|
|
楼主 |
发表于 2022-11-14 13:52:09
|
显示全部楼层
<?xml version="1.0"?>3 }, S; i* y! s
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
C9 k4 }1 q% M- D5 W, K0 ^<!--
3 N) g! A2 n. X8 A4 _' a( t Licensed to the Apache Software Foundation (ASF) under one or more
2 Q U% l( W$ g2 k" [ contributor license agreements. See the NOTICE file distributed with
9 S0 j4 S$ `& j6 t; D. Y* A7 ? this work for additional information regarding copyright ownership.' |5 I& z/ ^7 @7 d- |# r+ f# I. K
The ASF licenses this file to You under the Apache License, Version 2.08 T6 z/ g4 f. s1 W/ r
(the "License"); you may not use this file except in compliance with+ ?# Q$ r8 @& S$ F# y" z
the License. You may obtain a copy of the License at
5 B& \- q8 }0 L# Z. V http://www.apache.org/licenses/LICENSE-2.0/ I% }- H- e6 ^0 R `
Unless required by applicable law or agreed to in writing, software
- k; G$ z1 k/ r5 T* j9 ]' c& u+ O distributed under the License is distributed on an "AS IS" BASIS,
/ i$ b7 G7 ]" s) w' ? WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
3 V/ R1 N9 s) L, P8 x; L( o See the License for the specific language governing permissions and/ K |6 D2 P7 h3 |
limitations under the License.
' U2 C! Y' ^) y3 c; E( g6 h-->
H' m% R! E+ @/ G<!-- Do not modify this file directly. Instead, copy entries that you -->
6 |! Q9 }3 l5 n3 `. `' @% z<!-- wish to modify from this file into core-site.xml and change them -->
9 ?' w2 V8 h# e) ?' z( h( g<!-- there. If core-site.xml does not already exist, create it. -->/ [, M. K, k, q J% a! F
<configuration>% Q: w+ y; H" B. v
<!--- global properties -->6 E8 Q/ c" a' U7 J4 R
<property>
. p' f- w( d6 D% [ <name>hadoop.common.configuration.version</name>
8 g* R& Q8 C# N1 f+ C( \- L9 T <value>3.0.0</value>, T: Q2 |4 y5 w, K1 ?
<description>version of this configuration file</description>( u: e1 Q9 V: K1 k% _
</property>
: ?- W( A$ w3 ]<property>7 T# U8 L/ E# B
<name>hadoop.tmp.dir</name>
+ t6 T- M( `3 |: c4 K <value>/tmp/hadoop-${user.name}</value>
/ j8 i( i) k2 W( H C# K8 P <description>A base for other temporary directories.</description>
# K: ^# t R* j. u</property>% U% |! \' t% W
<property>
+ P! V8 c& A+ c# ` o# c* v5 ^8 v <name>hadoop.http.filter.initializers</name>$ [ @3 Q0 ^/ B' Z9 h. u
<value>org.apache.hadoop.http.lib.StaticUserWebFilter</value>! i( N) Y( c0 {+ Z% r/ @
<description>A comma separated list of class names. Each class in the list
( y) p0 `$ u$ q) q1 w must extend org.apache.hadoop.http.FilterInitializer. The corresponding
* j) c" l/ a* ~ Filter will be initialized. Then, the Filter will be applied to all user' O5 c' Q9 o$ _3 ?
facing jsp and servlet web pages. The ordering of the list defines the
( b) I4 \; T# s: A# R ordering of the filters.</description>
, A$ C. D- D" t) m9 t; ^% J: x</property>( A" _; P6 y& p6 N* B; o8 }+ `! ^. B
<!--- security properties -->* k) W; P. Z" \1 R
<property>7 X& k- y2 {! R: O4 _
<name>hadoop.security.authorization</name>; q4 E! o4 z0 M) ~" s. l
<value>false</value>
4 J, {5 N' _: v <description>Is service-level authorization enabled?</description>% x b4 V# Z4 b3 [, h
</property>7 y' p- M* ~( U! o3 I- q! j( L
<property>
; B9 z8 Y! `1 Q' K <name>hadoop.security.instrumentation.requires.admin</name>
7 X7 ^$ H, f s' M5 {( _0 O0 g* B <value>false</value>; ?6 [5 S$ ^* h, g8 b+ ?! c2 [
<description>
( n& L9 i4 j4 S3 t Indicates if administrator ACLs are required to access/ m6 j7 X* l7 }/ c9 F
instrumentation servlets (JMX, METRICS, CONF, STACKS).
8 P& O I2 Z3 a! m </description>
9 d4 _# t, k8 `</property>
: i! @! S2 Z* v( i. @! N% S$ N<property>6 f5 i; V' v6 I$ {
<name>hadoop.security.authentication</name>
4 M) {4 d9 b. ?( R, K$ ? <value>simple</value>6 ^2 S1 }" s, E
<description>Possible values are simple (no authentication), and kerberos/ ?3 z5 m# {0 N
</description>
8 D$ T8 s) q6 G, p0 Z</property>- `+ l' r( ~. j* o; ?% {. A
<property>
: k0 _6 R0 ~2 a; R) }: q9 f1 [2 f <name>hadoop.security.group.mapping</name>
9 O) L& l" ~! I8 T5 M: F, `; { <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>, ]! W( F& V3 e
<description>; ]7 ]* W1 X6 i+ w4 a2 y, S3 i
Class for user to group mapping (get groups for a given user) for ACL.8 H$ O& u" `1 X/ d
The default implementation,
$ c" V0 A: }* z5 U4 a, }7 q/ q org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback,
# {; C) N0 H6 c will determine if the Java Native Interface (JNI) is available. If JNI is
6 N* _6 ~" r6 M9 Z% {+ H( [# I available the implementation will use the API within hadoop to resolve a
/ L9 I1 S7 v0 N, i5 j2 g% y; [' L list of groups for a user. If JNI is not available then the shell. j& }# k* {& R4 Q I7 O
implementation, ShellBasedUnixGroupsMapping, is used. This implementation" B9 V, A3 e4 q1 _
shells out to the Linux/Unix environment with the
/ Q6 x( e! Z* C+ z& Q <code>bash -c groups</code> command to resolve a list of groups for a user.5 e+ H6 y( Q: k; q6 k, X
</description>
$ y2 K! t$ D) U( _</property>
z+ J# I3 \! g, {<property>
- A- M! y! l) V! t2 d <name>hadoop.security.dns.interface</name>8 V3 V4 P; `, i1 I3 i
<description>
7 v! G- _' W6 }+ L4 k6 [ The name of the Network Interface from which the service should determine
3 Y) ~; H# X4 ^* ] its host name for Kerberos login. e.g. eth2. In a multi-homed environment,( \7 S7 R& Y, b
the setting can be used to affect the _HOST substitution in the service7 @4 |& O* V! m( W2 m
Kerberos principal. If this configuration value is not set, the service" a# x. a, o0 T- o+ V* m
will use its default hostname as returned by
4 F% a- ]! D8 L InetAddress.getLocalHost().getCanonicalHostName().3 d3 Q9 f# m7 x5 F5 p
Most clusters will not require this setting.0 H' K& J: [0 e- }
</description>
* o' M5 S* d' M9 L' f: }: i& u</property>' [) u4 v. N! g2 v; N. l
<property>
) M/ v% y; Q6 W. K9 ^ <name>hadoop.security.dns.nameserver</name>( O- {: r: u# P( I0 E- u
<description>
; C+ d! ?) F0 F U$ \ The host name or IP address of the name server (DNS) which a service Node
$ o6 S- b7 t2 ^5 a# I0 `) z# y4 { should use to determine its own host name for Kerberos Login. Requires
7 ~7 o" f$ w: v$ f2 K. u hadoop.security.dns.interface.
; B- Y3 j0 }' y% c1 h! X Most clusters will not require this setting.2 Z: v- k5 O9 H6 `* F
</description>
5 p. d# e% [! r& J" ^5 D: d</property>
6 Q0 Q6 ]) V3 J. p1 i1 p<property>6 ?. F. `) J6 t) S; U1 F3 D
<name>hadoop.security.dns.log-slow-lookups.enabled</name>
% o! D |+ X$ r6 G0 l <value>false</value>$ C/ K/ O' F- y
<description>
" n, v! o* [, Q' f5 r Time name lookups (via SecurityUtil) and log them if they exceed the6 y: R5 ?4 k! @
configured threshold.
8 {/ `' s9 k) V </description>
$ L! A2 L+ X5 i6 B+ L</property>
3 m4 B$ k, w3 `& n* F9 j4 w<property>) q, d* c: v5 u7 `2 a! n. ^+ y- n2 V
<name>hadoop.security.dns.log-slow-lookups.threshold.ms</name>" g' e; K3 C5 O# S* W H9 i6 Z
<value>1000</value>0 n( z2 Y/ T' K
<description>1 j5 \/ t3 a8 M8 I
If slow lookup logging is enabled, this threshold is used to decide if a( m" P5 m7 O2 U: W1 C1 \
lookup is considered slow enough to be logged.
9 ^ c' F F( n# s( ^" D </description>
# m* R- p' L# N& h+ C: I+ e</property>5 @9 t/ p4 P! {0 B. O
<property>
2 }2 b6 X. a2 z( N+ l <name>hadoop.security.groups.cache.secs</name>
! f" I: v0 y- Q <value>300</value>0 E& R0 H) l( b
<description>% t1 K+ ^. w% t+ K
This is the config controlling the validity of the entries in the cache+ ?. a, A: O/ o6 Y; {' f; A2 c: d1 ?
containing the user->group mapping. When this duration has expired,5 Z1 Z% ^6 X" }7 e3 Z! s
then the implementation of the group mapping provider is invoked to get; G) ?$ g' E* Q7 p1 F
the groups of the user and then cached back.
' U7 z% l3 T: I0 e: u# F </description>
( _3 K, o% C5 ^& [/ L9 j5 ]" X</property>
& C1 E7 ~: i+ \ K5 l) s<property>( M! K; R& E# w" Z0 ?; _
<name>hadoop.security.groups.negative-cache.secs</name>) ?: C$ Z% I& C% A/ Q/ d
<value>30</value>1 }" `6 ~- A4 l2 C, P: m0 D5 B
<description>
2 H0 }. W, ?$ i- A" Z( P Expiration time for entries in the the negative user-to-group mapping
5 }, A: F! L. S% R1 \4 u caching, in seconds. This is useful when invalid users are retrying
, g, E! Z* R$ A; c% J frequently. It is suggested to set a small value for this expiration, since
- W v1 f) Q& I" w2 ?9 L) x a transient error in group lookup could temporarily lock out a legitimate
) U e" F6 W* s0 I& E user.5 r7 E" e% \! x2 K) }4 p) t
Set this to zero or negative value to disable negative user-to-group caching.0 w+ ~1 f5 Q: H1 e
</description>* V& E8 M H7 R/ C- S8 y, J0 d
</property>, _) M* W P G' N
<property>
' t2 a9 d2 N4 J; Z' X <name>hadoop.security.groups.cache.warn.after.ms</name>
5 L4 ?2 Q3 I7 O* O <value>5000</value>
" h8 C+ m2 c. `6 \2 {+ u2 \ <description>" A& F) V3 T; \% T
If looking up a single user to group takes longer than this amount of
. p% ^ Z1 E% G1 Y* q" f% U milliseconds, we will log a warning message.
3 _5 q' s6 w) Y5 W0 v9 A </description>( w3 v' Y! @. z0 z {* l& O
</property>
0 C6 g% ?- D$ e5 X<property> ~# I' D2 O( u7 l( B$ h0 n
<name>hadoop.security.groups.cache.background.reload</name>
5 l% G, G: T2 W2 g) P9 E! n <value>false</value>
. t" G/ X9 m* j$ E <description>) b; N( v0 p+ _& M. c3 ~
Whether to reload expired user->group mappings using a background thread
, V" m, \& q; y! X9 x pool. If set to true, a pool of& l$ H! E* m6 P- u6 |
hadoop.security.groups.cache.background.reload.threads is created to
' Z% y( a- k4 L# _9 _. w2 } update the cache in the background.
% g/ ?$ l9 W; x9 o; a7 e7 Z6 b3 `$ O A </description>
$ ]" J+ Y( k% A3 N* G9 B7 S" O$ D</property>
5 f" R3 T3 W ~% K% s2 ~<property>
, [* X. ~7 [5 }1 W( e0 D# V6 Z <name>hadoop.security.groups.cache.background.reload.threads</name>
3 V, T1 V2 p8 P. m: K& C+ H+ ] <value>3</value>0 {" r- V& V* V# e
<description>
6 _# ^" T1 Y+ \: t9 I Only relevant if hadoop.security.groups.cache.background.reload is true.9 A' Q4 Y( O0 F5 p! c
Controls the number of concurrent background user->group cache entry/ O" R* f9 v( i1 J L* b
refreshes. Pending refresh requests beyond this value are queued and! \# N2 {# W! ]& Z! m, ~7 U& G
processed when a thread is free.
- ?5 w4 A% H: o5 F) [ </description>" j3 y' j# E: H0 |. r* }& C8 ^
</property>* a0 }+ ]0 G/ w2 J5 H7 E
<property>) h1 b7 K4 B3 Q [" B/ @+ Q
<name>hadoop.security.groups.shell.command.timeout</name>* @5 U6 {* ~/ t7 g- h4 {. P
<value>0s</value>& h8 q( [6 R2 a7 t
<description># Y6 \7 L5 T- U& n8 D
Used by the ShellBasedUnixGroupsMapping class, this property controls how
6 r( n$ F. a* [$ l7 {% H long to wait for the underlying shell command that is run to fetch groups./ J2 P* s/ T& T x T. x
Expressed in seconds (e.g. 10s, 1m, etc.), if the running command takes
4 T+ I. R9 o( R longer than the value configured, the command is aborted and the groups
- _+ ~! R1 }; G- \: c2 [: P0 A resolver would return a result of no groups found. A value of 0s (default)5 Y% ~3 Q. |$ E3 j' n
would mean an infinite wait (i.e. wait until the command exits on its own).8 h8 y& o) |; j* C5 a2 b
</description>8 F; X6 x# }7 u& ^) n) }* d
</property>( f, Z( M" s% O; D1 U: F! }
<property>0 h1 K1 ^: P2 _$ T5 F+ t. X+ R1 R
<name>hadoop.security.group.mapping.ldap.connection.timeout.ms</name>
5 o8 Y: k0 b$ h" X" I0 Q <value>60000</value>1 P+ T5 F6 f4 N4 @4 `; a" |1 A* a
<description>5 z) [: E! f/ i9 S
This property is the connection timeout (in milliseconds) for LDAP
. }9 q. f% o" U3 ` ]0 G/ S operations. If the LDAP provider doesn't establish a connection within the3 d9 Q: T8 B- H# f
specified period, it will abort the connect attempt. Non-positive value- E8 y3 w- C0 {3 ?0 p
means no LDAP connection timeout is specified in which case it waits for the& J9 X' x6 W, A9 u
connection to establish until the underlying network times out.
" g; D9 [' l, t: Y </description>7 `' y) h) p6 O k" [
</property>
9 ^. f) A1 T+ [# _1 ~. r$ I, Z<property>
$ s/ t9 v3 c. _# v8 Y( _/ k <name>hadoop.security.group.mapping.ldap.read.timeout.ms</name>
5 n g. v! m$ W- W <value>60000</value>
, `( E" B1 a; r$ _7 _ <description>- X7 N) M1 i- m) C* G
This property is the read timeout (in milliseconds) for LDAP" T/ n! }* l" @' l, G$ G; G4 d# N
operations. If the LDAP provider doesn't get a LDAP response within the9 j! T+ I, ~% o9 f8 W* O% l, {
specified period, it will abort the read attempt. Non-positive value. R4 L' y' z5 i
means no read timeout is specified in which case it waits for the response! u. R+ {( G- B( N' @
infinitely.9 m7 s* ]9 j, q8 R9 g6 @
</description>* j# |' o9 J7 C: |+ y- N8 I
</property>
& e+ s: e% G+ z( U' S<property>3 r4 ?9 p/ H5 I: _
<name>hadoop.security.group.mapping.ldap.url</name>
/ b6 Z' p! X" s! D% N! X <value></value>
/ r' P8 V1 V! I: f1 s S <description>: K: x, d: ~, ^* N6 P* |
The URL of the LDAP server to use for resolving user groups when using
0 d' O6 j: _5 p" O+ |* A1 i6 S' f the LdapGroupsMapping user to group mapping.7 B0 I- E8 w. m6 A0 n
</description>% z+ Q3 @ @6 ~$ K; O/ r
</property>, V, b9 y* Q* w4 O4 K+ {6 s
<property>
5 n4 q& I7 z' \9 d3 c. ]+ N4 v <name>hadoop.security.group.mapping.ldap.ssl</name>9 Q( l# n, U/ e8 x6 J
<value>false</value>
6 e6 d# y7 p4 O4 g <description>
* R4 m. y' `( K7 @7 q Whether or not to use SSL when connecting to the LDAP server.
, X; H9 b7 w4 J% M- t r! z1 h </description>
4 B. }( E9 C9 m& f; g: k</property>" p+ A& [/ h( v8 U0 I* n
<property>+ p$ S3 u+ W: u8 ]- {) e
<name>hadoop.security.group.mapping.ldap.ssl.keystore</name>
4 u& t8 ]7 i* n$ i: ~ <value></value>
! N7 L8 t/ I7 n, r1 X! k" i <description>( K* ^) F N3 R8 p' M
File path to the SSL keystore that contains the SSL certificate required( t, }$ ]* h. J) @( r! q, o
by the LDAP server.9 C4 q( _: c: N6 k z X0 q
</description>) X, ~' F% v2 b0 ^+ d8 n2 V3 E
</property>) l' S2 |6 L4 ~$ L) {4 F( o0 L
<property>
+ k' L Z) a4 E* f1 Y) A <name>hadoop.security.group.mapping.ldap.ssl.keystore.password.file</name>2 \; ?- _0 C4 o4 G
<value></value>
* l: W/ [$ N8 ]. W, ^! t <description>
* ]# X) n' o! Y+ e, I* d4 Z O The path to a file containing the password of the LDAP SSL keystore. If
/ a: h- K: }* p% x% w the password is not configured in credential providers and the property4 u; w' D* T' R9 V5 u4 z
hadoop.security.group.mapping.ldap.ssl.keystore.password is not set,
" u& n5 j' d! O) M LDAPGroupsMapping reads password from the file.
, G0 c. C8 Z* m2 k% X0 a IMPORTANT: This file should be readable only by the Unix user running
+ s9 _1 P+ w$ r$ q the daemons and should be a local file.
0 _' m1 e( B' ?% ]% q- K </description>
& j) y8 a: c9 s- v8 w0 [* j</property>
A, P, b4 L' F# Q5 E y3 G) T<property>
, | b8 A- u: a) f( H+ g4 W2 d% [ <name>hadoop.security.group.mapping.ldap.ssl.keystore.password</name>
6 B* R h, E9 b' K6 a1 E. k, E <value></value>
4 j- V/ g7 N* s+ N( d1 X$ G <description>( H, p# U: e9 P! W- k5 b' i7 {
The password of the LDAP SSL keystore. this property name is used as an
# j5 |# h5 L- D3 Y3 m' j8 [ alias to get the password from credential providers. If the password can! U* O3 L h4 K- D
not be found and hadoop.security.credential.clear-text-fallback is true
* g- m' {/ @0 w; m5 Z. a4 ] LDAPGroupsMapping uses the value of this property for password.
( y) s: Q3 ?5 ` </description>
4 A( ?2 q. P, M! ]. k</property>" ?% V/ o* P. {" X; b% h3 }
<property>' o8 `; J8 |% o, @
<name>hadoop.security.group.mapping.ldap.conversion.rule</name>
; U: s7 L$ D* }% T f1 b9 k& z <value>none</value>8 J/ P8 |' x0 M' ?/ a" ?" u
<description> t% Z1 n8 C1 I* M
The rule is applied on the group names received from LDAP when7 G. n3 }3 J9 A
RuleBasedLdapGroupsMapping is configured.
1 Q) V3 |( x2 H4 y4 B; ^ C0 m Supported rules are "to_upper", "to_lower" and "none".
U1 k$ }; o8 o- K. n! s; s to_upper: This will convert all the group names to uppercase.. B; O7 T- A" \' u' d$ C- Q/ n7 l
to_lower: This will convert all the group names to lowercase.3 q8 y. E I( g5 R. }
none: This will retain the source formatting, this is default value.
' A- j5 m' r$ |4 R </description>( i& ~6 K6 A, n) F# j
</property>! M/ ]3 G6 j/ W. o: N7 N& j2 p' a
<property>: q9 i% l9 |, b4 q
<name>hadoop.security.credential.clear-text-fallback</name>
, y9 s( r5 D9 Q) k/ n3 _ <value>true</value>7 [2 }% L) l5 x* C
<description>
/ P8 R/ ]6 ?. a7 y# p) _: H1 {4 z5 a true or false to indicate whether or not to fall back to storing credential
7 \; ~, Y1 I$ F" t9 I password as clear text. The default value is true. This property only works
. W+ Q0 s1 R* t8 a/ F9 D+ L when the password can't not be found from credential providers.# n) N3 {, J7 J* w
</description>" T' @# v p0 M/ C
</property># z" W7 O9 E8 Z8 w3 @* k$ T$ }
<property>5 k; j* {+ P% T! k0 P
<name>hadoop.security.credential.provider.path</name>3 {2 S0 B+ g S9 o# F
<value></value>5 G. f2 a2 N" }+ \) E9 F6 o
<description>
* x3 Z) m: y3 ?( E A comma-separated list of URLs that indicates the type and6 @) |# H) Z! S! p# m7 M. d0 v
location of a list of providers that should be consulted.5 g" X+ l; ^& l' t3 G2 i0 c
</description>
9 o$ s8 C: H1 |1 p/ u. A a# y</property>2 ?5 ~" Q- Y5 X
<property>
. q% V1 U R. ^6 b; Q5 p* s& j <name>hadoop.security.credstore.java-keystore-provider.password-file</name>5 G& ?% v ^0 x( `0 O
<value></value>! O1 \" p/ O; V# H
<description>
& W. X+ W4 k$ k1 C The path to a file containing the custom password for all keystores
9 i' I& g! _! a3 Q5 d* H# r that may be configured in the provider path.+ T" \: \$ y; p8 H8 J
</description>& A! K z# B$ y+ F. S2 o" L, `, H4 `/ j
</property>0 h' }0 @. h4 j; `: b
<property>
1 V4 L8 i7 F% u" B <name>hadoop.security.group.mapping.ldap.ssl.truststore</name>3 q, X7 L9 O% E3 N' U/ G
<value></value>
" ?! E0 T: Q& m7 p <description>+ h+ f3 M, T# C. f
File path to the SSL truststore that contains the root certificate used to
+ ~& ^8 S4 q- _2 N: O sign the LDAP server's certificate. Specify this if the LDAP server's
) ^! D& R5 T# x! s! b' a1 z certificate is not signed by a well known certificate authority." C( {$ g8 E! t3 m5 Y- i: w( X, g
</description>5 Q7 U% v2 x2 _4 h/ o
</property>
2 |4 E& e- X4 q/ f<property>
( D4 ^. T5 B a8 p% [, n9 U <name>hadoop.security.group.mapping.ldap.ssl.truststore.password.file</name>* M- o- z- N0 E! i" O% k
<value></value>: w; N0 O1 ]+ }: ]+ l4 `6 X3 W/ a
<description>3 b& m V) U' a9 `+ w; _ I7 U
The path to a file containing the password of the LDAP SSL truststore.6 d, A, W' J5 L' Q3 Q' i8 j) M
IMPORTANT: This file should be readable only by the Unix user running/ x& b% Q/ ~+ c& \) p
the daemons.
, j: j' a4 ^: J7 [ </description>
# D% V, ]/ _) _/ X/ o2 {</property>, j; ]* l( F) w0 N! G
<property>
, S, ]# P1 I* q <name>hadoop.security.group.mapping.ldap.bind.user</name>
4 _2 t3 }( J2 ^: O7 ? <value></value>
I) M0 D6 a- m+ f9 u0 W9 ]6 b <description>
! M- _9 k; J1 W% y: n The distinguished name of the user to bind as when connecting to the LDAP
( R n5 q3 N) X2 t: n server. This may be left blank if the LDAP server supports anonymous binds.& L4 c( z/ S1 R: n M
</description>6 U7 j0 Z( ?4 t1 r; y M
</property>' B4 A! B. C. O
<property>
# c3 t- e& X/ o* j! l- O5 R& w <name>hadoop.security.group.mapping.ldap.bind.password.file</name>/ U- l1 U) S% H! G; ~
<value></value>
$ T- x" a5 z2 r" k) _ <description>5 R4 L# E, I/ h1 S% j
The path to a file containing the password of the bind user. If
- }1 S* k$ {. ^& L4 b3 _4 U the password is not configured in credential providers and the property
+ Y+ t" @$ T, a& H8 ` hadoop.security.group.mapping.ldap.bind.password is not set,
" _2 R& i& u* F3 f% D LDAPGroupsMapping reads password from the file.8 ]- O! \5 H3 V
IMPORTANT: This file should be readable only by the Unix user running
5 C' d6 s/ g+ n8 ~$ n the daemons and should be a local file.
. w; o9 Y1 w0 q4 s </description> ~1 d5 n3 H% o/ H. s e+ e% V
</property>$ V _- U4 P( b$ m7 o9 `
<property> p+ |$ P& N# y3 D9 P
<name>hadoop.security.group.mapping.ldap.bind.password</name>. c* q2 O1 W5 f- Q5 E
<value></value>" s2 N: f( |6 F- ^
<description>
$ ]% e+ L" U) B+ x. K/ k4 I5 ] The password of the bind user. this property name is used as an1 w7 X7 }/ w) w0 n& t
alias to get the password from credential providers. If the password can
% _' P" B$ ?: r% F ~* T not be found and hadoop.security.credential.clear-text-fallback is true2 ]4 m0 w( p: A* b& ?
LDAPGroupsMapping uses the value of this property for password.
" |5 D( G$ p/ D0 R' b6 P9 D, b </description>: B) j" B2 d1 j, n. T* v+ M* n, i5 x
</property>5 x4 J5 T- O# h9 h+ F, V9 B
<property>
3 O, F' q' ^3 q2 ^" T) I3 Q, ^ <name>hadoop.security.group.mapping.ldap.base</name>* V5 b( s4 q' t0 J% @
<value></value>
M, X0 Z/ K: f* ?2 L <description>' z4 p/ S+ E0 f* f' ]6 N' u) }
The search base for the LDAP connection. This is a distinguished name,
) X u5 ?4 d+ g2 F1 j and will typically be the root of the LDAP directory.
t7 f* O; x' Q+ F/ m </description>
- Y) A6 x! R; q7 U# w</property>
1 B' G# F2 q6 ]& G) L; u) E<property>
' z, j4 t5 E; k' W9 c# p <name>hadoop.security.group.mapping.ldap.userbase</name>
7 }# f/ |- R6 S0 V; `6 D! f a <value></value>$ E% I8 w) n; c. T8 x4 p; c0 Y
<description>
; \* z9 z1 A4 a5 Q8 s8 C7 I' T5 } The search base for the LDAP connection for user search query. This is a- H& U0 A6 G5 M: }% M2 L+ [
distinguished name, and its the root of the LDAP directory for users.* @, z' h3 ^' g, ]# v
If not set, hadoop.security.group.mapping.ldap.base is used.6 |" J- P* U `: C' o& o' B
</description>
# [0 c% i1 {2 T6 r; R, ]</property>
6 R1 C5 N6 K5 X+ |<property>6 P# z% Y+ }: ]
<name>hadoop.security.group.mapping.ldap.groupbase</name>
( i3 ^9 G' h3 ]) M <value></value>3 Y# | O8 [5 U Q; J* O
<description>
2 f9 M B4 w7 q9 u The search base for the LDAP connection for group search . This is a
/ s) O( ^5 V) E0 {- d/ p2 y o- P distinguished name, and its the root of the LDAP directory for groups.9 Y5 Z1 d- Q5 L" Y" l" x
If not set, hadoop.security.group.mapping.ldap.base is used.
2 m- k# N, ?# x: S5 l3 F2 b5 ~ </description>
4 b. f$ ~( f* t</property>! ~% T1 w% {; M% k
<property>/ W, B- M# q" i g2 P I
<name>hadoop.security.group.mapping.ldap.search.filter.user</name>+ t+ e: o' P' G V
<value>(&(objectClass=user)(sAMAccountName={0}))</value>
9 v5 t! I i6 {/ @- {5 | <description>
2 H$ [5 l; G/ n! R+ z1 e An additional filter to use when searching for LDAP users. The default will- G( g v0 B+ O4 [9 U3 J
usually be appropriate for Active Directory installations. If connecting to+ ^( Z5 F7 P; u% y2 Y6 w
an LDAP server with a non-AD schema, this should be replaced with v4 k% U) r, Q# E u4 k
(&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to
% g; B7 {7 u5 X; x. M. ] denote where the username fits into the filter. ^9 {! c/ M r9 S6 q# a4 Z$ S" S
If the LDAP server supports posixGroups, Hadoop can enable the feature by% Z( A; P, j& ?7 U
setting the value of this property to "posixAccount" and the value of
* c$ ~6 a/ K. \2 V) B' o/ ^ the hadoop.security.group.mapping.ldap.search.filter.group property to
3 q i( d1 L3 Z "posixGroup".( H# E, E2 g9 H( g, k9 r
</description>6 ?1 w7 h+ s2 W" y
</property>
5 a. |3 D, o J* w" d/ r<property>
. t, [( L8 b* z# e0 M Q. i <name>hadoop.security.group.mapping.ldap.search.filter.group</name>6 C! Y& r: Q* m' f' A5 D
<value>(objectClass=group)</value>4 L/ ?6 x) y! E$ c, o7 g
<description>. m6 B6 Z$ R8 |% @5 k) A
An additional filter to use when searching for LDAP groups. This should be
, Q5 Q, d( |, P$ S) V changed when resolving groups against a non-Active Directory installation. x0 m5 `" I4 t. I. k
See the description of hadoop.security.group.mapping.ldap.search.filter.user5 f, w8 L4 v/ o$ H& `
to enable posixGroups support.* A( T2 w- j$ Z, d
</description>
" y6 B1 J6 e y. D' e8 N+ b</property>
9 U3 i+ i7 H$ w' x<property># o7 P1 M' ~. N% a
<name>hadoop.security.group.mapping.ldap.search.attr.memberof</name>: K; U' l! d. c1 D) o$ |
<value></value>
7 ^9 Z u b; Q' B# g. o <description>
+ l, c8 O5 B n The attribute of the user object that identifies its group objects. By
5 `* i( q% [% |/ x* } default, Hadoop makes two LDAP queries per user if this value is empty. If5 Z# K8 d. o6 w; ~7 ^- i) v _+ }5 j
set, Hadoop will attempt to resolve group names from this attribute,0 M3 x& \3 x8 @7 H% t @7 C) M. G
instead of making the second LDAP query to get group objects. The value
8 U2 s! N( I" b2 S should be 'memberOf' for an MS AD installation.
* J; E' d/ T6 |& H& l </description>
}8 [$ N. k3 `+ \) q7 s: m6 M$ o</property>
+ f' q: X0 ^1 }% O2 T! h<property>5 G# N8 e9 I$ y/ w6 K
<name>hadoop.security.group.mapping.ldap.search.attr.member</name>- P# F6 V+ H4 M6 F5 n: z
<value>member</value>9 ` r3 O; q4 |# [
<description>5 ^% i' N+ ?9 F- r; Y
The attribute of the group object that identifies the users that are6 ~: p4 u% E. k& Z6 F7 q) r9 o
members of the group. The default will usually be appropriate for
0 R# I4 r* [$ b2 v6 R% ] o$ w6 r7 K any LDAP installation.
5 ^( b6 [5 d+ ?- R </description>
! s) o+ u/ i# I) L- C' L</property>& R6 `) _, D6 t5 C
<property>2 H- J% c5 t1 k8 z) [4 ^
<name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>2 u! ]- i( U7 ?" {- Q/ R+ D- V1 E
<value>cn</value>
+ c0 G) L+ f, o; ] <description>1 B$ _: b- r7 a$ f& w3 B
The attribute of the group object that identifies the group name. The
7 e0 d% c E" \7 K8 Y+ m3 y default will usually be appropriate for all LDAP systems.
6 R- V7 ^. `8 r* ] </description>
' j; {8 {% I) W0 G9 z4 V) S</property>6 b, Z- B% {. X9 o" A4 [
<property>
* U; V# d& B; r3 j6 |0 R <name>hadoop.security.group.mapping.ldap.search.group.hierarchy.levels</name>
" |* a+ [+ f* X N% R( I. P) R <value>0</value>5 U& l0 X, w4 j% N; D8 Y H! K
<description>
: h+ a0 N, l- ^# m) ^% ? The number of levels to go up the group hierarchy when determining+ K) K% a$ z6 S6 S
which groups a user is part of. 0 Will represent checking just the
0 u& V- R) J% I( f7 {- ? group that the user belongs to. Each additional level will raise the
; t+ R! `6 P T4 k% v6 @ time it takes to execute a query by at most
. r" x* W f0 Z0 z2 U hadoop.security.group.mapping.ldap.directory.search.timeout.1 N- A5 q4 @% ^
The default will usually be appropriate for all LDAP systems.
" m* [, b1 o9 E X6 v </description>% S/ o8 q1 x/ u, [
</property>/ N8 Y6 I. K9 h
<property>
+ L4 U% m) q' W+ ?( f; g- h5 u <name>hadoop.security.group.mapping.ldap.posix.attr.uid.name</name>; ~& l+ k8 n$ L$ d
<value>uidNumber</value>
, o" `, x7 U" B <description>9 a2 S& K- P( N! O2 H5 N) e0 ~
The attribute of posixAccount to use when groups for membership.% h5 n; X2 O2 ~3 c
Mostly useful for schemas wherein groups have memberUids that use an+ E1 v. p7 h o( P) _
attribute other than uidNumber.
( G5 x0 L+ J K- p9 x5 D) R9 T4 T </description>
# x6 m; q* Q" B j</property>0 G6 t& c3 S, F; s5 n' V
<property>
6 v/ {1 q( x5 j6 \ A3 c <name>hadoop.security.group.mapping.ldap.posix.attr.gid.name</name>. T9 s+ e! ?! l W' x/ M2 w* ?
<value>gidNumber</value>9 t; ?3 y9 u! L+ O& C5 U
<description>1 F, z5 I9 R1 z l
The attribute of posixAccount indicating the group id.
0 O) K. O$ g' ?; r' Y/ r, S2 I1 u </description>' V1 f: ?1 Y* ]) `. K$ E# N; d8 H
</property>
% l: Y* V4 a' Q/ o<property>: }! q! s; X/ i# Z' C
<name>hadoop.security.group.mapping.ldap.directory.search.timeout</name>
# D7 t) i) g4 c' | <value>10000</value>% a4 r4 V7 t3 X& b: k( Y
<description>
& W1 Q$ [! D' i( e; M! j( U3 E1 x The attribute applied to the LDAP SearchControl properties to set a
* `3 `$ q# c. H maximum time limit when searching and awaiting a result.+ H. h" r" z- u' e5 ]% L5 E1 |
Set to 0 if infinite wait period is desired.: M# p8 L' H+ l( x& R
Default is 10 seconds. Units in milliseconds.$ B" G ` l8 m- S2 U
</description>9 A* j4 T8 [0 V0 r
</property>* d; N; S% D& o' {
<property>0 W8 B2 }* A- U* W- H( f) T8 u) q/ }
<name>hadoop.security.group.mapping.providers</name>
- M" D4 D: X3 V- O <value></value>3 j3 Q5 p6 P& T- Y9 U
<description>
( r" N1 y1 V( z" [9 p/ s3 ` Comma separated of names of other providers to provide user to group% L8 f! R* A0 } ~: C1 j$ v
mapping. Used by CompositeGroupsMapping.
- l! z @4 |0 [$ G9 @ </description>1 `5 Q5 L5 O0 ^' s7 y
</property>
+ Y$ A' ^7 o7 S: C, z<property>
9 n8 h0 N3 G3 g# _" N3 z <name>hadoop.security.group.mapping.providers.combined</name>
3 O1 Y: S+ J; t5 J* @' j <value>true</value>5 c. Q, ~# @* Z* n. q
<description>
" b) _& a4 m8 c) H7 K5 w true or false to indicate whether groups from the providers are combined or
# E H4 p; X- X not. The default value is true. If true, then all the providers will be
. x8 Y5 p. l6 ]* q i. J4 d8 ~ tried to get groups and all the groups are combined to return as the final
: H1 V: Y# f$ k* D1 j results. Otherwise, providers are tried one by one in the configured list8 f [, p, R7 p& H' A+ S
order, and if any groups are retrieved from any provider, then the groups5 R J' p' Z9 n
will be returned without trying the left ones.5 L* z% n! _7 M: F8 x/ N4 ~" S
</description>5 \& s6 ]) |, ?$ C& S/ |
</property># I& H- z5 x6 h0 {) m+ K+ U
<property>
' Y# J8 `( q7 N5 n# K <name>hadoop.security.service.user.name.key</name># h0 h9 `4 b! t' {+ U4 [
<value></value>
6 m; Z4 @' ?& w4 G <description>; m, q" w; P& J$ h9 D8 S
For those cases where the same RPC protocol is implemented by multiple
1 Q" p* E4 L6 a7 }$ H8 V servers, this configuration is required for specifying the principal& c* n& L7 r4 Z% ^. [: l
name to use for the service when the client wishes to make an RPC call.' z) Y1 A3 t+ y. z
</description>
1 O0 a5 @/ R! \& g* F( w4 [$ L: Z</property>
Q: C0 e/ d- A0 r) E& q. ^ <property>
3 @0 {# n) ?/ s <name>fs.azure.user.agent.prefix</name>
4 r' l+ N2 H9 R$ U4 F( h8 t <value>unknown</value>
, H# X( b. P+ z, U+ p& e- k <description>( p9 X, V3 F$ V s
WASB passes User-Agent header to the Azure back-end. The default value
% ~0 v$ V# G7 Y0 G9 z2 T* A' A contains WASB version, Java Runtime version, Azure Client library version,. r% f1 t* i `& R, B% T$ u% I% P: {
and the value of the configuration option fs.azure.user.agent.prefix.9 D* f/ c# T' i9 F: Z
</description>( ?. R6 }9 {+ I t. K$ a* s5 c
</property>7 w" X7 h3 r, w4 v% e
<property>
' x+ {3 @+ T6 j$ V, P( } <name>hadoop.security.uid.cache.secs</name>
5 M& ?" S' j: ^) T <value>14400</value>. D+ n' ~ J/ W0 @. l
<description>
4 X2 v7 u5 ?' a7 D1 g This is the config controlling the validity of the entries in the cache. ^( c* O" g& h& N) p& m' K# u( d! e$ l
containing the userId to userName and groupId to groupName used by
0 }* G" S7 f. K# `5 [/ `( k1 i2 H NativeIO getFstat().. I6 }; I; R8 ^* ]% l$ g
</description>
3 ?9 o& y4 ?/ d</property>
5 e3 ?7 u8 k9 I( K4 [<property>7 x/ O) L6 w# V4 z1 @
<name>hadoop.rpc.protection</name>
) b+ ]* C9 W# p; h/ m0 F1 N <value>authentication</value>1 K. D- L/ W# @1 P2 \
<description>A comma-separated list of protection values for secured sasl
2 b3 {" n& A( a9 B, e) R. c connections. Possible values are authentication, integrity and privacy./ w3 i& T* J5 a9 n' e" y+ x1 p% I
authentication means authentication only and no integrity or privacy;
) z4 s8 V: c/ j integrity implies authentication and integrity are enabled; and privacy
1 D2 k, F$ ^* s- ~3 \& M7 j' \ implies all of authentication, integrity and privacy are enabled.
+ _! `3 O% u. }( W4 P; [ hadoop.security.saslproperties.resolver.class can be used to override
% m$ P) n$ ?) m5 b$ E the hadoop.rpc.protection for a connection at the server side.2 p/ O& {+ V8 j% d- f
</description>
- l# }! h8 s8 b1 k U</property>* N4 Z8 \( {6 H6 O5 T
<property># y# L l4 Q3 Y+ C
<name>hadoop.security.saslproperties.resolver.class</name>
; X3 m9 ?) V- i: O1 D <value></value>
& n" C( E" B; e% A& K <description>SaslPropertiesResolver used to resolve the QOP used for a
: I3 U) \1 w2 d7 j7 o2 W. G connection. If not specified, the full set of values specified in; P# N- s2 F$ u+ o7 ~
hadoop.rpc.protection is used while determining the QOP used for the E9 i9 J' e0 [9 u
connection. If a class is specified, then the QOP values returned by
# Y) v8 k! }' h3 K the class will be used while determining the QOP used for the connection., y0 R( L1 j: | a% q* I
</description>8 B* B7 m) T4 v/ T! y4 h2 A4 Z ^
</property>
$ t6 Z L9 l, }$ _<property>8 a4 S5 r% T5 z; X! b% p) F
<name>hadoop.security.sensitive-config-keys</name>- G6 w6 r5 v0 j0 h/ p
<value>
3 i8 `! W6 m5 ]6 E& q secret$! l. L) k5 K L3 O0 h* _
password$$ Q% t5 O6 K1 K; _( X9 y
ssl.keystore.pass$
9 d( O5 Y4 e) ` fs.s3.*[Ss]ecret.?[Kk]ey% e- v0 C. Y* k0 G2 W* a1 Y
fs.s3a.*.server-side-encryption.key
/ ]" k; B: w! _( ^& O# ~ fs.azure.account.key.*5 T/ `: v; M/ w+ Y" c
credential$2 U4 G# k' v H1 U5 O' J( M- g7 i
oauth.*token$. {9 G3 Y7 F: E
hadoop.security.sensitive-config-keys8 N* d- g6 r% y" C
</value>
1 S9 e+ @) F& d, g <description>A comma-separated or multi-line list of regular expressions to) Y6 K6 e7 L" Q' y: O( | P( @
match configuration keys that should be redacted where appropriate, for
/ Q) U& L8 J0 f' M7 a) X8 Z, s example, when logging modified properties during a reconfiguration, F# l7 y3 e* \# Q9 f
private credentials should not be logged.
: L! q/ B2 d0 [; d3 l8 a </description>- Y& J g3 m: |
</property>
+ F9 K7 G5 L. e( U<property>
' A0 Z6 e, Z' c <name>hadoop.workaround.non.threadsafe.getpwuid</name>( \! u$ \9 C0 p" O! ~7 Y
<value>true</value>+ M, G& ^( g z$ e0 H0 e$ v
<description>Some operating systems or authentication modules are known to5 n9 x3 U" V1 Y+ [( u5 v4 \# c, l" j
have broken implementations of getpwuid_r and getpwgid_r, such that these
, Q1 q& F5 @5 q9 v7 j calls are not thread-safe. Symptoms of this problem include JVM crashes6 P1 K4 z* i& w7 \3 D+ C
with a stack trace inside these functions. If your system exhibits this S( W3 A) t, @, S4 d- N
issue, enable this configuration parameter to include a lock around the
9 j! s( h* I+ i" p calls as a workaround.. m) K6 c" e7 X7 W# g$ n( |
An incomplete list of some systems known to have this issue is available
' S0 ~0 Z5 c* E8 k) d at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations
+ @" Y8 M" ]1 I) D6 v* ] </description>
! r" W# |! ?% r+ N( A0 J</property>% o# e3 l! z4 R% H4 f+ _
<property>! t0 ~ U7 K1 k& a4 S9 D
<name>hadoop.kerberos.kinit.command</name>
- J3 { J' K- f, ^: F <value>kinit</value>
6 ^7 ^/ A( @- O <description>Used to periodically renew Kerberos credentials when provided6 V6 K9 H' U5 @; Z
to Hadoop. The default setting assumes that kinit is in the PATH of users3 p8 A' a+ j" |
running the Hadoop client. Change this to the absolute path to kinit if this. j' g, {. y! ?* C2 h( M
is not the case.& [/ V+ ]1 N: J" h
</description>& j+ V. n( p; x2 ]1 ~$ _
</property>4 Q+ g& y' {( f, d6 V$ \( t
<property>7 [7 }; \, ]+ C; F* q
<name>hadoop.kerberos.min.seconds.before.relogin</name>! Q/ n0 X" ?' M* f/ x: ^' f
<value>60</value>8 `# a. n# w. R. |- {1 b
<description>The minimum time between relogin attempts for Kerberos, in% L9 ~5 m% S9 k3 |& R0 ~2 N
seconds.
: n" y J2 m0 ~7 b$ j </description>( S& W y: n2 a
</property>
7 {& _2 Z/ Z* m<property>2 Y, a' D+ y, n
<name>hadoop.security.auth_to_local</name>
/ m+ a \2 G, ]' D: I1 n: Q <value></value>9 Z2 E# T1 B# h
<description>Maps kerberos principals to local user names</description>
& u# Z o9 p- f. Z U' @</property>
' I( e- W, b4 F0 e; y2 R<property>
L* E/ Z) U- W6 ~6 E+ z <name>hadoop.token.files</name>
* G. v; U8 D% S, |! m& b <value></value> {5 C: U8 G" C) S+ _
<description>List of token cache files that have delegation tokens for hadoop service</description>4 w& N$ k" e$ G% \- M9 v
</property>
% r0 w) s& w7 p$ V! {! Z; U<!-- i/o properties -->
$ ^' Z6 w; g. n9 ]# j<property>4 B6 G* j. T( i( k) C7 ^! `3 N5 X
<name>io.file.buffer.size</name>
5 M6 S! W7 K9 b* ]" W5 C) s <value>4096</value> X0 H9 K/ L0 p
<description>The size of buffer for use in sequence files.* S! J% p3 H- S( Y: w( G
The size of this buffer should probably be a multiple of hardware
/ e) }# s# a2 j1 i) r+ }- D page size (4096 on Intel x86), and it determines how much data is
( W' q- l2 s( z buffered during read and write operations.</description>
/ i. D9 V( ^$ Y</property>
! h/ _- h7 |3 u<property>- m& g0 \: A: _
<name>io.bytes.per.checksum</name>$ A9 F: \3 c* U& g) a: j6 p s
<value>512</value>; @6 M2 Y) P _; R
<description>The number of bytes per checksum. Must not be larger than6 A* O# P4 Z, Y) q# [0 E
io.file.buffer.size.</description> `& G, ~6 c0 i: h
</property>
6 L0 J* y* q1 y% \2 d. [<property>( u% U7 R" s5 K; T
<name>io.skip.checksum.errors</name>, }+ r/ ^- p1 F
<value>false</value>
# G* B( ~$ w3 J( r4 C* Y <description>If true, when a checksum error is encountered while7 j6 n4 P' v+ X. O
reading a sequence file, entries are skipped, instead of throwing an
( D, Q( v& f5 j* P exception.</description>
9 ^0 N: k: P( r" x# h</property>
+ w4 f7 E8 a1 l<property>
3 m2 i( ~' ]( `7 g$ z1 {3 j <name>io.compression.codecs</name>
9 Z; b# M6 i1 v9 r6 y' m7 H <value></value>! M7 A+ h7 ~8 \7 V0 f! Z/ {
<description>A comma-separated list of the compression codec classes that can/ I; t' R8 P. p
be used for compression/decompression. In addition to any classes specified
* M. Z6 D4 v' Z9 C$ N# N with this property (which take precedence), codec classes on the classpath# g% W# V; i' p+ D
are discovered using a Java ServiceLoader.</description>
/ Y& B" q N( A' x4 }$ a</property>& S! @4 S6 G' c) _" K
<property>. N/ K3 I( v! O% k! Q# F+ _
<name>io.compression.codec.bzip2.library</name>
: b: T' J3 Y2 o1 U <value>system-native</value>
+ u# A" k: K) O+ V* S <description>The native-code library to be used for compression and
* P8 g7 L6 p0 W) D/ N7 i decompression by the bzip2 codec. This library could be specified
7 d4 t3 o6 U# ]+ H+ H either by by name or the full pathname. In the former case, the
% L+ T; a0 |$ H4 W% V library is located by the dynamic linker, usually searching the
1 D1 k9 ?. x9 ?* o directories specified in the environment variable LD_LIBRARY_PATH.) G Y. X5 d/ T' ]. ?
The value of "system-native" indicates that the default system
, k5 `8 g) U, Q$ w) e3 r6 g library should be used. To indicate that the algorithm should
- g4 R' f Z$ L5 t operate entirely in Java, specify "java-builtin".</description>
9 a; |, ~8 _) S# G</property>+ T% Z* C0 V* G) B
<property>2 ]% ]0 K) V& m* p! C
<name>io.serializations</name>6 Z+ d3 W# K5 G, |) [+ B
<value>org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization</value>
. S2 p; H, o' G5 o) A: _$ ` <description>A list of serialization classes that can be used for9 M/ L8 j7 h8 r j1 J
obtaining serializers and deserializers.</description>7 P6 ], W0 t( q- D. H
</property>
( p! H: i+ N1 v<property>& ]/ B) P2 [3 z/ d8 }! b- _
<name>io.seqfile.local.dir</name>
7 H+ |; i" t. d2 H# s$ \ <value>${hadoop.tmp.dir}/io/local</value>- v Q, M5 m$ X' v1 u
<description>The local directory where sequence file stores intermediate( P* H# l. [2 K) g+ s4 L0 K0 e
data files during merge. May be a comma-separated list of. ?$ v4 q" S: \& o. ]6 N9 [
directories on different devices in order to spread disk i/o.
1 R- O& s0 h. T Directories that do not exist are ignored.& w; S% J4 ^9 F5 s8 u
</description>
% s* g+ K( H4 f, B7 m</property>
" _' ~$ d; H( U C4 r/ ]<property>
9 S; s E) R' D, L' X <name>io.map.index.skip</name>% E) p, |" m, `, n8 O) C0 N, D
<value>0</value>
' O/ o6 ^' z' s) \4 N* f <description>Number of index entries to skip between each entry.5 Q& D* R/ F- v; j# v4 b
Zero by default. Setting this to values larger than zero can" y! D* l: W" F1 H
facilitate opening large MapFiles using less memory.</description>
% y0 j) `! e9 r% i1 I: p% U</property>/ T# R' r% ]7 @+ k$ T1 [
<property>, l9 k5 j4 q4 T$ U/ ^2 j0 V# {1 \
<name>io.map.index.interval</name>
' ], V6 S4 _- L <value>128</value>+ a" e1 _! Z& M
<description>
7 P# P' @* D$ S5 y$ J# t MapFile consist of two files - data file (tuples) and index file! R! Q: L* W7 i$ L
(keys). For every io.map.index.interval records written in the
- F/ k9 g) u' ^, d$ } data file, an entry (record-key, data-file-position) is written
$ j* f: S2 H9 o) L) a in the index file. This is to allow for doing binary search later" b3 @0 Z. A5 L# q
within the index file to look up records by their keys and get their& z3 M" t: R/ Q% g7 C/ w
closest positions in the data file.
+ ?0 I! q5 y0 @4 K t" K </description>
, F6 O, B/ O. z$ h- ?</property>
2 P/ z& Z J$ x<property>
1 {' p% U' l8 b6 h" X <name>io.erasurecode.codec.rs.rawcoders</name>- E' y- T8 W: z2 W' t
<value>rs_native,rs_java</value>- a }4 _/ C+ H* x2 W2 X
<description>+ T$ J( }2 f" X
Comma separated raw coder implementations for the rs codec. The earlier
6 m/ x( p1 t8 W) l! I factory is prior to followings in case of failure of creating raw coders.
4 ^: Q4 i6 V, j8 U' D0 Y </description>4 z8 G R4 @% [' ~# { f
</property>7 ^' `& Y$ {- i
<property>
( [: ~5 I3 L* r/ s <name>io.erasurecode.codec.rs-legacy.rawcoders</name>
, [+ y+ v: O8 U3 M <value>rs-legacy_java</value>
; c7 x1 Q7 g: V6 B$ h <description>
6 ~ R4 F: e: q" h% D. Z: ~" n Comma separated raw coder implementations for the rs-legacy codec. The earlier* S% R* M. V( V7 j, W5 w' X
factory is prior to followings in case of failure of creating raw coders.2 g; R! ~" a% A2 x, ~
</description>0 d9 V# g. `& ^9 c j3 i( W( q
</property>& x7 \2 Q; J4 v3 V3 F1 l
<property># z0 y6 M7 u! B
<name>io.erasurecode.codec.xor.rawcoders</name>2 b: m* e2 z: L( n% ?
<value>xor_native,xor_java</value>
8 P* ^% ^% r% D, g <description>5 e; |: u2 u. Y
Comma separated raw coder implementations for the xor codec. The earlier2 O: c3 a0 {. [1 G5 c# c% @
factory is prior to followings in case of failure of creating raw coders.
1 l( x3 F( K3 [/ } </description>
3 N1 t. E5 d) T1 g</property>2 G; j# ^% i# Q/ g) q) O
<!-- file system properties -->2 Y" R2 H) ]; O3 `! r& T
<property>+ Y1 {/ J2 r4 I- T- i
<name>fs.defaultFS</name>
" F6 q. `& z3 X# i, W5 b <value>file:///</value>
. j$ R1 u; a6 W' l <description>The name of the default file system. A URI whose; D% _% [' q/ R1 w6 @1 ]( i
scheme and authority determine the FileSystem implementation. The
1 d% [# s( ]5 D1 K uri's scheme determines the config property (fs.SCHEME.impl) naming# Q' } P- `9 P" ~) g
the FileSystem implementation class. The uri's authority is used to( N5 V, s3 I. h. l% H- E7 p" \5 U
determine the host, port, etc. for a filesystem.</description>
| ` d: F0 c8 e S7 i</property>
! h# H# x( r; e! g% g$ Z9 v<property>4 Y8 Z9 g; D* D! X/ k1 c* I5 Y
<name>fs.default.name</name>) i; F8 o1 y6 W* H( I0 p" c
<value>file:///</value>
. k3 h! z2 m5 W+ a <description>Deprecated. Use (fs.defaultFS) property
% z+ K7 `6 |7 N instead</description>1 g% V3 J1 v. q. L
</property>
2 E( v8 ` r; C( T% ~. _<property>
8 K0 o- ]: w- @+ n7 ~8 M! Y <name>fs.trash.interval</name>
, b6 z' Z4 ?' M5 { <value>0</value>
7 _2 n+ Q ~; m- d7 s <description>Number of minutes after which the checkpoint
* Q- `& R+ j$ }" s/ Q4 j/ L gets deleted. If zero, the trash feature is disabled.5 q+ S7 Q0 o( H; I
This option may be configured both on the server and the
# U3 q7 d3 |5 _, S+ t5 _ client. If trash is disabled server side then the client1 A4 H1 [# H5 p* t
side configuration is checked. If trash is enabled on the$ s6 Q7 I$ ?3 |7 z: D6 @
server side then the value configured on the server is( e c0 r3 x2 |+ O
used and the client configuration value is ignored.
, O A+ _# K* S0 N5 `* W6 j </description>
4 |$ I8 `# R/ q2 u+ b6 t</property>% X# _& n3 @5 `; e4 c! V- C
<property>
/ W* l w J( ~( M: Y4 k$ I) q( ^ <name>fs.trash.checkpoint.interval</name>
, L0 d* n% b4 f- ?4 ?3 N <value>0</value>
. ]( a2 f4 T+ ?) y( D <description>Number of minutes between trash checkpoints.
+ A1 F/ \ n; `+ P! m9 A Should be smaller or equal to fs.trash.interval. If zero,
# y# L( }! |5 M the value is set to the value of fs.trash.interval.
+ {# d) S8 l4 R3 O0 X$ \6 e" b0 I) b Every time the checkpointer runs it creates a new checkpoint! H, t7 k) `: H
out of current and removes checkpoints created more than
% Q, L! b3 Q4 t( b9 @+ i fs.trash.interval minutes ago.( U, o' N3 z" c5 b
</description>$ `( v9 r x) }- e: E' Z
</property>
6 n; t+ @4 \' f2 l9 A0 Q$ R<property>
3 L* |/ j$ d# z4 Y) Z: \ <name>fs.protected.directories</name>5 f7 H% H# C3 n) x# o
<value></value>
N* F& F+ M c7 O6 }6 } <description>A comma-separated list of directories which cannot
% C) [: R' r0 J be deleted even by the superuser unless they are empty. This
5 J9 _( _. C3 }1 p+ |$ N setting can be used to guard important system directories# d( [7 X4 F- y% v
against accidental deletion due to administrator error.
8 y. u8 a- n6 ], q- p </description>
, u) M- q H% b! J& B</property>% V) `( P: c9 A2 v2 {# y1 i
<property># Q0 Y# T: `+ q2 z3 o# C7 a& a5 |
<name>fs.AbstractFileSystem.file.impl</name>( L1 E0 A9 z4 @$ y
<value>org.apache.hadoop.fs.local.LocalFs</value>& j& b+ w8 y' l- t0 k: u
<description>The AbstractFileSystem for file: uris.</description>3 g2 M5 }; w5 |0 }. ~9 P& Q
</property>7 P9 q. x, v( s) u8 ]; g1 O
<property>5 z4 w5 I5 c+ |& e: K$ X8 w- z
<name>fs.AbstractFileSystem.har.impl</name>
$ Q3 }3 x- q! j <value>org.apache.hadoop.fs.HarFs</value>
& l+ o4 t% [* S4 s2 R <description>The AbstractFileSystem for har: uris.</description>' G, E" ^4 P/ [( [7 p! ^
</property>
* h1 r: g+ h+ T9 x<property>
7 J+ f/ q2 b2 K4 _2 M1 } <name>fs.AbstractFileSystem.hdfs.impl</name>
4 \- [5 i3 i! n <value>org.apache.hadoop.fs.Hdfs</value>$ s7 Z4 ^6 |( \7 W6 _* ~( x6 z
<description>The FileSystem for hdfs: uris.</description>
6 v( p3 T( ~, m$ }5 d/ e) V- C4 f</property>* [4 _$ C' X% l% K* f4 t
<property>4 x9 Y4 ]3 F: i* I) B9 N
<name>fs.AbstractFileSystem.viewfs.impl</name>' G* U( M+ b! n {% f L7 j
<value>org.apache.hadoop.fs.viewfs.ViewFs</value>
' r, k; H: _5 C/ x1 C) [ <description>The AbstractFileSystem for view file system for viewfs: uris& R ?! s! n# o
(ie client side mount table:).</description>
) g8 D1 v! w- V8 z2 S</property>
9 V. v- M1 {' P- J7 W! W: L/ z3 Y- ?<property>+ V! [% v# z$ ?9 G
<name>fs.viewfs.rename.strategy</name>5 T- K0 f) X3 K
<value>SAME_MOUNTPOINT</value>
+ y2 x5 c% i2 }$ [ <description>Allowed rename strategy to rename between multiple mountpoints.& ?! T6 X( {6 p( }5 u* z
Allowed values are SAME_MOUNTPOINT,SAME_TARGET_URI_ACROSS_MOUNTPOINT and
% h/ E2 V: {' J! z9 @ SAME_FILESYSTEM_ACROSS_MOUNTPOINT.; F$ G, l: C$ C R/ r* K2 z
</description>1 x' I$ T; w- W% d1 b; d; R) B
</property>. ?4 ^' P# H: z2 c4 {
<property>( x0 |" B' T6 |. J8 |' V
<name>fs.AbstractFileSystem.ftp.impl</name>
, z% o+ ?) _ A3 b: k; \9 h# B% q <value>org.apache.hadoop.fs.ftp.FtpFs</value>
, |/ k: M; u5 N3 D9 G8 U3 v <description>The FileSystem for Ftp: uris.</description>4 [- V, M# O' J
</property>3 b+ e% l4 ~- b" j
<property># P- v( N/ z3 G
<name>fs.ftp.impl</name>
7 h- U7 s3 {" \8 J4 Z" C' D <value>org.apache.hadoop.fs.ftp.FTPFileSystem</value>- ?2 U: [/ L, o3 ~, j# V) @
<description>The implementation class of the FTP FileSystem</description>$ D& Z+ y2 }( [& E; X
</property>) N5 x& G$ f. C. a5 n; F
<property>
; s# l: \* m1 a <name>fs.AbstractFileSystem.webhdfs.impl</name>6 h+ A& t) n1 f+ }" l: h
<value>org.apache.hadoop.fs.WebHdfs</value>
: ~ S- j: D! h# r! G" a <description>The FileSystem for webhdfs: uris.</description>
4 d! T) x! c, u, L</property>
, g: G9 D; `. V* o% I<property>1 J7 o! n+ Q& h9 z1 [& h/ l
<name>fs.AbstractFileSystem.swebhdfs.impl</name>
; Y+ W2 K5 A3 K4 H: a: P <value>org.apache.hadoop.fs.SWebHdfs</value>0 |! {9 R- `: O) A
<description>The FileSystem for swebhdfs: uris.</description>
1 Y7 h" \$ M4 @7 H+ y1 p9 f7 j1 N</property>: v/ U2 Q5 ~4 G' W. Q$ ?4 A
<property>
3 G1 b1 q% h- y! w/ C: }" v <name>fs.ftp.host</name>4 A9 ^2 Z3 y0 b4 @# W4 ?0 Q- r
<value>0.0.0.0</value>
/ o6 i3 J& u4 [, } <description>FTP filesystem connects to this server</description>5 r# O3 D* k% P! m
</property>9 o/ _: D+ [1 L& ?, s; {
<property># b4 U" u4 z* |2 E1 B7 \% A
<name>fs.ftp.host.port</name>
: z i5 w+ Q. Z" f <value>21</value>
5 Z8 U1 T! R! w% U* C <description>9 u* c# T3 D, c# o' }
FTP filesystem connects to fs.ftp.host on this port$ @- i' I9 T& U4 R8 ]8 L Z' }$ O
</description>: M4 X' Y& z3 E0 r4 r' `
</property>
: H: F+ ^$ t# M, _6 L# ^0 L<property>- |4 \5 i! x& N* M6 E8 K
<name>fs.ftp.data.connection.mode</name>
0 `( X! z0 N. [+ L( r <value>ACTIVE_LOCAL_DATA_CONNECTION_MODE</value>
8 M# G# E7 M/ Y6 x7 i( ?' { <description>Set the FTPClient's data connection mode based on configuration.( {$ r2 u, t8 ?/ C- L0 q
Valid values are ACTIVE_LOCAL_DATA_CONNECTION_MODE,
% ]' E% n3 O1 K/ A$ T' t PASSIVE_LOCAL_DATA_CONNECTION_MODE and PASSIVE_REMOTE_DATA_CONNECTION_MODE." C9 h( G$ N: P& r6 E- N4 J4 R
</description>" B! b" m- @9 `7 p: P1 _
</property>
7 Q, o1 ^8 V) w& p# t<property>' {% L9 V; Q8 e+ v% x
<name>fs.ftp.transfer.mode</name>
: x8 G4 A- t( `- `& U* A3 b <value>BLOCK_TRANSFER_MODE</value>5 t- R. X5 |3 X" L6 e0 F6 I# w
<description>. ]* c+ a$ _: u5 j9 v, P2 u" j
Set FTP's transfer mode based on configuration. Valid values are
% j) [$ w; @, ?/ m5 w STREAM_TRANSFER_MODE, BLOCK_TRANSFER_MODE and COMPRESSED_TRANSFER_MODE.3 ]; p5 e2 L! ^: k% x
</description>. @; w/ o% E6 O( z; a
</property>
( ]: D0 F( E9 ~( d1 a/ {<property>
/ V( a0 q4 _8 a7 G3 Z) K1 _ <name>fs.df.interval</name>
3 R3 M" M( d" {8 N) c2 S4 @ <value>60000</value>
# T( `3 s3 c' p' @+ w5 I2 t! C- X <description>Disk usage statistics refresh interval in msec.</description>
! @5 Q, l* D, ]: _, Q2 s& W</property>
8 W5 X9 |9 M. k<property>8 ^& R2 ^1 C" r& Z
<name>fs.du.interval</name>
4 Q! |5 {" _- x% ]! E% w; C$ S! O <value>600000</value>
8 g; H" m9 c, w+ A: u <description>File space usage statistics refresh interval in msec.</description>* Y/ X+ o) F8 m/ l( j) Y) h
</property>
; E+ @& e' Y7 t+ G! p' p4 h ?6 }<property>. |* z, |) V* j1 P" E0 z3 {
<name>fs.swift.impl</name>) o% L% P$ b. ~: H
<value>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</value>
" `& f: \* J) o <description>The implementation class of the OpenStack Swift Filesystem</description>! n( F" ]3 \! |) D
</property>6 E- _/ ~1 L/ K1 N: t- P5 P, d) h9 F
<property>% w X$ m z! T. I+ m5 h
<name>fs.automatic.close</name>
7 X. E( v. k0 w+ S, O <value>true</value>: L1 ^6 C' n6 D" B6 M
<description>By default, FileSystem instances are automatically closed at program5 H% h+ U1 C$ ^* }7 t( u
exit using a JVM shutdown hook. Setting this property to false disables this
- N4 P3 [' M; g! m# e6 a$ `" k9 P& @ behavior. This is an advanced option that should only be used by server applications _1 R. B( t" X4 J l' i/ s* J
requiring a more carefully orchestrated shutdown sequence.
, i$ i. e* r& L </description>$ o, ^: c" J0 t1 a
</property>
- E1 M" b( ?3 ]<property>
4 Z- ?/ N1 M8 C4 u6 z <name>fs.s3a.access.key</name>
7 ?. p& U& e8 b# N$ C+ M# I <description>AWS access key ID used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
! z1 r1 [: c) C$ ~4 B$ R; l8 \: h& s- }</property>' p- |2 X7 ~# G3 R7 a
<property>: W ?* a7 R' m* G0 Z* ~
<name>fs.s3a.secret.key</name>; `# Y/ O. Z5 B P6 u
<description>AWS secret key used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>) _3 n0 ]$ z# |7 e% s3 S
</property>
# @5 G4 j c) N. N- P+ \* m' k<property>
3 O* k/ }9 s' n! e' L <name>fs.s3a.aws.credentials.provider</name>
/ U! }; |. [# n <description>
v0 I+ b% ^+ q7 F Comma-separated class names of credential provider classes which implement
% C( \( p; m ^ I- b" g) u com.amazonaws.auth.AWSCredentialsProvider.4 t2 Z. G+ ]0 w# d) i$ G/ N
These are loaded and queried in sequence for a valid set of credentials.
( T# X# n$ l( ]' J0 b3 B o Each listed class must implement one of the following means of
, o& a9 t _! x3 V construction, which are attempted in order:
: V; Q3 s/ R' p3 M: F. \ 1. a public constructor accepting java.net.URI and
4 R( Z8 Y4 c0 Y" k org.apache.hadoop.conf.Configuration," Z7 }% K5 m& o( y3 I- ~! P. F6 A" v
2. a public static method named getInstance that accepts no
, Q: y) t* f4 U* g) d arguments and returns an instance of
( S. E" q- D' y1 l0 q com.amazonaws.auth.AWSCredentialsProvider, or
% A- ]- k6 q0 l# ^4 w 3. a public default constructor.3 {$ K9 M& f' _) q( y) Y
Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows
_% C. _, V8 g# Y* Z$ F6 O* t! W anonymous access to a publicly accessible S3 bucket without any credentials.
, f+ ~" w i1 l) N5 q) w8 ]* q Please note that allowing anonymous access to an S3 bucket compromises
2 |. T: u& x2 B9 K( u security and therefore is unsuitable for most use cases. It can be useful
% C$ Z4 N" S, ^3 r0 O! ~ for accessing public data sets without requiring AWS credentials.$ p: U8 H0 [0 m8 {, V3 U
If unspecified, then the default list of credential provider classes,
R# }6 F" u# E% O1 ~ queried in sequence, is:3 i" X, f& a- E* ^
1. org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider: supports static9 s3 U# J$ O& v( w& X+ C; p
configuration of AWS access key ID and secret access key. See also
/ n9 Z+ n& D) E* x' R7 T6 f8 y7 F fs.s3a.access.key and fs.s3a.secret.key.; |2 M& s- c8 G$ l( V
2. com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports
8 [4 f7 u# l( h8 p# y configuration of AWS access key ID and secret access key in
1 g8 R0 S' R, ]5 w: ^3 ~ environment variables named AWS_ACCESS_KEY_ID and
3 k% G# p4 L+ r. P) j- j6 M) I AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK.
3 h o! X" s2 ~ 3. com.amazonaws.auth.InstanceProfileCredentialsProvider: supports use/ m! S7 h! R& Z- z5 {! Z; {2 _
of instance profile credentials if running in an EC2 VM.% [( r4 L- A; F9 I: S1 D& Z' T
</description>
/ ]/ ]" C4 C7 ]7 k6 B8 W</property>
2 v: O* h9 T: n2 I% [" W5 t6 b<property>2 ^# S. W& y; M9 ~: o
<name>fs.s3a.session.token</name>- A& o6 _& G, e3 _6 y1 a) V
<description>Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
/ j! U! {, V# G. l1 u2 S \ as one of the providers.
% }# B% C3 L9 N8 M J( M5 @* @ y </description>
, O) p9 e" T1 v) g% h/ k7 `</property>
! \8 g$ b+ |1 z4 c<property>" I: ^& z% S, L; ~8 w" j5 L- A
<name>fs.s3a.security.credential.provider.path</name>
% P& p* I3 T$ f; Q( c) ~2 | L0 W) C <value />
[! q' B5 \9 j2 a( X- W s <description>
2 Z# p% c/ y5 ?5 q6 {! g2 {6 F Optional comma separated list of credential providers, a list% {3 F5 \7 C+ Q. o, C8 Z' ~0 F1 k* a
which is prepended to that set in hadoop.security.credential.provider.path
6 r$ g6 X- N+ C9 `! k% }2 A" ~: L </description>4 k/ Q2 v1 x9 C# O8 @" j
</property>& j9 D H. O$ Y2 q
<property>7 {$ D, Q4 D' U3 R8 A
<name>fs.s3a.assumed.role.arn</name>- Y2 a* u) V6 g9 {4 W% G
<value />
* n2 L- f" Z' F <description>
) S) W( b: D9 o AWS ARN for the role to be assumed.8 E) A% Y( R; y3 g0 |
Required if the fs.s3a.aws.credentials.provider contains5 O( S$ d' E4 W) N: s
org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider+ @' `" ?* [4 G8 g2 H1 t
</description>
9 Z- b$ l5 d' V</property>
/ P7 s. ~8 g5 o, b( A- \/ m<property>) g* l( F* n$ V! L
<name>fs.s3a.assumed.role.session.name</name>; }7 X- p2 v0 W
<value />
: F2 [, e3 E2 L$ B, }/ G <description>
) o9 y( A1 t# ~) E( l1 H& ^" a Session name for the assumed role, must be valid characters according to$ @4 t9 X* E F) i1 c* w
the AWS APIs.
( p+ p$ X/ x7 [1 m" K2 e Only used if AssumedRoleCredentialProvider is the AWS credential provider.
# [7 D( x: { O+ ~* Y If not set, one is generated from the current Hadoop/Kerberos username.- O! R( g6 M# @. _$ l6 A4 g9 `
</description>
$ ?- y! K& D4 r( t</property>
( f( M* ~5 u) g4 k<property>0 _5 w4 E, ^7 y7 i8 {
<name>fs.s3a.assumed.role.policy</name>6 b. a u, l& e# p% p+ s3 N3 E# M
<value/>
" _' H C& V( w! d8 w <description>
4 g% N, X" j- \- r; } Z JSON policy to apply to the role./ _8 `% V ~; r5 @ k3 P
Only used if AssumedRoleCredentialProvider is the AWS credential provider.
' `3 J5 v7 u+ Q" O </description>+ A* G' ~3 t' u8 j3 [
</property>; x) Z4 U% i, N" N! l: `0 T
<property>2 W1 d% F) R: @5 f
<name>fs.s3a.assumed.role.session.duration</name>
5 x: j! U: I7 g% k. h% c <value>30m</value>
1 a" h2 T( `9 I2 E <description>/ r! N; s" f$ z% ]. m+ j9 i
Duration of assumed roles before a refresh is attempted.
U: B. ~& s2 }/ X& s' g$ i* o4 h Only used if AssumedRoleCredentialProvider is the AWS credential provider.
- h" X* [6 z6 A6 P% H- ]' V. l Range: 15m to 1h
% Z# ]: y* O' W& d* Q9 Y T </description>/ l y- S* k1 C+ Q! I+ f
</property>$ Q; i' X. {6 B( N+ j# d% X
<property>$ u! o# [0 n; U" t& l
<name>fs.s3a.assumed.role.sts.endpoint</name>) A- U' H# a5 B* Y) |+ ]% y
<value/>; E( a# k3 q7 t3 d" `- Q
<description>8 h/ @+ W% |# _* b |* i! ?
AWS Simple Token Service Endpoint. If unset, uses the default endpoint.$ h9 c& G! {; A# t- P, q
Only used if AssumedRoleCredentialProvider is the AWS credential provider.0 a" I* W0 V( {7 x# X6 ]
</description>4 F, z! i0 Q2 ^5 [( B0 S+ h/ h) F
</property>
- v; c! \0 E9 p1 p<property>
. N! W3 W, N% i m3 k# G9 m <name>fs.s3a.assumed.role.credentials.provider</name>
9 g; A5 [/ b" X ] <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
' Y! t: @7 L: t <description>
" A: g. y4 K; h$ b; i) W List of credential providers to authenticate with the STS endpoint and
. |3 P/ [4 d* j+ x: U [) T6 I retrieve short-lived role credentials./ ~5 @3 y" C% |( ]& Z4 ~
Only used if AssumedRoleCredentialProvider is the AWS credential provider.( }6 a& y% U0 D6 V; X% L1 b r
If unset, uses "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider".& X h2 X2 \$ ?4 i, e
</description>* Y" L$ h0 O( J: E
</property>0 c0 I+ F- t& g* Q$ g2 C
<property>, E! y3 H9 d7 U. u# K- ?
<name>fs.s3a.connection.maximum</name>
J0 W) d( B9 |, I1 x: J7 b <value>15</value>
5 g7 \5 q6 e3 A Z <description>Controls the maximum number of simultaneous connections to S3.</description>5 \; ]' J; C6 G6 T0 {, ~2 z- ]
</property>) R3 c8 D S* f1 E3 o/ k& ^9 T; ^) G
<property>
4 @0 ]( w! G* w0 i* Z <name>fs.s3a.connection.ssl.enabled</name>
$ G( {- N7 O5 N# z2 T* m( t2 V0 U <value>true</value>
% _- G+ G6 }: x- {/ J <description>Enables or disables SSL connections to S3.</description>
* P/ [0 T1 k! w) D5 q( e/ ]</property> B: ^( Y( v4 m& j) N
<property> K0 d' o/ p" g' i: P7 |1 ~; V* l5 O- `
<name>fs.s3a.endpoint</name>: {7 i$ B3 H- z6 }4 ]0 l
<description>AWS S3 endpoint to connect to. An up-to-date list is. }( Y, z; N& M3 c1 R# F
provided in the AWS Documentation: regions and endpoints. Without this# [- j) g- X8 r1 M0 Z
property, the standard region (s3.amazonaws.com) is assumed.4 a" V I0 H: Z! z- e, r
</description>
" g# ]6 ]4 p, I" R, C</property>3 x1 u7 l5 [5 f( O+ A, G1 o
<property>
8 q' u. V$ {( n$ c4 t/ K- `8 G3 t: K5 I <name>fs.s3a.path.style.access</name>% r( t3 [, e( U- l) P" }
<value>false</value>; I! p$ y1 v4 Q _6 R
<description>Enable S3 path style access ie disabling the default virtual hosting behaviour." f/ Q- F/ a/ h
Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting.; `+ n+ }& |( i v
</description> w0 J/ b. ~' Y% ~$ z+ r+ n
</property>
8 {8 Q: L2 R& g: [8 J, C m<property>
! k) q- Y6 u4 f1 }4 k <name>fs.s3a.proxy.host</name>
( \7 C0 `$ n2 ?, X9 y! z. O <description>Hostname of the (optional) proxy server for S3 connections.</description>" B. s" _& f$ v) j& E4 ]* k
</property>9 w4 c9 V: s9 @6 E# _* w- Y
<property>* y4 ~' n- ]1 k: O, b
<name>fs.s3a.proxy.port</name>9 m# ^$ @3 c m7 }& i# p' |; x5 D
<description>Proxy server port. If this property is not set! h- O5 _) q# y+ _! B( K
but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with
8 h! d7 b4 j. ~/ r1 ? the value of fs.s3a.connection.ssl.enabled).</description>& `- o8 V3 A% R. O, a% T& |
</property>
b* K% {' l( j- F5 R# q<property>
+ G+ n' O' C( l; T. z4 L, z <name>fs.s3a.proxy.username</name>) T$ t4 N g. r4 P/ a
<description>Username for authenticating with proxy server.</description>
; x5 `0 e6 }8 r) Q6 Z& [$ s</property>$ K" v3 [% s2 M! M
<property>
+ y z O/ q/ ~2 ]3 b( _ <name>fs.s3a.proxy.password</name>, N* |6 A( O0 o, r2 q$ Z0 W
<description>Password for authenticating with proxy server.</description>8 I& R' ?9 U) @7 a- F
</property>5 g- Z: P" t% E$ u
<property>) b: Q) N5 i: @4 ] y1 C
<name>fs.s3a.proxy.domain</name># m7 w! x, o8 c! w' @0 ^- I( B
<description>Domain for authenticating with proxy server.</description>
: L9 J# v+ J* _( X5 K( i</property>. I3 P1 t X5 M6 x6 q9 l4 w
<property>
5 [) `5 W( X/ M) d$ [" J7 [ <name>fs.s3a.proxy.workstation</name>
3 ~+ W* v# A2 z; c) F <description>Workstation for authenticating with proxy server.</description>! _" D" p" \$ W. X1 v, T3 E
</property>) d' F/ u+ ~2 h
<property>! T* h+ \+ F# ^# r! R6 o
<name>fs.s3a.attempts.maximum</name>
6 |; ^+ I6 P# ], n <value>20</value>
& ^4 d3 |% G7 Q <description>How many times we should retry commands on transient errors.</description>
4 X( o* {0 e$ k. P: Q; i2 R4 t: h</property>
. [- I- Z: J# m<property>+ B$ l8 Z% [3 Y2 X4 L! c5 c
<name>fs.s3a.connection.establish.timeout</name>$ S$ O7 c& O+ P% l$ H3 Z+ z0 R$ b
<value>5000</value>
$ N9 B" e; s4 p* U+ [1 x <description>Socket connection setup timeout in milliseconds.</description>
N q5 D$ p/ {- V2 M" u1 C/ d</property>/ ^3 q/ h" H" A, p
<property>% V; Q. ?4 q/ `9 R$ a4 L
<name>fs.s3a.connection.timeout</name>
! B, C* J# F" W- X <value>200000</value>" H/ P. O' D& _) W
<description>Socket connection timeout in milliseconds.</description>6 }. ]7 q& e" O% q; `. w: ~6 t2 Q
</property>
- a6 \ j1 o* H* D# a7 i. f. `<property>
& x* C% x: _ M5 y; A# C <name>fs.s3a.socket.send.buffer</name>
* x0 `6 v% Y, e' \+ v! E9 B <value>8192</value>
# E; ?2 [; ]- {; J <description>Socket send buffer hint to amazon connector. Represented in bytes.</description>
9 X" Y) h9 @; O( d3 T" H2 @</property>
) p! @, h+ R: m5 V* G% d% o3 L<property>
" e/ n. V% l4 f2 L" Y9 z+ ]2 z' ` <name>fs.s3a.socket.recv.buffer</name>
; a" g3 q/ j6 Z' _' \ <value>8192</value>. e) Q! L: ~3 n5 G! I3 n! e
<description>Socket receive buffer hint to amazon connector. Represented in bytes.</description>
+ F4 f$ t" y2 V; K! E% ~! U- e$ _</property>
3 ]' f3 |7 S& ?* Z$ b) d8 U3 h<property>
2 x2 U" N7 P S: ?: h% P0 r0 Y <name>fs.s3a.paging.maximum</name>9 c3 V. z; h. b8 o# l3 w( @+ t
<value>5000</value>& s2 T% [' ]* _, e5 q; W0 u
<description>How many keys to request from S3 when doing* o* m$ l3 W1 \* A, b
directory listings at a time.</description>9 d9 V, E2 }4 |) P/ ]
</property>" k! R8 k" ?# P
<property>
9 K8 ~2 R& n; ^3 h9 | <name>fs.s3a.threads.max</name> J8 y$ v( X. p' q
<value>10</value> n2 r# l1 B. Q, |7 K n
<description>The total number of threads available in the filesystem for data& e# ?' M3 e4 f4 _
uploads *or any other queued filesystem operation*.</description>. t. r8 }4 e ]2 {% @
</property>8 t0 Z4 W1 Q n# ?! u8 R) q
<property>4 e& P2 e4 B4 e- }0 E: D; q
<name>fs.s3a.threads.keepalivetime</name>6 C" a6 H1 t/ B
<value>60</value>0 X3 @0 w$ b& H* w$ W4 [
<description>Number of seconds a thread can be idle before being
: F" [; I9 @" M$ V5 P/ v( j R6 J7 j terminated.</description>
" a$ `) }4 x% F4 K+ O& `8 G+ I- T</property>
# J2 O) |1 S* U I- @ f! ]- l; Q<property>
, U9 p; s8 y3 D <name>fs.s3a.max.total.tasks</name>5 {: n0 A ], S1 f8 [
<value>5</value>+ n/ g( @4 Y+ a$ u: s, f
<description>The number of operations which can be queued for execution</description>5 N2 d3 A+ [; e4 } e
</property>, h B6 }$ Q) n* z, L1 ?7 n6 g
<property>. G6 \, A/ o! _) A& h' @
<name>fs.s3a.multipart.size</name>: A T j+ v/ M
<value>100M</value>
/ t8 f, H$ ~4 w/ ^* I7 D* L <description>How big (in bytes) to split upload or copy operations up into.
5 q+ L% c7 Q* N- P5 |. Z A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
0 }9 w7 w5 h* K, h6 u </description>
. ?& Z% I' ^' ?</property>
$ ^3 T& B: Z& W5 E<property>
4 C# }& V) A9 h3 e <name>fs.s3a.multipart.threshold</name>( _2 h! E9 J$ {% d
<value>2147483647</value>. J( s; u4 [2 h1 @& O
<description>How big (in bytes) to split upload or copy operations up into.
- X. U. {9 K4 l) i& |0 w4 D This also controls the partition size in renamed files, as rename() involves& `: ~7 r6 F) G9 f- c# F
copying the source file(s).
0 w* V$ W2 ~9 h/ ^ A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
7 E* m" m' z6 H* ` </description>3 o8 x+ T5 L) D$ g* Q; e8 B* c
</property>
: |, w$ B+ ?- V* `: R: t" z<property>9 `) S/ q8 m3 T+ N
<name>fs.s3a.multiobjectdelete.enable</name>
6 X X" N7 Y. l H& } <value>true</value>
. s7 Q) {/ N# F3 m7 L <description>When enabled, multiple single-object delete requests are replaced by
4 Z& u' @* r# v9 T& H a single 'delete multiple objects'-request, reducing the number of requests.
& n) S5 p% p6 n/ K* F- j. f$ @ Beware: legacy S3-compatible object stores might not support this request.4 B/ `# M; ~0 G4 A8 F9 D1 t% O
</description>
s% e/ w0 z+ I' N# \$ \3 K$ }* r</property>2 \; `" o' S3 i. Q1 L
<property>: v K* V# F. n
<name>fs.s3a.acl.default</name>
* d5 H2 L( v* O1 {3 S0 n3 N6 W5 z <description>Set a canned ACL for newly created and copied objects. Value may be Private,* q3 l. X0 j/ |$ C5 P; l) D' B* B5 n
PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead,$ D% l, c5 v3 n+ B* Y; b
or BucketOwnerFullControl.</description>6 |) U- E+ s* }: U
</property>* {0 R1 K0 g! l' i+ o& V s. P
<property>& d5 u) w$ i; S( e" }* S
<name>fs.s3a.multipart.purge</name>
5 e+ `) C: `* B1 g" w/ r! ? <value>false</value>; i- r0 ^1 `7 z% T4 e
<description>True if you want to purge existing multipart uploads that may not have been
5 [6 c+ w; U2 y. r: Q8 E+ G completed/aborted correctly. The corresponding purge age is defined in9 ?, I4 _- n, M6 e
fs.s3a.multipart.purge.age.
. x4 ^# I2 h! ^2 I0 b( e If set, when the filesystem is instantiated then all outstanding uploads: E) h1 k& {6 ]; }2 O
older than the purge age will be terminated -across the entire bucket.
2 P4 j) g5 P- @1 H; D This will impact multipart uploads by other applications and users. so should
2 K, p. T* k- f" R9 D be used sparingly, with an age value chosen to stop failed uploads, without
4 t9 I9 S: B3 P b breaking ongoing operations. p* P- u- \; j2 p# U& D9 k
</description>/ [4 ?! _$ P9 U: J. ]
</property>
0 `! m( C- F5 j. Q9 ]6 z7 |/ C<property>( ^ U; y9 S$ h/ J: Z# v
<name>fs.s3a.multipart.purge.age</name>
; |7 G$ t5 h3 v5 c <value>86400</value>
1 r H' u1 P( o; a0 d! S$ _ <description>Minimum age in seconds of multipart uploads to purge) }5 P9 u+ N# _9 ?7 |2 \
on startup if "fs.s3a.multipart.purge" is true6 }1 k1 r) a8 C" E
</description>4 H8 D1 A' a! |8 Z) c
</property>
; J S: M, }- k3 v% b: {* Z<property>, L& R; s; n+ }3 [
<name>fs.s3a.server-side-encryption-algorithm</name>
6 ^% C. `6 A \$ D/ g9 h* _, m" V <description>Specify a server-side encryption algorithm for s3a: file system.
6 Y v# {% v( y Unset by default. It supports the following values: 'AES256' (for SSE-S3),% s" _5 B6 c- u L
'SSE-KMS' and 'SSE-C'.
* |* E8 h3 R0 H* @ </description>9 n( G- S+ c; l1 N3 H
</property>
+ _7 e; B6 o0 l8 M3 _" [2 z% e4 }% A6 n<property>4 p" x/ Q+ Q( `3 E
<name>fs.s3a.server-side-encryption.key</name>2 M' q3 J5 X$ K! H) L
<description>Specific encryption key to use if fs.s3a.server-side-encryption-algorithm
" p. K, I& h9 O8 p! `, ^ has been set to 'SSE-KMS' or 'SSE-C'. In the case of SSE-C, the value of this property' D0 r' x- o0 m# H' U$ u
should be the Base64 encoded key. If you are using SSE-KMS and leave this property empty,
6 Q' m( l7 z7 U& X you'll be using your default's S3 KMS key, otherwise you should set this property to7 X4 c r! o7 J- R
the specific KMS key id.: U( r) h, U" Y% l4 V! ~$ j* T
</description>( d$ y K8 P ]- e+ M
</property>* e+ r: A$ h- }
<property>
5 q, U4 R9 p: d& o <name>fs.s3a.signing-algorithm</name>
+ }( l* O% y( q& W <description>Override the default signing algorithm so legacy: h p$ A% t# n$ M4 v
implementations can still be used</description>$ m( T4 |% i4 } W4 e8 X
</property>% \5 Z) J* U+ y5 ^- X2 r
<property>
6 t1 W O0 n% K <name>fs.s3a.block.size</name>. t" u) H1 z- t9 f* E6 _5 Q& d
<value>32M</value>
1 m% J" f) h1 n$ p0 L& m- G: O( J <description>Block size to use when reading files using s3a: file system.
+ C& E+ [) w" j2 V- \( ^4 c% w A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
+ s/ w( _4 _5 Y6 s1 I2 O& Z </description>/ }. C3 P* T) x
</property>
1 \6 I( Z5 @5 k<property>
! D6 P. X9 m8 D1 \! V# i& p <name>fs.s3a.buffer.dir</name>$ p; G7 @! U$ }, y% Y& o
<value>${hadoop.tmp.dir}/s3a</value>
" g2 v1 N4 p, B; |# l <description>Comma separated list of directories that will be used to buffer file
7 u. q/ u$ u: I% B2 ]+ F uploads to.</description>
2 z/ f. l- p0 u4 L</property>/ l5 Z: y R3 ^/ |( E
<property>0 Q& q9 h9 [0 b E3 A
<name>fs.s3a.fast.upload.buffer</name>2 U2 R2 C8 q2 f2 b; i6 @$ K) H/ y
<value>disk</value>
* N/ e2 s& `& j Y' n <description>
0 M/ H2 p" ~1 |) r The buffering mechanism to for data being written. g" X0 E1 L, r' J
Values: disk, array, bytebuffer.% v# ~, E' d+ l3 q9 l- K( `
"disk" will use the directories listed in fs.s3a.buffer.dir as
2 E+ Q+ j3 z& f. r ^8 N the location(s) to save data prior to being uploaded.: Q/ r1 p& h; I! Q, o. }
"array" uses arrays in the JVM heap
p$ M- T: N0 p9 v, e. K "bytebuffer" uses off-heap memory within the JVM.9 T1 M$ A q- D8 D7 I, k# `1 E
Both "array" and "bytebuffer" will consume memory in a single stream up to the number
* C9 a/ ?( q. B( } of blocks set by:
/ W% H" k3 J7 e0 _! [% p. K6 R% o0 Q" v fs.s3a.multipart.size * fs.s3a.fast.upload.active.blocks.4 c6 ^9 Q; h. E: x$ n* D
If using either of these mechanisms, keep this value low
0 }, @5 m% p- K* E' b# X9 S The total number of threads performing work across all threads is set by2 I3 G) G# [1 x' w6 r2 F
fs.s3a.threads.max, with fs.s3a.max.total.tasks values setting the number of queued( ^" }. g. D/ c
work items.- a$ X0 J8 C {) ^- Q4 _8 W3 X
</description>1 ?2 r) `- R7 o3 k, o w( {
</property> I) r; M2 ]( r4 I. ?1 R
<property>
; W! m0 B5 r& z% ]8 h <name>fs.s3a.fast.upload.active.blocks</name>
$ v0 A5 M( b; ^' T2 Q! s <value>4</value>
" I3 p2 o% V# O7 i <description>
: n8 j/ n, u$ k) ]$ T: D7 i% i Maximum Number of blocks a single output stream can have" w: g$ X9 j- }5 w1 r( P
active (uploading, or queued to the central FileSystem
! M0 p& @9 a+ e" K& U- n instance's pool of queued operations.
/ l$ W: H/ `: I" \; K This stops a single stream overloading the shared thread pool.: a' D* s/ t& j0 {% f
</description>% {" s% W6 ]. n
</property>
% F5 J; Z1 l( ?' @! w& y<property>
$ }7 Y( x" E) Q' q# K, q <name>fs.s3a.readahead.range</name>
0 Z$ Q) H* X& R7 Y1 @ <value>64K</value>
. T' B# }) R* w, Z5 d/ e% w$ c4 a <description>Bytes to read ahead during a seek() before closing and7 O! F {1 i# J% N
re-opening the S3 HTTP connection. This option will be overridden if
7 D8 C# { m. C* G any call to setReadahead() is made to an open stream.
! z7 s0 j7 X5 Y: n* h9 O A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
8 H- y- D3 H" D$ W3 w, C </description>* y% f! }% `1 _& r8 o* v
</property>: ^1 u: B r4 k
<property>8 I( t) W0 `3 c2 f/ e8 \' r
<name>fs.s3a.user.agent.prefix</name>
/ R! F5 y6 ^' k2 z" n" e <value></value>2 k: g* E# K( D* H, r
<description>
" j9 c5 k, w% w# f/ K0 w Sets a custom value that will be prepended to the User-Agent header sent in
- E5 |9 g6 s) W% v' s2 t8 D- @8 A5 { HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header
( S0 L( r+ a5 P5 h9 W& A# J/ | always includes the Hadoop version number followed by a string generated by
3 |- l+ [/ \4 X# G the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6".' m2 d4 k5 G' T0 r g
If this optional property is set, then its value is prepended to create a
/ Q0 z( e# q! I$ j" k customized User-Agent. For example, if this configuration property was set
3 t% n, c' a3 }& M, h0 W7 \3 q to "MyApp", then an example of the resulting User-Agent would be
) v) e- ~( c8 q$ c6 a "User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6".+ T6 Y% m$ g( @ }: Y) I% b& R
</description>
! d i# z0 H1 t- W</property>
2 `; f, |+ |( R9 j: ]<property>& ^6 c1 `2 [ S6 }" P$ Q' l
<name>fs.s3a.metadatastore.authoritative</name>3 L7 x# n6 P [+ x2 J- W
<value>false</value>
& j& M3 {2 s/ H <description>
* W9 y o6 b- j' o3 w When true, allow MetadataStore implementations to act as source of3 @: d: N; V# A$ \7 `/ q* ^
truth for getting file status and directory listings. Even if this
9 D: O- b, A. S) x( y& b; y" ] is set to true, MetadataStore implementations may choose not to
. A0 @% G1 z$ i, l% N- C return authoritative results. If the configured MetadataStore does3 B" ]# S) Z0 i0 g. e2 u+ ^
not support being authoritative, this setting will have no effect.
/ z4 s; q' D" i6 a7 _: N* Z3 | </description>
2 H) W- }& s% x. ]2 W</property>& e# {5 J" B& n" m4 `
<property>! I7 ^/ {! T4 d
<name>fs.s3a.metadatastore.impl</name>% A1 n3 k, d: l1 ~
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
{# K) O( G4 }7 f2 @5 x3 T <description>
7 O8 A+ g, R! f8 H: j+ R* I/ s' R Fully-qualified name of the class that implements the MetadataStore9 ~6 F8 ^* M( O$ z, P) Y6 E" u
to be used by s3a. The default class, NullMetadataStore, has no
8 d: s! G# `1 G3 e% H# p G1 f effect: s3a will continue to treat the backing S3 service as the one) p8 k% C1 T$ _0 Z
and only source of truth for file and directory metadata.
* `7 \0 D! r. c4 N, E </description>! H, ^4 |' a1 r4 n
</property>7 M( ^+ f5 f& H$ N' m$ f" t
<property>
6 V5 V1 f4 Z! y <name>fs.s3a.s3guard.cli.prune.age</name>* C% e. f" o* {* [* v, R. q0 e, V
<value>86400000</value>( M! T- t8 {% U/ d/ N7 b3 I
<description>4 i, p$ f) r2 f' W8 ^9 h0 [
Default age (in milliseconds) after which to prune metadata from the
/ P' N. [ ^* Z N metadatastore when the prune command is run. Can be overridden on the9 K* _4 K0 i) s$ |- L; T
command-line., N0 }3 o2 W! Y0 h, ]' O
</description>
3 _+ u5 @0 p5 C5 s* Z5 v</property>$ e# j7 F4 U3 \1 E" C
<property>
9 Q/ z. H7 R1 w3 b/ A \ <name>fs.s3a.impl</name>
( N5 p8 z. M# c' R7 O4 w <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
; r& }& I. ?( D( c$ I# B4 x. R <description>The implementation class of the S3A Filesystem</description>$ C) W0 a& D0 g; K2 A
</property>7 W: \1 r' b$ v, n; a. q
<property>
/ s3 n% \- c( Y! R6 ~/ G0 i <name>fs.s3a.s3guard.ddb.region</name>
2 Y: d3 j$ N' S( t# o8 ~ <value></value>
6 t5 g( S; d& y, p! Z <description>. H9 R, V$ W& m+ e: v5 m* g, F
AWS DynamoDB region to connect to. An up-to-date list is
# I1 s, G: E% e8 e) U2 _8 w5 V provided in the AWS Documentation: regions and endpoints. Without this
2 {8 V$ e* a0 K6 L) d property, the S3Guard will operate table in the associated S3 bucket region.0 [# [" H- z2 E6 g4 u& U
</description>: [" s2 Z( @/ S: p
</property>
! U& P- x5 ?4 i( C' c3 X: O" f7 ?5 x<property>+ X' {; M5 c# w) R+ s4 s
<name>fs.s3a.s3guard.ddb.table</name>( e6 k) {4 u: v. M4 _- t
<value></value>
- J. }' X" U+ l! } ?. y& P6 c5 T <description>% c1 E N2 h R8 _. v: a5 z3 U, ]+ D
The DynamoDB table name to operate. Without this property, the respective
2 R) I% M, }. d9 k S3 bucket name will be used.8 D* p' Z4 e5 O1 e' C
</description>
, e; y7 _$ S1 W" w# d, s</property>8 Z( ]% b" }* H! r
<property>
3 }" G6 t5 v- n9 c7 Z# ? <name>fs.s3a.s3guard.ddb.table.create</name>4 Z/ C' Y7 V0 ~% z% R1 V! q* i$ h( a
<value>false</value>7 T) q/ f) y- ^) N. I9 P
<description>
# ^- v2 Q5 q& ]- A! |/ K If true, the S3A client will create the table if it does not already exist.
' \3 u( U4 n3 x: e. p </description>
7 f7 n2 v$ `6 g* b, @</property>& ]: `( Y. E* G8 @* \
<property>5 E$ X, y- i5 m/ R0 q
<name>fs.s3a.s3guard.ddb.table.capacity.read</name>- L8 L/ G" Z5 o
<value>500</value>
8 C# c% [7 L9 _; Z0 x8 d( K <description>
* d4 S7 y( u& S- c) ^ Provisioned throughput requirements for read operations in terms of capacity: \0 |# p: A6 y' `. b {* Q3 n
units for the DynamoDB table. This config value will only be used when! I. I3 {+ Q4 q+ y" Q4 O
creating a new DynamoDB table, though later you can manually provision by# U& x2 T$ s/ t8 d) _
increasing or decreasing read capacity as needed for existing tables.
( j y* S1 d N: y, c See DynamoDB documents for more information.
; A5 @: C$ i$ q& n </description>
8 i& O: V1 N1 f$ N</property>
# ~ ^4 D4 E0 W9 P* z<property>
% f0 o* d8 f$ ?" d2 V <name>fs.s3a.s3guard.ddb.table.capacity.write</name>
# i" W2 f9 y8 w- |# r <value>100</value>
6 _& B) \6 E' ~ W <description>) X; o' {# h8 I, e
Provisioned throughput requirements for write operations in terms of
7 t- Q' r& b6 r capacity units for the DynamoDB table. Refer to related config
1 A: Q" \* |8 }' V fs.s3a.s3guard.ddb.table.capacity.read before usage., b0 n8 n! S j7 n) K
</description>
! G+ Q' U6 ?9 q1 @</property>" z( y# W6 N {' z9 f
<property>" r' I' J0 L* K/ o2 H
<name>fs.s3a.s3guard.ddb.max.retries</name>4 n/ G7 ]* v$ e- H* j% _6 T! l) g6 f
<value>9</value>& p) w* ^8 g& Y* F6 [
<description>
: v6 S, X6 V- E. t. N4 ?9 z. L Max retries on batched DynamoDB operations before giving up and8 D% _: S/ G& s. O
throwing an IOException. Each retry is delayed with an exponential
7 ]' w( L! `: x2 X5 J8 ] backoff timer which starts at 100 milliseconds and approximately
4 i: ~) c+ o3 W7 e9 B4 M doubles each time. The minimum wait before throwing an exception is
0 @- E( t4 |7 [3 {/ \- _4 z' ? sum(100, 200, 400, 800, .. 100*2^N-1 ) == 100 * ((2^N)-1)* K; I- Z" w* r/ ?, t" X
So N = 9 yields at least 51.1 seconds (51,100) milliseconds of blocking5 T4 y* P7 k: k. y
before throwing an IOException.1 u4 f1 t' x7 ^: [
</description>
+ h& l; i6 {* U( s</property> W9 \" N3 O/ M) o0 c. l- u/ m
<property>
* D* ]+ P! s! z7 k7 E <name>fs.s3a.s3guard.ddb.background.sleep</name>
, b) h& }& P$ v* K+ K <value>25</value>
1 d9 a8 a4 h; @% s4 \0 F: w2 x <description>+ _! c4 `0 T* `8 |9 ~( h1 \ O
Length (in milliseconds) of pause between each batch of deletes when; t; Z" s3 h4 g6 y
pruning metadata. Prevents prune operations (which can typically be low
# _% D2 _& c v priority background operations) from overly interfering with other I/O
! ^& C4 A; V) J, F operations.2 }0 r- h* Q6 X1 W! e+ X% F, L; M
</description>( j3 t" ~' V% K
</property>& p2 G0 [1 f7 M. e- t$ _( M
<property> m; L! w |3 l, f6 v" G( U- x
<name>fs.s3a.retry.limit</name>
) A; ~+ K4 A$ m- ]1 S <value>${fs.s3a.attempts.maximum}</value>
6 V$ w( }" L! Y) E0 s6 {3 d <description>
; h: ^" Z8 G( u* Y( @; ^# T; { Number of times to retry any repeatable S3 client request on failure,
. B7 q6 a) E5 y9 d excluding throttling requests.1 v2 ?" V( F: C3 o! x! N, A# m
</description>. }$ n. Z- f( U. C) g2 l
</property>) e( _* F4 c. {: W* n5 g: ` P
<property>2 o. ?( V* [( g! y' R- d
<name>fs.s3a.retry.interval</name>' |" Z: W i" q' s1 F
<value>500ms</value>
2 l9 {5 S& Q6 h+ }) D& r; N <description>9 J9 j& r1 j4 \7 E+ Y* l# I
Interval between attempts to retry operations for any reason other* `* t0 x( i* Z ^0 h ?
than S3 throttle errors., e4 l2 o- T: O% ^2 y
</description>5 G+ [ I3 C& U+ ?8 L
</property>
! I; B8 \& d( _* D<property>
: G, o+ n d+ |4 g0 ~ <name>fs.s3a.retry.throttle.limit</name>
. A$ f( A( R: r1 M! b <value>${fs.s3a.attempts.maximum}</value>
. {7 V5 N5 @2 G8 t+ _& `2 c1 a$ } <description> }& B6 L( T0 H3 W' j7 i
Number of times to retry any throttled request.# J" ?4 u+ B) x
</description>. H& p0 b6 }1 J: e
</property>
0 h/ [3 ?- z: K; U; ~. X3 M. r<property>
0 |+ z! U) t3 E <name>fs.s3a.retry.throttle.interval</name>
! @4 y; ?# R, i# K4 y$ _8 _ <value>1000ms</value>! u! O( F- Q& P" Q
<description>
0 g2 D% Y2 y- K! [' j Interval between retry attempts on throttled requests.
! Y. }+ [2 M- q% r c/ x+ B7 { </description>& x: ~: p$ [% ]# K2 [; \7 J2 E
</property>
1 _5 G u, C [+ r( V/ k9 I0 q8 Z<property>
& k& a" J9 J$ ]6 F <name>fs.s3a.committer.name</name>
% W' v# m/ z0 I& K. B <value>file</value>3 Q( f) U0 _3 k
<description>3 k9 Y; r+ H4 k: T2 H, r' _ |8 ^) l
Committer to create for output to S3A, one of:% X$ U& k- q" B" X2 y5 g8 `
"file", "directory", "partitioned", "magic".
" K H% b) S. D </description>2 o7 k8 W5 W( n- N6 s( f/ o2 }
</property>- X V: N: {7 Y: P8 B6 s
<property> {1 X% e& E% O j
<name>fs.s3a.committer.magic.enabled</name>- b) h4 I2 J4 |
<value>false</value>; I" Q7 O" f, Y$ `3 R
<description>
2 |/ v0 l5 z' g Enable support in the filesystem for the S3 "Magic" committer.
, F L% Q0 a5 g: w9 V$ W. Q When working with AWS S3, S3Guard must be enabled for the destination
1 X3 q c: ~1 p L0 S bucket, as consistent metadata listings are required." w: \! U! M# j& {3 g4 B
</description>
' c( B A! m. z7 Q3 v</property>
6 Y# d& W8 p9 Z t<property>0 U- d% C5 E7 U8 o# a2 i1 m/ p
<name>fs.s3a.committer.threads</name>/ B. x" A2 `- E: P& m
<value>8</value>
) ]9 N: w( Q* z <description>
. L* K0 W( ^$ [1 l' ^- t" ]* F Number of threads in committers for parallel operations on files( c; x0 Y6 \1 X4 ?- M7 I
(upload, commit, abort, delete...)
- @3 E* Z) N) }( ` </description>+ P9 R, ?* C9 T0 m
</property>; E# z' L% P, N4 h* A
<property>
/ z5 S7 t+ V( Z& K <name>fs.s3a.committer.staging.tmp.path</name>- `: d+ d5 d" b* u. c# U' n9 O$ N
<value>tmp/staging</value>) y" d1 c8 r2 }4 V+ q/ Y
<description>
4 @- T7 j3 N0 @. c Path in the cluster filesystem for temporary data.# [. l& w, q$ c: I
This is for HDFS, not the local filesystem.
$ G* p6 z6 b; n T5 [ It is only for the summary data of each file, not the actual
% m8 V5 ^) D+ h( b4 E% Z1 Z data being committed.
, y! G) k5 p, z) T Using an unqualified path guarantees that the full path will be% A0 h0 K0 P/ u' ?$ _
generated relative to the home directory of the user creating the job," {% @ W0 P1 e* Z6 ?
hence private (assuming home directory permissions are secure).. b: G5 f8 s8 k, ^
</description>& g5 W5 C2 q+ l( u! u
</property>+ Y v, l2 u0 o, I
<property>
/ d% B! G: i; I <name>fs.s3a.committer.staging.unique-filenames</name>3 L( M' F1 x- S1 r; b
<value>true</value>4 V7 t: c7 E) B/ P. A9 q
<description>+ j" ~/ c+ j8 W, R N, y- B6 m
Option for final files to have a unique name through job attempt info," A! o/ U( m* N
or the value of fs.s3a.committer.staging.uuid
. z3 ]; n J \2 S0 O When writing data with the "append" conflict option, this guarantees" O/ H* p& ~' ^8 F- A9 }
that new data will not overwrite any existing data.8 y/ `9 t# Q2 U+ F( {
</description>" j) R3 {3 j* r6 P8 }( D( ^
</property>
2 h# x( S) Y4 j, G<property>5 B z: Z( A9 l+ P; ~/ c! k
<name>fs.s3a.committer.staging.conflict-mode</name>" ^) n4 I( `4 }
<value>fail</value>
! j1 e$ S; l2 F <description>
8 z, ~% Y A$ L2 E- K Staging committer conflict resolution policy.
5 g/ }6 A, x" }5 ^% g l7 t Supported: "fail", "append", "replace".' y8 K4 G9 E" J
</description>8 L- N# ]/ h# k4 y- g* ?0 _
</property>
: a2 M7 J& d" C! o9 _& U* N9 `; I<property>
% x2 S1 Y) Z3 z3 a0 ~! K <name>fs.s3a.committer.staging.abort.pending.uploads</name>* |. D; a. Z9 ]. F
<value>true</value>$ m+ ?# G* s9 P2 R; U6 p5 _
<description>
+ m, p5 K3 f m5 A) \ Should the staging committers abort all pending uploads to the destination
) x% p1 O' Z% R6 X0 k" A( q directory?
' f C3 K: G% b, C/ Z Changing this if more than one partitioned committer is) e/ f/ I3 w) P- F3 z9 u. b P
writing to the same destination tree simultaneously; otherwise
; Q8 N& Y( G3 g) C$ P the first job to complete will cancel all outstanding uploads from the
6 G8 G: u4 l6 B2 G! ^$ d+ E" e others. However, it may lead to leaked outstanding uploads from failed- e. K& {8 ^; a; L
tasks. If disabled, configure the bucket lifecycle to remove uploads
$ z: K+ f9 z3 i: @# v+ o2 F after a time period, and/or set up a workflow to explicitly delete
1 P; T) s5 B5 Q8 [% [1 n* S entries. Otherwise there is a risk that uncommitted uploads may run up) {% Z7 v T2 ^* B
bills.2 \2 o% B' N: a2 Q" P! X& C7 e
</description> k4 B9 i. P8 g/ a, ]6 w3 ^
</property>
3 h' }* v0 D0 U( U7 L<property>
% v% H+ v) R- ?5 B <name>fs.AbstractFileSystem.s3a.impl</name>
' q3 i$ M# g8 K O2 R9 K <value>org.apache.hadoop.fs.s3a.S3A</value>8 l- j+ L9 }4 B( k u
<description>The implementation class of the S3A AbstractFileSystem.</description>
# S' Y7 s- J& `6 M A5 |' X& e% ^</property>
) _- ?0 t4 {+ F; @, Y. b<property>
. {0 D/ r1 b4 M% {4 S <name>fs.s3a.list.version</name>5 L/ A" J6 l- C& N$ ]
<value>2</value>
" _4 x+ e8 t* | <description>, ?8 h; G/ @3 X7 o' o' q, C1 t
Select which version of the S3 SDK's List Objects API to use. Currently
$ f$ s: t5 n/ E) y# j support 2 (default) and 1 (older API).) {7 v1 p) H8 |! d7 D
</description>' t: u1 v. T0 u4 c
</property>
- n c/ q4 T, X: V) ^6 v. {7 A<property>
- m' m" s/ p5 w2 }+ [ <name>fs.s3a.etag.checksum.enabled</name>+ G0 I) k3 B2 Y' A
<value>false</value>
) {: C$ D( V9 c! }* d8 | <description>
* u* z" z( p7 h" |1 D( C Should calls to getFileChecksum() return the etag value of the remote( e3 m/ \+ I7 H; L, T% w
object., Y; `2 S9 c' n/ I. |) L
WARNING: if enabled, distcp operations between HDFS and S3 will fail unless
9 U, A% q; D8 N v% b% P* V -skipcrccheck is set.: K" `8 V2 f. C
</description>( M; S% `1 Z% U& f% z
</property>$ ?; R7 l. V& o9 A8 K( w% Z; u7 K
<!-- Azure file system properties -->
- } |# J: k, M+ Q6 F<property>
1 u. g+ w G% w9 @ <name>fs.wasb.impl</name>
1 z/ I1 S O; f% T# {% E <value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>$ r: s* {/ V: ?# | Q0 G
<description>The implementation class of the Native Azure Filesystem</description>
: E% M+ L+ U; ^( y* D</property>
' B/ ]1 Q$ _9 Z) G! _<property>5 p* j& A6 g- z& v
<name>fs.wasbs.impl</name>$ E8 ~6 c3 h+ \
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure</value>
2 F' a9 X# f' ` <description>The implementation class of the Secure Native Azure Filesystem</description>
6 P3 y' c1 R/ J) u s& {- j% r</property>& F& r. T( s, g
<property>
9 k/ y+ Y% r4 `. V3 @ <name>fs.azure.secure.mode</name># \/ d L7 ? X7 G+ X% _' `. n
<value>false</value>( ?4 ]# g/ w) y" R+ X5 u, \
<description>
) D7 g8 G W, u' {, E# q% } Config flag to identify the mode in which fs.azure.NativeAzureFileSystem needs3 g! r0 W8 x4 ?% H2 {7 Y
to run under. Setting it "true" would make fs.azure.NativeAzureFileSystem use ~/ t! ^, |) u
SAS keys to communicate with Azure storage.
: ^: P! Z8 N* E; e3 `. x6 A </description>, i4 ] q$ N7 w: t" ~$ b8 Y. F
</property>
+ x7 t9 ^# x7 E$ K/ c9 b<property>! w7 Z1 f: o+ Q- O3 J4 c
<name>fs.azure.local.sas.key.mode</name>
I4 Q: p, ^( u# S. |0 h8 I3 q <value>false</value>
$ ^& Q" }" J# }; R0 S& _6 S) U <description>3 k" ~$ `! r: U# a+ a
Works in conjuction with fs.azure.secure.mode. Setting this config to true
1 O/ J- `' R5 ^) _; n) K% ]0 [4 H% K results in fs.azure.NativeAzureFileSystem using the local SAS key generation! c, d1 @4 t$ V! `
where the SAS keys are generating in the same process as fs.azure.NativeAzureFileSystem.
: V% o& f/ ~8 H# H( X5 e3 V' ^9 G2 l If fs.azure.secure.mode flag is set to false, this flag has no effect. C& o. ?% N* ^4 a
</description>
, N: ~' E* P2 U9 Y" G2 l3 f</property>( l+ k. t J8 }9 G. c" H; W! s
<property>
& t' c$ T* ~' h8 \4 ~ <name>fs.azure.sas.expiry.period</name>
/ W: n, Y6 [ t1 _! t0 H8 @ <value>90d</value>
9 f" P. _$ M V. p! J: Q! T <description>
/ }3 v5 w2 W U3 [ The default value to be used for expiration period for SAS keys generated.
; f" R! I2 ]8 y Can use the following suffix (case insensitive):" ~9 J' j# i3 ~/ _9 p y
ms(millis), s(sec), m(min), h(hour), d(day)* o1 w0 ~' O+ x" d/ z
to specify the time (such as 2s, 2m, 1h, etc.).# b, z- _3 B7 q4 p* q$ Y6 q
</description>1 `) w) L$ F6 V" l
</property>; Y' J8 d7 n6 s: v/ u/ }
<property>
" Y' H& a% ^5 _# |7 C <name>fs.azure.authorization</name>
2 j' p* W6 `( n A' b7 g <value>false</value>: M, M0 I2 q" E9 t+ ]* O
<description>; n- Q# \8 |. L8 Y- H$ o
Config flag to enable authorization support in WASB. Setting it to "true" enables
0 e X2 l. T7 j( _ authorization support to WASB. Currently WASB authorization requires a remote service; t) {+ u8 J& _5 O2 o
to provide authorization that needs to be specified via fs.azure.authorization.remote.service.url
$ W% u+ ~+ `4 d, m% \) ]% J3 j# ? configuration. x* U3 e' S3 M* W: B/ f, A
</description>, y8 w% }+ m7 N1 @; u M8 o
</property>5 m* E Y0 q! o
<property>$ L* {6 |6 J$ q' Q; I* ]
<name>fs.azure.authorization.caching.enable</name>
$ F. t( f4 {5 s6 Q; a <value>true</value>
0 Q2 x1 ~! W, d4 M' D <description>5 K; k# {' @7 i& j3 n5 Z- i
Config flag to enable caching of authorization results and saskeys in WASB.
- k1 B( q* B, K" G- t This flag is relevant only when fs.azure.authorization is enabled.
, ~. E- J( T z: \% g </description>& s1 j( h' W+ K( y
</property>5 N8 ~6 d9 T$ {: @
<property>
& x v3 o" V7 j* W, c <name>fs.azure.saskey.usecontainersaskeyforallaccess</name>
; I, l# |* @: Y5 u2 Q <value>true</value>
' ]" \8 z6 s0 Z <description> a, R* {/ g) w5 C% `9 `1 A
Use container saskey for access to all blobs within the container.
# V" y2 B6 N" t5 J8 A2 s) _ Blob-specific saskeys are not used when this setting is enabled.. D. y1 J7 W$ K9 h
This setting provides better performance compared to blob-specific saskeys. h, V$ M8 G7 @. O0 U# j
</description>
& O) N. R$ W8 O</property>% X, ?2 I3 v, g8 u& j
<property>
$ s, A+ t, e" {" W$ g <name>io.seqfile.compress.blocksize</name>. r$ K6 u% |- W3 w( Z' _$ o
<value>1000000</value>
& O8 c$ Z# y5 S+ } <description>The minimum block size for compression in block compressed
4 \! u R+ A# ?. E% W% } SequenceFiles.% P0 i+ m2 M. w+ M; M; @# C
</description>
+ c E8 i2 [$ s8 ]</property>. A( q8 K1 C/ `7 _
<property>1 N0 G) Z6 K) t0 a. Z) p$ M% L
<name>io.mapfile.bloom.size</name>: O$ y6 u \4 }" @# }1 T
<value>1048576</value>( K; n5 z8 h+ ]( m# @
<description>The size of BloomFilter-s used in BloomMapFile. Each time this many
$ I( c, r7 v; ]$ } keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter).
! M7 m) S- g/ y. i7 H Larger values minimize the number of filters, which slightly increases the performance,
# m0 Y+ r- y' I+ S7 j% Q but may waste too much space if the total number of keys is usually much smaller9 B8 M3 v/ r) k2 ` f% v* [
than this number. j. v6 V" b8 Z, Q- s6 _4 P
</description>
8 u8 l" }1 X- a</property>
3 [5 k5 M* {' S% h! r<property>( I: J9 h+ _ x* K0 q5 J$ Z
<name>io.mapfile.bloom.error.rate</name>/ R: T. k; T9 j. J' l" a0 p
<value>0.005</value>
# f0 X5 g) V4 }9 I" Y6 T n4 @ <description>The rate of false positives in BloomFilter-s used in BloomMapFile.
" B; D# V \; N8 Q$ W As this value decreases, the size of BloomFilter-s increases exponentially. This
7 j4 D/ i( s% H& f: g value is the probability of encountering false positives (default is 0.5%).
' u! q' e$ R9 n! r7 \: J </description>
1 x* ~& A& J0 F) D& F0 Y7 g# j7 A</property>9 C) |* z' r2 f( c1 l7 W5 v
<property>) i7 C% g& m$ M5 P
<name>hadoop.util.hash.type</name>
; w0 o. E9 e! j6 n4 Q! A <value>murmur</value>1 S' h4 t: ?0 @/ H
<description>The default implementation of Hash. Currently this can take one of the
- v$ o' t Y5 ~; F/ g! } two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash.6 W3 u% ^. W& T
</description>
7 M& G1 @" l6 Y. f1 I4 U3 d</property>; Q" T/ Y- {! ~/ o+ M( \4 \ j4 i
<!-- ipc properties -->9 ^# a8 f7 ?7 Q7 q5 e1 }
<property>& o2 S' {# s0 Z# T s# Z0 c
<name>ipc.client.idlethreshold</name>. J$ e" K% y9 M1 ?* @8 Q0 B( N3 X2 \& T
<value>4000</value>
0 O! k Y/ o9 K2 I& E/ N. z7 P$ r <description>Defines the threshold number of connections after which! B z, [1 B E! T/ Y0 F
connections will be inspected for idleness. j6 d! m" U3 g
</description>, y# ~! l8 c. F. J
</property>
9 }. Z \$ G: M. C, h# \! j<property>- `9 U& j8 P# c/ v
<name>ipc.client.kill.max</name>4 u& ?* s/ C, L- P
<value>10</value>
6 T7 W2 y2 I" q% G! j1 \5 j <description>Defines the maximum number of clients to disconnect in one go.
0 \ V2 z. ?8 e7 q" _* D </description>
2 C' ~3 V, C9 n s* K: q7 J) @9 Z</property>) B+ {4 J; j; N3 p
<property># ] _' T$ h7 u6 N% h* w& E
<name>ipc.client.connection.maxidletime</name>
3 N1 |1 d$ V* k0 i+ [/ x' m <value>10000</value>* W* k0 L! e K
<description>The maximum time in msec after which a client will bring down the) E* X, J; p' l$ [, S8 ^. g
connection to the server.
+ G) E6 p- Z ]/ G5 _; W+ E </description>
6 L: e3 f1 A+ _: h& N</property>
' ]4 z$ e1 `1 F6 ?# ` y<property>! }4 B+ s, R' ^$ W0 E3 c. y2 t9 J
<name>ipc.client.connect.max.retries</name>) J5 o6 h5 g7 ]3 `" l" t+ R
<value>10</value>2 y$ @3 O4 a# l( S/ {% p( I
<description>Indicates the number of retries a client will make to establish
0 ~1 L ?6 Z6 \8 z& A4 P a server connection.
0 D0 A+ o; |! m6 d6 u </description>
& j* I. Z- [" A2 [8 r3 s# G. a</property>
$ q* d; e0 f& A- Y8 e<property>+ B- v0 s$ C! p( ?4 X
<name>ipc.client.connect.retry.interval</name>
+ G9 G5 r# b' H5 c5 @* H7 H <value>1000</value>
6 N; s' T9 K/ N; x( i <description>Indicates the number of milliseconds a client will wait for5 s0 }2 @) ]% }
before retrying to establish a server connection.6 w# o% i3 ?3 u/ p/ `6 t* s
</description> A( n/ {* n& n9 k% j
</property>+ i$ q- H+ C* v" Q# O t `
<property>
, N3 Z: k* t+ \# j6 \+ G8 b <name>ipc.client.connect.timeout</name>
/ _( ^ K. M: |6 U, W" ^ <value>20000</value>
( |, Y: Q3 w. ?* B) }/ [. w <description>Indicates the number of milliseconds a client will wait for the
& `% S7 O5 G a: e) R8 \4 v( p; I( N socket to establish a server connection.
! }% T7 I! o% F g/ F: L1 I1 h6 O </description>
& `9 A# R5 c: I/ {</property>+ |0 t) E" C* s5 y
<property>) o3 r) E) }; f+ I: h. y
<name>ipc.client.connect.max.retries.on.timeouts</name>
& H2 I/ R! [! i$ h s+ k <value>45</value>5 X4 q6 Z# t, ~. N3 E
<description>Indicates the number of retries a client will make on socket timeout
; [. @$ i$ }: a. I6 d to establish a server connection.: b4 Y* q" |; E* ]+ f
</description>& k0 Q7 }# I# k% H$ j, ~- h3 i7 \' z
</property>& h6 f( S0 u" I1 H/ M
<property>. ~, A) i$ f! @" |$ f: S5 d% T
<name>ipc.client.tcpnodelay</name>0 B' N! o! ]9 q
<value>true</value>5 n {3 H5 k* u+ I3 s+ | G* C. {
<description>Use TCP_NODELAY flag to bypass Nagle's algorithm transmission delays.
9 Y3 I& u7 C; W" Y </description>
2 _: Q( y# q% F0 a/ O; R</property>) X9 H. R0 G7 ]& P- U5 \
<property>- k7 O+ P" |7 p
<name>ipc.client.low-latency</name>
4 I; ?5 D5 H( [) J( V0 ^ <value>false</value> L5 B4 K: _/ t7 D3 j1 Q1 }
<description>Use low-latency QoS markers for IPC connections.
; g- A, [# P$ @5 |% \ </description>7 O( D: `: e1 {$ q. o
</property>7 {, x6 O1 K! t5 k" [1 ~
<property>- s5 E. M. ^! [, o
<name>ipc.client.ping</name>
9 o9 X2 Z" e1 F" `% O7 S3 T4 W, _8 ? <value>true</value>- b& v/ E/ m$ w2 G! o1 @$ a1 [3 g9 y
<description>Send a ping to the server when timeout on reading the response,
# i/ M. I6 H5 t; E0 e if set to true. If no failure is detected, the client retries until at least
0 z1 `& e0 z g W) i% V# p a byte is read or the time given by ipc.client.rpc-timeout.ms is passed.+ x1 v+ l) G1 |9 `
</description>. ~! ~; t6 x. v9 G2 x
</property>. U& {7 i# L6 I2 \
<property>
- n s+ _: h0 M% o <name>ipc.ping.interval</name>5 R* M1 j' [. h& O( Z
<value>60000</value>5 x/ q. t: [/ a, G+ Y4 ?# {
<description>Timeout on waiting response from server, in milliseconds./ p' q" k& _+ x( t: }( y
The client will send ping when the interval is passed without receiving bytes,! v9 W/ J7 h$ \ ~9 t
if ipc.client.ping is set to true.
+ V# D, h- L% a; b5 J1 \# Z </description># G+ K) Z$ x1 V% n6 m
</property>
$ ?. l2 p5 m' \. {% A: k% Q<property>
% e0 Y& z* }8 Y8 Q8 j <name>ipc.client.rpc-timeout.ms</name>
2 V+ P/ K. g4 ~+ X$ } <value>0</value>1 n( y# O; Q6 A
<description>Timeout on waiting response from server, in milliseconds.
D/ Z. g6 r4 V' ` If ipc.client.ping is set to true and this rpc-timeout is greater than
' O& D5 p+ O; \# F# _ the value of ipc.ping.interval, the effective value of the rpc-timeout is
+ x' H0 t- o0 h# ^+ x rounded up to multiple of ipc.ping.interval.
8 z! B6 f) \# f! p </description>2 w# F; c% z ]1 O
</property>
( R5 v; @- P4 [5 ^$ s; ~8 \<property>
' X# H9 ?% K3 b; h* C% u: y <name>ipc.server.listen.queue.size</name>( [7 h5 b/ Z% u
<value>128</value>
3 p$ {8 d: t; W <description>Indicates the length of the listen queue for servers accepting
& w( h# U5 @! ~) _/ i' J7 \( G- a client connections.
- m' ]7 S/ e7 {2 q# ~/ q" A </description>
) |3 h* X6 x& f5 p</property>7 \! A2 V: D; s2 O1 x/ ~
<property>( t' r1 A/ K+ ?$ p
<name>ipc.server.log.slow.rpc</name>
/ x; j; i/ l& J( r4 |! U4 v) v, k <value>false</value>0 y1 c I6 G( H# M. U- ?
<description>This setting is useful to troubleshoot performance issues for
: n) d8 J+ G# { various services. If this value is set to true then we log requests that
" v+ s' ~) @7 j3 P, K8 K: I R fall into 99th percentile as well as increment RpcSlowCalls counter.
! a" t5 ]2 b0 J* e. L0 L </description>
/ S1 s" C+ Y1 y" T5 j, N</property>; V8 y W7 o" `' S! U! r; F% U
<property>
- H# l+ J0 }& p& | <name>ipc.maximum.data.length</name>
$ M8 D" o- G A# I/ L9 {' e& s <value>67108864</value>& j( U8 U. C& J4 [
<description>This indicates the maximum IPC message length (bytes) that can be( ?' u( F0 m5 o
accepted by the server. Messages larger than this value are rejected by the0 v7 X* m9 Z; V0 t( o9 ]% {
immediately to avoid possible OOMs. This setting should rarely need to be! l/ u9 W. Z) X9 @" b! ~) Q
changed.
( ]0 d: |9 Y5 ]9 `, O4 j+ K+ p </description>
2 s. ]. y; ?! p7 a</property>& S2 t8 y l; d: U# i
<property>' `; O) o1 z0 q3 j
<name>ipc.maximum.response.length</name>
L2 L* P! C' `' r <value>134217728</value>
$ s2 _& F* B; R+ U z' q <description>This indicates the maximum IPC message length (bytes) that can be
, _& Y- j* h2 V7 {0 L8 R2 Q- d accepted by the client. Messages larger than this value are rejected k) T. Q$ W7 c
immediately to avoid possible OOMs. This setting should rarely need to be
4 W& U( [3 P4 t# X& P changed. Set to 0 to disable.
0 I8 q3 T: E9 f* q8 ^9 W6 c+ O </description>
% W! ?! t8 i/ i6 @) ~</property>
1 h% \# s9 ^2 `0 w. e, A& g<!-- Proxy Configuration -->) t. g" J' {3 Q |: A' u
<property>
, E* `- \ P2 L7 A6 M; \ e <name>hadoop.security.impersonation.provider.class</name>6 f' ?' R, U6 {" S, ~, [, u+ |, ~
<value></value>; P' r9 P! }( [% d, q1 J8 w
<description>A class which implements ImpersonationProvider interface, used to4 K" u4 a1 p. A+ O" Y
authorize whether one user can impersonate a specific user.
f$ L# ^7 _( w7 P. P If not specified, the DefaultImpersonationProvider will be used.
" l- c! l- l9 t- v9 v" Y If a class is specified, then that class will be used to determine5 B0 R( E/ g' D6 D0 f
the impersonation capability.
6 y9 |$ c9 T. f( Z: K6 D& _0 \ </description>
6 K/ R! z; b& f- g) g! x9 j</property>
8 y5 z( n$ ^1 |7 v: T2 E! l<property>9 [1 s1 T: Q1 G+ n) ]
<name>hadoop.rpc.socket.factory.class.default</name>( c+ V0 D, a8 j0 Q9 f
<value>org.apache.hadoop.net.StandardSocketFactory</value>- r2 m, X; B1 o8 }' A, D: m& L5 i/ Q
<description> Default SocketFactory to use. This parameter is expected to be
' Q. ^- O1 x5 A+ h1 C formatted as "package.FactoryClassName".
- _4 Y& g& \. k6 Q. x# N </description>
! ]' n; }2 J1 K9 O# e% G</property>
/ R/ P1 x: H# y: g1 r<property>
% e Z$ B1 D/ Z; h <name>hadoop.rpc.socket.factory.class.ClientProtocol</name>
x3 x8 I3 \, t) P5 u, w <value></value>( m/ [/ n% Q2 | U; ^
<description> SocketFactory to use to connect to a DFS. If null or empty, use
, O b+ t1 e" E# G' D7 b hadoop.rpc.socket.class.default. This socket factory is also used by/ \. q* W6 F: o
DFSClient to create sockets to DataNodes.( [( Y5 a9 T6 Z& j4 B3 D' S
</description>. U& s0 }; ~2 A% T# l
</property>
9 i- p8 n2 X/ A- A& R5 `5 h' J, i<property>
3 \4 J" \+ O2 k* { <name>hadoop.socks.server</name>
: F+ V' D( n: a F5 C <value></value>6 c# ~+ B+ n$ H9 C" Y1 k: t& r
<description> Address (host:port) of the SOCKS server to be used by the
8 Q9 P6 \+ s5 C7 N& ? SocksSocketFactory.; F `8 b! m5 C; I
</description>
# \# \0 |) u, t1 G A; G</property>: z, p) k+ L0 y W6 s4 g9 |" J
<!-- Topology Configuration -->, Y9 g7 |$ {* ]7 Z: y
<property>$ H1 e- W: Q3 H& x) J
<name>net.topology.node.switch.mapping.impl</name>
9 V. A. H" S8 H- Z <value>org.apache.hadoop.net.ScriptBasedMapping</value>
$ J' n3 k" `) Y7 H& m. ? <description> The default implementation of the DNSToSwitchMapping. It; y% R9 E s- I8 U& {
invokes a script specified in net.topology.script.file.name to resolve
" k( c# G2 Z0 f! N X9 n node names. If the value for net.topology.script.file.name is not set, the
! b6 Y: ~! l6 @' I+ J+ A) |' I0 _ default value of DEFAULT_RACK is returned for all node names.
5 E i2 ^# I! }5 Z </description>
9 w; A. E g& q6 B; ], P</property>
9 ^8 i0 K: D' b" G4 S<property>
; `* g# E& B3 t, ^5 _! u <name>net.topology.impl</name>
; [ {% o# s- L* s) E- N <value>org.apache.hadoop.net.NetworkTopology</value>
2 v5 i" F8 H" k1 p) X C <description> The default implementation of NetworkTopology which is classic three layer one.
3 x( A: O/ Q1 a, Y! ]: L) E# y </description>
4 {8 N5 b& N4 W* b, |* V</property>
8 J6 _. s) c! G4 z ]( Y* D<property>
7 m: @+ r" @8 O, G# l0 _ <name>net.topology.script.file.name</name>4 y; b! `6 Q- E3 q1 S
<value></value>
; E I d* U* J( Q <description> The script name that should be invoked to resolve DNS names to4 K" B0 {" \; T6 d- v4 y5 T
NetworkTopology names. Example: the script would take host.foo.bar as an
$ Y6 G/ Y7 [9 c/ U8 U1 A- ? argument, and return /rack1 as the output.
" q" g' ~1 d) w$ Q </description>
( A- r: D: g2 }- t' U1 S3 E8 K</property>
0 `! U) x7 u" d9 x<property>, @/ P* Q) t- o* H: x
<name>net.topology.script.number.args</name>% e& E- @6 t( ~3 z5 x
<value>100</value>4 }% V* U. S7 G
<description> The max number of args that the script configured with
6 u% n- e" W: }! X$ M4 y2 V( _ net.topology.script.file.name should be run with. Each arg is an7 _: L. Q5 U! P: C, s9 u0 t% w( [
IP address.1 W0 Q8 u* {+ g
</description>1 v4 j; D# Q) w7 ^
</property>! y0 {$ r( |) S& v7 }& q
<property>$ G( m; F% H; @
<name>net.topology.table.file.name</name>
7 I" p- r t+ N6 z; c0 a <value></value>
0 k( U/ P! N. N <description> The file name for a topology file, which is used when the
1 N' b2 ~/ M/ w$ f! s. q1 N* E* x net.topology.node.switch.mapping.impl property is set to D2 | {/ D7 h9 V8 O
org.apache.hadoop.net.TableMapping. The file format is a two column text
) g0 N# \6 U. L* H" h file, with columns separated by whitespace. The first column is a DNS or
' Z0 [4 ]+ x0 D/ ^6 f' u IP address and the second column specifies the rack where the address maps.! a5 h2 l' G& a2 s' W
If no entry corresponding to a host in the cluster is found, then( c/ C( a+ B/ w; |+ P' B
/default-rack is assumed.
- a! V! U; ] \9 ^5 { </description>7 g3 h4 X+ U" D8 N3 s
</property>: D) c; d# m( o5 l, R7 F$ l' s3 W
<!-- Local file system -->
+ z! Z4 u5 j2 t0 v<property>
6 g0 j5 Y& A. e* q. g <name>file.stream-buffer-size</name>
z8 a7 [$ B6 P <value>4096</value>+ J: }; T( Q( `- O+ a- v
<description>The size of buffer to stream files.: W! A% A: a8 U$ w3 A6 g9 C) k
The size of this buffer should probably be a multiple of hardware& t b! @0 c: f6 s6 \
page size (4096 on Intel x86), and it determines how much data is
- L7 p5 p8 r, Z buffered during read and write operations.</description>7 }' L( ?2 {. s& w% z
</property>+ ?: R8 { u$ G# \+ V
<property>. d" @( t7 x. {# o- R
<name>file.bytes-per-checksum</name>% W$ i, s4 D8 `) S& k+ D$ r
<value>512</value>
* C5 S' e% ~9 T <description>The number of bytes per checksum. Must not be larger than# i5 S" ?) Z# Q) |' @9 S3 P
file.stream-buffer-size</description>
1 e/ C! R% u0 e2 M1 Y</property>
& b( S3 b0 [( I$ W0 D3 e. b<property>
5 l6 P. @2 C2 m9 y( B9 E <name>file.client-write-packet-size</name>5 D% C% u3 U7 ]
<value>65536</value>
( I* {1 J# U, \ <description>Packet size for clients to write</description>6 g4 J3 s+ f4 j
</property>
' Y7 s; r& M$ i, _<property>
' j- I; ?& k _& y( q- s <name>file.blocksize</name>, A- G# @2 E# ?9 G h& ?& a2 {
<value>67108864</value>3 v9 J- q3 b9 Z
<description>Block size</description>
0 ~! H7 h* n, E% N</property>$ j: b0 Q& w! | Q
<property>
. W4 e5 o" `* R4 k <name>file.replication</name>: s4 b4 r: o+ p H9 g, G Q
<value>1</value>
; H% S+ u& j% ?$ g# ^4 ]/ Y <description>Replication factor</description># b' s( {; M' G, { W
</property>0 G3 [" n; d3 k- v
<!-- FTP file system -->
D0 A4 M$ L( Q3 ~4 e<property>
1 a2 t0 h; \) P3 C& F <name>ftp.stream-buffer-size</name>
6 n% n0 @% c T3 [9 p4 J1 W1 r <value>4096</value>
. O1 _! m4 y- s; I <description>The size of buffer to stream files.% Q7 T% M9 Z! Q, d; B3 o
The size of this buffer should probably be a multiple of hardware& {3 b4 @, Y1 M
page size (4096 on Intel x86), and it determines how much data is
2 G+ B# K' e! v3 X! ] buffered during read and write operations.</description>
% V$ H ^4 @; H: U7 [ X+ D5 W7 Q</property>' l/ ~$ j* e7 _# p
<property>! T+ e; U" v7 t4 d" ~9 {! `! ~4 s
<name>ftp.bytes-per-checksum</name>
0 N3 |4 i7 G! f5 O p, x7 Q7 k <value>512</value>
( ^& E" q0 W* m8 T$ Q; T <description>The number of bytes per checksum. Must not be larger than
/ P; F4 T* G$ E" z" `4 L; Y ftp.stream-buffer-size</description>
7 U2 E5 R5 \2 R# m4 ~</property>
1 g8 j5 A+ F2 |# K7 c<property>
7 b+ u7 p3 }) G <name>ftp.client-write-packet-size</name>5 l) d3 B7 l$ ^$ a8 j2 P! ?/ _
<value>65536</value>8 U7 J2 z1 l; d5 A4 M) Y
<description>Packet size for clients to write</description>
S6 U. ?5 s" g @$ B/ ]</property>
3 |8 `% P5 w/ U, Z* n6 {* b<property>0 n9 a7 d% o/ `% K/ D. W% Q
<name>ftp.blocksize</name>
. y, {# R! ^1 \ <value>67108864</value>
7 F4 l) D6 C( \0 w <description>Block size</description>
( {, K1 ~% ?! t" b: J</property>
; `- ^0 X( T; \<property>. o2 M7 p9 O# X
<name>ftp.replication</name>
3 N+ [% {* G/ [' O <value>3</value>, K2 p) z9 S+ S. f9 W
<description>Replication factor</description>9 p$ a: S$ B8 U; ^
</property>
Q/ l6 J6 N' _& Y: _' h<!-- Tfile -->
" Q O. k' H* W% u3 t<property>
" @/ f# h# U) t; O& S/ t& p <name>tfile.io.chunk.size</name>, b0 O4 \$ ]9 Y1 l4 i0 c. b0 y
<value>1048576</value>
% T5 O& s) t: D9 a <description>2 I+ J3 D' Y' X* \* K8 [- j
Value chunk size in bytes. Default to
_) r. [# N) K q! k 1MB. Values of the length less than the chunk size is
" D. }; u5 G4 j) N! h guaranteed to have known value length in read time (See also
% U# t6 Q1 y; j6 ] TFile.Reader.Scanner.Entry.isValueLengthKnown()).! l2 s( Q g. ]7 C. o, q, M
</description>' f4 \5 ~4 g& I% ?: M
</property>; Q5 |8 P; A7 c; |- |9 \
<property>
0 R8 G5 ~2 }" b) y; S$ ^ <name>tfile.fs.output.buffer.size</name>
3 Z( k1 D: B n Y! | <value>262144</value>1 I0 H1 u6 P W
<description>6 J/ U/ D& M& f! L0 j5 _5 L
Buffer size used for FSDataOutputStream in bytes.! J1 r! `- b2 p2 ^/ a: ~
</description>1 S4 F9 E2 y1 l+ y' c+ d7 R: |. b
</property>
$ N; \3 {, W1 E d( w- ^. H# v<property>
4 A2 ~# r# M3 `+ d* D8 w <name>tfile.fs.input.buffer.size</name>; E0 v6 r% B) i6 Z5 S5 o
<value>262144</value>
; ~ l" z; K3 Q/ v9 I9 \& ~$ t <description> o! R0 Z2 R7 m, m
Buffer size used for FSDataInputStream in bytes.: K; m2 o# Z- K1 a/ V
</description> [1 P- v1 r) Q% X
</property>' ~2 C: [% i b1 R
<!-- HTTP web-consoles Authentication -->
( t- d! }9 Q' ` W" h+ a0 Y5 ]<property>
3 I9 S' D |; c, t, M <name>hadoop.http.authentication.type</name>" R3 t, o) D' h! W p* } ^
<value>simple</value>
: q$ U0 O$ w& Z <description>1 s9 @: S; m, F4 F# d
Defines authentication used for Oozie HTTP endpoint.& R9 S) o p/ ` u; ^
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
* r0 _- Q4 m. Y; A) S </description>
9 M. q. P( D9 Q4 w( b0 U</property>5 v5 Q( z2 _, h2 D6 L3 A
<property>$ @6 \: Y4 ?+ C# U# n7 U# U" m
<name>hadoop.http.authentication.token.validity</name> j5 s6 K! p k" u# ]
<value>36000</value>& }) ?$ G# g$ J0 g1 e1 x, ?
<description>. p. H9 i0 [( ?! f/ ?0 C
Indicates how long (in seconds) an authentication token is valid before it has" H( ^' C/ S, D1 _3 t
to be renewed.5 |+ c, T( U1 e: Y" O3 a
</description>5 T: b. S$ `! h. \7 T
</property>+ s) q0 `5 \+ h1 M: B8 i7 a! C$ Z
<property> ?' O) V. q8 V% N8 j
<name>hadoop.http.authentication.signature.secret.file</name>
! ] g" T8 i3 D0 Z2 Y/ u' C) @. [$ i0 d <value>${user.home}/hadoop-http-auth-signature-secret</value>; y8 x' p) r% O# e6 X
<description>' a8 k' U- S' k. t( C
The signature secret for signing the authentication tokens.
5 k$ R- ?6 Y4 K1 D# O The same secret should be used for JT/NN/DN/TT configurations.
- E- }; k: E+ v& K </description>
! J8 k3 G4 {6 s- \2 L# A$ B. n2 ?</property>
' ]8 S1 m% i2 `& K+ q' s4 t" D<property>4 {6 u$ M: c# w* w- C1 ^
<name>hadoop.http.authentication.cookie.domain</name>0 s8 {8 t/ Q5 y# g) d5 s
<value></value>
2 Y( K" D. o. o. K9 P6 E <description>, s8 V1 g- K- Y1 ^0 \% I+ {
The domain to use for the HTTP cookie that stores the authentication token.
5 t/ u$ _8 P7 S t" y2 h In order to authentiation to work correctly across all Hadoop nodes web-consoles
/ g# L# b5 @$ e. P* d$ w3 d the domain must be correctly set.
: @: t4 A& k# D) M$ d" l5 [ IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings.
- _) \" C* [& \3 W, S For this setting to work properly all nodes in the cluster must be configured- b2 I- I3 M* i. l
to generate URLs with hostname.domain names on it.4 x i8 h- Y) Z! C' Y4 _" e0 t3 M; V
</description>0 Z6 V: Z8 y8 L$ ?% B
</property>7 v7 C; H2 w- ^) n
<property>
8 E( L) s V! f/ b" s& W; a <name>hadoop.http.authentication.simple.anonymous.allowed</name>! C5 F1 n5 O2 @
<value>true</value>
4 \; f7 e7 H/ g4 F7 _ <description>
4 r7 _9 n- U* G" E8 r Indicates if anonymous requests are allowed when using 'simple' authentication.
7 x$ F: ^7 y8 i- N8 w) n </description>
" @8 O+ x5 t- B2 ?6 V- a</property>% ]5 [- }: d8 H1 R6 ?3 B
<property>
/ O. c3 S* D) r$ L5 S4 F; J <name>hadoop.http.authentication.kerberos.principal</name>
! T3 e$ v4 \8 ~, t <value>HTTP/_HOST@LOCALHOST</value>
) z0 h6 c q5 E* D1 ~; H <description>8 y, W) c/ Z' \. r* t+ C& n
Indicates the Kerberos principal to be used for HTTP endpoint.1 A# Q3 d, v) S& |2 i3 q
The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
q1 q6 v6 D L* F* J b/ \0 m </description>
' i$ r3 D5 k" d, q6 q</property>
) N/ ^5 Q9 w; D; r- `6 h' f<property>& s" d. ]& T+ X- W$ N
<name>hadoop.http.authentication.kerberos.keytab</name>
6 W9 T/ j$ k- C5 J) H0 k# u <value>${user.home}/hadoop.keytab</value>4 Y0 o: O) z" L4 H& Y* q9 G3 ?
<description>) ]3 U; G" m2 N1 h4 H2 T; ?
Location of the keytab file with the credentials for the principal.
8 }) Q7 R* r; L6 U2 M6 ? Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.+ }: j& g9 ~/ @) a5 D6 ?
</description>4 z& u$ M/ P" r8 z- o
</property>: k! h' A6 |. _
<!-- HTTP CORS support -->4 b9 H) h- N2 t2 }9 N4 y
<property>
# J4 `) o9 u$ | B$ |- _ <name>hadoop.http.cross-origin.enabled</name>4 Q5 H3 i; j! d- j* P
<value>false</value>
% s( {3 Y/ p5 \3 K3 h* s9 N% j; A <description>Enable/disable the cross-origin (CORS) filter.</description>2 h/ D q, v/ C" S
</property>
) U) ?1 s7 V- o4 D5 _3 v8 Y<property>! Y- P. _" J( C
<name>hadoop.http.cross-origin.allowed-origins</name>
7 }5 X! k" J8 R* @: V" t5 f <value>*</value>
9 F9 ~ z9 E% j _# f4 m <description>Comma separated list of origins that are allowed for web services
6 ^$ B, K4 Q. S& v+ [% J6 s needing cross-origin (CORS) support. If a value in the list contains an
) v' q% Y* J. H' B% i7 K asterix (*), a regex pattern, escaping any dots ('.' -> '\.') and replacing' I1 G) P6 `% `. f" T; e
the asterix such that it captures any characters ('*' -> '.*'), is generated." |" F6 f( {! c; H' N4 x
Values prefixed with 'regex:' are interpreted directly as regular expressions,- u y! D5 L. ?+ g
e.g. use the expression 'regex:https?:\/\/foo\.bar:([0-9]+)?' to allow any" L2 m9 c7 v; |2 m
origin using the 'http' or 'https' protocol in the domain 'foo.bar' on any
& ~3 R6 t3 y& Z, D- U! a: n u port. The use of simple wildcards ('*') is discouraged, and only available for6 I. l% V( `) R) [1 h, F' n
backward compatibility.</description>/ W1 g5 E. X- ~$ h& r& D% N' c
</property>
- J/ l5 a2 b7 f e3 Y O<property>; X; H' O( O* V2 i
<name>hadoop.http.cross-origin.allowed-methods</name>4 C! {: `. K7 c# B) ^; z
<value>GET,POST,HEAD</value>) l( P* @( w# I* ~
<description>Comma separated list of methods that are allowed for web
6 Z1 M5 d4 w3 `& Y) q0 I services needing cross-origin (CORS) support.</description>7 B! w9 G) K3 M3 J2 o8 D9 H( ?
</property>
1 m0 H: E v& \ p0 ^<property>" q9 [% n; L9 h
<name>hadoop.http.cross-origin.allowed-headers</name>3 _0 u4 k, |3 t
<value>X-Requested-With,Content-Type,Accept,Origin</value>
% I7 `/ Q6 U) ~, @4 H/ ]' e. @2 V- { <description>Comma separated list of headers that are allowed for web% b: y/ Q! n) F, p2 H7 ]
services needing cross-origin (CORS) support.</description>* B. J s# h3 X$ K Q- L% P
</property>
u) }* ^( w4 l& Y8 d<property>
$ p3 n' M6 N8 u5 H- x( n <name>hadoop.http.cross-origin.max-age</name>' U! ?1 j. ~2 p k
<value>1800</value>
& Q5 N5 L" D4 ]: {: A9 l <description>The number of seconds a pre-flighted request can be cached
, z. o2 f) [2 q e$ R1 X' ] s7 N for web services needing cross-origin (CORS) support.</description>8 o# [8 _* ]( i+ q/ X" ?
</property>
5 o, s) m3 U: O" A( o- l<property>
. z$ w2 {% c4 @0 W <name>dfs.ha.fencing.methods</name>
1 J7 n1 a, _6 e" ^ <value></value>$ P8 n* \7 e8 ~" O' y$ ]
<description>
& \: e/ X! ]! U. r5 E* b List of fencing methods to use for service fencing. May contain3 O' p1 ^" d6 o
builtin methods (eg shell and sshfence) or user-defined method.
) Z6 J: V& }. l- |' X& G </description>8 j \" r$ Y% u! e7 h& ~9 y
</property>. }, q7 K9 _: g+ y z5 Y
<property>0 W+ D& J( B7 ]7 b$ m
<name>dfs.ha.fencing.ssh.connect-timeout</name>
$ q$ u H' G. ]8 R; w* t4 X <value>30000</value>4 H/ T$ d3 J; A" A3 q* k8 {7 i
<description>
5 Q& Z# ?, ]. h: L SSH connection timeout, in milliseconds, to use with the builtin% \1 ~2 c- ^6 H% d; @
sshfence fencer.
% R0 D$ H8 B' T/ y, q </description>
# j- J& J- X4 h. g L</property>( ~0 d1 U( z( n$ K( [0 J
<property>! l H" W% Y" q! {) r, d! q
<name>dfs.ha.fencing.ssh.private-key-files</name>
4 p5 X/ x6 V5 X' u7 X2 p+ Z <value></value> W5 V5 L' Z: g4 f
<description>
- r1 i1 k& S1 b9 ^- s; s5 E The SSH private key files to use with the builtin sshfence fencer.
- \7 D8 ^$ ` A$ d </description>4 r+ i) a' p. A! u8 i
</property>
. }7 d8 F ^9 \0 X3 W" h f<property>
0 P2 r: |) o/ t* z6 x& S <name>ha.zookeeper.quorum</name>
' @( K8 k% [+ N( H, F$ g <description># H% V( f" B) Q# Z$ a6 {2 t
A list of ZooKeeper server addresses, separated by commas, that are
- P9 L, l( f) {. c to be used by the ZKFailoverController in automatic failover.
6 Q1 O' U# K% b2 o1 ^/ k </description>
2 Y+ B. ?) B, |2 {</property>, u7 h9 @) D: j7 s% c
<property>
( Y% U6 u. j" H& u3 [ <name>ha.zookeeper.session-timeout.ms</name>
5 S9 n& _ u0 u5 G( M& P( j <value>10000</value>( J! P; b8 F3 `) o8 u7 d2 i0 z; M
<description>
& t1 d" i1 p/ A2 [( i7 k) B The session timeout to use when the ZKFC connects to ZooKeeper.' ^2 F: d$ I) }7 t
Setting this value to a lower value implies that server crashes0 z0 M& g! e% _9 S
will be detected more quickly, but risks triggering failover too
, S. z$ d/ e* w1 L aggressively in the case of a transient error or network blip.+ t ^& |9 r$ g
</description>9 Y; c5 e! e3 _. Q# V5 D
</property>
' d# b( T( i: v3 M( I8 M4 v( o<property>
}# Q3 O$ R; J* N: O: S ] a <name>ha.zookeeper.parent-znode</name>" m) p& n9 j4 x
<value>/hadoop-ha</value>( j. ^& f) e8 D( J# X
<description>6 w( K6 h$ }$ \- @+ E$ n& C
The ZooKeeper znode under which the ZK failover controller stores
# F2 [9 [) R j) d! U7 Z' v9 x its information. Note that the nameservice ID is automatically
E7 V4 ^+ n# c& p: y# p appended to this znode, so it is not normally necessary to3 {! r- z9 M9 l
configure this, even in a federated environment.
" {) L( F- J6 ]$ b4 Q) i; ` </description>
# R1 G) D2 }! v' _( @</property>
. _7 z. P. P5 D: O/ e& l<property>
; |) c# ^3 {: n3 E7 W <name>ha.zookeeper.acl</name>
, J$ L2 q( N. _( V' g' m% c <value>world:anyone:rwcda</value>
* s# g3 q$ D$ s! D$ a <description>, G! B) \% ]4 ^5 ~, C1 I
A comma-separated list of ZooKeeper ACLs to apply to the znodes2 O9 I8 y |$ R) @
used by automatic failover. These ACLs are specified in the same6 [5 l; V; b! z% D _: F- n: t
format as used by the ZooKeeper CLI.
. T% q/ |, ~# n; d If the ACL itself contains secrets, you may instead specify a
t4 k5 u5 Y8 y+ {, p( B9 |8 B* A( b path to a file, prefixed with the '@' symbol, and the value of5 W4 p& |2 g, b+ T, i' s8 Z8 f
this configuration will be loaded from within./ @+ \6 K# p( j, A) E
</description> D7 r x' ^7 Q. B8 C! Z
</property>
( q0 x9 G9 c* e<property>; Z$ z5 h" A9 o% w. }# b4 ^
<name>ha.zookeeper.auth</name>! R, h5 j( N( {. L9 I
<value></value>7 _& P: o3 f! E! {; I1 D* g1 G
<description>6 P/ s2 Z# P% s) l% Q: S
A comma-separated list of ZooKeeper authentications to add when: R) T6 p4 E& X
connecting to ZooKeeper. These are specified in the same format/ k6 X1 s, t5 t1 M, F' d5 H
as used by the "addauth" command in the ZK CLI. It is
# S ?6 S3 [. y important that the authentications specified here are sufficient$ r* w* J7 Z) y/ V" C j, ]& I
to access znodes with the ACL specified in ha.zookeeper.acl.; }$ r" h, O* c3 W$ C' X9 \
If the auths contain secrets, you may instead specify a
8 f6 _7 I: y% \; y4 V$ o+ Y( q path to a file, prefixed with the '@' symbol, and the value of) I# k( E2 O; P2 ]5 ]6 O
this configuration will be loaded from within.2 @( ~4 P8 N. \) g6 p) O
</description>3 X$ F( @) r( [, i& x
</property>6 N. y! r( l( @0 e, ^
<!-- Static Web User Filter properties. -->: ?* J6 Y" D- P* f* b! V
<property> ~9 X& R( l& E" W1 Q
<name>hadoop.http.staticuser.user</name>
8 J* [2 n( K y# _* w' y' l" i <value>dr.who</value>
' c7 C9 Z! ` l0 |1 ~7 X <description>' w6 w6 b# e% @+ A4 L
The user name to filter as, on static web filters' U0 B7 G3 [0 i" R; p
while rendering content. An example use is the HDFS( v1 ?* w& Y2 i0 P1 u
web UI (user to be used for browsing files).
0 i0 e6 k5 N- {" { </description>( b( i2 v0 M1 T! y
</property>6 E. [2 F, u. u
<!-- SSLFactory configuration -->4 k: X4 o* I" q& {
<property>3 v# V; }3 p) `: U8 _
<name>hadoop.ssl.keystores.factory.class</name>% k8 |5 ? s* b/ p- b4 H
<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>+ o. d7 f; P! ~# N9 Q: D I6 S
<description>( S# s% u4 V G
The keystores factory to use for retrieving certificates.2 S) H% r n' W% K8 M4 V% k
</description>* T$ p$ H: l1 i5 q: V! d& p" U
</property>; o' j1 t6 y) A: x( K5 ~" `
<property>
# e7 O/ x* _. o A0 N, g <name>hadoop.ssl.require.client.cert</name>
: {3 i8 \- y; G <value>false</value>- e7 z: g/ q8 c: ~6 j. \. U- G
<description>Whether client certificates are required</description>
2 f# L$ K1 ?( `( V5 W</property>) e" e: A- y5 @
<property>
* F( r' g* i' c# J% e <name>hadoop.ssl.hostname.verifier</name>% p! m' Q$ H0 K) t& d3 F8 X
<value>DEFAULT</value>
: p4 d, a3 `$ N- u <description>2 H& R% D7 }! e# ?; w. l6 J
The hostname verifier to provide for HttpsURLConnections.
8 a% C8 g: Y6 o7 } Valid values are: DEFAULT, STRICT, STRICT_IE6, DEFAULT_AND_LOCALHOST and
. [! m$ h% i1 i) j ALLOW_ALL" a5 P/ S7 w8 _' t$ y
</description>
0 h2 n8 D4 m+ @/ f% i</property>$ T+ X) \7 D% V0 @: m* K% O5 H
<property>
( r( y1 r! ?4 \: ^) c( `0 a9 N4 h9 A <name>hadoop.ssl.server.conf</name>7 @1 q3 U- |8 Y+ e0 u. W
<value>ssl-server.xml</value>5 W% x: N1 K3 d- b" L
<description>
; k! c. F- `) X! M W% L Resource file from which ssl server keystore information will be extracted.7 P! v" q2 Q8 d2 k# Y- c6 A2 s% m
This file is looked up in the classpath, typically it should be in Hadoop5 k" ~* L! N: ]8 F+ W& W
conf/ directory.( Z! t0 z+ } g0 a
</description>/ d6 Q- b% b @# m. O# C6 n
</property>
5 Y; J& l3 E; G5 `/ u2 j: s<property>2 {: }, N( ?$ W% B: b
<name>hadoop.ssl.client.conf</name>/ Z; T* \7 a, m+ x
<value>ssl-client.xml</value>
' L0 W% L4 E8 _% z3 |* E <description>
7 r% y5 A+ C0 f9 ?$ S" ^ Resource file from which ssl client keystore information will be extracted
3 ?' e7 m ?5 a$ r This file is looked up in the classpath, typically it should be in Hadoop
- P, W t. u) S conf/ directory.
; o! C. W1 g3 a: y6 Q& Z </description>/ u; p o Z3 g: z2 G
</property>
) G+ \% f8 e V3 |<property>
. g# o% q' v* V! d H, \4 v$ N6 m1 L0 z <name>hadoop.ssl.enabled</name>
6 n3 ~ h. e J/ v" r <value>false</value>- O7 q! y% a/ ?2 ^' F
<description>
* x' ~5 l# g1 n' b, } Deprecated. Use dfs.http.policy and yarn.http.policy instead.
7 B# {4 F/ r; n4 k </description>
3 K: t$ a3 @9 f; l `. |5 ?</property>6 }7 ?+ m5 |& g% }* n$ M3 O
<property>
# V9 ~; ~* h) p4 P1 y3 @ <name>hadoop.ssl.enabled.protocols</name>
' ~% f R9 [# V3 P <value>TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2</value>2 I; z) `8 z7 ?& F
<description>
. j' C. ?* f2 H6 d1 R The supported SSL protocols.3 v+ F! B4 `+ G- n% s6 x) J% |5 ]4 N7 e
</description>: g. U, D9 m4 X7 s! s+ C
</property>* h8 m! G$ e- P& ]
<property>
, D7 B! e6 j7 e3 } <name>hadoop.jetty.logs.serve.aliases</name>( [: w: Z! E* x% V
<value>true</value>
O4 i" y- ^1 m" v2 y: ` <description>
5 c9 T) X7 X8 A/ ^! J' y Enable/Disable aliases serving from jetty2 Y- J4 d# k( C! e, [! P, X
</description>) S, A' ]/ [1 l
</property>7 l, B! w1 D+ o* p4 @, A
<property>
3 I s2 J5 S! L- Z" y+ o <name>fs.permissions.umask-mode</name># Z ~6 r& |, A9 k( _6 D
<value>022</value>; L. w+ p1 K) l3 ~# q
<description>8 n) r/ _& S! H4 }3 E d% A
The umask used when creating files and directories.5 o" G0 j' E) q. k0 r8 j
Can be in octal or in symbolic. Examples are:
8 p/ d( D' R6 o, p "022" (octal for u=rwx,g=r-x,o=r-x in symbolic),* v; E- X5 P! G
or "u=rwx,g=rwx,o=" (symbolic for 007 in octal).+ h7 e" C5 D/ F' P7 X/ W$ m8 ^
</description>
; {& ^8 N5 w1 B+ n$ L8 @</property>0 Z1 p3 n! d( O( `$ O- U
<!-- ha properties -->
% ?; ]5 Z, ~9 r5 ^. ^: z y( ~; j, i<property>
1 f8 {- p3 @- u% A <name>ha.health-monitor.connect-retry-interval.ms</name>
; M+ R0 T/ S: l; y) E+ F$ l <value>1000</value>) H: M2 G$ k, S, ]! {2 m2 e" l
<description>
6 L) V5 Q$ c3 D; o3 d; Y5 U How often to retry connecting to the service.
/ @4 J: w, G0 @2 G# }6 S v4 h% o' ^$ t </description>2 e" o1 Y( n; l0 @
</property>
. S2 _, n0 i$ |$ J<property>
/ n5 a# q4 H& K Q% i9 t, e$ T <name>ha.health-monitor.check-interval.ms</name>2 z' N' L7 X3 q6 J/ y
<value>1000</value>/ |! N0 e7 _5 w+ [* ?. D7 u
<description>" X0 _4 B+ B3 ]9 K
How often to check the service.5 T! [0 F! ~- h+ f$ ~/ u
</description>3 ?& O; T. ], M' e
</property>& _- v+ T+ W' j1 g6 f2 P1 {+ X
<property>
# M! q+ r+ p$ u <name>ha.health-monitor.sleep-after-disconnect.ms</name>, g W# e. u, e3 @" ~2 F
<value>1000</value>
# z+ I2 u8 K) E: \' ?' c4 N4 F <description>& E1 h: K% s; \4 `- C
How long to sleep after an unexpected RPC error.
8 A9 d/ {5 q/ S6 i2 I7 W* H- Z: k </description>6 _- _; ]% I( `' R
</property>1 ?( e3 `3 D8 R1 l
<property>( ], k2 n5 t c9 M1 p8 O
<name>ha.health-monitor.rpc-timeout.ms</name>8 ~% d% l4 c' K& F2 H1 u0 u) w0 T
<value>45000</value>0 S+ N4 L9 B) O- [6 h- x' Z) @: Y
<description>$ j& g( J2 P) E! i$ H& g m4 U5 H
Timeout for the actual monitorHealth() calls.8 v' S o+ n% D& Z
</description>8 Y/ D& k( H8 z% c2 P$ O1 W' u. m, I
</property>
, F5 T- X3 P1 K) j0 \2 B8 z$ ]* l<property>( X" t8 i; ~4 x- G N1 k3 h3 `
<name>ha.failover-controller.new-active.rpc-timeout.ms</name>
6 o8 ?. Y8 t( J- q8 u( |5 A8 M <value>60000</value># | d2 |! c. D7 u. |# e- a B) o
<description>) [8 m& [4 K }9 g. Z
Timeout that the FC waits for the new active to become active5 E4 @5 y9 s# K- J; ?) g
</description>
2 G2 @* W( S D ` Y</property>
5 B) J( d/ P: \, N4 H<property>! Q- K# T) w6 J t" E
<name>ha.failover-controller.graceful-fence.rpc-timeout.ms</name>
* D* r4 l0 {# |- N" D <value>5000</value>
2 D4 N& s' K+ I& E) j4 ? <description>9 O( r2 [0 t4 X- M8 ^6 |3 {
Timeout that the FC waits for the old active to go to standby+ |& p& b U/ w, [
</description>
" h& H' Y8 ?; z( K+ W</property>! b7 S! d9 L5 T. \) _$ r$ p* [
<property>4 Y0 S. g% b6 _4 ~; F. d' y
<name>ha.failover-controller.graceful-fence.connection.retries</name>. S# y0 H4 ]4 q9 y" S
<value>1</value>
- d( E& m+ ?; s6 p5 v <description>
' D- j) _: u3 w. e" E/ C" g* \ FC connection retries for graceful fencing7 C- c" e1 G3 u9 H7 G
</description>. r r1 N1 a6 R9 ~8 F& n
</property>) o; D. m3 B& y" A* o
<property>
" Z, `2 ]6 Y, ?( D; ^ <name>ha.failover-controller.cli-check.rpc-timeout.ms</name>
# `4 Q% ^% W& N% @2 D <value>20000</value>
0 _; u. _( K: q9 d, | <description>2 k8 ^9 a6 A% @, t
Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState
& `2 f0 l$ h! K( t8 }4 _ </description>* j; J Y7 ?# X) {& J! p
</property>/ ` v2 D. o( T
<property>
1 |( x9 }; C7 p" ^. i( T <name>ipc.client.fallback-to-simple-auth-allowed</name>: K1 n7 y' C& J9 H' V* M; A
<value>false</value>
8 x" h6 G* B$ O! H* U; s0 V$ U" x <description>( @( F4 O4 @( g
When a client is configured to attempt a secure connection, but attempts to
3 }$ s' U' X9 i" w5 w- V% B ` connect to an insecure server, that server may instruct the client to/ \# D2 K( b4 P9 n
switch to SASL SIMPLE (unsecure) authentication. This setting controls
V' t. n% S+ X) b) R1 _ whether or not the client will accept this instruction from the server.
% ?0 E8 R1 W! b1 d: P$ E) }" j When false (the default), the client will not allow the fallback to SIMPLE& M5 M2 y* P/ D. f/ q0 H0 j
authentication, and will abort the connection.
* U& r4 c4 ?# Y# O7 } </description>
$ r* V& v1 M/ G y2 `- ` R</property>0 W% Q& _3 I/ h" A: r
<property>
/ j; D$ R2 ?) A- u5 X% M <name>fs.client.resolve.remote.symlinks</name>
2 y! ~" M0 d: n; n9 ^ <value>true</value> I' w9 c, E1 y! t; X- L" g
<description>- p1 b, W2 K; f- n
Whether to resolve symlinks when accessing a remote Hadoop filesystem. E1 e' j Z3 I6 u' ~6 U( z
Setting this to false causes an exception to be thrown upon encountering
9 S" w# a' K: t0 V a symlink. This setting does not apply to local filesystems, which
+ r) {( w- v8 r2 _$ ^' U automatically resolve local symlinks.
; f7 {1 T. Z$ O, ^% Z% O$ T </description> x$ s0 Z# E. ] ~$ [/ f
</property>/ b7 i( {7 g* z' f
<property>
( k U0 M" I1 @7 i, E# _# ^% @ <name>nfs.exports.allowed.hosts</name>
- }9 o/ I. N {( R5 z$ ^ <value>* rw</value>- c o, J7 f+ M- v9 h9 I( s/ m
<description>
5 j+ ^; y& Y: } By default, the export can be mounted by any client. The value string
0 k) J- z' F: b% A+ _+ T8 B contains machine name and access privilege, separated by whitespace
- ^8 d! x1 x" g8 Y7 W+ u7 f characters. The machine name format can be a single host, a Java regular& M D9 U+ }5 F' }
expression, or an IPv4 address. The access privilege uses rw or ro to
( x% d7 L ?3 h/ p* E8 j( G specify read/write or read-only access of the machines to exports. If the& v+ @+ f, O8 B9 D" Z6 J% U& d
access privilege is not provided, the default is read-only. Entries are separated by ";".
: v- W6 e; b8 ~6 q6 Q8 ^( w For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;".* n, y% _ C0 h3 ]4 \* Y" Z8 G
Only the NFS gateway needs to restart after this property is updated.
7 q/ p$ o* B: C </description>6 I$ r, v+ t; y
</property>
_" ~7 m1 O$ F, d+ x' d0 w<property>
, j) H# ^7 v4 K8 a <name>hadoop.user.group.static.mapping.overrides</name>
^' q; n) O0 w <value>dr.who=;</value>
. C6 }2 M r( m S <description>
n( a/ q% [& r! A+ X& p Static mapping of user to groups. This will override the groups if
) U5 s; W; w$ }0 S5 a available in the system for the specified user. In other words, groups/ g5 t2 S) V, W" D1 H2 Z
look-up will not happen for these users, instead groups mapped in this4 H1 D& ]9 O# F: O X' G
configuration will be used.
- r( u! w8 p) H- P* A4 @) q Mapping should be in this format.; e4 K5 n5 G1 G0 j2 C4 n
user1=group1,group2;user2=;user3=group2; J7 E* y O6 \) U
Default, "dr.who=;" will consider "dr.who" as user without groups.
0 K9 f0 D0 z3 {/ {( L </description>7 R( A6 W) X0 O9 ^0 K& N
</property>2 G/ l) S" ]: w# I1 F! b
<property>; g& P9 q; ?- {( e% m: d. s
<name>rpc.metrics.quantile.enable</name>& q$ H) k# L, j5 ^$ Q1 X1 c
<value>false</value>1 A1 ^/ P% s$ J: t, r, U/ U
<description>0 A) b# H1 {5 ?: X8 C7 ]/ e
Setting this property to true and rpc.metrics.percentiles.intervals/ a4 V1 x) i& }$ i; N2 ?
to a comma-separated list of the granularity in seconds, the
! P. ~) x, \5 H f8 O3 \ y 50/75/90/95/99th percentile latency for rpc queue/processing time in
8 e% D' g: L# H7 z! ?) u milliseconds are added to rpc metrics.
, t+ X. h) B6 W. g/ o1 Q </description>3 D% a0 Q! x4 A5 U9 _" ^
</property>- @6 i7 \- X$ a! {0 d/ i9 f# X
<property>- U) u S& \5 A8 b; ^
<name>rpc.metrics.percentiles.intervals</name>
6 ]6 i7 x: f# Z5 [4 W, _ <value></value>$ ?, d: O. d3 ^! z' d+ d* \ t2 @
<description>
8 v; l5 v Y7 ~ A comma-separated list of the granularity in seconds for the metrics which
6 P1 J$ k; n) w3 y% T# ^ describe the 50/75/90/95/99th percentile latency for rpc queue/processing
$ a6 e0 B' H1 ^. m time. The metrics are outputted if rpc.metrics.quantile.enable is set to
( ?, J! D* ~ E! X true.+ g$ g% N3 d% j( v5 k' e" B. p
</description>' B7 h1 K3 ~/ c' U- F9 q* w1 s
</property>! j4 v; K' d0 j3 P/ |
<property>" a8 i" X' v, _8 ^! Z& c
<name>hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE</name>
, u% ^8 n* Z2 r <value></value>
6 ]6 {! x& a7 b5 O% D <description>
5 \/ R' Z) I+ D( U The prefix for a given crypto codec, contains a comma-separated
4 H, @( u# G2 _4 x) D/ ?4 ? list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE).
4 c J0 M/ `, p- k1 W% j The first implementation will be used if available, others are fallbacks.
$ B# N9 U( {1 Y* N' z3 o0 ` </description>) Y) [2 g- i2 D7 G% k
</property>
% |, Q* C' s- E# A) b$ ?7 F<property>
/ p. n$ e, F" `% J3 Z* A/ V6 Z( u& X <name>hadoop.security.crypto.codec.classes.aes.ctr.nopadding</name>
# c8 k- K1 O: A9 z" \: x <value>org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec</value>
* {4 G+ z6 R; G& g+ G, o <description>3 X9 J: x: R# D& v9 |8 O
Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.4 ?8 e$ I5 V& ?3 m- O
The first implementation will be used if available, others are fallbacks.
7 ?& }' z4 q2 S# ? </description>/ N/ D7 m" X; n
</property>
: |2 V E# h: a" A<property>
& _! s4 h+ c1 V6 f <name>hadoop.security.crypto.cipher.suite</name>' G( Q- u! k) m, o
<value>AES/CTR/NoPadding</value>
- E" \7 q8 h; S <description>. t& l) ]- ^- o: r5 X
Cipher suite for crypto codec.
* Y4 C; g/ s6 v/ w- @$ u4 I </description>
. A+ |7 _5 W& F: y* L, l</property>2 @% m. q4 w/ ~& _& h: Q
<property>+ f. K2 h8 |1 Q- s# B
<name>hadoop.security.crypto.jce.provider</name>
; i9 ?1 W) l- x( [, N& { <value></value>
) k& q+ [3 A$ `- O, C+ S+ N. K6 p <description>
% G' C4 g7 w) D/ V1 N1 Y8 N7 H% r The JCE provider name used in CryptoCodec.7 |/ _' A: y1 E
</description>
# ]3 Z$ c2 Z$ L$ g' U- A6 o* s; c</property>+ u) ?) \6 }& ~, U
<property>
/ q0 w7 s6 c' v# j/ t7 V <name>hadoop.security.crypto.jceks.key.serialfilter</name>
+ ^: A; q2 L5 o: i' b <description>
# [" ^9 K: b& _ Enhanced KeyStore Mechanisms in JDK 8u171 introduced jceks.key.serialFilter./ }& d6 {8 Y1 ^8 Z' \
If jceks.key.serialFilter is configured, the JCEKS KeyStore uses it during
2 Y6 t2 q* {$ i( U& j; g6 K the deserialization of the encrypted Key object stored inside a- h& a7 G) V: Q9 \" P
SecretKeyEntry.9 f) l+ e7 a2 B
If jceks.key.serialFilter is not configured it will cause an error when
* U7 r, r5 o$ b4 T0 s Z# P* j, g recovering keystore file in KeyProviderFactory when recovering key from/ K7 y1 ^. D' @( H( C! M& A7 U
keystore file using JDK 8u171 or newer. The filter pattern uses the same# I- }* [! K+ E. u# l
format as jdk.serialFilter.. t+ b5 K: U0 z
The value of this property will be used as the following:0 K0 o: i" T* X0 w4 a) l" E
1. The value of jceks.key.serialFilter system property takes precedence$ ^7 b9 @. m r# z Q
over the value of this property.
( G/ H4 ~& I% i6 x x3 A 2. In the absence of jceks.key.serialFilter system property the value of* R* u$ D! M2 ?
this property will be set as the value of jceks.key.serialFilter.
$ H2 I- w+ c0 e- r) ~6 h: J# ` M 3. If the value of this property and jceks.key.serialFilter system2 ]1 R" v7 P( c4 ?; |* p# P2 Z( F! x
property has not been set, org.apache.hadoop.crypto.key.KeyProvider
9 t( j9 u) @) y ?7 r. i) n; d sets a default value for jceks.key.serialFilter.; a* Y T- Z; h+ \3 } L
</description>
0 s9 z/ J4 P/ k C</property>! A5 ]9 I4 \# z$ ^" C
<property>9 k' o) Y# ^: g
<name>hadoop.security.crypto.buffer.size</name>
: |+ `& f3 T" K: ?2 w <value>8192</value>) B0 [2 b, r7 H/ q
<description>0 ?8 I5 [0 r! L: ]
The buffer size used by CryptoInputStream and CryptoOutputStream.
9 e" T; x$ p1 ` g </description>* {2 q% v- r# Z: s$ L0 F( k
</property>
9 h; }& Q& S- F! ]1 X- M* i<property>) B$ C9 G* `. Z3 S6 i
<name>hadoop.security.java.secure.random.algorithm</name>% j, T$ m5 y& h1 O4 J/ Z
<value>SHA1PRNG</value> r# h9 j& n- |3 `* }* l Y
<description>
1 |2 G/ z. X0 A A; i) Y The java secure random algorithm.5 o, q' A# t0 v5 U. t1 F
</description>
. V! @. v8 ?) o' t</property>
, K9 f6 x1 M! C h7 S; h<property>1 n6 j; f8 s M+ v
<name>hadoop.security.secure.random.impl</name>! y' R! J# @* e [9 r' j
<value></value>
" R' T) M/ G& ~ <description>* |7 |3 A( Z; R2 [5 h5 @* c1 F0 G
Implementation of secure random.0 [" \2 X7 g0 R. a+ z7 v3 A
</description>, [- U+ q: _( |+ q8 D* D
</property>* o0 F% e* h& r7 y* z
<property>; t8 o2 ?/ l" ?9 G$ |0 z9 P
<name>hadoop.security.random.device.file.path</name>
) s# T3 S4 H& n N <value>/dev/urandom</value>
( o3 {+ n3 W) N% V <description>
7 j k% }$ ]6 ^ OS security random device file path.# b" B1 x5 i2 f
</description>
( W4 @# o+ Q0 N5 F</property>- a* c5 Z& u) S9 g9 Y
<property>
) d, V6 {2 P# d <name>hadoop.security.key.provider.path</name>
$ I ]/ S4 Q* Q) ]0 M/ X <description>/ y2 \' h; Q3 A: M' S
The KeyProvider to use when managing zone keys, and interacting with( s$ y8 k8 S9 o
encryption keys when reading and writing to an encryption zone.' Q9 S& j P) a: {
For hdfs clients, the provider path will be same as namenode's- s. R; ~$ u5 P
provider path.& C! T) \9 j$ q$ S. R
</description>, X [4 w' H I# {, P
</property>
1 w+ T3 v: ~# H<property>
' A3 f) U A; c2 h' `$ ] <name>hadoop.security.key.default.bitlength</name>8 S2 n) a, w7 }: O# T
<value>128</value>) x! q& I8 D+ g+ [% H1 f
<description>
7 y- y/ E; p7 p0 Q The length (bits) of keys we want the KeyProvider to produce. Key length0 S; k, J3 Q1 @0 \3 `
defines the upper-bound on an algorithm's security, ideally, it would. w, f3 q8 U* v* e3 w
coincide with the lower-bound on an algorithm's security.: n- S5 G1 Z- {8 S/ \3 B, J
</description>
" e: T% |, n. [0 \2 T, Z</property>0 q4 y0 }5 I0 I4 w2 x9 d
<property>0 ^9 J0 Y% D5 c, P' v$ y, [
<name>hadoop.security.key.default.cipher</name>, `4 n$ _- Z$ v
<value>AES/CTR/NoPadding</value>
5 [+ }) Y$ M; Q8 h+ q# L <description>8 l U+ L; v2 k
This indicates the algorithm that be used by KeyProvider for generating
+ ]6 D3 A1 B) G: x key, and will be converted to CipherSuite when creating encryption zone.
% m0 [7 Q' z3 C) b, t4 c/ \& x </description>- Q9 D0 t a0 w- A
</property>9 }. d! ]+ {6 x5 I6 R: S
<property>
! y3 G7 U0 o; k; e% n/ P <name>fs.har.impl.disable.cache</name>6 \& |" L( ]& D
<value>true</value>
2 t: X/ r( @; e5 f4 R <description>Don't cache 'har' filesystem instances.</description>
! d q4 k( S4 T. K0 ^</property>: D) L" |7 a9 d/ o* p/ M" V' K
<!--- KMSClientProvider configurations -->
) m6 X- ?( H0 m0 l<property>- V* J' n0 T( b, N
<name>hadoop.security.kms.client.authentication.retry-count</name>7 k1 N1 d5 K' ?$ V, Q; l, _
<value>1</value>
$ w) C* u3 d! w( N& a <description>. ?+ C9 ]/ h; Z6 [. j' K1 D, R# i2 h
Number of time to retry connecting to KMS on authentication failure; G& R5 d+ ~; L
</description>
4 w/ f' c. Z2 V! O5 W</property>
' d4 L9 o+ s- j' C! K<property>
0 C$ ^4 ?3 D; L* C4 f <name>hadoop.security.kms.client.encrypted.key.cache.size</name>4 h% }5 N4 [" g" _4 I- R* q
<value>500</value>) o4 {! \/ S' G% b5 H! _
<description>' u4 @) b! J2 C) C( N) p
Size of the EncryptedKeyVersion cache Queue for each key- s1 O! _9 v% H0 b( }
</description>
$ ]4 E S# i M6 r</property>
& h+ D* j# V+ j8 @& Y. r<property>
1 e0 Z; ]+ p( n# ^& m: i <name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>
! h9 e) o) J" F <value>0.3f</value>
+ j( B7 c. N: i8 Q/ N, Z% w <description>
% Z, U2 m- `, U, j6 P/ ]+ y If size of the EncryptedKeyVersion cache Queue falls below the" W1 |# g6 z3 T1 R# u8 z
low watermark, this cache queue will be scheduled for a refill
. I$ I: C! [% g# n </description>8 I4 P% `7 U7 m) `9 E; ]
</property>) T" Q4 a. r* ]: g
<property> @) D1 ^+ f- ]& {/ J
<name>hadoop.security.kms.client.encrypted.key.cache.num.refill.threads</name>
4 |0 m1 e; u6 J. s6 W5 N! ?$ u <value>2</value>8 j/ j. _+ |0 `# R! d0 h+ y/ p) T
<description>7 L: I$ a# @2 X I' ]% ^
Number of threads to use for refilling depleted EncryptedKeyVersion9 S. |9 T' W7 O( ~
cache Queues3 q' P. \. I; M p% u- }
</description>( D2 G: S6 U1 t' @) C& l8 L1 N
</property>9 P% E2 |) T9 b9 N2 R2 i. [4 D# \
<property>
. a: p( q- a X# j <name>hadoop.security.kms.client.encrypted.key.cache.expiry</name>
& b w6 _/ O- u/ V; ^) ^ <value>43200000</value>5 [# z8 D3 N5 w
<description>4 H: L7 {$ g" \9 ~& q
Cache expiry time for a Key, after which the cache Queue for this
; f3 g8 g. g+ o( @/ T8 m o key will be dropped. Default = 12hrs
! B, F! c8 \8 T1 Q3 D# w) U! y </description>, L: m4 P$ s7 [5 ]2 J9 p; W; e& ~
</property>
# H' K# B: S l$ L4 Q# e/ q<property>" j, d% k2 F: s+ M
<name>hadoop.security.kms.client.timeout</name>
& R3 m% r3 Y" a2 n. x( b <value>60</value>
8 D* ~: E- L( T0 E" | <description>
' j. O2 m8 A# X Sets value for KMS client connection timeout, and the read timeout% I0 R; [( E8 K) d1 C
to KMS servers.
0 s# E8 ^6 m# m" q" c </description>
& Y' u6 ?- V9 o# X4 S* i; I</property>
S/ U { p& w& ~0 D<property>
( P f( r+ s, h/ K <name>hadoop.security.kms.client.failover.sleep.base.millis</name>
$ m8 [, r4 ~ K/ `9 r <value>100</value>
. y: X, N/ Q3 [1 H3 I! M( K" F <description>) z. @8 o2 k* F6 ~& `$ C
Expert only. The time to wait, in milliseconds, between failover
* T- `7 n* [" J attempts increases exponentially as a function of the number of/ q( ?8 [8 Z' f1 ?4 V6 x& R+ Q
attempts made so far, with a random factor of +/- 50%. This option$ K6 ^* f, K5 n9 l4 [* C, R
specifies the base value used in the failover calculation. The) { w& B' D0 z
first failover will retry immediately. The 2nd failover attempt# Z* E0 h; z- i7 j. _8 y( q
will delay at least hadoop.security.client.failover.sleep.base.millis. e, D# u. C7 I* z
milliseconds. And so on.! X) \3 E) X4 U4 m5 E) ?* j, p
</description>1 s( @4 U5 `' Y3 t
</property>( N" T9 c7 i/ H( s1 r- b' v
<property>: r# _. V* C3 S5 X
<name>hadoop.security.kms.client.failover.sleep.max.millis</name>) y* g. I* a4 E, g6 Y* J. z
<value>2000</value>' D; e0 `% K. E6 C% T
<description>
, d! i6 c+ W5 z3 a Expert only. The time to wait, in milliseconds, between failover# Z t& U0 d* `( H' a8 ]; Z
attempts increases exponentially as a function of the number of
% d: ]4 ]5 P _) K attempts made so far, with a random factor of +/- 50%. This option
; h" L6 @# q7 B. q specifies the maximum value to wait between failovers.1 \9 ^. k' H$ E1 R
Specifically, the time between two failover attempts will not) ]& P: i- U, I# Y& R: [( N7 ~
exceed +/- 50% of hadoop.security.client.failover.sleep.max.millis
% V! }/ ], ?- d% B milliseconds.
/ a. w- h8 o+ ]* @ </description>8 E( I& W- o9 ~$ N* r
</property>4 ?5 p4 |) M+ R0 m1 k* a
<property>
0 \0 o. i" p5 W <name>ipc.server.max.connections</name>
2 Z' U) R1 d [0 p, h <value>0</value>" F. ]" J5 f- Z- w: U1 {/ c
<description>The maximum number of concurrent connections a server is allowed
" D% |, k; n( }" O4 Q' J to accept. If this limit is exceeded, incoming connections will first fill
/ D% S% ]& l" l$ Y9 E& i the listen queue and then may go to an OS-specific listen overflow queue.2 O1 O8 Z' m$ A6 M# q
The client may fail or timeout, but the server can avoid running out of file
1 y* U+ Z( T7 c+ J descriptors using this feature. 0 means no limit.
2 b$ D7 J/ \% A; r8 N( J </description>
* }* \+ |& v9 @7 b Y</property>
, k+ \6 T/ g* V- x5 j7 U5 F' [ <!-- YARN registry -->
' C" ?: d" T4 l% e4 R' S9 r9 Y <property> z7 y- d r0 \! O! S
<name>hadoop.registry.rm.enabled</name>( G6 g! N4 [' I+ P; l
<value>false</value>" \, Y& J+ F) |- j/ r9 }$ W' \
<description>
`+ f" h0 m) t/ _3 e5 m Is the registry enabled in the YARN Resource Manager?, d' ^' W: D; A. C# R
If true, the YARN RM will, as needed.) O+ I8 W) ]6 H- @5 J
create the user and system paths, and purge
5 S: A, A: X6 ` service records when containers, application attempts+ o" @3 n. Y D, M/ G. R& P5 e% a2 a
and applications complete.9 b- x; n. @) N: C- _- s/ `
If false, the paths must be created by other means,) n/ h g T6 O) G V
and no automatic cleanup of service records will take place.
2 @: _6 |$ f! } Q/ ^! I$ V </description>
, I! J" `# ?) T </property>$ ^1 E# Y. r7 r. i$ j6 H1 ?+ G
<property>
/ F9 g; |5 V( v: l2 e. }* ] <name>hadoop.registry.zk.root</name>
5 F) q* Y) V+ q$ P6 a" S& k- a <value>/registry</value>2 ^0 |( U7 @/ c7 u; Z6 N! w
<description>
$ g: W8 a3 V+ X+ m } The root zookeeper node for the registry$ M+ b& K, x: n2 }, H
</description># v) I' |7 D* R( h
</property>( \: o5 a& I: _9 q5 E1 { `
<property>
_/ D# U, H' e# x: R) m- V <name>hadoop.registry.zk.session.timeout.ms</name>
- A, e1 r, c/ F/ q <value>60000</value>
. G) g( d5 @5 S. v. R4 s1 z <description>
: q' m R8 E4 m1 l$ H; _ Zookeeper session timeout in milliseconds
3 v% T1 a2 b3 v9 d/ u; `" r- A </description>, [4 P: t. q+ s% P! R- O/ U; `& a. g
</property>
! ~) u! r+ o3 X4 K; M( f <property>
# D5 X) K+ C2 Z$ B a <name>hadoop.registry.zk.connection.timeout.ms</name>
5 N0 \; w. o3 a2 O7 l* M! x <value>15000</value>
8 H U! p1 F, E" [: x( |: @& _ <description>' E0 j8 }! y" I I) Q
Zookeeper connection timeout in milliseconds. M: Z$ f3 f+ T) r" w. ]: d+ x
</description>6 f8 B0 ^. W2 Z
</property>6 R& ]+ S( Q9 q T
<property>
7 g0 ?7 d# M/ f" m <name>hadoop.registry.zk.retry.times</name>$ Z4 x1 f7 z: M" ]+ Z" y2 ~
<value>5</value>
4 w) ~9 b2 A* o+ y4 U <description>
9 m! p4 B& z% V) C Zookeeper connection retry count before failing7 P& x1 I. W& X' U, p, L
</description>" w1 s, \& X5 y) ~( h4 V
</property>" |. I6 S7 b3 v G3 k
<property>
0 \8 y1 v* T3 i <name>hadoop.registry.zk.retry.interval.ms</name>
2 p: ]6 ~! K/ L" b- ^1 b* Y <value>1000</value>
4 ^8 W; P" [; E* j0 t <description>
4 ~6 \5 N6 s3 x& w </description>
0 w$ ?3 E( v' F: }$ H% M </property>5 P# B' U+ R2 L1 G: E" C
<property>
3 u- e8 j. W6 r; Q! z <name>hadoop.registry.zk.retry.ceiling.ms</name>. ^ e E) K6 r2 u+ i
<value>60000</value>
5 w8 Y1 ]* X9 D8 F <description>
( s0 r. E4 |- A6 J3 D3 R$ @+ Q Zookeeper retry limit in milliseconds, during- Y4 o. ?1 @* _. @- ]8 m2 c1 N
exponential backoff.6 I; V' {/ {4 ^! _
This places a limit even7 T$ N# V: k E: y- j: o% t& S
if the retry times and interval limit, combined5 O! c, n+ R: @2 o$ b3 Z
with the backoff policy, result in a long retry
" A6 j* C7 N$ n9 u period+ R( v6 t: P2 o+ X1 J# c( d
</description>/ _' o& U b4 {; r @
</property>
9 [) R6 z+ D& W, E( H <property>7 Q o& _+ N8 |3 U2 @9 |
<name>hadoop.registry.zk.quorum</name>
3 I8 q, u& S+ p* j <value>localhost:2181</value>% K! s+ o, n" z9 A* a' d
<description>" S: W' V! B5 D1 L6 d2 C0 I& |! F
List of hostname:port pairs defining the( m& }8 m% e1 K/ A! y- g
zookeeper quorum binding for the registry
- ?1 H) h' Q% s! l6 l+ B" y" ^ </description>7 x- [& M; H# u2 Z8 z4 p$ E6 V4 q; S
</property>( g4 t& ]4 C+ R% T' W, [2 E
<property>
: Z8 T/ v" N2 J <name>hadoop.registry.secure</name>/ B1 ^3 o* }" E& C
<value>false</value>
$ L' Y8 g# j6 e6 D <description>" L9 E' t1 F n9 D1 B( s: i4 n
Key to set if the registry is secure. Turning it on$ k3 y W% {# k/ I$ e
changes the permissions policy from "open access"
" f% d& M5 J4 i) p/ l3 D B to restrictions on kerberos with the option of
5 k0 N' I, l2 t3 A3 } a user adding one or more auth key pairs down their
! W/ J2 t, E9 h2 }2 u$ T7 B$ F own tree.
: V; ^) |" w$ y/ R+ @6 |7 O: Y </description>
, P# \* v7 C- I! d3 ] </property>' ]% _& Q" I3 v' C9 R0 u
<property>
% l. o" G) W$ `$ g' l1 W: m' B3 R <name>hadoop.registry.system.acls</name>
' {5 \ Z/ y; i) o. _- u, b <value>sasl:yarn@, sasl:mapred@, sasl:hdfs@</value>' e6 {: m1 ?- C
<description>3 w. j2 ^# I, Q) K) z
A comma separated list of Zookeeper ACL identifiers with6 {, E7 ~: q4 B5 Y2 a4 \
system access to the registry in a secure cluster.- t' K3 F& U3 _
These are given full access to all entries.
' x; l w, o+ U9 V- u# p4 n If there is an "@" at the end of a SASL entry it$ ~) s1 t: n+ _, c" r& x9 P! S" y
instructs the registry client to append the default kerberos domain.; w" x4 b- U9 A% t/ b7 h
</description>
5 B3 {7 n) b; S7 S4 | </property>
- m( F' V' ~8 L9 @% j0 b <property>
3 F, q& F# b. r: t <name>hadoop.registry.kerberos.realm</name>
/ _0 w K) U: D$ f8 G6 n$ ?7 G, S <value></value>
4 S' a6 w# r( c5 e0 m- V* |* c <description>
/ a& X+ }/ q% N7 | The kerberos realm: used to set the realm of; ~# B9 L0 ^4 a: z
system principals which do not declare their realm,5 j8 H5 F q7 n* E( [0 ~6 G, v0 L
and any other accounts that need the value.; W F0 n3 C- d6 h' d' l
If empty, the default realm of the running process
7 ?) l% }% \7 {5 H- i+ p- o is used.- n- ?0 H; _- `8 h' ]' ?
If neither are known and the realm is needed, then the registry! o& O. a1 A7 p2 L6 g* r; Z- L; b! h
service/client will fail.+ [& Y) t3 q3 V3 F5 \$ p1 S
</description>. }( k6 J% f8 m
</property>
3 _; s' d h% v: D" I( E1 n <property>, |: K4 W' ?- c; ?$ q
<name>hadoop.registry.jaas.context</name>
6 i; R) R4 j! N% q- g0 | <value>Client</value>* S" z1 C$ r8 N
<description>
+ h8 X, y) C! d7 l Key to define the JAAS context. Used in secure
) S4 g+ R2 K) d8 T" D9 C mode7 |9 q3 h$ |% o. r2 G" B
</description>* F; [1 g: g) Z8 h8 ]
</property># G5 q# x- `3 \: {( q1 M1 ?
<property>
& z+ }4 W) T7 C! S6 v" _ <name>hadoop.shell.missing.defaultFs.warning</name>5 q8 L! o j. O9 b
<value>false</value>$ v# E8 u& c# i \) ?
<description>1 R+ V! j- Y) p9 _! \
Enable hdfs shell commands to display warnings if (fs.defaultFS) property
6 E" `* L/ L5 q' \5 X0 v, u is not set.
5 K# w; K% i( a$ G </description>
% x8 h" Y7 U) J! D' W </property>" D* {. ?& c& m0 V- t- j
<property>1 G4 }8 E) ]8 c A# r* @
<name>hadoop.shell.safely.delete.limit.num.files</name>
0 A* E; y/ h F6 L <value>100</value>7 n3 m- |7 T0 m& K2 r8 B
<description>Used by -safely option of hadoop fs shell -rm command to avoid
) |9 o: k8 R! n7 W! h" s+ ]5 R accidental deletion of large directories. When enabled, the -rm command
5 |9 Y9 L: G t9 u requires confirmation if the number of files to be deleted is greater than
) A) s/ _) U6 [8 ^: {1 {+ y& o this limit. The default limit is 100 files. The warning is disabled if
. @; c3 `* o1 w0 H the limit is 0 or the -safely is not specified in -rm command.
. j; Y6 t& H" p: ] m </description>
) ]4 `1 P9 L) S- ^ </property>
# J& L; J0 n( G4 }6 D5 [5 p. W( l <property>- D6 f7 a3 @: s) Z, P* Y
<name>fs.client.htrace.sampler.classes</name>
7 J0 |% k g1 p <value></value>6 L E6 A& `+ o- u
<description>The class names of the HTrace Samplers to use for Hadoop; S+ h* v5 |! k/ } x4 n" g, L2 {' ?
filesystem clients.' f- P, G* S9 r N
</description>
8 V+ N, G" b+ O# { </property>
; z1 V" |$ o S/ F7 f <property>; Y# P2 w z) ?9 ]& ^4 K/ o$ G
<name>hadoop.htrace.span.receiver.classes</name>: \* D, L [$ z3 w5 X* A9 M& ~
<value></value>. o# h' F7 E& w$ y% H
<description>The class names of the Span Receivers to use for Hadoop.
4 M* q/ z$ K2 e( \. m' ~& { </description>! |' e6 X5 f4 v6 ^" `3 V* e" E
</property>! C& K4 X9 Q1 k6 {
<property>* u+ {! }; ?4 s G$ \9 |- f p
<name>hadoop.http.logs.enabled</name>+ y' p) q; \, [5 S+ g' G
<value>true</value>
* G5 T, u) |- h- {& G <description>) _ {& [$ X& c
Enable the "/logs" endpoint on all Hadoop daemons, which serves local
& n: P* O, l$ Q, L0 N' A0 t logs, but may be considered a security risk due to it listing the contents
; X/ O" V0 [; N0 d3 x E of a directory.9 x+ p1 w& i2 Y
</description>2 u N: F$ v- \# A- e
</property>8 R1 |7 D! b- f# n9 f
<property>; G4 n6 X9 b. B8 U+ ?9 N+ I5 B
<name>fs.client.resolve.topology.enabled</name>
) l, b! P7 o7 g. |5 ] <value>false</value> l2 ~4 w- z$ e! C" j
<description>Whether the client machine will use the class specified by2 N$ h7 d, ?1 B6 m4 Q* w. Z- ?
property net.topology.node.switch.mapping.impl to compute the network4 J. W8 H. L o* D8 S5 }
distance between itself and remote machines of the FileSystem. Additional
5 c7 I+ [! h; H) ~+ | properties might need to be configured depending on the class specified I9 A h/ w& b) ]) L- X9 h
in net.topology.node.switch.mapping.impl. For example, if
Y: j+ y/ L9 F: M8 [9 I org.apache.hadoop.net.ScriptBasedMapping is used, a valid script file
6 ?9 O1 Y% G$ C6 [ z4 h needs to be specified in net.topology.script.file.name.( ]2 H% J$ u: }, E
</description>/ h6 f$ e Z/ |: p- B' Z
</property>
8 S. K5 e0 E3 ?2 A- G <!-- Azure Data Lake File System Configurations -->3 ?, ?+ Z r- o1 d. u# e9 S
<property>: l, A* v* j4 l
<name>fs.adl.impl</name>4 \. L' u7 K E! k6 ?. w5 e( T# x# `
<value>org.apache.hadoop.fs.adl.AdlFileSystem</value>: X; }9 @% k+ D y" p; a
</property>7 s/ B+ \4 O4 g% f; l4 ]
<property>
6 L: W% Q4 F. U% ^- n; N <name>fs.AbstractFileSystem.adl.impl</name>6 C" d' _' {- F% p6 W
<value>org.apache.hadoop.fs.adl.Adl</value>& T) Y! P7 Y( M* m
</property>
% w. M$ q0 W, u C( m <property>
2 R/ F5 y; R5 p' R( @$ R8 ] <name>adl.feature.ownerandgroup.enableupn</name>
; t4 o3 g* X, z ^0 | k! d8 K) j <value>false</value>) t N0 S0 I- R8 e$ Q
<description>9 O& M G' u: H% i _- ]4 W
When true : User and Group in FileStatus/AclStatus response is0 |# Z5 n" i2 U6 o1 b
represented as user friendly name as per Azure AD profile.
0 x4 W# V1 z! u* ]0 e; J When false (default) : User and Group in FileStatus/AclStatus5 [% I% ]$ \% D* L! Y7 I! Q
response is represented by the unique identifier from Azure AD6 R1 S+ {; W2 U, L" v
profile (Object ID as GUID).
: j l* k; q: L) x, a) l For optimal performance, false is recommended.( ~+ o) x/ U* {/ n
</description>; ~( K2 o$ ]* h& n# Y7 J0 ]6 a& d
</property>% F, e7 Q f0 W3 c
<property>8 M0 Z" q" [) O1 O+ x
<name>fs.adl.oauth2.access.token.provider.type</name>& T5 A! D' i8 L- B% L* c( h
<value>ClientCredential</value>
# j9 i, P) n' j: M <description>. i; F2 `4 i; V, ^$ N) f- n
Defines Azure Active Directory OAuth2 access token provider type.
" l/ j- v1 U+ \. z# f Supported types are ClientCredential, RefreshToken, MSI, DeviceCode,
) R1 C4 l5 s1 m0 ~# ] s/ k$ k and Custom.( o6 S7 N i' U* k% Q
The ClientCredential type requires property fs.adl.oauth2.client.id,, M! n" x& t- R
fs.adl.oauth2.credential, and fs.adl.oauth2.refresh.url.
# E7 Y# ^5 j/ v4 x The RefreshToken type requires property fs.adl.oauth2.client.id and
5 B# |) k5 n3 J: h/ ]* r! d* U fs.adl.oauth2.refresh.token.
5 k5 w0 B# d" j+ l The MSI type reads optional property fs.adl.oauth2.msi.port, if specified.# s+ d6 S3 V3 @' f0 b+ Q
The DeviceCode type requires property
) o2 s8 u2 z3 P4 u* L fs.adl.oauth2.devicecode.clientapp.id.
) M" L1 R, J+ T1 o4 {: P The Custom type requires property fs.adl.oauth2.access.token.provider. N/ G8 K3 T) n: P
</description>2 L6 L$ s) O/ m
</property>9 \( o# j {5 {: e
<property>
1 n/ }/ n5 x, e9 Q' c& E <name>fs.adl.oauth2.client.id</name>
5 d4 I- f' a! P% Z# ]4 _4 I! k2 Z <value></value>
- P6 b' j( s3 @! w, f3 P0 E <description>The OAuth2 client id.</description>. Z( ]& N4 u: r
</property>
. H2 L1 k3 \3 T8 o <property>2 {" g5 A6 U7 c+ w( y' z
<name>fs.adl.oauth2.credential</name>9 r! Q. o6 @8 R3 ]* _
<value></value>
; {. @ E' A1 N2 {: S7 b <description>The OAuth2 access key.</description>
, h6 T3 W+ J/ P% O' i3 {# i' b </property>8 m6 w( ~! n# |3 z; ^
<property>
! t' F! h) e& K! g# s5 ]3 k <name>fs.adl.oauth2.refresh.url</name>6 O* o* `. v$ I( U8 `5 a) a
<value></value>2 D( C3 H+ ], d# a: I
<description>The OAuth2 token endpoint.</description>
) H1 c1 `* S' W E. \. G </property>
0 _8 c, M# i4 N" X) j' e( k <property>
! K: h3 _5 L2 z0 ^2 M& x2 Q <name>fs.adl.oauth2.refresh.token</name>
8 i/ M ` P) q* E; @& N- j <value></value>
; w" R* _2 J7 Y+ V F& l W <description>The OAuth2 refresh token.</description>* N" K4 M, q9 `. O* {% Z; z
</property># m [6 [5 ~5 \# e/ j
<property>" L! v9 @: U0 Q3 q( q
<name>fs.adl.oauth2.access.token.provider</name>
' O2 {' i2 a' l0 v; r6 X2 `9 j <value></value>
, R4 q& [+ @! u3 @3 c& A1 S O9 v <description>8 R% q, m* M2 L2 d0 s% x: o% U' C7 b6 S
The class name of the OAuth2 access token provider.
1 R3 G2 h$ ?/ d B% b/ y" `0 Z </description>. U$ }, I1 U0 y5 t
</property>0 Y9 G) A0 T* t/ g
<property>* ]& j% l" w9 E8 F d: `* r( }
<name>fs.adl.oauth2.msi.port</name>0 x; ^ m6 u" l% d
<value></value>' Z: x4 q2 B9 e$ e- X f
<description>" x& V& }# Z, j6 R$ t ?* c/ ]3 O
The localhost port for the MSI token service. This is the port specified
2 V2 Y7 ?& O5 N$ s5 f5 \) C6 E when creating the Azure VM. The default, if this setting is not specified,, ?6 ^, c; M f( H ^! i" Q/ O
is 50342.7 \) `+ |0 {2 ?4 Z) B9 y1 d! c9 I
Used by MSI token provider., v! U. R, B) m# j, Z6 r# D# p* O' B, L
</description>
0 }7 C3 |- _; c h+ k </property>
! s5 ~5 f) a! _& @" W3 l- N <property>
9 G* ]$ E3 t4 g. a s <name>fs.adl.oauth2.devicecode.clientapp.id</name>: R, \/ N" _: | G1 X( q }
<value></value>
8 S. S* v; v3 h, s <description>
3 J5 e$ n8 h" | ^. `2 J The app id of the AAD native app in whose context the auth request- N7 l/ x1 n# R* H* ?
should be made.
2 H& M, k) {3 p2 N Used by DeviceCode token provider.
6 u2 U X; Y1 \- \" H8 ] W </description>
6 A1 Q2 W, f' ^9 ~- s( `' n/ _' o </property>/ l; Q9 I0 P' s I6 M3 P
<!-- Azure Data Lake File System Configurations Ends Here-->' A0 N; |0 C& S. a3 U* m# I& e
<property>5 o& S( G$ J2 Z8 ^& F: T& d- |
<name>hadoop.caller.context.enabled</name>: ^* V2 G- r' u7 @( {
<value>false</value> |4 V0 j* L! u/ Y& X
<description>When the feature is enabled, additional fields are written into
' E% H% k8 d) z* e7 Y5 D name-node audit log records for auditing coarse granularity operations.
" f) y+ M& R# N6 _7 { </description>8 M( P! u3 i' ?1 h2 y4 \# t* ~
</property>2 P4 s6 x+ L% F! V2 J
<property>
& n8 v$ B# m- S8 @9 F, u <name>hadoop.caller.context.max.size</name> n8 W" y1 n# f7 o
<value>128</value>$ J$ ^( Z$ `0 n- V: D
<description>The maximum bytes a caller context string can have. If the% X/ J3 K/ c) B i
passed caller context is longer than this maximum bytes, client will
7 d, ~7 U( `4 H# K truncate it before sending to server. Note that the server may have a
3 ~0 X. J3 X& u# I7 u% q% J2 n different maximum size, and will truncate the caller context to the# {1 u1 K6 o8 N8 x9 S1 c
maximum size it allows.# X$ v/ l# R- p. k
</description>
6 G! d* O0 {8 `3 |2 [ </property>
3 u- B! g8 M, q- F' H, e# W <property>' Z& n0 f/ F8 ^: t
<name>hadoop.caller.context.signature.max.size</name>
5 K! V, _( Z6 i1 ]: r <value>40</value>1 j! h5 [5 [! L5 I; n
<description>
, w: }: z; i4 v) f( Y( W ~0 H The caller's signature (optional) is for offline validation. If the W! m( b7 {6 Q, A' S
signature exceeds the maximum allowed bytes in server, the caller context# T D3 V' W+ ]3 _5 G7 ^" \9 a
will be abandoned, in which case the caller context will not be recorded( O" M: h9 {2 \9 _5 d& g
in audit logs.
, }1 ?4 s e( M4 Q- a: {0 s# L </description>8 D& f4 C2 z2 x$ C
</property>0 @2 d" a' L% T: V. r6 E5 T
<!-- SequenceFile's Sorter properties -->
! Z* j; [& i7 F; T- l/ u <property>8 l' h5 R. g2 E0 [7 F# T9 ^1 h
<name>seq.io.sort.mb</name>% d% r2 u' v& x) x' l- F A7 M- \
<value>100</value>
7 g. t J5 Y. V, ^& k% B <description>$ u+ n& e$ }, `9 T
The total amount of buffer memory to use while sorting files,. h- r/ d# U' Z$ n3 G
while using SequenceFile.Sorter, in megabytes. By default,
5 j: Z4 ?" q/ _ gives each merge stream 1MB, which should minimize seeks.& P8 _' J1 v x- T) w, C# @
</description>
9 s" m, x, b; @# \3 f! h; B </property>7 c+ T9 E1 ~5 i6 Q1 t4 j) M
<property>8 v# m8 a2 Q) Q& [- h9 g& t
<name>seq.io.sort.factor</name>5 @# e- Y, j% i5 k
<value>100</value>7 \6 C$ x% Y$ p/ `
<description>5 B3 u0 v+ b8 [, B3 P
The number of streams to merge at once while sorting
& l* `4 |4 j3 i- A- I$ g; } files using SequenceFile.Sorter.
2 f: S0 |; a2 o This determines the number of open file handles.
& x$ b, m- N T g: q </description>
( {1 _* X2 G0 M. X6 O </property>
$ I* R+ n: f; |4 I7 _: m4 @2 Y1 Y <property>: }; N0 A$ ^8 n
<name>hadoop.zk.address</name>6 K) r. U i, g" W+ I" C
<!--value>127.0.0.1:2181</value-->$ b4 |* u4 ?; I- Y+ _
<description>Host:Port of the ZooKeeper server to be used.
! f' l. q+ I8 m+ M# m </description>% D; } Z" @, _, G; j& S& e+ j
</property>/ @9 u- g4 D+ Z9 m
<property>
+ m$ y. |: h8 {. a1 ~ <name>hadoop.zk.num-retries</name>
" @7 B% Z! X6 c2 `" Z2 W <value>1000</value># E2 | x4 W" n4 F& B" @
<description>Number of tries to connect to ZooKeeper.</description>
) R7 T! c- d% M2 x3 Q, x </property>
% {2 k" t/ w8 |$ I& G; z- R <property>( g M6 E% G, `% X0 c# r
<name>hadoop.zk.retry-interval-ms</name>
+ z, u( a; ^! G6 J1 n <value>1000</value>5 B) `3 {* I4 Z
<description>Retry interval in milliseconds when connecting to ZooKeeper.4 f- w: R& J Y+ Y
</description>
' W' f: u3 I# T </property>
% m9 E# T& A8 n/ M" @$ E <property>+ x/ L1 }4 h- [5 f* C$ v% C
<name>hadoop.zk.timeout-ms</name>
& G! C! p4 [+ {4 j, y$ I <value>10000</value>( ], u4 q; N: \6 x n5 f' g$ n/ e J* f
<description>ZooKeeper session timeout in milliseconds. Session expiration% f/ B9 C# Y }) ]) q
is managed by the ZooKeeper cluster itself, not by the client. This value is2 m$ B' T( Y) N+ t; E1 C" m1 K0 v
used by the cluster to determine when the client's session expires.
7 K: _6 ~5 o- l8 { Expirations happens when the cluster does not hear from the client within. C; D+ d- h6 x7 A @
the specified session timeout period (i.e. no heartbeat).</description>
! m W1 S7 M% {/ w" c1 Z </property># Z$ P5 H6 P- e9 e Z9 s0 Y7 W
<property>
( n# W7 E! E4 J5 Q) U* ` <name>hadoop.zk.acl</name>7 c/ O U# X8 E r
<value>world:anyone:rwcda</value>- a, V2 v+ W2 N5 p. p
<description>ACL's to be used for ZooKeeper znodes.</description>
" o: F0 ^' o g$ U) U: I </property>
" D9 X- b u: [6 _- k. F. W <property>
, v+ B2 E$ h5 t3 V! J1 \9 ]/ h <name>hadoop.zk.auth</name>
0 h/ S F' L3 R* ^: I j <description>
1 g1 u a7 ~. E, T+ \3 M Specify the auths to be used for the ACL's specified in hadoop.zk.acl.
5 P3 p- o2 w: {4 ~8 n7 q" _ This takes a comma-separated list of authentication mechanisms, each of the
2 E# J. H5 |1 Z6 K# S' z. ~9 G6 S( {" ^ form 'scheme:auth' (the same syntax used for the 'addAuth' command in
" N+ ?9 t, t- E4 u the ZK CLI).
" r8 ^( X$ Q4 V- a2 S% E </description>
, n L5 `; M+ n: ~7 d3 o# b: | </property>
% M% {$ @5 x7 S+ ]# o, ?3 W( T <property>
# x V8 w+ N/ {6 P5 M <name>hadoop.system.tags</name>
5 H* x0 d9 Q( C9 ]! P. O <value>YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT
8 I1 S7 j( _' Z7 k$ E9 O ,SERVER,DEBUG,DEPRICATED,COMMON,OPTIONAL</value>+ g" l3 ?- x; k2 q/ H/ f
<description>
3 ~' T9 r1 ?4 [* N+ z( E System tags to group related properties together.) N; l) P2 F& @; M( C) D, j, o9 W
</description>
: I( M$ V; m1 F" g </property>
' M. @( i9 t- C1 L) G+ u <property>
* U& t$ K& t v8 C& M2 Q <name>ipc.client.bind.wildcard.addr</name>
9 ~+ R1 k$ T9 C0 B7 M/ n <value>false</value>/ ?" \) Y1 B# M/ w8 Y' o
<description>When set to true Clients will bind socket to wildcard! R3 q# t) y$ {; l3 u5 v
address. (i.e 0.0.0.0)
( E- Z1 {! J$ S# l( s; }% c' g8 H </description>
) T5 ?2 x" m2 c7 A4 \' F. H: c </property>
/ a( I- q" Y3 C! Y5 Y9 q+ q</configuration>/ ~* Y, I* O* M* J
2.hdfs-default.xml
1 @( p7 q8 h" S, @. d; E
' s" X6 t% t9 t: \, E. S* u/ w<?xml version="1.0"?>3 [( Y9 b) Y( z1 {+ v$ r' E7 s3 z
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
& V7 E! L- T* i1 Y& ~3 B<!--
% l. E0 `! W( z3 @ Licensed to the Apache Software Foundation (ASF) under one or more6 }6 ?$ \0 C) Y7 o* X7 m6 s2 N3 U
contributor license agreements. See the NOTICE file distributed with% p7 o5 _( Y& d I
this work for additional information regarding copyright ownership.9 A' G' h+ c$ B! \' Y
The ASF licenses this file to You under the Apache License, Version 2.0# ?8 x3 j1 F7 f
(the "License"); you may not use this file except in compliance with0 `( X- Z+ b! Q: y- S
the License. You may obtain a copy of the License at2 n3 |, u2 s! @ Y) y' E& @
http://www.apache.org/licenses/LICENSE-2.0
9 q' F$ l; u7 n9 l4 I* Z' t Unless required by applicable law or agreed to in writing, software
9 B$ ]. E& d6 {* j6 a1 r distributed under the License is distributed on an "AS IS" BASIS,! ?7 e. c+ o# Y% s
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied." g* B0 I9 ]4 H* m$ k3 L
See the License for the specific language governing permissions and" V/ f6 x9 t- b6 s" ^" h c |
limitations under the License.
/ d9 P5 F6 I4 B-->
" s, P! t- I6 n1 p( e7 \<!-- Do not modify this file directly. Instead, copy entries that you -->
( ^" J; z `8 r<!-- wish to modify from this file into hdfs-site.xml and change them -->
+ E: A; ? b% y$ k0 @! i" ~<!-- there. If hdfs-site.xml does not already exist, create it. -->
% J. M3 N% G2 E1 |1 Z<configuration>
2 [$ R$ c) S0 H1 j+ M<property>( p" R1 `3 o2 i0 M, w5 N, q! s: |
<name>hadoop.hdfs.configuration.version</name>
; n+ ^6 x' b$ @ <value>1</value>
$ K* R' c# R# Y) W1 Y2 B1 D <description>version of this configuration file</description>
& f( e7 N- ]5 E' r' P. R$ q</property>0 @) ^/ U6 }& A9 L- ]$ D
<property>
. x/ U% \+ x" |8 P <name>dfs.namenode.rpc-address</name>0 a7 Z, N) f, H( r! q; B
<value></value>7 v& ^ n U' T
<description># T9 S U a. C7 F- V4 Q5 G4 l
RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist,
) {! k/ H5 f, g the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1
. I0 s1 J5 }& C3 n4 d dfs.namenode.rpc-address.EXAMPLENAMESERVICE+ w2 m L1 a6 s0 p
The value of this property will take the form of nn-host1:rpc-port. The NameNode's default RPC port is 8020.
! x' p7 P0 G4 e1 m </description> ?3 f7 t- N' `8 X3 z: V3 M( S+ c" W. u
</property>' L* |" C4 A2 o- W
<property>
" _8 R: l& I1 U( r <name>dfs.namenode.rpc-bind-host</name>
: {) I0 `; R9 p8 M3 S1 x <value></value>" R( p+ [. e0 s. e, x6 K3 U( t
<description>
3 X- M% R6 l0 T$ B9 D- U The actual address the RPC server will bind to. If this optional address is
8 L5 V( z1 g- ~- J% s5 W! G set, it overrides only the hostname portion of dfs.namenode.rpc-address.
1 n+ {) t+ @. G7 Q! E It can also be specified per name node or name service for HA/Federation.* W; f, C* P) u2 C2 f- t& @
This is useful for making the name node listen on all interfaces by
2 c5 W% g( K& O2 S2 _7 h# H* E setting it to 0.0.0.0.& ?4 Y0 I5 S9 r f" [
</description>
! N0 S- M4 o7 R- A$ j3 R</property>
( Z' w3 i) E5 C( O' \4 \<property>6 g; I1 I! s( R% `* u- C
<name>dfs.namenode.servicerpc-address</name>
) r& p; `; S4 T, Q3 ^+ G; f <value></value>
. j( y2 a( K4 |7 M" N <description># P+ b( y0 A$ p# S0 o( H
RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be) t0 S) G! @1 K6 O# F7 A% G& L
connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist,- c3 @$ F/ @ p
the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1
- z7 ~! L! t# r3 L dfs.namenode.rpc-address.EXAMPLENAMESERVICE) ^: t( Y* u# G
The value of this property will take the form of nn-host1:rpc-port.& Z. y5 v/ F8 q3 e
If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.2 w' p3 y+ L/ V" J
</description>6 P- J/ N, k' n
</property>$ j# U* ^" r6 k* ^4 H9 Z" ]
<property>
$ F$ n, O7 Q9 x# y7 x1 z3 j <name>dfs.namenode.servicerpc-bind-host</name>; S5 o6 D' }7 r! `) d0 c
<value></value>
" Z9 i' W# w0 T% V0 E \ <description>8 ~: c% y6 w# z
The actual address the service RPC server will bind to. If this optional address is3 k2 @# n9 \+ s1 J; `) a2 b
set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.
/ b# `& |+ w( z: w( _ It can also be specified per name node or name service for HA/Federation.! n: A( a4 c/ C6 L! D
This is useful for making the name node listen on all interfaces by- ~! y8 ~4 s4 ?2 b3 E1 ?0 s2 K! z
setting it to 0.0.0.0.
, N2 [; V$ {: [# B* k </description>& k) Y+ n2 Q: w- H0 D1 R3 F
</property>
$ f) d) O% D: l# H. U4 R1 X# B<property>
) H- t2 j1 @4 n <name>dfs.namenode.lifeline.rpc-address</name>
" L" e: S8 n+ q3 Y+ B/ y$ p <value></value>; N& ?$ X+ n2 ^. s+ Q4 \; \' K$ F( b; h
<description>
1 k1 F7 r7 q; e" ?% _7 ?' Q* G, o( j NameNode RPC lifeline address. This is an optional separate RPC address3 o( i- A) ]' o' N: Q$ ~& { }
that can be used to isolate health checks and liveness to protect against' W( ?/ F8 r& f5 S7 Z+ q9 R% F2 G
resource exhaustion in the main RPC handler pool. In the case of p; M& V2 P) G- _; ~$ Y! V; E
HA/Federation where multiple NameNodes exist, the name service ID is added
' l" ~" ^8 w' Y; F9 W9 Q' s to the name e.g. dfs.namenode.lifeline.rpc-address.ns1. The value of this
( b& U+ ~' _* y' j- B6 k property will take the form of nn-host1:rpc-port. If this property is not7 V& k, G& H# \9 |+ U
defined, then the NameNode will not start a lifeline RPC server. By# u4 [; P+ K5 N( ?% ~% n( S! \6 c
default, the property is not defined.2 b" g& {8 w7 z: _; E p/ s" I
</description>( \( `9 p' L$ M6 v& D
</property>
+ N1 p, b* Y5 x9 {<property>8 l/ }! z5 t q H! @' L# R2 ?* [
<name>dfs.namenode.lifeline.rpc-bind-host</name>
; i' n- Q$ z5 }' l/ V& l9 ~( \! h N3 q <value></value>* x5 M- v4 h8 Q. _+ \, h' t. @
<description>
+ Q( z5 n9 u6 n The actual address the lifeline RPC server will bind to. If this optional0 ]7 a9 p" t8 ?7 _
address is set, it overrides only the hostname portion of; D5 |# k5 {) V4 b1 S7 s* ~& s
dfs.namenode.lifeline.rpc-address. It can also be specified per name node8 l# G( ^( _3 E
or name service for HA/Federation. This is useful for making the name node
9 p" E$ T3 n! Y1 L9 l; d# i' |0 L listen on all interfaces by setting it to 0.0.0.0.
9 Y& s/ E4 ~$ ]% W6 n </description>
/ i {6 j8 a2 Z/ N' x3 O1 Y</property>9 `; |' ]- p( \1 O+ \
<property>
" f ?& @2 `' ] <name>dfs.namenode.secondary.http-address</name>
7 O5 P8 C& Y' T ^2 M8 I <value>0.0.0.0:9868</value>' o$ n8 `: {: V! L5 A/ o7 Y
<description>
0 J- D [0 F |6 T0 Q1 j The secondary namenode http server address and port.
) a+ }& [. L) o </description>4 e% \8 v% y* S: s& @5 J' z% t2 d
</property>
7 X" t% S0 F- N+ G<property>
! m+ Y. w) E4 C& o <name>dfs.namenode.secondary.https-address</name>
4 E5 } m3 ?% g1 U <value>0.0.0.0:9869</value>
3 e3 x1 O& |4 N <description>
5 ~8 t. f3 A8 l+ {6 l The secondary namenode HTTPS server address and port.9 R# k2 ?: e' D$ Y1 M# ]/ P8 L9 P* g
</description>% I) [ P8 F# z$ U4 |3 R
</property>( \* u. |& Z, ~) r& |
<property>* K8 {( P4 N/ {" z" E
<name>dfs.datanode.address</name>
. _. p8 Q5 C8 z: \" q7 w <value>0.0.0.0:9866</value>* @6 D2 T( K' a/ A
<description>! P! C5 G/ ^- j) k1 F% U( D, x6 H5 |
The datanode server address and port for data transfer.
, q6 c% S' Q, X ?" a$ E% | </description>
- D! Q& c# ]) t+ K" x% }6 c</property>, n: ?" G+ \0 o- @- g M- {
<property>
; A4 h4 z+ c9 s/ A! B+ n: [$ w <name>dfs.datanode.http.address</name>
7 h9 o& G+ L2 k. z <value>0.0.0.0:9864</value>
# l! e3 E L8 \( y+ Q0 Y6 O <description>
1 k/ r% R2 h* W) E8 J9 e- E* T The datanode http server address and port.$ D3 ^1 \3 r9 g- X2 \8 v* Y) t
</description>
0 z4 U" X* ^( U8 }0 j9 E2 A</property>" g0 Y6 ]% M' x `; d' i
<property>
. ]; }9 L% H6 U$ ]* D <name>dfs.datanode.ipc.address</name>. L# {, U7 Y9 k. C1 h/ J. m5 v# T
<value>0.0.0.0:9867</value>
( t+ O1 e9 N" h# e0 M. j <description>. D2 M2 V5 M. p) o9 P
The datanode ipc server address and port.6 u% C8 ?% C2 F5 `" ^
</description>! {2 p5 W9 o2 d6 p' f
</property>
! Y, t& J. H( r+ W1 g<property>
0 M/ D" K9 k( J$ _1 D2 ?9 j/ L+ H <name>dfs.datanode.http.internal-proxy.port</name>+ M! M' v+ w$ e7 R% E$ y
<value>0</value>/ w e. y0 X4 W7 f! \) S, y
<description>' n/ e) @: m7 j
The datanode's internal web proxy port.7 A+ v* ?" d) G) M/ G: Q
By default it selects a random port available in runtime.; r2 T$ ~! b8 |) `: _2 F5 A
</description>
- f. W8 O. ^5 j- H) Y1 B</property>
1 K) o. p. w$ O$ G2 ?<property>1 d& Z2 E3 D F9 k5 X6 B
<name>dfs.datanode.handler.count</name>$ \2 J: |6 t6 I7 i1 `
<value>10</value>9 Q9 N" v, ~% b) {$ B
<description>The number of server threads for the datanode.</description>
3 n+ T0 v2 y0 n* s: b% y</property>9 z/ B* B! ]. _" T3 L4 y
<property>
: A# e0 Z4 [& {+ q! V# p+ ^7 F% m5 ^ <name>dfs.namenode.http-address</name>
: z8 f+ U8 x, Z9 O& n <value>0.0.0.0:9870</value>
" M* S6 S/ Y; | <description>, C: }9 ~; Z6 O; ^! f( a9 D
The address and the base port where the dfs namenode web ui will listen on.
2 s, U+ W! f$ \& E </description>
1 { v) s3 k2 j" h6 A</property>
! G+ \$ m5 J* D, G6 Q) s<property>" Y& ]2 W. d h8 N0 E9 C& S' j
<name>dfs.namenode.http-bind-host</name>, ~, S: U1 e3 o( Q& s
<value></value>$ c( B* O& B% K" n, W" f6 {9 v
<description>* {! P5 _) @5 |
The actual address the HTTP server will bind to. If this optional address) u7 e, r' |( d- n9 Q+ t! l
is set, it overrides only the hostname portion of dfs.namenode.http-address.6 z' Q, M/ r- [$ j8 w
It can also be specified per name node or name service for HA/Federation.
: t/ g, t$ m& }" B( r- G This is useful for making the name node HTTP server listen on all
0 ^/ W0 L% t# ] g interfaces by setting it to 0.0.0.0.
( }( g. l, }0 d </description># {1 T% k1 W. p+ F |
</property>, C2 ?$ e) x+ O2 Y: e- {% k4 c
<property>
l* g$ `; B. [ <name>dfs.namenode.heartbeat.recheck-interval</name>- E& [- g: G$ l6 |- Z- ]/ e5 k
<value>300000</value>
* d7 o$ x& A" `$ f- }6 B <description>
/ H9 c- ~( X' _" u3 @/ I This time decides the interval to check for expired datanodes.0 P3 O( B0 \3 ? p3 a* Q
With this value and dfs.heartbeat.interval, the interval of7 o6 n% }1 ~( l" m
deciding the datanode is stale or not is also calculated.4 O: b" u4 ^* M: e1 M9 X0 T
The unit of this configuration is millisecond.
6 U( [- m& U! R+ k# o u- a# f) | </description># \- v* W1 _! `. m6 p
</property>, m0 I- g3 D7 f5 e1 y
<property>3 C( _1 R6 }2 N) r
<name>dfs.http.policy</name>6 M. {6 W5 {% e' \
<value>HTTP_ONLY</value>3 X) E4 f# q2 v% [$ r8 L
<description>Decide if HTTPS(SSL) is supported on HDFS1 k; G4 V: l: q
This configures the HTTP endpoint for HDFS daemons:; p+ E, n0 ^: k: {; F
The following values are supported:& u% d" O- b1 x2 O
- HTTP_ONLY : Service is provided only on http9 j7 U+ i, Z) O' k. W' B
- HTTPS_ONLY : Service is provided only on https1 r- c5 b* `& g9 |' x& o
- HTTP_AND_HTTPS : Service is provided both on http and https
% X+ z) r+ i% {$ f: h </description>( b: k$ _ t( U& B1 L
</property>9 p7 j) p6 a* ^
<property>
. ?; ]' L6 o- |, O/ y: \) [$ @ <name>dfs.client.https.need-auth</name>% f2 X2 Z$ y. W) g m6 ~9 M- t
<value>false</value>3 S/ e+ h+ q; z8 z8 U$ T- _! ]
<description>Whether SSL client certificate authentication is required
( i1 i. M' z. {; K% w( z </description>. H" z1 q+ D& C; J4 Z+ T
</property>0 z* _! l2 l2 k" Y( }0 ^
<property>
: ]+ o5 L( {9 Y <name>dfs.client.cached.conn.retry</name>, ~1 X) j4 D% _$ x% S7 t& U/ e) e
<value>3</value>$ m, f% ]0 D4 k9 z: `
<description>The number of times the HDFS client will pull a socket from the
8 Y+ v' k% }8 o% L, M6 P cache. Once this number is exceeded, the client will try to create a new7 @8 v9 {1 P& {; c) B+ ]4 ~
socket.
7 X5 A$ u3 k. y4 u7 }# V0 H8 c </description>
0 O: C% A, R7 K: j* R) I- v& m</property>
3 T8 \' X5 w. m+ d<property>0 Z0 e1 P! C q1 q7 O! n3 n
<name>dfs.https.server.keystore.resource</name>
& E8 w& C3 P( E% G! {) d <value>ssl-server.xml</value>
8 F# `7 c( i6 n <description>Resource file from which ssl server keystore
a. H. ?$ D) j( s- t' I8 r information will be extracted
% n- i# M) t9 B% F: X; O4 o </description>$ {" C" h) j' W: D' ~5 W
</property># y. ]+ w7 x, n* `
<property>0 X# @+ R; S" c1 J
<name>dfs.client.https.keystore.resource</name>
7 s& Z2 Z/ {- Q9 X1 T+ v4 } <value>ssl-client.xml</value>% e+ s* t/ g* S% V. z& f2 F# G
<description>Resource file from which ssl client keystore
, A" V: l; S, O% n! z information will be extracted
+ h9 O3 I- f4 i: ]: `: d/ q5 z </description>; |, `! W7 R1 m) D3 H" N1 |
</property>' \4 t1 j( Q# A8 a9 P
<property>" D" c) r/ N% S2 |: b! Z \
<name>dfs.datanode.https.address</name>. X& e3 Y+ A, F' D) }. e0 H
<value>0.0.0.0:9865</value>/ U& s0 U$ m, T/ v. \
<description>The datanode secure http server address and port.</description>3 S7 e P, l; A1 h$ ^5 c8 K
</property>$ T$ {) ]7 p; \3 {9 O, O
<property>1 w- n+ ?. y5 }0 r' E
<name>dfs.namenode.https-address</name>+ o& {% I" \( I9 K" V5 U5 p9 j& Y
<value>0.0.0.0:9871</value>4 @* k Y# V# H: @- G5 a; N
<description>The namenode secure http server address and port.</description>+ L* t# c; M+ }0 o8 M
</property>
% ^0 {3 v2 H5 W4 a$ a4 |<property>, ]! Z5 }5 t" N/ y( T
<name>dfs.namenode.https-bind-host</name>+ v* h1 g1 [# a1 X
<value></value>
: U ^* Z3 W- _& |+ N <description>
$ g; f5 c, h& O/ | The actual address the HTTPS server will bind to. If this optional address/ U5 K! m# ~! S% [8 T( g7 q' B
is set, it overrides only the hostname portion of dfs.namenode.https-address.9 C7 l y5 y9 K" X0 R! R* R( B* ^
It can also be specified per name node or name service for HA/Federation.
' }' f9 m/ Z9 c8 s- x \ This is useful for making the name node HTTPS server listen on all* Z; |3 H' l N/ j& F
interfaces by setting it to 0.0.0.0.
! G9 e% l+ W" \( j </description>4 T1 ^2 i$ M p; M: r" D7 K$ i
</property>
8 x8 T9 p: |, w O <property>( d- Z8 e5 k* d! O, G) C0 T& f6 o
<name>dfs.datanode.dns.interface</name>8 J' @ P8 b/ \- w e) H- }1 b
<value>default</value> a1 l F0 Y) J" \* \8 Y% _
<description>
7 T7 O- @6 O# R; t( U The name of the Network Interface from which a data node should6 B# |( q1 A: v6 b( b. b9 Q
report its IP address. e.g. eth2. This setting may be required for some+ {! d3 @, v4 @9 F+ l
multi-homed nodes where the DataNodes are assigned multiple hostnames4 H/ Q3 e; ?* K2 ^
and it is desirable for the DataNodes to use a non-default hostname.
- S* i, v8 _0 j8 Z Prefer using hadoop.security.dns.interface over
W- y3 O3 C+ k' v) d, H- q* [# \ dfs.datanode.dns.interface.
: v" \% V7 |+ ?. i2 K& D& P </description>
' x2 P& G( T f' Q* R; e @ </property>0 E- v3 z" G" z1 M7 I7 B6 n. q
<property>* F( S2 T: B8 N% ~! h j' S I
<name>dfs.datanode.dns.nameserver</name>/ `! f6 |1 L' K3 p1 l2 g4 z! v; x* v
<value>default</value>
& p" [6 W& q8 s! ]. e* q2 N <description>
- x* Y) d/ J. b The host name or IP address of the name server (DNS) which a DataNode
, \% V# a/ ]' K& j should use to determine its own host name.
) e) F5 ~! }, P6 `, ~ Prefer using hadoop.security.dns.nameserver over. A6 e+ l& |# U- o- M# N
dfs.datanode.dns.nameserver., v, M9 U' B0 y5 b1 B' C
</description># o3 f: |! V5 v
</property> `, G s- @1 ~0 X4 I+ ^
<property>
4 n1 D' K" _" {. w" u <name>dfs.namenode.backup.address</name>/ }5 ~9 g& M5 w1 t9 p0 ]
<value>0.0.0.0:50100</value>1 T5 ?9 r( g3 o9 m& h, o
<description>
4 M8 `6 l; g$ O: s The backup node server address and port.8 z* C2 i9 Z) @
If the port is 0 then the server will start on a free port.
1 E: s. M% Z: f$ P5 Z$ I2 e </description>2 }: y2 g4 y: C6 ^& z3 t S7 e
</property>/ Y( l6 S* ]' v
<property>
8 s5 }/ k4 _1 S/ H <name>dfs.namenode.backup.http-address</name>
, T; k8 e5 D @+ `; q8 t1 r8 H <value>0.0.0.0:50105</value>$ S( _8 N1 H! j$ H2 O' ?
<description>
/ V: B9 e' B6 N4 T5 ^ The backup node http server address and port.
J4 j) a# z- z9 v& y) z If the port is 0 then the server will start on a free port. g& e8 f1 n( I9 V
</description>( }2 l8 `# g2 d7 Q4 e& Z- L; n
</property>) S% u8 M3 Q* ]- k: ^
<property>
. e1 I% k4 p/ T- u+ K4 s6 E <name>dfs.namenode.redundancy.considerLoad</name>! G2 G* H0 |5 l. E+ _
<value>true</value>
( |! x* W% t( a/ S <description>Decide if chooseTarget considers the target's load or not
: }/ G! W; F, w$ q </description>
7 q: g3 q2 Q* D</property>
; |% Y6 }, u* D$ [ <property>
0 j% F# ]0 X* a) F) S <name>dfs.namenode.redundancy.considerLoad.factor</name>9 s1 {/ c9 u* F% J
<value>2.0</value>1 I( }% {2 m" i9 k0 K* R
<description>The factor by which a node's load can exceed the average
' |/ Q$ f7 B8 m( n5 t before being rejected for writes, only if considerLoad is true.
6 K; @7 o7 C/ v </description># u) H. t5 t; [) i
</property>
9 B: |, X. v. l, \<property>7 l- M9 q/ ~3 |9 y8 ]
<name>dfs.default.chunk.view.size</name>
% [. m3 O9 s5 z# G$ B) M <value>32768</value>+ M, y6 k, D5 b9 j" _7 o0 B3 Z
<description>The number of bytes to view for a file on the browser.
9 ^) Y: q8 N1 n$ s/ x5 s! l6 [ </description>9 \+ _9 o- P4 `% e% m9 U7 Z
</property>
: @$ j& F* d2 ]" P( k# C7 {, i<property>% s# w5 ^: _; x2 J9 E9 |
<name>dfs.datanode.du.reserved.calculator</name>& F/ H& O9 M3 h, Q
<value>org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator$ReservedSpaceCalculatorAbsolute</value>/ a/ z3 z# `* E6 m- V) O. E
<description>Determines the class of ReservedSpaceCalculator to be used for
6 M/ u; {3 E5 z `. m calculating disk space reservedfor non-HDFS data. The default calculator is
9 _+ `1 t I6 u6 \ ReservedSpaceCalculatorAbsolute which will use dfs.datanode.du.reserved
( ?: ^, T5 _4 ^ for a static reserved number of bytes. ReservedSpaceCalculatorPercentage n( |$ J1 P* H% \8 Z, \
will use dfs.datanode.du.reserved.pct to calculate the reserved number* Q/ u. H) I: N, d9 d( `2 Z, c
of bytes based on the size of the storage. ReservedSpaceCalculatorConservative and' V( W5 p ~. J% b
ReservedSpaceCalculatorAggressive will use their combination, Conservative will use
. R( F7 ]4 h% _; q! p3 q maximum, Aggressive minimum. For more details see ReservedSpaceCalculator.$ o% v9 {" w- K$ M) U! a
</description>+ t# o2 y; a* U% `, }& p7 h' w2 U
</property>! f4 \8 O( L( x% v/ b
<property>8 g2 X5 w6 @; c
<name>dfs.datanode.du.reserved</name>7 h+ f: U) n. m/ ]
<value>0</value>
- V2 w# ?- K; G$ x1 l2 i& K% L <description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.
7 ? C5 e7 y9 q' G+ L, U5 D Specific storage type based reservation is also supported. The property can be followed with
8 k* y: F2 O5 x8 U3 v5 s corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
. Y0 s6 j& i. K For example, reserved space for RAM_DISK storage can be configured using property
# E- f* Q* f5 g7 F, ^1 n& O 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured3 m# Q- H- C8 ^5 D
then dfs.datanode.du.reserved will be used.' @) z/ ^7 a4 a* M/ F! G+ z/ {
</description>5 c* P* |0 h+ h- V5 u* b
</property>$ p: Z8 V+ h" }: m# ^8 ]* {6 m
<property>5 B2 r8 N8 o- o. ]# J5 C$ Y7 g
<name>dfs.datanode.du.reserved.pct</name>4 B8 m. ~& o7 s! `, j
<value>0</value>
' G" f- V% P8 R/ j' f/ ^ <description>Reserved space in percentage. Read dfs.datanode.du.reserved.calculator to see
' a+ D( R( n* c when this takes effect. The actual number of bytes reserved will be calculated by using the1 N# {% ^/ ?6 L4 F2 S( s
total capacity of the data directory in question. Specific storage type based reservation
" G- e. _ T0 N) A0 d* B is also supported. The property can be followed with corresponding storage types
9 W [& h' u4 T ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.( j* O+ a$ ~* d9 g B
For example, reserved percentage space for RAM_DISK storage can be configured using property1 r2 @/ ^% K/ T$ p1 a0 n* @
'dfs.datanode.du.reserved.pct.ram_disk'. If specific storage type reservation is not configured7 G' |' ]/ E2 H# y( b8 W, Z2 M
then dfs.datanode.du.reserved.pct will be used.8 K% z3 {* c+ n7 D, q3 f4 _1 \
</description>
8 L1 R2 ?0 d: q8 g6 d2 A1 a</property>
% T+ v+ k, [" [" T8 i- b<property>6 s' D3 \) d' Q! x' ^9 `6 N$ Q% n; m
<name>dfs.namenode.name.dir</name> ^! i7 z l, ~* [$ L: t
<value>file://${hadoop.tmp.dir}/dfs/name</value>
9 C# h% m5 ]3 f1 C <description>Determines where on the local filesystem the DFS name node
# v8 e9 E/ I A1 K( j should store the name table(fsimage). If this is a comma-delimited list/ ?( r% _7 \2 Q/ k. i; Y$ N) u l
of directories then the name table is replicated in all of the3 u/ k- S' r2 j+ N/ o+ L! H! H- n
directories, for redundancy. </description>5 E+ T4 T6 _ H/ `. C2 g
</property>
9 F) ]. t& ?3 ?3 |+ s# b+ E<property>0 X0 y2 h8 L2 B! y: b
<name>dfs.namenode.name.dir.restore</name>" L- K6 m' [" q2 j+ t1 _
<value>false</value>& C1 u7 f: Y0 \$ n
<description>Set to true to enable NameNode to attempt recovering a' e/ G. Y+ K3 c
previously failed dfs.namenode.name.dir. When enabled, a recovery of any7 z" v3 F! L2 X8 q) |# z
failed directory is attempted during checkpoint.</description>
5 {. b0 X9 n. D/ Q% d- M8 _</property>
/ j G# [2 `- |7 O) j9 Y<property>
. g- r& r6 Q& T* @" ~# I <name>dfs.namenode.fs-limits.max-component-length</name># G* x% \& s! _ J0 c) S2 ?
<value>255</value>% D( E, [, q0 @/ h* \& l0 W1 p% [
<description>Defines the maximum number of bytes in UTF-8 encoding in each
" R6 n/ {9 z) }$ g( w0 J2 l6 |5 a component of a path. A value of 0 will disable the check.</description>
/ o3 s( P. O2 I/ h, S9 O! \</property>% t$ i h5 `& y
<property>
6 a. ^* X5 R% B2 H) Y <name>dfs.namenode.fs-limits.max-directory-items</name>8 u. J1 n, g% n5 j; O( d
<value>1048576</value>
7 l" }0 a5 N" Z <description>Defines the maximum number of items that a directory may
: a( H% V& x5 F+ C, Q4 A) i. h$ Z: A$ Y contain. Cannot set the property to a value less than 1 or more than: D, Y7 ]1 G: U4 m
6400000.</description>+ x. {5 v, J) L" \, ^% l2 }+ A
</property>8 b. P% s3 ], D2 r
<property>
. w2 o5 I3 g0 E+ o4 f <name>dfs.namenode.fs-limits.min-block-size</name>- f2 s$ P6 Z4 S( ^( R- U
<value>1048576</value>* U a8 n- Y0 K- d- `
<description>Minimum block size in bytes, enforced by the Namenode at create
! ^ [% Z7 M2 S1 P ~5 t5 v$ q/ e! J time. This prevents the accidental creation of files with tiny block: N( w& _3 }2 X& t
sizes (and thus many blocks), which can degrade
$ y! G. {9 N. [7 i q2 d1 O performance.</description>
3 R8 S l4 S' g3 U: N9 T</property>
+ R) X( V) i0 Y2 z% l, q, P0 z% ~<property>
4 U, R+ P: o% p% w <name>dfs.namenode.fs-limits.max-blocks-per-file</name>. Q4 |. B/ o' f+ f! g1 p8 L
<value>10000</value>4 O {2 T0 l9 n3 `' \- r
<description>Maximum number of blocks per file, enforced by the Namenode on
5 m; \5 ^" g, n/ c% ?& ~0 M' g write. This prevents the creation of extremely large files which can# |: ?( K5 G2 T! h
degrade performance.</description>7 q% R2 e1 H) _, b% `. I
</property>
' @7 Q- M2 T+ K<property>
6 @3 I# i8 j3 ?& P% a <name>dfs.namenode.edits.dir</name>
/ ^9 [" B, M8 `1 \ <value>${dfs.namenode.name.dir}</value>
Z8 a. Z7 k- P# o# p, B <description>Determines where on the local filesystem the DFS name node
, g8 F/ Q9 u6 V/ I) f should store the transaction (edits) file. If this is a comma-delimited list
# F q( y1 I/ T% s( W of directories then the transaction file is replicated in all of the 4 ]0 c T( B* s' D7 p6 n8 O
directories, for redundancy. Default value is same as dfs.namenode.name.dir
c5 t1 D0 F7 S- N' m9 l. u </description>5 m+ k" m* k# r
</property>- h5 r) l: s: m
<property>
" E$ ?" Y* z, d' S0 W- R* Y5 s <name>dfs.namenode.edits.dir.required</name>
( w' c! r7 U C* R) z/ A <value></value>1 i1 X4 T5 Z' V
<description>This should be a subset of dfs.namenode.edits.dir,& E8 Z! p8 }9 Y! Y: v$ X& Y
to ensure that the transaction (edits) file6 J% q, @% E0 a# m+ a% O2 c. a
in these places is always up-to-date.
# T4 S2 u- D) R" j </description>5 u5 X1 K1 f. p0 t+ M
</property>
" j& s$ o4 [, s, s3 x/ M# F<property>% D; r4 ~- E1 n- b5 G; w+ B
<name>dfs.namenode.shared.edits.dir</name>
5 x( K. p5 m2 x2 M$ h ]" r; G* Q <value></value>
. U- Y: T2 y, m( b, X K3 _ <description>A directory on shared storage between the multiple namenodes
" T; V& Q# j4 a. n0 n in an HA cluster. This directory will be written by the active and read7 Z! E% r& m# u% j5 z
by the standby in order to keep the namespaces synchronized. This directory
$ e" M* p) g% j! K5 x4 q6 Q does not need to be listed in dfs.namenode.edits.dir above. It should be
- {! ?0 A/ m$ _5 k3 i- H/ \ left empty in a non-HA cluster.
4 V9 i0 x& ^8 \1 F* l6 w& W </description>
; p- g1 ?# _. L0 J</property>
% g6 H% [4 k) ~4 h% }<property>
6 C* j2 {2 u7 n+ q! c8 M <name>dfs.namenode.edits.journal-plugin.qjournal</name>
# [9 ~! y) K6 Y" f: \- P- c <value>org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager</value>
; u- R5 D$ W9 R; D# a</property>- `" f/ h# c; V) ^* E" h7 S
<property>
1 H# P2 _8 X2 S, s <name>dfs.permissions.enabled</name>6 O, y: x. l7 x# E
<value>true</value>
- p0 _; Q- w z) f9 z <description>/ h. e0 \( z$ t% g
If "true", enable permission checking in HDFS./ u; e4 w5 T7 @' R4 e3 T
If "false", permission checking is turned off,% [- P ^( X( E
but all other behavior is unchanged.
% o, J( u+ F9 A( o' b! b: K) n Switching from one parameter value to the other does not change the mode,: b6 Z4 v0 C4 V+ c$ ?& g
owner or group of files or directories.& a7 J+ C2 u6 a2 {1 l" u s
</description>
9 ?* ^; p1 x) k" q: U; G</property>
; X8 \+ b$ `' ?( ~& a<property>
# C, I3 h9 f: ]; P% [ <name>dfs.permissions.superusergroup</name>2 ^/ x9 \. Y E: l; `
<value>supergroup</value>
, G+ }$ A1 C" P <description>The name of the group of super-users./ S, B8 q% u) F- Q
The value should be a single group name.
* | L$ y& f$ } </description>& Q5 ?2 S$ Y+ X4 O1 b
</property>- M, Q5 `: z5 W; m3 S, w0 D$ G
<property>/ _1 z. E/ q. N) X G5 |
<name>dfs.cluster.administrators</name>! J- l' h, `/ d3 {% U, ] }8 [# [
<value></value> `6 f) H& |3 t/ O$ B* o6 o3 g
<description>ACL for the admins, this configuration is used to control
( S: R5 [ D: [ who can access the default servlets in the namenode, etc. The value; R5 G0 y3 T! N
should be a comma separated list of users and groups. The user list/ L+ t) r* j0 e7 l0 z
comes first and is separated by a space followed by the group list,
) j2 d4 {+ ]% G* A+ N2 Y$ ] e.g. "user1,user2 group1,group2". Both users and groups are optional,- E \8 H5 U9 z- n
so "user1", " group1", "", "user1 group1", "user1,user2 group1,group2" Y7 j9 M. P" \/ ^5 _; O
are all valid (note the leading space in " group1"). '*' grants access# A3 n) m# j! [) D5 \& x
to all users and groups, e.g. '*', '* ' and ' *' are all valid.- D" ?$ ^$ M: i8 |& X/ l0 [' z: [
</description>
' g8 t! ?9 p& |$ _</property>7 w/ e+ \% p0 T! A, b5 x% m
<property>
# |8 q, s J: `7 C. G; _8 a3 Y" j <name>dfs.namenode.acls.enabled</name>8 c$ b6 m" I5 _, U$ `6 M, n( D
<value>false</value>- l$ X. Z* |8 O' @2 }! }7 z- a
<description>
6 ]" d8 t% Q$ t' M5 g3 A Set to true to enable support for HDFS ACLs (Access Control Lists). By
3 M( h* j7 e) {7 |! N& \" I default, ACLs are disabled. When ACLs are disabled, the NameNode rejects4 q5 n/ W6 t- n t+ {2 S. R
all RPCs related to setting or getting ACLs.
`6 K/ k |+ `2 W </description>
8 d( t- Z* {: @* _, A2 E* ` n</property>
" i- k q! p( E1 I6 L <property>! j _/ r; t6 b2 s, [. _& X
<name>dfs.namenode.posix.acl.inheritance.enabled</name>
! E8 E7 R! t, P8 O0 M6 _' r <value>true</value>
3 R- @: `6 a, o& m c# ?, m& s <description>
! E3 W- a+ e8 e# C4 Y# T Set to true to enable POSIX style ACL inheritance. When it is enabled
' w, T0 y' m2 Z" \3 O* D and the create request comes from a compatible client, the NameNode8 R/ Z9 p$ K+ M$ P
will apply default ACLs from the parent directory to the create mode
9 j- l! q9 _% z% @ and ignore the client umask. If no default ACL found, it will apply the
; _/ u3 V! j- E1 I' R client umask.
9 k' a3 A! n3 X </description>$ P! d* {9 T4 s4 Y1 @
</property>
5 y" Q0 f& o# _% P; \* P2 H <property>) s! h2 U" Z& M: f' U7 H* \
<name>dfs.namenode.lazypersist.file.scrub.interval.sec</name>
! H+ s3 W; @% v; p6 R: G3 B- I <value>300</value>( D! v" T9 j$ h" L8 f% a- W
<description>
; S9 P/ G. J- Q" D4 ?. N. t9 S The NameNode periodically scans the namespace for LazyPersist files with
. [2 @1 }" T0 R/ ~2 W; v, l missing blocks and unlinks them from the namespace. This configuration key
8 w0 ~6 f/ ^/ B" T controls the interval between successive scans. If this value is set to 0,- R" K2 {& P, {( `) n& J. O
the file scrubber is disabled.0 j* q, m* |, Z4 \$ D. y, H
</description>
" R; X7 i6 c; G2 \9 Y</property>
+ h- E4 x+ o# G H% i8 J8 q3 V<property>
" Q0 J- c! ~0 q' Q3 j2 G. ?7 f <name>dfs.block.access.token.enable</name>
+ f1 u$ Q. B$ d* H& i3 H1 c8 E <value>false</value>3 [" l8 E8 r, H/ \( `
<description>$ D% T2 R9 v7 f0 ^( r/ x$ f% j
If "true", access tokens are used as capabilities for accessing datanodes.& y, P! x/ p5 i
If "false", no access tokens are checked on accessing datanodes.0 \, B' ]. c! Y% X4 v9 J4 M
</description>
; a! _% q# Z k</property>
+ a! J+ b/ e8 a3 e' P8 \) p* m<property> K c, m' C* x# V7 {5 u
<name>dfs.block.access.key.update.interval</name>
) s+ T/ x" H: M$ t% b <value>600</value>3 M5 p! {5 m P* e
<description>
* m: s( ? J0 D! g- @- c' b* i Interval in minutes at which namenode updates its access keys.. Q _2 x' Z9 G
</description>/ O; D& H5 x$ k/ r1 }7 \
</property>4 `$ ?! u7 Y2 q& A
<property>
/ A7 W& r+ l+ p/ O; S' p+ O <name>dfs.block.access.token.lifetime</name>( z4 ^2 r9 V* M6 g" u: r) }* `
<value>600</value>
+ i: t: W% |" f9 w' r; [4 v1 M <description>The lifetime of access tokens in minutes.</description>3 z$ h, _7 \$ G" K: q+ s2 S$ _# @+ f
</property>
1 j+ S3 ^* y# f6 U<property>
- M* D4 D2 Q' b8 K <name>dfs.block.access.token.protobuf.enable</name>. Q. z3 `5 }3 S" e+ _' {0 s8 z5 q
<value>false</value>
; l: b; r0 d6 v5 L! p2 p& Q. ^1 [ <description>
Y& ]: {2 H( Y( v( C$ X h( L7 s( t If "true", block tokens are written using Protocol Buffers.
3 B p& X5 U7 v% V If "false", block tokens are written using Legacy format.
! |' K2 x6 w0 C+ t- X0 g </description>; R! V1 m9 }2 Q$ D
</property>
8 t6 u. N) ~: y' W<property>
* b$ a$ V* E0 d' O9 {# a0 P <name>dfs.datanode.data.dir</name># g2 O4 Q, G2 ?# M% Z4 {
<value>file://${hadoop.tmp.dir}/dfs/data</value>. o% u3 J) j; f; l: \+ _
<description>Determines where on the local filesystem an DFS data node
% O3 Y9 r0 D5 S: b should store its blocks. If this is a comma-delimited- o8 y! f5 d( x# r/ O/ u& j+ Q
list of directories, then data will be stored in all named
1 t* [8 j3 @, {, K6 g$ B' Y: z% m directories, typically on different devices. The directories should be tagged
3 @' z" m* E8 G- R/ |( W; r5 { with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
( q8 e# w; }) m; s( ^9 \/ M( v storage policies. The default storage type will be DISK if the directory does
; c. K' ^' _- c r not have a storage type tagged explicitly. Directories that do not exist will
5 e! U$ Y$ B/ F/ A2 {0 m8 ^ be created if local filesystem permission allows.
6 B1 o' M8 c; P! G( n2 U) @ </description>
" ]- `0 ~+ G" W# H$ [& v4 M0 ~</property>
0 m/ d* d2 h7 A3 H! s( W& B% k8 v<property>1 N$ H& G* \4 l: S+ i
<name>dfs.datanode.data.dir.perm</name>0 k# P$ J7 M1 y9 ~% x
<value>700</value>" N7 [3 b* e/ Z& T; E2 G
<description>Permissions for the directories on on the local filesystem where! [" T4 v; k' X( H6 b
the DFS data node store its blocks. The permissions can either be octal or
8 B0 M4 s: E( c9 I; n& i. |) ^ symbolic.</description>6 | M* l2 b2 g7 V2 n' v/ c, f
</property>3 h3 ]1 ~1 V& X1 A
<property> y* H# d# J: H$ X! D4 U
<name>dfs.replication</name>
# X/ }5 n# |" b. n! R5 B1 y <value>3</value>
% i1 H/ E" \% v3 f <description>Default block replication.
) k/ a8 Z$ u. h) c, K9 y6 G The actual number of replications can be specified when the file is created.
. }" }' Q% `7 V& K The default is used if replication is not specified in create time.0 {, [" W4 G; k; R7 b8 Q
</description>7 J9 c- p% T: ?/ q9 R0 J
</property>, s- m" x# g" |) J
<property>
3 f0 t2 q( s! ` u7 U2 W) z5 X <name>dfs.replication.max</name>0 s4 I3 ^+ ^( K8 ^1 o0 i4 y3 c/ I% z
<value>512</value>* Y/ c' i1 P. `$ k A" _2 m
<description>Maximal block replication. 6 q$ Q/ ` l5 _, x9 j
</description>: F6 O* d" m0 O8 S, g) T0 B- D
</property>
. E$ I; ~# `+ `3 E# a+ x" `<property>
+ w0 q) e4 x# E3 q$ ? d) ` <name>dfs.namenode.replication.min</name>' M- T( D! [$ I7 A8 x
<value>1</value>
5 k! `# G. k# s; U" z <description>Minimal block replication. - N& k j' `: n) a7 p8 x4 X
</description>
& n8 L8 d' n0 m1 E</property>
3 X b* w9 M- s0 }" c<property>
& Q2 z' O$ ~8 f <name>dfs.namenode.maintenance.replication.min</name>
( L f- n1 R4 z2 j4 @. J <value>1</value>8 g' D2 I7 ~4 Z2 t
<description>Minimal live block replication in existence of maintenance mode.' {# h8 A$ E2 l/ }5 K9 |
</description>
4 A0 I+ c/ {4 A- P</property>
. ~' H+ y* E% m+ F6 ~2 I<property>& w! l# `6 ^& S! g( X/ \
<name>dfs.namenode.safemode.replication.min</name>% K1 G; A! r0 i/ G5 y3 \
<value></value>' P* j, }3 Q2 V
<description>
5 i2 {9 o. A; z, p% F* i t+ Q a separate minimum replication factor for calculating safe block count.4 c# E/ J! i* p$ G4 i" M" l
This is an expert level setting.* `4 i. r" r) E5 D$ E1 W2 [0 w
Setting this lower than the dfs.namenode.replication.min
! }* G( \( a$ N* k/ h @ is not recommend and/or dangerous for production setups.9 p& x: g3 E0 ~' T1 T
When it's not set it takes value from dfs.namenode.replication.min
/ l6 b5 \1 D3 n8 J </description>: K$ [% ?6 y( x9 Y
</property>
; J. A7 n; R: ?<property>; d5 H' T8 K: R' K3 W
<name>dfs.blocksize</name>9 V( M8 f h6 Z3 {
<value>134217728</value>; M8 u; i2 n7 \
<description>
, B; A7 U$ C, D) ` The default block size for new files, in bytes. I# j) B3 l2 i
You can use the following suffix (case insensitive):4 H) O' _- V) x- B
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),
4 r2 L- P0 P, j& Q1 M1 h0 g Or provide complete size in bytes (such as 134217728 for 128 MB).
# o9 ?* d S9 y </description>) c0 M K% j! Y" f: a* D8 q
</property>9 l4 z- M! s p- v+ T
<property>
: u3 H) f/ ?/ t- ^' H <name>dfs.client.block.write.retries</name>
" P, c1 H. y/ g) O6 q <value>3</value>* G7 A/ s; z5 u! L
<description>The number of retries for writing blocks to the data nodes, . ?# M: Y! \1 t
before we signal failure to the application.- a. N. ^; K, N# K& x
</description>; e( Z' a2 o( o: ^9 C5 X
</property>" w9 ?: k4 u; {
<property>( ]" F: R& Y3 J0 B* V- Y/ H
<name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
' v! m% `% P e) x* G) }5 T <value>true</value>$ I5 u% n, g6 G. p
<description>& k, E& s( P" _3 K; h- f
If there is a datanode/network failure in the write pipeline,! f/ O9 Q+ a1 Z, _; X+ K6 G4 d
DFSClient will try to remove the failed datanode from the pipeline
8 k- c3 {4 n" ^$ E# A4 g' } and then continue writing with the remaining datanodes. As a result,
. P# J# `) ^3 ~, N the number of datanodes in the pipeline is decreased. The feature is
9 A+ i6 B$ b, E9 ]% r to add new datanodes to the pipeline.
- H3 _# V) M+ t This is a site-wide property to enable/disable the feature.
) P/ M6 B2 [3 ^0 m3 L When the cluster size is extremely small, e.g. 3 nodes or less, cluster3 v+ k0 n$ R* X
administrators may want to set the policy to NEVER in the default, ^+ Q4 x( U/ o! B
configuration file or disable this feature. Otherwise, users may* R& G6 p7 }+ _- _$ U* N
experience an unusually high rate of pipeline failures since it is
; b0 i0 x* |; A" Z! y2 \, y0 t, m impossible to find new datanodes for replacement.* x1 S/ X/ \3 i- k( k/ U! g8 ?
See also dfs.client.block.write.replace-datanode-on-failure.policy
6 E: Y7 x' q/ B( @' K% U </description>( E3 P' y' `" r/ a. @
</property>% j ]( r( O1 x# z, ^" H7 ^
<property>
2 R" H C' c2 i5 s/ R+ {. a4 ~ <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>' b# r9 x; t& t2 {8 A m; m
<value>DEFAULT</value>
8 ?& ]9 X7 C. ^4 r% u9 `) [7 H <description>8 ~' T3 S" F9 @2 B* I* \* b, H
This property is used only if the value of" R; F* o3 o/ V4 [7 s- O, b; w
dfs.client.block.write.replace-datanode-on-failure.enable is true." d+ [& x3 c* |! y* u n
ALWAYS: always add a new datanode when an existing datanode is removed.' a8 f8 T+ q: |9 K7 U
NEVER: never add a new datanode.+ M! t2 `2 a7 Z
DEFAULT: . Q7 `6 i; j' u& v: o% e
Let r be the replication number.9 R3 Q! H; m9 x5 L& f0 q: I0 }+ | _
Let n be the number of existing datanodes.
: a7 v- z& D. m( F# N Add a new datanode only if r is greater than or equal to 3 and either9 d% w" W* F' w, H
(1) floor(r/2) is greater than or equal to n; or9 {; B9 H6 S. u- o
(2) r is greater than n and the block is hflushed/appended., B3 [' q& v9 ]9 ^0 c2 N, G
</description>7 X8 Y, o% [2 p) N. w8 ]
</property>
3 L5 r3 y2 W q5 s! O. ?+ ], W<property>
& N: Z, Q) [) j0 O <name>dfs.client.block.write.replace-datanode-on-failure.best-effort</name>
o# h. v z* o. S# { <value>false</value>
+ J0 b z5 M4 @, |; ` <description>
7 [( @6 `% X, b/ I3 G$ c This property is used only if the value of4 ] J3 J1 F p
dfs.client.block.write.replace-datanode-on-failure.enable is true.
1 w9 I$ T2 u g7 d Best effort means that the client will try to replace a failed datanode
; d2 G9 D. b$ E0 A' K in write pipeline (provided that the policy is satisfied), however, it
/ i" J7 N0 g! u" l; V2 V continues the write operation in case that the datanode replacement also
/ Z1 _& J* b4 { fails.
& Q8 J" L# e; F, ]; J Suppose the datanode replacement fails.- L8 j! p. F2 D) h( ^. W
false: An exception should be thrown so that the write will fail.% t6 z% ~& r$ t" b
true : The write should be resumed with the remaining datandoes.
, z) ^8 c, ?, ^. y9 @( X6 W Note that setting this property to true allows writing to a pipeline
R9 F: M5 S. R, |8 r6 p" \+ ? with a smaller number of datanodes. As a result, it increases the7 ~+ K; E# n- p$ N( k
probability of data loss.8 j5 x1 L# }+ O/ |8 s0 I$ t8 \
</description>2 P2 h: ~6 j5 |) P
</property>+ ~6 K% L. f# N- W
<property>
0 _, j: i& K9 y* s4 { <name>dfs.client.block.write.replace-datanode-on-failure.min-replication</name>/ m# Y6 U: r$ [9 \7 J. I2 d
<value>0</value>
7 G) ] o* G* c <description>
; N+ D) G$ L8 d; @+ B( m% E+ _ The minimum number of replications that are needed to not to fail
; G! b' C# {4 g+ U8 A; |6 q5 J the write pipeline if new datanodes can not be found to replace k1 {4 R9 w3 p5 {& E$ s
failed datanodes (could be due to network failure) in the write pipeline.
# w$ b# b$ o3 a8 ~, z If the number of the remaining datanodes in the write pipeline is greater+ d( ^5 V* u9 r3 |7 m# k6 Q& A
than or equal to this property value, continue writing to the remaining nodes.. x6 K% @0 k% n! T) t6 N! R5 w( W1 C
Otherwise throw exception.& e+ c3 o- F \0 P, W3 J
If this is set to 0, an exception will be thrown, when a replacement
5 n- [+ F3 W5 {+ _, V* p can not be found., i8 H; }, m* c0 m4 K# n, O
See also dfs.client.block.write.replace-datanode-on-failure.policy
$ Q; r9 h! y$ q </description>, p T# y# m6 E( n5 X
</property>* _2 O1 o. {' K. U d/ p, Z
<property>
2 k) F o" {4 f3 }6 y1 c5 Q <name>dfs.blockreport.intervalMsec</name>
% |) E$ x9 {8 u* i- ]& q$ M <value>21600000</value>
9 o, |! q) E: G, \ <description>Determines block reporting interval in milliseconds.</description>$ `4 m5 q+ ]/ l+ b& |
</property>
# f: J4 G. c5 j<property>5 c$ x' w$ ^5 E, m; ?
<name>dfs.blockreport.initialDelay</name>
9 i0 q" u4 i: l# N <value>0s</value>* C3 N9 N6 E1 u5 j; r+ c( m& t! T) E
<description>: d8 P7 b5 h! {. `2 e- ?
Delay for first block report in seconds. Support multiple time unit, c+ y( V5 P/ N4 L' v
suffix(case insensitive), as described in dfs.heartbeat.interval.
$ P/ b0 _6 h+ | </description>
+ E$ H8 k1 R3 t5 F( n6 u j. Q$ j</property>/ x$ y$ c* m8 @! P$ ?. \
<property>: U9 }) y: X) j2 u4 I
<name>dfs.blockreport.split.threshold</name>
9 b1 C2 E# Q* ?+ y <value>1000000</value>- {7 F2 L$ r j
<description>If the number of blocks on the DataNode is below this, s% ?+ N4 ?, ]4 v$ w6 f3 |
threshold then it will send block reports for all Storage Directories
/ U7 ]" \, g4 d' B% J/ V' n' q in a single message.' ^' X2 A/ k& b5 A4 G! _
If the number of blocks exceeds this threshold then the DataNode will
. }$ A. o# J0 G) C/ g* Q$ g send block reports for each Storage Directory in separate messages.5 i R0 }3 ?: @: t! \( ~
Set to zero to always split.7 k% n) [6 K/ V' j
</description>! E: A, t( Y; x% t9 |- ~
</property>
( f8 c- y- u- a- i' M( z9 g) i<property>
3 A9 v4 m! Y# }$ r& [. x0 ^ <name>dfs.namenode.max.full.block.report.leases</name>0 B- v; B2 _1 f0 d: i8 ~# t9 I
<value>6</value>( J+ h. c( G) b' T. s4 O: T, m
<description>The maximum number of leases for full block reports that the
5 u2 X6 ]7 s7 E6 x; a) u NameNode will issue at any given time. This prevents the NameNode from
2 S5 G+ Y- M; H* I G! Q; W4 v! r being flooded with full block reports that use up all the RPC handler/ Q( l& e9 u% l: Q1 N, s
threads. This number should never be more than the number of RPC handler
! B4 e( ?6 f' v4 ]1 y9 e& e threads or less than 1.
3 [5 L& q0 y9 }8 Q) | F </description>
5 M4 q o7 w$ H( `</property>
R- ?! w8 {% n m<property>
' ?! _) Q0 [" C: C& s8 @0 `( { <name>dfs.namenode.full.block.report.lease.length.ms</name>0 J; i( V+ d/ y6 l% `
<value>300000</value>
2 u- f+ M! ?# X5 F; S, k( ^ <description>
) u$ A( x5 J1 d7 _ The number of milliseconds that the NameNode will wait before invalidating* N" g7 f2 R0 d6 R) t
a full block report lease. This prevents a crashed DataNode from
+ J" `8 m6 m) w$ y permanently using up a full block report lease.
7 Y" l) @( J* ]2 V" t; m </description>
- ^) I( ^5 m3 [6 W6 I</property>
! }# _% @6 b5 N% R<property>5 |2 ~8 W* g8 P- W6 h- t
<name>dfs.datanode.directoryscan.interval</name>$ F# o( ~: A3 b7 d/ ^8 `8 m2 W
<value>21600s</value>
& d3 C+ f! ^. u9 Y8 ?) l <description>Interval in seconds for Datanode to scan data directories and6 T* b5 q' A9 ^
reconcile the difference between blocks in memory and on the disk.
; m9 q# |# k& Y' T: V0 y Support multiple time unit suffix(case insensitive), as described
3 o/ B# _5 |" c; [/ U7 [$ ] in dfs.heartbeat.interval.
$ P" h5 B. f' v2 k$ v/ Y, N </description>
0 M! \0 Q2 N6 j( L+ Q6 I% S</property>1 S* F- G( i# ]7 B1 q4 |
<property>
0 P0 I0 `4 F9 b" v <name>dfs.datanode.directoryscan.threads</name>$ |8 k U" Y' K
<value>1</value>
1 D3 [+ j" G: U% q3 o <description>How many threads should the threadpool used to compile reports. w3 _" s# b, Q) _& w6 i" {/ O
for volumes in parallel have.$ B" A) G3 U2 O- T& X h
</description>
0 p5 T2 d1 T$ G4 q& _</property>
+ E I' o' t; R" R& h9 R<property>
. e" Q9 K/ K1 b* `7 T <name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name># [9 ^2 ]4 j5 w. f! o
<value>1000</value>9 C) n; |2 d# W
<description>The report compilation threads are limited to only running for
: S* @0 @+ @( X0 s( ^5 y: C a given number of milliseconds per second, as configured by the) ]9 D* d+ v; M( @
property. The limit is taken per thread, not in aggregate, e.g. setting( R5 O: {& c" k6 U/ T
a limit of 100ms for 4 compiler threads will result in each thread being
5 Z" a3 _. t8 o5 |7 T8 L9 H limited to 100ms, not 25ms.
. D" ]& J% y9 V Note that the throttle does not interrupt the report compiler threads, so the
# J$ f' e" d0 Q' v l3 | actual running time of the threads per second will typically be somewhat
$ Q/ Y# D% d4 m& v higher than the throttle limit, usually by no more than 20%.
$ k' |. U0 i1 { Setting this limit to 1000 disables compiler thread throttling. Only
+ h0 \5 a( o/ V values between 1 and 1000 are valid. Setting an invalid value will result
& _ V/ x6 ]# v/ W) E7 A8 z" M in the throttle being disabled and an error message being logged. 1000 is
. z; O/ W/ c+ }: e4 O6 f I( | the default setting.* ]' V$ X3 x6 H) _$ ]: u
</description>/ @) a4 L# `' [1 t) k \6 S
</property>3 f; t+ J* X6 |! g3 u6 ]
<property> w7 Z0 Y3 F, o4 v) h
<name>dfs.heartbeat.interval</name>1 d1 F, Z# |# v: w. z% k+ G+ Q
<value>3s</value>: r) T$ w' A( a+ {6 Q S
<description>
8 w! D( J: D6 U- R1 a Determines datanode heartbeat interval in seconds.
: s1 F/ d4 D% O- P+ O Can use the following suffix (case insensitive):
- x1 o. _. s- u2 q ms(millis), s(sec), m(min), h(hour), d(day)
% {+ j0 \0 V% J to specify the time (such as 2s, 2m, 1h, etc.).
; h) W! `; U4 m: R3 [ Or provide complete number in seconds (such as 30 for 30 seconds).
x$ ~3 k7 G% Q </description>
+ R; l8 b9 ] u8 L</property>2 u) L4 F5 K* X2 r
<property>) b+ s2 p% }, c; q! G
<name>dfs.datanode.lifeline.interval.seconds</name>
! R1 x3 u. L9 `" p+ b, ] <value></value>
" a+ Z4 E+ R2 Z& W1 A7 P$ B <description>
2 \5 b# C5 y& F- n0 P Sets the interval in seconds between sending DataNode Lifeline Protocol
! R0 f7 e7 F1 x6 f0 J messages from the DataNode to the NameNode. The value must be greater than
( Q0 d# G; ?) d$ B p the value of dfs.heartbeat.interval. If this property is not defined, then
( j5 x% A3 H$ M0 u' F the default behavior is to calculate the interval as 3x the value of
2 M+ ?9 @4 f) T3 w3 G5 t dfs.heartbeat.interval. Note that normal heartbeat processing may cause the) q. y* s4 s+ t2 r9 y- f9 e
DataNode to postpone sending lifeline messages if they are not required.
& S0 a( y k G( V% B) x+ } Under normal operations with speedy heartbeat processing, it is possible' Q& V' o; d+ K; c1 i/ ? q
that no lifeline messages will need to be sent at all. This property has no2 m9 l0 s& X) |& j
effect if dfs.namenode.lifeline.rpc-address is not defined.# d, v# o% k1 a" ~
</description>
3 T4 ~$ n0 X' ]+ S9 B! i" |</property>
+ z- B5 e6 b3 I2 j6 g8 j5 Z<property>0 z. h1 ]2 m. d: `3 n) J: I" c7 T2 v9 |
<name>dfs.namenode.handler.count</name> H1 L& G' I/ K1 K# @* \
<value>10</value>4 ^! b) Q* b' R9 f/ a# t
<description>The number of Namenode RPC server threads that listen to
! |) B5 v3 h N% h3 z requests from clients.- g5 A- H" z l* E) H* D; U
If dfs.namenode.servicerpc-address is not configured then& E) Q4 X! ^- R& |+ M
Namenode RPC server threads listen to requests from all nodes.1 v& I2 f; h6 o$ H9 |: K9 I4 q* a' m
</description>
2 R7 g8 k; u: ?2 ~0 g& A0 j# _</property>3 [. Y+ u% }- m; [1 P
<property>
5 Y8 s' T8 t" K( n <name>dfs.namenode.service.handler.count</name>
! ?( O) V4 K# b: G <value>10</value>
/ I5 h+ J. M+ c( I <description>The number of Namenode RPC server threads that listen to% [1 m6 Y6 O" ?
requests from DataNodes and from all other non-client nodes.: P1 W0 S, {/ ~: {
dfs.namenode.service.handler.count will be valid only if
' l t# Q9 R1 x9 P dfs.namenode.servicerpc-address is configured.. x9 c: U0 v6 r
</description>
& M' q8 k4 ]* d1 u, z* J9 j</property>, P0 I( u7 ^- l* L0 @; A3 C9 x
<property>
9 g# s/ U9 W; k* v \ <name>dfs.namenode.lifeline.handler.ratio</name>
) f' l- D2 w8 \5 H( r+ J <value>0.10</value>3 i# s P& m% h7 w% m5 b% J
<description>; C2 M2 O/ X1 y5 w5 c
A ratio applied to the value of dfs.namenode.handler.count, which then
7 s! i w5 w2 Q3 |9 D- X provides the number of RPC server threads the NameNode runs for handling the
! q' C3 l& k: A* A' A# N lifeline RPC server. For example, if dfs.namenode.handler.count is 100, and+ X3 L7 R0 p& f) M6 p& V- W1 ]
dfs.namenode.lifeline.handler.factor is 0.10, then the NameNode starts5 W5 w4 e9 d; R1 ]6 u3 s2 O
100 * 0.10 = 10 threads for handling the lifeline RPC server. It is common
! a; I* ^ A* t9 B0 g to tune the value of dfs.namenode.handler.count as a function of the number0 K: m& {! S: |
of DataNodes in a cluster. Using this property allows for the lifeline RPC6 K2 s' N2 X5 ^6 U1 K( Q
server handler threads to be tuned automatically without needing to touch a7 a6 |4 j# ~; O" [) H- \8 d4 R
separate property. Lifeline message processing is lightweight, so it is: H& X7 S2 m! @8 `( A* X/ A
expected to require many fewer threads than the main NameNode RPC server.
3 D, Z3 f0 y: m+ v! }: t This property is not used if dfs.namenode.lifeline.handler.count is defined,4 k, x" D, [1 b5 f
which sets an absolute thread count. This property has no effect if
1 V% Z* z( x7 s dfs.namenode.lifeline.rpc-address is not defined.) D2 |; g t- e$ Y! f
</description>6 A) `. g& { k& Z: |; ^6 _
</property>3 @. V' u& ]# V
<property>
- M8 p0 W2 O; {5 z' G6 p+ t <name>dfs.namenode.lifeline.handler.count</name>( G6 R7 x% b- T) O. [4 \
<value></value>1 c* r) _" \ ~5 [
<description>- R1 L2 t( c2 S3 [" K ^
Sets an absolute number of RPC server threads the NameNode runs for handling0 e2 ?2 u% M* d1 B; t$ r N
the DataNode Lifeline Protocol and HA health check requests from ZKFC. If
3 l5 g" S3 I7 p) k# {" V) u this property is defined, then it overrides the behavior of0 o; @* c3 f M6 P3 M# `
dfs.namenode.lifeline.handler.ratio. By default, it is not defined. This- A2 q$ _+ N# F0 a6 e
property has no effect if dfs.namenode.lifeline.rpc-address is not defined.5 J& N1 D# C- h. z. c/ U
</description>
* A1 t- V. u3 V3 u5 t0 l/ f! F</property>
) ]' F* w" A# Q* \6 z9 G<property>
, C6 [. R) Y* v4 j <name>dfs.namenode.safemode.threshold-pct</name>
6 f5 S0 a% L( V <value>0.999f</value>
" B) S( ?" j, b" d8 A: X <description>: ]$ M/ F4 f1 T; o' R. M
Specifies the percentage of blocks that should satisfy ! [3 t4 ~! \2 l
the minimal replication requirement defined by dfs.namenode.replication.min.
3 x6 K$ `) l7 o2 j Values less than or equal to 0 mean not to wait for any particular
6 {# H& x8 S* w, b9 q' _: ]! O percentage of blocks before exiting safemode.
; u8 A8 _* w r b" ^ Values greater than 1 will make safe mode permanent.' f: g8 [' I9 ?6 P4 O( j
</description>
" N k8 X. L W' d. O7 P9 c% g</property>
" [6 ~7 j- y( f' Z<property>
6 _. c8 ~3 g5 q/ k. k <name>dfs.namenode.safemode.min.datanodes</name>0 Z' X A2 I! K4 F% X: N6 E
<value>0</value>7 s( r$ _* t- i9 B4 C1 S) P
<description>
2 F; R9 F) q6 O+ |, M Specifies the number of datanodes that must be considered alive
5 l) ?- @8 p+ O" a1 ]' n before the name node exits safemode. N0 E% K; e! L0 M
Values less than or equal to 0 mean not to take the number of live/ j; @! k7 G" s: j s0 b9 S' O1 J8 H
datanodes into account when deciding whether to remain in safe mode5 c4 d% f' }4 P4 ~( v/ F7 _7 y
during startup.
1 E5 S" h. c, W) j) h Values greater than the number of datanodes in the cluster( y! J! R$ u: }! Q; l
will make safe mode permanent.$ `! G1 s: R, @5 t! X3 x+ ~
</description>5 J4 X2 i' m7 g2 @; W
</property>
+ P( r0 d" R" Z, U) ]" z8 L<property>
! ]" i; |4 `' A! h: Y+ d; ? <name>dfs.namenode.safemode.extension</name>8 `( C' L2 u" S; C
<value>30000</value>9 a. W: e# h0 {+ Y$ `6 Z [5 k
<description>
+ m, A1 ^* l/ I Determines extension of safe mode in milliseconds after the threshold level2 b- V& K: {7 n2 K0 ~' |
is reached. Support multiple time unit suffix (case insensitive), as
+ f A; t6 Y7 B# l! h! X described in dfs.heartbeat.interval.
; A7 Y- R1 I+ k) G- T' O </description> y& v' g. `- G
</property>* u& G8 F" L. ~+ n
<property>) o5 C3 s& y6 D
<name>dfs.namenode.resource.check.interval</name>
6 L' w1 w" ~2 a! ~ K <value>5000</value>4 c. V$ [- [! I `
<description>
% N9 @# B5 a/ _4 M5 |1 a# p The interval in milliseconds at which the NameNode resource checker runs./ U6 p; }8 c. p! b. l
The checker calculates the number of the NameNode storage volumes whose0 e7 O5 M9 ]& o. V
available spaces are more than dfs.namenode.resource.du.reserved, and
+ H5 _* N9 R2 D4 V: B/ B enters safemode if the number becomes lower than the minimum value a! s7 K+ D, y1 k5 P6 b' N
specified by dfs.namenode.resource.checked.volumes.minimum.# I% f6 a) _. _# z3 |" ^) L
</description>
- n& L1 ~% n5 W7 ~* b</property> K' R- W o8 |% [; d
<property>) W+ ^5 M& S! f5 S3 I& R7 `8 K; G0 m
<name>dfs.namenode.resource.du.reserved</name>
: a% N( B) W0 r: Q6 ?. C; s$ O <value>104857600</value>
, O8 P9 t( V) y/ k/ O5 z6 G <description>
8 P4 N0 f% e3 U6 I' I2 f The amount of space to reserve/require for a NameNode storage directory
" @9 ] d' c, K in bytes. The default is 100MB.. y/ s! j* A# [4 `! V
</description>
, i$ _( q2 L- J</property>
: T6 K: p4 S* w8 O) B<property>9 b# M( P+ C$ _
<name>dfs.namenode.resource.checked.volumes</name>
% u5 q) G5 l5 h! [9 D; y <value></value>
9 f7 r# ?( N3 h, A% ~ <description>
% w9 M9 F& S; R7 o9 g1 ^9 _# a& n2 D A list of local directories for the NameNode resource checker to check in
# C& ]" W) x% z* N addition to the local edits directories.5 f! _7 X B6 ~+ e- Z5 c
</description>
: c0 a4 s5 n4 q8 `+ T! {- M7 ^</property>$ L, u/ P/ m/ c2 d. q R( Y
<property>5 B, K" C) [& Y8 ?
<name>dfs.namenode.resource.checked.volumes.minimum</name>
3 j* b. O7 ^ I <value>1</value>
8 t. @" U3 q0 n; S$ P <description>
_4 v* w( C* D% p" I: Q The minimum number of redundant NameNode storage volumes required.
- N7 Q9 r; }6 T7 [ </description>% r$ i6 S. @4 `
</property>
& g/ P; S# y/ }7 Y+ e* @6 Z4 Y7 M<property>
$ }: }5 |0 D/ @* }8 k <name>dfs.datanode.balance.bandwidthPerSec</name>
% M. F; e2 P# t. G0 d. l6 K <value>10m</value>& r6 r8 N3 w! a% K
<description>
1 H9 w; M8 ]& k- U# p; z) f Specifies the maximum amount of bandwidth that each datanode
7 X' P" X Z% S6 T can utilize for the balancing purpose in term of
- M, R7 ^ j' ~4 x3 A2 ~/ h, n. L; d the number of bytes per second. You can use the following
: A% C) m8 {5 m9 `9 {6 U" t suffix (case insensitive):
% b9 E0 F! _0 j* I1 ]4 q k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)to specify the size& `3 I/ T4 h3 {' t8 d# V. @' E
(such as 128k, 512m, 1g, etc.).8 V8 T* i5 ^3 m" _6 _ X' p
Or provide complete size in bytes (such as 134217728 for 128 MB).' H1 C& t1 m5 Z' d
</description>
+ }) F2 t; T$ R+ x6 s8 p8 x</property>- j: f' k& d& j- J- l( r; H
<property>8 ^9 k4 V# M- K
<name>dfs.hosts</name>
6 f0 ]/ c. R0 C+ ^" Y <value></value>
% a6 I3 a+ x* Y+ m% ` <description>Names a file that contains a list of hosts that are
9 Y% Q. X# ~$ i) R6 n' t+ i9 i permitted to connect to the namenode. The full pathname of the file* \+ m& I8 w; @1 p- n
must be specified. If the value is empty, all hosts are5 K5 W% N5 c4 p, D
permitted.</description>7 s/ ?: [. f ?3 B8 x1 w; `
</property>
4 Y3 |0 L- ~* ^<property># e- x9 g& G" v0 M
<name>dfs.hosts.exclude</name>; I) }) D4 r. r# [0 o6 z9 b
<value></value>
H& Y [) |- k9 @% }, ]2 M <description>Names a file that contains a list of hosts that are
8 s& ]6 w) V+ J% o* B" S% v not permitted to connect to the namenode. The full pathname of the
: ^9 ?3 Y3 i0 q# n" p, Y1 I% n4 U2 _6 h file must be specified. If the value is empty, no hosts are1 _; {9 M9 ^+ m7 W. g: V
excluded.</description>4 ]; o4 N- W y2 Z7 X9 S
</property> 2 `1 I3 W7 R* |0 m$ @
<property>5 H: i- r' c: \* ^' p, g
<name>dfs.namenode.max.objects</name>8 @& J& F0 n4 J, ^" d- Z
<value>0</value>
# J. _4 Y/ T: z/ i# M <description>The maximum number of files, directories and blocks
7 }$ C6 ^, P* x P7 s7 i6 u3 w* ` dfs supports. A value of zero indicates no limit to the number+ P: T# S6 n) D. |5 X0 n
of objects that dfs supports.. A# Q) y( e, u. `4 i0 E! J
</description>
4 T3 }# D/ S3 d) A% N8 t) E! X</property>
. Y1 K6 {! ^( [: d5 U<property>
0 z% p( E$ v3 y2 ~ <name>dfs.namenode.datanode.registration.ip-hostname-check</name>+ A( `1 @7 P: D5 u" x p; I2 u
<value>true</value>
. |; ]+ x3 U! i5 S6 ~- v' m z, c <description>
2 U4 ]. e! X u" Y If true (the default), then the namenode requires that a connecting$ S9 M! [/ o, T: m6 y
datanode's address must be resolved to a hostname. If necessary, a reverse
/ ^' H; z5 `. J* w1 w DNS lookup is performed. All attempts to register a datanode from an
" {( g) H5 f. h2 V _) I% w' S) B unresolvable address are rejected.
3 w4 t# g$ a' |( u' q( V, z' z( n7 z5 r It is recommended that this setting be left on to prevent accidental W7 @/ }, i# ^- G, e# S7 O
registration of datanodes listed by hostname in the excludes file during a6 l$ O/ O& W1 F0 j* d- ] a
DNS outage. Only set this to false in environments where there is no
7 s1 Y, Z: @5 a& |% K3 ?9 y8 F infrastructure to support reverse DNS lookup.! W2 H! Q$ {1 Q* Y1 `/ p( R: p: o
</description>
3 b5 [. y8 A8 ^! j</property># P4 ~+ e* m6 t. P% E
<property>
: ^$ J4 W3 s/ {* K0 K- {5 A <name>dfs.namenode.decommission.interval</name>
$ f7 {( u; R- i; f9 J! V) s <value>30s</value>1 q! w$ q) P" d0 b+ C
<description>Namenode periodicity in seconds to check if
1 Q9 t9 X: M6 K1 b" y3 @4 X# b decommission or maintenance is complete. Support multiple time unit
, v$ e- e& T9 c. X; n: s suffix(case insensitive), as described in dfs.heartbeat.interval.0 j9 X& V6 _/ z
</description>
& Q4 `9 g. I5 w# u! ^+ c</property>( _& t0 _( P2 o; @( E# x
<property>+ G7 {6 r$ D! J7 Z
<name>dfs.namenode.decommission.blocks.per.interval</name>
' Z: g1 C# R. L <value>500000</value> ]8 S1 x1 q: u+ ^6 y% k( j, u7 _
<description>The approximate number of blocks to process per decommission
H4 I, O v9 N3 U7 ?8 N5 ^* u% ~ R or maintenance interval, as defined in dfs.namenode.decommission.interval.2 c7 A. ^5 e3 Z5 G% g- ?! I, J
</description>
: J( A1 k7 j2 ^3 m9 F1 g</property>: H7 `" E8 F9 ?" H Z
<property>' o! U2 W# c, \' F
<name>dfs.namenode.decommission.max.concurrent.tracked.nodes</name>
: K B8 o5 O$ [8 s' q5 H' W9 @ <value>100</value>
0 r1 A8 ?) n) X <description>
6 C9 {7 z, F( R6 K2 o The maximum number of decommission-in-progress or
# A5 T; \. K$ I4 N+ B9 Z) G entering-maintenance datanodes nodes that will be tracked at one time by
2 \8 w+ j, }/ C4 r @ the namenode. Tracking these datanode consumes additional NN memory( C/ C- |0 U( y1 J/ q
proportional to the number of blocks on the datnode. Having a conservative
" C5 }& F1 f, Q! I" R5 ^$ }/ A limit reduces the potential impact of decommissioning or maintenance of
* r1 D; T# D& ^1 y8 B' H1 | a large number of nodes at once.
' c* h: e- F! J2 a' G' J( O" } A value of 0 means no limit will be enforced.: q, h3 _- m( a( ~* M1 x
</description>
) g; T8 K% ?/ g</property>- ] F Z A+ z! G, I
<property>
9 {, q" Q2 {. K& F! W0 A- k <name>dfs.namenode.redundancy.interval.seconds</name># C* _) G2 E' Z: J% d
<value>3s</value>
. Z8 D9 b4 |0 ~) [7 i <description>The periodicity in seconds with which the namenode computes - G. h. P& Q3 @: A& P
low redundancy work for datanodes. Support multiple time unit suffix(case insensitive),+ W' Z$ T$ K3 Z4 O
as described in dfs.heartbeat.interval.0 t6 ?0 K9 R5 `
</description>+ [, g b; F, i2 Y
</property>
& @! f5 i8 c3 G7 |& F E<property>
0 r& z4 F* v, J; D' U6 w( ` <name>dfs.namenode.accesstime.precision</name>7 J) W4 c C {2 J3 ?( i
<value>3600000</value>
6 H2 M- S! M Q8 x0 x <description>The access time for HDFS file is precise upto this value.
; E( m# Z) q# Z The default value is 1 hour. Setting a value of 0 disables0 s4 X0 E: |8 Y9 z3 m5 @
access times for HDFS.- o4 I$ v8 i( T* k
</description>) X. \ Q1 A# C
</property>9 `4 `+ o! A9 M+ a
<property>4 v3 u0 z: f9 a
<name>dfs.datanode.plugins</name>1 u0 ~" j: Y! T6 F( _3 k" t
<value></value>% i. ^) N2 F. R8 L: g1 p
<description>Comma-separated list of datanode plug-ins to be activated.
3 G# y, G$ g7 @' t! G6 s! w r% l </description>' f. \3 C8 e h1 m2 R) _, r
</property>' O; m4 j: g1 r" J4 U; {
<property>2 L! P0 f5 d3 e& l$ ?
<name>dfs.namenode.plugins</name>: Y6 Z4 U( e' n% X( D: e C
<value></value>
w/ {" X2 E- `5 m% Q <description>Comma-separated list of namenode plug-ins to be activated.
& m% l F2 r' j( u( K </description> e$ O7 \# h/ z
</property>
9 q' `6 } j' j<property>
" R& p* g/ U3 W: |2 G1 T <name>dfs.namenode.block-placement-policy.default.prefer-local-node</name>
6 h5 J M, Y7 q' D: Y" G8 a5 V <value>true</value>
3 Y5 S3 b8 G9 U <description>Controls how the default block placement policy places
6 m+ n! I% y; C1 T+ m5 ^$ [ the first replica of a block. When true, it will prefer the node where
6 C5 j' }& H8 Z the client is running. When false, it will prefer a node in the same rack, L% \# J3 C2 j1 K8 }, K: b
as the client. Setting to false avoids situations where entire copies of
* X+ L# ^5 p/ G4 v4 Y large files end up on a single node, thus creating hotspots.
/ N- m8 z/ ?+ \) v </description>
& j' r& h4 G% D& K0 v</property>* o- f$ {3 o, F! ^: n" z8 V
<property>% u7 ], `0 E6 T% F& J
<name>dfs.stream-buffer-size</name>
A, L, `: b m/ L0 j+ r: R) b% S$ u <value>4096</value>
+ c& `6 L, C* @ I2 C <description>The size of buffer to stream files.+ y1 q" G8 [, m* X4 I
The size of this buffer should probably be a multiple of hardware
) Z5 z# w2 @5 a page size (4096 on Intel x86), and it determines how much data is1 t% g7 d; Z( u4 e" a
buffered during read and write operations.</description>2 |/ ?! C- L2 h* K% m' U6 T
</property>
3 P6 N9 w1 B5 c, h6 b<property>
! M) A- f9 d1 w0 p6 o0 {8 V; K <name>dfs.bytes-per-checksum</name>
- ^! O( t9 c Z; s! I) V; V0 Y7 t( n <value>512</value>) p, R1 o( O- C
<description>The number of bytes per checksum. Must not be larger than
' A: d. z9 N- s9 Q dfs.stream-buffer-size</description>
- L9 D3 k* e2 Q5 d) ]8 }</property>$ b: ^& o) `, `" L4 C+ _
<property>
1 K/ p; o( z+ _+ v& j <name>dfs.client-write-packet-size</name>- g# u( X: v: W0 }
<value>65536</value># X/ d) j9 C' m
<description>Packet size for clients to write</description>0 o( w/ E: K4 l, k
</property>
% A% m6 c& z: g- \' A; k# M3 x. t<property>6 I5 S4 X5 K/ |7 _% w& [
<name>dfs.client.write.exclude.nodes.cache.expiry.interval.millis</name>
4 A& G$ l) l; @8 h" m c <value>600000</value>) m6 ?1 \ v" |# t( `3 e
<description>The maximum period to keep a DN in the excluded nodes list
9 a! d7 _+ A3 y6 a* M0 [+ A: R6 O0 [ at a client. After this period, in milliseconds, the previously excluded node(s) will
) t& }5 b/ N7 m" t be removed automatically from the cache and will be considered good for block allocations
' e5 J0 _9 L, J6 S again. Useful to lower or raise in situations where you keep a file open for very long
+ R' \7 w, F2 o3 P5 v periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance
: |) ~, t8 y& T3 l8 V% m G3 @ restarts. Defaults to 10 minutes.</description>
: A3 y( P2 e1 T! F; ?( C8 ?# f</property>
* O+ f( e; w, x<property>
/ F4 k+ M1 D4 P9 ? <name>dfs.namenode.checkpoint.dir</name> Z c+ i% h! W" }# G. w
<value>file://${hadoop.tmp.dir}/dfs/namesecondary</value># m2 s% F" R$ T6 j
<description>Determines where on the local filesystem the DFS secondary
# A% e+ `) W& J- ]1 J9 Q- F4 Q name node should store the temporary images to merge.. h- @# f1 M4 S3 X" J
If this is a comma-delimited list of directories then the image is; U% I" {: K1 I4 T: B8 r2 c; c, _
replicated in all of the directories for redundancy.
( X/ |1 [ j% C </description>
* Y8 S. X/ _7 f/ p7 G* k</property>
. Y" s- [) |, e8 ?: O<property>
5 n" o; D+ x; ` ?( Z7 v <name>dfs.namenode.checkpoint.edits.dir</name>
$ }9 Z" L9 W0 K2 W1 U4 [ <value>${dfs.namenode.checkpoint.dir}</value># f; F! ?0 l5 v2 V. M
<description>Determines where on the local filesystem the DFS secondary
4 T9 t' X4 W. M A( T name node should store the temporary edits to merge., {" n. Z) T/ y5 Y% J
If this is a comma-delimited list of directories then the edits is
5 j! W' o( `. F' A' I replicated in all of the directories for redundancy.
, A" g% l% @& @; J/ W7 h3 b Default value is same as dfs.namenode.checkpoint.dir
p( ~, M$ J* L6 e </description>
9 r7 \) m8 w( ~. h</property>
7 P2 P# |4 s& \<property>
3 a5 H! b" `4 \ <name>dfs.namenode.checkpoint.period</name>! Y2 H8 G& `* c
<value>3600s</value>; X; c, D: ?4 y$ O( Z ]/ a
<description>
7 Z, F3 Y& _, N4 d4 P' \& t The number of seconds between two periodic checkpoints.% V$ v+ \( n) J+ @% R' S* J
Support multiple time unit suffix(case insensitive), as described
: Q" U/ m. m$ A/ J0 B in dfs.heartbeat.interval.! t) P3 Y$ L2 s2 j% x
</description>
0 d/ Q9 J3 O5 w5 s' s</property>
6 S; D4 C* }) I9 w. x/ E: N<property>
/ d& T$ `; _* w. c9 j) {- S2 i: R <name>dfs.namenode.checkpoint.txns</name>. G. ~ N! B5 i" p6 f' C. m9 k* F8 h
<value>1000000</value>
+ g# X1 x4 U+ H6 @" m. M, O( ^ <description>The Secondary NameNode or CheckpointNode will create a checkpoint- Q$ M) o) \, m, I
of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless4 N# Z0 Q( }# @7 f( L, Y0 w1 h
of whether 'dfs.namenode.checkpoint.period' has expired.& I# S# m+ \! U3 s
</description>/ g. ^6 q7 @! Q" B+ c
</property>3 w' a' U+ Y" s
<property>, b6 V. D8 k! Z4 ?
<name>dfs.namenode.checkpoint.check.period</name>
# w3 f6 e P+ H$ Z! _ <value>60s</value>' W% h' E1 h0 r! M3 }) |7 h
<description>The SecondaryNameNode and CheckpointNode will poll the NameNode
0 n& _' Q- B T0 x# _* E every 'dfs.namenode.checkpoint.check.period' seconds to query the number
: S) Y4 q: m1 P of uncheckpointed transactions. Support multiple time unit suffix(case insensitive),9 c$ d2 \$ Q# R) K+ f
as described in dfs.heartbeat.interval.- ]; g! p9 X- E% E! m& e1 z
</description>6 ]9 J/ P5 x* F$ {& T
</property>& k6 P; w/ d4 _% l% k3 ?3 D
<property>
2 z% W5 f7 j, Z6 `& ]+ d <name>dfs.namenode.checkpoint.max-retries</name>
' p" N0 }% P- P" @) Q1 X; E( I: e <value>3</value>
* Z/ d/ v1 A L6 E7 F, [% d8 W3 D <description>The SecondaryNameNode retries failed checkpointing. If the * ~; @/ I8 }. c( M k
failure occurs while loading fsimage or replaying edits, the number of
6 j1 o* n4 e! C/ o) S retries is limited by this variable. + d0 a! y% I% ~* l+ h
</description>
) F5 ^9 d; O3 V2 C: u3 \</property>) e9 Q, P$ O# c0 w
<property>
[! f `& h8 S) E <name>dfs.namenode.checkpoint.check.quiet-multiplier</name>" }3 N5 b& h0 B7 _7 l
<value>1.5</value>4 P- X6 }1 r/ ^( F+ Q2 i$ I, j
<description>
6 m8 q9 x$ M2 _) U6 g1 I0 g/ c Used to calculate the amount of time between retries when in the 'quiet' period
9 ?) k) h: T3 x7 o) b for creating checkpoints (active namenode already has an up-to-date image from another
* c; q G& C+ @9 ]5 |% x- q+ e checkpointer), so we wait a multiplier of the dfs.namenode.checkpoint.check.period before- ]5 Q5 I p0 z8 l6 Y7 R
retrying the checkpoint because another node likely is already managing the checkpoints,' _5 i8 m* D( e4 f* C; s
allowing us to save bandwidth to transfer checkpoints that don't need to be used.
% V9 f$ r; ]% g( H- y4 S* h6 l' x! n e </description>( g5 E5 z- Y7 F) D+ e6 f) b
</property>1 ]& P5 \5 f+ a8 r5 g* L5 }
<property>
) t# D/ n: V( h+ B <name>dfs.namenode.num.checkpoints.retained</name>
7 n$ g1 j. l) i6 \ <value>2</value>
- L. s4 d9 }4 P. Z, n: H/ S <description>The number of image checkpoint files (fsimage_*) that will be retained by7 K/ |9 s4 v. }' k
the NameNode and Secondary NameNode in their storage directories. All edit
2 l4 ?" e) a7 { logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained. m; R+ E$ C( Q& C: T, c a$ D
checkpoint will also be retained.8 [: c& W- U( P8 w( b. S
</description>
8 F) i$ h8 [& }) [4 b( x0 x</property>
3 H/ `5 z; d- I5 z<property>
5 L2 f1 K r0 n0 i$ A <name>dfs.namenode.num.extra.edits.retained</name>
; `: r4 N! Z `8 N+ N. n5 D <value>1000000</value>
( D. ^, |& e; h3 L* ^ <description>The number of extra transactions which should be retained
l" }( I; G, e( g" d beyond what is minimally necessary for a NN restart.- m; ^: d( B! A+ L- \4 q* H9 {" D. w% w l
It does not translate directly to file's age, or the number of files kept,
6 m; j$ S/ `$ e8 t but to the number of transactions (here "edits" means transactions).
; G/ O5 y; \0 x2 f- |4 Y! \ One edit file may contain several transactions (edits).
. y) ~9 D( C+ L6 C. `* |( C During checkpoint, NameNode will identify the total number of edits to retain as extra by4 w5 |3 E" ^2 N% S% ^) t
checking the latest checkpoint transaction value, subtracted by the value of this property.
1 V% i; n; Z6 c: Z9 f; k* X; W Then, it scans edits files to identify the older ones that don't include the computed range of D* x6 H9 F/ J- |1 t" W
retained transactions that are to be kept around, and purges them subsequently.5 X% }: R6 R: s5 q5 Z
The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have: G- m0 ~# s0 D# P7 |
been offline for some time and need to have a longer backlog of retained5 N; \+ H& ^( h7 n9 ^
edits in order to start again.
( K* Z- `7 N8 |7 g/ w2 d4 B Typically each edit is on the order of a few hundred bytes, so the default; j) x" t- B. J: s' n5 t6 a
of 1 million edits should be on the order of hundreds of MBs or low GBs.2 K+ k/ B& _* t+ E6 z
NOTE: Fewer extra edits may be retained than value specified for this setting
: D- u, d" j$ ^/ b) H4 j- ? if doing so would mean that more segments would be retained than the number2 C2 Q8 W$ P, k' C; v
configured by dfs.namenode.max.extra.edits.segments.retained.0 U+ W5 S' l. p+ n* R' Y) g+ Z0 Z
</description>8 Y4 l* K& \0 u7 w' ]9 a" _) k( E
</property>8 Z1 Q5 G; [( J! F4 I- Z
<property>
6 |( p: Q- \2 o! X0 Q <name>dfs.namenode.max.extra.edits.segments.retained</name>8 h, m3 z7 ?8 Z1 v& Q6 X7 c: u1 V
<value>10000</value>' P5 t/ i$ N: X
<description>The maximum number of extra edit log segments which should be retained% j6 ]' ?5 R1 K+ d" @; v
beyond what is minimally necessary for a NN restart. When used in conjunction with, m. ^% h( i- ]$ A6 ]( S
dfs.namenode.num.extra.edits.retained, this configuration property serves to cap) }& {- w' ~2 Y( F
the number of extra edits files to a reasonable value.) r5 p. G# Y5 w8 o
</description>- I% i2 T, J- i% [7 [/ `) q8 k H% ^
</property>0 s5 c9 g# r) W" E
<property>
- R# L6 |0 g5 |/ h# ?7 a5 F <name>dfs.namenode.delegation.key.update-interval</name>
2 a( Q X0 |# y; _$ ? <value>86400000</value>: y. @4 G- q. }1 A; e0 m* F
<description>The update interval for master key for delegation tokens - x) B/ _. a6 `9 k" F, T$ r1 Z
in the namenode in milliseconds.0 u1 k+ P8 U2 v$ Q/ L
</description>
7 ] S$ ?, g3 D. h1 ^$ [* ?8 U</property>
1 i& b o+ z# R+ ?2 C4 M9 \<property>
7 X! [7 a* f5 k+ ] D <name>dfs.namenode.delegation.token.max-lifetime</name>9 z& `2 Q; `5 ^
<value>604800000</value>" z( g1 s+ n' P9 l! k5 Y' I
<description>The maximum lifetime in milliseconds for which a delegation # e( a8 L1 C/ t* z& l4 ?, X
token is valid.3 \5 f. ~8 X8 ?* I. z+ j1 n4 s
</description>6 e. H3 C3 C4 Z7 i
</property>
; \% I$ a! F3 o b<property>
( q9 D% Q' ]1 |$ m <name>dfs.namenode.delegation.token.renew-interval</name>
1 m3 M* f" B: e7 S <value>86400000</value>
9 {# y; O6 c* {; o3 @+ x( }& Q! s# ]7 g <description>The renewal interval for delegation token in milliseconds.
7 Q/ R# n# e! [( q </description>: k# p5 i. g6 x
</property>
1 ]! h- W/ y% ]. U<property>* X$ G5 Z' J+ Q, N, s! b; H
<name>dfs.datanode.failed.volumes.tolerated</name>) p1 s2 z4 n! j$ i* s9 ^
<value>0</value>
- V, K' A& Y5 P( i8 K4 l <description>The number of volumes that are allowed to
# R" r" ` {0 J3 ] ? fail before a datanode stops offering service. By default
% D& ^/ |+ J2 ?8 M( h$ x! e$ b any volume failure will cause a datanode to shutdown. {* u0 k8 l, t# i* \
</description>0 N7 J; G1 K2 |6 ?
</property>
2 P0 ^" d4 k2 S {; G# X$ ?. q8 q& p) E<property>
; i7 m I# C2 R4 d3 t8 x <name>dfs.image.compress</name>5 c# Q4 w/ q% |
<value>false</value>! g! K$ K' h( {! N5 P3 W8 ?7 X, `
<description>Should the dfs image be compressed?3 c7 Z. l3 r/ c1 E
</description>
. P: j) T4 A+ {9 u8 i8 t" Z Z( q</property>
* X! S) E, \% M% C) E S<property>% A5 B8 K( d k1 I8 g8 y
<name>dfs.image.compression.codec</name>
' }8 X% M# G* n. x6 y0 { <value>org.apache.hadoop.io.compress.DefaultCodec</value>$ M' ~$ j1 N* s
<description>If the dfs image is compressed, how should they be compressed?
5 S) m; S; G1 N This has to be a codec defined in io.compression.codecs.3 u& c- \, H0 Q$ g) U
</description>
/ `4 t; |( i0 f" X, {</property>
0 ~1 N4 q( E; ?5 x/ H9 S<property>2 z9 E' G g& W' T* R. k
<name>dfs.image.transfer.timeout</name>! |3 ?/ E; V% o0 L! ^
<value>60000</value>: V# L- A3 f- J% F. Z' V* `1 g& I8 s
<description>
( @. R! u) O& o8 t Socket timeout for the HttpURLConnection instance used in the image
+ @8 o. P8 A0 | D transfer. This is measured in milliseconds.2 K: j5 Z. r6 U; x3 m
This timeout prevents client hangs if the connection is idle: F7 a- v5 Q5 ]
for this configured timeout, during image transfer.6 _6 z3 H# F* x4 j8 @+ T
</description>
' b( c/ A7 [. b) d, R/ K</property>( k; T; a. `) f1 }) U
<property>0 |* C5 E, e/ F0 p$ o, A* g
<name>dfs.image.transfer.bandwidthPerSec</name>
6 j: y; V3 Y. X# o1 h* I9 @9 u$ L1 d <value>0</value>- h. f; ?5 c9 m& L
<description>
$ p9 Y/ D' Y7 ]; q) q$ F6 D: G Maximum bandwidth used for regular image transfers (instead of
- }5 D, v+ m# q+ |: p+ F" L bootstrapping the standby namenode), in bytes per second.2 b* T1 o. t/ B$ V$ }) a5 J
This can help keep normal namenode operations responsive during
' E, z9 B8 l9 p# Q& { checkpointing.
% c% g# K6 J) j) ] A default value of 0 indicates that throttling is disabled.
: l# Z) @7 ?/ w/ J- k6 p5 Y The maximum bandwidth used for bootstrapping standby namenode is# E, P3 X2 v9 {' _$ ?# \- `0 l
configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.
0 a$ W% R9 m$ c# B( d </description>5 P9 l8 Q$ R1 p! \; x( P4 c
</property>
# z1 z6 a& h9 \6 z1 D <property>8 |- [9 x4 ?# F$ S! u4 ]
<name>dfs.image.transfer-bootstrap-standby.bandwidthPerSec</name>- H4 d+ g& a. M+ s: e0 W, z9 O+ B. T4 v3 O
<value>0</value># c n3 S% a, x
<description>( \( h0 F* m& r$ @* V5 j7 E; L
Maximum bandwidth used for transferring image to bootstrap standby
, a+ P. i$ u1 | namenode, in bytes per second.' S! x: Y- s5 z! V
A default value of 0 indicates that throttling is disabled. This default
) v1 N; U* X2 d1 L, B. g value should be used in most cases, to ensure timely HA operations.
. R+ B$ B% Y9 I The maximum bandwidth used for regular image transfers is configured* i6 n% r5 I: M q
with dfs.image.transfer.bandwidthPerSec.
1 v5 k( W1 o2 j3 }( s( w' {- E </description>
- b9 {4 g2 X" u6 K </property>4 s6 V* c" b, j$ e1 z
<property>& q/ o# a: b# c2 u, U5 U
<name>dfs.image.transfer.chunksize</name> l/ n( u# `' S( w& U
<value>65536</value>
; ^; v" Q* i3 s; u5 e* w! i <description>& C7 S5 Q4 ?* v8 e* O# `# E
Chunksize in bytes to upload the checkpoint.
2 ]! @$ L, j" R. ^2 ~* q Chunked streaming is used to avoid internal buffering of contents
( z- g/ c, m# q: Z/ v) f2 @ of image file of huge size.
1 `0 Z" `+ R7 c* f; M+ _; E </description>
: [. U4 ~2 F0 q9 Y; d& V/ `- Z</property>
/ |& \: a1 L" e9 M<property>
, n% s# S: `! M9 h <name>dfs.edit.log.transfer.timeout</name>
9 z x) q- A) Z1 a. }; N2 ~ <value>30000</value>
% W* Z u% i" x' ]% Y <description>" ~ `+ m5 H! t2 f5 |
Socket timeout for edit log transfer in milliseconds. This timeout
/ ^* V; m: n; }& ]6 s) i" r, Y should be configured such that normal edit log transfer for journal% Q) i9 ?* s" L5 {0 j2 ]6 g0 W/ o
node syncing can complete successfully.
/ V n4 {/ `' i: ]7 Z# o0 o </description>
N, |" N# ^1 X+ j4 f9 d( T</property>
6 o4 l1 X8 h+ p# C) m0 g6 k<property>
+ R Q! Y/ m) _" G9 A, x <name>dfs.edit.log.transfer.bandwidthPerSec</name>
h5 A% s2 p$ Q @ <value>0</value>
* e5 f; d( a ~4 H7 P <description>
) n( P. j' O8 B' p Maximum bandwidth used for transferring edit log to between journal nodes6 }) n6 x' [6 r7 w- w
for syncing, in bytes per second.! R0 y2 f/ [& ^. [1 \. L
A default value of 0 indicates that throttling is disabled.* E2 ~9 K- U2 n+ { b! A5 P
</description>
3 H5 M2 T' @6 b9 p4 x: V& Z% ^6 Q4 v* n</property>
- M7 z1 t; m! T: }<property>
: c2 ]# S) g( _9 w" T: A <name>dfs.namenode.support.allow.format</name>
- d' D* R% V5 a! P6 G0 j <value>true</value>
* c! `" ^" i0 h% z. \7 \ <description>Does HDFS namenode allow itself to be formatted?
# ^3 r+ r! N9 h8 O# l: M; C( ~) H# F You may consider setting this to false for any production
+ i* i3 L2 d+ F5 {7 \! { cluster, to avoid any possibility of formatting a running DFS.: v) n* g1 U5 I4 [$ K8 f
</description>. d) \: Y$ Z; K" U! G
</property>
! q+ w/ U8 a" e3 q; G9 c4 f/ E, h<property>$ o/ ^; z k% j# p* I5 {$ m8 T: Z
<name>dfs.datanode.max.transfer.threads</name>% t- y: I% @ g' k7 }" ^! [
<value>4096</value>
/ v4 r! y. U2 \* ]7 P) w* p <description>
( T% |2 {$ t0 \# L Specifies the maximum number of threads to use for transferring data
) j# W: F- F, F& ^ in and out of the DN.+ _6 P# N" {6 z" d- R1 K1 p4 U
</description>
4 A, D i7 A L$ G+ ?</property>" m) q6 V& Q5 ]
<property>) i; w/ m+ M) T5 [
<name>dfs.datanode.scan.period.hours</name>
+ }4 J( ~, g. e" D9 z <value>504</value>
5 B- n" }4 O5 F% N/ W <description>
, [/ e& \+ S4 V& a. [ If this is positive, the DataNode will not scan any; r4 o6 P: b6 H8 d
individual block more than once in the specified scan period.- _2 {2 V/ k7 D) j. ~
If this is negative, the block scanner is disabled.5 [% v4 N2 `$ H& V3 n7 a w
If this is set to zero, then the default value of 504 hours
" G+ O8 s( X+ N, M or 3 weeks is used. Prior versions of HDFS incorrectly documented
2 k3 e+ I; b, J" j; C% l) b$ l that setting this key to zero will disable the block scanner.4 I- x5 @: P, m
</description>! Y' A T: M4 p7 Q/ L; p% _
</property># ~* g# @: r# i; l0 f8 l
<property>' v k" K j3 j3 b
<name>dfs.block.scanner.volume.bytes.per.second</name>6 f8 J5 X, X5 R. L6 j
<value>1048576</value>! l0 a$ Z" r6 q2 H8 x6 v9 g
<description>
6 Y. h/ ?, z {0 w7 y If this is 0, the DataNode's block scanner will be disabled. If this- i, ~0 Y% | L3 k1 T( H
is positive, this is the number of bytes per second that the DataNode's' ^. V( H! T R: o. r0 P# \+ v
block scanner will try to scan from each volume.
! Z+ } w6 y1 S( u5 ^! c </description>
1 h6 g n1 _, g4 W+ ^</property>
- y& D, e, L- D4 t, v<property>& h- L p. s- X4 Y U
<name>dfs.datanode.readahead.bytes</name>% H: [7 o% B% t$ t, r
<value>4194304</value>, A3 P' |1 e2 ~( l2 \
<description>( x# k0 {4 G% Q" y w
While reading block files, if the Hadoop native libraries are available," c. f7 |4 c$ ], u" n
the datanode can use the posix_fadvise system call to explicitly U3 t& _4 @1 i+ K6 E4 r) k
page data into the operating system buffer cache ahead of the current
9 g o6 H. y. c4 x! Y( x4 r reader's position. This can improve performance especially when
- P' X6 K& T$ ~+ m8 @ disks are highly contended.- H; Y4 d) v! S8 H: W4 G% F
This configuration specifies the number of bytes ahead of the current
6 L3 i, ~# G7 I# a# B2 W" l5 W read position which the datanode will attempt to read ahead. This
3 D' n3 P+ i" p; q ~ feature may be disabled by configuring this property to 0.
8 ?3 q5 H5 Y4 E% g: g If the native libraries are not available, this configuration has no
+ y$ h5 q6 v" Q) F effect.
* P {# Q8 o O, w8 z/ [2 y </description>, J9 {$ L8 I" Y* d/ O
</property>/ p7 }( _0 t z8 D7 }! Y
<property>6 Q" V% B9 u ?4 s% ^' E
<name>dfs.datanode.drop.cache.behind.reads</name>
5 O" d2 y' m1 B <value>false</value>
: E6 ], K, x# f <description>1 {, ^ d' b- m t' y& ]
In some workloads, the data read from HDFS is known to be significantly& k, v1 |! y9 \# ?
large enough that it is unlikely to be useful to cache it in the: C0 R7 x. e6 d
operating system buffer cache. In this case, the DataNode may be' O( T' k9 d# f
configured to automatically purge all data from the buffer cache; p5 w4 o/ Q) ^
after it is delivered to the client. This behavior is automatically
0 _6 @1 i( j) s7 @ disabled for workloads which read only short sections of a block
8 ^5 Z- I0 S7 m2 {+ x (e.g HBase random-IO workloads).2 \$ G3 I6 ?5 K/ l; f" {& I$ }
This may improve performance for some workloads by freeing buffer
' q0 o" F5 H5 z- e cache space usage for more cacheable data.
3 y3 J2 a( k3 s! ] If the Hadoop native libraries are not available, this configuration
* N+ f% F' G9 s$ k7 O; W, M; _ has no effect.
$ E5 Y# F. I% f' @ </description>0 z) c6 G H# o7 L6 g- Z) w1 I6 K+ v
</property>
w7 H) F! {( [% W. i. ~, `<property>" U$ H5 i; Q- K2 A" j' p8 N
<name>dfs.datanode.drop.cache.behind.writes</name>; y4 ?! H5 f8 ~! V6 H
<value>false</value>
8 K" v8 h* s( V* d% t <description>
( c2 j% b A: z0 J0 { In some workloads, the data written to HDFS is known to be significantly" X3 B2 ~& Q( n) n* D" D! W& v) c
large enough that it is unlikely to be useful to cache it in the9 u4 F- q+ V( n. H8 p
operating system buffer cache. In this case, the DataNode may be: B7 `2 K' v6 I
configured to automatically purge all data from the buffer cache
) x! V) ?& @5 n1 b# R% _) M after it is written to disk.
) ]- A$ |; b# v6 H( w This may improve performance for some workloads by freeing buffer
( l4 | `2 N$ `( I- h {& ^ cache space usage for more cacheable data.+ Q# n9 s. N9 j* g. ?6 D1 L
If the Hadoop native libraries are not available, this configuration
3 g4 x% r4 E" |* \ has no effect.
* x+ }5 }6 }. c7 a. e </description>
/ a8 F, Z1 D+ X* |1 C! H6 [: `</property>- j! i4 j& Y) X6 p5 {4 \; L
<property>0 o0 y6 I+ d" K
<name>dfs.datanode.sync.behind.writes</name>
& n; A( }# F4 O4 D9 g <value>false</value>
1 e) H$ A( \, o1 R" ]* O0 P* ] <description>
7 H O! `) z8 P" r) _. u: k If this configuration is enabled, the datanode will instruct the
: B0 x. D1 J( e' h1 k1 L5 B0 E operating system to enqueue all written data to the disk immediately+ C$ X5 v1 n/ C+ D
after it is written. This differs from the usual OS policy which
: u1 y" q* L$ K, @ may wait for up to 30 seconds before triggering writeback.
4 x3 _- T) _5 ?8 N0 a4 M This may improve performance for some workloads by smoothing the+ b8 \7 i& y$ m* \' f
IO profile for data written to disk.
# R1 I7 X s* l/ `9 d8 M If the Hadoop native libraries are not available, this configuration
3 j, R2 N( V7 ^9 } V; ^ has no effect.
1 z( P/ C' G0 m7 c; l/ Q </description>8 ?! ?0 q$ G9 A' R3 \
</property>5 q5 _. y: Q; Z$ x) P/ ~5 H
<property>
9 J3 Y/ k h8 D& }% r2 c" Q <name>dfs.client.failover.max.attempts</name>0 \* \1 D' g+ f6 b
<value>15</value># u' N1 H/ `0 |- f- P; g0 h
<description>
) x: h" W' F$ Z) F- r' x5 m Expert only. The number of client failover attempts that should be
/ N+ R2 n# [% P$ u1 o) B* B+ C1 B made before the failover is considered failed.. L3 W# Y1 Z* K8 c! K( o0 A# [+ K- m& d
</description>" d9 }2 O# O; P+ b) X# h( B7 G, I
</property>
+ o$ W8 J5 T, n: o8 J4 N+ ~<property>
* E; q4 q7 L3 f5 f3 F+ j- [* l <name>dfs.client.failover.sleep.base.millis</name>
) F0 |/ n+ b5 t5 c, T <value>500</value>
# l8 W% e5 a( B! K& ~5 ^5 `! I <description>
; X' }# t- p; N6 f9 v1 G Expert only. The time to wait, in milliseconds, between failover
" l* d" N. m! d ]0 m% s7 w attempts increases exponentially as a function of the number of
" n v3 O5 v8 T7 f+ m attempts made so far, with a random factor of +/- 50%. This option
/ q4 X0 H/ w; S& i- \ specifies the base value used in the failover calculation. The
* k& G' t, ]& @7 y8 @9 g first failover will retry immediately. The 2nd failover attempt9 p5 M. l% Z9 f7 E# |
will delay at least dfs.client.failover.sleep.base.millis4 T, E' j' V) Z8 @5 { V/ F: M! Z
milliseconds. And so on.
9 Y* N# b9 q' I p </description>8 X x) `# Q$ {" c, ]' v6 z
</property> n& X% ?% x) y7 T5 d- X: w
<property>/ R+ a' [; x2 r- H3 V6 K! n, _
<name>dfs.client.failover.sleep.max.millis</name>1 t. w# W6 E6 R
<value>15000</value>
3 D& k; Z g# F# G% X* `* R <description>
1 G' o2 P; R! U( V8 L. T Expert only. The time to wait, in milliseconds, between failover5 S/ z7 Y: q3 H3 C" w
attempts increases exponentially as a function of the number of, P0 @' I E, W1 C/ C
attempts made so far, with a random factor of +/- 50%. This option1 {' @# }# i( U3 D
specifies the maximum value to wait between failovers.
7 K1 _0 t+ h6 H4 A2 b' W Specifically, the time between two failover attempts will not
* U2 d. U1 e6 O- W exceed +/- 50% of dfs.client.failover.sleep.max.millis7 a0 P6 Y) S* p
milliseconds.# x5 \5 d6 M0 K4 h& c) i9 q, e0 P
</description>
j+ @, I/ X" v% E0 Y+ T7 k</property>
" T$ I4 f0 `- E' j: I$ a/ _<property>+ u2 v7 T$ V2 [+ G, s: S
<name>dfs.client.failover.connection.retries</name>& i, ~. X$ s( Z, K2 L* T9 R3 W
<value>0</value>
/ j( _$ N! x4 z' Y. G+ w/ f <description>8 W9 o2 u6 @! R) u3 k) A
Expert only. Indicates the number of retries a failover IPC client
( w) L- }( A X6 m will make to establish a server connection.
/ N2 A. S8 x: d3 p. R </description>
9 k' K$ F6 s0 _8 Q+ |9 n( }9 _</property>, u' R- m& `( S1 w
<property>9 ^8 V. j7 S7 @+ F; P% ` s! Y/ L/ B
<name>dfs.client.failover.connection.retries.on.timeouts</name>
! M/ ]7 B) E' l& k <value>0</value>: y7 E7 C: K) B3 I- K+ S
<description>
8 V. L5 Y6 _. k+ L. H Expert only. The number of retry attempts a failover IPC client
3 a `# U5 o/ T/ {+ \) ^. ~* A+ ^1 d will make on socket timeout when establishing a server connection.( [" y( x/ ?- a5 M! M% r9 W2 X; Z
</description>& X0 P9 M' M) `2 _# g
</property>
Z7 i' ?% z& q! P& R<property>" V7 \1 v# W7 L0 X1 [# k; |7 m6 w8 ^
<name>dfs.client.datanode-restart.timeout</name>/ E2 Y4 C& J9 |. V* D0 Z3 W1 }
<value>30s</value>- |# p- n# [2 U3 j8 p# ^* G! t
<description>
9 y1 S5 K6 K7 _ S l Expert only. The time to wait, in seconds, from reception of an2 ]& |$ b3 O, k4 y2 _
datanode shutdown notification for quick restart, until declaring z7 K: `; y4 F& K( p
the datanode dead and invoking the normal recovery mechanisms.
. Q! C3 z: Q' j The notification is sent by a datanode when it is being shutdown2 f6 R! p+ W1 q# V) w8 ^; n# Z. _
using the shutdownDatanode admin command with the upgrade option.
( O0 Y% O4 z. B Support multiple time unit suffix(case insensitive), as described( ^/ H: ?: H% J% j/ I: c" C
in dfs.heartbeat.interval.
1 x/ t s. M$ j8 X' E8 B5 U! u </description>. i* z2 O8 O9 I' V/ l
</property>' }8 r3 C) r2 s( j3 g
<property>
1 z# H% b1 ^3 c; Z# A <name>dfs.nameservices</name> K' y3 b9 s+ J5 x B0 J
<value></value>
3 H$ G: s( }; o! b <description>
' u( @* I+ S% }' I8 T& \" L Comma-separated list of nameservices.4 q: H. H" q1 j2 ]/ `4 O
</description>, X; m, P7 U7 K. E. ^
</property>
8 U8 p/ L0 c" m' r<property>3 g$ f B5 t' J. a
<name>dfs.nameservice.id</name>, {' m: r9 Z6 m. v7 ^7 X2 N5 q
<value></value>
; p7 ^, y: q# W7 B; a+ `! @ <description>
$ ]6 x" J% w d The ID of this nameservice. If the nameservice ID is not
7 s& E8 S( P% N5 [9 x, z configured or more than one nameservice is configured for Z, j3 ?* x/ s
dfs.nameservices it is determined automatically by# `7 ~" _0 G( Q
matching the local node's address with the configured address.
) j- C1 J5 P& J) V </description>
( z7 i( N5 z# b& V6 p. t2 G</property>: w7 q+ L9 W+ h" s7 M4 G0 V
<property>
6 N) a7 W, I) d; A6 ~9 s <name>dfs.internal.nameservices</name>) w6 `% j5 U T0 V: K
<value></value>% B6 J# h& |2 S5 ?" a
<description>
2 W7 O' Q$ S5 t& w Comma-separated list of nameservices that belong to this cluster.
' q8 J: `4 w2 t. S, i3 V* c Datanode will report to all the nameservices in this list. By default5 L2 u, G$ M4 @9 o9 F4 y9 x% E: G
this is set to the value of dfs.nameservices.* o9 h; V1 M ]+ @
</description>- O9 X$ {2 \) r0 k
</property>
( _5 [0 Z/ n; ?, n* ?<property>
6 o7 ?; h- E2 L7 _! w4 U3 I <name>dfs.ha.namenodes.EXAMPLENAMESERVICE</name>9 [: o' t8 [1 n N+ C: A
<value></value>9 R: n0 s! ^# @" i/ I2 S
<description>. D. {+ {# M" [8 I3 i
The prefix for a given nameservice, contains a comma-separated: f; ?1 G$ c% a7 ~) I
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).6 ]7 s: `- M$ a( c
Unique identifiers for each NameNode in the nameservice, delimited by! R$ m/ F6 R5 c
commas. This will be used by DataNodes to determine all the NameNodes
' y8 A; h( k/ a" O3 P- C4 X# i0 }; W0 E* z in the cluster. For example, if you used “mycluster” as the nameservice
: r3 `8 p; t! w& S8 z* R ID previously, and you wanted to use “nn1” and “nn2” as the individual6 H/ d$ f$ ?9 z4 X, [3 k2 k% s
IDs of the NameNodes, you would configure a property
( M1 i4 n5 y. w4 S1 q+ @1 B7 Q dfs.ha.namenodes.mycluster, and its value "nn1,nn2".5 F. y x/ S) o
</description># K" _, [5 a7 ]' t9 S; y! v
</property># d- z" |& h/ U- K' Y2 S9 _
<property>
7 r+ Q, I; X6 x8 f6 g$ Z7 s <name>dfs.ha.namenode.id</name>0 L1 P) j& S0 h5 ]% y$ E( |
<value></value>
( I# _; a) U. q! A- i* C0 s <description>
! B( k; I) v7 i: { The ID of this namenode. If the namenode ID is not configured it
% I2 a2 C# l: l1 y2 C* O9 U! y is determined automatically by matching the local node's address
7 W) z3 F& ]0 J; M4 e. s with the configured address.
: @; l& b0 q6 Z% x. ~$ Z# v' @) k </description>" M2 F0 a" G4 P5 t& a
</property>
4 A( T6 ~1 j: i* i6 z* Q<property>
: T1 ^& o7 U' C' N% Z/ J <name>dfs.ha.log-roll.period</name>
) q& J5 u% i; z <value>120s</value>
+ Y: H+ f; l- S/ E5 \7 p$ X$ q9 ~2 [ <description>3 B7 W9 V: g" k0 P
How often, in seconds, the StandbyNode should ask the active to% W$ P) H/ Z/ f/ B. O
roll edit logs. Since the StandbyNode only reads from finalized, M! z- ~" G+ m+ p+ p V
log segments, the StandbyNode will only be as up-to-date as how
! Z' i' e6 i5 W, @ often the logs are rolled. Note that failover triggers a log roll
& j8 _& [# T3 \! U5 n+ V% U6 `+ _ so the StandbyNode will be up to date before it becomes active.
( a5 ~% V: n4 r. J+ N# B Support multiple time unit suffix(case insensitive), as described: s) L' E6 b* z& S) A4 m% \" L
in dfs.heartbeat.interval. \) v& B% `6 @ G$ L
</description>. P' g3 M6 N4 W: E
</property>
6 Q8 G) p" e# \. F<property>
. y( z" L8 A Z3 q( N; e4 i <name>dfs.ha.tail-edits.period</name>6 e' w( H: O, a7 L& y
<value>60s</value>" q# \; o, _5 ]
<description>
7 A/ q% D$ y9 j7 N1 o How often, in seconds, the StandbyNode should check for new* h, [6 }1 P1 P) k
finalized log segments in the shared edits log.6 ~; l) L) \. B+ Y1 j
Support multiple time unit suffix(case insensitive), as described+ p3 w! C0 @ V: V1 a0 d c% T+ x& |
in dfs.heartbeat.interval.# \7 D/ |4 [, ^7 j7 G
</description>
1 c3 [ T' a" E6 l7 ~" ]" {, t6 |</property>
( w. h) d/ X# [<property>5 E. v$ x5 X; v0 j5 Z
<name>dfs.ha.tail-edits.namenode-retries</name> P g; N: I2 {
<value>3</value>
1 i3 n' H( b* A/ ]2 I8 m <description>" g' a- r+ |; w# @9 h1 ^
Number of retries to use when contacting the namenode when tailing the log.9 B. {2 M* x, S: ~# I+ i S
</description>9 t' h6 X' C8 _& K
</property>* ~3 `' w( y) Z! H
<property>
, I$ [: p9 Q/ Q. D8 T, w' I) j <name>dfs.ha.tail-edits.rolledits.timeout</name>
2 R" ~& @! Q1 {' B: m <value>60</value>. ?1 v/ y C4 i/ Q7 D9 R
<description>The timeout in seconds of calling rollEdits RPC on Active NN.
, e7 O# P" ^- s& m. ` </description>+ T& ~) e- W# A( h; t) U* U
</property>
! N. P& p9 Y5 b<property>
7 E- a" S2 J5 }& G <name>dfs.ha.automatic-failover.enabled</name>2 `7 k! H% `0 x) h2 J9 y
<value>false</value>9 E) J, J- \# n5 u1 u8 m4 U
<description>' G$ l+ [) f( Q0 \8 @' g3 k
Whether automatic failover is enabled. See the HDFS High g; W7 r5 [. M# V
Availability documentation for details on automatic HA9 G; U8 _6 A, `3 p3 P. {
configuration.
& Y6 B- {6 S! G! ] </description>
" S/ V* ^; `& N6 k, }/ v& k</property>6 x. d5 @7 K% w' s" W
<property>
- E4 O7 A8 s' a <name>dfs.client.use.datanode.hostname</name>
4 h* r. B3 k9 l' {0 }0 a! _3 w$ {* A <value>false</value>
4 b% Q5 Q* q1 p* l <description>Whether clients should use datanode hostnames when
8 J2 {1 s( v; p. e- {% \/ c connecting to datanodes.) @) |8 s7 l, Q7 o: g) V3 v# W
</description> a2 U, y, Q# J2 N
</property>; N* Y/ l1 [. r9 P3 s8 [/ m# U% i
<property>
& I' x- {" L; i; O) A- h <name>dfs.datanode.use.datanode.hostname</name>8 O' m+ v" e' w6 ^& i) H
<value>false</value>' R; D! d8 K% ]% v8 S% H i
<description>Whether datanodes should use datanode hostnames when# H- o5 R+ |( N$ @. f
connecting to other datanodes for data transfer.
- i) ]5 G7 K J+ y( R% Q( k </description>' |; H6 h, r$ Q- M( _
</property>( O9 v2 n) K; Z$ W
<property>
8 |+ }! i% Z; k! c3 ~ <name>dfs.client.local.interfaces</name>/ z& t4 z- d. V. A. p+ ]
<value></value>
( J* |+ p% D0 ?7 i& ^ <description>A comma separated list of network interface names to use( g, L: }& g2 `' [/ M5 A7 b
for data transfer between the client and datanodes. When creating
1 T# Z" I) B9 S- ?- f a connection to read from or write to a datanode, the client
0 t# [/ j, J# q8 I chooses one of the specified interfaces at random and binds its6 \1 o1 l5 _, U; M- z
socket to the IP of that interface. Individual names may be& J& q8 V1 g( `' \
specified as either an interface name (eg "eth0"), a subinterface/ j2 c) `; ?0 z2 l
name (eg "eth0:0"), or an IP address (which may be specified using! r# }' h% A- G6 A' @0 b8 M: d
CIDR notation to match a range of IPs).
0 x/ x4 [: _5 [- y' H: M </description>
5 n3 P" W. K2 U. t: f</property>
% J: o' ?! e8 t<property>: f3 _% x' Y8 d) W; g9 S% [
<name>dfs.datanode.shared.file.descriptor.paths</name>' ~1 N, P: M7 u
<value>/dev/shm,/tmp</value>
! z, o1 _9 I! H& A6 y <description>
% k! y9 L! t. T/ Y( ~( A5 [% j0 G A comma-separated list of paths to use when creating file descriptors that
2 d( {: E$ ]8 ^) z2 ]! C. c will be shared between the DataNode and the DFSClient. Typically we use
0 R# g! m9 D7 ]( |1 X /dev/shm, so that the file descriptors will not be written to disk.+ T3 i$ h; B# R6 J* r
Systems that don't have /dev/shm will fall back to /tmp by default.
8 V- G. T( a# b# b </description>8 K6 K; y: V( }7 S! e8 _
</property>
8 C5 G+ o' q2 H2 _! @/ `<property>
. t5 d; j. x4 x <name>dfs.short.circuit.shared.memory.watcher.interrupt.check.ms</name>& B7 ` w0 E6 { ^! j+ y! o
<value>60000</value>$ h3 }: a) i( u
<description>) F. ~, Y# u( H6 w
The length of time in milliseconds that the short-circuit shared memory) e5 T1 q+ i. n: M- ?, `
watcher will go between checking for java interruptions sent from other6 V# M B0 G7 [/ i4 c! }* w
threads. This is provided mainly for unit tests.! [# s" m9 s, S. d
</description>
4 l2 \7 V- V3 x" ]( U</property>
4 S, W" t8 a2 V, D( Z; f1 Q/ D( @<property>0 b: ?7 k$ h/ L3 R" t% X& q1 A1 U! A
<name>dfs.namenode.kerberos.principal</name>+ z z5 v7 ]2 e3 \. L- d
<value></value>" T v: E8 s- `% h6 }
<description>2 k3 |8 P3 r" |. O
The NameNode service principal. This is typically set to4 g8 z2 u& U! h: S J
nn/_HOST@REALM.TLD. Each NameNode will substitute _HOST with its" \. O3 f l& i" M9 d0 M. B- n
own fully qualified hostname at startup. The _HOST placeholder5 Y5 L- f, ]4 h t9 l0 U0 [
allows using the same configuration setting on both NameNodes8 K3 [6 q7 O; i( R& f+ Y
in an HA setup.
G9 I6 e6 n% Q </description>
8 j; u3 g$ O- `% A) B4 R4 k</property>( Y* {! a5 E. @% l+ ^" t, o
<property>$ d n F& O4 Y- ?: \, J
<name>dfs.namenode.keytab.file</name>2 L4 j1 A5 X D% `- `' p
<value></value>
, N+ g+ x7 d+ ? M( Q <description>
) u4 @6 R, b7 d8 O6 v1 N The keytab file used by each NameNode daemon to login as its( {' H8 N1 Z2 n/ i/ c0 u
service principal. The principal name is configured with& z, ~( C$ \8 e( P- L4 l
dfs.namenode.kerberos.principal.6 d. b& Z0 h' H& \
</description>2 z. C! c: G+ K) y, b
</property>
+ P3 g7 `' O3 X<property>
" A; R/ ?1 \/ f3 L! l- j <name>dfs.datanode.kerberos.principal</name>- U, g2 f+ l: P" i! ?
<value></value>
+ e: R( j3 A" o3 \3 B <description>: @# t# m& s/ I! p
The DataNode service principal. This is typically set to
) @1 ~: v! u5 i/ N dn/_HOST@REALM.TLD. Each DataNode will substitute _HOST with its
" }# N3 x+ R4 ~1 t& P own fully qualified hostname at startup. The _HOST placeholder
- L# m- P3 J6 g; ^ allows using the same configuration setting on all DataNodes.
) `1 y+ a* g, j9 m </description>
4 z. ]" R. [2 D8 w& q, X</property>
( j& h* @3 z+ ~6 P<property>' j1 S6 ]/ c! n
<name>dfs.datanode.keytab.file</name>: Q) @2 ^* l' J! c
<value></value>/ M4 Y) [5 f+ F- S/ `$ L
<description>- z/ ^9 G9 j" D7 S0 `$ ^
The keytab file used by each DataNode daemon to login as its" U3 n) C$ R. j
service principal. The principal name is configured with
4 I- X: b+ z3 y. S; j$ f. I7 m( A dfs.datanode.kerberos.principal.5 f) |& Q; u+ r; c) u% e8 {
</description>0 U2 d/ v7 i n7 z% j* g( n
</property>
$ n% Y7 O d, N G<property>
9 y4 |+ R% V- k' ?& m: [8 `: m( y <name>dfs.journalnode.kerberos.principal</name>
$ F& B( j1 }8 q <value></value>9 G3 J1 |. a& q% R$ v
<description>, V* U% m: E; e$ ]2 N/ C
The JournalNode service principal. This is typically set to0 T T3 B4 |- ]. S+ j6 R
jn/_HOST@REALM.TLD. Each JournalNode will substitute _HOST with its
1 A7 i& r" w* {+ N own fully qualified hostname at startup. The _HOST placeholder' |+ Q6 |& J, j k* Z' o
allows using the same configuration setting on all JournalNodes.
8 v5 v: u7 _1 p& W </description># R7 e/ J; y% j% I2 r5 A) X" Z
</property>6 J I9 ^+ k0 |6 b
<property>+ m. r! V1 g/ i4 v
<name>dfs.journalnode.keytab.file</name>
; j( C. L# @% i <value></value>8 N9 j2 g. T# A% v
<description>$ m: Y$ ?" w, u) b8 w2 m* f
The keytab file used by each JournalNode daemon to login as its) s( h% I; i) F/ \4 B
service principal. The principal name is configured with- Z2 G% n) v6 m% x% ]+ i
dfs.journalnode.kerberos.principal.
$ N2 c& q3 E* U: y </description> q4 z- y6 y( Y" x$ k
</property>
) T9 P6 l; ?7 q3 M1 Q0 u1 L<property>5 m9 |6 [( `- C* i
<name>dfs.namenode.kerberos.internal.spnego.principal</name>" f3 G u) l/ ~+ l9 Y
<value>${dfs.web.authentication.kerberos.principal}</value>* ]1 ^, E2 h- X
<description>9 H" n9 p6 A: I) I+ V0 L% G# d
The server principal used by the NameNode for web UI SPNEGO3 T/ B. H' y5 L4 p' I" J4 W5 g: H
authentication when Kerberos security is enabled. This is
1 t4 s' Z, _ t$ V! [- N; i typically set to HTTP/_HOST@REALM.TLD The SPNEGO server principal$ H8 Q7 p% \ `- l- B
begins with the prefix HTTP/ by convention.
1 _ `3 b: L' {# ~5 { If the value is '*', the web server will attempt to login with
( c- ^7 O& c5 q/ z4 T/ w, M every principal specified in the keytab file
+ m: i6 G8 e/ V: G4 e dfs.web.authentication.kerberos.keytab.* x$ J, q. P/ M5 w
</description>; J2 H, {) J; m; h( _8 i) l- @
</property>2 J, g& `) g5 T
<property>
0 |/ A9 W9 x9 D! F <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
) e: r9 m& N; Q8 | <value></value>, P8 o) U" S; |1 D* E( ~
<description>
$ ]3 D- Y6 ] p" }$ Z, X" o The server principal used by the JournalNode HTTP Server for$ N; s; F" j' a; n' @. f' z
SPNEGO authentication when Kerberos security is enabled. This is% B$ C. |8 C6 D) L
typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal) m' u/ f" f9 T5 \6 x$ @
begins with the prefix HTTP/ by convention.
9 K$ r6 l3 _% H; o# q6 ~ If the value is '*', the web server will attempt to login with
3 K1 ~, X! ~: C, W every principal specified in the keytab file
8 b4 O* o/ G7 y, W7 H* N0 U# E dfs.web.authentication.kerberos.keytab." c! Q: l* I& i s/ j
For most deployments this can be set to ${dfs.web.authentication.kerberos.principal}% q6 ] A- C) X; _' @$ f1 ~) W
i.e use the value of dfs.web.authentication.kerberos.principal.
; V/ r+ u3 n. @9 a </description>
9 j7 D$ F0 ~# l- P</property>' I2 x2 `; |6 G" ?7 s3 m
<property># H- {1 W; b# a/ N
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>: f6 B* S8 R/ j
<value>${dfs.web.authentication.kerberos.principal}</value>
0 R+ ~7 i/ H& S$ J) c$ T <description>; @ v6 m( e. a/ ~! H
The server principal used by the Secondary NameNode for web UI SPNEGO
4 C; q2 t! T( F& h" Q$ D authentication when Kerberos security is enabled. Like all other: A2 _ [4 u/ e' \% b- c8 Q
Secondary NameNode settings, it is ignored in an HA setup.
6 i1 h! y4 Z* Q E" ? If the value is '*', the web server will attempt to login with& q7 v* n3 B1 ]2 t' D5 Y0 j' T% G& j
every principal specified in the keytab file0 {: J2 ~* E# h% D+ E, C
dfs.web.authentication.kerberos.keytab.
& E8 G* \: y- \& {4 J* F </description>6 R- Y8 i$ J$ X! W, U5 p( O5 [0 N0 r
</property>
( S) ~: Z* f% F5 F<property>
, V' I* J: G& l# x( k <name>dfs.web.authentication.kerberos.principal</name>- `1 v2 s& R2 w1 f
<value></value>& K( k; B( Q1 F) u# G" @1 U
<description>, B N' E0 p' l' u M
The server principal used by the NameNode for WebHDFS SPNEGO( g2 ]. F) n8 c. V! d r# o2 I2 \
authentication.
& N, h$ ~9 f8 u2 P1 E9 T Required when WebHDFS and security are enabled. In most secure clusters this0 Y$ L. Q' h! Q* I7 E
setting is also used to specify the values for+ U6 o9 v$ ?# G2 R. S3 g& H6 R4 ^
dfs.namenode.kerberos.internal.spnego.principal and
9 J% b( @1 n# f; W2 ^ ~9 Q( ^. w dfs.journalnode.kerberos.internal.spnego.principal.
0 n R$ g) X3 t+ a4 U1 _# l% [ </description>- F- b$ O# B6 x/ i1 Y7 I
</property>
s7 n- |; i# ]8 e<property>
/ Z5 m+ @9 `4 T <name>dfs.web.authentication.kerberos.keytab</name>
$ U* ]2 Y" ^7 G2 O) X$ P <value></value>
2 N6 G6 ?$ w! Q8 ]& Z4 \ <description>
8 g8 T l0 ^% B The keytab file for the principal corresponding to
0 S/ U3 V9 F( I. f) H dfs.web.authentication.kerberos.principal.0 r9 L0 D% z/ Q/ e/ x: X. O2 x
</description>
J% N% R/ `9 Y9 R+ r3 ^</property>
( X, v5 o! B+ k8 {* _2 U; p<property>
6 d: n; e+ E2 A. j8 L <name>dfs.namenode.kerberos.principal.pattern</name>
7 R6 ?) o% `" J <value>*</value>
1 @, [. \, ~1 A, h% w4 T" S/ p <description>% Q' q' N" T/ W( V$ y+ h) x
A client-side RegEx that can be configured to control
1 B1 G( }, I6 p5 o% J' y' O2 M t allowed realms to authenticate with (useful in cross-realm env.)
" i0 O+ g4 N0 Q5 A, k </description>
. M; O2 \% Y& Q: Y* q/ z8 I</property>
0 }; O5 w! ~. z7 m: \ B<property>
+ s7 A0 G- c0 z2 ~ <name>dfs.namenode.avoid.read.stale.datanode</name>
3 \0 {0 p8 Z( A8 t <value>false</value>
5 H% a2 t' u) T7 U! t& r <description>8 T; {8 O* I, O% @
Indicate whether or not to avoid reading from "stale" datanodes whose# z4 A& A4 V* q
heartbeat messages have not been received by the namenode
7 W8 m0 {# i' [! @; V for more than a specified time interval. Stale datanodes will be
( K! f' ^9 l) K: O- k# f. g moved to the end of the node list returned for reading. See
; _* n5 f# s2 Q2 B' `6 O dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.
+ w& \/ q- v2 d </description>$ Q* g4 L$ t5 n7 q% q7 h
</property>
/ W- t% L9 o, { a! `<property>
* P' b3 ]4 U1 d3 w& t2 Z <name>dfs.namenode.avoid.write.stale.datanode</name>" Q. n+ M. W- p5 y& I0 x, J' q
<value>false</value>
/ ~ p1 C3 B" p" ^; K <description>4 c( O. i! u7 |# ^0 J
Indicate whether or not to avoid writing to "stale" datanodes whose
& D. w$ L" x2 M2 K+ P# q/ I heartbeat messages have not been received by the namenode
8 Y. a6 C8 |! w$ V0 s( C2 [ for more than a specified time interval. Writes will avoid using 5 g! s. D' g9 p' C: P
stale datanodes unless more than a configured ratio
) p: e" _) \9 M+ d h4 X! ]; ` (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as
. l; o! R3 r# u4 d8 Q$ m5 N2 C stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting
$ R8 Y( l+ B7 A0 R& Q: L5 ?0 U) L2 n for reads.# T3 e; [9 M$ W8 {9 }- ?5 c; x
</description>" k" V4 d" r3 G# ]4 B/ J9 y" b
</property>
z9 C9 R# Y. X- j! V2 X<property>; m* `! R, v6 c" t# z2 [) f7 L
<name>dfs.namenode.stale.datanode.interval</name>
3 V: A2 t+ b$ N b <value>30000</value>& W: T6 c: l2 ]
<description>- O$ q4 D- e* Y- j, S6 {6 j/ h" O1 ?
Default time interval in milliseconds for marking a datanode as "stale",% Y9 k9 c" h* T) H
i.e., if the namenode has not received heartbeat msg from a datanode for
+ ]- j/ L4 s: h' t! q6 N. u9 P more than this time interval, the datanode will be marked and treated : W! a2 x0 o( P0 `9 U; H
as "stale" by default. The stale interval cannot be too small since # \/ y: U) k. R6 N4 p; d- i7 V2 j. H
otherwise this may cause too frequent change of stale states.
8 @ N$ W7 C" O We thus set a minimum stale interval value (the default value is 3 times 4 L# }9 J. c" i+ G5 ~8 {8 c
of heartbeat interval) and guarantee that the stale interval cannot be less% R2 x+ X2 Q% c) y6 d7 ]7 K8 l
than the minimum value. A stale data node is avoided during lease/block
1 \" J/ d7 ?* d8 Z# v0 h- r! E recovery. It can be conditionally avoided for reads (see
g# q+ |* K4 ]% l# o: K dfs.namenode.avoid.read.stale.datanode) and for writes (see
' m, k5 \" c' _! q: J4 D3 U dfs.namenode.avoid.write.stale.datanode)." v: {+ X: F- I7 @6 L% H6 O& G
</description>4 G7 {8 ]: A, s8 t! `
</property>/ c. ~; j% t2 K1 G0 f6 i+ P) ~
<property>
8 Q1 Q4 g, A$ _2 J/ a <name>dfs.namenode.write.stale.datanode.ratio</name>. h+ i) e O% V7 v: \
<value>0.5f</value>
. B6 x( X6 p# b <description>
! H- q; h9 U$ e' j: k% F When the ratio of number stale datanodes to total datanodes marked
% k* V4 C- ?0 l" h: m! K8 | is greater than this ratio, stop avoiding writing to stale nodes so/ c4 t- C: I% @* `
as to prevent causing hotspots.
# m! o- b1 S0 ]% ]( r8 j9 g- j7 v </description>
" w% e$ D' h( d* v3 N0 z. L! b</property>
# ?6 E4 c$ c& z3 A<property>) d+ ?4 J2 M4 Y0 J
<name>dfs.namenode.invalidate.work.pct.per.iteration</name>
% ?& B6 x& g7 C& [! A <value>0.32f</value>% Y |) ?% Y* v4 d
<description>
. \" N) M0 g) I' K *Note*: Advanced property. Change with caution.( f) L% D6 k% i# w* H8 u4 m
This determines the percentage amount of block
! @' K! B$ N- \* M& W) N% |+ Z8 a invalidations (deletes) to do over a single DN heartbeat; p. t+ Z5 F O
deletion command. The final deletion count is determined by applying this
3 o' q8 n9 U5 V& B( T% | percentage to the number of live nodes in the system.. }+ Q j7 f( l" H6 |" [
The resultant number is the number of blocks from the deletion list) V0 @; R5 M& L2 C* |8 ~
chosen for proper invalidation over a single heartbeat of a single DN.+ g/ P% h' D% N0 s0 C
Value should be a positive, non-zero percentage in float notation (X.Yf),
% y5 ^, X1 @% @+ K with 1.0f meaning 100%.
* l; D* R7 m% O' W* N1 x </description>
( r* i2 U( d$ x: Q</property>$ W9 ~2 L4 ~0 S; V2 s0 S9 V9 N
<property>2 q0 e) W; n7 ?
<name>dfs.namenode.replication.work.multiplier.per.iteration</name>
5 y; c5 J: R6 b2 e+ w7 e1 E <value>2</value>
" K# C9 ^" a0 e <description>
* D% j# R5 C# [+ u1 G8 C *Note*: Advanced property. Change with caution.
9 V7 [! s$ L# R6 n: ] This determines the total amount of block transfers to begin in; R6 f" g9 G0 `8 Q9 {, Y
parallel at a DN, for replication, when such a command list is being
1 j+ U: ]/ L4 p5 @) }$ s sent over a DN heartbeat by the NN. The actual number is obtained by' K- @. n1 |& w2 t
multiplying this multiplier with the total number of live nodes in the
4 f. q' H7 O/ Z F0 y8 Y cluster. The result number is the number of blocks to begin transfers
8 v: w3 v6 D: ]- j9 k1 B) w immediately for, per DN heartbeat. This number can be any positive,
4 K+ q4 ~- ~- [ N/ g non-zero integer. Q2 P! d6 r1 K8 I% I0 p* }2 c
</description>: M& }! b! c5 F( I) ~5 n( L
</property>
& h$ Z/ P% _. I+ B+ I<property>
, ]) f# `# U& w3 x- X7 q <name>nfs.server.port</name>; N( f+ m2 a6 M2 g& o
<value>2049</value>
. W% e: ]4 }) j7 J& v% A1 x <description>
* F7 V% O( i; D2 z Specify the port number used by Hadoop NFS.
% R" M! y2 k- r; | </description>- d! Q5 \$ r8 \" z2 m
</property>
7 q3 a/ z3 |; G$ ]5 L: S$ z$ E i<property>
3 }; y6 q' W, V$ C <name>nfs.mountd.port</name>" w/ D! R/ F' S' D' s3 @3 P
<value>4242</value>3 d8 k: Y* H$ ~8 O1 S
<description>/ \4 j/ Q' @1 F. c" @7 ^* R
Specify the port number used by Hadoop mount daemon.! _- q3 p' A4 b" A" t1 }3 G
</description>
F/ A, q( m! C9 L C3 a2 |$ M</property> X' m6 q b7 @- H! H/ Z
<property> " G8 ]7 ?& J$ c0 ~
<name>nfs.dump.dir</name>
- c3 _$ h6 |+ S+ }' T' b <value>/tmp/.hdfs-nfs</value># p( ?* k5 a. D. ]$ L* f$ B
<description>5 r! B5 Z$ L& W+ U
This directory is used to temporarily save out-of-order writes before
, g% p# l. E9 N* m/ D1 U writing to HDFS. For each file, the out-of-order writes are dumped after
& J1 X' w" _; `3 P/ y they are accumulated to exceed certain threshold (e.g., 1MB) in memory. $ o$ d2 m2 A5 F6 _
One needs to make sure the directory has enough space.' |4 U+ e- O S9 U* |
</description>
& T2 W* n% u0 g- C</property>6 [9 \+ n; q- D( W5 O
<property>7 }0 }. W7 h5 T2 O! `) h' y" Y( Y
<name>nfs.rtmax</name>
6 [2 l) b% n8 R7 z# d <value>1048576</value>3 V/ `2 R% P+ X6 n b( d
<description>This is the maximum size in bytes of a READ request1 [+ K: Z0 k" ~1 E e
supported by the NFS gateway. If you change this, make sure you ~( X* q* z' c) L) x
also update the nfs mount's rsize(add rsize= # of bytes to the . b4 D8 [" C" B6 z2 o
mount directive).
3 d' Y. A5 ~+ g& R- Q4 | </description>( x* p; A' @0 l2 [: ^0 L+ N
</property>) X% E. W; A0 Z& Q. g0 K5 ]( Z
<property>
! c# U, d0 Z8 O" K <name>nfs.wtmax</name>
4 o7 T9 q! g3 E3 \9 M <value>1048576</value>
" S8 c1 o' S! @, d( K <description>This is the maximum size in bytes of a WRITE request9 O$ H9 m) {* F+ l$ q6 E) x4 n
supported by the NFS gateway. If you change this, make sure you
5 S2 N" D7 J! g. S4 h, p% M2 x also update the nfs mount's wsize(add wsize= # of bytes to the % n( J! Z! R, K7 k% k. r) [
mount directive).: @2 B9 b7 V/ b5 Y
</description>
4 O* t) s1 S6 ]</property>
7 m+ e- l8 y/ e2 m0 {* d; D' X0 u<property>% j6 r) _/ x- z1 o' a1 S
<name>nfs.keytab.file</name>: }# b0 ?! Q, a2 e
<value></value>$ L2 t! g: t! g+ p% H4 R t+ u o
<description>
9 x7 q7 `' `% Z. i$ k8 P. t *Note*: Advanced property. Change with caution.' n1 L- @: l1 c
This is the path to the keytab file for the hdfs-nfs gateway.8 q. g' u& l. d. g: }
This is required when the cluster is kerberized.. f) i6 g$ n3 F) X$ a' G
</description>2 M- e+ l5 ~6 a8 w1 @7 I
</property>$ e' {+ S0 M; W w7 \. z# ?- I. ]
<property>
8 R% }& z2 x4 w; q" Y$ Y0 s <name>nfs.kerberos.principal</name>' T. B- @, c* Z8 B: c
<value></value>
; z" o2 I4 S) Y0 w& G <description>
+ _* u1 ?; w3 ? *Note*: Advanced property. Change with caution." Q& U3 S+ g, ?# q H" M
This is the name of the kerberos principal. This is required when* g; `: i) E; I# j
the cluster is kerberized.It must be of this format:
8 g6 o$ n7 w4 ^: o- U2 i nfs-gateway-user/nfs-gateway-host@kerberos-realm
# d( J- }6 U2 b/ N6 t% q </description>
0 c7 d5 j2 q/ f9 i7 s6 U# G) X4 a</property>
, f/ g# V x+ l9 I<property>( L* r% ~ r. e- @
<name>nfs.allow.insecure.ports</name>+ o6 L5 f9 Z: P. H% P
<value>true</value>* D" _: V$ g, ~8 R: V
<description>
F9 ~, ?+ z4 H7 J* Z- ~' U When set to false, client connections originating from unprivileged ports. M* a6 B: }6 a& g" v
(those above 1023) will be rejected. This is to ensure that clients
& ~0 ?3 [+ p, f) c7 d connecting to this NFS Gateway must have had root privilege on the machine
' D4 G/ b) b& ]' d" }8 ^ where they're connecting from., A; a. ^9 J) z- [2 g5 B
</description>, W, F1 S8 f( m% [& \
</property># }9 M/ V- a4 ?, G3 G0 ]
<property># N" h! Z( Z( M8 ]; q9 ^
<name>hadoop.fuse.connection.timeout</name>
9 }8 D* Q8 F" P' u1 g8 r( s/ ?9 u <value>300</value>
* s9 l3 ?( G, j$ M) }1 F <description>6 w; d4 B* |% ~9 |& m P- O
The minimum number of seconds that we'll cache libhdfs connection objects
" M. x- D" j( y, Y0 g1 O! [ in fuse_dfs. Lower values will result in lower memory consumption; higher+ _* o0 I: W4 ]" s, A/ D
values may speed up access by avoiding the overhead of creating new
; @1 F+ G. C$ p. ` K connection objects.. x7 q% I; J9 K* n; l& l P8 a
</description>+ N2 Q7 Z# T0 {6 U h. N
</property>1 f! _* H" d6 f- X; q
<property>% y5 b- O# t6 C: f
<name>hadoop.fuse.timer.period</name>2 b5 c& d! g3 `9 r# z2 m7 {" F
<value>5</value># B/ z. {# @6 q* ]& [) m
<description>
8 T6 O! w/ U2 w The number of seconds between cache expiry checks in fuse_dfs. Lower values5 |7 i z* h- L3 @! U+ X
will result in fuse_dfs noticing changes to Kerberos ticket caches more; y0 j* l& S0 G" ]5 ?
quickly.
5 l9 r8 A4 O a$ K </description>) s' a2 L. S- }' ~0 M. ^# U
</property>
/ `: I6 Z5 P( R/ x# {4 z<property>( b, }- h7 Q5 Q3 ?$ B2 {' T
<name>dfs.namenode.metrics.logger.period.seconds</name>" U! c5 l* `" O+ n: _. w a
<value>600</value>
/ {5 a+ V# }$ K' Y' d. o _ <description>
7 Z; C9 E1 x# i4 Y- `7 { This setting controls how frequently the NameNode logs its metrics. The
[: }' Z2 E, H( x# V/ g logging configuration must also define one or more appenders for6 o" A2 ], E; i( j# a# q; P
NameNodeMetricsLog for the metrics to be logged.
3 E% W( O7 O [ NameNode metrics logging is disabled if this value is set to zero or
% J C6 |! M7 P' J4 c" K$ r9 F/ A less than zero.
+ W5 R G8 F$ s3 H! m </description>* ]3 Q( q9 X2 ]8 j- r, n
</property>& }" \1 M! \9 f6 B6 P
<property>
0 t, B, r& }( G4 v7 m8 V <name>dfs.datanode.metrics.logger.period.seconds</name>* `, B4 B5 ]2 D) v: s3 b8 ^
<value>600</value>( @3 |/ ?6 ^3 \8 J3 z6 \0 g
<description>8 [6 l* F0 U0 K. c1 p
This setting controls how frequently the DataNode logs its metrics. The( d3 B' H% L& S9 C- ^) ~( X1 J
logging configuration must also define one or more appenders for
* @" w- E! z, o' e" L DataNodeMetricsLog for the metrics to be logged., f \1 R9 h, q- t% s0 Z, L: [
DataNode metrics logging is disabled if this value is set to zero or
' N8 i$ A5 d7 Q0 K1 T less than zero.
, t P" O& l) i" f9 A </description>2 p _" i# f* Z) r f3 I
</property>
9 u+ H0 o7 u6 A& p1 D+ H% ~<property>
1 z$ Z. L( d* a1 p) K <name>dfs.metrics.percentiles.intervals</name>
! _# I& z9 u; w7 p <value></value>
; l: A) k( M, H3 _+ A1 C0 ? <description>6 g z# t4 _2 l, [0 g5 L
Comma-delimited set of integers denoting the desired rollover intervals
6 b8 {! h% K. B% o( w (in seconds) for percentile latency metrics on the Namenode and Datanode.
" S. P& C" \5 s7 } By default, percentile latency metrics are disabled.
8 {# f) a& |/ p8 W9 ~ y' t </description>
6 f- E5 V6 | ?7 L4 m3 k. o6 ? D</property>( s- _( \3 w( g
<property>
& o/ z+ y2 P9 N: {+ @1 \ <name>dfs.datanode.peer.stats.enabled</name>9 s/ {, G! Y$ ]! Y5 I. j
<value>false</value>
% l4 @( a6 X- F0 p <description>6 x4 e3 h- `9 t; u
A switch to turn on/off tracking DataNode peer statistics.
q; \& |' q0 _. y. { </description>" d8 {( B4 C# s8 O. x* n0 {( ^6 m
</property>
9 D6 j; p* C0 _0 r<property>6 a6 E7 o; r; k6 G, b
<name>dfs.datanode.outliers.report.interval</name>! |% Q1 J; h2 h7 [
<value>30m</value>
8 {9 @/ `; M/ M& r <description>
5 I' K' _6 \1 @ This setting controls how frequently DataNodes will report their peer0 i9 |) R4 e4 R' O/ S' I0 _7 {
latencies to the NameNode via heartbeats. This setting supports$ `4 m8 \! r9 J: J( z
multiple time unit suffixes as described in dfs.heartbeat.interval.
) E7 P- ^7 D% t. T8 h' ]8 n) S, K+ t If no suffix is specified then milliseconds is assumed.5 z; H- m8 a R8 g" P
It is ignored if dfs.datanode.peer.stats.enabled is false.
& r1 o! l" Z; `$ j1 R </description># g5 h4 M/ w$ @4 j; e& c0 H @9 {
</property>
" J" X9 V/ G! z' Q" G<property>
4 B* Y$ @" O( ^! r <name>dfs.datanode.fileio.profiling.sampling.percentage</name>
( X& b( j: a7 s& ^% M# z <value>0</value>" V8 D0 n* Z. C% N
<description>
% Z N" z0 }9 [0 J$ F$ e$ F This setting controls the percentage of file I/O events which will be
8 f7 l/ B6 x1 ~+ W( K$ y0 U# ]/ A profiled for DataNode disk statistics. The default value of 0 disables
% k8 G* A4 x3 p3 M disk statistics. Set to an integer value between 1 and 100 to enable disk" k5 m8 l7 s3 u) H3 S- o
statistics.
# _% W/ d4 n, f, b9 @7 z" S </description>, t5 q. L \( S$ s( t
</property>; y0 ~/ A" S- L ~! Q0 _* R
<property>
& X- E3 e4 K* b, `- H <name>hadoop.user.group.metrics.percentiles.intervals</name>
8 \- @& D. b, h <value></value>
+ M5 {, _3 C; O- W <description>
8 n9 `( E5 {1 P9 @7 J3 H% W7 m" q A comma-separated list of the granularity in seconds for the metrics
; _1 A2 Z7 q M f3 T which describe the 50/75/90/95/99th percentile latency for group resolution
/ p1 }1 S( \+ i M' x1 A in milliseconds.
& j: V: r7 r$ ~; K V By default, percentile latency metrics are disabled.
9 @3 M1 U& j$ M) p </description>
% ?) c0 x) A8 M# X4 \</property>
: `; Q+ q- Q' c<property>% R3 l( Z' |5 @# ], M2 D! J
<name>dfs.encrypt.data.transfer</name>' {6 Z- ~% ~8 `5 P9 g2 x
<value>false</value>
7 p, [" O& [. V0 a2 D <description>
4 \2 b# [: u( w; j$ h, h Whether or not actual block data that is read/written from/to HDFS should
) e& [: y6 C I be encrypted on the wire. This only needs to be set on the NN and DNs,' P& u) f* ~' C# |# y) Z- `
clients will deduce this automatically. It is possible to override this setting ; E$ U7 e4 h6 D# ^% k
per connection by specifying custom logic via dfs.trustedchannel.resolver.class. 8 x- N/ X) L; [& u
</description>; J1 R2 R! b" z2 o* c7 P# c- }
</property>4 m( E4 f' P/ f5 d
<property> {" U: _* l( j
<name>dfs.encrypt.data.transfer.algorithm</name>
: @/ M& ?& B$ X: B/ o1 E <value></value>" p v4 O& [, U
<description>/ D* m! f4 \1 f3 d
This value may be set to either "3des" or "rc4". If nothing is set, then3 W+ u Y2 l/ \0 D0 C
the configured JCE default on the system is used (usually 3DES.) It is
. E7 c; Z/ v5 X, i3 [7 l& }: C! G6 n! i widely believed that 3DES is more cryptographically secure, but RC4 is
0 n, w3 {/ L X; o substantially faster.& x# g, N' V8 @: Q
Note that if AES is supported by both the client and server then this
* J* W1 \% S/ {# z8 W$ U encryption algorithm will only be used to initially transfer keys for AES.' N5 p: [: Q4 T* u( P# i5 {9 W
(See dfs.encrypt.data.transfer.cipher.suites.)
' G' \! @+ s6 c: S; d; T </description>& m8 ` Q. B S8 i
</property>* N3 _* x/ N$ I
<property>. v& p$ p6 q0 M( w
<name>dfs.encrypt.data.transfer.cipher.suites</name>* c9 U9 E! r6 ]: F" ]
<value></value>2 d3 n2 u) L7 o
<description>
( B& Q _/ t, S' n' p5 t6 q! b This value may be either undefined or AES/CTR/NoPadding. If defined, then1 w% @& a) e! M# \
dfs.encrypt.data.transfer uses the specified cipher suite for data/ }7 b3 N$ O7 V& e# B) { r' m$ Z
encryption. If not defined, then only the algorithm specified in" l* j! P& T+ N% ]" r
dfs.encrypt.data.transfer.algorithm is used. By default, the property is
1 D& N9 M; i- N4 f o y& I/ U not defined.' a) | }' a1 ~, s$ P# Y
</description>$ m& y0 k9 w1 r2 v% F# N
</property>
' r- y1 V# A, u( z1 f. E<property>
% h8 e/ ?: G- T6 U& S <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>- o% A$ C. x3 ?
<value>128</value># w$ W- J, P) {% `; l0 m P
<description>: g. M; ^8 R. ]" I3 N: }
The key bitlength negotiated by dfsclient and datanode for encryption.
# E6 T1 [' ]6 w2 z7 {( K This value may be set to either 128, 192 or 256.& n) e# \1 A6 c5 ?/ I" @
</description>+ C0 G% M; h6 k1 _/ z# x
</property>; f$ t% T! A( R, l7 H, M
<property>
$ A- h7 H, W* E$ m <name>dfs.trustedchannel.resolver.class</name>8 ]; d/ P D- [9 [- y% n1 o
<value></value>+ I; N* s3 A% _
<description>
( F- o' N1 I) b$ n TrustedChannelResolver is used to determine whether a channel 1 t% w9 h1 q* B3 e2 Y
is trusted for plain data transfer. The TrustedChannelResolver is" S5 H/ q. p5 g: g1 x( i
invoked on both client and server side. If the resolver indicates ! ], ?) Y3 S* }' ~" b" H
that the channel is trusted, then the data transfer will not be 4 o( @- I+ e* T2 q# P: k1 A
encrypted even if dfs.encrypt.data.transfer is set to true. The
$ g4 F- r3 b) P default implementation returns false indicating that the channel
2 L0 b! Y" v: r9 A is not trusted.
, c* h2 A* D8 a5 \/ K* A </description>
2 r j/ X. n, E! W6 P5 y</property>
7 b5 |) G4 F; J<property>2 Q# q$ e( V9 _) d8 T; N, @! _* B
<name>dfs.data.transfer.protection</name>0 }4 i A: z! C) x3 _* u
<value></value>8 R' o8 C! d9 S n/ Q6 [6 y y
<description>
) |- H# D+ a4 J @" \ A comma-separated list of SASL protection values used for secured
0 b( x! U) ~7 ~1 A' }3 H connections to the DataNode when reading or writing block data. Possible* ~; p& G% R! N3 r- g" [/ m0 ?
values are authentication, integrity and privacy. authentication means
; U+ B. w' H6 p1 Z; w+ l$ ]% a authentication only and no integrity or privacy; integrity implies
: X3 z7 ]* R5 V) n, d* R0 q authentication and integrity are enabled; and privacy implies all of
/ V& \8 U) l. s' y8 ~! |( _ authentication, integrity and privacy are enabled. If
# ?) [9 R% o+ u dfs.encrypt.data.transfer is set to true, then it supersedes the setting for ?0 ~+ ^2 F; W7 U6 r* w+ _
dfs.data.transfer.protection and enforces that all connections must use a* n$ C" i2 L+ a' _
specialized encrypted SASL handshake. This property is ignored for4 g+ S! t/ H, y( p9 N# H
connections to a DataNode listening on a privileged port. In this case, it! n' k1 q ]9 J+ j! l' f
is assumed that the use of a privileged port establishes sufficient trust.9 q9 L& W* x" w: U! L! [
</description>
8 ?: \) {1 h- f</property>& D; Y9 ^$ f# @4 A, Z. ~
<property> u! y+ I& S8 B& t2 j/ o1 S4 S
<name>dfs.data.transfer.saslproperties.resolver.class</name>. g/ C. B4 L) M- t! J
<value></value>+ v! w) ]# K& T3 m. Q8 C! L6 k
<description>
/ L9 ^' S0 w7 O) F* z N! X SaslPropertiesResolver used to resolve the QOP used for a connection to the5 u! d) H& a9 ]; Y2 h; [3 A
DataNode when reading or writing block data. If not specified, the value of
1 U. g I4 y( v1 _7 H4 z2 C8 [ hadoop.security.saslproperties.resolver.class is used as the default value. M: Q2 m, B _% Z
</description>6 v0 N! z7 H$ e" k! X& j& y
</property>& m2 X# M$ S$ B
<property>
* R4 D0 e) p3 d <name>dfs.journalnode.rpc-address</name>; _: v7 e E# x* w3 Y6 r9 T6 G& D
<value>0.0.0.0:8485</value>4 {& }+ v; n- [9 M" W% g& x
<description># q* l1 \4 _. k* f( e
The JournalNode RPC server address and port./ ^( Q1 ]4 G9 M2 D' P8 w+ g, Z
</description>
. ^8 j9 X* E k) | J& }; A5 M</property>
. O: z. D2 k4 T7 c4 G# v$ e<property>
) q. V0 M5 t) {- u <name>dfs.journalnode.rpc-bind-host</name>
/ C' g8 b! f( M. u) C" L& o <value></value>, K+ _* j- d4 p' j+ `8 c! X
<description>/ O. f) Z( h4 \% r) d% o0 z6 s/ C
The actual address the RPC server will bind to. If this optional address is
0 t% [3 ?9 k7 A* R0 h set, it overrides only the hostname portion of dfs.journalnode.rpc-address.
+ d- V! A/ x3 _& a e: P This is useful for making the JournalNode listen on all interfaces by) ]: m3 I5 o0 C0 r" b! M
setting it to 0.0.0.0.& x6 c3 h7 N8 O! ^0 [3 Y2 _
</description>
3 E5 j1 H: I0 \% l</property>
- Z8 y; s- T4 @4 Q& Z<property>
/ V) t% c5 b; ^( ]+ r7 Z( _ <name>dfs.journalnode.http-address</name>) I, e3 R1 Q+ w: \9 {: R; c
<value>0.0.0.0:8480</value>
7 Z; R O8 N& t3 v/ t7 r5 h9 h <description>/ N1 Q5 w" n+ }- i: P2 w& h
The address and port the JournalNode HTTP server listens on.+ A5 I) ?# o9 N! m3 V
If the port is 0 then the server will start on a free port.
2 G; X# c6 L0 [! ]' V </description>- h, f4 ~5 M/ z5 H& s
</property>
# V0 N) n1 I0 c- ~<property>9 z- G$ x, E# `7 T: o; M
<name>dfs.journalnode.http-bind-host</name>
0 R* q: [$ E | K8 j <value></value>9 `) Z9 }3 U, E8 V
<description>9 R2 Y$ y$ K1 k6 O
The actual address the HTTP server will bind to. If this optional address
8 ]/ y4 B! T8 B% }5 ?3 b" \ is set, it overrides only the hostname portion of
& ^; m! v( c. ~ x" C5 ~1 q! X, L dfs.journalnode.http-address. This is useful for making the JournalNode
( ?) [1 Z3 y7 w7 H HTTP server listen on allinterfaces by setting it to 0.0.0.0.
2 t% E0 p- \) R# G8 p' k </description>; `& V7 g: K2 Y3 i! C
</property>
9 j! t9 J; [6 h. \( R<property>/ F' v: j* u, B8 i
<name>dfs.journalnode.https-address</name>
% Y/ a" P% C% `9 B <value>0.0.0.0:8481</value>
+ t8 F* t5 M+ ]; ]3 v( Q V <description>
L Y6 ?, Q/ f6 P- o2 S7 W The address and port the JournalNode HTTPS server listens on.
/ s7 y6 {, M7 Z" k% l If the port is 0 then the server will start on a free port.# I! R! \3 T' ]: \% x- V6 h& @
</description>/ ~/ s! ^5 Z5 X/ M3 G" l+ p$ H
</property>4 ^; E6 ~* p7 h% a$ c
<property>3 w; |. M! G3 ]$ S0 `7 K! V4 C7 e
<name>dfs.journalnode.https-bind-host</name>+ u& i9 ~2 D8 C7 u
<value></value>1 \1 S0 }0 F0 s8 e2 X7 l" F4 Z
<description>$ O( q" T. I$ @9 l: ~
The actual address the HTTP server will bind to. If this optional address" |! e7 F0 Y- u5 q' x. ^
is set, it overrides only the hostname portion of* y! |4 K+ X9 `- B# O
dfs.journalnode.https-address. This is useful for making the JournalNode/ X& @( [3 @1 A! O- l. l
HTTP server listen on all interfaces by setting it to 0.0.0.0.+ t& Y% F* k+ \/ B
</description>% y, l: N0 J) _
</property>
- a: T, N) u+ s4 _+ F<property>
# Q# ]9 @2 e3 Q8 x" _: O7 l9 o <name>dfs.namenode.audit.loggers</name>$ y2 W) a4 k( Y# J) q ] v) e* O
<value>default</value>
2 A% X* w5 _* m$ s, n2 j5 e <description>8 N" ]# p7 I+ s. s7 \' L3 y s
List of classes implementing audit loggers that will receive audit events.
: b! V( A, X9 m5 @. w/ o1 D u These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger.$ T: W2 Z2 D4 U# x% n+ X! L
The special value "default" can be used to reference the default audit, {$ p6 ?5 m/ X6 P! F$ w O) ~
logger, which uses the configured log system. Installing custom audit loggers
" f, Q# q7 B) n# {' j" D- K% [) b may affect the performance and stability of the NameNode. Refer to the custom, e8 R+ ]. e7 Y" t/ X' R1 e8 g7 N
logger's documentation for more details.
+ U8 c$ Z: i) U </description>
# L2 o0 u. m) ^, U9 ^</property>
# \+ J7 N0 z2 b. W# n$ q: P<property>
! l) G: L2 r) b2 }+ U2 P* X7 A <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold</name>2 a& ?# L2 p% e, O; g+ k
<value>10737418240</value> <!-- 10 GB -->
, o, D8 e* ?1 L: J <description>' X8 `( @5 n; Q8 ~. m
Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
( i2 j' B0 H' I org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.
3 g* {% E- ?( n& G4 a6 |. D! N This setting controls how much DN volumes are allowed to differ in terms of
4 H% O; }3 v* Y/ U, J/ d bytes of free disk space before they are considered imbalanced. If the free
; L+ @& n) E9 k( a' D space of all the volumes are within this range of each other, the volumes/ j0 j2 x4 W3 n% p
will be considered balanced and block assignments will be done on a pure
& t* M! D' r$ P6 I2 a# \" s round robin basis.
4 C' b% X. c) k </description>
: a. [1 a4 Y0 `, d</property>
; X2 y# P7 s6 j<property>
) B6 F1 [+ T* L5 } <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name>: W3 O# M8 w; A) [
<value>0.75f</value>
7 H8 Z4 v' C+ \5 g2 d7 i <description>2 u6 M* M# |% }, i% Y- T
Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
8 c6 ~) _4 D' [" q org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.* C( v- a6 j N ^) a, ~- ]
This setting controls what percentage of new block allocations will be sent
9 J" t) ]4 l) S9 @9 h to volumes with more available disk space than others. This setting should7 x1 E2 R; ^9 x" a8 P9 o4 B
be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should
4 K/ Q3 R$ Z O+ y# ~ be no reason to prefer that volumes with less available disk space receive1 K' y( _2 o- o5 c$ w; O
more block allocations.
j+ e" C. t1 f6 H/ z </description>
! |, c4 `$ _. K% M: K</property>
* V% d6 j+ o' ` V<property>6 e" g* ]: q. c r' i- ]# f
<name>dfs.namenode.edits.noeditlogchannelflush</name># E' z5 R8 C2 _& Q) e7 r
<value>false</value># k" c; X+ A/ B
<description>
' X! C7 }7 b' n" A8 {* I Specifies whether to flush edit log file channel. When set, expensive
- E% X H6 W, g) O. m9 E" Z# z FileChannel#force calls are skipped and synchronous disk writes are
% S) q& t# w8 m: X d4 u- q enabled instead by opening the edit log file with RandomAccessFile("rws")
2 |6 U* A" j' O6 u' Y flags. This can significantly improve the performance of edit log writes
6 a$ B' g0 V) X- q' l on the Windows platform.
+ b( z$ L2 x. D1 s0 t2 `0 W- q Note that the behavior of the "rws" flags is platform and hardware specific
( c8 m$ J' ] v5 F and might not provide the same level of guarantees as FileChannel#force.
' @7 I$ G7 }4 x4 N; r For example, the write will skip the disk-cache on SAS and SCSI devices
+ q6 B7 G F: [" V# d3 Y while it might not on SATA devices. This is an expert level setting,7 s* m& T5 H9 E5 |& W/ k
change with caution.2 l! m. F9 l7 W( ?: N( l8 m
</description>
& g' O5 h h* L" Y4 J) ]</property>
( z, }3 w2 ^ ~6 O0 w. v0 n4 _<property>
+ {! y5 T) L9 A/ o1 U, ? <name>dfs.client.cache.drop.behind.writes</name>* L$ {7 U2 A/ Y y5 H/ |& y( u
<value></value>2 ]9 P5 F5 v8 g- Z* ?& B
<description>
* O5 d) T/ o1 ]8 p/ K- n Just like dfs.datanode.drop.cache.behind.writes, this setting causes the2 U- ^; B4 P+ d( Y9 W
page cache to be dropped behind HDFS writes, potentially freeing up more; e2 D v: c! g* o0 l! C
memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this7 c/ q6 W2 @1 j
is a client-side setting rather than a setting for the entire datanode.% e& f( c& B3 G4 w% U7 j
If present, this setting will override the DataNode default.
) q6 w) ]0 n: ]- J' \ If the native libraries are not available to the DataNode, this% q# |* Y, v4 g Q
configuration has no effect.! W9 F. h# \# x$ l. R' b1 o
</description>, A2 G% K9 b/ T9 j& `
</property>
# Q# |' k' n( o% @6 d<property> |$ F0 m3 s( @6 U( k7 B: _$ ?
<name>dfs.client.cache.drop.behind.reads</name>
`, k$ N% Y& T& \ <value></value>2 o% V, q' X: [
<description>& o& C0 _7 E" H( h
Just like dfs.datanode.drop.cache.behind.reads, this setting causes the
8 ]1 ]" ` Q4 M f4 P page cache to be dropped behind HDFS reads, potentially freeing up more
3 y1 k1 F; g! [% e memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this. m) O: C& D. i' Y) v
is a client-side setting rather than a setting for the entire datanode. If
8 J4 I6 g% h! Q0 L present, this setting will override the DataNode default.
' z' o n, ?) Y9 `: ] If the native libraries are not available to the DataNode, this) \9 X8 [3 _! J. j
configuration has no effect.
8 w' q% a X( n8 `" O </description>
" _- L/ ?" m/ p% d6 _4 i</property>" Y0 }3 W( b0 |0 h% N0 y
<property>
1 t8 l1 E1 ~( _( m/ C# h+ j <name>dfs.client.cache.readahead</name>
# Z$ s$ k+ L. n8 z4 P- a6 j <value></value>; }: Q$ |$ g0 L% M
<description>
; n0 }1 G8 \: c- C) n When using remote reads, this setting causes the datanode to ~0 s9 @+ B8 J) q$ ]1 {) H
read ahead in the block file using posix_fadvise, potentially decreasing0 f, B- j5 _$ O. I( U M' M( b. k
I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side
; f: I5 v/ {# H% \) ~3 \* c: x setting rather than a setting for the entire datanode. If present, this- ^8 Y8 ~$ `" K
setting will override the DataNode default.
0 x2 j; b5 o3 |8 v" E/ m$ I When using local reads, this setting determines how much readahead we do in/ o3 f# W' N+ O4 d
BlockReaderLocal.$ j+ A' _0 S4 @6 p/ M% t- \
If the native libraries are not available to the DataNode, this
$ H0 G& ?) J2 ]: N7 } configuration has no effect.2 c) g! P% A0 K1 J3 c% C. n1 o# K
</description>
, ?: O) P2 D' N4 v1 g5 w</property>
+ j2 @9 Y( T/ Y/ i4 S2 w<property>) @) J% G ]! [4 B9 G3 O
<name>dfs.client.server-defaults.validity.period.ms</name>
0 h/ w, I* V+ ~4 L- y- G4 Y/ x <value>3600000</value>
3 ~" @1 c& i9 t9 c <description>. h0 x$ f5 @0 K, ?
The amount of milliseconds after which cached server defaults are updated.
0 B3 |: D: K/ J; Q5 d' X. M5 f By default this parameter is set to 1 hour.
# Z* x, n" | J6 T </description>" w9 G5 [) {( j" E
</property>! X% W% c# {4 |3 u+ n% _5 R
<property>0 |: p, B( E8 o* y) @) h+ U
<name>dfs.namenode.enable.retrycache</name>/ A3 o& K2 d- n3 O% M' d4 x
<value>true</value>% W" C. V. j) v2 M6 W
<description>
) g) a) C7 t k; [, b This enables the retry cache on the namenode. Namenode tracks for' B# n) c6 |6 ?! j/ y; x
non-idempotent requests the corresponding response. If a client retries the
+ x, J6 G$ l* c1 `. I request, the response from the retry cache is sent. Such operations; ]3 N3 g5 Q+ Z, N6 @
are tagged with annotation @AtMostOnce in namenode protocols. It is4 c# _. y2 t. n/ W
recommended that this flag be set to true. Setting it to false, will result
& [4 V- |/ R) B- Y8 ?) g, \$ K& M in clients getting failure responses to retried request. This flag must
% `$ ?' `5 I3 P% m+ h) I be enabled in HA setup for transparent fail-overs.3 d4 f4 N) e5 Y1 z; H G" T
The entries in the cache have expiration time configurable$ v: f$ E6 V; A" G
using dfs.namenode.retrycache.expirytime.millis.
7 u% e; {4 _; U </description>+ Y. _# k, w. O$ I* H6 F
</property>
2 X7 r! W$ Z2 @% h5 O! ?) p+ h1 s<property>
2 d- v6 T% K3 |; P: j <name>dfs.namenode.retrycache.expirytime.millis</name>+ D$ M% o* m1 l$ `0 `& \
<value>600000</value>; Z+ e, `# t# Y6 i: `
<description>/ X, E9 S6 v4 j# d, e! I3 @5 x
The time for which retry cache entries are retained.
! e* ?0 w P6 p) U& r </description>
0 o, A0 I4 |! K6 M9 [& Q1 N B: M</property>9 V' D/ ~2 G% w) D
<property>9 s6 T2 A5 {7 R, ~
<name>dfs.namenode.retrycache.heap.percent</name>, z# s! F2 j- }, V5 t3 I! J: h
<value>0.03f</value>$ j2 r4 z. K" P- ^& h/ Q6 @6 s
<description>
e7 o1 A" g& K, {4 l0 m b This parameter configures the heap size allocated for retry cache& Q/ G# b& r; D
(excluding the response cached). This corresponds to approximately1 i; s% j6 O, e3 W Z3 }: `
4096 entries for every 64MB of namenode process java heap size.1 {2 r x% @- U6 t o( {% t$ w
Assuming retry cache entry expiration time (configured using
; F3 O: X: r1 F dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this
0 I9 l% F2 z& i4 `8 `" B8 O enables retry cache to support 7 operations per second sustained
1 y( @' d6 J1 ?* P for 10 minutes. As the heap size is increased, the operation rate
/ u/ B9 S, f6 {$ R6 W' M linearly increases.* D$ ~! b' e, P% e! N( N
</description>0 z0 _8 ` O i7 ?2 @
</property>+ `* j" E0 U. L; Z5 L
<property>
) @* c% ?3 c! ` <name>dfs.client.mmap.enabled</name>! m0 | P' |- Y' @' ^. N
<value>true</value>
( O: }( k! ^! Z# D( c/ o <description>7 r$ V+ m, V3 @9 g! ?& {" o7 J _
If this is set to false, the client won't attempt to perform memory-mapped reads.4 {2 [3 b5 s& O' N1 z* y9 z& E
</description>$ i+ l' U& ~/ c8 r# J
</property>7 `, U- I% k7 o
<property>
m5 n& U$ F- x ? <name>dfs.client.mmap.cache.size</name>' q r! x3 g3 N
<value>256</value>
% c4 Y2 s6 W$ _( \6 E1 d <description>
" n9 N8 R. w" n4 I; _7 e R When zero-copy reads are used, the DFSClient keeps a cache of recently used# c" N1 v* v6 S6 `: Z$ h
memory mapped regions. This parameter controls the maximum number of1 t# r' E! K8 Z* I& c
entries that we will keep in that cache.
) ^* R9 P4 g2 @- u5 X The larger this number is, the more file descriptors we will potentially, u6 z8 n/ m6 Z' a& L
use for memory-mapped files. mmaped files also use virtual address space.. F+ B2 F) m3 \# j+ q# {
You may need to increase your ulimit virtual address space limits before
8 t/ v! V# J& N( A increasing the client mmap cache size.
. R5 D7 P5 {5 X& P0 A( Z. ] Note that you can still do zero-copy reads when this size is set to 0.- I l m' a7 @7 S# w& T2 |& }
</description>
/ E+ H# g$ y I' [4 g2 r, Q1 @</property>) c& j% y0 N U! C2 b: `
<property>
! `4 p& t: Q; c7 Z3 D. g+ k0 e <name>dfs.client.mmap.cache.timeout.ms</name>
# a" H. O5 m; ` m9 C <value>3600000</value>
& h; l+ C- X% Q, l8 d d <description>
5 V" V/ r1 g( d! n! G The minimum length of time that we will keep an mmap entry in the cache c: G3 \* d) _, e$ V
between uses. If an entry is in the cache longer than this, and nobody `6 L* e: h1 X2 y
uses it, it will be removed by a background thread.' @' s3 Z2 H k9 F: \& u9 J
</description>
+ u! x+ c6 E: Q }0 [1 U- O" Y! d8 _</property>1 B! ^+ E9 a" O% T6 C- E/ B! T
<property>
% ^1 A+ l: v; j3 t <name>dfs.client.mmap.retry.timeout.ms</name>
+ n1 P* t0 w# K0 L <value>300000</value>! m7 _1 a7 c' e) z; A- ]5 g( ~) h
<description>/ ^* \9 h2 G% u, C8 r3 u
The minimum amount of time that we will wait before retrying a failed mmap
1 l7 e: J* a+ V) Q' S2 k operation.. b0 f, \+ ?9 w- h
</description>" }9 J7 p$ r$ z' L: i5 ~
</property>
- S8 Z8 N G0 N& X<property>
& Y; y& f) A3 H <name>dfs.client.short.circuit.replica.stale.threshold.ms</name># u8 v3 G+ J! R- {# S2 F, { k
<value>1800000</value>% c W6 g# @; Q. N$ y4 \/ N' s
<description>
- U; P8 m/ u6 @3 h: B" v The maximum amount of time that we will consider a short-circuit replica to! {+ D) \* b% O, L$ w
be valid, if there is no communication from the DataNode. After this time
, H; S) m5 I8 m has elapsed, we will re-fetch the short-circuit replica even if it is in
. [9 h) N/ _& b; r2 }: { the cache.& c7 j- M" R- D. G
</description>
3 n( x: v$ ?$ U, |2 Z1 t' _</property>
) @& ]$ d2 t8 ^! V0 W; J6 w<property>
2 u( e$ @1 s. F) I6 X: X <name>dfs.namenode.path.based.cache.block.map.allocation.percent</name>, x; R& s b2 s- J' D5 \: C
<value>0.25</value>
3 Q. i$ \0 n9 o' |+ Y <description>
3 s5 p9 [$ m2 \: x) v9 N5 h1 O* k The percentage of the Java heap which we will allocate to the cached blocks
$ c+ H) r) z* k! L map. The cached blocks map is a hash map which uses chained hashing.
# S, G" \% `1 z; s' X, e% L/ ` Smaller maps may be accessed more slowly if the number of cached blocks is
$ e/ l- Y0 Q5 g" ~; V3 n$ A large; larger maps will consume more memory.* O* K' D5 T8 ?5 Q
</description>
& {' Q% L5 x- j. m3 r</property>
1 o& O/ @# i7 |9 ?1 l+ ?<property>1 i" q, J: E& Q" y1 V
<name>dfs.datanode.max.locked.memory</name>. @3 P# k1 H8 D- p
<value>0</value>- f4 ?! Q) o# q1 o3 P- x2 T
<description>
% Y+ A2 P! \' Q! }5 N5 | The amount of memory in bytes to use for caching of block replicas in7 o$ r3 F% A$ \5 a
memory on the datanode. The datanode's maximum locked memory soft ulimit
% J) [2 a% g! I+ o/ S* u8 m (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode
2 i" r2 v( a$ \! T will abort on startup.
% n- v. r4 ~9 {7 ^ By default, this parameter is set to 0, which disables in-memory caching.- ^# C4 W% z. k- z& H4 R0 L
If the native libraries are not available to the DataNode, this
: C# d6 z( g+ S configuration has no effect.
9 L% }& x8 N: Z) x6 Z: D) B </description>+ z3 y& D4 w R+ O3 c
</property>4 a8 f/ w3 Z% ^* u8 \+ y. E% }
<property>$ X" V" J4 ~: L N) L1 Z
<name>dfs.namenode.list.cache.directives.num.responses</name>
0 _7 ?. Z$ S) b <value>100</value>
% m0 \. L' i, X: _2 O <description>8 F5 D( m. C) ]- Z6 _' I9 l
This value controls the number of cache directives that the NameNode will
7 B% y: E x+ {. J3 @ send over the wire in response to a listDirectives RPC.
$ b2 ^8 C( W( J/ d </description>( }! G3 x, Q8 _- H: c( R! l. k
</property>
: ?0 b% I0 m1 ~ O8 v7 q% O1 F* b<property>
6 o7 t v9 f) c3 g <name>dfs.namenode.list.cache.pools.num.responses</name>& H$ @7 \& S8 K8 g( R/ k- h
<value>100</value>9 [ t. ?+ o! i# A% J, V6 y! o
<description>. H' ]) s4 f' g5 v6 o
This value controls the number of cache pools that the NameNode will
1 T$ {1 ?3 S, E4 r* G M. ~. B6 h send over the wire in response to a listPools RPC.
, `& O: O- v* |( |* W </description>: v% J8 r. S9 P+ t0 O
</property>3 S" X) Z' ^/ g$ G! C' {% Y8 X( M
<property>+ ^: A x1 [' x" T S; Q4 O
<name>dfs.namenode.path.based.cache.refresh.interval.ms</name>
' V9 ^1 q, c$ b <value>30000</value>0 L# {8 v6 G$ Y2 Q8 ^
<description>
0 \; A( |7 `+ ~. }" [" I- I, p The amount of milliseconds between subsequent path cache rescans. Path
2 V. g2 O. Y7 u: F; } cache rescans are when we calculate which blocks should be cached, and on
/ N9 w; T/ K# _4 x( Q# |7 j what datanodes.
, i4 v5 T8 f k0 X( ?! h By default, this parameter is set to 30 seconds.( c. ^' I& s$ K# t
</description>8 A# x* \ E5 x% ?) S: H
</property>& u. B" V$ k3 m6 ~2 s& _9 _7 k
<property>
\! o, q' b1 G3 Q: `+ f1 `6 ?5 k; T <name>dfs.namenode.path.based.cache.retry.interval.ms</name>: O( d* d% J5 Q6 W
<value>30000</value>
- T$ N/ d8 t( O% W9 H9 |/ X <description> u9 G* f* v3 ~+ z4 m; b
When the NameNode needs to uncache something that is cached, or cache4 ^& O: [* I3 O3 O6 N' y
something that is not cached, it must direct the DataNodes to do so by
3 K) W& r! F% e sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode
* v* e5 P \# n. C0 W heartbeat. This parameter controls how frequently the NameNode will% [* c; s9 |- Q
resend these commands.
& k7 V ? F) t0 C! O( Q7 P, | </description>& h& Y& Y/ C: Q& E7 g8 }3 h, L
</property>1 o, I" Q% Q; G& {
<property>3 X m1 {- w. H) a
<name>dfs.datanode.fsdatasetcache.max.threads.per.volume</name>
) @1 k0 j$ J7 F% C; |" p <value>4</value>
3 I) G4 _, U! @. r% m( s <description>9 L/ |8 |; W S1 W e
The maximum number of threads per volume to use for caching new data& Q8 o/ u1 q; M
on the datanode. These threads consume both I/O and CPU. This can affect
# N! J+ K- S$ V: Q normal datanode operations.$ n* Y3 v a3 O4 }1 ?+ n: {- h
</description> _7 H% K: S$ u; k ]9 A$ q0 n! i1 x
</property>* K" C! B) @) }, Z
<property>+ b' S: C& R& X5 P
<name>dfs.cachereport.intervalMsec</name>0 z( ]0 @& ?1 K. ]
<value>10000</value>; q% Y" R# E+ _4 F+ N
<description>
1 Q' i+ S5 n9 Z! N& ^ Determines cache reporting interval in milliseconds. After this amount of. G" b6 Z Z- ]1 c" u9 O
time, the DataNode sends a full report of its cache state to the NameNode.0 k0 @: h5 Q& @2 L# ~8 d
The NameNode uses the cache report to update its map of cached blocks to/ w) I4 K5 `' o$ q! r; ]
DataNode locations.7 ~2 F: Q% m2 Y) L/ ^( l
This configuration has no effect if in-memory caching has been disabled by% h- J8 B7 |8 M f# j$ V
setting dfs.datanode.max.locked.memory to 0 (which is the default).
9 h2 p( z* \$ x" o! W. q If the native libraries are not available to the DataNode, this
- A/ Z# d# \+ f% s4 n( L% _6 ^+ d2 o configuration has no effect.
3 j6 U! e! u1 j0 q" q8 b+ H </description>
$ }; J4 [9 x# h6 e- n- ^& J) S</property>
" a) P& a& W, h& \<property>) r* ?) h6 M. i; j t
<name>dfs.namenode.edit.log.autoroll.multiplier.threshold</name>
: B. w) x9 j, K+ P/ T8 R9 j <value>2.0</value> m3 }, z9 K* r0 h
<description>7 @/ ^( M: x" K5 K: ?
Determines when an active namenode will roll its own edit log.- ~- I) R2 \0 a$ o
The actual threshold (in number of edits) is determined by multiplying
8 E" h I2 N5 k% O) N' r0 l this value by dfs.namenode.checkpoint.txns.
5 L& u( u6 ?- P9 I. h1 d This prevents extremely large edit files from accumulating on the active {) n' `9 s- M- d& q
namenode, which can cause timeouts during namenode startup and pose an
$ l) O6 I* q% v1 z" x5 n administrative hassle. This behavior is intended as a failsafe for when! d% V2 l& Z+ Q8 ~
the standby or secondary namenode fail to roll the edit log by the normal
* P8 _9 V% c0 T6 u0 _& @ checkpoint threshold./ _" }; Z) C4 L8 U: _% X% w5 I' S2 K
</description>+ t+ ?: H$ ~. M0 v( @
</property>' d" s6 p# D, z ~% b4 g
<property>
% x" s$ U' ?2 e5 v3 m <name>dfs.namenode.edit.log.autoroll.check.interval.ms</name>( X# T, l1 o7 v/ O# t( J
<value>300000</value>/ W+ a# z* X! }7 D1 P
<description>1 e6 _1 |% U, a1 A
How often an active namenode will check if it needs to roll its edit log,- {3 d8 g* O, q& h6 D9 K' I% f P$ c
in milliseconds.1 E% f8 `* o! E
</description>
" k+ d: U& x! j) _9 h/ U</property>
7 @: v% K2 x5 ?' Q; q<property>
4 ]+ s; k8 \! ]& y* d7 S <name>dfs.webhdfs.user.provider.user.pattern</name>
6 {3 I; W4 i: h3 a! }( R* @ <value>^[A-Za-z_][A-Za-z0-9._-]*[$]?$</value>. K1 Y( E( z, ^: M/ q6 k7 l" r
<description>0 n# j6 F; P) j; j8 P
Valid pattern for user and group names for webhdfs, it must be a valid java regex.; f$ @6 n7 M# m
</description>" g, l P' W! T1 A
</property>
. c! N! w" d8 K<property>/ d9 T0 U& ^) U! t, k0 z0 h+ k. E/ ^
<name>dfs.webhdfs.acl.provider.permission.pattern</name>
. F& w7 b+ l+ w6 @/ | <value>^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$</value>
# H" B1 Z$ G) |0 k% \, G <description>. Z4 ` l4 n7 C& {, r
Valid pattern for user and group names in webhdfs acl operations, it must be a valid java regex.
1 Z% v- k/ e: M% @, d; M1 i </description>4 J1 @6 ]2 `; h( ?4 J
</property>4 v0 M+ i6 @6 x w( E
<property>& V; Q- c7 D( P7 }' z' r
<name>dfs.webhdfs.socket.connect-timeout</name>% |5 M; @9 w9 |* q, @( o5 `) _2 p
<value>60s</value>8 l) ~( U# m' y1 o; I. v, v
<description>
" G8 ?$ x2 h/ e Socket timeout for connecting to WebHDFS servers. This prevents a
1 L+ C( S, j4 n0 V1 U WebHDFS client from hanging if the server hostname is3 ?* A5 z: X4 f' J7 V1 O
misconfigured, or the server does not response before the timeout
; Q8 m/ j' a+ H1 b1 l% J; u expires. Value is followed by a unit specifier: ns, us, ms, s, m,7 X: W, E* s% }5 N1 q5 R) u5 N
h, d for nanoseconds, microseconds, milliseconds, seconds,
1 y; N: _9 A# ~" `* n, ~* Y) ^ minutes, hours, days respectively. Values should provide units," c5 c7 U X) b( |# w8 q0 ~+ z( U
but milliseconds are assumed.$ `# U8 h! y) R8 I0 L1 {
</description>; v. A- y# G6 G4 x9 {8 i
</property>1 i2 B0 G& D6 k8 i- O, M
<property>
: D/ G" |" _& Z <name>dfs.webhdfs.socket.read-timeout</name>3 I! k1 u( p- u3 }5 ]" \/ K: p3 W
<value>60s</value>& W9 |# u( g. D, {- F% f2 I
<description> ~0 f X+ o1 I9 T$ W# z9 t; q
Socket timeout for reading data from WebHDFS servers. This
# x# I4 V" ~$ N8 L1 g prevents a WebHDFS client from hanging if the server stops sending
# [! I; w" `5 i* X) [# E data. Value is followed by a unit specifier: ns, us, ms, s, m, h,0 l G2 P" ^3 P& C5 J- M5 W8 f" D! W
d for nanoseconds, microseconds, milliseconds, seconds, minutes,( G' z; A: Z; o; ^2 }- }6 T
hours, days respectively. Values should provide units,! h7 a+ |. b0 ] c, N
but milliseconds are assumed.2 Z5 I7 p' T' c/ e8 A
</description>
- n0 b, r: Q3 [- [( m8 W8 n</property> u* s- o9 O, o: u9 X+ W" `$ ~ z
<property>
4 f3 z0 t i2 u8 [ <name>dfs.client.context</name>
" q: u; {4 T* L7 } <value>default</value>
2 t- \" R* w' z6 V9 q <description>
* P }1 p: ]$ d$ M/ `# } The name of the DFSClient context that we should use. Clients that share
% o; x& V1 G: x* F. Z- l- C a context share a socket cache and short-circuit cache, among other things.
# z1 ^1 q- n: a- G+ h0 W/ r You should only change this if you don't want to share with another set of+ W! |% U2 M9 k2 F
threads.( l3 M" N' j! n
</description>$ u5 a6 W& i" l$ f6 q
</property>! n8 A( D5 n# F6 i
<property>
# q3 p- [5 `/ `( b8 r Q <name>dfs.client.read.shortcircuit</name>1 i0 @; \/ A" \/ e; k
<value>false</value>
! P( q/ f" g) t, k% m <description>' t* a' M3 y7 u( q& ?
This configuration parameter turns on short-circuit local reads.# F ?- y: Q" \* _& n. N+ K
</description>5 e; C1 g: y9 {( ]6 P: {4 m) h! V
</property>. L8 W' w0 n/ o0 [
<property>! u8 a+ `* |. U' `6 l
<name>dfs.client.socket.send.buffer.size</name>
9 ^7 i7 l Z( i9 U; u8 i <value>0</value>
2 h. {6 e2 P% z% L5 ~9 l) P0 ] <description>
$ ~' G. l7 ^9 e4 y Socket send buffer size for a write pipeline in DFSClient side.' ?4 \1 m6 s1 k$ N' K
This may affect TCP connection throughput.9 C7 i5 ?9 v7 V8 _& B
If it is set to zero or negative value,8 i% j* s. p" a! G* s9 K6 Y2 t
no buffer size will be set explicitly,
( @. Z4 @2 o6 g1 L thus enable tcp auto-tuning on some system.
4 M& m$ D, n$ R% G The default value is 0.
2 P& P+ P6 Z3 b+ K1 _7 l% j </description>( \0 A; T2 `2 T3 x3 Z
</property>2 x0 T% |) g7 J
<property>$ Q4 \! U( m+ u1 ] o2 b
<name>dfs.domain.socket.path</name>) F! Z+ w4 W8 C# g' y5 y5 b
<value></value>
$ s1 P- n& ]" x' |7 f <description>
2 g) M- O& u& O3 A& _% J Optional. This is a path to a UNIX domain socket that will be used for
3 b% K& ?& h6 ]' h communication between the DataNode and local HDFS clients.
1 y4 {, Y4 w6 T! J If the string "_PORT" is present in this path, it will be replaced by the
+ C8 V7 v' }% Q1 s TCP port of the DataNode.
( Y: V2 X% Z( |: i7 q, [' J2 c </description>
) P- T% g# Y7 a9 X" g W</property>
" `, T! g9 i1 f) r<property>4 `- }& z( P% ~5 k$ ]
<name>dfs.domain.socket.disable.interval.seconds</name>- g" ^. b$ `* @, }' h% B. J8 h$ E
<value>600</value>/ q3 w ~3 W& X$ L
<description>
8 U/ i) @3 g: }. y( X9 e The interval that a DataNode is disabled for future Short-Circuit Reads,4 f* k. X, }. @" ]8 @( k3 @
after an error happens during a Short-Circuit Read. Setting this to 0 will
: c8 O& t4 C$ W/ D9 w* ^$ Q3 W not disable Short-Circuit Reads at all after errors happen. Negative values! U9 f7 }2 l+ X. f
are invalid.
+ |; h+ I0 V: R9 X9 { </description>6 O6 j2 R3 N9 I( h7 Y
</property>
2 E3 E }/ J8 H6 d<property>) h4 j) M1 a. \# Z. y
<name>dfs.client.read.shortcircuit.skip.checksum</name>7 X$ K! S- j7 Q% A- |
<value>false</value>
/ [+ I. X9 l" v5 O% L0 K$ m <description>
9 ?) w5 n8 f* y6 u( y! J If this configuration parameter is set,
! F3 @& I8 h$ g short-circuit local reads will skip checksums." @, M2 S. h( v; w) p2 x
This is normally not recommended,- k/ K9 ^ T, H) x/ W$ Y
but it may be useful for special setups.
% P" e' l S7 f4 X! a ` You might consider using this; E. g* c3 Y; Y$ u+ n
if you are doing your own checksumming outside of HDFS.
; ~" K+ R- F( t2 }! V$ w </description>
' j! T& U/ r& W4 y</property>2 B6 e- ^5 O% j2 |+ Q
<property>5 Q' J( J8 Q5 {% ?3 V
<name>dfs.client.read.shortcircuit.streams.cache.size</name>3 N8 B8 A# v e8 O9 K9 x# Q
<value>256</value>' R S, l- j, a' P0 Q5 `* L& @: ]! F
<description>
# k' N% W6 T1 H) l# Q The DFSClient maintains a cache of recently opened file descriptors.
) E) ]$ B, c" ] G+ b% A& V This parameter controls the maximum number of file descriptors in the cache.9 f& [1 U& G* r4 r1 h. @
Setting this higher will use more file descriptors,0 C n R/ m$ c1 [9 f
but potentially provide better performance on workloads$ y9 t" L# y& n
involving lots of seeks.
3 o k- d6 o# P7 E" K </description># y& k# ]" k( x; M# n- u0 _/ h6 m
</property>7 V1 ^" I* L' z. Z) x* {
<property># r4 P# |+ O% E) P
<name>dfs.client.read.shortcircuit.streams.cache.expiry.ms</name>
( t7 Y- V0 C, n0 L4 y- _ <value>300000</value>
$ _, r/ ]8 p4 K <description>
% q) F1 z( p9 ? This controls the minimum amount of time# i- M& L& r3 @8 M
file descriptors need to sit in the client cache context7 M4 E; r! ^4 {: E8 Z
before they can be closed for being inactive for too long.1 b1 f0 u+ P1 y0 f8 W, t
</description>1 K& e) `5 b6 U
</property>3 w/ H" c: j/ P
<property>
9 v- e, i1 p( m' `5 o <name>dfs.datanode.shared.file.descriptor.paths</name>3 _3 g8 o% p6 k+ ] ~1 X( r
<value>/dev/shm,/tmp</value>
+ Z5 \+ z' I# i* d8 k; S* y <description>
" _- }, u% G' s2 q Comma separated paths to the directory on which
2 Z( ^7 R. b, f' m- G shared memory segments are created.. y6 e& S' T4 p# B; B$ k
The client and the DataNode exchange information via
+ J; X0 o4 A! K: [ this shared memory segment.4 e" h/ v( H0 u5 K8 ^$ S( [
It tries paths in order until creation of shared memory segment succeeds.7 m7 c E4 J8 i! j
</description>' M: i) W" v+ x8 V8 T; C7 b& ?
</property>, W( k% `% N/ ~' s
<property>
* H0 Q% J( o+ [# ^- a <name>dfs.namenode.audit.log.debug.cmdlist</name>
' N" m7 |. F {- j5 J4 [$ |8 Q, R( e C <value></value>
" }8 O$ T1 H$ ` R0 a <description>
$ @; h E, B0 I5 E6 z% k A comma separated list of NameNode commands that are written to the HDFS! x" D/ c e( P% A
namenode audit log only if the audit log level is debug.
6 h0 \3 ^0 H" J' k8 ` </description>( r3 |% q4 x2 @* d. L. q5 R
</property>
3 l( k, ?- p& Y4 D; @<property>
4 c" ~4 } G% D& }0 t <name>dfs.client.use.legacy.blockreader.local</name>: T7 w ?2 b# `* }, ]2 L
<value>false</value>
2 C i/ r4 S5 _8 e5 O, {, b <description>
' C |. j! R; y M3 y$ j; C Legacy short-circuit reader implementation based on HDFS-2246 is used
4 u8 s8 r' O3 h7 u; M if this configuration parameter is true.+ a* w T# D% A5 [. o
This is for the platforms other than Linux
. y* [2 A8 B0 |$ R where the new implementation based on HDFS-347 is not available.' z7 H& K- N- G% {, A4 q, V% k
</description>
$ b! N6 w2 D. y+ {; `3 }</property>; G7 i3 {+ ^+ S+ c# z" _
<property>
* h$ {8 A6 F. d& E4 Z. E' Z <name>dfs.block.local-path-access.user</name>- y, R4 o7 I. }/ C& p; R7 s
<value></value>$ t# U; ]3 {- \! x2 F1 t- F
<description>
( h8 @5 i% q% O Comma separated list of the users allowed to open block files0 O6 P# f2 r9 b: ~7 ^6 c& v
on legacy short-circuit local read.
, F5 b& e E; L/ K1 l$ R( y6 j </description>/ I* {. k# t- P7 W: V) o
</property>7 c; g& M( {8 B2 ^4 w. p9 z4 ?
<property>
# ^. I+ d- F! c: E/ E$ h <name>dfs.client.domain.socket.data.traffic</name>
7 B0 y/ J- q- d7 Y& k+ p2 {7 Q <value>false</value>9 K* z* o! d% J0 [* ~
<description>1 T2 e! k! F. ~' b# b
This control whether we will try to pass normal data traffic! S7 V& j0 H2 @; y
over UNIX domain socket rather than over TCP socket* K) u" g4 H$ x) R$ G( Z( s9 T
on node-local data transfer.
* m/ P0 q8 d. k2 X7 X* J2 X This is currently experimental and turned off by default.
) P' `# T6 o( i0 @ </description>
* s6 n* ~8 w5 `& W: @</property>( S; x- Z0 w8 Y* O! e; B
<property>
; S k( D w3 ]! V- D1 b. ` <name>dfs.namenode.reject-unresolved-dn-topology-mapping</name>' J5 \( P3 s5 _% D
<value>false</value>6 R( V+ f7 d% ?9 f
<description># ` s9 T( r1 v$ M
If the value is set to true, then namenode will reject datanode 3 ~' T% y3 e& P/ m* ]
registration if the topology mapping for a datanode is not resolved and
* F( |0 A, S$ t, I1 M NULL is returned (script defined by net.topology.script.file.name fails 8 r0 B/ c; l T* k. B- K/ @
to execute). Otherwise, datanode will be registered and the default rack 6 _7 t8 k6 }/ l2 ]3 b3 `
will be assigned as the topology path. Topology paths are important for 5 S0 g* P; b! A
data resiliency, since they define fault domains. Thus it may be unwanted & r1 k# K6 [ r
behavior to allow datanode registration with the default rack if the
3 I# G: ]6 J$ x6 J0 Q. n9 j resolving topology failed.
1 R8 J8 p' }" q3 ]2 R' W; f </description>
8 q/ P0 w0 U$ ]4 s3 Z</property>8 S0 K a: w& ~8 d r/ T
<property>
1 V' K6 W( F: U' R5 o ? <name>dfs.namenode.xattrs.enabled</name>
+ |7 a; r4 J3 h: K, t0 }/ x <value>true</value>
0 w$ X: v, {2 @. @4 v) L <description>
0 N2 X& O3 N# ` F! f Whether support for extended attributes is enabled on the NameNode.
3 l4 t5 _+ {! ]: A+ G </description>
* [+ u, o7 s) n, y$ t9 z% t$ a</property>
- `) J4 K7 Z6 v' O8 b2 r1 Z- g<property>+ b5 m! n: I: Q- x
<name>dfs.namenode.fs-limits.max-xattrs-per-inode</name>
0 w/ [% w. [4 o9 S }0 E% `0 V <value>32</value>0 ~6 s0 L/ ^- L$ U
<description>
3 I- l- }7 B4 O, s9 w* g Maximum number of extended attributes per inode. ~ l- l' Z( c, h# a* q
</description>0 g! `0 k5 X; J7 B: U. b2 ^
</property>
" y4 Q4 }3 a$ q6 J: a+ k$ T# Q- d<property>
" S) @& J2 H1 Z: M <name>dfs.namenode.fs-limits.max-xattr-size</name>
- c* [1 d$ H1 `/ _5 v. f$ K0 a) Y <value>16384</value>
+ f X$ P/ Y) Z6 I- G <description>
0 Y6 \' a6 ?5 u; W& M The maximum combined size of the name and value of an extended attribute2 _/ s5 z# P! J" I
in bytes. It should be larger than 0, and less than or equal to maximum3 b& o. V" ^, @" e, v) a
size hard limit which is 32768.
* z9 Q+ x* I) s9 ^8 T8 r8 b" Q$ X </description>$ t3 `( l. Z* i& @& N
</property>! N5 R$ B& k4 W
<property>
# q! `: e! [) m! i2 Q5 u <name>dfs.client.slow.io.warning.threshold.ms</name> [' s$ Y- U# l1 g& G' E! t
<value>30000</value>4 u/ K H- z0 v$ U
<description>The threshold in milliseconds at which we will log a slow
5 c$ s! K6 P) p# f io warning in a dfsclient. By default, this parameter is set to 30000
4 {$ U- z1 S& I- D \+ \) ?( @ milliseconds (30 seconds).- p7 ?" Q; s& q& g- x
</description>, y G1 d7 p3 K- s. @ W, {# K
</property>
5 N& Y& }5 T- z; v' |0 s1 \<property>
: s+ u! B' }- s8 w* p <name>dfs.datanode.slow.io.warning.threshold.ms</name>' |0 H8 |' w0 G6 f& }8 G
<value>300</value>/ }8 {5 l8 n( r$ M: V5 @6 ]0 n$ r0 M
<description>The threshold in milliseconds at which we will log a slow3 D2 [& y# m) ?
io warning in a datanode. By default, this parameter is set to 300, v1 v5 R1 T6 ~, ]
milliseconds.
* \, [6 m) k* ~! z2 i4 @9 x </description>
! Y3 k$ X1 E# y/ N7 F" Y</property>
' u: f: l8 E8 X<property>
k! S6 {+ I4 d1 N <name>dfs.namenode.lease-recheck-interval-ms</name>
# R# h- E% g5 V# W+ t# q) n <value>2000</value>
9 v8 B% h! u/ ?; Q- D. d, M# e! o$ A <description>During the release of lease a lock is hold that make any
3 q. x7 w* m$ c" y operations on the namenode stuck. In order to not block them during3 l7 T( Q1 g9 I% |9 y7 V
a too long duration we stop releasing lease after this max lock limit.
+ }$ D; ?1 R9 T- C" o </description>3 n* q. G2 W' |; i' T% l z
</property>
6 x+ F) C" S( _$ r- y7 {; @' R4 I( U<property>5 I. Q" {+ }4 D' l
<name>dfs.namenode.max-lock-hold-to-release-lease-ms</name>8 E6 _; I1 K/ T( e C
<value>25</value>8 {, G5 G) k5 F0 w& _
<description>During the release of lease a lock is hold that make any% q/ c/ h$ s. R# h
operations on the namenode stuck. In order to not block them during
+ A- B, P I5 {* V. e a too long duration we stop releasing lease after this max lock limit.
1 v* A" T- z4 C) }# [# ^* F. V& u </description> H2 p" m4 q4 g& T% X" t
</property>
& }# i& z; y2 l& z. p<property>3 p2 h6 [$ m( x4 m( H
<name>dfs.namenode.write-lock-reporting-threshold-ms</name># p; M9 w% @ l, M& m3 g
<value>5000</value>1 j0 h% z" w j$ T, u
<description>When a write lock is held on the namenode for a long time,
4 X# V M. r4 s& k; J: E+ X) J this will be logged as the lock is released. This sets how long the
) V! T8 C8 _# F! W2 ]$ C, e# {, \ U lock must be held for logging to occur.
% p: U' ?1 @0 _ b2 Z </description>
4 r! y7 W7 W. n: X0 W! Z+ S, s% d</property>
: w( `* d5 C0 H! ^/ l5 r<property>
1 i) p. X. y( B8 w6 d2 d/ S <name>dfs.namenode.read-lock-reporting-threshold-ms</name>
' y% ], x! L- B1 w9 y; r8 _ <value>5000</value>
& \$ ~3 X/ l1 z" h2 g: ~) ~( ~ <description>When a read lock is held on the namenode for a long time,' O- x' ]0 h& p; L* A1 m
this will be logged as the lock is released. This sets how long the6 v1 p: I" N$ E) W G. O( u5 e/ a
lock must be held for logging to occur.
2 B4 e) F6 g Q: f( W* d/ v! } </description>
* o9 p5 Z5 ` L% | {2 Q8 P</property> }( ]0 |/ ]( i$ U# ] y
<property>
/ A+ S7 t5 ^. ] y. p/ K: S6 Z <name>dfs.namenode.lock.detailed-metrics.enabled</name>
: h0 a2 {3 d9 A: B- { <value>false</value>
+ z1 Q4 u$ b/ V3 v( T <description>If true, the namenode will keep track of how long various; m# U4 [/ h' U8 q" x& t; ~4 M
operations hold the Namesystem lock for and emit this as metrics. These: i+ N5 M3 o4 j, ]$ x# H; |/ e/ D
metrics have names of the form FSN(Read|Write)LockNanosOperationName,
% p* \4 x8 P! n6 J where OperationName denotes the name of the operation that initiated the9 P; f) I+ ^6 z5 K- T& f
lock hold (this will be OTHER for certain uncategorized operations) and z9 F6 v5 a" F% T) o$ {
they export the hold time values in nanoseconds.# N( O7 b9 y: }1 w6 g. u
</description>
2 g' i: }* V7 ^5 [</property>
' I3 M/ X4 I" _2 ^8 S, _3 b<property>
{- i0 k3 W: E: Q. g: i <name>dfs.namenode.fslock.fair</name>7 R* Q6 W9 U* k6 o$ S n
<value>true</value>
; D7 C- j$ p# ?3 c' e1 ~& `5 [ <description>If this is true, the FS Namesystem lock will be used in Fair mode,/ y R7 }+ A% z- X+ @
which will help to prevent writer threads from being starved, but can provide5 Y7 B$ r, h) q* Z/ I$ }( Z
lower lock throughput. See java.util.concurrent.locks.ReentrantReadWriteLock/ R# e0 u- H) V! m$ d6 l
for more information on fair/non-fair locks.
- U1 z4 [) A i. g* P' U+ s6 I </description>
: y' ^ L4 z9 ~8 \0 T; s; R8 T</property>
' F: g$ l% m, \/ w- |1 I<property>( _- F1 v5 m2 Q+ G! h
<name>dfs.namenode.startup.delay.block.deletion.sec</name>/ B- G/ @ K. o) E) d6 ?* O8 U, P
<value>0</value>
. l& E ^7 ^ L2 |, R, e* @ <description>The delay in seconds at which we will pause the blocks deletion0 o6 f$ ~; d7 P9 I& D
after Namenode startup. By default it's disabled.: S2 [1 x$ m, S. u8 b2 l* L- E
In the case a directory has large number of directories and files are
2 R7 O" d m1 f/ X% o- t8 G deleted, suggested delay is one hour to give the administrator enough time6 J9 c3 s, a" P6 E( [! e) \* U
to notice large number of pending deletion blocks and take corrective9 a1 X# r1 v8 }
action.+ J6 m' W# a2 f2 H" T' {' g
</description>
) S1 k0 f3 `/ X- T1 E3 r7 ~</property>2 {0 E* j4 f$ p+ l2 {, O2 u
<property>/ I/ [. z, \9 U2 }( Y# P& X" s
<name>dfs.datanode.block.id.layout.upgrade.threads</name>) J- B+ }1 ~$ k3 s' n3 i- l5 f
<value>12</value>! p7 a7 i- ?, s% B4 T7 }
<description>The number of threads to use when creating hard links from/ K( v# Y9 `. r$ I' t
current to previous blocks during upgrade of a DataNode to block ID-based( h+ t( z, X0 h9 R! _& m" C
block layout (see HDFS-6482 for details on the layout).</description>
! x$ `! |& {) f. L</property>4 |6 V, X- d! c+ [
<property>, K+ k# j* X" _9 h3 d
<name>dfs.namenode.list.encryption.zones.num.responses</name>
" z# P5 V1 ~. B" t H w H, d <value>100</value>
2 f9 H$ ~1 }" ?# {0 ? e7 P# p <description>When listing encryption zones, the maximum number of zones
6 [- |6 f* |( f( k that will be returned in a batch. Fetching the list incrementally in; G/ ~4 p3 s' _: X3 p$ |
batches improves namenode performance., R( \1 ^8 \7 P: F
</description>
$ E- C2 ?" r. |8 F0 s</property>
2 s* {& a2 V7 Q( `<property>4 x, k4 S+ h. ^# }+ M# V
<name>dfs.namenode.list.reencryption.status.num.responses</name>. U) f. x+ ]* E
<value>100</value>
! f3 [2 ? }1 c5 D <description>When listing re-encryption status, the maximum number of zones
1 n* Z6 } S, i u- |. D! L" x3 ^$ k that will be returned in a batch. Fetching the list incrementally in( k4 d4 F' j# W
batches improves namenode performance.
) K$ t9 r6 M- B6 L1 c: u' Q </description>
n; }( x( t5 s {" P: b( d</property>- M2 ?" f: R" p8 P, J+ }1 H# o
<property>
1 z7 y. r& D" v- _- @5 t$ E <name>dfs.namenode.list.openfiles.num.responses</name>
7 p9 W- {2 S; G2 ]1 p$ E% o <value>1000</value>
' L6 X/ a3 q: j- f- O <description>* A# s; z( c; y4 U+ v4 [
When listing open files, the maximum number of open files that will be
4 A! F% q) L0 y6 Z returned in a single batch. Fetching the list incrementally in batches; V7 Y3 f6 G* T
improves namenode performance.
% @3 `0 d) [; X/ ?8 ^1 H! }+ a </description>2 |" n' z ~$ u1 z* k+ U) J
</property>
4 J( } v8 t# P/ E<property>9 Z: [+ `- b( d) A
<name>dfs.namenode.edekcacheloader.interval.ms</name>2 y! _, `4 g$ J, R
<value>1000</value>& Z g& a8 R9 H4 y+ G- g) ~, F
<description>When KeyProvider is configured, the interval time of warming- o( |& v1 ?+ [+ A9 [# {
up edek cache on NN starts up / becomes active. All edeks will be loaded
& x1 s, H) A2 b" R. O9 x* V8 Y& j from KMS into provider cache. The edek cache loader will try to warm up the3 G. H+ R7 V9 _7 q; J5 y
cache until succeed or NN leaves active state.
8 o' ] h& B7 o4 L4 m; x5 J </description> _" |0 k8 d2 H- z. q
</property>
. @+ R' L2 z! m1 p5 |4 o$ Y- ^<property>1 ^/ m7 L- T* U( w n) u
<name>dfs.namenode.edekcacheloader.initial.delay.ms</name>
+ J. l) k' ^5 m7 v <value>3000</value>
& B. Q% b) G3 e# y, R. h+ S <description>When KeyProvider is configured, the time delayed until the first5 d6 `- a0 l1 G9 I
attempt to warm up edek cache on NN start up / become active.
c$ Z8 D6 h( L5 W% t8 u </description>
# \( O, d& E; z% K$ b, T$ d</property>
0 D# T' c0 g N1 J. o6 D<property>$ D! q# x2 Y% `. s6 U
<name>dfs.namenode.reencrypt.sleep.interval</name>( B& y; R0 ^+ S% p# [# Z! G6 T# l
<value>1m</value>1 E8 z3 q, w$ L; M
<description>Interval the re-encrypt EDEK thread sleeps in the main loop. The
' Z c' H- c( U2 n/ H interval accepts units. If none given, millisecond is assumed.
4 f+ l' y( C, |, \: y </description>
0 C1 A( x* ~% J S& J$ s</property>
2 J: v0 m8 M( u- U<property>7 H) k* e) _ d3 I/ a( W' A/ M, j
<name>dfs.namenode.reencrypt.batch.size</name>
6 M" N! o4 }/ l; h0 p3 Y2 f# ` <value>1000</value>: m/ X9 @# p' c4 U! m4 f
<description>How many EDEKs should the re-encrypt thread process in one batch.
! t) g. t5 n9 Z; m8 q </description>3 t6 G, ]# s8 }2 H( B; U/ d/ X
</property>
# A; Z9 X0 }' i<property>
9 x8 ]% ^: X# s9 s$ V, ]9 }' m7 k <name>dfs.namenode.reencrypt.throttle.limit.handler.ratio</name>7 J. E( u& E9 o7 H+ G, z& E3 Y
<value>1.0</value>
# G7 M9 A& m: ?/ m; U <description>Throttling ratio for the re-encryption, indicating what fraction% N: d X0 P: a
of time should the re-encrypt handler thread work under NN read lock.
s7 f8 k7 O6 `' p6 ~% x. b4 c Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are
1 C" P& R( P( l& v% G8 T! r5 { invalid values and will fail NN startup.
' ~( Q! Y+ ~% L% f! ?: q9 V </description># p2 \* N9 z) y4 a
</property>
; ~2 f; j2 P7 V/ ?& s( m- L- M3 I<property>
# Y9 D8 A2 [% c/ w" t# z <name>dfs.namenode.reencrypt.throttle.limit.updater.ratio</name>
, P: N; X, R/ ^& ]% i# ^& _1 p <value>1.0</value>
2 J0 X7 f8 j2 H/ Z <description>Throttling ratio for the re-encryption, indicating what fraction
8 {$ y* v! _7 S of time should the re-encrypt updater thread work under NN write lock.
6 i! s: \5 V, v4 t$ o, K Y Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are# Y1 Z z& j3 k( y
invalid values and will fail NN startup.
" h$ m8 W8 X& N' u8 s& o @ </description>
0 s; w$ |- m# `- ~3 I3 ^</property>/ x- L. v8 k. N, J" J
<property>, ^- e6 b+ E. k; e5 K' N' I! Y
<name>dfs.namenode.reencrypt.edek.threads</name>
# A: A/ X& O9 }5 D <value>10</value>; |1 E8 q! e+ Y a
<description>Maximum number of re-encrypt threads to contact the KMS2 @9 ^' n: Y" }8 T, g, i, y
and re-encrypt the edeks.! n! m. B; E6 s
</description>9 ^7 C, h) e4 x: [& F5 o# E3 t
</property>7 g0 P- p' q0 U2 D; ^
<property>
) l; M8 w' V% Z1 H+ D/ E* V <name>dfs.namenode.inotify.max.events.per.rpc</name>( P6 q7 \& K% j% i i+ q
<value>1000</value>2 ]7 i; @( o2 \0 B9 k! t
<description>Maximum number of events that will be sent to an inotify client
- m0 r1 q+ j) R! d- u4 ~ in a single RPC response. The default value attempts to amortize away+ k- ^8 [* F F/ y0 i' N' x1 p" Y
the overhead for this RPC while avoiding huge memory requirements for the0 l8 S% D- l+ x1 ~' n
client and NameNode (1000 events should consume no more than 1 MB.)* x+ O! R3 t" V1 y+ X" ?% l; |
</description>2 Y. g' Z) D* H# G
</property>5 U' z. E0 z2 C$ M* J
<property>/ Q, \1 d, x) A2 V- J
<name>dfs.user.home.dir.prefix</name>
; Z+ \4 A8 E1 a8 C1 x <value>/user</value>: Z' r$ c) E% [. n$ \( \
<description>The directory to prepend to user name to get the user's3 {* m9 X2 { l, C% y# n" g. g
home direcotry.* W' w; Y$ m/ k6 b) ]
</description>
$ Y& @2 R8 @, u, G; {' ?</property>1 g, F. Y; H0 K! c h# i' d5 b! [
<property>- v7 n9 r4 x0 }. z, b
<name>dfs.datanode.cache.revocation.timeout.ms</name> U) }0 i+ x* B/ b s" e
<value>900000</value>
$ N; J4 r, L, E4 R/ L* b! H <description>When the DFSClient reads from a block file which the DataNode is7 h! R" z4 y5 R
caching, the DFSClient can skip verifying checksums. The DataNode will" D9 j$ I, k# B, y" }% v4 w, a
keep the block file in cache until the client is done. If the client takes
- g8 `- G" m% X- ]* j; E# ^ an unusually long time, though, the DataNode may need to evict the block
4 w; r. g8 I5 [) C6 R, O file from the cache anyway. This value controls how long the DataNode will
7 T% e) x- ^) e( ? wait for the client to release a replica that it is reading without2 }0 A. v! j* ^$ P& v7 M
checksums.
) S$ ]% [ _) G& n( }6 T </description>
; z) d. N& `# L+ m</property>
( h3 u4 Y; ^! J+ b( x' ?8 M; j u<property>* k" }6 J7 U5 m& R
<name>dfs.datanode.cache.revocation.polling.ms</name>
; I+ p, X' y5 ~. Y3 V <value>500</value>- A+ [$ U5 P2 x" P2 z
<description>How often the DataNode should poll to see if the clients have( \" H; r n! N
stopped using a replica that the DataNode wants to uncache.0 }" ~5 [4 s9 l# [* G
</description>
8 k( e, r$ ?/ W' e1 C</property>
( h6 S: z3 U) M+ ~( F) H<property>, Y9 C' o3 f( D3 W* ^( _
<name>dfs.storage.policy.enabled</name>6 Q0 l! l! [$ X7 U* g+ l" O. g* s7 S! C
<value>true</value>
4 H- b9 |7 n& }2 {' _3 A <description>1 T) w7 H. @; ^: `2 e
Allow users to change the storage policy on files and directories.
4 H: s1 r L$ r3 {! l9 S </description>
# U X7 @6 |" Q% z F</property>
# @ o! a# {! ^3 ?0 F* J* [1 V) x! R<property>/ a- ^4 I# ]6 g1 H; G" G* C4 ]
<name>dfs.namenode.legacy-oiv-image.dir</name>2 g- Z. h$ v+ N/ ~: x8 P, _
<value></value>8 j: X; }% k2 r F( D- ~
<description>Determines where to save the namespace in the old fsimage format5 d3 F3 q) Q# Z7 y+ B
during checkpointing by standby NameNode or SecondaryNameNode. Users can, K% b( z# F2 m
dump the contents of the old format fsimage by oiv_legacy command. If* m3 e* M$ ]6 f" E' m5 ]
the value is not specified, old format fsimage will not be saved in* m% i4 K, t/ h9 v9 S# u: b
checkpoint.
. p' r j* _( v" b$ \, ` </description>
/ h1 _6 d8 h; H L* x9 o4 L4 Y</property>
5 F, X! z5 `3 l% q! ^! H<property># Q! |% I# X0 t( w1 K& W! v
<name>dfs.namenode.top.enabled</name>" Y8 s; N4 R2 i5 o6 ^) t5 e- C
<value>true</value>$ ^4 d) Y9 n" G( \% n( F/ f1 X
<description>Enable nntop: reporting top users on namenode
, i( |; [% S. O" T: ]/ i </description>' A6 k" e" J9 b+ M! p
</property>
0 F" [& k' Q- k8 Q8 a- Z<property>% E: G1 H w$ q" q
<name>dfs.namenode.top.window.num.buckets</name>
8 }8 c/ m, b8 t r <value>10</value>. a! {* G) \) x% N2 e% x- u
<description>Number of buckets in the rolling window implementation of nntop
N- F( j/ d/ U% G$ b; } </description>3 f3 Z* n! n7 D
</property>
3 f6 n: ?" t1 D, Q% K0 z; Z<property>
% f$ C3 ?# s) k. H: v <name>dfs.namenode.top.num.users</name>( y, p6 K6 R: ~5 ^
<value>10</value>
( L x3 t3 p% P( n9 D8 [; K <description>Number of top users returned by the top tool* V7 M( f5 P; S. A6 s
</description>3 J- g) ]- p. r7 I% z1 M3 ]; {
</property>2 B Z. s N+ l7 r
<property>
1 I. ?) G) u2 X1 V' R6 b <name>dfs.namenode.top.windows.minutes</name>
) l+ x/ C; F- V V/ t6 F- ] <value>1,5,25</value>% c4 U: J) U2 _
<description>comma separated list of nntop reporting periods in minutes
% q) g) Y) d+ M$ g3 { </description>
# B' f1 A: A* f q4 w</property>! ^. z+ I* b' h; @! |% r6 L
<property>
" |3 q% k& ?' I/ g* w' U0 Q, {7 c& p <name>dfs.webhdfs.ugi.expire.after.access</name>, s* [' z% R6 F4 H& G' x# ~
<value>600000</value>
% \. ]9 D0 H, K <description>How long in milliseconds after the last access
2 h& U) \ g& L the cached UGI will expire. With 0, never expire.
* ^: U1 J( \% s& m/ W# E1 p; K </description>/ d* c& b2 G3 X- M" \4 n
</property>
1 W. X# D- Z- G<property>* W% B1 @( [1 l) T) d; s
<name>dfs.namenode.blocks.per.postponedblocks.rescan</name> K# b7 y2 Q- Q0 O4 u( @
<value>10000</value>
p' E% l5 m$ u* r <description>Number of blocks to rescan for each iteration of$ e4 y& v1 e1 K; y7 a! j6 @
postponedMisreplicatedBlocks.
* p+ B& J1 c( \* v! u </description>/ B8 z- v" @* W% D$ m# |
</property>
2 Z t; ?, f' Y, Y w<property>, e! B/ N# n- O* P( @8 M
<name>dfs.datanode.block-pinning.enabled</name>+ h2 D: a9 j# P: G4 T( k F& W' k" L
<value>false</value>' \0 P7 Y' W, l* O0 a ?
<description>Whether pin blocks on favored DataNode.</description>
$ a/ x8 _6 s& e8 y3 B- O</property>
7 e0 P! r, w& i<property>6 y! M) Q' M) e
<name>dfs.client.block.write.locateFollowingBlock.initial.delay.ms</name>
, ^* @6 }4 v1 Y) S <value>400</value>
9 V2 C) g. \* A4 |: D2 F2 P <description>The initial delay (unit is ms) for locateFollowingBlock,% m1 d/ `8 d1 `9 m7 O: W
the delay time will increase exponentially(double) for each retry.
T3 i S/ A' l- E; a </description>
7 U: u1 {" q. s# \2 l; o</property>. A( a) |, {: s& S c4 y B
<property>
/ y' X! W( R2 J# j! x3 X% m <name>dfs.ha.zkfc.nn.http.timeout.ms</name>5 p7 d2 W, M. h
<value>20000</value>
6 @) u! I7 o/ k <description>
9 z& f9 E5 F! ^/ A3 M& N' y" j. n The HTTP connection and read timeout value (unit is ms ) when DFS ZKFC
0 m; K+ W6 T A: C! A& d8 r tries to get local NN thread dump after local NN becomes
0 A" B3 j' n! ]* P+ R SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY.8 l* s) g1 F. Z1 v0 A- a
If it is set to zero, DFS ZKFC won't get local NN thread dump.
, z& I- D! R, v( P& \ </description>; W% u+ r5 e3 F0 c
</property>
% p+ e5 s$ y7 G8 r9 w<property>3 O" T' S4 X. y! d, Y, Y
<name>dfs.ha.tail-edits.in-progress</name>9 W2 Z2 M. M' }; \3 n
<value>false</value>
5 v0 ]8 b0 a6 d( L <description>
; g" W( O; C! t0 V/ G8 K0 J Whether enable standby namenode to tail in-progress edit logs.! ^- d, Z" b- I: w
Clients might want to turn it on when they want Standby NN to have
2 ~* _) Z8 w. o& g5 j more up-to-date data.
, ^* U1 }) Y9 p' c4 e% }8 M1 m </description>1 _ }- x6 \0 Q) D
</property>6 E9 A' G9 ^3 B2 b/ k( w* h, l" Z
<property>6 ^' B; a, w* e! Q/ `' ]
<name>dfs.namenode.ec.system.default.policy</name>
) o" q& P( g, c/ s7 r4 P x <value>RS-6-3-1024k</value>
7 i* _2 w, R) w- s <description>The default erasure coding policy name will be used/ A! R _& [# @* y; N" [3 w6 }' m
on the path if no policy name is passed.* L0 s! c f* T( g3 c' P+ A# q, N" [
</description>) C6 @4 h" E. S
</property>" S# m8 A5 u: Q0 d
<property>
& c3 L7 p4 A8 H9 H9 W+ l: M <name>dfs.namenode.ec.policies.max.cellsize</name>
( X9 y3 q: D, {0 G8 F- b1 P/ O( R8 Z <value>4194304</value>
& h, i2 D6 z. F5 G <description>The maximum cell size of erasure coding policy. Default is 4MB.
- R$ A6 ], \! m( D, H3 C </description>; R/ h0 M/ ]" T7 ?% k- N& K
</property>. d: B- S7 V! B4 S4 v9 d7 O, h
<property>7 C4 ~- Z/ R" C. g% v; |) [
<name>dfs.datanode.ec.reconstruction.stripedread.timeout.millis</name>
0 w6 E3 d0 w$ J2 _. _" ?+ V <value>5000</value>; P* p+ q" j+ ~
<description>Datanode striped read timeout in milliseconds.+ a$ N9 W5 ?/ f2 G
</description>/ v3 ^; }. p" A. l* i6 \& z
</property>; m! k+ d- Q' q2 r* Z X
<property>1 K+ ]2 m) E' f4 r+ h4 A# s" B
<name>dfs.datanode.ec.reconstruction.stripedread.buffer.size</name>5 T* k" `+ T0 n
<value>65536</value>0 y3 v& C/ z: `& R. W
<description>Datanode striped read buffer size./ r6 y# R" D5 `; g' {% Q4 U
</description>
0 y# k# f* k" L</property>! u! V# z0 |, |2 K. c4 r$ H5 d
<property>: I {, F# G8 k! E+ a
<name>dfs.datanode.ec.reconstruction.threads</name>0 T+ V% s7 j, Y0 G, s) ?+ \9 s
<value>8</value>0 h! [% X* L3 X) R+ z1 R
<description>$ E. T' V+ d7 [5 C1 Q( k% K
Number of threads used by the Datanode for background+ ?& ~4 n3 w3 U" C& w0 G
reconstruction work.* |% G5 R# j7 ~
</description>
: x! V7 O/ S* d% |: |/ O( B</property>
0 V* t5 e |% y1 ^6 W1 b8 ]<property>
0 k) e# g+ N7 h4 O0 V G h <name>dfs.datanode.ec.reconstruction.xmits.weight</name> E: s3 j: S# o7 }5 |4 t
<value>0.5</value>
# B- ?6 U9 r9 X! `: l/ Y; @ <description>
! f4 R$ I4 a. ^" [, {& S" E% A Datanode uses xmits weight to calculate the relative cost of EC recovery+ W, ?: b* X0 O. i( `
tasks comparing to replicated block recovery, of which xmits is always 1.
& I; z9 A: m8 z% V. {1 P! o; | Namenode then uses xmits reported from datanode to throttle recovery tasks
3 L% r+ Y: E1 r5 ~$ ]) ]/ { for EC and replicated blocks.
# }" o B6 ~- B7 {1 C1 W0 N0 X The xmits of an erasure coding recovery task is calculated as the maximum, n2 h( r$ [- T* l" I
value between the number of read streams and the number of write streams.1 e9 |7 E6 l) p' V3 z5 m
</description># I1 Z. Q5 V0 g; |) c |, {6 _
</property>
9 _8 W' y7 j4 M<property>
2 ]$ p9 C4 M* m$ v: @: W+ Z5 G, _* J <name>dfs.namenode.quota.init-threads</name>
; r+ X5 N9 z$ d, o. l <value>4</value>( Y& _' Q& x2 {3 r5 N8 B9 x% o
<description>4 ~2 N1 G8 Q- y- ]
The number of concurrent threads to be used in quota initialization. The
; W; B4 P! \2 t' D3 c& i+ w/ ~ speed of quota initialization also affects the namenode fail-over latency.
3 X' Y( T T# ?* h/ b If the size of name space is big, try increasing this.+ e! l8 X" e0 M+ g2 `9 F, [
</description>& E; W1 Y, h- t3 A( ~
</property>
" t2 o1 K5 O! v! }& ?: S9 f<property>. |0 N o: @( A3 A, j: V
<name>dfs.datanode.transfer.socket.send.buffer.size</name>! Q+ A0 b2 [$ o8 F& n1 R0 h/ X, M
<value>0</value>! z$ S4 ]* p$ g8 }3 \9 U
<description>( ~) M; }1 x" g. @: U& g- v
Socket send buffer size for DataXceiver (mirroring packets to downstream
$ ~/ r) V4 V5 O1 h9 \ in pipeline). This may affect TCP connection throughput.6 S0 x; z* ~& J) E
If it is set to zero or negative value, no buffer size will be set
( f; k9 A7 ~* L4 T- X$ y- H: d5 D explicitly, thus enable tcp auto-tuning on some system.
5 u" U: i& M- C The default value is 0.
$ j6 b: D' n" g. K# Y! R9 H </description>
9 w( l2 B5 l5 \% z</property>
% @1 ~) U) u3 c<property>7 h% S3 q; s9 t
<name>dfs.datanode.transfer.socket.recv.buffer.size</name>/ x$ y3 ?0 e# Y. n! D
<value>0</value># f$ q& |1 A+ s+ Y9 z: j
<description>; U9 d% \( Y- \8 y& V4 u. C0 ?
Socket receive buffer size for DataXceiver (receiving packets from client
: W+ x8 G9 {3 t: z9 L( ^ v during block writing). This may affect TCP connection throughput. a7 ]( j) D7 Y: ~
If it is set to zero or negative value, no buffer size will be set
/ j9 L) S( a9 R( F$ J' k4 s explicitly, thus enable tcp auto-tuning on some system.
/ U9 T/ B# [& Q4 G! b3 b2 N The default value is 0.: d, b& M! c" |/ s# G
</description>- \" B! r' x* ^4 G) c
</property>
- f7 h8 Q, D+ t<property>
6 w- T- T" V& S- L' W <name>dfs.namenode.upgrade.domain.factor</name>+ X; p' u+ t! n5 u4 H2 v6 }8 N3 w
<value>${dfs.replication}</value>+ f* e5 m4 n% [/ c
<description>5 R7 i1 A# u D' K2 Z2 {/ \
This is valid only when block placement policy is set to' N$ F( x n5 K9 u
BlockPlacementPolicyWithUpgradeDomain. It defines the number of
9 o+ w* e% f+ A1 V* f: o unique upgrade domains any block's replicas should have.
; D" T2 r6 F+ ]: g" ]$ v When the number of replicas is less or equal to this value, the policy8 V7 B" h( R0 h+ g& r {3 I$ o6 @
ensures each replica has an unique upgrade domain. When the number of+ _) k) o ?; l
replicas is greater than this value, the policy ensures the number of
% d% J2 o: X- A- u3 M0 ~) U unique domains is at least this value.4 t+ c' @2 Z) J+ a
</description>6 y2 _1 C' @( q2 ]) k; c. ]3 G
</property>& X- m: w" J" h0 M5 c' ]9 T% L! Z
<property>
7 C. v4 C$ |3 H- m) t) [ <name>dfs.ha.zkfc.port</name># Y/ W. g$ o; S) G7 K. v6 t* ?
<value>8019</value>+ N& G) R3 {5 G1 C' ]; \. t
<description>
- d7 j. x0 M+ ] J7 Y2 w/ e RPC port for Zookeeper Failover Controller.0 O+ W; }& \) E
</description>
, P* v! ~ _2 F- h W</property>
. o+ M( L8 {1 d<property>
- H7 j/ Z! K r0 ~ <name>dfs.datanode.bp-ready.timeout</name>
; S' Z6 r# _; r) g7 V <value>20s</value>" E4 w8 x2 { C9 r
<description>9 d5 Q: E0 a$ |1 E3 E: h
The maximum wait time for datanode to be ready before failing the* G3 }1 Z6 ]& V' `! d
received request. Setting this to 0 fails requests right away if the/ P6 a# y0 J; b/ X, k: x* E
datanode is not yet registered with the namenode. This wait time5 x9 O* ?. ]% B( ]6 c- i
reduces initial request failures after datanode restart.
( l0 t& j: x# V& w6 y, `- N3 S Support multiple time unit suffix(case insensitive), as described
4 q9 c8 @" L7 y8 ` in dfs.heartbeat.interval.: h. Y k+ K5 H( V/ |" U1 U# R
</description>7 l7 j1 K. P3 a& }
</property>
7 |$ q8 u, Q, h M. c& \* ~+ B<property>: L( G2 n2 I/ K; w
<name>dfs.datanode.cached-dfsused.check.interval.ms</name>/ u! C- U2 f! x3 K- O
<value>600000</value>
7 q+ o4 e$ z* A6 V; u <description>, x- }% H+ E) G& w* g/ h' b
The interval check time of loading DU_CACHE_FILE in each volume.; `0 O k5 E* M; z9 A1 ^
When the cluster doing the rolling upgrade operations, it will
; _7 K6 D% [, {/ p( y2 V usually lead dfsUsed cache file of each volume expired and redo the, o/ f* d5 ]( ]8 Q% I
du operations in datanode and that makes datanode start slowly. Adjust# h7 Q% W! ~) P2 O% j9 b
this property can make cache file be available for the time as you want.
8 t, h$ f) z+ W. S, i" ] </description>
' O) F! |4 `) H9 f8 P5 y; h% c</property>
& E2 |; \0 q P9 b) m; {. `<property>1 K3 ?$ n# R7 t- v U% F* N
<name>dfs.webhdfs.rest-csrf.enabled</name>' L. m) C h* [& `! T' {
<value>false</value>! E2 k, }5 \1 w
<description>
" ?* ?/ H3 `" F. ? L3 O If true, then enables WebHDFS protection against cross-site request forgery
8 R3 x6 d3 Y: E2 H# g (CSRF). The WebHDFS client also uses this property to determine whether or1 G; T" \5 v# h
not it needs to send the custom CSRF prevention header in its HTTP requests.
8 C" N% C% [; i( P) p5 {" t </description>
+ h8 Q; ^4 R4 H% C</property>/ W: d+ b. l- k! {* ~; x
<property>
% _9 j0 ]" i$ K! K3 y3 \9 F+ z <name>dfs.webhdfs.rest-csrf.custom-header</name>+ x' _7 E: c9 b5 ?
<value>X-XSRF-HEADER</value>+ @" C( `4 B9 I6 v8 @
<description>6 U4 M0 _ U/ }. @/ O
The name of a custom header that HTTP requests must send when protection* q! L2 _ p' q$ X k9 w1 E" d
against cross-site request forgery (CSRF) is enabled for WebHDFS by setting! X/ D8 i' X9 m! V% `
dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this
$ y3 }) h6 {; l1 [' s( Q" M( j property to determine whether or not it needs to send the custom CSRF
7 l+ U8 b5 q4 w1 j0 i prevention header in its HTTP requests.$ v0 O* u: f/ g$ {# C' s
</description>: C( m, @" H# ^0 K$ `$ d' O9 W
</property>, h0 y5 p$ W( V! z' m
<property>' k4 b2 d# n, z& Z* @& Y" y
<name>dfs.webhdfs.rest-csrf.methods-to-ignore</name>+ N$ E# f# {* F0 {) |
<value>GET,OPTIONS,HEAD,TRACE</value>4 z) v) _& ^" u k3 m7 H9 _0 \! t% Q5 n
<description>
) [& A9 [/ J9 f" `+ I A comma-separated list of HTTP methods that do not require HTTP requests to
6 X6 u( x" n) l- R; z6 f* f9 L7 H include a custom header when protection against cross-site request forgery' S i3 q1 Q. v, p* o; I$ Q
(CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to; c6 o% P5 K# n J
true. The WebHDFS client also uses this property to determine whether or% [4 q- ]& w6 [* W
not it needs to send the custom CSRF prevention header in its HTTP requests.3 l: A5 | A+ } @+ w. [
</description>
/ k! k' @ B1 T3 f</property>5 S, D: V& t0 K4 G' |
<property>. m- H9 r b8 i) D/ M/ T
<name>dfs.webhdfs.rest-csrf.browser-useragents-regex</name>
, Y) I) e; h: S% r <value>^Mozilla.*,^Opera.*</value>- Z& k$ m M0 s! C3 \2 `. U
<description>
7 z( |) W8 x5 f' n A comma-separated list of regular expressions used to match against an HTTP
- y4 V0 B8 b) H- }( c0 w request's User-Agent header when protection against cross-site request3 o, @ J" e) B: g1 \" C
forgery (CSRF) is enabled for WebHDFS by setting
* o# E O- N c# r T; } dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches
4 H6 {7 p2 C7 l0 P M5 \ any of these regular expressions, then the request is considered to be sent
* v0 y# X' I3 f4 S$ \ by a browser, and therefore CSRF prevention is enforced. If the request's
( G4 [0 K$ L$ ] User-Agent does not match any of these regular expressions, then the request
1 s# I2 Y" J5 U/ O is considered to be sent by something other than a browser, such as scripted6 ^9 [1 B- ?/ x0 ?
automation. In this case, CSRF is not a potential attack vector, so
. F" y6 r9 e- Q$ w% |2 ]8 k the prevention is not enforced. This helps achieve backwards-compatibility
& r( V2 E. z: @: S! W+ F with existing automation that has not been updated to send the CSRF
6 s/ }2 P k2 I" t3 {% p1 X prevention header.
Q' O( W# m4 i2 f4 x/ Q7 h </description>) p6 W3 f8 j: ^* k8 i$ n
</property>
3 U+ J$ [8 E- B3 [7 J- `9 N <property>7 \4 x/ I1 F" ]) _8 J4 O, W4 }
<name>dfs.xframe.enabled</name>
: c* i9 s- W- q; |) d: Z2 q, O" a x <value>true</value>
) u, N& ]; A+ n2 H9 K6 L5 ] <description>6 S8 V" V$ o" p) ~. r
If true, then enables protection against clickjacking by returning
0 P6 ^, p+ b' J. Y X_FRAME_OPTIONS header value set to SAMEORIGIN.' r8 d, a' \/ c$ R( z u
Clickjacking protection prevents an attacker from using transparent or
0 q) Z* _1 N' M, v1 U1 G opaque layers to trick a user into clicking on a button
8 G2 W2 l$ M+ e1 L# y8 T/ g or link on another page.
$ Q$ t& Z5 ^7 ^2 j6 p) i# c" {2 \ </description> y2 H8 g2 b; S' e( l
</property>
$ k* E) d5 R. l( C3 o <property>
: k; ]1 ?; }" j( r <name>dfs.xframe.value</name>
- o1 \2 @ u& ]3 i( I; G3 C1 S4 i" U <value>SAMEORIGIN</value>$ E6 r1 m& T0 ~& [! ~ q
<description>4 H5 ?. l/ e" k4 y& W! |
This configration value allows user to specify the value for the" b. c! s% f" E! e1 U
X-FRAME-OPTIONS. The possible values for this field are7 w4 e% N# w) d+ U6 p- V, U: f7 P
DENY, SAMEORIGIN and ALLOW-FROM. Any other value will throw an
; I3 \/ {6 Y9 a+ j/ F/ K exception when namenode and datanodes are starting up.
7 J6 n }5 `' [% u' E5 {& X </description>. _. m# s- P& f: M/ e9 t
</property>
2 J: ~$ J. o ]<property>3 a5 h% B( `2 J/ b
<name>dfs.balancer.keytab.enabled</name>
) U% i G0 y* u( u9 r <value>false</value>
9 A% h' F% ~5 `8 P( Y <description>8 v* e6 ?; f& R, \7 \% y
Set to true to enable login using a keytab for Kerberized Hadoop.5 D; G$ J/ X; Z
</description>* i( h6 y2 w2 q( T
</property>5 B9 S! `. k3 Z5 @7 l$ v
<property>
. @$ t4 C- g/ t2 R; K <name>dfs.balancer.address</name>! s8 N' Y9 ?/ S# u; ~0 X
<value>0.0.0.0:0</value>
3 B$ v* y: s7 {+ B <description>
# n+ G- d# h; W3 a The hostname used for a keytab based Kerberos login. Keytab based login+ ]6 _2 g, \1 Q6 A3 G
can be enabled with dfs.balancer.keytab.enabled.7 S+ ?8 t# T# |3 n) U
</description>
. s! _3 J; S6 Q4 U</property>
7 [, S4 E" f6 q5 {* j C<property>
% n0 ~* o7 H4 k% X/ f; \4 E <name>dfs.balancer.keytab.file</name>
5 ` a8 A! X7 }' \0 p <value></value>
5 T$ \ f3 ^, i <description>
9 w: Y% V. m, H) } The keytab file used by the Balancer to login as its
1 t* L8 u/ w; Z service principal. The principal name is configured with& d! q; w8 E0 d: f
dfs.balancer.kerberos.principal. Keytab based login can be
7 T) x5 o( I' y- X0 L# l6 k$ A. H. F: ^ enabled with dfs.balancer.keytab.enabled.
' ~. k! f- Q- B' h7 Z( _" [ </description>1 z4 I) B- @" G6 V
</property>% v0 [$ n/ @* @4 |+ n
<property>
! v3 J1 Y; _& x2 Q# I6 h7 b <name>dfs.balancer.kerberos.principal</name>
) G; N) h1 h6 `4 l8 G: o <value></value>
4 \) ]/ \* m0 m1 G5 r <description>
4 m- U# i3 t% {3 e The Balancer principal. This is typically set to
1 j, G+ a; U" S/ c balancer/_HOST@REALM.TLD. The Balancer will substitute _HOST with its1 y" _+ }/ c5 R
own fully qualified hostname at startup. The _HOST placeholder3 P5 X* o6 ?) A1 }
allows using the same configuration setting on different servers.7 e. B6 _- |9 B3 `9 A, l) l
Keytab based login can be enabled with dfs.balancer.keytab.enabled. }, K3 C/ U" ]0 h- [3 X
</description>
- e, {$ D1 s( W4 R) T</property>: a4 Q, k1 @& b, F2 ^. r+ Z9 P( x0 i
<property>
) L0 x v6 h: W+ D3 }" ~ <name>dfs.http.client.retry.policy.enabled</name>
3 @( E1 ]7 U6 U2 \% q5 T2 F+ y. w <value>false</value>! K1 C( l' p y5 a) }$ b
<description>
7 W+ q j# I U! E If "true", enable the retry policy of WebHDFS client.4 I4 L9 z8 n5 e) b) H3 e
If "false", retry policy is turned off.
$ S2 @( k: m3 {* d7 B5 X Enabling the retry policy can be quite useful while using WebHDFS to% V& z' Q$ A' [8 G1 d
copy large files between clusters that could timeout, or
5 h' d' W; d' R8 x copy files between HA clusters that could failover during the copy.6 H" `/ J. R; t. W
</description>
' \1 `9 P5 f" h. q0 X</property>2 f+ M% L8 ]$ n. e O# A' Y
<property>
! ]" @9 L. U0 w: B9 Y <name>dfs.http.client.retry.policy.spec</name>
* g" q% R" P) ]- f5 Z4 B <value>10000,6,60000,10</value>" d; t- J/ p7 U6 Y
<description>
7 T5 [8 e) J$ S/ r2 |# E Specify a policy of multiple linear random retry for WebHDFS client,
$ z' a5 K0 A" D: X# A5 M6 u8 _% S e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1),
! l/ I! J1 w3 q, }3 H0 ^8 p ..., the first n0 retries sleep t0 milliseconds on average,
( U6 P; j D0 U$ g1 a3 d the following n1 retries sleep t1 milliseconds on average, and so on.
' h2 b5 P4 F ~2 t$ E </description>, l- i+ m# I" m4 t
</property>
+ e8 p4 H, v5 U2 o0 z C* j<property>4 J. c* V6 k! E* Y" W
<name>dfs.http.client.failover.max.attempts</name>
! Q1 O% S( W+ P% R <value>15</value>
+ E! I4 U) c6 T+ L3 } <description>/ ^& V9 t" D6 ?0 m: {- I; a
Specify the max number of failover attempts for WebHDFS client1 E M/ X) S" ^5 b5 c
in case of network exception.
* y0 E- e8 g8 g+ S% o4 } </description>7 n$ @9 N' m8 j' I7 a7 `
</property>
# Y8 M8 D2 z2 ]<property>
( x% j) v6 j/ ~0 E% L* X <name>dfs.http.client.retry.max.attempts</name>" n. v& C( z/ a% j( l
<value>10</value>, |: Y0 T( W. W" G2 ]4 I
<description> Q* }, D* l2 O% b0 b
Specify the max number of retry attempts for WebHDFS client,
$ v+ b* i% C/ \+ k2 H( X# l if the difference between retried attempts and failovered attempts is7 ?4 e5 ?! k& X9 o
larger than the max number of retry attempts, there will be no more
3 O5 {: W! A# `+ b. P retries.9 }0 a9 ]8 Z/ y" r3 O1 N' W
</description>: V$ d2 G8 F' H+ Y: b
</property>( U2 Q( Z9 Y$ S ~) U
<property>
3 ^+ D* ]& L/ ]2 l <name>dfs.http.client.failover.sleep.base.millis</name>
, }9 d; b7 u; O2 I7 ? <value>500</value>
; T# X, Q: {! Z; `& g <description>
2 x i e6 F& n3 e Specify the base amount of time in milliseconds upon which the" l1 F$ M! ]- m
exponentially increased sleep time between retries or failovers
6 ^& r: @. K8 m6 |0 ? r, v is calculated for WebHDFS client., c* n, ~# G' @' P. ]
</description>
$ L% S+ L1 [# k! o; Y</property>+ j8 K( ]" s5 G; J) I( J' R
<property>
5 T/ _5 \2 a7 u' o$ E2 S <name>dfs.http.client.failover.sleep.max.millis</name>; a: {& \1 i8 d3 r6 t4 D9 P# j
<value>15000</value>
. k0 y7 y6 z6 B' o4 ~ <description>
+ D8 h1 g% l; P3 Q3 K6 v) v8 f! z Specify the upper bound of sleep time in milliseconds between/ ^% q% S H: `0 ~. W
retries or failovers for WebHDFS client.
7 N& }! \; ], C% q. ?( b, V/ L </description>
* y5 k+ G D3 C: _# p</property>
5 {' U6 M0 z2 V- J6 K; D<property>
) d' T" d! }# g) @; x6 f <name>dfs.namenode.hosts.provider.classname</name>
6 v2 D2 {: @/ J/ y2 H4 F <value>org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager</value>7 B1 u3 {) E, \0 ]; g* L3 c l$ g
<description>
7 n7 O; h9 y5 }- L2 S# s1 x* r The class that provides access for host files.
) _0 k; Q6 D7 A) K, ]. K% ], q$ R! P org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used2 T j9 t' v6 Z' @
by default which loads files specified by dfs.hosts and dfs.hosts.exclude.2 [ @+ D \) T( [
If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is
& K* c4 j* `' x$ a! G3 S1 a used, it will load the JSON file defined in dfs.hosts.
9 Y( G, V2 s6 i To change class name, nn restart is required. "dfsadmin -refreshNodes" only' V3 g! F! M; k+ s
refreshes the configuration files used by the class.
6 R$ J5 J7 A/ ~+ r; ] C </description>7 y% n6 [* p! L) S
</property>
; ^1 O/ }3 O7 S; r<property>
; C- c! w' j* M& b+ k- [; O& ? <name>datanode.https.port</name>
- F: T1 K& ?, h! D1 M c <value>50475</value>& z4 i# B3 i* Y# [
<description>! a( c. H" H9 N7 V% g: y
HTTPS port for DataNode.
$ z. h: S( o. r8 ?4 D% O" w! D P3 l </description>
[9 b0 y% v& b4 n</property>
7 a1 s+ f1 t% F9 g; q* e5 k<property>
+ S# @, x% h- H# W+ ? <name>dfs.balancer.dispatcherThreads</name>% o8 O# X$ w+ D6 C2 o
<value>200</value>
, @5 I$ M2 T/ d0 k- P1 x: [; k( c <description>
9 [/ X4 H% n9 p) k) S Size of the thread pool for the HDFS balancer block mover.
0 n6 L0 H1 W4 X8 p* i: S0 S" | dispatchExecutor
1 D, _" f$ n2 Q </description> @! P+ A" l! U' ^
</property>% T8 {2 I. |2 g1 P. A1 x; V
<property>
+ |0 Z& v0 B0 P0 d% [ <name>dfs.balancer.movedWinWidth</name>
9 t! \, E1 }$ q M. G <value>5400000</value>- j9 L" h2 r- ?
<description>
, L! H% e) D& ~' [) [- b/ _: a Window of time in ms for the HDFS balancer tracking blocks and its/ e4 h; d( M1 d% b4 r/ E E8 p
locations.% l" e/ v$ e7 D: i8 h/ x
</description>6 r% z% ^8 } e0 |( c# A
</property>- R% d/ A* {3 a" F$ P+ V0 W
<property>; D- ~1 _* l) @+ {' R
<name>dfs.balancer.moverThreads</name>
+ u( S" x" V* J. P <value>1000</value>; z3 r0 b2 g% `; F9 M
<description>1 N" I* y5 l3 o% A; S
Thread pool size for executing block moves.
' M4 n, g2 e4 R w moverThreadAllocator& Y. C X7 ]; F( C' P1 P
</description>* ~4 U/ m S/ p- z6 K3 l
</property>
; v. v5 g' B' N& \# a. n- z<property>
8 a% a, i$ {2 K9 i0 I# K <name>dfs.balancer.max-size-to-move</name>5 J5 H+ k; a# c; k; |; M% Y' J
<value>10737418240</value>8 F# e) x) }/ M$ J( Z7 [6 G" y
<description>2 b Z9 W! W k O5 \3 q
Maximum number of bytes that can be moved by the balancer in a single: z; y* X- W' N" R7 B
thread.) Q" u5 C7 q4 R. z0 o9 i
</description>8 y C T# g7 w9 U
</property>5 N. v8 z% P( y# _, A4 P R, e
<property>
# ?* J6 A" O5 u2 I <name>dfs.balancer.getBlocks.min-block-size</name>
F, T4 Q# z8 k, ^$ ? <value>10485760</value>
, E/ S9 z) D' ~- T7 ] <description>. l6 i) k1 n$ Z8 @& Z1 t0 b( J
Minimum block threshold size in bytes to ignore when fetching a source's
. i& G, m! I4 I2 L" w6 O block list. ~ Z7 N) M7 v* }# z
</description>
% T; Z- }4 R( F</property>
0 N# y+ k4 `; d& Y( A! [<property>
- x B# f' Q# O' {( W <name>dfs.balancer.getBlocks.size</name>& |, q2 |: F+ i) ?/ I5 ~; d1 k
<value>2147483648</value>
# d) G8 `/ Z" ~6 P3 O# {! v' Z <description>' e" H: Y" f X3 x$ w/ n% J
Total size in bytes of Datanode blocks to get when fetching a source's
, o- c" B/ E; ^8 V h x' x" Q t block list.
( @5 b1 w ^0 W+ ^; I </description>6 G1 |% U) x8 b! B: i
</property>
. d) }& K) o6 v: G& C<property>" P" R) g4 l) {3 f9 j
<name>dfs.balancer.block-move.timeout</name>- O0 b! T5 S, I5 ?/ B. c. a, p
<value>0</value>. Y5 e: Y' \3 n$ b2 s: A8 s( M
<description>
2 k5 m2 B( K& Y7 s5 i Maximum amount of time in milliseconds for a block to move. If this is set
( ~. y$ u( W6 s( P greater than 0, Balancer will stop waiting for a block move completion& n$ G O& T! |, X G& C* ?$ f* Y
after this time. In typical clusters, a 3 to 5 minute timeout is reasonable.* a* U! Y8 l% _! N; X
If timeout happens to a large proportion of block moves, this needs to be3 J0 o. h2 `% Y: T/ h5 f2 b
increased. It could also be that too much work is dispatched and many nodes
% v9 `$ }' |) z) X are constantly exceeding the bandwidth limit as a result. In that case,4 B; p. J) V; a0 `& p; v3 G
other balancer parameters might need to be adjusted.2 ^ m8 W! t: i/ F
It is disabled (0) by default.5 v2 Y& S# p/ o l& `
</description>
2 E# Q; h d/ l$ s9 o</property>
6 |4 O6 o7 g9 L<property>( B _9 A) S8 d
<name>dfs.balancer.max-no-move-interval</name>
* \7 \) t* S2 O4 z& h <value>60000</value># c, j$ K: Q5 v, I
<description># @9 x, T; v( g- ^8 a7 [
If this specified amount of time has elapsed and no block has been moved6 E/ I- k* C- d( G
out of a source DataNode, on more effort will be made to move blocks out of4 C1 r; Q U/ J" P4 I# O: l
this DataNode in the current Balancer iteration.8 I) r* J, H: a, \
</description>
5 V7 g- r; Z3 U- ^# h1 p</property>$ `4 T4 z5 ?* E
<property>0 z2 f H& m+ w: ?; Z- |
<name>dfs.balancer.max-iteration-time</name>. l0 R$ H5 Q3 @
<value>1200000</value>& I" T0 }7 { [$ j$ H
<description>$ v( k4 j/ j! ~# ]
Maximum amount of time while an iteration can be run by the Balancer. After
4 q) {3 j% S B this time the Balancer will stop the iteration, and reevaluate the work
, W- O4 `/ E, n# Y9 ~ needs to be done to Balance the cluster. The default value is 20 minutes.
$ ]( P1 J# f9 Q/ j; n P2 ^ </description>( U% l* @5 v5 r( X8 e3 J7 x
</property>
) ]4 K9 {$ F& V: @$ E5 E3 L<property>8 w- \" f- Q" X/ m' n. }; x- D
<name>dfs.block.invalidate.limit</name>
6 _- @1 N, I% O3 Z <value>1000</value>
3 y! v2 ]! D8 i; m r <description>
! s+ U9 c% l' p. g7 a, z+ \ The maximum number of invalidate blocks sent by namenode to a datanode
; h0 V' C3 h8 [ per heartbeat deletion command. This property works with2 ^3 y3 Z: m: U% L
"dfs.namenode.invalidate.work.pct.per.iteration" to throttle block
$ X4 N$ g( X+ H5 S/ O5 s8 P* y, e deletions.
# D" y* R" D8 x </description>, M( w0 T! W8 ?
</property>
. E) j: l1 ]* h<property>) F; V I7 G# b% F( A9 d0 m
<name>dfs.block.misreplication.processing.limit</name>, W2 c+ Q7 N6 s/ B
<value>10000</value>
, O C# u; b! ^# V <description>7 m! S) p L# r
Maximum number of blocks to process for initializing replication queues.
; ]1 T$ e% `9 d$ B# i </description>
; C3 y4 h' C2 Z t2 M+ t1 D</property>% D# e2 _2 \ g, @
<property>
3 T* @6 ]* v7 H1 B- g. Z <name>dfs.block.placement.ec.classname</name>
7 t; {; T' u4 ~. F$ M: ^ <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant</value>
: t5 B0 ~2 o1 o: P: T <description>
9 r- o- `2 {7 F# c* s8 P+ z Placement policy class for striped files.2 V d% w& i! f/ L+ @, I$ w
Defaults to BlockPlacementPolicyRackFaultTolerant.class9 R6 ? ~, A: B! I6 w v* G5 P$ D
</description>- f) @' K: M7 N6 d
</property>
, s7 a5 x5 Q: d! a8 F" X6 ?<property>& z0 B7 w4 s1 Y) S/ Y7 B
<name>dfs.block.replicator.classname</name>
( O* W, ?) c- l- U- E' Y <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault</value>
$ B$ I+ l( W. z <description>2 D$ O1 z$ I c& i/ g8 y
Class representing block placement policy for non-striped files.3 l( j* @+ A P$ m' B* o6 Z
There are four block placement policies currently being supported:
" P1 O! | V- U6 Y$ o, h BlockPlacementPolicyDefault, BlockPlacementPolicyWithNodeGroup,1 [! j6 r, h* m4 B
BlockPlacementPolicyRackFaultTolerant and BlockPlacementPolicyWithUpgradeDomain.
u# z4 Q6 Z1 a6 D BlockPlacementPolicyDefault chooses the desired number of targets
2 L6 @% \9 w& B; l0 g$ J- n1 p for placing block replicas in a default way. BlockPlacementPolicyWithNodeGroup5 W# a7 h! L2 @ K+ L! h3 i. y* l5 y
places block replicas on environment with node-group layer. BlockPlacementPolicyRackFaultTolerant9 T( j/ z8 ]7 D! J
places the replicas to more racks.
; j$ N1 T6 v. u8 o$ X; I: d BlockPlacementPolicyWithUpgradeDomain places block replicas that honors upgrade domain policy.
9 U) ]1 c- N. i* R7 A$ r6 R7 T- N The details of placing replicas are documented in the javadoc of the corresponding policy classes.& E0 z( f4 s2 O# @( V% Y
The default policy is BlockPlacementPolicyDefault, and the corresponding class is
" H- T( c2 q; x: D4 Q* r# Z org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
7 A0 M: Q" B! ?3 v% O$ b- g G </description>
% i( q( \0 `& A& b7 I+ I</property>
* M& B1 q2 j5 \* F<property>
9 i& ~0 I! L# Y, k" Q <name>dfs.blockreport.incremental.intervalMsec</name> G8 X9 _# E+ ]8 \. X; x
<value>0</value>0 G# n% @* O H2 ]0 l
<description>3 @0 e1 b6 s. L
If set to a positive integer, the value in ms to wait between sending
& m: ]7 ^# v+ I& Y5 v incremental block reports from the Datanode to the Namenode.* x1 b7 D$ J$ N0 [5 Y
</description>, J" {& ]& x7 Z! e8 i% P2 \9 s! N7 k
</property>
' Y6 d7 _% P! j: W<property> C, [' r i! t/ z7 ~
<name>dfs.checksum.type</name>
/ X1 _+ Q' v6 ]% I0 G7 E" w <value>CRC32C</value>1 h! s# h8 X# \1 ^
<description>
! r6 A" w. l" R6 ^8 v! o Checksum type" f; S% ^0 Y$ R7 g
</description>
( O! v' K" w0 F* E3 `</property>5 l: p9 _( {8 c* b8 d. ~$ L
<property>
0 [, B, V! X2 v. Z8 ` <name>dfs.checksum.combine.mode</name>
/ q W( h( N! w: C- O <value>MD5MD5CRC</value>% k( G/ Q0 l) ?4 b+ p
<description>
9 R& U0 R8 |# i0 ~6 @5 e Defines how lower-level chunk/block checksums are combined into file-level
" b, b6 f8 U$ U- J3 D* H% { checksums; the original MD5MD5CRC mode is not comparable between files, c. `+ W" z, D" w2 k
with different block layouts, while modes like COMPOSITE_CRC are
9 P. g: ?+ Z) U1 j comparable independently of block layout.4 d; ]7 ~. a* J
</description>
4 g- m. K3 G" y6 I+ Z4 M' o</property>, d8 }# Z) W: l) ]
<property>% y B- T" b2 m
<name>dfs.client.block.write.locateFollowingBlock.retries</name>
3 _1 p) n+ R1 l. E3 t/ n7 O* J <value>5</value>0 f2 @- c) `4 \
<description>
" d2 R( p/ e7 d. M+ I$ U& C Number of retries to use when finding the next block during HDFS writes.
8 D/ `' S7 }+ X4 g P- ^5 d; n4 V </description>+ V, q, m: }/ k& y, C* j2 G
</property>
2 m R9 k. ]) ]8 a6 J<property>
- j8 T8 R9 ]0 x7 F+ S$ G) c <name>dfs.client.failover.proxy.provider</name>
* }7 u. _) j1 _2 T$ l <value></value>
" C( x, N4 _6 v <description>
/ r2 Y0 m. D) z& d6 l. X The prefix (plus a required nameservice ID) for the class name of the) g. _) k2 u/ X
configured Failover proxy provider for the host. For more detailed
: O) \6 W$ X; O( s& _0 h information, please consult the "Configuration Details" section of
2 i( e p+ [& p3 a! P the HDFS High Availability documentation.* l/ q# \; @, j+ D( ` @( d( W& s* g
</description>
5 U+ r ~* j7 i" A2 X: a</property>
& p/ \5 ?: x; F% p1 W<property>
0 g8 J {5 U2 M1 a5 Z" d <name>dfs.client.failover.random.order</name>
3 S. A2 L' H% w V! d <value>false</value>( T3 I* R5 W- Q& v
<description>
3 b7 x$ r1 j5 z Determines if the failover proxies are picked in random order instead of the
4 \% V/ z1 n/ a9 I0 w configured order. The prefix can be used with an optional nameservice ID
% g7 `6 W2 H1 v! x3 |# }! T (of form dfs.client.failover.random.order[.nameservice]) in case multiple# d p5 E7 Z- X5 [( d" D
nameservices exist and random order should be enabled for specific
1 O& q" u4 y1 q3 r4 [ L nameservices.
& k( d3 H( V7 O$ ?4 p+ O1 A: Z: ] </description>
6 \) {9 ?- k3 ~* M) I w0 I$ A& t# o</property>
) l% {. t9 U" d2 |1 R9 k. m<property>
6 m, [5 C. y4 C! Q0 k <name>dfs.client.key.provider.cache.expiry</name>. | p: @/ R w: W6 o+ l! ^9 A
<value>864000000</value>1 _6 b* P! M# i# c& z! k
<description>8 |5 s8 A; L& k8 _+ F
DFS client security key cache expiration in milliseconds.9 ^) c: f2 |3 \' \
</description>
* z5 F& R! [4 ~- u9 u5 J7 K</property># j. B% j. @+ }! I! P
<property>8 r" V" c6 T0 ~* h' _
<name>dfs.client.max.block.acquire.failures</name>0 s3 b, a6 i$ l- W- _+ j( E% U
<value>3</value>2 Q2 g0 J! R) Q; J5 j, I
<description>
, y# }/ \, J1 W1 J2 I Maximum failures allowed when trying to get block information from a specific datanode.; ~# Z" @- @) t# S8 v5 i6 i4 _' Y
</description>/ R0 Y8 e) C2 N8 ]* d D7 b
</property>2 m4 ?+ r0 M9 l& v# h2 p( G
<property>
( }. r" U. g+ S: D1 ]5 H% s7 _( C4 U9 c <name>dfs.client.read.prefetch.size</name>/ ]- H7 s5 t( P! G8 ?
<value></value>
. W& g; g( }; A/ ^* C r6 z# \) p+ x/ T9 S <description>! T1 _+ }2 B: ~' p& g! O0 }
The number of bytes for the DFSClient will fetch from the Namenode
9 I9 C; _3 z1 c d4 g7 p! ?! J during a read operation. Defaults to 10 * ${dfs.blocksize}.5 [3 \) S; `7 r0 G" H% y
</description>
* q+ H; q( p; `: O9 b</property>
' S# V( y: r7 r+ ?! m( X. p* l<property>
, ~7 A% L2 l) K, A- {: k( w4 w8 \ <name>dfs.client.read.short.circuit.replica.stale.threshold.ms</name>
+ w! r2 K! ~4 Y- \" y/ z <value>1800000</value>
6 V7 B- g. {6 H" K <description>
3 t6 z3 ^1 [/ r( C) R Threshold in milliseconds for read entries during short-circuit local reads.
7 Y, ~- Z& H5 ~+ f- |! A </description>
6 W6 n( a- `: @8 N" K+ R/ N+ G2 w+ b</property>
8 ?" h+ Y: R8 Q; @; ~, o! d2 I# y0 ]( `! \<property># e; W$ T7 [5 \, _ j6 u* g, T
<name>dfs.client.read.shortcircuit.buffer.size</name>
# n( q. O$ @$ t; M9 m: ^7 Z' ? <value>1048576</value>8 [! q8 h7 O7 a
<description># }) c7 u ^. V6 ]! r5 q4 z$ g
Buffer size in bytes for short-circuit local reads.1 e, R; G) f) X. Y5 C$ b2 }
</description>, p( U3 e* U& x# R# U( ^
</property>8 ^. _, {' ]4 M0 L" ?5 y, [0 Z- O
<property>
2 Q" `5 i. d( G$ @1 B& F" D7 ], k <name>dfs.client.read.striped.threadpool.size</name># w' A" g/ a3 X2 x7 U, g% W
<value>18</value>. p: Y9 c9 T; M1 ?" N$ ?
<description>* u7 d5 i9 ]. `8 ^% y( b5 m2 F6 K
The maximum number of threads used for parallel reading5 g/ ] `: k; r1 j
in striped layout.3 I5 J$ X4 }4 f0 W4 M# \! F
</description>
) y z3 A: \% ~4 \1 c; q3 g/ T</property>
* Y- d0 `3 k1 g<property>3 C7 Z1 w. y: J
<name>dfs.client.replica.accessor.builder.classes</name>! J0 E. w6 J# ~1 V5 u
<value></value>
- m/ A S' C% F8 D! o0 u <description>, ~7 P {: o. u( b
Comma-separated classes for building ReplicaAccessor. If the classes; c5 i/ t' t/ R6 d) [
are specified, client will use external BlockReader that uses the
: u5 v) F! S" Q# M* {$ n ReplicaAccessor built by the builder.
, N" e* t! [( w/ ^" [ </description>
0 L$ k3 ~8 B- l" S K</property>7 _! u! f) ?' a
<property>
8 q6 b h3 N% g: X. v! J <name>dfs.client.retry.interval-ms.get-last-block-length</name>
1 e" j8 Q6 g" |# ~' q% {; ~: E/ A <value>4000</value>/ e4 W- A% K3 {: t3 M3 y' b6 X
<description>
! A" B: Z! U1 p( \- F Retry interval in milliseconds to wait between retries in getting
$ {* S+ i- F3 u o5 W block lengths from the datanodes.
$ _* X5 }; L5 l+ o z0 d </description>7 P* s% {; N: d6 H7 @7 F7 z
</property>" [: F( m2 \5 z* g/ n9 v% K7 V
<property>" X/ b: W" J/ W
<name>dfs.client.retry.max.attempts</name>* R- B5 s1 A/ S3 `$ q6 G( |: y
<value>10</value>" G# X2 k; V7 c" J. M7 I' C
<description>6 n6 v+ s7 f. [9 s
Max retry attempts for DFSClient talking to namenodes.
: l/ n( c; t: A& o v4 ?2 M </description>; X; u* E$ ]: L. P: }# U( O2 M
</property>
. J$ s8 ^% y! b' X. I" g<property>
, p+ a6 Q- }% f( d* w! { <name>dfs.client.retry.policy.enabled</name>
# _+ E3 P3 K0 D1 V5 Q <value>false</value>
5 G; }# r! S% g3 j. W/ k% S <description>
7 |+ c' u4 e4 J K. X If true, turns on DFSClient retry policy.
6 |. G5 x; Y; J* [9 s; y </description>
, J8 A; w7 h8 Q( n</property>
% M7 k S3 M, u" [4 x" U<property>
! u# v. y4 e s9 m <name>dfs.client.retry.policy.spec</name>
' X/ Q& `6 |! s- ~5 Z) L) g <value>10000,6,60000,10</value> B5 m. s1 c& u3 T1 A# l Y. N" k
<description>
) W- v! D7 x% M Y Set to pairs of timeouts and retries for DFSClient.
! e6 }0 z6 T! g& k </description>0 [' n! I( ]) A2 W- J7 D# _
</property>
* O/ I/ F6 ^5 \3 O1 L! R<property>
( W3 O! O6 M, g' K, { <name>dfs.client.retry.times.get-last-block-length</name>1 t% m% f( C) M
<value>3</value>
7 e% S; Y4 w; W; i# s a& }$ }) Q <description>
* L# ]0 L: \' n# Q7 X& s, R/ U6 F2 y Number of retries for calls to fetchLocatedBlocksAndGetLastBlockLength().
3 J) r* c" a9 k- e </description>5 w L9 ]8 I' ^7 P0 E
</property>
2 a5 Y/ _6 M/ y% k1 }9 J0 C<property>
0 J, d- Q+ R4 w% Z <name>dfs.client.retry.window.base</name>
4 D1 V& V" t0 X! X2 e% e8 A3 G <value>3000</value>
. T$ C1 h& C/ |4 |& r% m <description>; C: E+ m8 k. _6 M2 H# {
Base time window in ms for DFSClient retries. For each retry attempt,) |+ \9 C- f* z( _/ N% G8 ]& h( ]9 y
this value is extended linearly (e.g. 3000 ms for first attempt and! r3 d3 Y+ p* A/ v2 _+ H2 B
first retry, 6000 ms for second retry, 9000 ms for third retry, etc.).
) I5 k& Z2 M% ^ </description>
2 ` y& F3 N6 }2 L9 L ^0 v</property>
$ I" Q' s+ e, H$ w* ~<property>
! S9 V( N8 a2 y <name>dfs.client.socket-timeout</name> r; z1 ?! r7 ]) ?& s3 L$ ?4 I
<value>60000</value>$ Z% H, l8 K/ ]1 M
<description>
4 r1 m; e" G3 [9 A6 s4 u6 ~ Default timeout value in milliseconds for all sockets.
1 o4 _% o, `8 R3 `. z* J" T8 t </description>8 D6 ?5 @1 k8 m; }" e
</property>3 U5 Q# |% r5 t' |7 v/ B' V
<property>+ N- m& ~ c- H, V% B4 x4 q, c
<name>dfs.client.socketcache.capacity</name>
5 |4 t7 V( Q, i <value>16</value>
: _' j! J$ Y6 w' Y <description>
: D* b9 M; h" c Socket cache capacity (in entries) for short-circuit reads.
4 y' L- F6 p: G, W G7 B$ x& r$ Q </description># m6 S: }2 z6 k& O5 W; {
</property>
! C# _1 W b2 a) a9 p$ f9 G% ]<property>
% i& A; u0 ^7 g8 s <name>dfs.client.socketcache.expiryMsec</name>4 i2 v2 t! j! W- L3 Z" b9 i3 S( E( k
<value>3000</value>
8 R. L- x3 j+ [! `- b0 `2 ] <description>
' _. V# A8 S! A j$ a A4 Q Socket cache expiration for short-circuit reads in msec.
, t8 n4 @; }+ R! ^ </description>
; V) @! B2 A4 v o! S</property>. t8 v* B, a) n; X5 [" L4 w1 K
<property>6 n7 K$ O; Z( I) Q$ ~; ?+ u0 d+ I I
<name>dfs.client.test.drop.namenode.response.number</name>2 E) r# c% r W. \+ Q
<value>0</value>
2 d- E2 m! T6 O) K8 | <description>- |$ J6 m9 f* C2 i7 q* }- Y: Y d
The number of Namenode responses dropped by DFSClient for each RPC call. Used' b" `, g$ c) L0 g$ D8 _ Z' ?
for testing the NN retry cache.
3 J y/ i8 e, Q" W </description>
0 W9 j' R2 |; y1 C</property>
! j+ D$ |: s+ N9 R/ k3 O<property>4 Y7 W0 P; n3 \* o% v, H' s; E; \
<name>dfs.client.hedged.read.threadpool.size</name>( q' u) f8 V1 o1 x/ p/ U
<value>0</value>& x; X& l- P" [
<description>+ `* y2 ^ u( p. O
Support 'hedged' reads in DFSClient. To enable this feature, set the parameter# N# {% B. \, c: Z& Y6 S
to a positive number. The threadpool size is how many threads to dedicate* K+ |( T& U- P8 w* N3 D7 R
to the running of these 'hedged', concurrent reads in your client., r8 h/ {6 ~8 O, {, m, H; I) Q! z
</description>
% b. j: j* k4 m, Z* {' U</property>' M% z% h- ~1 ?* B9 s- E' V' u/ \
<property>
N1 o& e; e( z( h <name>dfs.client.hedged.read.threshold.millis</name>; F+ j$ ^: `# R( U* w3 i
<value>500</value>: f1 B1 q7 c( J( C
<description>( c$ U! u+ t6 Z
Configure 'hedged' reads in DFSClient. This is the number of milliseconds+ e, B% _" [- x7 m1 {, z5 F% `/ s
to wait before starting up a 'hedged' read.
# Y* O7 Z# @! S( E9 ~7 x </description>
$ s; o( B/ k: Z$ A</property>
. r+ {7 w |" h! k<property>& k: Y2 V# }. [* ~' d' G, E
<name>dfs.client.write.byte-array-manager.count-limit</name>; K/ k4 O) P, A/ d: ?( d% T
<value>2048</value>8 {* |' y7 C6 i9 k
<description>' b2 M& z4 _6 m9 V3 n* M
The maximum number of arrays allowed for each array length.
9 |- h. ?# _2 l0 \3 G </description>
% S, X# Q. Y# m) L) H</property>
$ u% ?$ T; |4 b- J6 e<property>1 `% [* ^7 p. \" U# ^
<name>dfs.client.write.byte-array-manager.count-reset-time-period-ms</name>
$ a% [6 Z6 ^) f6 O% K$ k <value>10000</value>
9 T* T; A& t0 L: e <description>3 K {" g0 n% N9 n( e( w
The time period in milliseconds that the allocation count for each array length is) o# W6 `3 Q7 I! F$ ?
reset to zero if there is no increment.- \* d; }# ?4 s: S% g) d
</description>% {" }# B* [) x, U
</property>
$ T6 R F! i. c+ T<property>
7 d# Y4 ?$ T% S- C1 h+ v <name>dfs.client.write.byte-array-manager.count-threshold</name>& ~- f( M: h. ]3 @
<value>128</value>
( H/ G, g# _: ~; Z' s) E/ o( |5 I! y <description>
& Q& x* ^: k0 I/ H, v1 J4 v The count threshold for each array length so that a manager is created only after the
( q" b0 E$ t+ w- G0 ^( m+ T/ e allocation count exceeds the threshold. In other words, the particular array length! S% e& I' l. l- H1 R: n0 N( h" G
is not managed until the allocation count exceeds the threshold.- R; \+ I9 i, U" a+ O
</description>& p' R) K3 }* p5 b. _/ D/ S
</property>
* _3 O1 e7 H( R<property>
" w1 m3 u) D9 V" I <name>dfs.client.write.byte-array-manager.enabled</name>( [+ J5 t& ]# {" n0 h& }
<value>false</value>
4 E V O& l |- o <description>
, K5 a# [0 ^9 g5 z/ G( E If true, enables byte array manager used by DFSOutputStream.
" V0 {5 R0 E8 J6 k </description>1 X8 `' U/ _2 e/ E' @
</property>& _3 \: L/ a: C
<property>
/ T& }- L7 r* S <name>dfs.client.write.max-packets-in-flight</name>
3 z- q8 \ H1 E/ X: N <value>80</value>2 B8 M' @5 c* `1 A# G, {/ k$ {
<description>+ I" e# D& _. a* }( ~+ o, z
The maximum number of DFSPackets allowed in flight.9 G3 ^( S+ u/ p2 H
</description>" T! d. |4 v. ?7 p* r
</property>
/ g* c* K, ?3 `9 c" Q+ _6 s<property>7 X |0 h0 B5 ]4 }8 M. y
<name>dfs.content-summary.limit</name>
9 [; h' P1 N. B5 H. V <value>5000</value>
2 z4 V' f, D3 ] <description>
S a7 F. t) q' k" h( S4 a The maximum content summary counts allowed in one locking period. 0 or a negative number/ F: o5 \2 }- d' o
means no limit (i.e. no yielding).
# l2 f. g* R" x </description> h1 o7 F6 B0 n: Q4 O% D
</property>
% U+ ~3 `3 T/ K5 b' r& {1 Y<property>2 h, A, ?- l1 `+ s
<name>dfs.content-summary.sleep-microsec</name>7 i. j7 j- ?5 v. ?9 F3 t8 @
<value>500</value>$ K6 A+ {4 Q. i0 q) o& I! l% |0 l
<description>
% H/ H9 M# G7 ]; Y p# j7 y The length of time in microseconds to put the thread to sleep, between reaquiring the locks2 g4 h# \4 }1 T- w4 Q0 x' w5 N$ t
in content summary computation.
8 w- u {4 k( W; |$ i& J </description>
G- u3 t) m U. i6 U& H* E</property>
) M( Q, `; d6 n( a$ m<property>1 h8 h; s* O. i7 @6 R) d9 u: l" ]
<name>dfs.data.transfer.client.tcpnodelay</name>
8 C; T# h# _8 K+ E) r <value>true</value>
- y B j5 H. r- B6 L4 ~ <description>: E( g: U$ H8 H! S# ~
If true, set TCP_NODELAY to sockets for transferring data from DFS client.5 P1 C I9 v- j
</description>
# M' b2 E( G0 ?- e0 {4 K</property>- P2 w5 T+ Z3 d: b `
<property>
5 [" X- i) |, X* ~ <name>dfs.data.transfer.server.tcpnodelay</name># v% s A2 t7 k
<value>true</value>( O8 d4 s& c$ F8 y% G e( z
<description>/ E5 Y e3 ^) w" D$ `; b& `# O
If true, set TCP_NODELAY to sockets for transferring data between Datanodes.* z$ d0 x) t2 e9 V/ f$ @
</description>" p8 I: r; [- Z$ A
</property>" N! i! w+ Y4 P, E4 ?) Q% K
<property>* @2 z6 }6 X K3 ~
<name>dfs.datanode.balance.max.concurrent.moves</name>+ ?) o, Q) j# M) l
<value>50</value>! a0 [0 Z: P! o% g& y
<description>
* v1 G( q# Y: y8 v" G U Maximum number of threads for Datanode balancer pending moves. This
) S7 t3 v2 A- `2 \( U' A value is reconfigurable via the "dfsadmin -reconfig" command.
3 E B7 c: y1 Q) A </description>
( d! C* v) o4 B: }# v0 D</property>
/ O1 x4 F& q7 {7 a/ k' d<property>
5 }1 W2 x' R, X# E <name>dfs.datanode.fsdataset.factory</name>
# W& k% r |4 O <value></value>
7 @' S1 T3 S9 u: T! B7 G <description>
+ }/ P# ?2 S; i) Y+ i& d5 e The class name for the underlying storage that stores replicas for a7 `; u( O' U3 T) D) _5 J4 T9 L
Datanode. Defaults to3 o# Y8 \" m' r! h' z% e) f5 e
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.
# K, b5 y. w: C1 i* h( l/ p$ w6 d </description>
9 w! d& h# ~, O6 W6 K* ?</property>$ ^/ k2 ~# n7 z1 h8 o% Z' j7 q
<property>* k3 l2 U& `( T% P4 T! U* y
<name>dfs.datanode.fsdataset.volume.choosing.policy</name>* A K) j: }: [- E B2 t% y
<value></value>
, r: ~5 }5 E9 W <description>
1 r5 d, J: y! D! N The class name of the policy for choosing volumes in the list of
, P/ `5 s' {6 b" ?. s4 j% Z directories. Defaults to
1 O7 @" @& U/ ^, Z3 p org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.' C$ r7 }7 y1 n% s* ]' @3 h! T) r9 n
If you would like to take into account available disk space, set the" z# e5 b) c" t! S# v
value to
, R, _7 V% f& `! b& t) S! ` "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
! [1 x* V' s9 b3 W* W, ^ </description>
+ \/ k7 O6 e) T! \6 C' W: O</property> k1 n" a8 H/ [ O0 S7 M
<property>6 }) @9 _# A! O
<name>dfs.datanode.hostname</name>
8 [6 R! ~8 G, H/ y <value></value>- c7 w6 G6 _; K* ^3 g1 M+ E
<description>/ I( j# h a- @" V1 Y6 F
Optional. The hostname for the Datanode containing this
: S% m% R% c+ N# u2 h+ i; Z configuration file. Will be different for each machine.
' G8 e/ _, i V' `0 U! M4 _" A Defaults to current hostname. t* h# j2 o6 B+ d- T
</description>
. `1 F# C- f& c5 B</property>- U9 t8 U7 \( N; w3 `) C3 m! D
<property>
8 ~7 V9 }5 O7 {$ E. ?) L, W7 z <name>dfs.datanode.lazywriter.interval.sec</name>) G! w4 n6 N9 \( V3 j3 b& _
<value>60</value>
3 x. H3 e9 g8 ~7 m% J1 U" l0 L <description>' J, `# R3 c9 x( C7 b( C- ]) \8 o
Interval in seconds for Datanodes for lazy persist writes." J0 @ ~. W/ v$ j
</description>2 r; o/ i# L' q8 N+ l9 O( ~+ V+ j r8 Y
</property>
! P8 `+ Z9 a/ j<property>/ ~0 I/ I5 \2 l0 c
<name>dfs.datanode.network.counts.cache.max.size</name>3 [( c- v+ i9 q; f4 A
<value>2147483647</value> P6 u; i) Q% e9 n" j7 ^% a
<description>5 ^# r# Z# U7 J0 I3 Z
The maximum number of entries the datanode per-host network error
3 }- \# W9 s; \ ~$ Y& Z" ] A count cache may contain.
1 u! z; C# h6 e- C; D4 f </description>
2 w3 |* p1 F& h& C! R1 I</property>
+ Z7 Z9 Q9 A% U- L$ X<property>
& g5 u. O2 [& N/ B$ e8 ^/ [ <name>dfs.datanode.oob.timeout-ms</name>4 X; w1 ]3 r/ P
<value>1500,0,0,0</value>3 k7 P: u8 I# @& J
<description>6 L% U5 A* ?& ?
Timeout value when sending OOB response for each OOB type, which are
( S" y2 B2 }0 n( M+ g& i, E) n OOB_RESTART, OOB_RESERVED1, OOB_RESERVED2, and OOB_RESERVED3,
" _0 h+ C4 y [0 t$ f3 j respectively. Currently, only OOB_RESTART is used.+ @% x, Q2 _: N3 M0 q5 c
</description>
. E0 C$ F: r6 H8 z5 t3 m/ M. Z/ r</property>
5 G" {# D- n' f d# Z<property>
$ b1 i; w b2 p, x& v0 f& A <name>dfs.datanode.parallel.volumes.load.threads.num</name>, ]' _3 }" O! l( q% H: z
<value></value>
. A6 b4 T3 p4 _* k, Q+ s <description>$ E6 ?, _& [* l$ G9 J
Maximum number of threads to use for upgrading data directories.
: s6 y* D$ x# u- v The default value is the number of storage directories in the3 V' J9 x+ Z+ d- y" K5 N
DataNode.$ d+ F, V9 |( x0 X' r
</description>" i- R1 ^, X* _: ?# j& B
</property>
4 |3 O' q1 h; S7 H# H5 G<property>
: T0 h& n- r; W8 X) N <name>dfs.datanode.ram.disk.replica.tracker</name>
6 l9 s6 z7 X& |3 p; F <value></value>
9 n8 \) B0 @! ?0 b y; F- @ <description>9 F/ N# Z- [6 l7 a
Name of the class implementing the RamDiskReplicaTracker interface.0 P2 e- {* e( I M
Defaults to
' Q( {+ C. e7 p: s. ]( O* x9 U org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker.
6 H5 T7 u$ H+ e/ u9 F o/ n </description>
6 o; x9 Z* G4 E( j% C</property>$ u- n8 p6 d+ f/ l9 j0 E
<property>1 {8 Y2 a# k% G( E$ G [
<name>dfs.datanode.restart.replica.expiration</name>5 _. o% ]2 _: a/ K& ^
<value>50</value>0 \5 n& o' G! i: y7 b/ R2 o9 D g
<description>% F, m& A, e6 J; d3 |
During shutdown for restart, the amount of time in seconds budgeted for6 r/ z2 ~- L' }& F; u, G6 b
datanode restart.4 @& j2 u) F$ p: R I$ [5 o' z1 ^5 L
</description>0 G; U" F) w" c6 ~" s, \
</property>* G4 w/ e$ U+ W
<property>* }% w* E8 Y7 E# ^8 d/ z
<name>dfs.datanode.socket.reuse.keepalive</name>
* e% J+ B& q; a/ Y. H <value>4000</value>* V7 L9 ^+ L" R' L
<description>8 {/ @$ Y/ T' a, C" c
The window of time in ms before the DataXceiver closes a socket for a& G- N! K# i( i g, T2 y3 p) f
single request. If a second request occurs within that window, the( f. W$ W5 j* E$ K0 Y8 G
socket can be reused.4 K& R! V' Q2 r2 k
</description>( w6 B7 _) V: Q: D8 ]6 W3 {: }
</property>
; u, @/ W0 ]3 G<property>
. r+ ~8 Q0 n z0 v+ P" g <name>dfs.datanode.socket.write.timeout</name>8 p: r% g: {* B5 r" @! I/ y9 `: {
<value>480000</value>( k7 q3 S6 E9 W# j
<description>
( U1 b1 g. n( M, w: Z! j Timeout in ms for clients socket writes to DataNodes.
" J1 [3 u( W- e& i: q </description>
( x. ?3 T# B) w, ]4 J) D. W</property>8 r. W& n% y% G8 t
<property>
f. ] R2 C7 {) S/ Q1 v: C$ R3 c <name>dfs.datanode.sync.behind.writes.in.background</name>
) C8 P/ }, ~( s" i# v% ? <value>false</value>* x7 a& c9 X. i# ^- {
<description>
7 B) \! h1 M3 ?" f1 o6 T! ~ If set to true, then sync_file_range() system call will occur
; N" T8 c+ R# P9 J) f9 T asynchronously. This property is only valid when the property
& v! _5 | M1 n( |9 A/ g3 i dfs.datanode.sync.behind.writes is true.! V- R" g/ p4 X- `+ N
</description>
! o- N+ k% D6 ^# m</property>9 v/ s! |' S# ?
<property>
9 k& }9 B; b$ r <name>dfs.datanode.transferTo.allowed</name>
3 d h2 J8 |: i <value>true</value>4 e1 @. W: r, |6 a3 b
<description>7 p. j: P! @7 Q" ?: B6 p
If false, break block transfers on 32-bit machines greater than3 V1 U2 S9 g/ a- g1 B) a5 W6 t
or equal to 2GB into smaller chunks.
1 q3 j) m! ]5 V% _9 S9 E </description>
. w1 k8 T3 b5 f& S</property>) F2 g4 |$ q. R: ~1 f5 ^
<property>
: C D" Z! ~9 x; G <name>dfs.ha.fencing.methods</name>0 i0 c. n: X8 c/ F( x
<value></value>
- m2 n/ ]" w8 G# u$ g U$ n6 B3 A <description>
' d6 [/ q$ a" L5 r8 y A list of scripts or Java classes which will be used to fence T2 P- x8 G& t% _% T: g5 R8 z* _
the Active NameNode during a failover. See the HDFS High
, Q4 s [1 T& i6 i6 q& R Availability documentation for details on automatic HA
# Q% d2 i/ z$ f: w configuration.
& e; Z# D* K* S& U8 B* p f7 f </description>
* g m' d: n' d R$ k& y+ c( g' f</property>* {+ x7 c" U. }& C$ u9 f
<property>3 T6 D5 Q' ]$ h- h8 F4 h8 u
<name>dfs.ha.standby.checkpoints</name>- }: J( o Y- K( N D
<value>true</value>
$ f* j) U& m. U <description>
8 m) s3 v* T6 Q2 p' k- [$ K If true, a NameNode in Standby state periodically takes a checkpoint
0 E& T% Q( K+ a4 T# m of the namespace, saves it to its local storage and then upload to' K7 p5 u2 }8 Q7 _/ ?8 ^$ G5 N% }% C
the remote NameNode.
9 c; c$ v) f8 i </description>) K# a( G6 G; P! \4 y P8 u
</property>
1 | @/ ^0 u, t, R) Z/ T, C<property>5 e: Q5 D8 @$ F; g5 a& L1 {
<name>dfs.ha.zkfc.port</name>
, D; e D% A, F8 X8 Q; p; D; v <value>8019</value>8 P& D+ L: G' ~$ L1 t- K0 y& }) r
<description>2 d" Y$ G% J* u. d
The port number that the zookeeper failover controller RPC' a2 h3 J; z D
server binds to.
" M8 s' d+ {) P, g4 x( h2 }; j6 h </description>
) h/ C: H* X" }5 T6 b6 c8 Q f</property>6 h" I3 W- @0 j5 D6 @' X* i
<property>
3 \( k0 E5 ~- f; U <name>dfs.journalnode.edits.dir</name>
# b" f) j+ S! \ <value>/tmp/hadoop/dfs/journalnode/</value>
# o5 L6 V% N: p* u/ v/ F2 {1 b" [% \ <description>
# c( O; X/ L( [+ B! N( S) G" { The directory where the journal edit files are stored.8 v& H' C5 y5 Y b6 N5 q
</description>
/ ?. I0 y, E7 @- S2 d& P5 V</property>
( K: l7 @" g" h9 q9 s<property>' H6 R* a" I) T
<name>dfs.journalnode.enable.sync</name>
8 O p( I4 Q' J <value>true</value>
, Q2 P) ~9 i9 L- c% @ <description>
& V, _! d! f: j1 v( X+ A If true, the journal nodes wil sync with each other. The journal nodes, p6 t6 Q! E( y0 ]* _7 Y
will periodically gossip with other journal nodes to compare edit log+ n. Y/ H, _: u$ w2 U) w
manifests and if they detect any missing log segment, they will download6 @/ V( o3 L* d7 `
it from the other journal nodes.7 @/ b$ {& h5 [8 }
</description>% `2 [2 \$ v# ?- b5 V4 M, x4 i
</property>4 E' ?9 {. p1 R
<property>4 |* @( \. a) G ^4 t
<name>dfs.journalnode.sync.interval</name>
& D. U2 j5 s, X# q- a9 u9 N" Z' z8 z. | <value>120000</value>
- z+ m) n0 {9 U$ b, O) x <description>7 P/ u F0 G. ~ K( F3 u4 c
Time interval, in milliseconds, between two Journal Node syncs.
8 p2 z2 L& N% |, s! y This configuration takes effect only if the journalnode sync is enabled
- v: C0 z" {6 Z3 F! l& x# @ by setting the configuration parameter dfs.journalnode.enable.sync to true., c5 @* b$ X; }! g" m8 g) l
</description>2 a$ a I# e% f: \3 i& |/ Z
</property>
& E- D6 }, v. o<property>
, N, R/ h+ E0 J$ R& \2 S <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
% G" C& X* ]! H# F' r% y* ` <value></value>
2 \' V) j, e; E7 k. ~ <description>
2 e6 H- H- j3 _3 j Kerberos SPNEGO principal name used by the journal node.
/ a, O2 Y3 M& S: H+ C7 e </description>( i, h6 w) u7 [* z7 o- k
</property>0 Q6 \" ?' i' J' d
<property>: k& P& w2 H0 A3 w6 [. |
<name>dfs.journalnode.kerberos.principal</name>8 W6 K; U5 \: c' t
<value></value>
+ a) M: m% v; l+ @& r( w <description> |0 F$ }# _1 l$ p1 \, P& I! [
Kerberos principal name for the journal node.
9 E. l' ]7 _8 u8 L7 M </description>; J5 m0 [* E# ]( H* t
</property>
" L+ K- Z1 m. y<property>( A- g; X! t8 s% u; n
<name>dfs.journalnode.keytab.file</name>; V9 ? ^! j& j9 t5 ]- d* J& P
<value></value>
5 U6 n$ z) Q5 e8 f+ R3 K. N; C- V <description>% o3 h' |+ C5 c! a! Q" W
Kerberos keytab file for the journal node.9 r8 l- q3 \5 f+ m
</description>2 m' @5 P; P! L
</property># U3 H. o6 {% b9 W
<property>7 R, a% `" |3 D6 H# L, v' W
<name>dfs.ls.limit</name>
; q3 S3 J) A' l0 z% i6 n <value>1000</value>
, x1 r( _" u" x4 r <description>
3 F) J& k: W7 m6 w Limit the number of files printed by ls. If less or equal to" P; m* H2 R. d, z: [. E" o
zero, at most DFS_LIST_LIMIT_DEFAULT (= 1000) will be printed.2 p# T* L& b. H6 A. J4 P
</description>1 d* B/ r, d3 G/ Z+ g
</property>
: D8 T+ k3 Z: p; ?7 V& |% l9 y" v- `<property>& l$ W3 Q: A" j* s5 d5 i) [, `, P% ~
<name>dfs.mover.movedWinWidth</name>! d# N$ Q9 ~# l
<value>5400000</value>
. `+ d* B- K: p" r( U <description>
8 \. \6 N' J4 q) I5 r5 \4 n, R The minimum time interval, in milliseconds, that a block can be$ B m' N" G0 D. G
moved to another location again.
% T4 `. s( c* j) O/ y </description>, W3 X! Y- ]- J+ V
</property>7 m( X0 E' y" m) a
<property>
1 K% E2 Q2 J& ^, ~, p <name>dfs.mover.moverThreads</name>
; t( [& h' U5 r <value>1000</value>
M+ v. Z0 U8 \ O6 n. _( n, k7 N <description>
$ l9 }# E% u3 u Configure the balancer's mover thread pool size.
& \, Z2 b8 }/ C$ R2 G, x9 |6 T </description>1 H L7 n/ g/ z0 \; C! h7 s
</property>
8 N$ W4 m) Q4 L, w; k3 s# U: K S<property>
p3 {. l$ D. B9 o B <name>dfs.mover.retry.max.attempts</name>- }0 `! i3 M R- x/ x
<value>10</value>
& n# x6 p& k" K$ p. Q# C# i2 H <description>
8 q$ R, W7 o* A/ I The maximum number of retries before the mover consider the. b+ d0 X; ]- B$ N2 k
move failed.
( ^( q" F# V) q; b) i </description>3 s: Z) W) ?1 e0 q# ^( V5 T
</property>+ w- f3 z5 H2 c" R3 v; z( D; M
<property>; p3 ?' K. L2 O$ m0 k5 s+ U
<name>dfs.mover.keytab.enabled</name>
5 P5 s6 a N$ u <value>false</value>7 x1 X8 e* B* C& ~1 Y
<description>6 N4 t f- k! {( ?
Set to true to enable login using a keytab for Kerberized Hadoop.
Y8 i: E# y* t) T' K </description>
; b/ ?- n% u- ?, U/ R" H) E</property>; y# V; }3 ?5 k p |8 l" u; n
<property>
' c$ |4 R, \% ?5 R <name>dfs.mover.address</name>6 l/ D2 N8 |" D* B. M( \
<value>0.0.0.0:0</value>
0 P! j7 d- }2 q <description>" o# }# H$ V. L, [+ y4 e
The hostname used for a keytab based Kerberos login. Keytab based login& i- C% W7 h% m' M% y
can be enabled with dfs.mover.keytab.enabled.1 B! s- k3 e2 N
</description>
4 u5 y. ^3 H' h3 t! G6 T% D</property>
1 F* x- Y( t/ ^! E2 O+ V1 a& k, J3 {<property>
! N# e6 u5 k [& Z& f z6 Z( d <name>dfs.mover.keytab.file</name>
) R8 R) V/ j/ M: d# q' T% U <value></value>
' S. ?0 y& ^7 M* n8 \* i3 G! s <description>
+ ?0 x0 ~. {% m' j& \7 C The keytab file used by the Mover to login as its
- V3 k" x' i5 b service principal. The principal name is configured with
. ~# `' k8 Z! a, G3 ?* g" V dfs.mover.kerberos.principal. Keytab based login can be
; ~/ l5 R% V$ U: z0 e% g6 l enabled with dfs.mover.keytab.enabled.5 F4 V1 i; {2 A6 @5 N
</description>
: `+ b+ @3 s# z, W9 b" v</property>
& F3 B' s" [; {* o6 O8 G- f8 h& ~<property>, L$ m$ f( A# o" d1 T0 Q4 k
<name>dfs.mover.kerberos.principal</name>* k2 w! j) ^- H ]9 T6 n# V
<value></value>8 R4 f2 b& D6 Q# [7 o
<description>0 Q" Y, D* N# r) v; D
The Mover principal. This is typically set to
# Y, W! a4 P9 v' T mover/_HOST@REALM.TLD. The Mover will substitute _HOST with its4 n( c( I' D+ x( O- T
own fully qualified hostname at startup. The _HOST placeholder
8 F* ]" S6 C0 n8 Z/ f) U7 G allows using the same configuration setting on different servers.% H# o# y6 y; x
Keytab based login can be enabled with dfs.mover.keytab.enabled.9 U# E) Z+ C" }+ L3 K4 R% E
</description>
" K: u' I5 P0 B; o; |</property>1 r% {& S% f- d8 u+ s
<property>9 N8 ]$ w7 R7 z. ?+ `& w+ w
<name>dfs.mover.max-no-move-interval</name>
v, l" F* G+ ^2 M) h; J2 ` <value>60000</value>0 S7 M! m' c' x" N
<description>
0 h) Q6 F2 e2 j! z7 s If this specified amount of time has elapsed and no block has been moved
- A: I W2 h9 e4 j! e out of a source DataNode, on more effort will be made to move blocks out of
" v( l4 X! i% r+ t1 K/ d8 `2 G this DataNode in the current Mover iteration.; U& C5 i# ~7 f0 Z# k
</description>' m( u" g# \" h( x
</property>
$ L7 B& _# f0 i<property>
" P7 R. V4 Z1 E+ m <name>dfs.namenode.audit.log.async</name>; S& J* M7 t" s( T' h0 ]
<value>false</value>7 Q5 b, E) L* X
<description>
& S9 p6 g1 Y9 y0 u If true, enables asynchronous audit log.' Y. H8 I6 s8 B! i6 J
</description>
7 D: T, S5 V6 A2 W- B</property>
* _# e. S2 S+ n) z3 o6 l6 U( V<property>3 P1 _' m6 m- _, | C) U
<name>dfs.namenode.audit.log.token.tracking.id</name>
- N! i! {/ K4 Q* a7 L# P7 J <value>false</value>
. a$ M2 v1 j/ K& ] <description>8 `. G$ }3 t( t* g- G
If true, adds a tracking ID for all audit log events.) ^5 ~' A6 N) N4 k3 M
</description>5 x/ x5 l3 L6 G8 x
</property>
/ Y. R, P$ A$ g3 O6 t- |% @( B<property>
/ z* f2 H3 l8 e d <name>dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction</name>( p" p4 M! X5 F- p- b
<value>0.6</value>
D5 `& B/ w2 n& | F) Z5 P- g3 N <description>
+ a7 M, a% Q' \' X* P5 @. A5 @( u Only used when the dfs.block.replicator.classname is set to
6 t5 m( @# U K org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy." k2 t4 o% Q+ U" f
Special value between 0 and 1, noninclusive. Increases chance of
9 |4 }* h# \% ~ v8 Z placing blocks on Datanodes with less disk space used.8 ]5 \+ T7 ^- B; O2 ^3 i) B
</description>
2 X; y! G1 |; A& `5 N& q! t% _</property>; r% {/ G: Y5 L: O
<property># ~! b4 P v& ]# A. y9 c
<name>dfs.namenode.backup.dnrpc-address</name>
* u1 o- \6 S" H v <value></value>
5 F3 P' [8 @$ r9 }# X1 q7 I5 k <description>
7 b. V' ?" l* I3 z7 l! K: ~ Service RPC address for the backup Namenode.( ]% ^) S8 v+ S9 j7 |4 D z) a# N
</description>
5 Q2 i) u5 p. N( I</property>
3 u$ f7 G) B1 ^. M& q8 ^( X, q2 l<property>4 C8 q1 |* B6 q- L8 C( y3 S
<name>dfs.namenode.delegation.token.always-use</name>
. W3 o/ w& X4 ~+ } <value>false</value>
, g/ n" G& g6 ]7 J <description> D% E+ j0 T' u/ [3 E( U
For testing. Setting to true always allows the DT secret manager
/ b2 A4 F4 m% |) ^0 Z" l# }, f2 i to be used, even if security is disabled.5 S* G3 R2 x! }
</description>2 g [2 R$ ~* C( @ U' z
</property>, V7 Z% u# x; C4 _
<property>. U& ?) `9 t5 h( }: g9 X: Z ?
<name>dfs.namenode.edits.asynclogging</name>/ @5 _& D% }) g) N2 U
<value>true</value>
! l2 W! ~2 P; | k8 j <description>
( v- r/ o$ w" K5 f, r/ t' A If set to true, enables asynchronous edit logs in the Namenode. If set; f$ S# s9 b$ B0 N; Q
to false, the Namenode uses the traditional synchronous edit logs.9 f4 p2 N/ L8 N0 T
</description>
O! Q% N2 o) J3 Z7 w1 h [" o d4 u</property>
8 v) \; d4 c A/ N' [<property>
8 C! F0 T/ u7 \4 ]+ O) M3 q: ? <name>dfs.namenode.edits.dir.minimum</name>! n0 \1 ^3 E! G9 I) Z, [5 `' G8 Y! G
<value>1</value>
- B; X% Q' s4 ^3 z% N4 J+ [9 |4 c <description>
. r: a2 \2 ?0 ]3 k r: `* B1 k* \ dfs.namenode.edits.dir includes both required directories: I% l% x' A, }# b
(specified by dfs.namenode.edits.dir.required) and optional directories.
- }, G+ A# b" ?: L0 o The number of usable optional directories must be greater than or equal/ M- L6 @! q2 Y% Z: Y) c0 B
to this property. If the number of usable optional directories falls1 l& A7 t& }8 I8 ? F. i! P2 ?! }! }
below dfs.namenode.edits.dir.minimum, HDFS will issue an error.
: d; Q, N' O. K$ l2 D b- } This property defaults to 1.
, P1 p2 d* u$ g </description>
& e; }8 P+ @9 _( I" v C</property> F+ ~' p+ ~ I5 t- \- m* i$ [
<property>) J0 l# t4 @/ D# y
<name>dfs.namenode.edits.journal-plugin</name>& [6 |1 E2 b5 p
<value></value> c T+ Z8 d( P" B- r" m$ z- w( y* g
<description>
% I; {5 m% i& Q* V0 V When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir,
: y* {- H/ u8 ^8 A. f4 M and it encounters a URI with a schema different to "file" it loads the
, p% U9 U; W) J name of the implementing class from
5 O3 ~, K5 K+ |# X4 k* [- Q0 d "dfs.namenode.edits.journal-plugin.[schema]". This class must implement9 c" B9 k1 L) K1 D0 p A8 \
JournalManager and have a constructor which takes (Configuration, URI).
- Y, u$ V! `1 U: s( W </description>
4 r& z. |: K0 f, Z5 N9 X</property>
4 Q2 T0 K- `7 M0 P6 x<property>
1 H9 X* I4 r3 q& h4 f <name>dfs.namenode.file.close.num-committed-allowed</name>7 V& t1 m$ ^; V" t
<value>0</value>
) g7 j4 U) v+ K <description>
3 l" @( H7 ?- k+ t# @5 A) U( _ Normally a file can only be closed with all its blocks are committed.
1 B5 ?# ^9 }; \5 j+ S When this value is set to a positive integer N, a file can be closed; R# ^) l' Z* Z
when N blocks are committed and the rest complete.) Y+ J3 {6 T$ w* b
</description>
: ~- f( Z) c/ c5 V2 Y7 r</property>
- E. \. y& D# l/ e<property>
; f5 A! X$ f3 O <name>dfs.namenode.inode.attributes.provider.class</name>$ u' W/ p9 O, p4 |- C
<value></value>
; R) V; C0 V/ N& G" H <description>
% z/ i' i& A; S0 B/ n6 e Name of class to use for delegating HDFS authorization., ^% T) |3 A/ p# J1 t
</description>
9 Q/ z, K+ A, \* l3 P6 P, z/ u</property>
5 N `1 w% Z+ K, s* R( G<property>4 T% r0 }# q! `1 b9 G G& A
<name>dfs.namenode.inode.attributes.provider.bypass.users</name>" ^; z$ b( F* o% [/ @' u2 s
<value></value>. u, V# W2 r; D9 \
<description>
R6 v4 f; i( \6 M A list of user principals (in secure cluster) or user names (in insecure
5 v0 R- B% i Z- @ cluster) for whom the external attributes provider will be bypassed for all
) N7 y/ ~4 {4 H) J x3 Z operations. This means file attributes stored in HDFS instead of the, j7 \ n2 Y/ d
external provider will be used for permission checking and be returned when
/ J/ }7 d( z8 u8 ~" J requested.8 j6 \5 P2 j, t7 E
</description>
/ v2 l% Y' ^1 J</property>3 l3 k' r& Y" H/ b4 s3 r4 g* k
<property>
( R7 V1 K/ J5 X3 P <name>dfs.namenode.max-num-blocks-to-log</name>
. N2 |* b1 c$ v7 v3 k& p <value>1000</value>
' A7 }1 T) Q5 u! L6 Z <description>
6 {1 A$ d9 b) C& H6 q3 r% o Puts a limit on the number of blocks printed to the log by the Namenode- A# g9 @. j$ m2 D
after a block report.! x# f) F6 e1 T7 ^
</description>/ F$ z% l0 f4 @! d8 }' r
</property>5 J( \* _" N% @7 y: a" d, R
<property>- V9 s* y( A3 Q- ?- z* h, ^8 ~
<name>dfs.namenode.max.op.size</name> v8 L5 n# A5 n. \9 r
<value>52428800</value>
5 M6 i5 m( o+ F z9 V <description>+ M6 g2 m# b1 X; k; V
Maximum opcode size in bytes.$ z( y6 e$ x, m. f* ~ ?7 b- p X9 ?
</description> }' `/ S9 \. }' y/ h+ ]% E
</property>) K5 p/ O' G) w) T/ v, J7 \) ]* n
<property>
' p% U: m/ b' {: C" E+ C <name>dfs.namenode.missing.checkpoint.periods.before.shutdown</name>
# O2 _: A6 }; W" g <value>3</value>
1 h! E5 o: c% @- t2 k- J <description>
2 m/ r2 H5 r' ^+ [( J6 k The number of checkpoint period windows (as defined by the property
2 o6 g* Y' T; @1 g% [% ], O3 S X dfs.namenode.checkpoint.period) allowed by the Namenode to perform
2 k7 V# n0 }* O; I saving the namespace before shutdown./ {/ e8 j0 m2 F: W
</description>. D4 f* ]; F0 x1 E( y
</property>
- F4 A6 X5 p) Z* W8 ^; q5 i<property>5 x' \6 w2 m5 G# g& I
<name>dfs.namenode.name.cache.threshold</name>+ z: X9 O" d) m6 Z8 f
<value>10</value>
* Y5 ?" d( U% t <description>/ K% u$ f4 E% U9 u( B4 J
Frequently accessed files that are accessed more times than this
, B# h+ E# X+ V threshold are cached in the FSDirectory nameCache.5 U. [/ ?/ {$ H$ [5 N
</description>
% Y) l; X/ S! {</property>
+ m9 b) k2 r1 q8 `% H, w0 ~<property>5 f7 z B; [% ?# O" P" p
<name>dfs.namenode.replication.max-streams</name>
" V; [) Q$ h' g) \, ? <value>2</value>" Q+ l- R0 \; J: K* W& {
<description>
. ]2 E3 T5 ?! P% A7 I2 j* L Hard limit for the number of highest-priority replication streams.
; W1 x; V( u% E" H: ] </description>! p6 e9 W0 G p
</property>
N2 g& ^, G( r- ]# Q. s2 ^<property>2 K& R6 a! d4 F/ [
<name>dfs.namenode.replication.max-streams-hard-limit</name>& G8 s. _& F# T
<value>4</value>
* n, Z5 g) d0 N% G) x' d; M1 O <description>8 P* H% w: J, K5 Q! C7 N% o
Hard limit for all replication streams.
; N. X6 j& Z2 E2 x0 a% c8 S </description>
- u- k9 J# E: M3 F" n</property>
+ ?: e y0 ]( n/ g+ [<property>
' W q/ o- ~( @- m <name>dfs.namenode.reconstruction.pending.timeout-sec</name>
/ I" G4 e9 i1 ? A' ~5 L' O% ?: k <value>300</value>
0 d$ H8 N+ v* S <description>0 Z" Z4 w a" \/ y
Timeout in seconds for block reconstruction. If this value is 0 or less,5 _) l& v6 ?& ^/ o( S" m
then it will default to 5 minutes.
) g4 b% }) D. ^( i5 S8 ~! _, Q0 } </description> } Z7 `& Z- C/ y
</property>
9 F' H* ]6 V, U: k* h8 w<property>
5 g: D7 e" k& G0 H/ ]$ _9 ]6 {9 o <name>dfs.namenode.stale.datanode.minimum.interval</name>
/ ^& Z( z6 @6 ] b, u) j" k- W <value>3</value>8 W" M9 m! t. G1 J% i }1 {
<description>
6 k: x; K; g; C$ S7 h, P# S Minimum number of missed heartbeats intervals for a datanode to
" }. i7 H$ F6 Y u6 J% S* ^ be marked stale by the Namenode. The actual interval is calculated as% \4 x. X# @: |! `
(dfs.namenode.stale.datanode.minimum.interval * dfs.heartbeat.interval)
# B) D! R( ~ N in seconds. If this value is greater than the property
" U1 Z' H/ Q" D* v! O& [" I dfs.namenode.stale.datanode.interval, then the calculated value above2 V4 A* L1 O6 G( c- E- W0 Q" N" G
is used.( ?! @, D' j& k7 T7 s
</description># n. a: m1 e) B/ a" U0 D# }
</property>7 `8 l+ p8 N4 b9 t7 V0 I! d0 V+ P7 w
<property>
# V% C2 Q6 ^9 b% C5 { <name>dfs.namenode.storageinfo.defragment.timeout.ms</name>
8 a6 P3 B' v1 p. Y5 m <value>4</value>9 V6 C) o8 {, s7 z1 ^
<description>7 g8 ~1 E& E3 `! C9 e. F* A5 E+ x. W
Timeout value in ms for the StorageInfo compaction run.) Z& }. h9 w1 P6 X. u7 `
</description>
- ?4 p6 b* a2 A, \- U</property>
+ V% u% Z- z: z" Y8 h& m( h<property>2 _6 w3 W( q) m3 M
<name>dfs.namenode.storageinfo.defragment.interval.ms</name>
, o: f* E- o7 |& C <value>600000</value>
5 g& u$ q0 O, {- G6 z# @ ` <description>/ z% O& h* ] R W5 j) ~% y) [
The thread for checking the StorageInfo for defragmentation will
( D" t4 m; l) u; {! Y$ Y1 ^( G run periodically. The time between runs is determined by this
% y( ]7 f/ ~" z: P- c* f property.1 ?+ T+ e* Y7 v, v! n+ T
</description># N) C/ h1 z7 b) o
</property>: [+ E; [3 c0 s# E1 @
<property>) ?0 D+ }& `8 }, a" X G' `2 A
<name>dfs.namenode.storageinfo.defragment.ratio</name>
( y) T7 g! X# Q- B1 ~, f# d <value>0.75</value>+ V7 n# G* @& z0 _+ s
<description>" D0 z! P( w/ A7 h& w
The defragmentation threshold for the StorageInfo.( d% s$ X1 ?" O# q
</description>8 p7 y+ z' v# o" |7 |" W# l
</property>
+ u& \- I4 c$ E7 P9 _<property>+ j n/ B( q, H# m
<name>dfs.namenode.snapshot.capture.openfiles</name>0 S# i% C# j/ A+ @$ s
<value>false</value>
- w Z2 ?4 [' g+ Q( e2 I* c: v* s <description>. P; | v4 i, e% C! z
If true, snapshots taken will have an immutable shared copy of- _2 [" m C; D1 }/ U3 ?7 {
the open files that have valid leases. Even after the open files
# N0 Y/ V- I( a; K r grow or shrink in size, snapshot will always have the previous& B' Y' N. F) E: m( I- S
point-in-time version of the open files, just like all other3 Q9 ]: Z" k+ w5 A
closed files. Default is false.: ~- o$ k8 @1 ]9 Q- c
Note: The file length captured for open files in snapshot is
# C( [ u( }& x8 N: @( g whats recorded in NameNode at the time of snapshot and it may
$ ^. k) r' {' M6 n6 ~0 I be shorter than what the client has written till then. In order1 z: j' J& a/ {/ ]: N3 S9 J3 y+ u
to capture the latest length, the client can call hflush/hsync
" ]4 r6 ?( u: _# g ~/ R4 E4 I with the flag SyncFlag.UPDATE_LENGTH on the open files handles. W+ A: ]: _- D4 }! {! L
</description>7 m- O$ t1 t8 u4 A1 X( M/ ]
</property>
) S u5 A' d$ z& m3 B# K5 s5 s K i' e<property>3 J2 r! O, K( a& B4 _9 v
<name>dfs.namenode.snapshot.skip.capture.accesstime-only-change</name>
! Y$ ]4 I" N/ k( U# e <value>false</value>/ n0 X+ M' S0 ]5 n' \' y% R
<description>
% e. P( q" L4 K) I w If accessTime of a file/directory changed but there is no other0 L5 y: t7 M l6 A, u* z5 s
modification made to the file/directory, the changed accesstime will
5 G% j6 l2 f8 c0 Y not be captured in next snapshot. However, if there is other modification9 s+ S+ M1 S% }2 }2 [
made to the file/directory, the latest access time will be captured q$ k! i% R* m/ A' S, M
together with the modification in next snapshot.2 E/ {$ Q" R- z# A
</description>* g3 a# A2 X+ a }; u+ f+ k4 C
</property>
3 C9 t! ~/ C' `: }0 D3 n# a# m<property>
" f5 e" ^! z" q- D9 X <name>dfs.namenode.snapshotdiff.allow.snap-root-descendant</name>* z9 I8 m0 v1 l
<value>true</value>
% L9 n* v. j; K N <description>, I- q! q+ Q- _. S* z; b* U
If enabled, snapshotDiff command can be run for any descendant directory+ R) @9 c$ O b) w" u% t& ^
under a snapshot root directory and the diff calculation will be scoped
& G% b# P2 m2 d) _' e5 ]" I0 | to the given descendant directory. Otherwise, snapshot diff command can
/ |# E" v0 a5 u: H6 ^8 K only be run for a snapshot root directory.
" T5 F: f1 y; R( j& o- K# L </description>9 K7 p+ a9 Z- @7 f
</property>5 N8 Z3 l; ]8 J6 E/ B( t
<property>3 X3 U8 d4 W2 ^8 j4 o6 r; A
<name>dfs.namenode.snapshotdiff.listing.limit</name>* ~* B( u/ H8 t
<value>1000</value>' ^8 I& I: L7 [8 `7 v
<description>
7 f: Q* U1 H7 l) d; F7 C Limit the number of entries generated by getSnapshotDiffReportListing within4 A2 j( s6 }0 V# w" e1 _
one rpc call to the namenode.If less or equal to zero, at most% n4 a( Y' i: ^; Q8 P2 p/ p
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT (= 1000) will be sent
. M4 v! |, H$ Q; x1 b' K+ W+ W across to the client within one rpc call.
1 B: L `. m. J' S </description>' C+ y; l! V; A$ y
</property>" b4 S& D1 }$ Q& I/ L) T
<property>6 V" ~, k) F8 L+ @
<name>dfs.namenode.snapshot.max.limit</name>
0 d. c& D8 |. x; A" N1 @ <value>65536</value>
& D$ c, Y- |4 v( V <description>- U4 }- E" d3 N) Y# x' a. c$ a
Limits the maximum number of snapshots allowed per snapshottable8 M7 a1 X% R3 V7 i( _
directory.If the configuration is not set, the default limit
0 L' G# ]- B, m9 R for maximum no of snapshots allowed is 65536.
8 q+ w0 Y! W9 u- }% p </description>
4 ] o$ A% N/ s9 J. g/ L</property>4 n+ r* }# [5 J, U- C" e
<property>
# M/ O! N) M7 @8 Q/ w. o1 O! ^ <name>dfs.namenode.snapshot.skiplist.max.levels</name>
V5 U' M) I, e <value>0</value>) ~) p% L" M# x" R" l5 D: W; f- c
<description>
1 V, W% K2 Q z- S+ T0 t Maximum no of the skip levels to be maintained in the skip list for6 L- p- Y+ |8 p2 y' n
storing directory snapshot diffs. By default, it is set to 0 and a linear
/ B7 C. ]5 \7 Z0 x1 j( l' \7 z. R list will be used to store the directory snapshot diffs.
9 L3 F% F! U. O+ |: ?0 ] </description>
$ N' l) ?$ P5 C+ m</property>0 I- V2 a" @* ^2 i5 [: U
<property>
4 {( q0 O( h6 a8 H$ O <name>dfs.namenode.snapshot.skiplist.interval</name>7 [7 y6 L% X2 {" B! O- g
<value>10</value>
* K' @ J/ ]! i4 n <description>2 i S2 `' W6 c2 {' C
The interval after which the skip levels will be formed in the skip list
% A3 b' W! K3 `$ |( v for storing directory snapshot diffs. By default, value is set to 10.
8 x5 U$ L. t- p6 B+ j2 t. Y; y </description>: |" a+ ] v0 h# h9 `
</property>
: ?% i- e3 O: M: c$ r+ q<property>/ E& o# I1 r6 o: |/ h$ i
<name>dfs.pipeline.ecn</name>
9 j: Q5 R' ` N2 t8 I# X" m <value>false</value>
' V# `% t5 n3 @1 n m <description>/ c3 Q* \& z7 g) x
If true, allows ECN (explicit congestion notification) from the
! U3 [2 r5 {6 X5 W5 P; Q, Q Datanode.
T: @& R8 l9 C$ v% ] </description>
+ q0 E& s/ s- |</property>
+ r% c" y3 u) w8 H4 i<property>
. R4 O9 ?$ e& Q9 U- C2 w <name>dfs.qjournal.accept-recovery.timeout.ms</name>2 q" ~+ l0 j3 ]0 H$ x' u4 c
<value>120000</value>
$ y5 A6 r4 z! g I8 W1 F' b4 K <description>$ O( u6 L# M! E) ~4 Y
Quorum timeout in milliseconds during accept phase of
% M7 L6 x2 x+ {) Y t5 } recovery/synchronization for a specific segment.1 \& h$ Y$ l. l8 O# B
</description>* d) b1 R, Y" s7 L/ j) H
</property># D# U# E( d- ?0 \" B- K# z
<property>- ^$ D% _: d: h/ b Z
<name>dfs.qjournal.finalize-segment.timeout.ms</name>
- p) y1 K! s( j( a5 B+ L5 D2 h <value>120000</value>
1 ?1 t, W8 v0 g8 g1 D4 f9 _( b( P <description>
b$ V! U+ W3 l# A Quorum timeout in milliseconds during finalizing for a specific1 }$ b5 }! [- Z" j: T+ T4 ]
segment.
# C5 ^0 D; c7 J( \ </description>( p) G! _+ ~& }( h$ S
</property>
& p, _' D: ?! a @& j& n* r' d<property>
: E2 s. y5 W, m3 Y( q! x% f: ~ <name>dfs.qjournal.get-journal-state.timeout.ms</name>2 S9 W: u0 n: i+ c$ o' h
<value>120000</value>. X( @. G4 Y; V: U/ U$ U
<description># B7 T7 g" s, _: x. ?, b7 N
Timeout in milliseconds when calling getJournalState().5 S3 |8 @: _$ y8 w! k
JournalNodes.- Z. r0 I3 j, r1 j# _0 _
</description>
8 t1 J: Z; @: A; _</property>$ {, V6 B2 q( t9 K7 b ?. X$ c
<property>5 K7 R% w: [# ?# j/ Q2 a
<name>dfs.qjournal.new-epoch.timeout.ms</name>% s6 [0 p: D9 f6 q% `* v$ W- `& A+ Z
<value>120000</value>
( R0 z3 c: v9 [/ d0 ? <description>
! Y, `9 w1 h/ ^# A/ \9 p- E Timeout in milliseconds when getting an epoch number for write& y6 @7 i! S3 U) u' K6 `0 }
access to JournalNodes.
# L8 J$ |( y! }* U: g; x' A </description> r$ {" R& M1 K" w# V3 N
</property>. u6 w' l. s0 M \1 w. u
<property>& c' ~- t$ u$ o5 Y
<name>dfs.qjournal.prepare-recovery.timeout.ms</name>
6 q. G9 N) D5 f5 Q; t <value>120000</value>
6 ]8 a5 ^% X5 T t: `: H8 Y3 @ <description>3 L$ Z0 g' }9 c u" S: _
Quorum timeout in milliseconds during preparation phase of) J, M. z" {0 l1 e
recovery/synchronization for a specific segment.
1 `) p* y9 V6 G& I0 P/ \2 F </description>$ {/ S2 K; @7 P, u; X) Q! R
</property>
/ O9 I% R4 G% P, n6 z, N<property>* u8 j4 t$ T& |* f: s9 {
<name>dfs.qjournal.queued-edits.limit.mb</name># R1 l' G9 g7 d! W! C; b- ^
<value>10</value>
9 [0 S) u& ]5 I8 s5 _! Q <description>1 D9 K. }" f) `! u' R6 B' _( @6 o
Queue size in MB for quorum journal edits.
) i2 `5 i* g2 K$ O3 f+ u </description># G! f: l: f) z7 t# [9 i
</property>$ P8 ?) T, w3 d$ I( |; h0 f
<property>
' ^0 v3 V8 z( t6 R; f <name>dfs.qjournal.select-input-streams.timeout.ms</name>, |' G* b( S; j+ e
<value>20000</value>
2 f- b& ]3 F2 _$ `# d <description>7 G7 j& v# y1 K: }. ?* U; f
Timeout in milliseconds for accepting streams from JournalManagers.
) _4 F' V0 h- T3 i# ]7 w+ f% a </description>
6 u( P7 j. L( c# Y4 f3 \</property>8 M6 L8 Q) e( R, W& ~( V4 `
<property>
) z+ w |/ S3 L- k; U <name>dfs.qjournal.start-segment.timeout.ms</name>
5 z' B. s% H1 l+ v5 [4 M <value>20000</value>$ z0 { X& |5 d6 E" J! V; e8 X+ c9 s
<description>7 _7 c# V; m( L) X: r8 C; H
Quorum timeout in milliseconds for starting a log segment.9 e6 i8 a4 M( S/ h* n7 r7 |
</description>* d9 e0 a2 \5 `! u% C& N
</property>
3 \' V9 v7 B2 y- _<property>6 h* p, l, `' Q
<name>dfs.qjournal.write-txns.timeout.ms</name>
/ @/ Y' W5 h- b3 W' z <value>20000</value>
5 d* m& {- J( U% L2 Y <description>
0 \0 Z, C) h, [. C) y& s; ^ Write timeout in milliseconds when writing to a quorum of remote# F9 q8 C$ M5 {" l: B- V
journals.
3 H& f$ w5 N9 L! [ </description>- q( g% b: ]$ r5 K0 u8 u; D
</property>
& M% |: P/ e! Q& L, n# y b7 g; p<property>5 P" `2 B! m# B2 B
<name>dfs.quota.by.storage.type.enabled</name>7 }' i. n7 H) O9 H
<value>true</value>
, y, D) s8 G7 i1 W/ U7 n3 G0 N <description>
2 i3 U) U! {4 O4 b If true, enables quotas based on storage type.
8 z1 C0 b' _& v </description>% j' E, N/ B& d/ I! G
</property>8 n: J- F# S% n, ]) E
<property>$ d1 |. Q1 Q% E! s0 s) ~4 S5 |
<name>dfs.secondary.namenode.kerberos.principal</name>
3 g% {2 k% A1 ~$ y% l <value></value>
s! i+ b# P1 x J8 a6 P5 z <description>
4 D% N. n5 `2 h6 d- ~ Kerberos principal name for the Secondary NameNode./ E j3 ]! \& i1 c% `: R
</description>0 f- r7 ?; w" V A( z& U& l
</property>
: g6 H7 j& x+ e2 Y ^( x<property>
( s7 S" O9 I9 j7 k: g( c. g$ F <name>dfs.secondary.namenode.keytab.file</name>
1 ^6 K& f7 h: z W/ h; a/ } <value></value>
2 t/ U7 `7 Y V# F* y* k <description>; u; _8 \8 |4 u1 p+ Y' W2 ]
Kerberos keytab file for the Secondary NameNode.$ E) t Q0 O. p1 [4 h" ?
</description>
& R: b r* g. L& O- b1 q5 F</property>
# b; W( z8 a4 y/ U2 K# R<property>* }4 G2 h2 L! W [( W" Q. c
<name>dfs.web.authentication.filter</name>3 ^2 c- t6 m# w2 |
<value>org.apache.hadoop.hdfs.web.AuthFilter</value>
/ q. m% ]2 e H: [+ ? <description>! G3 E) R8 d5 ~; W
Authentication filter class used for WebHDFS.4 K& b* L) ~% \/ i+ ^" R. n6 o# l
</description>, X4 A: V; Z9 H/ I7 f
</property>
7 @; _2 k! E! |8 d2 N1 r: C<property>. }1 w* J; ^3 E! Q
<name>dfs.web.authentication.simple.anonymous.allowed</name>; ~& T6 \( b& z3 V- r3 S0 F
<value></value>
8 G* l! q) X/ @+ b0 n+ H+ F. a <description>
- S9 U: e# J* W$ k! z2 l$ | If true, allow anonymous user to access WebHDFS. Set to) b$ k, n0 ?) b
false to disable anonymous authentication.
" O3 H: ]- C, C" x4 \ </description>
- Z! v4 ^6 ^8 X</property>" q2 K7 k; Q, G8 m, A6 i/ W- g
<property>
- B0 o8 H5 b a9 r <name>dfs.web.ugi</name>7 {# T* z! f* Q" w5 A3 Z
<value></value>
w2 D! u; Q& e <description>
! Q, t) B% K# ?* A dfs.web.ugi is deprecated. Use hadoop.http.staticuser.user instead.8 E4 ~4 {+ E& Z& F% V! |+ s& G
</description>
8 c' o5 e7 n4 f: D* E( c</property>
6 `4 i7 f2 v9 f, d" a" g<property>
1 D7 j% k2 r5 p* J4 u6 V <name>dfs.webhdfs.netty.high.watermark</name>( A$ C3 \5 [! o: p. S* Q
<value>65535</value>* t: R* m8 f- `* K& P
<description>
, S, ^. e: l4 }1 S+ K High watermark configuration to Netty for Datanode WebHdfs.; N! g r% U& k- g
</description>
# y" h/ r4 i: N5 i; T! {9 [4 o' Q0 O</property>: W6 |" j( N" x. k! W
<property>
& Q( ^) m* R; P* O8 M <name>dfs.webhdfs.netty.low.watermark</name>( C7 A4 I1 \7 p4 P1 G3 o
<value>32768</value>% R2 U8 Z& v, {6 \; c. D
<description>
( y4 i9 C) c3 W% t) t Low watermark configuration to Netty for Datanode WebHdfs.
# S+ g+ H" o2 K) K/ O- }8 H: A5 f </description>- ~0 G- d2 y5 q( @0 Z6 V
</property>4 A9 w7 t) v) c4 N1 M, I
<property>
8 P5 ]; l! ?0 C8 ?& W1 u) j <name>dfs.webhdfs.oauth2.access.token.provider</name>
+ R! t* a; `0 t7 B z, U4 d8 H7 u( `0 f <value></value>6 I; y6 E( R. k% V" y3 U2 j6 z
<description>& \/ E: L1 O/ k w+ M( R
Access token provider class for WebHDFS using OAuth2.
/ O. H' x8 q+ g8 ~% C9 H" x" X Defaults to org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider., F" ^4 v& c: W9 y1 g/ @
</description>
' }7 u$ n2 V0 B. n& W0 m# i* y+ \0 ^</property>
A6 f9 C* W( n( x<property>
) Z+ }9 j1 p( D8 B <name>dfs.webhdfs.oauth2.client.id</name>4 w0 M+ q, I" v$ O" t, k
<value></value>
. P: e+ w* z8 a0 L' h4 ` <description>- @9 S* [9 A8 c( P
Client id used to obtain access token with either credential or
+ m/ T: w# z8 w+ y' N refresh token.% ]6 I$ p5 s9 r" C( E! b
</description>- \) v n: ?( i. O" i9 }1 X
</property>
; D9 D) N" |$ d' [- D9 r<property>$ I" e* g* c% n- g
<name>dfs.webhdfs.oauth2.enabled</name>. o) Q# \: p' X. b! h& _" }- [; c8 f
<value>false</value>0 R8 {* v0 h0 `& l: f
<description>% d) o/ h, ?7 _5 x, H0 Q3 ]
If true, enables OAuth2 in WebHDFS
: V9 X$ ] A5 w/ x& v </description>
% t9 L A3 u! o: Y/ {</property>) }8 c+ u& n" D8 Y' v7 _
<property>5 ?! `; c% a8 u7 I/ t
<name>dfs.webhdfs.oauth2.refresh.url</name>: L6 X9 L# h8 k5 q3 Q
<value></value>6 p2 L9 R2 y+ Z2 z( [- Y* B3 ^4 l
<description>
8 y: d. Z6 B3 l' p2 m4 Q1 p p URL against which to post for obtaining bearer token with
3 o7 f0 j5 f9 V7 y, u8 G. R. } either credential or refresh token.' M$ ~+ b z3 m( j# g( u+ M
</description>. D* H+ T( h8 ?, ]1 Q
</property>: \+ P. L5 B$ l; t
<property>7 T% F4 i5 o/ Z9 `
<name>ssl.server.keystore.keypassword</name>
3 W0 k6 V0 b3 j( ]5 I& X( ?( Z, T <value></value>
: `3 P+ j2 z- ~2 p% B4 e5 c <description>& o8 j# I1 ]* Z! W5 @* F
Keystore key password for HTTPS SSL configuration
F. {+ ^8 V( S0 k3 K* G5 ~' } </description>6 P3 O/ S+ B2 d% a$ |* \
</property>7 i3 Y6 g d) I+ r# F' B
<property>
+ h T4 n, O! `% S: H <name>ssl.server.keystore.location</name>
$ X3 \$ R6 F6 }) K! s' y* a <value></value> R6 O; |: @) p6 p# H
<description>+ u3 q' a0 B& m6 n2 L& F
Keystore location for HTTPS SSL configuration
$ T4 K+ ~: o4 e4 @1 N </description>
0 N& u; A# K5 b: m, n" P; J</property>* Q3 |( k" Z( o/ j
<property>
) v) L* u$ h9 i2 R2 H1 n9 W- r <name>ssl.server.keystore.password</name>
+ }" @' B' {2 c# v1 d" b9 i" q <value></value>
+ u: V! x5 N1 K& H8 C. Q. b+ Z <description>3 {0 Q! i) |3 e! C8 w, A; A, T" S4 O
Keystore password for HTTPS SSL configuration: `) P- c) b) t3 p \, z5 t
</description>' d9 Z; G' v+ E2 C+ A6 K1 d
</property>
; K0 P5 e& w: L5 L5 [' W4 L& z* U<property>0 V- n$ _9 M5 R+ j& B0 [: ~3 ~
<name>ssl.server.truststore.location</name>. b9 z( L+ d& Z9 k# z3 `. V" H
<value></value>
/ y# N- {4 ^1 n. w1 k- B. ]: k, f <description>1 B7 J. u8 t# @3 l; O. d. P
Truststore location for HTTPS SSL configuration w: k: V& o1 D9 B8 g2 v
</description>
5 ?5 @. w0 i3 X$ q( C</property>
; f: u ]5 n* w( Q<property>9 \7 m. G% g, G% [! d L/ N9 D9 r
<name>ssl.server.truststore.password</name>9 `( \+ _; r9 M% b5 A! S
<value></value>
% V$ X; k# P% D/ A& M. p1 i/ H2 {+ j <description>
5 {. M6 e4 Q- e1 b Truststore password for HTTPS SSL configuration+ G% S8 c; g# F- e9 n
</description>
1 o( N) @$ c8 |8 p0 p) L</property>
" |/ h- p) p) M1 [" [, Z4 W<!--Disk baalncer properties-->/ w+ a8 o; `& P1 W% H, X
<property>
% e" h$ |! I. C# @+ Q <name>dfs.disk.balancer.max.disk.throughputInMBperSec</name>
# Z5 X, B* d7 l <value>10</value>
; V1 m# s, S# v# ?$ Z1 }2 z* C <description>Maximum disk bandwidth used by diskbalancer- I) T1 ?; g2 Y& P. {; {, ]
during read from a source disk. The unit is MB/sec.6 C0 o) G& B1 ] n* l0 `
</description>5 ?# s" b/ H T: ~$ R& R
</property>7 _. X, v3 e5 z
<property>
7 g0 z2 q8 r& \6 e <name>dfs.disk.balancer.block.tolerance.percent</name>$ F) y9 A- X2 X' k4 {
<value>10</value>
}9 [6 L3 m9 X3 U: w" M <description>
5 t6 {- i$ u/ A! V+ N1 q" `; Y When a disk balancer copy operation is proceeding, the datanode is still
v( g/ o4 e+ c( S& o6 ? active. So it might not be possible to move the exactly specified
* ]$ }, I% Z4 [0 W amount of data. So tolerance allows us to define a percentage which
# y) B$ u" _$ ?: n" X# L& C defines a good enough move.
2 a% }2 E; G. O! \; y </description>; H( O( u' s6 J3 R- s! a+ h
</property>
5 ?0 @8 o: w$ X8 l- } <property>
) _& o' T& _ \7 f5 W, @$ w <name>dfs.disk.balancer.max.disk.errors</name>& F* g+ A4 J8 ?
<value>5</value>! ]! a. @1 Y4 {% S8 U( u
<description>8 A. `% l# T$ c3 W2 C
During a block move from a source to destination disk, we might
3 [7 O( e D0 J A( h8 | encounter various errors. This defines how many errors we can tolerate
, U# ?: K) S* f) {" h" W1 k before we declare a move between 2 disks (or a step) has failed.
6 V8 |4 I2 x; E/ P. v. @ </description>* T( L0 R$ ^. [/ U' _
</property>
: w4 _! @, j6 G <property>
( A, E$ d3 P) f! S& J/ x <name>dfs.disk.balancer.plan.valid.interval</name>5 k# |4 f0 Y5 J
<value>1d</value>
B6 C `: W& ~) | A1 j( F+ i <description>
( a# W* W# r4 \4 Q8 v0 m Maximum amount of time disk balancer plan is valid. This setting
& N- O7 |8 ?- n+ Q0 _7 F0 } supports multiple time unit suffixes as described in
' j8 W3 b6 a& H7 q dfs.heartbeat.interval. If no suffix is specified then milliseconds3 d" p! A: I. D# M" ^7 u
is assumed.4 |* Q) _! g+ N7 r- X" F
</description>
( U. Q' W) E" P/ z </property>1 _( B* M: [: q, g9 x/ N/ g- W
<property>+ F2 c4 c' E& U- U
<name>dfs.disk.balancer.enabled</name>
- V' x d, j4 e* U0 u7 j. ~0 z0 P <value>true</value>: j; L1 q% O) x
<description>7 Z$ K- u- D' {* p& `" y0 I- \
This enables the diskbalancer feature on a cluster. By default, disk5 a2 F2 D& k9 g- @. C
balancer is enabled.
% T' J6 [# {9 m5 W) S, m </description>2 z- F& O$ C& I. o: K7 e7 c2 R
</property>
. y4 c; m9 Z q6 W# J& y <property>
9 s7 ?- w5 O7 a* n8 w- y6 x' @( Z; E( _/ f <name>dfs.disk.balancer.plan.threshold.percent</name>
( i& q: w3 f) l" C9 i% A1 g# K <value>10</value>
3 s; X5 W& g/ H5 N) ]5 u <description>" ]6 N+ L$ D* ^
The percentage threshold value for volume Data Density in a plan.
* M; I. v8 z8 |% N, e4 @* [ If the absolute value of volume Data Density which is out of, _* Y9 y( {4 _+ J
threshold value in a node, it means that the volumes corresponding to" Y. C- U3 V( T# T
the disks should do the balancing in the plan. The default value is 10.
6 j1 r, w& i& H5 I! V' K </description> s* B9 d& n; M t
</property>
7 c& ~2 I: M8 [# n& A <property>+ r9 s T; C0 U1 h$ p8 g8 o4 T8 L
<name>dfs.namenode.provided.enabled</name>
, a8 w+ f w9 V) U$ P <value>false</value>
. ~# H5 X* t, g2 c B$ M <description>
% C4 N$ ~ [) N Enables the Namenode to handle provided storages.
* `5 }+ R4 ]2 s4 w' L </description>
( S3 G) R7 v8 _+ S </property>6 j& L5 _/ a1 R) z" K6 [, R/ L
<property>" s2 F& ]" n; H
<name>dfs.provided.storage.id</name>& j! a" Q& [. s$ ]8 {
<value>DS-PROVIDED</value>
0 A/ T* }0 F H6 ~ <description>& n9 p0 I2 g z$ U8 N
The storage ID used for provided stores. O6 u5 [! Y3 ~1 x3 q
</description>
' M/ G# g) P# A3 z2 { </property>4 Y8 T X% ]8 r/ v0 m
<property>
% D7 E- H! j% T" p i% W5 H0 X <name>dfs.provided.aliasmap.class</name>1 g& T+ b6 m Q' E7 [$ C G' C
<value>org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap</value>
# n1 W8 X2 O6 _! y' ?) W ] <description>/ A5 r4 [8 ^# G; S2 b0 B
The class that is used to specify the input format of the blocks on0 g7 E$ w8 E1 X: _8 |7 y8 ?
provided storages. The default is. U7 P5 f- Z3 \* z$ v7 r; C% y
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap which uses% F, u e/ B! ^
file regions to describe blocks. The file regions are specified as a
2 w; j L$ N" s* `) |+ M# Q delimited text file. Each file region is a 6-tuple containing the" h+ ^9 }4 b* m2 k* u
block id, remote file path, offset into file, length of block, the9 P3 Z# V, E: A: ~$ y8 C
block pool id containing the block, and the generation stamp of the
& {' r4 U9 ^6 A/ p block.
0 Y! Z$ l8 t+ L# h! N$ N/ a </description>/ w) }. N& m/ p
</property>: N, A) h9 u1 e9 c; A6 f
<property>$ d/ ~2 s8 S7 i
<name>dfs.provided.aliasmap.inmemory.batch-size</name>8 t2 l& Y' u5 r( O [2 s
<value>500</value>
3 ^; C/ Q3 G" k; b0 t" ~) N% x <description>1 C7 L4 ~7 H. J3 U
The batch size when iterating over the database backing the aliasmap K$ {6 T9 z8 b; v4 D
</description>, A4 z+ _7 e+ y! o
</property>
( R5 w: O0 G7 {# ]( k <property>
% H5 { S) F# K$ F$ A5 v. k# Q: R" \ <name>dfs.provided.aliasmap.inmemory.dnrpc-address</name>/ r w* z( L1 S6 ]
<value>0.0.0.0:50200</value>
! j$ Q2 P" p8 w9 [: M <description>
! t4 y/ r; E$ d3 v9 M F: d4 V( R& u The address where the aliasmap server will be running7 `7 Z% H, d4 |8 j1 {) u$ x$ I) W
</description>4 Y0 ~9 ]& ~. f N2 I- H
</property>* _8 k1 W! j/ z6 l/ m
<property>
( ~: a, W/ R, i1 S8 _; U <name>dfs.provided.aliasmap.inmemory.leveldb.dir</name>
2 l o" B$ O6 l# S8 C. l0 e5 _. V <value>/tmp</value> u- m8 a, F% J5 d- } r3 [
<description>
' L p1 R* z- W The directory where the leveldb files will be kept/ J4 i! W0 O# L7 V/ g, R
</description>
& g* a) Z/ V) h/ I2 G </property>% ~! S7 v( x2 |
<property>
- D9 K: g+ D3 _: ` <name>dfs.provided.aliasmap.inmemory.enabled</name>8 A t: X% ]/ p1 \, Z: z
<value>false</value> X1 e: v" U. s# V' x* ~ u
<description>
3 T8 {8 b5 ^( W2 o Don't use the aliasmap by default. Some tests will fail
/ S$ X9 H; S& W6 s* L5 Z because they try to start the namenode twice with the' |! w; ~. Q( S% r2 v
same parameters if you turn it on.
& i/ M" K3 X2 F, ]) E# b" w </description>
2 n$ \% q1 Q( I </property>" S' R" s4 N0 x5 {9 Q9 n% t& N
<property>
3 l$ r+ x, q2 E8 P. C% ~" [* @; k <name>dfs.provided.aliasmap.text.delimiter</name>; Q1 c+ R" G d9 h
<value>,</value>
. N; o- p! a1 y; O3 h, d/ ~7 Z4 C <description>4 Y, C& L) f- s- B( s
The delimiter used when the provided block map is specified as
! b9 a; d# [4 z! X6 z7 w a text file.; o, d& Q& F t2 Q% l% `
</description>5 @0 t6 `9 T0 O. I
</property>
9 @" B/ x- D/ g- s( A& ] <property>; q- u) G# n! M+ k) g6 a+ b
<name>dfs.provided.aliasmap.text.read.file</name>
# M7 y- O' r7 r8 {! L0 b <value></value>
2 p& G$ Q% I- |& l' y+ @5 ]. k <description>
% D6 C3 R w9 _# A) b* E. }; S/ m The path specifying the provided block map as a text file, specified as1 j, @0 y% G6 e |5 q
a URI.
( H; o( P ^' \* u </description>
! N- x, o0 I1 i. O. i _ </property>( r9 D B0 o$ A9 ~' J( Q+ e# C" Q
<property>6 P: J% i' d) l9 N$ i
<name>dfs.provided.aliasmap.text.codec</name>& L) H6 n0 j/ I4 [
<value></value>2 K3 c$ y& o: A
<description>6 C% i5 x, p2 a1 `
The codec used to de-compress the provided block map.2 Y; S1 y, S: I- a0 I
</description>
. C- r7 e) m& i Y, S m" Y! J </property>3 l. r$ E. U) J$ F/ Q+ k& M( ?
<property>
9 M0 l! J# V) _( R* T <name>dfs.provided.aliasmap.text.write.dir</name>
( N1 o1 r- Z$ c# ~' ~8 y <value></value>0 o1 u6 y2 z; w& `& _2 Y
<description>5 L5 ^8 t* b* |# L; i) d5 s
The path to which the provided block map should be written as a text
. `5 G; u7 J# K. u file, specified as a URI.
- D( l! n2 K5 R </description>
) H+ i- X5 H" |! Q! S# u' f; C </property>9 l# s' {; S; G, r
<property>
- s$ C. ~/ |! U! J; N, |4 a- q$ V <name>dfs.provided.aliasmap.leveldb.path</name>
5 M. E7 F5 ^5 G8 y4 X <value></value>
+ u- \2 \: B: W7 u, y2 [8 Z <description>
. Y# n4 h9 e; K7 Q The read/write path for the leveldb-based alias map
+ Y4 F1 h6 G' C( ` (org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap).
) \9 G, J8 P7 z8 v4 {% n& V The path has to be explicitly configured when this alias map is used.
* @$ e; C$ X( B' Z </description>/ v/ u0 h& [" K
</property>
3 A; D1 A" W1 O: c- X/ p <property>- [; J( O% |' Y$ ?& \0 E' r
<name>dfs.provided.aliasmap.load.retries</name>3 B3 g+ ]* K- d6 e0 N) k. d
<value>0</value>2 r7 P* O; e; q3 H! G l
<description>. ~ M' Y' |$ G9 W* g* i, q8 X
The number of retries on the Datanode to load the provided aliasmap;' |0 Z1 O, u' S
defaults to 0.9 I. M9 A5 G' | O
</description>5 q, j, e D- q5 \
</property>* |; C8 y! [$ c) K+ r
<property>
4 F% g! H/ s X) x3 J2 @ <name>dfs.lock.suppress.warning.interval</name>. A) L% K# p( S/ }, P. O1 k- ^( A
<value>10s</value>- b: F6 M7 D! L' h- U$ S1 w
<description>Instrumentation reporting long critical sections will suppress
( e1 B7 V5 i! P0 p+ x+ P consecutive warnings within this interval.</description>- _) N4 V; ?2 J+ f
</property>- |% x! a( B7 A% x2 x# ~) ~
<property>
' Y q& g! d! x$ ~8 k& k <name>httpfs.buffer.size</name>8 U, _4 j7 J& a% r' Z# y) j% Y: P" x
<value>4096</value>9 |, k+ y5 }+ b. A
<description>% p0 @3 L3 u3 O) `1 L# s9 s9 Y) A1 O
The size buffer to be used when creating or opening httpfs filesystem IO stream.
! M4 r' m. m" e </description>
& u; _* {4 j. @6 a' T4 m </property>7 z$ ?: |( `5 z& E
<property>
2 l8 X0 w3 z) J$ x4 x <name>dfs.webhdfs.use.ipc.callq</name>* K8 ~+ ~& N- @1 t+ \' q' u
<value>true</value>0 @3 T, }9 T1 r# ^- t* I
<description>Enables routing of webhdfs calls through rpc0 \* I( `& v3 ?1 v- p# ~) {
call queue</description>' h" b2 K' m. j4 P @: Z$ B; |& O2 a
</property>
+ \& {7 z* w4 X" p' K <property>
6 }# e2 l, ~8 o. h+ c: U. Q# X( L <name>dfs.datanode.disk.check.min.gap</name>
* M! i1 T- ~4 G( w/ m; z <value>15m</value>6 w" Z+ w1 H9 \1 l" T4 Y( |
<description>9 G0 d p, ~* a9 P d
The minimum gap between two successive checks of the same DataNode
. j2 T8 J% M6 Z; d" d9 S, y$ x# ^ volume. This setting supports multiple time unit suffixes as described$ @0 x' T0 ?5 W9 b L
in dfs.heartbeat.interval. If no suffix is specified then milliseconds. ~4 s$ o7 r* \, p
is assumed.8 `. R1 |: u0 A1 W
</description>. ~' @+ P9 m% r& t9 R1 \: m
</property>8 Y7 A6 O. f1 l. O# ]) U5 j
<property>$ ]. r4 S$ Q9 p7 q- ? b. C% H
<name>dfs.datanode.disk.check.timeout</name>
( X. L; ~) k1 J3 w6 q <value>10m</value>( x2 J2 y0 A7 ^% h: ~5 A
<description>6 g: {2 e+ l2 |+ m8 _8 _1 _3 j
Maximum allowed time for a disk check to complete during DataNode
6 G# [# Y$ s7 [% b" h" S startup. If the check does not complete within this time interval# Z# ?& h7 U% c3 \$ ^ P4 p
then the disk is declared as failed. This setting supports
( b7 k3 B$ j+ c multiple time unit suffixes as described in dfs.heartbeat.interval.
/ K0 h" D' u+ G9 W) I& `$ Q If no suffix is specified then milliseconds is assumed.
, f5 e' L. S6 G6 v" }( H" s </description>( n8 E0 I, _2 u% h" {
</property>/ S+ [# B& v0 A$ i' b
<property>; V2 l+ c6 V8 R2 \7 ^" z
<name>dfs.use.dfs.network.topology</name>
; ~" y: F: S% O% J1 [3 F <value>true</value>
- O, o8 e S, F6 w <description>$ B+ m+ u# d! G4 `
Enables DFSNetworkTopology to choose nodes for placing replicas.
( I+ B4 L, |$ K- v/ l When enabled, NetworkTopology will be instantiated as class defined in8 A8 U: C9 Z- V- b8 Q$ E
property dfs.net.topology.impl, otherwise NetworkTopology will be$ T. o9 t) K, Y7 H: R0 C" m$ D" q d
instantiated as class defined in property net.topology.impl.
4 F2 t3 w7 T/ z0 S% w </description>
! Q, I; ]# Y# g </property>0 h \. t, M# N' z
<property>
2 e2 m, c8 f( C2 w7 T6 L$ M, S, { <name>dfs.net.topology.impl</name>
/ F9 V* M6 ]( S0 C <value>org.apache.hadoop.hdfs.net.DFSNetworkTopology</value>
! l1 f e, M) [7 O, q7 G% @ <description>2 {4 w/ n, `3 |4 x9 M
The implementation class of NetworkTopology used in HDFS. By default,
& Q/ h5 j3 ~$ m+ @) o8 E- x7 g the class org.apache.hadoop.hdfs.net.DFSNetworkTopology is specified and
7 B+ U9 Y- x: w. y used in block placement.
0 S+ k/ t9 K# P6 Y: u+ O& U( ] This property only works when dfs.use.dfs.network.topology is true.9 p6 l; L; S0 ?# [% q
</description>& Q) L2 f3 y' D
</property>
" l. }; i: y4 C4 f3 h+ s) X; D, w% U <property>5 }1 s) E# D3 d8 D$ ~0 o0 `
<name>dfs.qjm.operations.timeout</name>
& u; h3 g$ W* `. \4 D: X <value>60s</value>
9 ?( s |, J: w9 c <description># w5 x( V5 ?$ d: @" R* L" }
Common key to set timeout for related operations in
9 Y6 R1 O) q4 `# \7 t QuorumJournalManager. This setting supports multiple time unit suffixes
, u J3 H: |4 T as described in dfs.heartbeat.interval.
/ ~! Y) S4 }. i If no suffix is specified then milliseconds is assumed.
* D/ r; _/ i. U5 \ </description>1 s4 U% x$ ^/ g) Z
</property>9 d/ O# J$ ?9 A6 ~: @- o [7 _% @
<property>
' }0 p8 c0 M2 Z3 \$ u" K <name>dfs.reformat.disabled</name>1 E \3 v' s7 ]
<value>false</value>2 o% T2 m) ~+ m+ @
<description>" x; N) }1 z o" g& Y2 y
Disable reformat of NameNode. If it's value is set to "true"
# V$ b7 Y! m2 O7 n3 S6 I1 K and metadata directories already exist then attempt to format NameNode
1 I& y' ?+ R z( B6 J5 R# g% v will throw NameNodeFormatException.; Q/ U! q# O6 W8 l) F) ?% z- i
</description>
0 W- d+ j; U4 X0 m% P- d </property>
Y$ E+ \* |5 @' @( _& q( W</configuration>
$ j! ]* u9 ?' \9 y4 ~7 h6 o' b" i8 k& l3.mapred-default.xml6 ]. K% Q$ k: `/ _; J" ]! Z7 f
/ ^' j9 F" p5 [ P; h- Y<?xml version="1.0"?>( ~) n$ U& j" B0 |+ Z: W
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>3 O( N" {. P" Y2 a) d; A7 }& E
<!--! G! \1 a, F& f! Z3 a! v$ @' o# Z
Licensed to the Apache Software Foundation (ASF) under one or more7 S; s3 c9 c6 _0 S! Z
contributor license agreements. See the NOTICE file distributed with5 Q' F+ C) m/ g& E& J: Q
this work for additional information regarding copyright ownership.
X; @0 C" ]5 t7 m- {6 V" _ The ASF licenses this file to You under the Apache License, Version 2.0
" K9 h* C2 g& M% F. x5 b$ E; Y (the "License"); you may not use this file except in compliance with
+ P% P+ K: ` P- c3 q the License. You may obtain a copy of the License at5 Y* E$ O5 s) q5 y0 n- Z& S
http://www.apache.org/licenses/LICENSE-2.0
% X0 o. J# C. s& C2 D2 f$ w" k) w Unless required by applicable law or agreed to in writing, software
: n, a6 _1 H8 j# a distributed under the License is distributed on an "AS IS" BASIS,* [! E0 t0 D/ Y. C7 t: y3 j1 E# @
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
4 H/ ?; ]3 Z4 Y* x& N y See the License for the specific language governing permissions and
2 n# A8 z) @; }0 Z limitations under the License.
2 [. n* T' z0 z' F Y! V-->( O9 P6 ^1 b/ Y7 R# T
<!-- Do not modify this file directly. Instead, copy entries that you -->6 F! [- s( _( ?0 r
<!-- wish to modify from this file into mapred-site.xml and change them -->8 d( T5 d# X5 L
<!-- there. If mapred-site.xml does not already exist, create it. -->1 y) O, [ g8 S3 O e7 u
<configuration>9 @" }2 g1 b0 \8 Q" s0 X/ ^8 G
<property>) {5 d1 U) ? w
<name>mapreduce.job.hdfs-servers</name>5 v1 M. Q& s9 Q n
<value>${fs.defaultFS}</value>/ G* F( M& K. C+ X* [
</property>6 x- R. e u4 \
<property>: f3 U# A4 W$ G6 X
<name>mapreduce.job.committer.setup.cleanup.needed</name>* e" K! U2 {/ l& h/ P
<value>true</value>
) o3 f- ~; E' l/ t# B, E9 B, x <description> true, if job needs job-setup and job-cleanup.
3 `( C) b) Z7 j/ M# ^. K9 n" i false, otherwise8 k" U9 D/ P, a# Z
</description>
, |/ Q3 a6 T; H) u7 v</property>
1 V! X: O. D. k3 V* S0 I<!-- i/o properties -->
7 F4 y. I O2 V2 ?<property>
1 S4 b0 p: v2 d( ? K r' l <name>mapreduce.task.io.sort.factor</name>
9 z$ {" E" u# w* l0 M X, W <value>10</value>
9 g- Z/ A- r5 [+ L <description>The number of streams to merge at once while sorting
. u! f/ N- R3 M files. This determines the number of open file handles.</description>3 s& o( y; T* `
</property>& R/ T. j2 y$ d
<property>6 ]9 l0 g8 u( O$ k; R! K% ?* A+ N
<name>mapreduce.task.io.sort.mb</name>
% f" m- T$ w8 |: S$ } <value>100</value>$ g) E- i# b2 Y' k4 J
<description>The total amount of buffer memory to use while sorting
& U+ _! a3 b9 ~5 r; \' I; ^ files, in megabytes. By default, gives each merge stream 1MB, which
' G0 u/ E- b* a. |# g2 J" M should minimize seeks.</description>2 P" ?" Z* s; k0 [6 K* S# M
</property>6 d! @+ L5 C- v3 L# C$ C' N, y' i! O* A
<property>
1 s0 b( B# |1 g" s <name>mapreduce.map.sort.spill.percent</name>
$ A9 T/ R2 S: X0 Z& B <value>0.80</value>
+ @* a. l _# R" }3 g- @ <description>The soft limit in the serialization buffer. Once reached, a; m: f% n& }! u
thread will begin to spill the contents to disk in the background. Note that
* L0 B& e; e% j4 g" t( D collection will not block if this threshold is exceeded while a spill is% r5 u6 D' p: d
already in progress, so spills may be larger than this threshold when it is
9 Z' k. H) `6 {. o+ R X5 J7 C set to less than .5</description>! F6 L8 r! C0 V9 k6 F! [) \3 V1 @
</property>1 [9 v2 h( s0 q
<property>
! g# |! z* z- h6 y5 s% ?4 s <name>mapreduce.job.local-fs.single-disk-limit.bytes</name>2 g. }2 a' F) f: ]
<value>-1</value>0 u! S9 m. P" w. }2 A
<description>Enable an in task monitor thread to watch for single disk* s# {) z& `' Z: ^
consumption by jobs. By setting this to x nr of bytes, the task will fast
, B; P( U7 J% u* x3 F1 n1 e( E fail in case it is reached. This is a per disk configuration.</description>6 J0 Y7 B& t+ \( V2 b
</property>- d; m- f. e7 W( A+ a4 S- z
<property>6 Q! s; l$ K+ u& t5 z8 d
<name>mapreduce.job.local-fs.single-disk-limit.check.interval-ms</name>
: ^, A: x S, [ <value>5000</value>
% W4 g" S) x9 y, L <description>Interval of disk limit check to run in ms.</description>1 _6 _: k# z G" W3 `( q: ~( O5 n
</property>
& w, j: D# Y( `! u<property>" M2 V4 ^/ i/ n6 {, K2 T& N" L
<name>mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed</name>2 O0 |! y8 |6 X+ r; H
<value>true</value>; p& }8 ?# s% W5 ?! h5 V0 e
<description>If mapreduce.job.local-fs.single-disk-limit.bytes is triggered, T4 a9 x# _! N; ^* e, d2 ]3 z
should the task be killed or logged. If false the intent to kill the task
3 J& W+ p& N6 w3 l R. s' J G. M9 F- Z is only logged in the container logs.</description>
) @' ?8 {" {; g4 r</property>% \6 t: V! h2 m
<property>4 U$ h% } u- h/ b, x* @+ N
<name>mapreduce.job.maps</name>
2 ~2 B" w* q% {# H2 T# Q <value>2</value>
6 T9 }4 U3 V: W5 ] <description>The default number of map tasks per job.& R) e$ }1 a- j/ [/ @& s
Ignored when mapreduce.framework.name is "local".% D/ O% T4 e ]2 j+ f6 ^
</description>! p. h! T6 {! t
</property>; |2 u3 e3 D& ]5 K1 L# A$ e7 h
<property>
: B9 | }. r( H% b* e <name>mapreduce.job.reduces</name>
$ J$ D# v; E F1 H! R <value>1</value>
, Z+ t, h% B6 ]& o {' }7 T5 K6 n4 | <description>The default number of reduce tasks per job. Typically set to 99%* J, ?6 z2 N! x
of the cluster's reduce capacity, so that if a node fails the reduces can: \) t- g3 E6 c. Z6 s
still be executed in a single wave.) x: H) u7 _9 `$ f! G: S* j- y
Ignored when mapreduce.framework.name is "local".5 F+ l+ A0 c, K& {2 Z0 N3 @
</description>
) q) W/ Q* ~& X2 q& U* b. O8 W</property>* D( M2 B, U F, F' h# m2 t
<property># r; p. B! r/ T8 J+ M+ P
<name>mapreduce.job.running.map.limit</name>
& V: J! I7 T0 i <value>0</value>6 b) Y9 n# ~4 l$ x3 Z, T! e1 A4 v
<description>The maximum number of simultaneous map tasks per job.
! h! w3 k. c0 ?" a w, u- F* { There is no limit if this value is 0 or negative.
+ t( t2 r* @" ] </description>3 u1 U; J6 b' e' m1 `' Z) ~4 D* t
</property>+ i, D/ i% p' L- c
<property>2 B9 V' A( m9 c
<name>mapreduce.job.running.reduce.limit</name>* c* z. h( N$ l; c6 |% k
<value>0</value>
- q$ B! _' p# q; ^% K; h! w0 Y2 c <description>The maximum number of simultaneous reduce tasks per job.
( c6 z \7 w% v There is no limit if this value is 0 or negative.
3 A6 _' Y( S* [: h! w; I% D: | </description>: A8 L: K: q+ R2 R2 [; x
</property>
$ R& ~" V+ N& N<property>
- D& O1 T2 v5 x: K/ G <name>mapreduce.job.max.map</name>
% b9 K4 t h, ]4 `4 }& m" i <value>-1</value>
* M7 M9 g2 p# R <description>Limit on the number of map tasks allowed per job.3 g! \* C* E+ B
There is no limit if this value is negative.1 F B! H9 b9 t) r' f
</description>' D3 X( R* e/ M
</property>$ W- H% E4 s2 q/ I. S) {
<property>( N* U. l6 x) h3 B( L
<name>mapreduce.job.reducer.preempt.delay.sec</name>8 v4 o3 E& O$ r, u* i
<value>0</value>; z% Q$ e* T2 A2 ^- ~4 V
<description>The threshold (in seconds) after which an unsatisfied
$ S" ?* q* Y( Z/ N0 D7 f mapper request triggers reducer preemption when there is no anticipated
& u9 G P+ w+ x0 O headroom. If set to 0 or a negative value, the reducer is preempted as
* M( r$ w; o# L( h7 l, B# G soon as lack of headroom is detected. Default is 0., @8 U5 u6 A3 k
</description>
! ]/ E- G7 f5 l. x; l3 a </property>
$ `* g; R# {" k9 _. Q <property># H7 M6 | S# b8 z( o
<name>mapreduce.job.reducer.unconditional-preempt.delay.sec</name>
% c6 ?1 }1 z. L* R4 M7 p" u <value>300</value>* R' V- G$ B5 S! ?" Y3 ]# z
<description>The threshold (in seconds) after which an unsatisfied4 z& J8 d f, p9 D, {7 E: u4 k* A
mapper request triggers a forced reducer preemption irrespective of the
7 s* W7 U" R5 h, Z0 a: A anticipated headroom. By default, it is set to 5 mins. Setting it to 0
: T2 `3 F2 c8 n leads to immediate reducer preemption. Setting to -1 disables this
0 \+ B7 X. h9 @6 E, t+ O preemption altogether.1 C, V& j, D. O9 u, \$ u/ r
</description>/ r7 I" B+ ^; m9 w$ f! j- R
</property>) K0 G* V% _4 i0 l, K6 `
<property>( S+ D2 i* ]) _: p1 C
<name>mapreduce.job.max.split.locations</name>
* k5 r$ X) w& ?- B4 d <value>10</value>
3 L- A* ^5 [. y9 i! X+ { <description>The max number of block locations to store for each split for
0 U) u" K% b' u locality calculation.+ j, z' g& X% [1 J$ L$ Y
</description>
, P; l! H& @" P0 }</property>6 m1 e J. B% S
<property> Y1 `* r0 ~% u) v6 o% F* o
<name>mapreduce.job.split.metainfo.maxsize</name>
- X$ V, e0 i7 [ <value>10000000</value>
/ I" { s7 {- Y. e+ D. T <description>The maximum permissible size of the split metainfo file.% M/ u7 ^$ n( B' {7 y- S' L7 n. G
The MapReduce ApplicationMaster won't attempt to read submitted split metainfo1 E4 n. K8 A2 z. H$ B
files bigger than this configured value.
& B7 J$ p1 B" L No limits if set to -1.( h1 d' _; e4 D) e
</description>; g5 }0 S$ @+ u: }, ^: d$ `
</property>
- j- M+ G$ o- p& ^<property>
7 v5 }- Q. e( S' g: u0 m <name>mapreduce.map.maxattempts</name>, _; x% j) u: ^8 E- A' N
<value>4</value>1 U r0 R) {* U, N' b- U5 P v+ V
<description>Expert: The maximum number of attempts per map task.8 N: R4 ]# [1 B2 K* t
In other words, framework will try to execute a map task these many number
^5 b! n4 {! y- y# u" ]5 |6 B/ Q) c of times before giving up on it.+ k8 F0 f2 c0 d3 ^# u' |3 }
</description>
2 U) L& q. C( a9 y3 F9 S. U</property>
+ Z& z5 ], f" Y0 i1 X9 m<property>
8 Y: F$ O s# } <name>mapreduce.reduce.maxattempts</name>
* j! P. j4 n, h <value>4</value>' U( N; ?: @$ w- s5 s# ]
<description>Expert: The maximum number of attempts per reduce task.
/ k( I' ]/ v% H8 i$ | In other words, framework will try to execute a reduce task these many number
( S& I; R( w+ P: I of times before giving up on it.
( Z# v: i8 j2 q </description>9 g% K- Z T0 z' U
</property>
5 B1 ]3 `2 W" {3 n" J<property>( U9 s* I s j1 m$ O3 j$ ~
<name>mapreduce.reduce.shuffle.fetch.retry.enabled</name>5 l6 J9 d$ _' P- f
<value>${yarn.nodemanager.recovery.enabled}</value>' P% n Y# Z% H- ^
<description>Set to enable fetch retry during host restart.</description>" O; X" D" q! }$ c! l. B( ]
</property>
- |3 Z3 ?. O4 a& U* ]2 U<property>
% |: Q8 y l9 \# N) C <name>mapreduce.reduce.shuffle.fetch.retry.interval-ms</name>
/ S S0 c4 ]: \ <value>1000</value>4 K+ s( {3 z' b$ C
<description>Time of interval that fetcher retry to fetch again when some8 V, y; ^9 m' @4 z# J( k" t
non-fatal failure happens because of some events like NM restart.
& `# p9 ?& P) ~& m8 M" O </description>. T! E3 }/ m( y7 z
</property>$ O5 m& Z% ~: ?9 d5 m
<property>1 j' S& v+ P. J* j7 h9 B
<name>mapreduce.reduce.shuffle.fetch.retry.timeout-ms</name>
! F, I3 T- e0 c" @# o <value>30000</value>
7 h" l9 |2 }$ B( K) ` <description>Timeout value for fetcher to retry to fetch again when some8 A2 S# a2 q$ k; y4 s
non-fatal failure happens because of some events like NM restart.</description>0 g2 k; | W3 T
</property>/ b' t( ?5 A7 T/ Y3 m" a) z
<property>
) u. [& a Y7 N3 N% w, K+ K* u9 C <name>mapreduce.reduce.shuffle.retry-delay.max.ms</name>6 o# ^! O/ Y# d7 Y" L/ N
<value>60000</value>5 ~( b6 ^; F! p& i P% P2 O
<description>The maximum number of ms the reducer will delay before retrying
O( o" ~- n* e$ ]1 ? S/ I; r4 s to download map data.8 J/ @" s3 Y) _$ L8 W
</description>
( Z- g+ E6 D' R" W7 w</property>6 q0 p* k$ ` l F/ N( q: Q
<property>4 ?* k* H, z5 O7 h
<name>mapreduce.reduce.shuffle.parallelcopies</name>% r; k* q8 }% O9 ^
<value>5</value>; n2 n& b& d1 t1 y- m" Z. L
<description>The default number of parallel transfers run by reduce
; ]- m% ]+ i8 E V l/ f* ?# ~ during the copy(shuffle) phase.. j! r1 q* v. e$ S/ T. d
</description>
9 I" K- ^8 K: e1 t7 ?</property>
3 `( s5 z, m0 @$ ~! _* S<property>
x* X S. \3 q3 Q9 s <name>mapreduce.reduce.shuffle.connect.timeout</name>
: l+ k. ~* O: G# z <value>180000</value>! q0 Q- T9 c; l- @) V! t
<description>Expert: The maximum amount of time (in milli seconds) reduce
" ~4 u3 Q$ x, v6 s) h- a' ~' B8 a task spends in trying to connect to a remote node for getting map output. C! W; z2 A0 x/ i5 h
</description># M5 o+ ?# G3 {& v: d+ o
</property>
' w$ g. G0 m# F$ x- ?. G6 j<property>
; a* L& d. H3 S2 B1 C* J9 u <name>mapreduce.reduce.shuffle.read.timeout</name>5 m; F" x+ N; u
<value>180000</value>
) j8 N3 u1 ?8 U0 {6 `( v f <description>Expert: The maximum amount of time (in milli seconds) reduce' q8 _( R" D3 Z6 Y2 b! e
task waits for map output data to be available for reading after obtaining9 P7 v0 `) N3 _6 K9 y/ H
connection.$ z( t' O/ F3 j Q1 W
</description>
" a& j8 L, E9 x3 d# C</property>
. v- Y& k) t. J0 [: X- b/ h<property>/ }' l: ]3 o( w
<name>mapreduce.shuffle.listen.queue.size</name>6 U, g% ?& b7 j# S% ] [0 [, B
<value>128</value>
% b% E& s3 |' s2 X8 N9 @ <description>The length of the shuffle server listen queue.</description>; V! f# ~8 m; |& f6 P) a Q- _1 d
</property>: G0 f- b! I$ y+ t5 Y
<property>
+ f. e- }8 }( X0 G2 J- o <name>mapreduce.shuffle.connection-keep-alive.enable</name>
0 g' Y% v! p: j9 |1 ~$ ^ <value>false</value>
) i& ]* a( V4 E& N8 w6 y <description>set to true to support keep-alive connections.</description>' i- ^2 m3 ]. j( C( V5 n, P
</property>
8 z# q" w9 K% Q- k<property>
8 v3 x7 k* ~# [, R6 V* _; ] <name>mapreduce.shuffle.connection-keep-alive.timeout</name>9 W" z6 ]7 a; h* z2 p
<value>5</value>
8 k. w7 F$ N- \* p4 E, e <description>The number of seconds a shuffle client attempts to retain2 t0 f7 e; w) C- `* t4 o
http connection. Refer "Keep-Alive: timeout=" header in" v7 }( V8 f. O2 g" n; {1 q
Http specification
T; L2 y' ?+ I( j0 H) n </description>
- `1 I5 f! g9 @</property>5 j4 ]# T, g/ {" J; U' L
<property># d% s" M* z w2 M8 R
<name>mapreduce.task.timeout</name>
$ v( P+ d# t# {, s& Q6 Z <value>600000</value>1 M1 F5 P* F B' ?8 @: Q
<description>The number of milliseconds before a task will be
3 E5 s3 D8 `9 {' q# S terminated if it neither reads an input, writes an output, nor' s; X" }6 ~; `
updates its status string. A value of 0 disables the timeout.
3 X8 B: @8 _2 _7 t& e: I( d </description>
6 X6 q+ M5 n/ r4 X& v) j2 X7 T' Z</property>) K' ?+ c4 O4 l1 v6 M! g
<property>
. B. h u( Z/ z* O <name>mapreduce.map.memory.mb</name>
. X2 Q- {7 D+ f# [8 m, T; p <value>-1</value>
6 @8 j$ v U5 E) n, R- D" k <description>The amount of memory to request from the scheduler for each
# R; ?8 g1 N" u. u0 M# x map task. If this is not specified or is non-positive, it is inferred from
5 k. B. a ~* I" A" @ mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
4 J& j" m! \4 i" E0 \4 ]( L! L% G If java-opts are also not specified, we set it to 1024.
0 x+ l/ t" X" a </description>' B. A, w6 B. J9 n% l
</property>2 p! L- w5 ^: G0 P3 R9 q
<property>3 e5 p2 w2 q& e8 Y0 K9 Q
<name>mapreduce.map.cpu.vcores</name>4 i$ ?& l2 \" e* [; E
<value>1</value>
7 ~5 ~% P; |* v% I! h/ F/ [ <description>The number of virtual cores to request from the scheduler for
) i) ^0 f& `3 n" n$ C each map task.6 d9 ?/ a: m1 k) T3 t0 S9 ?+ F
</description>
" y) `" d4 N C) i</property>* Y: t6 i3 g/ ?( o
<property>/ q9 `; O: l. |; X
<name>mapreduce.reduce.memory.mb</name>& R0 d- F; a" ^
<value>-1</value>& W n9 \4 m7 B+ B ^! S5 p y) u
<description>The amount of memory to request from the scheduler for each
" E2 k2 U5 @" K reduce task. If this is not specified or is non-positive, it is inferred
3 c' l+ H# Y7 r# ^ B! \) Y from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
8 h2 j. T3 l4 G) s9 ? If java-opts are also not specified, we set it to 1024.) u! _0 M1 E! @
</description>
' n* u7 u7 {3 z8 N G) ^- J* t</property>
# q/ n- `/ c8 s1 Y+ i8 B0 h& V<property>) b* l5 N9 j/ A( L
<name>mapreduce.reduce.cpu.vcores</name> T! j% _4 ^; \' i k
<value>1</value>" ~' U3 @; \9 N2 u
<description>The number of virtual cores to request from the scheduler for
0 z* h+ `0 _" [) p! S Z each reduce task.
' C0 u* o0 _6 @: x e </description>" j7 L1 T3 u% S H' }8 ]
</property>; C/ `8 Q8 J. Y: w+ S3 V
<property>
( Q3 K/ H: E$ v- E: Q6 y <name>mapred.child.java.opts</name>- M- ^6 ]/ @. ]3 G, l$ d
<value></value>
& _3 F' e1 p6 \$ ~6 R <description>Java opts for the task processes.
/ ?4 A/ ]) z4 M, G7 ? |' \( | The following symbol, if present, will be interpolated: @taskid@ is replaced
6 M2 l: \' a: _* L" [ by current TaskID. Any other occurrences of '@' will go unchanged.
( F9 O% |' B& {+ G For example, to enable verbose gc logging to a file named for the taskid in+ T/ ^- |, I i# T/ G2 e& v# W5 h1 h& X
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:% H4 q/ l& r" O9 @4 B& b
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
7 Q' f+ [. Z9 S1 ~! ]- p Usage of -Djava.library.path can cause programs to no longer function if8 @$ A& ^ A& ?) d5 q/ U
hadoop native libraries are used. These values should instead be set as part
8 y4 P. f9 { k8 m) r, R of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
0 c3 }7 E- T x. z mapreduce.reduce.env config settings.; n$ u2 R6 N0 L$ v K8 X, ^
If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and
( q# v; m. K% n8 z mapreduce.job.heap.memory-mb.ratio.
- G: A+ q9 e. t* E. e </description>% B8 u, Q: V8 A$ x m
</property>
+ X, N6 _( k4 v$ I<!-- This is commented out so that it won't override mapred.child.java.opts.6 t% Y2 Y4 H# Z) ~
<property>' _% V0 Q( @/ ^- q
<name>mapreduce.map.java.opts</name>9 u7 @" [' l: A, h% D! O6 W
<value></value>
' c! n! \! J) y; ^' B <description>Java opts only for the child processes that are maps. If set,
% a" d4 N# ` v this will be used instead of mapred.child.java.opts. If -Xmx is not set,
7 o5 K) \7 s5 \: @' h" Y0 s; k. Q it is inferred from mapreduce.map.memory.mb and
* N i1 P( z' a3 X, o% K9 q$ ~" v2 T mapreduce.job.heap.memory-mb.ratio.
5 x" x( R& k+ t( {9 P1 K0 L </description>
. u9 I; \/ m l: d, Q</property># M) i, y4 k" z0 U! V5 ^
--> w$ V, U- f: C7 S" z
<!-- This is commented out so that it won't override mapred.child.java.opts.; i9 M- h: ~6 K2 X; j# J0 n; e y* \& U
<property>% k: U! b. A# a1 z/ @" o
<name>mapreduce.reduce.java.opts</name> [. y1 a( P6 g- c2 m
<value></value>; M0 ?$ S3 n" C t& G( s
<description>Java opts only for the child processes that are reduces. If set,
2 }1 w( l I7 K8 P/ W this will be used instead of mapred.child.java.opts. If -Xmx is not set,
' l! c( F |7 a. R, k7 y it is inferred from mapreduce.reduce.memory.mb and
+ y5 Y. D( H% e; v2 H. t mapreduce.job.heap.memory-mb.ratio.
l7 t0 t; a5 U5 Z( v# b; o5 \ </description>
7 u4 Q5 R4 x: o! _</property>- D$ L0 I5 a8 J0 ]
-->
# T; M7 X- Q" [: G/ ~0 E, A: D<property>0 y5 N7 v% }% \9 j x9 G9 j
<name>mapred.child.env</name>: g) m1 f7 p! V. N
<value></value>& b& `- J0 t6 f1 ^5 p) ^. z+ D, F
<description>User added environment variables for the task processes.! ] K6 T3 A" C5 T* _) F
Example :
[0 k/ [; |4 Y2 t0 l 1) A=foo This will set the env variable A to foo
9 A: q9 P# }, v7 c* ?1 O- P* Q+ ^ 2) B=$B:c This is inherit nodemanager's B env variable on Unix.
{4 P v# r3 v: k/ e$ U# P3 d; A 3) B=%B%;c This is inherit nodemanager's B env variable on Windows.6 j( E% Q9 |6 ^1 n0 }
</description>
8 o' B9 D; Z6 J7 K$ z- X0 i# b</property>3 T) l7 d' {! y5 T
<!-- This is commented out so that it won't override mapred.child.env.
: Q% H, y& }3 V( i<property>
" F6 C: J, {2 m& g: y <name>mapreduce.map.env</name>
# K$ s2 C9 [+ B8 T$ @/ d <value></value>
8 h- R) @# M- B7 ]3 v$ a5 G <description>User added environment variables for the map task processes.4 `8 t; ~# P! T0 h: ]+ k8 q
</description> o% g( r5 B5 d& ]+ b8 j4 }/ i
</property>
. ]& _# J( I& g! {, F% P z0 x( J-->
, }. U0 k3 W+ c" m3 E<!-- This is commented out so that it won't override mapred.child.env.
! o' x' N) h! G6 Z# K8 ]& |- n<property>
9 [3 u& A. k4 q7 E <name>mapreduce.reduce.env</name>, Z/ z6 L" P5 ]9 `- N8 q: a) I! [( Y
<value></value>
* |' {" |/ L. w <description>User added environment variables for the reduce task processes.
" H' ^8 y1 R# ` </description>+ G3 L. k8 H( d) L
</property>
/ q0 w8 D. u- o-->9 o4 g. k# L$ |* N
<property>% W/ U, \4 F, s' G# A
<name>mapreduce.admin.user.env</name>/ q2 T O/ y' ^2 R
<value></value>4 ?. ?* W+ R8 h0 @0 Z8 G
<description>: }. Q* `- z4 M/ ?, b+ V
Expert: Additional execution environment entries for
$ S; G x2 v @4 M; m map and reduce task processes. This is not an additive property." I( k; e Y& k( p' Q; a
You must preserve the original value if you want your map and* H1 r2 M7 y) C
reduce tasks to have access to native libraries (compression, etc).& I& A c& x4 B4 Z8 s
When this value is empty, the command to set execution
9 y: Z/ s' A" [ envrionment will be OS dependent:7 [' G/ G5 x+ Q, @' h! s; L3 R
For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
: t E% b% M; W- I9 k. q For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.
' i K* l3 H4 y' r </description>
, X: X/ S, V [$ S4 L</property>, ?$ m1 M; j7 o8 g/ l
<property>
8 X7 |4 `: x. A' F; C& R <name>yarn.app.mapreduce.am.log.level</name>
% O. l! [% v9 I4 l; @6 e w) W <value>INFO</value>
5 _0 e/ W$ M j0 { <description>The logging level for the MR ApplicationMaster. The allowed
% L5 |4 C3 G6 z4 X levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
) d, D! l. L4 T" }7 {8 D0 ^ The setting here could be overriden if "mapreduce.job.log4j-properties-file"8 P. [. H% f1 ]* h6 u
is set.) a( g7 B8 k7 \: V& o; H& ^ O, |- a
</description>; p* O2 ^: q9 F- y; J
</property>$ {' U% E% Y# ^+ _9 ]2 `5 y& Y/ x
<property>$ S% a; G# B- {
<name>mapreduce.map.log.level</name>
# D' ?6 W8 c) t+ i0 \ <value>INFO</value>% f7 j# x6 T }% U4 X
<description>The logging level for the map task. The allowed levels are:. A( B( v1 ]- n- I" n) d
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
" e, n0 R; O9 P" x The setting here could be overridden if "mapreduce.job.log4j-properties-file"
) B* i8 [" l4 ~. Q1 S is set.
$ g! @ `: @" u$ ?7 m </description>
* P3 Q) U2 y0 U4 e6 }7 K</property>
' B. P3 \- g; K* t4 s<property># Z) V# B) |8 X) E' f
<name>mapreduce.reduce.log.level</name>
; }! o9 Z( \% p' x( k" U- K <value>INFO</value>
- ^) \- o3 l# H <description>The logging level for the reduce task. The allowed levels are:
+ x3 Z4 v5 V, B) `! w! [ OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.
! d% Y$ f9 l H4 ]8 Q P* J; b* r The setting here could be overridden if "mapreduce.job.log4j-properties-file" u8 v% c1 s: i9 |& ]% H7 q
is set.
2 i+ b% ^ r* o4 s: i) f9 z </description>
& F9 i( V& C1 f</property> a t, W7 m, h% h m
<property>
- }5 U D: C7 o, A3 N <name>mapreduce.reduce.merge.inmem.threshold</name>
; _. F- W; }; Z8 c/ N2 h7 i5 { <value>1000</value>8 c4 w5 H# X: {0 `! H0 Q
<description>The threshold, in terms of the number of files
- Z. b E7 L3 Q6 N5 A. x+ y/ a for the in-memory merge process. When we accumulate threshold number of files. ^2 S0 O5 K1 K+ A* N$ _
we initiate the in-memory merge and spill to disk. A value of 0 or less than
0 F; x) u: ~/ z! S; f: w. f; m 0 indicates we want to DON'T have any threshold and instead depend only on7 S$ j5 {$ `8 D) R2 O5 H6 q
the ramfs's memory consumption to trigger the merge.
: y2 [& ]! d. _& R" T P </description>1 m0 m% t$ N/ I$ N; k: _2 E
</property>
L+ z! z+ [+ {/ x; P5 V: O0 K<property>
$ H) T3 J+ G1 B/ q <name>mapreduce.reduce.shuffle.merge.percent</name>. ]/ r! T. R' r: d6 J ^
<value>0.66</value>, X+ i5 l# L$ c
<description>The usage threshold at which an in-memory merge will be
. N# m. t3 z+ p) V/ V( @' J initiated, expressed as a percentage of the total memory allocated to
7 X$ e0 N0 L2 r2 m+ C* v storing in-memory map outputs, as defined by
) G- O O% w! k1 p0 k+ j! E mapreduce.reduce.shuffle.input.buffer.percent.* K$ e4 Q: L0 m# Q
</description>
" f5 x/ \" E# B</property>( N: k: [1 Q0 {- u7 A# Q
<property>
; k2 W x! @8 f; r5 r5 X, c+ u3 \ <name>mapreduce.reduce.shuffle.input.buffer.percent</name>5 C8 X+ E, Y: Q+ S+ J: X5 P
<value>0.70</value>
6 `. s5 \: X: V+ H7 R <description>The percentage of memory to be allocated from the maximum heap
! G8 R# ]6 N' E size to storing map outputs during the shuffle.# [+ L+ M. W; A2 b, E8 p) V
</description>9 h8 |1 G, y9 a3 I) M' _- x
</property>" g$ M3 \ f3 }# e2 D, P
<property>
4 a( f: k, E6 L: i7 l <name>mapreduce.reduce.input.buffer.percent</name>$ J6 P2 Q R. v! i
<value>0.0</value>
+ w) k* \- t' X7 y7 e <description>The percentage of memory- relative to the maximum heap size- to
4 q# P0 N J% x retain map outputs during the reduce. When the shuffle is concluded, any
% [1 X& I2 j/ I3 L0 O remaining map outputs in memory must consume less than this threshold before
0 b6 g+ v- R; _+ S/ ^' a6 _0 ?) } the reduce can begin.
# z) ^) X6 b$ T1 n- z% W# j' K </description>% {7 s- a% i( m' _
</property>6 d* `8 B7 W& d$ U" `1 c% v
<property>+ [# Y: A; U- I; _3 A" \* n. t# r
<name>mapreduce.reduce.shuffle.memory.limit.percent</name>
% c1 e( `/ g/ ]+ S% g0 ^ <value>0.25</value>$ Z: E! d# z1 W. Y& z! Y! s
<description>Expert: Maximum percentage of the in-memory limit that a
$ ]: o6 h+ `& a5 F, |1 e0 Y! A: S single shuffle can consume. Range of valid values is [0.0, 1.0]. If the value
5 ?( F# T% k' f, F( ` is 0.0 map outputs are shuffled directly to disk.</description>
+ W3 L( ^9 J. f3 j8 y: P</property>
3 J3 a5 n; O2 F* M+ S6 q<property>! a+ `8 i( `# W9 t
<name>mapreduce.shuffle.ssl.enabled</name>- s" q. l4 s5 a, K5 o
<value>false</value>) Z' _/ b0 i& W4 y" {/ A6 I- c
<description>+ q4 w1 d& Y8 B8 {
Whether to use SSL for for the Shuffle HTTP endpoints.2 N5 ]8 J& b* Z. p0 Z( [( F5 a* O2 _
</description>
' @3 Z8 q# h! O) R; Q</property>2 m9 K" x' N) E, h# v u( a
<property>. l) n7 v* c* P# ]" A+ t
<name>mapreduce.shuffle.ssl.file.buffer.size</name>8 P' j. Y/ d. I6 `$ N8 l: Z0 B5 u+ @
<value>65536</value>* G& U2 S% v* b. u; M9 l
<description>Buffer size for reading spills from file when using SSL.0 _/ S( E) [# {7 ?
</description>
& K; ?; N( p, j( Z$ _; o</property>7 y0 N% d7 j- y7 Q
<property>; Q% @# l! k# p3 r9 C% r
<name>mapreduce.shuffle.max.connections</name>
) n% {( q2 E7 L9 t. t% W* q <value>0</value>
! p8 @4 _! t% \# F5 s9 p; ~" a <description>Max allowed connections for the shuffle. Set to 0 (zero)1 P$ ^* d1 b; G1 z% V/ P
to indicate no limit on the number of connections.
9 _# z; [* R( A4 l1 q' q5 L; v0 G# Q </description>0 U9 ^ q+ {( K9 R9 Q/ P
</property>
: n0 R+ J& Q: a. F. h% ~7 o4 }; ~<property>
7 \5 Z) U/ O- W5 s <name>mapreduce.shuffle.max.threads</name>
7 ^2 [5 h& @7 s4 ~ <value>0</value>2 u& s, w; h7 w5 E$ S6 N. X
<description>Max allowed threads for serving shuffle connections. Set to zero
, l) _2 b5 t- [' i- x8 ?' d, Y to indicate the default of 2 times the number of available+ c0 {' }- I& R$ a5 ~# M& n
processors (as reported by Runtime.availableProcessors()). Netty is used to
4 x I2 h, \5 u7 G A( z serve requests, so a thread is not needed for each connection.
- ?- o( P9 ?5 ?) J# h% ?5 v </description>
' a) W# X E% d9 i</property>
% i# {3 R6 D4 c3 Z<property>
O; O" p1 o8 R <name>mapreduce.shuffle.transferTo.allowed</name>
- W& p6 W3 v- l( K6 z* d4 T" e; ? <value></value>5 f! v: R9 n% T2 r( G7 y r
<description>This option can enable/disable using nio transferTo method in1 L$ q! ^; l9 Z1 j
the shuffle phase. NIO transferTo does not perform well on windows in the
# ^0 ]* ?6 X; o+ N" `5 i shuffle phase. Thus, with this configuration property it is possible to1 q$ ^" y' v# Q1 R: x/ X
disable it, in which case custom transfer method will be used. Recommended1 v5 f% q( E3 `. m" ^+ d2 R
value is false when running Hadoop on Windows. For Linux, it is recommended
z9 [; C$ S! p# n) g to set it to true. If nothing is set then the default value is false for4 y8 b0 b5 R. q P; z
Windows, and true for Linux.
1 [4 M! d) Z1 ] ~3 `7 O' d/ K </description>( ]7 `5 z; O5 |; j) R3 T! d- C
</property>6 y% d8 J! }7 S+ D
<property>; B. }# y- f2 U% j
<name>mapreduce.shuffle.transfer.buffer.size</name>6 q; M& ~) ?1 l( i) D: d
<value>131072</value>) C p* }' e, ]- U- X+ |+ _# l
<description>This property is used only if G* O! F1 h+ Y+ P# ^$ |5 t8 b1 B
mapreduce.shuffle.transferTo.allowed is set to false. In that case,
; T- d4 ]/ H! P$ n5 G( v this property defines the size of the buffer used in the buffer copy code6 W$ J$ {/ k2 M! X+ M
for the shuffle phase. The size of this buffer determines the size of the IO
: y) D @2 w" P! L N2 R( I) ] s requests.
5 A, X) i. Q/ i8 X- y </description>7 g) Q4 k2 e; a( e w7 c5 J
</property>. C5 G# P* |6 q1 d
<property>: M% u4 Q, K0 }2 x" _+ _8 {7 K+ p
<name>mapreduce.reduce.markreset.buffer.percent</name>1 f' t1 B; }5 ~, P6 x( Z
<value>0.0</value>: `8 F( E c' G, O
<description>The percentage of memory -relative to the maximum heap size- to
0 ~: u8 P6 y) `* V" ~! P/ N/ k be used for caching values when using the mark-reset functionality.
1 h& j; a e: J5 ]+ J8 w3 @/ \ </description>3 p! {) ^1 t7 ~' z& O
</property>
/ |3 D% {6 [7 R<property>
; ]8 a6 N# L+ {$ e4 R) x <name>mapreduce.map.speculative</name>) V. J/ Q2 T/ e6 m" [( v
<value>true</value>6 J8 ~! K# D5 S1 M" k
<description>If true, then multiple instances of some map tasks: P1 X& G' ^# _" ^
may be executed in parallel.</description>" U2 q- r0 n, |+ i+ l& }7 ?& b
</property>
3 c, w; A# n0 _1 n" f<property>$ m5 s' Y1 u) V
<name>mapreduce.reduce.speculative</name>
# I+ r, Q' O/ F. B1 B; Y <value>true</value>! p' {% f2 U' R, ?: v j
<description>If true, then multiple instances of some reduce tasks: b1 V- Q: R# H$ H6 n: s
may be executed in parallel.</description>2 i8 X) a& S! i' B4 [8 T
</property>
7 J8 I3 Z/ I2 U' v: ~<property>1 f, d6 N8 i; a4 g- D
<name>mapreduce.job.speculative.speculative-cap-running-tasks</name>! }( r% N$ |# L& G A+ N
<value>0.1</value>4 p% C) e9 }# |- c
<description>The max percent (0-1) of running tasks that/ C5 N% T! I8 K: d8 L7 q
can be speculatively re-executed at any time.</description>* M$ y& c( e T+ h/ v# V
</property>8 x5 [6 Z7 O! @' F( j
<property>& v; z* U- r9 Z" L5 W3 V) j7 F
<name>mapreduce.job.speculative.speculative-cap-total-tasks</name>7 M, ^, V2 ^# u: e! h
<value>0.01</value>
K0 R: i3 x6 U7 d2 W <description>The max percent (0-1) of all tasks that c0 s4 w6 O4 n' G. L/ P
can be speculatively re-executed at any time.</description>
* {8 F6 l& J- Z& u" e% G! m( L</property>% e; o i/ [$ {, t8 ]
<property>
0 g! A* B8 F. H) H# I6 _# l <name>mapreduce.job.speculative.minimum-allowed-tasks</name>
* G0 E$ k; K8 {! b" R: ~ <value>10</value>
5 o5 v+ ?* r% P/ D# u0 \/ a <description>The minimum allowed tasks that
7 f* C- d) I/ S5 z can be speculatively re-executed at any time.</description>$ h7 i7 T; D) y; [( N3 ]% s
</property>9 P4 ]7 {+ y8 X8 g( k/ W
<property>
2 W! f% u1 H1 T( w8 h+ _ <name>mapreduce.job.speculative.retry-after-no-speculate</name>
: R2 E# D7 D- P- W6 J' U <value>1000</value>. X2 K% H6 a: |9 V* [
<description>The waiting time(ms) to do next round of speculation
/ Y+ \/ L# V- l/ z% I+ P0 h1 { if there is no task speculated in this round.</description>
& _( g: t" Q; F a2 H</property>) Q" G ~: Y8 P0 x( t
<property>
+ m% A% z$ b* {) W- G6 I; |8 m8 {* Q <name>mapreduce.job.speculative.retry-after-speculate</name>
# G' [- V9 N0 F) p; @2 P4 n; t! [ <value>15000</value>5 a# T9 F: f: _4 g; Q
<description>The waiting time(ms) to do next round of speculation3 I% f6 m9 Q. p0 ?# a
if there are tasks speculated in this round.</description>
; K+ E! V7 `; G6 ~- w' c</property>
) `! M" V4 _3 @6 x<property>
+ {7 V4 I! M& S3 y# J- c, {4 k% f <name>mapreduce.job.map.output.collector.class</name>0 w" ~2 i* K/ O
<value>org.apache.hadoop.mapred.MapTask$MapOutputBuffer</value>
9 s A# \7 R- B, O# p$ Y <description>, G6 i$ Z0 T p. J3 G; k0 v
The MapOutputCollector implementation(s) to use. This may be a comma-separated
' F" {, K" a# Q0 D5 t* o list of class names, in which case the map task will try to initialize each( J, T7 w9 @& a
of the collectors in turn. The first to successfully initialize will be used.
8 |* T# w: z! u' y8 t5 q </description>( H. r6 ^' t3 H! W
</property>$ \# [% L" b* m
<property>
# |! @) y: K- w! Z4 r" ^ <name>mapreduce.job.speculative.slowtaskthreshold</name>+ ]2 O7 ^: P1 z8 q* B6 p
<value>1.0</value>7 |. ]8 w/ ?" I) F( m( q: p4 D
<description>The number of standard deviations by which a task's
& ]0 T( ]9 K/ O3 @% ]) v& `8 ~ j ave progress-rates must be lower than the average of all running tasks'" L8 f' i$ D1 e3 c5 D: `: Q/ E4 ]9 v
for the task to be considered too slow.: ]* W$ o9 ^9 W5 {: c! H+ X/ D1 \
</description>
R, l0 |) S9 c6 c! e</property>
% o; k" T5 [+ ^3 U1 m$ Z<property>
6 G& z' E( l5 }% t: i! x7 B, W: T <name>mapreduce.job.ubertask.enable</name>
' h" J8 z$ f7 N7 N <value>false</value>
! V5 @& j2 j0 j1 p' H: F, U <description>Whether to enable the small-jobs "ubertask" optimization,
' U0 k9 `/ \7 C6 B Z8 f; U. {! H+ _ which runs "sufficiently small" jobs sequentially within a single JVM.
7 W: q6 r" I" ~: F0 R6 w4 |9 D "Small" is defined by the following maxmaps, maxreduces, and maxbytes+ H* T) a. v1 s7 z2 |' [
settings. Note that configurations for application masters also affect
1 d) m/ U7 _5 H ^ the "Small" definition - yarn.app.mapreduce.am.resource.mb must be
! B# f, u$ M! T larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb,2 k$ g4 p$ R7 U
and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than8 x7 \( h% @) d' P4 m6 r* W
both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable9 E( y( B/ c4 b; T
ubertask. Users may override this value.* G0 f6 n) Y6 e+ i4 I" R
</description>
) p, Y1 @2 L2 r, X</property>. ~; _, g$ ~3 \8 ^8 ~& K
<property>
2 t% U! a/ K& [ <name>mapreduce.job.ubertask.maxmaps</name>4 T4 n! K/ G% q7 H- M# K! ?& ?9 d; Y
<value>9</value>( ^; j- g% [) d* X/ u# B, r( H/ s
<description>Threshold for number of maps, beyond which job is considered
7 J6 Y' f8 d" A6 _ Z3 s, S7 o too big for the ubertasking optimization. Users may override this value,7 `$ }3 ~( T% X, e% {
but only downward.# x5 D) x! P& {# g- w& ?" w s
</description>9 |0 M6 g! P8 b' |9 L2 V
</property>
$ e, y+ j6 D/ f- [<property>1 X! x! k0 Y% ~/ u) O' s
<name>mapreduce.job.ubertask.maxreduces</name>
* @/ A0 b. Q" B" f( P/ I2 G5 \: u <value>1</value>
- a; I$ e6 v/ {) f1 n1 S, r( f <description>Threshold for number of reduces, beyond which job is considered
/ w! @7 j# K8 a1 A ? too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT7 ~1 _0 u- g( [! }* `$ ~% }
MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max,9 ?+ O) ]" q0 P- Z$ g" ? n
however.) Users may override this value, but only downward.
- d% ?% q+ f6 J6 S- O9 } </description>7 D* n; h9 Y' I7 W' }
</property>
/ Z s9 L+ T7 @5 X i<property>
# r- ~- K3 y& i' L0 ?4 I <name>mapreduce.job.ubertask.maxbytes</name>! Z7 F6 V) o4 ]& N0 W4 m
<value></value>. m; A, M0 R( B) ^* _# f/ ]3 o
<description>Threshold for number of input bytes, beyond which job is
/ ]) G3 |6 K* a7 K6 q3 ^ v considered too big for the ubertasking optimization. If no value is; h: O% d' A. s' s. C1 b
specified, dfs.block.size is used as a default. Be sure to specify a
/ u/ s0 y1 i6 r% f* g default value in mapred-site.xml if the underlying filesystem is not HDFS.
. _# ~$ s2 e( d' h7 Q) V2 B Users may override this value, but only downward.$ r, {7 B+ ^% }' F. T* n0 W
</description>4 O5 ~0 D: b: Z
</property>
5 v+ Y" f) [& r: ?( k<property>! n: |8 j2 u3 ]6 T& ?1 K
<name>mapreduce.job.emit-timeline-data</name>
) j" z# a6 V) e( T, l) D0 b/ P# ] <value>false</value>4 B7 I5 o6 P$ H+ K
<description>Specifies if the Application Master should emit timeline data
9 w& y/ G' S. m& d5 }* n to the timeline server. Individual jobs can override this value.
+ s4 X& P7 ]- X" ?9 q1 Z </description># v* q# O. B$ H! \4 {
</property> j* _7 B' X# s, K( X+ q
<property>% a) p1 c& v/ ^0 h* \
<name>mapreduce.job.sharedcache.mode</name>! e1 D# j; C# E; R" ^. k9 q( N9 K6 X
<value>disabled</value>* [" _, q# @) h
<description>
/ F- `. Y0 B0 ]7 R' o8 P A comma delimited list of resource categories to submit to the shared cache.* j3 m1 e) P3 u2 ]# @( H8 _' ]
The valid categories are: jobjar, libjars, files, archives.
% V2 r; A0 \5 B9 \! E2 }! @ If "disabled" is specified then the job submission code will not use
( L& S; I! q: }) ~& v3 q* d the shared cache.
! c2 \0 q+ W+ c4 z9 z4 a$ X </description> a2 x2 i e, ? z8 p5 G9 g: |' C/ K
</property>' g0 X4 e. |: E5 @/ m7 _
<property>% j! C0 i8 ~* b: W* y* U
<name>mapreduce.input.fileinputformat.split.minsize</name>& e0 n! P( T) L3 ?
<value>0</value>
8 B6 Z" O2 L5 o/ E, d+ @ <description>The minimum size chunk that map input should be split/ J6 N* S8 M, P
into. Note that some file formats may have minimum split sizes that
" Z5 u8 A( S u6 K% R4 H3 Q6 l take priority over this setting.</description>+ M2 j0 x. J& T' g5 w3 M2 q) D
</property>
5 f9 n8 U; K, b8 S( i<property>) P2 h" B; \" T0 M5 `! C
<name>mapreduce.input.fileinputformat.list-status.num-threads</name>
2 i5 q4 {2 Y. f: m: C/ L y <value>1</value>
: `; s2 t# ^# _ <description>The number of threads to use to list and fetch block locations" Q0 m) J0 t2 i5 X; ]$ x
for the specified input paths. Note: multiple threads should not be used
) L' |! Y1 D( d: O if a custom non thread-safe path filter is used.
0 d+ v1 a, V9 N9 F2 D* Z$ r; _ </description>
* e/ }1 J- O& p: T1 d</property>" B- T7 B0 v0 x/ x
<property>' j( p2 b: J" A4 j
<name>mapreduce.input.lineinputformat.linespermap</name>- e1 V( b2 ~! {* x' t& @+ P5 G
<value>1</value>( ?; s# t3 P- y5 ~* {0 i5 I
<description>When using NLineInputFormat, the number of lines of input data0 { }/ t2 D. _8 p* x; o# ^
to include in each split.</description>
! p. w; U3 ?9 e; ]</property>- v L0 p: S3 x( U9 F
<property>6 p' ~% B) I1 J. L$ a
<name>mapreduce.client.submit.file.replication</name>% F3 }; l" x' ^* k; K6 b
<value>10</value>
8 N' M8 b3 C4 s& P" F8 w9 D3 \8 B4 y <description>The replication level for submitted job files. This8 c2 y% k1 [8 s K6 j& Z
should be around the square root of the number of nodes.
9 j$ ^& _; m. q1 w, m) J </description>
4 V* F6 {4 T5 I$ d2 _+ |</property>
N7 G) g6 T2 `* t7 @2 p* |<property>$ O! d5 o2 T3 Y# c; o
<name>mapreduce.task.files.preserve.failedtasks</name>( e/ s$ e4 _! P, G
<value>false</value>3 G; n' _! X4 Y
<description>Should the files for failed tasks be kept. This should only be$ c5 @1 K/ S5 R1 |
used on jobs that are failing, because the storage is never
1 E7 q- T( {) }8 @4 B- F# a$ \4 ~ reclaimed. It also prevents the map outputs from being erased
2 B8 N2 R8 A% I3 {% s' E, g from the reduce directory as they are consumed.</description>8 m8 T) g% O) P- {" d: v
</property>! h( e9 ^2 G( y1 n5 e1 U
<!--
8 z! b, R; ~3 k% a! C. o <property>5 X. f7 b' b1 |
<name>mapreduce.task.files.preserve.filepattern</name>" k0 R# j F0 X' G+ ]; x: G3 \$ L
<value>.*_m_123456_0</value>
8 D1 u; C; B: _" F <description>Keep all files from tasks whose task names match the given& E% {; P) f+ x! P- U* i
regular expression. Defaults to none.</description>$ u5 a, o1 q9 p" w; G
</property>2 H$ ?4 e8 S( M, U( U: ]& U1 x& L
-->
5 P1 o2 \, }8 {' [$ c<property>
! ?( E- G. G4 i/ Z3 P4 i <name>mapreduce.output.fileoutputformat.compress</name>/ A9 ~9 ^0 L x$ D3 h3 Q' r
<value>false</value>" G. z' N" X+ f R
<description>Should the job outputs be compressed?
) G/ ^* e4 x$ J! N) B: S2 | </description>/ l$ W) J! A& N1 P) Y
</property>
# c( N5 H9 [$ Z8 D<property>
V8 b# ^9 M9 X Z <name>mapreduce.output.fileoutputformat.compress.type</name>% g, S, B3 t# b* p
<value>RECORD</value>$ j9 e; L) j* Z
<description>If the job outputs are to compressed as SequenceFiles, how should
. o2 s; l+ A% b$ _ they be compressed? Should be one of NONE, RECORD or BLOCK.
+ F1 ]& w) {/ E. _. V% R </description>8 g6 \ t( f5 [
</property>
% N/ j3 r' x4 m<property>) |( J/ {' Q5 ~2 v$ P& S# o8 q& V
<name>mapreduce.output.fileoutputformat.compress.codec</name>3 J! f! t S8 L9 ]$ q' X; ?5 a( r
<value>org.apache.hadoop.io.compress.DefaultCodec</value>
U' ~( _- l3 S+ T2 { <description>If the job outputs are compressed, how should they be compressed?
u3 K6 M4 W! |; H </description>
" q' `" A, B; q, T8 ^</property>
! e1 Z# {2 s( j' y) L5 ?<property>% m) l. a. n+ C7 t# S
<name>mapreduce.map.output.compress</name>- g. r0 G$ e" |
<value>false</value>
8 R/ _0 Y! f7 Z2 J <description>Should the outputs of the maps be compressed before being/ L0 S2 p% g! `3 k9 ^
sent across the network. Uses SequenceFile compression.
2 u5 Z* U) W6 y5 U+ D0 H </description>: W6 w7 E# c. y5 [5 q3 s9 g
</property>
# T% q1 D( N8 F' |2 Y5 I<property>) |3 R3 A0 ^/ H7 Q
<name>mapreduce.map.output.compress.codec</name>; c+ y- j; q( _* P
<value>org.apache.hadoop.io.compress.DefaultCodec</value>5 Q5 w- K# D9 c( B+ U. \# `4 V1 a- {
<description>If the map outputs are compressed, how should they be# Y7 r+ A- ~# y) a
compressed?
5 R) X0 m) T4 X </description>
5 d# r4 E0 P* l, X& P" y</property>" `' s- L& `& u! {: G n7 K
<property>
5 s* c. j" \) e4 ]2 d6 t) n8 n' N$ c <name>map.sort.class</name>/ H6 V- M/ [$ y: F- W: f
<value>org.apache.hadoop.util.QuickSort</value>1 D0 Z# }1 X' e
<description>The default sort class for sorting keys.: @# \+ H6 k( V8 [& x
</description>6 V, r* z1 u8 @3 l, T6 q( M
</property>
$ a& F( t5 f8 @/ f<property># U' [) d% w S; H2 S
<name>mapreduce.task.userlog.limit.kb</name>
- I& N! u4 J- `8 b <value>0</value>. c* ^* U7 [( }6 R R+ k! a
<description>The maximum size of user-logs of each task in KB. 0 disables the cap.
# A4 Y/ Y! l# ^2 n: X9 t </description>
" j& K) [6 v' l7 _+ y0 l</property>
& G" m, C' n3 l, m<property>
3 b# I! L8 T7 P7 v5 P, m <name>yarn.app.mapreduce.am.container.log.limit.kb</name>
, m2 k5 n' H! }/ o <value>0</value>6 @# s) f4 x" N4 C
<description>The maximum size of the MRAppMaster attempt container logs in KB.4 b. z; K0 \$ Q' b4 i6 D# T! R
0 disables the cap.) R8 S, z0 t ]$ T) p6 r" }0 R) K
</description>& V+ I; m1 x. a3 E5 n
</property>
6 `" E! r" Y/ d, W6 ~<property>) k4 N$ b% r1 ]% k/ C# T
<name>yarn.app.mapreduce.task.container.log.backups</name>
2 A/ B" d- g9 ~! V# I" y <value>0</value>9 \$ o0 v3 K8 E; F) t5 z# Z: U
<description>Number of backup files for task logs when using
2 n8 @$ q- Q/ g5 R" H ContainerRollingLogAppender (CRLA). See4 V7 \. m8 L$ R% B) Y+ Y5 U- w
org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,2 }% L4 H2 Q" I. H5 x
ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA
* F4 [0 C& Y: N6 L! { is enabled for tasks when both mapreduce.task.userlog.limit.kb and
3 E. p3 O0 [5 d: f yarn.app.mapreduce.task.container.log.backups are greater than zero.2 l) i' V( j, M) r }
</description>2 S X4 I; W9 m5 p i* g
</property>
1 |$ X" a" I0 o) q3 p8 A* d<property>
9 ~+ s, j! Y) |! F* x5 M <name>yarn.app.mapreduce.am.container.log.backups</name>
" {# c; L$ H% N0 ~ <value>0</value>% O; z! `' ?) U$ e( y
<description>Number of backup files for the ApplicationMaster logs when using: Q4 P% q, ^- P# X0 `
ContainerRollingLogAppender (CRLA). See
( [/ p5 h* f. { org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,: M5 s5 W( f6 I$ ^; M3 x
ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA! _" W" u: g& {/ z0 o2 @$ R
is enabled for the ApplicationMaster when both
9 i7 X1 }: f# p yarn.app.mapreduce.am.container.log.limit.kb and* d( m8 w" O. j$ r
yarn.app.mapreduce.am.container.log.backups are greater than zero.
) c0 P' X7 E; ~6 A5 \) S6 t5 |# a </description>
* k& X& [! p5 k p4 N4 x' b% }</property>
; z/ a7 _- ^0 V( q<property>
6 n; d- ~# ^; B: ^- x. U <name>yarn.app.mapreduce.shuffle.log.separate</name>4 G1 C2 p. ?9 H5 H s
<value>true</value>
: i, l. _* N2 Z) T7 h+ U <description>If enabled ('true') logging generated by the client-side shuffle4 S9 d7 Z7 X/ m0 Y$ p
classes in a reducer will be written in a dedicated log file
9 n8 i7 L% i3 @) X3 P& [+ r 'syslog.shuffle' instead of 'syslog'.* ~0 J/ w5 L% q% w s) Y2 w
</description>
& T% }5 ]! c0 x2 f+ A6 \. Y</property>' ^4 i! i% C5 u7 `- m
<property>
# ` ~- ], R. y <name>yarn.app.mapreduce.shuffle.log.limit.kb</name>
[5 e. N- ~! P7 c6 \% f <value>0</value>. |* g2 b# ^) I* A5 A* A( D o
<description>Maximum size of the syslog.shuffle file in kilobytes
/ {8 W1 X, }; Z% @! l (0 for no limit).
. u# e0 }" Q/ x- M: N </description>
$ `' R9 ~' D e/ |! `1 g</property>( p' O. u6 g9 L, s+ o# y7 g: L. j7 p2 o
<property>
1 W) W& c% j( d+ m <name>yarn.app.mapreduce.shuffle.log.backups</name>$ E) P: ?5 P& X9 |8 y$ ], G
<value>0</value>
& h+ n* b J) W3 g/ u2 i" A. K <description>If yarn.app.mapreduce.shuffle.log.limit.kb and
5 k, o: s4 P# _: A yarn.app.mapreduce.shuffle.log.backups are greater than zero
9 Y6 w: Y. l, U0 T4 u then a ContainerRollngLogAppender is used instead of ContainerLogAppender
4 ^" e8 J6 L% z* C; j3 @( ^3 \ for syslog.shuffle. See
7 W' r: j k8 ~, ]5 ^2 K/ H3 b org.apache.log4j.RollingFileAppender.maxBackupIndex
* w# R7 h! t9 J$ A, K </description>
1 W$ r' F. z( r! ? W</property>
* B$ _& Z! u4 [' Z4 O( B8 q$ x, W<property>
) e! j, c- }8 ^* G6 a <name>mapreduce.job.maxtaskfailures.per.tracker</name>
2 l. Q: u7 O! x6 l3 g+ W1 T <value>3</value>$ S( X) G- M2 L7 {0 N L# H, R4 |
<description>The number of task-failures on a node manager of a given job
7 ]( |: x& y, O( |2 M' v) ^' g8 M after which new tasks of that job aren't assigned to it. It
/ W/ D- a* X/ X: q1 c MUST be less than mapreduce.map.maxattempts and' R* E( l2 |& B- H/ E$ L5 W5 K/ L
mapreduce.reduce.maxattempts otherwise the failed task will* _. e0 @& r6 _4 i" K/ O7 x1 C: x
never be tried on a different node.
* }0 c1 P* q( k7 J* @ </description>& q0 }- }) W1 k6 \2 u0 u- Y0 G( h
</property>0 o& w* q: ~: N, W! c
<property>
7 ?+ q1 D0 ?) D2 r! ?3 ~ <name>mapreduce.client.output.filter</name>
% [' T! O- n$ c- d( r2 s0 g <value>FAILED</value> d8 p5 m0 R2 O) @& a
<description>The filter for controlling the output of the task's userlogs sent
( C$ @+ j4 D0 T1 C* |$ J1 | to the console of the JobClient.8 W+ v( K9 i: |, Z/ V* q
The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and
) m; q( F( X$ j" S+ e& m ALL.
- @- K* a; v& g' L+ B </description>
0 q9 n5 I7 u0 ?6 E3 B</property>7 H/ b/ }& [2 p
<property>
$ a/ e$ V3 t0 Z/ ^( ] <name>mapreduce.client.completion.pollinterval</name>
3 d! `5 L1 r, b5 y7 f <value>5000</value>
7 O- P4 O6 A# M' p" t <description>The interval (in milliseconds) between which the JobClient' u! B$ R# U& z" X
polls the MapReduce ApplicationMaster for updates about job status. You may want to
( [/ ?6 R' x: c& l6 k9 g$ E7 s2 s- E( Y5 ~ set this to a lower value to make tests run faster on a single node system. Adjusting
! D; W6 k: p. ^$ N this value in production may lead to unwanted client-server traffic.
3 }' T- d x4 Z2 _" y </description>) F1 \+ R1 m' D; N! s5 \/ A( z/ n; a
</property>
[$ [1 O' G6 l i' l5 r <property>
$ i& v( C- s: l <name>mapreduce.client.progressmonitor.pollinterval</name>* t* V1 P9 b+ O: e, W- z
<value>1000</value>
0 S1 c6 ^/ b& G$ d: E2 W% q! ~ <description>The interval (in milliseconds) between which the JobClient
( W. y- d0 ~6 K3 q reports status to the console and checks for job completion. You may want to set this; H8 b( s: V, B, r% G5 v, S& d
to a lower value to make tests run faster on a single node system. Adjusting
! m" t. g9 e9 H6 v: l& b this value in production may lead to unwanted client-server traffic.
+ }" s5 v$ q+ A; m" I4 R </description>
* \1 m9 s- f( x+ z; i. b* P6 L </property>
$ F( X8 B x! t: d$ F# E! x% S2 l <property>
[( B# B8 l6 ~+ `! O2 V t: o <name>mapreduce.client.libjars.wildcard</name>- o. ]% o% K# h8 y% t' d, ]
<value>true</value>
) p X5 y# n$ ]. Q, u4 Q! E6 y6 C <description>
, H& ]6 L' z% z# H; O% C Whether the libjars cache files should be localized using
9 m. A2 e0 Z% J a wildcarded directory instead of naming each archive independently., m: v/ H, n3 L! v9 C9 z
Using wildcards reduces the space needed for storing the job" _ Y3 \# Y7 \3 c% f$ r' L
information in the case of a highly available resource manager' \: E# R: Y7 m
configuration.
0 ?! b6 E" A2 m$ u1 \ This propery should only be set to false for specific0 M7 R( q) v' V/ |
jobs which are highly sensitive to the details of the archive% l5 z" P2 z; s
localization. Having this property set to true will cause the archives
0 o. O6 R( A8 L to all be localized to the same local cache location. If false, each. R1 B r9 G7 R- E: G; j
archive will be localized to its own local cache location. In both
$ K) m$ o% L6 A, _* ~6 ]* \ y: ~ cases a symbolic link will be created to every archive from the job's
! a X" J' e% Z" s4 j A working directory.
6 }9 h( r& ?" m </description>! k L+ P2 \' R. ?, f
</property>
, S, a6 G2 w+ S4 {5 v6 I2 \3 T4 d <property>. z3 I$ ^' g0 S3 [' W5 n- E
<name>mapreduce.task.profile</name> G& H, `# ~. g5 e
<value>false</value>
" U3 I# n: Y3 M) G8 P <description>To set whether the system should collect profiler9 k6 q& q m, P' P7 }0 Y( i( N1 h
information for some of the tasks in this job? The information is stored3 ? W0 s" Q, v* r, g0 l7 l- `) Y
in the user log directory. The value is "true" if task profiling4 t2 R! P) x5 O6 g) H
is enabled.</description>
" l5 R' S) _# o! O) B: }. B: N </property>* c9 k8 @$ R0 _" T% J5 z5 p
<property>: G7 w" j& N' f1 X' k
<name>mapreduce.task.profile.maps</name>3 w. d5 @: u4 U
<value>0-2</value>
5 [/ p' [& M3 z+ k <description> To set the ranges of map tasks to profile.' `$ G3 q8 u6 Z+ t- P6 t9 Z
mapreduce.task.profile has to be set to true for the value to be accounted.9 T) V2 L& i" V$ V7 K2 J
</description>
" f5 D4 Y$ V0 Y( [4 v </property>0 M$ M/ s ^7 t2 S0 j9 q: K' ]
<property>+ ^ ]& E& p; B% S) V- ^
<name>mapreduce.task.profile.reduces</name>) ]. ~" r, a+ N1 I; ~
<value>0-2</value>& _ w8 H( q! y/ C
<description> To set the ranges of reduce tasks to profile.
. L. J' {9 \) g8 |6 o* C3 a0 s8 l mapreduce.task.profile has to be set to true for the value to be accounted.
9 R- X: r& O( ~* t1 ?3 ~ </description>
, A: ~! N1 |4 g" i' w </property>
% o0 ?+ d d( a5 G+ T <property>
$ A) \8 D& L( P* v& E4 I4 A+ F" N <name>mapreduce.task.profile.params</name>, k2 m- m2 D% i& ^% F
<value>-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s</value>
3 Y8 D% A' x! }4 |* x <description>JVM profiler parameters used to profile map and reduce task! ~7 b/ ^, Y0 S3 u; I
attempts. This string may contain a single format specifier %s that will
: m" f- \" j6 N' Q; \& g be replaced by the path to profile.out in the task attempt log directory.
. W8 d. z3 G; s- S" {5 \9 k9 k To specify different profiling options for map tasks and reduce tasks,
9 `& C/ B. o) y! G more specific parameters mapreduce.task.profile.map.params and4 s# N+ s6 N' T- {2 k& q& R
mapreduce.task.profile.reduce.params should be used.</description>% h% l/ t9 b r) U9 M
</property>
5 a2 h1 @* b8 M; v <property>- F5 J' u+ `2 q5 s: b0 n
<name>mapreduce.task.profile.map.params</name>
8 s0 @& {* s3 p <value>${mapreduce.task.profile.params}</value>
1 l* B* p6 [2 m* V4 Z <description>Map-task-specific JVM profiler parameters. See
' E' A7 {' K; \6 Y" n mapreduce.task.profile.params</description>: @; t( N0 p- J
</property>
/ c6 Q; Z+ N1 ~4 H- s1 w0 E <property>
$ ^9 l4 P, O3 ^ <name>mapreduce.task.profile.reduce.params</name>
- n# N8 E4 y$ ^ a' o8 e <value>${mapreduce.task.profile.params}</value>
! t: B! v o7 j, f, W <description>Reduce-task-specific JVM profiler parameters. See
9 D' u* A/ l+ {: E mapreduce.task.profile.params</description># {' m" D6 ], H# a7 o8 t6 T) T: x
</property>( t3 y0 d, p- o$ ]7 O1 C9 i
<property>& A; j) g$ M# O
<name>mapreduce.task.skip.start.attempts</name>4 p2 V9 w/ s0 U$ E" m
<value>2</value>
* N7 i, U! B1 E3 q" W <description> The number of Task attempts AFTER which skip mode
9 |4 Q4 U: ]; F1 n6 a: u will be kicked off. When skip mode is kicked off, the
9 L1 F# \. P2 r" c, o, |# w tasks reports the range of records which it will process! ^8 L: g6 R5 T. ~) Q
next, to the MR ApplicationMaster. So that on failures, the MR AM0 V3 H% y$ k4 X. L: Y1 A8 o
knows which ones are possibly the bad records. On further executions,* t) k7 ~; \0 R& z7 c
those are skipped.
" }8 r2 q0 U0 G$ Q </description>& E) x/ y! X" Z! v# s
</property>
9 w' C( J9 G) {# x. x9 ?4 r8 }" j <property>
9 L1 o$ d' H6 X( y5 [3 ^ <name>mapreduce.job.skip.outdir</name>
' Z8 v* P- ~* N# S- p: B8 ?, M <value></value>
5 \- W0 h3 Q5 O Z* D <description> If no value is specified here, the skipped records are
7 U' o* A! _ E2 d0 `4 T' E$ l written to the output directory at _logs/skip.
' Y7 R B; n' n. S% X User can stop writing skipped records by giving the value "none".# {, Y2 _& C# f1 M7 j% y. _
</description>
$ N+ C+ v# c/ U$ C, d6 ` </property>
/ h. q4 L" L( a <property>
6 [8 F4 ^3 S0 G* r1 [4 P( B* H6 z <name>mapreduce.map.skip.maxrecords</name>( k1 s5 f4 P- O- \
<value>0</value>
$ a3 w; @+ ]) y: O s7 y <description> The number of acceptable skip records surrounding the bad5 c4 i& v! q& B
record PER bad record in mapper. The number includes the bad record as well.- J' k9 X: V/ E, |' ?: k7 }
To turn the feature of detection/skipping of bad records off, set the
2 ^1 `/ ^. s! N- E value to 0.3 p9 Z: q* r& u4 E: U. ^) L# a1 _
The framework tries to narrow down the skipped range by retrying& R5 K9 n; x1 M1 P
until this threshold is met OR all attempts get exhausted for this task.
, L& y* o& U! r4 l5 s Set the value to Long.MAX_VALUE to indicate that framework need not try to
% q. R! `. F, D( N narrow down. Whatever records(depends on application) get skipped are1 Z/ q* L; G+ f8 ]9 C! V( X- N
acceptable.1 Q3 l& m% V, w- a7 ~) b/ ]5 |8 h& h
</description>
4 P/ o& t9 Y( Z# g5 d: @1 y* N </property>3 Z# `4 L9 `2 J3 t8 {% F. s
<property>
% R4 ]: S2 }/ t: G& Q! ] <name>mapreduce.map.skip.proc-count.auto-incr</name>
, ?5 J9 @* I: Y( {; _, ^% a <value>true</value>- D' O0 v7 p- c$ Q) s
<description>The flag which if set to true,
" t L( y3 h9 } SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by
; L7 N: J+ ]( @: W$ |' R9 N- B MapRunner after invoking the map function. This value must be set G- \3 g( ^% u1 ~; y
to false for applications which process the records asynchronously
! Z0 Y0 v* j4 d/ ?6 o or buffer the input records. For example streaming. In such cases
" E- {" V! j$ _( C8 E2 G9 R% R* K applications should increment this counter on their own.
- W: }( w5 y) j </description>
1 A, F8 a; E2 { </property># B2 O; V! Z j2 Q" Q
<property>2 Z8 S- B* n' G$ M; C
<name>mapreduce.reduce.skip.maxgroups</name>
0 V' `% b0 R* I- L$ W- X# d <value>0</value>
f( l- D3 @7 R3 V0 ]) I <description> The number of acceptable skip groups surrounding the bad; T/ X2 O; t+ S7 g
group PER bad group in reducer. The number includes the bad group as well.
4 s5 L( ~4 d# y* y! S, D# p To turn the feature of detection/skipping of bad groups off, set the0 K5 n7 u5 D! v4 `! p. ^' Z; y1 N' y, Y- e
value to 0.
& ]2 e7 I4 T5 w- `' o The framework tries to narrow down the skipped range by retrying# |# T" m* A+ M
until this threshold is met OR all attempts get exhausted for this task.
2 P$ f1 s9 z7 B9 d8 }! p Set the value to Long.MAX_VALUE to indicate that framework need not try to
, S4 t7 t+ j C: }! w narrow down. Whatever groups(depends on application) get skipped are# Z* `3 T% d& P8 A
acceptable., J u, g% g _% z# ^
</description>
0 a4 ?* [6 d. q+ Q2 Y0 v$ y </property>6 S; e* X+ S* N
<property>
% b; p. Y% U, p# F4 Z8 \) M <name>mapreduce.reduce.skip.proc-count.auto-incr</name>- q: S8 U6 v, q
<value>true</value>+ D! P/ D) a! d: ~6 z
<description>The flag which if set to true.
" l5 l8 N1 p1 q0 \ SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework
1 G! C; l% \5 o5 L6 d after invoking the reduce function. This value must be set to false for4 q6 o' {) Q+ K' S' m# Q' H: |
applications which process the records asynchronously or buffer the input- F: L' D' }( E' D
records. For example streaming. In such cases applications should increment
% q" Z& V3 U3 G, O4 ^6 Y this counter on their own.: l# b/ | o: x7 p; c. z% L
</description> c- h& a$ Y9 c9 H
</property>
7 v& l# S1 {3 K N3 ^$ n$ A <property>
7 r& r/ V$ {6 x) C) C+ G n2 |' ~ <name>mapreduce.ifile.readahead</name>
3 S, I3 ?6 F8 c* S <value>true</value>
' U- Q5 v( f8 L3 f% x2 w/ p) I <description>Configuration key to enable/disable IFile readahead.' X9 L3 V D. H* @# x3 J; }( M
</description>
& C1 E8 Y+ z; ~2 T </property>
7 V# w/ d9 _. _; ]8 |( Z. f <property>! @% I. J5 w, U5 C; o
<name>mapreduce.ifile.readahead.bytes</name>
. Q' [9 [; A. ?- ~9 }" F p7 y <value>4194304</value># D+ R0 E: Z8 a a8 I6 T4 N; Z; s
<description>Configuration key to set the IFile readahead length in bytes.7 p0 A" ~/ H& W3 E
</description>7 @2 {1 d. R! a1 V/ P( s
</property>+ q7 ]9 O) o, K' l# y
<property>
' d) e1 N% Z% p* i <name>mapreduce.job.queuename</name>
: {9 s1 w+ E; \) m3 D' `0 P- C <value>default</value>% o7 [; I* @7 t2 Y7 X
<description> Queue to which a job is submitted. This must match one of the
6 A( {$ L5 J2 b: e0 ?- F queues defined in mapred-queues.xml for the system. Also, the ACL setup' w3 [; t# Z4 ^6 ]
for the queue must allow the current user to submit a job to the queue.
. p1 E6 R+ R: C; z& b& A Before specifying a queue, ensure that the system is configured with4 j, I/ d. P* `1 z" Z; V
the queue, and access is allowed for submitting jobs to the queue.
4 c$ Q& J: G5 \. _6 a. U </description>
3 B( \5 @. R2 {</property>
4 Q' I4 J- i* p& } <property>1 G' B/ H1 w4 W, V* n) f( T
<name>mapreduce.job.tags</name>
" g3 s$ H. a, G( Z7 K <value></value>' J' D4 s# g4 H! s1 B; S- d% V
<description> Tags for the job that will be passed to YARN at submission
6 d- l4 Y( E- [+ h+ U time. Queries to YARN for applications can filter on these tags.4 `. M$ l3 Z9 ]$ h+ Y a# t
If these tags are intended to be used with The YARN Timeline Service v.2,
# Y5 Q: a& n5 O H0 K9 L5 G( e) D* K prefix them with the appropriate tag names for flow name, flow version and
* r- @6 |, y' O+ z3 o flow run id. Example:% T. F) L M# Z8 h! a
timeline_flow_name_tag:foo,9 U8 u+ U$ h8 p$ D( D4 M# x% S0 _' w
timeline_flow_version_tag:3df8b0d6100530080d2e0decf9e528e57c42a90a,/ u& n6 A3 j$ y4 c: ?6 k1 P# \
timeline_flow_run_id_tag:14652463485990 i0 z3 i/ ]& m$ z! P
</description>
8 K' v% D3 l9 I1 A </property>, p5 `# H5 b0 {6 z, v
<property>
/ k% N! Q( F1 L. n. n8 K, n <name>mapreduce.cluster.local.dir</name>% o) ?- E! T. c# X
<value>${hadoop.tmp.dir}/mapred/local</value>* f3 W% z2 E6 V: @3 T, \' S
<description>6 P7 z7 e9 B: j$ {4 @5 \5 K
The local directory where MapReduce stores intermediate
* M% r. w9 f( t. a( [+ H6 K% | data files. May be a comma-separated list of7 X9 _$ _+ B0 m% ~
directories on different devices in order to spread disk i/o.% I( ]* ^+ T: t8 m' Z0 h
Directories that do not exist are ignored.( v' g1 y5 o( ?
</description>
# @. k, E/ x+ S! @</property>
% O+ C) A1 {" J5 }3 ]( G/ {- A. g<property>
" M# z0 R% B5 v; C <name>mapreduce.cluster.acls.enabled</name># H+ j( `+ d8 U ~1 x7 n E
<value>false</value># D% D( \- G7 g) I3 k
<description> Specifies whether ACLs should be checked
) h! T8 h* B9 T1 r& P for authorization of users for doing various queue and job level operations.1 Y; {& m+ ^3 t" ^; h# k
ACLs are disabled by default. If enabled, access control checks are made by, H5 ]0 W, [+ i$ v7 e1 ?4 Y
MapReduce ApplicationMaster when requests are made by users for queue
, s! w+ t$ M9 q" E6 O$ V/ o; H9 ` operations like submit job to a queue and kill a job in the queue and job' o& v; U# V6 k2 T: b5 Y. W
operations like viewing the job-details (See mapreduce.job.acl-view-job)# t8 O% ~- `& {$ W9 t+ |) L
or for modifying the job (See mapreduce.job.acl-modify-job) using
& ?8 p. F! b+ d8 k. \+ l Map/Reduce APIs, RPCs or via the console and web user interfaces.# G) h# ~: [+ J h" _. S/ {* A2 a
For enabling this flag, set to true in mapred-site.xml file of all
2 z" _0 v8 @; T7 i% S! n/ a MapReduce clients (MR job submitting nodes). e# A- s) \3 l8 _7 ^( ^! j0 X& [9 {: L
</description>" h* e- V9 z% r- i$ G: G. I8 N
</property>
7 y t) q- [' u1 V9 f+ M1 y# C<property>
, I) y; r: S* A- @2 V! e <name>mapreduce.job.acl-modify-job</name>
$ [9 Y1 p$ G7 |2 d* V9 } <value> </value>
5 b: }$ K, d2 `0 P. h6 d- b <description> Job specific access-control list for 'modifying' the job. It' N- h+ x# g( w, R* m& v' |& v
is only used if authorization is enabled in Map/Reduce by setting the
% }" R7 q. _' N V- ? configuration property mapreduce.cluster.acls.enabled to true.+ \+ I; Q1 N7 K8 y
This specifies the list of users and/or groups who can do modification _$ ~$ N# o' r" [# ~: H; q( W. x3 ]
operations on the job. For specifying a list of users and groups the( I; }+ e8 e- M0 v
format to use is "user1,user2 group1,group". If set to '*', it allows all
, k1 D! y+ U! W. M# e. G+ \ users/groups to modify this job. If set to ' '(i.e. space), it allows! ^$ A7 n" `( X6 o4 v6 }
none. This configuration is used to guard all the modifications with respect
6 r( {/ b) { N to this job and takes care of all the following operations:
_. r N c, J5 f4 u o killing this job& q [0 O2 e! }$ O
o killing a task of this job, failing a task of this job
6 B" H# ~( W7 \/ `+ y8 ]* c o setting the priority of this job
6 A) m4 x+ S* t% w6 r" Q- X Each of these operations are also protected by the per-queue level ACL
# \3 ^5 i9 k) y* u "acl-administer-jobs" configured via mapred-queues.xml. So a caller should7 k1 j2 l2 v' Q" P% I7 T/ ]
have the authorization to satisfy either the queue-level ACL or the2 l8 Y6 i7 }* R# b" I
job-level ACL.
6 ~* P) @/ A+ P d Irrespective of this ACL configuration, (a) job-owner, (b) the user who
$ R( x( ~6 m5 E2 F1 p started the cluster, (c) members of an admin configured supergroup
' {! y- i! v+ ^7 E6 Q {; l configured via mapreduce.cluster.permissions.supergroup and (d) queue
7 _ C+ @# W( r( ] administrators of the queue to which this job was submitted to configured- ~# I6 \! i+ h$ s2 [6 i8 K
via acl-administer-jobs for the specific queue in mapred-queues.xml can
0 q2 R' x/ K% m do all the modification operations on a job.
; b. i! l2 U/ b" y* S+ U/ t By default, nobody else besides job-owner, the user who started the cluster,& `# m) T6 G* g) D
members of supergroup and queue administrators can perform modification
8 n! ~/ y1 U( D- Q operations on a job.
9 j. J* f' n: t+ n </description>
6 g. F, ?* }, w& z" u</property>; N! z: U1 I) \
<property>0 H/ V, ?! y* |; s' D
<name>mapreduce.job.acl-view-job</name>
" j" k0 T4 I% g& K5 X7 W4 C9 F <value> </value>
* b( `; u0 x9 S2 z- ~ <description> Job specific access-control list for 'viewing' the job. It is2 o1 }# {; B, V: L, i& D6 {& h, u
only used if authorization is enabled in Map/Reduce by setting the
8 z! m1 R1 p; s/ y8 Q configuration property mapreduce.cluster.acls.enabled to true.
0 f3 {7 Z5 z+ v! u7 C. ^' F This specifies the list of users and/or groups who can view private details
4 I+ u2 J* H3 a1 w% e about the job. For specifying a list of users and groups the
/ W/ s# J- S2 p1 m$ z- v* H$ Y M format to use is "user1,user2 group1,group". If set to '*', it allows all
$ d8 M B, t q0 `0 R8 \& [ users/groups to modify this job. If set to ' '(i.e. space), it allows
$ s! I: P# `6 z) r. d none. This configuration is used to guard some of the job-views and at
4 @/ t9 {- b+ W! x# n6 e/ ` present only protects APIs that can return possibly sensitive information
5 d% \4 h! `" i- Q1 n of the job-owner like
. p0 l( Z: ?$ k9 q% `0 ]+ F o job-level counters- ?8 d* s( W, |# S
o task-level counters
5 y, g) I& p' T8 y% V! @" I o tasks' diagnostic information
3 G4 M& o4 l7 @6 R) {6 [ o task-logs displayed on the HistoryServer's web-UI and
3 N+ y# N) a0 I. l, V* r o job.xml showed by the HistoryServer's web-UI- U. |/ D9 b3 u
Every other piece of information of jobs is still accessible by any other
# R# A* u9 g$ C. W user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc.0 k8 D) _; K5 ^- E4 [
Irrespective of this ACL configuration, (a) job-owner, (b) the user who
3 v3 k# v. I8 M. f- `( \+ Q# Y2 N started the cluster, (c) members of an admin configured supergroup$ k# [3 u' \/ r
configured via mapreduce.cluster.permissions.supergroup and (d) queue0 u; U8 x4 ~, o
administrators of the queue to which this job was submitted to configured
7 N1 c$ G. k* U) Q: [; O Y" x, Z& ] via acl-administer-jobs for the specific queue in mapred-queues.xml can
3 v7 _( b" L1 Z5 M. h% a4 \' v4 y+ b do all the view operations on a job.
% x4 |0 c( H1 x; q2 h- h6 t By default, nobody else besides job-owner, the user who started the4 _" y5 E" Q5 Y. D; [
cluster, memebers of supergroup and queue administrators can perform
4 ~( F# }3 }! D! M( E view operations on a job.4 |) }6 s7 [ `
</description>
9 { l) L& B( U7 K</property>9 O4 ?9 b# N& h5 R. @* U* P
<property>5 E8 t3 T3 |0 v3 w3 A& S9 @8 u
<name>mapreduce.job.finish-when-all-reducers-done</name>. d! H+ v+ N8 b4 `1 W# _
<value>true</value>- V; n/ m U ~$ n ~( g" g
<description>Specifies whether the job should complete once all reducers: s) B& B/ L2 o" `
have finished, regardless of whether there are still running mappers.' q4 H. O# j# g, v+ O5 `: Z
</description>
0 y1 y: s8 U2 y& {8 b</property>8 j! x0 O2 J6 k3 c" d% M9 n }
<property>
/ N8 ~+ Z9 P! j# Y/ q <name>mapreduce.job.token.tracking.ids.enabled</name>
3 k' [! k" o6 K7 q( P6 ?! ?$ H <value>false</value>1 b! j, D4 T0 T( O/ P! O
<description>Whether to write tracking ids of tokens to
4 B& W- n" w2 \! H1 m% _ job-conf. When true, the configuration property i# Y! |& G9 x$ Z, c) U! _+ l
"mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of; c7 |, i$ ]; J: i
the job</description>
8 ~9 y" D7 ^6 M</property>! h" `5 s) e, \# H( Y! z4 a9 O) }
<property>" V; ^% U: B$ ~4 J* Q
<name>mapreduce.job.token.tracking.ids</name>: t; v8 i( E3 G! N. O2 p
<value></value>( W5 C j' w1 o# d- C
<description>When mapreduce.job.token.tracking.ids.enabled is
/ G# a/ s! h6 z# F set to true, this is set by the framework to the
- W8 f5 |, _) ^2 T token-tracking-ids used by the job.</description>
3 t' K+ _+ z1 B; g9 ]4 T</property>
* T9 U% l/ T# b* U. a; [! o( ]<property>( r. L0 |, `' l- {4 G5 w8 \& `6 T
<name>mapreduce.task.merge.progress.records</name>
: T6 }! v1 i$ R5 m9 ]; O! b <value>10000</value>3 h6 c3 |; S* G9 n6 ~
<description> The number of records to process during merge before. R6 m9 o @# t5 _
sending a progress notification to the MR ApplicationMaster.
7 a" \4 j* m/ P* X% W8 o. I9 C1 m </description>. D0 t; c& u, @% C/ [
</property>
! |4 T6 X, T/ i m6 K( l; l<property>/ {/ t5 p y7 y1 h+ V- ^
<name>mapreduce.task.combine.progress.records</name>% I& o T) w& t- u0 D( r, |7 O0 A/ s
<value>10000</value>3 X7 k: O% Z3 Z' o4 K
<description> The number of records to process during combine output collection
6 Y) k7 @" I) {; h: x3 Q* j2 v! F before sending a progress notification.! Q- F+ I0 i2 }
</description>
7 C- U( E7 U) w5 k) u. u</property>3 E' m- K. t/ A& d: Y
<property>7 S E8 W5 z! U$ W% x. m. `
<name>mapreduce.job.reduce.slowstart.completedmaps</name>7 {; ?% m4 S, w, B0 C
<value>0.05</value>8 V' m* R% b; F# ]6 v. j) P
<description>Fraction of the number of maps in the job which should be
) @3 S4 H& ~. b' M) g3 e: g' w9 ?8 ^ complete before reduces are scheduled for the job.
' t. z) m8 k5 U! p5 K9 @, c# |3 L7 h </description>- ^9 |$ p. [& V& p
</property>
* v, D- w$ k# ?/ g<property>: Q4 x. @: P% b
<name>mapreduce.job.complete.cancel.delegation.tokens</name>2 d# _) L$ B; l/ R0 G a
<value>true</value>
5 E7 n: g3 v" s8 L, k <description> if false - do not unregister/cancel delegation tokens from7 S9 Y9 p2 n# j# }' f7 G
renewal, because same tokens may be used by spawned jobs! C! e& N# B$ a: p6 \- D
</description>7 e0 C" Z; ~+ x7 z
</property>
" H( P. E( \+ Z<property>
- A9 m6 n+ O% | H, {: m& ` <name>mapreduce.shuffle.port</name>
) v; s0 P) g6 j4 g, o. K- j <value>13562</value>
% c6 T1 P. }* _2 _) A <description>Default port that the ShuffleHandler will run on. ShuffleHandler. }7 p4 Z. v& r$ t0 _7 @ E$ n0 |: o
is a service run at the NodeManager to facilitate transfers of intermediate1 ?+ T3 t4 u! U
Map outputs to requesting Reducers.
2 a: P: H0 E- j </description>
4 _; N6 [5 S- {8 J: Y, i! B</property>3 `+ m3 e3 X6 f# [6 a
<property>
: U% ?9 X3 }& O. F# g% E <name>mapreduce.job.reduce.shuffle.consumer.plugin.class</name>
1 J; l6 R& L2 O Y9 n+ p0 ]# h' g <value>org.apache.hadoop.mapreduce.task.reduce.Shuffle</value>7 Q5 K7 V7 H+ U3 b; F
<description>8 w5 c, p+ R# v, `
Name of the class whose instance will be used
% y1 G, O' I, ?, U' n6 p& b5 v! _ to send shuffle requests by reducetasks of this job.1 r; |/ b4 {' E, @, o8 F& x
The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.$ l: \4 q. ~2 G) a
</description>
2 [; p3 b. u+ T0 f</property>9 N @7 y7 J1 X' F/ l% o
<!-- MR YARN Application properties -->
/ W+ K5 a. H0 o* a/ a<property>
9 i: ? l, Q, J, r2 q- V! P' J <name>mapreduce.job.node-label-expression</name>
. Z- R/ ^% V- }8 B <description>All the containers of the Map Reduce job will be run with this: i `; H: s1 g" z. M
node label expression. If the node-label-expression for job is not set, then
) U6 `( _% ] q" ` it will use queue's default-node-label-expression for all job's containers.' f* q1 x3 e6 V
</description>
5 [. ]; m1 T$ ^/ P# i</property>9 v. B( j( t; |) j# T. D
<property>
% Y2 R9 U$ q& T& a4 s; d# z, P <name>mapreduce.job.am.node-label-expression</name>
$ H2 W" @' d, U <description>This is node-label configuration for Map Reduce Application Master
4 c& `; Y: a6 s) c* i container. If not configured it will make use of
1 {. w2 j9 W0 b5 p& b( O$ C mapreduce.job.node-label-expression and if job's node-label expression is not( L1 U8 I; z0 k- b, [" C
configured then it will use queue's default-node-label-expression.0 f' C. l8 s- _5 D) \# K; M
</description>
" z% F: t p9 k, x. Q' s, \2 V</property>
; {& L8 L& w+ i# z: |9 K- ?9 ~3 z' G<property>
6 E3 X; H1 t! C& [9 l; u9 Z <name>mapreduce.map.node-label-expression</name>; q8 n$ L: ]" }7 g. |& D& R
<description>This is node-label configuration for Map task containers. If not
- @! g0 A. A3 I configured it will use mapreduce.job.node-label-expression and if job's2 {: V( y4 E( X/ f
node-label expression is not configured then it will use queue's) ]2 m8 k0 r+ H2 L% C# Y. H
default-node-label-expression.! w- `' M: B1 ?- K+ }+ f
</description>
2 B+ c( S+ S; v</property>
$ P) P% J" T; @% d3 q0 F0 @& h9 K<property>
% X' a( c, a w* q8 k) w9 M <name>mapreduce.reduce.node-label-expression</name>
. J3 ^ i' w+ _. d <description>This is node-label configuration for Reduce task containers. If# d0 C9 ~8 c* }% [+ K
not configured it will use mapreduce.job.node-label-expression and if job's
. u. A) K3 g0 D6 m+ r. e node-label expression is not configured then it will use queue's
* O ^8 j l9 g/ g default-node-label-expression." t, r& X. a4 y/ ^6 S
</description>
; n" I+ X. u1 L0 M</property>) Y; P& Z# [3 }# m# i, v0 d
<property>
7 Y2 X m* r4 T, }8 a0 N1 o# i <name>mapreduce.job.counters.limit</name>
+ ]1 }* `) u5 s; S <value>120</value>
& u* C/ E* ?& L& \ <description>Limit on the number of user counters allowed per job.
; B- W |: k! H# Q; v6 o l </description>
; ^/ v& \ [6 u</property>
. n; R3 H! U0 P9 ?1 d<property>
/ Y" d, A9 {+ R" }) S9 o# m <name>mapreduce.framework.name</name>
, r6 V6 e* {1 V& w' l <value>local</value>
: f& q, |+ A, K, d. p <description>The runtime framework for executing MapReduce jobs.
7 T/ x( j% ~7 Q0 F, w- w Can be one of local, classic or yarn.0 R. W: U8 f9 l _5 y0 H
</description>6 e8 _1 k9 z" | p4 q+ Y) V
</property>( ]& j* P% _. S" }5 K
<property>
* H* `4 P6 O# g! d# X i1 w, F <name>yarn.app.mapreduce.am.staging-dir</name>! \* x4 x# X* v+ f1 N3 @1 Z
<value>/tmp/hadoop-yarn/staging</value>
d# M ?, R% |- j' g5 W <description>The staging dir used while submitting jobs.. g6 s- [8 i9 ]
</description>4 C. p. T I$ B# V2 k9 b8 Y
</property>
- }/ u( H( e4 H; A9 f<property>
/ c8 p3 X/ x9 c, `* V6 k4 I( I <name>yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled</name>$ y$ f9 N5 S/ I
<value>false</value>
- e% N4 n/ p: o6 B3 k0 B+ u <description>Whether Erasure Coding should be enabled for
: y# L; G2 I. L/ H* P5 P5 O. n files that are copied to the MR staging area. This is a job-level
]# z, H- d/ L/ C setting.$ v, P `& O% z! r; D$ W" s
</description>
" j f7 u7 i+ f! G! ?7 {</property>+ f: }! B$ u! N4 g$ B
<property>
y* w, c' S# e# @# O t( I <name>mapreduce.am.max-attempts</name>" E6 ]0 W x2 s0 \) C$ U
<value>2</value>
+ J2 |& m4 e ?: S: [+ _) y: p <description>The maximum number of application attempts. It is a6 G/ e, k# k' t" i) A
application-specific setting. It should not be larger than the global number8 d: z' O3 j* h. B l9 V
set by resourcemanager. Otherwise, it will be override. The default number is
2 V% V" f9 Q I4 U set to 2, to allow at least one retry for AM.</description>
+ b$ e" k, y- z! l* y* A</property>2 h0 E ` `. j: i5 P" F
<!-- Job Notification Configuration -->
4 `% C ` f# U' ]/ Y0 _1 @<property>
4 g/ Q( i3 z3 g% ~2 f) a d <name>mapreduce.job.end-notification.url</name>
* ]' Y: u6 a' `! u <!--<value>http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus</value>-->
: p) A# g# Z1 ~ <description>Indicates url which will be called on completion of job to inform. l: m- W% c( Z' F0 K, l9 |
end status of job.: U' |1 n% p3 Z% F8 [( Q, b. \- F3 u
User can give at most 2 variables with URI : $jobId and $jobStatus.
5 H/ Q9 D$ C4 @/ _& r f If they are present in URI, then they will be replaced by their8 N9 K& }/ H+ W) t5 E/ I$ q. Y% |
respective values.8 u* Q" E& }& q8 {7 z* x
</description>
# k0 z {2 L1 Y</property>6 A1 w( a7 s1 f! h' f" ?
<property>
7 [7 V1 x& C" s" ]* X. y. ~ <name>mapreduce.job.end-notification.retry.attempts</name>2 I7 F* c2 r5 z' r" Z
<value>0</value>; p* s/ H( u+ Y2 a& ]; m$ }, U
<description>The number of times the submitter of the job wants to retry job% C3 h" a1 k* P3 Y3 _
end notification if it fails. This is capped by
0 ^2 V4 Z/ C- v7 o& [ x; k9 | mapreduce.job.end-notification.max.attempts</description>
; Z- h3 d) a% F4 C: ^</property>
. Z2 t* D9 p5 `/ y d<property>
* N7 y* w9 f8 _! ~6 d <name>mapreduce.job.end-notification.retry.interval</name>
+ H* G, Z4 o7 ^0 b <value>1000</value>+ u- _, [! W/ `5 j( k/ A) f, U7 p. U
<description>The number of milliseconds the submitter of the job wants to" ?- O: _3 ?& u* Z3 S
wait before job end notification is retried if it fails. This is capped by
9 N% t: T' A- F6 A7 [( m, L N mapreduce.job.end-notification.max.retry.interval</description>) e2 L& b- v) a6 d5 D
</property>
* f' A& ~8 H* i6 N( \5 [<property>% }9 }. R% b7 j7 Z
<name>mapreduce.job.end-notification.max.attempts</name>
) L- t0 d( x) |7 p <value>5</value># ]' V9 x5 m& B! `% Y/ m
<final>true</final>
2 A7 I% O$ j& f4 h4 O <description>The maximum number of times a URL will be read for providing job9 O6 Q' x) `% m7 C" s& \- N, Q
end notification. Cluster administrators can set this to limit how long
2 I8 q5 W3 i Y3 [ after end of a job, the Application Master waits before exiting. Must be
6 d3 q) X! V' n2 X marked as final to prevent users from overriding this.
9 H& A* f# ~, l! U n- ` </description>
0 S7 B9 i' H7 o% ]- } ?- l</property>
1 ^4 W; F# I0 C! z <property>" b& @0 \2 M- T; _
<name>mapreduce.job.log4j-properties-file</name>
% H- S. H9 p5 f6 ^, ~ <value></value>
! N- @) D" |7 ]' Z' S <description>Used to override the default settings of log4j in container-log4j.properties
) E- n! ^, j. E9 A( b. P1 }+ h8 i for NodeManager. Like container-log4j.properties, it requires certain
/ T7 u* W( u# a, C+ i6 [ framework appenders properly defined in this overriden file. The file on the0 t8 ]6 U) @7 J( }( S+ Q' n! q
path will be added to distributed cache and classpath. If no-scheme is given) i4 g& t! Z5 O# R
in the path, it defaults to point to a log4j file on the local FS.
* M6 P3 J# }7 g& m7 \* m/ a: m </description>. S0 \# b2 p) r% s# N$ c3 U8 F
</property>- a0 F! V/ X) z9 m
<property>
% G) w: {! J9 ? <name>mapreduce.job.end-notification.max.retry.interval</name>( i( ]4 T/ ?- J
<value>5000</value># `. F* `" _5 S) f6 k
<final>true</final>
) P- @% h. k x7 C9 a* _+ y <description>The maximum amount of time (in milliseconds) to wait before9 I1 S" A3 X& p7 k# V* l' O+ W7 Y
retrying job end notification. Cluster administrators can set this to. e. f% @+ g. |
limit how long the Application Master waits before exiting. Must be marked5 Y6 v; q0 m, X' A: e0 |& O2 V
as final to prevent users from overriding this.</description>" L0 b" q1 u. V% ?- K5 h
</property>
- h: x d% Y D4 M<property>
: K. L8 g! Q% {8 z1 l4 l0 y <name>yarn.app.mapreduce.am.env</name>9 m r7 P4 h. g; }
<value></value>
( A; ]1 V" S2 j* U <description>User added environment variables for the MR App Master
6 r( w) U% n" y% x9 I/ j; ~. I processes. Example :6 A7 n0 q7 L: J9 j
1) A=foo This will set the env variable A to foo
* N0 E2 O8 ^7 B& s( u8 {) s 2) B=$B:c This is inherit tasktracker's B env variable.
% `8 }/ T. Q$ b7 z2 q </description>! j U* P" B2 A' `
</property>3 R ]/ _+ Y1 E0 {! p
<property>4 g$ r, s. S, D4 Z
<name>yarn.app.mapreduce.am.admin.user.env</name>, ]. s6 r8 b% m% ~" `
<value></value>
1 g. B6 c) a0 {) D- c <description> Environment variables for the MR App Master
+ L& ~; j* {6 G( ?8 s$ g. } processes for admin purposes. These values are set first and can be' O; D% O: a4 R$ s# _6 N1 K
overridden by the user env (yarn.app.mapreduce.am.env) Example :) s& G. m1 N7 U( `- F& E
1) A=foo This will set the env variable A to foo Y& Y0 t' `4 H. @" s: \
2) B=$B:c This is inherit app master's B env variable.- y2 f6 y0 p# _) r, I: w8 W
</description>/ c7 z9 s e6 k% v
</property>
% U2 n- j( \( O$ o<property>
4 `- L, B- y" x+ f9 `! Q1 C <name>yarn.app.mapreduce.am.command-opts</name>8 \) u$ a6 e, Y
<value>-Xmx1024m</value>
/ }' G* {- D0 O$ X) k <description>Java opts for the MR App Master processes.& c4 r6 I7 S, c) n* E) U$ m( x! ?
The following symbol, if present, will be interpolated: @taskid@ is replaced+ y$ G3 x; b) d9 m
by current TaskID. Any other occurrences of '@' will go unchanged.
0 @/ }9 Z4 f0 T1 y For example, to enable verbose gc logging to a file named for the taskid in
- [" J) g; c% P* \ /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:) ?; T+ `: }+ e% N8 Q
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
9 `) S! |, |' \6 a, t' r8 D- k Usage of -Djava.library.path can cause programs to no longer function if! \; ^+ N4 Z/ h
hadoop native libraries are used. These values should instead be set as part6 z9 P% ~# k7 Y+ X* j' k/ ]
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
. q* o# i. u# D$ b7 ~ mapreduce.reduce.env config settings.) ] C$ W2 G+ I# B) {; W* w
</description>3 \9 b, f8 o% A
</property># p+ V% G) u. o
<property>
, |6 r( c# n% e; @7 V) _; O* _. n <name>yarn.app.mapreduce.am.admin-command-opts</name>
( V3 u0 g( P3 S, q- I <value></value>7 R) P$ r+ e; R8 D
<description>Java opts for the MR App Master processes for admin purposes.# T2 k$ D8 _, d2 L4 V" v
It will appears before the opts set by yarn.app.mapreduce.am.command-opts and, A. F( o( I/ j& W& O, m
thus its options can be overridden user.. ]' F% q4 m& O- H z( R
Usage of -Djava.library.path can cause programs to no longer function if
2 D8 I& K G5 g# D hadoop native libraries are used. These values should instead be set as part. B8 K$ N8 F" g
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
1 Z d9 @0 e& f( J$ C; j) f5 Y+ @ mapreduce.reduce.env config settings.. y, \' _' `" y' o% R S
</description>
. C P9 M8 c! l9 K5 B) y</property>& c# {1 @4 N' d8 O3 z( O
<property>: ~ U9 T! Z. J ]: `# e
<name>yarn.app.mapreduce.am.job.task.listener.thread-count</name># K% f/ m4 k7 c+ M) L& L& k2 _7 Q! e5 ]
<value>30</value>
6 v, q, K, ~$ j1 D <description>The number of threads used to handle RPC calls in the2 L2 [' F3 o7 z7 T$ [
MR AppMaster from remote tasks</description>5 q1 U5 d2 q1 w3 b x
</property>' n d- z, q( f/ t
<property>
! X# x% L9 Z/ o- u <name>yarn.app.mapreduce.am.job.client.port-range</name>9 u; W3 f7 [0 k
<value></value>: S; P! I6 n7 T( p' n
<description>Range of ports that the MapReduce AM can use when binding.
; V2 L, y2 o& q1 ` Leave blank if you want all possible ports.
" k. G. q" S% c1 d+ G. K; p) C For example 50000-50050,50100-50200</description>
2 c3 ^8 o& q( V9 e T; k</property>
; J {9 H* ^% Z+ `<property>
; A3 m3 x" V2 e+ ? <name>yarn.app.mapreduce.am.webapp.port-range</name>( }2 m: W8 k6 Y* ^ ^
<value></value>
1 D1 [* A& G3 h# A/ r <description>Range of ports that the MapReduce AM can use for its webapp when binding.
0 J( s! A% C1 Z6 u8 B- v Leave blank if you want all possible ports.$ O: V* O3 J) }4 m) n6 A0 X
For example 50000-50050,50100-50200</description>: h! Z% w! `9 H7 X2 a6 a, [% @
</property>
0 @3 d7 L) u ^/ \<property>6 H& ?- I, R7 M* `& w
<name>yarn.app.mapreduce.am.job.committer.cancel-timeout</name>' |/ | \/ ] u f; A
<value>60000</value>( C1 n7 R7 Y" t9 ^$ r6 ^
<description>The amount of time in milliseconds to wait for the output
6 B) x f$ `, l7 Y+ u# J$ Z committer to cancel an operation if the job is killed</description>
0 Y: Z. i M" \7 Y2 Z/ _+ d/ l5 \" K</property>$ b0 [* K3 X1 r' p/ B6 R- S$ z
<property>: s; w* m9 ~+ q
<name>yarn.app.mapreduce.am.job.committer.commit-window</name>9 l8 p! C5 v1 T3 ]4 A, w( O2 F0 d
<value>10000</value>& I" R" W9 j9 y/ Y2 [
<description>Defines a time window in milliseconds for output commit7 p$ @) f: ]0 @# M! I: y- [
operations. If contact with the RM has occurred within this window then
: H$ A" E4 b6 H% r% \4 s' H0 b( P commits are allowed, otherwise the AM will not allow output commits until3 A4 \% M8 q- [$ [
contact with the RM has been re-established.</description>% \7 A* r: h: l, \' a+ z4 J4 k
</property>
3 l0 b/ J8 x% P" {7 N, |<property>
: p: ]$ P# K, z3 U <name>mapreduce.fileoutputcommitter.algorithm.version</name>3 x. I# o, x, ^# A' i
<value>2</value>
3 \' D& U4 f( [; v3 V- ~ <description>The file output committer algorithm version4 m& Q0 g3 x9 u0 E+ C& K$ x( M2 S
valid algorithm version number: 1 or 2
' G8 `$ d0 Z3 J/ a9 c1 }/ b" D/ h default to 2, which is the original algorithm
% Z% n) u/ E$ }) r$ t In algorithm version 1,, D0 G' W& E4 X, t9 u" P# ]
1. commitTask will rename directory
9 V7 m5 E( {4 D, r, I$ ~ $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
% x$ U o5 e `. a1 q to* F) ^. ~. |' G% g3 @5 Z
$joboutput/_temporary/$appAttemptID/$taskID/
! _2 X$ h; M/ Y9 l) n 2. recoverTask will also do a rename
$ E! c' w, a1 h( R, U' i $joboutput/_temporary/$appAttemptID/$taskID/
3 b. E, @* l/ s* R$ _; o/ R. d to8 s6 J5 o4 Z# E1 q W S5 {
$joboutput/_temporary/($appAttemptID + 1)/$taskID/3 b' o* k# N) z/ S2 o5 s5 Z
3. commitJob will merge every task output file in
7 t* `# }/ Y2 e5 U $joboutput/_temporary/$appAttemptID/$taskID/
. V+ c, _4 e1 ? to
: J$ l4 X, n( `2 P $joboutput/, then it will delete $joboutput/_temporary/
, ?. E" U* L7 g. N- W and write $joboutput/_SUCCESS
7 p! y. O. t. k" K5 @" W+ l8 m It has a performance regression, which is discussed in MAPREDUCE-4815., C) i& r1 T, n6 Y& w; V
If a job generates many files to commit then the commitJob0 B9 W6 c& n) c8 N$ h3 U
method call at the end of the job can take minutes.
4 @* F/ s# P7 F+ e) y the commit is single-threaded and waits until all
6 w- X: w6 z6 {: B2 O) Z tasks have completed before commencing.; m7 q; R, g9 H9 U
algorithm version 2 will change the behavior of commitTask,7 _% j( Z4 ]/ s9 ~9 d/ P ?
recoverTask, and commitJob.
5 l5 u3 n# a8 K& |+ @ 1. commitTask will rename all files in
: w/ h1 l: `$ m) c" ]4 K2 V $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/$ s4 k% I% S& B, J. I! {
to $joboutput/
) r/ U3 z: J7 {6 o; o 2. recoverTask actually doesn't require to do anything, but for8 K; N9 |5 [' k" ]2 w
upgrade from version 1 to version 2 case, it will check if there
. h: j! h8 \$ e$ G" K, p) m D are any files in
+ X _5 Z8 c7 a' w0 H5 ^- w $joboutput/_temporary/($appAttemptID - 1)/$taskID/
" I+ u& y$ ?1 H! U and rename them to $joboutput/
2 \& j' A$ J3 I+ c( X 3. commitJob can simply delete $joboutput/_temporary and write9 Q/ I/ d9 c& h. l% A6 Y+ T9 Q
$joboutput/_SUCCESS
) ]8 e9 w; J. F) J% i1 J) L This algorithm will reduce the output commit time for
5 u' F1 i/ e1 k large jobs by having the tasks commit directly to the final
+ u& R' D' k' e4 M1 S7 s output directory as they were completing and commitJob had" C9 K1 b7 |( N2 B5 N
very little to do.. J7 }8 G1 e8 f/ r ]" v( F! y
</description>
: T: P6 W; ^8 ?6 U& \! O- i; l2 }2 }</property>
% Y% W4 D9 Y/ o: {+ e<property>9 s- i- O( @' a8 p3 s
<name>mapreduce.fileoutputcommitter.task.cleanup.enabled</name>3 o) i) z4 V! D; v K
<value>false</value>
& X, n+ g d1 f7 }7 I <description>Whether tasks should delete their task temporary directories. This is purely an
* W! ?& X/ p- e! I' d optimization for filesystems without O(1) recursive delete, as commitJob will recursively delete( g, T6 e* x; }( z6 b# ?: |
the entire job temporary directory. HDFS has O(1) recursive delete, so this parameter is left
; g; o6 |$ d; F+ P9 D$ f# V false by default. Users of object stores, for example, may want to set this to true.) z+ h( D( g6 X8 I6 V! ?
Note: this is only used if mapreduce.fileoutputcommitter.algorithm.version=2</description>- R7 P) p0 t; R3 G0 ?( l9 M
</property>
' n3 v. D3 {/ l6 D9 Y3 _- |) Z<property>
& P, X* S9 g. ]8 g8 a& s <name>yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms</name>0 Z( t( \& F- n1 I, q
<value>1000</value>
1 S% u' W& ]9 S9 b' @* U <description>The interval in ms at which the MR AppMaster should send
# e# g3 S0 l# ?/ t4 i% O0 D heartbeats to the ResourceManager</description>
# A5 O! n6 e+ r6 X: q+ |$ m</property> @ U1 | | ]4 h* S* q- R+ O
<property>! m& g$ P* M2 t) m* H$ g" U
<name>yarn.app.mapreduce.client-am.ipc.max-retries</name>/ m7 o4 a; k1 `! U) C; r. i- ?% p
<value>3</value>1 K- q* U L' g8 g
<description>The number of client retries to the AM - before reconnecting
; A7 @7 l( w# T) l6 R" U to the RM to fetch Application Status.</description>
9 @) R+ X' W+ u) w5 _</property>1 Q3 Q/ O4 Y6 n" [, W3 F; R6 D/ r
<property>
9 l( |1 _& s6 y. o, E* ` <name>yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts</name>9 u7 z% s+ q0 [ w4 w0 f* V# s
<value>3</value>' e- y0 r% m+ c( `9 l: [' u
<description>The number of client retries on socket timeouts to the AM - before
1 j% x- n) S0 v8 f8 L reconnecting to the RM to fetch Application Status.</description>9 I& G0 i4 O& ^2 J, _
</property>& b |9 F2 L. H" X L
<property>
9 g& g/ X0 c5 G4 u6 d* V <name>yarn.app.mapreduce.client.max-retries</name>
# x2 A+ i) u+ y# a <value>3</value>5 j6 x* z3 _5 ?. J
<description>The number of client retries to the RM/HS before
" u; F H$ z9 n7 Y, S throwing exception. This is a layer above the ipc.</description>
1 u; s3 a1 u+ U( p0 N</property>
' v Q2 p3 C/ }0 l<property>
8 s2 _ s7 s' Q4 O <name>yarn.app.mapreduce.am.resource.mb</name>- y. Y9 b Y1 P
<value>1536</value>7 P5 U$ i) `1 z/ C/ B7 B
<description>The amount of memory the MR AppMaster needs.</description>0 ^$ {( ?% T _) K3 u b5 f
</property>( F! R ~) @& P2 ^* C, H
<property>
' b; V; o8 ~/ r+ Q% ]# K <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>; g" D/ |1 D+ S- c8 L' Y5 f1 q$ p/ l
<value>1</value>
5 f4 D) V2 \" T) _+ Q6 o5 i6 _, S <description>
, L# }! y8 h. }7 j! s0 e/ ^: s5 m The number of virtual CPU cores the MR AppMaster needs.9 J6 m8 S; z+ w1 B( y5 w
</description>! n$ {) T/ T( m- L! s( b+ e
</property>
# N9 b6 M5 U1 c/ t<property>
6 g& W8 y6 I4 W0 y3 @! F' X4 l# E <name>yarn.app.mapreduce.am.hard-kill-timeout-ms</name>
/ D1 U O& ~& t <value>10000</value>& }" J7 m' ~" |6 k3 Q8 e L3 r
<description>5 I, M/ r8 o3 z2 G, @( {
Number of milliseconds to wait before the job client kills the application. \/ v0 {( a( W% M, M7 S( W# [# d
</description>4 v# L" N# f0 Q: Y7 y/ ]
</property>
& F% q( g( I+ x<property>
* B" W6 y. ?% s H8 K" V# o6 H <name>yarn.app.mapreduce.client.job.max-retries</name>
8 |. l6 c7 f1 f' S. Y+ J, l; L. D <value>3</value>* \; r$ ^- w" |9 g. i* W W
<description>The number of retries the client will make for getJob and
' w$ Z7 b+ `- @1 `: g0 F. ? dependent calls.3 z, P% i) n$ E6 A9 n6 ]+ H
This is needed for non-HDFS DFS where additional, high level D, u9 m/ u$ F. i* T# G
retries are required to avoid spurious failures during the getJob call.+ V# q# H7 _8 ~; {7 @8 z
30 is a good value for WASB</description>3 R& y' P: [* z- d% u% |: a
</property>- R* v' V! V0 k9 w" l
<property>
' ?2 i5 w, [! u" V% Y# V0 c <name>yarn.app.mapreduce.client.job.retry-interval</name>
5 `" i" B! T: p9 t5 c <value>2000</value>
- l3 w8 F% L9 ?9 L5 | <description>The delay between getJob retries in ms for retries configured
6 T6 e$ r. l: t& p* ^8 \% l with yarn.app.mapreduce.client.job.max-retries.</description>3 {6 v* P5 U( u2 `
</property>" X" n7 e! `: d' A6 A1 S2 n; ~+ b
<property>
" \" T2 Q5 i( I <description>CLASSPATH for MR applications. A comma-separated list
3 E; i+ R9 Z; V; {. r; Z of CLASSPATH entries. If mapreduce.application.framework is set then this. p) V8 i6 {" |1 H! @
must specify the appropriate classpath for that archive, and the name of
& f4 o2 _6 g4 ^ the archive must be present in the classpath.: n+ V- j H2 X6 x2 \, f
If mapreduce.app-submission.cross-platform is false, platform-specific! J6 F& M0 p8 V: H5 L& H* ]
environment vairable expansion syntax would be used to construct the default7 D8 \& C9 U/ N1 Y3 V5 D+ R8 {
CLASSPATH entries.
1 n% _8 @3 I4 f; J; C5 G) i For Linux:
, u$ W, z8 D4 z6 d- v- k $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,; a( m- a4 \( Y) Q2 ^" h
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*.9 j' d, g5 D1 T( [
For Windows:
7 T3 s5 j4 v: p. s) p %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,9 t. S' B& K4 }9 }
%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*.9 T {/ [6 E& r+ `
If mapreduce.app-submission.cross-platform is true, platform-agnostic default
w0 n* A; w! i0 e; m+ A CLASSPATH for MR applications would be used:
( K4 F$ h. t) | B$ ] {
. z. \: D. d2 A3 x7 H5 q2 | {HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,
+ J u: {, e9 M4 _ {; ?" I5 n3 d7 I5 f- r& {( q0 b
{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*6 O" d1 t) A: l7 `/ \7 S7 A
Parameter expansion marker will be replaced by NodeManager on container
4 b$ q1 B1 a5 y& l/ J launch based on the underlying OS accordingly.: c5 m1 D' l6 G+ c2 Q
</description>
' @ W( n- ~& |7 j <name>mapreduce.application.classpath</name>' o4 n9 A7 }" Q8 X9 e! I
<value></value>: z& _8 p0 Z2 [; `
</property>3 `& A9 d9 b) k4 d
<property>6 }( A6 A3 N% X" P
<description>If enabled, user can submit an application cross-platform
( p* G' F* J- @ i.e. submit an application from a Windows client to a Linux/Unix server or
* H: z' P1 d/ f+ Z& M vice versa.9 l: A3 d) i7 |) f
</description>5 }1 @. ?! P& \$ i/ ~
<name>mapreduce.app-submission.cross-platform</name>: h5 |, C- s& f8 l" k
<value>false</value>
$ H3 A& d7 H, \* U3 V, E' E</property>5 g; ^3 W7 O9 ~! n! c0 S- H- D
<property>
+ b n7 [1 P8 {) z <description>Path to the MapReduce framework archive. If set, the framework. e. F- d% T8 f J; Y9 X
archive will automatically be distributed along with the job, and this: W, F! |1 {0 }
path would normally reside in a public location in an HDFS filesystem. As
5 P" o: Z8 B2 i8 `( l with distributed cache files, this can be a URL with a fragment specifying
8 @( i/ x6 ^9 U2 }( _% f% N; w4 W. R the alias to use for the archive name. For example,
! \7 {: j& \5 g$ r5 S hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would
$ j" H" R7 d. K alias the localized archive as "mrframework"./ g1 o T# _: P3 A! o: V5 `
Note that mapreduce.application.classpath must include the appropriate W! _* i/ Y4 c& B$ b, G
classpath for the specified framework. The base name of the archive, or6 d. C# T, w! X4 M3 D8 I5 k
alias of the archive if an alias is used, must appear in the specified
7 y$ j8 {8 H: u/ t4 a classpath.! T* a8 d( v) ~% A0 l
</description>
) @. s: S" m; O) @+ f/ V/ n r <name>mapreduce.application.framework.path</name>
7 k9 ?4 X+ l: g5 R3 w <value></value>* {3 L# e5 s7 H1 u3 z4 b W
</property>% |. u& X8 x( r! T/ t
<property> j' u+ O: C; ^8 ~7 R& d
<name>mapreduce.job.classloader</name>
; Q% }6 T* d; t9 j2 _7 R7 Y <value>false</value>
- J! Y8 Q& r5 W8 T9 J7 P0 H! C <description>Whether to use a separate (isolated) classloader for
! L! n2 o6 w+ H2 \9 \* M% g user classes in the task JVM.</description>( W# n9 F0 d- U! L6 O2 f
</property>
- \7 k1 N3 V4 W" [3 |! f<property>* ], C: q8 b5 J P+ e: c- B7 n
<name>mapreduce.job.classloader.system.classes</name>: z7 E9 e0 ]' E. ?6 _
<value></value>
' l( n0 T: B+ K; `0 R <description>Used to override the default definition of the system classes for
. d$ n' l. g9 L5 y, y& s the job classloader. The system classes are a comma-separated list of
& E% f5 i0 @; f' y patterns that indicate whether to load a class from the system classpath,
4 @- L% `/ B$ j' l \5 f instead from the user-supplied JARs, when mapreduce.job.classloader is
2 l- v: ?$ J$ b/ A+ ?7 g! I W1 ?/ \ enabled.
$ B# `, V) O2 q; C+ o0 } A positive pattern is defined as:
! M: @& n6 ?3 D# n _, d 1. A single class name 'C' that matches 'C' and transitively all nested
' H* s1 J% v; `' o% e classes 'C$*' defined in C;( b0 e5 [7 ~# D
2. A package name ending with a '.' (e.g., "com.example.") that matches
* z# s# J' P2 b7 b all classes from that package.
! }: o& I7 ^* l. W; \% X. d, h2 } A negative pattern is defined by a '-' in front of a positive pattern
1 G" m* A% x1 {) w! u (e.g., "-com.example.").
, E' p, Q) P# d. H A class is considered a system class if and only if it matches one of the* L3 I6 m! [' v3 Q+ N' V
positive patterns and none of the negative ones. More formally:: H6 |" S% A. t( f; O Z; O. k2 ]
A class is a member of the inclusion set I if it matches one of the positive
' o. z2 s/ p+ w patterns. A class is a member of the exclusion set E if it matches one of
7 G% i2 K2 D& L2 {) \% a the negative patterns. The set of system classes S = I \ E.
1 T" x) f+ M% _ </description>
7 N7 b" {; ?5 H</property>
* Y- ~# |8 ^! ^0 _ {) A6 k8 S<property>
* @' _! R u& v2 D8 @7 ^- E$ l <name>mapreduce.jvm.system-properties-to-log</name>
V* Z% x" A5 q <value>os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name</value>
: d% m& m5 S) h8 I) q/ _8 i <description>Comma-delimited list of system properties to log on mapreduce JVM start</description>
0 f, B x! k3 Y0 n" B( b3 A) y</property>* z/ Z9 n& Y) u6 e+ D8 Q9 b
<!-- jobhistory properties -->$ F1 }. h4 v4 I @( A
<property>8 u3 } B) c+ G, `
<name>mapreduce.jobhistory.address</name>
7 r: g0 J0 L S: t+ A0 X( K+ V' O, w <value>0.0.0.0:10020</value>; \9 E0 ]( h5 H+ Y' Z
<description>MapReduce JobHistory Server IPC host:port</description>9 ~) C- _; u8 M" ^, S! C
</property>
8 l |# V* Q. Y5 K8 A<property>
$ Q) K; }9 h: y3 U <name>mapreduce.jobhistory.webapp.address</name>
) O, A/ ~) j+ ^2 ~# _2 h* `/ F5 a <value>0.0.0.0:19888</value>
0 G# b- m* k5 ]- d# i <description>MapReduce JobHistory Server Web UI host:port</description>6 u+ K |3 k+ D3 E L6 S: f: H
</property>, _/ |( Q' d$ g* U2 @" P g% n" ]! Z
<property>
; S/ Q6 Q- S! @. u P <name>mapreduce.jobhistory.webapp.https.address</name>7 w0 w3 J+ `- i0 k
<value>0.0.0.0:19890</value>. _! }" X+ \( `
<description>
. ?5 Z a' `: m: H The https address the MapReduce JobHistory Server WebApp is on.& x2 l+ K, n5 J) e
</description>
+ {: ~6 Z* s9 d' M# D</property>( S# N& T" o' s0 o* A& L% [
<property>' { N5 W: Y' u; `* |
<name>mapreduce.jobhistory.keytab</name>) e- `+ h" t, G" g b: x
<description>
$ e3 E" T1 {4 C' S" q Location of the kerberos keytab file for the MapReduce) _* g2 Y) i7 Q4 B* k
JobHistory Server.# g7 h+ ~0 X' ~ r, i4 K
</description>9 F* J e9 ^* m! Z2 d4 t8 a
<value>/etc/security/keytab/jhs.service.keytab</value>
; ~1 Q6 s/ ^0 [$ t( m</property>
, C2 j ~% U6 I+ ~+ _9 d<property>
* ]$ x" k, S) s <name>mapreduce.jobhistory.principal</name>
) ?- r) J8 T5 e8 ~7 t) j; }$ g <description>
2 @ p$ g1 ?7 G' w j Kerberos principal name for the MapReduce JobHistory Server.
, w2 Y# p7 B h. d9 D </description>
0 E; ?( |+ s2 `. @5 G# S <value>jhs/_HOST@REALM.TLD</value>
' K; f! e" r; `) l# S% f- s+ \</property>
! M5 Q2 f$ |0 k: v<property>
. P9 _# z! y" U( L" Y/ H <name>mapreduce.jobhistory.intermediate-done-dir</name>
+ _2 _1 [+ R0 r* E <value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>
: G2 |' t2 A# `# b( I <description></description>
0 ^7 ]; ?# d( N9 L+ t</property>
0 ?9 t" x/ F# j$ N<property>. p r+ _0 u) h0 X
<name>mapreduce.jobhistory.intermediate-user-done-dir.permissions</name>' ?2 M4 ]& w, d3 d1 u
<value>770</value>1 |! l6 \" M; y R
<description>The permissions of the user directories in2 [; P, r. {7 e
${mapreduce.jobhistory.intermediate-done-dir}. The user and the group
) E/ Q. j/ s4 W) Q5 ?# j permission must be 7, this is enforced.
8 A% [( x6 H3 u, M3 N- S </description>" c8 v% F _9 Y5 W
</property>
2 J' G, G0 o$ s; z. c! u, A<property>
+ V& e5 L7 b2 k# q, `2 r <name>mapreduce.jobhistory.always-scan-user-dir</name>
! c) E6 k) t, i/ V: L1 J, {1 M; B <value>false</value>7 @! t, H4 z4 o1 L
<description>Some Cloud FileSystems do not currently update the
3 o* I. \# H6 f, k" D# H modification time of directories. To support these filesystems, this
5 ^8 P. g) |3 W configuration value should be set to 'true'.
8 M1 g5 q' E. ^5 _: h7 ` </description>3 J4 X/ O% S( O, Y8 {6 f" o
</property>
% o; B0 R6 g6 X% v- F<property>
: t' M1 }) G& O* l- [) A, Q7 x; M- } <name>mapreduce.jobhistory.done-dir</name>7 C+ }8 x" R7 G
<value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>
: S! Q: f9 o! b% w) F& X <description></description>
8 e+ _ w! {" V. J2 k</property>4 Y2 L6 L2 e% K" {
<property>' m& D' v/ C$ x
<name>mapreduce.jobhistory.cleaner.enable</name>8 s9 S- y. x% d8 x8 `, h
<value>true</value>5 d6 I2 A- E* @
<description></description>+ M4 \# T$ |, T& n$ w2 a: {, n
</property>
) e2 Q& F( H$ h9 B2 S; D# d( ~/ V<property>$ ?8 m# ^! E# N0 e: c/ z. A, i
<name>mapreduce.jobhistory.cleaner.interval-ms</name>0 I, z# j+ U# E# F% \
<value>86400000</value> _5 L n1 A7 A0 z/ }7 {: J. B$ m1 w
<description> How often the job history cleaner checks for files to delete,
5 w! n1 q! [5 P- i/ \8 d0 G in milliseconds. Defaults to 86400000 (one day). Files are only deleted if
* O$ |% [ o/ k they are older than mapreduce.jobhistory.max-age-ms.' T/ [% q! x0 O' n3 H% H
</description>3 L0 I5 ?6 @# _# G1 X! f; F
</property>
3 @4 U$ @: `* C$ t o<property>
3 H# e, _+ G' B9 ] <name>mapreduce.jobhistory.max-age-ms</name>
* ^$ G' `6 U4 G0 Q7 m/ o <value>604800000</value>* k2 q9 W" K' C- n' K7 T- U2 X
<description> Job history files older than this many milliseconds will# P% Q+ f) K% M. k
be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
. n! ?2 [2 O, c- n) L </description>! D1 T3 J3 Q# U: k
</property>! O* V* u* f5 i$ `0 C. {7 }
<property>
5 u* }. G, G$ F6 W <name>mapreduce.jobhistory.client.thread-count</name>
5 U4 {* q- }. l- n7 m <value>10</value>( L( {, X5 y7 ]8 y7 l4 g& N
<description>The number of threads to handle client API requests</description>
: q% j# e+ c ]" t: g</property>7 _" Q9 @/ d+ k o
<property>
7 u5 A2 q3 l0 ]: v <name>mapreduce.jobhistory.datestring.cache.size</name>2 T9 ]) y5 k5 z3 B V) t
<value>200000</value>/ |" T( b2 ^' {' g$ s
<description>Size of the date string cache. Effects the number of directories5 z! c/ S) G. n( u
which will be scanned to find a job.</description>
; D2 _. {) E8 g$ c/ k: \</property>
+ S" W6 x! r4 H' z<property>
: M- _6 J7 t6 T. E <name>mapreduce.jobhistory.joblist.cache.size</name>
9 V; G2 _( {1 Y( ], G! ~3 I9 [ <value>20000</value>
2 j, y0 R0 k" Z4 p7 Y4 p* a <description>Size of the job list cache</description>
; S# o6 X/ S' s$ R' E! [. ]</property>- D. ]2 u; U0 g! m1 f
<property># L3 g# Y# b( Z5 v( T3 O
<name>mapreduce.jobhistory.loadedjobs.cache.size</name>( u* a# `* H3 {! m! |0 F
<value>5</value>
7 Z* o# p) ]9 y" E, o3 n8 y: r2 G8 B <description>Size of the loaded job cache. This property is ignored if; t) }" e' H$ I* V- A' D3 b
the property mapreduce.jobhistory.loadedtasks.cache.size is set to a6 ^) v0 d7 ]) ^" p( `$ B1 Q3 l
positive value.( D& ]- |' z2 T
</description>% K Q. _! b+ |% S E: f3 _
</property>2 D' x9 t' d4 m! q8 ^- s
<property>. ~/ } E! V' e( P+ T/ S
<name>mapreduce.jobhistory.loadedtasks.cache.size</name>
% [; D1 f- C( `' r. m" h <value></value>' G) U0 v" @& ~4 ^
<description>Change the job history cache limit to be set in terms/ @2 D) N- ?0 ^0 p
of total task count. If the total number of tasks loaded exceeds4 m2 D' {4 l' k+ y3 I9 `& R
this value, then the job cache will be shrunk down until it is! ^. q. Y. G+ u7 m3 H
under this limit (minimum 1 job in cache). If this value is empty" y; A8 L$ ^ S5 G7 y' T! X
or nonpositive then the cache reverts to using the property
X, a3 o' c/ ~- M mapreduce.jobhistory.loadedjobs.cache.size as a job cache size./ q# J g5 F; V3 o( I, w, m- q, f
Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size6 D0 `( z3 ]4 A
property:
/ W- W& u! j* M1 Q" D! b 1) For every 100k of cache size, set the heap size of the Job History6 P! V% u; z% `) m$ j" h3 ~9 A8 i
Server to 1.2GB. For example,* r7 t% c: p* s; j0 c4 E8 r6 S
mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB.
4 H/ ?) i) q D4 B v# | 2) Make sure that the cache size is larger than the number of tasks a; `2 ]/ J, l: w/ Q
required for the largest job run on the cluster. It might be a good
6 _! K$ W5 f, }' N idea to set the value slightly higher (say, 20%) in order to allow7 R- b. [& {" i" v7 i4 p3 U
for job size growth.
0 }' D+ [8 U G7 n( S9 v </description>
( L1 d+ H2 F- U" d+ m4 I</property>7 l e% a' D8 e! F& ~
<property>& i$ y5 X" m8 c) C* {
<name>mapreduce.jobhistory.move.interval-ms</name>9 H# m5 s! z' c$ U4 A6 E9 k' `
<value>180000</value>
T% X7 ?) |6 ?2 B/ ] <description>Scan for history files to more from intermediate done dir to done
9 D* d; C/ `9 Z5 s dir at this frequency.
: U/ k. d4 X0 m: f- o3 d, H </description>( o# I+ W6 }: C
</property>
5 l+ Q& @4 l# Q<property>8 C) \1 F c# ~% K1 e
<name>mapreduce.jobhistory.move.thread-count</name>( V0 K& ?5 G; g/ Q! _: {' v& L
<value>3</value>
. Q1 O! Z$ m9 c! q# }1 Q <description>The number of threads used to move files.</description>
! v- w5 z6 J6 x1 I* t$ h! V</property>% D4 S; r& T" B+ X
<property>. W, e @9 x/ Q$ F$ o
<name>mapreduce.jobhistory.store.class</name>4 g4 Y3 x5 @0 T: t' w( N8 w
<value></value>
8 S& _+ J! Y* I# b5 ^ O( J <description>The HistoryStorage class to use to cache history data.</description>
g g* d7 S. x: {7 W$ _' P</property>
Z2 i; O: w3 p; ?8 @5 D# S" V<property>' L7 }; _% f" m4 Y5 B/ f
<name>mapreduce.jobhistory.minicluster.fixed.ports</name>
1 P! v* x \8 L' e! Z( @/ q <value>false</value>
M9 f5 N9 w- t, G- X <description>Whether to use fixed ports with the minicluster</description>
0 Q2 }! O! ^ h: t, g</property>( y6 j/ r& s; y8 ]$ D e6 c7 m
<property>
4 ^, b, T- i9 S. i; \ <name>mapreduce.jobhistory.admin.address</name>
% F. ~/ A: e B3 c <value>0.0.0.0:10033</value>
' S6 X3 k! L. N3 y0 E9 H <description>The address of the History server admin interface.</description>
/ p: U( o( r ^0 B- o2 f, D</property>0 N9 l; M( k# m& n& R. l. L: l, v
<property>
; O2 Y2 v ]; X ?1 y& {, W6 O/ l <name>mapreduce.jobhistory.admin.acl</name># Y- R) V8 l/ p" U
<value>*</value>
; u/ l- o3 ^' _" x- X( V% X <description>ACL of who can be admin of the History server.</description>
% U( s9 G% A; |( S</property>
3 B$ n& W' d: y3 R6 F<property>7 v1 Y: P7 c4 z
<name>mapreduce.jobhistory.recovery.enable</name>
* @4 G9 a* x" M) ?: }9 X <value>false</value>
; {1 [$ u; [; ^- O+ r* } e <description>Enable the history server to store server state and recover8 \6 n/ T0 z" f; g7 |
server state upon startup. If enabled then
$ R" n3 k! q( A8 Q# X mapreduce.jobhistory.recovery.store.class must be specified.</description>, d, i- m% h4 o. U; f: h3 w" _2 R
</property>7 W' d- [3 C1 ?* ?
<property>2 S9 K; n8 U& G/ G
<name>mapreduce.jobhistory.recovery.store.class</name>1 ]! I- y- {, ~
<value>org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService</value>
3 ?8 `- o9 c: G2 l <description>The HistoryServerStateStoreService class to store history server
- i: k$ T. U# m! J* \9 f state for recovery.</description>+ G# O' H. \7 A; p: k
</property>
! J& i+ L1 v) t H* u; i. ?<property>" ~' n8 S; r7 M( b, s. I4 I
<name>mapreduce.jobhistory.recovery.store.fs.uri</name>
- H7 N# u5 J2 D q" v6 x3 [# z3 w <value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>3 ]% w( o) b! Q7 B9 W [
<!--value>hdfs://localhost:9000/mapred/history/recoverystore</value--> w; _ ]& h: Q/ k
<description>The URI where history server state will be stored if
4 t0 R# i+ ?& v HistoryServerFileSystemStateStoreService is configured as the recovery
5 {; M) a# `6 w0 q* Z storage class.</description>1 i0 r/ X1 ?( I8 p
</property>
7 `6 e: g. I- H+ t# k) l<property>) L1 f3 C. g$ i$ Q- x( f3 t
<name>mapreduce.jobhistory.recovery.store.leveldb.path</name>. F' d# x" S- Z) }
<value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>
- J u$ F& R9 m1 C <description>The URI where history server state will be stored if
& c% K {) G- j* k! U" p" \ HistoryServerLeveldbSystemStateStoreService is configured as the recovery
3 M, x; k, d$ |% l7 @. Y storage class.</description>
0 [) h# P9 l* Z% y N/ W4 @; s/ R% w</property>
1 J( C( d7 i* P% Q7 ^6 y<property>
) ~0 [. n, Y8 _ O/ Q' h& q <name>mapreduce.jobhistory.http.policy</name>
6 m; |$ i7 {& E4 u7 C$ }4 o <value>HTTP_ONLY</value>2 ]7 O; m$ E- ^1 u) l1 p9 Q
<description>: D& d" f; \/ D; a2 u
This configures the HTTP endpoint for JobHistoryServer web UI.* L! L' |. f: g) f
The following values are supported:; B `2 q% ^) ~
- HTTP_ONLY : Service is provided only on http) Q k" J# s8 \7 X
- HTTPS_ONLY : Service is provided only on https
( ]# {+ r- D' P, D4 ? </description>2 q5 y2 q' L! G( Q+ P7 t
</property>- b+ f& }* Z6 Y. X/ s
<property>
$ \' c+ ~! m g7 c8 F <name>mapreduce.jobhistory.jobname.limit</name>
) X5 ^" i/ V+ `8 Q <value>50</value>3 Y R1 o! Z- }$ F( w" @" V
<description>
5 {% J+ ~. ?! r$ i4 @' X Number of characters allowed for job name in Job History Server web page.
2 F5 P6 V! r& z4 A </description>
) }$ Y+ ^% x- n6 B( i, @</property>
2 L( ^& I; ]% m* w/ C# \3 T: m<property>0 u3 e$ Z. t8 K' R# G. V5 r% R
<description>2 j% x% j& U/ C/ y4 V! { W- _: w
File format the AM will use when generating the .jhist file. Valid, @1 k$ {7 Q5 p" k( _
values are "json" for text output and "binary" for faster parsing.
" w# D$ i/ z6 T# d E </description>
! ^ K0 L$ u8 ~ <name>mapreduce.jobhistory.jhist.format</name>
" u' c6 {2 N- F( @1 F <value>binary</value>
! }1 m# O" k, y$ ?' C. I</property>2 D1 S4 n( u6 T( T: [; l2 o4 {
<property>
* g3 K# u6 s* E# ^9 } <name>mapreduce.job.heap.memory-mb.ratio</name>
2 i1 F3 X1 [& b7 |, B1 V <value>0.8</value>& K* t& K* k/ \$ `! y" n% m$ Y
<description>The ratio of heap-size to container-size. If no -Xmx is
" X# `7 X5 z, u; I specified, it is calculated as8 U: k- V, N( e$ A( Z% J$ K$ g
(mapreduce.{map|reduce}.memory.mb * mapreduce.heap.memory-mb.ratio).# B+ F4 c* I/ z* \. {
If -Xmx is specified but not mapreduce.{map|reduce}.memory.mb, it is
8 H/ }& h$ U# L9 X& D calculated as (heapSize / mapreduce.heap.memory-mb.ratio).
1 b% `0 p* o3 r, x& h8 l </description>
+ K# R3 B8 O2 U+ @6 v</property>4 h9 f6 }# B) I1 u6 c! o% I
<property>$ Y7 b9 j2 E! a( @
<name>yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size</name>& o2 G# v& I3 L8 x* t% a' \1 _
<value>10</value>: e8 @( f8 Y4 }6 C' N" C! }$ C; u
<description>The initial size of thread pool to launch containers in the
0 w0 @6 s: s2 g& e1 X2 C# a2 z l app master.9 t. Y1 H6 {* b6 e- ^: Q5 x9 [6 F
</description>
9 _2 Q9 y) h" M. D7 x9 D</property>* z* c& E4 u3 n7 p! d& F3 E
<property>
: a& I2 G; {; |/ D <name>mapreduce.task.exit.timeout</name>( h( a9 C: O$ l: q
<value>60000</value>
$ B2 f, K3 |1 `; F' Q( V- o6 P <description>The number of milliseconds before a task will be5 { G" x# r0 X# Y
terminated if it stays in finishing state for too long.
& C; m) Z/ K% Y% i, j ? _) B) r After a task attempt completes from TaskUmbilicalProtocol's point of view,4 o6 \6 N' P' S7 b4 i Q
it will be transitioned to finishing state. That will give a chance for the% K: |4 z* N7 }" V
task to exit by itself./ y, q+ j# f& i8 Z* y3 m
</description>/ Y0 M' W# T4 \- v) a; |
</property>
& g5 ^! _9 u! u* _3 T8 G5 x) c<property>9 ]. y. W' o! v$ c, K1 _
<name>mapreduce.task.exit.timeout.check-interval-ms</name>) W7 j- ]5 u% P q' f: L! m- Y
<value>20000</value>1 b3 M f: e" m' K5 u O/ ]# f
<description>The interval in milliseconds between which the MR framework
" T, w6 ]/ F/ b4 f2 g( V checks if task attempts stay in finishing state for too long.# b* i5 i1 N% J! \5 T
</description>
5 @3 J+ q# B+ ^% N6 U</property>
- ~7 Y& W$ j' X" b<property>
% J8 d5 ^0 G1 F: [, h/ r6 K <name>mapreduce.job.encrypted-intermediate-data</name>6 V9 G: m: ]& f& G5 Z
<value>false</value>- M8 \7 p- \- Z! G6 |3 Z6 P1 r, G' ~
<description>Encrypt intermediate MapReduce spill files or not
* }( D4 O( _% X* E- C' B default is false</description>
8 @6 Y" ^5 g* ~! m1 N1 M</property>1 J3 W. i" h- Z- C
<property>
$ ^+ b+ w% _+ {6 Z, c( { <name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
\0 B' k: c/ m( S <value>128</value> h) o& ?* M' N; ^
<description>Mapreduce encrypt data key size default is 128</description>5 ^# E; v# {) j# `" A' e" E
</property>2 d z) o, T6 E! |2 x6 X
<property>4 w0 s3 G1 L- ~& h( k: @0 Y6 W& f
<name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>5 ~3 d: n. G; A4 Q/ n7 K& i: Z
<value>128</value>4 _$ x/ Q" z8 Q
<description>Buffer size for intermediate encrypt data in kb
, e: g! ?( o# U: P default is 128</description>! d" J" t+ k% W3 k
</property>0 u) I1 T9 `: ?* w! z
<property>
8 I8 l4 ] m7 g% T' ^ <name>mapreduce.task.local-fs.write-limit.bytes</name>
& x2 p4 [0 p9 B6 t) V0 G <value>-1</value>
& E9 i1 V$ ^ D# A <description>Limit on the byte written to the local file system by each task.
9 ?; d( @# `2 ]: _0 n% g/ Z- G This limit only applies to writes that go through the Hadoop filesystem APIs
2 e; J% C8 C, }" b: h within the task process (i.e.: writes that will update the local filesystem's
! N8 a2 D2 J: m U6 C BYTES_WRITTEN counter). It does not cover other writes such as logging,
. v8 `2 E7 {; \4 |* U: m9 y sideband writes from subprocesses (e.g.: streaming jobs), etc.
2 _: A' E, @* P: }/ T Negative values disable the limit.. Y7 A* {) {* G9 U
default is -1</description>
9 r `& ^ t* v; ]! C9 m0 I% i</property>/ ?, w' L( ]/ T/ H/ h$ @
<property>
+ N: X$ l/ J5 Q- J/ {* B8 d <description># p3 b; W1 b0 f, E; i2 a. Z
Enable the CSRF filter for the job history web app
- Y& \# u' K9 v8 e. a' f& } </description>
" f5 j" D9 m9 R" W9 u% Q <name>mapreduce.jobhistory.webapp.rest-csrf.enabled</name>. b5 H, r9 L4 G+ ?: N( D1 ?, b
<value>false</value>
+ q7 {2 @; Y! c1 V( M</property>' i& |2 }7 x- ^$ b. ~+ O' U
<property>
( z" E. N, r$ {4 k$ u <description>
* D P1 Q9 _; {0 C% V Optional parameter that indicates the custom header name to use for CSRF
: U. Q1 O3 w H- e9 {' I protection.
' v% {1 O' x0 p. }+ a" { </description>: U+ _9 j3 a% q$ |+ U
<name>mapreduce.jobhistory.webapp.rest-csrf.custom-header</name>- X9 q- [, P) E9 }* v
<value>X-XSRF-Header</value>
7 b$ h' Q+ R' n. o- o</property>/ H3 U6 @1 j7 l4 F7 k
<property>/ k0 R, J! A4 e" b$ p6 k& h* Z
<description>$ m3 _7 z& Q$ O) q z$ O- D( h
Optional parameter that indicates the list of HTTP methods that do not5 y5 n" W3 v0 S) v8 {
require CSRF protection
( a2 m7 z6 `( {/ L' e; R; k" M+ w </description>
. H/ B ]7 D1 Y <name>mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore</name>! R$ B7 j; J5 E& l% s2 p
<value>GET,OPTIONS,HEAD</value>
4 t+ a8 D m' u& l! G$ G! P</property>1 X, F/ F7 n; }
<property>) v% X, k7 X W {
<name>mapreduce.job.cache.limit.max-resources</name>" A( H- ~. v5 f8 R/ N& U. d, d9 z
<value>0</value>1 v4 t" l5 P- i* B
<description>The maximum number of resources a map reduce job is allowed to- N4 m$ n' O$ _* k$ ^% Q' V
submit for localization via files, libjars, archives, and jobjar command
* r4 c4 e" n: W3 E# T4 D# o line arguments and through the distributed cache. If set to 0 the limit is% D1 {9 ]+ D. |; |
ignored.
7 t d" J/ d) [! k </description># O; w# ]' m$ @6 e; u& g
</property>/ W+ ]" V; |- F7 m
<property>" e' }' ~3 |: g2 [
<name>mapreduce.job.cache.limit.max-resources-mb</name>" X0 w0 q/ Y( I$ K7 {2 k- X+ }- k
<value>0</value>
7 M: v& \# \, H9 c/ M; { <description>The maximum size (in MB) a map reduce job is allowed to submit
( c3 u: w9 a7 A2 n8 V! Z4 C$ L9 Q for localization via files, libjars, archives, and jobjar command line, w, t& P) c. ]1 |; Q6 G
arguments and through the distributed cache. If set to 0 the limit is! D3 S* Y2 c# V. U( P5 S+ e1 H/ [1 ?
ignored.- h6 [4 \/ O0 o& s' T
</description>
* y4 N1 H5 B0 K4 C; U- f: A</property>$ r2 z) g0 J! V5 _" `* z4 N8 H# O4 W
<property># o, ]" e# T& o$ L2 }; J5 H L
<name>mapreduce.job.cache.limit.max-single-resource-mb</name>
D5 p+ ^ K9 ?4 }) V' M <value>0</value>- B$ i" N' }* E Z) o
<description>The maximum size (in MB) of a single resource a map reduce job
. x, ]0 I; v4 d f1 I% M is allow to submit for localization via files, libjars, archives, and
9 f" X: o* s% T% U- S' d jobjar command line arguments and through the distributed cache. If set to" s, `& w1 b' H) V) a. d7 n
0 the limit is ignored.
$ {* F4 l9 r! w4 o' b3 A% N </description>
& h$ ^& b/ L6 |2 T</property>* h6 `* q( J! Z7 M
<property>
& a$ Y3 S' z. @" ] <description>. W( @% x; W0 _ t( D
Value of the xframe-options1 _; K# x5 Y2 c9 H* L
</description>/ K* u+ c5 l, n. R$ `0 m9 f
<name>mapreduce.jobhistory.webapp.xfs-filter.xframe-options</name>6 E J( S7 J2 m( A# J
<value>SAMEORIGIN</value>5 v2 Y4 T: G! }1 m7 g
</property>- r- c6 Q" b8 b% \# b- [8 z' t
<property>; [- R8 R1 r0 x4 t# {
<description>; y1 b3 Z9 i8 _6 p; U( y! |$ k C
The maximum number of tasks that a job can have so that the Job History
5 @0 D" y2 K& K Server will fully parse its associated job history file and load it into
# }. {3 i( e6 @2 w' w* |4 [ memory. A value of -1 (default) will allow all jobs to be loaded.
3 X/ o' C) V( R( k" Y </description>
1 j" b. K) ~% j& p" _ <name>mapreduce.jobhistory.loadedjob.tasks.max</name>
! ~+ e+ p- \! y# L4 i9 R1 G <value>-1</value>
2 r* Z" ?& V$ A* I% S3 H, T</property>7 J7 W* z1 W8 }; i( V
<property>) r6 d; C; d9 d$ e8 q( ]/ v
<description>4 n8 D/ M9 U, w1 q
The list of job configuration properties whose value will be redacted.3 B- E2 q) l+ T* t0 Z$ |! v+ ~7 ?
</description>
" W+ R, t. v# h- N <name>mapreduce.job.redacted-properties</name>4 }4 I! k t0 J2 h, T }
<value></value>
6 i& v0 K. X" P y* C' v# m</property>
/ ], M9 B6 o& e$ o& o<property>
8 L: r6 o( L* y8 {8 e' \" ~ <description>3 H$ R, E2 ^& g' |7 z. \# S
This configuration is a regex expression. The list of configurations that
! L$ D- t7 B( [ match the regex expression will be sent to RM. RM will use these' y, Y5 F$ ~! T( j# @
configurations for renewing tokens.
1 }- O0 X( |4 O& P3 p This configuration is added for below scenario: User needs to run distcp
% O {# N! w/ o3 p jobs across two clusters, but the RM does not have necessary hdfs: T6 D9 }4 [& I
configurations to connect to the remote hdfs cluster. Hence, user relies on2 R4 K% p! @+ [) l7 a
this config to send the configurations to RM and RM uses these
, w( h3 P* _+ q configurations to renew tokens.+ ]9 b% l- A0 M* z) c" h
For example the following regex expression indicates the minimum required, c, x7 S8 V& M
configs for RM to connect to a remote hdfs cluster:
' \% m& S X/ R1 Y- C, z dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$|dfs.namenode.kerberos.principal
; ^/ x" q- X( s0 b% @- ? </description>* z0 O' V3 F' q5 M9 N6 y- c
<name>mapreduce.job.send-token-conf</name>
6 \$ K; ?% N. ] {; g% u, _* n# n <value></value>, e* h4 p8 H4 u% A
</property>
2 \' z O6 k9 ~ z<property>
5 h7 D& w2 L% G0 D! J/ B <description>' I) G. O0 H) |8 B! M( ]3 E7 N/ p
The name of an output committer factory for MRv2 FileOutputFormat to use
0 r. H; P/ c6 G for committing work. If set, overrides any per-filesystem committer w$ J- V/ C6 W' l0 {; `
defined for the destination filesystem.
$ E$ a$ g$ `' \5 ?. B0 O </description>6 z# e ]* U* c2 F6 i
<name>mapreduce.outputcommitter.factory.class</name>
7 a/ S* `+ H5 \2 @, C <value></value>/ l% b( S8 ?2 Y) d4 M! n: E8 N
</property>
* m: @, o3 H0 V0 T' Y8 t<property>
3 K, }: P- [- ^% e5 ]* }0 W% P7 g <name>mapreduce.outputcommitter.factory.scheme.s3a</name>
. }0 K; I; M2 u <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value>
: e3 q |! t. G% g0 Q: r N <description>4 [/ p6 ?8 i; D
The committer factory to use when writing data to S3A filesystems.
/ F9 ^4 h" o H' g; S; [ If mapreduce.outputcommitter.factory.class is set, it will
& k) h) l1 o8 _0 |6 X override this property.
$ |4 y/ V( J6 V o9 E0 E1 N </description>
+ t3 Z# ?& g8 Y& Q' M& C/ u$ ]5 z</property>9 P0 x; N1 L6 G1 b3 }6 {
</configuration>
! R" L! A; c, ?$ ^4 q4.yarn-default.xml
* i# J0 f8 Y! V* q/ y \5 i( s* q# h" l
<?xml version="1.0"?>( m6 V3 {: k# \7 i
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>. ]3 R( m0 b) Q- @1 N
<!--" N! K5 [5 x) m+ Y$ s
Licensed to the Apache Software Foundation (ASF) under one or more7 @) I2 ~& U# X/ z
contributor license agreements. See the NOTICE file distributed with* S8 C% O4 g, n8 ]8 |; R8 W
this work for additional information regarding copyright ownership.3 k# i3 E3 R; c4 C ?( `6 g
The ASF licenses this file to You under the Apache License, Version 2.0
9 q% n4 z4 p$ q3 q5 q! a. t0 l (the "License"); you may not use this file except in compliance with
; l3 w* h1 A; m; I# w: C5 p+ E the License. You may obtain a copy of the License at
2 n6 Z/ j3 `6 {3 }: Q http://www.apache.org/licenses/LICENSE-2.0$ U I1 G8 b% f0 w: B+ |
Unless required by applicable law or agreed to in writing, software* ^& j* z2 O0 l; K
distributed under the License is distributed on an "AS IS" BASIS,
9 W' s# [ \# W! n/ e' u9 G+ X WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- r- ]& @) ?* \
See the License for the specific language governing permissions and6 C. u* `( N& i
limitations under the License.. J+ V# R1 V. D3 x
-->. ]- D8 C" ?, M% ?
<!-- Do not modify this file directly. Instead, copy entries that you -->
$ C, G$ N6 C S<!-- wish to modify from this file into yarn-site.xml and change them -->
# k% b5 W! ?" l1 x<!-- there. If yarn-site.xml does not already exist, create it. -->
! ~: L. z2 E% t" V7 H+ H1 Z<configuration>
2 A* V1 }6 l! Y' v& k- ? <!-- IPC Configuration -->
% B$ p- c. Y* O; z: g* M. n* |! l <property>3 d( k9 K' m0 l+ x% G7 i
<description>Factory to create client IPC classes.</description>
, A4 |, @3 x/ u6 ]" [8 k <name>yarn.ipc.client.factory.class</name>* |, g0 j: G: }9 j: X
</property>7 X: F% I: ]. e( R2 F7 M0 C
<property>
8 H# V9 s, J% ~. F) z <description>Factory to create server IPC classes.</description>
$ [) V w% u! r% }. ~" P' S" ^5 R <name>yarn.ipc.server.factory.class</name>9 B' G( U' O1 y4 W, P0 v
</property>
$ ?# W/ ^/ i8 _6 i* B; ^9 H0 x2 N3 L <property>7 z% g! [9 l7 i+ s, ]
<description>Factory to create serializeable records.</description>
1 K, z0 I8 k' I) U+ c( i, N! V <name>yarn.ipc.record.factory.class</name>
. N! T2 U( q, I: G# ]+ y </property>
7 G9 k+ u; F* I0 P- R6 R! j4 g- W: } <property>
5 b: v4 X; e/ Y, v+ F* M <description>RPC class implementation</description>: @ O, L: g: f, Q& d9 ?3 T
<name>yarn.ipc.rpc.class</name>; {3 d H; K% X7 P! W. C
<value>org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC</value>
8 p' n. _( a3 d( u3 u% o </property>2 O9 m2 b) E) |4 w; F+ @$ l4 l7 p
<!-- Resource Manager Configuration -->6 q* B) L* _% d& H7 j
<property>% n+ c* ~$ X$ E7 n3 N# Q! U( f
<description>The hostname of the RM.</description>
( p# N/ d# u# `& T C <name>yarn.resourcemanager.hostname</name>
, K' ]: W% M3 w, }0 L <value>0.0.0.0</value>
0 |/ s2 @) u1 f! g& n </property> ; {1 E. D n J+ g6 p' b9 f% A
<property>' E; ? u1 h% c& O
<description>The address of the applications manager interface in the RM.</description>4 Z! S+ { U& n' b3 {
<name>yarn.resourcemanager.address</name>" o4 e* v, |( l# l' T+ F5 e n
<value>${yarn.resourcemanager.hostname}:8032</value>9 L, o( M1 s" R! J$ R; M
</property>
. c2 z4 K8 W/ b' o" j- D <property>: t4 i% x3 @. r/ T; \! M2 n
<description>
2 `- [% E) k4 a The actual address the server will bind to. If this optional address is
( p/ M- y n$ N2 J6 V0 h2 K% {, z set, the RPC and webapp servers will bind to this address and the port specified in
, h. w" s" C8 X& N& L yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This& W2 q/ I4 [0 Z! e; a. {. Z" @
is most useful for making RM listen to all interfaces by setting to 0.0.0.0.6 U6 |/ h0 |7 y- M+ q9 T( {
</description>
1 h) O9 s% b& ?3 d) I <name>yarn.resourcemanager.bind-host</name>
3 j/ y9 I! M8 W3 G* \; t3 @1 F <value></value>1 S. N6 A, n/ e- p2 _- ~9 M. c/ I' M
</property>3 W. X6 J- s: U; r2 p( q
<property>
1 P2 ~! i& T1 {& _ <description>- V" @6 o4 R( \2 l( A1 B* s
If set to true, then ALL container updates will be automatically sent to& E- |* ^6 \! W/ T% }7 y, K8 v
the NM in the next heartbeat</description>
T8 z% Q r8 | <name>yarn.resourcemanager.auto-update.containers</name>
# l9 n- B- m6 T; q) g <value>false</value>
4 E( P ~1 b3 O) L* q </property>
: H- k3 _) h% _% j <property>
: Q! \; V+ x; S! O/ b! d <description>The number of threads used to handle applications manager requests.</description>4 Q( \9 A# T( E2 \* s
<name>yarn.resourcemanager.client.thread-count</name>
4 D# j1 y) Y% q+ y9 ^ <value>50</value>$ }3 K) ~4 C# ^/ p; v+ O
</property>! I1 n" ~( V3 S+ J. n, h, }+ J
<property>* n- r# e; O+ |2 p' k
<description>Number of threads used to launch/cleanup AM.</description>* \6 c! J1 [/ b# A* U/ ?% \
<name>yarn.resourcemanager.amlauncher.thread-count</name>2 @2 t( M3 b) [" D5 o
<value>50</value>
1 t) \; p9 Z7 f/ `$ c </property>
% _* T5 v: o6 _- h/ e <property>- Q5 _6 V8 q& A: Y5 Z. @' R* G3 u
<description>Retry times to connect with NM.</description>
+ C- F8 p$ k# [) C. l) K <name>yarn.resourcemanager.nodemanager-connect-retries</name>
" i( \; F: w6 a0 X3 f$ B' W <value>10</value>
7 ]. n+ T Z- M" ^% O1 R3 H( O. f </property>6 Z$ s2 g K F( V# g1 `) ] ~$ x
<property>
3 P/ H' `' \9 R2 \ ^7 v `' N <description>Timeout in milliseconds when YARN dispatcher tries to drain the
8 c0 u6 s8 J+ a7 N5 p0 r5 d events. Typically, this happens when service is stopping. e.g. RM drains" ^ e" i" b& R) T6 R% B
the ATS events dispatcher when stopping.4 W7 u/ p& h6 c4 J+ U
</description>
@1 c+ A' B7 o/ {9 m8 g <name>yarn.dispatcher.drain-events.timeout</name>5 i) ^* N3 s3 s. l6 x+ K
<value>300000</value>% \ l! S1 M, N3 C9 D( W
</property>8 M7 H6 B7 }/ n) ]" B2 R% y
<property>' Y! e8 W- O9 y" G
<description>The expiry interval for application master reporting.</description>) C: _. b* r2 n; a4 @7 S
<name>yarn.am.liveness-monitor.expiry-interval-ms</name>* z& T0 t) J: S' e
<value>600000</value>
- k8 G* M% v# i5 e P+ a( D0 j </property>3 Z" z7 A# G/ y
<property>5 @ J3 G( ]( u/ M
<description>The Kerberos principal for the resource manager.</description>
6 y9 f2 `4 x- i- f# [3 a4 u' b <name>yarn.resourcemanager.principal</name>% O9 k+ s" w ?1 R( v- U
</property>6 X7 w0 Q1 w( h0 r* ?
<property>
* L3 Q4 w4 T) o6 r1 U9 r2 W <description>The address of the scheduler interface.</description>$ I% V& @! N: i' q: h1 n! ^4 S
<name>yarn.resourcemanager.scheduler.address</name>6 `2 J8 D4 u5 U. M
<value>${yarn.resourcemanager.hostname}:8030</value>2 p: | L! u. Y h5 U9 R
</property>2 t8 c B H' z, R7 P. ]# L5 C0 _6 P
<property>
# z/ h8 ]0 L! s5 U7 I <description>Number of threads to handle scheduler interface.</description>
- }7 ?6 ^/ ~ N# Y0 z <name>yarn.resourcemanager.scheduler.client.thread-count</name>
0 J& O# A0 C* O* c+ L1 q1 u$ e6 ?& | <value>50</value>! ?/ ~5 B$ e; n
</property>
) W/ N2 P; J& @. {5 f" _ <property>% _' c6 K! }2 Q+ d3 k: j
<description>
3 v" E2 F1 ?4 u Specify which handler will be used to process PlacementConstraints.4 M- r# a( ]5 ^; T4 U5 [
Acceptable values are: `placement-processor`, `scheduler` and `disabled`.
0 V% X+ {6 z* S& G4 M% F For a detailed explanation of these values, please refer to documentation.8 J6 {2 B4 |0 H! y( n) u5 I/ R7 B
</description>
% C5 X: S' ^! z, X" c% i; T( q; M! c. { <name>yarn.resourcemanager.placement-constraints.handler</name>
2 o" o, _& i0 r' P% ]- Z <value>disabled</value>0 G6 }- X) R1 K% M7 _
</property>3 y) Z; I4 _3 b4 T) G$ w
<property>
J( j$ l/ g" v8 |* I <description>Number of times to retry placing of rejected SchedulingRequests</description>
0 J: i0 `8 A7 Q# ?. j <name>yarn.resourcemanager.placement-constraints.retry-attempts</name>3 |, B7 v! [; ~( P1 o, l8 |1 m
<value>3</value>
/ t! E% M' i6 [$ F </property>5 |4 `, o0 R) m* V9 `# ^7 z
<property>3 r ]6 z& {/ ]+ U' D
<description>Constraint Placement Algorithm to be used.</description>
" c" W8 L. X ]& M0 A4 O <name>yarn.resourcemanager.placement-constraints.algorithm.class</name> ~5 G, u+ j7 B; \8 A
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.DefaultPlacementAlgorithm</value>
+ o' U% f/ M: l" t7 f </property>8 k# R) ~! ]! M. s* I* P3 `0 ?9 ]; F( N
<property>
8 q5 Y2 u& D' R6 y4 z( ~ <description>Placement Algorithm Requests Iterator to be used.</description>$ Q- U4 q, w3 I& u8 a
<name>yarn.resourcemanager.placement-constraints.algorithm.iterator</name>1 N9 Z& X1 [$ O8 L: V2 t, C) T/ c
<value>SERIAL</value>
) J" t0 m+ q1 }; | </property>; M$ g$ ^ ^! U4 H! ^
<property>1 | B' _$ o# M0 ?1 u6 A
<description>Threadpool size for the Algorithm used for placement constraint processing.</description>
! r& Q! t2 E% ?, v' V6 {$ y <name>yarn.resourcemanager.placement-constraints.algorithm.pool-size</name>
( h4 {7 O3 b1 n0 H <value>1</value>
, p7 o5 O9 Y9 d( k" X </property>
" D; x/ N% m& e" n7 B <property># B! E* K3 W3 [3 v0 v
<description>Threadpool size for the Scheduler invocation phase of placement constraint processing.</description>
0 n) g: L' k% P0 M$ g, b: u <name>yarn.resourcemanager.placement-constraints.scheduler.pool-size</name>* b8 J7 c/ K* g2 a2 {7 C
<value>1</value>& k% B) I$ F& f) B- ~0 Y2 b0 M5 i
</property>3 d, p8 M$ f5 d( O% N9 A2 ^; F
<property>1 M6 U* V: h' ~- J7 Z( `+ h
<description>
, \, H+ }# }, P( E Comma separated class names of ApplicationMasterServiceProcessor1 V, |: L9 Z J, F. e; B
implementations. The processors will be applied in the order
9 D; o9 R6 x& S* G# I$ n) Z: \ they are specified.) X9 g4 N' O9 Q' d
</description>: \3 x) V I; {7 ~3 V( P
<name>yarn.resourcemanager.application-master-service.processors</name>) Q5 B" `8 V$ X8 |9 P
<value></value>/ P2 p1 t/ n' U5 L% ]/ o+ H& R; K
</property>
8 {, r3 b$ p" V8 B3 d; {( d <property>
" d, e0 `. A, V$ Z) c <description>. n0 z( \1 S/ A+ I2 n
This configures the HTTP endpoint for YARN Daemons.The following
j" u7 { b% s" M; | values are supported:: f7 S, l e' X9 c
- HTTP_ONLY : Service is provided only on http
4 A+ c4 A9 D1 B$ Q$ _ - HTTPS_ONLY : Service is provided only on https+ H( n3 X' W; Z- h& i4 [/ H, A
</description>
3 m) W0 b5 @9 e' E! l <name>yarn.http.policy</name>; P% r# y1 s5 M0 }2 V$ V9 N
<value>HTTP_ONLY</value>
" o1 l% J4 D2 K6 x& k4 X% i: m </property>
; C. o& I# A8 n$ |, D <property>
! L, z, V, B0 l w <description>, J: b8 h1 f' d, R/ C
The http address of the RM web application.) z7 X6 d) d4 _$ S2 B: q& l( p0 o
If only a host is provided as the value,
/ ^$ a4 W$ x' @ the webapp will be served on a random port. m* @7 a' k* @8 i/ O
</description>$ s: l' D* B( G% S
<name>yarn.resourcemanager.webapp.address</name>
9 W6 P) ? K2 l; [5 W <value>${yarn.resourcemanager.hostname}:8088</value>
. s) |4 R$ R% D/ D) v" e </property>
/ Q. s7 w% d+ v: Q <property>
# Z% L$ Q6 j# ?! W- h3 U <description>0 X _* d5 H* V$ x& u, m; ]
The https address of the RM web application. _ ^ g, g/ x9 o" I3 Z
If only a host is provided as the value,4 O) b3 b* k% p; M; F" z* k
the webapp will be served on a random port.
- Y! q6 |. E, {8 P0 x5 d </description>
4 T& j; g8 t6 H5 Y8 k <name>yarn.resourcemanager.webapp.https.address</name>
3 J' T( Q2 j/ t, W: l <value>${yarn.resourcemanager.hostname}:8090</value>
4 ~# F2 \8 s6 A </property>
, ~/ U# k( [1 j) o <property>2 O1 [$ E# n4 x6 g Y
<description>- p5 l) n p5 H# J8 A3 D2 E4 W8 m
The Kerberos keytab file to be used for spnego filter for the RM web
- Z5 z U" `+ x% M( Y' Q interface.
1 P/ t, W1 K, k1 W </description>
8 j! L6 f, j/ t/ W8 H& m <name>yarn.resourcemanager.webapp.spnego-keytab-file</name>
4 O& f6 l! {7 Z( ~( g <value></value>
/ O. e5 c$ c$ z! Q. x </property>
5 m9 u2 W) ~7 l+ R* B0 T) ]9 ?/ D8 l: A4 ~ <property>. z& S% E+ f! e5 b6 Y) K
<description>
4 k6 U1 A( }8 r' R# V+ Y! r The Kerberos principal to be used for spnego filter for the RM web
. K+ C9 A0 A% _$ i; ~ interface.: _/ \1 j7 c3 m
</description>
& i! X- p* l+ P3 E <name>yarn.resourcemanager.webapp.spnego-principal</name>2 a+ O3 S8 \( S' H
<value></value>0 Y+ o! b4 E* E( D
</property>- O5 [6 W8 E: m' u+ A
<property>
* I( [( d6 {. o) d% S7 V8 p5 K <description># {5 ]( {$ Y: b5 A
Add button to kill application in the RM Application view.' W, I% B K& w- E3 @1 U& Z( d; B
</description>
$ d5 j) M- }, P* M* X, L9 u$ p <name>yarn.resourcemanager.webapp.ui-actions.enabled</name>; K8 ]) h% ~6 d5 i% \" }9 x; W
<value>true</value>
/ \+ W$ G- ]; F! q1 {1 D9 [ </property>0 c: r3 ?2 K$ i) b5 ]& ^) A
<property>; @. W9 l" z3 D7 P% A, O! M5 c
<description>To enable RM web ui2 application.</description>
8 ^8 t* ^( t+ D+ ?9 k <name>yarn.webapp.ui2.enable</name>
/ d5 y; X+ b; b* F J: r' s7 x0 z <value>false</value>) B! v9 A& b- j5 _0 _0 H$ z
</property>
0 `- L7 ?& ~ p9 F" H# | <property>
8 F B% H- R' h, i5 m7 G0 f <description>. }' S- G- B. t" P5 g
Explicitly provide WAR file path for ui2 if needed., r4 A$ J& S2 {
</description>
" v0 j7 i. y# ` i <name>yarn.webapp.ui2.war-file-path</name>+ L i# W# o! U3 ] q
<value></value>3 R' D$ L( `" K/ U
</property>6 |% K4 p& u" N! f* ^" J" N
<property>0 W) h+ E6 v2 \5 l" O
<description>4 l* [# G' O8 _! Q0 M
Enable services rest api on ResourceManager.* O( A" o& N# ^4 o
</description>
9 T( b L) {3 s1 A/ U& \1 A <name>yarn.webapp.api-service.enable</name>- \- _& x1 h V* d2 S" s& \9 S& g
<value>false</value>
( v6 B) U' c% Q </property>" Y- T" Q( L+ x2 M, ? D
<property>
* T9 n8 Z" M6 R2 V6 Z( ~5 C; y; U1 c <name>yarn.resourcemanager.resource-tracker.address</name>
0 q1 [( j, |9 D# n8 P8 u <value>${yarn.resourcemanager.hostname}:8031</value>/ G/ N5 ^+ e& m
</property>
- h. W8 Y0 L" u2 h8 q <property>1 } x( Q. V' N9 K+ T. P
<description>Are acls enabled.</description>
; W2 a, X) ?1 a* j' w <name>yarn.acl.enable</name>* A M( }. c- l$ a! b
<value>false</value>
0 i: @( J4 T3 P; p" ?. u </property>; G. r I- l/ }" J: y
<property>
5 _' E6 Y7 e6 ]( q4 E1 y J <description>Are reservation acls enabled.</description>* f8 `' X3 ]6 ^8 Q; _" Q8 T) P) O$ v
<name>yarn.acl.reservation-enable</name>
1 K4 E3 M( r) f' H0 n2 } <value>false</value>
1 j5 f! T+ v+ ]* }) Z1 H. y </property>8 K$ q: I3 O# j9 Z0 N7 O
<property>
0 {5 v; `" {+ ?6 a$ J/ U" m <description>ACL of who can be admin of the YARN cluster.</description>
* A E5 v, A$ h# N! k3 T9 {. l <name>yarn.admin.acl</name>
+ [. Q' ~3 D' i- k$ h I- \ <value>*</value>
4 Z; C4 N% ~- F. e6 k' f </property>/ _- N- c: F4 M! B( {$ Q
<property>
4 D' Y+ X1 K2 b6 L) l2 D <description>The address of the RM admin interface.</description>, Y/ ]) c: \1 A' l6 H7 @0 O
<name>yarn.resourcemanager.admin.address</name>
) Q! `* `7 A& M; {& Q$ O <value>${yarn.resourcemanager.hostname}:8033</value>
* k1 h3 T7 f. H" [* g: { </property>. ~4 o& ?2 p; d
<property>8 K) R+ L, E6 k. ^4 \
<description>Number of threads used to handle RM admin interface.</description>. j0 @/ Y$ a& c
<name>yarn.resourcemanager.admin.client.thread-count</name>
1 j7 _ e; \* i: \! I' s <value>1</value>/ |- Q3 m6 H) O# N# O
</property>
/ [! i/ m& Z# V; Z* B- J, \ <property>
* y) P+ i$ t- p <description>Maximum time to wait to establish connection to
8 A8 b- S6 q) j' d ResourceManager.</description>
/ V( G1 S |7 `$ { O. N/ i% q <name>yarn.resourcemanager.connect.max-wait.ms</name>5 E. \: X9 Q9 a8 y6 L
<value>900000</value>
( e# z) y- J) ] </property>3 @1 K2 f3 J- }5 \1 l& r0 Q
<property>
5 t; V$ X5 m6 o* s: X6 u# i/ Q8 Q <description>How often to try connecting to the
& b% _5 }) L8 O) ^ ResourceManager.</description>
% y/ Z' ]5 Q" A y: t# L <name>yarn.resourcemanager.connect.retry-interval.ms</name>
# j ^1 [: L* r+ h7 K) Z( C" N' Y <value>30000</value>
7 w$ q J3 N# N </property>
4 m$ I9 Q/ T/ W4 c7 N" [ <property>" l) p& [- \/ C% c- x; }2 @9 |
<description>The maximum number of application attempts. It's a global
0 i! _7 c! E+ t- W6 C2 { setting for all application masters. Each application master can specify
6 ?2 U3 W7 e! X( \/ v+ S/ K its individual maximum number of application attempts via the API, but the0 {* O3 e: l0 x( b3 X3 e$ f
individual number cannot be more than the global upper bound. If it is,: h# V4 }1 L) }$ x" a3 L
the resourcemanager will override it. The default number is set to 2, to: G0 z; {& ~6 P) S& m4 m# v0 w' r
allow at least one retry for AM.</description>
0 i8 ~6 x3 i$ E* u* @6 I <name>yarn.resourcemanager.am.max-attempts</name># g8 H6 M- ?, Z: p% ^, H+ u
<value>2</value>
9 r& {% }% F0 H" @, t- X9 Z </property>/ N4 C# \% \$ l( `5 D, X: @
<property>( C. j3 f/ [3 q$ t3 p
<description>How often to check that containers are still alive. </description>- X7 d: ]. \' \3 I! X
<name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>
; X7 W: Y1 u: h: I% b <value>600000</value>8 b/ `! n' n- @. g
</property>% ], W1 l, f# n! c
<property>) i1 v6 P0 ^& z0 e" O! h. H! y1 m9 S
<description>The keytab for the resource manager.</description>
1 `$ m6 n1 y1 j9 t1 x0 r <name>yarn.resourcemanager.keytab</name>4 h* }+ O4 R W1 @) `! F7 P
<value>/etc/krb5.keytab</value>( j% D- @5 _1 \
</property>( B; @8 i" \& H- C5 D' K2 K4 w8 O
<property>& N* V4 C8 u" o! O2 M5 G
<description>Flag to enable override of the default kerberos authentication
$ e; F% `- H* L4 B4 q8 C7 @5 K' l filter with the RM authentication filter to allow authentication using
! Y0 [& a# Z+ _8 m delegation tokens(fallback to kerberos if the tokens are missing). Only
" C* I$ D0 {% i, i6 N applicable when the http authentication type is kerberos.</description>
* U) S; Q& g/ T4 [$ j <name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name>
& }0 C% q% q6 S, S2 @ <value>true</value>9 l: _3 q6 o8 u' G
</property>
1 T; n) y6 C5 O7 [! I <property>1 V9 z* n2 t, S$ g: I; l
<description>Flag to enable cross-origin (CORS) support in the RM. This flag
0 m" e0 P: L6 m; A/ j requires the CORS filter initializer to be added to the filter initializers. q8 y/ ?& j8 \0 [2 {. }
list in core-site.xml.</description>
- L& d$ T1 a' K3 v: A <name>yarn.resourcemanager.webapp.cross-origin.enabled</name>& P/ i2 F: V# C2 @% u
<value>false</value>
) B$ @" m9 b! ?+ i$ M) p+ _, E </property>; n- X+ I* g( }% H- }
<property>
! k6 R6 x/ p: G+ R) N7 s$ F% e <description>How long to wait until a node manager is considered dead.</description>8 q* b* \' q; V
<name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
; J' H: v1 V- j <value>600000</value>; T" ~! U' [; D5 D* x0 p; J
</property>4 `: G2 o. _6 m; ^) {" N- C
<property>
: n( n' v1 T" Y# L/ C <description>Path to file with nodes to include.</description>' D$ }( z# l) y# L
<name>yarn.resourcemanager.nodes.include-path</name>0 t9 j# j7 A+ R9 B5 t& Y1 c
<value></value>' |2 j- D: ~$ n& C+ \' L8 c
</property>* ^( e8 z" {1 n( _& R
<property>7 Z- J' s( O0 z; K& ]: |
<description>Path to file with nodes to exclude.</description>
& R8 S5 U/ Z( [# m! v# ?" A <name>yarn.resourcemanager.nodes.exclude-path</name>
& g2 }! m4 O0 h( g% ] <value></value>, F: h! [* [: C/ K
</property>
# Y* L8 c9 [6 L% e. Y+ x( a <property>
- h6 z! L# e0 C& r1 ^ <description>The expiry interval for node IP caching. -1 disables the caching</description>2 `0 {* v) z4 o& ^3 s
<name>yarn.resourcemanager.node-ip-cache.expiry-interval-secs</name>
2 J- H$ g; _5 u$ M' A- w4 e <value>-1</value>9 _/ S8 P% ^7 b( k. `! ?( s1 t0 u
</property>8 n4 {, n( F2 J. W1 q! N6 x3 Y
<property>
" c- e A, m( o8 `; g <description>Number of threads to handle resource tracker calls.</description>+ }6 L" c& z4 `, S+ a
<name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
6 U3 @2 }! d4 M" {" Z+ R <value>50</value>' m/ u' l( L6 P" x& _
</property>4 |9 `+ J9 E: w
<property>: N1 z, Y* q4 x
<description>The class to use as the resource scheduler.</description>
# N/ N0 x: O. B/ R <name>yarn.resourcemanager.scheduler.class</name>
. ~: }% c q2 C Y* w) c1 o <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>$ V+ ~7 _4 H5 R6 g4 A5 u9 p
</property>! i5 Q% ~0 h' e- f4 D
<property>- |7 ?. N# k7 U: ]' u% q% O3 Q
<description>The minimum allocation for every container request at the RM
3 B' q9 @! F& S' T7 m) }9 [; e; u in MBs. Memory requests lower than this will be set to the value of this" C. F+ M/ A& ~6 z. n/ V3 O
property. Additionally, a node manager that is configured to have less memory' h3 ^( \" y0 T0 _$ b# j; x9 M
than this value will be shut down by the resource manager.</description>
" a" |" _2 c1 z0 P# z <name>yarn.scheduler.minimum-allocation-mb</name>
4 o; |# \) P; d S1 j( i <value>1024</value>
0 ?5 L+ n0 d/ G </property>
, O2 d8 ?5 E X; `; ^+ V <property>0 f0 s3 V4 e2 C
<description>The maximum allocation for every container request at the RM
) i% N7 ~& }# K" M# u in MBs. Memory requests higher than this will throw an
- @& ]& J- a; n; y* ^) J: t InvalidResourceRequestException.</description>
0 h( ^4 a; \" E& C, z. z3 p <name>yarn.scheduler.maximum-allocation-mb</name>
) N8 o1 J7 D4 g" X' g' j+ g# N! d <value>8192</value>6 u7 N' c! l& i) t# ^' N9 d2 D
</property>
" V: y+ V7 H+ B {' f' | <property>7 k# {5 n7 B( c
<description>The minimum allocation for every container request at the RM
; | Q/ u0 A$ v: T in terms of virtual CPU cores. Requests lower than this will be set to the
- }- M% b' i2 m# a1 Z- L% e value of this property. Additionally, a node manager that is configured to7 ]1 i( x5 P0 Q2 l [: w! o, O$ j- ?
have fewer virtual cores than this value will be shut down by the resource; [( ?1 N% L/ X* |
manager.</description>
! c7 ?* W- Y( s- V: B <name>yarn.scheduler.minimum-allocation-vcores</name>
P7 y7 _! p% ~: A5 q <value>1</value>1 K7 }; M6 V, [$ Y3 k$ f
</property>
! {2 r$ O' F5 X8 E) h* `9 C <property>1 I" @9 M& e4 y/ P! P. u4 r* \' Y
<description>The maximum allocation for every container request at the RM6 z* r. D/ H7 X6 ?6 T5 T
in terms of virtual CPU cores. Requests higher than this will throw an
4 q$ B. e4 O8 m6 i InvalidResourceRequestException.</description>2 q! P! r! G/ v3 ^" u8 K
<name>yarn.scheduler.maximum-allocation-vcores</name>+ T* o7 c O9 |* _0 r/ H, P' w
<value>4</value>; n7 q" @0 F; n" v+ t" N
</property>
- f1 P% L/ W; Z: m# b4 W% | <property>
2 V7 H1 J+ ~, k! C <description>2 s9 T' T. y' t4 ~4 q
Used by node labels. If set to true, the port should be included in the
% b3 q) J, z8 S% Z/ E node name. Only usable if your scheduler supports node labels.
% l. N2 j5 z$ m* `/ n </description>! @$ Y9 K" m: T
<name>yarn.scheduler.include-port-in-node-name</name>. t' q- V* U+ M
<value>false</value>
5 u7 e5 k% Q( ]+ k. W+ e </property>; b) X7 S+ v9 J8 r
<property>
' k* C- o7 Q5 N& L$ w+ n3 J <description>Enable RM to recover state after starting. If true, then9 [7 l5 v" M" y4 ?( `
yarn.resourcemanager.store.class must be specified. </description>7 \1 P- W# a4 r$ R7 m9 L
<name>yarn.resourcemanager.recovery.enabled</name>
- ]2 L/ I1 W5 q6 L6 H, C <value>false</value>
8 y+ E/ a. L7 j$ Z" [% R5 D, O </property>" B3 e) {+ `3 o9 d3 l
<property>% n% A% t, _6 V: Q- q
<description>Should RM fail fast if it encounters any errors. By defalt, it
7 H; @2 K: Q1 {4 T$ W' i: m points to ${yarn.fail-fast}. Errors include:
) `& A9 m/ I7 r% _" Q9 L+ M 1) exceptions when state-store write/read operations fails.9 `) V+ N% ^! q6 \" i/ x2 B* A
</description>0 d- Z3 _* i" m8 G6 m: X9 r5 J
<name>yarn.resourcemanager.fail-fast</name>& |2 `- d. Y4 U8 N
<value>${yarn.fail-fast}</value>
& a- x5 M5 s8 \& Q) `. c </property>
9 F4 i+ O8 [$ i% _5 w <property>: g6 I' t9 r, G* D- C3 R( y4 a& M
<description>Should YARN fail fast if it encounters any errors.& B2 P+ E' l4 i$ g D( ]
This is a global config for all other components including RM,NM etc.* n2 X% r5 H& z& L; b
If no value is set for component-specific config (e.g yarn.resourcemanager.fail-fast),
# ?, Z+ A O, x# f0 }* r2 w this value will be the default.0 e! ?* I* Q+ C" t% r' z7 {
</description>
( y" D$ ]" m2 `* v2 u <name>yarn.fail-fast</name>+ ]" R J4 P$ V5 L6 i
<value>false</value>9 _& s( v' @% [, C( P
</property>
# d7 C: X3 P2 Y0 K <property>
" V# V6 k5 G9 a; k4 Q <description>Enable RM work preserving recovery. This configuration is private
9 E1 b3 E5 E/ e* U D to YARN for experimenting the feature.$ B U% r# J, {+ u
</description>
% _) A h. B3 }2 i; l <name>yarn.resourcemanager.work-preserving-recovery.enabled</name>! q4 w1 n4 B# _# Z1 j* C' X, U: o) X
<value>true</value>
" `5 d& {5 _, {' Y4 p5 B </property>
! D+ u2 Q% E( N <property>7 Q- A. V2 k' ^ z; f
<description>Set the amount of time RM waits before allocating new7 R/ G! e5 \7 b+ s, Y
containers on work-preserving-recovery. Such wait period gives RM a chance
; K D4 j% `6 M7 m: i. ^ to settle down resyncing with NMs in the cluster on recovery, before assigning
1 w( L n# ]7 t; ~ `. L+ C new containers to applications.. I# D$ p* x8 h( x' S& `" a4 w
</description>, K' z2 Q5 ^5 {- a* P' D
<name>yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms</name>
' t& s3 ?4 e( m <value>10000</value>& n2 i! S- n' o- n# K
</property> a% A; D$ b ^0 F5 ^* I& [6 g4 W; V
<property>1 j1 X: k) I& ]( Q2 J4 k0 H( Y% H- z8 O
<description>The class to use as the persistent store." J; k% p3 C* x& {$ [
If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
/ d$ _* G1 B: p* Q is used, the store is implicitly fenced; meaning a single ResourceManager
) ~! x* e9 u: i+ x$ ] is able to use the store at any point in time. More details on this
% w' u. n G4 h7 ? implicit fencing, along with setting up appropriate ACLs is discussed& f) S1 \; l* |
under yarn.resourcemanager.zk-state-store.root-node.acl.! C0 t! |$ U8 ~/ k) C7 i. X
</description>1 d& U" Y" q* a' k, E
<name>yarn.resourcemanager.store.class</name>
( Z0 @ w, x9 [: v6 W <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
) `% O- x% k; n" M- K, q; _0 ]4 Z </property>
9 C" a6 j0 W0 t <property># D, J! J) l+ C6 x7 @7 v& K/ R( u
<description>When automatic failover is enabled, number of zookeeper" \" f, z9 r6 \, M* ?2 I
operation retry times in ActiveStandbyElector</description>$ k- Q3 O5 R8 ?( ^
<name>yarn.resourcemanager.ha.failover-controller.active-standby-elector.zk.retries</name>
( D( G6 K6 h) }' F) V }! X1 j <!--<value>3</value>-->- q2 X. O" E4 S" d' V6 [4 x# ~
</property>. ^& @; Y1 j2 x* o# _
<property>
8 T6 \; g6 L: o. `' B+ y$ Y0 Q <description>The maximum number of completed applications RM state
1 A! y7 d9 q! d& t+ F) E store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}.+ r2 r3 O# ?; F1 q
By default, it equals to ${yarn.resourcemanager.max-completed-applications}.: L) ?3 a1 p) J# R9 h1 W
This ensures that the applications kept in the state store are consistent with
( P9 |# P) F5 L' n2 o0 g# v the applications remembered in RM memory.! X0 S* F+ [- U$ X2 X
Any values larger than ${yarn.resourcemanager.max-completed-applications} will# l+ f/ l! ^) ]" O1 i" W
be reset to ${yarn.resourcemanager.max-completed-applications}.# c- U3 }( B/ k$ n2 S
Note that this value impacts the RM recovery performance. Typically,3 B% |) |- w" Y& f
a smaller value indicates better performance on RM recovery." a0 T) H) |9 F: B) n o" l
</description> Y% i( R+ q+ ~- v6 U2 o
<name>yarn.resourcemanager.state-store.max-completed-applications</name>+ t' T% G, ?2 x' g$ Q |/ |0 s
<value>${yarn.resourcemanager.max-completed-applications}</value>5 g4 W5 S B1 _7 W. W6 P- P! |$ r
</property>
2 q( ]9 l7 ?9 K# U8 g; t <property>
- V8 ], g# a2 p <description>Full path of the ZooKeeper znode where RM state will be, c0 b: v9 J# N o: b5 f& m
stored. This must be supplied when using* W/ ~! B) I m( a8 p, M) |
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore3 |3 U' b$ n0 i# |: b
as the value for yarn.resourcemanager.store.class</description>4 g; K7 u2 K. {
<name>yarn.resourcemanager.zk-state-store.parent-path</name>) H @% g5 m2 E9 O+ `' W
<value>/rmstore</value>
! a/ k/ r/ P+ J# m1 O0 g </property>$ N) l1 n% `$ S1 g0 a7 I7 s/ c
<property>
+ ~0 C( a# l' a <description>6 i! g$ L7 R3 }/ ]/ i
ACLs to be used for the root znode when using ZKRMStateStore in an HA
1 A, n8 a: d. E* v6 m* ]+ I scenario for fencing.7 u' Q8 j+ v) E, C( ]# C' y* ]
ZKRMStateStore supports implicit fencing to allow a single! ^ I) C, t% X- C
ResourceManager write-access to the store. For fencing, the" o2 s c% J: d$ n3 G
ResourceManagers in the cluster share read-write-admin privileges on the
/ _( t" g: T6 X$ [( I/ k root node, but the Active ResourceManager claims exclusive create-delete
+ A6 b9 s% o- z3 \3 ? permissions.
' f- i4 I N# ?( P# E By default, when this property is not set, we use the ACLs from
+ @* ~' d& M- O. y9 _" b2 L8 t yarn.resourcemanager.zk-acl for shared admin access and
5 l1 E+ p" K2 _/ r- o2 }( f rm-address:random-number for username-based exclusive create-delete: F) Z; I3 O! C; d* x
access.- K/ ~/ `" G2 U3 ], ]% I
This property allows users to set ACLs of their choice instead of using! g: x3 u/ S! ]; b- n6 g" ~+ U
the default mechanism. For fencing to work, the ACLs should be
R/ G" `, @$ I* S9 Q carefully set differently on each ResourceManger such that all the1 E1 p/ O' k$ m9 G: f
ResourceManagers have shared admin access and the Active ResourceManger
, V+ d5 o' @) g# |& Q2 X/ k takes over (exclusively) the create-delete access.
/ m9 a% s+ g' Z5 a* k% c1 z" i </description>2 S; h( i6 c! ]- D' N, k# m
<name>yarn.resourcemanager.zk-state-store.root-node.acl</name>: N$ y6 ^) i. l
</property>
8 _: Y/ I. x$ d+ G( r <property>
1 O' I0 K: D- h9 i* B, g <description>URI pointing to the location of the FileSystem path where
, ^0 A* N/ J3 g% b$ i RM state will be stored. This must be supplied when using1 _6 B9 T: O4 X5 L# S
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore) m* j7 |& j$ c; K- _
as the value for yarn.resourcemanager.store.class</description>
8 ~; I6 h. c, n7 k" }, k. p' \ <name>yarn.resourcemanager.fs.state-store.uri</name>
, X8 T9 C( X4 ^, f <value>${hadoop.tmp.dir}/yarn/system/rmstore</value>, U' c) w- F8 M% Z* B
<!--value>hdfs://localhost:9000/rmstore</value-->
) [' x* w' O* r) e/ U* t3 j+ ?6 O </property>
" Q) x9 L6 \/ ]2 ?& p4 L! V! Y4 z <property>5 W& p* f2 W; L
<description>the number of retries to recover from IOException in/ V) h; Y, H- S: A1 E
FileSystemRMStateStore.
/ w7 g- Q6 L8 h/ @: t9 E/ e </description>. H2 l( W$ Y+ }* Q1 `; @" x3 f
<name>yarn.resourcemanager.fs.state-store.num-retries</name>
' A! p6 _) C; _& E <value>0</value>. f0 L# H6 g# a% K4 t0 H0 m
</property>8 N5 K% W; V9 U& s) i3 i! i
<property>* m+ i+ n% j' d' S) o
<description>Retry interval in milliseconds in FileSystemRMStateStore.
! ]" i4 @: f2 v8 v$ U </description>
. C9 F" ?- F4 ~ <name>yarn.resourcemanager.fs.state-store.retry-interval-ms</name>% N; A+ v8 ~' ?
<value>1000</value>
9 x7 j+ J; M; p5 Y+ b) _# ^' Z </property>
# t" `% [# f8 c8 m0 R- t <property>& V4 w$ T3 W/ M3 ~& f3 }; K
<description>Local path where the RM state will be stored when using
4 w% K( W) a9 s org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore
6 u) L& } v$ z/ R& e as the value for yarn.resourcemanager.store.class</description># h5 A7 F2 ]% d4 \
<name>yarn.resourcemanager.leveldb-state-store.path</name>$ i- z9 g ~; ?/ F* Y% W0 p- j) @
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>
7 o% r8 a& M! K; { </property>
, E: u% g; p! u# X; Q& f6 | J <property>4 A: Y' W t- V) D/ u
<description>The time in seconds between full compactions of the leveldb
$ C: S* [; i- h8 l database. Setting the interval to zero disables the full compaction R* Y6 f3 J" D+ `) Z: i9 p" T
cycles.</description>
# Q4 u0 A3 p; L [. B <name>yarn.resourcemanager.leveldb-state-store.compaction-interval-secs</name>8 u. E+ o1 h% u! n* t+ v4 l, s
<value>3600</value>
% a4 |. k, O( Z) a ?$ O& ~ </property>' w" T6 X% T0 z5 T" O. n- Y
<property>1 u0 T: ~, p0 F5 V; u# M
<description>Enable RM high-availability. When enabled,
! P/ w3 w, T( a& t1 K% K9 H( i (1) The RM starts in the Standby mode by default, and transitions to) U3 a# r+ d# D, y( }6 k; l" M, O
the Active mode when prompted to.2 _ q* A1 o- ~
(2) The nodes in the RM ensemble are listed in
& e! z' I# a6 O c5 Z yarn.resourcemanager.ha.rm-ids8 S7 d; R" r# d' i3 a! @, H
(3) The id of each RM either comes from yarn.resourcemanager.ha.id# c. f Q; U `% K, G
if yarn.resourcemanager.ha.id is explicitly specified or can be
) Y) u! E0 t# N# k/ ^6 S# c figured out by matching yarn.resourcemanager.address.{id} with local address
2 Y! U- Z# T0 T- o7 \- g7 g/ W% e3 ]! L1 j (4) The actual physical addresses come from the configs of the pattern0 ?9 ]- ~6 h5 S, l+ d0 N; @
- {rpc-config}.{id}</description>
+ j- g. [( q% e/ a% d" P <name>yarn.resourcemanager.ha.enabled</name>
D( p# m: o7 ]( I% X) k: y9 I <value>false</value>6 C7 O' D$ a+ x, f/ H+ M
</property>
3 V0 F6 h! t- O" d <property>; | R0 c# z1 k% a
<description>Enable automatic failover.2 w; ?' Y1 X7 t. ~2 z
By default, it is enabled only when HA is enabled</description>
5 T, @9 z- z7 Y" V9 z7 H <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>6 a- q1 }. M8 o' Z c9 n2 K
<value>true</value>
$ }/ F# y3 c) @ </property>
/ a) L7 T L' P6 d; Y' A1 ?& A <property>
$ m- a. j0 P% d: `% a+ o# ? <description>Enable embedded automatic failover.
# n) a) e$ T3 l' ?4 U i1 C% }* _8 H0 n By default, it is enabled only when HA is enabled.- f: W0 |4 M/ u' }1 v
The embedded elector relies on the RM state store to handle fencing,
9 X7 b$ \7 x4 K) n! h& c and is primarily intended to be used in conjunction with ZKRMStateStore.
. ?( D$ Z" R* l# O </description>0 I5 ?* ?$ P6 z: N0 o
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>5 M' ?) t9 i9 o' [3 X1 k7 h! u
<value>true</value>5 E; J+ p; e+ \3 [: }
</property>! R+ ~' K; V5 ]: @
<property>- N% K' S7 X0 E) j6 B
<description>The base znode path to use for storing leader information,
! r; x) Z8 z& C* N when using ZooKeeper based leader election.</description>( Y3 U$ L1 _$ v- W. c; L1 J
<name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>4 s; z* p: ?; b$ I/ j7 T
<value>/yarn-leader-election</value>
! a1 O P _8 }4 J& q" r </property>
7 L0 Y5 o5 u: E: i7 E+ c, ~ <property>
. D$ q& i: a) F <description>Index at which last section of application id (with each section* b( k% v( S& Q, Y
separated by _ in application id) will be split so that application znode
, q9 e! D! l4 Y+ o, U$ e) Q stored in zookeeper RM state store will be stored as two different znodes
' [* E5 f; e/ E2 U+ @4 j! b (parent-child). Split is done from the end.
0 H9 {' t/ }' i For instance, with no split, appid znode will be of the form8 S) C0 Y# P6 H P3 j
application_1352994193343_0001. If the value of this config is 1, the
' S4 \2 v" [% A. v! a8 }3 D5 C+ D$ L appid znode will be broken into two parts application_1352994193343_0004 U- d' D! n7 U
and 1 respectively with former being the parent node.$ A1 [8 i; X, }- Q1 [8 [) C, f
application_1352994193343_0002 will then be stored as 2 under the parent
: h5 e, V& {9 E* k9 ^8 q2 } node application_1352994193343_000. This config can take values from 0 to 4.
) m7 }- k- u3 Z. D Y% j; R. h c 0 means there will be no split. If configuration value is outside this
& d) j, O% T( H2 Q$ j range, it will be treated as config value of 0(i.e. no split). A value
O- T* g4 w( V3 i4 Z4 m2 x larger than 0 (up to 4) should be configured if you are storing a large number
% V+ S. R0 _0 B( X, k5 _% K of apps in ZK based RM state store and state store operations are failing due to6 b3 O8 n* w3 U# C
LenError in Zookeeper.</description>
3 i7 z3 R+ L( z+ Z4 Q <name>yarn.resourcemanager.zk-appid-node.split-index</name>
' X2 X2 }# L; m$ e <value>0</value>
, ~2 b9 Z" R$ X3 a </property>
6 p% t/ e1 A4 v7 H& o <property>% P. j5 m l$ j" M9 Q0 U& P
<description>Index at which the RM Delegation Token ids will be split so6 ]7 ^0 m# I+ m2 r' H
that the delegation token znodes stored in the zookeeper RM state store
' m q# n% G3 d) {' O. e will be stored as two different znodes (parent-child). The split is done6 q& h4 K) Y6 J, r) E
from the end. For instance, with no split, a delegation token znode will
+ }5 ?- Y/ m. p% r be of the form RMDelegationToken_123456789. If the value of this config is/ W- g0 k; P8 G8 y
1, the delegation token znode will be broken into two parts:# ~$ c# X; V) n: i& k
RMDelegationToken_12345678 and 9 respectively with former being the parent
2 [; g6 K* B! H- V7 Z$ B& Z+ L node. This config can take values from 0 to 4. 0 means there will be no
2 ~, [0 P. ?/ H. c6 J" D split. If the value is outside this range, it will be treated as 0 (i.e.
8 R1 }/ |- ~7 z! P no split). A value larger than 0 (up to 4) should be configured if you are# Z2 h8 d) b* O2 ?
running a large number of applications, with long-lived delegation tokens8 S" Y; z8 I" B
and state store operations (e.g. failover) are failing due to LenError in
. H* \1 f, U5 i5 j# O4 r Zookeeper.</description>* }$ v1 l8 S6 S* u5 e
<name>yarn.resourcemanager.zk-delegation-token-node.split-index</name>
( A, P8 f/ A( Z, p% f. G <value>0</value>" C& {& u- T2 \0 z
</property>! x( \. I% N B }% g7 f
<property>
- p! S0 c! E; } <description>Specifies the maximum size of the data that can be stored
! q/ n# l- i! F% @* ^ in a znode. Value should be same or less than jute.maxbuffer configured# q7 U" a# `/ Y! ^& \9 I, O
in zookeeper. Default value configured is 1MB.</description>
0 g$ C- N! O0 H, r5 w6 S <name>yarn.resourcemanager.zk-max-znode-size.bytes</name>. s3 |1 Y4 m# W: T! i: @. y% v
<value>1048576</value>& c, ]& Q( X! S+ u
</property>9 t- m% [* k0 A& k
<property>
4 n2 v4 L- e; R" e; j q <description>Name of the cluster. In a HA setting,) @2 d' g# d: f9 ?2 @5 v, C5 P
this is used to ensure the RM participates in leader: [8 }9 |: J: g2 X0 l! o2 a& }
election for this cluster and ensures it does not affect/ h1 @8 }" C; p w5 b
other clusters</description>/ ]$ K* [6 q! a( R7 v
<name>yarn.resourcemanager.cluster-id</name>. Q$ O9 u$ C+ p1 ]
<!--value>yarn-cluster</value-->; I$ i, Y5 `7 u7 G3 y, B8 x2 X! `5 M
</property> G. p4 ^8 K' h, y& ^# V2 C1 h
<property>5 {2 I# ~+ W o1 r2 c ]
<description>The list of RM nodes in the cluster when HA is
- q. g7 Z! B2 o6 C8 i" Z) f enabled. See description of yarn.resourcemanager.ha
. ~, D; x# \: i) ~: ~" ~ .enabled for full details on how this is used.</description>
7 c6 e: _/ r; y4 W4 E! K8 r <name>yarn.resourcemanager.ha.rm-ids</name>
. z& |4 w4 K7 Y- @' ` <!--value>rm1,rm2</value--># u; z% G/ \1 i; t" d) b- X
</property>' F5 v: K$ \2 W+ T/ {2 A7 ~
<property>
4 g( m }% _+ I5 t <description>The id (string) of the current RM. When HA is enabled, this5 H2 ]; x' f" ?7 g
is an optional config. The id of current RM can be set by explicitly8 p! O7 Y( Q" K! z$ _# E
specifying yarn.resourcemanager.ha.id or figured out by matching" z$ v" x5 H5 N2 K: v3 {4 Z
yarn.resourcemanager.address.{id} with local address* v/ p/ U0 P% {# D: P1 A6 @& f
See description of yarn.resourcemanager.ha.enabled- V7 i) T! N# z" _! @, w) s
for full details on how this is used.</description>
. f q5 l" J/ V4 ]! S <name>yarn.resourcemanager.ha.id</name>
- d1 L% f. E1 E" ~ S <!--value>rm1</value-->
3 Q# v+ D* E0 |9 M </property>
; J! K5 i, g. q4 e# t) K <property>
2 H6 O/ r8 L) J" N+ l <description>When HA is enabled, the class to be used by Clients, AMs and) v; A% o2 \! K% W, a2 N, w
NMs to failover to the Active RM. It should extend8 ~& Y0 S1 |! m8 q- U& I
org.apache.hadoop.yarn.client.RMFailoverProxyProvider</description>% m; _6 L" X& ^: `
<name>yarn.client.failover-proxy-provider</name>
3 }" I- P1 ?0 u/ P9 L <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>. ]& `( b. f R* G
</property>
* p- Q% `, @2 S( p' D <property>' ]- c" _" z2 G2 [7 `
<description>When HA is enabled, the max number of times0 o( B; Y. B _; D" r1 [9 [
FailoverProxyProvider should attempt failover. When set,
4 Q+ c' N. C& r' I2 E. y0 k9 p this overrides the yarn.resourcemanager.connect.max-wait.ms. When
n4 k1 S1 S8 r0 e5 { not set, this is inferred from6 x& O* K9 Z* h% {
yarn.resourcemanager.connect.max-wait.ms.</description>
5 {" B4 l. F, s! s <name>yarn.client.failover-max-attempts</name>
6 K c5 i% ~; t, @) }- Y <!--value>15</value-->% I( F: U J/ f% b }6 h: Q1 Y
</property>! e# A9 L9 w/ r9 f
<property>
7 q8 ?+ ?9 P9 N3 z <description>When HA is enabled, the sleep base (in milliseconds) to be
$ N1 u8 J& Z, V( N used for calculating the exponential delay between failovers. When set,+ m+ c5 i; A# y) p( v K% z: v
this overrides the yarn.resourcemanager.connect.* settings. When
, c0 G" ~+ b2 q( y0 z not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.+ c# d( @3 Q3 y" c- o1 }
</description>4 l/ J- l% h/ p/ v. _: y( v
<name>yarn.client.failover-sleep-base-ms</name>
' @* n \2 ~4 y <!--value>500</value-->
( p) [# o, i0 ?* a </property>. N% l9 _7 y, [( S( e' R
<property>0 j% Y# D6 D2 K% O/ F9 N4 k8 Q
<description>When HA is enabled, the maximum sleep time (in milliseconds)
& Y' n# N- q* P: p3 Y between failovers. When set, this overrides the+ u8 V9 C4 k) h, m' i8 M
yarn.resourcemanager.connect.* settings. When not set,1 m7 I$ Z' k. `3 V
yarn.resourcemanager.connect.retry-interval.ms is used instead.</description>1 ?* g! K/ a, \6 m% a
<name>yarn.client.failover-sleep-max-ms</name>
0 d+ A: X) g( J9 s <!--value>15000</value-->
8 L8 X Y% `4 H5 ^ </property>
0 Q& b) o3 ]$ U$ y& s3 G2 h% s8 q <property>) r) t7 C" N8 x9 f! v( a5 O
<description>When HA is enabled, the number of retries per% Q, z9 F- F8 l) l& N- X( K: x
attempt to connect to a ResourceManager. In other words,
2 ]# O0 Z! c2 I- T* x2 d" i. z it is the ipc.client.connect.max.retries to be used during
0 g! Y, q& |0 U5 X) V; t failover attempts</description> m9 U6 d- e" F# C$ b# i' f2 r/ ~
<name>yarn.client.failover-retries</name>2 G" |' l: u, I1 ^# O2 v5 q O
<value>0</value>, z' w- r2 p( S3 l& H' O6 }; c$ |
</property>
, R- D; I; l; h# s8 K8 e% h2 N/ H <property>
7 k, f$ J( [& K W <description>When HA is enabled, the number of retries per/ C3 h' L5 k6 M! L' T+ [
attempt to connect to a ResourceManager on socket timeouts. In other
" j( f2 Y8 X1 E6 o words, it is the ipc.client.connect.max.retries.on.timeouts to be used
5 h# G* [8 G" J4 b7 d during failover attempts</description>/ n5 Z# H( }+ j
<name>yarn.client.failover-retries-on-socket-timeouts</name>" l3 R7 u: U9 g
<value>0</value>
) j# s6 _( ^8 Z' x </property>6 h* q, ?1 h6 V* Q' h
<property>
+ l6 L! G, h2 H' @, m3 h7 ^ <description>The maximum number of completed applications RM keeps. </description>
! ]% Q% }" J' L0 S' D4 m, X5 k- ? <name>yarn.resourcemanager.max-completed-applications</name>
0 O1 b& ^* N4 R# t& ^ <value>1000</value>! m6 @: h% [1 o# N) B( A
</property>+ k7 N. \& b/ ]8 A) M+ K- d7 Z
<property>
) b0 |( C& V3 i8 b* A6 p i <description>Interval at which the delayed token removal thread runs</description>1 M6 @6 e* T. A
<name>yarn.resourcemanager.delayed.delegation-token.removal-interval-ms</name>
, ^, { A" l# b& [) l+ c <value>30000</value>
4 e, z+ a$ C0 y& H& t </property>
& ]) ^/ \6 m( K1 R7 g' `" l <property>( p! ]; l6 l& e
<description>Maximum size in bytes for configurations that can be provided4 k! t7 n4 h) {1 Z) b/ B6 |# N( a) Z
by application to RM for delegation token renewal.$ \+ E$ j" q" U
By experiment, it's roughly 128 bytes per key-value pair.5 k ~( ?8 `) U6 c9 x3 J! c
The default value 12800 allows roughly 100 configs, may be less.4 ]+ s1 j1 w' w2 i* P3 ^% o
</description>; [$ N' W5 O0 p8 e
<name>yarn.resourcemanager.delegation-token.max-conf-size-bytes</name>' I" A! i5 |8 K' ^
<value>12800</value>
+ c+ Q* S9 Y& I+ r/ h% Q: u7 {2 }3 E G </property>; k! d1 V( ^6 ^8 U
<property>
) n# \$ L V' j+ _' w <description>If true, ResourceManager will have proxy-user privileges.
0 m! T' V- p. [8 L Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to+ y) P+ A: c! s3 |! [* ?- Z' ^
do localization and log-aggregation on behalf of the user. If this is set to true,
- R5 N# J! P3 x6 `- g/ f ResourceManager is able to request new hdfs delegation tokens on behalf of
8 `8 m Q- z1 ?- P the user. This is needed by long-running-service, because the hdfs tokens6 M. t# r! D1 R9 u4 W& E
will eventually expire and YARN requires new valid tokens to do localization
5 a1 e- n4 U) i1 D2 r and log-aggregation. Note that to enable this use case, the corresponding
/ b! Q% X/ U; k8 ^ m1 X' ` HDFS NameNode has to configure ResourceManager as the proxy-user so that
2 T+ C" z' r8 Y$ o; U0 q# } ResourceManager can itself ask for new tokens on behalf of the user when
! p) Q0 m7 M8 j: ?3 J tokens are past their max-life-time.</description>
* ]3 L/ I: Z1 |; t' ?: S1 l; v <name>yarn.resourcemanager.proxy-user-privileges.enabled</name>4 G: q* M5 e$ P
<value>false</value>
( v! N* |3 Z7 P </property>. Z/ l! b! @/ K2 L
<property>
/ L4 \- t b5 A" y% {1 P& M <description>Interval for the roll over for the master key used to generate* J" g5 E3 N S- j* ?2 R( a
application tokens/ f% G" i3 e: F$ }. Y! H
</description>
. I# P. p% l; g5 u! c+ T <name>yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs</name>
% B0 y" j! | r; O9 F$ H <value>86400</value>1 q# S2 I: W) }& A1 W
</property>
9 {" m4 ]2 T3 X0 Q: Q& r <property>
! B# C4 |5 J, R' T <description>Interval for the roll over for the master key used to generate4 O+ V1 B, `* R, |! X k4 k
container tokens. It is expected to be much greater than% E# g9 f0 l: d+ Z; r' t* Q
yarn.nm.liveness-monitor.expiry-interval-ms and
. t1 W$ a. E5 s V yarn.resourcemanager.rm.container-allocation.expiry-interval-ms. Otherwise the
; q( z8 W# H5 e) M% B/ a+ d2 W behavior is undefined.
, O. x+ \# m( g' ]0 E </description>$ u; J4 L' T9 Q5 N8 U6 {
<name>yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs</name>- W8 G9 I" U6 e& ?9 G) _
<value>86400</value>
3 B9 s) R1 [/ v </property>
9 a4 N1 ~+ P4 z5 u; |( O9 s <property>
, ~: R. s/ g5 K, n4 { c" M' | <description>The heart-beat interval in milliseconds for every NodeManager in the cluster.</description>
4 S: @# K& h- t <name>yarn.resourcemanager.nodemanagers.heartbeat-interval-ms</name>
+ i6 G& K! d) J <value>1000</value>4 i% c8 Z5 W$ H4 R' c
</property>
1 h% b! J; x, J7 V4 {& s6 a9 J( N <property>
N7 s, d3 [( V% p <description>The minimum allowed version of a connecting nodemanager. The valid values are
' u6 N5 C/ V& B# K! Z# w NONE (no version checking), EqualToRM (the nodemanager's version is equal to
3 b7 W3 w7 U0 A/ U or greater than the RM version), or a Version String.</description>
7 J0 p4 k# P7 y. a" o1 H <name>yarn.resourcemanager.nodemanager.minimum.version</name>$ O: a( f( g- L% B w
<value>NONE</value>
$ v) h+ z" P" k ~ </property>. A4 f+ ^9 p0 b( T3 X1 K
<property>% A+ L' F/ Z* M5 e
<description>Enable a set of periodic monitors (specified in& C8 |" W& u2 ^5 Y6 J" y
yarn.resourcemanager.scheduler.monitor.policies) that affect the
8 c; `4 S6 T8 x" w# a scheduler.</description>
* _% b* M$ z* K- a" L! T <name>yarn.resourcemanager.scheduler.monitor.enable</name>: B$ ^( F: o" q* V9 b
<value>false</value>
' }9 g7 z* ]8 u; W, o. p0 h </property>
1 \/ d' s& p" Y: _' P+ @ <property>
; U2 S6 m9 J1 J <description>The list of SchedulingEditPolicy classes that interact with
8 O8 L, }, d2 d8 K the scheduler. A particular module may be incompatible with the
3 f$ j) q+ ?' F9 I0 e$ K scheduler, other policies, or a configuration of either.</description>' N0 I6 C, k7 L; P, _
<name>yarn.resourcemanager.scheduler.monitor.policies</name>
7 s: z% b5 F. s( L <value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value>- m3 x U4 j$ }
</property>7 u' C2 m+ p3 J9 @$ \$ W2 n: J
<property>
1 l7 H4 V& {* G5 g' F <description>The class to use as the configuration provider.
) y& l$ r3 A" D3 j3 T# G7 |# H If org.apache.hadoop.yarn.LocalConfigurationProvider is used,' h0 \8 w7 U+ o. x
the local configuration will be loaded.) e. v6 l, Z w- ]% {
If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used,
7 S7 l! N7 w9 J9 J1 |& [ the configuration which will be loaded should be uploaded to remote File system first.2 ~( r, Q( L1 u$ e
</description>
4 k8 _7 v5 r9 w J d; w& ?4 d <name>yarn.resourcemanager.configuration.provider-class</name>) n6 I# ]( {! h- G5 {; b# ~
<value>org.apache.hadoop.yarn.LocalConfigurationProvider</value>' V4 W' H4 Y( `$ b& p% B
<!-- <value>org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider</value> -->
& w' ?3 T h# j </property>% }2 @$ u, C( u* R' L. Q" i
<property>: S6 A' K ]- d: }
<description>
) Q+ _; |3 q5 r2 Y ]$ ] The value specifies the file system (e.g. HDFS) path where ResourceManager+ ?4 ^. @3 z$ S9 y! o5 b
loads configuration if yarn.resourcemanager.configuration.provider-class
0 o# `5 g: D# G- q- t/ Q( b. p is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider.
. ~3 s* d5 ^! u. [, j7 u4 W </description>+ o( |! J1 ^3 D& T
<name>yarn.resourcemanager.configuration.file-system-based-store</name>" U5 H6 X8 |" _2 G4 P8 G! Q
<value>/yarn/conf</value>
/ x0 q8 Y2 e8 c" ?, \" L" G </property>
0 G4 m$ r) ^# T6 h' |* O7 R* X4 C' c <property>
) s1 j4 |# a4 g2 \& { <description>The setting that controls whether yarn system metrics is
0 ^; I) Q. o) t+ \ published to the Timeline server (version one) or not, by RM.
8 q+ U! R$ C, d0 r8 [4 T2 k This configuration is now deprecated in favor of
7 b; J7 ?9 d5 J" ]& v0 g yarn.system-metrics-publisher.enabled.</description>
* r5 E/ x4 b5 j6 Y4 x <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
2 r7 q0 n0 w7 r9 P8 S) U+ w! R0 J <value>false</value>
9 _* D6 V* T$ T- V, N: C$ Q </property>6 H0 E) f0 C) [$ P: [0 @" W& W3 z+ K
<property>
! O5 O' ~; y; ~/ \) A <description>The setting that controls whether yarn system metrics is
" o2 b2 J$ r1 N2 C3 K& \9 C published on the Timeline service or not by RM And NM.</description>
# ~5 V, X2 i" c, ]& L* v0 |9 {- a _2 P <name>yarn.system-metrics-publisher.enabled</name>$ W; n) w% I: P" L# j- h
<value>false</value>
7 V. i+ d+ m# X, } </property>
: g$ }4 i& B& n2 S* p <property>& S# U. ?2 W! R+ i
<description>The setting that controls whether yarn container events are. X0 t: [1 C0 Z
published to the timeline service or not by RM. This configuration setting) q7 k' Y, i, g: ?
is for ATS V2.</description>& U3 |3 G6 X8 l; s2 M
<name>yarn.rm.system-metrics-publisher.emit-container-events</name>2 E0 Q; x9 x( D7 n- g' y: i0 w; `
<value>false</value> {5 i" U, U9 \/ Q9 r% n! K
</property>
6 s( @5 G" A: G/ p3 \ <property>
v+ u; A& t0 ^) o# h0 x6 P4 c0 v2 p <description>Number of worker threads that send the yarn system metrics
- z$ |5 n9 I0 p5 f, F2 n( I& H4 K data.</description>
) O, y, r3 N& z" ]7 L- g <name>yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size</name>
6 N) B6 S; v( s: t <value>10</value>
0 A2 _; H1 O' m4 q& B$ V7 R) p </property>$ S- d1 E& K$ n- R6 e( t1 I: n0 Y
<property>5 g) S9 ^, X5 N! G! _" J
<description>Number of diagnostics/failure messages can be saved in RM for
2 [( [" n' }- n( i; z4 }: V log aggregation. It also defines the number of diagnostics/failure* U5 ]8 U. m. R! \
messages can be shown in log aggregation web ui.</description>
5 i- }& C+ S6 G1 B, [0 N7 g <name>yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory</name>" b) o; `2 m# `
<value>10</value>/ V9 \7 r2 n* U& @. f
</property>0 i: {6 ]( \9 i$ y+ X
<!-- Node Manager Configs -->
) F; U _$ T: v! }" O, C8 L) _ <property>3 e |( Q. j0 V( S0 a" z& Q6 _
<description>
( E2 E/ |$ p; x* W/ f3 i- U6 P RM DelegationTokenRenewer thread count( v/ P! V: W2 `6 t
</description>! T- W i: Z7 A) \; u r
<name>yarn.resourcemanager.delegation-token-renewer.thread-count</name>
& |+ P' C6 o7 k" Q- Y* ^ <value>50</value>
; c4 q& v# s8 b) e) T </property>
& q; c$ G! ~, Q. R: Y+ T! x" R <property>( c8 _+ E) b8 Q0 v6 H8 w- N4 \
<description>. b: {( p6 o; v. Y, j! z
RM secret key update interval in ms
2 e3 V) ?2 G6 b </description># q$ c+ K% \7 b9 v& z
<name>yarn.resourcemanager.delegation.key.update-interval</name>( L9 ~, c8 g0 B$ s
<value>86400000</value>8 V: h4 P8 l; I. L
</property> D. w% k t: w8 F0 r$ a
<property>
* d' `" c* l ~7 {3 `3 F7 G <description>. c8 ?. Z( f7 z8 N7 o% a
RM delegation token maximum lifetime in ms
+ B6 K# O) [" \; H </description>: M2 w6 [6 O# P3 B
<name>yarn.resourcemanager.delegation.token.max-lifetime</name>
7 X/ A M+ Y. Q$ O* c: `* X <value>604800000</value>% m$ }+ a! Z8 M
</property>
S' `# h# }9 s3 x: g3 w; S9 k <property>
6 q# ]0 J" U+ O( k5 N2 j% F <description># B+ k+ b5 C# B# S2 w9 c+ }
RM delegation token update interval in ms9 X7 g) x3 O8 V
</description>8 r2 n' A! ]0 P
<name>yarn.resourcemanager.delegation.token.renew-interval</name>
. ^' }; q& `$ T <value>86400000</value>% P/ `$ m( j% W' W
</property>/ l& Q6 I( j& O3 [- l% o
<property>' |% \: E3 ^6 J% ^, f) V- n$ k
<description>
* M# Z) } l4 J Thread pool size for RMApplicationHistoryWriter.
4 J9 c0 H4 ?$ A </description>
) Y! N S8 G9 `( Z, F" u# f <name>yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size</name>
1 O! U: y: c* S$ y <value>10</value>( p9 u8 V- s$ f5 ]5 d1 a
</property>
* r4 \# u! A6 ^; e" T' @4 Z <property>
# ?' T' L0 u/ l+ M/ Q( p <description>
, p$ I8 _/ n: ^ \1 C& G- N/ c Comma-separated list of values (in minutes) for schedule queue related
$ S5 H" ~( C8 M) h) ]" J- v: a metrics.
; f( L8 B& y# p </description>
: l+ n8 I+ B s) i0 J6 s, D <name>yarn.resourcemanager.metrics.runtime.buckets</name>; J; \, N/ m) G# D: b# c0 d
<value>60,300,1440</value>
G* {0 l" W* `5 p+ O </property>
- ~$ W" a- V$ W9 A S$ m <property>- l' N6 D3 T& O( S& T
<description>( s5 @5 s8 `; N3 k9 P& U
Interval for the roll over for the master key used to generate2 u& [3 D8 I8 o2 E$ X
NodeManager tokens. It is expected to be set to a value much larger7 f5 g" L( t0 k1 P
than yarn.nm.liveness-monitor.expiry-interval-ms.1 Y& Q6 q r6 Y& d& K8 q; L
</description>4 g2 D l& ~% y( `" M* \
<name>yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs</name>
5 @! K( o5 J0 d <value>86400</value>* t* ]) Z1 J% x% `; C+ d: ^5 ?2 O
</property>
# u% u, t- K; Z4 I2 N0 K5 w, J <property>
/ D( w# m& s" I7 \- `2 H6 P* [. S <description> M/ Z y- l6 D. ^3 T9 j' G' ]
Flag to enable the ResourceManager reservation system.
2 |+ z5 G5 f9 Q4 F: y </description>
" e! p3 z" q+ O8 y2 k2 }4 j9 B <name>yarn.resourcemanager.reservation-system.enable</name>' F% t# P- J: ~ @5 Q2 ^# L9 \# v
<value>false</value> N* ?* [; W& s* n( w w' P
</property># [4 r& w2 D" e4 l3 u1 ~/ X
<property>
. |$ e# \' R. B1 w6 U, a( G0 I <description>
. o3 e7 B. D8 q7 k! ~+ g The Java class to use as the ResourceManager reservation system.; ~0 f# }1 o- w, c: L( w
By default, is set to* u! }5 }+ B V, U" n1 S9 G
org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem: J m9 m# Z1 l7 h
when using CapacityScheduler and is set to
8 A+ P! y! z1 Z2 ]. G% T1 z org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem
, T: R6 Z& ]. a: w6 v when using FairScheduler.$ T& P- T4 u5 j* G7 s' K
</description>+ `$ X: U% S2 F! A) q' r
<name>yarn.resourcemanager.reservation-system.class</name>2 W& q' E5 u9 K" {
<value></value> _* Q9 K y1 a0 P& L% H% F
</property>* v6 d: w7 \( s2 p
<property> y; c' T8 V+ G. g: ~3 N Z
<description>
% h9 x6 Y5 ~5 U' ^# t; }& A$ i4 M The plan follower policy class name to use for the ResourceManager& |, h3 I/ E( T6 y' D! J* e
reservation system.
$ v1 ?: i( W- Z6 _/ U4 B4 E4 j1 U By default, is set to
; {+ p0 v$ K, N7 w" d. }, y# e org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower2 x7 c' n; y& h3 A
is used when using CapacityScheduler, and is set to7 {8 ], f9 p) h( V# Y( N" I
org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower( X8 ]2 G3 ]8 z* ~7 V& O
when using FairScheduler.
/ |$ c, u. d9 D& Y0 M7 O' k5 y </description>3 H% g* x6 s, K( i
<name>yarn.resourcemanager.reservation-system.plan.follower</name>
$ m, A; s! @- u9 Z# V. o <value></value>! B- Q/ L) U' p
</property>
/ S( L5 @9 @9 R9 S <property>: j3 ~$ X8 C' q) g \* v# b/ L
<description> K& Z0 N; x- i! H
Step size of the reservation system in ms& `+ R: l% i) S- d
</description>5 Q T# f a0 T( Z$ V+ u) U
<name>yarn.resourcemanager.reservation-system.planfollower.time-step</name>
3 l' |' U7 U; q/ S <value>1000</value>
4 J0 L J; o# B% E, [% g3 [/ N </property>& p) T! C1 n x5 K3 F# Z5 u
<property>
' Z- L. Y7 u u) v <description>
^ c7 s" ]+ s! }4 b9 B The expiry interval for a container, t& a& L2 I* o9 p2 ^
</description>
, D" D q2 F6 e <name>yarn.resourcemanager.rm.container-allocation.expiry-interval-ms</name>8 t9 V# l6 M1 a' o% \! u
<value>600000</value>
" a2 Z6 |1 t. X </property>. c/ c$ C# c9 \6 a4 R
<property>
- n% @4 `" g( t' \ <description>
' [. e! P/ j: g, c9 O" z' F Flag to enable/disable resource profiles ~/ t: K1 k; |# ?1 _, |6 M
</description>
; y2 \8 `1 O" m- {8 y& s <name>yarn.resourcemanager.resource-profiles.enabled</name>
* Y: S# {2 g4 T R9 Q <value>false</value>
( u& X: w8 r$ v </property>0 V7 A, e& Q4 ~# \
<property>: x* ~' P& ^0 R9 ^/ W
<description>5 W- K/ c7 x0 {& o/ x1 J' G0 q$ u
If resource profiles is enabled, source file for the profiles
! [1 i+ n0 l! z' z) D </description>6 t# k2 X8 T9 D) k0 {6 ^7 s" d
<name>yarn.resourcemanager.resource-profiles.source-file</name>" A1 \- p+ Z1 ^9 `- @; \4 y
<value>resource-profiles.json</value>8 s2 T- [1 Z" I3 g2 n
</property>
" C# E/ M+ |! {: B# t" V0 r <!-- Node Manager Configuration -->
# E, P: h% |6 L! p <property>& s7 d. H* L( l. A9 q% h2 | H
<description>The hostname of the NM.</description>
, q" U% k; O/ I% D0 u2 M& S <name>yarn.nodemanager.hostname</name>
2 b9 h9 _' h r- f% h% j <value>0.0.0.0</value>
8 x! o$ @% O9 @8 t% E1 O" o% h </property>, k/ {0 T" d B- H0 X# k. x
<property>
" Q& c2 A" C. O7 r <description>The address of the container manager in the NM.</description>. ^. v8 }0 Y4 P% u( ~
<name>yarn.nodemanager.address</name>
' [2 }3 x3 c5 i4 ]/ `! y! A2 V1 ` <value>${yarn.nodemanager.hostname}:0</value>
& L) H- F/ G+ g& N( s) d. ? </property>- L) X; j; u; S3 L6 t
<property>5 u% _3 o9 t+ P4 N. |5 Z8 h
<description>5 u$ F( J, o O6 ]
The actual address the server will bind to. If this optional address is* ~ g/ W0 B# Q$ l& ?
set, the RPC and webapp servers will bind to this address and the port specified in
4 N& g, q1 m5 d) W1 G% K! w yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is+ t8 e' \% _! Z% l' Z
most useful for making NM listen to all interfaces by setting to 0.0.0.0.
7 i9 Q; I( F- | </description>5 A J1 W- T/ d7 G0 q7 f! {& O0 O
<name>yarn.nodemanager.bind-host</name>" |3 e) y" Y5 d' T
<value></value>6 Q- j6 m3 x. w" a5 M) c) c
</property>
* ]( P0 ~9 L" ~3 X+ K4 M <property>
6 g( S3 ?. M2 h u- z% O2 l <description>Environment variables that should be forwarded from the NodeManager's environment to the container's.</description># E. b! ~9 q# h; d9 c9 A; q
<name>yarn.nodemanager.admin-env</name>
4 Z+ s. u6 j' \+ r6 {$ y$ b <value>MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX</value>7 ^1 a: \1 B5 `) z
</property>" F+ q2 k3 M$ M+ J$ L
<property>
* D" M; E! Z# P( b: l <description>Environment variables that containers may override rather than use NodeManager's default.</description>1 `1 r$ G* O8 d8 W
<name>yarn.nodemanager.env-whitelist</name>
5 @$ G5 P5 s: k) S8 [7 ^ <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>/ \8 j b6 d5 n" m: n$ W
</property>: B1 r0 ]2 K( A
<property># a" e4 E" F, f/ i+ F) g, I! O, z& Z
<description>who will execute(launch) the containers.</description>
5 {2 a0 q9 D9 N8 h+ q G; K3 b <name>yarn.nodemanager.container-executor.class</name>4 R# R7 A: u1 k8 e+ O' [* N
<value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value>
+ }5 q3 t6 C y0 T </property>
6 e. M5 S5 N+ H <property>
; J5 E" \) {" s: Z, e <description>Comma separated List of container state transition listeners.</description>
) Q" v0 \9 y4 m* |2 ?( Y <name>yarn.nodemanager.container-state-transition-listener.classes</name>
" s, Z# e z, x <value></value>$ u. O# W- d+ x% j. R
</property>. ~8 U: d, x0 n* ?7 H
<property>
& n/ P9 K# ^( }. b6 |: n <description>Number of threads container manager uses.</description> ^, a, T0 D) z% U, L7 v" \: E5 l( p' Q
<name>yarn.nodemanager.container-manager.thread-count</name>
: P! r W% K" C/ a/ k4 ? <value>20</value>
3 S5 F r0 Q; h+ z( O2 j* b </property>. @7 c+ K. m* X8 J& i' m0 H U. l
<property>
2 T( k- n& J! x$ a7 v2 Q( v <description>Number of threads collector service uses.</description>
* q% U6 i/ E# M( Z$ w <name>yarn.nodemanager.collector-service.thread-count</name>
5 \! R5 o% y7 G3 x <value>5</value>( x' }2 F; x) N
</property>
8 U3 R! L3 ~5 d3 \: i' H+ V <property>' m, x) W9 [$ f+ u; W" e- u2 [2 Y
<description>Number of threads used in cleanup.</description>+ q7 \8 g; \8 c) d0 ^# r
<name>yarn.nodemanager.delete.thread-count</name>
' q; f) X1 m' y0 X <value>4</value>
9 ^' j& n, s8 J; n2 f </property>/ E% u. _) c9 p8 }
<property>
9 n3 M/ ~: ?6 w. i( n <description>Max number of OPPORTUNISTIC containers to queue at the, y/ @- b0 l( `$ @$ I+ H- R1 ~2 ]# d
nodemanager.</description>
% N9 u. ^) ?' @5 y: F <name>yarn.nodemanager.opportunistic-containers-max-queue-length</name>* r# Y/ j1 x m! z" D5 j. o
<value>0</value>; z( \( @+ C+ l4 V% l" [5 `5 `; M
</property>. w/ U: i6 ^5 X3 Y. y7 ]
<property># N& h) r- p) m8 O: F
<description>' T1 Q) @: c+ X, ]8 o$ Z
Number of seconds after an application finishes before the nodemanager's
! \9 }: C, ~3 y" ^. \ DeletionService will delete the application's localized file directory: |* G0 ]+ r/ q* [4 z3 a' N& ?
and log directory.
0 g l1 Q% m# w8 g+ y To diagnose YARN application problems, set this property's value large
, X1 V% b8 M9 H4 z enough (for example, to 600 = 10 minutes) to permit examination of these
& z" \' o% M1 O9 `+ J' i directories. After changing the property's value, you must restart the
) W% T4 w6 |5 L7 g- ~2 ]3 J nodemanager in order for it to have an effect.
7 i; L2 [& F, z The roots of YARN applications' work directories is configurable with
8 g0 M% N6 _$ F6 ~1 U; I9 F the yarn.nodemanager.local-dirs property (see below), and the roots: v- G& i2 J$ U+ \- b8 ^; C
of the YARN applications' log directories is configurable with the$ o8 G* o6 t) I2 _% g0 F+ V) ]
yarn.nodemanager.log-dirs property (see also below).( b9 ]/ x# _* d/ v( q: Y4 q4 S* t
</description>
8 t8 }' F5 d3 m! Q- ~9 q$ ]' O& s1 i <name>yarn.nodemanager.delete.debug-delay-sec</name>) o5 F, H, H2 E4 k+ j# J
<value>0</value>. N: Q0 m" n r
</property>
' i$ [* J# L9 V! L; H2 c! q3 L. L6 h/ P <property>
. o+ J, @& d# D. I+ P, i+ a <description>Keytab for NM.</description>4 C2 l/ _, ]5 v% \5 t' f, L
<name>yarn.nodemanager.keytab</name>
% x5 F/ C4 H F/ }6 z <value>/etc/krb5.keytab</value>
0 a3 ^2 l1 i3 n </property>
4 Q" k4 B. [1 i7 J- I2 ~8 o <property>
* b/ t0 c- |; E) n <description>List of directories to store localized files in. An : L6 {+ X. g5 l1 w0 I
application's localized file directory will be found in:0 ?( A% I- i0 u9 ?$ R4 [5 B
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.. ?* c$ F8 C0 a. {; ?
Individual containers' work directories, called container_${contid}, will
5 o; {$ n0 p0 U$ C& B5 z! k1 C be subdirectories of this.0 W; `) G( q% F; }8 ^
</description>
9 k6 A: C- K" s( B. g0 N <name>yarn.nodemanager.local-dirs</name>+ R6 a' |7 M0 g1 u& ^
<value>${hadoop.tmp.dir}/nm-local-dir</value>
- n! a4 L8 s0 d' ~3 ` </property>/ O) d2 c2 A+ @
<property>
2 w; l M' [& v0 m( j* \ <description>It limits the maximum number of files which will be localized0 t* R4 W2 M0 m/ D' C
in a single local directory. If the limit is reached then sub-directories
. s7 v2 ]2 t/ F0 |3 K% T$ N# | will be created and new files will be localized in them. If it is set to
2 s) j& e' W. Y: N a value less than or equal to 36 [which are sub-directories (0-9 and then
% F9 |4 s# f# L! O, u& ` a-z)] then NodeManager will fail to start. For example; [for public c0 Y5 K2 _# Z3 S8 X
cache] if this is configured with a value of 40 ( 4 files ++ Z" N& o. r, }
36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will
6 G8 b$ q" ~% h9 M, S allow 4 files to be created directly inside "/tmp/local-dir1/filecache".& k+ I6 J+ I; X% y) _8 X
For files that are localized further it will create a sub-directory "0"8 _7 {% R# S F. L* W( v
inside "/tmp/local-dir1/filecache" and will localize files inside it
; p- ~$ A0 G3 H3 X3 B until it becomes full. If a file is removed from a sub-directory that
/ y/ V% U; F' i, t5 G$ D is marked full, then that sub-directory will be used back again to; I/ Y) F; m3 `6 w
localize files.
( n; V/ W) X& J& B+ s% I </description>
$ b- s8 d8 o$ q; T <name>yarn.nodemanager.local-cache.max-files-per-directory</name>3 f3 R$ R. f- }- f4 i# D d- s
<value>8192</value>7 w; w# A5 T( e7 k4 }# t
</property>
1 v& T( E1 ^$ ?% s7 ?% W9 s <property>/ P2 L: t4 v6 s. ` E9 `! D6 N
<description>Address where the localizer IPC is.</description>
( F- n$ A) T M; P1 H <name>yarn.nodemanager.localizer.address</name>0 `+ m* A" D$ L
<value>${yarn.nodemanager.hostname}:8040</value>; a( L& L* T9 z; ]4 C" |7 M' G+ x
</property>
: A6 j* w+ Q7 I, F <property>; G/ ]% b3 d( e2 Z* f
<description>Address where the collector service IPC is.</description>0 h' ^# V _" r- B% S3 Z8 K) f1 H
<name>yarn.nodemanager.collector-service.address</name>
& u( K. J/ ~, |0 b* L: d! Q <value>${yarn.nodemanager.hostname}:8048</value>
; [, Y! Q( A& X) r+ g </property>6 E+ l( L' y: ~, b7 s7 g
<property>
5 O3 T/ s$ @+ x) U& u# {/ o+ O& I <description>Interval in between cache cleanups.</description>
" H$ K4 H) h/ H/ j* J <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>- W5 ^$ `/ a: u+ n
<value>600000</value>, i' b+ g& M! W. N3 H- r; k
</property>& {: k- Q! a2 ]
<property>
) N: u( U. \8 b" z7 `% ? <description>Target size of localizer cache in MB, per nodemanager. It is
+ l; q) D1 Z4 F a target retention size that only includes resources with PUBLIC and
5 F5 h% n& b j9 { PRIVATE visibility and excludes resources with APPLICATION visibility2 P. u1 r0 k$ _1 ~; Q0 ^+ r) I
</description>
# N( G- M4 Q1 z# y <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
' F/ ?8 d5 s7 m! @* V8 M: B& Q <value>10240</value>
) h3 O3 U$ i8 S6 c </property>+ i! y/ v/ G+ P
<property>! o$ _3 u' i8 k8 i. q( C
<description>Number of threads to handle localization requests.</description>+ p1 k) p# j( a6 {+ }# S
<name>yarn.nodemanager.localizer.client.thread-count</name>' i0 y1 z* K4 B5 F( r! {6 I
<value>5</value>
t9 [$ I) t7 Q5 X# P$ x </property>
" j. \7 j7 b) d4 ~( j <property>
+ l7 B' x9 m& @' y+ {# p" S1 c) g <description>Number of threads to use for localization fetching.</description>
. y" v! I' Y& v- H. m" [ <name>yarn.nodemanager.localizer.fetch.thread-count</name> A, @$ n5 A# a5 s. ~8 k
<value>4</value>5 I; r# t* k4 W) b# ?
</property>
/ y5 \+ _. F! U <property>
* }6 I C B) [- b' Q <description>/ t9 b+ H0 ^9 I' n6 { t3 h
</description>4 G' d8 K6 a0 a
<name>yarn.nodemanager.container-localizer.java.opts</name>- A- ?: y' w7 A
<value>-Xmx256m</value>
6 P7 p7 }+ @! g2 P9 a' U: S </property>7 V1 W# q, R( b8 r
<property>
: W9 x9 w, L+ Q7 U <description>
4 A9 m/ w( s5 u' R! O* N" F& J; H The log level for container localizer while it is an independent process.
( r$ h x o: `) V2 P( G </description>
' Y7 J( B/ Q# H- @ <name>yarn.nodemanager.container-localizer.log.level</name>' H3 P4 a _/ F1 I
<value>INFO</value>
, v2 {- @+ }/ U# p </property>
% M! N9 F4 m7 D r& v0 ^ <property> n: B0 ]4 w! q; P: |
<description>
" ] x. @% C5 q4 m, [) e& J0 v Where to store container logs. An application's localized log directory. K) o2 G" L5 w3 L) I$ r
will be found in ${yarn.nodemanager.log-dirs}/application_${appid}.
- w Z( W& u) }+ U' w x Individual containers' log directories will be below this, in directories
. F3 L# l% m1 J1 S. P s named container_{$contid}. Each container directory will contain the files
8 ^7 n& a2 z) |6 G( u1 m" Q1 W stderr, stdin, and syslog generated by that container.* R; k- e7 u/ g9 ?. K% I5 |) k% |
</description>; i9 u$ E2 r' M, [3 e
<name>yarn.nodemanager.log-dirs</name>
+ f" b6 H4 Z+ S$ B! ` <value>${yarn.log.dir}/userlogs</value>; @5 Y* W" ` S. K
</property>) W9 S) u1 D. A- ]0 o6 J/ ]$ F
<property>
: P- G( G' m# ~; s5 R9 y: r <description>
) \& H, M t( Z, U% k B- v The permissions settings used for the creation of container
7 T9 E8 N9 c$ y9 b1 _ directories when using DefaultContainerExecutor. This follows6 P0 G1 F1 Z7 Q$ Y, ?+ U% \1 a
standard user/group/all permissions format. \! h; L" r2 ?! J; T& m5 K* A
</description>- U; k6 q: [: Q8 ^7 U. l
<name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>
) p3 |& ~/ {* M# ^( g <value>710</value>5 C$ w# ^# d$ H% k
</property>
$ V& ?6 I3 {: H1 z( [$ @ <property>$ R3 L( v" s9 h6 z9 z8 \
<description>Whether to enable log aggregation. Log aggregation collects
$ f; V, w: X4 `5 Z l each container's logs and moves these logs onto a file-system, for e.g.
6 @$ L/ g Z* v& p HDFS, after the application completes. Users can configure the; S* s: ~- u! _) r
"yarn.nodemanager.remote-app-log-dir" and& f- U: Y5 M8 q# g2 F0 b
"yarn.nodemanager.remote-app-log-dir-suffix" properties to determine
4 ]* K; ]3 n+ K6 l; T* P where these logs are moved to. Users can access the logs via the
2 i# h# B: D# K' B( g0 R" W4 b! ? Application Timeline Server.
2 i( g' |/ R* J7 L </description>* n. }* K- H6 }9 Q$ o* L
<name>yarn.log-aggregation-enable</name>
0 D# C5 H3 M {0 K0 \: a <value>false</value>7 F1 ^' Q5 E0 a9 ?" n% |
</property>2 X+ K% ~8 w3 V% P1 }6 C- e
<property>
) b5 j/ }) p% V3 d; u% \: L: J <description>How long to keep aggregation logs before deleting them. -1 disables. 4 I( a. j+ e4 o4 }
Be careful set this too small and you will spam the name node.</description>
! m4 I7 E' i$ M3 U; G/ @3 [ <name>yarn.log-aggregation.retain-seconds</name>
7 e8 I# I5 B7 h% r <value>-1</value>& x# M# q/ [ g6 ^/ I4 y
</property> - P3 S" d- i: z$ @0 J0 c
<property>7 d: ^: o+ q8 i) p( B
<description>How long to wait between aggregated log retention checks.$ ^& _' Y2 V1 D. F' W! t& c
If set to 0 or a negative value then the value is computed as one-tenth
& I9 ^6 r8 Z5 b) q U a) u5 Z+ E of the aggregated log retention time. Be careful set this too small and
. c, b* \3 g" d- U d7 R! `5 G/ T; X, G you will spam the name node.</description>, z! N& H! M* K: }/ T
<name>yarn.log-aggregation.retain-check-interval-seconds</name>7 ^# o1 I, N* d' R/ \ g4 a0 k% p
<value>-1</value># Q: y3 n0 r2 P/ Y" k# J4 y" ]5 Z
</property>+ I' G8 [, m) f, D1 f$ ?
<property>. d- t. R1 v' j3 f* j
<description>Specify which log file controllers we will support. The first4 V" |' Y1 |) n4 N. `
file controller we add will be used to write the aggregated logs.
; O7 a5 a# A$ W' k2 h% A This comma separated configuration will work with the configuration:& e c. z q9 ?
yarn.log-aggregation.file-controller.%s.class which defines the supported
* ]2 ^8 S, E+ P file controller's class. By default, the TFile controller would be used.
7 I4 o7 m$ }' \0 E1 A" e- B The user could override this configuration by adding more file controllers.
. @. |9 j2 ]& N; F4 X To support back-ward compatibility, make sure that we always
, z# [% p' {7 M) |& h3 D add TFile file controller.</description>
1 g$ \* Z/ W/ ~/ z, P8 p <name>yarn.log-aggregation.file-formats</name>
: b# g( R7 C, ] <value>TFile</value>+ \" K/ [6 x' ?/ k
</property>
$ B @- v& Y8 U: a <property># g0 [( U' Y: |% k6 S3 L
<description>Class that supports TFile read and write operations.</description>
# j9 w. J9 S- Z3 z/ K$ _4 O$ S <name>yarn.log-aggregation.file-controller.TFile.class</name>6 s! `6 u* _9 e' t) \
<value>org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController</value>9 E5 i/ S. w* P$ f: D; ?" M7 Y
</property>; j. d; a: @, ?$ y3 x9 l
<property>
; W% w$ V& O; g4 `; M) b <description>' T" T1 u7 p6 a0 Z9 ?& y7 \
How long for ResourceManager to wait for NodeManager to report its# R) A' c! K" ^* g; w/ A
log aggregation status. If waiting time of which the log aggregation+ d* M: K' j8 A( u z7 S/ n
status is reported from NodeManager exceeds the configured value, RM, |# R6 [5 L& O% i8 m- [
will report log aggregation status for this NodeManager as TIME_OUT.
+ @: U: P* z1 j e. |9 g' q9 y This configuration will be used in NodeManager as well to decide
1 D8 y+ E8 p: l2 v8 r6 s6 B2 i whether and when to delete the cached log aggregation status.
" B+ Q6 L( c0 o& U8 c </description>
: u% J8 H1 \) H: J% q- m <name>yarn.log-aggregation-status.time-out.ms</name>
* G& @3 K4 {8 y# i) | <value>600000</value>$ k2 z6 u1 Q$ ~4 J3 C3 E+ P
</property>
r, [ A5 Y* J ~ <property>
4 V$ o4 p( e4 p, ?6 m' R! _ <description>Time in seconds to retain user logs. Only applicable if
4 y# _3 n1 z' R! e" W6 ^$ I. M log aggregation is disabled5 s S8 H+ u+ [: W @% A
</description>
: s. x1 z2 M! Q4 ?0 Z' [/ E <name>yarn.nodemanager.log.retain-seconds</name>
& }- {" T* g& X. j/ @ <value>10800</value>
+ A+ {! | C3 W </property>
# b' z9 ?7 P% W: Q <property>
8 G) K- B0 Z2 G% {5 ]2 X <description>Where to aggregate logs to.</description>8 [. _& a! @: |1 `1 c5 {
<name>yarn.nodemanager.remote-app-log-dir</name>
: m5 W4 r2 ^# l, O <value>/tmp/logs</value>
! y0 T" Y4 k+ q# F </property>* U' ?+ ~) Q7 ]: `" ]
<property>. ?1 V& v, z- r/ e" N
<description>The remote log dir will be created at
. v( [9 e$ R0 H$ I {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
! x& d8 Z1 G9 ]2 N/ a </description>2 q* I( r9 v% i0 B
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
* B* Q# O/ F: p3 ?" D <value>logs</value>
. {( m# s4 p) P( L. D1 Y </property>
" t' M5 |' p5 r; c8 u& E9 |/ H <property>% p2 j, e8 e4 @ U; }
<description>Generate additional logs about container launches.
0 p' p, t% @" G7 f2 m4 T8 c Currently, this creates a copy of the launch script and lists the4 }; x5 {4 r0 j, U0 b
directory contents of the container work dir. When listing directory* O6 v$ V& W2 u9 U' T
contents, we follow symlinks to a max-depth of 5(including symlinks
- q- n6 O, {2 H3 L C8 d which point to outside the container work dir) which may lead to a
. @( Y' D7 T g+ p slowness in launching containers.
8 E' K* l9 w; b5 v4 J+ a </description>
6 _' c) A6 H/ g% q9 [2 o# S9 S& {- d <name>yarn.nodemanager.log-container-debug-info.enabled</name>
0 F* L! ^3 s6 ` g <value>true</value>- q! r; I" r" Y$ }
</property> r! S; V. O0 c. ?; f6 e
<property>
2 t" I8 T/ \5 R5 K$ j4 G6 j <description>Amount of physical memory, in MB, that can be allocated , V9 ~4 J; I) F$ b' o& I
for containers. If set to -1 and z7 {! p9 q0 `2 \0 b% i0 h
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
! D) z7 e5 r$ {/ j E( v2 X6 y' l) M automatically calculated(in case of Windows and Linux).: X% V. K# F$ n9 H4 x4 z5 V9 P/ M
In other cases, the default is 8192MB.% V$ f( q+ e) }) U
</description>
3 S0 n B: R4 D# }/ W <name>yarn.nodemanager.resource.memory-mb</name>; d6 [. F8 O$ ~9 h
<value>-1</value>
1 }) V" L3 Q, v9 X9 e. R </property>
) c) H+ Z% x D" c- W! R <property>5 a. `8 g% K$ X* i; `6 `7 m
<description>Amount of physical memory, in MB, that is reserved# [2 a. e% L; x0 L$ \$ C
for non-YARN processes. This configuration is only used if
0 `- p# S: G# C5 A& c) Q yarn.nodemanager.resource.detect-hardware-capabilities is set
, q5 B5 h( e. ]4 Y6 Z, J2 x- d- x: L to true and yarn.nodemanager.resource.memory-mb is -1. If set
* M3 E9 v) I& p% ` to -1, this amount is calculated as7 K2 |3 C# @7 f$ J
20% of (system memory - 2*HADOOP_HEAPSIZE)8 X( G0 g0 ]" n
</description>
" r! I+ M% z! I7 h& m <name>yarn.nodemanager.resource.system-reserved-memory-mb</name>2 ^* P1 A$ \8 W) \% L0 v+ h, s
<value>-1</value>, [6 d, E, s$ }) W& N
</property>9 d; D8 x8 C( A. L6 v# P
<property>8 x7 A% S1 d5 |0 T: n3 ?
<description>Whether YARN CGroups memory tracking is enabled.</description>
8 E; h) o; A( V1 W% [$ C. s% D8 o <name>yarn.nodemanager.resource.memory.enabled</name>- P5 Q0 c5 T: R. M- n- N; m
<value>false</value> p$ G# d4 W6 ^" S, w. ]
</property>( c/ Y/ z+ h1 y/ K+ S9 d# R
<property>
; O H4 ]6 Q% q0 [$ {- R <description>Whether YARN CGroups strict memory enforcement is enabled.
5 Q* j& C# S% M5 _! n U3 b- c2 O0 X </description>5 P$ x, s3 m7 B5 p3 F, X
<name>yarn.nodemanager.resource.memory.enforced</name>& w, N; q' x# r2 \8 E
<value>true</value>7 N2 t/ b0 e$ h
</property>
5 q, C3 L9 N* b7 T <property>
3 d* T; z% _$ @+ w) u <description>If memory limit is enforced, this the percentage of soft limit1 h5 R* k# n, E1 f& X3 u
compared to the memory assigned to the container. If there is memory
& U; F$ P# l- ^5 u pressure container memory usage will be pushed back to its soft limit! N @. Q t* o& P
by swapping out memory.
% s" s% I7 g+ G% |; K </description>- M+ Z# V% P+ ?
<name>yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage</name>$ f" v+ A5 B1 U- s Z" i; A! x) r
<value>90.0</value>
( `; h O- q% |; O) A </property>
7 \/ l( k, l3 j( `& t' L <property>3 [/ G" j+ n x
<description>Container swappiness is the likelihood a page will be swapped. |+ e6 p8 p/ H B! F3 W! n) m
out compared to be kept in memory. Value is between 0-100.
: Q( ? Z7 Y7 o W8 i- x: h+ w </description>
6 m( y2 h( m/ u- o+ ]" m <name>yarn.nodemanager.resource.memory.cgroups.swappiness</name>, L: g7 \6 @9 e* p P2 [
<value>0</value>. X6 [, v3 S4 q8 u r* O
</property>
. L. K/ B4 b) ~ <property>
) p# c- [% q x% p3 m <description>Whether physical memory limits will be enforced for
( v2 {% i0 r. L containers.</description>
0 ~* X, ~% c+ B2 F2 \+ r( ?4 E <name>yarn.nodemanager.pmem-check-enabled</name>
0 @5 Z6 r6 j$ \( p1 ~: v2 c2 z <value>true</value>
7 l. Q" _0 I- K" C7 c& x! { </property>- X0 w8 o, i+ o7 B
<property>
# Q1 P# h' Z) M( q8 f' F3 V <description>Whether virtual memory limits will be enforced for
. Y# t- u" I: v" V* L containers.</description>
H3 u6 L( h& a- r) O <name>yarn.nodemanager.vmem-check-enabled</name>9 u, [$ z' z% Y0 q+ Q+ K
<value>true</value>) e1 p0 |- y# w x" ^
</property>
4 k R9 h( c5 }# d* W; c% R <property>1 q3 X' F4 a/ E) n0 B
<description>Ratio between virtual memory to physical memory when
# s4 Y1 c7 p" v2 ~6 _7 l) h! d setting memory limits for containers. Container allocations are* M" \( l' @: A) G7 y$ W
expressed in terms of physical memory, and virtual memory usage$ O; p+ L1 @# ]8 C. d
is allowed to exceed this allocation by this ratio.6 H- Q) d4 o& ] V( B: s/ I4 o
</description>& @, ?) A- {4 @! H1 e) H
<name>yarn.nodemanager.vmem-pmem-ratio</name>7 q; Y; m* I' E8 }0 c4 w9 M* L
<value>2.1</value> ^& Z, g) |2 y4 V: [. E
</property>3 D1 C+ f5 x1 Y |, z+ u
<property>
3 X7 D* x5 q9 }" P1 C <description>Number of vcores that can be allocated, c& `0 T7 E' q6 \/ h0 s; {) b
for containers. This is used by the RM scheduler when allocating
& c+ m4 V- v. f! U8 x5 ~ resources for containers. This is not used to limit the number of
' j* ?- V6 V4 X {$ Y0 ?7 S+ ?2 M CPUs used by YARN containers. If it is set to -1 and" b6 G1 P% u j d. ]! X# W
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is6 }& q5 t1 P9 c) Z5 ?2 e
automatically determined from the hardware in case of Windows and Linux. B! k5 {6 w2 f5 Z
In other cases, number of vcores is 8 by default.</description>) D6 c4 D; Y" `2 ~; V! e% r& T( m- `: s
<name>yarn.nodemanager.resource.cpu-vcores</name>
; b7 R/ C. v4 e0 t4 m% w <value>-1</value>
. \ ^9 c. m L </property>
( ?# c; c' }& {4 N) [ <property>
/ u1 i Y( ?* A. q; O6 G; s$ B! j <description>Flag to determine if logical processors(such as
: o* L: i- q3 l/ B hyperthreads) should be counted as cores. Only applicable on Linux, b( q& K% `( O+ }
when yarn.nodemanager.resource.cpu-vcores is set to -1 and
+ j, N& k0 e' b, ? yarn.nodemanager.resource.detect-hardware-capabilities is true.2 F1 I- z! v1 s
</description>
; }, Z! ]+ d( o6 ?3 r4 d <name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>
/ J* y. s0 l9 x6 V0 h3 d: ]) U <value>false</value>% V" K1 k, E) Y3 e W
</property>- j. Z; e" z9 [" O' F* m9 A
<property>
7 t6 @) F" w$ r3 I <description>Multiplier to determine how to convert phyiscal cores to8 t# g; F; o+ m- ^
vcores. This value is used if yarn.nodemanager.resource.cpu-vcores
* n( e9 C7 P" Q8 ^ is set to -1(which implies auto-calculate vcores) and
4 u4 P7 i* s$ w7 F; e Q yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The) ~+ d: K% v! {' ~$ q/ e' A
number of vcores will be calculated as
/ w( v1 a; H4 J' H7 y4 O number of CPUs * multiplier.7 @& p+ P; [3 @! t% S0 u- E
</description>
4 B( x5 j- \6 g: Y9 h# e <name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>: ^' n, k3 i6 s5 Q4 E% D/ a7 X
<value>1.0</value>7 a4 d$ l, ], j; E% n; v
</property>
* q; |) F! [% n( P; t c: u <property>0 ?) }5 m$ ]4 X6 c, n% f0 W) c
<description>
- v' S; y0 _" G* [4 @3 y9 w Thread pool size for LogAggregationService in Node Manager.. }. i$ B; A" k' R
</description>5 _4 E( u- X+ `5 @
<name>yarn.nodemanager.logaggregation.threadpool-size-max</name>. a* k( L7 X/ H1 ]5 s/ y5 o
<value>100</value>
$ C, p" U; v3 e; I" _" f$ u </property>
) Q/ O$ `" ~6 k& z S' k <property>
1 J! L+ u# ^6 M6 }' f <description>Percentage of CPU that can be allocated, X" ?4 z* m3 ~
for containers. This setting allows users to limit the amount of2 }) {) F& a( A0 h9 R$ j
CPU that YARN containers use. Currently functional only3 A( x" @0 {- E; k; H
on Linux using cgroups. The default is to use 100% of CPU.
( u2 V1 U$ l9 D+ h+ z! m2 y </description>
: [' N6 E, `7 c/ I8 T7 L1 V' ] <name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>
; j- F, g3 A/ Q: h; c( P/ N( I <value>100</value>1 u: H( Z$ }9 i, Z, o4 k+ M8 R
</property>( x" ^; l$ P. n8 X! \0 y9 y. E
<property>2 I; O! y3 k+ M1 m2 R3 E1 I
<description>Enable auto-detection of node capabilities such as
7 E) ] k. F3 A% t0 Y8 x J b memory and CPU.8 L% k; a) L4 r& i' u' x
</description>
5 x) x2 o# v7 O2 @+ R% ~ <name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
' y8 {/ k! F3 r/ V: \; h, R <value>false</value>
2 n+ s8 A+ F+ k& J </property>
/ C# g2 `& m% e' |6 [' \9 e8 _2 O <property>
3 C' A( r' o- C! a! z G9 H <description>NM Webapp address.</description>6 n" ]& H8 c3 i* ~
<name>yarn.nodemanager.webapp.address</name>: K4 j4 w0 r/ e
<value>${yarn.nodemanager.hostname}:8042</value>) @7 T; b+ T1 c( H- j) d" a; |* R
</property>
- C/ w% O- f# Y <property>
' _ s7 q! |. b <description>
& ?6 }* {3 B2 T. K9 I The https adddress of the NM web application.
3 v+ P9 w- A7 l </description>8 Y# S. A+ y2 x
<name>yarn.nodemanager.webapp.https.address</name>9 O4 q. ]5 ]: c4 o$ x* a% K
<value>0.0.0.0:8044</value>
7 s% p+ `! a* Q$ ^- \4 ? </property>
; |! \0 k' r. v. p3 C6 _ <property>
" u3 L3 U( y/ O4 d+ Y <description>
; h0 z9 g3 r8 v$ [ d6 ` The Kerberos keytab file to be used for spnego filter for the NM web* i7 a+ O5 L. I
interface.) J: D; J' Q$ c# {( C
</description>* V; F! q7 } W+ D, W* L3 H
<name>yarn.nodemanager.webapp.spnego-keytab-file</name># m# o+ L. k* H# W3 [
<value></value>
9 U9 E/ I9 R0 F/ V* j% P& B </property>
( |& Y! N9 A9 `6 s! V w o <property>1 W+ z5 n3 M. |6 }. w8 v
<description>0 D, U/ ` ]2 D0 ?( t
The Kerberos principal to be used for spnego filter for the NM web
- ?1 }; y4 f0 m interface.
6 p5 E7 c1 k- B </description>
: b* H$ a K4 q& X/ B <name>yarn.nodemanager.webapp.spnego-principal</name>
5 a& F" [& N: B4 l <value></value>
N, b5 q! N6 e+ m3 ? `% C </property>
0 C ?1 A# L, \- F9 V0 X. P <property>
9 m# ]- w5 D9 I, \* Y$ V3 R i <description>How often to monitor the node and the containers.
2 I( Q, H t9 L/ B If 0 or negative, monitoring is disabled.</description> J# d5 }" Y7 D0 }3 @* m) Z1 p* O6 @
<name>yarn.nodemanager.resource-monitor.interval-ms</name>
5 J6 v x6 x: K9 A, b, p <value>3000</value>
3 ^& ^. C* t, g/ O8 W1 z* T </property>
5 Q+ L7 s2 S3 Z$ L* h' x( l& }5 J <property>7 L$ h1 S V9 \- C0 ~2 M0 ]5 U6 M( p
<description>Class that calculates current resource utilization.</description>0 }* ~" a( S7 R% U% O! c. t3 B" }
<name>yarn.nodemanager.resource-calculator.class</name>& V- J( i" ]1 ^/ K- C7 X
</property>
2 h& [- {: L) R! W: u5 h: r+ | <property>
! d% p9 Z& `* e: L& E }% y( c- }0 e <description>Enable container monitor</description>
( m# d i1 r3 R- _8 ~& T; E <name>yarn.nodemanager.container-monitor.enabled</name>! R& N% Q& ] C
<value>true</value># g0 `1 H3 l( Y5 p
</property>5 q+ ~2 ]* D% B4 y$ s. |, j) N% k
<property>+ Q" m) m% l( e5 Y
<description>How often to monitor containers. If not set, the value for7 o% n8 y6 L! e {* _4 y
yarn.nodemanager.resource-monitor.interval-ms will be used.
9 z, G$ U) J. M$ b If 0 or negative, container monitoring is disabled.</description>: n: |& x$ z, }
<name>yarn.nodemanager.container-monitor.interval-ms</name>
& G+ }- @9 ^5 e$ F7 c4 Y </property>& r" c: q3 Q! n
<property>0 r( Q( y- S: e3 d- o
<description>Class that calculates containers current resource utilization.; R5 _5 p9 l/ k
If not set, the value for yarn.nodemanager.resource-calculator.class will" H! s: m6 V; K( Z
be used.</description>
* a0 ^3 q1 M4 v [* e <name>yarn.nodemanager.container-monitor.resource-calculator.class</name>
& J" U1 \, _8 k </property>
4 {3 ]9 k3 h$ s1 L <property>
7 C; g B& c! |7 S8 D3 i) S <description>Frequency of running node health script.</description>( L) a/ A* J2 t- H
<name>yarn.nodemanager.health-checker.interval-ms</name>
% J o8 B/ l0 f) ^, L <value>600000</value>" r& F/ Q! j) P, H- n4 r" a7 V
</property> {% ^5 D: ^3 u8 t3 @% O& x, v2 z
<property>
, c; ?) z+ F4 G+ J% F) ?$ ^- h) G <description>Script time out period.</description>7 }+ O; V8 I% A$ H
<name>yarn.nodemanager.health-checker.script.timeout-ms</name>; u+ ~0 v# L/ Q ?
<value>1200000</value>
7 O' g' C {3 b$ L* ^2 E' W </property>$ m/ J, B. a! O+ t. w% ?3 h0 s/ \" A
<property>2 P8 \0 X0 m5 m8 ^* [' n
<description>The health check script to run.</description>
' U1 Q) q4 }) R/ ?* V <name>yarn.nodemanager.health-checker.script.path</name>. c: X1 A0 n2 z" P( s7 r9 R0 C" g
<value></value>
! `7 M! e5 u8 z; D* I- h; r, o </property>( D$ V4 X. c' h' `% T* f+ \
<property>; y9 |5 ]( I$ c+ u0 Q( X% j
<description>The arguments to pass to the health check script.</description>
+ y! w/ l; f& Q$ a! O <name>yarn.nodemanager.health-checker.script.opts</name>2 d V, c4 C1 e4 f* v% ?
<value></value>
1 f. j' E6 ?! ~4 F3 S </property>
# d" u4 a3 d; y( ^) d' v5 H7 T5 p <property>, x5 u/ _0 h& I; ?! ]
<description>Frequency of running disk health checker code.</description>) I, N* t- C" X4 p: @ ^' s
<name>yarn.nodemanager.disk-health-checker.interval-ms</name>
2 L7 d" W' p) J2 @ <value>120000</value>7 K3 ]2 r: q8 F$ Y7 ]8 I; Q# R4 V
</property>7 v& A8 z g5 ~% o1 B+ C+ L
<property>1 t' p2 c' ~6 g2 {( E4 w
<description>The minimum fraction of number of disks to be healthy for the0 B! I# |( m0 a8 z& K3 k9 A p! x
nodemanager to launch new containers. This correspond to both9 @/ C2 p/ p; `" |* M$ @
yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there p! r7 _6 J, x& _
are less number of healthy local-dirs (or log-dirs) available, then
6 }$ F- c2 J4 \/ [$ i new containers will not be launched on this node.</description>
! V( M5 f2 S L9 q1 Y% a: U <name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>7 h5 ?6 s4 u- c, [" s* f" z% g
<value>0.25</value>
; Z8 p; h9 ?* y/ ~" [) q& U </property>
" d- Y* c3 o, k$ a$ M f <property>5 U0 G! |* J" f. v) p$ M! W
<description>The maximum percentage of disk space utilization allowed after / V w: m9 y" }/ L! F9 X5 F
which a disk is marked as bad. Values can range from 0.0 to 100.0. " {3 V; ~5 O; s9 z) Y* F4 K
If the value is greater than or equal to 100, the nodemanager will check
8 Q- v9 u3 y3 ^- J, A f for full disk. This applies to yarn.nodemanager.local-dirs and
* @+ d# N/ Y/ K, w yarn.nodemanager.log-dirs.</description>/ n' k1 ~- {( m6 d8 |. h0 }
<name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
+ c+ f; V* j3 k& l! S$ L <value>90.0</value>+ j% A6 T! z: I* ]
</property>. W' u. \4 e- n2 g
<property>' k- y2 O# I) [
<description>The low threshold percentage of disk space used when a bad disk is
8 _* r. f. C' h5 z6 W! v- k/ U; c marked as good. Values can range from 0.0 to 100.0. This applies to; i" L0 R% a( V- y/ O
yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.
# ]1 {5 s2 p* F7 a/ u# w Note that if its value is more than yarn.nodemanager.disk-health-checker.
4 d% u) P0 g% y max-disk-utilization-per-disk-percentage or not set, it will be set to the same value as
6 K# m& k# Z/ C y( ?. h1 } yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage.</description>
6 s8 @; ^7 n+ Z <name>yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage</name>9 B; V+ t) [, U+ M" Q& u+ h
<value></value>
4 r* _1 S+ U9 [& N </property>
& J6 @" v: R% a! C+ d/ i: Q <property>* a2 k, f' J& B% n
<description>The minimum space that must be available on a disk for
+ q, s; s+ [# Y: Y3 j it to be used. This applies to yarn.nodemanager.local-dirs and* L4 R, R/ w: D/ [% M
yarn.nodemanager.log-dirs.</description>
9 K! q* ]/ t" n/ Z& A <name>yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb</name>
6 X4 c* n1 m4 h. p7 O) |- r( c4 P <value>0</value>
% m# Q. g3 K! i) w) _& F" e* s. P </property>
' k* [8 D7 I( R2 M* w! s) c <property>% x7 j: P# Q3 z( c' r. d- }( I
<description>The path to the Linux container executor.</description>% p- {) E9 {3 r: N
<name>yarn.nodemanager.linux-container-executor.path</name>! h% E- x8 n- `" a6 [$ b$ E
</property>
" B% e$ H. A. K. l0 n. [1 N <property>' {* e O0 F; M3 |4 i2 j
<description>The class which should help the LCE handle resources.</description>
2 w& B8 G; N3 K- n, A <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
3 c! S/ N. Z6 ^ <value>org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler</value>
. O3 q( Z7 w7 U% d <!-- <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value> -->1 O- p0 S, u: \& w
</property>
2 V! g7 V1 Y* B5 f3 } <property>: S- z Z' H! j+ F# c
<description>The cgroups hierarchy under which to place YARN proccesses (cannot contain commas)., p! k7 T4 q9 b3 z& z2 T
If yarn.nodemanager.linux-container-executor.cgroups.mount is false/ d2 o# k, ]$ j( j8 K( ^4 A
(that is, if cgroups have been pre-configured) and the YARN user has write$ P8 ]: ?0 w; d2 ]1 {; s/ b
access to the parent directory, then the directory will be created.
3 |6 H, A, a. z6 ]# u If the directory already exists, the administrator has to give YARN
$ g- i9 M( ?8 w write permissions to it recursively.
6 q" B, A; R& m B; J0 X This property only applies when the LCE resources handler is set to
& L- \9 E, p& ]3 V9 [7 | @$ e CgroupsLCEResourcesHandler.</description>% l* k& `6 `2 D' u, p
<name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>* ~- V; F( t2 n1 O/ E0 {% K
<value>/hadoop-yarn</value>4 m X4 [) n! o& u+ G
</property>
* ^# g3 N( }4 Y# A. S$ m3 P6 x <property>) d( N8 P3 E: n, O+ d
<description>Whether the LCE should attempt to mount cgroups if not found.& K/ z+ m) ? o( ^, k
This property only applies when the LCE resources handler is set to
+ \- K, T! V" V9 H5 ]; P+ n0 {, g# b CgroupsLCEResourcesHandler.% c) C+ z' K# v! \: Z U) x- h
</description>
3 R7 S& b: x. e <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>6 X+ K" j( r1 }
<value>false</value>4 y0 m( g3 d% b- s4 V" ]
</property>/ J2 r; P, o4 I' z/ p* k" T5 W
<property>
/ x: l, o( i z' P; M <description>This property sets the path from which YARN will read the$ H3 b( B- A- H$ o. m8 `+ W
CGroups configuration. YARN has built-in functionality to discover the- E" y- Y$ }) C* |
system CGroup mount paths, so use this property only if YARN's automatic
) `$ O2 {2 R4 S- N, s3 g mount path discovery does not work.
' `+ m, L/ l1 s9 X4 e7 ~$ i" s1 S The path specified by this property must exist before the NodeManager is/ E" l. N- M% v: B
launched.
/ V5 K7 d* d2 C If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true,
0 I. Z9 x: F* f5 F YARN will first try to mount the CGroups at the specified path before5 ]" u: S( v6 Z) ?' K# F& U
reading them.4 W% _2 I2 ]& u3 `
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to
9 s$ k H }& W9 C; g4 } false, YARN will read the CGroups at the specified path.
& B6 r9 u s& e If this property is empty, YARN tries to detect the CGroups location.
* ]) B; X- `6 u( O* ? Please refer to NodeManagerCgroups.html in the documentation for further
# e& `4 T- N5 c, p; u details.
0 _$ n" u% [) j; \ This property only applies when the LCE resources handler is set to h4 y, v2 ` g: y
CgroupsLCEResourcesHandler.
! Y3 v' }5 W6 v- M1 q( G </description>
* l0 Y T# G" `, K% E <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>0 c2 A6 r6 S1 B7 y1 {
</property>
8 c% p" `! k* m7 J$ t <property>
& t/ ^" ]: D ]" Y& y& E) n <description>Delay in ms between attempts to remove linux cgroup</description>+ v' c! ~1 b# `
<name>yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms</name>
. t1 l5 v+ V- y; |$ A5 K <value>20</value>
5 z( }% W4 r; k+ w. d </property>1 a; f2 c* l7 P
<property>
1 K" H. ?( m9 a& k <description>This determines which of the two modes that LCE should use on
/ l8 D g. ~7 h. h/ J7 q a non-secure cluster. If this value is set to true, then all containers/ C1 k0 @* L# J
will be launched as the user specified in
4 N5 V$ L, h# o% e% p) u" M9 W/ u yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If( ?- O8 ~) Y/ S3 p
this value is set to false, then containers will run as the user who
' d5 F+ a6 a/ P- \9 T submitted the application.</description>
( {, t) @$ N2 V$ p7 r ?- ?# L5 a <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
! m7 g7 F& q. i. p* c <value>true</value>4 i7 ^& c* c; n. O! Q
</property> X8 l8 }% C; H) M3 @" \0 n
<property>
: Q8 e. Q7 W4 z+ o: }6 R. \ <description>The UNIX user that containers will run as when
4 @9 ?* W: E4 m' n# Q+ [: n Linux-container-executor is used in nonsecure mode (a use case for this
2 d% w5 s* e1 X# e is using cgroups) if the1 ~$ V' L& ?- }4 r& U
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is
* ~* Y4 M- b, T7 U; R set to true.</description>& j3 _+ n7 _5 n+ T" ~( U* l# }2 G+ d4 b& J
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name>
4 ^) k+ ^1 e- W1 Q* X6 E <value>nobody</value>
& N. f- K7 Q/ B5 Q1 f) V) _ m </property>
: m- _" H: S- c% o8 G <property>
7 Q1 P& G6 w3 k) R$ |2 g+ Y/ B <description>The allowed pattern for UNIX user names enforced by9 ^5 |, L" R0 r' i, j; }- G
Linux-container-executor when used in nonsecure mode (use case for this
( N! W8 u, _8 x) b! f4 e is using cgroups). The default value is taken from /usr/sbin/adduser</description>
% q8 t3 Z ^0 T& Y6 i% U <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern</name>; h1 T0 ?1 U" B* a
<value>^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$</value>
5 T" Q- I7 d+ y4 a7 [2 f </property>: `) l% }0 _) k/ B8 X
<property>& d7 T* k4 n/ a, X. q# C; w
<description>This flag determines whether apps should run with strict resource limits. J. U6 H2 W' t4 \6 u8 c+ j! S
or be allowed to consume spare resources if they need them. For example, turning the
) N T! R2 c8 n; u' c+ n flag on will restrict apps to use only their share of CPU, even if the node has spare
$ I7 c; d6 A1 K" Y$ [2 K) Y- o CPU cycles. The default value is false i.e. use available resources. Please note that. f& @3 F+ a. Z* X V3 u0 t, S
turning this flag on may reduce job throughput on the cluster. This setting does
* a$ v, d. s- x* M' H @" F not apply to other subsystems like memory.</description>- C% p3 O# C3 t5 h! F4 Q& `
<name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>
. B6 Z" v, l* S# w <value>false</value>0 ~: X) Z! N9 \& L3 z$ A8 s
</property>% w6 @* Y f L3 Z
<property>6 ~- W9 ?8 a: e3 q( E7 }% d, g. @
<description>Comma separated list of runtimes that are allowed when using
0 ^5 [& I! ^# G- `6 H% g1 d LinuxContainerExecutor. The allowed values are default, docker, and
' h+ F: m5 q, d% j7 g! ? javasandbox.</description>8 E; F8 k- e" M1 ~# l
<name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
7 g& \. @! H5 m) e- g4 V <value>default</value>
$ {7 Q6 x. `' P: b" L) B K </property>' X4 |' a: w# i4 u1 ^+ O4 J
<property>- z; E& U) N/ T S4 V# a, v
<description>This configuration setting determines the capabilities! M0 h5 I c/ g: M2 v
assigned to docker containers when they are launched. While these may not* A* l( Q( c. M# [
be case-sensitive from a docker perspective, it is best to keep these* c( U0 H o4 s- z( H& @
uppercase. To run without any capabilites, set this value to' S8 Z: }7 @+ O& B; X8 l- f+ L
"none" or "NONE"</description>
! [2 x$ o# U+ Z <name>yarn.nodemanager.runtime.linux.docker.capabilities</name>1 k+ Y- y( `8 @% ^" U( o4 Z% [
<value>CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE</value>
, [% ?$ { M/ q' j </property>
- x8 c0 M ^ G( r <property> P5 H( h7 S8 u9 y& W
<description>This configuration setting determines if: y' r, M1 O1 |* A. W
privileged docker containers are allowed on this cluster.
/ m) g& B* v' T1 y" C$ z Use with extreme care.</description>9 X+ z/ M% Y2 I- P' \ R" v
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed</name>
0 v& `8 W- u. p; `* _. j1 T) X <value>false</value>
) g2 u; }/ Z2 K* b& v9 n& | </property>
- q: Y: `0 I O. `8 `) t, p8 ?/ C0 C <property>8 X$ `* N# ^3 k7 ? K+ Z# W
<description>This configuration setting determines who is allowed to run& X7 u! \9 G' q9 Q$ G
privileged docker containers on this cluster. Use with extreme care.# Y- X5 a7 {" a4 M2 _
</description>. P" n. b9 C, [
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.acl</name>
" i* |. e) r& X% O! M2 H% C <value></value>
; u; N- E! W( n/ `! N5 z% W </property>+ y5 j* a1 F }" w- |
<property>, g- r* H. @2 a) b5 j6 Y% \6 W
<description>The set of networks allowed when launching containers using the* F$ ^# `- `# A
DockerContainerRuntime.</description>1 f: l; u) S1 q/ S' y/ {( `# T- ^
<name>yarn.nodemanager.runtime.linux.docker.allowed-container-networks</name>
o r/ T8 {1 Y) W/ z% ^ <value>host,none,bridge</value>$ r" P& _ O' f0 n, A
</property>
. k2 E4 { d2 R" x& F& L! w" ` <property>' ^) U. J; ^: i$ f1 U- N
<description>The network used when launching containers using the
6 {" ]& y$ J* M5 y3 z' q1 u) N DockerContainerRuntime when no network is specified in the request5 L9 m1 \+ P# }! |
. This network must be one of the (configurable) set of allowed container
7 `9 h7 L8 d3 K* y6 {& b networks.</description>
1 j$ X: v& p( Q/ _/ h <name>yarn.nodemanager.runtime.linux.docker.default-container-network</name>2 v. o% h1 P! D: o; C* Q9 k
<value>host</value>
. T2 X L0 T. N( L# o; v: T </property>
* [4 s1 V) g( `, ] <property>( a8 Q" O8 P; Y# |2 D
<description>This configuration setting determines whether the host's PID
1 }( z: y. U" k6 F# _% X" ^ namespace is allowed for docker containers on this cluster.
9 ^; {2 }, g) H6 N Use with care.</description>+ s; m. Y; a- w; W
<name>yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed</name>
( O* G7 S* K) T <value>false</value>6 Z/ a& @* c* \- h& u+ H
</property>
6 ?% T; G0 d, Q! @+ j <property>2 M- m2 X9 G# _# j6 b
<description>Property to enable docker user remapping</description>
# H! R( N, c! {) e <name>yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed</name>
( D: I- b. D- [' W8 ?& | <value>true</value>
# u4 Q% I2 w' R! v g% H </property>
& x: `4 W& @6 D g8 j ?. f <property>
5 r' z$ |% T \' t) Q+ m <description>lower limit for acceptable uids of user remapped user</description>, N# K0 h( \& Q9 H7 F
<name>yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold</name>
8 b1 p0 R0 t* r5 U6 d; w9 A% o <value>1</value>
+ P* V2 {% U- p6 } </property>
. T7 a4 i' E; ]! g) k1 g <property>9 S( ]4 L5 ?8 n$ G. P% Q# [/ c
<description>lower limit for acceptable gids of user remapped user</description>
- Q5 s# U3 h1 D: J <name>yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold</name>& J% T0 M2 H2 X5 [- m/ h" B
<value>1</value>7 u* N+ [* L/ d' w- J& O/ i! Q; b
</property>4 }1 c5 q/ z( R. }3 h$ w
<property>
9 r6 v/ a% d, q+ L B7 Q$ X/ g <description>Whether or not users are allowed to request that Docker
a) B2 c- f' ?, l1 E$ e; i# X6 H containers honor the debug deletion delay. This is useful for
' P; c& V3 f5 y( l" e% S troubleshooting Docker container related launch failures.</description>4 U" B x0 u, j' q" s* `! R. k
<name>yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed</name>5 q$ K) y; G- a
<value>false</value>) i+ W, v& r2 U8 b9 P4 M* V
</property>. D6 K; i( t5 ? ]. }
<property>
; G; g; K" m3 i# W- Q# g, O1 |+ j: ^ <description>The default list of read-only mounts to be bind-mounted
0 I. \- z4 e2 j- U" i, o" {& d into all Docker containers that use DockerContainerRuntime.</description>. F# l* w0 \1 G) c* |" p" M
<name>yarn.nodemanager.runtime.linux.docker.default-ro-mounts</name>
% `' H q: p0 s4 y1 w+ w* F <value></value>
* y5 s+ B+ q8 _; | </property>% I) g8 [# d, a' v
<property> P; [" M) t, l c& O
<description>The default list of read-write mounts to be bind-mounted
# b. k0 [0 ~& G$ O$ M9 Y: K8 E# U, T into all Docker containers that use DockerContainerRuntime.</description>
3 q7 m# x" i7 m <name>yarn.nodemanager.runtime.linux.docker.default-rw-mounts</name>
" ^( u% y( j1 r( l" \ <value></value>
- {4 |2 e. A( E: ?' ?1 ~# L, M </property>, i+ c" Q4 g% Z3 }7 i5 H( w. P
<property>
) F M: x7 m _! T F <description>The mode in which the Java Container Sandbox should run detailed by4 O* R) D: s+ y
the JavaSandboxLinuxContainerRuntime.</description>" n3 Z+ g. }: X
<name>yarn.nodemanager.runtime.linux.sandbox-mode</name>4 w; ?, n2 k3 } C( P. J, ?
<value>disabled</value>
& c" r4 i- K& C# w; V, O: R </property>& [: p2 j7 y: i3 k
<property>. b3 ^" _8 d! @
<description>Permissions for application local directories.</description>& [) Q1 E; ^0 k! F0 W: h! N
<name>yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions</name>
' w/ V# V- u1 K7 {1 [8 K <value>read</value>
5 K: b& i4 N6 { </property>
3 n6 g3 j) Q3 o4 q/ r <property>% S4 e; ^) T: Y4 ?" K+ F- O+ B
<description>Location for non-default java policy file.</description>
% @/ y: C; D+ e; }% P1 D <name>yarn.nodemanager.runtime.linux.sandbox-mode.policy</name>
$ t+ Q: r6 ]5 C$ M% ? <value></value>) P- \3 E2 h6 a
</property>
+ J8 \! Y; `/ \; T <property>4 H! L4 o) K* G: b5 }. t
<description>The group which will run by default without the java security
! c0 a2 p0 c1 j8 W; B manager.</description>! W' s9 X% K! }* ~
<name>yarn.nodemanager.runtime.linux.sandbox-mode.whitelist-group</name>+ }# N% n( c- [6 t* V4 N1 g
<value></value>
2 b- ~* L5 l8 J* ^3 Q$ j </property>
5 D- n% g% i- u) x- p0 V* U' q; G5 [ <property>
8 Z8 j, J/ B5 U <description>This flag determines whether memory limit will be set for the Windows Job
4 x1 } a0 S% }) e' J7 @3 U/ x% l9 ~ Object of the containers launched by the default container executor.</description>* J6 X9 c/ f# M
<name>yarn.nodemanager.windows-container.memory-limit.enabled</name>
) }+ p. E! r+ |# h, k. ~ <value>false</value>
+ K. O5 r- [- F s, Z0 U </property>
( L' Z: f/ h$ K$ G <property>
& s; |, x8 w- \( I& x <description>This flag determines whether CPU limit will be set for the Windows Job2 W$ N9 q, T* D h
Object of the containers launched by the default container executor.</description>% T. y! h- u/ R5 e i2 k0 E
<name>yarn.nodemanager.windows-container.cpu-limit.enabled</name>% z' O. f& z& b
<value>false</value>
U, V3 X; S O </property>
( E# {" j) z8 b <property>
2 @9 T6 P. x: z5 u, k <description>
0 J, T4 j( o; P Interval of time the linux container executor should try cleaning up8 Y2 V9 A+ r- d; g. U
cgroups entry when cleaning up a container.
" f0 [) G- `5 c) d </description>- @1 q9 v' B h& ^/ l3 h9 |0 F2 V
<name>yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms</name>2 O9 W/ D/ O# l3 K
<value>1000</value>
: n2 X0 b: i5 j </property>
, K3 k( ~8 S1 P3 r+ W! k <property>/ z# J( w0 ~" q2 a' }& }" o
<description>+ y) }7 w7 Q9 W; ~9 @& g
The UNIX group that the linux-container-executor should run as.8 H9 ^( U7 e, @( |( H
</description>1 F. D0 e8 S6 q K" u9 b
<name>yarn.nodemanager.linux-container-executor.group</name>
. C) l9 I0 p: m; o3 L <value></value>
8 X- n6 x4 j2 {* W& c6 v- J& i </property>0 L: i% u% Z# B
<property>
3 |8 x- [+ X! ^: U <description>T-file compression types used to compress aggregated logs.</description>
9 Q w4 o! Y. F% s/ t5 w6 y <name>yarn.nodemanager.log-aggregation.compression-type</name>
3 @/ g$ a5 p& e) L8 {( B <value>none</value>$ g# {2 {) m. x: y8 s
</property>% v8 h6 a) G* S0 x: ^2 K3 @* }1 L
<property>) g. r* P' ] ^$ x& F
<description>The kerberos principal for the node manager.</description>
) U! g ^/ d4 `# R6 M3 B! p8 Y <name>yarn.nodemanager.principal</name>% j' y% ]7 f. c
<value></value>+ ^7 |6 O u! s, p
</property>
% T+ z' U- U5 ]' \2 S <property>
8 j, y2 }; N+ A- C- I# { <description>A comma separated list of services where service name should only
C5 w8 }7 p' }8 j contain a-zA-Z0-9_ and can not start with numbers</description>
( P u0 \3 O# g' S <name>yarn.nodemanager.aux-services</name>
+ L, `5 `' W' ]7 v) C <value></value>" k2 Z9 ]2 W4 G; q! ~
<!--<value>mapreduce_shuffle</value>-->
1 m/ }" C9 A- U6 H T! W b7 ]0 r1 l </property>
+ N- V$ }. A; k4 I <property>
$ x; F* x# \% }, A <description>No. of ms to wait between sending a SIGTERM and SIGKILL to a container</description>
8 u# m% c: C. K# F. ?3 L; }7 k' X <name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>$ w: ?! ~) T% h- C! y( n
<value>250</value>
. Y& {) s' O2 z' T0 T! S7 |7 Q, R </property>9 `9 ]8 W" [ D! o. T
<property>
7 G$ c8 C5 x6 ]8 _$ w <description>Max time to wait for a process to come up when trying to cleanup a container</description>8 K! z1 h3 |% \# n0 O( T
<name>yarn.nodemanager.process-kill-wait.ms</name>
. {; h( [. r4 v" p# g1 d; b <value>5000</value>, ^, }. {% ^! c
</property>( }4 r3 ~/ U1 \$ ~: c8 X4 X1 S( i" s
<property>
% A8 ]9 J' m% K6 K, j7 @ <description>The minimum allowed version of a resourcemanager that a nodemanager will connect to.
$ I& b+ E, _2 l3 ?6 y1 x( T The valid values are NONE (no version checking), EqualToNM (the resourcemanager's version is 0 ^6 S, \. r/ a3 A2 `# R4 C" q" S
equal to or greater than the NM version), or a Version String.</description>
4 d" k: ]3 b4 w; H <name>yarn.nodemanager.resourcemanager.minimum.version</name>
. ]" E/ D) G* l. F' J, r9 D, F <value>NONE</value>
) N! m! F+ O9 H* H* _- t </property>" t+ q4 m1 d9 K9 Y3 f9 j5 {( U0 |* [
<property>. V0 K' {7 {) i. K: |
<description>Maximum size of contain's diagnostics to keep for relaunching
5 Q! U% m. a/ s6 M) J container case.</description>
% [5 ]) }/ `: K% L <name>yarn.nodemanager.container-diagnostics-maximum-size</name>+ ~: J. }2 |: h
<value>10000</value>- e8 r0 T7 _5 Q0 W" W
</property>
5 A& D* q8 C* L' m4 p; u# t' _ <property>
( ]8 z* y2 \( \# g <description>Minimum container restart interval in milliseconds.</description>7 _' y7 o8 m7 A4 g* o: V* U7 `1 o
<name>yarn.nodemanager.container-retry-minimum-interval-ms</name>1 n- H* s3 _" @2 L$ u
<value>1000</value>9 ^; e* Y0 {4 N; ~. o
</property>
0 k c) S2 \+ r" x$ d5 M7 a9 t, x <property>
# k1 ^& B+ P! y5 h0 k" D <description>Max number of threads in NMClientAsync to process container1 ]9 ~5 U0 A9 E7 S
management events</description>* j% e5 @6 M2 N9 I3 ~0 D
<name>yarn.client.nodemanager-client-async.thread-pool-max-size</name>
4 C/ G: m9 K& \3 a! L" {# s/ {: \ <value>500</value>8 C6 O: ]! [* W& {- c8 y- s
</property>
8 j" r* C7 V! ?* R <property>: @( A! E+ H2 C9 {) A4 }/ a
<description>Max time to wait to establish a connection to NM</description> C3 P! h8 W( ^' b2 P
<name>yarn.client.nodemanager-connect.max-wait-ms</name>7 h/ u4 [3 ]0 M1 K+ A
<value>180000</value>
- m# E. @ I( t( c" |; g </property>1 C6 H7 B5 O* K# `, V% U, k
<property>% c" x3 T$ z U1 _+ c& R
<description>Time interval between each attempt to connect to NM</description>
: \& |: ~' i/ s; y. i) A- V; y <name>yarn.client.nodemanager-connect.retry-interval-ms</name>
1 \2 x9 ~9 p. a; I8 G# } <value>10000</value>
4 x( O2 G' p/ h </property>
/ g" _! s; r* E$ w1 u/ i& J6 C <property>
, D, V2 ^4 j; w* i <description>% k2 @% K+ V7 v8 J0 K2 `& W
Max time to wait for NM to connect to RM." w9 Y% J) T A% G/ d9 g* {
When not set, proxy will fall back to use value of
6 r6 O0 z. u2 Z1 d yarn.resourcemanager.connect.max-wait.ms.* i" S; i' C3 K# g: p' |( x, |7 \
</description>
* y, j) ]+ O8 T3 [2 o$ | <name>yarn.nodemanager.resourcemanager.connect.max-wait.ms</name>
( O0 y" Y/ E, q" p. D8 e2 D! w <value></value>
/ T1 b" z- X* s3 O Q# q- d </property>
- ?) n. v* l, o3 r! L$ I9 ` <property>
' K6 W0 }: O" {+ ?9 Y5 a9 q <description>
- A* `( F$ c. E( p& S( s, u$ A2 h Time interval between each NM attempt to connect to RM.4 Z2 P* M7 D2 E2 { x5 p% h
When not set, proxy will fall back to use value of
2 w, R y$ }, O* J' d yarn.resourcemanager.connect.retry-interval.ms.( n/ W/ M4 D3 i1 D# ]6 Y
</description>
( O @7 n" h( c5 d% {1 U <name>yarn.nodemanager.resourcemanager.connect.retry-interval.ms</name>
1 m( }) e! O" Y9 y <value></value>
2 u" D) [" R' h& a9 c </property>" t* Q" [. o/ [: Y
<property>( c9 d0 m" b! d) h" n% H
<description>
7 h7 \1 G q& z3 U Maximum number of proxy connections to cache for node managers. If set. H6 ?, }8 k- i" n# ^6 y. p
to a value greater than zero then the cache is enabled and the NMClient
5 _3 _& q$ |& l4 z2 G1 W: Q% ? and MRAppMaster will cache the specified number of node manager proxies.
6 _1 j( { ~1 Y( n There will be at max one proxy per node manager. Ex. configuring it to a
, s" T: z; A) J5 t' x value of 5 will make sure that client will at max have 5 proxies cached
, S8 `% {9 p( f1 p: e with 5 different node managers. These connections for these proxies will
/ |+ K4 D9 k( h/ D$ s be timed out if idle for more than the system wide idle timeout period.
$ Q& T" N7 l2 d: y Note that this could cause issues on large clusters as many connections) {7 b& {% A3 P% H( b, O
could linger simultaneously and lead to a large number of connection
4 g! R; W. ^$ X4 }! ` threads. The token used for authentication will be used only at0 b) L: ~* @: x- N @# S+ N
connection creation time. If a new token is received then the earlier
, a7 v5 C, V; E connection should be closed in order to use the new token. This and
9 ^ }4 `" ^( k4 N3 f (yarn.client.nodemanager-client-async.thread-pool-max-size) are related
- R3 n4 Z# ~+ z# r+ B; A and should be in sync (no need for them to be equal).
4 W& j1 E: p P% G% C If the value of this property is zero then the connection cache is
2 r* Q, `( V; E" \) }1 {% x& b disabled and connections will use a zero idle timeout to prevent too
" K& n6 V( m2 H3 y4 S! v4 A5 r4 ? many connection threads on large clusters., s4 i) z% y2 }0 [: p
</description>- F: U/ e" D, P/ b& E
<name>yarn.client.max-cached-nodemanagers-proxies</name>, }, L# S! {* L2 y
<value>0</value>
! K8 [, c1 \2 f2 G* c </property>
' b2 u3 `" G Q2 S9 q& g9 T+ A <property>6 d. o% K& J1 \" `
<description>Enable the node manager to recover after starting</description>6 X& Q% l4 }) d, u+ H( u
<name>yarn.nodemanager.recovery.enabled</name>' P" w+ b! p" M! [; y9 S# v
<value>false</value>" Y `& F; c% {
</property>0 r3 ~# f4 J; f
<property>
! r% G. K V; c8 v' q <description>The local filesystem directory in which the node manager will
0 `# a' h4 Z% u x7 Y3 c3 ^1 o store state when recovery is enabled.</description>
% r8 T9 Y8 i$ }# w" k7 u O, F& v <name>yarn.nodemanager.recovery.dir</name>) f# U& y( D! O7 G
<value>${hadoop.tmp.dir}/yarn-nm-recovery</value> Y5 s0 S( |& ?/ g) l! Z) B( j" g
</property>- ~$ J0 P5 G9 ?$ I4 l% A
<property>
4 N/ }9 C3 d# o% f0 E! G <description>The time in seconds between full compactions of the NM state; r C# y# \% R9 D# y% E
database. Setting the interval to zero disables the full compaction8 ~( e# o$ ]% c2 w* }
cycles.</description>
# ?, e( u- `, w- C <name>yarn.nodemanager.recovery.compaction-interval-secs</name>
6 R' F( H6 G1 V; q, `* T <value>3600</value>
! @; g D! T1 L3 o m </property>5 ]7 W% x1 c; o. V% G" ]2 R- y
<property>
+ D* T* Y; [1 M# z+ E. e* y4 ` <description>Whether the nodemanager is running under supervision. A
H9 I+ z8 _" V# l7 b% C nodemanager that supports recovery and is running under supervision# O H, H' W3 X3 z( c, N1 M3 I
will not try to cleanup containers as it exits with the assumption1 z T _9 `$ ?8 ^
it will be immediately be restarted and recover containers.</description>3 A. ?4 C, r# V2 Q2 G
<name>yarn.nodemanager.recovery.supervised</name>
7 T' y' W% `0 q6 U <value>false</value>
" R4 L, O- n' X. `) v5 q </property>
3 U) M# x" ]" z <!--Docker configuration-->
& s" k; Z1 s& U5 L2 f) k <property># A/ B$ @& b8 o! ?/ s$ r: G
<description>
8 i: g; V% L" k( {; H Adjustment to the container OS scheduling priority. In Linux, passed M5 l, h) T" {3 x8 | i
directly to the nice command. If unspecified then containers are launched+ E9 x* D# d8 z3 H: j1 s/ \
without any explicit OS priority.
' s. U! I, S( o' N </description>) w0 p6 C6 E3 G7 G: |5 ]& C# f& u
<name>yarn.nodemanager.container-executor.os.sched.priority.adjustment</name>
' V& X; F9 `/ c% p. r$ d </property>4 c0 ~. x0 Q2 R- O) E
<property>9 i7 x0 f! D- W$ v* M
<description>2 Y3 n2 @# W' E* m0 m g
Flag to enable container metrics
+ h2 ]- O% Y: \. e2 A5 M+ e, O </description>" G1 ~6 X2 P+ t' f4 U2 Z/ f" P
<name>yarn.nodemanager.container-metrics.enable</name>
# |$ v: w" `8 b6 Q1 g2 e4 c <value>true</value>
" K, g/ H) n( X; }& I& e$ ` </property>3 k1 G( c8 K! q/ v9 J9 R- h) y, X
<property>
$ e' R* |, ^* Z6 D+ i. T6 y <description>; ]! b* K0 i1 B- z- W6 v8 @- O+ c
Container metrics flush period in ms. Set to -1 for flush on completion.
; \! z2 r! \3 K5 d5 f' e </description>, s2 E5 O7 f3 D" G) E/ E2 W
<name>yarn.nodemanager.container-metrics.period-ms</name>! F% I3 L0 u$ [( r7 T" K; C
<value>-1</value>" ^! G1 R' E0 ?5 ]2 P, p% B
</property>
3 y4 B& n8 s' [) W <property>
0 P6 O1 ~9 s1 q2 l- s <description>
7 C2 X; _( Z" [ The delay time ms to unregister container metrics after completion." y s d; e+ c# g5 u& [$ \
</description>
- n+ w5 g& J h( W <name>yarn.nodemanager.container-metrics.unregister-delay-ms</name>
! h: G( M- t5 v% G <value>10000</value>
% K+ P& }; w# c& h7 A( |. ^ </property>
% l+ y8 K. _6 G0 d/ s2 ]7 \4 a/ ^ <property>* e Z e" a* x
<description>
$ U u& \1 a& c( E3 A5 _6 O; o Class used to calculate current container resource utilization.
9 [: R! }7 G% ]- I# z </description>4 ]: ~! ^! ^, N+ _6 S- S; U
<name>yarn.nodemanager.container-monitor.process-tree.class</name>7 q& H; U; {) ^& c
<value></value>
6 q' o: q0 C' B& l9 F </property>( Y' Z3 ]" z3 r
<property>1 ]9 o4 I Q" I( Y
<description>: K* d0 M7 q: }& L& A& W4 [; c5 t
Flag to enable NodeManager disk health checker! @& B' [3 C' W k8 O$ y
</description>& B E* M! G( D1 M
<name>yarn.nodemanager.disk-health-checker.enable</name>
% ?/ E. c* l( u/ t7 t9 i) d <value>true</value>
* [/ F& T6 a( k. L </property>
+ d- P2 J0 ~. E& u6 a+ J) _. _' D. g <property>- J9 J' s7 G2 H2 R2 X
<description>, C: R0 c4 Q: }- m
Number of threads to use in NM log cleanup. Used when log aggregation. I2 k: |3 h# d) |
is disabled.' p9 h+ c( x* E6 F$ C
</description>) i5 x9 G& R4 V3 x5 {
<name>yarn.nodemanager.log.deletion-threads-count</name>
/ E3 B" F9 m3 p! {+ U% ` <value>4</value>
. l2 M/ l9 w4 t' ? </property>
; I* W8 o4 R0 i, Z <property>
: X2 d% M$ @1 L( m+ K2 y# B' O4 C <description>
, J! ~) X9 ~0 ? f8 H1 R) C The Windows group that the windows-container-executor should run as.8 R4 E, V0 X% e4 d) ]6 q5 F5 e
</description>; _2 h* `* y, e0 j" ?
<name>yarn.nodemanager.windows-secure-container-executor.group</name>9 u! M, ]& J8 U$ P! D, i* u
<value></value>% j" r* e3 |/ v& n2 G
</property>6 l/ s$ U* ~% X; S8 [( f! ~
<!-- Map Reduce Configuration -->! A& Y C6 i {6 R
<property>1 M8 X1 u) W9 u- I+ v: ^, y& r
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
- y, K* ` ]* |+ F+ x& c <value>org.apache.hadoop.mapred.ShuffleHandler</value>& }0 L, {3 C/ r/ ^: Q! H" j2 |3 ~* m
</property>7 f7 i& c, s3 `9 {
<!-- WebAppProxy Configuration --># k$ }4 i: G7 w( h- ]$ W
<property>
7 A* ?) q$ d! S6 E( g: \3 k+ D. | <description>The kerberos principal for the proxy, if the proxy is not
+ e5 Z* j& n% ] O4 ?! n running as part of the RM.</description>$ M6 V( R! y; t+ t! c# X
<name>yarn.web-proxy.principal</name>& v# N J* z5 O1 x( V
<value/>
# G8 Y) r8 M) q0 M3 S( Z" M </property>
; V1 \# _: `7 M6 Y0 z: p- ~) L# A) } <property>
( K# _% x5 R" S1 g, P/ G* p <description>Keytab for WebAppProxy, if the proxy is not running as part of ! H3 I. i: d$ ^; s- H0 {4 N
the RM.</description>, V: p9 @) c1 N8 }8 X, X8 e, S
<name>yarn.web-proxy.keytab</name>& v3 t+ t: x& P7 f% N
</property>% t$ c1 i# @# B1 U
<property>
8 [( a. L& t# ] <description>The address for the web proxy as HOST:PORT, if this is not
' b9 [$ ^& ?: P* X" \7 b given then the proxy will run as part of the RM</description>
2 q- o0 m) d2 R <name>yarn.web-proxy.address</name>5 T& |7 y2 o7 `+ ^
<value/>, I9 [2 `! m0 n, M. l7 @
</property>
8 Q) Q8 d2 b2 |7 S4 C <!-- Applications' Configuration -->; @, r+ [9 \' \6 F) C3 m- o
<property>+ k. @; e1 z2 {$ S: g/ W; \* o
<description>, Y6 p/ I, f3 R& [/ {! ?
CLASSPATH for YARN applications. A comma-separated list
: U& h5 b2 M) A4 {6 g- { of CLASSPATH entries. When this value is empty, the following default
4 H6 Z+ k( j5 ~) a9 S& E CLASSPATH for YARN applications would be used.
; M7 x* W, Q& w) e, |. c For Linux:. x7 @4 Y& {* A6 [: }, g' w7 J# z
$HADOOP_CONF_DIR,: K) _- A8 m: a/ N9 y1 C
$HADOOP_COMMON_HOME/share/hadoop/common/*,
3 y0 k Y s# P& t2 |- P $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,: q l9 _* Z5 q4 o/ b, `" O! X
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,' F! n9 p7 H) c" ]5 V# r! G
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
5 v0 n X% U7 i $HADOOP_YARN_HOME/share/hadoop/yarn/*,
3 p' k1 A- I" X/ ] $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
( H: Y* e) Q% m2 ?4 A; ~4 e9 | For Windows:
6 H5 m7 P& X: L# k %HADOOP_CONF_DIR%,1 e- G; U* q4 T2 b1 C
%HADOOP_COMMON_HOME%/share/hadoop/common/*,) s- X% j3 P- ?/ j
%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,1 x1 S' C) i" Y7 T! M0 q6 U
%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,; _; Q2 y# B8 M, B; X
%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,# J; a+ b. }& Z( E3 }) m7 f
%HADOOP_YARN_HOME%/share/hadoop/yarn/*,/ Z8 O0 |% W; q& b) f# Y" y$ n
%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*
2 O3 T; i& h2 f2 |0 ]5 ^" F1 ` </description>
5 |8 ^, v) w: o; X <name>yarn.application.classpath</name>
7 T( D3 d* _- l2 { <value></value>% s3 Z) X/ M8 ^2 @* `. Y8 e, `
</property>; @* x9 ^, f" ~' C
<!-- Timeline Service Configuration -->
5 {* L6 i' I+ q# A( e" ?* T ? <property>2 m& x) v8 V1 a
<description>Indicate what is the current version of the running: E; W- V2 @7 p, u2 Q5 L* z: N
timeline service. For example, if "yarn.timeline-service.version" is 1.5,5 L* l, D- ?3 m7 T' K; g$ b
and "yarn.timeline-service.enabled" is true, it means the cluster will and7 L" x h/ x4 Y/ z) w1 A/ ~# k
should bring up the timeline service v.1.5 (and nothing else).: O: o+ s3 W$ y; z j
On the client side, if the client uses the same version of timeline service,
1 i, q( J m+ q0 O' o# V it should succeed. If the client chooses to use a smaller version in spite of this,' v4 ~" {9 ^% c
then depending on how robust the compatibility story is between versions,
( M, L* A% X- `" o9 v5 @: W; d the results may vary.& V7 }- \" }2 Z/ j
</description>' L& |/ f2 T8 X8 X5 T8 C _, a, m
<name>yarn.timeline-service.version</name>$ C! G# O; ]: F4 U6 H$ |
<value>1.0f</value>
. _; y, W- z, g </property>$ K* n6 T+ Z& H+ _
<property>
+ Y0 m: J; [% S2 n; V+ s <description>4 M! @% O+ G h
In the server side it indicates whether timeline service is enabled or not.1 u6 f; F7 c c( g7 N
And in the client side, users can enable it to indicate whether client wants
% S. C$ @# R" S% J+ I5 C2 F to use timeline service. If it's enabled in the client side along with
2 L$ d- \" ?* ]& z5 g5 v! D security, then yarn client tries to fetch the delegation tokens for the ~8 p" r3 b5 H M) ~8 B4 V! r1 A
timeline server.9 c" }# r, ]: Q1 S* G3 N
</description>
) m$ y8 g" `1 j$ |9 Z9 w+ o( L <name>yarn.timeline-service.enabled</name>
- F5 a8 L) l# |, l" O- W <value>false</value>$ P3 N/ m9 \1 h* d* R/ G+ e7 v
</property>
. d. z( i0 }( B <property>
R' b( [3 K# C6 e1 _! u2 e <description>The hostname of the timeline service web application.</description>
' z+ d' q' v E, A" C <name>yarn.timeline-service.hostname</name>
0 h4 e- E: L& n/ J& T <value>0.0.0.0</value>$ J# Q. D5 F: P6 Z4 T$ s/ U
</property>8 y) f$ r# @) H4 @! Q& ^1 Q
<property>6 `% \* M, c: _- g K
<description>This is default address for the timeline server to start the
6 E$ P( X h& ]( }6 D. \+ O1 S+ L RPC server.</description>) g" V: [" f1 K* b. X4 r
<name>yarn.timeline-service.address</name>
4 i& q4 }- u7 M$ ~: s) ^* P- x# { <value>${yarn.timeline-service.hostname}:10200</value>. V7 t' g* X! y
</property>3 y/ B+ o- T# R: X6 ?# q4 i
<property>- }7 b, u& R& {" s' S6 s
<description>The http address of the timeline service web application.</description>
6 f9 \* R7 d! x; R; O <name>yarn.timeline-service.webapp.address</name>
2 q1 @/ ^5 R N& v' F5 _ <value>${yarn.timeline-service.hostname}:8188</value>% Y/ P2 T) S! d E% b5 m- B4 e# u
</property>
6 e( o9 c4 I' ]2 t; U2 U% T5 G6 I% D& d <property>
- i) f' S+ f0 ?, p) t) } <description>The https address of the timeline service web application.</description> t3 U$ H. Y& ^
<name>yarn.timeline-service.webapp.https.address</name>
, E4 e6 O2 `* r# |# a8 i2 k <value>${yarn.timeline-service.hostname}:8190</value>( V: b+ j. Z! ^- w- v/ R; e
</property>
+ h# K) G% m `" s4 _ <property> L* \$ Y, A1 D4 J' {! i' w7 @
<description>. w6 G& a5 ~! J6 Q& q4 r0 k3 {# Y
The actual address the server will bind to. If this optional address is
7 N' O' h6 ~: E set, the RPC and webapp servers will bind to this address and the port specified in
$ ~( P! u1 D- c6 Q" S! K' ] yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively.
! Z5 h; R) E& f0 b5 K This is most useful for making the service listen to all interfaces by setting to
; N1 f6 X2 M% W 0.0.0.0.
' c% k0 h- Z: u6 S2 |- g </description>
; _) z6 `1 M" D- ~8 H! ?0 P" | <name>yarn.timeline-service.bind-host</name># q% y$ r9 F9 r
<value></value>* Y( Q2 s: R" I
</property>
9 ]( C6 Y% {' v l3 Y <property>- Z+ g6 p J1 h) O
<description>
% W. n7 f5 h) f$ X! n& U Defines the max number of applications could be fetched using REST API or, h; @) M/ v( l
application history protocol and shown in timeline server web ui.
' \8 M; F7 b1 T </description>
* N* t. @) J. c: J, ~ <name>yarn.timeline-service.generic-application-history.max-applications</name>
1 V/ w) x4 E- V( X v. b <value>10000</value>3 g1 k0 V$ \* {/ ]& c
</property>( E; J; G: Y! {2 T( O, f
<property>% X. D0 ]% l- H* p
<description>Store class name for timeline store.</description>
6 ]2 b" i/ {; G6 ]3 B <name>yarn.timeline-service.store-class</name>% N1 ^0 u) S4 M9 _6 \
<value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value> R2 O/ I" @$ @( f; X2 ]
</property>
& y2 ^; h3 r$ o7 V* _) P <property>9 G+ s" D: Q3 A7 k( z2 S
<description>Enable age off of timeline store data.</description>
7 ?$ O7 E. M3 H: u+ D1 l <name>yarn.timeline-service.ttl-enable</name>
8 {% S% Z7 A; m& F; Z1 a/ \. E% l" _' I$ \ <value>true</value>& u' W3 ?' m+ i/ ~. b: U% L) f
</property>, u. y9 c( t2 X* j" T' a% q
<property>7 a3 a- t, z$ r
<description>Time to live for timeline store data in milliseconds.</description>
8 m. I4 _' ^' a <name>yarn.timeline-service.ttl-ms</name>" }5 }5 k* D# O* ^8 v" V
<value>604800000</value>' o" C+ \' y/ _; Y8 a" d& _' R6 B3 U v
</property>! {5 l5 y8 F# M, B- Q0 g! x+ H
<property>
( t3 H7 m+ n- Z5 w3 m k <description>Store file name for leveldb timeline store.</description>
& D6 h% X* l' t' W: Y7 \* i <name>yarn.timeline-service.leveldb-timeline-store.path</name>3 G- D6 f5 o* T4 E6 U
<value>${hadoop.tmp.dir}/yarn/timeline</value>
t+ ]* o6 z' p1 K# S </property>
0 `0 W+ {: A" S$ B <property>1 V3 e" T+ `$ r5 d
<description>Length of time to wait between deletion cycles of leveldb timeline store in milliseconds.</description>
: J- @# T- c5 c! `! k+ o! @ <name>yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms</name>
1 a' ~, {5 `) P: q( o) N7 [2 ~9 e <value>300000</value>
; w9 M2 S. R# p0 K, Z/ X+ h </property>7 o# N8 O/ o1 u) M% E3 o& B3 F
<property>( Y( i5 Q: Y8 P* M( \5 `
<description>Size of read cache for uncompressed blocks for leveldb timeline store in bytes.</description>
- K. ^6 R* B: W& A( _ <name>yarn.timeline-service.leveldb-timeline-store.read-cache-size</name>
! s; h# o) X6 t' ^; l <value>104857600</value>% O9 C" R$ z7 ^" ~2 R% y( p3 O. T
</property>
- E& d3 `: b; o8 n& u( G <property>
0 x3 p' e3 M; o$ R. i9 X <description>Size of cache for recently read entity start times for leveldb timeline store in number of entities.</description>
! @; F6 U& L% v/ Q9 [9 j <name>yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size</name>& e {9 `8 L$ u& k6 a5 k3 n
<value>10000</value>
+ O; L- ~& `1 ]' o </property>
/ F# s c7 K0 u% }4 `: p" z <property>
# [: C; G' d! g8 w, t$ Q% C6 O( U <description>Size of cache for recently written entity start times for leveldb timeline store in number of entities.</description>
9 C8 ~4 Y2 K: u8 x+ E$ x <name>yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size</name>
& x) {4 I8 b1 @9 H: K+ v <value>10000</value>
. M2 ~& P7 n" Q! R& g1 Y% v; l, p </property> M- N2 O3 F( P& ]3 n3 o h
<property>8 l$ f* y5 ^; y
<description>Handler thread count to serve the client RPC requests.</description>6 f( H) P) v+ }" V0 {
<name>yarn.timeline-service.handler-thread-count</name>
8 ~; Y' D% n3 I Y) d/ K. w( a, _ <value>10</value>
* y ~- Y; l. v* H6 N </property>/ `( {, r% g, f) Y; C5 z
<property>( _* R0 U3 ^: X" v5 N) U3 u
<name>yarn.timeline-service.http-authentication.type</name>
: o* N9 a9 a. u- T2 V& d L! Y4 j <value>simple</value>& g8 I( s4 a4 y$ K3 r! ]- S) x9 E
<description>
% v5 g) P% S5 {0 ]9 s8 ~+ b5 U% x Defines authentication used for the timeline server HTTP endpoint.6 ?8 C) k) @4 w% ^" ^& ~- Q1 |9 G. A
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#8 \. H6 i$ `7 P' z' k/ a: b
</description>; ]" v( b2 X7 z2 }4 _
</property>* }% w7 F1 c3 v9 Y0 d8 i( o2 Y3 {
<property>
: A0 r; Q1 p8 ?2 j- D <name>yarn.timeline-service.http-authentication.simple.anonymous.allowed</name>
3 f* m4 d1 N8 T, Q! u5 J, }1 Y <value>true</value>. V( ]/ t' a& _3 X6 C9 Q" e( M& B/ R
<description>, p2 q$ h+ j' `& K2 B, r' C
Indicates if anonymous requests are allowed by the timeline server when using
% }' T7 X" c( r! L$ o1 m, q 'simple' authentication.
6 i' ?0 |) v+ `0 B9 j( o </description>
" R& m( I/ h/ ~5 A9 M </property>
7 c4 `* D9 B9 x" U3 _' @2 f* b$ C <property>, X# x. ]3 }8 s1 K. z% ]9 ~- K! L6 k8 p
<description>The Kerberos principal for the timeline server.</description>
# v. B/ X& l# Q, k. _& U <name>yarn.timeline-service.principal</name># m6 D1 ?& F( V5 ~
<value></value>+ c2 _0 g0 u, _" o
</property>7 V' _) K) V( A) F
<property>. Q# e3 m4 a6 K R# Z% G! f% t
<description>The Kerberos keytab for the timeline server.</description>4 w8 I2 z% \. b
<name>yarn.timeline-service.keytab</name>
8 s2 W* Z" c& s; j: T% x <value>/etc/krb5.keytab</value>
! e! N6 h. k; x4 {+ {# V) }" U5 J </property>( w! _! K5 x ^/ W- i) m/ K
<property>
1 W/ q1 P' J5 `0 |% { <description>Comma separated list of UIs that will be hosted</description>9 c- T& x: f4 B6 [; r% `
<name>yarn.timeline-service.ui-names</name>
Z4 i3 [1 B. E+ U+ p! L% c% _+ x <value></value>
& V# L0 {# Y% P </property>- m* Y3 B' {1 o7 s8 H" b P/ `
<property>
9 m8 u3 x5 L9 R" C4 [/ D/ {0 l <description>3 c3 } S& m# Y
Default maximum number of retries for timeline service client0 Z# g! t! b- o! t6 Q% X8 d3 f
and value -1 means no limit.
e* |: R# m c V# \ </description>
: \! f0 x! D3 \7 T5 ~6 S: q <name>yarn.timeline-service.client.max-retries</name>
/ ]6 {; ^9 {" p4 x% [1 }0 t <value>30</value>
1 g+ I" U3 e* m1 N0 } </property>
& w- i3 `6 U6 A% S <property>
1 M' I6 Q5 N4 ~& ? <description>Client policy for whether timeline operations are non-fatal.
, `$ E. v! W& t3 h Should the failure to obtain a delegation token be considered an application1 d& Q& ^& R& ]( s2 r8 A- |
failure (option = false), or should the client attempt to continue to" @: u2 [. O" r6 s7 S, X, r& g0 e
publish information without it (option=true)</description>
! a' o1 A4 V' o4 B" E" h <name>yarn.timeline-service.client.best-effort</name>
9 p3 {- {' G. j1 j t3 O' x9 ~- m# \8 d <value>false</value>
: j4 G6 F5 e) W; c! i! w3 ` </property>. i" c, O3 A5 d3 r* q
<property>
2 W- E# P2 g' ?& N <description>4 f( ]9 ~6 n+ W2 D& b9 D4 i. J: [
Default retry time interval for timeline servive client.3 u+ X. Q, s% Q% p3 Q
</description>5 H5 n3 k4 L1 A: }
<name>yarn.timeline-service.client.retry-interval-ms</name>
3 ^8 u. `& x) b. j <value>1000</value>
- ~' o! B( W( H) Y </property>- `. K- l1 e6 V6 l$ g0 a
<property>
; _; Z5 _5 V4 l <description>9 @# o4 d. T, Y w7 J
The time period for which timeline v2 client will wait for draining
. D9 L3 R8 g4 ?8 g, D( Y leftover entities after stop.. K9 i6 G$ z7 {! Y0 t1 `6 r
</description>
! R0 a$ h8 J& ^. }$ k ~% h: Y" { <name>yarn.timeline-service.client.drain-entities.timeout.ms</name>
/ ~4 i) L% @* ~1 B K <value>2000</value>
* j" n+ U' h5 L) L5 J3 f </property>5 A _( S- J7 |6 P- z: \; |3 S4 ?
<property>8 P* Q5 h& ?8 G" s, u* O0 p
<description>Enable timeline server to recover state after starting. If
& R7 o! A4 D( w+ C* B. L: f4 ^ true, then yarn.timeline-service.state-store-class must be specified.+ Q0 p" D2 V* Z! V5 m) C6 P( c
</description>
/ h% d5 n( |6 j( x. ^1 M4 \1 Z* A <name>yarn.timeline-service.recovery.enabled</name>' w- q8 M3 T0 v; ~- o n
<value>false</value>
( Q* P+ j1 q* { </property>9 |7 j$ d8 ^: O4 y1 p$ a7 B( O
<property>9 O0 C4 \1 C8 l% o' c' d
<description>Store class name for timeline state store.</description>
! I% ?! X u2 J0 a: ] <name>yarn.timeline-service.state-store-class</name>
+ B! g7 W( C" w9 X N: p <value>org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore</value>
) w. k( ~$ K" t* u) t6 ] </property>; O/ s$ R* c8 o) m7 v
<property>1 _" |$ T8 W$ O
<description>Store file name for leveldb state store.</description>% l% z8 K! c1 U# i* y, p p% ~
<name>yarn.timeline-service.leveldb-state-store.path</name>. m: q& J4 Q% o$ d$ T1 x
<value>${hadoop.tmp.dir}/yarn/timeline</value>
# s6 y; e" y" Z4 O e2 d </property>
3 r4 B0 ]$ I( ?& ]- w+ T/ p <!-- Timeline Service v1.5 Configuration -->- E7 P1 f4 g8 L C. ?; y
<property>% e! f" r6 Q% ^6 Q H
<name>yarn.timeline-service.entity-group-fs-store.cache-store-class</name>8 d8 T6 ?+ c$ a, h
<value>org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore</value>+ Y R' z3 t$ d$ s! O- a d
<description>Caching storage timeline server v1.5 is using. </description>! ^$ n/ |/ h8 i0 j, H
</property>
: Q' A) r+ N9 Y8 }% M2 m' T" R <property>" S: B1 l( ^6 l
<name>yarn.timeline-service.entity-group-fs-store.active-dir</name>
& X/ |8 \2 @9 Y( ]( v' C4 k' r) I; J <value>/tmp/entity-file-history/active</value>3 r0 B( ]" B6 Y
<description>HDFS path to store active application’s timeline data</description>
( Y7 p6 \9 G! ?0 Z" a, N </property>
8 w" g+ u4 q7 k/ Z' ^. f <property>
: b: [2 c2 D. ]% p/ }) f <name>yarn.timeline-service.entity-group-fs-store.done-dir</name>4 l1 ]; ^5 \5 i4 f1 d
<value>/tmp/entity-file-history/done/</value>% y" ]+ Q/ F* K3 `( `; O
<description>HDFS path to store done application’s timeline data</description>
+ v: x2 g c2 I$ b) ] </property>
I' I r' ~; M. Q* V2 O6 i/ _ <property>* i! O/ E. Q; R- p/ Z
<name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>) V" E" h( a3 w2 h( e3 n
<value></value>0 h7 }+ L; ~2 b% \+ U8 [
<description>
4 B6 F# [7 Z8 T+ R: n2 s W& B0 s Plugins that can translate a timeline entity read request into
9 X" h9 J( d4 d1 t1 A a list of timeline entity group ids, separated by commas.
) W4 K6 o) W$ `# v- N/ P </description>
1 e; S, t7 J8 Z: S8 B4 D: Y6 z </property>
9 }2 ~1 b8 |7 X: G9 F <property>
3 A4 B2 |6 r/ P" D: t( M* a1 q <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath</name>
' }9 \6 j1 [: s" S0 Z/ S4 e2 p <value></value>8 p! p* x- O# q7 m9 s; j. F
<description>
/ B! D0 F6 m3 B, U; B Classpath for all plugins defined in6 `% W2 y/ L* T9 Q S! f N: r* V
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes.8 w: W( R& N, Q7 Y' ~
</description>& O7 w# x) Q) j K, a
</property>5 O/ E1 \. C% u8 |! L) A" m
<property>) N7 a- b- t% E2 u
<name>yarn.timeline-service.entity-group-fs-store.summary-store</name>* z, t8 s' [4 L |
<description>Summary storage for ATS v1.5</description>
# v) J" G: g# G8 C0 C/ @ <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>) g& q9 n# |& B# Y/ Q( l. Y
</property>
1 x5 T" m$ O' ` <property>. R$ ?8 u) I; g. x
<name>yarn.timeline-service.entity-group-fs-store.scan-interval-seconds</name>& [( R- e" |$ W" _9 G
<description>
1 R# S+ n* T1 U4 [ Scan interval for ATS v1.5 entity group file system storage reader.This* D- D& A- n& V! q% K" _
value controls how frequent the reader will scan the HDFS active directory, L) Z# g% n: r W8 W7 B
for application status.
" r- ~% S) a' M( x& ? </description>5 I. S/ d3 V$ q$ y/ v4 Q
<value>60</value>
( F- |& |5 ^! t/ I( g9 h </property>
% i0 X2 |! g* O <property>
: J# e% a# C/ v _ <name>yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds</name>9 @* I( k5 t/ i3 q# H
<description>0 O" g! H0 S- s& W
Scan interval for ATS v1.5 entity group file system storage cleaner.This
/ Z( B; w8 B5 A) p- E5 G0 U) d$ x value controls how frequent the reader will scan the HDFS done directory
/ W- _$ l6 ^% p9 I% H for stale application data.5 R, x: L5 t/ P
</description>8 _8 b2 f M/ q1 ^: x' m1 U
<value>3600</value>
/ Y1 d4 J3 k1 k# F( K# K9 t8 b0 s </property>- j# ?1 @. s7 B
<property>
- z7 T+ x( H( U+ Q) }: g; M. J9 d <name>yarn.timeline-service.entity-group-fs-store.retain-seconds</name>
( ]8 b. {% F* i7 ]0 M6 [ <description>
- g, k4 e9 z5 D0 i. | How long the ATS v1.5 entity group file system storage will keep an
. Q. w B/ v# h1 y5 S* o application's data in the done directory. @0 w ]9 L+ h+ n. g' h$ h; d
</description>* d9 ]$ E) D% |0 K
<value>604800</value>3 P2 r! z* W5 _. K- T- @
</property>
/ x0 w1 J ]/ E$ V( V <property>' W* C6 z7 x0 G' Z# Q9 \8 @1 W
<name>yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size</name>
' @- E& ?0 F7 Q" g7 T1 w% b7 W% U <description>4 s: ~+ E- N' W7 [. U/ p/ s
Read cache size for the leveldb cache storage in ATS v1.5 plugin storage.
) L2 G5 [$ W$ H \* }& J </description>
2 N3 `* B( k: z9 @0 w' _6 G: N <value>10485760</value>0 B% ~. d; m$ G6 i& h
</property>
# d0 _6 u& y/ I# @1 D <property>
' Z+ Z1 F; q, A. M3 T. L& L: b <name>yarn.timeline-service.entity-group-fs-store.app-cache-size</name>
: S0 a1 }$ Z0 J2 F( M <description>9 i2 P( R: F/ e% J8 H6 q9 s+ |& K
Size of the reader cache for ATS v1.5 reader. This value controls how many
# M6 p" K) t, L- r% | entity groups the ATS v1.5 server should cache. If the number of active9 k# _* j. t6 a3 ]% w7 d/ e
read entity groups is greater than the number of caches items, some reads* j7 e) d- s& \' U+ M0 V6 I) }
may return empty data. This value must be greater than 0.! N! B7 D7 t' O8 t6 u8 q
</description>
6 d6 | ?& j: y/ o9 b1 U3 u0 O <value>10</value>
- x% W6 h' |3 Q8 w# I </property>5 l4 S& W# c) x
<property>
# O" X5 l* ~. R1 N% G% ~! k. P <name>yarn.timeline-service.client.fd-flush-interval-secs</name>
, Q. A2 A T# |+ } <description>
3 n: E2 ~" b/ I) T Flush interval for ATS v1.5 writer. This value controls how frequent& m6 P8 \4 o7 m
the writer will flush the HDFS FSStream for the entity/domain.
1 \5 {2 v8 x# X, ?4 z </description>
4 w8 _8 F Q+ F# [8 t( ^ <value>10</value>; R. y! A# f" N& i5 w
</property>
* C* [" g/ x/ C z- F <property>& q1 M6 }8 h$ D, N4 k: _5 C k
<name>yarn.timeline-service.client.fd-clean-interval-secs</name>7 d5 y5 S7 I1 b# h% _( z0 h
<description>: |- R: Z- Q* `6 r1 n
Scan interval for ATS v1.5 writer. This value controls how frequent
0 Y' \+ v' K: c# t5 E5 l the writer will scan the HDFS FSStream for the entity/domain.
Q% M% ]8 E U( ^" r' w If the FSStream is stale for a long time, this FSStream will be close.
9 m! C- \* S5 o2 U" D: \4 u' {7 R </description>
- M6 t) W# s2 k! f <value>60</value>) c( J5 O+ t: p3 Q6 k2 D ^: F
</property>
7 m% D6 v5 m* {" M <property>
, j/ j2 K2 i" U5 n6 ~8 T <name>yarn.timeline-service.client.fd-retain-secs</name>
8 J& S8 i# Y0 y! u- z/ C# O" s) L <description>6 z' t K* h$ u2 R% ~
How long the ATS v1.5 writer will keep an FSStream open.+ Z7 ~) z6 Y$ N! P! X( Y7 R$ s7 @
If this fsstream does not write anything for this configured time,
9 i' `( S4 q* J. f$ P it will be close.+ |* H7 e, w0 w" l1 |6 S
</description>
8 @( i: f8 v% I* T/ G& Y <value>300</value>
% W: V) u+ ?+ u; ] </property>$ B2 F# C2 n; \0 Q& P
<!-- Timeline Service v2 Configuration -->
! c3 v3 h% f$ l& z8 E( l" i: @; h <property>$ Y! C4 P( ?$ j) t
<name>yarn.timeline-service.writer.class</name>% ^, e% C9 H. q% h; L! h
<description>
% _) L/ i6 ?: \* S- C8 S$ u Storage implementation ATS v2 will use for the TimelineWriter service.
' R; b$ X" ^' n: n </description>
7 d% K; t4 n$ e <value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl</value>
M; @+ u$ R5 g- G* z </property>
! j5 D+ C Z8 U q$ ^* g0 \ <property>
( ~7 |3 h4 Y; A <name>yarn.timeline-service.reader.class</name>* y" |" V2 z; k6 k/ h5 Q" [
<description>
' |% I+ ~+ G6 x/ W1 x5 |# P Storage implementation ATS v2 will use for the TimelineReader service.
+ g" [: H; @& D- P </description>
% N. t1 ~& j' T& L <value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl</value>2 D B: n1 R& ]4 B
</property>
7 e" o& v8 }7 ~0 t$ o <property>
/ i6 Q! i& s4 I* h. N* m <name>yarn.timeline-service.client.internal-timers-ttl-secs</name>
% w. A6 k+ z9 V1 m <description>& Z8 _2 H/ \& c+ Q$ q* }: b' Z
How long the internal Timer Tasks can be alive in writer. If there is no3 K4 Q4 F$ i% l8 z' a
write operation for this configured time, the internal timer tasks will
1 j5 [- ]1 q5 ?( M: y9 P be close.# Z w' w E& h X9 p9 Q
</description>) ^8 w j# c$ W! a, k% W
<value>420</value>8 V8 B3 L- u t1 t0 T3 @
</property>& c" c1 f$ P+ W4 v3 H' S- L( `- C
<property>0 V, R! L3 u8 }# O
<description>The setting that controls how often the timeline collector$ X4 l' b0 H- N6 y8 K
flushes the timeline writer.</description>/ T: H0 G' N( o
<name>yarn.timeline-service.writer.flush-interval-seconds</name>
4 ]- v( J7 h4 N$ c <value>60</value>0 T+ ~) R6 _, m- Y
</property>1 [+ z) c/ a( c8 I* z& C
<property>( Z' p( z7 A7 A+ b
<description>Time period till which the application collector will be alive0 |, i z9 L+ L7 p
in NM, after the application master container finishes.</description>
8 ?$ V# y; y( _7 H* U$ ~ <name>yarn.timeline-service.app-collector.linger-period.ms</name>; B( [, ]! q- I" {
<value>60000</value>+ w8 H9 `7 n4 w7 y6 w9 [% f
</property>
F; E1 D9 o5 r y+ V. z" C0 S <property>
+ d! b; p$ _: a1 f% ] <description>Time line V2 client tries to merge these many number of+ o6 i! _: H) s6 X
async entities (if available) and then call the REST ATS V2 API to submit.$ q- J/ G5 e0 m# K% g
</description>$ a4 b1 R: j. _8 f4 S- ]" i; q0 M
<name>yarn.timeline-service.timeline-client.number-of-async-entities-to-merge</name>
/ ^' a: O8 a g <value>10</value>
* T5 k1 Z, |7 k, j+ \ </property>1 y! X" N, h9 }
<property>1 F4 c$ z9 f6 p6 E' K/ p, S
<description>
) e1 C0 C: c( U% b The setting that controls how long the final value- N% ~/ |! `/ Y; y! S
of a metric of a completed app is retained before merging into
5 g: j$ s) j7 F the flow sum. Up to this time after an application is completed
( D8 O9 h1 ?( X% A# H$ V. E H out-of-order values that arrive can be recognized and discarded at the
5 o9 `9 a; i0 l; M7 f r9 }, i, q! t cost of increased storage.
0 Q% k" }; e5 c s" y( m3 b, d4 @ </description>( r/ w2 }% L s" U4 W
<name>yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds0 v b! S* a% U0 ~
</name>
& `1 K# M+ Z( O$ w/ t S <value>259200000</value>
0 x) Y3 _/ y# ^8 ^: p </property>+ x! B$ p! A o7 D& q7 W
<property>
) t. o7 ^3 \3 e$ E <description>! a+ @ a/ i) y7 h
The default hdfs location for flowrun coprocessor jar.
3 x" K3 _3 i8 g4 U i </description>
: K- t1 u! g1 `! g <name>yarn.timeline-service.hbase.coprocessor.jar.hdfs.location- u0 U& M( C n1 l/ o" i
</name>
+ }) Q# s1 b" t <value>/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar</value>
+ D6 F6 Y0 w. e1 r </property>* Q' c) w; x T3 R' E8 r" U
<property>
& w( w: C, t1 i% Q <description>
0 F. i% c# _) R. v The value of this parameter sets the prefix for all tables that are part of1 L. J: a7 {) e( j+ v
timeline service in the hbase storage schema. It can be set to "dev."
5 c( y5 u( ~3 n5 ^ or "staging." if it is to be used for development or staging instances.
$ G& b( n, k1 N7 I. w This way the data in production tables stays in a separate set of tables2 k& W: E2 L8 w1 H3 w
prefixed by "prod.".
; b9 A7 `- S( N+ ~ </description>9 |% F3 n! x8 x$ `$ e. h; G
<name>yarn.timeline-service.hbase-schema.prefix</name>- `5 G$ a' O9 j# H
<value>prod.</value>& N1 z$ x, ]! U e4 c9 Q
</property>: |1 A0 J0 z: I0 ~- }3 D% Y
<property>) G2 T/ q6 ~( C2 T
<description> Optional URL to an hbase-site.xml configuration file to be
" p0 ~/ K3 m, W+ j; w+ e used to connect to the timeline-service hbase cluster. If empty or not9 N7 o0 w' P* H1 n4 @: X7 I
specified, then the HBase configuration will be loaded from the classpath.
5 M/ {$ n$ d9 z6 a v When specified the values in the specified configuration file will override* ?( b! F6 u1 K/ e, ?) r9 Z
those from the ones that are present on the classpath.
7 I, h! I1 S! s7 m4 M4 X </description> X1 C% U4 J3 b; F( q
<name>yarn.timeline-service.hbase.configuration.file
$ \( e. s3 i* s H </name>
3 _* E2 Y1 \6 R( w* ? j <value></value>. [/ ^6 J/ T9 l K
</property>8 }6 b4 r: w" R S, G% W
<!-- Shared Cache Configuration -->- o' ^+ O5 o! R. L6 U
<property>
: p3 O# w. Q1 f( s, g/ q <description>Whether the shared cache is enabled</description>& B% ~# a; @1 C6 V; x3 y
<name>yarn.sharedcache.enabled</name>
0 X, l {& E2 ^$ T# P <value>false</value>
+ D, {, _, N: D* P8 R `1 r( { </property>
- U7 E9 \: }/ ` x) {% F0 j3 Y6 M <property>
% F1 K3 J0 r# m& V [" x' v <description>The root directory for the shared cache</description>
8 n* l7 y* |, w% b0 V0 _1 P) P <name>yarn.sharedcache.root-dir</name>
) W( O# K! ^! p' ]. | <value>/sharedcache</value>3 ~* e/ e! X7 ^7 u# v
</property>1 b6 h9 X2 H$ K& e/ G# [
<property>* B% K4 r$ n4 {" q# Q
<description>The level of nested directories before getting to the checksum" g9 `6 t! Y/ Z* A f) ]9 J' b
directories. It must be non-negative.</description># J: |8 F# L/ X* c) x9 `6 F' {
<name>yarn.sharedcache.nested-level</name>
# Z- p ]) c- x9 ~- C! z <value>3</value>
% Q' f5 f: ^8 P: w' j% o, M </property>
8 @/ m; V; {3 N# ~, e5 f+ n& R0 \ <property>
2 k9 m9 Z; Q3 Z9 @ <description>The implementation to be used for the SCM store</description>9 U/ K$ y3 S) ?
<name>yarn.sharedcache.store.class</name>, X7 Z0 c0 R! [/ b9 q
<value>org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore</value>% b! k; G7 w; d$ |# H+ O& d
</property>
/ d, E5 I3 \+ q: j' f) L <property>* n9 L' _! m r# r8 p- K
<description>The implementation to be used for the SCM app-checker</description>7 `4 P# a/ t4 x+ ^) q
<name>yarn.sharedcache.app-checker.class</name>1 e- ^$ Q: s; w/ [3 m
<value>org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker</value>
, P/ n+ n! ], `9 E6 S </property>5 t* B9 z/ T8 {& D& u) E6 Z6 y
<property>
" o0 E: A% W1 ^' | f- p+ R <description>A resource in the in-memory store is considered stale
/ x, Q2 |/ Y3 b+ U8 q: O# K if the time since the last reference exceeds the staleness period.3 ?- } p4 N% O
This value is specified in minutes.</description>6 Z9 p" L2 H4 n' p
<name>yarn.sharedcache.store.in-memory.staleness-period-mins</name>
& ^9 N A1 P; n& d+ y <value>10080</value>7 x* o% X" m* s- F+ i8 D" Z& x+ }
</property>
3 W r! C4 a) }3 Z <property>0 K. @$ l1 }" D+ L
<description>Initial delay before the in-memory store runs its first check! l) l+ h7 X, z$ }
to remove dead initial applications. Specified in minutes.</description>
2 C; G" t, q q* w% W" f <name>yarn.sharedcache.store.in-memory.initial-delay-mins</name>
" H4 c0 I0 P5 h7 p( W3 a: e <value>10</value>+ b0 Q0 R N/ m; v6 k9 c9 {
</property>
1 s: Q1 u& s( _1 j$ x. I% d, ` <property>
! J. Y1 z) `( D4 K1 W4 W7 c <description>The frequency at which the in-memory store checks to remove: i# h' t+ ]8 I* w' O
dead initial applications. Specified in minutes.</description>
M+ H% J$ H5 s; ~1 C# v/ `2 e6 | <name>yarn.sharedcache.store.in-memory.check-period-mins</name>
) z" b# J( @% N' |$ I# w9 E <value>720</value>7 }3 _% x3 A/ T5 Q! c/ u
</property>
+ @7 p$ V5 _8 [# Z7 E <property>
0 R, E" ]8 o2 g' {( @1 I& N, Y/ H <description>The address of the admin interface in the SCM (shared cache manager)</description>+ f/ n! z% V- J5 u, y: h& w; h
<name>yarn.sharedcache.admin.address</name>
, K2 \1 t1 H) ]9 m, Z- r# B% }& I <value>0.0.0.0:8047</value>3 u6 ~. v$ S! v5 h/ k9 C% P
</property>
( _, {$ A ~9 ^0 | <property>
/ d% ]+ H' }, c& s5 h- H; \ <description>The number of threads used to handle SCM admin interface (1 by default)</description>
- H* D& i2 g" J' P$ A% { <name>yarn.sharedcache.admin.thread-count</name>
- O8 C5 f- }% T; H% [ <value>1</value>
5 h% r, a; ~/ J( j4 w </property>( _; y) d* K$ g; g' S
<property>
9 J3 ]# L# v% I. P6 x <description>The address of the web application in the SCM (shared cache manager)</description>
e( g; y8 K- t$ z! k, r3 p6 T <name>yarn.sharedcache.webapp.address</name>
4 ]) p. _. W: o9 u <value>0.0.0.0:8788</value>* O/ E+ \& P- {6 q
</property>9 g" F. P- w( n; B3 t/ |
<property>
. m' y$ ]$ O1 L) j8 E <description>The frequency at which a cleaner task runs.
) M5 M$ b: P, f% z% w( @/ z Specified in minutes.</description>
" I2 {. D& Y4 } <name>yarn.sharedcache.cleaner.period-mins</name>
3 n+ s6 _; {2 G <value>1440</value>! A6 M& Q: P: J4 j( Y+ K
</property>8 m k2 C; z, F' o+ @; V: @0 D8 w1 J
<property>
; x1 y* ^- j8 R' M <description>Initial delay before the first cleaner task is scheduled.0 a7 X* ]1 b' U% b' f
Specified in minutes.</description>
; p. x8 n3 E+ g+ D5 ^# b+ T. q. V <name>yarn.sharedcache.cleaner.initial-delay-mins</name>" q/ L9 d5 F/ w# {' ]$ j
<value>10</value>
1 W) \$ A7 T% i6 H+ n3 M- C! ]2 M </property>
9 H" m ^/ H v0 k, e: m9 J <property>
! ]' Y, H1 j7 ^* l, T3 s <description>The time to sleep between processing each shared cache
6 F$ }, T# @7 S9 z) ~; g' A, y+ O resource. Specified in milliseconds.</description>6 h( g m: ?0 F' K5 Z8 I% C
<name>yarn.sharedcache.cleaner.resource-sleep-ms</name>/ x# x1 a4 z; Y
<value>0</value>0 b4 L* c. d/ }' A$ S% n" u1 ~4 F
</property>) M, A; [3 Y" i4 P0 ^
<property>
# q- i2 k+ l3 _9 q: ] <description>The address of the node manager interface in the SCM
4 n; J; I* _2 b% f( f (shared cache manager)</description>! L: R, x+ }( Q# B$ t
<name>yarn.sharedcache.uploader.server.address</name>
1 @ P1 v% ~7 y+ k& T <value>0.0.0.0:8046</value>8 P. S1 h: v8 H% l* q1 w) F) b7 w' B
</property>
* @0 ~2 H( L. C' M+ Y1 e7 K# W <property>
* r$ o5 x0 K$ e! C" i <description>The number of threads used to handle shared cache manager
+ |( G6 f4 H3 ^. H5 W requests from the node manager (50 by default)</description>8 z+ [5 M8 e+ d1 z8 E
<name>yarn.sharedcache.uploader.server.thread-count</name>- z7 |8 ]; z3 ~8 K8 b" n
<value>50</value>8 V F. K' D2 m$ F. ^
</property># ~8 d8 A7 s/ g9 x2 E6 n/ G
<property>" T: u7 Y5 ? q, C' Q
<description>The address of the client interface in the SCM
8 B$ z9 r0 _% s% S6 { (shared cache manager)</description>/ ~( w7 T# e$ }( ?( C
<name>yarn.sharedcache.client-server.address</name>8 ` u1 D T3 V
<value>0.0.0.0:8045</value>
3 Z$ D/ Q8 `: I1 t' k </property>
/ |3 b# E# }2 G6 f+ Y: m Y' | <property>
) f# R; u& `0 T/ Y <description>The number of threads used to handle shared cache manager
8 O* f' H. n% z5 y. E8 b! e requests from clients (50 by default)</description>
' c. s6 H2 N: ~. f" T: L7 Q) B9 ] <name>yarn.sharedcache.client-server.thread-count</name>
3 `. b3 w9 S0 v( a <value>50</value>' o# _/ S2 a; ]: ]/ f
</property>7 k+ d+ j5 I: x- z/ m* [
<property>
: ~& ?5 u+ r- S) k <description>The algorithm used to compute checksums of files (SHA-256 by/ t Q( f9 b5 j0 H- }
default)</description>
0 L5 P) S A: @% I$ F <name>yarn.sharedcache.checksum.algo.impl</name>! q% b* w% d2 o- p1 L
<value>org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl</value>& y. i, r* p. ^- u0 B
</property>
; a8 T5 V* L+ ? <property>
7 U( p/ @4 j/ u" M7 C <description>The replication factor for the node manager uploader for the) E& p- R' J4 H. Y+ ~8 @
shared cache (10 by default)</description>
! M+ H) I5 E' w* P" J) W$ m$ H <name>yarn.sharedcache.nm.uploader.replication.factor</name>
+ e) x1 O/ ?- W+ v: e <value>10</value>
1 Y) C3 R: L& g8 F' h* j! { </property>" ~/ Y* |. [9 }
<property># ?# l- |; x1 q# j# L j P
<description>The number of threads used to upload files from a node manager4 x1 j: N3 r& @6 `% U
instance (20 by default)</description>
' j& E1 W4 |" t8 S <name>yarn.sharedcache.nm.uploader.thread-count</name>6 D1 z, Y% o$ d7 E* Q" D" ]& R# [. G
<value>20</value>1 F# g+ V+ O% w J
</property>
. `1 G. {3 q1 @. A# ~ <property>
0 T8 a% t% G; y, D$ n <description># l4 b4 O3 s' E6 z" ?
ACL protocol for use in the Timeline server." @4 x" k6 y, {' L& v
</description>
: l2 y9 F; N- @7 N; i+ q- H h <name>security.applicationhistory.protocol.acl</name>- _% S, F$ ^8 ]. q3 n) z7 o1 d
<value></value>. X, D; t ]% m+ Q& a
</property>
- d; }- K- g+ Y# N$ A3 k <!-- Minicluster Configuration (for testing only!) -->
" D3 Z5 ^' h1 ]7 H6 j <property>+ `- ~9 r2 z' g, V0 U4 L
<description>5 i" c/ `$ {: T( q/ c8 p
Set to true for MiniYARNCluster unit tests; @9 t' X# d: A/ a( V
</description>
8 S( u, |6 O0 c7 o" s <name>yarn.is.minicluster</name>
2 O9 _5 D4 m5 r <value>false</value>
9 n' S% n8 | A9 [! @1 { </property> S5 g, k& l8 ^9 S" y
<property>& M+ m# Y; |! m; \
<description>
. f" g2 Q6 U" v6 G) R& l/ t Set for MiniYARNCluster unit tests to control resource monitoring
0 e0 @8 p. P) W7 L7 m </description> Z; Y2 N( F) }% {+ Q7 q, \
<name>yarn.minicluster.control-resource-monitoring</name>' M1 ~ z* {0 C! x" P% W. Z+ x% J
<value>false</value>9 [: J; A( H5 G# v
</property>6 g. y' t3 N; O+ m
<property> ?1 ?. r4 A1 i: {& ?6 u; g1 w
<description>
+ ~: x R/ S9 z Set to false in order to allow MiniYARNCluster to run tests without w: d: F; s! p' J: @4 x$ ]
port conflicts.7 R2 E8 X U, D5 D: p7 s
</description>, U0 ~) y# A0 {2 a
<name>yarn.minicluster.fixed.ports</name>7 y% I w+ l) @! f, P3 L
<value>false</value>* r" v4 @/ v& p+ W, H& X/ a, q0 P6 t
</property>
3 r' h2 F; @0 ^) p P3 Y <property>: ~7 E+ e; M" T; n& E$ l
<description>
4 v% X t, Z7 G1 X9 z$ r Set to false in order to allow the NodeManager in MiniYARNCluster to
! D, g8 G5 V7 _9 z- E3 ]; ^, v use RPC to talk to the RM.
& ^: q0 B6 q4 T/ C* v </description>0 u W9 U6 Y! m1 u! \
<name>yarn.minicluster.use-rpc</name>- G2 y" u6 J; o( y
<value>false</value>
8 O' ]8 }: {6 p1 K0 F5 R, [( b </property>) y8 L* @5 x: g$ C
<property>
, B& p+ C- {& x3 ~ n0 M <description>
5 S# Y& G3 J+ x& x6 y As yarn.nodemanager.resource.memory-mb property but for the NodeManager
- @0 t9 K9 f) w% k in a MiniYARNCluster.
8 D! M. V5 k& B' g9 }- c </description>) d# M' K5 I& d2 H8 w Y; I
<name>yarn.minicluster.yarn.nodemanager.resource.memory-mb</name>
7 w2 s& g, k L# }* m, e4 I <value>4096</value>
& z7 y; ? x7 G4 l; c( S8 s </property>
! n7 W: z: Q/ a- x# s6 R B <!-- Node Labels Configuration -->
' ]& Z! P/ b1 q% U9 j# ? {* T <property>- ~# S1 d# z0 e) k4 I! H8 A$ W& Y7 v
<description>
; O7 t$ R7 F+ ]' K Enable node labels feature5 d% v: Z, d. ]# P! @+ I, p
</description>2 c% a/ s& O" d% u
<name>yarn.node-labels.enabled</name>
" h1 a) B: ?# Y: Z; S/ y <value>false</value>( S, B4 ^) r6 E* R4 o
</property>; K! y4 u1 R$ l- o
<property>
c/ Y8 A' e- _! \8 x$ N% \ <description>; Z* ?5 b9 f* C9 X- ~5 D1 x& l9 D
URI for NodeLabelManager. The default value is
% [& X1 m8 _) Y9 C" e0 \ /tmp/hadoop-yarn-${user}/node-labels/ in the local filesystem.* T6 G' a( V9 O5 B4 J" B
</description>% N) _ o. W) ~2 `3 H
<name>yarn.node-labels.fs-store.root-dir</name>4 m. t8 b0 m( @5 A3 a. S
<value></value>9 l9 V- W, s* q9 P; r( M( Y
</property>4 J8 t. ]: t5 G5 F
<property>
% \/ y1 o/ m# A <description>
2 T: R5 b8 e0 |& L& [( r, L Set configuration type for node labels. Administrators can specify
) S" z4 ]5 ?9 j "centralized", "delegated-centralized" or "distributed".
1 v! X7 G' H7 l) F$ G* W3 D& V% h </description>
J4 Y8 p, n: V3 E* e0 y <name>yarn.node-labels.configuration-type</name>
' a/ y2 O/ v8 L+ ?7 B+ C( M5 q <value>centralized</value>5 G5 T- @0 r$ i7 v# M% C$ U
</property>
9 g4 Y( K) f, L: `( V" ~4 _1 S <!-- Distributed Node Labels Configuration -->* ^* t* L1 k0 O: X8 ?
<property>3 V/ L% N: H, r2 p3 F* E
<description>: v" b* L9 S4 |5 m
When "yarn.node-labels.configuration-type" is configured with "distributed": ^. n, V4 ?" c% \) S9 w" X$ A% o
in RM, Administrators can configure in NM the provider for the9 e6 X1 k2 ]) ?
node labels by configuring this parameter. Administrators can
" ?, A8 q. r' d$ @4 _) \: m3 V6 o configure "config", "script" or the class name of the provider. Configured
4 ?8 p: O- g. F4 l class needs to extend
$ `5 X* O7 }5 _4 m$ h" g+ [9 w org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider.8 ^7 i `% p% p5 Y; W: s- i" O
If "config" is configured, then "ConfigurationNodeLabelsProvider" and if
/ Y6 _" d) ?# S, G5 T# e; i" f "script" is configured, then "ScriptNodeLabelsProvider" will be used.
' P6 ^! A0 ~, ^ </description>
5 `% i! @9 U$ J0 F4 T <name>yarn.nodemanager.node-labels.provider</name>7 c7 [$ b' E1 d6 S! Q' p
</property>& T: f8 ]. d4 E3 m2 @) p
<property>. N, {) I8 ^# M" |' k
<description>( j J' i7 T" C9 b" G, D8 w: L
When "yarn.nodemanager.node-labels.provider" is configured with "config",0 j2 t; e) z1 l8 T
"Script" or the configured class extends AbstractNodeLabelsProvider, then3 n0 B9 |$ F9 G2 r6 V
periodically node labels are retrieved from the node labels provider. This
; w- L& ~+ ~' N7 |5 x9 t configuration is to define the interval period.
, I' |# i( t1 I5 K If -1 is configured then node labels are retrieved from provider only8 Q1 K3 {% w9 V1 @
during initialization. Defaults to 10 mins.& A9 ]- Y( d; ^$ ]9 v5 `' P
</description>+ E/ a$ O: E% [
<name>yarn.nodemanager.node-labels.provider.fetch-interval-ms</name>
2 `. a I' C* y+ H8 [! f <value>600000</value>4 y" S o7 H/ Y& h
</property>+ |$ f2 U5 _7 H6 [* }
<property>
1 C+ d' Z" Y& {/ g, W: \. ~ <description>6 M @4 X- T1 j- F5 r, V
Interval at which NM syncs its node labels with RM. NM will send its loaded
1 U- w" Q* l8 O2 l+ J* L, t0 M labels every x intervals configured, along with heartbeat to RM.
% @7 h( U ^7 p% P& g& D# I </description># }6 U- m, T/ L( C% S6 c$ V/ o0 T
<name>yarn.nodemanager.node-labels.resync-interval-ms</name>
& |. }5 ~& Q" A <value>120000</value>
" r( q* V: @1 K r- p </property>- ?, `- i4 r* i1 K
<property>
Q9 w4 c" h$ Y- b$ [* c9 Y" C4 A( H <description>
! @* E* P E/ K/ |* ` When "yarn.nodemanager.node-labels.provider" is configured with "config"3 R: R6 a x% g8 x R* s
then ConfigurationNodeLabelsProvider fetches the partition label from this
8 z4 h5 }: j1 t4 z parameter.6 A# G% o& Z. g' d1 l
</description>7 n' d; _2 G# D2 J1 o0 {# ?% ?9 f
<name>yarn.nodemanager.node-labels.provider.configured-node-partition</name>
$ `9 I; T1 t3 y. A2 W% B </property> P: w" {" p% d, O, f9 p
<property>
5 {& {4 r: M% ^" q <description>
& C" y k% a6 D5 k When "yarn.nodemanager.node-labels.provider" is configured with "Script"! |. t) S7 A4 r4 K: V: b
then this configuration provides the timeout period after which it will
: j9 c% y" }! k interrupt the script which queries the Node labels. Defaults to 20 mins.5 e! O" G8 S4 o3 \1 X. Q1 O$ u
</description>- M4 ]- M" |* G& \; p+ y7 M
<name>yarn.nodemanager.node-labels.provider.fetch-timeout-ms</name>% Y/ P( P; x$ J" a
<value>1200000</value>( f4 q s$ K2 ^
</property>6 k' j, }$ s: ~* U) X
<!-- Delegated-centralized Node Labels Configuration -->/ x' U8 z5 \8 v2 O& r# |6 V
<property>0 l' @% |. j% v. \* O' k
<description>
7 L$ P( t3 e- e" P* Y When node labels "yarn.node-labels.configuration-type" is
5 W8 V9 k. s+ } of type "delegated-centralized", administrators should configure G/ m6 b; k+ T) O5 R- t
the class for fetching node labels by ResourceManager. Configured
# M/ T( d/ Y, P" P class needs to extend; ~" J9 w/ S3 c) f
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.. g% Z0 a; M' E4 N. L/ V
RMNodeLabelsMappingProvider.2 V. U" K' N7 o7 J% @, B/ q: M- v0 E
</description>
+ U3 i2 u; P- J7 R9 b" O8 ?* O% p <name>yarn.resourcemanager.node-labels.provider</name>% O$ G7 d- C; f% X! S8 p
<value></value>2 j, m5 V! Y, G/ h3 j' V, t" @
</property>, {/ a2 [$ r0 J) ?) t9 j/ O& B' M
<property>
( P, V M* E9 b2 c6 s. h/ Y, ? <description>" G- O" ~$ C0 D
When "yarn.node-labels.configuration-type" is configured with
; l1 i* D2 V& e! i, ^ J "delegated-centralized", then periodically node labels are retrieved
& R+ C( P ~ N from the node labels provider. This configuration is to define the
; I/ _* t# ^: T) v8 _8 _* f interval. If -1 is configured then node labels are retrieved from( b8 }$ A& Z, N* X4 x' {; S+ P9 Z
provider only once for each node after it registers. Defaults to 30 mins.
4 M: r. u ?" } P0 y </description>5 u p# B# w( l
<name>yarn.resourcemanager.node-labels.provider.fetch-interval-ms</name>
2 k4 f" b8 w% z! l7 t/ w6 Z4 e/ g7 @ <value>1800000</value>3 F7 t7 p. p7 F; i' e
</property>
% W9 p; M3 G0 [, g <property>% d) Z2 t& v$ L3 W5 H* w
<description>& A0 A( c# t$ [# f5 [, e
Timeout in seconds for YARN node graceful decommission.
- F! I' n5 S) _, ?# m3 Q This is the maximal time to wait for running containers and applications to complete) T1 u( ]7 x# p9 N. _
before transition a DECOMMISSIONING node into DECOMMISSIONED.
; N! i% a8 d, r </description>* A+ j: |8 g, h1 D( K
<name>yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs</name> S% D: q1 }- \) s: q
<value>3600</value>
- u5 I5 P/ c% g/ ?* d </property>5 T: x% u' v; ~: F! v& c$ P! }- B
<property>
/ B+ M# K6 O8 b0 Y3 Z5 b <description>0 m# g! b. d; E- ^3 K
Timeout in seconds of DecommissioningNodesWatcher internal polling.
8 X7 Q, n7 n/ Z( s& n8 U </description>) A9 d/ M+ j: N9 i8 W+ h
<name>yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs</name>1 X' L7 B. ^: z6 C
<value>20</value>' f- Y+ q8 n# t2 k% a+ F
</property>, w0 n, I" j ~+ A% D: z
<property>
e7 W( o A0 k4 k& K <description>The Node Label script to run. Script output Line starting with: Z- D: @2 a/ @. o
"NODE_PARTITION:" will be considered as Node Label Partition. In case of
8 ~" f0 Y% y9 h* V multiple lines have this pattern, then last one will be considered
7 }& q; b! P+ ?+ X3 ~ </description>
! N t; W; M1 B& b7 E: |( f <name>yarn.nodemanager.node-labels.provider.script.path</name>
: [+ P2 p' {- [# ~! i* ]1 } </property>
7 k/ C5 S2 s$ C <property>7 \! b5 M }* x: @7 w) m& Q3 L
<description>The arguments to pass to the Node label script.</description>
' l. y( R5 x' M% Y5 b <name>yarn.nodemanager.node-labels.provider.script.opts</name>5 l X) B. ^! w# I: z* E
</property>
8 D$ W3 @: ?: I6 g <!-- Federation Configuration -->
5 ^! ~- O6 \1 l% H <property>& o2 ?+ o2 _4 V9 q1 G
<description>
7 Z8 O1 K4 F: n; h' D Flag to indicate whether the RM is participating in Federation or not.! H, K3 P2 D& n$ B/ ]
</description>, v; m9 O; E( v8 u7 C
<name>yarn.federation.enabled</name>7 C3 E+ x; N7 X0 P, t: q" X1 [
<value>false</value>
/ }9 Z$ O' u( }5 ]6 k9 T8 P$ C0 h9 H </property>& [; f d2 x1 y
<property>
* F5 I4 w% p1 e8 T4 m2 ?! E <description>
; `; X& d1 @; V0 [! ` Machine list file to be loaded by the FederationSubCluster Resolver
1 z) e# E+ U) g1 M6 B9 t </description>
4 e1 ~5 j# u f$ b) t( t <name>yarn.federation.machine-list</name>+ ^. P! n% I! o" D* f) [! U+ L& g
</property>! A& z6 G& M/ S; _$ z( q: i. F8 O
<property>9 s y$ {/ W2 P& K
<description>
# u% ~/ V2 F$ ?, ^& d: W Class name for SubClusterResolver
1 B, x- {# {$ ?5 j2 | </description>
7 x5 h- c9 V& } <name>yarn.federation.subcluster-resolver.class</name>. u: k1 I* e& J
<value>org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl</value>2 x, Q0 G8 q% H7 N/ h% u7 Q' c
</property>" t$ Y1 J& `- S: T' j- e3 Q
<property>) @" O c/ k6 ~: K3 b; R9 |6 p8 u
<description>: Q- t; ~; `, C' g3 l6 h
Store class name for federation state store
0 w, c Z7 V$ c& k </description>3 H, R: b( Z3 H# x- b. q0 V
<name>yarn.federation.state-store.class</name>
; w H9 T. H; W+ K7 z! x2 \% V& i. v <value>org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore</value>- K" p8 b- C' L- O+ b6 u
</property>! m6 S1 Q h1 G& L5 T
<property>
! w8 ?) M! m( v* C( m <description>
/ D4 d+ z! ]$ S9 k- g- a/ d+ g The time in seconds after which the federation state store local cache) [; w) @" }% G5 E% H! U3 b
will be refreshed periodically
# [* J/ F. Z! f4 s K0 q2 f5 g7 b </description>
1 [) o7 F; B' Z7 K# f" w; k <name>yarn.federation.cache-ttl.secs</name>
, M8 ~5 i7 D' h3 k8 F" \3 k. R <value>300</value>
0 V/ k3 m6 A7 v& W! N </property>
2 V% z& W1 R& D% B( R& X- ]4 V <property>+ D4 z1 @% L) X) f
<description>The registry base directory for federation.</description>
2 G, P3 G! c# o7 O" e+ C <name>yarn.federation.registry.base-dir</name>
- B7 r# U; S7 l/ t# P <value>yarnfederation/</value>6 W' j. _- P- @9 f% B# g3 U0 H
</property>
* \2 M ~6 \" A <!-- Other Configuration -->
l# |: F9 U0 Y/ N2 W; G <property>* l$ d' P/ b# a8 h G
<description>The registry implementation to use.</description>9 k( I' i' r3 d( B$ R
<name>yarn.registry.class</name>
+ |6 u0 \- X' h& |9 ? <value>org.apache.hadoop.registry.client.impl.FSRegistryOperationsService</value>
6 |, o& z- S* w% w9 {) a1 l </property>
, `/ D8 o; k* N# E; j <property>
9 X6 a7 n1 R5 p# H; h8 M& Q <description>The interval that the yarn client library uses to poll the
$ b$ ~! @$ p) k completion status of the asynchronous API of application client protocol.
2 Z- v; f W- l$ I </description>
9 v+ C& u) `4 C. V. C <name>yarn.client.application-client-protocol.poll-interval-ms</name>2 c/ x7 p. i* w$ p$ D
<value>200</value>4 V9 A- C* a2 l. K
</property>" [+ `# D+ b1 s5 k" @
<property>$ P3 Y. H. q( { c& P( M& N8 r
<description>
2 G! z8 H- e) X. z* r0 ^/ u* K The duration (in ms) the YARN client waits for an expected state change2 {7 o: b0 L% e. n2 Q
to occur. -1 means unlimited wait time.
5 z( i' p$ N- K( Y </description>, B# f9 w9 E4 B5 v
<name>yarn.client.application-client-protocol.poll-timeout-ms</name>9 j3 [' J% @' v8 z$ L* [2 m% b' T
<value>-1</value>
& I& s$ M8 G! V! x' ?: l& L" P </property>4 v L3 H9 m. N+ R' ^/ k( r
<property>1 g4 ^0 w1 _& k" @1 O- M
<description>RSS usage of a process computed via$ z8 ? v7 t, `* P
/proc/pid/stat is not very accurate as it includes shared pages of a4 X, Q% o; Z' ?% o% r+ Y2 \
process. /proc/pid/smaps provides useful information like
) O7 w+ C5 o& V+ \0 T# N Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used
* X& I" B% a% \$ o for computing more accurate RSS. When this flag is enabled, RSS is computed2 _# s4 J# N+ M5 R) ^( j
as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes# J& ]; D; K# Z7 i' p) ?
read-only shared mappings in RSS computation. & c, o1 T% s5 |
</description>
9 W/ @6 C. \) S" A1 j <name>yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled</name>
* M+ w6 b& K' J2 p2 a <value>false</value>+ a( i Q" W/ m* h1 h2 {( n' E
</property>
- R9 n, D8 \1 |+ E <property>
( I' }8 \% w% }; T! ]: J/ s, I <description>) p# ^5 `% s; M2 e9 l" x) }0 J
URL for log aggregation server
r7 X! R' k+ f9 \) o, v </description>% I& b5 ^4 r, y) q- |
<name>yarn.log.server.url</name>
s7 B# k. Z7 Z3 H2 w0 |$ f <value></value>, F4 P1 J' k( n8 o% a9 S8 K
</property>7 y- X4 F- n g2 D
<property>
1 f3 d$ b9 o8 S# B& d0 m <description>
* X* ?2 ~ m5 x# n6 V+ w URL for log aggregation server web service4 f/ T+ \/ G' ^% c- M7 c2 N6 H
</description>( p, f9 g9 T/ s: _- W) @
<name>yarn.log.server.web-service.url</name>
( E1 d. W0 S) o <value></value>7 s# O6 n. V k' i6 N
</property>
# T2 R1 b0 e* {, L+ s9 D <property>* S. ?; t0 i. P: _ b, O% }4 u# `
<description>' r3 r2 Z4 A7 g
RM Application Tracking URL
% M3 D6 p6 ]; i# h </description>
# W! _* O g, z# }5 c <name>yarn.tracking.url.generator</name>
& Q6 [6 y- { T$ L$ ^; u <value></value>
% B0 W9 e A8 a' r( @ </property>
( O1 \9 u. J. c1 S; e6 w; j <property>) W1 W3 c! \* Y" m2 `9 U
<description>
, w5 l# X! M& ]. x2 T* q ` Class to be used for YarnAuthorizationProvider/ |+ W6 s/ s/ `, O
</description>
2 a: j# s; r; f# ] <name>yarn.authorization-provider</name>. G+ w" k, O; q, E7 {1 W
<value></value>5 L# o4 m6 [% E! W
</property>
' {- [/ S, {9 @7 u% w <property>, z4 z( i* E* M' B q+ J# B
<description>Defines how often NMs wake up to upload log files.& O% X/ u5 ^/ j% B6 S# C1 l
The default value is -1. By default, the logs will be uploaded when
" h+ ^1 v: S z/ n; _# E the application is finished. By setting this configure, logs can be uploaded
7 r/ C# i' S) J6 |) A _ periodically when the application is running. The minimum rolling-interval-seconds
2 `0 ]; H. g% |$ f! [$ z2 P& n# c7 L can be set is 3600.
1 A7 L1 ?# ?* t& z6 @ \# Y </description>
; G( U" t; i9 G4 o3 V! Y) K) B <name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
% d/ h# E5 n# F: [4 K6 |: M4 g <value>-1</value>9 p7 e5 q: K3 L: t( ^
</property>
. I7 \/ C1 s2 A3 ~' s0 J; L3 @: P, D <property>* s8 Q/ h r7 `* A; u
<description>Define how many aggregated log files per application per NM
0 `( |: B& }# }/ w1 X5 w we can have in remote file system. By default, the total number of
( t: L# A2 ~# ?2 A' {) z& [ p aggregated log files per application per NM is 30.
" H5 V; |9 j8 H8 [; e </description># B- A- d- ?9 `5 E
<name>yarn.nodemanager.log-aggregation.num-log-files-per-app</name>" x$ w. d$ o7 X1 d9 y
<value>30</value>
% I& H0 P- D" M2 L' O4 S. t </property>
0 g# p+ f- d# h <property>
1 [ C! o7 Z8 U: J# w; B9 \ <description>
4 B& e) x' G" Z7 q Enable/disable intermediate-data encryption at YARN level. For now, K+ A0 _9 J) y# ~4 b4 Q0 v- b
this only is used by the FileSystemRMStateStore to setup right
0 k3 R/ A# E( v0 x: H# K, W6 b file-system security attributes.1 s; _ w2 k. }% x. {5 W
</description>7 W7 I" k3 C5 A2 n3 ]& E
<name>yarn.intermediate-data-encryption.enable</name>7 y/ U% b1 ]. t" Z! ?
<value>false</value>9 ?9 l! T* Q5 h
</property>
2 q: r/ ?" `! i% |+ ~* |$ } <property>
3 C% `! L, ~; t4 j7 G* I" A3 R) o; O <description>Flag to enable cross-origin (CORS) support in the NM. This flag- Q1 v0 K, k( F- K1 j4 l
requires the CORS filter initializer to be added to the filter initializers: Z' g4 g# U" Y& h" i5 L* n& @
list in core-site.xml.</description>: L* \4 Y/ G' H* [- t8 t
<name>yarn.nodemanager.webapp.cross-origin.enabled</name>
; Q* n, q- ], p+ P8 D <value>false</value>
' m3 J. K$ f( h( ~- k% z0 B </property>
j9 U" c( c2 e R( r7 E) s <property>
) E0 p6 `" x2 a4 i7 ~: d <description>, d3 a9 y) J+ ]; A
Defines maximum application priority in a cluster.
$ | m' z4 j/ N8 j% R If an application is submitted with a priority higher than this value, it will be
9 a9 m: x+ p' U) J1 {- r2 \2 U reset to this maximum value.
9 X1 s/ [5 O8 Z6 N. n </description>7 I1 `4 P& z7 O8 j
<name>yarn.cluster.max-application-priority</name>/ `: I; E0 L" R
<value>0</value>
9 ~* U% l6 ]5 J& R3 C. I </property>9 J3 P0 N: o1 e" C( l, j
<property>
Z) u& X) @/ Q9 N8 {9 I$ h <description>
6 T. K4 l6 I7 M* E- X$ ?; O0 V The default log aggregation policy class. Applications can8 |' m" R" F7 J" V7 p
override it via LogAggregationContext. This configuration can provide8 o- r2 R8 s0 O3 K
some cluster-side default behavior so that if the application doesn't
" A- c5 \& B) e% p2 n specify any policy via LogAggregationContext administrators of the cluster$ P* m8 h9 M. _6 Y& w) m
can adjust the policy globally.! K; l. C/ q( x# c) Q/ c
</description>
$ r& N; F" h9 F1 o; Z <name>yarn.nodemanager.log-aggregation.policy.class</name>
. H% D4 V+ v C8 X <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy</value>
- I/ P' H" o4 ` </property># u( }. u5 }4 c% R9 G
<property>
& d7 r I" w" D <description># v% ~3 F& ~8 Q) w6 ?2 F
The default parameters for the log aggregation policy. Applications can
- s. ]- @% |- ]; q# {& D/ h! l override it via LogAggregationContext. This configuration can provide" I" U" H+ X% Q, X( K8 Y
some cluster-side default behavior so that if the application doesn't; g! }4 z# u$ Q# g* I9 D+ j
specify any policy via LogAggregationContext administrators of the cluster
- z0 V( }0 b! P4 @ o8 M3 M, W can adjust the policy globally.
- l' b. r) X z& B </description> J4 g5 r3 T! _% B( ?
<name>yarn.nodemanager.log-aggregation.policy.parameters</name>
% b/ G7 b5 k$ q; w <value></value>7 W% c# f% l- Q# Z. g: z# D
</property>
1 v' y$ i% W% b+ K <property>
& A0 D' C- {% G/ ~! R <description>7 Y3 b# U8 J% Q- u
Enable/Disable AMRMProxyService in the node manager. This service is used to
1 I) _% B5 U/ z5 O/ e) R0 ]' {/ c intercept calls from the application masters to the resource manager.
9 ~7 T5 q/ w: j! K3 \0 n6 v </description>; ^/ [7 S( a+ u# R4 F
<name>yarn.nodemanager.amrmproxy.enabled</name>5 S3 p. v' h5 P; H+ C3 j
<value>false</value>! d `3 K+ L0 j# z
</property>
2 R8 @0 q- W; r <property>! ?2 |% {& `* w' t" ?
<description># W; f5 H. k; W+ @' d3 l$ k
The address of the AMRMProxyService listener.: v1 h+ x5 J# V. b+ i
</description>. v( L, F) X* E q7 U
<name>yarn.nodemanager.amrmproxy.address</name>
7 u/ ?# W5 g- i$ l8 y <value>0.0.0.0:8049</value>
0 J, o; u. k. |8 K </property>9 g0 \; R: Z; ~ b
<property>1 _ q8 ]* M, \3 |
<description>
3 N) `4 f# t# ]3 }/ k; c; S The number of threads used to handle requests by the AMRMProxyService.; A- L& b/ t) c$ B% U7 [: C+ V
</description>- K3 q2 X8 g _" r* H
<name>yarn.nodemanager.amrmproxy.client.thread-count</name>
. j3 K. P W( H <value>25</value>" ~7 m* e- C% G9 T
</property>% p: t& l! y( q# L4 o& L6 H, c
<property>
, S4 P3 d) {8 E+ G2 U! t6 c <description>8 o* p2 | y6 W! W3 i
The comma separated list of class names that implement the+ |. T7 [6 m; l
RequestInterceptor interface. This is used by the AMRMProxyService to create6 v, C6 p( o6 E$ N% k' q
the request processing pipeline for applications.! f, K- N1 Q! v- @2 r) y/ ]. w
</description>5 N( E+ k! Z# h8 G
<name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>
1 M- z" C0 J9 a7 G, w) E <value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor</value>
/ A( k0 L' {, P! O- S2 e6 K </property>
! c0 I! x2 y) N4 ?8 [ <property>
2 u7 e9 p7 X3 f9 r, n1 A3 {0 C5 p <description>
+ ^6 V: V% ^. y2 G$ S$ M Whether AMRMProxy HA is enabled.# |; {. p* X! h2 W+ [1 v
</description>$ t+ t; y, ]: t# G: z
<name>yarn.nodemanager.amrmproxy.ha.enable</name>
5 B( e8 G0 w, J; |9 i; R <value>false</value>& V' m' i1 E1 C( w$ ~( r
</property>
1 L; T D* F+ m/ q <property>
7 G: O* c% P( n/ l2 n" D4 o <description>
' j- W! A+ k, D/ Q$ c7 S" O Setting that controls whether distributed scheduling is enabled.
2 A; M. t0 @" ]/ _& }; z( Q </description>6 h2 y8 J: H! E* ^8 O" x
<name>yarn.nodemanager.distributed-scheduling.enabled</name>
2 C. S' F/ m; S4 V* w <value>false</value>
" T! E) g1 V" X0 f s" V </property>% ]; J% m( d) x: E m
<property>0 w! P9 i* d: c& X
<description>
$ ?/ ?; H, R* l- V& u0 P* g! H! m Setting that controls whether opportunistic container allocation
" e; }5 Y2 `. c is enabled.) X! X9 Q, w# O
</description>7 L$ ]3 Q$ c5 d0 t5 q. z8 T5 R
<name>yarn.resourcemanager.opportunistic-container-allocation.enabled</name>
+ l: a: ^2 h6 T* x) j* u1 E) B4 F! g <value>false</value>
7 M; k0 P& s/ c( ~! x; i </property>
2 T9 \4 S" I' v/ J <property>5 @. e. x) h5 X2 V
<description>. [! r$ V# P. a
Number of nodes to be used by the Opportunistic Container Allocator for) ?: J2 b2 ?* ~4 r5 g6 P; ?
dispatching containers during container allocation.
1 L# z' q+ S: O8 X6 m: y& ^ </description>
5 c5 r9 {7 k% t! p) T) H" y1 k <name>yarn.resourcemanager.opportunistic-container-allocation.nodes-used</name>: W6 Y4 g! @4 A- M- F
<value>10</value>2 u$ S" f" ^* d2 E
</property>
+ }- m; p" Z/ a" ~- I* f5 q7 m <property>; m3 ]- W) X2 G6 O4 g
<description>) M& J5 m* K; Q Z4 g$ p
Frequency for computing least loaded NMs.
( z2 N/ s# s! y% O+ e7 O( ?, ?6 J </description>6 |0 Q8 K& A& W# V( \
<name>yarn.resourcemanager.nm-container-queuing.sorting-nodes-interval-ms</name>
6 A3 w. E" I N <value>1000</value>7 E7 X/ }6 I9 } i
</property>6 f! F/ B( i/ _
<property>
- y& v7 J( U; c6 p6 ] <description>
9 k; A5 l6 e/ n) ]) t Comparator for determining node load for Distributed Scheduling.( _5 Y, q( `# J$ n$ w9 R
</description>
0 v3 P9 R8 P3 b! c B" v; H <name>yarn.resourcemanager.nm-container-queuing.load-comparator</name>6 S' e( C$ q3 V# y Z8 L4 n
<value>QUEUE_LENGTH</value>
0 m+ O, W; E4 R- r& e, w: z </property>9 S6 z% q( p! j5 ]
<property># j1 o; ]- O; e) O- B
<description>
1 U7 Z+ s5 C: ?/ V3 E6 U Value of standard deviation used for calculation of queue limit thresholds. ^7 y8 D' J* J! u: z
</description>
: }7 G. u7 D/ L* j' n6 D( _ <name>yarn.resourcemanager.nm-container-queuing.queue-limit-stdev</name>; T) P; l! I# E) D1 L
<value>1.0f</value>
0 p E% U% r' l& M' Y0 p </property> x" i) L7 T) P$ W
<property>$ X2 T6 }+ x+ N% x2 k
<description>
7 n8 r% b2 n" i; V6 A0 U Min length of container queue at NodeManager.
. V7 g6 [9 y: B. K; T$ S! f </description>. K' Y+ D# y, L C h' F2 j
<name>yarn.resourcemanager.nm-container-queuing.min-queue-length</name>" { \ y7 H4 e# E0 E) B7 `3 m# p
<value>5</value>9 Y! V' |8 n3 V3 g
</property>
3 L# w: M5 P* O0 M6 u3 u <property>! x$ L% I+ ^2 ?$ ?$ n" v( Y
<description>0 b' n5 D) x$ y
Max length of container queue at NodeManager.2 Y8 B* p! X; V* d
</description>
1 G. d$ _& I5 x6 K/ I8 G6 ? U <name>yarn.resourcemanager.nm-container-queuing.max-queue-length</name>+ @- K4 T3 u" o' t# x
<value>15</value>
, {: ^, @6 o' P- T4 D! v E </property>
0 N* V* u7 |( ^# b <property>
( _% D u; F) P S7 o* u2 z ~ <description>5 p8 d, g. X- n/ i3 J$ F; e
Min queue wait time for a container at a NodeManager.
U' A6 E, `) t' f$ x" r! N; y </description>
: x5 K- g5 `/ }! L <name>yarn.resourcemanager.nm-container-queuing.min-queue-wait-time-ms</name>
7 E, N( {; L' n# d <value>10</value>
% n' s$ y$ S6 O9 q+ V( N1 Z </property>
6 K; X+ L* H q. H9 i <property>
* ?8 A8 V7 V! c5 e7 I <description>! D- L) Z C! p
Max queue wait time for a container queue at a NodeManager.
. j# m$ C6 x3 ^& k </description># Y# u- \. v. T+ o. Q" J( |$ N
<name>yarn.resourcemanager.nm-container-queuing.max-queue-wait-time-ms</name>- `9 y5 C" Z+ _4 K' P( o( H9 N
<value>100</value>/ M2 d" O) I& l. c3 d ]$ ^
</property>
9 Z @! @' K+ u) `7 i <property>
7 j, X) I, g% t: g' k <description>9 G2 N7 D3 o# m, V7 B( e! p C- S5 ?
Use container pause as the preemption policy over kill in the container! H w1 A6 S7 h
queue at a NodeManager.( S$ c+ d G2 f" m- Q: E2 B2 q
</description>
- E# ]' Z% B9 ?) {5 q <name>yarn.nodemanager.opportunistic-containers-use-pause-for-preemption</name>
7 f: j4 y$ y( R$ T, { <value>false</value>
" c7 T9 T7 i" W2 n- u, e; J </property>8 H" b/ ?) G5 v3 _6 g, v
<property>0 {- n3 \( m- T& v
<description>
) V3 U7 |- T5 \% ? Error filename pattern, to identify the file in the container's- s: n' J, _# E( G+ p) J7 O
Log directory which contain the container's error log. As error file
3 n/ X# A- u* L2 ?7 @ redirection is done by client/AM and yarn will not be aware of the error
; m, z1 h+ O( Y% j' w file name. YARN uses this pattern to identify the error file and tail
4 _& c; P9 k- T7 O+ n) a% ` the error log as diagnostics when the container execution returns non zero
9 C. k; f% q& h; f value. Filename patterns are case sensitive and should match the
# z6 _& V ]( P( Q4 N specifications of FileSystem.globStatus(Path) api. If multiple filenames
0 a5 z7 G6 N& c4 H+ { matches the pattern, first file matching the pattern will be picked.5 Q- @6 F) y9 Y7 F
</description>
, r8 P7 |, r( t7 Q; ] <name>yarn.nodemanager.container.stderr.pattern</name>
% L" j& k% b; K8 T c <value>{*stderr*,*STDERR*}</value>
! A4 ?5 o% R ^8 J4 R7 U; v# i </property>
3 Q3 ?- ~# d) _) m* ]6 L <property>
* d# G; S7 v' G <description>
) }6 n9 P2 i, C: ^, V3 ^ Size of the container error file which needs to be tailed, in bytes.8 G8 F+ i8 x8 D9 }
</description>
' t. B" r) v2 P4 P. Q( E# Y <name>yarn.nodemanager.container.stderr.tail.bytes </name>
" ^6 @! d% K1 ?; S5 C6 c' z" t7 k <value>4096</value>" [" C( Z- {/ _! V, W/ i
</property>
9 a$ ^ ?) t7 c <property>
1 q6 S% K6 e ]: E5 q <description>
) U& M; { H3 v$ h$ Y: E% w Choose different implementation of node label's storage+ K0 x, z: n5 Q0 c! @1 B# N
</description>* L# b2 R3 P. y% T4 F
<name>yarn.node-labels.fs-store.impl.class</name>- Y+ ~4 s3 {& k! N+ K% h. o
<value>org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore</value>
) z! ^6 P9 ?* g </property>3 K r" G1 _2 v
<property>
) T c! E6 C6 A" w <description>
" E- P- S1 i5 t/ E$ d+ j! [ Enable the CSRF filter for the RM web app6 Z# l, U# b) M2 c9 W2 H
</description>) m7 t) D x: t6 f
<name>yarn.resourcemanager.webapp.rest-csrf.enabled</name>9 p- W7 T- l ?# ?
<value>false</value>( o1 e, e9 Y4 C1 R
</property>
* N9 z% F; i# j5 M <property>+ q' R" H& ]! C1 P. m9 I2 K
<description>
0 G' B# Z! X* [3 h+ k Optional parameter that indicates the custom header name to use for CSRF" F& k' F. K/ ^" l
protection.) j& B \0 H! u) Q# w& o; u
</description>
/ Z" G7 n& j7 e2 X: O$ _+ @2 @1 c <name>yarn.resourcemanager.webapp.rest-csrf.custom-header</name>
) e0 \) g! t$ u" M0 h/ M <value>X-XSRF-Header</value>
: T/ J. Q+ I7 z7 }2 p2 o. Q </property>
9 R; Y5 h+ d8 b7 V( ] <property>
8 _8 Y @0 P% ^! y0 H3 { <description>
3 G, L. b, \! R9 R Optional parameter that indicates the list of HTTP methods that do not4 ^7 q- X+ E/ R0 K! b+ U' g$ e- R( ]
require CSRF protection- f/ ]) M- b9 M) \
</description>
1 `: A8 X1 Y6 R2 T' z$ ~* D3 B <name>yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore</name>1 T! D, C$ K, D' W8 I, Z6 Z
<value>GET,OPTIONS,HEAD</value>
0 b7 M& W4 u. y+ a0 I5 u1 \ </property>* J0 N3 D6 m T' x
<property>
1 g8 I5 p0 j$ j4 u2 p* R( q% I <description>) {; q* B7 l5 F& ]4 X, Z. K
Enable the CSRF filter for the NM web app: ~/ b+ f6 W/ V2 ~- P* r. I5 O
</description>
4 V' l, a3 t& k# S5 `8 K& C u <name>yarn.nodemanager.webapp.rest-csrf.enabled</name>
: b* j k! ^( G# r/ \0 \5 R <value>false</value>
( t5 ^3 l7 V) w </property>6 f6 w3 v3 B6 V
<property>
; \) ^8 M$ }) g) Q <description>
( R( J& N: [% r8 z Optional parameter that indicates the custom header name to use for CSRF) [2 T# q; ~. ?" P/ g- _" N2 P
protection.$ i: T( [7 d, h. w6 e: U
</description>
% `0 o: k5 I) c4 D/ s <name>yarn.nodemanager.webapp.rest-csrf.custom-header</name>* l2 x) v5 G/ l3 g3 ]- @" o
<value>X-XSRF-Header</value>" l* ?# H, w2 Y" }) p
</property>
; q1 N3 _; ^# a <property>
; D9 E" h$ \0 C5 [0 @1 [+ k <description>
* i3 L( M, Q! U9 R5 ? T Optional parameter that indicates the list of HTTP methods that do not
# b, N$ d9 p4 H( U- U require CSRF protection
: x% I# v6 p7 q* z/ c+ h8 U0 l% m& y, E </description>
; u( y5 v* V2 r. o& r; @& @ <name>yarn.nodemanager.webapp.rest-csrf.methods-to-ignore</name>
: A- i& q! {/ u A8 p7 r <value>GET,OPTIONS,HEAD</value>0 [7 p9 [6 T+ o! {
</property>
2 K" m1 Y) G' j" i& I, y <property>1 [0 o3 p' ^7 \
<description>
0 ]: L& ~ E- o: |+ N# I: I% S! m The name of disk validator.8 Q! j9 ?/ \8 g: N$ X- H' w
</description>
% H0 G" g7 H, P5 n# t E <name>yarn.nodemanager.disk-validator</name>
& A7 Z: @: f3 b: C& a8 [ <value>basic</value>& R0 Z- o2 c0 P
</property>
7 d h6 c$ F" m+ } <property>& C5 `1 ^, Z- s$ r3 ?* O, B% g% b% V
<description>
0 O+ x6 b* R0 ?9 @. ]6 ^" p Enable the CSRF filter for the timeline service web app
# |% s$ T0 l$ ]% d </description>* U8 O3 ]* w7 ^$ |% z! n
<name>yarn.timeline-service.webapp.rest-csrf.enabled</name>
" ]$ t p! Z2 k <value>false</value>) b- |1 T# p- a% o! @* |
</property>9 J2 @4 [, ], y; V6 R/ N
<property>
) B- _+ k- H& O+ R3 ^/ y <description>
! m4 j9 H7 `% H3 ^0 a Optional parameter that indicates the custom header name to use for CSRF# X5 l: q3 s" z& i7 T4 |
protection.
: ^3 S$ O$ I! H7 f6 P </description>$ G9 z+ x+ G8 K
<name>yarn.timeline-service.webapp.rest-csrf.custom-header</name>
, L& ~2 {: X5 Y- j* l9 K" I <value>X-XSRF-Header</value>
" Z2 i0 F$ q2 D/ P8 L </property>
7 o" r8 X ~4 |2 W5 G7 v <property>
7 S1 w. n; P3 F; ^" j' z <description>
/ p5 D7 E8 P- V6 e- } a Optional parameter that indicates the list of HTTP methods that do not+ P2 n5 ?+ C8 a3 n0 n
require CSRF protection
1 s+ T5 _& [: k* `9 b0 s </description>
' o3 L }5 E* |+ p <name>yarn.timeline-service.webapp.rest-csrf.methods-to-ignore</name>
+ \, J5 |, c! b0 z, O% k <value>GET,OPTIONS,HEAD</value>* A$ n& Z) _; {6 ]4 G! a! b
</property>
: h S' g3 X5 e <property>
: g7 T; V o$ T' i4 ~0 [9 _ <description>9 p% i; [* `8 O6 @4 o4 V8 I
Enable the XFS filter for YARN
) u: m' X; a. R5 u9 u0 b- y </description>
2 ?7 W: T! c8 q* f) ~( Q% u7 s' ^) f <name>yarn.webapp.xfs-filter.enabled</name>
( v, F T- G7 D* } W <value>true</value>
2 B- p9 X3 f1 w2 p+ Q# V, i5 {& Z </property>, N" `+ v& V7 m! p6 ?$ a* }
<property>
, Y" x [. F7 @ <description>: H5 d7 N! q4 H1 X$ A9 R2 s
Property specifying the xframe options value.; q* b- h: g" C7 m( ~$ @ H2 d- q
</description>/ ~( Y2 \5 E, a$ ~5 i1 W4 ]
<name>yarn.resourcemanager.webapp.xfs-filter.xframe-options</name>
9 h5 k( k- O7 Y+ O5 B! t <value>SAMEORIGIN</value>
7 g; g6 ~9 d, O- l v& c </property>7 z" ?: P' |* ~9 l
<property>- A$ t! N. s4 o
<description>
7 R5 Z/ V) z8 P Property specifying the xframe options value.
2 b, E" ]" R7 w- O) v7 { </description>% J" y8 `" p, r5 C0 H" C* e% s
<name>yarn.nodemanager.webapp.xfs-filter.xframe-options</name>% M9 D$ ^% O% }6 i" q
<value>SAMEORIGIN</value>( l* ]/ A% f" ]9 n* P' l! T4 @
</property># c3 A+ ?7 @; i$ l1 D# W, N
<property>
- G+ B8 O6 k" k! a* P4 G2 v+ x <description>
2 o. c! o! }4 x# y0 E6 |3 ? Property specifying the xframe options value.
9 h! E7 ?" W4 b7 A% p </description>
. @0 }* P7 `: F, q& P7 \5 M <name>yarn.timeline-service.webapp.xfs-filter.xframe-options</name>9 i$ i* ~& k% k% S; C
<value>SAMEORIGIN</value>
3 H9 M. \, y( ?8 | </property>) ?1 x$ l. p" @. {# G6 w
<property>
/ [+ l! _) q" `' m+ H3 Z' x; \ <description>% a3 a1 l; @7 T2 \% o
The least amount of time(msec.) an inactive (decommissioned or shutdown) node can
' { f% ^ N: h! j5 c- R3 I stay in the nodes list of the resourcemanager after being declared untracked.
9 _9 w1 ]! e. _" n A node is marked untracked if and only if it is absent from both include and$ A9 D& z" N" h7 p4 f
exclude nodemanager lists on the RM. All inactive nodes are checked twice per
4 H0 i. i6 P i/ Y' b timeout interval or every 10 minutes, whichever is lesser, and marked appropriately.* U7 X& k8 R# X
The same is done when refreshNodes command (graceful or otherwise) is invoked.) w; T3 B3 o. ?5 G
</description>6 d) N# r) F7 L" y9 h F
<name>yarn.resourcemanager.node-removal-untracked.timeout-ms</name>
, p' ?( h5 i! ^+ C K! Y9 b6 E# l- c4 I <value>60000</value># I' y1 k6 c3 X, T8 I/ G+ [
</property>
- U! q1 c3 Y5 c% c5 H Z* g <property>' f, T3 @2 l; h* y
<description>
+ D6 q6 t; k1 q1 Y0 p The RMAppLifetimeMonitor Service uses this value as monitor interval
& E) R. K3 j: B' X8 @ </description>
% V" i4 i3 }' m4 h* [ <name>yarn.resourcemanager.application-timeouts.monitor.interval-ms</name>
" a, w% @( g8 `3 n <value>3000</value>8 c7 ?- ^$ ^+ A6 n3 X: F
</property>
" k" W D, N1 F( B! H7 s+ q: a5 c <property>' [* R0 j1 k% p V5 r; _
<description>/ y% h' r! m- u; C
Defines the limit of the diagnostics message of an application* R' Z- ?$ o2 n! k5 n
attempt, in kilo characters (character count * 1024).
; u/ W/ k o0 E8 K1 |4 K) v, P When using ZooKeeper to store application state behavior, it's
% x+ c6 L# f h+ ]" R% N! t important to limit the size of the diagnostic messages to
3 S |7 R4 a9 S2 g0 F( }7 @3 Q prevent YARN from overwhelming ZooKeeper. In cases where6 D3 `' D8 m8 F2 _ W) T
yarn.resourcemanager.state-store.max-completed-applications is set to
& a2 m& |/ z4 @* ]2 m7 p a large number, it may be desirable to reduce the value of this property% N" F! ]! c2 h. h
to limit the total data stored.
8 x4 X7 ] M0 q2 j# h# Y. Y" w </description>: y) o% a% n0 t% [ o! [
<name>yarn.app.attempt.diagnostics.limit.kc</name>
9 K% m9 {; b: Q" L1 ~ <value>64</value>1 ^) L0 \( W' x0 Y5 }
</property>- P! p+ G( j4 a# [* F; V
<property>
2 E* H9 G& m) r3 o1 `' G <description>
& b j1 o% Z( h- N. E Flag to enable cross-origin (CORS) support for timeline service v1.x or
: ]- ~- W2 B; l8 K4 W Timeline Reader in timeline service v2. For timeline service v2, also add
! P7 z7 G) i$ y org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the8 N/ R: m8 d4 [1 n1 @: B
configuration hadoop.http.filter.initializers in core-site.xml.! L( b, z8 P8 |. Z: [
</description>1 Y9 {; D4 [8 U8 S1 n
<name>yarn.timeline-service.http-cross-origin.enabled</name>
3 b) j7 U8 W- `+ O1 M <value>false</value>" B& D3 c8 r+ Y% v8 ]4 _
</property>* x& [5 Q7 }7 l1 L9 }4 k( q
<property>
; Z2 \3 `) g6 k5 C$ [7 R <description>
2 E- A% l/ s8 D$ M1 @# g c; T Flag to enable cross-origin (CORS) support for timeline service v1.x or
X/ a. o9 |- f. j$ e Timeline Reader in timeline service v2. For timeline service v2, also add2 i j% {: |$ u
org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the
E6 x) ?1 O* u3 q5 ? \9 ~, Z configuration hadoop.http.filter.initializers in core-site.xml.+ F2 w- c! X- m& H' O/ |' t
</description>
4 i9 j& R* t g/ O! O) V/ [4 i) ~ A <name>yarn.timeline-service.http-cross-origin.enabled</name>
: |% j! @* `9 q* ], j6 Z <value>false</value>7 ]5 ]: s! k- }( P
</property>6 e) @, p6 F* s' @! s4 R
<property>( G% E0 W5 u( d2 v; ^' M3 T/ H) a
<description>
( f: n6 ~; S; A# v+ M The comma separated list of class names that implement the( h! c- t: w% Q# ]# s' d; t6 y8 L
RequestInterceptor interface. This is used by the RouterClientRMService- y* _8 A' R; a4 O ], M
to create the request processing pipeline for users.
2 L/ N- A& b$ X- X1 [ </description>
5 ^; U) b7 b0 H n/ c <name>yarn.router.clientrm.interceptor-class.pipeline</name>
8 [2 B6 z& C. t7 E3 Y' S* { <value>org.apache.hadoop.yarn.server.router.clientrm.DefaultClientRequestInterceptor</value>( i, P/ a% M- Y6 v1 m
</property>
8 q4 L% D( J* c3 ~ <property>
/ l2 ]; Y! _2 B) Q, C <description>7 {" t4 k& j* }# N( B' D8 A
Size of LRU cache for Router ClientRM Service and RMAdmin Service.0 u* a& p! x; I O) r) y$ X
</description>2 r& L! \0 d+ H1 a$ a4 z/ o8 m2 A# [; E y
<name>yarn.router.pipeline.cache-max-size</name>: ^4 \) V& k0 ~' l
<value>25</value>- H- \4 T, B+ L: }* ~* g
</property>
) ?4 X& g/ P$ S1 ?6 o% ]3 k <property>
" U4 h8 X' h* S' J <description>
# m* s3 W& o" P$ G5 d4 r The comma separated list of class names that implement the. g; M( R/ p/ F5 H5 S6 W# F
RequestInterceptor interface. This is used by the RouterRMAdminService
6 Q! R! `$ Y- f4 T2 t2 M to create the request processing pipeline for users.2 s/ ]( Y! g6 @' I! o' @
</description>
; G, J/ D2 f) f6 { <name>yarn.router.rmadmin.interceptor-class.pipeline</name>! k$ k9 X1 I8 w1 m; m
<value>org.apache.hadoop.yarn.server.router.rmadmin.DefaultRMAdminRequestInterceptor</value>
7 B0 o u# [1 l, U7 s </property>
( l7 m x( ?" v1 n& |. d& t <property>1 |4 d* z4 _. ?
<description>% B3 ]# x( ~1 j/ n( a# D3 @% H& [
The actual address the server will bind to. If this optional address is
; J/ t5 W3 o+ v5 g5 m+ a9 _, X set, the RPC and webapp servers will bind to this address and the port specified in! T* \0 j4 B1 J2 ~5 e1 }/ W- L
yarn.router.address and yarn.router.webapp.address, respectively. This is
1 w5 p8 [8 Y8 j5 Q, Q c most useful for making Router listen to all interfaces by setting to 0.0.0.0.
# B; u J# S+ u) g1 `* r* X </description>9 p& b' C- u3 ~
<name>yarn.router.bind-host</name>
% I- {/ j5 s+ q a5 n4 o0 ]: V1 { <value></value>
! y# {* P* Y2 I- Y2 w6 j( V </property>$ p) o ~, x, R0 M4 m9 g
<property>
& n4 G+ P% T+ T% G P <description>6 r2 n( \( _5 b* r% `/ w1 C
Comma-separated list of PlacementRules to determine how applications" k# ]5 h4 |! L
submitted by certain users get mapped to certain queues. Default is
+ X. R$ U) D' D$ u# {, I5 [9 u user-group, which corresponds to UserGroupMappingPlacementRule.
2 R Z# c% | o; u4 [ </description>4 S; F* n6 [ G& B+ t
<name>yarn.scheduler.queue-placement-rules</name>) Z, Q3 a. X' a5 e
<value>user-group</value>6 j3 \- z& M* Z6 C
</property>3 \% L) ^9 m( p! s/ h5 G
<property>4 ` R/ B+ Y% _# z& R- V. R
<description>/ f/ H, U2 ]3 b
The comma separated list of class names that implement the
2 N5 i# S$ ~8 c) z5 q& r' o RequestInterceptor interface. This is used by the RouterWebServices
3 E% R5 b+ S8 e! [- G7 Y. ^8 L to create the request processing pipeline for users.8 M* m$ M8 O K
</description>5 c, ]% P' y' R/ @0 A8 u+ A
<name>yarn.router.webapp.interceptor-class.pipeline</name>
. t- \4 _9 N; S7 ^ <value>org.apache.hadoop.yarn.server.router.webapp.DefaultRequestInterceptorREST</value>( c: l3 ~& R7 |# D- @
</property>
2 {, I: q2 i1 u9 i. {1 a <property>
5 R/ y+ x+ B+ x) ] <description>) R3 b b1 O" w' R
The http address of the Router web application.6 z. F7 O6 ^) k, ~) Q! @
If only a host is provided as the value," N$ D/ T$ M" x M
the webapp will be served on a random port.
7 W1 E& A; G. b( K </description>
, s5 A( u0 U# y1 N' B$ x( r <name>yarn.router.webapp.address</name>
' o* ` o8 ]. C8 n" K& Z! X/ g( u <value>0.0.0.0:8089</value>
; O9 D! N' j% L; l </property>
7 q {; k- K! Q# K) | <property>; `1 Q. t) M! C( o* p4 s
<description>
, V ^. S, M4 m4 J The https address of the Router web application.5 L6 h: v) Z3 j% Q" ~; h! P8 [
If only a host is provided as the value,* _ f; z/ p1 o
the webapp will be served on a random port.
+ L; A, m6 a# X# B4 c9 H" L' A </description>
) B7 H5 C/ @, q. X) R! x& s8 c+ Q <name> yarn.router.webapp.https.address</name>
6 T" e1 _ h7 z, }# E" r6 W& s <value>0.0.0.0:8091</value>1 `3 |* T+ J0 S) w0 F9 k6 \' Q1 o
</property>. ~( o* I' e# W3 ]0 ~
<property>1 X$ S; J6 N( j" G8 @0 m& _- S% h( r# o
<description>
0 F7 f! D! V3 g: s5 d% m" p# I5 [ It is TimelineClient 1.5 configuration whether to store active
4 @2 X$ s6 b2 k: Z, C2 H application’s timeline data with in user directory i.e# X' V# J1 q" I) b7 [. ~' W
${yarn.timeline-service.entity-group-fs-store.active-dir}/${user.name}
; C4 I1 F, c6 x3 g; F/ e8 ~3 [ </description>
- ] B' g# g0 ~) y <name>yarn.timeline-service.entity-group-fs-store.with-user-dir</name>
4 c+ G, r4 V2 J' P; P9 n <value>false</value>. R7 W6 _2 [3 _: c2 A
</property>
1 |1 `% [* i! `; f6 w4 Z <!-- resource types configuration -->/ Z1 a g6 {8 u) l% x% {
<property>6 m6 N5 O: R8 B% x5 |
<name>yarn.resource-types</name>
- z! S8 l8 h7 Z4 O* | <value></value>
5 [# X( h# N6 _ <description>
2 i- |0 ]$ u, z- U7 y7 m3 M The resource types to be used for scheduling. Use resource-types.xml* N& v" e: A' i3 H) |3 S+ u
to specify details about the individual resource types.
3 Q) @( }/ I( ^! x+ E </description>4 t6 l. c1 `/ b- u2 [4 e$ Y
</property>! y# o. \4 o( o: M/ ]
<property>
* _, K. o0 C# k" S7 R* G. a <name>yarn.webapp.filter-entity-list-by-user</name>
6 U% V7 Z* `8 I, v' u! w4 l, M <value>false</value>; f( I' b" ]& ^8 }
<description>
) F6 d( V( Q- @. B1 P# ? Flag to enable display of applications per user as an admin
# ~3 p8 u0 Y$ A3 i, V' E1 W) x configuration.$ x( Y' z' d( G3 H' \3 c
</description>2 K6 I& A8 B$ [# [% b* B( N2 O
</property>" u- e. [# T" _8 Z7 b3 B0 a
<property>2 h7 T* ]8 F( ^ v
<description>
! q+ c5 m; G! e4 a% S8 d5 e: I3 H9 L The type of configuration store to use for scheduler configurations.
! h a1 _& q( q5 d0 P" [/ S- `' T Default is "file", which uses file based capacity-scheduler.xml to5 z; N4 c! X! T |6 S, k" Z" w
retrieve and change scheduler configuration. To enable API based! r8 E* |' E. c! ~
scheduler configuration, use either "memory" (in memory storage, no
- Q4 S0 q' {8 D1 u" s6 u persistence across restarts), "leveldb" (leveldb based storage), or
* D1 O0 R. K' V9 z: \! W1 U "zk" (zookeeper based storage). API based configuration is only useful
% s& y o$ z8 ]1 |4 \ when using a scheduler which supports mutable configuration. Currently; J/ ]5 R8 w* A/ T% a. g
only capacity scheduler supports this.
& L$ N' p. Y7 B/ [ </description>
6 N/ I. y3 B% u) x* j <name>yarn.scheduler.configuration.store.class</name>( m- T% a5 L' g6 O* @
<value>file</value>
8 I2 Z4 Q( c& o4 P# C4 U) P </property>
6 u9 F, A6 `: `8 G% P9 E) o/ U3 K <property>' p! q' O2 H! c, ]4 B1 k
<description> K$ h) A5 C1 e( @( X+ U$ B S
The class to use for configuration mutation ACL policy if using a mutable+ e, `0 e: u4 M1 L, G3 A4 p
configuration provider. Controls whether a mutation request is allowed.
0 e( i, c* o. I, u# Q' l4 t/ V8 n- z The DefaultConfigurationMutationACLPolicy checks if the requestor is a/ V; q2 ^9 ?0 X( |1 V( D
YARN admin.
8 |" _1 |. X) s* i7 F( |7 i </description>+ p% `2 Y" C9 I% h! z4 B
<name>yarn.scheduler.configuration.mutation.acl-policy.class</name>/ x! d6 a9 H7 A0 n
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy</value>
3 }4 _) S; w4 ^$ c </property>* P8 h3 x9 x8 N* c8 c
<property>3 n# [' x4 f# m! C7 U, J) t
<description>- y2 T9 u t6 Z
The storage path for LevelDB implementation of configuration store,! }7 X! o/ N$ [. r5 T5 `, R8 @
when yarn.scheduler.configuration.store.class is configured to be
; n! n& ]8 G8 ~" J/ L1 w8 Z* X "leveldb".' P7 Q- p. h3 N! F9 U
</description>1 k$ e( ~! e6 A4 L) U4 o S
<name>yarn.scheduler.configuration.leveldb-store.path</name>
; d* n( X1 @9 g: d% P' y <value>${hadoop.tmp.dir}/yarn/system/confstore</value>1 M6 i3 M7 g% Q1 e3 W
</property>
) x+ z* U3 o# x/ k7 ` c3 p <property>. X3 T0 V0 E* ]$ ]/ o
<description>
& z, `& r) @- U( A The compaction interval for LevelDB configuration store in secs,' {* Z5 w4 r. i0 D/ e% S
when yarn.scheduler.configuration.store.class is configured to be- u( P# L! r# `! f7 Y" B! C5 w/ c
"leveldb". Default is one day.7 ]% D$ W1 X1 p/ z w# O
</description>
# j: L4 Q7 @( t- M. o- e8 R <name>yarn.scheduler.configuration.leveldb-store.compaction-interval-secs</name>2 F" m4 @- c( Z, }/ O8 D' g
<value>86400</value>
& m* K# ^+ h( T2 w: \% \5 { J </property>' A$ k0 I! g1 i9 V
<property>0 l Q3 M$ h' V9 w- U
<description>
! L) }7 j' F* P" J% _ The max number of configuration change log entries kept in config, w7 T$ @! i; o. N
store, when yarn.scheduler.configuration.store.class is configured to be
5 A0 Z3 l7 v/ t3 s+ F: ~" E "leveldb" or "zk". Default is 1000 for either.
% Z. ?) ]5 k1 U& n# D2 q </description>
* W, r5 Y4 y/ ]5 M; [) f <name>yarn.scheduler.configuration.store.max-logs</name>! O7 _7 a5 H0 w/ d7 Z
<value>1000</value>
9 T9 k5 }7 H3 V2 D </property>
0 F9 `& ^0 G" _" U5 K! [4 @ <property>0 g1 E+ N6 r& H8 t+ d1 x
<description> v( p+ _# C' ^' P& L
ZK root node path for configuration store when using zookeeper-based/ }* s8 B" a0 f6 T, _- ?8 m4 q* t0 H
configuration store.1 i! A$ q6 X* X# `! J" G/ V
</description>
9 p. ^, P; J) \& H+ J: J <name>yarn.scheduler.configuration.zk-store.parent-path</name>8 g9 F+ t4 o' i+ ?' |
<value>/confstore</value>' M7 ^3 ~0 V3 A5 ?* L i
</property>
& A9 X2 P) L& j1 q: e q <property># t; |) j5 ~5 n4 U7 k j
<description># k% H" z h b/ \
Provides an option for client to load supported resource types from RM
0 ~! l2 f2 j7 ]5 K3 d0 u instead of depending on local resource-types.xml file.
/ ~" I) a/ g& P0 A* @8 w </description>
9 I4 J( m1 i" ?6 Q% z; ~2 `# u9 l' r <name>yarn.client.load.resource-types.from-server</name>
* _2 a1 p% p- o& F' g3 M4 c9 e5 L <value>false</value>
* g* ]) [$ l) k" A) b# R' K </property>5 h' Y0 Z% k$ b
<property>
2 d7 R. ?, y# P) g9 l7 T <description>% W2 F; v% J4 E8 I3 i* d
When yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto specified,, { M; I. q. ^4 d# s0 f
YARN NodeManager needs to run GPU discovery binary (now only support3 H& _) l, {) J# `
nvidia-smi) to get GPU-related information.& j' Y N+ }3 [6 [
When value is empty (default), YARN NodeManager will try to locate" ]- \9 Y# }2 L6 ~2 s3 u9 H
discovery executable itself.
9 Q. Y9 `2 v$ w4 N5 G An example of the config value is: /usr/local/bin/nvidia-smi" R9 n; K8 j, r
</description>+ h) y) ^9 J( P- L: c
<name>yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables</name>& {: x0 X9 C2 b% t9 [1 P
<value></value>2 u+ T" v( R" P/ d# Z& o% c
</property>
( c- k- `3 I, \+ ], ^3 } <property># X* h2 Q' ]. \ \- y6 |
<description>7 F1 @. o. B& \+ j
Enable additional discovery/isolation of resources on the NodeManager,
0 \& p7 P0 o) u" l split by comma. By default, this is empty.7 Q- w: B. j# s4 Z5 m2 a R# v. [
Acceptable values: { "yarn-io/gpu", "yarn-io/fpga"}.
; L* E# ^( j, B# n, l3 m </description>
9 r; W. p3 l# \4 O2 V/ [ W <name>yarn.nodemanager.resource-plugins</name>* B. A2 i) g9 h* |: `# L
<value></value>; S4 h/ G/ W% T; P! s! c4 j% O$ M
</property>5 f' G0 b7 s# J: V& e+ _# F
<property>
2 Z8 {; y, H% @$ @ <description>
0 e! E3 [ ^$ d3 `3 ?) t Specify GPU devices which can be managed by YARN NodeManager, split by comma
* c4 Q+ d) U, S% @+ H5 ~' `7 h Number of GPU devices will be reported to RM to make scheduling decisions." H# F4 q, N5 L/ W
Set to auto (default) let YARN automatically discover GPU resource from
) o- H; P$ l; v4 {/ P system.. ~0 Y/ y& n1 Y; B4 P( c" Y
Manually specify GPU devices if auto detect GPU device failed or admin- P, r$ J! I. S, \% T- X
only want subset of GPU devices managed by YARN. GPU device is identified$ ^! M [ m/ O/ D0 Y
by their minor device number and index. A common approach to get minor
0 M! i# `9 B3 {, {' Z( r device number of GPUs is using "nvidia-smi -q" and search "Minor Number"
0 g' g* p+ \' v- o output.
4 ]+ ^; @' ^1 x* a When manual specify minor numbers, admin needs to include indice of GPUs
5 y' }' R8 A6 W! ]. {# f as well, format is index:minor_number[,index:minor_number...]. An example7 f. w& C L' o3 k; F
of manual specification is "0:0,1:1,2:2,3:4" to allow YARN NodeManager to
3 {0 E$ i# B; x$ y manage GPU devices with indice 0/1/2/3 and minor number 0/1/2/4.2 X4 l/ F9 o) [$ [1 X
numbers .
5 V. v5 S% c7 F& `2 X' E' j1 c </description>
: m' S# `9 x; r. I) A1 w- x <name>yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices</name>
* z) |* {; d3 f. P/ u <value>auto</value>8 b# i2 F: d9 G) [+ G4 j* J
</property>! u, t" M" }8 }+ L: f% j' S2 \8 q
<property>, c n1 `" D" p
<description>% t1 K/ [* C* ?/ F T
Specify docker command plugin for GPU. By default uses Nvidia docker V1.7 u3 ~( z d: x: f y7 \: I
</description>
% O- K* b+ E, f% s3 o* m) e <name>yarn.nodemanager.resource-plugins.gpu.docker-plugin</name>
- R3 M) X) `! p( ?+ V <value>nvidia-docker-v1</value>( U! J) a& h% P4 h' u+ b
</property>1 w! k& O _- e9 ?1 Z7 {+ [+ L
<property>0 ^# v0 E8 b" f; r X O3 R
<description>
8 ?9 j# o, K5 P/ e Specify end point of nvidia-docker-plugin.
8 [! L# B3 Y: T3 \- g Please find documentation: https://github.com/NVIDIA/nvidia-docker/wiki5 j& f k6 |! z$ H* Z5 m
For more details.2 |1 u6 p0 w9 n) _
</description>+ X8 f4 U" X. l
<name>yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint</name>! F0 P& [" V/ y7 J( d
<value>http://localhost:3476/v1.0/docker/cli</value>; M1 n$ F+ }3 K$ [
</property>- ?0 F" z) t; N Z9 H% V4 y
<property>" s: H" E9 t) ^. X6 `
<description> R! J. u$ d5 C: U0 u
Specify one vendor plugin to handle FPGA devices discovery/IP download/configure.
9 b; J/ c4 ^9 z5 E Only IntelFpgaOpenclPlugin is supported by default.
9 b m/ Q1 u5 `/ ]/ h+ \ We only allow one NM configured with one vendor FPGA plugin now since the end user can put the same4 a. e+ r# `. M+ f( H& U
vendor's cards in one host. And this also simplify our design.- k+ O2 p" ~( @, b/ f F! A! V5 r
</description>' W! Y7 K) n6 \8 x; A8 v8 ?
<name>yarn.nodemanager.resource-plugins.fpga.vendor-plugin.class</name>1 r( t) r5 @/ ~" k. b3 k+ Y+ Z
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.IntelFpgaOpenclPlugin</value>
) U- Y& n1 ^7 v2 J </property>
) ^' t2 W! L5 G8 l$ o <property>
8 n4 P% y" @8 Z4 j+ C <description>* m6 b$ u! h9 i O/ G; O5 }
When yarn.nodemanager.resource.fpga.allowed-fpga-devices=auto specified,/ Q4 K$ y$ ]- c6 c
YARN NodeManager needs to run FPGA discovery binary (now only support* o" F G; W- d- p0 E! r& P
IntelFpgaOpenclPlugin) to get FPGA information.) J& s4 k% L5 G6 ?$ t
When value is empty (default), YARN NodeManager will try to locate" ~8 @ y$ Y; A% G5 g: x
discovery executable from vendor plugin's preference, h2 C" `% b$ l# X
</description>& ]" B1 B; x7 ?* {( D
<name>yarn.nodemanager.resource-plugins.fpga.path-to-discovery-executables</name># b3 U+ ^. o( _5 c1 T' u/ K" Q
<value></value>
3 W& t3 _4 U4 f8 F3 F </property>8 t! N8 s( @$ ]5 P
<property>
: a5 a. A, r+ w <description>
1 A( ^. H" ?" c- B- n1 w Specify FPGA devices which can be managed by YARN NodeManager, split by comma
2 S, D; v9 h& G' e3 X$ a Number of FPGA devices will be reported to RM to make scheduling decisions.
; v/ n8 A" T0 h5 W6 n0 A Set to auto (default) let YARN automatically discover FPGA resource from5 X4 e4 G+ D1 s+ r
system.
) `4 d( X2 K& s! G+ S. @ Manually specify FPGA devices if admin only want subset of FPGA devices managed by YARN.
) \) c6 _$ _; x( o1 b" \0 S, D At present, since we can only configure one major number in c-e.cfg, FPGA device is4 X. w+ {/ R8 T/ _/ O
identified by their minor device number. A common approach to get minor. O( P N* h, I/ h6 R v
device number of FPGA is using "aocl diagnose" and check uevent with device name.
0 M1 C, ~+ `& u" J7 `6 J </description>
& M8 @2 i# t. I* t1 a <name>yarn.nodemanager.resource-plugins.fpga.allowed-fpga-devices</name>
3 L) o# e/ w9 L# U <value>0,1</value>1 D7 [9 o' e& q, T0 \" X; H
</property>; y# z$ D# K( Q" j4 [1 Z( Q
<property># y M" H$ \4 [) j9 _% y) @
<description>The http address of the timeline reader web application.</description>
. p8 R; J1 R I0 W; { <name>yarn.timeline-service.reader.webapp.address</name>
% g0 n0 n$ J4 g <value>${yarn.timeline-service.webapp.address}</value>' _* a$ ]6 O# Y% G. Q: I) H% b
</property>& z# c' q2 }4 o( S: F
<property>
3 @5 [: \9 n* L$ K% @3 ] <description>The https address of the timeline reader web application.</description>
, Z1 [0 O8 ~7 z" \ <name>yarn.timeline-service.reader.webapp.https.address</name>1 `: y- c% g4 o7 @( }( X' ~, z
<value>${yarn.timeline-service.webapp.https.address}</value> j; l7 y7 k, R# e
</property>
6 f& X- ]/ W: O& P7 R' c3 V <property>
; {) L5 U2 c4 b: u7 f <description>
( m; V* O1 ]5 _4 C( v The actual address timeline reader will bind to. If this optional address is0 Q! H) _- Y9 {6 m
set, the reader server will bind to this address and the port specified in/ j) \9 x% @3 ^) I* q; M( y
yarn.timeline-service.reader.webapp.address.% E& v# o: v9 _) M: v1 X6 x; f' {4 s
This is most useful for making the service listen to all interfaces by setting to
. T6 b* K8 S4 m$ V 0.0.0.0.
8 @: b1 n. U: }4 `6 O7 i </description>
# _, Q8 k$ C+ X# J6 G! i <name>yarn.timeline-service.reader.bind-host</name>
6 e& O* T9 x# T# L <value></value>
( _ {( ~; [4 |9 B6 E6 N& o4 ?0 z9 { </property>3 ]+ d1 F- q" j/ W7 {3 Q
<property>
6 |8 V6 H, f$ m <description>% \* w5 q6 K7 f- a( k+ }
Whether to enable the NUMA awareness for containers in Node Manager.8 S- Q( M/ S8 J
</description>
3 ?4 q3 W1 |5 @/ [; ?3 ?5 q+ }; E <name>yarn.nodemanager.numa-awareness.enabled</name>1 Y" G! z- L. V! f; f J" i/ W
<value>false</value>5 W2 Z8 @1 u" y7 Y1 M T5 }
</property>- v% n' T- M b
<property>. m2 Q( |% e1 s3 @" g e
<description>6 \" a) R3 @+ ~2 _
Whether to read the NUMA topology from the system or from the
3 o! }/ j8 M$ r, D: s configurations. If the value is true then NM reads the NUMA topology from' F0 T- ^5 L+ q! f: W; w
system using the command 'numactl --hardware'. If the value is false then NM! T: u4 V# \ I$ ^" S# [( X# T, O
reads the topology from the configurations+ L) a8 a1 f5 ~2 p! a
'yarn.nodemanager.numa-awareness.node-ids'(for node id's),
M: x% F. E/ L' }4 @- c 'yarn.nodemanager.numa-awareness.<NODE_ID>.memory'(for each node memory),
& ^( u# N# D% e; ^# E2 h 'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus'(for each node cpus).. e$ z6 N2 |( `; h9 \
</description>
$ b4 o9 e+ F1 z <name>yarn.nodemanager.numa-awareness.read-topology</name>
) S& I! c7 v4 v <value>false</value>
! n* m" w5 \. L3 o3 f. l- n </property>
% T* k6 a3 D6 n7 c8 j& {4 p <property>
' a# N, }& ?2 s3 l6 m <description>! h3 `( h: M' o" _9 P/ H4 ~+ G2 K
NUMA node id's in the form of comma separated list. Memory and No of CPUs
; H1 K+ E9 X' D' o8 O will be read using the properties) |; a& @: |9 E4 h# D( D
'yarn.nodemanager.numa-awareness.<NODE_ID>.memory' and
F/ u; p K! F. x0 g 'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus' for each id specified( I9 A; ^' X# e. k' {6 `
in this value. This property value will be read only when
4 a! d3 m! j/ U i F: a0 g* v/ P 'yarn.nodemanager.numa-awareness.read-topology=false'.
* ?1 D5 _2 X$ W' p+ R& u For example, if yarn.nodemanager.numa-awareness.node-ids=0,1
* t% Q4 z7 G/ C" e- f then need to specify memory and cpus for node id's '0' and '1' like below,4 B1 Q# b# X4 m& P) W q
yarn.nodemanager.numa-awareness.0.memory=73717
: m% o9 E7 J' V( D8 {& a" N/ z yarn.nodemanager.numa-awareness.0.cpus=4
) G0 U( U; w; O$ E0 {/ \) u" l yarn.nodemanager.numa-awareness.1.memory=73727
9 i% a) d/ ]) M: A yarn.nodemanager.numa-awareness.1.cpus=4, V( ]2 Y0 o* k* ~: H
</description>
8 q8 a% t- E0 z. u" ? <name>yarn.nodemanager.numa-awareness.node-ids</name>
! |# _' z; Y; X/ J" H. @( z <value></value>: F7 P8 d0 m# j- r: N0 V
</property>
6 P# L9 I6 p A. P& Q; M <property>9 G0 @9 z3 u# u& N7 }' B; a0 G
<description>
6 A' F$ D6 K6 K4 P) v( h" k The numactl command path which controls NUMA policy for processes or0 l* G) ?; T! d" s9 ]. X% d# }2 P: a
shared memory. M' Q9 K g; c7 _6 l b8 L
</description>, E5 B* i$ ]0 O% ]
<name>yarn.nodemanager.numa-awareness.numactl.cmd</name>
: K. T: ^( n" u! M, n <value>/usr/bin/numactl</value>
" j) w2 w: r; g. v: E) W </property>
6 A2 H1 T# v3 |' j2 X% _7 G; @</configuration>
$ W- s' V1 y8 p2 \) x, C5 ?2 m4 E' v
|
|
发表于 2022-11-7 11:18:00