|
|
楼主 |
发表于 2022-11-14 13:52:09
|
显示全部楼层
<?xml version="1.0"?>
V3 l! W( d( Q& a: |+ D<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
" W) f6 L2 o: O1 C" R5 T0 [; R- _<!--; n6 e3 e8 ?- f4 G9 j* v2 | b
Licensed to the Apache Software Foundation (ASF) under one or more
* K9 a# D4 k7 @$ F! |* g# ~ contributor license agreements. See the NOTICE file distributed with
+ c4 o, l" }$ V5 B+ `) }% C* E% {1 F this work for additional information regarding copyright ownership.
) b4 P2 O3 e; N* j The ASF licenses this file to You under the Apache License, Version 2.00 b' y; \- G$ Y# T$ z: j/ ~
(the "License"); you may not use this file except in compliance with
! L0 s% E7 [* K. C! v. w, q the License. You may obtain a copy of the License at4 D/ @) e0 l' _# E# O8 g0 k8 E! \
http://www.apache.org/licenses/LICENSE-2.0
2 A+ [% ]: c9 D0 ~* X Unless required by applicable law or agreed to in writing, software
$ u, {$ I1 D6 b2 I+ X' b/ m distributed under the License is distributed on an "AS IS" BASIS,
4 \0 d3 J: P$ Y' d WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.5 v1 `+ q* } U% P0 }+ D/ o
See the License for the specific language governing permissions and
! l* w8 [' z2 V+ ?, u+ J5 z8 `. o5 k limitations under the License./ r" |* H7 J# ^$ o c/ D$ r
-->* N0 |" U6 ]( m1 t" g" ]
<!-- Do not modify this file directly. Instead, copy entries that you -->/ [+ b; Y2 h5 B* d
<!-- wish to modify from this file into core-site.xml and change them -->+ N- K/ k- {5 M6 W6 D6 n
<!-- there. If core-site.xml does not already exist, create it. --> P5 s, d& y, S" j9 J
<configuration>8 L/ ^3 G4 l4 o4 d4 y( ?5 d' V
<!--- global properties -->/ f# J, ~) R8 Q; _- Y: ^
<property>
; ]5 ]! H: J9 [# s <name>hadoop.common.configuration.version</name>( ?- A1 l' v! y
<value>3.0.0</value>4 D7 _ i" y8 R. g- J6 q6 @( Y0 r
<description>version of this configuration file</description>) J% ]' J. {, @
</property>
& `9 F' S7 H/ v: Q: k3 a+ {) H8 e<property>" e' L+ y9 Q4 Y( B, a0 A+ X, X# V
<name>hadoop.tmp.dir</name>
2 H, f8 `& N$ ? _ <value>/tmp/hadoop-${user.name}</value>
* F& Y; m6 E& \% s: e2 ]; e3 Y9 s <description>A base for other temporary directories.</description>7 v/ C. ~# f7 ^3 i
</property>
+ v% ]6 Z; k/ D8 M2 k0 U, e# S$ Z<property> c3 [/ z4 x3 A; L7 ?
<name>hadoop.http.filter.initializers</name>- Q- J" K5 G$ F, ], L0 V/ M
<value>org.apache.hadoop.http.lib.StaticUserWebFilter</value>" s7 R* C$ _' D/ q7 ~1 H0 N6 j, s
<description>A comma separated list of class names. Each class in the list
. l9 l& E# m$ r( q% H& f- b$ v must extend org.apache.hadoop.http.FilterInitializer. The corresponding* O0 n6 m. F; r/ d
Filter will be initialized. Then, the Filter will be applied to all user* H9 d1 A0 r. E
facing jsp and servlet web pages. The ordering of the list defines the
9 P* I8 R; O# @. D/ d- D3 F4 F% t ordering of the filters.</description>& Z, [: U6 L6 B
</property>2 F- \/ E+ X. {/ h
<!--- security properties -->4 x6 T1 F) ]& a; R
<property>. }& R t' p& U2 \4 Q B
<name>hadoop.security.authorization</name>. ?+ k" E1 W, x* k2 M2 f
<value>false</value>
- M6 I9 @: `, |! I <description>Is service-level authorization enabled?</description>
8 O) _0 ?2 h" x }4 a8 n: W</property>
" G9 ~& H# O: j; v<property>
" \9 t {; a+ i- e& b <name>hadoop.security.instrumentation.requires.admin</name>
: f! s: c6 F, ^5 r <value>false</value>
4 |# c8 T7 ^: E# I' K <description>
; e' q. G9 m: j1 c. J! N( _ Indicates if administrator ACLs are required to access4 o5 O, t) q& \3 l: _3 m, l& z; v' f
instrumentation servlets (JMX, METRICS, CONF, STACKS).
0 A; m9 r$ M* [1 {# W2 z </description>
+ H) u: N# I/ F) W2 D* @4 D</property>6 w. y# k" j- E5 ^4 N4 g
<property>
7 ? v# O7 K I/ }; o9 } <name>hadoop.security.authentication</name>
2 q; v( @' Q& Q/ X5 @$ ?: B <value>simple</value>
, |* ~5 A( H5 K <description>Possible values are simple (no authentication), and kerberos
9 R! r) `# x( D# C+ [% i </description>
$ N/ ]0 V4 M$ V/ w/ T- u9 h% k7 ?</property>
8 C# {8 O5 T9 H# x6 y* T<property>
' U' _+ C! H6 i, E* p6 v. Y9 { <name>hadoop.security.group.mapping</name>. n; I9 N: a7 d- X
<value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>
! N1 w5 Q1 ?& W5 w1 b0 ? <description>
8 @2 X0 c2 M9 \8 G! @: P Class for user to group mapping (get groups for a given user) for ACL.
4 P3 R E9 d" N) O* e& d The default implementation," I& T* ]; N7 [; l* P8 H) f
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback,% w: N3 h* G( P3 C7 v4 _- o4 P
will determine if the Java Native Interface (JNI) is available. If JNI is) o6 i% r. x6 d9 ^
available the implementation will use the API within hadoop to resolve a
/ c ^5 r/ r6 J* j! f list of groups for a user. If JNI is not available then the shell/ V& C. q8 R% H6 m2 \
implementation, ShellBasedUnixGroupsMapping, is used. This implementation" g8 V2 Z% e+ ^9 T1 }+ G o
shells out to the Linux/Unix environment with the$ T% [+ G W& @- [( x
<code>bash -c groups</code> command to resolve a list of groups for a user.- r2 W( S% Y- r4 ~
</description>9 W( z$ l$ E0 f# K, s
</property>
* F2 A9 R4 p) I9 S; J* U3 f9 S<property>
N( i9 n# L0 v5 D: o <name>hadoop.security.dns.interface</name>& v# i5 C/ B0 t; i1 |- C, v
<description>
' X i1 _+ A6 r W1 [) ~% Z6 d! M$ T9 @ The name of the Network Interface from which the service should determine& x# S. V; h1 b
its host name for Kerberos login. e.g. eth2. In a multi-homed environment,+ v" C }- r9 e& H( e) X
the setting can be used to affect the _HOST substitution in the service
: e- \8 e- {6 M2 G: U$ O Kerberos principal. If this configuration value is not set, the service
: N' [" F" U9 ]& V will use its default hostname as returned by
, A' R" T* J8 V6 b* ` |( t8 M InetAddress.getLocalHost().getCanonicalHostName().( g& x8 K. W8 ]) x# i
Most clusters will not require this setting.
- X+ {3 [6 t7 V* ?# F: ^0 f* N </description>
4 W: r0 p# [! D4 Z- S, w</property>
- _( x' \+ X" M: T1 O4 T8 O. Y+ k<property>
; R6 i z# M Z j0 S. E) H <name>hadoop.security.dns.nameserver</name>0 f% {' L7 F, C: f7 S
<description>( }8 U9 y* b% _
The host name or IP address of the name server (DNS) which a service Node5 t d+ x) w6 u E. E: i
should use to determine its own host name for Kerberos Login. Requires. g. k2 o9 V" A8 S5 L
hadoop.security.dns.interface. y( o2 w+ W9 B: g
Most clusters will not require this setting.1 U! ^8 g0 M. J; w
</description>
( }6 f/ g" e6 S% h: R</property>
/ J$ @' L- S2 Y6 n/ c<property>
# U5 T/ q% C3 t; r; i, h. p <name>hadoop.security.dns.log-slow-lookups.enabled</name>- E/ o3 d7 h3 ]( M! i
<value>false</value>- H+ x. R% V. ?- v& C( s
<description>
& W3 j" V. _. ^/ K/ O: r Time name lookups (via SecurityUtil) and log them if they exceed the
5 C# @3 l; u5 H$ ?5 S configured threshold.
! k5 d D& Q( [3 k* Z </description>
. A3 }2 o2 k6 m* ^</property>9 X& Z& b' k+ u- v
<property>
' s4 H8 N1 V5 W& ?+ E <name>hadoop.security.dns.log-slow-lookups.threshold.ms</name>
6 I0 O m# z7 p5 A <value>1000</value>7 g$ b( v0 A7 u
<description>; |# ], S0 q- I. t
If slow lookup logging is enabled, this threshold is used to decide if a) H$ Z" b. {# ^9 {$ e) O3 A
lookup is considered slow enough to be logged.6 M1 a/ E1 t7 |& Y3 ]( Z6 d/ E
</description>
. r0 o p Q+ e- G</property>& |5 u* s0 s3 N# G8 r
<property>
) A" p q# m8 O2 G/ G7 X <name>hadoop.security.groups.cache.secs</name>
9 q' Y* f4 r p6 Y: J: S3 @ <value>300</value>) K. w2 j' Q3 \' G* N
<description>
" h. H# E4 x) K This is the config controlling the validity of the entries in the cache
1 i% ^+ m- S" |# G containing the user->group mapping. When this duration has expired," r7 Y) z& N, A% l, |+ \
then the implementation of the group mapping provider is invoked to get9 O) @# Y+ p3 P5 \# D/ \. m
the groups of the user and then cached back.+ P/ i* k' c/ r( }8 H! v
</description>- P( ^) B' S$ g) [) n
</property>. s r- O: T" Y( e1 E" D
<property>6 h) v/ N# P' B5 ?& p2 k$ L- g- t
<name>hadoop.security.groups.negative-cache.secs</name>+ B% s2 |0 r, a& R' W; y
<value>30</value>) B/ F/ E o1 E( u8 X. ^
<description>, c( M1 {) s$ z
Expiration time for entries in the the negative user-to-group mapping8 q; |, }* p6 U* j! r7 n) d/ Z, \& Q
caching, in seconds. This is useful when invalid users are retrying, h, Y1 o+ ~7 n! R9 S8 ^
frequently. It is suggested to set a small value for this expiration, since. ~/ u) J) Y6 k8 V. |/ C7 x4 i) r0 B
a transient error in group lookup could temporarily lock out a legitimate$ ` k" W: @7 H5 P
user.+ ~& @/ b: _5 y
Set this to zero or negative value to disable negative user-to-group caching.' Q7 Y# z; `7 x2 H7 i r
</description>
4 k( W, ^$ `( ?: J8 h! Y# B$ R</property>
( h; K, F! x: r# l- N: b" S<property>/ N! y. H. j. k1 k$ A0 O* X
<name>hadoop.security.groups.cache.warn.after.ms</name>2 b5 |5 h! T; z5 q: \
<value>5000</value>! {3 G' d2 ^* R1 f8 q1 a! S' D
<description>
5 \2 U6 c6 l4 s7 _ If looking up a single user to group takes longer than this amount of e1 J0 G+ j/ R
milliseconds, we will log a warning message.
. p( t, o& k- Q8 r' a# y </description>
% a6 |5 s9 o+ ~6 h</property>
) y% Z- V* Z, L! H' E3 [<property>/ V1 [* r3 Z8 r9 S+ B
<name>hadoop.security.groups.cache.background.reload</name>
" H" k" z1 |/ v <value>false</value>
) R+ w2 s$ o* J% Y3 { <description>( S% h6 Q- }1 g! ^
Whether to reload expired user->group mappings using a background thread
) ^% a: K+ \* P: h M6 p& d0 v3 q pool. If set to true, a pool of
7 Z; N% ^3 _) |5 ]2 e8 J) X hadoop.security.groups.cache.background.reload.threads is created to& H+ ]! z# ?3 Q3 }! E
update the cache in the background." B+ r$ W* M! ]) U8 b
</description>
+ l" f! t% r6 A: u7 @( l' o, V1 ?</property>
8 _3 ^0 @2 W$ p9 F<property>
" [1 J" q1 e- U2 q% r: W <name>hadoop.security.groups.cache.background.reload.threads</name>" U5 \8 U: g$ p* l3 _
<value>3</value>2 g% B& l$ e0 w
<description>
, H+ v% _# j" x2 W Only relevant if hadoop.security.groups.cache.background.reload is true.; U1 C& {) _. @ {6 _5 \# X% s) d& l
Controls the number of concurrent background user->group cache entry5 J5 s8 k. g3 B: U5 a% a) r
refreshes. Pending refresh requests beyond this value are queued and- s2 h# ?5 q: s, D' a
processed when a thread is free./ h4 _9 @3 }1 N( R$ Z
</description>
5 V+ r1 F: _3 B( O8 {</property>0 O9 w( r- ]8 x/ l1 x) L8 m+ ?: g6 l
<property>
. M3 n4 F/ ^" I) @% Y1 k5 f <name>hadoop.security.groups.shell.command.timeout</name>+ Y$ F' M0 k. E/ x6 m$ ~
<value>0s</value>
3 ]% n0 O5 M' e# D8 M <description>
8 m( z9 r6 |* S% A5 H9 m Used by the ShellBasedUnixGroupsMapping class, this property controls how: J% l7 d7 v/ }) M- ]
long to wait for the underlying shell command that is run to fetch groups.6 E8 ?3 O7 s" w- p' ?
Expressed in seconds (e.g. 10s, 1m, etc.), if the running command takes
. D r8 C5 J4 [$ S9 n longer than the value configured, the command is aborted and the groups' t2 J4 b# `* t) D) Y% C) W
resolver would return a result of no groups found. A value of 0s (default)
; U6 O/ x9 h& Q would mean an infinite wait (i.e. wait until the command exits on its own).9 g) s9 c. v/ R- X7 ~' k
</description>
$ J6 W. _% \3 G! Z</property>" A2 V* K) ~' m2 e, S G3 Y
<property>0 U. \7 O4 D' X
<name>hadoop.security.group.mapping.ldap.connection.timeout.ms</name>) m1 j! k" |" G5 Z0 Q- S
<value>60000</value>
" t% U9 z2 g; j( } <description> K4 x. a3 g. _$ C& g4 V
This property is the connection timeout (in milliseconds) for LDAP
5 K8 U6 s9 M8 i* p/ W2 Y operations. If the LDAP provider doesn't establish a connection within the
: _4 p! ]6 j' {1 ? specified period, it will abort the connect attempt. Non-positive value5 w# p! v) r8 M' d5 v8 v
means no LDAP connection timeout is specified in which case it waits for the
3 ~0 t* t2 `' X0 V connection to establish until the underlying network times out.
6 }5 p. B$ m# K9 l2 B </description>1 W# |% q3 ]. x$ E' E& l
</property>" u" g* G4 K% Q& m: G: i
<property>8 l! z4 n& z5 i; h" w7 l
<name>hadoop.security.group.mapping.ldap.read.timeout.ms</name>
1 C/ ?" R$ B g9 Q2 N) g( z( \ <value>60000</value>! w }; J& K. F6 ?- F0 x! Y5 n; u
<description>
! ?- n$ s( b6 ^$ n6 D8 g, m' d# I+ S This property is the read timeout (in milliseconds) for LDAP6 I- K. ~1 p, c/ n9 p5 G8 C
operations. If the LDAP provider doesn't get a LDAP response within the
) W& q& m8 e" d1 d) G specified period, it will abort the read attempt. Non-positive value/ D H) ?! H% L$ G/ z
means no read timeout is specified in which case it waits for the response9 H/ H; ^2 Q0 U) x$ u C
infinitely.
* C. m& j3 L3 Z0 [: X </description>
9 D8 r* a& W' k( r6 U( ~; J# {</property>
/ Z2 U" q; _- O<property>
- B+ P5 u) a; ?/ O <name>hadoop.security.group.mapping.ldap.url</name>; B0 V4 \! m/ {6 V/ U3 K
<value></value>/ d) H# y2 |0 t4 ~; q8 b
<description>
9 r6 |/ z1 H; v! f The URL of the LDAP server to use for resolving user groups when using
9 C3 h' V1 N: r/ l the LdapGroupsMapping user to group mapping.. P3 Z# O- N9 z$ Z$ m% t+ m) }
</description>
- V; D& ?1 _" R; \" C</property>7 H7 Y+ Q4 U2 @" Y& t O
<property>$ s8 n8 C1 {9 t' |0 U
<name>hadoop.security.group.mapping.ldap.ssl</name>; A1 x+ p" r# F, X" R/ E+ x) h/ g
<value>false</value>! e9 L2 Z' ]& ]& t9 I
<description>
3 E" ^4 z0 c. o+ y) G Whether or not to use SSL when connecting to the LDAP server.
( S5 _( r4 B' i </description>
! F/ B; {# Y& ]4 p }: S/ w</property>
6 T0 S& t; D! M `' x<property>
/ x7 I9 T8 ~2 w9 Y8 D) @ { <name>hadoop.security.group.mapping.ldap.ssl.keystore</name>! u- h; j7 _8 n$ }
<value></value>' O! Y+ h( u; v; e3 ~' l# x
<description>/ o, H; g5 I% r) k2 F' T9 c3 D& F
File path to the SSL keystore that contains the SSL certificate required+ d0 D' ~% w- b' L$ |0 |+ _
by the LDAP server.8 c! d7 b$ n' l
</description>5 y5 W$ I1 E s4 x
</property>9 T3 S _! o# J/ p
<property>3 i! l/ P0 C3 A1 ]: M7 t# H# o
<name>hadoop.security.group.mapping.ldap.ssl.keystore.password.file</name>5 H0 x& A$ r- Y/ h$ o- t) c
<value></value>7 @# U1 s; V J# O
<description>
% N7 A( H0 o; H2 U) `- A$ Q The path to a file containing the password of the LDAP SSL keystore. If2 F+ p6 {, ?$ r( S# I0 `
the password is not configured in credential providers and the property8 r1 D% `' \, D& l
hadoop.security.group.mapping.ldap.ssl.keystore.password is not set,
- s, G) P) ~3 y0 ~5 g7 q0 {9 F" ] LDAPGroupsMapping reads password from the file.
: f! ?8 [2 ^: G" i7 J IMPORTANT: This file should be readable only by the Unix user running
' k! H2 f2 H8 e" k% Y the daemons and should be a local file.
+ ?* {1 a# x2 `1 @, o </description>) w; x5 O8 A5 N3 g
</property>% n4 ^+ K9 |! ?, d' f
<property>
/ U: H0 g' I) n <name>hadoop.security.group.mapping.ldap.ssl.keystore.password</name>/ ~# u# ~' L2 T' n k9 ^
<value></value>: T+ K9 T8 B4 E# z/ l
<description>
6 \; G0 \2 e1 r The password of the LDAP SSL keystore. this property name is used as an
3 P4 W$ K9 |5 Z6 \4 ? alias to get the password from credential providers. If the password can+ S# m! Y* P1 R" I* H1 D
not be found and hadoop.security.credential.clear-text-fallback is true
! o- |2 s! U* ~& [ U# p- T( E | LDAPGroupsMapping uses the value of this property for password.4 q: ^: c3 w- M# f7 P+ B$ p
</description>
. z- d6 U' C) Y" t) W8 ?</property>
$ M: H- S9 [4 a0 t% f<property>" e! x8 R3 F0 w7 `+ ?- I
<name>hadoop.security.group.mapping.ldap.conversion.rule</name>
- `0 K8 P% p5 w2 u <value>none</value>& M+ z! ]4 y7 p# _/ k2 G
<description>
" J4 |+ s$ ]! |9 c The rule is applied on the group names received from LDAP when
% P3 o8 L+ ^) s: @ RuleBasedLdapGroupsMapping is configured.
; g7 j0 v" M. G6 z/ v/ K Supported rules are "to_upper", "to_lower" and "none".1 {% I" o% S( T$ X1 m M
to_upper: This will convert all the group names to uppercase.' ~/ D7 Z- o2 j6 K! v0 v1 C
to_lower: This will convert all the group names to lowercase.5 `- s% y1 v7 S4 X8 K5 q
none: This will retain the source formatting, this is default value.
% T H8 s5 ~1 r) c </description>; m/ ?7 Y. ]) Z5 q" v
</property>; E. h2 J; T8 O% w5 L
<property>1 ^5 w* d2 P9 y" f B0 s
<name>hadoop.security.credential.clear-text-fallback</name>
5 G3 E& z- k7 U8 I <value>true</value>+ v" C7 Q5 b2 x- P
<description>2 S/ X; f$ y# M1 Y
true or false to indicate whether or not to fall back to storing credential& ?. g+ Z' f3 v
password as clear text. The default value is true. This property only works! O4 t" g8 ^1 k- z# ]
when the password can't not be found from credential providers., G/ H) N& B+ C6 K4 @
</description> P1 n% l, K2 C! l0 i+ F) \, {
</property>3 T6 P$ H9 s7 G
<property>
* e; E4 y! C/ u* @* y+ H <name>hadoop.security.credential.provider.path</name>! F7 Z3 q8 f1 f# _
<value></value>3 G; i0 E9 ^; v2 @% n1 u
<description>
& A( o* C5 v" A0 G6 u3 S A comma-separated list of URLs that indicates the type and7 A0 c# n5 L k4 E4 p
location of a list of providers that should be consulted.
4 N- x. u6 y3 \3 H </description>
. Q" z5 d% L" o- `. i L0 e</property>/ J6 p/ P( A' @6 V" r' w
<property>
6 A$ @1 B, M7 P. c' v <name>hadoop.security.credstore.java-keystore-provider.password-file</name>! F. P ?2 u* Z% @- [8 N7 F
<value></value>
+ T; H& g) M" _% o3 l: q <description>
. F9 n; Y" m9 Q3 m, M8 i: [' f5 n1 f The path to a file containing the custom password for all keystores
* q& y+ W7 a3 k8 m that may be configured in the provider path.
1 k- j/ m3 g- v9 v3 a( p, p </description>7 O! l$ T: L- j) c
</property>
0 b9 c( Z6 T! A, n<property>
* Y/ v" U' c( b0 y Y <name>hadoop.security.group.mapping.ldap.ssl.truststore</name>
, [: E9 s1 Z* Q' i) `- c <value></value>0 a( T* ~8 o; `9 ^ Z- f
<description>' J2 k3 u; l+ j
File path to the SSL truststore that contains the root certificate used to3 \2 U+ z* \- c7 ^% G, U; }
sign the LDAP server's certificate. Specify this if the LDAP server's4 y5 Q% T+ E" O* G5 q1 F- t
certificate is not signed by a well known certificate authority.
* k* r5 k0 Z. j% T2 \! i6 o </description>: E$ {1 Y9 d n3 L% ~
</property>& H' c6 T& i" |; b
<property>
5 a, y6 K: ~; n7 z. {% l <name>hadoop.security.group.mapping.ldap.ssl.truststore.password.file</name>+ k( [4 k: V9 R8 x. d
<value></value>7 }" G9 G d3 j. l! ]
<description>$ B$ I' W# Y6 t$ o- w) `8 m
The path to a file containing the password of the LDAP SSL truststore.
$ C5 L- y1 Y$ L9 E3 w IMPORTANT: This file should be readable only by the Unix user running
, x8 O, H) V5 U& x the daemons.
# B n# n! |- w) I. |( D </description>1 R8 t$ F& u' N& @3 z( S
</property>
3 W6 z, u9 u7 n; ~: m* u<property>( i/ N, X( K' C
<name>hadoop.security.group.mapping.ldap.bind.user</name>8 o6 c4 L. d) j; p+ y6 w
<value></value>
+ a9 T6 S8 ?" l! R/ H; M <description>) D# |8 K. b9 ?; S. g; D& \
The distinguished name of the user to bind as when connecting to the LDAP( S0 R" ?9 a7 f Z
server. This may be left blank if the LDAP server supports anonymous binds.
; I& E @$ I7 ], n% J </description>5 Y# |8 a; A8 E/ o0 H4 b5 S5 b# S* i
</property>
- E. e! Z9 t6 E; W K$ E. w<property> O: g) ]8 ~, y3 m" J) i
<name>hadoop.security.group.mapping.ldap.bind.password.file</name>
4 q( _" f9 q; t* Z5 W <value></value>
% J* K* S. h2 u) v$ m <description>
" a8 q1 q% G/ D5 E The path to a file containing the password of the bind user. If
6 {5 o1 A' _/ T the password is not configured in credential providers and the property
. ^& |/ c! V |* T1 G5 _( h hadoop.security.group.mapping.ldap.bind.password is not set,. Q% Y& s( N/ v6 }
LDAPGroupsMapping reads password from the file.) i- n2 V8 O7 j- l% D: {& U G; j9 C
IMPORTANT: This file should be readable only by the Unix user running
) X( Q$ d( b, e k$ ?; K the daemons and should be a local file.
! T+ Q% Y1 V! h; T" i </description>
5 W, h( ?7 J5 s4 {: l2 ]7 N</property>9 b2 o$ L! S$ P: P5 b# F
<property> v- K A% _* E2 ^& Y
<name>hadoop.security.group.mapping.ldap.bind.password</name>
! ^+ k/ m2 _$ o2 w <value></value>
3 m& p1 x! B' g) \6 X- r <description>
+ K* X3 c1 C/ J9 A! W The password of the bind user. this property name is used as an
3 `+ n# P& r5 i l alias to get the password from credential providers. If the password can; A$ b0 n' S. |, e d( y
not be found and hadoop.security.credential.clear-text-fallback is true
+ n7 Y5 e+ `# h! l8 C7 f% h6 t LDAPGroupsMapping uses the value of this property for password.
# s: N4 G# M( X' D. L0 N3 {: Y/ f$ A </description>
* k$ F# G9 y3 `; S |* e, \& w$ _</property>: {( h! c3 }1 o( x
<property>
" o' Q9 M6 w! Q6 {8 q+ { <name>hadoop.security.group.mapping.ldap.base</name>: M2 u0 Z, [6 q: ^
<value></value>9 k! v" V: _- G1 x0 g0 b
<description>
6 f: L& r" ?7 ~; o& [5 N The search base for the LDAP connection. This is a distinguished name,( z# z5 p8 E. {( }9 A2 L
and will typically be the root of the LDAP directory.8 n; s+ m0 S9 Q8 i8 l7 V0 K
</description>( u* L2 P% x1 p6 Z
</property>
/ C) N5 L7 R% h6 e) n<property>; Q1 ^0 n$ @2 [0 _; f" \
<name>hadoop.security.group.mapping.ldap.userbase</name>% f1 W2 Q; H; E! K9 p7 I
<value></value>$ T! p# n+ R. d, a9 H/ K
<description>
! ]( `$ ~) x" h7 ~+ L3 V The search base for the LDAP connection for user search query. This is a- W( {$ c/ w' U. U/ z
distinguished name, and its the root of the LDAP directory for users.
$ x, V. u0 `. L! `7 \3 q If not set, hadoop.security.group.mapping.ldap.base is used.
" u/ N/ _) {. k% e* n- p4 M$ E </description>8 r+ K( u3 E3 M6 m
</property>
7 T. B, q/ O4 p3 i: P! y<property>
# s e8 r/ s+ x* ~$ }* G <name>hadoop.security.group.mapping.ldap.groupbase</name>1 d3 I% \4 {! x& q
<value></value>! W- k/ ]3 }. A- O; ?4 [5 y
<description>
' J# c' {0 R' X7 f% j2 l+ O The search base for the LDAP connection for group search . This is a
9 C6 s. S2 K8 @9 ?2 R: K g- t distinguished name, and its the root of the LDAP directory for groups.; R; e$ g8 p" b1 q
If not set, hadoop.security.group.mapping.ldap.base is used.9 V+ I" ^ J0 v; }8 v6 N1 A
</description>
4 F: G% \& U* }7 l/ P# Y: {! X</property>
7 L. g2 a- b+ t" G+ m3 q6 \, k<property>
+ U) x9 J4 [8 f) ~3 b <name>hadoop.security.group.mapping.ldap.search.filter.user</name>+ z; k4 o7 |3 k& I: ^% E. k; K/ E
<value>(&(objectClass=user)(sAMAccountName={0}))</value>- b4 j& g# e- k
<description> r% X/ r1 S7 q8 D
An additional filter to use when searching for LDAP users. The default will( e3 n1 d! E: q S
usually be appropriate for Active Directory installations. If connecting to% E- [% N P5 l4 o. d4 ~- F e
an LDAP server with a non-AD schema, this should be replaced with, {" j% Q0 h7 n8 @$ n! E4 D2 L. o
(&(objectClass=inetOrgPerson)(uid={0}). {0} is a special string used to, h E0 c0 R+ b. I
denote where the username fits into the filter.+ R/ V3 [5 i4 ~( ]9 C9 h
If the LDAP server supports posixGroups, Hadoop can enable the feature by& f+ L4 G1 a) d9 j3 H
setting the value of this property to "posixAccount" and the value of
) V7 {0 f. V+ u8 D* @0 l the hadoop.security.group.mapping.ldap.search.filter.group property to5 }2 j7 `7 O- b1 e
"posixGroup".( ]2 J! }( V; k/ r8 v, t, |
</description>
, d# W! Z# u+ q# b</property>
9 H2 _! }8 L( [4 q8 B5 E# S) j<property>
, F& H. {% r5 B3 n, Y2 q: e <name>hadoop.security.group.mapping.ldap.search.filter.group</name>
/ L: T/ b9 U9 Z <value>(objectClass=group)</value>
7 x( v6 e9 n% C7 u) K <description>) O5 `5 U4 r% N! ]' N! {+ C- q
An additional filter to use when searching for LDAP groups. This should be- T( Q( X5 C0 u- ]' E0 b5 o$ R$ }
changed when resolving groups against a non-Active Directory installation./ Q8 W+ D6 {3 r X, J
See the description of hadoop.security.group.mapping.ldap.search.filter.user
3 j" x0 p- t" ?, R5 O1 r" ~ to enable posixGroups support.( { e3 i/ B3 l. V8 _
</description>
+ q$ X4 d' B9 I7 y" e</property>( _' `+ `5 E# ^4 X# G- @
<property>
/ y" ?+ D: o& I$ k8 @8 q/ | <name>hadoop.security.group.mapping.ldap.search.attr.memberof</name>
, ~$ G) @! s5 E( \: L1 M% S" V <value></value>+ D0 V, S# o9 ?$ D1 y. X
<description>
9 j$ b9 m' S0 Y The attribute of the user object that identifies its group objects. By1 M- K2 j! ~7 @3 p$ Y
default, Hadoop makes two LDAP queries per user if this value is empty. If/ n' O" O9 z* e# ?5 f+ x
set, Hadoop will attempt to resolve group names from this attribute,/ ]: g* o/ M% Z! Z b' ~
instead of making the second LDAP query to get group objects. The value
% g' N( S( i1 l1 E should be 'memberOf' for an MS AD installation.
2 |1 W# Q3 \6 F) Z+ I o </description>* h, }: K0 i% o9 `4 K4 D0 B
</property>) ]9 ]4 |' |$ A
<property>2 g* D! H3 i& K! A) U/ v! D
<name>hadoop.security.group.mapping.ldap.search.attr.member</name>' ^1 ~) q% n9 Z2 S# A9 B
<value>member</value>/ F0 w7 e$ `$ \# e5 o9 n' m- f
<description>- `! i: V% W. W6 o7 p% |
The attribute of the group object that identifies the users that are
4 a( _- h2 Z& U members of the group. The default will usually be appropriate for
6 E1 a! C- v7 ]" F( ?( x4 r1 e, e any LDAP installation.
$ Z' J; e* P# U1 y n </description>
' J9 N0 x/ s; a9 j7 N, H' W4 m# F' T</property>3 c, d3 d7 h5 Y) i2 K
<property>
* @3 C/ j, t ]: P0 ]$ v <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>9 E1 t$ @5 y+ p: v4 \
<value>cn</value>
" q5 Y% t& k4 _/ {& Q: H <description>
# ?4 l: P* d* p' j0 T" { The attribute of the group object that identifies the group name. The
8 B" U) K/ B5 U default will usually be appropriate for all LDAP systems.
4 G/ Q$ l0 B" U- M; u" X" U </description>* q" P! k h8 K) ~) j. k
</property>
) m! t( }! S. M: ]" p<property>
. b! a. l- [% b) S <name>hadoop.security.group.mapping.ldap.search.group.hierarchy.levels</name>5 f2 I" i: @' i" N
<value>0</value>
, Q- |* Y8 J( n! | <description>' Q) s {. y9 h- N+ a3 t
The number of levels to go up the group hierarchy when determining
3 p9 Y! k! ^1 @4 N! t0 P which groups a user is part of. 0 Will represent checking just the; t& U2 S! W/ j' K
group that the user belongs to. Each additional level will raise the
; [" r6 _. ?/ A5 g; ?1 d7 X( { time it takes to execute a query by at most. B- n. l8 j* S' |
hadoop.security.group.mapping.ldap.directory.search.timeout.
6 C1 W& y) `* S The default will usually be appropriate for all LDAP systems.
1 F/ r* p# E% Z( V- w3 {+ S! | </description>
) }8 ^- s4 B: ?4 b</property>- t" P; U2 |: v0 a3 ]/ [
<property>
5 w4 e7 Q( A( k8 O( G) z <name>hadoop.security.group.mapping.ldap.posix.attr.uid.name</name>6 n- Z/ {6 ?% S1 u( D
<value>uidNumber</value>4 y4 B1 Z% J- u5 y8 h8 T
<description>
/ W' d& Y! Y: r+ _ The attribute of posixAccount to use when groups for membership.) t g! z1 p) X
Mostly useful for schemas wherein groups have memberUids that use an
. a6 m( J v& G/ S7 K attribute other than uidNumber.
( O, j7 q; m+ j7 x( _# a </description>/ N) N$ O4 G' E2 \7 Y
</property># ], x. T. ^) t6 C) {
<property>4 p+ N; i) H7 ?, l3 z
<name>hadoop.security.group.mapping.ldap.posix.attr.gid.name</name>, B+ A: s2 x) X
<value>gidNumber</value>3 `* O \- Y8 S% l2 g
<description>8 `# M5 l9 M- o; g+ E
The attribute of posixAccount indicating the group id.( i. H, u% v, {$ d6 S1 a
</description>
f0 O1 s) t6 C3 ]+ ]' q</property>9 s# i" F) j+ v4 L( W
<property>- Z6 k3 H! V3 w$ Z: ?: Z S
<name>hadoop.security.group.mapping.ldap.directory.search.timeout</name>
R6 L( a6 L% I- W- o) J8 ~6 w <value>10000</value>9 l( k, c; Y1 r! s% a
<description>1 d9 {* Q, x; i6 p& E9 r
The attribute applied to the LDAP SearchControl properties to set a1 {0 o8 H/ g( V6 t+ m8 l+ t/ o W9 ^
maximum time limit when searching and awaiting a result.
! }$ f1 X. C% t4 b9 O4 x Set to 0 if infinite wait period is desired.
3 w9 V. R9 @1 t6 A. g7 Q Default is 10 seconds. Units in milliseconds.; \6 i' T( A4 ?5 ]7 P
</description>
* b: ] Q0 S3 J# @+ o9 W' {9 Z8 T</property> A, R B- s1 B/ S G g9 t
<property>
. j' c& W f# T <name>hadoop.security.group.mapping.providers</name>) U/ v# H4 c" H% t# ?# l8 s9 b! V
<value></value>
. M7 J3 M# Z) b0 z" X5 | <description>( f3 ]% p3 a$ V8 M
Comma separated of names of other providers to provide user to group9 |0 Q" z, k2 h# W. T
mapping. Used by CompositeGroupsMapping.* s6 s% K; J2 E) G$ G
</description>
8 ^+ ?+ G9 D6 @1 u0 @</property>$ n0 x9 ~. ^# g7 B/ F
<property>6 z: T6 Z& c! m/ ]4 a2 }: ?
<name>hadoop.security.group.mapping.providers.combined</name>
u I& ?+ S8 B) t. i* \ <value>true</value>
+ G' l P) b: B$ T1 E1 r% e# N <description># @, h0 D( k% F) u4 I
true or false to indicate whether groups from the providers are combined or
& {; [- v$ Z2 w& n8 M9 I3 q* e not. The default value is true. If true, then all the providers will be
" f, s" {7 s3 C2 [ tried to get groups and all the groups are combined to return as the final# c; a/ u) O6 q& j: W
results. Otherwise, providers are tried one by one in the configured list5 L' W( O- J/ c+ v) F$ @
order, and if any groups are retrieved from any provider, then the groups
4 L( |- r: N4 e& l% X$ l will be returned without trying the left ones.3 |: S9 L q9 ~" j2 G
</description>8 f. M# E% w' `( ~
</property>
1 g; [$ [6 y3 n4 Q% I6 t1 f<property>
+ [0 S! q9 @) Y3 D <name>hadoop.security.service.user.name.key</name>/ p6 p5 y9 u. a* S+ ] h! m
<value></value>
: W' y+ H' z' o* d7 `6 h <description>
3 g0 v N b. n/ B For those cases where the same RPC protocol is implemented by multiple. }+ J* q* u) d
servers, this configuration is required for specifying the principal
3 P6 p6 \. @/ u& u% d+ I name to use for the service when the client wishes to make an RPC call.
. o( o" D! x* d$ P* I% _: {5 { </description>
5 S2 z( n7 i; @4 T# R$ Z$ I; o' h</property>
$ P5 y" @6 t1 T } t( s2 f! d6 ~ <property>
6 P$ Y% C8 r3 G7 y" j <name>fs.azure.user.agent.prefix</name>
' x; u) t1 G6 [; c* G <value>unknown</value>$ R1 L6 D! V }0 ]
<description>3 B8 Q# l" z; m2 p2 H/ n
WASB passes User-Agent header to the Azure back-end. The default value/ @* Y, f* r3 \8 K; c, y
contains WASB version, Java Runtime version, Azure Client library version,
' X& B/ t8 d4 E; z0 P3 b and the value of the configuration option fs.azure.user.agent.prefix.7 q* m6 h& e! q. h7 T/ |' f' a, f$ X
</description>
5 X# J1 M" l" f \. b I </property>
* t, j1 D2 ], }+ B<property>
3 V p6 Y# @5 S/ ] u5 T8 i z; N <name>hadoop.security.uid.cache.secs</name>, C( [" w; B6 {5 }4 \
<value>14400</value>
# Q; Q4 U9 G f; D* ]: q5 ]2 _ <description>) [* a. b5 A- D/ I, w
This is the config controlling the validity of the entries in the cache
' X2 j# z( e3 L/ C9 M containing the userId to userName and groupId to groupName used by3 C; }: o4 E5 e! f9 m8 m9 R
NativeIO getFstat().
) {$ R# N6 V i" U* e </description>1 C k A/ g+ G
</property>
# \- B! w3 T( W<property>3 \5 n( E8 s* ^' A
<name>hadoop.rpc.protection</name>
& s' ?% k8 @0 @& \ <value>authentication</value># T3 ^% T. ?, p* r
<description>A comma-separated list of protection values for secured sasl' B; V8 K; O3 F, J& W
connections. Possible values are authentication, integrity and privacy.
8 k: Y/ {2 b' B authentication means authentication only and no integrity or privacy;
0 ~2 h1 `6 F: z7 r integrity implies authentication and integrity are enabled; and privacy _1 s' W3 {( K2 X# Y
implies all of authentication, integrity and privacy are enabled.+ D4 ^# F E' J0 U+ D3 R. t
hadoop.security.saslproperties.resolver.class can be used to override5 w" L$ ]1 s, D1 c4 ]
the hadoop.rpc.protection for a connection at the server side.9 N0 s- {! ?8 c0 I
</description>
; Y) A+ h7 a: E# w</property>
1 x h! F6 b( e<property>+ W6 t( X H [! ^" C0 M% @
<name>hadoop.security.saslproperties.resolver.class</name>
$ e. _* {: \! e <value></value>
0 W* n4 T: c2 E& m/ o+ V <description>SaslPropertiesResolver used to resolve the QOP used for a
. A5 S E# Z$ a* Q% m9 m connection. If not specified, the full set of values specified in- t- M$ q5 C6 ]1 y- m/ n
hadoop.rpc.protection is used while determining the QOP used for the
4 |6 l. ^$ A! @ connection. If a class is specified, then the QOP values returned by
3 i+ m& N; v9 B4 P- a the class will be used while determining the QOP used for the connection.% U" }: \! E) {6 g% [) R7 x. Q4 U
</description>- K5 Z! [* r% [/ X/ p
</property>8 u1 [$ t/ W" }! V& V7 v2 N6 v9 Z4 L
<property>. L( ]/ }6 I; D- d2 d7 d& {! O9 |
<name>hadoop.security.sensitive-config-keys</name>
4 U- a% i/ I! i' z- C; H) d <value>
4 `/ i7 b: h( j# G8 N! @ secret$9 n) Y) N. m( L/ a& k
password$
f3 P/ p% G) K' F3 A" T, b3 P' ~+ t ssl.keystore.pass$
, i4 g, {$ _" o6 Y( E- D4 d0 ^ fs.s3.*[Ss]ecret.?[Kk]ey
+ b3 J8 ~' @6 h( m( G fs.s3a.*.server-side-encryption.key
# ?' ~& V% H( E5 J3 Z5 [: ^2 q fs.azure.account.key.*
# D+ P3 `9 E3 N8 Q/ F5 z# v0 p6 X credential$$ \: \0 u2 t, T Z& d% }
oauth.*token$- \; n, g; K# w; Q
hadoop.security.sensitive-config-keys
! L: c% F) c* v# n0 |8 m' f </value>; ~8 d8 p! G6 b" O
<description>A comma-separated or multi-line list of regular expressions to
E1 v9 A+ t" S# a" l2 H$ m( z match configuration keys that should be redacted where appropriate, for
# @# j- J2 H" Z, W+ F example, when logging modified properties during a reconfiguration,7 Y$ S# E3 s& D. V" Q
private credentials should not be logged.# v' |$ I5 y: W* S% V% ]' U
</description>6 W2 E1 v$ L4 o& n0 c- R' y+ |
</property>
2 i! L4 P' A( o' h9 F<property># q' U5 [( K/ z2 k
<name>hadoop.workaround.non.threadsafe.getpwuid</name>
( m( L# y" D( F: F. R* ]5 ` <value>true</value>, m8 ] \9 W5 b
<description>Some operating systems or authentication modules are known to
9 |& l* o- y6 k9 z3 E7 J have broken implementations of getpwuid_r and getpwgid_r, such that these% M: ~! \ y' E; `4 ~$ _) T
calls are not thread-safe. Symptoms of this problem include JVM crashes
: C- |4 k7 q+ x8 z: P, b# P; q/ b% k with a stack trace inside these functions. If your system exhibits this3 w( V2 Z3 s4 ?, Y2 n4 \- U2 @, a& L
issue, enable this configuration parameter to include a lock around the9 g( p# y+ [7 L* G$ @. f, e9 J
calls as a workaround.
% e2 b# }$ L4 P. g5 u An incomplete list of some systems known to have this issue is available
% L8 o2 H4 V; X6 S at http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations5 H6 ~ y/ c( _: y, T
</description>- R8 H0 U- v9 Z- u" o2 d$ c
</property>
2 B' ~) ?; Y! _- }" @2 y, T<property>. A- T5 b+ M- R w" `; N8 n% n ~
<name>hadoop.kerberos.kinit.command</name>/ b9 g, w/ H s5 F: Y
<value>kinit</value>, l9 y: p. z" L
<description>Used to periodically renew Kerberos credentials when provided
I0 P6 {8 c- P! e: W to Hadoop. The default setting assumes that kinit is in the PATH of users7 ` L$ g/ I3 a
running the Hadoop client. Change this to the absolute path to kinit if this
/ x$ q8 R! j! Q3 v# ^ is not the case.! \: `3 w3 s2 O$ A
</description>
, `5 N6 d7 W$ g6 ]- G7 v</property>
. K J4 m0 U( {" m% U3 Z<property>4 p' c+ Y; M) W( o2 U/ t
<name>hadoop.kerberos.min.seconds.before.relogin</name>* ^7 }6 w: s) w1 C' p
<value>60</value>$ C6 o+ ^0 `* G* z6 e: W
<description>The minimum time between relogin attempts for Kerberos, in3 }, _$ U2 K7 ^
seconds.
, n1 N/ K% K" n/ L( _ t: E' g </description>
! ~3 c1 P3 A$ o1 y" u</property>
! R+ O- T. y; [<property>
! p& t$ u, ]0 j/ ]& g1 N' I <name>hadoop.security.auth_to_local</name> g g. Z5 K" Z6 X3 T6 |
<value></value>
" E# T) p$ {6 K% @5 o: b8 x" B <description>Maps kerberos principals to local user names</description>6 a6 u5 q V- \- D
</property>
) t9 D& A# Y$ Q% @1 u$ r& F<property>
6 E; J- D0 |# Z* h# j4 n6 { <name>hadoop.token.files</name>
+ C* u, @6 Q( M: I) G$ y4 t* ~ <value></value>
" e/ L3 g$ J' a9 k# E; _# K5 F. G <description>List of token cache files that have delegation tokens for hadoop service</description>
( f7 h( I+ G5 j/ q( l b</property>8 W, c' g; P# r5 `
<!-- i/o properties -->* h5 W" t1 {' \
<property>7 g, k$ C9 p; e4 z5 m; j
<name>io.file.buffer.size</name>( k9 U+ v3 H/ w
<value>4096</value>; d- M6 n+ n$ ?8 [$ L% t6 t
<description>The size of buffer for use in sequence files.9 w" C" I8 u0 e" X8 Z: k
The size of this buffer should probably be a multiple of hardware
6 i7 C& Y. f9 f* I2 \: Y- H page size (4096 on Intel x86), and it determines how much data is
+ {. o, Z& \$ p( U+ ^$ O buffered during read and write operations.</description>) C8 P* M0 p- k
</property>
h/ m/ f8 [, M. W<property>& Z' ^+ h. D% l( a. A* e$ Q
<name>io.bytes.per.checksum</name>
% g+ P6 k- L f# v( x6 U <value>512</value>7 \) X* f# U9 h) ~8 }) ^
<description>The number of bytes per checksum. Must not be larger than w2 P: ?2 O* Y2 Y& d! d
io.file.buffer.size.</description>
- h( |& O* p7 J' { v7 C- O</property>
% b w7 s$ P% C0 z6 w<property>) `% y& e9 q" [7 L+ I# g' Z+ E+ n
<name>io.skip.checksum.errors</name>1 w, ?; S6 g8 @, |8 S) H ?; s. a
<value>false</value>
4 o/ N+ a% T' v9 K* {8 \) w { <description>If true, when a checksum error is encountered while, ^# e' K0 b4 P8 Y; k
reading a sequence file, entries are skipped, instead of throwing an J8 t1 Y/ W* C3 D
exception.</description>1 t" ]" c4 Y, Y4 q. |& w
</property>. V2 ^5 Z4 a: s, s4 P- `, ` }
<property>, [* u# T' G/ G/ G; A" H
<name>io.compression.codecs</name>
* b$ o+ m" R3 t3 \- O& ]5 Y# @ <value></value>" \. N, U% \. t8 d* U- H/ g) I# z
<description>A comma-separated list of the compression codec classes that can' _. F3 v# w8 ?$ c
be used for compression/decompression. In addition to any classes specified4 Z# P3 g" a7 G
with this property (which take precedence), codec classes on the classpath) e: m# {! ^+ e/ Y: J, w
are discovered using a Java ServiceLoader.</description>
0 C* m! Z0 @, p, o</property>
: v2 V% ?$ j; A<property>! {. h6 i/ b" R; e9 a3 O
<name>io.compression.codec.bzip2.library</name>
% N0 |- v$ Z% D <value>system-native</value>+ i+ p8 P7 o5 @
<description>The native-code library to be used for compression and
* p7 i5 @7 C8 y9 Z1 n9 c decompression by the bzip2 codec. This library could be specified' d" ^3 K* z, ^( t
either by by name or the full pathname. In the former case, the
/ Z' l S+ g1 p# o7 i# i+ p library is located by the dynamic linker, usually searching the
$ Z0 k, @8 H4 n- V7 y! W8 ] directories specified in the environment variable LD_LIBRARY_PATH.2 N/ Q4 @5 t$ s1 d
The value of "system-native" indicates that the default system
# E% C T; _ {, X library should be used. To indicate that the algorithm should
$ P3 s$ c; ~9 C* R operate entirely in Java, specify "java-builtin".</description>8 y; [8 o* Q8 {6 _; z
</property>
8 w, E& n1 u$ M7 o' \# y( j<property>
" G+ K7 k) @& X2 u <name>io.serializations</name>
# b! ]+ X9 q) i, ^$ L3 O <value>org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization</value>
7 S! u0 I Z1 d, x5 ~0 K <description>A list of serialization classes that can be used for
! @' T* a. J. U obtaining serializers and deserializers.</description>9 ?* t7 Z" h9 {1 s) R3 K
</property>: U3 ^3 M( B$ H4 H) p: _
<property>2 d d d: V Z, b+ @: K
<name>io.seqfile.local.dir</name>% H2 L1 n: w, c: N7 B
<value>${hadoop.tmp.dir}/io/local</value>. X, N; f3 s3 j1 Q1 e
<description>The local directory where sequence file stores intermediate, k2 {7 @+ O# Q& R
data files during merge. May be a comma-separated list of
2 b8 v( `2 P; `6 S: f B& B directories on different devices in order to spread disk i/o.2 e r' E4 v* \5 ^* V6 O- ]. o: Z
Directories that do not exist are ignored.) b# o) ^3 B6 y5 X: y; W) I
</description>& h: \( h3 C% b: _% W
</property>
& |/ B! F# ^' E# c% U% `7 }<property>
$ @2 K: v2 }! k <name>io.map.index.skip</name>5 `+ `$ `8 i: ?$ v
<value>0</value>! N ^0 O0 h5 @
<description>Number of index entries to skip between each entry.
6 F6 h( T& \! L% S0 m" ]% {6 R+ [ Zero by default. Setting this to values larger than zero can. ?8 H& Y7 ~; `" S+ w9 L+ l1 w0 P; Y
facilitate opening large MapFiles using less memory.</description>
- m! q' }+ `# I1 V t</property>
5 o# b4 H4 R B- I9 [2 |, g<property>- V2 G) M. \# |) P" r: E" Z) l- F
<name>io.map.index.interval</name>- V" b3 g* a4 P, C) M
<value>128</value>; U; i' }7 i( b$ a/ ^) h, k
<description># X& U6 U7 G/ p; s" v: Y$ i; `
MapFile consist of two files - data file (tuples) and index file' A* {) a7 b' v) K* f$ U
(keys). For every io.map.index.interval records written in the
; \$ V, @. h$ }6 v$ W6 O- x' O data file, an entry (record-key, data-file-position) is written; t M, t! ~0 O, s- I3 k
in the index file. This is to allow for doing binary search later. X' i6 Y, Y, ^4 z8 t2 O
within the index file to look up records by their keys and get their# T* ]9 a& ^3 P$ I; X$ i0 e; D5 O8 E6 r6 O
closest positions in the data file.
% W4 L# q8 f: \" J: C* q4 ] </description>: H$ t1 L1 f) ~: O- q
</property>
5 w$ N7 q3 e3 M; n2 x1 A<property>1 n' M7 ?5 s3 ?) g- O. y
<name>io.erasurecode.codec.rs.rawcoders</name>9 w9 O& Y e5 P" x" A, a+ b
<value>rs_native,rs_java</value>+ o2 A5 n& g8 `
<description>
( x3 ^ f0 \/ ?. ?7 o Comma separated raw coder implementations for the rs codec. The earlier
: M& ]4 r5 j7 J# | factory is prior to followings in case of failure of creating raw coders./ l. ]. {( P' j' ]
</description>
& z, ~3 a) s' Y) c) A</property>
0 S/ w/ a1 K& b: J<property>
( @; m' g, Y7 o8 q$ a, X <name>io.erasurecode.codec.rs-legacy.rawcoders</name>5 h5 d; B7 f; L
<value>rs-legacy_java</value>3 u0 Q4 }1 I0 r3 q" e, L# v
<description>
9 g. z& x' k) e; k/ W1 t Comma separated raw coder implementations for the rs-legacy codec. The earlier
+ T: e3 a+ C% \" C( A& @* a. `4 b, l/ r factory is prior to followings in case of failure of creating raw coders.
" |! A% L, D& a. _9 K+ Y/ w7 p </description>
7 p G7 Z$ c# T8 V4 R" @, z</property>
# {0 {( Q6 t6 ]3 b<property>
5 _3 K& a: v+ V- H <name>io.erasurecode.codec.xor.rawcoders</name>- X0 `5 r( b+ C# H. ?; k
<value>xor_native,xor_java</value>5 ]- |0 {4 s$ S+ |. ^- c& W; H
<description>
2 D* V0 D" M: k4 e: r- H% }) | Comma separated raw coder implementations for the xor codec. The earlier9 Z6 _3 ~- b% ], C5 {7 B
factory is prior to followings in case of failure of creating raw coders.; k a; a* H7 O; [
</description>1 C' T5 c- N ], n" O
</property>; Q+ Z2 t, s, b7 T
<!-- file system properties -->3 R5 P, j& I! r- ^
<property>
1 \( |/ O2 x; ]5 I: v$ R <name>fs.defaultFS</name>
% I1 f; W& c- _9 @9 A <value>file:///</value>) L1 Y5 ?. J% K* {! ]
<description>The name of the default file system. A URI whose/ i1 I* _# ~. p% j
scheme and authority determine the FileSystem implementation. The
+ z- O7 D9 u! @9 z uri's scheme determines the config property (fs.SCHEME.impl) naming
: s' x# w) ^8 `6 \& ]: d, x$ G the FileSystem implementation class. The uri's authority is used to
6 y q# X1 x0 H& ^& E2 l: W2 e V. D+ l1 V3 G determine the host, port, etc. for a filesystem.</description>
& ]* d% K1 `3 E</property>
) N+ a7 k" S7 y9 }, U9 N" e! R<property>
& ^2 f$ O2 M. ^( H) N' L. H$ j <name>fs.default.name</name>
$ f7 V, w9 k& _7 {5 A4 e <value>file:///</value>
+ x, t: D+ l% ` <description>Deprecated. Use (fs.defaultFS) property; p$ p- J, N) \/ R
instead</description>7 q7 \$ z$ i1 K/ K2 Z
</property>3 g* m) R$ J" o( Y" _6 P
<property>7 K3 ^9 ?8 L4 f
<name>fs.trash.interval</name>1 b; j6 H" e3 d c* J; P# @
<value>0</value>/ O( x! _. l% W0 [2 O4 ^
<description>Number of minutes after which the checkpoint3 r, x* n( @, L; Y
gets deleted. If zero, the trash feature is disabled.
3 p& e$ G- [" ~ This option may be configured both on the server and the
) D; V+ C& F1 `: Q& a client. If trash is disabled server side then the client
. H B1 {: Z0 V1 x side configuration is checked. If trash is enabled on the Q6 w4 y5 g8 N7 X
server side then the value configured on the server is7 m/ N& C6 j" {5 j: m( x% S }
used and the client configuration value is ignored.7 {; U) E7 m# d5 s, I: P+ @
</description>0 I+ k! L' W& {/ @
</property>! j: S7 J. O& }8 q6 r { m
<property>
" ?& ~/ [* Z9 q- V. S <name>fs.trash.checkpoint.interval</name>
# G4 E! U: Q1 U3 P9 E4 X9 t <value>0</value>
5 j0 H2 @$ S5 g1 u9 ]) o <description>Number of minutes between trash checkpoints.- S6 x# B; x7 V- h: d ?! d* {
Should be smaller or equal to fs.trash.interval. If zero,
5 `( E' |8 I: c5 U: I6 k1 r the value is set to the value of fs.trash.interval.
) q4 w+ i# n; `' W) ~2 t' R3 o Every time the checkpointer runs it creates a new checkpoint
( C& ^! ^; y- I p% G: } out of current and removes checkpoints created more than
( u* ]; [# W Z fs.trash.interval minutes ago.% k. d& U! e3 h- B: H% I
</description>
% N) r" _* S( g: `* w- K3 N</property>
( \8 `$ |" x- W# s' v* Z* y<property>
: D" h- i m1 X <name>fs.protected.directories</name>
3 [3 s2 D6 \; N' k <value></value>
# Y+ W) p8 v4 a! w8 P <description>A comma-separated list of directories which cannot
& Q; S9 K$ E9 T* r* _/ F0 T be deleted even by the superuser unless they are empty. This
1 H S3 h9 T$ h setting can be used to guard important system directories5 C' c5 g, _# r; a* F5 A" \* Y
against accidental deletion due to administrator error.0 N C& a5 w' g3 O X9 D
</description># Y- P! e8 y7 b0 j4 m$ \
</property>
9 y; m2 k* M* H% O* ~' w<property>, n/ E3 e K0 h8 N4 |" ~
<name>fs.AbstractFileSystem.file.impl</name>
0 b* P# X& L* O7 W <value>org.apache.hadoop.fs.local.LocalFs</value>3 q' |8 Y! L& d6 D/ O
<description>The AbstractFileSystem for file: uris.</description>
4 U ]9 q6 U: E5 A0 ]" g z</property>+ f( y+ V: {9 g1 M! E r6 u" c
<property>. R4 K) G8 E; w7 r2 Z
<name>fs.AbstractFileSystem.har.impl</name>5 z0 M1 G$ J' Y& k
<value>org.apache.hadoop.fs.HarFs</value>- h' `4 _4 b% j
<description>The AbstractFileSystem for har: uris.</description>* j$ w8 M8 R% S" q
</property>
1 T+ P. \7 Y B4 ~4 e+ [4 w1 e9 M6 E<property>0 P/ U) b/ L1 P+ a* P
<name>fs.AbstractFileSystem.hdfs.impl</name>, @. V$ j8 y+ g# O* x5 T7 F- Y
<value>org.apache.hadoop.fs.Hdfs</value>* E* a9 [% I Z& Q/ x" Z; E+ s8 |
<description>The FileSystem for hdfs: uris.</description>
3 J2 L) D' A$ x6 M. r- m( {9 }4 S</property>- k! \. _9 Z& _9 u
<property>. J2 n! g; O8 R2 ~ s
<name>fs.AbstractFileSystem.viewfs.impl</name>
$ F$ j5 P' M& ] <value>org.apache.hadoop.fs.viewfs.ViewFs</value>
- Y* h) U, R, @/ V5 t4 s <description>The AbstractFileSystem for view file system for viewfs: uris) A$ k' Z6 S. K j1 z% G( _7 f- i) Z
(ie client side mount table:).</description>
M Y6 Z# X/ `$ B6 v1 D, F</property> l. Y( O- Q/ Y4 ~1 A6 c1 R* N% E
<property>
: Q+ |) ?, _8 i: [2 b8 R4 o3 } <name>fs.viewfs.rename.strategy</name>
$ A* t9 Q9 T* s+ [7 M7 W9 j# n <value>SAME_MOUNTPOINT</value>
/ _* |: q0 [! B' U& Z/ h j <description>Allowed rename strategy to rename between multiple mountpoints.
6 w# e7 M; S6 h. ] Allowed values are SAME_MOUNTPOINT,SAME_TARGET_URI_ACROSS_MOUNTPOINT and# m; B3 f+ F& E! m
SAME_FILESYSTEM_ACROSS_MOUNTPOINT.
% @3 x; N: y* ^# G% n% A# q </description>
1 K: w7 Z. x- z$ w) V) L</property>
* J# b9 y3 r. h: [<property>5 k" T4 h6 L; [; x, ~
<name>fs.AbstractFileSystem.ftp.impl</name>' N, Y) H* A* g% p
<value>org.apache.hadoop.fs.ftp.FtpFs</value>! r0 u4 i3 t$ C0 X( b8 }3 Y& H6 ^
<description>The FileSystem for Ftp: uris.</description>
6 _) o' Z {/ t0 p; k</property># G* v: `3 D, r7 W
<property>
0 L% o4 m' R7 `" f& U <name>fs.ftp.impl</name>/ D: i0 Q3 Z; [# Y, @
<value>org.apache.hadoop.fs.ftp.FTPFileSystem</value>
2 E) ^, R5 O+ M. J9 v <description>The implementation class of the FTP FileSystem</description>
) U6 d( J9 u5 R& V# y. q5 @7 H</property>
$ j; Y- N5 a4 h$ K l" A5 B6 H/ N<property>% r$ h" v8 L! g
<name>fs.AbstractFileSystem.webhdfs.impl</name>1 M8 g" ^, \' p+ A* Z" _- n% o
<value>org.apache.hadoop.fs.WebHdfs</value>1 M {) ?2 R; s. g* t. K
<description>The FileSystem for webhdfs: uris.</description>/ w7 e- N* b# ]3 L, Q
</property>
8 m- D, s% ]: c. ~& X6 w8 X<property>
$ p& h$ z9 k( E1 ]+ L <name>fs.AbstractFileSystem.swebhdfs.impl</name>
/ m! O; _; N% \- |6 h8 R) F( S* Q3 R7 F <value>org.apache.hadoop.fs.SWebHdfs</value>8 s3 l( o+ G) X* n
<description>The FileSystem for swebhdfs: uris.</description>
7 q$ f( r8 V& o, |7 _</property>' \( x' h$ v6 n$ g# k, m6 k1 v& R' H
<property>) S, e% U# F! ?
<name>fs.ftp.host</name>
) a* i! U. b) u( Y, ~, F) e3 S <value>0.0.0.0</value>9 n( @- g- R, Q
<description>FTP filesystem connects to this server</description>
' b' l. h; j. j7 r9 Y! N</property>
3 h" ] z8 D, \8 f<property>
9 R! d+ W, X* B. M$ K <name>fs.ftp.host.port</name>
* g6 Q [9 s7 k) E& M+ m& e( S <value>21</value>
8 `( S" P% K$ H <description>- Y$ I( r2 X/ ^# H& p+ T; a
FTP filesystem connects to fs.ftp.host on this port
! d- k* A. e' U9 ~$ ?, W </description>/ \# b4 \ ^/ h+ b) |4 ]
</property>
+ t' u3 [) X& x* V2 X3 A<property>
; C7 n- `! f* ]' y; z6 {" F <name>fs.ftp.data.connection.mode</name>
$ N% c3 H% @1 p' \8 M <value>ACTIVE_LOCAL_DATA_CONNECTION_MODE</value>. j( j0 [' p) E0 D! e
<description>Set the FTPClient's data connection mode based on configuration.; t* C4 z# t. C7 c, u
Valid values are ACTIVE_LOCAL_DATA_CONNECTION_MODE,6 c0 V- p6 q, u9 M j! {
PASSIVE_LOCAL_DATA_CONNECTION_MODE and PASSIVE_REMOTE_DATA_CONNECTION_MODE.& |& p7 z) `8 k
</description>
# j5 d2 [# I! R# @% h/ p2 c$ o</property>
5 E0 O* b, N# o# t) U: W( g6 ?+ Y) s! w<property>9 r$ w9 H8 x4 T6 f
<name>fs.ftp.transfer.mode</name>
. k9 ~% R- X0 C1 y <value>BLOCK_TRANSFER_MODE</value>7 g4 \% X5 I; O+ S
<description>
4 a- a" m/ w1 J$ g Set FTP's transfer mode based on configuration. Valid values are
2 \2 @( \$ K9 U- y( i7 I STREAM_TRANSFER_MODE, BLOCK_TRANSFER_MODE and COMPRESSED_TRANSFER_MODE.2 x: Q5 @4 g4 O% S! Z/ ]2 G8 S% g6 b
</description>3 C. ~* w2 W ?/ T1 Z
</property>1 |! j' S0 P( m7 K' d4 n
<property>
7 m7 t% v' U9 W! P0 j! }$ W <name>fs.df.interval</name>8 r K* N% {& Y7 z/ G- {4 Q
<value>60000</value>
& l8 i, |! X0 _' K( v5 Y <description>Disk usage statistics refresh interval in msec.</description>( J' Y. S1 O V0 ]( |
</property>
' C7 z: V; D; A# N3 ?' [( U<property>8 }& Z- o9 k) f) c
<name>fs.du.interval</name>
4 D" |: \& v) N/ t4 ~9 w4 s! K <value>600000</value>, \6 `, \% {; i: D: U! E2 h: P( n
<description>File space usage statistics refresh interval in msec.</description>9 o+ _6 R- e& Y3 c- k; {3 x# [; Y
</property>
: U# i: `2 F) k# q" v<property>7 N" o: P. i( P6 z5 Y H
<name>fs.swift.impl</name>
% ]% B9 o. M9 B) u1 v$ C, t) r <value>org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem</value>
: |1 g( @2 L2 W/ M+ i1 R( ^+ F <description>The implementation class of the OpenStack Swift Filesystem</description>
0 e) }- _' ^. V1 l) l</property>
) S1 T( _+ Z9 W6 U7 u r<property>
: w8 T( O& Z6 q0 e <name>fs.automatic.close</name>
/ M/ z8 k! K; y/ i <value>true</value>
8 }7 r6 `( ]: L# p; T: q <description>By default, FileSystem instances are automatically closed at program
: f. j8 X$ m' ]5 a9 q8 d' O! U exit using a JVM shutdown hook. Setting this property to false disables this/ y3 V4 \1 a; r5 i6 a! _
behavior. This is an advanced option that should only be used by server applications
8 I ?7 v) j( O1 R5 S4 k requiring a more carefully orchestrated shutdown sequence.
; D' R- U2 w7 B3 R </description>, } _' S6 Q# C2 Z8 \- |
</property>. e8 F+ b, j( I8 I7 |, M. C* _2 p
<property>. G( y- r9 H8 _0 ?1 w
<name>fs.s3a.access.key</name>- m( W' X7 ^; \
<description>AWS access key ID used by S3A file system. Omit for IAM role-based or provider-based authentication.</description># g/ M: ^* `) g: a% b- f
</property>
6 i. x2 Q L4 N' o- u, Z% I; k<property>
9 ]/ q2 W4 W& p3 j <name>fs.s3a.secret.key</name>
i* w$ _5 G$ Q5 `9 U- p! _$ X <description>AWS secret key used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
+ U1 I4 O3 \8 j' s</property> Q3 }& n9 m4 i. n$ [! L
<property>7 u4 o% u% p1 ]" R) z8 v3 G) O
<name>fs.s3a.aws.credentials.provider</name>9 E3 d; V; V9 L/ A* W" T
<description>
* e2 p4 ^- G$ V5 t" V Comma-separated class names of credential provider classes which implement/ |& m6 J7 s- {2 F; X
com.amazonaws.auth.AWSCredentialsProvider.( D& } F1 C4 s* S% I$ s
These are loaded and queried in sequence for a valid set of credentials.
% U9 G9 t4 n9 M- Y0 |3 ^* @ Each listed class must implement one of the following means of) T+ U! c3 J2 |# j, B9 u1 ]; C
construction, which are attempted in order:$ j; Q ^- y2 J }$ m& g4 c; u
1. a public constructor accepting java.net.URI and
" k8 m+ Q* _9 T2 f- p org.apache.hadoop.conf.Configuration,
; i7 X; t* r& E& x$ @8 j 2. a public static method named getInstance that accepts no! ~$ ?/ H2 A g4 X
arguments and returns an instance of
3 h: U0 ~# z( | com.amazonaws.auth.AWSCredentialsProvider, or
0 D" q; ]' w* V2 c' @1 Z2 u% B: g 3. a public default constructor.* B- R O" n4 p" H2 K- Z
Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows
6 w6 z$ [1 m* y2 L0 U" T F9 h$ q6 z anonymous access to a publicly accessible S3 bucket without any credentials.( O1 ?5 [( `8 `3 `4 U
Please note that allowing anonymous access to an S3 bucket compromises1 a0 T# N. A; ^" K
security and therefore is unsuitable for most use cases. It can be useful; f7 K( ^4 G% W! q
for accessing public data sets without requiring AWS credentials.
$ n, `" ]5 j' Y* c1 z9 k3 t7 N1 Y If unspecified, then the default list of credential provider classes,
$ i5 C- v' M) j8 i; v# x queried in sequence, is:
9 `' h& f5 ?6 A2 @! f" x) K, ? 1. org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider: supports static8 L, s* K8 {/ O0 i* J1 W- C1 F
configuration of AWS access key ID and secret access key. See also0 B) e3 ^# A$ @6 D
fs.s3a.access.key and fs.s3a.secret.key.
O0 a4 m3 u7 s! ^ 2. com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports
$ u1 Z( g4 Z5 S configuration of AWS access key ID and secret access key in
7 f6 w# f0 g: X% W* T5 } environment variables named AWS_ACCESS_KEY_ID and* j; q9 ~( Q4 @% ~# S" Q7 \( f
AWS_SECRET_ACCESS_KEY, as documented in the AWS SDK.1 f" q+ V+ Y5 H" ?$ x
3. com.amazonaws.auth.InstanceProfileCredentialsProvider: supports use
6 S1 E0 X- W* y+ ^ of instance profile credentials if running in an EC2 VM.
- ~4 G2 Q/ h' _7 b </description>
8 x" `6 x3 J$ F5 K. o& {</property>
8 N2 o: _: j8 [5 z6 e& ]<property>2 Z# W; F2 G8 L0 y+ J
<name>fs.s3a.session.token</name>
& F( T3 ?" s. Q <description>Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
2 W* e# i; D0 t- B$ x8 P as one of the providers.
9 e6 O( [/ P g. v c </description>
6 r* ^6 i8 T; X</property>6 f' x7 V5 O7 r
<property>
4 x$ w+ f; O5 N& g0 u L' l <name>fs.s3a.security.credential.provider.path</name>5 @7 f; B7 `& A! ~0 {
<value />& Q, P" i6 b, T, s% R$ P* D# m8 B
<description>
) `; B- r. }0 M- m$ b- X Optional comma separated list of credential providers, a list) e; P; X( [# q
which is prepended to that set in hadoop.security.credential.provider.path
$ u2 @; d% }" w% X5 @! ^+ s! X </description>3 r1 a5 a* m. d; A3 g/ w
</property>$ v" ]6 x+ L# x0 L9 X/ N
<property>
2 H- F Q$ U& Z# Y8 e7 K8 o <name>fs.s3a.assumed.role.arn</name>" n5 C( H! g7 S& g9 K1 q6 X
<value />
/ ^2 s" A, E p. G <description>+ x( ^8 ?4 j% a( S' n0 p/ S
AWS ARN for the role to be assumed.2 o/ W) r ]: {/ j8 I" g3 ?
Required if the fs.s3a.aws.credentials.provider contains
: c+ y c- x) T% u2 {7 h org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider
. m$ U8 }& f1 M& J/ ? </description>
! N/ X; A' F' i& }4 v, V1 F</property>
; e/ w+ a/ f+ ?. f" H<property>
' i# n7 ?) P/ W" I <name>fs.s3a.assumed.role.session.name</name>! n- X6 n6 G {0 f$ [4 J6 c
<value />
/ i. \- O* G5 Y9 R: U% G2 S <description>
7 L+ P" Y2 O& E( D% P$ g" i Session name for the assumed role, must be valid characters according to
# H7 a$ M( G6 h, O$ W the AWS APIs.
# N5 A: J, X6 d( w' ] ^2 E/ o" X Only used if AssumedRoleCredentialProvider is the AWS credential provider.
8 W+ B0 l" z2 }9 `- H0 `# F' W2 N If not set, one is generated from the current Hadoop/Kerberos username.) S) \3 R2 ?5 J7 m
</description>
9 y$ e' k1 A8 s3 U2 f& l8 F</property>
& k% r% A( y. C$ |- F# `, U<property>8 j, u/ z, s( [" w7 y( Y
<name>fs.s3a.assumed.role.policy</name># f6 ^+ `/ ?, r$ a6 n/ _
<value/>" w$ |3 p( d- O# ^0 u( ~
<description>1 H0 _$ [4 @. i1 B
JSON policy to apply to the role.
( Y. d* r5 U: Y- O& t/ R1 u Only used if AssumedRoleCredentialProvider is the AWS credential provider.
1 ?$ [% M8 h4 n' L9 F! j" q </description>8 p, c: g5 Y+ S# ~- t
</property>1 c/ j1 z: [+ G/ p
<property>
. {$ N# d: R4 T0 s5 A- } <name>fs.s3a.assumed.role.session.duration</name>5 A. h. j) E1 u, ?/ @+ B: g
<value>30m</value>. H+ C: {0 w$ E7 |" t* B D/ Q
<description>5 e$ Q; f; o8 m, v2 \& c$ ]
Duration of assumed roles before a refresh is attempted.+ x, W" h* v6 V$ p! i" n& O
Only used if AssumedRoleCredentialProvider is the AWS credential provider.% h" i2 y2 D2 l3 i$ ~
Range: 15m to 1h. x$ T) H% n; g' R8 b3 F! L' |
</description>6 ^* i6 F& I! ?+ Y) T/ }- _$ v
</property>1 F. C( Y d1 t
<property>7 k( [( a( Y% R- [& W
<name>fs.s3a.assumed.role.sts.endpoint</name>
: b% H! h. u% ~. T' N3 b <value/>
! _/ S( J/ a. z5 ` <description>! X- [6 ?8 j) K0 p8 ~7 a5 j, H; @2 u W
AWS Simple Token Service Endpoint. If unset, uses the default endpoint./ d9 O1 _" R3 K: N
Only used if AssumedRoleCredentialProvider is the AWS credential provider.
: k3 C4 i t% u/ J* K </description>3 N3 G9 t! F6 ^/ C/ v& Y4 d
</property>
) \$ _; ~+ {" N6 @$ C<property>
* o# X. {" c5 x <name>fs.s3a.assumed.role.credentials.provider</name>
7 T* @' A- M4 s <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
# i+ q5 F( v$ G. J( N# ?5 A9 N2 k <description>
# u; P5 v0 w. T& d! { List of credential providers to authenticate with the STS endpoint and' ]4 X: V; C7 C9 Q
retrieve short-lived role credentials.
+ w. z' ?' |3 J# Q6 t8 [5 s Only used if AssumedRoleCredentialProvider is the AWS credential provider.
/ q! P7 o( ]7 \. ]! e9 V' E If unset, uses "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider".6 c5 w) }/ l! M* G
</description>0 n) J; ^0 R4 r9 K; g
</property>
( q0 M( L( v+ k" s<property>
e7 n0 B1 [: v <name>fs.s3a.connection.maximum</name>& N5 D3 E% `: H1 V4 G% L2 P) o
<value>15</value> Y3 N2 i' M/ j( y& e
<description>Controls the maximum number of simultaneous connections to S3.</description>9 |0 N& z* c3 T% c7 Q$ A8 x
</property>
: A$ j( e5 L' L; j<property>, u; U3 ~$ V/ @9 s
<name>fs.s3a.connection.ssl.enabled</name>
8 V* r1 A" D+ ]! g' {0 y1 g <value>true</value>1 q& c& \3 c- G, s! E
<description>Enables or disables SSL connections to S3.</description>
+ h3 P7 [6 \- @/ b</property>2 j( U% `9 v! a2 p, n
<property>- \9 d4 t: @" g; d5 O! o& p
<name>fs.s3a.endpoint</name>& t( `) ?3 G, o: N% U
<description>AWS S3 endpoint to connect to. An up-to-date list is$ N% M9 |$ t6 @9 x; x
provided in the AWS Documentation: regions and endpoints. Without this! @! K: Z% n7 o: z
property, the standard region (s3.amazonaws.com) is assumed. a* M9 U' M/ b" C* q
</description>
9 l3 m/ k4 S6 g' i* {</property>
) T. ~" w' N+ A" l<property>
# v% {* G0 b" }1 h& v; A0 \ <name>fs.s3a.path.style.access</name>( d. b: s6 W: k$ P5 P# T0 O
<value>false</value>
6 ?7 \' v" M `' N9 R- P <description>Enable S3 path style access ie disabling the default virtual hosting behaviour.
- W. h2 \" n* K1 a- x m Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting.
' e2 q9 P4 n9 [0 E9 o3 n5 U </description>
& o9 s3 X d6 X7 Y( p$ |: C</property>" H4 [, J2 i- Z$ a$ g* e
<property>% `3 S) K0 w7 _. `
<name>fs.s3a.proxy.host</name>3 ^5 g( b+ J. p, ^
<description>Hostname of the (optional) proxy server for S3 connections.</description>7 L" U$ V# j! U4 d# ?
</property>( V! C. d6 R8 @, [. J. H; B
<property>! P Q I5 F0 T5 O3 ]7 N
<name>fs.s3a.proxy.port</name>
( }& r+ [9 C4 l- o8 g5 Q. N <description>Proxy server port. If this property is not set
: g9 n: }, X. _: N) ]2 k% K but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with, i# E/ H! ^6 \5 {7 }
the value of fs.s3a.connection.ssl.enabled).</description>
( M; P) j( x' A- C! \$ [</property>" K5 _1 h7 |; Z: l* X. o
<property>' T! d& o, t" q
<name>fs.s3a.proxy.username</name>6 G) W$ f* | l" M) y' D* H
<description>Username for authenticating with proxy server.</description>
5 q+ O, I3 j6 V</property>6 ~6 Y0 i% i1 i# F, M5 K
<property>0 g5 l* o! V! z" L8 s& B
<name>fs.s3a.proxy.password</name>) v. g, x# Q6 N/ K
<description>Password for authenticating with proxy server.</description>/ [" A+ i4 w. T, C" N: p( J! Y
</property>$ x' {4 D- o5 Y# T% s4 a
<property>0 o4 v) T) F* T( @# @+ Q# W3 b, d6 s
<name>fs.s3a.proxy.domain</name>
B8 [1 ?5 f6 h5 U <description>Domain for authenticating with proxy server.</description>- k. r+ J( ^* x3 i, H0 \
</property>
( y5 ~6 {! n7 {+ Z<property>
! z6 g5 Y- t& }0 O6 X <name>fs.s3a.proxy.workstation</name>" @8 p& \# a6 D- ]9 T
<description>Workstation for authenticating with proxy server.</description>" N* x8 T5 N# }4 Y4 ^9 I& J% d7 T
</property>
5 V7 t- L" Q5 o5 u# t5 ?5 ~; v<property>+ V2 `( R3 [8 _+ g9 M7 K
<name>fs.s3a.attempts.maximum</name>* Z' C+ t6 O$ G' S( B# z
<value>20</value>
7 N- U* t9 I7 }% K <description>How many times we should retry commands on transient errors.</description>: h5 F9 T! H) R
</property>
3 L# M; M! I0 l" x2 n- S% R<property>
/ Y4 a; N2 `4 X/ f9 h" h8 s/ T <name>fs.s3a.connection.establish.timeout</name>- g$ n, P% T% z( s8 I9 q( D) X
<value>5000</value># {5 b, ?+ j' Y. s
<description>Socket connection setup timeout in milliseconds.</description>5 R' O6 m, v9 I( a
</property>. h: x$ b0 C# M" g* E- b& v
<property>2 [0 h- q+ y+ G7 g" g- e
<name>fs.s3a.connection.timeout</name>
0 @* K$ H" S4 `# b7 A <value>200000</value>
" B* w, n9 d- B6 f& _ ^ <description>Socket connection timeout in milliseconds.</description>1 X4 h U$ r' V+ X2 }+ J# k
</property>
4 O8 }. j7 }7 c<property>. b8 r, O# c! I6 u
<name>fs.s3a.socket.send.buffer</name>" P# n. W8 F& Y3 g# y5 M( m
<value>8192</value> ]9 ~8 S& d& ?6 g* D
<description>Socket send buffer hint to amazon connector. Represented in bytes.</description>
: L/ D0 M B# u7 W</property>5 x5 P0 P& d% U
<property>
( \) ]% g$ X9 r- ]& y7 X2 z <name>fs.s3a.socket.recv.buffer</name>3 m" N& Y3 S( H# [8 q
<value>8192</value>
9 u. Y! j' R/ c9 _2 \- `/ [ <description>Socket receive buffer hint to amazon connector. Represented in bytes.</description>8 M( N# o( n. {1 Q# I4 h
</property>
/ m1 N3 [0 V1 a( |# @0 N) K% P<property>* Z9 W# v+ w0 j7 X1 n( A
<name>fs.s3a.paging.maximum</name>
7 p$ X6 E; R' _ <value>5000</value>
7 p0 m$ u- W/ _ B2 m) C, N <description>How many keys to request from S3 when doing: k# p' u$ C! b6 t4 c' c
directory listings at a time.</description>! [- i2 |3 R$ ]& m
</property>0 h& G% t; o# p* c$ t/ [5 M+ E
<property>
5 W0 r( P( p1 {6 x* P" C, }( |3 m <name>fs.s3a.threads.max</name>
% Y; X% B6 I1 N$ x <value>10</value>; x+ c3 X L I- G1 V. [2 m+ c
<description>The total number of threads available in the filesystem for data1 r; X( R( P7 T' E* r
uploads *or any other queued filesystem operation*.</description>
$ N$ {% v; m0 o6 P9 S$ a8 W</property>
$ Z5 l# ]5 a' I1 H" J: _<property>
: Z$ H: Q8 Z( T n2 ? <name>fs.s3a.threads.keepalivetime</name>
- V. D( q6 A& G' n- V' P <value>60</value>4 w$ } j/ H+ x9 n" l
<description>Number of seconds a thread can be idle before being
2 J0 u; f Y% d, ^* R terminated.</description>
0 N# w2 f* O6 p</property> T# K( H8 p1 \, Q* J- y
<property>
7 h% H% Q% {3 `6 _ T) {" N <name>fs.s3a.max.total.tasks</name>+ B! h& t. D) H9 V, @2 h
<value>5</value>3 b6 Z# c2 T# ^" i$ k
<description>The number of operations which can be queued for execution</description>; T3 h4 V' F# {( t+ _# ~) D
</property>
1 m2 U, k) I/ X5 L<property>
' V( C0 w6 J+ |0 _2 | <name>fs.s3a.multipart.size</name>
7 O6 R( D# l' u7 z <value>100M</value>
& {# l6 g% R/ i <description>How big (in bytes) to split upload or copy operations up into., v6 B5 s6 d: O# e$ {2 L5 W
A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.8 b. J3 m. H5 ?. g2 {0 U
</description>6 \8 G) Z9 ]$ ^$ G0 Q" @: V
</property> [# p* y2 s8 e: @; B
<property>
2 I' W: z, S/ K8 O. T9 L& X <name>fs.s3a.multipart.threshold</name># K1 V3 b/ c7 I6 l1 T
<value>2147483647</value>0 F5 Z) N( ]' u( x1 x
<description>How big (in bytes) to split upload or copy operations up into.6 P( ^3 I. q% n
This also controls the partition size in renamed files, as rename() involves
9 X# B7 Z4 z& v! S2 X copying the source file(s).
) q# H! D X3 ^0 N. ] A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.5 P( V! |! q( Y. M
</description># e7 ?# e2 s3 v) R$ E" h1 z
</property>
+ @2 F- E+ {: n" X2 O1 R$ w& R- L2 m<property>
6 w& R3 U y" a; j9 H: g# a* @ <name>fs.s3a.multiobjectdelete.enable</name>
, f3 L7 P" M x <value>true</value>
, w# f r/ \& a! r( Z <description>When enabled, multiple single-object delete requests are replaced by
. |; o8 |/ t+ ? Q a single 'delete multiple objects'-request, reducing the number of requests." |8 H% D; ^6 N; @
Beware: legacy S3-compatible object stores might not support this request.
0 y8 T6 x6 L! C8 @) | </description>
1 [; ?8 v* w/ e7 J1 V</property>
" q2 |/ [+ F& V' X0 i0 V1 I/ h<property>6 i; o) ~, f# Z" v: H: B
<name>fs.s3a.acl.default</name>
' A0 i1 v* ]2 O+ Q! N1 M T8 d: k <description>Set a canned ACL for newly created and copied objects. Value may be Private,
' B- T4 b- B* N; y6 |. w PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead,
( b4 v V: L1 q or BucketOwnerFullControl.</description>% {8 |: ~& }. e& v" p
</property>
% [. O& [- z9 R- A$ T<property>" q& C. ~: L3 @0 O+ Y0 @
<name>fs.s3a.multipart.purge</name>, i- e5 ]/ @8 H7 t! H; v5 b
<value>false</value>' @" h% q! b( W/ t
<description>True if you want to purge existing multipart uploads that may not have been5 s% L" x7 D7 H# |/ n0 ~, v
completed/aborted correctly. The corresponding purge age is defined in6 g2 X! p/ P- c+ C8 o
fs.s3a.multipart.purge.age.
/ g2 w8 ~. k2 |2 o* M# O6 a8 j If set, when the filesystem is instantiated then all outstanding uploads
( o7 W0 E% v( e% y! Q7 J) |+ Z older than the purge age will be terminated -across the entire bucket.
/ E/ m7 ?) C" R This will impact multipart uploads by other applications and users. so should0 D5 q H! p8 a
be used sparingly, with an age value chosen to stop failed uploads, without
: S3 Q8 {- O, y* _* ? breaking ongoing operations.' k1 h/ ~& \( z
</description>
& [* w2 m4 g+ [1 X; s' e% Y</property>5 S5 ]% s! {* T: J7 g" T) g) B4 d1 y
<property># b4 K2 \4 ?' T
<name>fs.s3a.multipart.purge.age</name>, i/ _ \3 c9 ?+ J' S" T" s
<value>86400</value>
; N* h2 U0 Q2 M [ <description>Minimum age in seconds of multipart uploads to purge
% P- I! D! }! H' \# E6 y on startup if "fs.s3a.multipart.purge" is true
: B! P1 ^8 b+ i, c8 T! H. m2 W </description>) ?: p7 b( U9 ^9 p' F
</property>" V* i$ w8 u9 P, w
<property>
, y! r/ J1 Y8 C6 H& q% o <name>fs.s3a.server-side-encryption-algorithm</name>
# q: s9 ]$ [- o0 Y/ X" X <description>Specify a server-side encryption algorithm for s3a: file system.0 j6 P) W8 d+ o$ I4 L- \# b
Unset by default. It supports the following values: 'AES256' (for SSE-S3),
; M$ j a6 n9 k% s! `5 R4 J2 m# K 'SSE-KMS' and 'SSE-C'., ~! |$ |; L( K" _ ], Y
</description>2 B |& F' [1 H \: f2 `. L+ G
</property>
2 Z3 K6 N3 g0 b' Q4 G<property>: M: k/ j5 X% f/ s. I
<name>fs.s3a.server-side-encryption.key</name>) }- }! _7 A. g& _' e
<description>Specific encryption key to use if fs.s3a.server-side-encryption-algorithm
4 w; {$ T) M, N: z2 U has been set to 'SSE-KMS' or 'SSE-C'. In the case of SSE-C, the value of this property
5 \ z, i( ~4 R should be the Base64 encoded key. If you are using SSE-KMS and leave this property empty,) S0 q9 o( p. h7 j1 y4 b# g9 ^
you'll be using your default's S3 KMS key, otherwise you should set this property to
, \* _( v- N, f* J8 ^' K the specific KMS key id.+ i2 J) F6 W# j5 \- [
</description>( k& k' ^4 ^" C% |9 T/ x
</property>
' y- p( W! n/ o( F<property>& g. b# C$ U" j% H4 B$ i' N! L2 T2 ^
<name>fs.s3a.signing-algorithm</name>9 i* R( J/ Q) b+ j- E* o
<description>Override the default signing algorithm so legacy
$ B+ j7 t& a" H( N. @; K( W implementations can still be used</description># p% R1 [$ j/ H
</property>
( J$ G+ M" G2 Z2 B: ?' p' Q<property>" T. G9 H2 D+ y6 R! z# v6 ]" E
<name>fs.s3a.block.size</name>
$ h4 F8 f0 {5 m( U( l, {! V7 x <value>32M</value>
/ x7 G" F; c0 P) [* u% F9 H: I, J$ E <description>Block size to use when reading files using s3a: file system.2 A7 F* `$ k' L, S# @6 p
A suffix from the set {K,M,G,T,P} may be used to scale the numeric value.
! j% N7 ~" L0 q- u6 Y: @0 P' ] </description>4 p- Y6 N1 P7 v. O
</property>. s2 T/ }' n5 d8 K
<property>7 K" f2 R( ^" U0 u) O
<name>fs.s3a.buffer.dir</name>+ V6 }3 [: {- O& N t
<value>${hadoop.tmp.dir}/s3a</value>
! O5 z- s, Z2 i! q <description>Comma separated list of directories that will be used to buffer file) [3 D1 t I# Q9 }" K
uploads to.</description>
& S1 V3 ]" B) p# w2 u* Y; P7 I</property>
; ~; b3 o# s% H8 ?1 h+ w<property>
. m j4 a' \( B9 w6 N, _7 Q% I <name>fs.s3a.fast.upload.buffer</name># L r# M9 f t3 N7 H8 g
<value>disk</value>, K) ^& C5 B2 C3 l0 Q$ z6 m% T
<description># @) q: P) x3 m4 Y3 g" m
The buffering mechanism to for data being written.
: j& M, X j- d7 E Values: disk, array, bytebuffer.0 b4 Z q6 b% F
"disk" will use the directories listed in fs.s3a.buffer.dir as
, G+ T, @5 C8 r( T the location(s) to save data prior to being uploaded.7 i9 s. D, [. S8 ?! k3 c' R/ l
"array" uses arrays in the JVM heap
& q/ j/ b- C2 G6 x3 T2 x4 A# ~$ G "bytebuffer" uses off-heap memory within the JVM.
D3 T4 s- Q, z2 S6 d. A Both "array" and "bytebuffer" will consume memory in a single stream up to the number
$ d8 D7 H9 O( A of blocks set by:
" d* k$ z: h9 y6 b. x' W/ R fs.s3a.multipart.size * fs.s3a.fast.upload.active.blocks.
$ n+ {% T7 ~* T) r" N$ @+ [" \ If using either of these mechanisms, keep this value low& h) O3 l( D9 o' u. q
The total number of threads performing work across all threads is set by% D6 q& T$ H, |2 [
fs.s3a.threads.max, with fs.s3a.max.total.tasks values setting the number of queued
[" v* N8 m8 C0 J( R$ Q1 l' Z work items.0 p6 `4 ]* h( F1 o! C1 C
</description>8 X5 d. Q1 m2 |+ I! C
</property>
: y8 L1 Q4 i# {1 K4 I5 f<property>
, ]$ J/ l% V8 ]; m' C, X <name>fs.s3a.fast.upload.active.blocks</name>- H0 q, W; A5 h, P
<value>4</value>
1 M' Y1 ~8 l' z% z. t* Y <description>, j. F3 {" w) |, h
Maximum Number of blocks a single output stream can have# I: A; ]9 O4 ^0 i u1 X
active (uploading, or queued to the central FileSystem
5 D8 x* [- ?1 B3 a instance's pool of queued operations.
' L, j+ G4 h- X This stops a single stream overloading the shared thread pool.. Z# j4 K& R/ V" M+ O
</description>: {0 C; u3 K Y2 y8 d
</property>, C6 z# Z! V# Q4 D& i( u$ x
<property>
5 t$ j, e% m- w# n, u/ Y2 i <name>fs.s3a.readahead.range</name>
5 z4 T5 \' u7 R$ Z& E <value>64K</value>
/ f# g1 V4 I! G& o <description>Bytes to read ahead during a seek() before closing and
, X7 U) J1 T: [0 k7 z6 s" H3 { re-opening the S3 HTTP connection. This option will be overridden if. B) F& d- @2 i! Z& o( Y% O$ t
any call to setReadahead() is made to an open stream.
/ L& k, N" c; t2 A! O A suffix from the set {K,M,G,T,P} may be used to scale the numeric value., ^2 Z9 W4 K: P- v v/ }
</description>
- Y) }( H( \ @2 k</property>
( F% _+ I* g) H1 l& N<property>
# g% ^ x6 ^( A8 x4 N* E4 I <name>fs.s3a.user.agent.prefix</name>9 V% I5 ~9 W; a w8 v; a* s
<value></value>
7 f- Q5 ]8 q' ]6 G/ f <description>
- ^1 K1 `7 f. j- S/ G" f }- Z+ F Sets a custom value that will be prepended to the User-Agent header sent in6 i5 A. M* ]/ `6 }* a+ ?5 V" K% A
HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header# l5 O5 s% B! _; K* E
always includes the Hadoop version number followed by a string generated by2 p# M C2 X: n- S
the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6"." U! t3 O- I& C
If this optional property is set, then its value is prepended to create a
/ R8 c \% Z9 k; }" Y" C# L customized User-Agent. For example, if this configuration property was set+ v9 A0 B# F. G3 b! `. k* W! F
to "MyApp", then an example of the resulting User-Agent would be0 G' s9 F9 @3 q/ ?/ {
"User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6".
) X* H ]) V9 T </description>
+ G/ f% A3 B: L</property>
/ l4 y0 m1 G1 w<property> O2 U# B' b; i2 a
<name>fs.s3a.metadatastore.authoritative</name>* P8 A% P I+ A% o) ]& O
<value>false</value>
6 G# {. D( s0 A, F X <description>
: v O" |- ~7 W# ~ When true, allow MetadataStore implementations to act as source of
6 f0 s1 r Z" X& s/ w2 X4 a truth for getting file status and directory listings. Even if this
( ]' w; w' a$ C! ^3 O is set to true, MetadataStore implementations may choose not to
; c. E, }2 U4 O9 ~7 Y' c( {3 `1 p return authoritative results. If the configured MetadataStore does; x" k; E$ [( O7 K3 @. A
not support being authoritative, this setting will have no effect.) d* I9 Y1 c I- k0 b/ Q
</description>6 N ^. T( k) @' n- `: S
</property>; R* w4 z5 ?) m. ~5 Z" d# @
<property>
M2 u3 c( O' W& N* w* t/ S, w <name>fs.s3a.metadatastore.impl</name>1 w- z4 }3 W; G! ?- D' o! s
<value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
2 O4 n8 U/ ?- r. X <description>. f7 u. [ s% |6 \7 y; U
Fully-qualified name of the class that implements the MetadataStore0 {/ m3 d' G" T; @8 `# Y
to be used by s3a. The default class, NullMetadataStore, has no" @* K) A0 ]6 g9 F; }' T
effect: s3a will continue to treat the backing S3 service as the one
6 ` M! I7 w7 x- g and only source of truth for file and directory metadata.0 q3 v! |6 q$ E/ ?/ e) B
</description>/ y. W, _) c3 {, \+ Q
</property>
* e- ]' m# {' ^$ F8 \: u* s+ p, [! ~: g<property>" _$ w) x, D. \1 Q" w6 `! q
<name>fs.s3a.s3guard.cli.prune.age</name>6 {* c2 S7 N! |
<value>86400000</value>
$ m1 u; [6 H* X! y$ t& c <description>1 N. z* u+ P% b# }% F) l
Default age (in milliseconds) after which to prune metadata from the
: i' {! m& s- S. Z! N9 w metadatastore when the prune command is run. Can be overridden on the0 B8 }9 L$ ?1 `& b( ?) J" `# }
command-line.9 l7 M+ w5 D* p, a: n
</description>
" l' h1 w# n: c3 Q! i0 ~</property>2 j7 L4 m7 i3 {& ^& U, _. d
<property>) ]) J/ e5 U; c; s4 i4 M
<name>fs.s3a.impl</name>
9 e# s E5 M! G+ {! a <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>* V4 T5 ? c; H
<description>The implementation class of the S3A Filesystem</description>
! F/ P* k6 p( @" L# o7 q7 u1 m</property>
3 T' A8 _' E) e- M: m<property>1 O {/ r, h; X6 K( O) q
<name>fs.s3a.s3guard.ddb.region</name>
0 w* \2 ]' T% f/ G <value></value>3 z! o# c" I' ]* a8 T' m) Z
<description>
, b; ~( C7 j% P/ {' n( ] AWS DynamoDB region to connect to. An up-to-date list is
- A+ U5 w: A& M$ r" \' V2 J: g provided in the AWS Documentation: regions and endpoints. Without this- V; P- H! O1 m$ T5 }
property, the S3Guard will operate table in the associated S3 bucket region.* `0 E* j9 ^. p
</description>
2 a/ A8 {" L- ?3 H+ P</property>
, K3 T% d5 K) G+ q/ H1 o<property>
. `8 W' k+ {9 m- [! r <name>fs.s3a.s3guard.ddb.table</name>( E7 F$ N" J7 A2 m+ z% I
<value></value>
2 C; l+ }2 f% x3 E <description>7 ^! e# K2 \( {9 q' W4 I
The DynamoDB table name to operate. Without this property, the respective
9 A6 C4 z4 R& _2 N0 Z% i: F S3 bucket name will be used.
# z% c0 Y. H! w3 ?1 p6 f' @ </description>
, H% K: v5 l1 m</property>3 r, B; j- ~8 i+ K) w7 g+ r
<property>' B5 S# @5 r. M F8 g1 e6 ?
<name>fs.s3a.s3guard.ddb.table.create</name>3 W+ t' r6 b8 N
<value>false</value>. _8 [- U( d* T$ ~0 L9 W8 K) j. X z" T
<description>
4 T% E. r& r1 H7 |) m; | If true, the S3A client will create the table if it does not already exist.4 a! T( ^4 b1 ^! ~4 i, c5 K
</description>) l1 T& {# Y1 ~) o
</property>. Z% T& e) R* U" m1 b1 a, A
<property>5 n3 e' d2 d* S" W6 B) } U
<name>fs.s3a.s3guard.ddb.table.capacity.read</name>
% {* [. c/ ^; M ~& ?+ x8 O <value>500</value>
' T# ?( ?5 C t1 V9 ? <description>
7 R' S- G; l9 D; Y4 I9 z9 y5 c Provisioned throughput requirements for read operations in terms of capacity# s) g2 {/ A. Y8 k$ D Z
units for the DynamoDB table. This config value will only be used when
0 _6 p7 m7 Z! ^3 m7 j creating a new DynamoDB table, though later you can manually provision by9 x% R% L3 i& a+ n' [. j
increasing or decreasing read capacity as needed for existing tables./ u/ a" N- J, [+ D4 m
See DynamoDB documents for more information.( U* k5 P: d5 {" T3 A% @6 T
</description>* J1 `0 s1 F6 @! @6 ?" g' W
</property>
. N0 C3 _; Z" u9 x. i/ F x. Y1 |<property>7 m( n5 P+ U r/ \
<name>fs.s3a.s3guard.ddb.table.capacity.write</name>% d7 M% A& @7 |* O3 V: U( f
<value>100</value>8 @* v, p4 E& ~. {, n5 U# H
<description>' L- r1 m5 {9 U7 G( m9 \; S9 d
Provisioned throughput requirements for write operations in terms of
' w2 p: y* c- i2 V* p' p capacity units for the DynamoDB table. Refer to related config$ [# y- e* M9 C! E# X
fs.s3a.s3guard.ddb.table.capacity.read before usage.
6 |- Y: |" t) ^1 T/ I5 U </description>& K- @; O4 n2 Q. V5 p
</property>
$ g6 W" v( |5 @- @5 |# J# e<property>+ h m' k( {6 ?5 ~ i M7 r
<name>fs.s3a.s3guard.ddb.max.retries</name>4 r4 i2 ?, \. g" q
<value>9</value>
' V" A9 X+ }* k( S8 G5 C <description>: q% V% G T/ k# m8 B
Max retries on batched DynamoDB operations before giving up and
% \; ^ q8 c% U3 {0 x throwing an IOException. Each retry is delayed with an exponential
6 {; h& Y( A( m- P backoff timer which starts at 100 milliseconds and approximately" r* U5 l4 b- V" J+ G# W2 w
doubles each time. The minimum wait before throwing an exception is! d$ R2 c$ a, H1 f! o2 v
sum(100, 200, 400, 800, .. 100*2^N-1 ) == 100 * ((2^N)-1)# h# ^( B5 l$ ]" S( x! f, D6 b
So N = 9 yields at least 51.1 seconds (51,100) milliseconds of blocking6 P; |6 `4 q2 g3 i# @4 N$ I$ K
before throwing an IOException.
6 m! v, f9 W3 ^7 d/ P |6 i </description>
) X: K5 V, n, H8 k3 O1 A</property>
! Q% g: J# l* ^$ x) G+ k<property>
$ w1 }# ]0 H( c0 j0 t% C <name>fs.s3a.s3guard.ddb.background.sleep</name>' A7 m+ ^3 M* t7 ~/ Z
<value>25</value>
4 ?% A% l' y/ {6 A5 }4 o <description>
M. U& `; M5 Y Length (in milliseconds) of pause between each batch of deletes when- N2 c2 R$ A2 V; N! g
pruning metadata. Prevents prune operations (which can typically be low
/ T6 x7 B0 |0 D/ v) k priority background operations) from overly interfering with other I/O
! y) ~2 P% m5 ]" G4 a2 Y) ^& F3 r operations.( C" m x; C6 ?" Y. [
</description>
2 a' p2 S5 Q. a, a$ X</property>
5 E# u2 O$ r7 h2 P& g: x<property>
- _; D$ C2 f& ? <name>fs.s3a.retry.limit</name>
. s" V2 v4 a# `, w0 z" S5 R0 E <value>${fs.s3a.attempts.maximum}</value>
5 C& U5 j; f( l+ N5 w" r <description>8 [# ]- i9 `9 i4 p
Number of times to retry any repeatable S3 client request on failure,5 f1 m' f9 M5 W& \. n2 I' l
excluding throttling requests.
" {6 ?* X5 m# C: \2 I, L </description>& A9 J5 K( r6 |* R/ [( h5 g$ D8 W- i
</property>- k5 D( D- M) @2 e* C3 B$ P
<property>1 ?4 F8 b# b- S: z; s6 n
<name>fs.s3a.retry.interval</name>: @! q/ O( H1 ^
<value>500ms</value>0 n5 k" q) ^) \+ {
<description>
$ l' {- f- ?* F- i9 s# h Interval between attempts to retry operations for any reason other& u0 @6 _4 n) F7 w$ W
than S3 throttle errors.1 E& P/ V) ~6 V
</description>
) [5 D' g& D$ h% o) D# z</property>
4 {8 F9 E( }* U<property>' c3 A' U* Z9 r6 s/ x" E
<name>fs.s3a.retry.throttle.limit</name>
f6 q _- W' V2 g( I <value>${fs.s3a.attempts.maximum}</value>
9 T8 {8 n! {! |, [& l# ^ <description>
" K- s; i; t3 s9 R Number of times to retry any throttled request. I/ F/ k0 n3 w
</description>" c' `' E- Z7 O; X! x6 C' ]2 d
</property>6 @3 {2 h% r6 b
<property>) |$ S0 t* s% m9 s! ]
<name>fs.s3a.retry.throttle.interval</name>
8 H. [" ~1 o5 S$ { <value>1000ms</value>1 t1 e# b5 B6 Z, r) D5 K
<description>1 l2 ]. W1 T& j; m6 j
Interval between retry attempts on throttled requests.3 q+ B0 m# P) o/ e0 T6 _# C
</description>1 @% c) W# }7 W K1 F! \1 C
</property>
' o" _- j( B5 t. s% B7 y8 y- C<property> Z2 f, N$ h! I3 M% L8 i1 M
<name>fs.s3a.committer.name</name>
2 {; H$ z0 t' S <value>file</value>$ V2 R, ^* d) ^- o8 L1 ^ k' x0 Q
<description>
6 T" S! U/ W- {% _. U, P* _$ F Committer to create for output to S3A, one of:$ N6 A5 Y; F1 `1 a2 X& @6 t- k( u
"file", "directory", "partitioned", "magic".
) M5 d# ?6 z9 ?! [' w </description>
7 h' w1 l c; y! U</property>
) V# N% ~' s& d! i8 a* p' v" ^) M<property>
" `7 y0 y( F. d m2 I# ` <name>fs.s3a.committer.magic.enabled</name>9 y6 L! w8 w3 f0 D( y( P
<value>false</value>
5 l8 [: i# i7 A9 ^" i" _! y <description>; q7 M$ P6 l! t2 _: F+ [* v1 \
Enable support in the filesystem for the S3 "Magic" committer.
& B1 g, y( N. }; }" h When working with AWS S3, S3Guard must be enabled for the destination8 [2 B% Q$ b8 Y: f/ @, u2 p) X
bucket, as consistent metadata listings are required.2 n* O( A& E" g, G! w& X9 d- E% Y2 D. n
</description>
* Y# m6 I3 `$ c2 C</property>+ D5 _ |" Y1 F5 g* f$ t
<property>
# u |. j* s/ \1 i3 e, r3 M <name>fs.s3a.committer.threads</name>
% s) z2 M1 v; P0 ]- |6 u! U <value>8</value>
' l/ N7 h9 E# b <description>7 v, M, P" |5 O3 O/ \& v t& G
Number of threads in committers for parallel operations on files
' I; d: J! M/ {3 c) y (upload, commit, abort, delete...)
8 P8 Z( K d; ]/ ]; s' t4 d </description>+ E, O2 {) m6 @- ?* Y3 q. s
</property># ~9 g# F8 C' T/ {
<property>
, n2 s# e: u# [: L( E% `# n9 q <name>fs.s3a.committer.staging.tmp.path</name>
' u O) z# n2 T- U) }/ X( F6 t2 ?- H <value>tmp/staging</value>& U$ Y7 @0 s6 l# r5 B2 ?
<description>
9 g0 q2 E+ m" I1 X+ j0 n) s Path in the cluster filesystem for temporary data.0 ?6 n- n& C9 W2 B" o9 q
This is for HDFS, not the local filesystem.
1 T- {) c, R2 ?8 J$ ~) `% b1 A2 d5 } It is only for the summary data of each file, not the actual1 M1 y# Z( [/ k" P/ h
data being committed.
# T/ e9 U& H+ E/ K Using an unqualified path guarantees that the full path will be
0 k0 F3 n! g6 L; L generated relative to the home directory of the user creating the job,& m# y" T7 \9 u D
hence private (assuming home directory permissions are secure).; T4 [7 o0 F9 Z) C2 d. b
</description>
8 g# ]4 }0 k5 u$ k0 C: f( K; }</property>
. S' J3 ^( _9 p; i' x/ J0 ]<property>: z" H4 _/ @: `! b( g4 X
<name>fs.s3a.committer.staging.unique-filenames</name>
' r' O6 o5 w2 ~3 b" Z# ?- Q <value>true</value>8 r& Y* x9 d% O) \8 d g9 p+ F
<description>
. L. Z1 V) u7 J0 G Option for final files to have a unique name through job attempt info,! o6 [# W X3 ]# t, C4 v7 B
or the value of fs.s3a.committer.staging.uuid
/ T& a% H+ ?8 p When writing data with the "append" conflict option, this guarantees
" h/ k6 d: u/ ?" b9 y+ v# W that new data will not overwrite any existing data.
, A0 H# h/ Y( l3 e: z </description>
: } L7 H% Q0 G2 h, p</property>
- _, }8 o: y Q5 p<property>
* K4 O+ A& `! w Z0 u <name>fs.s3a.committer.staging.conflict-mode</name>4 t% c0 q& a) b0 E1 {$ `( D' @
<value>fail</value>- t5 {$ |$ g' ]5 h- A/ k. x' V
<description>6 S U, V) H+ Z# D7 q$ u
Staging committer conflict resolution policy.) t" K2 O/ \$ ?: d
Supported: "fail", "append", "replace".* m( @" k. k! }: ?! l; e
</description>
6 u' f9 N$ m* N/ L/ }</property>
* u; \1 n6 R3 U+ N<property>; B" d: L9 O1 d8 n3 l
<name>fs.s3a.committer.staging.abort.pending.uploads</name>4 [0 J! {3 W; W$ K. w) k, ~0 _
<value>true</value>
+ ~1 m* H- X4 |6 y6 ~ <description>, T$ a5 ?5 L g! N
Should the staging committers abort all pending uploads to the destination
" e* e9 p, o' k9 p4 u! U6 w directory?, z. E: o3 h* D( P
Changing this if more than one partitioned committer is
# A) O. H* K+ ?6 Y6 k- E; U writing to the same destination tree simultaneously; otherwise) y H D2 [& _
the first job to complete will cancel all outstanding uploads from the o! I2 `$ L8 B0 x9 p) d; u
others. However, it may lead to leaked outstanding uploads from failed: `) M. w) n3 `& S( s# [
tasks. If disabled, configure the bucket lifecycle to remove uploads/ A W) y# Z) }: S
after a time period, and/or set up a workflow to explicitly delete
# Q% @ O* a" ^/ p5 o entries. Otherwise there is a risk that uncommitted uploads may run up; l* w: W i% D* w6 Q) G3 E
bills.
% G( p3 U# g/ {6 a% J; p </description>
2 r' n$ c. G# c0 o" o</property>
# T! U# M/ E, w9 z( n<property>7 J: r9 J! S, U3 m% A1 \
<name>fs.AbstractFileSystem.s3a.impl</name>/ F% ^+ G# U1 I1 y& q* L( ^
<value>org.apache.hadoop.fs.s3a.S3A</value>
; T$ l F' I/ v0 d! y <description>The implementation class of the S3A AbstractFileSystem.</description>7 ~7 K% ` Y7 x. V
</property>
" F( `) @# t" b7 s. M. a/ d<property>1 H) a5 G6 X; l3 N# C
<name>fs.s3a.list.version</name>8 \1 p2 D4 c; ?& n0 z5 Y! Y0 T8 I
<value>2</value>! `# N0 ?, B! _
<description>2 T/ l+ C) H$ O6 p/ }; a2 Q
Select which version of the S3 SDK's List Objects API to use. Currently$ v) w' G0 c5 y+ B1 H3 Q
support 2 (default) and 1 (older API).
1 R: S$ u) x4 r4 d% v/ x </description>
$ w9 o8 k% x% D$ |; K- {0 m! k</property>! U. G2 C' V" a) }! \
<property>
2 g4 e& j+ `7 u1 v) Y <name>fs.s3a.etag.checksum.enabled</name>
( }. E& Q2 x( V <value>false</value>" X( ]3 Z$ W- A) h5 Z% _( w% P
<description>
' j6 H. i- J; R7 k" p Should calls to getFileChecksum() return the etag value of the remote* G2 N! U% y, D0 L) G/ `- d. R6 U/ M
object.1 s! n7 _: y# L e6 n
WARNING: if enabled, distcp operations between HDFS and S3 will fail unless
1 X( {5 T5 s* R1 l# Q -skipcrccheck is set.7 m7 o' I9 c: }( b# T
</description>
% f1 c$ c% V, t* z3 A. w( s</property>( J5 M- T* ?, s' j) h: E( s! l
<!-- Azure file system properties -->1 \5 e6 c8 j; c( J; l
<property>! } {! M9 q% G- Y S5 B
<name>fs.wasb.impl</name>3 q- N' w2 U6 i. j
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
) Y6 e; p: n- t4 R _ <description>The implementation class of the Native Azure Filesystem</description>- A. T4 t' w" i8 n% v
</property>
- w/ ^2 Z& j3 \2 }1 W6 g4 _<property>9 o; W5 e, }. W& E
<name>fs.wasbs.impl</name>% J- ~: B* w! h& H
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure</value>, S8 g( o& b# Z5 W. W. ^2 E$ w
<description>The implementation class of the Secure Native Azure Filesystem</description>- l! ?6 l$ P; ]
</property>
3 d0 o3 f9 y x. n# x( p" A. c9 u<property>3 H" P& P( L! @; A( A- G
<name>fs.azure.secure.mode</name>
/ y# ?5 o$ ?3 E <value>false</value>; ]% h" X$ b, I
<description>* Z6 t3 V( ~- Q w4 m0 {. V
Config flag to identify the mode in which fs.azure.NativeAzureFileSystem needs
0 f, _4 a& l1 p3 ? to run under. Setting it "true" would make fs.azure.NativeAzureFileSystem use
7 N; r( t4 j8 C7 A* I SAS keys to communicate with Azure storage.( R$ f5 x. G4 `" i; |! u& r! b
</description>. p6 y, Q; e5 b6 ~+ K* ?0 s
</property>
9 K& I ]6 m- t) l# ]<property>
; b1 L5 K* |, ]8 E( r# w <name>fs.azure.local.sas.key.mode</name>5 J$ i9 g+ T- S
<value>false</value>( h+ W( b+ a$ v* ]- k
<description>
( ?3 a5 m; d7 t t# c4 z Works in conjuction with fs.azure.secure.mode. Setting this config to true
& E# B8 X, M" o0 X: e0 i results in fs.azure.NativeAzureFileSystem using the local SAS key generation- r- S- {6 { V U7 N _
where the SAS keys are generating in the same process as fs.azure.NativeAzureFileSystem.
- O& _! p# } f If fs.azure.secure.mode flag is set to false, this flag has no effect.
0 y- i9 d, K9 b) b5 [1 J9 I/ f </description>3 e0 e$ y! Z7 O
</property>
6 k n! s& C+ f" n/ B: M/ K/ i<property>% U4 q5 b" W$ m, l* @5 P( m
<name>fs.azure.sas.expiry.period</name>
" e6 \7 I1 |! Q) l) V' S <value>90d</value>1 C+ {( k3 |6 y) X
<description>
6 P( C( w& k+ u* J The default value to be used for expiration period for SAS keys generated.
; }1 R3 N5 s- G Can use the following suffix (case insensitive):
' [; u: o/ Q1 T5 v ms(millis), s(sec), m(min), h(hour), d(day)
" Z/ F7 H4 v7 ~$ G to specify the time (such as 2s, 2m, 1h, etc.).
# ~. b) u/ l1 `9 B' k& r </description>7 |/ G. |6 A; y6 ~6 C; h# m" n1 d
</property>8 W; u3 W- K) s3 r
<property>' R' G" W: g" r$ _) x6 d. M; W
<name>fs.azure.authorization</name>4 F; e5 ]# r; E
<value>false</value>
3 ]% y3 S0 a7 b, m# r. p* f8 T <description># b4 R0 J/ A! L) F' j. J- P
Config flag to enable authorization support in WASB. Setting it to "true" enables
- n9 j8 g F2 ^( q0 N authorization support to WASB. Currently WASB authorization requires a remote service6 i$ \3 F" `. Z* k
to provide authorization that needs to be specified via fs.azure.authorization.remote.service.url5 x9 d0 j3 G2 b7 M4 H2 P3 V2 ]8 A
configuration; o T3 F$ L$ M4 [3 V B
</description>$ s) q0 w- P. d, H& L
</property>& C) B- G" E) r" r+ U0 [
<property>
& v- z* l2 @( \) p. [' g <name>fs.azure.authorization.caching.enable</name>
/ H) L) J+ _" y! p <value>true</value>& A; u; z2 i5 M, k0 ^$ z- s
<description>4 |+ v9 r+ Q+ z/ s7 W
Config flag to enable caching of authorization results and saskeys in WASB.
8 K2 l- ^* T& m This flag is relevant only when fs.azure.authorization is enabled.
: m; z( _- K: L! _( R </description>1 U: K9 l1 K8 ]$ U1 b
</property>; S0 {8 Z0 i# G7 N
<property>( V) `6 Q6 ?0 \4 E; ~9 x- K/ }
<name>fs.azure.saskey.usecontainersaskeyforallaccess</name>, W) s7 k5 Y- E
<value>true</value>' a# |, C$ o& ?9 N/ u
<description>; a' p' V+ k# X% @! B6 ^
Use container saskey for access to all blobs within the container.
$ c3 A( n6 I. |/ {, n) z8 l/ p Blob-specific saskeys are not used when this setting is enabled., d0 E/ c0 B) q2 S3 R3 q
This setting provides better performance compared to blob-specific saskeys.% Z$ f& Y8 s7 |( U2 \
</description>5 s) J$ {8 p. {
</property>
' p, t3 ^9 U) P<property>$ ]' I6 ^) j; K! p" w) I
<name>io.seqfile.compress.blocksize</name>% s- F3 r/ A0 |; _7 O( @# e. d
<value>1000000</value>
( ?2 C+ p9 }) L5 S: ]/ w2 G <description>The minimum block size for compression in block compressed
9 z: K4 D" Z- @- X. i- G5 J SequenceFiles.. ~; I' E* f2 y/ \5 t2 W! a
</description>* W) o) S! m) ~7 j' f
</property>% m7 W+ a7 E. `" i7 _! H
<property>* p0 p5 u5 O3 m' Y; j9 l( Q
<name>io.mapfile.bloom.size</name>+ h" g* r2 N3 ?' S1 [: B* \
<value>1048576</value>
}6 \2 x0 o3 `; g H# G, Q <description>The size of BloomFilter-s used in BloomMapFile. Each time this many7 N2 J" C4 d5 _/ G9 m1 }. o
keys is appended the next BloomFilter will be created (inside a DynamicBloomFilter).3 s& E$ s: p$ V' h+ s7 |: y) S) ?! y, I
Larger values minimize the number of filters, which slightly increases the performance,
/ M1 \$ J$ Z" }# X: A but may waste too much space if the total number of keys is usually much smaller
8 r9 a2 R) u5 }4 `5 q/ R5 W than this number. m, \: l S! U( `1 x6 R S
</description>
3 G6 r" G# Y: }1 v* Y3 H</property>
' g2 C5 M1 k& @* {; O<property>; P5 B/ D Q+ }0 Q
<name>io.mapfile.bloom.error.rate</name>
% z. I- m0 n! A+ S! b% u: i1 C <value>0.005</value>' l; Y; h% U0 |0 C/ r
<description>The rate of false positives in BloomFilter-s used in BloomMapFile.
; A: b7 m c! N/ O, t# U& `, H As this value decreases, the size of BloomFilter-s increases exponentially. This
- R% t8 u7 ~* [4 b9 f% H value is the probability of encountering false positives (default is 0.5%).$ Y" g+ J: U, `9 n2 S! j; z
</description>
& s# k* j l: ]+ l C</property>
, |' U8 a- s3 g2 A6 G0 n# z<property>/ L3 c5 m. M D, h9 ~ M- ~- Q
<name>hadoop.util.hash.type</name>6 Y; M6 H% `9 ?6 R! Y
<value>murmur</value>. [$ t1 @4 E4 j+ T& r X! H: G
<description>The default implementation of Hash. Currently this can take one of the+ M1 D8 f3 L8 V7 H- @& ?
two values: 'murmur' to select MurmurHash and 'jenkins' to select JenkinsHash.7 @. _0 [5 Z- B2 }; r) n
</description>
+ f$ C5 i2 V+ P7 t7 v8 v4 l1 |</property>0 D I) J( J% a5 `1 @! Z) G
<!-- ipc properties -->4 \, Z7 f: |3 J% O- [7 L
<property> L- K# K( N0 s
<name>ipc.client.idlethreshold</name>3 }8 ~3 o: v. t
<value>4000</value>
, D% ~) M. e# y) W7 n. d& j1 ^ <description>Defines the threshold number of connections after which0 E* k& `$ P% |
connections will be inspected for idleness.4 g4 e" r, z. c4 e" T5 p
</description>
2 n' {$ E& ` [/ m( D</property>
/ O0 Z* Y' K8 x5 r<property>
' D" d$ E) |! \% o$ L <name>ipc.client.kill.max</name>5 ~ S+ ?$ V: {1 o: t l4 V
<value>10</value>
# U" @2 B0 @& @7 ?7 d/ I1 S <description>Defines the maximum number of clients to disconnect in one go.
; ]8 _& b; J" J5 [4 L </description>- V7 J9 A9 E5 C$ b
</property>& j5 ]5 `) h" E0 \' S8 r0 c
<property>. S$ V) `) w: D9 t
<name>ipc.client.connection.maxidletime</name>- e F" H+ D% i" T
<value>10000</value># q; a4 d1 L: Z. k# \% h( m _8 G8 y
<description>The maximum time in msec after which a client will bring down the- E* `/ @1 ?( P' Y" y5 p. _; N
connection to the server.
' t9 ~! Z2 n( m0 m- K: S1 } </description>
o3 _3 J# y# F! l t</property>
8 a: j6 ]& J* i5 p" W" @<property>
, S$ D( g/ D0 B <name>ipc.client.connect.max.retries</name>
* e# \1 v4 r* O9 Y3 y' x <value>10</value># t6 X$ R" p6 l$ n. b: Q/ K
<description>Indicates the number of retries a client will make to establish
5 f6 |1 _' E) q2 @' r a server connection. f5 C9 p5 V" j- M
</description>7 G% y0 c* e4 T
</property>
8 z" X5 P9 m+ g' B) a1 V<property>% k8 I, N, q- l/ @& b" p
<name>ipc.client.connect.retry.interval</name>
+ `1 Y3 ]/ `4 C( ]" ^ <value>1000</value>
5 z8 f% W! S: I% l6 q) H' C4 v <description>Indicates the number of milliseconds a client will wait for
& I# m( ^" k; r) d8 k# S J before retrying to establish a server connection.
8 w7 }- w3 o, ]5 k </description>8 r1 y( Y4 Z1 E$ x% ?
</property>
- c$ l# |/ I; y" w9 `. Y9 w! F<property>
2 z8 n9 q0 N1 Z" |9 } <name>ipc.client.connect.timeout</name>
% W, @, H$ E, e: l4 K8 o, q) T7 k <value>20000</value>/ I, N( R y1 Y* @, R7 h; n
<description>Indicates the number of milliseconds a client will wait for the* j3 c" ~. }' T& D: R+ ]0 A: U
socket to establish a server connection. O3 Y$ A4 ^1 @6 ]' \' _$ e0 t
</description>
0 X7 s; a8 N; p8 H! o# x9 m( | |</property>
1 d* e' k5 f' p# |. R<property>4 R8 E; }' u: q/ ^9 d. D6 y
<name>ipc.client.connect.max.retries.on.timeouts</name>- s' b% K8 d, A
<value>45</value>8 `0 [2 {! E( J) ]1 [0 u7 t- H
<description>Indicates the number of retries a client will make on socket timeout
" t- T) o$ B, ]' U0 c8 L to establish a server connection./ N5 N8 I# F/ s8 G
</description>
; i: h% o- _+ p9 Q</property>
! A. N$ `: V& H$ a$ g/ m<property>+ X% N- {2 b4 {& E/ {6 ]
<name>ipc.client.tcpnodelay</name>
& u. `; d4 T2 x M7 r% d <value>true</value>9 V/ M1 `' X- h& a, F
<description>Use TCP_NODELAY flag to bypass Nagle's algorithm transmission delays.; u+ V, t3 e) g+ u9 Z4 ^3 o
</description># i8 u( q1 v6 Y7 S* d4 Q
</property>2 G" y0 g* ^' l/ n" k9 M! ^
<property>/ e6 f) m$ y$ ?0 l
<name>ipc.client.low-latency</name>
8 K3 W! X) |$ Y# W( {+ v <value>false</value>; p S0 O$ P! P( s
<description>Use low-latency QoS markers for IPC connections.. e. d& s* }5 c6 g" g! s) ^8 g3 {
</description>! R. G) Z$ U- w; {1 z% l% D4 Z% I
</property>
2 l9 z* v# }* t% U0 t<property>
( h* Q2 e/ Z# k' [5 g% e <name>ipc.client.ping</name>6 ~7 H! b0 H0 P/ ]( w) y
<value>true</value>- L; g$ E7 E3 |+ Z* I2 `
<description>Send a ping to the server when timeout on reading the response,3 M% j4 y# X$ J) s
if set to true. If no failure is detected, the client retries until at least
$ j! u9 |) z0 c# Z8 { a byte is read or the time given by ipc.client.rpc-timeout.ms is passed./ Z: G) x, `8 i E
</description>
8 `: S9 q: X9 k' M5 E) u$ q6 k</property>
, | C* e9 b3 R3 ^' H<property>; y, [; _2 z! A' _' A
<name>ipc.ping.interval</name>
' x* k4 s3 ]" C2 B <value>60000</value>0 w% L4 v. ?% a1 i, W
<description>Timeout on waiting response from server, in milliseconds.
9 G+ w) A2 k* e( y The client will send ping when the interval is passed without receiving bytes,
/ R, M1 v. @: N. m1 {+ y0 W4 A3 X if ipc.client.ping is set to true.
/ p3 Y: [/ X; A1 ~' I9 v- W7 { </description>
; h/ B: e* ]/ F9 [</property>
9 V o) K1 O2 z% e, `6 j* o# L7 M<property>
3 m0 P4 Q) ^" }+ B <name>ipc.client.rpc-timeout.ms</name>' \# i; L: a! Y; d5 p7 ^# }: c! d4 |
<value>0</value>) K- b, x4 s7 K8 c" k& n
<description>Timeout on waiting response from server, in milliseconds.0 E" E2 K6 R% l- @& y+ a. B$ n
If ipc.client.ping is set to true and this rpc-timeout is greater than0 K% L" Y, _) @- ]4 a+ \0 u
the value of ipc.ping.interval, the effective value of the rpc-timeout is1 e& y/ V9 { D k G; v, b
rounded up to multiple of ipc.ping.interval.
( l0 [4 |0 l$ o. ?# v2 q </description>
# W1 e6 G' D! w: I K</property>
/ f9 M9 [# J' `6 s7 O, q<property>8 O1 c" ]* h) \+ y, h; i7 V; f8 e
<name>ipc.server.listen.queue.size</name>
6 u) S9 A! ?% y, I7 g7 U/ _ <value>128</value>
) m+ T) k7 f* c0 A: D! z- B2 H <description>Indicates the length of the listen queue for servers accepting
7 M9 N# \# b6 ?% y2 Y0 I0 n9 { client connections.
0 p0 j& `, s) e! w R. x </description>, `) U. L$ z; A" C- T
</property>/ s8 }5 k* Z5 q1 ^2 \* u
<property>
% F) ]) X) m6 J* a- Z( V <name>ipc.server.log.slow.rpc</name>
1 ~2 R: e% _" M! V <value>false</value>* e: _$ i' M( v* m
<description>This setting is useful to troubleshoot performance issues for
V7 }) O J/ e- V: x' S various services. If this value is set to true then we log requests that
5 u4 e, I! W; A4 }: M fall into 99th percentile as well as increment RpcSlowCalls counter.0 S8 S1 f' p7 z5 k3 f0 K
</description>
/ h2 E9 p( L! W1 U) u</property>2 X3 Z7 [# U/ P" I8 M! O* a
<property>
3 I+ ?7 q/ O% K3 C) ^9 f% O0 l) Y <name>ipc.maximum.data.length</name>7 ?1 L* K, a, E7 q: r/ E8 H
<value>67108864</value>' w6 h9 c+ U- A4 r& w
<description>This indicates the maximum IPC message length (bytes) that can be) Q$ h4 g7 T$ s4 @
accepted by the server. Messages larger than this value are rejected by the
* N0 N' t, [+ r8 V0 ] immediately to avoid possible OOMs. This setting should rarely need to be" @: N3 z' r# A6 J: e
changed.
+ |' _" x0 c+ Y- b7 x+ { </description>
% I' h" |/ O$ L" m3 H</property>! `* }" n5 U6 j: a
<property>
2 ~' A' s& @/ l% F* O( e <name>ipc.maximum.response.length</name>
% v6 J2 b2 p, g2 c/ L6 a) m <value>134217728</value>
3 ~1 w4 v9 n3 [- R; v b+ H <description>This indicates the maximum IPC message length (bytes) that can be
& ]& z6 p2 f4 D/ f( h accepted by the client. Messages larger than this value are rejected/ ?- o6 L& C: J! y- }( l
immediately to avoid possible OOMs. This setting should rarely need to be `5 f: y9 \) N) e$ o* Y
changed. Set to 0 to disable.
! w4 m) ~* W9 Y2 z' D& r) }; [ </description>
0 }5 p& {7 ^2 q) x, e7 k4 H4 B+ @</property>
7 r* A# R4 Y o$ j) K<!-- Proxy Configuration -->( \% ~' Z2 W; ?
<property>: j1 Y9 W! J) d1 U! ?! A5 n
<name>hadoop.security.impersonation.provider.class</name>( \6 ~* I5 e T6 R2 M
<value></value>7 u" M- V% o' I: v! w
<description>A class which implements ImpersonationProvider interface, used to
& M Y0 g/ R5 S6 N% l! ? authorize whether one user can impersonate a specific user.( [6 D3 J- r& P# r2 c3 {3 d
If not specified, the DefaultImpersonationProvider will be used.& x' L5 k9 S5 Q9 p1 ~9 ?
If a class is specified, then that class will be used to determine
& X- @6 i+ H! h: c$ c( L the impersonation capability.2 U! J }: `3 L$ p H6 z
</description>
% p( R" v. o- P! T- ]! W/ n</property>0 J7 v+ C; ^- Z% j; e& a* I: A% n; S
<property>
/ ^6 N; i% V8 D7 m6 u/ a <name>hadoop.rpc.socket.factory.class.default</name>
$ H7 B* L1 x8 y <value>org.apache.hadoop.net.StandardSocketFactory</value>+ _+ K2 M# V2 `7 l* f! I' D
<description> Default SocketFactory to use. This parameter is expected to be
' r' Q7 n' J: _: l2 d formatted as "package.FactoryClassName".1 `3 Q c/ ~& Q: K" T* G) r
</description>
. ~+ X: d+ i( @. p</property>
0 A8 V0 z3 J/ d. E. ]+ W8 s<property>
7 I6 K+ A2 u! T* j% r/ K8 r) d <name>hadoop.rpc.socket.factory.class.ClientProtocol</name>
, t$ B: f; z( z3 P$ `; o$ ` <value></value>
7 G2 u0 H/ h1 u) E9 e: D- M' q <description> SocketFactory to use to connect to a DFS. If null or empty, use
( s( }- D: K' L" U9 U( h9 j& j9 n hadoop.rpc.socket.class.default. This socket factory is also used by
8 T+ X) A- t f o DFSClient to create sockets to DataNodes.
/ _) W# C4 z9 x# s- M </description>& R, {! V0 P& T2 Q
</property>! x% A. _" ^, l
<property>
) u3 g0 _( C' B5 o/ n <name>hadoop.socks.server</name>4 i' D8 f8 C* @, {' K
<value></value>0 B' m" _2 m% D( `. E
<description> Address (host:port) of the SOCKS server to be used by the
5 B& O2 e( r% o! O# W% _2 d6 F SocksSocketFactory.
+ Y* _. w6 ^3 t7 r </description>7 I# A' Q2 w1 w+ y
</property>
/ p! u. R, S1 |$ S, F: e<!-- Topology Configuration -->2 Z9 }. h. X; A9 W
<property>
2 K0 ], x( t" q" W <name>net.topology.node.switch.mapping.impl</name>
& y5 ~, f& x1 S <value>org.apache.hadoop.net.ScriptBasedMapping</value>
. S# b: F: C* t! G, D <description> The default implementation of the DNSToSwitchMapping. It
( |, v& V; b; \+ X% @# Q invokes a script specified in net.topology.script.file.name to resolve
( k5 A. F1 O4 d2 D1 {& ? node names. If the value for net.topology.script.file.name is not set, the
/ n! U( h' P- [/ H4 _3 W$ V default value of DEFAULT_RACK is returned for all node names.$ q) R2 Y/ P( {8 D+ k
</description>
0 Y3 |2 q. R7 [& E4 j</property>$ s5 a5 B! c4 r; O; p2 o9 B
<property>& m; m4 c. ^3 G3 g+ J
<name>net.topology.impl</name>
% ]. t" {) ?/ I$ @0 v! a+ X <value>org.apache.hadoop.net.NetworkTopology</value>2 B! E' b/ q. ^+ m6 z6 I; Q
<description> The default implementation of NetworkTopology which is classic three layer one.
# ~# e4 I6 P' C5 O1 u8 | </description>
5 h. R3 }# H' q ~% g1 Y8 ^ K</property>
8 W1 m4 ?3 t/ N: T+ Q3 j4 F5 r<property>; F# N, O" Q, K. c5 B e
<name>net.topology.script.file.name</name>
; H l% ^+ Q. D+ k4 C7 x1 x: X9 m <value></value>& b c7 J2 J1 o3 @0 G) R
<description> The script name that should be invoked to resolve DNS names to, }5 O% k4 |- c9 d+ d1 I8 ]
NetworkTopology names. Example: the script would take host.foo.bar as an7 D2 {" k9 K) m
argument, and return /rack1 as the output.8 s: e' e7 O3 `# P6 y
</description>) @# o* ^+ z" p
</property>7 F9 S% v* p, O8 |# I; w e
<property>1 Q2 z. ?: Q% Z9 }. I8 v. U3 v8 k
<name>net.topology.script.number.args</name>4 {3 g Q! S& v
<value>100</value>! |' {1 V7 m! W
<description> The max number of args that the script configured with
/ C% x% j, r6 d4 V% [* d net.topology.script.file.name should be run with. Each arg is an& g6 A' O7 p: |& W/ I1 ?; p
IP address.
4 C/ b3 ~: S/ d# J! I+ p </description>; a# g* } H% h7 d3 P
</property>: K [' W; h8 a1 e
<property>
' x! [+ G, L" ] <name>net.topology.table.file.name</name>2 Z! U& {, e% o) h" _4 `& f: u
<value></value>
7 s. d& U. m2 F; T <description> The file name for a topology file, which is used when the
' V+ I$ I" u# e, b( n net.topology.node.switch.mapping.impl property is set to- y5 @3 t/ Y' ?6 D8 m( `! I
org.apache.hadoop.net.TableMapping. The file format is a two column text
- f0 F) J6 Y2 [ file, with columns separated by whitespace. The first column is a DNS or
. ]9 P8 P* n2 Q' G! l* { IP address and the second column specifies the rack where the address maps.- m7 u' q1 M4 x' a
If no entry corresponding to a host in the cluster is found, then% D" |) }" f# y
/default-rack is assumed.
, k. D4 j3 x6 J, r/ N2 D </description>
! g, Q# Q) L# n</property>9 x( O3 V5 }$ L# i
<!-- Local file system -->) U/ l5 L) b) k6 C3 `
<property>
& @' x9 M+ ^8 b; r: f$ p; W5 l2 y <name>file.stream-buffer-size</name>! v0 }4 s; J* _; Z. N1 ^
<value>4096</value>* E9 r; i) @, L4 z& b
<description>The size of buffer to stream files." c6 F* s: r, x7 A6 z- A
The size of this buffer should probably be a multiple of hardware* H) V/ M6 V- P9 r$ d) y
page size (4096 on Intel x86), and it determines how much data is( E0 t1 D) [4 { r5 B
buffered during read and write operations.</description>
# t% ?3 @ Y3 u4 l& n1 {</property>
4 G, S" R% z( ?$ Q) A<property>6 m% B6 v$ a2 c0 f
<name>file.bytes-per-checksum</name>
, B5 N2 @" }- R/ [ <value>512</value>6 F4 g; i! B6 D6 A7 f
<description>The number of bytes per checksum. Must not be larger than
) z% ]" t5 M) d, w1 d file.stream-buffer-size</description>
) Y: @4 g; C) w' e# {</property>+ \; E* B8 w5 s; t1 M0 a2 ?* w: o; i! @
<property>+ J: Y9 y" d; Q" S4 u: x! s
<name>file.client-write-packet-size</name>. n, O7 Y, Q# O9 u- F& J5 [
<value>65536</value>1 I R6 ~7 X( B7 r7 W+ v5 C3 ^
<description>Packet size for clients to write</description>
, f/ z! K. {) m8 K- \</property>( {" T1 L `0 d! c
<property>
, C5 z4 g" I' `7 S <name>file.blocksize</name>% V1 s" Z; S5 H9 M, ~& y$ f3 c
<value>67108864</value>
4 W8 G2 B! Y/ N; I: E- Q <description>Block size</description>- k; Q* i, _# |2 W1 z0 m% Z6 I& ]
</property>
7 q5 S K' I i! H, A<property>
+ [4 g; J/ q6 \' ^$ ^% U8 m f z; y <name>file.replication</name>* c, G8 L, Z2 V, i; h" }
<value>1</value>/ Y# Q- K$ Y Y3 D
<description>Replication factor</description>
3 v8 Q5 d% p0 n0 I% l: z5 I</property>7 l+ c7 t( g" i5 X- M; w+ O
<!-- FTP file system -->
+ S, f6 `. Q% n2 e) F$ p" W' W<property>6 Q L7 L# A# f- B
<name>ftp.stream-buffer-size</name># ?2 ^- d% A; ]" e; [, |1 \
<value>4096</value>
, G4 {7 t" W% c r2 |; O. ~ <description>The size of buffer to stream files.& [/ U8 d6 a0 a: }7 V+ r& @
The size of this buffer should probably be a multiple of hardware
8 j7 B5 g j0 g7 A. c' Z page size (4096 on Intel x86), and it determines how much data is
! V& L( `; P" q F1 G0 D buffered during read and write operations.</description>
" c! Y; d; P# W$ U; m0 c</property>" S; g, c. L# y* x. V
<property>- ~. u+ \) E3 R' |
<name>ftp.bytes-per-checksum</name>/ q' a& @! R# G* V4 c5 ~+ E) N
<value>512</value>
1 ^ k k$ I6 V. ?6 ]- x <description>The number of bytes per checksum. Must not be larger than7 ]) H, x" P5 z' I: f9 F+ E2 v
ftp.stream-buffer-size</description>
/ ], a. d* B7 G' L; {5 N8 }</property> B+ S- d% Q* ]" W8 Z# ~
<property>
2 N5 {" T$ ?4 y& t+ g+ ? <name>ftp.client-write-packet-size</name>
1 r6 w- s s6 C <value>65536</value>
8 D" y0 o" c6 s# O <description>Packet size for clients to write</description>& d1 g; N; j2 Y) z
</property>
$ C" R$ m9 C) `0 a! b<property>
. Q H8 z2 q. r- `0 X <name>ftp.blocksize</name>
+ k# K) k8 v( ]% P <value>67108864</value>
/ l' B) W1 s0 b2 G: A <description>Block size</description>
' W0 G& M0 {+ g, m* p4 Y7 I</property>2 k" J, }* }, S3 Z$ Y1 W4 ]( A
<property>
8 L* |+ g9 q* @! z, w' ~8 D <name>ftp.replication</name>- B! x& N$ \$ q0 \/ r
<value>3</value>
8 A$ H% h& `( x% n, z" d% e <description>Replication factor</description>
# S: ?. ^" r! |% s" w</property>
K2 j# W" h2 M2 Q8 ~<!-- Tfile -->
4 T# V' I8 Z, ^: V<property>
) m2 q/ h+ [: ~, j; m9 x6 V <name>tfile.io.chunk.size</name>- e1 _0 X, a% _) P1 z
<value>1048576</value>
: y9 }: |8 k& n* { <description>
/ L' i3 Q9 K5 {- V3 R6 N, d h# S) z3 G Value chunk size in bytes. Default to
4 \; u, |: C3 _9 \1 b9 F 1MB. Values of the length less than the chunk size is
$ G: O* V, |6 Z2 ~5 K& f4 K- B guaranteed to have known value length in read time (See also" K* N: {' O) b# _$ e
TFile.Reader.Scanner.Entry.isValueLengthKnown()).
* j/ [; |7 l1 g( q$ J% U y& O </description>
6 M5 l7 ~2 D7 }! L% b$ i+ S* u0 b</property>$ A: u# P: H3 u% p- E" w! \
<property>
* M7 y+ ?2 ]0 l8 \$ i <name>tfile.fs.output.buffer.size</name>
2 ^4 q; R" G& e; U+ v <value>262144</value>
; H0 F. ~9 S9 `0 z6 O4 B4 K <description>
7 A5 I& C% r8 |7 { Buffer size used for FSDataOutputStream in bytes.
! c$ A- O6 ~) H </description>% N' n5 F+ C6 R
</property>
" Y' H$ v7 L# Y; f1 y h. w( t<property>
9 g9 Q, d& P# A" @8 t, v+ K& p, V <name>tfile.fs.input.buffer.size</name>
! i& ^# G! H# f% F. K <value>262144</value>
; g/ v7 b! d/ r* a! h5 K <description>
* c4 G& b d# C Buffer size used for FSDataInputStream in bytes.
2 U& U; I, v1 a) I </description>& W1 M) s+ \ m( g5 v& O: \% w
</property>- `! A( V5 I$ Z2 H; f& y4 k
<!-- HTTP web-consoles Authentication -->
: R% z/ s9 ?/ w/ ~: z- d- e) y<property>* R5 g3 U9 Y: D+ e! d
<name>hadoop.http.authentication.type</name>8 d! L$ P2 r7 [5 Z
<value>simple</value>0 T0 C# X$ {; a0 m1 Q* G5 n
<description>- M4 B9 j5 o5 v& i
Defines authentication used for Oozie HTTP endpoint.
' w) p2 \. \) p, F Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
3 C- H9 [1 J$ b, Q, q7 H# ]. A9 h </description>3 N% O" M, ^0 u* F7 ?: B5 N8 _
</property>
: k% H7 w1 s- b L<property>
2 w9 _$ ~* I/ ~" e) Y! o( X <name>hadoop.http.authentication.token.validity</name>, Q! Q9 a. ~1 b- z' a+ q/ Z
<value>36000</value>
) F' W+ e+ M) x u; ]! P <description>8 n) A5 \1 y+ }6 U* n# }8 M w. n
Indicates how long (in seconds) an authentication token is valid before it has: S2 t2 U1 C- M( l, e0 J, o
to be renewed., q( d9 V' f' q: r7 Y) n. [1 G; a
</description>
]6 [4 h. W+ y1 r: {; q4 c</property>7 B; w; f1 f j/ W- b1 Q$ d& i0 ]4 T; t
<property>
% ?& T x# n v/ z5 g <name>hadoop.http.authentication.signature.secret.file</name>2 V0 Q7 I& C% K7 ?) e0 r; p: b
<value>${user.home}/hadoop-http-auth-signature-secret</value>" s5 y; e& I% ]1 n6 c9 \
<description>1 a* h# ]+ Y5 A' Q
The signature secret for signing the authentication tokens., `9 c( c A7 Y: h8 w! t
The same secret should be used for JT/NN/DN/TT configurations.8 E$ h8 k: G. s- F
</description>
; d/ C" L4 O: a2 ^6 j; R</property>
/ Z1 p: U$ e% c6 J# z F<property>
8 ^& ~" _' M" v& _* A9 C9 J) G <name>hadoop.http.authentication.cookie.domain</name>4 q9 ^' [/ A Y) a
<value></value>- \1 V& w; P- J$ t+ }$ _. T$ ]
<description>
+ E$ l/ y- S" ]$ r The domain to use for the HTTP cookie that stores the authentication token.
8 X7 B( h9 k9 M, d3 ?! @4 v9 u1 K In order to authentiation to work correctly across all Hadoop nodes web-consoles
- o1 g0 R$ q$ M the domain must be correctly set.$ k6 `( F/ O6 k4 y* A+ S
IMPORTANT: when using IP addresses, browsers ignore cookies with domain settings.
+ O5 e( I4 s7 a4 {1 L For this setting to work properly all nodes in the cluster must be configured+ P+ b8 B8 {$ P3 ]4 v
to generate URLs with hostname.domain names on it.; q! K5 x9 c* C2 d
</description>
$ H% v, p& W- _</property>5 h$ j# h: ^" y+ ?+ U. A' l) \
<property>
4 A/ A4 m! Y. r' G3 z5 W <name>hadoop.http.authentication.simple.anonymous.allowed</name>2 u- l ]5 e' I
<value>true</value>9 i. c/ _# @5 v7 i6 W7 j
<description>
; L5 }: c& z9 h Indicates if anonymous requests are allowed when using 'simple' authentication.
l {7 g" @: H" o1 {( u2 T, X </description># @' M, ]/ t) |2 Z+ N1 ^
</property>' h# D! p0 ]: V! a' q: o. {- Q' k
<property>. N+ j% v- v) B0 x; A
<name>hadoop.http.authentication.kerberos.principal</name>
$ q% Z& U: [2 p9 m p |; { <value>HTTP/_HOST@LOCALHOST</value>
4 p) |7 ?. o" x- c <description>- M& i: L' [# ~! w4 g
Indicates the Kerberos principal to be used for HTTP endpoint.
- }4 w; X* g/ a1 V% P/ o The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.' H3 v$ ~3 \" N- f& M) K8 j) i
</description>
# k9 n3 z! l3 L( S5 x M' M& {</property>
4 r5 v: k* L3 D5 D) B' N<property>$ \$ r$ J$ d6 n( Y2 T( _( ?" c
<name>hadoop.http.authentication.kerberos.keytab</name>
( h0 A6 d- k6 J. h <value>${user.home}/hadoop.keytab</value>7 R9 ^% f: l; M9 u7 k
<description>
Y: Y% H5 p& {/ C; b Location of the keytab file with the credentials for the principal.
- Q9 M9 w/ J8 e# I) s Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.5 a! W8 I+ V R2 ~# A8 w
</description>
4 D5 o, r* P7 [</property>! r4 X- \& [. R
<!-- HTTP CORS support -->) j& p- h1 j/ \+ I+ \
<property>
+ W2 s0 C2 Y1 X/ {7 L <name>hadoop.http.cross-origin.enabled</name>. r" R) | o. [* [; G, E1 D; S
<value>false</value>2 `/ i c4 u/ A7 U. {
<description>Enable/disable the cross-origin (CORS) filter.</description>, W% r/ R" ]% ^( A* e0 }
</property>
$ _) o. d# d! z; N% P: \<property>3 D* a, q( M7 ]7 D) Y8 d
<name>hadoop.http.cross-origin.allowed-origins</name>; _" e3 n. C4 F1 X! Z- s
<value>*</value>
+ }; p+ }/ w) t, M: ?' E0 e9 | <description>Comma separated list of origins that are allowed for web services$ Q! f# D( {+ Z& Z
needing cross-origin (CORS) support. If a value in the list contains an" B7 `3 ?( q6 Y$ @
asterix (*), a regex pattern, escaping any dots ('.' -> '\.') and replacing9 D4 ]2 N$ J" e' U* b, V4 Z+ Q
the asterix such that it captures any characters ('*' -> '.*'), is generated.2 A8 |+ j! Y8 x, Z/ A' a2 t
Values prefixed with 'regex:' are interpreted directly as regular expressions,
( Z' q; p \# c$ W e.g. use the expression 'regex:https?:\/\/foo\.bar:([0-9]+)?' to allow any
( U' L# X* r, n origin using the 'http' or 'https' protocol in the domain 'foo.bar' on any
0 s/ g3 c+ \! @7 X port. The use of simple wildcards ('*') is discouraged, and only available for4 [8 T% A. L/ v! O' `: F9 B E
backward compatibility.</description>0 r% v% O k* ]' Q0 v5 x! |% {
</property>
3 i2 F U+ G0 \+ d5 ~. e* Z2 @+ P<property># J+ \# h/ e b+ y' J, z( O
<name>hadoop.http.cross-origin.allowed-methods</name>& O, j& x. y, k
<value>GET,POST,HEAD</value>& c* V( J' F( u/ V
<description>Comma separated list of methods that are allowed for web& f$ B2 K$ A9 i
services needing cross-origin (CORS) support.</description>
' ^% \$ I- P; A0 I</property>! {0 D* @. r* G; A& y" Q
<property>1 |. q |. ~2 S: v
<name>hadoop.http.cross-origin.allowed-headers</name>
* x: ^5 \9 Y5 E" o/ X4 ~& G <value>X-Requested-With,Content-Type,Accept,Origin</value>( i$ P2 X' q9 B. j1 [3 o* X1 _) j
<description>Comma separated list of headers that are allowed for web# e/ N/ C8 h( i0 Q& ^( O+ \
services needing cross-origin (CORS) support.</description>
3 d1 U/ p: m8 X) u& C7 p</property> j$ }( m/ x1 }; {3 x4 b- b
<property>3 ^( K! |" |" q6 {6 ]
<name>hadoop.http.cross-origin.max-age</name>
, o; |; n4 i2 s5 f* B. t <value>1800</value>
+ A; `6 I7 ]' v( m! r: F+ b <description>The number of seconds a pre-flighted request can be cached8 k: {# @+ K7 r/ ~+ d5 F- x" d6 ?* a
for web services needing cross-origin (CORS) support.</description>
, i9 Y6 D8 j/ S4 B6 k/ W</property>
9 p: m# ?! n/ u<property>
q' x9 k+ w: E+ X+ G <name>dfs.ha.fencing.methods</name>+ |* a, w9 [ |
<value></value>
% J* n" \9 M7 u& T M& K <description>4 @0 h- I% P7 |+ u6 D2 \' G3 A R
List of fencing methods to use for service fencing. May contain
; j/ X1 r2 G8 ^ C builtin methods (eg shell and sshfence) or user-defined method.7 d3 j' k* p# z) t5 I! Z
</description>
6 v4 \+ F( ?) S</property>
+ ]7 h2 i& d4 Q+ R# a<property>) t0 T6 W B7 ?4 Y+ g( L
<name>dfs.ha.fencing.ssh.connect-timeout</name>
: o3 z( U6 S s C2 x* E <value>30000</value>' _' B7 ~: G u8 m, c: a
<description>
* X' Q: N- O/ |1 P; [8 _ ^0 v7 B SSH connection timeout, in milliseconds, to use with the builtin
. V" V0 D# a& F6 ^2 Y sshfence fencer.
# u6 Z3 |7 a2 H* O1 j, \ </description>6 }( n# I, ~( h( N9 H$ W2 x3 m) I
</property>7 Q- l, h! `: R5 a: f1 I+ k
<property>
5 n* e* y% x5 O: l4 Z4 m3 B: { <name>dfs.ha.fencing.ssh.private-key-files</name>
& v6 F2 A3 }8 \( g <value></value>
, {, f0 a O' q4 U <description>+ ~. a/ v$ _# C. u" u
The SSH private key files to use with the builtin sshfence fencer.
4 S) e3 J& O% ?* s% t( q+ i4 \ </description>
- ?" f0 f6 N3 F</property>
- F) C5 R3 o3 J a T" y7 N<property>
' @9 S! E4 D- f0 e <name>ha.zookeeper.quorum</name>
- F4 J( V6 @- I' c2 m( w1 U' J <description>- B3 I( N$ [% k, V) `
A list of ZooKeeper server addresses, separated by commas, that are9 n4 v* V" L# P# I% n P
to be used by the ZKFailoverController in automatic failover.
( R$ ~* y2 ~" }/ S </description>' @2 c/ j# f; X$ H# `
</property>
+ M; O- v+ X5 L<property>
3 x5 r: ?; |5 S o3 A8 M& A9 K <name>ha.zookeeper.session-timeout.ms</name>
( P& ^7 `0 p, h0 Z1 M& j' L: h% l$ \ <value>10000</value>
# N9 z1 {4 o, Z8 j& L3 a <description>4 x; Y* @% p$ |( Q" ]
The session timeout to use when the ZKFC connects to ZooKeeper.; P: C! v& }, c/ M
Setting this value to a lower value implies that server crashes) W3 H) w, C8 C, \8 B0 S, Z
will be detected more quickly, but risks triggering failover too0 [6 o2 x: Q9 G3 S9 f8 o
aggressively in the case of a transient error or network blip.0 {8 F2 L0 e( S5 B1 m% ]
</description>2 q5 o* r' r; @9 q1 C3 C
</property>8 x" z& s/ Q0 ?# T
<property>
1 w( q: ]+ L. h; V4 m8 ^ <name>ha.zookeeper.parent-znode</name>' z# n/ [/ H7 X# w* e- P4 j
<value>/hadoop-ha</value>; t4 x3 a# F# }9 {% B: Z9 f1 V
<description>
; s1 K7 X) k0 B. b$ T1 f, R The ZooKeeper znode under which the ZK failover controller stores* D1 c/ ~1 A: E% i$ X, Z
its information. Note that the nameservice ID is automatically: T, J. {" Q% R9 O; Y( K
appended to this znode, so it is not normally necessary to0 F* }) {+ i- W9 n1 j/ ?9 @: a& H
configure this, even in a federated environment.
, j7 ?" ^2 K8 z- ^ </description>
! Z3 Q5 Q; Y- Y3 W0 p0 w</property>
- S* W7 l1 t9 _3 w<property>$ ^1 u) p. r8 h9 ^
<name>ha.zookeeper.acl</name>
; k3 @# _) V1 ~3 A+ N; \ <value>world:anyone:rwcda</value>
; |& z" }/ b9 U: X: ^. \( X/ z <description>
+ e) p9 B. J7 z# X" I A comma-separated list of ZooKeeper ACLs to apply to the znodes$ R+ Y/ r+ m6 C4 T. @
used by automatic failover. These ACLs are specified in the same
) ]2 K. x2 _3 L/ {' y+ I3 A format as used by the ZooKeeper CLI.6 i, F1 o3 ^5 v) {- ~ k& v, r
If the ACL itself contains secrets, you may instead specify a
/ R' { g: i- g$ ?, _- J path to a file, prefixed with the '@' symbol, and the value of
* ]. V8 e1 Z, C6 n! T this configuration will be loaded from within.% C3 p/ m, ~6 L( H
</description>
: N! o! j8 ~' F& i* A& T</property>$ E9 l5 B: P) \" l- X h2 `
<property>
# c, P0 N2 j5 _. d; K ~ <name>ha.zookeeper.auth</name>* g/ ]5 E6 [, t# X2 [
<value></value>
' i( r7 H/ A6 g/ {2 y: u" j <description>
% b! f3 I+ j# Y( m# `1 @, M, N* @ A comma-separated list of ZooKeeper authentications to add when" r/ O+ W: {0 I) U5 b% _% X
connecting to ZooKeeper. These are specified in the same format, U- D$ b4 G! Q3 @# z3 _
as used by the "addauth" command in the ZK CLI. It is
. h& X2 Q% N( x, g# L4 |3 P% d# k" v important that the authentications specified here are sufficient1 L9 u# [" H _& j4 ^1 O
to access znodes with the ACL specified in ha.zookeeper.acl.
( @% V7 _1 S% R If the auths contain secrets, you may instead specify a1 h3 Q* Y$ u% ~2 C5 x4 F& p
path to a file, prefixed with the '@' symbol, and the value of1 h- U5 |1 h/ O X; D: w" p# M1 {
this configuration will be loaded from within.( E4 d0 [6 g$ [# J+ m: l- s
</description>
7 u0 r5 X% w3 Q</property>1 h8 M' H+ A. L8 ?8 ~7 A4 ~
<!-- Static Web User Filter properties. -->7 P$ e+ P/ z' B" G" F) {! B
<property>; i0 E$ I5 c5 v, T4 n2 y+ s3 j( t
<name>hadoop.http.staticuser.user</name>
+ x, b M. B- l9 U <value>dr.who</value>
9 n3 E! G2 X( E3 s0 P O& E3 C <description>' B* s' P; {# b& K
The user name to filter as, on static web filters
5 f1 r/ D1 F2 J' {1 a while rendering content. An example use is the HDFS/ Y0 r. g4 B" i8 A+ g2 _
web UI (user to be used for browsing files).* d& B8 `2 ~- l% ?5 O3 q8 f
</description>
+ O3 p9 w6 P" M+ [: ~3 W% m</property>
9 \+ G2 _6 W, ]7 b: S/ g; O0 {) ]" J<!-- SSLFactory configuration -->
$ z" d9 p. ^# X<property>9 L2 g( u* S# h" p! |
<name>hadoop.ssl.keystores.factory.class</name>
: v* Q7 p3 h9 H) Y <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>1 S/ R5 d k5 Q# W+ V0 N: s
<description>0 c0 k8 {7 Y. p! B1 N; A |
The keystores factory to use for retrieving certificates.
! r. X m1 a6 w5 `. ~+ x </description>
7 M A- `- O5 c6 [</property>( @+ t/ J5 B! S
<property>
* ]6 H/ F+ C$ [' q% o <name>hadoop.ssl.require.client.cert</name>
* C; d1 P$ Y/ H" g <value>false</value>
8 F$ O# x! c- J- |1 q <description>Whether client certificates are required</description>
1 G7 V7 \/ y+ @</property>
2 g# o$ X' t' y( E% i% a<property>& B& r; E; ?) x. v9 L1 f! D! u i6 z
<name>hadoop.ssl.hostname.verifier</name>9 C) c6 _0 h7 U! W; g* S/ c4 r, f
<value>DEFAULT</value>
0 i/ \* C/ e- E* D/ ~4 E! k. ~ <description>8 H: R9 H; f9 M- R! ]9 R% {
The hostname verifier to provide for HttpsURLConnections.
0 o! o3 L! [+ j+ n Valid values are: DEFAULT, STRICT, STRICT_IE6, DEFAULT_AND_LOCALHOST and7 K& m$ b8 r0 K) Q! C! o6 O8 v' G
ALLOW_ALL
( B2 i; Y3 L' `3 g$ C </description>
, o# Q5 h* E0 E# f: P/ t* D</property>7 L9 Y9 R2 F% y3 Q3 r' v, H
<property>% L& f% G: _( a
<name>hadoop.ssl.server.conf</name>" v, k% M7 q3 n. d/ C, o+ z
<value>ssl-server.xml</value>7 L4 E9 r, |# Z+ C1 f
<description>
, {# G+ g: p& G, t Resource file from which ssl server keystore information will be extracted.5 H9 U8 g2 J" x0 Y( L) _( h
This file is looked up in the classpath, typically it should be in Hadoop; h3 ~5 Y# {% ~8 P
conf/ directory.
' B! I* S ]( K( l1 y9 ] </description>8 z! v3 R( Y9 `+ g7 C. S' n
</property>
& l% e* r9 {5 O" t4 m9 ~<property>2 J# q0 ]2 R9 U2 T2 W7 d
<name>hadoop.ssl.client.conf</name>
9 M% A: E8 }3 A5 S# c2 f6 {7 | <value>ssl-client.xml</value>1 L! a1 n6 Z' V0 h/ j }
<description>
8 ?; M. G [% p; | Resource file from which ssl client keystore information will be extracted
" ~4 M" z6 Z' H This file is looked up in the classpath, typically it should be in Hadoop# c' b: S" w- h2 o2 O; `3 ?
conf/ directory.
' L! [% Y* y* ] </description>
4 m7 T* e2 q: t' a* L. x</property>
) g4 Z0 b7 { N) s/ [3 I. T<property>; Y8 r4 ?" y9 E4 F: R8 f1 l
<name>hadoop.ssl.enabled</name>
4 q! E5 [+ x" ]/ C( q( h" s D <value>false</value>+ e5 O' b @: y; `5 K& Z$ x
<description>
+ @7 t, {/ n9 J( G' K; h( a* u/ Q! k" K Deprecated. Use dfs.http.policy and yarn.http.policy instead.9 [/ l5 B9 H3 u/ X* B$ I
</description>- {: U2 H5 y: S @7 L+ A
</property>% V% ~9 T( F4 v" J: x5 n2 C) h
<property>
( G0 |3 `1 [; N) o6 } <name>hadoop.ssl.enabled.protocols</name>
/ @/ L2 K8 x4 @, Y5 {5 d% W <value>TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2</value>: l- E7 r; [/ t- p" T! k
<description>
" p7 T/ }% q0 C4 D The supported SSL protocols.; l# s$ d' p# y, `( l
</description>
; @! f$ b' b9 p! c! Q8 S; t% J( [3 l% U* u</property>0 K7 j6 \* c$ Q" q1 O0 x
<property>
! `9 @$ n, V; z' l3 ^& H( `9 _9 u <name>hadoop.jetty.logs.serve.aliases</name>6 g4 z: t. S' x* _- V
<value>true</value>8 z8 e6 `% K! C& F
<description>
; L- J0 H( S' t Enable/Disable aliases serving from jetty2 u! K/ {! o6 [# m; ?1 U3 N* q9 A3 a
</description> _7 v3 V' A6 W% @& T8 K! L
</property>& {% V9 J3 \9 G
<property>
6 E2 t" \! g7 h* Z* L <name>fs.permissions.umask-mode</name>' D4 z( }$ V/ H( L- }' W6 a
<value>022</value>
% I! K* F5 f3 h: l( {" `3 } <description>
- y; g+ O4 l W The umask used when creating files and directories.
5 z/ _6 M, F; s5 w3 v( n Can be in octal or in symbolic. Examples are:
# ~, K5 H8 ~# V' Q6 W "022" (octal for u=rwx,g=r-x,o=r-x in symbolic)," r$ I/ F% F! @% o; u
or "u=rwx,g=rwx,o=" (symbolic for 007 in octal).7 }# c4 q- e! ~! E
</description>
( A; M8 l3 H& C6 _7 f</property>$ ^ |- ^8 C0 g" S% j3 [" R( b3 \2 G
<!-- ha properties -->8 F; G! Q6 `" q# Z* |
<property>3 u. i- ?6 T/ w5 F* Y( e
<name>ha.health-monitor.connect-retry-interval.ms</name>
9 @ E0 [* c7 x0 U! B$ Y: o <value>1000</value>
" {6 U; k3 ?) U9 q2 W6 k; _: E3 P <description>/ D) ?) O" m9 d* c8 m; O
How often to retry connecting to the service.
, {1 W, X$ f1 S# [. |. i </description>
/ u( p" J- A/ F' y. p+ [8 t9 g</property>' q. g) r( }$ v5 ?$ c, X! |, V
<property> K, {0 N: F3 i# i
<name>ha.health-monitor.check-interval.ms</name>% t4 ]/ \# d4 |( Z
<value>1000</value>
8 R, {/ R8 r$ T+ l, [ <description>
# I. O$ H) C, D6 w. E How often to check the service.8 r2 r3 G3 M. M f/ m* Y7 P& h
</description>4 a' F% v2 G4 h! @) `
</property>/ F) x: t& R3 \7 s9 w
<property>- f/ M4 X% g+ u S3 b
<name>ha.health-monitor.sleep-after-disconnect.ms</name>
& d3 e% B" E: S7 h- x <value>1000</value>
0 _) L% F- H6 m, m <description>
1 z" E6 w' g7 `& W! t; t How long to sleep after an unexpected RPC error.
; j+ y8 J, D( h9 B* @" u/ J </description>
5 Z, S) N( n, { |. F</property>
- E* A! U2 L `% }/ K6 k& b<property>
6 ^1 u) `7 U+ K' u# u <name>ha.health-monitor.rpc-timeout.ms</name>, _" o( l! g8 A/ X: Z
<value>45000</value>
, s( |/ F: B" ^8 Y4 Q$ J$ {6 ~ <description>( }' w7 m/ j+ p$ f a: a; m4 A
Timeout for the actual monitorHealth() calls.0 s! k4 l6 ]3 s
</description>' u2 X$ v' o( l. ]4 H: F
</property>
! C4 P1 h+ g0 ]<property>
* R8 Y. H* W) ~! B* e; t <name>ha.failover-controller.new-active.rpc-timeout.ms</name>
7 i' a: w% k/ g9 X( b7 v+ h% N. G0 t <value>60000</value>
! [8 N7 c2 ~8 [. g <description>$ J% ?" s) l8 P5 R8 e- D
Timeout that the FC waits for the new active to become active3 s/ s9 D+ }) I) U" \* s4 E/ o
</description>9 W1 R+ ^' n* ^9 { @' A
</property>$ s+ V! X) e3 S, G
<property>
' Q+ j4 W# b9 r <name>ha.failover-controller.graceful-fence.rpc-timeout.ms</name>
. q e* ~' x7 G- f- n/ q. I <value>5000</value>8 V+ g, p8 N# q& ~
<description>6 p* l5 N2 o8 y6 t F- i8 E
Timeout that the FC waits for the old active to go to standby$ z9 o2 o9 v) A
</description>
y& j$ _6 g N</property>/ B% |% Y! m* e& u _# y
<property>
& ~, m: I6 \; x- Z <name>ha.failover-controller.graceful-fence.connection.retries</name>. F4 e2 Y* ~% {$ [
<value>1</value>
, Y1 r$ T' i% X <description>
# a# d+ C$ R& E* e. Z( g. M( y FC connection retries for graceful fencing
! D, Q) L5 L, m" h p; Y: X" ~ </description>5 ?. l# t! B+ v/ B! W3 Y$ }
</property>( y$ e' E% J3 D* Z
<property>
5 t$ L( ~8 Z" Y9 B1 u! u <name>ha.failover-controller.cli-check.rpc-timeout.ms</name>
+ ^+ g0 S' F. C- N( k <value>20000</value>5 Z/ B7 I4 q2 T0 H
<description>; H7 d+ A/ ^- W/ ^. `* b1 T
Timeout that the CLI (manual) FC waits for monitorHealth, getServiceState, U1 B& \) q8 L4 I
</description>. Z- `+ ~1 w( C
</property>" _. Q( v# N0 G: T) U0 p
<property>0 c" K3 m F J5 X& W y5 M( A. a; {
<name>ipc.client.fallback-to-simple-auth-allowed</name>
4 R! V1 ?" F5 T6 V. H' ]" W+ Y <value>false</value>
' q" x$ K* W8 E3 U <description>
2 c9 ~+ r8 N/ G& B4 W& ~+ S When a client is configured to attempt a secure connection, but attempts to) \+ c* P; d) F
connect to an insecure server, that server may instruct the client to
- H0 n+ [" s/ ` switch to SASL SIMPLE (unsecure) authentication. This setting controls R2 B' b2 Q. [. P g
whether or not the client will accept this instruction from the server.. X0 m3 c" j- w) J$ v4 s, f
When false (the default), the client will not allow the fallback to SIMPLE
1 {* g3 ~2 m [; g authentication, and will abort the connection.
6 M4 Y) `, `* P2 p; _3 P5 l </description>1 @. [* \' F9 X f8 o; R4 ]
</property>
% `, D7 P- \4 \1 m# |( Y3 T+ w<property>% y0 W0 m6 [8 S! Q& L6 |# @9 r
<name>fs.client.resolve.remote.symlinks</name>
% ?% T6 g6 R5 T e4 W <value>true</value>: q% v0 n# a0 S6 y H3 J3 z
<description>
0 i3 O' X2 v6 ?1 y4 l8 Z Whether to resolve symlinks when accessing a remote Hadoop filesystem.
/ T ~$ p2 @; T1 B, ]/ M2 x' m* X Setting this to false causes an exception to be thrown upon encountering
% U) R0 Q$ K e/ ?, z! ~; q, B a symlink. This setting does not apply to local filesystems, which
6 d9 m8 H$ ~4 A+ C4 o" F1 z; J automatically resolve local symlinks.# \' W: C6 n, }' J8 G, ?
</description>
7 t! m6 A. @ m3 X8 R3 \: d g; s</property>3 K5 ^& `/ k" s7 U0 `0 B# ?
<property>0 v4 Z% T0 w. c$ \
<name>nfs.exports.allowed.hosts</name>
, X0 e7 P! N: m/ z <value>* rw</value>
( w6 i0 b/ Q6 K$ h <description>
, |' e; c( N1 g6 s7 ~ By default, the export can be mounted by any client. The value string
: f l7 s4 \+ f contains machine name and access privilege, separated by whitespace
! \( A# l# r$ N* t5 `" E characters. The machine name format can be a single host, a Java regular* F2 E: y8 T& k% s, W* j# S
expression, or an IPv4 address. The access privilege uses rw or ro to
8 v/ {' a) D" L2 N2 p6 v" K- C) q$ h specify read/write or read-only access of the machines to exports. If the
. C8 F) v* B7 _( y/ ]: L0 K& C access privilege is not provided, the default is read-only. Entries are separated by ";".
' y X2 l' K/ q% y: o# f1 @- N For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;".
+ r1 B0 L4 p: P/ N x Only the NFS gateway needs to restart after this property is updated.
, S5 _: D# d6 s# D6 J, h0 l% R& z </description>
A8 K7 I& R& Y* X8 P' b</property>
* [. m: t+ j1 @7 O9 F, H<property>! c4 U4 s9 p. C4 O
<name>hadoop.user.group.static.mapping.overrides</name>
8 j! [2 @2 ^. h0 Q% h1 i1 l" { <value>dr.who=;</value>
! z8 ]" f, K3 @6 l, E1 `: h <description>* W# y$ q) \( t# ]/ J: C
Static mapping of user to groups. This will override the groups if5 z! `) }; U% O# \* @0 }2 Y
available in the system for the specified user. In other words, groups
4 o! S5 x, U! a; F% k look-up will not happen for these users, instead groups mapped in this5 E. _3 \2 o0 b' }9 ], X
configuration will be used.
; V/ p+ Z2 ^, V+ d5 s+ c& w Mapping should be in this format.+ E: l3 t7 M% C' y% B2 s
user1=group1,group2;user2=;user3=group2;) p5 E' X' _$ l, }/ Q7 L0 W7 M
Default, "dr.who=;" will consider "dr.who" as user without groups.0 j; Y. |' k3 C& T
</description>, k" `! g# J) g% @; P
</property>8 Y& \# s4 x) k6 f/ U; g7 c
<property>; j# y6 t" e! H& Y$ P; w9 t- O" r' ?
<name>rpc.metrics.quantile.enable</name>4 ^( z1 f9 B5 V- x2 x. y' v- d& K9 |
<value>false</value>- }* l" u+ B4 D0 A$ g
<description>
; l( \# E; g4 Q Setting this property to true and rpc.metrics.percentiles.intervals3 L8 k: \$ ^6 \ ^7 j! L
to a comma-separated list of the granularity in seconds, the
) ~8 f7 c! P2 B 50/75/90/95/99th percentile latency for rpc queue/processing time in4 i9 Q. ~3 c/ d
milliseconds are added to rpc metrics.2 P/ }. ]9 q) S: v1 E
</description>( x2 z9 ~8 z3 j
</property>* u: g! _ g( Z: i$ @
<property>' _/ E4 _( m6 y& z
<name>rpc.metrics.percentiles.intervals</name>
1 ?- P3 a7 u% j- }8 h$ `: E/ b. y <value></value>
{5 y3 R' v) Q <description>
# U* K' H$ r* z, W$ Z0 T A comma-separated list of the granularity in seconds for the metrics which
( U) R7 V6 S l; l c+ V# G& U describe the 50/75/90/95/99th percentile latency for rpc queue/processing
0 V' D3 G& v3 H) H" K time. The metrics are outputted if rpc.metrics.quantile.enable is set to7 }- Z& X4 M5 _0 e; o1 ~; l8 r
true.0 U) y" t+ a i+ T
</description>& K, g3 h5 M% D: [/ X' U( r6 O
</property>* x$ Y. Y- |3 B1 F5 x- v! U
<property>0 ?; M7 i7 e% m' I9 L
<name>hadoop.security.crypto.codec.classes.EXAMPLECIPHERSUITE</name>- b/ A0 I" V Z" [1 O
<value></value>
8 s }) e, `# [' ~2 }( d% P <description>% b; `# x, H, U; O3 b+ e
The prefix for a given crypto codec, contains a comma-separated1 u: V! P; q2 w( S! C
list of implementation classes for a given crypto codec (eg EXAMPLECIPHERSUITE).
# S+ r# q' h* x The first implementation will be used if available, others are fallbacks.
. n$ l9 {3 |" k$ S3 |& ?! w1 b </description>- Q3 Q! ~: g0 F
</property>
2 L$ B$ w( e; J4 P5 e6 J<property>% y8 U5 N9 u) ]& o% L
<name>hadoop.security.crypto.codec.classes.aes.ctr.nopadding</name>! h' x4 f# y# C2 Y) e# h$ a& F
<value>org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec</value>
7 ?% V! E- w2 b% \ <description>
. k! L& U6 B: x5 ^! E9 N Comma-separated list of crypto codec implementations for AES/CTR/NoPadding.
. d' j; `' e( H6 c3 v" j The first implementation will be used if available, others are fallbacks.
$ F# u7 F! J2 C5 `& i7 L </description>
( S% e1 W2 E& D- s) u2 O; [' L</property>
- f3 i9 T% ?) a4 [: [<property>0 T- ]: ?, T/ F* b
<name>hadoop.security.crypto.cipher.suite</name>% f V* F7 r# d( p
<value>AES/CTR/NoPadding</value>
/ D7 o$ y4 |# z4 t$ E. q <description>
" O( C9 `3 i; j Cipher suite for crypto codec.. m6 Y' q6 k6 V. S. G7 P
</description>* `0 R/ V4 D1 X# p
</property>, T r- N& @' Y5 ~6 r# t* M
<property>
* R. l$ J* w! e9 V) x& C* S <name>hadoop.security.crypto.jce.provider</name>1 H2 t6 ~) t- U/ S4 M
<value></value>; I' h4 b' w, \& @
<description>0 D6 U$ _# c1 k$ U/ ~/ |
The JCE provider name used in CryptoCodec.; [/ D7 d# m+ f6 o
</description>! r8 B$ o: u8 k8 ]- [1 e7 |
</property># q, P7 [6 V- ^0 \2 k
<property>
: @2 F1 l( s- a+ m; Y <name>hadoop.security.crypto.jceks.key.serialfilter</name>
3 h1 ]8 ?$ s: X& ? <description> S `/ z& y1 @# O' s: M8 t
Enhanced KeyStore Mechanisms in JDK 8u171 introduced jceks.key.serialFilter.
6 s4 I; z; P2 \4 s If jceks.key.serialFilter is configured, the JCEKS KeyStore uses it during' B/ k* T/ k7 u, U
the deserialization of the encrypted Key object stored inside a& H) b& m$ B- R. L, d: N
SecretKeyEntry.
, ]; o' F9 i8 v/ V) ?2 P7 d" g If jceks.key.serialFilter is not configured it will cause an error when
' Q+ K# [5 h j# f! Y recovering keystore file in KeyProviderFactory when recovering key from& _4 L5 G6 k' {' b. i; f
keystore file using JDK 8u171 or newer. The filter pattern uses the same
5 }' _7 n% F' B5 \# }3 H$ |7 n9 W format as jdk.serialFilter.
. E+ [+ S- M3 Z. C3 D The value of this property will be used as the following:1 g* X! T3 D4 R, M( D
1. The value of jceks.key.serialFilter system property takes precedence
0 b1 A/ }+ @0 L5 z- V over the value of this property.
# y4 k* j7 v4 d7 p; G- x! B 2. In the absence of jceks.key.serialFilter system property the value of
: ]+ t [) O1 o- b" X+ F# d2 K this property will be set as the value of jceks.key.serialFilter.
7 B% K" ]+ F7 r4 m# C+ ]/ u 3. If the value of this property and jceks.key.serialFilter system) U8 ^9 x/ ?0 k: g
property has not been set, org.apache.hadoop.crypto.key.KeyProvider
0 _" ~+ e3 C; I# j sets a default value for jceks.key.serialFilter.
$ d+ }4 }% V# d7 d l& d- w </description>4 W1 k* u& |) A' P! Y
</property>
3 f1 g0 V9 e; N8 W/ I0 y. m2 ?<property>
8 w* ?4 U5 }- F0 q <name>hadoop.security.crypto.buffer.size</name>
6 k; r9 I& S' d/ _) c <value>8192</value>, ]% m9 X' X1 P. x8 i' M( t
<description>3 \6 r; T& g9 I+ n
The buffer size used by CryptoInputStream and CryptoOutputStream.
; X- e- d7 j! _) Q: t2 \# ?% g </description>
, Z$ z& Z4 s) \; Q; n</property>1 ]+ a0 t; p" V9 U8 O2 ]
<property>
: q* y) Z1 G# \1 z/ d4 p <name>hadoop.security.java.secure.random.algorithm</name>4 n$ l& O9 O* P8 z( Y2 a
<value>SHA1PRNG</value>; _$ z% t% c3 s
<description>
+ |: \, {& i$ S2 ^$ l# r The java secure random algorithm.
; I( q8 H* u! d L. J. N4 w </description>( o: s* c) E/ n, D
</property>
- b r: B& d7 k' V<property>
n, a9 V/ ~; \( h9 u6 u" i) m <name>hadoop.security.secure.random.impl</name>
- S4 e: G9 k; O3 F, q; s' I8 S <value></value>) o! Q* _% S5 w* q1 ]
<description>( K1 _# E _; S+ W
Implementation of secure random.' {& d7 x3 S* g8 \6 E# a5 }
</description>* k6 g. ?3 W9 J
</property>- k2 S7 W$ g5 L' a) N
<property>) q; D3 H% h7 K- \5 w
<name>hadoop.security.random.device.file.path</name>
5 i0 l% T/ L) F+ d" r+ V <value>/dev/urandom</value>
. H4 e- A+ y! ]) B <description> d5 P: v) }6 y" D% V5 B7 K+ ~
OS security random device file path.( f/ S6 V$ P# p. c4 y) G
</description>
* K! v8 n9 D9 z9 h, m</property>
% k c% j" C4 A6 N. p+ _6 i5 p<property>
: b4 X; u9 u7 ?. h" B+ [ <name>hadoop.security.key.provider.path</name>2 O9 m6 ?# d" ?4 h1 l
<description>
6 ]! P* Z8 Q$ n9 l+ b The KeyProvider to use when managing zone keys, and interacting with
! S0 ]/ O, e: D$ ], r2 h encryption keys when reading and writing to an encryption zone.( Z, ]) ~0 |9 G0 J* M7 c
For hdfs clients, the provider path will be same as namenode's1 u: d) K- j3 }
provider path.
% |, i" s- d9 u </description>
! \! r0 B" M7 Y, I! ^2 X6 z</property>! y3 X7 m* q! r8 l5 C0 ?) S! @: a% `
<property>
4 n; }6 V' h4 n* B- G <name>hadoop.security.key.default.bitlength</name>
" ^" V) b# _; j1 M+ H <value>128</value>- H# f6 S# a" \9 ]. U
<description>
3 s9 _8 C0 C# D# ? The length (bits) of keys we want the KeyProvider to produce. Key length& q5 t, L c, Z0 U
defines the upper-bound on an algorithm's security, ideally, it would" S' i" |3 \; P) ~, ~
coincide with the lower-bound on an algorithm's security.7 S3 ?$ `& ]2 I
</description>
9 m z: l% _! z8 Z2 |* }# q</property>- [8 u0 \& `( K E
<property>& n4 N9 C' F0 a. m' F. g9 S3 X
<name>hadoop.security.key.default.cipher</name>
/ l, R: c0 h# j3 H8 B; E <value>AES/CTR/NoPadding</value>1 h' S0 D" E- U4 y
<description>
% Y" r) \+ y& z3 A This indicates the algorithm that be used by KeyProvider for generating
3 f4 y8 @, z$ y, H. g key, and will be converted to CipherSuite when creating encryption zone.
+ o! c7 l: r6 S* I# P </description>3 M- o& d n. F' }! |# X0 s
</property>8 I, L' d- i& k( O* h3 Z
<property>+ f0 b6 p) H c" L5 R; v2 G9 b+ w
<name>fs.har.impl.disable.cache</name>5 m0 j' }+ x' i$ Y! ^. _9 e0 \
<value>true</value>& i: G3 ~! j0 c q
<description>Don't cache 'har' filesystem instances.</description>
5 E1 t# ]. C) h; P3 F</property>) @" z8 K6 p; x6 p* [
<!--- KMSClientProvider configurations -->
8 X7 |- R' T6 o& M7 r<property>. s* ~4 W3 G$ c, [5 L: }
<name>hadoop.security.kms.client.authentication.retry-count</name>
& y% s9 J2 t9 I7 q$ e <value>1</value>: n! j7 B6 D6 g' Q% b; h
<description>$ ~. R( K" |' u; I3 C9 B2 e" k9 _
Number of time to retry connecting to KMS on authentication failure# q5 g+ g( @4 J: i- ^% ]" c K
</description>& R$ i ]3 f- @: P; ]! L& r
</property>
% ~) v' E/ ?3 [; X. Y<property>
9 T$ M( r5 ^ `- e3 e( { <name>hadoop.security.kms.client.encrypted.key.cache.size</name>
0 s) y# R& R. s, p6 Q2 W9 R3 E# e <value>500</value>7 p% K6 Y' m" [7 V$ p6 y' V( Z
<description>
, w" {& y; B! V4 O4 L+ k Size of the EncryptedKeyVersion cache Queue for each key7 L3 C* ^# F$ X
</description>: l9 b+ i+ M; O% b# _ @4 z
</property>
2 J2 h d, i! p7 E% {<property>1 R, s+ H% Z' R6 U. O9 h; B/ ^
<name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>. z& x& x! P* \; Y0 E7 r
<value>0.3f</value>
) y- b2 N# W) _0 ^& ]* }8 V% f; Y1 z <description>
+ b! _ R1 t' `. w2 o! s2 C; a If size of the EncryptedKeyVersion cache Queue falls below the' M M* \) R8 |% j
low watermark, this cache queue will be scheduled for a refill
4 L+ i9 K! r/ ` </description>5 u `/ Y, s+ e8 r8 c& l
</property>% T' f) D8 f$ `4 r
<property>1 D; }5 ?/ L5 ~* F
<name>hadoop.security.kms.client.encrypted.key.cache.num.refill.threads</name>' u) }% W+ M* N9 A1 y) b6 a
<value>2</value>: I, d0 c6 |/ I$ z
<description>
6 F; e& b5 J' c1 P2 p" { Number of threads to use for refilling depleted EncryptedKeyVersion1 S7 c! [$ c# j
cache Queues2 l1 i3 o* Q* y7 v/ [2 ~# f
</description>
* u1 `1 _2 t' ~8 ?7 s</property>
) @" y4 U& V; z! b* [<property>
% c( L. t1 s$ @" |' E7 a" I0 ~ F4 f <name>hadoop.security.kms.client.encrypted.key.cache.expiry</name>' F: _( J8 {7 _/ a
<value>43200000</value>9 _9 h$ X# j# h
<description>& t/ \+ W) a( }
Cache expiry time for a Key, after which the cache Queue for this7 ~3 `0 {9 o) _* j7 j1 K
key will be dropped. Default = 12hrs
- a3 Z( i; Q% A7 n) |8 o" Y </description>
9 c1 U- R- j- e( G. S' o. B$ p8 J</property>
4 u& P& G$ p) x9 P<property>
, \& ]8 |3 M. {! c8 l7 O9 x <name>hadoop.security.kms.client.timeout</name>
& R4 g2 m' ]' J, w <value>60</value>' U( T& l! T4 k4 w9 Z
<description>
" z6 Y& w, M& f9 _! Z Sets value for KMS client connection timeout, and the read timeout
6 \- }+ \) ~: y$ k P4 F2 C7 y to KMS servers." N. o" b8 F7 ?0 {% E& v& w9 k
</description>
% U( _6 o1 [, {& `* f4 ?8 x& O0 S</property>
: I. w, D; @4 C/ Z<property>) q, ^* f" {2 P% ?
<name>hadoop.security.kms.client.failover.sleep.base.millis</name>
r0 d; P: O2 {% i <value>100</value>: r! l4 Q# k* l) W
<description>4 k4 l K& Q( c& {) D* P7 v1 H) d4 ?1 b
Expert only. The time to wait, in milliseconds, between failover) [1 F8 y( q9 h+ Q
attempts increases exponentially as a function of the number of: F% G% k o. b, \5 b
attempts made so far, with a random factor of +/- 50%. This option" X6 K1 K) w8 m6 i
specifies the base value used in the failover calculation. The
/ B: R8 [* y" A( g+ B6 G first failover will retry immediately. The 2nd failover attempt
6 N% E9 @6 G- \% Q# J8 n! j* I# K will delay at least hadoop.security.client.failover.sleep.base.millis: m6 ~2 t0 e# |1 [
milliseconds. And so on.
3 {& \( D" r: x </description>9 h ^" R) E7 Q7 S$ U" f+ F
</property>
# b0 X9 i: ~& r. V/ h! q4 q<property>
7 \5 Z2 M; i. q$ Q- y <name>hadoop.security.kms.client.failover.sleep.max.millis</name>! D# q6 G& P6 z; @; H
<value>2000</value>5 i' s6 V% j( R6 r8 h* \
<description>
$ E0 U' v2 X0 o( U c Expert only. The time to wait, in milliseconds, between failover7 ~6 @! Y- O2 U, U6 H+ ^$ u
attempts increases exponentially as a function of the number of
" b# ^1 K2 |8 w u# d attempts made so far, with a random factor of +/- 50%. This option
$ d1 U6 Q! c2 N& s% a/ ~2 h9 b specifies the maximum value to wait between failovers.: S8 W8 q# U- q( ~/ c
Specifically, the time between two failover attempts will not
& Z2 D) ?, r! U" ?. [( I0 F exceed +/- 50% of hadoop.security.client.failover.sleep.max.millis7 n( l. g1 h/ @: C
milliseconds.
+ N+ R2 F0 v& x, n3 P </description>
1 X8 F7 K% \' q3 t/ ]</property>
- T+ B3 h! A8 b: |& V1 p/ B: A <property>
6 q3 e3 A5 P, g2 D <name>ipc.server.max.connections</name>) k: X: o, O. J# `6 X4 ^/ T# ~
<value>0</value>; B" N' Y* [+ h* k1 C* n; E8 T
<description>The maximum number of concurrent connections a server is allowed2 l, [% \1 B/ r
to accept. If this limit is exceeded, incoming connections will first fill
- c6 d, C4 R8 | the listen queue and then may go to an OS-specific listen overflow queue.+ }7 z' y3 m0 O" ^$ G! V# x
The client may fail or timeout, but the server can avoid running out of file
$ p$ w0 `) {. h8 }) @, X2 W7 g descriptors using this feature. 0 means no limit.( w# r, X! d+ s2 S( `9 M$ y4 Z
</description>
* D! s7 w6 q+ f</property>
; ], P9 }- y4 ?3 R7 T <!-- YARN registry -->
5 ^: w. t7 p1 y$ F <property># w" `0 P; H1 e# Q% g8 h
<name>hadoop.registry.rm.enabled</name>
( {2 \+ e5 ]9 J- ?; r <value>false</value>
7 a2 K1 v0 z- k5 G <description>
4 z: X# h% L" k' t; d2 \2 e6 b Is the registry enabled in the YARN Resource Manager?( H% {3 v2 `& Q
If true, the YARN RM will, as needed.
5 G+ [8 L4 y# f9 ]4 e create the user and system paths, and purge, m5 }6 n1 n+ E: d1 i" c: ?& Y
service records when containers, application attempts
8 m7 U& L* j; I! |! p and applications complete.
% ~0 D; a, `6 {, F1 j; q! u8 X If false, the paths must be created by other means,
5 ^( a# a1 l" Z+ S% O( s1 b7 S+ B and no automatic cleanup of service records will take place./ N3 |7 c# ]' }! j) j
</description>
( B; E* W8 v! M! h* l </property>
1 |4 y% D2 W4 A3 ^8 U1 [2 E <property>
5 R+ a D$ o$ e5 Y <name>hadoop.registry.zk.root</name>2 x, v6 |* Y$ g* L
<value>/registry</value>
& c, y: E. H# o) q <description>
2 R O8 m" Q0 ?+ q! O The root zookeeper node for the registry
8 C) g- ^. j! t3 k6 ?) A9 O7 _' } </description>5 D4 E* p) I; P
</property># {- P. F! S5 T" K4 z0 D% G# i) b
<property>: U! y* q4 p0 i* g/ P, t
<name>hadoop.registry.zk.session.timeout.ms</name>) H5 e7 n) N) k* h. O* @4 q
<value>60000</value>( N+ M S- `* k1 \) [
<description>
: l- }4 S5 _) m3 F0 [& m5 @ Zookeeper session timeout in milliseconds- C) u" s+ d8 r6 ~- v" K& T
</description>
3 T* B! u2 `. t' m </property>
7 ~1 j+ H9 y _. Y1 m7 T2 ?8 V <property>
1 @7 m' h' ^5 A- A <name>hadoop.registry.zk.connection.timeout.ms</name>+ h+ V0 \ w3 q& [4 |# r0 H/ ~* R# c
<value>15000</value>! l( e- M! m. z- m
<description>
' J% v+ u% f i Zookeeper connection timeout in milliseconds7 z O6 |; l6 X2 h% }. ^6 K
</description>* \% G* C! l& i. H
</property>- q4 b- j0 O; g+ _1 X+ F3 A; ^# d
<property>+ S ~% }- b$ J. e. c) ?
<name>hadoop.registry.zk.retry.times</name>
5 ]# q/ S( _$ j1 M4 e <value>5</value>/ ]0 {' X" T9 O( @" x' |
<description>$ b V; v8 P8 j' f8 k% ?/ \. U
Zookeeper connection retry count before failing4 q3 X& F! M$ u1 [; M
</description>
5 i. c4 [% o/ x2 u# _$ U- c1 X2 h </property>+ _" n0 z2 O' }" _9 p9 N" G
<property>
% p @1 K& ^- C' r: l0 ~" u: c5 | <name>hadoop.registry.zk.retry.interval.ms</name>2 e$ \$ u2 @$ a% E
<value>1000</value>
. l% L9 }: ?/ A1 H; u( R: T <description>
5 C; H; E& t7 u$ B, d </description>+ x, _5 i% p# @, B7 m5 E; j* F; z
</property>
; x1 I2 n" ]( M3 U& x8 v0 ^1 t <property>
" ]" g- F) N( P( {* u <name>hadoop.registry.zk.retry.ceiling.ms</name>
* h: D, K6 i5 f" a <value>60000</value>
' W" a, L9 L @& G7 }: y <description>) L! }, W! K( [. ?0 w: }' V
Zookeeper retry limit in milliseconds, during
8 g. P; b1 l1 F: {, ]& h1 z exponential backoff.
2 S8 w4 L3 F% } This places a limit even }6 A+ v1 h( B, Z a7 g6 O
if the retry times and interval limit, combined9 |& y" f% w2 E, W! j) D" _
with the backoff policy, result in a long retry
H- N6 d9 y5 ~. b& P% e0 k; t8 @ period+ q* m- P% Q& L. P9 O1 C
</description>
* Z ?+ h6 S- F4 Y- S" n2 \; L </property>( H; P5 O4 \- a5 P" E0 d8 z8 W
<property>. M8 f& R1 z+ c$ M" t ~
<name>hadoop.registry.zk.quorum</name>
% z9 F) P: B* _6 x <value>localhost:2181</value>
" ?# ^! [, J0 w <description>
) Z$ d/ S( s a2 |. B- Y" f1 H List of hostname:port pairs defining the
& z/ z* f* V2 l2 _ zookeeper quorum binding for the registry
5 |9 Y8 Z3 s3 m8 L0 y1 F6 ? </description>
, d6 n* I. w! J& \5 y </property>) `5 T( w8 P) \/ q5 a7 J1 Z
<property>( t# c& @* q) V, q; m4 {9 n" u9 v, X
<name>hadoop.registry.secure</name>
, t" G- Y6 z+ _% P! }, d; s <value>false</value>
y! y, \( t- U* j; U/ [1 m+ n3 L' u3 @ <description>* U6 }1 Q! |) m2 }) R/ i
Key to set if the registry is secure. Turning it on% j/ l: u- D G& x# [; G
changes the permissions policy from "open access"4 ], ], J9 u2 u/ e0 z( X; b, x
to restrictions on kerberos with the option of
! l; @8 M1 ~; }& I& X# q) z7 I a user adding one or more auth key pairs down their6 u: s$ J( V) |) v$ p
own tree.- o( P" s: m2 m, T
</description>
; s( p6 z* z) ]% i9 ]/ W </property>
# F4 g' p2 m+ M6 p <property>
8 w5 @1 m" }' A" } <name>hadoop.registry.system.acls</name>
" s1 A- y6 q+ h5 T8 B0 }( g <value>sasl:yarn@, sasl:mapred@, sasl:hdfs@</value>" v/ x5 ?5 b Z5 k8 q% d- k7 [
<description>
' \# i; t( O0 K5 I3 P A comma separated list of Zookeeper ACL identifiers with% Z* ] D% V) @/ d# }
system access to the registry in a secure cluster.
/ j- a- m2 M9 d5 I; A& f! u These are given full access to all entries.
# T1 D, o6 ?' L If there is an "@" at the end of a SASL entry it
6 m3 s! W* p- W0 {- C, y; k instructs the registry client to append the default kerberos domain.; | K, \- W& V/ u6 o
</description>& q* N' {7 H T1 x% x+ i7 ?4 N8 S- `' C
</property>
6 p" i- w$ `3 ~5 \ <property>
( U2 I8 a. G1 ^" h) M <name>hadoop.registry.kerberos.realm</name>( s: ^: h0 e, K9 u
<value></value>' J. r% k9 l# }' ]$ B6 z: z; Q
<description>
b. m" v, @' t2 q* `3 ?$ t, D The kerberos realm: used to set the realm of
, x5 u& P9 ]( U, U% S system principals which do not declare their realm,- z9 ^& _4 Z" ?# S. q- B
and any other accounts that need the value.
6 {- [ |3 m2 R* N& H If empty, the default realm of the running process2 o$ {. R7 W# a$ E6 w& x
is used.- R8 l5 n2 T# X/ w& g
If neither are known and the realm is needed, then the registry
3 W9 P* M& C2 W j/ N" w' v service/client will fail.5 ]" p# t0 t, @- ]$ Q; j/ w
</description>6 U2 f- u8 S2 @$ e) z( ^
</property>% _( e3 ?" J8 i; ?/ c# h; R4 V" I
<property>; n! ^/ K- t- J* o% j/ I7 [% Q
<name>hadoop.registry.jaas.context</name>; q2 C& ~: b6 p6 V
<value>Client</value>% x* ?) Y# M6 K" t
<description>
( }( o+ {9 e) u4 e Key to define the JAAS context. Used in secure
, ~! C# e" E8 G5 n$ b4 |1 V mode
" L# F' k# x, r5 q! v8 \* \! K; s </description>; H1 W: c2 |; l" M4 _
</property>
. c7 R4 a. v- C5 y# @) b$ X <property>" P* N& a" a% ]( z9 t- `7 _
<name>hadoop.shell.missing.defaultFs.warning</name>4 O' M s u6 ?+ Q+ {" q
<value>false</value>6 ] L/ H- H7 f, J }. \
<description>
- y; e# u! i" r9 r0 ] Enable hdfs shell commands to display warnings if (fs.defaultFS) property
3 `8 h/ T/ F6 [# I) U1 p is not set.
4 f& i6 e) P" a </description>- k% n4 o+ l8 z# {% n3 f6 N2 q
</property>2 i" D0 Y k6 b/ X' @5 M, R
<property>. p5 i" K! U/ u4 B' E9 \& a
<name>hadoop.shell.safely.delete.limit.num.files</name>7 A9 B0 ?& F8 j) L3 K
<value>100</value>3 Q6 K- G: u6 ?7 |- @8 F+ V9 E. E
<description>Used by -safely option of hadoop fs shell -rm command to avoid# u* s+ d) R! q5 f
accidental deletion of large directories. When enabled, the -rm command
8 G8 \3 D% d: a" N0 I requires confirmation if the number of files to be deleted is greater than
w; D2 k4 G7 ^3 k5 C- I this limit. The default limit is 100 files. The warning is disabled if: y$ B7 q9 u% s. ^& L2 J% Y# o
the limit is 0 or the -safely is not specified in -rm command.& s0 ?* x. E8 C( @- d
</description>( Y) p" m3 C$ b9 q- J: P" R6 K8 W4 |; p
</property>
) p' C8 D. m) C3 C <property>) m5 D0 t' s' @3 r
<name>fs.client.htrace.sampler.classes</name>0 u2 l5 g& j1 k6 L! W
<value></value>4 [3 Q3 `8 f0 I* F+ d
<description>The class names of the HTrace Samplers to use for Hadoop
* Q: Y- _& s7 G filesystem clients./ x, P. ~2 d1 n0 R
</description>
1 n: r* |. G! h- W/ ` z1 Y6 T </property>
4 _. {& X. [" H- o$ u$ l) ~; y3 f <property>2 a# [$ ^# u- d/ c7 F8 j8 x2 x
<name>hadoop.htrace.span.receiver.classes</name>
% a& h% L+ f; |) x3 T3 R <value></value># [# s' q2 m- H1 {
<description>The class names of the Span Receivers to use for Hadoop.
0 v7 [$ A/ M8 m5 n6 | </description>
9 x* S d8 d' R* [ </property>' f% Z ~9 _ Q9 G! _
<property>
* _# ^+ M/ j ]# k, c- l5 D2 f <name>hadoop.http.logs.enabled</name>
4 f7 }3 }4 {! B0 D, _ <value>true</value>
& j% r4 @; A- q) d, } <description>6 O3 o$ C1 \% r4 ~7 E
Enable the "/logs" endpoint on all Hadoop daemons, which serves local
4 _7 \3 I- ~' f$ H logs, but may be considered a security risk due to it listing the contents4 i7 ^' c/ _, M
of a directory.
N6 ]/ W2 |! G( ~% M </description>
$ `) d: E1 k( [! U </property>
. p" p5 p# r# m' u' c/ H: x- L <property>, Q/ C# B9 r% B+ } {' d7 J% z% K# {6 y
<name>fs.client.resolve.topology.enabled</name>3 h, u6 O' p- R7 C( P, ]
<value>false</value>2 ~/ a/ B5 M2 z
<description>Whether the client machine will use the class specified by$ m5 o$ m2 W, q: u, S; E
property net.topology.node.switch.mapping.impl to compute the network7 ?8 W5 R' ]6 q) f0 `, A. ?
distance between itself and remote machines of the FileSystem. Additional
6 a: q) q: f: O4 t3 \4 e ^ properties might need to be configured depending on the class specified# r0 q2 e$ t1 `: t
in net.topology.node.switch.mapping.impl. For example, if/ y& J8 o g6 V. X& q* W' Z7 e9 L2 V
org.apache.hadoop.net.ScriptBasedMapping is used, a valid script file
$ `8 ?) {. t4 o! a: B needs to be specified in net.topology.script.file.name.0 I! C! T; N1 p1 R) ]
</description>0 S9 i8 E, m* G0 C
</property>
) @5 J% x1 Y9 o <!-- Azure Data Lake File System Configurations -->0 n' u/ M; ~7 ~
<property>
6 ~! s8 t* P+ [" u1 @0 H7 p& V <name>fs.adl.impl</name>
) Q: I. Z& Z; ~$ w <value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
; p. b; O3 a) a: y7 C </property>
1 u/ {" ?: d7 U2 N <property>8 I0 z2 Z6 N% _9 t3 c( I( @# D2 [
<name>fs.AbstractFileSystem.adl.impl</name>
* g2 m- d4 A/ t/ }/ p8 q4 j+ A <value>org.apache.hadoop.fs.adl.Adl</value>! d; W; h1 |2 Q. i- ~: F
</property>8 H9 H# \; v7 t& {
<property>
9 z6 g( `2 l# z+ W <name>adl.feature.ownerandgroup.enableupn</name>
) Z9 r: `7 [1 y1 r1 q' N! R5 z8 h <value>false</value>
5 N* S' g, \- ] <description>; l- l5 X6 K5 q4 b; I
When true : User and Group in FileStatus/AclStatus response is2 J/ x( W1 W* Q/ D3 S2 D
represented as user friendly name as per Azure AD profile./ {2 d9 [/ ~0 n2 a% ?
When false (default) : User and Group in FileStatus/AclStatus
; L- q6 l& B. t) |9 Y7 ]3 y response is represented by the unique identifier from Azure AD6 e+ d9 I0 F7 j" w) E N% m9 p
profile (Object ID as GUID).& f$ K: N: H% Z- _6 q
For optimal performance, false is recommended.: ^6 C6 `$ g6 ~5 @- e" ]
</description>: q8 n9 H0 Q/ x3 O% m" J
</property>
& I7 ^+ C: o; j; v* I6 s& n <property>
+ k! E# s! ]+ m$ h8 @ <name>fs.adl.oauth2.access.token.provider.type</name>
+ B. ^5 { @9 q$ H, G9 S <value>ClientCredential</value>( C [! ^4 G/ ~3 Y3 W5 [# B& o
<description>3 M% K, |4 P$ n( X/ V" ~$ j* F% D
Defines Azure Active Directory OAuth2 access token provider type. i2 N7 }6 T- R, p
Supported types are ClientCredential, RefreshToken, MSI, DeviceCode,
) d8 y i% s- @# I and Custom.: f9 e) ]- B& ?! u
The ClientCredential type requires property fs.adl.oauth2.client.id,
8 v7 G" G: I5 c3 G1 \6 z. v fs.adl.oauth2.credential, and fs.adl.oauth2.refresh.url.+ [7 B# e6 O0 L
The RefreshToken type requires property fs.adl.oauth2.client.id and$ I6 l! Q6 C# @1 L* O
fs.adl.oauth2.refresh.token.0 z" o4 p# d: F5 p$ R, E
The MSI type reads optional property fs.adl.oauth2.msi.port, if specified.
6 H! o7 I/ E8 N+ J8 s' F9 x4 a The DeviceCode type requires property# U" \) h0 P7 m+ t2 X, ^, [
fs.adl.oauth2.devicecode.clientapp.id.
0 R( ^5 Z( _3 S1 K The Custom type requires property fs.adl.oauth2.access.token.provider.5 {8 [& N" d: g* w9 M
</description>% L+ U7 G4 b" i) P' ?, B
</property>& k0 N" E6 s: e& f! a9 G) d! x
<property>
+ ]0 b- ^. i \! D! { <name>fs.adl.oauth2.client.id</name> A2 E' k( z3 y2 T P4 N+ d. ?
<value></value>
0 U7 q! ]" z/ O* d1 y <description>The OAuth2 client id.</description>" J9 y% K# [; }6 v
</property>4 s4 b2 O& H2 v' F+ F
<property>: t+ P/ o- D3 ~, R' t1 e- r2 b! v- V
<name>fs.adl.oauth2.credential</name>
4 g% C! V6 w; ^* P6 ]2 d# ]; X: W <value></value>
. ?- G; A* {* c! U <description>The OAuth2 access key.</description>
' \" ^( I3 G- H, q </property>$ o( M* e' r7 M: b) _5 I9 m) C
<property>; {9 A( T2 D; R, I6 W! [) Y, w
<name>fs.adl.oauth2.refresh.url</name>/ S) G, b: U8 i( Q
<value></value>
3 [, V; \7 K5 z# z- _ <description>The OAuth2 token endpoint.</description>* ^0 S: P& L2 r5 U) s0 L! T
</property>1 ?3 R, K- ~# h; M
<property>
' e0 T$ t; w$ ]7 d# c/ |$ A; G f8 e <name>fs.adl.oauth2.refresh.token</name>
2 I4 ^" _3 Z2 _3 z& D, B' p <value></value>
5 r$ V! Y; I3 X: x7 s <description>The OAuth2 refresh token.</description>7 O! T( I% Z( ]3 J/ g. y( }7 F
</property>6 R7 I# }/ G$ V( l3 A* {7 d
<property>. R/ V p, B1 g) c! f- Q* ^
<name>fs.adl.oauth2.access.token.provider</name>7 h9 a; m F7 p; M8 u7 V) o$ g% {$ r
<value></value>
& l' x! C% H% i# c- z: A <description>
/ k1 Q+ N+ W9 d' t Z, b" ] The class name of the OAuth2 access token provider.
; i% o3 Z1 _' \$ j8 k" [% K& Z </description>
! p& O) E; Y: z. T </property>, J* f) [3 Y5 `1 E3 j
<property>" r: h! T9 x$ m, g' ~) D+ L
<name>fs.adl.oauth2.msi.port</name>7 y$ U1 x# f' }* y- Y% E1 a
<value></value> W" U' v5 ~ p3 u3 S
<description>1 \% ?+ x6 F' o) C3 ]; _
The localhost port for the MSI token service. This is the port specified
5 D& I& I" a5 E1 ?! {# { when creating the Azure VM. The default, if this setting is not specified,
# ^; \% L' q2 V is 50342.
& i9 v3 E7 I1 _/ C* o) }3 S0 i Used by MSI token provider.2 o5 T3 G( U& N
</description>
. @6 Y K0 U T# d" J; c5 Y( Y </property>& L9 b2 z' Z$ q B
<property>: C5 H8 n8 X- w& }5 v
<name>fs.adl.oauth2.devicecode.clientapp.id</name>
% r5 o5 O8 m, G <value></value>0 o7 }7 x0 F9 h( m
<description>
2 f0 b0 o0 \7 |- a' _ The app id of the AAD native app in whose context the auth request. p ]( ` E- C! Z5 i) c
should be made.! f1 W7 b; K% Y! U' u+ q
Used by DeviceCode token provider.
, F6 `/ u+ T! j# b" D </description>
) Z2 r2 z$ i2 G1 f </property>( q/ i# @$ \& K* }1 Y6 Q- B
<!-- Azure Data Lake File System Configurations Ends Here-->
, ?( w( Q5 v! R <property>: O. ~" T0 J' m
<name>hadoop.caller.context.enabled</name>
3 U9 S% [3 m, f3 o <value>false</value>% p8 L4 b/ {# z4 H6 e
<description>When the feature is enabled, additional fields are written into |/ b1 I+ A/ X2 O( j: R3 m
name-node audit log records for auditing coarse granularity operations.+ y. j. P2 m; P& h7 _3 ]! r
</description>
% k# f) ]8 V9 J k6 i" s </property>
& x! H0 ~% U. m g4 R5 ^( v <property>7 g3 F5 y9 @. g o: a0 z
<name>hadoop.caller.context.max.size</name>
6 n3 O# B) x8 C# @ ~ <value>128</value>7 x) }, ?. b4 Z: {' T: A
<description>The maximum bytes a caller context string can have. If the" g" u6 ~) h1 X
passed caller context is longer than this maximum bytes, client will
1 |: d8 t. g! p; q/ v7 h truncate it before sending to server. Note that the server may have a; T7 D8 V. M2 K. E2 u
different maximum size, and will truncate the caller context to the
" I* W% n& j, G, C8 R1 l3 m9 G* { maximum size it allows.: e! v0 [& N" {
</description>
; P+ h8 X5 z9 b: Y5 Q </property>4 y% [3 b% A, c6 ?' W% i+ o. q0 _
<property>( u+ {6 d$ @- H3 q
<name>hadoop.caller.context.signature.max.size</name>; ?; j9 Q7 y! Z9 |, R. P! J. R
<value>40</value>
! [7 c; R; K, f9 h/ J <description>
' X0 z1 {( J! T The caller's signature (optional) is for offline validation. If the
# S _ E. `2 S& U4 }* y! o signature exceeds the maximum allowed bytes in server, the caller context
- p8 {/ ^% L4 P# K9 W will be abandoned, in which case the caller context will not be recorded
, H2 Y5 l( Y2 j4 k9 u- i in audit logs., r0 W) e+ ^, L2 m: @! u% B
</description>
( W8 h# U. d9 s </property>
* v! ]; S: N) B' [- t% K( Q9 I<!-- SequenceFile's Sorter properties -->
8 S9 K8 H) N# b L* b/ ]2 C( H <property>
- L7 Y8 W( f8 t' m f' h4 V <name>seq.io.sort.mb</name>" ?2 @. G( V1 h% R$ K2 m* |, V
<value>100</value>9 a& _5 f. H8 ~* g- w7 `$ \
<description>
1 ~3 H {8 A- }* t The total amount of buffer memory to use while sorting files,8 e2 o0 w( B2 \) c
while using SequenceFile.Sorter, in megabytes. By default,6 X- `; ]. K5 B4 {+ p& Z
gives each merge stream 1MB, which should minimize seeks.
( A1 N+ A* P D! O </description>. ^7 }# A- c3 b6 C6 V9 [) N5 t
</property>
! b8 ]" O- v+ F/ X5 ?1 L2 B <property>' T# b' i7 j2 p6 ~6 j+ W! ]3 c
<name>seq.io.sort.factor</name>
, ^9 \2 a" H1 f) e8 `1 u, r, ^; l: @ <value>100</value>3 E3 m5 r# Q; |6 `8 e
<description>1 V$ ^2 A4 r Y* n& L4 p
The number of streams to merge at once while sorting
8 z4 {" o( Q2 T Z files using SequenceFile.Sorter.$ z7 u( T& D( z
This determines the number of open file handles.
# ]4 w, O, C' j </description>
3 B) y- b/ z/ {) j </property>
0 U* x+ \, c; ~( e1 A <property>
( o `" [4 m% x <name>hadoop.zk.address</name>/ ~: n1 j; N5 r* N' `/ k0 N; i# s: ]
<!--value>127.0.0.1:2181</value-->
5 Y# ^5 y) z. {% f1 J5 D <description>Host:Port of the ZooKeeper server to be used.' i4 u# G$ T* @
</description>
) r$ q* j+ V( \$ W1 D </property>% `* C) |' @; N# e5 `
<property>: B8 o/ J! m2 `7 F! ?, [
<name>hadoop.zk.num-retries</name>, I' d2 v% o- W$ {( ?1 P
<value>1000</value>0 B K' N% s. z0 _
<description>Number of tries to connect to ZooKeeper.</description>$ H7 z% t& I: F+ Z
</property>
# G7 G( p T9 b4 }' f# H <property>3 M; g- s- \ `( B
<name>hadoop.zk.retry-interval-ms</name>
6 g5 a! n7 e/ |0 V; w <value>1000</value>
) P1 f" W# v/ j( Y* i6 c, B <description>Retry interval in milliseconds when connecting to ZooKeeper.
7 K! q' B" V. X) B9 _* X </description>
7 s+ c9 y* D# f1 ` </property>
' l! p$ r7 x( D& ^0 s; h <property>* L2 d# p5 L6 w" A4 T
<name>hadoop.zk.timeout-ms</name>
# ]" G% v' S( ~! M% _8 C% f( A/ I <value>10000</value>
: n/ U" Y, l( h( c5 P+ W <description>ZooKeeper session timeout in milliseconds. Session expiration6 C; U# I- ^, d: I9 e' B
is managed by the ZooKeeper cluster itself, not by the client. This value is: z. k! t6 o( O; J# c
used by the cluster to determine when the client's session expires.
% M8 I; X6 O' t* w J) V Expirations happens when the cluster does not hear from the client within- ~1 Z h8 x% o9 j3 Z6 \* ?
the specified session timeout period (i.e. no heartbeat).</description>
! d+ M: J! c/ o6 k </property>
N! t: v2 B6 t. F$ O( b <property>9 C5 k3 |: m3 {. A0 m% y# M
<name>hadoop.zk.acl</name>
5 t. [' S+ ^+ U! M <value>world:anyone:rwcda</value>
, O" }$ V6 W( s% u <description>ACL's to be used for ZooKeeper znodes.</description>( }8 d7 O A% i+ |/ _+ V+ E2 h
</property>- V1 I& v9 H. F2 N7 [* R h
<property>; n, h: g/ m! u: F8 _7 l% U
<name>hadoop.zk.auth</name>" L0 T) s! W+ ?. o t
<description>
7 S! B) [8 d, N Specify the auths to be used for the ACL's specified in hadoop.zk.acl.
* P/ x, m/ z( g4 k; B+ z2 R This takes a comma-separated list of authentication mechanisms, each of the* f, a' h4 ], ~. F7 Z" g- m
form 'scheme:auth' (the same syntax used for the 'addAuth' command in
! g$ q2 M! _5 K& K the ZK CLI).
% \( D# A' E, q0 M( {9 v5 s </description>& i1 {% p2 N% X7 a6 I. k: d
</property>: N m* ?7 f& _: e3 z9 l; L+ K- J
<property>
" i9 a* {" z0 A5 k3 s <name>hadoop.system.tags</name>4 P: H: @3 }1 z& U
<value>YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT9 _6 _' K: k* Y3 m
,SERVER,DEBUG,DEPRICATED,COMMON,OPTIONAL</value>
* J6 I/ e8 l2 A4 H9 A+ |2 M F- B <description>
* e6 p0 A; z3 Q9 ~% U6 Z6 e: N. q3 M System tags to group related properties together.
; \: b: ^# V5 V* p </description>2 _( T% \- M: R7 m
</property>
) ^2 d6 S) R% B- G- V <property>
6 S+ z2 {, i. x1 B <name>ipc.client.bind.wildcard.addr</name>
; c# L4 r4 u- w' h <value>false</value>, C6 g/ i& l: x9 ~$ c6 y
<description>When set to true Clients will bind socket to wildcard2 |4 K s: o9 } I9 o! J( w
address. (i.e 0.0.0.0)6 t! [6 [$ A. q. q3 J' o
</description>! c2 j& }8 |- e1 F5 ^
</property>
- @& E U0 C2 W9 ?</configuration>
6 h% v* \% ^; s' F/ y2.hdfs-default.xml, ?# F" Q9 c0 f. U
5 {6 K" u- d' L6 Y% ^1 ~. Z2 y<?xml version="1.0"?>
3 x* t$ ~' N; o+ |<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>& G( E- F @1 H# J! e; \3 s+ @
<!--
: S0 M6 z; d6 v5 N+ F; ~8 g# Y Licensed to the Apache Software Foundation (ASF) under one or more) C% v5 D" y. u/ f1 g
contributor license agreements. See the NOTICE file distributed with
" Z, B+ y" C* v: e$ ?. l5 v: L' A this work for additional information regarding copyright ownership." U q; d: V/ L
The ASF licenses this file to You under the Apache License, Version 2.0, J* _1 V" f$ \! }, K
(the "License"); you may not use this file except in compliance with! B. ^: Z1 u' o4 h# V! E/ d! v% J
the License. You may obtain a copy of the License at
4 k( m2 G( Z* b' B5 H* z http://www.apache.org/licenses/LICENSE-2.0
, v# Y, R. F s* D, U. t1 J" w Unless required by applicable law or agreed to in writing, software+ p. E9 N1 M' ]* @$ P9 b- p" d
distributed under the License is distributed on an "AS IS" BASIS,7 _0 l' B) z Q1 Z! a' ?: p
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* ^4 g( x( U3 y) Z See the License for the specific language governing permissions and4 ^1 [. T* T3 g( p
limitations under the License./ l; W, m/ ]7 P3 i C4 L) w
-->9 q! f/ h0 a& ~, h0 _( A2 [
<!-- Do not modify this file directly. Instead, copy entries that you -->1 r: W2 g8 `4 H. F8 ~
<!-- wish to modify from this file into hdfs-site.xml and change them -->
/ ?* \& Y; e( h R<!-- there. If hdfs-site.xml does not already exist, create it. -->. g% A5 G2 p7 W8 f+ t- j
<configuration>0 N, c. v: t1 \; F5 x
<property>/ ]& }8 p7 H8 w$ ~, n" d
<name>hadoop.hdfs.configuration.version</name>( a0 q1 U* }5 x' j: x5 r( n
<value>1</value>) h3 m! v9 ^. b, Z, H; |3 ?$ H* ?
<description>version of this configuration file</description>
" V8 t, p& H$ }+ x" |# \</property>3 y% x# O& b5 A" `/ N
<property>! d4 X+ H& |- P6 x
<name>dfs.namenode.rpc-address</name>2 t p" d7 J; `7 v9 Q) M1 o
<value></value>
( o5 i" v- j. O9 o0 I6 f+ e2 Q( s <description>
" n( s# o( v! b$ K7 a RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist,% u2 d8 R3 M# g) A; l, g5 E
the name service id is added to the name e.g. dfs.namenode.rpc-address.ns18 \* `3 `- Q$ g+ S: m/ r
dfs.namenode.rpc-address.EXAMPLENAMESERVICE
4 a( ?& E) w, r' s The value of this property will take the form of nn-host1:rpc-port. The NameNode's default RPC port is 8020.9 n0 h3 t! X' J# G' c- K2 [3 ~; ?
</description>
% W6 e4 ~! I% W2 i7 n</property> ]) Z2 t0 @' R( ^. K& o
<property>8 G* k9 l$ u& U( o+ X* U. ]" \( j( J
<name>dfs.namenode.rpc-bind-host</name>, J! b# B6 d, n8 _
<value></value>, r+ G0 M1 ~0 ]% C: ?
<description>
& s: H* l! k, Y! O3 e2 T3 F The actual address the RPC server will bind to. If this optional address is+ A1 ~: F% J. ]6 H
set, it overrides only the hostname portion of dfs.namenode.rpc-address.1 j: s+ j* _5 A* Q& P- @6 V# l
It can also be specified per name node or name service for HA/Federation.
9 y% q M1 s* R. x# s# P This is useful for making the name node listen on all interfaces by
( @/ `7 u" f! J1 X" y$ `3 {# v setting it to 0.0.0.0.
1 n1 M( d* _4 H </description>
" n1 \$ a7 {( @$ r2 V; k6 o</property>
4 Y8 _4 V7 ^" c( {8 o: u7 A6 N<property>
3 N) Q7 M9 }- o. ^) b <name>dfs.namenode.servicerpc-address</name>: {* v9 ?6 z, _# [# Z
<value></value>
; j% [" L: g8 y <description>
0 @% d" Y3 N+ {) u! a RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be: f( u( u3 u8 A4 S M6 h$ D6 k
connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist,
; v0 T J3 s6 V: Z$ y* z3 x the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1( ?7 D# w! O/ r% p' F
dfs.namenode.rpc-address.EXAMPLENAMESERVICE
/ E- y2 q& P: ]* n: g The value of this property will take the form of nn-host1:rpc-port.
0 T" M/ i3 N; @# y If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.
! a# A3 y% b0 y </description>" P2 P1 Q" t; _2 }( K- E7 F5 M* I
</property>
3 y0 A6 k" ^& ]1 n$ R<property>' {) C) {- A7 ~: A; `4 z' l1 o( ~3 z
<name>dfs.namenode.servicerpc-bind-host</name>
+ s- p0 A/ v) C& r( o <value></value>
* s1 o' s4 g1 Y/ { <description>
A. q) y$ U, Q" q6 s+ n B/ U The actual address the service RPC server will bind to. If this optional address is7 \. L/ G: A. E8 S5 `% |
set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.
% g, s7 S, Q$ W( A0 _9 x' ?, H It can also be specified per name node or name service for HA/Federation.
- G& Q( B+ Z$ G ^' c This is useful for making the name node listen on all interfaces by& G* s: S0 o/ t6 ^& U
setting it to 0.0.0.0.
4 ~7 D* e3 Z8 P6 ~ </description>
8 }$ r ?- n, v6 N7 C3 v. n, z</property>
/ @! o3 \) q! X2 Z( m( [<property>( S @4 D) H `
<name>dfs.namenode.lifeline.rpc-address</name>
0 H1 Z/ z% G+ W( V( w. f <value></value>
9 @& ?$ C" P' i <description>
5 o- u9 j: T4 e9 G5 h NameNode RPC lifeline address. This is an optional separate RPC address
" p: J+ M9 h1 V/ ? that can be used to isolate health checks and liveness to protect against& }% ~' O0 D4 [' g" \0 z' Y" b
resource exhaustion in the main RPC handler pool. In the case of: l9 |% ~1 f3 x2 k; A
HA/Federation where multiple NameNodes exist, the name service ID is added
v( i3 l2 {: C" N. K to the name e.g. dfs.namenode.lifeline.rpc-address.ns1. The value of this
- w; z; m0 O) q/ U. D* ? property will take the form of nn-host1:rpc-port. If this property is not
/ w ^% N, K# _ defined, then the NameNode will not start a lifeline RPC server. By( w/ b; X8 }/ b) i
default, the property is not defined.
1 {3 q9 }' V; N2 e </description>
2 [: f3 _1 z0 Y8 a8 c" H' w</property>- W: x% L( ]) \; g l7 `! }
<property>1 S3 X2 m1 _% q8 o
<name>dfs.namenode.lifeline.rpc-bind-host</name>
% U, a% @7 N& k# ~8 S <value></value>
5 c4 A, s: n5 e" y" { <description>% h: v0 M- p# @* T6 Q2 B d* [/ P
The actual address the lifeline RPC server will bind to. If this optional w$ J5 t3 M' }: J: D* e
address is set, it overrides only the hostname portion of
+ j& R8 Y Z, D9 J) G dfs.namenode.lifeline.rpc-address. It can also be specified per name node
) t6 N/ v+ Q/ n6 `" x) `$ ?% B9 c0 O or name service for HA/Federation. This is useful for making the name node8 o* t: i$ W2 l. Z- K
listen on all interfaces by setting it to 0.0.0.0.# k: A/ r- F2 g- b `- L
</description> I5 |8 t. ^6 z" l$ G
</property> r( D% @9 C: F2 ], L2 |3 |
<property>
0 _6 y1 |. B4 y8 J4 Y h6 g <name>dfs.namenode.secondary.http-address</name>
' e$ A' C4 n6 L0 x# C+ f <value>0.0.0.0:9868</value>7 R1 m' a7 A% z6 f; u0 g9 |
<description>
* j6 o1 k2 f( [9 \% j7 c The secondary namenode http server address and port.
4 `- C, B& P( |5 C6 k7 ~ </description>$ Q5 _9 j% {8 f8 I' T
</property>
. y$ j8 a2 T6 P- M C5 L<property># D \+ O+ Z' Z- H# u& q9 A
<name>dfs.namenode.secondary.https-address</name>
- M# s1 N$ h# H8 }5 F% `/ z <value>0.0.0.0:9869</value>
8 E6 h& q, r% o8 L6 s9 P6 Y <description>
; S6 ], }# U5 [5 {, l M The secondary namenode HTTPS server address and port. E0 [* M; `9 x% e6 x$ {
</description>
6 N7 @/ ]& O: W( b7 O, m</property>/ [2 X* V! I$ V s! B
<property>
6 d: F: v! h D$ P4 L <name>dfs.datanode.address</name>( Q1 }) ]/ X' f( i0 M6 R
<value>0.0.0.0:9866</value>% y; T' i; z9 I8 k5 x
<description># x" P! [3 U5 [! _
The datanode server address and port for data transfer.
1 O. E( ?1 ?) g0 h </description>9 y9 o6 K1 }4 |
</property>- z$ h) A% E8 Z2 E9 [* i. i
<property>
6 E O9 O! _6 O. C4 a6 W <name>dfs.datanode.http.address</name>
1 \* B( s5 R2 Z1 k <value>0.0.0.0:9864</value>9 B* o) z9 ~0 ?' Y" |0 l& \: O. ^
<description>
4 G- I' o6 Q. h9 B( ^ The datanode http server address and port.
7 s+ t" I" Y1 K" m' r+ M! N! W2 t </description>
4 s; H* R# [4 Z! W</property>
- T2 y. h+ a/ x: e! [<property>
7 E+ _& e7 T" y0 S% }$ p- s' X( S2 f <name>dfs.datanode.ipc.address</name>1 U7 X; z% s3 Z3 l+ x9 n
<value>0.0.0.0:9867</value>
; E. `2 T; ]: k( ? e <description>
) }) i# r/ ~0 T& {4 @/ b+ d The datanode ipc server address and port.& k" t# u0 A/ T# G1 j5 }
</description># D+ q6 Z! U+ o4 T, U
</property>
# s S. S1 U/ T3 J<property>6 A' Y) Z) F6 k2 o$ {- w
<name>dfs.datanode.http.internal-proxy.port</name>
% J' l) u3 C( y" ]' y <value>0</value>3 b7 I' K! T! y5 n/ E' a3 q$ m* q k
<description>/ I- F3 t1 u5 D+ L
The datanode's internal web proxy port.
_2 j# i9 r- M9 ~& q; M3 K! m0 z By default it selects a random port available in runtime.2 B. `! B3 `- t7 ~
</description>0 I4 P4 s* x% U: l
</property>
" j/ o( l" [* G4 d. g9 V, P+ O<property>
. E4 r1 G: A, b4 ` <name>dfs.datanode.handler.count</name>
7 H8 T* v3 p0 a+ S8 Q7 ] <value>10</value>
. T! H- c3 A# _4 ^7 ~: i6 N; Z <description>The number of server threads for the datanode.</description>) s, R! X! o [1 |/ I% u+ @2 x! C
</property>
* p' B* y! {& A5 Y: r% l/ e6 v<property>
5 d+ i! p/ u( K: o, _" g <name>dfs.namenode.http-address</name>
% d$ S: S+ {( ^5 z2 u <value>0.0.0.0:9870</value>- S$ I8 n/ o0 n# n0 C
<description>
2 Q5 a: \, }7 ]" \5 h The address and the base port where the dfs namenode web ui will listen on.
' Q" g u: P T </description>0 a4 f% V; X: \3 W
</property>
6 J, b; \! X. ~ Z<property>" {2 O$ W5 O0 N) [2 ?. H" K/ v
<name>dfs.namenode.http-bind-host</name>; k: m* V2 t" p$ i/ R
<value></value>- n; |( D9 A6 i# M* ~' F
<description>% \5 }: q' _+ ^$ v6 \
The actual address the HTTP server will bind to. If this optional address( a( V8 n7 u' r/ k5 e a
is set, it overrides only the hostname portion of dfs.namenode.http-address.5 m" m' Z+ z7 _* a$ V1 K$ {1 h4 E
It can also be specified per name node or name service for HA/Federation.
* t H2 d. e% @( | This is useful for making the name node HTTP server listen on all
5 t o; M- x7 `* ]4 I! o5 ^ interfaces by setting it to 0.0.0.0.
9 \& A. N$ _. v- e* v4 f1 y5 z </description>
( F- h, L$ }8 s/ M</property>! x1 ^& z; e. G0 A
<property>* ~" c; f4 K6 l H. e9 ]7 p$ ~# f
<name>dfs.namenode.heartbeat.recheck-interval</name>
3 o# V. ^& g( r- r7 x" c <value>300000</value>
0 v9 u0 X r1 o9 s; F g$ y3 @ <description>
( a0 J2 P z$ t ]9 K! _ This time decides the interval to check for expired datanodes.
- h# ~' j" H: Q1 s' x With this value and dfs.heartbeat.interval, the interval of3 g& o' F! P8 K, l. h7 Y
deciding the datanode is stale or not is also calculated.
. O5 ?; E( j K# G. T, f- h- ?& J The unit of this configuration is millisecond.
, K* `2 ^ Y) P </description>8 y; Y4 I9 G! `+ P1 G5 d S# Z1 Q
</property>) x" \1 c/ `. |& v: Q* o. _, z
<property>* }* m6 _0 I0 D0 h1 ~0 T( K
<name>dfs.http.policy</name>
' x/ K& g+ P( D o/ ?: U <value>HTTP_ONLY</value>
7 r6 m/ P( e) ?! G! G7 n6 X <description>Decide if HTTPS(SSL) is supported on HDFS4 K4 V% b( y4 O ~9 G" W
This configures the HTTP endpoint for HDFS daemons:
" D8 S6 P- W; z The following values are supported:% s9 u# g+ u( k/ l j
- HTTP_ONLY : Service is provided only on http
& d7 w) O: i- S - HTTPS_ONLY : Service is provided only on https5 S9 y" z( P7 W# L' x& \
- HTTP_AND_HTTPS : Service is provided both on http and https
$ k, \) f9 o7 W6 r" }; V, M </description>) I r; t6 K4 i, B0 E
</property>; Y0 B3 N3 u1 m/ ]; H
<property>
/ J, d t1 [9 j0 E1 j! L& G <name>dfs.client.https.need-auth</name>* u0 O4 O3 } ~8 B/ U7 k
<value>false</value>3 _ i1 f, n6 N; O
<description>Whether SSL client certificate authentication is required
. ]! _, g# u+ Q7 ]/ j </description>
* z' t" s- D% ^# ]5 s2 g</property>$ s- l9 q! w2 b8 B0 R c' \
<property>1 e& c" [/ K2 ?8 d8 ~9 J; Q
<name>dfs.client.cached.conn.retry</name>' l# `) o0 c- E+ ^1 ~0 f8 G
<value>3</value>
3 [( e. Y' c: L6 Q+ T& `( r <description>The number of times the HDFS client will pull a socket from the
( H- Z# `8 M8 }$ x! S! k cache. Once this number is exceeded, the client will try to create a new' A9 Z. p v7 J+ @# A6 @
socket.
; G s+ b1 u+ C' n# m </description>
/ ]! e) y* K5 T</property>
E: ^- g) b/ M: u" p4 A" a8 J<property>5 c7 f9 ~4 p0 t, R
<name>dfs.https.server.keystore.resource</name>& z- ]0 c0 f8 f
<value>ssl-server.xml</value>
4 ^. j2 c& D1 r2 m6 U1 D: o3 h <description>Resource file from which ssl server keystore
; H+ ]! t$ B! h7 l7 i information will be extracted
, `. H' o6 b$ g7 n; x4 O9 ` </description>
9 `6 d G6 x! v: R! r3 u</property>: j3 V. n2 B' k& a, }: f
<property>
; @# q) M c5 `7 B <name>dfs.client.https.keystore.resource</name>6 X( m- q; \: f* e
<value>ssl-client.xml</value>
m; M6 K& N* A- F# p2 r8 g <description>Resource file from which ssl client keystore
3 c) ~- Z6 n) l: W, j2 N8 N0 l! i information will be extracted/ _' J* n( u f
</description>* P. ~. P H9 I+ Z4 h e
</property>
: j. p8 Q% k- t- I9 e<property>
% [, o- {: I0 J+ z& `1 ^) z: R <name>dfs.datanode.https.address</name>: W, m/ T3 B9 b- g# N8 [4 J0 E
<value>0.0.0.0:9865</value>; @& e5 Y6 Y( O0 I3 |
<description>The datanode secure http server address and port.</description>8 I* j2 g/ L+ H0 k. W
</property>- R& K' K6 O+ c4 \
<property>
' S4 y% e; i8 _2 G <name>dfs.namenode.https-address</name>1 R7 E) Z$ U4 F9 p
<value>0.0.0.0:9871</value>
0 I( z. n2 ]' }5 o1 F <description>The namenode secure http server address and port.</description>7 z3 H% B3 O1 ]3 ~1 F+ j& `
</property>
9 o4 b0 k2 x8 @) g<property>; G* _ h/ a3 ^) Z0 A i- Z; S
<name>dfs.namenode.https-bind-host</name>
, s0 _6 `- E9 ]6 y6 W% r1 P* V <value></value>
: p/ B/ J0 E$ G; c0 j0 o$ N <description>
8 ~2 A6 |) u/ G9 r9 ` The actual address the HTTPS server will bind to. If this optional address/ k% u: i) `9 k8 ?& I" ]
is set, it overrides only the hostname portion of dfs.namenode.https-address.
' {6 T2 p1 `3 R It can also be specified per name node or name service for HA/Federation." q. U' |) i3 z d( B( m
This is useful for making the name node HTTPS server listen on all: e9 c2 ]# |; A8 O: u
interfaces by setting it to 0.0.0.0.0 U; Q& E( q- B
</description>
8 @: T& Y" {4 k3 \+ o</property>
! |& L" q! w9 q9 @1 u9 H; E <property>5 n- m f* ~2 E; q
<name>dfs.datanode.dns.interface</name>
4 M3 E8 p' u. g! o! A2 U <value>default</value>
2 L) Y H' t! s' T$ {' l/ n <description>
; y |0 _7 ~ q The name of the Network Interface from which a data node should
?) Y/ s) N/ H report its IP address. e.g. eth2. This setting may be required for some
* W7 s" A3 |- E- W- Z3 x& E multi-homed nodes where the DataNodes are assigned multiple hostnames% ^0 j- p$ p0 T5 o
and it is desirable for the DataNodes to use a non-default hostname.) Y# \- }/ B: V1 g" N3 U4 u
Prefer using hadoop.security.dns.interface over% u0 f0 @; K; q4 t; \9 E$ [
dfs.datanode.dns.interface.% K- w" M9 k% M* a7 l9 Z N0 [
</description>
0 O' A" z3 R9 \. N9 Q </property>/ h4 T; f( N& N
<property>
2 U5 }: Q. `! x! O. h: k e <name>dfs.datanode.dns.nameserver</name>% X4 L1 _. p; G$ B
<value>default</value>' n) ?& \: ]! R x5 q
<description># ]' @( n1 O& _9 w5 v* y* x* y
The host name or IP address of the name server (DNS) which a DataNode, U; u2 U: E1 Q% N2 \9 W E2 }6 t
should use to determine its own host name.
t4 Y# p4 @' |* ^$ j* C Prefer using hadoop.security.dns.nameserver over+ ~, g$ e7 v* M& \; ~
dfs.datanode.dns.nameserver." H8 S* Y1 y8 W! l9 R& o
</description>
+ h2 h4 n7 \. _( J0 O, E$ P </property>
2 Z/ @+ N4 h Y6 h <property>8 z- u1 v( u T% c: Y3 b
<name>dfs.namenode.backup.address</name>5 G5 A' x4 R. ~ N6 u7 z
<value>0.0.0.0:50100</value>: u. g# l1 G' \
<description>0 B& J. i& N; p7 _# J$ g7 i/ g
The backup node server address and port.- @% B# q7 M* P: W+ O- X
If the port is 0 then the server will start on a free port.
9 a, k' i0 g5 | </description>5 s/ r* h/ D) E# g G
</property>( b# U. G1 u& k* I5 t2 m9 _/ K
<property>
4 s( x( M3 K4 F2 i& }5 v( x6 z <name>dfs.namenode.backup.http-address</name>5 i0 W0 t) c! S: n* w
<value>0.0.0.0:50105</value>
* R1 l! I4 ?, T/ _; F <description>
( J, ]+ ]. J. C: x4 O. U% R3 T# g The backup node http server address and port.- B/ N" C! b1 _& e
If the port is 0 then the server will start on a free port.4 a2 N. b& s' X s2 [
</description>6 J" V9 q! y- `4 r1 n
</property>
+ a! b& ?6 A1 t# o( F* O' l0 ?<property>
' k$ \5 S; L4 m8 B5 m r <name>dfs.namenode.redundancy.considerLoad</name>
2 S* G( q( A( J3 t, V <value>true</value>
3 j; R2 K; X& a4 w+ Y, F <description>Decide if chooseTarget considers the target's load or not
) v" ^( y9 J+ x" E </description>
|/ D* U$ V1 i" I; R</property>
j% k' X ]# f, x+ i4 D4 r! C! X2 ^% O <property>
- H' {! C# f! @6 ^4 Q <name>dfs.namenode.redundancy.considerLoad.factor</name>& d. d O- _; a: k, J* [! d
<value>2.0</value>- Y& \' t9 L+ q+ I; ^5 F1 B0 A
<description>The factor by which a node's load can exceed the average2 O7 g' f+ ]1 G0 v" ]
before being rejected for writes, only if considerLoad is true.
, _7 {& ~' f: Z5 A3 I, Y3 S) ] </description>' b. \8 u+ k9 o# l5 R1 o
</property>
! h2 h% Z" H7 T( {0 H& U- q6 ^- q<property>
$ G" I+ y' [2 I3 V/ Z* |; Y8 e <name>dfs.default.chunk.view.size</name>; v: ]4 S$ B1 E) P4 W6 @! w
<value>32768</value>
' T* r7 {* \3 |5 [( w <description>The number of bytes to view for a file on the browser.6 L9 `# A* v/ m4 L1 |: [
</description>" i, ?: I; ~2 z0 `2 H4 y" Y
</property>
3 u( ]) M; _* S+ }<property>3 u. t# d: h g: d- T( T. b/ s9 Y' Y
<name>dfs.datanode.du.reserved.calculator</name>8 M, z; r* Y9 n2 Y# c
<value>org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator$ReservedSpaceCalculatorAbsolute</value>+ i2 L4 H. B: t3 E M. F: X. S; P
<description>Determines the class of ReservedSpaceCalculator to be used for
+ E" ^ K9 w2 O7 P' a calculating disk space reservedfor non-HDFS data. The default calculator is7 k: o# z& @- `. r
ReservedSpaceCalculatorAbsolute which will use dfs.datanode.du.reserved
3 [$ ^! U* F( j! m' r0 e. L for a static reserved number of bytes. ReservedSpaceCalculatorPercentage
; y) W: I, x m* b will use dfs.datanode.du.reserved.pct to calculate the reserved number: d/ \# L; Q( i
of bytes based on the size of the storage. ReservedSpaceCalculatorConservative and
2 C/ a/ b+ z% j ReservedSpaceCalculatorAggressive will use their combination, Conservative will use
& B3 _0 W' B; q maximum, Aggressive minimum. For more details see ReservedSpaceCalculator.$ N+ G B) u8 g
</description>
4 }; f3 Q; e4 x( L6 ^</property>+ Q9 V7 F( H0 E$ s, H$ y
<property>
( @3 D2 a1 e& ~$ U <name>dfs.datanode.du.reserved</name>
, A5 l- O) ^, F; X: v# j3 _; I <value>0</value>" B8 T2 T; w' N1 g! C" Q
<description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.
6 b& N: c! O, ~4 X" m/ ` Specific storage type based reservation is also supported. The property can be followed with1 c0 E/ ^! [8 [; f
corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
8 J/ Y7 J6 B# r( A3 [# @1 ~* F For example, reserved space for RAM_DISK storage can be configured using property
$ n+ v, ~1 {9 Y8 k7 ]& ] 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured0 T7 s2 D* |: a+ h, c
then dfs.datanode.du.reserved will be used.
% E$ u3 Z) P# v3 z$ \ </description>4 c5 a- O- F. R
</property>
" L4 ^- J% d* ^& f<property>" L- W9 s# l; G" G, z# L
<name>dfs.datanode.du.reserved.pct</name>
- I$ {4 x: T; ? E <value>0</value>
% g$ E& D: ]' R' \, a6 h <description>Reserved space in percentage. Read dfs.datanode.du.reserved.calculator to see
/ d1 w1 }! G- K9 w when this takes effect. The actual number of bytes reserved will be calculated by using the
0 D( F% u: B3 Q3 M5 P6 S# Q total capacity of the data directory in question. Specific storage type based reservation
6 i0 l" M1 }4 l/ W: U is also supported. The property can be followed with corresponding storage types) `3 o' e! j( |, K8 T5 N) B
([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
, y6 _4 J! X2 Q4 l7 S# \) r For example, reserved percentage space for RAM_DISK storage can be configured using property
, u+ P% m5 Q9 W c$ D 'dfs.datanode.du.reserved.pct.ram_disk'. If specific storage type reservation is not configured- c3 e5 r% r2 k5 u# V- ?* K
then dfs.datanode.du.reserved.pct will be used.
' V. g* M3 `4 B3 F% Y: i. z; n4 _ </description>2 K7 S2 Z0 w( i+ N8 d
</property>2 c: O) a. w' p" w4 p
<property>
2 T* T; x- w0 C$ u <name>dfs.namenode.name.dir</name>% R0 M! y/ T" K+ q( c
<value>file://${hadoop.tmp.dir}/dfs/name</value>- b I! f( N+ H6 G2 b
<description>Determines where on the local filesystem the DFS name node
. J& k7 T! _0 ~) @! C$ D5 ~+ f should store the name table(fsimage). If this is a comma-delimited list5 E4 X. c7 i0 K$ z$ z" ~ |0 q0 P3 c
of directories then the name table is replicated in all of the
( ~ z7 k+ G/ | directories, for redundancy. </description>! X* f2 i6 W8 k: E( b4 a
</property>) g7 G2 K( e/ L0 I
<property>
/ @* K* k. y% N; Y <name>dfs.namenode.name.dir.restore</name>3 L$ X3 j& R8 C
<value>false</value>! u$ X( T0 L9 A3 N8 k- E
<description>Set to true to enable NameNode to attempt recovering a
: T0 M- u; C- O. i( a0 X/ ] previously failed dfs.namenode.name.dir. When enabled, a recovery of any; |$ A& S5 _! `" R; d; S' u5 \
failed directory is attempted during checkpoint.</description>$ w& k! r" R Z$ m
</property>
6 Y4 F+ ^. o' N4 o' f<property>' l' e% M7 v% o
<name>dfs.namenode.fs-limits.max-component-length</name>
/ k2 r' f8 u) H <value>255</value>
; W3 Y# d5 p) Z6 b <description>Defines the maximum number of bytes in UTF-8 encoding in each+ }# v }; v. e. S' T7 g
component of a path. A value of 0 will disable the check.</description>
0 N% L% m, t# f# f6 T</property>
# K- ~5 H1 p8 ?; {0 M* ~<property>
* U' ]% Z/ E0 C/ a <name>dfs.namenode.fs-limits.max-directory-items</name>
7 A# s6 D+ g3 E6 b3 t <value>1048576</value>
5 {$ f r: F1 t& ? <description>Defines the maximum number of items that a directory may
; J8 p8 D v0 `5 s contain. Cannot set the property to a value less than 1 or more than
3 Q: I; V& s( A 6400000.</description>
/ X& Y6 {1 }, D1 k9 S& C& G</property>+ ?2 D$ C! _) }, U% ?7 `
<property>4 K. \$ g- a0 \5 d3 `; I- r
<name>dfs.namenode.fs-limits.min-block-size</name>
9 n8 \/ g4 p4 G L# K$ I4 E <value>1048576</value>
- p' Y. F: \" O6 U1 Y <description>Minimum block size in bytes, enforced by the Namenode at create
1 ]' v4 s6 e; V9 h: r- b+ u0 t time. This prevents the accidental creation of files with tiny block
% c) S! r* b3 F+ \* c& E2 ? sizes (and thus many blocks), which can degrade. p( _: }' g! Z5 K* {+ \
performance.</description>
9 W8 r [0 f) a9 O</property>3 L9 a8 W# [1 @/ m5 ?
<property>: A% Q% y1 o( }4 o- y
<name>dfs.namenode.fs-limits.max-blocks-per-file</name>
- K4 [5 ]. u$ R4 G1 s! z <value>10000</value>3 \, t% ?7 \5 ?$ Q0 l3 j( d: u# W
<description>Maximum number of blocks per file, enforced by the Namenode on, D8 h/ \+ w2 R- X8 S+ b
write. This prevents the creation of extremely large files which can! S- `/ U& \& y: Q, z+ _- h' W! o- W3 |
degrade performance.</description>- _& V1 i1 n) s" t7 n0 a, {
</property>
. t6 J' U$ T$ E2 D" k; Y<property>
- A4 o8 r* q e/ W. N <name>dfs.namenode.edits.dir</name>
+ W. C- l$ U4 e' `. D, C* k2 A* R <value>${dfs.namenode.name.dir}</value>5 G$ o' L4 {2 T% \6 X7 J
<description>Determines where on the local filesystem the DFS name node
& O3 e7 s; v( Q0 G should store the transaction (edits) file. If this is a comma-delimited list
$ K' D( U) ~" e1 g* x( u0 F of directories then the transaction file is replicated in all of the
' I7 s# ^- F$ y6 k! V directories, for redundancy. Default value is same as dfs.namenode.name.dir
% D; z" ?+ I+ I </description>* p1 ~ O" ^) n
</property>; V* }' I7 _7 h
<property>
7 B; K% S2 z- q( @" V% } <name>dfs.namenode.edits.dir.required</name>: I6 F% M: _% Z) \
<value></value>% g: S- Y+ D- A$ X8 d
<description>This should be a subset of dfs.namenode.edits.dir,
$ v% c- x( L- s: p to ensure that the transaction (edits) file* W+ L- x* {7 C8 R: T9 ]' B: y
in these places is always up-to-date.
2 ]$ M( e/ S( Q2 T- y1 b </description>
0 V, s% T" e# d8 ?2 _$ s/ {6 h</property>
4 Q0 N5 f5 m4 F' E4 r- W<property>% T) w. a1 I3 _3 H+ ^
<name>dfs.namenode.shared.edits.dir</name>
3 N& d2 X! v$ u: v <value></value>
, \) z7 ?- G( w, } <description>A directory on shared storage between the multiple namenodes6 g& t# A- m8 x9 f8 g
in an HA cluster. This directory will be written by the active and read, h9 ]* h4 ?. o4 A9 l+ s
by the standby in order to keep the namespaces synchronized. This directory
" W+ s+ g& L: i# z2 g does not need to be listed in dfs.namenode.edits.dir above. It should be1 p# G+ [+ i" z7 L$ L2 K8 v
left empty in a non-HA cluster.
* R" A) f! t4 [0 Q+ c2 d6 X </description>
{- o# x1 P5 W: |* v</property>& ~1 @6 V& C3 t% V. N3 ]8 ~: q
<property>5 |6 U3 ]; s4 a' O9 p7 L N5 q
<name>dfs.namenode.edits.journal-plugin.qjournal</name>3 i$ J( {+ F8 ?5 h2 E
<value>org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager</value>: @7 V9 ^- ?) e. S+ ^
</property>$ l2 x* G) g8 ]; }) _/ h* o3 Z/ _
<property>
! U) c3 |1 ]) J5 s. h <name>dfs.permissions.enabled</name>
3 \: S) S8 [% }4 l/ L0 J4 Z <value>true</value>
3 o8 M& X3 s Y& `9 `; @& k <description>' Q8 W+ E8 Y) Z4 N' b- l
If "true", enable permission checking in HDFS.# P7 a% u8 O( A( ^
If "false", permission checking is turned off,# L6 |$ J0 X1 o; k4 @* }
but all other behavior is unchanged.
# ?5 U: ?. O( z3 v' M& e1 T Switching from one parameter value to the other does not change the mode,% d s6 u' J; \8 M) e: ?; O
owner or group of files or directories.. A( \ F6 @* u4 q# j0 o$ a! |9 I: \
</description>
' Z0 W/ l$ n2 h</property>
$ {6 `- \3 {& n$ r! J; _<property>5 c5 e# T( @: g
<name>dfs.permissions.superusergroup</name>
' M6 t0 d. L" Z3 Z6 C <value>supergroup</value>- h: x. ~ s8 h+ Q! b
<description>The name of the group of super-users.* E9 x2 q5 c8 \* _5 N9 k/ i3 g
The value should be a single group name.0 a8 j" L8 |/ D9 D- b
</description>
2 z3 u. j# Q6 e$ h. a( ^+ \; |2 N</property>: N. A/ s; p# R, }2 w. ^8 a
<property>
8 f% E& ?, D) Z4 j( k- M& f4 ~ <name>dfs.cluster.administrators</name>. `; W9 F' J+ _0 d9 p. Y$ t$ z9 W
<value></value>: f+ x% n) L) N9 t$ O7 {
<description>ACL for the admins, this configuration is used to control
. S9 g1 I* g* P: V/ D" \ who can access the default servlets in the namenode, etc. The value. t$ e* \6 F( m7 _
should be a comma separated list of users and groups. The user list
7 ]& E8 c% Z) V' g8 v ^8 f7 } comes first and is separated by a space followed by the group list,+ r1 q, H+ b3 X6 ~: p' ^
e.g. "user1,user2 group1,group2". Both users and groups are optional,. s" W4 b/ v& x9 o, D# d: N
so "user1", " group1", "", "user1 group1", "user1,user2 group1,group2"8 L$ s6 A+ C0 l. C& F: p7 u+ W
are all valid (note the leading space in " group1"). '*' grants access9 d( @ c8 \' ~! D0 _: O: t B
to all users and groups, e.g. '*', '* ' and ' *' are all valid.
5 `! f; q: I; v; D [8 s2 ~/ S </description>
, ]% D! E0 N$ U0 }; m</property>
2 s) G/ S F$ _1 j' Z0 o<property>% }# c5 ?3 f* k& U6 b6 M: G
<name>dfs.namenode.acls.enabled</name># Q* X/ A+ K$ X7 F3 ^1 y8 k6 g
<value>false</value>
+ f1 F8 \) p3 {# {4 p <description>
1 w# I5 E3 x5 T) T; t! @$ [ Set to true to enable support for HDFS ACLs (Access Control Lists). By
2 [/ a% n$ }1 x; _' Q default, ACLs are disabled. When ACLs are disabled, the NameNode rejects
8 p- p/ M( K: J. J$ l all RPCs related to setting or getting ACLs.
, ]* X% D. P2 L& }' E! j5 P4 W </description>- ?2 ?$ w6 ]# U# G
</property>- _% f( {, R% a7 \; Q' \
<property>
/ Z4 m; J( K, L: U% g- G <name>dfs.namenode.posix.acl.inheritance.enabled</name>
; g7 f- U4 U+ \, ?" G: _ <value>true</value>
1 l6 P+ E s/ N @5 J; e$ \ <description>3 L. s; X% a! h' c6 Z# q
Set to true to enable POSIX style ACL inheritance. When it is enabled
. V7 \9 l& h- C9 q# w( l0 i and the create request comes from a compatible client, the NameNode6 e* c* O% G; D: _! i- b. D
will apply default ACLs from the parent directory to the create mode
) _9 B6 p1 ]0 ^+ M and ignore the client umask. If no default ACL found, it will apply the6 p( x& B8 k2 c% o4 R. d4 O) v
client umask.
' ]( x/ L* P6 \ </description>6 Z% A/ e ~. o R3 x
</property>. `7 \1 I. ^- w. \& [# U" y2 _
<property>6 ?6 y; \" C: L8 U5 K$ Q
<name>dfs.namenode.lazypersist.file.scrub.interval.sec</name>2 D9 c8 t, M6 u5 e& L0 c* ]
<value>300</value>
1 D$ G. u! L1 o, e# }; b3 c <description>
. o9 \; q' z' F- ?) k. L The NameNode periodically scans the namespace for LazyPersist files with3 e! U. Y, x+ m7 S
missing blocks and unlinks them from the namespace. This configuration key
" Y- q' O" r# g: _ controls the interval between successive scans. If this value is set to 0,
7 Z. S: x0 C J, M' ? the file scrubber is disabled.( Z+ v+ I: r6 y9 i0 J
</description>
3 u$ e, \, r/ w0 D- r% w; A4 a+ ]: r8 u</property>
3 O; g- @6 ^0 J8 S6 M<property>4 j- R4 O: r2 y9 j( n
<name>dfs.block.access.token.enable</name>: ]/ U2 X% c# q4 z
<value>false</value>
- g4 @. K% l: A0 p/ O6 Q <description>
; v, z: Y6 e5 _+ |! J7 ? If "true", access tokens are used as capabilities for accessing datanodes.) X2 A+ z- H" Z
If "false", no access tokens are checked on accessing datanodes.
! _' ~8 i' v6 X3 ~ </description>/ x: }7 ]9 }9 @
</property>
+ U( B7 \8 e: Z8 @ Z. C# F$ T- d<property>1 Q( [/ v. Z, s/ y2 R- v& y+ _
<name>dfs.block.access.key.update.interval</name>
; e. S6 A- m( @( W) o, u <value>600</value>
c2 W8 A! b9 U <description>
4 J. M/ j6 V* B' s& ?/ Q1 u Interval in minutes at which namenode updates its access keys.) v1 E* x1 o. L' g8 n+ t% S1 c& ^
</description>
. y, i' Z* h* l* e7 H</property>
/ g" {7 |/ C/ X3 j2 y: \$ D<property>
1 ^; S: l- s. s- y <name>dfs.block.access.token.lifetime</name>
4 c' j2 d/ M3 S <value>600</value>
, F9 L$ t4 |9 ~+ ?/ T <description>The lifetime of access tokens in minutes.</description>
. ?8 @% z' {6 i0 \/ [5 w3 o</property>8 e3 W0 h! I" R* m# C
<property>7 V& ~+ d0 S. f0 t$ F% q5 r
<name>dfs.block.access.token.protobuf.enable</name>3 g: _: a( L3 Y4 |$ Y2 T' r2 j
<value>false</value>/ F, ^$ E( F: E
<description>; Z9 x& n1 y0 G* C& i( U
If "true", block tokens are written using Protocol Buffers.& h: B/ e/ y+ @: w2 Z& d5 j( n5 R
If "false", block tokens are written using Legacy format./ M% i7 b) L. m/ c& P! X
</description>
4 d2 F* k# s/ s0 W5 p& V, u( U</property>: \+ j6 X3 t' @
<property>1 f- m5 z7 W2 L6 m3 J
<name>dfs.datanode.data.dir</name>
8 ]7 r: V$ A; t! n$ [* f ?( O <value>file://${hadoop.tmp.dir}/dfs/data</value>
$ C% a! b& ^3 k5 ?+ I' b <description>Determines where on the local filesystem an DFS data node
' B0 O; J3 Q$ p# x- D should store its blocks. If this is a comma-delimited
% `" [; }" L1 b! F; @& C- L list of directories, then data will be stored in all named
' {' a$ A8 u# S& x directories, typically on different devices. The directories should be tagged' L/ t" L8 r4 a5 u+ F9 ~8 R! _
with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
: s$ a# i& ]% k) s, e2 G storage policies. The default storage type will be DISK if the directory does
9 i; K% B% q2 H8 j" v not have a storage type tagged explicitly. Directories that do not exist will
, Q- Q$ O6 g ]# L) k; y" p, Q3 o- ^ be created if local filesystem permission allows.8 s6 X( z; U. i3 Y. i. p
</description>/ E) B* M+ u0 A' e/ [; k1 Y7 k" J- |4 t
</property>
9 _( |) F/ D' u/ f; ?<property>
7 s4 b- S7 S" E# V2 w( o) d, } <name>dfs.datanode.data.dir.perm</name>3 d+ w0 p6 q/ ^" Z- ^& h
<value>700</value>
/ X l* A) x3 n# c1 n5 D0 c+ K <description>Permissions for the directories on on the local filesystem where/ u" R4 ~6 p% k) ^
the DFS data node store its blocks. The permissions can either be octal or/ L# o! r# X' ~8 B' e1 L: M0 Y; \
symbolic.</description>
3 z& l" w7 j7 q1 H6 I4 w& G</property>, w' P$ c9 @4 J) w# R7 M
<property>1 G( K. {7 a" x0 \9 \- V/ `
<name>dfs.replication</name>
; _+ g2 o3 {$ {: i, W <value>3</value>
4 R* y( ]0 v# T& `; C8 `. P <description>Default block replication. : _" d/ q! l8 l* D6 i6 Q6 n
The actual number of replications can be specified when the file is created.0 m: g8 a: F; @- a
The default is used if replication is not specified in create time.
9 A \9 q8 f, ^" W. y: f9 \) B </description>: u% w9 l7 l$ a; b* E/ b
</property>
( Z" e0 n6 q1 r<property>+ v4 `$ `$ k5 @0 f
<name>dfs.replication.max</name>' i/ J) c& d `1 N8 H
<value>512</value>
; n0 h7 v$ v8 @8 _ <description>Maximal block replication.
+ S! I3 F! S1 [) P* K </description>
0 A7 X# N3 y2 Y( H/ i! G3 v</property>, r! B0 T- p1 c6 \
<property>( V; p0 D2 X7 @9 G6 P! G9 i) K2 r
<name>dfs.namenode.replication.min</name>
+ k7 K5 \6 B& x <value>1</value>
! E4 M. j4 y4 c, V9 e# C <description>Minimal block replication. ! r$ e- v+ p4 w7 r+ V
</description>: ]; X+ [) N" V- Q
</property>
$ p- K1 j; @1 y<property>
" F. j& Z7 i5 {) R5 R" ` <name>dfs.namenode.maintenance.replication.min</name>+ I& w0 N& u1 ^ k3 X
<value>1</value>
: s- {% N6 U! I6 [$ ^+ S6 s# _1 B1 Z' } <description>Minimal live block replication in existence of maintenance mode.! V. S1 Q) b' |$ v$ t( _
</description>
% {4 f' \# P: e5 E; C" }5 J</property>7 W% x. x$ {! r$ a7 ?" b p
<property>
$ E# i, D4 [! `3 b <name>dfs.namenode.safemode.replication.min</name>
, t. ?* U6 C3 E( p9 `6 n. ]) I+ V <value></value>
B! y4 Y" U8 F4 Y. k6 @ <description>1 O0 v6 X: l @% I- ]1 ?8 l2 E
a separate minimum replication factor for calculating safe block count.
0 N' L& M/ m, L+ E4 @ a This is an expert level setting.
$ } Y4 \, M8 [$ n" N; f Setting this lower than the dfs.namenode.replication.min
7 j/ H; `# V2 I; @8 R2 ? is not recommend and/or dangerous for production setups.
/ B3 s2 }' j3 e When it's not set it takes value from dfs.namenode.replication.min
, l- n/ |% g, R1 T8 c7 D </description>1 U7 o3 _3 }# X- O3 u ?
</property> ~$ w0 t& b5 ]7 V* N
<property>2 ^2 |9 F& a0 K/ |
<name>dfs.blocksize</name>
- v, s9 b: o( d* s |3 z3 w2 n <value>134217728</value>0 I( \. G- m( A! D3 m
<description>4 s/ O" ?4 R s+ P! M, D
The default block size for new files, in bytes.
6 E; l1 E9 A6 l" e( Q, ^. U You can use the following suffix (case insensitive):$ M1 W/ O5 a2 j' S
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),
& r+ V: E6 m. _ Or provide complete size in bytes (such as 134217728 for 128 MB).
- R) o/ n1 x$ { U6 H# W5 i9 K </description>
% v1 J4 H5 Z+ O</property>$ w1 l& T6 E9 `3 i
<property>$ H" B( u, ~0 {: \+ H7 e6 U9 s
<name>dfs.client.block.write.retries</name>
- B% d/ g' M9 {0 C$ _ <value>3</value>9 f# B9 x' I g& h
<description>The number of retries for writing blocks to the data nodes,
+ L N5 y% \4 H) G: S before we signal failure to the application. i! T8 Y0 Q- R4 m4 y+ z
</description>
7 }5 y D. B. E) R- n</property>
" O# t: K: D0 _- X a) F, H<property>+ ~9 @( D# ^* `0 w0 J
<name>dfs.client.block.write.replace-datanode-on-failure.enable</name>
3 K4 Z' _7 H8 }% U# g! i <value>true</value>7 m' a1 C4 P2 M- r2 g
<description>* e9 t: R0 l8 R
If there is a datanode/network failure in the write pipeline," k, C8 {: Y; S4 c4 ^6 @
DFSClient will try to remove the failed datanode from the pipeline
* h( f) {& h" E8 v, U7 x4 @ and then continue writing with the remaining datanodes. As a result,: r' J3 z9 T+ {, C( T# K, Z
the number of datanodes in the pipeline is decreased. The feature is
* V2 e7 s, ^, n" y. b+ P' q/ X to add new datanodes to the pipeline.
9 V, b2 C3 B3 C$ I( k3 W) I This is a site-wide property to enable/disable the feature.
! T# D" r/ N9 c [ When the cluster size is extremely small, e.g. 3 nodes or less, cluster4 f B! Z' x! N% t0 Y
administrators may want to set the policy to NEVER in the default
# d+ C0 n7 \; g( Q configuration file or disable this feature. Otherwise, users may
! l! Z7 @! B7 U6 { experience an unusually high rate of pipeline failures since it is
% S7 `+ n9 E$ ^2 T- a, \% O impossible to find new datanodes for replacement.
9 N4 U7 o1 a* \! @ V See also dfs.client.block.write.replace-datanode-on-failure.policy
9 c/ F/ I/ ?& s </description>
9 _ a# y6 j' U* D+ v</property>2 R# n3 k' q$ X6 a# j) Q
<property>
- H7 ?! y# s' J; N' k+ R <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
* S; `- c# D8 S) E7 S1 j/ e <value>DEFAULT</value>
0 B9 l* F8 S, k# @, A# J! } s0 q <description>
4 s! G) a5 V' M! J- o; Y i. D This property is used only if the value of9 ?- `+ y Z1 E3 ~! w
dfs.client.block.write.replace-datanode-on-failure.enable is true.0 p; C& q1 }! h" |6 U* g
ALWAYS: always add a new datanode when an existing datanode is removed.
' M3 r$ Z0 ~) c8 K& N NEVER: never add a new datanode.
6 S1 S; S' M( | ?' H DEFAULT:
& ]( i# }. Y, C, \& o& [( I Let r be the replication number.6 ?* A( x l) n+ \
Let n be the number of existing datanodes.
9 H, ^; H8 j7 @' }5 i0 W/ v Add a new datanode only if r is greater than or equal to 3 and either" n4 b R8 d+ _
(1) floor(r/2) is greater than or equal to n; or
9 Z6 F+ O7 v$ j/ q4 C (2) r is greater than n and the block is hflushed/appended.: Y2 E2 ~! ~4 W+ f8 u# k
</description>4 M+ b2 {5 n. W9 K8 W. C
</property>1 B, x [3 c) s% E5 q
<property>5 ^9 c! w1 ^" k# ^
<name>dfs.client.block.write.replace-datanode-on-failure.best-effort</name>
3 J* I3 e. k% Z# U! f: h& d <value>false</value>, n, A* \6 w9 E0 L. {
<description>
7 E4 \% I9 ~! p/ I" C0 G This property is used only if the value of! H4 @* a2 X' D d1 b
dfs.client.block.write.replace-datanode-on-failure.enable is true.
9 y2 _+ I/ k( d; M4 x% W% H$ N4 Z. X! X Best effort means that the client will try to replace a failed datanode
' d, U4 v% h Q. I# r in write pipeline (provided that the policy is satisfied), however, it $ S7 a9 ]% w2 F$ \! {; s: ?. T9 p
continues the write operation in case that the datanode replacement also3 ~/ t: R( _. q% H
fails.
L. B7 U+ m, g Suppose the datanode replacement fails.7 R1 O+ X0 Y9 K' D
false: An exception should be thrown so that the write will fail.4 v/ K$ ^3 z3 p# v
true : The write should be resumed with the remaining datandoes.
! U Z: r% D( k1 h Note that setting this property to true allows writing to a pipeline. K; C$ C4 |8 e' n" P2 I) L7 i
with a smaller number of datanodes. As a result, it increases the
1 z! [: P; O2 F, H) e) A6 b probability of data loss.
% p* @# ?1 y& k/ k0 q1 v% } </description>- R. [- Y: U2 u: Q
</property>' i( p1 x0 `: ]* ~! K
<property>0 e$ S' s; K( W) X @
<name>dfs.client.block.write.replace-datanode-on-failure.min-replication</name>6 E$ S, g; X. O/ v
<value>0</value>
! R; K/ Q8 Y$ H3 S9 f/ f5 z5 a# G <description>
* q- R6 \$ C6 }* q" U The minimum number of replications that are needed to not to fail$ K) _) t7 H: w# y" l2 e, p2 t
the write pipeline if new datanodes can not be found to replace4 |) h% m0 M9 N: G9 A2 S9 G, L
failed datanodes (could be due to network failure) in the write pipeline.
; }0 T4 ?! g$ c$ t6 R2 J. }/ \: }7 K* q If the number of the remaining datanodes in the write pipeline is greater" ?9 w( V, R8 e+ ~1 P
than or equal to this property value, continue writing to the remaining nodes.4 ?/ ^7 [+ v$ _
Otherwise throw exception. a) ^9 M/ t: R
If this is set to 0, an exception will be thrown, when a replacement; N" \6 X: E- g6 f
can not be found. _5 H4 S# O( E3 j" k4 j7 q) h. F
See also dfs.client.block.write.replace-datanode-on-failure.policy
2 Y! U V, p: e' j# J </description>8 e+ X9 t# x3 x' C8 C1 r2 l
</property>
+ N' l+ T) i: n9 P<property>
* s0 a1 u8 w! j" w <name>dfs.blockreport.intervalMsec</name>( G M5 w' g2 q
<value>21600000</value>
2 X. ^" k q9 k7 d# t {- T4 U <description>Determines block reporting interval in milliseconds.</description>( D% z+ e1 j& ]" U
</property>, g' d% I9 Q9 I& I. m3 w4 b; ~
<property>) f( r% J7 g: l5 Y* D
<name>dfs.blockreport.initialDelay</name>
: V8 Z0 ^1 d: T" D- X7 ?4 ` <value>0s</value> s/ b0 L0 T- f% [& M) t, e
<description>% }1 z0 `7 z w8 _# J
Delay for first block report in seconds. Support multiple time unit
8 ~% e! k3 k: d suffix(case insensitive), as described in dfs.heartbeat.interval.* w _0 Z- z* A3 h5 w5 F
</description>
- z9 ?% D" O' V( k</property>6 |' n9 `8 Y9 z3 h4 k
<property>2 v9 q0 ? I2 j3 Q* |6 W' ]4 u
<name>dfs.blockreport.split.threshold</name>
& i! I0 e3 n8 t <value>1000000</value>
, c+ m- }; H! r <description>If the number of blocks on the DataNode is below this5 p4 v" c3 W3 I) Z" h9 W
threshold then it will send block reports for all Storage Directories( o& O S/ V: z: L' ?
in a single message.1 A) k# x- R& o1 ]* C7 T- B
If the number of blocks exceeds this threshold then the DataNode will8 {# `9 ~: g7 ^
send block reports for each Storage Directory in separate messages.4 y6 N+ a3 }4 d% Q( a
Set to zero to always split." k( F7 L7 S S- v
</description>4 x0 v* n) z# @ J$ t' d
</property>
8 X% m+ C6 k+ @2 U* R: H$ m' x<property>. z- B8 M( ~0 q) K
<name>dfs.namenode.max.full.block.report.leases</name>. }! b! @8 `5 p
<value>6</value>
. B7 ^, F2 H0 L1 x R4 B: S u# c1 d <description>The maximum number of leases for full block reports that the
) O# i* y# |/ }3 M NameNode will issue at any given time. This prevents the NameNode from9 Z- E( v* M9 V: k* K
being flooded with full block reports that use up all the RPC handler& J& c5 c5 d5 a* p
threads. This number should never be more than the number of RPC handler
2 Y7 L* O) L) H threads or less than 1.! y8 ]+ F6 n; v" p; ^2 r3 V) ]
</description>
$ s( X Y6 W( B; a</property>8 F+ m& i) R8 U" ]; g4 e
<property>
% u8 |! n( y# q" W0 ^ <name>dfs.namenode.full.block.report.lease.length.ms</name>
/ e5 B6 e9 C' W8 ?' q* d <value>300000</value>- L3 g Y5 w" c3 e" ]. w& O
<description>
( S2 S! P& ~$ w( J2 x: Y% x The number of milliseconds that the NameNode will wait before invalidating
) \' r c n* M2 J a full block report lease. This prevents a crashed DataNode from; H' ~4 e) X( ~- F7 \0 i% V
permanently using up a full block report lease.+ m, d/ J- X0 s, U
</description>, C9 y( f8 `& w$ M C. @
</property>7 C% i! L5 u( t4 H
<property>
; `$ u$ r& Q# P* E$ D <name>dfs.datanode.directoryscan.interval</name># v8 a2 L& E% A; |
<value>21600s</value>
- w1 ]; [6 a5 d# V <description>Interval in seconds for Datanode to scan data directories and
$ f: v# F1 I3 L' J5 T reconcile the difference between blocks in memory and on the disk.+ U' E5 s6 g! @) }; e
Support multiple time unit suffix(case insensitive), as described
# d6 q- ?- P) ]1 [ in dfs.heartbeat.interval.
, _, D; g- G3 s </description>2 u) ]1 B; p( D, q6 {# c
</property>) T" q6 \, D) L) q O7 x! H }* D
<property>
* Y$ p" E5 t6 d, u K7 L <name>dfs.datanode.directoryscan.threads</name>; }2 ]9 `" c! k
<value>1</value>
a' E, Y$ w1 [; Z" P! {: _ <description>How many threads should the threadpool used to compile reports
5 g" l6 y4 d; M8 N4 b' y; f for volumes in parallel have.
$ z( w( ~$ K1 t' f7 i5 } </description>
) I- ?1 w- D0 p</property>
! w9 s8 J+ m- l1 M7 |; t% x9 E<property>- f( ]! m3 @: J9 t# _8 c3 e# F/ s
<name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name>+ a( h9 t2 ~. W: F1 N; _
<value>1000</value>0 `9 U. M; @, o" _3 c
<description>The report compilation threads are limited to only running for- R4 E, Q% B( t8 o" F' E
a given number of milliseconds per second, as configured by the& N% D0 X( _8 I4 L9 c
property. The limit is taken per thread, not in aggregate, e.g. setting6 Z8 h8 U, r$ h! m
a limit of 100ms for 4 compiler threads will result in each thread being
X% n' d$ A* r1 }, v0 y limited to 100ms, not 25ms./ q; Y2 q, E q8 j
Note that the throttle does not interrupt the report compiler threads, so the1 D6 j! m* P1 \3 A5 V# Q1 u
actual running time of the threads per second will typically be somewhat
/ }. n" C" x$ I( Q+ p C higher than the throttle limit, usually by no more than 20%.
1 N2 x1 V& y: I Setting this limit to 1000 disables compiler thread throttling. Only( _1 m; p) I3 G2 j) V% w4 H
values between 1 and 1000 are valid. Setting an invalid value will result% ^, }' w9 B( A& r9 r* o+ A* R
in the throttle being disabled and an error message being logged. 1000 is
" p% b1 Q: x; X the default setting.1 {- B6 m2 s! d& P# \2 W
</description>
+ m0 P& W5 | l7 G</property>8 |0 J% F+ m# z1 R! X2 ^6 ^
<property>
- s, K7 S# x" ^ j- U( x <name>dfs.heartbeat.interval</name>
$ [% ]+ w. }) I: V6 ?9 K <value>3s</value>7 M9 S' C9 c" N2 {- i
<description>
a7 {/ \( `# @) D( w$ J) K, F Determines datanode heartbeat interval in seconds.
) `* U0 T' M( {( }9 m Can use the following suffix (case insensitive):
; r5 @5 B+ } \ Q" n& m4 }) | ms(millis), s(sec), m(min), h(hour), d(day)
2 n# I4 ]* Q+ F to specify the time (such as 2s, 2m, 1h, etc.).
, T! r' B, n! p" j5 q. r Or provide complete number in seconds (such as 30 for 30 seconds).1 K4 |( K4 }( D4 `2 B
</description>
+ j7 @. e" r# ?6 F</property>( y, P& D8 m/ `' a2 ~) q
<property>) b0 ~' l1 v* p. t
<name>dfs.datanode.lifeline.interval.seconds</name>
! C3 _% f. n3 e: r <value></value>
1 U) I7 n8 q3 F z <description>2 L1 Z9 L' z, p2 u2 H- ?8 a
Sets the interval in seconds between sending DataNode Lifeline Protocol" \2 J+ z; ?& p+ v
messages from the DataNode to the NameNode. The value must be greater than
n$ N4 C5 `. B' A- {: ` the value of dfs.heartbeat.interval. If this property is not defined, then
& l) s% j4 T% V; x, \2 P5 @7 C) O the default behavior is to calculate the interval as 3x the value of
2 X& }. W0 {3 B6 D) s5 k dfs.heartbeat.interval. Note that normal heartbeat processing may cause the d q& @- p W5 ]* [9 o
DataNode to postpone sending lifeline messages if they are not required.
# j e6 U+ W' X' J; S* z e; d Under normal operations with speedy heartbeat processing, it is possible$ w" n2 L [! e
that no lifeline messages will need to be sent at all. This property has no
" c5 U; L& |7 d" [4 h effect if dfs.namenode.lifeline.rpc-address is not defined.
2 v5 _- \5 Y) u5 N6 a4 L </description>8 y, t6 P" y+ E! H. h* X6 ]
</property>
U2 D( ]$ v" M) e# D2 X& p- x<property>
. X5 A* T8 b/ Z7 I8 {, F9 s <name>dfs.namenode.handler.count</name>) S1 P3 o: p; k( F6 \) Y) r
<value>10</value>
G* {* X! m- P9 j4 \ <description>The number of Namenode RPC server threads that listen to
* ~. @) {3 C3 E/ r requests from clients.
+ [ q0 G9 [* g" T If dfs.namenode.servicerpc-address is not configured then
8 C& M2 J' A6 Y& H- x% t Namenode RPC server threads listen to requests from all nodes.2 i9 e% X* y$ O7 V/ d' `% m R
</description>/ [! M* k- `; V& F$ K# r
</property>
, Q2 X2 k5 {7 \" h$ I4 N; K<property>
3 I* |! T2 s" `, f: k$ I: R2 a <name>dfs.namenode.service.handler.count</name>
4 D/ M, f' g5 W <value>10</value>
& ~5 }2 [3 T6 Q7 F7 k <description>The number of Namenode RPC server threads that listen to% P3 u2 z7 [7 z6 d/ ]# r
requests from DataNodes and from all other non-client nodes.1 O b- @" x: b5 {
dfs.namenode.service.handler.count will be valid only if
3 t4 i* X, W3 d( Z' y dfs.namenode.servicerpc-address is configured.' V% v n/ c9 J' G$ z' N5 N' M
</description>
1 a( Z w. r& F7 ]</property>
& H% x2 B. c" P) U% D, y3 x9 F; l<property>2 _' M* R" C4 u1 {6 ?4 n- q
<name>dfs.namenode.lifeline.handler.ratio</name>
1 Y# h1 r0 X" p5 ^ <value>0.10</value>, Z. K' i1 p1 B
<description>) [+ V u* O6 f- O
A ratio applied to the value of dfs.namenode.handler.count, which then
, I; s7 v6 J8 ~2 X) _ provides the number of RPC server threads the NameNode runs for handling the1 {( F- s! Q% `- x0 m
lifeline RPC server. For example, if dfs.namenode.handler.count is 100, and+ e2 i7 ]: [& v# y+ J
dfs.namenode.lifeline.handler.factor is 0.10, then the NameNode starts
4 o6 J, m/ N* S" I. a( ]$ y( W 100 * 0.10 = 10 threads for handling the lifeline RPC server. It is common' m. f1 v. g* r: ]5 S
to tune the value of dfs.namenode.handler.count as a function of the number
2 E: y( `0 K( }% |3 B of DataNodes in a cluster. Using this property allows for the lifeline RPC/ r1 C7 I8 \+ {$ U! K' R
server handler threads to be tuned automatically without needing to touch a
. L8 h D5 T: C separate property. Lifeline message processing is lightweight, so it is) k; ]4 x6 G0 h- O0 [" C
expected to require many fewer threads than the main NameNode RPC server.
5 ?' L) }6 U9 Y6 L4 S This property is not used if dfs.namenode.lifeline.handler.count is defined,
' e* `, R0 I. i1 b* _+ { which sets an absolute thread count. This property has no effect if3 S: b: Q- l6 ] y/ ]1 L* Y
dfs.namenode.lifeline.rpc-address is not defined.
3 t% \! I; n6 t+ I8 F/ N, K( L </description>5 k% |. I) ~" s L
</property>
% h- G% [% @& x9 i6 B<property>
4 q$ ~5 A$ P" z8 O3 I- k4 w <name>dfs.namenode.lifeline.handler.count</name>
6 L3 y. |; f, k( p- e, S* ` <value></value>1 W- m: Y0 ~) D* Y
<description>
* D- E, T5 a; _/ L2 o$ x* M4 Y Sets an absolute number of RPC server threads the NameNode runs for handling( r1 i( D& i6 [' S/ K2 U
the DataNode Lifeline Protocol and HA health check requests from ZKFC. If
) q' G+ S9 H: A. P |" [ this property is defined, then it overrides the behavior of
" n9 \0 F% \5 ^- ?: } x: b4 e dfs.namenode.lifeline.handler.ratio. By default, it is not defined. This
1 `7 l2 A( m2 P: f property has no effect if dfs.namenode.lifeline.rpc-address is not defined.$ {3 F/ l# ^" o3 N8 Q6 W x
</description># }6 f! x( J& n
</property>" U9 U, c' x' F
<property>+ w; K8 Y- c. X/ v ^& ~0 U
<name>dfs.namenode.safemode.threshold-pct</name>$ n! Y8 \9 H& ~, n( y/ x/ g& j7 X; Z
<value>0.999f</value>' D1 f0 w; O3 ]
<description>$ x, B0 g" n: {% H
Specifies the percentage of blocks that should satisfy
% q1 q9 {3 h# T# ~& y the minimal replication requirement defined by dfs.namenode.replication.min.7 `* Q6 o2 C* p9 Y
Values less than or equal to 0 mean not to wait for any particular) P& x5 u) d! ?! d' H4 ]3 `
percentage of blocks before exiting safemode.
. I, K4 |5 a3 W1 W Values greater than 1 will make safe mode permanent.
5 S0 ]4 _' h% p. X </description>8 ^0 {2 Z3 {. t+ e& h) C% O7 t% @
</property>
, n/ d4 ]4 Y2 A% i4 D1 Z5 G: z R<property>
& ?$ g" ^* P$ l" o+ m5 b <name>dfs.namenode.safemode.min.datanodes</name>
0 P& ?$ y& P) n; E- f% u% o <value>0</value>; }( B; u7 A' s: v. y9 Z; \" y" ]- y
<description>
, Q7 o, f1 Z& x& F: b" L3 u Specifies the number of datanodes that must be considered alive
# \* r# M9 Z d before the name node exits safemode.
8 g2 F W* u/ e# a A0 K$ ` Values less than or equal to 0 mean not to take the number of live
( a0 u7 O: K9 Z Y" s datanodes into account when deciding whether to remain in safe mode2 R& U3 _; E( q, @& a. S
during startup.4 d- A9 v' D/ ]/ m1 ]# j
Values greater than the number of datanodes in the cluster) t( m" V& J6 _ d: b/ F3 P( n
will make safe mode permanent.: ^# j K; k" t6 _( a) C: w
</description>& z0 V& u# f! L8 n# Y" z/ I7 `. B. f
</property>; Y5 `" M/ N9 K, _
<property>9 K5 v5 l9 h- D; n
<name>dfs.namenode.safemode.extension</name>; k9 f( t0 B7 O% j4 @
<value>30000</value># G+ X. ^- U _
<description>
" T4 _9 [, ^" J, J9 _ J* S0 r+ t Determines extension of safe mode in milliseconds after the threshold level' D4 d- r" {( E& R- \
is reached. Support multiple time unit suffix (case insensitive), as
7 @5 f$ a# ?. `' j7 B described in dfs.heartbeat.interval.8 t; {) Y0 v$ e) i
</description>
& i' F" @) b* e! M! ^% C6 I</property>
8 Q: t; c) J9 q3 @; ~. U<property>* e5 T+ {% p5 Z5 p6 J
<name>dfs.namenode.resource.check.interval</name>" p/ z9 `, u7 ?5 P$ N6 i. z
<value>5000</value>
, m0 J% c% ~% F6 P$ C( M <description>8 D2 z& R3 S, Z% Y* }
The interval in milliseconds at which the NameNode resource checker runs.) k' r/ M+ W. G: ]; z
The checker calculates the number of the NameNode storage volumes whose2 Y4 \) k5 d* q2 {. w3 n g
available spaces are more than dfs.namenode.resource.du.reserved, and
+ J$ {4 Q7 A% j( P enters safemode if the number becomes lower than the minimum value, j% e3 s! @; @# v7 T
specified by dfs.namenode.resource.checked.volumes.minimum.
( }$ d6 e5 P( u4 e2 D5 x </description>
. e9 p! t5 [+ o' z6 T% |</property>6 G- K' d" F+ a% j8 t" p
<property>4 g3 a) J z' K8 e9 ^5 Z
<name>dfs.namenode.resource.du.reserved</name>
4 D' e9 m0 H4 I! z <value>104857600</value>
+ a. K5 M9 g! M: Q <description>* u# d( y" [6 l4 V1 Z+ |
The amount of space to reserve/require for a NameNode storage directory$ d& a$ T( Y) \: b
in bytes. The default is 100MB.
$ u' v' f* V# {! U0 t4 V </description>! ~& ]6 p4 `# W0 y) H, n5 Z+ i
</property>
C2 a( P( \5 o8 C) i! c" J<property>1 d }" G3 j& G( H2 n3 o& ?4 l% X
<name>dfs.namenode.resource.checked.volumes</name>/ x' y0 g2 e5 t- x
<value></value>8 M3 h4 Q6 A; b4 r* B' p* E
<description>& C" E4 G( s3 L7 Y7 q2 m8 z& i1 T
A list of local directories for the NameNode resource checker to check in
! }8 J& |" N$ g* h# j addition to the local edits directories.; j1 @* \0 w' f# z' a8 G' r
</description>
0 @ Q* S" g: f; A, S4 v, a</property>
7 F4 H- A/ h5 g8 Z5 a0 c- t1 C<property>
7 D! z/ O9 j0 r <name>dfs.namenode.resource.checked.volumes.minimum</name>
5 S! h3 ]* S1 } <value>1</value>
8 g& K4 k; s1 [# v G% O8 K <description> O6 {% Y9 H2 e* m' o$ V$ M2 z, B
The minimum number of redundant NameNode storage volumes required.: f. t0 _; J ^. F- h2 k2 z. B$ `' I
</description>
) ~- S& k, a+ M</property>( \6 z. a+ v' R* x) I& D/ l! K8 F
<property>
- }, e2 Y: s) ~( }) ?/ v" [: s <name>dfs.datanode.balance.bandwidthPerSec</name>
o C0 M) j; v <value>10m</value> p' y( S) U, t) F
<description>
4 m3 z9 M7 Z3 a* q Specifies the maximum amount of bandwidth that each datanode
* E3 G% _3 O' Z; Q( K0 U$ e can utilize for the balancing purpose in term of
4 x/ g& g0 D/ _ the number of bytes per second. You can use the following- B! l, ]. f! a0 I) ^8 c" n
suffix (case insensitive):
4 i; n- S5 c' |5 n k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)to specify the size. P" n7 E, }5 o
(such as 128k, 512m, 1g, etc.).
/ x: K# B" x Y; Y' t4 T Or provide complete size in bytes (such as 134217728 for 128 MB).' ~5 ^0 V8 z$ `0 K* q: W: J
</description>3 v3 k9 a) L9 `- L
</property>; E/ z: l* `6 x" W% V
<property>2 Y P9 P, Z. b4 d$ K* ^
<name>dfs.hosts</name>
$ j" Y! [* C4 B4 l! q <value></value>
- [( q1 e) F+ C7 C( G# G <description>Names a file that contains a list of hosts that are4 j0 t! v. p6 ~: i7 O9 X4 a
permitted to connect to the namenode. The full pathname of the file$ R, c( M/ ^) b# m$ \! i
must be specified. If the value is empty, all hosts are3 C3 x9 h% K0 R' Y4 {( i
permitted.</description>8 [" p& K s# U+ i' x8 L9 @
</property>6 g4 N4 I, F2 y; T4 m4 P
<property>
( m# ~/ v, B$ e <name>dfs.hosts.exclude</name>! ]* _; Y: s, d/ `+ {& P
<value></value>
! [$ f; E; L, N/ [( c7 s <description>Names a file that contains a list of hosts that are s5 U" f( i. G. q* s0 Z( P/ W
not permitted to connect to the namenode. The full pathname of the2 u& ^9 ^8 Y" h1 m) m$ D
file must be specified. If the value is empty, no hosts are5 `3 e7 U* |- ?; b0 Q6 U
excluded.</description>
: }' t- u. q4 q. ?; h2 v: ^# E6 |</property> - `* c. u+ Q0 q+ U" r* a+ u
<property>2 a& a' D$ O0 c0 o+ X9 U! x
<name>dfs.namenode.max.objects</name>6 i. c" @7 x! I& {; h& P
<value>0</value> h3 X9 {. K' N9 s
<description>The maximum number of files, directories and blocks+ D2 J4 t7 Q( c7 V: W% ?6 M9 G w' N
dfs supports. A value of zero indicates no limit to the number6 ?: G0 q8 _" z0 `- f
of objects that dfs supports.0 l4 k6 k2 Q( D" `$ m
</description>( y7 m5 N: ~2 S: ~. M
</property>
( M0 s& F! ]* ^) a! E1 g8 q( W7 m<property>/ D$ q' l' I+ p
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>- t- F; N b7 J& W
<value>true</value>; \* W' @! Y) Y& R
<description>
! i4 `+ R3 i' U z, ?! z! h+ m" h If true (the default), then the namenode requires that a connecting
. R0 A6 y" z* W datanode's address must be resolved to a hostname. If necessary, a reverse% H" P3 \; Y" N6 ?( X% _
DNS lookup is performed. All attempts to register a datanode from an
; i" r n! A! V1 f& m5 f unresolvable address are rejected.
2 _: Q! E, Y- \: `# }/ I2 O& D2 A It is recommended that this setting be left on to prevent accidental/ Q) X* U) e' ]+ g) Z! X% h/ V
registration of datanodes listed by hostname in the excludes file during a
& {8 e# ?/ O- | C# l% { DNS outage. Only set this to false in environments where there is no
7 Z# L# i! l4 i W infrastructure to support reverse DNS lookup.
: l: V# g2 B7 ^5 C5 ]+ ` </description>
: u4 @1 I) ?9 K' \8 F8 _7 u* ]</property>
/ p' x2 \! f+ O% f' Z! T2 s/ ]- n<property>1 r$ c; ]& x. w3 ^ B% n& s
<name>dfs.namenode.decommission.interval</name>) |0 v7 q2 J* y" q' y
<value>30s</value>
. u( [4 H% g- y- t) J <description>Namenode periodicity in seconds to check if
% g6 Q0 }* V% c1 A( ^1 Y decommission or maintenance is complete. Support multiple time unit
% y2 J7 V( ~5 N suffix(case insensitive), as described in dfs.heartbeat.interval.0 M6 R, u6 a! w0 A4 F9 j. W/ |6 q
</description>' c1 f, J- e& U- s
</property>$ T& r4 [5 W ^ ]. X
<property>! C, r5 r/ o: ?! k$ o
<name>dfs.namenode.decommission.blocks.per.interval</name>2 m! d, |; o, g+ V
<value>500000</value>
- }8 G/ y6 h) B# g( A5 L <description>The approximate number of blocks to process per decommission' o# n9 z# S( A
or maintenance interval, as defined in dfs.namenode.decommission.interval.# R6 U! E* D' }9 b
</description>
* Y7 d# J" \7 ?* X# @) }+ {. R</property>6 d. _" S" b. \6 j( ^
<property>6 y a& D( H {) w0 O1 y7 a- q% J. K ~
<name>dfs.namenode.decommission.max.concurrent.tracked.nodes</name>( P0 G0 Q+ G" a, e
<value>100</value>4 {! t* \' Q. d: s- g; o; y
<description>. e2 `9 m, ]& K2 e4 B- k
The maximum number of decommission-in-progress or
8 T- S6 ^; R2 w; h& v- w8 c6 I entering-maintenance datanodes nodes that will be tracked at one time by
9 h+ T( u. O+ A0 o3 b the namenode. Tracking these datanode consumes additional NN memory9 B$ B/ Y; f. i; i: v/ L
proportional to the number of blocks on the datnode. Having a conservative# M7 z! y& v; n/ r9 V; _3 u
limit reduces the potential impact of decommissioning or maintenance of
+ [5 S0 U- e X. N; @. m u a large number of nodes at once.
$ x/ G- @( \! Y2 m A value of 0 means no limit will be enforced., f3 Z: g. | f
</description>
% F7 S3 r: k r& u7 }+ D</property>+ @6 l: _) ] u3 c* y
<property>& e0 M% @. _3 B8 Z( l% k
<name>dfs.namenode.redundancy.interval.seconds</name>
1 [; c! k7 y4 d <value>3s</value>
l* o: g5 `# H6 W* A; p9 [ <description>The periodicity in seconds with which the namenode computes " @ z% A( \, k1 a; P0 E& [
low redundancy work for datanodes. Support multiple time unit suffix(case insensitive),9 ?& H L6 R' z2 M# ^1 _% ^2 a
as described in dfs.heartbeat.interval.4 b0 [& D/ [, Y/ R' g% J4 k
</description>
0 e& a' f X6 u+ u/ ~! s</property>8 D, |, J' \- w6 `
<property>6 r- t5 c1 O+ z$ v9 v; `# w6 `: u9 p1 ?
<name>dfs.namenode.accesstime.precision</name>7 V7 W( E2 ~7 H! ]2 O) B% f% v' u
<value>3600000</value>
6 b4 w: O. w2 \2 E" ] <description>The access time for HDFS file is precise upto this value.
: Y0 e) \3 G. x6 B* \1 A The default value is 1 hour. Setting a value of 0 disables0 M' s, b: x* ~0 B9 S+ x; `, K3 ~
access times for HDFS.1 ^, y$ X# l* U$ S& U2 w `1 S
</description>
) h, i, h, ^! V5 I9 t8 c, j1 y</property>
1 C# B% _8 a3 c9 p7 N; g% r& m<property>$ a, A6 S- i+ a( H- k% D: q
<name>dfs.datanode.plugins</name>
1 ~4 W7 g0 {, ~/ D* G6 W( [$ T <value></value>$ R; I+ f- p1 n9 l& w
<description>Comma-separated list of datanode plug-ins to be activated.
+ {& N. X t' c3 K </description>* q9 U) [( @- p( E+ {4 H' l+ z
</property>
& V# F/ J/ I k7 v5 ^, C+ q* z4 }$ P0 x<property>
0 `% O W0 Q# k. t, _7 T6 `! D <name>dfs.namenode.plugins</name>
9 r7 f3 Z& j# U; V' S/ _# l8 G <value></value>
( l* s9 k9 m+ _( [) H, \ <description>Comma-separated list of namenode plug-ins to be activated.9 f: U) y0 z' @ P \
</description>
3 k8 |8 _3 h' D</property>
( n7 D. U+ D7 v) ?! m7 [$ V4 ]<property>) i z3 ~6 b8 u7 h, w
<name>dfs.namenode.block-placement-policy.default.prefer-local-node</name>! R: d8 ^( z# N5 ^
<value>true</value>
3 R+ M7 U9 J0 X, }; a: w% j <description>Controls how the default block placement policy places
+ p( W! y5 b# v the first replica of a block. When true, it will prefer the node where8 i' l. T) ~. t6 k* ~. z
the client is running. When false, it will prefer a node in the same rack
9 c6 v2 B" c X as the client. Setting to false avoids situations where entire copies of+ L1 \" ~$ B+ w9 x- ~) g6 z
large files end up on a single node, thus creating hotspots.. a. T4 X6 G/ H$ n/ Y
</description>% \# B* h: T5 |% L/ d
</property>
* _0 }' _$ ^ ~) R+ E<property>+ r p; p/ i8 Y2 r
<name>dfs.stream-buffer-size</name>
) q8 u$ p; p9 f! D0 a- h <value>4096</value>: G# n; G6 ]/ [# e" e
<description>The size of buffer to stream files.5 r, b# ^- O% b& p* R5 B3 q7 J
The size of this buffer should probably be a multiple of hardware$ E" Y) k0 m; x' n8 z
page size (4096 on Intel x86), and it determines how much data is
: _9 M' ?7 j0 H6 ]- k# G5 W6 b% } buffered during read and write operations.</description>; T* I$ X# L$ S
</property>! D/ N( e& b8 h- `+ }6 t( z+ ?8 `: ~5 p& F5 e
<property>
- b/ i8 W) ] J3 u* j K6 D+ n <name>dfs.bytes-per-checksum</name>
6 r6 s1 _, O9 g. E <value>512</value>
0 z; u1 S+ ^3 Y/ d <description>The number of bytes per checksum. Must not be larger than* t0 P2 c0 p$ G7 A s0 K3 e
dfs.stream-buffer-size</description>6 c+ R4 D! }# @2 K2 f
</property>
& r5 Y7 F, K2 \' F6 z t) L8 I<property>
% b4 G, N7 _. l6 K <name>dfs.client-write-packet-size</name>
7 `, s) J# w" w" _8 b <value>65536</value> G/ F7 u; p, k% v" q+ s8 H* ~2 S( Q. Y
<description>Packet size for clients to write</description>7 @9 S8 w7 W% K& i" R
</property>& F( n" C- ^$ _
<property>
% Q. ?, T z2 R( _) o: z <name>dfs.client.write.exclude.nodes.cache.expiry.interval.millis</name>) P- A) w* R/ [# F0 I `
<value>600000</value>/ S0 X# @& f, T3 P8 U3 j/ z
<description>The maximum period to keep a DN in the excluded nodes list
% |, I% m# M- J2 m; h Q at a client. After this period, in milliseconds, the previously excluded node(s) will
5 h; _3 Y9 k6 |! D be removed automatically from the cache and will be considered good for block allocations6 H |" q2 i9 s7 E
again. Useful to lower or raise in situations where you keep a file open for very long
% f, W+ q" }! M: \ periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance
) Y* o" [" j" G8 K restarts. Defaults to 10 minutes.</description>
; [. j- B2 r7 Q</property>2 B) I" C% i; u
<property>
* B) y( w2 Q d0 `$ q" f9 ? <name>dfs.namenode.checkpoint.dir</name>
" P* B* a6 o, C! ~$ c* Z, u <value>file://${hadoop.tmp.dir}/dfs/namesecondary</value>; n8 s8 }9 x$ k" f
<description>Determines where on the local filesystem the DFS secondary
( F- o! _, s( _4 g) m' D name node should store the temporary images to merge.
6 y, p3 W7 {( l& f- F- K If this is a comma-delimited list of directories then the image is. }' ^* u6 d" s
replicated in all of the directories for redundancy.
( J6 u* t4 C& [2 o </description>+ C u0 V& x& K, m5 p/ @: D+ K
</property>: |. c& y$ j9 `) T O
<property>
7 P: n6 l1 @ }( ?+ s <name>dfs.namenode.checkpoint.edits.dir</name>
! ]) v- c2 {' v0 P! k* P <value>${dfs.namenode.checkpoint.dir}</value>
7 u" m5 t1 `& K: a5 z! t <description>Determines where on the local filesystem the DFS secondary! q3 N' G( a9 C1 x/ B& b& v
name node should store the temporary edits to merge.
7 a7 ]8 R+ C- l& E If this is a comma-delimited list of directories then the edits is6 V% e& f. o4 ]. o; z
replicated in all of the directories for redundancy.
+ h' u- E' u* l8 {! p Default value is same as dfs.namenode.checkpoint.dir
# c8 w( g9 X# H6 X8 Z3 l( p2 k </description>
& a2 ?" C: d# E; g, O2 {' I& }</property>: d: U6 ?: y. k! r) D
<property>7 f6 L3 ~- ~: i: u9 R( t1 _$ G8 U
<name>dfs.namenode.checkpoint.period</name>
. ?7 C/ ~# H0 E/ u' p, J* p <value>3600s</value>
0 C( s# r* |! p; ~3 d <description>
1 @9 P; ~2 A* w The number of seconds between two periodic checkpoints.$ v2 f5 F* J$ ?) l6 K/ |
Support multiple time unit suffix(case insensitive), as described1 }7 e7 v( ~' Q% |+ d
in dfs.heartbeat.interval.
2 Q' Y# }1 s9 [# j9 K </description>
- e+ }" v' k+ I1 C7 F8 r; S</property>5 _7 \0 h: W9 g/ k1 c3 v4 k
<property>+ r. D+ }. Q+ T: s5 p* Q2 c
<name>dfs.namenode.checkpoint.txns</name>
, o/ Q& o5 ?6 g" j* Y9 E, [& A <value>1000000</value>& u3 G% O/ M* ]& K! }* W9 {
<description>The Secondary NameNode or CheckpointNode will create a checkpoint
" C1 A) C. `& K/ l" |4 ] of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless
2 O6 W+ D1 D5 M+ x$ K5 u4 y" I4 U of whether 'dfs.namenode.checkpoint.period' has expired., |+ \- j( y2 B' R$ N7 X' ?1 N: I
</description>
^# \5 K) G) Z9 n( Y: w</property>
! b1 p% ^/ Z4 m<property>. Y. x3 o1 o+ l2 e4 E; W, ^
<name>dfs.namenode.checkpoint.check.period</name>" g5 \2 q/ X1 E
<value>60s</value>
4 j, w! u/ E' U$ u' X <description>The SecondaryNameNode and CheckpointNode will poll the NameNode4 t/ G$ i. \/ O( \! V
every 'dfs.namenode.checkpoint.check.period' seconds to query the number$ c. p% L0 ^$ V# m1 h9 D
of uncheckpointed transactions. Support multiple time unit suffix(case insensitive), I' f) Z3 K, j! t
as described in dfs.heartbeat.interval.3 q, W) }3 `( l, ^8 V
</description>
8 z4 K2 Y* A9 _0 `</property>, H/ X. g2 m# t$ c `$ w
<property>
; a! v2 @1 `5 x <name>dfs.namenode.checkpoint.max-retries</name>
" D& l' T" U' L4 h <value>3</value>: o! M7 z/ i9 y5 ^
<description>The SecondaryNameNode retries failed checkpointing. If the
" |$ \) I3 C- o! s+ k failure occurs while loading fsimage or replaying edits, the number of
1 @6 C% [6 O4 q4 W7 l' O retries is limited by this variable. " v' _( z. m2 t* P' B1 v) R
</description>. O7 j- h6 ?$ m: A% K
</property>/ L1 p- y0 ~2 y* X
<property>
& j; i `. i1 P4 d' k, v <name>dfs.namenode.checkpoint.check.quiet-multiplier</name> c0 M7 m( i/ C& m$ h* @8 d- h
<value>1.5</value>
2 J" [: f6 b9 s C+ t, ^ <description>
7 B/ D5 B5 R1 k7 E Used to calculate the amount of time between retries when in the 'quiet' period
/ D' W8 D! T* Q- m5 D# u; | for creating checkpoints (active namenode already has an up-to-date image from another% d! n; t4 w% C4 P* v9 v3 A# S) V% k
checkpointer), so we wait a multiplier of the dfs.namenode.checkpoint.check.period before
/ E4 q$ j6 Y0 N3 Q' F% a: c3 F" F retrying the checkpoint because another node likely is already managing the checkpoints,7 Q3 G2 q- m/ S+ D+ E8 J
allowing us to save bandwidth to transfer checkpoints that don't need to be used.2 |+ @# |0 n) m
</description>
) b! C1 F9 K3 h' G</property>
/ X& \0 d8 q1 w( T1 x8 v: l: f' f<property>0 X* z, B: h% T- }5 V$ z; I' X
<name>dfs.namenode.num.checkpoints.retained</name>3 w" G% v" C2 b: I& _, m1 h3 q
<value>2</value>6 T( H$ A3 V* M5 e# y0 r' [
<description>The number of image checkpoint files (fsimage_*) that will be retained by
( f4 V( O* J% ^% E4 F7 @ the NameNode and Secondary NameNode in their storage directories. All edit% M- J( p4 m; z* C$ t
logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
/ A. ~6 [: D: m1 t. c* N checkpoint will also be retained.
1 v# v2 D: N+ Q/ d5 G! L: W </description>8 n5 B( Y$ t& a0 L1 w) f A
</property>
6 h( G8 [/ c0 s; a, J<property>) j$ E- t9 {' k/ j/ x# T
<name>dfs.namenode.num.extra.edits.retained</name>
$ y* y& J! J @+ ?" X0 c <value>1000000</value>
! p4 n5 \" I+ d$ [4 h <description>The number of extra transactions which should be retained
2 x( @/ _* c+ e& F beyond what is minimally necessary for a NN restart." [7 s. y9 p9 m5 a( n
It does not translate directly to file's age, or the number of files kept,
* p6 \2 L) ^/ O* g/ D/ J+ { but to the number of transactions (here "edits" means transactions).! ^9 l p6 V% P1 q
One edit file may contain several transactions (edits). z# I% \, f# B8 N: e" _# ], m6 h
During checkpoint, NameNode will identify the total number of edits to retain as extra by
" R6 a3 l! V3 y( H checking the latest checkpoint transaction value, subtracted by the value of this property.2 q$ G; M1 }# N8 h
Then, it scans edits files to identify the older ones that don't include the computed range of
/ f6 y& \* l8 r: \% ?# M retained transactions that are to be kept around, and purges them subsequently.
. U Y* ^5 z$ C3 ^6 ? The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have: a/ O( |/ I$ \ A4 \
been offline for some time and need to have a longer backlog of retained1 i9 e5 S2 O' r2 p
edits in order to start again.
( s# ]5 u: y8 | @ Typically each edit is on the order of a few hundred bytes, so the default$ y- x2 G- Y, k) }6 ]
of 1 million edits should be on the order of hundreds of MBs or low GBs.
- K% q7 f$ k8 A- M3 h2 A NOTE: Fewer extra edits may be retained than value specified for this setting6 c" S6 o" Y! f9 W. M
if doing so would mean that more segments would be retained than the number `/ o8 ^' T0 ?+ h+ d0 O
configured by dfs.namenode.max.extra.edits.segments.retained.
+ P" ?3 J l: _; a2 B/ S0 }' R </description>
) E: |* `6 Z+ j6 B7 Q) j a( _4 w</property>
8 F" h. C0 z9 {1 R<property>
5 r, M# s& F' B4 M) i <name>dfs.namenode.max.extra.edits.segments.retained</name>
' G/ T# o+ Q) b1 ?" ] <value>10000</value>9 t1 \) R! }& @2 w
<description>The maximum number of extra edit log segments which should be retained
( ?' N1 s: k5 F5 A! k8 X& j& y) v beyond what is minimally necessary for a NN restart. When used in conjunction with& U; ~( |# e: x% n6 V$ Y
dfs.namenode.num.extra.edits.retained, this configuration property serves to cap& Q, K5 o t" p3 Q+ y: ?: k
the number of extra edits files to a reasonable value., u0 s$ b1 C1 M3 C
</description>
) g, Q) d" k( ]% m1 D6 @: I+ Y* E0 t</property>; z* Y! S6 L0 }% m+ Z* s
<property>
, D6 H1 ^" H7 M; j$ m7 m1 o <name>dfs.namenode.delegation.key.update-interval</name>
/ ?! A7 A0 g& N' `# _8 u. ` <value>86400000</value>
, K2 i$ Y4 Z0 Z- ?3 ?' x Y <description>The update interval for master key for delegation tokens $ a. S% D7 U* ^% N8 x
in the namenode in milliseconds.
' g' P2 V) q. W* }7 x </description>
7 q \$ j9 V; M# M</property>
% S4 Y* \) x/ N- C% i# C' q<property>
_2 Y* u5 c( d9 p h9 y7 u2 U. E/ G$ p <name>dfs.namenode.delegation.token.max-lifetime</name>
' @& X# y8 x; A* x# Y. W1 z <value>604800000</value>3 R4 [* @; r& G6 {
<description>The maximum lifetime in milliseconds for which a delegation
& _, R. g3 o g token is valid.
6 Z* W+ t0 p Z </description>, `; y6 ^8 k3 c6 b
</property>
' [8 }% Z5 x. P) [9 S5 ~# d<property>
/ a0 U' e1 g6 S2 ^' \6 D <name>dfs.namenode.delegation.token.renew-interval</name>
6 g* O3 P. `* B3 V <value>86400000</value>: p$ L' T1 q0 N9 y
<description>The renewal interval for delegation token in milliseconds./ Z/ V0 H) G: J& l/ s/ S" ~
</description>9 ^* ?" O: Y" P; s2 d4 w
</property>, i* U8 q* O( k
<property>5 D! o0 T3 P3 H
<name>dfs.datanode.failed.volumes.tolerated</name>
# E8 J/ ^/ m: r* t5 X$ ^9 q <value>0</value>! z6 k6 \8 l! l7 K) K0 T1 A
<description>The number of volumes that are allowed to
9 {* v* @- U5 M/ G- C3 E fail before a datanode stops offering service. By default) ~8 n% Y. x, }' _: A3 l
any volume failure will cause a datanode to shutdown.9 Z' j' ]4 Q1 H0 J: {+ P3 i
</description>5 A) C6 A3 e- v( ^* _) H# _ x
</property>
4 q' G7 M3 b8 _* I<property>
6 P7 v/ u$ U7 T <name>dfs.image.compress</name>6 x" Q" k [& Y7 K
<value>false</value>) k5 ]; @9 Z4 w" G$ e
<description>Should the dfs image be compressed?
9 D2 [4 P7 ]* W" ]% H. g/ d </description>+ J9 B3 `; m1 S& i. r6 h
</property>
, I9 B% ]# o$ H4 k ~<property>$ R2 ^; D: f$ O! J( @4 C
<name>dfs.image.compression.codec</name>
* ^ J+ R& p. x/ T5 f <value>org.apache.hadoop.io.compress.DefaultCodec</value>
" E( i3 V/ s/ B <description>If the dfs image is compressed, how should they be compressed?
h ?- y- }: W# m This has to be a codec defined in io.compression.codecs.6 A6 h- @7 \* b( @
</description>* J1 i/ l" O7 k
</property>- U. W: M) Z+ G: \( n b8 {
<property>
) _9 W' m7 P% \8 v* K <name>dfs.image.transfer.timeout</name>
" a/ s! r0 { W9 p( A <value>60000</value>0 l5 w$ Y) O. A9 i* e
<description>& }. w- F+ m7 U3 n) a( t. S! G
Socket timeout for the HttpURLConnection instance used in the image
# {2 D x4 Q- m; j# l X" h( ^ transfer. This is measured in milliseconds.
0 n1 o. k4 v( S4 R! c: d This timeout prevents client hangs if the connection is idle
, Q' Z& B0 ~( m- m8 A for this configured timeout, during image transfer.
5 B7 `. H3 a5 R9 T! d </description>
9 Y2 ~0 A7 |5 m; e. @4 W</property>
( p3 }3 L# Y; {: E" C/ s<property>
4 K- g$ F5 F1 P' a+ A+ v$ [ <name>dfs.image.transfer.bandwidthPerSec</name>
x1 q7 Y9 i b3 ?# M% y9 n0 a <value>0</value>
# j. n! O: g- ]3 Y# R% s; [ <description>
, c' ]4 W) A" X# I: [8 v" }" } Maximum bandwidth used for regular image transfers (instead of6 z- i- Z& E) f( I. U4 M# Q1 s
bootstrapping the standby namenode), in bytes per second.1 a* R7 M; `* Y8 A0 j* S
This can help keep normal namenode operations responsive during
/ K; y: D$ @* B8 `# H checkpointing.* j) ]6 o1 M) w/ J; t" `! L: I2 a
A default value of 0 indicates that throttling is disabled.
% K: ~* z7 M- q' d% T9 K0 Y) d The maximum bandwidth used for bootstrapping standby namenode is
1 i6 S4 Y$ h( v8 O) G" _ configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.5 s2 f; \ v4 I" b
</description>$ S" {: n9 _0 |& L
</property>! h2 Q# ^0 k9 \" i
<property>
& Y8 q( O ?- o) O6 ] <name>dfs.image.transfer-bootstrap-standby.bandwidthPerSec</name>$ q' r5 \3 Z6 p& G% W: X( ?3 q
<value>0</value>8 X$ e6 [# x2 c( H" D
<description>3 {) [1 T# k( y- g z/ [
Maximum bandwidth used for transferring image to bootstrap standby
0 m# X8 J/ d- K" o namenode, in bytes per second.4 f7 @# L9 K7 ^4 X9 b! S# E
A default value of 0 indicates that throttling is disabled. This default, a/ u" U& x* x* l" i
value should be used in most cases, to ensure timely HA operations.
m$ i; h" s1 u# G The maximum bandwidth used for regular image transfers is configured
o9 x8 d& O7 W0 C# W. x with dfs.image.transfer.bandwidthPerSec.. h# n; l7 b! n. `, i* c( f S
</description>
8 O' Q) }0 R- A# `2 A% S7 d </property>, d5 F8 n! f0 d% `$ @ t
<property>9 }4 U( I O- U" h; j {' P: x: i
<name>dfs.image.transfer.chunksize</name>2 z! }) {0 s( s# ?+ T: ~
<value>65536</value>
s$ l: b4 @, `" C, d <description>8 _9 v# z/ y! {$ R
Chunksize in bytes to upload the checkpoint.7 o4 w% @& ~% s6 u* h) _
Chunked streaming is used to avoid internal buffering of contents! y ], ^+ J) v) N) g7 f0 L
of image file of huge size.! E& i: f) S! J' I* N
</description>1 v; u$ o+ W' o5 K
</property>8 x8 G2 \. C# [% K4 x
<property>
8 V8 ]6 ]4 C6 Z& x' a# u <name>dfs.edit.log.transfer.timeout</name>
G7 @; C' P- n, {# {% y! c# J <value>30000</value>
9 I: I. n% l2 l: ~0 K$ C <description>
& _0 K# u. P& K6 Y9 h4 N) ? Socket timeout for edit log transfer in milliseconds. This timeout# Q; L- U. N1 Q/ j5 x
should be configured such that normal edit log transfer for journal
! e, t8 G% b6 U node syncing can complete successfully.) o' Y; k6 z; @/ H% ]# F
</description>! z! P: w! ^% r! m* M% W
</property>4 I3 z4 E* r% x+ N u9 p; C
<property>. D- Y8 Z4 x* l4 v5 G
<name>dfs.edit.log.transfer.bandwidthPerSec</name>
X/ B" l/ a$ @ <value>0</value>. q% _! v) P2 g' J8 Y; O1 B
<description>
5 F3 J; ~9 L4 t4 t1 P Maximum bandwidth used for transferring edit log to between journal nodes
! |- [7 d- ~* v" F _# Q9 F, D3 y for syncing, in bytes per second.9 B' D9 W/ O* q6 X1 d6 @/ z8 n
A default value of 0 indicates that throttling is disabled.9 w1 q. P/ c$ f
</description>
- ]/ T; D5 D. F W</property># k% `4 `/ ^* [: F2 Y% S
<property>
2 A3 m/ }; m1 G3 g6 Q: P9 Z' [ <name>dfs.namenode.support.allow.format</name>: W" k3 M1 z+ ~; J+ u1 h" g, {
<value>true</value>
5 k4 [+ Z1 F0 @1 X6 n <description>Does HDFS namenode allow itself to be formatted?) `: q4 u0 N9 {! @ o
You may consider setting this to false for any production" |! T. g( |6 _: E( A9 H' Z
cluster, to avoid any possibility of formatting a running DFS.
( g9 j: O# z# f$ `* v </description>
: _% i5 H6 s w. m</property>
+ Q. [. C+ {" \* r<property> D0 l6 o2 m) O/ E% V* Y4 g/ P
<name>dfs.datanode.max.transfer.threads</name>, i8 m4 A3 \/ m- d' u
<value>4096</value>3 ]$ [. Q+ l- ^ d$ z
<description>3 O, n5 z. z, h: e" ]( z# g
Specifies the maximum number of threads to use for transferring data
% n2 I V5 {- F in and out of the DN.
4 t8 W8 F1 e0 N9 V$ n </description>
# u c( {- w) i4 O- S9 \% w</property>
+ D5 `! N( X3 `' n8 ]. h ~<property>! Q& S" ^/ i% |, }% Q! p; S
<name>dfs.datanode.scan.period.hours</name>. K* G- P$ [% h3 }
<value>504</value>
1 q) C( ]* @" W* X9 p! S0 Z z% l <description>
! J Q& a% c4 e If this is positive, the DataNode will not scan any1 n, e6 M4 i" V) o3 M9 r# y
individual block more than once in the specified scan period.
$ R8 j$ t9 x* E$ \- V; b If this is negative, the block scanner is disabled.- S/ \& o+ E: X4 d% _& _* D( K9 D
If this is set to zero, then the default value of 504 hours. y6 p6 ?/ i- {) t& _% m, p
or 3 weeks is used. Prior versions of HDFS incorrectly documented
: e" U1 s9 Y" ^7 n/ N+ L that setting this key to zero will disable the block scanner.
& {; g( F4 ?* ] J& B& | </description>' e; {, n1 ~' z+ G; O& C
</property>
/ h+ B6 B; N8 |( G<property>
6 ]) y0 h+ z: u( b <name>dfs.block.scanner.volume.bytes.per.second</name>, M( h7 }# u' i) ?5 W! B
<value>1048576</value># }7 f) H8 |5 v# p
<description>( g: n" d" k* s+ s0 l4 g/ E
If this is 0, the DataNode's block scanner will be disabled. If this
q6 x7 [4 R9 M# E) ~ is positive, this is the number of bytes per second that the DataNode's N& v* m, h4 M. l7 x1 Y! `
block scanner will try to scan from each volume.( {4 ]9 @+ `4 c9 U. E
</description>
" ` p) u) }/ k* N</property>9 |$ b6 o0 V6 K2 @+ P7 V3 d
<property>
3 Q) g+ l' B$ k5 X* M, X7 p <name>dfs.datanode.readahead.bytes</name>8 ^% r5 q1 L9 `) C# q! w1 y& ]
<value>4194304</value>" G& @- l) L; q6 Y9 b( F9 b9 u# H$ H
<description>" d1 r* e" h- O& I- h8 z: X6 p
While reading block files, if the Hadoop native libraries are available,
: ]- K" p0 b3 s7 ^& J- K1 @3 I' ` the datanode can use the posix_fadvise system call to explicitly; l0 D+ J: B1 \$ o4 R# O4 f, e: `: p
page data into the operating system buffer cache ahead of the current' ^) U; t# F# O. \9 G. D8 b2 H
reader's position. This can improve performance especially when8 T! u/ e# q" g
disks are highly contended.* x3 Q: a& {( F, \1 }! n* M7 s
This configuration specifies the number of bytes ahead of the current
5 c& P' M. @. o& `! \ read position which the datanode will attempt to read ahead. This+ I$ v) } ?: n* e- ?2 p! y% w& b
feature may be disabled by configuring this property to 0.
* B l) {" V0 b; y7 Z2 r) U ^ If the native libraries are not available, this configuration has no* h) w- ]$ E% H/ d" H: i: ^' r+ G
effect.6 s2 r' n! \. P/ [8 H$ `
</description>
0 V! G- H) s$ F</property>
4 j: r, i4 v# r& c/ N0 y# u* X) ~<property>2 @$ h5 ^1 K* }, Y+ m
<name>dfs.datanode.drop.cache.behind.reads</name> q4 q, D+ Z+ V2 R. d6 V( A& \
<value>false</value>& `0 F; S6 i* Z# A
<description>
& a& Y1 W5 B5 e3 A& b3 g0 d- @- z In some workloads, the data read from HDFS is known to be significantly' |- u4 Y' H8 b+ R5 M0 p5 A4 F+ o
large enough that it is unlikely to be useful to cache it in the
$ \- Z. ?1 L. j+ s operating system buffer cache. In this case, the DataNode may be3 T4 s5 G/ f9 U7 b d! L4 @( u
configured to automatically purge all data from the buffer cache
5 i4 ?' y+ b, s8 H7 `- e after it is delivered to the client. This behavior is automatically
/ s, M3 K; h3 q! B$ a/ z disabled for workloads which read only short sections of a block: |7 b: B/ L# E. d3 O! X
(e.g HBase random-IO workloads).
0 J$ f0 \# x1 N/ ^4 x( V% |" s This may improve performance for some workloads by freeing buffer3 o# a: [7 d2 _! U+ p% W& K- a7 M
cache space usage for more cacheable data.
9 ]* `! `9 u* N If the Hadoop native libraries are not available, this configuration" T' D6 Z1 S. T0 ` J; U
has no effect.
0 u$ h% N% R$ B8 X# M </description>
' K# H( N' a: T9 F& T" B. d" P) m</property>- P3 S$ Y: F- H6 z( t5 ~
<property>5 t' Y$ v2 D4 K) b; ~- H
<name>dfs.datanode.drop.cache.behind.writes</name>( z" C- V+ J8 i+ w* [" c
<value>false</value>
B3 [% q7 d8 c. G6 @) o <description>1 @! k4 _9 w/ g! P7 X0 R8 R+ f2 d/ n
In some workloads, the data written to HDFS is known to be significantly3 j! {* Z* h5 b" }( w- ~& P
large enough that it is unlikely to be useful to cache it in the
; H0 M2 G; j5 d# i. W operating system buffer cache. In this case, the DataNode may be5 r9 ]+ ]. `. b4 o$ T/ Y+ t( L
configured to automatically purge all data from the buffer cache9 Q7 m; P; L& f3 K% `
after it is written to disk.
* e$ F ]: y6 k" P; Q4 {& v This may improve performance for some workloads by freeing buffer
. i( S$ Y6 D) W7 A7 i% E/ r cache space usage for more cacheable data.1 l+ w, Q3 K3 I" k3 i1 C
If the Hadoop native libraries are not available, this configuration, ?) s% t( A0 _5 q. V$ ^/ h
has no effect.
. j& A( g) j K( U </description>0 p, J" a: s0 T, C
</property>) ?* g0 l* Y5 l; v& j8 U
<property> \2 ~* c& l3 ]0 h
<name>dfs.datanode.sync.behind.writes</name>
. F+ n4 ~" i% G" c9 S <value>false</value>3 W+ ?" [/ f: X) T& f/ m, @
<description>
' t6 @$ A4 L$ O5 k% o) I& Q- n( T If this configuration is enabled, the datanode will instruct the
' C' {; u" `$ M# G operating system to enqueue all written data to the disk immediately
4 h8 s, C1 r; o$ ^8 I: [+ i0 ] after it is written. This differs from the usual OS policy which
B' v' F2 R) u$ s3 H may wait for up to 30 seconds before triggering writeback.
$ f. q: G7 g; ?; ? This may improve performance for some workloads by smoothing the
, c# L5 w, _3 B6 X; ^/ L IO profile for data written to disk.2 h6 ^5 X* _6 q* g! H4 @# \% V
If the Hadoop native libraries are not available, this configuration
! K k+ |* p; J, Z0 a has no effect.. |8 C# D5 r; P3 K
</description>, t7 Q9 e' @- X5 p8 h# v
</property>
2 _& N& v2 W( i7 F6 p<property>0 Z a4 I. F7 B) b' z7 G" z
<name>dfs.client.failover.max.attempts</name>
3 V w' W! Z' `. m* \8 H! A4 i <value>15</value>" B) }2 Z& ~7 ]7 t
<description>
7 S6 ]% A0 l* e! h/ ~, w Expert only. The number of client failover attempts that should be
4 O5 k; g4 N' w% Y$ S made before the failover is considered failed.
5 {0 ^3 e" |7 k- E7 ^8 g4 y </description>
& d+ B6 m' r$ z+ o</property>
% R( n- N4 ]5 n R<property># j3 j$ T. l3 _7 n. r+ N, Z" j3 {6 m" `
<name>dfs.client.failover.sleep.base.millis</name>
; M- b7 n$ _. _" |2 M: [ <value>500</value>
! j+ n' V5 @, d4 u- N2 b <description>% t9 X) u2 [7 s5 U, ^7 ~& Y
Expert only. The time to wait, in milliseconds, between failover4 G3 n3 ]* l6 j+ @; L- |" c" J7 {
attempts increases exponentially as a function of the number of* v/ o0 I4 F9 V C/ n
attempts made so far, with a random factor of +/- 50%. This option
~; I& T$ e- U! H! @ specifies the base value used in the failover calculation. The
1 w' d% v1 X. h4 z) p first failover will retry immediately. The 2nd failover attempt
0 T" w: H* K7 c9 K will delay at least dfs.client.failover.sleep.base.millis* ?! h. J6 `4 T/ I
milliseconds. And so on.
& ~: H5 C) a# r+ T+ A; W: ^ </description>' M( x9 h( `- Z6 {
</property>0 U3 k; Z* A8 V2 O9 V
<property>
" f( @( u U2 [: n <name>dfs.client.failover.sleep.max.millis</name>
% t+ d. s f4 H7 |4 C4 o <value>15000</value>
( N3 o" T: a. W% v <description>, Q& \8 f9 o3 R8 t
Expert only. The time to wait, in milliseconds, between failover' S+ e( ?+ W! W B9 P* X/ H- O' d
attempts increases exponentially as a function of the number of
3 g# o8 x# o0 g attempts made so far, with a random factor of +/- 50%. This option
6 q7 @, A4 t9 _: J$ z, U2 n specifies the maximum value to wait between failovers.
. S, e) p! _( W4 v3 P Specifically, the time between two failover attempts will not
1 D N' f4 o- Q4 I; @: j. q) O exceed +/- 50% of dfs.client.failover.sleep.max.millis. {* @4 b- ^2 ^9 s9 z+ w9 l! `, ~
milliseconds.
+ O/ d% V: X$ H- ] </description>
+ \7 j+ q$ ^' H</property>
2 o8 |% w4 f4 S% N. K9 K, S2 F<property>
2 [6 S% e# Y) N# s. g <name>dfs.client.failover.connection.retries</name>) U8 P! d: O& ?+ h
<value>0</value> m" X* }( z- ~. I9 x
<description>
- q% j4 Q, \2 Y, M Expert only. Indicates the number of retries a failover IPC client
4 @* X' ~; Y0 y' \( N* ` will make to establish a server connection.
* a' }" ~" P" V$ a: g, y% H; [3 u n </description>/ a; a. H1 _) k$ Z
</property>% t! r C. S5 M7 @' T% ?* q5 V* E* ~6 m
<property>
( ^# q1 u3 ^; b W* _: L <name>dfs.client.failover.connection.retries.on.timeouts</name>2 Q4 N9 E3 B$ q" Y2 X1 L
<value>0</value>
. L- w* M, }3 w& v" ?7 u <description>
! `+ V( o. C! i Expert only. The number of retry attempts a failover IPC client
- v6 g. | d5 N7 ~; G+ k will make on socket timeout when establishing a server connection.
* z, k+ }' y% G6 z U6 J1 \7 i </description>
8 T! B2 N. h7 V1 a: ]% r% d( ]</property>
$ _! T% z; O% F<property>
/ x! l* \8 P% P0 D# F$ B <name>dfs.client.datanode-restart.timeout</name>
. ~) p q9 B1 s4 A; Z4 j <value>30s</value>& H$ l$ D; C6 c& ^
<description>
" `; W. }& x2 q/ c6 f" l! D5 O/ s0 X- @ Expert only. The time to wait, in seconds, from reception of an
6 _; w( \3 M8 J0 I# z; K datanode shutdown notification for quick restart, until declaring
: ]$ _. v6 b! V the datanode dead and invoking the normal recovery mechanisms.# h# l+ U) ^5 I2 f2 E: X/ g
The notification is sent by a datanode when it is being shutdown
4 r8 I. t9 ?$ f* F. R using the shutdownDatanode admin command with the upgrade option.
l4 X) c1 q/ l! y: e# T Support multiple time unit suffix(case insensitive), as described ^8 U& ]3 ~ v8 C
in dfs.heartbeat.interval.. S( N. q0 h/ Z% z$ C o( L: {
</description>
5 O: p' d5 e8 o) K% T8 R2 Z/ B- f</property> ^6 O% O5 H: N$ [0 [
<property>
" e% X. C G& e( x1 F <name>dfs.nameservices</name>. M# c% b5 ]0 v: B, V9 y
<value></value>6 y! }6 l+ c6 p- e& X ?+ r
<description>- y2 J/ y8 J5 x+ C
Comma-separated list of nameservices.: T9 T/ s* Y0 d1 U' z
</description>$ ~: ~. m2 k( l7 _' F
</property>( N; A: r' e# e* u
<property>
V+ P5 e; R: q. f; L$ b <name>dfs.nameservice.id</name>- ]0 f- l, c4 [* Y, Y
<value></value>+ P w' L$ W6 m8 a, g" w
<description>& t; @% R2 D C1 I& e# w4 C
The ID of this nameservice. If the nameservice ID is not
& j' Y6 r3 H9 h$ O3 _ G1 M7 d configured or more than one nameservice is configured for
4 R) D+ [8 N' ~! l4 K+ A dfs.nameservices it is determined automatically by
7 D" S, ^/ V6 r: v( n8 K- j4 H matching the local node's address with the configured address.
6 T, V: o \9 X$ t/ l( F </description>
: E3 `5 ^% q+ B5 l& L+ {6 S, D</property>
+ S3 q9 n* }; \; g* p% Z6 }<property>
1 S" ?& K2 e2 o- w9 L, f <name>dfs.internal.nameservices</name>
# p* k1 |& J. X' A$ ^3 ~ T <value></value>- F; V: B# |5 Y/ V1 u! G
<description>
8 s% V# I/ y J# t, J) b: Y Comma-separated list of nameservices that belong to this cluster.
# p G& z; d, L0 _% h' ^0 j Datanode will report to all the nameservices in this list. By default, }- p0 ?& g0 G' C; T9 K1 j1 A
this is set to the value of dfs.nameservices.) Z/ _8 b1 {; r+ i" X
</description>& I5 A' n9 g$ m1 Y+ m0 z1 m% t
</property>0 a; |$ D* A% L8 X
<property>
# g1 Y2 l6 l. V) G3 a3 `! g <name>dfs.ha.namenodes.EXAMPLENAMESERVICE</name>3 r7 ?7 c% d7 y( a3 F% R( B! H; [
<value></value>
0 h* h4 a4 V7 g. P' D9 V <description>
, w) t( H/ C7 b; e) p/ h- L- f The prefix for a given nameservice, contains a comma-separated( K% W% w: q. D4 x; o
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).& i* d/ \& z+ f
Unique identifiers for each NameNode in the nameservice, delimited by/ n5 m, t7 T- D5 j# N) w$ n
commas. This will be used by DataNodes to determine all the NameNodes$ P, x2 O+ {" K" M
in the cluster. For example, if you used “mycluster” as the nameservice
1 P! [: l# i0 ]) ^9 G# ?8 y ID previously, and you wanted to use “nn1” and “nn2” as the individual
# D+ j+ t+ ]$ ? IDs of the NameNodes, you would configure a property4 L6 a+ f9 j2 p$ {
dfs.ha.namenodes.mycluster, and its value "nn1,nn2".
7 ` w1 t1 Q& _$ Y* l </description>
# _- g+ ]/ ~% Q; I</property>
: J4 v. @; g7 z4 l4 J% @<property>8 ?0 S. e" Q( r1 Y
<name>dfs.ha.namenode.id</name>
1 n) Y2 A1 b9 g4 P7 G <value></value>
3 b% V( w% H: p' c8 R7 y, ~: ~ <description>
: X' a6 v( s, ?; S) j# F0 Z# Y The ID of this namenode. If the namenode ID is not configured it' Y2 }( Y; e+ Y2 [: A5 }* B
is determined automatically by matching the local node's address3 V: A: X+ o5 j2 A$ s* b' C
with the configured address.2 n) i7 W8 c! B
</description>
- s* ] ^4 D/ z1 F- |</property>
9 _* u& K8 M: S4 ^5 |<property>2 @+ [' ]8 [# g
<name>dfs.ha.log-roll.period</name>
( L1 M1 k7 ], C' @ <value>120s</value>3 b0 T* R% q; Z& V6 ^% H. ]% e
<description>
2 p( b# @: s2 w3 W! \' |* n8 R How often, in seconds, the StandbyNode should ask the active to/ g- H' }/ J8 [' g7 A2 }; V
roll edit logs. Since the StandbyNode only reads from finalized
( h6 F8 q2 A2 N/ H! i+ ?7 d log segments, the StandbyNode will only be as up-to-date as how# v- m0 m' A7 K* R$ e6 c. @
often the logs are rolled. Note that failover triggers a log roll6 i, z( w+ P F" O# H" E' x, t
so the StandbyNode will be up to date before it becomes active.
# Y7 X, P; y, \3 @ [3 Y( c Support multiple time unit suffix(case insensitive), as described
/ m- {* b0 i4 }- v1 d in dfs.heartbeat.interval.; V5 k; X* X: S$ V! ?9 U+ u
</description> d$ b) Q3 E6 b1 D+ c' E9 C) o7 r7 d
</property>5 [; F; q1 M6 i; [2 ]$ y/ J$ e
<property>. X: J' \, k0 ]2 P
<name>dfs.ha.tail-edits.period</name>
# ]! F1 w9 [4 m8 K: q& ]# j <value>60s</value>
f! M$ N7 F' K& j9 b <description>
& H( g, U: _9 o3 j7 e; w, O* V$ Z How often, in seconds, the StandbyNode should check for new- l- U8 { `' g
finalized log segments in the shared edits log.9 v8 X/ n1 B1 ^5 J: y% A+ i( {- c+ O
Support multiple time unit suffix(case insensitive), as described
+ d" G9 b; [9 [% w, V7 I* p in dfs.heartbeat.interval.
m6 J( Q' i7 r+ J6 m7 S! @! j8 W2 h </description>' }/ k" k; U: z' b
</property>1 j/ d$ M/ N+ l& s& @, Z
<property>& s) P: i6 P. A8 k& c
<name>dfs.ha.tail-edits.namenode-retries</name> n, L2 X1 }8 N( @! }/ Z
<value>3</value>3 A. q" w4 ]- X ]
<description>
2 A$ | U- A" b8 H0 \ Number of retries to use when contacting the namenode when tailing the log.+ A( J: r! A% W6 K/ ~& J9 C
</description>0 J. Z3 J6 z3 ~& |
</property>
: C; V3 ]/ V" h8 m<property>
; ]' q4 X! \6 V4 R" b: p <name>dfs.ha.tail-edits.rolledits.timeout</name>
4 B. ?( Z% r! y4 p x2 }4 J( z <value>60</value>
0 a# ]- E( f4 A/ I. k7 a9 M& x- e6 v <description>The timeout in seconds of calling rollEdits RPC on Active NN.( M4 ~& r1 Z9 K
</description>, Z% O2 `$ d( r1 p5 n6 l, J
</property>! f. f' r- a0 C- w8 a/ x6 V
<property>
; [8 Q3 c6 U) q& O <name>dfs.ha.automatic-failover.enabled</name>
7 b- p4 `' z. k) h# l9 h <value>false</value>
1 V* ~5 t: Q% {0 D% |- p! `4 a: e( Q <description>% P" ^, O- D4 v' p5 d. m8 ?! o7 a
Whether automatic failover is enabled. See the HDFS High$ V4 B3 g3 [ o" O% U" Z
Availability documentation for details on automatic HA1 w" Y; ^" u2 l
configuration.
\' W; S" f4 d4 r. n9 W </description>& P( B: W4 t" m& {6 f6 S, a* O" h" t0 l
</property>
1 Y! `! c T' |* p8 j, G<property>
+ v" Z$ b4 E$ {7 I+ H5 Q <name>dfs.client.use.datanode.hostname</name>
& R& p0 X, G! i0 |: @! a4 F <value>false</value>
# _- c$ G& C1 ]8 m+ E9 } <description>Whether clients should use datanode hostnames when/ ?' x/ S' A8 i% T
connecting to datanodes.) u# D% G: j& K4 N4 s1 g6 q
</description>$ _; J) D9 M, y
</property># V2 e0 Y% E( d0 w1 n3 V
<property>8 C& J# M, e8 X7 ? U
<name>dfs.datanode.use.datanode.hostname</name>) P O7 A" l6 }; R7 W4 `$ P+ U
<value>false</value>0 j( x3 P# Z9 |2 b& @/ \" ~
<description>Whether datanodes should use datanode hostnames when
; h) ?! E% _ q connecting to other datanodes for data transfer.: g5 z* d( i, {% x: e/ ~
</description>
6 I6 }2 F1 B. U, R# D</property>
+ o$ u% m7 e) H7 w @: g<property>' `- h0 { ]* a; u3 Z$ ~
<name>dfs.client.local.interfaces</name>+ i Y8 J8 ], ~0 k( _; W, e: w
<value></value>
2 F0 {, ]# q7 ?# I, @! d* {9 Z <description>A comma separated list of network interface names to use% }) g1 i/ y* K: _+ @
for data transfer between the client and datanodes. When creating
" A! ?- R9 ^) {7 d1 K7 x a connection to read from or write to a datanode, the client
) g: l+ T1 K) n- x chooses one of the specified interfaces at random and binds its
! u2 e' D, W8 ]: b" n7 x& z8 c2 J socket to the IP of that interface. Individual names may be9 \+ q/ t& g; o/ `
specified as either an interface name (eg "eth0"), a subinterface
; C* Z6 `& Y7 l. L- x2 A2 {; C name (eg "eth0:0"), or an IP address (which may be specified using
2 y8 j" ]2 t J# ~ CIDR notation to match a range of IPs).; q( r: X6 B3 r6 t4 a/ s# ]3 F
</description>4 Y1 x% d; t0 s3 v2 z
</property>
4 P* t8 I1 d2 j0 P# _ g<property>
7 t% w2 w; P7 f( L- t* _. a, M) b <name>dfs.datanode.shared.file.descriptor.paths</name>
! F& T& g/ J+ K% C <value>/dev/shm,/tmp</value>
' @- N5 h. |$ D4 t J. [ <description>8 |3 ^9 h: r, r0 ?
A comma-separated list of paths to use when creating file descriptors that- |% o! O# j6 O, x' N. C4 N' v; R9 y, C* R
will be shared between the DataNode and the DFSClient. Typically we use
, s: V0 o) `; T4 o /dev/shm, so that the file descriptors will not be written to disk.
. S( r* M; J }$ K* ^" v Systems that don't have /dev/shm will fall back to /tmp by default./ p- a9 ]: j: |) j7 r1 y
</description>
. f9 ]% r0 h0 h4 J) V) ^</property>
. n: a3 B# ]" R2 ^( w! D4 G<property>
) C, d6 f, _, w& B9 `" J4 V, R1 k8 } <name>dfs.short.circuit.shared.memory.watcher.interrupt.check.ms</name>( T6 a6 a9 p2 q4 a3 G6 t3 k
<value>60000</value>
( H/ K# g$ t# f <description> j d6 s$ F2 L% A$ ^, s3 G
The length of time in milliseconds that the short-circuit shared memory
5 U$ G# x5 F. }. B watcher will go between checking for java interruptions sent from other" I9 U* o( ?# i1 p; k% N5 ]
threads. This is provided mainly for unit tests.
* k. C1 N3 n( [5 X. p1 b </description>
% S; L, J. k4 `</property>
5 q* |' k* H* @& J s<property>
9 U0 y% n7 G# c) Y+ X+ F <name>dfs.namenode.kerberos.principal</name> Z# @5 [8 E% w7 c
<value></value>
+ s' G" O* F3 C! G! d <description>
3 j& V7 z. H5 G8 O The NameNode service principal. This is typically set to' ?! J& I% E4 `3 w, s
nn/_HOST@REALM.TLD. Each NameNode will substitute _HOST with its
# v3 T* y+ ~' l7 b own fully qualified hostname at startup. The _HOST placeholder) |5 f; E) ^5 M5 [* O
allows using the same configuration setting on both NameNodes9 K1 R' M. j4 w/ E% Q/ ? f
in an HA setup.
" x/ c( l C7 b5 H$ ^$ p6 q </description>4 t! b8 L' l' B
</property>
% }$ B: N2 U8 t9 M: P* v<property>1 Z! r, P5 }$ K' L |
<name>dfs.namenode.keytab.file</name># P+ B% P! ^, n. [* s9 J
<value></value>
3 y( W, Z9 ?6 ^$ K- }; w3 r3 A <description>. z' [; k& A- G7 U
The keytab file used by each NameNode daemon to login as its0 A" ?1 a+ Q: N' ^( v9 O2 C2 Y: a G& k
service principal. The principal name is configured with* ?' ^ ?; W1 P. |( _' p
dfs.namenode.kerberos.principal.
* I" D* o$ ~ a </description>
( `. A4 K0 @5 e: c. q</property>
7 D3 J7 P6 t( ^1 |3 ~ h<property>& j) p0 N/ W3 g" d
<name>dfs.datanode.kerberos.principal</name>6 I) t/ s$ F: D" J( O
<value></value>0 m! n; y1 E" m( n
<description>
# M1 H0 j% K* j- ~% l The DataNode service principal. This is typically set to' e/ ~! \5 J v9 F7 Z7 G( ^2 w
dn/_HOST@REALM.TLD. Each DataNode will substitute _HOST with its; f# ]3 m3 a( [# o. ~1 n* b
own fully qualified hostname at startup. The _HOST placeholder8 c4 a% k1 K& F# K+ c! b
allows using the same configuration setting on all DataNodes.1 {* I) z. `1 v. E/ w. ^9 ?
</description>; Q$ p. s9 v- a
</property>
4 ], x1 D$ |4 y9 f& D7 `<property>) h% u Y3 k' Y
<name>dfs.datanode.keytab.file</name>( t5 `2 X) W! `1 N, X$ q' p
<value></value>* u# z) X; M! g/ a) L4 d' \5 F
<description>
- Y. s7 p2 k- f2 K0 f+ C4 P5 I The keytab file used by each DataNode daemon to login as its
5 Q# K6 x2 F. @! c* i service principal. The principal name is configured with7 B% ?5 Y+ j" I1 m
dfs.datanode.kerberos.principal.# f [( O0 ?7 m# E+ c/ O
</description>
" S0 U) }, w& ]# [" J! ?</property>3 e2 b$ w& }1 w+ q2 V) z' d
<property>$ j- v4 f& ~- ]
<name>dfs.journalnode.kerberos.principal</name>; r0 L8 \8 _/ V2 d- t# i
<value></value>9 N" Y& Y8 `: O) t
<description>
# C6 Y# j/ V. C N The JournalNode service principal. This is typically set to8 `) F* I* }0 a5 X& V3 ^
jn/_HOST@REALM.TLD. Each JournalNode will substitute _HOST with its" D6 `; k7 l5 h8 ?* e
own fully qualified hostname at startup. The _HOST placeholder
9 V9 Q; t5 C/ p7 q1 n4 T allows using the same configuration setting on all JournalNodes.
?) W. D% Z/ @2 y6 J" ]+ n </description>; V- m% X" [# i# s" u) I
</property>- F" [- ?5 b7 B- B2 u4 ^8 u0 F
<property>
2 C+ B( w4 \0 a8 q! D9 W <name>dfs.journalnode.keytab.file</name># }7 |5 L7 L$ ~3 B$ C2 w7 `. X# D
<value></value> l/ {% u1 h+ \$ ^
<description>
" g: c* l. k% T4 U The keytab file used by each JournalNode daemon to login as its: [& ^% M' E: y$ Q5 S& j
service principal. The principal name is configured with
O: l) ~4 }) q+ Z) H! e1 ] dfs.journalnode.kerberos.principal.
% Z) s5 Q) z7 K* p </description>* P0 c0 G: Z5 ~& F
</property>
2 }8 D5 \1 O0 [3 W; x+ R6 J' v<property>+ C8 z# Z( c% r2 \. _& @
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
9 p! s% T5 i1 k7 g0 @0 M1 i' } <value>${dfs.web.authentication.kerberos.principal}</value>
7 a9 o S4 Q3 E0 U <description>
) X- M# o$ Q! k) C The server principal used by the NameNode for web UI SPNEGO
$ J$ ]( ^" }" y4 _ authentication when Kerberos security is enabled. This is
) V" ^& f3 p3 c typically set to HTTP/_HOST@REALM.TLD The SPNEGO server principal9 b. C2 p9 w% e( s% R5 S6 X
begins with the prefix HTTP/ by convention.
5 e" X! C! P1 W) H- e8 Q2 o k If the value is '*', the web server will attempt to login with9 ?8 {* e' ~9 r9 k# T
every principal specified in the keytab file
: o$ ^! G( @7 n V } dfs.web.authentication.kerberos.keytab.
: e" @! u" p S7 R</description>
& P {3 F9 O1 T5 C0 X8 S) s/ d0 M</property>
" Q# T/ t, g: _; W<property>& m. _; K& a; Z- n$ n. v8 s; U
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>" ^" z6 i3 K' G+ [$ j( m$ @$ ]& u
<value></value>; J7 w" \. _$ a
<description>* h3 m) D, k5 P, E4 O3 L) c
The server principal used by the JournalNode HTTP Server for
6 O9 m+ N8 J* a/ L, Z; f6 \ SPNEGO authentication when Kerberos security is enabled. This is
8 R! {+ |- g- C S typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal* G0 ~8 J% W& U
begins with the prefix HTTP/ by convention.4 F) d6 q* ` Q; K) M3 j& c# }
If the value is '*', the web server will attempt to login with
; q! N8 T! w+ P& e$ v: e1 k every principal specified in the keytab file6 ]9 E8 S' n0 T7 K E& O1 ]
dfs.web.authentication.kerberos.keytab.: {; c" p& N2 w
For most deployments this can be set to ${dfs.web.authentication.kerberos.principal}
% a1 g* l# P6 `6 _( U6 H, X, n i.e use the value of dfs.web.authentication.kerberos.principal.
) H# V" U, H/ l </description>4 S+ J8 e2 A( F( G( p
</property>
8 N7 p7 J- i3 _: ~<property>
, {" D y6 ]7 p% X <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>5 ~; A/ T$ j$ E" w
<value>${dfs.web.authentication.kerberos.principal}</value>
: E! A$ S K5 |6 r- ~ <description># d' }& N. m& {$ k8 U
The server principal used by the Secondary NameNode for web UI SPNEGO& K7 E1 M: H% p) f
authentication when Kerberos security is enabled. Like all other
: A) G; K* t8 v$ a' I; ]+ G Secondary NameNode settings, it is ignored in an HA setup." d7 z: l4 U" N& g. c
If the value is '*', the web server will attempt to login with8 J1 W" p1 k: ^) v
every principal specified in the keytab file
. x* d' m$ \8 n6 N! o dfs.web.authentication.kerberos.keytab.
/ u; J) G( f: q5 a, \- \/ U </description>
' s' }$ G; S2 l) s: f</property>% c- A1 o9 ?% y3 \
<property>" O! m. G: Z2 P. `5 i3 m* Q
<name>dfs.web.authentication.kerberos.principal</name>
! R" _, h; B5 ~/ a3 U" s <value></value> q6 ^4 I3 f) E% `6 g
<description>9 p( V2 {/ g' ?1 K h
The server principal used by the NameNode for WebHDFS SPNEGO
, R% c0 |. G2 b+ @ authentication.
: r5 L# s% V& j Required when WebHDFS and security are enabled. In most secure clusters this
9 {5 n- b% U0 A. o: F6 y setting is also used to specify the values for6 _3 o* Y/ s2 ?/ R5 `) _
dfs.namenode.kerberos.internal.spnego.principal and
3 B, c9 L% p# Q) o% D dfs.journalnode.kerberos.internal.spnego.principal.% {. B0 g ]6 I: T
</description>% N" a& L: z4 P: _ h, M
</property>
3 d8 m4 i0 D3 |<property>
# p p( e/ S* u$ x! c2 E <name>dfs.web.authentication.kerberos.keytab</name>
' {, \1 b$ J# _* d <value></value>
8 I4 j8 G; m7 a3 p! T H( g6 Y& ~* G <description>- p$ P4 u' w% q% A* ]( N& m0 u
The keytab file for the principal corresponding to
, x! T3 R9 h; c2 w. a0 q7 i9 D. ]# j4 f dfs.web.authentication.kerberos.principal.
/ s& `3 N* H$ y0 e& ?" e7 ^, D </description>
: H# d* r1 O6 V. `* e0 U</property>
# S/ z# _+ O9 n5 S9 v<property>! Q/ I/ y/ k4 b7 }8 N
<name>dfs.namenode.kerberos.principal.pattern</name>8 Y8 D# V- }* Q- b/ N: C# m+ I& r
<value>*</value>
; }" F% G8 E/ H) {2 C <description>
- i6 ?" u1 @* v1 z6 | A client-side RegEx that can be configured to control
2 E% y; C" T9 O" j; `+ ] allowed realms to authenticate with (useful in cross-realm env.)4 R2 h) ]" t/ M( K1 B
</description>
2 T* g9 {) l- N; ]7 d</property>( I a/ s1 w2 K. L
<property>
# ?* ?' A/ `& t: F* w# P <name>dfs.namenode.avoid.read.stale.datanode</name>% Q( I, k6 C2 E$ \
<value>false</value>
9 E9 X; V8 A+ w4 M+ H <description>
9 H7 q1 Z$ A0 {3 H, G Indicate whether or not to avoid reading from "stale" datanodes whose) v- r$ a6 I' A
heartbeat messages have not been received by the namenode f( e9 E: t) A. }0 r/ A/ |6 {
for more than a specified time interval. Stale datanodes will be; W! x( X0 h) m) O, J
moved to the end of the node list returned for reading. See- u& n, Y' m' Y2 x7 [ q
dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.
6 A9 _/ S5 T& d, N5 e$ T b* q7 V </description>2 V" F. Y2 |* [( H3 s
</property>: v L' s2 _( R1 V( s! f- w; n
<property>
2 d+ D( ]: v. C- O; d7 c$ C <name>dfs.namenode.avoid.write.stale.datanode</name>
3 ~) l3 x: o6 E& w& s9 ]6 R <value>false</value>
" q) j& f3 M) f. G( J2 [2 L <description>
8 r- a! M5 ~5 r1 i- {" c; u' m Indicate whether or not to avoid writing to "stale" datanodes whose 5 s8 B/ p0 K. l. | A" ?
heartbeat messages have not been received by the namenode
6 H" v9 C0 i7 c, Q4 e# J7 i for more than a specified time interval. Writes will avoid using 5 @1 C+ E- W1 t
stale datanodes unless more than a configured ratio ) T5 U A) Q& I- a6 ?9 u
(dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as
. u$ P' N- X2 N/ a8 E* E stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting
: G) H6 e- n& R/ J4 S( p2 T for reads.- k/ {% j* m5 Q- k; B
</description>2 g# B. x8 W* ~' G' S& q
</property>
, I& H+ ]# W2 E. j<property>
0 v" b( z5 F9 u& ~& r <name>dfs.namenode.stale.datanode.interval</name>. m0 V. E$ C& T- T
<value>30000</value>
- A% }! y; [1 f) g9 ~( S8 b1 a; X" [ <description>
: M% ^* k$ O; n Default time interval in milliseconds for marking a datanode as "stale",
1 h$ n) t' g! w& | i.e., if the namenode has not received heartbeat msg from a datanode for2 D' g0 }2 ], X$ [! D! x6 d
more than this time interval, the datanode will be marked and treated
: r2 ?2 d: K! n as "stale" by default. The stale interval cannot be too small since
, F- k/ X7 c, o2 A; F otherwise this may cause too frequent change of stale states. + W7 C, M6 n9 D; w8 _0 {+ `
We thus set a minimum stale interval value (the default value is 3 times
4 k8 ~2 @3 v# y of heartbeat interval) and guarantee that the stale interval cannot be less( v/ S6 W0 |) F% w5 W5 R3 c7 O. @
than the minimum value. A stale data node is avoided during lease/block
- Q" l9 b% \. h; I J, { recovery. It can be conditionally avoided for reads (see1 x3 {* p$ h+ E/ {. s7 C
dfs.namenode.avoid.read.stale.datanode) and for writes (see# {8 @3 {) l6 t* n/ E
dfs.namenode.avoid.write.stale.datanode).
( X$ s3 ], o8 N </description>4 B4 o- W) G/ x& E8 g' ~
</property>7 q q! r5 w3 c4 k
<property>
: O0 j2 | L* N$ k <name>dfs.namenode.write.stale.datanode.ratio</name>$ x( R0 {0 D; A: ~) F
<value>0.5f</value>5 b; p* ^" m2 c/ T( t! {, S+ F
<description>) C- @% N6 d% V; [/ _: V: p
When the ratio of number stale datanodes to total datanodes marked9 z0 @: Y) O. i. j# j) t; l2 q- ~
is greater than this ratio, stop avoiding writing to stale nodes so" I, I; J$ A6 y9 I7 |4 F
as to prevent causing hotspots.$ l$ k3 _2 A$ U2 k T0 K8 ^
</description>5 d7 E4 b: U3 \ T$ d+ p/ @
</property>$ d/ Q2 F N- `+ r6 h# U4 ]" l3 n7 o
<property>
% Z! B, F/ {9 \: s <name>dfs.namenode.invalidate.work.pct.per.iteration</name>7 M6 H, l* y: i; A% M$ m2 _
<value>0.32f</value>* r4 B, M b$ \6 J9 A
<description>2 F+ A! _* u1 I: Y1 @2 S' \
*Note*: Advanced property. Change with caution.- y" _% d8 Q" k, \+ x2 w
This determines the percentage amount of block' z, I+ ]6 a: _4 j' V" a
invalidations (deletes) to do over a single DN heartbeat
: J5 k W6 _6 Q deletion command. The final deletion count is determined by applying this
. x+ G$ a% Z2 Y9 U9 K3 u% o percentage to the number of live nodes in the system.( L6 v2 o6 j/ Q" L7 p z
The resultant number is the number of blocks from the deletion list1 F( F3 C" G6 A7 T- q) Y
chosen for proper invalidation over a single heartbeat of a single DN.
* w/ K+ `1 z& q Value should be a positive, non-zero percentage in float notation (X.Yf),0 Z. P3 C5 l5 H$ h, W* `3 s. b
with 1.0f meaning 100%.: n# H3 b2 E, L! n
</description>
+ p" P2 a! _! i& h3 y& f) K</property>
; D$ c& F4 U& W( C( H<property>
0 @' x0 U# H" ^0 j2 i) k: S* y$ S <name>dfs.namenode.replication.work.multiplier.per.iteration</name>0 n$ C6 x7 x0 y, _
<value>2</value>5 T0 m7 J: ^, |1 W
<description>9 k% ?2 P1 S8 v$ }. M
*Note*: Advanced property. Change with caution.
7 e3 L& j5 a5 T/ C+ B, U( Y This determines the total amount of block transfers to begin in; `! \8 z' j; `( o) _( x$ c0 G
parallel at a DN, for replication, when such a command list is being
: |. J+ f! r; t; n sent over a DN heartbeat by the NN. The actual number is obtained by& ~: ^. \: d3 b% K+ I) T5 c7 F6 Q, o
multiplying this multiplier with the total number of live nodes in the
2 `6 H& V* a0 R1 D, ]3 M. G( H+ E cluster. The result number is the number of blocks to begin transfers
; {9 z" S. [& |: y; T; e! d' v6 O1 O immediately for, per DN heartbeat. This number can be any positive,
# h% s2 d9 n5 P# c/ s! Y0 Z non-zero integer.
$ P3 X0 R# ~0 H7 D6 l# L9 i8 } </description>
) L* l3 G% O2 e- C" S7 n</property>, ?5 K2 R8 W9 p4 Z2 `
<property>
- [$ O' S7 n- W <name>nfs.server.port</name>
! k' ?* |& {, h7 Q <value>2049</value>
3 {% G, b2 U( F1 I2 R |( a <description>& N" }; n6 \5 D7 x
Specify the port number used by Hadoop NFS.
/ _# i3 G* Q5 i* v2 j </description>4 }( p' z2 B0 @# \# f
</property>1 z6 r& `3 {/ v5 f5 S# x
<property>
" T1 W; r0 \0 H# B0 l3 X <name>nfs.mountd.port</name>- P5 X# J, H' d1 m
<value>4242</value>
# J# v, ?. Z) O% ~) s* H$ U <description>* g5 S7 p" q2 R6 P
Specify the port number used by Hadoop mount daemon.( ~0 z$ z) Z. o* N. h c4 S p
</description>/ [9 g; u' {! I$ G
</property>
6 Y% f1 d1 h+ Z<property> 9 R% J( ?! \% M+ a
<name>nfs.dump.dir</name>
2 K" ?8 F& `' y/ r/ X) r <value>/tmp/.hdfs-nfs</value>
) A9 e5 h6 A; D% }/ Q <description>
; e$ W) @- t/ m( i5 F This directory is used to temporarily save out-of-order writes before. r F6 q* I( R$ C; O
writing to HDFS. For each file, the out-of-order writes are dumped after- d1 H$ M- D3 h- ]1 c
they are accumulated to exceed certain threshold (e.g., 1MB) in memory.
: u* ?) G' {, m! ]; Z' O* } One needs to make sure the directory has enough space.
: y( j2 S9 F1 j: Q5 M- t6 U. L </description> L- }& |5 l8 I! l* l
</property>
0 \+ h8 u6 N3 k# s4 I: R<property>
4 M: c. I9 `( Q, H0 v( ~ <name>nfs.rtmax</name>/ w, U$ B: ]8 [
<value>1048576</value>, t% y3 G5 Q& w, G
<description>This is the maximum size in bytes of a READ request u$ R2 m+ Y9 ^' y1 i& _0 t
supported by the NFS gateway. If you change this, make sure you
3 ]! R$ f. ?5 H# Q" P8 Q/ P also update the nfs mount's rsize(add rsize= # of bytes to the 5 d' `/ \1 m+ b& ]
mount directive).$ U7 ]. i4 o# ]; d! \% S
</description> e3 `8 m: x. ]5 O, R
</property>, n9 ^7 ?6 x) m
<property>
5 o+ H- v# Q/ Y; r) o, [ <name>nfs.wtmax</name>
: N( S' ^. x, |/ l' D0 E <value>1048576</value>/ h( _* }3 M; T3 h) M& `6 w
<description>This is the maximum size in bytes of a WRITE request; I, ^& D3 `; _) n' B
supported by the NFS gateway. If you change this, make sure you3 F( g3 @2 \% V* V7 l, H/ `
also update the nfs mount's wsize(add wsize= # of bytes to the
5 @* ^9 J8 G6 t8 z/ { mount directive).3 D: ~' N4 e: f
</description>
% a% ? g5 [/ T& _6 h6 v9 X</property>1 g8 u% D. q, j$ ~) o. \
<property>
D: ^ N- H3 Y+ } <name>nfs.keytab.file</name>
4 k+ ]# \% K5 M+ X <value></value>
* W* {1 f: s8 Q9 `; y/ E <description>
) J& R/ Q. Y% j( x *Note*: Advanced property. Change with caution.* F8 F! a; v1 @1 n# F
This is the path to the keytab file for the hdfs-nfs gateway. z. o: W& r/ j4 p& c
This is required when the cluster is kerberized.
& b5 `/ ^5 p2 A4 H# `. Z </description>
1 t& D+ K, P6 k5 @5 y) {! G4 _</property>
% g/ n1 U7 b% V) l# s<property>
: ~1 x* v0 m5 F, H <name>nfs.kerberos.principal</name>0 x2 F ]( r7 @, i. ]+ Z
<value></value>3 h1 F2 C# G: k; t8 z2 Q- Z
<description>- Y# z# f" D% l V% r( \
*Note*: Advanced property. Change with caution.
, Y: K' x5 U" j- k% {. G5 f This is the name of the kerberos principal. This is required when
, F1 W" V* ~/ @( L+ P the cluster is kerberized.It must be of this format:
; I+ U0 u% i* ?$ [ nfs-gateway-user/nfs-gateway-host@kerberos-realm
6 u3 A+ l5 W, N C" c </description>( w, p/ Y! k. P9 Z x
</property>4 b' S/ `/ S7 ^4 {2 _* K
<property>
$ F3 j# o4 H3 B6 h <name>nfs.allow.insecure.ports</name># _( R9 a" h6 R4 A. a
<value>true</value>* Z. b+ ~9 H8 I
<description>
% U, L" P) q/ l8 D. p) ` When set to false, client connections originating from unprivileged ports
, F) i5 b/ a) P- e" ?$ b (those above 1023) will be rejected. This is to ensure that clients% }+ M% ]: E" r. h* c) n' i
connecting to this NFS Gateway must have had root privilege on the machine
- o: g ?% b& @& _0 W where they're connecting from./ k: D+ B! |9 z' w
</description>& c+ S" D8 {$ v8 d7 y) Q
</property>
, l. H8 z& J: s0 i$ h4 B+ B# ^<property>
0 F$ A# Z. p. F8 @+ [4 p( a <name>hadoop.fuse.connection.timeout</name>
' S' i2 X1 J. w <value>300</value>/ }* s( U2 ^5 s; e+ ?
<description>( Y1 Z3 h k; n
The minimum number of seconds that we'll cache libhdfs connection objects6 [6 {/ a' k) Y1 D5 z I
in fuse_dfs. Lower values will result in lower memory consumption; higher
2 ~2 P& l' G6 |6 {1 T values may speed up access by avoiding the overhead of creating new& O1 F' I+ d/ q
connection objects.
H7 q- q2 ?' H* C0 Z( z4 l </description>
7 f( e) _8 I* Z( U" L</property>/ l0 ~1 W( w$ Y
<property>/ R+ Q, I; m6 X9 \$ Z
<name>hadoop.fuse.timer.period</name>
) \1 V3 M' Z1 q0 c7 w, S% U& c <value>5</value>
8 F4 _" n' ]; G& a <description>
5 }; ^: t# n! p, j! D The number of seconds between cache expiry checks in fuse_dfs. Lower values2 k6 P0 h R5 H' m L, S$ I
will result in fuse_dfs noticing changes to Kerberos ticket caches more1 @- P* V& i o2 ?& J4 ^6 [2 ~0 E
quickly.( @7 j- T4 a8 `( b) P
</description>$ A* ` d' O( g- ?9 d [8 k
</property>$ q8 s2 F8 [: n
<property>
! p9 P& Z* L& ~ <name>dfs.namenode.metrics.logger.period.seconds</name>
; h' j) V/ |, X( G' U <value>600</value>$ D! F0 u. v& W
<description>+ _* p, i4 \- q- o
This setting controls how frequently the NameNode logs its metrics. The+ p! t1 U/ E1 ^3 q
logging configuration must also define one or more appenders for' f" R1 H9 C! F8 @" y0 X) N
NameNodeMetricsLog for the metrics to be logged.
; C/ D5 b4 k: b N* u NameNode metrics logging is disabled if this value is set to zero or) O9 e. P" b" I( A2 c
less than zero.; ` N' T @4 c2 p$ L7 R
</description>
" F1 O) O/ B$ V6 c8 o) K, P2 C9 h- o</property>" t, y- w. m- k
<property>1 S. ?/ e* d ]7 x6 K
<name>dfs.datanode.metrics.logger.period.seconds</name>
- U' a9 G& l! y- w1 s# m: a4 s+ z( n <value>600</value>
J ^9 @8 H* @9 v* p9 Q <description># B( N1 { V: d4 z% Q
This setting controls how frequently the DataNode logs its metrics. The
' K" C6 z, {+ ?: r9 w, u( M" E logging configuration must also define one or more appenders for
6 z+ V5 ^4 o: J$ e) h- q DataNodeMetricsLog for the metrics to be logged.
* M4 y4 {6 C" m: ~ DataNode metrics logging is disabled if this value is set to zero or- [& X; g ?3 b( [3 f
less than zero., q' x( G+ D. a
</description>
3 L* T) b5 F$ C( k: W; l</property>
& J/ ~3 X3 v" y1 H' J" ?1 X! m. L0 W<property>
! z$ {$ C; [# s% e$ a! G4 b <name>dfs.metrics.percentiles.intervals</name>
, `5 u" k J0 M+ N4 k <value></value>% [9 a! p5 @1 t& {# {( {
<description>
- D5 o# K. c/ |7 ~2 R Comma-delimited set of integers denoting the desired rollover intervals % D4 t4 U' d9 B' y
(in seconds) for percentile latency metrics on the Namenode and Datanode.6 D* w7 B2 `% q& |# h: o8 R
By default, percentile latency metrics are disabled.
. l5 F6 R: [; f } </description>' r& x( G z* s' a+ h5 T
</property>+ Q- x0 T8 ^: L# n+ I' z: v
<property>5 o" X9 n) {# `
<name>dfs.datanode.peer.stats.enabled</name>2 B; i( x4 E! P0 F7 q3 I
<value>false</value>
! {' a3 A* H5 R7 f# }7 S% C <description>3 B% c6 H. F: B: a: i
A switch to turn on/off tracking DataNode peer statistics.
8 G& @$ o! S% g7 c </description>
/ W3 O. X, E; h. C( t* u</property>2 K6 I) C1 S' F9 z% P0 O3 ]; ^
<property>
2 M+ N2 b# s9 U <name>dfs.datanode.outliers.report.interval</name>
$ y2 n( k5 ]1 ^ <value>30m</value>
7 h' I- A+ K5 O+ r3 b3 F <description>$ B) @* `/ |4 f) j$ Q
This setting controls how frequently DataNodes will report their peer
: g3 q0 H* q) Y- C, g% w latencies to the NameNode via heartbeats. This setting supports/ x. V( x/ w. n# h, E) N- q/ {
multiple time unit suffixes as described in dfs.heartbeat.interval.
1 n. _9 Q, \9 {9 B! R If no suffix is specified then milliseconds is assumed.4 |3 j+ T+ t; y# ?
It is ignored if dfs.datanode.peer.stats.enabled is false.) @- }) Z# x/ }7 r! ?
</description>+ ], F' S" s6 A, D3 j% R
</property>
* L9 [! G( H1 X0 w+ w) S0 B<property>
! [* |7 y9 Y. [5 E7 ~ <name>dfs.datanode.fileio.profiling.sampling.percentage</name>
7 r( H" @: O' K <value>0</value> z% ~" |- I: F
<description>8 }* u& N3 }0 B" ^* ]8 A8 N
This setting controls the percentage of file I/O events which will be: m6 H9 ^: ]" x
profiled for DataNode disk statistics. The default value of 0 disables
( P8 f' O* L6 V" Z9 O disk statistics. Set to an integer value between 1 and 100 to enable disk
6 ~) B, i. u* z3 ~ statistics.5 T8 p/ O8 H! O
</description>
& f( N0 ^" e5 }2 M</property>7 F" C+ _1 E3 r c8 Y( C
<property>
9 S+ L4 ?1 ], W- Q5 }3 k: v! |* { <name>hadoop.user.group.metrics.percentiles.intervals</name>
/ ` v! N1 F# l, x% ? <value></value>& N" {9 x2 r/ D" Q" p
<description>
% T, d9 ]' ~: L+ H* W1 F9 ?5 r A comma-separated list of the granularity in seconds for the metrics
6 N/ O* j4 t9 G. H9 | which describe the 50/75/90/95/99th percentile latency for group resolution
i+ S% ?* L/ C: n in milliseconds.
2 {( ~" u# z) u b By default, percentile latency metrics are disabled." E: P$ R: s8 Y, ?" x
</description>
& i! N0 @# X2 p5 | M7 t</property>
& T2 ]" z! m4 W5 u3 @6 U6 h<property>
, e8 E9 X/ n7 Y* i <name>dfs.encrypt.data.transfer</name>/ j" X1 V' a' J3 N# x
<value>false</value>% P' ^* ~4 G* w3 c+ x0 i
<description>
" }2 g$ i$ L0 l Whether or not actual block data that is read/written from/to HDFS should
( @: k' Y9 M% I- C0 P2 l be encrypted on the wire. This only needs to be set on the NN and DNs,) x# ]4 F+ n- e
clients will deduce this automatically. It is possible to override this setting 3 V% M4 T! _& Z/ U
per connection by specifying custom logic via dfs.trustedchannel.resolver.class. # Y9 o" ]) H: |9 \$ [( Q# A
</description>0 Z; B" U6 [) U
</property>
1 Q8 h2 E$ Q2 X9 H<property>
8 d7 [# ~5 y$ L0 o' _2 X9 G2 v0 m <name>dfs.encrypt.data.transfer.algorithm</name>, \1 q" n, l8 r: F E
<value></value>- K; Y& `! O( D. b
<description>1 |- D) f4 D f# e% ?: ?6 t
This value may be set to either "3des" or "rc4". If nothing is set, then/ {; M, f4 D, ]9 k) q
the configured JCE default on the system is used (usually 3DES.) It is
) G: i! v) K4 `* m' j- r" x widely believed that 3DES is more cryptographically secure, but RC4 is v5 p' Z( r! m; B. |
substantially faster.$ c; S6 I L% X, x, R9 q7 R
Note that if AES is supported by both the client and server then this
# |- `9 C/ T0 d encryption algorithm will only be used to initially transfer keys for AES.
3 a! z; }& Z9 f: C5 s5 m (See dfs.encrypt.data.transfer.cipher.suites.)# G+ J8 ~1 [! d3 s" p4 m
</description>7 X/ n5 x* I2 j0 F( ?
</property>! ?# o# c: f( x
<property>
& E }4 ^( F. _8 p: { <name>dfs.encrypt.data.transfer.cipher.suites</name>( K/ @& g2 l/ A# U6 f
<value></value>$ }# n, k& x2 t& y, A. J; P
<description>& V! M- f) D# o2 h
This value may be either undefined or AES/CTR/NoPadding. If defined, then0 b9 [# N6 g% f! }
dfs.encrypt.data.transfer uses the specified cipher suite for data) M6 T0 O( y* i) q
encryption. If not defined, then only the algorithm specified in& X: u2 I, d0 l2 j6 w
dfs.encrypt.data.transfer.algorithm is used. By default, the property is) O( ^! h# S& l, D5 e6 d$ b5 O
not defined.1 i( z8 z' |+ B }) r
</description>
# m0 W( b# N: v' u5 K, x. c' \</property>8 {% l" Z! m: k! p5 ?
<property>
8 J. A' M% L4 R! X$ j+ @; q <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>
$ Y* g' r* H: w" Z) g <value>128</value>
4 P% F- [6 R3 e6 I! j8 Y <description>3 p# W" e) h& v( X7 G6 ~9 o$ H, f
The key bitlength negotiated by dfsclient and datanode for encryption.
8 W$ J! S; a% ~9 ?4 i2 Y% J This value may be set to either 128, 192 or 256.( C- n( J2 _/ j: Z. k* D
</description>
" g4 y3 ^% q. x# C+ l</property>
, k5 v) C5 ~) h9 \" A; i<property>1 t: h/ v5 s# o; C" C2 c% X% F
<name>dfs.trustedchannel.resolver.class</name>
+ _$ E1 m0 _! t; t) V, y( z <value></value>
8 A. @; T& ~. G ^ <description>9 x) h' T3 t7 i3 x7 x6 H0 j
TrustedChannelResolver is used to determine whether a channel / ^$ O+ q. c$ x }' Y
is trusted for plain data transfer. The TrustedChannelResolver is
1 ~' \: }4 g' J3 T! ?% _0 H invoked on both client and server side. If the resolver indicates
; k" W% t7 K2 [1 z that the channel is trusted, then the data transfer will not be ) Y) L; i5 \2 E( H3 ?! @* v
encrypted even if dfs.encrypt.data.transfer is set to true. The- u1 q! j$ d& m1 Z% B5 H
default implementation returns false indicating that the channel
& p/ C* c; V$ ?# L is not trusted.7 ?! k7 Y/ c6 x$ w
</description>
* M; b) ?; o* Z+ s</property>
9 {% { [. z* g& f; o<property> ?. [2 Z- l% T# F
<name>dfs.data.transfer.protection</name>; o, q8 `( m- M7 _
<value></value>
+ a# B8 d0 z3 r4 q: K$ h/ q <description>
z( g* x& G( u! O& s A comma-separated list of SASL protection values used for secured
& m3 L# \4 j* A8 u1 M* r2 \ connections to the DataNode when reading or writing block data. Possible$ O+ v& b5 X' u6 a( u2 ?1 Y$ [1 j7 W; Y. P
values are authentication, integrity and privacy. authentication means
! o- ?( z1 R6 X5 Y0 r: K authentication only and no integrity or privacy; integrity implies
" X: K5 Z4 r( T7 s authentication and integrity are enabled; and privacy implies all of8 d* q: \ b* o
authentication, integrity and privacy are enabled. If
. u2 d" Y) v/ m1 Y# x0 O dfs.encrypt.data.transfer is set to true, then it supersedes the setting for
+ U) f! X4 O8 l! u dfs.data.transfer.protection and enforces that all connections must use a
. O p, M7 x* e' [/ u6 | s specialized encrypted SASL handshake. This property is ignored for. g3 r' z6 i Q T. }' P
connections to a DataNode listening on a privileged port. In this case, it6 S5 U2 }' z5 n1 w: }% g/ D+ [
is assumed that the use of a privileged port establishes sufficient trust.
/ ?! g3 E* u- _( n" c </description>6 K2 N2 _7 M9 q& x. u! \
</property>" Q+ L0 o# g2 s
<property>$ d( L9 _! o- J5 n& W% p
<name>dfs.data.transfer.saslproperties.resolver.class</name>/ U; e. G1 L5 l# l( t6 F( @
<value></value>
" D g# V0 `8 D# T- ^! `5 D# O <description>
( N5 I1 ^' h; @( n9 k SaslPropertiesResolver used to resolve the QOP used for a connection to the0 y) B! t$ N4 }6 R% O5 `) O
DataNode when reading or writing block data. If not specified, the value of
3 f6 f+ E) \3 V1 o0 ? hadoop.security.saslproperties.resolver.class is used as the default value./ [+ s4 k& F. ?8 O. S+ @/ j7 b; p
</description>
# ~1 y# X" ]0 j0 D</property>7 h+ X) v3 h# k* D' o7 w# @2 i9 p; _ D
<property>9 D( R" S* I! }+ [6 G& F. Q( S9 z
<name>dfs.journalnode.rpc-address</name>
( n1 L3 F, x) H: S) Y8 J <value>0.0.0.0:8485</value>
; t- q( e. X2 s6 `6 B& P _5 G4 c <description>
1 b8 b: t" }& J" {: x6 n4 r The JournalNode RPC server address and port.
2 @4 @ Q. ^( t9 U- R; ~" J </description>
) K2 b; i5 j1 j# ^, l( i</property>
! ~. V2 y6 b& \<property>
% b L# y; a. G3 q <name>dfs.journalnode.rpc-bind-host</name>
1 h6 b7 y2 U Q/ c7 j+ \; y <value></value>
- r2 p7 g o/ Q% S% {8 R9 w1 d { <description>5 z" _- n+ B9 @0 {( [* n
The actual address the RPC server will bind to. If this optional address is' I1 q# S: U- p0 u
set, it overrides only the hostname portion of dfs.journalnode.rpc-address.
4 `0 s% y- }7 F' N This is useful for making the JournalNode listen on all interfaces by0 H; P3 x0 f( L/ U) ^
setting it to 0.0.0.0.
6 }3 y* `# t3 L1 `& \! Z </description>( ^0 n9 N1 M2 ^
</property>
- `3 P+ H! h2 l& z; v<property>
% o9 y; U& r# Y" \. ~8 f <name>dfs.journalnode.http-address</name>& }) n8 @( m3 B8 O- _4 B
<value>0.0.0.0:8480</value>
5 q) e6 B# c* ^! o+ Q <description>
. n7 B3 R% v! o3 n, ? The address and port the JournalNode HTTP server listens on.% e1 V# c5 X) X n4 D8 n& h- I% U
If the port is 0 then the server will start on a free port.1 v6 U% v$ G# M
</description>
/ F9 D+ n# r& {3 Z</property>
) v% F: M# [# ?, g0 N2 ^( B<property>" y7 B: v5 X. U7 }, O8 c) W
<name>dfs.journalnode.http-bind-host</name>
- r4 E" _, d& W5 H <value></value>6 b# D; r. M& ?( m! B7 x6 B
<description>
' R8 W1 w0 }; |) \/ I p The actual address the HTTP server will bind to. If this optional address5 w! |! a* B. j) J% u. _
is set, it overrides only the hostname portion of
5 n9 \) X/ C" h' H2 i, v dfs.journalnode.http-address. This is useful for making the JournalNode
2 Z' [6 R2 n. M5 ?4 z$ ~/ u7 B HTTP server listen on allinterfaces by setting it to 0.0.0.0.; K# @, F* G5 E, Q0 }3 r" ?9 \
</description>
! P& h2 b5 D$ N4 b* o+ j4 s1 d</property>0 j6 _; z5 V2 r, H
<property>$ Y- D. y$ j6 o' p; k! Q3 z- C
<name>dfs.journalnode.https-address</name>% H6 o" d8 A8 ~, \# V6 f. T" ^+ G( M
<value>0.0.0.0:8481</value>! _0 S8 q+ C% A, u: z
<description>* q& v6 C. {1 S! C6 B/ b; c
The address and port the JournalNode HTTPS server listens on.
5 O1 f# V8 U$ C If the port is 0 then the server will start on a free port.5 F/ W4 u# V5 W. O
</description>
' s4 e: F! a& a% M& [, G. G6 f/ t* W</property>6 t7 g0 n$ _- \) `! D# M
<property>
. l0 i* C8 o9 d4 U" z <name>dfs.journalnode.https-bind-host</name>5 a( d$ y5 a3 ?9 ~2 w7 M7 G- ^; c
<value></value>3 N' R, A; z" V, w& Z& Q
<description>
) S$ K T+ e' V" `: a The actual address the HTTP server will bind to. If this optional address
1 _# _1 R& Q' c is set, it overrides only the hostname portion of' V9 q/ E5 ?( g1 W Y
dfs.journalnode.https-address. This is useful for making the JournalNode
$ M- D8 o5 v! [% z" f8 C HTTP server listen on all interfaces by setting it to 0.0.0.0.
3 ^- W1 V8 {; Y! N* t </description>
5 Y1 z } ], x</property>
5 p% T+ s! l$ b" J' n<property>' K9 a; H, \: L' P8 ^
<name>dfs.namenode.audit.loggers</name>0 Q, d4 ]1 l% B6 V" _
<value>default</value>; \. D9 J# L% ?7 _5 h0 V+ \2 }" Q
<description>2 a4 c, q: Q' ]6 N6 @
List of classes implementing audit loggers that will receive audit events.% q$ `+ M$ o M: \
These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger.4 T1 _% j% L0 U4 J; k8 D4 K5 A
The special value "default" can be used to reference the default audit
. s) P8 v G. ~! X# S) V& N( X- G logger, which uses the configured log system. Installing custom audit loggers
2 G# a0 A5 {( }; Z8 u may affect the performance and stability of the NameNode. Refer to the custom
6 A* g9 b* ?* w, C8 K logger's documentation for more details.
W A& ?- e( \7 [5 _ </description>
; R. M- B8 Z/ C1 v2 w$ w8 {' i) l5 k</property>
# \& b# a; U$ B+ v/ a3 l<property>5 |$ B, f. ? Y$ k
<name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold</name>- J T4 _2 H8 z
<value>10737418240</value> <!-- 10 GB -->& V# w2 D% d$ {6 e A# g' s. A( h
<description>3 u6 q1 l7 J- ]" c L% W
Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to( {* H- a! b1 \' h k1 ?# [: h
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.6 v2 U0 x2 S- ^. {
This setting controls how much DN volumes are allowed to differ in terms of k. f5 c: {( G2 j- N, \* d, Y
bytes of free disk space before they are considered imbalanced. If the free9 i% e9 V6 L: g' p1 R. s" L0 t
space of all the volumes are within this range of each other, the volumes
1 \8 @6 o v9 u" ?% d+ ? will be considered balanced and block assignments will be done on a pure# O: ~$ ?* F6 H
round robin basis.
) ?4 _* n4 j$ `" U1 C </description>6 P+ J- A( O# T' ]" z" p8 F
</property>
% t6 ~: b( o8 p5 h Y% P: `) z0 u<property>
5 H2 Q! w9 f- i, r' O2 }& n <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction</name>
3 y: ? D8 {& F- V7 G <value>0.75f</value>
" a9 O1 q/ o5 H: X6 `5 f7 f <description>$ n8 n' c& d0 u/ Q( S# S4 S: t
Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to
0 b: L' u) T; {3 K0 x4 ]2 R org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.
n3 B0 H3 \. ?0 ~ This setting controls what percentage of new block allocations will be sent
' P8 D0 l+ G' {5 G% z$ V% |& [, S to volumes with more available disk space than others. This setting should9 C. B; J9 @" b8 x& j- E" a3 r% L
be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should
) u, n1 C+ p f; F9 b) y be no reason to prefer that volumes with less available disk space receive8 t( P7 m, V& C l
more block allocations.
' y' k$ M% Z v6 w/ p </description>
9 f5 O. \& y1 \& q; V8 B, y</property>! s! l. X# n- @' ^4 N
<property>
4 `+ r. \( F1 j1 u# M) N( s <name>dfs.namenode.edits.noeditlogchannelflush</name>
' f3 b$ s5 ?2 @/ S5 f0 c5 ] <value>false</value>
, L' e1 A. U9 a) T) |' R+ C* ]' G <description>* S3 c* [4 y1 \. D- U
Specifies whether to flush edit log file channel. When set, expensive- x) V/ o) s" J
FileChannel#force calls are skipped and synchronous disk writes are( B7 { c& f1 y* S3 f
enabled instead by opening the edit log file with RandomAccessFile("rws")% ]! `3 ^/ ~- N: i8 m5 H8 W2 @. ^
flags. This can significantly improve the performance of edit log writes; F1 y" i: j3 S. @3 W/ T+ ~% V8 h
on the Windows platform.
, {1 F) [1 K0 u7 V* T; T; e+ G! ] Note that the behavior of the "rws" flags is platform and hardware specific
# F; j! b* h; w' g% |+ u; e and might not provide the same level of guarantees as FileChannel#force.
" [' C, `3 H: Y, E For example, the write will skip the disk-cache on SAS and SCSI devices
/ s& v2 d( q7 \3 p d+ }( f while it might not on SATA devices. This is an expert level setting,9 R' q* a' k; }
change with caution.
, F I: o3 f K: F/ }& L8 ~ </description>
/ _% v4 _: B3 p# L</property>
0 ~. R" z+ A, O; Q<property>
& y, z; Q3 y. Q; Z. l% s <name>dfs.client.cache.drop.behind.writes</name>; u& {% `/ R, O% T& V
<value></value>
8 \' U0 |3 c+ P: G% t <description>6 ^/ E# E7 _; V0 ~ A4 w
Just like dfs.datanode.drop.cache.behind.writes, this setting causes the) j. M m$ j6 S2 g( S' c
page cache to be dropped behind HDFS writes, potentially freeing up more
$ J7 f, ?1 l9 G E7 q# O memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this
4 A8 E, H1 U, ~# j3 d is a client-side setting rather than a setting for the entire datanode.6 p) ^/ j5 i+ U4 I$ _
If present, this setting will override the DataNode default.
: p1 g! ~5 F9 R2 [ If the native libraries are not available to the DataNode, this
3 | ]1 Q4 P" C, s configuration has no effect.
0 x3 Q( \/ l, I1 h </description>/ X4 ` N& W0 _1 d8 Z9 Y
</property>6 ]# R0 q" }8 [2 f* i' }) c7 {. }
<property>
8 z R; t0 m" s9 E5 B6 \" O <name>dfs.client.cache.drop.behind.reads</name>
9 g) T3 J) Y9 Z4 }( g6 ^+ u <value></value>
$ p4 {# ^8 {" v! `& d <description>7 H. p; F( t( v9 ?8 R
Just like dfs.datanode.drop.cache.behind.reads, this setting causes the3 x+ r1 w$ W/ f- F+ e3 @/ l, _
page cache to be dropped behind HDFS reads, potentially freeing up more
: W6 s& ~" U6 P2 p memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this
' ~5 l9 v9 Y. I' l6 A! e2 \ is a client-side setting rather than a setting for the entire datanode. If* m Y k! g( E) n) a% u% H! [
present, this setting will override the DataNode default.
2 _% w# x: `( u* \3 V5 M" ?1 q If the native libraries are not available to the DataNode, this1 Z7 v, X/ i" b' |: B
configuration has no effect.
# m& \# q q! p6 A5 W: }/ G+ V </description>
& O, x* r5 n: N& k. y1 R% H</property>) C" A9 S8 m8 G( h. H
<property>
7 P* `4 W: t( F$ R0 Z <name>dfs.client.cache.readahead</name>
1 \$ s" ]( C: \6 W. O/ Z) J0 b+ L <value></value>; z; M2 t+ u: K( }7 F% t2 y
<description>
# P9 b3 \8 G$ P# C When using remote reads, this setting causes the datanode to5 l4 G7 ~- p F# b9 g$ e
read ahead in the block file using posix_fadvise, potentially decreasing1 S c* R& \( p1 y& S
I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side; K# _5 ~( A% B% s' R7 X( B, H
setting rather than a setting for the entire datanode. If present, this7 @' R3 c. O9 j9 o/ M
setting will override the DataNode default.
& K: L( V8 e$ q' Q- `" D% K When using local reads, this setting determines how much readahead we do in- ]& z/ G9 M c
BlockReaderLocal.
2 j5 x1 v7 D: a$ H) W0 b$ k7 ^8 x If the native libraries are not available to the DataNode, this7 U, t; V2 m2 l' |8 k9 I0 ?, W# a
configuration has no effect.& M6 y1 ]& d# j9 Z
</description>
* v r5 e$ S* a! p4 T4 L</property>( t, X) h( b! O: ]
<property>" Y; d0 Z: A* P5 {" i M
<name>dfs.client.server-defaults.validity.period.ms</name>
/ `/ f* k. T8 S. i <value>3600000</value> W3 T1 x9 m6 L
<description>& F$ b! D: u( V
The amount of milliseconds after which cached server defaults are updated.3 Z; W9 j; e% g2 `9 `
By default this parameter is set to 1 hour.8 h" @8 h# `# x. I; @& h! T; L
</description>
, ~- J1 [* {8 s</property>8 M, T. K) r' s. m3 q% S J
<property>& h9 n& k7 k7 c) [$ |$ s4 o" z
<name>dfs.namenode.enable.retrycache</name>
+ X F* X! p! f% p <value>true</value>) H& T- O9 O) N1 |( j4 C2 G6 Y
<description>: s2 g' J6 g' p" N- N( G" \
This enables the retry cache on the namenode. Namenode tracks for
4 Z R) T4 ?! a7 i5 t1 z% c+ ?! N non-idempotent requests the corresponding response. If a client retries the( G, y; P$ w: a
request, the response from the retry cache is sent. Such operations# F1 G3 }9 ~# S7 v
are tagged with annotation @AtMostOnce in namenode protocols. It is6 _3 h N0 K( O u1 ^! N" I# @
recommended that this flag be set to true. Setting it to false, will result
. U+ A" l. U* T3 C. E in clients getting failure responses to retried request. This flag must 4 K0 |2 B( z9 `1 w+ M3 H
be enabled in HA setup for transparent fail-overs.' @& I% v: E- ^5 S, O$ J5 s. z* ^
The entries in the cache have expiration time configurable9 z( h C; s: X) p0 `
using dfs.namenode.retrycache.expirytime.millis., N; _5 B& y/ d# R
</description>
1 s; V" I7 X0 V0 P" x- L3 M</property>
0 Y8 x1 Z" n6 P" Q+ c<property>+ R6 M+ o# h5 T* S) t8 m, g: _
<name>dfs.namenode.retrycache.expirytime.millis</name>
& {4 B5 T1 Q) u. G5 F <value>600000</value>& X/ a" v2 k) [) w
<description>% {% O( ]5 n( z: @( u5 }
The time for which retry cache entries are retained.! _5 P& g! {& U2 v% l% t( ]& m n5 j
</description>: O. Z3 N( r7 z& e; l+ ?1 g
</property>" m+ A- V, b, T8 x4 C2 o
<property>
b' }5 E i1 ]3 u3 l# D <name>dfs.namenode.retrycache.heap.percent</name>
^+ Q3 S0 [! Q$ E( o' j% T' W# x <value>0.03f</value>* S5 }8 y' B. \7 w9 e/ {1 f3 S$ `
<description>
4 }. d0 \1 R0 A! o; H& y This parameter configures the heap size allocated for retry cache" V* _0 V1 v# t" P* }" D
(excluding the response cached). This corresponds to approximately# F X0 H# S G i& a
4096 entries for every 64MB of namenode process java heap size.7 l* @' C2 s g, U3 F( _
Assuming retry cache entry expiration time (configured using
$ M8 A8 R2 S9 Q: {" d# ^ dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this/ @8 l1 e7 m, c; {4 B0 i+ @" ?
enables retry cache to support 7 operations per second sustained
# {) F% h; v" X4 a9 u# G9 x for 10 minutes. As the heap size is increased, the operation rate: S0 n$ ^3 u" L
linearly increases.9 r1 l3 l ^+ k2 [" q
</description>5 A0 X9 |- T. E) o, I" z
</property># U* S- z0 B+ ?
<property>
9 I* \: h4 r; q5 d# l" G! Y <name>dfs.client.mmap.enabled</name>- ~& ~8 J! i6 d0 b6 b2 ?
<value>true</value>
/ c3 k( M/ n0 f9 F1 `: v! A <description>9 T z/ k1 W4 h. @
If this is set to false, the client won't attempt to perform memory-mapped reads.) O$ x, s/ H' f' q
</description>5 T2 f" f L# i5 w* W s
</property>. \# J" T7 s0 d3 |) @
<property>
" f ^9 z0 O8 _+ ] <name>dfs.client.mmap.cache.size</name>
4 _) X0 y H' X, H' v5 C! o <value>256</value>! _0 f+ e2 B( C' Z- ^
<description>
0 o. z7 X# `& ^. J- r# i3 h$ v When zero-copy reads are used, the DFSClient keeps a cache of recently used
0 \: c9 N4 l: w" C% h8 F' l7 T memory mapped regions. This parameter controls the maximum number of
; J) S2 y; K! r r- A) ~- i" F9 @2 K entries that we will keep in that cache.* R6 S$ p0 H$ ~6 M0 Q
The larger this number is, the more file descriptors we will potentially
7 u* k; H+ t# w9 Q( w) [ use for memory-mapped files. mmaped files also use virtual address space.: T; w+ Q9 L) i2 ], E! ^
You may need to increase your ulimit virtual address space limits before
4 O1 `' C0 c! S0 y increasing the client mmap cache size.
3 C, m% U2 m# `- Y0 \" F5 j. t9 z Note that you can still do zero-copy reads when this size is set to 0.2 x0 e6 w/ W1 T/ |/ h
</description>
0 o: U* ? R: g: M$ K7 o</property>* U! y' B% h) \; B
<property>, D9 Z5 Q( g' o, X) P& N/ N! q
<name>dfs.client.mmap.cache.timeout.ms</name>7 o- L! T' d8 z A9 h* M7 d
<value>3600000</value>
8 ^5 w# P3 B" h# w- x/ e <description>
0 }% I; c4 d" O- [* }1 m* s The minimum length of time that we will keep an mmap entry in the cache. b) Q* z2 a) [5 `) }
between uses. If an entry is in the cache longer than this, and nobody
) Z- h+ o% M* ]- J uses it, it will be removed by a background thread.! P$ k' q/ A. ^( J
</description>( Z# X8 g. ~9 f9 B l d
</property>
0 q7 \$ W( _8 ^' P% r' ~ o<property>
! Z c A9 A' h2 B1 s3 V* I, e0 y <name>dfs.client.mmap.retry.timeout.ms</name># ` ^3 c4 T8 N5 j8 R- `$ v/ V
<value>300000</value>* ?3 z# H; ^) r6 E5 y. o
<description>
4 N3 h: C% W: r. e2 V2 w; R. T9 O The minimum amount of time that we will wait before retrying a failed mmap
5 u, P" ^; a6 `2 q operation.
7 h8 e$ X! @4 ^" }1 [ </description>
1 R) V: j; {; U3 X3 t/ z</property>
$ E" w5 y7 Y3 \2 a7 K6 r! G<property>/ \1 P$ @$ b1 G$ W' t6 g
<name>dfs.client.short.circuit.replica.stale.threshold.ms</name>. E$ G% e- E3 x
<value>1800000</value>& @1 F( m( k; o: @: B2 [
<description>
% [- H/ }7 {( I The maximum amount of time that we will consider a short-circuit replica to
# G8 U5 y7 A3 |; ` be valid, if there is no communication from the DataNode. After this time% ]7 U7 }8 p1 O9 v/ }5 F* b
has elapsed, we will re-fetch the short-circuit replica even if it is in, I: G( l& h0 i. s/ h, b+ N
the cache.
]% p# H* h0 A( c; e7 k </description>
- n" i# `) D. \</property>* Z+ v: M7 F2 V J$ p3 i/ r
<property>
) `" O/ d7 p1 ^0 N# W <name>dfs.namenode.path.based.cache.block.map.allocation.percent</name>
2 I( q0 [+ O2 z) b6 U% w3 }5 z <value>0.25</value>
3 w0 m* _! M8 d: z' A5 L/ | <description>
% k6 d. X& i" X, }$ H The percentage of the Java heap which we will allocate to the cached blocks; I0 v. Q" v& D0 L% Y4 K$ f/ ~
map. The cached blocks map is a hash map which uses chained hashing.
% A9 r& |/ u4 w# ^2 | U5 t" S) q( k Smaller maps may be accessed more slowly if the number of cached blocks is3 @. W: ^6 t3 h" [+ Z5 z& C
large; larger maps will consume more memory.
3 S, R; M5 y/ j6 z, v, M7 h# i </description>* ]. M1 K" n/ z8 k/ }3 r
</property>. y0 _' G/ N6 G+ r7 m L+ H7 M
<property>
. Z3 r, X; q- ^% t9 X, W <name>dfs.datanode.max.locked.memory</name>
0 D! \$ U( @, F1 w( d$ d# P <value>0</value>3 _" ]9 c9 A$ R5 h8 }! }8 A# p
<description>
0 T4 B$ m: R0 c The amount of memory in bytes to use for caching of block replicas in
0 ]' `5 c3 S! r2 u0 H' A. ^ memory on the datanode. The datanode's maximum locked memory soft ulimit( o! R# {" I3 u* Y, e
(RLIMIT_MEMLOCK) must be set to at least this value, else the datanode7 _+ h# l) R- q9 D
will abort on startup.
4 E$ O2 U& k o) Q' l7 Y+ W' D5 r By default, this parameter is set to 0, which disables in-memory caching.
1 d3 h! Q: |$ c* ~9 ]9 K. r4 c If the native libraries are not available to the DataNode, this
, E/ [ D' `/ W configuration has no effect.7 x9 l4 H0 J( n
</description>
# H/ |6 [1 G9 e8 F5 j N+ a</property>( x: e9 ?: E1 b: S' M# Y3 e* T
<property>
4 M0 ]; P( d& A7 h( @' R7 w <name>dfs.namenode.list.cache.directives.num.responses</name>/ c- d3 g: k" e1 N# o! b: A0 H+ w
<value>100</value># G8 z& @' D6 F3 P
<description>
3 K! o; I( I3 a This value controls the number of cache directives that the NameNode will. g0 P3 j v9 U3 \$ B+ _
send over the wire in response to a listDirectives RPC. Y. D& U! e+ B; {* b$ e
</description>+ l( [0 X2 g+ e D* N+ ]
</property>; r; |# _) ~, B3 G
<property>
+ G& t* f: t- j" U' | <name>dfs.namenode.list.cache.pools.num.responses</name>
9 V ~9 T' g+ F- H# |. y5 q) j: i <value>100</value>
/ o3 b: @. k' P& w R' z$ L9 W <description>
0 d7 Z) Z8 C/ {" M2 {/ o This value controls the number of cache pools that the NameNode will
7 r/ [! I6 S9 E, \ send over the wire in response to a listPools RPC.
, s L/ ^" b; f9 g+ w# Y% }- Q </description>+ T& z( j$ p/ o% i: H/ ], r
</property>) u7 w, }2 {) h r) p7 `' p
<property>
! P9 a* J; G* _0 D% N t; W <name>dfs.namenode.path.based.cache.refresh.interval.ms</name>- M; b! r$ e9 |* v
<value>30000</value>( Z3 N P& P2 `8 h/ F
<description>
6 L2 Z+ H( `2 ]; }" Q The amount of milliseconds between subsequent path cache rescans. Path
9 M) x( U3 I a9 R M$ y% q cache rescans are when we calculate which blocks should be cached, and on4 g. d% u b( K8 k
what datanodes.) [: K( s+ {/ S3 k
By default, this parameter is set to 30 seconds.) s6 D+ O9 s* ^( C2 {- R. c! {+ ~; H
</description>& T; k5 Y G8 I# z
</property>
4 r! h9 A7 ]' W<property>
. t7 v# o# N4 b) u- p <name>dfs.namenode.path.based.cache.retry.interval.ms</name>) d, C2 c. g' m R; Q
<value>30000</value>) J; |5 X2 e7 D" T. ^% K% @
<description>
7 C: ?9 @2 ?4 T* g: t2 R6 R When the NameNode needs to uncache something that is cached, or cache% L! d3 f/ }* P; ~, F4 e( p! K
something that is not cached, it must direct the DataNodes to do so by
. s5 S g/ r% q' a: D1 J5 N sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode
% q! g( c& g5 ^9 E* T. r; m* N8 N heartbeat. This parameter controls how frequently the NameNode will, a9 s- N: w) d
resend these commands., V; Z) E/ k4 s# H v
</description>& ^1 p* K/ Z9 k0 G6 x! j
</property>; I5 r6 g; a5 O5 Z8 V
<property>
& R) K/ G$ d$ @) k, D <name>dfs.datanode.fsdatasetcache.max.threads.per.volume</name>; F6 C7 `) v# j( a
<value>4</value>0 F4 l0 n" @& p$ G
<description>
% n7 V2 B. t, [ K The maximum number of threads per volume to use for caching new data2 P- Y! A _. |7 O
on the datanode. These threads consume both I/O and CPU. This can affect z" F/ J) x% @) b# `0 r: z
normal datanode operations.* v/ M, p" E- A+ `/ C' u( E1 }
</description>0 Z8 [5 C- w) b+ S0 ?
</property>
: @* h7 o/ V0 f$ A<property>2 d1 F% z! \4 v+ e/ v7 v
<name>dfs.cachereport.intervalMsec</name>
. A8 E* a1 C4 h: A2 w$ v& O <value>10000</value>
( j# b; s+ _3 b O <description>. {0 v/ i" N: j/ y
Determines cache reporting interval in milliseconds. After this amount of2 @1 ~& O3 ~% [) d4 G' E7 r3 T1 n
time, the DataNode sends a full report of its cache state to the NameNode.
, C4 O* u; \" c- @8 V3 D+ Z$ L+ U The NameNode uses the cache report to update its map of cached blocks to
6 |; U J! M; X6 ?0 r _ a DataNode locations.
8 d, Q3 k. ^- D) o$ O This configuration has no effect if in-memory caching has been disabled by
& I+ M" d0 Z" @/ S setting dfs.datanode.max.locked.memory to 0 (which is the default).
& z# n( I/ ^9 i+ y+ [' x; X If the native libraries are not available to the DataNode, this- E# J( M9 C8 _: u5 j) }+ R# {
configuration has no effect.' m. n# D; a" a) h
</description>
" U4 |% k/ J0 F" r</property>
) E9 A; q% x7 @<property>( @& d4 |% J6 P. e x; y; U
<name>dfs.namenode.edit.log.autoroll.multiplier.threshold</name>' H4 `( }' A6 Y
<value>2.0</value># @3 A) V) ]* {- A: \
<description>. P4 E5 L+ F0 E
Determines when an active namenode will roll its own edit log.4 v9 M! ?% d7 s0 e
The actual threshold (in number of edits) is determined by multiplying3 |' w, o/ k9 o V0 n, A
this value by dfs.namenode.checkpoint.txns.4 b4 e& P5 A, M& A7 B% m& Z# {- l9 E, B. [
This prevents extremely large edit files from accumulating on the active
8 d K/ R7 M8 v! t, h& W/ J! `( p3 i; Z namenode, which can cause timeouts during namenode startup and pose an
6 J* r, E9 w( W% y4 p administrative hassle. This behavior is intended as a failsafe for when
$ Z& S# a9 |, T5 ]; Y; s4 e the standby or secondary namenode fail to roll the edit log by the normal
7 [ j9 I* H3 q2 B/ l! T, n) h checkpoint threshold.
" F- O: I$ Z6 n7 A </description>
2 J" S, `( P9 x</property>
, r4 W; K L" k8 v, p/ f: M# p<property>& ^4 y8 ]) p2 T* i. K% `6 z% W
<name>dfs.namenode.edit.log.autoroll.check.interval.ms</name>
/ x4 F) K: X# j4 R: j <value>300000</value>: ^$ e8 d5 S. M
<description>
& N. f9 Y1 c$ I* o2 C& l4 p+ I9 ?7 X How often an active namenode will check if it needs to roll its edit log,& L( a9 F9 e9 i& y8 Y$ e4 t
in milliseconds.3 c3 H8 Z/ ~1 s9 z: C6 @2 C
</description>7 s- b% B7 w- `/ J
</property>
C$ R* p/ f, K; u# f a, I) O<property>
# L2 T, [" V6 ~6 G4 f" ^ <name>dfs.webhdfs.user.provider.user.pattern</name>
9 S% z0 I( s4 _) p6 y <value>^[A-Za-z_][A-Za-z0-9._-]*[$]?$</value>
m; t& q: N2 M0 l" j$ z; Z <description>
/ E6 V7 B( u4 q9 Z9 d Valid pattern for user and group names for webhdfs, it must be a valid java regex.. N2 J) `9 k# j9 _
</description>; W. U9 ]% k/ G9 w w
</property>
8 S! T7 L" N2 Z( O4 s7 f. n; U: R<property>
6 m6 |3 Y" E' R! E2 } <name>dfs.webhdfs.acl.provider.permission.pattern</name>
4 F- u6 ], w" ?" U" e. u( y <value>^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$</value>+ B7 j" \& q& l; Y2 ^7 M
<description>
) W. U, H# s! u8 ]6 [8 n% Q Valid pattern for user and group names in webhdfs acl operations, it must be a valid java regex.
3 G6 P7 n2 E4 e k' W8 @ </description>
8 ^* ]; J, R- h/ N5 I$ _ _+ u</property>
( v) O' g) Q0 d, l2 @) L i<property>- f: h9 Y7 L7 C
<name>dfs.webhdfs.socket.connect-timeout</name>
4 h- V8 f' L. ^) r2 y( Z6 v. w <value>60s</value>( n2 K1 s5 M) r$ h6 V4 s: ?3 i- ~
<description>
; z" l4 D+ {* n. t3 h: s Socket timeout for connecting to WebHDFS servers. This prevents a
% N% z3 ]; x. \ Q$ ?7 m; z% G WebHDFS client from hanging if the server hostname is" l" d8 b7 Y: [4 R* l- O) S: L
misconfigured, or the server does not response before the timeout2 m/ x/ K7 @5 r/ J h$ o
expires. Value is followed by a unit specifier: ns, us, ms, s, m,+ C3 q4 {4 t+ ?8 ~. p! k
h, d for nanoseconds, microseconds, milliseconds, seconds,
4 K! c$ [7 ?! \% l9 o; U minutes, hours, days respectively. Values should provide units,
) J: x2 `* \, `. x" E! R but milliseconds are assumed.- n j6 ]& N! p2 i
</description>
6 V! N# q7 _, U' `( ]" A; s</property>4 j) w6 U- R& L# x; l, A
<property>
/ T+ k0 G) F7 a: m4 E <name>dfs.webhdfs.socket.read-timeout</name>
# Y1 d# C F% M: b6 g: K <value>60s</value>7 F/ Q- U c( v; }$ [
<description>. G. ?% c" Z' M( Z/ |
Socket timeout for reading data from WebHDFS servers. This/ H7 _1 P9 Y; P) Q i" w+ V
prevents a WebHDFS client from hanging if the server stops sending1 P! T0 v. n$ e1 }
data. Value is followed by a unit specifier: ns, us, ms, s, m, h,. M1 {0 T, P8 m( a5 R6 f5 x
d for nanoseconds, microseconds, milliseconds, seconds, minutes,& d/ P c: h1 h/ @: ~* S) _
hours, days respectively. Values should provide units,
8 ]( q2 w; h5 g' H8 ?5 f3 s but milliseconds are assumed.
, g6 u) O6 Q3 E </description>
8 S" n5 z5 Z$ v4 k" i</property> y) Q+ _. h. f. C
<property>/ T5 o: y& `" `9 x+ P W
<name>dfs.client.context</name>0 S% }/ | u+ p+ U% {
<value>default</value>
7 I2 E& J1 ~/ [" Q" Y0 q- {1 ` <description>$ v! f! q( x8 L4 a4 q& Z9 h
The name of the DFSClient context that we should use. Clients that share
. G6 B* h8 `- F a context share a socket cache and short-circuit cache, among other things.* b/ D/ y9 x: e5 _
You should only change this if you don't want to share with another set of3 q8 v1 b3 r8 \) k3 D3 d
threads.7 m& K! x- e8 W% L" e$ \
</description>1 \' ]0 j5 b2 C% x) C' I# P" |+ i
</property>
/ h- ]6 P" F {$ [9 c. k; y<property>* i! {" L* A; ?# N0 Z, T. J4 l
<name>dfs.client.read.shortcircuit</name>4 H* H5 g G# X- t$ G
<value>false</value>
0 e( O8 q) ` A" N3 c( u <description>6 l7 A" w0 T5 G8 p7 E& `
This configuration parameter turns on short-circuit local reads.
2 X' D. X7 Q* @- }! ?" Y, Q3 T* V1 V </description>
& u& a( |: |8 {, w3 g4 Z, F</property>
' S/ x1 g5 T7 m+ h- s<property>
" g+ r6 }& R$ _0 z) R4 I <name>dfs.client.socket.send.buffer.size</name>
. s4 B+ F, ~! r) f# Z; C4 E) Z+ K <value>0</value>
% d5 ^9 ?4 R' N <description>6 j. M$ c/ r" l- y2 a
Socket send buffer size for a write pipeline in DFSClient side.; E1 I6 C$ e9 l6 X
This may affect TCP connection throughput.
; u) }/ j' V: g5 `$ O If it is set to zero or negative value,* _2 |' y' u( h) c# l) R& m
no buffer size will be set explicitly,3 g! {+ Z# n& I, F" v, `" D5 h! A
thus enable tcp auto-tuning on some system.
$ ?: J2 e: z. w8 Z The default value is 0.
4 M2 c$ w/ K8 x: p </description>
/ J, I% L( ]2 u</property>& K* z* k+ W2 {& |7 g% t3 a
<property>' H2 o) I; A* A7 F' c; i8 I
<name>dfs.domain.socket.path</name>' f) B6 i' |! P- M7 H+ U
<value></value>
) B! N0 ~$ x" k, F <description>
\& B" U1 h' `. Y Optional. This is a path to a UNIX domain socket that will be used for9 z6 I# r) a, v. X4 m
communication between the DataNode and local HDFS clients.
$ f, E# G8 M+ F I! k- L( X* v If the string "_PORT" is present in this path, it will be replaced by the' u0 [, b/ q) k' M3 }6 H8 e+ \' H
TCP port of the DataNode.8 C+ w8 K; ^! ?5 O/ f5 K5 K
</description>; q& `5 e5 S0 I' l
</property>
* I# X- u4 y9 ], K. v/ M+ ?<property>8 O) W% |! t) E. l4 C* }- f" A
<name>dfs.domain.socket.disable.interval.seconds</name>
: W6 o3 t( F, K6 D1 Q& u <value>600</value>
% Y# R) G8 K7 r, K) R: @ <description>
( r0 \ w, K( ]+ E The interval that a DataNode is disabled for future Short-Circuit Reads,
- O- p: `, P0 m; Y& Y4 E0 g after an error happens during a Short-Circuit Read. Setting this to 0 will- d% B4 j O2 P. ~8 f; q6 P; u
not disable Short-Circuit Reads at all after errors happen. Negative values9 X0 I% u2 V9 u! c/ j0 J
are invalid.
. Q2 S# G/ n ~ </description>
1 g, z! \; \5 o8 s. {, X: ^</property>
6 }( B, W# [7 y6 D<property>
& @5 @- W7 l! n) L2 C <name>dfs.client.read.shortcircuit.skip.checksum</name>
|5 D0 }5 Z/ y <value>false</value> H! t% Z) h4 \% b" C+ i1 b! f. B( j4 a
<description>6 ]5 ]' z( s) b# h% S1 [* d
If this configuration parameter is set,. H/ e& x# ?6 e" j( G
short-circuit local reads will skip checksums.
. D. i2 S: V. B* i) @+ }7 q; {; e This is normally not recommended,
/ U0 O8 O& B+ p3 K- n" m but it may be useful for special setups.* V) d! s4 X# h0 v. l. [
You might consider using this
6 p9 L. ?6 y: w% n) ^ if you are doing your own checksumming outside of HDFS.$ \2 G) ^, e) C- \8 j
</description>
: L/ h) }* R: m4 u2 e; ^' j4 B$ I</property>5 _9 ^: A" |, ]& l
<property>- \3 p" f: ^2 F
<name>dfs.client.read.shortcircuit.streams.cache.size</name>" F$ }% M: }8 _ T Y. R4 n! \
<value>256</value>
8 w; J" L! h% W, f& \. h9 Y; Y& U <description>6 k, c& E; w- Y# \' R+ Z' W2 ~& z
The DFSClient maintains a cache of recently opened file descriptors.
3 q8 {* s; J* r! A2 F2 O This parameter controls the maximum number of file descriptors in the cache.: g0 f! R; V! o
Setting this higher will use more file descriptors,
$ A1 z3 x4 q/ _* z3 E R# E but potentially provide better performance on workloads8 y1 {) B9 Y. Z( m- Q- w% J
involving lots of seeks.
) G" {! f2 x: I' g7 K </description>" P. |8 O2 W7 j+ [% G) x- O
</property>6 p( t+ C: h& U2 a% w+ S* d
<property>
( \. K( n0 O0 l3 I& x. L/ g/ l6 d <name>dfs.client.read.shortcircuit.streams.cache.expiry.ms</name>
. E/ R! m5 k$ N# p9 n <value>300000</value>
! ^: M# g2 K* t$ n* O$ ]& D <description>
* F+ `% m0 l& {# A This controls the minimum amount of time
H! j! ~! i) z2 h; |; E+ Q2 a file descriptors need to sit in the client cache context
# j1 o, o, V$ i8 P; ?; W before they can be closed for being inactive for too long.$ H; x- \ p) b9 ?& G- j
</description>6 a, n& G9 {5 g) U% r8 u% {
</property>8 ?/ [. ~! ~6 T5 J$ b* U/ O
<property>9 @5 W2 W$ h1 O
<name>dfs.datanode.shared.file.descriptor.paths</name>
% v3 K- B7 _+ ?/ m M ?. a <value>/dev/shm,/tmp</value>! R7 Z/ b7 N+ H- M8 p3 S! n% s
<description>
$ [# z. B6 t4 T" U3 w' L0 t/ a! \' [- { Comma separated paths to the directory on which* G1 a) m+ x$ A# S7 b
shared memory segments are created.: C$ a4 i$ k0 j3 M9 y
The client and the DataNode exchange information via$ N0 {5 W7 e4 j" M
this shared memory segment.
[7 }# t- a9 |3 g It tries paths in order until creation of shared memory segment succeeds.
( g: Y; p+ u6 T8 z </description>( r3 T4 I6 s0 s8 h1 c5 F
</property># O8 O; u% n7 E2 P; ~! P3 @
<property>
) P* ^; ~- e* m. Y, }( k \ <name>dfs.namenode.audit.log.debug.cmdlist</name>9 V/ V7 o5 d+ T& z
<value></value>! E% V- N" C9 R/ H/ F# ~
<description>
8 y$ c, E8 c2 I, K2 |/ D. E A comma separated list of NameNode commands that are written to the HDFS
8 F2 }5 ]# t) x4 z- d' l namenode audit log only if the audit log level is debug.
4 B9 \: J1 N9 |0 u1 i8 i5 | </description>3 b- r% F9 R5 `7 l3 o, s
</property>
- Y! z9 A) F+ j<property>5 G9 X v6 ^9 s4 t5 U% X8 u7 |
<name>dfs.client.use.legacy.blockreader.local</name>
4 r+ \ M5 I8 y/ v4 v$ J <value>false</value>
; m3 D& u) p4 ]2 ^! m <description>
- b R2 }1 q; y% P% y8 ` Legacy short-circuit reader implementation based on HDFS-2246 is used8 w1 M$ W B2 T& a+ a
if this configuration parameter is true., } \0 h, {" ^9 I, X
This is for the platforms other than Linux
i+ y7 ~4 v3 i% `# C9 }& X where the new implementation based on HDFS-347 is not available.: ~1 H- f; Y% o+ k
</description>0 \0 P+ k8 X: D( C) `7 C A8 v0 j# v
</property>9 y: L$ N2 P. k0 @# i r
<property>
5 G/ D1 t0 P! l, ?) `( H: Q <name>dfs.block.local-path-access.user</name>
. ^/ N, z/ L( ~3 {, T <value></value>! s4 G; B! B. T0 I: n
<description>
* y. O; T% ?7 W/ A1 h9 {! [ Comma separated list of the users allowed to open block files
5 A% u1 v; }. a2 G on legacy short-circuit local read.
; E- t8 ]. ^2 o2 I </description>0 A9 K6 b/ Y q4 g O; f: z
</property>
, Y3 q: y( B# `0 ?: U6 {* G<property>
% z1 I" }5 a( |# K: w! s% v <name>dfs.client.domain.socket.data.traffic</name>5 C& B3 R( C- r% I& r1 W
<value>false</value>0 F& _! V" B" X4 b; h3 |
<description>+ |% i$ E; j* H0 p4 X
This control whether we will try to pass normal data traffic/ h, E( m2 M9 Y: E
over UNIX domain socket rather than over TCP socket
# f0 |1 B! n. }5 D on node-local data transfer.2 d, ^$ S. m, W/ W* ]& g
This is currently experimental and turned off by default.- y; c6 P# L& U0 c8 A
</description>
( I! t0 S" q G2 O( I& g0 P* T5 t</property>
3 X( b' g0 V' q$ F<property>
O3 j2 q J5 J, V* F W' ?! r l <name>dfs.namenode.reject-unresolved-dn-topology-mapping</name>( H ~. b0 e' E
<value>false</value>
) s3 ~8 H0 L- v' R. D/ Z <description>
7 }6 ^: ^9 L7 n0 C/ i. K! _8 O If the value is set to true, then namenode will reject datanode " _& r; d9 J E6 D
registration if the topology mapping for a datanode is not resolved and
Y* Y$ q5 Y$ o/ O" i NULL is returned (script defined by net.topology.script.file.name fails " e+ S9 V( J/ B: w2 X& s% ~) s
to execute). Otherwise, datanode will be registered and the default rack 4 x# i8 b$ a1 R
will be assigned as the topology path. Topology paths are important for
, m2 u5 \& T: {: B+ X data resiliency, since they define fault domains. Thus it may be unwanted ! Q1 G9 ?+ Q/ M; F( M$ |1 Z6 I# l
behavior to allow datanode registration with the default rack if the
& P, u& ]0 V. m resolving topology failed.
3 P' d- `8 K. P. x* Q0 S </description>2 H3 b3 Y* @1 R9 X7 C1 L
</property>
1 }/ ^% s! C% d, E<property>
% O( A l0 H+ q. p( q/ w4 S/ M <name>dfs.namenode.xattrs.enabled</name>* j0 N$ n5 F* l8 q* Q% {
<value>true</value>
/ f' U: n {" H5 s+ f- ? <description>8 c% l( @4 ?+ p# o1 R' v6 H* z
Whether support for extended attributes is enabled on the NameNode.
# N$ S0 Y& y b* m8 P7 o+ P </description>
( C, N# _* e- t</property>7 g& W; H% Z) J
<property>
6 u, S* W7 j9 P1 u+ i <name>dfs.namenode.fs-limits.max-xattrs-per-inode</name>
: ] t; ]* C0 I/ @( }6 d <value>32</value>
$ q4 H2 p. u7 {- P <description>7 k/ F( y4 d. t) |
Maximum number of extended attributes per inode.
8 @" p6 C% m) |9 i+ ^1 q1 X </description>) B( m" \3 g, p5 f$ d: z) \7 Q, u/ q
</property>8 C) q z% x0 i0 s
<property>( b3 q* _6 u8 t( U* W2 |
<name>dfs.namenode.fs-limits.max-xattr-size</name>
% c2 @# c6 E6 Z <value>16384</value>
+ {( Q! ?' \0 d# f( v <description>
8 i- b6 R$ A) a# Z2 ~$ r The maximum combined size of the name and value of an extended attribute6 u$ j e D$ v0 b( T
in bytes. It should be larger than 0, and less than or equal to maximum0 e5 S2 I* ^0 g! X% V4 ?7 c( u
size hard limit which is 32768.4 l9 `7 v7 N' K6 S
</description>
3 p9 s6 g3 O0 n+ j</property>7 ~* Z6 k; r7 _' B+ P0 R$ B$ S' B, v: r
<property>
$ s m7 x6 K- S4 q <name>dfs.client.slow.io.warning.threshold.ms</name>
, ], [: k# }3 }2 }" a$ I, ~* {- T <value>30000</value>' y3 D- n3 u/ `& ]# @$ x
<description>The threshold in milliseconds at which we will log a slow
6 e0 D. \8 I! Q& H1 q io warning in a dfsclient. By default, this parameter is set to 30000 i- k# @$ @0 _! F# `
milliseconds (30 seconds)." M# b: Y: C0 N1 l
</description>0 {* | A. _7 H" d B% W. i
</property>
6 l/ o5 b* w6 b<property>
5 e3 ^" B# t/ V6 s! g- N5 a7 s2 H <name>dfs.datanode.slow.io.warning.threshold.ms</name>2 K5 d' N- q$ _* `
<value>300</value>& v; G/ V9 K# k) Z$ c/ Y
<description>The threshold in milliseconds at which we will log a slow
1 Q" \( R2 A$ g io warning in a datanode. By default, this parameter is set to 300
7 [6 k5 S) ]9 A9 S: Z! U' o milliseconds.9 ^5 c. ~7 d; p/ E4 ?) O
</description>) r7 R: p' f" b3 L
</property>% [& H2 b) P2 L
<property>
; l( ^* p* V. k/ n <name>dfs.namenode.lease-recheck-interval-ms</name>
2 r; v% r) k1 H0 g7 e0 i9 | <value>2000</value>
# u7 s0 A; O7 f+ ^9 \8 G <description>During the release of lease a lock is hold that make any+ ]2 H2 ~; X' N+ \# P6 S* f k
operations on the namenode stuck. In order to not block them during
' |7 A+ a0 \9 @% u0 x3 ^ a too long duration we stop releasing lease after this max lock limit.
& k2 f$ ?' X8 E- t) Y- _7 ~$ | </description>/ W& G4 Y7 c7 K; W) b; k
</property>. N6 K, d; B) `
<property>
* D8 q0 F* f Q3 S6 U2 K q( I; z <name>dfs.namenode.max-lock-hold-to-release-lease-ms</name>: i3 X: ?$ R8 P* {0 p; o' R
<value>25</value>& G% `; f5 W3 e P4 O7 A
<description>During the release of lease a lock is hold that make any/ Y2 _1 u% v9 A8 H
operations on the namenode stuck. In order to not block them during
3 E5 n1 p! U8 y+ Z" I: C( J a too long duration we stop releasing lease after this max lock limit.
9 ?: U |# ?. x& E: n </description>- H( a8 o. |! n7 p
</property>
8 `, I9 v$ t% `- ]" M) o<property>* P$ }: v0 S. H2 n! {
<name>dfs.namenode.write-lock-reporting-threshold-ms</name>- k) a' w: }) X+ x/ s9 d2 z# ~; s
<value>5000</value>' d3 E( w' H# g( G) }9 ?( c! d/ @
<description>When a write lock is held on the namenode for a long time,
5 }. n! A8 w% _4 s this will be logged as the lock is released. This sets how long the
! \! P# P' A) V lock must be held for logging to occur.
8 `( ?, _; r0 }2 C+ d! u6 a </description>: S* [$ g2 G: |& B: ^
</property>: a9 @6 H+ X2 \* {# Q1 L( G/ L8 S
<property>
9 V3 b/ w7 E9 A" n9 o <name>dfs.namenode.read-lock-reporting-threshold-ms</name>
8 `' R s7 M8 T) ^ <value>5000</value>! H h$ b# ~' S3 v% U) g
<description>When a read lock is held on the namenode for a long time,
0 Y- C6 W* M% [* f this will be logged as the lock is released. This sets how long the
3 f/ t! C1 c4 }6 }8 c+ ` lock must be held for logging to occur.
, P& n Z- T( s" f1 }: I% |; G </description>
; t. e7 R8 ?9 a. Y</property>- y8 ~$ S7 Z6 u3 ~
<property>0 P* z ~# R+ W% J9 X1 d/ e
<name>dfs.namenode.lock.detailed-metrics.enabled</name>
( @" D" |) R- O4 x, s7 u <value>false</value>
( z& D$ ~7 c1 k) X" \ <description>If true, the namenode will keep track of how long various. R" }# H; y- a& r& {( A
operations hold the Namesystem lock for and emit this as metrics. These
6 |* E$ m! ] l4 p metrics have names of the form FSN(Read|Write)LockNanosOperationName,4 G" g' C, [: ~& }7 g: U
where OperationName denotes the name of the operation that initiated the
- n; d' o0 `4 u lock hold (this will be OTHER for certain uncategorized operations) and
; O( L" b0 Y# D they export the hold time values in nanoseconds.
& \6 o1 @$ ^6 M8 v; G& Q7 z! r+ D </description>5 `. Z' \) i- a8 s
</property>
% N( W( L$ O# k% q: _, P<property>
, b& l' Q/ u6 i" A. X2 a <name>dfs.namenode.fslock.fair</name>
+ M: A/ P4 Q' J; o% e0 ] <value>true</value>7 f q& H, G" S. P: c. A/ c
<description>If this is true, the FS Namesystem lock will be used in Fair mode,$ H- s+ N5 [( l+ s& n
which will help to prevent writer threads from being starved, but can provide
6 n' @# h/ C% h) U5 m, T lower lock throughput. See java.util.concurrent.locks.ReentrantReadWriteLock. U5 ?2 b3 \* k: y& N
for more information on fair/non-fair locks.- D( ~6 X: E/ x
</description>
' G. v7 ]& Q( F( I/ k6 O- ~0 o G</property>/ S: |# V0 k2 N
<property>3 p0 C& w6 G3 B8 h6 u" u+ w) j$ ]
<name>dfs.namenode.startup.delay.block.deletion.sec</name>
9 `% {/ x8 _3 S' T1 ~% i <value>0</value>2 g1 ^: C2 u) ~
<description>The delay in seconds at which we will pause the blocks deletion) U; |& Y+ u0 E
after Namenode startup. By default it's disabled.
" {. E3 p! [! a: B In the case a directory has large number of directories and files are
9 }3 C+ }( b1 L# \3 I deleted, suggested delay is one hour to give the administrator enough time+ c6 V) ?+ ]! K6 B1 k, u
to notice large number of pending deletion blocks and take corrective
" {7 L1 X( B/ `7 M9 K8 t action./ O$ O1 k1 F, R9 D' Q% z3 s. o
</description>
5 f e2 m6 x9 J' F$ d</property>' C2 m9 u4 |) o5 l/ _
<property>3 e- O+ m- X( T4 O, D# Q+ k
<name>dfs.datanode.block.id.layout.upgrade.threads</name>
( J- P) U6 O# H2 G/ ]+ u <value>12</value>
+ C5 z' Q4 d$ N% f9 H8 B3 ^ <description>The number of threads to use when creating hard links from" g# i+ G! S. g/ Z9 O
current to previous blocks during upgrade of a DataNode to block ID-based
0 ]8 F" Y1 L1 s0 m. B. E block layout (see HDFS-6482 for details on the layout).</description>* {$ W4 V% Z! \
</property>
9 R2 E; B/ X' F! l, E<property>
: Q( V3 S& u# q <name>dfs.namenode.list.encryption.zones.num.responses</name>5 B+ k6 K; I! M7 w/ c
<value>100</value>2 W( z7 n5 r4 l$ l
<description>When listing encryption zones, the maximum number of zones
4 [1 O" L" ?; E' C/ L4 p that will be returned in a batch. Fetching the list incrementally in
( I2 U9 H3 e) ^6 D$ y2 ? batches improves namenode performance.
6 H8 _* q- m+ C1 ~0 O* i% Z </description>* c$ X, j, r. H8 ?/ |" Y
</property>
1 ^' [. H# H" N' y, V<property>, G. v7 }8 e5 o/ C4 N/ H( Q
<name>dfs.namenode.list.reencryption.status.num.responses</name>
: Z/ ~- T+ l! q' }' z& g8 M <value>100</value>3 `/ z, q5 v/ U6 `2 T
<description>When listing re-encryption status, the maximum number of zones; L% C/ {, S" l0 [
that will be returned in a batch. Fetching the list incrementally in
* a! F) x+ X- h+ ^) b2 K) \6 } batches improves namenode performance.
! V& q9 v4 [9 X </description>6 j, P2 e: H, p2 [
</property>- j! Q, h5 k! ^5 F
<property>
5 d; h! u, G( i. L <name>dfs.namenode.list.openfiles.num.responses</name>
+ _/ e5 R, r+ C. H5 J0 H& i <value>1000</value>: t( _ s3 _- ?
<description>
8 T% b2 l$ S% d8 R m7 D5 S When listing open files, the maximum number of open files that will be
2 ]$ }. s, l4 D$ N returned in a single batch. Fetching the list incrementally in batches6 [2 i0 y8 F/ P
improves namenode performance.
; q, |' y) n! ^7 m( V </description>7 y/ N! G; Y: @; G
</property>
1 M' j+ M: u/ I<property>' [3 i; a$ L& }
<name>dfs.namenode.edekcacheloader.interval.ms</name>
( d, I. _& l8 x- r# [ <value>1000</value>
4 q# b" X, J0 X+ C" z <description>When KeyProvider is configured, the interval time of warming
1 i& L8 n& ]; W m! L: m) W up edek cache on NN starts up / becomes active. All edeks will be loaded
z+ r {: C) o from KMS into provider cache. The edek cache loader will try to warm up the* o! U& x2 U( U" E* B. B" p) }
cache until succeed or NN leaves active state.
4 y$ F( o! Y( G) j' Q M </description>
" l* m1 S7 C B# r, Z+ a: J6 _</property>
; K+ O' C M7 Q: c9 \" _. y5 G& q<property> a8 e+ r" Q( z; X
<name>dfs.namenode.edekcacheloader.initial.delay.ms</name>
6 A3 G1 j1 q) I( M0 B <value>3000</value>! T* V- f* f3 K v O- [
<description>When KeyProvider is configured, the time delayed until the first! d4 s) n1 l7 c; D( P! R- p2 N
attempt to warm up edek cache on NN start up / become active.
1 Y/ W" ~; w% K, ^" V. U* P </description>
) n. I, o5 F3 d h' G3 {</property>
1 W- {1 |' g! f<property># p2 K& {/ I$ e
<name>dfs.namenode.reencrypt.sleep.interval</name>
" S' c9 n5 p' {. q <value>1m</value>7 G) Z# m) u/ ~' ^# v: V4 G
<description>Interval the re-encrypt EDEK thread sleeps in the main loop. The
V3 v! W) Z. C; [; `0 t interval accepts units. If none given, millisecond is assumed.* u6 E; y* R! _* g: f0 O
</description>* f3 f# ]- J5 w ~
</property>
& h" V3 A' m O4 w<property>
. N6 }) j: u$ M& k$ E6 W <name>dfs.namenode.reencrypt.batch.size</name> C' [7 Q. h; J3 M& a
<value>1000</value># `* w5 e8 x N) x8 U# u
<description>How many EDEKs should the re-encrypt thread process in one batch.
7 s* I0 H. l0 R+ q s; c& n </description>; @) A% P% }7 r0 e$ C" Q
</property>8 w& ^/ u+ y6 G- T, C
<property>' |' C2 p: h) M& e/ n
<name>dfs.namenode.reencrypt.throttle.limit.handler.ratio</name>
. F4 R0 i8 f; G; k# H- [ <value>1.0</value>
& {. [) b+ a& o- A" B) \1 ] <description>Throttling ratio for the re-encryption, indicating what fraction j& k2 t9 s# ^
of time should the re-encrypt handler thread work under NN read lock.
! l1 ]3 C4 a. z/ G# P Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are
. P7 p! J8 g6 x( v, A Z$ v i5 V invalid values and will fail NN startup.
) X- s& ?+ M3 H </description>
; L# A( R4 o# Q o; k+ [3 Z% F</property>
; u" w, O: ~- ^' v' t! J: X+ p<property>5 F' C9 O* Z' u& X d# a6 f. [
<name>dfs.namenode.reencrypt.throttle.limit.updater.ratio</name>
' `# q# H( }2 ^ <value>1.0</value>
5 e6 w q( D& n/ [/ {$ o% A& F0 ] <description>Throttling ratio for the re-encryption, indicating what fraction: N2 s0 r3 Q: E1 C4 E
of time should the re-encrypt updater thread work under NN write lock.
- U( e5 r' I& }# _" G4 V; { Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are% o3 h* O( Z; ?% C, o/ }
invalid values and will fail NN startup.- C0 a/ O8 ~( ^+ R7 s7 f9 B
</description>! @" L* ]( j+ \9 v9 C' r6 K: l
</property>
1 Y: l/ [+ g( S' h$ Z<property>' W) l6 V+ d( E. T. P9 N
<name>dfs.namenode.reencrypt.edek.threads</name>6 t( v: |; m( T& ~
<value>10</value>4 P; a# G" I5 c3 M+ r
<description>Maximum number of re-encrypt threads to contact the KMS7 N% h6 |3 N0 h& U! h
and re-encrypt the edeks.5 z! T$ U ^. {
</description>
. f2 s% ^/ {1 C' S# E7 _</property>
3 K E5 J& m. x8 z6 g: z<property>
' [5 g' ]9 U" v) a: d <name>dfs.namenode.inotify.max.events.per.rpc</name>4 e1 _$ I5 V3 n% Z5 Y
<value>1000</value>
0 V. V6 e0 K3 p/ }, @ <description>Maximum number of events that will be sent to an inotify client
0 }) w. E! R' z! T0 l in a single RPC response. The default value attempts to amortize away6 e8 p" I% Y, P& F/ J
the overhead for this RPC while avoiding huge memory requirements for the4 s0 B# s( a! y6 g H% K9 D
client and NameNode (1000 events should consume no more than 1 MB.)
: g! N( Z+ L9 W$ ` f% U6 z </description>9 B$ D- t% u' _- u5 u
</property>
' y) o9 p* D5 p<property>8 m2 U% g" \2 g6 q6 ]. W1 I3 O
<name>dfs.user.home.dir.prefix</name>& Z2 X; ]% F) c8 r! f# ]8 J
<value>/user</value>
. C* P2 L! }1 ?/ I0 U <description>The directory to prepend to user name to get the user's' [# y2 ~' k9 ~8 K0 m
home direcotry.% u/ t) s1 E% I8 ~
</description>
; a1 J3 W2 n7 x7 X& n2 W</property>7 F4 e. l. m6 U0 I. T
<property>
, v& L: H* T- L4 G d <name>dfs.datanode.cache.revocation.timeout.ms</name>
# Y* `. O% T; `6 w& |# K) ?9 }* r <value>900000</value>5 G/ `! r" e! T) N0 y1 i
<description>When the DFSClient reads from a block file which the DataNode is
, k+ T+ X1 x0 J0 K7 B' K( n caching, the DFSClient can skip verifying checksums. The DataNode will
) A) Y1 f1 N) V5 b. K8 s7 l keep the block file in cache until the client is done. If the client takes
* g" [8 ^- ?" c% K- X6 k9 a an unusually long time, though, the DataNode may need to evict the block& `" N- P# h) Q) ~2 }( w
file from the cache anyway. This value controls how long the DataNode will
$ ^/ ~% A8 d' k& e wait for the client to release a replica that it is reading without
+ C0 p- Q9 l- x# K. r2 {! A# @3 y checksums.% M8 n ^, Z h7 w+ h$ \9 o5 U
</description>) L; R1 M2 y! t5 U7 E- j# q# \
</property>- d! ?& R) Q. N* i$ w
<property>
: \9 Y3 h3 `( f" U <name>dfs.datanode.cache.revocation.polling.ms</name>
( ? `2 Y; h1 s* {2 ]# P6 ^ <value>500</value>
9 _$ k$ T( ^& N <description>How often the DataNode should poll to see if the clients have
/ ^8 z" d3 F9 ]! g7 c' } stopped using a replica that the DataNode wants to uncache.2 b. L( \7 i2 d* c5 ~
</description>- j& j# m/ s7 R k( p5 J
</property>
/ m( f- u1 u( Y3 Q/ d' j<property>( t; r9 N. C& E' ?, k3 z$ N- Q
<name>dfs.storage.policy.enabled</name>/ w, \0 Z' V. K# o' Q9 I
<value>true</value>
( R0 G4 ?: {5 N7 E& D8 K) u2 X7 @/ K5 S <description>
9 b' R" t+ Y) t( j' s# c# j% R/ X Allow users to change the storage policy on files and directories.0 k R- Q; l4 a: d8 A
</description>
+ }. P3 q: W1 M, y' V: h" O</property>1 {. Q4 K- F$ D! k
<property>
1 ^. V9 Q& v# c, Y0 @4 N <name>dfs.namenode.legacy-oiv-image.dir</name>
; {5 T# k+ i# n7 n <value></value>" ]* L. \6 p* f$ K
<description>Determines where to save the namespace in the old fsimage format
1 C1 u0 z s2 y+ H$ Q during checkpointing by standby NameNode or SecondaryNameNode. Users can* v5 R0 e7 L& y4 M# K; o; o
dump the contents of the old format fsimage by oiv_legacy command. If
4 a3 M& v1 f$ ]2 q. r a the value is not specified, old format fsimage will not be saved in a- l9 \7 S5 |9 A% x. O5 [0 `4 w
checkpoint.5 F% f( @) g: j* \& H
</description>0 `; Y0 p: Z" k+ }& |6 o2 o
</property>
; ?6 X/ v% e' u W<property>) E4 w: q# Y- a9 m) l
<name>dfs.namenode.top.enabled</name>0 i3 V- k+ u* |6 i9 w* t
<value>true</value>
: y' d; i% I# B! v$ Y <description>Enable nntop: reporting top users on namenode# z0 C4 J5 h4 v
</description>" h0 k" Q# x9 e$ s& A( T# n
</property>
9 [5 n% t( O8 e& N" w<property>( A% f0 o5 T6 G) {3 Z; v" E! p
<name>dfs.namenode.top.window.num.buckets</name>
9 j% n( W2 Q- c6 U <value>10</value>
: [% D2 x) W$ q/ Z. p9 P6 t <description>Number of buckets in the rolling window implementation of nntop
W) g- N3 \& f9 _ </description>
% x, s9 b5 V! [5 W% `1 U, m</property>
) v4 ]0 x" e9 B1 y( K: N<property>2 N' E; p! ~/ i
<name>dfs.namenode.top.num.users</name>
+ {5 P" j8 P: P$ F C( s <value>10</value>/ ]8 g& [) S, t/ Y4 F7 m$ [- O
<description>Number of top users returned by the top tool) D, i& v7 G# b3 \
</description>8 K5 V: B, w7 B: U
</property>' h j$ D! w. E- a. B
<property>
; R/ [# c9 @" m% v; {* E; c <name>dfs.namenode.top.windows.minutes</name>
5 _4 H. Z7 Z9 Q, C% _' X2 ^9 c: t& o, _ <value>1,5,25</value>' }, n8 T N$ V9 a) ~7 C
<description>comma separated list of nntop reporting periods in minutes
5 ]6 x9 F* N& F </description>+ ?7 A% U: {* I# t4 O) D( U5 |
</property>5 r; z+ B6 { z; p% c# c
<property>
6 o4 \/ m. K$ V* ^9 h% l <name>dfs.webhdfs.ugi.expire.after.access</name>
3 G) G" g& m! ~* W g% y <value>600000</value>+ e% d) K: U/ ?' I8 K8 e
<description>How long in milliseconds after the last access
% o. M3 c0 \/ s9 f7 H the cached UGI will expire. With 0, never expire.
( y4 P& ]1 t& w' N2 W! I7 U </description>3 \1 W. s! ~: y! u
</property>7 q) z# s) k! a0 ]/ c
<property>
' Q# q+ }' r" O4 Y! k <name>dfs.namenode.blocks.per.postponedblocks.rescan</name>
* I ^* z% |7 G0 G m8 C+ p <value>10000</value>
+ ]) h* o; P# Q& D <description>Number of blocks to rescan for each iteration of
$ J0 f7 q4 Z) A3 E& V- J+ L postponedMisreplicatedBlocks.. O: \" ^6 F7 K: p$ h
</description>
7 y& C; d) C$ F; ] h</property>1 e2 t. L, V7 X0 N
<property>
' ~& R l$ t* D+ S8 D2 B" ~7 m <name>dfs.datanode.block-pinning.enabled</name>, r" r% j; d: H" D3 W# b$ m
<value>false</value>, F% o$ V! v V) ~' {% ~
<description>Whether pin blocks on favored DataNode.</description>
. v0 _: W' K8 {: p</property>
' h1 r) f2 r% e1 {4 x# p: H<property>8 g1 G+ B: r, k8 `5 {/ j
<name>dfs.client.block.write.locateFollowingBlock.initial.delay.ms</name>
- w& w# p3 G q) z9 b6 {$ m <value>400</value>+ `) m+ K: E( i* d+ Q
<description>The initial delay (unit is ms) for locateFollowingBlock,: h* x7 g% U: Q& u. f
the delay time will increase exponentially(double) for each retry.
0 Y9 l: \3 H( q$ x% S </description>
" h/ B4 g! i% {1 ^8 t3 P( F</property>
5 g$ L8 l- j# h( U- F# b( d( M<property>
2 _4 n3 {8 k$ `& j' ` <name>dfs.ha.zkfc.nn.http.timeout.ms</name>, o5 K) n5 _8 ?( y/ T$ H
<value>20000</value>
6 D E" v1 R" k7 |: R5 @. d <description>
- v7 M) I5 g4 X The HTTP connection and read timeout value (unit is ms ) when DFS ZKFC9 P4 y+ v1 R9 k( u R+ R8 u6 Q- |
tries to get local NN thread dump after local NN becomes7 g! ^7 Z$ e! q) G9 A& X
SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY.9 v* G4 f6 J7 e/ q2 v$ X* d) `
If it is set to zero, DFS ZKFC won't get local NN thread dump.2 H& t+ K( O1 Z( @" W$ O
</description># ?4 i: M. e6 {* F* ]* u% O$ O
</property>' g& O9 U3 e2 f4 F& u, A( F
<property>
% j% l( X7 \4 L8 `* j <name>dfs.ha.tail-edits.in-progress</name>
/ k) |' _) S3 m1 ]/ k4 i) a <value>false</value>
P& F) S0 s; j( I4 k <description>4 ?7 p8 f i6 U; E4 @$ ?& \, P
Whether enable standby namenode to tail in-progress edit logs.
9 N! ^# o0 Y* A Clients might want to turn it on when they want Standby NN to have% f" G/ U1 g5 C5 q
more up-to-date data.
7 E" _5 D" J) r0 I g) I- d </description># U$ K* W$ X' ^: u( l9 e( ?
</property>
# f3 q( T( [+ `: k& B% c. M<property>
8 b4 A! T9 G+ ]2 _7 K <name>dfs.namenode.ec.system.default.policy</name>0 R) B, L9 R8 R+ F7 D
<value>RS-6-3-1024k</value>2 d: m7 _+ V6 J. c0 b
<description>The default erasure coding policy name will be used& v# v) A" {! O# \, p7 K, `
on the path if no policy name is passed.
- U5 ~% f6 i- b! X. e </description>
, l. z j) s5 F/ c- S+ q! ~0 ]# r: j7 @</property>
# C8 K4 E! x+ p9 |% i% N<property>
: G9 r' F H7 B/ x1 B1 I' A0 C <name>dfs.namenode.ec.policies.max.cellsize</name>
1 {. n D! w$ Y$ @ <value>4194304</value>
6 b0 X% s& W+ }3 \+ Z8 Z <description>The maximum cell size of erasure coding policy. Default is 4MB.
5 S$ B! a1 m- g8 H </description>: \0 h( y7 g- ~" d1 J
</property>/ ]: }" a% M! j, @; Q/ ]" H
<property>- m8 v/ o# R( A* u E
<name>dfs.datanode.ec.reconstruction.stripedread.timeout.millis</name>0 o9 q0 l! M% s3 r& j y
<value>5000</value> d& B2 @1 a3 V, N
<description>Datanode striped read timeout in milliseconds.
# f5 ?, U7 _5 V' G8 \5 T8 I/ q }9 | </description>
8 {5 k, P( F" g p7 N* [</property>
" s) d4 g! W$ F<property>
) D0 d+ U2 m; Q+ ^# { X* e/ l <name>dfs.datanode.ec.reconstruction.stripedread.buffer.size</name>
# ?! X& C, D1 e8 E9 c <value>65536</value>% x7 k9 t8 V7 W p9 G
<description>Datanode striped read buffer size.! e1 ?5 f% e! ]. i: @' F* n$ o& }
</description>$ Q! K3 a) _: X8 O
</property>
+ F) j ^! b0 a/ Z( x<property>
6 |' S/ |# ~* e <name>dfs.datanode.ec.reconstruction.threads</name>
) ^0 J3 v1 G" ^# e- j) o <value>8</value>
K; ]+ E- ?' F3 E1 L5 Q; W U <description>
( g4 I/ E2 v8 k4 D Number of threads used by the Datanode for background( Z2 D' m: q* c5 u2 p7 H. a
reconstruction work.
0 V; v4 a) Z0 p# i" h3 ` </description>
& M- }9 X: Q/ u3 t9 r- d- `( k6 E, T</property>
7 W7 `+ x3 k! h6 y4 U3 B. X<property>
% O6 _8 D+ X1 }' v* s <name>dfs.datanode.ec.reconstruction.xmits.weight</name>
* Z5 t4 R( d% t7 Z <value>0.5</value>
$ M% P L+ O5 N <description>' ]: {: `" p7 p# j0 m0 P% o
Datanode uses xmits weight to calculate the relative cost of EC recovery; ^8 X8 f; K. q: ]3 Y, j' z
tasks comparing to replicated block recovery, of which xmits is always 1.# K# o* ]9 e/ d( M. F% n
Namenode then uses xmits reported from datanode to throttle recovery tasks" C9 f' t! P. V
for EC and replicated blocks.- Q* F& a3 f! a
The xmits of an erasure coding recovery task is calculated as the maximum, x9 E' g6 M% w J$ d) U
value between the number of read streams and the number of write streams.( p0 V4 v) D' ^8 `( i9 a
</description>
) V J/ a3 `7 v</property>% p' R, O: T! A9 F* g" h
<property>5 p8 B. z( f& h8 {
<name>dfs.namenode.quota.init-threads</name>2 y/ p$ N, @3 [) N' {5 i. M' }
<value>4</value>
4 X, J2 S/ a6 O; e: ~( T; h* z% Q1 G <description>0 \ e' C! i( l4 X, K
The number of concurrent threads to be used in quota initialization. The
3 d6 e5 j& a1 D8 s c2 P" L speed of quota initialization also affects the namenode fail-over latency.
! L) T, A/ x5 ~5 V e6 N If the size of name space is big, try increasing this.
! Z) O, E& z0 k' Q </description>
2 L) b$ \' V7 s+ z* Q</property>
. i" j' m! O/ S; z1 c+ g2 n<property>
8 a; X$ `7 Y+ {; R( Q. ~7 p* j' r <name>dfs.datanode.transfer.socket.send.buffer.size</name>* |' m) N+ c! z2 F& n
<value>0</value>0 ~. v7 o, r1 |. G4 W' ^
<description>, ^8 b+ b% R* M# ?. X+ A* k
Socket send buffer size for DataXceiver (mirroring packets to downstream$ |# ~; Z" O* p7 K$ S# J/ ]. h: m
in pipeline). This may affect TCP connection throughput.
0 G0 a S7 q$ v8 w3 _ If it is set to zero or negative value, no buffer size will be set
. |9 r% M* f' I9 c. w/ K, B explicitly, thus enable tcp auto-tuning on some system.* E2 i8 D! r; A+ W; ]3 ^& K" F/ }+ R
The default value is 0.* k9 O% l# E' z$ y T& V
</description>
( `0 k& |8 M4 |* ~0 T9 X: P% N& Q4 R</property>) ^. Z4 W! }, S3 ]. n( v% x
<property>! w, l1 n# X; z' V- q# A4 n
<name>dfs.datanode.transfer.socket.recv.buffer.size</name>* b9 w& M* H6 V p; c" F# Y
<value>0</value>" N- l6 g& y- ^0 T8 ~$ f5 F9 u/ g
<description>
H% L6 H. d/ ^" H. s3 H$ C Socket receive buffer size for DataXceiver (receiving packets from client
+ j9 |/ `. p* K5 ` during block writing). This may affect TCP connection throughput.
, S: X2 Z5 w& t. _3 @* f If it is set to zero or negative value, no buffer size will be set( x6 I4 Q' r2 _9 E, d
explicitly, thus enable tcp auto-tuning on some system.
* d; k$ @5 z5 k, ~1 H The default value is 0.
- r7 k+ U, h3 u* E </description>
& e2 o) c+ [$ `1 t: q; E4 A) J</property>
4 E3 z' p$ W3 V& K<property>
' t6 v# u* K( [ j: U <name>dfs.namenode.upgrade.domain.factor</name>
: W% x) x) Z4 }& t( g4 A2 f <value>${dfs.replication}</value>
/ ~) F5 ^8 H" w6 B: ? <description>! [+ P6 c1 \& }! R7 P
This is valid only when block placement policy is set to V7 [$ m: I# z( g' z$ t# |; {
BlockPlacementPolicyWithUpgradeDomain. It defines the number of9 @7 G8 R7 H/ B2 M4 k& A, N
unique upgrade domains any block's replicas should have.
. K. _6 f9 I" F4 x b* M9 H" y When the number of replicas is less or equal to this value, the policy, h/ S+ X! h7 s; a+ L, o6 A( Z
ensures each replica has an unique upgrade domain. When the number of
1 S" W( N+ J% N7 { replicas is greater than this value, the policy ensures the number of& b5 i; J6 q7 x6 ?6 y! V
unique domains is at least this value." d) L+ p7 F2 H8 a% h
</description>
* ^$ O3 R0 R! x</property>9 g& g( E, D. |+ ?- d- L- L9 T
<property>, U6 a+ ^ k, }4 ^7 n% A% \
<name>dfs.ha.zkfc.port</name>1 N- v# N+ U9 X* v, E) `
<value>8019</value>$ W' w& i$ n, |& y6 f
<description>4 g B) Y- N5 e2 P2 V3 L6 ?
RPC port for Zookeeper Failover Controller.
" C7 d$ S' ?$ C </description>! @9 h, L8 u4 ~# C2 E- ?0 M5 J7 q
</property>+ W1 R: A+ J$ w
<property>$ t6 p" Q/ q9 H6 q
<name>dfs.datanode.bp-ready.timeout</name>% N" }* q+ h9 v
<value>20s</value>, h3 P* B* \* q/ _+ h% u' u
<description>0 V8 T* P; J4 U, {
The maximum wait time for datanode to be ready before failing the) X0 c0 r" X( Q. `2 t
received request. Setting this to 0 fails requests right away if the9 A0 P4 H/ E! s! T2 K) a! z/ d
datanode is not yet registered with the namenode. This wait time/ e' r; v6 i! U3 k: Z/ V
reduces initial request failures after datanode restart.; N8 K! g. k. y2 z' t
Support multiple time unit suffix(case insensitive), as described
! n1 U! `7 T, l# H9 x1 P in dfs.heartbeat.interval.4 w( N* v2 A6 o* d; y
</description>4 f6 ]1 A2 q) i! M( m2 y1 ]8 U
</property>
0 @+ L" l7 ]9 Q2 i# P: `+ q<property>1 U; j- X: H) Z8 t
<name>dfs.datanode.cached-dfsused.check.interval.ms</name># |9 c4 y" |" A4 h' G
<value>600000</value>$ E4 u" A" R" }3 }% u2 B
<description>9 ]* Y# G( W6 S$ o! I
The interval check time of loading DU_CACHE_FILE in each volume.8 i& u4 n/ {0 W/ [
When the cluster doing the rolling upgrade operations, it will
7 e/ x) j: s6 R. n$ c0 V1 g- H usually lead dfsUsed cache file of each volume expired and redo the
+ d1 M$ G" J$ b& B du operations in datanode and that makes datanode start slowly. Adjust
8 i. }4 |8 P9 Z% Q this property can make cache file be available for the time as you want.
5 A% y9 R& Z8 l" ^ A </description>
' s' z* s) l/ l& { s</property>: ~ l$ J" r0 Z$ ?' k0 Y1 P
<property>0 _" K7 H+ H1 }9 `. b8 _
<name>dfs.webhdfs.rest-csrf.enabled</name>9 I6 d. t, t5 J* |
<value>false</value>
2 t$ g0 P" i4 b9 u$ k% N2 h <description>
+ l9 k, B" B) B) H; V If true, then enables WebHDFS protection against cross-site request forgery
i* [8 k' w" v (CSRF). The WebHDFS client also uses this property to determine whether or: R" a" r& G! p% Y
not it needs to send the custom CSRF prevention header in its HTTP requests.
& x; @) v/ t& {* ~ </description>0 @* C: t; V# y0 u' H
</property>
2 O7 n! `4 X5 [( [<property>
/ v' [& o7 Q5 ^5 w/ X9 @$ r <name>dfs.webhdfs.rest-csrf.custom-header</name>
: K! _- K3 i( h X; A. Z <value>X-XSRF-HEADER</value>/ E) S& [3 t1 s0 l! l3 b
<description>, d+ O U/ i0 U5 r: w
The name of a custom header that HTTP requests must send when protection/ G$ F% W* y8 T/ w
against cross-site request forgery (CSRF) is enabled for WebHDFS by setting* V0 |$ _7 O) \3 f( ?. @
dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this
$ X4 E& O, Z% r4 i( Q) h2 H& _ property to determine whether or not it needs to send the custom CSRF7 l6 d H4 G g; }
prevention header in its HTTP requests., v: y1 m" U8 t- i$ ~0 Z5 D3 F
</description># |% {- ^3 C# U" c) k
</property>
" F! y6 { ^% ~$ P8 S* m* _ u% n<property>
' r9 {; Q) a, x8 J <name>dfs.webhdfs.rest-csrf.methods-to-ignore</name>
, y; o0 I3 l: r- ` <value>GET,OPTIONS,HEAD,TRACE</value>$ w- J) I; U {1 V' e
<description>! m8 N6 ]3 S6 J1 `
A comma-separated list of HTTP methods that do not require HTTP requests to
: M! z; d4 C h; y include a custom header when protection against cross-site request forgery
# w5 h: B% [% G: C$ Z9 S; @7 E( { (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to
) Q& k! |4 x) W, L- P; ] true. The WebHDFS client also uses this property to determine whether or
: e2 N- r! |( C6 n not it needs to send the custom CSRF prevention header in its HTTP requests.7 h2 S0 X7 O4 G5 Q
</description>( [- H" D: ]3 e U4 {* l
</property>
, m5 Z' Q) Y- y9 O9 p, l' m: R<property> `) q/ P) w# Y5 Q" E' J" b4 c" e
<name>dfs.webhdfs.rest-csrf.browser-useragents-regex</name>$ S7 ~5 C) k3 t5 H2 `# a
<value>^Mozilla.*,^Opera.*</value>
$ g% B& k V5 Y( \6 e; I& S <description>
% A3 {$ F w( _3 c3 x: k5 { A comma-separated list of regular expressions used to match against an HTTP
$ D& E. {+ e5 g9 s1 x: u" n; @ request's User-Agent header when protection against cross-site request: j: E: R& Z% o' S( B# X
forgery (CSRF) is enabled for WebHDFS by setting
* w' P+ g8 g6 p dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches, z# `- ~8 |1 b$ C
any of these regular expressions, then the request is considered to be sent
& }# f! Z3 n9 @+ p q" P5 {; | by a browser, and therefore CSRF prevention is enforced. If the request's
6 z; J- P' h. s) E2 V8 A. h User-Agent does not match any of these regular expressions, then the request
* f W7 J- h# I% g( ? is considered to be sent by something other than a browser, such as scripted+ a" g5 H5 o% W
automation. In this case, CSRF is not a potential attack vector, so9 ~* z! y5 X, Q( d# X n# n
the prevention is not enforced. This helps achieve backwards-compatibility
/ K: t% g- |; d! U! {1 {! G) g with existing automation that has not been updated to send the CSRF- T! s. F8 g9 X$ g
prevention header.: V2 S* _3 K, D# q; P5 l. o, u( k9 C
</description>9 _& h4 I [' O6 Z' V
</property>5 ~: W% C/ v( S
<property>* F% c, S; p7 }" n4 ?. |8 s
<name>dfs.xframe.enabled</name>
! O- \, l0 f' E2 x) j& v/ b% o: Q <value>true</value>
/ m4 l& n0 e, {% @1 ?. R7 x <description>
' B4 `# o$ I, u If true, then enables protection against clickjacking by returning- {8 t- L" H$ s! X0 g j, A. q
X_FRAME_OPTIONS header value set to SAMEORIGIN.4 n; V1 N0 `' |3 V1 a& a. C
Clickjacking protection prevents an attacker from using transparent or$ x+ ]5 R; Z, P6 S4 x w, g0 h
opaque layers to trick a user into clicking on a button) F v1 n2 w7 }* I* ~1 g
or link on another page.
: u9 X& C9 O8 l </description>
$ `0 }2 T* [: j# l$ ^5 S' Y </property>0 v% |# q \+ i
<property>; ~5 B% ]- E# I6 W: Z
<name>dfs.xframe.value</name>
/ d$ z9 a6 _. C# E; K _7 t <value>SAMEORIGIN</value>6 H) _% q( ^7 H/ u9 @+ ]# \
<description>
4 ]7 Z+ h- N) z9 {* p0 w This configration value allows user to specify the value for the2 C' H% v: x! h
X-FRAME-OPTIONS. The possible values for this field are
8 J; m; A) \2 Q+ P" _: j# q; ^, s DENY, SAMEORIGIN and ALLOW-FROM. Any other value will throw an7 c T: Y6 X7 X/ _9 d6 P
exception when namenode and datanodes are starting up.
/ s K2 D0 s6 }- y1 H) F </description>
# ]+ _% _: D- M8 X# Y( M </property>5 M" `# K8 F* q+ U" o
<property>: ~- k1 K6 u0 P6 c
<name>dfs.balancer.keytab.enabled</name>3 C( T1 p6 L1 }! s& K/ ~" {
<value>false</value>
3 c/ f; V6 R9 O* S5 W( a <description>0 l$ T& a7 m1 A. g0 ?5 {! M- t
Set to true to enable login using a keytab for Kerberized Hadoop.3 Q8 N7 s' G6 j+ g: O% v
</description>
9 _2 ~0 B1 S" d+ |</property>
, N6 F9 k% ]3 y/ g7 L! L8 ]& M/ O<property>
9 o; q9 I0 A1 `" e% e/ z <name>dfs.balancer.address</name> t% V! J/ G+ _, J, }5 [
<value>0.0.0.0:0</value>
" f$ R P8 q8 ^- u; w <description>3 x: f6 ~9 Q. O0 N1 i: I+ Q
The hostname used for a keytab based Kerberos login. Keytab based login/ g2 l3 p$ Z: e t# b7 p
can be enabled with dfs.balancer.keytab.enabled.
" d4 v! @8 i+ O/ |/ n </description>
. C# h8 d, W S1 c) T3 ~! R</property>
`1 D, u- J" z) N+ w) w. C: f4 Q<property>" y9 D: l% c/ b n0 g! X, V4 l2 W9 E0 M
<name>dfs.balancer.keytab.file</name>
/ k% I# @* v# c% R9 q4 | <value></value>! Z2 D U& e: u6 w* n" j
<description>
" P* k' i5 z# h$ ^3 r; d6 V The keytab file used by the Balancer to login as its4 O# d, o. d$ t, v: I
service principal. The principal name is configured with v" P/ z& h, z4 y$ [
dfs.balancer.kerberos.principal. Keytab based login can be9 w4 [. Y1 w' s4 B
enabled with dfs.balancer.keytab.enabled.
1 O9 m: E6 w6 }! R' L, Q R* B </description>
% v# M7 k: @# T/ v/ n</property>: W. D/ t" w1 Q/ R% W/ D7 g6 |
<property>
0 ~6 {3 q( t% J+ ]4 @+ c, A <name>dfs.balancer.kerberos.principal</name>
' L1 Y# ~! O0 @( W5 |! f <value></value>
, W x' M: _8 V0 Q s <description>8 ?: w% y$ A# d
The Balancer principal. This is typically set to) V; h5 I. w; u( Q p/ _* N& X+ O/ `
balancer/_HOST@REALM.TLD. The Balancer will substitute _HOST with its
$ _+ W9 B) I) M own fully qualified hostname at startup. The _HOST placeholder
7 ^, E+ k! D2 h; }% M7 z allows using the same configuration setting on different servers.
% G- Q% c& I. ^4 Y0 O9 N Keytab based login can be enabled with dfs.balancer.keytab.enabled.7 i: n' S& L2 N, v9 Y
</description>
! ?) N' `5 B+ S7 k* u</property>
( m9 z+ ]7 k, c0 F" j/ \<property>
9 X( H4 d; }2 R <name>dfs.http.client.retry.policy.enabled</name>$ ^; O7 W' i J; _* D
<value>false</value>) n# [/ ~7 H5 v# i+ m
<description>- b4 i5 X; M. c# h) J& c) F
If "true", enable the retry policy of WebHDFS client.( Y3 E N* C1 s4 n3 k
If "false", retry policy is turned off.3 p4 F, v. f& p& u! B8 s
Enabling the retry policy can be quite useful while using WebHDFS to
4 c7 D% d5 I# Q+ q) L copy large files between clusters that could timeout, or
4 I# ]& d* l/ b. A! _1 R copy files between HA clusters that could failover during the copy.6 Q) P7 n1 m" a4 g
</description>
; e6 @" N. {2 u* H</property>' n6 Z4 X" _9 s' z8 _6 ~
<property>6 T. |( i- }' z _0 v. ~; F
<name>dfs.http.client.retry.policy.spec</name>( E3 }! g* F+ t+ I" x0 V
<value>10000,6,60000,10</value>* R8 N. w+ l; S' ]1 V5 h
<description>
9 ~: S9 X$ x1 T" F Specify a policy of multiple linear random retry for WebHDFS client,/ F+ m3 s3 [, S% N9 [5 d
e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1),& n1 a0 F0 a, l
..., the first n0 retries sleep t0 milliseconds on average,$ `" K3 }" O$ e1 }# X5 }' M
the following n1 retries sleep t1 milliseconds on average, and so on.
$ |5 G& K1 z2 a9 B' d </description>0 g6 f3 Y' O+ l8 F8 |' k5 \! \
</property>9 y* A: `) g: \$ o, N+ A7 r
<property>
& S6 ]* w9 r6 w$ c1 G! X9 K <name>dfs.http.client.failover.max.attempts</name>2 C* O2 f. t1 A" ]5 v
<value>15</value>, E/ ~; }* e0 M: H% {. [
<description># l1 H6 a c A+ A }1 F
Specify the max number of failover attempts for WebHDFS client/ S1 \9 p/ k- r" ~" t* E
in case of network exception./ q+ A- V- w+ U f3 Y
</description>
; K m" ?/ ]0 ~, V' O2 ~* c</property>
% I& k" K! J- M0 Z<property>
" k8 L, R" m4 E: X. i' F8 x5 u <name>dfs.http.client.retry.max.attempts</name>
% ^, q$ B, o: G; _ <value>10</value>
* e- F1 Z! O1 U) n4 A6 f" F <description>
7 a* K0 T% C6 K, U: b Specify the max number of retry attempts for WebHDFS client,5 t2 f2 _# ~2 T4 d. q
if the difference between retried attempts and failovered attempts is
! o, `' S. Q4 u; B- Y" |* x9 ?% i& { larger than the max number of retry attempts, there will be no more5 K2 U# s2 p2 b5 C
retries.$ h) L- g$ [9 H1 d
</description>& @; A/ h1 e6 g7 |" {3 g* b0 l
</property>+ A, l( H8 h, T) E0 F
<property>+ {5 a# u% R6 _( |' ~
<name>dfs.http.client.failover.sleep.base.millis</name>, r- f2 D" I& T) u* v0 F# K l
<value>500</value>
6 ^9 c- z- i: D# F% t# C! q <description>5 f7 _9 Q+ u2 V- F+ v/ V0 U7 v
Specify the base amount of time in milliseconds upon which the
7 r4 u/ O- u/ W exponentially increased sleep time between retries or failovers/ s7 V; t$ O4 A4 D* R
is calculated for WebHDFS client.6 f& K) C: Y; J3 x+ K+ [3 G
</description>9 b3 F+ d& ^8 r7 {. }, b: g
</property>3 i# s* A8 N, S j' N. }
<property>9 X5 m% b$ h9 W. v( ~ D O
<name>dfs.http.client.failover.sleep.max.millis</name>! r( N: m1 g' M+ \3 y) x
<value>15000</value>5 |6 w8 E8 J* v" `" N: A
<description>
* p1 J3 m1 n9 @8 Z1 U9 y2 L" l Specify the upper bound of sleep time in milliseconds between- u; \# a0 [1 N# T
retries or failovers for WebHDFS client.
' e0 I. h% U+ \4 P% J! l: N </description>
\, P9 E% e8 `: ]. m</property>& N" u/ U; X! H2 W9 w5 I) T6 n
<property>
, F5 S8 Y4 a5 @1 ~1 j# Z <name>dfs.namenode.hosts.provider.classname</name>+ C' N" v% _' \+ x" I
<value>org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager</value>/ {4 h" b2 H' H2 d3 [) F
<description>
- u) q! ]8 w) U: ] The class that provides access for host files.& X. F: b& |. @5 s
org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used5 K$ ^2 T# _( N
by default which loads files specified by dfs.hosts and dfs.hosts.exclude.' K2 X* z# P' R: l3 k
If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is, W8 [) p( J- u% x/ V
used, it will load the JSON file defined in dfs.hosts.% t( @7 ]! H4 N" l$ s! I0 A$ G
To change class name, nn restart is required. "dfsadmin -refreshNodes" only
$ F5 E; I( D* _ @ refreshes the configuration files used by the class.
7 `8 _7 ^8 Z7 W: e </description>( w+ Y- D0 h! q
</property>
, j# f8 k" _5 M; i& r6 H0 C5 C<property>
/ M! w+ ?1 K. F( y. W6 A <name>datanode.https.port</name>
5 c5 c! y+ I# t7 B+ V <value>50475</value>8 e# ^7 ^4 j/ F$ P, v" K
<description>
' S& ]: X% k( X* A/ W HTTPS port for DataNode.7 R9 j8 R7 v# s
</description>) s. Z6 K, E( i
</property>
: Y* z, y+ }. t+ r3 R# h<property>& Z) \8 S/ t0 `# ^1 m3 `' [
<name>dfs.balancer.dispatcherThreads</name>! R1 G# U: q1 U( e: `7 @
<value>200</value>
0 X+ D2 O6 A6 o# d4 a% [$ b <description>2 Y; y$ t/ l2 ?- }) i# W0 o, ]
Size of the thread pool for the HDFS balancer block mover.
4 [4 R4 W v: F' k1 D dispatchExecutor1 V. M" a& [- N5 K0 ~1 T
</description>. O( p1 R6 U0 S. g4 x; |* W3 D2 q
</property>
& A# x. v" u4 B$ E! h<property>
$ A# `8 @/ E1 f6 W& o <name>dfs.balancer.movedWinWidth</name>0 _9 X2 P- {2 ~( ?5 ^. E
<value>5400000</value>
9 Z2 q6 T' f. |1 ] <description>: M: }/ S$ \ s3 Q# r% R! ?
Window of time in ms for the HDFS balancer tracking blocks and its
/ J8 s8 {' f. D locations.
$ u [$ e3 Z( v( a: _+ | </description>9 T; } T; Y* e+ L3 ^
</property>
5 m! P7 i4 P9 o0 Q<property>
. @% f: l j! O <name>dfs.balancer.moverThreads</name>
; p" i- A; D. y <value>1000</value>. T9 N+ n: K; |6 p6 l6 o
<description>+ h/ H- m; b' I! O$ z
Thread pool size for executing block moves.
: k2 ]% q2 ?& C* m$ y moverThreadAllocator$ }8 o9 T. p, D
</description>. v, M5 `# F# }2 [# f7 Z' k
</property>4 I3 ~) ]7 l: `: t
<property>( _6 ?. R+ ~# t# Z
<name>dfs.balancer.max-size-to-move</name>& S) Z. T2 F, W, E' b0 ]( N
<value>10737418240</value>- M5 W7 e/ ?6 l8 W( n& x
<description># [$ N$ G. `; t+ x& l0 ~8 y/ p! r% E
Maximum number of bytes that can be moved by the balancer in a single
$ F# c5 X& e1 ^, O+ v& b8 H* M! { thread./ |7 @5 `4 }, j$ X+ T
</description>3 x% v | b% C
</property>: }$ W& G5 `5 |; r; v8 `
<property>
( }! O9 Z$ X& [2 V* W <name>dfs.balancer.getBlocks.min-block-size</name>
7 j. v* f0 _6 l. [( z4 G' c* H <value>10485760</value>& i: O6 O- N; T9 h6 j; |2 m3 x
<description>
7 ]6 b1 D) I& I( A/ o) ], N Minimum block threshold size in bytes to ignore when fetching a source's! I# T/ J7 f J9 F+ ]2 Z
block list.3 G. U1 T. t) O8 d" G& @, q! E
</description>
) D; N+ P+ o1 z6 P</property>1 w I: \% s; @2 q1 i8 G
<property>
' K4 p, \& V5 G) `& ? <name>dfs.balancer.getBlocks.size</name>
E& ~ B) c+ c+ @' n1 Y <value>2147483648</value>
, {4 x# g5 h* ~ <description>
1 P0 d( L4 B9 G Total size in bytes of Datanode blocks to get when fetching a source's$ P; n3 t' O0 |$ d& F
block list.
: k' A) W5 l* ` }6 L% Q" j, w </description>! W8 T! s0 X9 ^& P2 e+ |! h
</property>5 m. D! f. w, `5 h/ G$ k
<property>
3 C, U/ G3 U5 v) ]" V <name>dfs.balancer.block-move.timeout</name>
9 J B3 ^. V7 t# F. m <value>0</value>" v" \0 ]" M/ G1 U
<description>
7 _) B: f$ T0 U; A: G+ c Maximum amount of time in milliseconds for a block to move. If this is set
! F# I7 l+ J! N. m3 `6 [* @: k6 } greater than 0, Balancer will stop waiting for a block move completion( ~) k) X8 Z1 l5 c' ?3 E9 U
after this time. In typical clusters, a 3 to 5 minute timeout is reasonable.
4 i+ `' q4 P i4 w9 V' P If timeout happens to a large proportion of block moves, this needs to be0 [. M% R' s' o
increased. It could also be that too much work is dispatched and many nodes
7 ^2 w+ n6 r" ? are constantly exceeding the bandwidth limit as a result. In that case,
7 y2 m7 h( T3 a) O$ @+ { other balancer parameters might need to be adjusted.
) f+ m# H0 \2 i, t+ V" t It is disabled (0) by default.
, _- {7 B+ M8 a, t1 u! P </description>
T/ {7 r; J0 \3 P</property>
. U$ R* P9 @) {! r- u+ W3 l) m<property>' s: K8 {8 {4 n$ q9 }* _! s
<name>dfs.balancer.max-no-move-interval</name>
% u) E$ A! t+ w$ U% A# d5 @ <value>60000</value>
9 x2 ?* c* L5 \- q <description>
0 K8 p8 J6 Y) k' h If this specified amount of time has elapsed and no block has been moved
6 \+ F7 Q+ `, `) u* j out of a source DataNode, on more effort will be made to move blocks out of
! l( c2 t3 U. Y; m& I this DataNode in the current Balancer iteration.! C5 r! p( u+ M% a! ^# Z
</description>& y' q/ E9 U5 p% o" `
</property>: i* }& V# d8 a- I
<property>
/ {# o) N i7 @1 n% K <name>dfs.balancer.max-iteration-time</name>2 g; q: o$ L( O3 i6 Z! r
<value>1200000</value># v$ _6 a+ h4 x9 w
<description>
" v3 Z- S. ` b9 m% i* d0 F! V Maximum amount of time while an iteration can be run by the Balancer. After
& ]) f% j6 l% s% m _+ O$ I this time the Balancer will stop the iteration, and reevaluate the work( a: u& h W+ F0 g; W2 d# H
needs to be done to Balance the cluster. The default value is 20 minutes.' E" L& Y/ e0 v4 m* W- i8 e
</description>' h( x; _$ P2 y" Z
</property>
$ J+ m' {9 a/ {8 O( X. G<property>
4 }+ [7 [: E1 s- y5 M( R <name>dfs.block.invalidate.limit</name>% Y+ p2 Y6 c# V
<value>1000</value>
, }; a- U# O, M <description>
8 |. l. C& X6 R The maximum number of invalidate blocks sent by namenode to a datanode
! g a. x1 S3 S( {0 R+ |$ m per heartbeat deletion command. This property works with) H- f3 R, [4 \# L( `$ N; o% h
"dfs.namenode.invalidate.work.pct.per.iteration" to throttle block
$ I+ o i7 K5 d) b; C: J deletions.+ O. R, r- z! }
</description>3 v/ A# g. Y$ q* j) C
</property>- f' \2 s, W, S
<property>
" _7 r6 L1 m; K5 x( B% n+ v& n <name>dfs.block.misreplication.processing.limit</name>
- ?% D; \' C! L1 X3 P7 w <value>10000</value># u6 p# I3 N8 F# ]+ L5 B
<description>5 `! E. I1 }7 E: I# h* M5 [
Maximum number of blocks to process for initializing replication queues.
9 l' Z! y0 f& [ z( C+ {# [ </description>3 c( y2 L* e- m9 z; y* X& v6 p5 j
</property>6 c0 v6 x8 v, F3 j
<property>; j% U. _( r* C& r! i
<name>dfs.block.placement.ec.classname</name>
* l/ L+ V z/ G" \0 X! K$ {( o <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant</value>( O }! n- A. q; g# d
<description>+ \5 O3 e0 t+ v \, @
Placement policy class for striped files.
$ i; Q* q7 z* |! O' K Defaults to BlockPlacementPolicyRackFaultTolerant.class
- O" D& r. {* \' E3 N8 [ </description> G8 r5 }9 l8 ~/ P2 ?: |
</property>
$ _% ]# ?# A) _! E* b, J" E+ B<property>( t6 c5 P2 b( U2 _+ H X
<name>dfs.block.replicator.classname</name>
1 S5 t3 a7 Q2 ]4 e4 e <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault</value>5 T- v% B# \; Y' W9 o
<description>7 E! x0 z) U9 u ]+ z& R& ^
Class representing block placement policy for non-striped files.
, a/ Z3 M) {9 C; r; E There are four block placement policies currently being supported:
. }% J U9 ?( a' X7 V; K BlockPlacementPolicyDefault, BlockPlacementPolicyWithNodeGroup,
( z. I7 H: k& g# r. t BlockPlacementPolicyRackFaultTolerant and BlockPlacementPolicyWithUpgradeDomain.
( D$ I; v! l5 o. q4 L1 v BlockPlacementPolicyDefault chooses the desired number of targets; ], z# V- k; m: f
for placing block replicas in a default way. BlockPlacementPolicyWithNodeGroup+ ]5 a- D( ]/ q* D3 q" Y K; T
places block replicas on environment with node-group layer. BlockPlacementPolicyRackFaultTolerant
+ }$ I3 D. O) z$ @ places the replicas to more racks.+ w7 T' Z r# K# G5 `
BlockPlacementPolicyWithUpgradeDomain places block replicas that honors upgrade domain policy.1 s6 v; U" Z! }% `9 t4 p1 z
The details of placing replicas are documented in the javadoc of the corresponding policy classes.
2 F$ ~; Y; [+ G" R( @) F The default policy is BlockPlacementPolicyDefault, and the corresponding class is, A! F0 \- ]5 [) | ?4 C& R
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
: b- k/ A3 J3 }2 \. B2 l. B+ F# U </description># u3 n- s4 e1 I9 M$ A) I, u
</property>5 ~. b/ c' r6 e
<property>, F3 H) X4 a1 ^ A( [( e
<name>dfs.blockreport.incremental.intervalMsec</name>
8 L; D6 R2 T0 Q8 U, }2 g <value>0</value>8 i" c7 J$ p. @' R) v
<description>) e/ q8 j& h- C [5 {+ } a
If set to a positive integer, the value in ms to wait between sending* d/ J1 }: n. v/ c$ ]% W% t% @
incremental block reports from the Datanode to the Namenode.
5 c! z7 T1 d- j/ \/ d, G </description>2 a$ g8 E8 o) u" G3 ~) m
</property>
9 ? W! T! P& [2 B4 v5 M+ t9 x# ~<property>
4 K% X8 q$ K& Q; n/ X6 E <name>dfs.checksum.type</name>: e$ g" b9 {. H, F3 v. R. \( E, W
<value>CRC32C</value># S8 C( S- B* V) T$ Z
<description>
2 I- Q& V2 }$ \0 A6 | Checksum type
- e& d! g, a% }* x" I. I </description>' u% ?- y9 f, o
</property>
9 L E$ i& I! P9 h- u) b, g( t- n3 Y<property>$ |: a$ V# z |( i* \- M8 J
<name>dfs.checksum.combine.mode</name>
) @1 O8 \" b% W, Q- l4 ` <value>MD5MD5CRC</value>
( G R/ d2 e9 I% |/ L$ r2 O <description>
0 [* d5 c2 d, _# p0 I" }- M Defines how lower-level chunk/block checksums are combined into file-level
# o2 R+ U: ~# |5 u/ p( C9 E3 h% @ checksums; the original MD5MD5CRC mode is not comparable between files! a+ O$ [$ @, o8 t/ K8 J k, ~
with different block layouts, while modes like COMPOSITE_CRC are7 Q) |7 U9 @" j5 v) @& O* t4 [
comparable independently of block layout.8 f" W: w0 ~* | Z
</description>
% v% @) o" A" M8 W</property>
; F4 d0 H3 z3 | B0 h8 [5 O. G<property>
0 s" X8 U& k% \) r <name>dfs.client.block.write.locateFollowingBlock.retries</name>* \9 O) r/ V9 [3 I. U! E
<value>5</value>9 c$ _, \8 T" t4 }# j7 x
<description>8 ?% J" m* Z& H- l3 x% h6 r
Number of retries to use when finding the next block during HDFS writes.
6 k' ^" U% `. h! |. Q/ \7 O </description>1 p0 F t) g) e9 x% h
</property>) Z( o2 a) [6 ^. H5 P) G; Y) r
<property>
- A- y; N' ]$ L$ g& N- Y G/ @ <name>dfs.client.failover.proxy.provider</name>
# F8 N& Z% Y1 D- | <value></value>5 I! a, \9 d7 n6 Y1 @
<description>
, b# h8 k3 m* J7 G The prefix (plus a required nameservice ID) for the class name of the& B) Y2 u/ C7 h: _- \ {- c
configured Failover proxy provider for the host. For more detailed
- J" H& B. W, [2 V information, please consult the "Configuration Details" section of
) E, G5 @. D. t the HDFS High Availability documentation.) z9 l |; f/ g9 l2 {$ N: r
</description>+ \% v+ X7 X: d/ s7 H, m' {' B) `
</property> M1 {; ^0 f* V, e" h$ A* w( O. s
<property> R* d% ~5 n2 J
<name>dfs.client.failover.random.order</name> h) K0 { a6 c
<value>false</value>( Z- j2 b( v9 J) W- l
<description>5 w8 a/ E8 q: _1 v
Determines if the failover proxies are picked in random order instead of the/ ]# L6 O6 z4 H/ @
configured order. The prefix can be used with an optional nameservice ID/ u3 d" c2 t6 }$ n- m+ \3 Z- E
(of form dfs.client.failover.random.order[.nameservice]) in case multiple# T8 ^0 Y, z, t3 U; u: j
nameservices exist and random order should be enabled for specific5 D. T+ T, M& T/ N9 E+ y ~
nameservices.
( _' {' }8 ?2 w% W. {' @* x) C </description>
5 i1 V) s& S1 z</property>
3 f+ ]. N+ T+ K2 r, @: z2 L<property>
: X9 f$ x+ ]- t# k4 r, N3 b <name>dfs.client.key.provider.cache.expiry</name>, x4 |9 G; E1 C6 V( l
<value>864000000</value>
! K% ~6 h2 v5 Y5 ]; L* F. ^ <description>
( [& G( Q. B. ?1 q2 V DFS client security key cache expiration in milliseconds.0 s2 G0 D$ u# T' q; U# m
</description>% I2 S" C2 m# L3 `0 E* ?8 }
</property>
: D0 G6 z4 M% k- o4 g9 i v<property>2 }. e8 L% D; Q, I
<name>dfs.client.max.block.acquire.failures</name>- H8 m' J7 h$ F3 v/ ^
<value>3</value>
2 x4 e* ^( J/ r# X% b <description>
& W5 R- a6 G1 V& y/ Q Maximum failures allowed when trying to get block information from a specific datanode.: r+ b1 k9 z: {3 ?3 z6 ~+ i
</description>
7 M E( m& @2 z3 _; ?8 k5 w/ G</property>
1 c% p+ V( z8 v: t+ @! O<property>
* o7 `+ v! W0 D( D0 n <name>dfs.client.read.prefetch.size</name>3 g, N [) m4 h U3 Q4 O9 h) B
<value></value>4 i6 a$ u1 l e% Z
<description>
! h' m A! m/ L; ^9 `& H The number of bytes for the DFSClient will fetch from the Namenode: j k+ c/ Q X- h" o3 J
during a read operation. Defaults to 10 * ${dfs.blocksize}.5 R% r5 ]1 e8 @- h9 V
</description># b( j, ^6 h, T
</property>( e: [3 t4 N8 V. }0 Z( {; M4 {
<property>
9 l9 q$ U6 z- p$ h6 Z# M <name>dfs.client.read.short.circuit.replica.stale.threshold.ms</name>, J% e' k8 |4 H+ C
<value>1800000</value>5 ~( \' K! b2 I( T
<description>
& B- p: f) c! | r) u8 R Threshold in milliseconds for read entries during short-circuit local reads.
6 ~! M b$ l0 Z( G, z- |: P </description>7 Q# d3 ]( q; G- T$ a% F3 l
</property>
; M" L$ T$ N4 _5 o<property>
6 e+ W. c. a& W1 t* S( [; n <name>dfs.client.read.shortcircuit.buffer.size</name>9 g( A$ h* `. |/ W9 z
<value>1048576</value>+ w2 N( ]( W) D2 N+ D. q. f
<description>
6 q5 \& g8 S3 l% W Buffer size in bytes for short-circuit local reads.1 B2 p F7 M# P& h) v- \
</description>, a' P8 M/ \; x' c3 {
</property>
. E6 e* d1 |& s1 `<property>
& T; z8 K: U+ P% j <name>dfs.client.read.striped.threadpool.size</name>
B9 |4 l9 b0 ?- a# A& j <value>18</value>* |- r' N8 f9 q
<description>$ `0 S L2 o( e) U; Q
The maximum number of threads used for parallel reading
: j3 }( e( E. J0 Z) P% N in striped layout.6 Z# W5 e# P7 f7 ~0 A' \
</description>
$ x3 ~0 P3 r; P L) g4 @</property>* F. C. V3 I$ w" g4 d
<property>
8 i6 j0 d- _8 W3 n' e" i& p <name>dfs.client.replica.accessor.builder.classes</name>8 i* y$ l) C3 a! j7 b5 x
<value></value>
8 f4 N5 w7 ^0 T2 I! {+ e3 J; E <description>$ Y, ]& s+ N) t% t! @" t
Comma-separated classes for building ReplicaAccessor. If the classes' z$ ^, _5 ?8 f0 }" J$ I
are specified, client will use external BlockReader that uses the
( z% L6 X+ o1 v% ?4 t1 B+ m ReplicaAccessor built by the builder.3 F3 j2 B( j' I6 W3 d
</description>
) H/ q. `) L7 O7 C' a# z</property>
7 w$ \+ |7 d% T' `<property>
8 b; G& h! |" y8 C7 Y, N <name>dfs.client.retry.interval-ms.get-last-block-length</name>
4 j3 s z; u# v# Q# w6 _ <value>4000</value>, Z% W* M2 v+ }- P! {9 w) Q
<description>) u7 R9 \6 L! B7 R3 k
Retry interval in milliseconds to wait between retries in getting" P* m8 I) j+ D$ C0 `' r5 C! j
block lengths from the datanodes.
/ ~- l3 \7 ~5 ?& g9 o% ?% J+ |6 Y( P </description>
5 a+ z) y; }3 X% L</property>: j4 U% z2 d! r: a
<property>
. I! |) F( C6 x' A5 r' h <name>dfs.client.retry.max.attempts</name>
: [8 b# Z5 Q+ k& @' H/ }" Q <value>10</value>
+ P' Y6 G' Y- ?' [ <description>4 K# a! B7 \) t& Q/ B ~, ]) W b
Max retry attempts for DFSClient talking to namenodes.
8 w9 u4 O, d2 b5 m( G </description>! K) G7 Y3 ]# i4 d
</property>
9 h1 o0 i/ h2 [4 ]7 _; f<property>
7 c- F) o) L2 b. Y( g3 b <name>dfs.client.retry.policy.enabled</name>- d% D$ @+ ?; h R
<value>false</value>, J9 s9 v) v2 `: A7 T( g% ~ F
<description>( Q5 T3 ~- r5 f
If true, turns on DFSClient retry policy.
' j3 Y* j! Q. b1 h- }! O1 k0 k </description>. d# K# v% G" e0 w
</property>' i h' z) _. h! w& i; l
<property>
) J+ _+ Y' n# U' E- ~ <name>dfs.client.retry.policy.spec</name>
, u5 f! e S5 N k- C# | <value>10000,6,60000,10</value>
: b* z7 Y, ^" w K# m <description>
! Q5 k- U* }; l1 h% f+ J9 v3 T Set to pairs of timeouts and retries for DFSClient.
# e' y3 c" F1 F' E! F$ S </description>
' Y' b% t7 k) @. N& n. K- ^# i! L</property>
5 M! M( y* p' v<property>
/ h; S, a7 I8 Q6 M6 \5 ^% r <name>dfs.client.retry.times.get-last-block-length</name>4 \# i K0 P; d4 O# _$ k* M
<value>3</value>
3 j- q) x: g8 _6 X: c; L <description>: T, J T& X/ R7 ^/ C; n5 d' l8 q
Number of retries for calls to fetchLocatedBlocksAndGetLastBlockLength().
6 Q' r) l3 Y0 s; H" M1 e, j </description>
7 i) j/ K6 T& O# h</property>
$ f% d0 [ r% B# K/ K* y' t<property>
* P) w; l6 ], @6 u2 L" n9 Z {. h <name>dfs.client.retry.window.base</name>
: i' L6 g' u4 @7 P <value>3000</value>+ `7 F' S1 M3 L- t
<description>' A, q2 b& j4 x7 W
Base time window in ms for DFSClient retries. For each retry attempt,+ H; D, E. V2 c: W$ n% X$ c6 P- C. ^
this value is extended linearly (e.g. 3000 ms for first attempt and
8 ^3 w$ Z! b) \* i% F first retry, 6000 ms for second retry, 9000 ms for third retry, etc.).8 O9 l5 e7 [; W- J" }
</description>
1 o4 F9 y3 _, E& e9 M& D9 ?</property>6 S+ g1 y0 u3 I' X- X* z( @
<property>
( F# v8 x, o& l) g% B# w5 A <name>dfs.client.socket-timeout</name>
! C" o% J7 p& c$ ]/ f: a$ ] <value>60000</value>$ Z& v/ q/ K0 |
<description>
- A) J" A4 `5 q- |2 o Default timeout value in milliseconds for all sockets.
1 Y, K0 D& n# a6 K5 a- L9 M- B4 G </description>
$ g% `, P+ L# M( w</property>1 O5 a: ]* p% D2 W' K% Y' Z9 ]
<property>
I) n+ J8 K& Q% \. F <name>dfs.client.socketcache.capacity</name>0 q3 z: N* s# I- P' Q7 S" M
<value>16</value>
, C* X) }. d# W7 u* ~" q! X <description>0 ]7 ? r. }( M+ {+ e) N* d" y
Socket cache capacity (in entries) for short-circuit reads.
. ?/ V: Q& b6 W* \6 o4 b5 l. z </description>1 q6 X6 r4 P* O6 r
</property>
. E/ D$ w/ A1 L' L+ l+ q0 r<property>
# W5 t |; K! i4 J* q <name>dfs.client.socketcache.expiryMsec</name>
8 i; _& H4 g: ?, X <value>3000</value>+ A5 |. ~' d& x& l( N6 u
<description>
$ N' q, Q7 N9 `$ f/ G5 E' a1 I2 s Socket cache expiration for short-circuit reads in msec.
; J7 o/ I$ `$ C9 d6 I </description>
( D/ R# d: H6 [: t0 r</property>
) x) x& M' O" E<property>
S2 ]( A( k' r/ j <name>dfs.client.test.drop.namenode.response.number</name>
% }8 _8 t5 P. C. N5 ]" c <value>0</value>
% a" y. `# D( v! r$ b+ |* H" t+ U <description>! b; \5 u" b9 [$ t7 ~' N L+ k+ i1 g
The number of Namenode responses dropped by DFSClient for each RPC call. Used
0 C$ p9 K6 w# g) ~. V for testing the NN retry cache.
& D; H2 X% H% q2 I" o' J </description>
( P: s: C/ r/ D3 U</property>1 P+ U& {1 v6 \8 ?7 M2 j
<property>1 H$ @1 u1 u8 f5 \7 u+ g: U2 v
<name>dfs.client.hedged.read.threadpool.size</name>
$ V7 _" V1 p# M# V1 |2 N' @# Q8 `$ Y <value>0</value>
6 _4 l. g) d& l3 F <description> k2 F8 [1 x' A2 W* r D
Support 'hedged' reads in DFSClient. To enable this feature, set the parameter
( g1 n" d/ }2 o R4 ? to a positive number. The threadpool size is how many threads to dedicate
1 F1 f* j. K Q! E" P to the running of these 'hedged', concurrent reads in your client.
3 Z+ k2 e4 [) h4 M </description>% T; q+ G, G" k7 N! O0 b: d
</property>
/ B$ T7 Z$ w+ ?6 s<property>8 O5 D4 j" w7 y2 h5 T, j
<name>dfs.client.hedged.read.threshold.millis</name> B# x( P4 N7 S8 j" v& \
<value>500</value># t/ v$ W& K% c: @8 o
<description>' E# c, _* Y4 E7 s- K; s2 F
Configure 'hedged' reads in DFSClient. This is the number of milliseconds
: U' k# C2 w9 h X3 b0 A& B to wait before starting up a 'hedged' read.
6 d! Y+ d# [" M3 \& b( y </description>1 p) c6 C% C1 I' t
</property>
5 o5 e) \# l7 X0 i2 E0 M2 V+ j# {<property>
2 W2 Q3 k$ R- {' E; `* X# S4 l <name>dfs.client.write.byte-array-manager.count-limit</name>
0 Q2 R# H: {/ v) U# i, g <value>2048</value>6 j3 h5 g4 q9 B# |" u
<description>- Y) A% b( w% D0 Q* \& S+ s. T& K
The maximum number of arrays allowed for each array length.
7 }- D$ B9 c: e </description>" E# P4 L% f* n" L: a; Q$ N9 U) Z
</property>& a; n% S: Q: T
<property>
6 o1 I9 x' Y* o% l9 M) F <name>dfs.client.write.byte-array-manager.count-reset-time-period-ms</name>; w0 H9 b) U6 H( A5 ?
<value>10000</value>
u2 h0 H. v; Y6 f <description>1 ~4 `5 q" ?1 z; j5 c
The time period in milliseconds that the allocation count for each array length is
4 w! }: n! a. T. s8 [* ]$ q# @( ~ reset to zero if there is no increment.# I3 `0 u1 D% a6 G2 z0 K
</description>% E! v' {4 V7 q, t9 ?# e5 k
</property># i, O+ m/ _* r0 ^& W
<property>3 [" a2 [. _+ ]; k8 R% S: n( m
<name>dfs.client.write.byte-array-manager.count-threshold</name>( L! l8 T2 P+ Z' @' D3 p8 I
<value>128</value>
/ e5 c0 T0 h1 R/ s0 M2 _ <description>6 e% ?! Q8 t1 N$ d4 X* I" ?
The count threshold for each array length so that a manager is created only after the
; B0 {% R- j, y7 D" w+ D; J allocation count exceeds the threshold. In other words, the particular array length$ A/ [5 f8 d2 X5 t, K
is not managed until the allocation count exceeds the threshold.
& \0 A$ o; t* S3 ? </description>$ T" U5 U; {; E0 c9 t
</property>& ]' I" X1 V9 V2 O1 V/ ]
<property>6 D9 r; S9 u0 k+ e. G) p* v
<name>dfs.client.write.byte-array-manager.enabled</name>
1 ?+ _$ j+ n5 ^: C" k4 G8 g% X <value>false</value>
1 N M) B. V9 a <description>( ~, g' h. ]. x/ o4 }$ g6 G5 B& ]
If true, enables byte array manager used by DFSOutputStream.
$ U) g/ ^$ Q1 F: \5 Q* k& _3 M </description>
) k- }7 I4 K" C0 P8 s! F7 V) q5 T</property>6 X5 |& |' Q6 v0 p
<property>
( o% t+ R5 ~2 I! }8 ?5 F, h <name>dfs.client.write.max-packets-in-flight</name>
% f4 ]7 S6 m2 o, G% V2 ~5 Q1 h6 Q <value>80</value>
, R' A+ j: B: I! Y; Z" F' S <description>8 Y$ c. S: J% ]# f3 i' o" z
The maximum number of DFSPackets allowed in flight.' m* M' p6 {0 Z+ a) d' [6 P
</description>5 A# t4 Z5 a2 p0 y
</property>
+ I5 f f: M5 ^: x% e- F! E<property>
% m1 @7 H0 ?* X# o, f <name>dfs.content-summary.limit</name>" {) H, d7 F$ `# w
<value>5000</value>
! @5 k$ F$ ?( u8 F# V <description>. K A" h+ p G9 Z% ~( q2 C: e
The maximum content summary counts allowed in one locking period. 0 or a negative number& H7 w& y; m( }7 W
means no limit (i.e. no yielding).
0 S* B. C. G: v- L% l% h </description>
, e2 n5 l( {8 T1 c' i</property>
6 U7 X0 t1 F. Y<property>
4 O; _8 o# {9 p2 ^% ]7 f% H" c <name>dfs.content-summary.sleep-microsec</name>+ z) d9 q6 z3 T5 g" h0 I% ]
<value>500</value>
* Q- r4 ?* W' X7 f <description>
9 E- s8 J; N$ y+ \, E The length of time in microseconds to put the thread to sleep, between reaquiring the locks
3 N8 o* m' i Y- V! Z" q' Z in content summary computation.
' o4 w! Q8 A# { z/ | </description>9 Y4 V4 Y. L6 V- s, B5 H
</property>2 {3 I6 O- G F g: F6 ]/ S' l! ?3 {
<property>- G4 |9 ~' V5 @7 U: x
<name>dfs.data.transfer.client.tcpnodelay</name>
$ J' h" i1 p8 ?0 |' }0 ~; y+ I, N <value>true</value>
; o5 [: V" W7 Z1 t0 N <description>1 P8 Z- C3 e4 C+ B; E
If true, set TCP_NODELAY to sockets for transferring data from DFS client.
3 e; v- Q- J u V6 ^: e </description>, G. C9 j3 |' H' q g
</property>
0 y% s+ Z ~3 L9 x5 @6 g<property>
9 U, J8 j0 z" t6 J# [3 | <name>dfs.data.transfer.server.tcpnodelay</name>
1 j( a) r; p( } F <value>true</value>
6 ~/ }8 M, t# Q s( _% W& d <description>: K/ j/ I4 h( U, v, v
If true, set TCP_NODELAY to sockets for transferring data between Datanodes.( g7 Z- `, _$ C7 l# c
</description>3 J$ S5 G2 `* f
</property>
8 ^% h. P* d+ D, d" ~<property>1 s0 U; h- K E( K0 K) s
<name>dfs.datanode.balance.max.concurrent.moves</name>
% G6 @0 K( \ R8 d <value>50</value># D5 p8 p% E: p5 g8 U
<description>, o ?0 Y: E S+ a$ P2 @! r
Maximum number of threads for Datanode balancer pending moves. This
7 q8 r' W* b! A9 v value is reconfigurable via the "dfsadmin -reconfig" command.' o7 z8 u& O5 E
</description>4 x/ Y5 I( f7 h( ]
</property>
* I5 v4 X$ N6 r1 i- g! n$ u<property>
3 V, G) s, E Q4 ? <name>dfs.datanode.fsdataset.factory</name>6 X9 g+ n2 N# s
<value></value>
' C+ p1 p G2 \$ h+ @/ Z <description>
0 X% @% R" l) m) c The class name for the underlying storage that stores replicas for a) j/ `, c9 N! k$ P2 C/ b& p
Datanode. Defaults to; S8 f B% ` j H G: ^
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.0 W" h6 O- x `3 S# V" P
</description>1 `, D0 D1 D, y' B) `5 G* v
</property>$ |- C N* H6 E1 `: Q# G
<property>
' {; X$ s6 k! Z <name>dfs.datanode.fsdataset.volume.choosing.policy</name>6 l# ?9 q, s) R; z1 t E
<value></value>: ~, `( A3 O! W% i s) j1 D
<description>
3 R% k2 A# _/ s+ w$ E The class name of the policy for choosing volumes in the list of, |+ V3 j' t- Z" x
directories. Defaults to
' f' i9 Z9 S; Y- y, d org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.
2 `' U' k% p! c7 @# Q8 k. d If you would like to take into account available disk space, set the# f9 A3 W2 p' A! }; n6 r F
value to
# h9 @- l6 N3 j "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
4 Q1 D& F( O) Y6 z9 p, c </description>
a% t* j" r+ E# [. }</property>
% l2 U% J7 S; }5 T5 Q<property>
2 M6 v/ Z1 d7 r- g8 ? <name>dfs.datanode.hostname</name>
- l; f* ^* Y& z3 V: e I. y <value></value>
! Q8 T: M+ R# d: \. C8 O. x/ f <description>
+ @: X0 M$ j9 r Optional. The hostname for the Datanode containing this' g* ]% c }: o
configuration file. Will be different for each machine.
4 g5 p/ I' a& p! [ Defaults to current hostname.
* D! ]. n3 P( W: i </description>4 X3 X" }1 P; E$ c
</property>
9 i( c8 H1 U$ O9 q<property>
4 q, J- ?; e% i" w* r. d/ q: `" E <name>dfs.datanode.lazywriter.interval.sec</name>
" B/ ^# U( Y! t9 w0 H6 u! p4 c <value>60</value>
5 t/ A9 ], x' n" R8 x <description>
/ j1 Y( j0 l. {! w Interval in seconds for Datanodes for lazy persist writes.
% I( I8 B3 [' C# t4 j m) X </description>( u7 R C' Y1 ]) K
</property>% `/ t* J( e! x' t6 L3 Y6 Q
<property>
9 w4 I/ t& N$ u/ L& V k$ n <name>dfs.datanode.network.counts.cache.max.size</name>
% f0 @0 _, U: D/ p3 o <value>2147483647</value>
$ C8 v- U' X( S) m) t <description>
. i$ h! e- |/ c+ f- ` The maximum number of entries the datanode per-host network error2 w) g6 f6 M7 ]
count cache may contain.9 \+ j2 I; L P" G/ f
</description>/ g! [1 u8 h2 A0 d; c" z; s
</property>
, X5 w+ |2 u' ]' S' p<property># m# j8 e" ?6 d6 C, v
<name>dfs.datanode.oob.timeout-ms</name>
, M& L9 d2 x" x& K4 ` <value>1500,0,0,0</value>& z; S4 b: P* R) ^. S3 ?8 c
<description>3 }* I( x4 {4 G7 H5 G
Timeout value when sending OOB response for each OOB type, which are
- Q0 H' I- w( C( }) j7 v OOB_RESTART, OOB_RESERVED1, OOB_RESERVED2, and OOB_RESERVED3," s+ P' e6 e$ [ B Z; B8 d6 Y
respectively. Currently, only OOB_RESTART is used.
: c6 G) Q7 F* \. _ </description>. e; ~. D7 N6 N2 I, P h" K
</property>! n$ _6 b$ Y+ G% X/ L3 T. \2 x
<property>/ J. h) J0 s6 M' k8 M
<name>dfs.datanode.parallel.volumes.load.threads.num</name>
5 b& D. j2 M( w <value></value>
, v& g* w$ q0 t3 @, n <description>
( X2 A2 ?$ W | j! ]+ M. B Maximum number of threads to use for upgrading data directories.. D: A7 q. Q9 P+ |3 |( [& g. i8 X
The default value is the number of storage directories in the6 q8 }/ t& L& K- H5 b) N7 Y
DataNode.
; V% k- O4 e+ A, V2 K4 k( {* Q7 _ </description>
2 N7 t+ o. r/ Y/ k* F. g. b" l+ S- d</property>
# B. n7 _1 o1 s6 N. H& R* B7 L<property>
" o% t) E1 _. ]- s0 z <name>dfs.datanode.ram.disk.replica.tracker</name>
4 F; R: D; U( B6 f* e- Z3 n <value></value>
& Q! J8 U5 h9 b, {* }4 g <description>
5 L [) M: W" W' ]/ g Name of the class implementing the RamDiskReplicaTracker interface., n) m; ?! q2 |$ E" X& H; Z
Defaults to
8 T- l3 S' d2 p! c org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker.; H" m- o/ @; i" v
</description>0 h# m8 S( p1 J
</property>
# j' k: z8 t2 L<property>" M4 n0 d0 G5 H) Y$ G# T
<name>dfs.datanode.restart.replica.expiration</name>2 i8 a$ H* `( u# s2 F" E
<value>50</value>5 e' f. I4 }6 m0 C" i+ ~( T' `9 ?
<description>
- L7 S V5 V$ Z During shutdown for restart, the amount of time in seconds budgeted for, n( H, k+ e) z# c2 v! o2 C0 \' O
datanode restart." F: ~2 H7 C6 Z
</description>8 D8 y( \! q$ |1 ^: ] J9 e; X; K
</property>
$ f" \; ~0 ^: m- l<property>7 h1 I5 `7 j5 U* @
<name>dfs.datanode.socket.reuse.keepalive</name>5 x3 d( E* h8 ^, V
<value>4000</value>
4 ^: l& \8 g* s" W4 I+ k( u4 B <description>
1 Y5 L8 W! `" T7 G) O The window of time in ms before the DataXceiver closes a socket for a O6 f. L2 f5 a1 |* g
single request. If a second request occurs within that window, the% c3 n6 F6 X4 Y- b4 }' @. D$ s
socket can be reused.( _$ U" T5 i) z$ f( o" H; x$ p
</description>0 e/ N# M* C# d, o, x/ M. p
</property>
2 _& n; Y; M& l1 [4 l<property>! N5 S: _% f/ P) B
<name>dfs.datanode.socket.write.timeout</name>- i7 \3 n$ P( M0 @; D
<value>480000</value>$ q/ E& e$ Q) [" r
<description>- q6 E3 y% m ~" ]- J
Timeout in ms for clients socket writes to DataNodes.! b" `/ }* t. Q4 _& l, }
</description># N2 a' ^% u! g1 ~ c
</property>
! Y3 ^; c; ]3 j1 j. c' W# E1 |<property>
! ^- S. }# f' x. f; Q, U0 Y+ u <name>dfs.datanode.sync.behind.writes.in.background</name>% a" q# |# A, J9 M' \1 }3 K3 b
<value>false</value>& O! }4 X M# V7 D
<description>
& j' T! R+ S t8 ^; {4 ]9 ] If set to true, then sync_file_range() system call will occur+ Y5 d+ q( Z+ B7 L; ~
asynchronously. This property is only valid when the property7 ^8 R. y! K) O, H1 \! R
dfs.datanode.sync.behind.writes is true.
! d! @! c6 ^* L5 X: Y r0 ] </description>
5 T+ O1 {; {* @5 ]; o</property>6 h3 S' g2 ?* ^# V; r
<property> y% z8 X& a6 y9 u0 p+ ?) o
<name>dfs.datanode.transferTo.allowed</name>; x% t/ u. h( [- `& Q
<value>true</value>/ Q3 F& y" D# z
<description># \: r* G" b4 U2 u( W9 w
If false, break block transfers on 32-bit machines greater than
7 f S8 B. [7 o4 b- ~ or equal to 2GB into smaller chunks.
& ^5 i$ |/ S1 Y `, q </description>! c; D l0 ?, t1 n- J3 K
</property>
) i5 h. @/ `# z1 B<property>
7 `* O! P% U6 @& v2 g# x8 h1 Q$ v <name>dfs.ha.fencing.methods</name>
4 ]$ u! N( O4 j <value></value>1 R, e3 z. g5 l% N
<description>
; k8 T M4 a" I4 y9 [ A list of scripts or Java classes which will be used to fence
, ~- J" [) N( X: K1 M5 r the Active NameNode during a failover. See the HDFS High1 f3 s4 a; b- u6 Y# }, ]% g# _
Availability documentation for details on automatic HA
3 @4 }/ G: s& E& a8 R7 \) V configuration.
. w; N6 k8 o! }: S# h8 R( m </description>2 _$ \' V# F4 A( L O8 T5 P- O
</property>
# c1 {* R2 I) E+ L<property>
6 W7 h/ ^1 f* R6 Q1 y <name>dfs.ha.standby.checkpoints</name>
`& C& b0 \ f+ d4 q <value>true</value>
" {( J# `+ V* F1 Q <description>( x Y8 X @. n$ ^" y ]6 M# z P
If true, a NameNode in Standby state periodically takes a checkpoint% g* J" M: |. i4 {# Q0 R3 U
of the namespace, saves it to its local storage and then upload to
6 `- @1 [1 N4 l& g) z6 ~; E2 z the remote NameNode.
% `% e8 k8 g4 M7 C </description>
) O2 w- n/ ?5 s5 B) s& l</property>
/ A% U+ j! Y& H& L) v<property>- l2 ^% A0 ]7 K% O* X& U3 ]
<name>dfs.ha.zkfc.port</name>
' k, R1 [1 u$ D' G <value>8019</value>
3 G' h9 r. P* x% V2 b3 D* u. V <description>" C7 f! A7 n9 i3 X1 K2 H' M- z
The port number that the zookeeper failover controller RPC- _* u Y' @6 Y2 ~1 V; h C$ Y
server binds to.0 ^0 `; }8 w. X5 }) N
</description>
' S5 a4 H4 O- x5 K6 t6 A</property>) A1 R4 |. J; r2 h9 T/ |
<property>) l2 ^5 ]2 D, b9 f. K( K
<name>dfs.journalnode.edits.dir</name>5 _3 g" K; o( q! ]4 u! H3 V7 D
<value>/tmp/hadoop/dfs/journalnode/</value>/ N" Z, W4 b4 m y s* X
<description>
0 ^+ |1 Q0 `# M The directory where the journal edit files are stored.. `2 U( W3 U, v! o+ C! x3 y
</description>. l) J3 d# K) N& J+ J
</property>5 N, d! M: a6 x; ]+ |% ]5 m
<property>
* N. D6 T# \4 R9 M% P7 Z( q0 G <name>dfs.journalnode.enable.sync</name>" }$ u. H7 E I+ S" W
<value>true</value>7 H K( u, K2 c; s E% e6 W* ?
<description>
0 v: y$ L! _5 J0 [/ S' _; Q$ h" o If true, the journal nodes wil sync with each other. The journal nodes. V* t, `" v( a$ K' a. G
will periodically gossip with other journal nodes to compare edit log: G; A. g% {! H) o1 I
manifests and if they detect any missing log segment, they will download
/ b% W( H7 X2 n5 I5 [ it from the other journal nodes.
s/ T5 i- N) z </description>
: K8 L% a& a2 [</property>
# W( W" `9 h. K) h% h<property>
' A0 I! `6 {6 p <name>dfs.journalnode.sync.interval</name>
, o+ D/ `2 Z8 w5 |( X <value>120000</value>4 U5 ]! V% N- ^
<description>1 o5 w0 O" @/ @6 k! ^) h
Time interval, in milliseconds, between two Journal Node syncs.- }0 T. X: S! x0 {# q0 k
This configuration takes effect only if the journalnode sync is enabled
y$ t: e( ^9 K0 ]# f5 Q. O by setting the configuration parameter dfs.journalnode.enable.sync to true./ \" Q, ~) O; y! I4 B' ^
</description>
9 B6 o6 \1 }+ I, {. t4 s. r</property>
, L6 l9 l6 x9 x Q3 f/ k<property>9 Z7 ~1 m5 m2 F6 r, _0 m/ O. l4 q
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>
* K& c& ?( S/ [5 Y; }5 w2 Z& H* S <value></value>! w7 a; p8 \, Z9 L3 k
<description>3 F9 q z. E+ d) e3 h
Kerberos SPNEGO principal name used by the journal node.
& m+ T7 w! S9 y4 u </description>7 ]; ~5 }* K& L2 P
</property>
: f* |& R8 }* V( N7 A0 D& a8 M<property>
/ k! L5 O. s( p; i" j0 m% j <name>dfs.journalnode.kerberos.principal</name>
M. H: n) ]: u2 @5 i <value></value>
/ C+ F+ K: a0 o) @ <description>2 S5 ]" y3 T Z' |7 u
Kerberos principal name for the journal node." j3 T: K" @7 S6 X% b- P
</description>8 w1 U* P! f1 l3 J K' Y; o. z
</property>8 t) Y1 @" K% O e i* K
<property>; R0 [) g6 i# u" _4 E- V7 A% b
<name>dfs.journalnode.keytab.file</name>7 r' n/ Q& P5 s+ d2 N6 {/ H- r
<value></value>6 S- q- f- e) t: n
<description>+ d, h7 B' o5 A+ S1 f z& m- q
Kerberos keytab file for the journal node.
3 ?" D, g/ T# y% q# V8 t M5 O5 g' Y </description>7 B* A0 y; S9 r) k! ~. h5 N3 w
</property>) S- G3 F m. x% l& D
<property>
1 T7 v X$ d( B <name>dfs.ls.limit</name>( K o3 r" L# i' A8 w
<value>1000</value>5 X; A7 Y: a6 v5 u
<description>! N$ }# a, J- S1 _! G
Limit the number of files printed by ls. If less or equal to
\' i9 p4 m7 s% ^5 U$ h4 |: ` zero, at most DFS_LIST_LIMIT_DEFAULT (= 1000) will be printed.
8 } g6 g8 V5 b8 S) Z4 c: q </description>6 P6 J, A6 E% W2 p$ n1 O! Y' j$ T) x
</property>
- r. E3 P/ j: u. F3 e<property>
; b* Z1 d& i9 F" r1 E <name>dfs.mover.movedWinWidth</name>
- Y& V- Y0 @ G5 R# J) g, S <value>5400000</value>
5 u( m6 \5 \" L1 k; s6 }/ m0 v% G <description>7 P1 m7 I/ ` M$ j2 u5 u" a$ d
The minimum time interval, in milliseconds, that a block can be
8 D; \2 ~# M, G8 E moved to another location again.4 L$ A. K+ ~1 y6 r
</description>; T$ k9 s: E% B
</property>
7 `6 V# l5 j3 C<property>! G; Z0 |! {, T( Q6 c2 A4 x9 ]
<name>dfs.mover.moverThreads</name>
* M5 u+ ]9 x5 z3 x/ b! r9 m" F <value>1000</value>2 ~2 K/ Y7 l! D; x+ `+ X/ o% Q+ @
<description>
+ P: O2 q( U4 W& W* _- e. Y Configure the balancer's mover thread pool size.
8 w0 Z, J( a" u: |+ \7 s' L3 c </description>
/ G7 V& Q) k5 K5 r( |4 L( m</property>
; e0 |! _% `4 u" n<property> E. @* E2 y; F$ K$ d4 p- ^
<name>dfs.mover.retry.max.attempts</name>9 o+ ]! J7 ~$ K8 J5 X4 f+ r
<value>10</value>% y: m5 C% K( E% a7 `
<description>
' q+ i5 P5 @) Y1 H+ p& h7 i The maximum number of retries before the mover consider the1 M/ I- p( p% j- N# l. P/ _3 B
move failed.
9 u4 Q J# \! ` </description>: G5 K H7 ?$ {: [' R/ x
</property>- f5 a0 y1 c! j9 j% p
<property>
& q# N0 o' @: e6 E/ \$ D, P& e <name>dfs.mover.keytab.enabled</name>
& }1 c$ X( q* O$ u( T. K <value>false</value>
0 P3 S- q _ B0 h4 a9 t3 X$ R <description>5 D3 W1 o% e* u( m
Set to true to enable login using a keytab for Kerberized Hadoop.) l+ a' L6 X: f% X2 h/ u
</description>5 U; g" D2 b4 J/ c
</property>* h% w, O4 C9 U5 d
<property>
% j4 x4 t" X9 d2 }5 Q2 G" K <name>dfs.mover.address</name>1 w" j" W" N- |# o; p. m
<value>0.0.0.0:0</value>
' W7 F3 U7 n! a/ s" Y x( V <description>
" h' x' X A8 K( a The hostname used for a keytab based Kerberos login. Keytab based login% z* d; y, g9 E# b1 r
can be enabled with dfs.mover.keytab.enabled.
. `& B6 b5 ~/ ?$ [ </description>7 V$ \, p" H9 W7 [& U4 q+ ^
</property>
& Y: i- ]- j! l3 R<property>! n: B9 Y; a5 E
<name>dfs.mover.keytab.file</name>
8 R) `5 S. a- y4 \- S/ z% @3 h <value></value>9 R1 j' u0 X$ I5 u3 W
<description>+ V1 b( O$ W! ?; {9 N+ @: z
The keytab file used by the Mover to login as its
5 Z: R* y. Z* q- a% G1 [ service principal. The principal name is configured with& T) P( z. H5 Z' k
dfs.mover.kerberos.principal. Keytab based login can be$ O% N) e! @" k; z, T" }$ u5 l5 E
enabled with dfs.mover.keytab.enabled.& w2 f0 d) K5 M/ g
</description>
3 E' p$ ?/ _! a. Q</property>
3 g2 c! x' T$ u+ y( p& U<property>3 B2 A. n7 l" m5 K" i. X. }4 ]2 `
<name>dfs.mover.kerberos.principal</name>
, G i1 s7 g& y! k3 b <value></value>
0 u6 S% p7 K5 L" L9 Z' U <description>
2 ~: f8 M# X. ^0 _: j The Mover principal. This is typically set to
: m# ?) T" L# T1 [/ [ mover/_HOST@REALM.TLD. The Mover will substitute _HOST with its
) H3 H# |9 I& r. B* x6 Z6 X+ L own fully qualified hostname at startup. The _HOST placeholder3 u. S4 \% P/ J
allows using the same configuration setting on different servers.
6 u; ?; D- S9 n+ o: O Keytab based login can be enabled with dfs.mover.keytab.enabled.
5 Q; U' Z" I" C" t. c </description>9 i' p! S. [7 k; e: |1 i
</property>9 I+ O- ~' w& Y: E/ N$ y
<property>! K/ W' |! i+ g- ^" g
<name>dfs.mover.max-no-move-interval</name>9 A# y$ t$ n! n* c- O1 T
<value>60000</value>
0 u" K6 C9 o1 I3 a <description>: q T% w: L/ B/ y. n3 P4 R. |( i
If this specified amount of time has elapsed and no block has been moved
! x+ l; `+ z: @; P4 W ?' J out of a source DataNode, on more effort will be made to move blocks out of: W1 `4 t# q2 l4 h9 k
this DataNode in the current Mover iteration.
" K8 l: U/ Q0 p) i1 a' F6 R </description>9 h S4 g3 |1 S# ^
</property>
" _& ]: u, R8 n! V. e2 F<property>
. N1 Z& |: G* V' d- e% O4 I <name>dfs.namenode.audit.log.async</name>
) g# q7 {5 R4 E6 t8 L1 ` <value>false</value>, v% }6 J* s/ D% L
<description>
3 @3 s$ `" X) g2 Z) k- P If true, enables asynchronous audit log." i }- c( ^& t* |+ J6 D H0 ]
</description>3 u3 P4 p$ X" D2 [8 b. G+ @
</property>
/ f8 c$ M( P% a, n<property>
. x8 }2 n, _/ Y7 ]1 O) G6 L# S' s <name>dfs.namenode.audit.log.token.tracking.id</name>, y& X* u. ^; S; f( N
<value>false</value>
/ [3 X6 b: V3 p- h. q <description>
4 I5 E: d' {% B If true, adds a tracking ID for all audit log events.
' o* d7 w! H- | </description>) Y8 D* K& x1 r* B. |
</property>6 ]7 K$ _* e' q
<property>& d4 S |9 ]4 r5 ~$ Y' U, ?9 v
<name>dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction</name>
' k, o! T9 H& Q" {$ [7 a) ~ <value>0.6</value>, Y5 y) t# Z# j
<description>
" w- _2 `2 `7 t; Y Only used when the dfs.block.replicator.classname is set to
: K+ ?; u' k7 c' e' y. z org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.
, q0 V+ d: T5 G9 Y* K* b Special value between 0 and 1, noninclusive. Increases chance of/ E% l1 R: F4 N) h( x7 s4 s) k
placing blocks on Datanodes with less disk space used.
- }. H! M, n4 K9 a8 N" S </description>3 ~, E* i# [3 g. p2 R9 [9 A
</property>8 I" A1 t6 w+ ]
<property>
5 @* x8 k9 Y5 O8 r) _7 z/ Y <name>dfs.namenode.backup.dnrpc-address</name>) j6 F# Y+ s+ _$ u4 I
<value></value>" l$ F0 r6 a% w f; t5 o p: }
<description>, E3 g3 K# r3 l
Service RPC address for the backup Namenode.
3 a* p$ K* b: k, E# q. P </description>
2 _4 @) u$ I0 i6 D D5 x</property>
' Y$ y H. z6 Z2 z' E8 p! b<property>& R3 g2 D6 I { ?7 z: p5 X
<name>dfs.namenode.delegation.token.always-use</name>
$ T+ O' |8 f* s" {# ` <value>false</value>
" @, ^$ D0 p# {+ G: B- f$ L0 u7 S <description>6 k) q+ D; ]+ c, k' N3 Y
For testing. Setting to true always allows the DT secret manager
6 y. @9 I8 n* u# W. U5 W to be used, even if security is disabled.
6 C: ^7 g% Q# ]& k' V </description>
1 Q) k0 w0 h9 O</property>
$ v7 i- O3 ? |) j& v2 P<property>& G; V9 }2 z' N
<name>dfs.namenode.edits.asynclogging</name>
3 P5 A+ s( a/ m" _2 _4 B' S+ A, g <value>true</value>. r2 ?5 A/ F6 ~9 ]6 v* D8 r
<description>
% B I+ W& C9 {% H' S' s If set to true, enables asynchronous edit logs in the Namenode. If set* E5 ]4 |$ o; [
to false, the Namenode uses the traditional synchronous edit logs.
+ J `: v' \/ x# Z" V5 a </description>
8 q2 R& T: \/ v- m( w5 v* z</property>3 \6 n9 o% K# g9 g7 x2 b% x6 o
<property>
% x! q/ f' {$ Q, e+ b' U6 o4 d <name>dfs.namenode.edits.dir.minimum</name>0 b8 Q& F0 V7 X2 Q
<value>1</value>1 K- C+ C' a) V' m# [- G. m+ O
<description>
6 P$ j, r' r: A% I9 I a e dfs.namenode.edits.dir includes both required directories$ f# J$ Z* {0 g, U
(specified by dfs.namenode.edits.dir.required) and optional directories.0 x' i4 |; L9 l' z4 G% L
The number of usable optional directories must be greater than or equal
- N' G# K2 X& Z to this property. If the number of usable optional directories falls+ D6 f8 x- H. M3 D/ o
below dfs.namenode.edits.dir.minimum, HDFS will issue an error.; Y/ o( ^. ^. x! b' q- X- S% R
This property defaults to 1.
# F9 G: W0 M& |3 @- \, @) r$ U5 A </description>4 C. I, Q }3 M# Y. B! l
</property>
' E0 Z( L! ?2 H8 n<property>
& N% y' M% B6 i9 F# Q% o4 h/ Q2 T <name>dfs.namenode.edits.journal-plugin</name>
- U6 V& x7 ^* _0 U <value></value>
: o$ g- x+ D- Z& W4 g' O <description>
) }! J1 t) A: J N When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir,' I6 f# o, _4 R @8 S- K. w
and it encounters a URI with a schema different to "file" it loads the# _+ o6 h! B$ ]
name of the implementing class from( l) p) G/ D) q8 q; v7 ~
"dfs.namenode.edits.journal-plugin.[schema]". This class must implement, G8 R7 i; |# Q) b
JournalManager and have a constructor which takes (Configuration, URI).5 }- n' O/ z+ w& L/ x5 s7 d
</description>
9 p1 e7 @7 ]3 y$ K: Z6 p</property>3 X! j$ X" V/ {- R V! o
<property>6 K2 @4 B. I0 i2 u8 a1 Z+ p+ {( j
<name>dfs.namenode.file.close.num-committed-allowed</name>
% f/ i+ j- R9 {. n; e <value>0</value>/ _* T8 {0 ?7 g
<description>
! p9 A/ Y! h4 Y; W Normally a file can only be closed with all its blocks are committed.
9 W4 O! C0 }1 U1 x, B3 F' }- ?) Y% k When this value is set to a positive integer N, a file can be closed
/ Y0 Z' i' O- F3 y9 r when N blocks are committed and the rest complete.- {" X! m. ~7 b1 O5 g$ b+ ?- b
</description>2 Q H: L& T% z6 m9 z, i' {. b! J# x4 `
</property># j% ~4 t2 \; Y. Q
<property>' s4 [5 H; K4 T3 d% ^
<name>dfs.namenode.inode.attributes.provider.class</name>. m' J' K( b* j
<value></value> x& I# X/ l( \0 ?, \1 Z7 X! n" p
<description>
+ ~' M3 k" W# o8 h1 W Name of class to use for delegating HDFS authorization.% p3 D; I9 W6 C$ i7 N
</description>
5 }! P1 P7 |; s4 G8 I) B</property>
) S: G( Y, x) G6 c5 W _* W<property>5 X6 k5 [1 [$ q' v
<name>dfs.namenode.inode.attributes.provider.bypass.users</name>$ A0 R& N8 G2 t U6 ^
<value></value>
2 N! J" p; b* ] <description>1 V5 n L+ w2 O
A list of user principals (in secure cluster) or user names (in insecure
3 W0 u* a4 ^1 _' b# ^7 m cluster) for whom the external attributes provider will be bypassed for all
) f4 B+ ]) f& Q# d6 P operations. This means file attributes stored in HDFS instead of the9 f6 F0 C7 ]) S& _
external provider will be used for permission checking and be returned when
' c0 _+ v6 }0 [6 C) F: q requested.
, s: E o& t4 {; M </description>
: _& ]& \, W. e) x</property>; R9 e7 x1 F* d% n) L) d
<property>
7 E2 W/ m* v+ u <name>dfs.namenode.max-num-blocks-to-log</name>, I. @- C, N6 _& L0 g% W {
<value>1000</value>2 V0 ?& r( v7 j8 B3 e0 Y
<description>0 F! ^! R3 t; ?% [
Puts a limit on the number of blocks printed to the log by the Namenode- q7 y* x8 q# L1 O
after a block report.
/ _% M- y! P) E" t, L P1 k </description>
/ t6 a$ ?3 [2 b( ^! |) y</property>
5 X# n1 A, h+ B8 k# H' Q<property>
1 m: \# h, m! z& I8 r8 L9 W; m$ _ <name>dfs.namenode.max.op.size</name>& D1 D3 m' D8 T: ?0 ~: O
<value>52428800</value>
+ q B9 ?$ U) \; R5 L <description>9 W- v" c) G( C4 ]& g2 T, {
Maximum opcode size in bytes.
: z+ ~' b6 \" ~ </description>5 u+ R% v1 p+ Z& s& L2 m& C0 O, a
</property>
. D" s- L/ U, b* s# Z2 k- q<property>
3 _4 D$ }5 T6 h& w1 x0 c <name>dfs.namenode.missing.checkpoint.periods.before.shutdown</name>: u& y9 q! @* P! c: Q W
<value>3</value>
7 Y8 r1 s( g! S# ] <description>
, p! p. H9 ^$ W3 e/ g! f The number of checkpoint period windows (as defined by the property+ P0 j# M+ L. L: a
dfs.namenode.checkpoint.period) allowed by the Namenode to perform
# T0 U0 u5 o6 j5 {: k- ^/ } saving the namespace before shutdown.
/ q: |* L* u/ K b- t4 P </description>/ [' S$ `( N- K" B7 K
</property>) _+ ?* o! S0 i
<property>
' a9 X2 F$ X$ K3 {0 e- G. r <name>dfs.namenode.name.cache.threshold</name>
5 G7 \1 G4 a) {& @ <value>10</value>; h b8 E- B9 p* z3 F2 P7 S8 S) B
<description>% d0 @6 [4 J7 a: B7 ?) E
Frequently accessed files that are accessed more times than this
! b J' B& }) H threshold are cached in the FSDirectory nameCache.
% b" i! k) B& B4 D3 C& u </description>* G+ j O7 P) t# c$ o3 o& J
</property>
6 E. }" C6 V$ I6 O$ ]<property>
" Q: m9 A4 f* J2 I1 P <name>dfs.namenode.replication.max-streams</name>
$ `& |; f$ \1 _, f- U9 k; t <value>2</value>( R: A) @! `8 T. g
<description>
9 s5 G$ K j3 O Hard limit for the number of highest-priority replication streams.
3 G7 J3 h6 T" t W8 D2 J5 U </description>
5 d% L( |5 a9 U/ [ n' e1 \</property>9 ?3 E( s0 _# X1 P1 R
<property>
& \+ Y+ T; a; R. f- {' d! ?5 a <name>dfs.namenode.replication.max-streams-hard-limit</name>
6 K' D2 _; Q, T% B1 T <value>4</value>
# s' C E3 r5 I1 m2 o6 a8 Q <description>
- ?6 T8 ` s7 y Hard limit for all replication streams.
6 T! {: j2 c8 j0 y. N </description>1 Z! `3 O- }) x: x1 h
</property>8 W+ X7 w3 R, B- T
<property>( c; f X8 D6 s: ?1 D5 T* {
<name>dfs.namenode.reconstruction.pending.timeout-sec</name>
* W- x7 ]. _4 V% y% p <value>300</value>: B0 q, f5 n; }1 g( G
<description>5 S l+ j, s5 d9 U+ P( ]
Timeout in seconds for block reconstruction. If this value is 0 or less,
( s) `) V7 C, l* _6 t+ V" Y* E then it will default to 5 minutes.$ e% X: R. r; r, P6 ]0 g8 ~
</description>! b( @$ u# Z" K& z4 D; t
</property>
, g5 k2 X" q) E1 i1 k<property>* P* M. R, l& L( K7 q$ t8 ]" z
<name>dfs.namenode.stale.datanode.minimum.interval</name>
5 e* g6 s, P/ K4 V' R- q( M <value>3</value>1 v' o. F- o% e3 L; A/ u! A
<description>" y& S4 x- z- a! ^
Minimum number of missed heartbeats intervals for a datanode to: O1 O. @- A$ r. y; h# c9 [5 K
be marked stale by the Namenode. The actual interval is calculated as8 ]4 D/ v0 O3 ?- \) }, T% O
(dfs.namenode.stale.datanode.minimum.interval * dfs.heartbeat.interval)- c4 I& k4 Z7 w1 f1 O
in seconds. If this value is greater than the property6 a5 m m2 b. t5 k
dfs.namenode.stale.datanode.interval, then the calculated value above
7 g1 \9 V0 ~# l$ c6 M" V. H is used.. u5 s5 S3 o# t7 p( @/ |
</description>
) ~! k0 c% i& A. u</property>& j% @' T. W/ v1 N; T- A4 }
<property>
& |0 B2 r+ ^* j. ~' V: G <name>dfs.namenode.storageinfo.defragment.timeout.ms</name>
) F* O9 |9 R( o4 j# k) Y <value>4</value>" B4 N( e1 q1 h: d: m: U1 r# l% G
<description>
7 o% k0 l5 ^) D# D Timeout value in ms for the StorageInfo compaction run.7 p, ], T! Q' I$ @; N
</description>& |) M I% a( o9 u! u# R
</property>9 G: \$ g8 f K" ~* u
<property>
! g$ C: c9 I* ~, y8 U/ l <name>dfs.namenode.storageinfo.defragment.interval.ms</name>9 \ ]- M) Y! `5 M
<value>600000</value>" y0 h: b$ P: n! S% C7 \+ u
<description>
, A8 o! }' x# q' O% n$ a8 o' d. K: B | The thread for checking the StorageInfo for defragmentation will( V* ~4 q( V W A
run periodically. The time between runs is determined by this$ d) }! G( i: u+ p' \7 F% L
property.6 K& ? z8 W) M9 J: E% ]1 h
</description>
: a6 E# S# K7 n0 C. M# N$ ?</property>
* o! x/ d( x# e' k6 C<property>
' f- [, d6 C3 U <name>dfs.namenode.storageinfo.defragment.ratio</name>/ M0 z9 x6 O& U- b$ n6 a! r
<value>0.75</value>9 ^& I1 J8 b2 n- | s
<description>
! c) ~9 L$ r: R8 p0 I4 P: E9 \0 B3 w The defragmentation threshold for the StorageInfo.4 P3 v1 \, N$ m( C# L
</description>( N" u) ~; D: }6 w* l- t. @2 I& M
</property>: k }1 i5 g* \- o) H
<property>
) |$ b0 @4 h& B1 M# g <name>dfs.namenode.snapshot.capture.openfiles</name>
( s6 Y0 Y2 Q, n. g <value>false</value>7 m- d# L2 B1 F9 p9 t
<description>
! r) I% v# Y& |% h If true, snapshots taken will have an immutable shared copy of
6 S$ F6 j( e. b1 q1 h: o the open files that have valid leases. Even after the open files: ^! |& }$ f7 p3 b
grow or shrink in size, snapshot will always have the previous
; o- A/ r c% p4 V. t point-in-time version of the open files, just like all other
3 m; c% O7 \/ Z3 C2 Y; d closed files. Default is false.
i- E1 Z, q, t3 d) c- B* } Note: The file length captured for open files in snapshot is
- J- }2 {8 _, r7 O- \; s0 \# R. a4 l whats recorded in NameNode at the time of snapshot and it may: `" w2 n8 r2 k: Q% P3 K% p9 J7 ]
be shorter than what the client has written till then. In order% K+ \: b' c+ H) Y( ~
to capture the latest length, the client can call hflush/hsync
% k/ j* m& V4 K with the flag SyncFlag.UPDATE_LENGTH on the open files handles.4 j; X, v5 D& X2 [$ e! p
</description>
' ?. D* s b3 x5 K" W3 F o1 {</property>
/ u% |5 ~1 ?3 e" q# d. V<property>: d8 I9 w5 Z/ @* L1 y Q
<name>dfs.namenode.snapshot.skip.capture.accesstime-only-change</name>, P) W/ z3 @8 ] D) b
<value>false</value>
$ g3 X0 u% g' f: O <description>
2 g- b$ Z' X- |5 L. b8 X# E If accessTime of a file/directory changed but there is no other' X3 a) Y0 }; n5 h
modification made to the file/directory, the changed accesstime will# k5 q! C6 }8 w1 F
not be captured in next snapshot. However, if there is other modification, @3 n; }2 w( a- K/ e
made to the file/directory, the latest access time will be captured+ C+ {% ^) S# C- H
together with the modification in next snapshot." o9 i V5 T! D* G( g v
</description>
. M0 ?0 ]( P% G( m</property>
( O: a1 p2 b% h" _9 {" X R<property>
* N4 v0 T, K# [1 I <name>dfs.namenode.snapshotdiff.allow.snap-root-descendant</name>
5 Z& [' u! \3 K8 R <value>true</value>- I& {, l g; `3 \' n
<description>* L2 ~- F& Y+ \: H+ M' i2 \/ a- }
If enabled, snapshotDiff command can be run for any descendant directory/ b/ B# k) k# Q7 b z7 q" k$ q& s
under a snapshot root directory and the diff calculation will be scoped
3 _5 U9 q; G& m8 S( z3 ~ to the given descendant directory. Otherwise, snapshot diff command can
& T1 q* n3 J% {! _. r only be run for a snapshot root directory.3 V( m9 B- F8 {7 o
</description>
& ~9 u- P# _: l: D6 ~/ g</property>
& J1 p6 Y! Z3 t' x3 j- g<property>. f( L8 i2 X P
<name>dfs.namenode.snapshotdiff.listing.limit</name>
$ k! f. e- C% c2 C$ ` <value>1000</value>
1 ~+ X+ D: @. M3 z <description>
' G' r" s! X5 y Limit the number of entries generated by getSnapshotDiffReportListing within
# K! Y3 N9 |0 g one rpc call to the namenode.If less or equal to zero, at most3 b* n& T# U4 U& T0 C1 g0 e
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT (= 1000) will be sent. i# f) a! Q' I! ?7 m5 g
across to the client within one rpc call.1 X4 o% {9 u( z7 n9 Y% l1 u
</description>( K- i/ j, p( t" C0 g& z3 \
</property>
% w! e8 n+ p' K, e& y" F# O<property>5 @9 a0 Y% G) E* E. w0 S6 o9 j
<name>dfs.namenode.snapshot.max.limit</name>
* y% U* ^7 y" D5 _2 Y, e <value>65536</value>
v6 S5 V) C* W$ X) Z; U6 Y& R <description>
* ?* z# M1 m. T3 o d2 t Limits the maximum number of snapshots allowed per snapshottable1 `1 h: r' n* N, E+ d9 t5 b, l. k
directory.If the configuration is not set, the default limit* I+ B* |6 f' \; \3 W
for maximum no of snapshots allowed is 65536.
; A* z( T, ]) Y3 X; Z! a </description>
3 G! n* V( f# B4 j" [</property>
3 {$ o8 f# B/ G% J0 |<property>$ G( M2 V1 d$ q5 C$ R* \7 F
<name>dfs.namenode.snapshot.skiplist.max.levels</name>, _+ X# a. S* `" C- p
<value>0</value>
0 A5 }1 n# |1 f; ~& _ <description>. r( ~% T1 _9 D' q, G" v
Maximum no of the skip levels to be maintained in the skip list for
& Z2 }+ E, J& }0 m storing directory snapshot diffs. By default, it is set to 0 and a linear
0 \; ]& t; ~4 Q6 o# P, x( j# z list will be used to store the directory snapshot diffs.
' t" F* f6 W9 H4 Z; K U9 f </description>( M+ Y" i m, X. }; x
</property>
% \% t+ j4 k( S' j9 W<property>
/ M; D0 c5 R } W- y <name>dfs.namenode.snapshot.skiplist.interval</name>
! Q. D* U7 c8 p! o7 e" D1 d <value>10</value>
) w- ?3 n5 S5 W& `7 O( |9 Y5 t <description>0 F& S# f2 b9 e: s) k* ]
The interval after which the skip levels will be formed in the skip list
`0 X# ~1 Z* i" X- ~5 q for storing directory snapshot diffs. By default, value is set to 10.
7 ]# I- N, V8 w( J" p$ O </description>
; V" Y: ?+ s3 T% j</property>
0 e a. C0 p; ~) Y. G0 t<property>% b5 t/ m. t9 u2 }& W0 R+ S
<name>dfs.pipeline.ecn</name>
3 y9 v8 O8 r$ V: I3 ~/ g <value>false</value>& ^, N9 o% S; i
<description>
- t; I: n0 M$ t- s" O2 W7 q; m' X ~# o If true, allows ECN (explicit congestion notification) from the
3 i; G7 K7 B& N) [" c5 M0 ? Datanode.
' F, Z( _3 k: p O/ e6 G- F </description>
$ I0 N& g% C/ ?! E( j4 }4 o! B</property>, M% D4 k- r5 I& V1 d/ ]
<property>, }/ Y2 B) {0 c* O- h
<name>dfs.qjournal.accept-recovery.timeout.ms</name> n: u$ y$ r" K
<value>120000</value>0 r$ W& ~& @8 E
<description>3 }1 z8 |, t) r) ]) _+ U! c8 n
Quorum timeout in milliseconds during accept phase of" q; u# w6 ~2 [( y& _/ @
recovery/synchronization for a specific segment.
4 h U9 l# N1 p9 } </description>' B }: f* u( k# H8 A; [2 i) K" q+ @
</property>. t' K C1 Q& L7 M$ X; A" {
<property>1 ^6 P( e3 [3 A" z' B
<name>dfs.qjournal.finalize-segment.timeout.ms</name>
9 @; y9 J) j- w! t) i <value>120000</value>
$ ]3 y( |# X; l- \7 ?8 D0 |7 @5 p <description>) {' Z8 ?. p0 ]! F& q- h7 G
Quorum timeout in milliseconds during finalizing for a specific
" x8 T8 j0 n, O$ ^$ x0 K4 v segment.
o2 ^, i# ]: F6 Z. `/ l </description>* ~3 H- p7 c- b0 D7 ^) o
</property>2 h; I" E+ i4 K4 E4 E
<property>5 a$ \4 W8 {5 L
<name>dfs.qjournal.get-journal-state.timeout.ms</name>! z" B* C2 M$ F8 D
<value>120000</value>
" t0 c4 O9 j; w4 B <description>1 O t4 P! b# k2 ~- i9 Z/ g, I
Timeout in milliseconds when calling getJournalState()./ C& l: O' U8 _. q
JournalNodes.
6 |/ p6 {5 q/ {7 T* H# z, X </description>
6 n. s$ o' Q; T7 T</property>- p Q+ K+ ]1 y6 F
<property>
! j F( r( R/ }6 J <name>dfs.qjournal.new-epoch.timeout.ms</name>
6 o- r5 i1 }1 V9 ^ <value>120000</value>
" }- t: v5 w1 v <description>
, k5 u/ u |6 G+ Y! q" h `9 A Timeout in milliseconds when getting an epoch number for write' k P2 P+ `) C4 ?0 R8 R7 z. m
access to JournalNodes.
: X. p b% A' u" L% E8 e </description>
& f% G# a4 j/ g</property>
. Y' z; g8 p' @<property>2 \0 E( o. E+ ]
<name>dfs.qjournal.prepare-recovery.timeout.ms</name>2 e0 \2 L! o' u& X
<value>120000</value>
! [* A% `% q9 ^( c9 H& w4 i <description>4 l8 t$ L" F7 N! \( u P3 K
Quorum timeout in milliseconds during preparation phase of7 _. F8 b |; d; O# e- Z, g+ {
recovery/synchronization for a specific segment. v7 {6 I M3 T7 l/ u
</description>
( a/ s& X, P$ ?. i; l</property>
6 ~& p4 A/ m/ h- f<property>
# _3 g* E- O4 ~2 e! k' q$ P <name>dfs.qjournal.queued-edits.limit.mb</name>
. S9 s+ J3 `0 M <value>10</value>
2 L, _; {0 g% ]4 \& { <description>, ~/ ` v' X8 Q. \) `
Queue size in MB for quorum journal edits.( v* ~" s& e& D G' c
</description>
% F0 ~. _8 u& h</property>
6 Q1 G1 A; S5 ^9 E H- e<property>$ T3 j8 D$ b" U
<name>dfs.qjournal.select-input-streams.timeout.ms</name>
+ q! h$ n, e. [+ ] <value>20000</value>
3 r2 H( {1 |% i <description>
# J7 }3 @. h9 h& ~+ `+ o Timeout in milliseconds for accepting streams from JournalManagers.! f1 f2 X1 A% R6 _4 {
</description>
$ [* B2 c0 X" k, ~2 Y2 [</property>
( Z7 P- r/ A1 R- @6 ^2 q<property>/ j" }4 Y3 \5 e
<name>dfs.qjournal.start-segment.timeout.ms</name>
1 X! Z/ `6 [; I <value>20000</value>
, _1 V' F' A% X% ~! t9 P/ b <description>5 Z9 v9 ?( @# f# `
Quorum timeout in milliseconds for starting a log segment.& O! `" {) ~2 B
</description>2 E9 s. T5 X& Y- o$ c! }
</property>
7 E1 k1 m; f2 r+ F3 o. l<property>( g" \: i6 ~6 ~( A
<name>dfs.qjournal.write-txns.timeout.ms</name>
9 f" X; a* e/ \$ g( x& |7 V <value>20000</value>
5 |. G6 y2 s6 Z( W- y/ F/ T4 b% C <description>" B: ^6 H2 P% t# B" z! u
Write timeout in milliseconds when writing to a quorum of remote
( A4 { C- }/ J \7 F ^ journals.: Q4 f9 `) C3 |, Y7 U
</description>
* _/ f( ]+ G) h) g" ]: H& s$ o6 b</property>9 w+ h7 e! ^8 G5 v
<property>
) y( |# n3 x: U) k4 x$ v, S6 o, Y <name>dfs.quota.by.storage.type.enabled</name>
2 e- M! e6 Y; T <value>true</value>
5 f, h$ H/ W. [! }/ F% A <description>
; ~, a0 t" T w9 X1 [ If true, enables quotas based on storage type.
" e6 C: }1 ~* [- M% O+ e </description>) e0 n5 E$ f* p- Q
</property>
" U+ {6 Y% [; J7 Y. ^<property>
/ l. G B# Y- t- {# u <name>dfs.secondary.namenode.kerberos.principal</name>* [4 Q; |) k' O2 H) W Y; q
<value></value>4 x5 E! S1 R/ O1 f; k0 G+ b
<description># e0 Y* p$ n9 t8 f
Kerberos principal name for the Secondary NameNode.
! u5 e- F# g+ Z9 Z7 g& n </description>+ k/ O. F7 Y% H5 ~
</property># Y4 u* m( @/ b3 e
<property>
! F) h0 s8 i! I7 y! w( y/ S7 t/ J <name>dfs.secondary.namenode.keytab.file</name>
( {' ]( F1 Y* ]- S c <value></value>
/ e4 c% X( q4 ~5 g6 o4 v# ~ <description>
+ l7 p! A- {' @3 ^ Kerberos keytab file for the Secondary NameNode.
/ c9 w/ L0 c( V/ v0 K8 _3 ` </description>
G* s9 H- P, Q9 O$ C</property>
: I% G5 u: ^' o+ \; k! }2 y<property>9 j& J* Q/ F, r! T; d a/ A
<name>dfs.web.authentication.filter</name>8 g U0 x+ k: {# Y4 E% v+ Y$ t
<value>org.apache.hadoop.hdfs.web.AuthFilter</value>
; x9 K+ k/ ?1 U: b7 \ <description>" ]) f. y$ t; m& _0 S6 u! |& z: E8 {
Authentication filter class used for WebHDFS.+ z) K, J# c8 w# s5 ^1 a+ y/ D
</description>& P+ ?. x& ~; b" a: M: M4 L5 [
</property>
& n) i" j- z) H8 @; R% f P<property>
8 S! R4 T6 _% U. f- K- h <name>dfs.web.authentication.simple.anonymous.allowed</name>1 K' Q7 x/ z! ? L
<value></value>
8 c) e; ~& {3 ? n# D0 T <description>
: D E6 R+ y# r' H% j$ @2 K If true, allow anonymous user to access WebHDFS. Set to7 [% t3 X$ d1 V$ ~
false to disable anonymous authentication.
7 k+ W H9 ]! S% o$ p/ O( q </description>) E$ h; G; B+ Z
</property>
9 k1 H: q9 Z' o$ J2 m<property>/ z- K% u( d# b8 M; T
<name>dfs.web.ugi</name>; [' }$ B) k" a
<value></value>. U( z/ f. {; V7 Z2 O& \
<description>3 U5 |+ U0 T5 E/ q
dfs.web.ugi is deprecated. Use hadoop.http.staticuser.user instead.$ O; R! S% Y) g% ]# |( H
</description>0 A9 K- b$ H- |: R9 p/ a# ?, u
</property>
3 n" V# ]' z v( c& t0 h<property>
8 r$ P( ]4 |) y- t3 |4 l <name>dfs.webhdfs.netty.high.watermark</name>4 P$ U8 ]0 J& w% X; G
<value>65535</value>1 I* b8 N2 Q! M' T* ^% r
<description>! U0 \0 m& C( x! ^4 x
High watermark configuration to Netty for Datanode WebHdfs.7 `3 Y! q3 K( T" i& q
</description>- S# y) X* @" J3 p9 D0 K3 C. e+ {
</property>
: L$ _8 r5 ?3 _( [8 |& G, e<property>
9 a! Z9 Y7 K8 L' L& ? <name>dfs.webhdfs.netty.low.watermark</name># J9 @" a4 z+ q$ C; s! d
<value>32768</value>. T$ Y; B4 V" l. X0 p5 T$ v& a( F7 S, f. U
<description>6 `5 f5 R& Y. U( i. z; O
Low watermark configuration to Netty for Datanode WebHdfs.
! O& x1 u* c! T3 L </description>
$ y# r+ \# }, M) i8 x. z6 r, V</property>. n, `" k6 u, Z/ b+ g; R
<property>- Y+ ~$ I; G6 H% f0 e! [4 O: ^' v; @
<name>dfs.webhdfs.oauth2.access.token.provider</name>; O( N, O9 |* m
<value></value>9 ?- Z' |6 a/ d6 H+ h
<description>7 S2 t0 {! g5 ?2 a `+ Z2 R
Access token provider class for WebHDFS using OAuth2.
3 J6 M) I8 R! P+ d V4 ^) S1 j Defaults to org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider. K* \' J5 F6 l! f8 Y+ I4 d
</description>) j1 F- y4 b1 y5 y( u# E
</property>
4 @" t2 q2 D1 s) \- b. w<property>$ d/ \* |! ]/ e, Q) O
<name>dfs.webhdfs.oauth2.client.id</name>
6 M% s' W5 u3 M9 F* y; \ <value></value>
& ~3 _9 p' c+ h <description>
6 G, c+ j& B2 Y0 m: L Client id used to obtain access token with either credential or
) y, }! m6 g) O+ ]& ~ refresh token.
2 Q& f4 n+ U8 E9 U: w5 k. | </description># D' x' Q& R: K: Y9 l* e5 I
</property>
8 ^" G3 T; a8 e" C0 S, q3 r N4 H<property>
$ z$ `$ H& e+ x) c; f <name>dfs.webhdfs.oauth2.enabled</name>! N& W3 [$ O- q( y9 g; w
<value>false</value>
8 c0 k; }* J7 U6 v' @, j+ k <description>5 {" X6 A! |" q2 t+ A4 t1 W1 P
If true, enables OAuth2 in WebHDFS
* X+ `' R, }9 t </description>( h# A* l( O$ _0 J/ B
</property>- R' e* q" d: G5 k; [8 j! _6 t
<property>& ` d$ P& n4 H. n: M C% z* s
<name>dfs.webhdfs.oauth2.refresh.url</name>! ^# B) M$ l3 C0 o. M- ~; d
<value></value>
^2 A- X, l# D' v, A8 A <description>
0 P% ^# q( I" N: X# g URL against which to post for obtaining bearer token with
; n: j5 [. a2 i; u either credential or refresh token.
8 f- {3 w+ G; T; A1 |/ f6 J </description>+ j. o- M: y3 `$ P, C4 ^9 M& _7 H6 Q
</property>
( H; I, R5 u& L$ a* P# w<property>8 Y' q6 v y0 x0 G& j/ o, x
<name>ssl.server.keystore.keypassword</name>. f; R+ N2 q1 G" ? m
<value></value>5 i5 G& G3 s' Q. h- r/ @ d
<description>5 b" u+ H" o# V4 ~6 p2 u' k/ b
Keystore key password for HTTPS SSL configuration. d7 x3 o& ^( |2 ~- Y
</description>
6 @ g# q* s1 t- I! W</property>; n: O% \2 X7 K+ J2 b! ~' l4 B( V7 C" T
<property>& f- a4 _8 R( j) J1 z% u7 L# v
<name>ssl.server.keystore.location</name>7 m8 q* y( C3 ?( I6 h/ H# H
<value></value>1 h- |8 n! t; e* w
<description>5 Q# f( U1 V0 M2 o( e1 R4 U
Keystore location for HTTPS SSL configuration9 [+ s2 H5 Q7 ~) P
</description>, {' O; N8 _) f! k9 i
</property>* D" g: \# D8 a
<property>
. @' c2 t$ J1 d6 J% C/ c( w4 { <name>ssl.server.keystore.password</name>0 x* R% M% ~5 h |4 @$ d
<value></value>( g7 p( g" P( c- W- `. S
<description>
* `1 A7 T6 p2 H! u& E( Y Keystore password for HTTPS SSL configuration% I8 R8 }, W) z: K
</description>
- M& C n& r2 }1 U2 J</property>
- K7 Z$ J9 n- w/ r<property>
: ]5 |9 n* h, L; e3 v( Z <name>ssl.server.truststore.location</name>2 r1 i* Q# a5 _! _
<value></value>
5 k# m; ?; w& f) x9 p4 t) p! y <description>8 C$ K- C+ B% I. S d8 Y) @2 d
Truststore location for HTTPS SSL configuration( ?+ S; p9 f3 m4 T7 j; Z
</description>" E- V: o: h5 p) F% ^# [, ^
</property>+ r! y5 V$ f* T* h/ y
<property>$ L0 F: \) {- t, |, I5 m
<name>ssl.server.truststore.password</name>9 M5 A6 U3 a3 k3 `4 f/ @3 Q
<value></value>4 w, n2 W' f0 W! \: W% X% C; C" }+ [
<description>
3 t Z, ?% F8 E# N5 X Truststore password for HTTPS SSL configuration. R+ J5 C1 G# ?9 \, f) O' Y
</description>
. k; m9 I2 ]" t% s0 E+ P. |</property> a1 F4 R* y3 L0 ]% C
<!--Disk baalncer properties-->1 C$ {. f# G! J# c7 K' L' x' v
<property>& D" a2 n; w6 w( Q9 f1 i/ ~
<name>dfs.disk.balancer.max.disk.throughputInMBperSec</name>. i* T( x7 |4 U$ D5 Q
<value>10</value>5 D1 A- H! T0 h4 p
<description>Maximum disk bandwidth used by diskbalancer
% X# b! g$ o& |, b% f0 j) K4 K during read from a source disk. The unit is MB/sec.7 L! ?; T: d- r$ X
</description>4 D4 k2 Z F. B' \' p9 ^; F
</property>. x7 C' y; @* W- U
<property>
* X2 y. {- ~* {" y" C6 Z/ L# S+ K <name>dfs.disk.balancer.block.tolerance.percent</name>, J, k. K2 v) S3 f# X% E' W# Y/ F
<value>10</value>
- s) r- l/ O0 r+ L <description>
4 \7 z( Q5 j0 ~* g: S5 N7 y; N When a disk balancer copy operation is proceeding, the datanode is still+ K+ y$ C2 r- N4 c X$ w4 g$ t* p
active. So it might not be possible to move the exactly specified
/ p! B6 _0 U% p2 R$ E4 k amount of data. So tolerance allows us to define a percentage which2 i8 |) A4 r! L; R' `% S9 E6 u3 [1 a
defines a good enough move.
& K$ Z1 u Y* e+ f) F9 B, F @ </description>
% Q8 t) n% d m% ^* x( k; I </property>" }2 O, l: Q5 z8 F$ j
<property>
0 [, K# M M/ q% B% A- r <name>dfs.disk.balancer.max.disk.errors</name>
: m: V& i" ]4 E; t3 i <value>5</value>
* W, Y/ ?$ f$ j D0 z <description>
Z p8 I/ A; G$ y* q' L% O During a block move from a source to destination disk, we might
$ ]- s" ^- ]8 O+ z8 n encounter various errors. This defines how many errors we can tolerate- w$ }$ }+ k8 h3 d2 O$ U
before we declare a move between 2 disks (or a step) has failed.
; ]! s2 z' g2 q </description>% ^) c9 c3 Q4 p2 H
</property>4 Y( g2 [8 X! e% E: ]
<property>
) E0 ~" I+ r' d- G9 J- |% M <name>dfs.disk.balancer.plan.valid.interval</name>
2 t; Z2 Q- Z, h& P <value>1d</value>
1 `5 |6 y# ] q; e, f4 } <description>- T5 d8 G0 i7 k' z5 J) ?
Maximum amount of time disk balancer plan is valid. This setting
7 [0 Y! c( |* Y! a( Z supports multiple time unit suffixes as described in
/ K1 ]0 I1 p* ] dfs.heartbeat.interval. If no suffix is specified then milliseconds
7 R. [2 F9 ^, m5 F' } is assumed.# I) }) q8 c% X/ f/ n9 ^ H! ?
</description>
7 D: u) N( s A5 m </property>
0 _4 p, i1 a; ^+ R+ |8 n <property>7 x. y9 J4 g% n# t* c
<name>dfs.disk.balancer.enabled</name>
. ?! h; G7 w- E7 t( |9 Y6 [2 \ <value>true</value>
3 k* a" ?( D( w. H <description>
* J) W, q f5 A This enables the diskbalancer feature on a cluster. By default, disk
2 p+ }, Z2 T1 D. u balancer is enabled.& W, R' m) B3 }3 @& K; q" a
</description>
2 W& D5 i6 T0 _ </property>1 i( V1 S7 R2 X5 E
<property>
, i) \: `" O- q' O <name>dfs.disk.balancer.plan.threshold.percent</name>8 h8 |: G) v) ^" ?1 _& S
<value>10</value>- J' f1 h% ~+ y
<description>! |3 ]2 f4 \# U) C6 g, W
The percentage threshold value for volume Data Density in a plan.: O5 y4 } Z/ U+ _1 ?, v* U7 s9 S
If the absolute value of volume Data Density which is out of
/ q& H7 ?( p* `1 m threshold value in a node, it means that the volumes corresponding to& b. k9 }/ b, c# b" Q3 t$ o+ w% ]
the disks should do the balancing in the plan. The default value is 10.
" L, z$ R% t" t7 k </description>
. I; L7 i: D: f( j5 o: ] @ </property>/ s# X8 Z' d) Q4 Z, X [
<property>
2 `2 H, u4 l* n# K" I& R- o/ { <name>dfs.namenode.provided.enabled</name>
4 z- V L% C' K, m0 U& I9 @9 x <value>false</value>+ O' F$ L6 r) n$ w6 u
<description>5 L2 v) W, P" \5 X" u6 P' A
Enables the Namenode to handle provided storages.
, K5 D) C8 j1 ^8 `3 _: x7 j% v' U M* k6 z </description>* _9 C9 y0 A7 w1 Q
</property>
& H- _% W5 F7 n. g7 z& w <property>9 o& @9 a* U2 e& Y' ?4 K
<name>dfs.provided.storage.id</name>1 C' L" o8 ^. i3 B* r' M
<value>DS-PROVIDED</value>
, ]& W2 s/ D2 P+ x& z+ Z# O <description>% W) r! p2 C' x5 k0 F$ u! G
The storage ID used for provided stores.
% g7 S6 n) K! S5 e/ p3 j4 I! A8 L- U </description># j- o% j y! r6 j$ ^0 g6 W
</property>
{% `! t" u* q, q5 q# ~ <property>
- r9 `# f& \* B+ G' u, A \/ q6 [; D' ^ <name>dfs.provided.aliasmap.class</name>
- O$ W5 a' |# r+ Q% \; y! u% R <value>org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap</value>2 c9 X( |/ \; F
<description>
. [% H; f) s4 Q( }/ k. _8 o: f; U The class that is used to specify the input format of the blocks on
! M6 a! w) [& k5 Z, ] provided storages. The default is8 E& }( n' M) G# D7 \1 s
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap which uses& a8 o3 ^, F0 l% N; s* }: i+ ]- E8 a% S
file regions to describe blocks. The file regions are specified as a/ z. h4 [: ?& X3 k) m
delimited text file. Each file region is a 6-tuple containing the
0 o! V: Q% a# F L block id, remote file path, offset into file, length of block, the
( v" @3 U1 F- Z3 h9 u" m block pool id containing the block, and the generation stamp of the
/ |3 |2 z8 O; F3 z- r. D* u block.
" P* d8 D& F+ z, [1 z </description>
2 Z! y$ X0 T$ j5 }! B& q' c/ c </property>5 m. ]9 z) a( R
<property>
/ q; ^8 P" V! b8 }% W0 `4 Q8 k <name>dfs.provided.aliasmap.inmemory.batch-size</name>, n. v: S* A( n) y
<value>500</value>( l1 V `+ M9 P8 o# w. e
<description>
2 t3 }1 C+ ^3 f3 v The batch size when iterating over the database backing the aliasmap
" w7 G( h+ Q: ~8 r) \ </description>
/ N( j5 T) F6 \8 I </property>& {5 }& ?+ G/ M, ]9 f/ X
<property>4 V) m( }: [* H5 v& o
<name>dfs.provided.aliasmap.inmemory.dnrpc-address</name>( ^- {( e/ ^ J
<value>0.0.0.0:50200</value>
+ q6 V- T, c( Z/ P! a <description>
% I( l& k3 [7 B8 k, C The address where the aliasmap server will be running, p, ?; _3 ]; Q' l& L; |
</description>
4 g( j2 V$ j( e% {+ m' d6 q3 K8 r0 F </property>
; t+ V7 s/ Q+ A/ [1 b$ N <property>
& [9 P2 d7 d' Y; R <name>dfs.provided.aliasmap.inmemory.leveldb.dir</name>
8 s2 e5 U1 p% q9 L5 |/ w' b0 A <value>/tmp</value>1 B& @5 k7 d7 x$ f* P; Y: a
<description>
1 Q5 j2 g( t( M The directory where the leveldb files will be kept7 y# m. V5 R; v9 `2 { Q
</description>' ^& r9 A& M: A, G8 w4 `8 G+ G' i
</property>
$ h4 ]' B6 @4 t/ k6 b <property>
+ W) R# w9 V* s$ q% m' e" c <name>dfs.provided.aliasmap.inmemory.enabled</name>
, u( k$ c0 D3 e0 Y0 U. U' W <value>false</value>0 k( W% {# i( f: R
<description>( G. _+ m# t; Y/ v, g) \3 g) X
Don't use the aliasmap by default. Some tests will fail8 j/ G& J* B4 q! G, W9 k# {
because they try to start the namenode twice with the4 {! z- z/ V2 w1 a1 @( Y
same parameters if you turn it on.
. M; b p& }8 n+ r) h- K </description>0 o# R. `/ o+ I9 m" D( e7 V
</property>
) T7 S; Y/ O# k* b$ k7 n <property>1 _$ d- Y% q& G4 L0 X' v$ m8 M8 D
<name>dfs.provided.aliasmap.text.delimiter</name>
4 o# k* v9 W" D" A2 V, o <value>,</value>
. h) ]$ j8 R8 c$ S8 Z/ s: |" g+ g <description>
J% i2 A: y( h P' F The delimiter used when the provided block map is specified as
$ p M6 L) h1 I* O# ~" \ a text file.
$ E5 j# w+ O5 v" t) x5 y; a Y) l' D* k </description>' B, }7 j! R$ V+ O$ y
</property>1 F: B+ @8 x" |/ Y2 A% r
<property>8 d: g, g- [1 e2 `+ f& ]7 W
<name>dfs.provided.aliasmap.text.read.file</name>6 g1 y/ X" E2 T: H& W
<value></value>
0 {% K) g; e5 j2 I6 d <description>. v1 P" [- F0 W( w5 S+ d0 s: j
The path specifying the provided block map as a text file, specified as
2 a4 D# M; _) z; K" i. L a URI.% z- b7 t7 a0 {3 o
</description> o) D1 {. ^4 h( [7 h8 ]+ J
</property>8 Y9 ?; R1 z, i; [
<property>
# T+ ^; G6 _0 z5 ]: K9 {0 X <name>dfs.provided.aliasmap.text.codec</name>8 f4 @3 s% Z) {3 Z, Z; R6 I
<value></value>, H0 V7 d3 \: s% i* W- Y Y
<description>
- C2 l% U! `5 K. l The codec used to de-compress the provided block map.
& E; q0 o% |. c3 _5 ]* n2 i$ ` </description>
0 J( t& k1 g J# y4 O4 D/ Z- J. z3 D </property>
$ @1 u: r3 {3 b" Z- b5 |, y <property>' z( e v' g; p- \
<name>dfs.provided.aliasmap.text.write.dir</name>
4 j N3 I1 s5 h0 H) Y <value></value>
1 ^( W2 H& U" e <description>
6 t% q! f/ j% c0 b2 i! f1 ~8 E: u The path to which the provided block map should be written as a text
0 N% D k+ L; T7 F! J: u, C- b/ f file, specified as a URI.
6 m! ]0 J; y8 l4 S7 K </description>8 a# o" D7 Y* X* P* L7 B' V
</property>4 `) F! }. i* z: J2 M0 k
<property>4 ?4 @& X( ?2 B! q' V9 F
<name>dfs.provided.aliasmap.leveldb.path</name>
9 ^: h& R8 c1 i) B" M' x <value></value>
' k9 |. b: k* ^6 v! C* j <description>: q7 B; N& g, [$ I
The read/write path for the leveldb-based alias map1 i, R6 u& ~0 X7 a+ y( [* D, g
(org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap).: p4 y$ H: Z* t* A/ J% @; I
The path has to be explicitly configured when this alias map is used.
* q" }/ ?7 e7 z t </description>
6 C/ ]: f, w; C( } </property>
, B2 j s1 A! |, l2 k U1 r) J <property>
5 k0 c, Q9 z% L" S. f <name>dfs.provided.aliasmap.load.retries</name>
% G! V* s& I2 c4 r( g9 M I) S- l <value>0</value>
. y1 x7 `7 z" P- `; U+ B <description>6 y, D( S* [: G. j g1 f' _- Q& \; }
The number of retries on the Datanode to load the provided aliasmap;- a9 K! ]! f0 @. c5 w* @& N
defaults to 0.
. J2 G j z- o; b: ^0 m# s, o& ^ </description>
- i4 C+ z5 d: T* S z% q9 Q9 V* [ </property>7 G' a0 C7 m7 X$ |* A6 M
<property>
$ I% x% [: V4 l: p/ y, j. g9 r5 B- A <name>dfs.lock.suppress.warning.interval</name>& o& E V8 |% P5 h% a1 ^
<value>10s</value>3 J, m: m' E) C
<description>Instrumentation reporting long critical sections will suppress
. X/ |& {7 v- x% ^ H4 M/ M consecutive warnings within this interval.</description>
2 E5 e, f- a4 @ u </property>' P A. J( ^; I7 f+ `0 [
<property>
% j3 \/ O* x3 `3 _) K <name>httpfs.buffer.size</name>" o, P# }8 M$ Y8 [ q
<value>4096</value>: `) K. @; E2 I0 B9 i# @, H, b, `
<description>/ J0 {1 i8 Q" P0 m) t7 g
The size buffer to be used when creating or opening httpfs filesystem IO stream./ V+ B# G6 j; V. l
</description>5 F, E6 N8 j; P0 V0 W
</property>, i, B# q: V# O Q4 G
<property>
3 B3 c( X" i8 O( _5 \4 r <name>dfs.webhdfs.use.ipc.callq</name>
E/ f3 v( ?' Z2 G# ]" E6 ]6 A <value>true</value>
3 e4 U. k1 h* i <description>Enables routing of webhdfs calls through rpc- g, U- ~, r6 V5 A3 y
call queue</description>5 F: s. n3 k$ \% l/ d/ m
</property>
3 d; C' g) N$ A <property>
3 ~( L, d" i, c% O0 C <name>dfs.datanode.disk.check.min.gap</name>
' }1 a& g( O9 ^7 R7 c1 V5 k <value>15m</value>/ b- j; E, s5 V. U$ p, _
<description>
8 A! }' O5 {, U! t# X The minimum gap between two successive checks of the same DataNode
2 J3 P* T) e. G0 r8 E, B2 B" w6 A volume. This setting supports multiple time unit suffixes as described' o% n$ B7 |2 ~( q$ ~
in dfs.heartbeat.interval. If no suffix is specified then milliseconds3 E2 H; D* O' m& p/ t
is assumed.. }4 {- O7 U ]4 D R$ F% B8 C
</description>1 U( r5 C; y( R: _: D, g
</property> q, U+ ^; ` o% q
<property>
5 o* @# v' {6 ]* e <name>dfs.datanode.disk.check.timeout</name>
0 u% w$ q' z5 | <value>10m</value>
% W" z v9 J$ c- z5 O <description>1 |4 N1 H; V5 F! `# f
Maximum allowed time for a disk check to complete during DataNode" j, S. T0 S# l* Z7 e/ K
startup. If the check does not complete within this time interval
C! Q m$ b9 t then the disk is declared as failed. This setting supports
: S) P' x4 b/ K$ C# e' P2 s0 X multiple time unit suffixes as described in dfs.heartbeat.interval.' E+ I& {. Q- h( i/ a( g
If no suffix is specified then milliseconds is assumed.
' j# t( ]0 `1 E4 A F </description>* Y" \& c" L- k8 S' H+ ^
</property>
* d2 f$ x8 q# h6 P9 c <property>
. R: V, {, ^# j1 F$ A; S3 } <name>dfs.use.dfs.network.topology</name>/ x! v! z) g% L* P' h! V
<value>true</value>4 F+ R0 n8 {1 ~8 }8 ^$ I
<description>- I7 l5 d: A% N# N* G1 S
Enables DFSNetworkTopology to choose nodes for placing replicas.
( {# A2 X2 y/ G$ s) b, p& |* ^ When enabled, NetworkTopology will be instantiated as class defined in
. P9 q: `5 K; h/ q9 ^! b, \ property dfs.net.topology.impl, otherwise NetworkTopology will be
, `& H4 Y9 {7 z- G instantiated as class defined in property net.topology.impl.
5 ?0 D+ H9 I: d' q1 g </description>+ q, p4 m' f! m5 ]2 R) A. T+ e7 J
</property>& Q8 S, ~8 U' ]: D: J# `4 M9 V
<property>
+ H* R+ [4 p/ \ <name>dfs.net.topology.impl</name>6 i* {% t7 m8 ~' x O
<value>org.apache.hadoop.hdfs.net.DFSNetworkTopology</value>. G' ]% `0 O. q. I1 o. u
<description> o( Z* ?5 T6 u& n/ G4 j
The implementation class of NetworkTopology used in HDFS. By default,' F- X: M% q% t' J
the class org.apache.hadoop.hdfs.net.DFSNetworkTopology is specified and
) J1 Z! i5 [% ] used in block placement.
7 P0 Q, x0 |% ?+ p2 l This property only works when dfs.use.dfs.network.topology is true.
' M, e/ S/ }, T5 A# T% P </description>
4 w. S: {0 y$ M: ~ </property>( c) F0 J* F$ p1 }1 t
<property>
" X0 j4 v! y' j' b/ e* g1 N* ?9 k <name>dfs.qjm.operations.timeout</name>+ \ j$ ?, n0 Z$ {% B
<value>60s</value>" x- O& V( s P, y( b0 P" a
<description>4 T/ w; R: U( d( @
Common key to set timeout for related operations in ?( x$ U+ z. \+ b/ [
QuorumJournalManager. This setting supports multiple time unit suffixes
$ S( S e4 q4 i# e' D" ^ as described in dfs.heartbeat.interval.
, R& j$ |7 t% m" h) A If no suffix is specified then milliseconds is assumed.- ?) M1 x1 k" y4 F1 u9 P* {$ q- X
</description>
2 [5 Q) ]: u9 C1 }# \ </property>8 x. \' J3 `: [0 p6 y; b8 ]$ {5 m
<property>
`0 t/ y' L! M4 f <name>dfs.reformat.disabled</name>7 H' C, ?7 F# ~7 @. y
<value>false</value>
5 ^0 P( ^8 ^. t% I, |% ] <description>
Q P: }! U5 A- L' Q Disable reformat of NameNode. If it's value is set to "true"
0 n; i0 I. G& R6 G& ] and metadata directories already exist then attempt to format NameNode
) x6 L/ C$ W: r7 r* M will throw NameNodeFormatException.& s" C7 L* W8 D) S' w3 A7 v! g) u
</description>
' B+ z5 _4 S+ [6 p0 P+ z </property>- x \2 s* T( T* |* R! {+ ~
</configuration>/ F2 ]5 V9 ]+ G; x3 F
3.mapred-default.xml
7 ~- |" |4 x" P9 Q+ b# R0 o% q
. W1 S* [/ a5 v; _& q& I3 n<?xml version="1.0"?>- w! q8 M7 J7 B& V
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>- i/ a2 g' j6 n1 s" K0 u6 y
<!--
# _! @0 l4 R$ f4 y Licensed to the Apache Software Foundation (ASF) under one or more
y' D1 T U, t' f+ J5 E contributor license agreements. See the NOTICE file distributed with
) G: @$ \# j2 w" G( ^7 j this work for additional information regarding copyright ownership.' [ n Q4 O' U8 Z6 `9 l; b
The ASF licenses this file to You under the Apache License, Version 2.0
+ f, H7 C) _/ a, j (the "License"); you may not use this file except in compliance with& t2 n' j3 @. g9 ~$ v4 a# g
the License. You may obtain a copy of the License at& E# A# I: _1 r- M5 u% Z
http://www.apache.org/licenses/LICENSE-2.0
7 i3 o2 t+ k2 j5 p) G Unless required by applicable law or agreed to in writing, software
, }" J/ B7 \' G; O& D distributed under the License is distributed on an "AS IS" BASIS,$ E, T' V) |: y2 k4 c5 T: T
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- n$ m* M% m9 y6 s m! T See the License for the specific language governing permissions and
" N9 f7 m6 L" W! N4 E# e limitations under the License.
4 I- C3 b0 K) \8 N! ?* P-->
1 k0 K3 V- }3 q( f+ R. i6 o9 b<!-- Do not modify this file directly. Instead, copy entries that you -->
: t& A' m) r7 O! Y2 r" F<!-- wish to modify from this file into mapred-site.xml and change them -->) D8 |7 \+ u$ I# K T
<!-- there. If mapred-site.xml does not already exist, create it. -->5 r3 H: |* {: L+ Z6 d; `/ K
<configuration>/ ?) J4 Z1 N Q$ D1 c
<property>( i9 M ?/ X% X1 o0 |: M" c
<name>mapreduce.job.hdfs-servers</name>
) m" v: y1 a2 C# x) q <value>${fs.defaultFS}</value>8 I, }# j* Q2 T! c1 ] y5 p
</property>
# X$ C4 f1 G. W3 o/ m7 h! A* m- j<property>
9 }- j9 H( P+ g. z" U5 z <name>mapreduce.job.committer.setup.cleanup.needed</name>
; U% y5 d: K* ~2 _- V9 A# F <value>true</value>2 E9 Z5 V( ]+ K# w
<description> true, if job needs job-setup and job-cleanup.4 V% ~: g( U& H. i% |) p9 Y
false, otherwise. m) v& e9 Z( C
</description>+ ?; ^: F9 @ U' q: F2 H4 u" u
</property>
, k% y$ G& D+ {" u4 z<!-- i/o properties -->
3 s2 D s, e8 G; a4 C ~6 E<property>$ r7 o. B6 K) j$ V, f
<name>mapreduce.task.io.sort.factor</name>
4 a/ ^# P/ c! Q8 Y <value>10</value>
6 G; v) R4 _; H; b! j+ z. q <description>The number of streams to merge at once while sorting
8 Z. ~3 @$ N# F" j files. This determines the number of open file handles.</description>
+ F; H3 {' q$ I2 K</property>
4 K, V! I; D& }+ v$ I9 z<property>
1 x( s8 E; x1 Y" N <name>mapreduce.task.io.sort.mb</name>4 ]2 t& J$ q2 O: S9 W
<value>100</value>$ c$ f# k9 S9 E7 H- z
<description>The total amount of buffer memory to use while sorting
2 j4 _, e3 |; I2 Y! F files, in megabytes. By default, gives each merge stream 1MB, which8 O: _- `4 X9 P. Y
should minimize seeks.</description>
* ~) {6 u8 s0 E# Z' u* y1 j, P</property>
; h" d! J1 t0 V& v6 |' A4 K<property>
$ Z1 B" v9 ^1 ` l <name>mapreduce.map.sort.spill.percent</name>
# ]$ r, T/ f9 ] <value>0.80</value>/ \& L5 B9 H6 }! D$ ~( F$ k' O
<description>The soft limit in the serialization buffer. Once reached, a
% V. }- T( d H$ `" T' s thread will begin to spill the contents to disk in the background. Note that
) O( }+ K5 k# |/ Y3 }. E* Q: f collection will not block if this threshold is exceeded while a spill is
. a, ^. q2 U J: |4 F7 o+ R7 w already in progress, so spills may be larger than this threshold when it is
9 F& u- S5 Y0 `: l; F set to less than .5</description>
& ?( ^6 V7 d8 Q, \</property>
& B) C; @: E2 K i$ r, q U<property>
* h" _- A9 K: W# z' d% f <name>mapreduce.job.local-fs.single-disk-limit.bytes</name>5 \8 r4 M/ ~; i f
<value>-1</value>8 S; p* t% u! R! N& Z
<description>Enable an in task monitor thread to watch for single disk
1 n$ }6 N/ r2 L consumption by jobs. By setting this to x nr of bytes, the task will fast4 J, `1 h5 a' G z O4 P
fail in case it is reached. This is a per disk configuration.</description>: W( a# y6 _4 S5 C' T; q
</property>
) I, u# s% ^3 r: A( R+ e. p<property>
0 E) E, t( |! @' g# @* h <name>mapreduce.job.local-fs.single-disk-limit.check.interval-ms</name>9 w9 T, g" ^% H, w2 }
<value>5000</value>" R. Z1 D3 V9 [* e6 Y: s
<description>Interval of disk limit check to run in ms.</description>
$ v+ I* u& t R! b* i5 b</property>
% [/ ?, L$ ] q- A/ N, K<property>
3 W$ A6 V3 Q) F: R3 c <name>mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed</name>
5 @& P5 A2 t- o$ e4 _2 u2 m <value>true</value>
' D$ x( D' L' ~! g) ? <description>If mapreduce.job.local-fs.single-disk-limit.bytes is triggered+ u8 \6 x" a. k. A9 p: J7 J
should the task be killed or logged. If false the intent to kill the task
3 ?& r8 d x9 z& H/ S" D, F, d is only logged in the container logs.</description>' R: J9 d( e2 @( I7 Q+ I
</property>- m6 a' d3 J0 Z+ p4 n! A6 v
<property> `% o* i2 `4 R7 t' o
<name>mapreduce.job.maps</name>. ?3 J2 r! |( ^: E) q/ o t! g8 P
<value>2</value>
0 X. K* K2 }- B, D6 M2 E# c8 o <description>The default number of map tasks per job.
% ~3 `9 W# I: w7 R Ignored when mapreduce.framework.name is "local".
" k* ]" R6 t, }8 ]. N </description>6 K, L A! k* j8 C8 A
</property>8 K. P, c% s! U: a" E( t |% r$ w
<property>! g: b* m7 d' h3 k, @( e
<name>mapreduce.job.reduces</name>. m& q' r" `4 x9 r+ d$ I- I# F% X0 h
<value>1</value>
) z0 J! A( x+ `8 Q <description>The default number of reduce tasks per job. Typically set to 99%* `; @3 _0 t6 ]- e! C" |
of the cluster's reduce capacity, so that if a node fails the reduces can9 M' G X) N8 d2 E/ a8 S
still be executed in a single wave.
$ B2 ]! [7 u1 i5 p# d$ A Ignored when mapreduce.framework.name is "local".
4 h. S% @6 o( W6 X: ~, q7 x </description>
4 I! w8 ?0 a9 A. y! z0 k# {</property>
2 `) X- F# [' z0 S; C<property>
$ G7 r( U Y$ U1 i: C7 t$ L6 f" F <name>mapreduce.job.running.map.limit</name># U* Q" ^8 Q* j1 |$ H: @( d$ Z5 f4 j; l/ \
<value>0</value>
3 X/ d1 r6 Z# c. `# [& z! ] <description>The maximum number of simultaneous map tasks per job.
- s9 }4 i( o3 r: Q7 D3 I5 X There is no limit if this value is 0 or negative.
$ T" P. I4 U/ v8 B7 G </description>. p$ n; Q# P) v
</property>
7 n: t5 r, J+ X& a( k5 ]<property>
" k o6 ~9 |! d8 G3 J6 i <name>mapreduce.job.running.reduce.limit</name>+ H& L) F' }9 w) E& v; h4 t
<value>0</value>
& M! N4 g Z ^ <description>The maximum number of simultaneous reduce tasks per job.
; `! ?5 ]" U9 K* i There is no limit if this value is 0 or negative.
" a& M! J0 G7 X; o. I/ p' @. Z4 p </description>" q% ]& ?9 h0 e& k2 G0 a
</property>
& l- c5 |2 L) y& C) u# T0 ]8 B. c9 m1 Q5 P<property>
, X, o8 \& i0 g& h- k4 ~ <name>mapreduce.job.max.map</name>
: ^9 P1 r# b! R( i5 K; J- X <value>-1</value>
+ j- K# D; F, p, y' o% o j, u <description>Limit on the number of map tasks allowed per job.
. n8 s; j; O0 ~+ C There is no limit if this value is negative.2 D% j: q9 o! Z, j5 E" K
</description>
, L" F7 m! ?( r! P* v</property>* S4 h# _6 J$ j+ t# H( K5 z
<property>
* w' r% ~0 u4 f' U5 Q5 e$ _ <name>mapreduce.job.reducer.preempt.delay.sec</name>$ } y" Z3 y$ |5 E& i
<value>0</value>
$ k( N6 ~5 w2 C+ f <description>The threshold (in seconds) after which an unsatisfied7 _+ ~) Q J. C" \2 X
mapper request triggers reducer preemption when there is no anticipated6 q8 B# [2 g2 \' q
headroom. If set to 0 or a negative value, the reducer is preempted as& R. ~* i6 n+ l6 t4 {
soon as lack of headroom is detected. Default is 0.
- l* Y' Z9 X5 M- T </description>
9 b3 ]! }1 f3 h$ `+ N% E" z </property>1 b9 R" h' [& ~* ^! x: e
<property>
, E! g8 q( O7 T5 t- E3 O6 ~& V <name>mapreduce.job.reducer.unconditional-preempt.delay.sec</name>
/ d. k, D( b9 w4 A <value>300</value>) m) O1 x9 ~2 J
<description>The threshold (in seconds) after which an unsatisfied
2 D9 ^; w" a; c. _ s mapper request triggers a forced reducer preemption irrespective of the
6 r! t/ H( u, W* b% r+ z9 B4 U anticipated headroom. By default, it is set to 5 mins. Setting it to 0
' Y% _2 ~0 c( l0 o3 h1 @7 G8 s leads to immediate reducer preemption. Setting to -1 disables this4 r8 [' Q4 D6 g$ C6 c/ v1 j
preemption altogether.
0 m) x0 Z7 t. ^" F </description>; U8 |+ {7 M% B+ ?- _ F: r
</property>
9 d, ^/ ?- c9 Z$ F1 X3 c <property>5 m) l0 q! s: Q* e$ |
<name>mapreduce.job.max.split.locations</name>
9 j9 S% ]; m: B: H8 M- e <value>10</value>, S2 Y6 Y0 |' K0 L' L
<description>The max number of block locations to store for each split for
8 a# i6 _$ T& e& [ A2 w locality calculation., i. q8 D7 g2 |
</description>/ o8 F4 q+ _- e7 R9 D5 _3 b
</property>
7 Q ]+ j6 x$ D: s1 G" Q7 |<property>. v. @! X* {- s5 T; U/ O/ f
<name>mapreduce.job.split.metainfo.maxsize</name>- V; x0 M5 f6 B$ A
<value>10000000</value>5 m( P# F4 J! r: e. m
<description>The maximum permissible size of the split metainfo file.
3 Z8 |4 ^! g( W, V The MapReduce ApplicationMaster won't attempt to read submitted split metainfo
0 g, X5 J1 E9 {; [ files bigger than this configured value.
, p$ [5 p8 }- F No limits if set to -1.
: f5 x6 \, s, I </description>) d% \% z8 d/ L, W1 d& }& ? ?
</property>% i: U& } s% P3 L
<property>
( e/ Y+ f) K$ T1 J6 R <name>mapreduce.map.maxattempts</name>9 Z% Y9 s" K* Q3 ?9 W' }- U
<value>4</value>
" h7 c4 E; s5 f# q; r h* R <description>Expert: The maximum number of attempts per map task.
# y6 Q' R3 n( J6 t( A$ T In other words, framework will try to execute a map task these many number
* z0 `7 O" R: c" y" E* S6 G7 K5 u of times before giving up on it.6 ]& ` f9 U$ b, w1 q
</description>
9 z9 ]* ?8 H% K* d U' o, B</property>5 g+ u3 z* I( R2 F1 a1 ~ |% g
<property>* a: A' k% ]3 h% }0 T: t) _; b% |
<name>mapreduce.reduce.maxattempts</name>* @, Z) V$ [+ G% @0 E
<value>4</value>+ j& m4 O) m0 r, c# _
<description>Expert: The maximum number of attempts per reduce task.) \: U$ ?5 P- d* a+ w$ m% E! j
In other words, framework will try to execute a reduce task these many number
' V; z! \* }! a, G4 }. F of times before giving up on it.
& `9 q! ~! M1 K5 ~5 J. X </description>9 o! N, {9 t2 [( L h/ Q
</property>: i" H" E4 o. n1 V( h
<property>
* [- q! e. b: g3 g: B$ v- ? <name>mapreduce.reduce.shuffle.fetch.retry.enabled</name>7 c6 P- o' U% e( y0 P: n$ R( T
<value>${yarn.nodemanager.recovery.enabled}</value>
# F% [/ a+ w+ V <description>Set to enable fetch retry during host restart.</description>" \, U) y; X5 c: W6 ~! u
</property>
* e) b9 p- d/ H* e5 n2 h; ~<property>" Y* m8 \( e* Q
<name>mapreduce.reduce.shuffle.fetch.retry.interval-ms</name>+ C. q. j7 b& ~6 Z+ k5 \% Z
<value>1000</value>
, S+ a" j2 p- a5 N9 x <description>Time of interval that fetcher retry to fetch again when some
" F# S; p3 V: s' u ]" a; ]2 w non-fatal failure happens because of some events like NM restart.
5 l& a' Q, x! C7 B% p </description>- q( r( {! c# _* r' X% _( Q! @8 n
</property>
2 a( [% d8 Y4 T) q<property>
( c. C% a+ I6 _4 G# F <name>mapreduce.reduce.shuffle.fetch.retry.timeout-ms</name>
( [# W7 [4 L/ L! E: S& D; {! N <value>30000</value>
! j7 S% b8 F2 R$ F+ }7 }+ v5 ]. ` <description>Timeout value for fetcher to retry to fetch again when some
+ L* k4 R* Y* I9 q6 D5 Z8 X non-fatal failure happens because of some events like NM restart.</description>
5 L0 v! O# X1 i c7 f; Q</property>
' L8 Y' h& b0 q3 T<property>( `8 A; L- e; Z- w
<name>mapreduce.reduce.shuffle.retry-delay.max.ms</name>" n( J# m9 f5 p
<value>60000</value>1 G1 `& u* @6 ? _! H) Y- D ^
<description>The maximum number of ms the reducer will delay before retrying: [$ Q, h9 u2 H# s2 R
to download map data.
2 U. @4 Z+ O7 B+ I% N </description>6 v) Y6 f8 P* ~
</property>
. b0 C2 ^% [: v: s8 g J3 y<property>
& R+ K, {+ G) i, j2 Q3 {! c7 P, ?) { <name>mapreduce.reduce.shuffle.parallelcopies</name>( \ q$ p' i+ y+ H
<value>5</value>
* w& a; x7 v9 l <description>The default number of parallel transfers run by reduce g* Q% J* U! `5 b$ o0 t& k
during the copy(shuffle) phase.
+ U/ \6 e" I# F </description>: k# \* p/ {; L: z* X4 _; Y
</property># [. ^, s/ ?: ?0 r4 t
<property>" B. \- r9 x5 r+ x
<name>mapreduce.reduce.shuffle.connect.timeout</name>
( x/ N3 V" A) E, A <value>180000</value>$ {/ H( e6 D: w& s1 B
<description>Expert: The maximum amount of time (in milli seconds) reduce+ x( D& }. w0 F- T$ i+ j
task spends in trying to connect to a remote node for getting map output.) O3 _- o0 [8 i5 g
</description>
: y3 a2 \. ~5 f* }) H( }. V& s</property>
/ p/ Q% ^" k5 i9 E* k6 T1 D3 q/ X<property>9 {5 |7 D; M- j! p
<name>mapreduce.reduce.shuffle.read.timeout</name>
9 g8 } l; C' u' `6 p7 @: N6 [ <value>180000</value>
5 }" Y* o% ^$ U) B1 i- Y, ^ <description>Expert: The maximum amount of time (in milli seconds) reduce a* r- m5 J v1 D
task waits for map output data to be available for reading after obtaining
* ?0 {, w9 ^2 [' d5 T3 ]& { connection.
+ V4 A6 N+ K7 l+ e0 D </description>
# I% F: I/ Y% C/ d. l</property>
: {- H ~# T# T<property>
$ H" }3 p, V) [; t <name>mapreduce.shuffle.listen.queue.size</name>
& v3 S: p# y: Y f' W <value>128</value>
& R+ f4 f; |! Y& d- W <description>The length of the shuffle server listen queue.</description>
% o3 G5 |, G5 U</property>3 @! @$ T, u6 W& Y
<property>
2 s, b7 |/ c1 [# U <name>mapreduce.shuffle.connection-keep-alive.enable</name>
0 b; z; d, ^1 ^: q! [/ P' u5 q: ^ <value>false</value>
2 b0 L2 X; b% h" d8 M+ ` <description>set to true to support keep-alive connections.</description>
# a5 m+ i7 o, T</property>/ X; `- M, y3 U- b% O9 t( M, @
<property>/ u2 B: Z* x$ @) s# l$ R8 g
<name>mapreduce.shuffle.connection-keep-alive.timeout</name>3 ~# x4 K+ x ~. }8 u2 u
<value>5</value>' W8 C/ N5 R2 |- F1 p( v
<description>The number of seconds a shuffle client attempts to retain
' c6 A2 ~/ q2 K http connection. Refer "Keep-Alive: timeout=" header in R ], z, a+ g* @" w: Z
Http specification& ~/ Y$ H% y" S7 \; M! I
</description>/ }+ A( s; |% d) R; C
</property>, U- a9 y6 K4 ^8 i1 n# C
<property>7 ?+ ]+ E5 h4 \4 a- z
<name>mapreduce.task.timeout</name>; I) m K+ \) I
<value>600000</value>* W0 S6 l1 B M, C! G
<description>The number of milliseconds before a task will be
2 m* M9 s ?2 i3 ?7 ? terminated if it neither reads an input, writes an output, nor
6 J9 O8 Q# g2 d* \' j updates its status string. A value of 0 disables the timeout.
4 g0 J: \( @$ G) b/ X% ~0 u </description>
( X. k, W/ r6 s$ \; d1 Y</property>
% P, \' `* h) F1 c r7 t+ J<property>) D( \' y& ?* r& {9 l. z
<name>mapreduce.map.memory.mb</name>
+ Z2 G+ J( L3 o- b F <value>-1</value> S# Y$ B E: `- I* P- M; I+ b
<description>The amount of memory to request from the scheduler for each6 L3 J; c( ~- h
map task. If this is not specified or is non-positive, it is inferred from: y1 |% }+ Q: _% B
mapreduce.map.java.opts and mapreduce.job.heap.memory-mb.ratio.
7 t* Z4 C$ U. r" i If java-opts are also not specified, we set it to 1024.
( x n. P; Y8 k </description>
: ~; Z8 M0 o. f5 }# h</property>8 v+ P/ |, s6 S% n/ J: h- j& S# g% S
<property>
8 _* F& ~! l/ e4 b' V+ A <name>mapreduce.map.cpu.vcores</name>
# P4 U9 N( ^8 y; G% Z5 d/ D: ` <value>1</value>& g/ u. B, R: z/ `9 h
<description>The number of virtual cores to request from the scheduler for$ M/ Z. D+ |( n! E- G
each map task., Z2 n- j) S9 v/ ^
</description>
- S1 \6 B" D- s/ e6 j" h3 I' R</property>0 j+ ~5 Y( {3 d x2 l: c7 H
<property>" E; ]" G" u( [. Q+ [# h( Z
<name>mapreduce.reduce.memory.mb</name>
" _7 H3 s* K8 V <value>-1</value>
. o8 b$ G f& W8 O& c J <description>The amount of memory to request from the scheduler for each
+ s. S; D. _# a8 N reduce task. If this is not specified or is non-positive, it is inferred
2 |3 ?5 v' [1 x( q from mapreduce.reduce.java.opts and mapreduce.job.heap.memory-mb.ratio.
) V2 o4 {- W" s8 G& _ If java-opts are also not specified, we set it to 1024.
+ @7 j j( o4 _8 _& C8 n; n9 ? </description> a2 b' ]5 l+ w0 J+ m) |( c& ?
</property>
/ C( u) q& o0 u<property>' C6 ]: ^! ^" ]9 O- e
<name>mapreduce.reduce.cpu.vcores</name>) ?* |8 z1 C3 g6 g3 ^2 A# t- _4 W
<value>1</value>5 n8 Y/ f! v- w- l& ]' P$ o- [3 u
<description>The number of virtual cores to request from the scheduler for
/ W3 |( H1 g6 t' ` each reduce task.7 r; B5 v" a% `$ A% ^5 o
</description>
% p- y0 N2 @, D7 z4 v</property>
. m! b' I2 n5 g<property>! u. u+ E( J6 h* W
<name>mapred.child.java.opts</name>
9 V8 D& Z3 z. k2 e" D) Q$ \ <value></value>
# q+ e1 `8 e8 f1 r <description>Java opts for the task processes.0 ?: O& M. t$ L) a
The following symbol, if present, will be interpolated: @taskid@ is replaced
9 j2 Z: x3 W$ I5 y# W* Y by current TaskID. Any other occurrences of '@' will go unchanged.
7 I* P! r# Q1 z For example, to enable verbose gc logging to a file named for the taskid in
" _6 ~! }8 w! e /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
+ V# g" g. b& }' E4 T -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc _! j* C; U5 \) u ~/ S
Usage of -Djava.library.path can cause programs to no longer function if5 n8 R- j8 Y$ m0 r7 l6 u
hadoop native libraries are used. These values should instead be set as part3 _9 ~, t/ j& e* o
of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
8 j, Y& ^, M5 y mapreduce.reduce.env config settings.3 G4 H# P1 O! ^1 O) |) c5 G
If -Xmx is not set, it is inferred from mapreduce.{map|reduce}.memory.mb and' l1 R6 K# X: |6 u m# V4 Z
mapreduce.job.heap.memory-mb.ratio.
$ A8 U$ Q( V$ ]# I </description>( T0 _" Y7 D4 v* ?! u
</property>9 `% r2 r2 f6 g- F+ _ y
<!-- This is commented out so that it won't override mapred.child.java.opts.
U# M3 i# i& B/ M# `9 k' u<property>) x Q) K, r6 n1 G
<name>mapreduce.map.java.opts</name>
/ ?6 B; G- { S <value></value>9 j& e# \3 H8 W$ R3 H# j( u
<description>Java opts only for the child processes that are maps. If set,
7 t0 q7 [2 g8 G d4 T& b this will be used instead of mapred.child.java.opts. If -Xmx is not set,
9 ^: H7 Q+ K# D+ ?; J$ |4 G: B it is inferred from mapreduce.map.memory.mb and
# a: f- O0 y0 u" d mapreduce.job.heap.memory-mb.ratio.
. y; D, O" x8 Y f0 i </description>
0 u( b) r* Y$ n5 D/ f& I/ ~, {</property>; E& a. v% t. v2 s% O/ [6 }
-->
/ j$ d& r# w- I( t: Z<!-- This is commented out so that it won't override mapred.child.java.opts., h: C8 h% d& H4 _8 a- {+ @# d
<property>6 H! _3 f0 u# a& t
<name>mapreduce.reduce.java.opts</name>& P; Q% I" A0 C6 h/ L; H
<value></value>. Z/ w3 @' }5 g( A# h8 X0 X
<description>Java opts only for the child processes that are reduces. If set,1 X7 e% a& D: c, h3 }
this will be used instead of mapred.child.java.opts. If -Xmx is not set,- }+ [4 D# x. |, M$ n; s
it is inferred from mapreduce.reduce.memory.mb and. J$ h6 M3 f3 G0 O/ @# O3 A
mapreduce.job.heap.memory-mb.ratio.
( o+ D$ Z, v- ]9 y </description>- y: i/ T; t& u: O4 E1 p
</property># c4 M, `+ a6 ^0 x- T
-->
u6 o8 N, I. x2 o2 e<property>8 }. o) P% U6 k u7 ]
<name>mapred.child.env</name>5 E d1 ^9 _1 k. u3 j
<value></value>) X3 H) s- H! h. [! F
<description>User added environment variables for the task processes.
6 u' Y* z* e; q Example :
& C2 p* d8 ?# p5 D1 m# a/ F 1) A=foo This will set the env variable A to foo
6 E! o1 L: y( G' W7 b7 B8 z 2) B=$B:c This is inherit nodemanager's B env variable on Unix.% x/ Y2 `4 y( C7 }% m* O: e. X: ]
3) B=%B%;c This is inherit nodemanager's B env variable on Windows.
& [+ U. [0 b5 L </description>
! N& `4 L0 M3 K1 U4 R8 _</property>9 Z; M9 @5 y$ L5 {2 K+ W
<!-- This is commented out so that it won't override mapred.child.env.
" x$ v2 i# Q+ ~/ y1 }' ~! }" h1 Z<property>3 m M7 E3 c* Z: B: Z. a% ?
<name>mapreduce.map.env</name>7 {' w8 d3 p6 T5 e& z
<value></value>
$ z$ r4 s0 e8 D" z <description>User added environment variables for the map task processes.: E* Q- X; i+ X) W; L4 M( F- ]) p5 a
</description>6 O* k9 g, w2 M5 X
</property>$ \- m! r8 G% Y) F: \+ y
-->
" D+ i1 C8 e: l5 M<!-- This is commented out so that it won't override mapred.child.env.! K: V0 h# A- c
<property>; f+ X' R- R- i
<name>mapreduce.reduce.env</name>" ^& i( D5 r2 D
<value></value>2 O3 w5 q6 k# U. f" o3 Z
<description>User added environment variables for the reduce task processes.8 y k E0 X7 O( M3 i ^: ]$ t
</description>( r. P' ?4 t0 w! e `* t
</property>: M* }/ g* B& S% M! k: G4 g, M
-->: E# D0 r5 A& O; y+ |; p
<property>5 ]# ?* H" Q1 V& m
<name>mapreduce.admin.user.env</name>
1 o7 x" C, q+ R$ [" ?+ N <value></value>
& j; |9 s4 e4 E2 [& N <description>. R! `# y- ]! _! y
Expert: Additional execution environment entries for
! L* v) Z" u8 p( n6 ?" d2 q+ g map and reduce task processes. This is not an additive property.
3 |$ @( C( H- f8 H r. p. |) Y& j You must preserve the original value if you want your map and
2 p) ?3 U" e1 T8 ] T" R reduce tasks to have access to native libraries (compression, etc).
7 d( D2 y$ n, J6 |, J When this value is empty, the command to set execution
) @$ D" O$ m# y' p$ X7 [$ I" D envrionment will be OS dependent:9 G( q+ w# b8 K) s/ @9 s
For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
4 W7 M5 x8 Q3 T: D4 Q For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.' c, C8 h$ ^; V0 x+ L/ n+ ?
</description>
& Q# l2 A5 |* h5 ~</property>0 _+ j( o- @: B( }
<property>' }7 o O5 F! N( \) Z- D
<name>yarn.app.mapreduce.am.log.level</name>6 Y i2 P7 d7 H% x
<value>INFO</value>
& f j! i/ v: I" ^: N' V4 g% x <description>The logging level for the MR ApplicationMaster. The allowed6 u9 q6 A% P4 a
levels are: OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.: q! @6 y& u/ q% C' M
The setting here could be overriden if "mapreduce.job.log4j-properties-file": v2 a; v5 b% i5 {+ F$ S* ]; d
is set./ a# \# p/ |7 F( @: h5 R
</description>! y0 P+ X/ l- F% @. x8 C- D
</property>- m* {7 g" ]+ {+ u% C
<property>( a) i0 ^6 e4 n% N
<name>mapreduce.map.log.level</name>! z- Z) c% G# q; ~
<value>INFO</value>
2 h# u4 ]# G& K% c8 y+ b: B: h <description>The logging level for the map task. The allowed levels are:
+ c/ L7 \! N# O' S& P! S( { OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.' b& [: y! i3 N# J& E" T
The setting here could be overridden if "mapreduce.job.log4j-properties-file"
5 g: G6 t( C; @* e1 A% N& C is set.9 o4 _3 t& w- i: Z4 V
</description>) k6 c3 h }$ u& H! Y5 N. y# Y1 b
</property>
0 f# M$ q9 O0 z/ s3 n<property>7 m6 o: o1 |8 [" ?% c" ~6 r5 M5 z1 N5 m
<name>mapreduce.reduce.log.level</name>
% [1 ?, x! A0 v$ B; Y# ^ <value>INFO</value>8 j B5 \: l& d/ Q: i }+ R
<description>The logging level for the reduce task. The allowed levels are:- t5 P: v, B* S, C2 Z1 j* H
OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL.$ S1 u) Q( @/ C4 x( D5 F0 O
The setting here could be overridden if "mapreduce.job.log4j-properties-file"
4 v' H- [3 O+ d, F' V is set.
; R& f2 ]8 @7 y/ j! t3 I/ Z4 O$ H( w </description>2 P+ v: B7 v5 s6 d- U% q X
</property>+ w* S) Y1 G- \
<property>, q- c# F+ X. ]* j- T5 o
<name>mapreduce.reduce.merge.inmem.threshold</name>
" s; H- s+ w' i' o4 j( B- a: s <value>1000</value>- ]7 e. d# Y0 r8 m' r
<description>The threshold, in terms of the number of files9 D- {$ `8 k1 I1 ^6 R2 R; q5 n
for the in-memory merge process. When we accumulate threshold number of files; ?. r9 }% W3 r/ q3 Z/ l
we initiate the in-memory merge and spill to disk. A value of 0 or less than3 O; E1 ?: m# M8 }
0 indicates we want to DON'T have any threshold and instead depend only on
- k7 S. A- O6 a% D& p; e5 t0 b the ramfs's memory consumption to trigger the merge.6 B2 b- k. W: C
</description>3 x: ?( G$ d% a
</property>
- }( l$ i8 }: Z( O" f& e d<property>$ g9 q- s; C$ B& t
<name>mapreduce.reduce.shuffle.merge.percent</name>
; N; R9 g) b: G2 N* f: y6 Y <value>0.66</value>
1 Z. r; h' K' T4 \; B# S; | <description>The usage threshold at which an in-memory merge will be
2 ]2 @/ V. I2 t6 {. j- M q initiated, expressed as a percentage of the total memory allocated to
2 J9 p4 E' N- n& V8 H$ w. e" j& _ storing in-memory map outputs, as defined by
" U }: ~2 D) w9 G2 D3 G% U! ~ mapreduce.reduce.shuffle.input.buffer.percent.
: |9 r# Y W0 Q) S% _: `: e3 H9 O </description>; }) }& I+ _* Z) E/ T. E8 Q9 ~6 K/ ?
</property>4 @$ M n) Y2 `5 s: j5 y
<property>" u* i/ Z" m; ?9 M3 h# }
<name>mapreduce.reduce.shuffle.input.buffer.percent</name>
/ @4 W" v4 P' ~9 C4 X2 a: w! J <value>0.70</value>- {: \1 y- A" {( d
<description>The percentage of memory to be allocated from the maximum heap2 h% h7 \4 s- }
size to storing map outputs during the shuffle.
~7 S7 P8 U; w- U/ u% d </description>
* d$ {+ B- @+ W4 y1 m</property>
& E0 ~; W! K. p3 Q% _# d<property>
: t$ S: U) x' N8 W <name>mapreduce.reduce.input.buffer.percent</name>
) m1 A+ P3 s* h <value>0.0</value>/ i; B* y3 A0 ~6 @# K+ B
<description>The percentage of memory- relative to the maximum heap size- to9 U* m$ K" y" o3 W0 H) o) E5 o
retain map outputs during the reduce. When the shuffle is concluded, any
. I5 {3 O& a" o remaining map outputs in memory must consume less than this threshold before
' t Y r0 n3 H- J6 a6 j2 O9 q the reduce can begin.
) |4 g6 G7 L, W; e& U/ m- T0 ? </description>
: b Y5 w# J. l0 C* o</property>
. U; a, l+ Z, h; E: D6 x<property>
& I! P/ @% h4 q: ~ <name>mapreduce.reduce.shuffle.memory.limit.percent</name>( ?# o' O& [9 y
<value>0.25</value>
8 ]. u: e& D, X4 D( A' B$ G <description>Expert: Maximum percentage of the in-memory limit that a
5 |" V1 A! t ~( ~ single shuffle can consume. Range of valid values is [0.0, 1.0]. If the value& u2 j" i6 b! g8 @7 C/ V
is 0.0 map outputs are shuffled directly to disk.</description>) Y9 b& q: r% \+ X* a4 Y1 z) X
</property>
" ~1 I' j. e+ D5 g2 F<property>
9 y" W! [. z+ d5 T& i2 C <name>mapreduce.shuffle.ssl.enabled</name>, \8 q9 e" x' J7 j
<value>false</value>
1 h5 C1 @6 z. B* |0 i <description># C; V$ ~# z. b8 s. x
Whether to use SSL for for the Shuffle HTTP endpoints.! j: K5 _( p! x0 b# y; h* Q4 U) R
</description>1 n7 v4 r9 i+ e2 [; i& d/ n* k/ x
</property>4 Z S0 U8 Z; D4 f; _
<property>0 O% _$ B* y* I2 y) U+ e6 t/ ]
<name>mapreduce.shuffle.ssl.file.buffer.size</name>
, M3 O4 k& B6 @% e <value>65536</value>
5 |$ _# g. k8 h0 W' e <description>Buffer size for reading spills from file when using SSL.
" b3 Z7 G, m' d; `$ ?! i( [* L </description>+ e8 S E: T: h
</property>
* {6 ^% ^/ K0 h% J<property>
( Y i% j) U( e <name>mapreduce.shuffle.max.connections</name>* W1 \1 J% ^4 _9 o& q
<value>0</value>5 d5 v9 A6 I% D! `, j: {/ w% ^
<description>Max allowed connections for the shuffle. Set to 0 (zero). G$ P: [" h, `: n
to indicate no limit on the number of connections.) j9 r; X$ @' E/ T; [0 y
</description>
@5 T, ?* f1 C</property>
7 }! ~$ {* ?& v w! U: ?! n! x3 c2 b# s<property>' |6 ~1 F# V9 y+ Z
<name>mapreduce.shuffle.max.threads</name>: v5 |! M q, m9 s' s
<value>0</value>
/ x! Z% U: d6 E) @1 @% l, e6 T <description>Max allowed threads for serving shuffle connections. Set to zero
; y- o3 C2 `5 u" q3 ^. y9 M7 k to indicate the default of 2 times the number of available
8 A& |. f# J, H processors (as reported by Runtime.availableProcessors()). Netty is used to
$ W6 _- B. k3 v6 @ serve requests, so a thread is not needed for each connection.
* I5 P% Q4 ^' m* \% }, r8 r) ]; K </description>1 o+ F5 w" N* w. _$ e
</property>
8 K0 D0 i. R, L9 I<property> B/ H; P1 }4 k* x
<name>mapreduce.shuffle.transferTo.allowed</name>
$ o1 Z8 T4 C7 g+ Z1 y <value></value>
/ q: [/ r0 d+ O3 h" j: ] <description>This option can enable/disable using nio transferTo method in
; a4 u( K" k: [2 K1 _ the shuffle phase. NIO transferTo does not perform well on windows in the
5 J# Q V0 P; [8 Y9 d+ Z, j. x- A* S- l shuffle phase. Thus, with this configuration property it is possible to- i4 S ]: p( l0 I! `( R
disable it, in which case custom transfer method will be used. Recommended9 u' V3 {8 H8 \( U& a/ _" a$ [
value is false when running Hadoop on Windows. For Linux, it is recommended
1 ~, `1 n# h* e to set it to true. If nothing is set then the default value is false for
, [, h+ Y w4 D% O3 F+ Q) c Windows, and true for Linux.5 p$ ]) j% g/ t
</description>' Y! Q: }8 S9 N3 ]7 s) Q5 l& A, w1 s
</property>
) q, c3 ^0 V( R( G2 }<property>2 P9 v6 B6 z" f% q! H
<name>mapreduce.shuffle.transfer.buffer.size</name>% F/ m* |% U" L; M& T! t( N
<value>131072</value>* b: g& K7 P, k+ F
<description>This property is used only if/ s5 \# M7 Y& I7 Y& n( ~* p- o
mapreduce.shuffle.transferTo.allowed is set to false. In that case,
& {8 w" t7 q6 d6 f5 w( Y% X3 K this property defines the size of the buffer used in the buffer copy code0 Q5 l# z, e, v- |$ t7 s
for the shuffle phase. The size of this buffer determines the size of the IO$ Y3 G9 Z. X: ~6 p# v# T! h; g V: V
requests.
4 O4 d5 w4 o% m- R: B z2 C5 u </description>5 j1 ], O( f+ D+ q4 ^9 d
</property>% y6 R1 F, R0 I r( h% i; L# ^& ^
<property>
1 @/ U/ M+ b m! h# y <name>mapreduce.reduce.markreset.buffer.percent</name>
' y) i0 @) W. K2 V8 w <value>0.0</value>$ T. @0 c+ ^# z/ q
<description>The percentage of memory -relative to the maximum heap size- to( _* e j' ]' G8 K& _1 v
be used for caching values when using the mark-reset functionality.
& M( ]3 y$ z# o/ g+ A) ^/ ~ </description>
% R/ `+ c. L5 c7 W7 [& R m</property>
. {5 w" P5 @8 o! i5 n# R T<property>: }7 N1 ~4 u6 b$ ]: s# E
<name>mapreduce.map.speculative</name>0 @+ O. O. I1 P q$ m1 T t- Z
<value>true</value>9 i# I/ `9 u; p4 e. G, a- A+ n8 a
<description>If true, then multiple instances of some map tasks
; {6 i3 G. g4 X( o9 g2 }+ d9 ` may be executed in parallel.</description>
9 ?& K) f7 _1 S- \</property>0 {( v% h" E5 L1 M4 E7 Y/ b- Y- d
<property>
3 }# N7 R/ }, @5 E4 c <name>mapreduce.reduce.speculative</name>
, c3 o% K3 q/ F; e* s- x <value>true</value>7 V1 Y6 b9 c" m) w, S- L
<description>If true, then multiple instances of some reduce tasks; u: n9 ~! x( E. U# {0 L
may be executed in parallel.</description>
) K# i9 z& Z3 z) z0 n7 R$ ?, D</property>* B4 }! d7 u+ B2 x3 W3 C
<property>
. Y. S& ?0 e' z9 z <name>mapreduce.job.speculative.speculative-cap-running-tasks</name>; |( [* ^/ q) y$ r3 j; \1 T& E
<value>0.1</value>
( H& V* r/ j0 m4 h ] <description>The max percent (0-1) of running tasks that
& W3 g; d' P0 c5 g/ p' p can be speculatively re-executed at any time.</description>
) C; d( _, b. s" S: }4 a2 ?0 b</property>: @( C$ O8 I+ |- b
<property>6 ~* Z4 T: `6 p
<name>mapreduce.job.speculative.speculative-cap-total-tasks</name>9 L' W# \6 A O3 ~& N
<value>0.01</value>$ u8 |- t" V9 h' x& O
<description>The max percent (0-1) of all tasks that- F ^$ y2 l! e
can be speculatively re-executed at any time.</description>0 y* V2 O& W5 m4 r k& q
</property>5 _/ Y( q! n& \0 h% y
<property>/ b! D3 r o5 z( U5 N
<name>mapreduce.job.speculative.minimum-allowed-tasks</name>- F( a% n3 ]9 m$ `
<value>10</value>
- R: [4 @2 b G) s3 l' E <description>The minimum allowed tasks that6 ]# |; f. T# N" R* a
can be speculatively re-executed at any time.</description>( U( B5 ?: _6 S
</property>
4 k7 N( V0 M9 m* Q<property>
, O5 t1 s! B1 Z- b" N. X% d$ w <name>mapreduce.job.speculative.retry-after-no-speculate</name>, p( f( i* Y! y: E# K8 g
<value>1000</value>$ `) _1 e# f+ _" l, F
<description>The waiting time(ms) to do next round of speculation. d+ R" D* X6 J2 ~ f
if there is no task speculated in this round.</description>$ [) Y; J( ~; G
</property>
2 a) L; w4 Q0 F1 M+ F<property>
( C3 U* ^& `4 Y: |9 Z+ Y! W <name>mapreduce.job.speculative.retry-after-speculate</name>
9 g6 A0 {, n" y: H" A <value>15000</value>; H- p* v9 e" V9 B. u' S
<description>The waiting time(ms) to do next round of speculation0 A$ B; n# P% j) y0 ~" m
if there are tasks speculated in this round.</description>8 B5 v+ V- {0 g, E/ K6 ~' {' k1 Z
</property>
+ ~$ K% n* _8 s6 [<property># ]: B3 y6 P4 d: u5 V
<name>mapreduce.job.map.output.collector.class</name>
5 @9 W/ p) [. m8 ?/ c$ v4 e Z <value>org.apache.hadoop.mapred.MapTask$MapOutputBuffer</value>
$ V/ K) T O M* ?& O5 f8 ]8 O <description>0 a) m3 X& z% G3 J9 V
The MapOutputCollector implementation(s) to use. This may be a comma-separated
9 `: Q1 p7 | O& N4 q list of class names, in which case the map task will try to initialize each
& p ]6 t( X( `% f+ L of the collectors in turn. The first to successfully initialize will be used.
3 M* I2 L& u7 Y% O' n8 R) u </description>
2 D1 E6 H& d- A</property>
& a h- R- v. }+ ?% q, f<property>. L+ d9 A" Y$ U9 B6 T& M7 Z- q
<name>mapreduce.job.speculative.slowtaskthreshold</name># O/ }, a, ^7 m. w
<value>1.0</value>
! u: Q( H2 v$ W) B& p: j <description>The number of standard deviations by which a task's
1 ?$ Q' }, n$ x# s4 { R ave progress-rates must be lower than the average of all running tasks'
' ?9 H) G: C) S( M/ a for the task to be considered too slow.
1 Z- K3 E" O' e </description>
0 V, o9 u' f% f3 y1 W</property>
8 ~6 M! O W+ u+ n0 Y<property>
: R6 ~' w' d8 n) I: L5 W <name>mapreduce.job.ubertask.enable</name>9 c1 Z c- W6 S/ B' S) ]
<value>false</value># X4 h. Q8 D* I5 H. u+ g
<description>Whether to enable the small-jobs "ubertask" optimization,
' Z. e* ?; L/ L7 k O which runs "sufficiently small" jobs sequentially within a single JVM.! u1 |: G9 `% C( r6 [+ V0 U
"Small" is defined by the following maxmaps, maxreduces, and maxbytes9 h! V1 B1 A3 a& n$ H9 ]8 S/ \" J
settings. Note that configurations for application masters also affect5 E3 R* p1 g# F8 S" E( n* v
the "Small" definition - yarn.app.mapreduce.am.resource.mb must be+ I) ^% ]1 [* h( I; R! t9 |3 D! y
larger than both mapreduce.map.memory.mb and mapreduce.reduce.memory.mb,5 X; h6 o% c" v A0 V
and yarn.app.mapreduce.am.resource.cpu-vcores must be larger than
( v* M: e. B/ X. Y; Q both mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores to enable
7 c& m+ a' I8 X0 V7 ~ e ubertask. Users may override this value.
3 F# J5 T; E5 V: ` </description>& E( e* H4 L/ x7 b4 h% h2 Q# J) ^; V
</property>- I I$ {& f% {, g
<property>
# @+ }; l' ?- U, \& D8 ? <name>mapreduce.job.ubertask.maxmaps</name>3 x5 c- b A, R( X, ]" J
<value>9</value>% P) Z4 `0 a/ {4 X# \
<description>Threshold for number of maps, beyond which job is considered
0 C/ I5 c2 U: n( I; E) K3 q too big for the ubertasking optimization. Users may override this value,
0 M. A' O# J. R6 p$ B$ a7 J but only downward.
9 m# M& d& H$ j; U' A </description>+ x/ E: E& l/ m! ?' J
</property>
D& T% S# [7 l<property>
$ W1 Q+ @" `9 a) p) T) q <name>mapreduce.job.ubertask.maxreduces</name>' E. ^2 F6 v( g
<value>1</value>& H5 u; r9 n# e4 |" L
<description>Threshold for number of reduces, beyond which job is considered9 O6 x$ E3 `2 ~4 \9 a
too big for the ubertasking optimization. CURRENTLY THE CODE CANNOT SUPPORT4 c7 L+ d j: R& i& Y
MORE THAN ONE REDUCE and will ignore larger values. (Zero is a valid max,
8 H) L3 [6 s0 I* l6 w* w however.) Users may override this value, but only downward.
# I0 i$ O& I: e </description>8 I+ F* N. _% A3 A0 D, n
</property>1 h$ D- x2 f3 M7 @1 l
<property>
% @7 L- {! _4 Z- I2 y( B <name>mapreduce.job.ubertask.maxbytes</name># H/ E8 j \8 G5 E) Y: t
<value></value>+ J' G- r/ N7 \2 a* Q: ~! P3 ]
<description>Threshold for number of input bytes, beyond which job is: ~ g1 {( S4 H0 I" C
considered too big for the ubertasking optimization. If no value is, o# N2 ]* d1 \( Y5 v3 F
specified, dfs.block.size is used as a default. Be sure to specify a
, J S' T& ?0 W; z5 q: [ default value in mapred-site.xml if the underlying filesystem is not HDFS.
$ _! |! c' `3 o, ]* B Users may override this value, but only downward.9 T( x6 O c8 Q2 q% A
</description>
$ ^2 [4 s' J' W) d1 P</property>- k. h+ o- ?! [ d/ B3 r$ t H& f
<property>4 i; Q% n( l) f4 a# i, _1 V
<name>mapreduce.job.emit-timeline-data</name>
8 E: M" H ]' _) M! e* o0 V <value>false</value>
8 ?6 s+ v* F: r2 y$ F* I3 h <description>Specifies if the Application Master should emit timeline data
) K# F' e; p9 \& t) M6 L) M to the timeline server. Individual jobs can override this value." D9 l6 ]( H/ b9 U! e8 G& j
</description>
3 H V, P3 @& _5 P1 s# g* O8 {7 h</property># H1 Z! r) O( m! U
<property>
+ y/ m! k7 o) n <name>mapreduce.job.sharedcache.mode</name>1 T J7 @2 | b+ L3 ~
<value>disabled</value>
2 f8 e- @& j( a6 o/ d <description>% W1 [3 C1 B6 \ K- r
A comma delimited list of resource categories to submit to the shared cache.
5 L7 L) Z! `/ O) c+ a' s( p1 F The valid categories are: jobjar, libjars, files, archives.
# a2 }% @3 [$ u- U5 w, c If "disabled" is specified then the job submission code will not use2 O$ Y$ D, X% O; P
the shared cache.' O, U n3 n. N. P( Q
</description>
7 M# D4 |, P7 D7 [& y& R( @$ E</property>! ]2 @1 A5 ?6 Q5 X/ X" \
<property>
0 R7 o# E% g; Q# z4 V3 t6 v1 p <name>mapreduce.input.fileinputformat.split.minsize</name>
6 Q; v, t& s+ X9 d ~+ ~1 K& d( D <value>0</value>5 [+ l: ^: s3 R; s; V3 o& D, x% `
<description>The minimum size chunk that map input should be split& I: ~ R% h1 R! f0 C8 T x
into. Note that some file formats may have minimum split sizes that
9 Q8 |" q( @5 q5 P0 U' R% Q take priority over this setting.</description>( o0 Z" [" I; g# b2 b
</property>+ R5 s5 s5 I* f6 Y, E5 z! W: Z7 z
<property>
8 m$ v7 F0 x' }5 T( O. g4 ^# u3 @ <name>mapreduce.input.fileinputformat.list-status.num-threads</name> ~+ |: m9 t" _% z* I
<value>1</value>
% [" K# m8 {* H, |2 k <description>The number of threads to use to list and fetch block locations
# D2 W2 Z' i$ Q2 Z# E! Z for the specified input paths. Note: multiple threads should not be used S. V `) Q* ~3 \( O: m
if a custom non thread-safe path filter is used.; Y5 C; C& f6 T3 A* Y
</description>) Y/ E$ @$ ]; W
</property>+ K; Y9 ^$ J$ ~; a6 I' G8 O3 G6 u
<property>
* l. u/ p9 ?& v- Y5 S B <name>mapreduce.input.lineinputformat.linespermap</name>! S( a) B" K4 w9 U0 p
<value>1</value>
2 g. U, H0 T( K0 a <description>When using NLineInputFormat, the number of lines of input data _1 E" W$ W+ V7 _, d% c1 ?9 r
to include in each split.</description>1 x' |% O1 I+ p
</property>. P9 B. F, x$ S( n! g; \# }
<property>
. J* K b. {, a0 _# s7 P <name>mapreduce.client.submit.file.replication</name>9 W6 E& Y9 V. A; }# s. Y
<value>10</value>
+ H* h3 C" v* B6 X3 _. b, {' z& t <description>The replication level for submitted job files. This; J X) J/ Y8 A: B& Z7 [: O
should be around the square root of the number of nodes.+ U# L( i5 G, n7 K
</description>
3 F! Q' ^: W9 T! S$ C$ _4 Y</property>
- Q0 B& q3 e l8 M% p- j0 u1 N<property>6 H! c5 Z; k3 t W/ h
<name>mapreduce.task.files.preserve.failedtasks</name>' A0 ]5 P7 ^0 c( q6 g0 E
<value>false</value>
1 ?) o7 c$ J! c8 J3 B6 C$ B <description>Should the files for failed tasks be kept. This should only be6 P/ {' B, v- x9 S# v* ]
used on jobs that are failing, because the storage is never9 S/ o* ~5 d/ \5 h/ j$ S
reclaimed. It also prevents the map outputs from being erased' I& j- ^, _7 }. K
from the reduce directory as they are consumed.</description>+ J; {1 f, Z; j( z
</property>
7 x5 s0 n3 f3 @* F! S# f<!--' T0 M I% B+ c; _: g
<property>
2 H0 @$ K9 g. I7 d2 O2 C <name>mapreduce.task.files.preserve.filepattern</name>5 \1 P' U: I6 [# F
<value>.*_m_123456_0</value>. x" m, l- ^/ o
<description>Keep all files from tasks whose task names match the given, g! J) C) ?% U1 q7 L' K+ t
regular expression. Defaults to none.</description>
4 |4 Z( J) [' T9 k </property>
, f! ]- i8 J) O1 R% T-->2 W7 W/ B2 V9 o3 }6 F9 p1 j
<property>
- Z1 |9 a' h3 a8 d4 A <name>mapreduce.output.fileoutputformat.compress</name>
4 U# r( Z6 n/ ]& @. ~) {4 D% ~5 ^ <value>false</value>5 ?9 ?4 D0 D" b/ E3 u& T
<description>Should the job outputs be compressed?
( {' n, E/ U; Z3 M2 y* L) Z) \ </description>
) l7 D% R$ K l5 t</property>$ w, ~' L" S- t4 {& T
<property>
$ ^1 a. l1 r" F7 v) g <name>mapreduce.output.fileoutputformat.compress.type</name>
- j' }, Z$ {3 [4 A <value>RECORD</value>
9 {, K0 Z% U, d% s1 n/ P3 q <description>If the job outputs are to compressed as SequenceFiles, how should: ~& ^1 m/ R# E6 {* f V3 h9 j
they be compressed? Should be one of NONE, RECORD or BLOCK.
+ B) F# a9 b5 M% x' y% O </description>
/ n+ W- L+ n/ l& T. B</property>% u) ]- D1 d1 S$ b& w* d
<property>
* H7 ~/ C% G! h" v7 F r1 [0 ` <name>mapreduce.output.fileoutputformat.compress.codec</name>
0 ]8 A' p/ S4 R! l( t <value>org.apache.hadoop.io.compress.DefaultCodec</value>6 a1 u6 R. z" L6 R2 t
<description>If the job outputs are compressed, how should they be compressed?
# U% K# f) t+ W, _2 q1 o5 Z </description>% ?8 M( M% s$ r4 ?/ A
</property>
" t. }4 _3 p7 Y! a<property>
6 P2 p% }7 [# ]! G$ O0 } <name>mapreduce.map.output.compress</name>
) }# }6 N& B; @ <value>false</value>; r4 ~9 q2 V# u5 B8 x
<description>Should the outputs of the maps be compressed before being
8 k' D [6 y# I1 G, w5 g# g sent across the network. Uses SequenceFile compression.
4 L" w. b% P ^4 P, j& j </description>8 b1 M9 H: e6 x. `
</property>
' O2 J& ~0 v5 y% S& Y<property>4 \; g( L6 S* y
<name>mapreduce.map.output.compress.codec</name>
6 ] B3 z+ X: x' j8 r4 o <value>org.apache.hadoop.io.compress.DefaultCodec</value>
3 G+ @1 i$ v- e <description>If the map outputs are compressed, how should they be
* h4 C2 K8 D, w2 J5 H! I( { X compressed?, u8 o9 h0 a) I$ |
</description>) S" x# D& f) Q7 `
</property>% s" L! q8 A3 C K. b
<property>/ A* l6 o- {& Y' l) n/ B
<name>map.sort.class</name>1 N0 c; W' H3 j2 m& h+ k0 W
<value>org.apache.hadoop.util.QuickSort</value>
- X7 @' \* X" t) s& R7 O <description>The default sort class for sorting keys.
* ^6 ^1 [9 E5 X. x </description>8 N, j$ C0 z) J0 L0 W
</property>
, Q( G! o" f+ t' N( Y. q: }<property># y6 {( i' P* H- Z3 ]% T& h/ y
<name>mapreduce.task.userlog.limit.kb</name>8 d% } l2 W( ~2 Z5 A; V5 R
<value>0</value>
. I8 U# ]1 v! D$ m- P3 H% s <description>The maximum size of user-logs of each task in KB. 0 disables the cap.8 r6 m, R: S6 O6 t5 }
</description>
; \, I' T$ n* D- Q n</property>
' h; U: T: v' j& b+ n<property>4 ^+ K( [$ ]1 d% Q# ]! o V! V
<name>yarn.app.mapreduce.am.container.log.limit.kb</name>) P7 }$ X3 f( E4 U1 y* K# u
<value>0</value>% a" u$ D/ ]% c# Y
<description>The maximum size of the MRAppMaster attempt container logs in KB.) `9 e: q: s; L" k4 Q# S
0 disables the cap.9 Q2 i' W# B# g, y
</description>
, G: |, ^& ?8 `2 ?</property>
% l/ p4 C5 u! ]8 T4 R<property>9 Q' Y3 V( X3 E! H
<name>yarn.app.mapreduce.task.container.log.backups</name>
7 ~- U; R# `; O' P" y6 B# a <value>0</value>( f8 W9 A( ~( s0 U+ N# n( G
<description>Number of backup files for task logs when using
" {! I7 X& Q% u; [1 C- C. t ContainerRollingLogAppender (CRLA). See
" x6 \9 T F7 q6 | org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,% m- _' e7 }! R# |: _
ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA
/ Y. U7 a; W5 G is enabled for tasks when both mapreduce.task.userlog.limit.kb and
; s3 _" O' J) [; d* F" i) z: U4 ` yarn.app.mapreduce.task.container.log.backups are greater than zero.5 g) N) a) {% I
</description>0 a; n! Z2 R. ~9 B; t! Z1 S2 E/ u
</property>
* M: H% U0 h2 M$ I8 `<property>
! g9 G& a9 o1 E: [ <name>yarn.app.mapreduce.am.container.log.backups</name>
& H& o; w m1 `. d: ~# O <value>0</value>* j' u4 P& |* @6 U
<description>Number of backup files for the ApplicationMaster logs when using' t/ x3 e' M9 g- G0 Q/ i1 o
ContainerRollingLogAppender (CRLA). See
& G# G* b# r. g' ~3 I% ] org.apache.log4j.RollingFileAppender.maxBackupIndex. By default,
- V2 ]3 x2 N, ?2 e7 R: ] ContainerLogAppender (CLA) is used, and container logs are not rolled. CRLA6 y. g- O( ^$ {% T; U
is enabled for the ApplicationMaster when both6 _. c6 Q+ j8 H
yarn.app.mapreduce.am.container.log.limit.kb and
# |* w0 D9 i+ L6 k; T- I1 K0 p yarn.app.mapreduce.am.container.log.backups are greater than zero./ A, V6 v- i# c3 N! i
</description>
0 h3 p) {% ~9 y</property>
/ n& e4 D+ H) u2 n' x. y<property>( e& _# G# z1 r' x; y" E' a
<name>yarn.app.mapreduce.shuffle.log.separate</name>
+ P6 o, b n" e <value>true</value>
# u2 D" J' a1 J+ x/ w0 k <description>If enabled ('true') logging generated by the client-side shuffle. H& o) B0 q1 T% i# [
classes in a reducer will be written in a dedicated log file, L# T% Z7 [/ Q- Y- G
'syslog.shuffle' instead of 'syslog'.
8 ~! k G2 i1 T' _$ b0 R </description>
7 ^" J& f' y" k9 p: c3 x</property>
4 s- n; E! M; O% C* C5 i3 h<property>0 O8 I2 X! c8 |& U5 p, r9 \0 ?
<name>yarn.app.mapreduce.shuffle.log.limit.kb</name>' w$ X. t$ \' v. w
<value>0</value>
, s& y' W3 ]/ G7 g( s <description>Maximum size of the syslog.shuffle file in kilobytes0 G" U7 p: T, z& f2 P m
(0 for no limit).1 X3 }0 p* J* u, u
</description>
, v& D& M: F1 `8 U- i* i2 y</property>6 s" ~* E( B% y9 l, K
<property>/ b+ m4 ?% A& p* [+ M
<name>yarn.app.mapreduce.shuffle.log.backups</name>
0 B9 a' b3 e3 H <value>0</value>
" Q/ V+ Z% n/ f/ ]0 W <description>If yarn.app.mapreduce.shuffle.log.limit.kb and
+ T' l! I- X F# B& x4 d; Y, e yarn.app.mapreduce.shuffle.log.backups are greater than zero' G* g9 t% T' K6 o! `7 e7 K o% W
then a ContainerRollngLogAppender is used instead of ContainerLogAppender1 {5 u# D' _3 g$ K6 j- }' a* b5 o
for syslog.shuffle. See
7 ^2 w2 {) p6 \) m org.apache.log4j.RollingFileAppender.maxBackupIndex
6 p! }$ |4 C: i/ i </description>
: S8 n) Y& j* M1 d3 H$ i6 z) H! Y</property>
, M0 A# U* _5 F: ?' }<property>
( p* O8 n' r9 m <name>mapreduce.job.maxtaskfailures.per.tracker</name>
2 H3 q* k" M7 V( C# ~% n <value>3</value>
; u2 E# p1 a. H* `1 z. z. Y <description>The number of task-failures on a node manager of a given job& o) |4 k0 x* q. [
after which new tasks of that job aren't assigned to it. It' ]& T4 o( E4 ]9 Q3 E1 Q! L
MUST be less than mapreduce.map.maxattempts and3 P, p! h* c4 w
mapreduce.reduce.maxattempts otherwise the failed task will5 ~, \2 s; i7 I9 E( P5 d$ F& ?% n
never be tried on a different node.
8 V1 M4 A3 y: e- d </description>% O3 [- D( f! T ^. S( n* ~. c
</property>! @; |- r M' h9 j& `5 d
<property>; J$ r$ c$ K. x+ S
<name>mapreduce.client.output.filter</name>" y, x7 d4 j% U/ q
<value>FAILED</value>7 b# j* s* v" s8 `
<description>The filter for controlling the output of the task's userlogs sent3 I; z0 s! U2 d
to the console of the JobClient.
* ]/ j) r: X$ U* {; `. U% I/ x The permissible options are: NONE, KILLED, FAILED, SUCCEEDED and% b5 Q4 z% f- s
ALL.
: {# \7 V/ N2 I4 D6 ~0 _& @ </description>
# I5 k" { `) a4 w8 ? P1 N</property>( h, B" Y" c, J3 G% ?
<property>
0 u+ Y8 q8 `/ f6 a5 N7 ` <name>mapreduce.client.completion.pollinterval</name>9 M$ q8 @; f0 |& R" U! l) c
<value>5000</value>7 {# o# Y$ J0 Y' \! j
<description>The interval (in milliseconds) between which the JobClient/ y, C* C2 J) K# Z0 {4 \3 y
polls the MapReduce ApplicationMaster for updates about job status. You may want to
1 D; p# Z# H8 r' C! T set this to a lower value to make tests run faster on a single node system. Adjusting
6 @* l: X E. Q( B this value in production may lead to unwanted client-server traffic.6 n5 v; @8 p) _) y
</description>; T0 n% L" U9 h* ^9 h/ N$ R
</property>4 J, m) d( c7 `, ~/ X2 o) h. s! b* n4 Q
<property>" l6 G" b4 y+ W% y* |# K% r
<name>mapreduce.client.progressmonitor.pollinterval</name>
3 `. Y, {/ x v( _5 s$ l <value>1000</value>: x5 Q7 _4 L$ {) P4 m
<description>The interval (in milliseconds) between which the JobClient! \; b) j$ o, y% z: \- Q8 M
reports status to the console and checks for job completion. You may want to set this: F! c9 g0 K0 ?5 R3 w- q
to a lower value to make tests run faster on a single node system. Adjusting
3 ? v. i7 s- D9 L s U this value in production may lead to unwanted client-server traffic./ Z7 o* s0 v {( c3 F6 y' W
</description>" I4 \1 _2 M# `. {+ J0 J2 ]
</property>
* ] M3 F: X5 K <property>9 `: J0 w( a: `$ ~6 D
<name>mapreduce.client.libjars.wildcard</name>
) y. H/ m) c4 r& z ~$ V <value>true</value>
8 \3 v8 ~& P) ^ <description>3 }% t) q9 f/ O
Whether the libjars cache files should be localized using
$ W% k( _* s6 d: y7 O. N a wildcarded directory instead of naming each archive independently.( X- G4 q9 w* P6 d% n: m' f
Using wildcards reduces the space needed for storing the job
3 c, O) N% z- x& O/ ` information in the case of a highly available resource manager
- v. w) e+ y! E2 J8 j8 o; ^, z configuration.& Q& j. f2 x, @+ t1 o
This propery should only be set to false for specific$ G! H, ]/ P+ g' X# u- F2 j6 {
jobs which are highly sensitive to the details of the archive9 {1 w& G0 `' M1 q9 C$ Y% b) I
localization. Having this property set to true will cause the archives' C, x) z. c! q* u- @
to all be localized to the same local cache location. If false, each* c. h- F' W1 W8 y4 `
archive will be localized to its own local cache location. In both
5 j% Q7 i" Z' a# m9 q cases a symbolic link will be created to every archive from the job's
+ F7 w7 x7 Q$ u* Q# n5 u working directory.
2 @0 l8 a& q8 |7 K4 s9 c! b </description>$ _% O9 w9 z" z5 ~4 \* |% S6 d
</property>
. H i# z1 Z) c& e9 U( e- X0 U9 r <property>
' r: m& @# s9 H0 w <name>mapreduce.task.profile</name>6 p& r8 b" Z: T# ]# D2 h% v5 }
<value>false</value>
- |0 Q% {! ]- a4 F7 h" s4 f <description>To set whether the system should collect profiler* H( Z! k+ A; C5 q9 E+ ^0 W
information for some of the tasks in this job? The information is stored/ j! y$ \1 K2 L# L
in the user log directory. The value is "true" if task profiling
9 C0 ?# t' a$ P0 _ is enabled.</description>
) T/ V' K; f) N </property>" H; L6 y2 m, t3 }+ ^, K7 ^
<property>
; b. Y" i Q- o) M3 z9 D <name>mapreduce.task.profile.maps</name>
/ k" d' |, I+ M# W- I, C <value>0-2</value># u- l3 D2 @, N7 R
<description> To set the ranges of map tasks to profile.
% U0 j' Y1 B2 m mapreduce.task.profile has to be set to true for the value to be accounted.9 _+ I8 o9 o" E6 K
</description>
! e+ v s+ C0 d) M! b/ k </property>, i6 q; u6 ]! T: ?
<property>3 c+ I- y# I0 Y
<name>mapreduce.task.profile.reduces</name>; I$ @7 A6 k8 ?5 B$ u) q" I
<value>0-2</value>/ X/ l7 {- E2 g6 W+ u! |% X( {. `
<description> To set the ranges of reduce tasks to profile.
. C4 g. F! K7 A- @7 q U+ h mapreduce.task.profile has to be set to true for the value to be accounted.6 y5 I1 L, q/ c5 p
</description>
) J; ?. E/ _/ e </property>: d3 A- V# y* W' W" v0 n, P+ p
<property>
- W; f j+ W( h7 i, F0 T+ b: V <name>mapreduce.task.profile.params</name>
( o7 W3 L7 f) r+ X5 V- G <value>-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s</value>+ ^! p7 X6 k, j( P# j
<description>JVM profiler parameters used to profile map and reduce task3 R: W; A0 i- I$ E
attempts. This string may contain a single format specifier %s that will7 {; b5 |5 h0 ]0 S$ a% V
be replaced by the path to profile.out in the task attempt log directory.& X3 Y5 B, T, g# |& t$ L
To specify different profiling options for map tasks and reduce tasks,- ~; }$ @# d" g
more specific parameters mapreduce.task.profile.map.params and& P' c8 ~8 Y" Y
mapreduce.task.profile.reduce.params should be used.</description>
0 h+ {6 P4 R' X% w" |8 U </property>
3 ]4 A2 ^' {& M7 a- D <property>9 K7 G6 U6 y# d' d+ m
<name>mapreduce.task.profile.map.params</name>
1 E4 b/ S8 J$ T/ H4 a% K <value>${mapreduce.task.profile.params}</value>, `# w; E) ^7 t9 {5 V
<description>Map-task-specific JVM profiler parameters. See, T5 J4 ^! [$ m$ r6 O& T
mapreduce.task.profile.params</description>
8 z2 ~, Y$ \+ {. C5 J- f </property>
8 p* @" [% E7 }. k3 ]: v <property># L: @- {& M7 ~7 ~9 W# b
<name>mapreduce.task.profile.reduce.params</name>0 B) t+ F. ~( Z, T
<value>${mapreduce.task.profile.params}</value>; @ w# M5 ~0 I9 ?+ v
<description>Reduce-task-specific JVM profiler parameters. See, Y0 r$ I r- l% g2 A
mapreduce.task.profile.params</description>, z C) H. e4 p0 \) l; u
</property>/ b( O0 O; i( y+ _ \% _$ ~
<property>
8 ^$ }$ K0 I( ]1 u5 f) r+ A <name>mapreduce.task.skip.start.attempts</name>
1 i2 @/ |, x6 S( T; a0 I' z" T <value>2</value>3 Y, T4 W% E% s' G) N9 w _
<description> The number of Task attempts AFTER which skip mode% A) ^3 n; G, @1 u
will be kicked off. When skip mode is kicked off, the; V9 C) ^* p$ z) }
tasks reports the range of records which it will process
2 I# Q, s6 S/ u. p3 [ next, to the MR ApplicationMaster. So that on failures, the MR AM
$ a; }4 j9 n, J. G knows which ones are possibly the bad records. On further executions,
- ~: l5 Z* @; e# V those are skipped.
. j, A5 z1 s! s) I/ ^- @" f( C </description>- Q* B' } l! j8 K
</property>
x" l, T' h$ i/ u$ u <property>
$ p, v) N4 q% `! g; \! J <name>mapreduce.job.skip.outdir</name>
' ?) K, a- N4 X( M; S! ` <value></value>4 s# d* {$ C& N/ X8 |/ w- ^/ B8 z
<description> If no value is specified here, the skipped records are9 j4 k3 L3 s% R- w+ f
written to the output directory at _logs/skip.
K( h0 ^6 |" u/ }4 O( I User can stop writing skipped records by giving the value "none".
/ _3 Y1 U% G8 `* r6 c7 y </description>% h& `* c$ `) B# ]% [4 S
</property>7 i; k# Q% X# b" M# J6 n+ b+ L2 \
<property>8 @4 n) w; }6 J X# w6 }
<name>mapreduce.map.skip.maxrecords</name>
! }- l3 F3 m6 m/ h. \" O' h, s <value>0</value>
$ `& W. W0 D0 U- V <description> The number of acceptable skip records surrounding the bad
+ N5 ^- h5 L8 l record PER bad record in mapper. The number includes the bad record as well.0 k1 Q0 Z3 t4 t* T: G' z
To turn the feature of detection/skipping of bad records off, set the8 |$ h( {/ r9 d0 c! b" o0 Q7 M7 r
value to 0.
; U" _7 u2 G" V7 u# q The framework tries to narrow down the skipped range by retrying
6 {( p1 d k1 P6 ?$ F3 T& |1 } until this threshold is met OR all attempts get exhausted for this task.5 i3 I% |6 h9 N2 Z& o. M
Set the value to Long.MAX_VALUE to indicate that framework need not try to
& d/ j7 w& {% ]6 w/ i" j narrow down. Whatever records(depends on application) get skipped are
7 O! m9 i7 n" G4 P+ Y& g/ m) e! |9 Q acceptable.7 A: J7 k- t- W4 v6 K9 D1 F
</description>9 d1 C. F. N5 N" C( y
</property>
3 U$ t, n- a* y4 W <property>, T, r9 W- O- a8 R; X; f. J+ D
<name>mapreduce.map.skip.proc-count.auto-incr</name># H1 N& U {, U ~, [# L
<value>true</value>
" C% G9 y" y! i k+ J) N6 o <description>The flag which if set to true,# ~' \: P( c" Q& f" n
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented by. z, {; ~- Q+ O) s$ J9 n4 s- U
MapRunner after invoking the map function. This value must be set
, v1 y% ?) x2 y) }0 ^4 R& K! I2 V to false for applications which process the records asynchronously4 p* U/ c: H! }
or buffer the input records. For example streaming. In such cases$ B# Y' D- n$ A/ a+ Z+ J
applications should increment this counter on their own.
" u, H! j- F2 ] </description>
5 L( r$ [ n6 a' ~1 M </property>
9 @9 F- u+ ^) t) i X <property>" ?9 q4 r" z! s! u
<name>mapreduce.reduce.skip.maxgroups</name>
: p* e0 c* d/ J9 C: R7 h4 k0 u) M <value>0</value>
& p* y; t6 p9 I) d7 S; i1 s <description> The number of acceptable skip groups surrounding the bad9 ^5 S9 h7 p5 I) V. |( F2 x! B
group PER bad group in reducer. The number includes the bad group as well.
5 Y6 T+ K7 y) g To turn the feature of detection/skipping of bad groups off, set the
3 M( m" E$ w- R" T: E& x value to 0.
0 {3 N% H+ q7 I% j& l The framework tries to narrow down the skipped range by retrying
/ X9 e. j, u0 q* W; @ until this threshold is met OR all attempts get exhausted for this task.
S1 [& v3 V7 k T/ H4 i! Z6 _ Set the value to Long.MAX_VALUE to indicate that framework need not try to
0 h/ O1 U, m+ { narrow down. Whatever groups(depends on application) get skipped are
8 ?4 B6 e- q+ X ^2 ? acceptable.
1 l$ U" w0 d( Y; G) {4 y </description>
/ @8 I+ l; s, B </property>
9 C6 F8 X& `+ P% c <property>
* r0 j/ g- [( j; V, ?2 e <name>mapreduce.reduce.skip.proc-count.auto-incr</name>: b: B. d- c- v
<value>true</value>
' z ]; ^% q4 `" h5 C7 Y2 n <description>The flag which if set to true.+ g+ g6 v3 C) \ z" h( Q. X# |
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework
* V. `) ^4 Q" O: c6 z0 e" C after invoking the reduce function. This value must be set to false for
+ o7 ?* q7 o- c! ?6 q applications which process the records asynchronously or buffer the input/ r7 l9 Z7 ^8 ^( P8 ?8 |
records. For example streaming. In such cases applications should increment
- k+ ?! v6 K5 E% E3 X# b f this counter on their own.# A) o6 f5 G/ |* Z+ v/ |
</description>
0 ~/ P' h' L+ o+ ?; G </property>6 G3 A0 A1 e6 b
<property>3 v; n( U3 P) _4 Q% V0 K: G
<name>mapreduce.ifile.readahead</name>
3 J( D. j0 @8 r( R) B. X6 C9 | <value>true</value>* R9 x' N2 d v( ~' u( S5 U3 G+ P
<description>Configuration key to enable/disable IFile readahead.
8 s# f- j3 g( v. Q </description>" W' H" m; U' q1 `; M
</property>! Z- z4 }" O3 _# _1 G# O
<property>
& @+ M0 s) D9 B6 X <name>mapreduce.ifile.readahead.bytes</name>
# v4 p5 ?) \. @8 M* r" k! } <value>4194304</value>
r' A5 l" R8 @* r' s# {! u <description>Configuration key to set the IFile readahead length in bytes.* q. e; ], J7 \9 @% t6 E2 q, Y( N
</description>
" l) V/ v# P2 S X) P </property>
, q! G+ ?' C/ k4 g3 Q9 p<property>$ y9 t! Y6 Z& e( s# H1 L: N
<name>mapreduce.job.queuename</name>
2 N$ K5 x9 a" I: g/ C; i3 U) K6 ~ <value>default</value>. G: I6 Y. x3 X* _ J# K
<description> Queue to which a job is submitted. This must match one of the) l9 @% A/ p0 T- j8 f. @, }
queues defined in mapred-queues.xml for the system. Also, the ACL setup
# v; r+ K/ }* [0 y* o3 a: m$ n for the queue must allow the current user to submit a job to the queue.2 o( m* I- x/ _( X- i' f% E
Before specifying a queue, ensure that the system is configured with d% O. x" X: ~: P
the queue, and access is allowed for submitting jobs to the queue.
) x8 c* e3 H, E: A </description>
1 E/ s8 A' s9 L8 l; e" i% ^& h</property>
( T! f! z2 W) R <property>
2 Y) V X I7 ?9 m4 h! o <name>mapreduce.job.tags</name>* g; q' l: D9 P; b6 |( d- O' [
<value></value>
6 t6 i& d) t- D) _- O <description> Tags for the job that will be passed to YARN at submission3 t' V6 k6 \5 V
time. Queries to YARN for applications can filter on these tags.
3 s# ?0 e8 x" D" @; Z3 D. x If these tags are intended to be used with The YARN Timeline Service v.2,# p$ M& K! V! |/ R
prefix them with the appropriate tag names for flow name, flow version and
6 Q2 x; C+ i4 O6 w flow run id. Example:. J* l1 ?& ]' R( ?
timeline_flow_name_tag:foo,/ g. F! e9 u( D5 l" _" s3 I& Q
timeline_flow_version_tag:3df8b0d6100530080d2e0decf9e528e57c42a90a,; H/ x0 K8 u! b* S$ ^5 t
timeline_flow_run_id_tag:1465246348599* P7 G( i' L9 W
</description>
: O9 C% x P% a+ y </property>2 S/ }& f7 P$ S+ k) P
<property>
; p- s1 {7 s' Q5 q <name>mapreduce.cluster.local.dir</name>7 O. E& G5 ]. z. h( ^! ?
<value>${hadoop.tmp.dir}/mapred/local</value>7 @+ e L. i5 G
<description>
4 ~( X3 n% {0 E" K2 G The local directory where MapReduce stores intermediate
: n' b8 T8 G. J9 B/ x. P data files. May be a comma-separated list of
6 B/ a0 J: L( s6 w3 g' H directories on different devices in order to spread disk i/o.6 |. T n G' A: B. `( V7 Q
Directories that do not exist are ignored.% ?# \& }- _) d2 S7 n* s) Q; Y. c
</description>0 } m4 M$ q1 V
</property>
/ Z! `9 J; {# ?2 s/ z7 Y<property>
+ ~: j, P2 N' u <name>mapreduce.cluster.acls.enabled</name>
( O; f* x ?3 u, t <value>false</value>
- K, I$ ?* u" Y8 J <description> Specifies whether ACLs should be checked- L' L' a) ~. I5 _( T. C3 H* R
for authorization of users for doing various queue and job level operations." a' p8 c h ]$ K: @& f3 P
ACLs are disabled by default. If enabled, access control checks are made by
" q" u. U+ W3 G3 c MapReduce ApplicationMaster when requests are made by users for queue
& L2 L; H1 ~: N# A$ I9 n* @, j operations like submit job to a queue and kill a job in the queue and job
- Z" v- p S* w) Y operations like viewing the job-details (See mapreduce.job.acl-view-job)
$ `" y4 H( N. `8 b" [ or for modifying the job (See mapreduce.job.acl-modify-job) using
' I/ i+ S- Z! o3 ?) e Map/Reduce APIs, RPCs or via the console and web user interfaces.
) j& M, g$ w% o" B6 q" l For enabling this flag, set to true in mapred-site.xml file of all' j, _1 K7 ]5 F8 ^% }: d
MapReduce clients (MR job submitting nodes).
; _0 D5 r- k/ [" x </description>
8 [4 A. {" \: f" ~' l</property>7 o, s6 H9 H, M+ \' o, [5 R
<property>, H; k+ E1 ]1 L# e" w
<name>mapreduce.job.acl-modify-job</name># A9 u' ?! o8 z4 r* K
<value> </value>. E+ o. L- C- p; H2 b
<description> Job specific access-control list for 'modifying' the job. It9 |6 \- a& u+ V% }
is only used if authorization is enabled in Map/Reduce by setting the+ C1 E2 p- v3 E/ F
configuration property mapreduce.cluster.acls.enabled to true.4 M2 N6 q- G& |' _ \
This specifies the list of users and/or groups who can do modification, p8 Y6 T% e- ?: j1 ^2 O# ~! K
operations on the job. For specifying a list of users and groups the
$ c! ^! `5 v! O# o format to use is "user1,user2 group1,group". If set to '*', it allows all% p9 H% u8 x7 s7 m- p2 ^
users/groups to modify this job. If set to ' '(i.e. space), it allows8 c, @# p6 }3 q0 I5 k: }
none. This configuration is used to guard all the modifications with respect
5 a5 l) Z& N \ to this job and takes care of all the following operations:& a4 o# ~: V" z1 m7 K: _/ Q
o killing this job
4 t$ e/ G; v( E; G+ P' I4 K. F o killing a task of this job, failing a task of this job
9 |# k1 b' S- \5 e5 N1 ^7 z o setting the priority of this job
& S6 G! Y e+ @; R Each of these operations are also protected by the per-queue level ACL
$ {/ `3 I/ C+ e5 x1 G "acl-administer-jobs" configured via mapred-queues.xml. So a caller should
( n# m: y# ?" A have the authorization to satisfy either the queue-level ACL or the) g W: k# N- ?
job-level ACL.
8 k6 O) _; d) u: Z Irrespective of this ACL configuration, (a) job-owner, (b) the user who
' `) ^0 C: H3 J, A started the cluster, (c) members of an admin configured supergroup$ l& ~; j* {# K
configured via mapreduce.cluster.permissions.supergroup and (d) queue
) e) ?' s8 B4 q$ d( A+ Z: E administrators of the queue to which this job was submitted to configured
6 U5 Z/ T3 F' s/ z# v- h* p7 W; t8 X via acl-administer-jobs for the specific queue in mapred-queues.xml can
! f" I7 P1 f/ O5 c2 { do all the modification operations on a job.
, H; {( F9 P" @ z, j6 t By default, nobody else besides job-owner, the user who started the cluster,% u4 G# j7 Y" h4 S
members of supergroup and queue administrators can perform modification" X* V( J. M) d5 `/ c! K m
operations on a job.
1 E0 B* _) ^3 R </description>( C `; A5 f( L
</property>& t6 P5 C7 N. U$ `( F) C4 T
<property>
" R: q9 J/ l+ d& ]0 j# w6 E <name>mapreduce.job.acl-view-job</name>0 `% D4 o, X" S5 U# ~$ Y! D/ O
<value> </value>8 {6 D0 ~" f: n; g" E
<description> Job specific access-control list for 'viewing' the job. It is
" k( e; G2 c! b5 x only used if authorization is enabled in Map/Reduce by setting the
: a; P+ E1 m3 h# K configuration property mapreduce.cluster.acls.enabled to true.
% {" v0 j/ H4 e This specifies the list of users and/or groups who can view private details& N9 u& q u' r8 v. V: ?0 ^
about the job. For specifying a list of users and groups the. b" } ?8 V9 o
format to use is "user1,user2 group1,group". If set to '*', it allows all9 {) {4 [! `9 u5 o! S1 S0 t
users/groups to modify this job. If set to ' '(i.e. space), it allows2 A( X, c3 S. ^6 C
none. This configuration is used to guard some of the job-views and at6 z5 E0 O% V: V; ~/ m) ?
present only protects APIs that can return possibly sensitive information
( G5 e# j/ s0 Q5 |9 Y1 D1 R0 v of the job-owner like% c) \8 \0 C; h1 }% h$ y! n
o job-level counters& z9 Y7 |6 D% [7 S# K: F: s
o task-level counters. O* b* k- H/ m) M: {
o tasks' diagnostic information
e! e* o" p3 ?' A+ Q# u9 o o task-logs displayed on the HistoryServer's web-UI and
! D4 c6 M, n8 Y! [8 J0 V1 U o job.xml showed by the HistoryServer's web-UI8 k4 F1 O# H$ M( v/ M
Every other piece of information of jobs is still accessible by any other
' h3 `# E8 \/ m0 v% X) X- a; ?$ W9 w user, for e.g., JobStatus, JobProfile, list of jobs in the queue, etc.( }. Y8 P. H0 B6 K: {9 o; `
Irrespective of this ACL configuration, (a) job-owner, (b) the user who
. n1 z, y7 p5 D0 E. e# {6 m: c started the cluster, (c) members of an admin configured supergroup
( J: p+ U# |6 r7 ]- y configured via mapreduce.cluster.permissions.supergroup and (d) queue7 B% ]# g3 h- b, l# N
administrators of the queue to which this job was submitted to configured. R+ o+ w, D& f9 ^) `1 Q9 u
via acl-administer-jobs for the specific queue in mapred-queues.xml can
+ h! q G; X5 h: c* |5 F8 y8 l do all the view operations on a job.
. E7 E( V+ O+ E2 f; K. D By default, nobody else besides job-owner, the user who started the
$ W1 s, |- N/ Q, J7 J) s cluster, memebers of supergroup and queue administrators can perform- o& X5 r9 L) x% g
view operations on a job.
7 }( z9 ]! x, M0 }5 j( ~ </description>9 {6 @# v) k" L( O: e7 |
</property>
$ C2 L" D4 A5 v: p* g% ?<property>
3 y) Y+ ?1 l6 S4 w, t. L6 |* L N <name>mapreduce.job.finish-when-all-reducers-done</name>9 |8 O: G+ _. a
<value>true</value>8 C; A! r& I+ R, }
<description>Specifies whether the job should complete once all reducers
/ n$ p$ X2 U3 g6 c) U$ ? have finished, regardless of whether there are still running mappers.: `/ w) `9 ]* g5 x4 j+ A3 f
</description>
, T, C/ s8 ~3 u5 Y& c j6 m</property>
3 Q R. {: D! i9 z5 E+ z5 G<property>
& e/ o$ b7 I" F | <name>mapreduce.job.token.tracking.ids.enabled</name># |0 O8 o- `& s% C. }
<value>false</value>* s- x: H1 a1 L E: A1 g3 z
<description>Whether to write tracking ids of tokens to! k$ K! R( x! h6 M5 g$ ~, u
job-conf. When true, the configuration property
% ]/ L% b' @" d, F7 _! v "mapreduce.job.token.tracking.ids" is set to the token-tracking-ids of' Q {" p, ^( T
the job</description>
9 L2 ? p7 i5 P3 T8 p</property>8 a- I' h! c* j2 i, E! ~: n+ m
<property>
8 \3 X% d9 g+ [0 q& ] <name>mapreduce.job.token.tracking.ids</name># `! x8 `: L* m8 R) ]
<value></value>1 E( S& X- U+ T' ?4 |
<description>When mapreduce.job.token.tracking.ids.enabled is9 m! y5 G& ?. ?% }6 ^ O
set to true, this is set by the framework to the$ L2 y# S$ ~* B% f+ |8 y H' }( O
token-tracking-ids used by the job.</description>
) N3 M; K9 M2 _' Y' B3 x$ u+ X</property>% v, p! x+ @# p3 N! O- \
<property>$ C$ O& }2 N9 W) b
<name>mapreduce.task.merge.progress.records</name>
/ U/ N q1 e" [ g <value>10000</value>
2 t8 K/ Q2 O( P4 f) g <description> The number of records to process during merge before/ Y6 }1 y$ M( C5 L D1 ~
sending a progress notification to the MR ApplicationMaster.& i% w" z+ M) `8 D+ c
</description>( J" u& p' b3 l }0 e5 ]( [" p4 ?3 w
</property>1 g: h, {5 z, i4 y
<property># t& V# \. i7 D- ?
<name>mapreduce.task.combine.progress.records</name>
) R" Y' L5 d4 ?& J2 H+ t8 s <value>10000</value>
& s, M- D4 {$ Z! p" W <description> The number of records to process during combine output collection
( B( B1 o$ f Z; E) k5 A7 v before sending a progress notification.
$ `: a, B+ i$ f& p </description>" k+ \4 z" C. \2 b
</property>
/ u9 U* P* R) F& e+ o- l<property>* ^7 p# J& K# q2 h, a/ _& s- q
<name>mapreduce.job.reduce.slowstart.completedmaps</name>+ h: z3 v0 @" W5 u7 _
<value>0.05</value>) w3 ]- t* l6 `3 w. x, x+ {
<description>Fraction of the number of maps in the job which should be
8 e7 X" U( E1 _3 `9 w complete before reduces are scheduled for the job.: h# |3 L) d9 R: ]8 r6 Y! }! x; U4 l
</description>5 I+ F9 B$ e) D v) o& }
</property>2 e9 N! }: w, \6 T0 K9 I4 `" j2 Q
<property>1 Y3 a4 d8 x8 z
<name>mapreduce.job.complete.cancel.delegation.tokens</name>
; G+ Z' x$ v+ C1 [( ~ <value>true</value>
/ a( G: r0 g ~/ o/ r2 q5 `" ~- F' V <description> if false - do not unregister/cancel delegation tokens from
3 F" i0 S& j1 ^- V% \) d renewal, because same tokens may be used by spawned jobs) d9 @0 X+ P* v" H9 w
</description>0 x2 b* ?1 L6 P1 K( Z
</property>+ G8 n, T8 d9 b8 @6 v. m: C
<property>: `! I/ l, X3 ]1 p- f9 t
<name>mapreduce.shuffle.port</name>
. i x+ H; \" n% s+ h/ \ <value>13562</value>
3 {* J9 e M9 g! H <description>Default port that the ShuffleHandler will run on. ShuffleHandler6 K, i) g! A( C2 f% J) a% l7 b3 m- _
is a service run at the NodeManager to facilitate transfers of intermediate
- j( ~+ t6 W1 z! A Map outputs to requesting Reducers.
: d" }/ m' ]' F* h& P6 O. K. t1 ~ </description>0 y. Q4 d* X+ I( X+ y1 e% C4 I
</property>6 B) P" W. D) C" ?+ |- y- S
<property>+ p- e" p, @: u/ S5 h- v
<name>mapreduce.job.reduce.shuffle.consumer.plugin.class</name>& s5 Z O' ~: l1 Y
<value>org.apache.hadoop.mapreduce.task.reduce.Shuffle</value>
G# r# D. Z" t6 E+ h% H <description>
7 t, K" R* e, Q2 s Name of the class whose instance will be used3 \! y7 @' O- D u- i
to send shuffle requests by reducetasks of this job.
- c! u: \8 u' L6 u The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.* S E) g! X! O
</description>1 }. L0 V# b5 e: T7 H: ^) G0 b% G
</property>/ W4 Q5 D: W2 P3 x- d: r
<!-- MR YARN Application properties -->. _4 Q! m" d7 M- E; E
<property>% u" `8 d# a. s; ^
<name>mapreduce.job.node-label-expression</name>. G$ K6 n% Z4 o' |/ e
<description>All the containers of the Map Reduce job will be run with this
' ~ Y3 v: G& R+ [( a$ {0 I node label expression. If the node-label-expression for job is not set, then
7 i1 R3 y/ ^; b' e' Q) v) a it will use queue's default-node-label-expression for all job's containers.
: @/ e; Z5 j( e, C2 U </description>% [$ k( j5 x! c: v/ d- t! U2 o
</property>
# Y3 h9 W2 x( e0 d6 s<property>
" D7 Z7 A8 \0 f% l <name>mapreduce.job.am.node-label-expression</name>% \ U- }4 ~9 y5 Q. E
<description>This is node-label configuration for Map Reduce Application Master
/ V) B2 ]' W6 t. W8 S container. If not configured it will make use of
; f0 B1 E1 S) ^" O% e& |+ N mapreduce.job.node-label-expression and if job's node-label expression is not
5 Q: l& w: A9 T" E- E) J, Y: ~ configured then it will use queue's default-node-label-expression.
9 N5 J- ?" x( I/ D6 O Q% H; K+ w </description>2 P0 ~" q3 L& C+ p
</property>5 S4 h8 \& M& Z( E4 D6 Y
<property>4 k" t" M' x; V) u+ Z
<name>mapreduce.map.node-label-expression</name>
, [ r" D1 E/ V' J- J <description>This is node-label configuration for Map task containers. If not
7 v6 D: j; b% P" \ configured it will use mapreduce.job.node-label-expression and if job's, U# L0 H+ F3 M+ l7 S% r0 W
node-label expression is not configured then it will use queue's
- J, ^( y) `5 _3 T0 Z; I default-node-label-expression.
0 R% o8 c: X) O3 S1 P8 m </description>) r$ ]* }% k1 P& J/ G" F8 r
</property>0 t( D4 D, f! I2 `- L
<property>1 \6 v4 g2 A2 F; m
<name>mapreduce.reduce.node-label-expression</name>5 ~2 m9 w" m. E- Z& r. p0 e
<description>This is node-label configuration for Reduce task containers. If6 ~, j- M, {% c/ e: @% x
not configured it will use mapreduce.job.node-label-expression and if job's
" d) e5 O: ?& V$ M$ [: P node-label expression is not configured then it will use queue's* p h$ ?( @# y2 z3 C
default-node-label-expression.) A8 z8 }1 {+ I2 \
</description>
3 R1 d w$ [! @! m7 i/ j% A: a</property>8 _4 S, t3 n0 p& }- h' o( b
<property>
M& ?3 S* g+ j( x6 O; ` v& Y% | <name>mapreduce.job.counters.limit</name>
) n# c3 _" Z6 ~! q, m/ P% E <value>120</value>
4 W7 [ y7 Q% \# o: T <description>Limit on the number of user counters allowed per job.8 Z8 D% A6 P. a$ F9 W
</description>1 [) ]& K/ A, a% W, s K
</property>- N/ ]0 ~/ L8 x3 r& z# f6 J
<property>% @. h4 Z" ]% I" K& G* m' F
<name>mapreduce.framework.name</name>
% P3 H0 m6 O! ]) j1 D+ d6 g9 ^7 ]/ T! ` <value>local</value>& @% \3 h' F- l1 B/ r
<description>The runtime framework for executing MapReduce jobs.. b p$ ?) Y3 Y, k/ f9 [" \/ \( L* r
Can be one of local, classic or yarn.
& |9 z5 O. }, H. J" ^ i) j </description>, H/ g% I" c; ]
</property>
& F5 l2 S- ^; L<property>
% L% C' x- ?3 A' a h8 @: U& P7 J <name>yarn.app.mapreduce.am.staging-dir</name>% ?! e% R# ?# [, V
<value>/tmp/hadoop-yarn/staging</value>
; }/ w8 }6 u2 z& t <description>The staging dir used while submitting jobs.
; _3 `6 P8 M$ Z7 a+ n </description> P6 x0 t: q- ^3 t) p
</property>6 ~( k2 V1 }+ O' N& ~
<property>( x+ ]7 m! Z6 W2 b* N
<name>yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled</name>) Q# ~* @- y& |) s$ [3 I: g
<value>false</value>* U( I4 r1 K; H9 n V9 l- B. n
<description>Whether Erasure Coding should be enabled for5 Z7 o j; D* {
files that are copied to the MR staging area. This is a job-level0 C. x* b8 u7 L z+ Q
setting.
0 T. A# T5 Y: S1 J l) g# r* V7 T </description>7 c+ m( b( a7 e$ i) V5 V; \
</property>
* v4 n- K6 b) k. W% h% D: o<property>
! v- Y* }0 c$ U <name>mapreduce.am.max-attempts</name>
8 ?; N2 N4 A. @- Q <value>2</value>
, x5 N! i+ x8 H' ~7 k <description>The maximum number of application attempts. It is a/ d6 L1 U0 \+ z
application-specific setting. It should not be larger than the global number2 T, S5 N# [* d8 \. i( c
set by resourcemanager. Otherwise, it will be override. The default number is2 S4 C% X! \. i& _
set to 2, to allow at least one retry for AM.</description>
3 O/ p# M- i" P; r) f8 [- i</property>
/ S' m! m: k4 W2 D# O2 U( ^<!-- Job Notification Configuration -->6 F7 f, i o9 x, r! W, p( W& e
<property>4 S3 O9 i9 S; Q
<name>mapreduce.job.end-notification.url</name>) V( F" E' s0 Q! `) t2 m" x. D
<!--<value>http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus</value>-->& |- b+ E f' o, U9 A
<description>Indicates url which will be called on completion of job to inform# F9 h+ T9 X3 b6 u+ o9 j
end status of job.) s1 E# O. o' F
User can give at most 2 variables with URI : $jobId and $jobStatus.
( Y J+ [3 u, F If they are present in URI, then they will be replaced by their
6 E& q/ {; c5 P3 q. m' B respective values.$ [5 S7 N U( d
</description>" |3 D. H9 P: o% n+ G! t7 _
</property>8 |; y. b. }& n4 E" {- }! l1 D/ _
<property>5 D9 r/ s8 V; F% J
<name>mapreduce.job.end-notification.retry.attempts</name>% D2 ~! d, F( f' F- J
<value>0</value>
# E: z7 K% e- t* H; T <description>The number of times the submitter of the job wants to retry job6 ~6 ]. ?; C2 A
end notification if it fails. This is capped by" J4 { ~" Z( h
mapreduce.job.end-notification.max.attempts</description>
, m8 s9 ? B- |6 L</property>
* w+ I( H* ^! L! E4 L2 m1 {/ J2 r<property># ?1 }$ I( X* k4 {5 s1 k
<name>mapreduce.job.end-notification.retry.interval</name>
l: y, p2 L2 I1 ` <value>1000</value>( Y0 u5 w* b" Y* L+ I1 I/ }! y
<description>The number of milliseconds the submitter of the job wants to& I G$ ]( t7 W9 N
wait before job end notification is retried if it fails. This is capped by
7 l4 }. i% `! V5 c: W5 J5 s% }3 { mapreduce.job.end-notification.max.retry.interval</description>
+ n2 c7 x, t, L0 ~. L2 o</property>
2 q @3 B7 M4 z( p" l. s! L4 |$ H4 K<property>! J# f/ h n+ Q0 D$ C4 ~
<name>mapreduce.job.end-notification.max.attempts</name>
7 H6 f' g, @1 V! }& v1 a <value>5</value>
, n5 i2 r0 |7 r! K4 c u# V <final>true</final>
: I, l/ ?/ ~. P+ ]3 d <description>The maximum number of times a URL will be read for providing job" Z( b" f9 h& _2 r* S2 X
end notification. Cluster administrators can set this to limit how long
5 q0 _2 Y; M" _ after end of a job, the Application Master waits before exiting. Must be
4 x7 i+ G* N" B$ E9 E' d3 p& f4 [7 u marked as final to prevent users from overriding this.
. L$ n B, d& o+ _* `2 x7 g </description># u5 L7 z: L4 {, v7 Y6 W$ x/ j6 T
</property>
" |# k1 a7 t9 v c5 r6 h <property>7 s& G( o- R: b# A/ O7 ?
<name>mapreduce.job.log4j-properties-file</name>& T* n3 k( {7 `8 [' R8 R8 ]$ K
<value></value>
- W: \# g- x. C* O& r6 Y <description>Used to override the default settings of log4j in container-log4j.properties
. F6 M! a8 U9 Y1 B for NodeManager. Like container-log4j.properties, it requires certain
* \$ s1 [. Y7 l. L framework appenders properly defined in this overriden file. The file on the! P* h! G" \, F3 p
path will be added to distributed cache and classpath. If no-scheme is given( T* _: x. ^7 `/ B4 c0 |3 }; E/ F
in the path, it defaults to point to a log4j file on the local FS.! M& }, N5 c6 v: \* Z
</description>7 {+ T- K' h Y" }
</property>; Q8 p" y, m6 E
<property>
6 K; X$ [; p0 M# Y7 O/ _- v" J7 ?6 K <name>mapreduce.job.end-notification.max.retry.interval</name>
1 J. O9 m/ a4 J& a <value>5000</value>
6 g2 Z! Z6 K } <final>true</final>% [+ P6 }) p$ x) A
<description>The maximum amount of time (in milliseconds) to wait before# ]5 H, c% o# k' k
retrying job end notification. Cluster administrators can set this to- G' s- e' }- x8 a* Z3 Q
limit how long the Application Master waits before exiting. Must be marked7 `4 A7 R7 O' D0 h
as final to prevent users from overriding this.</description>: U2 i r2 o" M+ L7 @" U4 z
</property>
4 ~: b% \0 a8 I" ^5 v3 y: Y, J3 p<property>
. H5 w! c; h `& F' p- E* t4 N0 e <name>yarn.app.mapreduce.am.env</name>
& a2 K$ ]$ i- C2 P8 A3 @# F ? <value></value>( e' l3 i2 k& q
<description>User added environment variables for the MR App Master; f9 v5 p) y/ M( Q1 O
processes. Example :
* A2 H, A8 i1 h9 T% k2 x 1) A=foo This will set the env variable A to foo* d* F5 E( |" Y5 \7 j
2) B=$B:c This is inherit tasktracker's B env variable.! |$ c6 ~6 j0 I+ I; ?/ |0 g5 }
</description>
$ ?. x3 k+ I8 ?; U1 G9 J, D; Q1 B7 V</property>
' _: l& B5 {% W3 O% o<property>
' v6 R2 W9 W* f <name>yarn.app.mapreduce.am.admin.user.env</name>$ |( O6 h! q5 e" U$ Z3 Y* b# O
<value></value>
) j" a- J( s) j0 _: n9 A <description> Environment variables for the MR App Master
2 |6 o# V8 O* ]3 B8 V% p processes for admin purposes. These values are set first and can be
; |4 p0 j' a7 P/ E+ X! \% D overridden by the user env (yarn.app.mapreduce.am.env) Example :5 F+ V+ S" x6 m1 }' H
1) A=foo This will set the env variable A to foo( ^+ r h I7 a9 E$ m
2) B=$B:c This is inherit app master's B env variable.( k! y/ L; P) [1 j( I" n
</description>
3 ^$ U. C* p0 H% _1 _* W</property>
- H( H) o' O2 V# S( J+ ^<property>
* E8 |4 o" H6 j9 O: j5 \( _ <name>yarn.app.mapreduce.am.command-opts</name>! A+ T% r, u: ^
<value>-Xmx1024m</value># S/ _! _" e5 s$ ]" L# c: h
<description>Java opts for the MR App Master processes.
+ p( R. e' d+ b7 Q8 s# G+ E; x+ X The following symbol, if present, will be interpolated: @taskid@ is replaced
x9 x1 [+ m' P% [ by current TaskID. Any other occurrences of '@' will go unchanged.
7 J! w8 j- R4 o* q& F. n3 e8 C3 F For example, to enable verbose gc logging to a file named for the taskid in- M5 p6 a, k! T% o
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
! Z. {5 H/ ?1 T5 _ -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
$ l: c+ n# `& E Usage of -Djava.library.path can cause programs to no longer function if
9 k5 }" [! d% ]: o# J2 u3 i hadoop native libraries are used. These values should instead be set as part
! }# `# {# M8 E4 b0 Z7 `' C of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
* d9 l4 q2 `: ], F mapreduce.reduce.env config settings.$ T$ Y' D' F6 d( y$ |5 `
</description>. Z! [9 G( v9 b4 W
</property>
c. z! l. [$ W6 j<property>0 b* G; m+ U8 t7 M
<name>yarn.app.mapreduce.am.admin-command-opts</name>5 W/ W" K& w6 v$ u* b* m; m3 `
<value></value>3 {1 o5 Q5 R. q; [0 S% s
<description>Java opts for the MR App Master processes for admin purposes.
6 a4 X" x/ f/ F It will appears before the opts set by yarn.app.mapreduce.am.command-opts and3 {: r( c1 B/ M7 C
thus its options can be overridden user.
6 `* m0 j; z% y ~ Usage of -Djava.library.path can cause programs to no longer function if# O; l" L& _$ Y# d# e) a+ x* v! l( O
hadoop native libraries are used. These values should instead be set as part
* C; n8 v. ~% a7 Z u* C5 u of LD_LIBRARY_PATH in the map / reduce JVM env using the mapreduce.map.env and
6 r" v8 U# D' E% |2 r mapreduce.reduce.env config settings.7 \5 ]# B0 |4 i0 N( {
</description>' R$ Y1 U8 D* q: A, T6 y
</property>$ o* i) _1 O2 @6 c$ Q
<property>
t5 d$ s" A( o" z0 x <name>yarn.app.mapreduce.am.job.task.listener.thread-count</name>( Q0 P5 Z2 Z9 ]$ z$ @8 Q) O
<value>30</value>
' _6 d. m5 f9 H% l <description>The number of threads used to handle RPC calls in the* ~2 g; f. q& u# a( Z6 Y
MR AppMaster from remote tasks</description>
4 G) s. T' c7 `+ G5 K+ W</property>
$ b1 }# K7 A! ^9 j' z<property>, |& h. @ W! E" N3 M( p5 n8 g
<name>yarn.app.mapreduce.am.job.client.port-range</name>& R. y- y6 V1 E9 D
<value></value>6 {- i0 e- t; K8 i
<description>Range of ports that the MapReduce AM can use when binding.
/ a% r3 U# o4 Q8 B$ S0 j0 m' D Leave blank if you want all possible ports., w# ?) B5 M7 b: i. l! T ^: K
For example 50000-50050,50100-50200</description>
9 J( B0 s$ P+ i" g& `</property>) s% p, c9 `- N2 j* m& R: ^
<property># C/ @" Z- ~7 P) t- a9 i
<name>yarn.app.mapreduce.am.webapp.port-range</name>
9 O: v. j" U' f. I' [+ f" I* w$ J <value></value>
: |2 c% W6 z3 [# ] <description>Range of ports that the MapReduce AM can use for its webapp when binding.
+ z+ q% }! t+ H( J Leave blank if you want all possible ports.
" W6 }4 h0 {" e4 t( }- ~7 Z" z For example 50000-50050,50100-50200</description>0 E7 x. D2 Y) r0 M, {, \ ~
</property>
- D2 H6 C7 H! t* F, q0 ?<property>3 P# R( W* t4 \# F6 R$ [0 V- D
<name>yarn.app.mapreduce.am.job.committer.cancel-timeout</name># n9 I# i( N P. H0 n# _6 E
<value>60000</value>
: l. m Q. d& `/ m3 c8 h <description>The amount of time in milliseconds to wait for the output2 k% m8 v9 ]* L3 Q0 {
committer to cancel an operation if the job is killed</description>! o, _! x4 Y s" t- O2 A- P
</property>2 V; B" ?1 ]8 p9 t
<property>
$ u" o3 u( W8 Z. i: G <name>yarn.app.mapreduce.am.job.committer.commit-window</name>! Y0 N; t) X; G+ p- k5 y9 |! p f/ P
<value>10000</value>
% ?$ @5 A- I9 W- d' f <description>Defines a time window in milliseconds for output commit8 { G) W/ y* ?+ _
operations. If contact with the RM has occurred within this window then
- X8 U/ Z8 U& H* X z commits are allowed, otherwise the AM will not allow output commits until
! S) E3 Z# c, a4 L contact with the RM has been re-established.</description>( q% |- T7 p/ A: z$ W; q' n
</property>/ B6 x8 y5 f' M5 V5 c
<property>; L0 h# v# P2 g: I- `
<name>mapreduce.fileoutputcommitter.algorithm.version</name>: f8 ^/ _. w+ F7 Q' x9 K
<value>2</value>2 P: p. \" Z, g2 l5 O" B
<description>The file output committer algorithm version
& T! b9 ` T1 J% g$ p valid algorithm version number: 1 or 22 m% n4 O1 V2 E+ L% K
default to 2, which is the original algorithm5 q% Z4 {4 f$ o) o3 O% m; y$ |
In algorithm version 1,
( y8 \. }7 r1 w( Q 1. commitTask will rename directory& F7 R* e. H3 M& \$ v" `: z5 s6 S
$joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/) Z2 P1 Q1 y) j; u8 j7 M
to
+ A8 ?% c( h/ W7 B: M) l/ E $joboutput/_temporary/$appAttemptID/$taskID/+ ]( n) V( p/ C) y0 ?, Z0 l* K. G
2. recoverTask will also do a rename
1 ^9 _* ~+ {, [ $joboutput/_temporary/$appAttemptID/$taskID/
5 k7 @( w3 Q4 o: b( b to
: Z t2 F0 q7 @# e $joboutput/_temporary/($appAttemptID + 1)/$taskID/
) r2 X% i9 k5 ?" j6 O7 y 3. commitJob will merge every task output file in& I; u& i5 s5 M
$joboutput/_temporary/$appAttemptID/$taskID/0 p" s4 y6 k, ^
to+ n! F8 u+ i! q
$joboutput/, then it will delete $joboutput/_temporary/0 g& C. `( W' J/ p3 U2 W
and write $joboutput/_SUCCESS
) `5 K3 _8 }0 } It has a performance regression, which is discussed in MAPREDUCE-4815.
6 q" r. U2 [9 R; G If a job generates many files to commit then the commitJob
+ W6 I+ d% p$ j6 A: S, L A; m method call at the end of the job can take minutes.
# Z" a. x- G/ ~* p- L" R/ X the commit is single-threaded and waits until all
0 l- ]: @7 Q p$ O tasks have completed before commencing.( X/ M0 B- W+ G2 A% E
algorithm version 2 will change the behavior of commitTask,
$ d% W+ G, s" w0 k! J o recoverTask, and commitJob.0 K* R' ?1 k3 m2 I! N% q7 \8 T7 u% d
1. commitTask will rename all files in
5 |, f% n- z$ }6 r, I* S $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
# c) \6 I' P, b6 d' I4 \1 Q$ _ to $joboutput/- m* E' x. k$ ~6 A4 s e4 r- D& }; g3 m
2. recoverTask actually doesn't require to do anything, but for3 ^9 H- n& ]( C( n: K
upgrade from version 1 to version 2 case, it will check if there
& S) v; E8 S8 `5 I are any files in5 N* e0 q3 }& m3 e6 l; C. W
$joboutput/_temporary/($appAttemptID - 1)/$taskID/" Q# q$ ?* E: @& U3 P
and rename them to $joboutput/& g; Z# |- \( P% d& r
3. commitJob can simply delete $joboutput/_temporary and write- _2 p. @" V& `9 m- }
$joboutput/_SUCCESS
; v* I3 a" L- U4 H This algorithm will reduce the output commit time for7 U y. G y) h7 S
large jobs by having the tasks commit directly to the final2 B v6 j% ~ y8 @$ k0 `. T
output directory as they were completing and commitJob had' r# }& k# r5 n: k9 z2 t8 t x4 Z- N& V; ?
very little to do.
+ h8 j S$ a7 ] F </description># R. s* h1 \0 q
</property>, @( b1 C9 o; `: R; w
<property>0 |2 ] ]) M. t( ]7 r
<name>mapreduce.fileoutputcommitter.task.cleanup.enabled</name>
/ c) k' ~; Z# G( [ <value>false</value>8 s- p; c2 D7 W Z
<description>Whether tasks should delete their task temporary directories. This is purely an& Y4 n* b4 N1 {
optimization for filesystems without O(1) recursive delete, as commitJob will recursively delete) B# ?" a/ P* k7 M+ `% E+ u
the entire job temporary directory. HDFS has O(1) recursive delete, so this parameter is left) y: J3 k1 s$ i r6 p" n1 r
false by default. Users of object stores, for example, may want to set this to true.
: a3 ?; a$ N7 p6 {. ~8 A& y; s Note: this is only used if mapreduce.fileoutputcommitter.algorithm.version=2</description>
& T! [9 u4 g, A# y3 Q% X g</property>
1 y5 i+ i; @2 {- |+ b9 e1 N( ^9 ~ y<property>" Y4 q9 \7 T B+ i
<name>yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms</name>
1 F! p2 b' D# M <value>1000</value>1 g5 F6 S! e# i. `6 ^
<description>The interval in ms at which the MR AppMaster should send
$ U2 f8 p0 T( y5 Z heartbeats to the ResourceManager</description>) a x" M4 l8 ^+ g0 X* O
</property>; k9 \" x! u( O
<property>9 D1 n, v Y2 y- X) v( U
<name>yarn.app.mapreduce.client-am.ipc.max-retries</name>
: Y' B# ^( ] q) H <value>3</value>
4 q" A5 J- T7 l! L# v <description>The number of client retries to the AM - before reconnecting+ C8 z' O0 G9 q8 B
to the RM to fetch Application Status.</description>* V; g+ l. c' D; G
</property>
3 w, o) \* v* ?- D; _: i; K- S& t* e( @<property>* m5 |% `; q8 v E% X! O6 ]
<name>yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts</name>
9 R* p. [& N `4 j8 f! s7 m, k& ] <value>3</value>
, C& {7 r( H0 m9 z L, P% s8 I <description>The number of client retries on socket timeouts to the AM - before
" s8 j5 U3 C+ o reconnecting to the RM to fetch Application Status.</description>, h" ^) E! I+ g
</property>( | h3 q1 i/ h+ A. Q' N
<property>, G4 [/ h5 L" B8 M9 k8 k( m6 e$ U
<name>yarn.app.mapreduce.client.max-retries</name>
( [2 J$ G: ]- O6 h+ V5 E+ X. ^1 z$ F( I <value>3</value>7 S3 a8 P7 S% d5 {5 R, |: `/ w
<description>The number of client retries to the RM/HS before3 Y7 m* J$ A$ }
throwing exception. This is a layer above the ipc.</description>
2 V; {5 Z+ O J6 i9 W% l</property> W" P: C5 Q( r. u& y- C* R
<property>( R) A4 h* _. D
<name>yarn.app.mapreduce.am.resource.mb</name>
; i: B K" |+ U <value>1536</value>
: N2 I# _7 h# B+ I' H <description>The amount of memory the MR AppMaster needs.</description>
% e. T8 J4 q# M* U5 `5 L+ a: K</property>! Q7 C: I- }/ _
<property>7 e* {( i7 z/ C' @1 B
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>. a; `# y$ ` C; Y' i+ g, g: H: F) ]0 q
<value>1</value>* a; F4 Y* S) Q# V7 c* ?7 X
<description>' N7 ]- u+ g. g7 I& B2 X
The number of virtual CPU cores the MR AppMaster needs.# D9 r, C7 o7 E( E/ W9 m& I
</description>
8 Y3 y9 F9 Q: a7 L6 \& T( b3 l</property>
`3 I! b Q& A. V<property>
; H. ~ k9 h% S7 j7 W9 l, ~ <name>yarn.app.mapreduce.am.hard-kill-timeout-ms</name>+ A. [* O6 Z2 w( @# Z9 [# @
<value>10000</value>+ _& E8 P$ e# t6 w! G; M
<description>2 t$ z* f* }8 ~* K. ~; d" i
Number of milliseconds to wait before the job client kills the application.; _" D" U5 G, [8 S* ~
</description>' T, Z6 r. n4 }; l
</property>, k- Q }5 v9 A4 }
<property>0 t8 d5 i+ T2 X( F+ {/ m6 ~" `
<name>yarn.app.mapreduce.client.job.max-retries</name>
/ w% E3 r& g6 Q, O, C" Q$ Q <value>3</value>% }+ X- {' S: m6 H3 _2 |
<description>The number of retries the client will make for getJob and
0 v9 Y3 e/ c, U3 T) C7 A- B0 d* I dependent calls.4 U) b& Y$ \7 }& n
This is needed for non-HDFS DFS where additional, high level; ~6 U# K1 K( }: m# M; Q
retries are required to avoid spurious failures during the getJob call.
1 u$ r4 t. |# O9 J$ m 30 is a good value for WASB</description>
% I% Y5 n& O. d5 Q( o</property>
) z% P+ U/ K1 G<property>
( o$ P# X; ?0 x3 j <name>yarn.app.mapreduce.client.job.retry-interval</name>
0 F: b8 ]: D6 Y4 C1 B <value>2000</value>
# b1 T' ]+ e0 y) d7 r* }3 i" ] <description>The delay between getJob retries in ms for retries configured
' N% X- s( _. u! t2 m y with yarn.app.mapreduce.client.job.max-retries.</description>* r* \4 Q' w" O S% [, c) C
</property>1 S1 q, D8 F0 Z$ c
<property> g, S" U+ O. s
<description>CLASSPATH for MR applications. A comma-separated list% v- C1 D3 @- M6 m
of CLASSPATH entries. If mapreduce.application.framework is set then this# A6 \; ?& c. x1 B4 n4 t {
must specify the appropriate classpath for that archive, and the name of
( F) v, B. _2 ^( H1 [ z% ~ the archive must be present in the classpath.
8 f# a' I5 [1 q If mapreduce.app-submission.cross-platform is false, platform-specific" F$ O$ J! q0 W; U: D
environment vairable expansion syntax would be used to construct the default5 p2 ]: D1 l' w% s5 [
CLASSPATH entries.
: [8 g; f3 e9 l$ ^+ |, B& H For Linux:
% q4 z/ c5 C, ]- f6 ^: g $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,4 o8 B) L+ M7 N1 G' ?- Y/ y' a
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*.- Y1 S$ K7 C" K
For Windows:
% u3 [( V0 w% ?, V2 i% Q %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,
! x, A5 C% d) v/ M, L %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*.9 _- r. Z, T5 C( o0 f {
If mapreduce.app-submission.cross-platform is true, platform-agnostic default
, P0 j, J" Z1 }9 a( Z; _& u! @ CLASSPATH for MR applications would be used:+ z- j" F/ x/ j. I! T
{$ W2 T+ n" W" |; _# f+ t5 B
{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,
! x; f; Z5 B, A* B: ]8 J# i {5 B% U1 _) n1 N* Y
{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*6 b ^ B8 N2 p" n8 q/ a6 o! ?4 Y/ Z3 c
Parameter expansion marker will be replaced by NodeManager on container7 B# X. S/ H% q V" b l+ C
launch based on the underlying OS accordingly.
; |8 N9 i& w8 B* e8 g1 D$ h </description>
4 {5 ]% j" E; @6 U( d" Z( }+ C( D <name>mapreduce.application.classpath</name>+ c! h& ~+ U( L a4 e) _3 P
<value></value>5 ]2 s% m' Q! V9 s/ B
</property>
8 K; w3 i" x9 r6 [<property>0 I" {5 g8 a: R" t
<description>If enabled, user can submit an application cross-platform
3 D# ]: V9 ~* Y" P4 g i.e. submit an application from a Windows client to a Linux/Unix server or
/ b @ G0 _# ]! G vice versa.) ]* Y9 Q7 F/ J& y( V8 B
</description>
8 h! N' f% G1 c3 G$ b+ L: T <name>mapreduce.app-submission.cross-platform</name>/ n! M. ~9 M8 I3 ?% a! K
<value>false</value>
$ a5 p; ?4 ~, ^</property>) W) x, p* Q( T# k, r( I
<property>7 U/ a7 Y1 Y' o: i1 Q3 U" m( o
<description>Path to the MapReduce framework archive. If set, the framework
+ X& V6 G: c% `8 L" T4 K archive will automatically be distributed along with the job, and this" j* v/ Y& }6 R* G% o: @2 |! n
path would normally reside in a public location in an HDFS filesystem. As
0 S0 |9 m; ?& D: O with distributed cache files, this can be a URL with a fragment specifying
) t4 G( [+ z& s/ B3 R& j# [0 c+ [: W9 Y the alias to use for the archive name. For example," p6 p$ k: C+ S8 B, \; w: u; G& m; m
hdfs:/mapred/framework/hadoop-mapreduce-2.1.1.tar.gz#mrframework would5 c+ a* D1 Y# R) j, g6 t- R
alias the localized archive as "mrframework".9 X1 P$ V5 t# u. o
Note that mapreduce.application.classpath must include the appropriate
& f( O2 q( k9 ^$ Z! g1 O0 P; Y% P | classpath for the specified framework. The base name of the archive, or
% U1 [: D b+ M5 x6 Y4 P alias of the archive if an alias is used, must appear in the specified
7 d; g4 }' [ f classpath./ e8 u' x4 b; M7 D. x4 b
</description>0 c" n+ Q9 a9 C; l2 p* v7 j
<name>mapreduce.application.framework.path</name>
* Q0 X/ E( ?; ]+ a <value></value>5 m7 K- l7 L* I, E9 J; z0 Z0 }
</property>
. _# S* F5 a+ B+ Y<property>
" M! d2 A2 V( m8 c( l! y <name>mapreduce.job.classloader</name>
& ?1 r8 t+ C+ [6 @ Q9 N- Q <value>false</value>+ e# ~8 |* K) Z" N3 `" M$ m( a7 j
<description>Whether to use a separate (isolated) classloader for
& U) @' j" h0 i2 k Z9 D1 F7 ~0 k7 B user classes in the task JVM.</description> d& q3 D* Q& N- H) |1 Y
</property>
& v M( X2 g# {3 J' Z9 q* v7 L<property>+ h) M% {' W5 l- c" O
<name>mapreduce.job.classloader.system.classes</name>7 K. w+ f: K; D( r8 e
<value></value>
1 Z9 R, \! M. Q T7 h" p$ l <description>Used to override the default definition of the system classes for
4 ^. t4 {* E% e: O5 r0 T the job classloader. The system classes are a comma-separated list of; n& l2 ~1 v' i" S& @3 u* |
patterns that indicate whether to load a class from the system classpath,- y7 U) Z$ M5 v1 E) b7 x+ J
instead from the user-supplied JARs, when mapreduce.job.classloader is! l0 E) _ s$ A, L# [
enabled.
; i- X9 }9 d. N+ [% y( w A positive pattern is defined as:1 `. }, P4 A5 [( A" \% k
1. A single class name 'C' that matches 'C' and transitively all nested
, s. g2 K3 ?. P8 J# S classes 'C$*' defined in C;
: ~7 n8 y3 A' @: h 2. A package name ending with a '.' (e.g., "com.example.") that matches' ]2 f# d6 C: Y5 {
all classes from that package. n6 Y2 H6 ~( W6 e+ ^6 u7 ?
A negative pattern is defined by a '-' in front of a positive pattern
" ] X% @2 \" E' @) u. F (e.g., "-com.example.").5 }8 C4 H/ _$ s" s2 F
A class is considered a system class if and only if it matches one of the9 f8 q& N$ ?3 O; R1 x u
positive patterns and none of the negative ones. More formally:
* h+ F$ M1 N" |' z: w4 Y+ I A class is a member of the inclusion set I if it matches one of the positive
3 N! f0 U/ }* z7 U% T patterns. A class is a member of the exclusion set E if it matches one of2 K u8 h! C2 W, `
the negative patterns. The set of system classes S = I \ E.) W; H, K% v R, U
</description>! p. M% Z/ B" W1 F9 Y' e
</property>
1 {, G& P. _; F5 v<property>! \& c q8 U" V4 b( h0 A1 I
<name>mapreduce.jvm.system-properties-to-log</name>$ r! h6 x: t6 D A
<value>os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name</value>
4 X( P1 a, K' k2 H <description>Comma-delimited list of system properties to log on mapreduce JVM start</description>, d1 L* j2 k% v8 g$ R5 t# ]
</property>9 u; ~' p* y4 y8 w6 w
<!-- jobhistory properties -->' {3 D" x6 W1 A: _4 j
<property>
0 V8 f2 F+ A9 V5 B& L5 r4 D5 g ?0 c <name>mapreduce.jobhistory.address</name>9 X" Y" T- @ Z4 E1 l7 S. y( k
<value>0.0.0.0:10020</value>) J1 Z+ |- T9 ]. y
<description>MapReduce JobHistory Server IPC host:port</description>
' h9 u8 H5 F1 ~5 K s0 N+ K* t</property>9 T4 P3 d1 r) ?$ z) ]$ v
<property>
( s0 E5 r" h# u2 r) F5 `9 {1 S3 W <name>mapreduce.jobhistory.webapp.address</name>2 M8 w% ?( g7 p2 v% h
<value>0.0.0.0:19888</value>
. a$ W" S! |! {4 O, O, D <description>MapReduce JobHistory Server Web UI host:port</description>
# M& v* A! x9 j8 H& X! B$ ?</property>' E+ T: y. R/ L$ _8 K7 |; O
<property>
/ k4 i, R2 e/ y) a- M <name>mapreduce.jobhistory.webapp.https.address</name>* u! H* u0 o2 p" g5 X$ m7 X" ?5 Q" i% C
<value>0.0.0.0:19890</value>. R4 |) r8 z( R: E# g4 N w- t0 j
<description>( L* O6 y7 R( J: H% p4 O- Q8 v2 Y
The https address the MapReduce JobHistory Server WebApp is on.& | p2 ?) f+ e* t* R
</description>
9 V: }% X! V4 H0 B0 m7 N' h</property>1 J1 V+ v' Y: x
<property>
& r- ~ t7 U: Z% q. Z <name>mapreduce.jobhistory.keytab</name>
) ~1 s: k/ j8 L <description>% t8 W2 U3 {& `( e+ ~5 V- P* ~
Location of the kerberos keytab file for the MapReduce
4 U5 R" H. r k0 n0 i JobHistory Server.8 r5 y. k/ u3 E6 W, @" X5 S5 N& E- r
</description>$ ]+ \7 P4 V. m
<value>/etc/security/keytab/jhs.service.keytab</value>
* A& h% H$ ~3 ?* i5 D) ?9 t</property>( J- r! `" m9 g8 i8 Y5 J( Q
<property>, F' B1 f9 H: U* x( B
<name>mapreduce.jobhistory.principal</name>3 n& B$ d" Z9 \3 x" @0 f( T
<description>
( m* w( ]7 o' q9 g Kerberos principal name for the MapReduce JobHistory Server." X# n, M- |: V3 n
</description>
) n2 s( B! }; k6 j& o0 m. k <value>jhs/_HOST@REALM.TLD</value>
: O4 l( Z6 Y2 F$ R! B4 f</property>- {6 z' n g+ Y4 |$ P( L$ S0 \
<property>) G4 w( l0 H _% s
<name>mapreduce.jobhistory.intermediate-done-dir</name>
+ I2 y a/ i3 Q) h- r/ T1 |" X <value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>6 ` L2 r6 M" Q2 ~1 L
<description></description>
, m% }- i1 G1 G' i</property>: W4 L4 r, Z! G+ o
<property>
9 p: F5 ]& _$ j2 @9 `8 I/ b <name>mapreduce.jobhistory.intermediate-user-done-dir.permissions</name>. q( ^; w9 T; o* u( d7 {
<value>770</value>
/ {$ U: B- D( y ] ~ <description>The permissions of the user directories in
! u6 y0 s( t* c% u/ T/ H; i" e5 V ${mapreduce.jobhistory.intermediate-done-dir}. The user and the group& r$ x8 i# ]0 n& p6 C4 s1 x; f
permission must be 7, this is enforced.) O* r/ a! g5 d5 F6 B" K
</description>, Z. z2 d0 K& {. g- d
</property>( v" b) s. ?; H5 z2 x$ y. n
<property>
2 s( L5 I) ~+ M' s2 d, R1 s <name>mapreduce.jobhistory.always-scan-user-dir</name>
% }- j# `% j5 F# {$ `' l <value>false</value>
4 e5 X- S7 g* ^3 w* p c3 \ <description>Some Cloud FileSystems do not currently update the4 P8 a) O1 Z" q ]1 {
modification time of directories. To support these filesystems, this
- c& v9 X+ M1 P( ] Q' T configuration value should be set to 'true'.
6 t: i. ~8 P m </description>' O; r' A2 G9 V9 F
</property># i% O6 a; u8 U$ R2 k4 V
<property>6 w: A$ L% B+ I6 [- b8 l
<name>mapreduce.jobhistory.done-dir</name>; ?- {8 C: P# c% f
<value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>
# D+ Y3 A/ m# V4 q, G+ J5 k <description></description>
/ G/ i* Q; r# D3 w9 M$ L& m</property>
3 m, P3 R' c; n @4 Y {- p2 N* Z: y<property>6 g- n6 M6 ]" \2 {8 J6 c3 ]
<name>mapreduce.jobhistory.cleaner.enable</name>
, A l' L. j, P& i+ ]8 Y <value>true</value>
' ^/ j) e! a( M0 G! Z% ]) q% J <description></description>, N' ^( @ Z) m6 z: X
</property>
0 R5 X" c5 E: I* }2 X/ d<property>
: m( w$ m) q5 l! W# x: Z <name>mapreduce.jobhistory.cleaner.interval-ms</name>
- [6 O- n# o' { o <value>86400000</value>( [ c9 t% s1 ^( z" x, J* l. J, ]
<description> How often the job history cleaner checks for files to delete,
2 h- j6 q+ M3 W( X4 g in milliseconds. Defaults to 86400000 (one day). Files are only deleted if; y( Z) a: [) o+ L1 y& A
they are older than mapreduce.jobhistory.max-age-ms.+ l' A6 ]5 ^. W( E0 \0 \' `6 Z
</description>" |, U8 i( `6 V/ n; g& V: b V( k* P% y
</property>3 A+ J4 V+ e- w2 ?8 P/ ?
<property>
- o1 `; x- X! U% ] <name>mapreduce.jobhistory.max-age-ms</name>
- K0 N* E8 i& N0 b, d4 X <value>604800000</value>0 f: D3 q* { Y! g' @+ Y: @' L
<description> Job history files older than this many milliseconds will) N, y7 P3 s2 t, p1 [
be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
9 y' I c3 h6 X6 W7 D9 o+ N: g </description>) V9 p( q4 O0 J, j, U
</property>% L- |" e2 v; b; ?
<property>! L. p `! r+ ]1 h$ p. ]7 T
<name>mapreduce.jobhistory.client.thread-count</name>
) w( P# k5 _, m) e: H4 D1 J) r <value>10</value>1 S+ R% B5 c6 K( `. R6 B& l- t
<description>The number of threads to handle client API requests</description>8 Q( b" n( p/ X( V3 ^7 ^
</property>- K! z; Z/ w( [# P' ^5 g9 F; p
<property>
5 M6 v; K" _9 B# M: k, j% A <name>mapreduce.jobhistory.datestring.cache.size</name>
) E/ |6 p- Q' x+ n' \+ ` <value>200000</value>
. k- O2 x/ g& w$ d' \! g& s <description>Size of the date string cache. Effects the number of directories
" S0 Z, f- {+ L* z& F7 ^0 | which will be scanned to find a job.</description>- r: g1 i7 w/ n+ d R
</property>
* t+ o$ z Y3 Z$ ~; x- k<property>. K/ g* d# i P/ `
<name>mapreduce.jobhistory.joblist.cache.size</name>6 v. [1 ^ s4 w. E8 C3 ]6 r* v
<value>20000</value>" [4 ]' h$ {$ r9 e* B
<description>Size of the job list cache</description>
7 y& I9 N1 }0 q</property>
' M) i0 X5 H$ M! W+ P<property>
, J$ [5 G' z, y: R/ d9 A% F+ s% K <name>mapreduce.jobhistory.loadedjobs.cache.size</name>
P4 l1 r# e# B& ^ <value>5</value>
2 \5 ?9 _3 m. T0 o: ^. w Y+ u5 t/ i <description>Size of the loaded job cache. This property is ignored if
2 C. g6 P! P5 e! F) M the property mapreduce.jobhistory.loadedtasks.cache.size is set to a
& l6 o7 [+ J3 i3 b% [6 \ positive value.$ l; W# N4 `* C$ {0 i
</description>
* m2 N$ l' e) E4 t6 ^% h0 x* v</property>
1 ~: I8 F o, q' C<property>+ F6 {2 T3 |. X* |
<name>mapreduce.jobhistory.loadedtasks.cache.size</name>: r8 z* F6 |. U5 D) I9 P
<value></value>/ {- Y6 i( B: `# [; c! L
<description>Change the job history cache limit to be set in terms; o5 y' E1 V2 \8 n/ e# S/ H+ s
of total task count. If the total number of tasks loaded exceeds% l& {8 p; S, r2 o4 H5 W7 e
this value, then the job cache will be shrunk down until it is3 }' R/ H0 }# |: S
under this limit (minimum 1 job in cache). If this value is empty! C6 n4 X! y( X" J' _! @$ ~
or nonpositive then the cache reverts to using the property
' F M e1 }$ o+ A; z mapreduce.jobhistory.loadedjobs.cache.size as a job cache size.
! U3 \! F3 P1 m) c: | Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size3 g$ Q! F3 @% n o) l
property:
# I1 H" r: r4 |6 s# c) j! | 1) For every 100k of cache size, set the heap size of the Job History2 W2 J. ? l0 C- b" N* K& A
Server to 1.2GB. For example,7 d' ~, ?3 b H* K r, D
mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB.
6 F s6 t: [2 d 2) Make sure that the cache size is larger than the number of tasks8 |& ?! E' e- z% R9 K/ h! h( i
required for the largest job run on the cluster. It might be a good
5 V2 l& y; P* m N/ @* W idea to set the value slightly higher (say, 20%) in order to allow' K$ M9 f9 x2 }& x
for job size growth.
& g5 R) l" D- g5 G" Z/ N </description>( ?# m" |. L; c. m2 P
</property>
# r6 f) u, I a( U; U1 L<property>0 e' C6 p! M/ H* }. x
<name>mapreduce.jobhistory.move.interval-ms</name>
1 p' S& \" e# h0 `8 a <value>180000</value> n) B+ Q8 W% s! i+ q6 y7 f
<description>Scan for history files to more from intermediate done dir to done
/ v. I4 D" g+ q. f1 u dir at this frequency.( N j' C. H$ I* p. z0 [
</description>5 ^* @' w) k$ h; R# S1 a
</property>: c* b' S, z* w( |0 n$ d+ b* e \
<property>
1 Y0 e" S" {7 s6 {* a: C1 [ <name>mapreduce.jobhistory.move.thread-count</name>
- ? L3 E8 t$ ] <value>3</value>- m5 m- C1 M! y: |5 r' N) y
<description>The number of threads used to move files.</description>5 l# ?1 x1 A9 [$ m
</property>1 x' i: |0 r, H0 f! Y, Z# |
<property>& {; s# b8 V3 o% \- I2 P& D0 k% a. j
<name>mapreduce.jobhistory.store.class</name>
0 k) B8 P) x1 T1 C! s: ? <value></value>/ n% T; q; Q! y, z2 E
<description>The HistoryStorage class to use to cache history data.</description>+ n3 P! Y& X. {1 [. e! a
</property>
; T% Z/ W, r2 H<property>; x; M$ m! i6 B" ^+ `7 ]$ A
<name>mapreduce.jobhistory.minicluster.fixed.ports</name>- v% }5 E* Z$ L; v
<value>false</value>
/ M& f# N1 u# |7 u }! r+ t <description>Whether to use fixed ports with the minicluster</description>
# ?+ n7 m/ u% V# G$ p( }* i- e1 s</property>
4 t& B: c4 N; _, t6 h6 ?4 L2 I<property>
. j9 j1 j/ y. E D" g* ?9 H5 U <name>mapreduce.jobhistory.admin.address</name>
+ f) S9 d0 v; P# |3 f$ R' p4 @ <value>0.0.0.0:10033</value>7 s. N6 n! m7 d! p) K/ A' j: [
<description>The address of the History server admin interface.</description>1 b) j) d: c. N( S/ U! r
</property>
6 V u! ~0 p Y) ^0 I# e* f/ f<property>
6 c" [& y& _7 P1 i& l) o2 x9 B <name>mapreduce.jobhistory.admin.acl</name>
6 T- q+ u1 B* d9 r& u: A& P <value>*</value>
/ I) [( M, f& d6 n5 H <description>ACL of who can be admin of the History server.</description>
( ?+ K9 J b x# Y* M</property>
! D& _. B6 a1 B, B/ A7 @<property>
# |) e! p8 j6 o5 k1 v <name>mapreduce.jobhistory.recovery.enable</name>. r( p; }! d, L) |
<value>false</value>: M( d9 v2 K) z: b
<description>Enable the history server to store server state and recover
% f) h: J1 y8 z5 n1 L server state upon startup. If enabled then
n9 Z4 t# |; q/ j mapreduce.jobhistory.recovery.store.class must be specified.</description>6 h) O8 `, b# f' t1 c
</property>5 L+ }) R5 D( s
<property>
% A" k# Z" V4 a4 |5 n8 g& u: [ <name>mapreduce.jobhistory.recovery.store.class</name>6 R& o7 M7 n9 P+ h' B
<value>org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService</value>
- i) g7 Z" R+ Q. D2 G <description>The HistoryServerStateStoreService class to store history server' A: ~ A5 r. W s5 Z l
state for recovery.</description>2 i# T0 Y# i5 V b6 b
</property>
& N) \. v7 k& ?* u0 i& u, B& L0 [<property>
, S3 G# l$ w4 O, d <name>mapreduce.jobhistory.recovery.store.fs.uri</name>
8 p. s9 \* W- t" ]/ Z* | <value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>( [; \# L" }1 O3 ~9 w: a- o' e
<!--value>hdfs://localhost:9000/mapred/history/recoverystore</value-->
7 u& ~- c# I' Q+ N <description>The URI where history server state will be stored if7 Z. ~5 a9 c# [- j$ X
HistoryServerFileSystemStateStoreService is configured as the recovery
j+ A+ m" W" g' Z storage class.</description>
y7 J6 U) f+ o _7 x. e6 i9 Y* D</property>6 j2 p& c, R% A& B, h- r$ E
<property>: g6 H7 Z. t9 ?) M% F
<name>mapreduce.jobhistory.recovery.store.leveldb.path</name>
. h3 |: S7 u9 b/ T6 E3 F/ ~ <value>${hadoop.tmp.dir}/mapred/history/recoverystore</value>1 J8 M) t/ S# x1 l5 w+ l
<description>The URI where history server state will be stored if. Z. U6 _8 v" K
HistoryServerLeveldbSystemStateStoreService is configured as the recovery
) v& H& \$ F# W9 N storage class.</description> k% C& ~0 x2 r( o: }
</property>4 W' b* ?1 j' A$ u( a
<property>
$ e0 L" Y; ~8 w- b' d: M; @ <name>mapreduce.jobhistory.http.policy</name>* S3 u) m0 g! i1 A, Z
<value>HTTP_ONLY</value>
9 y/ q6 k( W- ]! e( z- \ <description>% q! B2 p5 n6 l$ m
This configures the HTTP endpoint for JobHistoryServer web UI. M6 N! `$ T8 d% R4 [, _
The following values are supported:% D( e8 r+ _, l% ~4 ~: a/ j" m
- HTTP_ONLY : Service is provided only on http
2 |. F! F6 D# v/ [. p, x- h. l - HTTPS_ONLY : Service is provided only on https- U' x D! c, c. h( t$ Y: V, a
</description>
% {. K1 O6 c7 [' V4 S</property>3 ^/ C7 H3 k ^( a& k1 `) P5 |2 r
<property>
. Z, n, A. u+ Z <name>mapreduce.jobhistory.jobname.limit</name>
. r) s9 p& i1 }) f1 N* ^ <value>50</value>
( t4 `" r) S# q: r& A3 M0 ^7 O <description>2 f$ Q. h3 F; R( c! d. v7 s, _! t" |
Number of characters allowed for job name in Job History Server web page.# ]+ [! S# P# o
</description>
% R0 F( J+ x( H L- C7 s e5 s</property>
) Z- W6 q( w# B, }* ]2 w+ c& D<property>! \. `/ W' Y _8 _
<description>0 A0 n8 E( G) a& T4 Y. F* |
File format the AM will use when generating the .jhist file. Valid" |% U D% i) q7 z9 I* u
values are "json" for text output and "binary" for faster parsing." N* e% S! m. g( a4 q
</description>/ [( i2 I: A: s8 U5 T! d4 f4 q
<name>mapreduce.jobhistory.jhist.format</name>
9 v+ x1 w( L& I# e1 A; f. v <value>binary</value>$ A) a* T- h$ w+ @% l
</property>
( @0 Y) u! X* E/ R6 o9 d<property>0 ~( H6 i" G4 g3 w
<name>mapreduce.job.heap.memory-mb.ratio</name>2 [! e9 g& }; X6 Q* O J- r
<value>0.8</value>
5 E8 u$ M# m3 w9 `! i( k4 p <description>The ratio of heap-size to container-size. If no -Xmx is. a3 ?/ |2 W9 G& s/ o
specified, it is calculated as
! e: c6 r6 @9 G9 z3 q# i0 c (mapreduce.{map|reduce}.memory.mb * mapreduce.heap.memory-mb.ratio).$ p# ?8 j1 X: e( Z4 Y' ?) M( b
If -Xmx is specified but not mapreduce.{map|reduce}.memory.mb, it is+ { l: z$ I8 O. t8 Z) u" G
calculated as (heapSize / mapreduce.heap.memory-mb.ratio).
* L! u4 C( \2 c( o. }0 Q. B4 q8 o </description>5 Y+ K+ g1 G+ m( \
</property>& A7 [. c) Y. s- ?6 L6 y4 a
<property>
# C: U) U* _& ?8 m <name>yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size</name>
% q" ~" L& t* a9 _. W <value>10</value>
5 p, v; v6 K: `# D( M: m <description>The initial size of thread pool to launch containers in the# {5 y: R: Q# `
app master.
- ` z- u4 k7 K8 g9 Y </description>
$ E) W" F; P! F1 y</property>
' {8 H/ [" x; I<property>
7 v5 A3 i8 U) Y! H" M <name>mapreduce.task.exit.timeout</name>4 C' H6 a4 e N0 T% F. M
<value>60000</value>$ @4 m+ D; l. c( Z) n+ [
<description>The number of milliseconds before a task will be w5 A: {- L2 Q. Q2 p+ \ p
terminated if it stays in finishing state for too long.8 p6 d1 l" d, A: F
After a task attempt completes from TaskUmbilicalProtocol's point of view,
3 z! \. `$ R% @0 |" Q/ T. s it will be transitioned to finishing state. That will give a chance for the
# a* F' B. ~: @. G" _ task to exit by itself.
% o/ B% v+ Z. U0 H; n* G! o0 v </description>4 g! O5 T4 Y, n8 s
</property>4 d' ?8 s6 u4 I( L% Y
<property>
2 K; l" s) c& i7 q <name>mapreduce.task.exit.timeout.check-interval-ms</name>
; w1 D7 @& M- W/ ~, n <value>20000</value>% w, c- H h! P* v7 f
<description>The interval in milliseconds between which the MR framework1 \6 b k. f$ e# |. b
checks if task attempts stay in finishing state for too long.
0 e$ B2 {. L( B h4 u# ^6 Z8 s </description>
H3 i1 c2 X- `6 r) s- o</property>
5 w: \0 h! i8 D<property>- {4 {* A" f' k+ Y1 s! G V- N
<name>mapreduce.job.encrypted-intermediate-data</name>* _" P- o" E# ~! o
<value>false</value>4 T$ \4 R& p, u
<description>Encrypt intermediate MapReduce spill files or not
) [( v5 F+ j' t# g3 [% L, g4 D default is false</description>
% b$ \: _8 T! T; N) f</property>
( A/ Q: A! K) n" |<property>. n% M# P- b) r6 `2 K* J
<name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
- @- Y' q: F7 S0 k8 Q- [, s <value>128</value>9 ?! U! N/ n& k4 ^
<description>Mapreduce encrypt data key size default is 128</description>4 J5 X$ Z% H; y* i! d4 c
</property>* n8 s; _9 W/ f8 N" S
<property>
: \0 d& V' v1 W8 s <name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>
" T& M% E- o' d: E+ r" ~- g <value>128</value>( N& B% h: D$ n, b" M4 Z
<description>Buffer size for intermediate encrypt data in kb
) H* H( [' a9 k. R; ` { default is 128</description>
7 T" M) A( _- X+ d1 p6 m</property>
" o" w$ ^$ j- @6 @1 N<property>
6 i) K. h4 n4 M3 q4 P <name>mapreduce.task.local-fs.write-limit.bytes</name>
; s& V: R4 j$ N, ]5 h. d <value>-1</value>1 ~) R& I% b1 U# \
<description>Limit on the byte written to the local file system by each task.9 y" v; Y6 V! \% p$ _. O1 d
This limit only applies to writes that go through the Hadoop filesystem APIs+ u- N0 p( {7 \
within the task process (i.e.: writes that will update the local filesystem's
+ @- [; H. _, \- ?: Q4 d* H2 l BYTES_WRITTEN counter). It does not cover other writes such as logging,
; g, s" l) [4 ]% [( b* y& k sideband writes from subprocesses (e.g.: streaming jobs), etc.
4 n1 N5 B5 Y$ ]8 K' _ Negative values disable the limit./ \! Z, h( h/ S6 ~8 F
default is -1</description>
5 e* J# J2 ^; e! O2 K( }; R</property>
$ {$ X4 U5 f, n<property>
$ W1 T# s3 p# p <description>. }5 e1 k8 S, c4 M
Enable the CSRF filter for the job history web app! P1 \4 Z$ E$ \1 N* u
</description>
) [$ R; @5 k! K$ B* A, p- ] <name>mapreduce.jobhistory.webapp.rest-csrf.enabled</name>
: ?9 {9 t+ N7 f# s- T <value>false</value>
8 d9 U* [! A! d$ S+ s5 p</property>
& c7 T4 @" S7 n- _1 i" \<property>
' c$ i E% W* k7 I( s <description>
7 D" n; ~1 Y( ^8 C, l g& Z6 }/ { Optional parameter that indicates the custom header name to use for CSRF
- ^0 t! F1 u9 M$ \9 ?7 L protection.2 {" {& k( t c3 j- h
</description>- L& D$ _. z9 S. k7 A
<name>mapreduce.jobhistory.webapp.rest-csrf.custom-header</name>
, O' }, D- o$ I8 c! Q/ A! p <value>X-XSRF-Header</value>
$ x* Q$ M. j3 E: X, a+ {</property>) h5 J; g3 m& E% }# C" S u7 l& N
<property>: z4 _- g& b+ ~' h- }! m
<description>
" p4 E9 ^! Y% {8 L' Q Optional parameter that indicates the list of HTTP methods that do not
& A5 y! m T8 E) { require CSRF protection6 u; Y4 g- b0 o' E
</description>8 N; | n* F# J' e
<name>mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore</name>* u5 I# y# Q+ {6 {* F D9 }
<value>GET,OPTIONS,HEAD</value>* b- H! X1 U4 l* C! n9 J
</property>
: V( q" c% y7 M+ |<property>
) P+ T9 o% N5 u" t2 L; ]5 U$ j) A <name>mapreduce.job.cache.limit.max-resources</name>
( T" z9 x/ g. ^+ { <value>0</value>6 a. M9 I* G7 K
<description>The maximum number of resources a map reduce job is allowed to
7 W! b1 x: W; p" U9 @; O5 |1 e submit for localization via files, libjars, archives, and jobjar command" n; q7 }+ a0 b& Q. o. Y
line arguments and through the distributed cache. If set to 0 the limit is
3 Q7 i6 G) a% K1 k* G, m l ignored.
s' g4 s% ~, T </description>$ L' ^" N$ c" z/ k- w+ Y; H
</property>
& S, _! e) L" g# _; K( [<property>
/ l9 F- V7 e b7 R <name>mapreduce.job.cache.limit.max-resources-mb</name>
5 C8 i" R" n0 n! h2 u2 ~7 p3 o* S <value>0</value>& }/ i1 Q8 u; g4 C* J
<description>The maximum size (in MB) a map reduce job is allowed to submit" j4 l: M2 Y/ |& L/ o: m3 Y
for localization via files, libjars, archives, and jobjar command line, w g" d; h, S
arguments and through the distributed cache. If set to 0 the limit is
6 E) ^6 _, L) o9 i/ n+ s7 Q ignored.0 q$ }* p3 U- f7 g
</description>( y' a% ]# C0 n. d) L- M3 e& T4 ^
</property>
0 n6 V* h7 }5 G3 f% P- S1 O! ]<property>) D# W8 v: o6 z1 D u
<name>mapreduce.job.cache.limit.max-single-resource-mb</name>
i5 K4 c" w3 g4 W <value>0</value> s; m! z P+ @
<description>The maximum size (in MB) of a single resource a map reduce job, E9 m* Q+ I5 A+ g" ]' B! H( a8 l
is allow to submit for localization via files, libjars, archives, and
( u5 ^4 `8 @% x/ X' D jobjar command line arguments and through the distributed cache. If set to
; ~/ M* A9 a# S# _3 O 0 the limit is ignored.
& Q7 L! ~% k+ m+ Q- ] Z* A </description>( @* \% v* {0 Q$ q1 u" V
</property>
4 ~, c, U$ v9 @1 X$ }; K, b3 v" \<property>
/ o/ W$ [1 x/ z: ] <description>
8 \4 ?1 g. c3 g Value of the xframe-options
) ]' i, e2 S2 a; D o& ^# i </description> J* f# q0 v' x2 b
<name>mapreduce.jobhistory.webapp.xfs-filter.xframe-options</name>3 R8 F5 N3 z& {1 I( k8 ]& O2 s& h/ G
<value>SAMEORIGIN</value>
, K6 z0 ~ v. ]& j</property>
4 e& U7 j+ ?) f6 B$ f<property>
; f, M( H! o) m <description>! K; w3 |& X0 V8 B. p; \
The maximum number of tasks that a job can have so that the Job History
H8 W6 ]# Y" `6 t Server will fully parse its associated job history file and load it into
. ^( M+ R; L6 Z$ q: ^ memory. A value of -1 (default) will allow all jobs to be loaded.# p- w& c' Y; c$ ^. s4 _" W4 X( F) \
</description>
) k! ^; Y o- e <name>mapreduce.jobhistory.loadedjob.tasks.max</name>% c! Z, _/ K% r6 a$ g8 Q) v
<value>-1</value>
; `7 u( Z+ ?: y! u</property>+ `! T" z1 f5 Z, M# @$ d1 S( A) R
<property>* f8 v+ j' k# O9 P1 E8 O
<description>
( A5 f/ K5 N+ e: w The list of job configuration properties whose value will be redacted.
6 b* l9 S+ }- j4 } </description>
1 ^% u4 V) R* \& x: G5 `8 p <name>mapreduce.job.redacted-properties</name>1 M. ~0 q5 L I; ~3 w
<value></value>
% g. s1 r8 u- L, V0 \</property>
$ [" ^5 y! @$ d* o6 T7 L/ `<property>
0 y+ ?6 M; G! z3 P% G; B+ n$ d <description>2 Y, z$ M$ C( o) z# X- x
This configuration is a regex expression. The list of configurations that
; [! i! w4 X4 q; X& f: h3 J0 W1 F I4 A match the regex expression will be sent to RM. RM will use these: G+ ^, I3 n1 F9 `$ a5 s
configurations for renewing tokens.
4 x9 T0 H. M( m4 ]5 l- {& V This configuration is added for below scenario: User needs to run distcp
3 W5 u0 y# k! ? jobs across two clusters, but the RM does not have necessary hdfs
/ h6 n1 m* ^0 J. b) @ X configurations to connect to the remote hdfs cluster. Hence, user relies on& {1 ]* u2 Q- [& m z5 K3 l3 F2 h
this config to send the configurations to RM and RM uses these
' K2 t: Y. T. T7 A" | configurations to renew tokens.
- ^* d' o, _; h* @5 W$ i2 N- W For example the following regex expression indicates the minimum required
# \8 ~ Y8 R) _) L; _ configs for RM to connect to a remote hdfs cluster:
- k8 @8 K9 \+ x0 ^ dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$|dfs.namenode.kerberos.principal$ U0 ^, N+ T& w9 J8 v3 ~5 |, {
</description>
; o9 L% [! s" w. U/ y+ l# U <name>mapreduce.job.send-token-conf</name>% P' ^1 ]+ d6 Q' j! f7 G
<value></value>
! U( j5 g4 z- R</property>
* w! w( ?; W1 b5 |1 I2 \8 M, P- k<property>
# a s+ X v. T <description>
) p- v2 F7 {: L- U( P, L( t The name of an output committer factory for MRv2 FileOutputFormat to use" E) ]0 Y3 t g3 G
for committing work. If set, overrides any per-filesystem committer
9 f, q3 x7 ?3 @9 } defined for the destination filesystem.
% T1 L1 g# j; E% E( a$ x! X' p: {7 K </description>
q% t# g# l! C# m1 w9 |5 Z k U <name>mapreduce.outputcommitter.factory.class</name>8 p! S* D" a6 l' @' `/ r
<value></value>
& }' M$ N+ h& `1 |, S</property>
* C$ N4 J$ Q* S# n% A! I# x7 x<property>
5 ?3 Q$ {$ M0 O$ k" C) Y2 F <name>mapreduce.outputcommitter.factory.scheme.s3a</name>
' s; E/ O/ f) e( F% Y2 c0 V6 }/ m <value>org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory</value># H. m4 } ]( P" U. H3 o& A
<description>+ S4 w0 Z# n7 j2 l
The committer factory to use when writing data to S3A filesystems.
% P% R% Y- f) n, {# ?- i$ S If mapreduce.outputcommitter.factory.class is set, it will
s# [' V# w' v% ]1 A- J# X override this property.) e: t$ x% X; H' @4 p
</description>( ^: } J. o4 c# t: \* N, U' M
</property>' x7 M9 t4 r, z1 U. L
</configuration>
0 c1 Q) d7 ]3 x% o- K4.yarn-default.xml
4 M1 A9 R% r( M& p
- u! P2 q# {; A; Q9 f0 R<?xml version="1.0"?>
$ V4 ^2 M' H, G( w* ?<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
* M6 f; O" k8 E' I; z' m0 n9 r<!--
$ N' z; N3 ?0 B; T: _+ L Licensed to the Apache Software Foundation (ASF) under one or more* q/ h* j- E! n1 W: S
contributor license agreements. See the NOTICE file distributed with! n! u; T; Z: G
this work for additional information regarding copyright ownership.
! v; p" i1 W6 C* _% G% ^' G/ X- d. v The ASF licenses this file to You under the Apache License, Version 2.0
( S- _4 h+ D. }2 a8 T9 _ (the "License"); you may not use this file except in compliance with: |6 u! X* o$ W1 `, U
the License. You may obtain a copy of the License at
0 r9 ^: w% q! j* z0 r- Z6 H http://www.apache.org/licenses/LICENSE-2.02 a8 y5 T! V" Y0 \$ |& E1 a
Unless required by applicable law or agreed to in writing, software
% i w# `7 B. ^2 y) O( w distributed under the License is distributed on an "AS IS" BASIS,0 j2 g5 z6 }2 j* W O
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.9 r+ V8 O$ X1 k
See the License for the specific language governing permissions and
T- P. h/ c( O* [ limitations under the License.
8 L* A) {) C! G. R-->$ |- p8 F7 y" E$ V+ M
<!-- Do not modify this file directly. Instead, copy entries that you -->
4 g0 K1 s/ q6 ~: x5 f/ @' i: G<!-- wish to modify from this file into yarn-site.xml and change them -->
: |; _9 x; J! w4 |" @<!-- there. If yarn-site.xml does not already exist, create it. -->. v1 L9 H& R$ ? R' ^' B6 X1 b1 b
<configuration>
) O5 L2 v/ x/ V/ w% D <!-- IPC Configuration -->/ ~6 W- w* Q$ c, `) O; v) a3 Z
<property>$ k! L2 m/ u" b6 `2 @( v- h
<description>Factory to create client IPC classes.</description>
2 T( D0 d2 V8 F& f' B <name>yarn.ipc.client.factory.class</name>( ~2 m: v- U5 e7 A) I
</property>
6 W% E/ M1 |1 ]0 T$ M <property>
( S, S3 l, L: Y <description>Factory to create server IPC classes.</description>
. v; J9 W0 ~( v: ? <name>yarn.ipc.server.factory.class</name>. W" L7 I+ M' @' f5 S$ v: k2 R% h
</property>
g, R$ E/ y, W1 @: a& e <property>
& E6 p: D# u' J$ d( _ <description>Factory to create serializeable records.</description>
9 n* T" ?. L: \' t <name>yarn.ipc.record.factory.class</name>3 l& R( d. y' U" t( v
</property>& C8 I: Q( L* H3 z! O- S2 t
<property>
9 Y6 K o7 e T6 L: z <description>RPC class implementation</description>+ k4 N, n7 Y; w! {6 ?! R! B
<name>yarn.ipc.rpc.class</name>
0 @5 |9 j9 J: D <value>org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC</value>
, r1 I9 N$ ~2 K- P </property> F, U! p. Y4 \& w
<!-- Resource Manager Configuration -->
% H1 H7 F+ T# k, c5 ` <property>0 G% \' p& b; m! P( ?
<description>The hostname of the RM.</description>5 v$ z6 u# j$ f3 f5 P* a
<name>yarn.resourcemanager.hostname</name>
8 E" K1 e5 X7 B8 V; T: r <value>0.0.0.0</value>* E' u$ \8 \) l& D/ q2 X
</property>
) h6 T8 b: `2 l/ f <property>; g9 ^8 F! `2 f7 ~1 Z) j
<description>The address of the applications manager interface in the RM.</description>* c( M. N7 H, w# |
<name>yarn.resourcemanager.address</name>- Z5 \# U& U4 _. q% S+ R. r
<value>${yarn.resourcemanager.hostname}:8032</value>& ~) |# u7 b4 M
</property>; A8 f$ N* u& R: _! b6 ^3 y0 t
<property>: S. m H/ `4 h ^* @
<description>2 q( I# [5 U- G: c# i" G
The actual address the server will bind to. If this optional address is
, R2 T6 C. H3 g! Z. [' Q set, the RPC and webapp servers will bind to this address and the port specified in
6 f. R: Y% h" W3 C- ]3 \ yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This% A4 A( p% h1 h, k+ B9 T% @6 G& F( W
is most useful for making RM listen to all interfaces by setting to 0.0.0.0./ `' f/ p! Y* L4 N' }! k+ K0 _) Y- v
</description>" K( q4 B$ v8 ^
<name>yarn.resourcemanager.bind-host</name>' `3 P& [5 @ f0 M1 S( O
<value></value>
/ Z+ ~$ X) p# v) V: t+ o- j% } </property>4 T! z' e6 V0 x! o
<property>
1 @ W; p3 j8 U, x w <description>! J! i) Z7 p' Y8 k5 M$ M
If set to true, then ALL container updates will be automatically sent to
* m3 ]8 p% E+ b# E1 l! f6 \ the NM in the next heartbeat</description>
1 d- k. ~0 r8 i6 A/ C* e1 b <name>yarn.resourcemanager.auto-update.containers</name>
# K7 g. Y) c& _2 @/ L b2 n" Q <value>false</value>' x) I: a2 _4 O4 e
</property>
3 |' i2 {& G6 Q- q1 n; a: a6 l <property>
! G3 b; z B1 n6 {- P8 ] <description>The number of threads used to handle applications manager requests.</description>; L4 v" b; {& r
<name>yarn.resourcemanager.client.thread-count</name>
( U7 k4 q5 `. |8 j <value>50</value>
F* O4 E9 P( g% z, Z" m9 m2 O4 u </property>
5 U. |$ B0 M" j <property>
3 l% z- Y8 B7 P/ M& N+ s <description>Number of threads used to launch/cleanup AM.</description>
) \! L, ^$ [! V L/ ^ <name>yarn.resourcemanager.amlauncher.thread-count</name>+ j. f, U$ T4 \2 ]
<value>50</value>
\$ S0 }" h5 j5 J </property>
8 Q! K( Y5 ?- V/ ], V- C <property>
6 h) Q$ m* k1 M2 h: ^( t <description>Retry times to connect with NM.</description>
6 m& U6 s+ W( r4 h+ E8 o <name>yarn.resourcemanager.nodemanager-connect-retries</name>7 E3 n- z1 s& Y; Z* ^3 i" q5 ?
<value>10</value> d* m$ \& F# q9 r a( Q
</property>
6 p2 M. e( x2 e( b' j& J <property>
, z) U% j( o- f8 [& _/ \0 v <description>Timeout in milliseconds when YARN dispatcher tries to drain the/ p# y, t L4 G/ E/ Z3 _$ h1 k/ u
events. Typically, this happens when service is stopping. e.g. RM drains7 r) R0 s3 X* q. L/ n+ O& E
the ATS events dispatcher when stopping.
E" x+ c2 h8 [% O' s* L. U </description>
! K+ E& H# W; ^( u8 Y$ ?- r <name>yarn.dispatcher.drain-events.timeout</name>3 y4 e6 k- k: _& B `8 s
<value>300000</value>
6 ?6 E! g W& v+ \" _ </property>$ Z& z- C) y* ~" |
<property>9 o; R. E9 e' u2 |
<description>The expiry interval for application master reporting.</description>
3 A3 x9 @3 K8 f/ \4 E; w <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
9 l3 }: ^3 Z* p) r5 h' e <value>600000</value>) j2 I1 A' h( o/ D s
</property>3 _7 |' Z- n/ M; b) p
<property>
: Q4 K, [ p8 F. F# O <description>The Kerberos principal for the resource manager.</description>" ?& a' U P6 Y/ }2 Y: Y) X& J
<name>yarn.resourcemanager.principal</name>
8 a1 {6 z# R% [9 l; b </property>
# K" O3 j' {( M5 Q: A4 g4 t <property> _) x: h( T0 Z7 U* K4 t+ J$ N0 {
<description>The address of the scheduler interface.</description>
) k5 }/ h& G t$ c <name>yarn.resourcemanager.scheduler.address</name>
/ Y3 R: O: m5 |" W9 n <value>${yarn.resourcemanager.hostname}:8030</value>, X6 e! E9 q' l; X3 I
</property>3 F( C* ^) D5 a: O' V- L4 n
<property>
f$ \/ }1 W& |9 S# \ <description>Number of threads to handle scheduler interface.</description>, a# q& X( G5 S3 N
<name>yarn.resourcemanager.scheduler.client.thread-count</name>7 q2 M/ T4 v5 t2 L, C, [& }- y6 n
<value>50</value>
: M( v! n2 s5 ~( f </property>/ X+ k- I" f! g5 `0 d. P- \
<property>+ {8 O4 T; Z' M5 k
<description>
( e) e1 [) y! z l5 b1 Q$ ^( G8 m Specify which handler will be used to process PlacementConstraints.
# U" n1 k) _) x Acceptable values are: `placement-processor`, `scheduler` and `disabled`.
9 ]0 b% k. Q* i3 | For a detailed explanation of these values, please refer to documentation.1 W5 |$ \+ N$ F$ \( c0 c4 b/ ~9 Y& Z
</description>
6 f3 L7 _ g' r <name>yarn.resourcemanager.placement-constraints.handler</name>1 y0 B) ]! U! G; ?
<value>disabled</value>
0 n, j! K- Z' S! K3 O/ {" q% [/ z </property>, _1 \! T9 n9 |. `
<property>
, i' [ i* D9 K3 f. R1 r3 ] <description>Number of times to retry placing of rejected SchedulingRequests</description>& e1 M6 J) X* |) w, ~9 r, k
<name>yarn.resourcemanager.placement-constraints.retry-attempts</name>
3 ^' x, _' h+ o0 W9 e <value>3</value>- ]- ^; ? o) q# Z! R
</property>
s6 g% {' j! J: J <property>: `9 U- W/ f( p4 w% Q
<description>Constraint Placement Algorithm to be used.</description>
, K( ]0 T6 T# D$ z9 C+ J# f+ v1 R <name>yarn.resourcemanager.placement-constraints.algorithm.class</name>
3 q9 Q" y8 ~, e9 B) D6 U <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.DefaultPlacementAlgorithm</value>
, t. B' [$ h& Z3 ^6 q4 d </property>; e$ X" Q/ N) I7 w D+ T
<property>
6 Q- h8 O4 v! o2 h- X <description>Placement Algorithm Requests Iterator to be used.</description>: e0 Y9 ]6 q! z: n7 K% ]
<name>yarn.resourcemanager.placement-constraints.algorithm.iterator</name>1 K& Y w/ J/ A @- w6 q& ]
<value>SERIAL</value>
/ s& e6 W; d% e </property>6 T6 c) o+ a% A. N
<property>+ v4 k( g9 m' p% M% d E4 N4 p
<description>Threadpool size for the Algorithm used for placement constraint processing.</description>9 Y6 V. L& n( a% k$ C
<name>yarn.resourcemanager.placement-constraints.algorithm.pool-size</name>) a! Y6 L9 i$ p3 v+ ?- x
<value>1</value>
! j/ Q8 L) j6 O. R; m! D- j </property># _: w. o% c' \
<property>& C7 {8 S. v7 z+ {* y. T' U
<description>Threadpool size for the Scheduler invocation phase of placement constraint processing.</description>/ d$ B! G4 |" h
<name>yarn.resourcemanager.placement-constraints.scheduler.pool-size</name>; S) O/ i$ k. E# q
<value>1</value>
+ c0 M% F3 e- U$ u </property>5 y1 `4 l8 c( ~$ J, T( r/ Y
<property>
# u; K7 S+ N3 L* ~ <description>6 p# c- H T- g# p. B* a& s
Comma separated class names of ApplicationMasterServiceProcessor' @/ |. j$ M* ^2 q) ?) H
implementations. The processors will be applied in the order1 X% ^! J8 N) j$ |% u. e/ A( [, i
they are specified.7 e( q Z) f) O% h
</description>
8 L9 J, E8 l% Z <name>yarn.resourcemanager.application-master-service.processors</name>
; z% B+ U0 n1 C' R) M <value></value>0 o$ K/ O+ R: t7 K, F) ~
</property>7 \. [3 ?9 o' l, H, \8 _5 Q0 V
<property>
+ C3 }+ X8 ~% \9 a- D0 [ <description>( G8 f3 s2 C+ P# m8 i4 h; x4 T- ~) k
This configures the HTTP endpoint for YARN Daemons.The following
" ?/ j: X5 ]' I* k1 r9 v/ G8 t values are supported:6 z7 g# D9 P: ^ c2 F: z' l! l7 N
- HTTP_ONLY : Service is provided only on http
# ?9 O+ e* o9 ?; o) f - HTTPS_ONLY : Service is provided only on https# p b) p* L# Z
</description>
+ }) L. G7 B- H- c F <name>yarn.http.policy</name>
K$ d5 M+ ?" p3 O0 f, I% W7 v <value>HTTP_ONLY</value>
% f' F" |9 _3 X" u7 d- W) y </property>
( F$ V6 X+ L$ n# h" ~* L) l% U# t$ } <property>& `4 \* G# V% z* o# c8 G
<description>% N3 e) N& u2 p
The http address of the RM web application.
/ H! o+ R" } i4 I( _! m; J If only a host is provided as the value,: i' }1 y+ r0 p- g7 o2 `# I
the webapp will be served on a random port.
" O3 l1 L T; S0 o3 C </description>
$ B- U9 G8 | h0 ?" _, X1 p* r. Q <name>yarn.resourcemanager.webapp.address</name>/ D2 l* Q9 ^! k2 d( l
<value>${yarn.resourcemanager.hostname}:8088</value>( g* K3 H( L e& a3 ~8 E
</property>
! g* ~$ o1 G4 O% ~* j% a; k <property>& G, U3 O/ \3 X! P$ g3 F
<description>
% p% W; p/ {' L5 r The https address of the RM web application./ d1 }0 e- @. ^$ _
If only a host is provided as the value,
8 A6 a3 D( y" x( c the webapp will be served on a random port.: J1 T6 X- h" I$ T+ n
</description>- G+ s' `9 V% o, n' I' e4 q6 x2 P
<name>yarn.resourcemanager.webapp.https.address</name>
, z' q2 u. C6 ?% | <value>${yarn.resourcemanager.hostname}:8090</value>9 H) v' G5 X8 J" U
</property>
9 r0 w/ U) t# Y: a <property>, b1 S( |, w" @* D+ o: `
<description>' G2 t1 p3 z1 e k- ?- n& k9 k. z( N
The Kerberos keytab file to be used for spnego filter for the RM web2 a2 U& N G/ g) W& y
interface.
. \! q6 W/ I9 `0 ]& Q </description>6 V" t7 v. l8 v0 K: O+ E
<name>yarn.resourcemanager.webapp.spnego-keytab-file</name>
5 M5 ?. a5 g/ T6 t9 y <value></value>
# o8 {' s* _1 N$ F </property>
' g% {( g# a; R& g <property>$ ^2 W! w+ G6 F- a1 O0 _5 K
<description>
# `' x) a9 ^% P The Kerberos principal to be used for spnego filter for the RM web1 m0 c( v8 L+ S8 e; q
interface.
# r# [" n3 ?0 b1 V. \ </description>; I2 v/ N" `& V% n8 R
<name>yarn.resourcemanager.webapp.spnego-principal</name>
F+ F. `/ A9 H: m4 H4 ~3 A& o( _ <value></value>' M7 i! J4 o' A1 T0 o2 r m5 C
</property>
: g2 T) Y5 G0 i: u' ]0 r8 h- S <property>
& x" ~; ?$ q T9 L; e+ K# Y, d# H9 c <description>
b1 Q3 [ i; a& L: s7 h8 R1 H Add button to kill application in the RM Application view.4 o- G3 l: }2 n" X# L
</description>
- H$ K2 _' Z% B4 ^+ J <name>yarn.resourcemanager.webapp.ui-actions.enabled</name>- U9 i) b1 m! K! F, g$ l% Y
<value>true</value>
7 ^' O' |9 E d) {; k </property>; q# H5 [7 q* }9 ^5 X
<property>) i' Y3 X" O: }7 l! R @
<description>To enable RM web ui2 application.</description>9 }$ h8 X3 H0 L3 u4 a& w
<name>yarn.webapp.ui2.enable</name>7 M5 W# k! n/ P7 p/ L# D2 j1 A
<value>false</value>
1 ^* [+ r" T% x1 T. |! S5 Z+ y8 v9 K </property>; o- L' h; n4 t# a6 i+ D
<property>
# V2 N! S+ D; q5 o <description>
# F* ? G! w# @ Explicitly provide WAR file path for ui2 if needed.$ V: u9 \- I) d/ R2 o% n
</description>! t& b2 X* h; u
<name>yarn.webapp.ui2.war-file-path</name># N& E$ ^# s0 \2 F+ W; C
<value></value>
% M7 q- @$ H2 Q </property>
9 U& ~& `$ L7 R; m4 h6 f) s& y <property>
% ]' K+ I/ C$ E( @) Y% O# k <description>
2 ~- y3 R9 j1 N8 e n, `1 v0 s Enable services rest api on ResourceManager.
8 b1 ~2 O* x6 a" v </description>
' \; C" b$ c: W, X1 P. \* J <name>yarn.webapp.api-service.enable</name>- O* ^* G7 h( x
<value>false</value>7 n5 z0 t& m0 A' r7 J. C
</property>
; C# x2 b2 g) a" i <property>
* S+ Q4 O* u, K! \; P G# m; z <name>yarn.resourcemanager.resource-tracker.address</name>3 D- F+ h/ p# M) u6 z
<value>${yarn.resourcemanager.hostname}:8031</value>0 |- E# z; H2 f) X; Y/ R4 o
</property>
3 c1 b2 a7 O$ L6 g/ \ <property>: Z! a) T& i6 D! q: ]1 n! P" S
<description>Are acls enabled.</description>
6 h: {8 k) S3 A <name>yarn.acl.enable</name>1 m0 k# B# h r' ?4 g# v
<value>false</value>
9 a* m3 q+ C) @$ l </property>6 n" D' s8 E( k7 K# W) w( B
<property>/ M! Q. X' c. d# [; b; a
<description>Are reservation acls enabled.</description>4 T& Y# \4 W5 g3 P7 {7 E4 M
<name>yarn.acl.reservation-enable</name>; E4 j% _, N& `0 `; d0 i
<value>false</value>
1 I$ p, s/ A7 K, f6 r# o4 Q </property>( K! q, g: E8 {6 m4 V9 b& t) H+ j
<property>
) z. i R( @1 p+ y" E; F' ?/ J <description>ACL of who can be admin of the YARN cluster.</description>
9 v5 A& o0 N) E: j4 s4 P <name>yarn.admin.acl</name> c- ]! B# }, p- J3 Q
<value>*</value>' d( a% J7 x- P
</property>4 H$ J* Y; Z) P0 Q
<property>* [# g# n( J0 U1 _6 k
<description>The address of the RM admin interface.</description>$ C ~" P7 M) V/ o7 V
<name>yarn.resourcemanager.admin.address</name>1 p3 s1 u2 P% h0 f2 f
<value>${yarn.resourcemanager.hostname}:8033</value>
# z: i0 F7 T" U" |1 r </property>
) n \- L: o s9 K4 r5 o9 E: V <property>
9 [6 j, H9 F( f8 }8 P <description>Number of threads used to handle RM admin interface.</description>
4 y% o) y* a$ N1 q <name>yarn.resourcemanager.admin.client.thread-count</name>
. g- h4 m5 z8 k. h& Z' J <value>1</value>/ G& W- W( K' y n" p$ f, X3 ~
</property>
( S3 p0 R# V1 ^! M p <property>' I6 x" Y9 t2 G2 b) o
<description>Maximum time to wait to establish connection to3 C/ Q: j, Q4 r! n! i7 L
ResourceManager.</description>- y& @( e+ A% g
<name>yarn.resourcemanager.connect.max-wait.ms</name>
. r- _- l, a6 Y7 i8 C3 o1 S <value>900000</value>
' S. r2 {& K) P# g7 \4 x, l </property>
+ C7 d8 T/ I# f% N4 O% \ <property>
4 t# P D6 k& R <description>How often to try connecting to the A+ C$ ]$ H0 @
ResourceManager.</description>% N e& L9 q* Q7 A' U
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
# P9 L! t$ ]# W d/ J( c <value>30000</value>
% L) e4 f1 y' I7 P) T6 o </property>' D. S$ n7 Y8 P1 P; G% N
<property>/ v7 T" b8 T9 j" R3 ?' u6 z
<description>The maximum number of application attempts. It's a global
1 X/ I1 |" e) Z1 \9 B; I9 v setting for all application masters. Each application master can specify
2 N4 p e2 O3 T6 [ its individual maximum number of application attempts via the API, but the$ ^7 C7 `$ U& |$ C) P
individual number cannot be more than the global upper bound. If it is,
4 a( a) @5 R! `% Z d n the resourcemanager will override it. The default number is set to 2, to* q- h4 l# w$ L/ d. d( i
allow at least one retry for AM.</description>
4 H. l- b8 z0 {5 H( N <name>yarn.resourcemanager.am.max-attempts</name>
- A5 z; ^% v+ f <value>2</value>
+ F* L5 U; C! M! ?" O$ e </property>/ K" \* S9 _; j% A: l+ T8 b& }2 z. a
<property>
& O! L+ X9 a( {! G- Y: S <description>How often to check that containers are still alive. </description>
3 i( |8 }5 q, y& ^ <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>; v3 r5 ^. A+ p! [6 i8 n
<value>600000</value>
- p2 u' f' z! T$ x7 B2 n </property>) R; d; ]: g. D/ d5 g7 j# `
<property>
0 W" C5 y! \, U9 ]" } H6 K4 S) ^1 N- G <description>The keytab for the resource manager.</description>" v1 U- M/ e4 Q- W) s) ^9 X
<name>yarn.resourcemanager.keytab</name>
" S9 [3 }4 \3 M/ f <value>/etc/krb5.keytab</value>) h3 v0 A; V) y
</property>7 {% Y0 p& O+ ?; |9 u! u1 f
<property>
( `" Q: Y% O. i <description>Flag to enable override of the default kerberos authentication
3 q' H' d9 ^% {- |: D filter with the RM authentication filter to allow authentication using
: W* e+ f) W0 }+ }! U2 R- r/ ` delegation tokens(fallback to kerberos if the tokens are missing). Only
2 s9 C5 W+ ^. W! A( i applicable when the http authentication type is kerberos.</description>
' v- y* |! l, c. @- k <name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name>
3 Z. f3 }, j6 M3 _ <value>true</value>2 ^8 a+ u! R: X# a6 _% c
</property>+ x7 w6 a& L( Q! h. Z# q% L7 F' p
<property>4 k- O# o" ^! w" o5 s8 O, C* C
<description>Flag to enable cross-origin (CORS) support in the RM. This flag
% D8 H8 n" U/ T: O* ?* k requires the CORS filter initializer to be added to the filter initializers
! K6 N7 w1 x; L( ? list in core-site.xml.</description>; k7 j3 O2 f9 o5 E! B1 Q, _
<name>yarn.resourcemanager.webapp.cross-origin.enabled</name>
+ V" J/ L( \, d$ g. [' R <value>false</value># E& _* G5 v0 `6 o- }4 O
</property>- W- f( E. w- d5 d
<property>" {2 z$ E: q$ ?6 U3 ~. m1 q+ {
<description>How long to wait until a node manager is considered dead.</description>8 ]* z9 n4 e; J- I" f# a
<name>yarn.nm.liveness-monitor.expiry-interval-ms</name>& w+ m$ ~0 n1 }* Y
<value>600000</value>9 B' P+ ]4 F, M' r
</property>; W. O9 c5 F2 D; i
<property>1 T F0 A( K2 K& f
<description>Path to file with nodes to include.</description>7 g3 X: b0 w3 K1 X2 i
<name>yarn.resourcemanager.nodes.include-path</name>
% B$ A5 r7 g* @% Z& G# H, V <value></value>
! i1 z4 Q9 z/ n5 C </property>. K5 K# L/ Y2 b' }
<property>' t7 L( C& [9 Y4 Y9 l$ b' b
<description>Path to file with nodes to exclude.</description>
9 R0 G5 D" B* q% C" h" w6 H8 c <name>yarn.resourcemanager.nodes.exclude-path</name>. m `( ]' v8 g% y
<value></value>0 n) \/ P1 c3 M9 ~; P- v
</property>; c; X8 t: d8 X
<property>
* Q$ l1 x6 [9 O3 E$ R% q; P) H <description>The expiry interval for node IP caching. -1 disables the caching</description># W7 N' S6 c5 i$ m9 n: \" Z4 n
<name>yarn.resourcemanager.node-ip-cache.expiry-interval-secs</name>* m& _. x4 ]2 ]7 M. p) n% H
<value>-1</value>
( q% u! _% p0 n& t6 r0 f- p: e/ N </property>
+ R5 i, ~/ V( p$ H <property>
' p# l. P; Q# Z# l <description>Number of threads to handle resource tracker calls.</description>
7 l/ ^5 b! D0 q- N <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>) Y% ]6 S$ M7 ^! S! t" h
<value>50</value>
. }* w6 @; k. g! o% G3 |( F7 S \. x </property>
( v: J7 Y& k) W: e& W j' @3 r; v <property>
* F0 C- b9 x' v" S9 Z: C. F <description>The class to use as the resource scheduler.</description>
( F V& O& E4 J% w <name>yarn.resourcemanager.scheduler.class</name>
) Q& ?+ C2 [* Q" a <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
0 z! m8 R! h5 M# o& e, g1 w1 c </property>
( R7 I% f' u& G+ z/ q3 Y! @2 } <property>1 q& q3 C8 }4 C# z
<description>The minimum allocation for every container request at the RM8 \ S: W2 }4 l. p) Z: d1 {3 m
in MBs. Memory requests lower than this will be set to the value of this M& v% D7 N: S8 X; p
property. Additionally, a node manager that is configured to have less memory; Q% ` p0 E3 n; o' p7 \
than this value will be shut down by the resource manager.</description>: X1 z# R2 v! a0 `4 C$ o
<name>yarn.scheduler.minimum-allocation-mb</name>- J8 R' _# A3 M) _" l+ x; O
<value>1024</value>
1 f* F, ]8 u2 k% q1 I9 c. D </property>" b: o/ D; }3 u0 g2 B* ^
<property>
& M& r2 M5 B; i$ N3 x3 k) f <description>The maximum allocation for every container request at the RM
- m6 u/ P3 Q$ c6 X5 S% | in MBs. Memory requests higher than this will throw an1 w) ], p' J4 u
InvalidResourceRequestException.</description>! k* X( o4 g. o- a
<name>yarn.scheduler.maximum-allocation-mb</name>
* o9 J& h0 j, w- Y0 { <value>8192</value> S7 ^1 Y) D; m! A$ p0 j
</property>1 S0 [( a, d6 P" _! P1 {' x
<property>
2 g2 r) `1 ]- @' v" P: l% Y7 M <description>The minimum allocation for every container request at the RM9 e2 _* Y/ M, i
in terms of virtual CPU cores. Requests lower than this will be set to the" [8 s t7 u B/ J* v: r$ j
value of this property. Additionally, a node manager that is configured to; l" y. v6 u9 G4 }
have fewer virtual cores than this value will be shut down by the resource; K8 K) y. K2 W# W
manager.</description>( E0 |: V3 h% g8 o
<name>yarn.scheduler.minimum-allocation-vcores</name>
- v% j5 z+ ^: v3 M: N# { <value>1</value>
, d V& Y9 }0 k' c, A$ @; I </property>% z/ n# F+ f, k7 x
<property>
0 Z% \9 t: R9 O/ k! l3 e3 U <description>The maximum allocation for every container request at the RM; X4 S& x/ \- r' t
in terms of virtual CPU cores. Requests higher than this will throw an) a( Q% @5 @. U7 V* r0 W* Y+ l1 c
InvalidResourceRequestException.</description>& G- X) t- ^! p: i( b- D9 I
<name>yarn.scheduler.maximum-allocation-vcores</name>
- [$ y% p' z: S& K5 q' i) Y <value>4</value>% W( s7 w- P* Q5 p! }; X9 s& N J
</property>% v3 x+ q, m! m+ |! n& p* X) C% V
<property>
* n5 O, b& W! C <description>
~) ]4 `9 j& V. A6 g" I$ O Used by node labels. If set to true, the port should be included in the
) t. x. V" x- _2 ^8 u+ X* U; V- r5 u node name. Only usable if your scheduler supports node labels.
' l* R6 P" k4 E9 B </description>; ` }4 F* K% ]$ Y z0 f
<name>yarn.scheduler.include-port-in-node-name</name> E) _" x! ]. c6 g5 N' l
<value>false</value>
1 M- m6 E/ [( K9 u9 p2 i </property>
$ Z5 }$ x6 K" o! Y <property>
% V# v' T0 p! c' C# ] <description>Enable RM to recover state after starting. If true, then b5 j$ x* K( d
yarn.resourcemanager.store.class must be specified. </description>1 s/ f* ^1 O2 x" @1 Q
<name>yarn.resourcemanager.recovery.enabled</name>. z0 h5 | B# M' A' a
<value>false</value>- T1 v2 e2 D8 W
</property>, b$ q. O7 l4 |; E/ P8 X
<property>
4 Y' Z# I' w2 @" [# \ <description>Should RM fail fast if it encounters any errors. By defalt, it0 ]2 o( J$ r6 Y6 |8 H% |" }
points to ${yarn.fail-fast}. Errors include:! C" F2 O0 t( j& P8 s) r! {! Y
1) exceptions when state-store write/read operations fails.
; {# e- t) i/ s7 a/ I M </description>
# A- a! x T) Z" _0 P0 ] <name>yarn.resourcemanager.fail-fast</name>
1 E4 h8 r' l2 O8 z <value>${yarn.fail-fast}</value>
2 ^6 K9 j$ H. M" D+ j5 L </property>, x; N4 Q& X' t' J" ~
<property>) A% O: Q' P$ f( m( @* s$ h8 {. x
<description>Should YARN fail fast if it encounters any errors.( v& a: [/ U0 b* z" ]1 x) V
This is a global config for all other components including RM,NM etc.1 h% _3 T# z" i7 m+ b5 n3 v3 C5 d4 i
If no value is set for component-specific config (e.g yarn.resourcemanager.fail-fast),- ?" l% B) I. g: _) |
this value will be the default./ f4 @5 |1 t% {2 \
</description> L2 n. p; Z; `) J \8 p1 Y7 i2 d& e5 B
<name>yarn.fail-fast</name>( ]4 \1 Z5 m: y$ d% e
<value>false</value>
6 N n; [2 M8 \% h' L- { </property>7 c1 t# U$ Q6 U( C E' A2 L
<property>" G$ h4 U$ H k, w
<description>Enable RM work preserving recovery. This configuration is private
; `& b* f- a& Z; j4 A0 m4 V to YARN for experimenting the feature.! V+ }* M5 k& @$ |& d) K4 m3 x* S+ k
</description>1 j l) H& e% d
<name>yarn.resourcemanager.work-preserving-recovery.enabled</name>
8 x4 m0 P0 |' z4 [% ~' { <value>true</value>
3 t9 u4 M+ Y/ K! H8 ? </property>
- K: \$ h3 L. j6 S5 w <property>
' s# ] S/ s- f1 Z6 s4 C4 g <description>Set the amount of time RM waits before allocating new% C, c: A- l1 R3 k4 s
containers on work-preserving-recovery. Such wait period gives RM a chance
, H) Y' U3 z$ |& q1 v& b* a to settle down resyncing with NMs in the cluster on recovery, before assigning
; |2 @1 |; G4 v, @8 U new containers to applications.6 l2 D @; e; h+ g2 a. W2 ~
</description>
' }: r& | X2 u9 Q <name>yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms</name>* q- t$ W r! O* y3 H
<value>10000</value>
$ _: B$ F# c& \' \# z </property>$ r/ E. v( ^; h" U, w
<property>+ h! G4 I# Y; B- d5 ^4 w8 b
<description>The class to use as the persistent store.; a/ T7 [) i# q
If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
, K$ t* [+ w* [0 k% C is used, the store is implicitly fenced; meaning a single ResourceManager
9 q/ _' K7 X* t' _ is able to use the store at any point in time. More details on this
- y {$ q0 z7 g3 }: x% Q, s implicit fencing, along with setting up appropriate ACLs is discussed! y, |2 l* t5 J4 B; Y
under yarn.resourcemanager.zk-state-store.root-node.acl.
- t: o* i1 y O& R </description>: S4 t4 _6 [" {) x/ [. a5 z# Q
<name>yarn.resourcemanager.store.class</name>( q3 }5 c/ N; o8 c0 U
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
3 D6 _4 ^1 P7 {7 P* {' s7 Y- c5 w </property>
, s( `. R6 x6 g <property>6 ^) ~. t. N3 g( o
<description>When automatic failover is enabled, number of zookeeper
# V9 ~7 F. G9 H) B operation retry times in ActiveStandbyElector</description>) E; w5 P5 h. a+ Y6 b
<name>yarn.resourcemanager.ha.failover-controller.active-standby-elector.zk.retries</name>$ ^/ J- [- a5 _3 ~& [7 t, v# v, N
<!--<value>3</value>-->
# U# b0 I* v, R8 @1 ` </property>
% L- |0 H0 E6 _" F4 i H <property>
( L7 x. ~) f4 H1 V: [6 T <description>The maximum number of completed applications RM state
8 b; p. u' j1 s. `3 V$ r9 t/ R! A store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}., p6 _0 h) k% ]; F5 A
By default, it equals to ${yarn.resourcemanager.max-completed-applications}.
7 c- z2 Z/ K4 M$ _ p0 g9 m This ensures that the applications kept in the state store are consistent with! t9 l4 T3 g) H" N/ j% p
the applications remembered in RM memory.
( y2 p' ?5 t0 T. G- y4 v Any values larger than ${yarn.resourcemanager.max-completed-applications} will8 L0 f% q3 _7 G* X2 O/ _
be reset to ${yarn.resourcemanager.max-completed-applications}.6 I$ v0 ^. H$ v0 C" d+ h) b
Note that this value impacts the RM recovery performance. Typically,
1 h: \2 I) s# ]) H- `6 x. r9 Z" z3 M a smaller value indicates better performance on RM recovery.
) u: j' D9 [3 J: y </description>) l/ J+ v0 E' W
<name>yarn.resourcemanager.state-store.max-completed-applications</name>9 e2 J/ G4 G, j; j9 y! f; G5 ` I
<value>${yarn.resourcemanager.max-completed-applications}</value>2 n6 {& K% B3 D8 {
</property>
7 }' r% o7 ~/ _ <property>
* W- O* n1 a" K+ S: a' A <description>Full path of the ZooKeeper znode where RM state will be- Q1 [: k9 z' `: N4 S
stored. This must be supplied when using
8 k% F, L3 ~; s# a% o& x org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore _- [3 V O( Z. w) i3 G ~
as the value for yarn.resourcemanager.store.class</description>
/ u8 F4 K t3 o" H1 w <name>yarn.resourcemanager.zk-state-store.parent-path</name>
6 R* N* y: O# i" {+ G <value>/rmstore</value>4 r u/ r# M$ B% m% o& [" F
</property>1 m* K* C& Z- U, N! Z0 ~/ U
<property>) R# `( t( {; Y* A& W* q% Z1 N8 J
<description>
u3 F4 n& O% O E# P4 s) B ACLs to be used for the root znode when using ZKRMStateStore in an HA
k: J& ]9 B& ?; h6 u+ P scenario for fencing.- A, }; l' ?; P7 I# G, o6 F0 S
ZKRMStateStore supports implicit fencing to allow a single
+ Q: l! _8 y) W8 p! ], u+ q ResourceManager write-access to the store. For fencing, the
- H; K" o, y1 f" z2 p# F ResourceManagers in the cluster share read-write-admin privileges on the6 a2 }! C4 {, V) Q Z# U {
root node, but the Active ResourceManager claims exclusive create-delete
6 N% b4 y. t, O% ?7 H permissions.
& ~* G5 c( N6 A. l0 _ By default, when this property is not set, we use the ACLs from4 g3 l1 x" P8 k. {2 e( {. |4 d0 ?. `
yarn.resourcemanager.zk-acl for shared admin access and$ |& X( Z+ O# j- p
rm-address:random-number for username-based exclusive create-delete
$ J z t- I3 ` access.
# Z' q3 m) D! F# [ This property allows users to set ACLs of their choice instead of using* T. p: M3 Z7 X. J: O0 j# f
the default mechanism. For fencing to work, the ACLs should be3 m& ~* i, a* @8 P( p/ n
carefully set differently on each ResourceManger such that all the
U9 F7 Z3 r& m+ l$ c5 h/ e. ^ ResourceManagers have shared admin access and the Active ResourceManger
9 O0 _8 v3 _4 {' ~4 b# ~. h |1 W takes over (exclusively) the create-delete access.
! k, U$ B+ ?* ?& Z; G; V </description>
' z+ W: j3 E, c( O- {/ {; j0 Q k <name>yarn.resourcemanager.zk-state-store.root-node.acl</name>, N" K8 T- @$ \$ O, T) z
</property>
7 f/ R; ]: U) J. ?8 w- ?, v <property>* V( E! c# ]9 v6 G. w
<description>URI pointing to the location of the FileSystem path where
- R- D* }$ Z3 d) N5 S RM state will be stored. This must be supplied when using. V, g# ~1 f) {- ^% a* e
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore; `: g/ D: f! _# K5 Q" Y" m/ Q
as the value for yarn.resourcemanager.store.class</description>: Q: k! h7 X8 L
<name>yarn.resourcemanager.fs.state-store.uri</name>4 {9 o. K8 y# O7 Q' K1 {
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>) B8 l f+ R. \4 V( z
<!--value>hdfs://localhost:9000/rmstore</value-->8 }% t$ E) s7 }
</property>5 E; @. K4 u6 N# s* d; k3 {: {
<property>
7 k8 |$ P1 ^8 i* }5 P7 _. Q <description>the number of retries to recover from IOException in; [8 y' e. k0 L* J
FileSystemRMStateStore.
6 s' m. x- Y0 L, U0 ~8 ? </description>
$ Z4 a& m( s9 k4 p4 \ <name>yarn.resourcemanager.fs.state-store.num-retries</name>
; S' _9 @$ t& W0 P <value>0</value>
m1 }) D3 h* {; C1 d0 I; F </property>
" o0 h Q/ \( a% [( w& T( W- m3 d9 k' s <property>
" S4 B- ]& _2 V$ `$ q8 g* A" Y <description>Retry interval in milliseconds in FileSystemRMStateStore.
! ^; ^+ e0 r5 U1 E: j2 Y& P' T0 S </description>+ ]5 v7 a" [: b8 y9 _
<name>yarn.resourcemanager.fs.state-store.retry-interval-ms</name>
8 H0 h: u6 k D% ? <value>1000</value>2 m s6 U/ ?) f# m+ _
</property># c/ C V: b4 y+ t3 x6 |5 {6 m- |
<property>
* v. B/ l. A2 x G8 J4 @ <description>Local path where the RM state will be stored when using
2 W1 C8 r. j. y" G; z org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore
, g/ K% G. L4 }7 Y/ N0 n& @; q% p as the value for yarn.resourcemanager.store.class</description>
' z, C/ {' ^) R- j% U& u <name>yarn.resourcemanager.leveldb-state-store.path</name>6 V5 v$ M4 C6 g+ E% U; x) g
<value>${hadoop.tmp.dir}/yarn/system/rmstore</value>) \2 ^$ t' f4 }( x8 h
</property>* ~; i* S/ F+ u* B1 X, ^ ^
<property>
" I' _7 M$ @5 p <description>The time in seconds between full compactions of the leveldb$ M5 B" J9 o. k) _' ^" F9 J; W
database. Setting the interval to zero disables the full compaction
6 b' O, ^; S2 Q cycles.</description>
' d$ e* l& Z' @3 V7 l! j2 o; h4 c <name>yarn.resourcemanager.leveldb-state-store.compaction-interval-secs</name>
; X+ e. f" j3 c: ~; g <value>3600</value>) {/ b2 r0 a) c# f
</property>$ l3 Q' h: r- q% j4 H
<property>
3 h. K: `9 {) D9 A& z, H5 X <description>Enable RM high-availability. When enabled,
+ G9 a% b# j0 J' _9 J* Z* y (1) The RM starts in the Standby mode by default, and transitions to
) w& K% ?# @' p- f0 b the Active mode when prompted to.5 p v& D. O/ z, d
(2) The nodes in the RM ensemble are listed in# }$ K; t; z, x% f% S) g
yarn.resourcemanager.ha.rm-ids- ?3 C/ i" X9 o, N, \8 p- r) `
(3) The id of each RM either comes from yarn.resourcemanager.ha.id: U1 T' q: e3 d' t
if yarn.resourcemanager.ha.id is explicitly specified or can be
. H& S2 q0 Y/ B; C% _) ^ figured out by matching yarn.resourcemanager.address.{id} with local address& o' H3 C% h9 D/ v. Y1 U- m
(4) The actual physical addresses come from the configs of the pattern
( @+ [) k( A0 K% O& D0 Z - {rpc-config}.{id}</description>
& A( l y [1 \- x( G7 x <name>yarn.resourcemanager.ha.enabled</name>
* }2 j, r$ {- C- g1 p$ L% y6 y <value>false</value>
7 E" P; k0 w4 O# C8 H5 s </property>
; _' D, d, J* N, p& _# @. e <property>9 w. `+ s% V; R
<description>Enable automatic failover.
& {( f% N% c* ]6 H2 \+ Y9 i! A' Q( t By default, it is enabled only when HA is enabled</description>2 K: |5 s/ ?! o( {: F" h
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>( a) T, K0 y$ Z; W3 V. }
<value>true</value>8 K: K* d7 E6 @3 a3 Y
</property>8 O- | N3 [: A1 c
<property>( d: e, C' a* v0 N5 M) I. B
<description>Enable embedded automatic failover.0 n, p4 E- f- m& q
By default, it is enabled only when HA is enabled.$ _. V9 ]! K# E5 d
The embedded elector relies on the RM state store to handle fencing,
- F0 _1 N* T# @( U and is primarily intended to be used in conjunction with ZKRMStateStore.: ?& Z* t9 J7 V% R9 B+ `
</description>
3 f+ Y- Z* l3 |& |+ n <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
! S; R4 n8 Q' b8 b) O \ <value>true</value>* E3 O( Y+ G! Z
</property>
0 F u# F- |, o0 Y) D0 ? <property>" E2 V* T5 _! c0 B; M
<description>The base znode path to use for storing leader information,
* Y+ F5 x2 j. s( F when using ZooKeeper based leader election.</description>2 u( H2 [1 z% [, A
<name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>- u* @; ]. C7 x" R
<value>/yarn-leader-election</value>$ O: j( O7 O" f' I; `9 I
</property>
( N @4 @) y. y8 ~ <property>% `- X w1 z5 R" ?0 r
<description>Index at which last section of application id (with each section& w8 Y. x& [/ u5 b, x
separated by _ in application id) will be split so that application znode
2 O: p) o0 w& x- o3 s stored in zookeeper RM state store will be stored as two different znodes$ H7 j$ J1 n4 K! B5 y l: u
(parent-child). Split is done from the end.
; g: w: T5 x+ `+ Y3 I For instance, with no split, appid znode will be of the form
' A1 z" I1 X' j) O application_1352994193343_0001. If the value of this config is 1, the
) m2 |$ q/ E$ w3 @ appid znode will be broken into two parts application_1352994193343_000, k# s# W& l- W7 Z0 w
and 1 respectively with former being the parent node.
% P3 a( p& }. [ application_1352994193343_0002 will then be stored as 2 under the parent
! X/ c d9 u: r& A node application_1352994193343_000. This config can take values from 0 to 4.4 w! O2 b$ Y6 P- k0 M4 H6 A0 L1 Q
0 means there will be no split. If configuration value is outside this
7 M+ h) M& y+ Y2 E4 V% s% H range, it will be treated as config value of 0(i.e. no split). A value' v! q. d, D _/ x4 V
larger than 0 (up to 4) should be configured if you are storing a large number
1 Z$ s3 S2 ]' A of apps in ZK based RM state store and state store operations are failing due to
: ~: u6 s p1 v" F8 R3 g LenError in Zookeeper.</description>
: u. r3 T1 u7 V" {. N# v: O <name>yarn.resourcemanager.zk-appid-node.split-index</name>1 ^$ z X {% H2 N6 Q9 M
<value>0</value>
0 Q3 j+ G7 _# V8 k( W2 ]% |; j </property>
, H/ @7 H2 `' t. w <property>" T3 @: }$ u0 C: j- E! `
<description>Index at which the RM Delegation Token ids will be split so
a0 P6 f6 l2 G- D0 H' e" h that the delegation token znodes stored in the zookeeper RM state store
/ V8 a, X8 H. [0 E& D7 P will be stored as two different znodes (parent-child). The split is done8 U D# E/ j" c0 C4 u) C5 P9 n4 }: y
from the end. For instance, with no split, a delegation token znode will! J" S5 J" }2 F" z* P3 z
be of the form RMDelegationToken_123456789. If the value of this config is
7 N$ \8 X, V8 U 1, the delegation token znode will be broken into two parts:
* K8 G% {/ T! e RMDelegationToken_12345678 and 9 respectively with former being the parent
% {( f9 ]) Y. ]1 ]( T1 P T v node. This config can take values from 0 to 4. 0 means there will be no
$ A' r" j( ]: \ ^ split. If the value is outside this range, it will be treated as 0 (i.e.0 A: g* \& @7 K1 k/ l* p
no split). A value larger than 0 (up to 4) should be configured if you are. L! w, L) {6 {* L3 ?
running a large number of applications, with long-lived delegation tokens5 B; Z* r& k) M
and state store operations (e.g. failover) are failing due to LenError in% _% {0 g3 j* _! N
Zookeeper.</description>8 Y, o, H( p; L, I
<name>yarn.resourcemanager.zk-delegation-token-node.split-index</name>
8 ^- M$ l% C+ U2 ^4 o <value>0</value>& l0 g4 ^& h7 Z# O8 w6 \" i$ \
</property>
$ j2 L# A) c4 R6 Y" B! v) l <property>6 j9 y" T+ z% f2 p, Q" ~/ c/ y
<description>Specifies the maximum size of the data that can be stored+ G8 ?6 Q8 y A. p
in a znode. Value should be same or less than jute.maxbuffer configured
6 K' w) I1 Y* K, t in zookeeper. Default value configured is 1MB.</description>5 F J1 ~" z& g" R) V1 E3 @4 [
<name>yarn.resourcemanager.zk-max-znode-size.bytes</name>, m# g* A( l" m$ @4 s
<value>1048576</value>
u" c/ Y+ E& F. V0 G( A7 w </property>
7 R. P- @5 y2 w! v: S <property>$ g* p2 G: z0 h, F6 x* H
<description>Name of the cluster. In a HA setting,8 Z5 D; J5 W! E
this is used to ensure the RM participates in leader
- F9 d- {4 ?! n; d# i6 i! i election for this cluster and ensures it does not affect
0 n4 n) w" i0 P) Q$ _5 F9 Y; k other clusters</description>( g3 x5 e5 b6 A, o/ @5 _
<name>yarn.resourcemanager.cluster-id</name>
* i8 L8 }; ?" j* I: b |6 V <!--value>yarn-cluster</value-->
1 g! I K, }- N! @# R& P </property>
) i. `( ?, V8 ]4 v <property>
8 O# ~( f) ?( ~! U1 x) e8 x <description>The list of RM nodes in the cluster when HA is8 }$ R% b5 r4 Q& f) g1 ~3 X
enabled. See description of yarn.resourcemanager.ha1 Y2 J4 Q, A/ \' h3 Z+ K2 e0 ]) c
.enabled for full details on how this is used.</description>3 m) N( e% U4 Y1 x0 M( m
<name>yarn.resourcemanager.ha.rm-ids</name>0 B( i' z" @7 {, A* }: A
<!--value>rm1,rm2</value-->/ T! N& m! X; \: C
</property>8 C* a1 `' s$ {+ z& Z/ w& y) q: e, q3 x
<property>
: o8 E# v7 P- G2 ^8 h' P6 c <description>The id (string) of the current RM. When HA is enabled, this9 x! u/ I( C9 D4 D% X3 r- C
is an optional config. The id of current RM can be set by explicitly
' \2 F' s1 J7 g9 {) W0 v/ | specifying yarn.resourcemanager.ha.id or figured out by matching
. ]+ f$ Z4 A+ U O) i* ^/ t8 a yarn.resourcemanager.address.{id} with local address( a" ^ A5 ^1 l% E
See description of yarn.resourcemanager.ha.enabled
- s) |7 v5 r9 [. ?) F5 J for full details on how this is used.</description>
6 Z V( O9 w5 f5 Z2 _ <name>yarn.resourcemanager.ha.id</name>' ~" n' m# J* H: M; e+ _( d
<!--value>rm1</value-->
: ?7 X4 R( D, r7 o( P( `: _ </property>
! d3 V- G5 U1 a! X: X" d <property>
' \. V& ?& i) |) N' q/ } <description>When HA is enabled, the class to be used by Clients, AMs and _4 D8 _& x! L1 F
NMs to failover to the Active RM. It should extend9 ?$ ?+ C3 f4 K8 N* G/ |) i
org.apache.hadoop.yarn.client.RMFailoverProxyProvider</description>
8 D$ z2 O5 ^/ _& g2 |- Z <name>yarn.client.failover-proxy-provider</name>) y* B$ ?- ?0 w& o: k: f/ P
<value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
- l9 B4 a) A/ S3 f% E2 o </property>1 ?! D; w4 M5 s( Z5 }6 R
<property>) |$ g. j0 ~" C' u. _: A
<description>When HA is enabled, the max number of times
: [& q4 J U. b4 E5 M& X) z% x1 l FailoverProxyProvider should attempt failover. When set,3 w7 C# n/ G7 Y: A" @
this overrides the yarn.resourcemanager.connect.max-wait.ms. When
% o& }0 m+ P" y' `3 i not set, this is inferred from! T0 v9 m( M% Q8 K* r7 d0 q
yarn.resourcemanager.connect.max-wait.ms.</description>
. l4 Z' q( z3 a7 w <name>yarn.client.failover-max-attempts</name>% H: s& B) w6 X J# B
<!--value>15</value-->& `' }+ I) o) M; ?+ ~ @/ o
</property>+ U$ |9 f; e1 Y
<property>6 o. l0 f" P2 [4 T. d4 n) o
<description>When HA is enabled, the sleep base (in milliseconds) to be K6 y1 p& ` [+ ]/ b
used for calculating the exponential delay between failovers. When set,3 _; `' e, U2 n. c
this overrides the yarn.resourcemanager.connect.* settings. When
. [: @0 T; T1 B3 ~. L X) K not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.
# S( t5 Q" y1 a" V# x) |, V# g </description>; J. |* \' j% \0 U1 Z& E
<name>yarn.client.failover-sleep-base-ms</name>, c" s( Z; O, V
<!--value>500</value-->; r/ v7 H3 e4 k9 q2 `
</property>
! j, N: K! X/ S: Y <property>
9 b( ?9 B- R4 Q, d <description>When HA is enabled, the maximum sleep time (in milliseconds)
: e: p( Y+ A m& }& p0 b between failovers. When set, this overrides the# P# X- K$ I: E! I7 E; r
yarn.resourcemanager.connect.* settings. When not set,
/ c$ _0 F1 D9 |4 M: S yarn.resourcemanager.connect.retry-interval.ms is used instead.</description>
) e- e, N) j R) ^4 v" q2 f <name>yarn.client.failover-sleep-max-ms</name>. Y& N, ^- i" O
<!--value>15000</value-->
1 I* k1 k) S- _; r </property>
; u1 W- q0 {6 [: d. ]& A% ~ <property>
2 L; F0 {3 a+ i. v9 R$ n g; c1 J <description>When HA is enabled, the number of retries per1 x$ J4 { d5 Q1 N3 K
attempt to connect to a ResourceManager. In other words,
' h* [3 d& ~" X' f" u8 c6 ?2 k0 h it is the ipc.client.connect.max.retries to be used during1 {( B9 i- G5 F
failover attempts</description>, t2 v* o4 u/ }) d% l8 ]: K
<name>yarn.client.failover-retries</name>5 j! A- y: O- h* O3 [" r) d- f0 p
<value>0</value>1 m5 |+ V, C0 v* S; C; \( y* {9 ?
</property>
# ?0 c7 y0 K9 W% r <property>
9 a4 K) F$ ~4 v <description>When HA is enabled, the number of retries per
~$ O3 u0 u& x- {% E2 o attempt to connect to a ResourceManager on socket timeouts. In other# G9 i& T U# u" ~* F7 q/ R8 |2 e1 C, r
words, it is the ipc.client.connect.max.retries.on.timeouts to be used: ^3 f8 Q8 k2 C
during failover attempts</description>
- h* f; r4 x0 `! A5 `" O0 |0 \ <name>yarn.client.failover-retries-on-socket-timeouts</name>2 `4 ^) ^5 x& S0 ~
<value>0</value>* `9 N2 Z) g8 C6 R
</property>" R% G% I7 q: d
<property>* E E& \/ |- D. Z, u
<description>The maximum number of completed applications RM keeps. </description>; k4 p2 _8 r+ ]# M
<name>yarn.resourcemanager.max-completed-applications</name>
! Y/ A2 [; F1 ^, k <value>1000</value>
) {7 y- a: i5 I: l </property>! H+ Z- R" q! N5 h0 Z
<property>
F& ?/ q$ p) @; S <description>Interval at which the delayed token removal thread runs</description>, ] B; x. W- N
<name>yarn.resourcemanager.delayed.delegation-token.removal-interval-ms</name>
7 L: \( W8 w& k4 l% x9 ] <value>30000</value>- ^4 ]2 j/ S N* |: F
</property>
/ [0 ]5 x" R8 \% q9 P3 N4 V# a' J <property>
# z' h$ F+ ~. M( t3 O4 Z w <description>Maximum size in bytes for configurations that can be provided8 M. m& L* Y& Z# s% U1 A! a
by application to RM for delegation token renewal.& n! ~4 |9 h( d V9 ^
By experiment, it's roughly 128 bytes per key-value pair., y3 A1 o1 M" O# X) u
The default value 12800 allows roughly 100 configs, may be less.: }3 U' Z4 O0 F1 i, K, }( O
</description>
" Q) J! @; i8 n# G: l Z <name>yarn.resourcemanager.delegation-token.max-conf-size-bytes</name>
V6 T, e& J. x/ @1 | <value>12800</value>, z9 M1 f3 H% j' v# f
</property>$ ?5 u, \9 x; b% m# e% t) j% m
<property>
( W: g, ~ u4 o, {& h! }7 p <description>If true, ResourceManager will have proxy-user privileges.$ h" N+ r C6 ?9 ]' P
Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to
, K- t t" `2 L: M do localization and log-aggregation on behalf of the user. If this is set to true,
i% K5 Q0 M7 U# Y ResourceManager is able to request new hdfs delegation tokens on behalf of
" @, v u% a, ^+ G' x' f4 x the user. This is needed by long-running-service, because the hdfs tokens0 _. ?& S% Z3 I. p
will eventually expire and YARN requires new valid tokens to do localization
& i( G+ \ C/ D& g2 s1 O( |: v and log-aggregation. Note that to enable this use case, the corresponding
* M: ~+ X" K3 e4 z& b HDFS NameNode has to configure ResourceManager as the proxy-user so that
# j7 C5 i% M4 I( A" W ResourceManager can itself ask for new tokens on behalf of the user when
0 w9 ]0 l" ]4 Q tokens are past their max-life-time.</description>) [: x9 r' X" i V7 z
<name>yarn.resourcemanager.proxy-user-privileges.enabled</name>
: t" J4 W8 M6 f1 I% l <value>false</value>
& w( M( k0 C5 c6 X </property>
9 M- b" h: P. Z: f7 e( s <property>
7 j8 f8 ~0 H5 l* V <description>Interval for the roll over for the master key used to generate
5 y0 A! k/ F. U7 u; F1 }; [8 K application tokens1 G: W; W3 z2 E- ]: S
</description>
8 B$ `9 I8 i* k4 m3 j. V$ v <name>yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs</name>8 S' y6 o" w& ^
<value>86400</value>
8 @% n y u+ b </property>
5 G8 R, a, X4 t! K {4 R <property># E- J2 B7 n- z7 k$ ]
<description>Interval for the roll over for the master key used to generate
9 L( C# q. ]6 V+ J) J! D) z container tokens. It is expected to be much greater than
) G# }% ~5 z2 J" f8 x3 q yarn.nm.liveness-monitor.expiry-interval-ms and
, [& }( m. }7 [ yarn.resourcemanager.rm.container-allocation.expiry-interval-ms. Otherwise the: m! ` I' i$ g k4 a; }8 W
behavior is undefined., c! ?6 g+ E: z& _& U2 b* c+ g
</description>
1 G3 q A- G% w8 b2 Q& r <name>yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs</name>
- u3 A- X; u- w <value>86400</value>+ \$ a% c$ j3 h0 r* Y
</property>. _6 h1 y8 z9 C; W
<property>
+ H* P, i5 X$ {) z( D2 Q* |2 s& J <description>The heart-beat interval in milliseconds for every NodeManager in the cluster.</description>
& [, t/ k. I0 L S2 f <name>yarn.resourcemanager.nodemanagers.heartbeat-interval-ms</name>! m! W% W |: g9 w/ `6 R4 }
<value>1000</value>5 h; K3 z! ?: P. b
</property>, q& L9 n" P, E
<property>
. j6 Z K+ M7 @0 e <description>The minimum allowed version of a connecting nodemanager. The valid values are7 O, U' S8 Q. C/ ?8 }
NONE (no version checking), EqualToRM (the nodemanager's version is equal to+ x9 S1 q9 s* Q0 q% e
or greater than the RM version), or a Version String.</description>
: l# a6 j. \6 Z! B$ @ <name>yarn.resourcemanager.nodemanager.minimum.version</name>7 H/ s7 P: q6 n$ d& p5 \3 C$ J
<value>NONE</value>
/ x8 W$ t; p' w! V </property>
/ v3 q% \0 g4 `) `' }$ b9 q <property>
; {, t0 [, o. s( d6 H <description>Enable a set of periodic monitors (specified in
" J$ ^) k9 a8 D8 v yarn.resourcemanager.scheduler.monitor.policies) that affect the( G7 Y Y2 H7 Y
scheduler.</description>
5 a( a) c4 Z& F( z: h* ?. E1 t) W <name>yarn.resourcemanager.scheduler.monitor.enable</name>
/ J/ C; \- I% p& @; F' ` <value>false</value>
% L% a! _! N- R( |) K# ^$ Q- E </property>
) F1 ^. Z# c r <property>/ n# H- j: A0 T N
<description>The list of SchedulingEditPolicy classes that interact with! V" a0 z! p. H/ j
the scheduler. A particular module may be incompatible with the% `% x/ |& l( d/ B+ L/ Q. j
scheduler, other policies, or a configuration of either.</description>
a. x* Y+ A! o) ~3 X$ ~ <name>yarn.resourcemanager.scheduler.monitor.policies</name>) d- ^8 v% e& {4 T
<value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value>
! A8 z- c5 g- d </property>, X# n+ a6 s% O7 ?9 K9 t
<property>9 K# t4 G! H" K) Y6 z
<description>The class to use as the configuration provider.( q; r3 X4 h0 N! T1 ?4 z
If org.apache.hadoop.yarn.LocalConfigurationProvider is used,
! C2 i3 _9 Z" m1 s8 h2 X6 d" x the local configuration will be loaded./ @- g- T9 O8 p% B2 Q
If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used,
( r+ }# v. D* ~, p: m0 Z' H, m9 i the configuration which will be loaded should be uploaded to remote File system first.% L6 U& i: ^, x
</description>
( W+ ^6 f% `/ @8 F <name>yarn.resourcemanager.configuration.provider-class</name>
4 f* p1 n" T$ h1 R, I4 z <value>org.apache.hadoop.yarn.LocalConfigurationProvider</value>* R1 _+ p( b; y# F" b
<!-- <value>org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider</value> -->4 D5 k: t& c2 n W/ e' ~- u* u
</property>
k5 r! a5 M+ W& n' x <property>
' i$ S. F3 @' R0 F2 T7 W( \ <description>* ?: N5 W6 K; E
The value specifies the file system (e.g. HDFS) path where ResourceManager
; u1 r; M% X4 T8 }- B3 S loads configuration if yarn.resourcemanager.configuration.provider-class n; u3 M' B6 t1 n
is set to org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider.
- O! R& T0 K( f4 W8 K2 y </description>7 ?. w+ A7 T( m5 v8 x R
<name>yarn.resourcemanager.configuration.file-system-based-store</name>
/ [+ Y- z4 Y3 S$ O8 T1 z <value>/yarn/conf</value>, v8 K1 B! l( O4 I- g, Y# L6 {
</property>
5 M) x3 a' W9 K# j# ] <property>
4 {+ \8 e& M: d# r <description>The setting that controls whether yarn system metrics is
$ c y/ f' s1 O6 u* ~ published to the Timeline server (version one) or not, by RM.
1 ~8 T5 l' Q2 I2 r8 B% q, V* W This configuration is now deprecated in favor of
% R! E* ~6 ]' L7 {5 e2 J yarn.system-metrics-publisher.enabled.</description>
- R0 z9 @, X/ @/ l <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
V: M3 g2 z( }( M8 v7 s <value>false</value>
0 f, j$ Y4 M( r, w' k# N8 s* a </property>: o' N, x# ~0 d$ z
<property># z4 K% A* x9 G# c7 S
<description>The setting that controls whether yarn system metrics is
2 j/ i8 A6 a" U+ } published on the Timeline service or not by RM And NM.</description>
9 J2 v) U8 H. \$ D+ c0 Q( ^ <name>yarn.system-metrics-publisher.enabled</name>2 b9 X g5 s* O3 f5 p% I
<value>false</value>9 v1 ]3 ]7 ^: \4 V& d- m9 m& P
</property>
- e( {/ c- y3 M8 P" o <property>
) d2 k9 H/ V+ N" o$ i% J <description>The setting that controls whether yarn container events are
! _ {5 P- ] j3 w, c( L% f published to the timeline service or not by RM. This configuration setting
) |' u0 ]/ u" U: O8 o8 i% j is for ATS V2.</description>. Y. J* ^, F6 ^% B/ S) ]; @
<name>yarn.rm.system-metrics-publisher.emit-container-events</name>
( P( m1 R2 t2 ]0 W i) q- g <value>false</value>
, e V- y. N* X, L; [9 C </property>
! Y+ i0 D5 Z- D2 j, C! m" e <property>" e j& s5 g& ?: U
<description>Number of worker threads that send the yarn system metrics
- {" s7 Q2 M; j7 S1 u; c data.</description>
0 ]8 @6 u1 P. P: ? <name>yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size</name>* s; M0 j) M- o5 N% X( m! o4 O* k
<value>10</value>
8 X! h& O/ p* D4 F. c3 x1 }- g </property>
5 d- F D0 r E: O# F <property>- N5 ~, q. U& F2 g- P' r" {
<description>Number of diagnostics/failure messages can be saved in RM for- P7 {1 i, ^2 Q0 D- X$ X8 ]
log aggregation. It also defines the number of diagnostics/failure
7 u' d! s* R" A8 q" X5 U1 [2 |, t messages can be shown in log aggregation web ui.</description>8 X8 ?3 z4 y9 s& \( Z4 W9 R; H" h
<name>yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory</name>( F8 h" _+ Z; E3 N- o; `4 n
<value>10</value>
; f: X T7 a( D' Q/ @ </property>0 ?6 Y. A1 G* _, |2 D4 F9 F
<!-- Node Manager Configs -->8 X( Q4 J" K8 R# r
<property>
0 k# A+ B" S z; z* |: q <description>
* J3 n+ |8 F6 @& d6 J, g" t RM DelegationTokenRenewer thread count
" c* y% t- |( v$ q </description>
# r+ V' m q0 A9 L: D <name>yarn.resourcemanager.delegation-token-renewer.thread-count</name>; V/ x- q" b+ E( Q
<value>50</value>: b6 `' @+ q% j. u- e
</property>5 f; A2 N4 Q6 ^
<property>
( K; c) F: x! _ <description>( M! d! z$ g, s( Q! a: w
RM secret key update interval in ms1 `" L2 w7 q* O* K. X+ [
</description>0 j1 x7 [6 ?+ i2 ~
<name>yarn.resourcemanager.delegation.key.update-interval</name>
7 g" ~$ T% g: x" r% t0 V <value>86400000</value>
, K% B1 C: e4 j- a </property>
; s6 N. j/ Q# b8 S. k2 O5 ~$ I <property>
9 e8 D6 w4 v8 a* G Y* x <description>
! R% q* k. c4 E9 E; ]7 c9 l, q RM delegation token maximum lifetime in ms
4 h9 |0 { f2 e </description>6 N& Z( q, S2 ^/ O! W7 @
<name>yarn.resourcemanager.delegation.token.max-lifetime</name>* l1 W1 V" d% K& Q6 c" q
<value>604800000</value>. T8 k* ?7 ?+ B9 C; p
</property>
; |% l( G- P& Q. D' M$ k3 m <property>
: t3 f0 W- X0 L0 t! C' [ <description>% O) N: g- P. j$ r
RM delegation token update interval in ms" I9 Q: ]& m, I/ m6 c
</description>1 j# r" E) ~3 t6 {1 w0 m( Q
<name>yarn.resourcemanager.delegation.token.renew-interval</name>
: W. {$ x& r1 `. q7 I& `1 H1 j. | <value>86400000</value>
' d4 e3 f3 n7 B- u, ~- m </property>1 g5 C' _( J z: E B3 _* o
<property>
, D" j+ B4 c% \3 V <description>
* u/ H& z8 b4 n Z( t Thread pool size for RMApplicationHistoryWriter.$ s8 a& Y: M1 a5 M. ?, p
</description> L! G2 `7 t, k( X' D( \5 ~! Y
<name>yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size</name>
- }, \4 q$ K: c3 ? V& ^, r <value>10</value>$ Z/ Q* b8 H0 S. F
</property>
# `& L6 U- e3 f* J <property>
; I2 V/ p) g" g: u3 m' h/ \. c: @4 ~ <description>
! y1 D: ], m2 m V& R0 J& a Comma-separated list of values (in minutes) for schedule queue related+ P$ {0 M. S7 @* `. N& v, C: c7 p
metrics.
4 v1 f0 g3 n* B' Q9 U </description>2 S2 g& q; u# M8 E2 ~) c8 q
<name>yarn.resourcemanager.metrics.runtime.buckets</name>
* o; Z: m; x. O& N! j9 n+ l( p! T( B <value>60,300,1440</value>2 i9 o5 y( v! ?) }8 c2 d& r
</property>
' s& Y/ _9 }. r4 {" h <property>
) `9 Y& ?: m0 K; X% `; N0 x <description>
$ S; R8 ^: F z! l Interval for the roll over for the master key used to generate" S# p0 q& q: r
NodeManager tokens. It is expected to be set to a value much larger
0 Y: Q/ w" b4 L5 G than yarn.nm.liveness-monitor.expiry-interval-ms.2 Q& e9 [2 V1 t5 y' o
</description>
# P$ z) W; L7 M! M% m' m <name>yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs</name>
" a8 Z; t% G# H* ] <value>86400</value>" H5 ~; q$ O3 q- Q3 |: s
</property>
- }) n! y9 a6 ]* Y. `7 f$ B, D6 Y <property>6 T7 [8 X1 Y: P% P8 s) w2 o
<description>! B1 Z! ~! Q. G+ y
Flag to enable the ResourceManager reservation system.0 B. L. c) W. w
</description>
, j# Z" o2 v; }. S <name>yarn.resourcemanager.reservation-system.enable</name>
- K) S# |8 Z( U a+ I' I# T3 G8 E <value>false</value>3 @. V) l- o$ @1 U( Z
</property>/ `& M; v& b; n; S: Y V) `
<property>. t8 ], Y3 ] D: O7 {9 c
<description>5 D$ \: q- H; j% ]: |
The Java class to use as the ResourceManager reservation system.
4 x$ L3 D5 I" D( F! W By default, is set to! x) I* W1 M0 ?6 o X, T4 B
org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityReservationSystem" ]* [% c+ u' H8 O7 X- V9 V" { ~
when using CapacityScheduler and is set to5 j; X5 d) w8 Z
org.apache.hadoop.yarn.server.resourcemanager.reservation.FairReservationSystem
* x7 A+ R* ^* J" x. m when using FairScheduler.; {4 r4 r' z7 a: Q$ ~! r# t
</description>
5 C$ m {! t! E1 u4 r! E <name>yarn.resourcemanager.reservation-system.class</name>3 G9 X, v- o1 _4 a
<value></value>& L& H3 {: L5 r1 Z4 B+ [- i( h
</property>
& `1 K) H- V* v& D5 l e. N/ f. l <property>
9 O# G: D9 l0 t <description>* H- z! Z% S9 E2 Y
The plan follower policy class name to use for the ResourceManager
( ~& ^1 _/ P4 _9 w reservation system.
5 n8 x& s* J5 B; I3 L By default, is set to
1 S8 a; W" y0 M: @: c6 u7 J org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacitySchedulerPlanFollower
7 ]; a: O; H$ Z6 w2 i8 Y. s' ?9 W is used when using CapacityScheduler, and is set to
7 ~' J, I2 @! ^- s org.apache.hadoop.yarn.server.resourcemanager.reservation.FairSchedulerPlanFollower( e2 Z2 a% D5 `0 ]% P. W+ |
when using FairScheduler.
4 Y% ^" x: V* {0 Y. L </description>9 }! g) C _3 Z
<name>yarn.resourcemanager.reservation-system.plan.follower</name>3 h( \& f8 u5 L8 T8 \
<value></value>
2 B, T6 J1 B$ b7 m) a: Y </property>
5 \! b6 _5 ]- m+ }( V* z% ]7 | <property>
) t6 l' Y2 b! g- q <description>1 X3 B1 R. o+ V, p5 H9 }
Step size of the reservation system in ms
# [' O Y4 J" k1 y, b8 n' O7 K7 x </description>
$ P" ^3 M/ L7 X4 Y% s1 r1 a <name>yarn.resourcemanager.reservation-system.planfollower.time-step</name>
5 i: x6 B4 a3 w3 l1 d <value>1000</value>
4 h6 A2 V8 f' }1 }- Z0 _( x </property>
0 M! Y# R6 v0 ~7 K, l. [ <property>0 G' Q, e7 a8 f; e+ h5 r: h
<description>
' x/ P! H& `9 r+ \) h& Q The expiry interval for a container% I. R# z2 @9 y
</description>5 _0 `+ L4 d& ^/ }
<name>yarn.resourcemanager.rm.container-allocation.expiry-interval-ms</name>- ?1 T) H5 V1 n4 V* p
<value>600000</value>
! m1 E8 s5 x: g% \( L: e. n </property>- s- y% e9 v6 Z
<property>7 `4 x ?% p2 u- ~3 B' s
<description>
+ G) @2 n7 \& V8 G Flag to enable/disable resource profiles. L j, q0 r8 X1 i- L
</description>
4 W: I9 g8 z2 G! A <name>yarn.resourcemanager.resource-profiles.enabled</name>: e9 X& g# ?* G2 F: w1 L' o5 e1 J
<value>false</value>
$ ~8 u7 U! p6 V3 ~# D5 E# i& ] </property>
/ f& p6 M7 }9 L% ?* @ <property># ~3 L2 ~+ r! _% |9 t
<description>
2 u7 {, I N8 Z& @5 q6 ^- n, @, U If resource profiles is enabled, source file for the profiles5 }2 o! f; d9 z/ o! S
</description>
: ?6 ?. O; b/ t! @( I9 ?$ T' T <name>yarn.resourcemanager.resource-profiles.source-file</name>
1 M+ o/ D6 u3 L9 f0 F" C4 D; W0 T; q <value>resource-profiles.json</value>( ]: [- b, V& Z9 X! Q s# V+ m d
</property>
( B+ k. a2 T }1 D4 c <!-- Node Manager Configuration -->2 Z [6 A, r; t
<property>, j9 d! z/ d R0 c3 d1 N7 [
<description>The hostname of the NM.</description>
! v v K% T* g <name>yarn.nodemanager.hostname</name>6 g7 e8 ]. Z! u* d- s
<value>0.0.0.0</value>6 r/ R* p Y( q: a( }) U# B! z
</property>
8 p9 I& n. e- V/ `3 D <property>
1 X! S5 s; x# i5 t, | @- ~ <description>The address of the container manager in the NM.</description>. }4 e$ Q' w7 P- U
<name>yarn.nodemanager.address</name>" B. K( S5 P7 c$ I# A! C( q: n+ j
<value>${yarn.nodemanager.hostname}:0</value>! ?$ I0 d {. d+ f2 p
</property>" r/ R% {* p! d; a
<property>
3 D2 Z8 A9 T' G0 v( d3 ], r8 m- I% y; } <description>
0 K9 z' H# R1 r/ T7 T% |, J7 S The actual address the server will bind to. If this optional address is; z0 f# o& r4 I6 Z2 o1 z
set, the RPC and webapp servers will bind to this address and the port specified in9 s7 j6 x0 B9 N- @! ` N( S" l3 t
yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is' T# ?! _1 v5 t( }# c) y- U/ a
most useful for making NM listen to all interfaces by setting to 0.0.0.0.
% w9 E/ D; R" Q, R, ^3 k* j/ M </description># k: d" N1 [" b2 D
<name>yarn.nodemanager.bind-host</name>2 f( q. r; q: M' v: N
<value></value>: S- s) W s/ X0 V$ H/ v8 F% X$ i% E
</property>
9 m5 O T8 I) D <property>
9 l; m5 u6 [0 d" p6 B; f- e <description>Environment variables that should be forwarded from the NodeManager's environment to the container's.</description>
$ w* c6 Q6 e9 k5 ] <name>yarn.nodemanager.admin-env</name>$ ?# t/ j3 |; [
<value>MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX</value>6 a4 t$ k) J6 L4 R0 ?" ~% ^
</property>
; U) h+ ?* I6 h* ] <property>- r' b9 e2 S* C9 {% Y: J
<description>Environment variables that containers may override rather than use NodeManager's default.</description>8 y( a; c5 F9 \* p. ~4 P' m
<name>yarn.nodemanager.env-whitelist</name>6 p- ~% }6 x# m. y! l
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>3 a: o0 n" x* p' U8 P1 P. _( a: _. }7 R
</property>, U. V% |( s( J% F4 a
<property>
! n: N: X/ r$ g! Z# P% Z6 d <description>who will execute(launch) the containers.</description>( j+ a H! y; e3 {% [6 d
<name>yarn.nodemanager.container-executor.class</name>
& A- [5 S: M( z+ d7 a3 k. G1 c <value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value>7 ~: M- Q7 d) f9 N, A' g; c6 U" ^8 O
</property>
) p6 a3 y# N* N% ]" Q' O8 g <property>& A2 e8 O2 o6 B1 k
<description>Comma separated List of container state transition listeners.</description>
- |! x" O4 E& B <name>yarn.nodemanager.container-state-transition-listener.classes</name>
. v$ ]/ a- l6 r' |3 I1 r1 p <value></value>/ k/ s! I$ F. w% H3 B8 ?
</property>5 c0 J* I) a m" f
<property>$ w: z) m- f. {3 y. v3 p
<description>Number of threads container manager uses.</description>
. q( R6 Q; k8 j <name>yarn.nodemanager.container-manager.thread-count</name>
/ j+ ]4 b# ]7 V4 J2 \3 D+ o( R/ i <value>20</value>
8 W5 I8 y O* F0 y- E </property>4 S, ~& K6 c9 H6 ?
<property>
- c( N7 o+ ]! F. e* B n <description>Number of threads collector service uses.</description>/ H! S0 _$ b: C1 T) T; ~
<name>yarn.nodemanager.collector-service.thread-count</name>4 E# h+ ]( k: B1 B& |
<value>5</value>8 _/ i4 G0 A4 J5 I6 m* q/ c3 `
</property>
7 J6 D7 B3 p) W <property>
3 K! a! a% C3 r) H9 h' k <description>Number of threads used in cleanup.</description>
' K" ^; _9 w; V; ? <name>yarn.nodemanager.delete.thread-count</name>; H, k, V; D& M$ ~5 Z# }
<value>4</value>
% W! k! B' h8 T- Q, {) ^5 j </property>
0 K5 l3 U q) z" }! c0 ~ <property>5 D, I6 \" G/ c1 k+ _, s% ]: k+ ]
<description>Max number of OPPORTUNISTIC containers to queue at the) v8 r' K, z: C
nodemanager.</description>9 V ^9 j; C5 V! E
<name>yarn.nodemanager.opportunistic-containers-max-queue-length</name>
7 `1 _3 g( W( i0 |) K- F# J <value>0</value>7 |( N( v( ~1 T9 E- K# p
</property>
# O6 v7 y! q8 {5 x <property>
& \7 N! J I L u9 A( \/ a <description>; }1 o0 R( `9 [: K
Number of seconds after an application finishes before the nodemanager's ' t% ^0 \6 P& `* o3 O6 T
DeletionService will delete the application's localized file directory$ F3 H3 B: {9 e* e1 s8 Q/ R g
and log directory.
( {, X+ l/ u1 ~! N. ~ To diagnose YARN application problems, set this property's value large
+ v) W4 S3 p) |' }% J2 G enough (for example, to 600 = 10 minutes) to permit examination of these3 |, z9 O( Q7 z. Z3 i# ]
directories. After changing the property's value, you must restart the : I% W, ?$ ^4 B7 D/ B- y8 ]& n- A
nodemanager in order for it to have an effect.6 R7 W, o5 v; r# E
The roots of YARN applications' work directories is configurable with! J" H2 u# V3 S4 N, K" J
the yarn.nodemanager.local-dirs property (see below), and the roots
, k' Z- h4 r8 h% t7 ?; k/ w of the YARN applications' log directories is configurable with the% O+ H6 u5 ] M1 {. C( Q) [
yarn.nodemanager.log-dirs property (see also below).+ D3 \- ^- t; i5 N1 a' p; k0 A
</description>
. T6 [3 ^! b! ]: M7 ~( I <name>yarn.nodemanager.delete.debug-delay-sec</name>
2 g8 G6 B7 e/ V1 u% R6 l' P' T <value>0</value>
, c7 u1 H2 h) M2 Q% x& L [ </property>
7 n/ L5 o4 C% i/ ^ U <property>" b* K' p" K& K @. K
<description>Keytab for NM.</description>
. p4 Q0 [7 [: V9 o& W <name>yarn.nodemanager.keytab</name>
3 z+ l2 _- c* ]0 T" R* @( ~ <value>/etc/krb5.keytab</value>
: U) u z" }7 l8 o </property>
3 y- o: _/ `4 ]2 W! c, o <property>
$ b9 t; y0 g8 m! E# k9 J2 s4 E2 @ <description>List of directories to store localized files in. An
9 `* j1 Q' D: B% ^3 P: L/ G application's localized file directory will be found in:. l: s6 ~) V7 i5 T+ X7 D
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.9 l' f7 D& \- x! ~& M N
Individual containers' work directories, called container_${contid}, will0 T4 J4 m: y! [7 ~' }& b5 }5 |
be subdirectories of this.
3 F8 |* L3 R6 d% `: H </description>
* @' @% h' ^5 U+ ] <name>yarn.nodemanager.local-dirs</name># j1 Z6 p: `! G
<value>${hadoop.tmp.dir}/nm-local-dir</value>
! D+ V! p( Q5 t </property>0 }* }4 c+ s% s: h; ]* b/ V
<property>3 ~: ~" ~) [/ U$ I1 P
<description>It limits the maximum number of files which will be localized
& Q5 T3 a* k4 @3 a$ T in a single local directory. If the limit is reached then sub-directories
9 h8 ?7 f& y7 ]4 t6 d F will be created and new files will be localized in them. If it is set to9 E8 h. \1 L u& S0 C2 e
a value less than or equal to 36 [which are sub-directories (0-9 and then
: r; v& ]4 F7 Y8 E. s% N2 h a-z)] then NodeManager will fail to start. For example; [for public
4 |% L' Q$ c$ U4 z4 \ q7 E cache] if this is configured with a value of 40 ( 4 files +
% t* q/ k) q' S% u! {5 P$ S- a+ F: i0 b 36 sub-directories) and the local-dir is "/tmp/local-dir1" then it will
% S: j3 l# u) U6 w allow 4 files to be created directly inside "/tmp/local-dir1/filecache".
$ @. A! y: {2 u' j1 R For files that are localized further it will create a sub-directory "0"( \% |( N4 W/ P4 j
inside "/tmp/local-dir1/filecache" and will localize files inside it/ V) V! i" g9 a( K
until it becomes full. If a file is removed from a sub-directory that
! `2 @% I/ o; P: k is marked full, then that sub-directory will be used back again to
5 q; q5 ~6 o# S2 a localize files.* y0 Y8 A9 R" K
</description>/ }" \' ~ v) j2 ^
<name>yarn.nodemanager.local-cache.max-files-per-directory</name>5 p; e, n5 Z5 c% k9 T/ r# D) U
<value>8192</value>4 d7 w5 y3 n. a0 f& S* T( K1 t: K
</property>
. h; m5 A/ x, z- T <property> \& E6 a1 L1 N
<description>Address where the localizer IPC is.</description>+ e, {( w/ z. L
<name>yarn.nodemanager.localizer.address</name>6 d4 j" ]* m/ {$ P$ R
<value>${yarn.nodemanager.hostname}:8040</value>
% v* v/ @2 y) Q' N. w </property>
: n" w# s, m8 d7 f, I8 [% y <property>
`, {/ W9 K6 P, K/ ?% M <description>Address where the collector service IPC is.</description>2 j. j7 T6 y( u2 d
<name>yarn.nodemanager.collector-service.address</name>" d# s' ^( K6 P! z, l6 K' e
<value>${yarn.nodemanager.hostname}:8048</value>
' l2 d3 J# X Y6 \8 C8 l" H5 X7 D# [ </property>; U1 d+ e5 D( i+ d3 K5 X
<property>" U1 V A& l" o8 J% S
<description>Interval in between cache cleanups.</description>4 p) x/ P% V# q1 @- S* G
<name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>8 e; q) Z+ _% z
<value>600000</value>
9 e% y$ U) t: Q: `) U </property>$ \0 _( N) q; w( a) T
<property>
6 ?% \# K3 _" `' t$ @9 |% y' g <description>Target size of localizer cache in MB, per nodemanager. It is
2 F1 \1 F- ~; p8 @: e a target retention size that only includes resources with PUBLIC and
+ [, b* m8 P4 W PRIVATE visibility and excludes resources with APPLICATION visibility
7 m7 C; M, g, i2 T </description>
% W$ M4 p2 U1 m <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
8 S8 C/ b2 i& s4 z; r& x <value>10240</value>6 i6 w+ K. ]( _7 ]5 h
</property>; ^! X- |' y' c+ }% f( t; u
<property>% t1 s/ Y+ l/ ~( |4 j
<description>Number of threads to handle localization requests.</description>$ _- v) n) c* e! k& v
<name>yarn.nodemanager.localizer.client.thread-count</name>
4 H( U( S5 R3 u! F <value>5</value>
3 j2 l! u7 ^, E </property>' L: d7 z5 G7 z
<property> B' P+ |1 c4 u! X7 a7 h% S
<description>Number of threads to use for localization fetching.</description>
* n& Z4 G7 Q5 O2 d$ K <name>yarn.nodemanager.localizer.fetch.thread-count</name>
$ G. L+ a) Z% G8 u <value>4</value>, z9 d! g, c7 v5 v) t1 U# }
</property>
, Y! y6 z: H( L <property>3 e0 S8 }6 {2 w( w# d
<description>
5 r, [& B6 x: g# B5 ^6 _4 G </description>+ }/ a) ^5 x6 c5 u/ g% i& Y
<name>yarn.nodemanager.container-localizer.java.opts</name>
' `9 U) O8 w" f5 [$ h3 A <value>-Xmx256m</value>
6 J6 A% h0 M/ G; n9 A9 _! l8 k </property>7 Y% U0 r6 u% O* X' ?
<property># {: ?3 X+ N2 h# z, w
<description>
; Q- e3 Q* a+ c The log level for container localizer while it is an independent process.7 b- N, Q, v& l: b) s5 o
</description>9 n( N8 Z9 B/ ?. }8 s7 }0 n
<name>yarn.nodemanager.container-localizer.log.level</name>
c1 i4 r+ \1 ^ d8 A# T. _: T1 ? <value>INFO</value>
# K4 |5 d7 N2 C0 q; v$ y, b3 c$ y </property>, O# k) t, U* H# D1 s
<property>9 |9 w' r: @$ Z
<description>
; r# r0 M0 W1 W; f6 Q Where to store container logs. An application's localized log directory
5 ?; h; q9 \3 ?5 f1 l will be found in ${yarn.nodemanager.log-dirs}/application_${appid}.
, J2 J/ @: {& @0 v Individual containers' log directories will be below this, in directories - y- I. U; p6 ?& F+ T3 D; d
named container_{$contid}. Each container directory will contain the files6 }$ W3 S3 D) B3 c! k0 i2 v) _
stderr, stdin, and syslog generated by that container.
W& Q7 y' c" V, t* R% Q: N </description>! J4 N3 Y' Z& C
<name>yarn.nodemanager.log-dirs</name>
* S) M m; `$ ?& m- t$ w <value>${yarn.log.dir}/userlogs</value>/ |# g) }- X9 n/ q& {* V& q3 s3 ^
</property>
7 Y1 P# p) ~7 ^8 V6 L; s <property>; N) ?$ D, N- v* G+ V
<description>
! V+ F1 e- z3 Q$ U# ^ The permissions settings used for the creation of container4 `9 p0 e3 ^7 \
directories when using DefaultContainerExecutor. This follows
0 [3 n6 C6 k) l$ n; G! g standard user/group/all permissions format.8 { z" R( n$ i3 S
</description>
; X+ \' \8 C1 a" G0 b7 U <name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>
3 E: ~5 G) R D+ t+ J <value>710</value>
0 H9 @, Y4 E# ?, `/ S </property>( j e- j2 u( ?6 X+ O, K
<property>) z. W k( e C7 K4 y
<description>Whether to enable log aggregation. Log aggregation collects; a3 I& ?8 x( Z# K1 K% f
each container's logs and moves these logs onto a file-system, for e.g.. d. w0 f k0 _+ y
HDFS, after the application completes. Users can configure the/ o1 a6 v" b2 b- w6 g. g
"yarn.nodemanager.remote-app-log-dir" and
( u5 M4 e3 T* h( e "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine
8 k# ^* h- B3 h where these logs are moved to. Users can access the logs via the
% ]; R+ H1 a- G$ n Application Timeline Server.. O! U- f: K/ p/ K! V) I
</description>/ Q Z' c: W/ w4 z
<name>yarn.log-aggregation-enable</name>5 t9 g( h& V$ D6 Z# ]
<value>false</value>
8 s9 R- `' B2 Z# V9 i# m. {3 x </property>. ^7 l9 N- e. T- j0 l' |3 J
<property>
0 n4 I4 s9 H% {( H6 J <description>How long to keep aggregation logs before deleting them. -1 disables. 2 Z0 t/ A4 x. m3 a) U! g2 P" c& R. O
Be careful set this too small and you will spam the name node.</description>
5 c% a6 ^+ y6 N- x( t% @+ C: [ <name>yarn.log-aggregation.retain-seconds</name>
! Y- E5 U' P1 j# b6 ?# r) T1 h <value>-1</value>, ~- e" ~% D$ A1 T
</property> 4 p# O E$ l; a) t, T# M2 {; g
<property>' v; O' v: |& O, v$ q0 g
<description>How long to wait between aggregated log retention checks.
8 O9 g* V0 E8 q+ s- v# Z B- b If set to 0 or a negative value then the value is computed as one-tenth% F. }& d. d1 L; R
of the aggregated log retention time. Be careful set this too small and8 N" q& T& H! l8 N
you will spam the name node.</description>
`' x& r- A* t% ~& P2 I0 O <name>yarn.log-aggregation.retain-check-interval-seconds</name>4 |! b' U( _7 H
<value>-1</value>( k E' d' G! @( B1 k
</property>) D; |$ Y' V& c
<property>
; L3 {! e3 J% v <description>Specify which log file controllers we will support. The first7 m5 a6 q3 g9 ?! w
file controller we add will be used to write the aggregated logs./ }# O2 r _& Y5 ]( D! _: G
This comma separated configuration will work with the configuration:
. N3 e$ O* U, o1 Y yarn.log-aggregation.file-controller.%s.class which defines the supported
( ^3 r# s9 }' `" S/ |$ T file controller's class. By default, the TFile controller would be used.
) t9 ]) J$ s+ t% h( R& Q3 H* @( s The user could override this configuration by adding more file controllers.
% R4 x$ O! }' x. ?( L- S+ b Z) g2 o To support back-ward compatibility, make sure that we always
+ @) d) d, |9 `4 [' e8 J j6 ^ add TFile file controller.</description>3 F" L& s4 @. F( _1 ^
<name>yarn.log-aggregation.file-formats</name>
9 N- S9 Z2 y. {9 X <value>TFile</value>+ }4 C d# O% S# m2 I6 D3 h
</property>
4 c9 R* n$ i6 X+ ] <property>, N0 X: ^: J2 G8 [) ]
<description>Class that supports TFile read and write operations.</description>
0 N1 t: T2 b! \8 ?; Y+ @ <name>yarn.log-aggregation.file-controller.TFile.class</name>
$ ]0 }* r. Z d, s, Y+ v <value>org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController</value>; W: Z/ y' T/ d, g6 C) [
</property>; H+ C8 z! Y$ j: b P) z4 W4 e+ G
<property>, K E& U7 P# R% e, ]
<description>
# Q: B; d$ b) _& k4 Z How long for ResourceManager to wait for NodeManager to report its
- q& [% \# i- w& P4 o+ {8 a. L log aggregation status. If waiting time of which the log aggregation
! `* ^7 |/ Z# b# Z status is reported from NodeManager exceeds the configured value, RM! ~ D0 s' z3 T& n8 d1 T
will report log aggregation status for this NodeManager as TIME_OUT.
" h# e4 j8 k" }; K p1 A3 w# I This configuration will be used in NodeManager as well to decide: @5 m c9 ^) C& d3 h: u) G
whether and when to delete the cached log aggregation status.
9 t, q; ]: c$ Z# q6 m </description>
2 ~0 e" [/ z# ?- J$ e: x <name>yarn.log-aggregation-status.time-out.ms</name>
1 m! q( L0 ?7 p- r <value>600000</value>
3 Y' k$ F* ~% y( T/ D2 J </property>: P8 V0 Z' A( X/ N% y
<property>8 L: u7 b8 v; v. h. b
<description>Time in seconds to retain user logs. Only applicable if6 m- p' E% l; b9 F+ {6 R& P( Y. S$ C
log aggregation is disabled
; i ] V; {/ r2 l% H </description>
, ^0 [! c$ Q! |+ Z, d <name>yarn.nodemanager.log.retain-seconds</name>
5 F! @1 [$ s5 t% A <value>10800</value>* v/ `2 K9 i, o7 E! U0 p J1 D
</property>
6 M8 T: A% R/ A! M- K1 z <property>
4 M9 f E) H* t <description>Where to aggregate logs to.</description>
& a/ E5 }+ M. u2 v* X1 n# B. k3 @ <name>yarn.nodemanager.remote-app-log-dir</name>2 k7 v% P+ c/ g) p) A
<value>/tmp/logs</value>
1 C! P4 R* I3 F$ A6 V </property>9 _- \9 y7 k/ E3 q! y8 u
<property>
' f4 _4 U b" U1 |1 J, @ <description>The remote log dir will be created at
0 _# ]5 g) C7 H( c$ d" G j& |4 b {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}7 L3 j# T5 K5 i) i# U$ U L
</description>
1 @' Z+ e- n% l! ?2 n9 N <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
, L6 w* m9 T7 M5 X- a, H4 [ <value>logs</value>
' D" P3 M' _" ^0 d </property>
: V$ O1 i$ u8 a6 c, T! W <property>
9 ~. |( \4 ^- X3 @ <description>Generate additional logs about container launches.! Y. _/ ~6 D9 ?$ w1 w9 o
Currently, this creates a copy of the launch script and lists the5 ?) N2 {% t- z) X+ a; r2 z
directory contents of the container work dir. When listing directory
0 |* L7 J/ P/ r) G& ]2 b contents, we follow symlinks to a max-depth of 5(including symlinks
( @' U8 c m' X. I' J- p- C" G+ M which point to outside the container work dir) which may lead to a2 i9 L- q }. E8 {
slowness in launching containers.
; `; E% v: a# M </description>
* E7 m7 J4 a `1 |( d% C- K/ K <name>yarn.nodemanager.log-container-debug-info.enabled</name>1 T, r! q! ]# ?( A" h. |: }
<value>true</value>9 Q+ X1 P5 q$ P7 M3 l/ W( ?
</property>
# }* \/ |$ [6 G& T4 ` <property>
5 l& N! h# l$ ? <description>Amount of physical memory, in MB, that can be allocated & m% J8 B3 ^" T( C* Z
for containers. If set to -1 and
/ P( U9 W: g z yarn.nodemanager.resource.detect-hardware-capabilities is true, it is6 h! Y# W" Z+ ]$ n
automatically calculated(in case of Windows and Linux).
& s h. z6 X" g; D- q In other cases, the default is 8192MB. S/ ?0 U, P( a: X4 W" T4 F
</description>
% h' J8 \( v! K: L) B <name>yarn.nodemanager.resource.memory-mb</name>( u; [, u5 h4 C; ]0 D# [7 v
<value>-1</value>+ @# U3 Q2 G3 I
</property>* {& q7 Q% v. ?
<property>
) {% g6 y, k, p8 [ <description>Amount of physical memory, in MB, that is reserved
; b* A4 y+ P: B' i for non-YARN processes. This configuration is only used if
3 V' M; A% D! \. g yarn.nodemanager.resource.detect-hardware-capabilities is set
6 k$ n' r9 V4 H' ]3 ]+ r1 s to true and yarn.nodemanager.resource.memory-mb is -1. If set
: m2 w1 [" B% g! ^, R; ?/ L' O8 u to -1, this amount is calculated as
' s3 Q; `: \3 B1 i k 20% of (system memory - 2*HADOOP_HEAPSIZE)
8 r! J+ U4 @% @. ?8 d ? </description>; k$ S& o4 J3 d6 w `4 G, I
<name>yarn.nodemanager.resource.system-reserved-memory-mb</name>
0 ~) E$ Y L& r9 I <value>-1</value>
1 d; C, v. t6 n </property>
. e% T8 H% q4 @; X& r. k- M <property> y2 i% ]2 d; N* S; G
<description>Whether YARN CGroups memory tracking is enabled.</description>
5 s+ K# |# N! R. M; y: c. ~ <name>yarn.nodemanager.resource.memory.enabled</name>
+ B9 F0 b+ h0 s4 u2 C. ^ <value>false</value>
/ E+ y: T( C0 z; v! ]) C6 n </property>
& D- S* O3 [) o# Z <property>
1 V7 X, w, x8 D/ {$ | <description>Whether YARN CGroups strict memory enforcement is enabled.& o. T" X# } ]: e" H4 {: r/ \1 o
</description>' k L+ e6 d0 D) M" X a1 P R+ F
<name>yarn.nodemanager.resource.memory.enforced</name>
* W( e* O- x7 z% z+ v0 k6 R1 G <value>true</value>& a1 X" s1 I+ g7 c+ k
</property>
, N7 f) C# `# G0 u2 C <property>6 r* W- D9 j$ m; D4 u: d8 V
<description>If memory limit is enforced, this the percentage of soft limit4 n, P9 z2 Q/ C+ g
compared to the memory assigned to the container. If there is memory ]& c: r& [; v# O7 m! n
pressure container memory usage will be pushed back to its soft limit
^1 j D* m3 K( u by swapping out memory.+ r U- |9 N) b" Z2 s# U3 D
</description>
/ Y; H4 T7 }1 B1 W <name>yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage</name>: e# L8 b$ e: M' E0 g- ?. G
<value>90.0</value>
4 T3 c! [# q, A" d </property>3 @$ r( N' c0 A. C" B! V: W
<property>
5 ?! _) }% }1 E1 R0 v* c, k; _ <description>Container swappiness is the likelihood a page will be swapped9 [. ?% a1 g/ S) a! _
out compared to be kept in memory. Value is between 0-100., v) d6 u( U* p! Q. h0 |5 ], a
</description>
, |& s, l* E1 w" w0 z8 u <name>yarn.nodemanager.resource.memory.cgroups.swappiness</name>
* w* g# q# v) N, }9 b <value>0</value>
! J" k8 P" x% f* _9 U. R$ c7 X; x </property>
7 e4 {4 H" `# T <property>- U3 e- ~( U2 V1 E3 I4 I. [
<description>Whether physical memory limits will be enforced for
; Q3 U; G6 c5 ]0 W' f containers.</description>' O8 Z0 y/ S3 r+ {
<name>yarn.nodemanager.pmem-check-enabled</name>
: C: D3 j; C( r) S <value>true</value>
8 j2 X5 x, ]/ |5 `/ N </property>& ^$ y. T7 O, i6 E/ I4 j0 A3 E; l
<property>
2 d, E4 m& [% G% x3 W <description>Whether virtual memory limits will be enforced for
# \- d% Z, n1 Y0 u' z containers.</description>
4 ?5 g0 a# y4 E- k# H! o <name>yarn.nodemanager.vmem-check-enabled</name>
- ?. c7 j9 k4 u; y E" k <value>true</value>* o2 g, L, V6 u5 d; @: C# E# H
</property>; D9 M2 S& J, ^3 [
<property>
! b3 V4 X" c2 K$ R <description>Ratio between virtual memory to physical memory when. l1 Z, C4 h' W
setting memory limits for containers. Container allocations are
, ]) D& z$ E1 _$ a+ I0 Z expressed in terms of physical memory, and virtual memory usage* B5 I8 J; w7 q* a, A" T! d
is allowed to exceed this allocation by this ratio.1 r/ z# s2 d" q! L
</description>
& |8 O5 B1 Z- I P% C8 j" c <name>yarn.nodemanager.vmem-pmem-ratio</name>8 y! u, [: a( H$ S9 N" c- e
<value>2.1</value>" _/ @. u, U; p& F0 u
</property>7 }$ g' Y% s _% W, }0 l7 R+ V, S% K
<property>5 N( g& R: ^5 k( L
<description>Number of vcores that can be allocated
) ^" C+ r+ Z! y! a( b+ h( }1 S for containers. This is used by the RM scheduler when allocating$ I2 C+ n8 x2 N: ~; ~: F5 h7 k- d
resources for containers. This is not used to limit the number of f. E, A. U) a7 W
CPUs used by YARN containers. If it is set to -1 and; k3 G8 o* C Y' G! a
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is& e. k; |. H G3 g
automatically determined from the hardware in case of Windows and Linux.* P+ x5 W7 f( _( Z9 p) E1 r3 @
In other cases, number of vcores is 8 by default.</description>
) D" }6 ~' r& \4 N, j: e& P+ N2 j4 {+ w <name>yarn.nodemanager.resource.cpu-vcores</name>' A ]# F7 u7 y4 B9 o
<value>-1</value>% V. t2 v, q: M- ^
</property>! M" ^) I4 ^7 @. b" F
<property>
: ~" e# |* i3 S <description>Flag to determine if logical processors(such as
o% M& m6 M8 r6 N) o% W6 Q hyperthreads) should be counted as cores. Only applicable on Linux
8 v4 z- d% q+ S4 A2 P! u; N# i; N when yarn.nodemanager.resource.cpu-vcores is set to -1 and0 N. s/ s" C: }; E1 _+ Y
yarn.nodemanager.resource.detect-hardware-capabilities is true.4 c6 S3 y" b. \. B% Y( y/ k
</description>
1 c5 S% x" ]1 m( B <name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>8 p3 U& ^5 S/ G' o
<value>false</value>
9 r/ _! s0 g4 g* b2 _3 O% A8 g </property>+ d/ r/ o+ g8 |* j9 K- I8 p. J# t
<property>; t' H: K; L8 K% c9 V! Z& T8 O6 A
<description>Multiplier to determine how to convert phyiscal cores to6 x! }) i9 z1 h
vcores. This value is used if yarn.nodemanager.resource.cpu-vcores
* z: d& E- C4 ~, [, o3 c is set to -1(which implies auto-calculate vcores) and
; f1 c# _" v9 R5 Q: h, ^- n yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The
2 U* m* \- [" V: H" C. S, K: ]* M number of vcores will be calculated as
8 U+ Y/ Q3 ^/ S! q" Z# M% [ number of CPUs * multiplier.% v* d+ ^% W2 q; A0 w
</description>
" Z! o9 T/ y! b/ U8 o" i <name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>7 R3 g- C/ ~! `& @( t/ m
<value>1.0</value>" W5 Z1 E* E8 j% p' N9 e9 I+ E I' @
</property>
7 d& Z; d: C, Q+ i' w# I <property>/ t2 t7 t& P! M, }% Y
<description>5 ?" d2 @' R3 i6 `# U+ G2 W
Thread pool size for LogAggregationService in Node Manager.( y& x* c E. ~7 c, h4 e/ f4 U
</description> H; G- U3 F. [+ K% e
<name>yarn.nodemanager.logaggregation.threadpool-size-max</name>; h2 h& X6 V: ~" t, P* j* a
<value>100</value>
$ n! z* y/ N e8 j3 M </property>& ?3 t% @+ i9 D% ~) W
<property>
/ [0 ?/ Z+ W" J <description>Percentage of CPU that can be allocated0 g( y. U2 g! J+ i8 O: d
for containers. This setting allows users to limit the amount of
7 N; k, T5 L0 \6 z$ ~ CPU that YARN containers use. Currently functional only
8 n t- G- i* X* D* s- I on Linux using cgroups. The default is to use 100% of CPU.. [; e9 l/ j+ ?& Q) ^
</description>0 [, O$ F2 h) _
<name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>
% ?3 Q% o# s3 |9 |9 f! k" U <value>100</value>/ Z2 |% N& I1 A4 b6 K3 {! y
</property>2 V0 k& H+ G5 ~! W+ T7 O" O
<property>9 V) Q) Y9 N" S# g6 f
<description>Enable auto-detection of node capabilities such as6 E! R' B1 H9 W4 K" F, Z H
memory and CPU.
1 j# ^/ l3 |, z' ?5 ]8 h, ^/ ?0 R </description>
6 {2 x% y) r& R4 W. r$ V2 f) I <name>yarn.nodemanager.resource.detect-hardware-capabilities</name>6 E4 E: h" r3 u! q3 j0 l, |- w
<value>false</value>9 d/ W0 K5 V# Y9 m
</property>
' R1 K! v& i0 a( _: e8 C8 c <property>
+ ?3 [* [7 F4 b' Y, y <description>NM Webapp address.</description>
) k$ h0 b& A) D( Q, p! V <name>yarn.nodemanager.webapp.address</name>1 t3 V1 i+ T( R7 c
<value>${yarn.nodemanager.hostname}:8042</value>
" x+ c' S u& c( N1 x </property>
J6 k. W) p! {2 N; Q, ^ <property>/ E2 F R8 a3 ]7 y& L6 U. L! s" b& \
<description>
% \1 I0 r! `6 o* A2 @ The https adddress of the NM web application.: \' y k' C( F6 C0 S5 o
</description>5 L L, F2 N& G) W+ p4 M) u
<name>yarn.nodemanager.webapp.https.address</name>% k. H8 I4 a0 L
<value>0.0.0.0:8044</value>$ ]- ^7 S0 L* y0 ?8 Z. V$ I! U
</property>$ b$ W9 y1 \$ E% P# i( L
<property>
4 }& \1 b+ r- U. X6 K8 g/ j" g <description>
% z0 A; K* U/ P, k6 g The Kerberos keytab file to be used for spnego filter for the NM web5 \+ f' t/ c. l' h, _
interface.* j% t; S# S9 D
</description>2 s+ X# o" R: N7 K4 L
<name>yarn.nodemanager.webapp.spnego-keytab-file</name>
( S ~" J9 K% A+ {: ^6 @ <value></value>
; u, t" D) i! j+ s$ B6 m </property>2 Y* y( Y( E$ w* T; p% ?$ a
<property>" |/ p1 \: D p
<description>
8 |" ~# g* b0 i6 e/ Q The Kerberos principal to be used for spnego filter for the NM web
" j- S0 `% i2 {, j2 L interface.
: @' t) |" b) Y4 S4 R( N# e/ F$ D </description>/ E0 u2 M* @% K% z/ R
<name>yarn.nodemanager.webapp.spnego-principal</name>
( k- |; B( Y% I% M/ q3 O7 O3 g% O <value></value>- L( N1 o0 z, \1 N3 u7 P
</property>* m+ `- Z$ T2 w) d; m4 ~
<property>
; x# f4 k0 x; H( n0 x; t g <description>How often to monitor the node and the containers.
* [8 n" d5 C8 \/ Z: x1 {, K M# ~ If 0 or negative, monitoring is disabled.</description>
. }, k' g& M' O* r7 B5 d9 z <name>yarn.nodemanager.resource-monitor.interval-ms</name>
+ w& N9 Q6 r6 B" m+ e# S3 B0 c3 G <value>3000</value>5 R. T) {( k9 Y, J$ X) X
</property>& B9 u% w' j4 T( n
<property>
& p0 }/ y1 [- } <description>Class that calculates current resource utilization.</description>( Y' j- @ v- |- G, C
<name>yarn.nodemanager.resource-calculator.class</name>
0 g1 a* [& ^) C; u6 Q/ S. ?' s* T# U </property>
4 E( U8 n. |9 t( b8 j: T& @9 M <property>, P6 ` @7 A7 @+ u7 P
<description>Enable container monitor</description>
9 M3 R) z3 C6 [3 u <name>yarn.nodemanager.container-monitor.enabled</name>
/ l* f" w, z4 O4 L <value>true</value>4 s$ j8 C4 T5 K; |0 L! b' {
</property>
7 X o1 _: L* q <property>
, W* h% }8 H* ~ <description>How often to monitor containers. If not set, the value for
6 A' R2 g1 E- o yarn.nodemanager.resource-monitor.interval-ms will be used.
B0 D9 i: p' [$ x3 x' o If 0 or negative, container monitoring is disabled.</description>
3 B3 K7 x9 M* d R <name>yarn.nodemanager.container-monitor.interval-ms</name>8 a% W6 r5 O; T6 n7 O
</property>
& X! D' B% a. a' ` <property>7 }" N; H2 o; t- J" Q
<description>Class that calculates containers current resource utilization.- u: t1 e* ?$ b Y( @) X
If not set, the value for yarn.nodemanager.resource-calculator.class will
/ k# X3 l3 Q% @! B be used.</description>
2 ]/ c2 u3 i, R <name>yarn.nodemanager.container-monitor.resource-calculator.class</name>) ?7 @) F. I3 O2 A f# c4 e
</property>) n8 z1 S# x& d' s6 l
<property>2 ?: a' r9 s: _# X& f1 F% F+ T
<description>Frequency of running node health script.</description>0 _* e! q$ S8 [$ A
<name>yarn.nodemanager.health-checker.interval-ms</name>* n* M7 N" ?& @& F$ g$ v1 \
<value>600000</value>
# l: z, z# h" E; B* U </property>- @$ J5 {' ~5 E5 {2 P" A5 \2 N* `
<property>
. G( `% T( e% {) u2 g9 Z% D <description>Script time out period.</description>
* R. J: q% j- l3 L <name>yarn.nodemanager.health-checker.script.timeout-ms</name>
. s+ K( y1 P; b- N+ ~1 l( [ <value>1200000</value>' N2 p m8 {5 W) V% i- M
</property>
+ o0 P- m, k/ ~# T <property>( }" G" n0 N9 w# L$ @2 U( }
<description>The health check script to run.</description>
9 O1 T& }1 j( W3 Y. d <name>yarn.nodemanager.health-checker.script.path</name>
5 W# n# ?' J( ^& K* a) D2 i <value></value>
+ w T* G$ H' D! A ]& D </property>' B2 k5 S1 o2 F) ~0 |
<property>* k. l" |, z- X8 S: g4 j
<description>The arguments to pass to the health check script.</description>' a `1 t) B" V
<name>yarn.nodemanager.health-checker.script.opts</name>. e, }' f9 S, Q1 X: G/ @
<value></value>
$ f' o- l1 `+ k" C- c9 Z </property>
/ r: E# ?" I0 S+ y7 p. j8 `# A1 Q* u <property>1 A8 q @2 P- R4 u2 O
<description>Frequency of running disk health checker code.</description>
) V" t) V U; e5 m <name>yarn.nodemanager.disk-health-checker.interval-ms</name>
* R" t# m: o3 f <value>120000</value>+ c; l4 i# U5 q+ u$ I' Q" o
</property>
! ?5 I" W1 [2 v <property>
! \3 n% n1 U: a! Q" B) m) @% N4 y" i: z <description>The minimum fraction of number of disks to be healthy for the' t9 \$ \! i3 i: o- f) k
nodemanager to launch new containers. This correspond to both
3 n3 C6 F, g+ k2 L* x& W yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there y' ]% }7 z! X) `/ \8 S% u
are less number of healthy local-dirs (or log-dirs) available, then
, ]% V4 _" w& m. t. D# k. o. \ new containers will not be launched on this node.</description>
1 m4 N2 ?! |! ~7 I <name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>7 \' ?. S' I# d1 e
<value>0.25</value>
- d }) T! z/ E, A V( S/ B </property>0 a: e! Y" d. W9 k' S5 A$ P
<property>: Y6 Y, o0 j7 G6 u. }0 f' K
<description>The maximum percentage of disk space utilization allowed after % u8 G0 f1 C! d5 T/ p7 d
which a disk is marked as bad. Values can range from 0.0 to 100.0.
4 X8 j9 g( V7 _ If the value is greater than or equal to 100, the nodemanager will check
9 [4 A- K+ _0 w1 K0 A# n' p for full disk. This applies to yarn.nodemanager.local-dirs and6 [: A+ q' y4 v2 {
yarn.nodemanager.log-dirs.</description>: }( a' L+ w& ]7 H. j+ p
<name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>5 T2 w$ z# w! l9 g% s& Z- B
<value>90.0</value>2 H1 u! Q) n5 O; U
</property>, F5 t b7 o% g0 L x3 M
<property>9 M( h3 B# {# r" b! e2 s
<description>The low threshold percentage of disk space used when a bad disk is
; q5 c6 P+ t6 R3 A% B marked as good. Values can range from 0.0 to 100.0. This applies to: i0 z% e5 [! Z7 h' q
yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.
: T5 `7 l. x, w6 b% y1 u2 {" b Note that if its value is more than yarn.nodemanager.disk-health-checker.7 j* c8 |) s0 M" Z4 F" [/ Y& c, |) H6 W
max-disk-utilization-per-disk-percentage or not set, it will be set to the same value as
+ S% H4 d8 p6 y( d& g5 v& m# C yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage.</description>; W" D! `1 e9 i0 ^& Y$ X0 O
<name>yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage</name>
( T' i. Q) b" T7 T- R6 }' i <value></value>& |9 g9 E. O" c
</property>
, }; N' s& Q1 S8 n+ {( U1 ] <property>
# w5 Z4 t8 T, m! X <description>The minimum space that must be available on a disk for
( ?/ A- s- r. w) _7 ~ it to be used. This applies to yarn.nodemanager.local-dirs and& z/ U( e3 E% f" N. l0 e8 b2 B3 y
yarn.nodemanager.log-dirs.</description># A4 u. V1 z) n2 t. @: ] @( y2 x
<name>yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb</name>
5 Z. s f8 D7 a <value>0</value>$ B& d2 t8 d0 g8 @5 \, G
</property>, n" f- L. ^+ ^3 @) X
<property>
2 X6 F3 x ~+ A v <description>The path to the Linux container executor.</description>
5 ~+ J4 D4 R- T# y; s, m6 X& ] <name>yarn.nodemanager.linux-container-executor.path</name>
7 Y/ @" S/ U0 s/ Y </property>
5 j8 p8 X: O7 i; H2 A <property>1 g% p- m7 P2 N9 C& J& g1 b
<description>The class which should help the LCE handle resources.</description>6 R+ n" z, W1 K
<name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>; p) W0 N) _; D$ ?
<value>org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler</value>
" c4 e) ^% c4 x" Q/ e q# {1 ? <!-- <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value> -->/ H3 {/ J& i0 f9 _7 V( e5 t3 y
</property>
! j0 T! C' K: j- ^# {2 E6 k, e: Z <property>* L6 c/ ~; f0 }6 {* i
<description>The cgroups hierarchy under which to place YARN proccesses (cannot contain commas).
# k* M* c0 D" B s2 | If yarn.nodemanager.linux-container-executor.cgroups.mount is false# ~6 F* j; Y/ c7 I
(that is, if cgroups have been pre-configured) and the YARN user has write3 _/ {( O7 a! l6 A; v
access to the parent directory, then the directory will be created.
; m2 K3 O6 b6 S* e If the directory already exists, the administrator has to give YARN: m3 c. c" ^: g- n( ]( r
write permissions to it recursively.
4 J' A9 @" ^" Q) b This property only applies when the LCE resources handler is set to' D) N# g$ u2 w' @3 Y1 M; x |
CgroupsLCEResourcesHandler.</description>
$ u a! X- [ u7 m8 w <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
% i) ?% w0 C/ r, v! f4 i <value>/hadoop-yarn</value>
: V; z% N& z, O* O9 y8 \( s </property>
P/ V3 ]' Z d' l4 v0 W% |3 _& E <property>
4 _( f. s) i$ N; X. @4 z# H3 y <description>Whether the LCE should attempt to mount cgroups if not found.
% B; ^9 h) T1 s This property only applies when the LCE resources handler is set to
' y( l" e- h% B9 y CgroupsLCEResourcesHandler.( l2 v L { w, M- [' ~
</description>
, B! @- _$ z( l7 Y <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>7 |: G0 E* a& l
<value>false</value>
, e+ `( P/ M- ^- Z7 k </property>
7 z8 e ]' Z5 g* ^' G <property>4 [$ a) ~0 O* V
<description>This property sets the path from which YARN will read the
0 G6 ^4 e7 ^* e/ N, M/ O CGroups configuration. YARN has built-in functionality to discover the: S) |; G# j5 c) |' C. x( B: R
system CGroup mount paths, so use this property only if YARN's automatic
5 c" w# q+ p8 e% [; U6 m+ @ mount path discovery does not work.9 _3 ?# U& K8 p# ^
The path specified by this property must exist before the NodeManager is+ o, l' e2 n, \: a2 y" T# O6 F5 q9 T: [0 _
launched.' y( m' L( t% x+ i
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true,
& d: Z0 S- {2 t3 p5 U YARN will first try to mount the CGroups at the specified path before
6 S; v% ~' T$ k. k ] reading them." G% a! c8 B7 S& d# I$ I) j
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to9 h7 A$ L) {2 W" g
false, YARN will read the CGroups at the specified path.0 l+ j7 X" J: K) E7 o* |
If this property is empty, YARN tries to detect the CGroups location.
8 t$ q$ E* k4 G; e4 s3 l8 s Please refer to NodeManagerCgroups.html in the documentation for further" a$ s: `; i0 g# z
details.
: L; t0 ]8 O8 t8 ], P This property only applies when the LCE resources handler is set to
2 c, J1 S8 e/ m; c$ E CgroupsLCEResourcesHandler.
8 x1 R& i0 h8 Y% Q) c" w </description>( w9 m2 A9 ^! v+ {2 b$ k* [- ^
<name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
6 p/ r( D# h& o% ]7 R3 K </property>1 Y4 m5 {1 F9 t
<property>* j( E& o" _) F9 X# W8 S/ p$ G: a
<description>Delay in ms between attempts to remove linux cgroup</description>
8 P/ X8 q' u$ G+ O, X <name>yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms</name>7 J* ?) J7 w, X3 K' Q
<value>20</value>
m7 K: H+ @* A2 N1 d </property>. L, C" R) y* r% Q7 h; V- L' ^
<property>
8 s& E2 g4 j9 B6 ^- q% s <description>This determines which of the two modes that LCE should use on7 x* O+ V/ q q7 m8 @
a non-secure cluster. If this value is set to true, then all containers* d/ C& R+ Q7 r4 [0 c
will be launched as the user specified in
* J* a. Z5 m b yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If
6 }* U+ l9 c) e9 }* h; V this value is set to false, then containers will run as the user who
& K) T6 m% v' V! m! O submitted the application.</description>& d1 B' ]# g, u" D
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
( F1 W/ m% {' X <value>true</value>
6 }+ o9 x5 q i2 Y: s2 v </property>7 ~5 O* w$ r! Z# t% B0 _5 Q7 o
<property>3 n- D- M% W* Z
<description>The UNIX user that containers will run as when1 L7 d# Z$ z7 g' A/ t% m
Linux-container-executor is used in nonsecure mode (a use case for this
/ w2 E* M$ G# w% j; i is using cgroups) if the3 \0 u. h' A. L: Z: F
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is3 `) V7 t+ E5 F+ m: c
set to true.</description>9 o5 u t& R! P, Y1 `5 r( P B
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name>
+ P4 o& O% y" q0 O* k1 d <value>nobody</value># F4 N# G' ?7 z2 [/ P4 n
</property>
' ^6 r9 P& b+ `% @ <property>
+ Z; |8 b0 S* F& p4 z <description>The allowed pattern for UNIX user names enforced by
; K2 L! ~/ u( ? Linux-container-executor when used in nonsecure mode (use case for this
5 [+ d {: e1 p* E is using cgroups). The default value is taken from /usr/sbin/adduser</description>0 }# ]3 y; T$ n' T2 J
<name>yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern</name>
0 H7 D! {5 t! y$ T1 ?4 e$ c2 I: @ <value>^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$</value>
* r7 q1 R7 X/ F </property>
( \5 M$ O' W" S! U <property>% y. g* [, s' u: @2 ]" [
<description>This flag determines whether apps should run with strict resource limits7 `' u+ x1 A1 X" a8 W0 X
or be allowed to consume spare resources if they need them. For example, turning the
/ p. h3 j( } q# M! x. J flag on will restrict apps to use only their share of CPU, even if the node has spare
8 y. {% z: ?; W' T1 N2 E, }7 G CPU cycles. The default value is false i.e. use available resources. Please note that! y, ~% Q9 ~) J# z5 l
turning this flag on may reduce job throughput on the cluster. This setting does( E% {3 o0 b2 R; N# K9 ^" t% J3 L3 x
not apply to other subsystems like memory.</description>
/ M9 c O! }/ B }& l9 L <name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>
1 I- y7 C s* \" J9 y& p <value>false</value>
) t6 V1 z! r/ i </property>5 ^: ?9 X' }3 A1 P+ p8 T0 _/ d
<property>' Q. A! x' `; z9 X5 ]* O
<description>Comma separated list of runtimes that are allowed when using+ w% k0 }7 x9 s+ a- h( b
LinuxContainerExecutor. The allowed values are default, docker, and; j! J0 h) c9 r6 W" ?
javasandbox.</description>; ^6 r% e3 A* z2 R* c
<name>yarn.nodemanager.runtime.linux.allowed-runtimes</name># I/ r6 [$ Q& {+ } C
<value>default</value>8 C; X0 ?8 q( u/ D& B2 h: g0 I
</property>8 j/ Q5 F6 T# y+ Y1 m
<property>( q9 F, h& m& n
<description>This configuration setting determines the capabilities# z( z. p/ E1 p2 N6 G
assigned to docker containers when they are launched. While these may not
4 `9 L' F6 Z) ^9 |1 s be case-sensitive from a docker perspective, it is best to keep these
2 V) q! [3 F& C4 t uppercase. To run without any capabilites, set this value to! U8 X( f# y! @& b; C
"none" or "NONE"</description>
. V2 j! I# X5 J0 a" r& C$ U <name>yarn.nodemanager.runtime.linux.docker.capabilities</name>
# K8 h' D0 P8 G4 p- U <value>CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE</value>) }9 L" [' I& u3 f! i) N9 d U
</property>
/ a. O; ?: m0 m8 k4 t2 O3 P# [ <property>" g7 m8 v9 T6 C) p8 ^
<description>This configuration setting determines if
$ ?, _2 W) P! z- t1 E privileged docker containers are allowed on this cluster.
$ `7 d1 s5 d( n5 }0 Y/ } Use with extreme care.</description>9 h2 `# Y7 C' D, F/ {' J
<name>yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed</name>
/ q R/ B2 ~- _' s <value>false</value>
/ F1 E% c8 o% u </property>4 c% K+ q7 m7 Y5 m2 b8 j" U9 m
<property>% C! l" A7 J5 @8 G2 U! q. {
<description>This configuration setting determines who is allowed to run
* ]/ k' P4 ?, b0 ^* }8 I1 ^ privileged docker containers on this cluster. Use with extreme care.
. v+ G. n& u B* H </description>
. L9 A( c: R; |1 f% R <name>yarn.nodemanager.runtime.linux.docker.privileged-containers.acl</name>! r: p m, V: W9 U3 u7 h+ ]! G
<value></value>& S5 O8 ?* W7 R5 w, U! |# Z7 ?9 G
</property> m+ K ?3 V, p. e6 S
<property>& C/ J) A* R* u. [$ s. l9 _
<description>The set of networks allowed when launching containers using the Q4 P" }* R$ ~
DockerContainerRuntime.</description>. V; W$ D4 |8 E0 o" N+ O6 v1 e- N
<name>yarn.nodemanager.runtime.linux.docker.allowed-container-networks</name>
/ M: [3 o6 P# F$ g3 F# m" Q <value>host,none,bridge</value>
* Q! V3 `# E) Q& Z1 C+ t* ` </property>
% E; o! m% K7 {. h2 P& I! w' U <property>
1 Q( t Y) r2 R9 q' f8 I% D <description>The network used when launching containers using the
2 o4 Q) P H' v; E% d) V$ K. c DockerContainerRuntime when no network is specified in the request8 b9 u; y% W9 o( J- s& H0 S* p
. This network must be one of the (configurable) set of allowed container6 q. e8 ^0 L p) n" `
networks.</description>7 }" {; q! }" @
<name>yarn.nodemanager.runtime.linux.docker.default-container-network</name>% V- ?) L3 U2 ?
<value>host</value>
3 F* L4 c+ c$ A. B </property>
4 U; N( m6 h4 w1 C2 i <property>
" w. R, w4 q, t& \5 h! q! K, \ <description>This configuration setting determines whether the host's PID
' X5 B9 _. [/ y namespace is allowed for docker containers on this cluster., N1 J+ L' T% R: A/ V1 p9 \( y' _
Use with care.</description>
$ K6 n4 R, |$ E& }+ G <name>yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed</name>! k+ E+ Y/ H5 H9 O, `3 W2 T5 Q
<value>false</value>- }# d& i2 M% |3 ~" m! Z) y1 O
</property> Z- n( c2 c, I% I4 L
<property>
5 l3 Q8 j- K/ ~6 P: r3 r <description>Property to enable docker user remapping</description>; K! `, b" g% Z
<name>yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed</name>' Z" _) F$ ?. ^
<value>true</value>4 a! [0 `1 h9 S
</property>
" F1 [/ d, u" N" }& P, B <property>
8 E3 h' ~1 e) T1 ~9 s <description>lower limit for acceptable uids of user remapped user</description>
) j5 @6 a* \6 r: i/ ]3 ^ <name>yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold</name>( [ Y6 j/ W1 |2 g
<value>1</value>. @7 n* x& G; j4 A. Y, ?) B% P4 m
</property>
" b6 T$ V* [8 q9 ], T! i" _" n- h <property>* b6 `4 o; i' G( u, f: S4 z
<description>lower limit for acceptable gids of user remapped user</description>
+ q# h6 I, |2 i5 V# Z$ r; d# }1 n <name>yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold</name> m# f0 V0 D* q
<value>1</value>
9 L. f1 k2 D1 `# P: `4 ^ </property>1 Q* K/ u* ]. a2 c2 s% A' o
<property>
" o, F- `$ o3 a( w9 b3 p <description>Whether or not users are allowed to request that Docker% `5 C+ J: u4 t
containers honor the debug deletion delay. This is useful for5 J. `6 r4 U% z0 _ `7 W+ y0 \
troubleshooting Docker container related launch failures.</description>% t( q: c$ H( s! l
<name>yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed</name>
% n, Y" ]0 B7 [& } <value>false</value>
! R( `/ H) Q& N4 p6 L( V/ i </property>
, H5 z% I% w+ N3 G) }- Q <property>: {; c, n$ x E1 N
<description>The default list of read-only mounts to be bind-mounted
: ?# q7 X% @ p/ i) }5 P8 m into all Docker containers that use DockerContainerRuntime.</description>
; v) I6 H4 |+ f1 z# N. o, m) V! ?4 g <name>yarn.nodemanager.runtime.linux.docker.default-ro-mounts</name>
9 d9 K& Q% I' P <value></value>
) @9 c# ~$ C' E. k9 \4 v2 N7 G# P, K </property>
" A' A. x3 F `. V$ ^ <property>6 z; G$ p1 e6 p5 K, U
<description>The default list of read-write mounts to be bind-mounted
. u; G9 m6 {5 V& d7 H0 t5 A into all Docker containers that use DockerContainerRuntime.</description>
, w6 N# L6 X/ \5 E3 t& Q <name>yarn.nodemanager.runtime.linux.docker.default-rw-mounts</name># E; E3 d4 i- D8 U0 H
<value></value>
, L5 m: g7 j9 l. r" B </property>2 ?1 l+ l" a# V% S U
<property>
8 t# C+ C- s/ S$ d1 K( D3 b9 l <description>The mode in which the Java Container Sandbox should run detailed by7 x2 Y: u6 e- w% N' g) A
the JavaSandboxLinuxContainerRuntime.</description>9 q/ I1 g% Z# B
<name>yarn.nodemanager.runtime.linux.sandbox-mode</name>* t/ S( P4 l+ a# V9 P( i0 D9 C
<value>disabled</value>
7 s8 g( N( O) T: \# t </property>. O* ], U. J: M( d
<property>. v( U" }1 U- K( k
<description>Permissions for application local directories.</description>
6 c+ i/ \ }3 x+ T L" t1 s. B <name>yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions</name>
7 x$ M& c4 K- Z7 W1 l6 S <value>read</value>
7 X) Y9 q( ]6 s+ ~" n </property>
) V9 z9 W) X5 @1 V+ k% [ <property>
0 `# Q$ B# p6 l" W+ x( o <description>Location for non-default java policy file.</description>
# |) J/ r, \* | <name>yarn.nodemanager.runtime.linux.sandbox-mode.policy</name>
) J7 T9 p- s+ \! s4 } <value></value>! I2 }! k( a) G7 {& }0 @0 i
</property>+ i: o7 l% W. {0 y$ i
<property>
% P/ E- j0 h! g$ C! {9 m$ l6 f <description>The group which will run by default without the java security# X+ F' @" ~) b$ ]' l1 S# M8 s
manager.</description>9 D5 s8 H! {. G* Y, K6 e
<name>yarn.nodemanager.runtime.linux.sandbox-mode.whitelist-group</name>/ W0 Q8 }6 u+ v! G2 |
<value></value>. {( u/ }( c! L- l S* Q% q
</property>
) s V8 W Q! B- N <property>
. a' {0 v9 B0 i& O% G* c <description>This flag determines whether memory limit will be set for the Windows Job
) {6 x' m; B( p1 ?( S Object of the containers launched by the default container executor.</description>% ~3 [$ `! }6 o! P! h/ J1 M1 e6 |
<name>yarn.nodemanager.windows-container.memory-limit.enabled</name>
! R2 Y$ v' R6 }. c <value>false</value>9 T* y0 a- M+ T- i6 p; l% K A8 v
</property>
5 K) b2 O* x( l' y% } <property>
9 H) o# a) V7 N# p0 }( A) { k <description>This flag determines whether CPU limit will be set for the Windows Job
! k2 p- Q/ }$ M* Z Object of the containers launched by the default container executor.</description>1 Z+ r' O3 c) a9 v$ `) Z4 u
<name>yarn.nodemanager.windows-container.cpu-limit.enabled</name>
9 M' e! d% m' D* h5 x' a5 F <value>false</value>
1 b6 y7 _) ]6 J$ I </property>
$ F- u' \ g; Y' f8 o! _# a. _9 ] <property>% y" \: U& H- S" N. g
<description>
6 P0 J) w* p, c0 }; {! M, i& \- j Interval of time the linux container executor should try cleaning up
! g' F. {% B1 c6 U) d) n$ h' z cgroups entry when cleaning up a container.4 w. s# `# U% h% a( R
</description>
1 r# `$ p$ A/ s, R! e+ C# ~ <name>yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms</name>1 j# G$ ?. ?5 i4 _8 O& }3 M
<value>1000</value>. d/ i, x& Q K
</property>4 M4 T/ ^1 [, R6 l' u# W
<property>
& H7 w+ V8 }" s. l <description>& ?9 k. z6 t/ g. T! ?
The UNIX group that the linux-container-executor should run as.1 M1 _/ o& _! i% s0 {
</description>* _1 A! Q9 a+ L# T, T0 Q
<name>yarn.nodemanager.linux-container-executor.group</name>
6 [1 C% u% G* S6 t0 q/ u: Y <value></value>
( f1 L: W) y* @ </property>8 Y& F4 D( H6 f2 s
<property>
; `# v5 M/ O; b4 l* ^. R) p0 ^ <description>T-file compression types used to compress aggregated logs.</description>
+ O4 N6 N4 E+ y4 ~! \ <name>yarn.nodemanager.log-aggregation.compression-type</name>
! a) E5 O0 k7 l" T7 j; L& \* h <value>none</value>
/ J2 e/ X- K: u: h$ ^% ` </property>9 V6 H/ C! f0 `9 L3 \7 k+ n! @
<property>
8 \$ V$ o+ e2 f K4 L4 d: u <description>The kerberos principal for the node manager.</description>( F6 \7 Q/ H+ ~; H
<name>yarn.nodemanager.principal</name>
8 I8 M: p: x$ X2 Z; i <value></value>, s4 V" g9 x% b. z- Y* j, f6 r
</property>
2 Y% P* U l; @' } <property>
+ m( K* S( o5 p+ F( k( p <description>A comma separated list of services where service name should only
- Z. n1 J. S& q contain a-zA-Z0-9_ and can not start with numbers</description>
B0 \" `) m( \: V <name>yarn.nodemanager.aux-services</name>
. H* R/ g2 i8 A+ ~ <value></value>' b6 W; G# ~, q- `# i) ?+ r h4 w+ @
<!--<value>mapreduce_shuffle</value>-->9 ~- ^1 v5 {9 h) ]: l4 U) v
</property>
- r2 f0 q X& f2 X" x <property>
9 A) O5 {4 z$ _( j* z" x <description>No. of ms to wait between sending a SIGTERM and SIGKILL to a container</description>3 L- R; q2 [" [, D: z) X7 z! P
<name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>$ ?* Q8 ~' Z$ O' w: a
<value>250</value>
8 s# k$ k, Z# p2 X7 C7 F </property>' X) v1 z- I1 W' h3 {
<property>! d1 [7 z j9 ^5 y
<description>Max time to wait for a process to come up when trying to cleanup a container</description>$ g& u" z% o1 \5 \, u
<name>yarn.nodemanager.process-kill-wait.ms</name>5 Y! m' \% I; T% J5 z( x2 ~7 W0 [
<value>5000</value>
+ K6 B+ w3 }6 {0 c; V </property>' d- E3 C4 {0 H
<property>8 }8 V1 S- \( E- A) ?7 ]
<description>The minimum allowed version of a resourcemanager that a nodemanager will connect to.
' m' m7 ?% H; }6 C, h! | The valid values are NONE (no version checking), EqualToNM (the resourcemanager's version is
. v/ s8 E+ a; t9 p( t M, u& M+ n; N& a equal to or greater than the NM version), or a Version String.</description>, X$ r' O1 `( i6 b% \
<name>yarn.nodemanager.resourcemanager.minimum.version</name>
5 L. q. M5 w! ]) P" b <value>NONE</value>
t# t! p8 |" v6 E' j+ L </property>' B5 e' k- F+ B! ^3 U: K' \
<property>
2 f1 ^& w+ a6 @: e <description>Maximum size of contain's diagnostics to keep for relaunching- N* `" Y% L- u. A/ I! `
container case.</description>' ?( V9 D7 a* Q$ E; A0 O
<name>yarn.nodemanager.container-diagnostics-maximum-size</name>* Y! l: {2 ?' t: M1 D/ R$ ^0 }
<value>10000</value>
0 V/ Q; i* e. A! ]5 v </property>* b0 ^* P( @ P, r) m4 H* k
<property>! D* R7 V' c0 t8 s* }% f
<description>Minimum container restart interval in milliseconds.</description>
4 X8 f; m; w! D! }9 i* `3 V <name>yarn.nodemanager.container-retry-minimum-interval-ms</name>
6 T/ D( f5 K. g* l <value>1000</value>
x+ H2 c! v7 {$ [) r6 |% D, b6 O" R </property>
$ B! I9 s$ F6 u <property>
* E/ u6 v9 m8 R7 J& C9 ~ <description>Max number of threads in NMClientAsync to process container
' _8 t3 ]' L% B# W management events</description>' ?' h5 n( D! w( c; h. v
<name>yarn.client.nodemanager-client-async.thread-pool-max-size</name>! O* L v0 c1 `+ ^
<value>500</value>3 v/ `7 J6 Z. ^( o* Y
</property>$ r7 ~% v- g9 s# d2 [2 R3 V0 u' H1 ]' ^; ]
<property>3 H7 a4 t2 `3 S/ |* f8 @
<description>Max time to wait to establish a connection to NM</description>% ]/ K* ]- C$ \' ^
<name>yarn.client.nodemanager-connect.max-wait-ms</name>
0 M ]- ~4 o. f5 f <value>180000</value>
, R7 T- F* Q' s7 X; v </property>
( j' P' ]! }( Y, s* s& ~ <property>
6 ]. e" r) e; _/ n1 E <description>Time interval between each attempt to connect to NM</description>2 o" D9 c$ U5 h% f3 ]
<name>yarn.client.nodemanager-connect.retry-interval-ms</name>7 k( W7 A( E: ]6 D4 H. e
<value>10000</value>0 r! v4 C7 Y. T+ R; v7 ^( I( m
</property>
4 j+ m, o# [* s! ]( H <property>
! V) V h( [ D- D3 @/ j4 f/ S <description>
# r) ?+ N; A6 Y% T+ D Max time to wait for NM to connect to RM.
- y% [( Y; L( O6 |% B: I0 K0 v# S When not set, proxy will fall back to use value of
7 g% P4 S1 Q4 \1 p yarn.resourcemanager.connect.max-wait.ms., [8 i' V- r+ f: Z# ^9 `
</description>: P+ `9 g2 w( C0 ]/ T
<name>yarn.nodemanager.resourcemanager.connect.max-wait.ms</name>
5 t- S Z4 [+ t2 A0 u. x <value></value>
- ^8 |: _& `4 M1 v </property>
1 f8 D2 M( D$ u: f5 I <property>2 i- c4 H* |! u- F; K1 g
<description>
% @( C- u& k, o) V2 J Time interval between each NM attempt to connect to RM.& a# i- l/ J. m6 B, B K. A
When not set, proxy will fall back to use value of
/ O2 _' U; p4 z7 Z! X& H yarn.resourcemanager.connect.retry-interval.ms.+ d2 m: J. ^1 `& y; A' x' f
</description>: A& P/ l3 R' t7 t: B
<name>yarn.nodemanager.resourcemanager.connect.retry-interval.ms</name>* |8 l% r/ ^# J( C& P3 B
<value></value>3 e0 y* q+ P0 b- q
</property>
' U( p" Y& G1 S5 Z$ ?' P! i <property>
' Q4 z& \' K' b- S <description>
' i2 [8 o# J( s5 i. M8 {# P- Q; W# } Maximum number of proxy connections to cache for node managers. If set
' A1 C$ l2 {4 T% i- W3 d- H to a value greater than zero then the cache is enabled and the NMClient5 f% c5 o U( Q" E+ A
and MRAppMaster will cache the specified number of node manager proxies.+ V/ ` l8 N8 _3 D1 Z1 w
There will be at max one proxy per node manager. Ex. configuring it to a/ a; n$ h. O+ c4 g$ Y
value of 5 will make sure that client will at max have 5 proxies cached: {: G% w$ q3 s. a/ i$ a
with 5 different node managers. These connections for these proxies will
" l( ~: w! ` n0 R" l7 N9 q7 D be timed out if idle for more than the system wide idle timeout period.2 j. D8 J/ ?3 F" a# _7 x
Note that this could cause issues on large clusters as many connections" g+ y' ^, y4 T8 u, s$ F
could linger simultaneously and lead to a large number of connection4 B+ _$ M; x: \
threads. The token used for authentication will be used only at
# f4 T4 F# v9 Q9 G4 |8 v- x connection creation time. If a new token is received then the earlier& U, _1 j' w! U7 |; n4 e! o9 `4 E
connection should be closed in order to use the new token. This and6 g1 s; y/ K R
(yarn.client.nodemanager-client-async.thread-pool-max-size) are related
; y# \# n3 t6 v% e) ~% o$ \( ^6 w and should be in sync (no need for them to be equal).; v( ^' u; d/ j" D- ^" ^, V) p
If the value of this property is zero then the connection cache is
$ {1 @: y4 X0 o0 m/ x4 U disabled and connections will use a zero idle timeout to prevent too$ X- t3 D. V# ]1 V) T: t& e
many connection threads on large clusters.5 j) M4 O6 ?) ?
</description>! O1 V$ e0 r) J
<name>yarn.client.max-cached-nodemanagers-proxies</name>' T! K+ t( r& X5 d+ B p& K
<value>0</value>
8 v+ c1 `1 U6 t7 s </property>. a% f' k( ^! L5 T+ b' [
<property>
3 A, L5 a1 ^; h1 P0 M6 k) `+ M <description>Enable the node manager to recover after starting</description>
6 C1 l1 _3 V5 g( z- U <name>yarn.nodemanager.recovery.enabled</name>
! v ]' }# x; @ U; y <value>false</value>
2 P8 }9 G k4 E/ c </property>
* a% N: @/ p& X; `; ^* X3 c% v <property>( _6 Z$ _: Z$ n, e$ `8 O8 J
<description>The local filesystem directory in which the node manager will
/ ~/ m B: d/ i( ` store state when recovery is enabled.</description>
2 R0 G) G1 i; Q# i) ]! T <name>yarn.nodemanager.recovery.dir</name>
. j( ^. w v( a4 p <value>${hadoop.tmp.dir}/yarn-nm-recovery</value>+ p; U/ ]! I1 q2 z# A8 k0 K% T
</property>9 ^/ T/ ^3 k p- G9 U( K, Y( d
<property>0 T5 B S2 C& n+ R, ]$ e
<description>The time in seconds between full compactions of the NM state1 Q5 `, ^7 M3 V- s4 U9 z b. o1 A0 t
database. Setting the interval to zero disables the full compaction2 b, y9 f" y( \1 ^0 p& d( k
cycles.</description>
# @$ w: m7 ~# S& M <name>yarn.nodemanager.recovery.compaction-interval-secs</name>6 R3 E. w0 t! Y6 U# k& f$ d0 ~
<value>3600</value>; w: Z4 [: Y- o2 R7 q9 M
</property>
" ?9 ^7 x+ G. R' }8 P/ M8 {6 X, e% N <property>
# D( c7 c/ A. T# p <description>Whether the nodemanager is running under supervision. A$ C7 ?6 F, g( |) w% Z U
nodemanager that supports recovery and is running under supervision0 b0 }3 |6 Z& o9 @ f
will not try to cleanup containers as it exits with the assumption
* R4 o. l, Q9 o+ Q1 P. }7 j it will be immediately be restarted and recover containers.</description>( a R( f" \# H
<name>yarn.nodemanager.recovery.supervised</name>$ L2 h5 D$ Q- I M. |- d5 k
<value>false</value>
9 P$ G: T2 _7 y) ~3 r </property>" A; B, e! `) E1 A5 J
<!--Docker configuration-->* ?( }1 y: o. y7 ?5 V+ ^
<property>+ B- E, X. L) A* s, Y
<description>
' f6 y- Z% K- o/ S0 O Adjustment to the container OS scheduling priority. In Linux, passed4 w- J3 A& j2 {- \5 @! T& G, \. J2 ~; B
directly to the nice command. If unspecified then containers are launched. ?7 `0 S7 ~1 R. i+ p6 t
without any explicit OS priority.
$ l8 E3 F; U% Y. p </description>
2 r' q+ a0 M! l# F- E9 [ <name>yarn.nodemanager.container-executor.os.sched.priority.adjustment</name>) L0 h6 d/ g) ?) b: v# q5 C
</property>
( E( O7 }( s/ z( I; L <property>
1 Y8 f# j* n4 T" ^3 i <description>
8 b# w/ { E9 W3 c; s% t7 A' n Flag to enable container metrics
1 A$ ^ U4 j3 Y4 ~! c: O </description>: _. i9 C e( o
<name>yarn.nodemanager.container-metrics.enable</name>/ n9 |" C" L- D2 [7 ^5 z! y
<value>true</value>
/ X, C5 f- N2 L ]- g </property>' U' h1 v2 f, Q$ y2 k! a4 G
<property>
# J. r: d4 h& r. x* e <description>' n% ?( P: Z0 }5 |, s
Container metrics flush period in ms. Set to -1 for flush on completion.
9 o+ j4 H9 b: d% U: J4 k </description>
4 v; k( V5 {7 Z* H: {" v <name>yarn.nodemanager.container-metrics.period-ms</name>
2 U! V8 A @9 M9 C. i <value>-1</value>
]6 S; I8 e; X </property>
+ q1 q$ s1 o: T. v8 e <property>/ `8 R. k X5 d7 b7 b7 P- u
<description>) z6 W6 A( j5 b- {8 ]- g9 }
The delay time ms to unregister container metrics after completion.
7 ]" K1 q! L F1 Q </description>6 l) {5 t' P3 O/ e: v! x/ r
<name>yarn.nodemanager.container-metrics.unregister-delay-ms</name>
2 ?& V& Q! v4 P& X <value>10000</value>& Q, D( b; o; e0 j) s1 X
</property>
2 P! g, f2 I+ [& A7 f3 I2 @: g <property>6 h. g2 h+ f2 s0 Y$ o
<description>
3 x! G3 c' a+ ^# K" i* M Class used to calculate current container resource utilization.4 s+ V( @; r0 _0 G" s% J I# h( k8 ~* ~
</description>2 d4 `' F# }. k2 a
<name>yarn.nodemanager.container-monitor.process-tree.class</name>1 @/ r! J* e" M7 G1 d# C: h& t; `
<value></value>
" q5 E$ N( h) U' w </property>3 U9 Y/ `6 `/ l9 l
<property>
: y, a+ A8 \1 O" k9 O <description>! o, N5 x/ m; }( q8 h: I) j
Flag to enable NodeManager disk health checker; t/ w8 d/ Y! ?, |) o/ w0 K
</description>
. @( _2 v# h: k4 O+ Z <name>yarn.nodemanager.disk-health-checker.enable</name>3 a% [3 Q6 B: Z/ `2 ]
<value>true</value>
], Q G8 q+ n) M' u </property>4 U# v/ s9 c$ H. l0 r2 p
<property>+ C1 P% Y& a. {1 \8 {1 X1 ~+ b
<description>
3 C* u8 T; |. z5 b& Q Number of threads to use in NM log cleanup. Used when log aggregation) E5 l$ F5 V. ?8 G
is disabled.
" Z- O: T8 Z6 ?3 e. Y </description>9 h/ u; e' y/ |, r% u0 K4 i! S
<name>yarn.nodemanager.log.deletion-threads-count</name>" I* y2 l$ Y; X, k
<value>4</value>
: y( k1 I3 j; ?7 Y D$ r </property>
, A/ m+ l+ A. a# V# Z" U <property>
6 }6 }/ L* q8 z. j) D <description>8 K# c2 _* s# i2 L+ ?% D
The Windows group that the windows-container-executor should run as.& F6 [; G9 h, i% A; B8 @3 ]
</description>7 _+ l8 r i4 w$ P7 W
<name>yarn.nodemanager.windows-secure-container-executor.group</name>
1 `% G' q" X* K& D7 C <value></value>
- [) v) x& N6 ? q- q: {2 I </property># `8 [& n* _! L
<!-- Map Reduce Configuration -->2 d* s) v6 r3 n, h5 j( r/ o3 G8 Y- l) g
<property> h! B# i& A B/ Z9 O
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
! @" F' l% j# ^5 [9 J <value>org.apache.hadoop.mapred.ShuffleHandler</value>' e: S% I; R/ ^6 g
</property>
/ J4 i; m6 b5 T& Y# |7 a9 c: a" C <!-- WebAppProxy Configuration -->+ `# i# u _- N. j5 G. [9 D- x
<property>
w" Q5 E& V6 ~! N" s <description>The kerberos principal for the proxy, if the proxy is not0 f- G8 ~$ {# i2 J: T+ h( P
running as part of the RM.</description>
: _. d4 Q) a, ]3 E <name>yarn.web-proxy.principal</name>
' {; ?% ~: ~8 H5 O! I. {2 x <value/>$ v" k; b( i+ }5 U M- {+ K
</property>
" r# x! x: N1 v" f' n* I1 w <property>: y1 b2 f8 q9 c$ V! u6 e
<description>Keytab for WebAppProxy, if the proxy is not running as part of
6 E$ F+ M$ z2 ]# J4 ^7 Q the RM.</description>
8 S1 s" X# y* x) y7 v <name>yarn.web-proxy.keytab</name>
5 p& l7 f- U; o </property>1 q, }; A4 X% h7 q( x/ ]
<property>& Z' R# |* @/ m/ L/ S" k, r+ S
<description>The address for the web proxy as HOST:PORT, if this is not
/ b' l9 I M M; A" z- e' j given then the proxy will run as part of the RM</description>
' U7 y" X/ R0 p h8 w! k8 I& E <name>yarn.web-proxy.address</name>
+ U6 a7 U$ V/ i <value/>, n/ Z, t& v( k, g u
</property>
. s( s/ ^; u: p9 t" o <!-- Applications' Configuration -->1 C; ~0 k* m; B
<property>* h( y' F4 e/ ]2 U1 h
<description># u @ D. s: @* z
CLASSPATH for YARN applications. A comma-separated list
1 w/ h# ?2 J' _! \, k of CLASSPATH entries. When this value is empty, the following default0 B( v9 J$ m9 k5 k
CLASSPATH for YARN applications would be used. 2 o# q' @% ^% s) E" x5 Z5 r
For Linux:5 Z9 @7 `2 k. N, t# p3 K
$HADOOP_CONF_DIR,
9 A0 }- `* w+ X% [ $HADOOP_COMMON_HOME/share/hadoop/common/*,
% ~8 W8 ?( B8 ^0 q $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
& Y- O1 l) z2 \" p+ ?3 p $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,5 w, G: | _5 O. l9 x: Q" m# z+ {) [
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$ b# j# ?" t& M4 I- I
$HADOOP_YARN_HOME/share/hadoop/yarn/*,$ y$ \/ N, {2 a3 T6 B1 I0 I; i
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
! C. r: O; U2 R% z, e For Windows:" d1 G. u5 G' ~0 ^9 H7 ~ P
%HADOOP_CONF_DIR%,- M5 |: D W; m8 b
%HADOOP_COMMON_HOME%/share/hadoop/common/*,
8 d4 J/ X' i! K %HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,
' g" x* u) J" K5 @3 @0 R %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,5 t$ W/ u4 i: ~% T
%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,) f; W H e' }, x' [: r
%HADOOP_YARN_HOME%/share/hadoop/yarn/*,
# O1 k7 d4 t6 f4 B( [3 d7 m %HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*3 E/ {# V2 [/ b4 v, Y
</description>6 u1 V$ C' {* e/ g
<name>yarn.application.classpath</name>+ b6 j" A- W( M: Z& @
<value></value>
- R+ v- Z5 s/ U+ S8 x </property>' D. D; Q! S+ z( A( k) I
<!-- Timeline Service Configuration -->
2 a0 S a l: W* E' F1 @7 l <property>6 h- J5 e: t: R3 k A
<description>Indicate what is the current version of the running) j6 M `' S2 W
timeline service. For example, if "yarn.timeline-service.version" is 1.5,
- M2 e% _; H7 H- K6 g' p and "yarn.timeline-service.enabled" is true, it means the cluster will and5 ^- T% s* w! L: N
should bring up the timeline service v.1.5 (and nothing else).
" M2 t" n, i* ?* m/ U" }1 I On the client side, if the client uses the same version of timeline service,
* V7 W) Q4 v* S# o; J) n% x( H it should succeed. If the client chooses to use a smaller version in spite of this,
! O* e- }) ]: v3 T$ P+ Y$ s then depending on how robust the compatibility story is between versions,
& U$ d, w' g" z( R8 g the results may vary./ B3 z4 w# y" f1 J+ y% o C) M8 G1 v4 c
</description>
* B W8 |( m2 i* l <name>yarn.timeline-service.version</name>
/ w. N$ X( Y3 {8 z7 ]7 m/ d5 J! o/ x <value>1.0f</value>
- c R3 F9 O3 z; d( { </property>" Q, d! u4 Z# ^
<property>& R, n4 H2 `. p% O7 u0 `* v/ [3 |7 m
<description>
! c! Q* t8 r8 P$ |! p4 D+ C- | In the server side it indicates whether timeline service is enabled or not.
+ z; J# w& i( ]! C And in the client side, users can enable it to indicate whether client wants
+ J; }. ^0 p- e6 j7 |) E to use timeline service. If it's enabled in the client side along with
# k6 @; Y8 r- S/ X( `9 f4 C: e security, then yarn client tries to fetch the delegation tokens for the
9 p" K/ s, R; h% d+ F2 j5 |: N0 Y timeline server.3 u" L! C: N& s2 u8 t5 b
</description>
: v- j1 `$ o t. F0 x; y$ ? <name>yarn.timeline-service.enabled</name>
! |, A6 R% ^1 K( D9 F& m5 p <value>false</value>
6 x2 n } b9 O2 D' z; M </property>
9 J# ]3 \( s1 \4 J/ V <property>
U4 \, ~# J- m <description>The hostname of the timeline service web application.</description> u, z9 @% A0 `) N3 K
<name>yarn.timeline-service.hostname</name>6 H: f. p* E5 {4 K0 `: }! B
<value>0.0.0.0</value>: {* c X& d6 T" d: a
</property>2 R% G. e! b% M1 u0 u
<property>
* q, N9 I* K0 O8 W- b <description>This is default address for the timeline server to start the
1 y) d9 T6 N8 x# m% g/ Y RPC server.</description>
& k) [/ w/ q! |7 M* x( ^% \ <name>yarn.timeline-service.address</name>
( S1 p, O) |2 B0 f1 p <value>${yarn.timeline-service.hostname}:10200</value>
0 C0 k, y" G% @; Q6 Z7 x </property>
. z% a1 G3 M1 p! a$ I- C2 g, f <property>
* m% \, g F0 `1 _) } <description>The http address of the timeline service web application.</description>4 J- q# K( E& P+ m: b
<name>yarn.timeline-service.webapp.address</name>
1 U8 [; e* Z* c6 w3 U& [ d <value>${yarn.timeline-service.hostname}:8188</value>
6 l% ?0 i# ~, x, G; @% | </property>+ h( }) w' b; i+ j Q' A9 S
<property>
+ r$ o+ f0 _0 ?1 `) ?3 O! _( ^ <description>The https address of the timeline service web application.</description>
+ R8 [4 e+ @2 p2 z9 ^% D- Y$ d; ^# h <name>yarn.timeline-service.webapp.https.address</name>3 l# I7 M1 f6 a4 h, ^- ?
<value>${yarn.timeline-service.hostname}:8190</value> u, F! }( x: X8 J2 P* R6 u
</property>
3 }$ X2 |1 k" ~2 M* I# ]- H0 F8 x <property>& T% N9 O M; F* Q* w
<description>0 w. \( S1 ^& C" x: N. S0 Q* L
The actual address the server will bind to. If this optional address is
( \+ | A( z, F7 W set, the RPC and webapp servers will bind to this address and the port specified in
s# ~1 N( p8 y% b( t# n yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively.. E! M" ~) {8 \7 |3 a! E
This is most useful for making the service listen to all interfaces by setting to; l% O, h1 [# l& }4 l2 k3 \" n r
0.0.0.0.: w% I! z7 u! Z5 S) `
</description>
- b7 X+ E4 u0 ^4 M <name>yarn.timeline-service.bind-host</name>3 F1 W% f' h- O3 N7 S# l, r8 V
<value></value>8 G9 v+ H9 C+ ]9 U
</property>
/ Z; z* f( S/ R# y <property>
. Q! j5 [% `4 g# _ <description>
1 i6 S- @8 [" Q$ v/ S' v Defines the max number of applications could be fetched using REST API or, }1 F& q5 g" ^0 j3 l% N' ` h1 b+ C3 ^3 h
application history protocol and shown in timeline server web ui.
/ S6 S: h! u0 ?4 w4 q' F </description>4 H _! b5 M0 D: k, @
<name>yarn.timeline-service.generic-application-history.max-applications</name>
% J8 G( I) i; t2 t <value>10000</value>7 [$ ~' M. @- K, \' h
</property>
) V) q+ \3 ?& r- l <property>' _% N" I0 L7 Y& q3 t* `& `
<description>Store class name for timeline store.</description>6 ]1 J; C) G# G% L) b9 A; x
<name>yarn.timeline-service.store-class</name>
& ^4 h0 k Z7 D) `5 H. X5 D <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
2 F$ D& z7 n( v </property>
1 u v2 X& g g; O0 y$ R# C7 ?* M8 | <property>
/ l/ W8 g1 h# y: T8 i1 w ~ <description>Enable age off of timeline store data.</description>0 Z0 i" H* `* b* o
<name>yarn.timeline-service.ttl-enable</name>
/ B1 v. e% ]. r+ m# c <value>true</value>
# d: T9 o4 \% ]4 Q </property>' E3 R2 |3 Y- }
<property>
0 F, }' l. Y: `" @; s! I' p6 [& [ <description>Time to live for timeline store data in milliseconds.</description>
0 m9 s# X1 N( f" ]# A* B <name>yarn.timeline-service.ttl-ms</name>
% \7 p9 k0 _& i3 g; e2 d <value>604800000</value>
# m5 V- T' _* S9 H' p/ R: p. ? </property>
* W) R% E- q& D1 j <property>
: z" U! p, ~ }' | H/ @" u <description>Store file name for leveldb timeline store.</description>$ L! q: H- ~; }* j/ W
<name>yarn.timeline-service.leveldb-timeline-store.path</name>
. Y( {6 I. r, H <value>${hadoop.tmp.dir}/yarn/timeline</value>3 P" W6 @5 t* g% C8 d1 A) E9 K& b
</property>9 `, u/ i; I* C; _/ t* v' F
<property>: b% U3 q6 f8 c
<description>Length of time to wait between deletion cycles of leveldb timeline store in milliseconds.</description>$ {' b' w# C- G
<name>yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms</name>
7 D; e. k$ h/ h: F: b9 l k <value>300000</value>
6 b7 c N8 a5 O2 {" b5 u: B </property>
% S, y8 S/ C5 [* v$ w/ [' B& R <property>& k& e( M$ M: N. [! ~
<description>Size of read cache for uncompressed blocks for leveldb timeline store in bytes.</description>
. I# K$ X3 ^8 m- Z6 v+ m <name>yarn.timeline-service.leveldb-timeline-store.read-cache-size</name>
, z W+ ^ l/ |7 q. ^7 V <value>104857600</value>( h. u3 ?& b2 F. w! o8 H3 t* f5 d
</property>
/ g0 H4 F* |# Q! B. Z- H% ?$ Z <property>
6 P' C' _; [3 Z* O. {6 H <description>Size of cache for recently read entity start times for leveldb timeline store in number of entities.</description>
v' ^& L& B; Z3 i) F/ ~1 \" E( q0 b9 S9 | <name>yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size</name> x! T! `* p$ D4 W
<value>10000</value>
$ R/ H2 X i6 C7 W7 d5 C* C2 ] </property>8 p* x* k; }$ B: b# Z4 e" ^
<property>
( T R) v! E2 g* f <description>Size of cache for recently written entity start times for leveldb timeline store in number of entities.</description>
( S4 `7 O/ P+ | <name>yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size</name>
) A# f& |# c9 [& a <value>10000</value>( `- z# k; A8 ?4 x# c+ b+ ]
</property>
5 h" r$ G2 D/ Y/ a <property>
- _4 m7 }% N2 z E <description>Handler thread count to serve the client RPC requests.</description>3 u! J2 i) [! |
<name>yarn.timeline-service.handler-thread-count</name>
1 g! |8 j l, n <value>10</value>( f) C0 [! M. k8 k5 `. d; ?' Q
</property>9 |+ P" e8 v/ {* X/ M( I* W
<property>
1 X5 M' q& d6 _9 F <name>yarn.timeline-service.http-authentication.type</name>
, `6 c! K: S E2 v& ]" \, G7 B, u2 X <value>simple</value>
: B9 t% o3 P. X: f& v* v <description>; J" i$ r1 U1 v( Z+ C/ p7 v
Defines authentication used for the timeline server HTTP endpoint.) ]) e2 n, ^& O
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
+ S2 U, d/ t0 h( b$ Y& e </description>
1 o# U4 n8 J7 x </property>8 [# l M2 z4 ]# P; \- `/ E
<property>
) |. v' x) \* N `; C <name>yarn.timeline-service.http-authentication.simple.anonymous.allowed</name>
" s& v" d; {2 Y4 C <value>true</value>
+ H( j/ ^- b9 |" b3 p, r5 f7 U8 E. S <description>
0 @8 r. i! Z, ~1 r/ f Indicates if anonymous requests are allowed by the timeline server when using3 r) Y4 y) X i$ y" d, M9 m& V0 Z2 O
'simple' authentication.) [5 H$ J8 }" y8 J3 Y
</description>
{1 C" a* Z7 P6 |3 T. ^% U </property>( ]2 G+ D1 C, ^
<property>
" x6 c4 [- w4 S2 \2 ^) | <description>The Kerberos principal for the timeline server.</description>
; ], [9 Y$ Q& _ <name>yarn.timeline-service.principal</name>6 U% h1 G* Q) i& i1 a
<value></value>
- u" T% a, z8 B9 ^ B8 _9 n </property>
& e5 c% V* P# |" b <property>3 f& c; Q' n+ f0 W; p' T5 ^- U/ _$ l' O
<description>The Kerberos keytab for the timeline server.</description>" }. t* T! o; q( |" `
<name>yarn.timeline-service.keytab</name>
9 [+ Z' b1 Y1 w# w1 L& T" ^ <value>/etc/krb5.keytab</value>8 s/ h* a" i/ F9 @
</property>' P+ d( ?. D: s+ V+ x
<property>1 I+ @! x9 ?2 R7 i# Q
<description>Comma separated list of UIs that will be hosted</description>
: y) j9 h, s9 f" v* `' i0 P <name>yarn.timeline-service.ui-names</name>
9 Z, } a' {: v <value></value>; `/ S1 }) R: {0 P
</property>3 A0 }# R1 Y) R6 W
<property># w3 | S3 D6 P0 R+ y
<description>
( J" L, y# E6 @# Z2 r7 m Default maximum number of retries for timeline service client. m( h8 d; V8 @3 d! \/ P
and value -1 means no limit. J9 ]; J; ^ t% d% c0 y
</description>; ?8 c3 g1 E- v4 I/ x, x5 w4 z
<name>yarn.timeline-service.client.max-retries</name>
, ]0 g& C' w( Z- Z1 ] <value>30</value>
6 z" e/ S( c0 C9 X5 @ </property>: `% G2 ?, o# N
<property>* z3 g6 b6 O3 o0 h
<description>Client policy for whether timeline operations are non-fatal.0 @$ |& l% I, T$ A
Should the failure to obtain a delegation token be considered an application
0 D7 o8 e0 M# n5 ?3 Z failure (option = false), or should the client attempt to continue to/ B ]* |3 L/ M1 J5 G
publish information without it (option=true)</description>
! N7 V& {: H: C/ `7 d& N, H <name>yarn.timeline-service.client.best-effort</name>2 R i K1 H/ Q, a5 V8 w
<value>false</value>9 m2 M2 m/ A$ }- i" Q( c, x8 \
</property>
' A/ |2 h" i9 C! e <property>
$ }% D& C3 K9 | <description>1 d3 p: g0 G% l3 n# W- ^
Default retry time interval for timeline servive client.
: U6 }$ Q* i: j/ F </description>/ f5 y: w" K) c- F
<name>yarn.timeline-service.client.retry-interval-ms</name>
& i% ?: x( k' K! |9 M$ H- w2 B+ r <value>1000</value>
- k$ n2 Y! |' v& E+ C </property>
9 I* ^" d: V, U3 @; A1 ^3 E! x d& c <property>
( K) h& U8 Y: C2 t1 m; x$ g% d <description>. f% s. p) E- n) V/ y# b( b; Z
The time period for which timeline v2 client will wait for draining
0 l, Q1 m* A: g, R% k leftover entities after stop.
e9 m; r9 L1 b+ x0 i, s; B& Z </description>
* n; r. f% V7 p+ P <name>yarn.timeline-service.client.drain-entities.timeout.ms</name>7 C% B+ s" K M0 e1 g- D: M
<value>2000</value>
+ D8 m7 @7 K& H+ B! l2 K </property>
( [7 Q4 H* ?0 K <property>0 r( S d: {" J5 _
<description>Enable timeline server to recover state after starting. If$ M: d) Z& r! L( J3 C) Z6 Z9 v/ b
true, then yarn.timeline-service.state-store-class must be specified.
' r0 \* |1 p$ e* k" N1 j, F </description>. k% x* _4 q( y2 V) T( ^% z
<name>yarn.timeline-service.recovery.enabled</name>9 w, N4 N1 G8 e$ u: S2 @% o; p
<value>false</value>7 o5 I& ?& Z, x, P# F
</property>. W/ h, r4 o3 b/ T4 D
<property>
& ~- Y% P* d6 n7 q" F( _" {' ? <description>Store class name for timeline state store.</description>
+ L( L/ p" Q1 [9 y) x1 X2 R6 x" I <name>yarn.timeline-service.state-store-class</name># g* m3 L1 t% p2 J
<value>org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore</value>, \+ w1 i6 [8 |4 W* z
</property>
/ A* G; U& W2 v" H <property>0 W/ r% c- [! E
<description>Store file name for leveldb state store.</description>/ \ J: L$ A5 l' Z( {9 ]8 t
<name>yarn.timeline-service.leveldb-state-store.path</name>
' V5 q: {& T' T6 b/ c) K } <value>${hadoop.tmp.dir}/yarn/timeline</value>
. k* u6 a! t( \, D, _& { </property>
/ N2 h! q7 Z' h0 E* P- P2 f3 Z; @ <!-- Timeline Service v1.5 Configuration -->
! c6 M4 x/ y. f <property>6 @% v/ i9 L1 W( C+ ^& U; Y
<name>yarn.timeline-service.entity-group-fs-store.cache-store-class</name># T8 G$ b. r" i% m( J1 [! n
<value>org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore</value>
( p& {4 I0 G3 V( N3 S6 r7 q2 s <description>Caching storage timeline server v1.5 is using. </description>
. v5 Z* n6 R9 Q+ \ </property>
, J0 B4 u( R) `7 { <property>
* Z' @; b! |. v% D: y: R <name>yarn.timeline-service.entity-group-fs-store.active-dir</name>' B+ {9 X7 M/ L
<value>/tmp/entity-file-history/active</value>
1 V. M( M6 M, B <description>HDFS path to store active application’s timeline data</description>
( D- R" {5 T. b; X% ? </property>) i1 }4 C7 Z& t% j" C4 u5 K
<property>2 f) r4 E; _% O
<name>yarn.timeline-service.entity-group-fs-store.done-dir</name>
/ V" k& b8 M; u: v <value>/tmp/entity-file-history/done/</value>' s+ F4 |% f0 J- e9 N- `
<description>HDFS path to store done application’s timeline data</description>2 g% Q4 [0 ^+ ]
</property>8 l# t1 ]3 w/ k" M3 Q! ^6 K
<property>
1 ^6 R+ Q+ x( j1 |$ k. {" L <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>
9 z1 L' P) M0 F6 X, p <value></value>
5 v# o' R9 D* d) B$ m7 k- y <description>( D9 j5 }1 Y% O. m
Plugins that can translate a timeline entity read request into
" ~3 r- ]: H! X a list of timeline entity group ids, separated by commas.' R' x- \/ z- g0 p, |( y9 @6 z
</description>" M: P1 y2 h" G7 W. L
</property>* K, `% \ c) t" ^# s$ F: K
<property>6 q- T8 h8 R( C0 a |# q' ^
<name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath</name>
( p% }' X. U5 s/ S/ j* ^5 W <value></value>+ O5 A x, ^/ v
<description>+ |3 _( W& v3 {% {. x8 X9 W
Classpath for all plugins defined in' }" e- z1 ?. n0 x' E: v
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes.2 n$ y5 i- `: W8 D0 ]
</description>
* S @: v, n- E1 |' _ </property>
1 O$ t$ V9 [5 J; W* |9 g2 M <property>2 f* g6 H5 O0 ^/ Q g3 [
<name>yarn.timeline-service.entity-group-fs-store.summary-store</name>
' W( M. T7 D3 m; e <description>Summary storage for ATS v1.5</description>
, v' o+ I& Y* N <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
+ ^0 B8 m) e. u' O$ R </property>
8 M U+ ?+ m- g0 e: U& u* `0 @1 V <property>
+ k6 z# k% R- z/ V O2 k. l9 ]" u <name>yarn.timeline-service.entity-group-fs-store.scan-interval-seconds</name>
3 a/ U/ G7 x0 k+ f9 ^' w <description>
+ }8 C8 b/ P- ]6 _* @ Scan interval for ATS v1.5 entity group file system storage reader.This
$ C0 O2 r: n: D- L value controls how frequent the reader will scan the HDFS active directory
]# m9 S* e& H5 l9 t' n7 Y for application status.
$ D, M* {/ P& b9 i </description> {0 a( F$ d" v, c) |
<value>60</value>
: }# h& d( \5 Y. W& o" ]4 Q } </property>
" R- d, N2 E1 b, l$ F, v" K <property>
) e4 ]- w) q7 J- N! _ <name>yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds</name>6 h% p9 V4 b% V% p& d8 J: n0 A
<description>/ B g9 ~6 L O) M
Scan interval for ATS v1.5 entity group file system storage cleaner.This: L/ q; ~8 `, w0 ~+ m
value controls how frequent the reader will scan the HDFS done directory9 A j0 Z: H6 j3 Y( z0 ] r
for stale application data.4 x0 q& c! O1 U6 p/ b0 V, C
</description>3 ~+ k3 b" C6 x( G! t# E ?! F. G
<value>3600</value>& f+ s* R5 G4 F
</property>
* l* K$ h/ ?1 f9 w9 d <property>. k* Y6 C0 d* s9 z7 U1 i5 G
<name>yarn.timeline-service.entity-group-fs-store.retain-seconds</name>6 V; t% \( c" b/ Y# V
<description>
9 r9 w- j/ ?7 W5 _/ Y How long the ATS v1.5 entity group file system storage will keep an: r" D( }$ h7 e9 y' f
application's data in the done directory.' f6 \/ P8 K7 ]6 v$ S' {
</description>
$ U0 k/ _4 O; H, D2 h% m: h* _ <value>604800</value>
8 N6 ^9 ~. Q) L' T </property>
! R( p7 D& f% ?: _! S+ ` <property>
" m! Q5 ~) L$ e <name>yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size</name>" M/ f9 _; _ Y0 M: o; u
<description>
, Q- n9 l a* k; Z7 y Read cache size for the leveldb cache storage in ATS v1.5 plugin storage.2 [, L" z4 j; @
</description>
X' P ~1 d7 m. U @: `$ O% h/ s <value>10485760</value>
0 p' i% i+ \# h7 ^3 E </property>' m3 ]0 E( j0 m* l
<property>. {9 m* z% K- X" ^3 D; q
<name>yarn.timeline-service.entity-group-fs-store.app-cache-size</name>% W2 C4 \* [. M! ` X
<description>) j7 U2 b9 d8 H! g' ~0 \1 |
Size of the reader cache for ATS v1.5 reader. This value controls how many4 i5 C1 L, A' O
entity groups the ATS v1.5 server should cache. If the number of active
3 _4 j' ?/ ?6 W8 d% v, P3 P, z read entity groups is greater than the number of caches items, some reads" n# a% {9 @/ Q/ N3 O
may return empty data. This value must be greater than 0.( R! [% ^% N& k# s3 G; X9 P ]
</description>9 ?/ o- S& G! \; \
<value>10</value>' I* d" r8 d' `. u8 n; [$ O3 o H
</property>! Y5 e5 X/ u/ x% W5 c$ I
<property>% r! z- _) r2 L2 y- v
<name>yarn.timeline-service.client.fd-flush-interval-secs</name>6 {! t3 Q) T; {0 [ m1 d X4 Q
<description>+ a D& N6 a0 V. O/ c
Flush interval for ATS v1.5 writer. This value controls how frequent
' `2 R' m. L0 M( W& X the writer will flush the HDFS FSStream for the entity/domain.9 N, m! d3 U c" _9 Z9 @
</description>
/ [0 f* A: l' Z0 n; J( a; K9 |, w <value>10</value>
1 }+ H; ~6 u0 W! d% o </property>4 t1 {& n9 K5 S1 ^1 v! I b0 O7 r
<property>
8 ^& f) E' L: _/ q h <name>yarn.timeline-service.client.fd-clean-interval-secs</name>1 e; B* B. z0 X6 M$ j6 @- u7 ~2 t
<description>) D- R" E/ v! l* F& c
Scan interval for ATS v1.5 writer. This value controls how frequent
3 s6 {5 N; V. X( p the writer will scan the HDFS FSStream for the entity/domain.$ ~5 T1 R6 A0 B
If the FSStream is stale for a long time, this FSStream will be close.
1 {; F, O+ Q% ~7 c6 n </description>
# m% A8 k6 L1 x c j4 ?1 w <value>60</value>
, e) L8 n" H; O; L </property>
5 f1 X) S* l6 x8 y# G6 z. _ <property>4 U% X$ W7 V- o1 z' M0 Y5 L% Z' t
<name>yarn.timeline-service.client.fd-retain-secs</name>8 F3 L% S6 m1 \
<description>
/ `* l% l8 D% o6 A How long the ATS v1.5 writer will keep an FSStream open.3 t1 a9 M% L ^1 I+ z, R
If this fsstream does not write anything for this configured time,7 S! D& \1 A/ t% b6 t
it will be close.4 }" m" _' Z* B. S
</description>
4 [. Y' `! W! [' p% u2 b <value>300</value>& R7 F* s5 W# g9 I8 V/ Z1 C
</property>, U* D" V( n2 j4 e* r! a. m
<!-- Timeline Service v2 Configuration -->( R5 z' F2 d4 j! W5 g2 M% L( i
<property>" l4 S! J) W6 B7 [
<name>yarn.timeline-service.writer.class</name>4 n2 p. D6 f& c( ]0 v Z3 j
<description>$ ^( {6 C" t* P& h$ A I
Storage implementation ATS v2 will use for the TimelineWriter service.
1 r& f; x- Z& Q4 q* m </description>
+ {6 T& {# n3 Y <value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl</value>: `4 q; N8 k+ } X
</property>' e/ B5 w9 }+ G
<property>
, O6 |5 X( x* G <name>yarn.timeline-service.reader.class</name>
8 S$ _/ |1 M* Q* e! b! D <description>
' V: A* H9 \- o' w1 E Storage implementation ATS v2 will use for the TimelineReader service.
R+ T% e/ `! T5 i( f% l. n! w/ I/ N </description>( x$ g: \. p& {+ Q
<value>org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl</value>0 A" I9 ^+ H) {$ @& ?* i1 a
</property>
2 b' H" i0 a" ?4 S8 v0 G/ I <property>
1 j& b8 ]' ]) S9 G7 T/ p9 C <name>yarn.timeline-service.client.internal-timers-ttl-secs</name>
. M/ u9 H4 r- q& I+ n# z5 b% { <description>
0 E$ D! Z6 \! @: \) z8 x How long the internal Timer Tasks can be alive in writer. If there is no
( f" m3 e5 t8 j: D1 K9 O8 z; S+ L write operation for this configured time, the internal timer tasks will
1 c% s7 o$ W9 E be close.
& i# q; _# ]' r# T$ g </description>2 k* m( n8 {9 U, ?* ~
<value>420</value>
" n: f- N) W) T g9 q </property>
Z: t# @ k% o: D" | <property>" q4 o- r1 }( c
<description>The setting that controls how often the timeline collector7 y- w1 r& C% v- n" P, L
flushes the timeline writer.</description>
! S' q l8 I" ^9 t' n5 K <name>yarn.timeline-service.writer.flush-interval-seconds</name>
0 W% w6 B/ J6 Y i% N4 W6 ^/ Y L7 x <value>60</value>
4 d9 g# ]: P' y# M; O4 [ </property>
+ H6 u' `% y, Y; V1 ?8 s; l* F <property>
* V2 Y5 t9 ?& J; e$ g) L& X6 B- o+ u$ ? <description>Time period till which the application collector will be alive
# R8 O: l' H% k! d in NM, after the application master container finishes.</description> A' K* B# V9 b1 U
<name>yarn.timeline-service.app-collector.linger-period.ms</name>+ o- s' H6 g4 `; z% P
<value>60000</value>( F( F' C. E8 S3 a* C2 H
</property>
, W( w+ ~! Y1 u$ N; q( n <property>$ J: C J6 L- p O7 L2 A: F& {
<description>Time line V2 client tries to merge these many number of. ~' C( z2 f5 o z- c2 @ q4 j+ J
async entities (if available) and then call the REST ATS V2 API to submit.
% K0 a5 L/ ?/ o' e4 K </description>& f9 h; Q- ^: K! c7 F
<name>yarn.timeline-service.timeline-client.number-of-async-entities-to-merge</name>! E* K# e, X! h$ U
<value>10</value>! I8 F! y) [' e/ Q+ c5 c+ r$ M
</property>
2 @6 a( M& W3 H+ d$ ^ <property>
, A% ~8 Q9 S- ~& n9 q7 u <description>
( U3 F% X6 v% R# C. J The setting that controls how long the final value4 r* P: m2 y2 e9 t
of a metric of a completed app is retained before merging into( x/ ~& Q) }% N Y. }% m l
the flow sum. Up to this time after an application is completed
[: L8 X" L+ g out-of-order values that arrive can be recognized and discarded at the# T1 I! i) d: u9 S6 D: `
cost of increased storage.
- X4 M$ Z7 {" \* X+ @( c) D' G </description>
4 I l b8 u/ K5 U) f8 F6 \* Q <name>yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds: m# ]) k2 u! l2 f' A
</name>" z9 S8 @: ^% P; v! Y) Q9 s
<value>259200000</value>
$ o4 r$ L3 W1 f8 ~4 m% N </property>$ G$ w* Q ~* p! |+ G$ ^
<property>
' U! m2 z# ~& B0 Z9 }( Q <description>
# ~$ i- ~5 p& X1 ^: H/ K- ` The default hdfs location for flowrun coprocessor jar.- B) y; r+ F2 B% p8 w0 ~; p
</description>4 n/ n3 U* m% s! A" Z% i1 e) A
<name>yarn.timeline-service.hbase.coprocessor.jar.hdfs.location& J' e. _$ U' f' G: K x1 O1 U3 N
</name>7 w, \- _8 A. u
<value>/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar</value>/ X4 n" F6 ^( g! Z% \8 R+ s
</property>
% c/ B6 K% s! @8 | <property>
6 z- C, ^: }( I- K9 K; K. I& ^ <description>
4 `% T8 X3 V" W/ @$ Q! r S The value of this parameter sets the prefix for all tables that are part of
) v2 l, z! ~3 d: U4 A4 c- {9 X timeline service in the hbase storage schema. It can be set to "dev."
3 L2 b2 a1 a+ U) _+ r b1 U8 i or "staging." if it is to be used for development or staging instances.
% o# V. C) G6 ?5 S U) P! _# y1 [ This way the data in production tables stays in a separate set of tables) c9 x2 C3 V& m+ ?+ v, ^+ h- z) t/ m
prefixed by "prod.".
! b D2 ~8 j2 F </description>. J! ~1 T8 T3 M- }
<name>yarn.timeline-service.hbase-schema.prefix</name>
& a; f: n. s9 J% ]5 _7 I- ~3 Y% D <value>prod.</value>
+ h; ]6 J: K3 i </property>. u; Q% K" q' o/ c6 f( ^; I
<property>
+ K8 y, v; `2 @) I <description> Optional URL to an hbase-site.xml configuration file to be9 N7 @' V' G' u
used to connect to the timeline-service hbase cluster. If empty or not
) j" D7 Q/ z8 }( L+ I3 r specified, then the HBase configuration will be loaded from the classpath.! G( h$ A; ?& K9 `) [+ @: o( T' Z
When specified the values in the specified configuration file will override8 \6 y( l; X$ {0 Z8 [; ?
those from the ones that are present on the classpath.
( |& r) r6 t) o1 \ </description>
- j# d. f$ o4 r% w" e <name>yarn.timeline-service.hbase.configuration.file
4 c: y* X" w. L L0 @; A5 e, ] </name>. J, Q* @7 f. `7 ^) V
<value></value>% A1 ^! y5 L; A5 ^6 X) |
</property>+ q- a/ O* H0 C- |% ^) y$ o
<!-- Shared Cache Configuration -->0 n5 n6 h! B" o
<property>
9 h; ]; D6 ~% S. }6 x" u& k <description>Whether the shared cache is enabled</description>
: |: E7 B3 s6 z- b <name>yarn.sharedcache.enabled</name>. d4 I E; ?" Z7 i- }; y$ N
<value>false</value>
. _+ U0 g9 J0 i6 y </property>6 Y* l$ Z" X- A t4 o
<property>
0 \& X0 e. w# D" C <description>The root directory for the shared cache</description>. J5 R( B- y# v, i
<name>yarn.sharedcache.root-dir</name>4 G: F1 p/ d/ e
<value>/sharedcache</value>
8 g. B% C; ]: V: n6 A9 M9 n! S- I2 k </property> G! I6 _) s; J4 k
<property>
. c' p# |# {$ o6 ~% U [4 V <description>The level of nested directories before getting to the checksum
% `5 q# T$ \+ j) ^' i' r' g directories. It must be non-negative.</description>
# w0 A3 a: c/ s5 r* | B <name>yarn.sharedcache.nested-level</name>
- N; a) b0 L' n8 U <value>3</value>) A8 l0 f7 @4 \. c6 P( _9 p
</property>
* y( b9 a/ O g <property>& g" T+ v9 C) h R. o1 U6 N
<description>The implementation to be used for the SCM store</description>
6 {) P2 w+ w+ {/ f% X/ B% A <name>yarn.sharedcache.store.class</name>$ t8 n; {( D6 I- e
<value>org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore</value>' w, K8 d0 _3 g. g: c1 V/ K/ V% b w
</property>( S4 P o1 n% m0 I$ d# \( B
<property>* Q, u* N9 K- e. Q7 z
<description>The implementation to be used for the SCM app-checker</description>- W7 D8 k( v/ F# E: f1 h" p
<name>yarn.sharedcache.app-checker.class</name>/ a9 m4 g7 ]( H. Z$ d c
<value>org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker</value>8 D% Q) N# C0 H1 ]
</property>
* z) L! H9 e! Y/ Z" L0 I <property>. T! b' H" i- p+ t/ S) o* V
<description>A resource in the in-memory store is considered stale6 a j% N; ` o7 Y! E
if the time since the last reference exceeds the staleness period.' W" ~9 w L; u+ T& b* X
This value is specified in minutes.</description>: F" H( K, j6 |( G
<name>yarn.sharedcache.store.in-memory.staleness-period-mins</name>& m: P4 o( T/ t' O4 L
<value>10080</value># L+ g7 ?. N( L
</property>1 y/ ~6 c0 D; d1 i
<property>
: S$ v1 V! b* m, M* v+ Q <description>Initial delay before the in-memory store runs its first check5 }- ], P! A! T. d
to remove dead initial applications. Specified in minutes.</description>
5 n* C' I5 |! g0 ~ <name>yarn.sharedcache.store.in-memory.initial-delay-mins</name>
. I `& Y! c7 T' W! m <value>10</value>
! d/ B4 g5 l* K5 r7 w </property>
8 U4 }; |, @9 B5 E8 A6 N <property>
4 |7 h; ~3 h: |: H0 r- F <description>The frequency at which the in-memory store checks to remove0 }7 x L' B8 o3 n! K% x
dead initial applications. Specified in minutes.</description>7 F* M- R4 r. e3 {7 O
<name>yarn.sharedcache.store.in-memory.check-period-mins</name>$ O# ?' y) p" D4 ~
<value>720</value>
" B# M3 v5 }5 A! M' G </property>$ v* n8 E' w6 |' g" b* h
<property>
9 W5 c# m2 M. a- _ <description>The address of the admin interface in the SCM (shared cache manager)</description>
0 Q2 m6 [( X8 y+ M, h* l9 _( n <name>yarn.sharedcache.admin.address</name>: e9 z- t; h5 C! x5 l
<value>0.0.0.0:8047</value>
+ {# ~+ u+ p) r! ? H3 ~# G </property>
* ~. t3 Q% ?1 B& P. ` } <property>+ }0 N/ M) l+ o1 `* o, A* Y0 V
<description>The number of threads used to handle SCM admin interface (1 by default)</description>/ k7 \9 R" S' s' i5 {* A* z9 Y8 T
<name>yarn.sharedcache.admin.thread-count</name>& R: X% J* N- O- D1 m- q* R
<value>1</value>9 q3 e, }# A9 I8 d- a5 r- g5 }, r" C
</property>! v, C" ]/ b, R. }3 c, D
<property>
/ m. a& k( ^7 a1 e# l6 e <description>The address of the web application in the SCM (shared cache manager)</description>
6 {8 p' g2 B( C6 W/ y <name>yarn.sharedcache.webapp.address</name>
* ~ m1 W9 Z& P5 x: b <value>0.0.0.0:8788</value>% M, ^( w$ `7 Q0 u
</property>
; \$ c2 P4 \* n$ A- J <property>
- [. e& _8 A# {9 p( q# c2 S9 b3 X <description>The frequency at which a cleaner task runs.
( t$ Z9 H* o1 P) g/ g4 } Specified in minutes.</description>
7 T1 p- Z& H* i+ c" { <name>yarn.sharedcache.cleaner.period-mins</name>
; z+ Y7 B0 c: }& w$ S9 Y <value>1440</value>: m* j* Y" z! c8 X: M
</property>
% q7 Z- Q! w5 L2 E# ^ <property>" A. Y5 b5 U& i6 Z1 T# a
<description>Initial delay before the first cleaner task is scheduled.
8 r. i# U$ V. ^% H- J5 \8 ` Specified in minutes.</description>
\; V9 I: g& l8 X <name>yarn.sharedcache.cleaner.initial-delay-mins</name>
0 ]" D k6 u7 V6 e <value>10</value>+ ^+ `, c+ h* W* [
</property>0 n4 e% A6 W0 M; g0 E
<property>
! ~# s' N; _2 ~) v' h! j <description>The time to sleep between processing each shared cache
* z! C8 k/ L: n) P3 z* b* k4 q resource. Specified in milliseconds.</description>5 F& ` H1 D4 V# b
<name>yarn.sharedcache.cleaner.resource-sleep-ms</name>6 E3 f' { I( h7 ?6 ]# W
<value>0</value>
( e+ v& o5 S7 H </property>
( c/ O" T2 k) s: P: o. v <property>( v/ G+ C* G7 J6 q% i v( g$ T x, p
<description>The address of the node manager interface in the SCM- H. q1 |2 C, h& i% S0 j# ^
(shared cache manager)</description>" n$ }* q7 e% `' G
<name>yarn.sharedcache.uploader.server.address</name>- l! b* w6 G! B5 X3 m
<value>0.0.0.0:8046</value>, F( M3 e" Q9 s E4 {( L, Y; b" J
</property>
# Y @* [9 p8 v' r3 t/ V <property>- L9 o6 J' Z# I' L% X* b& t
<description>The number of threads used to handle shared cache manager' c/ W8 ` x8 w! p8 }+ ]
requests from the node manager (50 by default)</description>
; X. L* `; ?- Z0 W% C <name>yarn.sharedcache.uploader.server.thread-count</name>
5 |/ Z; h7 O n7 Q <value>50</value>7 q, G+ ]$ Z5 r" p6 U" k
</property>% G& m& F2 Z* I6 {5 q. m3 |. L1 u
<property>7 V2 [* u& O2 A9 `* R
<description>The address of the client interface in the SCM1 j1 Y2 Z+ j$ |
(shared cache manager)</description>) m% I/ d4 {! v$ B! n9 d
<name>yarn.sharedcache.client-server.address</name>8 x9 X* ?( n. k
<value>0.0.0.0:8045</value>
, ~4 o) I( a' ^. Z9 X+ i( a+ R </property>
, d7 U5 D" @* i; x) j <property>/ q( i( ] p: Y; M6 |
<description>The number of threads used to handle shared cache manager
; ~* _( b/ i& U# `) o d0 U requests from clients (50 by default)</description>5 }1 b8 H: E; Y# u# I+ B
<name>yarn.sharedcache.client-server.thread-count</name>6 ?; m! Y- H; g4 |' |0 a+ I; k
<value>50</value>& i7 u3 l: v" `$ X
</property>
1 ]* D! L7 n& G# q9 v <property>
+ }6 D* M+ l5 P6 r# C" J" _ <description>The algorithm used to compute checksums of files (SHA-256 by+ t% n; C |+ X6 j
default)</description>0 z( ?6 u0 { C
<name>yarn.sharedcache.checksum.algo.impl</name>
9 J( O" e) i, { D x( P <value>org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl</value>/ Q$ O4 j O8 r3 `
</property>' t* S1 e- v5 X, j: M
<property>
! \% y/ a& n* b6 x <description>The replication factor for the node manager uploader for the
2 J+ u. H( E3 T6 N) J shared cache (10 by default)</description>
/ Y, {' p$ ^5 h <name>yarn.sharedcache.nm.uploader.replication.factor</name> n7 L! o0 P! Y7 J
<value>10</value>
" m5 Z! \* B- V0 x: e! s& }( n* F </property>, i# Z7 t S, z) q4 H' c0 t
<property>5 Z: r ~6 ~! |5 m1 U
<description>The number of threads used to upload files from a node manager
1 h+ M$ n2 z6 s3 H. g instance (20 by default)</description>" f- q8 g! H$ k9 D3 s
<name>yarn.sharedcache.nm.uploader.thread-count</name>$ y. C. l6 p0 t, {1 Z
<value>20</value>
6 u& g& T' M c* v" G0 E- ~" b7 J </property>
. |* x& j& z+ K7 u s' k <property>
. _& `: `# {& e: Y0 F- d <description>
, ?" \0 e) w3 b& z6 Q ACL protocol for use in the Timeline server.
- L' v! k/ a5 r( }# B </description>
5 Z7 s6 W, `2 g <name>security.applicationhistory.protocol.acl</name>
" _: B% ~7 Z; J# s# E% Y# j+ z <value></value>6 |9 ]0 B! O9 I' ~- m
</property>4 S- q0 }, U# c( q6 U5 N) B
<!-- Minicluster Configuration (for testing only!) -->, s) }' Z) b% Q5 f% w4 b
<property>; X3 n$ l) [1 U8 ~# ~8 o! b
<description>
: _# S$ o' M) e- K Set to true for MiniYARNCluster unit tests( [$ P6 Q* Q7 ]) o- i9 ]' \
</description>$ z- U) w7 x5 e, h) o0 o/ O
<name>yarn.is.minicluster</name>
# ?; D9 x0 V# R. |4 W, e <value>false</value>9 b7 [2 T+ ~ w1 |. Y; F' F. r; O
</property>
9 U' y+ Z1 {; ]+ p9 n: U' m4 I <property>
+ `0 o" ?( S( F, O <description>, n; c9 l; {8 S( x% [6 L3 i- B
Set for MiniYARNCluster unit tests to control resource monitoring
( ` L! y0 B6 u </description>
7 k2 k+ U% l/ ]2 ?0 `) z <name>yarn.minicluster.control-resource-monitoring</name>/ W. h7 D! k0 h; A3 z( _+ f5 I0 D
<value>false</value>3 A/ @2 X, d& D1 U) \( @% K5 G; E: Y
</property>5 Q) e7 J9 {# U' u
<property>2 V1 N/ p7 E$ o J! k& G
<description>6 L# r. G* G( h
Set to false in order to allow MiniYARNCluster to run tests without
6 H9 Y! \. [0 A port conflicts.; e+ r1 O3 `8 e7 {, Z
</description> m3 p8 |5 f1 O; r9 L- K
<name>yarn.minicluster.fixed.ports</name>
0 f' s, S; ?5 O/ W R <value>false</value>+ ^# P6 ]8 p5 w; @2 u) m) s
</property>$ X8 q' M( z. a/ i/ p
<property>) F- E% ] o/ p9 Z% F7 b
<description>
) z: S% s4 L9 z7 e0 @9 F Set to false in order to allow the NodeManager in MiniYARNCluster to9 ?- T( F" n \( A" {+ c+ h! ? H
use RPC to talk to the RM.6 [7 m& B, w' |* \ D, v
</description>
& {9 K8 M" t6 J' ^( q5 q8 ^ <name>yarn.minicluster.use-rpc</name>3 J; b2 A$ g9 @. q" x& r/ l
<value>false</value>6 v1 e4 r' C# I8 M1 ~7 ?, ~
</property>1 ~* a D; n5 w7 h
<property>% m& W# F8 w% b6 e% k2 s
<description>( x5 [8 @- U2 l4 G
As yarn.nodemanager.resource.memory-mb property but for the NodeManager
" j }& c" B( q4 y: z in a MiniYARNCluster.
2 V+ s/ U( t) `5 M" q </description>
6 b, \/ |* W; M" ] q0 X <name>yarn.minicluster.yarn.nodemanager.resource.memory-mb</name>
7 |/ z- w( z. z* G6 N* A4 F$ C <value>4096</value>0 w. t H7 U+ n7 r/ x0 a3 H4 P
</property>
# V6 S3 C6 W$ S$ S% `& T <!-- Node Labels Configuration --># m: ]. s; v/ d/ |# ]
<property>) G0 o- U$ l' o" ^: A* a! o
<description>
; r* M3 w' _9 K& s$ ]( q, q, a/ ? Enable node labels feature# J7 x2 ^2 m7 D# w* J* i' R1 b/ u0 _
</description>" s* e' L! O0 P3 ^- T
<name>yarn.node-labels.enabled</name>
9 o& m0 N5 E) E0 v0 m/ v <value>false</value>
" H! l: c B8 C9 M </property>6 ?, c2 b$ v- w8 r
<property>3 w4 `5 i7 k) n5 p ?( X8 j
<description>
+ h% | }3 ~+ } URI for NodeLabelManager. The default value is
5 F3 V7 ~/ l7 t0 ` /tmp/hadoop-yarn-${user}/node-labels/ in the local filesystem., z9 f# Z' \( C' b
</description>
( z: ]) x& |+ o' D, ~ <name>yarn.node-labels.fs-store.root-dir</name>. w+ o9 G# E* S* v, a5 r
<value></value>2 V2 G) T9 c3 e P
</property># Y0 ~* G5 M' V$ m4 s/ a: a
<property>9 N S: P( y. m( T' ~
<description>& h+ O- h* b# ]! W, P7 \. Z8 b
Set configuration type for node labels. Administrators can specify ~. t: D( o# m: |# G. N8 @% L: |
"centralized", "delegated-centralized" or "distributed".# {, R" J! q, w% [0 h3 J1 [3 Q; m
</description>( G7 j( g4 E5 M* O# f: _
<name>yarn.node-labels.configuration-type</name>
" ]! d$ C! W# G6 r1 r/ ~ <value>centralized</value>, R) N' q! m4 U1 d8 Q3 E& k
</property>
! t7 e6 K% @8 e2 u( X <!-- Distributed Node Labels Configuration -->2 C' j1 _4 I8 p m" z- Z$ C, ~
<property>
/ U& S. N1 r" O <description>& C2 R0 w1 _4 l" L# ~
When "yarn.node-labels.configuration-type" is configured with "distributed"3 M% @0 M0 ?- Z9 a6 P7 n
in RM, Administrators can configure in NM the provider for the; m( ^1 B* d- o, [/ e c1 S- l3 O) r
node labels by configuring this parameter. Administrators can% c9 j: a3 o4 f3 m, w8 t
configure "config", "script" or the class name of the provider. Configured" V q( q4 m. S, `8 ]
class needs to extend& r9 w+ H3 M5 G# z0 W
org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider.# D) j, I# d& T. ~9 M
If "config" is configured, then "ConfigurationNodeLabelsProvider" and if+ z, k n( s% ]. Z
"script" is configured, then "ScriptNodeLabelsProvider" will be used.. P0 M" L0 J( `; x6 m
</description>
8 X/ X$ ~3 T3 T6 Y <name>yarn.nodemanager.node-labels.provider</name>
0 e# t J& ~: `! F l </property>' P4 r1 `* M N% {3 F$ M7 \, D
<property>$ }, a& T) |- U4 h' j
<description># ]$ ?( h' [3 |
When "yarn.nodemanager.node-labels.provider" is configured with "config",! X* F8 v& |( K
"Script" or the configured class extends AbstractNodeLabelsProvider, then
, i/ M) a9 J0 a; O$ Y& m' ^ periodically node labels are retrieved from the node labels provider. This, y ~6 J1 P# H! {4 r& k& o
configuration is to define the interval period.8 l0 H; W0 [3 c! ]% e2 B0 h/ O
If -1 is configured then node labels are retrieved from provider only
: d1 z# k5 n: G! \/ E0 z# [2 Y; G during initialization. Defaults to 10 mins.
' g: ?2 E8 J t& P+ L3 }$ ~ </description>
# c! G) n _3 p) { S <name>yarn.nodemanager.node-labels.provider.fetch-interval-ms</name>
1 P; W/ v1 h3 n) E3 h <value>600000</value>
) @, P9 _5 o- L7 P </property>9 C* s2 m K0 c6 U& j Z
<property>6 P/ ~6 r+ \9 ?6 f! u
<description>
8 ]2 w4 ]/ I, O/ Y Interval at which NM syncs its node labels with RM. NM will send its loaded
6 `( C4 j% Q0 B0 ^ labels every x intervals configured, along with heartbeat to RM.
$ k6 y4 S6 ~# r3 U9 x$ W7 Y </description> q1 @" o+ X& X' l* H
<name>yarn.nodemanager.node-labels.resync-interval-ms</name>
6 @) I5 u. A& _2 i- T <value>120000</value>
1 G9 t6 t9 R% u( N8 Z, ?- C </property>7 g+ w$ h; {* U7 M: p) Y$ F& L$ Q
<property>
- S" g6 [6 W# b# K7 a$ P, K" B <description>/ h# t1 k$ F; J/ L; K/ j" h4 n
When "yarn.nodemanager.node-labels.provider" is configured with "config"
: G- }+ B6 L/ Y5 d* K1 F then ConfigurationNodeLabelsProvider fetches the partition label from this5 X4 n# a6 y9 F: G
parameter.3 E$ P7 L" ^. t$ [; S
</description>) M, l8 {! W" p
<name>yarn.nodemanager.node-labels.provider.configured-node-partition</name>
) |( H/ F4 N/ n( `2 y </property>2 k9 L6 O. j; z* Q, o
<property># d3 ?( S& Z3 o9 R6 G' Q+ a6 n
<description>
; O6 O: B) V1 o5 G+ a. ]0 G+ h+ ? { When "yarn.nodemanager.node-labels.provider" is configured with "Script"
3 g" z" W; F$ [9 K* w+ T7 o& r- G8 d then this configuration provides the timeout period after which it will# ~" x$ r; ~' K3 O. b0 S: O
interrupt the script which queries the Node labels. Defaults to 20 mins.3 q- W! I+ @* V* }1 I, {
</description>
; n! r1 i* B7 ]3 {! Q3 B. g) | <name>yarn.nodemanager.node-labels.provider.fetch-timeout-ms</name>1 E" m1 s8 l! S5 E7 Z6 j
<value>1200000</value>3 | O& A: q, E: \
</property>" x4 @% H# j" Z* P* A
<!-- Delegated-centralized Node Labels Configuration -->8 L2 A0 \0 I9 y+ s2 t( V) N; s
<property>
9 M( f* v* L, u2 r) s1 K8 p <description>
4 n& l) R+ U! n+ ?& Q/ [ When node labels "yarn.node-labels.configuration-type" is( D+ d* d# E: x
of type "delegated-centralized", administrators should configure. ^7 u* x, P) ~6 r3 F1 | i
the class for fetching node labels by ResourceManager. Configured
9 m7 T# p1 [. j, T; G/ p& A* j class needs to extend
3 Z H! T5 W! s7 K' t# Q, | org.apache.hadoop.yarn.server.resourcemanager.nodelabels./ [0 v1 I( [2 J N" d( p0 W+ I
RMNodeLabelsMappingProvider.
' O u# H5 ]# J% G8 \ e7 V </description>
0 c" w2 j$ k \1 Z <name>yarn.resourcemanager.node-labels.provider</name>
8 s2 {0 I! A* s |0 ^, ? <value></value>
3 s! [( Z) c' P, a </property>5 N- t z1 d$ t7 U3 ?4 o; S
<property>
4 i4 M4 g7 y; c! i <description>
' a- P; ]9 d6 p! S1 m' n When "yarn.node-labels.configuration-type" is configured with
) c' q5 L$ J# [% Z+ e( I "delegated-centralized", then periodically node labels are retrieved
/ D' m' u6 X; _* R# P from the node labels provider. This configuration is to define the
+ _0 ^/ G* z1 H, ?9 F- K interval. If -1 is configured then node labels are retrieved from
; f( _; J# R: [6 @# N; ~0 i provider only once for each node after it registers. Defaults to 30 mins.7 [2 D. h& N) ^, ?# u3 l
</description>
, b2 M6 d9 M) b4 q <name>yarn.resourcemanager.node-labels.provider.fetch-interval-ms</name>9 @/ k4 E! y7 J& q
<value>1800000</value>
2 A8 S$ {0 c$ d2 `7 w </property>. l8 B! T, z7 ~
<property>
+ h, }" t2 @1 @, H" d, _3 U5 Q <description>3 m, S' ~0 o; d3 ]3 v" k1 M2 \$ S6 g5 \
Timeout in seconds for YARN node graceful decommission.
% y. ^+ r6 Q" h+ L/ O This is the maximal time to wait for running containers and applications to complete0 u7 r0 [# Z2 _' r& U% }, I) g+ J
before transition a DECOMMISSIONING node into DECOMMISSIONED.
3 ^2 }' Q' _2 e! W, _ </description>) S6 ]5 B4 h4 x( I0 i) D f
<name>yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs</name>
+ Z7 W! N, x0 v+ Y9 h3 `% l <value>3600</value>' d. x7 K8 i0 z- D7 d
</property>: s6 G. o( W/ ^( D {" C
<property>
- m6 w2 h5 b2 F+ G4 P. | <description>9 r& V0 X! }$ o' Y0 L
Timeout in seconds of DecommissioningNodesWatcher internal polling.6 ?- C8 t* _* z/ Y# W$ b6 i
</description>
0 R5 l5 D! b! R" C <name>yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs</name>
6 y5 v, z6 g* ` Y <value>20</value>
" Y4 Y7 ~" Y+ c- P# |3 o" E' ~ </property>+ [, Z! i" y4 ?# z l
<property>
3 R( f: [: t* ^ p <description>The Node Label script to run. Script output Line starting with/ E. t. F+ `8 n8 i* j
"NODE_PARTITION:" will be considered as Node Label Partition. In case of4 p+ J. m( Z1 j+ O, l
multiple lines have this pattern, then last one will be considered
c7 e1 ^6 q& z0 q </description># n. \5 M; m5 n% d
<name>yarn.nodemanager.node-labels.provider.script.path</name>
+ a8 {1 y$ `+ ~/ a( ~1 R </property>* V/ a" O4 U' k3 ~! G/ T/ P
<property>/ F6 {3 z# l( P* `; k4 t+ }" K+ Y
<description>The arguments to pass to the Node label script.</description>+ _" z3 ?! j$ J5 U5 N
<name>yarn.nodemanager.node-labels.provider.script.opts</name>4 |% o* ]( j- W" W% k/ B
</property>" m% a: X6 b6 {' G! a) t& Z# H! Q
<!-- Federation Configuration -->
}# l% s2 i! a <property>
\* n( y! ?6 j( O& ?- X <description>
0 y: g9 t3 i5 h& K9 T# } Flag to indicate whether the RM is participating in Federation or not.
8 r* [4 O S; X8 c. h </description>
) B* s6 G, x7 g% P <name>yarn.federation.enabled</name>, Z$ P( c0 C# o8 I. j
<value>false</value>- C/ h$ q8 k: Y: R. {( C! W
</property>
6 @& A* W7 j- d* I: x- c S" K <property>: X4 o u" H- p
<description>" v2 w) N a S% c# m
Machine list file to be loaded by the FederationSubCluster Resolver
2 B' k4 S. _) d) S0 S </description>
1 Q8 d; J3 p5 j; e, u# Y% d2 ` <name>yarn.federation.machine-list</name>: P& y8 \ x2 Z4 {9 ~2 b* p @
</property>
& z+ g+ w' {% Y" }4 h5 e: O <property>
( y7 ?! c/ b( Z u' \" N2 a <description>6 F1 Q& j) N8 e) z# E% v7 O; V
Class name for SubClusterResolver, |4 {- r z* h1 p* l; j( e
</description>
1 L9 O) {. ]. h3 w9 A' c9 x) M. Q9 \ <name>yarn.federation.subcluster-resolver.class</name>* b$ i1 j) d+ k5 k) _3 D
<value>org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl</value>
6 l6 {* X$ Y- H# ]# c2 e k </property>3 x& _+ x8 |! N# e- {
<property>$ s& I: W, e& }4 Q1 y! x; D$ p
<description>
. W7 e- a- e/ v Store class name for federation state store
: ~ O1 Q: \2 d, M0 R$ X3 o* x </description>$ f6 h Q( v, m( O; B8 s
<name>yarn.federation.state-store.class</name>
x) I. p4 I( k @% B) p <value>org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore</value>- ?1 N I& ?: S5 h, u9 m _
</property>
8 g! b! r; ?8 i- F2 V+ W* N% C <property># v) N/ ]! s) D* W* W
<description>) |; q! Y% x) c" F! [! u
The time in seconds after which the federation state store local cache- }" w- x/ J) K+ H9 f
will be refreshed periodically
$ Z/ O4 x! [" Y6 s- y0 R </description>! N, ]# h" o: g* A% @9 i* c1 H- Y
<name>yarn.federation.cache-ttl.secs</name>
; W( i# y6 Y" Z% A$ K <value>300</value>
0 O; u6 A: h% d* H0 `, b) _ </property>
. h0 } h; j1 i2 s <property>9 }$ O! f# G) E. w" o$ l( I6 A2 j6 }
<description>The registry base directory for federation.</description>8 }2 u, B) O; S9 U4 k8 N
<name>yarn.federation.registry.base-dir</name>
/ l" L3 `% w) K& s% A <value>yarnfederation/</value>: G3 y+ e0 p3 I' v1 M
</property>& c# Y( q8 ~: K! y1 P$ u
<!-- Other Configuration -->* A4 [* g2 w0 j9 H. L
<property>
$ u$ R/ U6 C6 i4 { <description>The registry implementation to use.</description># z/ M/ F% u& s+ m" i
<name>yarn.registry.class</name>0 J# _0 I( i4 E4 H6 W3 L5 A
<value>org.apache.hadoop.registry.client.impl.FSRegistryOperationsService</value>
# L( x$ N# _% D' _$ B6 ^ </property>
8 t; f0 T' @3 M5 u5 U" T <property>
* a$ c) |5 r" [ <description>The interval that the yarn client library uses to poll the: O/ b+ v% V2 \( X
completion status of the asynchronous API of application client protocol.: Z. @ V: J( c7 z6 Z! |
</description>
; Y8 ~( b ~$ U9 Z <name>yarn.client.application-client-protocol.poll-interval-ms</name>
: z7 [7 z) s0 F <value>200</value>
& ?6 u$ H, G2 X </property>
, ?6 d/ r- O9 `2 ] v( M3 o. [ <property>
7 E, _0 U" w/ [7 s; e <description>7 p" ?( y8 g/ X; E! |' y1 c
The duration (in ms) the YARN client waits for an expected state change
f& W( t% W" p: d6 u- O to occur. -1 means unlimited wait time.; J# w$ `0 L/ s! ]2 L" u
</description>
3 @) B j" C z4 N7 O- C. W. G- Q <name>yarn.client.application-client-protocol.poll-timeout-ms</name>6 \+ b ]* O4 D: e7 U- H/ }
<value>-1</value>
! `9 r# q* O" W* A+ x; b+ i </property>
" k3 W0 h3 N& a7 a <property>: r( Q7 u# K# z% @
<description>RSS usage of a process computed via! H2 A4 U; C* _: E7 D1 s
/proc/pid/stat is not very accurate as it includes shared pages of a+ x4 l: H: Q J- W4 [
process. /proc/pid/smaps provides useful information like2 }1 q! p8 ~2 H
Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used
$ x" K: g& S8 @% B! G1 g+ {1 u for computing more accurate RSS. When this flag is enabled, RSS is computed
* @1 r+ B @$ \ as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes* K# x0 }. n9 d- J5 Y3 C
read-only shared mappings in RSS computation. : { t y5 w. ^) d
</description>/ Q2 q7 C# p2 R" Y. s6 b
<name>yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled</name>4 m7 D- O0 X- W$ Z. D
<value>false</value>
/ B' _0 V7 s3 I; K! O; v </property>4 u' V; V6 C4 e$ W+ G* _3 `, n
<property>
& g0 K2 v9 U$ v( j0 N, \5 y% h <description>/ E# M' L$ X1 }! \+ ?, K
URL for log aggregation server
" f2 ?5 l- K. U: j( x& e$ X8 s r) L </description>( W6 d; H+ R- N4 z9 K
<name>yarn.log.server.url</name>
4 a( [ @% @% D7 l$ }! f <value></value>) E( p' i2 k5 j9 I5 m5 a
</property>; r: y# E) M& C1 X& X7 e- h$ n
<property>
9 W L8 @8 z4 V7 _ <description>& Q9 f" k/ O0 L1 l& B4 J1 K
URL for log aggregation server web service
! r- {: e" A" F' d </description>' N: G. {6 E' F; q% u6 H3 l
<name>yarn.log.server.web-service.url</name>
! [5 N( a1 F0 t$ [7 H( A <value></value>
4 Q7 g. i9 N, G& q+ m i. P- `4 c </property>
: ?" ? F a0 ~- `* }# Q6 l <property>
% k* J4 g' f7 t/ k+ \8 k2 ~( y+ I <description>
F) Z4 M" O0 Y, `) L2 t RM Application Tracking URL
4 C3 T, j! a7 T) g </description>
/ L) N* e o' N <name>yarn.tracking.url.generator</name>
# k) J2 \0 l7 _# \8 G <value></value>
% X! K, u0 x" f+ s( E7 A </property>5 U: ?5 l1 x" o
<property>1 ^" t( B% ?- m- N$ y' G
<description>
. K5 Y. L0 w; y7 s, W/ s! B Class to be used for YarnAuthorizationProvider; T4 Y; B+ C0 R+ P! J0 y) ?: l4 L
</description>
' z: g9 D* i# z8 {- a5 B! a4 \ <name>yarn.authorization-provider</name>
- w* a" F' d6 `5 a2 z <value></value>
; I0 g' H& \: ~: Q' B </property>2 F& ?# ~/ u- g, w, O! g
<property>
* o" a h7 a: r7 K3 ] <description>Defines how often NMs wake up to upload log files.
. b1 f+ g' U4 d- h! ?1 h3 {( x$ z& N The default value is -1. By default, the logs will be uploaded when
# E5 M5 b2 S4 K1 x the application is finished. By setting this configure, logs can be uploaded
- W7 x0 V& M2 w/ a' }6 m periodically when the application is running. The minimum rolling-interval-seconds1 z: {6 T# M* A( D1 I
can be set is 3600.( @- \% S; j$ A3 @' F i; O! F- G2 X: X
</description>9 L3 V% @8 J% |* g* q8 K6 Z
<name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>' ^5 O' E( x& E& r/ m" v' s0 w
<value>-1</value>! h: W8 f+ ^1 f
</property>
, i' t& I, z5 H+ }- R <property>
% c5 c* r9 I) h <description>Define how many aggregated log files per application per NM* b9 L) i$ K1 k# l# e3 F8 g6 V
we can have in remote file system. By default, the total number of; F9 X, y+ Y/ a
aggregated log files per application per NM is 30.- r1 M. `" G: `& ?
</description>
; w/ ?* ~5 {6 A; X" E O <name>yarn.nodemanager.log-aggregation.num-log-files-per-app</name>
" g" _+ O% `5 m4 h W0 t3 W <value>30</value>" ~: e$ A. A' `: z! b% ?6 V: x
</property>. u7 Y8 P2 e* ^6 A! b6 i
<property>1 h$ e3 l. X2 g% ?
<description>
8 ?1 G! p- k1 c1 d' m Enable/disable intermediate-data encryption at YARN level. For now,$ [% T+ J$ V$ U% W% @8 d3 e3 l
this only is used by the FileSystemRMStateStore to setup right" I; t" A0 d" X0 L6 U
file-system security attributes.
% b* y$ g' y2 h: v4 A( p </description>
J0 l$ J( X8 n u9 E <name>yarn.intermediate-data-encryption.enable</name>( n% T# D6 u- l! Y( ]# m) g
<value>false</value>
& W6 J. e' t' N! B8 E% c% b9 @ </property>
7 b- v/ g# l" p <property>
2 f- K, @. U- p: T <description>Flag to enable cross-origin (CORS) support in the NM. This flag d# ]% s* c' o3 h X9 ], J1 O! Q
requires the CORS filter initializer to be added to the filter initializers
! w% k" ?- `$ T: N; |9 ^ list in core-site.xml.</description>
- } B/ [% ~0 F; `% A2 M( F <name>yarn.nodemanager.webapp.cross-origin.enabled</name>
- m9 I6 `% M- j <value>false</value>$ H" R3 a% N' l1 }. @1 e% m/ E
</property>7 h3 o$ p" o6 z7 \
<property>
* d" }' V. ], C <description>
5 `& t1 \0 N/ [8 X F, }, {9 z Defines maximum application priority in a cluster.4 a7 J4 M& l) x1 w) Q" P
If an application is submitted with a priority higher than this value, it will be+ A& ]" V7 N0 B' d, ^) K1 _
reset to this maximum value.1 Y" n1 x1 u+ @1 O
</description># ?- ~' j, O& x ]0 i1 h$ P
<name>yarn.cluster.max-application-priority</name>; N2 l4 E6 X+ t# L, Y
<value>0</value>3 M, Z1 l, `# E3 t* Y
</property>
# \( M" R/ K+ m4 `- d <property>
7 D* v. H; k9 b& ]8 x <description>( q+ b1 l5 W7 A2 Z9 ^* {' h1 l
The default log aggregation policy class. Applications can# I; u5 J6 L( i; T" u! t! N( s, F
override it via LogAggregationContext. This configuration can provide. j" N3 c& x# G, Y
some cluster-side default behavior so that if the application doesn't+ Y/ {" r, l U. `8 C1 b! ^
specify any policy via LogAggregationContext administrators of the cluster4 u; H6 D* F+ U) l, r1 K8 ~, V
can adjust the policy globally.5 _7 }# T; d2 A
</description>
7 S8 X/ w7 {( i- D <name>yarn.nodemanager.log-aggregation.policy.class</name>, w% e$ b6 W2 [. [( B
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy</value>
1 {% d* k1 _$ v T) E* F' [3 r' k </property>
8 K# |+ w" m/ a <property>
m5 M* G2 j, U9 u1 k <description>
9 e& |- X+ F' G% s. [ `0 O. |# o The default parameters for the log aggregation policy. Applications can
) K5 B4 _! ]$ r. w override it via LogAggregationContext. This configuration can provide
3 u& N3 ]9 `) ]3 Q- [" @ some cluster-side default behavior so that if the application doesn't9 B- n' W1 M4 X! }. ?: f' O
specify any policy via LogAggregationContext administrators of the cluster
, C( J/ N$ X- A can adjust the policy globally., f* r* P# K3 ~" i% l& C: H
</description>
) `! o8 O% n! T. O; R! \ <name>yarn.nodemanager.log-aggregation.policy.parameters</name>6 L z7 o7 `, r! O8 v
<value></value>
% R/ h" `" ]# z0 h) h </property>
! `, u1 u( E; s' z% W <property>( z0 j8 I5 [( j& r5 z; k4 \" h2 k
<description>
: w |3 W! `4 B d! M( y; Y" A Enable/Disable AMRMProxyService in the node manager. This service is used to7 b! z) g% I% ^
intercept calls from the application masters to the resource manager. m4 v. Z* O* [% } F) l
</description>
; a( J& g6 r% p3 c <name>yarn.nodemanager.amrmproxy.enabled</name>
" W. L! S5 {7 h <value>false</value>
3 v( [2 S) Z" `' Z </property>4 A# x0 L$ J- F: r- v" K
<property>( b3 N+ h; T1 S# }
<description>
, |. ^/ S6 S d" w/ z+ y/ I% {9 H The address of the AMRMProxyService listener.
3 T+ ~3 W: ~* v/ o7 f5 R: ` y </description>' ^( F6 O7 i1 ?" A6 W5 Z
<name>yarn.nodemanager.amrmproxy.address</name>$ U. V7 J: U: b! N2 z# J4 J) X
<value>0.0.0.0:8049</value>7 X7 S" ?2 k% e) U! w
</property>
$ `9 m; v T; u4 l8 K4 J6 Q <property>) Q/ \# R6 D4 E
<description>2 V6 j% A1 |) H2 _, K Z, ]; W
The number of threads used to handle requests by the AMRMProxyService.
/ W9 A, |9 d9 V </description>
2 U, B1 V q* n( o1 l" ] <name>yarn.nodemanager.amrmproxy.client.thread-count</name>6 J9 G1 \: O0 T: K* V& [
<value>25</value>) J9 m S( |& a4 z7 H9 H
</property>
: X& B) `* K ^+ }/ \2 A( \ <property>( K' T# ~% J) V% l
<description>, P( C' I; I3 L$ V3 z1 f# A
The comma separated list of class names that implement the9 ]) W' D9 q6 y) w
RequestInterceptor interface. This is used by the AMRMProxyService to create" I( r" t! J: J: W# |: _0 i
the request processing pipeline for applications.
6 J$ B' k' {3 P1 f2 y4 X9 V </description>
+ Z% O3 u# [! A$ P8 b <name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>1 k/ y0 U- W: R: K. X. l$ f
<value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor</value>
# n* {( D1 R( w& I+ u3 j* g8 U+ G </property>
0 }/ K* Y8 E7 @' Z h" N/ L8 M+ m <property>
/ h( Z) ^2 w8 F& B6 H" u5 G <description>
/ C3 a- s" e* o; R6 U* |6 C Whether AMRMProxy HA is enabled.
( Q$ P+ v: e& P- Q) I; C7 t </description>
5 K; k( j+ u" N4 d J <name>yarn.nodemanager.amrmproxy.ha.enable</name>$ M' A6 g- Q) b m# h) f/ n: w
<value>false</value>7 C5 o/ X$ H- S: F, w
</property>) \" y, v8 e2 l( O# ~
<property>
; g8 I4 h @1 V- G: K) o$ w <description>
7 O+ `) A0 Z' i! L/ [2 B Setting that controls whether distributed scheduling is enabled.2 @- S% G/ _! V
</description> _' k, k# w3 J N0 T5 g
<name>yarn.nodemanager.distributed-scheduling.enabled</name># Z- F$ Y5 T5 T" e, O/ B
<value>false</value>8 {, s0 w5 F$ ~
</property>$ ^3 t; l+ z p9 X
<property>
& e5 F* k& n/ ~ <description>$ S7 B3 E3 o- n( R4 v
Setting that controls whether opportunistic container allocation! q) V* x0 u7 S4 ~ X6 {# N
is enabled.
1 E1 B+ v2 H1 N# ~) j7 _6 G8 r </description>
! _/ V: s3 C3 }( x <name>yarn.resourcemanager.opportunistic-container-allocation.enabled</name>, _! N9 O8 T! w5 a8 o
<value>false</value>! u% d% z3 z& ?" |
</property>
) z1 M! F6 U$ g* O: s" k <property>
& o' w1 `# I3 r% X9 r <description>/ {5 P) U- Y& E8 |+ u9 q
Number of nodes to be used by the Opportunistic Container Allocator for
2 R: L" b; S9 ?' l dispatching containers during container allocation.& E) |5 b8 z2 ~# w3 l( V* C
</description>
2 V& e- Y8 {0 _( Y <name>yarn.resourcemanager.opportunistic-container-allocation.nodes-used</name>
% c& @- o& I* T. y! l <value>10</value>" w, ^# x4 l+ Z, K2 p
</property>
; w$ H/ b) u" F4 H' Z+ g <property>, @) l8 V8 K3 J9 d `0 k
<description>" M$ n( |& w. B+ S
Frequency for computing least loaded NMs.
* S t5 i: S5 C& }* M/ {+ e </description>
5 J5 D! q' d2 O! h4 [1 J <name>yarn.resourcemanager.nm-container-queuing.sorting-nodes-interval-ms</name>) q/ ~7 Q" S. [9 ^* i8 ~& S8 d% t
<value>1000</value>
- U) i. ~3 D5 Y </property>
/ z9 i3 c7 e: f <property>- \( g1 `' k8 J# c
<description>! t# `0 ?7 E* F0 f# W" i8 J
Comparator for determining node load for Distributed Scheduling.
7 M6 J* @' H0 C, X( z0 P A6 w. s </description>9 @! ^8 `, C' N3 }% n7 ?0 M
<name>yarn.resourcemanager.nm-container-queuing.load-comparator</name>+ A, ] T6 R/ ]# J, S! G
<value>QUEUE_LENGTH</value>
+ L A, |6 A8 {, C% h </property>
* I# U9 g! p; F7 B7 U/ ~ <property>! q$ y' z' W) y+ [6 {
<description>7 c1 L9 J+ F4 Q6 w4 y' M; a
Value of standard deviation used for calculation of queue limit thresholds.( ^" @& P% S2 `. U
</description>
/ Z) v) s4 }1 s <name>yarn.resourcemanager.nm-container-queuing.queue-limit-stdev</name>, s0 P& K Z) r7 w2 Q
<value>1.0f</value>. |- r; k0 T1 L1 b* ^( V; ~
</property>
- \, [# S5 j5 g3 S <property>
_# }' @* q7 ]$ A2 {( ~1 T8 G# ? <description>
% Q& k: ]2 d* G8 j7 Z0 } g9 r Min length of container queue at NodeManager.
! B' w, C5 {' i. Y& S </description>
4 B5 w# g0 E; U: g8 a) ~! s/ ?, y. l <name>yarn.resourcemanager.nm-container-queuing.min-queue-length</name>
" ^& D1 R: S2 d) C1 G1 ^! f2 t <value>5</value>' `4 }# i7 I( A" W
</property>: S5 K' k% ~1 B* m$ {( L( D7 I
<property>* b, U; \2 Z3 O3 }4 O7 A% J
<description>+ c y" U8 ~( s% \* g* X% a4 Y
Max length of container queue at NodeManager.
7 p+ g! o! z" e& N% T </description>+ @% p* t6 j1 F) }
<name>yarn.resourcemanager.nm-container-queuing.max-queue-length</name>
6 o. m# \6 ~. o/ n# @ <value>15</value>
! X: _) C% V$ |, E) z </property>8 M6 Y p7 U- x9 S% o
<property>
' q8 [5 h% d' j. h; P* G. @ <description>: [/ c( D: f! n( X0 D; T" S2 J5 X
Min queue wait time for a container at a NodeManager.' a; C9 k$ f1 `' Z6 |2 o2 h
</description># ^5 h# m7 x; ~ M. ^ o2 w$ a
<name>yarn.resourcemanager.nm-container-queuing.min-queue-wait-time-ms</name>% x: x8 h8 W+ W' N
<value>10</value>' e8 n4 Y7 i- l' s/ I
</property>! r$ }+ @7 h- O- ]5 G5 ?
<property>/ G9 _3 e: m3 Y2 Y( z4 z
<description>$ W! u' l3 z5 W7 W* V. w
Max queue wait time for a container queue at a NodeManager.+ W( y) N0 s8 q, b, s1 H
</description>
1 L; [% W1 Z/ o0 Z1 d; ` <name>yarn.resourcemanager.nm-container-queuing.max-queue-wait-time-ms</name>
# G0 L) |1 L* ~8 {2 x+ a1 a <value>100</value>* D1 V P4 R2 U& ^7 [3 ?, z
</property>3 K, ^$ r+ y& u) n) ^2 [
<property>
, j# m* y" E+ P. q( c <description>$ m' V4 J) l2 p/ X* E- R. A
Use container pause as the preemption policy over kill in the container
$ L/ z6 Y4 g' {- ^0 M3 Q queue at a NodeManager.1 E. I, \% O7 z) {8 P
</description>, }" x% N) W" M. }$ f
<name>yarn.nodemanager.opportunistic-containers-use-pause-for-preemption</name>1 B6 k! M3 B5 [9 p
<value>false</value>
" a/ s; ] K+ _% d4 l/ a8 t4 T </property>9 ?7 L! Y) ~ j: L7 E5 v1 |! q( D
<property>
6 P2 J# ^' A& o7 S: k <description>
/ x0 C' J$ L$ D! G5 `0 I Error filename pattern, to identify the file in the container's
w' V, S2 w( ^1 v1 F9 F) g6 D Log directory which contain the container's error log. As error file
, H5 c; L$ R( Z4 `6 a redirection is done by client/AM and yarn will not be aware of the error
8 F$ | Q# e5 b' R- I) o* O# n file name. YARN uses this pattern to identify the error file and tail
& P) A- N' ?' i- v8 A the error log as diagnostics when the container execution returns non zero! Z5 i4 c7 g7 T2 ~ b9 h
value. Filename patterns are case sensitive and should match the- ~) A7 {- m/ P/ m7 P; r3 p; }
specifications of FileSystem.globStatus(Path) api. If multiple filenames
1 e% `3 G- v/ s* X matches the pattern, first file matching the pattern will be picked.+ X' z6 B+ v; o) T% k# C; b
</description>! `1 @7 {- ^% A$ H& d
<name>yarn.nodemanager.container.stderr.pattern</name>
! x5 p5 ~, b& l+ K0 Q$ z: V9 o/ P <value>{*stderr*,*STDERR*}</value>
# m0 Q( h( w1 Q4 X0 P2 I </property>- y( t2 n4 z& O( ~
<property>
; b* r. f: P6 [/ w, e2 k <description>
0 j1 b- |# i7 d0 E3 i% i Size of the container error file which needs to be tailed, in bytes.
! B* p* K) i, _6 A, G" [ </description>
2 ~2 \$ a6 Q& I1 `0 v9 V. E+ ` <name>yarn.nodemanager.container.stderr.tail.bytes </name>& ?) C% u/ Z5 t$ p
<value>4096</value>( b+ }9 B& u4 g O9 ^
</property>
2 g0 ?8 D9 l9 [& B. ] <property> d: T$ B6 L; e% M/ B! o
<description> M( `# u& l/ C: j
Choose different implementation of node label's storage
) A4 m1 F4 E; X. ] </description>0 }0 q9 U8 U5 ?4 O
<name>yarn.node-labels.fs-store.impl.class</name>' O, X# A' F5 T" d5 G4 E
<value>org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore</value># }% O6 M9 S6 L L& V U8 t; I
</property>
/ E+ d0 W; K4 N1 h( N <property>
6 B( q; e1 V1 {6 X <description>
! H' @ H! h2 u1 s, | I Enable the CSRF filter for the RM web app
2 m! Y* |- l6 }: z </description>
: m) d- u: _" h4 w; z <name>yarn.resourcemanager.webapp.rest-csrf.enabled</name>
% S! Q7 @- a7 C+ x! j6 I! M! t <value>false</value>' \5 ]2 p2 o. X( s& V; \- D; @
</property>
/ y; G4 n3 K. j1 a0 A <property>
1 m% t4 C' E- K <description>' S. y0 d: l N* ?4 C* k7 B
Optional parameter that indicates the custom header name to use for CSRF
8 l* H5 X5 O. m2 k protection.
* n& }& q2 y# i- t Q/ | </description>; @) b$ Z; w a
<name>yarn.resourcemanager.webapp.rest-csrf.custom-header</name>
4 X6 t1 O" h7 M7 j$ L <value>X-XSRF-Header</value>
% g& H! l- R1 m. L </property>" [( V4 ]; u) C0 @2 x% m
<property>0 i$ c+ \; t3 S
<description>
4 \4 }) j. x/ O6 C* c9 s Optional parameter that indicates the list of HTTP methods that do not6 R( _, z8 V7 Q- r! z
require CSRF protection: |( ^& X$ l4 l" |0 Q: [
</description>( u( _: z% M! C, A! t9 W; Z4 h* e W
<name>yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore</name>& O7 S% t7 E4 e! V
<value>GET,OPTIONS,HEAD</value>
& q3 C7 ^; A) j* U8 ?; M' t </property>
2 M4 H4 k5 a7 N3 @& }! @0 A: _ <property>, C" f: d0 d4 Q7 c+ f7 B2 T1 r
<description>! G6 [* z5 |% r3 Y9 l5 O& y
Enable the CSRF filter for the NM web app/ V. _) c' R3 c, p7 a7 h2 d
</description>. l/ W% x ?' L8 V S
<name>yarn.nodemanager.webapp.rest-csrf.enabled</name>3 K6 ^( L/ m5 k9 G/ g5 {! T' l
<value>false</value>
* u& K- B/ B. Q1 c </property>1 W" T* }/ \ a. p3 ?; h
<property>1 Q2 C/ w2 c* k' y- @ b) K( D
<description>1 S) G& k8 ~! Y1 `0 k
Optional parameter that indicates the custom header name to use for CSRF
" Q/ l* J& |; E5 }+ O( [6 C protection.
& }+ d3 a$ |3 _ </description>$ A, B1 I3 |- c) H1 v) ]0 ^
<name>yarn.nodemanager.webapp.rest-csrf.custom-header</name>
8 E$ \- T0 T8 n: i <value>X-XSRF-Header</value>3 {0 x8 o: s4 J, T; h# Q6 C
</property>9 c3 P! ~& p3 |' v. w0 {
<property>
% l$ H* ^0 A4 D) G) W$ R <description>2 b' c( ]/ O- W3 P9 M
Optional parameter that indicates the list of HTTP methods that do not
$ A- m0 H4 r% w) y! S& O- l) @ require CSRF protection
# \ e) D2 A$ h) t' U8 a! o </description>0 I- W! O1 f! f0 g2 M
<name>yarn.nodemanager.webapp.rest-csrf.methods-to-ignore</name>
% Q5 j: z! a t, B8 r8 g; u <value>GET,OPTIONS,HEAD</value>
$ |+ ^6 b b" M$ _9 m, X" ?# r </property>- F) n' i7 l6 X6 i7 [
<property>
4 Y( E4 _% k3 C/ D# t9 e& ^6 F% t <description>
7 n$ Y+ X/ f, }' J% y The name of disk validator.; W2 S: H2 {* l/ {5 C* C+ G
</description>8 q: K6 g; q+ Q
<name>yarn.nodemanager.disk-validator</name>
& I8 W1 O/ d/ l; P4 h <value>basic</value>4 }4 y. m5 E; j9 a/ X! c
</property>, c! M* b$ a0 U( l
<property>: R( r% O$ ~. G0 c+ f5 O9 [
<description>) Z- {2 v& \$ `! w
Enable the CSRF filter for the timeline service web app
! A6 @% x$ I8 V. f </description>
: B, [* D( T8 M <name>yarn.timeline-service.webapp.rest-csrf.enabled</name>/ ^/ |# y4 H, D: m' h6 ~# g
<value>false</value>
3 K: H4 t% l6 _) J7 \ p5 m </property>& ?; E* r, F7 N" b: T7 M
<property>
+ Y8 G9 E9 A# ~8 D8 R+ v5 m& R7 Z <description>
7 Q; V0 c0 k E Optional parameter that indicates the custom header name to use for CSRF
, }/ s+ J; S e9 h protection.
; q, J: D+ f e* o </description>
! W' u& y# W5 j <name>yarn.timeline-service.webapp.rest-csrf.custom-header</name>1 g Y# R# z: L6 j5 u$ E
<value>X-XSRF-Header</value>
( s$ w5 k; d2 `! o0 f) f v7 z2 w </property>
& w" Q# X/ Z% a <property>& i( t+ N0 x* D2 K1 M
<description>, m! {2 D- Y6 V
Optional parameter that indicates the list of HTTP methods that do not
7 \; A9 y9 g7 P- Q) r require CSRF protection" i& G) R S1 n% G1 N
</description># |: t* S% U0 o4 d \/ _3 Z4 W0 J
<name>yarn.timeline-service.webapp.rest-csrf.methods-to-ignore</name>
$ |% p! L2 p6 L* e" E! M <value>GET,OPTIONS,HEAD</value> M- M& E8 F8 M! y4 }' U
</property>
& p/ m) ?7 r* |, x( ?* D" q4 A <property>
" @" E8 B( x B: Q2 Y+ s <description>7 S% [: f. m9 [5 [8 d
Enable the XFS filter for YARN
0 i d9 t4 V8 V; ` </description>/ F/ @9 y6 E# }" E! y
<name>yarn.webapp.xfs-filter.enabled</name>
2 D4 o& f: ]0 Y1 ` <value>true</value>
, k T5 y2 K5 A0 T0 T6 T5 @ </property>; M% X; h9 t) b! v4 p
<property>% Q* W6 M `* y7 y& j: x
<description>
3 L. D& E g- ^+ m Property specifying the xframe options value.
: _0 I0 s- P+ v: B </description>
0 ?! Y, w1 F5 [0 v5 [- x <name>yarn.resourcemanager.webapp.xfs-filter.xframe-options</name>
+ B# x" B2 b$ ], |8 B <value>SAMEORIGIN</value>
0 e# m; ^6 Z# q </property>0 t/ h1 D/ Y( v! G% E' ?
<property>, P& D2 k9 z" l% `) f, H9 w
<description>
& b4 W5 u5 A) t& v! V+ N$ Q0 W- s Property specifying the xframe options value.
* a" _% }0 a% B4 i: J3 ~) U- ] </description>+ F$ W3 D& }- H
<name>yarn.nodemanager.webapp.xfs-filter.xframe-options</name>
- j' s3 {% L/ B/ Z' U <value>SAMEORIGIN</value>" D: e) Q# d% A+ ?
</property>
, R# Q4 g/ W( |/ Z# a( O <property>
- b+ |7 g: J$ U7 L3 \- _0 R9 Q( p+ b <description>% \1 F. F- Y. S0 I- j- M# @
Property specifying the xframe options value.4 f( C6 E( r1 O+ ~
</description>% P6 ]! n; d. c2 r8 V5 c Z! {
<name>yarn.timeline-service.webapp.xfs-filter.xframe-options</name>& B/ ^& M6 H$ _8 _' k2 @5 M0 _
<value>SAMEORIGIN</value>9 L! |% o$ y( f" N, A/ z; B) W+ g3 `
</property>$ u# ` ], n f! m5 S2 [
<property>! U$ W& Z% x6 }$ Y6 Q
<description>
) L9 k' ?' I# k7 n7 W( o The least amount of time(msec.) an inactive (decommissioned or shutdown) node can
! U5 `; u/ H, O stay in the nodes list of the resourcemanager after being declared untracked.
e, i$ t" v+ d" P6 b! ^ A node is marked untracked if and only if it is absent from both include and
2 [6 O" e8 n! t5 [, `7 I. K% i' D# M exclude nodemanager lists on the RM. All inactive nodes are checked twice per# }& X7 O( v' I9 N
timeout interval or every 10 minutes, whichever is lesser, and marked appropriately.
' ?8 |/ Y) u2 ^* y) f* } The same is done when refreshNodes command (graceful or otherwise) is invoked.
N1 a8 C; N% R2 L </description>$ p, N1 G. L# o: F" o& N
<name>yarn.resourcemanager.node-removal-untracked.timeout-ms</name>" B, Z8 j, Y! |2 ]/ e7 T
<value>60000</value>7 }# d s2 i2 N/ \+ e$ P7 g
</property>
& J7 d8 g `- `9 O0 a <property>- m/ @& p: l2 {1 D# S3 f5 x& ~ d
<description>
0 `0 T" h, Z3 H; w& x% g+ { The RMAppLifetimeMonitor Service uses this value as monitor interval2 K( E6 y7 x8 P* @7 c
</description>/ ?* Z% j& i: c8 Q* d
<name>yarn.resourcemanager.application-timeouts.monitor.interval-ms</name>) q' V8 q" V. B: L5 r: c5 V& \, z
<value>3000</value>
" T l+ M2 O% L </property>
' ~1 K Z& ^/ V0 \+ a6 v <property>8 N$ J5 k) F* o2 v% Q1 S
<description>
& L3 d4 F1 i9 p9 F0 ] Defines the limit of the diagnostics message of an application
/ d% p- U' l* I3 c/ I- h- H% L attempt, in kilo characters (character count * 1024).0 T, R2 ]9 ?( _- a5 m6 _
When using ZooKeeper to store application state behavior, it's
, |6 b3 M7 i& k( u+ s important to limit the size of the diagnostic messages to5 L. Z- s) b) p) I4 v2 h# [" x
prevent YARN from overwhelming ZooKeeper. In cases where! O8 { y# k1 @( t
yarn.resourcemanager.state-store.max-completed-applications is set to
F4 J8 I$ n- A% D) u! ]" h) N. L a large number, it may be desirable to reduce the value of this property
9 c5 m' G! y) K/ U to limit the total data stored.
1 L* H7 M/ F; c7 @! B2 p6 w </description>9 J/ q3 `) b# a6 J9 u5 Z% A
<name>yarn.app.attempt.diagnostics.limit.kc</name>
! }/ x! _9 B, E- h$ U8 C <value>64</value>+ |1 N' Z) N5 k2 r1 t* U% N
</property>
. `8 F# `0 p/ s- Z <property>
/ p. \# I0 _7 l: A <description>
; o1 {) k; e( r; e; O1 J1 u Flag to enable cross-origin (CORS) support for timeline service v1.x or5 h6 {9 `1 m! N k1 W2 f F
Timeline Reader in timeline service v2. For timeline service v2, also add3 r" o$ g" U& X
org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the6 e. _) Y! v; i7 L& L5 ]
configuration hadoop.http.filter.initializers in core-site.xml.6 ]' k7 R7 Y; Z# x' I H9 n
</description>
+ `- T w0 `; q1 n, y& p) y2 e <name>yarn.timeline-service.http-cross-origin.enabled</name>- u( A1 b* q. w8 O6 |
<value>false</value>
: B/ @$ B; W2 }" u+ }% u </property>/ B0 ]& a5 q( f& B4 f2 C0 B
<property>
- x+ x7 x& D: }* M# o <description>
' g, H& h0 @- y5 \% @ Flag to enable cross-origin (CORS) support for timeline service v1.x or* O. |4 E- A8 K( g4 {* |+ ]
Timeline Reader in timeline service v2. For timeline service v2, also add
8 k9 A0 j. c; U& b# a4 U6 ` org.apache.hadoop.security.HttpCrossOriginFilterInitializer to the
1 A: w9 y5 b* d configuration hadoop.http.filter.initializers in core-site.xml.. e, T6 E# ^) h" k5 b ~; a
</description>3 l5 c( F6 ]' K% j6 J0 ~/ X
<name>yarn.timeline-service.http-cross-origin.enabled</name> V* Z7 ]8 U% d; N* P( O& v
<value>false</value>5 [- u( z$ J5 v, [% v0 {& a# T
</property># z5 @5 {. o! I+ F" g( A/ s
<property>
! ~) s; Q$ @; V' V# L5 A9 [ <description># g( m8 w. D d" Z$ r4 Y
The comma separated list of class names that implement the6 f. G+ F. O- u. ?6 D3 z
RequestInterceptor interface. This is used by the RouterClientRMService$ V0 H3 ^7 o7 n, b3 D3 U3 {
to create the request processing pipeline for users.: _8 `7 I& a8 V3 }; b. ^# I4 Q
</description>9 F v& P* N; D) P( V
<name>yarn.router.clientrm.interceptor-class.pipeline</name>
6 [6 _# Q% k' b* n <value>org.apache.hadoop.yarn.server.router.clientrm.DefaultClientRequestInterceptor</value>
# E! Y/ Y! D, I </property>8 K- `( s; G5 S }4 Z
<property>0 P% x, @4 _& z$ k* b7 k
<description>
& _1 u2 M. u) \% _# x" |/ ^ Size of LRU cache for Router ClientRM Service and RMAdmin Service.) f4 f* ~5 T- `/ a; E9 r. |
</description>* l1 P. E# |: e; x# p% ]% s
<name>yarn.router.pipeline.cache-max-size</name>5 ^7 `& n$ w$ P5 Y7 \- e6 y
<value>25</value>9 x$ T$ x+ h! p
</property>& l1 E- E2 j) k1 L' e5 x+ z
<property>1 P9 `/ @2 E/ {/ q7 A# }
<description>
6 x3 l9 q. A, y) j. ` The comma separated list of class names that implement the
3 v1 y+ b _7 [# t RequestInterceptor interface. This is used by the RouterRMAdminService B& b$ Z+ |% U, i c
to create the request processing pipeline for users./ c" T3 g( z8 z" P- L
</description>
$ j3 e( X( S) M5 X$ A9 u <name>yarn.router.rmadmin.interceptor-class.pipeline</name>
. t( d# L, N) N( s! J <value>org.apache.hadoop.yarn.server.router.rmadmin.DefaultRMAdminRequestInterceptor</value>; _1 b. K1 S. ^; [0 P
</property>
4 B _& w( l. u. A' @ <property>; d9 N2 l9 r( ]9 E0 M, R
<description>
& Y# \8 C- E+ k5 ~! R: t: I The actual address the server will bind to. If this optional address is
! b5 d) ~( v2 e3 [" Z% t8 l set, the RPC and webapp servers will bind to this address and the port specified in& P, e: n3 [. F: a7 E* P- J
yarn.router.address and yarn.router.webapp.address, respectively. This is( J* r, ]/ ~2 X# u' y# k: J- s
most useful for making Router listen to all interfaces by setting to 0.0.0.0.
) m0 n8 w- i4 Z6 W# A4 D3 p7 k) J </description>
( l6 H% ^1 M% y9 }) K1 ^. m <name>yarn.router.bind-host</name>
) e" Z" y" h9 s, N$ L" u* b1 l <value></value>) b! Q* x# f: I0 Y$ X, A. E; w! n
</property>
0 U/ p% ?* T0 v; K7 Y- Y <property>9 s6 S, }2 Q; h
<description>
- V" y. Y. H% R `- ` Comma-separated list of PlacementRules to determine how applications' L6 l! @+ C1 N2 \$ f
submitted by certain users get mapped to certain queues. Default is
' W' E0 V$ A5 _* B( r2 q: U user-group, which corresponds to UserGroupMappingPlacementRule.
, O/ H7 q; ]! G6 A! |) M b. i </description>8 p6 G$ i# d: j$ R2 K$ d
<name>yarn.scheduler.queue-placement-rules</name>! j. E+ W+ q4 y8 I) y6 y
<value>user-group</value>2 X! @: o# X" Z: [8 g# k
</property>' Z3 p: r4 B" ]5 d+ b4 t
<property>/ c% u+ ~3 H7 s9 l+ Y" t0 l5 _
<description>
- B, K5 t* I! g) @8 ~6 J" D: d The comma separated list of class names that implement the
s. k7 w4 v+ q4 w" ~ RequestInterceptor interface. This is used by the RouterWebServices
. ~) ?3 P9 b( s" X+ g o to create the request processing pipeline for users.5 H$ T" P0 B; N" Y
</description>9 _' {" i# [7 k0 l
<name>yarn.router.webapp.interceptor-class.pipeline</name>
( a- k2 ^! }% N9 P o <value>org.apache.hadoop.yarn.server.router.webapp.DefaultRequestInterceptorREST</value>4 n) K6 E( K/ q4 |
</property>) P: m1 \% a8 _5 E- i( E
<property>
/ k% y5 Y: b6 l. C; e <description>
8 } l K: {% Z+ W The http address of the Router web application.
$ {5 m3 O: C& L) Z2 {9 ~ If only a host is provided as the value,
0 D2 X/ O2 V6 ^3 R5 u6 n the webapp will be served on a random port.
8 @) x8 O$ M* E P# ?) n* b </description>+ |3 v$ y3 f" a0 u! m3 a! K
<name>yarn.router.webapp.address</name>& I! J( }7 G' S/ y' d& P
<value>0.0.0.0:8089</value>; ?% a" p# ^! t
</property> I) O8 ^5 U9 D
<property># | \2 B2 z( R5 j
<description>
3 F/ u$ x$ n8 a, t3 R2 j' b! M2 E The https address of the Router web application.
' y3 M( F1 `2 n4 ^7 q' o If only a host is provided as the value,4 m' h4 m5 l2 ?3 M
the webapp will be served on a random port.
! k& d G% M0 L1 D </description>& }# e7 m7 S* y1 @+ j I
<name> yarn.router.webapp.https.address</name>! W+ s. Y B" L4 {2 i% l3 |7 }; U8 z
<value>0.0.0.0:8091</value>
1 g7 V0 r3 Z$ H3 q" k </property>
( g5 U. L! t9 v2 f <property>+ C9 H: D6 N$ T% M- b
<description>
0 x/ u: z, y5 o- O) Z. p: c It is TimelineClient 1.5 configuration whether to store active- Y, {; j6 Z3 @; X3 \' _& C
application’s timeline data with in user directory i.e" X3 T0 q5 l9 |2 |/ C
${yarn.timeline-service.entity-group-fs-store.active-dir}/${user.name}) Z2 {* m1 R, Y% p
</description>
+ R P4 Y- Y# p+ N8 N1 Z <name>yarn.timeline-service.entity-group-fs-store.with-user-dir</name>
& x+ y1 L, k* x# x+ G <value>false</value>. f5 U7 F! B( |* x5 s a
</property>0 r* x. w9 O4 H1 H
<!-- resource types configuration -->7 j3 H7 y! e5 ~: `
<property>2 u8 |+ K6 |7 b$ i
<name>yarn.resource-types</name>
0 N6 u, Z- ~: V+ b. L5 j% \ <value></value>2 b; Z9 c% }4 j
<description>* x) o1 D( j3 m9 {
The resource types to be used for scheduling. Use resource-types.xml2 }* q7 q/ I3 c- t
to specify details about the individual resource types.. _1 y0 f( l8 l* b
</description>
! Y2 ?! U1 |: Y' u2 U/ f9 `, ] </property>
) o" o' c9 x$ w) v+ ? <property>& K, m3 H4 C& h* q1 t
<name>yarn.webapp.filter-entity-list-by-user</name>+ E0 J. R6 c8 k; B! m
<value>false</value>: |" n7 j3 H1 ^) g+ ~
<description>
0 b9 j5 }9 r* Z Flag to enable display of applications per user as an admin9 a! L4 K) j% S) h6 O% ^
configuration.
4 U! u6 }, d+ {7 U </description>7 ~! w }; m8 K7 D0 I3 i
</property>
, S4 Y. j! n4 |5 I2 E8 t3 Z5 r; s <property># H% z& F) D N
<description>
" T3 O3 J, w. p/ m' f+ D The type of configuration store to use for scheduler configurations.
8 T2 \0 L! r: y( [: K; g- V Default is "file", which uses file based capacity-scheduler.xml to: b& @8 Y& c1 z5 S# O
retrieve and change scheduler configuration. To enable API based5 P) V$ B8 G q' h9 e
scheduler configuration, use either "memory" (in memory storage, no: e) N, v5 C2 I, K t; i
persistence across restarts), "leveldb" (leveldb based storage), or
* @6 m7 `3 y# M: r# w "zk" (zookeeper based storage). API based configuration is only useful6 x% A# ~! V5 S+ Q
when using a scheduler which supports mutable configuration. Currently. m" b, P# l& c3 H/ `0 y
only capacity scheduler supports this.! n8 P* ^) i6 f# P/ x2 w3 w
</description>1 |) ~6 u* O2 D/ o. h2 L- e
<name>yarn.scheduler.configuration.store.class</name>
. h* q) D9 N* }6 ]' k <value>file</value>5 N0 t# N3 D0 H0 i9 C& M5 F { }
</property>+ q; x% I% Y' \: z5 Y. M* c) q3 m
<property>
* M- L* `% T/ O! e0 d! c <description>% e0 w" C2 X* U! s
The class to use for configuration mutation ACL policy if using a mutable* G" M3 n' T* _( Z+ ~- \% b
configuration provider. Controls whether a mutation request is allowed.
" h7 K- `- ^9 g* v k- F The DefaultConfigurationMutationACLPolicy checks if the requestor is a6 I0 n5 p" }: Z8 a5 [
YARN admin.
7 Y, W$ w( z- `. I3 g! @% }0 L </description>
}/ Y- K: g1 H: ~$ e2 ~ <name>yarn.scheduler.configuration.mutation.acl-policy.class</name>
+ u5 Y; |9 ~) i R8 ^ X: B5 z <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy</value>
7 G5 L, ?; k' ]; [1 w% L </property>
9 R8 b, _. i8 o( L- ?9 [ <property>, |) `$ @. L/ ] G. P, O: u; [' |
<description>+ D: c3 J: s q
The storage path for LevelDB implementation of configuration store,
@& z! [, h0 @% ^ when yarn.scheduler.configuration.store.class is configured to be
3 s+ }5 ~' G/ T! r/ |6 S/ O+ |4 b1 ^ "leveldb".8 f2 i1 P8 g: V- N4 S
</description>. r# e; O0 a9 l0 y+ `
<name>yarn.scheduler.configuration.leveldb-store.path</name>
6 u" `, g( `( s9 S) C <value>${hadoop.tmp.dir}/yarn/system/confstore</value>
) d* q" p2 r8 {$ [: V </property>
# J/ A1 a6 W/ v; c <property>, b$ e; A" r9 x& K
<description>$ O+ s! W8 f. s* T. x0 L( [
The compaction interval for LevelDB configuration store in secs,
2 A% m9 K% r. B/ z- @# H when yarn.scheduler.configuration.store.class is configured to be' R1 `- @" h: N
"leveldb". Default is one day.
8 a! z6 ^, ?% ?0 ?( O6 A C( @ </description>
0 J$ R! ?/ L1 g: G$ ?/ a1 L <name>yarn.scheduler.configuration.leveldb-store.compaction-interval-secs</name>1 a: d# Z8 K( ]+ ]/ P) b9 E
<value>86400</value>
$ Y6 N, N9 Z+ `6 M& B$ q </property>* I/ O8 P. T( |4 ?! n+ Z u
<property>5 g2 G: ?- S1 r/ t5 e! o8 f
<description>' w3 z* A9 w \3 }
The max number of configuration change log entries kept in config4 ] B% K) e" k: g. {+ ~$ S; ~
store, when yarn.scheduler.configuration.store.class is configured to be/ S& ]5 E) m9 N$ M- b
"leveldb" or "zk". Default is 1000 for either.% R$ o0 Q4 U5 B6 p7 k7 A2 G
</description>- L; B$ m% G* ~+ B8 n
<name>yarn.scheduler.configuration.store.max-logs</name>; e, d' }7 p9 o/ q2 C
<value>1000</value>0 |- l9 X+ W4 D' t! c, K
</property>
) t- a3 j5 ~* Y! u! ] <property>
7 V$ Z5 I7 x' N2 F* `! H9 L" e, F+ B <description>7 M g9 l7 e: x- c4 N+ ?4 D. [
ZK root node path for configuration store when using zookeeper-based! W! t( R7 P- c$ i. S
configuration store.8 z& ~+ R- K5 F2 E0 C
</description>
+ b7 I9 @! H5 e <name>yarn.scheduler.configuration.zk-store.parent-path</name>& W/ q+ s5 B2 X' H. D$ W1 d+ E/ ~3 S3 W
<value>/confstore</value>
+ c" s8 d3 w* [6 T% m6 t6 z3 ? </property>8 {, {' E# d5 ]" }; @
<property>( P5 S# A* Q4 \/ l7 n7 y
<description>: h! U# ?/ A5 h* v& l- i
Provides an option for client to load supported resource types from RM: w; L% R! M$ s
instead of depending on local resource-types.xml file.3 `7 m# f# U& Y1 c7 D
</description>
1 m: J/ f. r& n- ?% ~- C ^6 ~ <name>yarn.client.load.resource-types.from-server</name>
2 b3 a, _$ W& s0 M' d8 @# ?8 c$ d <value>false</value>8 f% g! U; m) L: z6 u
</property>
. f. u2 S8 d1 f7 P' m" k! { <property>4 q+ s& E* F8 @& s& Q* X, T# q
<description>
3 s! g$ j" e' e( m- H5 i0 ? When yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto specified,
9 r9 W" ~2 d' R$ b YARN NodeManager needs to run GPU discovery binary (now only support, Q( y4 d1 l* Q
nvidia-smi) to get GPU-related information.
1 }0 r8 K" A/ H/ v* {& f; q! a When value is empty (default), YARN NodeManager will try to locate' }9 F' O- i# `( `% V( J( u. S" z
discovery executable itself. G- W9 U4 _$ L% }9 Y' v
An example of the config value is: /usr/local/bin/nvidia-smi7 g9 |- P6 k% i3 A' o- ~
</description>. C6 b9 n, L6 F
<name>yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables</name>) J% x: m& y, v) q; u; j
<value></value>8 S9 s5 m0 T! E% ^9 M! r+ V7 l
</property>
9 P, \5 o) Z' x( G8 x! ?( ^- c <property>; X2 y G$ z$ s& x, e# h$ b
<description>/ t K; I; i0 Q( A8 e3 b, C
Enable additional discovery/isolation of resources on the NodeManager,* G6 F3 j$ I. U- `5 |: H
split by comma. By default, this is empty.
2 ]% v y2 J5 d! ?/ R Acceptable values: { "yarn-io/gpu", "yarn-io/fpga"}.
, y' Z6 b, w3 E( f. d: E4 I s3 s </description>
. R+ s5 @1 u4 U; I) a1 ~7 P" v$ | <name>yarn.nodemanager.resource-plugins</name>. }0 b2 f1 |3 c( b
<value></value>2 S' P7 H- m V1 b" Y
</property>
/ k* v v% U& U+ @ C, p <property>& s" @/ `* G! z" m3 d; @- ^) Q' f
<description>
* m1 U8 X# k# Z8 N* y& I* o Specify GPU devices which can be managed by YARN NodeManager, split by comma1 }- ?7 ^( B5 b( o+ Q6 d+ q4 u& e' v
Number of GPU devices will be reported to RM to make scheduling decisions.
1 q- r: }' b) O' i0 B Set to auto (default) let YARN automatically discover GPU resource from
) U3 {9 S% x, Z& i! T8 ?( } system.
2 L2 ]& B1 c9 w& v; v Manually specify GPU devices if auto detect GPU device failed or admin
q" ~* E) }) k& h1 Z only want subset of GPU devices managed by YARN. GPU device is identified! s( F6 X& P, ~0 ~1 K: R, k
by their minor device number and index. A common approach to get minor
& j+ g+ o9 i9 q( U3 o1 a. A9 G device number of GPUs is using "nvidia-smi -q" and search "Minor Number"
% J7 F; o! \6 ~- l, K k' J2 B output.
4 g: C! D2 @1 W/ Y% Z When manual specify minor numbers, admin needs to include indice of GPUs
, w8 t/ n% A2 A: ~0 O0 a" b as well, format is index:minor_number[,index:minor_number...]. An example
7 C3 S6 n- [; @( O- n; S, Z: C" a0 S of manual specification is "0:0,1:1,2:2,3:4" to allow YARN NodeManager to
- V+ g m$ @+ E1 G manage GPU devices with indice 0/1/2/3 and minor number 0/1/2/4.* v- d8 M- s3 L; |6 ?7 O
numbers .7 N- B Y. N$ F: M- s5 h, B
</description>
r3 Q; {8 b" v" B) g <name>yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices</name>
" t2 S a4 `" E' d( O, D& ]2 N/ }" v <value>auto</value>* E! [6 n9 W' t/ W- G( i" a
</property>3 O. c( N' ?* V& @* p4 b
<property>
5 b7 h3 n( Z' S3 G; F% h/ | <description>4 a4 b Z9 H4 E$ }5 _3 u+ i
Specify docker command plugin for GPU. By default uses Nvidia docker V1.
) S, r# B3 A0 l; b </description>
$ E( B# Q" e5 r0 q, E& V) C <name>yarn.nodemanager.resource-plugins.gpu.docker-plugin</name>
! t/ ?$ m4 L1 | C' v) d <value>nvidia-docker-v1</value>
& H, K4 ~1 c8 q R0 ] </property>
( i' c2 `. L9 K" d( ? <property>
& e1 z U! U) w9 U8 W# u <description>
. p j1 I* G9 J& x3 A Specify end point of nvidia-docker-plugin.
m B8 a5 n3 U3 J6 z: e$ G' ?3 Z9 G2 j! W Please find documentation: https://github.com/NVIDIA/nvidia-docker/wiki) N# w' u2 u5 {; N
For more details.
- D/ `( t- ~5 E8 ^ </description>; B9 q% C- {4 r8 q3 R
<name>yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint</name>
% D- i; w0 l* h9 a/ o# H7 e <value>http://localhost:3476/v1.0/docker/cli</value>
* X1 a3 k5 N, W8 N- d9 b </property>
; r$ u4 ^6 O3 a8 b <property>
& u2 U |; n* f e/ n& ? <description>
; y2 P% n2 B- n- V5 j5 g Specify one vendor plugin to handle FPGA devices discovery/IP download/configure.: R: g3 o8 P. `4 s8 y* @. _
Only IntelFpgaOpenclPlugin is supported by default.
& n* c! E9 T& e0 [9 ^ We only allow one NM configured with one vendor FPGA plugin now since the end user can put the same
V) R3 N" W2 r' S+ G* ~) X vendor's cards in one host. And this also simplify our design.
0 Z3 L" M5 ^' A: j/ r: d </description>
1 |, I" Q* d( d2 l% B% h <name>yarn.nodemanager.resource-plugins.fpga.vendor-plugin.class</name>8 Y: j% C- L* j* \
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.IntelFpgaOpenclPlugin</value>
/ `0 S/ K* v9 K$ C, e* k% H </property>; m: V+ M) R* R+ r6 F, Z, J. n
<property>
/ }8 l- N) |5 A% Y% P6 v8 B7 f+ Z <description>0 L2 s" T9 T$ @% X0 S! b
When yarn.nodemanager.resource.fpga.allowed-fpga-devices=auto specified,
7 N) v/ D9 O' L( Q# z8 d4 j ^& G YARN NodeManager needs to run FPGA discovery binary (now only support1 D3 D* S1 Q' n
IntelFpgaOpenclPlugin) to get FPGA information.
0 B4 l0 \% W) O$ V4 } When value is empty (default), YARN NodeManager will try to locate& O9 ~+ A' U$ }' S/ K" M8 q$ [
discovery executable from vendor plugin's preference
( Q6 j, F9 s+ }' f* H3 }: B+ G7 y </description>9 |" ]( f) m( P: f( U5 {8 e) \ _
<name>yarn.nodemanager.resource-plugins.fpga.path-to-discovery-executables</name>; x, E6 d5 d+ w6 p
<value></value>$ S; a- d. r9 f' ~( t$ a# \$ K
</property>& |0 |1 m( I& j' a$ f: W) }
<property>$ [, M }3 _2 m T4 q
<description>0 s' T! t: v3 \
Specify FPGA devices which can be managed by YARN NodeManager, split by comma2 s+ U3 ]- k e* s, |6 m- Q4 |
Number of FPGA devices will be reported to RM to make scheduling decisions.) L0 q5 Z# Q- h2 o8 k1 q
Set to auto (default) let YARN automatically discover FPGA resource from% V9 \ G0 p. S7 b7 ]2 R2 ~9 v
system.
/ S; c4 u. [- v# D' q: K9 }/ j Manually specify FPGA devices if admin only want subset of FPGA devices managed by YARN.4 T9 h3 X0 @- o. F
At present, since we can only configure one major number in c-e.cfg, FPGA device is1 y" S7 Z1 N4 B- a
identified by their minor device number. A common approach to get minor
( ^# L$ {% K9 s* |# o device number of FPGA is using "aocl diagnose" and check uevent with device name.
# n: c; h6 J/ _8 u6 [) v/ D </description>" C" L5 k* H$ l' k& O
<name>yarn.nodemanager.resource-plugins.fpga.allowed-fpga-devices</name>. L# s2 a/ y/ T
<value>0,1</value>& t' z2 V, ?% B) p, ]) ^( f
</property>
; F( x2 W8 c# M, `/ v% s <property>
% q$ j; ^1 z4 M- a <description>The http address of the timeline reader web application.</description>3 e ?3 n( d: h$ y: i6 h
<name>yarn.timeline-service.reader.webapp.address</name>) q5 g8 M8 Z" p: ]% S
<value>${yarn.timeline-service.webapp.address}</value>8 R! h# r- j. ~% t1 a* Q
</property>$ v1 }6 V' I( K2 Q# k7 X" U4 e
<property>: m3 @: t6 L9 k& K- V2 u
<description>The https address of the timeline reader web application.</description>7 A1 G/ w7 V" \. a3 X3 Q2 j2 z k# o
<name>yarn.timeline-service.reader.webapp.https.address</name>
& b2 N+ c" H+ W- w; @& k/ N <value>${yarn.timeline-service.webapp.https.address}</value>% e9 Q7 z, k5 H& }4 `8 l' I7 g- }
</property>
* b1 U6 G! J3 M <property>1 T* S% g% ?' W, e) Y- j
<description>+ \% @& N3 _* ^
The actual address timeline reader will bind to. If this optional address is2 a7 r8 \! G7 A. j( _( G4 `+ B
set, the reader server will bind to this address and the port specified in
7 b# H0 f3 n" D" A6 y yarn.timeline-service.reader.webapp.address.- y/ s1 d4 Y8 X3 A" ?
This is most useful for making the service listen to all interfaces by setting to
2 [, d" O$ k- x6 z$ [2 \; a$ S 0.0.0.0.
5 a8 l! y4 I& }! d% [& d </description>: j" ]' [) w, n% Q
<name>yarn.timeline-service.reader.bind-host</name>( }0 |8 x& }% E3 n
<value></value>( I1 q# D( C! s. p5 ?. s
</property>5 D ]3 Y2 P( h/ x$ g3 j2 l
<property>2 A% w( A/ K& ^! Y8 ]- P
<description>
c) t l8 @. z. o Whether to enable the NUMA awareness for containers in Node Manager.: K! n6 a7 u5 e/ d: V+ P
</description>6 ^7 b' E4 R c! ^; H# Q
<name>yarn.nodemanager.numa-awareness.enabled</name>
( g. Y4 o5 y* a/ k9 o. R <value>false</value>
$ g' x( Z8 q' [; t/ w </property>
" O2 y: Q+ W8 K3 i1 k f <property>8 J4 h5 A+ ?2 r" @2 Q- ?0 G
<description>' h- u, q3 u7 \" t) T$ P/ N
Whether to read the NUMA topology from the system or from the
5 Y, C( K* i8 I& ^* l configurations. If the value is true then NM reads the NUMA topology from
! ]. R7 _" x& f9 T% Y2 t* n system using the command 'numactl --hardware'. If the value is false then NM; Q% z- F+ I5 n) K. j. |# E
reads the topology from the configurations- E( b8 S! h5 s0 {
'yarn.nodemanager.numa-awareness.node-ids'(for node id's),
0 b- z9 F* n* P8 O6 @% r7 k3 }( v 'yarn.nodemanager.numa-awareness.<NODE_ID>.memory'(for each node memory),
, B. n7 v& t. V$ m# |/ X) t: V3 K 'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus'(for each node cpus).
. K5 g9 I, P( @( V8 T </description>3 X# A; q" u* C
<name>yarn.nodemanager.numa-awareness.read-topology</name>
# r6 K# [3 D! x. O <value>false</value>
2 x k: v Z6 c+ j( K </property>
C! y8 k0 m9 r) G ^* l' m <property>9 M% U4 W& P8 m
<description>
; \) l+ A, X! D" W4 F" _ NUMA node id's in the form of comma separated list. Memory and No of CPUs3 Q8 _0 X$ B2 b1 O
will be read using the properties
3 l! E: K) A+ G' q' \4 s4 o 'yarn.nodemanager.numa-awareness.<NODE_ID>.memory' and/ U6 Z. R* g/ U, `$ G/ K% ?
'yarn.nodemanager.numa-awareness.<NODE_ID>.cpus' for each id specified$ o3 B y3 ^ @. L5 E+ I
in this value. This property value will be read only when/ J3 ]' t6 U5 Y" ^
'yarn.nodemanager.numa-awareness.read-topology=false'.
3 {: ~- r# N; _ For example, if yarn.nodemanager.numa-awareness.node-ids=0,14 v7 n& q( i) s8 h% J6 l
then need to specify memory and cpus for node id's '0' and '1' like below,
2 u. Q4 i6 J4 B3 W/ Q0 k4 v% g yarn.nodemanager.numa-awareness.0.memory=73717, `- K# \4 k$ \4 Q0 R8 s) R1 k
yarn.nodemanager.numa-awareness.0.cpus=4, j; M/ j! {8 ?! q( A/ z) l+ s1 J
yarn.nodemanager.numa-awareness.1.memory=73727
, h* I! p7 M- }% f4 y# d) g yarn.nodemanager.numa-awareness.1.cpus=4: B4 B# z9 A4 V
</description>
' f" D7 ?4 {7 V <name>yarn.nodemanager.numa-awareness.node-ids</name># R2 y+ }8 D1 L$ h. x [
<value></value>$ s- [4 _0 u0 r9 W1 S
</property>4 n) `) U$ i2 J8 `" F
<property>
- Y# p, M3 [0 i& y+ c0 K; J# C <description>7 l K' m3 T% u5 r1 z' G* k
The numactl command path which controls NUMA policy for processes or
- l2 m2 F, ?! C" f) e- _9 B shared memory.
% Y: z. R. Z0 o. l </description>
0 M6 a" J6 W8 S <name>yarn.nodemanager.numa-awareness.numactl.cmd</name>
3 F' Q( z9 y& p* Z3 F" Q* W <value>/usr/bin/numactl</value># u( Z# m( P3 T, s4 \- ^3 {
</property>
5 E4 T' @$ n) M( F$ O</configuration>4 z' `! M& |/ m! ~ E( n2 D
5 E6 p5 W/ D+ X& |
|
|
发表于 2022-11-7 11:18:00