|
|
filebeat是轻量级日志收集框架，go语言开发。需要在每个日志收集的终端部署，配置日志文件路径。可以将日志收集到es，logstash，这里以收集到elasticsearch为例。配置主要分为input和output两块。解压后有filebeat.yml配置文件，主要针对该文件进行配置。
/ Q2 q. U* e) A' [
& {+ y3 L' }; E- type: log4 |2 ?# H2 F& J% x' s4 v
#日志文件位置 h3 k! I) R2 R! u; v) z& B/ ]
paths:
1 L1 f' F9 @2 _3 `( G - /data/logs/*/*.log
& @# G8 Y; t% U6 toutput.elasticsearch:
( R; B! S) ?. Z m #es连接信息8 W6 k: c/ d! O3 P, C
hosts: ["localhost:9200"]
1 n; h6 h# ^7 _3 b3 x. S7 @0 S protocol: "http". z1 [1 A$ o1 \; H
username: "elastic"
2 J6 y. v" D) r8 }* p: Z/ v password: "888888" 0 W# W i5 t1 H9 c
会自动创建一个 "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}"
6 T! J$ O& V( }/ a4 E( c- P/ B! \4 {/ L! N
7 P g' n( ]2 Q- X5 z7 J1 Y
m ~. W( y- D' t( l3 U i9 W
例子：/ P- d4 Y* B$ r* }. S6 ]6 x% O" n
/ k' z( Y$ D* ?* l5 u8 \vim /etc/filebeat/filebeat.yml
$ G+ T r' @; X- Q, Z- }* g. V, Q4 N% F0 D Y; h7 W: o
filebeat.inputs:
/ M- C) X, k7 b" \: O6 U6 \ - type: log
4 w# Y7 \; a: K G enabled: true2 Q# q% o1 u: R- g9 P% @
paths:
2 \4 N: q" z$ w: W6 z, A - /var/log/messages
; t0 p* v) }+ q; l7 h! B tags: ["messages"]( M K4 Z k4 f, S
fields_under_root: true
( o2 X, j! H+ K" i3 e: R( }7 U L, e0 |& W% r
- type: log+ |) S2 C0 @% O: q7 b
enabled: true
# Q o, u# F/ m2 W paths:2 m' J3 B, `+ K$ A3 ]& \
- /var/log/nova/nova-compute.log: [& ~( m% s1 X' x
tags: ["nova-compute"]0 T, m; a/ m+ R5 S
fields_under_root: true
! ?) u2 O( ~* n4 W
' i& B1 [; Y& M2 B5 h& S: `% ? - type: log
$ l5 V5 T5 }. G! U enabled: true
$ u$ {+ ^7 T! v" t; T L paths:
+ ?4 s% o! w7 o - /var/log/nova/nova-manage.log3 Q/ y( k0 M, u) ^8 e' t
tags: ["nova-manage"]' S! n4 i# O; X" ?( l" y6 F h
fields_under_root: true
4 X: Y2 O3 u8 }, m; L: \4 g! O8 P* h. L% A$ W ~
- type: log
5 |+ j* x2 h3 ~ enabled: true
& o9 [4 d4 D& i$ b paths:
; L/ V" R$ U8 O# b - /var/log/nova/scheduler.log7 S, r- J& r9 G, i: U* {% D
tags: ["scheduler"]
) ~$ x, L+ b' D fields_under_root: true3 G. s7 g3 Z+ F. }6 I) h5 n
$ L! B+ x: l- M5 o& z
- type: log; P+ h4 B% u$ c" G/ J8 e9 R5 a
enabled: true6 b6 w6 {# g5 n# T5 ^: c
paths:* k- B2 Z+ D' @; I J5 P
- /var/log/nova/conductor.log0 _! |$ z) |7 |
tags: ["conductor"]+ P; }8 h8 J$ P) q
fields_under_root: true. C6 O0 w4 f0 X0 H5 R
- i8 o) [7 q; Z3 X - type: log! T# |$ o3 f1 T# D, }' x$ D; ~$ \
enabled: true
1 N& v3 c+ v, H0 S5 h paths:
: V! H- Q5 N5 @ - /var/log/nova/cert.log4 Z2 Y, o' M y% M1 b. K) ^* E
tags: ["cert"]- g: T' d3 G' _8 H
fields_under_root: true
3 ]8 [7 D# c6 a5 A) S+ Z/ [
5 j+ Y9 m4 f/ b9 U - type: log
1 U+ q! s. W M; r* A enabled: true. B- y: ?- O) F V D1 A$ l
paths:
" o2 a* T# P& k5 C - /var/log/nova/consoleauth.log . V+ _( M) ]. M j; M5 P) s
tags: ["consoleauth"]3 j7 A; s( H0 Q7 ]( \, @6 g
fields_under_root: true
5 A: b* j. C7 [/ w6 ?4 j8 n; x1 g6 y: Y9 I% ^) i9 J3 F9 L- M
- type: log9 X4 q% B" m; O5 B$ _
enabled: true7 o7 M: U) c8 I
paths:2 V5 `; k/ Z7 y7 b, p' n0 r
- /var/log/nova/nova-novncproxy.log+ G# l/ L: \' ?5 g2 w9 ]8 Y$ Y
tags: ["nova-novncproxy"]$ b# M: n! k3 s5 Z& v0 @7 v7 Z
fields_under_root: true1 A" T- ^3 t5 D, z$ j8 I" @# l
7 Y/ c' z7 \" y W) I* | - type: log8 b, D( D9 {# {/ n; n
enabled: true
# c8 {( N9 v5 d: K6 n5 ^% ?8 g paths:
) b0 w( P' }7 I/ w x - /var/log/rabbitmq/rabbit*.log
' `- w% i: A( P, B; { tags: ["rabbit"]3 [' S# C0 |( s1 h9 g
fields_under_root: true- u) f( @$ K+ B* S
! F' n7 S' V7 ?: f- @6 G - type: log
( ?0 A. D/ F) x0 H1 N8 p enabled: true2 G# v' y1 L4 h
paths:
. p) {( l/ I! D3 e' h! v+ |/ g! p - /var/log/glance/*.log
/ l: y" e* x9 N1 D9 B( @& ~ tags: ["glance"]
* }7 Y: ?( N4 C8 ^! Q fields_under_root: true- |( e6 n; X$ z; }; P8 m
& z% s/ |0 D' h1 V" i
- type: log+ O$ t8 T* C6 E0 H Y
enabled: true! h& N" F2 [/ H
paths:
( n/ F2 {) N7 C% N4 j - /var/log/neutron/openvswitch-agent.log
/ ~# } x" {1 X tags: ["openvswitch-agent"]6 H! w' t: M3 f: S
fields_under_root: true
2 A! X3 ~- M; S0 h4 U0 G4 j" T1 _) M9 i( o- n2 E5 r& t& [; u
- type: log3 N9 E0 d; j: m* E. C' {
enabled: true- X8 z( _1 v( Y; h, z9 T" d/ j& o
paths:/ y5 n- g- d9 f$ E
- /var/log/kuryr/kuryr-controller.log; j0 \2 d, z6 p' y. b
tags: ["kuryr-controller"] F, N8 X+ y- H/ c' h- G
fields_under_root: true- k; R* K$ z7 {; ?) A
- u6 S5 r$ V: j! I2 @' _; x) O
- type: log3 p) k) ?6 n& b3 w& Z* |$ s
enabled: true
5 A7 R+ M% x! {' F8 B paths:4 w& w$ y, l4 h) A; D4 j" H4 h
- /var/log/keystone/keystone.log
7 L+ I0 Y) T, U2 | tags: ["keystone"]! p' `+ ]+ P7 ~1 W
fields_under_root: true5 S0 [! [3 x+ c8 j
" C' Q% R3 K4 E" I5 `
output.elasticsearch:
' E$ M7 K( V. t" [- R hosts: ["172.24.110.12:9200", "172.24.110.12:9200"]
5 {# t" \ o" F# S6 Z) B/ W username: elastic
: i$ W2 _9 t. q9 ~% G% |- s o password: xxxxxxx
' u! S9 [1 Y4 D0 L% t indices:
- y8 ^( h* Y( A4 N* m - index: "compute_messages-error-%{[agent.version]}-%{+yyyy.MM.dd}"
4 a- d/ T( ?: S# | `3 H( ~ when:
# N1 ]4 y e- u: a; b or:
" o6 V' ^+ q# A V - contains:/ P$ |% l9 e9 a7 i' b& a
tags: "messages"9 k$ i: j* T7 @9 l; ^
message: "err"
" w5 P4 {7 _$ @: C8 r2 z - contains:# e5 P9 z7 T% n8 T, h X
tags: "messages"
/ i2 c1 `" U( m* q message: "ERR"
3 k7 [# c3 c) g - contains:) r; m0 T; N% i- B7 l' y% F' T
tags: "messages"
7 p5 r7 x. A7 Y% I message: "fail"3 R( H+ y* J+ O6 i
- index: "compute_messages-%{[agent.version]}-%{+yyyy.MM.dd}") X9 d C( _; U
when.contains:
% G3 {. K. f6 M7 u1 V$ b tags: "messages"
; P( z2 E3 x8 n3 p6 s - index: "compute_nova-compute-error-%{[agent.version]}-%{+yyyy.MM.dd}"& C4 [9 n6 z$ c, k: H6 Z
when:
& ?! X' X [* z; A& y; c$ J or:
4 @9 a3 Q! u- R: E; _* q* r - contains:
! n! o- P1 `) x- n tags: "nova-compute"; y- ?1 {' {* O0 s& U) q
message: "err"
* i2 ^6 W* g) @" B* D+ b - contains:% }1 T' R( T+ g+ n
tags: "nova-compute"
: h2 g7 a- U+ G! S+ L* ` message: "ERR"1 F7 ?! k" m: D9 S# |, z
- contains:/ ]% H: K+ Z" U
tags: "nova-compute"5 }' {) w' @$ k
message: "fail"
0 x0 S; {$ P, J. x( ?. @8 F" K - index: "compute_nova-compute-%{[agent.version]}-%{+yyyy.MM.dd}"9 z7 F( h4 Z/ z3 f
when.contains:$ p- o" T" N, J) f; W% l6 ~: P. D# U _
tags: "nova-compute"
6 q: R0 I! X. k" W
2 F( {: Z8 I/ A; D) _: X* s: n - index: "controller_nova-manage-error-%{[agent.version]}-%{+yyyy.MM.dd}"
: X* A% i8 C! q: z: ?. i when:
5 d* O6 r9 S i( m or:" [# x+ K$ ?! A0 g" D
- contains:
0 b7 V9 |5 H# ^0 m* A; r tags: "nova-manage"! R" h! P0 d- L/ w& p
message: "err"
) n$ T E8 o5 \$ k/ L$ \2 b( ` - contains:" l* T* |! u0 |
tags: "nova-manage"/ h( N" N4 u6 q) n L
message: "ERR"6 P& @, T8 G- M) U) F
- contains:
0 m5 }9 d( a! P- P* f [8 ]; o tags: "nova-manage"
2 n! b0 i# \. P2 n: |3 O* X& h! @ message: "fail"
! R" @8 n) j- E) h - index: "controller_nova-manage-%{[agent.version]}-%{+yyyy.MM.dd}"
U- v+ M/ [1 R5 f4 ?& v when.contains:
7 T: @- u/ w8 W; F7 r; U( G1 i tags: "nova-manage") C! F3 Q4 Q2 M5 X* C6 E
8 d7 Q8 w5 `) F - index: "controller_scheduler-error-%{[agent.version]}-%{+yyyy.MM.dd}"2 x% {2 W$ Z; [! @
when:
5 N6 ?! ~; J; x- t, B" n2 j6 k( X or:% X/ H2 ^9 A; `4 f$ u
- contains:2 I6 [4 v& {, S% H! P: G1 @
tags: "scheduler"
+ C% _( d$ v0 E5 H# O message: "err"
) F* J% l$ G. P6 X - contains:
. y& Q# c% i7 z; c( a tags: "scheduler"
1 b: y1 ]$ O' L. O, j, r/ G( M; c: R message: "ERR"
# D* Y J) @# G, U: R) G - contains:
5 m1 G, G. O* y! M& s a tags: "scheduler"5 B. b* ~; [- D2 _! b8 @, _
message: "fail"7 O' L9 l3 w* u* B8 p
- index: "controller_scheduler-%{[agent.version]}-%{+yyyy.MM.dd}"
: j" w4 E- D6 _- L2 S& { when.contains:
6 T1 G `6 T0 h E+ D: I tags: "scheduler" ~' \1 d" m% S/ x, i
- z" f3 z/ ~; _8 U: E - index: "controller_conductor-error-%{[agent.version]}-%{+yyyy.MM.dd}"
( D: [6 b+ S- g+ Z9 h" y when:
7 h% g9 \7 }' ]& a or:
& f1 D9 `; }- R& ]0 b4 Q - contains:
8 V8 u4 v' \% ~4 ~1 q tags: "conductor"
9 P0 R1 U& o6 r: c: Q$ r4 e message: "err"3 \" P: K- o5 M. Q. N
- contains:8 m4 j, f X! E5 H/ `. ?
tags: "conductor"- E3 H% K X( u+ w/ N
message: "ERR". Q$ n8 ], U* H* q9 P
- contains:4 d3 n7 [4 U% V( `
tags: "conductor"2 n3 l! b! u4 ^$ v; ]
message: "fail"
- T! @' O+ W; Q! z# p. p - index: "controller_conductor-%{[agent.version]}-%{+yyyy.MM.dd}"
9 c' w" F9 t4 n2 \ when.contains:% \; b0 |* ~6 t/ p& v. G
tags: "conductor"
/ V5 G0 X0 C& J+ X3 O
! c" a: m& Q' W" v% B: C2 J. B - index: "controller_cert-error-%{[agent.version]}-%{+yyyy.MM.dd}"2 k/ R1 w8 m P. o# I2 }7 @
when:3 G; i/ j/ [' |; P7 p5 @
or:! w& Q" {: d% ~" N/ p) \( I1 x7 E
- contains:
2 _% B- ?' |+ ^; ~' M/ L tags: "cert"( J1 a& p' K# }" D8 Q. h# i9 u
message: "err"0 M/ ]. e, g: g6 v4 N
- contains:5 |3 V/ Y1 t+ x9 h% [2 N5 \* A9 `: o
tags: "cert"7 F Y0 L \7 @! U/ X# G
message: "ERR": ?, F. c {# j8 P$ @% l9 x
- contains:
+ f0 k2 v" {* a5 M tags: "cert"9 Z: D; j4 G8 d* i
message: "fail"
( t( G& i7 k9 C% M! d - index: "controller_cert-%{[agent.version]}-%{+yyyy.MM.dd}"
7 I/ P( t4 r7 O when.contains:; h8 g3 W2 q8 ^- C" {( E$ o
tags: "cert"
* T: S' {3 t5 ^5 J
; }6 T% t9 e4 K: x6 d - index: "controller_consoleauth-error-%{[agent.version]}-%{+yyyy.MM.dd}"* ^2 U1 I, j$ N0 |5 [
when:* |6 V5 L% E; H- r" c
or:
4 ^. ~1 `" e& k9 t8 P1 a8 Z- o ^ - contains:4 ]) b" [6 `+ }& ]3 t. E( `
tags: "consoleauth"5 s0 p/ @. H ^. A3 E& Z! Z
message: "err"" R- V) T1 X+ H3 Z+ Q) |
- contains:8 P; ^& q; I4 N( W4 @$ g
tags: "consoleauth"
5 F8 X5 d( X+ [, C* Z U) [ message: "ERR"
, B1 x9 f( m3 q5 |! E- J5 L - contains:. _% \% Q' A" x4 S: g' f
tags: "consoleauth"
5 f' i7 m, k- U( U- @+ Y message: "fail"
2 \% B( S7 O M: O9 ` - index: "controller_consoleauth-%{[agent.version]}-%{+yyyy.MM.dd}"
! L/ ]) y, T( ^ when.contains:
: f6 V& V( Y5 G6 y& L tags: "consoleauth"
' [+ X& ?2 X1 N% G j2 E. D! I: |% V- `
- index: "controller_nova-novncproxy-error-%{[agent.version]}-%{+yyyy.MM.dd}"
$ s' } p$ f% y7 S3 @! D when:
' ~7 ^5 q' F+ j# _' p or:
! @- W3 U) P5 n- H! j- M$ b) i - contains:
9 C4 s, P! m# I+ u6 ]' ? tags: "nova-novncproxy"
) ]1 [7 d$ K3 N( f2 |. w. c' S. p message: "err"
" P4 `* Z- B0 v, k6 F - contains:
: a7 ~# {% ^) ~8 j K( X tags: "nova-novncproxy"! c: [0 X, }0 R# w9 A
message: "ERR"
! y5 |( X# V5 B `& f6 ^ - contains:( R9 e X! ?/ M2 L' l
tags: "nova-novncproxy"$ j' k: f% W0 A2 u
message: "fail"
5 c6 v0 |/ [- i% B - index: "controller_nova-novncproxy-%{[agent.version]}-%{+yyyy.MM.dd}"% I3 t$ H( I) n! ?9 O( z
when.contains:% i$ c1 W% g! Y. H
tags: "nova-novncproxy", Y2 Z @: q" o$ u" @, f6 i
- |+ p2 W5 l- K% U) |
- index: "controller_rabbit-error-%{[agent.version]}-%{+yyyy.MM.dd}"0 m: l' n3 t% D5 q2 G9 t8 S8 X
when:6 V6 m% l( F) \
or:- v, M2 P% c8 |3 |% W/ N3 [
- contains:% Q* T( ~- p5 i: e5 [
tags: "rabbit"
1 A; c& ^& w" @ u) v. O message: "err"
3 @: c& ]; {( Y - contains:
7 Q$ b& C+ @# z6 T; E3 l* Q tags: "rabbit"/ R1 Q5 J3 M6 R
message: "ERR"
4 k$ w4 M- \& m) l* q6 [2 ] n - contains:
6 k+ ?" ?. P( z tags: "rabbit"
( Y. d8 F2 F( T. @1 f- q: K message: "fail"
0 Q7 p) m+ d; }: `$ R - index: "controller_rabbit-%{[agent.version]}-%{+yyyy.MM.dd}"
) {" e2 U& @9 b( ]3 Q when.contains:4 U; c; E9 R, z/ h4 m* M
tags: "rabbit"0 i4 k w& x+ G# e2 r
- v0 U$ `+ a4 \" W* E) n( e
- index: "controller_glance-error-%{[agent.version]}-%{+yyyy.MM.dd}"
3 i/ y1 X& K. E+ G( g( a when:! m* N# q# o& y; A& u% f
or:
6 M" m8 V# Q" y* p: X m' `6 A+ h - contains:$ v4 g4 M+ S- e' }8 R, c
tags: "glance") q8 s3 n/ A2 X0 ` I
message: "err"
7 v+ f1 p; E1 m0 a - contains:
/ E8 c8 t' K" G tags: "glance"
% G( a0 }* l* D- v9 p' U8 G4 U+ f) ~ message: "ERR"/ ^3 C" x. F/ J' R
- contains:1 O+ c3 ~+ e6 r) C; R
tags: "glance". r3 a0 W- }8 `. g
message: "fail"
; b" s+ M+ L: f( x9 x* d+ v) m - index: "controller_glance-%{[agent.version]}-%{+yyyy.MM.dd}"( b) ^7 s0 y: _9 k# ?+ l
when.contains:
% N' r! x2 x" k7 j B* W tags: "glance"
3 w: @3 T& I0 m' v7 |1 E, t! I: {# W7 w2 c# B9 |1 B! c
- index: "controller_openvswitch-agent-error-%{[agent.version]}-%{+yyyy.MM.dd}"# g! n7 B) P% h) k3 L$ @
when:$ f$ \( t8 ^5 L* F' S' o/ n, S
or:9 Y4 y2 \0 S* u& H! c: [( ?
- contains:
( C8 e7 K3 |! y4 ~1 M& o/ M tags: "openvswitch-agent"
# A G, w% ^# f) }$ B" b message: "err"
p8 C2 M, ~6 g- ?% O- f+ o - contains:/ R' a9 u5 \" l% G
tags: "openvswitch-agent"6 `9 [0 N- \" k: r. n, i
message: "ERR"
! `6 w% w4 J( m: {4 J4 g) ?; P - contains:
2 T8 Z9 o, P, N S" S8 N. K tags: "openvswitch-agent"
6 w/ p9 }6 e% i1 a2 Z message: "fail"4 j7 g4 `* F: C0 }* m
- index: "controller_openvswitch-agent-%{[agent.version]}-%{+yyyy.MM.dd}"* z9 [5 m; z! r! u# y( i
when.contains:
/ N& d8 ^! D+ {8 t1 X tags: "openvswitch-agent"
( n9 L8 l3 a3 ^+ k4 J7 F5 j1 f$ ~
- index: "controller_kuryr-controller-error-%{[agent.version]}-%{+yyyy.MM.dd}"
3 b8 ~& n! u% C3 r when:
( G" N c; A. Y or:
$ v6 d7 E5 h6 r - contains:
4 |" @' a' ?" n! y9 i. Y# F' G( O5 i tags: "kuryr-controller"
' j0 ?. S2 y1 b5 r$ q3 N- X message: "err"/ x8 _( ]6 `6 s6 `6 u, |& ]
- contains:
8 l6 {' \* D1 y5 T$ y+ n8 E tags: "kuryr-controller"1 y7 B" Y+ ?! }$ b" n+ ^4 U
message: "ERR"
5 {5 h$ e* k# L$ ^ - contains:( i+ e4 Q$ ]/ k5 {- H
tags: "kuryr-controller"
' l/ \$ |2 k& _3 n; _3 N message: "fail"
, C ?* x/ n1 J0 {% B4 B - index: "controller_kuryr-controller-%{[agent.version]}-%{+yyyy.MM.dd}"& N0 T6 f z) p
when.contains:1 Z3 I5 F# C8 g0 I$ `: I
tags: "kuryr-controller"
# k8 D) Z0 q" a) {0 v& M# M
3 d* E/ y' R8 o7 Z - index: "controller_keystone-error-%{[agent.version]}-%{+yyyy.MM.dd}"' d9 C% w" @1 b+ c& g1 M! y% `" z
when:
4 K, f) i5 K& a! ` i or:
7 E2 F$ j& f6 R; k; G, L. a; ? - contains:
# ]/ i5 G: ]% S3 D* G tags: "keystone"2 b. n9 `. l2 P( `& ^- ~
message: "err"
U8 B$ l, c% k - contains:" E5 q$ B/ `9 B! B8 [
tags: "keystone"
3 s7 v" G: h( t$ R4 O; h( @ message: "ERR": D& P7 g, B/ z! x" Z+ e: D* m5 B
- contains:
9 Q5 l+ x1 Z3 d N tags: "keystone"* P/ D. ?1 m% Y& t, \$ Q& `8 Q+ X( v: \+ y
message: "fail"
9 p5 R8 `) c1 C - index: "controller_keystone-%{[agent.version]}-%{+yyyy.MM.dd}"
* x5 x; `& i& o2 }* i* s$ @; U: G" N when.contains:
. g1 [+ x- t2 a7 S0 U U1 k5 G% [( z tags: "keystone"# R! P: B/ Z3 H; _* l1 P
: ^( ^) c) U, n t6 Y$ i3 Osetup.ilm.enabled: false9 U( R1 R3 p) W5 w1 b# g/ T5 V
setup.template.name: system
2 f) B0 {# n2 Y E' vsetup.template.pattern: system-*
! v8 m+ ]' \- M) q4 a" d: ]5 b* }2 Z9 |1 u8 S, O |
3 N% Y. ]- Q. C
8 h) J/ X+ W8 W% Z: T: T) h! } }& P' [' Y8 @, \( Q
例：filebeat-7.12.1-2023.05.16-000001索引文件, s, Z% M- {, J9 S; @
0 I! b% X) Y& ]8 d
索引创建规则4 b& r$ H5 G& n i) Q8 F# g
6 ]& g; _* W5 y$ `# n默认使用es的索引声明周期策略
/ X$ D- N$ g2 P; ?+ B4 H/ y G4 v, h4 a5 W
index lifecycle management (ILM) 生成索引
3 m4 }/ _7 y' ]% i% S5 k# u5 R. V/ H! g& W: z; u: D5 E
配置ILM8 V" A5 l5 z3 W+ {: I
5 X; Z" E8 X! ^/ Q3 n#auto false true; T5 C2 ^* @) b" S X8 C- b
setup.ilm.enabled: auto2 p" F5 p$ N' |' [0 a6 O1 t% _) T
#索引别名
: b0 J( _4 T# N/ R+ Vsetup.ilm.rollover_alias: "filebeat". k+ ~ @9 U& l+ [
#索引增加策略
- v, A, i: k' R& h7 A; }7 Isetup.ilm.pattern: "{now/d}-000001"( i2 p0 t; G- K4 Y: W6 Z/ `* I
setup.ilm.enabled默认值auto，自动使用es中filebeat生命周期策略创建索引
+ f) m" J9 D8 D" S7 C5 y
; W" `7 V' s. ~4 P. x b( ksetup.ilm.rollover_alias默认值filebeat-%{[agent.version]} ，创建索引时指定索引别名。
/ n, w* f1 i- {, j
, t* k2 t: {8 q5 n) F, T" Asetup.ilm.pattern默认值%{now/d}-000001，索引rollover增加策略。8 F8 U3 J) J: V8 D x* }& j
0 { k6 |; ?; H7 r自动生成的索引名就是使用alias+pattern。类似filebeat-7.12.1-2023.05.16-000001这种。9 l" ~& N" O; z7 j% q
5 B" m0 L; V6 _3 a8 Y6 \' ?
更多配置参考：https://www.elastic.co/guide/en/beats/filebeat/7.17/ilm.html. x& p6 B: @: C( E1 Q
$ ?7 r, ]: w: Q% F/ s/ A自定义索引文件$ d: ^7 I+ c# h8 U
; h' g( Q- }" W g1 I
output.elasticsearch可以指定index，使用自定义索引第一步就是要关闭ILM,
* D- o' x2 O0 z+ B- b" i5 n; W7 |, E J# r$ k6 }2 a$ o& @6 L" U
setup.ilm.enabled: false' Q: Y0 B8 o2 F. c4 {; u' z5 @. U
下一步要配置setup.template.name和setup.template.pattern" r: c' Q* m1 z- T2 A( \7 Q" L
+ ?8 n: \" x$ ]0 w
setup.template.name: "filebeat"
9 F7 Q; o: W9 K- Osetup.template.pattern: "filebeat-*"- b% w# N. W7 Q# A+ N8 J9 x
setup.template.overwrite: false
) o+ ]7 { i' i- [在output.elasticsearch指定index' V: v) u0 R" y8 l2 l* ]
1 V0 j+ |6 g! U( Mindex: "spring-%{[agent.version]}-%{+yyyy.MM.dd}"
3 L) R' U- Q* Z# S$ b" n0 h运行就会自动生成索引spring-7.12.1-2023.05.16。index定义可以使用上下文定义变量。可以在input里自定义field A% {5 M8 t9 p3 A
7 v$ J& P0 R+ t; R
fields:
1 w" L' w9 t" u3 _: A% K- R level: system
, ~1 c8 z& L- l4 G region: A12 Y$ |# H' u" Y( B7 w' L" a4 g
自定义的fields会一并push到索引中，index中使用自定义的fields' \ c/ L r7 a* w5 s: u
0 S7 H- Q0 C1 C% W( D/ [5 R
index: "spring-%{[fields.region]}-%{[agent.version]}-%{+yyyy.MM.dd}"0 }8 c4 P2 ^- A* ]; Z
会生成索引：spring-a1-7.12.1-2023.05.16。这里A1自动转成小写了。( {& k8 x: P$ d7 U3 g1 p
3 X( g: x8 T+ j
日志多行合并$ {7 S( S+ n! ] i) H, Q
1 F1 J5 X- i8 x; n默认情况下收集日志一行一条记录，有些情况下比如格式化输出，异常栈。一条完整的日志会包含多行数据。这时候就需要配置多行匹配。配置项在filebeat.inputs里6 U& l E' _9 J# t
. {0 \1 z% h# f* w% m# N; X# z$ @: F
multiline.pattern: '^\['3 M9 p1 V( O& T% \0 B2 b7 |' F
multiline.negate: true2 K" V& |2 y6 q Q0 m# C
multiline.match: after K5 x6 b' R, i& c" Q/ C
multiline.pattern指定日志匹配正则，这里'^['就是匹配以 [ 开头的行。这个地方的具体格式就要合实际输出的日志格式相匹配了。: `9 q1 W9 [1 D v
& [: J7 Y8 U9 T
negate和match两个参数结合使用，没太看懂，理解其来感觉有点绕，自己看官方演示例子吧https://www.elastic.co/guide/en/ ... iline-examples.html，有个表格图例。大体意思就是遇到不匹配的是向上合并还是向下合并，归属于那一条。这里配置true和after就是不匹配的格式行归属到上一个匹配的结果行。
) d* j' h/ ?5 i: }* T- E `0 N
I; W+ _) D0 V% E; T根据条件写入不同索引4 \7 |$ J- j9 \* _# }( b5 C3 b
$ {1 E" l0 r+ ~1 V$ i* U7 P* ~! Ooutput.elasticsearch:
/ u2 W' j" m a hosts: ["http://localhost:9200"]
6 U) y" r$ ?: b0 [ indices:; H! o! z* I/ S5 J; c' C+ P
- index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}"$ ^- o- |6 W, @+ s( k
when.contains:
/ ^* ]2 Q: X& \; F message: "WARN"" v2 Y& e+ M# B9 i$ Y
- index: "error-%{[agent.version]}-%{+yyyy.MM.dd}"4 V+ N% I. A$ X; s: @; ?$ ^# b
when.contains:. _* K& @, U! r; y
message: "ERR"6 T" {+ `% C; I* s! t
# V( r" Y" |& W/ H6 e& T5 n判断message内容，是否包含某些内容。不做演示。
2 L& O. B0 [' ~6 T7 g1 j. }( g
1 D8 F7 b0 r* |8 z+ K% G收集到的日志可在kibana 日志功能界面化查看检索。需要配置日志索引匹配模式，例如上面的我们就需要新增匹配日志模式spring-*。
' |( C6 X( }2 v5 s9 s) q8 a
& s6 m4 n( f3 u最后filebeat.yml有效配置大概这样$ Q8 U5 ^; ?8 d; i. J3 a V8 G5 i
- X+ e. V- d# o$ w. v( m
filebeat.inputs: G9 J g( @0 Z+ w% ^# Q7 C
- type: log/ P- s% d$ W# a
enabled: true! y1 W. y: Q. F: i& g7 x% ]% y
paths:
# x9 w) w6 J8 E8 a) r9 i; y - /data/logs/*/*.log
: D3 b& A( e. t4 V* b$ Z/ T: G
! m3 `/ C. b7 y6 g' y) C% r' r) E fields:0 n" M$ v2 n- \- h
level: system$ H; ~+ N4 ]+ }7 i
region: A1: U3 v+ t2 j% P8 f
4 s' [4 L7 n) H7 g6 i3 F3 g# C
multiline.pattern: '^#\['
: }) {+ |8 s% g6 u6 b multiline.negate: true3 v/ A8 j- x, i
multiline.match: after! Q8 G A& H8 N( H; X
( r! O6 Z# ^8 N) N, L0 C9 c; ]
output.elasticsearch:
X" F5 r4 Q C hosts: ["localhost:9200"]7 A! a7 Q$ N |- F* v! s
protocol: "http"
; e. N5 F2 z2 M8 C username: "elastic"
: G% t# L9 z1 u& ~9 |& v ` password: "888888"/ J+ x4 ^: O) y2 V, Z5 g8 R
index: "spring-%{[fields.region]}-%{[agent.version]}-%{+yyyy.MM.dd}"
- A% q4 t5 e- ]" z) N% b
& b! F1 K4 U2 Asetup.ilm.enabled: false: y" ?. V' ~+ I9 r+ o
setup.template.name: "filebeat"
& l$ J' D! M K1 b5 U* wsetup.template.pattern: "filebeat-*". _2 N- h5 g0 h8 C0 f0 b3 F
setup.template.overwrite: false5 g2 E8 x1 t4 C5 ]& t
) J* w$ w( t! I6 A( S
" m" I0 ?2 _* Y2 y9 N2 G |
|
发表于 2024-11-8 10:18:15