elasticsearch8版本安装详细步骤（集群安装）

admin · 发表于 2024-10-31 15:53:27

elasticsearch8版本安装详细步骤

1、设置虚拟内存
sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" >> /etc/sysctl.conf，
echo "vm.max_map_count = 262144" >> /etc/sysctl.conf

加入：vm.max_map_count=262144
使配置生效：sysctl -p

2、设置文件打开数：
cat >>/etc/security/limits.conf<<EOF
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
EOF

设置pam配置：
echo "session required pam_limits.so" >> /etc/pam.d/login

3、禁用swap交换分区
swapoff -a             #临时禁用
vi /etc/fstab       #永久禁用
找到swap这一行前面使用#符号禁用掉

4、设置TCP重传超时
sysctl -w net.ipv4.tcp_retries2=5
编辑配置文件：echo "net.ipv4.tcp_retries2 = 5" >> /etc/sysctl.conf

5、创建一个用户
useradd es

6、创建安装目录
mkdir /data/elasticsearch/
mkdir /data/elasticsearch/elastic-cluster1

mkdir /data/elasticsearch/elastic-cluster2

[root@it-elassearch ~]# ls -p /data/elasticsearch
elastic-cluster1/  elastic-cluster2/

7、下载官网安装包
官网下载地址，选择对应版本的elasticsearch和kibana：https://www.elastic.co/cn/downloads/past-releases#elasticsearch
https://www.elastic.co/cn/downloads/past-releases#elasticsearch

[root@it-elassearch-2 ~]# chown es:es elasticsearch-8.15.0-linux-x86_64.tar.gz
[root@it-elassearch-2 ~]# mv elasticsearch-8.15.0-linux-x86_64.tar.gz /data/elasticsearch/

切换到es账号：

7.1 解压：我的安装包下载到了目录下，解压到/elasticsearch/elastic-cluster1/目录下

cd /data/elasticsearch/elastic-cluster1
[es@it-elassearch elastic-cluster1]$ ls

elasticsearch-8.15.0-linux-x86_64.tar.gz

/data/elasticsearch/elastic-cluster2
[es@it-elassearch-2 elastic-cluster2]$ ls
elasticsearch-8.15.0-linux-x86_64.tar.gz

解压：

[es@it-elassearch elastic-cluster1]$ tar -zxvf elasticsearch-8.15.0-linux-x86_64.tar.gz

[es@it-elassearch-2 elastic-cluster2]$ tar -zxvf elasticsearch-8.15.0-linux-x86_64.tar.gz

进入对应目录下：cd /elasticsearch/elastic-cluster1

配置es参数文件：

节点一配置：
#vim elasticsearch-8.15.0/config/elasticsearch.yml

cluster.name: essearch
node.name: it-elassearch
path.data: ./elasticsearch-8.15.0/data
path.logs: ./elasticsearch-8.15.0/logs
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["172.24.110.125", "172.24.110.126"]
cluster.initial_master_nodes: ["it-elassearch", "it-elassearch-2"]
#node.master: true
#node.data: true

xpack.security.transport.ssl.enabled: false
xpack.security.enabled: false

下面是加鉴权的配置：
cluster.name: essearch
node.name: it-elassearch
path.data: ./elasticsearch-8.15.0/data
path.logs: ./elasticsearch-8.15.0/logs
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["172.24.110.125", "172.24.110.126"]
cluster.initial_master_nodes: ["it-elassearch", "it-elassearch-2"]
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Type,Content-Length
xpack.security.enabled: true
#xpack.security.authc.accept_default_password: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /data/elasticsearch/elastic-cluster1/elasticsearch-8.15.0/config/certificates/elastic-certificates.p12

xpack.security.transport.ssl.truststore.path: /data/elasticsearch/elastic-cluster1/elasticsearch-8.15.0/config/certificates/elastic-certificates.p12

节点二配置
#vim elasticsearch-8.15.0/config/elasticsearch.yml

cluster.name: essearch
node.name: it-elassearch-2
path.data: ./elasticsearch-8.15.0/data
path.logs: ./elasticsearch-8.15.0/logs
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["172.24.110.125", "172.24.110.126"]
cluster.initial_master_nodes: ["it-elassearch", "it-elassearch-2"]
#node.master: true
#node.data: true

xpack.security.transport.ssl.enabled: false
xpack.security.enabled: false

下面是鉴权的配置：
cluster.name: essearch
node.name: it-elassearch-2
path.data: ./elasticsearch-8.15.0/data
path.logs: ./elasticsearch-8.15.0/logs
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["172.24.110.125", "172.24.110.126"]
cluster.initial_master_nodes: ["it-elassearch", "it-elassearch-2"]
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Type,Content-Length
xpack.security.enabled: true
#xpack.security.authc.accept_default_password: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /data/elasticsearch/elastic-cluster2/elasticsearch-8.15.0/config/certificates/elastic-certificates.p12

xpack.security.transport.ssl.truststore.path: /data/elasticsearch/elastic-cluster2/elasticsearch-8.15.0/config/certificates/elastic-certificates.p12

JVM堆内存大小设置
设置内存大小：

vim elasticsearch-8.15.0/config/jvm.options

-Xms4g
-Xmx4g

实际业务线上环境，建议所有Elasticsearch节点都是独立节点，不要部署其他程序、其他后台进程，以提高性能。如果内存足够大，比如：128GB、256GB，单节点是浪费，建议通过虚拟化方式切分开。

解释：

目录结构
Type       Description       Default Location       Setting
home       Elasticsearch 主目录或 $ES_HOME       Directory created by unpacking the archive
bin       二进制脚本，包括用于启动节点的 elasticsearch 和用于安装插件的 elasticsearch-plugin       $ES_HOME/bin
conf       配置文件，包括但不限于elasticsearch.yml       $ES_HOME/config       ES_PATH_CONF
conf       为传输层和 HTTP 层生成 TLS 密钥和证书       $ES_HOME/config/certs
data       节点上分配的每个索引/分片的数据文件的位置       $ES_HOME/data       path.data
logs       日志文件位置       $ES_HOME/logs       path.logs
plugins       插件文件位置。每个插件将包含在一个子目录中       $ES_HOME/plugins
repo       共享文件系统存储库位置。可以容纳多个位置。文件系统存储库可以放置在此处指定的任何目录的任何子目录中       Not configured       path.repo

集群名称设置：cluster.name:
节点名称：node.name:
网络主机设置：network.host:
发现形成集群：discovery.seed_hosts:
选举主节点master资格的节点：cluster.initial_master_nodes:
设置集群间通信端口：transport.port:
设置数据存放位置：path.data:
设置日志存放位置：path.logs:

cluster.name: CollectorDBCluster
path.data:/data/elasticsearch/data
path.logs: /data/cusc-logs
network.host: 10.153.61.71
http.pち中o砗rt: 9200
node.name:node-1
cluster.initial master nodes: ["node-1”1
各配置项含义:
cluster.name 集群名称，各节点配成相同的集群名称。
node.name 节点名称，各节点配置不同。
node.master 指示某个节点是否符合成为主节点的条件。
node.data 指示节点是否为数据节点。数据节点包含并管理索引的一部分,
path.data 数据存储目录。
path.logs 日志存储目录。
bootstrap.memory lock 内存锁定，是否禁用交换
bootstrap.system call filter 系统调用过滤器。
network.host 绑定节点IP。
http.port 端囗。

启动集群：

配置文件属组权限，并启动
[root@it-elassearch elasticsearch]# chown -R es:es elastic-cluster1/
[root@it-elassearch-2 elasticsearch]# chown -R es:es elastic-cluster2/

如果直接用es账号配置的，可以不做上面的操作：

如果添加鉴权配置请按照此处配置，不配置即可略过：

es开启认证详细步骤：

1、生成证书：
[es@it-elassearch elasticsearch-8.15.0]$ ./bin/elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
* The CA certificate
* The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: 【直接回车】
Enter password for elastic-stack-ca.p12 : 【输入密码】

完成后会生成elastic-stack-ca.p12 文件

2、生成密匙：

[es@it-elassearch elasticsearch-8.15.0]$ ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
* By default, this generates a single certificate and key for use
   on a single instance.
* The '-multiple' option will prompt you to enter details for multiple
   instances and will generate a certificate and key for each one
* The '-in' option allows for the certificate generation to be automated by describing
   the details of each instance in a YAML file

* An instance is any piece of the Elastic Stack that requires an SSL certificate.
   Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
   may all require a certificate and private key.
* The minimum required value for each instance is a name. This can simply be the
   hostname, which will be used as the Common Name of the certificate. A full
   distinguished name may also be used.
* A filename value may be required for each instance. This is necessary when the
   name would result in an invalid file or directory name. The name provided here
   is used as the directory name (within the zip) and the prefix for the key and
   certificate files. The filename is required if you are prompted and the name
   is not displayed in the prompt.
* IP addresses and DNS names are optional. Multiple values can be specified as a
   comma separated string. If no IP addresses or DNS names are provided, you may
   disable hostname verification in your SSL configuration.

* All certificates generated by this tool will be signed by a certificate authority (CA)
   unless the --self-signed command line option is specified.
   The tool can automatically generate a new CA for you, or you can provide your own with
   the --ca or --ca-cert command line options.

By default the 'cert' mode produces a single PKCS#12 output file which holds:
* The instance certificate
* The private key for the instance certificate
* The CA certificate

If you specify any of the following options:
* -pem (PEM formatted output)
* -multiple (generate multiple certificates)
* -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (elastic-stack-ca.p12) : 【输入密码】
Please enter the desired output file [elastic-certificates.p12]:    【直接回车】
Enter password for elastic-certificates.p12 :  【输入密码】

Certificates written to /data/elasticsearch/elastic-cluster1/elasticsearch-8.15.0/elastic-certificates.p12

This file should be properly secured as it contains the private key for
your instance.
This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.

此操作中间需要输入刚才设置的密码就直接输入即可，需要输入路劲的地方可以不输入，直接回车就会生成一个文件在当前目录下：
elastic-certificates.p12

3、将凭证移至相应的目录即可：

##将凭证迁移到指定的目录：
创建目录：
mkdir -p ./config/certificates/
移动凭证到指定目录下：

[es@it-elassearch elasticsearch-8.15.0]$ mv elastic-certificates.p12 elastic-stack-ca.p12 ./config/certificates/

4、复制凭证到每个节点上：（使用scp或者rsync方式即可）
[es@it-elassearch elasticsearch-8.15.0]$ rsync -azvP -e 'ssh -p 60028' config/certificates/ es@172.24.110.126:/data/elasticsearch/elastic-cluster2/elasticsearch-8.15.0/config/certificates/
es@172.24.110.126's password:
sending incremental file list
./
elastic-certificates.p12
      3,596 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3)
elastic-stack-ca.p12
      2,672 100% 2.55MB/s 0:00:00 (xfr#2, to-chk=0/3)

sent 6,314 bytes  received 57 bytes  1,415.78 bytes/sec
total size is 6,268  speedup is 0.98

5、修改配置文件：
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Type,Content-Length
xpack.security.enabled: true
#xpack.security.authc.accept_default_password: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /data/elasticsearch/elastic-cluster/elasticsearch-8.15.0/config/certificates/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /data/elasticsearch/elastic-cluster/elasticsearch-8.15.0/config/certificates/elastic-certificates.p12

加权一样需要切换到es账号：

在各个节点上添加密码：

[es@it-elassearch elasticsearch-8.15.0]$ ./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
The elasticsearch keystore does not exist. Do you want to create it? [y/N]y
Enter value for xpack.security.transport.ssl.keystore.secure_password:

输入密码：第一次输入密码上面配置的

[es@it-elassearch elasticsearch-8.15.0]$ ./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
Enter value for xpack.security.transport.ssl.truststore.secure_password:

输入密码：第二次输入上面的密码：

接下来和没有做鉴权的一样，逐个启动集群：

切换到其它用户，root用户不能启动ES：su es

[es@it-elassearch elasticsearch-8.15.0]$ bin/elasticsearch -d
.......
Oct 24, 2024 5:33:34 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
[2024-10-24T17:33:40,246][INFO ][o.e.n.NativeAccess    ] [it-elassearch] Using native vector library; to disable start with -Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false
[2024-10-24T17:33:40,727][INFO ][o.e.n.NativeAccess    ] [it-elassearch] Using [jdk] native provider and native methods for [Linux]
[2024-10-24T17:33:41,119][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [it-elassearch] Java vector incubator API enabled; uses preferredBitSize=128; floating-point vectors only
[2024-10-24T17:33:42,185][INFO ][o.e.n.Node             ] [it-elassearch] version[8.15.0], pid[8520], build[tar/1a77947f34deddb41af25e6f0ddb8e830159c179/2024-08-05T10:05:34.233336849Z], OS[Linux/3.10.0-1160.24.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/22.0.1/22.0.1+8-16]
.......
[2024-10-24T17:34:27,594][WARN ][o.e.c.c.ClusterFormationFailureHelper] [it-elassearch] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [it-elassearch, it-elassearch-2] to bootstrap a cluster: have discovered [{it-elassearch}{1TZ7_AjMQBm4NUw73Dr9eQ}{wrEeokvZTM-NfqrlNd_FSQ}{it-elassearch}{172.24.110.125}{172.24.110.125:9300}{cdfhilmrstw}{8.15.0}{7000099-8512000}]; discovery will continue using [172.24.110.126:9300] from hosts providers and [{it-elassearch}{1TZ7_AjMQBm4NUw73Dr9eQ}{wrEeokvZTM-NfqrlNd_FSQ}{it-elassearch}{172.24.110.125}{172.24.110.125:9300}{cdfhilmrstw}{8.15.0}{7000099-8512000}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/ ... roubleshooting.html
[2024-10-24T17:34:27,609][INFO ][o.e.h.AbstractHttpServerTransport] [it-elassearch] publish_address {172.24.110.125:9200}, bound_addresses {[::]:9200}
[2024-10-24T17:34:27,637][INFO ][o.e.n.Node             ] [it-elassearch] started {it-elassearch}{1TZ7_AjMQBm4NUw73Dr9eQ}{wrEeokvZTM-NfqrlNd_FSQ}{it-elassearch}{172.24.110.125}{172.24.110.125:9300}{cdfhilmrstw}{8.15.0}{7000099-8512000}{ml.max_jvm_size=4294967296, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8200949760, ml.allocated_processors=4, ml.allocated_processors_double=4.0}

[es@it-elassearch elasticsearch-8.15.0]$ netstat -ntlp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address          Foreign Address       State    PID/Program name
tcp       0    0 127.0.0.1:25          0.0.0.0:*             LISTEN    -
tcp       0    0 0.0.0.0:60028          0.0.0.0:*             LISTEN    -
tcp6    0    0 :::9300                :::*                   LISTEN    8520/java
tcp6    0    0 ::1:25                :::*                   LISTEN    -
tcp6    0    0 :::60028             :::*                   LISTEN    -
tcp6    0    0 :::9200                :::*                   LISTEN    8520/java

[es@it-elassearch-1 elastic-cluster1]$ ./elasticsearch-8.15.0/bin/elasticsearch-create-enrollment-token -s kibana

ERROR: [xpack.security.enrollment.enabled] must be set to `true` to create an enrollment token, with exit code 78

在elasticsearch-8.15.0/config/elasticsearch.yml 添加配置：

[es@it-elassearch-1 elastic-cluster1]$ vim elasticsearch-8.15.0/config/elasticsearch.yml

xpack.security.enrollment.enabled: true

保存再次执行：

[es@it-elassearch-1 elastic-cluster1]$ ./elasticsearch-8.15.0/bin/elasticsearch-create-enrollment-token -s kibana
Unable to create enrollment token for scope [kibana]

ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore, with exit code 73

如果你的kibana不在同一台主机上就需要加上http或https的访问路径生成，用""扩起来

admin · 发表于 2024-10-31 16:09:57

登录时提示要求身份验证与此站点的连接不安全：

设置密码：
[es@it-elassearch elasticsearch-8.15.0]$ ./bin/elasticsearch-setup-passwords interactive
******************************************************************************
Note: The 'elasticsearch-setup-passwords' tool has been deprecated. This command will be removed in a future release.
******************************************************************************

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

密码要记住。

admin · 发表于 2024-10-31 16:12:02

访问地址：http://172.24.110.125:9200/    输入账号密码即可：
{
"name": "it-elassearch",
"cluster_name": "essearch",
"cluster_uuid": "AB2CR6VfTCeR57LVvOGUuw",
"version": {
      "number": "8.15.0",
      "build_flavor": "default",
      "build_type": "tar",
      "build_hash": "1a77947f34deddb41af25e6f0ddb8e830159c179",
      "build_date": "2024-08-05T10:05:34.233336849Z",
      "build_snapshot": false,
      "lucene_version": "9.11.1",
      "minimum_wire_compatibility_version": "7.17.0",
      "minimum_index_compatibility_version": "7.0.0"
},
"tagline": "You Know, for Search"
}

admin · 发表于 2024-12-2 15:50:06

3）更改 elasticsearch 主配置文件
[root@elk-node1 ~]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster 集群名字
node.name: elk-node1 节点名字
path.data: /data/elk_data 数据存放路径
path.logs: /var/log/elasticsearch/ 日志存放路径
bootstrap.memory_lock: false 不在启动的时候锁定内存
network.host: 0.0.0.0 提供服务绑定的 IP 地址，0.0.0.0 代表所有地址
http.port: 9200 侦听端口为 9200
discovery.zen.ping.unicast.hosts: ["elk-node1", "elk-node2"] 集群发现通过单播实现
[root@elk-node2 elasticsearch]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: elk-node2
path.data: /data/elk_data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elk-node1", "elk-node2"]

admin · 发表于 2024-12-3 09:30:45

http://172.24.110.125:9200/_cluster/health?pretty
{
"cluster_name": "elasticsearch",
"status": "green",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 107,
"active_shards": 214,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}

admin · 发表于 2024-12-3 09:36:37

编写 logstash 配置文件 apache_log.conf 搜集 apache 日志并执行

[root@apache conf.d]# cat apache_log.conf
input {
file {
path => "/etc/httpd/logs/access_log"
type => "access"
start_position => "beginning"
}
file {
path => "/etc/httpd/logs/error_log"
type => "error"
start_position => "beginning"
}
}
output {
if [type] == "access" {
elasticsearch {
hosts => ["192.168.10.181:9200"]
index => "apache_access-%{+YYYY.MM.dd}"
}
}
if [type] == "error" {
elasticsearch {
hosts => ["192.168.10.181:9200"]
index => "apache_error-%{+YYYY.MM.dd}"
}
}
}

[root@apache ]# /usr/share/logstash/bin/logstash -f apache_log.conf

admin · 发表于 2024-12-5 14:19:46

安装 logstash 并做一些日志搜集输出到 elasticsearch 中
1)在 elk-node1 上安装
[root@elk-node1 ~]# rpm -ivh logstash-5.5.1.rpm
警告：logstash-5.5.1.rpm: 头 V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中... ################################# [100%]
正在升级/安装...
1:logstash-1:5.5.1-1 ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
[root@elk-node1 ~]# systemctl start logstash.service
[root@elk-node1 ~]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/
2）logstash 基本使用
Logstash 使用管道方式进行日志的搜集处理和输出。有点类似 linux 系统的管道命令 xxx | ccc
| ddd，xxx 执行完了会执行 ccc，然后执行 ddd。
在 logstash 中，包括了三个阶段:
输入 input --> 处理 filter（不是必须的） --> 输出 output

每个阶段都由很多的插件配合工作，比如 file、elasticsearch、redis 等等。
每个阶段也可以指定多种方式，比如输出既可以输出到 elasticsearch 中，也可以指定到 stdout
在控制台打印。
由于这种插件式的组织方式，使得 logstash 变得易于扩展和定制。
logstash 命令行中常用的命令:
-f：通过这个命令可以指定 Logstash 的配置文件，根据配置文件配置 logstash
-e：后面跟着字符串，该字符串可以被当做 logstash 的配置（如果是“” 则默认使用 stdin
作为输入，stdout 作为输出）
-t：测试配置文件是否正确，然后退出
启动一个 logstash,-e：在命令行执行；input 输入，stdin 标准输入，是一个插件；output 输
出，stdout：标准输出
[root@elk-node1 ~]# logstash -e 'input { stdin{} } output { stdout{} }'
22:50:19.889 [Api Webserver] INFO logstash.agent - Successfully started Logstash API
endpoint {:port=>9600}
www.baidu.com 输入
2017-08-09T14:51:26.277Z elk-node1 www.baidu.com 输出
www.sina.com.cn 输入
2017-08-09T14:51:30.971Z elk-node1 www.sina.com.cn 输出
使用 rubydebug 显示详细输出，codec 为一种编解码器
[root@elk-node1 ~]# logstash -e 'input { stdin{} } output { stdout{ codec
=>rubydebug} }'
22:54:23.075 [Api Webserver] INFO logstash.agent - Successfully started Logstash API
endpoint {:port=>9600}
www.baidu.com 输入
{
"@timestamp" => 2017-08-09T14:54:31.236Z,
"@version" => "1",
"host" => "elk-node1",
"message" => "www.baidu.com"
}
www.sina.com.cn 输入
{
"@timestamp" => 2017-08-09T14:54:39.412Z,
"@version" => "1",
"host" => "elk-node1",
"message" => "www.sina.com.cn"
}
使用 logstash 将信息写入到 elasticsearch 中
[root@elk-node1 ~]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=>
["192.168.10.181:9200"]} }'
22:59:03.834 [Api Webserver] INFO logstash.agent - Successfully started Logstash API
endpoint {:port=>9600}
www.baidu.com
www.sina.com.cn
www.google.com
在 elasticsearch 中查看 logstash 新加的索引

3）logstash 配置文件使用
logstash 配置文件基本上由三部分组成，input、output 以及用户需要才添加的 filter，
因此标准的配置文件格式如下：
input {...}
filter {...}
output {...}
在每个部分中，也可以指定多个访问方式，例如我想要指定两个日志来源文件，则可以这样写：
input {
file { path =>"/var/log/messages" type =>"syslog"}
file { path =>"/var/log/apache/access.log" type =>"apache"}
}
下面是一个收集系统日志的配置文件例子，将其放到/etc/logstash/conf.d/目录中，
logstash 启动的时候便会加载。注意要给 logstash 读取日志文件的权限。
[root@elk-node1 logstash]# chmod o+r /var/log/messages
[root@elk-node1 logstash]# ll /var/log/messages
-rw----r--. 1 root root 2566754 8 月 9 23:25 /var/log/messages
[root@elk-node1 ~]# cd /etc/logstash/conf.d/
[root@elk-node1 conf.d]# cat system.conf
input {
file { 从文件中读取
path => "/var/log/messages" 文件路径
type => "system"
start_position => "beginning" 是否从头开始读取
}
}
output {
elasticsearch { 输出到 elasticsearch 中
hosts => ["192.168.10.181:9200"] elasticsearch 主机地址和端口
index => "system-%{+YYYY.MM.dd}" 索引名称
}
}
重启 logstash 服务
[root@elk-node1 logstash]# systemctl restart logstash
在浏览器中即可看到新加索引及其内容

		自动登录	找回密码
密码			注册