elk日志系统实践

1.Elk Stack简介

Filebeat: 一个轻量级开源日志文件数据搜集器,基于 Logstash-Forwarder 源代码开发,是对它的替代。在需要采集日志数据的 server 上安装 Filebeat,并指定日志目录或日志文件后,Filebeat 就能读取数据,迅速发送到 Logstash 进行解析,亦或直接发送到 Elasticsearch 进行集中式存储和分析
Logstash: 数据收集引擎。它支持动态的从各种数据源搜集数据,并对数据进行过滤、分析、丰富、统一格式等操作,然后存储到用户指定的位置
Elasticsearch: 分布式搜索和分析引擎,具有高可伸缩、高可靠和易管理等特点。基于 Apache Lucene 构建,能对大容量的数据进行接近实时的存储、搜索和分析操作。通常被用作某些应用的基础搜索引擎,使其具有复杂的搜索功能
Kibana: 数据分析和可视化平台。通常与 Elasticsearch 配合使用,对其中数据进行搜索、分析和以统计图表的方式展示

2.安装

1.filebeat安装    
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz
    tar -xzf filebeat-6.4.0-linux-x86_64.tar.gz
    官方文档
        https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-getting-started.html
 2.logstash安装
    wget https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.tar.gz
    tar -xzf logstash-5.2.2.tar.gz
    官方文档
        https://www.elastic.co/guide/en/logstash/5.2/index.html
 3.elasticsearch安装
    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.zip
    unzip elasticsearch-5.2.2.zip
    官方文档
        https://www.elastic.co/guide/en/elasticsearch/reference/5.2/index.html
4.kibana安装
    wget https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-linux-x86_64.tar.gz
    tar -xzf kibana-5.2.2-linux-x86_64.tar.gz
    官方文档
        https://www.elastic.co/guide/en/kibana/5.2/index.html

3.配置

1. filebeat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# filebeat.yml
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

# Change to true to enable this input configuration.
enabled: true

# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log 日志文件路径
- /tmp/ccapi.log
#- c:\programdata\elasticsearch\logs\*

document_type: ccapi
tags: ["ccapi"] # 增加tags

### Multiline options
multiline.pattern: ^\[ # 多行异常错误堆栈文本处理
multiline.negate: true
multiline.match: after

# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
fields:
type: "ccapi" # fields下面新增type字段, 用于区分业务

#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
setup.template.name: "logstash" # 设置es索引的名称, 默认为filebeat
setup.template.pattern: "logstash-*"
setup.template.json.enabled: true # 加载json的模版配置, 这里使用Logstash默认生成的es模板
setup.template.json.path: "/home/phenix/logstash/logstash.template.json" # json的模版配置路径

#============================== Kibana =====================================
setup.kibana:
host: "localhost:5601"
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["localhost:5044"]

2. logstash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 input {
beats {
port => 5044
}
}

filter {

if [fields][type] == "ccapi" { # 如果业务是ccapi相关
json { # 因为日志已经json化,且以字符串的形式,存在message字段
source => "message" # json处理message字段
target => "msg" # 将json完的数据, 存在msg字段下(避免如果
# json内容中本身也含有type,host,path字段,
# 那么解析后将覆盖掉logstash默认的这三个字段)
remove_field => ["message"] # 将message字段移除
}
}
}


output {
stdout { codec => rubydebug }

if [fields][type] == "ccapi" { # 如果业务是ccapi
elasticsearch {
hosts => ["192.168.1.33:9200"]
index => "logstash-ccapi-%{+YYYY.MM.dd}"# 指定索引, 业务+日期切片
document_type => "ccapi" # 指定索引下的类型
flush_size => 2000 # 攒到2000条数据后发送
idle_flush_time => 10 # 每10s发送一次
sniffing => true
template_overwrite => true
}
}
}
```

运行结果
{"msg": {
            "msg": "Request",
            "respContentType": "application/json",
            "reqUri": "/api/room/create?",
            "reqMethod": "POST",
            "respTimeMs": 23,
            "logSource": "web",
            "ascTime": "2019-06-04T05:54:58.623Z",
            "type": "request",
            "userId": null,
            "respStat": 200,
            "token": null,
            "reqId": "49136896868d11e98923b083febfc53a",
            "respSizeB": 239,
            "reqData": null,
            "service": "ccapi",
            "clientIp": "192.168.1.73",
            "serverIp": "localhost",
            "device": "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)"
          },
          "input": {
            "type": "log"
          },
          "@timestamp": "2019-06-04T05:54:55.678Z",
          "offset": 2720807,
          "@version": "1",
          "host": {
            "name": "localhost"
          },
          "beat": {
            "hostname": "localhost",
            "name": "localhost",
            "version": "6.4.0"
          },
          "prospector": {
            "type": "log"
          },
          "source": "/tmp/ccapi.log",
          "fields": {
            "type": "ccapi"
          },
          "tags": [
            "ccapi",
            "beats_input_codec_plain_applied"
          ]}
1
2
3
4
5
6
### 3. elasticsearch

```shell
# elasticsearch.yml
network.host: 0.0.0.0
http.port: 9200

4.kibana

1
2
3
# kibana.yml
server.host: 0.0.0.0
elasticsearch.url: "http://192.168.1.33:9200"

4.其余

1.手动导出logstash默认模版

https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template-es5x.json

2.手动导入模版到es

curl -XPUT 'http://localhost:9200/_template/logstash' -d@模版路径

3.查看es索引

curl 'localhost:9200/_cat/indices?v'

4.查看某个索引下的数据

curl 'localhost:9200/索引名/类型/_search?'

5.删除某个索引

curl -XDELETE 'localhost:9200/索引名'

5.参考

ES学习 https://fuxiaopang.gitbooks.io/learnelasticsearch/
ES模版设置https://birdben.github.io/2016/12/22/Logstash/Logstash%E5%AD%A6%E4%B9%A0%EF%BC%88%E5%85%AD%EF%BC%89elasticsearch%E6%8F%92%E4%BB%B6%E2%80%94%E2%80%94%E8%AE%BE%E7%BD%AEES%E7%9A%84Template/
https://www.jianshu.com/p/1f67e4436c37
logstash字段详解 http://www.51niux.com/?id=205
ELK架构演进 https://github.com/jasonGeng88/blog/blob/master/201703/logstash_deploye_scale.md
ELK/kibana表盘设置
http://docs.flycloud.me/docs/ELKStack/logstash/get-start/index.html
https://github.com/jasonGeng88/blog/blob/master/201703/elk_parse_log.md
https://www.ibm.com/developerworks/cn/opensource/os-cn-elk-filebeat/index.html

坚持原创技术分享,您的支持将鼓励我继续创作!