k8s nginx ingress日志收集到ELK并分析

# k8s nginx ingress日志收集并分析

## 版本说明

| 组件名称      | 版本     |
| ------------- | -------- |
| ELK           | 7.9.3    |
| filebeat      | 7.9.3    |
| k8s           | v1.24.17 |
| ingress-nginx | v1.8.0   |

## 效果图
![](/media/202402/2024-02-04_085819_753026.png)

## 修改ingress日志输出格式

在ingress-nginx-controller中添加

```shell
log-format-escape-json: true
log-format-upstream: |
{
  "@timestamp": "$time_iso8601",
  "domain": "$server_name",
  "hostname": "$hostname",
  "remote_user": "$remote_user",
  "client": "$remote_addr",
  "proxy_protocol_addr": "$proxy_protocol_addr",
  "@source": "$server_addr",
  "host": "$http_host",
  "request": "$request_uri",
  "url": "$uri",
  "args": "$args",
  "upstreamaddr": "$upstream_addr",
  "status": "$status",
  "upstream_status": "$upstream_status",
  "size": "$body_bytes_sent",
  "responsetime": "$request_time",
  "upstreamtime": "$upstream_response_time",
  "proxy_upstream_name": "$proxy_upstream_name",
  "proxy_alternative_upstream_name": "$proxy_alternative_upstream_name",
  "xff": "$http_x_forwarded_for",
  "upstream_response_length": "$upstream_response_length",
  "referer": "$http_referer",
  "http_user_agent": "$http_user_agent",
  "request_length": "$request_length",
  "req_id": "$req_id",
  "request_method": "$request_method",
  "request_body": "$request_body",
  "scheme": "$scheme",
  "file_dir": "$request_filename",
  "k8s_namespace": "$namespace",
  "k8s_ingress_name": "$ingress_name",
  "k8s_service_name": "$service_name",
  "k8s_service_port": "$service_port",
  "https": "$https"
}
```

```shell
备注：
log-format-escape-json: true
这个参数表示让nginx日志中的字符都转义后打印
不然传到logstash 中 json插件解析的时候会报错 
报错 JSON parse error：Unrecognized character escape ‘x‘
nginx 配置文件中可以用下面的方法改
log_format json escape=json '{"@timestamp":"$time_iso8601",'
                        '"@source":"$server_addr",'
                        '"hostname":"$hostname",'
                        '"ip":"$remote_addr",'
                        '"xff":"$http_x_forwarded_for",'
                        '"client":"$remote_addr",'
                        '"remote_user":"$remote_user",'
                        '"request_method":"$request_method",'
                        '"request_body": "$request_body",'
                        '"scheme":"$scheme",'
                        '"domain":"$server_name",'
                        '"referer":"$http_referer",'
                        '"request":"$request_uri",'
                        '"args":"$args",'
                        '"size":$body_bytes_sent,'
                        '"status": $status,'
                        '"responsetime":$request_time,'
                        '"upstreamtime":"$upstream_response_time",'
                        '"upstreamaddr":"$upstream_addr",'
                        '"upstream_status": "$upstream_status",'
                        '"http_user_agent":"$http_user_agent",'
                        '"http_host":"$host",'
                        '"url":"$uri",'
                        '"connection_requests": $connection_requests,'
                        '"https":"$https"'
                        '}';

```

## 在ingress 机器上部署filebeat

```shell
rpm -ivh filebeat-7.9.3-x86_64.rpm
cd /etc/filebeat/
ls
vim filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/pods/ingress-nginx_ingress-nginx-controller-*/controller/0.log
  tags: ["ingress"]
  fields_under_root: true
  fields:
    project: k8s-ingress-prd
    app: ingress

output.logstash:
  hosts: ["10.7.6.123:5060"]

```

## logstash 接收日志并格式化

```shell
2024-02-03T17:07:13.022151969+08:00 stdout F {......}
containerd会把我们上面的log_format 格式化的lson日志前面添加时间戳变成text日志
所以我们需要在logstash 中用 grok 匹配
   grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} stdout F %{GREEDYDATA:log_message}" }
  }

```

logstash 配置文件如下

```shell
input {
    beats {
        host => "0.0.0.0"
        port => 5060
    }
}
filter {
    if [app]  == "ingress"
        {
    #grok 匹配出日志中的json方便解析    
    grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} stdout F %{GREEDYDATA:log_message}" }
  }
    json {
        source => "log_message"
        target => "parsed_log"
    }
    #把公网ip匹配出来
    grok {
    match => { "message" => "\"xff\":\"%{IP:remote_ip}" }
  }
    #用geoip解析公网地址
    geoip {
    source => "remote_ip" #这个字段一定要在第一层，如果是在第二层parsed_log.xff geoip会识别不到
    target => "geoip"
    database => "/tqls_system/apps/logstash/GeoLite2-City.mmdb"

}
    mutate {
   remove_field => ["log_message"]
   add_field => {
         "[@metadata][target_index]" => "prd-k8s-ingress-%{+YYYY.MM.dd}"
          }
    }
  }
 else {
mutate {
add_field => {
"[@metadata][target_index]" => "prd-k8s-%{[app]}-%{+YYYY.MM.dd}"
             }
        }
      }
}
output {
    elasticsearch {
        hosts => ["10.7.6.219:9200"]
        index => "%{[@metadata][target_index]}"
      }
}
```

## 画热力图

在画热力图时有个坑

>https://www.zhihu.com/tardis/bd/art/412912721?source_id=1001
>
>根据官方文档的指引，想要在Kibana中使用地图相关的功能，在索引里面必须包含`geo_point`类型的字段。被正确识别的es内容在Kibana中会被展示为如下的样子......
>
>但当我们通过修改logstash配置文件，把日志内容存储到`logstash-nginx-log-*`，我们会看到，`geoip.location`会被自动识别成geo_point类型......

我们的索引名称是 prd-k8s-ingress-* elasticsearch在处理这种自定义的索引的时候，如果没有额外进行设置，会默认把`geoip.location`这一字段存储为float或者string

我们需要新建模板 在开发工具里控制台里运行以下代码

```shell
PUT  /_template/ingress
{
    "order" : 0,
    "template": "ingress" ,
    "version" : 60001,
    "index_patterns" : [
      "prd-k8s-ingress-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "path_match" : "message",
            "mapping" : {
              "norms" : false,
              "type" : "text"
            },
            "match_mapping_type" : "string"
          }
        },
        {
          "string_fields" : {
            "mapping" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "match_mapping_type" : "string",
            "match" : "*"
          }
        }
      ],
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "geoip" : {
          "dynamic" : true,
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        },
        "@version" : {
          "type" : "keyword"
        }
      }
    },
    "aliases" : { }
}
```

```shell
#获取全部模板
GET /_template/*
#获取某一模板的具体信息
GET /_template/logstash
#删除模板
DELETE  /_template/api
```