nginx - Logstash Grokked Data Does Not Reach ElasticSearch -
i have simple logstash 1.4.1 setup , data seems correct in stdout, in es/kibana "message" field not overwritten.
logstash.conf
input { file { path => "/var/log/nginx/access.log" type => "nginx_access" } } filter { if [type] == "nginx_access" { grok { match => [ "message", "%{nginxaccess}" ] overwrite => [ "message" ] } } } output { elasticsearch { protocol => "http" } stdout { codec => rubydebug } }
nginx pattern
ngusername [a-za-z\.\@\-\+_%]+ nguser %{ngusername} nginxaccess %{iporhost:clientip} %{nguser:ident} %{nguser:auth} \[%{httpdate:timestamp}\] "%{word:verb} %{uripathparam:request} http/%{number:httpversion}" %{number:response} (?:%{number:bytes}|-) (?:"(?:%{uri:referrer}|-)"|%{qs:referrer}) %{qs:agent}
stdout
{ "message" => "127.0.0.1 - - [27/nov/2014:13:43:53 +0100] \"get /favicon.ico http/1.1\" 502 574 \"-\" \"mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, gecko) chrome/38.0.2125.122 safari/537.36\"", "@version" => "1", "@timestamp" => "2014-11-27t12:43:54.230z", "type" => "nginx_access", "host" => "laptop", "path" => "/var/log/nginx/access.log", "clientip" => "127.0.0.1", "ident" => "-", "auth" => "-", "timestamp" => "27/nov/2014:13:43:53 +0100", "verb" => "get", "request" => "/favicon.ico", "httpversion" => "1.1", "response" => "502", "bytes" => "574", "agent" => "\"mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, gecko) chrome/38.0.2125.122 safari/537.36\"" }
what doing wrong? how can debug?
edit
get api showing fields in _source, not indexed. do?
{ "_index":"logstash-2014.11.27", "_type":"nginx_access4", "_id":"aunxr34z5dzktb-2teug", "_version":1, "found":true, "_source":{ "message":"127.0.0.1 - - [27/nov/2014:13:43:53 +0100] \"get / http/1.1\" 502 574 \"-\" \"mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, gecko) chrome/38.0.2125.122 safari/537.36\"", "@version":"1", "@timestamp":"2014-11-27t12:43:54.230z", "type":"nginx_access", "host":"laptop", "path":"/var/log/nginx/access.log", "clientip":"127.0.0.1", "ident":"-", "auth":"-", "timestamp":"27/nov/2014:13:43:53 +0100", "verb":"get", "request":"/", "httpversion":"1.1", "response":"502", "bytes":"574", "agent":"\"mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, gecko) chrome/38.0.2125.122 safari/537.36\"" } }
reloading index pattern's field list helped. created 1 before logging data.
Comments
Post a Comment