Hey guys, so I created an ELK server to centralize logging for my FreeIPA server. I got Filebeat running on the IPA server to send log data to my ELK server. It was running great, super fast, had pumped in 750k hits worth of log data in a few minutes.
UNTIL I ran a
curl -XDELETE elk:9200/filebeat-*/ to delete the data so I could fix the timestamp formatting in my Logstash config. I fixed the timestamp thing, cleared the data, and restarted Filebeat.
Now, the log data is being pumped in at like 1/50th the speed it did the first time (750k vs 5k in the same amount of time). I rebooted my ELK server, tried it again, and it was fast. Ran the XDELETE and it was slow AGAIN???
What's causing this? I have no idea because everything is working, but it's just SLOW. But at the same time I don't see resources being really used, it's all <1.0 LOAD with >16GB of free MEM??? I have reason to believe it's something with the Elasticsearch because rebooting the ELK server performance was immediately better.
What I'm running:
- CentOS 6.7
- elasticsearch 2.1
- logstash 2.1
- kibana 4.3
- filebeat 1.0.1
Things I've tried before rebooting:
- restarting elasticsearch service on ELK
- restarting logstash service on ELK
- restarting kibana service on ELK
- restarting filebeat service on IPA
- making sure all 4 processes are not running with ps aux | grep [service]
- deleting all filebeat-* data in elasticsearch with XDELETE (causes the slowdown??)
- deleting and re-XPUT-ing the filebeat index template into elasticsearch
Here are some relevant config files:
- logstash config (on ELK server): http://pastebin.com/0VSDqZLU
- filebeat config (on IPA server): http://pastebin.com/M55yNVC3
[link][2 comments]