Architecture Overview
The Forwarder
- runs on every application server
- send logs somewhere else
- typically file based
- lightweight
The Log Aggregator
- filters/modifies logs
- send logs to index/s3
- horizontally scalable
- should have robust buffering
The Index
- takes in structured log messages
- not a log archive
- just use Elasticsearch
The Analyzer
- UI on top of index
- robust querying
- data visualizations
Our Central Logging Architecture
Fluentd
- written in Ruby (!!!)
- 700+ community plugins
- TLS support and flexible auth
Kibana
- Angular and Node
- Lucene query syntax
- plugin architecture