While playing around with fluentd the need came up to extract data nested deep into the logging input and add it as a flat field to the output. If you’re not familiar with fluentd (I’m not): it’s similar to logstash in that it collects logs from a variety of sources, filters, transforms and categorises them and either stores them in files or stream them to other sinks. It also supports a variety of plug-ins.
I find the concepts and architecture well suited to the task, the documentation less so – I feel I spent way too much time on figuring out how to do something simple, but at least the effort wasn’t in vain, this post came out 🙂
Fluentd reads log entries and splits them into records; so each entry is a record with a tag, a timestamp and several fields. Records are best imagined as JSON objects: fields can be flat (numbers and strings) or structured objects. I had the need to extract data deeply nested into the record and move it into a first level field “at the root” of the record.
The input records look something like this:
{
"field1":"value1",
...
"appdata":
{
"metadata":["the name of the app","some","other","entries"]
}
}
The field I’m looking for is accessible as (javascript notation) this.appdata.metadata[0]
The solution requires a plugin (installed by default) called record transformer:
<source>
@type http # reads log entries over http
port 9880 # opens a socket
bind 0.0.0.0
<parse>
@type json # records are expected in JSON format
</parse>
tag logs
</source>
<filter logs> # extracts the input record appdata.metadata[0] into a new field app_name
@type record_transformer
enable_ruby true
<record>
app_name ${record["appdata"]["metadata"][0]}
</record>
</filter>
<match logs> # writes results to the output
@type stdout
</match>
George's Techblog 