Available hekad plugins compiled with this version of hekad.
Connects to a remote AMQP broker (RabbitMQ) and retrieves messages from the specified queue. If the message is serialized by hekad’s AMQPOutput then the message will be de-serialized, otherwise the message will be run through the specified LoglineDecoder’s. As AMQP is dynamically programmable, the broker topology needs to be specified.
Parameters:
An AMQP connection string formatted per the RabbitMQ URI Spec.
AMQP exchange name
AMQP exchange type (fanout, direct, topic, or headers).
Whether the exchange should be configured as a durable exchange. Defaults to non-durable.
Whether the exchange is deleted when all queues have finished and there is no publishing. Defaults to auto-delete.
The message routing key used to bind the queue to the exchange. Defaults to empty string.
How many messages to fetch at once before message acks are sent. See RabbitMQ performance measurements for help in tuning this number. Defaults to 2.
Name of the queue to consume from, an empty string will have the broker generate a name for the queue. Defaults to empty string.
Whether the queue is durable or not. Defaults to non-durable.
Whether the queue is exclusive (only one consumer allowed) or not. Defaults to non-exclusive.
Whether the queue is deleted when the last consumer un-subscribes. Defaults to auto-delete.
List of logline decoder names used to transform a raw message body into a structured hekad message. These are skipped for serialized hekad messages.
Since many of these parameters have sane defaults, a minimal configuration to consume serialized messages would look like:
[AMQPInput]
url = "amqp://guest:guest@rabbitmq/"
exchange = "testout"
exchangeType = "fanout"
Or if using a logline decoder to parse OSX syslog messages may look like:
[AMQPInput]
url = "amqp://guest:guest@rabbitmq/"
exchange = "testout"
exchangeType = "fanout"
decoders = ["logparser", "leftovers"]
[logparser]
type = "LoglineDecoder"
MatchRegex = '\w+ \d+ \d+:\d+:\d+ \S+ (?P<Reporter>[^\[]+)\[(?P<Pid>\d+)](?P<Sandbox>[^:]+)?: (?P<Remaining>.*)'
[logparser.MessageFields]
Type = "amqplogline"
Hostname = "myhost"
Reporter = "%Reporter%"
Remaining = "%Remaining%"
Logger = "%Logger%"
Payload = "%Remaining%"
[leftovers]
type = "LoglineDecoder"
MatchRegex = '.*'
[leftovers.MessageFields]
Type = "drop"
Payload = ""
Listens on a specific UDP address and port for messages. If the message is signed it is verified against the signer name and specified key version. If the signature is not valid the message is discarded otherwise the signer name is added to the pipeline pack and can be use to accept messages using the message_signer configuration option.
Parameters:
An IP address:port on which this plugin will listen.
Optional TOML subsection. Section name consists of a signer name, underscore, and numeric version of the key.
The hash key used to sign the message.
Example:
[UdpInput]
address = "127.0.0.1:4880"
[UdpInput.signer.ops_0]
hmac_key = "4865ey9urgkidls xtb0[7lf9rzcivthkm"
[UdpInput.signer.ops_1]
hmac_key = "xdd908lfcgikauexdi8elogusridaxoalf"
[UdpInput.signer.dev_1]
hmac_key = "haeoufyaiofeugdsnzaogpi.ua,dp.804u"
Listens on a specific TCP address and port for messages. If the message is signed it is verified against the signer name and specified key version. If the signature is not valid the message is discarded otherwise the signer name is added to the pipeline pack and can be use to accept messages using the message_signer configuration option.
Parameters:
An IP address:port on which this plugin will listen.
Optional TOML subsection. Section name consists of a signer name, underscore, and numeric version of the key.
The hash key used to sign the message.
Example:
[TcpInput]
address = ":5565"
[TcpInput.signer.ops_0]
hmac_key = "4865ey9urgkidls xtb0[7lf9rzcivthkm"
[TcpInput.signer.ops_1]
hmac_key = "xdd908lfcgikauexdi8elogusridaxoalf"
[TcpInput.signer.dev_1]
hmac_key = "haeoufyaiofeugdsnzaogpi.ua,dp.804u"
Tails logfiles, creating a message for each line in each logfile being monitored. Logfiles are read in their entirety, and watched for changes. This input gracefully handles log rotation via the file moving but may lose a few log lines of using the truncation method of log rotation. It’s recommended to use log rotation schemes that move the logfile to another location to avoid possible loss of log lines.
In the event the logfile does not currently exist, it will be placed in an internal discover list, and checked for existence every discoverInterval milliseconds (5000ms or 5s) by default.
Parameters:
Each LogfileInput can have a single logfile to monitor.
The hostname to use for the messages, by default this will be the machines qualified hostname. This can be set explicitly to ensure its the correct name in the event the machine has multiple interfaces/hostnames.
During logfile rotation, or if the logfile is not originally present on the system, this interval is how often the existence of the logfile will be checked for. The default of 5 seconds is usually fine. This interval is in milliseconds.
How often the file descriptors for each file should be checked to see if new log data has been written. Defaults to 500 milliseconds. This interval is in milliseconds.
List of logline decoder names used to transform the log line into a structured hekad message.
Each LogfileInput may specify a logger name to use in the case an error occurs during processing of a particular line of logging text. By default, the logger name is set to the logfile name.
Heka will write out a journal to keep track of the last known read position of a logfile. By default, this will default to writing in /var/run/hekad/seekjournals/. The journal name will be the logger name with path separators and periods replaced with underscores.
When heka restarts, if a logfile cannot safely resume reading from the last known position, this flag will determine whether hekad will force the seek position to be 0 or the end of file. By default, hekad will resume reading from the start of file.
[LogfileInput]
logfile = "/var/log/opendirectoryd.log"
logger = "opendirectoryd"
[LogfileInput]
logfile = "/var/log/opendirectoryd.log"
Listens for statsd protocol counter, timer, or gauge messages on a UDP port, and generates Stat objects that are handed to a StatAccumulator for aggregation and processing.
Parameters:
An IP address:port on which this plugin will expose a statsd server. Defaults to “127.0.0.1:8125”.
Name of a StatAccumInput instance that this StatsdInput will use as its StatAccumulator for submitting received stat values. Defaults to “StatAccumInput”.
Example:
[StatsdInput]
address = ":8125"
stat_accum_input = "custom_stat_accumulator"
Provides an implementation of the StatAccumulator interface which other plugins can use to submit Stat objects for aggregation and roll-up. Accumulates these stats and then periodically emits a “stat metric” type message containing aggregated information about the stats received since the last generated message.
Parameters:
Specifies whether or not the aggregated stat information should be emitted in the message fields of the generated messages. Defaults to true. NOTE: At least one of ‘emit_in_payload’ or ‘emit_in_fields’ must be true or it will be considered a configuration error and the input won’t start.
Percent threshold to use for computing “upper_N%” type stat values. Defaults to 90.
Time interval (in seconds) between generated output messages. Defaults to 10.
String value to use for the Type value of the emitted stat messages. Defaults to “heka.statmetric”.
Starts a HTTP client which intermittently polls a URL for data. The entire response body is parsed by a decoder into a pipeline pack. Data is always fetched using HTTP GET and any errors are logged and are not fatal for the plugin.
Parameters:
A HTTP URL which this plugin will regularly poll for data. No default URL is specified.
Time interval (in seconds) between attempts to poll for new data. Defaults to 10.
The name of the decoder used to transform the response body text into a structured hekad message. No default decoder is specified.
Example:
[HttpInput]
url = "http://localhost:9876/"
ticker_interval = 5
decoder = "JsonDecoder"
A decoder may be specified for each encoding type defined in message.pb.go. Unless you are using a custom decoder you probably won’t need to specify these by hand, by default the JsonDecoder and ProtobufDecoder will be configured as if you had included the following configuration.
Example:
[JsonDecoder]
encoding_name = "JSON"
[ProtobufDecoder]
encoding_name = "PROTOCOL_BUFFER"
The JsonDecoder converts JSON serialized Heka messages to Message struct objects. The encoding_name setting means that this decoder should be used for any Heka protocol messages that have the encoding header of JSON. The ProtobufDecoder converts protocol buffers serialized messages to Message struct objects. The hekad protocol buffers message schema in defined in the message.proto file in the message package.
Note
These sections remain configurable explicitly in the configuration file for possible future use where a different Decoder may want to handle one of these encodings.
Decoder plugin that accepts messages of a specified form and generates new outgoing messages from extracted data, effectively transforming one message format into another. Can be combined w/ message_matcher capture groups (see matcher_capture_groups) to extract unstructured information from message payloads and use it to populate Message struct attributes and fields in a more structured manner.
Parameters:
Regular expression that must match for the decoder to process the message.
Subsection defining severity strings and the numerical value they should be translated to. hekad uses numerical severity codes, so a severity of WARNING can be translated to 3 by settings in this section.
Subsection defining message fields to populate and the interpolated values that should be used. Valid interpolated values are any captured in a regex in the message_matcher, and any other field that exists in the message. In the event that a captured name overlaps with a message field, the captured name’s value will be used. Optional representation metadata can be added at the end of the field name using a pipe delimiter i.e. ResponseSize|B = “%ResponseSize%” will create Fields[ResponseSize] representing the number of bytes. Adding a representation string to a standard message header name will cause it to be added as a user defined field i.e., Payload|json will create Fields[Payload] with a json representation.
Interpolated values should be surrounded with % signs, for example:
[my_decoder.message_fields]
Type = "%Type%Decoded"
This will result in the new message’s Type being set to the old messages Type with Decoded appended.
A formatting string instructing hekad how to turn a time string into the actual time representation used internally. Example timestamp layouts can be seen in Go’s time documetation.
Time zone in which the timestamps in the text are presumed to be in. Should be a location name corresponding to a file in the IANA Time Zone database (e.g. “America/Los_Angeles”), as parsed by Go’s time.LoadLocation() function (see http://golang.org/pkg/time/#LoadLocation). Defaults to “UTC”. Not required if valid time zone info is embedded in every parsed timestamp, since those can be parsed as specified in the timestamp_layout.
Example (Parsing Apache Combined Log Format):
[apache_transform_decoder]
type = "LoglineDecoder"
match_regex = '/^(?P<RemoteIP>\S+) \S+ \S+ \[(?P<Timestamp>[^\]]+)\] "(?P<Method>[A-Z]+) (?P<Url>[^\s]+)[^"]*" (?P<StatusCode>\d+) (?P<RequestSize>\d+) "(?P<Referer>[^"]*)" "(?P<Browser>[^"]*)"/'
timestamplayout = "02/Jan/2006:15:04:05 -0700"
[apache_transform_decoder.severity_map]
DEBUG = 1
WARNING = 2
INFO = 3
[apache_transform_decoder.message_fields]
Type = "ApacheLogfile"
Logger = "apache"
Url|uri = "%Url%"
Method = "%Method%"
Status = "%Status%"
RequestSize|B = "%RequestSize%"
Referer = "%Referer%"
Browser = "%Browser%"
Once a second a CounterFilter will generate a message of type heka.counter- output. The payload will contain text indicating the number of messages that matched the filter’s message_matcher value during that second (i.e. it counts the messages the plugin received). Every ten seconds an extra message (also of type heka.counter-output) goes out, containing an aggregate count and average per second throughput of messages received.
Parameters: None
Example:
[CounterFilter]
message_matcher = "Type != 'heka.counter-output'"
Filter plugin that accepts messages of a specfied form and uses extracted message data to generate statsd-style numerical metrics in the form of Stat objects that can be consumed by a StatAccumulator.
Parameters:
Subsection defining a single metric to be generated
Metric type, supports “Counter”, “Timer”, “Gauge”.
Metric name, must be unique.
Expression representing the (possibly dynamic) value that the StatFilter should emit for each received message.
Name of a StatAccumInput instance that this StatFilter will use as its StatAccumulator for submitting generate stat values. Defaults to “StatAccumInput”.
Example (Assuming you had TransformFilter inserting messages as above):
[StatsdInput]
address = "127.0.0.1:29301"
stat_accum_name = "my_stat_accum"
[my_stat_accum]
flushInterval = 5
[Hits]
type = "StatFilter"
stat_accum_name = "my_stat_accum"
message_matcher = 'Type == "ApacheLogfile"'
[Hits.Metric.bandwidth]
type = "Counter"
name = "httpd.bytes.%Hostname%"
value = "%Bytes%"
[Hits.Metric.method_counts]
type = "Counter"
name = "httpd.hits.%Method%.%Hostname%"
value = "1"
Note
StatFilter requires an available StatAccumulator to be running.
The sandbox filter provides an isolated execution environment for data analysis.
The sandbox manager provides dynamic control (start/stop) of sandbox filters in a secure manner without stopping the Heka daemon.
Connects to a remote AMQP broker (RabbitMQ) and sends messages to the specified queue. The message is serialized if specified, otherwise only the raw payload of the message will be sent. As AMQP is dynamically programmable, the broker topology needs to be specified.
Parameters:
An AMQP connection string formatted per the RabbitMQ URI Spec.
AMQP exchange name
AMQP exchange type (fanout, direct, topic, or headers).
Whether the exchange should be configured as a durable exchange. Defaults to non-durable.
Whether the exchange is deleted when all queues have finished and there is no publishing. Defaults to auto-delete.
The message routing key used to bind the queue to the exchange. Defaults to empty string.
Whether published messages should be marked as persistent or transient. Defaults to non-persistent.
Whether published messages should be fully serialized. If set to true then messages will be encoded to Protocol Buffers and have the AMQP message Content-Type set to application/hekad. Defaults to true.
Example (that sends log lines from the logger):
[AMQPOutput]
url = "amqp://guest:guest@rabbitmq/"
exchange = "testout"
exchangeType = "fanout"
message_matcher = 'Logger == "/var/log/system.log"'
Logs messages to stdout using Go’s log package.
Parameters:
If set to true, then only the message payload string will be output, otherwise the entire Message struct will be output in JSON format.
Example:
[counter_output]
type = "LogOutput"
message_matcher = "Type == 'heka.counter-output'"
payload_only = true
Writes message data out to a file system.
Parameters:
Full path to the output file.
Output format for the message to be written. Supports json or protobufstream, both of which will serialize the entire Message struct, or text, which will output just the payload string. Defaults to text.
Whether a timestamp should be prefixed to each message line in the file. Defaults to false.
File permission for writing. A string of the octal digit representation. Defaults to “644”.
Example:
[counter_file]
type = "FileOutput"
message_matcher = "Type == 'heka.counter-output'"
path = "/var/log/heka/counter-output.log"
prefix_ts = true
perm = "666"
Output plugin that serializes messages into the Heka protocol format and delivers them to a listening TCP connection. Can be used to deliver messages from a local running Heka agent to a remote Heka instance set up as an aggregator and/or router.
Parameters:
An IP address:port to which we will send our output data.
Example:
[aggregator_output]
type = "TcpOutput"
address = "heka-aggregator.mydomain.com:55"
message_matcher = "Type != 'logfile' && Type != 'heka.counter-output' && Type != 'heka.all-report'"
Specialized output plugin that listens for certain Heka reporting message types and generates JSON data which is made available via HTTP for use in web based dashboards and health reports.
Parameters:
Specifies how often, in seconds, the dashboard files should be updated.
An IP address:port on which we will serve output via HTTP. Defaults to “0.0.0.0:4352”.
File system directory into which the plugin will write data files and from which it will serve HTTP. The Heka process must have read / write access to this directory. Defaults to ”./dashboard”.
Example:
[DashboardOutput]
ticker_interval = 60
message_matcher = "Type == 'heka.all-report' || Type == 'heka.sandbox-output' || Type == 'heka.sandbox-terminated'"
Output plugin that serializes messages into JSON structures and uses HTTP requests to insert them into an ElasticSearch database.
Parameters:
ElasticSearch cluster name. Defaults to “elasticsearch”
Name of the ES index into which the messages will be inserted. Defaults to “heka-%{2006.01.02}”.
Name of ES record type to create. Defaults to “message”.
Interval at which accumulated messages should be bulk indexed into ElasticSearch, in milliseconds. Defaults to 1000 (i.e. one second).
Number of messages that, if processed, will trigger them to be bulk indexed into ElasticSearch. Defaults to 10.
Message serialization format, either “clean” or “raw”, where “clean” is a more concise JSON representation of the message. Defaults to “clean”.
If the format is “clean”, then the ‘fields’ parameter can be used to specify that only specific message data should be indexed into ElasticSearch. Available fields to choose are “Uuid”, “Timestamp”, “Type”, “Logger”, “Severity”, “Payload”, “EnvVersion”, “Pid”, “Hostname”, and “Fields” (where “Fields” causes the inclusion of any and all dynamically specified message fields. Defaults to all.
Format to use for timestamps in generated ES documents. Defaults to “2006-01-02T15:04:05.000Z”.
Example:
[ElasticSearchOutput]
message_matcher = "Type == 'sync.log'"
cluster = "elasticsearch-cluster"
index = "synclog-%{2006.01.02.15.04.05}"
type_name = "sync.log.line"
server = "http://es-server:9200"
format = "clean"
flush_interval = 5000
flush_count = 10
WhisperOutput plugins parse the “stat metric” messages generated by a StatAccumulator and write the extracted counter, timer, and gauge data out to a graphite compatible whisper database file tree structure.
Parameters:
Path to the base directory where the whisper file tree will be written. Defaults to “/var/run/hekad/whisper”.
Default aggregation method to use for each whisper output file. Supports the following values:
Default specification for new whisper db archives. Should be a sequence of 3-tuples, where each tuple describes a time interval’s storage policy: [<offset> <# of secs per datapoint> <# of datapoints>] (see whisper docs for more info). Defaults to:
[ [0, 60, 1440], [0, 900, 8], [0, 3600, 168], [0, 43200, 1456]]
The above defines four archive sections. The first uses 60 seconds for each of 1440 data points, which equals one day of retention. The second uses 15 minutes for each of 8 data points, for two hours of retention. The third uses one hour for each of 168 data points, or 7 days of retention. Finally, the fourth uses 12 hours for each of 1456 data points, representing two years of data.
Permission mask to be applied to folders created in the whisper database file tree. Must be a string representation of an octal integer. Defaults to “700”.
Example:
[WhisperOutput]
message_matcher = "Type == 'heka.statmetric'"
default_agg_method = 3
default_archive_info = [ [0, 30, 1440], [0, 900, 192], [0, 3600, 168], [0, 43200, 1456] ]
folder_perm = "755"
Specialized output plugin that listens for Nagios external command message types and generates an HTTP request against the Nagios cmd.cgi API. Currently the output will only send passive service check results. The message payload must consist of a state followed by a colon and then the message i.e., “OK:Service is functioning properly”. The valid states are: OK|WARNING|CRITICAL|UNKNOWN. Nagios must be configured with a service name that matches the Heka plugin instance name and the hostname where the plugin is running.
Parameters:
An HTTP URL to the Nagios cmd.cgi. Defaults to “http://localhost/nagios/cgi-bin/cmd.cgi”.
Username used to authenticate with the Nagios web interface. Defaults to “”.
Password used to authenticate with the Nagios web interface. Defaults to “”.
Specifies the amount of time, in seconds, to wait for a server’s response headers after fully writing the request. Defaults to 2.
Example configuration to output alerts from SandboxFilter plugins:
[NagiosOutput]
url = "http://localhost/nagios/cgi-bin/cmd.cgi"
username = "nagiosadmin"
password = "nagiospw"
message_matcher = "Type == 'heka.sandbox-output' && Fields[payload_type] == 'nagios-external-command' && Fields[payload_name] == 'PROCESS_SERVICE_CHECK_RESULT'"
Example Lua code to generate a Nagios alert:
output("OK:Alerts are working!")
inject_message("nagios-external-command", "PROCESS_SERVICE_CHECK_RESULT")
CarbonOutput plugins parse the “stat metric” messages generated by a StatAccumulator and write the extracted counter, timer, and gauge data out to a graphite compatible carbon daemon. Output is written over a TCP socket using the plaintext protocol.
Parameters:
An IP address:port on which this plugin will write to. Defaults to: localhost:2003
Example:
[CarbonOutput]
message_matcher = "Type == 'heka.statmetric'"
address = "localhost:2003"
hekad(1), hekad.config(5)