0.10.0 (2015-12-30)

Bug Handling

* Fixed issue where LogstreamerInput doesn't notice when a zero-length file is
  deleted and replaced by a new zero-length file before any data is appended

0.10.0b2 (2015-11-20)

Backwards Incompatibilities

* `DecoderRunner` interface changed `SetSendFailure` to `SetFailureHandling`.


* Allow TcpOutput to re-establish the connection after a configurable number
  of successfully delivered messages.

* Added `set_hostname` option to UdpInput configuration (#1668).

* Added `linux_netdev` and `linux_netstat` decoders (#1712).

* Improved error output for ElasticSearchOutput HTTP failures.

* Added `log_decode_failures` config value to decoders.

* Adder `log_flags` to hekad config, to control the prefix for STDOUT and
  STDERR logs.

Bug Handling

* Fixed issue where ElasticSearchOutput was trying to send zero length requests

* Fixed race condition in ElasticSearchOutput (#1786).

* AMQPInput `Run` method now returns an error when the input channel closes but
  `Stop` hasn't been called to successfully trigger restarts (#1757).

* Fixed error where restarting plugins were losing specified configuration

* Fixed config error where global `max_pack_idle` setting was the wrong type
  and was being ignored (#1778).

* Fixed race condition in InputRunner's default deliverer initialization.

* Set hostname correctly in the Graylog decoder (#1663).

* Add sample rate support for gauge type in StatsdInput.

* Fixed some cases where SandboxDecoder wasn't correctly setting
  `pack.TrustMsgBytes` to false when needed.

* Fixed broken queue rolling and added test.

* Force ElasticSearch index name to lower case, as required by ElasticSearch.

* Fixed default buffer config values.

* Updated docker configuration so docker build works again (#1634).

* Add generic json lua sandbox decoder (#1651).

* Fixed bug where a SandboxInput configured with a `ticker_interval` would
  get stuck in an infinite loop on shutdown (#1705).

* Fixed race condition in the BufferReader's queue file lookup code (#1639).

* Changes in StatAccumInput, FileOutput, TcpInput, ProcessDirectoryInput tests
  to minimize intermittent Travis failures.

* Ensure the response is read until it is complete in the HttpOutput plugin.
  Previously, the deferred Body.Close() may have been called on incomplete
  responses. This resulted in the connections not returning to the client's
  Transport connection pool.

* Fixed panic that was occurring when loading a config file or directory that
  exists but which registers no plugins (#1597).

* Delay start up when a buffered plugin's buffer is at capacity to give the
  back-pressure time to resolve (#1738).

* Fixed bug where LogStreamerInput would sometimes loop infinitely reading the
  same file over and over when reading gzipped log files.

0.10.0b1 (2015-08-07)

Backwards Incompatibilities

* `PluginHelper.PipelinePack` method now returns `(*PipelinePack, err)` values
  instead of just `*PipelinePack`.


* Allow multiple sandbox module directories to be specified (#1525).

* Add Nginx stub status lua sandbox decoder

* Add Nginx stub status lua sandbox decoder.

* Added support for SIGUSR2 signal for use when Heka is wedged, triggers a Heka
  pipeline report to the console, abort and serialize of all sandboxes, and a

* InputRunner and router's `Inject` methods each now return an error that can
  be checked to verify successful message injection.

* Added `SplitStreamNullSplitterToEOF` to SplitterRunner interface so input
  plugins can avoid generating messages with whatever happens to come back from
  a `Read()` call.

* Validate the buffer max_file_size is greater than MAX_RECORD_SIZE (#1623).

Bug Handling

* Validate the buffer max_file_size is greater than MAX_RECORD_SIZE (#1623).

* Fixed ProcessInput hangs when splitter is defined (#1620, #1644).

* `heka.all-report` and `heka.memstat` messages are now protobuf encoded before
  being injected into the router (#1659).

* Fixed ProcessInput not getting process's full output when NullSplitter is in
  use (#1645).

0.10.0b0 (2015-07-13)

Backwards Incompatibilities

* Major overhaul of filter and output plugin APIs to support disk buffering

* `queue_max_buffer_size` and `queue_full_action` config options for TcpOutput
  and ElasticSearchOutput are now moved to `max_buffer_size` and `full_action`
  in the `buffering` subsection (#1378).

* Go 1.4 now required for building.

* Deprecated the read_next_field() sandbox API; it is targeted for removal in
  0.11.0 (#1602).

* Removed unused PipelinePack.Decoded attribute.

* LogOutput will write data to stdout instead of stderr (#1515).

* Using strftime literals for filenames during rotation in FileOutput plugin

* Implemented strftime format codes in: filenames in FileOutput plugin,
  ESJsonEncoder, ESLogstashV0Encoder, Payload encoder (#1469, #1508).

* The package created by 'make deb' creates an "heka" user and ships an init
  script and a systemd unit.

* The 'make deb' target requires fakeroot and debhelper to be installed.

* SplitterRunner interface now provides a `Done` method that should be called
  whenever the splitter is no longer needed.

* `"Fields"` choice for the `fields` setting renamed to `"DynamicFields"` for
  both ESJsonEncoder and ESLogstashV0Encoder.


* Added support for `write_message` to delete fields when passed nil values.

* Added a timer_event_on_shutdown configuration option for Filter/Output
  sandboxes (#1460).

* Added `replace_dot` setting to StatFilter.

* A protobuf encoding of the current message is now always stored in
  pack.MsgBytes prior to injection into the message router (#265).

* ProtobufEncoder now just copies the pack.MsgBytes into a new byte slice and
  returns that.

* Added Linux CPU Stats Decoder and CPU Stats Filter.

* Centralized common functions used by the Schema Carbon Line Encoder and
  Schema InfluxDB Line Encoder into new ts_line_protocol and field_util modules.

* Added a new sandbox encoder, Schema Carbon Line Encoder, which provides
  support for converting fields in a Heka message to metrics formatted to
  send to Carbon/Graphite.

* Added a new sandbox encoder, Schema InfluxDB Line Encoder, which updates
  compatibility of sending data in Heka message fields to InfluxDB 0.9.0+
  write API. It is required to use this encoder when integrating Heka with an
  InfluxDB 0.9.0+ instance as the API to commit metrics has changed. This
  encoder was also updated to use the line protocol which is now the default
  series format for the write API.

* Added support for abstract Unix domain sockets to the UdpInput.

* Added support for `can_exit` to inputs (defaults to false, except on
  ProcessDirectoryInput spawned processes where it defaults to true)

* Added `field_mappings` setting to the ESJsonEncoder allowing custom names for
  Heka message fields in ElasticSearch.

* Termination messages are now generated for SandboxFilters that fail
  initialization by a SandboxManagerFilter.

* Added alert_throttle setting to HTTP status graph filter.

* Added DockerEventInput.

* Added splitters to the reports displayed by Heka after SIGUSR1 signal.

* Added `graphite` module with helpers allowing to generate graphite metrics
  for counters and timeseries (#1461).

* Added Basic Auth and API key authentication to HttpListenInput (#1533).

* Added SSL/TLS support to HttpListenInput (#1534)

* Allow to overwrite ContainerName using a Container environment variable from
  within the Docker container for the DockerLogInput. (#1545)

* Allow extracting environment variables from a Docker container to use as
  fields for the DockerLogInput. (#1569)

* Dashboard now shows sandbox plugins in alphabetical order.

* Added 'dynamic_fields' config to ESJsonEncoder for selecting a subset of
  Fields values

Bug Handling

* Fixed visibility of synchronous decoders in reports (#1312).

* Fixed hang on SandboxFilter termination (#1509)

* More ProcessInput/ProcessDirectoryInput retry logic fixes (#1412 & #1418).

* ProcessInput fixed to no longer leak decoder goroutines when reconfigured
  via ProcessDirectoryInput-driven config changes (#1444).

* Check configuration, ticker_interval in Stat Accumulator Input  must be
  greater than 0 (#1474).

* Switch from unmaintained `crowdmob` fork of GoAMZ dependency to `AdRoll`

* Added ability to reconnect after lost connection in `heka-flood` (#1536).

* Respect ElasticSearch URL path (#1558).

0.9.3 (2015-??-??)

Bug Handling

* Correctly honor "user-agent" heading config in HttpInput (#1520).

* Removed state from PluginMaker to remove race conditions during plugin
  construction (#1532).

* Get decoder lock before cleaning up decoders during pipeline shutdown to
  avoid race condition panics during exit (#1531).

0.9.2 (2015-04-22)

Bug Handling

* Added bounds check for truncated inputs lines to StatsdInput.

* Fixed ElasticSearch recovery after full queue when `queue_full_action` is
  set to "drop" or "block".

* Fixed TcpOutput recovery after full queue when `queue_full_action` is set to
  "drop" or "block" (#1484).

* Fixed bug where LogstreamerInput wasn't honoring `oldest_duration` setting

* ElasticSearch payload encoder will ensure there is a newline in the
  end of the payload in order for the bulk API to work correctly (#1457).

* Fix a gzipped file seeking in logstreamer may cause an OOM exception.

* Fixed config parsing typo bug in heka-logstreamer command (#1436).

* Always check for decoder existence when a decoder is specified for an input
  plugin (#1439).

* Check `IsStoppable()` on all input, filter, output plugins if they error at
  startup time, only shut down Heka if false.

* Fixed typo causing panic in AMQPInput when splitter returns an error

* Handle chunked error responses in http_output.

* ElasticSearchOutput buffering now only retries messages when it's clear that
  the problem related to failed communication with the ElasticSearch server

* Added handling of incomplete/trailing data for SplitBytes() method in
  SplitterRunner (#1455)

0.9.1 (2015-03-13)


* Added configurable max_message_size to heka-cat.

* Added `count` setting to TokenSplitter to allow splitting on every Nth
  instance of the given delimiter.

* Added `deliver_incomplete_final` setting to SplitterRunner.

* Added `max_message_size` setting to UdpOutput.

Bug Handling

* Fix the message matcher parser to clear the yySymType structure on each call

* Protect read_message("raw") from reading an empty pack.MsgBytes (#1405).

* Moved `SetSplitterRunner` out of the config's MakeRunner and into the
  `NewSplitterRunner` function to make sure a splitter always has access to a
  SplitterRunner if needed.

* Fixed SplitterRunner buffer readPos so it only increments when a read call
  doesn't return an error (#1367).

* Ensured plugin list file is closed after writing to it in Dashboard plugin.

* Fixed exiting / restart behavior for ProcessInput to match documentation

* UdpOutput drops message if its size exceeds allowed UDP data size (#1393).

0.9.0 (2015-02-25)

Backwards Incompatibilities

* Major overhaul of Heka's configuration loading code. This doesn't impact
  most plugins, it's only a breaking change for plugins that happen to
  instantiate and manage the lifecycles of other embedded plugins, e.g.
  MultiDecoder, ProcessDirectoryInput, SandboxManagerFilter.

* All inputs now support `decoder`, `synchronous_decode`, and
  `send_decode_failures` config options, automatically extracted by Heka's
  config system.

* InputRunner now handles message decoding and delivery to the router,
  according to the specifications of the input's configuration. This is
  accomplished either directly via the `Deliver` method or, in cases where
  decoding might need to happen in separate goroutines, through `Deliverer`
  objects available from the `NewDeliverer` method.

* Hekad and all clients started using stdout for informational messages and
  stderr for error messages.

* Stats Accum Input treats gauge inputs as float64 rather than int64 to match
  statsd spec and other statsd implementations (#850).

* Introduction of Splitter plugins, accompanied by major changes to how input
  plugins work to support the use of Splitter plugins. (#424)

* TcpInput and UdpInput now set a non-protobuf encoded message's Type field to
  the configured input plugin name instead of the hard-coded string

Bug Handling

* Fix the shutdown hang caused by a Filter/Output plugin failure (#1324).

* Fix the RPM dependency errors on Centos7 (#1311).

* Don't output empty messages when use_framing = true (#1326).

* KafkaInput uses the oldest offset available instead of zero when no
  checkpoint file exists and the offset method is set to manual. KafkaInput
  also correctly closes the checkpoint file before removing it when it is
  invalid (#1325).

* Handle empty byte fields in the sandbox interface (#1284).

* Reset header when discarding valid but oversized messages (#1221).

* Prevent the protobuf stream encoder from creating messages over

* AMQPOutput now uses defined constants for delivery mode instead of hard
  coded (and wrong) integer values (#1235).

* Hekad now respects --config parameter and loads default config path
  instead of printing help and exit (#1239).

* MessageProtoParser now handles EOF gracefully, and returns all
  complete records up to the end of the stream without having to
  use GetRemainingData() (#1305).

* HttpListenInput no longer URL unescapes the HTTP request body (#1124).

* SmtpOutput now encodes email subject when necessary (#1277).

* All config files are now pre-loaded before any of the config is actually
  loaded to prevent default plugins from being registered for each separate
  config file, possibly overriding custom config.


* Added decode_message to the sandbox API (#1344).

* Added a SandboxInput plugin (#1333).

* Added a SandboxOutput plugin (#1303).

* `heka/plugins/amqp` package now exposes `GetAmqpHub` and `NewAMQPDialer`
  functions for use by external packages.

* Added heartbeat monitoring sandbox filter plugin.

* Include the new lua_sandbox with support for shared library modules.

* Allow the sandbox process_message function to set the last error string
  when it returns (#1191).

* Improve the sandbox inject_* error messages (#1156).

* Added ability to specify unit type and aggregation methods for stats graph

* Added option (immediate_start) to start process immediately in ProcessInput

* Added `hostname` setting to global `[hekad]` config options to allow
  overriding the hostname value provided by Go's `os.Hostname` call (#1123).

* Added 'connect_timeout' option in ElasticSearch output plugin.

* Slightly improve error output in ElasticSearch plugin.

* Added support for HTTP authentication to ElasticSearchOutput.

* TcpOutput's disk buffering now supports specification of a max buffer size
  and an action to take if the max size is reached, from 'shutdown', 'drop',
  or 'block' (#1110).

* Added `send_interval` setting to SmtpOutput.

* Added `timestamp` format setting to ESLogstashV0Encoder (#1142).

* LogstreamerInput now uses first two "magic" bytes to identify gzip files
  instead of relying on `.gz` suffix.

* Added on-disk buffering to ElasticSearchOutput. It is enabled by default.

* Added TLS support to ElasticSearchOutput (#1259).

* Added ability for HttpListenInput to capture specified HTTP request headers
  and write them as message fields.

* Added ability for Statsd input handler to treat a malformed stat line
  one at time without skipping all the "good" stats in a multi-line input
  (more like statsd itself).

* Added `message_interval` setting support to heka-flood test configuration.

* Added `cert_path` setting for TLS support to DockerLogInput.

* Added new fields to messages generated by HttpListenInput, including
  RemoteAddr, Path, EnvVersion, Hostname (the name of the server that
  handled the request), and Host (the host to which the client sent the
  request) (#1328).

* Added `hostname_keep` option to RsyslogDecoder (#1338).

* Added an option to Schema InfluxDB Encoder to send the data to InfluxDB as
  a list of series rather than the default of a single series for all fields
  in the message.

* Added an option to Schema InfluxDB Encoder that excludes all base fields
  from being sent to InfluxDB to reduce the network traffic and storage
  demands if these fields aren't useful.

* Added `read_only` setting to AMQPInput to support read-only users.

* Added ability to specify MAX_MESSAGE_SIZE via configuration file of hekad
  and heka-flood (#1208).

* Changed default plugin_chansize setting from 50 to 30 (#1242).

0.8.3 (2015-01-08)

Bug Handling

* Fixed LogstreamerInput to use a separate DecoderRunner for every
  LogstreamInput created, rather than sending all of the streams through a
  single decoder.

0.8.2 (2015-01-06)

Bug Handling

* Fix rsyslog sandbox decoder test to use current year and location for
  timestamp parsing, since syslog timestamp format makes that assumption.

* Ensure that geoip_decoder is not included in release binaries.

0.8.1 (2014-12-17)

Bug Handling

* Fix leaky file descriptor bug on http_output.go.

* Only stamp PipelinePack diagnostics in cases when a pack actually matches a
  message matcher, instead of for every matcher every time (#1167).

* StatsdInput allows to specify (via max_msg_size option) size of message read
  from UDP (#1165).

* AMQPOutput now recycles packs even when a publish error causes the output to
  exit / restart (#1178).

* Fixed HttpOutput TLS section parsing (#1163).

* Fixed LogstreamInput extraneous journal saves that caused high disk IO
  when EOF is reached.

0.8.0 (2014-10-29)

Backwards Incompatibilities

* Audited the use of Logger and Type headers on internal message (#1024). These
  changes may break some existing message matchers with regards to
  plugin/sandbox termination messages.

* Sandbox decoder linux_cpustats has been renamed to linux_loadavg and the
  corresponding filter cpustats was renamed to loadavg (#1094).

* ESLogstashv0 encoder will now set the @type field to match the ES record
  type specified in the `type_name` setting. This is Logstash behaviour in
  V0. You can revert back to the old method with `use_message_type = true`

Bug Handling

* linux_diskstats.lua will now parse a line with no leading space (#1141).

* Protect globals.Stopping with a RWMutex.

* Added support for `fields` config setting to ESLogstashV0Encoder; it was
  already documented, but hadn't been implemented (#1096).

* Fixed deadlock race condition in AMQPOutput shutdown sequence (#824).


* Added Kafka Input/Output plugins (#1148).

* Optimized prefix/suffix testing in the message matcher (#1040).

* LogstreamerInput now supports seekjournal file hashing for files that are
  less than 500 bytes in length; hash is generated against file contents with
  0 bytes prepended to make 500 bytes of hash content (#972).

* Support environment variables in config files (#1023).

* Add Dockerfiles and example Docker usage to repo

* Encoders can now return (nil, nil) to express that they've swallowed the
  input message without generating any output.

* SandboxEncoder now supports return value of -2 from the process_message call
  to specify that no output is being generated.

* Added schema_influx.lua encoder.

* Added CBUF Librato Encoder.

* Added option to disable re-using TCP connections to the ElasticSearch output

* Added separate env loading file and support for NUM_JOBS env var to Windows
  build (#971).

* Added DockerLogInput (issue #1092).

* Added support for "Epoch", "EpochMilli", "EpochMicro", and "EpochNano"
  timestamp formats to the time parsing code used by the Payload*Decoder
  plugins (#963).

* Added ability to drop big message entirely or to keep first part of it (#1134)

0.7.3 (2014-10-28)

Bug Handling

* Fail gracefully on LogstreamerInput missing `file_match` setting (#1105).

* Fixed issue where ProcessInput wasn't propagating environment variables and
  working directory to repeatedly run (vs. one time, long running) processes

* UdpInput now removes a Unix datagram socket file at shutdown time if one is
  created at startup.

* Fixed issue with keeping previous EOF state in LogStreamer (#1119).

* ProcessInput now honors documented restart behavior (#1117).

0.7.2 (2014-10-01)

Bug Handling

* Fail more gracefully if user doesn't specify a scheme in the server URL for
  the ElasticSearchOutput (#1069).

* Correctly compute http timeout interval for ElasticSearchOutput when using
  HTTP indexing.

* Fixed issue w/ inaccurate payload sizes in heka-flood.

* Allow severity to be modified by the MessageTemplate logic for use in
  ScribbleDecoder (#1084).

* Render pack diagnostic idle time in seconds (as specified) instead of

* Prevent orphaned matchers from remaining in the router's matcher lists in
  cases where a filter or output registered in one TOML file overrides a
  filter or output of the same name in an earlier TOML file.

0.7.1 (2014-09-02)

Bug Handling

* Fixed handling of TcpInput and TcpOutput keep_alive_period default value
  handling, and added docs (#1054).

* Fixed OSX lua_sandbox build error.

* Fixed load ordering of nested MultiDecoders (#1045).

0.7.0 (2014-08-27)

Backwards Incompatibilities

* Switched to using fork of gomock, import location changed from
  `` to ``.

* Move *GlobalConfigStruct out of pipeline package's global namespace, it is
  now stored as an attribute on PipelineConfig. Any code that used to call
  pipeline.Globals() now must instead access *PipelineConfig.Globals. The
  `WantsPipelineConfig` interface has been introduced to give plugins access
  to the active *PipelineConfig if it's needed in the ConfigStruct or Init
  methods (#951).

* Removed deprecated `pipeline.GetHekaConfigDir` API call, which has been
  replaced by `PrependBaseDir` and `PrependShareDir` for a few versions

* Removed the regular expression template support from the message matcher
  (issue #960).

* Updated the AMQP input / output configuration values to use underscore
  delimited words for long config options (issue #953).

* LogstreamerInput now errors on nonexistent `log_directory` instead of
  creating the folder (issue #1066).

Bug Handling

* Fixed the MySQL slow query grammar to handle logs with no newline.

* Fixed packet tracking idle packs error output formatting.

* Prevent panics during shutdown when a restarting plugin has been restarted
  (issue #957).

* Added support for a Logstreamer symbolically linked log_directory root
  (issue #741).

* Fixed lots of race conditions in the tests.

* Eliminated race condition in use of global AMQPHub in AMQP input / output
  plugins (#953).


* Added StatMetric Influx Encoder.

* Added RstEncoder.

* Added support for '__ignore_root' tag in add_external_plugin to allow
  external packages to be added which do not have any .go files
  in their root directory (issue #955).

* Switched from using goprotobuf to gogoprotobuf with marshal_all and
  unmarshal_all extensions for significant performance increase.

* Added stats graph sandbox filter to make it easier to generate graphs of
  statsd / graphite metrics (issue #966).

* Added support for the majority of repository URI formats when cloning external
  Go packages (issue #937).

* Added the ability to test for field existence in message matcher (issue #958).

* Added support to ESJSonEncoder and ESLogstashV0Encoder to properly encode
  field arrays.

* Added a filter to monitor all process message failures (issue #948).

* Added UdpOutput.

* Added IrcOutput.

* Added support for filter and output plugins to optionally exit without
  causing Heka to shutdown via config option.

* Added support for custom HTTP Headers in HttpListenInput and DashboardOutput.

0.6.1 (2014-08-27)

Bug Handling

* CarbonOutput using UDP transport now uses multiple packets to send stats
  data to Carbon server when output would be longer than the 64KiB max size
  for UDP packets (issue #1035).

* StatsdInput now trims all whitespace around incoming stat names instead of
  just trailing newlines (issue #1011).

* Fixed silent failures in ElasticSearchOutput bulk indexing operations.

* Fixed permanent loss of HTTP connection in certain cases when ElasticSearch
  has been restarted (issue #1008).

* Fixed default sandbox script type when loading dynamic plugins.

0.6.0 (2014-07-09)

Backwards Incompatibilities

* Go 1.3 now required for building.

* Changed handling of message stream framing. Before stream framing was
  presumed to always be required when encoding with protocol buffers, and not
  required when using other encodings. Now all outputs support an optional
  `use_framing` setting that will determine whether or not framing will be
  applied when using the OutputRunner's Encode method (issue #922).

* All files in a config directory that do not end in ".toml" will now be
  ignored and not loaded as a part of the Heka configuration (issue #750).

* Removed the PayloadJsonDecoder which is replaced by the Lua sandbox cjson
  module (issue 826).

* Removed LogfileInput and LogfileDirectoryManagerInput which were replaced
  by LogstreamerInput in 0.5.0 (issue 914).

* MultiDecoder no longer sets prepends name of the decoder instance to the
  message type value for every message. This was impacting performance, is
  almost never what is needed, and is almost always overwritten by the nested
  subdecoders anyway.

* MultiDecoder now has a `subs` option that refers to other top level decoder
  config definitions, instead of nested sub-configs and a separate `order`
  option specifying the order. This makes for less typing and the ability to
  reuse decoder definitions as both standalone decoders and across multiple
  MultiDecoders (issue #485).

* ElasticSearchOutput now uses an Encoder plugin instead of MessageFormatters.
  Created ESJsonEncoder, ESLogstashV0Encoder, and ESPayloadEncoder
  (implemented as a SandboxEncoder in es_payload.lua) to replicate prior
  formatter behavior.

* Changed the Lua sandbox API.  inject_message is no longer overloaded, output
  has been renamed to add_to_payload, and inject_payload has been introduced.

* Changed TcpOutput, FileOutput, and LogOutput to use Encoder plugins for
  output formatting instead of bespoke formatting implementations.

* The rsyslog decoder now further parses the %SYSLOGTAG% variable. The
  fields.syslogtag no longer exists and is replaced by fields.programname and
  message.Pid (issue #677).

* Removed no-longer-used 'decoder_poolsize' global config setting.

Bug Handling

* Fixed the ElasticSearchOutput to always use UTC times in the bulk API header
  (issue #504).

* Fixed the SandboxDecoder panic when failing the Decode() after successfully
  injecting a message (issue #910).

* StatsdInput no longer spins up a new goroutine for each stat (issue #359).

* MultiDecoder no longer prevents shutdown if a nested SandboxDecoder crashes
  during startup (issue #896).

* MultiDecoder using `cascade_strategy = "all"` now passes all generated packs
  to all nested decoders instead of skipping the remainder if an earlier pack
  in the sequence fails to decode (issue #896).

* MultiDecoder no longer tries to recycle the original pack, leaving that job
  to the DecoderRunner as intended (issue #896).

* Updated SmtpOutput to use a slice instead of a map to generate SMTP headers
  so the order will remain consistent when using Go 1.3 or later.

* Use FindProcess instead of syscall.Kill to make the 'pid_file' configuration
  setting work on Windows (issue #807)

* Fix the SandboxFilter ReportMsg panic on termination/shutdown (issue #816)

* MultiDecoder with no 'order' set now fails on init instead of panicking on
  first message.

* Fix panic on SIGUSR1 caused by no reports in a given plugin category (issue

* Fix file_match config of plugin logstreamer_input (issue #893).

* StatFilter now correctly handles values from integer or float fields (issue


* Added general purpose HttpOutput (issue #820).

* Added message processing stat reporting to the MultiDecoder for each
  subdecoder and in aggregate (issue #719).

* Added memory statistics to the Heka report output, for self monitoring.

* Added a message type configuration option to the rsyslog decoder (issue #907).

* Added `elasticsearch` Lua module to generate BulkAPI index JSON (issue #875).

* Added support for Lua sandbox preservation versioning (issue #701).

* Added support for unix datagram sockets to the UdpInput using net "unixgram"
  (issue #790).

* Add LogstreamInput seekjournal reporting to dashboard output (issue #445).

* Added a HyperLogLog library to the Lua sandbox using the Redis

* Added an alert encoder to make the alert messages easier to read.

* Introduced Encoder plugin type (issue #417).

* Added a bloom filter to the Lua sandbox and created a unique items filter.

* Turned the MySQL slow query log examples into a deployed decoder and filter.

* Added an Nginx error log decoder (issue #785)

* Added the ability to preserve the webserver log line in message payload
  (issue #784)

* Added the optional 'pid_file' configuration setting (issue #777).

* Add anomaly, alert, annotation modules (issue #677)

* Added 'sample_denominator' global config setting to allow tweaking the
  sample interval when computing timing of certain operations, replacing prior
  hard-coded DURATION_SAMPLE_DENOMINATOR constant (issue #625).

* Added an Apache access log decoder based on the Apache 'LogFormat'
  configuration directive.

* Added BufferedOutput. Extracts the queuing functionality out of TcpOutput.go
  into a general purpose lib for use in any output module. Callers get
  messages buffered to disk while another goroutine consumes and forwards
  data. Any errors encountered can cause the sending goroutine to backoff and
  resend data.

* Added http_timeout to elasticsearch output to prevent slow or stale
  connections from holding up the flow. (issue #769)

* Add query parameters to the Message as Fields in the HttpListenInput

* Add QueueTTL option to AMQPInput to allow specifying message expiration.

* Added optional tls sub-section to AMQPInput and AMQPOutput for configuring
  AMQPS TLS settings.

* Add detection and handling of gzipped files to LogstreamerInput (issue #648).

* Added optional TCP Keep-Alive parameters to TcpInput/TcpOutput plugins.

0.5.2 (2014-05-16)

Bug Handling

* FileOutput no longer panics when using `format = "text"` and payload is nil
  (issue #843).

* Fix SandboxDecoder pass-through case so decoders that only use write_message
  and not inject_message will emit messages correctly (issue #844).

* Fixed TcpInput so it will only override the default delimiter for a
  RegexpParser if a delimiter is specified in the config.

* Fix the FileOutput panic when HUP'ed.  Modified the CheckWritePermission
  utility function to use a unique filename for each check. (issue #808)

* Terminated plugins are now removed from the SandboxManager quota (issue

* Fixed SIGUSR1 triggered text report to stdout to work with updated
  heka.all-report JSON data structure (issue #762).

* MultiDecoder now uses message.GetType() instead of Message.Type so
  emitted message type will be generated correctly (issue #761).

* LogstreamerInput no longer causes Heka to fail to start on empty or
  whitespace-only journal files (issue #755).

* The cbufd_host_aggregator filter now properly reclaims expired hosts.

* Pull in a new lua_sandbox to fix JSON encoding of empty strings in the
  sandbox plugin output() call.

* Escape regexp meta characters (notably '\') to prevent a panic in the LogstreamerInput on Windows

0.5.1 (2014-03-18)

Bug Handling

* Skip *.bak, *.tmp, *~, and .files in a config dir as a non-breaking band-aid
  until we require an explicit naming convention in 0.6 (see issue #750).

* heka-logstreamer command now supports config directories in addition to
  single config files, just like hekad itself (issue #742).

* Logstreamer package's NewLogstreamSet function no longer lowercases the
  match part names when constructing match translation maps since the
  PopulateMatchParts method doesn't actually expect the names to be

* Added support for use of "missing" value in Logstreamer translation maps to
  allow users to place missing values at the end of the list instead of the
  beginning (issue #735).

0.5.0 (2014-03-06)

Backwards Incompatibilities

* ProcessInput no longer supports a separate'name' config setting, it uses the
  specified plugin name from the TOML section header like all of the other

* Removed Stdout_chan and Stderr_chan from ManagedCmd and CommandChain, client
  code now has direct access to Stdout_r and Stderr_r io.Reader providers.

* The PluginHelper interface DecoderRunner prototype has changed (issue #717)
  to allow for the base name e.g. "ProtobufDecoder" and a full instance name
  e.g. "MyInput-ProtobufDecoder" of the decoder to be specified.  This allows
  multiple decoders of the same type to show up on the DashboardOutput and
  sandbox decoder state preservation to work properly. Also the DecoderRunner
  UUID interface method was removed.

* FileOutput `flushinterval` config setting changed to `flush_interval` to
  match config naming conventions.

* SandboxDecoder, SandboxFilter, and SandboxManagerFilter now all use
  `${SHARE_DIR}/lua_modules` as the default `module_directory` setting.
  SandboxDecoder and SandboxFilter both now interpret relative paths to lua
  source code to be relative to ${SHARE_DIR} instead of ${BASE_DIR}.

* DashboardOutput `static_directory` setting now defaults to
  `${SHARE_DIR}/dasher` instead of `${BASE_DIR}/dashboard_static`.

* The sandbox preservation data is now stored in the
  {base_dir}/sandbox_preservation directory instead of with the plugin source.
  On the initial restart no preserved data will be restored unless it is
  manually moved to this directory first. (issue #626)

* The Heka utilities (flood, sbmgr, sbmgrload, inject) have been namespaced
  with a `heka-` prefix for their respective binaries. I.e. Flood has been
  renamed heka-flood., etc.

* MultiDecoder now gets its name from the TOML section name like the rest of
  the plugins, instead of a separate 'name' config option.

* Major reorganization of the `pipeline` package, moving the implementation of
  most plugins to sub-packages of a separate `plugins` package.

* Removed the wrapper 'table' element from the JSON serialization (issue #525)
  i.e., {"table":{"value":1}} is now simply {"value":1}.  The change also
  removes the special '_name' metadata arribute; the top level _name element
  should be created in the Lua structure if it is required.

* In the process of removing the core sandbox code from Heka (issue #464), the
  sandbox was streamlined to only load the base library by default. All sandbox
  plugins must now explictly load additional libraries with the require

* Removed DecoderSet method from PluginHelper interface (and DecoderSet
  abstraction entirely) and replaced it with Decoder and DecoderRunner methods
  that return a Decoder or a DecoderRunner by name.

* Changed Decoder interface to support one input pack generating multiple
  output packs.

Bug Handling

* Network parsers now return all records available in the parse buffer
  (issue #732).

* TcpInput now stops a given connection's decoder when the connection is
  closed, preventing memory pooling (issue #713).

* StatsdInput now doesn't fail with multiple stats in a single UDP packet.

* Set default StatAccumInput stat namespace prefix values even when
  `legacy_namespaces` is set to false (issue #630).

* Fixed cpuprof file from being closed right after opening so no data was
  being logged.

* Fixed LogfileInput so it will only override the default delimiter for a
  RegexpParser if a delimiter is specified in the config.

* PluginWrapper will now check for the WantsName interface when creating a
  plugin, and will set the plugin's name if appropriate.

* SandboxDecoder now explicitly logs a fatal error before triggering shutdown
  to ensure error message is actually emitted.

* Message severity now defaults to the highest RFC 5424 value (i.e. 7) implies
  low severity, rather than zero, which implies `emergency`, (issue #518).

* 'flood' command now outputs every send error (even 'connection refused'), and
  always increments 'messages sent' count even when there is a sending error
  (so setting 'num_messages' still works even if hekad stops responding,

* stat_accum_input will not fail when flushing a timer where the percentile
  is equal to the min value.


* Added ProcessDirectoryInput.

* InputRunners now support being specified as 'transient', meaning their
  lifespan should be managed by the code that creates the InputRunner and not
  Heka's pipeline code.

* HttpInput: now supports configuring the HTTP method, HTTP headers and HTTP
  Basic Authentication

* TLS Listeners can specify a 'client_cafile' which limits the CAs that a
  client cert can be chained to. This provides a mechanism for TLS Client AUTH.

* TLS Senders can specify a 'root_cafile' which limits the CAs that a
  server cert can be chained to. This provides a mechanims for TLS Server AUTH.

* Added StopDecoderRunner function to PluginHelper API so inputs can manually
  decommission decoders when they're no longer being used.

* The SandboxManagerFilter can now control the sandbox usage limits (issue #95)

* Added support for send_nsca to NagiosOutput as an alternative to direct http
  submission; also a way to explicitely specify service_description and host
  to match Nagios config

* Added configurable network types to TcpInput "tcp", "tcp4", "tcp6", "unix" and
 "unixpacket" (issue #539)

* Added configurable network types to UdpInput "udp", "udp4", "udp6"
  (issue #539)

* Added flush_count config setting to FileOutput to complement existing
  flush_interval. Also added flush_operator setting which can be "AND" or "OR"
  to specify how flush_count and flush_interval should be combined.

* Introduced `share_dir` global config setting, which specifies where Heka's
  read only resources should live. Defaults to `/usr/share/heka`. Also added
  `pipeline.PrependShareDir()` function for use within plugin initialization

* Added an rsyslog decoder based on the rsyslog string configuration template
  (issue #432).

* Added an Nginx access log decoder based on the Nginx 'log_format'
  configuration directive.

* heka-cat: A command-line utility for counting, viewing, filtering, and
  extracting Heka protobuf logs.

* TcpOutput has been redesigned to handle unavailable endpoints and dropped
  connections without data loss (issue #355).

* CarbonOutput now supports submitting messages via UDP, persistent TCP

* Added Logstreamer Input [LogstreamerInput]: An input that replaces the
  Logfile and Logfile Directory Inputs and supports sequential reading of
  logstreams that span sets of ordered logfiles (issue #372).

* TcpInput, TcpOutput, and flood client now all support TLS encrypted TCP
  connections using Go's crypto/tls package.

* Added Http Listen Input [HttpListenInput]: An input that listens for
  HTTP requests on the specified address. If a decoder is not specified the
  input generates a message with the HTML request body as the payload.
  This input is especially useful for consuming and processing webhooks.
  (Issue #431)

* Added support for local external packages (issue #393)

* Inject: A command-line utility for injecting arbitrary messages into
  a Heka pipeline.

* Added Go ScribbleDecoder for efficient setting of purely static message
  field values.

* Exposed `write_message` API function to Lua decoders to allow mutation of
  existing decoded message (issue #577).

* HttpInput: Added urls (array) option.

* HttpInput: Failed and successful HTTP GET actions produce messages of Type
  "", Logger = Request URL, and severity appropriate to
  HTTP status 200 or not. I.e. Connections responding with a status of 200
  produce a message with Severity 6, non-200 statuses Severity 1. Failure to
  connect produces a message with Severity 1 and Type "heka.httpinput.error".

* HttpInput: Fields[ResponseTime] populates with time duration in seconds for
  HTTP GET of URL, Fields[Status] with HTTP Status, Fields[Protocol] with HTTP
  protocol and version, Fields[StatusCode] with HTTP Response Status Code for
  successful HTTP GETs. The Circular Buffer Graph Annotation (Alerts)
  ( plugin
  is compatible with the HttpInput plugin.

* HttpInput: Added success_severity and error_severity options for GET actions.

* HttpInput: Messages now set Logger, UUID, and Timestamp.

* Added log_errors option to PayloadregexDecoder to allow skipping invalid

* Added "id" flag to elasticsearch output (issue #386).

* Added SmtpOutput (issue #472)

* Added preserve_data option to SandboxDecoder (issue #668).

* Added delete_idle_stats to StatAccumInput.

* Added sum and count_ps metrics to timers in stat_accum_input.

0.4.2 (2013-12-02)

Bug Handling

* Changed CPack configuration such that the 'make package' target no longer
  creates deb packages, and added a new 'make deb' target that creates debs
  with the filename expected by the deb package naming conventions (see

* Doc clarifications re: required use of ProtobufDecoder (see

* Explicitly exclude system level folders from those that CPack will include
  as part of the Heka RPM (requires CMake >= 2.8.12) (see

0.4.1 (2013-11-05)

Bug Handling

* Updated Mozilla Sphinx theme submodule reference and configuration paths to
  work around bug in ReadTheDocs rendering (see

* Changed default DashboardOutput.StaticDirectory value to
  `/usr/share/heka/dasher`, to match where the packages put the files.


* add require_all_fields flag to PayloadJsonDecoder
  (issue #528)

0.4.0 (2013-10-30)

Backwards Incompatibilities

* DashboardOutput now requires a `static_directory` setting to specify where
  the Heka dashboard source code can be found.

* AMQPInput now only supports a single decoder to be specified (issue #414).

* UdpInput configuration now requires a parser_type and decoder to be specified
  (issue #411).

* TcpInput configuration now requires a parser_type and decoder to be specified
  (issue #165).

* Removed the LogfileInput decoders array configuration in favor of a single
  decoder (issue #308).

* SandboxManagerFilter now auto-generates a default working directory based on
  the plugin name, in the ${BASE_DIR}/sbxmgrs folder.

* LogfileInput seek journals now support a boolean `use_seek_journal` config
  flag, and only the name of the journal file is now settable via
  `seek_journal_name`, replacing the `seekjournal` option which specified the
  full path.

* Build is now entirely cmake based and lives directly in the heka repository,
  replacing the separate primarily gnu make driven separate heka-build repo.

* Core Heka plugins now default to writing data to /var/cache instead of
  /var/run since /var/run is usually deleted on server restart.

* StatAccumInput now defaults to emitting stats data in the payload instead of
  in the message fields.

* Renamed LoglineDecoder to PayloadRegexDecoder.

Bug Handling

* Added check for nil RemoteAddr value since UDP connections won't have one
  (issue #462).

* Statsd messages are now parsed manually (issue #44).

* Added mDRunner so MultiDecoder will work with sub-decoders that implement
  the WantsDecoderRunner interface.

* Removed all code that attempted to catch and recover from panics in called
  plugin code, since a) the recovery often left Heka in an undefined, broken
  state b) the safety was a lie, any panic in a separate goroutine would not
  be caught and c) panic traceback actually makes it easier to debug.

* WhisperOutput and CarbonOutput no longer require stats to use the `stats`
  namespace prefix.

* LogfileInput now truncates and overwrites the seek journals instead of

* `protoc` now an optional dependency for the build, only required if the
  protobuf definition is actually changed.

* MessageTemplate field name interpolation was only supporting alphabetic
  characters for field names, now supports all "word" characters (i.e.
  alphanumeric plus underscore).

* ElasticSearchOutput now always sends timestamps in expected UTC time zone

* Fixed a CarbonOutput memory / socket leak.

* Fixed panic caused by an invalid config file path.


* Added a sandbox message field iterator (issue #460)

* Added ProcessInput. (issue #406)

* Replaced DashboardOutput UI with new Backbone.js-based `dasher` interface.
  (issue #378)

* Add delta output to circular buffers. (issue #467)

* Added json decoding to the Lua sandbox (issue #463). If you have an old build
  directory you will have to run make clean to pick up the new patch.

* Added a 'config' variable section to the Sandbox decoder/filter plugin.
  (issue #444)

* Added StatsToFieldsDecoder that will convert a message with statsd formatted
  string in the payload to one that also has the data available as message
  fields, in the same format that would be generated by a StatAccumInput with
  `emit_in_fields` set to true.

* MultiDecoder now supports `cascade_strategy` config options of `first-wins`
  and `all`, defaults to `first-wins`.

* Added stream parsing to UdpInput. (issue #411)

* Added stream parsing to TcpInput. (issue #165)

* Added SandboxDecoder. (issue #370)

* Support Heka protobuf message data in the LogfileInput plugin. (issue #261)

* Added "payload" format option to ElasticSearchOutput to pass message payload
  through to ES unchanged.

* Allow LogfileInput to use configurable delimiters including regexp (issue #263)
  This includes a backwards incompatible change for the journal file, old
  journals will not be honored and the new journal will overwrite it.

* Allow add_external_plugin to specify sub-packages. (issue #384)

* Added `base_dir` global (i.e. `[hekad]` section) config option to specify a
  single location Heka plugins should use to persist data to the file system.

* The `hekad` commands `config` flag now supports use of a config directory in
  addition to a single config file. If a directory path is specified, every
  file in the directory will be parsed and they will all be merged to a single
  running configuration.

* Added LogfileDirectoryManagerInput (issue #344).

* LPEG library added to Lua sandbox (issue #144).

* ElasticSearchOutput now supports `logstash_v0` output format that is modeled
  after the original Logstash ElasticSearch message schema.

* Added PayloadJsonDecoder that can extract information from JSON text in a
  message payload during decoding.

* Make index type_name option for ElasticSearchOutput definable from Field value (issue #356)

0.3.0 (2013-07-16)

* Fixed default (and only supported, for now) sample rate in the stats
  generated by the StatFilter.

* Fixes to the platform specific build constraints around signal handling.

* Added HttpInput that can make HTTP requests to a remote server, placing
  the response body into a message payload.

* Filters and outputs now require a specified message_matcher or they will
  raise an error applying the configuration.

* ticker_interval now exposed as a global config option for inputs, as well
  as outputs and filters.

* Overhaul of StatAccumInput implementation to fix bugs and allow for better
  handling of output format options.

* LoglineDecoder will now default to the current day, month, and year when
  parsing text that contains bare timestamps w/ no date information.

* Added permission checks to FileOutput and WhisperOutput plugins to fail
  gracefully when they've been asked to write to a location where they do not
  have write permission.

* Fixed folder creation permission bugs and allow configurable folder
  permission settings in FileOutput and WhisperOutput.

* `hekad` now emits help message if started w/ no command line options.

* Consistently use underscore_separated_words for plugin TOML configuration

* Use strings instead of integers to specify octal permission values in the
  config, since TOML doesn't support octal integer values.

* LoglineDecoder config now allows specification of a time zone in case the
  parsed text contains non-UTC timestamps that don't include embedded time
  zone information.

* Removed match capture group support from the router's message matching
  functionality, since this is better done in the decoder layer where the
  back-pressure won't slow down all message delivery. This gets rid of the
  `PipelineCapture` object altogether, so now filter and output plugins deal
  w/ PipelinePacks directly.

* Added ElasticSearch output.

* Lua filters can now emit tables, which will be serialized to JSON and
  put in an injected message payload.

* Custom dashboard javascript now generated by Heka instead of hosted

* Improvements to flood, a Heka protocol load test client.

* Config loading now fails if a config section specifies an unrecognized
  configuration option.

* StatAccumulator now supports stat metric messages with stats data embedded
  in message fields as well as message payload.

* Added support for plugin code to provide message_matcher and ticker_interval
  default values.

* Reimplemented StatMonitor as StatAccumInput, providing a StatAccumulator

* SIGUSR1 signals now generate Heka report on stdout instead of sending
  a message so it can be viewed even when message delivery is hosed.

* Added explicit Close() calls to whisperdb files in WhisperOutput.

* Redesigned message field metadata to use string specifiers instead of
  an enum.

* Lua filters can now emit multiple named outputs to the Heka dashboard.

* LogfileInput now supports just one file per input.

* Removed race conditions exposed by Go's race detector.

* Improved self monitoring / dashboard output.

* Improved shutdown message flushing.

* Added Nagios output.

* Added support for LogfileInput to resume parsing where it left off.

* Added AMQP input and output plugins.

* Improved control over sandbox filter message injection restrictions.

* Added support for restartable plugins.

* Moved regular expression and capture group parsing out of the filter layer
  (i.e. in the TransformFilter) and into the decoder layer (i.e.
  LoglineDecoder) to prevent back pressure from impacting the entire router.

0.2.0 (2013-06-26)

* Fix stat name regex to capture "." characters in the name.

0.2.0rc2 (2013-05-23)

* Fix Lua sandbox C code to work on 32 bit systems.

* Trivial release documentation ReST formatting fix.

0.2.0rc1 (2013-05-21)

* Use non-https links in docs to prevent bad rendering due to mixed http/https

* A number of documentation tweaks and updates.

* Static linking of cgo wrapped Lua environment.

* Added LICENSE.txt and CHANGES.txt for better project hygiene.

* Changed default interval for log file reading from 1ms to 500ms so we don't
  churn the machine in default configuration.

* Moved StatPacket channel setup into StatsdInput's Init method (from Run) to
  avoid race conditions.

* Added support for mingw-based Windows build.

* Perform message injection in a goroutine to prevent blocking the router when
  the plugin that is doing the injection has a full in channel.

* Added required pack recycling to TcpOutput.

0.2.0b1 (2013-04-30)

* Initial public release