Overview

The sources field is an array, and multiple source configurations can be filled in a Pipeline.

Therefore, please note that name of all sources is required as the unique identifier of the source in the pipeline.

Source Common Configuration

The following configurations are available for all sources.

enabled

field	type	required	default	description
enabled	bool	false	true	whether to enable the source

name

field	type	required	default	description
name	string	true		source name

fields

field	type	required	default	description
fields	map	false		customize additional fields added to the event

For example:

Example

sources:
- type: file
  name: access
  paths:
  - /var/log/*.log
  fields:
    service: demo

service: demo will be added to all collected logs.

fieldsFromEnv

field	type	required	default	description
fieldsFromEnv	map	false		additional fields added to the event, the value is the key of the env

For example:

Example

sources:
- type: file
  name: access
  paths:
  - /var/log/*.log
  fieldsFromEnv:
    service: SVC_NAME

Loggie will get the value ${SVC_NAME} from the environment variable where Loggie is located, and then add a field to all log events:service: ${SVC_NAME}.

fieldsUnderRoot

field	type	required	default	description
fieldsUnderRoot	bool	false	false	whether the additional fields are placed at the root of the event

For example, by default, the output log format is:

{
    "body": "hello world",
    "fields": {
        "service": "demo"
    }
}

If you set fieldsUnderRoot=true, the output log format is:

{
    "body": "hello world",
    "service": "demo"
}

fieldsUnderKey

field	type	required	default	description
fieldsUnderKey	string	false	fields	if fieldsUnderRoot=false, the key of the field

For example, you can modify the default field fields to be tag:

{
    "body": "hello world",
    "tag": {
        "service": "demo"
    }
}

codec

field	type	required	default	description
codec		false		When the source receives data, it is used for parsing and preprocessing
codec.type	string	false	none

Please note: Currently only file source supports source codec.

type: json

field	type	required	default	description
codec.bodyFields		false		Use this field in the json data asbody

Configuration example:

type: json

  sources:
  - type: file
    name: nginx
    paths:
    - /var/log/*.log
    codec:
      type: json
      bodyFields: log

If the collected log is:

{"log":"I0610 08:29:07.698664 Waiting for caches to sync\n", "stream":"stderr", "time:"2021-06-10T08:29:07.698731204Z"}

Then the event after codec is:

body: "I0610 08:29:07.698664 Waiting for caches to sync"

Note: Currently fields other than bodyFields are discarded.

type: regex

field	type	required	default	description
codec.pattern		true		regular expression
codec.bodyFields		true		use the field extracted by the regular as body

Configuration example:

type: regex

  sources:
  - type: file
    name: nginx
    paths:
    - /var/log/*.log
    codec:
      type: regex
      pattern: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$
      bodyFields: log

If the collected log is:

2021-12-01T03:13:58.298476921Z stderr F INFO [main] Starting service [Catalina]

Then the event after codec is:

body: "INFO [main] Starting service [Catalina]"

Note: Currently fields other than bodyFields are discarded.