Expand JSON fields - Simple Log Service - Alibaba Cloud Documentation Center

You can use the processor_json plugin to expand JSON fields. This topic describes the parameters of the processor_json plugin and provides configuration examples.

Important

Form-based configuration: Available when you collect text logs and container standard output.
JSON-based configuration: Unavailable when you collect text logs.

Entry point

To use a Logtail plugin to process logs, you can add a plugin configuration when you create or modify a Logtail collection configuration. For more information, see Overview of Logtail plugins for data processing.

Configurations

Important

The processor_json plugin is supported in Logtail 0.16.28 and later versions.

Form-based configuration

Parameters

Set the Processor Type to Expand JSON Field. The following table describes the parameters.

Parameter	Description
Original Field	The name of the original field to expand.
JSON Expansion Depth	The depth for JSON expansion. The default value is 0, which indicates no limit. A value of 1 indicates the current level.
JSON Expansion Connector	The connector for expanded keys. The default value is an underscore (_).
JSON Expansion Field Prefix	The prefix to add to the names of expanded fields.
Expand Array	Specifies whether to expand arrays. This parameter is supported in Logtail 1.8.0 and later versions.
Keep Original Field	If you select this option, the original field is retained in the parsed log.
Error On Missing Original Field	If you select this option, an error is reported if the raw log does not contain the specified original field.
Use Original Field Name As Prefix For Expanded Fields	If you select this option, the system uses the original field name as the prefix for all expanded JSON field names.
Keep Raw Log On Parsing Failure	If you select this option, the raw log is retained if parsing fails.

Example
In this example, the s_key field is expanded. The prefix j is added to the expanded field names. The original field name s_key is also used as a prefix. The following shows a sample configuration:
- Raw log (the file path from which Logtail reads logs)
```
{"s_key":"{\"k1\":{\"k2\":{\"k3\":{\"k4\":{\"k51\":\"51\",\"k52\":\"52\"},\"k41\":\"41\"}}}}"}
```
- Logtail plugin processing configuration
- Processing result

JSON-based configuration

Parameters

Set type to processor_json. The following table describes the parameters in detail.

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the original field to expand.
NoKeyError	Boolean	No	Specifies whether to report an error if the raw log does not contain the specified original field. true (default): An error is reported. false: No error is reported.
ExpandDepth	Int	No	The depth of JSON expansion. The default value is 0, which indicates no limit. A value of 1 indicates the current level, and so on.
ExpandConnector	String	No	The connector for expanded keys. The default value is an underscore (_).
Prefix	String	No	The prefix to add to the names of expanded fields.
KeepSource	Boolean	No	Specifies whether to keep the original field in the parsed log. true (default): The field is kept. false: The field is not kept.
UseSourceKeyAsPrefix	Boolean	No	Specifies whether to use the original field name as a prefix for all expanded JSON field names.
KeepSourceIfParseError	Boolean	No	Specifies whether to keep the raw log if parsing fails. true (default): The log is kept. false: The log is not kept.
ExpandArray	Boolean	No	Specifies whether to expand arrays. This parameter is supported in Logtail 1.8.0 and later versions. false (default): Arrays are not expanded. true: Arrays are expanded. For example, `{"k":["1","2"]}` is expanded to `{"k[0]":"1","k[1]":"2"}`.

Example

In this example, the s_key field is expanded. The prefix j is added to the expanded field names. The original field name s_key is also used as a prefix. The following shows a sample configuration:

Raw log (the file path from which Logtail reads logs)

{"s_key":"{\"k1\":{\"k2\":{\"k3\":{\"k4\":{\"k51\":\"51\",\"k52\":\"52\"},\"k41\":\"41\"}}}}"}

Logtail plugin processing configuration

{
  "processors":[
    {
      "type":"processor_json",
      "detail": {
        "SourceKey": "content",
        "NoKeyError":true,
        "ExpandDepth":0,
        "ExpandConnector":"-",
        "Prefix":"j",
        "KeepSource": false,
        "UseSourceKeyAsPrefix": true
      }
    }
  ]
}

Processing result

References

Configure a Logtail pipeline using API operations:
Configure a processor plugin in the console:
- Collect text logs from a host
- Collect container logs from a cluster in the console (stdout/file)
Collect container logs from a Kubernetes cluster using a CRD (stdout/file)