CloudWatchLogs / Client / put_transformer

put_transformer#

CloudWatchLogs.Client.put_transformer(**kwargs)#

Creates or updates a log transformer for a single log group. You use log transformers to transform log events into a different format, making them easier for you to process and analyze. You can also transform logs from different sources into standardized formats that contains relevant, source-specific information.

After you have created a transformer, CloudWatch Logs performs the transformations at the time of log ingestion. You can then refer to the transformed versions of the logs during operations such as querying with CloudWatch Logs Insights or creating metric filters or subscription filers.

You can also use a transformer to copy metadata from metadata keys into the log events themselves. This metadata can include log group name, log stream name, account ID and Region.

A transformer for a log group is a series of processors, where each processor applies one type of transformation to the log events ingested into this log group. The processors work one after another, in the order that you list them, like a pipeline. For more information about the available processors to use in a transformer, see Processors that you can use.

Having log events in standardized format enables visibility across your applications for your log analysis, reporting, and alarming needs. CloudWatch Logs provides transformation for common log types with out-of-the-box transformation templates for major Amazon Web Services log sources such as VPC flow logs, Lambda, and Amazon RDS. You can use pre-built transformation templates or create custom transformation policies.

You can create transformers only for the log groups in the Standard log class.

You can also set up a transformer at the account level. For more information, see PutAccountPolicy. If there is both a log-group level transformer created with PutTransformer and an account-level transformer that could apply to the same log group, the log group uses only the log-group level transformer. It ignores the account-level transformer.

Request Syntax

response = client.put_transformer(
    logGroupIdentifier='string',
    transformerConfig=[
        {
            'addKeys': {
                'entries': [
                    {
                        'key': 'string',
                        'value': 'string',
                        'overwriteIfExists': True|False
                    },
                ]
            },
            'copyValue': {
                'entries': [
                    {
                        'source': 'string',
                        'target': 'string',
                        'overwriteIfExists': True|False
                    },
                ]
            },
            'csv': {
                'quoteCharacter': 'string',
                'delimiter': 'string',
                'columns': [
                    'string',
                ],
                'source': 'string'
            },
            'dateTimeConverter': {
                'source': 'string',
                'target': 'string',
                'targetFormat': 'string',
                'matchPatterns': [
                    'string',
                ],
                'sourceTimezone': 'string',
                'targetTimezone': 'string',
                'locale': 'string'
            },
            'deleteKeys': {
                'withKeys': [
                    'string',
                ]
            },
            'grok': {
                'source': 'string',
                'match': 'string'
            },
            'listToMap': {
                'source': 'string',
                'key': 'string',
                'valueKey': 'string',
                'target': 'string',
                'flatten': True|False,
                'flattenedElement': 'first'|'last'
            },
            'lowerCaseString': {
                'withKeys': [
                    'string',
                ]
            },
            'moveKeys': {
                'entries': [
                    {
                        'source': 'string',
                        'target': 'string',
                        'overwriteIfExists': True|False
                    },
                ]
            },
            'parseCloudfront': {
                'source': 'string'
            },
            'parseJSON': {
                'source': 'string',
                'destination': 'string'
            },
            'parseKeyValue': {
                'source': 'string',
                'destination': 'string',
                'fieldDelimiter': 'string',
                'keyValueDelimiter': 'string',
                'keyPrefix': 'string',
                'nonMatchValue': 'string',
                'overwriteIfExists': True|False
            },
            'parseRoute53': {
                'source': 'string'
            },
            'parsePostgres': {
                'source': 'string'
            },
            'parseVPC': {
                'source': 'string'
            },
            'parseWAF': {
                'source': 'string'
            },
            'renameKeys': {
                'entries': [
                    {
                        'key': 'string',
                        'renameTo': 'string',
                        'overwriteIfExists': True|False
                    },
                ]
            },
            'splitString': {
                'entries': [
                    {
                        'source': 'string',
                        'delimiter': 'string'
                    },
                ]
            },
            'substituteString': {
                'entries': [
                    {
                        'source': 'string',
                        'from': 'string',
                        'to': 'string'
                    },
                ]
            },
            'trimString': {
                'withKeys': [
                    'string',
                ]
            },
            'typeConverter': {
                'entries': [
                    {
                        'key': 'string',
                        'type': 'boolean'|'integer'|'double'|'string'
                    },
                ]
            },
            'upperCaseString': {
                'withKeys': [
                    'string',
                ]
            }
        },
    ]
)

Parameters:

logGroupIdentifier (string) –
[REQUIRED]

Specify either the name or ARN of the log group to create the transformer for.
transformerConfig (list) –
[REQUIRED]

This structure contains the configuration of this log transformer. A log transformer is an array of processors, where each processor applies one type of transformation to the log events that are ingested.
- (dict) –
  
  This structure contains the information about one processor in a log transformer.
  - addKeys (dict) –
    
    Use this parameter to include the addKeys processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of objects, where each object contains the information about one key to add to the log event.
      - (dict) –
        
        This object defines one key that will be added with the addKeys processor.
        
        key (string) – [REQUIRED]
        
        The key of the new entry to be added to the log event
        
        value (string) – [REQUIRED]
        
        The value of the new entry to be added to the log event
        
        overwriteIfExists (boolean) –
        
        Specifies whether to overwrite the value if the key already exists in the log event. If you omit this, the default is false.
  - copyValue (dict) –
    
    Use this parameter to include the copyValue processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of CopyValueEntry objects, where each object contains the information about one field value to copy.
      - (dict) –
        
        This object defines one value to be copied with the copyValue processor.
        
        source (string) – [REQUIRED]
        
        The key to copy.
        
        target (string) – [REQUIRED]
        
        The key of the field to copy the value to.
        
        overwriteIfExists (boolean) –
        
        Specifies whether to overwrite the value if the destination key already exists. If you omit this, the default is false.
  - csv (dict) –
    
    Use this parameter to include the CSV processor in your transformer.
    - quoteCharacter (string) –
      
      The character used used as a text qualifier for a single column of data. If you omit this, the double quotation mark " character is used.
    - delimiter (string) –
      
      The character used to separate each column in the original comma-separated value log event. If you omit this, the processor looks for the comma , character as the delimiter.
    - columns (list) –
      
      An array of names to use for the columns in the transformed log event.
      
      If you omit this, default column names ( [column_1, column_2 ...]) are used.
      - (string) –
    - source (string) –
      
      The path to the field in the log event that has the comma separated values to be parsed. If you omit this value, the whole log message is processed.
  - dateTimeConverter (dict) –
    
    Use this parameter to include the datetimeConverter processor in your transformer.
    - source (string) – [REQUIRED]
      
      The key to apply the date conversion to.
    - target (string) – [REQUIRED]
      
      The JSON field to store the result in.
    - targetFormat (string) –
      
      The datetime format to use for the converted data in the target field.
      
      If you omit this, the default of yyyy-MM-dd'T'HH:mm:ss.SSS'Z is used.
    - matchPatterns (list) – [REQUIRED]
      
      A list of patterns to match against the source field.
      - (string) –
    - sourceTimezone (string) –
      
      The time zone of the source field. If you omit this, the default used is the UTC zone.
    - targetTimezone (string) –
      
      The time zone of the target field. If you omit this, the default used is the UTC zone.
    - locale (string) –
      
      The locale of the source field. If you omit this, the default of locale.ROOT is used.
  - deleteKeys (dict) –
    
    Use this parameter to include the deleteKeys processor in your transformer.
    - withKeys (list) – [REQUIRED]
      
      The list of keys to delete.
      - (string) –
  - grok (dict) –
    
    Use this parameter to include the grok processor in your transformer.
    - source (string) –
      
      The path to the field in the log event that you want to parse. If you omit this value, the whole log message is parsed.
    - match (string) – [REQUIRED]
      
      The grok pattern to match against the log event. For a list of supported grok patterns, see Supported grok patterns.
  - listToMap (dict) –
    
    Use this parameter to include the listToMap processor in your transformer.
    - source (string) – [REQUIRED]
      
      The key in the log event that has a list of objects that will be converted to a map.
    - key (string) – [REQUIRED]
      
      The key of the field to be extracted as keys in the generated map
    - valueKey (string) –
      
      If this is specified, the values that you specify in this parameter will be extracted from the source objects and put into the values of the generated map. Otherwise, original objects in the source list will be put into the values of the generated map.
    - target (string) –
      
      The key of the field that will hold the generated map
    - flatten (boolean) –
      
      A Boolean value to indicate whether the list will be flattened into single items. Specify true to flatten the list. The default is false
    - flattenedElement (string) –
      
      If you set flatten to true, use flattenedElement to specify which element, first or last, to keep.
      
      You must specify this parameter if flatten is true
  - lowerCaseString (dict) –
    
    Use this parameter to include the lowerCaseString processor in your transformer.
    - withKeys (list) – [REQUIRED]
      
      The array caontaining the keys of the fields to convert to lowercase.
      - (string) –
  - moveKeys (dict) –
    
    Use this parameter to include the moveKeys processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of objects, where each object contains the information about one key to move.
      - (dict) –
        
        This object defines one key that will be moved with the moveKey processor.
        
        source (string) – [REQUIRED]
        
        The key to move.
        
        target (string) – [REQUIRED]
        
        The key to move to.
        
        overwriteIfExists (boolean) –
        
        Specifies whether to overwrite the value if the destination key already exists. If you omit this, the default is false.
  - parseCloudfront (dict) –
    
    Use this parameter to include the parseCloudfront processor in your transformer.
    
    If you use this processor, it must be the first processor in your transformer.
    - source (string) –
      
      Omit this parameter and the whole log message will be processed by this processor. No other value than @message is allowed for source.
  - parseJSON (dict) –
    
    Use this parameter to include the parseJSON processor in your transformer.
    - source (string) –
      
      Path to the field in the log event that will be parsed. Use dot notation to access child fields. For example, store.book
    - destination (string) –
      
      The location to put the parsed key value pair into. If you omit this parameter, it is placed under the root node.
  - parseKeyValue (dict) –
    
    Use this parameter to include the parseKeyValue processor in your transformer.
    - source (string) –
      
      Path to the field in the log event that will be parsed. Use dot notation to access child fields. For example, store.book
    - destination (string) –
      
      The destination field to put the extracted key-value pairs into
    - fieldDelimiter (string) –
      
      The field delimiter string that is used between key-value pairs in the original log events. If you omit this, the ampersand & character is used.
    - keyValueDelimiter (string) –
      
      The delimiter string to use between the key and value in each pair in the transformed log event.
      
      If you omit this, the equal = character is used.
    - keyPrefix (string) –
      
      If you want to add a prefix to all transformed keys, specify it here.
    - nonMatchValue (string) –
      
      A value to insert into the value field in the result, when a key-value pair is not successfully split.
    - overwriteIfExists (boolean) –
      
      Specifies whether to overwrite the value if the destination key already exists. If you omit this, the default is false.
  - parseRoute53 (dict) –
    
    Use this parameter to include the parseRoute53 processor in your transformer.
    
    If you use this processor, it must be the first processor in your transformer.
    - source (string) –
      
      Omit this parameter and the whole log message will be processed by this processor. No other value than @message is allowed for source.
  - parsePostgres (dict) –
    
    Use this parameter to include the parsePostGres processor in your transformer.
    
    If you use this processor, it must be the first processor in your transformer.
    - source (string) –
      
      Omit this parameter and the whole log message will be processed by this processor. No other value than @message is allowed for source.
  - parseVPC (dict) –
    
    Use this parameter to include the parseVPC processor in your transformer.
    
    If you use this processor, it must be the first processor in your transformer.
    - source (string) –
      
      Omit this parameter and the whole log message will be processed by this processor. No other value than @message is allowed for source.
  - parseWAF (dict) –
    
    Use this parameter to include the parseWAF processor in your transformer.
    
    If you use this processor, it must be the first processor in your transformer.
    - source (string) –
      
      Omit this parameter and the whole log message will be processed by this processor. No other value than @message is allowed for source.
  - renameKeys (dict) –
    
    Use this parameter to include the renameKeys processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of RenameKeyEntry objects, where each object contains the information about a single key to rename.
      - (dict) –
        
        This object defines one key that will be renamed with the renameKey processor.
        
        key (string) – [REQUIRED]
        
        The key to rename
        
        renameTo (string) – [REQUIRED]
        
        The string to use for the new key name
        
        overwriteIfExists (boolean) –
        
        Specifies whether to overwrite the existing value if the destination key already exists. The default is false
  - splitString (dict) –
    
    Use this parameter to include the splitString processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of SplitStringEntry objects, where each object contains the information about one field to split.
      - (dict) –
        
        This object defines one log field that will be split with the splitString processor.
        
        source (string) – [REQUIRED]
        
        The key of the field to split.
        
        delimiter (string) – [REQUIRED]
        
        The separator characters to split the string entry on.
  - substituteString (dict) –
    
    Use this parameter to include the substituteString processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of objects, where each object contains the information about one key to match and replace.
      - (dict) –
        
        This object defines one log field key that will be replaced using the substituteString processor.
        
        source (string) – [REQUIRED]
        
        The key to modify
        
        from (string) – [REQUIRED]
        
        The regular expression string to be replaced. Special regex characters such as [ and ] must be escaped using \ when using double quotes and with when using single quotes. For more information, see Class Pattern on the Oracle web site.
        
        to (string) – [REQUIRED]
        
        The string to be substituted for each match of from
  - trimString (dict) –
    
    Use this parameter to include the trimString processor in your transformer.
    - withKeys (list) – [REQUIRED]
      
      The array containing the keys of the fields to trim.
      - (string) –
  - typeConverter (dict) –
    
    Use this parameter to include the typeConverter processor in your transformer.
    - entries (list) – [REQUIRED]
      
      An array of TypeConverterEntry objects, where each object contains the information about one field to change the type of.
      - (dict) –
        
        This object defines one value type that will be converted using the typeConverter processor.
        
        key (string) – [REQUIRED]
        
        The key with the value that is to be converted to a different type.
        
        type (string) – [REQUIRED]
        
        The type to convert the field value to. Valid values are integer, double, string and boolean.
  - upperCaseString (dict) –
    
    Use this parameter to include the upperCaseString processor in your transformer.
    - withKeys (list) – [REQUIRED]
      
      The array of containing the keys of the field to convert to uppercase.
      - (string) –

Returns:

None

put_transformer#

Request Syntax

Exceptions