OpenSearchIngestion / Client / create_pipeline

create_pipeline#

OpenSearchIngestion.Client.create_pipeline(**kwargs)#

Creates an OpenSearch Ingestion pipeline. For more information, see Creating Amazon OpenSearch Ingestion pipelines.

See also: AWS API Documentation

Request Syntax

response = client.create_pipeline(
    PipelineName='string',
    MinUnits=123,
    MaxUnits=123,
    PipelineConfigurationBody='string',
    LogPublishingOptions={
        'IsLoggingEnabled': True|False,
        'CloudWatchLogDestination': {
            'LogGroup': 'string'
        }
    },
    VpcOptions={
        'SubnetIds': [
            'string',
        ],
        'SecurityGroupIds': [
            'string',
        ],
        'VpcAttachmentOptions': {
            'AttachToVpc': True|False,
            'CidrBlock': 'string'
        },
        'VpcEndpointManagement': 'CUSTOMER'|'SERVICE'
    },
    BufferOptions={
        'PersistentBufferEnabled': True|False
    },
    EncryptionAtRestOptions={
        'KmsKeyArn': 'string'
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
Parameters:

  • PipelineName (string) –

    [REQUIRED]

    The name of the OpenSearch Ingestion pipeline to create. Pipeline names are unique across the pipelines owned by an account within an Amazon Web Services Region.

  • MinUnits (integer) –

    [REQUIRED]

    The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

  • MaxUnits (integer) –

    [REQUIRED]

    The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

  • PipelineConfigurationBody (string) –

    [REQUIRED]

    The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n.

  • LogPublishingOptions (dict) –

    Key-value pairs to configure log publishing.

    • IsLoggingEnabled (boolean) –

      Whether logs should be published.

    • CloudWatchLogDestination (dict) –

      The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.

      • LogGroup (string) – [REQUIRED]

        The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/vendedlogs/OpenSearchService/pipelines.

  • VpcOptions (dict) –

    Container for the values required to configure VPC access for the pipeline. If you don’t specify these values, OpenSearch Ingestion creates the pipeline with a public endpoint.

    • SubnetIds (list) – [REQUIRED]

      A list of subnet IDs associated with the VPC endpoint.

      • (string) –

    • SecurityGroupIds (list) –

      A list of security groups associated with the VPC endpoint.

      • (string) –

    • VpcAttachmentOptions (dict) –

      Options for attaching a VPC to a pipeline.

      • AttachToVpc (boolean) – [REQUIRED]

        Whether a VPC is attached to the pipeline.

      • CidrBlock (string) –

        The CIDR block to be reserved for OpenSearch Ingestion to create elastic network interfaces (ENIs).

    • VpcEndpointManagement (string) –

      Defines whether you or Amazon OpenSearch Ingestion service create and manage the VPC endpoint configured for the pipeline.

  • BufferOptions (dict) –

    Key-value pairs to configure persistent buffering for the pipeline.

    • PersistentBufferEnabled (boolean) – [REQUIRED]

      Whether persistent buffering should be enabled.

  • EncryptionAtRestOptions (dict) –

    Key-value pairs to configure encryption for data that is written to a persistent buffer.

    • KmsKeyArn (string) – [REQUIRED]

      The ARN of the KMS key used to encrypt buffer data. By default, data is encrypted using an Amazon Web Services owned key.

  • Tags (list) –

    List of tags to add to the pipeline upon creation.

    • (dict) –

      A tag (key-value pair) for an OpenSearch Ingestion pipeline.

      • Key (string) – [REQUIRED]

        The tag key. Tag keys must be unique for the pipeline to which they are attached.

      • Value (string) – [REQUIRED]

        The value assigned to the corresponding tag key. Tag values can be null and don’t have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

Return type:

dict

Returns:

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ],
                    'VpcAttachmentOptions': {
                        'AttachToVpc': True|False,
                        'CidrBlock': 'string'
                    },
                    'VpcEndpointManagement': 'CUSTOMER'|'SERVICE'
                }
            },
        ],
        'BufferOptions': {
            'PersistentBufferEnabled': True|False
        },
        'EncryptionAtRestOptions': {
            'KmsKeyArn': 'string'
        },
        'VpcEndpointService': 'string',
        'ServiceVpcEndpoints': [
            {
                'ServiceName': 'OPENSEARCH_SERVERLESS',
                'VpcEndpointId': 'string'
            },
        ],
        'Destinations': [
            {
                'ServiceName': 'string',
                'Endpoint': 'string'
            },
        ],
        'Tags': [
            {
                'Key': 'string',
                'Value': 'string'
            },
        ]
    }
}

Response Structure

  • (dict) –

    • Pipeline (dict) –

      Container for information about the created pipeline.

      • PipelineName (string) –

        The name of the pipeline.

      • PipelineArn (string) –

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) –

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) –

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) –

        The current status of the pipeline.

      • StatusReason (dict) –

        The reason for the current status of the pipeline.

        • Description (string) –

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) –

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) –

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) –

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) –

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) –

      • LogPublishingOptions (dict) –

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) –

          Whether logs should be published.

        • CloudWatchLogDestination (dict) –

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.

          • LogGroup (string) –

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/vendedlogs/OpenSearchService/pipelines.

      • VpcEndpoints (list) –

        The VPC interface endpoints that have access to the pipeline.

        • (dict) –

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) –

            The unique identifier of the endpoint.

          • VpcId (string) –

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) –

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) –

              A list of subnet IDs associated with the VPC endpoint.

              • (string) –

            • SecurityGroupIds (list) –

              A list of security groups associated with the VPC endpoint.

              • (string) –

            • VpcAttachmentOptions (dict) –

              Options for attaching a VPC to a pipeline.

              • AttachToVpc (boolean) –

                Whether a VPC is attached to the pipeline.

              • CidrBlock (string) –

                The CIDR block to be reserved for OpenSearch Ingestion to create elastic network interfaces (ENIs).

            • VpcEndpointManagement (string) –

              Defines whether you or Amazon OpenSearch Ingestion service create and manage the VPC endpoint configured for the pipeline.

      • BufferOptions (dict) –

        Options that specify the configuration of a persistent buffer. To configure how OpenSearch Ingestion encrypts this data, set the EncryptionAtRestOptions. For more information, see Persistent buffering.

        • PersistentBufferEnabled (boolean) –

          Whether persistent buffering should be enabled.

      • EncryptionAtRestOptions (dict) –

        Options to control how OpenSearch encrypts buffer data.

        • KmsKeyArn (string) –

          The ARN of the KMS key used to encrypt buffer data. By default, data is encrypted using an Amazon Web Services owned key.

      • VpcEndpointService (string) –

        The VPC endpoint service name for the pipeline.

      • ServiceVpcEndpoints (list) –

        A list of VPC endpoints that OpenSearch Ingestion has created to other Amazon Web Services services.

        • (dict) –

          A container for information about VPC endpoints that were created to other services

          • ServiceName (string) –

            The name of the service for which a VPC endpoint was created.

          • VpcEndpointId (string) –

            The unique identifier of the VPC endpoint that was created.

      • Destinations (list) –

        Destinations to which the pipeline writes data.

        • (dict) –

          An object representing the destination of a pipeline.

          • ServiceName (string) –

            The name of the service receiving data from the pipeline.

          • Endpoint (string) –

            The endpoint receiving data from the pipeline.

      • Tags (list) –

        A list of tags associated with the given pipeline.

        • (dict) –

          A tag (key-value pair) for an OpenSearch Ingestion pipeline.

          • Key (string) –

            The tag key. Tag keys must be unique for the pipeline to which they are attached.

          • Value (string) –

            The value assigned to the corresponding tag key. Tag values can be null and don’t have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

Exceptions