I have successfully created a serverless OpenSearch collection. In my collection dashboard there is a URL, similar to the one below, specified in "OpenSearch endpoint" (note: the a123 is a fake prefix substituted in for anonymity of original URL):

I am now trying to create an S3 ingestion pipeline. When I specify the host in the pipeline yaml configuration

sink:
  - opensearch:
      hosts: [ "; ]

I receive the following pipeline validation error:

The "opensearch.hosts" does not conform to the standard regex pattern ^https://(vpc-|search-)(?[0-9a-z-]{3,28})-[a-z0-9]{26}.(?[a-z0-9-]{4,20}).es.amazonaws$ or ^https://(vpc-|search-)(?[0-9a-z-]{3,28})-[a-z0-9]{26}.aos.(?[a-z0-9-]{4,20}).on.aws$ for an Amazon OpenSearch domain endpoint. Custom domain endpoints are not compatible with Amazon OpenSearch Ingestion at this time.

From the looks of it, I think the validator is complaining that aoss (the additional 's' for serverless) should instead be aos.

How can an integration pipeline configuration contain a serverless opensearch endpoint without failing validation?

I have successfully created a serverless OpenSearch collection. In my collection dashboard there is a URL, similar to the one below, specified in "OpenSearch endpoint" (note: the a123 is a fake prefix substituted in for anonymity of original URL):

https://a123.us-east-1.aoss.amazonaws

I am now trying to create an S3 ingestion pipeline. When I specify the host in the pipeline yaml configuration

sink:
  - opensearch:
      hosts: [ "https://a123.us-east-1.aoss.amazonaws" ]

I receive the following pipeline validation error:

The "opensearch.hosts" https://a123.us-east-1.aoss.amazonaws does not conform to the standard regex pattern ^https://(vpc-|search-)(?[0-9a-z-]{3,28})-[a-z0-9]{26}.(?[a-z0-9-]{4,20}).es.amazonaws$ or ^https://(vpc-|search-)(?[0-9a-z-]{3,28})-[a-z0-9]{26}.aos.(?[a-z0-9-]{4,20}).on.aws$ for an Amazon OpenSearch domain endpoint. Custom domain endpoints are not compatible with Amazon OpenSearch Ingestion at this time.

From the looks of it, I think the validator is complaining that aoss (the additional 's' for serverless) should instead be aos.

How can an integration pipeline configuration contain a serverless opensearch endpoint without failing validation?

• amazon-web-services
• amazon-opensearch

Share Improve this question Follow edited Nov 20, 2024 at 17:11 Ramón J Romero y Vigil asked Nov 20, 2024 at 15:56 Ramón J Romero y VigilRamón J Romero y Vigil 18k77 gold badges8383 silver badges137137 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

I ran into a similar issue. Just make sure serverless: is set to true and the schema will be correct:

 sink:
    - opensearch:
        hosts:https://a123.us-east-1.aoss.amazonaws
        aws:
          sts_role_arn: arn:aws:iam::123:role
          region: us-east-1
          serverless: true