I have a spring-boot application running on elastic beanstalk, and it communicates with other instances in the ELB-cluster through hazelcast. Or rather, it did.

We are in the process of upgrading the application to newer spring-boot versions, and in the process moved to hazelcast 5.1.7. Now we're facing a peculiar issue with the auto-discovery.

I configured hazlecast without role, access key or secret key:

    private fun Config.configureAwsDiscovery(awsProperties: AwsProperties) {
        // enable discovery
        setProperty("hazelcast.discovery.enabled", "true")

        networkConfig.join.apply {
            tcpIpConfig.isEnabled = false
            multicastConfig.isEnabled = false
            awsConfig = AwsConfig().apply {
                isEnabled = true
                setProperty("region", awsProperties.region)
                setProperty("host-header", awsProperties.hz.hostReader)
                setProperty("security-group-name", awsProperties.securityGroup)
                setProperty("tag-key", awsProperties.hz.tagKey)
                setProperty("tag-value", awsProperties.hz.tagValue)
                setProperty("hz-port", awsProperties.hz.port)
            }
        }
    }

The idea being that it picks up on the role attached to the elastic beanstalk instance and uses that. The role has DescribeInstance permissions for everything, and was in fact used prior to that for the same job. Hazelcast picks up the role well enough, but then seems unable to get its credentials to perform AWS requests:

com.hazelcast.aws.AwsDiscoveryStrategy   : Using AWS discovery plugin with configuration: AwsConfig{accessKey='***', secretKey='***', iamRole='arn:aws:iam::491898696816:role/webcam-service-ec2-role-development', region='eu-central-1', hostHeader='ec2.amazonaws', securityGroupName='webcam-service-sg-development', tags='[(key=hazelcast, value=webcam-service-development)]', hzPort=5701-5701, cluster='null', family='null', serviceName='null', connectionTimeoutSeconds=10, connectionRetries=3, readTimeoutSeconds=10}

com.hazelcast.aws.AwsClientConfigurator  : AWS plugin performing discovery in EC2 environment for region: 'eu-central-1' filtered by: 'hz-port:5701-5701, security-group-name:webcam-service-sg-development, tag-key:hazelcast, tag-value:webcam-service-development'

c.hazelcast.aws.AwsCredentialsProvider   : Fetching AWS Credentials using EC2 IAM Role: arn:aws:iam::491898696816:role/webcam-service-ec2-role-development

com.hazelcast.config.InvalidConfigurationException: Unable to retrieve credentials from IAM Role: 'arn:aws:iam::491898696816:role/webcam-service-ec2-role-development', please make sure it's attached to your EC2 Instance

It says to make sure that the role is attached to the instance, which is a bit ironic, since if it wasn't, it wouldn't even have known about it. That arn is nowhere in the code or the configuration. It's also a weird point for the process to fail. It has the role, but it seems unable to retrieve its current credentials, which is perplexing. It can do that neither with or without IMDSv1 enabled (because I know that the older version we used fetched the credentials through IMDSv1).

After a while of fruitlessly trying to figure out what's going wrong with that role, I created a user with the same policy, created credentials for it, and handed the access and secret keys to hazelcast explicitly. And this works fine, discovery and all:

    private fun Config.configureAwsDiscovery(awsProperties: AwsProperties) {
        // enable discovery
        setProperty("hazelcast.discovery.enabled", "true")

        networkConfig.join.apply {
            tcpIpConfig.isEnabled = false
            multicastConfig.isEnabled = false
            awsConfig = AwsConfig().apply {
                isEnabled = true
                setProperty("access-key", awsProperties.accessKey)
                setProperty("secret-key", awsProperties.secretKey)
                setProperty("region", awsProperties.region)
                setProperty("host-header", awsProperties.hz.hostReader)
                setProperty("security-group-name", awsProperties.securityGroup)
                setProperty("tag-key", awsProperties.hz.tagKey)
                setProperty("tag-value", awsProperties.hz.tagValue)
                setProperty("hz-port", awsProperties.hz.port)
            }
        }
    }

While that is... fine, I would really much rather get rid of that user again and instead use the instance role. Can anybody take a guess at what's going wrong here?