I have configured an AWS IAM policy to allow all access to secretsmanager on all resources:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:*"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-2:MYACCOUNTID:secret:*"
            ],
            "Effect": "Allow"
        }
    ]
}

And running the AWS CLI command get-secret-value works fine, while batch-get-secret-value gives the error:

An error occurred (AccessDeniedException) when calling the BatchGetSecretValue operation:
 User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not
 authorized to perform: secretsmanager:BatchGetSecretValue because no identity-based policy allows the
 secretsmanager:BatchGetSecretValue action

Here's how it goes:

ubuntu@vapp-3-145-XX-XXX:~$ /snap/bin/aws --region us-east-2 secretsmanager get-secret-value --secret-id secret-secret-account
{
    "ARN": "arn:aws:secretsmanager:us-east-2:MYACCOUNTID:secret:secret-secret-account-2eHacl",
    "Name": "secret-secret-account",
    "VersionId": "b72697d0-4d9d-4bbe-8504-d174d7444510",
    "SecretString": "FOOBAR",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-11-12T14:05:56.132000+00:00"
}
ubuntu@vapp-3-145-XX-XXX:~$ /snap/bin/aws --region us-east-2 secretsmanager batch-get-secret-value --secret-id-list secret-secret-account
An error occurred (AccessDeniedException) when calling the BatchGetSecretValue operation: User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not authorized to perform: secretsmanager:BatchGetSecretValue because no identity-based policy allows the secretsmanager:BatchGetSecretValue action

What gives?

Searching the web I found suggestions to add a "Principal" to the policy, but if I try to do that, the IAM policy editor says that "Unsupported Principal: The policy type IDENTITY_POLICY does not support the Principal element. Remove the Principal element".

Update 1

I verified that there is only one policy that applies to said instance that has anything about secretsmanager - and it is what I expect to find:

$ (
function aws() { /snap/bin/aws --profile MYPROFILE "$@"; };
iamProfile=$(aws ec2 describe-instances --instance-id i-XXXXXXXXXXXXXXXXX | \
  jq -r '.Reservations[].Instances[].IamInstanceProfile.Arn|split("/")|last');
aws iam get-instance-profile --instance-profile "$iamProfile" | \
  jq -r '.InstanceProfile.Roles[].RoleName' | while read roleName; do
aws iam list-attached-role-policies --role-name vapp-role-staging-us-east-2 | \
  jq -r '.AttachedPolicies[].PolicyArn' | while read policyArn; do
aws iam get-policy --policy-arn "$policyArn" | \
  jq -r '.Policy.DefaultVersionId' | while read defVer; do
aws iam get-policy-version --policy-arn "$policyArn" --version-id "$defVer" | \
  jq -r '.PolicyVersion.Document.Statement[].Action|if type == "array" then . else [.] end|.[]';
done;
done;
done;
)|grep secretsmanager
secretsmanager:*

Update 2

We do have Service Control Policies enabled in the anization:

But there's only one - the AWS managed "FullAWSAccess":

And that is applied to the root account, all sub accounts and all anizational unit.

Update 3

I found out that if I replace more ARN parts with wild cards in the Resource field, it doesn't help, but doing "Resource": [ "*" ] does allow the EC2 instance to successfully call BatchGetSecretValue.

Even more interestingly, using the original policy (that I started with but didn't post here) that allows secretsmanager:GetSecretValue on the specific secrets ARNs that I have, and then also adding a statement for secretsmanager:BatchGetSecretValue with no resource limit - i.e.:

{"Version": "2012-10-17","Statement": [
  { "Action": [ "secretsmanager:GetSecretValue" ],
    "Resource": [ "arn:aws:secretsmanager:us-east-2:MYACCUOUNTID:secret:secret-secret-account", ... ],
    "Effect": "Allow" },
  { "Action": [ "secretsmanager:BatchGetSecretValue" ],
    "Resource": [ "*" ],
    "Effect": "Allow" }
]}

This causes the batch-get-secret-value call to return this weird error message:

{
    "SecretValues": [],
    "Errors": [
        {
            "SecretId": "secret-secret-account",
            "ErrorCode": "AccessDeniedException",
            "Message": "User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not authorized to perform: secretsmanager:GetSecretValue on resource: secret-secret-account because no identity-based policy allows the secretsmanager:GetSecretValue action"
        }
    ]
}

The format is fine, I believe, it is the batch API returning a successful result that has an error for one of the requested secrets (the only one), but the internal error is "access denied" even though there is access. Even weirder is that with this configuration, the get-secret-value call fails sometimes (!!!).

My current workaround is to use the horrible policy statement "Action":["secretsmanager:GetSecretValue","secretsmanager:BatchGetSecretValue"],"Resource":"*","Effect":"Allow" which is not good solution because I'm running several applications on the same region and account, and they should not have access to each other's secrets.

Update 4

The inconsistent behavior seems to be about the calls hitting different API servers that take differing amounts of time to get updated with new policy versions - sometimes as long as 10 minutes, but it eventually settles.

I have configured an AWS IAM policy to allow all access to secretsmanager on all resources:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:*"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-2:MYACCOUNTID:secret:*"
            ],
            "Effect": "Allow"
        }
    ]
}

And running the AWS CLI command get-secret-value works fine, while batch-get-secret-value gives the error:

An error occurred (AccessDeniedException) when calling the BatchGetSecretValue operation:
 User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not
 authorized to perform: secretsmanager:BatchGetSecretValue because no identity-based policy allows the
 secretsmanager:BatchGetSecretValue action

Here's how it goes:

ubuntu@vapp-3-145-XX-XXX:~$ /snap/bin/aws --region us-east-2 secretsmanager get-secret-value --secret-id secret-secret-account
{
    "ARN": "arn:aws:secretsmanager:us-east-2:MYACCOUNTID:secret:secret-secret-account-2eHacl",
    "Name": "secret-secret-account",
    "VersionId": "b72697d0-4d9d-4bbe-8504-d174d7444510",
    "SecretString": "FOOBAR",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-11-12T14:05:56.132000+00:00"
}
ubuntu@vapp-3-145-XX-XXX:~$ /snap/bin/aws --region us-east-2 secretsmanager batch-get-secret-value --secret-id-list secret-secret-account
An error occurred (AccessDeniedException) when calling the BatchGetSecretValue operation: User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not authorized to perform: secretsmanager:BatchGetSecretValue because no identity-based policy allows the secretsmanager:BatchGetSecretValue action

What gives?

Searching the web I found suggestions to add a "Principal" to the policy, but if I try to do that, the IAM policy editor says that "Unsupported Principal: The policy type IDENTITY_POLICY does not support the Principal element. Remove the Principal element".

Update 1

I verified that there is only one policy that applies to said instance that has anything about secretsmanager - and it is what I expect to find:

$ (
function aws() { /snap/bin/aws --profile MYPROFILE "$@"; };
iamProfile=$(aws ec2 describe-instances --instance-id i-XXXXXXXXXXXXXXXXX | \
  jq -r '.Reservations[].Instances[].IamInstanceProfile.Arn|split("/")|last');
aws iam get-instance-profile --instance-profile "$iamProfile" | \
  jq -r '.InstanceProfile.Roles[].RoleName' | while read roleName; do
aws iam list-attached-role-policies --role-name vapp-role-staging-us-east-2 | \
  jq -r '.AttachedPolicies[].PolicyArn' | while read policyArn; do
aws iam get-policy --policy-arn "$policyArn" | \
  jq -r '.Policy.DefaultVersionId' | while read defVer; do
aws iam get-policy-version --policy-arn "$policyArn" --version-id "$defVer" | \
  jq -r '.PolicyVersion.Document.Statement[].Action|if type == "array" then . else [.] end|.[]';
done;
done;
done;
)|grep secretsmanager
secretsmanager:*

Update 2

We do have Service Control Policies enabled in the anization:

But there's only one - the AWS managed "FullAWSAccess":

And that is applied to the root account, all sub accounts and all anizational unit.

Update 3

I found out that if I replace more ARN parts with wild cards in the Resource field, it doesn't help, but doing "Resource": [ "*" ] does allow the EC2 instance to successfully call BatchGetSecretValue.

Even more interestingly, using the original policy (that I started with but didn't post here) that allows secretsmanager:GetSecretValue on the specific secrets ARNs that I have, and then also adding a statement for secretsmanager:BatchGetSecretValue with no resource limit - i.e.:

{"Version": "2012-10-17","Statement": [
  { "Action": [ "secretsmanager:GetSecretValue" ],
    "Resource": [ "arn:aws:secretsmanager:us-east-2:MYACCUOUNTID:secret:secret-secret-account", ... ],
    "Effect": "Allow" },
  { "Action": [ "secretsmanager:BatchGetSecretValue" ],
    "Resource": [ "*" ],
    "Effect": "Allow" }
]}

This causes the batch-get-secret-value call to return this weird error message:

{
    "SecretValues": [],
    "Errors": [
        {
            "SecretId": "secret-secret-account",
            "ErrorCode": "AccessDeniedException",
            "Message": "User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not authorized to perform: secretsmanager:GetSecretValue on resource: secret-secret-account because no identity-based policy allows the secretsmanager:GetSecretValue action"
        }
    ]
}

The format is fine, I believe, it is the batch API returning a successful result that has an error for one of the requested secrets (the only one), but the internal error is "access denied" even though there is access. Even weirder is that with this configuration, the get-secret-value call fails sometimes (!!!).

My current workaround is to use the horrible policy statement "Action":["secretsmanager:GetSecretValue","secretsmanager:BatchGetSecretValue"],"Resource":"*","Effect":"Allow" which is not good solution because I'm running several applications on the same region and account, and they should not have access to each other's secrets.

Update 4

The inconsistent behavior seems to be about the calls hitting different API servers that take differing amounts of time to get updated with new policy versions - sometimes as long as 10 minutes, but it eventually settles.

• amazon-web-services
• security
• amazon-iam
• aws-cli
• aws-secrets-manager

Share Improve this question Follow edited Mar 27 at 0:12 Guss asked Mar 26 at 16:45 GussGuss 32.5k1919 gold badges114114 silver badges141141 bronze badges 5

• Do you have any other active IAM policies on the my-server-staging-us-east-2 role? Are there any relevant SCPs in play (within your AWS anization)? FYI you can only add Principal to resource-based policies (resources are things like KMS keys and S3 buckets). – jarmod Commented Mar 26 at 17:01
• @jarmod what are SCPs? There are other policies in that server's IAM role, but none that mention secretsmanager. I'll post an update showing that in a bit. – Guss Commented Mar 26 at 17:12
• Quick search will explain SCPs in detail but basically they're anization-level Service Control Policies that serve as guardrails around IAM policies. If you aren't part of an AWS Organization then they're not relevant. – jarmod Commented Mar 26 at 22:28
• 1 @jarmod I updated the question with the result of reviewing all policies that apply to the instance, and also review of SCPs. I do not believe either are at fault. – Guss Commented Mar 26 at 23:27
• Related to your update 4, yes IAM is an eventually consistent system. – jarmod Commented Mar 27 at 12:06

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

I've raised the same issue on AWS re:Post where I was pointed at the documentation and BKM - which is that BatchGetSecretValue by definition does not act on a specific resource and therefor breaks if it has any resource limit. Instead, what BatchGetSecretValue does is that it just takes the argument list and perform multiple GetSecretValue API calls for the caller, with the caller's security context - so access to specific secrets can be controlled by setting the GetSecretValue permissions correctly.

The BKM is to use the following security policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:BatchGetSecretValue",
        "secretsmanager:ListSecrets"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "SecretARN1",
        "SecretARN2",
        "SecretARN3"
     ]
   }
 ]
}

The BatchGetSecretValue API needs to be able to list all the secrets, so that it can tell which of the secrets that were asked for actually exist, and then it could try to get each of them and will return an errors item in the response (as can be seen in the OP) if it can't access a secret that it saw.