Storage.signUrl() fails with default credentials in java8 standard runtime

Invocation:

// initialization
storage = StorageOptions.getDefaultInstance().getService();

// later
storage.signUrl(info, 5L, TimeUnit.MINUTES,
                Storage.SignUrlOption.httpMethod(HttpMethod.valueOf(method)),
                Storage.SignUrlOption.withContentType()
        );

Stacktrace:

Caused by: java.lang.IllegalStateException: Signing key was not provided and could not be derived
	at com.google.common.base.Preconditions.checkState(Preconditions.java:444)
	at com.google.cloud.storage.StorageImpl.signUrl(StorageImpl.java:508)

I tracked the problem down to the com.google.auth.oauth2.GoogleCredentials.getDefaultCredentialsUnsynchronized()(https://github.com/google/google-auth-library-java/blob/51a5445b33d10f252cadfdcca82dd9e68512e483/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L182) where it skips over tryGetAppEngineCredential() to return an instance of com.google.auth.oauth2.AppengineCredentials which is one of the implementations of ServiceAccountSigner required by the signUrl call with default credentials.

This may also affect other services assuming an instance of com.google.auth.oauth2.AppengineCredentials.

Is there any specific reason why to check for java7 only?

Background: We moved to the java8 runtime on GAE and upgraded our api clients to the google-cloud-java api clients version 1.10.0.

@neozwu

@neozwu Do you have context around this? It looks like the relevant auth library code was last updated in these PRs:

@lesv

@phimar For GAE java 8 standard, metadata server is used to retreive credentials. The current implementation for AppengineCredentials uses appengine APIs, which only work for java 7. It seems that ComputeEngineCredentials didn't implement ServiceAccountSigner. @lesv @ludoch do you have any context regarding this?

/cc @lesv @ludoch

No, but its a very important thing to fix.

Thank you for your comments and the linkage to the issue on google/google-auth-library-java.

I can successfully sign urls with the default storage options (on Java8 GAE Standard) using google-cloud 0.20.3-alpha. When I upgrade to 0.32.0-alpha I see this error.

I've reverted to the old version.

Same issue is present in flexible runtime too?

@vchudnov-g

@vchudnov-g FYI
@jabubake FYI
@garrettjonesgoogle FYI

@frankyn

@frankyn FYI

google.auth.version has been bumped. Please update the library to the latest version (1.38.0) and see if the problem is resolved.

I had the same problem (using 1.35.0):
Caused by: java.lang.IllegalStateException: Signing key was not provided and could not be derived

Using AppEngine standard, I solved like this:

        static final AppIdentityService identityService = AppIdentityServiceFactory.getAppIdentityService();

        SignUrlOption signWith = SignUrlOption.signWith(new ServiceAccountSigner() {
            @Override
            public byte[] sign(byte[] toSign) {
                return identityService.signForApp(toSign).getSignature();
            }

            @Override
            public String getAccount() {
                return identityService.getServiceAccountName();
            }
        });

It would seem that this method documentation no longer applies to GCE, right?
https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/Storage.java#L2045-L2047

Based on the above PRs and commentary, when running on GCE with google-cloud-storage v1.38.0, the application default credentials (ADC) should be sufficient to sign a blob, yes?

Wondering if anything else has to be done as we are still having trouble (different error message, getting a little farther in the code as far as execution) and not sure why things fail when running on GCE using ADC vs. when we provide the key file directly via GOOGLE_APPLICATION_CREDENTIALS.

thanks y'all for the support.

@NicolaSpreafico

The ComputeEngineCredentials signing method requires 2 things:

The Identity and Access Management (IAM) API must be enabled for the project
The Compute Engine service account (the default App Engine service account for GAE Standard Java 8) needs the iam.serviceAccounts.signBlob permission (available to the "Service Account Token Creator" role).

@NicolaSpreafico try updating the com.google.cloud:google-cloud-storage dependency to 1.38.0 (or com.google.auth:google-auth-library-oauth2-http to 0.10.0) and ensuring the default service account has the necessary IAM access.

@noahdietz ADC prefers the key file provided by GOOGLE_APPLICATION_CREDENTIALS over the Compute Engine credentials and uses 2 different strategies for signing URLs. If you're seeing a SigningException you can inspect the underlying cause by catching the SigningException and looking at the Throwable from getCause(). It's likely that you need to set up the IAM config above.

Created googleapis/google-auth-library-java#175 for documentation and will open a new issue for docs here as well.

@chingor13

@chingor13 fixed! we had set all of the scopes and IAM policies we thought were necessary (devstorage.full_control & bucket objectAdmin). Didn't work. Turns out we needed to use the cloud-platform scope on the instance instead. At least that was the only thing that changed and it started working.

Thanks for the help.

Greetings folks! It looks like this issue has been resolved. If you're still running into problems... let us know!

I needed to grant iam.serviceAccounts.signBlob in order for the default service account to sign URLs while running in App Engine, but running locally with the same credentials didn't require this---is this a bug?

deps: update core dependencies to v1.25.0 (#2629)

chore(main): release 3.9.1 (#2638)

chore: remove old native image kokoro downstream jobs (#2629)

chore(deps): update dependency com.google.cloud:libraries-bom to v26.…

chore(main): release 2.62.1-SNAPSHOT (googleapis#2629)

Navigation Menu

Repository navigation

Storage.signUrl() fails with default credentials in java8 standard runtime #2629

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Storage.signUrl() fails with default credentials in java8 standard runtime #2629

Description

Activity

michaelbausor commented on Nov 15, 2017

neozwu commented on Nov 15, 2017

lesv commented on Nov 18, 2017

philnieh commented on Nov 21, 2017

stickfigure commented on Jan 8, 2018

ValdasK commented on Jan 23, 2018

lesv commented on Jan 24, 2018

jabubake commented on Jan 25, 2018

11 remaining items

yihanzhen commented on Aug 7, 2018

NicolaSpreafico commented on Aug 9, 2018

noahdietz commented on Aug 9, 2018

chingor13 commented on Aug 9, 2018

noahdietz commented on Aug 9, 2018

JustinBeckwith commented on Sep 13, 2018

khoek commented on Nov 1, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions