The downloaded data did not match the data from the server

Hey,
Any files that I get from the bucket "pubsite_prod_rev_", gets error code: CONTENT_DOWNLOAD_MISMATCH.

bucket.file("reviews/reviews_com.sample.android_201409.csv").download({
  destination: './reviews.cvs'
},
function(err, content){
  console.log(err.code);//CONTENT_DOWNLOAD_MISMATCH
});

:(

@stephenplusplus

Any thougths @stephenplusplus ?

@beshkenadze

@beshkenadze sorry that it's taken so long to get on this. It's hard to say what could be causing this or if it's related to #651.

The error is being returned because either an MD5 or CRC32c validation check isn't passing. In other words, the data you've received isn't matching the data that's stored in the bucket. You can workaround this by disabling validation:

bucket.file("reviews/reviews_com.sample.android_201409.csv").download({
  destination: './reviews.cvs',
  validation: false
},
function(err, content){
  // No more mismatch error (hopefully)
});

If you want to do some debugging, I put up a branch you can swap out your gcloud dependency for. It will just do a little console.log-ing to help see what's going on:

$ npm install --save stephenplusplus/gcloud-node#spp--654

@stephenplusplus

Hey @stephenplusplus,
How is getting of the hash from a local file?
Play Сloud gives files compressed in GZ format.
Possible hashes are calculated from the extracted file?

@beshkenadze

Not sure I understand the question @beshkenadze ...

To give some background (sorry if you already know this, just adding for context):

Play is uploading files to a bucket that they own
Play then shares access to the bucket with your Google account (not your project's service accounts)
When you download a file, gcloud-node does some validation to make sure that the data wasn't tampered with along the way (see https://github.com/GoogleCloudPlatform/gcloud-node/blob/d4d3d06b466834fd8a19771e75ff5cd9f89013fa/lib/storage/file.js#L491)
If the validation fails (ie, MD5 or CRC32 don't match what the file should have), we raise the error you're seeing (see https://github.com/GoogleCloudPlatform/gcloud-node/blob/d4d3d06b466834fd8a19771e75ff5cd9f89013fa/lib/storage/file.js#L548)

I don't see anything that would indicate that we're looking at the uncompressed file, as we're treating the data as nothing more than bytes and ignoring the file type all together.

It could be possible that Play (when uploading the data) is somehow bypassing the part where they set the CRC32 and MD5 hash for the file (which would cause this error to happen on all Play-uploaded files). Could you tell us what you get back in the headers that start with x-goog-hash when you GET the files from GCS?

/cc @stephenplusplus

Now find the real file and will try to show an example.

Could you tell us what you get back in the headers that start with x-goog-hash when you GET the files from GCS?

That's what this will do:

$ npm install --save stephenplusplus/gcloud-node#spp--654

@stephenplusplus

@stephenplusplus version ("version": "0.8.1") to old :)

That branch (spp--654) is tracking master: https://github.com/stephenplusplus/gcloud-node/tree/spp--654

I used request-debug and got this:

{
  response: {
    debugId: 1,
    headers: {
      'x-guploader-uploadid': 'XXXX',
      expires: 'Mon, 20 Jul 2015 13:40:35 GMT',
      date: 'Mon, 20 Jul 2015 13:40:35 GMT',
      'cache-control': 'private, max-age=0',
      'last-modified': 'Sun, 19 Jul 2015 18:53:59 GMT',
      etag: 'W/"XXXX"',
      'x-goog-generation': '1437332039288000',
      'x-goog-metageneration': '1',
      'x-goog-stored-content-encoding': 'gzip',
      'x-goog-stored-content-length': '5939',
      'content-type': 'text/csv; charset=utf-16le',
      'x-goog-hash': 'crc32c=66rJzQ==, md5=2T/NKanU9vTItoiF7+tMAA==',
      'x-goog-storage-class': 'STANDARD',
      vary: 'Accept-Encoding',
      'content-length': '24148',
      server: 'UploadServer',
      'alternate-protocol': '443:quic,p=1',
      connection: 'close'
    },
    statusCode: 200
  }
}

Nice :) Using my branch will show the hashes that are being built locally as well.

@stephenplusplus

@stephenplusplus ;)

Headers: { 'x-guploader-uploadid': 'XXXX',
  expires: 'Mon, 20 Jul 2015 13:46:35 GMT',
  date: 'Mon, 20 Jul 2015 13:46:35 GMT',
  'cache-control': 'private, max-age=0',
  'last-modified': 'Sun, 19 Jul 2015 18:53:59 GMT',
  etag: 'W/"XXXX"',
  'x-goog-generation': '1437332039288000',
  'x-goog-metageneration': '1',
  'x-goog-stored-content-encoding': 'gzip',
  'x-goog-stored-content-length': '5939',
  'content-type': 'text/csv; charset=utf-16le',
  'x-goog-hash': 'crc32c=66rJzQ==, md5=2T/NKanU9vTItoiF7+tMAA==',
  'x-goog-storage-class': 'STANDARD',
  vary: 'Accept-Encoding',
  'content-length': '24148',
  server: 'UploadServer',
  'alternate-protocol': '443:quic,p=1',
  connection: 'close' }

Local CRC32c Hash: Fw==
Local MD5 Hash: Hwt6cw9joXTy4EOtQqh0pg==
crypto.js:126
  return this._handle.digest(outputEncoding);
                      ^
Error: Not initialized
    at Error (native)

@beshkenadze

Those don't match even a little bit! Like you pointed out @beshkenadze, I think we're running into issues because request automatically decodes the file as it's being downloaded, resulting in different hashes. I can't think of a great solution immediately for how we can work around this, other than:

shut off the auto-decoding for all downloads (don't think we want this),
ignore the hash mismatch if we see the file was gzip'd in the response headers,
branch off from the request download stream, and run the calculation on the native http.IncomingMessage response stream (which won't do the decoding)

Can we run a quick test of uploading a zipped file, and seeing that the verification fails ? I'd feel much more confident if this was happening on a non-magic bucket (the one in question here is actually owned by Google Play, so it might be doing something wonky).