datastore: blob value is lost

I'm trying to save a Buffer as a property. From what I can see this should be possibe. In this example the Buffer contains 21 bytes:

dataset.save({
  key: dataset.key('Location'),
  data: {
    id: '113-16',
    geometry: new Buffer('010100000000000000000059400000000000006940', 'hex')
  }
}, function (err, key) {
  console.log(err || key);
});

However when I retrieve the entity I get a Buffer with zero length.

dataset.get(dataset.key([ 'Location', 5707702298738688 ]), function (err, entity) {
  console.log(err || entity.data.geometry.length);
});

In your lib/datastore/entity.js, function propertyToValue at line 362 you're creating a Buffer with the property.blob_value as base64:

  if (exists(property.blob_value)) {
    return new Buffer(property.blob_value, 'base64');
  }

When I debugged at this point I found that property.blob_value was a ByteBufferNB object, not a base64 string. I'm wondering if you meant to base64 encode the original value in function valueToProperty at line 430:

  if (v instanceof Buffer) {
    p.blob_value = v;    //  should it be v.toString('base64') ?
    return p;
  }

But that doesn't make sense since the datastore_v1.proto file defines blob_value as bytes not string at line 111:

  optional string blob_key_value = 16;

I suppose I could simply store my data directly as a base64-encoded string. But I'd prefer the compactness and efficiency of binary storage if possible.

I can reproduce this issue. It appears that the property.blob_value is for some reason the entire ByteBuffer object for the entire response. When I decoded the protobuf response and looked for the geometry value, it's set to:

blob_value:
   { buffer: <Buffer 0a 68 0a 66 0a 2d 0a 16 1a 14 73 7e 73 6f 79 2d 74 72 61 6e 73 64 75 63 65 72 2d 36 39 34 12 13 0a 08 4c 6f 63 61 74 69 6f 6e 10 80 80 80 80 af c8 80 09 ...>,
     offset: 85,
     markedOffset: -1,
     limit: 106,
     littleEndian: true,
     noAssert: false }

This is what propertyToValue also sees. property.blob_value instanceof ByteBuffer is true in this case.
In fact, if you print out the hex value for the entire ByteBuffer, you can see your data is in tact:

property.blob_value.buffer.toString('hex')

Output:

0a680a660a2d0a161a14737e736f792d7472616e7364756365722d36393412130a084c6f636174696f6e108080808080e4d108120f0a02696422098a01063131332d313612240a0867656f6d657472792218920115010100000000000000000059400000000000006940

That 010100000000000000000059400000000000006940 at the end is your location.

But then things get really weird when you decode the original response buffer.

In datastore/lib/request.js#makeAuthorizedRequest, we get a buffer. We then decode it using pbResponse.decode(buffer) and get a nice decoded object we can traverse in javascript.

If we set that decoded to var decoded = pbResponse.decode(buffer); and then traverse to the geometry value decoded.found[0].entity.property[1].value.blob_value we get an ByteBuffer object much like the one at the top of this post. In fact, if we get the buffer property in that object, that is decoded.found[0].entity.property[1].value.blob_value.buffer and compare it to our ORIGINAL buffer we got fresh from the server, they are exactly the same. That is, the decoded buffer contains the original buffer. Recursion is awesome!

I don't know enough about protobufs to fix this issue or even to say this is right or wrong but it's blowing my mind. From first glance, it looks like a bug in the protobuf.js library but a second opinion would be great.

Wow, nice job digging into this. Thanks! My whole experience with protocol buffers extends back to just last Wednesday so consider me an interested bystander for now.

I figured it out! As you can see in the ByteBuffer object above, it has an offset and a limit. So we just need to use that to return the bytes you want! Luckily, ByteBuffer objects have a method to do just that, and viola, a one-line fix :) See 2d76a12