Datastore error on great amount of data

Hi,
i'm facing a real strange issue.
I've a datastore save request that goes well when the entity is small,
but when we try to save a great amount of data we receive this error

{ handle: 2,
type: 'error',
 className: 'Error',
 constructorFunction: { ref: 8 },
 protoObject: { ref: 9 },
 prototypeObject: { ref: 1 },
 properties: 
  [ { name: 'stack', attributes: 2, propertyType: 3, ref: 1 },
    { name: 'message', attributes: 2, propertyType: 0, ref: 10 },
    { name: 'code', propertyType: 0, ref: 11 },
    { name: 'metadata', propertyType: 0, ref: 12 } ],
 text: 'Error: Bad Request' }

that breaks the node process with this stacktrace:

  < /Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:300
  <   throw new Error('Unsupported field value, ' + value + ', was provided.');
  <   ^
  < Error: Unsupported field value, undefined, was provided.
  <     at Object.encodeValue (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:300:9)
  <     at /Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:652:24
  <     at Array.map (native)
  <     at Object.queryToQueryProto (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:646:33)
  <     at makeRequest (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/request.js:495:21)
  <     at Object.makeRequest (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/common/util.js:579:21)
  <     at DestroyableTransform.<anonymous> (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/request.js:485:13)
  <     at DestroyableTransform.g (events.js:286:16)
  <     at emitOne (events.js:96:13)
  <     at DestroyableTransform.emit (events.js:188:7)

the error shows up when i try to save an entity with two string field, not indexed of 1MB of size
ie

{
  data: [0.9MB string not indexed]
  data2: [0.9MB string not indexex]
  [other strings indexed of no more than 40bit]
}

We chunck our data on string of less than 1MB to respect the maximum string length property on datastore.
I'm pretty sure that the issue is about the amount of data because saving only one chunk no error was raised.

There's an upper limit on the total entity size or there is some error?

Thanks in advance

Sorry you're having this issue :( Is it possible to put a small sample together, or otherwise the actual gcloud-node code you're using?

hi,
here is a really small test file to verify the behaviour.

        'use strict';

        const gcloud = require('gcloud')({
            projectId: [project ID],
            keyFilename: [keyfilePath]
        });
        const datastore = gcloud.datastore();
        const _ = require('lodash');

        var str = new Array(500000).join("a");

        debugger;

        function toDatastore(obj, indexedFields) {
            indexedFields = indexedFields || [];
            const results = [];
            Object.keys(obj).forEach((k) => {
                if (obj[k] === undefined) {
                    return;
                }
                results.push({
                    name: k,
                    value: obj[k],
                    excludeFromIndexes: indexedFields.indexOf(k) < 0
                });
            });
            return results;
        }

        function create(kind, data, indexedFields, cb) {
            const key = datastore.key([kind]);
            debugger;
            const entity = {
                key: key,
                data: toDatastore(data, indexedFields)
            };

            datastore.save(
                entity,
                (error) => {
                    debugger;
                    if (error) {
                        return cb ? cb(error) : error;
                    }

                    return cb ? cb(null) : true;
                }
            );
        };

        create("testKind", {
            data0: str,
            data1: str,
            data3: str,
            key: '250277808676421',
            key1: 'b4wbj0b5wy04',
            key2: 'sdjhfkahsdjkfhjkas',
            key3: "setstts",
            key4: "sdfasdfa"
        }, ["key", "key1", "key2", "key3", "key4"], (err, result) => {
            debugger;
        });

In this case the error was raised whth string of 500000 chars.
Thanks

@pcostell

Thanks for putting that together. @pcostell any insight on this one? Thanks!

@stephenplusplus

It looks like you've exceeded the maximum size for an entity: https://cloud.google.com/datastore/docs/concepts/limits

Not sure why you don't get a nicer error though. @stephenplusplus can see what the request/response from Datastore is?

@pcostell

@pcostell would that relate to #1378? I think as of now, we don't have a way to pull out any detailed error messaging.

Ah yes that's likely the issue -- although I don't understand why a Bad request turns into the given node stack trace.

Yeah, I don't understand that either. That didn't happen for me from the sample code provided, maybe it was from some temporarily malformed code, @mik115?

No temporary malformed and yes I reached the upper entity limit size of 1MB.
I've already fixed the problem chunking my data and saving them in separate entities.

Thanks anyway for the really quick responses

Great :) We're working on providing more helpful error messages in #1378. Feel free to re-open this if you think there's more we can do.

It breaks node process for me as well! With same error, on same line:

Error: Unsupported field value, undefined, was provided.
  <     at Object.encodeValue

Ideas how to fix it?

@kirillgroshkov

@kirillgroshkov could you share reproduction code?

chore(deps): update dependency python to v3.10.4 (#1386) (#815)

chore: update license headers (#1386)

chore(deps): update dependency python to v3.10.4 (#1386) (#1602)

chore(main): release 10.3.3 (#1386)

fix(build): move branch references to main (#1174) (#1386)

feat: Trusted Private Cloud support, use the universeDomain parameter (…

fix(build): move branch references to main (#1174) (#1386)

feat: Trusted Private Cloud support, use the universeDomain parameter (…

Navigation Menu

Repository navigation

Datastore error on great amount of data #1386

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Datastore error on great amount of data #1386

Description

Activity

stephenplusplus commented on Jun 15, 2016

michele-pini commented on Jun 16, 2016

stephenplusplus commented on Jun 16, 2016

pcostell commented on Jun 16, 2016

stephenplusplus commented on Jun 16, 2016

pcostell commented on Jun 16, 2016

stephenplusplus commented on Jun 16, 2016

michele-pini commented on Jun 17, 2016

stephenplusplus commented on Jun 20, 2016

kirillgroshkov commented on Nov 13, 2017

stephenplusplus commented on Nov 14, 2017

1 remaining item

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions