Repository navigation
BigQuery Python Client v0.28.0: num_rows, schema property of a Table returning None and empty list #4373
Description
Hi,
I am using the BigQuery Python client v0.28.0
Using the bigquery client to list all tables in a dataset using method list_dataset_tables, I get an iterator of table references in the dataset. When I try to access the num_rows and schema properties of the Table object it returns None and [].
But when I access the table using client.get_table() I am able to access both the num_rows and schema properties.
I am confused since both the methods return a google.cloud.bigquery.table.Table object but are inconsistent.
for table in bqclient.list_dataset_tables(dataset_ref):
print(table.num_rows)
print(table.schema)
Returns None and []
table = bqclient.get_table(table)
print(table.num_rows)
print(table.schema)
Returns the number of rows and list of schemaFields
Thanks
Activity
tswast commentedon Nov 13, 2017
Thanks for the report.
It appears as though the BigQuery tables.list method only returns a subset of the fields of a table.
I see a few potentially sane fixes:
list_dataset_tablesreturn a list of table references and discard the few extra table fields that the list call returns.I'll send a note to my contacts on the BigQuery backend team about doing (1), but the most likely fix will be (2).
tswast commentedon Nov 13, 2017
I guess there's another option (4) document that only a subset of fields are exposed in the table response objects from the list request.
I dislike that option the most, as it leaves a lot of room for confusion.
tswast commentedon Nov 13, 2017
I got confirmation from the backend team that (1) is infeasible. Properties like the schema and total rows of the table take much longer to fetch in the backend. The list API call includes only properties which are fast for the backend to fetch.
I think I'd like to propose another option (5) Introduce a new type containing this subset of properties. Maybe
PartialTableorTableListItemwhich includes only those properties present in the list API response. That way it will be much clearer from the documentation which properties are included.tseaver commentedon Nov 15, 2017
I had (naively, I guess) assumed that this issue was the exact reason
TableReferenceeven exists.tswast commentedon Nov 15, 2017
No, I unfortunately hadn't thought that listing would have some but not all properties.
TableReferencewas added for the very similar problem of where properties such as destination table on the query job only include the table ID.I'm thinking we really do need to introduce a third type.
tswast commentedon Nov 20, 2017
Just sent #4427 to add
TableListItem. This issue is also present on datasets and will need a similar change for them.