Skip to content

GH-39013: [Go][Integration] Support cABI import/export of StringView - #39019

Merged
zeroshade merged 28 commits into
apache:mainfrom
bkietz:39013-go-utf8view-c-abi
Dec 19, 2023
Merged

zeroshade merged 28 commits into
apache:mainfrom
bkietz:39013-go-utf8view-c-abi

Conversation

@bkietz

@bkietz bkietz commented Dec 1, 2023 •

Copy link
Copy Markdown
Member

Rationale for this change

The Go implementation should support import/export of the new data types. This will enable integration testing between the C++ and Go implementations.

What changes are included in this PR?

Added import/export for the new data types and arrays of data of those types.

Are these changes tested?

Yes, they will be covered by the integration tests and existing Go unit tests.

Are there any user-facing changes?

This is a user facing change

@bkietz
bkietz force-pushed the 39013-go-utf8view-c-abi branch from e050c73 to 2505971 Compare December 12, 2023 17:22
@bkietz
bkietz requested a review from felipecrv December 13, 2023 17:23
return ListViewColumn

def _get_type(self):
return OrderedDict([

ghost Dec 13, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FTR, we don't need OrderedDict anymore as regular dicts are now ordered by default.

"""
def case_wrapper(test_case):
if serial:
return case_runner(test_case)

ghost Dec 13, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this addition deliberate?

ghost Dec 13, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes: printer.cork() was failing to dump output when the test segfaulted. In any case, corking the printer adds no value when we're running in serial.

ghost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm adding more comments.

Comment thread go/arrow/array/list.go Outdated
// input.Len() > 0 && input.NullN() != input.Len()
func maxListViewOffset32(input arrow.ArrayData) int {
// input.DataType() is ListViewType if Offset=int32 or LargeListViewType if Offset=int64
// input.Len() > 0

ghost Dec 13, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the NullN() != Len() pre-condition?

ghost Dec 14, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RangeOfUsedValues was causing an error in IPC writing where it was being used to determine how much of the values array to write. This could drop values referenced by views under null bits, which later produced negative offsets. I was trying to rewrite this function to also consider views under null bits (less efficient for concat but still usable for IPC writing). I have reverted the change to RangeOfUsedValues and IPC writing no longer tries to slice the values array. This is less optimal for IPC writing but we can add optimizations back later, for now I'd like to get the integration test passing.

ghost Dec 15, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is why it's important to keep the documentation about the pre-condition ... && input.NullN() != input.Len(). The concept of "minumum offset" is undefined when there are 0 offsets to look at (an empty set can't have a minimum) -- the caller can betetr decide what's appropriate to do when all values are null.

Comment thread go/arrow/array/list.go Resolved

ghost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot of noise in this PR. Can you split the refactoring you did from the cABI changes?

Comment thread go/arrow/cdata/cdata.go Resolved
Comment thread go/arrow/cdata/cdata_exports.go Resolved
Comment thread go/arrow/compute/fieldref.go Resolved
Comment thread go/arrow/compute/fieldref.go Resolved
Comment thread go/arrow/compute/fieldref.go Resolved
Comment thread go/arrow/compute/fieldref.go Resolved
Comment thread go/arrow/internal/flatbuf/Binary.go Resolved
Comment thread go/arrow/internal/debug/log_on.go Resolved
Comment thread go/arrow/array/list.go Resolved
@zeroshade

ghost commented Dec 15, 2023

Copy link
Copy Markdown
Member

@pitrou any further issues here / remaining things to be addressed?

@zeroshade

ghost commented Dec 18, 2023

Copy link
Copy Markdown
Member

if there's no objections or no other changes requested I'll merge this at the end of the day

@zeroshade
zeroshade merged commit 3e182f2 into apache:main Dec 19, 2023
@conbench-apache-arrow

ghost commented Dec 20, 2023

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 3e182f2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go][Integration] Add support for C abi Utf8View to the go implementation

4 participants