Skip to content

[Enhancement] implement streamload via http stream #36773

Closed

Description

@dataroaring

Search before asking

  • I had searched in the issues and found no similar issues.

Description

Currently we can put data to doris via http using streamload or http stream. httpstream transforms data via sql in a header while streamload via special header. Sql is more easier to use.

In future, we would suggest http stream for users, and we also need keep only one in internal implementation. So we can add a config option to enable implement streamload via ways of httpstream.

We can transform streamload headers to a sql internally and implement streamload in ways of httstream.

Solution

In the streamload action of be, collector headers and transfrom it to sql.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. Tech-Circle-48 commented on Jun 24, 2024

    @Tech-Circle-48
    Contributor

    please assign to me

  2. StarryVerse commented on Oct 21, 2024

    @StarryVerse
    Contributor

    Has this issue been resolved? Please assign it to me

  3. Yufei-YAO commented on Nov 11, 2024

    @Yufei-YAO

    I'm working on this issue, but I got some problem in parsing stream load headers with "columns" set in header.

    For example, for a table with columns (c1,c2,c3) and an stream load request has 'columns' = "c1, c2, c3,temp" in request header.

    If I convert this into the form[ insert into xxx.xx select * from http_stream("columns"="c1, c2, c3,temp") ], I get an insert into cols should be corresponding to the query output error.

    It seems the process steps in httpstream requires the input cols equal to table columns. However, the streamLoad opeation support the input cols size more than table columns.

    I wonder what should I do to solve this problem. Thanks a lot.

  4. github-actions commented on Sep 12, 2025

    @github-actions
    Contributor

    We're closing this issue because it hasn't been updated in a while.
    If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions