Skip to content

[Proposal] Get the hot spot of the doris cluster. #3517

Closed

Description

@chaoyli

Nowadays, We have no insight about the hotspot the cluster.
During the time, we don't know the scanner rows of one table during time.
Now we should collect the scanner rows of the table and scanner bytes of the table for query.
Using this information, we can got the insight about the hot spot of cluster.

Activity

  1. chaoyli commented on May 8, 2020

    @chaoyli
    ContributorAuthor

    The metric is like this

    [
       {
          "tags":{
             "table_id":10010,
             "metric":"scanner_rows"
          },
          "value":1000,
          "unit":"number"
       },
       {
          "tags":{
             "table_id":10010,
             "metric":"scanner_bytes"
          },
          "value":1024,
          "unit":"bytes"
       }
    ]
    

    Supposing there are 10000 tablets in one BE node and collecting metrics at every 15 seconds.
    Every time we will gather 10000 * 200 Bytes = 2MB, one hundred nodes will collect 200MB, this will be consume huge bandwidth. And also user will take care of table granularity than tablet. So we will collect table metrics instead of tablet metrics.
    For metric in one BE node, I will use a map to store the information.

    std::unordered_map<table_id, metric> table_metrics
    

    The incremental metric value should be calculated at every collecting interval.
    Like this

    2020-05-08 14:00:00  10000  1024(scanner_rows)
    2020-05-08 14:00:15   10000  2048(scanner_rows)
    The incremental value at this cycle is 1024.
    
  2. chaoyli commented on May 8, 2020

    @chaoyli
    ContributorAuthor

    There also exists another function using log.
    If every query collecting the tablet scanner rows/bytes, and write it to log.
    We can access the log to get the message, it needs log analysis which is not integrated in Doris.
    So I will use the above method to solve this problem.

  3. imay commented on May 8, 2020

    @imay
    Contributor

    This is a good job.
    However I think something maybe discussed clearly before writing code.

    1. Who will update this metrics?
    2. How will updater get the table id?
    3. How will the final metric be summary?
  4. chaoyli commented on May 9, 2020

    @chaoyli
    ContributorAuthor

    This is a good job.
    However I think something maybe discussed clearly before writing code.

    1. Who will update this metrics?
    2. How will updater get the table id?
    3. How will the final metric be summary?
    1. Scanner Thread will update the metrics, it is the same as the metric like the query profile.
    2. Nowadays, tablet meta has save the table id in the metadata, so new table will know the table_id.
    3. If metric has been collected by agent, you can store to another database, or can be summed by a central node. There are ok both.
  5. morningman commented on May 9, 2020

    @morningman
    Contributor

    I think we could refer to
    https://pingcap.com/blog-cn/observability-of-distributed-system/
    and
    https://cloud.google.com/bigtable/docs/keyvis-overview

    to learn something

  6. added
    area/profileIssues or PRs related to the profile
    kind/featureCategorizes issue or PR as related to a new feature.
    on May 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/profileIssues or PRs related to the profilekind/featureCategorizes issue or PR as related to a new feature.proposalCategorizes an issue is a proposal

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions