Skip to content

feat: show engine generation speed #271

Closed

Description

@outofcoffee

We should show the tokens per second generation speed from engines.

If this is not exposed as a metric, our existing log handling for engines like llama.cpp should enable us to parse it from the structured logs, e.g.

86.15.860.290 I slot print_timing: id  0 | task 326 | n_gen =   9438, tg =   3.87 t/s, tg_3s =   5.29 t/s
86.19.263.970 I slot print_timing: id  0 | task 326 | n_gen =   9456, tg =   3.88 t/s, tg_3s =   5.29 t/s
86.22.573.260 I slot print_timing: id  0 | task 326 | n_gen =   9474, tg =   3.88 t/s, tg_3s =   5.44 t/s

The tg figure is the value we want.

If supported by the engine, we should show it as one of the metrics, with a sparkline historical graph, e.g.

  Speed: 12.34 t/s  ▂▃▂▂▂▂▂▁▂▂▁▂▂▇▇▃▂▂▂▂▂▃▂▃▂

The max value for the sparkline should be the high watermark of the metric value relative to the current values shown in the graph.

The sparkline should use the default colour (e.g. grey) for all values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions