We should show the tokens per second generation speed from engines. If this is not exposed as a metric, our existing log handling for engines like llama.cpp should enable us to parse it from the structured logs, e.g. ``` 86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s 86.19.263.970 I slot print_timing: id 0 | task 326 | n_gen = 9456, tg = 3.88 t/s, tg_3s = 5.29 t/s 86.22.573.260 I slot print_timing: id 0 | task 326 | n_gen = 9474, tg = 3.88 t/s, tg_3s = 5.44 t/s ``` The `tg` figure is the value we want. If supported by the engine, we should show it as one of the metrics, with a sparkline historical graph, e.g. ``` Speed: 12.34 t/s ▂▃▂▂▂▂▂▁▂▂▁▂▂▇▇▃▂▂▂▂▂▃▂▃▂ ``` The max value for the sparkline should be the high watermark of the metric value relative to the current values shown in the graph. The sparkline should use the default colour (e.g. grey) for all values.
Activity