Skip to content

Support for RPC Backend #392

Closed
Closed

Description

@ecyht2

Add support for RPC backend for GGML. This allows for distributed inferencing on multiple GPU over the network. Not so important now as the current models are still kinda small but would be interesting to have this feature.

Activity

  1. Green-Sky commented on Sep 3, 2024

    @Green-Sky
    Contributor

    First we need partial/split offloading. (but yea, rpc would be one of the usable backends)

  2. ManuXD32 commented on Feb 15, 2026

    @ManuXD32

    Hey, are there any plans on adding distributed inference? Just checking

  3. stduhpf commented on Feb 15, 2026

    @stduhpf
    Contributor
  4. leejet commented on Sep 27, 2026

    @leejet
    Owner

    Supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions