cache-type-k and cache-type-p parameters support #1305

Activity

1. Limour-dev commented on Mar 27, 2024

   

   Limour-dev

   on Mar 27, 2024

   Last edited by Limour-dev

   Contributor

   More actions

   Setting type_v quantization below f16 will result in an error: GGML_ASSERT: llama.cpp\ggml.c:7553: false
2. Limour-dev

   added a commit that references this issue on Mar 28, 2024

   add KV cache quantization options

   ...

   94a9519
3. abetlen

   added a commit that references this issue on Apr 1, 2024

   feat: add support for KV cache quantization options (#1307)

   ...

   f165048
4. abetlen commented on Apr 1, 2024

   

   abetlen

   on Apr 1, 2024

   Owner

   More actions

   @riverzhou thanks to @Limour-dev this should be in the latest v0.2.58 release by setting llama_cpp.Llama(..., type_k=llama_cpp.GGML_TYPE_I8)
5. abetlen

   closed this as completedon Apr 1, 2024
6. xhedit

   added 2 commits that reference this issue on Apr 6, 2024

   feat: add support for KV cache quantization options (abetlen#1307)

   ...

   96fb705

   merge test (#2)

   ...

   bf766bd
7. francomattar

   added a commit that references this issue on Dec 13, 2024

   feat: add support for KV cache quantization options (#1307)

   ...

   c0f7009
8. nayanzin33sergey

   added a commit that references this issue on Dec 18, 2024

   feat: add support for KV cache quantization options (#1307)

   ...

   4734c90
9. jamesdev9

   added a commit that references this issue on Dec 22, 2024

   feat: add support for KV cache quantization options (#1307)

   ...

   5ffac6f
10. NoBrainer242

    added a commit that references this issue on Dec 27, 2024

    feat: add support for KV cache quantization options (#1307)

    ...

    8fcc6ff
11. NoBrainer24

    added a commit that references this issue on Dec 28, 2024

    feat: add support for KV cache quantization options (#1307)

    ...

    691136d