Repository navigation
Kinded type behaving differently from non-kinded type in an invariant wrapper #2042
Description
Hi Nikita, long time! Thanks for the interesting library bringing Haskell-like FP patterns to Python.
Bug report
What's wrong
I'm trying to extract the type T from a Kind1 type (say, Foo[T]) that is wrapped in an invariant type, e.g., list[Foo[T]]. However, that's leading to some unexpected behavior.
Repro:
from typing import Any, TypeVar
from returns.primitives import hkt
T = TypeVar("T")
class Foo(hkt.SupportsKind1["Foo[Any]", T]):
pass
# These are just helper functions to extract the type `T`.
def expect_list_foo(foo_list: list[Foo[T]]) -> list[T]:
return []
@hkt.kinded
def expect_list_foo__kinded(foo_list: list[hkt.Kind1[Foo[Any], T]]) -> list[T]:
return []
@hkt.kinded
def expect_tuple_foo__kinded(foo_tuple: tuple[hkt.Kind1[Foo[Any], T], str]) -> list[T]:
return []
def inferred_types(foo_list: list[Foo[int]], foo_tuple: tuple[Foo[int], str]) -> None:
x = expect_list_foo(foo_list)
y = expect_list_foo__kinded(foo_list) # <-- Problem
z = expect_tuple_foo__kinded(foo_tuple)
reveal_type(x)
reveal_type(y)
reveal_type(z)
print(x, y, z)How is that should be
I expected all three calls to work and return list[int].
Actual output:
30: error: Argument 1 to "expect_list_foo__kinded" has incompatible type "list[Foo[int]]"; expected "list[KindN[Foo[Any], int, Any, Any]]" [arg-type]
30: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
30: note: Consider using "Sequence" instead, which is covariant
34: note: Revealed type is "builtins.list[builtins.int]"
35: note: Revealed type is "builtins.list[builtins.int]"
36: note: Revealed type is "builtins.list[builtins.int]"
Found 1 error in 1 file (checked 1 source file)
Seems like list[Foo[int]] is not treated as compatible with list[hkt.Kind1[Foo[Any], T]]. tuple[Foo[int], str] is treated as compatible with tuple[hkt.Kind1[Foo[Any], T], str]. So, it seems like this might be due to the invariance of list.
Is this expected behavior? How can I get this to work?
System information
-
pythonversion: 3.10.8 -
returnsversion: 0.24.0 -
mypyversion: 1.15.0 -
hypothesisversion (if any): -
pytestversion (if any):
Activity
sobolevn commentedon Feb 17, 2025
Hi, @pradeep90! Great to see you 😊
Well, this is a very hard question, actually. I've never expirienced this before.
Several ideas that I have right now:
KindNwith@kindedfully eleminates itself, so in this contextlist[hkt.Kind1[Foo[Any], T]]islist[Foo[T]]@kindedthere might be a different logic?You are very welcome to investigate further. I would love to get a PR with the fix / docs update :)
pradeep90 commentedon Feb 17, 2025
Thanks for getting back. I appreciate it.
I dug into the code and found the root cause :)
We are passing an argument of type
list[Foo[int]]to a parameter of typelist[Kind1[Foo, T]]. The type checker needs to check whether the argument type is compatible. The way it checks depends on the variance of the container:tuple, it just checks the inner types in one direction: IsFoo[int]compatible withKind1[Foo, int]? That is true sinceFoois a descendant ofKind1[Foo, Int].list, it also checks the inner types in the other direction: IsKind1[Foo, int]compatible withFoo[int]. This fails becauseKindN[Foo, int]is not a descendant ofFoo[int]:|In short, the
returnsMypy plugin adds HKT support in one direction not the other. So,Foo[int]andKind1[Foo, int]are not truly substitutable for each other :| The same problem will occur any time we have an invariant or contravariant container (e.g.,Callable) around a kinded type.Some options I can think of to fix this:
ctx.api.errors.error_info_map. However, this would be hacky.KindN[IO, int]is a "subclass" of or in some way compatible withIO[int]. I couldn't find a way to express that using the Mypy plugin API, but others may know a way.If you have any ideas, let me know. If not, I'll probably not be digging further, but I'll leave this root-cause information here for future use.
sobolevn commentedon Feb 17, 2025
Hm, thank you for indepth analysis, maybe we should use two types? Like
KindNwhich is a supertype andSubKindNwhich will be subtype? This is also an open question.pradeep90 commentedon Feb 17, 2025
Hmm... Interesting direction. I'm not clear how that would help solve
Kind1[IO, int] <: IO[int], though. Perhaps you could clarify that.Another option: Dynamically add one overload per HKT that is in scope. This is the old idea from your blog post, but generating the overloads dynamically in the Mypy plugin instead of hardcoding them. This would be a principled solution and would work as expected, but it could explode the number of overloads :) If there is a way to get the argument type in the method call, it should be doable to specialize the method with overloads for just the possible HKTs that are in the argument type.
However,
get_method_signature_hookis called before argument types are available, andget_method_hookis called after the error has been added. So, I don't know of a clean solution here.sobolevn commentedon Feb 19, 2025
This is cool! It can work indeed.