Friday, 7 January 2022

Mypy 0.931 Released

We’ve just uploaded mypy 0.931 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes bug fixes and fixes to regressions only. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Fixes to Regressions

  • Fix mypyc binary wheel (Michael R. Crusoe, PR 11862)
  • Revert to treating multi-line strings as a single value for exclude in TOML files (Matt Bogosian, PR 11828)
  • Revert to treating exclude in .ini as a single value (Matt Bogosian, PR 11881)
  • Fix __slots__ regression (Nikita Sobolev, PR 11824)
  • Fix regression with __module__ and similar non-final Enum attributes (Nikita Sobolev, PR 11823)
  • Fix regression when a contextmanager yields a generic function (Anthony Sottile, PR 11870)
  • Fix __reduce__ regression (Shantanu, PR 11866)
  • Fix NoReturn type aliases (Nikita Sobolev, PR 11912)
  • Fix regression in None handling with --no-strict-optional (Nikita Sobolev, PR 11717)
  • Fix crash related to generics (Nikita Sobolev, PR 11924)
  • Restore removed builtin_type() plugin API method (Jukka Lehtosalo, PR 11932)

Other Fixes

  • Always allow the use of type[T] in stubs (Nikita Sobolev, PR 11863)
  • Fix PEP 585 type aliases in stubs (Shantanu, PR 11918)
  • Fix bug with literal types in stubtest (Nikita Sobolev, PR 11931)

We’d like to thank our employer, Dropbox, for funding the mypy core team.

Wednesday, 22 December 2021

Mypy 0.930 Released

We’ve just uploaded mypy 0.930 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features and bug fixes. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

New Feature: Explicit Type Aliases (PEP 613)

You can now explicitly mark an assignment as a type alias, by using typing.TypeAlias:

    # On Python 3.9 and earlier, use "from typing_extensions ..."
    from typing import TypeAlias

    A: TypeAlias = int | str

For more context, see the docs or PEP 613.

This was contributed by Shantanu in PR 11305.

New Feature: NotRequired in TypedDicts

You can now define individual TypedDict items as non-required by using typing_extensions.NotRequired. Previously this could only be configured at TypedDict level (by using total=). Example:

    from typing import TypedDict
    from typing_extensions import NotRequired

    class D(TypedDict):
        x: str
        y: NotRequired[int]

    a: D = {"x": "value"}  # OK
    b: D = {"x": "value", y: 0}  # OK
    c: D = {}  # Error: "x" is required

You can also use typing_extensions.Required to mark an item as required in a non-total TypedDict.

For more information, see the draft PEP 655.

This was contributed by David Foster in PR 10370.

New Feature: ParamSpec (Experimental)

Mypy now partially support ParamSpec, which was introduced in Python 3.10. This makes it possible to define type variables that range over the parameter specification of a callable type. In particular, it allows giving precise types for some decorators that change the return type of the decorated function.

Here’s an example copied (and slightly modified) from PEP 612:

    from typing import Awaitable, Callable, ParamSpec, TypeVar

    P = ParamSpec("P")
    R = TypeVar("R")

    def add_logging(f: Callable[P, R]) -> Callable[P, Awaitable[R]]:
        async def inner(*args: P.args, **kwargs: P.kwargs) -> R:
            await log_to_database()
            return f(*args, **kwargs)
        return inner

    @add_logging
    def takes_int_str(x: int, y: str) -> int:
        return x + 7

    await takes_int_str(1, "A")  # Accepted
    await takes_int_str("B", 2)  # Rejected

See Python documentation and PEP 612 for additional details. Note that typing.Concatenate and some other use cases are still not supported.

Support for semantic analysis of ParamSpec was contributed by Shantanu. Jukka Lehtosalo implemented type checking of ParamSpec types. Additional fixes were contributed by Nikita Sobolev.

Miscellaneous New Features

  • Define a GitHub Action that makes it easier to run mypy in GitHub CI workflows (Elvis Pranskevichus, PR 11320)
  • Allow mypy to run self check on PyPy 3.8 beta (Ethan Smith, PR 11350)
  • Check that enum values are unique (Nikita Sobolev, PR 11267)
  • Make enum classess with values implicitly final (Nikita Sobolev, PR 11247)
  • Make enum members implicitly final (Nikita Sobolev, PR 10852)
  • Allow NamedTuple to be used as a type that accepts all named tuples, as an experimental extension (Nikita Sobolev, PR 11162)
  • Allow booleans to be narrowed to literal types (Ethan Leba, PR 10389)
  • Add slots=True support for @dataclass (Nikita Sobolev, PR 11483)
  • Add slots=True support for @attr.s (Nikita Sobolev, PR 11489)
  • Support the typing_extensions.OrderedDict alias (Nikita Sobolev, PR 11533)

Usability and Documentation Improvements

  • Add better NamedTuple error messages (Nikita Sobolev, PR 11127)
  • Show all overloads in error message (Akuli, PR 9177)
  • Fix error message for dataclasses.field with positional argument (Hiroshi Ogawa, PR 11180)
  • Fix error message for f-string and str-bytes-safe (Nikita Sobolev, PR 11139)
  • Add documentation for type: ignore (Tushar Sadhwani, PR 11358)
  • Support --exclude more than once on command line (Nipunn Koorapati, PR 11329)
  • Use list[int] instead of List[int] in documentation (PR 11377, PR 11450) (Nick Crews)
  • Remove builtins. from error messages (97littleleaf11, PR 11522)
  • Document explicit type aliases (Nikita Sobolev, PR 11800)
  • Improve documentation of annotating generator functions (Alex Waygood, PR 11623)
  • Improve error message for nested TypedDict (97littleleaf11, PR 11658)
  • Exit gracefully on KeyboardInterrupt (Marc Mueller, PR 10725)

Optimizations

This release includes some optimizations that make mypy a bit faster (in addition to mypyc performance improvements discussed below):

  • Enable --fast-exit by default to speed up mypy (Jukka Lehtosalo, PR 11541)
  • Only read the stdlib versions dictionary once per run (Jukka Lehtosalo, PR 11542)
  • Optimize commonly used function (Jukka Lehtosalo, PR 11543)
  • Minor optimization/cleanup (Jukka Lehtosalo, PR 11544)
  • Reduce the number of executed local imports (Jukka Lehtosalo, PR 11545)

Mypyc Fixes and Improvements

  • Add missing type coercions (Jared Hance, PR 11176)
  • Support --allow-redefinition (Jared Hance, PR 11175)
  • Constant fold integer operations and string concatenation (Jukka Lehtosalo, PR 11194)
  • Use optimized implementation for builtins.sum (Sara Sinback and 97littleleaf11, PR 10268)
  • Move mypyc to console_scripts to fix running on Windows 10 (谭九鼎, PR 11494)
  • Make min(x, y) faster(Chetan Khanna and 97littleleaf11, PR 10265)
  • Make max(x, y) faster (97littleleaf11, PR 11530)
  • Speed up reference counting operations by inlining them in commonly executed blocks (Jukka Lehtosalo, PR 11540)
  • Add support for attrs classes (Chad Dombrova, PR 11328)
  • Port mypyc to Python 3.11 (Victor Stinner, PR 11652)
  • Reduce the amount of debug information included in compiled extension modules (Nehal J Wani, PR 11526)

Other Notable Fixes and Improvements

  • Fix semantic analysis of assignment expressions (Shantanu, PR 11153)
  • Fixes mypy crash on protocol with contravariant variable (Nikita Sobolev, PR 11135)
  • Warn about unused ignores when not all specified error codes are used (Hiroshi Ogawa, PR 11178)
  • Improve type narrowing for walrus operator in conditional statements (Shantanu, PR 11202)
  • Fix literal type compatibility checking special case (Nikita Sobolev, PR 11236)
  • Rename API method builtin_type to named_type (97littleleaf11, PR 11224)
  • Fix narrowing information not propagated in assignment and boolean expressions (Ran Benita, PR 11207)
  • Fix narrowing of a nested union of TypedDicts (Ran Benita, PR 11204)
  • Fix case mismatching modules during namespace package search (Nipunn Koorapati, PR 11261)
  • Disallow invalid identifiers from getting implicit bazel __init__.py (Nipunn Koorapati, PR 11268)
  • Don’t type check lambdas nested in unchecked functions (Nikita Sobolev, PR 11213)
  • Update None.__bool__ to return Literal[False] (Nikita Sobolev, PR 11290)
  • Relax type checking of % formatting when the right operand is an iterable (97littleleaf11, PR 11319)
  • Fix crash with namespace packages when generating html reports (Nikita Sobolev, PR 11338)
  • Allow slice syntax in Annotated types (Zac Hatfield-Dodds, PR 11345)
  • Show warning if self / cls argument is missing (Nikita Sobolev, PR 11317)
  • Improve subtype checking of TypeGuard types (Nikita Sobolev, PR 11314)
  • Fix type narrowing for overlaping runtime types (Nikita Sobolev, PR 11273)
  • Do not use TypeGuard context for lambda (Nikita Sobolev, PR 11417)
  • Check reachability in module bodies (Nikita Sobolev, PR 11361)
  • Check that __new__ in a metaclass returns a subtype of type (Nikita Sobolev, PR 11420)
  • Improve support of contextlib.asynccontextmanager (Barnaby Shearer, PR 11352)
  • Discard deprecated builtin_type (97littleleaf11, PR 11343)
  • Don't look in user site packages when using a virtual env (Dimitri Merejkowsky, PR 11444)
  • Update stubgen to preserve string literals in annotations (Imad Eddine Rezgui, PR 11292)
  • Fix issue with exporting names in the presence of __getattr__ (Shantanu, PR 11411)
  • Reject duplicate bases when defining TypedDict (97littleleaf11, PR 11485)
  • Fix type inference for index expression with a bounded TypeVar (Ilya Labun, PR 11434)
  • Fix get_dynamic_class_hook in some scenarios (Jade Lin, PR 10904)
  • Check print >> properly (Nikita Sobolev, PR 11576)
  • Correctly handle cls in protocol classmethod (Ilia Novoselov, PR 11119)
  • Use current Python version, rather than hardcoding 3.6, in stubgen (Shantanu, PR 10907)
  • Don't use ModuleType.__getattr__ if we know module symbols (Jukka Lehtosalo, PR 11597)
  • Fix strict equality when using the latest typeshed (Jukka Lehtosalo, PR 11599)
  • Make sure ClassVar does not contain type variables (Nikita Sobolev, PR 11585)
  • Allow overriding attributes with methods (Nikita Sobolev, PR 11561)
  • Don’t generate an error for PEP 593 Annotated with a string literal second argument (Seth Yastrov, PR 10777)
  • Install types to the correct environment (Konstantin Weddige, PR 11457)
  • Support decorators in additional contexts (Nikita Sobolev, PR 11150)
  • Fix crash involving unreachable binary operations (Shantanu, PR 11680)
  • Fix None assignments with mypy daemon cache (Christian Bundy, PR 11574)
  • Handle OSError when accessing the mtime of a mypy cache file (Zac Hatfield-Dodds, PR 11718)
  • Special case some special Enum properties to be non-final (Nikita Sobolev, PR 11713)
  • Fix crash involving undefined cyclic import * (Shantanu, PR 11681)
  • Remove incorrect assumption about file system case sensitivity (Shantanu, PR 11708)
  • Fix compatibility of bool and Literal[True, False] (Nikita Sobolev, PR 11709)
  • Allow subclassing enums with annotations and no values (Nikita Sobolev, PR 11579)

Typeshed Updates

Typeshed is now modular and distributed as separate PyPI packages for everything except the standard library. Please see git log for full list of typeshed changes.

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • 97littleleaf11
  • Abdul Rehaman
  • Akuli
  • Alex Waygood
  • Barnaby Shearer
  • Chad Dombrova
  • Chetan Khanna
  • Christian Bundy
  • Daniel Mendelsohn
  • David Foster
  • DetachHead
  • Dimitri Merejkowsky
  • Elisha Hollander
  • Elvis Pranskevichus
  • Ethan Leba
  • Ethan Smith
  • Hiroshi Ogawa
  • Ilia Novoselov
  • Ilya Labun
  • Imad Eddine Rezgui
  • Infinil
  • Jared Hance
  • Ken Jin
  • Konstantin Weddige
  • KotlinIsland
  • Louis Sautier
  • luzpaz
  • Marc Mueller
  • Nehal J Wani
  • Neil Girdhar
  • Nick Crews
  • Nikita Sobolev
  • Nipunn Koorapati
  • Noé Rubinstein
  • parthaxntn
  • Peter Law
  • pranavrajpal
  • Ran Benita
  • Sara Sinback
  • Sebastian Rittau
  • Seth Yastrov
  • Shantanu
  • Taneli Hukkinen
  • Tushar Sadhwani
  • Victor Stinner
  • William Woodruff
  • Zac Hatfield-Dodds
  • 林玮 (Jade Lin)
  • 谭九鼎

We’d also like to thank our employer, Dropbox, for funding the mypy core team.

Tuesday, 21 December 2021

Mypy 0.921 Released

We’ve just uploaded mypy 0.921 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes bug fixes only. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Bug Fixes Included in This Release

  • Fix regression in PathLike (Shantanu, PR 11785)
  • Allow calling a function with name _ (Jukka Lehtosalo, PR 11810)
  • Fix signature in curses and corresponding dmypy bug (Shantanu, PR 11785)

We’d like to thank our employer, Dropbox, for funding the mypy core team.

Wednesday, 15 December 2021

Mypy 0.920 Released

We’ve just uploaded mypy 0.920 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features and bug fixes. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Making a Variable Optional in an Else Block

Now mypy allows more general conditionally defined variables with optional types. Previously mypy allowed this only if a None assignment happened first. Now this is also supported:

    def func() -> int: ...
    
    if test():
        x = func()
    else:
        x = None  # Ok, 'x' has type int | None

Implemented by Michael J. Sullivan in PR 11002.

Type Checking __slots__ Assignment

For classes that define the special __slots__ attribute, mypy will now report an error on assigning to attributes that are not found in the slots definitions (this matches runtime semantics). Example:

    class SuperClass:
        __slots__ = ("a",)
    
    class SubClass(SuperClass):
        __slots__ = ("b",)
        def __init__(self) -> None:
            self.a = 1  # OK
            self.b = 2  # OK
            self.c = 3  # Error, previously mypy allowed this.

Note that this feature does not work for dynamically computed slots. Implemented by Nikita Sobolev in PR 10864.

Partial Python 3.10 Support

We now ship binary wheels for Python 3.10 and mypy supports the new Python 3.10 union type syntax:

  • X | Y is the new syntax for Union[X, Y]
  • X | None is the new syntax for Optional[X].

Some new features, including the match statement, ParamSpec and TypeAlias are not supported yet.

Python 3.5 Is No Longer Supported

Python 3.5 reached its end of life more than a year ago. Its support was deprecated in mypy 0.910 and it is no longer supported in mypy 0.920.

Efficient String and Bytes Formatting in Mypyc

Most string and bytes formatting methods now use fast C-level logic when compiled with mypyc. This includes % formatting, the format() method, and f-strings. This was implemented by 97littleleaf11 as part of a Google Summer of Code project.

Work Towards singledispatch Support

Mypy and mypyc now have experimental support for functools.singledispatch. Here is a simple example:

    from functools import singledispatch
    
    class Expression: pass
    
    class Literal(Expression):
        def __init__(self, value: float) -> None:
            self.value = value
    
    class Add(Expression):
        def __init__(self, left: Expression, right: Expression) -> None:
            self.left = left
            self.right = right
    
    @singledispatch
    def evaluate(e: Expression) -> float:
        assert False, f"unsupported expression: {e}"
    
    @evaluate.register
    def _(e: Literal) -> float:
        return e.value
    
    @evaluate.register
    def _(e: Add) -> float:
        return evaluate(e.left) + evaluate(e.right)
    
    print(evaluate(Literal(2)))
    print(evaluate(Add(Literal(3), Literal(5))))

The signature of the initial function variant handles the generic case and describes the external signature of the singledispatch function.

This was implemented Pranav Rajpal as part of a Google Summer of Code project.

Stubgen Improvements

  • Add support for yield statements in stubgen (Sebastian Rittau, PR 10745)
  • Handle commas in namedtuple field definition in stubgen (Vanessa Ung, PR 10828)

Other Notable Fixes and Improvements

  • Allow plugin signature hooks to return FunctionLike (to support overloads) (pranavrajpal, PR 10717)
  • Narrow type with type variable when the upper bound is a subtype of current type (ethframe, PR 10658)
  • Make --cache-fine-grained imply --local-partial-types (Michael J. Sullivan, PR 10737)
  • Support new union syntax in stubs independent of target Python version and in runtime context (PR 10770, PR 10771) (Jukka Lehtosalo)
  • Support new union type syntax with isinstance() (Jukka Lehtosalo, PR 10775)
  • Better message if method is incompatible with base class (Anmol Takiar, PR 10572)
  • Allow redefinition of underscore functions (named '_') (pranavrajpal, PR 10811)
  • Properly track positional-only arguments for unannotated functions (Michael J. Sullivan, PR 10802)
  • Fixes to type checking %c string and bytes interpolation (97littleleaf11, PR 10869)
  • Use better error message for %c interpolation (97littleleaf11, PR 10875)
  • Skip overlapping overload checks in ignored files (Shantanu, PR 10922)
  • Fix ad hoc instance intersection logic (Christoph Tyralla, PR 9909)
  • Support tuple multiplication with literal integers (hatal175, PR 10361)
  • Fix recursion issue with nested instances and unions (Peilonrayz, PR 9663)
  • Fix caching behavior of PEP561-installed namespace packages (Michael J. Sullivan, PR 10937)
  • Add __dataclass_fields__ and __attrs_attrs__ to dataclasses (Timofey Kukushkin, PR 8578)
  • Fix argument checking on empty dict with double stars (Momoko Hattori, PR 9629)
  • Fix some type-guard-related crashes (Shantanu, PR 11061)
  • Reject raise Err if Err can’t be called without arguments (Nikita Sobolev, PR 11125)
  • Fix TypedDict crash with function definition (Nikita Sobolev, PR 11126)
  • Report attribute access errors for type variable bound to a union (Christoph Tyralla, PR 11140)
  • Fix crash on dataclasses.field(unpack) (Nikita Sobolev, PR 11137)
  • Fix crash related to ParamSpec in mypy daemon (Jukka Lehtosalo, PR 11567)
  • Properly type check *CustomType and CustomType function arguments (Nikita Sobolev, PR 11151)
  • Fix crash with overload and callable object decorators (Shantanu, PR 11630)
  • Fix crash involving explicit reexport, import cycle, and a wildcard (Shantanu, PR 11632)

Mypyc Features, Fixes, and Performace Improvements

  • Fix class-based named tuples (Jukka Lehtosalo, PR 10746)
  • Add a special case for len() of a string value (97littleleaf11, PR 10710)
  • Reject instance attribute access through class object (Jukka Lehtosalo, PR 10798)
  • Speed up the construction of list objects (97littleleaf11, PR 10807)
  • Add bytes primitive type (97littleleaf11, PR 10881)
  • Add bytearray support (97littleleaf11, PR 10891)
  • Optimize construction via list() and dict() (Richard Si, PR 10918)
  • Speed up bytes join() method (jhance, PR 10929)
  • Speed up len(bytes) (97littleleaf11, PR 10936)
  • Speed up bytes indexing and slicing (PR 10966, PR 10950) (97littleleaf11)
  • Speed up bytes.decode() and str.encode() (PR 10974, PR 10951) (97littleleaf11)
  • Speed up bytes equality checks (jhance, PR 10928)

Documentation Improvements

  • Add docs about exhaustive literal and enum checks (Nikita Sobolev, PR 10860)
  • Add ‘or’ regular expression example to --exclude documentation (Niklas Gustafsson, PR 10903)
  • Document per-module follow_imports more explicitly (Shantanu, PR 10845)
  • Add docs for the TypeGuard type (Nikita Sobolev, PR 10758)
  • Update type narrowing section in common issues (Nikita Sobolev, PR 11014)
  • Document how --no-implicit-reexport handles from X import Y as Z (Max Marrone, PR 11083)
  • Document strict config file option (Anders Kaseorg, PR 11132)
  • Add better type narrowing documentation (Nikita Sobolev, PR 11088)
  • Add documentation about := and type guards (Nikita Sobolev, PR 11161)

Typeshed Updates

Typeshed is now modular and distributed as separate PyPI packages for everything except the standard library. Please see git log for full list of typeshed changes.

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • 97littleleaf11
  • Abdul Rehaman
  • Anders Kaseorg
  • Anmol Takiar
  • Anton Agestam
  • Chad Dombrova
  • Christoph Tyralla
  • Daniel Hahler
  • Dong-hee Na
  • Elisha Hollander
  • Ethan Smith
  • ethframe
  • geo7
  • hatal175
  • Hugues
  • Ilya Konstantinov
  • Jared Hance
  • Kevin Wojniak
  • KotlinIsland
  • Louis Sautier
  • Max Marrone
  • Mehdi ABAAKOUK
  • Michael J. Sullivan
  • Michael R. Crusoe
  • Momoko Hattori
  • Nikita Sobolev
  • Niklas Gustafsson
  • Nipunn Koorapati
  • Peilonrayz
  • pranavrajpal
  • Richard Si
  • ryan-gunderson
  • Sebastian Rittau
  • sed-i
  • Shantanu
  • Taneli Hukkinen
  • Tim Gallant
  • Timofey Kukushkin
  • Vanessa Ung
  • Zac Hatfield-Dodds

We’d also like to thank our employer, Dropbox, for funding the mypy core team.

Tuesday, 22 June 2021

Mypy 0.910 Released

We’ve just uploaded mypy 0.910 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This is a small feature release that includes a new command-line option --non-interactive for installing stub (type) packages without asking for a confirmation, fixes to regressions, and a few other improvements. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Installing Stub Packages Non-Interactively

Mypy 0.900 added --install-types to install missing stub packages. We received feedback that this wasn’t a good fit for all use cases, since it asks for interactive confirmation from the user, and it requires another mypy invocation to actually perform type checking.

This release adds the new option --non-interactive that can be used with --install-types to install suggested stub packages without asking for confirmation. This can be useful in Continuous Integration jobs. The option causes mypy to both install stub packages and perform type checking within a single invocation. When not using --non-interactive, you’d have to run mypy again to get up-to-date results with the installed stubs.

Here’s an example of using --non-interactive:

    $ mypy --install-types --non-interactive src/
    Installing missing stub packages:
    /Users/jukka/venv/mypy/bin/python3 -m pip install types-requests
    
    Collecting types-requests
      Downloading types_requests-2.25.0-py3-none-any.whl (22 kB)
    Installing collected packages: types-requests
    Successfully installed types-requests-2.25.0
    
    Success: no issues found in 15 source files

This option provides a new way to migrate existing mypy runner scripts after updating to mypy 0.9xx: just add the --install-types --non-interactive options to your mypy command line.

Note that --install-types currently always installs the latest stubs for all supported packages. If you want reproducible results from your builds, we recommend explicitly pinning stub package versions in your requirements.txt file, for example.

Python 3.5 Deprecation

Running mypy on Python 3.5 is now deprecated. A future mypy release will drop Python 3.5 support. We haven’t decided when this will happen, but this might happen in the next feature release after 0.910.

Stubgen Improvements

  • Don't annotate unknown argument and return types (Sebastian Rittau, PR 10626)
  • Never generate a variable initializer (Sebastian Rittau, PR 10623)
  • Use NamedTuple class syntax (Sebastian Rittau, PR 10625)
  • Use T | None (PEP 604) instead of Optional[T] (Sebastian Rittau, PR 10624)

Other Fixes and Improvements

  • Fix some crashes from faulty casts (Shantanu, PR 10560)
  • Update docs for deferral of PEP 563 to 3.11 (Smart, PR 10655)
  • Don't suggest to install stubs for packages with py.typed files (Sebastian Rittau, PR 10652)
  • Correct the type package name of pyopenssl ( Sebastian Rittau, PR 10656)
  • Improve error reporting when --install-types has no cache (PR 10667)
  • Suggest types-setuptools for pkg_resources (Sebastian Rittau, PR 10681)
  • Fix crash with assignment to variable guarded with TypeGuard (PR 10683)
  • Don't ask to install a stub package if stubs are installed (PR 10670)
  • Fix crash when inferring multiple assignment with overloaded function (PR 10689)

Typeshed Updates

There are no typeshed updates in this release.

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Sebastian Rittau
  • Shantanu
  • Smart

We’d also like to thank our employer, Dropbox, for funding the mypy core team.

Tuesday, 8 June 2021

Mypy 0.900 Released

We’ve just uploaded mypy 0.900 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes many new features and bug fixes. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Update: Released mypy 0.901 that fixes a crash when using --install-types.

Third-party Library Stubs in Stub Packages (Breaking Change)

Mypy now only ships with type stubs for stdlib modules (plus typing_extensions and mypy_extensions). If you use third-party libraries that don’t include stubs or inline type annotations, you can explicitly install stub packages, usually called types-<name>.

For example, if you use requests, you can install stubs like this:

    python3 -m pip install types-requests

If mypy can’t find stubs, it will suggest the name of the stub package to install:

  t.py:1: error: Library stubs not installed for "requests" 
      (or incompatible with Python 3.8)
  t.py:1: note: Hint: "python3 -m pip install types-requests"
  t.py:1: note: (or run "mypy --install-types" to install all 
      missing stub packages)
  t.py:1: note: See https://mypy.readthedocs.io/en/...

You can also now run mypy --install-types to install all missing stub packages in the mypy run (or the previous run, if you don’t pass any files to check):

  $ mypy --install-types
  Installing missing stub packages:
  /Users/jukka/venv/mypy/bin/python3 -m pip install types-requests

  Install? [yN] y

  Collecting types-requests
    Using cached types_requests-0.1.8-py2.py3-none-any.whl (22 kB)
  Installing collected packages: types-requests
  Successfully installed types-requests-0.1.8

Note: --install-types isn’t supported by mypy daemon.

Note: For security reasons, the suggested stubs are limited to only a small subset of manually selected packages that have been verified by the typeshed team. These packages include only stub files and no executable code.

Note: Even if you are using --ignore-missing-imports, mypy will ask to install stubs for packages that had bundled stubs in previous mypy releases. This helps you avoid losing type checking precision from missing stubs when upgrading to mypy 0.900.

For more information, read the blog post about this change and why we did this. There is more detail in the docs.

Python 2 Support is Opt-in (Breaking Change)

If you want to type check Python 2 code, you now have to install mypy[python2] using pip (on Python 3.8 and later):

    python3 -m pip install -U mypy[python2]

This will install the additional typed-ast pip dependency that is required for Python 2 support. Type checking Python 2 code is still fully supported by mypy.

This was contributed by Shantanu (PR 10183).

TypeGuard

Mypy now supports defining user-defined functions that perform type narrowing, similar to isinstance. Read PEP 647 for the details. Here’s an example adapted from the PEP:

    from typing_extensions import TypeGuard

    def is_str_list(val: List[object]) -> TypeGuard[List[str]]:
        """Are all list items strings?"""
        return all(isinstance(x, str) for x in val)

    def func(val: List[object]) -> None:
        if is_str_list(val):
            print(" ".join(val))  # OK

Since TypeGuard will be included in Python 3.10, which hasn’t been released yet, you will have to import it from a recent version of typing_extensions for now.

This was contributed by Guido van Rossum (PR 9865).

Support for pyproject.toml

You can now use a pyproject.toml file instead of mypy.ini for mypy configuration. Here is an example from the documentation:

    # mypy global options:

    [tool.mypy]
    python_version = "2.7"
    warn_return_any = true
    warn_unused_configs = true

    # mypy per-module options:

    [[tool.mypy.overrides]]
    module = "mycode.foo.*"
    disallow_untyped_defs = true

    [[tool.mypy.overrides]]
    module = "mycode.bar"
    warn_return_any = false

    [[tool.mypy.overrides]]
    module = [
        "somelibrary",
        "some_other_library"
    ]
    ignore_missing_imports = true

This was contributed by Adam Weeden (PR 10219).

Binary Wheels for Apple Silicon

We now include binary wheels for Apple Silicon. This can make mypy runs over 4x faster on supported hardware. This requires a native Apple Silicon build of Python 3.9 to work.

This was contributed by Jingchen Ye (97littleleaf11, PR 10145).

Type Narrowing with type(x)

Mypy now recognizes that checks such as type(x) is str narrow the type of x to str. Previously you had to use isinstance(x, str). Example:

    def example(x: Union[int, str]) -> bool:
        if type(x) is str:
            # Type of "v" is "str" here
            ...
        else:
            # Type of "v" is Union[int, str] here
            # "v" could be a instance of a "str" subclass!
            ...

This was contributed by Pranav Rajpal (PR 10284).

Hosted Documentation for Mypyc

We now have hosted user documentation for mypyc, the compiler we use to speed up mypy. Mypyc is still experimental, but we are looking for early adopters.

New Mypyc Features

  • Support __setitem__, __delitem__, __len__ and __contains__ in native classes (PR 10451)
  • Basic support for class-based named tuples (PR 10252)
  • Support class-based TypedDict definitions (PR 10226)

Mypyc Performance Improvements

  • Pre-allocate space in list comprehensions (97littleleaf11, PR 10295)
  • Faster dict true test (97littleleaf11, PR 10333)
  • Implement dict.setdefault primitive (97littleleaf11, PR 10286)
  • Optimize truth value testing for strings (pranavrajpal, PR 10269)
  • Faster set creation with generator expression (97littleleaf11, PR 10261)
  • Add a primitive for loading the bool type (97littleleaf11, PR 10257)
  • Faster creation of a tuple from list/tuple (97littleleaf11, PR 10217)
  • Use vectorcalls to speed up calls to Python objects (PR 10153)
  • Add primitive for str.replace (97littleleaf11, PR 10088)
  • Add primitives for list.index() and list.remove() (PR 9961)
  • Intern string literals (PR 9960)
  • Add fast path for simple calls in wrapper functions (PR 9948)
  • Use METH_FASTCALL with __call__ (PR 9946)
  • Use faster METH_FASTCALL wrapper functions on Python 3.7+ (PR 9894)
  • Add primitives for list.sort() and list.reverse() (PR 9897)
  • Recognize six.moves.xrange as an alias of range (PR 9896)
  • Some tagged integer micro-optimizations (PR 9801)
  • Add primitive for dict.copy (vsakkas, PR 9721)

Other Mypyc Fixes and Improvements

  • Fix reference counting in dict.setdefault (97littleleaf11, PR 10334)
  • Don't coerce types checked with isinstance (Sara Sinback, PR 10245)
  • Simplify argument parsing in legacy wrapper functions (PR 10234)
  • Simplify generated C by omitting gotos to the next block (PR 10230)
  • Box initializers of final variables when required (PR 10229)
  • Fix compilation of unreachable expressions (PR 10228)
  • Foundational support for tuple literals (and None and bool literals, PR 10148)
  • Use tables to construct and store literals (PR 10147)
  • Fix overflow error handling in list.insert (PR 9895)
  • Always add implicit None return type to __init__ method (Thomas Johnson, PR 9866)
  • Small documentation updates (Thomas Johnson, PR 10184)
  • Updates to introduction in the documentation (PR 10179)
  • Various small doc updates (PR 10177)
  • Make mypyc docs use the same theme as mypy docs (PR 10176)
  • Fix mypyc failing to compile on CPython 3.10.0a6 (Adrian Freund, PR 10202)
  • Avoid declaring variable in C for loop initialization (Thomas Johnson, PR 10091)

Stubgen Improvements

  • Properly convert overloaded functions (Chad Dombrova, PR 9613)
  • Fix invoking mro() (Gen Xu, PR 10138)
  • Render a bit better stubs (Sergei Izmailov, PR 9903)
  • Fixed Any and Dict missing from stubgen imports list (Ashley Whetter, PR 9900)
  • Fix type error (Shantanu, PR 10523)
  • Fixes and typos (Sergei Izmailov, PR 9877)

Stubtest Improvements

  • Improve build error messages (Shantanu, PR 10365)
  • Check overloads are functions, don't early return (Shantanu, PR 10258)
  • Fix signature construction for overload + implicit classmethod (Shantanu, PR 9921)

Documentation Updates

  • Add security documentation for --install-types (Adam Weeden, PR 10555)
  • Improve documentation of extending mypy (Nikita Sobolev, PR 10222)
  • Update documentation links (Shantanu, PR 10159)
  • Fix incomplete error codes documentation (Iulian Onofrei, PR 9799)
  • Document explicit_package_bases config file option (Sam Bull, PR 10020)
  • Clarify that mypy_path is relative to current working directory (Gustav Gränsbo, PR 9959)
  • Emphasise that users probably want --follow-imports=normal (Shantanu, PR 9955)

Other Notable Fixes and Improvements

  • Make ignore_missing_imports work for third-party libraries which previously had bundled stubs (PR 10582)
  • Restrict the number of errors shown when there are missing stubs (PR 10579)
  • Mention fullname of PlaceholderNode in assertion to simplify debugging (Adam Weeden, PR 10565)
  • Use double quotes in error messages instead of single quotes (Dixith)
  • Use double quotes in various error messages (Prasanth Chettri, PR 10053)
  • Fix crash on TypeGuard plus "and" (Jelle Zijlstra, PR 10496)
  • Fix spurious name undefined error in class body within import cycle (PR 10498)
  • Remove assertion from TypeGuard serialization (Jelle Zijlstra, PR 10486)
  • Do not allow to use Final and ClassVar at same time (Yurii Karabas, PR 10478)
  • Allow running mypy if Python 2 typeshed stubs aren't installed (PR 10494)
  • Allow inline comments in VERSIONS (Sebastian Rittau, PR 10472)
  • Infer unreachability for await no_return() (Shantanu, PR 10458)
  • Fix unresolved reference + tuples crashing (EXPLOSION, PR 10401)
  • Fix crash with type alias inside __init__ in incremental mode (PR 10432)
  • Fix crash with nested NamedTuple in incremental mode (PR 10431)
  • Fix crash related to module-level __getattr__ in incremental mode (PR 10430)
  • Add support for functools.cached_property (Marc Mueller, PR 10408)
  • Don't narrow Type[X] with a metaclass (PR 10424)
  • Remove note: setup.cfg does need [mypy] section (James Cooke, PR 10364)
  • Fix inference of IntEnum value attribute type (Python 2, PR 10417)
  • Don't report error when using total_ordering in Python 2 mode (PR 10416)
  • Fix narrowing down TypedDict unions with enum literal types (PR 10415)
  • Fix inference of IntEnum value attribute type (PR 10412)
  • Use typeddict-item error code for more errors (PR 10410)
  • Fix a bunch of tests broken by python 3.10 (Adrian Freund, PR 10404)
  • Support maximum version marker of stdlib typeshed modules (Sebastian Rittau, PR 10402)
  • First look into @python2 subdirectory of search path items in Python 2 mode (PR 10392)
  • Make enum type compatible with union of all enum item literals (PR 10388)
  • Fix an issue with union types containing literals (Adrian Freund, PR 10373)
  • Fix re-exporting __all__ in a stub file (PR 10382)
  • Extend the dataclass plugin to deal with callable properties (Aaron Ecay, PR 10292)
  • Make TypedDict incompatible type message more understandable (Akshay K, PR 10326)
  • Make --explicit-package-bases invertible (Gustav Gränsbo, PR 9969)
  • Fix crash for protocol classes and Hashable false negative (Christoph Tyralla, PR 10308)
  • Fix strict equality false positive when strict optional disabled (Kamil Turek, PR 10174)
  • Issue an error when overriding typeguard with non-typeguard in subclass (Kamil Turek, PR 10300)
  • Support functools.total_ordering (Ashley Whetter, PR 7831)
  • Support reversed operand order when comparing against sys.version_info (Kamil Turek, PR 10288)
  • Only import distutils if it is needed (Adrian Freund, PR 10203)
  • Recombine complete union of enum literals into original type (PR 9097, PR 9063) (Ethan Leba)
  • Support union type arguments when showing protocol member conflicts (97littleleaf11, PR 10154)
  • Fix recently added enum value type prediction (Paddy, PR 10057)
  • Support subclassing of NodeVisitor and TraverserVisitor in plugins (Nikita Sobolev, PR 10125)
  • Speed up processing of new files in daemon by caching ASTs (PR 10128)
  • Add narrowing and closures to common issues (Shantanu, PR 9956)
  • Add Python script to build wheels using cibuildwheel (PR 10096)
  • Don't try to follow imports to encodings.* (PR 10094)
  • Speed up case sensitive is-file check in file system cache (PR 10093)
  • Use fast path for finding modules in mypy daemon (PR 10095)
  • Fix following stub imports in daemon when using --follow-imports=skip (PR 10092)
  • Use sys.executable instead of python3 for --install-types (Anthony Sottile, PR 10086)
  • Add build-requirements.txt to MANIFEST.in (PR 10071)
  • Add separate requirements file for mypy self-compile (PR 10069)
  • Expand user home for cache_dir (Marco Zatta, PR 10051)
  • Fix daemon crash when deleting packages (PR 10036)
  • Keep track of fine-grained dependencies for modules in typeshed (PR 10034)
  • Fix import following issue in daemon (PR 10032)
  • Fix mypy daemon crash when following from a new module (PR 10030)
  • Detect invalid value for --custom-typeshed-dir (PR 9985)
  • Fix pip install for new typeshed (Ivan Levkivskyi, PR 9976)
  • Report incompatible assignments for descriptors with overloaded __set__ methods (Christoph Tyralla, PR 9893)
  • Fix TypedDict.get("missing_key") with string literal (Adrian Freund, PR 9906)

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Aaron Ecay
  • Adam Weeden
  • Adrian Freund
  • Akshay K
  • Akuli
  • Anthony Sottile
  • Arjun
  • Ashley Whetter
  • Chad Dombrova
  • Christoph Tyralla
  • Dixith
  • Ethan Leba
  • EXPLOSION
  • Gen Xu
  • Gustav Gränsbo
  • hatal175
  • Iulian Onofrei
  • James Cooke
  • Jelle Zijlstra
  • Jesús Cea
  • Jingchen Ye (97littleleaf11)
  • Kamil Turek
  • Kyle Verhoog
  • Marc Mueller
  • Marco Gorelli
  • Marco Zatta
  • Nikita Sobolev
  • Paddy
  • Philip Jägenstedt
  • Pranav Rajpal
  • Prasanth Chettri
  • Raphael Gaschignard
  • Reda Bouaida
  • Saiprasad Kale
  • Sam Bull
  • Sara Sinback
  • Sebastian Rittau
  • Sergei Izmailov
  • Shantanu
  • Thomas Johnson
  • vsakkas
  • wwuck
  • Xuanda Yang
  • Yurii Karabas

Additional thanks to all contributors to typeshed:

  • Adrian Freund
  • Akuli
  • Alexander Reynolds
  • Andre Delfino
  • Andrew Crabtree
  • Andrew Zhou
  • Andrey
  • Ankur Singh
  • Anthony Sottile
  • Anton Agestam
  • Arie Bovenberg
  • Árni Már Jónsson
  • Ashwin Ramaswami
  • baterflyrity
  • Cássio Botaro
  • Christopher Dignam
  • Christopher Gurnee
  • Christopher Sang
  • Daniel Rosenwasser
  • danieleades
  • Dante Camarena
  • David Barker
  • Dean Way
  • dearoneesama
  • dod-wdwilcox
  • Dominic Davis-Foster
  • Edgar Handal
  • Eric Traut
  • Erik Soma
  • Giel van Schijndel
  • Gilad Reti
  • Greg Ward
  • Guilhem C
  • Guillaume Ayoub
  • harens
  • hatal175
  • Henry Schreiner
  • heroesm
  • Igor Nehoroshev
  • Ilya Konstantinov
  • Iwan
  • jack1142
  • Jacob Hayes
  • Jake Bailey
  • James
  • James Morris
  • Jason Huang
  • Jelle Zijlstra
  • Jia Chen
  • John T. Wodder II
  • Jon Banafato
  • Jon Dufresne
  • JonathanMeans
  • Jörg Herbel
  • Jürgen Gmach
  • Kaleb Barrett
  • Ken Odegard
  • krouziciorel
  • Kyle Altendorf
  • ldruschk
  • Mahmoud Harmouch
  • Marat Sharafutdinov
  • Mark Vismonte
  • Marks Polakovs
  • Marti Raudsepp
  • Mathieu Parent
  • Maxime Arthaud
  • Mohit Goyani
  • MrGreenTea
  • n0ob404
  • naglis
  • Nick Crews
  • Nick M
  • Nipunn Koorapati
  • Philipp Hahn
  • Rebecca Chen
  • Ricardo
  • rijenkii
  • Rune Tynan
  • Ryan McCampbell
  • sachindavra
  • Sam Bull
  • Sean B. Palmer
  • Sebastian Rittau
  • Shantanu
  • Sorin Sbarnea
  • spaghEddieDoyle
  • Stanislav Levin
  • Steve B
  • Tadek Kijkowski
  • Takumi Kato
  • Terence Honles
  • Thomas Cellerier
  • Tigran Kostandyan
  • Tushar Sadhwani
  • Udi Fuchs
  • Utsav
  • Vasily Zakharov
  • Viktor Roytman
  • Vyom Pathak
  • Wilfredo Sánchez Vega
  • Wolf Honore
  • Yehuda Davis

Wednesday, 26 May 2021

The Upcoming Switch to Modular Typeshed

Mypy has been bundling stubs for many third-party libraries from typeshed, a repository of type definitions for Python packages. Mypy uses the stubs to type check code that uses these libraries, and they allow more types to be inferred by mypy. Typeshed is not just for mypy — it’s also used by several other tools in the Python typing ecosystem.

Mypy will no longer include stubs for third-party libraries in the soon-to-be-released mypy 0.900 release. Instead, mypy will only ship stubs for standard library modules, while stubs for third-party libraries can be independently installed using pip. For example, if you use requests, you can install the stubs like this:

  python3 -m pip install types-requests

The stub packages are named types-<project name>. Note that the project name may be different from the runtime package name. For example, stubs for the yaml package are available in types-PyYAML.

Steps for Mypy Users

When you update to mypy 0.900 (once it has been released), you’ll likely get errors about missing stubs. Mypy will suggest the stub package that should be installed. Here’s an example output from mypy:

  a.py:1: error: Library stubs not installed for "requests"
  a.py:1: note: Hint: "python3 -m pip install types-requests"
  a.py:1: note: (or run "mypy --install-types" to install all 
      missing stub packages)
  a.py:1: note: See https://mypy.readthedocs.io/en/...

You can now run mypy --install-types to install all the missing dependencies (unless you are using mypy daemon). Alternatively, you can install the stubs manually using pip as suggested by mypy.

As a final step, you may want to add the stub packages to the build or test dependencies of your project, often in requirements.txt. You can use pip freeze to help pin to the current installed versions of the stubs. For example, you might add this line to your requirements.txt:

  types-requests==0.1.8

Currently stub packages are mostly versioned independently of the implementation package. However, it’s possible to update stub metadata in typeshed to have the versioning better reflect the target implementation package version.

Motivation

The bundling of stubs with mypy has been convenient, but it also has caused a lot of friction, especially for users with many third-party dependencies. Here are the main benefits of unbundling stubs:

  • Updating stubs without updating mypy. Right now it’s not straightforward to update bundled stubs to a newer version without also updating mypy. Soon you can use pip to update to the latest stubs, without having to update to mypy as well.
  • Updating mypy without updating stubs. Updating stubs or installing new stubs sometimes results in additional mypy errors, due to newly added but too restrictive types in stubs, or newly introduced but correct types that expose incorrect types in your code. Previously mypy users had to deal with these errors when updating mypy, which complicated the update process. Soon you’ll be able to update mypy without updating third-party stubs. Standard library stubs will still be bundled for convenience, since all mypy users need them, and they tend to be easier to validate for correctness.
  • Faster access to new stubs and improvements. If new stubs or stub improvements are contributed to typeshed, previously you had to wait for a new mypy release that includes them, or deal with a custom, unsupported typeshed set-up that hasn’t been properly tested. With mypy 0.900, you will be able to install or update stub packages whenever you want. As soon as a change to a stub is merged, typeshed will automatically upload a fresh stub package to PyPI. If you are not happy with some stub changes or encounter an issue, you can easily contribute fixes and start using the updated stubs, or continue to use older stubs if you so wish.
  • Lower barrier to contributing new stubs. Since previously every user would get all contributed stubs, each new stub was a somewhat risky proposition. If some types were incorrect, or if there were missing definitions in the stubs, this could result in many false positives. Modular typeshed lets each user individually decide whether to start using the contributed stubs, making the impact of glitches in stubs lower. The faster iteration speed should also make it more likely that issues will be fixed soon.
  • Consistency. Previously some third-party library stubs were bundled with mypy, some were bundled with the main implementation (upstream) package, and some were available on PyPI as separate stub packages. Now there will be one fewer option: you’d either use stubs or types included with the implementation package, or install a stub package. Types included with the main package are the preferred option, but many packages still don’t have them. Stub packages will be common for some time. Installing, versioning and updating stubs and implementation packages will also be more similar.
  • Support for multiple library versions. Libraries may introduce backward incompatible API changes, such as removing deprecated features. Sometimes, for various reasons, you may want to continue using an older version of a library. Now it will be easy to also continue using an older version of stubs, since library stubs are independently versioned. This also opens the option of maintaining separate stubs for versions 1.x and 2.x of a library, for example.

Vendoring Stubs

If you don’t want to deal with installing stub packages, mypy 0.900 will support vendoring stubs (i.e., maintaining a local copy in your repository). First, you can copy some stubs from typeshed into a directory, and point the MYPYPATH environment variable (or mypy_path in the mypy configuration file) to it. Mypy will look for stubs in that directory. For example, you could use a directory named stubs in your repository structured like this:

  • stubs/toml.pyi
  • stubs/yaml/__init__.pyi

If your project supports both Python 2 and 3, you can have stubs for Python 2 in a @python2 subdirectory of the main stub directory:

  • stubs/foobar.pyi (used for Python 3)
  • stubs/@python2/foobar.pyi (used for Python 2)

If you include stubs in the mypy search path, stubs/@python2 will automatically be used for Python 2 stubs.

Feedback?

Feel free to open a GitHub issue if you have concerns about this new approach of distributing stubs.

Acknowledgements

Ivan Levkivskyi set up the infrastructure for automatically uploading stub packages to PyPI and implemented the necessary changes to typeshed.