Friday, 12 July 2019

Mypy 0.720 Released

We’ve just uploaded mypy 0.720 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release features a new semantic analyzer and optional warnings about unreachable code. It also includes many other new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

New Semantic Analyzer Used by Default

The new semantic analyzer was introduced in mypy 0.710 and is now the default. The old analyzer is still available under --no-new-semantic-analyzer, but it will be removed in the next mypy release. This release includes many improvements to the new semantic analyzer.

The semantic analyzer is a component of mypy that binds names to definitions. It also keeps track of what sort of thing each name defines, such as a function or a type alias. The new semantic analyzer is a major redesign and achieves several things.

The old semantic analyzer ordinarily processes files in a linear order, from the beginning to the end. This is subtly different in the mypy daemon, which sometimes processes an individual function or a module top level (code not nested within a function) as a single unit. The new semantic analyzer unifies mypy behavior by always analyzing module top-levels separately, before analyzing functions. As a bonus, this more closely reflects Python runtime behavior, since modules are usually fully initialized before any functions are called.

The old semantic analyzer processes each module in an import cycle twice, in an order determined by complicated heuristics. This can cause bogus errors when there are references to another module that hasn’t been processed yet in a dependency cycle. The new analyzer processes modules in a flexible order, and references across an import cycle work significantly more reliably.

The old semantic analyzer can’t always deal with forward references to definitions. The new semantic analyzer improves support for forward references. Example where this helps:

    from typing import NamedTuple
    
    f = lambda: A(1)  # This forward reference now works
    
    A = NamedTuple('A', [('x', int)])
    
    f()

The new semantic analyzer is stricter than the old semantic analyzer. It catches additional type errors and quality issues. Existing code that passes type checking may require small adjustments.

New Feature: Warn about Unreachable Code

If you run mypy with --warn-unreachable, mypy will warn you about unreachable statements, and expressions that may be never evaluated because of short circuiting. For example:

    from typing import Text
    
    def emphasize(text: Text) -> Text:
        if isinstance(text, int):
            text += 1  # Error: statement is unreachable
        return text.upper()

See the documentation for more information. This feature was contributed by Michael Lee.

Plugin API Changes

The new semantic analyzer may require changes to mypy plugins. We have a GitHub issue with more information for maintainers of plugins.

Notable Improvements and Bug Fixes

  • Have --package usage respect mypy_path (Aaron Batilo, PR 6926)
  • Support flexible option specification in config files. Now all the flags support invertible forms so that one can specify either strict_optional = False or no_strict_optional = True. This matches how flags work on the command line (PR 7054)
  • Fix reachability inference with isinstance(Any, Any) (Michael Lee, PR 7048)
  • Fix mypyc crash with plugins using get_customize_class_mro_hook() (PR 7075)
  • Fix slicing tuples with non-literal expressions (Ethan Smith, PR 7066)
  • Fixes to column numbers in error messages (PR 7078, PR 7081)
  • Make Need type annotation for variable error more consistent (PR 7079, PR 7113)
  • Fix error reporting context for missing generic type arguments (PR 7100)
  • Use only directories containing no __init__.py as namespace packages (Arne Welzel, PR 7108)
  • Support method plugin hooks on union types (PR 6560)
  • Fix some corner cases in --strict-equality (PR 7156, PR 7164)
  • Fix binding of self-types on unions (PR 7152)
  • Add more details for Invalid type errors (PR 7166)
  • Fix custom __init__() in dataclasses (Ryan Gonzalez, PR 7168)
  • Better support for indexing with unions of literal types (Michael Lee, PR 6558)

Other Updates

You can browse the full commit log here.

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Aaron Batilo
  • Arne Welzel
  • Chad Birch
  • Daniel Hahler
  • Deepyaman Datta
  • Ethan Smith
  • Michael Lee
  • Rafi Blecher
  • Ryan Gonzalez

Additional thanks to all contributors to typeshed:

  • Benjamin Woodruff
  • Chad Dombrova
  • Cole Maclean
  • crusaderky
  • Eric Arellano
  • Florian Ludwig
  • Francis Colas
  • Gordon Ball
  • Hynek Schlawack
  • ikelos
  • Jason Gilholme
  • Jelle Zijlstra
  • Maarten ter Huurne
  • Martijn Pieters
  • Mathieu Bridon
  • Matthew Wilkes
  • Motoki Naruse
  • Rafi Blecher
  • Ran Benita
  • Rebecca Chen
  • redshiftzero
  • Rune Tynan
  • Sam Zhou
  • Savo Kovačević
  • Sebastian Rittau
  • 秋葉
— Ivan Levkivskyi, on behalf of the mypy team

Monday, 24 June 2019

Mypy 0.711 Released

We’ve just uploaded mypy 0.711 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This fixes two minor issues with mypy 0.710. You can install it as follows:

python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Details

The following two issues in mypy 0.710 were fixed:
  • Revert typeshed PR 2878 (“Define functools.partial as overloaded function instead of its own class”). This caused too many false positive errors in real-world code. (PR 3077)
  • Fix MYPYC_BLACKLIST on Windows. This broke running dmypy on Windows. (PR 7032)

If you’re upgrading from mypy 0.701, mypy 0.700 or earlier, please read the blog post about mypy 0.710.

Wednesday, 19 June 2019

Mypy 0.710 Released: New Semantic Analyzer

We’ve just uploaded mypy 0.710 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release features a new, experimental semantic analyzer and inline configuration options. It also includes many other new features, bug fixes and library stub (typeshed) updates. You can install it as follows:
    python3 -m pip install -U mypy
You can read the full documentation for this release on Read the Docs.

New Semantic Analyzer (Experimental)

We invite everyone with more than a passing interest in mypy to try out the new semantic analyzer. Please report any issues you might find, since we will likely make it the default in the next mypy release, and we will drop the old semantic analyzer soon after.
The semantic analyzer is a component of mypy that binds names to definitions. It also keeps track of what sort of thing each name defines, such as a function or a type alias. This release officially supports a revamped implementation of the semantic analyzer, dubbed the new semantic analyzer. In particular, it deals better with import cycles and forward references.
You can enable the new semantic analyzer by passing --new-semantic-analyzer on the command line or by setting new_semantic_analyzer = True in your mypy config file (typically mypy.ini). (docs)

Inline Configuration Options

You can now specify per-file configuration flags in # mypy: comments. For example, you can now put
    # mypy: no-strict-optional
at the top of a file to disable strict optional checking in that specific file. Previously you would have to specify this on the command line (e.g., mypy --strict-optional) or in the mypy config file (e.g., strict_optional = True in the [mypy] section or in a file-specific section). Multiple flags can be separated by commas or placed on separate lines.
    # mypy: no-warn-no-return
    # mypy: disallow-any-generics, always-false="FOO,BAR"
For more info, see the docs.

Type inference improvements and bug fixes

  • Improvements to the handling of Literal types. (Michael Lee; docs)
  • You can now ignore a file with a single # type: ignore comment at the top of the file. (Brandt Bucher, PR 6830)
  • Fix crashes with callable() and issubclass(). (Jelle Zijlstra, PR 6981)
  • Allow class-based TypedDicts in all Python versions. (PR 6971)
  • Fix yield from <tuple>. (Brandt Bucher, PR 6902)
  • Add more precise inference for enum attributes. (Michael Lee, PR 6867)
  • When strict optional is disabled, anything can be falsey; previously this caused some incorrect assumptions about code being unreachable. (PR 6876)
  • Document extended callable types as deprecated. (PR 6890)
  • Allow redefinition within with statements. This is strictly speaking unsafe, but allowing this is convenient in the vast majority of cases. (PR 6871)
  • Allow del with a list, e.g. del [x, y]. The semantics are the same as del x, y. (Brandt Bucher, PR 6858)
  • Don't crash when partial types are used in inherited attribute. (Ekin Dursun, PR 6766)
  • Allow ellipsis default arguments in "trivial" non-stub functions. (Michael Lee, PR 6665)
  • Don't treat the first argument of a function nested inside a method as self. (Ekin Dursun, PR 6769)
  • Emit an error message when a class member is redefined. (Ekin Dursun, PR 6686)
  • Don’t complain about overriding a private method name in a subclass. (Rafael Carício, PR 6790)
  • Refine the interaction between Literal and Final. (Michael Lee, PR 6763)
  • Fix the dataclass plugin’s interaction with member expressions. (PR 6739)
  • Support decorated constructors. (Previously, decorating __init__ would cause the class’s constructor to be unchecked!, PR 6645)
  • Fix multiple inheritance check for generic classes. (PR 6720)
  • Add basic support for enum literals. (Michael Lee, PR 6668)
  • Loosen base class attribute compatibility checks when attribute is redefined (Seth Yastrov, PR 6585)
  • Fix crash when a star expression is used in isinstance. (Ekin Dursun, PR 6659)
  • Fix crash when namedtuple, classmethod and generic types used. (Ekin Dursun, PR 6669)
  • Fix __getattr__ and operators related infinite recursion. (Ekin Dursun, PR 6655)
  • Don't type check a return statement containing a lambda . (This would treat everything after the lambda as unreachable., PR 6633)

Error message refinements

  • Messages produced by reveal_type() and reveal_locals() now use the note level rather than error. (Daniel Hahler, PR 6919)
  • Make the error for too few or too many arguments in type comments non-fatal. (PR 6813)
  • Improve the error message when a function is missing a return type. (Allison King, PR 6773)
  • Better error message for assignments and returns incompatible due to invariance. (Felipe de Morais, PR 6803)
  • Update the error message for incompatible arguments. (Matas Kairaitis, PR 6796)
  • Show the specific argument names that are missing in the error message for missing arguments. (Rafael Carício, PR 6795)
  • When a duplicate module is detected, suggest that a missing __init__.py file could be the cause. (Alexander Bliskovsky, PR 6785)
  • Extend the list of third-party packages with the top 200 packages from PyPI. (Matas Kairaitis, PR 6781)
  • Show closest candidates for misspellings in the case of from … import … . (marco-dd, PR 6778)
  • Improve the error message when an annotation is expected. (Rafael Carício, PR 6782)
  • When a type comment contains a syntax error, show the text of the bad comment in the error message. (kcaebe, PR 6779)
  • Include a note when user forgets to import something from the typing module. (Christopher Sabater Cordero, PR 6770)
  • Display both instances of duplicate modules in the error message. (Alexander Bliskovsky, PR 6772)
  • Turn all warnings and bare notes into errors. (PR 6650)

Command line/configuration related changes

This section also covers documentation, stubgen and plugins.
  • Update the typed_ast dependency to 1.4.0+. This allows installing mypy using Python 3.8. (PR 6937)
  • Add --no-implicit-reexport flag. (Jared Hance; docs)
  • Only honor --warn-unused-configs in non-incremental mode. In incremental mode it doesn't work right, and it isn't worth fixing. (PR 6829)
  • Fix sys.stdout overriding in mypy.api. (Amanda Walker, and El Khadiri Yassine, PR 6750)
  • Miscellaneous small improvements to stubgen.
  • Command-line syntax errors now consistently use exit code 2. (PR 6738)
  • Tweaks to --strict-equality based on user feedback. (PR 6674)
  • Allow specifying the list of files to check in the mypy config file. (Samuel Williams, docs)
  • Make directory checks case sensitive. (PR 6684)
  • Allow disabling the config file using --config-file="". (PR 6664)
  • Add some more plugin docs. (PR 6618)
  • Add a get_additional_deps hook for plugins, to support django-stubs. (PR 6598)

Python 3.8 related changes

  • Support / for positional-only arguments (PEP 570). (Brandt Bucher, PR 6900)
  • Support importing Protocol, TypedDict, Literal, Final and @final from typing (typing_extensions is still supported). Ironically, this is also supported Python 2.7. These were recently standardized by PEP 544 (Protocol and @runtime_checkable), PEP 586 (Literal), PEP 589 (TypedDict) and PEP 591 (Final/@final).
  • Fix line numbers assigned to class and def and ignoring for Python 3.8. (Brandt Bucher, PR 6753)
  • Expression-scoped ignores in Python 3.8. (Brandt Bucher, PR 6648)

Other Updates

You can browse the full commit log here.

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.
Thanks to all mypy contributors who contributed to this release:
  • Alexander Bliskovsky
  • Allison King
  • Amanda Walker
  • Anthony Sottile
  • Bernát Gábor
  • Brandt Bucher
  • Brooke
  • cclauss
  • Charles-Axel Dein
  • Christopher Sabater Cordero
  • crusaderky
  • Daniel Hahler
  • duke9509
  • El Khadiri Yassine
  • Ekin Dursun
  • Emil Goldsmith Olesen
  • Ethan Smith
  • Felipe de Morais
  • Ivar
  • Jan Szopinski
  • Jared Hance
  • Jason Michalski
  • Jelle Zijlstra
  • Joe Juzl
  • kcaebe
  • marco-dd
  • Matas Kairaitis
  • Max Mautner
  • Michael Lee
  • Nick Gaya
  • PattenR
  • Rafael Carício
  • Robin Chen
  • Rémi Lapeyre
  • Samuel Williams
  • Sanjit Kalapatapu
  • Seth Yastrov
  • viourr
Additional thanks to all contributors to typeshed:
  • Alex Chamberlain
  • Alexander Fasching
  • Anirudh Padmarao
  • Anthony Sottile
  • Benjamin Peterson
  • berdario
  • Brandt Bucher
  • Brendan Long
  • Brian Brunner
  • Callum Ryan
  • Carl Meyer
  • Chad Birch
  • Chad Dombrova
  • Chandan Singh
  • Connor Skees
  • Dan Crosta
  • Danny Weinberg
  • Dominic
  • Eric Arellano
  • Ethan Madden
  • Evan Moses
  • ijl
  • J Rob Gant
  • Jadiker
  • Jan Szopinski
  • Jean Hominal
  • Jelle Zijlstra
  • Jennifer Taylor
  • Jia Chen
  • Joe Juzl
  • John Freeman
  • Jon Dufresne
  • Jonathan Slenders
  • Josh Morton
  • Mark Mendoza
  • Mark Vismonte
  • markedwards
  • Masashi SHIBATA
  • Mathieu Bridon
  • Max Rydahl Andersen
  • Michael A. Smith
  • Michael Lee
  • nabnut
  • Paul Dagnelie
  • Philipp Hahn
  • Philipp Schrader
  • Radu Matei Lăcraru
  • Ran Benita
  • Rebecca Chen
  • Rune Tynan
  • Saul Shanabrook
  • Scott Belden
  • Sean Vig
  • Sebastian Rittau
  • Sergey Machulskis
  • Shahar Evron
  • Simon Kohlmeyer
  • Stephen Thorne
  • Sushain Cherivirala
  • The Fox in the Shell
  • Utkarsh Gupta
  • Viktor Roytman
  • Walter Scott Johnson
  • Yegor Roganov
  • zadamah
  • 秋葉
— Guido van Rossum, on behalf of the mypy team












Tuesday, 16 April 2019

Mypy 0.701 Released

We've just uploaded mypy 0.701 to PyPI. This is a bugfix release containing no new features, only fixes for some important bugs and regressions in the (default) mypyc-compiled version of 0.700. You can install it as follows:

    python3 -m pip install --upgrade mypy

Mypy Bugs Fixed

  • Fix a crash in mypyc-compiled stubgen (PR 6642)
  • Compile with a newer version of mypyc that fixes several memory leak bugs (PR 6642)

Wednesday, 3 April 2019

Mypy 0.700 Released: Up To 4x Faster

We’ve just uploaded mypy 0.700 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release switches to a compiled version of mypy, which is up to 4x faster than the previous release. It also includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on Read the Docs.

Much Improved performance

We are now shipping a mypy binary compiled with mypyc by default, which is up to 4x faster than the interpreted version shipped previously. The compiled version has been available as a separate PyPI distribution for a while now, and this is first release where it’s installed by default. Mypyc is a compiler that compiles type-annotated Python modules into efficient CPython C extension modules.

Note that the legacy, optional mypy_mypyc package will not be updated any more. If you used the mypy_mypyc package for an earlier mypy release, you should uninstall it first, before updating to the new package (you can safely ignore this if don’t know what this is):

    python3 -m pip uninstall mypy_mypyc

The compiled version is supported on 64-bit macOS, Windows and Linux platforms (Python 3.5 and later). On other platforms an interpreted version is used instead, which doesn’t provide any performance improvement. The compiled version is mostly compatible with the interpreted version, but some mypy plugins may be incompatible with it. You can still install the interpreted version of 0.700 like this:

    $ python3 -m pip install --no-binary mypy -U mypy

Stricter Equality Checks

If you run mypy with --strict-equality, mypy will detect equality and in checks that are likely to always result in True or False, since the operand types are not overlapping. For example, it will warn about comparing a str value to a bytes value, which can never be equal. See the documentation for more information.

Python 3.4 Unsupported for Running Mypy

You can no longer run mypy using Python 3.4, since Python 3.4 has reached its end of life. You can still type check code that targets Python 3.4, and we have no plans to drop support for this.

Plugin System Improvements

  • Add a get_additional_deps() plugin hook to support django-stubs (PR 6598)
  • Use the get_attribute_hook() plugin hook for dynamic classes (lincolnq, PR 6371)

Other Improvements

  • Allow # type: ignore comments after type comments (Ethan Smith, PR 6591)
  • Don't resolve callable NamedTuple fields to their return type (Danny Weinberg, PR 6576)
  • Support indexing unions containing tuples (PR 6475)
  • Fix member access on generic classes (PR 6418)
  • Fix interaction of isinstance() with Type[...] (PR 6419)
  • Fixes to tuple fallback types (PR 6442)
  • Fix missing built-in type aliases (PR 6441)
  • Make processing mypy cache faster (PR 6405)
  • Add an experimental mechanism for avoiding hashing sources on mypy daemon start (PR 6389)
  • Add flag --skip-cache-mtime-checks to skip mtime checks on cache files (PR 6391)
  • Give a better error message (and don't crash) if there is bad JSON in the cache (Ethan Smith, PR 6277)
  • Stubgen: improvements to stubs generated from docstrings (Wiktor Niesiobędzki, PR 6368)

Typeshed Updates (Highlights)

Here are selected typeshed updates (there are many more):

  • orjson: Add new stub file (ijl, PR 2747)
  • bleach: Add new stub file (Sebastian Rittau, PR 2709)
  • pycurl: Support Python 3 (JinyuanShanghai, PR 2867)
  • pydoc: Support Python 3 (Rebecca Chen, PR 2873)
  • multiprocessing: Add multiprocessing.spawn submodule and multiprocessing.pool stubs (Sean McLemon, PR 2823)
  • builtins: Add isascii methods to str, bytes, and bytearray for Python 3.7 (Brandt Bucher, PR 2834)
  • builtins: Add missing BaseException.__suppress_context__ attribute (wouter bolsterlee, PR 2876)
  • platform: Add types to some cross-platform functions (Paul Dagnelie, PR 2781)
  • codecs: Support the errors argument in factory functions (Aymeric Augustin, PR 2752)
  • charset: Improve the stubs under Python 3 (Josh Morton, PR 2768)
  • asyncio: Fix signature of asyncio.create_server (Aymeric Augustin, PR 2763)
  • asyncio: Update asyncio.streams stub (Gleb Chipiga, PR 2845)
  • werkzeug: Some small tweaks (lincolnq, PR 2784)
  • urllib[2]: Add stubs for HTTP Handler classes in Python 2 urllib2 and Python 3 urllib.request (Michael Brandt, PR 2710)
  • urllib2: Add some annotations (Philipp Hahn, PR 2688)
  • cyaml: Add some missing annotations (秋葉, PR 2769)
  • calendar: Add itermonthdays3 and itermonthdays4, fix incorrect cssclass_today (Sean McLemon, PR 2825)
  • enum: Add underscore names (秋葉, PR 2822)
  • logging: Changed parameter name of several methods from 'lvl' to 'level' to match the implementation (Eric Traut, PR 2840)
  • attr: Update attr stubs to 19.1.0 (Gleb Chipiga, PR 2846)
  • ast: Support new features and node types introduced in Python 3.8 (PR 2859)
  • flask: Fix several types (Gabriel Corona, PR 2858)
  • webbrowser: Update signature of webbrowser.register for Python 3.7 (Cary Yang, PR 2865)
  • pdb: Add missing class Pdb (Rebecca Chen, PR 2872)
  • yaml: add full_load and full_load_all stubs (Christopher Dignam, PR 2892)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Anthony Sottile
  • Bernát Gábor
  • Danny Weinberg
  • Ethan Smith
  • Jelle Zijlstra
  • lincolnq
  • Mark Mendoza
  • Michael
  • Michael Lee
  • oremanj
  • Richard Eames
  • Sean McLemon
  • Sebastian Rittau
  • Wiktor Niesiobędzki

Additional thanks to all contributors to typeshed:

  • Alun Champion
  • Andrew Svetlov
  • Antoine Fourmy
  • Aymeric Augustin
  • Benjamin Peterson
  • Bertrand Bonnefoy-Claudet
  • Brandt Bucher
  • Cary Yang
  • Chen Li
  • Christopher Dignam
  • Eric Traut
  • Gabriel Corona
  • Gleb Chipiga
  • herr kaste
  • Hynek Schlawack
  • ijl
  • Ingmar Steen
  • Jeff
  • Jelle Zijlstra
  • JinyuanShanghai
  • Josh Morton
  • Joshua Oreman
  • Lawrence Chan
  • lincolnq
  • Luke Granger-Brown
  • Mark Mendoza
  • Matt Robinson
  • Maxim Kurnikov
  • Michael Brandt
  • Nipunn Koorapati
  • Pascal Corpet
  • Paul Dagnelie
  • Philipp Hahn
  • Rebecca Chen
  • Sean McLemon
  • Sebastian Rittau
  • Tom Briggs
  • Utkarsh Gupta
  • Vasily Zakharov
  • William Ayd
  • wouter bolsterlee
  • yoshiyuho
  • Łukasz Hanuszczak
  • 秋葉
— Jukka Lehtosalo, on behalf of the mypy team

Friday, 1 March 2019

Extending mypy with plugins

Python is a famously dynamic language. It’s easy to write DSL-like frameworks that are hard to understand by static type checkers. Although with recent mypy features, such as protocols and literal types, and with basic metaclass and descriptor support, we can more often provide precise types, it’s still often hard to avoid false positives and negatives. To solve this problem, while avoiding the need for custom type system features for every framework, mypy supports a plugin system. Plugins are Python modules that provide callbacks (plugin hooks) that mypy will call when type checking classes and functions that interact with a library or framework. For example, such hooks can provide a more precise function return type that is otherwise hard to express, or auto-generate some methods in a class, to reflect the effects of a class decorator. To read more about plugin system architecture and for the full list of available hooks, see these docs.

Bundled plugins for standard library

Mypy comes with a default plugin for some builtin functions and classes, and the ctypes, contextlib, and dataclasses modules. It also includes a plugin for attrs (for historical reasons — it was the first third-party plugin written for mypy). These plugins allow mypy to infer more precise types and correctly type check code using these library features. To illustrate this, consider this snippet:

    from dataclasses import dataclass
    from typing import Generic, TypeVar
    
    @dataclass
    class TaggedVector(Generic[T]):
        data: List[T]
        tag: str
    
    position = TaggedVector([0, 0, 0], 'origin')

Above, get_class_decorator_hook() is called when analyzing the class definition. It adds autogenerated methods, including __init__(), to the class body. Mypy uses this generated constructor to correctly infer TaggedVector[int] as the type of position. As you can see, plugins work even with generic classes.

Here’s another snippet:

    from contextlib import contextmanager
    
    @contextmanager
    def timer(title: str) -> Iterator[float]:
        ...
    with timer(9000) as tm:
        ...

The get_function_hook() hook provides a precise return type for the contextmanager decorator, so that calls to the decorated function can be type-checked precisely. Mypy can now detect an error: the argument to timer() must be a string.

Combining plugins and stubs

In addition to relying on dynamic Python features, frameworks often have the challenge of having large APIs. Mypy needs stub files for the libraries to check code that uses those libraries (unless the library contains inline annotations, which is still often not the case). Distributing the stubs for large frameworks via typeshed is not very practical:

  • Typeshed has a relatively slow release cycle (it’s bundled with mypy).
  • Incomplete stubs can cause false positives that are hard to avoid.
  • There’s no easy way to mix and match stubs from different typeshed versions.
Stub packages, introduced in PEP 561, help with these issues:
  • Maintainers can release a stub package as often as they want.
  • Users who haven’t opted in to using the package won’t see any false positives.
  • You can freely install arbitrary versions of multiple different stub packages.

Moreover, pip allows combining stubs for a library and the corresponding mypy plugin into a single distribution. Stubs for a framework and a corresponding mypy plugin can now be easily developed and distributed together, which is often useful since plugins fill in missing or imprecise definitions in the stubs.

A recent example of such a package is SQLAlchemy stubs and plugin, the first public 0.1 release of which was published on PyPI earlier this month. Although the project is still in early alpha, we are already successfully using it at Dropbox to improve type checking precision. Currently the plugin understands basic ORM declarations:

    from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy import Column, Integer, String
    
    Base = declarative_base()
    
    class User(Base):
        __tablename__ = 'users'
        id = Column(Integer, primary_key=True)
        name = Column(String)

In the above snippet, the plugin uses the get_dynamic_class_hook() hook to tell mypy that Base is a valid base class, even though it doesn’t look like one. Then get_base_class_hook() is called on the definition of User, and it adds some autogenerated attributes. Next we instantiate the model:

    user = User(id=42, name=42)

The get_function_hook() hook is called, so that mypy can spot the error here: an integer is given instead of a user name.

The stubs define Column as a generic descriptor, so that attributes on a model get correct types:

    id_col = User.id  # Inferred type is "Column[int]"
    name = user.name  # Inferred type is "Optional[str]"

We welcome PRs that add more precise types to the stubs (the progress for core modules is tracked here).

Here are some gotchas that we found while working on the stubs:

  • Use a module-level __getattr__() to avoid false positives during the very early stages, when stubs are incomplete (this suppresses mypy errors due to missing module attributes). You can also use it in __init__.py files if some submodules are missing.
  • Descriptors often help with giving more precise types for customized attribute access (like in the Column example above), and it’s OK to use them even if the actual runtime implementation uses a more complex mechanism involving a metaclass, for example.
  • Don’t hesitate to declare framework classes as generic in stubs. Although they are not generic at runtime, this often allows us to give much more precise types for certain framework features, while runtime errors can easily be worked around. (We hope that frameworks will gradually add built-in support for generic types by explicitly inheriting relevant classes from typing.Generic.)

Recently released mypy plugins

There are already several plugins available for some popular Python frameworks. Apart from the above mentioned plugin for SQLAlchemy, other notable recent examples of packages with stubs and a bundled mypy plugin include stubs for Django and Zope Interfaces. All these projects are under active development.

Installing and enabling plugin + stub packages

Use pip to install a package with a mypy plugin and/or stubs in the virtual environment where mypy is installed:

    $ pip install sqlalchemy-stubs

Mypy should automatically discover the installed stubs. To enable the plugins you’ve installed, explicitly include them in your mypy.ini (or a custom config file):

    [mypy]
    plugins = sqlmypy, mypy_django_plugin.main 

Developing mypy plugins and writing stubs

If you want to develop a stubs + plugin package for a framework you use, you can use the sqlalchemy-stubs repository as a template. It includes the setup.py file, testing infrastructure using data-driven tests, and an example plugin class with a bunch of plugin hooks. We recommend using stubgen, an automatic stub generator that comes with mypy, to get started with stubs. Stubgen got several improvements in mypy 0.670.

For more details about mypy plugin system, see the docs. You can also just browse the source code of the plugins mentioned above. If you have any questions, feel free to ask them on the Python typing Gitter chat.

Friday, 8 February 2019

Mypy 0.670 Released

We’ve just uploaded mypy 0.670 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

(There are also new mypy-mypyc wheels; see the 0.660 release blog post.)

You can read the documentation for this release on Read the Docs.

New Feature: Variable Redefinition

As you might have experienced, PEP 484 and mypy require a variable to keep its initial type throughout a scope. Occasionally this can be annoying, as the traditional workaround is to rename the second (independent) use of the variable. We have introduced a new flag, --allow-redefinition, which relaxes this behavior in certain contexts. Here’s an example where this may be useful:

    def process(items: List[str]) -> None:
        # 'items' has type List[str]
        items = [item.split() for item in items]
        # 'items' now has type List[List[str]]
        ...

You can also enable this per module in the mypy.ini file using allow_redefinition = True. See the docs for the command-line flag and the config option.

Stubgen Improvements

We’ve done a fairly large overhaul of the stubgen utility, which can automatically generate draft stub files for almost any module or package:

  • Streamline the command-line interface; in particular, the source discovery/collection options now more closely resemble those of mypy itself: -m <module>, -p <package>, <file> or <directory>
  • Perform a lightweight semantic analysis to generate somewhat better stubs
  • Added documentation
  • When parsing types out of docstrings (this happens for C modules only), stubgen now handles overloaded signatures generated by pybind11 (Wiktor Niesiobędzki, PR 5975)

Other Improvements

  • Expand getting started docs to discuss type hints in more detail (Michael Lee, PR 6226)
  • Always infer in operator as returning bool (Joel Croteau, PR 5688)
  • Allow star args in ctypes.Array constructor (Alan Du, PR 6213)
  • Fix plugin invocation for __call__ methods (oremanj, PR 6334)
  • Implement the XDG directory spec for config files: $XDG_CONFIG_HOME/mypy/config is now included in the search path for config files (Ryan Delaney, PR 6304)
  • When using the --junit-xml flag, the Python version and platform in the junit.xml file are now formatted as mypy-py3_6-windows — previously this was mypy-py3.6-windows but the dot was misinterpreted by some tools (PR 6222)
  • Update the typed_ast dependency to version 1.3.1; this means we now officially support Python 3.7
  • Temporarily delete pyproject.toml from the repo in order to work around a pip bug (PR 6342)
  • Include mypy_bootstrap.ini in PyPI packages (John Reese, PR 6252)

Internal Improvements and Code Cleanup

  • Fix daemon overlapped I/O when more data needs to be read (Ethan Smith, PR 6272)
  • Move most message constants to mypy.message_registry (Chad Dombrova, PR 6194)
  • Fix all DeprecationWarning: invalid escape sequence (Mickaël Schoentgen, PR 6195)
  • Add strictness flags (for mypy itself) that can be added with no source code changes (Michael Lee, PR 6237)
  • Some daemon performance improvements for large code bases

Typeshed Updates

  • Create stubs for Flask (Pascal Corpet, PR 2740)
  • Fix type of indent in JSONEncoder (Vield, PR 2737)
  • Make metavar in argparse be Optional (cormoran, PR 2739)
  • As of Python 3.6 dump_stats() method allows PathLike object to be passed (Igor Davydenko, PR 2741)
  • Add back StopIteration.value in Python 3 (Jelle Zijlstra, PR 2744)
  • Fix logging.getLevelName() type hints (Michael Noseworthy, PR 2730)
  • Add missing explicit Optional to stubs for the xml.etree package (Michael R. Shannon, PR 2734)
  • logging: inherit TimedRotatingFileHandler from Handler (Евгений, PR 2738)
  • Add SSLCertVerificationError fields (Hynek Schlawack, PR 2745)
  • Fix six.raise_from() value type (Frazer McLean, PR 2746)
  • builtins.pyi: Update __iadd__() and __imul__() in class list (Utkarsh Gupta, PR 2754)
  • pkg_resources: Add PKG_INFO str attribute for Distribution class (Joachim Jablon, PR 2775)
  • pkg_resources: fix stub for get_metadata_lines() (Joachim Jablon, PR 2776)
  • Add type annotation for collections.deque.__iadd__() (Joel Rosdahl, PR 2774)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Alan Du
  • Chad Dombrova
  • Ethan Smith
  • Joel Croteau
  • John Reese
  • Michael Lee
  • Mickaël Schoentgen
  • oremanj
  • Ryan Delaney
  • Sebastian Witowski
  • Wiktor Niesiobędzki

Additional thanks to all contributors to typeshed:

  • cormoran
  • Евгений
  • Frazer McLean
  • Hynek Schlawack
  • Igor Davydenko
  • Jelle Zijlstra
  • Joachim Jablon
  • Joel Rosdahl
  • Michael Noseworthy
  • Michael R. Shannon
  • Pascal Corpet
  • Utkarsh Gupta
  • Vield
— Guido van Rossum, on behalf of the mypy team