Friday, 1 March 2019

Extending mypy with plugins

Python is a famously dynamic language. It’s easy to write DSL-like frameworks that are hard to understand by static type checkers. Although with recent mypy features, such as protocols and literal types, and with basic metaclass and descriptor support, we can more often provide precise types, it’s still often hard to avoid false positives and negatives. To solve this problem, while avoiding the need for custom type system features for every framework, mypy supports a plugin system. Plugins are Python modules that provide callbacks (plugin hooks) that mypy will call when type checking classes and functions that interact with a library or framework. For example, such hooks can provide a more precise function return type that is otherwise hard to express, or auto-generate some methods in a class, to reflect the effects of a class decorator. To read more about plugin system architecture and for the full list of available hooks, see these docs.

Bundled plugins for standard library

Mypy comes with a default plugin for some builtin functions and classes, and the ctypes, contextlib, and dataclasses modules. It also includes a plugin for attrs (for historical reasons — it was the first third-party plugin written for mypy). These plugins allow mypy to infer more precise types and correctly type check code using these library features. To illustrate this, consider this snippet:

    from dataclasses import dataclass
    from typing import Generic, TypeVar
    
    @dataclass
    class TaggedVector(Generic[T]):
        data: List[T]
        tag: str
    
    position = TaggedVector([0, 0, 0], 'origin')

Above, get_class_decorator_hook() is called when analyzing the class definition. It adds autogenerated methods, including __init__(), to the class body. Mypy uses this generated constructor to correctly infer TaggedVector[int] as the type of position. As you can see, plugins work even with generic classes.

Here’s another snippet:

    from contextlib import contextmanager
    
    @contextmanager
    def timer(title: str) -> Iterator[float]:
        ...
    with timer(9000) as tm:
        ...

The get_function_hook() hook provides a precise return type for the contextmanager decorator, so that calls to the decorated function can be type-checked precisely. Mypy can now detect an error: the argument to timer() must be a string.

Combining plugins and stubs

In addition to relying on dynamic Python features, frameworks often have the challenge of having large APIs. Mypy needs stub files for the libraries to check code that uses those libraries (unless the library contains inline annotations, which is still often not the case). Distributing the stubs for large frameworks via typeshed is not very practical:

  • Typeshed has a relatively slow release cycle (it’s bundled with mypy).
  • Incomplete stubs can cause false positives that are hard to avoid.
  • There’s no easy way to mix and match stubs from different typeshed versions.
Stub packages, introduced in PEP 561, help with these issues:
  • Maintainers can release a stub package as often as they want.
  • Users who haven’t opted in to using the package won’t see any false positives.
  • You can freely install arbitrary versions of multiple different stub packages.

Moreover, pip allows combining stubs for a library and the corresponding mypy plugin into a single distribution. Stubs for a framework and a corresponding mypy plugin can now be easily developed and distributed together, which is often useful since plugins fill in missing or imprecise definitions in the stubs.

A recent example of such a package is SQLAlchemy stubs and plugin, the first public 0.1 release of which was published on PyPI earlier this month. Although the project is still in early alpha, we are already successfully using it at Dropbox to improve type checking precision. Currently the plugin understands basic ORM declarations:

    from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy import Column, Integer, String
    
    Base = declarative_base()
    
    class User(Base):
        __tablename__ = 'users'
        id = Column(Integer, primary_key=True)
        name = Column(String)

In the above snippet, the plugin uses the get_dynamic_class_hook() hook to tell mypy that Base is a valid base class, even though it doesn’t look like one. Then get_base_class_hook() is called on the definition of User, and it adds some autogenerated attributes. Next we instantiate the model:

    user = User(id=42, name=42)

The get_function_hook() hook is called, so that mypy can spot the error here: an integer is given instead of a user name.

The stubs define Column as a generic descriptor, so that attributes on a model get correct types:

    id_col = User.id  # Inferred type is "Column[int]"
    name = user.name  # Inferred type is "Optional[str]"

We welcome PRs that add more precise types to the stubs (the progress for core modules is tracked here).

Here are some gotchas that we found while working on the stubs:

  • Use a module-level __getattr__() to avoid false positives during the very early stages, when stubs are incomplete (this suppresses mypy errors due to missing module attributes). You can also use it in __init__.py files if some submodules are missing.
  • Descriptors often help with giving more precise types for customized attribute access (like in the Column example above), and it’s OK to use them even if the actual runtime implementation uses a more complex mechanism involving a metaclass, for example.
  • Don’t hesitate to declare framework classes as generic in stubs. Although they are not generic at runtime, this often allows us to give much more precise types for certain framework features, while runtime errors can easily be worked around. (We hope that frameworks will gradually add built-in support for generic types by explicitly inheriting relevant classes from typing.Generic.)

Recently released mypy plugins

There are already several plugins available for some popular Python frameworks. Apart from the above mentioned plugin for SQLAlchemy, other notable recent examples of packages with stubs and a bundled mypy plugin include stubs for Django and Zope Interfaces. All these projects are under active development.

Installing and enabling plugin + stub packages

Use pip to install a package with a mypy plugin and/or stubs in the virtual environment where mypy is installed:

    $ pip install sqlalchemy-stubs

Mypy should automatically discover the installed stubs. To enable the plugins you’ve installed, explicitly include them in your mypy.ini (or a custom config file):

    [mypy]
    plugins = sqlmypy, mypy_django_plugin.main 

Developing mypy plugins and writing stubs

If you want to develop a stubs + plugin package for a framework you use, you can use the sqlalchemy-stubs repository as a template. It includes the setup.py file, testing infrastructure using data-driven tests, and an example plugin class with a bunch of plugin hooks. We recommend using stubgen, an automatic stub generator that comes with mypy, to get started with stubs. Stubgen got several improvements in mypy 0.670.

For more details about mypy plugin system, see the docs. You can also just browse the source code of the plugins mentioned above. If you have any questions, feel free to ask them on the Python typing Gitter chat.

Friday, 8 February 2019

Mypy 0.670 Released

We’ve just uploaded mypy 0.670 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

(There are also new mypy-mypyc wheels; see the 0.660 release blog post.)

You can read the documentation for this release on Read the Docs.

New Feature: Variable Redefinition

As you might have experienced, PEP 484 and mypy require a variable to keep its initial type throughout a scope. Occasionally this can be annoying, as the traditional workaround is to rename the second (independent) use of the variable. We have introduced a new flag, --allow-redefinition, which relaxes this behavior in certain contexts. Here’s an example where this may be useful:

    def process(items: List[str]) -> None:
        # 'items' has type List[str]
        items = [item.split() for item in items]
        # 'items' now has type List[List[str]]
        ...

You can also enable this per module in the mypy.ini file using allow_redefinition = True. See the docs for the command-line flag and the config option.

Stubgen Improvements

We’ve done a fairly large overhaul of the stubgen utility, which can automatically generate draft stub files for almost any module or package:

  • Streamline the command-line interface; in particular, the source discovery/collection options now more closely resemble those of mypy itself: -m <module>, -p <package>, <file> or <directory>
  • Perform a lightweight semantic analysis to generate somewhat better stubs
  • Added documentation
  • When parsing types out of docstrings (this happens for C modules only), stubgen now handles overloaded signatures generated by pybind11 (Wiktor Niesiobędzki, PR 5975)

Other Improvements

  • Expand getting started docs to discuss type hints in more detail (Michael Lee, PR 6226)
  • Always infer in operator as returning bool (Joel Croteau, PR 5688)
  • Allow star args in ctypes.Array constructor (Alan Du, PR 6213)
  • Fix plugin invocation for __call__ methods (oremanj, PR 6334)
  • Implement the XDG directory spec for config files: $XDG_CONFIG_HOME/mypy/config is now included in the search path for config files (Ryan Delaney, PR 6304)
  • When using the --junit-xml flag, the Python version and platform in the junit.xml file are now formatted as mypy-py3_6-windows — previously this was mypy-py3.6-windows but the dot was misinterpreted by some tools (PR 6222)
  • Update the typed_ast dependency to version 1.3.1; this means we now officially support Python 3.7
  • Temporarily delete pyproject.toml from the repo in order to work around a pip bug (PR 6342)
  • Include mypy_bootstrap.ini in PyPI packages (John Reese, PR 6252)

Internal Improvements and Code Cleanup

  • Fix daemon overlapped I/O when more data needs to be read (Ethan Smith, PR 6272)
  • Move most message constants to mypy.message_registry (Chad Dombrova, PR 6194)
  • Fix all DeprecationWarning: invalid escape sequence (Mickaël Schoentgen, PR 6195)
  • Add strictness flags (for mypy itself) that can be added with no source code changes (Michael Lee, PR 6237)
  • Some daemon performance improvements for large code bases

Typeshed Updates

  • Create stubs for Flask (Pascal Corpet, PR 2740)
  • Fix type of indent in JSONEncoder (Vield, PR 2737)
  • Make metavar in argparse be Optional (cormoran, PR 2739)
  • As of Python 3.6 dump_stats() method allows PathLike object to be passed (Igor Davydenko, PR 2741)
  • Add back StopIteration.value in Python 3 (Jelle Zijlstra, PR 2744)
  • Fix logging.getLevelName() type hints (Michael Noseworthy, PR 2730)
  • Add missing explicit Optional to stubs for the xml.etree package (Michael R. Shannon, PR 2734)
  • logging: inherit TimedRotatingFileHandler from Handler (Евгений, PR 2738)
  • Add SSLCertVerificationError fields (Hynek Schlawack, PR 2745)
  • Fix six.raise_from() value type (Frazer McLean, PR 2746)
  • builtins.pyi: Update __iadd__() and __imul__() in class list (Utkarsh Gupta, PR 2754)
  • pkg_resources: Add PKG_INFO str attribute for Distribution class (Joachim Jablon, PR 2775)
  • pkg_resources: fix stub for get_metadata_lines() (Joachim Jablon, PR 2776)
  • Add type annotation for collections.deque.__iadd__() (Joel Rosdahl, PR 2774)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Alan Du
  • Chad Dombrova
  • Ethan Smith
  • Joel Croteau
  • John Reese
  • Michael Lee
  • Mickaël Schoentgen
  • oremanj
  • Ryan Delaney
  • Sebastian Witowski
  • Wiktor Niesiobędzki

Additional thanks to all contributors to typeshed:

  • cormoran
  • Евгений
  • Frazer McLean
  • Hynek Schlawack
  • Igor Davydenko
  • Jelle Zijlstra
  • Joachim Jablon
  • Joel Rosdahl
  • Michael Noseworthy
  • Michael R. Shannon
  • Pascal Corpet
  • Utkarsh Gupta
  • Vield
— Guido van Rossum, on behalf of the mypy team

Wednesday, 16 January 2019

Mypy 0.660 Released

We’ve just uploaded mypy 0.660 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes an optional compiled version, new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on Read the Docs.

Introducing an Optional Compiled Version of Mypy — mypy-mypyc

As part of our ongoing attempts to speed up mypy, we have built mypyc, an ahead-of-time compiler from type-annotated Python to CPython C extensions modules. We will be writing more about mypyc in the near future, but for now the main thing to know is that mypy can be compiled with mypyc, speeding it up by up to 3-4x!

To install a compiled mypy, instead of installing the mypy package, install the mypy-mypyc one (mypy-mypyc packages share version numbers with the corresponding mypy package):

    python3 -m pip uninstall -U mypy  # if mypy is installed
    python3 -m pip install -U mypy-mypyc

Binaries are available for 64-bit Windows, macOS, and Linux for Python 3.5 to 3.7.

Literal Types

This release introduces a new experimental kind of type: literal types. Literal types let you declare that some expression is equal to a specific primitive value. For example, if a variable is annotated with the type Literal["foo"], mypy will understand that variable is not only of type str, but is also equal to specifically the string "foo".

This feature is particularly useful when you want to annotate functions where the return type depends on the exact value provided. For example, suppose we have a function fetch_data(…) that returns bytes if the first argument is True, and str if it’s False. Previously, there was no good way of annotating the return type of fetch_data: the best we can do is Union[bytes, str]. With literal types, we can produce a more precise signature:

    from typing import overload
    from typing_extensions import Literal
    
    @overload
    def fetch_data(raw: Literal[True]) -> bytes: ...
    @overload
    def fetch_data(raw: Literal[False]) -> str: ...
    # Fallback overload if the user provides a regular bool
    @overload
    def fetch_data(raw: bool) -> Union[bytes, str]: ...
    
    def fetch_data(raw):
        # Implementation is omitted
        ...
    
    reveal_type(fetch_data(True))   # Revealed type is 'bytes'
    reveal_type(fetch_data(False))  # Revealed type is 'str'
Note 1: Literal types have some important nuances and limitations. For example, Literal[…] may contain only primitive literal values, and mypy will not attempt to understand complex expressions using literal types on a deep level. For more details, please see the documentation. Note 2: In order to use this feature, you must install the latest typing_extensions package, version 3.7.2:
    python3 -m pip install -U typing_extensions

Mypy Daemon Windows Support

Thanks to work from Ethan Smith, the mypy daemon, dmypy, is now fully supported on Windows!

Quick Mode Removed

The quick mode (--quick-and-dirty) has been deprecated for two mypy releases and has now been removed. The mypy daemon, dmypy, offers high speed without the correctness compromises of the quick mode.

Plugin Improvements

  • Add documentation for plugin system (PR 6057)
  • Make name lookup available to all plugin hooks (PR 6044)
  • Add more information to FunctionContext and MethodContext (Maxim Kurnikov, PR 5918)

Other Improvements and Notable Bugs Fixed

  • Introduce an optional sqlite backed incremental cache, enabled with --sqlite-cache (PR 6023)
  • Fix a daemon crash when there is a decode error (PR 6064)
  • Allow setting python_executable from config file (Ethan Smith, PR 5777)
  • Short-circuit if expression for always true/always false variables and MYPY/TYPE_CHECKING (Samer Masterson, PR 5965)
  • Don't map actual kwargs to formal *args (PR 6096)
  • Disable cache when producing reports (Maarten ter Huurne, PR 6076)
  • Fix issues with pointer arrays in the ctypes plugin (Jakub Stasiak, PR 6097)
  • Support kw_only=True in the attrs plugin (David Euresti, PR 6107)
  • Fix some daemon crash bugs (PR 6098)
  • Better error messages when __eq__ has unexpected signature (David Wobrock, PR 6106)
  • Collect additional timing stats and allow reporting them from the daemon (PR 6137)
  • Fix dmypy run when bad options passed to mypy (Ethan Smith, PR 6153)
  • Improve error messages from multiple inheritance compatibility checks (Maxim Kurnikov, PR 5926)
  • Fix an incremental mode crash that can occur in situations with import cycles and star imports (PR 6179)

mypy_extensions Moved to a Separate Repository

This is likely only of interest to downstream packagers, but the mypy_extensions module has been moved from the extensions subdirectory of the mypy repository to its own repository, https://github.com/python/mypy_extensions.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Chad Dombrova
  • Chris Philip
  • David Euresti
  • David Wobrock
  • Ethan Smith
  • Jakub Stasiak
  • Jared Hance
  • Jonathan Striebel
  • Maarten ter Huurne
  • Maxim Kurnikov
  • Michael Lee
  • Mykhailo Havelia
  • Samer Masterson
  • Shashank Parekh
  • Vincent Perez

Additional thanks to all contributors to typeshed:

  • Alexander Lyon
  • Alex Sarkesian
  • Andrew Gaul
  • Andrew Svetlov
  • Brandt Bucher
  • Dan Čermák
  • Daniel Mouritzen
  • Dave Halter
  • David Euresti
  • Diogo Magalhães Martins
  • Dominik Gabi
  • Ethan Smith
  • gnattishness
  • Hynek Schlawack
  • Ilya Konstantinov
  • Jared Hance
  • Jelle Zijlstra
  • Juan Gonzalez
  • Kai Willadsen
  • Kostya Esmukov
  • Michael Lee
  • Philipp Hahn
  • Sander Voerman
  • Savo Kovačević
  • Sebastian Rittau
  • Tomer Keren
  • Utkarsh Gupta
  • Ville Skyttä
— Michael J. Sullivan, on behalf of the mypy team

Friday, 7 December 2018

Mypy 0.650 Released

We’ve just uploaded mypy 0.650 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on Read the Docs.

Skipping the Rest of a File Using Asserts

You can now use certain always-false, unindented top-level assertions to tell mypy to skip the rest of the file. This allows you to skip type checking based on the target platform, for example:

    import sys

    assert sys.platform != 'win32'

    # The rest of this file doesn't work on Windows.

Some other expressions exhibit similar behavior; in particular, typing.TYPE_CHECKING, variables named MYPY, and any variable whose name is passed to --always-true or --always-false. (However, True and False are not treated specially!)

Mypy Daemon Improvements

  • Exit with status 2 from dmypy on irregular errors, and with 1 only if type checking finished but generated errors (PR 5982)
  • Work towards dmypy support on Windows:
    • Add initial support for dmypy on Windows (Ethan Smith, PR 5859)
    • Create new, hidden window when starting daemon on Windows (Ethan Smith, PR 5994)
  • Add --remove/--update flags to dmypy recheck (to allow Watchman integration, for example) (PR 5840, PR 5745)
  • Fix issues with # type: ignore in dmypy (PR 5861)
  • Fix daemon dependency handling for super() (PR 5793)
  • Speed up dmypy by reducing constant overhead (PR 5907)

Stubgen Improvements

  • Update stubgen to output async def and coroutine decorators (Bryan Forbes, PR 5845)
  • Improve stub files generated by stubgenc for pybind11 modules (Wiktor Niesiobędzki, PR 5850)
  • Improve stub generation so it works better with pybind11 wrappers (Eric Chang, PR 5814)
  • Make stubgen less aggressive about walking C extension modules (PR 5900)

Plugin Improvements

The mypy plugin system is still not documented or officially supported for third-party plugin development, but it’s available for brave early adopters.

  • Add plugin for ctypes.Array (dgelessus, PR 4869)
  • Fix ctypes plugin trying to look up bytes in Python 2 (dgelessus, PR 5924)
  • Fix dataclass inherited field ordering (Jakub Stasiak, PR 5877)
  • Invalidate cache when a plugin is added, removed or changed (PR 5878, PR 5892)
  • Add plugin hook for dynamic class definition (PR 5875)
  • Add options attribute to all plugin APIs (dgelessus, PR 5890)
  • Fix plugin hooks not being called correctly for dunder methods (dgelessus, PR 5700)
  • Fix issues with plugins in daemon mode (PR 5828)

Documentation Updates

  • Document ClassVar (PR 6005, PR 5733) (Joannah Nanjekye)
  • Improve documentation of ABCs (PR 6004)
  • Document what to do if a class is not generic at runtime (PR 5833)
  • Remove the “revision history” chapter, which was not very useful (PR 6018)

Other Improvements and Notable Bugs Fixed

  • Re-order module search path to put MYPYPATH first (Ethan Smith, PR 5776)
  • Stop suggesting MYPYPATH on missing imports (Michael Lee, PR 5950)
  • Add --disallow-any-generics to --strict (Joel Croteau, PR 5685)
  • Change all TypedDicts to be subtypes of Mapping[str, object] (PR 5933)
  • Type check TypedDict as caller kwargs (PR 5925)
  • Fix crash in PEP 561 module resolution (Ethan Smith, PR 6000)
  • Fix crash when reveal_locals() encounters a class definition (PR 5920)
  • Make descriptor assignments more like normal assignments (PR 5853)
  • Fix issue with module __getattr__ (PR 5893)
  • Fixes to reports: don't crash on control characters, ensure directories exist earlier (PR 5885)
  • Preserve tabs when creating XML reports (Maarten ter Huurne, PR 5945)
  • Create missing directories for junit XML output (Evan Kepner, PR 5827)
  • Fix constraint inference for non-invariant instances (PR 5817)
  • Use actual variable name for failed enum calls (PR 5810)
  • Support None.__bool__() (Daniel Darabos, PR 5780)
  • Avoid "No such file or directory" if module ancestor doesn't exist (PR 5785)
  • Add option to raise exception on internal errors (Joel Croteau, PR 5768)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Bryan Forbes
  • Chris Philip
  • Daniel Darabos
  • dgelessus
  • Eric Chang
  • Ethan Smith
  • Evan Kepner
  • Florian Bruhin
  • Jakub Stasiak
  • Joannah Nanjekye
  • Joel Croteau
  • Maarten ter Huurne
  • Michael Lee
  • Sebastian Rittau
  • Tomasz Walotek
  • Wiktor Niesiobędzki

Additional thanks to all contributors to typeshed:

  • Adam Simpkins
  • Andrew Svetlov
  • anentropic
  • Brandt Bucher
  • Daniel Li
  • Dave Halter
  • David Zbarsky
  • Dev Aggarwal
  • dgelessus
  • Diego Elio Pettenò
  • edgchen1
  • Ethan Smith
  • Fionn Fitzmaurice
  • Florian Bruhin
  • Gregory P. Smith
  • Grzegorz Śliwiński
  • Hannes Karppila
  • Jakub Stasiak
  • Jelle Zijlstra
  • Jerome Leclanche
  • Jared Hance
  • Joel Rosdahl
  • Joey Wilhelm
  • Keith Gray
  • Lourens Veen
  • Maarten ter Huurne
  • Marco Leogrande
  • Matt Gilson
  • MatthewPScott81
  • Maxim Kurnikov
  • Michael
  • Michael Lee
  • Michael R. Crusoe
  • Opal Symes
  • Pascal Corpet
  • Peter Pentchev
  • PRAJWAL M
  • Rebecca Chen
  • Ruud van Asseldonk
  • Sebastian Rittau
  • Semyon Proshev
  • Siva Chandra
  • Summon528
  • thautwarm
  • Utkarsh Gupta
— Jukka Lehtosalo, on behalf of the mypy team

Friday, 12 October 2018

Mypy 0.641 Released

THIS RELEASE REPLACES 0.640. The bug reported in issue 5784 has been fixed. There are no other changes.

We’ve just uploaded mypy 0.641 to the Python Package Index (PyPI). Mypy is an optional static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:
    python3 -m pip install -U mypy
You can read the documentation for this release on ReadTheDocs.

New Feature: Final Qualifiers

In this release we’re rolling out an exciting new (experimental) feature for the type system: final qualifiers.

Installation

Using this feature requires installing the latest typing_extensions package, version 3.6.6:
    python3 -m pip install -U typing_extensions

Final variables

You can declare a variable “final” using the Final qualifier, like so:
    from typing_extensions import Final
    RATE: Final = 3000
    # ...
    RATE = 300  # error: Cannot assign to final name "RATE"
This also works for local variables, class variables and instance variables. For class variables it implies that subclasses cannot override the value either. You can also use a type comment (# type: Final).
Note that “final” variables may have a mutable type, and then they can still be mutated:
    PLATFORMS: Final = ['mac', 'windows']
    PLATFORMS.append('linux')  # OK

Final methods and classes

You can also declare methods “final”, meaning they cannot be overridden by a subclass:
    from typing_extensions import final  # Note lower-case 'f'
    class Base:
       @final
       def update(self) -> None:
           ...
    class Derived(base):
        def update(self) -> None:  # error: Cannot override final attribute "update"
            ...
The @final decorator can also be used for classes, where it means they cannot be subclassed.

Note

Final qualifiers only have an effect during type checking, in type-checked code; they don’t do anything at runtime.

Documentation

The full documentation has additional details about this feature.

New Feature: Allow Omitting Return Type for __init__

It is now possible to omit the return type of an annotated __init__ method without getting an error message. For example:
    class Circuit:
        def __init__(self, voltage: float):
            self.voltage = voltage
In previous mypy versions this would elicit an error message:
    error: The return type of "__init__" must be None
This error was just annoying as the only legal return declaration for __init__ is -> None, so we’ve removed it. Note that this only works if there’s at least one annotated argument! For __init__ methods without arguments you must still add -> None, otherwise the method will be treated as untyped, and its body will not be type checked at all. Examples:
    class UntypedExample:
        # This method is not type-checked at all!
        def __init__(self):
            self.voltage = 0.0
    
    class TypedExample:
        # This is how to ensure that a 0-argument __init__ is type-checked:
        def __init__(self) -> None:
            self.voltage = 0.0

New Feature: Namespace Package Support

Python 3 supports packages without __init__.py files, better known as namespace packages. These were introduced by PEP 420 as a mechanism for splitting a single Python package across multiple directories. There are some subtleties in their semantics (e.g. a package with __init__.py file are preferred even if they occur later on sys.path), and until this release mypy did not support them. Support is now provided using the command-line flag --namespace-packages (or the correponding config file phrase namespace_packages = True). Note that the default is still off. At some point in the future the default may change to on for Python 3. (Python 2 doesn’t support namespace packages, though mypy will let you combine this flag with --py2.)

Other Flags and Options Changes

  • New flag: --allow-untyped-globals suppresses errors caused by not being able to fully infer the types of global and class variables (PR 5670)
  • Deprecated flag: --quick-and-dirty ; it will be removed in a future release (PR 5737)
  • Removed flags, all of which have been deprecated for a long time: --disallow-any, -f/--dirty-stubs, --use-python-path, -s/--silent-imports, --almost-silent, --[no-]fast-parser, --strict-boolean (for the latter, see discussion in issue 3195) (PR 5740)
  • The -2/--py2 flag’s behavior now matches --python-version (sqwishy, PR 5619)
  • New experimental flag: --fast-exit makes mypy exit faster (by skipping final GC pass) (PR 5569)

Other Improvements and Notable Bugs Fixed

  • Get rid of Optional when assigning Any in an is None branch (PR 5629)
  • Don't crash on aliases of the form C = C (PR 5632)
  • Fix meet for tuple with instance (PR 5641)
  • Add missing fine-grained dependencies when using Type[X] (PR 5607)
  • Make NamedTuple provide __new__ instead of __init__ (PR 5643)
  • Fully analyze named tuple subclasses in third pass (PR 5644)
  • Defer subclass methods if superclass has not been analyzed to fix issues with checking method overrides in import cycles (PR 5637)
  • Fix type object signature when both __new__ and __init__ present (PR 5642)
  • Fix crash related to decorated functions (PR 5654)
  • Fix “Invalid type” errors in import cycles (PR 5635)
  • Make classmethod's first argument be Type[...] and fix __init__ defined after a class method (PR 5646)
  • Allow comparisons to refine Optional types without strict-optional (PR 4523)
  • Fix untyped decorator check for class instances (Daniel Izquierdo, PR 5509)
  • Fix bug with inferring bad arguments to overloads (Michael Lee, PR 5660)
  • Add the current directory to sys.path when running stubgen (PR 5673)
  • Allow returning inferred None from functions (PR 5663)
  • Support stubs marked with py.typed in PEP-420 nested packages (Chris Philip, PR 5591)
  • Don't report "function does not return a value" in unannotated functions (PR 5683)
  • Make return type implicitly None for type checked __init__ and __init_subclass__ (Ekin Dursun, PR 5677)
  • Improve usage of outer context for inference (PR 5699)
  • Reformat and reorganize the config file docs (Michael Lee, PR 5595)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.
Thanks to all mypy contributors who contributed to this release:
  • ceh
  • Chris Philip
  • Daniel Izquierdo
  • Ekin Dursun
  • Elazar Gershuni
  • Ethan Smith
  • Jan Teske
  • Joel Croteau
  • Michael Lee
  • sqwishy
Additional thanks to all contributors to typeshed:
  • Adam Dangoor
  • Benjamin Peterson
  • Christian Haudum
  • Dmitry Shachnev
  • Emil Hessman
  • Grzegorz Śliwiński
  • Israel Tsadok
  • Jan Teske
  • Jelle Zijlstra
  • Jon Dufresne
  • kitsuyui
  • Manuel Vázquez Acosta
  • Martin DeMello
  • Matt Robinson
  • Matthew Christopher
  • Michael R. Crusoe
  • Nathaniel Manista
  • Ran Benita
  • Rebecca Chen
  • Richard Levasseur
  • Ruben Berenguel
  • Scott Colby
  • Sebastian Kreft
  • Sebastian Rittau
  • Siva Chandra
  • spdkils
  • Teddy Sudol
  • Wim L
  • Zac Hatfield-Dodds
— Guido van Rossum, on behalf of the mypy team
















Monday, 17 September 2018

Mypy 0.630 Released

We’ve just uploaded mypy 0.630 to the Python Package Index (PyPI). Mypy is an optional static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:
    python3 -m pip install -U mypy
You can read the documentation for this release on ReadTheDocs.

New Feature: Callback Protocols

You can now use protocols to specify callable types. In some situations it was hard (or even impossible) to express types of complex callbacks using the Callable[[Arg, ...], Ret] syntax. Examples include overloaded callbacks, complex generic callbacks, or just callables that don’t accept a fixed number of positional arguments. You can define a callback protocol for situations like these — it’s a protocol with only __call__ defined:
    from typing import Optional, Iterable, List
    from typing_extensions import Protocol
    
    class Combiner(Protocol):
        def __call__(self, *vals: bytes, 
                     maxlen: Optional[int] = None) -> List[bytes]: ...
    
    def batch_proc(data: Iterable[bytes],
                   cb_results: Combiner) -> bytes:
        ...        
    def good_cb(*vals: bytes,
                maxlen: Optional[int] = None) -> List[bytes]:
        ...
    batch_proc([], good_cb)  # OK
For more information see the docs.

Other Improvements and Notable Bugs Fixed

  • Support classes and type aliases in plugin hooks (PR 5379)
  • Treat IntEnum type object as iterable (Elazar Gershuni, PR 5388)
  • Improve Tuple/Sequence/Iterable type overlap checks (Ethan Smith, PR 5315)
  • Improve checks for overlapping types (Michael Lee, PR 5476)
  • Improve error message for overload overrides with swapped variants (Michael Lee, PR 5369)
  • Add support for optional converters in attrs plugin (David Euresti, PR 5411)
  • Allow calling Type[T] where T has a generic bound (Elazar Gershuni, PR 5309)
  • Support additional arguments to namedtuple() (Jelle Zijlstra, PR 5215)
  • Fix crash related to decorated functions (PR 5427)
  • Fix daemon crash related to named tuples (PR 5429)
  • Avoid infinite recursion when attempting to define recursive types (PR 5434)
  • Don't treat non-method functions whose name resembles a reverse operator (e.g. __radd__) as such (Sebastian Rittau, PR 5421)
  • Allow plugin registration in mypy.ini using module names or entry points (Chad Dombrova, PR 5358)
  • Invalidate cache when a previously missing stub is added (PR 5465)
  • Fix issues related to reverse operator methods (Michael Lee, PR 5475)

Documentation Fixes and Internal Improvements

  • Remove runtests.py in favor of pytest (Elazar Gershuni, PR 5274)
  • Improve stub testing to cover all of typeshed (Ethan Smith, PR 5051)
  • Reorganize command line documentation (Michael Lee, PR 5333)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team. Thanks to all mypy contributors who contributed to this release:
  • Alex Jurkiewicz
  • Anthony Sottile
  • Batuhan Osman Taşkaya
  • Bernát Gábor
  • Bruce Merry
  • Chad Dombrova
  • David Euresti
  • Elazar Gershuni
  • Elias Zamaria
  • Emil Hessman
  • Ethan Smith
  • Georg Molau
  • Janus Troelsen
  • Jelle Zijlstra
  • Matvey Tolstolytski
  • Michael Lee
  • Sebastian Rittau
  • William Schwartz
  • ceh
  • snarkmaster
Additional thanks to all contributors to typeshed:
  • Amol Bhave
  • Anthony Sottile
  • Brandon Lin
  • Bruce Merry
  • Chelsea Voss
  • Daniel Li
  • David Euresti
  • David Zbarsky
  • Devin Fee
  • Dmitry Shachnev
  • Dominik Gabi
  • EFanZh
  • Emil Hessman
  • Ethan Smith
  • Gary van der Merwe
  • Goldstein
  • Hynek Schlawack
  • Ilya Konstantinov
  • Jelle Zijlstra
  • Linda_pp
  • Martin DeMello
  • Matt Gilson
  • Max Murin
  • Michael Lee
  • MinJune Kim
  • Ollie Ford
  • Olmo Kramer
  • Omar Sandoval
  • Philipp Hahn
  • Ran Benita
  • Rebecca Chen
  • Sebastian Kreft
  • Sebastian Rittau
  • Siva Chandra
  • Stig Johan Berggren
  • Teddy Sudol
  • Tomasz Trębski
  • Ville Skyttä
  • Yusuke Miyazaki
  • Zac Hatfield-Dodds
  • Zsolt Dollenstein
  • justinpawela
  • stevenjackson121
  • tikki
— Ivan Levkivskyi, on behalf of the mypy team

Friday, 13 July 2018

Mypy 0.620 Released

We’ve just uploaded mypy 0.620 to the Python Package Index (PyPI). Mypy is an optional static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:
    python3 -m pip install -U mypy
You can read the documentation for this release on ReadTheDocs.

New Features

Support for data classes in Python 3.7

The recently released Python 3.7 added a new module dataclasses that allows writing simple boilerplate-free classes. Mypy now supports this new feature:
    from dataclasses import dataclass
    from typing import List
    
    @dataclass
    class FitResult:
        optimum: List[float]
        chi: float
        method: str = "TRF"
    
    FitResult([0.1, 0.2], 1.2)  # OK
    FitResult([0.1, 0.2], 1.2, "LM")  # Also OK
    FitResult(1, 2)  # Error!
Note: there are some limitations in supported features — see the docs. (Contributed by Bogdan Popa in PR 5010.)

Overloads on generic types and other overload improvements

Mypy previously rejected certain patterns involving overloaded functions, in particular defining overloads on generic types, and calling overloads on union and optional types. These (and a few others) are now supported. In addition, the error messages for overloads are now more detailed:
    from typing import List, Union, overload
    
    @overload
    def summarize(data: List[int]) -> float: ...
    @overload
    def summarize(data: List[str]) -> str: ...
    def summarize(data):
        # Implementation goes here
        ...
    
    gen_data: Union[List[int], List[str]]
    res = summarize(gen_data)  # OK, inferred type is Union[float, str]
    
    bad_data: int
    summarize(bad_data)
    # error: No overload variant of "summarize" matches argument type "int"
    # note: Possible overload variants:
    # note:     def summarize(data: List[int]) -> float
    # note:     def summarize(data: List[str]) -> str
See the updated docs for more details. (Contributed by Michael Lee.)

Incomplete and partial packages

Writing complete stubs for an existing large library may be hard and sometimes impractical. To allow gradual improvements in library stubs without generating spurious errors, two mechanisms are now supported. Adding a __getattr__ function to __init__.pyi indicates that the corresponding package (or subpackage) is incomplete, thus silencing Missing library stub errors for this package:
    # pack/__init__.pyi
    from typing import Any
    def __getattr__(arrr: str) -> Any: ...
    
    # pack/subpack/__init__.pyi
    # empty
    
    # pack/subpack/mod.pyi
    class Test: ...
    
    # main.py
    from pack import other  # OK, pack is incomplete
    other.func(1, 2)  # OK, all types in incomplete packages are Any
    
    from pack.subpack import mod  # OK
    from pack.subpack import another  # Error: missing library stub file
In addition, a PEP 561 stub package can declare itself as partial, allowing fallbacks to other sources of typing information such as inline annotations and typeshed stubs. See PEP 561 for the details. (Contributed by Ethan Smith.)

Other Improvements and Notable Bugs Fixed

  • Fix running mypy from editable install directory (Ethan Smith, PR 5381)
  • Support egg/setuptools packages for PEP 561 searching (Ethan Smith, PR 5282)
  • Refactor and reorder search path to make it compliant with PEP 561 (Ethan Smith, PR 5256)
  • Silence errors in modules in site-packages and typeshed (Ethan Smith, PR 5303)
  • Fix two option handling bugs in dmypy (PR 5172)
  • Fix a daemon crash bug (PR 5285)
  • Fix incorrect handling of attrs attributes with init=False and default (David Euresti, PR 5154)
  • Various additional overload bug fixes (PR 5236, PR 5254, PR 5224, PR 5163, PR 5166) (Michael Lee)
  • Several module __getattr__ fixes (PR 5332, PR 5306, PR 5295, PR 5292)

Internal Improvements

  • Test suite cleanups (PR 5142, PR 5271) (Elazar Gershuni)
  • Improve PEP 561 testing infrastructure (PR 5060, PR 5225, PR 5237) (Ethan Smith)
  • Support for PyCharm test debugging, and add tox environment setup (Bernát Gábor, PR 5189)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team. Thanks to all mypy contributors who contributed to this release:
  • Alex Tereshenkov
  • Bogdan Popa
  • David Euresti
  • Elazar Gershuni
  • Emil Hessman
  • Ethan Smith
  • Gábor Bernát
  • Herst
  • Jelle Zijlstra
  • Julian Ospald
  • Michael Lee
  • Nate White
  • Sebastian Rittau
Additional thanks to all contributors to typeshed:
  • Andrew Svetlov
  • Anthony Sottile
  • Bertrand Bonnefoy-Claudet
  • Cyril Jouve
  • Daniel Li
  • Ethan Smith
  • Froger David
  • George King
  • Hanaasagi
  • Ilya Konstantinov
  • Jason Fried
  • Jelle Zijlstra
  • John Reese
  • Josh Holland
  • Martin DeMello
  • Mathieu Leduc-Hamel
  • Max
  • Max R
  • Michael
  • Michael Hirsch, Ph.D
  • Michael Lee
  • NAKAMURA Yoshitaka
  • Nipunn Koorapati
  • Philipp Hahn
  • Rebecca Chen
  • Robert Collins
  • Roy Williams
  • Scott Belden
  • Sebastian Rittau
  • Sekou Diao
  • Stephen Thorne
  • Steven Karas
  • Sushain Cherivirala
  • Yusuke Miyazaki
  • mbarkhau
  • potykion
  • strager
— Ivan Levkivskyi, on behalf of the mypy team