Friday, 29 November 2019

Mypy 0.750 Released

We’ve just uploaded mypy 0.750 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes many features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

More Powerful Self-types

This version adds support for more patterns involving explicit annotations on self argument in methods. In particular, the mixin pattern got basic support via a protocol as a self annotation that specifies what the mixin expects from any class that subclasses it:

    class Lockable(Protocol):
        @property
        def lock(self) -> Lock: ...
    
    class AtomicCloseMixin:
        def atomic_close(self: Lockable) -> int:
            with self.lock:  # This is OK
                ... # perform some actions
    
    class File(AtomicCloseMixin):
        def __init__(self) -> None:
            self.lock = Lock()
    
    class Bad(AtomicCloseMixin):
        pass
    
    f = File()
    b = Bad()
    f.atomic_close()  # OK
    b.atomic_close()  # Error: Invalid self type for "atomic_close"

Another somewhat common pattern that got support via explicit self-types is overloaded generic constructors (in particular in library stubs):

    from typing import Any, AnyStr, Generic
    from typing_extensions import Literal
    
    class File(Generic[AnyStr]):
        @overload
        def __init__(self: File[bytes], raw: Literal[True]) -> None: ...
        @overload
        def __init__(self: File[str], raw: Literal[False]) -> None: ...
        @overload
        def __init__(self: File[Any], raw: bool = ...) -> None: ...
    
    reveal_type(File())  # type is File[Any]
    reveal_type(File(raw=True))  # type is File[bytes]

Read the docs for more examples and use cases.

Stubgen Revamped

Stubgen, the automatic stub generator that comes with mypy, has many fixes and improvements. In particular:

  • Recover from various errors that broke stub generation previously, such as inconsistent MROs or modules that crash on import.
  • Export imported names that aren't referenced in the module.
  • Export all names imported from the current package by default. Use --export-less to disable this behavior.
  • Fix various issues that resulted in syntactically invalid stubs being generated.
  • Fix signatures of various C extension methods.
  • Make the docstring parser more robust.
  • Add various heuristics to skip internal modules, such as tests and vendored packages.
  • Allow tweaking the level of logging. Add --verbose and --quiet flags.

Generating stub files has never been so easy and robust! Please read more docs here and try it on the libraries you maintain or use that don’t yet have stubs in Typeshed. Please consider contributing any stubs you create to Typeshed so that the community can share and improve them.

Mypy Daemon is No Longer Experimental

The mypy daemon has been the default way of running mypy locally at Dropbox for well over a year and it's quite stable. It also got several additional commands and flags for better customization (and existing flags got better documentation). In particular, mypy daemon now supports basic integration with external file system event watchers for even faster, sub-second type checking results over tens of thousands of files.

Use dmypy recheck --update FILE and dmypy recheck --remove FILE to directly tell the daemon which files got edited/deleted to avoid waiting for the built-in file system watcher. For more details, see the docs.

Static Inference of Annotations (Experimental)

The mypy daemon now supports (as an experimental feature) statically inferring a draft type annotation for a given function or method. Running dmypy suggest FUNCTION will produce a suggested signature in this format:

    (param_type_1, param_type_2, ...) -> ret_type

This is a low-level command that can be used by editors, IDEs, and other tools, such as the mypy plugin for PyCharm, to propose an annotation to the user, and/or to insert an annotation to a source file.

In this example, the function format_id() has no annotation:

    def format_id(user):
        return "User: {}".format(user)
    
    root = format_id(0)

Mypy uses call sites and return statements to infer that format_id() accepts an int and returns a str. For more details, see the docs.

Other Notable Improvements and Bug Fixes

  • Show closest candidates for misspelled module attributes (Theodore Liu, PR 7971)
  • Fix C string encoding in the mypyc compiler (Saleem Rashid, PR 7978)
  • Fix type-checking decorated generic methods (beezee, PR 7933)
  • Make revealed type of final variables distinct from non-final ones (Michael Lee, PR 7955)
  • Fix issubclass() to narrow down types of type variables (Xuanda Yang, PR 7930)
  • Refine types of parent expressions when narrowing expression types (Michael Lee, PR 7917)
  • Fix starting compiled mypy daemon on Windows (PR 7929)
  • Show closest candidates for misspelled keyword arguments (Martijn Wuis, PR 7888)
  • Fix type checking of ternary expressions with an empty collection (Xuanda Yang, PR 7892)
  • Support negative integers in literal types (Xuanda Yang, PR 7878)
  • Make sure type inference works for literals against protocols (PR 7882)
  • Don't add coercions when compiling expression statements via mypyc (jag426, PR 7872)
  • Fix crashes in mypy daemon when sorting union items (PR 7842)
  • Enable whole-file # type: ignore comments in Python 2.7 (Brandt Bucher, PR 7789)
  • Record internal Any types as precise in HTML report generator (ahmadF, PR 7708)
  • Add misc/dump-ast.py script to dump parse trees (Michael Wayne Goodman, PR 7820)
  • Use union component as a self-type in class methods (PR 7822)
  • Correctly handle union types in string interpolation checker (Vincent Barbaresi, PR 7809)
  • Fix capturing stderr in mypy.api.run (David Tucker, PR 7804)
  • Fix internal error when checking dataclasses with a duplicate field (Vincent Barbaresi, PR 7808)
  • Add cross references to mypy command line options in the documentation (Oleg Höfling, PR 7784, PR 7801, PR 7802)
  • Improve missing import error message (Lewis Cowles, PR 7698)
  • Add option to print absolute file paths (Maxim Koltsov, PR 7754)
  • Recommend using string escaping for classes that are non-generic at runtime (Rubin Raithel, PR 7707)
  • Fix accessing unannotated implicit class methods (PR 7739)
  • Ensure that non-total TypedDicts can be false (henribru, PR 7745)
  • Fix type checking __init_subclass__() (PR 7723)

Plugin API Changes and Improvements

This release includes three major mypy plugin API changes:

  • There is a new plugin hook for specifying per-module configuration data: report_config_data(). It can be used if the plugin has some sort of per-module configuration that can affect type checking.
  • *(breaking change)* New TypeAliasType type has been added and is now used for type aliases instead of eagerly expanding them. To get the expanded type use mypy.types.get_proper_type(). See this comment for more compatibility instructions.
  • *(breaking change)* Symbol node methods .fullname() and .name() were replaced with properties. See this comment for compatibility instructions.

These improvements are related to plugins:

  • Invoke the get_dynamic_class_hook() plugin hook also on method calls (Seth Yastrov, PR 7990)
  • Fix plugin invocation on class method calls through Type[...] (PR 7969)
  • Allow new plugins to be added by programs (like mypyc) that directly invoke build (PR 7875)

For more information on plugin hooks see the docs. Breaking API changes are announced in this GitHub issue. Please subscribe to it if you are an active mypy plugin developer.

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • ahmadF
  • Alexandre Viau
  • beezee
  • Bob Whitelock
  • Brandt Bucher
  • Chad Dombrova
  • Daniel Hahler
  • David Euresti
  • David Tucker
  • Djedouas
  • Ekin Dursun
  • Emmanuel Nosa Evbuomwan
  • Ethan Smith
  • Gábor Lipták
  • henribru
  • hoefling
  • Ilaï Deutel
  • jag426
  • Lewis Cowles
  • Maxim Koltsov
  • Michael Lee
  • Michael R. Crusoe
  • Michael Wayne Goodman
  • Rubin Raithel
  • Saleem Rashid
  • Sebastian Rittau
  • Seth Yastrov
  • TH3CHARLie
  • Theodore Liu
  • Thomas Hisch
  • Tim Gates
  • Tudor Brindus
  • Tuomas Suutari
  • Vasily Zakharov
  • Vincent Barbaresi
  • Wuisch
  • Xuanda Yang
  • Zac Hatfield-Dodds

Additional thanks to all contributors to typeshed:

  • Alexander Schlarb
  • Angela Ambroz
  • Anthony Sottile
  • Árni Már Jónsson
  • Benjamin Peterson
  • bianca rosa
  • Brett Cannon
  • Christopher Hunt
  • cptpcrd
  • Daniel Hahler
  • David Tucker
  • Denis Laxalde
  • Diego Elio Pettenò
  • Dima Boger
  • Eric N. Vander Weele
  • Florian Bruhin
  • Greg Ward
  • Ilaï Deutel
  • Jakub Stasiak
  • Jelle Zijlstra
  • Jeremy Lainé
  • Jon Dufresne
  • JR Heard
  • Lawrence Chan
  • Maarten ter Huurne
  • Markus Bauer
  • Martijn Pieters
  • Martin DeMello
  • Michael Lee
  • Michael Seifert
  • Michał Słapek
  • Nathaniel Brahms
  • Nikita Sobolev
  • Nikola Forró
  • Ran Benita
  • Rebecca Chen
  • Rune Tynan
  • Ryan Morshead
  • Sebastian Rittau
  • Taneli Hukkinen
  • Trim21
  • Utkarsh Gupta
  • Vasily Zakharov
  • 秋葉

Wednesday, 16 October 2019

Mypy 0.740 Released

Mypy 0.740 Released

We’ve just uploaded mypy 0.740 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes many features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Type Checking for str.format Calls

Calls to the format method on string literals are now type-checked based on the rules of the format string syntax and the format specification mini-language (PR 7418)

This also fixes some corner cases in % formatting and allows flagging bad interactions between bytes and strings.

Here are some errors mypy can now catch:

  • Likely accidental formatting of bytes ("{}".format(b'123') will produce "b'123'" on Python 3 which is likely unintended — the !r specifier should be used when it is intended)
  • Invalid format string syntax
  • Invalid conversion type or specifier
  • Incorrect number of arguments or field names (One thing this will complain about is too many arguments for a format string—this does not cause an exception at runtime but is still usually a bug.)
  • Type mismatches for numeric formatting
  • Type errors in index or attribute expressions

Improved check_untyped_defs

The self argument to methods without annotations is now correctly given the object’s type (instead of Any) when using check_untyped_defs (PR 7530).

This allows substantially better checking of untyped methods as well as enabling inference of attribute types in some situations where they would not have been determined in the past.

Additionally, we now suppress most “needs type annotation” errors caused by assigning empty collections (i.e, data = []), in untyped functions.

More Precise Error Locations (Breaking Change)

Reported line and column numbers are now more precise; in particular, incompatible assignment errors are now reported as being an error with the right hand side rather than the left hand side (PR 7578, PR 7569).

In some cases these changes can break # type: ignores, but from our experience they don’t do it too often.

Stricter Treatment of Context Manager Objects

Mypy 0.730 added support for using Literal types in the return type from __exit__ to determine whether it is possible for a with statement to swallow exceptions, which can affect whether mypy thinks code after a with statement is reachable.

To better pinpoint the source of issues caused by this, mypy now gives an error when __exit__ methods are declared to return bool but always return False. See the documentation for more information.

Documentation and Other Error Message Improvements

  • Add many cross-references to standard library to mypy docs (Oleg Höfling)
  • Update examples of mutable and non-mutable mapping in cheat sheet (Marco, Gorelli, PR 7620)
  • Fix syntax errors in code examples (Oleg Höfling, PR 7651)
  • Always report dmypy crashes to the client (ideally this fix will affect nobody, but…. 😉 , PR 7593)
  • Fix some notes not being ignored when using # type: ignore[code] (PR 7573)
  • Check syntax errors in # type: ignore[code] comments (PR 7460)
  • Fix incorrect error message when a positional argument is missing in a function with multiple default arguments (PattenR, PR 7547)
  • Fix message when second argument to isinstance is an alias to Any (Cohen Karnell, PR 7504)

Other Notable Improvements and Bug Fixes

  • Only return a declared type from __new__ if it is a subtype (PR 7372)
  • Give better types for pathlib.Path.open (Michael Lee, PR 7643)
  • Fix a crash on missing self with --check-untyped-defs (PR 7669)
  • Treat __class_getitem__ as an implicit class method, similar to __init_subclass__ (ahmadF, PR 7647)
  • Allow TypedDict key with literal type during construction (PR 7645)
  • Support NamedTuple class syntax in all stubs (Sebastian Rittau, PR 7646)
  • Support eq and order arguments to attr.s (David Euresti, PR 7619)
  • Fix some false positives when checking raise statements (Patrick Williams, PR 7515)
  • Fix spurious errors with inherited dataclasses in incremental mode (PR 7596)
  • Fix a crash in mypy daemon (PR 7543)
  • Type-check class keyword in __init_subclass__ calls (g.denis, PR 7452)
  • Fix crash related to __getattr__ and __call__ (Ekin Dursun, PR 7446)
  • Fix incremental mode crashes (PR 7499)

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • ahmadF
  • Andrzej Górski
  • Brad
  • Cohen Karnell
  • David Euresti
  • Ekin Dursun
  • g.denis
  • hoefling
  • Jelle Zijlstra
  • John Reese
  • Marco Gorelli
  • Michael Lee
  • Michael R. Crusoe
  • Patrick Williams
  • PattenR
  • Sebastian Rittau

Additional thanks to all contributors to typeshed:

  • Alan Du
  • Alex Garbutt
  • Alex Jurkiewicz
  • Andrew Svetlov
  • Anthony Sottile
  • Árni Már Jónsson
  • Augusto Hack
  • Benjamin Peterson
  • Brad
  • brianjyoung
  • Carl Meyer
  • Colin Gilgenbach
  • Eric Traut
  • Glyph
  • herr kaste
  • hh-h
  • Jelle Zijlstra
  • Jeremy Lainé
  • jjmortensen
  • Lukáš Lalinský
  • Parth Sharma
  • Philippe F
  • plokmijnuhby
  • Rafi Blecher
  • Rebecca Chen
  • robertschweizer
  • Rune Tynan
  • Russ Allbery
  • Ryan Morshead
  • Sebastian Rittau
  • sinancepel
  • Stefan T
  • Utkarsh Gupta
  • Vasily Zakharov
  • wouter bolsterlee
  • 秋葉

Thursday, 26 September 2019

Mypy 0.730 Released

We’ve just uploaded mypy 0.730 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes many features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

More Precise Error Locations (Breaking Change)

If you call a function with an incompatible argument type, mypy now points the error message to the argument with the incompatible type. Previously mypy pointed to the function being called, which could be confusing in multi-line calls.

Note: You may have to move some # type: ignore comments as a result of this change, since their behavior depends on the line number of reported errors.

Error Codes

Mypy can now optionally display error codes. They are shown within square brackets after each error message:

    prog.py:24: error: "str" has no attribute "trim"  [attr-defined]

Enable error codes using --show-error-codes (or show_error_codes = True in a configuration file). See the documentation for more information (documentation on individual codes here and here).

Ignoring Specific Error Codes

You can ignore only errors with specific error codes on a particular line by using a # type: ignore[code, ...] comment. This reduces the risk of ignoring unexpected, serious errors when using # type: ignore comments that ignore (almost) all possible errors on a line.

See the documentation for more details.

Colors in Output

Mypy now uses colored, more user-friendly output by default. Example:

  $ mypy mypy
  mypy/argmap.py:12: error: Incompatible return value type (got 
    "List[int]", expected "List[Optional[int]]")
  mypy/argmap.py:12: note: "List" is invariant -- see 
    http://mypy.readthedocs.io/en/latest/common_issues.html#variance
  mypy/argmap.py:12: note: Consider using "Sequence" instead, 
    which is covariant
  mypy/checkstrformat.py:238: error: Argument 1 to 
    "checkers_for_type" has incompatible type "str"; expected "bytes"
  Found 2 errors in 2 files (checked 154 source files)
  
  $ mypy mypy
  Success: no issues found in 154 source files

You can use --no-color to disable colored output. You can use --no-error-summary to hide the summary line with the number of errors.

Windows support was contributed by Ethan Smith (requires Windows 10).

Pretty Output Mode

You can use --pretty to display each line which had errors and a caret that points to the location of the error on each line. Example of pretty output:

  $ mypy --pretty mypy
  mypy/checkstrformat.py:238: error: Argument 1 to "checkers_for_type" 
  has incompatible type "str"; expected "bytes"
                  c = self.checkers_for_type(specifier.type, ...
                                             ^
  Found 1 error in 1 file (checked 154 source files)

This mode also wraps long error messages at word boundaries to improve readability.

Old Semantic Analyzer Removed

This release no longer includes the old semantic analyzer. See mypy 0.720 release notes for information on the new semantic analyzer.

Reachability and Context Managers

The --warn-unreachable option now behaves more correctly with “exception-swallowing” context managers. If a context manager is currently declared to return bool but it never swallows exceptions, you should annotate the return of __exit__ as Literal[False] instead of bool, or otherwise mypy may complain about missing return statements. Example:

    from typing import Optional, Type
    from types import TracebackType
    from typing_extensions import Literal

    class Resource:
        ...
        def __exit__(self,
                     exc_type: Optional[Type[BaseException]],
                     exc_val: Optional[BaseException],
                     exc_tb: Optional[TracebackType]) -> Literal[False]:
            ...
            return False

This was contributed by Michael Lee (PR 7317).

Miscellaneous New Features

  • Support PEP 572: Assignment expressions (a.k.a. the “walrus operator”) (Jelle Zijlstra, PR 6899)
  • Basic support for the PEP 593 draft: typing_extensions.Annotated (Brandt Bucher, PR 7292)
  • Don't re-export star-imported names when implicit re-exports are disabled (Ran Benita, PR 7361)
  • Honor the return type of __new__ (PR 7188)
  • Support recursive traversal into PEP 420 namespace packages (Barry Warsaw, PR 7219)
  • Support expansion of home directories (~) and environment variables in all path configuration options (hetmankp, PR 7022, PR 7273)
  • Cache directory may be set via environment (hetmankp, PR 7016, PR 7443)
  • Make type formatting use fully qualified names whenever there is ambiguity (PR 7226)
  • Infer better types when concatenating fixed-length tuples (Ekin Dursun, PR 7409)

Improvements to Strict Equality Checking

  • Apply --strict-equality special-casing for bytes also to bytearray (PR 7473)
  • Make --strict-equality stricter with literals (PR 7310)
  • Fix special case of --strict-equality and literal types (PR 7358)
  • Whitelist some equality checks under --strict-equality (PR 7302)
  • Disable --strict-equality inside functions that have type variables with value restrictions (PR 7304)
  • Fix overly restrictive --strict-equality with NewType and literals (PR 7351)

Changes Affecting Plugins

  • Add error code arguments to the plugin API (PR 7323)
  • Move some (undocumented) API functions as part of a refactoring, including make_simplified_union (PR 7397)
  • Plugins don’t need to support the old semantic analyzer, which was removed

Other Notable Improvements and Bug Fixes

  • Report undefined module attributes with --ignore-missing-imports (PR 7372)
  • Fix issue with list comprehension shadowing variables (PR 7476)
  • Fix false positive when overriding a class method that uses self type (PR 7474)
  • Fix combination of --follow-imports-for-stubs and --follow-imports=skip (g.denis, PR 7459)
  • Fix crash in multiple assignment when __iter__ returns Any (PR 7462)
  • Be more aggressive about site-packages in MYPYPATH (PR 7458)
  • Recognize a class as protocol if at least one of the bases is Protocol (Tom Lee, PR 7455)
  • Handle class with tuple base as TypeVar bound (Andriy Teraz, PR 7444)
  • Test Python 3.8 beta in Continuous Integration (PR 7421)
  • Fix crash in incremental mode related to redefined classes (PR 7413)
  • Assume list can be empty if passed as a *args argument (Ekin Dursun, PR 7392)
  • Fix crash when a dataclass with a no-init InitVar is inherited (Ekin Dursun, PR 7390)
  • Fix dataclass __init__ signature calculation issue with forward references (PR 7382)
  • Second expression in assert statement should be checked as conditional (g.denis, PR 7318)
  • Fix TypedDict with unicode name (PR 7369)
  • Improve overload target selection when using TypedDict (PR 7365)
  • Treat __init_subclass__ as an implicit class method (Tavian Barnes, PR 7355)
  • Be less aggressive when inferring literal types (PR 7354)
  • Analyze and check default arguments to lambdas (PR 7306)
  • Fix stubgen placement of decorators (g.denis, PR 7269)
  • Fix crash when interpolating bytes with a dictionary (PR 7297)
  • Reduce memory use by freeing state and ASTs when they are no longer needed (PR 7280)
  • Analyze super() expressions more precisely (Tavian Barnes, PR 7232)
  • Make per-line type checking precision statistics more precise (PR 7254, PR 7261)
  • Log whether a compiled mypy is being used (PR 7257)
  • Fix crash related to overriding an attribute of a generic class (PR 7235)
  • Fix spurious error about redefining final variable (PR 7230)
  • Fix crash when a broken overload appears in stub file (PR 7223)
  • Add Any type to imports when encountered by stubgen (Oleg Höfling, PR 7090, PR 7208)
  • Reprocess a module if a previously missing import was added (PR 7199)
  • Fix crash related to TypedDict (PR 7195)

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

The library stubs for subprocess were improved significantly. This may require fixes to existing annotations that weren’t checked by mypy previously.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Adam
  • Adam Smith
  • Amit jha
  • Andriy Teraz
  • Barry Warsaw
  • Brandt Bucher
  • Daniël van Eeden
  • ed1d1a8d
  • Ekin Dursun
  • Ethan Smith
  • g.denis
  • hetmankp
  • hoefling
  • Hugh Han
  • Jakub Molinski
  • Jared Hance
  • Jason Veatch
  • Jelle Zijlstra
  • Juan Luis Cano Rodríguez
  • kxing
  • Matthew Wright
  • Max Murin
  • Michael Lee
  • Michael Sullivan
  • nu_no
  • Paul Sokolovsky
  • Ran Benita
  • Sanjit Kalapatapu
  • Shuyao Bi
  • Sinan Cepel
  • Tapasweni Pathak
  • Tavian Barnes
  • Tom Lee
  • Tomer Chachamu
  • Tushar Sadhwani
  • yuxiang-he

Additional thanks to all contributors to typeshed:

  • Alan Du
  • Allison Kaptur
  • Andriy Teraz
  • Anthony Sottile
  • Benjamin Peterson
  • Bouteillebleu
  • Brad
  • Brandt Bucher
  • Cole Maclean
  • Colin Gilgenbach
  • Connor Brinton
  • Daniel Hahler
  • Daniel Holth
  • Evgeny Dedov
  • g.denis
  • Ilya Konstantinov
  • Jaromir Latal
  • Jelle Zijlstra
  • Jon Dufresne
  • Kamil Bar
  • Maarten ter Huurne
  • Michael Lee
  • Michael Nix
  • Michael R. Shannon
  • Naman
  • Nicklas Lindgren
  • Niels Buwen
  • Nipunn Koorapati
  • Pascal Corpet
  • Patrick Valsecchi
  • Phil Jones
  • plokmijnuhby
  • Ran Benita
  • Rebecca Chen
  • Rune Tynan
  • Samuel Freilich
  • Scott Belden
  • Sebastian Rittau
  • Shannon Zhu
  • Vasily Zakharov
  • Ville Skyttä
  • William Ayd
  • Yannack
  • Zsolt Dollenstein
  • 秋葉

Friday, 12 July 2019

Mypy 0.720 Released

We’ve just uploaded mypy 0.720 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release features a new semantic analyzer and optional warnings about unreachable code. It also includes many other new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

New Semantic Analyzer Used by Default

The new semantic analyzer was introduced in mypy 0.710 and is now the default. The old analyzer is still available under --no-new-semantic-analyzer, but it will be removed in the next mypy release. This release includes many improvements to the new semantic analyzer.

The semantic analyzer is a component of mypy that binds names to definitions. It also keeps track of what sort of thing each name defines, such as a function or a type alias. The new semantic analyzer is a major redesign and achieves several things.

The old semantic analyzer ordinarily processes files in a linear order, from the beginning to the end. This is subtly different in the mypy daemon, which sometimes processes an individual function or a module top level (code not nested within a function) as a single unit. The new semantic analyzer unifies mypy behavior by always analyzing module top-levels separately, before analyzing functions. As a bonus, this more closely reflects Python runtime behavior, since modules are usually fully initialized before any functions are called.

The old semantic analyzer processes each module in an import cycle twice, in an order determined by complicated heuristics. This can cause bogus errors when there are references to another module that hasn’t been processed yet in a dependency cycle. The new analyzer processes modules in a flexible order, and references across an import cycle work significantly more reliably.

The old semantic analyzer can’t always deal with forward references to definitions. The new semantic analyzer improves support for forward references. Example where this helps:

    from typing import NamedTuple
    
    f = lambda: A(1)  # This forward reference now works
    
    A = NamedTuple('A', [('x', int)])
    
    f()

The new semantic analyzer is stricter than the old semantic analyzer. It catches additional type errors and quality issues. Existing code that passes type checking may require small adjustments.

New Feature: Warn about Unreachable Code

If you run mypy with --warn-unreachable, mypy will warn you about unreachable statements, and expressions that may be never evaluated because of short circuiting. For example:

    from typing import Text
    
    def emphasize(text: Text) -> Text:
        if isinstance(text, int):
            text += 1  # Error: statement is unreachable
        return text.upper()

See the documentation for more information. This feature was contributed by Michael Lee.

Plugin API Changes

The new semantic analyzer may require changes to mypy plugins. We have a GitHub issue with more information for maintainers of plugins.

Notable Improvements and Bug Fixes

  • Have --package usage respect mypy_path (Aaron Batilo, PR 6926)
  • Support flexible option specification in config files. Now all the flags support invertible forms so that one can specify either strict_optional = False or no_strict_optional = True. This matches how flags work on the command line (PR 7054)
  • Fix reachability inference with isinstance(Any, Any) (Michael Lee, PR 7048)
  • Fix mypyc crash with plugins using get_customize_class_mro_hook() (PR 7075)
  • Fix slicing tuples with non-literal expressions (Ethan Smith, PR 7066)
  • Fixes to column numbers in error messages (PR 7078, PR 7081)
  • Make Need type annotation for variable error more consistent (PR 7079, PR 7113)
  • Fix error reporting context for missing generic type arguments (PR 7100)
  • Use only directories containing no __init__.py as namespace packages (Arne Welzel, PR 7108)
  • Support method plugin hooks on union types (PR 6560)
  • Fix some corner cases in --strict-equality (PR 7156, PR 7164)
  • Fix binding of self-types on unions (PR 7152)
  • Add more details for Invalid type errors (PR 7166)
  • Fix custom __init__() in dataclasses (Ryan Gonzalez, PR 7168)
  • Better support for indexing with unions of literal types (Michael Lee, PR 6558)

Other Updates

You can browse the full commit log here.

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Aaron Batilo
  • Arne Welzel
  • Chad Birch
  • Daniel Hahler
  • Deepyaman Datta
  • Ethan Smith
  • Michael Lee
  • Rafi Blecher
  • Ryan Gonzalez

Additional thanks to all contributors to typeshed:

  • Benjamin Woodruff
  • Chad Dombrova
  • Cole Maclean
  • crusaderky
  • Eric Arellano
  • Florian Ludwig
  • Francis Colas
  • Gordon Ball
  • Hynek Schlawack
  • ikelos
  • Jason Gilholme
  • Jelle Zijlstra
  • Maarten ter Huurne
  • Martijn Pieters
  • Mathieu Bridon
  • Matthew Wilkes
  • Motoki Naruse
  • Rafi Blecher
  • Ran Benita
  • Rebecca Chen
  • redshiftzero
  • Rune Tynan
  • Sam Zhou
  • Savo Kovačević
  • Sebastian Rittau
  • 秋葉
— Ivan Levkivskyi, on behalf of the mypy team

Monday, 24 June 2019

Mypy 0.711 Released

We’ve just uploaded mypy 0.711 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This fixes two minor issues with mypy 0.710. You can install it as follows:

python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Details

The following two issues in mypy 0.710 were fixed:
  • Revert typeshed PR 2878 (“Define functools.partial as overloaded function instead of its own class”). This caused too many false positive errors in real-world code. (PR 3077)
  • Fix MYPYC_BLACKLIST on Windows. This broke running dmypy on Windows. (PR 7032)

If you’re upgrading from mypy 0.701, mypy 0.700 or earlier, please read the blog post about mypy 0.710.

Wednesday, 19 June 2019

Mypy 0.710 Released: New Semantic Analyzer

We’ve just uploaded mypy 0.710 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release features a new, experimental semantic analyzer and inline configuration options. It also includes many other new features, bug fixes and library stub (typeshed) updates. You can install it as follows:
    python3 -m pip install -U mypy
You can read the full documentation for this release on Read the Docs.

New Semantic Analyzer (Experimental)

We invite everyone with more than a passing interest in mypy to try out the new semantic analyzer. Please report any issues you might find, since we will likely make it the default in the next mypy release, and we will drop the old semantic analyzer soon after.
The semantic analyzer is a component of mypy that binds names to definitions. It also keeps track of what sort of thing each name defines, such as a function or a type alias. This release officially supports a revamped implementation of the semantic analyzer, dubbed the new semantic analyzer. In particular, it deals better with import cycles and forward references.
You can enable the new semantic analyzer by passing --new-semantic-analyzer on the command line or by setting new_semantic_analyzer = True in your mypy config file (typically mypy.ini). (docs)

Inline Configuration Options

You can now specify per-file configuration flags in # mypy: comments. For example, you can now put
    # mypy: no-strict-optional
at the top of a file to disable strict optional checking in that specific file. Previously you would have to specify this on the command line (e.g., mypy --strict-optional) or in the mypy config file (e.g., strict_optional = True in the [mypy] section or in a file-specific section). Multiple flags can be separated by commas or placed on separate lines.
    # mypy: no-warn-no-return
    # mypy: disallow-any-generics, always-false="FOO,BAR"
For more info, see the docs.

Type inference improvements and bug fixes

  • Improvements to the handling of Literal types. (Michael Lee; docs)
  • You can now ignore a file with a single # type: ignore comment at the top of the file. (Brandt Bucher, PR 6830)
  • Fix crashes with callable() and issubclass(). (Jelle Zijlstra, PR 6981)
  • Allow class-based TypedDicts in all Python versions. (PR 6971)
  • Fix yield from <tuple>. (Brandt Bucher, PR 6902)
  • Add more precise inference for enum attributes. (Michael Lee, PR 6867)
  • When strict optional is disabled, anything can be falsey; previously this caused some incorrect assumptions about code being unreachable. (PR 6876)
  • Document extended callable types as deprecated. (PR 6890)
  • Allow redefinition within with statements. This is strictly speaking unsafe, but allowing this is convenient in the vast majority of cases. (PR 6871)
  • Allow del with a list, e.g. del [x, y]. The semantics are the same as del x, y. (Brandt Bucher, PR 6858)
  • Don't crash when partial types are used in inherited attribute. (Ekin Dursun, PR 6766)
  • Allow ellipsis default arguments in "trivial" non-stub functions. (Michael Lee, PR 6665)
  • Don't treat the first argument of a function nested inside a method as self. (Ekin Dursun, PR 6769)
  • Emit an error message when a class member is redefined. (Ekin Dursun, PR 6686)
  • Don’t complain about overriding a private method name in a subclass. (Rafael Carício, PR 6790)
  • Refine the interaction between Literal and Final. (Michael Lee, PR 6763)
  • Fix the dataclass plugin’s interaction with member expressions. (PR 6739)
  • Support decorated constructors. (Previously, decorating __init__ would cause the class’s constructor to be unchecked!, PR 6645)
  • Fix multiple inheritance check for generic classes. (PR 6720)
  • Add basic support for enum literals. (Michael Lee, PR 6668)
  • Loosen base class attribute compatibility checks when attribute is redefined (Seth Yastrov, PR 6585)
  • Fix crash when a star expression is used in isinstance. (Ekin Dursun, PR 6659)
  • Fix crash when namedtuple, classmethod and generic types used. (Ekin Dursun, PR 6669)
  • Fix __getattr__ and operators related infinite recursion. (Ekin Dursun, PR 6655)
  • Don't type check a return statement containing a lambda . (This would treat everything after the lambda as unreachable., PR 6633)

Error message refinements

  • Messages produced by reveal_type() and reveal_locals() now use the note level rather than error. (Daniel Hahler, PR 6919)
  • Make the error for too few or too many arguments in type comments non-fatal. (PR 6813)
  • Improve the error message when a function is missing a return type. (Allison King, PR 6773)
  • Better error message for assignments and returns incompatible due to invariance. (Felipe de Morais, PR 6803)
  • Update the error message for incompatible arguments. (Matas Kairaitis, PR 6796)
  • Show the specific argument names that are missing in the error message for missing arguments. (Rafael Carício, PR 6795)
  • When a duplicate module is detected, suggest that a missing __init__.py file could be the cause. (Alexander Bliskovsky, PR 6785)
  • Extend the list of third-party packages with the top 200 packages from PyPI. (Matas Kairaitis, PR 6781)
  • Show closest candidates for misspellings in the case of from … import … . (marco-dd, PR 6778)
  • Improve the error message when an annotation is expected. (Rafael Carício, PR 6782)
  • When a type comment contains a syntax error, show the text of the bad comment in the error message. (kcaebe, PR 6779)
  • Include a note when user forgets to import something from the typing module. (Christopher Sabater Cordero, PR 6770)
  • Display both instances of duplicate modules in the error message. (Alexander Bliskovsky, PR 6772)
  • Turn all warnings and bare notes into errors. (PR 6650)

Command line/configuration related changes

This section also covers documentation, stubgen and plugins.
  • Update the typed_ast dependency to 1.4.0+. This allows installing mypy using Python 3.8. (PR 6937)
  • Add --no-implicit-reexport flag. (Jared Hance; docs)
  • Only honor --warn-unused-configs in non-incremental mode. In incremental mode it doesn't work right, and it isn't worth fixing. (PR 6829)
  • Fix sys.stdout overriding in mypy.api. (Amanda Walker, and El Khadiri Yassine, PR 6750)
  • Miscellaneous small improvements to stubgen.
  • Command-line syntax errors now consistently use exit code 2. (PR 6738)
  • Tweaks to --strict-equality based on user feedback. (PR 6674)
  • Allow specifying the list of files to check in the mypy config file. (Samuel Williams, docs)
  • Make directory checks case sensitive. (PR 6684)
  • Allow disabling the config file using --config-file="". (PR 6664)
  • Add some more plugin docs. (PR 6618)
  • Add a get_additional_deps hook for plugins, to support django-stubs. (PR 6598)

Python 3.8 related changes

  • Support / for positional-only arguments (PEP 570). (Brandt Bucher, PR 6900)
  • Support importing Protocol, TypedDict, Literal, Final and @final from typing (typing_extensions is still supported). Ironically, this is also supported Python 2.7. These were recently standardized by PEP 544 (Protocol and @runtime_checkable), PEP 586 (Literal), PEP 589 (TypedDict) and PEP 591 (Final/@final).
  • Fix line numbers assigned to class and def and ignoring for Python 3.8. (Brandt Bucher, PR 6753)
  • Expression-scoped ignores in Python 3.8. (Brandt Bucher, PR 6648)

Other Updates

You can browse the full commit log here.

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.
Thanks to all mypy contributors who contributed to this release:
  • Alexander Bliskovsky
  • Allison King
  • Amanda Walker
  • Anthony Sottile
  • Bernát Gábor
  • Brandt Bucher
  • Brooke
  • cclauss
  • Charles-Axel Dein
  • Christopher Sabater Cordero
  • crusaderky
  • Daniel Hahler
  • duke9509
  • El Khadiri Yassine
  • Ekin Dursun
  • Emil Goldsmith Olesen
  • Ethan Smith
  • Felipe de Morais
  • Ivar
  • Jan Szopinski
  • Jared Hance
  • Jason Michalski
  • Jelle Zijlstra
  • Joe Juzl
  • kcaebe
  • marco-dd
  • Matas Kairaitis
  • Max Mautner
  • Michael Lee
  • Nick Gaya
  • PattenR
  • Rafael Carício
  • Robin Chen
  • Rémi Lapeyre
  • Samuel Williams
  • Sanjit Kalapatapu
  • Seth Yastrov
  • viourr
Additional thanks to all contributors to typeshed:
  • Alex Chamberlain
  • Alexander Fasching
  • Anirudh Padmarao
  • Anthony Sottile
  • Benjamin Peterson
  • berdario
  • Brandt Bucher
  • Brendan Long
  • Brian Brunner
  • Callum Ryan
  • Carl Meyer
  • Chad Birch
  • Chad Dombrova
  • Chandan Singh
  • Connor Skees
  • Dan Crosta
  • Danny Weinberg
  • Dominic
  • Eric Arellano
  • Ethan Madden
  • Evan Moses
  • ijl
  • J Rob Gant
  • Jadiker
  • Jan Szopinski
  • Jean Hominal
  • Jelle Zijlstra
  • Jennifer Taylor
  • Jia Chen
  • Joe Juzl
  • John Freeman
  • Jon Dufresne
  • Jonathan Slenders
  • Josh Morton
  • Mark Mendoza
  • Mark Vismonte
  • markedwards
  • Masashi SHIBATA
  • Mathieu Bridon
  • Max Rydahl Andersen
  • Michael A. Smith
  • Michael Lee
  • nabnut
  • Paul Dagnelie
  • Philipp Hahn
  • Philipp Schrader
  • Radu Matei Lăcraru
  • Ran Benita
  • Rebecca Chen
  • Rune Tynan
  • Saul Shanabrook
  • Scott Belden
  • Sean Vig
  • Sebastian Rittau
  • Sergey Machulskis
  • Shahar Evron
  • Simon Kohlmeyer
  • Stephen Thorne
  • Sushain Cherivirala
  • The Fox in the Shell
  • Utkarsh Gupta
  • Viktor Roytman
  • Walter Scott Johnson
  • Yegor Roganov
  • zadamah
  • 秋葉
— Guido van Rossum, on behalf of the mypy team












Tuesday, 16 April 2019

Mypy 0.701 Released

We've just uploaded mypy 0.701 to PyPI. This is a bugfix release containing no new features, only fixes for some important bugs and regressions in the (default) mypyc-compiled version of 0.700. You can install it as follows:

    python3 -m pip install --upgrade mypy

Mypy Bugs Fixed

  • Fix a crash in mypyc-compiled stubgen (PR 6642)
  • Compile with a newer version of mypyc that fixes several memory leak bugs (PR 6642)