Monday 21 November 2016

Mypy 0.4.6 Released

We’ve just uploaded mypy 0.4.6 to PyPI. This release adds many new features, bug fixes and library stub updates. As always, you can install it as follows, being careful to install mypy-lang, not mypy:

    python3 -m pip install --upgrade mypy-lang

Generic Type Aliases

You can now define type aliases that are themselves generic types. For example, consider this definition of Vector:

    from typing import TypeVar, List

    T = TypeVar('T')

    Vector = Tuple[T, T, T]

Now Vector[float], for example, is equivalent to Tuple[float, float, float]. Generic type aliases can make your type annotations shorter and easier to read. The following example defines two equivalent functions. The first one uses a generic type alias to simplify the signature:

    def negate1(vecs: List[Vector[float]]) -> List[Vector[float]]:
        return [(-x, -y, -z) for x, y, z in vecs]

    def negate2(vecs: List[Tuple[float, float, float]]
               ) -> List[Tuple[float, float, float]]:
        return [(-x, -y, -z) for x, y, z in vecs]

Check out the documentation for more details.

This was contributed by Ivan Levkivskyi and he also implemented the related changes to typing.

Detecting Missing Return Statements

Mypy now can generate an error if a function is missing a return statement, unless the return type is None or Any. Use the --warn-no-return command line option to enable this, or the warn_no_return config file setting. When the option is set, mypy warns about this function, since it doesn’t return anything if the condition is false:

    def f(n: int) -> int:
        if n > 3:
            return n + 1  # Missing return statement

Mypy currently doesn’t warn about empty function bodies or bodies that only have an ellipsis (), since these are sometimes used for abstract methods and aren’t necessarily a problem. The option can generate false positives if you rely on the implicit None return value when execution falls off the end of a function.

This was contributed by Reid Barton.

Improved Type Checking of Import Cycles

Previous versions of mypy were prone to giving errors like ‘Cannot determine type of “foo”’ if your program had import cycles. Another common problem was that innocent-looking changes inside import cycles would cause mypy to emit a new set of unrelated errors.

This release has two changes that should resolve most of these issues. They are a bit technical, and it’s not important to understand how they work in order to enjoy the benefits. Anyway, the two following subsections have a summary for those who are interested in the nitty-gritty internal details. Feel free to skip them. For even more details, have a look at these PRs: #2264 and #2167

Deferred Checking within Import Cycles

Previously mypy type-checked each module only once, a single module at a time. This could be a problem with import cycles. For example, module a could depend on inferred types in b, and b could depend on types of a. No linear ordering of modules would produce clean output, unless the program had type annotations for all the problematic variables — this was only a problem for things whose types weren’t immediately obvious to mypy. Mypy now type-checks an import cycle in two passes. If there is a reference to a variable in another module that mypy hasn’t yet processed, mypy defers the type-checking of the surrounding function, and finally runs another type-checking pass for only deferred functions.

Programs without cycles aren’t affected, since mypy has type-checked single modules in two passes for some time. We’ve now generalized it to multiple modules in a cycle.

Stable Processing Order within Import Cycles

The erratic mypy behavior mentioned earlier was caused by changes in the order of processing modules within a cycle. Mypy goes through each module in a cycle in a linear order, and some errors are only triggered for certain orderings. In previous versions of mypy a small change in a program could trigger big changes in the module processing order. Mypy now better approximates the runtime module initialization order, which is usually pretty stable.

For example, if module a has a statement like from b import f, module b likely will be initialized before module a at runtime, and so mypy will type-check b before a. On the other hand, if an import statement is within an if TYPE_CHECKING: block, mypy will not use this heuristic, since typing.TYPE_CHECKING is false at runtime and the import statement won’t affect runtime behavior.

Self Types (Experimental)

[This feature is experimental — the implementation still has significant limitations or bugs, and the feature may change in future mypy versions in incompatible ways.]

Sometimes you have a method that returns a value with exactly the same type as self. Previously there was no way to do this. Now you can annotate the self argument with a type variable to express this. Here’s an example taken from the documentation:

    from typing import TypeVar

    T = TypeVar('T', bound='Shape')

    class Shape:
        def set_scale(self: T, scale: float) -> T:
            self.scale = scale
            return self

    class Circle(Shape):
        def set_radius(self, r: float) -> 'Circle':
            self.radius = r
            return self

    class Square(Shape):
        def set_width(self, w: float) -> 'Square':
            self.width = w
            return self

    circle = Circle().set_scale(0.5).set_radius(2.7)  # type: Circle
    square = Square().set_scale(0.5).set_width(3.2)  # type: Square

By declaring self as a type variable, the set_scale can return a Circle when called on a Circle object and a Square when called on a Square, while only having a single method definition in the Shape base class.

For a class method, you can also now use Type[T] in a similar way as the annotation for the cls argument.

This feature was contributed by Elazar Gershuni.

Type Applications

Mypy now supports a type application syntax for user-defined generic classes. For example, consider a generic Stack class:

    from typing import TypeVar, Generic

    T = TypeVar('T')

    class Stack(Generic[T]):
        ...

When constructing a Stack instance, you can use the type application syntax Stack[<type>] to specify the type arguments. This constructs a Stack[int] instance using a type application:

    stack = Stack[int]()

Previously you had to use a type annotation, which is more verbose:

    stack = Stack()  # type: Stack[int]

This is now practical since the latest typing implementation caches the result of Stack[int], so it will be quick enough even for a frequently called function. This was actually supported in early mypy releases, but it has been unsupported for a while, primarily due to performance concerns.

You still have to use the type annotation syntax for standard-library classes such as list and Queue since these classes don’t support the indexing operator.

This was contributed by Ivan Levkivskyi. He also implemented the related changes to typing.

New NamedTuple Syntax (Python 3.6)

You can use a new syntax for named tuples with Python 3.6b1 or newer:

    from typing import NamedTuple

    class Point(NamedTuple):
        x: int
        y: int

    p = Point(x=1, y=2)

Run mypy with --fast-parser --python-version 3.6 to use this. You’ll need to install a recent version of typed_ast (pip3 install --upgrade typed_ast).

This was contributed by Ivan Levkivskyi.

Config File Changes

The filename patterns in the mypy configuration file were changed to module name patterns. You’ll have to update your config files if you use this feature.

Here’s an example mypy.ini file that rejects functions without a type annotation in the frobnicate package but allows them elsewhere:

    [mypy-frobnicate.*]
    disallow_untyped_defs = True

You can configure additional module search path entries in the mypy configuration file through the mypy_path configuration file option. This can be useful with local stub files stored in a separate directory, for example. Previously you had to use the MYPYPATH environment variable. mypy_path was contributed by Filip Figiel.

New Command Line Options

  • Add --cobertura-xml-report for outputting Cobertura XML reports with type checking coverage information (Roy Williams).
  • Add support for using a custom typeshed directory (--custom-typeshed-dir).
  • Add --junit-xml=PATH option for generating JUnit XML files with type checking results.
  • Add --find-occurrences that finds all references to a class member using static type information (experimental).
  • Stubgen now supports --fast-parser (Ivan Levkivskyi).

Other Changes in This Release

  • Improvements to --strict-optional (please go ahead and try it — it now works pretty well).
  • Allow subclassing Tuple[...] outside stubs again (Ivan Levkivskyi).
  • Show column number for reveal_type() expressions.
  • Various updates to the bundled typing module.
  • Support conditional import in functions/classes (Mark Laws).
  • Support matrix multiplication operator @ (Elazar Gershuni).
  • Several miscellaneous bug fixes.
  • Several documentation improvements.
  • Lots of updates to the library stubs in typeshed.

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Alex Jurkiewicz
  • Byungwoo Jeon
  • Calen Pennington
  • Chris Lamb
  • David Foster
  • Elazar Gershuni
  • Filip Figiel
  • Herbert Ho
  • Ivan Levkivskyi
  • Kevin Yap
  • Mark Laws
  • Reid Barton
  • Roy Williams
  • Ryan Gonzalez
  • Ulric Aleksandrov

Additional thanks to these contributors to typeshed:

  • Alvaro Caceres
  • Calen Pennington
  • David Percy
  • Dima Gerasimov
  • Eklavya Sharma
  • Evan Hubinger
  • Gerhard Hagerer
  • Hong Minhee
  • Jakub Stasiak
  • Jon Dufresne
  • Jordan Pittier
  • Joshua Smock
  • Kai Lautaportti
  • Matthias Kramm
  • Onno Kortmann
  • Reiner Gerecke
  • Ruud van Asseldonk
  • Ryan C. Thompson
  • Sebastian Meßmer
  • TrueBrain
  • Xavier Mehrenberger
  • Yegor Roganov
  • Joseph H Garvin
  • nobuggy
  • paavoap

— Jukka (on behalf of the rest of the mypy team: Guido, David and Greg)

Friday 7 October 2016

Mypy 0.4.5 Released

We’ve just uploaded mypy 0.4.5 to PyPI. This release adds many new features, bug fixes and library stub updates. As always, you can install it as follows, being careful to install mypy-lang, not mypy:

    python3 -m pip install -U mypy-lang

Gitter Chat Room

We’re embarking on a customer support paradigm shift: we now have a chat room at gitter.com. Please join us for an interactive conversation. Of course, we also still encourage filing issues in our tracker!

Configuration Files

You can now put common mypy options in a configuration file, so you won’t have to remember to pass all those pesky flags each time you run mypy. Many options can also be specified per file using filename matching patterns. For example, you might put this in mypy.ini at the root of your repo:

    [mypy]
    python_version = 2.7
    fast_parser = True
    [mypy-foo/*]
    disallow_untyped_defs = True

This sets --python-version 2.7 (a.k.a. --py2 ) and --fast-parser for all mypy runs in this tree, and selectively turns on the --disallow-untyped-defs flag for all files under the foo/ subdirectory. This prints an error for function definitions without type annotations in that subdirectory only.

On a related topic, if you have a file containing a list of files and directories to check, you can now invoke mypy on that list using mypy @filenames — this reads a list of filenames from the file filenames .

PEP 526 Variable Annotations

If you like to live in the future, you may have noticed that Python 3.6b1 was recently released with syntactic support for variable annotations specified by PEP 526. But syntactic support alone wouldn’t be very useful, so we’ve got experimental type checking support for this feature in mypy! Thanks a bundle to Ivan Levkivskyi for the implementation. Example:

    def sum(a: List[float]) -> List[float]:
        total = 0.0
        cumulative: List[float] = []  # Check it out!
        for x in a:
            total += x
            cumulative.append(total)
        return cumulative

Note: you need --fast-parser to use this (it’s easy to enable in mypy.ini , see above) and you need to install typed-ast version 0.6.1 (python3 -m pip install -U typed-ast ).

New and Changed Flags

  • Renamed --suppress-error-context to --hide-error-context .
  • Fixed some nasty bugs in --incremental mode that could cause irreproducible errors.
  • Multiple -c arguments are now joined using newlines into a single multiline command.
  • New flag --show-column-numbers causes error messages to include column numbers in addition to line numbers. Note that column numbers are zero-based: the start of the line is column 0. (Ben Duffield)
  • New flag --disallow-subclassing-any reports errors when a base class has type Any .
  • New flag --strict-optional-whitelist controls strict none checking for individual files using filename matching patterns. (In mypy.ini, use show_none_errors in pattern sections, in combination with a global strict_optional flag.)
  • New flag --scripts-are-modules gives command line arguments that appear to be scripts (i.e. files whose name does not end in .py) a module name derived from the script name rather than the fixed name __main__. (This is more useful although less strictly correct.)

Deprecations and Removals

  • We no longer support using Python 3.2 to run mypy. (You can still type check Python 3.2 programs by specifying --python-version 3.2 .)
  • There used to be an undocumented feature named “weak mode”, triggered by comments of the form # mypy: weak . We now have better way to configure the strictness of checking per file, using pattern sections in mypy.ini (see above) so we have removed this feature to simplify mypy’s internals somewhat.
  • There was an undocumented package mypy.codec that was meant to support the Python 3 function annotation syntax in Python 2 using a codec hack (# coding: mypy ). We stopped using this a long time ago in favor of type comments, and we’ve finally removed the code.

Miscellaneous Other Changes in This Release

  • typing.TYPE_CHECKING : a constant that is false at runtime but true while type checking. This is the PEP 484 spelling. Previously you had to use if MYPY instead of if TYPE_CHECKING. Requires version 3.5.2.2 of the typing package.
  • Support for bytes formatting (e.g. b'foo%d' % 42) in Python 3.5 and up, per PEP 461. (Buck Baskin)
  • Several improvements to namedtuple and NamedTuple , e.g. _replace() and _fields are now supported. (Elazar Gershuni)
  • Mypy is now less aggressive about shortening types in error messages. E.g. previously a union with four or more values would often be replaced with union type (4 items), and similar for tuples. This now doesn’t happen unless the text would be over 400 characters.
  • Modest performance improvements.
  • Several miscellaneous bug fixes.
  • Lots of updates to the library stubs in typeshed.

Stub Generator

  • Improvements to the generated stubs. (Valérian Rousset)
  • Add --recursive and --ignore-errors flags. (Raphael Gaschignard)

Acknowledgements

We’d like to thank Michael Lee whose internship ended during this release cycle. He made --incremental worthy of trust and also contributed many other fixes (including documentation). Thanks Michael!

Thanks to all mypy contributors who contributed to this release:

  • Ben Duffield
  • Buck Baskin
  • David Foster
  • Elazar Gershuni
  • Ivan Levkivskyi
  • John Hagen
  • Raphael Gaschignard
  • Valérian Rousset

Additional thanks to these contributors to typeshed:

  • Alvaro Caceres
  • claws
  • Danny Weinberg
  • David Euresti
  • dlinnemeyer
  • Drew Haven
  • Elazar Gershuni
  • Evgeniy Vasilev
  • Gustavo J. A. M. Carneiro
  • Ivan Levkivskyi
  • John Hagen
  • Manuel Krebber
  • Matthias Kramm
  • Ollie Wild
  • rchen152
  • Rich Li
  • Robin Alazard
  • Roy Williams
  • Samuel Colvin
  • Sebastian Meßmer
  • Stephen Thorne
  • Tim Abbott
  • Valérian Rousset
  • Yasushi Saito

— Jukka (on behalf of the rest of the mypy team: Guido, David and Greg)

Thursday 25 August 2016

Mypy 0.4.4 Released

I just uploaded version 0.4.4 of mypy-lang to PyPI! This release adds several new features, bug fixes and library stub updates.

Run this to upgrade to the new release using pip:

    python3 -m pip install -U mypy-lang

Experimental async and await Support

Mypy can now type check code using async and await (PEP 492). Here is an example from the documentation (this requires --fast-parser):

    import asyncio

    async def format_string(tag: str, count: int) -> str:
        return 'T-minus {} ({})'.format(count, tag)

    async def countdown_1(tag: str, count: int) -> str:
        while count > 0:
            my_str = await format_string(tag, count)  # has type 'str'
            print(my_str)
            await asyncio.sleep(0.1)
            count -= 1
        return "Blastoff!"

    loop = asyncio.get_event_loop()
    loop.run_until_complete(countdown_1("Millennium Falcon", 5))
    loop.close()

NewType

You can use NewType (see documentation) to define a special lightweight variant of an existing type. Mypy considers it to be a separate type, but there’s only minimal runtime overhead, as it doesn’t define a new class. We can create a new type called UserId, instances of which are actually int objects at runtime, but which is treated like a subclass of int by mypy:

    from typing import NewType

    UserId = NewType('UserId', int)

    def name_by_id(user_id: UserId) -> str:
        ...

    name_by_id(42)          # Type check error -- UserId expected
    name_by_id(UserId(42))  # OK
    print(UserId(42))       # 42 (the runtime representation is just int)

Additional Changes

Here is list of other notable changes in this release:

  • Support checks for sys.version_info and sys.platform, which are especially useful for stubs (see documentation)
  • Add the --platform command line option to explicitly set the target platform, instead of defaulting to the current platform
  • Support *expr within list, tuple and set expressions, and **expr within dict expressions (requires --fast-parser)
  • Incremental type checking is now more robust (--incremental)
  • Strict optional checking is also more robust (--strict-optional)
  • Many bugs fixed (especially crashes)
  • Lots of typeshed improvements
  • Documentation updates
  • Warn about unused type ignores on imports with --warn-unused-ignores
  • Add --suppress-error-context flag to suppress notes about class/function (this cleaner output mode may become the default in the future)
  • Less output by default on internal error (use --tb to show traceback)
  • Removed the need for -f /--dirty-stubs during mypy development
  • Started transitioning to using pytest for the mypy test suite (PR #1944)

Acknowledgements

Thanks to all mypy contributors who contributed to this release:
  • Daniel F Moisset
  • Fabian Heredia Montiel
  • Roy Williams
  • Ryan Gonzalez
  • Shrey Desai
  • Valérian Rousset
Additional thanks to typeshed contributors:
  • Alvaro Caceres
  • Antoine Catton
  • Daniel Horn
  • Daniël van Eeden
  • David Euresti
  • Elazar Gershuni
  • Emanuel Barry
  • Fu Yong Quah
  • Jakub Stasiak
  • Matthias Kramm
  • Max Wittek
  • Michael R. Crusoe
  • Nicholas Bishop
  • Tom Manderson
  • Tomasz Elendt
  • Tyler O'Meara
  • Wojciech Kaczmarek
  • Alvaro Caceres
  • jchien14
  • jdelic
  • John K Lai
  • kosaka
  • peterdotran
  • speezepearson
  • Tony Grue
  • wreed4

- Jukka (on behalf of the rest of the mypy team: Guido, David, Greg and Michael Lee)

Thursday 14 July 2016

Mypy 0.4.3 Released

I just uploaded version 0.4.3 of mypy-lang to PyPI! This release adds several new features (some of which are still experimental), bug fixes and library stub updates.

Run this to upgrade to the new release using pip:

    python3 -m pip install -U mypy-lang

Strict Checking of Optional Types

Mypy now supports strict checking of optional types (enable it through --strict-optional). The option makes mypy check that you don’t try to perform unsupported operations on None values, and that you don’t use None values when a non-Optional value is expected.

This feature is still experimental and has known issues. Our intention is to turn this on by default in a future mypy release, once it works well enough.

Here is an example buggy program that mypy will reject only if you use --strict-optional:

    from typing import Optional, List

    def read_data_file(path: Optional[str]) -> List[str]:
        with open(path) as f:  # Error
            return f.read().split(',')

Mypy will rightly complain about the program, since open() does not accept a None argument — and path may be None. Luckily, it’s easy to fix the program:

    from typing import Optional, List

    DEFAULT_PATH = '...'

    def read_data_file(path: Optional[str]) -> List[str]:
        if not path:
            path = DEFAULT_PATH
        with open(path) as f:  # OK!
            return f.read().split(',')

Mypy recognizes that if the not path condition isn’t true and the if statement body is not executed, path won’t be None. It also knows that after the assignment statement that I added path can’t be None, so now the open(path) call is always safe, no matter which execution path is taken.

The example shows that mypy can do pretty clever reasoning about code. Mypy also knows about other kinds of None checks — go ahead and give it a try.

Multi-line Comment Function Annotation Syntax

You can now use an alternative per-argument comment annotation syntax for code that needs to be Python 2 compatible. Previously, functions with a long argument list resulted in overly long type comments and it was tricky to see which argument type corresponds to which argument.

Note: This only works when using --fast-parser. This isn’t supported on Windows yet. (We intend to use --fast-parser by default in a future release, but only once it works on Windows.)

Here is an example (from PEP 484):

    def send_email(address,     # type: Union[str, List[str]]
                   sender,      # type: str
                   cc,          # type: Optional[List[str]]
                   bcc,         # type: Optional[List[str]]
                   subject='',
                   body=None    # type: List[str]
                   ):
        # type: (...) -> bool
        """Send an email message.  Return True iff successful."""
        <code>

Additional Changes

Here are the remaining notable changes in this release:

  • Improve how mypy deals with module cycles. When adding a new module to a cycle, mypy is less likely to generate unrelated-looking errors from other modules in the cycle.
  • Mypy more aggressively infers types for variables initialized with literals, instead of requiring a dummy annotation. Previously you sometimes had to write code like x = 0 # type: int which was a little silly.
  • Infer types from multiple isinstance checks or’ed together, such as isinstance(x, int) or isinstance(x, str). The inferred type is a union type.
  • Add option --warn-unused-ignores that gives a warning if you use # type: ignore on a line that actually doesn’t generate an error and thus may be redundant.
  • Add option --warn-redundant-casts that gives a warning if you have a cast that is superfluous and can be removed without generating an error.
  • Fixes and improvements to incremental type checking. Add option --cache-dir for specifying a custom custom incremental cache directory. Note that incremental type checking is still experimental.
  • Fixes to Type[C].
  • Other bug fixes.
  • Typeshed stub updates.
  • Python 3.2 is no longer officially supported for running mypy. (This does not affect type checking Python 3.2 programs.)

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Carol Willing
  • Eric Price
  • Herbert Ho
  • Max Wittek
  • Michael Lee
  • Russ Allbery

Additional thanks to typeshed contributors:

  • Alvaro Caceres
  • Amandine Lee
  • Amit Saha
  • Brett Cannon
  • Dakkaron
  • Drew Haven
  • Eklavya Sharma
  • Elazar
  • Fu Yong Quah
  • Herbert Ho
  • Håken Lid
  • Matthias Kramm
  • Michael Lee
  • Mickaël S
  • Oren Leaffer
  • Phil Jones
  • Philip House
  • Russ Allbery
  • Skip Montanaro
  • Valérian Rousset
  • garetht
  • tewe
  • Thomas Cellerier

- Jukka (on behalf of the rest of the mypy team: Guido, David, Greg and Reid)

Thursday 9 June 2016

Mypy 0.4.2 Released

I just uploaded version 0.4.2 of mypy-lang to PyPI! This is a minor release that focuses on bug fixes.

The biggest change is the addition of Type[C]. You can now accurately describe the type of an argument whose type is itself a type or class. Given a class C, a function argument annotated with Type[C] can be a class object that's either C or a subclass of C. You can call C's class methods on that argument, and instantiate it using the constructor signature(s) supported by C. Combined with a type variable with C as an upper bound you can do even more powerful things. The Type operator must be imported from the typing module. It will be available starting Python 3.5.2. See the description in PEP 484.

Note: To use typing.Type[C], you need Python 3.5.2 (to be released shortly) or you can pip install the latest version of typing.

These are the other notable changes in this release:

  • Add reveal_type(expr) for displaying the static type of an expression.
  • Better IntEnum support.
  • Get the latest version of typing from python/typing repo (for Python 3.4 and earlier that don't include typing in stdlib).
  • When both foo.py* and foo/__init__.py* exist, prefer the latter.
  • Suppress error about untyped defs in typeshed stubs.
  • Fix use of property in a NamedTuple class.
  • Fix how the fast parser interprets string literals in Python 2.
  • Other bug fixes.
  • Typeshed stub updates.
  • Documentation updates, such as a type annotation cheat sheet.
  • Stop testing running mypy with Python 3.2. We’ll likely drop support for running mypy itself with 3.2 pretty soon, though you'll still be able to type check Python 3.2 code.

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Akshit Khurana
  • Andrew Sprenchynat
  • Babak
  • Jakub Stasiak
  • Jeffrey McLarty
  • Josh Soref
  • Justus Adam
  • Margaret Sy
  • Max Wittek
  • Peter McCormick
  • Ryan Gonzalez
  • Tarashish Mishra
  • Thiago
  • Tim Abbott
  • Thomas Ballinger
  • Valérian Rousset

Additional thanks to typeshed contributors:

  • beckjake
  • Dakkaron
  • David Euresti
  • David Shea
  • detlefla
  • Eklavya Sharma
  • Erin Haswell
  • Florian Bruhin
  • Jakub Stasiak
  • James Tatum
  • jukebox
  • Max Payton
  • Michael R. Crusoe
  • Oren Leaffer
  • Ran Benita
  • Thomas Grainger
  • Tim Abbott
  • Tim Simpson
  • Vadim Chugunov
  • Valérian Rousset
- Jukka (on behalf of the rest of the mypy team: Guido, David, Greg and Reid)

Tuesday 31 May 2016

Slides for Mypy Talk At PyCon

The mypy team gave a talk about mypy at PyCon 2016 on Sunday, the day before the main conference. The slides are available here:

Thanks to all who were able to attend!

The talk wasn't recorded (which is particularly unfortunate since David Fisher gave an awesome live demo).

- Jukka

Thursday 5 May 2016

Mypy 0.4 Released

I just uploaded version 0.4 of mypy-lang to PyPI! This release focuses on performance improvements, usability improvements and bug fixes. Again, there are too many changes to describe them all. I'll focus here on the most important ones.

I'm also happy to welcome a new mypy core developer, Reid Barton. Reid is supported by Dropbox, like the rest of the core team.

As before, mypy is experimental and still has known bugs and limitations, though things are quickly getting better. Library stubs for many commonly used modules are still missing, but now there is the option --silent-imports (which was supported by 0.3.1, but poorly documented) that can work around this issue by ignoring missing stubs, at the cost of some type checking precision.

Please report any new issues you encounter -- we try to respond quickly to all user requests. If you create a new library stub, consider contributing it to typeshed, even if it isn’t very polished.

Release Highlights

These are some of the most exciting new things:

  • Experimental incremental type checking mode (enabled via --incremental or -i) speeds up type checking large programs significantly by reusing cached results from previous type checking runs. If you’ve only made small changes, type checking can now be much faster. We plan to enable this by default once we’ve given it a little more time to mature. (By Guido)
  • David Fisher implemented a new experimental parser that uses a tweaked version of the standard library ast module to parse programs. This gives a nice performance boost and can be enabled via --fast-parser. (You must first pip install typed-ast for this to work. Note that it doesn't support Python 3.2.)
  • Mypy now supports type variable bounds, i.e. TypeVar(…, bound=…), as specified by PEP 484. (By Reid Barton)
  • Many crashes and bugs fixed (and a few added :-).
  • A ton of typeshed updates.

Command Line Changes

There are several new command line options:

  • --fast-parser enables the new, faster parser (see above). This is still experimental but worth a try!
  • --disallow-untyped-defs generates errors for functions without type annotations. Consider using this if you tend to forget to annotate some functions.
  • --disallow-untyped-calls causes mypy to complain about calls to untyped functions. This is a boon for static typing purists, together with --disallow-untyped-defs :-)
  • --line-count-report DIRECTORY generates a report of annotated line counts.
  • --almost-silent behaves like --silent-imports, but it also reports every module that was silently ignored. This is useful for troubleshooting if mypy doesn’t seem to find all the code you are expecting.

Some old options or arguments are now a little more useful:

  • When given a directory as a positional argument, mypy now always looks into the directory and any subpackages of the directory and includes every .py or .pyi file that it finds there. It also does the right thing if the target directory is a package. (The old behavior was surprising and better not discussed any further.)
  • Documented the --silent-imports / -s command-line option for silently ignoring missing modules.
  • Support multiple -m options.
  • --implicit-any was renamed to --check-untyped-defs. It causes mypy to do rudimentary type checking even in unannotated functions. Be default, mypy mostly ignores unannotated functions. It’s still better to annotate functions, but this can provide some light type checking for almost free.

A few options were removed or deprecated:

  • --silent is deprecated (use -s or --silent-imports, which do the same thing)
  • --use-python-path is disabled because it’s rarely the right thing to do and has resulted in much confusion. --silent-imports often gives better results.

Python 2 Support

There were only a few changes to Python 2 language support — it’s now pretty mature:

  • A function annotation like # type: (...) -> rtype (that’s a literal ...) specifies only the return type of a function without having to enumerate all the argument types (they default to Any).
  • Parse lambda (x): y correctly.

(Support for PEP 484 long argument list annotations for Python 2 will be included in the next release.)

Type Checking Improvements

  • Support bare Callable types (#670)
  • The first argument of a class method now has a real type (it used to be Any)
  • Fix various issues around generics, type variables and type inference
  • Fix error messages when the type of *args or **kwds is wrong
  • Compute type of inherited constructor correctly (#1246)
  • Add typing.DefaultDict
  • Support calling dict() with positional arguments and keyword arguments or **kwds (#1391)
  • Some improvements to handling of import cycles, forward references and decorators
  • Some improvements to conditional imports and definitions (still more to do)
  • Improve type inference of [] if ... else [something]
  • Suppress (most) errors from semantic analysis for unannotated functions (#1334)

Other News

  • We’ve now got a nice CONTRIBUTING.md file (thanks, Greg Price!)
  • Fixed many bugs in report generation, e.g. correct support for multiple files (also by Greg)
  • Improved algorithm for finding data files (e.g. typeshed) when installed on various platforms
  • “Regular” errors are written to stdout; errors about command line or files to stderr
  • Added a .bat file for running mypy on Windows
  • Various speedups (also by Greg -- even the test suite was made much faster!)
  • Improved processing of internal errors

Acknowledgements

Here's a list of everybody who contributed to the mypy repository since the last release. Thanks to everybody who helped! Apologies if I've missed anybody.

  • Alex Brandt
  • Ben Darnell
  • Brett Cannon
  • Chris (allthedata)
  • Dmitriy Olshevskiy
  • Ivan Levkivskyi
  • Keith Philpott
  • Kevin Modzelewski
  • Max Wittek
  • Michał Zochniak
  • mr.Shu
  • Paul Kienzle
  • Reid Barton
  • Ryan Gonzalez
  • Samuel Colvin
  • Vytautas Astrauskas

Also thanks to everybody who contributed to typeshed:

  • Alessio Bogon
  • Anirudha Bose
  • Ben Darnell
  • Dakkaron
  • David Soria Parra
  • Drew Haven
  • FichteFoll
  • Filip Hron
  • Hong Minhee
  • Isaac Goldberg
  • Ismail
  • Ivan Levkivskyi
  • Jakub Stasiak
  • James Tatum
  • jukebox
  • Julien Hebert
  • Katherine Lai
  • Linjie Ding
  • Lorenzo Bolla
  • Martin Geisler
  • Matthias Kramm
  • Max Wittek
  • Maxim Kurnikov
  • Michael Lee
  • Michael R. Crusoe
  • mulhern
  • Pas
  • Reid Barton
  • Sidharth Kapur
  • tharvik
  • Valérian Rousset
  • Yasushi Saito
  • Yusuke Miyazaki

Additional thanks to Guido and the rest of the core team for helping with this blog post.

- Jukka (on behalf of the mypy team)

Friday 19 February 2016

Mypy 0.3 Released

I just uploaded mypy-lang 0.3 to PyPI! This release focuses on further PEP 484 compatibility and Python 2 compatibility, but it also contains many general improvements. The changes are too numerous to enumerate them all. Instead, I'll focus here on the most visible and useful ones.

I'm also happy to announce new mypy core developers Guido, David Fisher and Greg Price, who have joined the project thanks to support from our employer, Dropbox. This has given mypy development a big boost, and mypy is currently under very active development.

Mypy 0.3 breaks backward compatibility. You may have to modify your code to be PEP 484 compatible if you have been using mypy-lang from PyPI. As before, mypy is experimental and still has known bugs and limitations. In particular, we still don't have library stubs for many commonly used modules. Please report any new issues you encounter -- we try to respond quickly to all user requests, and bugs are often fixed rapidly.

Feature Highlights

  • Python 2 support -- now basically all Python 2.7 syntax is supported in Python 2 mode. Mypy also supports a comment-based function annotation syntax for Python 2 that was added to PEP 484 recently. It also works for code that aims to be both Python 2 and 3 compatible. More about this below.
  • Improved PEP 484 support. Mypy now supports most PEP 484 features (not yet, however, sys.version_info checks), and several non-standard features have been removed.
  • Partial Python 3.5 support (though async/await are sadly still not supported).
  • New experimental type checking modes (--silent-imports and --implicit-any command-line options) that make it much easier to type check subsets of a larger code base (see discussion of developer workflow improvements below).
  • Improved type inference -- fewer variable annotations are required, and mypy is much less likely to give the annoying "Cannot determine type of x" errors.
  • Many bug fixes and better Python compatibility.
  • Numerous standard library stub improvements (via typeshed, see below).

GitHub Changes

  • The mypy github repository has moved to the Python organization. The new URL is https://github.com/python/mypy.
  • Library stubs now live in a separate repo called typeshed (https://github.com/python/typeshed). The idea of the repository is make stubs a shared resource between all Python type checking or inference related projects.

Python 2 Support

Mypy now has experimental Python 2.7 support! For code that needs to be Python 2 compatible, function type annotations are given in comments, since the function annotation syntax was introduced in Python 3. Mypy has had partial, unofficial Python 2 support for a while, but only now it is actually usable and standard (after a recent PEP 484 update).

Run mypy in Python 2 mode by using the --py2 option:

    $ mypy --py2 program.py

To run your program, you must have the typing module in your module search path. Use pip install typing to install the module (it also works for Python 3).

Python 2 support was a lot of work and here is a summary of some of the most obvious changes:

  • Support pretty much all of Python 2 only syntax, including the print statement and the except Ex, err: syntax.
  • Provide Python 2 stubs for many library modules (now in typeshed, as mentioned earlier.)
  • Better Python 2 implementation of typing.
  • Document Python 2 support.
  • Make stub generator support generating Python 2 stubs (the stub generator is still highly experimental and not at all well documented, but it has already been pretty useful anyway.)

The example below illustrates Python 2 function type annotation syntax. This is also valid in Python 3 mode:

    from typing import List

    items = []  # type: List[str]

    def hello(): # type: () -> None
        print 'hello'

    class Example:
        def method(self, lst, opt=0, *args, **kwargs):
            # type: (List[str], int, *str, **bool) -> int
            ...

Here are some things to note:

  • Type syntax for variables is the same as Python 3.
  • Function annotations are in comments and look quite a bit like Python 3 annotations.
  • The module typing is available in Python 2 mode and things like Any must be imported from it, even if they are only used in comments.
  • You don't provide an annotation for the self/cls variable of methods.
  • The annotation can be on the same line as the function header or on the following line.
  • You don't need to use string literal escapes for forward references within comments.

Better PEP 484 Compatibility

Several changes improve PEP 484 compatibility. Mypy still doesn't implement quite everything in PEP 484, but we are pretty close.

Undefined Removed

As the final version of PEP 484 doesn't include Undefined, mypy no longer supports it. It's easy to migrate old code to not use Undefined -- just use type comments. For example, refactor this

      from typing import Undefined
      x = Undefined(int)

to this:

      x = None  # type: int

Support for Callable[..., t]

Mypy supports callable types with unspecified argument types. Just replace the arguments with an explicit ...:

    from typing import Callable
    def call_a_lot(func: Callable[..., int]) -> None:
        kwargs = {'arg': 1}
        func(1, x=y, **kwargs)  # OK, no checking of arguments

Support for Homogeneous Tuples (Tuple[t, ...])

Mypy now has a type for homogeneous, variable-length tuples: Tuple[t, ...] (with an explicit ...). These can be used like sequences, but the concrete type must be a tuple. These are useful for several library functions that expect concrete tuples, and also let us give precise types for *args arguments, which are tuples.

@no_type_check

You can use the @no_type_check decorator to skip all type checks within a function. It also lets you use function annotations for non-mypy uses:

    from typing import no_type_check

    @no_type_check
    def special_annotations(arg: {'value': 1}) -> 1234:   # Fine!
        1 + ''  # We can do all sorts of crazy things here, and mypy is silent!

Changes to Name Visibility in Stubs

As per PEP 484, mypy no longer export names imported in stubs, unless they are imported via as. For example, this stub won't export Tuple any more:

    from typing import Tuple

    def my_function() -> Tuple[int, str]: ...

This is especially nice together if you import a module with a stub using from module import *, as your namespace won't get polluted with random names imported by the stub from typing.

If in a stub you import using from m import *, everything in m will get exported. This makes it easy for a stub to re-export all definitions from another module.

Using ... in Stubs

PEP 484 recommends using an ellipsis (...) instead of pass for empty function and class bodies in stubs. It can can also be used as variable initializers and default argument values. This example illustrates all of these:

    # This is stub file
    var = ...  # type: int
    def function(arg: str = ...) -> int: ...
    class VeryBadError(Exception): ...

bytearray Is Compatible with bytes

Mypy considers bytearray to be compatible with bytes (as per PEP 484). This simplifies stub files and annotations, as previously we often had to use Union[bytes, bytearray] or something similar, which was hard to remember to do consistently.

Workflow Improvements

This release has several experimental improvements to running mypy against your code. These are subject to change in the future and not very well documented, but they are useful enough that I'll introduce them anyway:

  • You can give an arbitrary number of files or directories to type check on the mypy command line (e.g. mypy file1.py file2.py dir). Mypy will recurse into a directory argument if it contains an __init__.py or __init__.pyi (stub) file, looking for .py and .pyi files. If the directory is not a package, it'll only look for .py / .pyi files within the given directory but not in subdirectories.
  • You can ask mypy to ignore files outside those explicitly mentioned on the command line (or inferred by the directory traversal algorithm discussed above) via --silent-imports. Stubs are still consulted, but other than that mypy will not try to follow imports to other modules or complain about missing stubs. This makes it easy to start experimenting with mypy on a subset of a larger codebase without getting a huge number of errors about missing stubs and similar.
  • By default mypy type checks only functions with annotations. You can give mypy a the option --implicit-any to implicitly give an annotation with all Any types for every unannotated function. This way mypy can do useful (though a bit limited) type checking for unannotated code. As you add function annotations, mypy will do an incrementally better job. In the future, something like this may be the default mode of operation, once we get this good enough.

Miscellanea

Here are some additional improvements:

  • Mypy now continues processing files after errors and can report errors in multiple files in a single run
  • Rudimentary __new__ support
  • Better type inference for isinstance checks, including support for and and not operations involving isinstance and multiple checks per condition
  • Using a value with Any type as a base class (this is very useful together with # type: ignore)
  • Support for @abstractproperty

There are also some interesting work-in-progress features such as a new automatic stub generator.

Acknowledgements

First of all, I'd like to thank Dropbox for contributing very significant engineering resources to improving mypy, including some of my time! We've been experimenting with using mypy on Dropbox code and initial results are promising, even though there are some big unsolved challenges ahead, such as type checking performance and scaling to very large codebases (but we are working on these!).

Also, thanks to Guido and Matt Robben for feedback on earlier drafts of this article.

Many people contributed to this release. Thanks to everybody who helped! Apologies if I've missed anybody (this includes contributions to typeshed):

  • Andrew Aldridge
  • Anup Chenthamarakshan
  • Ben Darnell
  • Ben Longbons
  • ctcutler
  • Daniel Shaulov
  • Darjus Loktevic
  • David Fisher
  • David Shea
  • Della Anjeh
  • Eric Price
  • Gigi Sayfan
  • Greg Price
  • Guido van Rossum
  • Howard Lee
  • Ian Cordasco
  • icoxfog417
  • Igor Vuk
  • ismail-s
  • James Guthrie
  • Jared Hance
  • Jing Wang
  • Jukka Lehtosalo
  • Kyle Consalus
  • Li Haoyi
  • Matthew Wright
  • Matthias Bussonnier
  • Matthias Kramm
  • Michael Walter
  • Michal Pokorný
  • Mihnea Giurgea
  • Motoki Naruse
  • Prayag Verma
  • Robert T. McGibbon
  • Roy Williams
  • Ryan Gonzalez
  • Sander Kersten
  • Sebastian Reuße
  • Seo Sanghyeon
  • Tad Leonard
  • Tim Abbott
  • Vita Smid
  • Vlad Shcherbina
  • Wen Zhang
  • wizzardx
  • Yuval Langer

An up-to-date list of all mypy code contributions is available at the GitHub contributors page.