Friday, 15 December 2017

Mypy 0.560 Released

We’ve just uploaded mypy 0.560 to the Python Package Index (PyPI). This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on ReadTheDocs.

I'm also happy to welcome our newest mypy team member at Dropbox, Michael Sullivan.

Iterable and Friends Are Now Protocols

Various ABCs in typing such as Iterable, Iterator and Sized are now protocols instead of regular ABCs. This means that they support structural subtyping and you no longer need to explicitly subclass from typing.Iterable if you want your class to be treated as an Iterable, for example. You only need to define an __iter__ method:

    from typing import Iterator
    
    class MyCollection:  # Note: no base classes
        ...
        def __iter__(self) -> Iterator[str]: ...
    
    def dump(c: MyCollection) -> None:
        for item in sorted(c):  # Okay, sorted() takes any Iterable    
            print(item)

Previously you had to explicitly subclass MyCollection from Iterator[str] to make mypy accept it as an argument for sorted() (and in other contexts where an iterable object is expected), even though it’d work at runtime without subclassing:

    from typing import Iterable, Iterator
    
    class MyCollection(Iterable[str]):  # Base class is now optional
        ...

The new behavior matches how duck typing works in Python, and it also avoids potential metaclass conflicts that you may encounter if you inherit from one of the typing ABCs.

Read the documentation for more details, including a full list of protocols in typing. With this change, protocols are no longer considered an experimental feature.

This was contributed by Ivan Levkivskyi.

Other New Features

  • Widen type of x to Any when isinstance(x, Any) is asserted (Dominik Miedziński, PR 3751)
  • Add checks for overlapping tuple types in overloads (Elazar Gershuni, PR 4238)

Notable Bugs Fixed

  • Fix spurious abstract attribute errors with Any base class (PR 4230)
  • Fix crash caused by lambda that is deferred, i.e. it refers to a variable whose inferred type is not available yet for mypy (PR 4232)
  • Check the right operand of is expressions (Ethan Smith, PR 4243)
  • When disallow_untyped_defs is enabled, correctly warn about missing return types in async def (Jelle Zijlstra, PR 4218)
  • Check signatures of reverse operator methods (Ethan Smith, PR 4249)
  • Fix type inference problem with tuple types (Ivan Levkivskyi, PR 4087)
  • Report additional errors in lambda bodies that were previously omitted (Ethan Smith, PR 4257)
  • Fix crash related to relative imports (Ivan Levkivskyi, PR 4259)
  • Fix spurious errors about returning Any when --warn-return-any is used (Elazar Gershuni, PR 4295)
  • Issue a blocking error early if builtins can't be found, instead of crashing or generating confusing errors (PR 4327)

Improved Error Messages

  • Fix missing line number in covariance error (PR 4275)
  • Improve error message for an invalid callable type (Daniel Li, PR 4213)
  • Produce better error on invalid types like List(int) (Michael J. Sullivan, PR 4323)
  • Give better error message for omitted parentheses in function type comment (Michael J. Sullivan, PR 4329)
  • Suggest spurious trailing commas as a cause for invalid tuple types (Michael J. Sullivan, PR 4336)

Other Changes

  • Speed improvements in incremental mode (PR 4294, PR 4308, PR 4331)
  • Document empty tuple syntax: Tuple[()] (Ethan Smith, PR 4313)
  • Document how to silence invalid errors from linters (Ethan Smith, PR 4314)
  • Document how to ignore imports in specific modules only (Ethan Smith, PR 4316)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Daniel Li
  • Dominik Miedziński
  • Elazar Gershuni
  • Ethan Smith
  • Ivan Levkivskyi
  • Jelle Zijlstra
  • Nehal J Wani

Additional thanks to all contributors to typeshed:

  • Alex Willmer
  • Ben Leslie
  • Brandon W Maister
  • Daniel Axtens
  • Elazar Gershuni
  • Ethan Smith
  • FichteFoll
  • gossrock
  • Grzegorz Śliwiński
  • Henri Bai
  • Ivan Levkivskyi
  • Jelle Zijlstra
  • Josh Staiger
  • Kenny Do
  • Luka Sterbic
  • Masayuki Yamamoto
  • Nathan Henrie
  • Neil Pilgrim
  • Patrick Valsecchi
  • rchen152
  • Sebastian Rittau
  • Sebastian Steenbuck
  • Semyon Proshev
  • Tim Abbott

Wednesday, 15 November 2017

Dropbox releases PyAnnotate -- auto-generate type annotations for mypy

For statically checking Python code, mypy is great, but it only works after you have added type annotations to your codebase. When you have a large codebase, this can be painful. At Dropbox we’ve annotated over 1.2 million lines of code (about 20% of our total Python codebase), so we know how much work this can be. It’s worth it though: the payoff is fantastic.

To easy the pain of adding type annotations to existing code, we’ve developed a tool, PyAnnotate, that observes what types are actually used at runtime, and inserts annotations into your source code based on those observations. We’ve now open-sourced the tool.

Here’s how it works. To start, you run your code with a special profiling hook enabled. This observes all call arguments and return values and records the observed types in memory. At the end of a run the data is dumped to a file in JSON format. A separate command-line utility can then read this JSON file and use it to add inline annotations to your source code.

Once the automatic annotations have been added you should run mypy on the updated files to verify that the generated annotations make sense, and most likely you will have to tweak some annotations by hand. But once you get the hang of it, PyAnnotate can save you a lot of work when adding annotations to a large codebase, and that’s nothing to sneeze at.

PyAnnotate is intended to ease the work of adding type annotations to existing (may we say legacy”) code. For new code we recommend adding annotations as you write the code type annotations should reflect the intention of the author, and there’s no better time to capture those intentions than when initially writing the code.

Example

Here’s a little program that defines a gcd() function and calls it a few times:

# gcd.py
def main():
    print(gcd(15, 10))
    print(gcd(45, 12))

def gcd(a, b):
    while b:
        a, b = b, a%b
    return a

We also need a driver script:

# driver.py
from gcd import main
from pyannotate_runtime import collect_types

if __name__ == '__main__':
    collect_types.init_types_collection()
    with collect_types.collect():
        main()
    collect_types.dump_stats('type_info.json')

Now let’s install PyAnnotate and run the driver script. The standard output is just what the main()  function prints:

$ pip install pyannotate
$ python driver.py
5
3

At this point, if you’re curious, you can look in type_info.json to see what types were recorded:

[
    {
        "path": "gcd.py",
        "line": 1,
        "func_name": "main",
        "type_comments": [
            "() -> None"
        ],
        "samples": 1
    },
    {
        "path": "gcd.py",
        "line": 5,
        "func_name": "gcd",
        "type_comments": [
            "(int, int) -> int"
        ],
        "samples": 2
    }
]

Let’s go ahead and annotate the gcd.py file (the -w flag means go ahead, update the file”):

$ pyannotate -w gcd.py
Refactored gcd.py
--- gcd.py        (original)
+++ gcd.py        (refactored)
@@ -1,8 +1,10 @@
 def main():
+    # type: () -> None
     print(gcd(15, 10))
     print(gcd(45, 12))
 def gcd(a, b):
+    # type: (int, int) -> int
     while b:
         a, b = b, a%b
     return a
Files that were modified:
gcd.py
$ mypy gcd.py

Since the original file was so small, the unified diff printed by PyAnnotate happens to show the entire file. Note that the final command shows that mypy is happy with the results! If you’d rather not see the diff output, use the -q flag.

Where to get PyAnnotate

The full source code for PyAnnotate is on GitHub: https://github.com/dropbox/pyannotate
If you’d rather just install and use it, a source distribution and a universal wheel” file exist on PyPI: https://pypi.python.org/pypi/pyannotate

Caveats

PyAnnotate doesn’t actually inspect every call the profiling hooks have a lot of overhead and a test scenario for a large application would run too slowly. For details on the downsampling” it performs see the source code. For the same reasons it also doesn’t inspect every item of a list or dictionary in fact it only inspects the first four. And if a function is called with many different argument types, it will only preserve the first eight that differ significantly.

The generated annotations are only as good as the test scenario you’re using. If you have a function that is written to take either an integer or a string, but your test scenario only calls it with integers, the annotations will say arg: int. If your function may return None or a dictionary, but in your test scenario it always returns None, the annotation will say -> None.

Annotations for functions with *args or **kwds in their signature will likely require manual cleanup.

PyAnnotate currently doesn’t generate Abstract Base Classes (ABCs) such as Iterable or Mapping. If you’re a fan of these you will have to do some manual tweaking to the output.

Since NewType and TypedDict are invisible at runtime, PyAnnotate won’t generate those.

Because of a limitation of the profiling hook, the runtime collection code cannot tell the difference between raising an exception and returning None.

Code in __main__ is currently ignored, because the module name doesn’t correspond to the filename.

The current version of PyAnnotate has been designed to work with either Python 2 or Python 3, but we’ve only used it for Python 2 code ourselves (since that’s what we have) and the tool currently generates type comments that are compatible with Python 2. We’re happy to accept a pull request adding the capability to generate Python 3 style annotations though.

Surely there are some bugs left in our code. We’re happy to accept bug reports in our tracker and bug fixes as pull requests. Note that all code contributions require filling out the Dropbox Contributor License Agreement (CLA).

We consider the current release a prototype, and we don’t promise that future versions will be backwards compatible. In fact, we hope to find the time to make significant changes. We just didn’t feel we would do anyone a favor by sitting on the code longer than we already have.

Friday, 10 November 2017

Mypy 0.550 released

We’ve just uploaded mypy 0.550 to the Python Package Index (PyPI). This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on ReadTheDocs.

Notable Changes

  • The --old-html-report flag was removed (PR 4204)
  • Replace --disallow-any flags with separate boolean flags (PR 4178)
  • Running mypy on Python 3.3 is no longer supported. However Python 3.3 is still valid for the target of the analysis (i.e. the --python-version flag) (PR 4152)

Notable Bugs Fixed

  • Fix classmethod access for Tuple subclasses (PR 4164)
  • Ignore failure to remove temporary files (PR 4163)
  • Make stubgen return status 0 on --help (PR 4160)
  • Fix broken installation on Windows in virtualenv (PR 4149)
  • Fix recent crashes in forward refs (PR 4120)
  • The source of mypy itself is now fully strict-optional compatible (PR 4078)
  • Check code after inferred UninhabitedType (PR 4059)

Notable Typeshed Changes

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Aleksander Gondek
  • Elazar Gershuni
  • Gareth T
  • Gregory Werbin
  • Hugo vk
  • Ivan Levkivskyi
  • Jelle Zijlstra
  • Rahiel Kasim
  • Svyatoslav Ilinskiy

Additional thanks to all contributors to typeshed:

  • Alan Du
  • Carl Meyer
  • Ethan Smith
  • Grzegorz Śliwiński
  • Ilya Konstantinov
  • Jelle Zijlstra
  • Josh Staiger
  • Martin DeMello
  • Matthew Wright
  • Matěj Cepl
  • Patrick Valsecchi
  • Roy Williams
  • Sally
  • Sebastian Rittau
  • Sebastian Steenbuck
  • Wilfred Hughes
  • Zac Hatfield Dodds
  • anentropic
  • hashstat
  • marienz
  • rchen152

Friday, 20 October 2017

Mypy 0.540 Released

We’ve just uploaded mypy 0.540 to the Python Package Index (PyPI). This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on ReadTheDocs.

Switch to More Rapid Releases

Going forward, we are going to release a new mypy version roughly every 2-4 weeks. The motivation is to make the release process lighter-weight and more predictable, and to reduce the need for bug fix releases — the next normal release will be a short time away, so we’d only need bug fix releases for critical bugs. As a bonus, users will get new features and bug fixes sooner, and contributors will have their PRs released faster!

New Features

  • Allow assignments to multiple targets from union types (Ivan Levkivskyi, PR 4067)
  • Allow definitions of subscripted type aliases such as Item = Tuple[int, str] within functions (Ivan Levkivskyi, PR 4000)
  • Narrow type of optional left operand of in operator if the collection items aren’t optional (Ivan Levkivskyi, PR 4072)
  • Generate an error if <unknown> type would be inferred for a variable (PR 4112)

Notable Bugs Fixed

  • Fix errors not being reported for some modules with incremental mode by deleting cache file for a module if errors are found (Ivan Levkivskyi, PR 4045)
  • Fix incremental mode crash related to Type[...] types (Ivan Levkivskyi, PR 4038)
  • Fix crash related to tuple types (Ivan Levkivskyi, PR 4051)
  • Fix incorrect errors about __getattr__ methods (Ivan Levkivskyi, PR 4073)
  • Fixes to non-method attributes with callable types (Elazar Gershuni, PR 4016)
  • Fix function overloading based on Type[...] types (Ivan Levkivskyi, PR 4037)
  • Fix processing invalid overloaded function definitions (Elazar Gershuni, PR 4064)
  • Fix handling of disallow_any/disallow_untyped_defs combination in mypy config file (Jelle Zijlstra, PR 4076)

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Carl Meyer
  • Edward Betts
  • Elazar Gershuni
  • Ethan Smith
  • Ivan Levkivskyi
  • Jelle Zijlstra

Additional thanks to all contributors to typeshed:

  • Ashwini Chaudhary
  • FichteFoll
  • Gábor Bernát
  • Jelle Zijlstra
  • Lloyd
  • Max Rozentsveyg
  • Michael Lee
  • Naman Bharadwaj
  • Semyon Proshev
  • Suren Nihalani
  • Travis Parker
  • hashstat
  • rymdhund

Friday, 6 October 2017

Mypy 0.530 Released

We’ve just uploaded mypy 0.530 to the Python Package Index (PyPI). This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the documentation for this release on ReadTheDocs.

Major New Features

We present several new features in this release: Protocols, TypedDict, Per-file Control of Strict Optional Checking and Module Aliases. The features are described below.

Protocols (structural subtyping, a.k.a. static duck typing)

Protocols are a new, experimental feature, intended to better support duck typing. Protocols are on a standardization track (PEP 544), but we are still working on the details of how protocols work. They are not yet included in the standard library typing module. You can import them from the new typing_extensions package instead (not to be confused with mypy_extensions).

To use protocols, first install the package typing_extensions:

    python3 -m pip install -U typing_extensions

Once this is installed you can start defining protocols by inheriting from typing_extensions.Protocol. Here’s a simple example:

    from typing import Iterable
    from typing_extensions import Protocol
    
    class SupportsClose(Protocol):
        def close(self) -> None: ...  # Use a literal '...' here
    
    class Resource:  # No SupportsClose base class!
        # ... some methods ...
    
        def close(self) -> None:
           self.resource.release()
    
    def close_all(items: Iterable[SupportsClose]) -> None:
        for item in items:
            item.close()
    
    close_all([Resource(), open('some/file')])  # Okay!

In this example, Resource is compatible with the SupportClose protocol since it defines a compatible close() method. Regular file objects returned by open() are similarly compatible with the protocol, as they support close().

Protocols were implemented by Ivan Levkivskyi. Here’s the documentation.

TypedDict

TypedDict has been an undocumented feature for a while; it’s now officially supported. It’s still experimental, but we think that it’s robust enough to invite everyone to kick the tires.

Python programs often use dictionaries with string keys to represent objects. Here is a typical example:

    movie = {'name': 'Blade Runner', 'year': 1982}

You can now define a TypedDict to give a precise type for objects like movie, where the type of each dictionary value depends on the key:

    from mypy_extensions import TypedDict
    Movie = TypedDict('Movie', {'name': str, 'year': int})
    favorite = {'name': 'Blade Runner', 'year': 1982}  # type: Movie
Movie is a TypedDict type with two items: 'name' (with type str) and 'year' (with type int). Note that we used an explicit type annotation for the favorite variable. This type annotation is important -- without it, mypy will try to infer a regular, uniform Dict type for favorite, which is not what we want here.

By default mypy ensures that a TypedDict object has exactly all the specified keys. You can provide the total=False argument to TypedDict(...) to allow keys to be left out when creating a TypedDict object:

    GuiOptions = TypedDict(
       'GuiOptions', {'language': str, 'color': str}, total=False)
    options = {}  # type: GuiOptions  # Okay
    options['language'] = 'en'

Python 3.6 also supports an alternative, class-based syntax to define a TypedDict:

       from mypy_extensions import TypedDict
    
       class Movie(TypedDict):
           name: str
           year: int

This syntax also supports the total=False flag, as follows:

    class GuiOptions(TypeDict, total=False):
        ...

Several people contributed to TypedDict, including David Foster (original implementation), Ivan Levkivskyi, Roy Williams, Max Moroz and Jukka Lehtosalo. Here’s the documentation.

Per-file Control of Strict Optional Checking

Mypy has supported the option --strict-optional for a while that causes mypy to detect errors due to None values. However, many users still haven’t switched to using this option as it can require a bit of refactoring to adopt, such as adding Optional[…] to annotations. Now mypy makes this easier as you can enable strict optional checking on a per-file basis, allowing you to gradually migrate your codebase to strict optional checking, while ensuring that previously migrated modules remain strict optional clean.

You can control this through a mypy config file. Example:

    ; Enable strict optional checking in acme.util
    [mypy-acme.util]
    strict_optional = True

This was implemented by David Fisher (PR 3206). Most of the work was actually completed for the 0.520/0.521 release but we withheld the announcement because we hadn’t had enough opportunity to test it.

Module Aliases

Mypy now can handle aliases for modules. Example:

    import subprocess
    sp = subprocess
    sp.check_call(['whoami'])  # Okay

This was contributed by Carl Meyer, with help from Ivan Levkivskyi.

More New Features

  • Support module-level __getattr__() in stubs (Jelle Zijlstra, PR 3647; Ethan Smith, PR 3779)
  • Generate note if reveal_type() is used in an unchecked function (Svyatoslav Ilinskiy, PR 3743)
  • Add flag --disallow-incomplete-defs to disallow partially annotated functions (Svyatoslav Ilinskiy, PR 3744)
  • Add flag --disallow-untyped-decorators to output error if a typed function is used with an untyped decorator (Svyatoslav Ilinskiy, PR 3555)
  • Tag Any types with the kind of their origin (Svyatoslav Ilinskiy, PR 3786)
  • Make disallow_subclassing_any a per-module option (Daniel Watkins, PR 3801)
  • Report sources of Any in html-report per-line as hovertext (Svyatoslav Ilinskiy, PR 3798)
  • Add --warn-unused-configs flag (PR 3865)
  • Support six.add_metaclass() (Elazar Gershuni, PR 3842)
  • Type check arguments for super() (PR 3919)
  • Allow creating type aliases for None (Ivan Levkivskyi, PR 3754)

Stubgen Improvements

  • Add -o <dir> option (Brian Bruggeman, PR 3902)
  • Preserve existing annotations (Daniel F Moisset, PR 3169)

Notable Bugs Fixed

  • Fix crashes and failures caused by forward references to types (Ivan Levkivskyi, PR 3952)
  • Make configuration file section ordering consistent on Python 3.5 and earlier (PR 3678)
  • Re-broaden variable type to Any on assignment if it started as Any but was narrowed in an enclosing frame (lincolnq, PR 3361)
  • Suppress errors found in unreachable parts of a Boolean expression (Roy Williams, PR 3666)
  • Give a useful message if we can't infer a TypeVar-based type for attribute with value restriction (PR 3651)
  • Support docstrings in a class-based TypedDict (Dominik Miedziński, PR 3749)
  • Do not follow imports from stubs with --follow-imports=skip (Svyatoslav Ilinskiy, PR 3738)
  • Support exported names starting with an underscore in __all__ (David Ross, PR 3746)
  • Search lib directory suffixed by ABI (32 or 64), which is used in at least Gentoo Linux (Shunsuke Shimizu, PR 3758)
  • Fix crash with unanalyzed branches and old-html report (Svyatoslav Ilinskiy, PR 3782)
  • Hide imported names in stubs unless as id is used (to align with PEP 484) (Ivan Levkivskyi, PR 3706)
  • Use correct configuration file path when searching for plugins (Tin Tvrtković, PR 3770)
  • Use types of actual arguments to infer lambda argument types if callee is a lambda (Elazar Gershuni, PR 3116)
  • Don't supply a type context to the expression in yield from (PR 3815)
  • Erase type variables of generic types when used in Type[C] (Daniel Watkins, PR 3833)
  • Fix using stale metadata in incremental mode when errors are no longer ignored (Mark Hahnenberg, PR 3793)
  • Infer better return type for some calls to overloaded functions with Any actual arguments (PR 3917)
  • Fix problem with overload resolution when there are union types in signatures (PR 3300)
  • Fix type checking decorated method overrides when decorator changes signature (PR 3918)
  • Fix crash caused by attempted serialization of a partial type (PR 3954)
  • Support user install on Windows (Ethan Smith, PR 4005)
  • Fix crashes related to partial types (Ivan Levkivskyi, PR 3995)
  • Don't crash on unexpected PY2 except syntax (PR 4011)

Dependencies Etc.

  • This mypy release requires typed-ast version 1.1.0 or higher. Since mypy 0.52[01] required a version less than 1.1.0, this is a mandatory upgrade. The main reason for the upgrade is that the new version supports Python 3.3 and 3.4 on Windows, which required some API changes.
  • This is the last mypy release with official support for running on Python 3.3, which reached its End Of Life last week. Note that future mypy versions will still support checking older Python versions — they just won’t run on 3.3 or earlier. It’s easy to have multiple Python versions installed, so this won’t prevent anyone from running mypy.

Other Changes

  • Many more mypy unit tests are now run via pytest (but the runtests.py script is still not dead yet).
  • Almost all of the mypy implementation is now type checked using strict optional checking (Ivan Levkiskyi).
  • Improvements to error messages (Dominik Miedziński, PR 3515; Elazar Gershuni, PR 3783; Svyatoslav Ilinskiy, PR 3707; Jesse Lord, PR 3411; Ethan Smith, PR 3862; Daniel Watkins, PR 3873, PR 3893, PR 3898, and PR 3910).

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Brian Bruggeman
  • Carl Meyer
  • Chen Rotem Levy
  • Daniel F Moisset
  • Daniel Watkins
  • David Ross
  • Dominik Miedziński
  • Edward Betts
  • Elazar Gershuni
  • Ethan Smith
  • Greg Price
  • Ivan Levkivskyi
  • Jelle Zijlstra
  • Jesse Lord
  • lincolnq
  • Mark Hahnenberg
  • Max Moroz
  • Michael R. Crusoe
  • Roy Williams
  • Shunsuke Shimizu
  • Svyatoslav Ilinskiy
  • Thomas Grainger
  • Tin Tvrtković
  • Tobin Yehle

Additional thanks to all contributors to typeshed:

  • Adam Dangoor
  • Alex Ellis
  • Alex Grönholm
  • Amandine Lee
  • Artur Rodrigues
  • Ashwini Chaudhary
  • Ben Leslie
  • Brian C. Lane
  • Bruce Merry
  • Chris Kerr
  • Daniel Watkins
  • David Novakovic
  • Dominik Gabi
  • Elazar Gershuni
  • Ethan Smith
  • Evan Krall
  • FichteFoll
  • Garrett
  • Ivan Levkivskyi
  • Jaen
  • Jelle Zijlstra
  • John Reese
  • Jonathan Schoonhoven
  • Liran Nuna
  • Luka Sterbic
  • Martin DeMello
  • Matthias Kramm
  • Melvyn Sopacua
  • Michael Lee
  • Mikołaj Biel
  • Neil Pilgrim
  • Øystein Olsen
  • Pavel Pletenev
  • Pieter-Jan Briers
  • Robert T. McGibbon
  • Semyon Proshev
  • Svyatoslav Ilinskiy
  • Thomas Grainger
  • Tim Crawford
  • Zack Hsi
  • aubanel
  • blokeley
  • daid
  • eujing
  • rchen152
  • toejough
  • vim345

Tuesday, 25 July 2017

Mypy 0.521 Released

We’ve just uploaded mypy 0.521 to PyPI. This is a bugfix release containing no new features, only fixes for some important bugs and regressions in 0.520. You can install it as follows:

    python3 -m pip install --upgrade mypy

Mypy Bugs Fixed

  • Fix crash on name collision for self attributes (Ivan Levkivskyi, PRs 3700 and 3719)
  • Fix bug in cache updates and improve cache logging (PR 3708)
  • Fix per-file strict Optional interaction with default-None args (PR 3686)
  • Fix crash in --quick mode when aliases are re-exported (Ivan Levkivskyi, PR 3740)
  • Fix crash on incompatible redefinition in named tuple (Ivan Levkivskyi, PR 3760)

Typeshed Bugs Fixed

  • Fix regressions in csv.DictReader (Thomas Grainger, PRs 1475 and 1478)

Acknowledgments

First of all, I’d like to thank my employer Dropbox for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Ivan Levkivskyi

Additional thanks to all contributors to typeshed for this release:

  • Thomas Grainger

— Jukka (on behalf of the rest of the mypy team: Guido, David and Greg)

Monday, 10 July 2017

Mypy 0.520 Released

We’ve just uploaded mypy 0.520 to PyPI. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install --upgrade mypy

We are always happy to answer questions on Gitter — note that we’ve moved to a new channel, python/typing.

Major New Features

Fine-grained Control of Any Types

Mypy now has a --disallow-any option that gives you more fine-grained control over Any types in your code. In particular, it lets you disallow the following kinds of Any:

  • explicit disallows explicit Any in type positions such as type annotations and generic type parameters
  • expr disallows all expressions in the module that have type Any
  • unimported disallows Any types that come from unfollowed imports (they usually appear with --follow-imports=skip or --ignore-missing-imports)
  • generics disallows usage of generic types that do not specify explicit type parameters. For example, List (without type parameters) is disallowed but using List[int] (or even List[Any]) is allowed.
  • decorated disallows functions that have Any in their signature after decorator transformation

You can use the option from command-line as --disallow-any=expr,explicit or from config file as disallow_any = generics,expr.

To learn more about the --disallow-any option, read the documentation.

This was implemented by Svyatoslav Ilinskiy.

__setattr__ Support

Mypy now understands __setattr__. If a class defines it, mypy allows assignments to arbitrary attributes. Example:

    class A:
        ...
        # This will allow assignment to any A.x, if x is the same type as value.
        def __setattr__(self, name: str, value: int) -> None:
            ...
    a.foo = 42  # Works
    a.bar = 'Ex-parrot'  # Fails type checking

This was contributed by Ethan Smith (PR 3451).

Hash-based Incremental Cache

Previously the incremental cache (enabled by the -i option) detected changes in source files based on the file modification time and file size. This made it hard to share incremental cache files between systems, since modification times generally aren’t consistent across multiple repository checkouts. Now mypy requires a change to the md5 hash of a file to consider it changed. This makes it possible to “warm up” the incremental cache by downloading it from a central repository, which can speed up mypy runs significantly. Providing an implementation of such a repository is out of scope for mypy, though.

This was implemented by Max Moroz and Guido (PR 3437 and some others).

NamedTuple Methods

It is now possible to add methods to NamedTuples defined using the Python 3.6 class-based syntax:
    class Point(NamedTuple):
        x: float
        y: float

        def translate(self, dx: float, dx: float) -> 'Point':
            return Point(x=self.x + dx, y=self.y + dy)

    p = Point(1, 2).translate(Point(4, 5))
    print(p)

Note that runtime support for this feature requires typing 3.6.1.

This was contributed by Jelle Zijlstra (PR 3081).

Improved Error Messages

Various error messages generated by mypy have been improved. Here is a summary of what’s changed:

  • Use distinct formatting for regular types and type object types; previously they looked sometimes confusingly identical (Elazar Gershuni, PR 3374)
  • Display the union type and offending union item when reporting a missing attribute for a union type (Amir Rachum, PR 3402)
  • Include line number of previous definition when there is a redefinition error (Max Moroz, PR 3424)
  • Suggest a type annotation if the type of a returned variable is incompatible with the declared return type (Jesse Lord, PR 3428)
  • Improve error message for invalid package names passed to mypy (chernrick, PR 3447)
  • Fix several error messages that didn't display types correctly (Svyatoslav Ilinskiy, PR 3638)
  • Revamp TypedDict-related error messages (Jukka Lehtosalo)

More New Features

  • Experimental support for enabling strict optional checking on a per file basis (David Fisher, PR 3206)
  • Support simple aliases to modules (Carl Meyer, PR 3435)
  • Support six.with_metaclass (Guido, PR 3364)
  • Allow any subtype of typing.Mapping[str, Any] to be passed in as **kwargs (Roy Williams, PR 3604)
  • Add flag --no-implicit-optional to avoid interpreting arguments with a default of None as Optional — this may become the default behavior in the future (David Fisher, PR 3248)
  • Allow NewType that subclasses another NewType (Carl Meyer, PR 3465)
  • Infer better return types for certain common special signatures such as open (if the mode is a string literal), contextlib.contextmanager and ** with a literal exponent (Jukka Lehtosalo, PR 3299)
  • Add --include-private flag to stubgen to include private members (Chad Dombrova, PR 3367)
  • Add flag --any-exprs-report to generate report about Any expressions (Svyatoslav Ilinskiy, PR 3492)
  • Allow tuple indexing using non-literal expressions (David Fisher, PR 3514)
  • Normalize Type[Union[...]] internally to Union[Type[...], ...] (Max Moroz, PR 3209)
  • Improve handling of compile-time relational operators (Ryan Gonzalez, PR 3158)
  • Add --no-incremental flag to disable incremental mode (David Fisher, PR 3570)
  • Improve type inference of unconditional if statements without else block (Michael Lee, PR 3567)
  • Document the --shadow-file option (David Fisher, PR 3615)
  • Display unreachable code as Any in HTML coverage reports (Svyatoslav Ilinskiy, PR 3616)
  • Add --skip-version-check flag to ignore the mypy version when validating incremental cache metadata (Guido, PR 3641)
  • Add a very experimental (and still undocumented) plugin system for extending mypy (PR 3534 and Jukka Lehtosalo, PR 3517)
  • Many improvements to TypedDict and new documentation [this is still not an officially supported feature] (Jukka Lehtosalo, Max Moroz and Ivan Levkivskyi)

Notable Bugs Fixed

  • Make change of platform invalidate the cache (Guido, PR 3663)
  • Fix imported generic type aliases (Ivan Levkivskyi, PR 3524)
  • Fix Optional[...] type aliases (Ivan Levkivskyi, PR 3524)
  • Reject redefinition of type aliases (Ivan Levkivskyi, PR 3524)
  • Fix None slice bounds with --strict-optional (Dominik Miedziński, PR 3445)
  • Fix crashes related to referring to a type before the definition has been processed (Ivan Levkiskyi, PR 3322 and PR 3560)
  • Fix crash when displaying the location of a method definition (Dominik Miedziński, PR 3375)
  • Treat non-inplace augmented assignment as a simple assignment (Max Moroz, PR 3110)
  • Fixes to quick mode (Ivan Levkivskyi, PR 3304 and PR 3356)
  • Don't warn about returning Any if an Any return value is explicitly allowed (Ivan Levkivskyi, PR 3473)
  • Fix type of f-string with only {…} such as f'{x}' (lincolnq, PR 3390)
  • Make Type[X] compatible with the metaclass of X (Elazar Gershuni, PR 3346)
  • Support accessing modules imported in class bodies within methods (Carl Meyer, PR 3450)
  • Fix crash when serializing a type-ignored property setter (Jelle Zijlstra, PR 3493)
  • Fix spurious "cycle in function expansion" errors (Guido, PR 3504)
  • Fix to union type used in condition (David Fisher, PR 3510)
  • Fix crashes on lambda generators (Ivan Levkivskyi, PR 3523)
  • Fix accessing qualified import in incremental mode (Michael Lee, PR 3548)
  • Fix overload resolution for metaclass (Elazar Gershuni, PR 3511)
  • Allow instances of a class to access inner class definitions on that class (Roy Williams, PR 3636)
  • Fix to type checking calls to classmethods with self types (Elazar Gershuni, PR 3633)
  • Don't crash if isinstance() called with too few arguments (Jukka Lehtosalo, PR 3652)
  • Reject isinstance() with NewType (Jukka Lehtosalo, PR 3654)
  • Fix crash during deserialization of callable types (Jukka Lehtosalo, PR 3660)

Other Changes

  • Require version 1.0.4 of the typed-ast dependency to run mypy
  • Release version 0.3.0 of mypy_extensions
  • Publish a new mypy roadmap
  • The majority of the mypy implementation is now type checked using strict optional checking (Ivan Levkiskyi)

Acknowledgments

First of all, I’d like to thank my employer Dropbox for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Abhas Bhattacharya
  • Amir Rachum
  • Carl Meyer
  • Chad Dombrova
  • chernrick
  • Dominik Miedziński
  • Elazar Gershuni
  • Ethan Smith
  • fdeliege
  • Ivan Levkivskyi
  • Jelle Zijlstra
  • Jesse Lord
  • lincolnq
  • Max Moroz
  • Michael Lee
  • Nathaniel Smith
  • Patrick Williams
  • Roy Williams
  • Ryan Gonzalez
  • Svyatoslav Ilinskiy
  • Thomas Grainger

Additional thanks to all contributors to typeshed:

  • Alex Grönholm
  • Alex Khomchenko
  • amstree
  • Antoine Reversat
  • Ashwini Chaudhary
  • Ask Solem
  • Atul Varma
  • Benjamin Starling
  • Carl Meyer
  • Cooper Lees
  • Daniel Li
  • David Euresti
  • DmitriyS
  • Doug Freed
  • Eklavya Sharma
  • Elliot Marsden
  • elmar bucher
  • Emily Morehouse
  • Ethan Smith
  • George Caley
  • Harmen
  • Ivan Levkivskyi
  • James Elford
  • Jan Hermann
  • Jason Fried
  • Jelle Zijlstra
  • Joe Bateson
  • John Reese
  • khyox
  • li-dan
  • Luka Sterbic
  • Lukasz Langa
  • macheins
  • Mahmoud Afaneh
  • Matthew Page
  • Matthias Kramm
  • Matěj Cepl
  • Max Moroz
  • Max Rozentsveyg
  • Michael Lee
  • Miguel Gaiowski
  • mistermocha
  • mpage
  • Nathan Henrie
  • Peter Vilim
  • pkordas
  • Ray Kraesig
  • rchen152
  • Rhys Parry
  • Roy Williams
  • Ryan Jarvis
  • Sebastian Meßmer
  • Semyon Proshev
  • Sergey Passichenko
  • Suren Nihalani
  • Svyatoslav Ilinskiy
  • Teemu R
  • Thomas Grainger
  • Yusuke Miyazaki

— Jukka (on behalf of the rest of the mypy team: Guido, David and Greg)