Tuesday, 22 June 2021

Mypy 0.910 Released

We’ve just uploaded mypy 0.910 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This is a small feature release that includes a new command-line option --non-interactive for installing stub (type) packages without asking for a confirmation, fixes to regressions, and a few other improvements. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Installing Stub Packages Non-Interactively

Mypy 0.900 added --install-types to install missing stub packages. We received feedback that this wasn’t a good fit for all use cases, since it asks for interactive confirmation from the user, and it requires another mypy invocation to actually perform type checking.

This release adds the new option --non-interactive that can be used with --install-types to install suggested stub packages without asking for confirmation. This can be useful in Continuous Integration jobs. The option causes mypy to both install stub packages and perform type checking within a single invocation. When not using --non-interactive, you’d have to run mypy again to get up-to-date results with the installed stubs.

Here’s an example of using --non-interactive:

    $ mypy --install-types --non-interactive src/
    Installing missing stub packages:
    /Users/jukka/venv/mypy/bin/python3 -m pip install types-requests
    
    Collecting types-requests
      Downloading types_requests-2.25.0-py3-none-any.whl (22 kB)
    Installing collected packages: types-requests
    Successfully installed types-requests-2.25.0
    
    Success: no issues found in 15 source files

This option provides a new way to migrate existing mypy runner scripts after updating to mypy 0.9xx: just add the --install-types --non-interactive options to your mypy command line.

Note that --install-types currently always installs the latest stubs for all supported packages. If you want reproducible results from your builds, we recommend explicitly pinning stub package versions in your requirements.txt file, for example.

Python 3.5 Deprecation

Running mypy on Python 3.5 is now deprecated. A future mypy release will drop Python 3.5 support. We haven’t decided when this will happen, but this might happen in the next feature release after 0.910.

Stubgen Improvements

  • Don't annotate unknown argument and return types (Sebastian Rittau, PR 10626)
  • Never generate a variable initializer (Sebastian Rittau, PR 10623)
  • Use NamedTuple class syntax (Sebastian Rittau, PR 10625)
  • Use T | None (PEP 604) instead of Optional[T] (Sebastian Rittau, PR 10624)

Other Fixes and Improvements

  • Fix some crashes from faulty casts (Shantanu, PR 10560)
  • Update docs for deferral of PEP 563 to 3.11 (Smart, PR 10655)
  • Don't suggest to install stubs for packages with py.typed files (Sebastian Rittau, PR 10652)
  • Correct the type package name of pyopenssl ( Sebastian Rittau, PR 10656)
  • Improve error reporting when --install-types has no cache (PR 10667)
  • Suggest types-setuptools for pkg_resources (Sebastian Rittau, PR 10681)
  • Fix crash with assignment to variable guarded with TypeGuard (PR 10683)
  • Don't ask to install a stub package if stubs are installed (PR 10670)
  • Fix crash when inferring multiple assignment with overloaded function (PR 10689)

Typeshed Updates

There are no typeshed updates in this release.

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Sebastian Rittau
  • Shantanu
  • Smart

We’d also like to thank our employer, Dropbox, for funding the mypy core team.

Tuesday, 8 June 2021

Mypy 0.900 Released

We’ve just uploaded mypy 0.900 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes many new features and bug fixes. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Update: Released mypy 0.901 that fixes a crash when using --install-types.

Third-party Library Stubs in Stub Packages (Breaking Change)

Mypy now only ships with type stubs for stdlib modules (plus typing_extensions and mypy_extensions). If you use third-party libraries that don’t include stubs or inline type annotations, you can explicitly install stub packages, usually called types-<name>.

For example, if you use requests, you can install stubs like this:

    python3 -m pip install types-requests

If mypy can’t find stubs, it will suggest the name of the stub package to install:

  t.py:1: error: Library stubs not installed for "requests" 
      (or incompatible with Python 3.8)
  t.py:1: note: Hint: "python3 -m pip install types-requests"
  t.py:1: note: (or run "mypy --install-types" to install all 
      missing stub packages)
  t.py:1: note: See https://mypy.readthedocs.io/en/...

You can also now run mypy --install-types to install all missing stub packages in the mypy run (or the previous run, if you don’t pass any files to check):

  $ mypy --install-types
  Installing missing stub packages:
  /Users/jukka/venv/mypy/bin/python3 -m pip install types-requests

  Install? [yN] y

  Collecting types-requests
    Using cached types_requests-0.1.8-py2.py3-none-any.whl (22 kB)
  Installing collected packages: types-requests
  Successfully installed types-requests-0.1.8

Note: --install-types isn’t supported by mypy daemon.

Note: For security reasons, the suggested stubs are limited to only a small subset of manually selected packages that have been verified by the typeshed team. These packages include only stub files and no executable code.

Note: Even if you are using --ignore-missing-imports, mypy will ask to install stubs for packages that had bundled stubs in previous mypy releases. This helps you avoid losing type checking precision from missing stubs when upgrading to mypy 0.900.

For more information, read the blog post about this change and why we did this. There is more detail in the docs.

Python 2 Support is Opt-in (Breaking Change)

If you want to type check Python 2 code, you now have to install mypy[python2] using pip (on Python 3.8 and later):

    python3 -m pip install -U mypy[python2]

This will install the additional typed-ast pip dependency that is required for Python 2 support. Type checking Python 2 code is still fully supported by mypy.

This was contributed by Shantanu (PR 10183).

TypeGuard

Mypy now supports defining user-defined functions that perform type narrowing, similar to isinstance. Read PEP 647 for the details. Here’s an example adapted from the PEP:

    from typing_extensions import TypeGuard

    def is_str_list(val: List[object]) -> TypeGuard[List[str]]:
        """Are all list items strings?"""
        return all(isinstance(x, str) for x in val)

    def func(val: List[object]) -> None:
        if is_str_list(val):
            print(" ".join(val))  # OK

Since TypeGuard will be included in Python 3.10, which hasn’t been released yet, you will have to import it from a recent version of typing_extensions for now.

This was contributed by Guido van Rossum (PR 9865).

Support for pyproject.toml

You can now use a pyproject.toml file instead of mypy.ini for mypy configuration. Here is an example from the documentation:

    # mypy global options:

    [tool.mypy]
    python_version = "2.7"
    warn_return_any = true
    warn_unused_configs = true

    # mypy per-module options:

    [[tool.mypy.overrides]]
    module = "mycode.foo.*"
    disallow_untyped_defs = true

    [[tool.mypy.overrides]]
    module = "mycode.bar"
    warn_return_any = false

    [[tool.mypy.overrides]]
    module = [
        "somelibrary",
        "some_other_library"
    ]
    ignore_missing_imports = true

This was contributed by Adam Weeden (PR 10219).

Binary Wheels for Apple Silicon

We now include binary wheels for Apple Silicon. This can make mypy runs over 4x faster on supported hardware. This requires a native Apple Silicon build of Python 3.9 to work.

This was contributed by Jingchen Ye (97littleleaf11, PR 10145).

Type Narrowing with type(x)

Mypy now recognizes that checks such as type(x) is str narrow the type of x to str. Previously you had to use isinstance(x, str). Example:

    def example(x: Union[int, str]) -> bool:
        if type(x) is str:
            # Type of "v" is "str" here
            ...
        else:
            # Type of "v" is Union[int, str] here
            # "v" could be a instance of a "str" subclass!
            ...

This was contributed by Pranav Rajpal (PR 10284).

Hosted Documentation for Mypyc

We now have hosted user documentation for mypyc, the compiler we use to speed up mypy. Mypyc is still experimental, but we are looking for early adopters.

New Mypyc Features

  • Support __setitem__, __delitem__, __len__ and __contains__ in native classes (PR 10451)
  • Basic support for class-based named tuples (PR 10252)
  • Support class-based TypedDict definitions (PR 10226)

Mypyc Performance Improvements

  • Pre-allocate space in list comprehensions (97littleleaf11, PR 10295)
  • Faster dict true test (97littleleaf11, PR 10333)
  • Implement dict.setdefault primitive (97littleleaf11, PR 10286)
  • Optimize truth value testing for strings (pranavrajpal, PR 10269)
  • Faster set creation with generator expression (97littleleaf11, PR 10261)
  • Add a primitive for loading the bool type (97littleleaf11, PR 10257)
  • Faster creation of a tuple from list/tuple (97littleleaf11, PR 10217)
  • Use vectorcalls to speed up calls to Python objects (PR 10153)
  • Add primitive for str.replace (97littleleaf11, PR 10088)
  • Add primitives for list.index() and list.remove() (PR 9961)
  • Intern string literals (PR 9960)
  • Add fast path for simple calls in wrapper functions (PR 9948)
  • Use METH_FASTCALL with __call__ (PR 9946)
  • Use faster METH_FASTCALL wrapper functions on Python 3.7+ (PR 9894)
  • Add primitives for list.sort() and list.reverse() (PR 9897)
  • Recognize six.moves.xrange as an alias of range (PR 9896)
  • Some tagged integer micro-optimizations (PR 9801)
  • Add primitive for dict.copy (vsakkas, PR 9721)

Other Mypyc Fixes and Improvements

  • Fix reference counting in dict.setdefault (97littleleaf11, PR 10334)
  • Don't coerce types checked with isinstance (Sara Sinback, PR 10245)
  • Simplify argument parsing in legacy wrapper functions (PR 10234)
  • Simplify generated C by omitting gotos to the next block (PR 10230)
  • Box initializers of final variables when required (PR 10229)
  • Fix compilation of unreachable expressions (PR 10228)
  • Foundational support for tuple literals (and None and bool literals, PR 10148)
  • Use tables to construct and store literals (PR 10147)
  • Fix overflow error handling in list.insert (PR 9895)
  • Always add implicit None return type to __init__ method (Thomas Johnson, PR 9866)
  • Small documentation updates (Thomas Johnson, PR 10184)
  • Updates to introduction in the documentation (PR 10179)
  • Various small doc updates (PR 10177)
  • Make mypyc docs use the same theme as mypy docs (PR 10176)
  • Fix mypyc failing to compile on CPython 3.10.0a6 (Adrian Freund, PR 10202)
  • Avoid declaring variable in C for loop initialization (Thomas Johnson, PR 10091)

Stubgen Improvements

  • Properly convert overloaded functions (Chad Dombrova, PR 9613)
  • Fix invoking mro() (Gen Xu, PR 10138)
  • Render a bit better stubs (Sergei Izmailov, PR 9903)
  • Fixed Any and Dict missing from stubgen imports list (Ashley Whetter, PR 9900)
  • Fix type error (Shantanu, PR 10523)
  • Fixes and typos (Sergei Izmailov, PR 9877)

Stubtest Improvements

  • Improve build error messages (Shantanu, PR 10365)
  • Check overloads are functions, don't early return (Shantanu, PR 10258)
  • Fix signature construction for overload + implicit classmethod (Shantanu, PR 9921)

Documentation Updates

  • Add security documentation for --install-types (Adam Weeden, PR 10555)
  • Improve documentation of extending mypy (Nikita Sobolev, PR 10222)
  • Update documentation links (Shantanu, PR 10159)
  • Fix incomplete error codes documentation (Iulian Onofrei, PR 9799)
  • Document explicit_package_bases config file option (Sam Bull, PR 10020)
  • Clarify that mypy_path is relative to current working directory (Gustav Gränsbo, PR 9959)
  • Emphasise that users probably want --follow-imports=normal (Shantanu, PR 9955)

Other Notable Fixes and Improvements

  • Make ignore_missing_imports work for third-party libraries which previously had bundled stubs (PR 10582)
  • Restrict the number of errors shown when there are missing stubs (PR 10579)
  • Mention fullname of PlaceholderNode in assertion to simplify debugging (Adam Weeden, PR 10565)
  • Use double quotes in error messages instead of single quotes (Dixith)
  • Use double quotes in various error messages (Prasanth Chettri, PR 10053)
  • Fix crash on TypeGuard plus "and" (Jelle Zijlstra, PR 10496)
  • Fix spurious name undefined error in class body within import cycle (PR 10498)
  • Remove assertion from TypeGuard serialization (Jelle Zijlstra, PR 10486)
  • Do not allow to use Final and ClassVar at same time (Yurii Karabas, PR 10478)
  • Allow running mypy if Python 2 typeshed stubs aren't installed (PR 10494)
  • Allow inline comments in VERSIONS (Sebastian Rittau, PR 10472)
  • Infer unreachability for await no_return() (Shantanu, PR 10458)
  • Fix unresolved reference + tuples crashing (EXPLOSION, PR 10401)
  • Fix crash with type alias inside __init__ in incremental mode (PR 10432)
  • Fix crash with nested NamedTuple in incremental mode (PR 10431)
  • Fix crash related to module-level __getattr__ in incremental mode (PR 10430)
  • Add support for functools.cached_property (Marc Mueller, PR 10408)
  • Don't narrow Type[X] with a metaclass (PR 10424)
  • Remove note: setup.cfg does need [mypy] section (James Cooke, PR 10364)
  • Fix inference of IntEnum value attribute type (Python 2, PR 10417)
  • Don't report error when using total_ordering in Python 2 mode (PR 10416)
  • Fix narrowing down TypedDict unions with enum literal types (PR 10415)
  • Fix inference of IntEnum value attribute type (PR 10412)
  • Use typeddict-item error code for more errors (PR 10410)
  • Fix a bunch of tests broken by python 3.10 (Adrian Freund, PR 10404)
  • Support maximum version marker of stdlib typeshed modules (Sebastian Rittau, PR 10402)
  • First look into @python2 subdirectory of search path items in Python 2 mode (PR 10392)
  • Make enum type compatible with union of all enum item literals (PR 10388)
  • Fix an issue with union types containing literals (Adrian Freund, PR 10373)
  • Fix re-exporting __all__ in a stub file (PR 10382)
  • Extend the dataclass plugin to deal with callable properties (Aaron Ecay, PR 10292)
  • Make TypedDict incompatible type message more understandable (Akshay K, PR 10326)
  • Make --explicit-package-bases invertible (Gustav Gränsbo, PR 9969)
  • Fix crash for protocol classes and Hashable false negative (Christoph Tyralla, PR 10308)
  • Fix strict equality false positive when strict optional disabled (Kamil Turek, PR 10174)
  • Issue an error when overriding typeguard with non-typeguard in subclass (Kamil Turek, PR 10300)
  • Support functools.total_ordering (Ashley Whetter, PR 7831)
  • Support reversed operand order when comparing against sys.version_info (Kamil Turek, PR 10288)
  • Only import distutils if it is needed (Adrian Freund, PR 10203)
  • Recombine complete union of enum literals into original type (PR 9097, PR 9063) (Ethan Leba)
  • Support union type arguments when showing protocol member conflicts (97littleleaf11, PR 10154)
  • Fix recently added enum value type prediction (Paddy, PR 10057)
  • Support subclassing of NodeVisitor and TraverserVisitor in plugins (Nikita Sobolev, PR 10125)
  • Speed up processing of new files in daemon by caching ASTs (PR 10128)
  • Add narrowing and closures to common issues (Shantanu, PR 9956)
  • Add Python script to build wheels using cibuildwheel (PR 10096)
  • Don't try to follow imports to encodings.* (PR 10094)
  • Speed up case sensitive is-file check in file system cache (PR 10093)
  • Use fast path for finding modules in mypy daemon (PR 10095)
  • Fix following stub imports in daemon when using --follow-imports=skip (PR 10092)
  • Use sys.executable instead of python3 for --install-types (Anthony Sottile, PR 10086)
  • Add build-requirements.txt to MANIFEST.in (PR 10071)
  • Add separate requirements file for mypy self-compile (PR 10069)
  • Expand user home for cache_dir (Marco Zatta, PR 10051)
  • Fix daemon crash when deleting packages (PR 10036)
  • Keep track of fine-grained dependencies for modules in typeshed (PR 10034)
  • Fix import following issue in daemon (PR 10032)
  • Fix mypy daemon crash when following from a new module (PR 10030)
  • Detect invalid value for --custom-typeshed-dir (PR 9985)
  • Fix pip install for new typeshed (Ivan Levkivskyi, PR 9976)
  • Report incompatible assignments for descriptors with overloaded __set__ methods (Christoph Tyralla, PR 9893)
  • Fix TypedDict.get("missing_key") with string literal (Adrian Freund, PR 9906)

Acknowledgements

Thanks to all mypy contributors who contributed to this release:

  • Aaron Ecay
  • Adam Weeden
  • Adrian Freund
  • Akshay K
  • Akuli
  • Anthony Sottile
  • Arjun
  • Ashley Whetter
  • Chad Dombrova
  • Christoph Tyralla
  • Dixith
  • Ethan Leba
  • EXPLOSION
  • Gen Xu
  • Gustav Gränsbo
  • hatal175
  • Iulian Onofrei
  • James Cooke
  • Jelle Zijlstra
  • Jesús Cea
  • Jingchen Ye (97littleleaf11)
  • Kamil Turek
  • Kyle Verhoog
  • Marc Mueller
  • Marco Gorelli
  • Marco Zatta
  • Nikita Sobolev
  • Paddy
  • Philip Jägenstedt
  • Pranav Rajpal
  • Prasanth Chettri
  • Raphael Gaschignard
  • Reda Bouaida
  • Saiprasad Kale
  • Sam Bull
  • Sara Sinback
  • Sebastian Rittau
  • Sergei Izmailov
  • Shantanu
  • Thomas Johnson
  • vsakkas
  • wwuck
  • Xuanda Yang
  • Yurii Karabas

Additional thanks to all contributors to typeshed:

  • Adrian Freund
  • Akuli
  • Alexander Reynolds
  • Andre Delfino
  • Andrew Crabtree
  • Andrew Zhou
  • Andrey
  • Ankur Singh
  • Anthony Sottile
  • Anton Agestam
  • Arie Bovenberg
  • Árni Már Jónsson
  • Ashwin Ramaswami
  • baterflyrity
  • Cássio Botaro
  • Christopher Dignam
  • Christopher Gurnee
  • Christopher Sang
  • Daniel Rosenwasser
  • danieleades
  • Dante Camarena
  • David Barker
  • Dean Way
  • dearoneesama
  • dod-wdwilcox
  • Dominic Davis-Foster
  • Edgar Handal
  • Eric Traut
  • Erik Soma
  • Giel van Schijndel
  • Gilad Reti
  • Greg Ward
  • Guilhem C
  • Guillaume Ayoub
  • harens
  • hatal175
  • Henry Schreiner
  • heroesm
  • Igor Nehoroshev
  • Ilya Konstantinov
  • Iwan
  • jack1142
  • Jacob Hayes
  • Jake Bailey
  • James
  • James Morris
  • Jason Huang
  • Jelle Zijlstra
  • Jia Chen
  • John T. Wodder II
  • Jon Banafato
  • Jon Dufresne
  • JonathanMeans
  • Jörg Herbel
  • Jürgen Gmach
  • Kaleb Barrett
  • Ken Odegard
  • krouziciorel
  • Kyle Altendorf
  • ldruschk
  • Mahmoud Harmouch
  • Marat Sharafutdinov
  • Mark Vismonte
  • Marks Polakovs
  • Marti Raudsepp
  • Mathieu Parent
  • Maxime Arthaud
  • Mohit Goyani
  • MrGreenTea
  • n0ob404
  • naglis
  • Nick Crews
  • Nick M
  • Nipunn Koorapati
  • Philipp Hahn
  • Rebecca Chen
  • Ricardo
  • rijenkii
  • Rune Tynan
  • Ryan McCampbell
  • sachindavra
  • Sam Bull
  • Sean B. Palmer
  • Sebastian Rittau
  • Shantanu
  • Sorin Sbarnea
  • spaghEddieDoyle
  • Stanislav Levin
  • Steve B
  • Tadek Kijkowski
  • Takumi Kato
  • Terence Honles
  • Thomas Cellerier
  • Tigran Kostandyan
  • Tushar Sadhwani
  • Udi Fuchs
  • Utsav
  • Vasily Zakharov
  • Viktor Roytman
  • Vyom Pathak
  • Wilfredo Sánchez Vega
  • Wolf Honore
  • Yehuda Davis

Wednesday, 26 May 2021

The Upcoming Switch to Modular Typeshed

Mypy has been bundling stubs for many third-party libraries from typeshed, a repository of type definitions for Python packages. Mypy uses the stubs to type check code that uses these libraries, and they allow more types to be inferred by mypy. Typeshed is not just for mypy — it’s also used by several other tools in the Python typing ecosystem.

Mypy will no longer include stubs for third-party libraries in the soon-to-be-released mypy 0.900 release. Instead, mypy will only ship stubs for standard library modules, while stubs for third-party libraries can be independently installed using pip. For example, if you use requests, you can install the stubs like this:

  python3 -m pip install types-requests

The stub packages are named types-<project name>. Note that the project name may be different from the runtime package name. For example, stubs for the yaml package are available in types-PyYAML.

Steps for Mypy Users

When you update to mypy 0.900 (once it has been released), you’ll likely get errors about missing stubs. Mypy will suggest the stub package that should be installed. Here’s an example output from mypy:

  a.py:1: error: Library stubs not installed for "requests"
  a.py:1: note: Hint: "python3 -m pip install types-requests"
  a.py:1: note: (or run "mypy --install-types" to install all 
      missing stub packages)
  a.py:1: note: See https://mypy.readthedocs.io/en/...

You can now run mypy --install-types to install all the missing dependencies (unless you are using mypy daemon). Alternatively, you can install the stubs manually using pip as suggested by mypy.

As a final step, you may want to add the stub packages to the build or test dependencies of your project, often in requirements.txt. You can use pip freeze to help pin to the current installed versions of the stubs. For example, you might add this line to your requirements.txt:

  types-requests==0.1.8

Currently stub packages are mostly versioned independently of the implementation package. However, it’s possible to update stub metadata in typeshed to have the versioning better reflect the target implementation package version.

Motivation

The bundling of stubs with mypy has been convenient, but it also has caused a lot of friction, especially for users with many third-party dependencies. Here are the main benefits of unbundling stubs:

  • Updating stubs without updating mypy. Right now it’s not straightforward to update bundled stubs to a newer version without also updating mypy. Soon you can use pip to update to the latest stubs, without having to update to mypy as well.
  • Updating mypy without updating stubs. Updating stubs or installing new stubs sometimes results in additional mypy errors, due to newly added but too restrictive types in stubs, or newly introduced but correct types that expose incorrect types in your code. Previously mypy users had to deal with these errors when updating mypy, which complicated the update process. Soon you’ll be able to update mypy without updating third-party stubs. Standard library stubs will still be bundled for convenience, since all mypy users need them, and they tend to be easier to validate for correctness.
  • Faster access to new stubs and improvements. If new stubs or stub improvements are contributed to typeshed, previously you had to wait for a new mypy release that includes them, or deal with a custom, unsupported typeshed set-up that hasn’t been properly tested. With mypy 0.900, you will be able to install or update stub packages whenever you want. As soon as a change to a stub is merged, typeshed will automatically upload a fresh stub package to PyPI. If you are not happy with some stub changes or encounter an issue, you can easily contribute fixes and start using the updated stubs, or continue to use older stubs if you so wish.
  • Lower barrier to contributing new stubs. Since previously every user would get all contributed stubs, each new stub was a somewhat risky proposition. If some types were incorrect, or if there were missing definitions in the stubs, this could result in many false positives. Modular typeshed lets each user individually decide whether to start using the contributed stubs, making the impact of glitches in stubs lower. The faster iteration speed should also make it more likely that issues will be fixed soon.
  • Consistency. Previously some third-party library stubs were bundled with mypy, some were bundled with the main implementation (upstream) package, and some were available on PyPI as separate stub packages. Now there will be one fewer option: you’d either use stubs or types included with the implementation package, or install a stub package. Types included with the main package are the preferred option, but many packages still don’t have them. Stub packages will be common for some time. Installing, versioning and updating stubs and implementation packages will also be more similar.
  • Support for multiple library versions. Libraries may introduce backward incompatible API changes, such as removing deprecated features. Sometimes, for various reasons, you may want to continue using an older version of a library. Now it will be easy to also continue using an older version of stubs, since library stubs are independently versioned. This also opens the option of maintaining separate stubs for versions 1.x and 2.x of a library, for example.

Vendoring Stubs

If you don’t want to deal with installing stub packages, mypy 0.900 will support vendoring stubs (i.e., maintaining a local copy in your repository). First, you can copy some stubs from typeshed into a directory, and point the MYPYPATH environment variable (or mypy_path in the mypy configuration file) to it. Mypy will look for stubs in that directory. For example, you could use a directory named stubs in your repository structured like this:

  • stubs/toml.pyi
  • stubs/yaml/__init__.pyi

If your project supports both Python 2 and 3, you can have stubs for Python 2 in a @python2 subdirectory of the main stub directory:

  • stubs/foobar.pyi (used for Python 3)
  • stubs/@python2/foobar.pyi (used for Python 2)

If you include stubs in the mypy search path, stubs/@python2 will automatically be used for Python 2 stubs.

Feedback?

Feel free to open a GitHub issue if you have concerns about this new approach of distributing stubs.

Acknowledgements

Ivan Levkivskyi set up the infrastructure for automatically uploading stub packages to PyPI and implemented the necessary changes to typeshed.

Wednesday, 17 March 2021

Summer of Code with Mypy

TH3CHARLie, Xuanda Yang
th3charlie at gmail dot com
GitHub Profile
GSoC Project Page
GSoC Submission Page

Introduction

In Fall 2019, I was writing a course project in Python and was tired of the experience of finding type errors at the last minute, so I did some searching and tried mypy for the first time. At that time, I would not dream of doing a three-month coding project with the organization, which became reality last summer. In this post, I'd like to share the story of this very personal experience and talk about how I started to contribute to mypy and eventually became a Google Summer of Code (GSoC) student of this project.

Pre-application

After finishing my course project with mypy, out of curiosity, I took a look at its GitHub page and paid extra attention to the issue tracker. I searched for issues with the "good first issue" label and was surprised to find that most of them were equipped with comments from the maintainers telling how to tackle them. Reading through these comments, I found some issues that fit my level and started to take a try at them.

Before submitting a pull request, I read through the repository and checked every material that seemed useful for writing a PR, which includes the README file, the documentation and the wiki page. I learned about how to install the development version of mypy, how to test my PR locally and some high-level ideas about the codebase. This proves to save a lot of trouble in my future contributions to the project. My first PR to the project was Support Negative Int Literal and was reviewed by Ivan. He patiently guided me to refine the PR step by step and finally merged it into the main branch. The satisfaction of seeing my code entering the codebase of a tool used by people around the globe was huge and encouraged me to take more attempts on other good first issues. By the end of 2019, I had submitted around 20 PRs to the repository. Most of them were super simple tasks, like producing meaningful error messages or fixing a corner case of a specific type-checking rule.

Looking back now, writing all these simple PRs helped me gradually familiarize myself with the codebase and the development workflow, which became a valuable asset when applying to the GSoC project.

Applying to GSoC

Google Summer of Code (GSoC) is an annual event where students get to work with open source communities through the summer and even get some stipends from Google. Before Google announced last year's selected organizations and project ideas on Feb 20th, one of mypy's maintainers posted an issue named Project ideas listing several time-consuming project ideas. At that time, I was eager to try something more challenging than fixing small bugs so I spent some time reading them through and tried to figure out which of them fit my level. A few days later, the official organization list for GSoC 2020 was out and mypy was one of them and the previously mentioned issue included the potential project ideas. What a great match, I said to myself. So I made my mind and started to draft my application.

The student application of GSoC usually consists of two parts. The first is (are) mandatory project proposal(s). Each student can choose up to three project ideas from three different organizations. Finding suitable project ideas from hundreds of them is usually hard since understanding the idea descriptions already requires effort and even some domain knowledge. Thanks to my previous contribution experience at mypy, I didn't spend too much time on this matter and quickly narrowed my topic to compiler related ones. Generalize mypyc IR to Make non-C Backends Possible from mypy and another topic from LLVM were my choices. However, the latter one was picked by someone else through the mailing list before I even had the chance to take a closer look at it. So mypy became the only organization left for me to apply to.

To write a good proposal, I started to discuss directly with the maintainers (or future mentors) in the issue tracker. I'd say this is one of the best approaches to prepare your proposal. By doing this, you are essentially expressing your passion, demonstrating your background and skills directly to the mentors and the mentors can provide details of the project that help you get a better picture. Don't be afraid to discuss, open-source communities often have very nice people and mypy is definitely one of them. Jukka Lehtosalo and Michael (Sully) Sullivan provided tons of good advice that helped me formulate a plan for the project and therefore finish my proposal. I also came up with a prototype PR addressing one core change of my proposal before the deadline. Although it never got merged since we did it differently later, a working prototype is always welcomed and I'd say it would increase your chance of getting accepted.

The second and also the optional part is the organization-specific requirement(s). Last year for applicants to mypy, there were some posted requirements including course background, programming skills, submitted PRs and an email giving details.

After I submitted my proposal to GSoC's official site and sent an email to the mypy team, I did a short video interview with the mypy team. I talked about what I would do to make the project successful and also talked a little more about my background other than what was written in the email. The interview went quite well and several days later, I received an email from the mypy team and was told I was selected by the organization. Then all I had to do was to wait for the official announcement from Google. And days later, I officially became the first GSoC student of the mypy project and would start my coding summer soon.

Coding Through the Summer

After getting accepted to GSoC, what's left is to code through the summer. The project I worked on was Generalize mypyc IR to Make non-C Backends Possible. In short, it was a project to redesign and refactor some parts of the old mypyc IR to remove the embedded C backend-specific information. You can find the detailed technique report on the submission page. Here, I'd like to talk about other things that play as important a role as coding.

Plan the schedule and monitor the progress. Three months is just a blink of an eye, so it's important to have realistic goals to achieve at specific timestamps, to make sure the project goes smoothly. We had a rough schedule in the proposal and established monthly goals in later discussions. With the schedule and goals in mind, I was able to make progress continuously. I even had an extra week. Usually, before the coding period starts in June, there is one month for community bonding where students get to familiarize themselves with the workflow of the project and connect with the mentors. Thanks to my previous mypy contribution experience, I thought it would be fine to skip this period and start coding one week earlier than scheduled. This extra week gave my mentors and me a lot of flexibility to handle unexpected schedule delays.

Communication is the key. Jukka, one of my mentors, told me that it was always hard to come up with the perfect design at first as requirements and needs develop all the time. To face this developing nature, we communicated a lot. I had daily syncs with my mentors through Gitter chat, weekly meetings with Jukka, and monthly meetings with both Jukka and Sully. In all these meetings, I shared recent progress, they helped me remove any blockers, and we discussed component designs. I am one-hundred percent sure that this project would never have been this successful without such frequent and effective communication.

After the Summer

I completed this project and submitted the required materials in early September, with around 60 PRs merged into the repository. To me, the GSoC project ended pretty well, I received the certificate from Google as well as some stipends. More importantly, I've demonstrated my passion and skills to contribute to this organization more than just a community contributor. I was honored to become one of mypy's committers. That means a lot to me and I am looking forward to contributing more to the community.

That's my story with mypy through the Google Summer of Code 2020. I hope this post will provide some useful information to future potential applicants including applicants of this year since mypy has officially been selected as one of this year’s organizations. If you have any questions regarding this post, feel free to send me an email and I will be more than happy to discuss.

Acknowledgements

I'd like to express my sincere gratitude to every person and organization that made my GSoC 2020 adventure possible. I'd like to firstly thank Google and mypy for providing me this opportunity.

Especially, I'd like to thank my mentors Jukka Lehtosalo and Michael Sullivan. We had daily syncs every day on Gitter and weekly/monthly video meetings via Zoom. In every discussion, they helped me clean my thoughts and find out the best approach to meet our goals. They responded quickly to my PRs, giving high-quality review comments and suggestions. They mentored me with patience and passion and I feel connected even though we are several timezones/continents away. Their guidance is even beyond the scope of the project and helps me form good software engineering skills along the way.

I also would like to thank my parents for supporting me working on this project and Yaozhu Sun from the University of Hong Kong, who lit my passion for the field of compilers and programming languages two years ago. Finally, I'd like to thank Kanemura Miku from Hinatazaka 46 for all the mental support during this special summer.

Disclaimer

All content of this post only represents the author's personal opinions. The post does not, in any degree, constitute the view or opinion from Google or the mypy community.

Friday, 19 February 2021

Mypy 0.812 Released

We’ve just uploaded mypy 0.812 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes a fix to a regression in source file finding logic in mypy 0.800, and a new command-line option --exclude to exclude paths from the build. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Improved Source File Finding

Mypy 0.800 changed how mypy finds modules if you run mypy as mypy directory/ or mypy -p package. Mypy started looking for source files in directories without a __init__.py file. This is often the expected behavior, and it avoids excluding some files that should be type checked.

However, this caused issues for some users, such as when using mypy . to type check all files under the current directory. Mypy could now try to type check files inside nested virtual environments and node_modules directories, which is usually not desirable. This could result in mypy needlessly complaining about duplicate module names, in particular.

Now mypy will skip directories named site-packages or node_modules, and any directory beginning with a dot (such as .git) when recursively looking for files to check.

This doesn’t affect how mypy resolves imports — it only affects when mypy is given a directory or a package to type check. You can override the exclusions by explicitly passing the files on the command line.

This was contributed by Shantanu (PR 9992).

Excluding Paths

Mypy now supports the --exclude regex command line option to exclude paths matching a regular expression when searching for files to type check. For example, mypy --exclude '/setup\.py$' skips all setup.py files. This lets you exclude additional paths that mypy started finding after mypy 0.800 changed module finding behavior, as discussed above.

You can also specify this in the config file (exclude=regex). The option expects forward slashes as directory separators on all platforms, including Windows, for consistency.

This was also contributed by Shantanu (PR 9992). See the documentation for more details.

Typeshed Updates

There are no typeshed updates in this release.

Acknowledgments

Thanks to Shantanu who contributed to this release.

We’d also like to thank our employer, Dropbox, for funding the mypy core team.

Friday, 22 January 2021

Mypy 0.800 Released

We’ve just uploaded mypy 0.800 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Python 3.9 Support

Mypy 0.800 officially supports the recently released Python 3.9. We now provide compiled binary wheels for Python 3.9, improving type checking speed significantly.

Typing Usability Improvements (PEP 585 and PEP 604)

The necessity to repeatedly import various types and special forms from typing has been a long-term nuisance for users of static type checking and Python.

Two new Python features improve this situation and are now supported by mypy:

  • PEP 585 lets you use list[int] instead of List[int] (no need to import List and other generic collections from typing).
  • PEP 604 lets you write X | Y instead of Union[X, Y], and X | None instead of Optional[X] (no need to import Union or Optional from typing).

Note: Using list[int] requires Python 3.9 and X | Y requires Python 3.10 (alpha) in order to work at runtime. To use them on older versions of Python, use from __future__ import annotations. This allows them to be used in type annotations, but the older variants (or string literal escaping) may be required in non-annotation contexts, such as in type aliases. See the docs for more details.

Here is an example that uses the new features:

    from __future__ import annotations
    
    def fields(s: str | None) -> list[str]:
        if not s:
            return []
        else:
            return s.split(',')

These were implemented by Allan Daemon in PR 9564 and by Marc Mueller in PR 9647.

Improvements to Finding Modules

This release adds several improvements to how mypy finds Python source files to type check.

You can now pass paths to files within namespace packages on the command line, and mypy can better infer their module names. As before, use --namespace-packages to enable namespace packages.

When you use --explicit-package-bases together with --namespace-packages, mypy assumes that only the current directory and directories explicitly specified in MYPYPATH (or mypy_path in the config file) are valid package roots. This can help with situations where the module name of a file is ambiguous. For example, it may not be clear whether src/pkg/mod.py should be treated as src.pkg.mod or pkg.mod, and you can use this option to disambiguate between the two (more information in the docs).

The above improvements were implemented in PR 9742 by Shantanu.

Other related improvements (also implemented by Shantanu):

  • When you run mypy as mypy <directory>, look for source files recursively also inside directories without a __init__.py (PR 9614)
  • Support namespace packages with -p (PR 9683)
  • Log encountered source files when running mypy with -v (PR 9672)
  • Document the new module finding behavior (PR 9923)

Other Notable Improvements and Bug Fixes

  • Only treat import X as X as a re-export in stubs (Shantanu, PR 9515)
  • Fix package imports with aliases in stubgen (Chad Dombrova, PR 9534)
  • Require first argument of namedtuple() to match the variable name (Momoko Hattori, PR 9577)
  • Add error code for name mismatches in named tuples and TypedDicts to make it easy to disable these error messages (Jukka Lehtosalo, PR 9811)
  • Document local_partial_types config option (Momoko Hattori, PR 9551)
  • Improve ambiguous kwargs checking (Erik Soma, PR 9573)
  • Disable unreachable warnings in boolean operators for type variables with value restrictions (Vincent Barbaresi, PR 9572)
  • Allow assignment to an empty tuple (Tobin Yehle, PR 5617)
  • Use absolute path when checking source duplication error (Yuki Igarashi, PR 9059)
  • Add get_function_signature_hook() to the plugin API (Nikita Sobolev, PR 9102)
  • Speed up type checking of dictionary, set, and list expressions (Hugues, PR 9477)
  • Allow non-types as arguments in Annotated (Patrick Arminio, PR 9625)
  • Add support for next generation attrs API (David Euresti, PR 9396)
  • Fix case folding of missing keys error message for TypedDicts (Marti Raudsepp, PR 9757)
  • Fix generic inheritance of __init__() methods in dataclasses and attrs classes (Nate McMaster, PR 9383, PR 9380)
  • Add more information to error message on too few arguments (Abhinay Pandey, PR 9796)
  • Document PEP 585, 563, 604, and related functionality (Shantanu, PR 9763)

Mypyc Improvements

We use mypyc to compile mypy into fast C extension modules. This release includes many mypyc improvements.

Xuanda Yang finished the migration to use a new, lower-level compiler intermediate representation in his Google Summer of Code project.

New supported Python features:

  • Support the walrus operator (:=) (Michael J. Sullivan, PR 9624)

Performance improvements:

  • Add primitives for list sort and list reverse (Jukka Lehtosalo, PR 9897)
  • Recognize six.moves.xrange again as an alias of range (Jukka Lehtosalo, PR 9896)
  • Speed up some integer primitives (Jukka Lehtosalo, PR 9801)
  • Speed up if x for int values (Jukka Lehtosalo, PR 9854)
  • Implement dict clear primitive (Vasileios Sakkas, PR 9724)
  • Implement list insert primitive (Vasileios Sakkas, PR 9741)
  • Implement float abs primitive (Xuanda Yang, PR 9695)
  • Implement str-to-float primitive (Xuanda Yang, PR 9685)
  • Specialize some calls to frozenset (Michael J. Sullivan, PR 9623)
  • Speed up multiple assignment from tuple (Xuanda Yang, PR 9575)
  • Speed up multiple assignment from sequence (Jukka Lehtosalo, PR 9800)
  • Optimize startswith and endswith (Tomer Chachamu, PR 9557)
  • Add primitives for bitwise ops (Jukka Lehtosalo, PR 9529)
  • Speed up in operations for list/tuple (Johan Dahlin, PR 9004)
  • Add primitives for list, str and tuple slicing (Jukka Lehtosalo, PR 9283)
  • Speed up tuple equality checks (Xuanda Yang, PR 9343)

Bug fixes:

  • Always add implicit None return type to __init__ method (Thomas Johnson, PR 9866)
  • Fix deallocation of deeply nested data structures (Michael J. Sullivan, PR 9839)
  • Fix using package imported inside a function (Jukka Lehtosalo, PR 9782)
  • Fix type of for loop index register in for over range (Jukka Lehtosalo, PR 9634)

Typeshed Updates

Many improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Abdullah Selek
  • Abhinay Pandey
  • Adam
  • aghast
  • Akuli
  • Alexander
  • Allan Daemon
  • Aristotelis Mikropoulos
  • Ashley Whetter
  • Brian Mboya
  • Bryan Forbes
  • cdce8p
  • Chad Dombrova
  • David Euresti
  • Denis Laxalde
  • Eisuke Kawashima
  • Erik Soma
  • Ethan Pronovost
  • Florian Bruhin
  • Frank Dana
  • Greg Compestine
  • Guido van Rossum
  • Hugues
  • Jake Bailey
  • Jakub Stasiak
  • Jelle Zijlstra
  • Jeremy Metz
  • Johan Dahlin
  • Jon Shea
  • Jonathan Wong
  • Jürgen Gmach
  • Kamil Turek
  • Krzysztof Przybyła
  • Lawrence Chan
  • Marti Raudsepp
  • Matan Gover
  • Matt Gilson
  • Michael J. Sullivan
  • Momoko Hattori
  • Nate McMaster
  • Nikita Sobolev
  • Nils K
  • Nipunn Koorapati
  • Oleg Höfling
  • Patrick Arminio
  • Rajiv Singh
  • rhkleijn
  • Roland van Laar
  • Shantanu
  • Tobin Yehle
  • Tom Scogland
  • Tomer Chachamu
  • Tripp Horbinski
  • Ville Skyttä
  • Vincent Barbaresi
  • vsakkas
  • Wes Turner
  • willtryagain
  • Xiaodong DENG
  • Xuanda Yang
  • Yash Chhabria
  • Yuki Igarashi

Additional thanks to all contributors to typeshed:

  • Abraham Francis
  • Adam Dangoor
  • Adam Kliś
  • Adam Lichtl
  • Akuli
  • alexander-held
  • an onion
  • Anders Kaseorg
  • Andrew Mitchell
  • Árni Már Jónsson
  • Ash Berlin-Taylor
  • Ashwin Vishnu
  • Avery
  • cdce8p
  • Cebtenzzre
  • Changsheng
  • Christine
  • coiax
  • cptpcrd
  • crusaderky
  • Daniel O'Neel
  • David Caro
  • Dominic Davis-Foster
  • Eric Traut
  • Ethan Pronovost
  • Frank Maximilian
  • Gal Ben David
  • github-actions[bot]
  • Guido van Rossum
  • henribru
  • Hong Xu
  • Hugues
  • Hynek Schlawack
  • jack1142
  • Jake Bailey
  • Jason Fried
  • Jelle Zijlstra
  • Jia Chen
  • Jon Dufresne
  • Jonathan Schoonhoven
  • Jonathan Slenders
  • Joseph Haaga
  • Julien Danjou
  • Jun Jia
  • Jérome Perrin
  • karl ding
  • Katelyn Gigante
  • Kaushal Rohit
  • Kevin Wojniak
  • ky-gog
  • Kyle Fuller
  • kylec1
  • Lam Son Ho
  • Lourens Veen
  • Mahmoud Abduljawad
  • Mariam Maarouf
  • Marti Raudsepp
  • melassa
  • Mickaël Schoentgen
  • Mikhail Sveshnikov
  • Mikołaj Kuranowski
  • Moriyoshi Koizumi
  • Nate McMaster
  • Neel Somani
  • nicolas-harraudeau-sonarsource
  • Nikolaus Waxweiler
  • Nils K
  • Nipunn Koorapati
  • Oleg Höfling
  • Omar Sandoval
  • Paul
  • Pete Scopes
  • Peter Law
  • Philipp Hahn
  • Phillip Huang
  • proost
  • PythonCoderAS
  • Rajiv Bakulesh Shah
  • Ran Benita
  • Raphael Geronimi
  • Rebecca Chen
  • Sam Bull
  • Sebastian Rittau
  • Sergei Lebedev
  • Shantanu
  • Stefano Chiodino
  • Steve Dignam
  • Sténio Jacinto
  • Timur Kushukov
  • Tom Most
  • turettn
  • Unrud
  • Utsav
  • Vasily Zakharov
  • Vincent Barbaresi
  • Vincent Meurisse
  • 愚氓
  • Yuri Khan