Wednesday, 17 March 2021

Summer of Code with Mypy

TH3CHARLie, Xuanda Yang
th3charlie at gmail dot com
GitHub Profile
GSoC Project Page
GSoC Submission Page

Introduction

In Fall 2019, I was writing a course project in Python and was tired of the experience of finding type errors at the last minute, so I did some searching and tried mypy for the first time. At that time, I would not dream of doing a three-month coding project with the organization, which became reality last summer. In this post, I'd like to share the story of this very personal experience and talk about how I started to contribute to mypy and eventually became a Google Summer of Code (GSoC) student of this project.

Pre-application

After finishing my course project with mypy, out of curiosity, I took a look at its GitHub page and paid extra attention to the issue tracker. I searched for issues with the "good first issue" label and was surprised to find that most of them were equipped with comments from the maintainers telling how to tackle them. Reading through these comments, I found some issues that fit my level and started to take a try at them.

Before submitting a pull request, I read through the repository and checked every material that seemed useful for writing a PR, which includes the README file, the documentation and the wiki page. I learned about how to install the development version of mypy, how to test my PR locally and some high-level ideas about the codebase. This proves to save a lot of trouble in my future contributions to the project. My first PR to the project was Support Negative Int Literal and was reviewed by Ivan. He patiently guided me to refine the PR step by step and finally merged it into the main branch. The satisfaction of seeing my code entering the codebase of a tool used by people around the globe was huge and encouraged me to take more attempts on other good first issues. By the end of 2019, I had submitted around 20 PRs to the repository. Most of them were super simple tasks, like producing meaningful error messages or fixing a corner case of a specific type-checking rule.

Looking back now, writing all these simple PRs helped me gradually familiarize myself with the codebase and the development workflow, which became a valuable asset when applying to the GSoC project.

Applying to GSoC

Google Summer of Code (GSoC) is an annual event where students get to work with open source communities through the summer and even get some stipends from Google. Before Google announced last year's selected organizations and project ideas on Feb 20th, one of mypy's maintainers posted an issue named Project ideas listing several time-consuming project ideas. At that time, I was eager to try something more challenging than fixing small bugs so I spent some time reading them through and tried to figure out which of them fit my level. A few days later, the official organization list for GSoC 2020 was out and mypy was one of them and the previously mentioned issue included the potential project ideas. What a great match, I said to myself. So I made my mind and started to draft my application.

The student application of GSoC usually consists of two parts. The first is (are) mandatory project proposal(s). Each student can choose up to three project ideas from three different organizations. Finding suitable project ideas from hundreds of them is usually hard since understanding the idea descriptions already requires effort and even some domain knowledge. Thanks to my previous contribution experience at mypy, I didn't spend too much time on this matter and quickly narrowed my topic to compiler related ones. Generalize mypyc IR to Make non-C Backends Possible from mypy and another topic from LLVM were my choices. However, the latter one was picked by someone else through the mailing list before I even had the chance to take a closer look at it. So mypy became the only organization left for me to apply to.

To write a good proposal, I started to discuss directly with the maintainers (or future mentors) in the issue tracker. I'd say this is one of the best approaches to prepare your proposal. By doing this, you are essentially expressing your passion, demonstrating your background and skills directly to the mentors and the mentors can provide details of the project that help you get a better picture. Don't be afraid to discuss, open-source communities often have very nice people and mypy is definitely one of them. Jukka Lehtosalo and Michael (Sully) Sullivan provided tons of good advice that helped me formulate a plan for the project and therefore finish my proposal. I also came up with a prototype PR addressing one core change of my proposal before the deadline. Although it never got merged since we did it differently later, a working prototype is always welcomed and I'd say it would increase your chance of getting accepted.

The second and also the optional part is the organization-specific requirement(s). Last year for applicants to mypy, there were some posted requirements including course background, programming skills, submitted PRs and an email giving details.

After I submitted my proposal to GSoC's official site and sent an email to the mypy team, I did a short video interview with the mypy team. I talked about what I would do to make the project successful and also talked a little more about my background other than what was written in the email. The interview went quite well and several days later, I received an email from the mypy team and was told I was selected by the organization. Then all I had to do was to wait for the official announcement from Google. And days later, I officially became the first GSoC student of the mypy project and would start my coding summer soon.

Coding Through the Summer

After getting accepted to GSoC, what's left is to code through the summer. The project I worked on was Generalize mypyc IR to Make non-C Backends Possible. In short, it was a project to redesign and refactor some parts of the old mypyc IR to remove the embedded C backend-specific information. You can find the detailed technique report on the submission page. Here, I'd like to talk about other things that play as important a role as coding.

Plan the schedule and monitor the progress. Three months is just a blink of an eye, so it's important to have realistic goals to achieve at specific timestamps, to make sure the project goes smoothly. We had a rough schedule in the proposal and established monthly goals in later discussions. With the schedule and goals in mind, I was able to make progress continuously. I even had an extra week. Usually, before the coding period starts in June, there is one month for community bonding where students get to familiarize themselves with the workflow of the project and connect with the mentors. Thanks to my previous mypy contribution experience, I thought it would be fine to skip this period and start coding one week earlier than scheduled. This extra week gave my mentors and me a lot of flexibility to handle unexpected schedule delays.

Communication is the key. Jukka, one of my mentors, told me that it was always hard to come up with the perfect design at first as requirements and needs develop all the time. To face this developing nature, we communicated a lot. I had daily syncs with my mentors through Gitter chat, weekly meetings with Jukka, and monthly meetings with both Jukka and Sully. In all these meetings, I shared recent progress, they helped me remove any blockers, and we discussed component designs. I am one-hundred percent sure that this project would never have been this successful without such frequent and effective communication.

After the Summer

I completed this project and submitted the required materials in early September, with around 60 PRs merged into the repository. To me, the GSoC project ended pretty well, I received the certificate from Google as well as some stipends. More importantly, I've demonstrated my passion and skills to contribute to this organization more than just a community contributor. I was honored to become one of mypy's committers. That means a lot to me and I am looking forward to contributing more to the community.

That's my story with mypy through the Google Summer of Code 2020. I hope this post will provide some useful information to future potential applicants including applicants of this year since mypy has officially been selected as one of this year’s organizations. If you have any questions regarding this post, feel free to send me an email and I will be more than happy to discuss.

Acknowledgements

I'd like to express my sincere gratitude to every person and organization that made my GSoC 2020 adventure possible. I'd like to firstly thank Google and mypy for providing me this opportunity.

Especially, I'd like to thank my mentors Jukka Lehtosalo and Michael Sullivan. We had daily syncs every day on Gitter and weekly/monthly video meetings via Zoom. In every discussion, they helped me clean my thoughts and find out the best approach to meet our goals. They responded quickly to my PRs, giving high-quality review comments and suggestions. They mentored me with patience and passion and I feel connected even though we are several timezones/continents away. Their guidance is even beyond the scope of the project and helps me form good software engineering skills along the way.

I also would like to thank my parents for supporting me working on this project and Yaozhu Sun from the University of Hong Kong, who lit my passion for the field of compilers and programming languages two years ago. Finally, I'd like to thank Kanemura Miku from Hinatazaka 46 for all the mental support during this special summer.

Disclaimer

All content of this post only represents the author's personal opinions. The post does not, in any degree, constitute the view or opinion from Google or the mypy community.

Friday, 19 February 2021

Mypy 0.812 Released

We’ve just uploaded mypy 0.812 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes a fix to a regression in source file finding logic in mypy 0.800, and a new command-line option --exclude to exclude paths from the build. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Improved Source File Finding

Mypy 0.800 changed how mypy finds modules if you run mypy as mypy directory/ or mypy -p package. Mypy started looking for source files in directories without a __init__.py file. This is often the expected behavior, and it avoids excluding some files that should be type checked.

However, this caused issues for some users, such as when using mypy . to type check all files under the current directory. Mypy could now try to type check files inside nested virtual environments and node_modules directories, which is usually not desirable. This could result in mypy needlessly complaining about duplicate module names, in particular.

Now mypy will skip directories named site-packages or node_modules, and any directory beginning with a dot (such as .git) when recursively looking for files to check.

This doesn’t affect how mypy resolves imports — it only affects when mypy is given a directory or a package to type check. You can override the exclusions by explicitly passing the files on the command line.

This was contributed by Shantanu (PR 9992).

Excluding Paths

Mypy now supports the --exclude regex command line option to exclude paths matching a regular expression when searching for files to type check. For example, mypy --exclude '/setup\.py$' skips all setup.py files. This lets you exclude additional paths that mypy started finding after mypy 0.800 changed module finding behavior, as discussed above.

You can also specify this in the config file (exclude=regex). The option expects forward slashes as directory separators on all platforms, including Windows, for consistency.

This was also contributed by Shantanu (PR 9992). See the documentation for more details.

Typeshed Updates

There are no typeshed updates in this release.

Acknowledgments

Thanks to Shantanu who contributed to this release.

We’d also like to thank our employer, Dropbox, for funding the mypy core team.

Friday, 22 January 2021

Mypy 0.800 Released

We’ve just uploaded mypy 0.800 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Python 3.9 Support

Mypy 0.800 officially supports the recently released Python 3.9. We now provide compiled binary wheels for Python 3.9, improving type checking speed significantly.

Typing Usability Improvements (PEP 585 and PEP 604)

The necessity to repeatedly import various types and special forms from typing has been a long-term nuisance for users of static type checking and Python.

Two new Python features improve this situation and are now supported by mypy:

  • PEP 585 lets you use list[int] instead of List[int] (no need to import List and other generic collections from typing).
  • PEP 604 lets you write X | Y instead of Union[X, Y], and X | None instead of Optional[X] (no need to import Union or Optional from typing).

Note: Using list[int] requires Python 3.9 and X | Y requires Python 3.10 (alpha) in order to work at runtime. To use them on older versions of Python, use from __future__ import annotations. This allows them to be used in type annotations, but the older variants (or string literal escaping) may be required in non-annotation contexts, such as in type aliases. See the docs for more details.

Here is an example that uses the new features:

    from __future__ import annotations
    
    def fields(s: str | None) -> list[str]:
        if not s:
            return []
        else:
            return s.split(',')

These were implemented by Allan Daemon in PR 9564 and by Marc Mueller in PR 9647.

Improvements to Finding Modules

This release adds several improvements to how mypy finds Python source files to type check.

You can now pass paths to files within namespace packages on the command line, and mypy can better infer their module names. As before, use --namespace-packages to enable namespace packages.

When you use --explicit-package-bases together with --namespace-packages, mypy assumes that only the current directory and directories explicitly specified in MYPYPATH (or mypy_path in the config file) are valid package roots. This can help with situations where the module name of a file is ambiguous. For example, it may not be clear whether src/pkg/mod.py should be treated as src.pkg.mod or pkg.mod, and you can use this option to disambiguate between the two (more information in the docs).

The above improvements were implemented in PR 9742 by Shantanu.

Other related improvements (also implemented by Shantanu):

  • When you run mypy as mypy <directory>, look for source files recursively also inside directories without a __init__.py (PR 9614)
  • Support namespace packages with -p (PR 9683)
  • Log encountered source files when running mypy with -v (PR 9672)
  • Document the new module finding behavior (PR 9923)

Other Notable Improvements and Bug Fixes

  • Only treat import X as X as a re-export in stubs (Shantanu, PR 9515)
  • Fix package imports with aliases in stubgen (Chad Dombrova, PR 9534)
  • Require first argument of namedtuple() to match the variable name (Momoko Hattori, PR 9577)
  • Add error code for name mismatches in named tuples and TypedDicts to make it easy to disable these error messages (Jukka Lehtosalo, PR 9811)
  • Document local_partial_types config option (Momoko Hattori, PR 9551)
  • Improve ambiguous kwargs checking (Erik Soma, PR 9573)
  • Disable unreachable warnings in boolean operators for type variables with value restrictions (Vincent Barbaresi, PR 9572)
  • Allow assignment to an empty tuple (Tobin Yehle, PR 5617)
  • Use absolute path when checking source duplication error (Yuki Igarashi, PR 9059)
  • Add get_function_signature_hook() to the plugin API (Nikita Sobolev, PR 9102)
  • Speed up type checking of dictionary, set, and list expressions (Hugues, PR 9477)
  • Allow non-types as arguments in Annotated (Patrick Arminio, PR 9625)
  • Add support for next generation attrs API (David Euresti, PR 9396)
  • Fix case folding of missing keys error message for TypedDicts (Marti Raudsepp, PR 9757)
  • Fix generic inheritance of __init__() methods in dataclasses and attrs classes (Nate McMaster, PR 9383, PR 9380)
  • Add more information to error message on too few arguments (Abhinay Pandey, PR 9796)
  • Document PEP 585, 563, 604, and related functionality (Shantanu, PR 9763)

Mypyc Improvements

We use mypyc to compile mypy into fast C extension modules. This release includes many mypyc improvements.

Xuanda Yang finished the migration to use a new, lower-level compiler intermediate representation in his Google Summer of Code project.

New supported Python features:

  • Support the walrus operator (:=) (Michael J. Sullivan, PR 9624)

Performance improvements:

  • Add primitives for list sort and list reverse (Jukka Lehtosalo, PR 9897)
  • Recognize six.moves.xrange again as an alias of range (Jukka Lehtosalo, PR 9896)
  • Speed up some integer primitives (Jukka Lehtosalo, PR 9801)
  • Speed up if x for int values (Jukka Lehtosalo, PR 9854)
  • Implement dict clear primitive (Vasileios Sakkas, PR 9724)
  • Implement list insert primitive (Vasileios Sakkas, PR 9741)
  • Implement float abs primitive (Xuanda Yang, PR 9695)
  • Implement str-to-float primitive (Xuanda Yang, PR 9685)
  • Specialize some calls to frozenset (Michael J. Sullivan, PR 9623)
  • Speed up multiple assignment from tuple (Xuanda Yang, PR 9575)
  • Speed up multiple assignment from sequence (Jukka Lehtosalo, PR 9800)
  • Optimize startswith and endswith (Tomer Chachamu, PR 9557)
  • Add primitives for bitwise ops (Jukka Lehtosalo, PR 9529)
  • Speed up in operations for list/tuple (Johan Dahlin, PR 9004)
  • Add primitives for list, str and tuple slicing (Jukka Lehtosalo, PR 9283)
  • Speed up tuple equality checks (Xuanda Yang, PR 9343)

Bug fixes:

  • Always add implicit None return type to __init__ method (Thomas Johnson, PR 9866)
  • Fix deallocation of deeply nested data structures (Michael J. Sullivan, PR 9839)
  • Fix using package imported inside a function (Jukka Lehtosalo, PR 9782)
  • Fix type of for loop index register in for over range (Jukka Lehtosalo, PR 9634)

Typeshed Updates

Many improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Abdullah Selek
  • Abhinay Pandey
  • Adam
  • aghast
  • Akuli
  • Alexander
  • Allan Daemon
  • Aristotelis Mikropoulos
  • Ashley Whetter
  • Brian Mboya
  • Bryan Forbes
  • cdce8p
  • Chad Dombrova
  • David Euresti
  • Denis Laxalde
  • Eisuke Kawashima
  • Erik Soma
  • Ethan Pronovost
  • Florian Bruhin
  • Frank Dana
  • Greg Compestine
  • Guido van Rossum
  • Hugues
  • Jake Bailey
  • Jakub Stasiak
  • Jelle Zijlstra
  • Jeremy Metz
  • Johan Dahlin
  • Jon Shea
  • Jonathan Wong
  • Jürgen Gmach
  • Kamil Turek
  • Krzysztof Przybyła
  • Lawrence Chan
  • Marti Raudsepp
  • Matan Gover
  • Matt Gilson
  • Michael J. Sullivan
  • Momoko Hattori
  • Nate McMaster
  • Nikita Sobolev
  • Nils K
  • Nipunn Koorapati
  • Oleg Höfling
  • Patrick Arminio
  • Rajiv Singh
  • rhkleijn
  • Roland van Laar
  • Shantanu
  • Tobin Yehle
  • Tom Scogland
  • Tomer Chachamu
  • Tripp Horbinski
  • Ville Skyttä
  • Vincent Barbaresi
  • vsakkas
  • Wes Turner
  • willtryagain
  • Xiaodong DENG
  • Xuanda Yang
  • Yash Chhabria
  • Yuki Igarashi

Additional thanks to all contributors to typeshed:

  • Abraham Francis
  • Adam Dangoor
  • Adam Kliś
  • Adam Lichtl
  • Akuli
  • alexander-held
  • an onion
  • Anders Kaseorg
  • Andrew Mitchell
  • Árni Már Jónsson
  • Ash Berlin-Taylor
  • Ashwin Vishnu
  • Avery
  • cdce8p
  • Cebtenzzre
  • Changsheng
  • Christine
  • coiax
  • cptpcrd
  • crusaderky
  • Daniel O'Neel
  • David Caro
  • Dominic Davis-Foster
  • Eric Traut
  • Ethan Pronovost
  • Frank Maximilian
  • Gal Ben David
  • github-actions[bot]
  • Guido van Rossum
  • henribru
  • Hong Xu
  • Hugues
  • Hynek Schlawack
  • jack1142
  • Jake Bailey
  • Jason Fried
  • Jelle Zijlstra
  • Jia Chen
  • Jon Dufresne
  • Jonathan Schoonhoven
  • Jonathan Slenders
  • Joseph Haaga
  • Julien Danjou
  • Jun Jia
  • Jérome Perrin
  • karl ding
  • Katelyn Gigante
  • Kaushal Rohit
  • Kevin Wojniak
  • ky-gog
  • Kyle Fuller
  • kylec1
  • Lam Son Ho
  • Lourens Veen
  • Mahmoud Abduljawad
  • Mariam Maarouf
  • Marti Raudsepp
  • melassa
  • Mickaël Schoentgen
  • Mikhail Sveshnikov
  • Mikołaj Kuranowski
  • Moriyoshi Koizumi
  • Nate McMaster
  • Neel Somani
  • nicolas-harraudeau-sonarsource
  • Nikolaus Waxweiler
  • Nils K
  • Nipunn Koorapati
  • Oleg Höfling
  • Omar Sandoval
  • Paul
  • Pete Scopes
  • Peter Law
  • Philipp Hahn
  • Phillip Huang
  • proost
  • PythonCoderAS
  • Rajiv Bakulesh Shah
  • Ran Benita
  • Raphael Geronimi
  • Rebecca Chen
  • Sam Bull
  • Sebastian Rittau
  • Sergei Lebedev
  • Shantanu
  • Stefano Chiodino
  • Steve Dignam
  • Sténio Jacinto
  • Timur Kushukov
  • Tom Most
  • turettn
  • Unrud
  • Utsav
  • Vasily Zakharov
  • Vincent Barbaresi
  • Vincent Meurisse
  • 愚氓
  • Yuri Khan

Friday, 9 October 2020

Mypy 0.790 Released

We’ve just uploaded mypy 0.790 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Note that mypy 0.790 doesn’t yet officially support the recently released Python 3.9.

Enabling and Disabling Error Codes

You can now globally disable and enable the generation of error messages with specific error codes by using --enable-error-code and --disable-error-code command line options (and config file options). This lets you disallow certain checks that don’t provide value or are too noisy, and in the future these can be used to enable optional, stricter checks. For more information, read the documentation.

This was contributed by Callum Wilkinson (PR 9172).

Bug Fixes

  • Fix disable_error_code config file option (Aristotelis Mikropoulos, PR 9538)
  • Fix partial type crash during protocol checking (Shantanu, PR 9495)
  • Store the type for assignment expression (walrus) targets (Lawrence Chan, PR 9479)
  • Always type check arguments when using --disallow-untyped-calls (PR 9510)
  • Make None compatible with Hashable (PR 9371)
  • Don't infinite loop on self dependencies in --follow-imports=normal (Michael J. Sullivan, PR 9302)
  • Don't simplify away Any when joining union types (PR 9301)
  • Check for deleted vars in raise from (Alexandre Viau, PR 9272)
  • Fix corner case for comparing nested overloads (Nikita Sobolev, PR 9259)
  • Fix issues with async for and with statements (Guido van Rossum, PR 9268)
  • Fix internal error on list/dict comprehension with walrus operator in global scope (dhood, PR 9062)
  • Fix propagation of module-specific options to the parser (Guido van Rossum, PR 9247)
  • Fix crash when super is called outside a method (Weiss, PR 9173)
  • Fix untyped decorator overload error on class decorator with __call__ overloads (Ran Benita, PR 9232)
  • Clean up terminal width handling (Florian Bruhin, PR 9143)
  • Add keyword arguments for functional Enum API (LiuYuhui, PR 9123)
  • Validate follow_imports values in mypy.ini (Michael J. Sullivan, PR 9165)
  • Fix *expr in an assigned expression (LiuYuhui, PR 8827)
  • Don't consider comparing True and False as a dangerous comparison (PR 9021)
  • Make reveal_type work with call expressions returning None (Jakub Stasiak, PR 8924)

Error Reporting Improvements

  • Improve missing module error for subdirectories (Ethan Leba, PR 8927)
  • Clarify bytes formatting error messages (Shantanu, PR 9243)
  • Fix misleading follow_imports error message in dmypy (Michael J. Sullivan, PR 9167)
  • Use [arg-type] error code for additional argument type error messages (PR 9090)
  • Report some additional serious errors in junit.xml (PR 8950)
  • Report note about binary operation on the same location as error message (PR 8936)

Documentation Updates

  • Document disable_error_code config file option (Aristotelis Mikropoulos, PR 9539)
  • Add cross references to config file values (Oleg Höfling, PR 7859)
  • Add cross references to additional config values, and missing plugins config value, and document --help option for stubgen (Oleg Höfling, PR 9241)
  • Remove note that Final is experimental and suggest importing it from typing (Ran Benita, PR 9138)
  • Discuss unreachable code as a common issue (Chetan Khanna, PR 8899)

Stubgen Improvements

  • Improve property type detection in extension modules (Antoine Prouvost, PR 8999)
  • Fix type stubgen crash caused by invalid type annotation (Henry Schreiner, PR 8888)
  • Import Iterable and Iterator from typing in generated stubs (Ashley Whetter, PR 9088)

Other Improvements

  • Speed up type checking of unions containing many literal string types (Akuli, PR 9192)
  • Add scripts to misc/ (in the mypy GitHub repository) for computing and applying diffs of mypy caches to enable incremental cache downloads (Michael J. Sullivan, PR 8906)

Internal Changes

This release includes several improvements to mypyc, the compiler we use to speed up mypy. Most notably, Xuanda Yang has been migrating mypyc to use a new, lower-level intermediate representation in his Google Summer of Code project. This provides a solid foundation for a variety of future mypyc improvements.

Additional mypyc improvements:

  • Dynamically disallow instantiating traits/interpreted subclasses (Michael J. Sullivan, PR 9248)
  • Basic support for protocols (Michael J. Sullivan, PR 8914)
  • Various optimizations (PR 8903, PR 8870; Michael J. Sullivan and Xuanda Yang, PR 9245)
  • Make it easier to write mypyc test cases (PR 9094)

Typeshed Updates

Many improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Akuli
  • Alexandre Viau
  • Antoine Lecubin
  • Antoine Prouvost
  • Aristotelis Mikropoulos
  • Ashley Whetter
  • Ben Wolsieffer
  • Brian Mboya
  • Callum Wilkinson
  • Chetan Khanna
  • dhood
  • Ethan Leba
  • Florian Bruhin
  • Henry Schreiner
  • ishaan
  • Jakub Stasiak
  • Jelle Zijlstra
  • Lawrence Chan
  • LiuYuhui
  • Michael R. Crusoe
  • Michael Sullivan
  • Nikita Sobolev
  • Oleg Höfling
  • Ram Rachum
  • Ran Benita
  • Rune Tynan
  • Shantanu
  • Weiss
  • Xuanda Yang
  • 谭九鼎

Additional thanks to all contributors to typeshed:

  • Adam Hitchcock
  • Akuli
  • Alex Grönholm
  • Anders Kaseorg
  • Andre Delfino
  • Andreas Poehlmann
  • Artem Simonov
  • Batuhan Taskaya
  • Brad Solomon
  • Brian Wellington
  • Bruce Merry
  • Cerek Hillen
  • Christine
  • Christopher Head
  • Christopher Schramm
  • Dakkaron
  • Dan Palmer
  • David Hotham
  • Denis Laxalde
  • Dmitry Marakasov
  • Eric Masseran
  • Eric N. Vander Weele
  • Eric Traut
  • Federico Bond
  • frehoy
  • Gregory P. Smith
  • helafumpek
  • Ian Good
  • Ian Woloschin
  • Igor Nehoroshev
  • Ilaï Deutel
  • Ilya Konstantinov
  • Jakub Stasiak
  • James Weaver
  • Jaromir Latal
  • Jelle Zijlstra
  • Jia Chen
  • John Snow
  • Jonathan Slenders
  • Joshua Oreman
  • Julian Mehnle
  • Julin S
  • Jürgen Gmach
  • Karl Ding
  • Kevin Wojniak
  • Kunal Arya
  • Luciano Ramalho
  • Magnus Watn
  • MapleCCC
  • Mario Ishac
  • Mark Mendoza
  • Markus Pielmeier
  • Martijn Pieters
  • Matan Gover
  • Michael H
  • Mikhail Golubev
  • Milap Sheth
  • Nils K
  • Nipunn Koorapati
  • Omar Sandoval
  • Peter Law
  • Philipp Hahn
  • Phillip Huang
  • Ran Benita
  • Rebecca Chen
  • Rich Li
  • Rune Tynan
  • Russell Davis
  • Sabby6
  • Sam Bull
  • Sebastian Kreft
  • Sebastian Rittau
  • Selim Belhaouane
  • Shannon Zhu
  • Shantanu
  • Steve B
  • Steve Dignam
  • Tarcisio
  • Teddy Sudol
  • Toru Ogawa
  • Vegard Stikbakke
  • Vishal Kuo
  • Vlad Emelianov
  • Willy Wu
  • Wouter Bolsterlee
  • Áron Ricardo Perez-Lopez

Wednesday, 3 June 2020

Mypy 0.780 Released

Mypy 0.780 Released

We’ve just uploaded mypy 0.780 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Following Imports in Mypy Daemon

You can now leave out the --follow-imports option when running dmypy, and mypy daemon will follow imports, similar to a non-daemon mypy run:

    dmypy run -- prog.py pkg/*.py

Previously mypy daemon did not support this, and it was necessary to use --follow-imports and explicitly mention all files to check on the command line. (Following imports in mypy daemon is an experimental feature.)

See the docs about following import in mypy.

Miscellaneous New Features

  • Support environment variable interpolation for junit_xml configuration key (Mattwmaster58, PR 8479)
  • Add no_site_packages config file setting (davidzwa, PR 8524)
  • Allow .mypy.ini (with a dot prefix) as an alternative name to mypy.ini (dosisod, PR 8515)
  • Disallow unpacking of strings in multiple assignment, as this is usually a bug (Abtin, PR 8589)
  • Allow type aliases in a runtime (non-type) context (PR 8779)
  • Add narrowing unions with bool literals via identity check (Steve Dignam, PR 8821)
  • Always allow to cast to Any without warnings (oda, PR 8544)
  • Suggest solutions for a typo in a key of a TypedDict (Felicián Németh, PR 8483)
  • Provide more context about why incompatible with supertype is an error (Chetan Khanna, PR 8866)

Other Notable Improvements and Bug Fixes

  • Fix handling dependencies to __call__ in mypy daemon (PR 8494)
  • Various improvements to stubtest (PR 8502, PR 8886) (Shantanu)
  • Fix invalid type causing named tuple errors reported in wrong files (PR 8549)
  • Clarify documentation of Liskov substitution principle (PR 8563)
  • Fix type inference with lambda that returns None (PR 8562)
  • Fix incremental crash bug caused by NewType in functions (PR 8607)
  • Fix indexed assignment check when TypedDict is used as upper bound (Xuanda Yang, PR 8621)
  • Improve error message for bad indexed assignment (pando, PR 8641)
  • Fix crash when the same file is processed under multiple names (PR 8644)
  • Fix parser when using Python 3.9 (PR 8716)
  • Accept dict() as an empty dictionary in a TypedDict context (PR 8749)
  • Fix incorrect coroutine return type for nested function (Jens Widell, PR 8741)
  • Reject bytes literal as a TypedDict key (PR 8782)
  • Allow attrs keyword-only arguments at any position (Markus Schmaus, PR 8803)
  • Detect duplicate keys in call-based TypedDict definitions (PR 8849)
  • Look at arguments when generating constraints for an overload (PR 8845)
  • Fix potential wrong-file error message for unsupported types (PR 8854)

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Abtin
  • Brad Solomon
  • Chetan Khanna
  • Claudio Jolowicz
  • davidzwa
  • Diego Elio Pettenò
  • dosisod
  • Ethan Smith
  • Felicián Németh
  • Jelle Zijlstra
  • Jens Widell
  • Joy Bhalla
  • LiuYuhui
  • Markus Schmaus
  • Marten Kenbeek
  • Mattwmaster58
  • Max R
  • oda
  • pando
  • PattenR
  • prostomarkeloff
  • Shantanu
  • Steve Dignam
  • Timofey Kukushkin
  • Ville Skyttä
  • Xuanda Yang

Additional thanks to all contributors to typeshed:

  • Alex McGrath Kraak
  • Alexandre Yang
  • Andrew
  • Andrew Svetlov
  • Anis
  • Anthony Sottile
  • Ben Leslie
  • Ben Motz
  • Benjamin Poirier
  • Brian Turek
  • Bruce Merry
  • Christopher Whelan
  • coiax
  • David Euresti
  • David T.H. Kao
  • Debjyoti Biswas
  • Denis Laxalde
  • Diego Elio Pettenò
  • dosisod
  • Ethan Smith
  • Florian Ludwig
  • Florimond Manca
  • Gal Ben David
  • Graham Bleaney
  • Jakub Stasiak
  • Jan Verbeek
  • Jaromir Latal
  • Javier Honduvilla Coto
  • Jeff Hunter
  • Jelle Zijlstra
  • Jia Chen
  • Jocelyn Boullier
  • Joost Cassee
  • Julian Andres Klode
  • Julin S
  • karl ding
  • Katelyn Gigante
  • Kazushi Kitaya
  • Kjell Braden
  • Lars
  • Lawrence
  • Luciano Ramalho
  • Mark Mendoza
  • Markus Wamser
  • Max R
  • Mickaël Schoentgen
  • Nguyễn Gia Phong
  • Nipunn Koorapati
  • Oleg Höfling
  • Paul Ganssle
  • Peter Pentchev
  • petsuter
  • PGijsbers
  • Philipp Hahn
  • Rahix
  • Ran Benita
  • Razzi Abuissa
  • Rebecca Chen
  • Rodrigo Castro
  • Rune Tynan
  • Sahith Nallapareddy
  • Sam Bull
  • Sebastian Rittau
  • Selim Belhaouane
  • Serhiy Storchaka
  • Shantanu
  • Steve Dignam
  • Tim Hatch
  • Vishal Kuo
  • Zhiming Wang

Tuesday, 10 March 2020

Mypy 0.770 Released

We’ve just uploaded mypy 0.770 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Tagged Unions

Previously, the only way to distinguish between two types in a union was to use an isinstance check. Now, mypy supports distinguishing between two or more types based on the value of some common shared "tag" field.

This feature is most useful when you want to narrow down a union of TypedDicts: since TypedDicts are really just regular dicts at runtime, using isinstance checks won't work. Instead, you can now label or tag each TypedDict with a distinct Literal type and discriminate by checking that tag:

    from typing import Literal, TypedDict, Union
    
    class NewJob(TypedDict):
        event_type: Literal["new-job"]
        job_description: str
    
    class CancelJob(TypedDict):
        event_type: Literal["cancel-job"]
        job_id: int
    
    request: Union[NewJob, CancelJob]
    
    # We know request["event_type"] must be of type
    # Union[Literal["new-job"], Literal["cancel-job"]]
    if request["event_type"] == "new-job":
        # So if we narrow request["event_type"] down to type
        # Literal["new-job"], mypy will conclude 'request' must
        # therefore be of type 'NewJob' in this branch...
        print("Starting new job", request.job_description)
    else:
        # ...and inferred to be type 'CancelJob' in this one
        print("Cancelling job with id", request.job_id)

Note that you can import Literal from typing_extensions on Python 3.7 and earlier.

You can also use this technique to discriminate between unions of objects, tuples, or named tuples, so long as: 1. Every item in your union is tagged with a unique value 2. Mypy understands how to narrow the union of the tag values.

For more details and examples, see the documentation on tagged unions.

This feature was contributed by Michael Lee.

Type Inference Improvements

Type inference is now more powerful. Mypy can infer types in some cases where you previously needed an explicit type annotation.

Mypy can infer a built-in collection type from augmented assignment statements:

    x = []
    x += [1, 2]  # Infer type List[int] for x

Some dictionary construction idioms are better supported:

    d = {}  # No type annotation needed
    for k in things():
        if k not in d:
            d[k] = 0
        else:
            d[k] += 1

Mypy is better at inferring defaultdict types:

    from collections import defaultdict
    
    d = defaultdict(list)
    d['foo'].append(1)  # Infer type DefaultDict[str, List[int]] for d

Multiple Inheritance and isinstance() Checks

Previously, mypy assumed a variable cannot have two unrelated types when analyzing isinstance checks. For example, in the following example B is not a subclass of A, so mypy would incorrectly conclude the isinstance check is always false, and that the if-branch is unreachable — even though var could actually be a subtype of both types.

This meant that mypy skipped type checking anything inside the if statement, since it (by design) skips analyzing unreachable branches.

    class A: pass
    class B: pass
    
    class Ok(A, B): pass
    
    x: A = Ok()
    if isinstance(x, B):
        reveal_type(x)

Now, mypy will instead decide that the narrowed type of x is test.<subclass of "A" and "B">, instead of marking the branch as unreachable.

If it’s impossible for two types to be a subtype of one another, mypy will continue to not attempt to infer this "ad-hoc intersection":

    # Fails with a "TypeError: multiple bases have instance lay-out confict" error at runtime
    class Impossible(int, str): pass
    
    bad: int = Impossible()
    if instance(bad, str):
        # This branch is unreachable and so is not type checked.
        # If you want to be warned about unreachable branches, use
        # the --warn-unreachable flag.
        reveal_type(bad)

This feature was contributed by Michael Lee.

Other Notable Improvements and Bug Fixes

  • Fix some crash bugs involving import * and import cycles (PR 8450)
  • Fix interaction of descriptor methods with plugins (Jan Verbeek, PR 8365)
  • Allow strict in config file (Ville Skyttä, PR 8192)
  • Don’t crash when a module shadows builtin libraries (such as typing or types, PR 8405)
  • Fix type join between subclasses of unicode and str (PR 8402)
  • Fix type join of fixed-length tuples with mismatching lengths (Marti Raudsepp, PR 8333)
  • Fix type join of Sequence (e.g. variadic tuple) and fixed-length tuple (Marti Raudsepp, PR 8335)
  • Make mypy.api.run_dmypy actually capture the output (PR 8375)
  • Support determining whether a literal is truthy (Jan Verbeek, PR 8368)
  • Fix covariant overriding of decorated methods (Xuanda Yang, PR 8350)
  • Support typing.Annotated in addition to typing_extensions.Annotated (Jakub Stasiak, PR 8371)
  • Add add_method_to_class function to plugins.common (useful when writing mypy plugins) (Maksim Kurnikov, PR 8245)
  • Fix module alias as instance attribute (Uwe L. Korn, PR 8259)
  • Automatically write a file .gitignore file to cache directory, ignoring everything (Ville Skyttä , PR 8193)
  • Don't make dunder attributes enum members (Xuanda Yang, PR 8302)
  • Allow redefining TypedDict keys (while still generating an error) (Cohen Karnell, PR 8109)
  • Fix some crashes in dataclasses (PR 8271)
  • Use TypedDict union as type context when unambiguous (PattenR, PR 8212)
  • Fix false positive for subclasses of bytes overriding __str__ (Shantanu, PR 8222)
  • Fix deeply nested InitVar definitions in dataclasses with init=False (Jacob Beck, PR 8208)
  • Narrow types for walrus assignment in if statements in some cases (Shantanu, PR 8258)
  • Narrow types for walrus assignment in if statements in most of the rest of cases (PR 8458)
  • Fix incorrect error code indexing (Xuanda Yang, PR 8248)
  • Fix regression in container check logic (Michael Lee, PR 8232)
  • Make reachability code understand chained comparisons (Michael Lee, PR 8148)
  • Fix incorrect name lookup for decorated methods (Xuanda Yang, PR 8175)
  • Fix simplifying unions with type aliases (Xuanda Yang, PR 8146)
  • Fix crash when overriding __init__ in a dataclass subclass (Jacob Beck, PR 8159)
  • Fix some daemon crashes involving classes becoming generic (PR 8157)

Documentation and Error Reporting Improvements

  • Use fully qualified names in error messages for class names resembling builtins(Mukuntha N S, PR 8425)
  • Improve diagnostics involving missing stubs for a library that is installed in site-packages (Michael Lee, PR 8238)
  • Add a section to the documentation about incompatible overrides (RAHUL RAJA, PR 8377)
  • Add variable-sized tuples to the cheat sheet (Marcio Mazza, PR 8364)
  • Improve documentation of decorators (add decorator factories) (Marti Raudsepp, PR 8336)
  • Update documentation of variables and aliases (Xuanda Yang, PR 8200)
  • Report an error if a final class has abstract attributes (Denys Halenok, PR 8332)
  • Update common issues to include __init__ without arguments (Tan Yuanhong, PR 8303)
  • Add a new error code for errors about unreachability (Denys Halenok, PR 8312)
  • Fix error indicator position in code with tabs (dosisod, PR 8307)
  • Document --local-partial-types (Xuanda Yang, PR 8201)
  • Update documentation for Literal types (Michael Lee, PR 8152)

Stubtest rewrite

stubtest is a tool that compares stub definitions to what it finds at runtime with introspection and reports back inconsistencies. It got a complete rewrite. Some features of the new stubtest are:
  • Find missing, extraneous or mistyped classes, methods, functions and attributes in the stubs
  • Check presence, names and kinds of function arguments, accounting for overloads, decorators, *args and kwargs. Checks argument types against their default values, accounting for type vars.
  • Checks @property, @classmethod, @staticmethod declarations
  • Check types of module level and class level attributes and enums

Some results of this:

  • We now run stubtest in typeshed CI (for stdlib)
  • We’ve fixed about 1900 issues in typeshed definitions
  • We’ve greatly improved Python 3.8 support in typeshed and it’ll be easier to make the changes needed for future Python versions
  • We’ve uncovered a handful of issues in Python itself

The stubtest rewrite was contributed by Shanatu.

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • Anthony Sottile
  • Cohen Karnell
  • Denys Halenok
  • dosisod
  • Ethan Smith
  • FooTaGe
  • hauntsaninja
  • HeShan
  • Jacob Beck
  • Jakub Stasiak
  • Jan Verbeek
  • Jérome Perrin
  • lazytype
  • Maksim Kurnikov
  • Marcio Mazza
  • Marti Raudsepp
  • Michael Lee
  • Mukuntha N S
  • Nikita Sobolev
  • PattenR
  • RAHUL RAJA
  • Shantanu
  • Tan Yuanhong
  • Uwe L. Korn
  • Ville Skyttä
  • Xuanda Yang

Additional thanks to all contributors to typeshed:

  • abe
  • Alan Du
  • Alex Grönholm
  • Alex Willmer
  • Andrew Svetlov
  • Anthony Sottile
  • Artjoms Iskovs
  • Batuhan Taşkaya
  • Benjamin Peterson
  • Brian Maissy
  • Cal Paterson
  • Christopher Dignam
  • crusaderky
  • Cyril Roelandt
  • Daniel Däschle
  • Daniel Farley
  • Daniel Hahler
  • Daniël van Eeden
  • Dave Halter
  • dave-shawley
  • Erick
  • Faidon Liambotis
  • Fionn Fitzmaurice
  • François Freitag
  • Gisle Aas
  • hauntsaninja
  • Ian Good
  • ijl
  • Ilaï Deutel
  • Jacob Beck
  • Jacob Ilias Komissar
  • Jakub Stasiak
  • Jan Verbeek
  • Jaromir Latal
  • Jason
  • Jason Gilholme
  • Jelle Zijlstra
  • Jens Hedegaard Nielsen
  • Jeppe Fihl-Pearson
  • layday
  • lazytype
  • Maarten ter Huurne
  • Mark
  • Michael Heyns
  • Mickaël Schoentgen
  • Niklas Fiekas
  • Oleg Höfling
  • Ophir LOJKINE
  • Pavel Savchenko
  • Rebecca Chen
  • Reid Swan
  • rikhilraithatha
  • Romain
  • Rune Tynan
  • Sebastian Rittau
  • Shantanu
  • Squirrel
  • Thomas Schaper
  • tikki
  • Vury Leo
  • Wolf Honore

Tuesday, 17 December 2019

Mypy 0.760 Released

We’ve just uploaded mypy 0.760 to the Python Package Index (PyPI). Mypy is a static type checker for Python. This release includes new features, bug fixes and library stub (typeshed) updates. You can install it as follows:

    python3 -m pip install -U mypy

You can read the full documentation for this release on Read the Docs.

Type Signature Suggestions for Tools

Normally mypy only infers the types of variables, and you have to explicitly annotate functions. Mypy daemon can now generate a suggestion for a function or method signature using static analysis through dmypy suggest. This is a low-level feature that can be used by other tools and editor integrations to make it easier to annotate existing code. See the documentation for more information.

Type Inference Improvements

Mypy type inference got a few minor improvements. Mypy can now infer a variable with a collection type from two assignments:

    items = []  # No type annotation needed
    if check():
        items = [2, 3]

Previously mypy required a type annotation for items. Mypy can also more often infer an OrderedDict type:

    from collections import OrderedDict

    data = OrderedDict()  # No type annotation needed
    data['key'] = 4

Mypy can infer the type of an attribute in more cases when it’s first initialized through self.x:

    class Counts:
        def __init__(self, n: int) -> None:
            self.counts = {}  # Type inferred from initialization
            for i in range(n):
                self.counts[i] = 0

Fixes to Regressions

  • Fix regression introduced in 0.750: union (non-)simplification should not affect type inference (PR 8095)

Breaking Changes

  • Make the error code of a binary operation involving overload operator, for consistency (PR 8124)

Other Notable Improvements and Bug Fixes

  • Generate error when assigning an Enum or TypedDict type to an attribute (Xuanda Yang, PR 8107)
  • Fix relative path calculation if path is on a different drive on Windows (Netzeband, PR 8094)
  • Don’t infer an abstract type object type from concrete type object values (Xuanda Yang, PR 8096)
  • Fix an inconsistency with type checking lambda expressions (PR 8080)
  • Fix crash involving generic callable types (PR 8075)
  • Improve error message with long tuple initializer (Xuanda Yang, PR 7995)
  • Improve mypy daemon documentation (PR 8041)
  • Recommend typing_extensions instead of mypy_extensions for TypedDict (PR 8023)
  • Fix an inconsistency with self types (PR 8030, PR 8021)
  • Support type alias within Literal[...] (Xuanda Yang, PR 8014)
  • Fix --warn-return-any false positive when returning Any from a function declared to return object (Xuanda Yang, PR 8011)
  • Stubgen: Split @abstractproperty into @abstractmethod and @property (Xuanda Yang, PR 8066)
  • Stubgen: Give a better error message when using a .pyd file as target (Xuanda Yang, PR 8063)
  • Update stubgen documentation (PR 8031)

Typeshed Updates

Many small improvements were made to typeshed — too many to list. Browse the typeshed commit log here.

Acknowledgments

First of all, we’d like to thank our employer, Dropbox, for funding the mypy core team.

Thanks to all mypy contributors who contributed to this release:

  • jag426
  • Netzeband
  • Xuanda Yang

Additional thanks to all contributors to typeshed:

  • Alois Klink
  • Benjamin Peterson
  • cshesse
  • Denis Eliseev
  • Dylan Anthony
  • Eugene Ha
  • hauntsaninja
  • Jacob Ilias Komissar
  • Jason Fried
  • Jelle Zijlstra
  • Katelyn Gigante
  • Maksim Kurnikov
  • Michał Słapek
  • Mohammed El-Afifi
  • Ran Benita
  • Robert Schweizer
  • Rune Tynan
  • Ryan Hileman
  • Sebastian Rittau
  • Steven Kalt
  • toppk
  • Tuomas Suutari