Friday, 19 February 2016

Mypy 0.3 Released

I just uploaded mypy-lang 0.3 to PyPI! This release focuses on further PEP 484 compatibility and Python 2 compatibility, but it also contains many general improvements. The changes are too numerous to enumerate them all. Instead, I'll focus here on the most visible and useful ones.

I'm also happy to announce new mypy core developers Guido, David Fisher and Greg Price, who have joined the project thanks to support from our employer, Dropbox. This has given mypy development a big boost, and mypy is currently under very active development.

Mypy 0.3 breaks backward compatibility. You may have to modify your code to be PEP 484 compatible if you have been using mypy-lang from PyPI. As before, mypy is experimental and still has known bugs and limitations. In particular, we still don't have library stubs for many commonly used modules. Please report any new issues you encounter -- we try to respond quickly to all user requests, and bugs are often fixed rapidly.

Feature Highlights

  • Python 2 support -- now basically all Python 2.7 syntax is supported in Python 2 mode. Mypy also supports a comment-based function annotation syntax for Python 2 that was added to PEP 484 recently. It also works for code that aims to be both Python 2 and 3 compatible. More about this below.
  • Improved PEP 484 support. Mypy now supports most PEP 484 features (not yet, however, sys.version_info checks), and several non-standard features have been removed.
  • Partial Python 3.5 support (though async/await are sadly still not supported).
  • New experimental type checking modes (--silent-imports and --implicit-any command-line options) that make it much easier to type check subsets of a larger code base (see discussion of developer workflow improvements below).
  • Improved type inference -- fewer variable annotations are required, and mypy is much less likely to give the annoying "Cannot determine type of x" errors.
  • Many bug fixes and better Python compatibility.
  • Numerous standard library stub improvements (via typeshed, see below).

GitHub Changes

  • The mypy github repository has moved to the Python organization. The new URL is https://github.com/python/mypy.
  • Library stubs now live in a separate repo called typeshed (https://github.com/python/typeshed). The idea of the repository is make stubs a shared resource between all Python type checking or inference related projects.

Python 2 Support

Mypy now has experimental Python 2.7 support! For code that needs to be Python 2 compatible, function type annotations are given in comments, since the function annotation syntax was introduced in Python 3. Mypy has had partial, unofficial Python 2 support for a while, but only now it is actually usable and standard (after a recent PEP 484 update).

Run mypy in Python 2 mode by using the --py2 option:

    $ mypy --py2 program.py

To run your program, you must have the typing module in your module search path. Use pip install typing to install the module (it also works for Python 3).

Python 2 support was a lot of work and here is a summary of some of the most obvious changes:

  • Support pretty much all of Python 2 only syntax, including the print statement and the except Ex, err: syntax.
  • Provide Python 2 stubs for many library modules (now in typeshed, as mentioned earlier.)
  • Better Python 2 implementation of typing.
  • Document Python 2 support.
  • Make stub generator support generating Python 2 stubs (the stub generator is still highly experimental and not at all well documented, but it has already been pretty useful anyway.)

The example below illustrates Python 2 function type annotation syntax. This is also valid in Python 3 mode:

    from typing import List

    items = []  # type: List[str]

    def hello(): # type: () -> None
        print 'hello'

    class Example:
        def method(self, lst, opt=0, *args, **kwargs):
            # type: (List[str], int, *str, **bool) -> int
            ...

Here are some things to note:

  • Type syntax for variables is the same as Python 3.
  • Function annotations are in comments and look quite a bit like Python 3 annotations.
  • The module typing is available in Python 2 mode and things like Any must be imported from it, even if they are only used in comments.
  • You don't provide an annotation for the self/cls variable of methods.
  • The annotation can be on the same line as the function header or on the following line.
  • You don't need to use string literal escapes for forward references within comments.

Better PEP 484 Compatibility

Several changes improve PEP 484 compatibility. Mypy still doesn't implement quite everything in PEP 484, but we are pretty close.

Undefined Removed

As the final version of PEP 484 doesn't include Undefined, mypy no longer supports it. It's easy to migrate old code to not use Undefined -- just use type comments. For example, refactor this

      from typing import Undefined
      x = Undefined(int)

to this:

      x = None  # type: int

Support for Callable[..., t]

Mypy supports callable types with unspecified argument types. Just replace the arguments with an explicit ...:

    from typing import Callable
    def call_a_lot(func: Callable[..., int]) -> None:
        kwargs = {'arg': 1}
        func(1, x=y, **kwargs)  # OK, no checking of arguments

Support for Homogeneous Tuples (Tuple[t, ...])

Mypy now has a type for homogeneous, variable-length tuples: Tuple[t, ...] (with an explicit ...). These can be used like sequences, but the concrete type must be a tuple. These are useful for several library functions that expect concrete tuples, and also let us give precise types for *args arguments, which are tuples.

@no_type_check

You can use the @no_type_check decorator to skip all type checks within a function. It also lets you use function annotations for non-mypy uses:

    from typing import no_type_check

    @no_type_check
    def special_annotations(arg: {'value': 1}) -> 1234:   # Fine!
        1 + ''  # We can do all sorts of crazy things here, and mypy is silent!

Changes to Name Visibility in Stubs

As per PEP 484, mypy no longer export names imported in stubs, unless they are imported via as. For example, this stub won't export Tuple any more:

    from typing import Tuple

    def my_function() -> Tuple[int, str]: ...

This is especially nice together if you import a module with a stub using from module import *, as your namespace won't get polluted with random names imported by the stub from typing.

If in a stub you import using from m import *, everything in m will get exported. This makes it easy for a stub to re-export all definitions from another module.

Using ... in Stubs

PEP 484 recommends using an ellipsis (...) instead of pass for empty function and class bodies in stubs. It can can also be used as variable initializers and default argument values. This example illustrates all of these:

    # This is stub file
    var = ...  # type: int
    def function(arg: str = ...) -> int: ...
    class VeryBadError(Exception): ...

bytearray Is Compatible with bytes

Mypy considers bytearray to be compatible with bytes (as per PEP 484). This simplifies stub files and annotations, as previously we often had to use Union[bytes, bytearray] or something similar, which was hard to remember to do consistently.

Workflow Improvements

This release has several experimental improvements to running mypy against your code. These are subject to change in the future and not very well documented, but they are useful enough that I'll introduce them anyway:

  • You can give an arbitrary number of files or directories to type check on the mypy command line (e.g. mypy file1.py file2.py dir). Mypy will recurse into a directory argument if it contains an __init__.py or __init__.pyi (stub) file, looking for .py and .pyi files. If the directory is not a package, it'll only look for .py / .pyi files within the given directory but not in subdirectories.
  • You can ask mypy to ignore files outside those explicitly mentioned on the command line (or inferred by the directory traversal algorithm discussed above) via --silent-imports. Stubs are still consulted, but other than that mypy will not try to follow imports to other modules or complain about missing stubs. This makes it easy to start experimenting with mypy on a subset of a larger codebase without getting a huge number of errors about missing stubs and similar.
  • By default mypy type checks only functions with annotations. You can give mypy a the option --implicit-any to implicitly give an annotation with all Any types for every unannotated function. This way mypy can do useful (though a bit limited) type checking for unannotated code. As you add function annotations, mypy will do an incrementally better job. In the future, something like this may be the default mode of operation, once we get this good enough.

Miscellanea

Here are some additional improvements:

  • Mypy now continues processing files after errors and can report errors in multiple files in a single run
  • Rudimentary __new__ support
  • Better type inference for isinstance checks, including support for and and not operations involving isinstance and multiple checks per condition
  • Using a value with Any type as a base class (this is very useful together with # type: ignore)
  • Support for @abstractproperty

There are also some interesting work-in-progress features such as a new automatic stub generator.

Acknowledgements

First of all, I'd like to thank Dropbox for contributing very significant engineering resources to improving mypy, including some of my time! We've been experimenting with using mypy on Dropbox code and initial results are promising, even though there are some big unsolved challenges ahead, such as type checking performance and scaling to very large codebases (but we are working on these!).

Also, thanks to Guido and Matt Robben for feedback on earlier drafts of this article.

Many people contributed to this release. Thanks to everybody who helped! Apologies if I've missed anybody (this includes contributions to typeshed):

  • Andrew Aldridge
  • Anup Chenthamarakshan
  • Ben Darnell
  • Ben Longbons
  • ctcutler
  • Daniel Shaulov
  • Darjus Loktevic
  • David Fisher
  • David Shea
  • Della Anjeh
  • Eric Price
  • Gigi Sayfan
  • Greg Price
  • Guido van Rossum
  • Howard Lee
  • Ian Cordasco
  • icoxfog417
  • Igor Vuk
  • ismail-s
  • James Guthrie
  • Jared Hance
  • Jing Wang
  • Jukka Lehtosalo
  • Kyle Consalus
  • Li Haoyi
  • Matthew Wright
  • Matthias Bussonnier
  • Matthias Kramm
  • Michael Walter
  • Michal Pokorný
  • Mihnea Giurgea
  • Motoki Naruse
  • Prayag Verma
  • Robert T. McGibbon
  • Roy Williams
  • Ryan Gonzalez
  • Sander Kersten
  • Sebastian Reuße
  • Seo Sanghyeon
  • Tad Leonard
  • Tim Abbott
  • Vita Smid
  • Vlad Shcherbina
  • Wen Zhang
  • wizzardx
  • Yuval Langer

An up-to-date list of all mypy code contributions is available at the GitHub contributors page.

Friday, 10 April 2015

Mypy at PyCon 2015 in Montreal

I'm currently at PyCon 2015 in Montreal (and enjoying myself!). I'm happy to chat about anything related to mypy, PEP 484 (type hinting / typing), static typing or static analysis. Say hi if you see me, or meet me at the Dropbox booth or send me an email -- it can be quite difficult to find me among the crowds. There's probably also going to be an open space session that I will attend, maybe on Saturday. I'm going to be around until Wednesday morning.

Sunday, 5 April 2015

Mypy 0.2 Released

I just uploaded mypy 0.2 to PyPI! This release focuses on PEP 484 compatibility, but it also contains many general improvements. The changes are too numerous to enumerate them all. Instead, I'll focus here on the most visible and useful ones.

Install mypy using pip:

$ pip install mypy-lang

Mypy 0.2 breaks backward compatibility. You will probably have to modify your code to be PEP 484 draft compatible. As before, mypy is experimental and still has bugs. Backward compatibility won't be preserved until mypy hits 1.0. However, mypy is committed to becoming compatible with PEP 484.

PEP 484 Syntax Changes

Several changes improve PEP 484 compatibility. Mypy still doesn't implement all of PEP 484, but we've made a lot of progress.

'typevar' Replaced With 'TypeVar'

To define a type variable, use typing.TypeVar instead of typing.typevar. For example:

  from typing import TypeVar, Sequence

  T = TypeVar('T')

  def first(a: Sequence[T]) -> T:
      return a[0]

Function[...] Replaced With Callable[...]

Replace types like Function[...] with Callable[...]. Other than the name change, the types are identical.

No More Type Application Syntax

You can't write code like this any more:

  from typing import List

  names = List[str]()  # ERROR

Use a type comment instead:

  from typing import List

  names = []  # type: List[str]

Types such as List[str] are no longer ordinary classes. Previously, List[int] evaluated to just list at runtime, making it possible to support the idiom in the first example. List[str] now actually creates a new type object that preserves information about the str type argument. This is useful and allows using type information at runtime in a variety of ways. The tradeoff is that constructing instances using generic type objects is no longer possible. As a side benefit, there is usually a single obvious way to declare the type of a container object. Previously there were just too many ways of defining the types of variables and objects.

Any(x) Not Valid As a Cast

Use cast(Any, e) instead of Any(e); the prior is equivalent to the latter, and the latter syntax is no longer supported.

Overloading In Stubs Only

The @overload decorator is only supported in stubs. You can replace many uses of overloading using union types.

Silencing Mypy Errors Locally

You can use a "# type: ignore" comment to silence mypy errors on a particular line. For example, to make mypy not complain about a missing function in a stub:

  import acme

  # Don't complain about missing do_stuff
  acme.do_stuff()  # type: ignore

This is very useful if mypy gets confused about some particular behavior in your code. Instead of trying to refactor your code to pass type checking, you can just locally ignore individual errors.

Better Python Support

Mypy supports additional Python syntax. The mypy parser now can parse most of Python 3.2 standard library, and there's also support for a bunch of features introduced in later 3.x releases.

More Library Modules Supported

Additional stubs for standard library modules are included, and we've started adding support for 3rd party modules such as requests.

This mypy release also makes it easier to work with arbitrary library modules that aren't directly supported my mypy. You can silence errors due to missing library stubs using # type: ignore. For example, if you want to use frobnicate that has no stubs:

  from frobnicate import do_stuff  # type: ignore

  do_stuff(...)  # Okay!

Type Aliases

Mypy supports type aliases that make it easier to use more powerful types such as union types and callable types, and can be used to have more descriptive names for types. Example:

  from typing import Dict, List, Tuple

  DependencyMap = Dict[str, List[str]]
  Location = Tuple[str, str]

  def get_dependencies(base: Location) -> DependencyMap: ...

Miscellania

  • Mypy supports named tuples (both collections.namedtuple and typing.NamedTuple).
  • Mypy supports __call__ and __getattr__ (thanks Sander Kersten!).
  • There's an alternative stub file name extension .pyi. This means that you can have stubs and normal .py files in the same directory. This can also be used to override the default stubs for standard library modules that ship with mypy with local modified variants.
  • Optional[t] allows documenting whether None is a valid value for a type.
  • Better runtime introspection of types (for example, the type parameter int of List[int] is preserved; previously the type parameters were stripped at runtime).
  • The ABCs in typing are compatible with collections.abc. For example, now there's both Sequence and MutableSequence.
  • As always, many bugs were squashed, though there are still remaining open bugs.
  • The @ducktype class decorator was removed. Some built-in types have special handling in the type checker so that int is compatible with float, as it was before.
  • There is no longer typing.AbstractGeneric -- always use typing.Generic when defining generic classes.

There are also some interesting work-in-progress features such as a new automatic stub generator.

Acknowledgements

Many people contributed to this release. Thanks to everybody who helped! Here is a partial list:

  • Guido van Rossum, Łukasz Langa and others at typehinting (PEP 484)
  • Sander Kersten (continued with strong contributions, including several new Python features and a bunch of code cleanup)
  • Igor Vuk
  • Brodie Rao
  • Anders Schuller
  • Ryan Gonzalez
  • Marek Sapota

An up-to-date list of code contributions is available at the GitHub contributors page.

Saturday, 17 January 2015

Mypy and the PEP 484 (Type Hinting) Draft

Guido van Rossum just yesterday announced that the first draft of PEP 484 is available for review. This Python Enhancement Proposal defines a standard type annotation syntax, which is heavily inspired by mypy, for type hinting. The plan is to get it included in Python 3.5.

I've been contributing to the design and editing process and am very excited about the prospect of mypy supporting a standard annotation syntax. That's why (in addition to holidays) there hasn't been much activity in the mypy repo during the last month or so. Fear not -- there's a lot of things happening around mypy.

The PEP does not define a type checker or many of the details needed to implement a type system -- it gives a syntax and basic guidelines, around which various tools such as type checkers (including mypy), IDEs and maybe even optimizing (JIT) compilers can be implemented. There is still a lot of room for mypy to experiment and innovate around type checking Python code.

Why This Is Great

A standardized syntax could be boon for mypy. Here are just some benefits:

  • More people, projects and companies are likely to hear about and use a syntax that is standard. More users means more polish, more bugs fixed, and overall a better experience for everybody.
  • 3rd party libraries are likely to adopt type annotations when there is a standard. Mypy will catch more bugs when you use these static-typing-aware libraries in your programs.
  • Tools such as IDEs and documentation generators are likely to add support for standard type annotations.
  • There will be less fragmentation caused by multiple incompatible type syntax languages. This makes things less confusing and reduces duplicate work.

How Does This Affect Mypy Features?

The PEP will have some direct implicitations for mypy. As it's still a draft, we don't know what the final proposal will look like, but some aspects of mypy, mostly syntax, will likely change. Here are some of the most prominent differences (some of these are not yet reflected in the PEP draft):

  • The PEP uses Callable[...] instead of Function[...] for callable/function types.
  • Generic types are not valid in expressions. So code like
        x = List[int]()
        
    will be written like this (this has been supported by mypy since the early days):
        x = []  # type: List[int]
        
    Syntax for type annotations is still being discussed, and this is clearly one of the more controversial aspects of the proposal.
  • typevar is spelled TypeVar and has other syntactic changes.
  • Function overloading (@overload) is only available in library stubs. A future PEP may add runtime support for function overloading or multiple dispatch, but for now there's not going to be overloading for user code. This isn't really a major issue, since other mypy features such as union types (which are a recent addition and part of the PEP) make overloading rarely useful any more.
  • There's provisions for explicit type checking of None values. Currently mypy implicitly makes None compatible with every type, but this is likely going to change. For example, Optional[int] would be used for int or None (same as Union[int, None]).
  • There's a new type for variable length tuples: Tuple[int, ...] ('...' is part of the syntax).
  • Type arguments of generic types in function annotations will be available for introspection. Currently List[int] evaluates to just list which loses information at runtime.

I've added a bunch of issues that cover differences between the PEP and mypy.

Join the Discussion

Join the discussion of open PEP issues and suggest changes at https://github.com/ambv/typehinting!

Friday, 23 August 2013

Mypy Development Update #3

This is a short update about what's been happening in the mypy project.

I have (mostly) been on vacation recently, and now it's time to get back to mypy development. My PhD dissertation submission deadline also looms near, so I will have to work on that as well. Actually a lot of things are happening in my life — I will be relocating after finishing with my PhD, and this will involve (the typical) things such as finding a nice neighborhood, new schools/nurseries and shipping our pets and our belongings. I have already been glacial in responding to emails recently, and things may not get better until later this year.

I've been working on several new mypy features during the summer, though some of them are still incomplete. I want to get a "good-enough" prototype mypy implementation for my dissertation, and I really don't have time to polish things at this stage, which is a bit sad but can't really be helped. After submitting I can get back to doing things in a more orderly manner.

I'm really enthusisistic about some of the new features. Here is a sketch of some of them:

  • Multiple (implementation) inheritance now mostly works. This is a side effect of the switch to Python-compatible syntax, as mypy lost the distinction between classes and interfaces.
  • Abstract methods and abstract base classes (ABCs) are supported, using the abstractmethod decorator and the ABCMeta metaclass, as in Python.
  • Static methods are supported, using the staticmethod decorator.
  • There is support for read-only properties. Support for more complex properties is not there yet.
  • Type inference of code that uses isinstance checks is getting better. The nice result is that casts are often no longer needed in cases where they used to be required.
  • Conditional expressions of form x if y else z are supported.
  • For loops over tuple literals are supported.
  • The del statement is improved. You can now delete names and attributes. These don't affect type checking yet, but it makes it easier to adapt existing code that uses del statements to static typing.
  • I'm working on support for a feature which allows a kind of string polymorphism. It lets a single function implementation to support both str and bytes arguments with effective type checking. Previously either some code duplication or using dynamically typed code was needed in cases like this. This seems like a very useful feature especially for library code. I'll get back to this later when the implementation is more complete.
  • There is now some very limited Python 2.x support. It should be straightforward but moderately laborious to finish this, but other tasks are more urgent.
  • As always, several bugs have been fixed and standard library stubs have been improved.

As I had previously planned, I'm not currently working on the compiler. I focus on getting the type system and type checker reasonably feature-complete and stable first.

Tuesday, 2 July 2013

Mypy Switches to Python-Compatible Syntax

Mypy now has a Python-compatible syntax! The implementation is already self-hosting, but it should still be considered very experimental. This is a huge change which touched almost every source code file in the repository, and there are still many unfixed bugs.

Here is small example of the new syntax:

  import typing

  def fib(n: int) -> None:
      a, b = 0, 1
      while a < n:
          print(a)
          a, b = b, a+b

I have mostly updated the web site and documentation to reflect the new syntax. The Mypy Tutorial (formerly Mypy Overview) is largely rewritten:

http://www.mypy-lang.org/tutorial.html

The update also adds more content to the tutorial that isn't directly related to the syntax switch.

Since the mypy implementation is now a valid Python 3 program, there is no need for a translation step before running it. Thus also the old mypy-py repository is no longer needed to run mypy. Instead, mypy now uses the standard Python setup.py mechanism for installation. Have a look at the updated README for details:

https://github.com/JukkaL/mypy/blob/master/README.md

I have updated the example programs on the web site:

http://www.mypy-lang.org/examples.html

I also updated the mypy roadmap:

http://www.mypy-lang.org/roadmap.html

If you are thirsty for more code to look at, point your browser at the git repository:

From now on, the old syntax is considered legacy and will not be supported by the implementation. There will be a tool for migrating mypy code written using the old syntax to the new syntax. The tool is already functional, but it still needs a bit if polish before it's generally useful. I will notify when it's available on this blog. I'll also write a longer blog post discussing the new syntax in more detail at a later date.

Any feedback is appreciated! The syntax is still work in progress, and improvements are possible.

Many thanks to Guido van Rossum, Sebastian Riikonen and Ron Murawski for ideas and comments.

Friday, 28 June 2013

Python-Compatible Syntax for Mypy Is Almost Ready

This is just a quick update about what has been happening in the mypy project recently.

I have been working on the new Python-compatible syntax along with writing my dissertation during the last months. I've also done a lot of design and evaluation work related to potential new exciting mypy features. I won't go into details now, but these include much more powerful type inference, union types and better parametric polymorphism for string types.

The new syntax is currently mostly functional. I'm very happy with the initial results, and this now seems like the obvious way forward for mypy. The most important test was when I translated the entire mypy implementation to use the new syntax this Tuesday. I was able to do it in just a few hours using an automatic translator tool that I had written earlier, though there was quite a bit of manual clean-up work afterwards.

The code is available in the pythonsyntax branch for anybody who wants to play with it:

https://github.com/JukkaL/mypy/tree/pythonsyntax

Note that installing mypy now uses the standard setup.py approach instead of the old funky approach of using a git repository to distribute a runnable implementation. Have a look at the updated README for more information.

There is not much documentation, though. The wiki contains a short introruction to the new syntax:

http://www.mypy-lang.org/wiki/PythonCompatibleSyntax

The details of the syntax may still change, and I have many ideas of making the new syntax even better. I'll write about these later.