Skip to main content

PyPy gets a new compiler

Today, I merged the parser-compiler branch, which I have been working on over the summer. It contained a total rewrite of both PyPy's Python parser and AST compiler. PyPy's old parser was (in)famous internally for being complicated and slow (with many algorithmic complexities greater than O(n)). The new parser is a simple as I could make it LL(1) parser like CPython (though it doesn't share the hacks of CPython's parser).

The new compiler is based on the Abstract Syntax Trees (AST) that CPython 2.5 introduced instead of PyPy's old AST based on the compiler package's. This means that Python code running on PyPy will be able to use the same _ast interface as CPython. PyPy's _ast implementation supports AST features that CPython 2.6 added, including compiling modified AST to bytecode and executing it. In this rewrite, some more obscure compiler features were added, too. For example, jumps in bytecode can now be greater than 65535 bytes! (That's like an if statement with 7000 lines of code in the body.)

While the PyPy translation toolchain still has many obscure details and hacks, this merge completes the process of making the actual Python interpreter very clean. Hopefully, this will make adding new features much easier and make PyPy less frustrating to maintain as well as providing application level code with an improved AST interface!


Jeremy Cowles wrote on 2009-08-25 23:03:

Nice, keep up the good work!

Anonymous wrote on 2009-08-26 08:12:

Thank you.. Keep it up.

random user wrote on 2009-08-26 17:52:

Very nice. Thanks for all of your work!

tobami wrote on 2009-08-31 10:43:

Hi, the Gothenburg sprint news are very interesting.

What are your thoughts about a release roadmap?. Do you intend to release a pypy 1.2 with improved compatibility and speed but no JIT, and later include the JIT (version 1.5, maybe?)?.

I think publishing some kind of roadmap would be useful, as a project suffers when its release cycles are BOTH long and unpredictable.

tobami wrote on 2009-08-31 10:51:

Also, starting from the next stable release, it would be great to publish some kind of benchmarks page to keep track of performance across different versions (cpython 2.6 vs pypy 1.1 vs pypy 1.2 vs pypy with JIT).

Now that I think of it, do you need some kind of help with the website?. I think starting with the next pypy's release, the project will get a lot more visibility and a nicer and better structured website would be a definite plus. If you feel it would be a useful task I could help there.

Maciej Fijalkowski wrote on 2009-08-31 16:05:


Both, the benchmarks (that would also include say jython) and a nice website for people who actually want to use it would be a very nice addon. We definitely would appreciate some help with it.

If you have any ideas feel free to continue discussion on pypy-dev.


tobami wrote on 2009-08-31 21:46:

Hi Maciej, as you suggested, I have subscribed to the pypy-dev mailing list and have started the discussion.



Maciej Fijalkowski wrote on 2009-09-01 15:47:

Hey Miguel.

I fail to see your post.


tobami wrote on 2009-09-02 10:28:

it got rejected. I have written to pypy-dev-owner to see where the problem is.


Maciej Fijalkowski wrote on 2009-09-05 10:13:


you should subscribe to the list first.
We get far too much spam to accept
posts from non-members.


tobami wrote on 2009-09-05 21:21:


well, I subscribed first, that is the problem. I now get sent the pypy-dev mailing list, but my post got rejected anyway. And pypy-owner hasn't answered yet.

What can I do?

Maciej Fijalkowski wrote on 2009-09-06 19:57:


you did something wrong. pypy-dev
is not a moderated list (from
members, that is). Can you leave your mail, so we can no longer spam here? Mine is fijal at