The AST and Me

Emily Morehouse

Co-Founder, Director of Engineering

@ Cuttlesoft

We're going to cover:

Life cycle of a piece of Python code
Interacting with your code at various stages
Current Python optimizations
Practical applications

We're going to cover:

Life cycle of a piece of Python code
Interacting with your code at various stages
Current Python optimizations
Practical applications

We're going to cover:

Life cycle of a piece of Python code
Interacting with your code at various stages
Current Python optimizations
Practical applications

We're going to cover:

Life cycle of a piece of Python code
Interacting with your code at various stages
Current Python optimizations
Practical applications

Should you care about language internals?

A Peek Under The Hood

From Source Code To Execution:

Q: Interpreted or compiled?

A: Both!

Compiler → Generates Bytecode

Interpreter → Executes Bytecode

Life Cycle of a Piece of Python Code

What is an Abstract Syntax Tree?

Tree representing the structure of your source code.

What is... a tree?

(1 + 2) * 3

Let's dig in.

Primary Tools:

ast module
dis module

ast module
dis module

Secondary Libraries:

astor
meta
codegen

astor
meta
codegen

astor
meta
codegen

Code Objects:

Contains instructions and information needed to run the code.
Internal representation of a piece of Python code.

Contains instructions and information needed to run the code.
Internal representation of a piece of Python code.

co_name
co_varnames
co_stacksize
co_consts
co_argcount
co_code

co_name
co_varnames
co_stacksize
co_consts
co_argcount
co_code

co_name
co_varnames
co_stacksize
co_consts
co_argcount
co_code

co_name
co_varnames
co_stacksize
co_consts
co_argcount
co_code

co_name
co_varnames
co_stacksize
co_consts
co_argcount
co_code

co_name
co_varnames
co_stacksize
co_consts
co_argcount
co_code

print("may the force be with you")


                        if a == 23: 

                                print("may the force be with you")


                        a = 32 

                        if a == 23: 

                                print("may the force be with you")

Current Optimizations

Python's compiler is purposefully simple
Peephole Optimizer
Few AST optimizations besides constant folding

Python's compiler is purposefully simple
Peephole Optimizer
Few AST optimizations besides constant folding

Python's compiler is purposefully simple
Peephole Optimizer
Few AST optimizations besides constant folding

Peephole Optimizations

Looking around without moving your head.


                        x = 1

                        y = x + 2

y = 1 + 2

Constant Folding

Evaluating constant expressions at compile time.

a = 2 * 3

a = 6

How The AST Affects Your Code

Bytecode is created using an AST
Bytecode is stored in *.pyc files
VERY micro speed-ups

Bytecode is created using an AST
Bytecode is stored in *.pyc files
VERY micro speed-ups

Bytecode is created using an AST
Bytecode is stored in *.pyc files
VERY micro speed-ups

Example Applications / Projects

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

VOC (PyBee)
Transcrypt (Python to JS)

Code transformations

Jinja (templating engine)

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

VOC (PyBee)
Transcrypt (Python to JS)

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

VOC (PyBee)
Transcrypt (Python to JS)

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

VOC (PyBee)
Transcrypt (Python to JS)

Code transformations

Code generators

Hypothesis (test generation)
Cog (file/text generation)

Alternative interpreters
Transpiling / Alternative Compilation

VOC (PyBee)
Transcrypt (Python to JS)

Code transformations

Jinja (templating engine)

Code analysis

Code linting
Code formatters

Code transformations

Code analysis

Code linting
Code formatters

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Post-CSS
Babel (ESNext -> ESNow)
Migrations (e.g. Python 2 -> 3)
React Native

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Post-CSS
Babel (ESNext -> ESNow)
Migrations (e.g. Python 2 -> 3)
React Native

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Post-CSS
Babel (ESNext -> ESNow)
Migrations (e.g. Python 2 -> 3)
React Native

Code analysis

Python Security's PyT

Code linting
Code formatters

YAPF
Black
Prettier

Code transformations

Post-CSS
Babel (ESNext -> ESNow)
Migrations (e.g. Python 2 -> 3)
React Native

Improve and speed up your code (🤷🏻‍♀️)
Debug errors/oddities
Change the Python grammar
Round-Tripping

Improve and speed up your code
Debug errors/oddities
Change the Python grammar
Round-Tripping

Improve and speed up your code (🤷🏻‍♀️)
Debug errors/oddities
Change the Python grammar
Round-Tripping

Improve and speed up your code (🤷🏻‍♀️)
Debug errors/oddities
Change the Python grammar
Round-Tripping

Improve and speed up your code (🤷🏻‍♀️)
Debug errors/oddities
Change the Python grammar
Round-Tripping

Improve and speed up your code (🤷🏻‍♀️)
Debug errors/oddities
Change the Python grammar
Round-Tripping

Round-Tripping:

Comparing the AST from your code before and after making changes.

Go forth and build awesome tools!

These are not made for your eyes now, they're here if you want to reference something later! --- Use for info on interpreted vs compiled: * http://akaptur.com/blog/2013/12/03/introduction-to-the-python-interpreter-4/ - Allison Kaptur - Introduction to the Python Interpreter, Part 4: It's Dynamic! * https://www.python-course.eu/python3_execute_script.php Generic compilation process: * https://courses.cs.washington.edu/courses/cse401/16wi/lectures/05-implementing-coroutines-with-bytecode-interpreter-wi16.pdf Definition of AST: * https://en.wikipedia.org/wiki/Abstract_syntax_tree CPython's Compiler Documentation * https://devguide.python.org/compiler/ Use for showing pre- and post- AST with constant folding: * http://www.cs.dartmouth.edu/~mckeeman/cs48/mxcom/doc/CodeOptimization.html AST Visualizer - GUI * https://github.com/ivan111/vpyast * https://vpyast.appspot.com/ AST view - HTML * http://pythonhosted.org/pyRegurgitator/ Breaking compilation down * https://eli.thegreenplace.net/2009/11/28/python-internals-working-with-python-asts AST vs CST: * https://eli.thegreenplace.net/2009/02/16/abstract-vs-concrete-syntax-trees/ DIS example to show bytecode changes: * https://gist.github.com/ramalho/7ceaf014d125749f0b0c Why knowing about your bytecode (and therefore AST) can help find speedups): * http://multigrad.blogspot.com/2014/06/fun-with-python-bytecode.html Improving your code: * http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/ Use for showing weird peephole optimization: * https://github.com/akaptur/peephole-optimization - Allison Kaptur - Peephole Optimizer Github Project * http://akaptur.com/blog/2014/08/02/the-cpython-peephole-optimizer-and-you/ - Allison Kaptur - The CPython Peephole Optimizer and You Round-Trip Engineering: * http://www.lysator.liu.se/xenofarm/argouml/work/argouml/www/documentation/defaulthtml/manual/ch06.html

The end!

Talk is hosted at:
- emilyemorehouse.github.io/ast-and-me
Talk source code and experimentation scripts are at:
- github.com/emilyemorehouse/ast-and-me