From fbe4c4f64680e73a1e38a21cd1e76cbdcfed4391 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 24 Aug 2021 21:26:20 -0700 Subject: [PATCH 01/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7d8e172..9501e87 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# chocopy-python-frontend +# chocopy-python-compiler AOT compiler for Chocopy, written entirely in Python. [Chocopy](https://chocopy.org/) is a subset of Python 3.6 that is used for Berkeley's compilers course, and has a reference compiler written in Java. From 162ccca40a04a0080b1c89895e709bd3f78d4c18 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 24 Aug 2021 21:27:53 -0700 Subject: [PATCH 02/79] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9501e87..431c823 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ AOT compiler for Chocopy, written entirely in Python. [Chocopy](https://chocopy.org/) is a subset of Python 3.6 that is used for Berkeley's compilers course, and has a reference compiler written in Java. -This compiler matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation. Additionally, this compiler contains 2 backends not found in the reference implementation: +This compiler includes a frontend and a backend. The frontend matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation. The compiler currently supports 2 backends, which are not found in the reference implementation: - Untyped Python 3 source code - JVM bytecode, formatted for the Krakatau assembler @@ -12,7 +12,7 @@ Most of the test cases are taken from test suites included in the release code f ## Requires: - Python 3.6 - 3.8 -- [Krakatau JVM Assembler](https://github.com/Storyyeller/Krakatau) (only if you want to use the JVM backend) +- [Krakatau assembler](https://github.com/Storyyeller/Krakatau) (only if you want to use the JVM backend) ## Usage From 588cba0cb330bd63f00e06420a32ba47c25c4468 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Sun, 29 Aug 2021 22:29:42 -0700 Subject: [PATCH 03/79] Update README.md --- README.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 431c823..eb9b382 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,18 @@ # chocopy-python-compiler -AOT compiler for Chocopy, written entirely in Python. [Chocopy](https://chocopy.org/) is a subset of Python 3.6 that is used for Berkeley's compilers course, and has a reference compiler written in Java. +An ahead-of-time compiler for Chocopy, written entirely in Python. [Chocopy](https://chocopy.org/) is a subset of Python 3.6 that is used as a teaching language for Berkeley's compilers course, and has a reference implementation targeting RISC-V written in Java. It includes a relatively large set of features from Python, such as: lists, classes, methods, nested functions, and nonlocals. -This compiler includes a frontend and a backend. The frontend matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation. The compiler currently supports 2 backends, which are not found in the reference implementation: -- Untyped Python 3 source code -- JVM bytecode, formatted for the Krakatau assembler +This project is mostly a tool for me to learn/practice compiler writing, and I plan to extend it with additional backends if/when I have time. I hope that this project will also become a useful educational reference for other people. -That means that you can parse and typecheck the Chocopy file with this compiler, then use the reference implementation's backend to handle assembly code generation. +Progress is documented on my [blog](https://yangdanny97.github.io/blog/): +- [Part 1: Frontend/Typechecker](https://yangdanny97.github.io/blog/2020/05/29/chocopy-typechecker) +- [Part 2: JVM backend](https://yangdanny97.github.io/blog/2021/08/26/chocopy-jvm-backend) -Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. +This compiler's frontend matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation. That means that you can parse and typecheck the Chocopy file with this compiler, then use the reference implementation's backend to handle assembly code generation. + +This compiler currently supports 2 backends, which are not found in the reference implementation: +- Untyped Python 3 source code +- JVM bytecode, formatted for the Krakatau assembler ## Requires: - Python 3.6 - 3.8 @@ -62,9 +66,12 @@ The `compile_jvm.sh` script is a useful utility to compile and run files with th Note that in the above example commands & the `compile_jvm.sh` script all expect the Krakatau directory and this repository's directory to share the same parent - commands will differ if you cloned Krakatau to a different location. -### JVM Backend - Known Incompatibilities: +### JVM Backend - Known Issues: - Since bytecode for each class is stored in a separate file, on operating systems with case-insensitive file names (like MacOS) you cannot have 2 classes whose names only differ by case. +- Since the main JVM class for a Chocopy program shares the name of the file, do not define classes with the same name as the source file. - The special parameter `self` in methods and constructors may not be referenced by a `nonlocal` declaration. The Java equivalent, `this`, is final and cannot be assigned to. - Some large programs may cause the JVM to run out of stack space, since each frame currently has a maximum stack size of 500. +## Test Suite +Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. From 58c75d742334e2193b5b71e5d3852fc2b8478637 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 8 Feb 2022 08:49:00 -0800 Subject: [PATCH 04/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eb9b382..9fdcee2 100644 --- a/README.md +++ b/README.md @@ -74,4 +74,4 @@ Note that in the above example commands & the `compile_jvm.sh` script all expect ## Test Suite -Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. +Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. The runtime test suite for the JVM backend were evaluated using Java 8 on my local machine. From ed47f385b0a8f9dcdeabfd956be65993b5a83186 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 9 May 2022 21:21:25 -0700 Subject: [PATCH 05/79] rename --- README.md | 6 +++--- compile_jvm.sh => demo_jvm.sh | 0 2 files changed, 3 insertions(+), 3 deletions(-) rename compile_jvm.sh => demo_jvm.sh (100%) diff --git a/README.md b/README.md index 7d8e172..217eb53 100644 --- a/README.md +++ b/README.md @@ -57,10 +57,10 @@ The JVM backend for this compiler outputs JVM bytecode in plaintext formatted fo - Example: `java -cp ` - Example: `java -cp . binary_tree` -The `compile_jvm.sh` script is a useful utility to compile and run files with the JVM backend with a single command (provide the path to the input source file as an argument). -- To run the same example as above, run `./compile_jvm.sh tests/runtime/binary_tree.py` +The `demo_jvm.sh` script is a useful utility to compile and run files with the JVM backend with a single command (provide the path to the input source file as an argument). +- To run the same example as above, run `./demo_jvm.sh tests/runtime/binary_tree.py` -Note that in the above example commands & the `compile_jvm.sh` script all expect the Krakatau directory and this repository's directory to share the same parent - commands will differ if you cloned Krakatau to a different location. +Note that in the above example commands & the `demo_jvm.sh` script all expect the Krakatau directory and this repository's directory to share the same parent - commands will differ if you cloned Krakatau to a different location. ### JVM Backend - Known Incompatibilities: - Since bytecode for each class is stored in a separate file, on operating systems with case-insensitive file names (like MacOS) you cannot have 2 classes whose names only differ by case. diff --git a/compile_jvm.sh b/demo_jvm.sh similarity index 100% rename from compile_jvm.sh rename to demo_jvm.sh From 8db64bd4d06c9fc6e8461d5934cd75b3dc5abae8 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 9 May 2022 21:58:01 -0700 Subject: [PATCH 06/79] setup CIL backend, initial demo --- Makefile | 2 + compiler/cil_backend.py | 222 ++++++++++++++++++++++++++++++++++++++++ compiler/compiler.py | 9 +- demo_cil.sh | 8 ++ main.py | 17 ++- 5 files changed, 254 insertions(+), 4 deletions(-) create mode 100644 compiler/cil_backend.py create mode 100755 demo_cil.sh diff --git a/Makefile b/Makefile index bb37513..d728f56 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,8 @@ clean: rm -f *.j rm -f *.class + rm -f *.cil + rm -f *.exe rm -f *.ast rm -f *.ast.typed rm -f *.test.py diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py new file mode 100644 index 0000000..57801c9 --- /dev/null +++ b/compiler/cil_backend.py @@ -0,0 +1,222 @@ +from .astnodes import * +from .types import * +from .builder import Builder +from .typesystem import TypeSystem +from .visitor import Visitor +from collections import defaultdict +import json + + +class CilBackend(Visitor): + + def __init__(self, main: str, ts: TypeSystem): + self.classes = dict() + self.classes[main] = Builder(main) + self.currentClass = main + self.main = main # name of main class + self.locals = [defaultdict(lambda: None)] + self.counter = 0 # for labels + self.returnType = None + self.localLimit = 50 + self.stackLimit = 500 + self.ts = ts + + def indent(self): + self.instr("{") + self.currentBuilder().indent() + + def unindent(self): + self.currentBuilder().unindent() + self.instr("}") + + def currentBuilder(self): + return self.classes[self.currentClass] + + def visit(self, node: Node): + node.visit(self) + + def instr(self, instr: str): + self.currentBuilder().newLine(instr) + + def newLabelName(self) -> str: + self.counter += 1 + return "IL_"+str(self.counter) + + def label(self, name: str) -> str: + self.currentBuilder().unindent() + self.instr(name+":") + self.currentBuilder().indent() + + def enterScope(self): + self.locals.append(defaultdict(lambda: None)) + + def exitScope(self): + self.locals.pop() + + def returnInstr(self, exprType: ValueType): + raise Exception("unimplemented") + + def wrap(self, val:Expr, elementType:ValueType): + raise Exception("unimplemented") + + def store(self, name: str, t: ValueType): + raise Exception("unimplemented") + + def load(self, name: str, t: ValueType): + raise Exception("unimplemented") + + def arrayStore(self, elementType:ValueType): + raise Exception("unimplemented") + + def arrayLoad(self, elementType:ValueType): + raise Exception("unimplemented") + + def newLocalEntry(self, name: str) -> int: + raise Exception("unimplemented") + + def genLocalName(self, offset: int) -> str: + raise Exception("unimplemented") + + def newLocal(self, name: str = None, isRef: bool = True) -> int: + raise Exception("unimplemented") + + def visitStmtList(self, stmts:[Stmt]): + if len(stmts) == 0: + self.instr("nop") + else: + for s in stmts: + self.visit(s) + + def Program(self, node: Program): + # TODO + self.instr(f".assembly '{self.main}'") + self.instr("{") + self.instr("}") + self.instr(f".module {self.main}.exe") + self.instr(f".class public auto ansi beforefieldinit {self.main} extends [mscorlib]System.Object") + self.indent() + self.instr(".method public static hidebysig default void Main (string[] args) cil managed") + self.indent() + self.instr(".entrypoint") + self.instr(f".maxstack {self.localLimit}") + self.visitStmtList(node.statements) + self.instr("ret") + self.unindent() + self.unindent() + + def ClassDef(self, node: ClassDef): + raise Exception("unimplemented") + + def FuncDef(self, node: FuncDef): + raise Exception("unimplemented") + + def VarDef(self, node: VarDef): + raise Exception("unimplemented") + + # STATEMENTS + + def AssignStmt(self, node: AssignStmt): + raise Exception("unimplemented") + + def IfStmt(self, node: IfStmt): + raise Exception("unimplemented") + + def ExprStmt(self, node: ExprStmt): + self.visit(node.expr) + # TODO + + def BinaryExpr(self, node: BinaryExpr): + raise Exception("unimplemented") + + def IndexExpr(self, node: IndexExpr): + raise Exception("unimplemented") + + def UnaryExpr(self, node: UnaryExpr): + raise Exception("unimplemented") + + def CallExpr(self, node: CallExpr): + # TODO + for i in range(len(node.args)): + self.visitArg(node.function.inferredType, i, node.args[i]) + self.instr("call void class [mscorlib]System.Console::WriteLine(string)") + # raise Exception("unimplemented") + + def ForStmt(self, node: ForStmt): + raise Exception("unimplemented") + + def ListExpr(self, node: ListExpr): + raise Exception("unimplemented") + + def WhileStmt(self, node: WhileStmt): + raise Exception("unimplemented") + + def ReturnStmt(self, node: ReturnStmt): + raise Exception("unimplemented") + + def Identifier(self, node: Identifier): + raise Exception("unimplemented") + + def MemberExpr(self, node: MemberExpr): + raise Exception("unimplemented") + + def IfExpr(self, node: IfExpr): + raise Exception("unimplemented") + + def MethodCallExpr(self, node: MethodCallExpr): + raise Exception("unimplemented") + + # LITERALS + + def BooleanLiteral(self, node: BooleanLiteral): + raise Exception("unimplemented") + + def IntegerLiteral(self, node: IntegerLiteral): + raise Exception("unimplemented") + + def NoneLiteral(self, node: NoneLiteral): + raise Exception("unimplemented") + + def StringLiteral(self, node: StringLiteral): + self.instr(f"ldstr {json.dumps(node.value)}") + + # TYPES + + def TypedVar(self, node: TypedVar): + pass + + def ListType(self, node: ListType): + pass + + def ClassType(self, node: ClassType): + pass + + def emit(self) -> str: + return self.currentBuilder().emit() + + # SUGAR + + def NonLocalDecl(self, node: NonLocalDecl): + pass + + def GlobalDecl(self, node: GlobalDecl): + pass + + # BUILT-INS - note: these are in-lined + def emit_assert(self, arg: Expr): + raise Exception("unimplemented") + + def emit_exn(self, msg: str): + raise Exception("unimplemented") + + def emit_input(self): + raise Exception("unimplemented") + + def emit_len(self, arg: Expr): + raise Exception("unimplemented") + + def emit_print(self, arg: Expr): + raise Exception("unimplemented") + + def visitArg(self, funcType, paramIdx: int, arg: Expr): + self.visit(arg) + # TODO diff --git a/compiler/compiler.py b/compiler/compiler.py index 977bfa5..3ae7bea 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -8,6 +8,7 @@ from .nestedfunchoister import NestedFuncHoister from .typesystem import TypeSystem from .jvm_backend import JvmBackend +from .cil_backend import CilBackend from .python_backend import PythonBackend import ast from pathlib import Path @@ -64,5 +65,11 @@ def emitJVM(self, main:str, ast: Node): jvm_backend = JvmBackend(main, self.transformer.ts) jvm_backend.visit(ast) return jvm_backend.classes - + + def emitCIL(self, main:str, ast: Node): + self.closurepass(ast) + EmptyListTyper().visit(ast) + cil_backend = CilBackend(main, self.transformer.ts) + cil_backend.visit(ast) + return cil_backend.classes diff --git a/demo_cil.sh b/demo_cil.sh new file mode 100755 index 0000000..235780b --- /dev/null +++ b/demo_cil.sh @@ -0,0 +1,8 @@ +# utility for compiling a Chocopy file to .exe files and running it +base_name="$(basename $1 .py)" + +rm -f *.cil +rm -f *.exe +python3 main.py --mode cil $1 . +ls *.cil | xargs -L1 ilasm +mono $base_name.exe \ No newline at end of file diff --git a/main.py b/main.py index fdcc465..57b8cd8 100644 --- a/main.py +++ b/main.py @@ -11,6 +11,7 @@ 'python - output untyped Python 3 source code\n' + 'hoist - output untyped Python 3 source code w/o nonlocals or nested function definitions\n' + 'jvm - output JVM bytecode formatted for the Krakatau assembler\n' + 'cil - output CIL bytecode formatted for the Mono ilasm assembler\n' ) def out_msg(path, verbose): @@ -19,7 +20,7 @@ def out_msg(path, verbose): def main(): parser = argparse.ArgumentParser(description='Chocopy frontend') - parser.add_argument('--mode', dest='mode', choices=["parse", "tc", "python", "jvm", "hoist"], default="python", + parser.add_argument('--mode', dest='mode', choices=["parse", "tc", "python", "jvm", "hoist", "cil"], default="python", help=mode_help) parser.add_argument('--print', dest='should_print', action="iframe.php?url=https%3A%2F%2Fgithub.com%2Fstore_true", help="output to stdout instead of file") @@ -45,7 +46,6 @@ def main(): raise Exception("Error: input file must end with .py") infile_name = infile[:-3].split("/")[-1] - infile_no_extension = infile[:-3] if outdir is None: outdir = "./" @@ -113,7 +113,18 @@ def main(): fname = outdir + cls + ".j" with open(fname, "w") as f: out_msg(fname, args.verbose) - f.write(jvm_emitter.emit()) + f.write(jvm_emitter.emit()) + elif args.mode == "cil": + cil_emitters = compiler.emitCIL(infile_name, tree) + for cls in cil_emitters: + cil_emitter = cil_emitters[cls] + if args.should_print: + print(cil_emitter.emit()) + else: + fname = outdir + cls + ".cil" + with open(fname, "w") as f: + out_msg(fname, args.verbose) + f.write(cil_emitter.emit()) if __name__ == "__main__": main() From 9317cbe43c99b65f3d3abc7174b068bacc3dca65 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 11 May 2022 00:01:31 -0700 Subject: [PATCH 07/79] update readme, literals --- README.md | 62 +++++++++++++++++++++++++++++++++-------- compiler/cil_backend.py | 9 ++++-- 2 files changed, 57 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 217eb53..92cf27e 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,30 @@ -# chocopy-python-frontend +# chocopy-python-compiler -AOT compiler for Chocopy, written entirely in Python. [Chocopy](https://chocopy.org/) is a subset of Python 3.6 that is used for Berkeley's compilers course, and has a reference compiler written in Java. +Ahead-of-time compiler for [Chocopy](https://chocopy.org/), a subset of Python 3.6 with type annotations static type checking. -This compiler matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation. Additionally, this compiler contains 2 backends not found in the reference implementation: +Chocopy is used in compiler courses at several universities. This project has no relation to those courses, and is purely for my own learning/practice/fun. Detailed progress writeups are documented on [my blog](https://yangdanny97.github.io/blog/). + +## Features + +This compiler is written entirely in Python. Since Chocopy is itself a subset of Python, lexing and parsing can be entirely handled by Python's `ast` module. + +This compiler matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation's backend. That means that you can parse and typecheck the Chocopy file with this compiler, then use the reference implementation's backend to handle assembly code generation. + +Additionally, this compiler contains 2 backends not found in the reference implementation: - Untyped Python 3 source code - JVM bytecode, formatted for the Krakatau assembler +- CIL bytecode, formatted for the Mono ilasm assembler -That means that you can parse and typecheck the Chocopy file with this compiler, then use the reference implementation's backend to handle assembly code generation. +The test suite includes both static validation of generated/annotated ASTs, as well as runtime tests that actually execute the output programs to check correctness. Many of the AST validation test cases are taken from test suites included in the release code for Berkeley's CS164, with some additional tests written for more coverage. -Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. - -## Requires: +## Requirements: - Python 3.6 - 3.8 -- [Krakatau JVM Assembler](https://github.com/Storyyeller/Krakatau) (only if you want to use the JVM backend) +- JVM Backend Requirements: + - [Krakatau JVM Assembler](https://github.com/Storyyeller/Krakatau) + - Tested with Java 8 +- CIL Backend Requirements: + - [Mono](https://www.mono-project.com/) + - Tested with Mono 6.12 ## Usage @@ -22,6 +34,7 @@ The input file should have extension `.py`. If the output file is not provided, - AST JSON outputs will be written to a file of the same name/location as the input file, with extension `.py.ast` - Python source outputs will be written to a file of the same name/location as the input file, with extension `.out.py` - JVM outputs will be written to the same location as the input file, with the extension `.j` +- CIL outputs will be written to the same location as the input file, with the extension `.cil` **Flags:** @@ -35,6 +48,7 @@ The input file should have extension `.py`. If the output file is not provided, - `python` - output untyped Python 3 source code - `hoist` - output untyped Python 3 source code w/o nonlocals or nested function definitions - `jvm` - output JVM bytecode formatted for the Krakatau assembler + - `cil` - output CIL bytecode formatted for the Mono ilasm assembler ## Differences from the reference implementation: @@ -62,9 +76,35 @@ The `demo_jvm.sh` script is a useful utility to compile and run files with the J Note that in the above example commands & the `demo_jvm.sh` script all expect the Krakatau directory and this repository's directory to share the same parent - commands will differ if you cloned Krakatau to a different location. -### JVM Backend - Known Incompatibilities: -- Since bytecode for each class is stored in a separate file, on operating systems with case-insensitive file names (like MacOS) you cannot have 2 classes whose names only differ by case. +### JVM Backend - Known Issues/Incompatibilities: +- Since bytecode for each class is stored in a separate file, on operating systems with case-insensitive file names you cannot have 2 classes whose names only differ by case. - The special parameter `self` in methods and constructors may not be referenced by a `nonlocal` declaration. The Java equivalent, `this`, is final and cannot be assigned to. -- Some large programs may cause the JVM to run out of stack space, since each frame currently has a maximum stack size of 500. +- Some large programs may cause the JVM to run out of stack space, since each frame currently has a hardcoded maximum stack size of 500. +- Integers are compiled to regular ints instead of longs, so this backend will not work on 32-bit JVMs. + +## CIL Backend Notes: +The CIL backend for this compiler outputs CIL bytecode in plaintext formatted for the Mono ilsam assembler: +1. Use this compiler to generate plaintext bytecode + - Format: `python3 main.py --mode cil ` + - Example: `python3 main.py --mode cil tests/runtime/binary_tree.py .` +2. Run the ilasm assembler to generate `.exe` files - note that this must be done for EACH .cil file generated by the compiler + - Format: `ilasm <.cil file>` + - Example: `ls *.cil | xargs -L1 ilasm` +3. Run the `.exe` files + - Example: `mono <.exe file>` + - Example: `mono binary_tree.exe` + +The `demo_cil.sh` script is a useful utility to compile and run files with the CIL backend with a single command (provide the path to the input source file as an argument). +- To run the same example as above, run `./demo_cil.sh tests/runtime/binary_tree.py` + +## FAQ +- What is this for? + - The primary goal of the project is for me to practice compiler implementation. The secondary goal is to provide a reference to anyone else who is interested in the topics I explore through working on this project - I go into more detail about each part of the compiler on my blog. +- Why Chocopy? + - It has a detailed spec and is a relatively small language while being non-trivial enough to offer interesting compiler implementation problems. +- Why not design your own language? + - This project is focused on compiler implementation. I want to keep the project very focused and make each addition manageable so that I can make progress in my very limited spare time. +- Why implement this in Python? + - Since Chocopy is a subset of Python, implementing the compiler in Python means I do not have to write my own lexer and parser. This was explicitly something I wanted to experiment with while writing the frontend, and it worked wonderfully. The secondary reason is that writing it in Python means I can prototype new ideas faster. The lack of type safety in the compiler codebase is mitigated by an extensive test suite. diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 57801c9..236820f 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -168,13 +168,16 @@ def MethodCallExpr(self, node: MethodCallExpr): # LITERALS def BooleanLiteral(self, node: BooleanLiteral): - raise Exception("unimplemented") + if node.value: + self.instr("ldc.i4.1") + else: + self.instr("ldc.i4.0") def IntegerLiteral(self, node: IntegerLiteral): - raise Exception("unimplemented") + self.instr(f"ldc.i8 {node.value}") def NoneLiteral(self, node: NoneLiteral): - raise Exception("unimplemented") + self.instr("ldnull") def StringLiteral(self, node: StringLiteral): self.instr(f"ldstr {json.dumps(node.value)}") From 1ae429dabea17de30943295204203902429b8523 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 16 May 2022 16:07:57 -0400 Subject: [PATCH 08/79] strings and arithmetic --- compiler/cil_backend.py | 275 +++++++++++++++++++++++++++---- compiler/types/classvaluetype.py | 16 ++ compiler/types/functype.py | 1 - compiler/types/listvaluetype.py | 3 + 4 files changed, 266 insertions(+), 29 deletions(-) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 236820f..2ca10c1 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -20,6 +20,7 @@ def __init__(self, main: str, ts: TypeSystem): self.localLimit = 50 self.stackLimit = 500 self.ts = ts + self.defaultToGlobals = False # treat all vars as global if this is true def indent(self): self.instr("{") @@ -44,7 +45,7 @@ def newLabelName(self) -> str: def label(self, name: str) -> str: self.currentBuilder().unindent() - self.instr(name+":") + self.instr(name+": nop") self.currentBuilder().indent() def enterScope(self): @@ -53,17 +54,22 @@ def enterScope(self): def exitScope(self): self.locals.pop() - def returnInstr(self, exprType: ValueType): - raise Exception("unimplemented") - def wrap(self, val:Expr, elementType:ValueType): raise Exception("unimplemented") def store(self, name: str, t: ValueType): - raise Exception("unimplemented") + n = self.locals[-1][name] + if n is None: + raise Exception( + f"Internal compiler error: unknown name {name} for store") + self.instr(f"stloc {n}") def load(self, name: str, t: ValueType): - raise Exception("unimplemented") + n = self.locals[-1][name] + if n is None: + raise Exception( + f"Internal compiler error: unknown name {name} for load") + self.instr(f"ldloc {n}") def arrayStore(self, elementType:ValueType): raise Exception("unimplemented") @@ -75,10 +81,16 @@ def newLocalEntry(self, name: str) -> int: raise Exception("unimplemented") def genLocalName(self, offset: int) -> str: - raise Exception("unimplemented") + return f"__local__{offset}" - def newLocal(self, name: str = None, isRef: bool = True) -> int: - raise Exception("unimplemented") + def newLocal(self, name: str = None) -> int: + # store the top of stack as a new local + n = len(self.locals[-1]) + self.instr(f"stloc {n}") + if name is None: + name = self.genLocalName(n) + self.locals[-1][name] = n + return n def visitStmtList(self, stmts:[Stmt]): if len(stmts) == 0: @@ -95,11 +107,21 @@ def Program(self, node: Program): self.instr(f".module {self.main}.exe") self.instr(f".class public auto ansi beforefieldinit {self.main} extends [mscorlib]System.Object") self.indent() + + var_decls = [d for d in node.declarations if isinstance(d, VarDef)] + for v in var_decls: + self.instr(f".field public static {v.var.t.getCILName()} {v.var.identifier.name}") + self.instr(".method public static hidebysig default void Main (string[] args) cil managed") self.indent() self.instr(".entrypoint") self.instr(f".maxstack {self.localLimit}") + self.defaultToGlobals = True + for v in var_decls: + self.visit(v.value) + self.instr(f"stsfld {v.var.t.getCILName()} {self.main}::{v.var.identifier.name}") self.visitStmtList(node.statements) + self.defaultToGlobals = False self.instr("ret") self.unindent() self.unindent() @@ -115,31 +137,176 @@ def VarDef(self, node: VarDef): # STATEMENTS + def processAssignmentTarget(self, target: Expr): + if isinstance(target, Identifier): + if self.defaultToGlobals or target.varInstance.isGlobal: + self.instr( + f"stsfld {target.inferredType.getCILName()} {self.main}::{target.name}") + elif target.varInstance.isNonlocal: + raise Exception("unimplemented") + else: + self.store(target.name, target.inferredType) + def AssignStmt(self, node: AssignStmt): - raise Exception("unimplemented") + self.visit(node.value) + targets = node.targets[::-1] + if len(targets) > 1: + self.instr("dup") + for t in targets: + self.processAssignmentTarget(t) + else: + self.processAssignmentTarget(targets[0]) def IfStmt(self, node: IfStmt): - raise Exception("unimplemented") + if len(node.elseBody) == 0: + startLabel = self.newLabelName() + endLabel = self.newLabelName() + self.label(startLabel) + self.visit(node.condition) + self.instr(f"brfalse {endLabel}") + self.visitStmtList(node.thenBody) + self.label(endLabel) + self.instr("nop") + else: + startLabel = self.newLabelName() + elseLabel = self.newLabelName() + endLabel = self.newLabelName() + self.label(startLabel) + self.visit(node.condition) + self.instr(f"brfalse {elseLabel}") + self.visitStmtList(node.thenBody) + self.instr(f"br {endLabel}") + self.label(elseLabel) + self.visitStmtList(node.elseBody) + self.label(endLabel) + self.instr("nop") def ExprStmt(self, node: ExprStmt): self.visit(node.expr) - # TODO + if isinstance(node.expr, CallExpr) or isinstance(node.expr, MethodCallExpr): + self.instr("pop") + + def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: + return leftType.isListType() and rightType.isListType() and operator == "+" def BinaryExpr(self, node: BinaryExpr): - raise Exception("unimplemented") + operator = node.operator + leftType = node.left.inferredType + rightType = node.right.inferredType + if not self.isListConcat(operator, leftType, rightType): + self.visit(node.left) + self.visit(node.right) + if operator == "+": + if self.isListConcat(operator, leftType, rightType): + # var z = new int[x.Length + y.Length]; + # x.CopyTo(z, 0); + # y.CopyTo(z, x.Length); + # IL_000f: ldloc.0 + # IL_0010: ldlen + # IL_0011: conv.i4 + # IL_0012: ldloc.1 + # IL_0013: ldlen + # IL_0014: conv.i4 + # IL_0015: add + # IL_0016: newarr [mscorlib]System.Int32 + # IL_001b: stloc.2 + # IL_001c: ldloc.0 + # IL_001d: ldloc.2 + # IL_001e: ldc.i4.0 + # IL_001f: callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32) + # IL_0024: nop + # IL_0025: ldloc.1 + # IL_0026: ldloc.2 + # IL_0027: ldloc.0 + # IL_0028: ldlen + # IL_0029: conv.i4 + # IL_002a: callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32) + # TODO + pass + elif leftType == StrType(): + self.instr("call string [mscorlib]System.String::Concat(string, string)") + elif leftType == IntType(): + self.instr("add.ovf") + else: + raise Exception( + "Internal compiler error: unexpected operand types for +") + # other arithmetic operators + elif operator == "-": + self.instr("sub.ovf") + elif operator == "*": + self.instr("mul.ovf") + elif operator == "//": + self.instr("div") + elif operator == "%": + self.instr("rem") + # relational operators + elif operator == "<": + self.instr("clt") + elif operator == "<=": + self.instr("cgt") + self.instr("ldc.i4.0") + self.instr("ceq") + elif operator == ">": + self.instr("cgt") + elif operator == ">=": + self.instr("clt") + self.instr("ldc.i4.0") + self.instr("ceq") + elif operator == "==": + if leftType == StrType(): + self.instr("call instance bool [mscorlib]System.String::Equals(string)") + else: + self.instr("ceq") + elif operator == "!=": + self.instr("ceq") + self.instr("ldc.i4.0") + self.instr("ceq") + elif operator == "is": + self.instr("ceq") + # logical operators + elif operator == "and": + self.instr("and") + elif operator == "or": + self.instr("or") + else: + raise Exception( + f"Internal compiler error: unexpected operator {operator}") def IndexExpr(self, node: IndexExpr): - raise Exception("unimplemented") + self.visit(node.list) + self.visit(node.index) + self.instr("conv.i4") + if node.list.inferredType.isListType(): + raise Exception("unimplemented") + else: + self.instr("call instance char [mscorlib]System.String::get_Chars(int32)") + self.instr("ldc.i4.1") + self.instr("newobj instance void [mscorlib]System.String::.ctor(char, int32)") def UnaryExpr(self, node: UnaryExpr): - raise Exception("unimplemented") + self.visit(node.operand) + if node.operator == "-": + self.instr("neg") + elif node.operator == "not": + self.instr("ldc.i4.0") + self.instr("ceq") def CallExpr(self, node: CallExpr): - # TODO - for i in range(len(node.args)): - self.visitArg(node.function.inferredType, i, node.args[i]) - self.instr("call void class [mscorlib]System.Console::WriteLine(string)") - # raise Exception("unimplemented") + name = node.function.name + if node.isConstructor: + raise Exception("unimplemented") + if name == "print": + self.emit_print(node.args[0]) + elif name == "len": + self.emit_len(node.args[0]) + elif name == "input": + self.emit_input() + elif name == "__assert__": + self.emit_assert(node.args[0]) + else: + for i in range(len(node.args)): + self.visitArg(node.function.inferredType, i, node.args[i]) + raise Exception("unimplemented") def ForStmt(self, node: ForStmt): raise Exception("unimplemented") @@ -150,17 +317,39 @@ def ListExpr(self, node: ListExpr): def WhileStmt(self, node: WhileStmt): raise Exception("unimplemented") + def buildReturn(self, value: Expr): + if not self.returnType.isNone(): + if value is None: + self.NoneLiteral(None) + else: + self.visit(value) + self.instr("ret") + def ReturnStmt(self, node: ReturnStmt): - raise Exception("unimplemented") + self.buildReturn(node.value) def Identifier(self, node: Identifier): - raise Exception("unimplemented") + if self.defaultToGlobals or node.varInstance.isGlobal: + self.instr(f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.name}") + elif node.varInstance.isNonlocal: + raise Exception("unimplemented") + else: + self.load(node.name, node.inferredType) def MemberExpr(self, node: MemberExpr): raise Exception("unimplemented") def IfExpr(self, node: IfExpr): - raise Exception("unimplemented") + self.visit(node.condition) + l1 = self.newLabelName() + l2 = self.newLabelName() + self.instr(f"brtrue {l1}") + self.visit(node.elseExpr) + self.instr(f"br {l2}") + self.label(l1) + self.visit(node.thenExpr) + self.label(l2) + self.instr("nop") def MethodCallExpr(self, node: MethodCallExpr): raise Exception("unimplemented") @@ -206,19 +395,49 @@ def GlobalDecl(self, node: GlobalDecl): # BUILT-INS - note: these are in-lined def emit_assert(self, arg: Expr): - raise Exception("unimplemented") + label = self.newLabelName() + self.visit(arg) + self.instr(f"brtrue {label}") + msg = f"failed assertion on line {arg.location[0]}" + self.emit_exn(msg) + self.label(label) + self.NoneLiteral(None) def emit_exn(self, msg: str): - raise Exception("unimplemented") + self.instr(f'ldstr "{msg}"') + self.instr("newobj instance void [mscorlib]System.Exception::.ctor(string)") + self.instr("throw") def emit_input(self): - raise Exception("unimplemented") + self.instr("call string [System.Console]System.Console::ReadLine()") def emit_len(self, arg: Expr): - raise Exception("unimplemented") + t = arg.inferredType + is_list = False + if t.isListType(): + is_list = True + else: + if t == NoneType(): + is_list = True + elif t == EmptyType(): + is_list = True + elif t == StrType(): + is_list = False + else: + self.emit_exn( + f"Built-in function len is unsupported for values of type {arg.inferredType.classname}") + self.visit(arg) + if is_list: + self.instr("ldlen") + self.instr("conv.i8") + else: + self.instr("callvirt instance int32 [mscorlib]System.String::get_Length()") + self.instr("conv.i8") def emit_print(self, arg: Expr): - raise Exception("unimplemented") + self.visit(arg) + self.instr("call void class [mscorlib]System.Console::WriteLine(string)") + self.NoneLiteral(None) def visitArg(self, funcType, paramIdx: int, arg: Expr): self.visit(arg) diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index d1b0b83..fcdd3ef 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -64,6 +64,22 @@ def getJavaName(self, isList = False): return self.className else: return self.className + + def getCILName(self, isList = False): + if self.className == "bool": + return "bool" + elif self.className == "str": + return "string" + elif self.className == "object": + return "object" + elif self.className == "": + return "object" + elif self.className == "": + return "object[]" + elif self.className == "int": + return "int64" + else: + return self.className def __str__(self): return self.className diff --git a/compiler/types/functype.py b/compiler/types/functype.py index e53f507..dc2db25 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -34,7 +34,6 @@ def getJavaSignature(self)->str: else: sig = p.getJavaSignature() params.append(sig) - return "({}){}".format("".join(params), r) def methodEquals(self, other): diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 92c2552..3df402a 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -16,6 +16,9 @@ def getJavaSignature(self, _=False): def getJavaName(self, _=False): return "["+self.elementType.getJavaSignature(True) + def getCILName(self, _=False): + return self.elementType.getCILName(True) + "[]" + def isListType(self): return True From 0e199d2f53059bb201e536543b9f8e0d6c47b54a Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 16 May 2022 16:11:27 -0400 Subject: [PATCH 09/79] reenable exponent test --- tests/runtime/exponent.py | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/tests/runtime/exponent.py b/tests/runtime/exponent.py index aced79d..d8dc752 100644 --- a/tests/runtime/exponent.py +++ b/tests/runtime/exponent.py @@ -14,19 +14,19 @@ def geta() -> int: return f(y) # Input parameter -# n:int = 42 +n:int = 42 -# # Run [0, n] -# i:int = 0 +# Run [0, n] +i:int = 0 -# # Crunch -# while i <= n: -# print(exp(2, i % 31)) -# i = i + 1 +# Crunch +while i <= n: + print(exp(2, i % 31)) + i = i + 1 -# __assert__(exp(2,3) == 8) -# __assert__(exp(3,3) == 27) -# __assert__(exp(3,4) == 81) -# __assert__(exp(4,4) == 256) -# __assert__(exp(5,1) == 5) -# __assert__(exp(1,99) == 1) \ No newline at end of file +__assert__(exp(2,3) == 8) +__assert__(exp(3,3) == 27) +__assert__(exp(3,4) == 81) +__assert__(exp(4,4) == 256) +__assert__(exp(5,1) == 5) +__assert__(exp(1,99) == 1) \ No newline at end of file From 4eb2a7e61ae5751e0fade1735f927a3d766c5257 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 17 May 2022 00:31:54 -0400 Subject: [PATCH 10/79] functions, lists, looping --- compiler/astnodes/identifier.py | 6 + compiler/builder.py | 20 ++- compiler/cil_backend.py | 280 ++++++++++++++++++++++++------- compiler/jvm_backend.py | 7 +- compiler/types/classvaluetype.py | 8 +- compiler/types/functype.py | 4 + compiler/types/listvaluetype.py | 5 +- demo_cil.sh | 1 + demo_jvm.sh | 1 + tests/runtime/control_flow.py | 3 + tests/runtime/functions.py | 5 +- 11 files changed, 267 insertions(+), 73 deletions(-) diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index 0ad49d7..bf0abab 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -21,3 +21,9 @@ def copy(self): cpy.varInstance = self.varInstance return cpy + def getCILName(self): + banned = ["char"] + if self.name in banned: + return "__local__" + self.name + return self.name + diff --git a/compiler/builder.py b/compiler/builder.py index 41edd75..be911c9 100644 --- a/compiler/builder.py +++ b/compiler/builder.py @@ -1,13 +1,23 @@ +from unicodedata import name + + class Builder: def __init__(self, name:str): self.name = name - self.lines = [] + self.lines = [] # list of strings or children builders self.indentation = 0 def newLine(self, line=""): self.lines.append((self.indentation*" ") + line) return self + # returns a reference to the child builder + def newBlock(self): + child = Builder(self.name) + child.indentation = self.indentation + self.lines.append(child) + return child + def addText(self, text=""): if len(self.lines) == 0: self.newLine() @@ -23,4 +33,10 @@ def unindent(self): return self def emit(self)->str: - return "\n".join(self.lines) \ No newline at end of file + lines = [] + for l in self.lines: + if isinstance(l, str): + lines.append(l) + else: + lines.append(l.emit()) + return "\n".join(lines) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 2ca10c1..14e0f49 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -1,3 +1,4 @@ +from cmath import log from .astnodes import * from .types import * from .builder import Builder @@ -6,6 +7,15 @@ from collections import defaultdict import json +class CilStackLoc: + def __init__(self, name, loc, t, isArg): + self.name = name + self.loc = loc + self.isArg = isArg + self.t = t + + def decl(self): + return f"[{self.loc}] {self.t} {self.name}" class CilBackend(Visitor): @@ -57,40 +67,49 @@ def exitScope(self): def wrap(self, val:Expr, elementType:ValueType): raise Exception("unimplemented") - def store(self, name: str, t: ValueType): + def store(self, name: str): n = self.locals[-1][name] if n is None: raise Exception( f"Internal compiler error: unknown name {name} for store") - self.instr(f"stloc {n}") + if n.isArg: + self.instr(f"starg {n.loc}") + else: + self.instr(f"stloc {n.loc}") - def load(self, name: str, t: ValueType): + def load(self, name: str): n = self.locals[-1][name] if n is None: raise Exception( f"Internal compiler error: unknown name {name} for load") - self.instr(f"ldloc {n}") + if n.isArg: + self.instr(f"ldarg {n.loc}") + else: + self.instr(f"ldloc {n.loc}") def arrayStore(self, elementType:ValueType): - raise Exception("unimplemented") + self.instr(f"stelem {elementType.getCILName()}") def arrayLoad(self, elementType:ValueType): - raise Exception("unimplemented") + self.instr(f"ldelem {elementType.getCILName()}") - def newLocalEntry(self, name: str) -> int: - raise Exception("unimplemented") + def newLocalEntry(self, name: str, t: ValueType, isArg: bool = False) -> int: + # add a new entry to locals table w/o storing anything + n = len([k for k in self.locals[-1] if self.locals[-1][k].isArg == isArg]) + self.locals[-1][name] = CilStackLoc(name, n, t.getCILName(), isArg) + return n def genLocalName(self, offset: int) -> str: return f"__local__{offset}" - def newLocal(self, name: str = None) -> int: + def newLocal(self, name: str, t: ValueType): # store the top of stack as a new local - n = len(self.locals[-1]) + n = len([k for k in self.locals[-1] if not self.locals[-1][k].isArg]) self.instr(f"stloc {n}") if name is None: name = self.genLocalName(n) - self.locals[-1][name] = n - return n + self.locals[-1][name] = CilStackLoc(name, n, t.getCILName(), False) + return name def visitStmtList(self, stmts:[Stmt]): if len(stmts) == 0: @@ -100,7 +119,10 @@ def visitStmtList(self, stmts:[Stmt]): self.visit(s) def Program(self, node: Program): - # TODO + func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] + cls_decls = [d for d in node.declarations if isinstance(d, ClassDef)] + var_decls = [d for d in node.declarations if isinstance(d, VarDef)] + self.instr(f".assembly '{self.main}'") self.instr("{") self.instr("}") @@ -108,32 +130,82 @@ def Program(self, node: Program): self.instr(f".class public auto ansi beforefieldinit {self.main} extends [mscorlib]System.Object") self.indent() - var_decls = [d for d in node.declarations if isinstance(d, VarDef)] + # global vars (static members) for v in var_decls: - self.instr(f".field public static {v.var.t.getCILName()} {v.var.identifier.name}") + self.instr(f".field public static {v.var.t.getCILName()} {v.var.identifier.getCILName()}") + # main method, top level statements self.instr(".method public static hidebysig default void Main (string[] args) cil managed") self.indent() self.instr(".entrypoint") self.instr(f".maxstack {self.localLimit}") + locals = self.currentBuilder().newBlock() self.defaultToGlobals = True for v in var_decls: self.visit(v.value) - self.instr(f"stsfld {v.var.t.getCILName()} {self.main}::{v.var.identifier.name}") + self.instr(f"stsfld {v.var.t.getCILName()} {self.main}::{v.var.identifier.getCILName()}") self.visitStmtList(node.statements) self.defaultToGlobals = False + self.generateLocalsDirective(locals) self.instr("ret") self.unindent() + + # global functions (static funcs) + for d in func_decls: + self.visit(d) self.unindent() def ClassDef(self, node: ClassDef): raise Exception("unimplemented") + def generateLocalsDirective(self, locals): + # defer local declarations until we know what we need + locals.newLine(".locals init (").indent() + mapping = self.locals[-1] + localDecls = [mapping[k] for k in mapping if not mapping[k].isArg] + sortedDecls = sorted(localDecls, key=lambda x: x.loc) + for i in range(len(sortedDecls)): + comma = "," if i < len(sortedDecls) - 1 else "" + locals.newLine(sortedDecls[i].decl() + comma) + locals.unindent().newLine(")") + def FuncDef(self, node: FuncDef): - raise Exception("unimplemented") + self.instr(".method public hidebysig static") + self.instr(f"{node.type.getCILSignature(node.name.getCILName())} cil managed") + self.indent() + self.instr(f".maxstack {self.localLimit}") + self.enterScope() + + # initialize locals + locals = self.currentBuilder().newBlock() + + for i in range(len(node.params)): + self.newLocalEntry(node.params[i].identifier.getCILName(), node.type.parameters[i], True) + for d in node.declarations: + self.visit(d) + self.returnType = node.type.returnType + + # handle last return + self.visitStmtList(node.statements) + hasReturn = False + for s in node.statements: + if s.isReturn: + hasReturn = True + if not hasReturn: + self.buildReturn(None) + self.generateLocalsDirective(locals) + self.exitScope() + self.unindent() def VarDef(self, node: VarDef): - raise Exception("unimplemented") + varName = node.var.identifier.getCILName() + if node.isAttr: + raise Exception("unimplemented") + elif node.var.varInstance.isNonlocal: + raise Exception("unimplemented") + else: + self.visit(node.value) + self.newLocal(varName, node.var.t) # STATEMENTS @@ -141,11 +213,22 @@ def processAssignmentTarget(self, target: Expr): if isinstance(target, Identifier): if self.defaultToGlobals or target.varInstance.isGlobal: self.instr( - f"stsfld {target.inferredType.getCILName()} {self.main}::{target.name}") + f"stsfld {target.inferredType.getCILName()} {self.main}::{target.getCILName()}") elif target.varInstance.isNonlocal: raise Exception("unimplemented") else: - self.store(target.name, target.inferredType) + self.store(target.getCILName()) + elif isinstance(target, IndexExpr): + temp = self.newLocal(None, target.inferredType) + self.visit(target.list) + self.visit(target.index) + self.load(temp) + self.arrayStore(target.inferredType) + elif isinstance(target, MemberExpr): + raise Exception("unimplemented") + else: + raise Exception( + "Internal compiler error: unsupported assignment target") def AssignStmt(self, node: AssignStmt): self.visit(node.value) @@ -166,7 +249,6 @@ def IfStmt(self, node: IfStmt): self.instr(f"brfalse {endLabel}") self.visitStmtList(node.thenBody) self.label(endLabel) - self.instr("nop") else: startLabel = self.newLabelName() elseLabel = self.newLabelName() @@ -179,7 +261,6 @@ def IfStmt(self, node: IfStmt): self.label(elseLabel) self.visitStmtList(node.elseBody) self.label(endLabel) - self.instr("nop") def ExprStmt(self, node: ExprStmt): self.visit(node.expr) @@ -193,36 +274,39 @@ def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType rightType = node.right.inferredType - if not self.isListConcat(operator, leftType, rightType): - self.visit(node.left) - self.visit(node.right) + self.visit(node.left) + self.visit(node.right) if operator == "+": if self.isListConcat(operator, leftType, rightType): - # var z = new int[x.Length + y.Length]; - # x.CopyTo(z, 0); - # y.CopyTo(z, x.Length); - # IL_000f: ldloc.0 - # IL_0010: ldlen - # IL_0011: conv.i4 - # IL_0012: ldloc.1 - # IL_0013: ldlen - # IL_0014: conv.i4 - # IL_0015: add - # IL_0016: newarr [mscorlib]System.Int32 - # IL_001b: stloc.2 - # IL_001c: ldloc.0 - # IL_001d: ldloc.2 - # IL_001e: ldc.i4.0 - # IL_001f: callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32) - # IL_0024: nop - # IL_0025: ldloc.1 - # IL_0026: ldloc.2 - # IL_0027: ldloc.0 - # IL_0028: ldlen - # IL_0029: conv.i4 - # IL_002a: callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32) - # TODO - pass + """ + var x = new int[]; + var y = new int[]; + var z = new int[x.Length + y.Length]; + x.CopyTo(z, 0); + y.CopyTo(z, x.Length); + """ + r = self.newLocal(None, rightType) + l = self.newLocal(None, leftType) + self.load(l) + self.instr("ldlen") + self.load(r) + self.instr("ldlen") + self.instr("add") + self.instr("conv.i4") + merged_t = self.ts.join(leftType, rightType).elementType + self.instr(f"newarr {merged_t.getCILName()}") + merged = self.newLocal(None, ListValueType(merged_t)) + self.load(l) + self.load(merged) + self.instr("ldc.i4 0") + self.instr("callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32)") + self.load(r) + self.load(merged) + self.load(l) + self.instr("ldlen") + self.instr("conv.i4") + self.instr("callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32)") + self.load(merged) elif leftType == StrType(): self.instr("call string [mscorlib]System.String::Concat(string, string)") elif leftType == IntType(): @@ -277,7 +361,7 @@ def IndexExpr(self, node: IndexExpr): self.visit(node.index) self.instr("conv.i4") if node.list.inferredType.isListType(): - raise Exception("unimplemented") + self.arrayLoad(node.list.inferredType.elementType) else: self.instr("call instance char [mscorlib]System.String::get_Chars(int32)") self.instr("ldc.i4.1") @@ -292,10 +376,10 @@ def UnaryExpr(self, node: UnaryExpr): self.instr("ceq") def CallExpr(self, node: CallExpr): - name = node.function.name + name = node.function.getCILName() if node.isConstructor: - raise Exception("unimplemented") - if name == "print": + self.instr(f"newobj instance void [mscorlib]System.Object::.ctor()") + elif name == "print": self.emit_print(node.args[0]) elif name == "len": self.emit_len(node.args[0]) @@ -306,16 +390,87 @@ def CallExpr(self, node: CallExpr): else: for i in range(len(node.args)): self.visitArg(node.function.inferredType, i, node.args[i]) - raise Exception("unimplemented") + signature = node.function.inferredType.getCILSignature(f"{self.main}::{name}") + self.instr(f"call {signature}") + if node.function.inferredType.returnType.isNone(): + self.NoneLiteral(None) # push null for void return def ForStmt(self, node: ForStmt): - raise Exception("unimplemented") + # itr = {expr}, idx = 0 + self.visit(node.iterable) + itr = self.newLocal(None, node.iterable.inferredType) + self.instr("ldc.i8 0") + idx = self.newLocal(None, IntType()) + startLabel = self.newLabelName() + endLabel = self.newLabelName() + self.label(startLabel) + # while idx < len(itr) + self.load(idx) + self.instr("conv.i4") + self.load(itr) + if node.iterable.inferredType.isListType(): + self.instr("ldlen") + else: + self.instr("callvirt instance int32 [mscorlib]System.String::get_Length()") + self.instr("conv.i4") + self.instr("clt") + self.instr(f"brfalse {endLabel}") + # x = itr[idx] + self.load(itr) + self.load(idx) + self.instr("conv.i4") + if node.iterable.inferredType.isListType(): + self.arrayLoad(node.iterable.inferredType.elementType) + else: + self.instr("call instance char [mscorlib]System.String::get_Chars(int32)") + self.instr("ldc.i4.1") + self.instr("newobj instance void [mscorlib]System.String::.ctor(char, int32)") + if self.defaultToGlobals or node.identifier.varInstance.isGlobal: + self.instr( + f"stsfld {node.identifier.inferredType.getCILName()} {self.main}::{node.identifier.getCILName()}") + else: + self.store(node.identifier.getCILName()) + # body + self.visitStmtList(node.body) + # idx = idx + 1 + self.load(idx) + self.instr("ldc.i8 1") + self.instr("add") + self.store(idx) + self.instr(f"br {startLabel}") + self.label(endLabel) + + + def ListExpr(self, node: ListExpr): - raise Exception("unimplemented") + t = node.inferredType + length = len(node.elements) + self.instr(f"ldc.i4 {length}") + elementType = None + if isinstance(t, ClassValueType): + if node.emptyListType: + elementType = node.emptyListType + else: + elementType = ClassValueType("object") + else: + elementType = t.elementType + self.instr(f"newarr {elementType.getCILName()}") + for i in range(len(node.elements)): + self.instr("dup") + self.instr(f"ldc.i4 {i}") + self.visit(node.elements[i]) + self.arrayStore(elementType) def WhileStmt(self, node: WhileStmt): - raise Exception("unimplemented") + startLabel = self.newLabelName() + endLabel = self.newLabelName() + self.label(startLabel) + self.visit(node.condition) + self.instr(f"brfalse {endLabel}") + self.visitStmtList(node.body) + self.instr(f"br {startLabel}") + self.label(endLabel) def buildReturn(self, value: Expr): if not self.returnType.isNone(): @@ -330,11 +485,11 @@ def ReturnStmt(self, node: ReturnStmt): def Identifier(self, node: Identifier): if self.defaultToGlobals or node.varInstance.isGlobal: - self.instr(f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.name}") + self.instr(f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") elif node.varInstance.isNonlocal: raise Exception("unimplemented") else: - self.load(node.name, node.inferredType) + self.load(node.getCILName()) def MemberExpr(self, node: MemberExpr): raise Exception("unimplemented") @@ -349,7 +504,6 @@ def IfExpr(self, node: IfExpr): self.label(l1) self.visit(node.thenExpr) self.label(l2) - self.instr("nop") def MethodCallExpr(self, node: MethodCallExpr): raise Exception("unimplemented") @@ -409,7 +563,7 @@ def emit_exn(self, msg: str): self.instr("throw") def emit_input(self): - self.instr("call string [System.Console]System.Console::ReadLine()") + self.instr("call string [mscorlib]System.Console::ReadLine()") def emit_len(self, arg: Expr): t = arg.inferredType @@ -436,7 +590,7 @@ def emit_len(self, arg: Expr): def emit_print(self, arg: Expr): self.visit(arg) - self.instr("call void class [mscorlib]System.Console::WriteLine(string)") + self.instr(f"call void class [mscorlib]System.Console::WriteLine({arg.inferredType.getCILName()})") self.NoneLiteral(None) def visitArg(self, funcType, paramIdx: int, arg: Expr): diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 40016c1..5be301f 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -375,14 +375,11 @@ def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType rightType = node.right.inferredType - if not self.isListConcat(operator, leftType, rightType): - self.visit(node.left) - self.visit(node.right) + self.visit(node.left) + self.visit(node.right) # concatenation and addition if operator == "+": if self.isListConcat(operator, leftType, rightType): - self.visit(node.left) - self.visit(node.right) self.instr("dup2") arrR = self.newLocal(None, True) # stack is L, R, L diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index fcdd3ef..ebbc9a1 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -65,7 +65,13 @@ def getJavaName(self, isList = False): else: return self.className - def getCILName(self, isList = False): + def getCILSignature(self): + if self.className == "": + return "void" + else: + return self.getCILName() + + def getCILName(self): if self.className == "bool": return "bool" elif self.className == "str": diff --git a/compiler/types/functype.py b/compiler/types/functype.py index dc2db25..0c761e5 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -20,6 +20,10 @@ def dropFirstParam(self): f.freevars = self.freevars return f + def getCILSignature(self, name: str)->str: + paramSig = ", ".join([t.getCILSignature() for t in self.parameters]) + return f"{self.returnType.getCILSignature()} {name}({paramSig})" + def getJavaSignature(self)->str: r = None if self.returnType.isNone(): diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 3df402a..6c77b57 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -17,7 +17,10 @@ def getJavaName(self, _=False): return "["+self.elementType.getJavaSignature(True) def getCILName(self, _=False): - return self.elementType.getCILName(True) + "[]" + return self.elementType.getCILName() + "[]" + + def getCILSignature(self, _=False): + return self.getCILName() def isListType(self): return True diff --git a/demo_cil.sh b/demo_cil.sh index 235780b..f8b93ce 100755 --- a/demo_cil.sh +++ b/demo_cil.sh @@ -5,4 +5,5 @@ rm -f *.cil rm -f *.exe python3 main.py --mode cil $1 . ls *.cil | xargs -L1 ilasm +echo "Running program $base_name..." mono $base_name.exe \ No newline at end of file diff --git a/demo_jvm.sh b/demo_jvm.sh index abe6d6f..9d1e3cc 100755 --- a/demo_jvm.sh +++ b/demo_jvm.sh @@ -5,4 +5,5 @@ rm -f *.j rm -f *.class python3 main.py --mode jvm $1 . ls *.j | xargs -L1 python3 ../Krakatau/assemble.py -q +echo "Running program $base_name..." java -cp . $base_name \ No newline at end of file diff --git a/tests/runtime/control_flow.py b/tests/runtime/control_flow.py index 095ce08..7e8203e 100644 --- a/tests/runtime/control_flow.py +++ b/tests/runtime/control_flow.py @@ -81,6 +81,9 @@ b = b - 1 __assert__(b == 0) +for char in y: + pass + for char in y: z = char + z __assert__(z == "321") diff --git a/tests/runtime/functions.py b/tests/runtime/functions.py index bc6ef37..e61ffed 100644 --- a/tests/runtime/functions.py +++ b/tests/runtime/functions.py @@ -42,4 +42,7 @@ def f8(x:int)->int: __assert__(f7() == 5) __assert__(f8(0) == 0) __assert__(f8(1) == 1) -__assert__(f8(f7()) == 5) \ No newline at end of file +__assert__(f8(f7()) == 5) + +print(1) +print(True) \ No newline at end of file From c38023fccfed4ced7b9e3e73aeb37c805bbfe277 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 17 May 2022 15:51:29 -0400 Subject: [PATCH 11/79] add keywords to escape for identifiers, 17/20 passing --- compiler/astnodes/classdef.py | 28 ++++- compiler/astnodes/identifier.py | 126 +++++++++++++++++++- compiler/astnodes/vardef.py | 6 +- compiler/cil_backend.py | 192 ++++++++++++++++++++++--------- compiler/compiler.py | 2 +- compiler/jvm_backend.py | 16 +-- compiler/types/classvaluetype.py | 2 +- main.py | 18 ++- test.py | 64 +++++++++++ 9 files changed, 364 insertions(+), 90 deletions(-) diff --git a/compiler/astnodes/classdef.py b/compiler/astnodes/classdef.py index 16f666f..d29220e 100644 --- a/compiler/astnodes/classdef.py +++ b/compiler/astnodes/classdef.py @@ -2,16 +2,22 @@ from .identifier import Identifier from .vardef import VarDef from .funcdef import FuncDef +from .typedvar import TypedVar +from .classtype import ClassType +from ..types.classvaluetype import ClassValueType +from ..types.functype import FuncType +from ..types.Types import NoneType class ClassDef(Declaration): - def __init__(self, location:[int], name:Identifier, superclass:Identifier, declarations:[Declaration]): + def __init__(self, location: [int], name: Identifier, superclass: Identifier, declarations: [Declaration]): super().__init__(location, "ClassDef") self.name = name self.superclass = superclass for d in declarations: if isinstance(d, VarDef): d.isAttr = True + d.attrOfClass = name.name if isinstance(d, FuncDef): d.isMethod = True self.declarations = declarations @@ -34,10 +40,26 @@ def toJSON(self, dump_location=True): d = super().toJSON(dump_location) d["name"] = self.name.toJSON(dump_location) d["superClass"] = self.superclass.toJSON(dump_location) - d["declarations"] = [decl.toJSON(dump_location) for decl in self.declarations] + d["declarations"] = [decl.toJSON(dump_location) + for decl in self.declarations] return d def getIdentifier(self): return self.name - + def getDefaultConstructor(self)->FuncDef: + var_decls = [d for d in self.declarations if isinstance(d, VarDef)] + constructor = FuncDef(self.location, + Identifier(self.location, "__init__"), + [TypedVar(self.location, + Identifier(self.location, "self"), + ClassType(self.location, self.name.name) + )], + ClassType(self.location, ""), + var_decls, + [], True + ) + constructor.params[0].t = ClassValueType(self.name.name) + constructor.type = FuncType( + [ClassValueType(self.name.name)], NoneType()) + return constructor diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index bf0abab..9c2a635 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -1,8 +1,126 @@ from .expr import Expr +CIL_KEYWORDS = set(["char", "value"] + + ["add", + "and", + "any", + "arglist", + "as", + "be", + "beq", + "bge", + "bgt", + "ble", + "blt", + "bne", + "box", + "br", + "break", + "brfalse", + "brinst", + "brnull", + "brtrue", + "brzero", + "call", + "calli", + "callvirt", + "can", + "castclass", + "ceq", + "cgt", + "check", + "ckfinite", + "clt", + "constrained", + "conv", + "cpblk", + "cpobj", + "div", + "dup", + "endfault", + "endfilter", + "endfinally", + "execution", + "fault", + "initblk", + "initobj", + "instruction", + "isinst", + "jmp", + "ldarg", + "ldarga", + "ldc", + "ldelem", + "ldelema", + "ldfld", + "ldflda", + "ldftn", + "ldind", + "ldlen", + "ldloc", + "ldloca", + "ldnull", + "ldobj", + "ldsfld", + "ldsflda", + "ldstr", + "ldtoken", + "ldvirtftn", + "leave", + "localloc", + "mkrefany", + "mul", + "neg", + "newarr", + "newobj", + "no", + "nop", + "normally", + "not", + "nullcheck", + "of", + "or", + "ovf", + "part", + "performed", + "pop", + "rangecheck", + "readonly", + "ref", + "refanytype", + "refanyval", + "rem", + "ret", + "rethrow", + "shall", + "shl", + "shr", + "sizeof", + "skipped", + "specified", + "starg", + "stelem", + "stfld", + "stind", + "stloc", + "stobj", + "stsfld", + "sub", + "subsequent", + "switch", + "tail", + "throw", + "typecheck", + "un", + "unaligned", + "unbox", + "volatile", + "xor"] + ) + class Identifier(Expr): - def __init__(self, location:[int], name:str): + def __init__(self, location: [int], name: str): super().__init__(location, "Identifier") self.name = name self.varInstance = None @@ -22,8 +140,6 @@ def copy(self): return cpy def getCILName(self): - banned = ["char"] - if self.name in banned: - return "__local__" + self.name + if self.name in CIL_KEYWORDS: + return f"'{self.name}'" return self.name - diff --git a/compiler/astnodes/vardef.py b/compiler/astnodes/vardef.py index 57abe5c..7372b95 100644 --- a/compiler/astnodes/vardef.py +++ b/compiler/astnodes/vardef.py @@ -4,11 +4,12 @@ class VarDef(Declaration): - def __init__(self, location:[int], var:TypedVar, value:Expr, isAttr:bool=False): + def __init__(self, location:[int], var:TypedVar, value:Expr, isAttr:bool=False, attrOfClass=None): super().__init__(location, "VarDef") self.var = var self.value = value self.isAttr = isAttr + self.attrOfClass = attrOfClass def preorder(self, visitor): visitor.VarDef(self) @@ -30,3 +31,6 @@ def toJSON(self, dump_location=True): def getIdentifier(self): return self.var.identifier + + def getName(self)->str: + return self.var.identifier.name diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 14e0f49..4ea775e 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -7,6 +7,7 @@ from collections import defaultdict import json + class CilStackLoc: def __init__(self, name, loc, t, isArg): self.name = name @@ -17,12 +18,11 @@ def __init__(self, name, loc, t, isArg): def decl(self): return f"[{self.loc}] {self.t} {self.name}" + class CilBackend(Visitor): def __init__(self, main: str, ts: TypeSystem): - self.classes = dict() - self.classes[main] = Builder(main) - self.currentClass = main + self.builder = Builder(main) self.main = main # name of main class self.locals = [defaultdict(lambda: None)] self.counter = 0 # for labels @@ -34,29 +34,26 @@ def __init__(self, main: str, ts: TypeSystem): def indent(self): self.instr("{") - self.currentBuilder().indent() + self.builder.indent() def unindent(self): - self.currentBuilder().unindent() + self.builder.unindent() self.instr("}") - def currentBuilder(self): - return self.classes[self.currentClass] - def visit(self, node: Node): node.visit(self) def instr(self, instr: str): - self.currentBuilder().newLine(instr) + self.builder.newLine(instr) def newLabelName(self) -> str: self.counter += 1 return "IL_"+str(self.counter) def label(self, name: str) -> str: - self.currentBuilder().unindent() + self.builder.unindent() self.instr(name+": nop") - self.currentBuilder().indent() + self.builder.indent() def enterScope(self): self.locals.append(defaultdict(lambda: None)) @@ -64,7 +61,7 @@ def enterScope(self): def exitScope(self): self.locals.pop() - def wrap(self, val:Expr, elementType:ValueType): + def wrap(self, val: Expr, elementType: ValueType): raise Exception("unimplemented") def store(self, name: str): @@ -87,10 +84,10 @@ def load(self, name: str): else: self.instr(f"ldloc {n.loc}") - def arrayStore(self, elementType:ValueType): + def arrayStore(self, elementType: ValueType): self.instr(f"stelem {elementType.getCILName()}") - def arrayLoad(self, elementType:ValueType): + def arrayLoad(self, elementType: ValueType): self.instr(f"ldelem {elementType.getCILName()}") def newLocalEntry(self, name: str, t: ValueType, isArg: bool = False) -> int: @@ -111,7 +108,7 @@ def newLocal(self, name: str, t: ValueType): self.locals[-1][name] = CilStackLoc(name, n, t.getCILName(), False) return name - def visitStmtList(self, stmts:[Stmt]): + def visitStmtList(self, stmts: [Stmt]): if len(stmts) == 0: self.instr("nop") else: @@ -127,36 +124,84 @@ def Program(self, node: Program): self.instr("{") self.instr("}") self.instr(f".module {self.main}.exe") - self.instr(f".class public auto ansi beforefieldinit {self.main} extends [mscorlib]System.Object") + self.instr( + f".class public auto ansi beforefieldinit {self.main} extends [mscorlib]System.Object") self.indent() # global vars (static members) for v in var_decls: - self.instr(f".field public static {v.var.t.getCILName()} {v.var.identifier.getCILName()}") + self.instr( + f".field public static {v.var.t.getCILName()} {v.getIdentifier().getCILName()}") # main method, top level statements - self.instr(".method public static hidebysig default void Main (string[] args) cil managed") + self.instr( + ".method public static hidebysig default void Main (string[] args) cil managed") self.indent() self.instr(".entrypoint") self.instr(f".maxstack {self.localLimit}") - locals = self.currentBuilder().newBlock() + locals = self.builder.newBlock() self.defaultToGlobals = True for v in var_decls: self.visit(v.value) - self.instr(f"stsfld {v.var.t.getCILName()} {self.main}::{v.var.identifier.getCILName()}") + self.instr( + f"stsfld {v.var.t.getCILName()} {self.main}::{v.getIdentifier().getCILName()}") self.visitStmtList(node.statements) self.defaultToGlobals = False self.generateLocalsDirective(locals) self.instr("ret") - self.unindent() + self.unindent() # end of main method # global functions (static funcs) for d in func_decls: self.visit(d) - self.unindent() + + self.unindent() # end of main class + + for c in cls_decls: + self.visit(c) def ClassDef(self, node: ClassDef): - raise Exception("unimplemented") + def constructor(superclass: str, func: FuncDef): + func.type = func.type.dropFirstParam() + func.name.name = ".ctor" + # add call to parent constructor after child field initialization + # before other constructor statements + call = CallExpr(func.location, Identifier(func.location, node.superclass.getCILName()), []) + call.isConstructor = True + parentCall = ExprStmt(func.location, call) + func.statements.insert(0, parentCall) + self.FuncDef(func, "specialname rtspecialname instance", True) + + func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] + var_decls = [d for d in node.declarations if isinstance(d, VarDef)] + clsName = node.name.getCILName() + superclass = ClassValueType(node.superclass.getCILName()).getCILName() + + self.instr( + f".class public auto ansi beforefieldinit {clsName} extends {superclass}") + self.indent() + + constructor_def = None + # field decls + for v in var_decls: + self.instr( + f".field public {v.var.t.getCILName()} {v.getIdentifier().getCILName()}") + for d in func_decls: + if d.name.name == "__init__": + # constructor + constructor_def = d + d.declarations = var_decls + d.declarations + constructor(superclass, d) + else: + # method + d.type = d.type.dropFirstParam() + self.FuncDef(d, "virtual instance", True) + if constructor_def == None: + # give a default constructor if none exists + funcDef = node.getDefaultConstructor() + constructor(superclass, funcDef) + + self.unindent() # end class def generateLocalsDirective(self, locals): # defer local declarations until we know what we need @@ -169,18 +214,19 @@ def generateLocalsDirective(self, locals): locals.newLine(sortedDecls[i].decl() + comma) locals.unindent().newLine(")") - def FuncDef(self, node: FuncDef): - self.instr(".method public hidebysig static") - self.instr(f"{node.type.getCILSignature(node.name.getCILName())} cil managed") + def FuncDef(self, node: FuncDef, funcType: str = "static", isMethod: bool = False): + self.instr(f".method public hidebysig {funcType}") + self.instr( + f"{node.type.getCILSignature(node.name.getCILName())} cil managed") self.indent() self.instr(f".maxstack {self.localLimit}") self.enterScope() # initialize locals - locals = self.currentBuilder().newBlock() - + locals = self.builder.newBlock() for i in range(len(node.params)): - self.newLocalEntry(node.params[i].identifier.getCILName(), node.type.parameters[i], True) + self.newLocalEntry( + node.params[i].identifier.getCILName(), node.params[i].t, True) for d in node.declarations: self.visit(d) self.returnType = node.type.returnType @@ -198,9 +244,14 @@ def FuncDef(self, node: FuncDef): self.unindent() def VarDef(self, node: VarDef): - varName = node.var.identifier.getCILName() + varName = node.getIdentifier().getCILName() if node.isAttr: - raise Exception("unimplemented") + # codegen for initialization in constructors + className = ClassValueType(node.attrOfClass) + self.instr("ldarg 0") + self.visit(node.value) + self.instr( + f"stfld {node.var.t.getCILName()} {className.getCILName()}::{node.getIdentifier().getCILName()}") elif node.var.varInstance.isNonlocal: raise Exception("unimplemented") else: @@ -225,7 +276,11 @@ def processAssignmentTarget(self, target: Expr): self.load(temp) self.arrayStore(target.inferredType) elif isinstance(target, MemberExpr): - raise Exception("unimplemented") + temp = self.newLocal(None, target.inferredType) + self.visit(target.object) + self.load(temp) + self.instr( + f"stfld {target.inferredType.getCILName()} {target.object.inferredType.getCILName()}::{target.member.getCILName()}") else: raise Exception( "Internal compiler error: unsupported assignment target") @@ -264,8 +319,7 @@ def IfStmt(self, node: IfStmt): def ExprStmt(self, node: ExprStmt): self.visit(node.expr) - if isinstance(node.expr, CallExpr) or isinstance(node.expr, MethodCallExpr): - self.instr("pop") + self.instr("pop") def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: return leftType.isListType() and rightType.isListType() and operator == "+" @@ -299,16 +353,19 @@ def BinaryExpr(self, node: BinaryExpr): self.load(l) self.load(merged) self.instr("ldc.i4 0") - self.instr("callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32)") + self.instr( + "callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32)") self.load(r) self.load(merged) self.load(l) self.instr("ldlen") self.instr("conv.i4") - self.instr("callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32)") + self.instr( + "callvirt instance void [mscorlib]System.Array::CopyTo(class [mscorlib]System.Array, int32)") self.load(merged) elif leftType == StrType(): - self.instr("call string [mscorlib]System.String::Concat(string, string)") + self.instr( + "call string [mscorlib]System.String::Concat(string, string)") elif leftType == IntType(): self.instr("add.ovf") else: @@ -338,7 +395,8 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("ceq") elif operator == "==": if leftType == StrType(): - self.instr("call instance bool [mscorlib]System.String::Equals(string)") + self.instr( + "call instance bool [mscorlib]System.String::Equals(string)") else: self.instr("ceq") elif operator == "!=": @@ -363,9 +421,11 @@ def IndexExpr(self, node: IndexExpr): if node.list.inferredType.isListType(): self.arrayLoad(node.list.inferredType.elementType) else: - self.instr("call instance char [mscorlib]System.String::get_Chars(int32)") + self.instr( + "call instance char [mscorlib]System.String::get_Chars(int32)") self.instr("ldc.i4.1") - self.instr("newobj instance void [mscorlib]System.String::.ctor(char, int32)") + self.instr( + "newobj instance void [mscorlib]System.String::.ctor(char, int32)") def UnaryExpr(self, node: UnaryExpr): self.visit(node.operand) @@ -378,7 +438,8 @@ def UnaryExpr(self, node: UnaryExpr): def CallExpr(self, node: CallExpr): name = node.function.getCILName() if node.isConstructor: - self.instr(f"newobj instance void [mscorlib]System.Object::.ctor()") + self.instr( + f"newobj instance void {name}::.ctor()") elif name == "print": self.emit_print(node.args[0]) elif name == "len": @@ -390,7 +451,8 @@ def CallExpr(self, node: CallExpr): else: for i in range(len(node.args)): self.visitArg(node.function.inferredType, i, node.args[i]) - signature = node.function.inferredType.getCILSignature(f"{self.main}::{name}") + signature = node.function.inferredType.getCILSignature( + f"{self.main}::{name}") self.instr(f"call {signature}") if node.function.inferredType.returnType.isNone(): self.NoneLiteral(None) # push null for void return @@ -411,7 +473,8 @@ def ForStmt(self, node: ForStmt): if node.iterable.inferredType.isListType(): self.instr("ldlen") else: - self.instr("callvirt instance int32 [mscorlib]System.String::get_Length()") + self.instr( + "callvirt instance int32 [mscorlib]System.String::get_Length()") self.instr("conv.i4") self.instr("clt") self.instr(f"brfalse {endLabel}") @@ -422,9 +485,11 @@ def ForStmt(self, node: ForStmt): if node.iterable.inferredType.isListType(): self.arrayLoad(node.iterable.inferredType.elementType) else: - self.instr("call instance char [mscorlib]System.String::get_Chars(int32)") + self.instr( + "call instance char [mscorlib]System.String::get_Chars(int32)") self.instr("ldc.i4.1") - self.instr("newobj instance void [mscorlib]System.String::.ctor(char, int32)") + self.instr( + "newobj instance void [mscorlib]System.String::.ctor(char, int32)") if self.defaultToGlobals or node.identifier.varInstance.isGlobal: self.instr( f"stsfld {node.identifier.inferredType.getCILName()} {self.main}::{node.identifier.getCILName()}") @@ -440,9 +505,6 @@ def ForStmt(self, node: ForStmt): self.instr(f"br {startLabel}") self.label(endLabel) - - - def ListExpr(self, node: ListExpr): t = node.inferredType length = len(node.elements) @@ -485,15 +547,18 @@ def ReturnStmt(self, node: ReturnStmt): def Identifier(self, node: Identifier): if self.defaultToGlobals or node.varInstance.isGlobal: - self.instr(f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") + self.instr( + f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") elif node.varInstance.isNonlocal: raise Exception("unimplemented") else: self.load(node.getCILName()) def MemberExpr(self, node: MemberExpr): - raise Exception("unimplemented") - + self.visit(node.object) + self.instr( + f"ldfld {node.inferredType.getCILName()} {node.object.inferredType.getCILName()}::{node.member.getCILName()}") + def IfExpr(self, node: IfExpr): self.visit(node.condition) l1 = self.newLabelName() @@ -506,7 +571,19 @@ def IfExpr(self, node: IfExpr): self.label(l2) def MethodCallExpr(self, node: MethodCallExpr): - raise Exception("unimplemented") + className = node.method.object.inferredType.className + methodName = node.method.member.getCILName() + if methodName == "__init__" and className in {"int", "bool"}: + return + self.visit(node.method.object) + for i in range(len(node.args)): + self.visitArg(node.method.inferredType, i + 1, node.args[i]) + methodType = node.method.inferredType.dropFirstParam() + signature = methodType.getCILSignature( + f"{className}::{methodName}") + self.instr(f"callvirt instance {signature}") + if methodType.returnType.isNone(): + self.NoneLiteral(None) # push null for void return # LITERALS @@ -537,7 +614,7 @@ def ClassType(self, node: ClassType): pass def emit(self) -> str: - return self.currentBuilder().emit() + return self.builder.emit() # SUGAR @@ -559,7 +636,8 @@ def emit_assert(self, arg: Expr): def emit_exn(self, msg: str): self.instr(f'ldstr "{msg}"') - self.instr("newobj instance void [mscorlib]System.Exception::.ctor(string)") + self.instr( + "newobj instance void [mscorlib]System.Exception::.ctor(string)") self.instr("throw") def emit_input(self): @@ -585,12 +663,14 @@ def emit_len(self, arg: Expr): self.instr("ldlen") self.instr("conv.i8") else: - self.instr("callvirt instance int32 [mscorlib]System.String::get_Length()") + self.instr( + "callvirt instance int32 [mscorlib]System.String::get_Length()") self.instr("conv.i8") def emit_print(self, arg: Expr): self.visit(arg) - self.instr(f"call void class [mscorlib]System.Console::WriteLine({arg.inferredType.getCILName()})") + self.instr( + f"call void class [mscorlib]System.Console::WriteLine({arg.inferredType.getCILName()})") self.NoneLiteral(None) def visitArg(self, funcType, paramIdx: int, arg: Expr): diff --git a/compiler/compiler.py b/compiler/compiler.py index 3ae7bea..600a433 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -71,5 +71,5 @@ def emitCIL(self, main:str, ast: Node): EmptyListTyper().visit(ast) cil_backend = CilBackend(main, self.transformer.ts) cil_backend.visit(ast) - return cil_backend.classes + return cil_backend.builder diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 5be301f..3b9441b 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -197,21 +197,11 @@ def ClassDef(self, node: ClassDef): else: self.method(d) if constructor_def == None: - funcDef = FuncDef(node.location, - Identifier(node.location, "__init__"), - [TypedVar(node.location, - Identifier(node.location, "self"), - ClassType(node.location, self.currentClass) - )], - ClassType(node.location, ""), - var_decls, - [], True - ) - funcDef.type = FuncType([ClassValueType(self.currentClass)], NoneType()) + funcDef = node.getDefaultConstructor() self.constructor(superclass, funcDef) self.instr(".end class") - def funcDefHelper(self, node: FuncDef, isConstructor = False): + def funcDefHelper(self, node: FuncDef): for i in range(len(node.params)): self.newLocalEntry(node.params[i].identifier.name) for d in node.declarations: @@ -237,7 +227,7 @@ def constructor(self, superclass:str, node: FuncDef): # call superclass constructor self.instr("aload 0") self.instr(f"invokespecial Method {superclass} ()V ") - self.funcDefHelper(node, True) + self.funcDefHelper(node) self.instr(".end code") self.currentBuilder().unindent() self.instr(".end method") diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index ebbc9a1..693c13c 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -85,7 +85,7 @@ def getCILName(self): elif self.className == "int": return "int64" else: - return self.className + return "class "+self.className def __str__(self): return self.className diff --git a/main.py b/main.py index 57b8cd8..f355d03 100644 --- a/main.py +++ b/main.py @@ -115,16 +115,14 @@ def main(): out_msg(fname, args.verbose) f.write(jvm_emitter.emit()) elif args.mode == "cil": - cil_emitters = compiler.emitCIL(infile_name, tree) - for cls in cil_emitters: - cil_emitter = cil_emitters[cls] - if args.should_print: - print(cil_emitter.emit()) - else: - fname = outdir + cls + ".cil" - with open(fname, "w") as f: - out_msg(fname, args.verbose) - f.write(cil_emitter.emit()) + cil_emitter = compiler.emitCIL(infile_name, tree) + if args.should_print: + print(cil_emitter.emit()) + else: + fname = outdir + cil_emitter.name + ".cil" + with open(fname, "w") as f: + out_msg(fname, args.verbose) + f.write(cil_emitter.emit()) if __name__ == "__main__": main() diff --git a/test.py b/test.py index 64f0a4f..59fce23 100644 --- a/test.py +++ b/test.py @@ -17,6 +17,7 @@ def run_all_tests(): run_python_backend_tests() run_closure_tests() run_jvm_tests() + run_cil_tests() def run_parse_tests(): print("Running parser tests...\n") @@ -168,6 +169,26 @@ def run_jvm_tests(): print("\nNot all test cases passed. Please run `make clean` after inspecting the output") print("\nPassed {:d} out of {:d} JVM backend test cases\n".format(n_passed, total)) +def run_cil_tests(): + print("Running CIL backend tests...\n") + total = 0 + n_passed = 0 + cil_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() + for test in cil_tests_dir.glob('*.py'): + passed = run_cil_test(test) + total += 1 + if not passed: + print("Failed: "+ str(test) + "\n") + else: + n_passed += 1 + if total == n_passed: + subprocess.run("cd {} && rm -f *.cil && rm -f *.exe".format( + str(Path(__file__).parent.resolve()) + ), shell=True) + else: + print("\nNot all test cases passed. Please run `make clean` after inspecting the output") + print("\nPassed {:d} out of {:d} CIL backend test cases\n".format(n_passed, total)) + def run_parse_test(test, bad=True)->bool: # if bad=True, then test cases prefixed with bad are expected to fail compiler = Compiler() @@ -372,6 +393,49 @@ def run_jvm_test(test)->bool: return False return passed +def run_cil_test(test)->bool: + passed = True + try: + infile_name = str(test)[:-3].split("/")[-1] + outdir = "./" + compiler = Compiler() + astparser = compiler.parser + ast = compiler.parse(test) + if len(astparser.errors) > 0: + return False + compiler.typecheck(ast) + if len(ast.errors.errors) > 0: + print(ast.errors.toJSON(False)) + return False + cil_emitter = compiler.emitCIL(infile_name, ast) + fname = outdir + cil_emitter.name + ".cil" + with open(fname, "w") as f: + f.write(cil_emitter.emit()) + except Exception as e: + print("Internal compiler error:", test) + track = traceback.format_exc() + print(e) + print(track) + return False + try: + assembler_commands = ["ilasm {}.cil".format(str(test.name[:-3]))] + output = subprocess.check_output("cd {} && {} && mono {}.exe".format( + str(Path(__file__).parent.resolve()), + " && ".join(assembler_commands), + str(test.name[:-3]) + ), shell=True) + lines = output.decode().split("\n") + for l in lines: + for e in error_flags: + if e in l: + passed = False + print(l) + break + except Exception as e: + print(e) + return False + return passed + def ast_equals(d1, d2)->bool: # precondition: the input dict must represent a well-formed AST # d1 is the correct AST, d2 is the AST output by this compiler From 0a5f1eec46b58caca0428dfb75f6b29de43760e9 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 17 May 2022 16:08:00 -0400 Subject: [PATCH 12/79] some more keywords --- compiler/astnodes/identifier.py | 2 +- tests/runtime/var_decl.py | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index 9c2a635..bf28423 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -1,6 +1,6 @@ from .expr import Expr -CIL_KEYWORDS = set(["char", "value"] + +CIL_KEYWORDS = set(["char", "value", "int32", "int64", "string", "long", "null"] + ["add", "and", "any", diff --git a/tests/runtime/var_decl.py b/tests/runtime/var_decl.py index 8d17880..b09f53f 100644 --- a/tests/runtime/var_decl.py +++ b/tests/runtime/var_decl.py @@ -2,6 +2,15 @@ x:str = "mystring" y:int = 1 z:bool = True +# potentially colliding names +i8:int = 1 +i32: int = 1 +i64: int = 1 +int32: int = 1 +int64: int = 1 +long: int = 1 +string: str = "" +null: str = "" print(x) print(y) print(z) \ No newline at end of file From 5ea90205404a8ab0925d37bf4dd71d79b8df591e Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 17 May 2022 16:17:28 -0400 Subject: [PATCH 13/79] fix parent class init, doubling vector test case passes --- compiler/cil_backend.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 4ea775e..4121a3f 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -166,11 +166,8 @@ def constructor(superclass: str, func: FuncDef): func.name.name = ".ctor" # add call to parent constructor after child field initialization # before other constructor statements - call = CallExpr(func.location, Identifier(func.location, node.superclass.getCILName()), []) - call.isConstructor = True - parentCall = ExprStmt(func.location, call) - func.statements.insert(0, parentCall) - self.FuncDef(func, "specialname rtspecialname instance", True) + self.FuncDef(func, "specialname rtspecialname instance", + node.superclass.getCILName()) func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] var_decls = [d for d in node.declarations if isinstance(d, VarDef)] @@ -195,7 +192,7 @@ def constructor(superclass: str, func: FuncDef): else: # method d.type = d.type.dropFirstParam() - self.FuncDef(d, "virtual instance", True) + self.FuncDef(d, "virtual instance") if constructor_def == None: # give a default constructor if none exists funcDef = node.getDefaultConstructor() @@ -214,7 +211,7 @@ def generateLocalsDirective(self, locals): locals.newLine(sortedDecls[i].decl() + comma) locals.unindent().newLine(")") - def FuncDef(self, node: FuncDef, funcType: str = "static", isMethod: bool = False): + def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None): self.instr(f".method public hidebysig {funcType}") self.instr( f"{node.type.getCILSignature(node.name.getCILName())} cil managed") @@ -232,6 +229,9 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", isMethod: bool = Fals self.returnType = node.type.returnType # handle last return + if superConstructor: + self.instr("ldarg.0") + self.instr(f"call instance void {superConstructor}::.ctor()") self.visitStmtList(node.statements) hasReturn = False for s in node.statements: From b0c9c42f7a8856a3c4e4cd08d3544353b4cb6f05 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 17 May 2022 16:38:02 -0400 Subject: [PATCH 14/79] broken nonlocals --- compiler/cil_backend.py | 35 ++++++++++++++++++++++++++++++----- tests/runtime/nonlocal.py | 30 +++++++++++++++--------------- 2 files changed, 45 insertions(+), 20 deletions(-) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 4121a3f..83be05b 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -253,7 +253,9 @@ def VarDef(self, node: VarDef): self.instr( f"stfld {node.var.t.getCILName()} {className.getCILName()}::{node.getIdentifier().getCILName()}") elif node.var.varInstance.isNonlocal: - raise Exception("unimplemented") + elementType = node.var.t + self.wrap(node.value, elementType) + self.newLocal(varName, ListValueType(elementType)) else: self.visit(node.value) self.newLocal(varName, node.var.t) @@ -266,7 +268,11 @@ def processAssignmentTarget(self, target: Expr): self.instr( f"stsfld {target.inferredType.getCILName()} {self.main}::{target.getCILName()}") elif target.varInstance.isNonlocal: - raise Exception("unimplemented") + temp = self.newLocal(None, target.inferredType) + self.visit(target) + self.instr("ldc.i4 0") + self.load(temp) + self.arrayStore(target.inferredType) else: self.store(target.getCILName()) elif isinstance(target, IndexExpr): @@ -550,7 +556,9 @@ def Identifier(self, node: Identifier): self.instr( f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") elif node.varInstance.isNonlocal: - raise Exception("unimplemented") + self.load(node.name) + self.instr("ldc.i4 0") + self.arrayLoad(node.inferredType) else: self.load(node.getCILName()) @@ -674,5 +682,22 @@ def emit_print(self, arg: Expr): self.NoneLiteral(None) def visitArg(self, funcType, paramIdx: int, arg: Expr): - self.visit(arg) - # TODO + argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + paramIsRef = paramIdx in funcType.refParams + if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + # ref arg and ref param, pass ref arg + self.load(arg.name) + elif paramIsRef: + # non-ref arg and ref param, or do not pass ref arg + # unwrap if necessary, re-wrap + self.wrap(arg, arg.inferredType) + else: # non-ref param, maybe unwrap + self.visit(arg) + + def wrap(self, val:Expr, elementType:ValueType): + self.instr("ldc.i4 1") + self.instr(f"newarr {elementType.getCILName()}") + self.instr("dup") + self.instr("ldc.i4 0") + self.visit(val) + self.arrayStore(elementType) diff --git a/tests/runtime/nonlocal.py b/tests/runtime/nonlocal.py index dc8cdc2..bc3ec4a 100644 --- a/tests/runtime/nonlocal.py +++ b/tests/runtime/nonlocal.py @@ -83,23 +83,23 @@ def test14(): # nonlocals can be mutated __assert__(test(1) == 2) -__assert__(test3() == 3) +# __assert__(test3() == 3) -# nonlocals passed into functions cannot be mutated -a = 0 -test9(a) -__assert__(a == 0) +# # nonlocals passed into functions cannot be mutated +# a = 0 +# test9(a) +# __assert__(a == 0) -# array idx's can be mutated w/o nonlocal -test7() +# # array idx's can be mutated w/o nonlocal +# test7() -test10() +# test10() -a = 0 -b = Nonlocals() -b.testMethod(a) -b.testMethod(0) -__assert__(a == 0) -b.testMethod(1) -b.testMethod4() +# a = 0 +# b = Nonlocals() +# b.testMethod(a) +# b.testMethod(0) +# __assert__(a == 0) +# b.testMethod(1) +# b.testMethod4() From 62a3e94a5ef534cd9d815adace0d05911cbdeb76 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 17 May 2022 17:18:51 -0400 Subject: [PATCH 15/79] fix nonlocals --- compiler/cil_backend.py | 2 +- compiler/types/functype.py | 10 +++++++++- tests/runtime/nonlocal.py | 30 +++++++++++++++--------------- 3 files changed, 25 insertions(+), 17 deletions(-) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 83be05b..4ecdbeb 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -269,7 +269,7 @@ def processAssignmentTarget(self, target: Expr): f"stsfld {target.inferredType.getCILName()} {self.main}::{target.getCILName()}") elif target.varInstance.isNonlocal: temp = self.newLocal(None, target.inferredType) - self.visit(target) + self.load(target.name) self.instr("ldc.i4 0") self.load(temp) self.arrayStore(target.inferredType) diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 0c761e5..987f424 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -21,7 +21,15 @@ def dropFirstParam(self): return f def getCILSignature(self, name: str)->str: - paramSig = ", ".join([t.getCILSignature() for t in self.parameters]) + params = [] + for i in range(len(self.parameters)): + p = self.parameters[i] + if i in self.refParams and isinstance(p, ClassValueType): + sig = p.getCILSignature() + "[]" + else: + sig = p.getCILSignature() + params.append(sig) + paramSig = ", ".join(params) return f"{self.returnType.getCILSignature()} {name}({paramSig})" def getJavaSignature(self)->str: diff --git a/tests/runtime/nonlocal.py b/tests/runtime/nonlocal.py index bc3ec4a..dc8cdc2 100644 --- a/tests/runtime/nonlocal.py +++ b/tests/runtime/nonlocal.py @@ -83,23 +83,23 @@ def test14(): # nonlocals can be mutated __assert__(test(1) == 2) -# __assert__(test3() == 3) +__assert__(test3() == 3) -# # nonlocals passed into functions cannot be mutated -# a = 0 -# test9(a) -# __assert__(a == 0) +# nonlocals passed into functions cannot be mutated +a = 0 +test9(a) +__assert__(a == 0) -# # array idx's can be mutated w/o nonlocal -# test7() +# array idx's can be mutated w/o nonlocal +test7() -# test10() +test10() -# a = 0 -# b = Nonlocals() -# b.testMethod(a) -# b.testMethod(0) -# __assert__(a == 0) -# b.testMethod(1) -# b.testMethod4() +a = 0 +b = Nonlocals() +b.testMethod(a) +b.testMethod(0) +__assert__(a == 0) +b.testMethod(1) +b.testMethod4() From c9d4a1b9e3c8c6bd1908465a3e361f32c5221637 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 18 May 2022 00:51:44 -0400 Subject: [PATCH 16/79] cleanup --- compiler/cil_backend.py | 56 +++++++---------------------------------- compiler/jvm_backend.py | 50 ++++++------------------------------ compiler/visitor.py | 27 ++++++++++++++++++++ 3 files changed, 43 insertions(+), 90 deletions(-) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 4ecdbeb..857ac56 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -1,9 +1,8 @@ -from cmath import log from .astnodes import * from .types import * from .builder import Builder from .typesystem import TypeSystem -from .visitor import Visitor +from .visitor import CommonVisitor from collections import defaultdict import json @@ -19,18 +18,15 @@ def decl(self): return f"[{self.loc}] {self.t} {self.name}" -class CilBackend(Visitor): +class CilBackend(CommonVisitor): + stackLimit = 500 + defaultToGlobals = False # treat all vars as global if this is true def __init__(self, main: str, ts: TypeSystem): self.builder = Builder(main) self.main = main # name of main class - self.locals = [defaultdict(lambda: None)] - self.counter = 0 # for labels - self.returnType = None - self.localLimit = 50 - self.stackLimit = 500 self.ts = ts - self.defaultToGlobals = False # treat all vars as global if this is true + self.enterScope() def indent(self): self.instr("{") @@ -40,11 +36,8 @@ def unindent(self): self.builder.unindent() self.instr("}") - def visit(self, node: Node): - node.visit(self) - - def instr(self, instr: str): - self.builder.newLine(instr) + def currentBuilder(self): + return self.builder def newLabelName(self) -> str: self.counter += 1 @@ -55,15 +48,6 @@ def label(self, name: str) -> str: self.instr(name+": nop") self.builder.indent() - def enterScope(self): - self.locals.append(defaultdict(lambda: None)) - - def exitScope(self): - self.locals.pop() - - def wrap(self, val: Expr, elementType: ValueType): - raise Exception("unimplemented") - def store(self, name: str): n = self.locals[-1][name] if n is None: @@ -138,7 +122,7 @@ def Program(self, node: Program): ".method public static hidebysig default void Main (string[] args) cil managed") self.indent() self.instr(".entrypoint") - self.instr(f".maxstack {self.localLimit}") + self.instr(f".maxstack {self.stackLimit}") locals = self.builder.newBlock() self.defaultToGlobals = True for v in var_decls: @@ -216,7 +200,7 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None self.instr( f"{node.type.getCILSignature(node.name.getCILName())} cil managed") self.indent() - self.instr(f".maxstack {self.localLimit}") + self.instr(f".maxstack {self.stackLimit}") self.enterScope() # initialize locals @@ -610,28 +594,6 @@ def NoneLiteral(self, node: NoneLiteral): def StringLiteral(self, node: StringLiteral): self.instr(f"ldstr {json.dumps(node.value)}") - # TYPES - - def TypedVar(self, node: TypedVar): - pass - - def ListType(self, node: ListType): - pass - - def ClassType(self, node: ClassType): - pass - - def emit(self) -> str: - return self.builder.emit() - - # SUGAR - - def NonLocalDecl(self, node: NonLocalDecl): - pass - - def GlobalDecl(self, node: GlobalDecl): - pass - # BUILT-INS - note: these are in-lined def emit_assert(self, arg: Expr): label = self.newLabelName() diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 3b9441b..3d34cb2 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -2,35 +2,27 @@ from .types import * from .builder import Builder from .typesystem import TypeSystem -from .visitor import Visitor +from .visitor import CommonVisitor from collections import defaultdict import json -class JvmBackend(Visitor): +class JvmBackend(CommonVisitor): + classes = dict() + localLimit = 50 + stackLimit = 500 + defaultToGlobals = False # treat all vars as global if this is true def __init__(self, main: str, ts: TypeSystem): - self.classes = dict() self.classes[main] = Builder(main) self.currentClass = main self.main = main # name of main class - self.locals = [defaultdict(lambda: None)] - self.counter = 0 # for labels - self.returnType = None - self.localLimit = 50 - self.stackLimit = 500 self.ts = ts - self.defaultToGlobals = False # treat all vars as global if this is true + self.enterScope() def currentBuilder(self): return self.classes[self.currentClass] - def visit(self, node: Node): - node.visit(self) - - def instr(self, instr: str): - self.currentBuilder().newLine(instr) - def newLabelName(self) -> str: self.counter += 1 return "L"+str(self.counter) @@ -40,12 +32,6 @@ def label(self, name: str) -> str: self.instr(name+":") self.currentBuilder().indent() - def enterScope(self): - self.locals.append(defaultdict(lambda: None)) - - def exitScope(self): - self.locals.pop() - def returnInstr(self, exprType: ValueType): if exprType.isJavaRef(): self.instr("areturn") @@ -654,28 +640,6 @@ def NoneLiteral(self, node: NoneLiteral): def StringLiteral(self, node: StringLiteral): self.instr(f"ldc {json.dumps(node.value)}") - # TYPES - - def TypedVar(self, node: TypedVar): - pass - - def ListType(self, node: ListType): - pass - - def ClassType(self, node: ClassType): - pass - - def emit(self) -> str: - return self.currentBuilder().emit() - - # SUGAR - - def NonLocalDecl(self, node: NonLocalDecl): - pass - - def GlobalDecl(self, node: GlobalDecl): - pass - # BUILT-INS - note: these are in-lined def emit_assert(self, arg: Expr): label = self.newLabelName() diff --git a/compiler/visitor.py b/compiler/visitor.py index fd99526..4627d09 100644 --- a/compiler/visitor.py +++ b/compiler/visitor.py @@ -1,4 +1,6 @@ from .astnodes import * +from collections import defaultdict +from .builder import Builder class Visitor: @@ -96,3 +98,28 @@ def ListType(self, node: ListType): def ClassType(self, node: ClassType): pass + +class CommonVisitor(Visitor): + returnType = None # for tracking return types in functions + counter = 0 # for labels + + # helpers for handling locals + + locals = [] + + def enterScope(self): + self.locals.append(defaultdict(lambda: None)) + + def exitScope(self): + self.locals.pop() + + # helpers for building code + + def instr(self, instr: str): + self.currentBuilder().newLine(instr) + + def currentBuilder(self)->Builder: + raise Exception("unimplemented") + + def emit(self) -> str: + return self.currentBuilder().emit() \ No newline at end of file From 3a18dfa58dbf99a5e3ca986e02869acfdf4d5102 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 18 May 2022 08:46:27 -0400 Subject: [PATCH 17/79] format --- compiler/__init__.py | 2 +- compiler/astnodes/__init__.py | 2 +- compiler/astnodes/assignstmt.py | 3 +- compiler/astnodes/binaryexpr.py | 5 +- compiler/astnodes/booleanliteral.py | 5 +- compiler/astnodes/callexpr.py | 7 +- compiler/astnodes/classdef.py | 24 +-- compiler/astnodes/classtype.py | 5 +- compiler/astnodes/compilererror.py | 3 +- compiler/astnodes/declaration.py | 3 +- compiler/astnodes/errors.py | 4 +- compiler/astnodes/expr.py | 4 +- compiler/astnodes/exprstmt.py | 4 +- compiler/astnodes/forstmt.py | 4 +- compiler/astnodes/funcdef.py | 13 +- compiler/astnodes/globaldecl.py | 3 +- compiler/astnodes/identifier.py | 233 ++++++++++++++-------------- compiler/astnodes/ifexpr.py | 3 +- compiler/astnodes/ifstmt.py | 5 +- compiler/astnodes/indexexpr.py | 3 +- compiler/astnodes/integerliteral.py | 3 +- compiler/astnodes/listexpr.py | 5 +- compiler/astnodes/listtype.py | 4 +- compiler/astnodes/literal.py | 4 +- compiler/astnodes/memberexpr.py | 4 +- compiler/astnodes/methodcallexpr.py | 5 +- compiler/astnodes/node.py | 3 +- compiler/astnodes/noneliteral.py | 3 +- compiler/astnodes/nonlocaldecl.py | 3 +- compiler/astnodes/program.py | 8 +- compiler/astnodes/returnstmt.py | 4 +- compiler/astnodes/stmt.py | 5 +- compiler/astnodes/stringliteral.py | 3 +- compiler/astnodes/typeannotation.py | 5 +- compiler/astnodes/typedvar.py | 6 +- compiler/astnodes/unaryexpr.py | 5 +- compiler/astnodes/vardef.py | 5 +- compiler/astnodes/whilestmt.py | 4 +- compiler/builder.py | 6 +- compiler/cil_backend.py | 6 +- compiler/closuretransformer.py | 15 +- compiler/closurevisitor.py | 24 +-- compiler/compiler.py | 11 +- compiler/empty_list_typer.py | 8 +- compiler/jvm_backend.py | 40 +++-- compiler/nestedfunchoister.py | 20 +-- compiler/parser.py | 6 +- compiler/python_backend.py | 12 +- compiler/typechecker.py | 5 +- compiler/typeeraser.py | 3 +- compiler/types/Types.py | 6 + compiler/types/__init__.py | 2 +- compiler/types/classvaluetype.py | 11 +- compiler/types/functype.py | 13 +- compiler/types/listvaluetype.py | 7 +- compiler/types/symboltype.py | 6 +- compiler/types/valuetype.py | 5 +- compiler/typesystem.py | 18 ++- compiler/varcollector.py | 5 +- compiler/visitor.py | 10 +- 60 files changed, 346 insertions(+), 307 deletions(-) diff --git a/compiler/__init__.py b/compiler/__init__.py index fc80254..2ae2839 100644 --- a/compiler/__init__.py +++ b/compiler/__init__.py @@ -1 +1 @@ -pass \ No newline at end of file +pass diff --git a/compiler/astnodes/__init__.py b/compiler/astnodes/__init__.py index fab51ff..23f0175 100644 --- a/compiler/astnodes/__init__.py +++ b/compiler/astnodes/__init__.py @@ -33,4 +33,4 @@ from .globaldecl import GlobalDecl from .listtype import ListType from .program import Program -from .vardef import VarDef \ No newline at end of file +from .vardef import VarDef diff --git a/compiler/astnodes/assignstmt.py b/compiler/astnodes/assignstmt.py index 5e62823..c701526 100644 --- a/compiler/astnodes/assignstmt.py +++ b/compiler/astnodes/assignstmt.py @@ -1,9 +1,10 @@ from .stmt import Stmt from .expr import Expr + class AssignStmt(Stmt): - def __init__(self, location:[int], targets:[Expr], value:Expr): + def __init__(self, location: [int], targets: [Expr], value: Expr): super().__init__(location, "AssignStmt") self.targets = targets self.value = value diff --git a/compiler/astnodes/binaryexpr.py b/compiler/astnodes/binaryexpr.py index 597b43a..2cbd133 100644 --- a/compiler/astnodes/binaryexpr.py +++ b/compiler/astnodes/binaryexpr.py @@ -1,8 +1,9 @@ from .expr import Expr + class BinaryExpr(Expr): - def __init__(self, location:[int], left:Expr, operator:str, right:Expr): + def __init__(self, location: [int], left: Expr, operator: str, right: Expr): super().__init__(location, "BinaryExpr") self.left = left self.right = right @@ -28,5 +29,3 @@ def toJSON(self, dump_location=True): d["right"] = self.right.toJSON(dump_location) d["operator"] = self.operator return d - - diff --git a/compiler/astnodes/booleanliteral.py b/compiler/astnodes/booleanliteral.py index f95e106..22b6b6e 100644 --- a/compiler/astnodes/booleanliteral.py +++ b/compiler/astnodes/booleanliteral.py @@ -1,12 +1,11 @@ from .literal import Literal + class BooleanLiteral(Literal): - def __init__(self, location:[int], value:bool): + def __init__(self, location: [int], value: bool): super().__init__(location, "BooleanLiteral") self.value = value def visit(self, visitor): return visitor.BooleanLiteral(self) - - diff --git a/compiler/astnodes/callexpr.py b/compiler/astnodes/callexpr.py index 5075f1b..29540f0 100644 --- a/compiler/astnodes/callexpr.py +++ b/compiler/astnodes/callexpr.py @@ -1,14 +1,15 @@ from .expr import Expr from .identifier import Identifier + class CallExpr(Expr): - def __init__(self, location:[int], function:Identifier, args:[Expr]): + def __init__(self, location: [int], function: Identifier, args: [Expr]): super().__init__(location, "CallExpr") self.function = function self.args = args self.isConstructor = False - self.freevars = [] # captured free vars + self.freevars = [] # captured free vars def postorder(self, visitor): for a in self.args: @@ -29,5 +30,3 @@ def toJSON(self, dump_location=True): d["function"] = self.function.toJSON(dump_location) d["args"] = [a.toJSON(dump_location) for a in self.args] return d - - diff --git a/compiler/astnodes/classdef.py b/compiler/astnodes/classdef.py index d29220e..ee4df5f 100644 --- a/compiler/astnodes/classdef.py +++ b/compiler/astnodes/classdef.py @@ -8,6 +8,7 @@ from ..types.functype import FuncType from ..types.Types import NoneType + class ClassDef(Declaration): def __init__(self, location: [int], name: Identifier, superclass: Identifier, declarations: [Declaration]): @@ -47,19 +48,20 @@ def toJSON(self, dump_location=True): def getIdentifier(self): return self.name - def getDefaultConstructor(self)->FuncDef: + def getDefaultConstructor(self) -> FuncDef: var_decls = [d for d in self.declarations if isinstance(d, VarDef)] constructor = FuncDef(self.location, - Identifier(self.location, "__init__"), - [TypedVar(self.location, - Identifier(self.location, "self"), - ClassType(self.location, self.name.name) - )], - ClassType(self.location, ""), - var_decls, - [], True - ) + Identifier(self.location, "__init__"), + [TypedVar(self.location, + Identifier(self.location, "self"), + ClassType(self.location, + self.name.name) + )], + ClassType(self.location, ""), + var_decls, + [], True + ) constructor.params[0].t = ClassValueType(self.name.name) constructor.type = FuncType( - [ClassValueType(self.name.name)], NoneType()) + [ClassValueType(self.name.name)], NoneType()) return constructor diff --git a/compiler/astnodes/classtype.py b/compiler/astnodes/classtype.py index 01d87b8..3ff91b1 100644 --- a/compiler/astnodes/classtype.py +++ b/compiler/astnodes/classtype.py @@ -1,8 +1,9 @@ from .typeannotation import TypeAnnotation + class ClassType(TypeAnnotation): - def __init__(self, location:[int], className:str): + def __init__(self, location: [int], className: str): super().__init__(location, "ClassType") self.className = className @@ -13,5 +14,3 @@ def toJSON(self, dump_location=True): d = super().toJSON(dump_location) d["className"] = self.className return d - - diff --git a/compiler/astnodes/compilererror.py b/compiler/astnodes/compilererror.py index c4afa36..3cb0d62 100644 --- a/compiler/astnodes/compilererror.py +++ b/compiler/astnodes/compilererror.py @@ -1,8 +1,9 @@ from .node import Node + class CompilerError(Node): - def __init__(self, location:[int], message:str, syntax:bool=False): + def __init__(self, location: [int], message: str, syntax: bool = False): super().__init__(location, "CompilerError") self.message = message self.syntax = syntax diff --git a/compiler/astnodes/declaration.py b/compiler/astnodes/declaration.py index 60c4ee3..ac9a6e8 100644 --- a/compiler/astnodes/declaration.py +++ b/compiler/astnodes/declaration.py @@ -1,6 +1,7 @@ from .node import Node + class Declaration(Node): - def __init__(self, location:[int], kind:str): + def __init__(self, location: [int], kind: str): super().__init__(location, kind) diff --git a/compiler/astnodes/errors.py b/compiler/astnodes/errors.py index 367b936..335f0cd 100644 --- a/compiler/astnodes/errors.py +++ b/compiler/astnodes/errors.py @@ -1,9 +1,10 @@ from .node import Node from .compilererror import CompilerError + class Errors(Node): - def __init__(self, location:[int], errors:[CompilerError]): + def __init__(self, location: [int], errors: [CompilerError]): super().__init__(location, "Errors") self.errors = errors @@ -14,4 +15,3 @@ def toJSON(self, dump_location=True): d = super().toJSON(dump_location) d["errors"] = [e.toJSON(dump_location) for e in self.errors] return d - diff --git a/compiler/astnodes/expr.py b/compiler/astnodes/expr.py index 0938518..a9d81d7 100644 --- a/compiler/astnodes/expr.py +++ b/compiler/astnodes/expr.py @@ -1,8 +1,9 @@ from .node import Node + class Expr(Node): - def __init__(self, location:[int], kind:str): + def __init__(self, location: [int], kind: str): super().__init__(location, kind) self.inferredType = None self.shouldBoxAsRef = False @@ -12,4 +13,3 @@ def toJSON(self, dump_location=True): if self.inferredType is not None: d['inferredType'] = self.inferredType.toJSON(dump_location) return d - diff --git a/compiler/astnodes/exprstmt.py b/compiler/astnodes/exprstmt.py index e0e9b45..c7d18ea 100644 --- a/compiler/astnodes/exprstmt.py +++ b/compiler/astnodes/exprstmt.py @@ -1,9 +1,10 @@ from .stmt import Stmt from .expr import Expr + class ExprStmt(Stmt): - def __init__(self, location:[int], expr:Expr): + def __init__(self, location: [int], expr: Expr): super().__init__(location, "ExprStmt") self.expr = expr @@ -23,4 +24,3 @@ def toJSON(self, dump_location=True): d = super().toJSON(dump_location) d["expr"] = self.expr.toJSON(dump_location) return d - diff --git a/compiler/astnodes/forstmt.py b/compiler/astnodes/forstmt.py index a1c4be3..2b3b3e2 100644 --- a/compiler/astnodes/forstmt.py +++ b/compiler/astnodes/forstmt.py @@ -2,9 +2,10 @@ from .expr import Expr from .identifier import Identifier + class ForStmt(Stmt): - def __init__(self, location:[int], identifier:Identifier, iterable:Expr, body:[Stmt]): + def __init__(self, location: [int], identifier: Identifier, iterable: Expr, body: [Stmt]): super().__init__(location, "ForStmt") self.identifier = identifier self.iterable = iterable @@ -34,4 +35,3 @@ def toJSON(self, dump_location=True): d["iterable"] = self.iterable.toJSON(dump_location) d["body"] = [s.toJSON(dump_location) for s in self.body] return d - diff --git a/compiler/astnodes/funcdef.py b/compiler/astnodes/funcdef.py index 96d8ca9..60be06a 100644 --- a/compiler/astnodes/funcdef.py +++ b/compiler/astnodes/funcdef.py @@ -4,6 +4,7 @@ from .typeannotation import TypeAnnotation from .stmt import Stmt + class FuncDef(Declaration): # The AST for @@ -11,8 +12,8 @@ class FuncDef(Declaration): # DECLARATIONS # STATEMENTS - def __init__(self, location:[int], name:Identifier, params:[TypedVar], returnType:TypeAnnotation, - declarations:[Declaration], statements:[Stmt], isMethod:bool = False): + def __init__(self, location: [int], name: Identifier, params: [TypedVar], returnType: TypeAnnotation, + declarations: [Declaration], statements: [Stmt], isMethod: bool = False): super().__init__(location, "FuncDef") self.name = name self.params = params @@ -20,8 +21,8 @@ def __init__(self, location:[int], name:Identifier, params:[TypedVar], returnTyp self.declarations = declarations self.statements = [s for s in statements if s is not None] self.isMethod = isMethod - self.freevars = [] # used in AST transformations, not printed out - self.type = None # type signature of function + self.freevars = [] # used in AST transformations, not printed out + self.type = None # type signature of function def getFreevarNames(self): return set([v.name for v in self.freevars]) @@ -49,10 +50,10 @@ def toJSON(self, dump_location=True): d["name"] = self.name.toJSON(dump_location) d["params"] = [t.toJSON(dump_location) for t in self.params] d["returnType"] = self.returnType.toJSON(dump_location) - d["declarations"] = [decl.toJSON(dump_location) for decl in self.declarations] + d["declarations"] = [decl.toJSON(dump_location) + for decl in self.declarations] d["statements"] = [s.toJSON(dump_location) for s in self.statements] return d def getIdentifier(self): return self.name - diff --git a/compiler/astnodes/globaldecl.py b/compiler/astnodes/globaldecl.py index 9cdfa4f..d7b8a55 100644 --- a/compiler/astnodes/globaldecl.py +++ b/compiler/astnodes/globaldecl.py @@ -1,9 +1,10 @@ from .declaration import Declaration from .identifier import Identifier + class GlobalDecl(Declaration): - def __init__(self, location:[int], variable:Identifier): + def __init__(self, location: [int], variable: Identifier): super().__init__(location, "GlobalDecl") self.variable = variable diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index bf28423..e69d5e1 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -1,122 +1,123 @@ from .expr import Expr CIL_KEYWORDS = set(["char", "value", "int32", "int64", "string", "long", "null"] + - ["add", - "and", - "any", - "arglist", - "as", - "be", - "beq", - "bge", - "bgt", - "ble", - "blt", - "bne", - "box", - "br", - "break", - "brfalse", - "brinst", - "brnull", - "brtrue", - "brzero", - "call", - "calli", - "callvirt", - "can", - "castclass", - "ceq", - "cgt", - "check", - "ckfinite", - "clt", - "constrained", - "conv", - "cpblk", - "cpobj", - "div", - "dup", - "endfault", - "endfilter", - "endfinally", - "execution", - "fault", - "initblk", - "initobj", - "instruction", - "isinst", - "jmp", - "ldarg", - "ldarga", - "ldc", - "ldelem", - "ldelema", - "ldfld", - "ldflda", - "ldftn", - "ldind", - "ldlen", - "ldloc", - "ldloca", - "ldnull", - "ldobj", - "ldsfld", - "ldsflda", - "ldstr", - "ldtoken", - "ldvirtftn", - "leave", - "localloc", - "mkrefany", - "mul", - "neg", - "newarr", - "newobj", - "no", - "nop", - "normally", - "not", - "nullcheck", - "of", - "or", - "ovf", - "part", - "performed", - "pop", - "rangecheck", - "readonly", - "ref", - "refanytype", - "refanyval", - "rem", - "ret", - "rethrow", - "shall", - "shl", - "shr", - "sizeof", - "skipped", - "specified", - "starg", - "stelem", - "stfld", - "stind", - "stloc", - "stobj", - "stsfld", - "sub", - "subsequent", - "switch", - "tail", - "throw", - "typecheck", - "un", - "unaligned", - "unbox", - "volatile", - "xor"] - ) + ["add", + "and", + "any", + "arglist", + "as", + "be", + "beq", + "bge", + "bgt", + "ble", + "blt", + "bne", + "box", + "br", + "break", + "brfalse", + "brinst", + "brnull", + "brtrue", + "brzero", + "call", + "calli", + "callvirt", + "can", + "castclass", + "ceq", + "cgt", + "check", + "ckfinite", + "clt", + "constrained", + "conv", + "cpblk", + "cpobj", + "div", + "dup", + "endfault", + "endfilter", + "endfinally", + "execution", + "fault", + "initblk", + "initobj", + "instruction", + "isinst", + "jmp", + "ldarg", + "ldarga", + "ldc", + "ldelem", + "ldelema", + "ldfld", + "ldflda", + "ldftn", + "ldind", + "ldlen", + "ldloc", + "ldloca", + "ldnull", + "ldobj", + "ldsfld", + "ldsflda", + "ldstr", + "ldtoken", + "ldvirtftn", + "leave", + "localloc", + "mkrefany", + "mul", + "neg", + "newarr", + "newobj", + "no", + "nop", + "normally", + "not", + "nullcheck", + "of", + "or", + "ovf", + "part", + "performed", + "pop", + "rangecheck", + "readonly", + "ref", + "refanytype", + "refanyval", + "rem", + "ret", + "rethrow", + "shall", + "shl", + "shr", + "sizeof", + "skipped", + "specified", + "starg", + "stelem", + "stfld", + "stind", + "stloc", + "stobj", + "stsfld", + "sub", + "subsequent", + "switch", + "tail", + "throw", + "typecheck", + "un", + "unaligned", + "unbox", + "volatile", + "xor"] + ) + class Identifier(Expr): diff --git a/compiler/astnodes/ifexpr.py b/compiler/astnodes/ifexpr.py index 676066f..99785cc 100644 --- a/compiler/astnodes/ifexpr.py +++ b/compiler/astnodes/ifexpr.py @@ -1,8 +1,9 @@ from .expr import Expr + class IfExpr(Expr): - def __init__(self, location:[int], condition:Expr, thenExpr:Expr, elseExpr:Expr): + def __init__(self, location: [int], condition: Expr, thenExpr: Expr, elseExpr: Expr): super().__init__(location, "IfExpr") self.condition = condition self.thenExpr = thenExpr diff --git a/compiler/astnodes/ifstmt.py b/compiler/astnodes/ifstmt.py index ef61e86..4810dd8 100644 --- a/compiler/astnodes/ifstmt.py +++ b/compiler/astnodes/ifstmt.py @@ -1,15 +1,15 @@ from .stmt import Stmt from .expr import Expr + class IfStmt(Stmt): - def __init__(self, location:[int], condition:Expr, thenBody:[Stmt], elseBody:[Stmt]): + def __init__(self, location: [int], condition: Expr, thenBody: [Stmt], elseBody: [Stmt]): super().__init__(location, "IfStmt") self.condition = condition self.thenBody = [s for s in thenBody if s is not None] self.elseBody = [s for s in elseBody if s is not None] - def postorder(self, visitor): visitor.visit(self.condition) for s in self.thenBody: @@ -36,4 +36,3 @@ def toJSON(self, dump_location=True): d["thenBody"] = [s.toJSON(dump_location) for s in self.thenBody] d["elseBody"] = [s.toJSON(dump_location) for s in self.elseBody] return d - diff --git a/compiler/astnodes/indexexpr.py b/compiler/astnodes/indexexpr.py index 2768453..43860b1 100644 --- a/compiler/astnodes/indexexpr.py +++ b/compiler/astnodes/indexexpr.py @@ -1,8 +1,9 @@ from .expr import Expr + class IndexExpr(Expr): - def __init__(self, location:[int], lst:Expr, index:Expr): + def __init__(self, location: [int], lst: Expr, index: Expr): super().__init__(location, "IndexExpr") self.list = lst self.index = index diff --git a/compiler/astnodes/integerliteral.py b/compiler/astnodes/integerliteral.py index 14823ad..8755044 100644 --- a/compiler/astnodes/integerliteral.py +++ b/compiler/astnodes/integerliteral.py @@ -1,8 +1,9 @@ from .literal import Literal + class IntegerLiteral(Literal): - def __init__(self, location:[int], value:int): + def __init__(self, location: [int], value: int): super().__init__(location, "IntegerLiteral") self.value = value diff --git a/compiler/astnodes/listexpr.py b/compiler/astnodes/listexpr.py index b116e17..10b5963 100644 --- a/compiler/astnodes/listexpr.py +++ b/compiler/astnodes/listexpr.py @@ -1,8 +1,9 @@ from .expr import Expr + class ListExpr(Expr): - def __init__(self, location:[int], elements:[Expr]): + def __init__(self, location: [int], elements: [Expr]): super().__init__(location, "ListExpr") self.elements = elements self.emptyListType = None @@ -25,5 +26,3 @@ def toJSON(self, dump_location=True): d = super().toJSON(dump_location) d["elements"] = [e.toJSON(dump_location) for e in self.elements] return d - - diff --git a/compiler/astnodes/listtype.py b/compiler/astnodes/listtype.py index 3cc533e..c312267 100644 --- a/compiler/astnodes/listtype.py +++ b/compiler/astnodes/listtype.py @@ -1,8 +1,9 @@ from .typeannotation import TypeAnnotation + class ListType(TypeAnnotation): - def __init__(self, location:[int], elementType:TypeAnnotation): + def __init__(self, location: [int], elementType: TypeAnnotation): super().__init__(location, "ListType") self.elementType = elementType @@ -13,4 +14,3 @@ def toJSON(self, dump_location=True): d = super().toJSON(dump_location) d["elementType"] = self.elementType.toJSON(dump_location) return d - diff --git a/compiler/astnodes/literal.py b/compiler/astnodes/literal.py index 58773e6..de4795e 100644 --- a/compiler/astnodes/literal.py +++ b/compiler/astnodes/literal.py @@ -1,8 +1,9 @@ from .expr import Expr + class Literal(Expr): - def __init__(self, location:[int], kind:str): + def __init__(self, location: [int], kind: str): super().__init__(location, kind) self.value = None @@ -11,4 +12,3 @@ def toJSON(self, dump_location=True): if self.value is not None: d['value'] = self.value return d - diff --git a/compiler/astnodes/memberexpr.py b/compiler/astnodes/memberexpr.py index 77b996f..a059364 100644 --- a/compiler/astnodes/memberexpr.py +++ b/compiler/astnodes/memberexpr.py @@ -1,9 +1,10 @@ from .expr import Expr from .identifier import Identifier + class MemberExpr(Expr): - def __init__(self, location:[int], obj:Expr, member:Identifier): + def __init__(self, location: [int], obj: Expr, member: Identifier): super().__init__(location, "MemberExpr") self.object = obj self.member = member @@ -12,6 +13,7 @@ def preorder(self, visitor): visitor.MemberExpr(self) visitor.visit(self.object) return self + def postorder(self, visitor): visitor.visit(self.object) return visitor.MemberExpr(self) diff --git a/compiler/astnodes/methodcallexpr.py b/compiler/astnodes/methodcallexpr.py index e034688..5f89873 100644 --- a/compiler/astnodes/methodcallexpr.py +++ b/compiler/astnodes/methodcallexpr.py @@ -1,9 +1,10 @@ from .expr import Expr from .memberexpr import MemberExpr + class MethodCallExpr(Expr): - def __init__(self, location:[int], method:MemberExpr, args:[Expr]): + def __init__(self, location: [int], method: MemberExpr, args: [Expr]): super().__init__(location, "MethodCallExpr") self.method = method self.args = args @@ -29,5 +30,3 @@ def toJSON(self, dump_location=True): d["method"] = self.method.toJSON(dump_location) d["args"] = [a.toJSON(dump_location) for a in self.args] return d - - diff --git a/compiler/astnodes/node.py b/compiler/astnodes/node.py index 244d7b1..8834c7b 100644 --- a/compiler/astnodes/node.py +++ b/compiler/astnodes/node.py @@ -1,7 +1,7 @@ class Node: - def __init__(self, location:[int], kind:str): + def __init__(self, location: [int], kind: str): if len(location) != 2: raise Exception('location must be length 2') self.kind = kind @@ -25,4 +25,3 @@ def toJSON(self, dump_location=True): if self.errorMsg is not None: d['errorMsg'] = self.errorMsg return d - diff --git a/compiler/astnodes/noneliteral.py b/compiler/astnodes/noneliteral.py index fb6ca37..20ad082 100644 --- a/compiler/astnodes/noneliteral.py +++ b/compiler/astnodes/noneliteral.py @@ -1,8 +1,9 @@ from .literal import Literal + class NoneLiteral(Literal): - def __init__(self, location:[int]): + def __init__(self, location: [int]): super().__init__(location, "NoneLiteral") self.value = None diff --git a/compiler/astnodes/nonlocaldecl.py b/compiler/astnodes/nonlocaldecl.py index ff4f067..ce269e9 100644 --- a/compiler/astnodes/nonlocaldecl.py +++ b/compiler/astnodes/nonlocaldecl.py @@ -1,9 +1,10 @@ from .declaration import Declaration from .identifier import Identifier + class NonLocalDecl(Declaration): - def __init__(self, location:[int], variable:Identifier): + def __init__(self, location: [int], variable: Identifier): super().__init__(location, "NonLocalDecl") self.variable = variable diff --git a/compiler/astnodes/program.py b/compiler/astnodes/program.py index 1f596c5..9a13c11 100644 --- a/compiler/astnodes/program.py +++ b/compiler/astnodes/program.py @@ -4,9 +4,11 @@ from .errors import Errors # root AST for source file + + class Program(Node): - def __init__(self, location:[int], declarations:[Declaration], statements:[Stmt], errors:Errors): + def __init__(self, location: [int], declarations: [Declaration], statements: [Stmt], errors: Errors): super().__init__(location, "Program") self.declarations = [d for d in declarations if d is not None] self.statements = [s for s in statements if s is not None] @@ -32,8 +34,8 @@ def visit(self, visitor): def toJSON(self, dump_location=True): d = super().toJSON(dump_location) - d['declarations'] = [d.toJSON(dump_location) for d in self.declarations] + d['declarations'] = [d.toJSON(dump_location) + for d in self.declarations] d['statements'] = [s.toJSON(dump_location) for s in self.statements] d['errors'] = self.errors.toJSON(dump_location) return d - diff --git a/compiler/astnodes/returnstmt.py b/compiler/astnodes/returnstmt.py index 28f7cdb..58280ec 100644 --- a/compiler/astnodes/returnstmt.py +++ b/compiler/astnodes/returnstmt.py @@ -1,9 +1,10 @@ from .stmt import Stmt from .expr import Expr + class ReturnStmt(Stmt): - def __init__(self, location:[int], value:Expr): + def __init__(self, location: [int], value: Expr): super().__init__(location, "ReturnStmt") self.value = value self.isReturn = True @@ -30,4 +31,3 @@ def toJSON(self, dump_location=True): else: d["value"] = None return d - diff --git a/compiler/astnodes/stmt.py b/compiler/astnodes/stmt.py index ea86078..6d8d22e 100644 --- a/compiler/astnodes/stmt.py +++ b/compiler/astnodes/stmt.py @@ -1,9 +1,8 @@ from .node import Node + class Stmt(Node): - def __init__(self, location:[int], kind:str): + def __init__(self, location: [int], kind: str): super().__init__(location, kind) self.isReturn = False - - diff --git a/compiler/astnodes/stringliteral.py b/compiler/astnodes/stringliteral.py index 0f8aa08..ec901a2 100644 --- a/compiler/astnodes/stringliteral.py +++ b/compiler/astnodes/stringliteral.py @@ -1,8 +1,9 @@ from .literal import Literal + class StringLiteral(Literal): - def __init__(self, location:[int], value:str): + def __init__(self, location: [int], value: str): super().__init__(location, "StringLiteral") self.value = value diff --git a/compiler/astnodes/typeannotation.py b/compiler/astnodes/typeannotation.py index fb0f108..370d8ca 100644 --- a/compiler/astnodes/typeannotation.py +++ b/compiler/astnodes/typeannotation.py @@ -1,8 +1,7 @@ from .node import Node + class TypeAnnotation(Node): - def __init__(self, location:[int], kind:str): + def __init__(self, location: [int], kind: str): super().__init__(location, kind) - - diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 1b4ac66..1dd56aa 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -2,13 +2,14 @@ from .identifier import Identifier from .typeannotation import TypeAnnotation + class TypedVar(Node): - def __init__(self, location:[int], identifier:Identifier, typ:TypeAnnotation): + def __init__(self, location: [int], identifier: Identifier, typ: TypeAnnotation): super().__init__(location, "TypedVar") self.identifier = identifier self.type = typ - self.t = None # the typechecked type goes here + self.t = None # the typechecked type goes here self.varInstance = None def visit(self, visitor): @@ -19,4 +20,3 @@ def toJSON(self, dump_location=True): d["identifier"] = self.identifier.toJSON(dump_location) d["type"] = self.type.toJSON(dump_location) return d - diff --git a/compiler/astnodes/unaryexpr.py b/compiler/astnodes/unaryexpr.py index 7a84831..e5bd872 100644 --- a/compiler/astnodes/unaryexpr.py +++ b/compiler/astnodes/unaryexpr.py @@ -1,8 +1,9 @@ from .expr import Expr + class UnaryExpr(Expr): - def __init__(self, location:[int], operator:str, operand:Expr): + def __init__(self, location: [int], operator: str, operand: Expr): super().__init__(location, "UnaryExpr") self.operand = operand self.operator = operator @@ -24,5 +25,3 @@ def toJSON(self, dump_location=True): d["operator"] = self.operator d["operand"] = self.operand.toJSON(dump_location) return d - - diff --git a/compiler/astnodes/vardef.py b/compiler/astnodes/vardef.py index 7372b95..e955e40 100644 --- a/compiler/astnodes/vardef.py +++ b/compiler/astnodes/vardef.py @@ -2,9 +2,10 @@ from .expr import Expr from .typedvar import TypedVar + class VarDef(Declaration): - def __init__(self, location:[int], var:TypedVar, value:Expr, isAttr:bool=False, attrOfClass=None): + def __init__(self, location: [int], var: TypedVar, value: Expr, isAttr: bool = False, attrOfClass=None): super().__init__(location, "VarDef") self.var = var self.value = value @@ -32,5 +33,5 @@ def toJSON(self, dump_location=True): def getIdentifier(self): return self.var.identifier - def getName(self)->str: + def getName(self) -> str: return self.var.identifier.name diff --git a/compiler/astnodes/whilestmt.py b/compiler/astnodes/whilestmt.py index ce114c5..6878380 100644 --- a/compiler/astnodes/whilestmt.py +++ b/compiler/astnodes/whilestmt.py @@ -1,9 +1,10 @@ from .stmt import Stmt from .expr import Expr + class WhileStmt(Stmt): - def __init__(self, location:[int], condition:Expr, body:[Stmt]): + def __init__(self, location: [int], condition: Expr, body: [Stmt]): super().__init__(location, "WhileStmt") self.condition = condition self.body = [s for s in body if s is not None] @@ -29,4 +30,3 @@ def toJSON(self, dump_location=True): d["condition"] = self.condition.toJSON(dump_location) d["body"] = [s.toJSON(dump_location) for s in self.body] return d - diff --git a/compiler/builder.py b/compiler/builder.py index be911c9..8a38f74 100644 --- a/compiler/builder.py +++ b/compiler/builder.py @@ -2,9 +2,9 @@ class Builder: - def __init__(self, name:str): + def __init__(self, name: str): self.name = name - self.lines = [] # list of strings or children builders + self.lines = [] # list of strings or children builders self.indentation = 0 def newLine(self, line=""): @@ -32,7 +32,7 @@ def unindent(self): self.indentation -= 1 return self - def emit(self)->str: + def emit(self) -> str: lines = [] for l in self.lines: if isinstance(l, str): diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 857ac56..49ad597 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -646,17 +646,17 @@ def emit_print(self, arg: Expr): def visitArg(self, funcType, paramIdx: int, arg: Expr): argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg self.load(arg.name) elif paramIsRef: # non-ref arg and ref param, or do not pass ref arg # unwrap if necessary, re-wrap self.wrap(arg, arg.inferredType) - else: # non-ref param, maybe unwrap + else: # non-ref param, maybe unwrap self.visit(arg) - def wrap(self, val:Expr, elementType:ValueType): + def wrap(self, val: Expr, elementType: ValueType): self.instr("ldc.i4 1") self.instr(f"newarr {elementType.getCILName()}") self.instr("dup") diff --git a/compiler/closuretransformer.py b/compiler/closuretransformer.py index 7274a14..98bc906 100644 --- a/compiler/closuretransformer.py +++ b/compiler/closuretransformer.py @@ -3,11 +3,13 @@ from .astnodes import * from .types import * -def typeToAnnotation(t: ValueType)->SymbolType: + +def typeToAnnotation(t: ValueType) -> SymbolType: if isinstance(t, ListValueType): - return ListType([0,0], typeToAnnotation(t.elementType)) + return ListType([0, 0], typeToAnnotation(t.elementType)) elif isinstance(t, ClassValueType): - return ClassType([0,0], t.className) + return ClassType([0, 0], t.className) + class ClosureTransformer(TypeChecker): # rewriting function signatures to include free vars as explicit arguments @@ -28,10 +30,11 @@ def getSignature(self, node: FuncDef): t.refParams[i] = node.params[i].varInstance for i in range(len(node.freevars)): if node.freevars[i].varInstance.isNonlocal: - t.refParams[len(node.params) + i] = node.freevars[i].varInstance + t.refParams[len(node.params) + + i] = node.freevars[i].varInstance return t - def funcParams(self, node:FuncDef): + def funcParams(self, node: FuncDef): for fv in node.freevars: ident = fv.copy() annot = typeToAnnotation(ident.inferredType) @@ -63,5 +66,3 @@ def CallExpr(self, node: CallExpr): def MethodCallExpr(self, node: MethodCallExpr): self.callHelper(node) return super().MethodCallExpr(node) - - diff --git a/compiler/closurevisitor.py b/compiler/closurevisitor.py index 47686af..33e3f65 100644 --- a/compiler/closurevisitor.py +++ b/compiler/closurevisitor.py @@ -3,16 +3,19 @@ from .visitor import Visitor from .varcollector import VarCollector + class VarInstance: def __init__(self): self.isNonlocal = False self.isGlobal = False self.isSelf = False -def newInstance(tv: TypedVar)->VarInstance: + +def newInstance(tv: TypedVar) -> VarInstance: tv.varInstance = VarInstance() return tv.varInstance + def merge(d1, d2): combined = {} for k in d1: @@ -21,7 +24,8 @@ def merge(d1, d2): combined[k] = d2[k] return combined -def deduplicate(ids:[Identifier])->[Identifier]: + +def deduplicate(ids: [Identifier]) -> [Identifier]: seen = set() res = [] for i in ids: @@ -41,10 +45,10 @@ class ClosureVisitor(Visitor): def __init__(self): self.globals = {} - self.nonlocals = [] # uncaptured nonlocals + self.nonlocals = [] # uncaptured nonlocals self.decls = [] - def getInstance(self, name:str)->VarInstance: + def getInstance(self, name: str) -> VarInstance: for i in self.decls[::-1]: if name in i: return i[name] @@ -89,11 +93,12 @@ def FuncDef(self, node: FuncDef): elif isinstance(d, NonLocalDecl): varInstance = self.getInstance(d.variable.name) if varInstance.isSelf: - raise Exception("Special parameter 'self' may not be used in a nonlocal declaration") + raise Exception( + "Special parameter 'self' may not be used in a nonlocal declaration") varInstance.isNonlocal = True elif isinstance(d, VarDef): decls[d.getIdentifier().name] = newInstance(d.var) - + self.decls.append(decls) vars = VarCollector().getVarsFromList(node.statements) freevars = [] @@ -112,11 +117,6 @@ def FuncDef(self, node: FuncDef): node.freevars = deduplicate(freevars) # remove nonlocal decls node.declarations = [ - d for d in node.declarations + d for d in node.declarations if not isinstance(d, NonLocalDecl) ] - - - - - diff --git a/compiler/compiler.py b/compiler/compiler.py index 600a433..c5ceb3e 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -4,7 +4,7 @@ from .typechecker import TypeChecker from .parser import Parser, ParseError from .closurevisitor import ClosureVisitor -from .closuretransformer import ClosureTransformer +from .closuretransformer import ClosureTransformer from .nestedfunchoister import NestedFuncHoister from .typesystem import TypeSystem from .jvm_backend import JvmBackend @@ -13,6 +13,7 @@ import ast from pathlib import Path + class Compiler: def __init__(self): self.ts = TypeSystem() @@ -37,7 +38,8 @@ def parse(self, infile) -> Node: return astparser.visit(tree) except SyntaxError as e: e.filename = fname - message = "Syntax Error: {}. Line {:d} Col {:d}".format(str(e), e.lineno, e.offset) + message = "Syntax Error: {}. Line {:d} Col {:d}".format( + str(e), e.lineno, e.offset) astparser.errors.append(ParseError(message)) return None @@ -59,17 +61,16 @@ def emitPython(self, ast: Node): backend.visit(ast) return backend.builder - def emitJVM(self, main:str, ast: Node): + def emitJVM(self, main: str, ast: Node): self.closurepass(ast) EmptyListTyper().visit(ast) jvm_backend = JvmBackend(main, self.transformer.ts) jvm_backend.visit(ast) return jvm_backend.classes - def emitCIL(self, main:str, ast: Node): + def emitCIL(self, main: str, ast: Node): self.closurepass(ast) EmptyListTyper().visit(ast) cil_backend = CilBackend(main, self.transformer.ts) cil_backend.visit(ast) return cil_backend.builder - diff --git a/compiler/empty_list_typer.py b/compiler/empty_list_typer.py index 587d78a..6940c3b 100644 --- a/compiler/empty_list_typer.py +++ b/compiler/empty_list_typer.py @@ -3,6 +3,8 @@ from .visitor import Visitor # A visitor to refine the types of empty list literals + + class EmptyListTyper(Visitor): def __init__(self): @@ -20,8 +22,8 @@ def isEmptyListMultiAssign(self, node: Node): if len(node.value.elements) > 0: return False return True - - def transformMultiAssign(self, node:AssignStmt)->[AssignStmt]: + + def transformMultiAssign(self, node: AssignStmt) -> [AssignStmt]: statements = [] for t in node.targets: statements.append(AssignStmt(node.location, [t], node.value)) @@ -75,5 +77,3 @@ def MethodCallExpr(self, node: MethodCallExpr): def ReturnStmt(self, node: ReturnStmt): self.expectedType = self.expReturnType - - diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 3d34cb2..72e007d 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -38,7 +38,7 @@ def returnInstr(self, exprType: ValueType): else: self.instr("ireturn") - def wrap(self, val:Expr, elementType:ValueType): + def wrap(self, val: Expr, elementType: ValueType): self.loadInt(1) self.instr(f"anewarray {elementType.getJavaName(True)}") self.instr("dup") @@ -66,21 +66,24 @@ def load(self, name: str, t: ValueType): else: self.instr(f"iload {n}") - def arrayStore(self, elementType:ValueType): + def arrayStore(self, elementType: ValueType): # expect the stack to be array, idx, value if elementType == IntType(): - self.instr("invokestatic Method java/lang/Integer valueOf (I)Ljava/lang/Integer;") + self.instr( + "invokestatic Method java/lang/Integer valueOf (I)Ljava/lang/Integer;") elif elementType == BoolType(): - self.instr("invokestatic Method java/lang/Boolean valueOf (Z)Ljava/lang/Boolean;") + self.instr( + "invokestatic Method java/lang/Boolean valueOf (Z)Ljava/lang/Boolean;") self.instr("aastore") - def arrayLoad(self, elementType:ValueType): + def arrayLoad(self, elementType: ValueType): # expect the stack to be array, idx self.instr("aaload") if elementType == IntType(): self.instr("invokevirtual Method java/lang/Integer intValue ()I") elif elementType == BoolType(): - self.instr("invokevirtual Method java/lang/Boolean booleanValue ()Z") + self.instr( + "invokevirtual Method java/lang/Boolean booleanValue ()Z") def newLocalEntry(self, name: str) -> int: # add a new entry to locals table w/o storing anything @@ -103,7 +106,7 @@ def newLocal(self, name: str = None, isRef: bool = True) -> int: self.locals[-1][name] = n return n - def visitStmtList(self, stmts:[Stmt]): + def visitStmtList(self, stmts: [Stmt]): if len(stmts) == 0: self.instr("nop") else: @@ -126,7 +129,8 @@ def Program(self, node: Program): # main self.instr(".method public static main : ([Ljava/lang/String;)V") self.currentBuilder().indent() - self.instr(f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") + self.instr( + f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") self.defaultToGlobals = True self.visitStmtList(node.statements) self.defaultToGlobals = False @@ -203,13 +207,14 @@ def funcDefHelper(self, node: FuncDef): self.buildReturn(None) self.exitScope() - def constructor(self, superclass:str, node: FuncDef): + def constructor(self, superclass: str, node: FuncDef): self.enterScope() constructorSig = node.type.dropFirstParam() self.instr( f".method public : {constructorSig.getJavaSignature()}") self.currentBuilder().indent() - self.instr(f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") + self.instr( + f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") # call superclass constructor self.instr("aload 0") self.instr(f"invokespecial Method {superclass} ()V ") @@ -224,7 +229,8 @@ def method(self, node: FuncDef): self.instr( f".method public {node.name.name} : {methodSig.getJavaSignature()}") self.currentBuilder().indent() - self.instr(f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") + self.instr( + f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") self.funcDefHelper(node) self.instr(".end code") self.currentBuilder().unindent() @@ -235,7 +241,8 @@ def FuncDef(self, node: FuncDef): self.instr( f".method public static {node.name.name} : {node.type.getJavaSignature()}") self.currentBuilder().indent() - self.instr(f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") + self.instr( + f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") self.funcDefHelper(node) self.instr(".end code") self.currentBuilder().unindent() @@ -367,7 +374,8 @@ def BinaryExpr(self, node: BinaryExpr): self.instr(f"iload {lenR}") self.instr("iadd") # stack is L, total_length - self.instr(f"anewarray {self.ts.join(leftType, rightType).elementType.getJavaName(True)}") + self.instr( + f"anewarray {self.ts.join(leftType, rightType).elementType.getJavaName(True)}") newArr = self.newLocal(None, True) self.instr("iconst_0") self.instr(f"aload {newArr}") @@ -625,7 +633,7 @@ def BooleanLiteral(self, node: BooleanLiteral): else: self.instr("iconst_0") - def loadInt(self, value:int): + def loadInt(self, value: int): if value >= 0 and value <= 5: self.instr(f"iconst_{value}") else: @@ -705,12 +713,12 @@ def emit_print(self, arg: Expr): def visitArg(self, funcType, paramIdx: int, arg: Expr): argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg self.load(arg.name, ListValueType(arg.inferredType)) elif paramIsRef: # non-ref arg and ref param, or do not pass ref arg # unwrap if necessary, re-wrap self.wrap(arg, arg.inferredType) - else: # non-ref param, maybe unwrap + else: # non-ref param, maybe unwrap self.visit(arg) diff --git a/compiler/nestedfunchoister.py b/compiler/nestedfunchoister.py index 130ff43..be341f9 100644 --- a/compiler/nestedfunchoister.py +++ b/compiler/nestedfunchoister.py @@ -2,11 +2,13 @@ from .types import * from .visitor import Visitor + class HoistedFunctionInfo: def __init__(self, name, decl): self.name = name self.decl = decl + class NestedFuncHoister(Visitor): # hoist all nested funcs to be top level funcs # rename hoisted functions to be unique & rename call sites @@ -26,8 +28,8 @@ def visit(self, node: Node): else: return node.visit(self) - def genFuncName(self, name:str): - # example: + def genFuncName(self, name: str): + # example: # f2 declared inside f1 will be named f1__f2 # f4 declared inside C.f3 will be named C__f3__f4 if len(self.nestingNames) == 0: @@ -62,12 +64,13 @@ def ClassDef(self, node: ClassDef): self.nestingNames.pop() self.currentClass = None - def rename(self, node:FuncDef): + def rename(self, node: FuncDef): identifier = node.getIdentifier() oldname = identifier.name if self.nestingLevel != 0: identifier.name = self.genFuncName(identifier.name) - self.functionInfo[-1][oldname] = HoistedFunctionInfo(identifier.name, node) + self.functionInfo[-1][oldname] = HoistedFunctionInfo( + identifier.name, node) def FuncDef(self, node: FuncDef): identifier = node.getIdentifier() @@ -82,14 +85,14 @@ def FuncDef(self, node: FuncDef): for s in node.statements: self.visit(s) - + self.functionInfo.pop() self.nestingNames.pop() self.nestingLevel -= 1 if self.nestingLevel > 0: self.hoisted.append(node) - + node.declarations = [ d for d in node.declarations if not isinstance(d, FuncDef) @@ -106,6 +109,5 @@ def CallExpr(self, node: CallExpr): return if node.function.name in self.classes: return - raise Exception("Unable to find function declaration for " + node.function.name) - - + raise Exception( + "Unable to find function declaration for " + node.function.name) diff --git a/compiler/parser.py b/compiler/parser.py index 3b3a280..2ef3a87 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -127,10 +127,12 @@ def visit_ClassDef(self, node): location = self.getLocation(node) identifier = Identifier([location[0], location[1] + 6], node.name) if len(node.bases) > 1: - raise ParseError("Multiple inheritance is unsupported", node.bases[1]) + raise ParseError( + "Multiple inheritance is unsupported", node.bases[1]) base = None if len(node.bases) == 0: - base = Identifier([location[0], location[1] + 7 + len(node.name)], "object") + base = Identifier([location[0], location[1] + + 7 + len(node.name)], "object") else: base = self.visit(node.bases[0]) if node.keywords: diff --git a/compiler/python_backend.py b/compiler/python_backend.py index 330db02..4b42db2 100644 --- a/compiler/python_backend.py +++ b/compiler/python_backend.py @@ -4,14 +4,15 @@ from .visitor import Visitor import json + class PythonBackend(Visitor): def __init__(self): self.builder = Builder(None) def visit(self, node: Node): return node.visit(self) - - def addText(self, text:str): + + def addText(self, text: str): self.builder.addText(text) # TOP LEVEL & DECLARATIONS @@ -142,15 +143,15 @@ def visitArg(self, node, funcType, paramIdx: int, argIdx: int): return argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg self.addText(arg.name) - elif paramIsRef: + elif paramIsRef: # non-ref arg and ref param, or do not pass ref arg self.addText("[") self.visit(arg) self.addText("]") - else: # non-ref param, maybe unwrap + else: # non-ref param, maybe unwrap self.visit(arg) def CallExpr(self, node: CallExpr): @@ -220,7 +221,6 @@ def Identifier(self, node: Identifier): self.addText(node.name + "[0]") else: self.addText(node.name) - def MemberExpr(self, node: MemberExpr): self.visit(node.object) diff --git a/compiler/typechecker.py b/compiler/typechecker.py index a5113c6..a9658cf 100644 --- a/compiler/typechecker.py +++ b/compiler/typechecker.py @@ -4,6 +4,7 @@ from .typesystem import TypeSystem, ClassInfo from .visitor import Visitor + class TypeChecker(Visitor): def __init__(self, ts: TypeSystem): # typechecker attributes and their chocopy typing judgement analogues: @@ -37,7 +38,7 @@ def visit(self, node: Node): else: return node.postorder(self) - def funcParams(self, node:FuncDef): + def funcParams(self, node: FuncDef): pass def enterScope(self): @@ -164,7 +165,7 @@ def ClassDef(self, node: ClassDef): continue if not t.methodEquals(funcType): # excluding self argument self.addError(d.getIdentifier(), - F"Redefined method doesn't match superclass signature: {funcName}") + F"Redefined method doesn't match superclass signature: {funcName}") continue self.ts.classes[className].methods[funcName] = funcType if isinstance(d, VarDef): # attributes diff --git a/compiler/typeeraser.py b/compiler/typeeraser.py index b9ced38..a396fbd 100644 --- a/compiler/typeeraser.py +++ b/compiler/typeeraser.py @@ -3,6 +3,8 @@ from .visitor import Visitor # A utility visitor to erase the inferred types of expressions + + class TypeEraser(Visitor): def visit(self, node: Node): @@ -37,4 +39,3 @@ def NonLocalDecl(self, node: NonLocalDecl): def GlobalDecl(self, node: GlobalDecl): self.visit(node.variable) - diff --git a/compiler/types/Types.py b/compiler/types/Types.py index d0df0dd..bc311e0 100644 --- a/compiler/types/Types.py +++ b/compiler/types/Types.py @@ -2,20 +2,26 @@ # factories for types + def ObjectType(): return ClassValueType("object") + def IntType(): return ClassValueType("int") + def StrType(): return ClassValueType("str") + def BoolType(): return ClassValueType("bool") + def NoneType(): return ClassValueType("") + def EmptyType(): return ClassValueType("") diff --git a/compiler/types/__init__.py b/compiler/types/__init__.py index 3b3b53d..b9f7afe 100644 --- a/compiler/types/__init__.py +++ b/compiler/types/__init__.py @@ -3,4 +3,4 @@ from .listvaluetype import ListValueType from .symboltype import SymbolType from .valuetype import ValueType -from .Types import * \ No newline at end of file +from .Types import * diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 693c13c..8927cb5 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -1,7 +1,8 @@ from .valuetype import ValueType + class ClassValueType(ValueType): - def __init__(self, className:str): + def __init__(self, className: str): self.className = className def __eq__(self, other): @@ -9,10 +10,10 @@ def __eq__(self, other): return self.className == other.className return False - def isListType(self)->bool: + def isListType(self) -> bool: return self.className in {"", ""} - def getJavaSignature(self, isList = False)->str: + def getJavaSignature(self, isList=False) -> str: if self.className == "bool": if isList: return "Ljava/lang/Boolean;" @@ -43,7 +44,7 @@ def isSpecialType(self): def isJavaRef(self): return self.className not in ["int", "bool"] - def getJavaName(self, isList = False): + def getJavaName(self, isList=False): if self.className == "bool": if isList: return "java/lang/Boolean" @@ -86,7 +87,7 @@ def getCILName(self): return "int64" else: return "class "+self.className - + def __str__(self): return self.className diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 987f424..af3b524 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -2,12 +2,13 @@ from .valuetype import ValueType from .symboltype import SymbolType + class FuncType(SymbolType): - def __init__(self, parameters:[ValueType], returnType:ValueType): + def __init__(self, parameters: [ValueType], returnType: ValueType): self.parameters = parameters self.returnType = returnType self.refParams = {} - self.freevars = [] # used in AST transformations, not printed out + self.freevars = [] # used in AST transformations, not printed out def __eq__(self, other): if isinstance(other, FuncType): @@ -20,7 +21,7 @@ def dropFirstParam(self): f.freevars = self.freevars return f - def getCILSignature(self, name: str)->str: + def getCILSignature(self, name: str) -> str: params = [] for i in range(len(self.parameters)): p = self.parameters[i] @@ -32,7 +33,7 @@ def getCILSignature(self, name: str)->str: paramSig = ", ".join(params) return f"{self.returnType.getCILSignature()} {name}({paramSig})" - def getJavaSignature(self)->str: + def getJavaSignature(self) -> str: r = None if self.returnType.isNone(): r = "V" @@ -59,7 +60,7 @@ def isFuncType(): def __str__(self): paramStr = ",".join([str(t) for t in self.parameters]) return F"[{paramStr}]->{self.returnType}" - + def __hash__(self): paramStr = ",".join([str(t) for t in self.parameters]) return (F"[{paramStr}]->{self.returnType}").__hash__() @@ -69,4 +70,4 @@ def toJSON(self, dump_location=True): "kind": "FuncType", "parameters": [p.toJSON(dump_location) for p in self.parameters], "returnType": self.returnType.toJSON(dump_location) - } \ No newline at end of file + } diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 6c77b57..624f1e0 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -1,8 +1,9 @@ from .valuetype import ValueType + class ListValueType(ValueType): - - def __init__(self, elementType:ValueType): + + def __init__(self, elementType: ValueType): self.elementType = elementType def __eq__(self, other): @@ -38,4 +39,4 @@ def toJSON(self, dump_location=True): return { "kind": "ListValueType", "elementType": self.elementType.toJSON(dump_location) - } \ No newline at end of file + } diff --git a/compiler/types/symboltype.py b/compiler/types/symboltype.py index 36cf8dd..61f82f6 100644 --- a/compiler/types/symboltype.py +++ b/compiler/types/symboltype.py @@ -1,12 +1,12 @@ class SymbolType: # base class for types - + def isValueType(): return False def isListType(): return False - + def isFuncType(): return False @@ -20,4 +20,4 @@ def toJSON(self, dump_location=True): raise Exception("unsupported") def llvmType(self, typeSystem): - raise Exception("unsupported") \ No newline at end of file + raise Exception("unsupported") diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index 8903064..09ea240 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -1,5 +1,6 @@ from .symboltype import SymbolType + class ValueType(SymbolType): def isValueType(): return True @@ -10,10 +11,10 @@ def isNone(self): def toJSON(self, dump_location=True): raise Exception("unsupported") - def getJavaSignature(self)->str: + def getJavaSignature(self) -> str: raise Exception("unsupported") - def isJavaRef(self)->bool: + def isJavaRef(self) -> bool: raise Exception("unsupported") def isListType(self): diff --git a/compiler/typesystem.py b/compiler/typesystem.py index 8c0519c..fe291b9 100644 --- a/compiler/typesystem.py +++ b/compiler/typesystem.py @@ -1,17 +1,19 @@ from .types import * from collections import defaultdict + class ClassInfo: - def __init__(self, name:str, superclass:str = None): + def __init__(self, name: str, superclass: str = None): self.name = name self.superclass = superclass - self.attrs = defaultdict(lambda: None) # (attr type, init value) - self.methods = defaultdict(lambda: None) # type of method + self.attrs = defaultdict(lambda: None) # (attr type, init value) + self.methods = defaultdict(lambda: None) # type of method self.orderedAttrs = [] - + def __str__(self): return F"class {self.name}({self.superclass}): {self.attrs} {self.methods}" + class TypeSystem: def __init__(self): # information for each class @@ -19,7 +21,7 @@ def __init__(self): objectInfo = ClassInfo("object") objectInfo.methods["__init__"] = FuncType([ObjectType()], NoneType()) - + intInfo = ClassInfo("int", "object") intInfo.methods["__init__"] = FuncType([ObjectType()], NoneType()) @@ -151,7 +153,7 @@ def join(self, a: ValueType, b: ValueType): return ObjectType() def getAllMethods(self, className: str): - # return map of method names to tuples of + # return map of method names to tuples of # (signature, classname of their definition) methods = {} if self.classes[className].superclass is not None: @@ -160,7 +162,7 @@ def getAllMethods(self, className: str): methods[name] = (self.classes[className].methods[name], className) return methods - def getOrderedAttrs(self, className:str): + def getOrderedAttrs(self, className: str): # return list of (name, type, init value) triples attrs = [] if self.classes[className].superclass is not None: @@ -168,4 +170,4 @@ def getOrderedAttrs(self, className:str): for attr in self.classes[className].orderedAttrs: attrType, attrInit = self.classes[className].attrs[attr] attrs.append((attr, attrType, attrInit)) - return attrs \ No newline at end of file + return attrs diff --git a/compiler/varcollector.py b/compiler/varcollector.py index f493066..4f001be 100644 --- a/compiler/varcollector.py +++ b/compiler/varcollector.py @@ -3,6 +3,7 @@ from .types import * from .visitor import Visitor + class VarCollector(Visitor): # simple visitor to collect all the identifiers used as expressions or assignment targets @@ -17,9 +18,9 @@ def getVarsFromList(self, nodes: [Node]): for n in nodes: self.visit(n) return self.vars - + def visit(self, node: Node): return node.postorder(self) def Identifier(self, node: Identifier): - self.vars.append(node) \ No newline at end of file + self.vars.append(node) diff --git a/compiler/visitor.py b/compiler/visitor.py index 4627d09..84ee370 100644 --- a/compiler/visitor.py +++ b/compiler/visitor.py @@ -2,6 +2,7 @@ from collections import defaultdict from .builder import Builder + class Visitor: def visit(self, node: Node): @@ -99,8 +100,9 @@ def ListType(self, node: ListType): def ClassType(self, node: ClassType): pass + class CommonVisitor(Visitor): - returnType = None # for tracking return types in functions + returnType = None # for tracking return types in functions counter = 0 # for labels # helpers for handling locals @@ -117,9 +119,9 @@ def exitScope(self): def instr(self, instr: str): self.currentBuilder().newLine(instr) - - def currentBuilder(self)->Builder: + + def currentBuilder(self) -> Builder: raise Exception("unimplemented") def emit(self) -> str: - return self.currentBuilder().emit() \ No newline at end of file + return self.currentBuilder().emit() From 1edaf58be4cbea73a3c2978b19d9c580486344d4 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 18 May 2022 15:16:37 -0400 Subject: [PATCH 18/79] fix bug --- compiler/jvm_backend.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 72e007d..051d856 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -8,12 +8,12 @@ class JvmBackend(CommonVisitor): - classes = dict() localLimit = 50 stackLimit = 500 defaultToGlobals = False # treat all vars as global if this is true def __init__(self, main: str, ts: TypeSystem): + self.classes = dict() self.classes[main] = Builder(main) self.currentClass = main self.main = main # name of main class From e1ce617558a0dfea4cdf7450961c587edb7d46e3 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 22 May 2022 18:09:13 -0700 Subject: [PATCH 19/79] properly implement nonlocals in CIL backend --- compiler/cil_backend.py | 99 ++++-- compiler/types/functype.py | 2 +- nonlocal.cil | 545 +++++++++++++++++++++++++++++ nonlocal.exe | Bin 0 -> 5120 bytes tests/runtime/nonlocal_builtins.py | 12 + 5 files changed, 631 insertions(+), 27 deletions(-) create mode 100644 nonlocal.cil create mode 100644 nonlocal.exe create mode 100644 tests/runtime/nonlocal_builtins.py diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 49ad597..abb44f3 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -68,6 +68,41 @@ def load(self, name: str): else: self.instr(f"ldloc {n.loc}") + def loadVarAddr(self, node: Identifier): + if self.defaultToGlobals or node.varInstance.isGlobal: + self.instr( + f"ldsflda {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") + elif self.isFromRefArg(node): + self.load(node.getCILName()) + else: + self.loadAddr(node.getCILName()) + + def loadAddr(self, name: str): + n = self.locals[-1][name] + if n is None: + raise Exception( + f"Internal compiler error: unknown name {name} for load") + if n.isArg: + self.instr(f"ldarga {n.loc}") + else: + self.instr(f"ldloca {n.loc}") + + def loadInd(self, t: ValueType): + if t == BoolType(): + self.instr("ldind.i4") + elif t == IntType(): + self.instr("ldind.i8") + else: + self.instr("ldind.ref") + + def storeInd(self, t: ValueType): + if t == BoolType(): + self.instr("stind.i4") + elif t == IntType(): + self.instr("stind.i8") + else: + self.instr("stind.ref") + def arrayStore(self, elementType: ValueType): self.instr(f"stelem {elementType.getCILName()}") @@ -236,10 +271,6 @@ def VarDef(self, node: VarDef): self.visit(node.value) self.instr( f"stfld {node.var.t.getCILName()} {className.getCILName()}::{node.getIdentifier().getCILName()}") - elif node.var.varInstance.isNonlocal: - elementType = node.var.t - self.wrap(node.value, elementType) - self.newLocal(varName, ListValueType(elementType)) else: self.visit(node.value) self.newLocal(varName, node.var.t) @@ -253,10 +284,9 @@ def processAssignmentTarget(self, target: Expr): f"stsfld {target.inferredType.getCILName()} {self.main}::{target.getCILName()}") elif target.varInstance.isNonlocal: temp = self.newLocal(None, target.inferredType) - self.load(target.name) - self.instr("ldc.i4 0") + self.load(target.getCILName()) self.load(temp) - self.arrayStore(target.inferredType) + self.storeInd(target.inferredType) else: self.store(target.getCILName()) elif isinstance(target, IndexExpr): @@ -539,10 +569,9 @@ def Identifier(self, node: Identifier): if self.defaultToGlobals or node.varInstance.isGlobal: self.instr( f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") - elif node.varInstance.isNonlocal: - self.load(node.name) - self.instr("ldc.i4 0") - self.arrayLoad(node.inferredType) + elif self.isFromRefArg(node): + self.load(node.getCILName()) + self.loadInd(node.inferredType) else: self.load(node.getCILName()) @@ -643,23 +672,41 @@ def emit_print(self, arg: Expr): f"call void class [mscorlib]System.Console::WriteLine({arg.inferredType.getCILName()})") self.NoneLiteral(None) + def isFromRefArg(self, arg: Expr): + return self.isFromArg(arg) and arg.varInstance.isNonlocal + + def isFromArg(self, arg: Expr): + if not isinstance(arg, Identifier): + return False + if arg.varInstance.isGlobal: + return True + n = self.locals[-1][arg.name] + if n is None: + raise Exception( + f"Internal compiler error: unknown name {arg.name}") + return n.isArg + def visitArg(self, funcType, paramIdx: int, arg: Expr): - argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + argIsRef = self.isFromRefArg(arg) paramIsRef = paramIdx in funcType.refParams if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: - # ref arg and ref param, pass ref arg + # ref -> ref: pass through a ref to a nonlocal self.load(arg.name) - elif paramIsRef: - # non-ref arg and ref param, or do not pass ref arg - # unwrap if necessary, re-wrap - self.wrap(arg, arg.inferredType) - else: # non-ref param, maybe unwrap + elif paramIsRef and argIsRef: + # ref -> ref: + # deref, store value in new local, and pass ref to new local self.visit(arg) - - def wrap(self, val: Expr, elementType: ValueType): - self.instr("ldc.i4 1") - self.instr(f"newarr {elementType.getCILName()}") - self.instr("dup") - self.instr("ldc.i4 0") - self.visit(val) - self.arrayStore(elementType) + temp = self.newLocal(None, arg.inferredType) + self.loadAddr(temp) + elif paramIsRef: + # value -> ref + # store in new local, pass ref + if isinstance(arg, Identifier) and not self.isFromArg(arg): + self.loadVarAddr(arg) + else: + self.visit(arg) + temp = self.newLocal(None, arg.inferredType) + self.loadAddr(temp) + else: + # value/ref -> value : deref if necessary + self.visit(arg) \ No newline at end of file diff --git a/compiler/types/functype.py b/compiler/types/functype.py index af3b524..1b921c8 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -26,7 +26,7 @@ def getCILSignature(self, name: str) -> str: for i in range(len(self.parameters)): p = self.parameters[i] if i in self.refParams and isinstance(p, ClassValueType): - sig = p.getCILSignature() + "[]" + sig = p.getCILSignature() + "&" else: sig = p.getCILSignature() params.append(sig) diff --git a/nonlocal.cil b/nonlocal.cil new file mode 100644 index 0000000..45616e8 --- /dev/null +++ b/nonlocal.cil @@ -0,0 +1,545 @@ +.assembly 'nonlocal' +{ +} +.module nonlocal.exe +.class public auto ansi beforefieldinit nonlocal extends [mscorlib]System.Object +{ + .field public static int64 a + .field public static class Nonlocals b + .method public static hidebysig default void Main (string[] args) cil managed + { + .entrypoint + .maxstack 500 + .locals init ( + [0] int64 __local__0, + [1] int64 __local__1, + [2] int64 __local__2, + [3] int64 __local__3, + [4] int64 __local__4 + ) + ldc.i8 0 + stsfld int64 nonlocal::a + ldnull + stsfld class Nonlocals nonlocal::b + ldc.i8 1 + stloc 0 + ldloca 0 + call int64 nonlocal::test(int64&) + ldc.i8 2 + ceq + brtrue IL_1 + ldstr "failed assertion on line 84" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_1: nop + ldnull + pop + call int64 nonlocal::test3() + ldc.i8 3 + ceq + brtrue IL_2 + ldstr "failed assertion on line 86" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_2: nop + ldnull + pop + ldc.i8 0 + stsfld int64 nonlocal::a + ldsfld int64 nonlocal::a + stloc 1 + ldloca 1 + call void nonlocal::test9(int64&) + ldnull + pop + ldsfld int64 nonlocal::a + ldc.i8 0 + ceq + brtrue IL_3 + ldstr "failed assertion on line 92" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_3: nop + ldnull + pop + call void nonlocal::test7() + ldnull + pop + call void nonlocal::test10() + ldnull + pop + ldc.i8 0 + stsfld int64 nonlocal::a + newobj instance void Nonlocals::.ctor() + stsfld class Nonlocals nonlocal::b + ldsfld class Nonlocals nonlocal::b + ldsfld int64 nonlocal::a + stloc 2 + ldloca 2 + callvirt instance void Nonlocals::testMethod(int64&) + ldnull + pop + ldsfld class Nonlocals nonlocal::b + ldc.i8 0 + stloc 3 + ldloca 3 + callvirt instance void Nonlocals::testMethod(int64&) + ldnull + pop + ldsfld int64 nonlocal::a + ldc.i8 0 + ceq + brtrue IL_4 + ldstr "failed assertion on line 103" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_4: nop + ldnull + pop + ldsfld class Nonlocals nonlocal::b + ldc.i8 1 + stloc 4 + ldloca 4 + callvirt instance void Nonlocals::testMethod(int64&) + ldnull + pop + ldsfld class Nonlocals nonlocal::b + callvirt instance void Nonlocals::testMethod4() + ldnull + pop + ret + } + .method public hidebysig static + int64 test(int64&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + call void nonlocal::test__test2(int64&) + ldnull + pop + ldarg 0 + ldind.i8 + ret + } + .method public hidebysig static + int64 test3() cil managed + { + .maxstack 500 + .locals init ( + [0] int64 x + ) + ldc.i8 4 + stloc 0 + ldloca 0 + call void nonlocal::test3__test4(int64&) + ldnull + pop + ldloc 0 + ret + } + .method public hidebysig static + void test7() cil managed + { + .maxstack 500 + .locals init ( + [0] int64[] x + ) + ldnull + stloc 0 + ldc.i4 3 + newarr int64 + dup + ldc.i4 0 + ldc.i8 1 + stelem int64 + dup + ldc.i4 1 + ldc.i8 2 + stelem int64 + dup + ldc.i4 2 + ldc.i8 3 + stelem int64 + stloc 0 + ldloc 0 + call void nonlocal::test7__test8(int64[]) + ldnull + pop + ldloc 0 + ldc.i8 0 + conv.i4 + ldelem int64 + ldc.i8 0 + ceq + brtrue IL_5 + ldstr "failed assertion on line 33" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_5: nop + ldnull + pop + ret + } + .method public hidebysig static + void test9(int64&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + call void nonlocal::test9__test9helper(int64&) + ldnull + pop + ret + } + .method public hidebysig static + void test10() cil managed + { + .maxstack 500 + .locals init ( + [0] int64 y + ) + ldc.i8 1 + stloc 0 + ldloc 0 + ldloca 0 + call int64 nonlocal::test10__test11(int64, int64&) + ldc.i8 2 + ceq + brtrue IL_6 + ldstr "failed assertion on line 52" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_6: nop + ldnull + pop + ldloca 0 + call int64 nonlocal::test10__test12(int64&) + ldc.i8 4 + ceq + brtrue IL_7 + ldstr "failed assertion on line 53" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_7: nop + ldnull + pop + ldloc 0 + ldc.i8 2 + ceq + brtrue IL_8 + ldstr "failed assertion on line 54" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_8: nop + ldnull + pop + ret + } + .method public hidebysig static + void test13(class Nonlocals&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + call void nonlocal::test13__test14(class Nonlocals&) + ldnull + pop + ret + } + .method public hidebysig static + void test__test2(int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + ldc.i8 2 + stloc 0 + ldarg 0 + ldloc 0 + stind.i8 + ret + } + .method public hidebysig static + void test3__test3__test4__test5(int64&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + ldind.i8 + ldc.i8 4 + ceq + brtrue IL_9 + ldstr "failed assertion on line 15" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_9: nop + ldnull + pop + ret + } + .method public hidebysig static + void test3__test3__test4__test6(int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + ldc.i8 3 + stloc 0 + ldarg 0 + ldloc 0 + stind.i8 + ret + } + .method public hidebysig static + void test3__test4(int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + ldarg 0 + call void nonlocal::test3__test3__test4__test5(int64&) + ldnull + pop + ldarg 0 + ldind.i8 + stloc 0 + ldloca 0 + call int64 nonlocal::test(int64&) + pop + ldarg 0 + ldind.i8 + ldc.i8 4 + ceq + brtrue IL_10 + ldstr "failed assertion on line 21" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_10: nop + ldnull + pop + ldarg 0 + call void nonlocal::test3__test3__test4__test6(int64&) + ldnull + pop + ldarg 0 + ldind.i8 + ldc.i8 3 + ceq + brtrue IL_11 + ldstr "failed assertion on line 23" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_11: nop + ldnull + pop + ret + } + .method public hidebysig static + void test7__test8(int64[]) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + ldc.i8 0 + stloc 0 + ldarg 0 + ldc.i8 0 + ldloc 0 + stelem int64 + ret + } + .method public hidebysig static + void test9__test9helper(int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + ldc.i8 0 + stloc 0 + ldarg 0 + ldloc 0 + stind.i8 + ret + } + .method public hidebysig static + int64 test10__test11(int64, int64&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + ldarg 1 + ldind.i8 + add.ovf + ret + } + .method public hidebysig static + int64 test10__test10__test12__test13(int64, int64&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + ldarg 1 + ldind.i8 + add.ovf + ret + } + .method public hidebysig static + int64 test10__test12(int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + ldarg 0 + ldind.i8 + ldarg 0 + call int64 nonlocal::test10__test10__test12__test13(int64, int64&) + stloc 0 + ldarg 0 + ldloc 0 + stind.i8 + ldarg 0 + ldind.i8 + ldc.i8 2 + ceq + brtrue IL_12 + ldstr "failed assertion on line 50" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_12: nop + ldnull + pop + ldarg 0 + ldind.i8 + ldarg 0 + call int64 nonlocal::test10__test10__test12__test13(int64, int64&) + ret + } + .method public hidebysig static + void Nonlocals__testMethod__testMethod2(class Nonlocals, int64&, int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0, + [1] int64 __local__1 + ) + ldarg 0 + callvirt instance void Nonlocals::testMethod3() + ldnull + pop + ldc.i8 3 + stloc 0 + ldarg 1 + ldloc 0 + stind.i8 + ldc.i8 3 + stloc 1 + ldarg 2 + ldloc 1 + stind.i8 + ret + } + .method public hidebysig static + void test13__test14(class Nonlocals&) cil managed + { + .maxstack 500 + .locals init ( + [0] class Nonlocals __local__0 + ) + ldnull + stloc 0 + ldarg 0 + ldloc 0 + stind.ref + ret + } +} +.class public auto ansi beforefieldinit Nonlocals extends object +{ + .method public hidebysig virtual instance + void testMethod3() cil managed + { + .maxstack 500 + .locals init ( + ) + nop + ret + } + .method public hidebysig virtual instance + void testMethod(int64&) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 y + ) + ldc.i8 2 + stloc 0 + ldarg 0 + ldarg 1 + ldloca 0 + call void nonlocal::Nonlocals__testMethod__testMethod2(class Nonlocals, int64&, int64&) + ldnull + pop + ldloc 0 + ldc.i8 3 + ceq + brtrue IL_13 + ldstr "failed assertion on line 69" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_13: nop + ldnull + pop + ret + } + .method public hidebysig virtual instance + void testMethod4() cil managed + { + .maxstack 500 + .locals init ( + [0] class Nonlocals __local__0 + ) + ldarg 0 + stloc 0 + ldloca 0 + call void nonlocal::test13(class Nonlocals&) + ldnull + pop + ldarg 0 + ldnull + ceq + ldc.i4.0 + ceq + brtrue IL_14 + ldstr "failed assertion on line 73" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_14: nop + ldnull + pop + ret + } + .method public hidebysig specialname rtspecialname instance + void .ctor() cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg.0 + call instance void object::.ctor() + nop + ret + } +} \ No newline at end of file diff --git a/nonlocal.exe b/nonlocal.exe new file mode 100644 index 0000000000000000000000000000000000000000..f1dcdaf4b652830c71daedfa2c5ed2ee6374be28 GIT binary patch literal 5120 zcmd^DO>7%Q6n#a=4A zXUcPnc7Co@a;tXEvCC&mcA;dCADgxp-FYX~(P8a3rl$@OO{f43pZ_H1d0V51+Nr`s zQE&p9TT9_>`<%c8YG^V90{avF*asFtS8r4dj-VYn)dM z#>S?{dx#bQTw#p|McCTd-FH{L2}S&iB|Oy&phEErupdfbDWeu4s&x=0!3`(aHIJHl zb|+qCNG(+`S?^Q4Cy7Fa6S|spkx>?bo`@tw8UtJ!{O%~1xAE5G`45U z>v6GVX*X=+BKv}kIP2P6gU=Aj-UBvPm((nL){MrdmP{3ajSPcq4F@>va z$tS%s{mywpbtF%{jO#;~MOZN(f@f8F3O%&3r}iosL2bo|hyg^no*Yw?X%<5^wHQSF zh&(i(vIJ77(db-13OoQ9s|#~~BSKE{Os~{q2{P~>lN{sRbMRa*zbwdAShF^u`hk6- zE*cfIlDNz~1Acxzg!fn)Nze#6U;E)3LDkQsIE9^ht^ri)Bl+cc5aLF86A$+vpHv>==_(9gSAK0fzF;XCS`5Hxfbe{?! z*mt2e;(|`U{DdX2zF-cbfQ^^6)~u${|8qhbX<6#x|l7 zoPPg7#I#hrraC?CJFE>^qpj~AKGrf&n6c?s#}wh8-uH$(>{ZzMD|1E~PrHq;5~f?nuJy>|*nE=c# zP``j`r!_;Z!Ab{34`}cAKt*W4P@QTBxtuW6pgKx1x+K&`kDXMr$n~mFHz}v`pza!K z5qo=RpoeYVq;o=XhNnP#wcx$Y*Id#6lF=3#CH0y+LhHupq$&XS|2fBDZ5DNToSwo{ z|BPdO7|%E&(+C8HKXFg4AGRh-ZV5m5=87q2*&&ah=O*3xvqk3xTCC*Va=|0U%r`MC(R9ZVs8rtZD3IJAGkHN-n8ej-+0c{ zqt~2Cr@G+IH?I3>j&hXBS6z)-ONd#$)WGyQBZU|~4V?n+E{EpH#=3%!L(m7;r4o30 zE`ly$$EFb)AZki)s42Zs>L(iAI?goi4g}c*(oBBaP~Ki z6W=>c_)F7ZQ+h*$ujD5D4Rd_-(;c#}oL@ci`SRv>Hz!@JspHdJg#Ss~2k z!n~c**Cfw>@!ICq#^L?HmT2~%TRybx$ZaUMr{kpNi~O4efAKg#c2-u~cCyXi7GB)8 J8Q9i={{u;;aWwz{ literal 0 HcmV?d00001 diff --git a/tests/runtime/nonlocal_builtins.py b/tests/runtime/nonlocal_builtins.py new file mode 100644 index 0000000..3e383fa --- /dev/null +++ b/tests/runtime/nonlocal_builtins.py @@ -0,0 +1,12 @@ + + +def f(): + x: bool = True + y: str = "a" + def g(): + nonlocal x + nonlocal y + print(y) + __assert__(x) + g() +f() \ No newline at end of file From 3dc1bd325fdc4a777bc27b8fb6ea7656c970152e Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 22 May 2022 18:21:46 -0700 Subject: [PATCH 20/79] fix bug --- compiler/cil_backend.py | 20 +- exponent.cil | 177 +++++++++++++ exponent.exe | Bin 0 -> 3072 bytes nonlocal.cil | 545 ---------------------------------------- nonlocal.exe | Bin 5120 -> 0 bytes 5 files changed, 187 insertions(+), 555 deletions(-) create mode 100644 exponent.cil create mode 100644 exponent.exe delete mode 100644 nonlocal.cil delete mode 100644 nonlocal.exe diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index abb44f3..f679d26 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -73,9 +73,9 @@ def loadVarAddr(self, node: Identifier): self.instr( f"ldsflda {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") elif self.isFromRefArg(node): - self.load(node.getCILName()) + self.load(node.name) else: - self.loadAddr(node.getCILName()) + self.loadAddr(node.name) def loadAddr(self, name: str): n = self.locals[-1][name] @@ -242,7 +242,7 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None locals = self.builder.newBlock() for i in range(len(node.params)): self.newLocalEntry( - node.params[i].identifier.getCILName(), node.params[i].t, True) + node.params[i].identifier.name, node.params[i].t, True) for d in node.declarations: self.visit(d) self.returnType = node.type.returnType @@ -273,7 +273,7 @@ def VarDef(self, node: VarDef): f"stfld {node.var.t.getCILName()} {className.getCILName()}::{node.getIdentifier().getCILName()}") else: self.visit(node.value) - self.newLocal(varName, node.var.t) + self.newLocal(node.getIdentifier().name, node.var.t) # STATEMENTS @@ -282,13 +282,13 @@ def processAssignmentTarget(self, target: Expr): if self.defaultToGlobals or target.varInstance.isGlobal: self.instr( f"stsfld {target.inferredType.getCILName()} {self.main}::{target.getCILName()}") - elif target.varInstance.isNonlocal: + elif self.isFromRefArg(target): temp = self.newLocal(None, target.inferredType) - self.load(target.getCILName()) + self.load(target.name) self.load(temp) self.storeInd(target.inferredType) else: - self.store(target.getCILName()) + self.store(target.name) elif isinstance(target, IndexExpr): temp = self.newLocal(None, target.inferredType) self.visit(target.list) @@ -514,7 +514,7 @@ def ForStmt(self, node: ForStmt): self.instr( f"stsfld {node.identifier.inferredType.getCILName()} {self.main}::{node.identifier.getCILName()}") else: - self.store(node.identifier.getCILName()) + self.store(node.identifier.name) # body self.visitStmtList(node.body) # idx = idx + 1 @@ -570,10 +570,10 @@ def Identifier(self, node: Identifier): self.instr( f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") elif self.isFromRefArg(node): - self.load(node.getCILName()) + self.load(node.name) self.loadInd(node.inferredType) else: - self.load(node.getCILName()) + self.load(node.name) def MemberExpr(self, node: MemberExpr): self.visit(node.object) diff --git a/exponent.cil b/exponent.cil new file mode 100644 index 0000000..ce45fd3 --- /dev/null +++ b/exponent.cil @@ -0,0 +1,177 @@ +.assembly 'exponent' +{ +} +.module exponent.exe +.class public auto ansi beforefieldinit exponent extends [mscorlib]System.Object +{ + .field public static int64 n + .field public static int64 i + .method public static hidebysig default void Main (string[] args) cil managed + { + .entrypoint + .maxstack 500 + .locals init ( + ) + ldc.i8 42 + stsfld int64 exponent::n + ldc.i8 0 + stsfld int64 exponent::i + IL_1: nop + ldsfld int64 exponent::i + ldsfld int64 exponent::n + cgt + ldc.i4.0 + ceq + brfalse IL_2 + ldc.i8 2 + ldsfld int64 exponent::i + ldc.i8 31 + rem + call int64 exponent::exp(int64, int64) + call void class [mscorlib]System.Console::WriteLine(int64) + ldnull + pop + ldsfld int64 exponent::i + ldc.i8 1 + add.ovf + stsfld int64 exponent::i + br IL_1 + IL_2: nop + ldc.i8 2 + ldc.i8 3 + call int64 exponent::exp(int64, int64) + ldc.i8 8 + ceq + brtrue IL_3 + ldstr "failed assertion on line 27" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_3: nop + ldnull + pop + ldc.i8 3 + ldc.i8 3 + call int64 exponent::exp(int64, int64) + ldc.i8 27 + ceq + brtrue IL_4 + ldstr "failed assertion on line 28" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_4: nop + ldnull + pop + ldc.i8 3 + ldc.i8 4 + call int64 exponent::exp(int64, int64) + ldc.i8 81 + ceq + brtrue IL_5 + ldstr "failed assertion on line 29" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_5: nop + ldnull + pop + ldc.i8 4 + ldc.i8 4 + call int64 exponent::exp(int64, int64) + ldc.i8 256 + ceq + brtrue IL_6 + ldstr "failed assertion on line 30" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_6: nop + ldnull + pop + ldc.i8 5 + ldc.i8 1 + call int64 exponent::exp(int64, int64) + ldc.i8 5 + ceq + brtrue IL_7 + ldstr "failed assertion on line 31" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_7: nop + ldnull + pop + ldc.i8 1 + ldc.i8 99 + call int64 exponent::exp(int64, int64) + ldc.i8 1 + ceq + brtrue IL_8 + ldstr "failed assertion on line 32" + newobj instance void [mscorlib]System.Exception::.ctor(string) + throw + IL_8: nop + ldnull + pop + ret + } + .method public hidebysig static + int64 exp(int64, int64) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 a + ) + ldc.i8 0 + stloc 0 + ldc.i8 1 + stloc 0 + ldarg 1 + ldloca 0 + ldarg 0 + call int64 exponent::exp__f(int64, int64&, int64) + ret + } + .method public hidebysig static + int64 exp__exp__f__geta(int64&) cil managed + { + .maxstack 500 + .locals init ( + ) + ldarg 0 + ldind.i8 + ret + } + .method public hidebysig static + int64 exp__f(int64, int64&, int64) cil managed + { + .maxstack 500 + .locals init ( + [0] int64 __local__0 + ) + IL_9: nop + ldarg 0 + ldc.i8 0 + cgt + ldc.i4.0 + ceq + brfalse IL_10 + ldarg 1 + call int64 exponent::exp__exp__f__geta(int64&) + ret + br IL_11 + IL_10: nop + ldarg 1 + ldind.i8 + ldarg 2 + mul.ovf + stloc 0 + ldarg 1 + ldloc 0 + stind.i8 + ldarg 0 + ldc.i8 1 + sub.ovf + ldarg 1 + ldarg 2 + call int64 exponent::exp__f(int64, int64&, int64) + ret + IL_11: nop + } +} \ No newline at end of file diff --git a/exponent.exe b/exponent.exe new file mode 100644 index 0000000000000000000000000000000000000000..b0f186e0140861ce6c59309086295290a274885e GIT binary patch literal 3072 zcmeHJO>Y}T7=G4qoMzLQMpQ)=LYYkgKhTN;LWmRGIdba3;dy6v9k*1p^%3>8=lyWEvg%XL&G-8bG3pilDdongn4A|zp@5K58MFH6Vz`#SC%9fvG?W6rxVo>HH-H9lF z;Wo`)`u3WnDd2K0)XktC*HBa`a2D2lXIl;ZpeD-w0xc9*3XrE$C zP_x{4W^UnxOf7b)JDS38vVfVBkJ+kB+FR{+<7SRs26DHO3n_^u)?6&VU3B6W%5(bB ziCjK6pPxU*0|SQ9BrmZ1=`G+U<#pQA@rr%sWy+rvH zZRy8!u^H4Tg_!QF4X426dqlW-hcYH{^DbgrAwz75%Mfm521k(R!*$Z|H^_yaFkx5` zW6uO4OcAq%7azh$%)AEJ$E5CiNhPN4v;trIQBHR>5(BTU1nbwD`gLr!>p|FT)Nr=; zfv!g=f837rCKdy~9W*tTJ9XWP8i9|MYQsl0Vf>impWaHzmJQKxHuqmpj_r0dr%t&aF4)Ip;r^7oZA8d%_;E1Iqrl`1$2W L%zu*hzv{q0Nqr0J literal 0 HcmV?d00001 diff --git a/nonlocal.cil b/nonlocal.cil deleted file mode 100644 index 45616e8..0000000 --- a/nonlocal.cil +++ /dev/null @@ -1,545 +0,0 @@ -.assembly 'nonlocal' -{ -} -.module nonlocal.exe -.class public auto ansi beforefieldinit nonlocal extends [mscorlib]System.Object -{ - .field public static int64 a - .field public static class Nonlocals b - .method public static hidebysig default void Main (string[] args) cil managed - { - .entrypoint - .maxstack 500 - .locals init ( - [0] int64 __local__0, - [1] int64 __local__1, - [2] int64 __local__2, - [3] int64 __local__3, - [4] int64 __local__4 - ) - ldc.i8 0 - stsfld int64 nonlocal::a - ldnull - stsfld class Nonlocals nonlocal::b - ldc.i8 1 - stloc 0 - ldloca 0 - call int64 nonlocal::test(int64&) - ldc.i8 2 - ceq - brtrue IL_1 - ldstr "failed assertion on line 84" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_1: nop - ldnull - pop - call int64 nonlocal::test3() - ldc.i8 3 - ceq - brtrue IL_2 - ldstr "failed assertion on line 86" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_2: nop - ldnull - pop - ldc.i8 0 - stsfld int64 nonlocal::a - ldsfld int64 nonlocal::a - stloc 1 - ldloca 1 - call void nonlocal::test9(int64&) - ldnull - pop - ldsfld int64 nonlocal::a - ldc.i8 0 - ceq - brtrue IL_3 - ldstr "failed assertion on line 92" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_3: nop - ldnull - pop - call void nonlocal::test7() - ldnull - pop - call void nonlocal::test10() - ldnull - pop - ldc.i8 0 - stsfld int64 nonlocal::a - newobj instance void Nonlocals::.ctor() - stsfld class Nonlocals nonlocal::b - ldsfld class Nonlocals nonlocal::b - ldsfld int64 nonlocal::a - stloc 2 - ldloca 2 - callvirt instance void Nonlocals::testMethod(int64&) - ldnull - pop - ldsfld class Nonlocals nonlocal::b - ldc.i8 0 - stloc 3 - ldloca 3 - callvirt instance void Nonlocals::testMethod(int64&) - ldnull - pop - ldsfld int64 nonlocal::a - ldc.i8 0 - ceq - brtrue IL_4 - ldstr "failed assertion on line 103" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_4: nop - ldnull - pop - ldsfld class Nonlocals nonlocal::b - ldc.i8 1 - stloc 4 - ldloca 4 - callvirt instance void Nonlocals::testMethod(int64&) - ldnull - pop - ldsfld class Nonlocals nonlocal::b - callvirt instance void Nonlocals::testMethod4() - ldnull - pop - ret - } - .method public hidebysig static - int64 test(int64&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - call void nonlocal::test__test2(int64&) - ldnull - pop - ldarg 0 - ldind.i8 - ret - } - .method public hidebysig static - int64 test3() cil managed - { - .maxstack 500 - .locals init ( - [0] int64 x - ) - ldc.i8 4 - stloc 0 - ldloca 0 - call void nonlocal::test3__test4(int64&) - ldnull - pop - ldloc 0 - ret - } - .method public hidebysig static - void test7() cil managed - { - .maxstack 500 - .locals init ( - [0] int64[] x - ) - ldnull - stloc 0 - ldc.i4 3 - newarr int64 - dup - ldc.i4 0 - ldc.i8 1 - stelem int64 - dup - ldc.i4 1 - ldc.i8 2 - stelem int64 - dup - ldc.i4 2 - ldc.i8 3 - stelem int64 - stloc 0 - ldloc 0 - call void nonlocal::test7__test8(int64[]) - ldnull - pop - ldloc 0 - ldc.i8 0 - conv.i4 - ldelem int64 - ldc.i8 0 - ceq - brtrue IL_5 - ldstr "failed assertion on line 33" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_5: nop - ldnull - pop - ret - } - .method public hidebysig static - void test9(int64&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - call void nonlocal::test9__test9helper(int64&) - ldnull - pop - ret - } - .method public hidebysig static - void test10() cil managed - { - .maxstack 500 - .locals init ( - [0] int64 y - ) - ldc.i8 1 - stloc 0 - ldloc 0 - ldloca 0 - call int64 nonlocal::test10__test11(int64, int64&) - ldc.i8 2 - ceq - brtrue IL_6 - ldstr "failed assertion on line 52" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_6: nop - ldnull - pop - ldloca 0 - call int64 nonlocal::test10__test12(int64&) - ldc.i8 4 - ceq - brtrue IL_7 - ldstr "failed assertion on line 53" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_7: nop - ldnull - pop - ldloc 0 - ldc.i8 2 - ceq - brtrue IL_8 - ldstr "failed assertion on line 54" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_8: nop - ldnull - pop - ret - } - .method public hidebysig static - void test13(class Nonlocals&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - call void nonlocal::test13__test14(class Nonlocals&) - ldnull - pop - ret - } - .method public hidebysig static - void test__test2(int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - ldc.i8 2 - stloc 0 - ldarg 0 - ldloc 0 - stind.i8 - ret - } - .method public hidebysig static - void test3__test3__test4__test5(int64&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - ldind.i8 - ldc.i8 4 - ceq - brtrue IL_9 - ldstr "failed assertion on line 15" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_9: nop - ldnull - pop - ret - } - .method public hidebysig static - void test3__test3__test4__test6(int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - ldc.i8 3 - stloc 0 - ldarg 0 - ldloc 0 - stind.i8 - ret - } - .method public hidebysig static - void test3__test4(int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - ldarg 0 - call void nonlocal::test3__test3__test4__test5(int64&) - ldnull - pop - ldarg 0 - ldind.i8 - stloc 0 - ldloca 0 - call int64 nonlocal::test(int64&) - pop - ldarg 0 - ldind.i8 - ldc.i8 4 - ceq - brtrue IL_10 - ldstr "failed assertion on line 21" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_10: nop - ldnull - pop - ldarg 0 - call void nonlocal::test3__test3__test4__test6(int64&) - ldnull - pop - ldarg 0 - ldind.i8 - ldc.i8 3 - ceq - brtrue IL_11 - ldstr "failed assertion on line 23" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_11: nop - ldnull - pop - ret - } - .method public hidebysig static - void test7__test8(int64[]) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - ldc.i8 0 - stloc 0 - ldarg 0 - ldc.i8 0 - ldloc 0 - stelem int64 - ret - } - .method public hidebysig static - void test9__test9helper(int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - ldc.i8 0 - stloc 0 - ldarg 0 - ldloc 0 - stind.i8 - ret - } - .method public hidebysig static - int64 test10__test11(int64, int64&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - ldarg 1 - ldind.i8 - add.ovf - ret - } - .method public hidebysig static - int64 test10__test10__test12__test13(int64, int64&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - ldarg 1 - ldind.i8 - add.ovf - ret - } - .method public hidebysig static - int64 test10__test12(int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - ldarg 0 - ldind.i8 - ldarg 0 - call int64 nonlocal::test10__test10__test12__test13(int64, int64&) - stloc 0 - ldarg 0 - ldloc 0 - stind.i8 - ldarg 0 - ldind.i8 - ldc.i8 2 - ceq - brtrue IL_12 - ldstr "failed assertion on line 50" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_12: nop - ldnull - pop - ldarg 0 - ldind.i8 - ldarg 0 - call int64 nonlocal::test10__test10__test12__test13(int64, int64&) - ret - } - .method public hidebysig static - void Nonlocals__testMethod__testMethod2(class Nonlocals, int64&, int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0, - [1] int64 __local__1 - ) - ldarg 0 - callvirt instance void Nonlocals::testMethod3() - ldnull - pop - ldc.i8 3 - stloc 0 - ldarg 1 - ldloc 0 - stind.i8 - ldc.i8 3 - stloc 1 - ldarg 2 - ldloc 1 - stind.i8 - ret - } - .method public hidebysig static - void test13__test14(class Nonlocals&) cil managed - { - .maxstack 500 - .locals init ( - [0] class Nonlocals __local__0 - ) - ldnull - stloc 0 - ldarg 0 - ldloc 0 - stind.ref - ret - } -} -.class public auto ansi beforefieldinit Nonlocals extends object -{ - .method public hidebysig virtual instance - void testMethod3() cil managed - { - .maxstack 500 - .locals init ( - ) - nop - ret - } - .method public hidebysig virtual instance - void testMethod(int64&) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 y - ) - ldc.i8 2 - stloc 0 - ldarg 0 - ldarg 1 - ldloca 0 - call void nonlocal::Nonlocals__testMethod__testMethod2(class Nonlocals, int64&, int64&) - ldnull - pop - ldloc 0 - ldc.i8 3 - ceq - brtrue IL_13 - ldstr "failed assertion on line 69" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_13: nop - ldnull - pop - ret - } - .method public hidebysig virtual instance - void testMethod4() cil managed - { - .maxstack 500 - .locals init ( - [0] class Nonlocals __local__0 - ) - ldarg 0 - stloc 0 - ldloca 0 - call void nonlocal::test13(class Nonlocals&) - ldnull - pop - ldarg 0 - ldnull - ceq - ldc.i4.0 - ceq - brtrue IL_14 - ldstr "failed assertion on line 73" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_14: nop - ldnull - pop - ret - } - .method public hidebysig specialname rtspecialname instance - void .ctor() cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg.0 - call instance void object::.ctor() - nop - ret - } -} \ No newline at end of file diff --git a/nonlocal.exe b/nonlocal.exe deleted file mode 100644 index f1dcdaf4b652830c71daedfa2c5ed2ee6374be28..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 5120 zcmd^DO>7%Q6n#a=4A zXUcPnc7Co@a;tXEvCC&mcA;dCADgxp-FYX~(P8a3rl$@OO{f43pZ_H1d0V51+Nr`s zQE&p9TT9_>`<%c8YG^V90{avF*asFtS8r4dj-VYn)dM z#>S?{dx#bQTw#p|McCTd-FH{L2}S&iB|Oy&phEErupdfbDWeu4s&x=0!3`(aHIJHl zb|+qCNG(+`S?^Q4Cy7Fa6S|spkx>?bo`@tw8UtJ!{O%~1xAE5G`45U z>v6GVX*X=+BKv}kIP2P6gU=Aj-UBvPm((nL){MrdmP{3ajSPcq4F@>va z$tS%s{mywpbtF%{jO#;~MOZN(f@f8F3O%&3r}iosL2bo|hyg^no*Yw?X%<5^wHQSF zh&(i(vIJ77(db-13OoQ9s|#~~BSKE{Os~{q2{P~>lN{sRbMRa*zbwdAShF^u`hk6- zE*cfIlDNz~1Acxzg!fn)Nze#6U;E)3LDkQsIE9^ht^ri)Bl+cc5aLF86A$+vpHv>==_(9gSAK0fzF;XCS`5Hxfbe{?! z*mt2e;(|`U{DdX2zF-cbfQ^^6)~u${|8qhbX<6#x|l7 zoPPg7#I#hrraC?CJFE>^qpj~AKGrf&n6c?s#}wh8-uH$(>{ZzMD|1E~PrHq;5~f?nuJy>|*nE=c# zP``j`r!_;Z!Ab{34`}cAKt*W4P@QTBxtuW6pgKx1x+K&`kDXMr$n~mFHz}v`pza!K z5qo=RpoeYVq;o=XhNnP#wcx$Y*Id#6lF=3#CH0y+LhHupq$&XS|2fBDZ5DNToSwo{ z|BPdO7|%E&(+C8HKXFg4AGRh-ZV5m5=87q2*&&ah=O*3xvqk3xTCC*Va=|0U%r`MC(R9ZVs8rtZD3IJAGkHN-n8ej-+0c{ zqt~2Cr@G+IH?I3>j&hXBS6z)-ONd#$)WGyQBZU|~4V?n+E{EpH#=3%!L(m7;r4o30 zE`ly$$EFb)AZki)s42Zs>L(iAI?goi4g}c*(oBBaP~Ki z6W=>c_)F7ZQ+h*$ujD5D4Rd_-(;c#}oL@ci`SRv>Hz!@JspHdJg#Ss~2k z!n~c**Cfw>@!ICq#^L?HmT2~%TRybx$ZaUMr{kpNi~O4efAKg#c2-u~cCyXi7GB)8 J8Q9i={{u;;aWwz{ From 296469ccf89c1869a1dc999239be8565ba420b96 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 22 May 2022 18:22:06 -0700 Subject: [PATCH 21/79] remove artifacts --- exponent.cil | 177 --------------------------------------------------- exponent.exe | Bin 3072 -> 0 bytes 2 files changed, 177 deletions(-) delete mode 100644 exponent.cil delete mode 100644 exponent.exe diff --git a/exponent.cil b/exponent.cil deleted file mode 100644 index ce45fd3..0000000 --- a/exponent.cil +++ /dev/null @@ -1,177 +0,0 @@ -.assembly 'exponent' -{ -} -.module exponent.exe -.class public auto ansi beforefieldinit exponent extends [mscorlib]System.Object -{ - .field public static int64 n - .field public static int64 i - .method public static hidebysig default void Main (string[] args) cil managed - { - .entrypoint - .maxstack 500 - .locals init ( - ) - ldc.i8 42 - stsfld int64 exponent::n - ldc.i8 0 - stsfld int64 exponent::i - IL_1: nop - ldsfld int64 exponent::i - ldsfld int64 exponent::n - cgt - ldc.i4.0 - ceq - brfalse IL_2 - ldc.i8 2 - ldsfld int64 exponent::i - ldc.i8 31 - rem - call int64 exponent::exp(int64, int64) - call void class [mscorlib]System.Console::WriteLine(int64) - ldnull - pop - ldsfld int64 exponent::i - ldc.i8 1 - add.ovf - stsfld int64 exponent::i - br IL_1 - IL_2: nop - ldc.i8 2 - ldc.i8 3 - call int64 exponent::exp(int64, int64) - ldc.i8 8 - ceq - brtrue IL_3 - ldstr "failed assertion on line 27" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_3: nop - ldnull - pop - ldc.i8 3 - ldc.i8 3 - call int64 exponent::exp(int64, int64) - ldc.i8 27 - ceq - brtrue IL_4 - ldstr "failed assertion on line 28" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_4: nop - ldnull - pop - ldc.i8 3 - ldc.i8 4 - call int64 exponent::exp(int64, int64) - ldc.i8 81 - ceq - brtrue IL_5 - ldstr "failed assertion on line 29" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_5: nop - ldnull - pop - ldc.i8 4 - ldc.i8 4 - call int64 exponent::exp(int64, int64) - ldc.i8 256 - ceq - brtrue IL_6 - ldstr "failed assertion on line 30" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_6: nop - ldnull - pop - ldc.i8 5 - ldc.i8 1 - call int64 exponent::exp(int64, int64) - ldc.i8 5 - ceq - brtrue IL_7 - ldstr "failed assertion on line 31" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_7: nop - ldnull - pop - ldc.i8 1 - ldc.i8 99 - call int64 exponent::exp(int64, int64) - ldc.i8 1 - ceq - brtrue IL_8 - ldstr "failed assertion on line 32" - newobj instance void [mscorlib]System.Exception::.ctor(string) - throw - IL_8: nop - ldnull - pop - ret - } - .method public hidebysig static - int64 exp(int64, int64) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 a - ) - ldc.i8 0 - stloc 0 - ldc.i8 1 - stloc 0 - ldarg 1 - ldloca 0 - ldarg 0 - call int64 exponent::exp__f(int64, int64&, int64) - ret - } - .method public hidebysig static - int64 exp__exp__f__geta(int64&) cil managed - { - .maxstack 500 - .locals init ( - ) - ldarg 0 - ldind.i8 - ret - } - .method public hidebysig static - int64 exp__f(int64, int64&, int64) cil managed - { - .maxstack 500 - .locals init ( - [0] int64 __local__0 - ) - IL_9: nop - ldarg 0 - ldc.i8 0 - cgt - ldc.i4.0 - ceq - brfalse IL_10 - ldarg 1 - call int64 exponent::exp__exp__f__geta(int64&) - ret - br IL_11 - IL_10: nop - ldarg 1 - ldind.i8 - ldarg 2 - mul.ovf - stloc 0 - ldarg 1 - ldloc 0 - stind.i8 - ldarg 0 - ldc.i8 1 - sub.ovf - ldarg 1 - ldarg 2 - call int64 exponent::exp__f(int64, int64&, int64) - ret - IL_11: nop - } -} \ No newline at end of file diff --git a/exponent.exe b/exponent.exe deleted file mode 100644 index b0f186e0140861ce6c59309086295290a274885e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 3072 zcmeHJO>Y}T7=G4qoMzLQMpQ)=LYYkgKhTN;LWmRGIdba3;dy6v9k*1p^%3>8=lyWEvg%XL&G-8bG3pilDdongn4A|zp@5K58MFH6Vz`#SC%9fvG?W6rxVo>HH-H9lF z;Wo`)`u3WnDd2K0)XktC*HBa`a2D2lXIl;ZpeD-w0xc9*3XrE$C zP_x{4W^UnxOf7b)JDS38vVfVBkJ+kB+FR{+<7SRs26DHO3n_^u)?6&VU3B6W%5(bB ziCjK6pPxU*0|SQ9BrmZ1=`G+U<#pQA@rr%sWy+rvH zZRy8!u^H4Tg_!QF4X426dqlW-hcYH{^DbgrAwz75%Mfm521k(R!*$Z|H^_yaFkx5` zW6uO4OcAq%7azh$%)AEJ$E5CiNhPN4v;trIQBHR>5(BTU1nbwD`gLr!>p|FT)Nr=; zfv!g=f837rCKdy~9W*tTJ9XWP8i9|MYQsl0Vf>impWaHzmJQKxHuqmpj_r0dr%t&aF4)Ip;r^7oZA8d%_;E1Iqrl`1$2W L%zu*hzv{q0Nqr0J From d0fb63fc744771640fa4d06076743f42089899c1 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Mon, 23 May 2022 17:00:36 -0700 Subject: [PATCH 22/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c8ab411..fd51222 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Chocopy is used in compiler courses at several universities. This project has no Progress is documented on my [blog](https://yangdanny97.github.io/blog/): - [Part 1: Frontend/Typechecker](https://yangdanny97.github.io/blog/2020/05/29/chocopy-typechecker) - [Part 2: JVM backend](https://yangdanny97.github.io/blog/2021/08/26/chocopy-jvm-backend) -- Part 3: CIL backend - coming soon! +- [Part 3: CIL backend](https://yangdanny97.github.io/blog/2022/05/22/chocopy-cil-backend) ## Features From 27f7257131a326cdda0fb12983954a6f9c6707e8 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 7 Sep 2022 18:47:50 -0700 Subject: [PATCH 23/79] fix type annotations --- compiler/astnodes/assignstmt.py | 3 ++- compiler/astnodes/binaryexpr.py | 3 ++- compiler/astnodes/booleanliteral.py | 3 ++- compiler/astnodes/callexpr.py | 3 ++- compiler/astnodes/classdef.py | 3 ++- compiler/astnodes/classtype.py | 3 ++- compiler/astnodes/compilererror.py | 3 ++- compiler/astnodes/declaration.py | 3 ++- compiler/astnodes/errors.py | 3 ++- compiler/astnodes/expr.py | 3 ++- compiler/astnodes/exprstmt.py | 3 ++- compiler/astnodes/forstmt.py | 3 ++- compiler/astnodes/funcdef.py | 5 +++-- compiler/astnodes/globaldecl.py | 3 ++- compiler/astnodes/identifier.py | 3 ++- compiler/astnodes/ifexpr.py | 3 ++- compiler/astnodes/ifstmt.py | 3 ++- compiler/astnodes/indexexpr.py | 3 ++- compiler/astnodes/integerliteral.py | 3 ++- compiler/astnodes/listexpr.py | 3 ++- compiler/astnodes/listtype.py | 3 ++- compiler/astnodes/literal.py | 3 ++- compiler/astnodes/memberexpr.py | 3 ++- compiler/astnodes/methodcallexpr.py | 3 ++- compiler/astnodes/node.py | 4 +++- compiler/astnodes/noneliteral.py | 3 ++- compiler/astnodes/nonlocaldecl.py | 3 ++- compiler/astnodes/program.py | 3 ++- compiler/astnodes/returnstmt.py | 3 ++- compiler/astnodes/stmt.py | 3 ++- compiler/astnodes/stringliteral.py | 3 ++- compiler/astnodes/typeannotation.py | 3 ++- compiler/astnodes/typedvar.py | 3 ++- compiler/astnodes/unaryexpr.py | 3 ++- compiler/astnodes/vardef.py | 3 ++- compiler/astnodes/whilestmt.py | 3 ++- compiler/builder.py | 3 --- compiler/cil_backend.py | 10 +++++----- compiler/closurevisitor.py | 3 ++- compiler/empty_list_typer.py | 3 ++- compiler/jvm_backend.py | 6 +++--- compiler/parser.py | 5 +++-- compiler/types/functype.py | 3 ++- compiler/varcollector.py | 4 ++-- 44 files changed, 93 insertions(+), 55 deletions(-) diff --git a/compiler/astnodes/assignstmt.py b/compiler/astnodes/assignstmt.py index c701526..4142c3e 100644 --- a/compiler/astnodes/assignstmt.py +++ b/compiler/astnodes/assignstmt.py @@ -1,10 +1,11 @@ from .stmt import Stmt from .expr import Expr +from typing import List class AssignStmt(Stmt): - def __init__(self, location: [int], targets: [Expr], value: Expr): + def __init__(self, location: List[int], targets: List[Expr], value: Expr): super().__init__(location, "AssignStmt") self.targets = targets self.value = value diff --git a/compiler/astnodes/binaryexpr.py b/compiler/astnodes/binaryexpr.py index 2cbd133..8fbc6fb 100644 --- a/compiler/astnodes/binaryexpr.py +++ b/compiler/astnodes/binaryexpr.py @@ -1,9 +1,10 @@ from .expr import Expr +from typing import List class BinaryExpr(Expr): - def __init__(self, location: [int], left: Expr, operator: str, right: Expr): + def __init__(self, location: List[int], left: Expr, operator: str, right: Expr): super().__init__(location, "BinaryExpr") self.left = left self.right = right diff --git a/compiler/astnodes/booleanliteral.py b/compiler/astnodes/booleanliteral.py index 22b6b6e..a704bfb 100644 --- a/compiler/astnodes/booleanliteral.py +++ b/compiler/astnodes/booleanliteral.py @@ -1,9 +1,10 @@ from .literal import Literal +from typing import List class BooleanLiteral(Literal): - def __init__(self, location: [int], value: bool): + def __init__(self, location: List[int], value: bool): super().__init__(location, "BooleanLiteral") self.value = value diff --git a/compiler/astnodes/callexpr.py b/compiler/astnodes/callexpr.py index 29540f0..77412b1 100644 --- a/compiler/astnodes/callexpr.py +++ b/compiler/astnodes/callexpr.py @@ -1,10 +1,11 @@ from .expr import Expr from .identifier import Identifier +from typing import List class CallExpr(Expr): - def __init__(self, location: [int], function: Identifier, args: [Expr]): + def __init__(self, location: List[int], function: Identifier, args: List[Expr]): super().__init__(location, "CallExpr") self.function = function self.args = args diff --git a/compiler/astnodes/classdef.py b/compiler/astnodes/classdef.py index ee4df5f..71b24f3 100644 --- a/compiler/astnodes/classdef.py +++ b/compiler/astnodes/classdef.py @@ -7,11 +7,12 @@ from ..types.classvaluetype import ClassValueType from ..types.functype import FuncType from ..types.Types import NoneType +from typing import List class ClassDef(Declaration): - def __init__(self, location: [int], name: Identifier, superclass: Identifier, declarations: [Declaration]): + def __init__(self, location: List[int], name: Identifier, superclass: Identifier, declarations: List[Declaration]): super().__init__(location, "ClassDef") self.name = name self.superclass = superclass diff --git a/compiler/astnodes/classtype.py b/compiler/astnodes/classtype.py index 3ff91b1..0a81b2c 100644 --- a/compiler/astnodes/classtype.py +++ b/compiler/astnodes/classtype.py @@ -1,9 +1,10 @@ from .typeannotation import TypeAnnotation +from typing import List class ClassType(TypeAnnotation): - def __init__(self, location: [int], className: str): + def __init__(self, location: List[int], className: str): super().__init__(location, "ClassType") self.className = className diff --git a/compiler/astnodes/compilererror.py b/compiler/astnodes/compilererror.py index 3cb0d62..10fa433 100644 --- a/compiler/astnodes/compilererror.py +++ b/compiler/astnodes/compilererror.py @@ -1,9 +1,10 @@ from .node import Node +from typing import List class CompilerError(Node): - def __init__(self, location: [int], message: str, syntax: bool = False): + def __init__(self, location: List[int], message: str, syntax: bool = False): super().__init__(location, "CompilerError") self.message = message self.syntax = syntax diff --git a/compiler/astnodes/declaration.py b/compiler/astnodes/declaration.py index ac9a6e8..8e95696 100644 --- a/compiler/astnodes/declaration.py +++ b/compiler/astnodes/declaration.py @@ -1,7 +1,8 @@ from .node import Node +from typing import List class Declaration(Node): - def __init__(self, location: [int], kind: str): + def __init__(self, location: List[int], kind: str): super().__init__(location, kind) diff --git a/compiler/astnodes/errors.py b/compiler/astnodes/errors.py index 335f0cd..30491f6 100644 --- a/compiler/astnodes/errors.py +++ b/compiler/astnodes/errors.py @@ -1,10 +1,11 @@ from .node import Node from .compilererror import CompilerError +from typing import List class Errors(Node): - def __init__(self, location: [int], errors: [CompilerError]): + def __init__(self, location: List[int], errors: List[CompilerError]): super().__init__(location, "Errors") self.errors = errors diff --git a/compiler/astnodes/expr.py b/compiler/astnodes/expr.py index a9d81d7..217f81c 100644 --- a/compiler/astnodes/expr.py +++ b/compiler/astnodes/expr.py @@ -1,9 +1,10 @@ from .node import Node +from typing import List class Expr(Node): - def __init__(self, location: [int], kind: str): + def __init__(self, location: List[int], kind: str): super().__init__(location, kind) self.inferredType = None self.shouldBoxAsRef = False diff --git a/compiler/astnodes/exprstmt.py b/compiler/astnodes/exprstmt.py index c7d18ea..7ed2fe8 100644 --- a/compiler/astnodes/exprstmt.py +++ b/compiler/astnodes/exprstmt.py @@ -1,10 +1,11 @@ from .stmt import Stmt from .expr import Expr +from typing import List class ExprStmt(Stmt): - def __init__(self, location: [int], expr: Expr): + def __init__(self, location: List[int], expr: Expr): super().__init__(location, "ExprStmt") self.expr = expr diff --git a/compiler/astnodes/forstmt.py b/compiler/astnodes/forstmt.py index 2b3b3e2..d6bb634 100644 --- a/compiler/astnodes/forstmt.py +++ b/compiler/astnodes/forstmt.py @@ -1,11 +1,12 @@ from .stmt import Stmt from .expr import Expr from .identifier import Identifier +from typing import List class ForStmt(Stmt): - def __init__(self, location: [int], identifier: Identifier, iterable: Expr, body: [Stmt]): + def __init__(self, location: List[int], identifier: Identifier, iterable: Expr, body: List[Stmt]): super().__init__(location, "ForStmt") self.identifier = identifier self.iterable = iterable diff --git a/compiler/astnodes/funcdef.py b/compiler/astnodes/funcdef.py index 60be06a..9f6d384 100644 --- a/compiler/astnodes/funcdef.py +++ b/compiler/astnodes/funcdef.py @@ -3,6 +3,7 @@ from .typedvar import TypedVar from .typeannotation import TypeAnnotation from .stmt import Stmt +from typing import List class FuncDef(Declaration): @@ -12,8 +13,8 @@ class FuncDef(Declaration): # DECLARATIONS # STATEMENTS - def __init__(self, location: [int], name: Identifier, params: [TypedVar], returnType: TypeAnnotation, - declarations: [Declaration], statements: [Stmt], isMethod: bool = False): + def __init__(self, location: List[int], name: Identifier, params: List[TypedVar], returnType: TypeAnnotation, + declarations: List[Declaration], statements: List[Stmt], isMethod: bool = False): super().__init__(location, "FuncDef") self.name = name self.params = params diff --git a/compiler/astnodes/globaldecl.py b/compiler/astnodes/globaldecl.py index d7b8a55..bc9fd4a 100644 --- a/compiler/astnodes/globaldecl.py +++ b/compiler/astnodes/globaldecl.py @@ -1,10 +1,11 @@ from .declaration import Declaration from .identifier import Identifier +from typing import List class GlobalDecl(Declaration): - def __init__(self, location: [int], variable: Identifier): + def __init__(self, location: List[int], variable: Identifier): super().__init__(location, "GlobalDecl") self.variable = variable diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index e69d5e1..87ae1fe 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -1,4 +1,5 @@ from .expr import Expr +from typing import List CIL_KEYWORDS = set(["char", "value", "int32", "int64", "string", "long", "null"] + ["add", @@ -121,7 +122,7 @@ class Identifier(Expr): - def __init__(self, location: [int], name: str): + def __init__(self, location: List[int], name: str): super().__init__(location, "Identifier") self.name = name self.varInstance = None diff --git a/compiler/astnodes/ifexpr.py b/compiler/astnodes/ifexpr.py index 99785cc..ee972f8 100644 --- a/compiler/astnodes/ifexpr.py +++ b/compiler/astnodes/ifexpr.py @@ -1,9 +1,10 @@ from .expr import Expr +from typing import List class IfExpr(Expr): - def __init__(self, location: [int], condition: Expr, thenExpr: Expr, elseExpr: Expr): + def __init__(self, location: List[int], condition: Expr, thenExpr: Expr, elseExpr: Expr): super().__init__(location, "IfExpr") self.condition = condition self.thenExpr = thenExpr diff --git a/compiler/astnodes/ifstmt.py b/compiler/astnodes/ifstmt.py index 4810dd8..d110d08 100644 --- a/compiler/astnodes/ifstmt.py +++ b/compiler/astnodes/ifstmt.py @@ -1,10 +1,11 @@ from .stmt import Stmt from .expr import Expr +from typing import List class IfStmt(Stmt): - def __init__(self, location: [int], condition: Expr, thenBody: [Stmt], elseBody: [Stmt]): + def __init__(self, location: List[int], condition: Expr, thenBody: List[Stmt], elseBody: List[Stmt]): super().__init__(location, "IfStmt") self.condition = condition self.thenBody = [s for s in thenBody if s is not None] diff --git a/compiler/astnodes/indexexpr.py b/compiler/astnodes/indexexpr.py index 43860b1..1c4b25c 100644 --- a/compiler/astnodes/indexexpr.py +++ b/compiler/astnodes/indexexpr.py @@ -1,9 +1,10 @@ from .expr import Expr +from typing import List class IndexExpr(Expr): - def __init__(self, location: [int], lst: Expr, index: Expr): + def __init__(self, location: List[int], lst: Expr, index: Expr): super().__init__(location, "IndexExpr") self.list = lst self.index = index diff --git a/compiler/astnodes/integerliteral.py b/compiler/astnodes/integerliteral.py index 8755044..c7a6427 100644 --- a/compiler/astnodes/integerliteral.py +++ b/compiler/astnodes/integerliteral.py @@ -1,9 +1,10 @@ from .literal import Literal +from typing import List class IntegerLiteral(Literal): - def __init__(self, location: [int], value: int): + def __init__(self, location: List[int], value: int): super().__init__(location, "IntegerLiteral") self.value = value diff --git a/compiler/astnodes/listexpr.py b/compiler/astnodes/listexpr.py index 10b5963..ee4b3a6 100644 --- a/compiler/astnodes/listexpr.py +++ b/compiler/astnodes/listexpr.py @@ -1,9 +1,10 @@ from .expr import Expr +from typing import List class ListExpr(Expr): - def __init__(self, location: [int], elements: [Expr]): + def __init__(self, location: List[int], elements: List[Expr]): super().__init__(location, "ListExpr") self.elements = elements self.emptyListType = None diff --git a/compiler/astnodes/listtype.py b/compiler/astnodes/listtype.py index c312267..343d66f 100644 --- a/compiler/astnodes/listtype.py +++ b/compiler/astnodes/listtype.py @@ -1,9 +1,10 @@ from .typeannotation import TypeAnnotation +from typing import List class ListType(TypeAnnotation): - def __init__(self, location: [int], elementType: TypeAnnotation): + def __init__(self, location: List[int], elementType: TypeAnnotation): super().__init__(location, "ListType") self.elementType = elementType diff --git a/compiler/astnodes/literal.py b/compiler/astnodes/literal.py index de4795e..a182fea 100644 --- a/compiler/astnodes/literal.py +++ b/compiler/astnodes/literal.py @@ -1,9 +1,10 @@ from .expr import Expr +from typing import List class Literal(Expr): - def __init__(self, location: [int], kind: str): + def __init__(self, location: List[int], kind: str): super().__init__(location, kind) self.value = None diff --git a/compiler/astnodes/memberexpr.py b/compiler/astnodes/memberexpr.py index a059364..487cc1a 100644 --- a/compiler/astnodes/memberexpr.py +++ b/compiler/astnodes/memberexpr.py @@ -1,10 +1,11 @@ from .expr import Expr from .identifier import Identifier +from typing import List class MemberExpr(Expr): - def __init__(self, location: [int], obj: Expr, member: Identifier): + def __init__(self, location: List[int], obj: Expr, member: Identifier): super().__init__(location, "MemberExpr") self.object = obj self.member = member diff --git a/compiler/astnodes/methodcallexpr.py b/compiler/astnodes/methodcallexpr.py index 5f89873..deb70ac 100644 --- a/compiler/astnodes/methodcallexpr.py +++ b/compiler/astnodes/methodcallexpr.py @@ -1,10 +1,11 @@ from .expr import Expr from .memberexpr import MemberExpr +from typing import List class MethodCallExpr(Expr): - def __init__(self, location: [int], method: MemberExpr, args: [Expr]): + def __init__(self, location: List[int], method: MemberExpr, args: List[Expr]): super().__init__(location, "MethodCallExpr") self.method = method self.args = args diff --git a/compiler/astnodes/node.py b/compiler/astnodes/node.py index 8834c7b..6af5cca 100644 --- a/compiler/astnodes/node.py +++ b/compiler/astnodes/node.py @@ -1,7 +1,9 @@ +from typing import List + class Node: - def __init__(self, location: [int], kind: str): + def __init__(self, location: List[int], kind: str): if len(location) != 2: raise Exception('location must be length 2') self.kind = kind diff --git a/compiler/astnodes/noneliteral.py b/compiler/astnodes/noneliteral.py index 20ad082..475aea7 100644 --- a/compiler/astnodes/noneliteral.py +++ b/compiler/astnodes/noneliteral.py @@ -1,9 +1,10 @@ from .literal import Literal +from typing import List class NoneLiteral(Literal): - def __init__(self, location: [int]): + def __init__(self, location: List[int]): super().__init__(location, "NoneLiteral") self.value = None diff --git a/compiler/astnodes/nonlocaldecl.py b/compiler/astnodes/nonlocaldecl.py index ce269e9..90f3a18 100644 --- a/compiler/astnodes/nonlocaldecl.py +++ b/compiler/astnodes/nonlocaldecl.py @@ -1,10 +1,11 @@ from .declaration import Declaration from .identifier import Identifier +from typing import List class NonLocalDecl(Declaration): - def __init__(self, location: [int], variable: Identifier): + def __init__(self, location: List[int], variable: Identifier): super().__init__(location, "NonLocalDecl") self.variable = variable diff --git a/compiler/astnodes/program.py b/compiler/astnodes/program.py index 9a13c11..c7c4318 100644 --- a/compiler/astnodes/program.py +++ b/compiler/astnodes/program.py @@ -2,13 +2,14 @@ from .declaration import Declaration from .stmt import Stmt from .errors import Errors +from typing import List # root AST for source file class Program(Node): - def __init__(self, location: [int], declarations: [Declaration], statements: [Stmt], errors: Errors): + def __init__(self, location: List[int], declarations: List[Declaration], statements: List[Stmt], errors: Errors): super().__init__(location, "Program") self.declarations = [d for d in declarations if d is not None] self.statements = [s for s in statements if s is not None] diff --git a/compiler/astnodes/returnstmt.py b/compiler/astnodes/returnstmt.py index 58280ec..d124dfb 100644 --- a/compiler/astnodes/returnstmt.py +++ b/compiler/astnodes/returnstmt.py @@ -1,10 +1,11 @@ from .stmt import Stmt from .expr import Expr +from typing import List class ReturnStmt(Stmt): - def __init__(self, location: [int], value: Expr): + def __init__(self, location: List[int], value: Expr): super().__init__(location, "ReturnStmt") self.value = value self.isReturn = True diff --git a/compiler/astnodes/stmt.py b/compiler/astnodes/stmt.py index 6d8d22e..22b4cc1 100644 --- a/compiler/astnodes/stmt.py +++ b/compiler/astnodes/stmt.py @@ -1,8 +1,9 @@ from .node import Node +from typing import List class Stmt(Node): - def __init__(self, location: [int], kind: str): + def __init__(self, location: List[int], kind: str): super().__init__(location, kind) self.isReturn = False diff --git a/compiler/astnodes/stringliteral.py b/compiler/astnodes/stringliteral.py index ec901a2..2084813 100644 --- a/compiler/astnodes/stringliteral.py +++ b/compiler/astnodes/stringliteral.py @@ -1,9 +1,10 @@ from .literal import Literal +from typing import List class StringLiteral(Literal): - def __init__(self, location: [int], value: str): + def __init__(self, location: List[int], value: str): super().__init__(location, "StringLiteral") self.value = value diff --git a/compiler/astnodes/typeannotation.py b/compiler/astnodes/typeannotation.py index 370d8ca..7d105d4 100644 --- a/compiler/astnodes/typeannotation.py +++ b/compiler/astnodes/typeannotation.py @@ -1,7 +1,8 @@ from .node import Node +from typing import List class TypeAnnotation(Node): - def __init__(self, location: [int], kind: str): + def __init__(self, location: List[int], kind: str): super().__init__(location, kind) diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 1dd56aa..407b826 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -1,11 +1,12 @@ from .node import Node from .identifier import Identifier from .typeannotation import TypeAnnotation +from typing import List class TypedVar(Node): - def __init__(self, location: [int], identifier: Identifier, typ: TypeAnnotation): + def __init__(self, location: List[int], identifier: Identifier, typ: TypeAnnotation): super().__init__(location, "TypedVar") self.identifier = identifier self.type = typ diff --git a/compiler/astnodes/unaryexpr.py b/compiler/astnodes/unaryexpr.py index e5bd872..6b2caf6 100644 --- a/compiler/astnodes/unaryexpr.py +++ b/compiler/astnodes/unaryexpr.py @@ -1,9 +1,10 @@ from .expr import Expr +from typing import List class UnaryExpr(Expr): - def __init__(self, location: [int], operator: str, operand: Expr): + def __init__(self, location: List[int], operator: str, operand: Expr): super().__init__(location, "UnaryExpr") self.operand = operand self.operator = operator diff --git a/compiler/astnodes/vardef.py b/compiler/astnodes/vardef.py index e955e40..2cc5042 100644 --- a/compiler/astnodes/vardef.py +++ b/compiler/astnodes/vardef.py @@ -1,11 +1,12 @@ from .declaration import Declaration from .expr import Expr from .typedvar import TypedVar +from typing import List class VarDef(Declaration): - def __init__(self, location: [int], var: TypedVar, value: Expr, isAttr: bool = False, attrOfClass=None): + def __init__(self, location: List[int], var: TypedVar, value: Expr, isAttr: bool = False, attrOfClass=None): super().__init__(location, "VarDef") self.var = var self.value = value diff --git a/compiler/astnodes/whilestmt.py b/compiler/astnodes/whilestmt.py index 6878380..e531edd 100644 --- a/compiler/astnodes/whilestmt.py +++ b/compiler/astnodes/whilestmt.py @@ -1,10 +1,11 @@ from .stmt import Stmt from .expr import Expr +from typing import List class WhileStmt(Stmt): - def __init__(self, location: [int], condition: Expr, body: [Stmt]): + def __init__(self, location: List[int], condition: Expr, body: List[Stmt]): super().__init__(location, "WhileStmt") self.condition = condition self.body = [s for s in body if s is not None] diff --git a/compiler/builder.py b/compiler/builder.py index 8a38f74..23b790d 100644 --- a/compiler/builder.py +++ b/compiler/builder.py @@ -1,6 +1,3 @@ -from unicodedata import name - - class Builder: def __init__(self, name: str): self.name = name diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index f679d26..cd1beb0 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from collections import defaultdict +from typing import List import json @@ -127,7 +127,7 @@ def newLocal(self, name: str, t: ValueType): self.locals[-1][name] = CilStackLoc(name, n, t.getCILName(), False) return name - def visitStmtList(self, stmts: [Stmt]): + def visitStmtList(self, stmts: List[Stmt]): if len(stmts) == 0: self.instr("nop") else: @@ -693,7 +693,7 @@ def visitArg(self, funcType, paramIdx: int, arg: Expr): # ref -> ref: pass through a ref to a nonlocal self.load(arg.name) elif paramIsRef and argIsRef: - # ref -> ref: + # ref -> ref: # deref, store value in new local, and pass ref to new local self.visit(arg) temp = self.newLocal(None, arg.inferredType) @@ -707,6 +707,6 @@ def visitArg(self, funcType, paramIdx: int, arg: Expr): self.visit(arg) temp = self.newLocal(None, arg.inferredType) self.loadAddr(temp) - else: + else: # value/ref -> value : deref if necessary - self.visit(arg) \ No newline at end of file + self.visit(arg) diff --git a/compiler/closurevisitor.py b/compiler/closurevisitor.py index 33e3f65..57f6813 100644 --- a/compiler/closurevisitor.py +++ b/compiler/closurevisitor.py @@ -2,6 +2,7 @@ from .types import * from .visitor import Visitor from .varcollector import VarCollector +from typing import List class VarInstance: @@ -25,7 +26,7 @@ def merge(d1, d2): return combined -def deduplicate(ids: [Identifier]) -> [Identifier]: +def deduplicate(ids: List[Identifier]) -> List[Identifier]: seen = set() res = [] for i in ids: diff --git a/compiler/empty_list_typer.py b/compiler/empty_list_typer.py index 6940c3b..b424014 100644 --- a/compiler/empty_list_typer.py +++ b/compiler/empty_list_typer.py @@ -1,6 +1,7 @@ from .astnodes import * from .types import * from .visitor import Visitor +from typing import List # A visitor to refine the types of empty list literals @@ -23,7 +24,7 @@ def isEmptyListMultiAssign(self, node: Node): return False return True - def transformMultiAssign(self, node: AssignStmt) -> [AssignStmt]: + def transformMultiAssign(self, node: AssignStmt) -> List[AssignStmt]: statements = [] for t in node.targets: statements.append(AssignStmt(node.location, [t], node.value)) diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 051d856..fd6d9aa 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from collections import defaultdict +from typing import List import json @@ -13,7 +13,7 @@ class JvmBackend(CommonVisitor): defaultToGlobals = False # treat all vars as global if this is true def __init__(self, main: str, ts: TypeSystem): - self.classes = dict() + self.classes = dict() self.classes[main] = Builder(main) self.currentClass = main self.main = main # name of main class @@ -106,7 +106,7 @@ def newLocal(self, name: str = None, isRef: bool = True) -> int: self.locals[-1][name] = n return n - def visitStmtList(self, stmts: [Stmt]): + def visitStmtList(self, stmts: List[Stmt]): if len(stmts) == 0: self.instr("nop") else: diff --git a/compiler/parser.py b/compiler/parser.py index 2ef3a87..3abb5ae 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -1,5 +1,6 @@ from ast import * from .astnodes import * +from typing import List as Lst class ParseError(Exception): @@ -19,13 +20,13 @@ def __init__(self): # reduce a list of >2 expressions separated by a # left-associative operator into a BinaryExpr tree - def binaryReduce(self, op: str, values: [Expr]) -> Expr: + def binaryReduce(self, op: str, values: Lst[Expr]) -> Expr: current = BinaryExpr(values[0].location, values[0], op, values[1]) for v in values[2:]: current = BinaryExpr(values[0].location, current, op, v) return current - def getLocation(self, node) -> [int]: + def getLocation(self, node) -> Lst[int]: # input is Python AST node # get 2 item list corresponding to AST node starting location # make columns 1-indexed diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 1b921c8..4d83b65 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -1,10 +1,11 @@ from compiler.types.classvaluetype import ClassValueType from .valuetype import ValueType from .symboltype import SymbolType +from typing import List class FuncType(SymbolType): - def __init__(self, parameters: [ValueType], returnType: ValueType): + def __init__(self, parameters: List[ValueType], returnType: ValueType): self.parameters = parameters self.returnType = returnType self.refParams = {} diff --git a/compiler/varcollector.py b/compiler/varcollector.py index 4f001be..0b24e7f 100644 --- a/compiler/varcollector.py +++ b/compiler/varcollector.py @@ -1,4 +1,4 @@ - +from typing import List from .astnodes import * from .types import * from .visitor import Visitor @@ -14,7 +14,7 @@ def getVars(self, node: Node): self.visit(node) return self.vars - def getVarsFromList(self, nodes: [Node]): + def getVarsFromList(self, nodes: List[Node]): for n in nodes: self.visit(n) return self.vars From 570781694c7b9bac43ba3bb21cdae8688bfa571f Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 7 Sep 2022 21:40:59 -0700 Subject: [PATCH 24/79] support assert in parser --- compiler/parser.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/compiler/parser.py b/compiler/parser.py index 3abb5ae..bcdf5e0 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -368,6 +368,11 @@ def visit_arg(self, node): annotation = self.getTypeAnnotation(node.annotation) return TypedVar(location, identifier, annotation) + def visit_Assert(self, node): + location = self.getLocation(node) + func = Identifier(location, "__assert__") + return CallExpr(location, func, [self.visit(node.test)]) + # operators def visit_And(self, node): @@ -447,9 +452,6 @@ def visit_Raise(self, node): def visit_Try(self, node): raise ParseError("Unsupported", node) - def visit_Assert(self, node): - raise ParseError("Unsupported", node) - def visit_Import(self, node): raise ParseError("Unsupported", node) From 5c91d9a92ec765a64cfead68742554bbc4d08689 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 7 Sep 2022 21:44:09 -0700 Subject: [PATCH 25/79] int and bool only test, for wasm --- tests/runtime/int_and_bool.py | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 tests/runtime/int_and_bool.py diff --git a/tests/runtime/int_and_bool.py b/tests/runtime/int_and_bool.py new file mode 100644 index 0000000..e12936f --- /dev/null +++ b/tests/runtime/int_and_bool.py @@ -0,0 +1,31 @@ +x:int = 1 +y:int = 2 +a:bool = True +b:bool = False + +print(x) +print(y) +print(a) +print(b) + +assert x + y == 3 +assert x < y +assert y > x +assert x + x == 2 +assert y * y == 4 +assert 5 // 2 == y +assert 5 % 2 == x +assert x == x +assert x != y +assert not b +assert a +assert True +assert not False +assert a == a +assert a != b +assert a and a +assert a or b +assert not (b or b) +assert not (b and b) + + From d812d6faa07be27c0fb5c03f4c3d935efe20cc70 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 7 Sep 2022 21:51:25 -0700 Subject: [PATCH 26/79] fix assert parsing, update tests to use new assert, fix doc --- README.md | 2 +- compiler/parser.py | 2 +- tests/runtime/assignment.py | 26 ++--- tests/runtime/binary_tree.py | 14 +-- tests/runtime/classes.py | 18 +-- tests/runtime/contains.py | 32 +++--- tests/runtime/control_flow.py | 36 +++--- tests/runtime/doubling_vector.py | 68 +++++------ tests/runtime/exponent.py | 12 +- tests/runtime/functions.py | 22 ++-- tests/runtime/globals.py | 8 +- tests/runtime/incrementing_counter.py | 12 +- tests/runtime/linked_list.py | 18 +-- tests/runtime/lists.py | 156 +++++++++++++------------- tests/runtime/nested_list.py | 32 +++--- tests/runtime/nonlocal.py | 8 +- tests/runtime/operators.py | 52 ++++----- tests/runtime/ratio.py | 16 +-- tests/runtime/strings.py | 70 ++++++------ 19 files changed, 302 insertions(+), 302 deletions(-) diff --git a/README.md b/README.md index fd51222..bdbd348 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ The reference implementation represents a node's location as a four item list of The exact error messages from typechecking do not necessarily match the reference implementation, but the total number of messages (and the nodes that the messages are attached to) will match. -This compiler support an extra standard function, `__assert__`, which takes in a single `bool` argument and functions exactly like Python's `assert` statement. It is used in the test suite to assert values in runtime tests. +This compiler supports a limited version of Python's `assert` keyword. The `assert` may be followed by a single `bool` expression, which will raise an exception with an unspecified/generic message if the value is false. It is used in the test suite to assert values in runtime tests. ## JVM Backend Notes: diff --git a/compiler/parser.py b/compiler/parser.py index bcdf5e0..ab9c3af 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -371,7 +371,7 @@ def visit_arg(self, node): def visit_Assert(self, node): location = self.getLocation(node) func = Identifier(location, "__assert__") - return CallExpr(location, func, [self.visit(node.test)]) + return ExprStmt(location, CallExpr(location, func, [self.visit(node.test)])) # operators diff --git a/tests/runtime/assignment.py b/tests/runtime/assignment.py index 74218e1..9cbc059 100644 --- a/tests/runtime/assignment.py +++ b/tests/runtime/assignment.py @@ -8,31 +8,31 @@ b = a -__assert__(x != y) +assert x != y x = y -__assert__(x == y) +assert x == y x = 2 y = 2 -__assert__(x == y) +assert x == y x = y = 0 -__assert__(x == 0) -__assert__(y == 0) +assert x == 0 +assert y == 0 z = None -__assert__(z is None) +assert z is None -__assert__(a == b) +assert a == b b = False -__assert__(a != b) +assert a != b -__assert__(c == d) +assert c == d d = c = "123" -__assert__(c == d) +assert c == d d = "456" -__assert__(c != d) +assert c != d d = "123" -__assert__(c == d) +assert c == d z = print("1234") -__assert__(z is None) \ No newline at end of file +assert z is None diff --git a/tests/runtime/binary_tree.py b/tests/runtime/binary_tree.py index 8fccf04..177fd49 100644 --- a/tests/runtime/binary_tree.py +++ b/tests/runtime/binary_tree.py @@ -76,10 +76,10 @@ def makeNode(x: int) -> TreeNode: t.insert(i) i = i + 1 -__assert__(t.size == 175) -__assert__(t.contains(15)) -__assert__(t.contains(23)) -__assert__(t.contains(42)) -__assert__(not t.contains(4)) -__assert__(not t.contains(8)) -__assert__(not t.contains(16)) \ No newline at end of file +assert t.size == 175 +assert t.contains(15) +assert t.contains(23) +assert t.contains(42) +assert not t.contains(4) +assert not t.contains(8) +assert not t.contains(16) diff --git a/tests/runtime/classes.py b/tests/runtime/classes.py index 8c5ccd7..ec01a9e 100644 --- a/tests/runtime/classes.py +++ b/tests/runtime/classes.py @@ -29,29 +29,29 @@ def setZ(self: B, z:int): # constructors, getters, setters c1 = A() -__assert__(c1.y == 1) +assert c1.y == 1 c2 = B() -__assert__(c2.y == 5) -__assert__(c2.z == 5) +assert c2.y == 5 +assert c2.z == 5 c3 = B() -__assert__(c3.y == 5) +assert c3.y == 5 c2.y = 0 -__assert__(c2.y == 0) +assert c2.y == 0 # methods, dynamic dispatch c2.setZ(2) -__assert__(c2.z == 2) +assert c2.z == 2 x = 0 c1.t() -__assert__(x == 1) +assert x == 1 x = 0 c2.t() -__assert__(x == 2) +assert x == 2 x = 0 c3.t() -__assert__(x == 2) +assert x == 2 diff --git a/tests/runtime/contains.py b/tests/runtime/contains.py index 39a82ab..8cbe4eb 100644 --- a/tests/runtime/contains.py +++ b/tests/runtime/contains.py @@ -14,20 +14,20 @@ def contains2(items:[int], x:int) -> bool: return True return False -__assert__(contains([4, 8, 15, 16, 23], 15)) -__assert__(contains([4, 8, 15, 16, 23], 4)) -__assert__(contains([4, 8, 15, 16, 23], 8)) -__assert__(contains([4, 8, 15, 16, 23], 16)) -__assert__(contains([4, 8, 15, 16, 23], 23)) -__assert__(not contains([4, 8, 15, 16, 23], 999)) -__assert__(not contains([4], 15)) -__assert__(not contains([], 15)) +assert contains([4, 8, 15, 16, 23], 15) +assert contains([4, 8, 15, 16, 23], 4) +assert contains([4, 8, 15, 16, 23], 8) +assert contains([4, 8, 15, 16, 23], 16) +assert contains([4, 8, 15, 16, 23], 23) +assert not contains([4, 8, 15, 16, 23], 999) +assert not contains([4], 15) +assert not contains([], 15) -__assert__(contains2([4, 8, 15, 16, 23], 15)) -__assert__(contains2([4, 8, 15, 16, 23], 4)) -__assert__(contains2([4, 8, 15, 16, 23], 8)) -__assert__(contains2([4, 8, 15, 16, 23], 16)) -__assert__(contains2([4, 8, 15, 16, 23], 23)) -__assert__(not contains2([4, 8, 15, 16, 23], 999)) -__assert__(not contains2([4], 15)) -__assert__(not contains2([], 15)) +assert contains2([4, 8, 15, 16, 23], 15) +assert contains2([4, 8, 15, 16, 23], 4) +assert contains2([4, 8, 15, 16, 23], 8) +assert contains2([4, 8, 15, 16, 23], 16) +assert contains2([4, 8, 15, 16, 23], 23) +assert not contains2([4, 8, 15, 16, 23], 999) +assert not contains2([4], 15) +assert not contains2([], 15) diff --git a/tests/runtime/control_flow.py b/tests/runtime/control_flow.py index 7e8203e..10a7765 100644 --- a/tests/runtime/control_flow.py +++ b/tests/runtime/control_flow.py @@ -14,26 +14,26 @@ b = 0 if b == 0: b = 1 -__assert__(b == 1) +assert b == 1 b = 0 if b != 0: b = 2 -__assert__(b == 0) +assert b == 0 b = 0 if b != 0: b = 0 else: b = 1 -__assert__(b == 1) +assert b == 1 b = 0 if b == 0: b = 1 else: b = 0 -__assert__(b == 1) +assert b == 1 b = 0 if b > 0: @@ -42,7 +42,7 @@ b = 0 else: b = 1 -__assert__(b == 1) +assert b == 1 b = 0 if b == 0: @@ -51,7 +51,7 @@ b = 1 else: b = 1 -__assert__(b == 0) +assert b == 0 b = 5 if b == 0: @@ -60,7 +60,7 @@ b = 2 else: pass -__assert__(b == 5) +assert b == 5 b = -1 if b > 0: @@ -69,48 +69,48 @@ b = 1 else: b = 0 -__assert__(b == 1) +assert b == 1 b = -1 while b > 0: b = b - 1 -__assert__(b == -1) +assert b == -1 b = 5 while b > 0: b = b - 1 -__assert__(b == 0) +assert b == 0 for char in y: pass for char in y: z = char + z -__assert__(z == "321") -__assert__(char == "3") +assert z == "321" +assert char == "3" x = [1, 2, 3] for b in x: x[0] = b -__assert__(x[0] == 3) +assert x[0] == 3 x = [1, 2, 3] for b in x: c = c * 2 -__assert__(c == 800) -__assert__(b == 3) +assert c == 800 +assert b == 3 c = 100 x = [] for b in x: c = c * 2 -__assert__(c == 100) +assert c == 100 a = True d = [True, True, True, False] for a in d: pass -__assert__(not a) +assert not a a = True d = [True, True, True] @@ -122,4 +122,4 @@ f = [None, object(), None, object()] for e in f: pass -__assert__(not (e is None)) +assert not (e is None) diff --git a/tests/runtime/doubling_vector.py b/tests/runtime/doubling_vector.py index 2ae48ee..12342bb 100644 --- a/tests/runtime/doubling_vector.py +++ b/tests/runtime/doubling_vector.py @@ -57,55 +57,55 @@ def vrange(i:int, j:int) -> Vector: for num in [4, 8, 15, 16, 23, 42]: vec.append(num) -__assert__(vec.capacity() == 8) -__assert__(vec.size == 6) -__assert__(vec.items[0] == 4) -__assert__(vec.items[1] == 8) -__assert__(vec.items[2] == 15) -__assert__(vec.items[3] == 16) -__assert__(vec.items[4] == 23) -__assert__(vec.items[5] == 42) +assert vec.capacity() == 8 +assert vec.size == 6 +assert vec.items[0] == 4 +assert vec.items[1] == 8 +assert vec.items[2] == 15 +assert vec.items[3] == 16 +assert vec.items[4] == 23 +assert vec.items[5] == 42 # extras from doubling -__assert__(vec.items[6] == 15) -__assert__(vec.items[7] == 16) +assert vec.items[6] == 15 +assert vec.items[7] == 16 vec = Vector() for num in [4, 8, 15, 16, 23, 42]: vec.append(num) -__assert__(vec.capacity() == 6) -__assert__(vec.size == 6) -__assert__(vec.items[0] == 4) -__assert__(vec.items[1] == 8) -__assert__(vec.items[2] == 15) -__assert__(vec.items[3] == 16) -__assert__(vec.items[4] == 23) -__assert__(vec.items[5] == 42) +assert vec.capacity() == 6 +assert vec.size == 6 +assert vec.items[0] == 4 +assert vec.items[1] == 8 +assert vec.items[2] == 15 +assert vec.items[3] == 16 +assert vec.items[4] == 23 +assert vec.items[5] == 42 vec = vrange(0, 1) -__assert__(vec.capacity() == 1) -__assert__(vec.size == 1) -__assert__(vec.items[0] == 0) +assert vec.capacity() == 1 +assert vec.size == 1 +assert vec.items[0] == 0 vec = vrange(0, 2) -__assert__(vec.capacity() == 2) -__assert__(vec.size == 2) -__assert__(vec.items[0] == 0) -__assert__(vec.items[1] == 1) +assert vec.capacity() == 2 +assert vec.size == 2 +assert vec.items[0] == 0 +assert vec.items[1] == 1 vec = vrange(1, 3) -__assert__(vec.capacity() == 2) -__assert__(vec.size == 2) -__assert__(vec.items[0] == 1) -__assert__(vec.items[1] == 2) +assert vec.capacity() == 2 +assert vec.size == 2 +assert vec.items[0] == 1 +assert vec.items[1] == 2 vec = vrange(1, 1) -__assert__(vec.capacity() == 1) -__assert__(vec.size == 0) +assert vec.capacity() == 1 +assert vec.size == 0 vec = vrange(0, -1) -__assert__(vec.capacity() == 1) -__assert__(vec.size == 0) +assert vec.capacity() == 1 +assert vec.size == 0 vec = vrange(1, 100) -__assert__(vec.size == 99) +assert vec.size == 99 diff --git a/tests/runtime/exponent.py b/tests/runtime/exponent.py index d8dc752..a317a86 100644 --- a/tests/runtime/exponent.py +++ b/tests/runtime/exponent.py @@ -24,9 +24,9 @@ def geta() -> int: print(exp(2, i % 31)) i = i + 1 -__assert__(exp(2,3) == 8) -__assert__(exp(3,3) == 27) -__assert__(exp(3,4) == 81) -__assert__(exp(4,4) == 256) -__assert__(exp(5,1) == 5) -__assert__(exp(1,99) == 1) \ No newline at end of file +assert exp(2,3) == 8 +assert exp(3,3) == 27 +assert exp(3,4) == 81 +assert exp(4,4) == 256 +assert exp(5,1) == 5 +assert exp(1,99) == 1 diff --git a/tests/runtime/functions.py b/tests/runtime/functions.py index e61ffed..4d063b5 100644 --- a/tests/runtime/functions.py +++ b/tests/runtime/functions.py @@ -30,19 +30,19 @@ def f8(x:int)->int: y:object = None f1() -__assert__(f2() == 1) -__assert__(f3() is None) -__assert__(f4(1, None) == 2) -__assert__(f4(f2(), f3()) == 2) -__assert__(f5() is None) +assert f2() == 1 +assert f3() is None +assert f4(1, None) == 2 +assert f4(f2(), f3()) == 2 +assert f5() is None x = f4(f2(), f3()) y = f3() -__assert__(f4(x, y) == 3) -__assert__(f6() == 6) -__assert__(f7() == 5) -__assert__(f8(0) == 0) -__assert__(f8(1) == 1) -__assert__(f8(f7()) == 5) +assert f4(x, y) == 3 +assert f6() == 6 +assert f7() == 5 +assert f8(0) == 0 +assert f8(1) == 1 +assert f8(f7()) == 5 print(1) print(True) \ No newline at end of file diff --git a/tests/runtime/globals.py b/tests/runtime/globals.py index bf7e91c..20eac53 100644 --- a/tests/runtime/globals.py +++ b/tests/runtime/globals.py @@ -9,8 +9,8 @@ def t(): y = y + y __assert__(z == 0) -__assert__(x == 0) -__assert__(y == "a") +assert x == 0 +assert y == "a" t() -__assert__(x == 1) -__assert__(y == "aa") \ No newline at end of file +assert x == 1 +assert y == "aa" diff --git a/tests/runtime/incrementing_counter.py b/tests/runtime/incrementing_counter.py index 2c21d5d..07e8dcf 100644 --- a/tests/runtime/incrementing_counter.py +++ b/tests/runtime/incrementing_counter.py @@ -9,13 +9,13 @@ def inc(self : Counter): i : int = 0 c = Counter() c.inc() -__assert__(c.n == 1) +assert c.n == 1 c.inc() -__assert__(c.n == 2) +assert c.n == 2 c.inc() c.inc() -__assert__(c.n == 4) +assert c.n == 4 -# for i in [9,9,9,9,9,9]: -# c.inc() -# __assert__(c.n == 10) +for i in [9,9,9,9,9,9]: + c.inc() +assert c.n == 10 diff --git a/tests/runtime/linked_list.py b/tests/runtime/linked_list.py index fc3ba64..184e029 100644 --- a/tests/runtime/linked_list.py +++ b/tests/runtime/linked_list.py @@ -28,14 +28,14 @@ def add(self : "LinkedList", val : int): x:LinkedList = None x = LinkedList() -__assert__(x.is_empty()) -__assert__(x.length() == 0) +assert x.is_empty() +assert x.length() == 0 x.add(1) -__assert__(not x.is_empty()) -__assert__(x.length() == 1) -__assert__(x.head.val == 1) +assert not x.is_empty() +assert x.length() == 1 +assert x.head.val == 1 x.add(100) -__assert__(not x.is_empty()) -__assert__(x.length() == 2) -__assert__(x.head.val == 100) -__assert__(x.head.next.val == 1) \ No newline at end of file +assert not x.is_empty() +assert x.length() == 2 +assert x.head.val == 100 +assert x.head.next.val == 1 diff --git a/tests/runtime/lists.py b/tests/runtime/lists.py index e4712a9..9496e13 100644 --- a/tests/runtime/lists.py +++ b/tests/runtime/lists.py @@ -25,154 +25,154 @@ def getNestedIdx(lst:[[int]], idx:int)->[int]: a = [] b = [] -__assert__(len(x) == 0) -__assert__(len([]) == 0) -__assert__(len([1, 2, 3]) == 3) +assert len(x) == 0 +assert len([]) == 0 +assert len([1, 2, 3]) == 3 x = [1, 2, 3] -__assert__(len(x) == 3) -__assert__(x[0] == 1) -__assert__(x[1] == 2) -__assert__(x[2] == 3) +assert len(x) == 3 +assert x[0] == 1 +assert x[1] == 2 +assert x[2] == 3 x = [1, 2, 3] x = [0] + x -__assert__(len(x) == 4) -__assert__(x[0] == 0) -__assert__(x[1] == 1) -__assert__(x[2] == 2) -__assert__(x[3] == 3) +assert len(x) == 4 +assert x[0] == 0 +assert x[1] == 1 +assert x[2] == 2 +assert x[3] == 3 x = [1, 2, 3] x = x + [4] -__assert__(len(x) == 4) -__assert__(x[0] == 1) -__assert__(x[1] == 2) -__assert__(x[2] == 3) -__assert__(x[3] == 4) +assert len(x) == 4 +assert x[0] == 1 +assert x[1] == 2 +assert x[2] == 3 +assert x[3] == 4 y = ["1", "2", "3"] -__assert__(len(y) == 3) -__assert__(y[0] == "1") -__assert__(y[1] == "2") -__assert__(y[2] == "3") +assert len(y) == 3 +assert y[0] == "1" +assert y[1] == "2" +assert y[2] == "3" z = [None] -__assert__(len([None]) == 1) -__assert__([None][0] is None) +assert len([None]) == 1 +assert [None][0] is None z = [[]] -__assert__(len([[]]) == 1) -__assert__(len([[]][0]) == 0) +assert len([[]]) == 1 +assert len([[]][0]) == 0 a = [[object()]] -__assert__(len(a) == 1) -__assert__(len(a[0]) == 1) +assert len(a) == 1 +assert len(a[0]) == 1 y = ["1", "2", "3"] y = y + y -__assert__(len(y) == 6) -__assert__(y[0] == "1") -__assert__(y[1] == "2") -__assert__(y[2] == "3") -__assert__(y[3] == "1") -__assert__(y[4] == "2") -__assert__(y[5] == "3") +assert len(y) == 6 +assert y[0] == "1" +assert y[1] == "2" +assert y[2] == "3" +assert y[3] == "1" +assert y[4] == "2" +assert y[5] == "3" x = [1, 2, 3] x = x + x -__assert__(len(x) == 6) -__assert__(x[0] == 1) -__assert__(x[1] == 2) -__assert__(x[2] == 3) -__assert__(x[3] == 1) -__assert__(x[4] == 2) -__assert__(x[5] == 3) +assert len(x) == 6 +assert x[0] == 1 +assert x[1] == 2 +assert x[2] == 3 +assert x[3] == 1 +assert x[4] == 2 +assert x[5] == 3 w = [None] -__assert__(len(w) == 1) -__assert__(w[0] is None) +assert len(w) == 1 +assert w[0] is None w = [object()] -__assert__(len(w) == 1) -__assert__(not (w[0] is None)) +assert len(w) == 1 +assert not (w[0] is None) w = [object()] w[0] = None -__assert__(w[0] is None) +assert w[0] is None w[0] = object() -__assert__(not (w[0] is None)) +assert not (w[0] is None) x = [1, 2, 3] x[0] = 999 -__assert__(len(x) == 3) -__assert__(x[0] == 999) -__assert__(x[1] == 2) -__assert__(x[2] == 3) +assert len(x) == 3 +assert x[0] == 999 +assert x[1] == 2 +assert x[2] == 3 x2 = x x[1] = 999 -__assert__(x[1] == 999) -__assert__(x2[1] == 999) +assert x[1] == 999 +assert x2[1] == 999 x = [0, 1] x2[1] = 30 -__assert__(x[1] != 30) -__assert__(x2[1] == 30) +assert x[1] != 30 +assert x2[1] == 30 x = x2 x2 = x y = ["1", "2", "3"] y[2] = "a" -__assert__(len(y) == 3) -__assert__(y[0] == "1") -__assert__(y[1] == "2") -__assert__(y[2] == "a") +assert len(y) == 3 +assert y[0] == "1" +assert y[1] == "2" +assert y[2] == "a" y2 = y y2[1] = "aa" -__assert__(y2[1] == "aa") -__assert__(y2[1] == "aa") +assert y2[1] == "aa" +assert y2[1] == "aa" a = [None, [], [object(), None]] -__assert__(a[0] is None) -__assert__(len(a[1]) == 0) -__assert__(len(a[2]) == 2) -__assert__(not (a[2][0] is None)) -__assert__(a[2][1] is None) +assert a[0] is None +assert len(a[1]) == 0 +assert len(a[2]) == 2 +assert not (a[2][0] is None) +assert a[2][1] is None w = [] -__assert__(len(w) == 0) -__assert__(len(w + w) == 0) +assert len(w) == 0 +assert len(w + w) == 0 x = [] -__assert__(len(x) == 0) -__assert__(len(x + x) == 0) +assert len(x) == 0 +assert len(x + x) == 0 y = [] y = y + [""] -__assert__(len(y) == 1) -__assert__(len(y[0]) == 0) +assert len(y) == 1 +assert len(y[0]) == 0 x = [1,2,3] setIdx(x, 1, 0) -__assert__(x[1] == 0) +assert x[1] == 0 b = [[1, 1], [2], [3]] setNestedIdx(b, 0, 0, 1) -__assert__(b[0][0] == 1) +assert b[0][0] == 1 -__assert__(b[1][0] == 2) +assert b[1][0] == 2 x = getNestedIdx(b, 1) -__assert__(x[0] == 2) +assert x[0] == 2 x[0] = 1 -__assert__(b[1][0] == 1) +assert b[1][0] == 1 x = [1, 2, 3] x2 = x x[0] = 0 -__assert__(x2[0] == 0) +assert x2[0] == 0 x = [1] y = ["1"] diff --git a/tests/runtime/nested_list.py b/tests/runtime/nested_list.py index 1f7f26f..bba72af 100644 --- a/tests/runtime/nested_list.py +++ b/tests/runtime/nested_list.py @@ -11,40 +11,40 @@ # __assert__(len(a[0]) == 0) a = [[1]] -__assert__(len(a) == 1) -__assert__(len(a[0]) == 1) +assert len(a) == 1 +assert len(a[0]) == 1 a = [[1],[2, 2, 2],[3,3],[]] -__assert__(len(a) == 4) -__assert__(len(a[0]) == 1) -__assert__(len(a[1]) == 3) -__assert__(len(a[2]) == 2) -__assert__(len(a[3]) == 0) +assert len(a) == 4 +assert len(a[0]) == 1 +assert len(a[1]) == 3 +assert len(a[2]) == 2 +assert len(a[3]) == 0 -__assert__(a[0][0] == 1) -__assert__(a[1][0] == 2) -__assert__(a[1][1] == 2) -__assert__(a[1][2] == 2) +assert a[0][0] == 1 +assert a[1][0] == 2 +assert a[1][1] == 2 +assert a[1][2] == 2 a[0][0] = 5 -__assert__(a[0][0] == 5) +assert a[0][0] == 5 a[0] = [2, 2] -__assert__(len(a[0]) == 2) +assert len(a[0]) == 2 a[0] = a[0] + [3] -__assert__(len(a[0]) == 3) +assert len(a[0]) == 3 a = [[1],[1,1,1],[1,1],[]] c = 0 for b in a: c = c + len(b) -__assert__(c == 6) +assert c == 6 a = [[1],[2,3,4],[5,0],[]] c = 0 for b in a: for d in b: c = c + d -__assert__(c == 15) \ No newline at end of file +assert c == 15 diff --git a/tests/runtime/nonlocal.py b/tests/runtime/nonlocal.py index dc8cdc2..e5c3d4c 100644 --- a/tests/runtime/nonlocal.py +++ b/tests/runtime/nonlocal.py @@ -81,15 +81,15 @@ def test14(): b:Nonlocals = None # nonlocals can be mutated -__assert__(test(1) == 2) +assert test(1) == 2 -__assert__(test3() == 3) +assert test3() == 3 # nonlocals passed into functions cannot be mutated a = 0 test9(a) -__assert__(a == 0) +assert a == 0 # array idx's can be mutated w/o nonlocal test7() @@ -100,6 +100,6 @@ def test14(): b = Nonlocals() b.testMethod(a) b.testMethod(0) -__assert__(a == 0) +assert a == 0 b.testMethod(1) b.testMethod4() diff --git a/tests/runtime/operators.py b/tests/runtime/operators.py index 86c0db9..a7163e9 100644 --- a/tests/runtime/operators.py +++ b/tests/runtime/operators.py @@ -5,30 +5,30 @@ a:str = "123" b:str = "123" c:str = "456" -__assert__(w == x) -__assert__(y != x) -__assert__(b == b) -__assert__(a == b) -__assert__(b != c) -__assert__(y > x) -__assert__(y >= x) -__assert__(w >= x) -__assert__(x < y) -__assert__(x <= y) -__assert__(w <= x) -__assert__(w + x == y) -__assert__(y - x == w) -__assert__(w * x == x) -__assert__(5 // 2 == y) -__assert__(5 % 2 == x) -__assert__(z is z) -__assert__(not False) -__assert__(not (w != x)) -__assert__(-x == -1) -__assert__(True and True) -__assert__(True or False) -__assert__(False or True) -__assert__((False or True) and True) -__assert__(True if x != y else False) -__assert__(False if x == y else True) +assert w == x +assert y != x +assert b == b +assert a == b +assert b != c +assert y > x +assert y >= x +assert w >= x +assert x < y +assert x <= y +assert w <= x +assert w + x == y +assert y - x == w +assert w * x == x +assert 5 // 2 == y +assert 5 % 2 == x +assert z is z +assert not False +assert not (w != x) +assert -x == -1 +assert True and True +assert True or False +assert False or True +assert (False or True) and True +assert True if x != y else False +assert False if x == y else True diff --git a/tests/runtime/ratio.py b/tests/runtime/ratio.py index 1d28ae7..987c1ef 100644 --- a/tests/runtime/ratio.py +++ b/tests/runtime/ratio.py @@ -14,15 +14,15 @@ def mul(self : Rat, other : Rat) -> Rat: r3 : Rat = None r1 = Rat().new(4, 5) r2 = Rat().new(2, 3) -__assert__(r1.n == 4) -__assert__(r1.d == 5) -__assert__(r2.n == 2) -__assert__(r2.d == 3) +assert r1.n == 4 +assert r1.d == 5 +assert r2.n == 2 +assert r2.d == 3 r3 = r1.mul(r2) -__assert__(r3.n == 8) -__assert__(r3.d == 15) +assert r3.n == 8 +assert r3.d == 15 r3 = r3.mul(r2).mul(r2) -__assert__(r3.n == 32) -__assert__(r3.d == 135) \ No newline at end of file +assert r3.n == 32 +assert r3.d == 135 diff --git a/tests/runtime/strings.py b/tests/runtime/strings.py index 456f968..bd4d444 100644 --- a/tests/runtime/strings.py +++ b/tests/runtime/strings.py @@ -1,62 +1,62 @@ x:str = "123" y:str = "123" -__assert__(len(x) == 3) -__assert__(x == y) -__assert__(x == "123") -__assert__(y == x) -__assert__(x != "456") -__assert__(x[0] == "1") -__assert__(x[1] == "2") -__assert__(x[2] == "3") +assert len(x) == 3 +assert x == y +assert x == "123" +assert y == x +assert x != "456" +assert x[0] == "1" +assert x[1] == "2" +assert x[2] == "3" x = "123" x = x + "" -__assert__(x == "123") -__assert__(len(x) == 3) -__assert__(x[0] == "1") -__assert__(x[1] == "2") -__assert__(x[2] == "3") +assert x == "123" +assert len(x) == 3 +assert x[0] == "1" +assert x[1] == "2" +assert x[2] == "3" x = "123" x = "" + x -__assert__(x == "123") -__assert__(len(x) == 3) -__assert__(x[0] == "1") -__assert__(x[1] == "2") -__assert__(x[2] == "3") +assert x == "123" +assert len(x) == 3 +assert x[0] == "1" +assert x[1] == "2" +assert x[2] == "3" x = "123" x = x + "4" -__assert__(x == "1234") -__assert__(len(x) == 4) -__assert__(x[0] == "1") -__assert__(x[1] == "2") -__assert__(x[2] == "3") -__assert__(x[3] == "4") +assert x == "1234" +assert len(x) == 4 +assert x[0] == "1" +assert x[1] == "2" +assert x[2] == "3" +assert x[3] == "4" x = "123" x = "0" + x -__assert__(x == "0123") -__assert__(len(x) == 4) +assert x == "0123" +assert len(x) == 4 x = "123" x = x + y -__assert__(x == "123123") -__assert__(y == "123") -__assert__(len(x) == 6) -__assert__(len(y) == 3) +assert x == "123123" +assert y == "123" +assert len(x) == 6 +assert len(y) == 3 x = "123" x = x + x -__assert__(x == "123123") -__assert__(len(x) == 6) +assert x == "123123" +assert len(x) == 6 x = "123" y = x x = "0" -__assert__(y == "123") -__assert__(len(x) == 1) -__assert__(len(y) == 3) +assert y == "123" +assert len(x) == 1 +assert len(y) == 3 From f33c9249a746e4557bfb42a2a413cbeb881db36b Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 8 Sep 2022 00:48:44 -0700 Subject: [PATCH 27/79] v1 wasm - untested --- README.md | 20 +- compiler/compiler.py | 8 + compiler/jvm_backend.py | 3 +- compiler/parser.py | 6 +- compiler/types/classvaluetype.py | 16 + compiler/wasm_backend.py | 336 +++++++++++++++++++++ main.py | 14 +- tests/runtime/control_flow.py | 70 ----- tests/runtime/control_flow_2.py | 72 +++++ tests/runtime/int_and_bool.py | 7 + tests/runtime/int_and_bool_control_flow.py | 33 ++ tests/runtime/int_and_bool_funcs.py | 15 + 12 files changed, 522 insertions(+), 78 deletions(-) create mode 100644 compiler/wasm_backend.py create mode 100644 tests/runtime/control_flow_2.py create mode 100644 tests/runtime/int_and_bool_control_flow.py create mode 100644 tests/runtime/int_and_bool_funcs.py diff --git a/README.md b/README.md index bdbd348..b9a278e 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ The input file should have extension `.py`. If the output file is not provided, - `hoist` - output untyped Python 3 source code w/o nonlocals or nested function definitions - `jvm` - output JVM bytecode formatted for the Krakatau assembler - `cil` - output CIL bytecode formatted for the Mono ilasm assembler + - `wasm` - output WASM as plaintext in WAT format (WIP) ## Differences from the reference implementation: @@ -105,6 +106,23 @@ The CIL backend for this compiler outputs CIL bytecode in plaintext formatted fo The `demo_cil.sh` script is a useful utility to compile and run files with the CIL backend with a single command (provide the path to the input source file as an argument). - To run the same example as above, run `./demo_cil.sh tests/runtime/binary_tree.py` +## WASM Backend Notes: + +WIP + +Planned features: +- ints and bools +- binary operators and assignment +- control flow +- print and assert + +Not-currently-planned features: +- classes/objects +- arrays +- strings +- nested functions +- global/nonlocal + ## FAQ - What is this for? @@ -116,4 +134,4 @@ The `demo_cil.sh` script is a useful utility to compile and run files with the C - Why implement this in Python? - Since Chocopy is a subset of Python, implementing the compiler in Python means I do not have to write my own lexer and parser. This was explicitly something I wanted to experiment with while writing the frontend, and it worked wonderfully. The secondary reason is that writing it in Python means I can prototype new ideas faster. The lack of type safety in the compiler codebase is mitigated by an extensive test suite. -Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. The runtime test suite for the JVM backend were evaluated using Java 8 on my local machine. +Most of the test cases are taken from test suites included in the release code for CS164, with some additional tests written for more coverage. Tests include both static validation of generated/annotated ASTs, as well as runtime tests that check the correctness of output code. diff --git a/compiler/compiler.py b/compiler/compiler.py index c5ceb3e..1b37f84 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -10,6 +10,7 @@ from .jvm_backend import JvmBackend from .cil_backend import CilBackend from .python_backend import PythonBackend +from .wasm_backend import WasmBackend import ast from pathlib import Path @@ -74,3 +75,10 @@ def emitCIL(self, main: str, ast: Node): cil_backend = CilBackend(main, self.transformer.ts) cil_backend.visit(ast) return cil_backend.builder + + def emitWASM(self, main: str, ast: Node): + self.closurepass(ast) + EmptyListTyper().visit(ast) + wasm_backend = WasmBackend(main, self.transformer.ts) + wasm_backend.visit(ast) + return wasm_backend.builder diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index fd6d9aa..09f9035 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -675,8 +675,7 @@ def emit_input(self): "invokespecial Method java/util/Scanner (Ljava/io/InputStream;)V") l = self.newLocal() self.instr(f"aload {l}") - self.currentBuilder().addLine( - "invokevirtual Method java/util/Scanner nextLine ()Ljava/lang/String;") + self.instr("invokevirtual Method java/util/Scanner nextLine ()Ljava/lang/String;") def emit_len(self, arg: Expr): t = arg.inferredType diff --git a/compiler/parser.py b/compiler/parser.py index ab9c3af..b2abc52 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -1,6 +1,6 @@ from ast import * from .astnodes import * -from typing import List as Lst +import typing class ParseError(Exception): @@ -20,13 +20,13 @@ def __init__(self): # reduce a list of >2 expressions separated by a # left-associative operator into a BinaryExpr tree - def binaryReduce(self, op: str, values: Lst[Expr]) -> Expr: + def binaryReduce(self, op: str, values: typing.List[Expr]) -> Expr: current = BinaryExpr(values[0].location, values[0], op, values[1]) for v in values[2:]: current = BinaryExpr(values[0].location, current, op, v) return current - def getLocation(self, node) -> Lst[int]: + def getLocation(self, node) -> typing.List[int]: # input is Python AST node # get 2 item list corresponding to AST node starting location # make columns 1-indexed diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 8927cb5..19a1a25 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -88,6 +88,22 @@ def getCILName(self): else: return "class "+self.className + def getWasmName(self): + if self.className == "bool": + return "i32" + elif self.className == "str": + raise Exception("unsupported") + elif self.className == "object": + raise Exception("unsupported") + elif self.className == "": + raise Exception("unsupported") + elif self.className == "": + raise Exception("unsupported") + elif self.className == "int": + return "i64" + else: + raise Exception("unsupported") + def __str__(self): return self.className diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py new file mode 100644 index 0000000..dae2f5c --- /dev/null +++ b/compiler/wasm_backend.py @@ -0,0 +1,336 @@ +from .astnodes import * +from .types import * +from .builder import Builder +from .typesystem import TypeSystem +from .visitor import CommonVisitor +from typing import List + +class WasmBuilder(Builder): + def __init__(self, name: str): + super(WasmBuilder, self).__init__(name) + + def module(self): + self.newLine("(module") + self.indent() + + def block(self, name: str): + self.newLine(f"(block ${name}") + self.indent() + + def loop(self, name: str): + self.newLine(f"(loop ${name}") + self.indent() + + def param(self, name:str, type: str)->str: + return f"(param ${name} {type})" + + def func(self, name:str, params: List[str]=[], resType=None): + params = " ".join(params) + result = "" + if resType is not None: + result = f" (result {resType})" + self.newLine(f"(func ${name} {params}{result}") + self.indent() + + def end(self): + self.unindent() + self.newLine(")") + +class WasmBackend(CommonVisitor): + defaultToGlobals = False # treat all vars as global if this is true + localCounter = 0 + + def __init__(self, main: str, ts: TypeSystem): + self.builder = WasmBuilder(main) + self.main = main # name of main method + self.ts = ts + self.enterScope() + + + def currentBuilder(self): + return self.classes[self.currentClass] + + def newLabelName(self) -> str: + self.counter += 1 + return "label_"+str(self.counter) + + def instr(self, instr: str): + self.builder.newLine(instr) + + def store(self, name: str): + self.instr(f"local.set ${name}") + + def load(self, name: str): + self.instr(f"local.get ${name}") + + def genLocalName(self) -> str: + self.localCounter+=1 + return f"local_{self.localCounter}" + + def newLocal(self, name: str = None, t: str = "i64")->str: + # store the top of stack as a new local + if name is None: + name = self.genLocalName() + self.instr(f"(local ${name} {t})") + self.store(name) + return name + + def visitStmtList(self, stmts: List[Stmt]): + if len(stmts) == 0: + self.instr("nop") + else: + for s in stmts: + self.visit(s) + + def Program(self, node: Program): + func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] + var_decls = [d for d in node.declarations if isinstance(d, VarDef)] + self.builder.module() + self.instr('(import "console" "log" (func $log_int (param i64)))') + self.instr('(import "console" "log" (func $log_bool (param i64)))') + self.instr('(import "console" "assert" (func $assert (param i64)))') + for v in var_decls: + self.instr(f"(global ${v.var.identifier.name} {v.var.t.getWasmName()}") + self.builder.indent() + self.visit(v.value) + self.builder.end() + for d in func_decls: + self.visit(d) + self.builder.func("main") + self.defaultToGlobals = True + self.visitStmtList(node.statements) + self.defaultToGlobals = False + self.builder.end() + self.instr(f"(start $main)") + self.builder.end() + + def FuncDef(self, node: FuncDef): + params = [self.builder.param(p.identifier.name, p.t.getWasmName()) for p in node.params] + self.returnType = node.type.returnType + ret = None if self.returnType.isNone() else self.returnType.getWasmName() + self.builder.func(node.name.name, params, ret) + for d in node.declarations: + self.visit(d) + self.visitStmtList(node.statements) + self.builder.end() + + def VarDef(self, node: VarDef): + varName = node.var.identifier.name + if node.isAttr: + raise Exception("TODO") + elif node.var.varInstance.isNonlocal: + raise Exception("TODO") + else: + self.visit(node.value) + self.newLocal(varName, node.value.inferredType.getWasmName()) + + # # STATEMENTS + + def processAssignmentTarget(self, target: Expr): + if isinstance(target, Identifier): + if self.defaultToGlobals or target.varInstance.isGlobal: + self.instr(f"global.set ${target.name}") + elif target.varInstance.isNonlocal: + raise Exception("TODO") + else: + self.store(target.name) + elif isinstance(target, IndexExpr): + raise Exception("TODO") + elif isinstance(target, MemberExpr): + raise Exception("TODO") + else: + raise Exception( + "Internal compiler error: unsupported assignment target") + + def AssignStmt(self, node: AssignStmt): + self.visit(node.value) + targets = node.targets[::-1] + if len(targets) > 1: + name = self.newLocal(None, node.value.inferredType.getWasmName()) + for t in targets: + self.load(name) + self.processAssignmentTarget(t) + else: + self.processAssignmentTarget(targets[0]) + + def IfStmt(self, node: IfStmt): + self.visit(node.condition) + self.instr("(if") + self.builder.indent() + self.instr("(then") + self.builder.indent() + self.visitStmtList(node.thenBody) + self.builder.end() + self.instr("(else") + self.builder.indent() + self.visitStmtList(node.elseBody) + self.builder.end() + self.builder.end() + + def ExprStmt(self, node: ExprStmt): + self.visit(node.expr) + self.instr("drop") + + def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: + return leftType.isListType() and rightType.isListType() and operator == "+" + + def BinaryExpr(self, node: BinaryExpr): + operator = node.operator + leftType = node.left.inferredType + rightType = node.right.inferredType + self.visit(node.left) + self.visit(node.right) + # concatenation and addition + if operator == "+": + if self.isListConcat(operator, leftType, rightType): + raise Exception("TODO") + elif leftType == StrType(): + raise Exception("TODO") + elif leftType == IntType(): + self.instr("i64.add") + else: + raise Exception( + "Internal compiler error: unexpected operand types for +") + # other arithmetic operators + elif operator == "-": + self.instr("i64.sub") + elif operator == "*": + self.instr("i64.mul") + elif operator == "//": + self.instr("i64.div_s") + elif operator == "%": + self.instr("i64.rem_s") + # relational operators + elif operator == "<": + self.instr("i64.lt_s") + elif operator == "<=": + self.instr("i64.gt_s") + self.instr("i64.eqz") + elif operator == ">": + self.instr("i64.gt_s") + elif operator == ">=": + self.instr("i64.lt_s") + self.instr("i64.eqz") + elif operator == "==": + # TODO: refs + self.instr("i64.eq") + elif operator == "!=": + self.instr("i64.ne") + elif operator == "is": + raise Exception("TODO") + # logical operators + elif operator == "and": + self.instr("i64.and") + elif operator == "or": + self.instr("i64.or") + else: + raise Exception( + f"Internal compiler error: unexpected operator {operator}") + + def UnaryExpr(self, node: UnaryExpr): + if node.operator == "-": + self.instr("i64.const 0") + self.visit(node.operand) + self.instr("i64.sub") + elif node.operator == "not": + self.visit(node.operand) + self.instr("i64.eqz") + + def CallExpr(self, node: CallExpr): + name = node.function.name + if node.isConstructor: + raise Exception("TODO") + if name == "print": + self.emit_print(node.args[0]) + elif name == "len": + raise Exception("TODO") + elif name == "input": + raise Exception("TODO") + elif name == "__assert__": + self.emit_assert(node.args[0]) + else: + for i in range(len(node.args)): + self.visit(node.args[i]) + self.instr(f"call ${name}") + if node.function.inferredType.returnType.isNone(): + self.NoneLiteral(None) # push null for void return + + def WhileStmt(self, node: WhileStmt): + block = self.newLabelName() + loop = self.newLabelName() + self.builder.block(block) + self.builder.loop(loop) + self.visit(node.condition) + self.instr(f"i64.eqz") + self.instr(f"br_if ${block}") + for s in node.body: + self.visit(s) + self.instr(f"br ${loop}") + self.builder.end() + self.builder.end() + + def buildReturn(self, value: Expr): + if self.returnType.isNone(): + self.instr("return") + else: + if value is None: + self.NoneLiteral(None) + else: + self.visit(value) + self.instr("return") + + def ReturnStmt(self, node: ReturnStmt): + self.buildReturn(node.value) + + def Identifier(self, node: Identifier): + if self.defaultToGlobals or node.varInstance.isGlobal: + self.instr(f"global.get ${node.name}") + elif node.varInstance.isNonlocal: + raise Exception("TODO") + else: + self.instr(f"local.get ${node.name}") + + def IfExpr(self, node: IfExpr): + self.visit(node.condition) + self.instr("(if") + self.builder.indent() + self.instr("(then") + self.builder.indent() + self.visit(node.thenExpr) + self.builder.end() + self.instr("(else") + self.builder.indent() + self.visit(node.elseExpr) + self.builder.end() + self.builder.end() + + # # LITERALS + + def BooleanLiteral(self, node: BooleanLiteral): + if node.value: + self.instr(f"i64.const 1") + else: + self.instr(f"i64.const 0") + + def IntegerLiteral(self, node: IntegerLiteral): + self.instr(f"i64.const f{node.value}") + + def NoneLiteral(self, node: NoneLiteral): + self.instr(f"i64.const 0") + + # # BUILT-INS - note: these are in-lined + def emit_assert(self, arg: Expr): + self.visit(arg) + self.instr("call $assert") + self.instr("i64.const 0") + + def emit_print(self, arg: Expr): + if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: + raise Exception(f"Built-in function print is unsupported for values of type {arg.inferredType.classname}") + self.visit(arg) + self.instr(f"call $log_{arg.inferredType.className}") + self.instr("i64.const 0") + + + + diff --git a/main.py b/main.py index f355d03..dc0e485 100644 --- a/main.py +++ b/main.py @@ -12,6 +12,7 @@ 'hoist - output untyped Python 3 source code w/o nonlocals or nested function definitions\n' + 'jvm - output JVM bytecode formatted for the Krakatau assembler\n' 'cil - output CIL bytecode formatted for the Mono ilasm assembler\n' + 'wasm - output WASM in WAT format\n' ) def out_msg(path, verbose): @@ -20,7 +21,7 @@ def out_msg(path, verbose): def main(): parser = argparse.ArgumentParser(description='Chocopy frontend') - parser.add_argument('--mode', dest='mode', choices=["parse", "tc", "python", "jvm", "hoist", "cil"], default="python", + parser.add_argument('--mode', dest='mode', choices=["parse", "tc", "python", "jvm", "hoist", "cil", "wasm"], default="python", help=mode_help) parser.add_argument('--print', dest='should_print', action="iframe.php?url=https%3A%2F%2Fgithub.com%2Fstore_true", help="output to stdout instead of file") @@ -122,7 +123,16 @@ def main(): fname = outdir + cil_emitter.name + ".cil" with open(fname, "w") as f: out_msg(fname, args.verbose) - f.write(cil_emitter.emit()) + f.write(cil_emitter.emit()) + elif args.mode == "wasm": + wat_emitter = compiler.emitWASM(infile_name, tree) + if args.should_print: + print(wat_emitter.emit()) + else: + fname = outdir + wat_emitter.name + ".wat" + with open(fname, "w") as f: + out_msg(fname, args.verbose) + f.write(wat_emitter.emit()) if __name__ == "__main__": main() diff --git a/tests/runtime/control_flow.py b/tests/runtime/control_flow.py index 10a7765..7963992 100644 --- a/tests/runtime/control_flow.py +++ b/tests/runtime/control_flow.py @@ -11,76 +11,6 @@ x = [] -b = 0 -if b == 0: - b = 1 -assert b == 1 - -b = 0 -if b != 0: - b = 2 -assert b == 0 - -b = 0 -if b != 0: - b = 0 -else: - b = 1 -assert b == 1 - -b = 0 -if b == 0: - b = 1 -else: - b = 0 -assert b == 1 - -b = 0 -if b > 0: - b = 0 -elif b < 0: - b = 0 -else: - b = 1 -assert b == 1 - -b = 0 -if b == 0: - pass -elif b < 0: - b = 1 -else: - b = 1 -assert b == 0 - -b = 5 -if b == 0: - b = 1 -elif b < 0: - b = 2 -else: - pass -assert b == 5 - -b = -1 -if b > 0: - b = 0 -elif b < 0: - b = 1 -else: - b = 0 -assert b == 1 - -b = -1 -while b > 0: - b = b - 1 -assert b == -1 - -b = 5 -while b > 0: - b = b - 1 -assert b == 0 - for char in y: pass diff --git a/tests/runtime/control_flow_2.py b/tests/runtime/control_flow_2.py new file mode 100644 index 0000000..63e30f4 --- /dev/null +++ b/tests/runtime/control_flow_2.py @@ -0,0 +1,72 @@ +b:int = 0 + + +b = 0 +if b == 0: + b = 1 +assert b == 1 + +b = 0 +if b != 0: + b = 2 +assert b == 0 + +b = 0 +if b != 0: + b = 0 +else: + b = 1 +assert b == 1 + +b = 0 +if b == 0: + b = 1 +else: + b = 0 +assert b == 1 + +b = 0 +if b > 0: + b = 0 +elif b < 0: + b = 0 +else: + b = 1 +assert b == 1 + +b = 0 +if b == 0: + pass +elif b < 0: + b = 1 +else: + b = 1 +assert b == 0 + +b = 5 +if b == 0: + b = 1 +elif b < 0: + b = 2 +else: + pass +assert b == 5 + +b = -1 +if b > 0: + b = 0 +elif b < 0: + b = 1 +else: + b = 0 +assert b == 1 + +b = -1 +while b > 0: + b = b - 1 +assert b == -1 + +b = 5 +while b > 0: + b = b - 1 +assert b == 0 \ No newline at end of file diff --git a/tests/runtime/int_and_bool.py b/tests/runtime/int_and_bool.py index e12936f..14166a2 100644 --- a/tests/runtime/int_and_bool.py +++ b/tests/runtime/int_and_bool.py @@ -28,4 +28,11 @@ assert not (b or b) assert not (b and b) +x = y +assert x == y +assert x == 2 + +x = y = 3 +assert x == y +assert x == 3 diff --git a/tests/runtime/int_and_bool_control_flow.py b/tests/runtime/int_and_bool_control_flow.py new file mode 100644 index 0000000..387cc3a --- /dev/null +++ b/tests/runtime/int_and_bool_control_flow.py @@ -0,0 +1,33 @@ +x:int = 1 +y:int = 2 +a:bool = True +b:bool = False + +if a: + assert True + +if a: + assert True +else: + assert False + +if b: + assert False + +if b: + assert False +else: + assert True + +if x == y: + assert False +else: + assert True + +if x == x: + assert True +else: + assert False + +assert (5 if a else 0) == 5 +assert (0 if b else 5) == 5 \ No newline at end of file diff --git a/tests/runtime/int_and_bool_funcs.py b/tests/runtime/int_and_bool_funcs.py new file mode 100644 index 0000000..69b9c1f --- /dev/null +++ b/tests/runtime/int_and_bool_funcs.py @@ -0,0 +1,15 @@ +x:int = 1 +y:int = 2 +a:bool = True +b:bool = False + +def test1(a1:int, a2:int)->int: + return a1 + a2 + +def test2(a1:bool)->bool: + return not a1 + +assert test1(x, y) == 3 +assert test1(x, x) == 2 +assert test2(b) +assert not test2(a) From bdeecff74ac44a154be183ada3d4a224a79020e5 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 8 Sep 2022 15:23:39 -0700 Subject: [PATCH 28/79] [wasm] working int and bool funcs, control flow, printing, assert --- Makefile | 4 +- compiler/types/classvaluetype.py | 10 +-- compiler/wasm_backend.py | 80 ++++++++++++++------ demo_wasm.sh | 11 +++ int_and_bool_control_flow.wasm | Bin 0 -> 263 bytes int_and_bool_control_flow.wat | 121 +++++++++++++++++++++++++++++++ wasm.js | 31 ++++++++ 7 files changed, 230 insertions(+), 27 deletions(-) create mode 100755 demo_wasm.sh create mode 100644 int_and_bool_control_flow.wasm create mode 100644 int_and_bool_control_flow.wat create mode 100644 wasm.js diff --git a/Makefile b/Makefile index d728f56..558c1df 100644 --- a/Makefile +++ b/Makefile @@ -6,4 +6,6 @@ clean: rm -f *.ast rm -f *.ast.typed rm -f *.test.py - rm -f *.out.py \ No newline at end of file + rm -f *.out.py + rm -f *.wasm + rm -f *.wat \ No newline at end of file diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 19a1a25..8570bfd 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -92,17 +92,17 @@ def getWasmName(self): if self.className == "bool": return "i32" elif self.className == "str": - raise Exception("unsupported") + raise Exception("TODO") elif self.className == "object": - raise Exception("unsupported") + raise Exception("TODO") elif self.className == "": - raise Exception("unsupported") + raise Exception("TODO") elif self.className == "": - raise Exception("unsupported") + raise Exception("TODO") elif self.className == "int": return "i64" else: - raise Exception("unsupported") + raise Exception("TODO") def __str__(self): return self.className diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index dae2f5c..2cdaa08 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -36,16 +36,34 @@ def end(self): self.unindent() self.newLine(")") + def emit(self) -> str: + lines = [] + for l in self.lines: + if isinstance(l, str): + if " drop" in l and " i64.const 0" in lines[-1]: + lines[-1] = None + continue + lines.append(l) + else: + lines.append(l.emit()) + lines = [l for l in lines if l is not None] + return "\n".join(lines) + + def newBlock(self): + child = WasmBuilder(self.name) + child.indentation = self.indentation + self.lines.append(child) + return child + class WasmBackend(CommonVisitor): defaultToGlobals = False # treat all vars as global if this is true localCounter = 0 + locals = None def __init__(self, main: str, ts: TypeSystem): self.builder = WasmBuilder(main) self.main = main # name of main method self.ts = ts - self.enterScope() - def currentBuilder(self): return self.classes[self.currentClass] @@ -68,11 +86,10 @@ def genLocalName(self) -> str: return f"local_{self.localCounter}" def newLocal(self, name: str = None, t: str = "i64")->str: - # store the top of stack as a new local + # add a new local decl, does not store anything if name is None: name = self.genLocalName() - self.instr(f"(local ${name} {t})") - self.store(name) + self.locals.newLine(f"(local ${name} {t})") return name def visitStmtList(self, stmts: List[Stmt]): @@ -86,25 +103,33 @@ def Program(self, node: Program): func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] var_decls = [d for d in node.declarations if isinstance(d, VarDef)] self.builder.module() - self.instr('(import "console" "log" (func $log_int (param i64)))') - self.instr('(import "console" "log" (func $log_bool (param i64)))') - self.instr('(import "console" "assert" (func $assert (param i64)))') + self.instr('(import "imports" "logInt" (func $log_int (param i64)))') + self.instr('(import "imports" "logBool" (func $log_bool (param i32)))') + self.instr('(import "imports" "logString" (func $log_str (param i64)))') + + self.instr('(import "imports" "assert" (func $assert (param i32)))') for v in var_decls: - self.instr(f"(global ${v.var.identifier.name} {v.var.t.getWasmName()}") - self.builder.indent() + self.instr(f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()})") self.visit(v.value) - self.builder.end() + self.instr(f")") for d in func_decls: self.visit(d) + module_builder = self.builder + self.builder = module_builder.newBlock() + self.builder.func("main") self.defaultToGlobals = True + self.locals = self.builder.newBlock() self.visitStmtList(node.statements) self.defaultToGlobals = False self.builder.end() + + self.builder = module_builder self.instr(f"(start $main)") self.builder.end() def FuncDef(self, node: FuncDef): + self.locals = self.builder.newBlock() params = [self.builder.param(p.identifier.name, p.t.getWasmName()) for p in node.params] self.returnType = node.type.returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() @@ -122,7 +147,8 @@ def VarDef(self, node: VarDef): raise Exception("TODO") else: self.visit(node.value) - self.newLocal(varName, node.value.inferredType.getWasmName()) + n = self.newLocal(varName, node.value.inferredType.getWasmName()) + self.store(n) # # STATEMENTS @@ -147,6 +173,7 @@ def AssignStmt(self, node: AssignStmt): targets = node.targets[::-1] if len(targets) > 1: name = self.newLocal(None, node.value.inferredType.getWasmName()) + self.store(name) for t in targets: self.load(name) self.processAssignmentTarget(t) @@ -213,16 +240,22 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("i64.eqz") elif operator == "==": # TODO: refs - self.instr("i64.eq") + if leftType == BoolType(): + self.instr("i32.eq") + else: + self.instr("i64.eq") elif operator == "!=": - self.instr("i64.ne") + if leftType == BoolType(): + self.instr("i32.ne") + else: + self.instr("i64.ne") elif operator == "is": raise Exception("TODO") # logical operators elif operator == "and": - self.instr("i64.and") + self.instr("i32.and") elif operator == "or": - self.instr("i64.or") + self.instr("i32.or") else: raise Exception( f"Internal compiler error: unexpected operator {operator}") @@ -234,7 +267,7 @@ def UnaryExpr(self, node: UnaryExpr): self.instr("i64.sub") elif node.operator == "not": self.visit(node.operand) - self.instr("i64.eqz") + self.instr("i32.eqz") def CallExpr(self, node: CallExpr): name = node.function.name @@ -261,7 +294,7 @@ def WhileStmt(self, node: WhileStmt): self.builder.block(block) self.builder.loop(loop) self.visit(node.condition) - self.instr(f"i64.eqz") + self.instr(f"i32.eqz") self.instr(f"br_if ${block}") for s in node.body: self.visit(s) @@ -291,29 +324,34 @@ def Identifier(self, node: Identifier): self.instr(f"local.get ${node.name}") def IfExpr(self, node: IfExpr): + n = self.newLocal(None, node.inferredType.getWasmName()) self.visit(node.condition) self.instr("(if") self.builder.indent() self.instr("(then") self.builder.indent() self.visit(node.thenExpr) + self.store(n) self.builder.end() self.instr("(else") self.builder.indent() self.visit(node.elseExpr) + self.store(n) self.builder.end() self.builder.end() + self.load(n) + # # LITERALS def BooleanLiteral(self, node: BooleanLiteral): if node.value: - self.instr(f"i64.const 1") + self.instr(f"i32.const 1") else: - self.instr(f"i64.const 0") + self.instr(f"i32.const 0") def IntegerLiteral(self, node: IntegerLiteral): - self.instr(f"i64.const f{node.value}") + self.instr(f"i64.const {node.value}") def NoneLiteral(self, node: NoneLiteral): self.instr(f"i64.const 0") diff --git a/demo_wasm.sh b/demo_wasm.sh new file mode 100755 index 0000000..e639622 --- /dev/null +++ b/demo_wasm.sh @@ -0,0 +1,11 @@ +# utility for compiling a Chocopy file to .wasm files and running it +base_name="$(basename $1 .py)" + +rm -f *.wat +rm -f *.wasm + +python3 main.py --mode wasm $1 . +wat2wasm $base_name.wat -o $base_name.wasm +echo "Running program $base_name..." +node wasm.js $base_name.wasm + diff --git a/int_and_bool_control_flow.wasm b/int_and_bool_control_flow.wasm new file mode 100644 index 0000000000000000000000000000000000000000..c604791e5ca63f6f338db817c03d6ad726d3a304 GIT binary patch literal 263 zcmYL>F%E)26hvosSx~`{v&Y>uI` logString(x), + logInt: x => logInt(x), + logBool: x => logBool(x), + assert: x => console.assert(x) + } +}; + +const fs = require('fs'); + +const wasmBuffer = fs.readFileSync(wasm_path); +WebAssembly.instantiate(wasmBuffer, importObject); \ No newline at end of file From 4ddd4cb62dc8a60f3fedefd0b889e2ae6909190e Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 8 Sep 2022 17:42:02 -0700 Subject: [PATCH 29/79] simple string instantiation, length - TODO: concat, equality --- compiler/types/classvaluetype.py | 8 +- compiler/wasm_backend.py | 62 ++++++++++++-- int_and_bool_control_flow.wasm | Bin 263 -> 0 bytes int_and_bool_control_flow.wat | 121 --------------------------- simple_string.wasm | Bin 0 -> 385 bytes simple_string.wat | 136 +++++++++++++++++++++++++++++++ tests/runtime/simple_string.py | 9 ++ wasm.js | 10 ++- 8 files changed, 208 insertions(+), 138 deletions(-) delete mode 100644 int_and_bool_control_flow.wasm delete mode 100644 int_and_bool_control_flow.wat create mode 100644 simple_string.wasm create mode 100644 simple_string.wat create mode 100644 tests/runtime/simple_string.py diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 8570bfd..708e0ef 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -92,13 +92,13 @@ def getWasmName(self): if self.className == "bool": return "i32" elif self.className == "str": - raise Exception("TODO") + return "i32" elif self.className == "object": - raise Exception("TODO") + return "i32" elif self.className == "": - raise Exception("TODO") + return "i32" elif self.className == "": - raise Exception("TODO") + return "i32" elif self.className == "int": return "i64" else: diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 2cdaa08..870c833 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -40,7 +40,7 @@ def emit(self) -> str: lines = [] for l in self.lines: if isinstance(l, str): - if " drop" in l and " i64.const 0" in lines[-1]: + if " drop" in l and " i32.const 0" in lines[-1]: lines[-1] = None continue lines.append(l) @@ -99,18 +99,29 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) + def loadMemoryCounter(self): + self.instr("i32.const 0") # addr 0 + self.instr("i32.load") + + def incrMemoryCounter(self, n:int): + self.instr("i32.const 0") # addr 0 + self.instr(f"i32.const {n}") + self.loadMemoryCounter() + self.instr("i32.add") + self.instr("i32.store") # alignment: 64 bit + def Program(self, node: Program): func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] var_decls = [d for d in node.declarations if isinstance(d, VarDef)] self.builder.module() self.instr('(import "imports" "logInt" (func $log_int (param i64)))') self.instr('(import "imports" "logBool" (func $log_bool (param i32)))') - self.instr('(import "imports" "logString" (func $log_str (param i64)))') - + self.instr('(import "imports" "logString" (func $log_str (param i32)))') self.instr('(import "imports" "assert" (func $assert (param i32)))') + self.instr('(memory (import "js" "mem") 1)') for v in var_decls: self.instr(f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()})") - self.visit(v.value) + self.instr(f"{v.var.t.getWasmName()}.const 0") self.instr(f")") for d in func_decls: self.visit(d) @@ -120,6 +131,12 @@ def Program(self, node: Program): self.builder.func("main") self.defaultToGlobals = True self.locals = self.builder.newBlock() + self.instr("i32.const 0") # addr 0 + self.instr("i32.const 8") # store value 8 + self.instr("i32.store") + for v in var_decls: + self.visit(v.value) + self.instr(f"global.set ${v.getIdentifier().name}") self.visitStmtList(node.statements) self.defaultToGlobals = False self.builder.end() @@ -276,7 +293,7 @@ def CallExpr(self, node: CallExpr): if name == "print": self.emit_print(node.args[0]) elif name == "len": - raise Exception("TODO") + self.emit_len(node.args[0]) elif name == "input": raise Exception("TODO") elif name == "__assert__": @@ -341,7 +358,6 @@ def IfExpr(self, node: IfExpr): self.builder.end() self.load(n) - # # LITERALS def BooleanLiteral(self, node: BooleanLiteral): @@ -354,20 +370,48 @@ def IntegerLiteral(self, node: IntegerLiteral): self.instr(f"i64.const {node.value}") def NoneLiteral(self, node: NoneLiteral): - self.instr(f"i64.const 0") + self.instr(f"i32.const 0") + + def StringLiteral(self, node: StringLiteral): + length = len(node.value) + # store the length + self.loadMemoryCounter() # addr: mem + addr = self.newLocal(None, "i32") + self.instr(f"local.tee ${addr}") # store memory counter + self.instr(f"i32.const {length}") # value + self.instr(f"i32.store") # alignment: 32-bit + for i in range(length): + offset = i + 4 + val = ord(node.value[i]) + # addr: mem + 4 + idx + self.loadMemoryCounter() + self.instr(f"i32.const {offset}") + self.instr("i32.add") + self.instr(f"i32.const {val}") + self.instr("i32.store8") + memory = length + 4 + increase = 8 + (8 * (memory // 8)) + self.incrMemoryCounter(increase) + # load the address the string was stored at to the stack + self.load(addr) # # BUILT-INS - note: these are in-lined def emit_assert(self, arg: Expr): self.visit(arg) self.instr("call $assert") - self.instr("i64.const 0") + self.NoneLiteral(None) def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: raise Exception(f"Built-in function print is unsupported for values of type {arg.inferredType.classname}") self.visit(arg) self.instr(f"call $log_{arg.inferredType.className}") - self.instr("i64.const 0") + self.NoneLiteral(None) + + def emit_len(self, arg: Expr): + self.visit(arg) + self.instr("i32.load") + self.instr("i64.extend_i32_u") diff --git a/int_and_bool_control_flow.wasm b/int_and_bool_control_flow.wasm deleted file mode 100644 index c604791e5ca63f6f338db817c03d6ad726d3a304..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 263 zcmYL>F%E)26hvosSx~`{v&Y>uI`wxf>(u3#3C;^{Q!5Zf5dCJRcGh&Mj$J8EhSY zmi=K98Uo;ztNDSN18P9lNw#m^+pZ7kq20Y6dtd^YfmPccz=>QKS@+%kxSLtJ31Qpy zp!ndLZ6lx)N?N#qisjsLE-t^xFVYPP&7TE|Tfq&~O<5OS?_O|PXyWz$JfvP9;-Gmv z^LBHM>i37K;*Y;eRz^pGX!0L{ooMFu@vP3hKAqIXtd>ej6jIVktVYx4%Dok?WQ-W2 J#F$uI{s241KwJO- literal 0 HcmV?d00001 diff --git a/simple_string.wat b/simple_string.wat new file mode 100644 index 0000000..f761d96 --- /dev/null +++ b/simple_string.wat @@ -0,0 +1,136 @@ +(module + (import "imports" "logInt" (func $log_int (param i64))) + (import "imports" "logBool" (func $log_bool (param i32))) + (import "imports" "logString" (func $log_str (param i32))) + (import "imports" "assert" (func $assert (param i32))) + (memory (import "js" "mem") 1) + (global $x (mut i32) + i32.const 0 + ) + (global $y (mut i32) + i32.const 0 + ) + (global $z (mut i32) + i32.const 0 + ) + (func $main + (local $local_1 i32) + (local $local_2 i32) + (local $local_3 i32) + i32.const 0 + i32.const 8 + i32.store + i32.const 0 + i32.load + local.tee $local_1 + i32.const 3 + i32.store + i32.const 0 + i32.load + i32.const 4 + i32.add + i32.const 49 + i32.store8 + i32.const 0 + i32.load + i32.const 5 + i32.add + i32.const 50 + i32.store8 + i32.const 0 + i32.load + i32.const 6 + i32.add + i32.const 51 + i32.store8 + i32.const 0 + i32.const 8 + i32.const 0 + i32.load + i32.add + i32.store + local.get $local_1 + global.set $x + i32.const 0 + i32.load + local.tee $local_2 + i32.const 0 + i32.store + i32.const 0 + i32.const 8 + i32.const 0 + i32.load + i32.add + i32.store + local.get $local_2 + global.set $y + i32.const 0 + i32.load + local.tee $local_3 + i32.const 5 + i32.store + i32.const 0 + i32.load + i32.const 4 + i32.add + i32.const 49 + i32.store8 + i32.const 0 + i32.load + i32.const 5 + i32.add + i32.const 50 + i32.store8 + i32.const 0 + i32.load + i32.const 6 + i32.add + i32.const 51 + i32.store8 + i32.const 0 + i32.load + i32.const 7 + i32.add + i32.const 52 + i32.store8 + i32.const 0 + i32.load + i32.const 8 + i32.add + i32.const 53 + i32.store8 + i32.const 0 + i32.const 16 + i32.const 0 + i32.load + i32.add + i32.store + local.get $local_3 + global.set $z + global.get $x + call $log_str + global.get $y + call $log_str + global.get $z + call $log_str + global.get $x + i32.load + i64.extend_i32_u + i64.const 3 + i64.eq + call $assert + global.get $y + i32.load + i64.extend_i32_u + i64.const 0 + i64.eq + call $assert + global.get $z + i32.load + i64.extend_i32_u + i64.const 5 + i64.eq + call $assert + ) + (start $main) +) \ No newline at end of file diff --git a/tests/runtime/simple_string.py b/tests/runtime/simple_string.py new file mode 100644 index 0000000..3cd0d35 --- /dev/null +++ b/tests/runtime/simple_string.py @@ -0,0 +1,9 @@ +x:str = "123" +y:str = "" +z:str = "12345" +print(x) +print(y) +print(z) +assert len(x) == 3 +assert len(y) == 0 +assert len(z) == 5 \ No newline at end of file diff --git a/wasm.js b/wasm.js index b8a6830..e530e54 100644 --- a/wasm.js +++ b/wasm.js @@ -1,9 +1,8 @@ const wasm_path = process.argv[2]; function logString(offset) { - const length = 0; - // TODO: get length from buffer - const bytes = new Uint8Array(memory.buffer, offset, length); + const length = new Uint32Array(memory.buffer, offset, 1)[0]; + const bytes = new Uint8Array(memory.buffer, offset + 4, Number(length)); const string = new TextDecoder('utf8').decode(bytes); console.log(string); } @@ -16,13 +15,16 @@ function logBool(val) { console.log(val !== 0); } +const memory = new WebAssembly.Memory({ initial: 2, maximum: 100 }); + const importObject = { imports: { logString: x => logString(x), logInt: x => logInt(x), logBool: x => logBool(x), assert: x => console.assert(x) - } + }, + js: { mem: memory }, }; const fs = require('fs'); From 20ef0ddf15eb48f5a1d0bd03ba9dff635dd743e2 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 8 Sep 2022 17:53:47 -0700 Subject: [PATCH 30/79] add some more tests for is --- compiler/types/classvaluetype.py | 2 +- compiler/wasm_backend.py | 20 ++++++++++++-------- tests/runtime/lists.py | 4 ++++ tests/runtime/operators.py | 8 +++++++- 4 files changed, 24 insertions(+), 10 deletions(-) diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 708e0ef..f3d79c6 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -102,7 +102,7 @@ def getWasmName(self): elif self.className == "int": return "i64" else: - raise Exception("TODO") + return "i32" def __str__(self): return self.className diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 870c833..c1397cb 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -119,6 +119,7 @@ def Program(self, node: Program): self.instr('(import "imports" "logString" (func $log_str (param i32)))') self.instr('(import "imports" "assert" (func $assert (param i32)))') self.instr('(memory (import "js" "mem") 1)') + # initialize all globals to 0 for now, since we don't statically allocate strings or arrays for v in var_decls: self.instr(f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()})") self.instr(f"{v.var.t.getWasmName()}.const 0") @@ -131,9 +132,11 @@ def Program(self, node: Program): self.builder.func("main") self.defaultToGlobals = True self.locals = self.builder.newBlock() + # initialize memory counter self.instr("i32.const 0") # addr 0 self.instr("i32.const 8") # store value 8 self.instr("i32.store") + # initialize globals for v in var_decls: self.visit(v.value) self.instr(f"global.set ${v.getIdentifier().name}") @@ -256,18 +259,19 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("i64.lt_s") self.instr("i64.eqz") elif operator == "==": - # TODO: refs - if leftType == BoolType(): - self.instr("i32.eq") - else: + # TODO: refs, string + if leftType == IntType(): self.instr("i64.eq") - elif operator == "!=": - if leftType == BoolType(): - self.instr("i32.ne") else: + self.instr("i32.eq") + elif operator == "!=": + if leftType == IntType(): self.instr("i64.ne") + else: + self.instr("i32.ne") elif operator == "is": - raise Exception("TODO") + # pointer comparisons + self.instr("i32.eq") # logical operators elif operator == "and": self.instr("i32.and") diff --git a/tests/runtime/lists.py b/tests/runtime/lists.py index 9496e13..3313131 100644 --- a/tests/runtime/lists.py +++ b/tests/runtime/lists.py @@ -25,6 +25,10 @@ def getNestedIdx(lst:[[int]], idx:int)->[int]: a = [] b = [] +assert b is b +assert not (a is b) +assert not (a is None) + assert len(x) == 0 assert len([]) == 0 assert len([1, 2, 3]) == 3 diff --git a/tests/runtime/operators.py b/tests/runtime/operators.py index a7163e9..d331320 100644 --- a/tests/runtime/operators.py +++ b/tests/runtime/operators.py @@ -21,7 +21,6 @@ assert w * x == x assert 5 // 2 == y assert 5 % 2 == x -assert z is z assert not False assert not (w != x) assert -x == -1 @@ -32,3 +31,10 @@ assert True if x != y else False assert False if x == y else True +assert z is z +assert None is None +assert not (object() is object()) +assert z is None +z = object() +assert z is z + From 3f77434ad3ea0bed076f1fa1ac4333ef7ce71ee6 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 8 Sep 2022 17:58:21 -0700 Subject: [PATCH 31/79] update readme, clean --- README.md | 32 +++++++---- simple_string.wasm | Bin 385 -> 0 bytes simple_string.wat | 136 --------------------------------------------- 3 files changed, 21 insertions(+), 147 deletions(-) delete mode 100644 simple_string.wasm delete mode 100644 simple_string.wat diff --git a/README.md b/README.md index b9a278e..13c30c8 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ The test suite includes both static validation of generated/annotated ASTs, as w - CIL Backend Requirements: - [Mono](https://www.mono-project.com/) - Tested with Mono 6.12 +- WASM Backend Requirements: + - [WebAssembly Binary Toolkit (wabt)](https://github.com/WebAssembly/wabt) ## Usage @@ -108,20 +110,28 @@ The `demo_cil.sh` script is a useful utility to compile and run files with the C ## WASM Backend Notes: -WIP +This is very much WIP. -Planned features: -- ints and bools -- binary operators and assignment +Supported: +- int +- bool +- string (partial) +- most operators +- assignment - control flow -- print and assert - -Not-currently-planned features: -- classes/objects -- arrays -- strings +- print, len, and assert +- globals + +Unsupported: +- class/object +- array +- nonlocal +- string equality, concatenation +- input + +Unclear/Untested: +- `None` - nested functions -- global/nonlocal ## FAQ diff --git a/simple_string.wasm b/simple_string.wasm deleted file mode 100644 index 10040e9d223f60a243fb03f8e48d245daeefc5cc..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 385 zcmb7-y-veW4293}ZGNB(c?3iwxf>(u3#3C;^{Q!5Zf5dCJRcGh&Mj$J8EhSY zmi=K98Uo;ztNDSN18P9lNw#m^+pZ7kq20Y6dtd^YfmPccz=>QKS@+%kxSLtJ31Qpy zp!ndLZ6lx)N?N#qisjsLE-t^xFVYPP&7TE|Tfq&~O<5OS?_O|PXyWz$JfvP9;-Gmv z^LBHM>i37K;*Y;eRz^pGX!0L{ooMFu@vP3hKAqIXtd>ej6jIVktVYx4%Dok?WQ-W2 J#F$uI{s241KwJO- diff --git a/simple_string.wat b/simple_string.wat deleted file mode 100644 index f761d96..0000000 --- a/simple_string.wat +++ /dev/null @@ -1,136 +0,0 @@ -(module - (import "imports" "logInt" (func $log_int (param i64))) - (import "imports" "logBool" (func $log_bool (param i32))) - (import "imports" "logString" (func $log_str (param i32))) - (import "imports" "assert" (func $assert (param i32))) - (memory (import "js" "mem") 1) - (global $x (mut i32) - i32.const 0 - ) - (global $y (mut i32) - i32.const 0 - ) - (global $z (mut i32) - i32.const 0 - ) - (func $main - (local $local_1 i32) - (local $local_2 i32) - (local $local_3 i32) - i32.const 0 - i32.const 8 - i32.store - i32.const 0 - i32.load - local.tee $local_1 - i32.const 3 - i32.store - i32.const 0 - i32.load - i32.const 4 - i32.add - i32.const 49 - i32.store8 - i32.const 0 - i32.load - i32.const 5 - i32.add - i32.const 50 - i32.store8 - i32.const 0 - i32.load - i32.const 6 - i32.add - i32.const 51 - i32.store8 - i32.const 0 - i32.const 8 - i32.const 0 - i32.load - i32.add - i32.store - local.get $local_1 - global.set $x - i32.const 0 - i32.load - local.tee $local_2 - i32.const 0 - i32.store - i32.const 0 - i32.const 8 - i32.const 0 - i32.load - i32.add - i32.store - local.get $local_2 - global.set $y - i32.const 0 - i32.load - local.tee $local_3 - i32.const 5 - i32.store - i32.const 0 - i32.load - i32.const 4 - i32.add - i32.const 49 - i32.store8 - i32.const 0 - i32.load - i32.const 5 - i32.add - i32.const 50 - i32.store8 - i32.const 0 - i32.load - i32.const 6 - i32.add - i32.const 51 - i32.store8 - i32.const 0 - i32.load - i32.const 7 - i32.add - i32.const 52 - i32.store8 - i32.const 0 - i32.load - i32.const 8 - i32.add - i32.const 53 - i32.store8 - i32.const 0 - i32.const 16 - i32.const 0 - i32.load - i32.add - i32.store - local.get $local_3 - global.set $z - global.get $x - call $log_str - global.get $y - call $log_str - global.get $z - call $log_str - global.get $x - i32.load - i64.extend_i32_u - i64.const 3 - i64.eq - call $assert - global.get $y - i32.load - i64.extend_i32_u - i64.const 0 - i64.eq - call $assert - global.get $z - i32.load - i64.extend_i32_u - i64.const 5 - i64.eq - call $assert - ) - (start $main) -) \ No newline at end of file From b080cc9fb22a3a2ee27f9bceaedffcf847a175ba Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 8 Sep 2022 19:21:35 -0700 Subject: [PATCH 32/79] work on strings and arrays, setup test suite 7/26 --- README.md | 7 +- compiler/astnodes/typedvar.py | 3 + compiler/types/classvaluetype.py | 2 + compiler/wasm_backend.py | 195 +++++++++++++++++++++++++------ test.py | 105 ++++++++++++++++- wasm.js | 11 +- 6 files changed, 278 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 13c30c8..ddb09e9 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ This is very much WIP. Supported: - int - bool -- string (partial) +- string (literals, len) - most operators - assignment - control flow @@ -123,10 +123,11 @@ Supported: - globals Unsupported: +- for-loops - class/object -- array +- list - nonlocal -- string equality, concatenation +- string equality, concatenation, iteration - input Unclear/Untested: diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 407b826..ac4e8fc 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -21,3 +21,6 @@ def toJSON(self, dump_location=True): d["identifier"] = self.identifier.toJSON(dump_location) d["type"] = self.type.toJSON(dump_location) return d + + def getWasmParam(self): + return f"(param ${self.identifier.name} {self.t.getWasmName()})" diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index f3d79c6..f7d40da 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -89,6 +89,8 @@ def getCILName(self): return "class "+self.className def getWasmName(self): + # bools are i32, ints are i64 + # all others are pointers/refs, which are i32 if self.className == "bool": return "i32" elif self.className == "str": diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index c1397cb..56048d0 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -21,8 +21,17 @@ def loop(self, name: str): self.newLine(f"(loop ${name}") self.indent() - def param(self, name:str, type: str)->str: - return f"(param ${name} {type})" + def _if(self): + self.newLine(f"(if") + self.indent() + + def _then(self): + self.newLine(f"(then") + self.indent() + + def _else(self): + self.newLine(f"(else") + self.indent() def func(self, name:str, params: List[str]=[], resType=None): params = " ".join(params) @@ -75,10 +84,13 @@ def newLabelName(self) -> str: def instr(self, instr: str): self.builder.newLine(instr) - def store(self, name: str): + def setLocal(self, name: str): self.instr(f"local.set ${name}") - def load(self, name: str): + def teeLocal(self, name: str): + self.instr(f"local.tee ${name}") + + def loadLocal(self, name: str): self.instr(f"local.get ${name}") def genLocalName(self) -> str: @@ -150,7 +162,7 @@ def Program(self, node: Program): def FuncDef(self, node: FuncDef): self.locals = self.builder.newBlock() - params = [self.builder.param(p.identifier.name, p.t.getWasmName()) for p in node.params] + params = [p.getWasmParam() for p in node.params] self.returnType = node.type.returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() self.builder.func(node.name.name, params, ret) @@ -168,20 +180,32 @@ def VarDef(self, node: VarDef): else: self.visit(node.value) n = self.newLocal(varName, node.value.inferredType.getWasmName()) - self.store(n) + self.setLocal(n) # # STATEMENTS - def processAssignmentTarget(self, target: Expr): + def processAssignmentTarget(self, target: Expr, name: str): + # name is the name of the local that the value is stored in if isinstance(target, Identifier): + self.loadLocal(name) if self.defaultToGlobals or target.varInstance.isGlobal: self.instr(f"global.set ${target.name}") elif target.varInstance.isNonlocal: raise Exception("TODO") else: - self.store(target.name) + self.setLocal(target.name) elif isinstance(target, IndexExpr): - raise Exception("TODO") + lst, idx = self.validateIdx(target) + # 8 * idx + 4 + list addr + self.loadLocal(idx) + self.instr("i32.const 8") + self.instr("i32.mul") + self.instr("i32.const 4") + self.instr("i32.add") + self.loadLocal(lst) + self.instr("i32.add") + self.loadLocal(name) + self.instr("i32.store") elif isinstance(target, MemberExpr): raise Exception("TODO") else: @@ -191,25 +215,18 @@ def processAssignmentTarget(self, target: Expr): def AssignStmt(self, node: AssignStmt): self.visit(node.value) targets = node.targets[::-1] - if len(targets) > 1: - name = self.newLocal(None, node.value.inferredType.getWasmName()) - self.store(name) - for t in targets: - self.load(name) - self.processAssignmentTarget(t) - else: - self.processAssignmentTarget(targets[0]) + name = self.newLocal(None, node.value.inferredType.getWasmName()) + self.setLocal(name) + for t in targets: + self.processAssignmentTarget(t, name) def IfStmt(self, node: IfStmt): self.visit(node.condition) - self.instr("(if") - self.builder.indent() - self.instr("(then") - self.builder.indent() + self.builder._if() + self.builder._then() self.visitStmtList(node.thenBody) self.builder.end() - self.instr("(else") - self.builder.indent() + self.builder._else() self.visitStmtList(node.elseBody) self.builder.end() self.builder.end() @@ -262,11 +279,15 @@ def BinaryExpr(self, node: BinaryExpr): # TODO: refs, string if leftType == IntType(): self.instr("i64.eq") + elif leftType == StrType(): + raise Exception("TODO") else: self.instr("i32.eq") elif operator == "!=": if leftType == IntType(): self.instr("i64.ne") + elif leftType == StrType(): + raise Exception("TODO") else: self.instr("i32.ne") elif operator == "is": @@ -347,20 +368,121 @@ def Identifier(self, node: Identifier): def IfExpr(self, node: IfExpr): n = self.newLocal(None, node.inferredType.getWasmName()) self.visit(node.condition) - self.instr("(if") - self.builder.indent() - self.instr("(then") - self.builder.indent() + self.builder._if() + self.builder._then() self.visit(node.thenExpr) - self.store(n) + self.setLocal(n) self.builder.end() - self.instr("(else") - self.builder.indent() + self.builder._else() self.visit(node.elseExpr) - self.store(n) + self.setLocal(n) self.builder.end() self.builder.end() - self.load(n) + self.loadLocal(n) + + def ListExpr(self, node: ListExpr): + length = len(node.elements) + t = node.inferredType + elementType = None + if isinstance(t, ClassValueType): + if node.emptyListType: + elementType = node.emptyListType + else: + elementType = ClassValueType("object") + else: + elementType = t.elementType + # store the length + self.loadMemoryCounter() # addr: mem + addr = self.newLocal(None, "i32") + self.teeLocal(addr) # store memory counter + self.instr(f"i32.const {length}") # value + self.instr(f"i32.store") # alignment: 32-bit + # unlike strings, each item in the list gets 64 bits instead of 8 + for i in range(length): + offset = i * 8 + 4 + # addr: mem + 4 + idx * 8 + self.loadMemoryCounter() + self.instr(f"i32.const {offset}") + self.instr("i32.add") + self.visit(node.elements[i]) + self.instr(f"{elementType.getWasmName()}.store") + memory = length * 8 + 4 + increase = 8 + (8 * (memory // 8)) + self.incrMemoryCounter(increase) + # load the address the list was stored at to the stack + self.loadLocal(addr) + + def validateIdx(self, node: IndexExpr): + self.visit(node.list) + lst = self.newLocal(None, "i32") + self.teeLocal(lst) + self.instr("i32.load") + + self.visit(node.index) + self.instr("i32.wrap_i64") + idx = self.newLocal(None, "i32") + self.teeLocal(idx) + + # make sure idx < length + self.instr("i32.gt_s") + self.instr("i32.eqz") + self.builder._if() + self.builder._then() + self.visit(node.thenExpr) + self.instr("unreachable") + self.builder.end() + self.builder.end() + + # make sure idx >= 0 + self.instr('i32.const 0') + self.loadLocal(idx) + self.instr('i32.gt_s') + self.builder._if() + self.builder._then() + self.visit(node.thenExpr) + self.instr("unreachable") + self.builder.end() + self.builder.end() + return lst, idx + + def IndexExpr(self, node: IndexExpr): + lst, idx = self.validateIdx(node) + + if node.list.inferredType.isListType(): + # 8 * idx + 4 + list addr + self.loadLocal(idx) + self.instr("i32.const 8") + self.instr("i32.mul") + self.instr("i32.const 4") + self.instr("i32.add") + self.loadLocal(lst) + self.instr("i32.add") + self.instr("i32.load") + else: + # store the length + self.loadMemoryCounter() # addr: mem + addr = self.newLocal(None, "i32") + self.teeLocal(addr) # store memory counter + self.instr(f"i32.const 1") # value + self.instr(f"i32.store") + + # addr of single char + self.loadMemoryCounter() + self.instr(f"i32.const 4") + self.instr("i32.add") + + # idx + 4 + list addr + self.loadLocal(idx) + self.instr("i32.const 4") + self.instr("i32.add") + self.loadLocal(lst) + self.instr("i32.add") + self.instr("i32.load8_u") + + self.instr("i32.store8") + self.incrMemoryCounter(8) + # load the address the string was stored at to the stack + self.loadLocal(addr) # # LITERALS @@ -381,9 +503,9 @@ def StringLiteral(self, node: StringLiteral): # store the length self.loadMemoryCounter() # addr: mem addr = self.newLocal(None, "i32") - self.instr(f"local.tee ${addr}") # store memory counter + self.teeLocal(addr) # store memory counter self.instr(f"i32.const {length}") # value - self.instr(f"i32.store") # alignment: 32-bit + self.instr(f"i32.store") for i in range(length): offset = i + 4 val = ord(node.value[i]) @@ -397,9 +519,9 @@ def StringLiteral(self, node: StringLiteral): increase = 8 + (8 * (memory // 8)) self.incrMemoryCounter(increase) # load the address the string was stored at to the stack - self.load(addr) + self.loadLocal(addr) - # # BUILT-INS - note: these are in-lined + # # BUILT-INS def emit_assert(self, arg: Expr): self.visit(arg) self.instr("call $assert") @@ -413,6 +535,7 @@ def emit_print(self, arg: Expr): self.NoneLiteral(None) def emit_len(self, arg: Expr): + # the length of a string or array is always in the first 4 bytes self.visit(arg) self.instr("i32.load") self.instr("i64.extend_i32_u") diff --git a/test.py b/test.py index 59fce23..1df6b28 100644 --- a/test.py +++ b/test.py @@ -11,6 +11,35 @@ dump_location = True error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected"} +disabled_wasm_tests = [ + "assignment.py", + "functions.py", + "nested_list.py", + "binary_tree.py", + "globals.py", + "nonlocal.py", + "classes.py", + # "hello_world.py", + "nonlocal_builtins.py", + "contains.py", + "incrementing_counter.py", + "operators.py", + "control_flow.py", + # "int_and_bool.py", + "ratio.py", + # "control_flow_2.py", + # "int_and_bool_control_flow.py", + # "simple_string.py", + "doubling_vector.py", + # "int_and_bool_funcs.py", + "strings.py", + "exponent.py", + "linked_list.py", + # "var_decl.py", + # "expr_stmt.py", + "lists.py" +] + def run_all_tests(): run_parse_tests() run_typecheck_tests() @@ -18,6 +47,7 @@ def run_all_tests(): run_closure_tests() run_jvm_tests() run_cil_tests() + run_wasm_tests() def run_parse_tests(): print("Running parser tests...\n") @@ -149,6 +179,34 @@ def run_python_backend_tests(): print("\nNot all test cases passed. Please run `make clean` after inspecting the output") print("\nPassed {:d} out of {:d} Python backend runtime test cases\n".format(n_passed, total)) +def run_wasm_tests(): + print("Running WASM backend tests...\n") + total = 0 + n_passed = 0 + wasm_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() + for test in wasm_tests_dir.glob('*.py'): + skip = False + for disabled in disabled_wasm_tests: + if disabled in str(test): + skip = True + break + if skip: + print("Skipping: " + str(test) + "\n") + continue + passed = run_wasm_test(test) + total += 1 + if not passed: + print("Failed: "+ str(test) + "\n") + else: + n_passed += 1 + if total == n_passed: + subprocess.run("cd {} && rm -f *.wat && rm -f *.wasm".format( + str(Path(__file__).parent.resolve()) + ), shell=True) + else: + print("\nNot all test cases passed. Please run `make clean` after inspecting the output") + print("\nPassed {:d} out of {:d} WASM backend test cases\n".format(n_passed, total)) + def run_jvm_tests(): print("Running JVM backend tests...\n") total = 0 @@ -298,7 +356,6 @@ def run_closure_runtime_test(test)->bool: return False def run_python_emit_test(test)->bool: - infile_name = str(test)[:-3].split("/")[-1] try: compiler = Compiler() astparser = compiler.parser @@ -395,8 +452,9 @@ def run_jvm_test(test)->bool: def run_cil_test(test)->bool: passed = True + name = str(test.name[:-3]) try: - infile_name = str(test)[:-3].split("/")[-1] + infile_name = name.split("/")[-1] outdir = "./" compiler = Compiler() astparser = compiler.parser @@ -418,11 +476,11 @@ def run_cil_test(test)->bool: print(track) return False try: - assembler_commands = ["ilasm {}.cil".format(str(test.name[:-3]))] + assembler_commands = [f"ilasm {name}.cil"] output = subprocess.check_output("cd {} && {} && mono {}.exe".format( str(Path(__file__).parent.resolve()), " && ".join(assembler_commands), - str(test.name[:-3]) + name ), shell=True) lines = output.decode().split("\n") for l in lines: @@ -436,6 +494,45 @@ def run_cil_test(test)->bool: return False return passed +def run_wasm_test(test)->bool: + passed = True + name = str(test.name[:-3]) + try: + infile_name = name.split("/")[-1] + outdir = "./" + compiler = Compiler() + astparser = compiler.parser + ast = compiler.parse(test) + if len(astparser.errors) > 0: + return False + compiler.typecheck(ast) + if len(ast.errors.errors) > 0: + print(ast.errors.toJSON(False)) + return False + wasm_emitter = compiler.emitWASM(infile_name, ast) + fname = outdir + name + ".wat" + with open(fname, "w") as f: + f.write(wasm_emitter.emit()) + except Exception as e: + print("Internal compiler error:", test) + track = traceback.format_exc() + print(e) + print(track) + return False + try: + output = subprocess.check_output(f"wat2wasm {name}.wat -o {name}.wasm && node wasm.js {name}.wasm", shell=True) + lines = output.decode().split("\n") + for l in lines: + for e in error_flags: + if e in l: + passed = False + print(l) + break + except Exception as e: + print(e) + return False + return passed + def ast_equals(d1, d2)->bool: # precondition: the input dict must represent a well-formed AST # d1 is the correct AST, d2 is the AST output by this compiler diff --git a/wasm.js b/wasm.js index e530e54..3c33c37 100644 --- a/wasm.js +++ b/wasm.js @@ -1,21 +1,28 @@ const wasm_path = process.argv[2]; +// utils for pretty-printing ints, bools, strings + function logString(offset) { + // first 4 bytes is the length const length = new Uint32Array(memory.buffer, offset, 1)[0]; - const bytes = new Uint8Array(memory.buffer, offset + 4, Number(length)); + // next [length] bytes is the string contents, encoded as utf-8 + const bytes = new Uint8Array(memory.buffer, offset + 4, length); const string = new TextDecoder('utf8').decode(bytes); console.log(string); } function logInt(val) { + // cast BigInt to a number for pretty-printing w/o the "n" + // this may truncate or break for values that take >53 bits console.log(Number(val)); } function logBool(val) { + // this is a 32 bit number, either 1 or 0 console.log(val !== 0); } -const memory = new WebAssembly.Memory({ initial: 2, maximum: 100 }); +const memory = new WebAssembly.Memory({ initial: 10, maximum: 100 }); const importObject = { imports: { From df049903187aa829a69cb33625ea890509fd33a6 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 9 Sep 2022 19:39:33 -0700 Subject: [PATCH 33/79] finish string and list support --- README.md | 26 +- compiler/types/listvaluetype.py | 3 + compiler/wasm_backend.py | 556 +++++++++++++++++++++------- test.py | 5 +- tests/runtime/control_flow.py | 2 +- tests/runtime/globals.py | 2 +- tests/runtime/int_and_bool_funcs.py | 11 + tests/runtime/nested_list.py | 6 +- tests/runtime/nonlocal.py | 20 +- tests/runtime/nonlocal_builtins.py | 2 +- tests/runtime/simple_list.py | 27 ++ tests/runtime/simple_string.py | 33 +- 12 files changed, 541 insertions(+), 152 deletions(-) create mode 100644 tests/runtime/simple_list.py diff --git a/README.md b/README.md index ddb09e9..619246f 100644 --- a/README.md +++ b/README.md @@ -110,30 +110,38 @@ The `demo_cil.sh` script is a useful utility to compile and run files with the C ## WASM Backend Notes: -This is very much WIP. +This is WIP, not all features are supported. -Supported: -- int -- bool -- string (literals, len) +Features: +- int, bool, string, list - most operators - assignment - control flow - print, len, and assert - globals +- bounds checking for string and list indexing, null-safety for list length/indexing -Unsupported: -- for-loops +Unsupported/TODO: - class/object -- list - nonlocal -- string equality, concatenation, iteration - input +- string literal interning +- nicer exception messages Unclear/Untested: - `None` - nested functions +Memory format: + +- strings (utf-8) - first 4 bytes for length, followed by 1 byte for each character +- lists - first 4 bytes for length, followed by 8 bytes for each element +- ints - i64 +- pointers (objects, strings, lists) - i32 +- None - 0 (i32) + +Strings and lists are stored in the heap, aligned to 8 bytes. Note that memory does not get freed/garbage collected, so memory will run out for long-running programs. This is especially a problem with string iteration and string/list concatenation, since indexing a string in Chocopy requires a new string to be allocated. + ## FAQ - What is this for? diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 624f1e0..2f8d5e4 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -40,3 +40,6 @@ def toJSON(self, dump_location=True): "kind": "ListValueType", "elementType": self.elementType.toJSON(dump_location) } + + def getWasmName(self): + return "i32" diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 56048d0..e610dc9 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -5,10 +5,11 @@ from .visitor import CommonVisitor from typing import List + class WasmBuilder(Builder): def __init__(self, name: str): super(WasmBuilder, self).__init__(name) - + def module(self): self.newLine("(module") self.indent() @@ -33,7 +34,7 @@ def _else(self): self.newLine(f"(else") self.indent() - def func(self, name:str, params: List[str]=[], resType=None): + def func(self, name: str, params: List[str] = [], resType=None): params = " ".join(params) result = "" if resType is not None: @@ -64,6 +65,7 @@ def newBlock(self): self.lines.append(child) return child + class WasmBackend(CommonVisitor): defaultToGlobals = False # treat all vars as global if this is true localCounter = 0 @@ -90,14 +92,14 @@ def setLocal(self, name: str): def teeLocal(self, name: str): self.instr(f"local.tee ${name}") - def loadLocal(self, name: str): + def getLocal(self, name: str): self.instr(f"local.get ${name}") def genLocalName(self) -> str: - self.localCounter+=1 + self.localCounter += 1 return f"local_{self.localCounter}" - def newLocal(self, name: str = None, t: str = "i64")->str: + def newLocal(self, name: str = None, t: str = "i32") -> str: # add a new local decl, does not store anything if name is None: name = self.genLocalName() @@ -111,16 +113,283 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) - def loadMemoryCounter(self): - self.instr("i32.const 0") # addr 0 - self.instr("i32.load") - - def incrMemoryCounter(self, n:int): - self.instr("i32.const 0") # addr 0 - self.instr(f"i32.const {n}") - self.loadMemoryCounter() - self.instr("i32.add") - self.instr("i32.store") # alignment: 64 bit + def stdlib(self)->str: + return """ + ;; allocate and return addr of memory + ;; based on https://github.com/ucsd-cse231-s22/chocopy-wasm-compiler-A/blob/2022/stdlib/memory.wat + (func $alloc (param $bytes i32) (result i32) + (local $addr i32) + global.get $heap + local.set $addr + local.get $bytes + global.get $heap + i32.add + global.set $heap + local.get $addr + ) + ;; copy $size bytes from $src to $dest + ;; this just blindly copies memory and does not do any sort of validation/checks + (func $mem_cpy (param $src i32) (param $dest i32) (param $size i32) + (local $idx i32) + (local $temp i32) + i32.const 0 + local.set $idx + (block $block + (loop $loop + local.get $idx + local.get $size + i32.lt_s + i32.eqz + br_if $block + ;; read byte from $src + offset + local.get $idx + local.get $src + i32.add + i32.load8_u + local.set $temp + ;; write byte to $dest + offset + local.get $idx + local.get $dest + i32.add + local.get $temp + i32.store8 + ;; increment offset + local.get $idx + i32.const 1 + i32.add + local.set $idx + br $loop + ) + ) + ) + ;; return the length of a string or list as i32 + (func $len (param $addr i32) (result i32) + local.get $addr + call $nullthrow + i32.load + ) + ;; throw if $addr is null, otherwise return $addr + (func $nullthrow (param $addr i32) (result i32) + local.get $addr + i32.eqz + (if + (then + unreachable + ) + ) + local.get $addr + ) + ;; check the bounds of a string or list access, throwing if illegal + (func $check_bounds (param $addr i32) (param $idx i32) + local.get $addr + call $len + local.get $idx + i32.gt_s + i32.eqz + (if + (then + unreachable + ) + ) + i32.const 0 + local.get $idx + i32.gt_s + (if + (then + unreachable + ) + ) + ) + ;; index a string, returning the character as an i32 + (func $get_char (param $addr i32) (param $idx i32) (result i32) + local.get $addr + i32.const 4 + i32.add + local.get $idx + i32.add + i32.load8_u + ) + ;; index a string, allocating a new string for the single character and returning the address + (func $str_idx (param $addr i32) (param $idx i32) (result i32) + (local $new i32) + i32.const 8 + call $alloc + local.set $new + local.get $new + i32.const 1 + i32.store + local.get $new + i32.const 4 + i32.add + local.get $addr + local.get $idx + call $get_char + i32.store8 + local.get $new + ) + ;; concatenate two strings, returning the address of the new string + (func $str_concat (param $s1 i32) (param $s2 i32) (result i32) + (local $len1 i32) + (local $len2 i32) + (local $addr i32) + ;; allocate memory + local.get $s1 + call $len + local.tee $len1 + local.get $s2 + call $len + local.tee $len2 + i32.add + i32.const 4 + i32.add + i32.const 8 + i32.div_u + i32.const 8 + i32.add + call $alloc + local.tee $addr + ;; store length + local.get $len1 + local.get $len2 + i32.add + i32.store + ;; copy string 1 + local.get $s1 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + call $mem_cpy + ;; copy string 2 + local.get $s2 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + i32.add + local.get $len2 + call $mem_cpy + local.get $addr + ) + ;; compare two strings, returning true if the two strings have the same contents + (func $str_cmp (param $left i32) (param $right i32) (result i32) + (local $result i32) + (local $length i32) + (local $idx i32) + i32.const 1 + local.set $result + local.get $left + i32.load + local.tee $length + local.get $right + i32.load + i32.eq + (if + (then + i32.const 0 + local.set $idx + (block $block + (loop $loop + local.get $idx + local.get $length + i32.lt_s + i32.eqz + br_if $block + local.get $left + local.get $idx + call $get_char + local.get $right + local.get $idx + call $get_char + i32.eq + local.get $result + i32.and + local.set $result + local.get $result + i32.eqz + br_if $block + local.get $idx + i32.const 1 + i32.add + local.set $idx + br $loop + ) + ) + ) + (else + i32.const 0 + local.set $result + ) + ) + local.get $result + ) + ;; concatenate two lists, returning the address of the new list + (func $list_concat (param $l1 i32) (param $l2 i32) (result i32) + (local $len1 i32) + (local $len2 i32) + (local $addr i32) + ;; allocate 8 * (len1 + len2 + 1) bytes + local.get $l1 + call $len + local.tee $len1 + local.get $l2 + call $len + local.tee $len2 + i32.add + i32.const 1 + i32.add + i32.const 8 + i32.mul + call $alloc + local.tee $addr + ;; store length + local.get $len1 + local.get $len2 + i32.add + i32.store + ;; copy list 1 + local.get $l1 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + i32.const 8 + i32.mul + call $mem_cpy + ;; copy list 2 + local.get $l2 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + i32.const 8 + i32.mul + i32.add + local.get $len2 + i32.const 8 + i32.mul + call $mem_cpy + local.get $addr + ) + """ + + def alloc(self, local = None): + # consume i32 from top of stack, allocate that many bytes + self.instr("call $alloc") + if local is not None: + self.setLocal(local) + + def nullthrow(self): + # throw if top of stack is 0, otherwise returns top of stack + self.instr("call $nullthrow") def Program(self, node: Program): func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] @@ -131,11 +400,11 @@ def Program(self, node: Program): self.instr('(import "imports" "logString" (func $log_str (param i32)))') self.instr('(import "imports" "assert" (func $assert (param i32)))') self.instr('(memory (import "js" "mem") 1)') + self.instr(f"(global $heap (mut i32) (i32.const 4))") # initialize all globals to 0 for now, since we don't statically allocate strings or arrays for v in var_decls: - self.instr(f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()})") - self.instr(f"{v.var.t.getWasmName()}.const 0") - self.instr(f")") + self.instr( + f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()}) ({v.var.t.getWasmName()}.const 0))") for d in func_decls: self.visit(d) module_builder = self.builder @@ -145,8 +414,8 @@ def Program(self, node: Program): self.defaultToGlobals = True self.locals = self.builder.newBlock() # initialize memory counter - self.instr("i32.const 0") # addr 0 - self.instr("i32.const 8") # store value 8 + self.instr("i32.const 0") # addr 0 + self.instr("i32.const 8") # store value 8 self.instr("i32.store") # initialize globals for v in var_decls: @@ -157,6 +426,7 @@ def Program(self, node: Program): self.builder.end() self.builder = module_builder + self.instr(self.stdlib()) self.instr(f"(start $main)") self.builder.end() @@ -184,28 +454,35 @@ def VarDef(self, node: VarDef): # # STATEMENTS + def setIdentifier(self, target: Identifier): + # consume top of stack + if self.defaultToGlobals or target.varInstance.isGlobal: + self.instr(f"global.set ${target.name}") + elif target.varInstance.isNonlocal: + raise Exception("TODO") + else: + self.setLocal(target.name) + def processAssignmentTarget(self, target: Expr, name: str): # name is the name of the local that the value is stored in if isinstance(target, Identifier): - self.loadLocal(name) - if self.defaultToGlobals or target.varInstance.isGlobal: - self.instr(f"global.set ${target.name}") - elif target.varInstance.isNonlocal: - raise Exception("TODO") - else: - self.setLocal(target.name) + self.getLocal(name) + self.setIdentifier(target) elif isinstance(target, IndexExpr): - lst, idx = self.validateIdx(target) + self.visit(target.list) + iterable = self.newLocal() + self.setLocal(iterable) + idx = self.validateIdx(iterable, target) # 8 * idx + 4 + list addr - self.loadLocal(idx) + self.getLocal(idx) self.instr("i32.const 8") self.instr("i32.mul") self.instr("i32.const 4") self.instr("i32.add") - self.loadLocal(lst) + self.getLocal(iterable) self.instr("i32.add") - self.loadLocal(name) - self.instr("i32.store") + self.getLocal(name) + self.instr(f"{target.inferredType.getWasmName()}.store") elif isinstance(target, MemberExpr): raise Exception("TODO") else: @@ -238,6 +515,20 @@ def ExprStmt(self, node: ExprStmt): def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: return leftType.isListType() and rightType.isListType() and operator == "+" + def loadChar(self, string: str, idx: str): + self.getLocal(string) + self.getLocal(idx) + self.instr("call $get_char") + + def strCompare(self): + self.instr("call $str_cmp") + + def strConcat(self): + self.instr("call $str_concat") + + def listConcat(self): + self.instr("call $list_concat") + def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType @@ -247,9 +538,9 @@ def BinaryExpr(self, node: BinaryExpr): # concatenation and addition if operator == "+": if self.isListConcat(operator, leftType, rightType): - raise Exception("TODO") + self.listConcat() elif leftType == StrType(): - raise Exception("TODO") + self.strConcat() elif leftType == IntType(): self.instr("i64.add") else: @@ -276,18 +567,18 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("i64.lt_s") self.instr("i64.eqz") elif operator == "==": - # TODO: refs, string if leftType == IntType(): self.instr("i64.eq") elif leftType == StrType(): - raise Exception("TODO") + self.strCompare() else: self.instr("i32.eq") elif operator == "!=": if leftType == IntType(): self.instr("i64.ne") elif leftType == StrType(): - raise Exception("TODO") + self.strCompare() + self.instr("i32.eqz") else: self.instr("i32.ne") elif operator == "is": @@ -344,6 +635,54 @@ def WhileStmt(self, node: WhileStmt): self.builder.end() self.builder.end() + def ForStmt(self, node: ForStmt): + block = self.newLabelName() + loop = self.newLabelName() + + iterable = self.newLocal() + idx = self.newLocal() + length = self.newLocal() + + self.visit(node.iterable) + self.teeLocal(iterable) + self.nullthrow() + + self.instr("i32.load") + self.setLocal(length) + + # idx = 0 + self.instr("i32.const 0") + self.setLocal(idx) + + self.builder.block(block) + self.builder.loop(loop) + + # exit loop if idx >= length + self.getLocal(idx) + self.getLocal(length) + self.instr("i32.lt_s") + self.instr(f"i32.eqz") + + self.instr(f"br_if ${block}") + + isList = node.iterable.inferredType.isListType() + contentsType = node.identifier.inferredType.getWasmName() + self.idxHelper(iterable, idx, isList, contentsType) + self.setIdentifier(node.identifier) + + for s in node.body: + self.visit(s) + + # idx += 1 + self.getLocal(idx) + self.instr("i32.const 1") + self.instr("i32.add") + self.setLocal(idx) + + self.instr(f"br ${loop}") + self.builder.end() + self.builder.end() + def buildReturn(self, value: Expr): if self.returnType.isNone(): self.instr("return") @@ -378,7 +717,7 @@ def IfExpr(self, node: IfExpr): self.setLocal(n) self.builder.end() self.builder.end() - self.loadLocal(n) + self.getLocal(n) def ListExpr(self, node: ListExpr): length = len(node.elements) @@ -391,98 +730,66 @@ def ListExpr(self, node: ListExpr): elementType = ClassValueType("object") else: elementType = t.elementType + + # 8 bytes per element + 4 for the length, rounded up to nearest 8 + increase = (length + 1) * 8 + self.instr(f"i32.const {increase}") + + addr = self.newLocal() + self.alloc(addr) + # store the length - self.loadMemoryCounter() # addr: mem - addr = self.newLocal(None, "i32") - self.teeLocal(addr) # store memory counter - self.instr(f"i32.const {length}") # value - self.instr(f"i32.store") # alignment: 32-bit + self.getLocal(addr) + self.instr(f"i32.const {length}") # value + self.instr(f"i32.store") # alignment: 32-bit # unlike strings, each item in the list gets 64 bits instead of 8 for i in range(length): offset = i * 8 + 4 # addr: mem + 4 + idx * 8 - self.loadMemoryCounter() + self.getLocal(addr) self.instr(f"i32.const {offset}") self.instr("i32.add") self.visit(node.elements[i]) self.instr(f"{elementType.getWasmName()}.store") - memory = length * 8 + 4 - increase = 8 + (8 * (memory // 8)) - self.incrMemoryCounter(increase) + # load the address the list was stored at to the stack - self.loadLocal(addr) + self.getLocal(addr) - def validateIdx(self, node: IndexExpr): - self.visit(node.list) - lst = self.newLocal(None, "i32") - self.teeLocal(lst) - self.instr("i32.load") + def validateIdx(self, iterable: str, node: IndexExpr): + idx = self.newLocal() self.visit(node.index) self.instr("i32.wrap_i64") - idx = self.newLocal(None, "i32") - self.teeLocal(idx) - - # make sure idx < length - self.instr("i32.gt_s") - self.instr("i32.eqz") - self.builder._if() - self.builder._then() - self.visit(node.thenExpr) - self.instr("unreachable") - self.builder.end() - self.builder.end() - - # make sure idx >= 0 - self.instr('i32.const 0') - self.loadLocal(idx) - self.instr('i32.gt_s') - self.builder._if() - self.builder._then() - self.visit(node.thenExpr) - self.instr("unreachable") - self.builder.end() - self.builder.end() - return lst, idx + self.setLocal(idx) - def IndexExpr(self, node: IndexExpr): - lst, idx = self.validateIdx(node) + self.getLocal(iterable) + self.getLocal(idx) + self.instr("call $check_bounds") + return idx - if node.list.inferredType.isListType(): + def idxHelper(self, iterable: str, idx: str, isList: bool, contentsType: str): + if isList: # 8 * idx + 4 + list addr - self.loadLocal(idx) + self.getLocal(idx) self.instr("i32.const 8") self.instr("i32.mul") self.instr("i32.const 4") self.instr("i32.add") - self.loadLocal(lst) - self.instr("i32.add") - self.instr("i32.load") - else: - # store the length - self.loadMemoryCounter() # addr: mem - addr = self.newLocal(None, "i32") - self.teeLocal(addr) # store memory counter - self.instr(f"i32.const 1") # value - self.instr(f"i32.store") - - # addr of single char - self.loadMemoryCounter() - self.instr(f"i32.const 4") - self.instr("i32.add") - - # idx + 4 + list addr - self.loadLocal(idx) - self.instr("i32.const 4") - self.instr("i32.add") - self.loadLocal(lst) + self.getLocal(iterable) self.instr("i32.add") - self.instr("i32.load8_u") + self.instr(f"{contentsType}.load") + else: # must be a string, need to alloc a new string + self.getLocal(iterable) + self.getLocal(idx) + self.instr("call $str_idx") - self.instr("i32.store8") - self.incrMemoryCounter(8) - # load the address the string was stored at to the stack - self.loadLocal(addr) + def IndexExpr(self, node: IndexExpr): + self.visit(node.list) + iterable = self.newLocal() + self.setLocal(iterable) + idx = self.validateIdx(iterable, node) + self.idxHelper(iterable, idx, node.list.inferredType.isListType(), + node.inferredType.getWasmName()) # # LITERALS @@ -500,28 +807,32 @@ def NoneLiteral(self, node: NoneLiteral): def StringLiteral(self, node: StringLiteral): length = len(node.value) + memory = length + 4 + # 1 byte per char + 4 for length, rounded to nearest 8 + increase = 8 + (8 * (memory // 8)) + self.instr(f"i32.const {increase}") + + addr = self.newLocal() + self.alloc(addr) + # store the length - self.loadMemoryCounter() # addr: mem - addr = self.newLocal(None, "i32") - self.teeLocal(addr) # store memory counter - self.instr(f"i32.const {length}") # value + self.getLocal(addr) + self.instr(f"i32.const {length}") # value self.instr(f"i32.store") for i in range(length): offset = i + 4 val = ord(node.value[i]) # addr: mem + 4 + idx - self.loadMemoryCounter() + self.getLocal(addr) self.instr(f"i32.const {offset}") self.instr("i32.add") self.instr(f"i32.const {val}") self.instr("i32.store8") - memory = length + 4 - increase = 8 + (8 * (memory // 8)) - self.incrMemoryCounter(increase) + # load the address the string was stored at to the stack - self.loadLocal(addr) + self.getLocal(addr) - # # BUILT-INS + # BUILT-INS def emit_assert(self, arg: Expr): self.visit(arg) self.instr("call $assert") @@ -529,7 +840,8 @@ def emit_assert(self, arg: Expr): def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: - raise Exception(f"Built-in function print is unsupported for values of type {arg.inferredType.classname}") + raise Exception( + f"Built-in function print is unsupported for values of type {arg.inferredType.classname}") self.visit(arg) self.instr(f"call $log_{arg.inferredType.className}") self.NoneLiteral(None) @@ -537,9 +849,5 @@ def emit_print(self, arg: Expr): def emit_len(self, arg: Expr): # the length of a string or array is always in the first 4 bytes self.visit(arg) - self.instr("i32.load") - self.instr("i64.extend_i32_u") - - - - + self.instr("call $len") + self.instr("i64.extend_i32_u") \ No newline at end of file diff --git a/test.py b/test.py index 1df6b28..188e813 100644 --- a/test.py +++ b/test.py @@ -11,7 +11,7 @@ dump_location = True error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected"} -disabled_wasm_tests = [ +disabled_wasm_tests = { "assignment.py", "functions.py", "nested_list.py", @@ -37,8 +37,9 @@ "linked_list.py", # "var_decl.py", # "expr_stmt.py", + "simple_list.py", "lists.py" -] +} def run_all_tests(): run_parse_tests() diff --git a/tests/runtime/control_flow.py b/tests/runtime/control_flow.py index 7963992..94e15e2 100644 --- a/tests/runtime/control_flow.py +++ b/tests/runtime/control_flow.py @@ -45,7 +45,7 @@ a = True d = [True, True, True] for a in d: - __assert__(a) + assert a print(a) e = None diff --git a/tests/runtime/globals.py b/tests/runtime/globals.py index 20eac53..85ff920 100644 --- a/tests/runtime/globals.py +++ b/tests/runtime/globals.py @@ -7,7 +7,7 @@ def t(): global y x = x + 1 y = y + y - __assert__(z == 0) + assert z == 0 assert x == 0 assert y == "a" diff --git a/tests/runtime/int_and_bool_funcs.py b/tests/runtime/int_and_bool_funcs.py index 69b9c1f..4158540 100644 --- a/tests/runtime/int_and_bool_funcs.py +++ b/tests/runtime/int_and_bool_funcs.py @@ -9,7 +9,18 @@ def test1(a1:int, a2:int)->int: def test2(a1:bool)->bool: return not a1 +def test3(): + return None + +def test4(): + return + +test3() +test4() + assert test1(x, y) == 3 assert test1(x, x) == 2 assert test2(b) assert not test2(a) +assert test4() is None +assert test3() is None diff --git a/tests/runtime/nested_list.py b/tests/runtime/nested_list.py index bba72af..e521995 100644 --- a/tests/runtime/nested_list.py +++ b/tests/runtime/nested_list.py @@ -5,10 +5,10 @@ # TODO - check if these are legal # a = [] -# __assert__(len(a) == 0) +# assert len(a) == 0 # a = [[]] -# __assert__(len(a) == 1) -# __assert__(len(a[0]) == 0) +# assert len(a) == 1 +# assert len(a[0]) == 0 a = [[1]] assert len(a) == 1 diff --git a/tests/runtime/nonlocal.py b/tests/runtime/nonlocal.py index e5c3d4c..9b52279 100644 --- a/tests/runtime/nonlocal.py +++ b/tests/runtime/nonlocal.py @@ -12,15 +12,15 @@ def test3()->int: def test4(): nonlocal x def test5(): - __assert__(x == 4) + assert x == 4 def test6(): nonlocal x x = 3 test5() test(x) - __assert__(x == 4) + assert x == 4 test6() - __assert__(x == 3) + assert x == 3 test4() return x @@ -30,7 +30,7 @@ def test8(): x[0] = 0 x = [1, 2, 3] test8() - __assert__(x[0] == 0) + assert x[0] == 0 def test9(x:int): def test9helper(): @@ -47,11 +47,11 @@ def test12()->int: def test13(m:int)->int: return m + y y = test13(y) - __assert__(y == 2) + assert y == 2 return test13(y) - __assert__(test11(y) == 2) - __assert__(test12() == 4) - __assert__(y == 2) + assert test11(y) == 2 + assert test12() == 4 + assert y == 2 class Nonlocals: def testMethod3(self:"Nonlocals"): @@ -66,11 +66,11 @@ def testMethod2(): x = 3 y = 3 testMethod2() - __assert__(y == 3) + assert y == 3 def testMethod4(self:"Nonlocals"): test13(self) - __assert__(not (self is None)) + assert not (self is None) def test13(x:"Nonlocals"): def test14(): diff --git a/tests/runtime/nonlocal_builtins.py b/tests/runtime/nonlocal_builtins.py index 3e383fa..a37d11f 100644 --- a/tests/runtime/nonlocal_builtins.py +++ b/tests/runtime/nonlocal_builtins.py @@ -7,6 +7,6 @@ def g(): nonlocal x nonlocal y print(y) - __assert__(x) + assert x g() f() \ No newline at end of file diff --git a/tests/runtime/simple_list.py b/tests/runtime/simple_list.py new file mode 100644 index 0000000..6dbc176 --- /dev/null +++ b/tests/runtime/simple_list.py @@ -0,0 +1,27 @@ +w:[int] = None +x:int = 0 + +w = [] +assert len(w) == 0 + +w = [1, 2, 3] +assert len(w) == 3 +assert w[0] == 1 +assert w[1] == 2 +assert w[2] == 3 + +print(w[0]) +print(w[1]) +print(w[2]) + +w[1] = 10 +assert w[1] == 10 + +for x in w: + print(x) + +w = w + [5, 1, 0] +assert len(w) == 6 +assert w[3] == 5 +assert w[4] == 1 +assert w[5] == 0 diff --git a/tests/runtime/simple_string.py b/tests/runtime/simple_string.py index 3cd0d35..1306bc1 100644 --- a/tests/runtime/simple_string.py +++ b/tests/runtime/simple_string.py @@ -1,9 +1,40 @@ x:str = "123" y:str = "" z:str = "12345" +char:str = "c" print(x) print(y) print(z) assert len(x) == 3 assert len(y) == 0 -assert len(z) == 5 \ No newline at end of file +assert len(z) == 5 +assert char == char[0] + +print(x[0]) + +for char in y: + print(char) + +for char in z: + print(char) + +assert x[0] == "1" +assert x[1] == "2" +assert x[2] == "3" + +x = x + x[0] +assert x == "1231" + +x = x + "" +assert x == "1231" + +x = "" + x +assert x == "1231" + +assert y == "" +assert y != x +assert x != "321" +assert "123" == "123" +assert x == x +assert "" == "" +assert x != "" \ No newline at end of file From a28bb6a97184a833ab3337e1631b9c9eacade56a Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 9 Sep 2022 23:18:50 -0700 Subject: [PATCH 34/79] most nonlocals working --- README.md | 3 + compiler/astnodes/typedvar.py | 6 +- compiler/cil_backend.py | 7 +- compiler/jvm_backend.py | 6 +- compiler/typechecker.py | 4 + compiler/wasm_backend.py | 646 +++++++++++++++++---------------- test.py | 17 - tests/runtime/global_loop.py | 13 + tests/runtime/local_loop.py | 7 + tests/runtime/nonlocal_loop.py | 12 + 10 files changed, 387 insertions(+), 334 deletions(-) create mode 100644 tests/runtime/global_loop.py create mode 100644 tests/runtime/local_loop.py create mode 100644 tests/runtime/nonlocal_loop.py diff --git a/README.md b/README.md index 619246f..bb6aa42 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,9 @@ The exact error messages from typechecking do not necessarily match the referenc This compiler supports a limited version of Python's `assert` keyword. The `assert` may be followed by a single `bool` expression, which will raise an exception with an unspecified/generic message if the value is false. It is used in the test suite to assert values in runtime tests. +### Known Bugs: +- for-loops do not work with nonlocal + ## JVM Backend Notes: The JVM backend for this compiler outputs JVM bytecode in plaintext formatted for the Krakatau assembler. Here's how you can compile and run a file using this backend: diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index ac4e8fc..36ce1ff 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -22,5 +22,7 @@ def toJSON(self, dump_location=True): d["type"] = self.type.toJSON(dump_location) return d - def getWasmParam(self): - return f"(param ${self.identifier.name} {self.t.getWasmName()})" + def getWasmParam(self, paramIdx, funcType): + isRef = paramIdx in funcType.refParams + t = "i32" if isRef else self.t.getWasmName() + return f"(param ${self.identifier.name} {t})" diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index cd1beb0..cb9a1a0 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -263,7 +263,6 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None self.unindent() def VarDef(self, node: VarDef): - varName = node.getIdentifier().getCILName() if node.isAttr: # codegen for initialization in constructors className = ClassValueType(node.attrOfClass) @@ -510,11 +509,7 @@ def ForStmt(self, node: ForStmt): self.instr("ldc.i4.1") self.instr( "newobj instance void [mscorlib]System.String::.ctor(char, int32)") - if self.defaultToGlobals or node.identifier.varInstance.isGlobal: - self.instr( - f"stsfld {node.identifier.inferredType.getCILName()} {self.main}::{node.identifier.getCILName()}") - else: - self.store(node.identifier.name) + self.processAssignmentTarget(node.identifier) # body self.visitStmtList(node.body) # idx = idx + 1 diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 09f9035..7d7f54e 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -523,11 +523,7 @@ def ForStmt(self, node: ForStmt): self.instr("iadd") self.instr( "invokevirtual Method java/lang/String substring (II)Ljava/lang/String;") - if self.defaultToGlobals or node.identifier.varInstance.isGlobal: - self.instr( - f"putstatic Field {self.main} {node.identifier.name} {node.identifier.inferredType.getJavaSignature()}") - else: - self.store(node.identifier.name, node.identifier.inferredType) + self.processAssignmentTarget(node.identifier) # body self.visitStmtList(node.body) # idx = idx + 1 diff --git a/compiler/typechecker.py b/compiler/typechecker.py index a9658cf..d44eb4d 100644 --- a/compiler/typechecker.py +++ b/compiler/typechecker.py @@ -445,6 +445,10 @@ def CallExpr(self, node: CallExpr): def ForStmt(self, node: ForStmt): # set isReturn=True if any statement in body has isReturn=True iterType = node.iterable.inferredType + if not self.defInCurrentScope(node.identifier.name): + self.addError( + node.identifier, F"Identifier not mutable in current scope: {node.identifier.name}") + return if isinstance(iterType, ListValueType): if not self.ts.canAssign(iterType.elementType, node.identifier.inferredType): self.addError( diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index e610dc9..041b2e3 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -34,13 +34,15 @@ def _else(self): self.newLine(f"(else") self.indent() - def func(self, name: str, params: List[str] = [], resType=None): + def func(self, name: str, params: List[str] = [], resType=None) -> Builder: + # return new block for declaring extra locals params = " ".join(params) result = "" if resType is not None: result = f" (result {resType})" self.newLine(f"(func ${name} {params}{result}") self.indent() + return self.newBlock() def end(self): self.unindent() @@ -59,7 +61,7 @@ def emit(self) -> str: lines = [l for l in lines if l is not None] return "\n".join(lines) - def newBlock(self): + def newBlock(self) -> Builder: child = WasmBuilder(self.name) child.indentation = self.indentation self.lines.append(child) @@ -95,9 +97,10 @@ def teeLocal(self, name: str): def getLocal(self, name: str): self.instr(f"local.get ${name}") - def genLocalName(self) -> str: + def genLocalName(self, suffix=None) -> str: self.localCounter += 1 - return f"local_{self.localCounter}" + suffix = "" if suffix is None else ("_" + suffix) + return f"local{suffix}{self.localCounter}" def newLocal(self, name: str = None, t: str = "i32") -> str: # add a new local decl, does not store anything @@ -113,280 +116,12 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) - def stdlib(self)->str: - return """ - ;; allocate and return addr of memory - ;; based on https://github.com/ucsd-cse231-s22/chocopy-wasm-compiler-A/blob/2022/stdlib/memory.wat - (func $alloc (param $bytes i32) (result i32) - (local $addr i32) - global.get $heap - local.set $addr - local.get $bytes - global.get $heap - i32.add - global.set $heap - local.get $addr - ) - ;; copy $size bytes from $src to $dest - ;; this just blindly copies memory and does not do any sort of validation/checks - (func $mem_cpy (param $src i32) (param $dest i32) (param $size i32) - (local $idx i32) - (local $temp i32) - i32.const 0 - local.set $idx - (block $block - (loop $loop - local.get $idx - local.get $size - i32.lt_s - i32.eqz - br_if $block - ;; read byte from $src + offset - local.get $idx - local.get $src - i32.add - i32.load8_u - local.set $temp - ;; write byte to $dest + offset - local.get $idx - local.get $dest - i32.add - local.get $temp - i32.store8 - ;; increment offset - local.get $idx - i32.const 1 - i32.add - local.set $idx - br $loop - ) - ) - ) - ;; return the length of a string or list as i32 - (func $len (param $addr i32) (result i32) - local.get $addr - call $nullthrow - i32.load - ) - ;; throw if $addr is null, otherwise return $addr - (func $nullthrow (param $addr i32) (result i32) - local.get $addr - i32.eqz - (if - (then - unreachable - ) - ) - local.get $addr - ) - ;; check the bounds of a string or list access, throwing if illegal - (func $check_bounds (param $addr i32) (param $idx i32) - local.get $addr - call $len - local.get $idx - i32.gt_s - i32.eqz - (if - (then - unreachable - ) - ) - i32.const 0 - local.get $idx - i32.gt_s - (if - (then - unreachable - ) - ) - ) - ;; index a string, returning the character as an i32 - (func $get_char (param $addr i32) (param $idx i32) (result i32) - local.get $addr - i32.const 4 - i32.add - local.get $idx - i32.add - i32.load8_u - ) - ;; index a string, allocating a new string for the single character and returning the address - (func $str_idx (param $addr i32) (param $idx i32) (result i32) - (local $new i32) - i32.const 8 - call $alloc - local.set $new - local.get $new - i32.const 1 - i32.store - local.get $new - i32.const 4 - i32.add - local.get $addr - local.get $idx - call $get_char - i32.store8 - local.get $new - ) - ;; concatenate two strings, returning the address of the new string - (func $str_concat (param $s1 i32) (param $s2 i32) (result i32) - (local $len1 i32) - (local $len2 i32) - (local $addr i32) - ;; allocate memory - local.get $s1 - call $len - local.tee $len1 - local.get $s2 - call $len - local.tee $len2 - i32.add - i32.const 4 - i32.add - i32.const 8 - i32.div_u - i32.const 8 - i32.add - call $alloc - local.tee $addr - ;; store length - local.get $len1 - local.get $len2 - i32.add - i32.store - ;; copy string 1 - local.get $s1 - i32.const 4 - i32.add - local.get $addr - i32.const 4 - i32.add - local.get $len1 - call $mem_cpy - ;; copy string 2 - local.get $s2 - i32.const 4 - i32.add - local.get $addr - i32.const 4 - i32.add - local.get $len1 - i32.add - local.get $len2 - call $mem_cpy - local.get $addr - ) - ;; compare two strings, returning true if the two strings have the same contents - (func $str_cmp (param $left i32) (param $right i32) (result i32) - (local $result i32) - (local $length i32) - (local $idx i32) - i32.const 1 - local.set $result - local.get $left - i32.load - local.tee $length - local.get $right - i32.load - i32.eq - (if - (then - i32.const 0 - local.set $idx - (block $block - (loop $loop - local.get $idx - local.get $length - i32.lt_s - i32.eqz - br_if $block - local.get $left - local.get $idx - call $get_char - local.get $right - local.get $idx - call $get_char - i32.eq - local.get $result - i32.and - local.set $result - local.get $result - i32.eqz - br_if $block - local.get $idx - i32.const 1 - i32.add - local.set $idx - br $loop - ) - ) - ) - (else - i32.const 0 - local.set $result - ) - ) - local.get $result - ) - ;; concatenate two lists, returning the address of the new list - (func $list_concat (param $l1 i32) (param $l2 i32) (result i32) - (local $len1 i32) - (local $len2 i32) - (local $addr i32) - ;; allocate 8 * (len1 + len2 + 1) bytes - local.get $l1 - call $len - local.tee $len1 - local.get $l2 - call $len - local.tee $len2 - i32.add - i32.const 1 - i32.add - i32.const 8 - i32.mul - call $alloc - local.tee $addr - ;; store length - local.get $len1 - local.get $len2 - i32.add - i32.store - ;; copy list 1 - local.get $l1 - i32.const 4 - i32.add - local.get $addr - i32.const 4 - i32.add - local.get $len1 - i32.const 8 - i32.mul - call $mem_cpy - ;; copy list 2 - local.get $l2 - i32.const 4 - i32.add - local.get $addr - i32.const 4 - i32.add - local.get $len1 - i32.const 8 - i32.mul - i32.add - local.get $len2 - i32.const 8 - i32.mul - call $mem_cpy - local.get $addr - ) - """ - - def alloc(self, local = None): + def alloc(self, local=None): # consume i32 from top of stack, allocate that many bytes self.instr("call $alloc") if local is not None: self.setLocal(local) - + def nullthrow(self): # throw if top of stack is 0, otherwise returns top of stack self.instr("call $nullthrow") @@ -410,9 +145,8 @@ def Program(self, node: Program): module_builder = self.builder self.builder = module_builder.newBlock() - self.builder.func("main") + self.locals = self.builder.func("main") self.defaultToGlobals = True - self.locals = self.builder.newBlock() # initialize memory counter self.instr("i32.const 0") # addr 0 self.instr("i32.const 8") # store value 8 @@ -431,11 +165,12 @@ def Program(self, node: Program): self.builder.end() def FuncDef(self, node: FuncDef): - self.locals = self.builder.newBlock() - params = [p.getWasmParam() for p in node.params] + params = [] + for i in range(len(node.params)): + params.append(node.params[i].getWasmParam(i, node.type)) self.returnType = node.type.returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() - self.builder.func(node.name.name, params, ret) + self.locals = self.builder.func(node.name.name, params, ret) for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) @@ -446,7 +181,12 @@ def VarDef(self, node: VarDef): if node.isAttr: raise Exception("TODO") elif node.var.varInstance.isNonlocal: - raise Exception("TODO") + self.instr("i32.const 8") + self.instr("call $alloc") + addr = self.newLocal(varName) + self.teeLocal(addr) + self.visit(node.value) + self.instr(f"{node.value.inferredType.getWasmName()}.store") else: self.visit(node.value) n = self.newLocal(varName, node.value.inferredType.getWasmName()) @@ -454,23 +194,26 @@ def VarDef(self, node: VarDef): # # STATEMENTS - def setIdentifier(self, target: Identifier): - # consume top of stack + def setIdentifier(self, target: Identifier, val: str): + # val is the name of the local that the value is stored in if self.defaultToGlobals or target.varInstance.isGlobal: + self.getLocal(val) self.instr(f"global.set ${target.name}") elif target.varInstance.isNonlocal: - raise Exception("TODO") + self.getLocal(target.name) + self.getLocal(val) + self.instr(f"{target.inferredType.getWasmName()}.store") else: + self.getLocal(val) self.setLocal(target.name) - def processAssignmentTarget(self, target: Expr, name: str): - # name is the name of the local that the value is stored in + def processAssignmentTarget(self, target: Expr, val: str): + # val is the name of the local that the value is stored in if isinstance(target, Identifier): - self.getLocal(name) - self.setIdentifier(target) + self.setIdentifier(target, val) elif isinstance(target, IndexExpr): self.visit(target.list) - iterable = self.newLocal() + iterable = self.newLocal(self.genLocalName("iterable")) self.setLocal(iterable) idx = self.validateIdx(iterable, target) # 8 * idx + 4 + list addr @@ -481,7 +224,7 @@ def processAssignmentTarget(self, target: Expr, name: str): self.instr("i32.add") self.getLocal(iterable) self.instr("i32.add") - self.getLocal(name) + self.getLocal(val) self.instr(f"{target.inferredType.getWasmName()}.store") elif isinstance(target, MemberExpr): raise Exception("TODO") @@ -491,11 +234,12 @@ def processAssignmentTarget(self, target: Expr, name: str): def AssignStmt(self, node: AssignStmt): self.visit(node.value) + val = self.newLocal(self.genLocalName( + "val"), node.value.inferredType.getWasmName()) + self.setLocal(val) targets = node.targets[::-1] - name = self.newLocal(None, node.value.inferredType.getWasmName()) - self.setLocal(name) for t in targets: - self.processAssignmentTarget(t, name) + self.processAssignmentTarget(t, val) def IfStmt(self, node: IfStmt): self.visit(node.condition) @@ -616,7 +360,7 @@ def CallExpr(self, node: CallExpr): self.emit_assert(node.args[0]) else: for i in range(len(node.args)): - self.visit(node.args[i]) + self.visitArg(node.function.inferredType, i, node.args[i]) self.instr(f"call ${name}") if node.function.inferredType.returnType.isNone(): self.NoneLiteral(None) # push null for void return @@ -639,9 +383,12 @@ def ForStmt(self, node: ForStmt): block = self.newLabelName() loop = self.newLabelName() - iterable = self.newLocal() - idx = self.newLocal() - length = self.newLocal() + iterable = self.newLocal(self.genLocalName("iterable")) + idx = self.newLocal(self.genLocalName("idx")) + length = self.newLocal(self.genLocalName("length")) + # this temporarily stores the current value + temp = self.newLocal(self.genLocalName( + "temp"), node.identifier.inferredType.getWasmName()) self.visit(node.iterable) self.teeLocal(iterable) @@ -668,7 +415,8 @@ def ForStmt(self, node: ForStmt): isList = node.iterable.inferredType.isListType() contentsType = node.identifier.inferredType.getWasmName() self.idxHelper(iterable, idx, isList, contentsType) - self.setIdentifier(node.identifier) + self.setLocal(temp) + self.setIdentifier(node.identifier, temp) for s in node.body: self.visit(s) @@ -700,12 +448,14 @@ def Identifier(self, node: Identifier): if self.defaultToGlobals or node.varInstance.isGlobal: self.instr(f"global.get ${node.name}") elif node.varInstance.isNonlocal: - raise Exception("TODO") + self.instr(f"local.get ${node.name}") + self.instr(f"{node.inferredType.getWasmName()}.load") else: self.instr(f"local.get ${node.name}") def IfExpr(self, node: IfExpr): - n = self.newLocal(None, node.inferredType.getWasmName()) + n = self.newLocal(self.genLocalName("ifexpr_result"), + node.inferredType.getWasmName()) self.visit(node.condition) self.builder._if() self.builder._then() @@ -735,7 +485,7 @@ def ListExpr(self, node: ListExpr): increase = (length + 1) * 8 self.instr(f"i32.const {increase}") - addr = self.newLocal() + addr = self.newLocal(self.genLocalName("addr")) self.alloc(addr) # store the length @@ -756,7 +506,7 @@ def ListExpr(self, node: ListExpr): self.getLocal(addr) def validateIdx(self, iterable: str, node: IndexExpr): - idx = self.newLocal() + idx = self.newLocal(self.genLocalName("idx")) self.visit(node.index) self.instr("i32.wrap_i64") @@ -785,7 +535,7 @@ def idxHelper(self, iterable: str, idx: str, isList: bool, contentsType: str): def IndexExpr(self, node: IndexExpr): self.visit(node.list) - iterable = self.newLocal() + iterable = self.newLocal(self.genLocalName("iterable")) self.setLocal(iterable) idx = self.validateIdx(iterable, node) self.idxHelper(iterable, idx, node.list.inferredType.isListType(), @@ -812,7 +562,7 @@ def StringLiteral(self, node: StringLiteral): increase = 8 + (8 * (memory // 8)) self.instr(f"i32.const {increase}") - addr = self.newLocal() + addr = self.newLocal(self.genLocalName("addr")) self.alloc(addr) # store the length @@ -832,6 +582,26 @@ def StringLiteral(self, node: StringLiteral): # load the address the string was stored at to the stack self.getLocal(addr) + def visitArg(self, funcType, paramIdx: int, arg: Expr): + argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + paramIsRef = paramIdx in funcType.refParams + if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + # ref arg and ref param, pass ref arg + self.getLocal(arg.name) + elif paramIsRef: + # non-ref arg and ref param, or do not pass ref arg + # unwrap if necessary, re-wrap + self.instr("i32.const 8") + self.instr("call $alloc") + addr = self.newLocal(self.genLocalName("arg_" + str(paramIdx))) + self.teeLocal(addr) + self.visit(arg) + self.instr(f"{arg.inferredType.getWasmName()}.store") + self.getLocal(addr) + + else: # non-ref param, maybe unwrap + self.visit(arg) + # BUILT-INS def emit_assert(self, arg: Expr): self.visit(arg) @@ -850,4 +620,272 @@ def emit_len(self, arg: Expr): # the length of a string or array is always in the first 4 bytes self.visit(arg) self.instr("call $len") - self.instr("i64.extend_i32_u") \ No newline at end of file + self.instr("i64.extend_i32_u") + + def stdlib(self) -> str: + return """ + ;; allocate and return addr of memory + ;; based on https://github.com/ucsd-cse231-s22/chocopy-wasm-compiler-A/blob/2022/stdlib/memory.wat + (func $alloc (param $bytes i32) (result i32) + (local $addr i32) + global.get $heap + local.set $addr + local.get $bytes + global.get $heap + i32.add + global.set $heap + local.get $addr + ) + ;; copy $size bytes from $src to $dest + ;; this just blindly copies memory and does not do any sort of validation/checks + (func $mem_cpy (param $src i32) (param $dest i32) (param $size i32) + (local $idx i32) + (local $temp i32) + i32.const 0 + local.set $idx + (block $block + (loop $loop + local.get $idx + local.get $size + i32.lt_s + i32.eqz + br_if $block + ;; read byte from $src + offset + local.get $idx + local.get $src + i32.add + i32.load8_u + local.set $temp + ;; write byte to $dest + offset + local.get $idx + local.get $dest + i32.add + local.get $temp + i32.store8 + ;; increment offset + local.get $idx + i32.const 1 + i32.add + local.set $idx + br $loop + ) + ) + ) + ;; return the length of a string or list as i32 + (func $len (param $addr i32) (result i32) + local.get $addr + call $nullthrow + i32.load + ) + ;; throw if $addr is null, otherwise return $addr + (func $nullthrow (param $addr i32) (result i32) + local.get $addr + i32.eqz + (if + (then + unreachable + ) + ) + local.get $addr + ) + ;; check the bounds of a string or list access, throwing if illegal + (func $check_bounds (param $addr i32) (param $idx i32) + local.get $addr + call $len + local.get $idx + i32.gt_s + i32.eqz + (if + (then + unreachable + ) + ) + i32.const 0 + local.get $idx + i32.gt_s + (if + (then + unreachable + ) + ) + ) + ;; index a string, returning the character as an i32 + (func $get_char (param $addr i32) (param $idx i32) (result i32) + local.get $addr + i32.const 4 + i32.add + local.get $idx + i32.add + i32.load8_u + ) + ;; index a string, allocating a new string for the single character and returning the address + (func $str_idx (param $addr i32) (param $idx i32) (result i32) + (local $new i32) + i32.const 8 + call $alloc + local.set $new + local.get $new + i32.const 1 + i32.store + local.get $new + i32.const 4 + i32.add + local.get $addr + local.get $idx + call $get_char + i32.store8 + local.get $new + ) + ;; concatenate two strings, returning the address of the new string + (func $str_concat (param $s1 i32) (param $s2 i32) (result i32) + (local $len1 i32) + (local $len2 i32) + (local $addr i32) + ;; allocate memory + local.get $s1 + call $len + local.tee $len1 + local.get $s2 + call $len + local.tee $len2 + i32.add + i32.const 4 + i32.add + i32.const 8 + i32.div_u + i32.const 8 + i32.add + call $alloc + local.tee $addr + ;; store length + local.get $len1 + local.get $len2 + i32.add + i32.store + ;; copy string 1 + local.get $s1 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + call $mem_cpy + ;; copy string 2 + local.get $s2 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + i32.add + local.get $len2 + call $mem_cpy + local.get $addr + ) + ;; compare two strings, returning true if the two strings have the same contents + (func $str_cmp (param $left i32) (param $right i32) (result i32) + (local $result i32) + (local $length i32) + (local $idx i32) + i32.const 1 + local.set $result + local.get $left + i32.load + local.tee $length + local.get $right + i32.load + i32.eq + (if + (then + i32.const 0 + local.set $idx + (block $block + (loop $loop + local.get $idx + local.get $length + i32.lt_s + i32.eqz + br_if $block + local.get $left + local.get $idx + call $get_char + local.get $right + local.get $idx + call $get_char + i32.eq + local.get $result + i32.and + local.set $result + local.get $result + i32.eqz + br_if $block + local.get $idx + i32.const 1 + i32.add + local.set $idx + br $loop + ) + ) + ) + (else + i32.const 0 + local.set $result + ) + ) + local.get $result + ) + ;; concatenate two lists, returning the address of the new list + (func $list_concat (param $l1 i32) (param $l2 i32) (result i32) + (local $len1 i32) + (local $len2 i32) + (local $addr i32) + ;; allocate 8 * (len1 + len2 + 1) bytes + local.get $l1 + call $len + local.tee $len1 + local.get $l2 + call $len + local.tee $len2 + i32.add + i32.const 1 + i32.add + i32.const 8 + i32.mul + call $alloc + local.tee $addr + ;; store length + local.get $len1 + local.get $len2 + i32.add + i32.store + ;; copy list 1 + local.get $l1 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + i32.const 8 + i32.mul + call $mem_cpy + ;; copy list 2 + local.get $l2 + i32.const 4 + i32.add + local.get $addr + i32.const 4 + i32.add + local.get $len1 + i32.const 8 + i32.mul + i32.add + local.get $len2 + i32.const 8 + i32.mul + call $mem_cpy + local.get $addr + ) + """ diff --git a/test.py b/test.py index 188e813..17b2bcd 100644 --- a/test.py +++ b/test.py @@ -12,32 +12,15 @@ error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected"} disabled_wasm_tests = { - "assignment.py", - "functions.py", - "nested_list.py", "binary_tree.py", - "globals.py", "nonlocal.py", "classes.py", - # "hello_world.py", - "nonlocal_builtins.py", - "contains.py", "incrementing_counter.py", "operators.py", "control_flow.py", - # "int_and_bool.py", "ratio.py", - # "control_flow_2.py", - # "int_and_bool_control_flow.py", - # "simple_string.py", "doubling_vector.py", - # "int_and_bool_funcs.py", - "strings.py", - "exponent.py", "linked_list.py", - # "var_decl.py", - # "expr_stmt.py", - "simple_list.py", "lists.py" } diff --git a/tests/runtime/global_loop.py b/tests/runtime/global_loop.py new file mode 100644 index 0000000..a6eb769 --- /dev/null +++ b/tests/runtime/global_loop.py @@ -0,0 +1,13 @@ +x:int = 1 + +def test(): + y:[int] = None + def inner(): + global x + for x in y: + pass + y = [1, 2, 3] + inner() + assert x == 3 + +test() \ No newline at end of file diff --git a/tests/runtime/local_loop.py b/tests/runtime/local_loop.py new file mode 100644 index 0000000..b4b653c --- /dev/null +++ b/tests/runtime/local_loop.py @@ -0,0 +1,7 @@ +x:str = "" +y:str = "123" + +for x in y: + pass + +assert x == "3" \ No newline at end of file diff --git a/tests/runtime/nonlocal_loop.py b/tests/runtime/nonlocal_loop.py new file mode 100644 index 0000000..7a19205 --- /dev/null +++ b/tests/runtime/nonlocal_loop.py @@ -0,0 +1,12 @@ +def test(): + x:int = 1 + y:[int] = None + def inner(): + nonlocal x + for x in y: + pass + y = [1, 2, 3] + inner() + assert x == 3 + +test() \ No newline at end of file From 99ffdc239ab15a9235bfaf429d64ae19505a7fb6 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 9 Sep 2022 23:22:14 -0700 Subject: [PATCH 35/79] update README --- README.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index bb6aa42..ce13545 100644 --- a/README.md +++ b/README.md @@ -66,9 +66,6 @@ The exact error messages from typechecking do not necessarily match the referenc This compiler supports a limited version of Python's `assert` keyword. The `assert` may be followed by a single `bool` expression, which will raise an exception with an unspecified/generic message if the value is false. It is used in the test suite to assert values in runtime tests. -### Known Bugs: -- for-loops do not work with nonlocal - ## JVM Backend Notes: The JVM backend for this compiler outputs JVM bytecode in plaintext formatted for the Krakatau assembler. Here's how you can compile and run a file using this backend: @@ -120,20 +117,13 @@ Features: - most operators - assignment - control flow -- print, len, and assert +- stdlib: print, len, and assert - globals -- bounds checking for string and list indexing, null-safety for list length/indexing Unsupported/TODO: - class/object -- nonlocal -- input -- string literal interning -- nicer exception messages - -Unclear/Untested: -- `None` -- nested functions +- nonlocal (partial) +- stdlib: input Memory format: @@ -145,6 +135,8 @@ Memory format: Strings and lists are stored in the heap, aligned to 8 bytes. Note that memory does not get freed/garbage collected, so memory will run out for long-running programs. This is especially a problem with string iteration and string/list concatenation, since indexing a string in Chocopy requires a new string to be allocated. +To provide memory safety, string/list indexing have bounds checking and list operations have a null-check, which crashes the program with a generic "unreachable" instruction. + ## FAQ - What is this for? From a3028c39ff09858f4cf7eb745474f8445fd82ed3 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Wed, 14 Sep 2022 22:09:54 -0700 Subject: [PATCH 36/79] cleanup --- README.md | 35 +++++++++++++++++++++++------------ compiler/wasm_backend.py | 9 +++++++++ wasm.js | 25 +++++++++++++++++++++---- 3 files changed, 53 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index ce13545..9f5db52 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ The test suite includes both static validation of generated/annotated ASTs, as w - [Mono](https://www.mono-project.com/) - Tested with Mono 6.12 - WASM Backend Requirements: - - [WebAssembly Binary Toolkit (wabt)](https://github.com/WebAssembly/wabt) + - [WebAssembly Binary Toolkit (wabt)](https://github.com/WebAssembly/wabt), specifically the `wat2wasm` tool ## Usage @@ -94,7 +94,7 @@ Note that in the above example commands & the `demo_jvm.sh` script all expect th ## CIL Backend Notes: -The CIL backend for this compiler outputs CIL bytecode in plaintext formatted for the Mono ilsam assembler: +The CIL backend for this compiler outputs CIL bytecode in plaintext formatted for the Mono `ilasm` assembler: 1. Use this compiler to generate plaintext bytecode - Format: `python3 main.py --mode cil ` - Example: `python3 main.py --mode cil tests/runtime/binary_tree.py .` @@ -110,9 +110,23 @@ The `demo_cil.sh` script is a useful utility to compile and run files with the C ## WASM Backend Notes: -This is WIP, not all features are supported. +This is WIP, not all features are supported (the binary tree example itself actually does not work, but you can try another one). -Features: +The WASM backend for this compiler outputs WASM in plaintext `.wat` format which can be converted to `.wasm` using `wat2wasm`: +1. Use this compiler to generate plaintext WebAssembly + - Format: `python3 main.py --mode wasm ` + - Example: `python3 main.py --mode wasm tests/runtime/binary_tree.py .` +2. Run `wat2wasm` assembler to generate `.wasm` files + - Format: `wat2wasm <.wat file> -o <.wasm file>` + - Example: `wat2wasm binary_tree.wat -o binary_tree.wasm` +3. Run the `.wasm` files using a minimal JS runtime + - Example: `node wasm.js <.wasm file>` + - Example: `node wasm.js binary_tree.wasm` + +The `demo_wasm.sh` script is a useful utility to compile and run files with the WASM backend with a single command (provide the path to the input source file as an argument). +- To run the same example as above, run `./demo_wasm.sh tests/runtime/binary_tree.py` + +### WASM Backend - Supported Features: - int, bool, string, list - most operators - assignment @@ -120,22 +134,19 @@ Features: - stdlib: print, len, and assert - globals -Unsupported/TODO: +### WASM Backend - Unsupported Features: - class/object - nonlocal (partial) -- stdlib: input +- stdlib: input (node.js does not have synchronous I/O out of the box so this is difficult) -Memory format: +### WASM Backend - Memory Format, Safety, and Management: - strings (utf-8) - first 4 bytes for length, followed by 1 byte for each character - lists - first 4 bytes for length, followed by 8 bytes for each element - ints - i64 -- pointers (objects, strings, lists) - i32 -- None - 0 (i32) - -Strings and lists are stored in the heap, aligned to 8 bytes. Note that memory does not get freed/garbage collected, so memory will run out for long-running programs. This is especially a problem with string iteration and string/list concatenation, since indexing a string in Chocopy requires a new string to be allocated. +- pointers (objects, strings, lists) - i32, where `None` is 0 -To provide memory safety, string/list indexing have bounds checking and list operations have a null-check, which crashes the program with a generic "unreachable" instruction. +Strings, lists, objects, and refs holding nonlocals are stored in the heap, aligned to 8 bytes. Right now, memory does not get freed/garbage collected once it is allocated. To provide memory safety, string/list indexing have bounds checking and list operations have a null-check, which crashes the program with a generic "unreachable" instruction. ## FAQ diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 041b2e3..f921fc4 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -681,6 +681,7 @@ def stdlib(self) -> str: (func $nullthrow (param $addr i32) (result i32) local.get $addr i32.eqz + ;; throw if $addr == 0 (if (then unreachable @@ -695,6 +696,7 @@ def stdlib(self) -> str: local.get $idx i32.gt_s i32.eqz + ;; throw if !($len > $idx) (if (then unreachable @@ -703,6 +705,7 @@ def stdlib(self) -> str: i32.const 0 local.get $idx i32.gt_s + ;; throw if 0 > $idx (if (then unreachable @@ -794,9 +797,11 @@ def stdlib(self) -> str: local.get $left i32.load local.tee $length + ;; compare $length with len of $right local.get $right i32.load i32.eq + ;; only compare contents if lengths are equal (if (then i32.const 0 @@ -807,17 +812,21 @@ def stdlib(self) -> str: local.get $length i32.lt_s i32.eqz + ;; get left char br_if $block local.get $left local.get $idx call $get_char + ;; get right char local.get $right local.get $idx call $get_char + ;; $result = $result && left char == right char i32.eq local.get $result i32.and local.set $result + ;; if !$result then break local.get $result i32.eqz br_if $block diff --git a/wasm.js b/wasm.js index 3c33c37..e433d82 100644 --- a/wasm.js +++ b/wasm.js @@ -1,6 +1,18 @@ -const wasm_path = process.argv[2]; +/** + * Runtime support for running WASM compiled from Chocopy + * + * This is a very minimal runtime since the goal was to implement as much as + * possible directly in WASM. + * + * The only imports from JS to WASM are for `console.log` and `console.assert`, + * and the latter isn't even strictly necessary. + * + * The memory buffer is instantiated by JS, but after that it's never written + * to and only used to print strings. + */ + -// utils for pretty-printing ints, bools, strings +const wasm_path = process.argv[2]; function logString(offset) { // first 4 bytes is the length @@ -22,7 +34,10 @@ function logBool(val) { console.log(val !== 0); } -const memory = new WebAssembly.Memory({ initial: 10, maximum: 100 }); +const memory = new WebAssembly.Memory({ + initial: 10, + maximum: 100 +}); const importObject = { imports: { @@ -31,7 +46,9 @@ const importObject = { logBool: x => logBool(x), assert: x => console.assert(x) }, - js: { mem: memory }, + js: { + mem: memory + }, }; const fs = require('fs'); From 39615c609188198b89cad26b8ffb4db89a1eb66e Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 2 Oct 2022 20:15:20 -0700 Subject: [PATCH 37/79] implement classes --- README.md | 3 +- compiler/typesystem.py | 30 +++-- compiler/wasm_backend.py | 207 +++++++++++++++++++++++++++++++--- test.py | 29 ++--- tests/runtime/binary_tree.py | 118 +++++++++---------- tests/runtime/inherit_init.py | 51 +++++++++ tests/runtime/linked_list.py | 58 ++++++---- tests/runtime/strings.py | 5 + wasm.js | 8 +- 9 files changed, 381 insertions(+), 128 deletions(-) create mode 100644 tests/runtime/inherit_init.py diff --git a/README.md b/README.md index 9f5db52..214b3c7 100644 --- a/README.md +++ b/README.md @@ -135,8 +135,7 @@ The `demo_wasm.sh` script is a useful utility to compile and run files with the - globals ### WASM Backend - Unsupported Features: -- class/object -- nonlocal (partial) +- nonlocal referencing function param - stdlib: input (node.js does not have synchronous I/O out of the box so this is difficult) ### WASM Backend - Memory Format, Safety, and Management: diff --git a/compiler/typesystem.py b/compiler/typesystem.py index fe291b9..2caa998 100644 --- a/compiler/typesystem.py +++ b/compiler/typesystem.py @@ -152,22 +152,38 @@ def join(self, a: ValueType, b: ValueType): # this really shouldn't be returned return ObjectType() - def getAllMethods(self, className: str): - # return map of method names to tuples of - # (signature, classname of their definition) - methods = {} + def getOrderedMethods(self, className: str): + # (name, signature, defined in class) + methods = [] if self.classes[className].superclass is not None: - methods = self.getAllMethods(self.classes[className].superclass) + methods = self.getOrderedMethods(self.classes[className].superclass) for name in self.classes[className].methods: - methods[name] = (self.classes[className].methods[name], className) + hasExisting = False + for i in range(len(methods)): + if methods[i][0] == name: + methods[i] = (name, self.classes[className].methods[name], className) + hasExisting = True + break + if not hasExisting: + methods.append((name, self.classes[className].methods[name], className)) return methods + def getMappedMethods(self, className: str): + # map of name -> signature, defined in class + ordered = self.getOrderedMethods(className) + return { x: (y, z) for x, y, z in ordered } + def getOrderedAttrs(self, className: str): # return list of (name, type, init value) triples attrs = [] if self.classes[className].superclass is not None: - attr = self.getOrderedAttrs(self.classes[className].superclass) + attrs = self.getOrderedAttrs(self.classes[className].superclass) for attr in self.classes[className].orderedAttrs: attrType, attrInit = self.classes[className].attrs[attr] attrs.append((attr, attrType, attrInit)) return attrs + + def getMappedAttrs(self, className: str): + # map of name -> type, init value tuples + ordered = self.getOrderedAttrs(className) + return { x: (y, z) for x, y, z in ordered } diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index f921fc4..fe4bdfb 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -69,14 +69,44 @@ def newBlock(self) -> Builder: class WasmBackend(CommonVisitor): - defaultToGlobals = False # treat all vars as global if this is true - localCounter = 0 - locals = None - def __init__(self, main: str, ts: TypeSystem): self.builder = WasmBuilder(main) self.main = main # name of main method self.ts = ts + self.defaultToGlobals = False # treat all vars as global if this is true + self.localCounter = 0 + self.locals = None + + # (class name, attr name) -> class offset + self.attrOffsets = dict() + # (class name, method name) -> (class offset, table offset, inherited) + self.methodOffsets = dict() + + self.typeDefs = None + self.declaredTypes = set() + self.undeclaredFuncs = set() + + def initializeOffsets(self): + tblOffset = 0 + methodTableOffsets = dict() + # assign positions in the global method table + classes = [c for c in self.ts.classes if c != "" and c != ""] + for cls in classes: + for methName, _, defCls in self.ts.getOrderedMethods(cls): + if cls == defCls: + methodTableOffsets[(cls, methName)] = tblOffset + tblOffset += 1 + # calculate info for each class + for cls in classes: + attrs = self.ts.getOrderedAttrs(cls) + for idx, attrInfo in enumerate(attrs): + self.attrOffsets[(cls, attrInfo[0])] = idx + methods = self.ts.getOrderedMethods(cls) + for idx, methInfo in enumerate(methods): + name, _, defCls = methInfo + # get offset in global table + t = methodTableOffsets[(defCls, name)] + self.methodOffsets[(cls, name)] = (idx, t, defCls != cls) def currentBuilder(self): return self.classes[self.currentClass] @@ -127,14 +157,32 @@ def nullthrow(self): self.instr("call $nullthrow") def Program(self, node: Program): + self.initializeOffsets() + cls_decls = [d for d in node.declarations if isinstance(d, ClassDef)] func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] var_decls = [d for d in node.declarations if isinstance(d, VarDef)] + self.builder.module() self.instr('(import "imports" "logInt" (func $log_int (param i64)))') self.instr('(import "imports" "logBool" (func $log_bool (param i32)))') self.instr('(import "imports" "logString" (func $log_str (param i32)))') - self.instr('(import "imports" "assert" (func $assert (param i32)))') + self.instr('(import "imports" "assert" (func $assert (param i32) (param i32)))') self.instr('(memory (import "js" "mem") 1)') + + # initialize method table + self.instr(f"(table {len(self.methodOffsets)} funcref)") + funcNames = [(f"${k[0]}${k[1]}", v[1]) for k, v in self.methodOffsets.items() if not v[2]] + funcNames = sorted(funcNames, key=lambda x: x[1]) + + self.undeclaredFuncs = set([x[0] for x in funcNames]) + funcNames = " ".join([x[0] for x in funcNames]) + self.instr(f"(elem (i32.const 0) {funcNames})") + + # initialize typedefs + self.typeDefs = self.builder.newBlock() + self.typeDefs.newLine(f"(type $i32__ (func (param i32)))") + self.declaredTypes.add("$i32__") + self.instr(f"(global $heap (mut i32) (i32.const 4))") # initialize all globals to 0 for now, since we don't statically allocate strings or arrays for v in var_decls: @@ -142,6 +190,13 @@ def Program(self, node: Program): f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()}) ({v.var.t.getWasmName()}.const 0))") for d in func_decls: self.visit(d) + for c in cls_decls: + self.visit(c) + + # add stubs for any undeclared functions (no-op __init__) + for func in sorted(self.undeclaredFuncs): + self.instr(f"(func {func} (param $self i32))") + module_builder = self.builder self.builder = module_builder.newBlock() @@ -164,22 +219,39 @@ def Program(self, node: Program): self.instr(f"(start $main)") self.builder.end() + def ClassDef(self, node: ClassDef): + self.currentClass = node.name.name + func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] + for func in func_decls: + name = f"{node.name.name}${func.name.name}" + self.undeclaredFuncs.remove("$" + name) + self.funcDefHelper(func, name) + def FuncDef(self, node: FuncDef): + self.funcDefHelper(node, node.name.name) + + def funcDefHelper(self, node: FuncDef, name: str): params = [] for i in range(len(node.params)): params.append(node.params[i].getWasmParam(i, node.type)) self.returnType = node.type.returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() - self.locals = self.builder.func(node.name.name, params, ret) + self.locals = self.builder.func(name, params, ret) for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) + # implicitly return None if possible + if ret is not None and not isinstance(node.statements[-1], ReturnStmt): + if self.returnType.getWasmName() == "i32": + self.instr("i32.const 0") + else: + self.instr("unreachable") self.builder.end() def VarDef(self, node: VarDef): varName = node.var.identifier.name if node.isAttr: - raise Exception("TODO") + raise Exception("this should be handled elsewhere") elif node.var.varInstance.isNonlocal: self.instr("i32.const 8") self.instr("call $alloc") @@ -227,11 +299,27 @@ def processAssignmentTarget(self, target: Expr, val: str): self.getLocal(val) self.instr(f"{target.inferredType.getWasmName()}.store") elif isinstance(target, MemberExpr): - raise Exception("TODO") + cls = target.object.inferredType.className + attr = target.member.name + offset = self.attrOffsets[(cls, attr)] + self.visit(target.object) + self.instr(f"i32.const {offset * 8 + 8}") + self.instr("i32.add") + self.getLocal(val) + self.instr(f"{target.inferredType.getWasmName()}.store") else: raise Exception( "Internal compiler error: unsupported assignment target") + def MemberExpr(self, node: MemberExpr): + cls = node.object.inferredType.className + attr = node.member.name + offset = self.attrOffsets[(cls, attr)] + self.visit(node.object) + self.instr(f"i32.const {offset * 8 + 8}") + self.instr("i32.add") + self.instr(f"{node.inferredType.getWasmName()}.load") + def AssignStmt(self, node: AssignStmt): self.visit(node.value) val = self.newLocal(self.genLocalName( @@ -304,12 +392,12 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("i64.lt_s") elif operator == "<=": self.instr("i64.gt_s") - self.instr("i64.eqz") + self.instr("i32.eqz") elif operator == ">": self.instr("i64.gt_s") elif operator == ">=": self.instr("i64.lt_s") - self.instr("i64.eqz") + self.instr("i32.eqz") elif operator == "==": if leftType == IntType(): self.instr("i64.eq") @@ -346,18 +434,61 @@ def UnaryExpr(self, node: UnaryExpr): self.visit(node.operand) self.instr("i32.eqz") + def constructor(self, node: CallExpr): + cls = node.function.name + attrs = self.ts.getMappedAttrs(cls) + meths = self.ts.getMappedMethods(cls) + size = len(attrs) + len(meths) + 1 + increase = size * 8 + self.instr(f"i32.const {increase}") + addr = self.newLocal(self.genLocalName("addr")) + self.alloc(addr) + + # store starting position of vtable + self.getLocal(addr) + self.instr(f"i32.const {len(attrs) * 8 + 8}") + self.instr(f"i32.store") # alignment: 32-bit + + # initialize attrs + for name, t, v in self.ts.getOrderedAttrs(cls): + offset = self.attrOffsets[(cls, name)] + self.getLocal(addr) + self.instr(f"i32.const {offset * 8 + 8}") + self.instr(f"i32.add") + self.visit(v) + self.instr(f"{t.getWasmName()}.store") + + # initialize vtable + for name, _, _ in self.ts.getOrderedMethods(cls): + clsOffset, globalOffset, _ = self.methodOffsets[(cls, name)] + self.getLocal(addr) + self.instr(f"i32.const {len(attrs) * 8 + clsOffset * 8 + 8}") + self.instr(f"i32.add") + self.instr(f"i32.const {globalOffset}") + self.instr(f"i32.store") + + # call __init__, should always be index 0 + self.getLocal(addr) + self.getLocal(addr) + self.getLocal(addr) + self.instr("i32.load") + self.instr("i32.add") # addr + start of vtable offset + self.instr("i32.load") # load global table offset from vtable + self.instr(f"call_indirect (type $i32__)") + self.getLocal(addr) + def CallExpr(self, node: CallExpr): name = node.function.name if node.isConstructor: - raise Exception("TODO") - if name == "print": + self.constructor(node) + elif name == "print": self.emit_print(node.args[0]) elif name == "len": self.emit_len(node.args[0]) elif name == "input": - raise Exception("TODO") + raise Exception("user input is unimplemented") elif name == "__assert__": - self.emit_assert(node.args[0]) + self.emit_assert(node.args[0], node.location[0]) else: for i in range(len(node.args)): self.visitArg(node.function.inferredType, i, node.args[i]) @@ -365,6 +496,49 @@ def CallExpr(self, node: CallExpr): if node.function.inferredType.returnType.isNone(): self.NoneLiteral(None) # push null for void return + def MethodCallExpr(self, node: MethodCallExpr): + funcType = node.method.inferredType + className = node.method.object.inferredType.className + methodName = node.method.member.name + if methodName == "__init__" and className in {"int", "bool"}: + return + + self.visit(node.method.object) + obj = self.newLocal(self.genLocalName("obj")) + self.setLocal(obj) + + self.getLocal(obj) + for i in range(len(node.args)): + self.visitArg(funcType, i + 1, node.args[i]) + + # load indirect index + self.getLocal(obj) + self.getLocal(obj) + self.instr("i32.load") + methOffset, _, _ = self.methodOffsets[(className, methodName)] + self.instr(f"i32.const {methOffset * 8}") + self.instr("i32.add") + self.instr("i32.add") + self.instr("i32.load") + + # call indirect + self.instr(f";; call method {methodName}") + + # debug + temp = self.newLocal(self.genLocalName("idx")) + self.setLocal(temp) + self.getLocal(temp) + self.instr("i64.extend_i32_u") + self.instr("call $log_int") + self.getLocal(temp) + + params = " ".join([f"(param {t.getWasmName()})" for t in funcType.parameters]) + result = "" if funcType.returnType.isNone() else f"(result {funcType.returnType.getWasmName()})" + self.instr(f"call_indirect {params} {result}") + + if funcType.returnType.isNone(): + self.NoneLiteral(None) # push null for void return + def WhileStmt(self, node: WhileStmt): block = self.newLabelName() loop = self.newLabelName() @@ -603,8 +777,9 @@ def visitArg(self, funcType, paramIdx: int, arg: Expr): self.visit(arg) # BUILT-INS - def emit_assert(self, arg: Expr): + def emit_assert(self, arg: Expr, line: int): self.visit(arg) + self.instr(f"i32.const {line}") self.instr("call $assert") self.NoneLiteral(None) @@ -757,6 +932,8 @@ def stdlib(self) -> str: i32.const 8 i32.div_u i32.const 8 + i32.mul + i32.const 8 i32.add call $alloc local.tee $addr diff --git a/test.py b/test.py index 17b2bcd..8b4a9a8 100644 --- a/test.py +++ b/test.py @@ -9,28 +9,15 @@ from compiler.compiler import Compiler dump_location = True -error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected"} - -disabled_wasm_tests = { - "binary_tree.py", - "nonlocal.py", - "classes.py", - "incrementing_counter.py", - "operators.py", - "control_flow.py", - "ratio.py", - "doubling_vector.py", - "linked_list.py", - "lists.py" -} +error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected", "failed"} def run_all_tests(): - run_parse_tests() - run_typecheck_tests() - run_python_backend_tests() - run_closure_tests() - run_jvm_tests() - run_cil_tests() + # run_parse_tests() + # run_typecheck_tests() + # run_python_backend_tests() + # run_closure_tests() + # run_jvm_tests() + # run_cil_tests() run_wasm_tests() def run_parse_tests(): @@ -163,6 +150,8 @@ def run_python_backend_tests(): print("\nNot all test cases passed. Please run `make clean` after inspecting the output") print("\nPassed {:d} out of {:d} Python backend runtime test cases\n".format(n_passed, total)) +disabled_wasm_tests = [] + def run_wasm_tests(): print("Running WASM backend tests...\n") total = 0 diff --git a/tests/runtime/binary_tree.py b/tests/runtime/binary_tree.py index 177fd49..91ccc78 100644 --- a/tests/runtime/binary_tree.py +++ b/tests/runtime/binary_tree.py @@ -1,80 +1,82 @@ # Binary-search trees class TreeNode(object): - value:int = 0 - left:"TreeNode" = None - right:"TreeNode" = None + value: int = 0 + left: "TreeNode" = None + right: "TreeNode" = None - def insert(self:"TreeNode", x:int) -> bool: - if x < self.value: - if self.left is None: - self.left = makeNode(x) - return True - else: - return self.left.insert(x) - elif x > self.value: - if self.right is None: - self.right = makeNode(x) - return True - else: - return self.right.insert(x) - return False + def insert(self: "TreeNode", x: int) -> bool: + if x < self.value: + if self.left is None: + self.left = makeNode(x) + return True + else: + return self.left.insert(x) + elif x > self.value: + if self.right is None: + self.right = makeNode(x) + return True + else: + return self.right.insert(x) + return False + + def contains(self: "TreeNode", x: int) -> bool: + if x < self.value: + if self.left is None: + return False + else: + return self.left.contains(x) + elif x > self.value: + if self.right is None: + return False + else: + return self.right.contains(x) + else: + return True - def contains(self:"TreeNode", x:int) -> bool: - if x < self.value: - if self.left is None: - return False - else: - return self.left.contains(x) - elif x > self.value: - if self.right is None: - return False - else: - return self.right.contains(x) - else: - return True class Tree(object): - root:TreeNode = None - size:int = 0 + root: TreeNode = None + size: int = 0 + + def insert(self: "Tree", x: int) -> object: + if self.root is None: + self.root = makeNode(x) + self.size = 1 + else: + if self.root.insert(x): + self.size = self.size + 1 - def insert(self:"Tree", x:int) -> object: - if self.root is None: - self.root = makeNode(x) - self.size = 1 - else: - if self.root.insert(x): - self.size = self.size + 1 + def contains(self: "Tree", x: int) -> bool: + if self.root is None: + return False + else: + return self.root.contains(x) - def contains(self:"Tree", x:int) -> bool: - if self.root is None: - return False - else: - return self.root.contains(x) def makeNode(x: int) -> TreeNode: - b:TreeNode = None - b = TreeNode() - b.value = x - return b + b: TreeNode = None + b = TreeNode() + b.value = x + return b # Input parameters -n:int = 100 -c:int = 4 +n: int = 100 +c: int = 4 # Data -t:Tree = None -i:int = 0 -k:int = 37813 +t: Tree = None +i: int = 0 +k: int = 37813 # Crunch t = Tree() while i < n: - t.insert(k) - k = (k * 37813) % 37831 - if i % c != 0: - t.insert(i) - i = i + 1 + t.insert(k) + k = (k * 37813) % 37831 + if i % c != 0: + t.insert(i) + i = i + 1 assert t.size == 175 assert t.contains(15) diff --git a/tests/runtime/inherit_init.py b/tests/runtime/inherit_init.py new file mode 100644 index 0000000..f6eab03 --- /dev/null +++ b/tests/runtime/inherit_init.py @@ -0,0 +1,51 @@ +# this is the same as linked_list.py except LinkedList.__init__ is inherited + +class Link(object): + val: int = 0 + next: "Link" = None + + def __init__(self: "Link"): + pass + + def new(self: "Link", val: int, next: "Link") -> "Link": + self.val = val + self.next = next + return self + + +class LinkedList(object): + head: Link = None + + def __init(self: "LinkedList"): + pass + + def is_empty(self: "LinkedList") -> bool: + return self.head is None + + def length(self: "LinkedList") -> int: + cur: Link = None + length: int = 0 + cur = self.head + while not (cur is None): + length = length + 1 + cur = cur.next + return length + + def add(self: "LinkedList", val: int): + self.head = Link().new(val, self.head) + + +x: LinkedList = None + +x = LinkedList() +assert x.is_empty() +assert x.length() == 0 +x.add(1) +assert not x.is_empty() +assert x.length() == 1 +assert x.head.val == 1 +x.add(100) +assert not x.is_empty() +assert x.length() == 2 +assert x.head.val == 100 +assert x.head.next.val == 1 diff --git a/tests/runtime/linked_list.py b/tests/runtime/linked_list.py index 184e029..f04b7d3 100644 --- a/tests/runtime/linked_list.py +++ b/tests/runtime/linked_list.py @@ -1,31 +1,39 @@ class Link(object): - val : int = 0 - next : "Link" = None - def __init__(self : "Link"): - pass - def new(self : "Link", val : int, next : "Link") -> "Link": - self.val = val - self.next = next - return self + val: int = 0 + next: "Link" = None + + def __init__(self: "Link"): + pass + + def new(self: "Link", val: int, next: "Link") -> "Link": + self.val = val + self.next = next + return self + class LinkedList(object): - head : Link = None - def __init(self : "LinkedList"): - pass - def is_empty(self : "LinkedList") -> bool: - return self.head is None - def length(self : "LinkedList") -> int: - cur : Link = None - length : int = 0 - cur = self.head - while not (cur is None): - length = length + 1 - cur = cur.next - return length - def add(self : "LinkedList", val : int): - self.head = Link().new(val, self.head) - -x:LinkedList = None + head: Link = None + + def __init__(self: "LinkedList"): + pass + + def is_empty(self: "LinkedList") -> bool: + return self.head is None + + def length(self: "LinkedList") -> int: + cur: Link = None + length: int = 0 + cur = self.head + while not (cur is None): + length = length + 1 + cur = cur.next + return length + + def add(self: "LinkedList", val: int): + self.head = Link().new(val, self.head) + + +x: LinkedList = None x = LinkedList() assert x.is_empty() diff --git a/tests/runtime/strings.py b/tests/runtime/strings.py index bd4d444..c6eb073 100644 --- a/tests/runtime/strings.py +++ b/tests/runtime/strings.py @@ -59,4 +59,9 @@ assert len(x) == 1 assert len(y) == 3 +assert "1" + "2" + "3" + "4" + "5" == "12345" + +x = "123123" +assert x[0] + x[1] + x[2] == x[3] + x[4] + x[5] + diff --git a/wasm.js b/wasm.js index e433d82..ea577e9 100644 --- a/wasm.js +++ b/wasm.js @@ -34,6 +34,12 @@ function logBool(val) { console.log(val !== 0); } +function assert(val, line) { + if (!val) { + throw new Error("Assertion failed on line " + line.toString()); + } +} + const memory = new WebAssembly.Memory({ initial: 10, maximum: 100 @@ -44,7 +50,7 @@ const importObject = { logString: x => logString(x), logInt: x => logInt(x), logBool: x => logBool(x), - assert: x => console.assert(x) + assert: (x, y) => assert(x, y) }, js: { mem: memory From ebc8ac588f31db2da7454ed5bbf33641fc055dc3 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 3 Oct 2022 01:12:25 -0700 Subject: [PATCH 38/79] finish wasm backend --- README.md | 28 ++++++++++++---------------- compiler/astnodes/typedvar.py | 5 ----- compiler/types/functype.py | 14 ++++++++++++++ compiler/wasm_backend.py | 20 ++++++-------------- test.py | 12 ++++++------ 5 files changed, 38 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 214b3c7..adf4608 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # chocopy-python-compiler -Ahead-of-time compiler for [Chocopy](https://chocopy.org/), a subset of Python 3.6 with type annotations static type checking. +Ahead-of-time compiler for [Chocopy](https://chocopy.org/), a subset of Python 3.6 with type annotations and static type checking. Chocopy is used in compiler courses at several universities. This project has no relation to those courses, and is purely for my own learning/practice/fun. @@ -13,12 +13,13 @@ Progress is documented on my [blog](https://yangdanny97.github.io/blog/): This compiler is written entirely in Python. Since Chocopy is itself a subset of Python, lexing and parsing can be entirely handled by Python's `ast` module. -This compiler matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation's backend. That means that you can parse and typecheck the Chocopy file with this compiler, then use the reference implementation's backend to handle assembly code generation. +The frontend of this compiler matches the functionality of the first 2 passes (parsing & typechecking) Chocopy's reference compiler implementation, and outputs the AST in a JSON format that is compatible with the reference implementation's backend. That means that you can parse and typecheck the Chocopy file with this compiler, then use the reference implementation's backend to handle assembly code generation. -Additionally, this compiler contains 2 backends not found in the reference implementation: +This compiler contains multiple backends not found in the reference implementation: - Untyped Python 3 source code - JVM bytecode, formatted for the Krakatau assembler - CIL bytecode, formatted for the Mono ilasm assembler +- WASM, in WAT format The test suite includes both static validation of generated/annotated ASTs, as well as runtime tests that actually execute the output programs to check correctness. Many of the AST validation test cases are taken from test suites included in the release code for Berkeley's CS164, with some additional tests written for more coverage. @@ -42,6 +43,7 @@ The input file should have extension `.py`. If the output file is not provided, - Python source outputs will be written to a file of the same name/location as the input file, with extension `.out.py` - JVM outputs will be written to the same location as the input file, with the extension `.j` - CIL outputs will be written to the same location as the input file, with the extension `.cil` +- WASM outputs will be written to the same location as the input file, with the extension `.wat` **Flags:** @@ -56,7 +58,7 @@ The input file should have extension `.py`. If the output file is not provided, - `hoist` - output untyped Python 3 source code w/o nonlocals or nested function definitions - `jvm` - output JVM bytecode formatted for the Krakatau assembler - `cil` - output CIL bytecode formatted for the Mono ilasm assembler - - `wasm` - output WASM as plaintext in WAT format (WIP) + - `wasm` - output WASM as plaintext in WAT format ## Differences from the reference implementation: @@ -110,8 +112,6 @@ The `demo_cil.sh` script is a useful utility to compile and run files with the C ## WASM Backend Notes: -This is WIP, not all features are supported (the binary tree example itself actually does not work, but you can try another one). - The WASM backend for this compiler outputs WASM in plaintext `.wat` format which can be converted to `.wasm` using `wat2wasm`: 1. Use this compiler to generate plaintext WebAssembly - Format: `python3 main.py --mode wasm ` @@ -126,17 +126,10 @@ The WASM backend for this compiler outputs WASM in plaintext `.wat` format which The `demo_wasm.sh` script is a useful utility to compile and run files with the WASM backend with a single command (provide the path to the input source file as an argument). - To run the same example as above, run `./demo_wasm.sh tests/runtime/binary_tree.py` -### WASM Backend - Supported Features: -- int, bool, string, list -- most operators -- assignment -- control flow -- stdlib: print, len, and assert -- globals +The `wasm.js` file contains all the runtime support needed to run the WASM generated by this compiler. This backend was designed was to minimize runtime JavaScript dependencies, so the only imported functions are for assertions and printing strings/integers/booleans. ### WASM Backend - Unsupported Features: -- nonlocal referencing function param -- stdlib: input (node.js does not have synchronous I/O out of the box so this is difficult) +- `input` stdlib function (node.js does not have synchronous I/O out of the box so this is difficult) ### WASM Backend - Memory Format, Safety, and Management: @@ -144,8 +137,11 @@ The `demo_wasm.sh` script is a useful utility to compile and run files with the - lists - first 4 bytes for length, followed by 8 bytes for each element - ints - i64 - pointers (objects, strings, lists) - i32, where `None` is 0 +- objects - first 8 bytes for vtable offset, followed by 8 bytes for each attribute, followed by 8 bytes for each method index. inherited attribute/method positions are same as parent. + +Strings, lists, objects, and refs holding nonlocals are stored in the heap, aligned to 8 bytes. Right now, memory does not get freed/garbage collected once it is allocated, so large programs may run out of memory. -Strings, lists, objects, and refs holding nonlocals are stored in the heap, aligned to 8 bytes. Right now, memory does not get freed/garbage collected once it is allocated. To provide memory safety, string/list indexing have bounds checking and list operations have a null-check, which crashes the program with a generic "unreachable" instruction. +To provide memory safety, string/list indexing have bounds checking and list operations have a null-check, which crashes the program with a generic "unreachable" instruction. ## FAQ diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 36ce1ff..407b826 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -21,8 +21,3 @@ def toJSON(self, dump_location=True): d["identifier"] = self.identifier.toJSON(dump_location) d["type"] = self.type.toJSON(dump_location) return d - - def getWasmParam(self, paramIdx, funcType): - isRef = paramIdx in funcType.refParams - t = "i32" if isRef else self.t.getWasmName() - return f"(param ${self.identifier.name} {t})" diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 4d83b65..4acce1b 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -50,6 +50,20 @@ def getJavaSignature(self) -> str: params.append(sig) return "({}){}".format("".join(params), r) + def getWasmSignature(self, names=None) -> str: + params = [] + for i in range(len(self.parameters)): + p = self.parameters[i] + paramName = ("$" + names[i]) if names else "" + if i in self.refParams and isinstance(p, ClassValueType): + sig = f"(param {paramName} i32)" + else: + sig = f"(param {paramName} {p.getWasmName()})" + params.append(sig) + params = " ".join(params) + result = "" if self.returnType.isNone() else f" (result {self.returnType.getWasmName()})" + return params + result + def methodEquals(self, other): if isinstance(other, FuncType) and len(self.parameters) > 0 and len(other.parameters) > 0: return self.parameters[1:] == other.parameters[1:] and self.returnType == other.returnType diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index fe4bdfb..288c0cb 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -34,13 +34,9 @@ def _else(self): self.newLine(f"(else") self.indent() - def func(self, name: str, params: List[str] = [], resType=None) -> Builder: + def func(self, name: str, sig: str="") -> Builder: # return new block for declaring extra locals - params = " ".join(params) - result = "" - if resType is not None: - result = f" (result {resType})" - self.newLine(f"(func ${name} {params}{result}") + self.newLine(f"(func ${name} {sig}") self.indent() return self.newBlock() @@ -231,18 +227,16 @@ def FuncDef(self, node: FuncDef): self.funcDefHelper(node, node.name.name) def funcDefHelper(self, node: FuncDef, name: str): - params = [] - for i in range(len(node.params)): - params.append(node.params[i].getWasmParam(i, node.type)) self.returnType = node.type.returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() - self.locals = self.builder.func(name, params, ret) + paramNames = [x.identifier.name for x in node.params] + self.locals = self.builder.func(name, node.type.getWasmSignature(paramNames)) for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) # implicitly return None if possible if ret is not None and not isinstance(node.statements[-1], ReturnStmt): - if self.returnType.getWasmName() == "i32": + if self.returnType.getWasmName() == "i32" and not self.returnType.isSpecialType(): self.instr("i32.const 0") else: self.instr("unreachable") @@ -532,9 +526,7 @@ def MethodCallExpr(self, node: MethodCallExpr): self.instr("call $log_int") self.getLocal(temp) - params = " ".join([f"(param {t.getWasmName()})" for t in funcType.parameters]) - result = "" if funcType.returnType.isNone() else f"(result {funcType.returnType.getWasmName()})" - self.instr(f"call_indirect {params} {result}") + self.instr(f"call_indirect {funcType.getWasmSignature()}") if funcType.returnType.isNone(): self.NoneLiteral(None) # push null for void return diff --git a/test.py b/test.py index 8b4a9a8..96a0567 100644 --- a/test.py +++ b/test.py @@ -12,12 +12,12 @@ error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected", "failed"} def run_all_tests(): - # run_parse_tests() - # run_typecheck_tests() - # run_python_backend_tests() - # run_closure_tests() - # run_jvm_tests() - # run_cil_tests() + run_parse_tests() + run_typecheck_tests() + run_python_backend_tests() + run_closure_tests() + run_jvm_tests() + run_cil_tests() run_wasm_tests() def run_parse_tests(): From c0dea244bf37a3a875cf45ee2800135b7e6aceae Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 3 Oct 2022 23:03:02 -0700 Subject: [PATCH 39/79] compact wasm memory format for objects --- README.md | 2 +- compiler/wasm_backend.py | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index adf4608..89cddc0 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ The `wasm.js` file contains all the runtime support needed to run the WASM gener - lists - first 4 bytes for length, followed by 8 bytes for each element - ints - i64 - pointers (objects, strings, lists) - i32, where `None` is 0 -- objects - first 8 bytes for vtable offset, followed by 8 bytes for each attribute, followed by 8 bytes for each method index. inherited attribute/method positions are same as parent. +- objects - first 4 bytes for vtable offset, followed by 8 bytes for each attribute, followed by 4 bytes for each method index. inherited attribute/method positions are same as parent. Strings, lists, objects, and refs holding nonlocals are stored in the heap, aligned to 8 bytes. Right now, memory does not get freed/garbage collected once it is allocated, so large programs may run out of memory. diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 288c0cb..10d8fcc 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -297,7 +297,7 @@ def processAssignmentTarget(self, target: Expr, val: str): attr = target.member.name offset = self.attrOffsets[(cls, attr)] self.visit(target.object) - self.instr(f"i32.const {offset * 8 + 8}") + self.instr(f"i32.const {offset * 8 + 4}") self.instr("i32.add") self.getLocal(val) self.instr(f"{target.inferredType.getWasmName()}.store") @@ -310,7 +310,7 @@ def MemberExpr(self, node: MemberExpr): attr = node.member.name offset = self.attrOffsets[(cls, attr)] self.visit(node.object) - self.instr(f"i32.const {offset * 8 + 8}") + self.instr(f"i32.const {offset * 8 + 4}") self.instr("i32.add") self.instr(f"{node.inferredType.getWasmName()}.load") @@ -432,22 +432,22 @@ def constructor(self, node: CallExpr): cls = node.function.name attrs = self.ts.getMappedAttrs(cls) meths = self.ts.getMappedMethods(cls) - size = len(attrs) + len(meths) + 1 - increase = size * 8 + size = len(attrs) * 8 + len(meths) * 4 + 4 + increase = size if size % 8 == 0 else size + 4 self.instr(f"i32.const {increase}") addr = self.newLocal(self.genLocalName("addr")) self.alloc(addr) # store starting position of vtable self.getLocal(addr) - self.instr(f"i32.const {len(attrs) * 8 + 8}") + self.instr(f"i32.const {len(attrs) * 8 + 4}") self.instr(f"i32.store") # alignment: 32-bit # initialize attrs for name, t, v in self.ts.getOrderedAttrs(cls): offset = self.attrOffsets[(cls, name)] self.getLocal(addr) - self.instr(f"i32.const {offset * 8 + 8}") + self.instr(f"i32.const {offset * 8 + 4}") self.instr(f"i32.add") self.visit(v) self.instr(f"{t.getWasmName()}.store") @@ -456,7 +456,7 @@ def constructor(self, node: CallExpr): for name, _, _ in self.ts.getOrderedMethods(cls): clsOffset, globalOffset, _ = self.methodOffsets[(cls, name)] self.getLocal(addr) - self.instr(f"i32.const {len(attrs) * 8 + clsOffset * 8 + 8}") + self.instr(f"i32.const {len(attrs) * 8 + clsOffset * 4 + 4}") self.instr(f"i32.add") self.instr(f"i32.const {globalOffset}") self.instr(f"i32.store") @@ -510,7 +510,7 @@ def MethodCallExpr(self, node: MethodCallExpr): self.getLocal(obj) self.instr("i32.load") methOffset, _, _ = self.methodOffsets[(className, methodName)] - self.instr(f"i32.const {methOffset * 8}") + self.instr(f"i32.const {methOffset * 4}") self.instr("i32.add") self.instr("i32.add") self.instr("i32.load") From 667c484af7424acddc5ee7c8284bb6b9165970a0 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sat, 8 Oct 2022 14:24:29 -0700 Subject: [PATCH 40/79] only store a single copy of each vtable --- compiler/wasm_backend.py | 79 +++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 42 deletions(-) diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 10d8fcc..6994a0a 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -77,9 +77,8 @@ def __init__(self, main: str, ts: TypeSystem): self.attrOffsets = dict() # (class name, method name) -> (class offset, table offset, inherited) self.methodOffsets = dict() - - self.typeDefs = None - self.declaredTypes = set() + # class -> offset of start of vtable + self.vtables = dict() self.undeclaredFuncs = set() def initializeOffsets(self): @@ -93,16 +92,20 @@ def initializeOffsets(self): methodTableOffsets[(cls, methName)] = tblOffset tblOffset += 1 # calculate info for each class + memOffset = 0 for cls in classes: attrs = self.ts.getOrderedAttrs(cls) for idx, attrInfo in enumerate(attrs): self.attrOffsets[(cls, attrInfo[0])] = idx methods = self.ts.getOrderedMethods(cls) + self.vtables[cls] = [] for idx, methInfo in enumerate(methods): name, _, defCls = methInfo # get offset in global table t = methodTableOffsets[(defCls, name)] self.methodOffsets[(cls, name)] = (idx, t, defCls != cls) + self.vtables[cls].append((memOffset + idx * 4, t)) + memOffset += (len(methods) * 4) def currentBuilder(self): return self.classes[self.currentClass] @@ -174,12 +177,11 @@ def Program(self, node: Program): funcNames = " ".join([x[0] for x in funcNames]) self.instr(f"(elem (i32.const 0) {funcNames})") - # initialize typedefs - self.typeDefs = self.builder.newBlock() - self.typeDefs.newLine(f"(type $i32__ (func (param i32)))") - self.declaredTypes.add("$i32__") + # calculate first index of unallocated memory after vtables + fst = sum([len(t) * 4 for _, t in self.vtables.items()]) + fst = fst if fst % 8 == 0 else fst + 4 + self.instr(f"(global $heap (mut i32) (i32.const {fst}))") - self.instr(f"(global $heap (mut i32) (i32.const 4))") # initialize all globals to 0 for now, since we don't statically allocate strings or arrays for v in var_decls: self.instr( @@ -198,10 +200,7 @@ def Program(self, node: Program): self.locals = self.builder.func("main") self.defaultToGlobals = True - # initialize memory counter - self.instr("i32.const 0") # addr 0 - self.instr("i32.const 8") # store value 8 - self.instr("i32.store") + self.initializeVtables() # initialize globals for v in var_decls: self.visit(v.value) @@ -214,6 +213,13 @@ def Program(self, node: Program): self.instr(self.stdlib()) self.instr(f"(start $main)") self.builder.end() + + def initializeVtables(self): + for _, t in self.vtables.items(): + for memOffset, funcOffset in t: + self.instr(f"i32.const {memOffset}") + self.instr(f"i32.const {funcOffset}") + self.instr("i32.store") def ClassDef(self, node: ClassDef): self.currentClass = node.name.name @@ -430,9 +436,9 @@ def UnaryExpr(self, node: UnaryExpr): def constructor(self, node: CallExpr): cls = node.function.name + self.instr(f";; construct {cls}") attrs = self.ts.getMappedAttrs(cls) - meths = self.ts.getMappedMethods(cls) - size = len(attrs) * 8 + len(meths) * 4 + 4 + size = len(attrs) * 8 + 4 increase = size if size % 8 == 0 else size + 4 self.instr(f"i32.const {increase}") addr = self.newLocal(self.genLocalName("addr")) @@ -440,7 +446,7 @@ def constructor(self, node: CallExpr): # store starting position of vtable self.getLocal(addr) - self.instr(f"i32.const {len(attrs) * 8 + 4}") + self.instr(f"i32.const {self.vtables[cls][0][0]}") self.instr(f"i32.store") # alignment: 32-bit # initialize attrs @@ -452,23 +458,12 @@ def constructor(self, node: CallExpr): self.visit(v) self.instr(f"{t.getWasmName()}.store") - # initialize vtable - for name, _, _ in self.ts.getOrderedMethods(cls): - clsOffset, globalOffset, _ = self.methodOffsets[(cls, name)] - self.getLocal(addr) - self.instr(f"i32.const {len(attrs) * 8 + clsOffset * 4 + 4}") - self.instr(f"i32.add") - self.instr(f"i32.const {globalOffset}") - self.instr(f"i32.store") - # call __init__, should always be index 0 - self.getLocal(addr) - self.getLocal(addr) - self.getLocal(addr) - self.instr("i32.load") - self.instr("i32.add") # addr + start of vtable offset - self.instr("i32.load") # load global table offset from vtable - self.instr(f"call_indirect (type $i32__)") + self.getLocal(addr) # self argument + self.instr(f"i32.const {self.vtables[cls][0][1]}") + self.instr(f"call_indirect (param i32)") + + # return pointer to self self.getLocal(addr) def CallExpr(self, node: CallExpr): @@ -501,24 +496,28 @@ def MethodCallExpr(self, node: MethodCallExpr): obj = self.newLocal(self.genLocalName("obj")) self.setLocal(obj) + # args self.getLocal(obj) for i in range(len(node.args)): self.visitArg(funcType, i + 1, node.args[i]) # load indirect index self.getLocal(obj) - self.getLocal(obj) - self.instr("i32.load") + self.instr("i32.load") # load start of vtable + methOffset, _, _ = self.methodOffsets[(className, methodName)] self.instr(f"i32.const {methOffset * 4}") self.instr("i32.add") - self.instr("i32.add") - self.instr("i32.load") + self.instr("i32.load") # load table index of method - # call indirect self.instr(f";; call method {methodName}") + self.instr(f"call_indirect {funcType.getWasmSignature()}") + + if funcType.returnType.isNone(): + self.NoneLiteral(None) # push null for void return - # debug + # helper for debugging pointers, unused normally + def debug(self): temp = self.newLocal(self.genLocalName("idx")) self.setLocal(temp) self.getLocal(temp) @@ -526,11 +525,6 @@ def MethodCallExpr(self, node: MethodCallExpr): self.instr("call $log_int") self.getLocal(temp) - self.instr(f"call_indirect {funcType.getWasmSignature()}") - - if funcType.returnType.isNone(): - self.NoneLiteral(None) # push null for void return - def WhileStmt(self, node: WhileStmt): block = self.newLabelName() loop = self.newLabelName() @@ -770,6 +764,7 @@ def visitArg(self, funcType, paramIdx: int, arg: Expr): # BUILT-INS def emit_assert(self, arg: Expr, line: int): + self.instr(f";; assert line {line}") self.visit(arg) self.instr(f"i32.const {line}") self.instr("call $assert") From 21d7e1783cd3a2f2173efe97c3787c4902ce7398 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 9 Oct 2022 15:11:47 -0700 Subject: [PATCH 41/79] update readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 89cddc0..25e8e1a 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ The `wasm.js` file contains all the runtime support needed to run the WASM gener - lists - first 4 bytes for length, followed by 8 bytes for each element - ints - i64 - pointers (objects, strings, lists) - i32, where `None` is 0 -- objects - first 4 bytes for vtable offset, followed by 8 bytes for each attribute, followed by 4 bytes for each method index. inherited attribute/method positions are same as parent. +- objects - first 4 bytes for vtable addr, followed by 8 bytes for each attribute. inherited attribute/method positions are same as parent. Strings, lists, objects, and refs holding nonlocals are stored in the heap, aligned to 8 bytes. Right now, memory does not get freed/garbage collected once it is allocated, so large programs may run out of memory. From b70cb597990fce92553702036cb3e4c269b83708 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Wed, 12 Oct 2022 22:56:32 -0500 Subject: [PATCH 42/79] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 25e8e1a..402fa29 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ Progress is documented on my [blog](https://yangdanny97.github.io/blog/): - [Part 1: Frontend/Typechecker](https://yangdanny97.github.io/blog/2020/05/29/chocopy-typechecker) - [Part 2: JVM backend](https://yangdanny97.github.io/blog/2021/08/26/chocopy-jvm-backend) - [Part 3: CIL backend](https://yangdanny97.github.io/blog/2022/05/22/chocopy-cil-backend) +- [Part 4: WASM backend](https://yangdanny97.github.io/blog/2022/10/11/chocopy-wasm-backend) ## Features From 000ad4e23fe7c1f5dffbef4a7da5b83fce2ffd9b Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Wed, 12 Oct 2022 22:57:04 -0500 Subject: [PATCH 43/79] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 402fa29..6bde08b 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ The test suite includes both static validation of generated/annotated ASTs, as w - Tested with Mono 6.12 - WASM Backend Requirements: - [WebAssembly Binary Toolkit (wabt)](https://github.com/WebAssembly/wabt), specifically the `wat2wasm` tool + - NodeJS for the runtime ## Usage From fadec95eda44b054a5739ee109226be42d29e465 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 14 May 2023 00:07:06 -0700 Subject: [PATCH 44/79] formatting and lint fixes, entry point for new llvm backend --- compiler/builder.py | 2 +- compiler/cil_backend.py | 6 +- compiler/compiler.py | 3 + compiler/empty_list_typer.py | 2 +- compiler/jvm_backend.py | 9 ++- compiler/parser.py | 12 +-- compiler/typechecker.py | 2 +- compiler/types/classvaluetype.py | 2 +- compiler/types/functype.py | 3 +- compiler/types/listvaluetype.py | 4 +- compiler/typesystem.py | 18 +++-- compiler/wasm_backend.py | 56 +++++++------- main.py | 50 +++++++++---- test.py | 124 ++++++++++++++++++++----------- 14 files changed, 178 insertions(+), 115 deletions(-) diff --git a/compiler/builder.py b/compiler/builder.py index 23b790d..364b33a 100644 --- a/compiler/builder.py +++ b/compiler/builder.py @@ -5,7 +5,7 @@ def __init__(self, name: str): self.indentation = 0 def newLine(self, line=""): - self.lines.append((self.indentation*" ") + line) + self.lines.append((self.indentation * " ") + line) return self # returns a reference to the child builder diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index cb9a1a0..b03d37c 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -41,11 +41,11 @@ def currentBuilder(self): def newLabelName(self) -> str: self.counter += 1 - return "IL_"+str(self.counter) + return "IL_" + str(self.counter) def label(self, name: str) -> str: self.builder.unindent() - self.instr(name+": nop") + self.instr(name + ": nop") self.builder.indent() def store(self, name: str): @@ -212,7 +212,7 @@ def constructor(superclass: str, func: FuncDef): # method d.type = d.type.dropFirstParam() self.FuncDef(d, "virtual instance") - if constructor_def == None: + if constructor_def is None: # give a default constructor if none exists funcDef = node.getDefaultConstructor() constructor(superclass, funcDef) diff --git a/compiler/compiler.py b/compiler/compiler.py index 1b37f84..3151771 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -82,3 +82,6 @@ def emitWASM(self, main: str, ast: Node): wasm_backend = WasmBackend(main, self.transformer.ts) wasm_backend.visit(ast) return wasm_backend.builder + + def emitLLVM(self, main: str, ast: Node): + pass diff --git a/compiler/empty_list_typer.py b/compiler/empty_list_typer.py index b424014..59c4a4d 100644 --- a/compiler/empty_list_typer.py +++ b/compiler/empty_list_typer.py @@ -59,7 +59,7 @@ def AssignStmt(self, node: AssignStmt): self.expectedType = node.targets[0].inferredType def ListExpr(self, node: ListExpr): - if self.expectedType == None: + if self.expectedType is None: return expType = self.expectedType if isinstance(self.expectedType, ListValueType) and len(node.elements) == 0: diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 7d7f54e..a03f789 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -25,11 +25,11 @@ def currentBuilder(self): def newLabelName(self) -> str: self.counter += 1 - return "L"+str(self.counter) + return "L" + str(self.counter) def label(self, name: str) -> str: self.currentBuilder().unindent() - self.instr(name+":") + self.instr(name + ":") self.currentBuilder().indent() def returnInstr(self, exprType: ValueType): @@ -186,7 +186,7 @@ def ClassDef(self, node: ClassDef): self.constructor(superclass, d) else: self.method(d) - if constructor_def == None: + if constructor_def is None: funcDef = node.getDefaultConstructor() self.constructor(superclass, funcDef) self.instr(".end class") @@ -671,7 +671,8 @@ def emit_input(self): "invokespecial Method java/util/Scanner (Ljava/io/InputStream;)V") l = self.newLocal() self.instr(f"aload {l}") - self.instr("invokevirtual Method java/util/Scanner nextLine ()Ljava/lang/String;") + self.instr( + "invokevirtual Method java/util/Scanner nextLine ()Ljava/lang/String;") def emit_len(self, arg: Expr): t = arg.inferredType diff --git a/compiler/parser.py b/compiler/parser.py index b2abc52..f1a6a0a 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -74,7 +74,7 @@ def visit_Module(self, node): if (isinstance(body[i], GlobalDecl) or isinstance(body[i], NonLocalDecl)): raise ParseError( "Expected function, class, or variable declaration", node.body[i]) - if decl == False: + if not decl: raise ParseError( "All declarations must come before statements", node.body[i]) declarations.append(b) @@ -105,7 +105,7 @@ def visit_FunctionDef(self, node): if isinstance(b, ClassDef): raise ParseError( "Inner classes are unsupported", node.body[i]) - if decl == False: + if not decl: raise ParseError( "All declarations must come before statements", node.body[i]) declarations.append(b) @@ -143,7 +143,7 @@ def visit_ClassDef(self, node): node.decorator_list[0]) body = [self.visit(b) for b in node.body] # allow class bodies that only contain a single pass - if len(body) == 1 and body[0] == None: + if len(body) == 1 and body[0] is None: body = [] else: for i in range(len(body)): @@ -156,7 +156,7 @@ def visit_ClassDef(self, node): def visit_Return(self, node): location = self.getLocation(node) - if node.value == None: + if node.value is None: return ReturnStmt(location, None) else: return ReturnStmt(location, self.visit(node.value)) @@ -288,7 +288,7 @@ def visit_Constant(self, node): return BooleanLiteral(location, node.value) elif isinstance(node.value, int): return IntegerLiteral(location, node.value) - elif isinstance(node.value, str) and node.kind == None: + elif isinstance(node.value, str) and node.kind is None: return StringLiteral(location, node.value) elif node.value is None: return NoneLiteral(location) @@ -335,7 +335,7 @@ def visit_List(self, node): def visit_NameConstant(self, node): location = self.getLocation(node) - if node.value == None: + if node.value is None: return NoneLiteral(location) elif isinstance(node.value, bool): return BooleanLiteral(location, node.value) diff --git a/compiler/typechecker.py b/compiler/typechecker.py index d44eb4d..99cff01 100644 --- a/compiler/typechecker.py +++ b/compiler/typechecker.py @@ -283,7 +283,7 @@ def AssignStmt(self, node: AssignStmt): else: for t in node.targets: if isinstance(t, IndexExpr) and t.list.inferredType == StrType(): - self.addError(t, F"Cannot assign to index of string") + self.addError(t, "Cannot assign to index of string") return if isinstance(t, Identifier) and not self.defInCurrentScope(t.name): self.addError( diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index f7d40da..cae440b 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -86,7 +86,7 @@ def getCILName(self): elif self.className == "int": return "int64" else: - return "class "+self.className + return "class " + self.className def getWasmName(self): # bools are i32, ints are i64 diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 4acce1b..b02311b 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -61,7 +61,8 @@ def getWasmSignature(self, names=None) -> str: sig = f"(param {paramName} {p.getWasmName()})" params.append(sig) params = " ".join(params) - result = "" if self.returnType.isNone() else f" (result {self.returnType.getWasmName()})" + result = "" if self.returnType.isNone( + ) else f" (result {self.returnType.getWasmName()})" return params + result def methodEquals(self, other): diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 2f8d5e4..426d269 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -12,10 +12,10 @@ def __eq__(self, other): return False def getJavaSignature(self, _=False): - return "["+self.elementType.getJavaSignature(True) + return "[" + self.elementType.getJavaSignature(True) def getJavaName(self, _=False): - return "["+self.elementType.getJavaSignature(True) + return "[" + self.elementType.getJavaSignature(True) def getCILName(self, _=False): return self.elementType.getCILName() + "[]" diff --git a/compiler/typesystem.py b/compiler/typesystem.py index 2caa998..5ac7a8f 100644 --- a/compiler/typesystem.py +++ b/compiler/typesystem.py @@ -116,8 +116,7 @@ def canAssign(self, a: ValueType, b: ValueType) -> bool: return True if isinstance(b, ListValueType) and a == EmptyType(): return True - if (isinstance(b, ListValueType) and isinstance(a, ListValueType) - and a.elementType == NoneType()): + if (isinstance(b, ListValueType) and isinstance(a, ListValueType) and a.elementType == NoneType()): return self.canAssign(a.elementType, b.elementType) return False @@ -148,7 +147,7 @@ def join(self, a: ValueType, b: ValueType): bAncestors = bAncestors[::-1] for i in range(min(len(aAncestors), len(bAncestors))): if aAncestors[i] != bAncestors[i]: - return self.classes[aAncestors[i-1]] + return self.classes[aAncestors[i - 1]] # this really shouldn't be returned return ObjectType() @@ -156,22 +155,25 @@ def getOrderedMethods(self, className: str): # (name, signature, defined in class) methods = [] if self.classes[className].superclass is not None: - methods = self.getOrderedMethods(self.classes[className].superclass) + methods = self.getOrderedMethods( + self.classes[className].superclass) for name in self.classes[className].methods: hasExisting = False for i in range(len(methods)): if methods[i][0] == name: - methods[i] = (name, self.classes[className].methods[name], className) + methods[i] = ( + name, self.classes[className].methods[name], className) hasExisting = True break if not hasExisting: - methods.append((name, self.classes[className].methods[name], className)) + methods.append( + (name, self.classes[className].methods[name], className)) return methods def getMappedMethods(self, className: str): # map of name -> signature, defined in class ordered = self.getOrderedMethods(className) - return { x: (y, z) for x, y, z in ordered } + return {x: (y, z) for x, y, z in ordered} def getOrderedAttrs(self, className: str): # return list of (name, type, init value) triples @@ -186,4 +188,4 @@ def getOrderedAttrs(self, className: str): def getMappedAttrs(self, className: str): # map of name -> type, init value tuples ordered = self.getOrderedAttrs(className) - return { x: (y, z) for x, y, z in ordered } + return {x: (y, z) for x, y, z in ordered} diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 6994a0a..10f971c 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -23,18 +23,18 @@ def loop(self, name: str): self.indent() def _if(self): - self.newLine(f"(if") + self.newLine("(if") self.indent() def _then(self): - self.newLine(f"(then") + self.newLine("(then") self.indent() def _else(self): - self.newLine(f"(else") + self.newLine("(else") self.indent() - def func(self, name: str, sig: str="") -> Builder: + def func(self, name: str, sig: str = "") -> Builder: # return new block for declaring extra locals self.newLine(f"(func ${name} {sig}") self.indent() @@ -85,9 +85,10 @@ def initializeOffsets(self): tblOffset = 0 methodTableOffsets = dict() # assign positions in the global method table - classes = [c for c in self.ts.classes if c != "" and c != ""] + classes = [c for c in self.ts.classes if c != + "" and c != ""] for cls in classes: - for methName, _, defCls in self.ts.getOrderedMethods(cls): + for methName, _, defCls in self.ts.getOrderedMethods(cls): if cls == defCls: methodTableOffsets[(cls, methName)] = tblOffset tblOffset += 1 @@ -112,7 +113,7 @@ def currentBuilder(self): def newLabelName(self) -> str: self.counter += 1 - return "label_"+str(self.counter) + return "label_" + str(self.counter) def instr(self, instr: str): self.builder.newLine(instr) @@ -165,12 +166,14 @@ def Program(self, node: Program): self.instr('(import "imports" "logInt" (func $log_int (param i64)))') self.instr('(import "imports" "logBool" (func $log_bool (param i32)))') self.instr('(import "imports" "logString" (func $log_str (param i32)))') - self.instr('(import "imports" "assert" (func $assert (param i32) (param i32)))') + self.instr( + '(import "imports" "assert" (func $assert (param i32) (param i32)))') self.instr('(memory (import "js" "mem") 1)') # initialize method table self.instr(f"(table {len(self.methodOffsets)} funcref)") - funcNames = [(f"${k[0]}${k[1]}", v[1]) for k, v in self.methodOffsets.items() if not v[2]] + funcNames = [(f"${k[0]}${k[1]}", v[1]) + for k, v in self.methodOffsets.items() if not v[2]] funcNames = sorted(funcNames, key=lambda x: x[1]) self.undeclaredFuncs = set([x[0] for x in funcNames]) @@ -211,9 +214,9 @@ def Program(self, node: Program): self.builder = module_builder self.instr(self.stdlib()) - self.instr(f"(start $main)") + self.instr("(start $main)") self.builder.end() - + def initializeVtables(self): for _, t in self.vtables.items(): for memOffset, funcOffset in t: @@ -236,7 +239,8 @@ def funcDefHelper(self, node: FuncDef, name: str): self.returnType = node.type.returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() paramNames = [x.identifier.name for x in node.params] - self.locals = self.builder.func(name, node.type.getWasmSignature(paramNames)) + self.locals = self.builder.func( + name, node.type.getWasmSignature(paramNames)) for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) @@ -447,21 +451,21 @@ def constructor(self, node: CallExpr): # store starting position of vtable self.getLocal(addr) self.instr(f"i32.const {self.vtables[cls][0][0]}") - self.instr(f"i32.store") # alignment: 32-bit - + self.instr("i32.store") # alignment: 32-bit + # initialize attrs for name, t, v in self.ts.getOrderedAttrs(cls): offset = self.attrOffsets[(cls, name)] self.getLocal(addr) self.instr(f"i32.const {offset * 8 + 4}") - self.instr(f"i32.add") + self.instr("i32.add") self.visit(v) self.instr(f"{t.getWasmName()}.store") # call __init__, should always be index 0 - self.getLocal(addr) # self argument + self.getLocal(addr) # self argument self.instr(f"i32.const {self.vtables[cls][0][1]}") - self.instr(f"call_indirect (param i32)") + self.instr("call_indirect (param i32)") # return pointer to self self.getLocal(addr) @@ -503,12 +507,12 @@ def MethodCallExpr(self, node: MethodCallExpr): # load indirect index self.getLocal(obj) - self.instr("i32.load") # load start of vtable + self.instr("i32.load") # load start of vtable methOffset, _, _ = self.methodOffsets[(className, methodName)] self.instr(f"i32.const {methOffset * 4}") self.instr("i32.add") - self.instr("i32.load") # load table index of method + self.instr("i32.load") # load table index of method self.instr(f";; call method {methodName}") self.instr(f"call_indirect {funcType.getWasmSignature()}") @@ -531,7 +535,7 @@ def WhileStmt(self, node: WhileStmt): self.builder.block(block) self.builder.loop(loop) self.visit(node.condition) - self.instr(f"i32.eqz") + self.instr("i32.eqz") self.instr(f"br_if ${block}") for s in node.body: self.visit(s) @@ -568,7 +572,7 @@ def ForStmt(self, node: ForStmt): self.getLocal(idx) self.getLocal(length) self.instr("i32.lt_s") - self.instr(f"i32.eqz") + self.instr("i32.eqz") self.instr(f"br_if ${block}") @@ -651,7 +655,7 @@ def ListExpr(self, node: ListExpr): # store the length self.getLocal(addr) self.instr(f"i32.const {length}") # value - self.instr(f"i32.store") # alignment: 32-bit + self.instr("i32.store") # alignment: 32-bit # unlike strings, each item in the list gets 64 bits instead of 8 for i in range(length): offset = i * 8 + 4 @@ -705,15 +709,15 @@ def IndexExpr(self, node: IndexExpr): def BooleanLiteral(self, node: BooleanLiteral): if node.value: - self.instr(f"i32.const 1") + self.instr("i32.const 1") else: - self.instr(f"i32.const 0") + self.instr("i32.const 0") def IntegerLiteral(self, node: IntegerLiteral): self.instr(f"i64.const {node.value}") def NoneLiteral(self, node: NoneLiteral): - self.instr(f"i32.const 0") + self.instr("i32.const 0") def StringLiteral(self, node: StringLiteral): length = len(node.value) @@ -728,7 +732,7 @@ def StringLiteral(self, node: StringLiteral): # store the length self.getLocal(addr) self.instr(f"i32.const {length}") # value - self.instr(f"i32.store") + self.instr("i32.store") for i in range(length): offset = i + 4 val = ord(node.value[i]) diff --git a/main.py b/main.py index dc0e485..64e2c30 100644 --- a/main.py +++ b/main.py @@ -8,20 +8,26 @@ 'Modes:\n' + 'parse - output AST in JSON format\n' + 'tc - output typechecked AST in JSON format\n' + - 'python - output untyped Python 3 source code\n' + - 'hoist - output untyped Python 3 source code w/o nonlocals or nested function definitions\n' + - 'jvm - output JVM bytecode formatted for the Krakatau assembler\n' - 'cil - output CIL bytecode formatted for the Mono ilasm assembler\n' - 'wasm - output WASM in WAT format\n' + 'python - output untyped Python 3 source code\n' + + 'hoist - output untyped Python 3 source code w/o nonlocals or nested function definitions\n' + + 'jvm - output JVM bytecode formatted for the Krakatau assembler\n' + + 'cil - output CIL bytecode formatted for the Mono ilasm assembler\n' + + 'wasm - output WASM in WAT format\n' + + 'llvm - output LLVM\n' ) + def out_msg(path, verbose): if verbose: print("Output to {}".format(path)) + def main(): parser = argparse.ArgumentParser(description='Chocopy frontend') - parser.add_argument('--mode', dest='mode', choices=["parse", "tc", "python", "jvm", "hoist", "cil", "wasm"], default="python", + parser.add_argument('--mode', + dest='mode', + choices=["parse", "tc", "python", "jvm", "hoist", "cil", "wasm", "llvm"], + default="python", help=mode_help) parser.add_argument('--print', dest='should_print', action="iframe.php?url=https%3A%2F%2Fgithub.com%2Fstore_true", help="output to stdout instead of file") @@ -39,7 +45,7 @@ def main(): infile = args.infile outdir = args.outdir - if args.infile == None: + if args.infile is None: parser.print_help() raise Exception("Error: must specify input file") @@ -52,7 +58,7 @@ def main(): outdir = "./" elif outdir[-1] != "/": outdir = outdir + "/" - + outfile = None if args.mode == "tc": outfile = outdir + infile_name + ".ast.typed" @@ -62,6 +68,8 @@ def main(): outfile = outdir + infile_name + ".out.py" elif args.mode == "jvm": outfile = outdir + infile_name + ".j" + elif args.mode == "llvm": + outfile = outdir + infile_name + ".ll" compiler = Compiler() astparser = compiler.parser @@ -81,7 +89,7 @@ def main(): if args.mode in {"parse", "tc"}: ast_json = tree.toJSON(False) - if args.should_print: + if args.should_print: print(json.dumps(ast_json, indent=2)) else: with open(outfile, "w") as f: @@ -91,7 +99,7 @@ def main(): builder = compiler.emitPython(tree) if args.should_print: print(builder.emit()) - else: + else: with open(outfile, "w") as f: out_msg(outfile, args.verbose) f.write(builder.emit()) @@ -100,7 +108,7 @@ def main(): builder = compiler.emitPython(tree) if args.should_print: print(builder.emit()) - else: + else: with open(outfile, "w") as f: out_msg(outfile, args.verbose) f.write(builder.emit()) @@ -110,16 +118,16 @@ def main(): jvm_emitter = jvm_emitters[cls] if args.should_print: print(jvm_emitter.emit()) - else: + else: fname = outdir + cls + ".j" with open(fname, "w") as f: out_msg(fname, args.verbose) - f.write(jvm_emitter.emit()) + f.write(jvm_emitter.emit()) elif args.mode == "cil": cil_emitter = compiler.emitCIL(infile_name, tree) if args.should_print: print(cil_emitter.emit()) - else: + else: fname = outdir + cil_emitter.name + ".cil" with open(fname, "w") as f: out_msg(fname, args.verbose) @@ -128,11 +136,21 @@ def main(): wat_emitter = compiler.emitWASM(infile_name, tree) if args.should_print: print(wat_emitter.emit()) - else: + else: fname = outdir + wat_emitter.name + ".wat" with open(fname, "w") as f: out_msg(fname, args.verbose) - f.write(wat_emitter.emit()) + f.write(wat_emitter.emit()) + elif args.mode == "llvm": + llvm_emitter = compiler.emitLLVM(infile_name, tree) + if args.should_print: + print(llvm_emitter.emit()) + else: + fname = outdir + llvm_emitter.name + ".ll" + with open(fname, "w") as f: + out_msg(fname, args.verbose) + f.write(llvm_emitter.emit()) + if __name__ == "__main__": main() diff --git a/test.py b/test.py index 96a0567..fc9ce43 100644 --- a/test.py +++ b/test.py @@ -9,7 +9,9 @@ from compiler.compiler import Compiler dump_location = True -error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected", "failed"} +error_flags = {"error", "Error", "Exception", + "exception", "Expected", "expected", "failed"} + def run_all_tests(): run_parse_tests() @@ -20,6 +22,7 @@ def run_all_tests(): run_cil_tests() run_wasm_tests() + def run_parse_tests(): print("Running parser tests...\n") total = 0 @@ -30,7 +33,7 @@ def run_parse_tests(): passed = run_parse_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 # typechecker tests should all successfully parse @@ -39,7 +42,7 @@ def run_parse_tests(): passed = run_parse_test(test, False) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 # runtime tests should all successfully parse @@ -48,10 +51,12 @@ def run_parse_tests(): passed = run_typecheck_test(test, False) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 - print("\nPassed {:d} out of {:d} parser test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} parser test cases\n".format( + n_passed, total)) + def run_typecheck_tests(): print("Running typecheck tests...\n") @@ -62,7 +67,7 @@ def run_typecheck_tests(): passed = run_typecheck_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 tc_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() @@ -70,10 +75,12 @@ def run_typecheck_tests(): passed = run_typecheck_test(test, False) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 - print("\nPassed {:d} out of {:d} typechecker test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} typechecker test cases\n".format( + n_passed, total)) + def run_closure_tests(): print("Running closure transformation tests...\n") @@ -85,7 +92,7 @@ def run_closure_tests(): passed = run_closure_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 tc_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() @@ -93,10 +100,11 @@ def run_closure_tests(): passed = run_closure_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 - print("\nPassed {:d} out of {:d} closure transformation test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} closure transformation test cases\n".format( + n_passed, total)) if total == n_passed: subprocess.run("cd {} && rm -f *.test.py".format( str(Path(__file__).parent.resolve()) @@ -109,15 +117,17 @@ def run_closure_tests(): passed = run_closure_runtime_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 - print("\nPassed {:d} out of {:d} closure transformation runtime test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} closure transformation runtime test cases\n".format( + n_passed, total)) if total == n_passed: subprocess.run("cd {} && rm -f *.test.py".format( str(Path(__file__).parent.resolve()) ), shell=True) + def run_python_backend_tests(): print("Running Python backend tests...\n") total = 0 @@ -128,10 +138,11 @@ def run_python_backend_tests(): passed = run_python_emit_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 - print("\nPassed {:d} out of {:d} Python backend emit test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} Python backend emit test cases\n".format( + n_passed, total)) total = 0 n_passed = 0 tc_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() @@ -139,7 +150,7 @@ def run_python_backend_tests(): passed = run_python_runtime_test(test) total += 1 if not passed: - print("Failed: "+ str(test)) + print("Failed: " + str(test)) else: n_passed += 1 if total == n_passed: @@ -148,10 +159,13 @@ def run_python_backend_tests(): ), shell=True) else: print("\nNot all test cases passed. Please run `make clean` after inspecting the output") - print("\nPassed {:d} out of {:d} Python backend runtime test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} Python backend runtime test cases\n".format( + n_passed, total)) + disabled_wasm_tests = [] + def run_wasm_tests(): print("Running WASM backend tests...\n") total = 0 @@ -165,11 +179,11 @@ def run_wasm_tests(): break if skip: print("Skipping: " + str(test) + "\n") - continue + continue passed = run_wasm_test(test) total += 1 if not passed: - print("Failed: "+ str(test) + "\n") + print("Failed: " + str(test) + "\n") else: n_passed += 1 if total == n_passed: @@ -178,7 +192,9 @@ def run_wasm_tests(): ), shell=True) else: print("\nNot all test cases passed. Please run `make clean` after inspecting the output") - print("\nPassed {:d} out of {:d} WASM backend test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} WASM backend test cases\n".format( + n_passed, total)) + def run_jvm_tests(): print("Running JVM backend tests...\n") @@ -189,7 +205,7 @@ def run_jvm_tests(): passed = run_jvm_test(test) total += 1 if not passed: - print("Failed: "+ str(test) + "\n") + print("Failed: " + str(test) + "\n") else: n_passed += 1 if total == n_passed: @@ -198,7 +214,9 @@ def run_jvm_tests(): ), shell=True) else: print("\nNot all test cases passed. Please run `make clean` after inspecting the output") - print("\nPassed {:d} out of {:d} JVM backend test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} JVM backend test cases\n".format( + n_passed, total)) + def run_cil_tests(): print("Running CIL backend tests...\n") @@ -209,7 +227,7 @@ def run_cil_tests(): passed = run_cil_test(test) total += 1 if not passed: - print("Failed: "+ str(test) + "\n") + print("Failed: " + str(test) + "\n") else: n_passed += 1 if total == n_passed: @@ -218,9 +236,11 @@ def run_cil_tests(): ), shell=True) else: print("\nNot all test cases passed. Please run `make clean` after inspecting the output") - print("\nPassed {:d} out of {:d} CIL backend test cases\n".format(n_passed, total)) + print("\nPassed {:d} out of {:d} CIL backend test cases\n".format( + n_passed, total)) + -def run_parse_test(test, bad=True)->bool: +def run_parse_test(test, bad=True) -> bool: # if bad=True, then test cases prefixed with bad are expected to fail compiler = Compiler() astparser = compiler.parser @@ -240,7 +260,8 @@ def run_parse_test(test, bad=True)->bool: correct_json = json.load(f) return ast_equals(correct_json, ast_json) -def run_typecheck_test(test, checkAst = True)->bool: + +def run_typecheck_test(test, checkAst=True) -> bool: try: compiler = Compiler() astparser = compiler.parser @@ -267,7 +288,8 @@ def run_typecheck_test(test, checkAst = True)->bool: print(track) return False -def run_closure_test(test)->bool: + +def run_closure_test(test) -> bool: # check that typechecking passes with the transformed AST # for valid cases only try: @@ -295,7 +317,8 @@ def run_closure_test(test)->bool: print(track) return False -def run_closure_runtime_test(test)->bool: + +def run_closure_runtime_test(test) -> bool: infile_name = str(test)[:-3].split("/")[-1] try: compiler = Compiler() @@ -328,7 +351,8 @@ def run_closure_runtime_test(test)->bool: print(track) return False -def run_python_emit_test(test)->bool: + +def run_python_emit_test(test) -> bool: try: compiler = Compiler() astparser = compiler.parser @@ -346,7 +370,8 @@ def run_python_emit_test(test)->bool: print(track) return False -def run_python_runtime_test(test)->bool: + +def run_python_runtime_test(test) -> bool: infile_name = str(test)[:-3].split("/")[-1] try: compiler = Compiler() @@ -378,7 +403,8 @@ def run_python_runtime_test(test)->bool: print(track) return False -def run_jvm_test(test)->bool: + +def run_jvm_test(test) -> bool: passed = True try: infile_name = str(test)[:-3].split("/")[-1] @@ -397,7 +423,7 @@ def run_jvm_test(test)->bool: jvm_emitter = jvm_emitters[cls] fname = outdir + cls + ".j" with open(fname, "w") as f: - f.write(jvm_emitter.emit()) + f.write(jvm_emitter.emit()) except Exception as e: print("Internal compiler error:", test) track = traceback.format_exc() @@ -405,7 +431,8 @@ def run_jvm_test(test)->bool: print(track) return False try: - assembler_commands = ["python3 ../Krakatau/assemble.py -q ./{}.j".format(cls) for cls in jvm_emitters] + assembler_commands = [ + "python3 ../Krakatau/assemble.py -q ./{}.j".format(cls) for cls in jvm_emitters] output = subprocess.check_output("cd {} && {} && java -cp . {}".format( str(Path(__file__).parent.resolve()), " && ".join(assembler_commands), @@ -423,7 +450,8 @@ def run_jvm_test(test)->bool: return False return passed -def run_cil_test(test)->bool: + +def run_cil_test(test) -> bool: passed = True name = str(test.name[:-3]) try: @@ -441,7 +469,7 @@ def run_cil_test(test)->bool: cil_emitter = compiler.emitCIL(infile_name, ast) fname = outdir + cil_emitter.name + ".cil" with open(fname, "w") as f: - f.write(cil_emitter.emit()) + f.write(cil_emitter.emit()) except Exception as e: print("Internal compiler error:", test) track = traceback.format_exc() @@ -467,7 +495,8 @@ def run_cil_test(test)->bool: return False return passed -def run_wasm_test(test)->bool: + +def run_wasm_test(test) -> bool: passed = True name = str(test.name[:-3]) try: @@ -485,7 +514,7 @@ def run_wasm_test(test)->bool: wasm_emitter = compiler.emitWASM(infile_name, ast) fname = outdir + name + ".wat" with open(fname, "w") as f: - f.write(wasm_emitter.emit()) + f.write(wasm_emitter.emit()) except Exception as e: print("Internal compiler error:", test) track = traceback.format_exc() @@ -493,7 +522,8 @@ def run_wasm_test(test)->bool: print(track) return False try: - output = subprocess.check_output(f"wat2wasm {name}.wat -o {name}.wasm && node wasm.js {name}.wasm", shell=True) + output = subprocess.check_output( + f"wat2wasm {name}.wat -o {name}.wasm && node wasm.js {name}.wasm", shell=True) lines = output.decode().split("\n") for l in lines: for e in error_flags: @@ -506,26 +536,29 @@ def run_wasm_test(test)->bool: return False return passed -def ast_equals(d1, d2)->bool: + +def ast_equals(d1, d2) -> bool: # precondition: the input dict must represent a well-formed AST # d1 is the correct AST, d2 is the AST output by this compiler if isinstance(d1, dict) and isinstance(d2, dict): for k, v in d1.items(): if k not in d2 and k != "inferredType": - print("Expected field: "+k) + print("Expected field: " + k) return False # only check starting line of node if k == "location": if d1[k][0] != d2[k][0]: - print("Expected starting line {:d}, got {:d}".format(d1[k][0], d2[k][0])) + print("Expected starting line {:d}, got {:d}".format( + d1[k][0], d2[k][0])) return False # check number of errors, not the messages elif k == "errors": if len(d1[k]["errors"]) != len(d2[k]["errors"]): - print("Expected {:d} errors, got {:d}".format(len(d1[k]["errors"]), len(d2[k]["errors"]))) + print("Expected {:d} errors, got {:d}".format( + len(d1[k]["errors"]), len(d2[k]["errors"]))) return False elif k == "errorMsg": - pass # only check presence of message, not content + pass # only check presence of message, not content elif k == "inferredType": if k in d2 and not ast_equals(v, d2[k]): return False @@ -533,12 +566,13 @@ def ast_equals(d1, d2)->bool: return False for k in d2.keys(): if k not in d1 and k != "inferredType": - print("Unxpected field: "+k) + print("Unxpected field: " + k) return False return True if isinstance(d1, list) and isinstance(d2, list): if len(d1) != len(d2): - print("Expected list of length {:s}, got {:s}".format(len(d1), len(d2))) + print("Expected list of length {:s}, got {:s}".format( + len(d1), len(d2))) return False for i in range(len(d1)): if not ast_equals(d1[i], d2[i]): From fc0074509ab6b3728a4666189930e6a34a36c38c Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 14 May 2023 17:56:39 -0700 Subject: [PATCH 45/79] more setup --- compiler/compiler.py | 7 +- compiler/llvm_backend.py | 118 +++++++++++++++++++++++++++++++ compiler/types/classvaluetype.py | 100 ++++++++++++++++---------- compiler/types/functype.py | 18 ++++- compiler/types/listvaluetype.py | 20 +++--- compiler/types/symboltype.py | 3 - compiler/types/valuetype.py | 7 +- 7 files changed, 219 insertions(+), 54 deletions(-) create mode 100644 compiler/llvm_backend.py diff --git a/compiler/compiler.py b/compiler/compiler.py index 3151771..e76b351 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -11,6 +11,7 @@ from .cil_backend import CilBackend from .python_backend import PythonBackend from .wasm_backend import WasmBackend +from .llvm_backend import LlvmBackend import ast from pathlib import Path @@ -84,4 +85,8 @@ def emitWASM(self, main: str, ast: Node): return wasm_backend.builder def emitLLVM(self, main: str, ast: Node): - pass + self.closurepass(ast) + EmptyListTyper().visit(ast) + llvm_backend = LlvmBackend(main, self.transformer.ts) + llvm_backend.visit(ast) + return llvm_backend.builder diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py new file mode 100644 index 0000000..be4b0fc --- /dev/null +++ b/compiler/llvm_backend.py @@ -0,0 +1,118 @@ +from .astnodes import * +from .types import * +from .typesystem import TypeSystem +from .visitor import Visitor +from collections import defaultdict +from typing import List + +import llvmlite.ir as ir +import llvmlite.binding as llvm + + +class LlvmBackend(Visitor): + locals = [] + + def __init__(self, main: str, ts: TypeSystem): + self.module = ir.Module() + self.builder = None + + def enterScope(self): + self.locals.append(defaultdict(lambda: None)) + + def exitScope(self): + self.locals.pop() + + def visit(self, node: Node): + return node.visit(self) + + # TOP LEVEL & DECLARATIONS + + def Program(self, node: Program): + pass + + def VarDef(self, node: VarDef): + pass + + def ClassDef(self, node: ClassDef): + pass + + def FuncDef(self, node: FuncDef): + pass + + # STATEMENTS + + def NonLocalDecl(self, node: NonLocalDecl): + pass + + def GlobalDecl(self, node: GlobalDecl): + pass + + def AssignStmt(self, node: AssignStmt): + pass + + def IfStmt(self, node: IfStmt): + pass + + def ExprStmt(self, node: ExprStmt): + pass + + def BinaryExpr(self, node: BinaryExpr): + pass + + def IndexExpr(self, node: IndexExpr): + pass + + def UnaryExpr(self, node: UnaryExpr): + pass + + def CallExpr(self, node: CallExpr): + pass + + def ForStmt(self, node: ForStmt): + pass + + def ListExpr(self, node: ListExpr): + pass + + def WhileStmt(self, node: WhileStmt): + pass + + def ReturnStmt(self, node: ReturnStmt): + pass + + def Identifier(self, node: Identifier): + pass + + def MemberExpr(self, node: MemberExpr): + pass + + def IfExpr(self, node: IfExpr): + pass + + def MethodCallExpr(self, node: MethodCallExpr): + pass + + # LITERALS + + def BooleanLiteral(self, node: BooleanLiteral): + pass + + def IntegerLiteral(self, node: IntegerLiteral): + pass + + def NoneLiteral(self, node: NoneLiteral): + pass + + def StringLiteral(self, node: StringLiteral): + pass + + # TYPES + + def TypedVar(self, node: TypedVar): + pass + + def ListType(self, node: ListType): + pass + + def ClassType(self, node: ClassType): + pass diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index cae440b..1f40774 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -1,4 +1,14 @@ from .valuetype import ValueType +from llvmlite import ir + + +class SpecialClass: + BOOL = 'bool' + STR = 'str' + INT = 'int' + NONE = '' + EMPTY = '' + OBJECT = 'object' class ClassValueType(ValueType): @@ -11,54 +21,54 @@ def __eq__(self, other): return False def isListType(self) -> bool: - return self.className in {"", ""} + return self.className in {SpecialClass.EMPTY, SpecialClass.NONE} def getJavaSignature(self, isList=False) -> str: - if self.className == "bool": + if self.className == SpecialClass.BOOL: if isList: return "Ljava/lang/Boolean;" else: return "Z" - elif self.className == "str": + elif self.className == SpecialClass.STR: return "Ljava/lang/String;" - elif self.className == "object": + elif self.className == SpecialClass.OBJECT: return "Ljava/lang/Object;" - elif self.className == "int": + elif self.className == SpecialClass.INT: if isList: return "Ljava/lang/Integer;" else: return "I" - elif self.className == "": + elif self.className == SpecialClass.NONE: return "Ljava/lang/Object;" - elif self.className == "": + elif self.className == SpecialClass.EMPTY: return "[Ljava/lang/Object;" else: return "L" + self.className + ";" - def isNone(self): - return self.className == "" + def isNone(self) -> bool: + return self.className == SpecialClass.NONE - def isSpecialType(self): - return self.className in ["int", "str", "bool"] + def isSpecialType(self) -> bool: + return self.className in [SpecialClass.INT, SpecialClass.STR, SpecialClass.BOOL] - def isJavaRef(self): - return self.className not in ["int", "bool"] + def isJavaRef(self) -> bool: + return self.className not in [SpecialClass.INT, SpecialClass.BOOL] - def getJavaName(self, isList=False): - if self.className == "bool": + def getJavaName(self, isList=False) -> str: + if self.className == SpecialClass.BOOL: if isList: return "java/lang/Boolean" else: return "boolean" - elif self.className == "str": + elif self.className == SpecialClass.STR: return "java/lang/String" - elif self.className == "object": + elif self.className == SpecialClass.OBJECT: return "java/lang/Object" - elif self.className == "": + elif self.className == SpecialClass.NONE: return "java/lang/Object" - elif self.className == "": + elif self.className == SpecialClass.EMPTY: return "[Ljava/lang/Object;" - elif self.className == "int": + elif self.className == SpecialClass.INT: if isList: return "java/lang/Integer" else: @@ -66,42 +76,42 @@ def getJavaName(self, isList=False): else: return self.className - def getCILSignature(self): - if self.className == "": + def getCILSignature(self) -> str: + if self.className == SpecialClass.NONE: return "void" else: return self.getCILName() - def getCILName(self): - if self.className == "bool": + def getCILName(self) -> str: + if self.className == SpecialClass.BOOL: return "bool" - elif self.className == "str": + elif self.className == SpecialClass.STR: return "string" - elif self.className == "object": + elif self.className == SpecialClass.OBJECT: return "object" - elif self.className == "": + elif self.className == SpecialClass.NONE: return "object" - elif self.className == "": + elif self.className == SpecialClass.EMPTY: return "object[]" - elif self.className == "int": + elif self.className == SpecialClass.INT: return "int64" else: return "class " + self.className - def getWasmName(self): + def getWasmName(self) -> str: # bools are i32, ints are i64 # all others are pointers/refs, which are i32 - if self.className == "bool": + if self.className == SpecialClass.BOOL: return "i32" - elif self.className == "str": + elif self.className == SpecialClass.STR: return "i32" - elif self.className == "object": + elif self.className == SpecialClass.OBJECT: return "i32" - elif self.className == "": + elif self.className == SpecialClass.NONE: return "i32" - elif self.className == "": + elif self.className == SpecialClass.EMPTY: return "i32" - elif self.className == "int": + elif self.className == SpecialClass.INT: return "i64" else: return "i32" @@ -112,8 +122,24 @@ def __str__(self): def __hash__(self): return str(self).__hash__() - def toJSON(self, dump_location=True): + def toJSON(self, dump_location=True) -> dict: return { "kind": "ClassValueType", "className": self.className } + + def getLLVMType(self) -> ir.Type: + if self.className == SpecialClass.BOOL: + return ir.IntType() + elif self.className == SpecialClass.STR: + raise Exception("unsupported") + elif self.className == SpecialClass.OBJECT: + raise Exception("unsupported") + elif self.className == SpecialClass.NONE: + raise Exception("unsupported") + elif self.className == SpecialClass.EMPTY: + raise Exception("unsupported") + elif self.className == SpecialClass.INT: + return ir.IntType() + else: + raise Exception("unsupported") diff --git a/compiler/types/functype.py b/compiler/types/functype.py index b02311b..675aa6d 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -2,6 +2,7 @@ from .valuetype import ValueType from .symboltype import SymbolType from typing import List +from llvmlite import ir class FuncType(SymbolType): @@ -65,12 +66,12 @@ def getWasmSignature(self, names=None) -> str: ) else f" (result {self.returnType.getWasmName()})" return params + result - def methodEquals(self, other): + def methodEquals(self, other) -> bool: if isinstance(other, FuncType) and len(self.parameters) > 0 and len(other.parameters) > 0: return self.parameters[1:] == other.parameters[1:] and self.returnType == other.returnType return False - def isFuncType(): + def isFuncType() -> bool: return True def __str__(self): @@ -81,9 +82,20 @@ def __hash__(self): paramStr = ",".join([str(t) for t in self.parameters]) return (F"[{paramStr}]->{self.returnType}").__hash__() - def toJSON(self, dump_location=True): + def toJSON(self, dump_location=True) -> dict: return { "kind": "FuncType", "parameters": [p.toJSON(dump_location) for p in self.parameters], "returnType": self.returnType.toJSON(dump_location) } + + def getLLVMType(self) -> ir.Type: + params = [] + for i in range(len(self.parameters)): + p = self.parameters[i] + if i in self.refParams and isinstance(p, ClassValueType): + sig = p.getLLVMType().as_pointer() + else: + sig = p.getLLVMType() + params.append(sig) + return ir.FunctionType(self.returnType.getLLVMType(), params) diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 426d269..cd09ba9 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -1,4 +1,5 @@ from .valuetype import ValueType +from llvmlite import ir class ListValueType(ValueType): @@ -11,22 +12,22 @@ def __eq__(self, other): return self.elementType == other.elementType return False - def getJavaSignature(self, _=False): + def getJavaSignature(self, _=False) -> str: return "[" + self.elementType.getJavaSignature(True) - def getJavaName(self, _=False): + def getJavaName(self, _=False) -> str: return "[" + self.elementType.getJavaSignature(True) - def getCILName(self, _=False): + def getCILName(self, _=False) -> str: return self.elementType.getCILName() + "[]" - def getCILSignature(self, _=False): + def getCILSignature(self, _=False) -> str: return self.getCILName() - def isListType(self): + def isListType(self) -> bool: return True - def isJavaRef(self): + def isJavaRef(self) -> bool: return True def __str__(self): @@ -35,11 +36,14 @@ def __str__(self): def __hash__(self): return str(self).__hash__() - def toJSON(self, dump_location=True): + def toJSON(self, dump_location=True) -> dict: return { "kind": "ListValueType", "elementType": self.elementType.toJSON(dump_location) } - def getWasmName(self): + def getWasmName(self) -> str: return "i32" + + def getLLVMType(self) -> ir.Type: + raise Exception("unimplemented") diff --git a/compiler/types/symboltype.py b/compiler/types/symboltype.py index 61f82f6..dd906ca 100644 --- a/compiler/types/symboltype.py +++ b/compiler/types/symboltype.py @@ -18,6 +18,3 @@ def isSpecialType(): def toJSON(self, dump_location=True): raise Exception("unsupported") - - def llvmType(self, typeSystem): - raise Exception("unsupported") diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index 09ea240..70f1b34 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -11,11 +11,14 @@ def isNone(self): def toJSON(self, dump_location=True): raise Exception("unsupported") - def getJavaSignature(self) -> str: + def getJavaSignature(self): raise Exception("unsupported") - def isJavaRef(self) -> bool: + def isJavaRef(self): raise Exception("unsupported") def isListType(self): raise Exception("unsupported") + + def getLLVMType(self): + raise Exception("unsupported") From c1eadbb6e4b3e329b94531d960ada0e59a20db30 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 15 May 2023 00:28:30 -0700 Subject: [PATCH 46/79] printf works --- compiler/compiler.py | 6 +- compiler/llvm_backend.py | 114 ++++++++++++++++++++++++++++--- compiler/types/__init__.py | 2 +- compiler/types/classvaluetype.py | 8 +-- compiler/types/functype.py | 3 +- foobar.py | 1 + main.py | 8 +-- test.py | 75 ++++++++++++++++++-- 8 files changed, 187 insertions(+), 30 deletions(-) create mode 100644 foobar.py diff --git a/compiler/compiler.py b/compiler/compiler.py index e76b351..1c1880c 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -84,9 +84,9 @@ def emitWASM(self, main: str, ast: Node): wasm_backend.visit(ast) return wasm_backend.builder - def emitLLVM(self, main: str, ast: Node): + def emitLLVM(self, ast: Node): self.closurepass(ast) EmptyListTyper().visit(ast) - llvm_backend = LlvmBackend(main, self.transformer.ts) + llvm_backend = LlvmBackend(self.transformer.ts) llvm_backend.visit(ast) - return llvm_backend.builder + return llvm_backend.module diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index be4b0fc..41d89fb 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -9,10 +9,19 @@ import llvmlite.binding as llvm +int8_t = ir.IntType(8) # for booleans +int32_t = ir.IntType(32) # for ints +voidptr_t = ir.IntType(8).as_pointer() + + class LlvmBackend(Visitor): locals = [] + counter = 0 - def __init__(self, main: str, ts: TypeSystem): + def __init__(self, ts: TypeSystem): + llvm.initialize() + llvm.initialize_native_target() + llvm.initialize_native_asmprinter() self.module = ir.Module() self.builder = None @@ -25,10 +34,25 @@ def exitScope(self): def visit(self, node: Node): return node.visit(self) + def visitStmtList(self, stmts: List[Stmt]): + for s in stmts: + self.visit(s) + + def newLocal(self): + self.counter += 1 + return f"__local_{self.counter}" + # TOP LEVEL & DECLARATIONS def Program(self, node: Program): - pass + funcType = ir.FunctionType(ir.VoidType(), []) + func = ir.Function(self.module, funcType, "__main__") + self.enterScope() + bb_entry = func.append_basic_block('entry') + self.builder = ir.IRBuilder(bb_entry) + self.visitStmtList(node.statements) + self.builder.ret_void() + self.exitScope() def VarDef(self, node: VarDef): pass @@ -37,7 +61,24 @@ def ClassDef(self, node: ClassDef): pass def FuncDef(self, node: FuncDef): - pass + funcname = node.name + returnType = node.type.returnType.getLLVMType() + argTypes = [p.getLLVMType() for p in node.type.parameters] + funcType = ir.FunctionType(returnType, argTypes) + func = ir.Function(self.module, funcType, funcname) + self.enterScope() + bb_entry = func.append_basic_block('entry') + self.builder = ir.IRBuilder(bb_entry) + for i, arg in enumerate(func.args): + arg.name = node.proto.argnames[i] + alloca = self.builder.alloca( + node.type.parameters[i].getLLVMType(), name=arg.name) + self.builder.store(arg, alloca) + self.locals[-1][arg.name] = alloca + self.visitStmtList(node.statements) + # self.builder.ret(retval) + self.exitScope() + return func # STATEMENTS @@ -54,7 +95,7 @@ def IfStmt(self, node: IfStmt): pass def ExprStmt(self, node: ExprStmt): - pass + self.visit(node.expr) def BinaryExpr(self, node: BinaryExpr): pass @@ -66,7 +107,17 @@ def UnaryExpr(self, node: UnaryExpr): pass def CallExpr(self, node: CallExpr): - pass + if node.function.name == "print": + self.emit_print(node.args[0]) + return + callee_func = self.module.get_global(node.function.name) + if callee_func is None or not isinstance(callee_func, ir.Function): + raise Exception("unknown function") + if len(callee_func.args) != len(node.args): + raise Exception('Call argument length mismatch', + node.function.name) + call_args = [self.visit(arg) for arg in node.args] + return self.builder.call(callee_func, call_args, 'calltmp') def ForStmt(self, node: ForStmt): pass @@ -81,7 +132,8 @@ def ReturnStmt(self, node: ReturnStmt): pass def Identifier(self, node: Identifier): - pass + addr = self.locals[-1][node.name] + return self.builder.load(addr, node.name) def MemberExpr(self, node: MemberExpr): pass @@ -95,16 +147,19 @@ def MethodCallExpr(self, node: MethodCallExpr): # LITERALS def BooleanLiteral(self, node: BooleanLiteral): - pass + return ir.Constant(int8_t, 1 if node.value else 0) def IntegerLiteral(self, node: IntegerLiteral): - pass + return ir.Constant(int32_t, node.value) def NoneLiteral(self, node: NoneLiteral): - pass + return ir.Constant(int32_t, 0) def StringLiteral(self, node: StringLiteral): - pass + const = self.make_bytearray((node.value + '\00').encode('ascii')) + alloca = self.builder.alloca(ir.ArrayType(int8_t, len(node.value) + 1), name=self.newLocal()) + self.builder.store(const, alloca) + return alloca # TYPES @@ -116,3 +171,42 @@ def ListType(self, node: ListType): def ClassType(self, node: ClassType): pass + + # BUILT-INS + + def emit_print(self, arg: Expr): + if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: + raise Exception("unsupported") + if arg.inferredType.className == "bool": + raise Exception("TODO") + elif arg.inferredType.className == 'int': + return self.printf("%i\n", False, self.visit(arg)) + else: + return self.printf("%s\n", True, self.visit(arg)) + + # UTILS + + def make_bytearray(self, buf): + b = bytearray(buf) + n = len(b) + return ir.Constant(ir.ArrayType(int8_t, n), b) + + def is_null(self, value): + return self.builder.icmp_unsigned('==', value.type(None), value) + + def is_nonnull(self, value): + return self.builder.icmp_unsigned('!=', value.type(None), value) + + def printf(self, format: str, cast: bool, arg): + func_t = ir.FunctionType(int32_t, [voidptr_t], True) + fmt_bytes = self.make_bytearray((format + '\00').encode('ascii')) + alloca = self.builder.alloca(ir.ArrayType(int8_t, 4), name=self.newLocal()) + self.builder.store(fmt_bytes, alloca) + try: + fn = self.module.get_global('printf') + except KeyError: + fn = ir.Function(self.module, func_t, 'printf') + fmt_ptr = self.builder.bitcast(alloca, voidptr_t) + if cast: + arg = self.builder.bitcast(arg, voidptr_t) + return self.builder.call(fn, [fmt_ptr, arg]) diff --git a/compiler/types/__init__.py b/compiler/types/__init__.py index b9f7afe..732d0d6 100644 --- a/compiler/types/__init__.py +++ b/compiler/types/__init__.py @@ -1,4 +1,4 @@ -from .classvaluetype import ClassValueType +from .classvaluetype import ClassValueType, SpecialClass from .functype import FuncType from .listvaluetype import ListValueType from .symboltype import SymbolType diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 1f40774..8c9c64d 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -130,16 +130,16 @@ def toJSON(self, dump_location=True) -> dict: def getLLVMType(self) -> ir.Type: if self.className == SpecialClass.BOOL: - return ir.IntType() + return ir.IntType(8) elif self.className == SpecialClass.STR: - raise Exception("unsupported") + raise ir.IntType(8).as_pointer() elif self.className == SpecialClass.OBJECT: raise Exception("unsupported") elif self.className == SpecialClass.NONE: - raise Exception("unsupported") + raise ir.VoidType() elif self.className == SpecialClass.EMPTY: raise Exception("unsupported") elif self.className == SpecialClass.INT: - return ir.IntType() + return ir.IntType(32) else: raise Exception("unsupported") diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 675aa6d..7063d94 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -98,4 +98,5 @@ def getLLVMType(self) -> ir.Type: else: sig = p.getLLVMType() params.append(sig) - return ir.FunctionType(self.returnType.getLLVMType(), params) + returnType = self.returnType.getLLVMType() + return ir.FunctionType(returnType, params) diff --git a/foobar.py b/foobar.py new file mode 100644 index 0000000..125a75c --- /dev/null +++ b/foobar.py @@ -0,0 +1 @@ +print("foobar") diff --git a/main.py b/main.py index 64e2c30..71eec78 100644 --- a/main.py +++ b/main.py @@ -142,14 +142,14 @@ def main(): out_msg(fname, args.verbose) f.write(wat_emitter.emit()) elif args.mode == "llvm": - llvm_emitter = compiler.emitLLVM(infile_name, tree) + llvm_module = compiler.emitLLVM(infile_name, tree) if args.should_print: - print(llvm_emitter.emit()) + print(str(llvm_module)) else: - fname = outdir + llvm_emitter.name + ".ll" + fname = outdir + llvm_module.name + ".ll" with open(fname, "w") as f: out_msg(fname, args.verbose) - f.write(llvm_emitter.emit()) + f.write(str(llvm_module)) if __name__ == "__main__": diff --git a/test.py b/test.py index fc9ce43..a733bd0 100644 --- a/test.py +++ b/test.py @@ -7,6 +7,8 @@ from compiler.typeeraser import TypeEraser from compiler.typesystem import TypeSystem from compiler.compiler import Compiler +import llvmlite.binding as llvm +from ctypes import CFUNCTYPE dump_location = True error_flags = {"error", "Error", "Exception", @@ -14,13 +16,15 @@ def run_all_tests(): - run_parse_tests() - run_typecheck_tests() - run_python_backend_tests() - run_closure_tests() - run_jvm_tests() - run_cil_tests() - run_wasm_tests() + # run_parse_tests() + # run_typecheck_tests() + # run_python_backend_tests() + # run_closure_tests() + # run_jvm_tests() + # run_cil_tests() + # run_wasm_tests() + # run_llvm_tests() + test_eval_llvm() def run_parse_tests(): @@ -581,3 +585,60 @@ def ast_equals(d1, d2) -> bool: if d1 != d2: print("Expected {:s}, got {:s}".format(str(d1), str(d2))) return d1 == d2 + + +def run_llvm_tests(): + print("Running LLVM backend tests...\n") + total = 0 + n_passed = 0 + llvm_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() + for test in llvm_tests_dir.glob('*.py'): + passed = run_llvm_test(test) + total += 1 + if not passed: + print("Failed: " + str(test) + "\n") + else: + n_passed += 1 + if total != n_passed: + print("\nNot all test cases passed") + print("\nPassed {:d} out of {:d} LLVM backend test cases\n".format( + n_passed, total)) + + +def eval_llvm(module): + target = llvm.Target.from_default_triple() + target_machine = target.create_target_machine() + llvmmod = llvm.parse_assembly(str(module)) + with llvm.create_mcjit_compiler(llvmmod, target_machine) as ee: + ee.finalize_object() + fptr = CFUNCTYPE(None)(ee.get_function_address("__main__")) + fptr() + + +def run_llvm_test(test): + pass + + +def test_eval_llvm(): + llvm_debug("foobar.py") + + +def llvm_debug(test): + try: + compiler = Compiler() + astparser = compiler.parser + chocopy_ast = compiler.parse(test) + if len(astparser.errors) > 0: + return False + compiler.typecheck(chocopy_ast) + module = compiler.emitLLVM(chocopy_ast) + print("Module output:") + print(str(module)) + print("Evaluation output:") + eval_llvm(module) + except Exception as e: + print("Internal compiler error:", test) + track = traceback.format_exc() + print(e) + print(track) + return False From 7f0b71c03bc627db557c25db037526a15b533bf5 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 16 May 2023 17:45:17 -0700 Subject: [PATCH 47/79] add operators --- compiler/llvm_backend.py | 68 ++++++++++++++++++++++++++++++++++++-- foobar.py | 2 +- tests/runtime/operators.py | 1 + 3 files changed, 67 insertions(+), 4 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 41d89fb..0e38796 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -40,7 +40,7 @@ def visitStmtList(self, stmts: List[Stmt]): def newLocal(self): self.counter += 1 - return f"__local_{self.counter}" + return f".local_{self.counter}" # TOP LEVEL & DECLARATIONS @@ -97,14 +97,76 @@ def IfStmt(self, node: IfStmt): def ExprStmt(self, node: ExprStmt): self.visit(node.expr) + def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: + return leftType.isListType() and rightType.isListType() and operator == "+" + def BinaryExpr(self, node: BinaryExpr): - pass + operator = node.operator + leftType = node.left.inferredType + rightType = node.right.inferredType + lhs = self.visit(node.left) + rhs = self.visit(node.right) + # concatenation and addition + if operator == "+": + if self.isListConcat(operator, leftType, rightType): + raise Exception("unimplemented") + elif leftType == StrType(): + raise Exception("unimplemented") + elif leftType == IntType(): + return self.builder.add(lhs, rhs) + else: + raise Exception( + "Internal compiler error: unexpected operand types for +") + # other arithmetic operators + elif operator == "-": + return self.builder.sub(lhs, rhs) + elif operator == "*": + return self.builder.mul(lhs, rhs) + elif operator == "//": + return self.builder.sdiv(lhs, rhs) + elif operator == "%": + return self.builder.urem(lhs, rhs) + # relational operators + elif operator in {"<", "<=", ">", ">="}: + return self.builder.icmp_signed(operator, lhs, rhs) + elif operator == "==": + if leftType == IntType(): + return self.builder.icmp_signed(operator, lhs, rhs) + elif leftType == StrType(): + raise Exception("Unimplemented") + else: + return self.instr("i32.eq") + elif operator == "!=": + if leftType == IntType(): + return self.builder.icmp_signed(operator, lhs, rhs) + elif leftType == StrType(): + raise Exception("Unimplemented") + else: + # pointer comparisons + return self.builder.icmp_unsigned(operator, lhs, rhs) + elif operator == "is": + # pointer comparisons + return self.builder.icmp_unsigned("==", lhs, rhs) + # logical operators + elif operator == "and": + return self.builder.and_(lhs, rhs) + elif operator == "or": + return self.builder.or_(lhs, rhs) + else: + raise Exception( + f"Internal compiler error: unexpected operator {operator}") def IndexExpr(self, node: IndexExpr): pass def UnaryExpr(self, node: UnaryExpr): - pass + if node.operator == "-": + val = self.visit(node.operand) + return self.builder.neg(val) + elif node.operator == "not": + false = ir.Constant(int8_t, 0) + val = self.visit(node.operand) + return self.builder.icmp_unsigned('==', false, val) def CallExpr(self, node: CallExpr): if node.function.name == "print": diff --git a/foobar.py b/foobar.py index 125a75c..e38606f 100644 --- a/foobar.py +++ b/foobar.py @@ -1 +1 @@ -print("foobar") +print(-5 % 2) diff --git a/tests/runtime/operators.py b/tests/runtime/operators.py index d331320..a3a163a 100644 --- a/tests/runtime/operators.py +++ b/tests/runtime/operators.py @@ -21,6 +21,7 @@ assert w * x == x assert 5 // 2 == y assert 5 % 2 == x +assert -5 % 2 == 1 assert not False assert not (w != x) assert -x == -1 From c9e2fbaccd9fc09188aea73d3290a849b4bacb76 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 16 May 2023 17:52:45 -0700 Subject: [PATCH 48/79] add TODO for modulo operator fix --- Makefile | 3 ++- compiler/llvm_backend.py | 2 +- foobar.py | 2 +- tests/runtime/operators.py | 5 ++++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 558c1df..709a3c3 100644 --- a/Makefile +++ b/Makefile @@ -8,4 +8,5 @@ clean: rm -f *.test.py rm -f *.out.py rm -f *.wasm - rm -f *.wat \ No newline at end of file + rm -f *.wat + rm -f *.ll \ No newline at end of file diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 0e38796..b4696e0 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -125,7 +125,7 @@ def BinaryExpr(self, node: BinaryExpr): elif operator == "//": return self.builder.sdiv(lhs, rhs) elif operator == "%": - return self.builder.urem(lhs, rhs) + return self.builder.srem(lhs, rhs) # relational operators elif operator in {"<", "<=", ">", ">="}: return self.builder.icmp_signed(operator, lhs, rhs) diff --git a/foobar.py b/foobar.py index e38606f..3e8ac3b 100644 --- a/foobar.py +++ b/foobar.py @@ -1 +1 @@ -print(-5 % 2) +print(-5 % -2) diff --git a/tests/runtime/operators.py b/tests/runtime/operators.py index a3a163a..500fd91 100644 --- a/tests/runtime/operators.py +++ b/tests/runtime/operators.py @@ -21,7 +21,10 @@ assert w * x == x assert 5 // 2 == y assert 5 % 2 == x -assert -5 % 2 == 1 +# TODO: fix modulo operator behavior +# assert -5 % 2 == 1 +# assert 5 % -2 == -1 +# assert -5 % -2 == -1 assert not False assert not (w != x) assert -x == -1 From 731f07eb4b3b405a7a89849368c72282eeefe6f9 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 16 May 2023 23:28:05 -0700 Subject: [PATCH 49/79] variables, assignments, conditionals --- compiler/llvm_backend.py | 151 ++++++++++++++++++++++++------- compiler/types/classvaluetype.py | 6 +- foobar.py | 8 +- main.py | 2 +- 4 files changed, 127 insertions(+), 40 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index b4696e0..9fa5b22 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -8,8 +8,8 @@ import llvmlite.ir as ir import llvmlite.binding as llvm - -int8_t = ir.IntType(8) # for booleans +bool_t = ir.IntType(1) # for booleans +int8_t = ir.IntType(8) # chars int32_t = ir.IntType(32) # for ints voidptr_t = ir.IntType(8).as_pointer() @@ -17,6 +17,8 @@ class LlvmBackend(Visitor): locals = [] counter = 0 + globals = {} + externs = {} def __init__(self, ts: TypeSystem): llvm.initialize() @@ -45,17 +47,54 @@ def newLocal(self): # TOP LEVEL & DECLARATIONS def Program(self, node: Program): + # globals for commonly used strings + self.globals = { + 'true': self.global_constant('true', + ir.ArrayType(int8_t, 5), + self.make_bytearray('True\00'.encode('ascii'))), + 'false': self.global_constant('false', + ir.ArrayType(int8_t, 6), + self.make_bytearray('False\00'.encode('ascii'))), + 'fmt_i': self.global_constant('fmt_i', + ir.ArrayType(int8_t, 4), + self.make_bytearray('%i\n\00'.encode('ascii'))), + 'fmt_s': self.global_constant('fmt_s', + ir.ArrayType(int8_t, 4), + self.make_bytearray('%s\n\00'.encode('ascii'))), + 'fmt_assert': self.global_constant('fmt_assert', + ir.ArrayType(int8_t, 29), + self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))) + } + + printf_t = ir.FunctionType(int32_t, [voidptr_t], True) + self.externs['printf'] = ir.Function(self.module, printf_t, 'printf') + setjmp_t = ir.FunctionType(int32_t, [int32_t], True) + self.externs['setjmp'] = ir.Function(self.module, setjmp_t, 'setjmp') + # TODO make this the right type + longjmp_t = ir.FunctionType(int32_t, [int32_t], True) + self.externs['longjmp'] = ir.Function( + self.module, longjmp_t, 'longjmp') + funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") + self.enterScope() - bb_entry = func.append_basic_block('entry') - self.builder = ir.IRBuilder(bb_entry) + entry_block = func.append_basic_block('entry') + self.builder = ir.IRBuilder(entry_block) + self.visitStmtList( + [d for d in node.declarations if isinstance(d, VarDef)]) self.visitStmtList(node.statements) self.builder.ret_void() self.exitScope() def VarDef(self, node: VarDef): - pass + val = self.visit(node.value) + saved_block = self.builder.block + addr = self.create_entry_block_alloca( + node.getName(), node.var.t.getLLVMType()) + self.builder.position_at_end(saved_block) + self.builder.store(val, addr) + self.locals[-1][node.getName()] = addr def ClassDef(self, node: ClassDef): pass @@ -82,14 +121,18 @@ def FuncDef(self, node: FuncDef): # STATEMENTS - def NonLocalDecl(self, node: NonLocalDecl): - pass - - def GlobalDecl(self, node: GlobalDecl): - pass - def AssignStmt(self, node: AssignStmt): - pass + val = self.visit(node.value) + for var in node.targets[::-1]: + if isinstance(var, MemberExpr): + raise Exception("unimplemented") + elif isinstance(var, IndexExpr): + raise Exception("unimplemented") + elif isinstance(var, Identifier): + addr = self.locals[-1][var.name] + self.builder.store(val, addr) + else: + raise Exception("Illegal assignment") def IfStmt(self, node: IfStmt): pass @@ -164,7 +207,7 @@ def UnaryExpr(self, node: UnaryExpr): val = self.visit(node.operand) return self.builder.neg(val) elif node.operator == "not": - false = ir.Constant(int8_t, 0) + false = ir.Constant(bool_t, 0) val = self.visit(node.operand) return self.builder.icmp_unsigned('==', false, val) @@ -195,13 +238,42 @@ def ReturnStmt(self, node: ReturnStmt): def Identifier(self, node: Identifier): addr = self.locals[-1][node.name] + assert addr is not None return self.builder.load(addr, node.name) def MemberExpr(self, node: MemberExpr): pass def IfExpr(self, node: IfExpr): - pass + return self.ifHelper(lambda: self.visit(node.condition), + lambda: self.visit(node.thenExpr), + lambda: self.visit(node.elseExpr), + node.inferredType.getLLVMType()) + + def ifHelper(self, condFn, thenFn, elseFn, t): + cond = condFn() + + then_block = self.builder.append_basic_block() + else_block = self.builder.append_basic_block() + merge_block = self.builder.append_basic_block() + self.builder.cbranch(cond, then_block, else_block) + + self.builder.position_at_start(then_block) + then_val = thenFn() + self.builder.branch(merge_block) + then_block = self.builder.block + + self.builder.position_at_start(else_block) + else_val = elseFn() + self.builder.branch(merge_block) + else_block = self.builder.block + + self.builder.position_at_start(merge_block) + + phi = self.builder.phi(t) + phi.add_incoming(then_val, then_block) + phi.add_incoming(else_val, else_block) + return phi def MethodCallExpr(self, node: MethodCallExpr): pass @@ -209,19 +281,20 @@ def MethodCallExpr(self, node: MethodCallExpr): # LITERALS def BooleanLiteral(self, node: BooleanLiteral): - return ir.Constant(int8_t, 1 if node.value else 0) + return ir.Constant(bool_t, 1 if node.value else 0) def IntegerLiteral(self, node: IntegerLiteral): return ir.Constant(int32_t, node.value) def NoneLiteral(self, node: NoneLiteral): - return ir.Constant(int32_t, 0) + return ir.Constant(ir.PointerType(None), None) def StringLiteral(self, node: StringLiteral): const = self.make_bytearray((node.value + '\00').encode('ascii')) - alloca = self.builder.alloca(ir.ArrayType(int8_t, len(node.value) + 1), name=self.newLocal()) + alloca = self.builder.alloca(ir.ArrayType( + int8_t, len(node.value) + 1), name=self.newLocal()) self.builder.store(const, alloca) - return alloca + return self.builder.bitcast(alloca, voidptr_t) # TYPES @@ -238,13 +311,18 @@ def ClassType(self, node: ClassType): def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: - raise Exception("unsupported") + raise Exception("Only bool, int, or str may be printed") if arg.inferredType.className == "bool": - raise Exception("TODO") + text = self.ifHelper( + lambda: arg, + lambda: self.builder.bitcast(self.globals['true'], voidptr_t), + lambda: self.builder.bitcast(self.globals['false'], voidptr_t), + voidptr_t) + return self.printf(self.globals['fmt_s'], True, text) elif arg.inferredType.className == 'int': - return self.printf("%i\n", False, self.visit(arg)) + return self.printf(self.globals['fmt_i'], False, self.visit(arg)) else: - return self.printf("%s\n", True, self.visit(arg)) + return self.printf(self.globals['fmt_s'], True, self.visit(arg)) # UTILS @@ -259,16 +337,19 @@ def is_null(self, value): def is_nonnull(self, value): return self.builder.icmp_unsigned('!=', value.type(None), value) - def printf(self, format: str, cast: bool, arg): - func_t = ir.FunctionType(int32_t, [voidptr_t], True) - fmt_bytes = self.make_bytearray((format + '\00').encode('ascii')) - alloca = self.builder.alloca(ir.ArrayType(int8_t, 4), name=self.newLocal()) - self.builder.store(fmt_bytes, alloca) - try: - fn = self.module.get_global('printf') - except KeyError: - fn = ir.Function(self.module, func_t, 'printf') - fmt_ptr = self.builder.bitcast(alloca, voidptr_t) - if cast: - arg = self.builder.bitcast(arg, voidptr_t) - return self.builder.call(fn, [fmt_ptr, arg]) + def printf(self, format, cast: bool, arg): + fmt_ptr = self.builder.bitcast(format, voidptr_t) + return self.builder.call(self.externs['printf'], [fmt_ptr, arg]) + + def create_entry_block_alloca(self, name, t): + builder = ir.IRBuilder() + builder.position_at_start(self.builder.function.entry_basic_block) + return builder.alloca(t, size=None, name=name) + + def global_constant(self, name, t, value): + module = self.module + data = ir.GlobalVariable(module, t, name, 0) + data.linkage = 'internal' + data.global_constant = True + data.initializer = value + return data diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 8c9c64d..43450ce 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -130,13 +130,13 @@ def toJSON(self, dump_location=True) -> dict: def getLLVMType(self) -> ir.Type: if self.className == SpecialClass.BOOL: - return ir.IntType(8) + return ir.IntType(1) elif self.className == SpecialClass.STR: - raise ir.IntType(8).as_pointer() + return ir.IntType(8).as_pointer() elif self.className == SpecialClass.OBJECT: raise Exception("unsupported") elif self.className == SpecialClass.NONE: - raise ir.VoidType() + return ir.VoidType() elif self.className == SpecialClass.EMPTY: raise Exception("unsupported") elif self.className == SpecialClass.INT: diff --git a/foobar.py b/foobar.py index 3e8ac3b..c3ade35 100644 --- a/foobar.py +++ b/foobar.py @@ -1 +1,7 @@ -print(-5 % -2) +x: str = "1" +y: str = "0" +print(x) +print(y) +x = y = "2" +print(x) +print(y) diff --git a/main.py b/main.py index 71eec78..e699036 100644 --- a/main.py +++ b/main.py @@ -13,7 +13,7 @@ 'jvm - output JVM bytecode formatted for the Krakatau assembler\n' + 'cil - output CIL bytecode formatted for the Mono ilasm assembler\n' + 'wasm - output WASM in WAT format\n' + - 'llvm - output LLVM\n' + 'llvm - output LLVM IR\n' ) From 9e11ab5e3d00bebe56ed22a2dffead503da52417 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 18 May 2023 00:57:57 -0700 Subject: [PATCH 50/79] error handling --- compiler/llvm_backend.py | 114 +++++++++++++++++++++++++-------------- foobar.py | 7 --- 2 files changed, 75 insertions(+), 46 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 9fa5b22..47524dc 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -12,6 +12,7 @@ int8_t = ir.IntType(8) # chars int32_t = ir.IntType(32) # for ints voidptr_t = ir.IntType(8).as_pointer() +jmp_buf_t = ir.ArrayType(ir.IntType(8), 100) class LlvmBackend(Visitor): @@ -68,12 +69,10 @@ def Program(self, node: Program): printf_t = ir.FunctionType(int32_t, [voidptr_t], True) self.externs['printf'] = ir.Function(self.module, printf_t, 'printf') - setjmp_t = ir.FunctionType(int32_t, [int32_t], True) + setjmp_t = ir.FunctionType(int32_t, [jmp_buf_t.as_pointer()]) self.externs['setjmp'] = ir.Function(self.module, setjmp_t, 'setjmp') - # TODO make this the right type - longjmp_t = ir.FunctionType(int32_t, [int32_t], True) - self.externs['longjmp'] = ir.Function( - self.module, longjmp_t, 'longjmp') + longjmp_t = ir.FunctionType(ir.VoidType(), [jmp_buf_t.as_pointer(), int32_t]) + self.externs['longjmp'] = ir.Function(self.module, longjmp_t, 'longjmp') funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") @@ -81,11 +80,35 @@ def Program(self, node: Program): self.enterScope() entry_block = func.append_basic_block('entry') self.builder = ir.IRBuilder(entry_block) + + jmp_buf = self.global_constant("jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * 100))) + status = self.builder.call(self.externs['setjmp'], [jmp_buf]) + cond = self.builder.icmp_signed("!=", ir.Constant(int32_t, 0), status) + + error_block = self.builder.append_basic_block('error_handling') + program_block = self.builder.append_basic_block('program_code') + merge_block = self.builder.append_basic_block('end_program') + self.builder.cbranch(cond, + error_block, + program_block) + + self.builder.position_at_start(error_block) + self.printf(self.globals['fmt_assert'], status) + self.builder.branch(merge_block) + error_block = self.builder.block + + self.builder.position_at_start(program_block) + self.programHelper(node) + self.builder.branch(merge_block) + program_block = self.builder.block + self.builder.position_at_start(merge_block) + self.builder.ret_void() + self.exitScope() + + def programHelper(self, node: Program): self.visitStmtList( [d for d in node.declarations if isinstance(d, VarDef)]) self.visitStmtList(node.statements) - self.builder.ret_void() - self.exitScope() def VarDef(self, node: VarDef): val = self.visit(node.value) @@ -135,7 +158,12 @@ def AssignStmt(self, node: AssignStmt): raise Exception("Illegal assignment") def IfStmt(self, node: IfStmt): - pass + if len(node.elseBody) == 0: + self.ifHelper(lambda: self.visit(node.condition), lambda: self.visitStmtList( + node.thenBody)) + else: + self.ifHelper(lambda: self.visit(node.condition), lambda: self.visitStmtList( + node.thenBody), lambda: self.visitStmtList(node.elseBody)) def ExprStmt(self, node: ExprStmt): self.visit(node.expr) @@ -178,7 +206,7 @@ def BinaryExpr(self, node: BinaryExpr): elif leftType == StrType(): raise Exception("Unimplemented") else: - return self.instr("i32.eq") + return self.builder.icmp_signed(operator, lhs, rhs) elif operator == "!=": if leftType == IntType(): return self.builder.icmp_signed(operator, lhs, rhs) @@ -207,14 +235,16 @@ def UnaryExpr(self, node: UnaryExpr): val = self.visit(node.operand) return self.builder.neg(val) elif node.operator == "not": - false = ir.Constant(bool_t, 0) val = self.visit(node.operand) - return self.builder.icmp_unsigned('==', false, val) + return self.builder.icmp_unsigned('==', ir.Constant(bool_t, 0), val) def CallExpr(self, node: CallExpr): if node.function.name == "print": self.emit_print(node.args[0]) return + if node.function.name == "__assert__": + self.emit_assert(node.args[0]) + return callee_func = self.module.get_global(node.function.name) if callee_func is None or not isinstance(callee_func, ir.Function): raise Exception("unknown function") @@ -250,30 +280,37 @@ def IfExpr(self, node: IfExpr): lambda: self.visit(node.elseExpr), node.inferredType.getLLVMType()) - def ifHelper(self, condFn, thenFn, elseFn, t): + def ifHelper(self, condFn, thenFn, elseFn=None, returnType=None): cond = condFn() + if returnType is not None: + assert elseFn is not None - then_block = self.builder.append_basic_block() - else_block = self.builder.append_basic_block() - merge_block = self.builder.append_basic_block() - self.builder.cbranch(cond, then_block, else_block) + then_block = self.builder.append_basic_block('then') + if elseFn is not None: + else_block = self.builder.append_basic_block('else') + merge_block = self.builder.append_basic_block('merge') + self.builder.cbranch(cond, + then_block, + else_block if elseFn is not None else merge_block) self.builder.position_at_start(then_block) then_val = thenFn() self.builder.branch(merge_block) then_block = self.builder.block - self.builder.position_at_start(else_block) - else_val = elseFn() - self.builder.branch(merge_block) - else_block = self.builder.block + if elseFn is not None: + self.builder.position_at_start(else_block) + else_val = elseFn() + self.builder.branch(merge_block) + else_block = self.builder.block self.builder.position_at_start(merge_block) - phi = self.builder.phi(t) - phi.add_incoming(then_val, then_block) - phi.add_incoming(else_val, else_block) - return phi + if returnType is not None: + phi = self.builder.phi(returnType, 'phi') + phi.add_incoming(then_val, then_block) + phi.add_incoming(else_val, else_block) + return phi def MethodCallExpr(self, node: MethodCallExpr): pass @@ -296,33 +333,32 @@ def StringLiteral(self, node: StringLiteral): self.builder.store(const, alloca) return self.builder.bitcast(alloca, voidptr_t) - # TYPES - - def TypedVar(self, node: TypedVar): - pass - - def ListType(self, node: ListType): - pass + # BUILT-INS - def ClassType(self, node: ClassType): - pass + def emit_assert(self, arg: Expr): + self.ifHelper( + lambda: self.builder.icmp_unsigned('==', ir.Constant(bool_t, 0), self.visit(arg)), + lambda: self.longJmp(arg.location[0]) + ) - # BUILT-INS + def longJmp(self, line): + jmp_buf = self.module.get_global("jmp_buf") + self.builder.call(self.externs['longjmp'], [jmp_buf, ir.Constant(int32_t, line)]) def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: raise Exception("Only bool, int, or str may be printed") if arg.inferredType.className == "bool": text = self.ifHelper( - lambda: arg, + lambda: self.visit(arg), lambda: self.builder.bitcast(self.globals['true'], voidptr_t), lambda: self.builder.bitcast(self.globals['false'], voidptr_t), voidptr_t) - return self.printf(self.globals['fmt_s'], True, text) + return self.printf(self.globals['fmt_s'], text) elif arg.inferredType.className == 'int': - return self.printf(self.globals['fmt_i'], False, self.visit(arg)) + return self.printf(self.globals['fmt_i'], self.visit(arg)) else: - return self.printf(self.globals['fmt_s'], True, self.visit(arg)) + return self.printf(self.globals['fmt_s'], self.visit(arg)) # UTILS @@ -337,7 +373,7 @@ def is_null(self, value): def is_nonnull(self, value): return self.builder.icmp_unsigned('!=', value.type(None), value) - def printf(self, format, cast: bool, arg): + def printf(self, format, arg): fmt_ptr = self.builder.bitcast(format, voidptr_t) return self.builder.call(self.externs['printf'], [fmt_ptr, arg]) diff --git a/foobar.py b/foobar.py index c3ade35..e69de29 100644 --- a/foobar.py +++ b/foobar.py @@ -1,7 +0,0 @@ -x: str = "1" -y: str = "0" -print(x) -print(y) -x = y = "2" -print(x) -print(y) From 4e4b2618576c67894447fcf11cd59db990ca349e Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 18 May 2023 23:56:43 -0700 Subject: [PATCH 51/79] string len() --- compiler/llvm_backend.py | 13 ++++++++++++- foobar.py | 1 + 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 47524dc..1dbfb72 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -73,6 +73,8 @@ def Program(self, node: Program): self.externs['setjmp'] = ir.Function(self.module, setjmp_t, 'setjmp') longjmp_t = ir.FunctionType(ir.VoidType(), [jmp_buf_t.as_pointer(), int32_t]) self.externs['longjmp'] = ir.Function(self.module, longjmp_t, 'longjmp') + strlen_t = ir.FunctionType(int32_t, [voidptr_t]) + self.externs['strlen'] = ir.Function(self.module, strlen_t, 'strlen') funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") @@ -245,6 +247,8 @@ def CallExpr(self, node: CallExpr): if node.function.name == "__assert__": self.emit_assert(node.args[0]) return + if node.function.name == "len": + return self.emit_len(node.args[0]) callee_func = self.module.get_global(node.function.name) if callee_func is None or not isinstance(callee_func, ir.Function): raise Exception("unknown function") @@ -335,8 +339,15 @@ def StringLiteral(self, node: StringLiteral): # BUILT-INS + def emit_len(self, arg: Expr): + if arg.inferredType == StrType(): + val = self.builder.bitcast(self.visit(arg), voidptr_t) + return self.builder.call(self.externs['strlen'], [val]) + else: + raise Exception("unimplemented") + def emit_assert(self, arg: Expr): - self.ifHelper( + return self.ifHelper( lambda: self.builder.icmp_unsigned('==', ir.Constant(bool_t, 0), self.visit(arg)), lambda: self.longJmp(arg.location[0]) ) diff --git a/foobar.py b/foobar.py index e69de29..e59f49d 100644 --- a/foobar.py +++ b/foobar.py @@ -0,0 +1 @@ +print(len("1234")) \ No newline at end of file From 43fc1ac9f15539466bb9890f83eb99d3ff5a988f Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 19 May 2023 00:13:34 -0700 Subject: [PATCH 52/79] while loop --- compiler/llvm_backend.py | 43 +++++++++++++++++++++++++++++++++------- foobar.py | 5 ++++- 2 files changed, 40 insertions(+), 8 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 1dbfb72..599d8b5 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -89,21 +89,22 @@ def Program(self, node: Program): error_block = self.builder.append_basic_block('error_handling') program_block = self.builder.append_basic_block('program_code') - merge_block = self.builder.append_basic_block('end_program') + end_program = self.builder.append_basic_block('end_program') self.builder.cbranch(cond, error_block, program_block) self.builder.position_at_start(error_block) self.printf(self.globals['fmt_assert'], status) - self.builder.branch(merge_block) + self.builder.branch(end_program) error_block = self.builder.block self.builder.position_at_start(program_block) self.programHelper(node) - self.builder.branch(merge_block) + + self.builder.branch(end_program) program_block = self.builder.block - self.builder.position_at_start(merge_block) + self.builder.position_at_start(end_program) self.builder.ret_void() self.exitScope() @@ -230,7 +231,10 @@ def BinaryExpr(self, node: BinaryExpr): f"Internal compiler error: unexpected operator {operator}") def IndexExpr(self, node: IndexExpr): - pass + if node.list.inferredType == StrType(): + raise Exception("unimplemented") + else: + raise Exception("unimplemented") def UnaryExpr(self, node: UnaryExpr): if node.operator == "-": @@ -265,10 +269,35 @@ def ListExpr(self, node: ListExpr): pass def WhileStmt(self, node: WhileStmt): - pass + while_block = self.builder.append_basic_block('while') + do_block = self.builder.append_basic_block('do') + end_block = self.builder.append_basic_block('end') + self.builder.branch(while_block) + + self.builder.position_at_start(while_block) + cond = self.visit(node.condition) + self.builder.cbranch(cond, + do_block, + end_block) + while_block = self.builder.block + + self.builder.position_at_start(do_block) + self.visitStmtList(node.body) + self.builder.branch(while_block) + do_block = self.builder.block + + self.builder.position_at_start(end_block) def ReturnStmt(self, node: ReturnStmt): - pass + if self.returnType.isNone(): + self.builder.ret_void() + else: + val = None + if node.value is None: + val = self.NoneLiteral(None) + else: + val = self.visit(node.value) + self.builder.ret(val) def Identifier(self, node: Identifier): addr = self.locals[-1][node.name] diff --git a/foobar.py b/foobar.py index e59f49d..9bc1ae1 100644 --- a/foobar.py +++ b/foobar.py @@ -1 +1,4 @@ -print(len("1234")) \ No newline at end of file +x:int = 10 +while x > 0: + print(x) + x = x - 1 \ No newline at end of file From af2aa6163e824d08de8f11a14c16486ac52d69ca Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 19 May 2023 22:51:42 -0700 Subject: [PATCH 53/79] string index and iteration --- compiler/llvm_backend.py | 104 ++++++++++++++++++++++++++++++++------- foobar.py | 10 ++-- 2 files changed, 92 insertions(+), 22 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 599d8b5..adde3f2 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -64,15 +64,20 @@ def Program(self, node: Program): self.make_bytearray('%s\n\00'.encode('ascii'))), 'fmt_assert': self.global_constant('fmt_assert', ir.ArrayType(int8_t, 29), - self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))) + self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))), + 'fmt_err': self.global_constant('fmt_err', + ir.ArrayType(int8_t, 18), + self.make_bytearray('Error on line %i\n\00'.encode('ascii'))) } printf_t = ir.FunctionType(int32_t, [voidptr_t], True) self.externs['printf'] = ir.Function(self.module, printf_t, 'printf') setjmp_t = ir.FunctionType(int32_t, [jmp_buf_t.as_pointer()]) self.externs['setjmp'] = ir.Function(self.module, setjmp_t, 'setjmp') - longjmp_t = ir.FunctionType(ir.VoidType(), [jmp_buf_t.as_pointer(), int32_t]) - self.externs['longjmp'] = ir.Function(self.module, longjmp_t, 'longjmp') + longjmp_t = ir.FunctionType( + ir.VoidType(), [jmp_buf_t.as_pointer(), int32_t]) + self.externs['longjmp'] = ir.Function( + self.module, longjmp_t, 'longjmp') strlen_t = ir.FunctionType(int32_t, [voidptr_t]) self.externs['strlen'] = ir.Function(self.module, strlen_t, 'strlen') @@ -83,7 +88,8 @@ def Program(self, node: Program): entry_block = func.append_basic_block('entry') self.builder = ir.IRBuilder(entry_block) - jmp_buf = self.global_constant("jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * 100))) + jmp_buf = self.global_constant( + "jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * 100))) status = self.builder.call(self.externs['setjmp'], [jmp_buf]) cond = self.builder.icmp_signed("!=", ir.Constant(int32_t, 0), status) @@ -95,7 +101,15 @@ def Program(self, node: Program): program_block) self.builder.position_at_start(error_block) - self.printf(self.globals['fmt_assert'], status) + # if setjmp returns a positive status N, then a user-defined assertion failed on line N + # if setjmp returns a negative status N, then a built-in invariant (such as a bounds check) failed on line -N + self.ifHelper(lambda: self.builder.icmp_signed( + '>', ir.Constant(int32_t, 0), status), + lambda: self.printf(self.globals['fmt_assert'], status), + lambda: self.printf( + self.globals['fmt_err'], self.builder.neg(status)) + ) + self.builder.branch(end_program) error_block = self.builder.block @@ -232,10 +246,40 @@ def BinaryExpr(self, node: BinaryExpr): def IndexExpr(self, node: IndexExpr): if node.list.inferredType == StrType(): - raise Exception("unimplemented") + string = self.visit(node.list) + idx = self.visit(node.index) + return self.strIndex(string, idx, True, node.index.location[0]) else: raise Exception("unimplemented") + def strIndex(self, string, index, check_bounds=False, line: int = 0): + string = self.builder.bitcast(string, voidptr_t) + # bounds checks + # negate the line number for built-in checks + if check_bounds: + self.ifHelper( + lambda: self.builder.icmp_signed( + '>', ir.Constant(int32_t, 0), index), + lambda: self.longJmp(-line) + ) + self.ifHelper( + lambda: self.builder.icmp_signed('<=', + self.builder.call( + self.externs['strlen'], [string]), + index), + lambda: self.longJmp(-line) + ) + ptr = self.builder.gep(string, [index]) + char = self.builder.load(ptr) + alloca = self.builder.alloca(ir.ArrayType( + int8_t, 2), name=self.newLocal()) + alloca = self.builder.bitcast(alloca, voidptr_t) + char_ptr = self.builder.gep(alloca, [ir.Constant(int32_t, 0)]) + self.builder.store(char, char_ptr, 8) + t_ptr = self.builder.gep(alloca, [ir.Constant(int32_t, 1)]) + self.builder.store(ir.Constant(int8_t, 0), t_ptr, 8) + return alloca + def UnaryExpr(self, node: UnaryExpr): if node.operator == "-": val = self.visit(node.operand) @@ -263,26 +307,54 @@ def CallExpr(self, node: CallExpr): return self.builder.call(callee_func, call_args, 'calltmp') def ForStmt(self, node: ForStmt): - pass + var = self.locals[-1][node.identifier.name] + idx = self.builder.alloca(int32_t, None, 'idx') + self.builder.store(ir.Constant(int32_t, 0), idx) + iterable = self.visit(node.iterable) + + if node.iterable.inferredType == StrType(): + self.whileHelper( + lambda: self.builder.icmp_signed("<", + self.builder.load(idx), + self.builder.call(self.externs['strlen'], [iterable])), + lambda: self.forBody(node, + var, + lambda currIdx: self.strIndex( + iterable, currIdx), + idx)) + else: + raise Exception("unimplemented") + + def forBody(self, node: ForStmt, var, idxFn, idx): + currIdx = self.builder.load(idx) + self.builder.store(idxFn(currIdx), var) + self.visitStmtList(node.body) + self.builder.store(self.builder.add( + currIdx, ir.Constant(int32_t, 1)), idx) def ListExpr(self, node: ListExpr): pass def WhileStmt(self, node: WhileStmt): + self.whileHelper( + lambda: self.visit(node.condition), + lambda: self.visitStmtList(node.body)) + + def whileHelper(self, condFn, bodyFn): while_block = self.builder.append_basic_block('while') do_block = self.builder.append_basic_block('do') end_block = self.builder.append_basic_block('end') self.builder.branch(while_block) self.builder.position_at_start(while_block) - cond = self.visit(node.condition) + cond = condFn() self.builder.cbranch(cond, do_block, end_block) while_block = self.builder.block self.builder.position_at_start(do_block) - self.visitStmtList(node.body) + bodyFn() self.builder.branch(while_block) do_block = self.builder.block @@ -377,13 +449,15 @@ def emit_len(self, arg: Expr): def emit_assert(self, arg: Expr): return self.ifHelper( - lambda: self.builder.icmp_unsigned('==', ir.Constant(bool_t, 0), self.visit(arg)), + lambda: self.builder.icmp_unsigned( + '==', ir.Constant(bool_t, 0), self.visit(arg)), lambda: self.longJmp(arg.location[0]) ) - def longJmp(self, line): + def longJmp(self, line: int): jmp_buf = self.module.get_global("jmp_buf") - self.builder.call(self.externs['longjmp'], [jmp_buf, ir.Constant(int32_t, line)]) + self.builder.call(self.externs['longjmp'], [ + jmp_buf, ir.Constant(int32_t, line)]) def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: @@ -407,12 +481,6 @@ def make_bytearray(self, buf): n = len(b) return ir.Constant(ir.ArrayType(int8_t, n), b) - def is_null(self, value): - return self.builder.icmp_unsigned('==', value.type(None), value) - - def is_nonnull(self, value): - return self.builder.icmp_unsigned('!=', value.type(None), value) - def printf(self, format, arg): fmt_ptr = self.builder.bitcast(format, voidptr_t) return self.builder.call(self.externs['printf'], [fmt_ptr, arg]) diff --git a/foobar.py b/foobar.py index 9bc1ae1..eb7537e 100644 --- a/foobar.py +++ b/foobar.py @@ -1,4 +1,6 @@ -x:int = 10 -while x > 0: - print(x) - x = x - 1 \ No newline at end of file +x:str = "9819278632" +i:str = "" +for i in x: + print(i) +for i in x: + print(x) \ No newline at end of file From 4c1fc166e22f045ec33bba904f2e6470843345ef Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 19 May 2023 23:20:13 -0700 Subject: [PATCH 54/79] str concat --- compiler/llvm_backend.py | 30 ++++++++++++++++++++++++++++-- foobar.py | 9 +++------ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index adde3f2..b3c44d1 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -67,20 +67,33 @@ def Program(self, node: Program): self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))), 'fmt_err': self.global_constant('fmt_err', ir.ArrayType(int8_t, 18), - self.make_bytearray('Error on line %i\n\00'.encode('ascii'))) + self.make_bytearray('Error on line %i\n\00'.encode('ascii'))), + 'fmt_str_concat': self.global_constant('fmt_str_concat', + ir.ArrayType(int8_t, 5), + self.make_bytearray('%s%s\00'.encode('ascii'))) } printf_t = ir.FunctionType(int32_t, [voidptr_t], True) self.externs['printf'] = ir.Function(self.module, printf_t, 'printf') + setjmp_t = ir.FunctionType(int32_t, [jmp_buf_t.as_pointer()]) self.externs['setjmp'] = ir.Function(self.module, setjmp_t, 'setjmp') + longjmp_t = ir.FunctionType( ir.VoidType(), [jmp_buf_t.as_pointer(), int32_t]) self.externs['longjmp'] = ir.Function( self.module, longjmp_t, 'longjmp') + strlen_t = ir.FunctionType(int32_t, [voidptr_t]) self.externs['strlen'] = ir.Function(self.module, strlen_t, 'strlen') + sprintf_t = ir.FunctionType(voidptr_t, [voidptr_t, voidptr_t], True) + self.externs['sprintf'] = ir.Function( + self.module, sprintf_t, 'sprintf') + + malloc_t = ir.FunctionType(voidptr_t, [int32_t]) + self.externs['malloc'] = ir.Function(self.module, malloc_t, 'malloc') + funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") @@ -199,7 +212,20 @@ def BinaryExpr(self, node: BinaryExpr): if self.isListConcat(operator, leftType, rightType): raise Exception("unimplemented") elif leftType == StrType(): - raise Exception("unimplemented") + lhs = self.builder.bitcast(lhs, voidptr_t) + rhs = self.builder.bitcast(rhs, voidptr_t) + llen = self.builder.call(self.externs['strlen'], [lhs]) + rlen = self.builder.call(self.externs['strlen'], [rhs]) + total_len = self.builder.add(self.builder.add( + llen, rlen), ir.Constant(int32_t, 1)) + # this is a memory leak since we never free + new_str = self.builder.call( + self.externs['malloc'], [total_len]) + fmt = self.builder.bitcast( + self.globals['fmt_str_concat'], voidptr_t) + self.builder.call(self.externs['sprintf'], [ + new_str, fmt, lhs, rhs]) + return new_str elif leftType == IntType(): return self.builder.add(lhs, rhs) else: diff --git a/foobar.py b/foobar.py index eb7537e..31dd363 100644 --- a/foobar.py +++ b/foobar.py @@ -1,6 +1,3 @@ -x:str = "9819278632" -i:str = "" -for i in x: - print(i) -for i in x: - print(x) \ No newline at end of file +x:str = "123" +y:str = "456" +print(len(x + y + x + y + x + y)) \ No newline at end of file From 20f18c9c7c1ddde68664543aaca719cbe306887b Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 22 May 2023 00:24:24 -0700 Subject: [PATCH 55/79] lists, indexing, iterating, no concat yet --- compiler/llvm_backend.py | 130 +++++++++++++++++++++++--------- compiler/types/listvaluetype.py | 2 +- foobar.py | 19 ++++- test.py | 3 +- 4 files changed, 115 insertions(+), 39 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index b3c44d1..6ca10d7 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -3,16 +3,18 @@ from .typesystem import TypeSystem from .visitor import Visitor from collections import defaultdict -from typing import List +from typing import List, Union import llvmlite.ir as ir import llvmlite.binding as llvm +JMP_BUF_BYTES = 100 + bool_t = ir.IntType(1) # for booleans int8_t = ir.IntType(8) # chars int32_t = ir.IntType(32) # for ints voidptr_t = ir.IntType(8).as_pointer() -jmp_buf_t = ir.ArrayType(ir.IntType(8), 100) +jmp_buf_t = ir.ArrayType(ir.IntType(8), JMP_BUF_BYTES) class LlvmBackend(Visitor): @@ -41,10 +43,6 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) - def newLocal(self): - self.counter += 1 - return f".local_{self.counter}" - # TOP LEVEL & DECLARATIONS def Program(self, node: Program): @@ -102,7 +100,7 @@ def Program(self, node: Program): self.builder = ir.IRBuilder(entry_block) jmp_buf = self.global_constant( - "jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * 100))) + "jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * JMP_BUF_BYTES))) status = self.builder.call(self.externs['setjmp'], [jmp_buf]) cond = self.builder.icmp_signed("!=", ir.Constant(int32_t, 0), status) @@ -114,14 +112,7 @@ def Program(self, node: Program): program_block) self.builder.position_at_start(error_block) - # if setjmp returns a positive status N, then a user-defined assertion failed on line N - # if setjmp returns a negative status N, then a built-in invariant (such as a bounds check) failed on line -N - self.ifHelper(lambda: self.builder.icmp_signed( - '>', ir.Constant(int32_t, 0), status), - lambda: self.printf(self.globals['fmt_assert'], status), - lambda: self.printf( - self.globals['fmt_err'], self.builder.neg(status)) - ) + self.printf(self.globals['fmt_err'], status) self.builder.branch(end_program) error_block = self.builder.block @@ -276,29 +267,58 @@ def IndexExpr(self, node: IndexExpr): idx = self.visit(node.index) return self.strIndex(string, idx, True, node.index.location[0]) else: - raise Exception("unimplemented") + lst = self.visit(node.list) + idx = self.visit(node.index) + self.assert_nonnull(lst, node.list.location[0]) + return self.listIndex(lst, idx, + node.inferredType.getLLVMType(), + True, node.index.location[0]) + + def listIndex(self, list, index, arrType, check_bounds=False, line: int = 0): + length = self.list_len(list) + if check_bounds: + self.ifHelper( + lambda: self.builder.icmp_signed( + '>', ir.Constant(int32_t, 0), index), + lambda: self.longJmp(line) + ) + self.ifHelper( + lambda: self.builder.icmp_signed('<=', + length, + index), + lambda: self.longJmp(line) + ) + structType = ir.LiteralStructType([int32_t, arrType]) + list = self.builder.bitcast(list, structType.as_pointer()) + # get the actual array and cast + data = self.builder.gep(list, [ + ir.Constant(int32_t, 0), + ir.Constant(int32_t, 1)]) + data = self.builder.bitcast(data, arrType.as_pointer()) + # index the array + ptr = self.builder.gep(data, [index]) + return self.builder.load(ptr) def strIndex(self, string, index, check_bounds=False, line: int = 0): string = self.builder.bitcast(string, voidptr_t) # bounds checks - # negate the line number for built-in checks if check_bounds: self.ifHelper( lambda: self.builder.icmp_signed( '>', ir.Constant(int32_t, 0), index), - lambda: self.longJmp(-line) + lambda: self.longJmp(line) ) self.ifHelper( lambda: self.builder.icmp_signed('<=', self.builder.call( self.externs['strlen'], [string]), index), - lambda: self.longJmp(-line) + lambda: self.longJmp(line) ) ptr = self.builder.gep(string, [index]) char = self.builder.load(ptr) alloca = self.builder.alloca(ir.ArrayType( - int8_t, 2), name=self.newLocal()) + int8_t, 2)) alloca = self.builder.bitcast(alloca, voidptr_t) char_ptr = self.builder.gep(alloca, [ir.Constant(int32_t, 0)]) self.builder.store(char, char_ptr, 8) @@ -334,32 +354,60 @@ def CallExpr(self, node: CallExpr): def ForStmt(self, node: ForStmt): var = self.locals[-1][node.identifier.name] - idx = self.builder.alloca(int32_t, None, 'idx') - self.builder.store(ir.Constant(int32_t, 0), idx) + idx_var = self.builder.alloca(int32_t, None, 'idx') + self.builder.store(ir.Constant(int32_t, 0), idx_var) iterable = self.visit(node.iterable) if node.iterable.inferredType == StrType(): self.whileHelper( lambda: self.builder.icmp_signed("<", - self.builder.load(idx), + self.builder.load(idx_var), self.builder.call(self.externs['strlen'], [iterable])), lambda: self.forBody(node, var, lambda currIdx: self.strIndex( iterable, currIdx), - idx)) + idx_var)) else: - raise Exception("unimplemented") + self.assert_nonnull(iterable, node.iterable.location[0]) + length = self.list_len(iterable) + self.whileHelper( + lambda: self.builder.icmp_signed("<", + self.builder.load(idx_var), + length), + lambda: self.forBody(node, + var, + lambda currIdx: self.listIndex( + iterable, currIdx, node.identifier.inferredType.getLLVMType()), + idx_var)) - def forBody(self, node: ForStmt, var, idxFn, idx): - currIdx = self.builder.load(idx) + def forBody(self, node: ForStmt, var, idxFn, idx_var): + currIdx = self.builder.load(idx_var) self.builder.store(idxFn(currIdx), var) self.visitStmtList(node.body) self.builder.store(self.builder.add( - currIdx, ir.Constant(int32_t, 1)), idx) + currIdx, ir.Constant(int32_t, 1)), idx_var) def ListExpr(self, node: ListExpr): - pass + n = len(node.elements) + if n == 0: + elemType = node.emptyListType.getLLVMType() + else: + elemType = node.inferredType.elementType.getLLVMType() + listType = ir.LiteralStructType([int32_t, ir.ArrayType(elemType, n)]) + alloca = self.builder.alloca(listType) + for i in range(n): + value = self.visit(node.elements[i]) + idx_ptr = self.builder.gep(alloca, [ + ir.Constant(int32_t, 0), + ir.Constant(int32_t, 1), + ir.Constant(int32_t, i)]) + self.builder.store(value, idx_ptr) + len_ptr = self.builder.gep( + alloca, [ir.Constant(int32_t, 0), ir.Constant(int32_t, 0)]) + self.builder.store(ir.Constant(int32_t, n), len_ptr) + alloca = self.builder.bitcast(alloca, voidptr_t) + return alloca def WhileStmt(self, node: WhileStmt): self.whileHelper( @@ -455,28 +503,42 @@ def IntegerLiteral(self, node: IntegerLiteral): return ir.Constant(int32_t, node.value) def NoneLiteral(self, node: NoneLiteral): - return ir.Constant(ir.PointerType(None), None) + return ir.Constant(voidptr_t, None) def StringLiteral(self, node: StringLiteral): const = self.make_bytearray((node.value + '\00').encode('ascii')) alloca = self.builder.alloca(ir.ArrayType( - int8_t, len(node.value) + 1), name=self.newLocal()) + int8_t, len(node.value) + 1)) self.builder.store(const, alloca) return self.builder.bitcast(alloca, voidptr_t) # BUILT-INS def emit_len(self, arg: Expr): + val = self.visit(arg) if arg.inferredType == StrType(): - val = self.builder.bitcast(self.visit(arg), voidptr_t) + val = self.builder.bitcast(val, voidptr_t) return self.builder.call(self.externs['strlen'], [val]) else: - raise Exception("unimplemented") + return self.list_len(val) + + def assert_nonnull(self, val, line): + val = self.builder.bitcast(val, voidptr_t) + self.ifHelper( + lambda: self.builder.icmp_signed( + '==', ir.Constant(voidptr_t, None), val), + lambda: self.longJmp(line) + ) + + def list_len(self, arg): + val = self.builder.bitcast(arg, int32_t.as_pointer()) + return self.builder.load(val) def emit_assert(self, arg: Expr): + arg = self.visit(arg) return self.ifHelper( lambda: self.builder.icmp_unsigned( - '==', ir.Constant(bool_t, 0), self.visit(arg)), + '==', ir.Constant(bool_t, 0), arg), lambda: self.longJmp(arg.location[0]) ) diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index cd09ba9..0a12af8 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -46,4 +46,4 @@ def getWasmName(self) -> str: return "i32" def getLLVMType(self) -> ir.Type: - raise Exception("unimplemented") + return ir.IntType(8).as_pointer() diff --git a/foobar.py b/foobar.py index 31dd363..68ac931 100644 --- a/foobar.py +++ b/foobar.py @@ -1,3 +1,16 @@ -x:str = "123" -y:str = "456" -print(len(x + y + x + y + x + y)) \ No newline at end of file +x:str = '' +y:[str] = None +a: int = 1 +b: [int] = None +c: bool = True +d: [bool] = None +d = [True and True, False and False, True and False, False and True, True or True, False or False, True or False, False or True] +y = ["213", "123" + "86asdkhjasbdj", "415"] +for x in y: + print("asd") + print(x) +b = [123 + 2, 123 + 123, 1213123123] +for a in b: + print(a) +for c in d: + print(c) \ No newline at end of file diff --git a/test.py b/test.py index a733bd0..7b6432b 100644 --- a/test.py +++ b/test.py @@ -629,7 +629,8 @@ def llvm_debug(test): astparser = compiler.parser chocopy_ast = compiler.parse(test) if len(astparser.errors) > 0: - return False + print(astparser.errors) + assert len(astparser.errors) == 0 compiler.typecheck(chocopy_ast) module = compiler.emitLLVM(chocopy_ast) print("Module output:") From 0fad0fecb06784c8af73fc8e58bcdab3e23347ae Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 22 May 2023 23:04:28 -0700 Subject: [PATCH 56/79] list assignment --- compiler/astnodes/listexpr.py | 1 + compiler/closuretransformer.py | 2 +- compiler/empty_list_typer.py | 4 +++- compiler/llvm_backend.py | 43 +++++++++++++++++----------------- foobar.py | 26 ++++++++------------ 5 files changed, 37 insertions(+), 39 deletions(-) diff --git a/compiler/astnodes/listexpr.py b/compiler/astnodes/listexpr.py index ee4b3a6..d2af568 100644 --- a/compiler/astnodes/listexpr.py +++ b/compiler/astnodes/listexpr.py @@ -7,6 +7,7 @@ class ListExpr(Expr): def __init__(self, location: List[int], elements: List[Expr]): super().__init__(location, "ListExpr") self.elements = elements + # this is populated by the EmptyListTyper pass self.emptyListType = None def preorder(self, visitor): diff --git a/compiler/closuretransformer.py b/compiler/closuretransformer.py index 98bc906..0174098 100644 --- a/compiler/closuretransformer.py +++ b/compiler/closuretransformer.py @@ -12,7 +12,7 @@ def typeToAnnotation(t: ValueType) -> SymbolType: class ClosureTransformer(TypeChecker): - # rewriting function signatures to include free vars as explicit arguments + # rewrite function signatures to include free vars as explicit arguments # rewrite function calls to include new args def __init__(self): diff --git a/compiler/empty_list_typer.py b/compiler/empty_list_typer.py index 59c4a4d..e6e04b2 100644 --- a/compiler/empty_list_typer.py +++ b/compiler/empty_list_typer.py @@ -3,7 +3,9 @@ from .visitor import Visitor from typing import List -# A visitor to refine the types of empty list literals +# A visitor to refine the types of empty list literals [] +# based on what they are being assigned to +# Prior to this pass, they have the special type class EmptyListTyper(Visitor): diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 6ca10d7..4048b12 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -134,8 +134,8 @@ def programHelper(self, node: Program): def VarDef(self, node: VarDef): val = self.visit(node.value) saved_block = self.builder.block - addr = self.create_entry_block_alloca( - node.getName(), node.var.t.getLLVMType()) + addr = self.builder.alloca( + node.var.t.getLLVMType(), None, node.getName()) self.builder.position_at_end(saved_block) self.builder.store(val, addr) self.locals[-1][node.getName()] = addr @@ -171,7 +171,12 @@ def AssignStmt(self, node: AssignStmt): if isinstance(var, MemberExpr): raise Exception("unimplemented") elif isinstance(var, IndexExpr): - raise Exception("unimplemented") + lst = self.visit(var.list) + idx = self.visit(var.index) + self.assert_nonnull(lst, var.list.location[0]) + ptr = self.listIndex( + lst, idx, var.inferredType.getLLVMType(), True, var.index.location[0]) + self.builder.store(val, ptr) elif isinstance(var, Identifier): addr = self.locals[-1][var.name] self.builder.store(val, addr) @@ -270,11 +275,13 @@ def IndexExpr(self, node: IndexExpr): lst = self.visit(node.list) idx = self.visit(node.index) self.assert_nonnull(lst, node.list.location[0]) - return self.listIndex(lst, idx, - node.inferredType.getLLVMType(), - True, node.index.location[0]) + ptr = self.listIndex(lst, idx, + node.inferredType.getLLVMType(), + True, node.index.location[0]) + return self.builder.load(ptr) - def listIndex(self, list, index, arrType, check_bounds=False, line: int = 0): + def listIndex(self, list, index, elemType, check_bounds=False, line: int = 0): + # return pointer to list[index] length = self.list_len(list) if check_bounds: self.ifHelper( @@ -288,16 +295,15 @@ def listIndex(self, list, index, arrType, check_bounds=False, line: int = 0): index), lambda: self.longJmp(line) ) - structType = ir.LiteralStructType([int32_t, arrType]) + structType = ir.LiteralStructType([int32_t, elemType]) list = self.builder.bitcast(list, structType.as_pointer()) # get the actual array and cast data = self.builder.gep(list, [ ir.Constant(int32_t, 0), ir.Constant(int32_t, 1)]) - data = self.builder.bitcast(data, arrType.as_pointer()) - # index the array - ptr = self.builder.gep(data, [index]) - return self.builder.load(ptr) + data = self.builder.bitcast(data, elemType.as_pointer()) + # return pointer to value in array + return self.builder.gep(data, [index]) def strIndex(self, string, index, check_bounds=False, line: int = 0): string = self.builder.bitcast(string, voidptr_t) @@ -377,8 +383,8 @@ def ForStmt(self, node: ForStmt): length), lambda: self.forBody(node, var, - lambda currIdx: self.listIndex( - iterable, currIdx, node.identifier.inferredType.getLLVMType()), + lambda currIdx: self.builder.load(self.listIndex( + iterable, currIdx, node.identifier.inferredType.getLLVMType())), idx_var)) def forBody(self, node: ForStmt, var, idxFn, idx_var): @@ -502,7 +508,7 @@ def BooleanLiteral(self, node: BooleanLiteral): def IntegerLiteral(self, node: IntegerLiteral): return ir.Constant(int32_t, node.value) - def NoneLiteral(self, node: NoneLiteral): + def NoneLiteral(self, _: NoneLiteral): return ir.Constant(voidptr_t, None) def StringLiteral(self, node: StringLiteral): @@ -550,7 +556,7 @@ def longJmp(self, line: int): def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: raise Exception("Only bool, int, or str may be printed") - if arg.inferredType.className == "bool": + if arg.inferredType == BoolType(): text = self.ifHelper( lambda: self.visit(arg), lambda: self.builder.bitcast(self.globals['true'], voidptr_t), @@ -573,11 +579,6 @@ def printf(self, format, arg): fmt_ptr = self.builder.bitcast(format, voidptr_t) return self.builder.call(self.externs['printf'], [fmt_ptr, arg]) - def create_entry_block_alloca(self, name, t): - builder = ir.IRBuilder() - builder.position_at_start(self.builder.function.entry_basic_block) - return builder.alloca(t, size=None, name=name) - def global_constant(self, name, t, value): module = self.module data = ir.GlobalVariable(module, t, name, 0) diff --git a/foobar.py b/foobar.py index 68ac931..c198cb7 100644 --- a/foobar.py +++ b/foobar.py @@ -1,16 +1,10 @@ -x:str = '' -y:[str] = None -a: int = 1 -b: [int] = None -c: bool = True -d: [bool] = None -d = [True and True, False and False, True and False, False and True, True or True, False or False, True or False, False or True] -y = ["213", "123" + "86asdkhjasbdj", "415"] -for x in y: - print("asd") - print(x) -b = [123 + 2, 123 + 123, 1213123123] -for a in b: - print(a) -for c in d: - print(c) \ No newline at end of file +x:[int] = None +y:int = 0 +x = [1, 2, 3] +for y in x: + print(y) +x[0] = 2 +x[1] = 2 +x[2] = 9 +for y in x: + print(y) \ No newline at end of file From 9a10598e34e67bde25bbc58c4300cd71802340f0 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Thu, 25 May 2023 23:05:45 -0700 Subject: [PATCH 57/79] make strings and arrays heap allocated, list concat --- compiler/llvm_backend.py | 142 +++++++++++++++++++++---------- compiler/types/classvaluetype.py | 8 +- foobar.py | 8 +- 3 files changed, 103 insertions(+), 55 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 4048b12..40c1976 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -3,7 +3,7 @@ from .typesystem import TypeSystem from .visitor import Visitor from collections import defaultdict -from typing import List, Union +from typing import List import llvmlite.ir as ir import llvmlite.binding as llvm @@ -11,7 +11,7 @@ JMP_BUF_BYTES = 100 bool_t = ir.IntType(1) # for booleans -int8_t = ir.IntType(8) # chars +int8_t = ir.IntType(8) # chars, or booleans in arrays int32_t = ir.IntType(32) # for ints voidptr_t = ir.IntType(8).as_pointer() jmp_buf_t = ir.ArrayType(ir.IntType(8), JMP_BUF_BYTES) @@ -92,6 +92,12 @@ def Program(self, node: Program): malloc_t = ir.FunctionType(voidptr_t, [int32_t]) self.externs['malloc'] = ir.Function(self.module, malloc_t, 'malloc') + strcmp_t = ir.FunctionType(int32_t, [voidptr_t, voidptr_t]) + self.externs['strcmp'] = ir.Function(self.module, strcmp_t, 'strcmp') + + memcpy_t = ir.FunctionType(voidptr_t, [voidptr_t, voidptr_t, int32_t]) + self.externs['memcpy'] = ir.Function(self.module, memcpy_t, 'memcpy') + funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") @@ -197,6 +203,11 @@ def ExprStmt(self, node: ExprStmt): def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: return leftType.isListType() and rightType.isListType() and operator == "+" + def getListDataPtr(self, lst, elemType): + lst = self.builder.bitcast(lst, int32_t.as_pointer()) + lst = self.builder.gep(lst, [ir.Constant(int32_t, 1)]) + return self.builder.bitcast(lst, elemType.as_pointer()) + def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType @@ -206,19 +217,46 @@ def BinaryExpr(self, node: BinaryExpr): # concatenation and addition if operator == "+": if self.isListConcat(operator, leftType, rightType): - raise Exception("unimplemented") + lhs = self.toVoidPtr(lhs) + rhs = self.toVoidPtr(rhs) + llen = self.list_len(lhs) + rlen = self.list_len(rhs) + total_len = self.builder.add(llen, rlen) + if node.inferredType == EmptyType(): + elemType = node.emptyListType.getLLVMType() + else: + elemType = node.inferredType.elementType.getLLVMType() + assert elemType is not None + size = self.builder.add(ir.Constant(int32_t, 4), self.builder.mul( + total_len, self.sizeof(elemType))) + new_arr = self.builder.call(self.externs['malloc'], [size]) + size_ptr = self.builder.bitcast(new_arr, int32_t.as_pointer()) + self.builder.store(total_len, size_ptr) + + data = self.getListDataPtr(new_arr, elemType) + lhs_data = self.getListDataPtr(lhs, elemType) + rhs_data = self.getListDataPtr(rhs, elemType) + lhs_bytes = self.builder.mul(llen, self.sizeof(elemType)) + + self.builder.call(self.externs['memcpy'], [ + self.toVoidPtr(data), self.toVoidPtr(lhs_data), lhs_bytes]) + + data_rhs_start = self.builder.gep(data, [llen]) + rhs_bytes = self.builder.mul(rlen, self.sizeof(elemType)) + + self.builder.call(self.externs['memcpy'], [ + self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) + return new_arr elif leftType == StrType(): - lhs = self.builder.bitcast(lhs, voidptr_t) - rhs = self.builder.bitcast(rhs, voidptr_t) + lhs = self.toVoidPtr(lhs) + rhs = self.toVoidPtr(rhs) llen = self.builder.call(self.externs['strlen'], [lhs]) rlen = self.builder.call(self.externs['strlen'], [rhs]) total_len = self.builder.add(self.builder.add( llen, rlen), ir.Constant(int32_t, 1)) - # this is a memory leak since we never free new_str = self.builder.call( self.externs['malloc'], [total_len]) - fmt = self.builder.bitcast( - self.globals['fmt_str_concat'], voidptr_t) + fmt = self.toVoidPtr(self.globals['fmt_str_concat']) self.builder.call(self.externs['sprintf'], [ new_str, fmt, lhs, rhs]) return new_str @@ -243,16 +281,18 @@ def BinaryExpr(self, node: BinaryExpr): if leftType == IntType(): return self.builder.icmp_signed(operator, lhs, rhs) elif leftType == StrType(): - raise Exception("Unimplemented") + cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) + return self.builder.icmp_signed("==", cmp, ir.Constant(int32_t, 0)) else: return self.builder.icmp_signed(operator, lhs, rhs) elif operator == "!=": if leftType == IntType(): return self.builder.icmp_signed(operator, lhs, rhs) elif leftType == StrType(): - raise Exception("Unimplemented") + cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) + return self.builder.icmp_signed("!=", cmp, ir.Constant(int32_t, 0)) else: - # pointer comparisons + # pointer comparisons - TODO fix this op return self.builder.icmp_unsigned(operator, lhs, rhs) elif operator == "is": # pointer comparisons @@ -295,18 +335,12 @@ def listIndex(self, list, index, elemType, check_bounds=False, line: int = 0): index), lambda: self.longJmp(line) ) - structType = ir.LiteralStructType([int32_t, elemType]) - list = self.builder.bitcast(list, structType.as_pointer()) - # get the actual array and cast - data = self.builder.gep(list, [ - ir.Constant(int32_t, 0), - ir.Constant(int32_t, 1)]) - data = self.builder.bitcast(data, elemType.as_pointer()) + data = self.getListDataPtr(list, elemType) # return pointer to value in array return self.builder.gep(data, [index]) def strIndex(self, string, index, check_bounds=False, line: int = 0): - string = self.builder.bitcast(string, voidptr_t) + string = self.toVoidPtr(string) # bounds checks if check_bounds: self.ifHelper( @@ -323,14 +357,14 @@ def strIndex(self, string, index, check_bounds=False, line: int = 0): ) ptr = self.builder.gep(string, [index]) char = self.builder.load(ptr) - alloca = self.builder.alloca(ir.ArrayType( - int8_t, 2)) - alloca = self.builder.bitcast(alloca, voidptr_t) - char_ptr = self.builder.gep(alloca, [ir.Constant(int32_t, 0)]) + addr = self.builder.call(self.externs['malloc'], [ + ir.Constant(int32_t, 2)]) + addr = self.toVoidPtr(addr) + char_ptr = self.builder.gep(addr, [ir.Constant(int32_t, 0)]) self.builder.store(char, char_ptr, 8) - t_ptr = self.builder.gep(alloca, [ir.Constant(int32_t, 1)]) + t_ptr = self.builder.gep(addr, [ir.Constant(int32_t, 1)]) self.builder.store(ir.Constant(int8_t, 0), t_ptr, 8) - return alloca + return addr def UnaryExpr(self, node: UnaryExpr): if node.operator == "-": @@ -400,20 +434,21 @@ def ListExpr(self, node: ListExpr): elemType = node.emptyListType.getLLVMType() else: elemType = node.inferredType.elementType.getLLVMType() - listType = ir.LiteralStructType([int32_t, ir.ArrayType(elemType, n)]) - alloca = self.builder.alloca(listType) + assert elemType is not None + size = self.builder.add(ir.Constant(int32_t, 4), self.builder.mul( + ir.Constant(int32_t, n), self.sizeof(elemType))) + addr = self.builder.call(self.externs['malloc'], [size]) + addr = self.builder.bitcast(addr, int32_t.as_pointer()) for i in range(n): value = self.visit(node.elements[i]) - idx_ptr = self.builder.gep(alloca, [ - ir.Constant(int32_t, 0), - ir.Constant(int32_t, 1), - ir.Constant(int32_t, i)]) + data = self.getListDataPtr(addr, elemType) + idx_ptr = self.builder.gep(data, [ir.Constant(int32_t, i)]) self.builder.store(value, idx_ptr) len_ptr = self.builder.gep( - alloca, [ir.Constant(int32_t, 0), ir.Constant(int32_t, 0)]) + addr, [ir.Constant(int32_t, 0)]) self.builder.store(ir.Constant(int32_t, n), len_ptr) - alloca = self.builder.bitcast(alloca, voidptr_t) - return alloca + addr = self.toVoidPtr(addr) + return addr def WhileStmt(self, node: WhileStmt): self.whileHelper( @@ -512,24 +547,26 @@ def NoneLiteral(self, _: NoneLiteral): return ir.Constant(voidptr_t, None) def StringLiteral(self, node: StringLiteral): - const = self.make_bytearray((node.value + '\00').encode('ascii')) - alloca = self.builder.alloca(ir.ArrayType( - int8_t, len(node.value) + 1)) - self.builder.store(const, alloca) - return self.builder.bitcast(alloca, voidptr_t) + bytes = bytearray((node.value + '\00').encode('ascii')) + size = ir.Constant(int32_t, 1 + len(node.value)) + addr = self.builder.call(self.externs['malloc'], [size]) + for i in range(len(bytes)): + idx_ptr = self.builder.gep(addr, [ir.Constant(int32_t, i)]) + self.builder.store(ir.Constant(int8_t, bytes[i]), idx_ptr) + return addr # BUILT-INS def emit_len(self, arg: Expr): val = self.visit(arg) if arg.inferredType == StrType(): - val = self.builder.bitcast(val, voidptr_t) + val = self.toVoidPtr(val) return self.builder.call(self.externs['strlen'], [val]) else: return self.list_len(val) def assert_nonnull(self, val, line): - val = self.builder.bitcast(val, voidptr_t) + val = self.toVoidPtr(val) self.ifHelper( lambda: self.builder.icmp_signed( '==', ir.Constant(voidptr_t, None), val), @@ -541,11 +578,12 @@ def list_len(self, arg): return self.builder.load(val) def emit_assert(self, arg: Expr): + line = arg.location[0] arg = self.visit(arg) return self.ifHelper( lambda: self.builder.icmp_unsigned( '==', ir.Constant(bool_t, 0), arg), - lambda: self.longJmp(arg.location[0]) + lambda: self.longJmp(line) ) def longJmp(self, line: int): @@ -559,8 +597,8 @@ def emit_print(self, arg: Expr): if arg.inferredType == BoolType(): text = self.ifHelper( lambda: self.visit(arg), - lambda: self.builder.bitcast(self.globals['true'], voidptr_t), - lambda: self.builder.bitcast(self.globals['false'], voidptr_t), + lambda: self.toVoidPtr(self.globals['true']), + lambda: self.toVoidPtr(self.globals['false']), voidptr_t) return self.printf(self.globals['fmt_s'], text) elif arg.inferredType.className == 'int': @@ -576,7 +614,7 @@ def make_bytearray(self, buf): return ir.Constant(ir.ArrayType(int8_t, n), b) def printf(self, format, arg): - fmt_ptr = self.builder.bitcast(format, voidptr_t) + fmt_ptr = self.toVoidPtr(format) return self.builder.call(self.externs['printf'], [fmt_ptr, arg]) def global_constant(self, name, t, value): @@ -586,3 +624,17 @@ def global_constant(self, name, t, value): data.global_constant = True data.initializer = value return data + + def sizeof(self, t): + if isinstance(t, ir.IntType): + width = t.width + # each item in array must be at least 1 byte + if width == 1: + width = 8 + return ir.Constant(int32_t, width // 8) + null = t(None) + offset = null.gep([int32_t(1)]) + return self.builder.ptrtoint(offset, int32_t) + + def toVoidPtr(self, ptr): + return self.builder.bitcast(ptr, voidptr_t) diff --git a/compiler/types/classvaluetype.py b/compiler/types/classvaluetype.py index 43450ce..5c9ae58 100644 --- a/compiler/types/classvaluetype.py +++ b/compiler/types/classvaluetype.py @@ -134,12 +134,12 @@ def getLLVMType(self) -> ir.Type: elif self.className == SpecialClass.STR: return ir.IntType(8).as_pointer() elif self.className == SpecialClass.OBJECT: - raise Exception("unsupported") + return ir.IntType(8).as_pointer() elif self.className == SpecialClass.NONE: - return ir.VoidType() + return ir.IntType(8).as_pointer() elif self.className == SpecialClass.EMPTY: - raise Exception("unsupported") + return ir.IntType(8).as_pointer() elif self.className == SpecialClass.INT: return ir.IntType(32) else: - raise Exception("unsupported") + return ir.IntType(8).as_pointer() diff --git a/foobar.py b/foobar.py index c198cb7..778b04f 100644 --- a/foobar.py +++ b/foobar.py @@ -1,10 +1,6 @@ x:[int] = None y:int = 0 -x = [1, 2, 3] +x = [1, 2, 3] + [4, 5, 6] + [7, 8] for y in x: print(y) -x[0] = 2 -x[1] = 2 -x[2] = 9 -for y in x: - print(y) \ No newline at end of file +print(len(x)) \ No newline at end of file From 6e07a2ed07dd6e758e3e896b037f0214d1e9e809 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 26 May 2023 00:50:15 -0700 Subject: [PATCH 58/79] known bug for array concat, add sensible default for empty lists --- compiler/llvm_backend.py | 36 ++++++++++++-------- foobar.py | 13 +++---- test.py | 3 ++ tests/runtime/null_and_empty_list_compare.py | 6 ++++ tests/runtime/short_circuit.py | 7 ++++ 5 files changed, 45 insertions(+), 20 deletions(-) create mode 100644 tests/runtime/null_and_empty_list_compare.py create mode 100644 tests/runtime/short_circuit.py diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 40c1976..a0aeaea 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -223,7 +223,7 @@ def BinaryExpr(self, node: BinaryExpr): rlen = self.list_len(rhs) total_len = self.builder.add(llen, rlen) if node.inferredType == EmptyType(): - elemType = node.emptyListType.getLLVMType() + elemType = int8_t else: elemType = node.inferredType.elementType.getLLVMType() assert elemType is not None @@ -233,19 +233,19 @@ def BinaryExpr(self, node: BinaryExpr): size_ptr = self.builder.bitcast(new_arr, int32_t.as_pointer()) self.builder.store(total_len, size_ptr) - data = self.getListDataPtr(new_arr, elemType) + data_lhs_start = self.getListDataPtr(new_arr, elemType) lhs_data = self.getListDataPtr(lhs, elemType) rhs_data = self.getListDataPtr(rhs, elemType) lhs_bytes = self.builder.mul(llen, self.sizeof(elemType)) self.builder.call(self.externs['memcpy'], [ - self.toVoidPtr(data), self.toVoidPtr(lhs_data), lhs_bytes]) + self.toVoidPtr(data_lhs_start), self.toVoidPtr(lhs_data), lhs_bytes]) - data_rhs_start = self.builder.gep(data, [llen]) - rhs_bytes = self.builder.mul(rlen, self.sizeof(elemType)) + # data_rhs_start = self.builder.gep(data_lhs_start, [llen]) + # rhs_bytes = self.builder.mul(rlen, self.sizeof(elemType)) - self.builder.call(self.externs['memcpy'], [ - self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) + # self.builder.call(self.externs['memcpy'], [ + # self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) return new_arr elif leftType == StrType(): lhs = self.toVoidPtr(lhs) @@ -284,6 +284,7 @@ def BinaryExpr(self, node: BinaryExpr): cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) return self.builder.icmp_signed("==", cmp, ir.Constant(int32_t, 0)) else: + # bool return self.builder.icmp_signed(operator, lhs, rhs) elif operator == "!=": if leftType == IntType(): @@ -292,11 +293,14 @@ def BinaryExpr(self, node: BinaryExpr): cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) return self.builder.icmp_signed("!=", cmp, ir.Constant(int32_t, 0)) else: - # pointer comparisons - TODO fix this op - return self.builder.icmp_unsigned(operator, lhs, rhs) + # bool + return self.builder.icmp_signed(operator, lhs, rhs) elif operator == "is": # pointer comparisons - return self.builder.icmp_unsigned("==", lhs, rhs) + return self.builder.icmp_unsigned("==", + self.builder.ptrtoint( + lhs, int32_t), + self.builder.ptrtoint(rhs, int32_t)) # logical operators elif operator == "and": return self.builder.and_(lhs, rhs) @@ -431,7 +435,11 @@ def forBody(self, node: ForStmt, var, idxFn, idx_var): def ListExpr(self, node: ListExpr): n = len(node.elements) if n == 0: - elemType = node.emptyListType.getLLVMType() + if node.emptyListType: + elemType = node.emptyListType.getLLVMType() + else: + # fallback to voidptr + elemType = int8_t else: elemType = node.inferredType.elementType.getLLVMType() assert elemType is not None @@ -626,11 +634,11 @@ def global_constant(self, name, t, value): return data def sizeof(self, t): - if isinstance(t, ir.IntType): + if isinstance(t, ir.IntType) and not t.is_pointer: width = t.width # each item in array must be at least 1 byte - if width == 1: - width = 8 + if width < 8: + return ir.Constant(int32_t, 1) return ir.Constant(int32_t, width // 8) null = t(None) offset = null.gep([int32_t(1)]) diff --git a/foobar.py b/foobar.py index 778b04f..03e44f8 100644 --- a/foobar.py +++ b/foobar.py @@ -1,6 +1,7 @@ -x:[int] = None -y:int = 0 -x = [1, 2, 3] + [4, 5, 6] + [7, 8] -for y in x: - print(y) -print(len(x)) \ No newline at end of file +z:[str] = None +y:str = "" +z = ["asd"] + ["asd"] +print(len(z)) +print(z[0]) +y = z[0] +print(len(y)) \ No newline at end of file diff --git a/test.py b/test.py index 7b6432b..c1d8475 100644 --- a/test.py +++ b/test.py @@ -632,6 +632,9 @@ def llvm_debug(test): print(astparser.errors) assert len(astparser.errors) == 0 compiler.typecheck(chocopy_ast) + if len(compiler.typechecker.errors) > 0: + print(compiler.typechecker.errors) + assert len(compiler.typechecker.errors) == 0 module = compiler.emitLLVM(chocopy_ast) print("Module output:") print(str(module)) diff --git a/tests/runtime/null_and_empty_list_compare.py b/tests/runtime/null_and_empty_list_compare.py new file mode 100644 index 0000000..c9ddf68 --- /dev/null +++ b/tests/runtime/null_and_empty_list_compare.py @@ -0,0 +1,6 @@ +w:[int] = None +assert w is None +assert not w is [] +w = [] +assert not w is None +assert not w is [] diff --git a/tests/runtime/short_circuit.py b/tests/runtime/short_circuit.py new file mode 100644 index 0000000..e958cbe --- /dev/null +++ b/tests/runtime/short_circuit.py @@ -0,0 +1,7 @@ +def foo() -> bool: + assert False + return True + + +print(True or foo()) +print(False and foo()) From 31a6ed9f0ee788747e20e051ba6628a39749a7e7 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Fri, 26 May 2023 01:07:23 -0700 Subject: [PATCH 59/79] fix sizeof --- compiler/llvm_backend.py | 35 +++++++++++++++++++---------------- foobar.py | 5 ++++- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index a0aeaea..80f2a9d 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -221,15 +221,16 @@ def BinaryExpr(self, node: BinaryExpr): rhs = self.toVoidPtr(rhs) llen = self.list_len(lhs) rlen = self.list_len(rhs) - total_len = self.builder.add(llen, rlen) + total_len = self.builder.add(llen, rlen, 'total_len') if node.inferredType == EmptyType(): elemType = int8_t else: elemType = node.inferredType.elementType.getLLVMType() assert elemType is not None size = self.builder.add(ir.Constant(int32_t, 4), self.builder.mul( - total_len, self.sizeof(elemType))) - new_arr = self.builder.call(self.externs['malloc'], [size]) + total_len, self.sizeof(elemType)), 'bytes') + new_arr = self.builder.call( + self.externs['malloc'], [size], 'new_list') size_ptr = self.builder.bitcast(new_arr, int32_t.as_pointer()) self.builder.store(total_len, size_ptr) @@ -241,11 +242,11 @@ def BinaryExpr(self, node: BinaryExpr): self.builder.call(self.externs['memcpy'], [ self.toVoidPtr(data_lhs_start), self.toVoidPtr(lhs_data), lhs_bytes]) - # data_rhs_start = self.builder.gep(data_lhs_start, [llen]) - # rhs_bytes = self.builder.mul(rlen, self.sizeof(elemType)) + data_rhs_start = self.builder.gep(data_lhs_start, [llen]) + rhs_bytes = self.builder.mul(rlen, self.sizeof(elemType)) - # self.builder.call(self.externs['memcpy'], [ - # self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) + self.builder.call(self.externs['memcpy'], [ + self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) return new_arr elif leftType == StrType(): lhs = self.toVoidPtr(lhs) @@ -255,7 +256,7 @@ def BinaryExpr(self, node: BinaryExpr): total_len = self.builder.add(self.builder.add( llen, rlen), ir.Constant(int32_t, 1)) new_str = self.builder.call( - self.externs['malloc'], [total_len]) + self.externs['malloc'], [total_len], 'new_str') fmt = self.toVoidPtr(self.globals['fmt_str_concat']) self.builder.call(self.externs['sprintf'], [ new_str, fmt, lhs, rhs]) @@ -362,7 +363,7 @@ def strIndex(self, string, index, check_bounds=False, line: int = 0): ptr = self.builder.gep(string, [index]) char = self.builder.load(ptr) addr = self.builder.call(self.externs['malloc'], [ - ir.Constant(int32_t, 2)]) + ir.Constant(int32_t, 2), 'char']) addr = self.toVoidPtr(addr) char_ptr = self.builder.gep(addr, [ir.Constant(int32_t, 0)]) self.builder.store(char, char_ptr, 8) @@ -445,7 +446,7 @@ def ListExpr(self, node: ListExpr): assert elemType is not None size = self.builder.add(ir.Constant(int32_t, 4), self.builder.mul( ir.Constant(int32_t, n), self.sizeof(elemType))) - addr = self.builder.call(self.externs['malloc'], [size]) + addr = self.builder.call(self.externs['malloc'], [size], 'list_literal') addr = self.builder.bitcast(addr, int32_t.as_pointer()) for i in range(n): value = self.visit(node.elements[i]) @@ -557,7 +558,7 @@ def NoneLiteral(self, _: NoneLiteral): def StringLiteral(self, node: StringLiteral): bytes = bytearray((node.value + '\00').encode('ascii')) size = ir.Constant(int32_t, 1 + len(node.value)) - addr = self.builder.call(self.externs['malloc'], [size]) + addr = self.builder.call(self.externs['malloc'], [size], 'str_literal') for i in range(len(bytes)): idx_ptr = self.builder.gep(addr, [ir.Constant(int32_t, i)]) self.builder.store(ir.Constant(int8_t, bytes[i]), idx_ptr) @@ -583,7 +584,7 @@ def assert_nonnull(self, val, line): def list_len(self, arg): val = self.builder.bitcast(arg, int32_t.as_pointer()) - return self.builder.load(val) + return self.builder.load(val, 'len') def emit_assert(self, arg: Expr): line = arg.location[0] @@ -634,15 +635,17 @@ def global_constant(self, name, t, value): return data def sizeof(self, t): - if isinstance(t, ir.IntType) and not t.is_pointer: + if not t.is_pointer: width = t.width # each item in array must be at least 1 byte if width < 8: return ir.Constant(int32_t, 1) return ir.Constant(int32_t, width // 8) - null = t(None) - offset = null.gep([int32_t(1)]) - return self.builder.ptrtoint(offset, int32_t) + else: + null = t.as_pointer()(None) + offset = null.gep([int32_t(1)]) + size = self.builder.ptrtoint(offset, int32_t, 'sizeof') + return size def toVoidPtr(self, ptr): return self.builder.bitcast(ptr, voidptr_t) diff --git a/foobar.py b/foobar.py index 03e44f8..0c7973e 100644 --- a/foobar.py +++ b/foobar.py @@ -3,5 +3,8 @@ z = ["asd"] + ["asd"] print(len(z)) print(z[0]) +print(z[1]) y = z[0] -print(len(y)) \ No newline at end of file +print(len(y)) +# z: [[str]] = None +# z = [["str"]] \ No newline at end of file From 68a481b6b81b3840eef0d3d057888e6c1e7c4463 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sat, 27 May 2023 23:17:19 -0700 Subject: [PATCH 60/79] format tests, add modulo test, set up test suite for LLVM --- compiler/astnodes/typedvar.py | 3 + compiler/llvm_backend.py | 118 +++++++++++-------- foobar.py | 10 -- test.py | 62 ++++++---- tests/runtime/assignment.py | 14 +-- tests/runtime/classes.py | 20 ++-- tests/runtime/contains.py | 10 +- tests/runtime/control_flow.py | 20 ++-- tests/runtime/control_flow_2.py | 4 +- tests/runtime/doubling_vector.py | 28 +++-- tests/runtime/exponent.py | 47 ++++---- tests/runtime/expr_stmt.py | 2 +- tests/runtime/functions.py | 32 +++-- tests/runtime/global_loop.py | 9 +- tests/runtime/globals.py | 8 +- tests/runtime/hello_world.py | 2 +- tests/runtime/incrementing_counter.py | 19 +-- tests/runtime/int_and_bool.py | 9 +- tests/runtime/int_and_bool_control_flow.py | 10 +- tests/runtime/int_and_bool_funcs.py | 17 ++- tests/runtime/lists.py | 34 +++--- tests/runtime/local_loop.py | 6 +- tests/runtime/modulo.py | 20 ++++ tests/runtime/nested_list.py | 14 +-- tests/runtime/nonlocal.py | 50 +++++--- tests/runtime/nonlocal_builtins.py | 5 +- tests/runtime/nonlocal_loop.py | 8 +- tests/runtime/null_and_empty_list_compare.py | 2 +- tests/runtime/operators.py | 19 ++- tests/runtime/ratio.py | 21 ++-- tests/runtime/simple_list.py | 4 +- tests/runtime/simple_string.py | 10 +- tests/runtime/strings.py | 6 +- tests/runtime/var_decl.py | 12 +- 34 files changed, 380 insertions(+), 275 deletions(-) create mode 100644 tests/runtime/modulo.py diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 407b826..6034ba1 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -13,6 +13,9 @@ def __init__(self, location: List[int], identifier: Identifier, typ: TypeAnnotat self.t = None # the typechecked type goes here self.varInstance = None + def name(self): + return self.identifier.name + def visit(self, visitor): return visitor.TypedVar(self) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 80f2a9d..1cf3f7e 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -68,7 +68,9 @@ def Program(self, node: Program): self.make_bytearray('Error on line %i\n\00'.encode('ascii'))), 'fmt_str_concat': self.global_constant('fmt_str_concat', ir.ArrayType(int8_t, 5), - self.make_bytearray('%s%s\00'.encode('ascii'))) + self.make_bytearray('%s%s\00'.encode('ascii'))), + 'jmp_buf': self.global_constant( + "jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * JMP_BUF_BYTES))) } printf_t = ir.FunctionType(int32_t, [voidptr_t], True) @@ -98,6 +100,12 @@ def Program(self, node: Program): memcpy_t = ir.FunctionType(voidptr_t, [voidptr_t, voidptr_t, int32_t]) self.externs['memcpy'] = ir.Function(self.module, memcpy_t, 'memcpy') + # begin main function + + funcDefs = [d for d in node.declarations if isinstance(d, FuncDef)] + for d in funcDefs: + self.visit(d) + funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") @@ -105,10 +113,9 @@ def Program(self, node: Program): entry_block = func.append_basic_block('entry') self.builder = ir.IRBuilder(entry_block) - jmp_buf = self.global_constant( - "jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * JMP_BUF_BYTES))) - status = self.builder.call(self.externs['setjmp'], [jmp_buf]) - cond = self.builder.icmp_signed("!=", ir.Constant(int32_t, 0), status) + status = self.builder.call(self.externs['setjmp'], [ + self.globals['jmp_buf']]) + cond = self.builder.icmp_signed("!=", int32_t(0), status) error_block = self.builder.append_basic_block('error_handling') program_block = self.builder.append_basic_block('program_code') @@ -150,22 +157,33 @@ def ClassDef(self, node: ClassDef): pass def FuncDef(self, node: FuncDef): - funcname = node.name + # TODO - methods + self.returnType = node.type.returnType + shouldReturnValue = not self.returnType.isNone() + funcname = node.name.name returnType = node.type.returnType.getLLVMType() argTypes = [p.getLLVMType() for p in node.type.parameters] funcType = ir.FunctionType(returnType, argTypes) func = ir.Function(self.module, funcType, funcname) + self.globals[funcname] = func self.enterScope() bb_entry = func.append_basic_block('entry') self.builder = ir.IRBuilder(bb_entry) for i, arg in enumerate(func.args): - arg.name = node.proto.argnames[i] + arg.name = node.params[i].name() alloca = self.builder.alloca( node.type.parameters[i].getLLVMType(), name=arg.name) self.builder.store(arg, alloca) self.locals[-1][arg.name] = alloca + for d in node.declarations: + self.visit(d) self.visitStmtList(node.statements) - # self.builder.ret(retval) + # implicitly return None if possible + if shouldReturnValue is not None and ( + len(node.statements) == 0 or + not isinstance(node.statements[-1], ReturnStmt) + ): + self.builder.ret(self.NoneLiteral(None)) self.exitScope() return func @@ -205,7 +223,7 @@ def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) def getListDataPtr(self, lst, elemType): lst = self.builder.bitcast(lst, int32_t.as_pointer()) - lst = self.builder.gep(lst, [ir.Constant(int32_t, 1)]) + lst = self.builder.gep(lst, [int32_t(1)]) return self.builder.bitcast(lst, elemType.as_pointer()) def BinaryExpr(self, node: BinaryExpr): @@ -227,7 +245,7 @@ def BinaryExpr(self, node: BinaryExpr): else: elemType = node.inferredType.elementType.getLLVMType() assert elemType is not None - size = self.builder.add(ir.Constant(int32_t, 4), self.builder.mul( + size = self.builder.add(int32_t(4), self.builder.mul( total_len, self.sizeof(elemType)), 'bytes') new_arr = self.builder.call( self.externs['malloc'], [size], 'new_list') @@ -254,7 +272,7 @@ def BinaryExpr(self, node: BinaryExpr): llen = self.builder.call(self.externs['strlen'], [lhs]) rlen = self.builder.call(self.externs['strlen'], [rhs]) total_len = self.builder.add(self.builder.add( - llen, rlen), ir.Constant(int32_t, 1)) + llen, rlen), int32_t(1)) new_str = self.builder.call( self.externs['malloc'], [total_len], 'new_str') fmt = self.toVoidPtr(self.globals['fmt_str_concat']) @@ -283,7 +301,7 @@ def BinaryExpr(self, node: BinaryExpr): return self.builder.icmp_signed(operator, lhs, rhs) elif leftType == StrType(): cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) - return self.builder.icmp_signed("==", cmp, ir.Constant(int32_t, 0)) + return self.builder.icmp_signed("==", cmp, int32_t(0)) else: # bool return self.builder.icmp_signed(operator, lhs, rhs) @@ -292,7 +310,7 @@ def BinaryExpr(self, node: BinaryExpr): return self.builder.icmp_signed(operator, lhs, rhs) elif leftType == StrType(): cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) - return self.builder.icmp_signed("!=", cmp, ir.Constant(int32_t, 0)) + return self.builder.icmp_signed("!=", cmp, int32_t(0)) else: # bool return self.builder.icmp_signed(operator, lhs, rhs) @@ -331,7 +349,7 @@ def listIndex(self, list, index, elemType, check_bounds=False, line: int = 0): if check_bounds: self.ifHelper( lambda: self.builder.icmp_signed( - '>', ir.Constant(int32_t, 0), index), + '>', int32_t(0), index), lambda: self.longJmp(line) ) self.ifHelper( @@ -350,7 +368,7 @@ def strIndex(self, string, index, check_bounds=False, line: int = 0): if check_bounds: self.ifHelper( lambda: self.builder.icmp_signed( - '>', ir.Constant(int32_t, 0), index), + '>', int32_t(0), index), lambda: self.longJmp(line) ) self.ifHelper( @@ -363,12 +381,12 @@ def strIndex(self, string, index, check_bounds=False, line: int = 0): ptr = self.builder.gep(string, [index]) char = self.builder.load(ptr) addr = self.builder.call(self.externs['malloc'], [ - ir.Constant(int32_t, 2), 'char']) + int32_t(2)], 'char') addr = self.toVoidPtr(addr) - char_ptr = self.builder.gep(addr, [ir.Constant(int32_t, 0)]) + char_ptr = self.builder.gep(addr, [int32_t(0)]) self.builder.store(char, char_ptr, 8) - t_ptr = self.builder.gep(addr, [ir.Constant(int32_t, 1)]) - self.builder.store(ir.Constant(int8_t, 0), t_ptr, 8) + t_ptr = self.builder.gep(addr, [int32_t(1)]) + self.builder.store(int8_t(0), t_ptr, 8) return addr def UnaryExpr(self, node: UnaryExpr): @@ -377,12 +395,11 @@ def UnaryExpr(self, node: UnaryExpr): return self.builder.neg(val) elif node.operator == "not": val = self.visit(node.operand) - return self.builder.icmp_unsigned('==', ir.Constant(bool_t, 0), val) + return self.builder.icmp_unsigned('==', bool_t(0), val) def CallExpr(self, node: CallExpr): if node.function.name == "print": - self.emit_print(node.args[0]) - return + return self.emit_print(node.args[0]) if node.function.name == "__assert__": self.emit_assert(node.args[0]) return @@ -400,7 +417,7 @@ def CallExpr(self, node: CallExpr): def ForStmt(self, node: ForStmt): var = self.locals[-1][node.identifier.name] idx_var = self.builder.alloca(int32_t, None, 'idx') - self.builder.store(ir.Constant(int32_t, 0), idx_var) + self.builder.store(int32_t(0), idx_var) iterable = self.visit(node.iterable) if node.iterable.inferredType == StrType(): @@ -431,7 +448,7 @@ def forBody(self, node: ForStmt, var, idxFn, idx_var): self.builder.store(idxFn(currIdx), var) self.visitStmtList(node.body) self.builder.store(self.builder.add( - currIdx, ir.Constant(int32_t, 1)), idx_var) + currIdx, int32_t(1)), idx_var) def ListExpr(self, node: ListExpr): n = len(node.elements) @@ -444,18 +461,19 @@ def ListExpr(self, node: ListExpr): else: elemType = node.inferredType.elementType.getLLVMType() assert elemType is not None - size = self.builder.add(ir.Constant(int32_t, 4), self.builder.mul( - ir.Constant(int32_t, n), self.sizeof(elemType))) - addr = self.builder.call(self.externs['malloc'], [size], 'list_literal') + size = self.builder.add(int32_t(4), self.builder.mul( + int32_t(n), self.sizeof(elemType))) + addr = self.builder.call(self.externs['malloc'], [ + size], 'list_literal') addr = self.builder.bitcast(addr, int32_t.as_pointer()) for i in range(n): value = self.visit(node.elements[i]) data = self.getListDataPtr(addr, elemType) - idx_ptr = self.builder.gep(data, [ir.Constant(int32_t, i)]) + idx_ptr = self.builder.gep(data, [int32_t(i)]) self.builder.store(value, idx_ptr) len_ptr = self.builder.gep( - addr, [ir.Constant(int32_t, 0)]) - self.builder.store(ir.Constant(int32_t, n), len_ptr) + addr, [int32_t(0)]) + self.builder.store(int32_t(n), len_ptr) addr = self.toVoidPtr(addr) return addr @@ -479,14 +497,15 @@ def whileHelper(self, condFn, bodyFn): self.builder.position_at_start(do_block) bodyFn() - self.builder.branch(while_block) + if not self.builder.block.is_terminated: + self.builder.branch(while_block) do_block = self.builder.block self.builder.position_at_start(end_block) def ReturnStmt(self, node: ReturnStmt): if self.returnType.isNone(): - self.builder.ret_void() + self.builder.ret(self.NoneLiteral(None)) else: val = None if node.value is None: @@ -524,13 +543,15 @@ def ifHelper(self, condFn, thenFn, elseFn=None, returnType=None): self.builder.position_at_start(then_block) then_val = thenFn() - self.builder.branch(merge_block) + if not self.builder.block.is_terminated: + self.builder.branch(merge_block) then_block = self.builder.block if elseFn is not None: self.builder.position_at_start(else_block) else_val = elseFn() - self.builder.branch(merge_block) + if not self.builder.block.is_terminated: + self.builder.branch(merge_block) else_block = self.builder.block self.builder.position_at_start(merge_block) @@ -547,21 +568,21 @@ def MethodCallExpr(self, node: MethodCallExpr): # LITERALS def BooleanLiteral(self, node: BooleanLiteral): - return ir.Constant(bool_t, 1 if node.value else 0) + return bool_t(1 if node.value else 0) def IntegerLiteral(self, node: IntegerLiteral): - return ir.Constant(int32_t, node.value) + return int32_t(node.value) def NoneLiteral(self, _: NoneLiteral): - return ir.Constant(voidptr_t, None) + return voidptr_t(None) def StringLiteral(self, node: StringLiteral): bytes = bytearray((node.value + '\00').encode('ascii')) - size = ir.Constant(int32_t, 1 + len(node.value)) + size = int32_t(1 + len(node.value)) addr = self.builder.call(self.externs['malloc'], [size], 'str_literal') for i in range(len(bytes)): - idx_ptr = self.builder.gep(addr, [ir.Constant(int32_t, i)]) - self.builder.store(ir.Constant(int8_t, bytes[i]), idx_ptr) + idx_ptr = self.builder.gep(addr, [int32_t(i)]) + self.builder.store(int8_t(bytes[i]), idx_ptr) return addr # BUILT-INS @@ -578,7 +599,7 @@ def assert_nonnull(self, val, line): val = self.toVoidPtr(val) self.ifHelper( lambda: self.builder.icmp_signed( - '==', ir.Constant(voidptr_t, None), val), + '==', voidptr_t(None), val), lambda: self.longJmp(line) ) @@ -591,14 +612,14 @@ def emit_assert(self, arg: Expr): arg = self.visit(arg) return self.ifHelper( lambda: self.builder.icmp_unsigned( - '==', ir.Constant(bool_t, 0), arg), + '==', bool_t(0), arg), lambda: self.longJmp(line) ) def longJmp(self, line: int): jmp_buf = self.module.get_global("jmp_buf") self.builder.call(self.externs['longjmp'], [ - jmp_buf, ir.Constant(int32_t, line)]) + jmp_buf, int32_t(line)]) def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: @@ -609,11 +630,12 @@ def emit_print(self, arg: Expr): lambda: self.toVoidPtr(self.globals['true']), lambda: self.toVoidPtr(self.globals['false']), voidptr_t) - return self.printf(self.globals['fmt_s'], text) + self.printf(self.globals['fmt_s'], text) elif arg.inferredType.className == 'int': - return self.printf(self.globals['fmt_i'], self.visit(arg)) + self.printf(self.globals['fmt_i'], self.visit(arg)) else: - return self.printf(self.globals['fmt_s'], self.visit(arg)) + self.printf(self.globals['fmt_s'], self.visit(arg)) + return self.NoneLiteral(None) # UTILS @@ -639,8 +661,8 @@ def sizeof(self, t): width = t.width # each item in array must be at least 1 byte if width < 8: - return ir.Constant(int32_t, 1) - return ir.Constant(int32_t, width // 8) + return int32_t(1) + return int32_t(width // 8) else: null = t.as_pointer()(None) offset = null.gep([int32_t(1)]) diff --git a/foobar.py b/foobar.py index 0c7973e..e69de29 100644 --- a/foobar.py +++ b/foobar.py @@ -1,10 +0,0 @@ -z:[str] = None -y:str = "" -z = ["asd"] + ["asd"] -print(len(z)) -print(z[0]) -print(z[1]) -y = z[0] -print(len(y)) -# z: [[str]] = None -# z = [["str"]] \ No newline at end of file diff --git a/test.py b/test.py index c1d8475..dab8fc1 100644 --- a/test.py +++ b/test.py @@ -23,7 +23,7 @@ def run_all_tests(): # run_jvm_tests() # run_cil_tests() # run_wasm_tests() - # run_llvm_tests() + run_llvm_tests() test_eval_llvm() @@ -167,23 +167,12 @@ def run_python_backend_tests(): n_passed, total)) -disabled_wasm_tests = [] - - def run_wasm_tests(): print("Running WASM backend tests...\n") total = 0 n_passed = 0 wasm_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() for test in wasm_tests_dir.glob('*.py'): - skip = False - for disabled in disabled_wasm_tests: - if disabled in str(test): - skip = True - break - if skip: - print("Skipping: " + str(test) + "\n") - continue passed = run_wasm_test(test) total += 1 if not passed: @@ -587,17 +576,48 @@ def ast_equals(d1, d2) -> bool: return d1 == d2 +disabled_llvm_tests = [ + "/incrementing_counter.", + "/binary_tree.", + "/classes.", + "/doubling_vector.", + "/globals.", + "/nonlocal_builtins.", + "/nonlocal_loop.", + "/nonlocal.", + "/ratio.", + "/operators.", + "/inherit_init.", + "/linked_list.", + "/exponent.", + "/lists.", + "/short_circuit.", + "/global_loop.", + "/control_flow.", + "modulo" +] + + def run_llvm_tests(): print("Running LLVM backend tests...\n") total = 0 n_passed = 0 llvm_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() for test in llvm_tests_dir.glob('*.py'): - passed = run_llvm_test(test) + skip = False + for disabled in disabled_llvm_tests: + if disabled in str(test): + skip = True + break + if skip: + # print("Skipping: " + str(test) + "\n") + continue + passed = run_llvm_test(test, False) total += 1 if not passed: print("Failed: " + str(test) + "\n") else: + print("Passed: " + str(test) + "\n") n_passed += 1 if total != n_passed: print("\nNot all test cases passed") @@ -606,6 +626,7 @@ def run_llvm_tests(): def eval_llvm(module): + # eval the compiled LLVMLite from Python target = llvm.Target.from_default_triple() target_machine = target.create_target_machine() llvmmod = llvm.parse_assembly(str(module)) @@ -615,15 +636,11 @@ def eval_llvm(module): fptr() -def run_llvm_test(test): - pass - - def test_eval_llvm(): - llvm_debug("foobar.py") + run_llvm_test("foobar.py", False) -def llvm_debug(test): +def run_llvm_test(test, debug): try: compiler = Compiler() astparser = compiler.parser @@ -636,10 +653,11 @@ def llvm_debug(test): print(compiler.typechecker.errors) assert len(compiler.typechecker.errors) == 0 module = compiler.emitLLVM(chocopy_ast) - print("Module output:") - print(str(module)) - print("Evaluation output:") + if debug: + print("Module output:") + print(str(module)) eval_llvm(module) + return True except Exception as e: print("Internal compiler error:", test) track = traceback.format_exc() diff --git a/tests/runtime/assignment.py b/tests/runtime/assignment.py index 9cbc059..9c44bec 100644 --- a/tests/runtime/assignment.py +++ b/tests/runtime/assignment.py @@ -1,10 +1,10 @@ -x:int = 1 -y:int = 2 -z:object = None -a:bool = True -b:bool = False -c:str = "" -d:str = "" +x: int = 1 +y: int = 2 +z: object = None +a: bool = True +b: bool = False +c: str = "" +d: str = "" b = a diff --git a/tests/runtime/classes.py b/tests/runtime/classes.py index ec01a9e..4cd8f65 100644 --- a/tests/runtime/classes.py +++ b/tests/runtime/classes.py @@ -1,5 +1,5 @@ class A: - y:int = 1 + y: int = 1 def __init__(self: A): pass @@ -8,24 +8,26 @@ def t(self: A): global x x = 1 + class B(A): - z:int = 0 + z: int = 0 def __init__(self: B): self.z = 5 self.y = 5 - + def t(self: B): global x x = 2 - - def setZ(self: B, z:int): + + def setZ(self: B, z: int): self.z = z -x:int = 0 -c1:A = None -c2:B = None -c3:A = None + +x: int = 0 +c1: A = None +c2: B = None +c3: A = None # constructors, getters, setters c1 = A() diff --git a/tests/runtime/contains.py b/tests/runtime/contains.py index 8cbe4eb..e2f33ea 100644 --- a/tests/runtime/contains.py +++ b/tests/runtime/contains.py @@ -1,19 +1,21 @@ # Search in a list -def contains(items:[int], x:int) -> bool: - i:int = 0 +def contains(items: [int], x: int) -> bool: + i: int = 0 while i < len(items): if items[i] == x: return True i = i + 1 return False -def contains2(items:[int], x:int) -> bool: - i:int = 0 + +def contains2(items: [int], x: int) -> bool: + i: int = 0 for i in items: if i == x: return True return False + assert contains([4, 8, 15, 16, 23], 15) assert contains([4, 8, 15, 16, 23], 4) assert contains([4, 8, 15, 16, 23], 8) diff --git a/tests/runtime/control_flow.py b/tests/runtime/control_flow.py index 94e15e2..0f138d5 100644 --- a/tests/runtime/control_flow.py +++ b/tests/runtime/control_flow.py @@ -1,13 +1,13 @@ -x:[int] = None -y:str = "123" -z:str = "" -char:str = "" -a:bool = True -b:int = 0 -c:int = 100 -d:[bool] = None -e:object = None -f:[object] = None +x: [int] = None +y: str = "123" +z: str = "" +char: str = "" +a: bool = True +b: int = 0 +c: int = 100 +d: [bool] = None +e: object = None +f: [object] = None x = [] diff --git a/tests/runtime/control_flow_2.py b/tests/runtime/control_flow_2.py index 63e30f4..9dc3163 100644 --- a/tests/runtime/control_flow_2.py +++ b/tests/runtime/control_flow_2.py @@ -1,4 +1,4 @@ -b:int = 0 +b: int = 0 b = 0 @@ -69,4 +69,4 @@ b = 5 while b > 0: b = b - 1 -assert b == 0 \ No newline at end of file +assert b == 0 diff --git a/tests/runtime/doubling_vector.py b/tests/runtime/doubling_vector.py index 12342bb..47b316e 100644 --- a/tests/runtime/doubling_vector.py +++ b/tests/runtime/doubling_vector.py @@ -5,20 +5,20 @@ class Vector(object): size: int = 0 # Constructor - def __init__(self:"Vector"): + def __init__(self: "Vector"): self.items = [0] # Returns current capacity - def capacity(self:"Vector") -> int: + def capacity(self: "Vector") -> int: return len(self.items) # Increases capacity of vector by one element - def increase_capacity(self:"Vector") -> int: + def increase_capacity(self: "Vector") -> int: self.items = self.items + [0] return self.capacity() # Appends one item to end of vector - def append(self:"Vector", item: int): + def append(self: "Vector", item: int): if self.size == self.capacity(): self.increase_capacity() @@ -26,11 +26,13 @@ def append(self:"Vector", item: int): self.size = self.size + 1 # A faster (but more memory-consuming) implementation of vector + + class DoublingVector(Vector): - doubling_limit:int = 16 + doubling_limit: int = 16 # Overriding to do fewer resizes - def increase_capacity(self:"DoublingVector") -> int: + def increase_capacity(self: "DoublingVector") -> int: if (self.capacity() <= self.doubling_limit // 2): self.items = self.items + self.items else: @@ -39,18 +41,20 @@ def increase_capacity(self:"DoublingVector") -> int: self.items = self.items + [0] return self.capacity() -def vrange(i:int, j:int) -> Vector: - v:Vector = None + +def vrange(i: int, j: int) -> Vector: + v: Vector = None v = DoublingVector() - + while i < j: v.append(i) i = i + 1 return v - -vec:Vector = None -num:int = 0 + + +vec: Vector = None +num: int = 0 # Create a vector and populate it with The Numbers vec = DoublingVector() diff --git a/tests/runtime/exponent.py b/tests/runtime/exponent.py index a317a86..6286489 100644 --- a/tests/runtime/exponent.py +++ b/tests/runtime/exponent.py @@ -1,32 +1,35 @@ # Compute x**y def exp(x: int, y: int) -> int: - a: int = 0 - def f(i: int) -> int: - nonlocal a - def geta() -> int: - return a - if i <= 0: - return geta() - else: - a = a * x - return f(i-1) - a = 1 - return f(y) + a: int = 0 + + def f(i: int) -> int: + nonlocal a + + def geta() -> int: + return a + if i <= 0: + return geta() + else: + a = a * x + return f(i-1) + a = 1 + return f(y) + # Input parameter -n:int = 42 +n: int = 42 # Run [0, n] -i:int = 0 +i: int = 0 # Crunch while i <= n: - print(exp(2, i % 31)) - i = i + 1 + print(exp(2, i % 31)) + i = i + 1 -assert exp(2,3) == 8 -assert exp(3,3) == 27 -assert exp(3,4) == 81 -assert exp(4,4) == 256 -assert exp(5,1) == 5 -assert exp(1,99) == 1 +assert exp(2, 3) == 8 +assert exp(3, 3) == 27 +assert exp(3, 4) == 81 +assert exp(4, 4) == 256 +assert exp(5, 1) == 5 +assert exp(1, 99) == 1 diff --git a/tests/runtime/expr_stmt.py b/tests/runtime/expr_stmt.py index 60094c0..99a704f 100644 --- a/tests/runtime/expr_stmt.py +++ b/tests/runtime/expr_stmt.py @@ -2,4 +2,4 @@ 2 None "123" -False \ No newline at end of file +False diff --git a/tests/runtime/functions.py b/tests/runtime/functions.py index 4d063b5..c625b91 100644 --- a/tests/runtime/functions.py +++ b/tests/runtime/functions.py @@ -1,33 +1,41 @@ def f1(): - x:int = 1 + x: int = 1 -def f2()->int: + +def f2() -> int: return 1 -def f3()->object: + +def f3() -> object: return None -def f4(x:int, y:object)->int: + +def f4(x: int, y: object) -> int: return x + 1 + def f5(): return None -def f6()->int: + +def f6() -> int: return f4(5, None) -def f7()->int: - x:int = 0 - y:int = 0 + +def f7() -> int: + x: int = 0 + y: int = 0 x = f4(10, None) y = f6() return x - y -def f8(x:int)->int: + +def f8(x: int) -> int: return x -x:int = 0 -y:object = None + +x: int = 0 +y: object = None f1() assert f2() == 1 @@ -45,4 +53,4 @@ def f8(x:int)->int: assert f8(f7()) == 5 print(1) -print(True) \ No newline at end of file +print(True) diff --git a/tests/runtime/global_loop.py b/tests/runtime/global_loop.py index a6eb769..7174360 100644 --- a/tests/runtime/global_loop.py +++ b/tests/runtime/global_loop.py @@ -1,7 +1,9 @@ -x:int = 1 +x: int = 1 + def test(): - y:[int] = None + y: [int] = None + def inner(): global x for x in y: @@ -10,4 +12,5 @@ def inner(): inner() assert x == 3 -test() \ No newline at end of file + +test() diff --git a/tests/runtime/globals.py b/tests/runtime/globals.py index 85ff920..f84e9cb 100644 --- a/tests/runtime/globals.py +++ b/tests/runtime/globals.py @@ -1,6 +1,7 @@ -x:int = 0 -y:str = "a" -z:int = 0 +x: int = 0 +y: str = "a" +z: int = 0 + def t(): global x @@ -9,6 +10,7 @@ def t(): y = y + y assert z == 0 + assert x == 0 assert y == "a" t() diff --git a/tests/runtime/hello_world.py b/tests/runtime/hello_world.py index 7b6a0e7..e94acca 100644 --- a/tests/runtime/hello_world.py +++ b/tests/runtime/hello_world.py @@ -1 +1 @@ -print("hello, world!") \ No newline at end of file +print("hello, world!") diff --git a/tests/runtime/incrementing_counter.py b/tests/runtime/incrementing_counter.py index 07e8dcf..f18966f 100644 --- a/tests/runtime/incrementing_counter.py +++ b/tests/runtime/incrementing_counter.py @@ -1,12 +1,15 @@ class Counter(object): - n : int = 0 - def __init__(self : Counter): - pass - def inc(self : Counter): - self.n = self.n + 1 + n: int = 0 -c : Counter = None -i : int = 0 + def __init__(self: Counter): + pass + + def inc(self: Counter): + self.n = self.n + 1 + + +c: Counter = None +i: int = 0 c = Counter() c.inc() assert c.n == 1 @@ -16,6 +19,6 @@ def inc(self : Counter): c.inc() assert c.n == 4 -for i in [9,9,9,9,9,9]: +for i in [9, 9, 9, 9, 9, 9]: c.inc() assert c.n == 10 diff --git a/tests/runtime/int_and_bool.py b/tests/runtime/int_and_bool.py index 14166a2..ac28efc 100644 --- a/tests/runtime/int_and_bool.py +++ b/tests/runtime/int_and_bool.py @@ -1,7 +1,7 @@ -x:int = 1 -y:int = 2 -a:bool = True -b:bool = False +x: int = 1 +y: int = 2 +a: bool = True +b: bool = False print(x) print(y) @@ -35,4 +35,3 @@ x = y = 3 assert x == y assert x == 3 - diff --git a/tests/runtime/int_and_bool_control_flow.py b/tests/runtime/int_and_bool_control_flow.py index 387cc3a..9151aca 100644 --- a/tests/runtime/int_and_bool_control_flow.py +++ b/tests/runtime/int_and_bool_control_flow.py @@ -1,7 +1,7 @@ -x:int = 1 -y:int = 2 -a:bool = True -b:bool = False +x: int = 1 +y: int = 2 +a: bool = True +b: bool = False if a: assert True @@ -30,4 +30,4 @@ assert False assert (5 if a else 0) == 5 -assert (0 if b else 5) == 5 \ No newline at end of file +assert (0 if b else 5) == 5 diff --git a/tests/runtime/int_and_bool_funcs.py b/tests/runtime/int_and_bool_funcs.py index 4158540..7322e9b 100644 --- a/tests/runtime/int_and_bool_funcs.py +++ b/tests/runtime/int_and_bool_funcs.py @@ -1,20 +1,25 @@ -x:int = 1 -y:int = 2 -a:bool = True -b:bool = False +x: int = 1 +y: int = 2 +a: bool = True +b: bool = False -def test1(a1:int, a2:int)->int: + +def test1(a1: int, a2: int) -> int: return a1 + a2 -def test2(a1:bool)->bool: + +def test2(a1: bool) -> bool: return not a1 + def test3(): return None + def test4(): return + test3() test4() diff --git a/tests/runtime/lists.py b/tests/runtime/lists.py index 3313131..cb05561 100644 --- a/tests/runtime/lists.py +++ b/tests/runtime/lists.py @@ -1,21 +1,25 @@ -w:[object] = None -x:[int] = None -x2:[int] = None -y:[str] = None -y2:[str] = None -z:object = None -a:[[object]] = None -b:[[int]] = None - -def setIdx(lst:[int], idx:int, value:int): +w: [object] = None +x: [int] = None +x2: [int] = None +y: [str] = None +y2: [str] = None +z: object = None +a: [[object]] = None +b: [[int]] = None + + +def setIdx(lst: [int], idx: int, value: int): lst[idx] = value -def setNestedIdx(lst:[[int]], idx1:int, idx2:int, value:int): + +def setNestedIdx(lst: [[int]], idx1: int, idx2: int, value: int): lst[idx1][idx2] = value -def getNestedIdx(lst:[[int]], idx:int)->[int]: + +def getNestedIdx(lst: [[int]], idx: int) -> [int]: return lst[idx] + w = [] x = [] x2 = [] @@ -48,7 +52,7 @@ def getNestedIdx(lst:[[int]], idx:int)->[int]: assert x[3] == 3 x = [1, 2, 3] -x = x + [4] +x = x + [4] assert len(x) == 4 assert x[0] == 1 assert x[1] == 2 @@ -159,7 +163,7 @@ def getNestedIdx(lst:[[int]], idx:int)->[int]: assert len(y) == 1 assert len(y[0]) == 0 -x = [1,2,3] +x = [1, 2, 3] setIdx(x, 1, 0) assert x[1] == 0 @@ -181,5 +185,3 @@ def getNestedIdx(lst:[[int]], idx:int)->[int]: x = [1] y = ["1"] w = x + y - - diff --git a/tests/runtime/local_loop.py b/tests/runtime/local_loop.py index b4b653c..ac73f0a 100644 --- a/tests/runtime/local_loop.py +++ b/tests/runtime/local_loop.py @@ -1,7 +1,7 @@ -x:str = "" -y:str = "123" +x: str = "" +y: str = "123" for x in y: pass -assert x == "3" \ No newline at end of file +assert x == "3" diff --git a/tests/runtime/modulo.py b/tests/runtime/modulo.py new file mode 100644 index 0000000..886fad1 --- /dev/null +++ b/tests/runtime/modulo.py @@ -0,0 +1,20 @@ +# TODO: fix modulo operator behavior +assert -5 % 2 == 1 +assert 5 % -2 == -1 +assert -5 % -2 == -1 + +assert -5 % 3 == 1 +assert 5 % -3 == -1 +assert -5 % -3 == -2 + +assert -5 % 1 == 0 +assert -5 % -1 == 0 +assert 5 % -1 == 0 + +assert -5 % 4 == 3 +assert -5 % -4 == -1 +assert 5 % -4 == -3 + +assert -5 % 5 == 0 +assert -5 % -5 == 0 +assert 5 % -5 == 0 \ No newline at end of file diff --git a/tests/runtime/nested_list.py b/tests/runtime/nested_list.py index e521995..b1fd972 100644 --- a/tests/runtime/nested_list.py +++ b/tests/runtime/nested_list.py @@ -1,7 +1,7 @@ -a:[[int]] = None -b:[int] = None -c:int = 0 -d:int = 0 +a: [[int]] = None +b: [int] = None +c: int = 0 +d: int = 0 # TODO - check if these are legal # a = [] @@ -14,7 +14,7 @@ assert len(a) == 1 assert len(a[0]) == 1 -a = [[1],[2, 2, 2],[3,3],[]] +a = [[1], [2, 2, 2], [3, 3], []] assert len(a) == 4 assert len(a[0]) == 1 assert len(a[1]) == 3 @@ -36,13 +36,13 @@ assert len(a[0]) == 3 -a = [[1],[1,1,1],[1,1],[]] +a = [[1], [1, 1, 1], [1, 1], []] c = 0 for b in a: c = c + len(b) assert c == 6 -a = [[1],[2,3,4],[5,0],[]] +a = [[1], [2, 3, 4], [5, 0], []] c = 0 for b in a: for d in b: diff --git a/tests/runtime/nonlocal.py b/tests/runtime/nonlocal.py index 9b52279..3543e32 100644 --- a/tests/runtime/nonlocal.py +++ b/tests/runtime/nonlocal.py @@ -1,18 +1,23 @@ -a:int = 0 +a: int = 0 -def test(x:int)->int: + +def test(x: int) -> int: def test2(): nonlocal x x = 2 test2() return x -def test3()->int: - x:int = 4 + +def test3() -> int: + x: int = 4 + def test4(): nonlocal x + def test5(): assert x == 4 + def test6(): nonlocal x x = 3 @@ -24,27 +29,34 @@ def test6(): test4() return x + def test7(): - x:[int] = None + x: [int] = None + def test8(): x[0] = 0 x = [1, 2, 3] test8() assert x[0] == 0 -def test9(x:int): + +def test9(x: int): def test9helper(): nonlocal x x = 0 test9helper() + def test10(): - y:int = 1 - def test11(m:int)->int: + y: int = 1 + + def test11(m: int) -> int: return m + y - def test12()->int: + + def test12() -> int: nonlocal y - def test13(m:int)->int: + + def test13(m: int) -> int: return m + y y = test13(y) assert y == 2 @@ -53,12 +65,14 @@ def test13(m:int)->int: assert test12() == 4 assert y == 2 + class Nonlocals: - def testMethod3(self:"Nonlocals"): + def testMethod3(self: "Nonlocals"): pass - def testMethod(self:"Nonlocals", x:int): - y:int = 2 + def testMethod(self: "Nonlocals", x: int): + y: int = 2 + def testMethod2(): nonlocal x nonlocal y @@ -68,19 +82,21 @@ def testMethod2(): testMethod2() assert y == 3 - def testMethod4(self:"Nonlocals"): + def testMethod4(self: "Nonlocals"): test13(self) assert not (self is None) -def test13(x:"Nonlocals"): + +def test13(x: "Nonlocals"): def test14(): nonlocal x x = None test14() -b:Nonlocals = None -# nonlocals can be mutated +b: Nonlocals = None + +# nonlocals can be mutated assert test(1) == 2 assert test3() == 3 diff --git a/tests/runtime/nonlocal_builtins.py b/tests/runtime/nonlocal_builtins.py index a37d11f..e821860 100644 --- a/tests/runtime/nonlocal_builtins.py +++ b/tests/runtime/nonlocal_builtins.py @@ -3,10 +3,13 @@ def f(): x: bool = True y: str = "a" + def g(): nonlocal x nonlocal y print(y) assert x g() -f() \ No newline at end of file + + +f() diff --git a/tests/runtime/nonlocal_loop.py b/tests/runtime/nonlocal_loop.py index 7a19205..7583504 100644 --- a/tests/runtime/nonlocal_loop.py +++ b/tests/runtime/nonlocal_loop.py @@ -1,6 +1,7 @@ def test(): - x:int = 1 - y:[int] = None + x: int = 1 + y: [int] = None + def inner(): nonlocal x for x in y: @@ -9,4 +10,5 @@ def inner(): inner() assert x == 3 -test() \ No newline at end of file + +test() diff --git a/tests/runtime/null_and_empty_list_compare.py b/tests/runtime/null_and_empty_list_compare.py index c9ddf68..c9dc8b8 100644 --- a/tests/runtime/null_and_empty_list_compare.py +++ b/tests/runtime/null_and_empty_list_compare.py @@ -1,4 +1,4 @@ -w:[int] = None +w: [int] = None assert w is None assert not w is [] w = [] diff --git a/tests/runtime/operators.py b/tests/runtime/operators.py index 500fd91..7793491 100644 --- a/tests/runtime/operators.py +++ b/tests/runtime/operators.py @@ -1,10 +1,10 @@ -w:int = 1 -x:int = 1 -y:int = 2 -z:object = None -a:str = "123" -b:str = "123" -c:str = "456" +w: int = 1 +x: int = 1 +y: int = 2 +z: object = None +a: str = "123" +b: str = "123" +c: str = "456" assert w == x assert y != x assert b == b @@ -21,10 +21,6 @@ assert w * x == x assert 5 // 2 == y assert 5 % 2 == x -# TODO: fix modulo operator behavior -# assert -5 % 2 == 1 -# assert 5 % -2 == -1 -# assert -5 % -2 == -1 assert not False assert not (w != x) assert -x == -1 @@ -41,4 +37,3 @@ assert z is None z = object() assert z is z - diff --git a/tests/runtime/ratio.py b/tests/runtime/ratio.py index 987c1ef..4f3285f 100644 --- a/tests/runtime/ratio.py +++ b/tests/runtime/ratio.py @@ -1,17 +1,22 @@ class Rat(object): - n : int = 0 - d : int = 0 - def __init__(self : Rat): + n: int = 0 + d: int = 0 + + def __init__(self: Rat): pass - def new(self : Rat, n : int, d : int) -> Rat: + + def new(self: Rat, n: int, d: int) -> Rat: self.n = n self.d = d return self - def mul(self : Rat, other : Rat) -> Rat: + + def mul(self: Rat, other: Rat) -> Rat: return Rat().new(self.n * other.n, self.d * other.d) -r1 : Rat = None -r2 : Rat = None -r3 : Rat = None + + +r1: Rat = None +r2: Rat = None +r3: Rat = None r1 = Rat().new(4, 5) r2 = Rat().new(2, 3) assert r1.n == 4 diff --git a/tests/runtime/simple_list.py b/tests/runtime/simple_list.py index 6dbc176..44b45e9 100644 --- a/tests/runtime/simple_list.py +++ b/tests/runtime/simple_list.py @@ -1,5 +1,5 @@ -w:[int] = None -x:int = 0 +w: [int] = None +x: int = 0 w = [] assert len(w) == 0 diff --git a/tests/runtime/simple_string.py b/tests/runtime/simple_string.py index 1306bc1..5b292fe 100644 --- a/tests/runtime/simple_string.py +++ b/tests/runtime/simple_string.py @@ -1,7 +1,7 @@ -x:str = "123" -y:str = "" -z:str = "12345" -char:str = "c" +x: str = "123" +y: str = "" +z: str = "12345" +char: str = "c" print(x) print(y) print(z) @@ -37,4 +37,4 @@ assert "123" == "123" assert x == x assert "" == "" -assert x != "" \ No newline at end of file +assert x != "" diff --git a/tests/runtime/strings.py b/tests/runtime/strings.py index c6eb073..59715f8 100644 --- a/tests/runtime/strings.py +++ b/tests/runtime/strings.py @@ -1,5 +1,5 @@ -x:str = "123" -y:str = "123" +x: str = "123" +y: str = "123" assert len(x) == 3 assert x == y @@ -63,5 +63,3 @@ x = "123123" assert x[0] + x[1] + x[2] == x[3] + x[4] + x[5] - - diff --git a/tests/runtime/var_decl.py b/tests/runtime/var_decl.py index b09f53f..675905d 100644 --- a/tests/runtime/var_decl.py +++ b/tests/runtime/var_decl.py @@ -1,9 +1,9 @@ -w:object = None -x:str = "mystring" -y:int = 1 -z:bool = True +w: object = None +x: str = "mystring" +y: int = 1 +z: bool = True # potentially colliding names -i8:int = 1 +i8: int = 1 i32: int = 1 i64: int = 1 int32: int = 1 @@ -13,4 +13,4 @@ null: str = "" print(x) print(y) -print(z) \ No newline at end of file +print(z) From 7d25d912afede31652757ea828eeb0d394c1e095 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 28 May 2023 16:14:15 -0700 Subject: [PATCH 61/79] minimal class support, implement globals --- compiler/llvm_backend.py | 187 +++++++++++++++++++++++++++------------ foobar.py | 16 ++++ test.py | 11 +-- tests/runtime/modulo.py | 2 +- 4 files changed, 152 insertions(+), 64 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 1cf3f7e..453bb4c 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -4,11 +4,12 @@ from .visitor import Visitor from collections import defaultdict from typing import List +import json import llvmlite.ir as ir import llvmlite.binding as llvm -JMP_BUF_BYTES = 100 +JMP_BUF_BYTES = 200 bool_t = ir.IntType(1) # for booleans int8_t = ir.IntType(8) # chars, or booleans in arrays @@ -22,6 +23,7 @@ class LlvmBackend(Visitor): counter = 0 globals = {} externs = {} + constructors = {} def __init__(self, ts: TypeSystem): llvm.initialize() @@ -29,6 +31,30 @@ def __init__(self, ts: TypeSystem): llvm.initialize_native_asmprinter() self.module = ir.Module() self.builder = None + # (class name, method name) -> (idx in vtable, defining class name) + self.methodOffsets = dict() + self.ts = ts + + def initializeOffsets(self): + tblOffset = 0 + methodTableOffsets = dict() + # assign positions in the global method table + classes = [c for c in self.ts.classes if c != + "" and c != ""] + for cls in classes: + ctorType = ir.FunctionType(ir.VoidType(), [voidptr_t]) + ctor = ir.Function(self.module, ctorType, cls) + self.constructors[cls] = ctor + for methName, _, defCls in self.ts.getOrderedMethods(cls): + if cls == defCls: + methodTableOffsets[(cls, methName)] = tblOffset + tblOffset += 1 + # calculate info for each class + for cls in classes: + methods = self.ts.getOrderedMethods(cls) + for idx, methInfo in enumerate(methods): + name, _, defCls = methInfo + self.methodOffsets[(cls, name)] = (idx, defCls) def enterScope(self): self.locals.append(defaultdict(lambda: None)) @@ -46,32 +72,30 @@ def visitStmtList(self, stmts: List[Stmt]): # TOP LEVEL & DECLARATIONS def Program(self, node: Program): - # globals for commonly used strings - self.globals = { - 'true': self.global_constant('true', - ir.ArrayType(int8_t, 5), - self.make_bytearray('True\00'.encode('ascii'))), - 'false': self.global_constant('false', - ir.ArrayType(int8_t, 6), - self.make_bytearray('False\00'.encode('ascii'))), - 'fmt_i': self.global_constant('fmt_i', - ir.ArrayType(int8_t, 4), - self.make_bytearray('%i\n\00'.encode('ascii'))), - 'fmt_s': self.global_constant('fmt_s', - ir.ArrayType(int8_t, 4), - self.make_bytearray('%s\n\00'.encode('ascii'))), - 'fmt_assert': self.global_constant('fmt_assert', - ir.ArrayType(int8_t, 29), - self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))), - 'fmt_err': self.global_constant('fmt_err', - ir.ArrayType(int8_t, 18), - self.make_bytearray('Error on line %i\n\00'.encode('ascii'))), - 'fmt_str_concat': self.global_constant('fmt_str_concat', - ir.ArrayType(int8_t, 5), - self.make_bytearray('%s%s\00'.encode('ascii'))), - 'jmp_buf': self.global_constant( - "jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * JMP_BUF_BYTES))) - } + self.initializeOffsets() + self.global_constant('__true', + ir.ArrayType(int8_t, 5), + self.make_bytearray('True\00'.encode('ascii'))) + self.global_constant('__false', + ir.ArrayType(int8_t, 6), + self.make_bytearray('False\00'.encode('ascii'))) + self.global_constant('__fmt_i', + ir.ArrayType(int8_t, 4), + self.make_bytearray('%i\n\00'.encode('ascii'))) + self.global_constant('__fmt_s', + ir.ArrayType(int8_t, 4), + self.make_bytearray('%s\n\00'.encode('ascii'))) + self.global_constant('__fmt_assert', + ir.ArrayType(int8_t, 29), + self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))) + self.global_constant('__fmt_err', + ir.ArrayType(int8_t, 18), + self.make_bytearray('Error on line %i\n\00'.encode('ascii'))) + self.global_constant('__fmt_str_concat', + ir.ArrayType(int8_t, 5), + self.make_bytearray('%s%s\00'.encode('ascii'))) + self.global_constant( + "__jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * JMP_BUF_BYTES))) printf_t = ir.FunctionType(int32_t, [voidptr_t], True) self.externs['printf'] = ir.Function(self.module, printf_t, 'printf') @@ -101,11 +125,32 @@ def Program(self, node: Program): self.externs['memcpy'] = ir.Function(self.module, memcpy_t, 'memcpy') # begin main function - + # declare global variables, methods, and functions + varDefs = [d for d in node.declarations if isinstance(d, VarDef)] + for d in varDefs: + t = d.var.t.getLLVMType() + self.global_variable(d.var.name(), t) funcDefs = [d for d in node.declarations if isinstance(d, FuncDef)] + for d in funcDefs: + self.declareFunc(d) + classDefs = [d for d in node.declarations if isinstance(d, ClassDef)] + for cls in classDefs: + methodDefs = [d for d in cls.declarations if isinstance(d, FuncDef)] + for m in methodDefs: + if m.getIdentifier().name != "__init__": + raise Exception("TODO") + # provide default __init__ impl for classes + for cls in self.constructors: + ctor = self.constructors[cls] + if len(ctor.blocks) == 0: + bb = ctor.append_basic_block('entry') + ir.IRBuilder(bb).ret_void() + + # define functions for d in funcDefs: self.visit(d) + # main function funcType = ir.FunctionType(ir.VoidType(), []) func = ir.Function(self.module, funcType, "__main__") @@ -114,7 +159,7 @@ def Program(self, node: Program): self.builder = ir.IRBuilder(entry_block) status = self.builder.call(self.externs['setjmp'], [ - self.globals['jmp_buf']]) + self.module.get_global('__jmp_buf')]) cond = self.builder.icmp_signed("!=", int32_t(0), status) error_block = self.builder.append_basic_block('error_handling') @@ -125,13 +170,19 @@ def Program(self, node: Program): program_block) self.builder.position_at_start(error_block) - self.printf(self.globals['fmt_err'], status) + self.printf(self.module.get_global('__fmt_err'), status) self.builder.branch(end_program) error_block = self.builder.block self.builder.position_at_start(program_block) - self.programHelper(node) + # initialize globals + for d in varDefs: + val = self.visit(d.value) + addr = self.module.get_global(d.var.name()) + assert addr is not None + self.builder.store(val, addr) + self.visitStmtList(node.statements) self.builder.branch(end_program) program_block = self.builder.block @@ -139,11 +190,6 @@ def Program(self, node: Program): self.builder.ret_void() self.exitScope() - def programHelper(self, node: Program): - self.visitStmtList( - [d for d in node.declarations if isinstance(d, VarDef)]) - self.visitStmtList(node.statements) - def VarDef(self, node: VarDef): val = self.visit(node.value) saved_block = self.builder.block @@ -156,16 +202,20 @@ def VarDef(self, node: VarDef): def ClassDef(self, node: ClassDef): pass - def FuncDef(self, node: FuncDef): - # TODO - methods + def declareFunc(self, node: FuncDef): self.returnType = node.type.returnType - shouldReturnValue = not self.returnType.isNone() funcname = node.name.name returnType = node.type.returnType.getLLVMType() argTypes = [p.getLLVMType() for p in node.type.parameters] funcType = ir.FunctionType(returnType, argTypes) - func = ir.Function(self.module, funcType, funcname) - self.globals[funcname] = func + ir.Function(self.module, funcType, funcname) + + def FuncDef(self, node: FuncDef): + # TODO - methods + shouldReturnValue = not self.returnType.isNone() + func = self.module.get_global(node.getIdentifier().name) + self.returnType = node.type.returnType + shouldReturnValue = not self.returnType.isNone() self.enterScope() bb_entry = func.append_basic_block('entry') self.builder = ir.IRBuilder(bb_entry) @@ -202,7 +252,7 @@ def AssignStmt(self, node: AssignStmt): lst, idx, var.inferredType.getLLVMType(), True, var.index.location[0]) self.builder.store(val, ptr) elif isinstance(var, Identifier): - addr = self.locals[-1][var.name] + addr = self.getAddr(var) self.builder.store(val, addr) else: raise Exception("Illegal assignment") @@ -275,7 +325,7 @@ def BinaryExpr(self, node: BinaryExpr): llen, rlen), int32_t(1)) new_str = self.builder.call( self.externs['malloc'], [total_len], 'new_str') - fmt = self.toVoidPtr(self.globals['fmt_str_concat']) + fmt = self.toVoidPtr(self.module.get_global('__fmt_str_concat')) self.builder.call(self.externs['sprintf'], [ new_str, fmt, lhs, rhs]) return new_str @@ -397,13 +447,21 @@ def UnaryExpr(self, node: UnaryExpr): val = self.visit(node.operand) return self.builder.icmp_unsigned('==', bool_t(0), val) + def constructor(self, node: CallExpr): + # TODO - calculate size + obj = self.builder.call(self.externs['malloc'], [ir.Constant(int32_t, 1)], 'new_object') + self.builder.call(self.constructors[node.function.name], [obj]) + return obj + def CallExpr(self, node: CallExpr): - if node.function.name == "print": + if node.isConstructor: + return self.constructor(node) + elif node.function.name == "print": return self.emit_print(node.args[0]) - if node.function.name == "__assert__": + elif node.function.name == "__assert__": self.emit_assert(node.args[0]) return - if node.function.name == "len": + elif node.function.name == "len": return self.emit_len(node.args[0]) callee_func = self.module.get_global(node.function.name) if callee_func is None or not isinstance(callee_func, ir.Function): @@ -415,7 +473,7 @@ def CallExpr(self, node: CallExpr): return self.builder.call(callee_func, call_args, 'calltmp') def ForStmt(self, node: ForStmt): - var = self.locals[-1][node.identifier.name] + var = self.getAddr(node.identifier) idx_var = self.builder.alloca(int32_t, None, 'idx') self.builder.store(int32_t(0), idx_var) iterable = self.visit(node.iterable) @@ -514,9 +572,18 @@ def ReturnStmt(self, node: ReturnStmt): val = self.visit(node.value) self.builder.ret(val) + def getAddr(self, node: Identifier): + if node.varInstance.isGlobal: + return self.module.get_global(node.name) + elif node.varInstance.isNonlocal: + raise Exception("unimplemented") + else: + addr = self.locals[-1][node.name] + assert addr is not None + return addr + def Identifier(self, node: Identifier): - addr = self.locals[-1][node.name] - assert addr is not None + addr = self.getAddr(node) return self.builder.load(addr, node.name) def MemberExpr(self, node: MemberExpr): @@ -617,7 +684,7 @@ def emit_assert(self, arg: Expr): ) def longJmp(self, line: int): - jmp_buf = self.module.get_global("jmp_buf") + jmp_buf = self.module.get_global('__jmp_buf') self.builder.call(self.externs['longjmp'], [ jmp_buf, int32_t(line)]) @@ -627,14 +694,14 @@ def emit_print(self, arg: Expr): if arg.inferredType == BoolType(): text = self.ifHelper( lambda: self.visit(arg), - lambda: self.toVoidPtr(self.globals['true']), - lambda: self.toVoidPtr(self.globals['false']), + lambda: self.toVoidPtr(self.module.get_global('__true')), + lambda: self.toVoidPtr(self.module.get_global('__false')), voidptr_t) - self.printf(self.globals['fmt_s'], text) + self.printf(self.module.get_global('__fmt_s'), text) elif arg.inferredType.className == 'int': - self.printf(self.globals['fmt_i'], self.visit(arg)) + self.printf(self.module.get_global('__fmt_i'), self.visit(arg)) else: - self.printf(self.globals['fmt_s'], self.visit(arg)) + self.printf(self.module.get_global('__fmt_s'), self.visit(arg)) return self.NoneLiteral(None) # UTILS @@ -650,12 +717,20 @@ def printf(self, format, arg): def global_constant(self, name, t, value): module = self.module - data = ir.GlobalVariable(module, t, name, 0) + data = ir.GlobalVariable(module, t, name) data.linkage = 'internal' data.global_constant = True data.initializer = value return data + def global_variable(self, name, t): + module = self.module + data = ir.GlobalVariable(module, t, name) + data.linkage = 'internal' + data.initializer = t(None) + data.global_constant = False + return data + def sizeof(self, t): if not t.is_pointer: width = t.width diff --git a/foobar.py b/foobar.py index e69de29..7174360 100644 --- a/foobar.py +++ b/foobar.py @@ -0,0 +1,16 @@ +x: int = 1 + + +def test(): + y: [int] = None + + def inner(): + global x + for x in y: + pass + y = [1, 2, 3] + inner() + assert x == 3 + + +test() diff --git a/test.py b/test.py index dab8fc1..3b031de 100644 --- a/test.py +++ b/test.py @@ -24,7 +24,7 @@ def run_all_tests(): # run_cil_tests() # run_wasm_tests() run_llvm_tests() - test_eval_llvm() + # test_eval_llvm() def run_parse_tests(): @@ -581,19 +581,14 @@ def ast_equals(d1, d2) -> bool: "/binary_tree.", "/classes.", "/doubling_vector.", - "/globals.", "/nonlocal_builtins.", "/nonlocal_loop.", "/nonlocal.", "/ratio.", - "/operators.", "/inherit_init.", "/linked_list.", "/exponent.", - "/lists.", "/short_circuit.", - "/global_loop.", - "/control_flow.", "modulo" ] @@ -630,6 +625,7 @@ def eval_llvm(module): target = llvm.Target.from_default_triple() target_machine = target.create_target_machine() llvmmod = llvm.parse_assembly(str(module)) + llvmmod.verify() with llvm.create_mcjit_compiler(llvmmod, target_machine) as ee: ee.finalize_object() fptr = CFUNCTYPE(None)(ee.get_function_address("__main__")) @@ -637,10 +633,11 @@ def eval_llvm(module): def test_eval_llvm(): - run_llvm_test("foobar.py", False) + run_llvm_test("foobar.py", True) def run_llvm_test(test, debug): + print("running test", test) try: compiler = Compiler() astparser = compiler.parser diff --git a/tests/runtime/modulo.py b/tests/runtime/modulo.py index 886fad1..81440f1 100644 --- a/tests/runtime/modulo.py +++ b/tests/runtime/modulo.py @@ -17,4 +17,4 @@ assert -5 % 5 == 0 assert -5 % -5 == 0 -assert 5 % -5 == 0 \ No newline at end of file +assert 5 % -5 == 0 From e27efefe4ea9b0bb770e6480610c85d759127d1a Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 28 May 2023 23:42:37 -0700 Subject: [PATCH 62/79] nonlocals --- compiler/llvm_backend.py | 69 ++++++++++++++++++++++++++++++++-------- foobar.py | 6 ++-- test.py | 2 -- 3 files changed, 57 insertions(+), 20 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 453bb4c..923b545 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -135,7 +135,8 @@ def Program(self, node: Program): self.declareFunc(d) classDefs = [d for d in node.declarations if isinstance(d, ClassDef)] for cls in classDefs: - methodDefs = [d for d in cls.declarations if isinstance(d, FuncDef)] + methodDefs = [ + d for d in cls.declarations if isinstance(d, FuncDef)] for m in methodDefs: if m.getIdentifier().name != "__init__": raise Exception("TODO") @@ -193,11 +194,23 @@ def Program(self, node: Program): def VarDef(self, node: VarDef): val = self.visit(node.value) saved_block = self.builder.block - addr = self.builder.alloca( - node.var.t.getLLVMType(), None, node.getName()) - self.builder.position_at_end(saved_block) - self.builder.store(val, addr) - self.locals[-1][node.getName()] = addr + if node.isAttr: + raise Exception("this should be handled elsewhere") + elif node.var.varInstance.isNonlocal: + addr = self.builder.alloca( + node.var.t.getLLVMType(), None, node.getName()) + wrapper = self.builder.alloca( + node.var.t.getLLVMType().as_pointer(), None, node.getName() + "_wrapper") + self.builder.position_at_end(saved_block) + self.builder.store(val, addr) + self.builder.store(addr, wrapper) + self.locals[-1][node.getName()] = wrapper + else: + addr = self.builder.alloca( + node.var.t.getLLVMType(), None, node.getName()) + self.builder.position_at_end(saved_block) + self.builder.store(val, addr) + self.locals[-1][node.getName()] = addr def ClassDef(self, node: ClassDef): pass @@ -205,9 +218,7 @@ def ClassDef(self, node: ClassDef): def declareFunc(self, node: FuncDef): self.returnType = node.type.returnType funcname = node.name.name - returnType = node.type.returnType.getLLVMType() - argTypes = [p.getLLVMType() for p in node.type.parameters] - funcType = ir.FunctionType(returnType, argTypes) + funcType = node.type.getLLVMType() ir.Function(self.module, funcType, funcname) def FuncDef(self, node: FuncDef): @@ -222,7 +233,7 @@ def FuncDef(self, node: FuncDef): for i, arg in enumerate(func.args): arg.name = node.params[i].name() alloca = self.builder.alloca( - node.type.parameters[i].getLLVMType(), name=arg.name) + node.type.getLLVMType().args[i], name=arg.name) self.builder.store(arg, alloca) self.locals[-1][arg.name] = alloca for d in node.declarations: @@ -325,7 +336,8 @@ def BinaryExpr(self, node: BinaryExpr): llen, rlen), int32_t(1)) new_str = self.builder.call( self.externs['malloc'], [total_len], 'new_str') - fmt = self.toVoidPtr(self.module.get_global('__fmt_str_concat')) + fmt = self.toVoidPtr( + self.module.get_global('__fmt_str_concat')) self.builder.call(self.externs['sprintf'], [ new_str, fmt, lhs, rhs]) return new_str @@ -449,10 +461,33 @@ def UnaryExpr(self, node: UnaryExpr): def constructor(self, node: CallExpr): # TODO - calculate size - obj = self.builder.call(self.externs['malloc'], [ir.Constant(int32_t, 1)], 'new_object') + obj = self.builder.call(self.externs['malloc'], [ + ir.Constant(int32_t, 1)], 'new_object') self.builder.call(self.constructors[node.function.name], [obj]) return obj + def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): + argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + paramIsRef = paramIdx in funcType.refParams + if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + # ref arg and ref param, pass ref arg + return self.getAddr(arg) + elif paramIsRef: + # non-ref arg and ref param, or do not pass ref arg + # unwrap if necessary, re-wrap + saved_block = self.builder.block + val = self.visit(arg) + addr = self.builder.alloca( + node.var.t.getLLVMType()) + wrapper = self.builder.alloca( + node.var.t.getLLVMType().as_pointer(), None, "wrapper") + self.builder.position_at_end(saved_block) + self.builder.store(val, addr) + self.builder.store(addr, wrapper) + return wrapper + else: # non-ref param, maybe unwrap + return self.visit(arg) + def CallExpr(self, node: CallExpr): if node.isConstructor: return self.constructor(node) @@ -469,7 +504,10 @@ def CallExpr(self, node: CallExpr): if len(callee_func.args) != len(node.args): raise Exception('Call argument length mismatch', node.function.name) - call_args = [self.visit(arg) for arg in node.args] + call_args = [] + for i in range(len(node.args)): + call_args.append(self.visitArg( + node.function.inferredType, i, node.args[i])) return self.builder.call(callee_func, call_args, 'calltmp') def ForStmt(self, node: ForStmt): @@ -576,7 +614,10 @@ def getAddr(self, node: Identifier): if node.varInstance.isGlobal: return self.module.get_global(node.name) elif node.varInstance.isNonlocal: - raise Exception("unimplemented") + addr = self.locals[-1][node.name] + assert addr is not None + return self.builder.load(addr) + # return self.builder.gep(addr, [int32_t(0)]) else: addr = self.locals[-1][node.name] assert addr is not None diff --git a/foobar.py b/foobar.py index 7174360..7583504 100644 --- a/foobar.py +++ b/foobar.py @@ -1,11 +1,9 @@ -x: int = 1 - - def test(): + x: int = 1 y: [int] = None def inner(): - global x + nonlocal x for x in y: pass y = [1, 2, 3] diff --git a/test.py b/test.py index 3b031de..5b5b4cc 100644 --- a/test.py +++ b/test.py @@ -581,8 +581,6 @@ def ast_equals(d1, d2) -> bool: "/binary_tree.", "/classes.", "/doubling_vector.", - "/nonlocal_builtins.", - "/nonlocal_loop.", "/nonlocal.", "/ratio.", "/inherit_init.", From c3862ae31daac311f5796e4ea7143305dc842b65 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sat, 10 Jun 2023 18:52:21 -0700 Subject: [PATCH 63/79] add annotations across the codebase --- compiler/astnodes/classdef.py | 2 +- compiler/astnodes/funcdef.py | 8 ++-- compiler/astnodes/identifier.py | 3 +- compiler/astnodes/node.py | 2 +- compiler/astnodes/typedvar.py | 5 ++- compiler/astnodes/vardef.py | 6 ++- compiler/cil_backend.py | 7 +-- compiler/closurevisitor.py | 14 ++---- compiler/compiler.py | 3 +- compiler/empty_list_typer.py | 6 +-- compiler/jvm_backend.py | 8 ++-- compiler/llvm_backend.py | 40 ++++++++++++----- compiler/nestedfunchoister.py | 5 +++ compiler/parser.py | 80 ++++++++++++++++----------------- compiler/typechecker.py | 7 +++ compiler/types/__init__.py | 1 + compiler/types/functype.py | 8 +++- compiler/types/symboltype.py | 11 +++-- compiler/types/valuetype.py | 10 ++--- compiler/types/varinstance.py | 4 ++ compiler/typesystem.py | 31 ++++++++----- compiler/varcollector.py | 1 + compiler/visitor.py | 4 +- compiler/wasm_backend.py | 22 +++++---- foobar.py | 2 +- test.py | 2 +- 26 files changed, 173 insertions(+), 119 deletions(-) create mode 100644 compiler/types/varinstance.py diff --git a/compiler/astnodes/classdef.py b/compiler/astnodes/classdef.py index 71b24f3..772f389 100644 --- a/compiler/astnodes/classdef.py +++ b/compiler/astnodes/classdef.py @@ -46,7 +46,7 @@ def toJSON(self, dump_location=True): for decl in self.declarations] return d - def getIdentifier(self): + def getIdentifier(self) -> Identifier: return self.name def getDefaultConstructor(self) -> FuncDef: diff --git a/compiler/astnodes/funcdef.py b/compiler/astnodes/funcdef.py index 9f6d384..98cf3fa 100644 --- a/compiler/astnodes/funcdef.py +++ b/compiler/astnodes/funcdef.py @@ -3,10 +3,13 @@ from .typedvar import TypedVar from .typeannotation import TypeAnnotation from .stmt import Stmt +from ..types import FuncType from typing import List class FuncDef(Declaration): + freevars: List[Identifier] # used in AST transformations, not printed out + type: FuncType = None # type signature of function # The AST for # def NAME(PARAMS) -> RETURNTYPE: @@ -22,8 +25,7 @@ def __init__(self, location: List[int], name: Identifier, params: List[TypedVar] self.declarations = declarations self.statements = [s for s in statements if s is not None] self.isMethod = isMethod - self.freevars = [] # used in AST transformations, not printed out - self.type = None # type signature of function + self.freevars = [] def getFreevarNames(self): return set([v.name for v in self.freevars]) @@ -56,5 +58,5 @@ def toJSON(self, dump_location=True): d["statements"] = [s.toJSON(dump_location) for s in self.statements] return d - def getIdentifier(self): + def getIdentifier(self) -> Identifier: return self.name diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index 87ae1fe..a5065fa 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -1,5 +1,6 @@ from .expr import Expr from typing import List +from ..types import VarInstance CIL_KEYWORDS = set(["char", "value", "int32", "int64", "string", "long", "null"] + ["add", @@ -121,11 +122,11 @@ class Identifier(Expr): + varInstance: VarInstance = None def __init__(self, location: List[int], name: str): super().__init__(location, "Identifier") self.name = name - self.varInstance = None def visit(self, visitor): return visitor.Identifier(self) diff --git a/compiler/astnodes/node.py b/compiler/astnodes/node.py index 6af5cca..1a05981 100644 --- a/compiler/astnodes/node.py +++ b/compiler/astnodes/node.py @@ -19,7 +19,7 @@ def preorder(self, visitor): def postorder(self, visitor): return self.visit(visitor) - def toJSON(self, dump_location=True): + def toJSON(self, dump_location=True) -> dict: d = {} d['kind'] = self.kind if dump_location: diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 6034ba1..4903d90 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -1,17 +1,18 @@ from .node import Node from .identifier import Identifier from .typeannotation import TypeAnnotation +from ..types import ValueType, VarInstance from typing import List class TypedVar(Node): + t: ValueType = None # the typechecked type goes here + varInstance: VarInstance = None def __init__(self, location: List[int], identifier: Identifier, typ: TypeAnnotation): super().__init__(location, "TypedVar") self.identifier = identifier self.type = typ - self.t = None # the typechecked type goes here - self.varInstance = None def name(self): return self.identifier.name diff --git a/compiler/astnodes/vardef.py b/compiler/astnodes/vardef.py index 2cc5042..1c08816 100644 --- a/compiler/astnodes/vardef.py +++ b/compiler/astnodes/vardef.py @@ -1,10 +1,12 @@ from .declaration import Declaration from .expr import Expr +from .identifier import Identifier from .typedvar import TypedVar -from typing import List +from typing import List, Optional class VarDef(Declaration): + attrOfClass: Optional[str] def __init__(self, location: List[int], var: TypedVar, value: Expr, isAttr: bool = False, attrOfClass=None): super().__init__(location, "VarDef") @@ -31,7 +33,7 @@ def toJSON(self, dump_location=True): d["value"] = self.value.toJSON(dump_location) return d - def getIdentifier(self): + def getIdentifier(self) -> Identifier: return self.var.identifier def getName(self) -> str: diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index b03d37c..301e37d 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -8,13 +8,13 @@ class CilStackLoc: - def __init__(self, name, loc, t, isArg): + def __init__(self, name: str, loc: int, t: str, isArg: bool): self.name = name self.loc = loc self.isArg = isArg self.t = t - def decl(self): + def decl(self) -> str: return f"[{self.loc}] {self.t} {self.name}" @@ -26,6 +26,7 @@ def __init__(self, main: str, ts: TypeSystem): self.builder = Builder(main) self.main = main # name of main class self.ts = ts + self.locals = [] self.enterScope() def indent(self): @@ -612,7 +613,7 @@ def BooleanLiteral(self, node: BooleanLiteral): def IntegerLiteral(self, node: IntegerLiteral): self.instr(f"ldc.i8 {node.value}") - def NoneLiteral(self, node: NoneLiteral): + def NoneLiteral(self, _: NoneLiteral): self.instr("ldnull") def StringLiteral(self, node: StringLiteral): diff --git a/compiler/closurevisitor.py b/compiler/closurevisitor.py index 57f6813..33273f2 100644 --- a/compiler/closurevisitor.py +++ b/compiler/closurevisitor.py @@ -2,14 +2,7 @@ from .types import * from .visitor import Visitor from .varcollector import VarCollector -from typing import List - - -class VarInstance: - def __init__(self): - self.isNonlocal = False - self.isGlobal = False - self.isSelf = False +from typing import List, Dict def newInstance(tv: TypedVar) -> VarInstance: @@ -17,7 +10,7 @@ def newInstance(tv: TypedVar) -> VarInstance: return tv.varInstance -def merge(d1, d2): +def merge(d1: dict, d2: dict) -> dict: combined = {} for k in d1: combined[k] = d1[k] @@ -43,10 +36,11 @@ class ClosureVisitor(Visitor): # instances that are captured by nested functions are marked as refs # instances that correspond to global variables are marked as such + globals: Dict[str, VarInstance] + decls: List[Dict[str, VarInstance]] def __init__(self): self.globals = {} - self.nonlocals = [] # uncaptured nonlocals self.decls = [] def getInstance(self, name: str) -> VarInstance: diff --git a/compiler/compiler.py b/compiler/compiler.py index 1c1880c..7450283 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -17,11 +17,12 @@ class Compiler: + transformer: ClosureTransformer = None + def __init__(self): self.ts = TypeSystem() self.parser = Parser() self.typechecker = TypeChecker(self.ts) - self.transformer = None def parse(self, infile) -> Node: astparser = self.parser diff --git a/compiler/empty_list_typer.py b/compiler/empty_list_typer.py index e6e04b2..bb6a798 100644 --- a/compiler/empty_list_typer.py +++ b/compiler/empty_list_typer.py @@ -9,10 +9,8 @@ class EmptyListTyper(Visitor): - - def __init__(self): - self.expectedType = None - self.expReturnType = None + expectedType: ValueType = None + expReturnType: ValueType = None def visit(self, node: Node): return node.preorder(self) diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index a03f789..1eb00e7 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from typing import List +from typing import List, Dict import json @@ -11,16 +11,18 @@ class JvmBackend(CommonVisitor): localLimit = 50 stackLimit = 500 defaultToGlobals = False # treat all vars as global if this is true + classes: Dict[str, Builder] def __init__(self, main: str, ts: TypeSystem): - self.classes = dict() + self.classes = {} self.classes[main] = Builder(main) self.currentClass = main self.main = main # name of main class self.ts = ts + self.locals = [] self.enterScope() - def currentBuilder(self): + def currentBuilder(self) -> Builder: return self.classes[self.currentClass] def newLabelName(self) -> str: diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 923b545..d40d86c 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -3,8 +3,7 @@ from .typesystem import TypeSystem from .visitor import Visitor from collections import defaultdict -from typing import List -import json +from typing import List, Dict, Tuple import llvmlite.ir as ir import llvmlite.binding as llvm @@ -19,21 +18,26 @@ class LlvmBackend(Visitor): - locals = [] - counter = 0 - globals = {} - externs = {} - constructors = {} + locals: List[defaultdict] + externs: Dict[str, ir.Function] + constructors: Dict[str, ir.Function] + # (class name, method name) -> (idx in vtable, defining class name) + methodOffsets: Dict[Tuple[str, str], Tuple[int, str]] + # (class name, attr name) -> idx in struct + attrOffsets: Dict[Tuple[str, str], int] + builder: ir.builder = None def __init__(self, ts: TypeSystem): llvm.initialize() llvm.initialize_native_target() llvm.initialize_native_asmprinter() self.module = ir.Module() - self.builder = None - # (class name, method name) -> (idx in vtable, defining class name) - self.methodOffsets = dict() self.ts = ts + self.locals = [] + self.externs = {} + self.constructors = {} + self.methodOffsets = {} + self.attrOffsets = {} def initializeOffsets(self): tblOffset = 0 @@ -69,6 +73,13 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) + def getClassStructType(self, cls) -> ir.LiteralStructType: + elements = [voidptr_t] # pointer to vtable + attrs = self.ts.getOrderedAttrs(cls) + for attrInfo in attrs: + elements.append(attrInfo[1].getLLVMType()) + return ir.LiteralStructType() + # TOP LEVEL & DECLARATIONS def Program(self, node: Program): @@ -617,7 +628,6 @@ def getAddr(self, node: Identifier): addr = self.locals[-1][node.name] assert addr is not None return self.builder.load(addr) - # return self.builder.gep(addr, [int32_t(0)]) else: addr = self.locals[-1][node.name] assert addr is not None @@ -628,7 +638,13 @@ def Identifier(self, node: Identifier): return self.builder.load(addr, node.name) def MemberExpr(self, node: MemberExpr): - pass + cls = node.object.inferredType.className + attr = node.member.name + offset = self.attrOffsets[(cls, attr)] + obj = self.visit(node.object) + obj = self.builder.bitcast(obj, self.getClassStructType(cls).as_pointer()) + ptr = self.builder.gep(obj, [int32_t(offset)]) + return self.builder.load(ptr, attr) def IfExpr(self, node: IfExpr): return self.ifHelper(lambda: self.visit(node.condition), diff --git a/compiler/nestedfunchoister.py b/compiler/nestedfunchoister.py index be341f9..04ea0d5 100644 --- a/compiler/nestedfunchoister.py +++ b/compiler/nestedfunchoister.py @@ -1,6 +1,7 @@ from .astnodes import * from .types import * from .visitor import Visitor +from typing import List, Dict class HoistedFunctionInfo: @@ -12,6 +13,10 @@ def __init__(self, name, decl): class NestedFuncHoister(Visitor): # hoist all nested funcs to be top level funcs # rename hoisted functions to be unique & rename call sites + functionInfo: List[Dict[str, HoistedFunctionInfo]] + currentClass: str + nestingNames: List[str] + hoisted: List[FuncDef] def __init__(self): # map of function names to their modified names diff --git a/compiler/parser.py b/compiler/parser.py index f1a6a0a..12cd080 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -1,5 +1,5 @@ -from ast import * from .astnodes import * +import ast import typing @@ -14,25 +14,25 @@ def __init__(self, message, node=None): super().__init__(message + ".") -class Parser(NodeVisitor): +class Parser(ast.NodeVisitor): def __init__(self): self.errors = [] # reduce a list of >2 expressions separated by a # left-associative operator into a BinaryExpr tree - def binaryReduce(self, op: str, values: typing.List[Expr]) -> Expr: + def binaryReduce(self, op: str, values: typing.List[Expr]) -> BinaryExpr: current = BinaryExpr(values[0].location, values[0], op, values[1]) for v in values[2:]: current = BinaryExpr(values[0].location, current, op, v) return current - def getLocation(self, node) -> typing.List[int]: + def getLocation(self, node: ast.AST) -> typing.List[int]: # input is Python AST node # get 2 item list corresponding to AST node starting location # make columns 1-indexed return [node.lineno, node.col_offset + 1] - def visit(self, node): + def visit(self, node: ast.AST): try: return super().visit(node) except ParseError as e: @@ -40,15 +40,15 @@ def visit(self, node): return # process python AST nodes into chocopy type annotations - def getTypeAnnotation(self, node) -> TypeAnnotation: + def getTypeAnnotation(self, node: ast.expr) -> TypeAnnotation: location = self.getLocation(node) - if isinstance(node, List): + if isinstance(node, ast.List): if len(node.elts) > 1: raise ParseError("Unsupported List type annotation", node) return ListType(location, self.getTypeAnnotation(node.elts[0])) - elif isinstance(node, Name): + elif isinstance(node, ast.Name): return ClassType(location, node.id) - elif isinstance(node, Str): + elif isinstance(node, ast.Str): return ClassType(location, node.s) else: raise ParseError("Unsupported type annotation", node) @@ -56,7 +56,7 @@ def getTypeAnnotation(self, node) -> TypeAnnotation: # see https://greentreesnakes.readthedocs.io/en/latest/nodes.html # and https://docs.python.org/3/library/ast.html - def visit_Module(self, node): + def visit_Module(self, node: ast.Module) -> Program: location = [1, 1] if hasattr(node, "type_ignores") and node.type_ignores: raise ParseError("Cannot ignore type", node) @@ -88,7 +88,7 @@ def visit_Module(self, node): location = declarations[0].location return Program(location, declarations, statements, Errors([0, 0], [])) - def visit_FunctionDef(self, node): + def visit_FunctionDef(self, node: ast.FunctionDef) -> FuncDef: if node.decorator_list: raise ParseError("Unsupported decorator list", node.decorator_list[0]) @@ -124,7 +124,7 @@ def visit_FunctionDef(self, node): returns = self.getTypeAnnotation(node.returns) return FuncDef(location, identifier, arguments, returns, declarations, statements) - def visit_ClassDef(self, node): + def visit_ClassDef(self, node: ast.ClassDef) -> ClassDef: location = self.getLocation(node) identifier = Identifier([location[0], location[1] + 6], node.name) if len(node.bases) > 1: @@ -154,19 +154,19 @@ def visit_ClassDef(self, node): "Expected attribute or method declaration", node.body[i]) return ClassDef(location, identifier, base, body) - def visit_Return(self, node): + def visit_Return(self, node: ast.Return) -> ReturnStmt: location = self.getLocation(node) if node.value is None: return ReturnStmt(location, None) else: return ReturnStmt(location, self.visit(node.value)) - def visit_Assign(self, node): + def visit_Assign(self, node: ast.Assign) -> AssignStmt: location = self.getLocation(node) targets = [self.visit(t) for t in node.targets] return AssignStmt(location, targets, self.visit(node.value)) - def visit_AnnAssign(self, node): + def visit_AnnAssign(self, node: ast.AnnAssign) -> VarDef: if not node.value: raise ParseError("Expected initializing value", node) if not hasattr(node, "annotation") or not node.annotation: @@ -181,7 +181,7 @@ def visit_AnnAssign(self, node): raise ParseError("Expected literal value", node.value) return VarDef(location, var, value) - def visit_While(self, node): + def visit_While(self, node: ast.While) -> WhileStmt: location = self.getLocation(node) if node.orelse: raise ParseError("Cannot have else in while", node) @@ -192,7 +192,7 @@ def visit_While(self, node): raise ParseError("Illegal declaration", node) return WhileStmt(location, condition, body) - def visit_For(self, node): + def visit_For(self, node: ast.For) -> ForStmt: location = self.getLocation(node) if node.orelse: raise ParseError("Cannot have else in for", node) @@ -204,7 +204,7 @@ def visit_For(self, node): raise ParseError("Illegal declaration", node) return ForStmt(location, identifier, iterable, body) - def visit_If(self, node): + def visit_If(self, node: ast.If) -> IfStmt: location = self.getLocation(node) condition = self.visit(node.test) then_body = [self.visit(b) for b in node.body] @@ -216,7 +216,7 @@ def visit_If(self, node): raise ParseError("Illegal declaration", node) return IfStmt(location, condition, then_body, else_body) - def visit_Global(self, node): + def visit_Global(self, node: ast.Global) -> GlobalDecl: location = self.getLocation(node) if len(node.names) != 1: raise ParseError( @@ -226,7 +226,7 @@ def visit_Global(self, node): identifier = Identifier(idLoc, node.names[0]) return GlobalDecl(location, identifier) - def visit_Nonlocal(self, node): + def visit_Nonlocal(self, node: ast.Nonlocal) -> NonLocalDecl: location = self.getLocation(node) if len(node.names) != 1: raise ParseError( @@ -236,40 +236,40 @@ def visit_Nonlocal(self, node): identifier = Identifier(idLoc, node.names[0]) return NonLocalDecl(location, identifier) - def visit_Expr(self, node): + def visit_Expr(self, node: ast.Expr) -> ExprStmt: # this is a Stmt that evaluates an Expr location = self.getLocation(node) val = self.visit(node.value) return ExprStmt(location, val) - def visit_Pass(self, node): + def visit_Pass(self, _: ast.Pass) -> None: # removed by any AST constructors that take in [Stmt] return None - def visit_BoolOp(self, node): + def visit_BoolOp(self, node: ast.BoolOp) -> BinaryExpr: values = [self.visit(v) for v in node.values] op = self.visit(node.op) return self.binaryReduce(op, values) - def visit_BinOp(self, node): + def visit_BinOp(self, node: ast.BinOp) -> BinaryExpr: left = self.visit(node.left) right = self.visit(node.right) location = self.getLocation(node) return BinaryExpr(location, left, self.visit(node.op), right) - def visit_UnaryOp(self, node): + def visit_UnaryOp(self, node: ast.UnaryOp) -> UnaryExpr: operand = self.visit(node.operand) location = self.getLocation(node) return UnaryExpr(location, self.visit(node.op), operand) - def visit_IfExp(self, node): + def visit_IfExp(self, node: ast.IfExp) -> IfExpr: location = self.getLocation(node) condition = self.visit(node.test) then_body = self.visit(node.body) else_body = self.visit(node.orelse) return IfExpr(location, condition, then_body, else_body) - def visit_Call(self, node): + def visit_Call(self, node: ast.Call) -> Expr: location = self.getLocation(node) function = self.visit(node.func) if node.keywords: @@ -281,7 +281,7 @@ def visit_Call(self, node): return CallExpr(location, function, arguments) raise ParseError("Invalid receiver of call", node.func) - def visit_Constant(self, node): + def visit_Constant(self, node: ast.Constant) -> Expr: # support for Python 3.8 location = self.getLocation(node) if isinstance(node.value, bool): @@ -295,7 +295,7 @@ def visit_Constant(self, node): else: raise ParseError("Unsupported constant", node) - def visit_Compare(self, node): + def visit_Compare(self, node: ast.Compare) -> BinaryExpr: if len(node.ops) > 1 or len(node.comparators) > 1: raise ParseError("Unsupported compare between > 2 things", node) location = self.getLocation(node) @@ -304,36 +304,36 @@ def visit_Compare(self, node): right = self.visit(node.comparators[0]) return BinaryExpr(location, left, operator, right) - def visit_Attribute(self, node): + def visit_Attribute(self, node: ast.Attribute) -> MemberExpr: location = self.getLocation(node) obj = self.visit(node.value) member = Identifier(location, node.attr) return MemberExpr(location, obj, member) - def visit_Subscript(self, node): + def visit_Subscript(self, node: ast.Subscript) -> IndexExpr: location = self.getLocation(node) return IndexExpr(location, self.visit(node.value), self.visit(node.slice)) - def visit_Name(self, node): + def visit_Name(self, node: ast.Name) -> Identifier: location = self.getLocation(node) return Identifier(location, node.id) - def visit_Num(self, node): + def visit_Num(self, node: ast.Num) -> IntegerLiteral: location = self.getLocation(node) if not isinstance(node.n, int): raise ParseError("Only integers are supported", node) return IntegerLiteral(location, node.n) - def visit_Str(self, node): + def visit_Str(self, node: ast.Str) -> StringLiteral: location = self.getLocation(node) return StringLiteral(location, node.s) - def visit_List(self, node): + def visit_List(self, node: ast.List) -> ListExpr: location = self.getLocation(node) elements = [self.visit(e) for e in node.elts] return ListExpr(location, elements) - def visit_NameConstant(self, node): + def visit_NameConstant(self, node: ast.NameConstant) -> Expr: location = self.getLocation(node) if node.value is None: return NoneLiteral(location) @@ -342,10 +342,10 @@ def visit_NameConstant(self, node): else: raise ParseError("Unsupported name constant", node) - def visit_Index(self, node): + def visit_Index(self, node: ast.Index): return self.visit(node.value) - def visit_arguments(self, node): + def visit_arguments(self, node: ast.arguments) -> list: if node.vararg: raise ParseError("Unsupported vararg", node.vararg) if node.kwarg: @@ -359,7 +359,7 @@ def visit_arguments(self, node): args = [self.visit(a) for a in args] return args - def visit_arg(self, node): + def visit_arg(self, node: ast.arg) -> TypedVar: # type annotation is either Str(s) or Name(id) if not hasattr(node, "annotation") or not node.annotation: raise ParseError("Missing type annotation", node) @@ -368,7 +368,7 @@ def visit_arg(self, node): annotation = self.getTypeAnnotation(node.annotation) return TypedVar(location, identifier, annotation) - def visit_Assert(self, node): + def visit_Assert(self, node: ast.Assert) -> ExprStmt: location = self.getLocation(node) func = Identifier(location, "__assert__") return ExprStmt(location, CallExpr(location, func, [self.visit(node.test)])) diff --git a/compiler/typechecker.py b/compiler/typechecker.py index 99cff01..96a2cbe 100644 --- a/compiler/typechecker.py +++ b/compiler/typechecker.py @@ -3,9 +3,16 @@ from collections import defaultdict from .typesystem import TypeSystem, ClassInfo from .visitor import Visitor +from typing import List, Optional class TypeChecker(Visitor): + symbolTable: List[defaultdict] + currentClass: str + errors: List[CompilerError] + expReturnType: Optional[ValueType] + program: Program + def __init__(self, ts: TypeSystem): # typechecker attributes and their chocopy typing judgement analogues: # O : symbolTable diff --git a/compiler/types/__init__.py b/compiler/types/__init__.py index 732d0d6..fe3b8c2 100644 --- a/compiler/types/__init__.py +++ b/compiler/types/__init__.py @@ -4,3 +4,4 @@ from .symboltype import SymbolType from .valuetype import ValueType from .Types import * +from .varinstance import VarInstance diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 7063d94..08f49df 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -1,16 +1,20 @@ from compiler.types.classvaluetype import ClassValueType from .valuetype import ValueType from .symboltype import SymbolType -from typing import List +from .varinstance import VarInstance +from typing import List, Dict from llvmlite import ir class FuncType(SymbolType): + refParams: Dict[int, VarInstance] + freevars: list + def __init__(self, parameters: List[ValueType], returnType: ValueType): self.parameters = parameters self.returnType = returnType self.refParams = {} - self.freevars = [] # used in AST transformations, not printed out + self.freevars = [] def __eq__(self, other): if isinstance(other, FuncType): diff --git a/compiler/types/symboltype.py b/compiler/types/symboltype.py index dd906ca..faf77f4 100644 --- a/compiler/types/symboltype.py +++ b/compiler/types/symboltype.py @@ -1,19 +1,22 @@ +from typing import Optional + + class SymbolType: # base class for types - def isValueType(): + def isValueType() -> bool: return False - def isListType(): + def isListType() -> bool: return False - def isFuncType(): + def isFuncType() -> bool: return False def elementType(): return None - def isSpecialType(): + def isSpecialType() -> bool: return False def toJSON(self, dump_location=True): diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index 70f1b34..e22ee14 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -2,22 +2,22 @@ class ValueType(SymbolType): - def isValueType(): + def isValueType() -> bool: return True - def isNone(self): + def isNone(self) -> bool: return False def toJSON(self, dump_location=True): raise Exception("unsupported") - def getJavaSignature(self): + def getJavaSignature(self) -> str: raise Exception("unsupported") - def isJavaRef(self): + def isJavaRef(self) -> bool: raise Exception("unsupported") - def isListType(self): + def isListType(self) -> bool: raise Exception("unsupported") def getLLVMType(self): diff --git a/compiler/types/varinstance.py b/compiler/types/varinstance.py new file mode 100644 index 0000000..21e3028 --- /dev/null +++ b/compiler/types/varinstance.py @@ -0,0 +1,4 @@ +class VarInstance: + isNonlocal = False + isGlobal = False + isSelf = False diff --git a/compiler/typesystem.py b/compiler/typesystem.py index 5ac7a8f..db31ba4 100644 --- a/compiler/typesystem.py +++ b/compiler/typesystem.py @@ -1,8 +1,13 @@ from .types import * from collections import defaultdict +from typing import List, Dict, Tuple, Any class ClassInfo: + orderedAttrs: List[str] + attrs: Dict[str, Tuple[ValueType, Any]] + methods: Dict[str, FuncType] + def __init__(self, name: str, superclass: str = None): self.name = name self.superclass = superclass @@ -15,6 +20,8 @@ def __str__(self): class TypeSystem: + classes: Dict[str, ClassInfo] + def __init__(self): # information for each class self.classes = defaultdict(lambda: None) @@ -38,7 +45,7 @@ def __init__(self): self.classes[""] = ClassInfo("", "object") self.classes[""] = ClassInfo("", "object") - def getMethodHelper(self, className: str, methodName: str): + def getMethodHelper(self, className: str, methodName: str) -> Tuple[FuncType, str]: # requires className to be the name of a valid class if methodName not in self.classes[className].methods: if self.classes[className].superclass is None: @@ -46,16 +53,16 @@ def getMethodHelper(self, className: str, methodName: str): return self.getMethodHelper(self.classes[className].superclass, methodName) return (self.classes[className].methods[methodName], className) - def getMethod(self, className: str, methodName: str): + def getMethod(self, className: str, methodName: str) -> FuncType: # requires className to be the name of a valid class return self.getMethodHelper(className, methodName)[0] - def getMethodDefClass(self, className: str, methodName: str): + def getMethodDefClass(self, className: str, methodName: str) -> str: # returns the class that the method was originally defined in # requires className to be the name of a valid class return self.getMethodHelper(className, methodName)[1] - def getAttrHelper(self, className: str, attrName: str): + def getAttrHelper(self, className: str, attrName: str) -> Tuple[ValueType, Any]: # requires className to be the name of a valid class if attrName not in self.classes[className].attrs: if self.classes[className].superclass is None: @@ -63,17 +70,17 @@ def getAttrHelper(self, className: str, attrName: str): return self.getAttrHelper(self.classes[className].superclass, attrName) return self.classes[className].attrs[attrName] - def getAttr(self, className: str, attrName: str): + def getAttr(self, className: str, attrName: str) -> ValueType: # returns type of attribute # requires className to be the name of a valid class return self.getAttrHelper(className, attrName)[0] - def getAttrInit(self, className: str, attrName: str): + def getAttrInit(self, className: str, attrName: str) -> Any: # returns initial value of attribute # requires className to be the name of a valid class return self.getAttrHelper(className, attrName)[1] - def getAttrOrMethod(self, className: str, name: str): + def getAttrOrMethod(self, className: str, name: str) -> SymbolType: # returns type of attribute or method # requires className to be the name of a valid class if name in self.classes[className].methods: @@ -120,7 +127,7 @@ def canAssign(self, a: ValueType, b: ValueType) -> bool: return self.canAssign(a.elementType, b.elementType) return False - def join(self, a: ValueType, b: ValueType): + def join(self, a: ValueType, b: ValueType) -> ValueType: # return closest mutual ancestor on typing tree if self.canAssign(a, b): return b @@ -151,7 +158,7 @@ def join(self, a: ValueType, b: ValueType): # this really shouldn't be returned return ObjectType() - def getOrderedMethods(self, className: str): + def getOrderedMethods(self, className: str) -> List[Tuple[str, FuncType, str]]: # (name, signature, defined in class) methods = [] if self.classes[className].superclass is not None: @@ -170,12 +177,12 @@ def getOrderedMethods(self, className: str): (name, self.classes[className].methods[name], className)) return methods - def getMappedMethods(self, className: str): + def getMappedMethods(self, className: str) -> Dict[str, Tuple[FuncType, str]]: # map of name -> signature, defined in class ordered = self.getOrderedMethods(className) return {x: (y, z) for x, y, z in ordered} - def getOrderedAttrs(self, className: str): + def getOrderedAttrs(self, className: str) -> List[Tuple[str, ValueType, Any]]: # return list of (name, type, init value) triples attrs = [] if self.classes[className].superclass is not None: @@ -185,7 +192,7 @@ def getOrderedAttrs(self, className: str): attrs.append((attr, attrType, attrInit)) return attrs - def getMappedAttrs(self, className: str): + def getMappedAttrs(self, className: str) -> Dict[str, Tuple[ValueType, Any]]: # map of name -> type, init value tuples ordered = self.getOrderedAttrs(className) return {x: (y, z) for x, y, z in ordered} diff --git a/compiler/varcollector.py b/compiler/varcollector.py index 0b24e7f..5c2d21a 100644 --- a/compiler/varcollector.py +++ b/compiler/varcollector.py @@ -6,6 +6,7 @@ class VarCollector(Visitor): # simple visitor to collect all the identifiers used as expressions or assignment targets + vars: List[Identifier] def __init__(self): self.vars = [] diff --git a/compiler/visitor.py b/compiler/visitor.py index 84ee370..a72ad62 100644 --- a/compiler/visitor.py +++ b/compiler/visitor.py @@ -1,6 +1,7 @@ from .astnodes import * from collections import defaultdict from .builder import Builder +from typing import List class Visitor: @@ -106,8 +107,7 @@ class CommonVisitor(Visitor): counter = 0 # for labels # helpers for handling locals - - locals = [] + locals: List[defaultdict] = None def enterScope(self): self.locals.append(defaultdict(lambda: None)) diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 10f971c..6eab413 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from typing import List +from typing import List, Dict, Tuple, Set class WasmBuilder(Builder): @@ -65,20 +65,24 @@ def newBlock(self) -> Builder: class WasmBackend(CommonVisitor): + # (class name, attr name) -> class offset + attrOffsets: Dict[Tuple[str, str], int] + # (class name, method name) -> (class offset, table offset, inherited) + methodOffsets: Dict[Tuple[str, str], Tuple[int, int, bool]] + # class -> offset of start of vtable + vtables: Dict[str, int] + undeclaredFuncs: Set[str] + locals: WasmBuilder = None + def __init__(self, main: str, ts: TypeSystem): self.builder = WasmBuilder(main) self.main = main # name of main method self.ts = ts self.defaultToGlobals = False # treat all vars as global if this is true self.localCounter = 0 - self.locals = None - - # (class name, attr name) -> class offset - self.attrOffsets = dict() - # (class name, method name) -> (class offset, table offset, inherited) - self.methodOffsets = dict() - # class -> offset of start of vtable - self.vtables = dict() + self.attrOffsets = {} + self.methodOffsets = {} + self.vtables = {} self.undeclaredFuncs = set() def initializeOffsets(self): diff --git a/foobar.py b/foobar.py index 7583504..7a0869b 100644 --- a/foobar.py +++ b/foobar.py @@ -11,4 +11,4 @@ def inner(): assert x == 3 -test() +test() \ No newline at end of file diff --git a/test.py b/test.py index 5b5b4cc..69d74ec 100644 --- a/test.py +++ b/test.py @@ -24,7 +24,7 @@ def run_all_tests(): # run_cil_tests() # run_wasm_tests() run_llvm_tests() - # test_eval_llvm() + test_eval_llvm() def run_parse_tests(): From 2cf3939439899d9e0be16522442e0d232709d121 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sat, 10 Jun 2023 19:39:15 -0700 Subject: [PATCH 64/79] add members --- compiler/llvm_backend.py | 88 ++++++++++++++++++++++++++-------------- foobar.py | 14 ------- test.py | 2 +- 3 files changed, 58 insertions(+), 46 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index d40d86c..12f82c0 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -21,11 +21,14 @@ class LlvmBackend(Visitor): locals: List[defaultdict] externs: Dict[str, ir.Function] constructors: Dict[str, ir.Function] + methods: Dict[str, Dict[str, ir.Function]] + structs: Dict[str, ir.LiteralStructType] # (class name, method name) -> (idx in vtable, defining class name) methodOffsets: Dict[Tuple[str, str], Tuple[int, str]] # (class name, attr name) -> idx in struct attrOffsets: Dict[Tuple[str, str], int] builder: ir.builder = None + currentClass: str = None def __init__(self, ts: TypeSystem): llvm.initialize() @@ -35,7 +38,8 @@ def __init__(self, ts: TypeSystem): self.ts = ts self.locals = [] self.externs = {} - self.constructors = {} + self.methods = {} + self.structs = {} self.methodOffsets = {} self.attrOffsets = {} @@ -47,18 +51,18 @@ def initializeOffsets(self): "" and c != ""] for cls in classes: ctorType = ir.FunctionType(ir.VoidType(), [voidptr_t]) - ctor = ir.Function(self.module, ctorType, cls) - self.constructors[cls] = ctor - for methName, _, defCls in self.ts.getOrderedMethods(cls): + ctor = ir.Function(self.module, ctorType, cls + "____init__") + self.methods[cls]["__init__"] = ctor + self.structs[cls] = self.getClassStructType(cls) + for idx, (methName, methType, defCls) in enumerate(self.ts.getOrderedMethods(cls)): + self.methodOffsets[(cls, methName)] = (idx, defCls) if cls == defCls: methodTableOffsets[(cls, methName)] = tblOffset tblOffset += 1 - # calculate info for each class - for cls in classes: - methods = self.ts.getOrderedMethods(cls) - for idx, methInfo in enumerate(methods): - name, _, defCls = methInfo - self.methodOffsets[(cls, name)] = (idx, defCls) + if methName != "__init__": + funcType = methType.getLLVMType() + self.methods[cls][methName] = ir.Function(self.module, funcType, cls + "__" + methName) + # TODO - set up tables def enterScope(self): self.locals.append(defaultdict(lambda: None)) @@ -73,12 +77,12 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) - def getClassStructType(self, cls) -> ir.LiteralStructType: + def getClassStructType(self, cls: str) -> ir.LiteralStructType: elements = [voidptr_t] # pointer to vtable attrs = self.ts.getOrderedAttrs(cls) for attrInfo in attrs: elements.append(attrInfo[1].getLLVMType()) - return ir.LiteralStructType() + return ir.LiteralStructType(elements) # TOP LEVEL & DECLARATIONS @@ -146,14 +150,16 @@ def Program(self, node: Program): self.declareFunc(d) classDefs = [d for d in node.declarations if isinstance(d, ClassDef)] for cls in classDefs: + self.currentClass = cls.name.name methodDefs = [ d for d in cls.declarations if isinstance(d, FuncDef)] for m in methodDefs: - if m.getIdentifier().name != "__init__": - raise Exception("TODO") + self.visit(m) + self.currentClass = None + # provide default __init__ impl for classes - for cls in self.constructors: - ctor = self.constructors[cls] + for cls in self.methods: + ctor = self.methods[cls]["__init__"] if len(ctor.blocks) == 0: bb = ctor.append_basic_block('entry') ir.IRBuilder(bb).ret_void() @@ -227,15 +233,15 @@ def ClassDef(self, node: ClassDef): pass def declareFunc(self, node: FuncDef): - self.returnType = node.type.returnType funcname = node.name.name funcType = node.type.getLLVMType() ir.Function(self.module, funcType, funcname) def FuncDef(self, node: FuncDef): - # TODO - methods - shouldReturnValue = not self.returnType.isNone() - func = self.module.get_global(node.getIdentifier().name) + if node.isMethod: + func = self.module.get_global(self.curentClass + "__" + node.getIdentifier().name) + else: + func = self.module.get_global(node.getIdentifier().name) self.returnType = node.type.returnType shouldReturnValue = not self.returnType.isNone() self.enterScope() @@ -261,11 +267,20 @@ def FuncDef(self, node: FuncDef): # STATEMENTS + def getAttrPtr(self, node: MemberExpr): + cls = node.object.inferredType.className + attr = node.member.name + offset = self.attrOffsets[(cls, attr)] + obj = self.visit(node.object) + obj = self.builder.bitcast(obj, self.structs[cls].as_pointer()) + return self.builder.gep(obj, [int32_t(offset)]) + def AssignStmt(self, node: AssignStmt): val = self.visit(node.value) for var in node.targets[::-1]: if isinstance(var, MemberExpr): - raise Exception("unimplemented") + ptr = self.getAttrPtr(var) + self.builder.store(val, ptr) elif isinstance(var, IndexExpr): lst = self.visit(var.list) idx = self.visit(var.index) @@ -471,10 +486,10 @@ def UnaryExpr(self, node: UnaryExpr): return self.builder.icmp_unsigned('==', bool_t(0), val) def constructor(self, node: CallExpr): - # TODO - calculate size - obj = self.builder.call(self.externs['malloc'], [ - ir.Constant(int32_t, 1)], 'new_object') - self.builder.call(self.constructors[node.function.name], [obj]) + cls = node.function.name + size = self.sizeof(self.structs[cls]) + obj = self.builder.call(self.externs['malloc'], [size], 'new_object') + self.builder.call(self.constructors[cls + "____init__"], [obj]) return obj def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): @@ -638,12 +653,8 @@ def Identifier(self, node: Identifier): return self.builder.load(addr, node.name) def MemberExpr(self, node: MemberExpr): - cls = node.object.inferredType.className + ptr = self.getAttrPtr(node) attr = node.member.name - offset = self.attrOffsets[(cls, attr)] - obj = self.visit(node.object) - obj = self.builder.bitcast(obj, self.getClassStructType(cls).as_pointer()) - ptr = self.builder.gep(obj, [int32_t(offset)]) return self.builder.load(ptr, attr) def IfExpr(self, node: IfExpr): @@ -687,7 +698,22 @@ def ifHelper(self, condFn, thenFn, elseFn=None, returnType=None): return phi def MethodCallExpr(self, node: MethodCallExpr): - pass + obj = self.visit(node.method.object) + className = node.method.object.inferredType.className + methName = node.method.member.name + methIdx = self.methodOffsets[(className, methName)][0] + # TODO + callee_func = None + if callee_func is None or not isinstance(callee_func, ir.Function): + raise Exception("unknown method") + if len(callee_func.args) != len(node.args) + 1: + raise Exception('Call argument length mismatch', + node.function.name) + call_args = [obj] + for i in range(len(node.args)): + call_args.append(self.visitArg( + node.function.inferredType, i, node.args[i])) + return self.builder.call(callee_func, call_args, 'callmethodtmp') # LITERALS diff --git a/foobar.py b/foobar.py index 7a0869b..e69de29 100644 --- a/foobar.py +++ b/foobar.py @@ -1,14 +0,0 @@ -def test(): - x: int = 1 - y: [int] = None - - def inner(): - nonlocal x - for x in y: - pass - y = [1, 2, 3] - inner() - assert x == 3 - - -test() \ No newline at end of file diff --git a/test.py b/test.py index 69d74ec..1ffc3c0 100644 --- a/test.py +++ b/test.py @@ -23,7 +23,7 @@ def run_all_tests(): # run_jvm_tests() # run_cil_tests() # run_wasm_tests() - run_llvm_tests() + # run_llvm_tests() test_eval_llvm() From 3d806d1b156ffca97e93272fd1fd4408e437f9c4 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 11 Jun 2023 17:44:15 -0700 Subject: [PATCH 65/79] support classes --- compiler/llvm_backend.py | 124 +++++++++++++++++++++++------------- compiler/types/functype.py | 2 +- compiler/types/valuetype.py | 3 +- foobar.py | 59 +++++++++++++++++ test.py | 76 ++++++++++++++-------- 5 files changed, 192 insertions(+), 72 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 12f82c0..ed759a1 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -23,11 +23,11 @@ class LlvmBackend(Visitor): constructors: Dict[str, ir.Function] methods: Dict[str, Dict[str, ir.Function]] structs: Dict[str, ir.LiteralStructType] - # (class name, method name) -> (idx in vtable, defining class name) - methodOffsets: Dict[Tuple[str, str], Tuple[int, str]] - # (class name, attr name) -> idx in struct - attrOffsets: Dict[Tuple[str, str], int] - builder: ir.builder = None + # (class name, method name) -> idx in vtable, type + methodOffsets: Dict[Tuple[str, str], Tuple[int, ir.FunctionType]] + # idx in struct, initial value + attrOffsets: Dict[str, Dict[str, Tuple[int, Expr]]] + builder: ir.IRBuilder = None currentClass: str = None def __init__(self, ts: TypeSystem): @@ -44,25 +44,33 @@ def __init__(self, ts: TypeSystem): self.attrOffsets = {} def initializeOffsets(self): - tblOffset = 0 - methodTableOffsets = dict() # assign positions in the global method table classes = [c for c in self.ts.classes if c != "" and c != ""] for cls in classes: - ctorType = ir.FunctionType(ir.VoidType(), [voidptr_t]) - ctor = ir.Function(self.module, ctorType, cls + "____init__") - self.methods[cls]["__init__"] = ctor + self.attrOffsets[cls] = {} self.structs[cls] = self.getClassStructType(cls) - for idx, (methName, methType, defCls) in enumerate(self.ts.getOrderedMethods(cls)): - self.methodOffsets[(cls, methName)] = (idx, defCls) - if cls == defCls: - methodTableOffsets[(cls, methName)] = tblOffset - tblOffset += 1 - if methName != "__init__": - funcType = methType.getLLVMType() - self.methods[cls][methName] = ir.Function(self.module, funcType, cls + "__" + methName) - # TODO - set up tables + attrs = self.ts.getOrderedAttrs(cls) + for i, (name, _, val) in enumerate(attrs): + # offset by 1 because first field of struct is ptr to vtable + self.attrOffsets[cls][name] = (i + 1, val) + + self.methods[cls] = {} + orderedMethods = self.ts.getOrderedMethods(cls) + vtable = [] + for idx, (methName, methType, _) in enumerate(orderedMethods): + funcType = methType.getLLVMType() + self.methodOffsets[(cls, methName)] = (idx, funcType) + func = ir.Function(self.module, funcType, + cls + "__" + methName) + self.methods[cls][methName] = func + for methName, _, _ in orderedMethods: + func = self.methods[cls][methName] + vtable.append(func) + t = self.getClassVtableType(cls) + self.global_constant('__' + cls + '__vtable', + t, + ir.Constant(t, vtable)) def enterScope(self): self.locals.append(defaultdict(lambda: None)) @@ -77,8 +85,17 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) + def getClassVtableType(self, cls: str) -> ir.LiteralStructType: + orderedMethods = self.ts.getOrderedMethods(cls) + elements = [] + for _, methType, _ in orderedMethods: + funcType = methType.getLLVMType().as_pointer() + elements.append(funcType) + return ir.LiteralStructType(elements) + def getClassStructType(self, cls: str) -> ir.LiteralStructType: - elements = [voidptr_t] # pointer to vtable + elements = [self.getClassVtableType( + cls).as_pointer()] # pointer to vtable attrs = self.ts.getOrderedAttrs(cls) for attrInfo in attrs: elements.append(attrInfo[1].getLLVMType()) @@ -157,12 +174,12 @@ def Program(self, node: Program): self.visit(m) self.currentClass = None - # provide default __init__ impl for classes for cls in self.methods: + # provide default __init__ impl for classes ctor = self.methods[cls]["__init__"] if len(ctor.blocks) == 0: bb = ctor.append_basic_block('entry') - ir.IRBuilder(bb).ret_void() + ir.IRBuilder(bb).ret(voidptr_t(None)) # define functions for d in funcDefs: @@ -239,7 +256,8 @@ def declareFunc(self, node: FuncDef): def FuncDef(self, node: FuncDef): if node.isMethod: - func = self.module.get_global(self.curentClass + "__" + node.getIdentifier().name) + func = self.module.get_global( + self.currentClass + "__" + node.getIdentifier().name) else: func = self.module.get_global(node.getIdentifier().name) self.returnType = node.type.returnType @@ -267,19 +285,20 @@ def FuncDef(self, node: FuncDef): # STATEMENTS - def getAttrPtr(self, node: MemberExpr): - cls = node.object.inferredType.className - attr = node.member.name - offset = self.attrOffsets[(cls, attr)] - obj = self.visit(node.object) + def getAttrPtr(self, obj, cls: str, attr: str): + offset, _ = self.attrOffsets[cls][attr] obj = self.builder.bitcast(obj, self.structs[cls].as_pointer()) - return self.builder.gep(obj, [int32_t(offset)]) + attr_ptr = self.builder.gep(obj, [int32_t(0), int32_t(offset)]) + return attr_ptr def AssignStmt(self, node: AssignStmt): val = self.visit(node.value) for var in node.targets[::-1]: if isinstance(var, MemberExpr): - ptr = self.getAttrPtr(var) + cls = var.object.inferredType.className + attr = var.member.name + obj = self.visit(var.object) + ptr = self.getAttrPtr(obj, cls, attr) self.builder.store(val, ptr) elif isinstance(var, IndexExpr): lst = self.visit(var.list) @@ -489,7 +508,18 @@ def constructor(self, node: CallExpr): cls = node.function.name size = self.sizeof(self.structs[cls]) obj = self.builder.call(self.externs['malloc'], [size], 'new_object') - self.builder.call(self.constructors[cls + "____init__"], [obj]) + # initialize fields + for attr in self.attrOffsets[cls]: + _, val = self.attrOffsets[cls][attr] + ptr = self.getAttrPtr(obj, cls, attr) + self.builder.store(self.visit(val), ptr) + # set vtable pointer + vtable_ptr = self.builder.bitcast(obj, voidptr_t.as_pointer()) + vtable = self.module.get_global("__" + cls + "__vtable") + vtable = self.builder.bitcast(vtable, voidptr_t) + self.builder.store(vtable, vtable_ptr) + # call __init__ method + self.builder.call(self.methods[cls]["__init__"], [obj]) return obj def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): @@ -653,8 +683,10 @@ def Identifier(self, node: Identifier): return self.builder.load(addr, node.name) def MemberExpr(self, node: MemberExpr): - ptr = self.getAttrPtr(node) + cls = node.object.inferredType.className attr = node.member.name + obj = self.visit(node.object) + ptr = self.getAttrPtr(obj, cls, attr) return self.builder.load(ptr, attr) def IfExpr(self, node: IfExpr): @@ -698,21 +730,23 @@ def ifHelper(self, condFn, thenFn, elseFn=None, returnType=None): return phi def MethodCallExpr(self, node: MethodCallExpr): - obj = self.visit(node.method.object) className = node.method.object.inferredType.className + obj = self.visit(node.method.object) + obj = self.builder.bitcast(obj, self.structs[className].as_pointer()) + methName = node.method.member.name - methIdx = self.methodOffsets[(className, methName)][0] - # TODO - callee_func = None - if callee_func is None or not isinstance(callee_func, ir.Function): - raise Exception("unknown method") - if len(callee_func.args) != len(node.args) + 1: - raise Exception('Call argument length mismatch', - node.function.name) - call_args = [obj] + methIdx, _ = self.methodOffsets[(className, methName)] + + vtable_ptr = self.builder.gep(obj, [int32_t(0), int32_t(0)]) + vtable = self.builder.load(vtable_ptr) + + callee_func_ptr = self.builder.gep(self.builder.gep( + vtable, [int32_t(0), int32_t(methIdx)]), [int32_t(0)]) + callee_func = self.builder.load(callee_func_ptr) + + call_args = [self.builder.bitcast(obj, voidptr_t)] for i in range(len(node.args)): - call_args.append(self.visitArg( - node.function.inferredType, i, node.args[i])) + call_args.append(self.visitArg(node.method.inferredType, i, node.args[i])) return self.builder.call(callee_func, call_args, 'callmethodtmp') # LITERALS @@ -815,7 +849,7 @@ def global_variable(self, name, t): return data def sizeof(self, t): - if not t.is_pointer: + if not (t.is_pointer or isinstance(t, ir.LiteralStructType)): width = t.width # each item in array must be at least 1 byte if width < 8: diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 08f49df..0746709 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -93,7 +93,7 @@ def toJSON(self, dump_location=True) -> dict: "returnType": self.returnType.toJSON(dump_location) } - def getLLVMType(self) -> ir.Type: + def getLLVMType(self) -> ir.FunctionType: params = [] for i in range(len(self.parameters)): p = self.parameters[i] diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index e22ee14..7d0b810 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -1,4 +1,5 @@ from .symboltype import SymbolType +from llvmlite import ir class ValueType(SymbolType): @@ -20,5 +21,5 @@ def isJavaRef(self) -> bool: def isListType(self) -> bool: raise Exception("unsupported") - def getLLVMType(self): + def getLLVMType(self) -> ir.Type: raise Exception("unsupported") diff --git a/foobar.py b/foobar.py index e69de29..4cd8f65 100644 --- a/foobar.py +++ b/foobar.py @@ -0,0 +1,59 @@ +class A: + y: int = 1 + + def __init__(self: A): + pass + + def t(self: A): + global x + x = 1 + + +class B(A): + z: int = 0 + + def __init__(self: B): + self.z = 5 + self.y = 5 + + def t(self: B): + global x + x = 2 + + def setZ(self: B, z: int): + self.z = z + + +x: int = 0 +c1: A = None +c2: B = None +c3: A = None + +# constructors, getters, setters +c1 = A() +assert c1.y == 1 +c2 = B() +assert c2.y == 5 +assert c2.z == 5 +c3 = B() +assert c3.y == 5 + +c2.y = 0 +assert c2.y == 0 + +# methods, dynamic dispatch + +c2.setZ(2) +assert c2.z == 2 + +x = 0 +c1.t() +assert x == 1 + +x = 0 +c2.t() +assert x == 2 + +x = 0 +c3.t() +assert x == 2 diff --git a/test.py b/test.py index 1ffc3c0..84c43ec 100644 --- a/test.py +++ b/test.py @@ -9,22 +9,57 @@ from compiler.compiler import Compiler import llvmlite.binding as llvm from ctypes import CFUNCTYPE +from typing import List dump_location = True error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected", "failed"} +disabled_llvm_tests = [ + "/binary_tree.", + "/doubling_vector.", + "/nonlocal.", + "/exponent.", + "/modulo." +] + +disabled_jvm_tests = [ + "short_circuit", + "modulo" +] + +disabled_cil_tests = [ + "short_circuit", + "modulo" +] + +disabled_wasm_tests = [ + "short_circuit", + "modulo" +] + + +def should_skip(disabled_tests: List[str], test: Path) -> bool: + skip = False + for disabled in disabled_tests: + if disabled in str(test): + skip = True + print("Skipping " + str(test)) + break + return skip + + def run_all_tests(): - # run_parse_tests() - # run_typecheck_tests() - # run_python_backend_tests() - # run_closure_tests() - # run_jvm_tests() - # run_cil_tests() - # run_wasm_tests() - # run_llvm_tests() - test_eval_llvm() + run_parse_tests() + run_typecheck_tests() + run_python_backend_tests() + run_closure_tests() + run_jvm_tests() + run_cil_tests() + run_wasm_tests() + run_llvm_tests() + # test_eval_llvm() def run_parse_tests(): @@ -173,6 +208,8 @@ def run_wasm_tests(): n_passed = 0 wasm_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() for test in wasm_tests_dir.glob('*.py'): + if should_skip(disabled_wasm_tests, test): + continue passed = run_wasm_test(test) total += 1 if not passed: @@ -195,6 +232,8 @@ def run_jvm_tests(): n_passed = 0 jvm_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() for test in jvm_tests_dir.glob('*.py'): + if should_skip(disabled_jvm_tests, test): + continue passed = run_jvm_test(test) total += 1 if not passed: @@ -217,6 +256,8 @@ def run_cil_tests(): n_passed = 0 cil_tests_dir = (Path(__file__).parent / "tests/runtime/").resolve() for test in cil_tests_dir.glob('*.py'): + if should_skip(disabled_cil_tests, test): + continue passed = run_cil_test(test) total += 1 if not passed: @@ -576,21 +617,6 @@ def ast_equals(d1, d2) -> bool: return d1 == d2 -disabled_llvm_tests = [ - "/incrementing_counter.", - "/binary_tree.", - "/classes.", - "/doubling_vector.", - "/nonlocal.", - "/ratio.", - "/inherit_init.", - "/linked_list.", - "/exponent.", - "/short_circuit.", - "modulo" -] - - def run_llvm_tests(): print("Running LLVM backend tests...\n") total = 0 @@ -603,7 +629,7 @@ def run_llvm_tests(): skip = True break if skip: - # print("Skipping: " + str(test) + "\n") + print("Skipping: " + str(test) + "\n") continue passed = run_llvm_test(test, False) total += 1 From eb07a6cfe42f3e0bd1e0a1e396c57ec11b52b4f7 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 12 Jun 2023 22:16:20 -0700 Subject: [PATCH 66/79] fix mod and short circuit in java --- compiler/jvm_backend.py | 30 ++++++++++++++++++++++-------- test.py | 5 +---- tests/runtime/modulo.py | 1 - 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index 1eb00e7..ca83cb1 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -360,8 +360,10 @@ def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType rightType = node.right.inferredType - self.visit(node.left) - self.visit(node.right) + shortCircuitOps = {"and", "or"} + if operator not in shortCircuitOps: + self.visit(node.left) + self.visit(node.right) # concatenation and addition if operator == "+": if self.isListConcat(operator, leftType, rightType): @@ -412,7 +414,7 @@ def BinaryExpr(self, node: BinaryExpr): elif operator == "//": self.instr("invokestatic Method java/lang/Math floorDiv (II)I") elif operator == "%": - self.instr("irem") + self.instr("invokestatic Method java/lang/Math floorMod (II)I") # relational operators elif operator == "<": self.comparator("if_icmplt") @@ -439,9 +441,15 @@ def BinaryExpr(self, node: BinaryExpr): self.comparator("if_acmpeq") # logical operators elif operator == "and": - self.instr("iand") + condFn = lambda: self.visit(node.left) + thenFn = lambda: self.visit(node.right) + elseFn = lambda: self.instr("iconst_0") + self.ternary(condFn, thenFn, elseFn) elif operator == "or": - self.instr("ior") + condFn = lambda: self.visit(node.left) + thenFn = lambda: self.instr("iconst_1") + elseFn = lambda: self.visit(node.right) + self.ternary(condFn, thenFn, elseFn) else: raise Exception( f"Internal compiler error: unexpected operator {operator}") @@ -597,14 +605,20 @@ def MemberExpr(self, node: MemberExpr): f"getfield Field {node.object.inferredType.className} {node.member.name} {node.inferredType.getJavaSignature()}") def IfExpr(self, node: IfExpr): - self.visit(node.condition) + condFn = lambda: self.visit(node.condition) + thenFn = lambda: self.visit(node.thenExpr) + elseFn = lambda: self.visit(node.elseExpr) + self.ternary(condFn, thenFn, elseFn) + + def ternary(self, condFn, thenFn, elseFn): + condFn() l1 = self.newLabelName() l2 = self.newLabelName() self.instr(f"ifne {l1}") - self.visit(node.elseExpr) + elseFn() self.instr(f"goto {l2}") self.label(l1) - self.visit(node.thenExpr) + thenFn() self.label(l2) self.instr("nop") diff --git a/test.py b/test.py index 84c43ec..4d8e7f8 100644 --- a/test.py +++ b/test.py @@ -24,10 +24,7 @@ "/modulo." ] -disabled_jvm_tests = [ - "short_circuit", - "modulo" -] +disabled_jvm_tests = [] disabled_cil_tests = [ "short_circuit", diff --git a/tests/runtime/modulo.py b/tests/runtime/modulo.py index 81440f1..1e6b6f4 100644 --- a/tests/runtime/modulo.py +++ b/tests/runtime/modulo.py @@ -1,4 +1,3 @@ -# TODO: fix modulo operator behavior assert -5 % 2 == 1 assert 5 % -2 == -1 assert -5 % -2 == -1 From 3821553ebd5be73f22e1b79aa4514a78f29327bd Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 12 Jun 2023 22:28:01 -0700 Subject: [PATCH 67/79] fix modulo and short circuit for cil --- compiler/cil_backend.py | 37 ++++++++++++++++++++++++++++++------- test.py | 5 +---- 2 files changed, 31 insertions(+), 11 deletions(-) diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 301e37d..b2c64cb 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -348,8 +348,10 @@ def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType rightType = node.right.inferredType - self.visit(node.left) - self.visit(node.right) + shortCircuitOperators = {"and", "or"} + if operator not in shortCircuitOperators: + self.visit(node.left) + self.visit(node.right) if operator == "+": if self.isListConcat(operator, leftType, rightType): """ @@ -399,6 +401,15 @@ def BinaryExpr(self, node: BinaryExpr): elif operator == "//": self.instr("div") elif operator == "%": + b = self.newLocal(None, IntType()) + a = self.newLocal(None, IntType()) + # emulate Python modulo with ((a rem b) + b) rem b) + self.load(a) + self.load(b) + self.instr("rem") + self.load(b) + self.instr("add.ovf") + self.load(b) self.instr("rem") # relational operators elif operator == "<": @@ -427,9 +438,15 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("ceq") # logical operators elif operator == "and": - self.instr("and") + c = lambda: self.visit(node.left) + t = lambda: self.visit(node.right) + e = lambda: self.instr("ldc.i4.0") + self.ternary(c, t, e) elif operator == "or": - self.instr("or") + c = lambda: self.visit(node.left) + t = lambda: self.instr("ldc.i4.1") + e = lambda: self.visit(node.right) + self.ternary(c, t, e) else: raise Exception( f"Internal compiler error: unexpected operator {operator}") @@ -577,14 +594,20 @@ def MemberExpr(self, node: MemberExpr): f"ldfld {node.inferredType.getCILName()} {node.object.inferredType.getCILName()}::{node.member.getCILName()}") def IfExpr(self, node: IfExpr): - self.visit(node.condition) + c = lambda: self.visit(node.condition) + t = lambda: self.visit(node.thenExpr) + e = lambda: self.visit(node.elseExpr) + self.ternary(c, t, e) + + def ternary(self, condFn, thenFn, elseFn): + condFn() l1 = self.newLabelName() l2 = self.newLabelName() self.instr(f"brtrue {l1}") - self.visit(node.elseExpr) + elseFn() self.instr(f"br {l2}") self.label(l1) - self.visit(node.thenExpr) + thenFn() self.label(l2) def MethodCallExpr(self, node: MethodCallExpr): diff --git a/test.py b/test.py index 4d8e7f8..6061b08 100644 --- a/test.py +++ b/test.py @@ -26,10 +26,7 @@ disabled_jvm_tests = [] -disabled_cil_tests = [ - "short_circuit", - "modulo" -] +disabled_cil_tests = [] disabled_wasm_tests = [ "short_circuit", From 83bfa12fbf1a8d2b65fb43ebc95ec8c26cef8259 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 12 Jun 2023 22:41:16 -0700 Subject: [PATCH 68/79] fix modulo and short circuit for wasm --- compiler/astnodes/expr.py | 2 ++ compiler/astnodes/memberexpr.py | 2 ++ compiler/types/valuetype.py | 3 +++ compiler/wasm_backend.py | 47 ++++++++++++++++++++++++++------- test.py | 5 +--- 5 files changed, 46 insertions(+), 13 deletions(-) diff --git a/compiler/astnodes/expr.py b/compiler/astnodes/expr.py index 217f81c..af6a36a 100644 --- a/compiler/astnodes/expr.py +++ b/compiler/astnodes/expr.py @@ -1,8 +1,10 @@ from .node import Node from typing import List +from ..types import ValueType class Expr(Node): + inferredType: ValueType def __init__(self, location: List[int], kind: str): super().__init__(location, kind) diff --git a/compiler/astnodes/memberexpr.py b/compiler/astnodes/memberexpr.py index 487cc1a..8827dfc 100644 --- a/compiler/astnodes/memberexpr.py +++ b/compiler/astnodes/memberexpr.py @@ -1,9 +1,11 @@ from .expr import Expr from .identifier import Identifier from typing import List +from ..types import SymbolType class MemberExpr(Expr): + inferredType: SymbolType def __init__(self, location: List[int], obj: Expr, member: Identifier): super().__init__(location, "MemberExpr") diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index 7d0b810..4e06f13 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -23,3 +23,6 @@ def isListType(self) -> bool: def getLLVMType(self) -> ir.Type: raise Exception("unsupported") + + def getWasmName(self) -> str: + raise Exception("unsupported") diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 6eab413..f4ac308 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -122,9 +122,11 @@ def newLabelName(self) -> str: def instr(self, instr: str): self.builder.newLine(instr) + # set the value, consuming it from the stack def setLocal(self, name: str): self.instr(f"local.set ${name}") + # set the value and load it back onto the stack def teeLocal(self, name: str): self.instr(f"local.tee ${name}") @@ -373,8 +375,10 @@ def BinaryExpr(self, node: BinaryExpr): operator = node.operator leftType = node.left.inferredType rightType = node.right.inferredType - self.visit(node.left) - self.visit(node.right) + shortCircuitOperators = {"and", "or"} + if operator not in shortCircuitOperators: + self.visit(node.left) + self.visit(node.right) # concatenation and addition if operator == "+": if self.isListConcat(operator, leftType, rightType): @@ -394,6 +398,17 @@ def BinaryExpr(self, node: BinaryExpr): elif operator == "//": self.instr("i64.div_s") elif operator == "%": + a = self.newLocal(None, IntType().getWasmName()) + b = self.newLocal(None, IntType().getWasmName()) + self.setLocal(b) + self.setLocal(a) + # emulate Python modulo with ((a rem b) + b) rem b) + self.getLocal(a) + self.getLocal(b) + self.instr("i64.rem_s") + self.getLocal(b) + self.instr("i64.add") + self.getLocal(b) self.instr("i64.rem_s") # relational operators elif operator == "<": @@ -426,9 +441,17 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("i32.eq") # logical operators elif operator == "and": - self.instr("i32.and") + c = lambda: self.visit(node.left) + t = lambda: self.visit(node.right) + e = lambda: self.instr("i32.const 0") + resultType = BoolType().getWasmName() + self.ternary(c, t, e, resultType) elif operator == "or": - self.instr("i32.or") + c = lambda: self.visit(node.left) + t = lambda: self.instr("i32.const 1") + e = lambda: self.visit(node.right) + resultType = BoolType().getWasmName() + self.ternary(c, t, e, resultType) else: raise Exception( f"Internal compiler error: unexpected operator {operator}") @@ -622,16 +645,22 @@ def Identifier(self, node: Identifier): self.instr(f"local.get ${node.name}") def IfExpr(self, node: IfExpr): - n = self.newLocal(self.genLocalName("ifexpr_result"), - node.inferredType.getWasmName()) - self.visit(node.condition) + c = lambda: self.visit(node.condition) + t = lambda: self.visit(node.thenExpr) + e = lambda: self.visit(node.elseExpr) + resultType = node.inferredType.getWasmName() + self.ternary(c, t, e, resultType) + + def ternary(self, condFn, thenFn, elseFn, resultType): + n = self.newLocal(self.genLocalName("ifexpr_result"), resultType) + condFn() self.builder._if() self.builder._then() - self.visit(node.thenExpr) + thenFn() self.setLocal(n) self.builder.end() self.builder._else() - self.visit(node.elseExpr) + elseFn() self.setLocal(n) self.builder.end() self.builder.end() diff --git a/test.py b/test.py index 6061b08..9f17b1f 100644 --- a/test.py +++ b/test.py @@ -28,10 +28,7 @@ disabled_cil_tests = [] -disabled_wasm_tests = [ - "short_circuit", - "modulo" -] +disabled_wasm_tests = [] def should_skip(disabled_tests: List[str], test: Path) -> bool: From b9a806793892be0398d305028d5891c85ed246be Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 12 Jun 2023 23:05:18 -0700 Subject: [PATCH 69/79] fix modulo for llvm --- compiler/llvm_backend.py | 5 ++++- test.py | 3 +-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index ed759a1..c6a1d96 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -399,7 +399,10 @@ def BinaryExpr(self, node: BinaryExpr): elif operator == "//": return self.builder.sdiv(lhs, rhs) elif operator == "%": - return self.builder.srem(lhs, rhs) + # emulate Python modulo with ((a % b) + b) % b) + val = self.builder.srem(lhs, rhs) + val = self.builder.add(val, rhs) + return self.builder.srem(val, rhs) # relational operators elif operator in {"<", "<=", ">", ">="}: return self.builder.icmp_signed(operator, lhs, rhs) diff --git a/test.py b/test.py index 9f17b1f..3a706e6 100644 --- a/test.py +++ b/test.py @@ -20,8 +20,7 @@ "/binary_tree.", "/doubling_vector.", "/nonlocal.", - "/exponent.", - "/modulo." + "/exponent." ] disabled_jvm_tests = [] From 0e3a5280bdc3dcc1678b88f4cfb4ca48b7545e2a Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 19 Jun 2023 18:46:20 -0700 Subject: [PATCH 70/79] binary tree llvm - consistent block termination and implicit returns --- compiler/llvm_backend.py | 51 ++++++++------ foobar.py | 146 +++++++++++++++++++++++---------------- test.py | 27 ++++---- 3 files changed, 129 insertions(+), 95 deletions(-) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index c6a1d96..07b3a94 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -20,7 +20,6 @@ class LlvmBackend(Visitor): locals: List[defaultdict] externs: Dict[str, ir.Function] - constructors: Dict[str, ir.Function] methods: Dict[str, Dict[str, ir.Function]] structs: Dict[str, ir.LiteralStructType] # (class name, method name) -> idx in vtable, type @@ -58,14 +57,16 @@ def initializeOffsets(self): self.methods[cls] = {} orderedMethods = self.ts.getOrderedMethods(cls) vtable = [] - for idx, (methName, methType, _) in enumerate(orderedMethods): + for idx, (methName, methType, defCls) in enumerate(orderedMethods): funcType = methType.getLLVMType() self.methodOffsets[(cls, methName)] = (idx, funcType) - func = ir.Function(self.module, funcType, - cls + "__" + methName) + if defCls == cls: + func = ir.Function(self.module, funcType, + cls + "__" + methName) + self.methods[cls][methName] = func + for methName, _, defCls in orderedMethods: + func = self.methods[defCls][methName] self.methods[cls][methName] = func - for methName, _, _ in orderedMethods: - func = self.methods[cls][methName] vtable.append(func) t = self.getClassVtableType(cls) self.global_constant('__' + cls + '__vtable', @@ -220,8 +221,8 @@ def Program(self, node: Program): self.visitStmtList(node.statements) self.builder.branch(end_program) - program_block = self.builder.block self.builder.position_at_start(end_program) + assert not end_program.is_terminated self.builder.ret_void() self.exitScope() @@ -255,13 +256,14 @@ def declareFunc(self, node: FuncDef): ir.Function(self.module, funcType, funcname) def FuncDef(self, node: FuncDef): + fname = node.getIdentifier().name if node.isMethod: func = self.module.get_global( - self.currentClass + "__" + node.getIdentifier().name) + self.currentClass + "__" + fname) else: - func = self.module.get_global(node.getIdentifier().name) + func = self.module.get_global(fname) self.returnType = node.type.returnType - shouldReturnValue = not self.returnType.isNone() + implicitReturn = self.returnType not in {IntType(), BoolType(), StrType(), NoneType()} self.enterScope() bb_entry = func.append_basic_block('entry') self.builder = ir.IRBuilder(bb_entry) @@ -274,12 +276,14 @@ def FuncDef(self, node: FuncDef): for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) - # implicitly return None if possible - if shouldReturnValue is not None and ( - len(node.statements) == 0 or - not isinstance(node.statements[-1], ReturnStmt) - ): - self.builder.ret(self.NoneLiteral(None)) + # implicitly return None if needed, close all blocks + for block in func.blocks: + self.builder.position_at_end(block) + if not block.is_terminated: + if implicitReturn: + self.builder.ret(self.NoneLiteral(None)) + else: + self.builder.unreachable() self.exitScope() return func @@ -536,10 +540,13 @@ def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): # unwrap if necessary, re-wrap saved_block = self.builder.block val = self.visit(arg) + # print(val) addr = self.builder.alloca( - node.var.t.getLLVMType()) + arg.inferredType.getLLVMType()) + # print(addr) wrapper = self.builder.alloca( - node.var.t.getLLVMType().as_pointer(), None, "wrapper") + arg.inferredType.getLLVMType().as_pointer(), None, "wrapper") + # print(wrapper) self.builder.position_at_end(saved_block) self.builder.store(val, addr) self.builder.store(addr, wrapper) @@ -659,6 +666,7 @@ def whileHelper(self, condFn, bodyFn): self.builder.position_at_start(end_block) def ReturnStmt(self, node: ReturnStmt): + assert not self.builder.block.is_terminated if self.returnType.isNone(): self.builder.ret(self.NoneLiteral(None)) else: @@ -713,14 +721,14 @@ def ifHelper(self, condFn, thenFn, elseFn=None, returnType=None): self.builder.position_at_start(then_block) then_val = thenFn() - if not self.builder.block.is_terminated: + if not then_block.is_terminated: self.builder.branch(merge_block) then_block = self.builder.block if elseFn is not None: self.builder.position_at_start(else_block) else_val = elseFn() - if not self.builder.block.is_terminated: + if not else_block.is_terminated: self.builder.branch(merge_block) else_block = self.builder.block @@ -749,7 +757,8 @@ def MethodCallExpr(self, node: MethodCallExpr): call_args = [self.builder.bitcast(obj, voidptr_t)] for i in range(len(node.args)): - call_args.append(self.visitArg(node.method.inferredType, i, node.args[i])) + call_args.append(self.visitArg( + node.method.inferredType, i, node.args[i])) return self.builder.call(callee_func, call_args, 'callmethodtmp') # LITERALS diff --git a/foobar.py b/foobar.py index 4cd8f65..91ccc78 100644 --- a/foobar.py +++ b/foobar.py @@ -1,59 +1,87 @@ -class A: - y: int = 1 - - def __init__(self: A): - pass - - def t(self: A): - global x - x = 1 - - -class B(A): - z: int = 0 - - def __init__(self: B): - self.z = 5 - self.y = 5 - - def t(self: B): - global x - x = 2 - - def setZ(self: B, z: int): - self.z = z - - -x: int = 0 -c1: A = None -c2: B = None -c3: A = None - -# constructors, getters, setters -c1 = A() -assert c1.y == 1 -c2 = B() -assert c2.y == 5 -assert c2.z == 5 -c3 = B() -assert c3.y == 5 - -c2.y = 0 -assert c2.y == 0 - -# methods, dynamic dispatch - -c2.setZ(2) -assert c2.z == 2 - -x = 0 -c1.t() -assert x == 1 - -x = 0 -c2.t() -assert x == 2 - -x = 0 -c3.t() -assert x == 2 +# Binary-search trees +class TreeNode(object): + value: int = 0 + left: "TreeNode" = None + right: "TreeNode" = None + + def insert(self: "TreeNode", x: int) -> bool: + if x < self.value: + if self.left is None: + self.left = makeNode(x) + return True + else: + return self.left.insert(x) + elif x > self.value: + if self.right is None: + self.right = makeNode(x) + return True + else: + return self.right.insert(x) + return False + + def contains(self: "TreeNode", x: int) -> bool: + if x < self.value: + if self.left is None: + return False + else: + return self.left.contains(x) + elif x > self.value: + if self.right is None: + return False + else: + return self.right.contains(x) + else: + return True + + +class Tree(object): + root: TreeNode = None + size: int = 0 + + def insert(self: "Tree", x: int) -> object: + if self.root is None: + self.root = makeNode(x) + self.size = 1 + else: + if self.root.insert(x): + self.size = self.size + 1 + + def contains(self: "Tree", x: int) -> bool: + if self.root is None: + return False + else: + return self.root.contains(x) + + +def makeNode(x: int) -> TreeNode: + b: TreeNode = None + b = TreeNode() + b.value = x + return b + + +# Input parameters +n: int = 100 +c: int = 4 + +# Data +t: Tree = None +i: int = 0 +k: int = 37813 + +# Crunch +t = Tree() +while i < n: + t.insert(k) + k = (k * 37813) % 37831 + if i % c != 0: + t.insert(i) + i = i + 1 + +assert t.size == 175 +assert t.contains(15) +assert t.contains(23) +assert t.contains(42) +assert not t.contains(4) +assert not t.contains(8) +assert not t.contains(16) diff --git a/test.py b/test.py index 3a706e6..9964139 100644 --- a/test.py +++ b/test.py @@ -17,10 +17,7 @@ disabled_llvm_tests = [ - "/binary_tree.", - "/doubling_vector.", "/nonlocal.", - "/exponent." ] disabled_jvm_tests = [] @@ -41,15 +38,15 @@ def should_skip(disabled_tests: List[str], test: Path) -> bool: def run_all_tests(): - run_parse_tests() - run_typecheck_tests() - run_python_backend_tests() - run_closure_tests() - run_jvm_tests() - run_cil_tests() - run_wasm_tests() - run_llvm_tests() - # test_eval_llvm() + # run_parse_tests() + # run_typecheck_tests() + # run_python_backend_tests() + # run_closure_tests() + # run_jvm_tests() + # run_cil_tests() + # run_wasm_tests() + # run_llvm_tests() + test_eval_llvm() def run_parse_tests(): @@ -647,7 +644,7 @@ def eval_llvm(module): def test_eval_llvm(): - run_llvm_test("foobar.py", True) + run_llvm_test("foobar.py", "foobar.ll") def run_llvm_test(test, debug): @@ -665,8 +662,8 @@ def run_llvm_test(test, debug): assert len(compiler.typechecker.errors) == 0 module = compiler.emitLLVM(chocopy_ast) if debug: - print("Module output:") - print(str(module)) + with open(debug, "w") as f: + f.write(str(module)) eval_llvm(module) return True except Exception as e: From 158406e6a193f156fc1ad954ea2678b1fa511ef0 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 19 Jun 2023 22:57:21 -0700 Subject: [PATCH 71/79] fix llvm --- Makefile | 3 +- README.md | 39 ++++++++ compiler/compiler.py | 16 ++-- compiler/llvm_backend.py | 172 +++++++++++++++--------------------- compiler/types/valuetype.py | 2 +- demo_llvm.sh | 10 +++ foobar.py | 87 ------------------ main.py | 22 ++--- test.py | 126 +++++++++----------------- 9 files changed, 188 insertions(+), 289 deletions(-) create mode 100755 demo_llvm.sh delete mode 100644 foobar.py diff --git a/Makefile b/Makefile index 709a3c3..86cf09a 100644 --- a/Makefile +++ b/Makefile @@ -9,4 +9,5 @@ clean: rm -f *.out.py rm -f *.wasm rm -f *.wat - rm -f *.ll \ No newline at end of file + rm -f *.ll + rm -f *.s diff --git a/README.md b/README.md index 6bde08b..6e4938c 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ Progress is documented on my [blog](https://yangdanny97.github.io/blog/): - [Part 2: JVM backend](https://yangdanny97.github.io/blog/2021/08/26/chocopy-jvm-backend) - [Part 3: CIL backend](https://yangdanny97.github.io/blog/2022/05/22/chocopy-cil-backend) - [Part 4: WASM backend](https://yangdanny97.github.io/blog/2022/10/11/chocopy-wasm-backend) +- [Part 5: LLVM backend - coming soon!](https://yangdanny97.github.io/blog) ## Features @@ -21,6 +22,7 @@ This compiler contains multiple backends not found in the reference implementati - JVM bytecode, formatted for the Krakatau assembler - CIL bytecode, formatted for the Mono ilasm assembler - WASM, in WAT format +- LLVM IR, in text format The test suite includes both static validation of generated/annotated ASTs, as well as runtime tests that actually execute the output programs to check correctness. Many of the AST validation test cases are taken from test suites included in the release code for Berkeley's CS164, with some additional tests written for more coverage. @@ -35,6 +37,10 @@ The test suite includes both static validation of generated/annotated ASTs, as w - WASM Backend Requirements: - [WebAssembly Binary Toolkit (wabt)](https://github.com/WebAssembly/wabt), specifically the `wat2wasm` tool - NodeJS for the runtime +- LLVM Backend Requirements + - LLVM toolchain + - `llvmlite` + - Tested with LLVM 16.0.6 ## Usage @@ -61,6 +67,7 @@ The input file should have extension `.py`. If the output file is not provided, - `jvm` - output JVM bytecode formatted for the Krakatau assembler - `cil` - output CIL bytecode formatted for the Mono ilasm assembler - `wasm` - output WASM as plaintext in WAT format + - `llvm` - output LLVM IR in text format ## Differences from the reference implementation: @@ -145,6 +152,38 @@ Strings, lists, objects, and refs holding nonlocals are stored in the heap, alig To provide memory safety, string/list indexing have bounds checking and list operations have a null-check, which crashes the program with a generic "unreachable" instruction. +## LLVM Backend Notes: + +The LLVM backend for this compiler outputs LLVM IR in plaintext `.ll` format which can be compiled using `llc` or interpreted using `lli`: +1. Use this compiler to generate plaintext LLVM IR + - Format: `python3 main.py --mode llvm ` + - Example: `python3 main.py --mode llvm tests/runtime/binary_tree.py .` +2. Run the `.ll` files using `lli` + - Example: `lli <.ll file>` + - Example: `lli binary_tree.ll` + +The `demo_llvm.sh` script is a useful utility to compile and run files with the LLVM backend with a single command (provide the path to the input source file as an argument). +- To run the same example as above, run `./demo_llvm.sh tests/runtime/binary_tree.py` + +Generated programs should only depend on the C standard library, so there's no custom runtime to link to. + +### LLVM Backend - Unsupported Features: +- `input` stdlib function - TODO + +### LLVM Backend - Memory Format, Safety, and Management: + +- strings - null-terminated `char*`, same as C strings +- lists - first 4 bytes for length, followed by the contents as a packed array +- ints - 32 bits +- pointers (objects, strings, lists) - same as C pointers, where `None` is the null pointer +- objects - struct containing vtable address followed by attributes + +Memory does not get freed/garbage collected once it is allocated, so large programs may run out of memory. + +To provide some memory safety, string/list indexing have bounds checking and list operations have a null-check, which exits the program with a generic error message and line number. + +Error handling is done using the `setjmp`/`longjmp` strategy, with the line of the error/assertion used as the argument for `longjmp`. + ## FAQ - What is this for? diff --git a/compiler/compiler.py b/compiler/compiler.py index 7450283..64934f0 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -24,7 +24,7 @@ def __init__(self): self.parser = Parser() self.typechecker = TypeChecker(self.ts) - def parse(self, infile) -> Node: + def parse(self, infile) -> Program: astparser = self.parser # given an input file, parse it into an AST object lines = None @@ -46,46 +46,46 @@ def parse(self, infile) -> Node: astparser.errors.append(ParseError(message)) return None - def closurepass(self, ast: Node): + def closurepass(self, ast: Program): ClosureVisitor().visit(ast) NestedFuncHoister().visit(ast) self.transformer = ClosureTransformer() self.transformer.visit(ast) return ast - def typecheck(self, ast: Node): + def typecheck(self, ast: Program): # given an AST object, typecheck it # typechecking mutates the AST, adding types and errors self.typechecker.visit(ast) return ast - def emitPython(self, ast: Node): + def emitPython(self, ast: Program): backend = PythonBackend() backend.visit(ast) return backend.builder - def emitJVM(self, main: str, ast: Node): + def emitJVM(self, main: str, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) jvm_backend = JvmBackend(main, self.transformer.ts) jvm_backend.visit(ast) return jvm_backend.classes - def emitCIL(self, main: str, ast: Node): + def emitCIL(self, main: str, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) cil_backend = CilBackend(main, self.transformer.ts) cil_backend.visit(ast) return cil_backend.builder - def emitWASM(self, main: str, ast: Node): + def emitWASM(self, main: str, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) wasm_backend = WasmBackend(main, self.transformer.ts) wasm_backend.visit(ast) return wasm_backend.builder - def emitLLVM(self, ast: Node): + def emitLLVM(self, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) llvm_backend = LlvmBackend(self.transformer.ts) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 07b3a94..e075879 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -34,6 +34,7 @@ def __init__(self, ts: TypeSystem): llvm.initialize_native_target() llvm.initialize_native_asmprinter() self.module = ir.Module() + self.module.triple = llvm.get_process_triple() self.ts = ts self.locals = [] self.externs = {} @@ -188,7 +189,7 @@ def Program(self, node: Program): # main function funcType = ir.FunctionType(ir.VoidType(), []) - func = ir.Function(self.module, funcType, "__main__") + func = ir.Function(self.module, funcType, "main") self.enterScope() entry_block = func.append_basic_block('entry') @@ -224,6 +225,12 @@ def Program(self, node: Program): self.builder.position_at_start(end_program) assert not end_program.is_terminated self.builder.ret_void() + + for block in func.blocks: + self.builder.position_at_end(block) + if not block.is_terminated: + self.builder.unreachable() + self.exitScope() def VarDef(self, node: VarDef): @@ -263,7 +270,8 @@ def FuncDef(self, node: FuncDef): else: func = self.module.get_global(fname) self.returnType = node.type.returnType - implicitReturn = self.returnType not in {IntType(), BoolType(), StrType(), NoneType()} + implicitReturn = self.returnType not in { + IntType(), BoolType(), StrType()} self.enterScope() bb_entry = func.append_basic_block('entry') self.builder = ir.IRBuilder(bb_entry) @@ -318,12 +326,16 @@ def AssignStmt(self, node: AssignStmt): raise Exception("Illegal assignment") def IfStmt(self, node: IfStmt): + cond = self.visit(node.condition) if len(node.elseBody) == 0: - self.ifHelper(lambda: self.visit(node.condition), lambda: self.visitStmtList( - node.thenBody)) + with self.builder.if_then(cond): + self.visitStmtList(node.thenBody) else: - self.ifHelper(lambda: self.visit(node.condition), lambda: self.visitStmtList( - node.thenBody), lambda: self.visitStmtList(node.elseBody)) + with self.builder.if_else(cond) as (then, else_): + with then: + self.visitStmtList(node.thenBody) + with else_: + self.visitStmtList(node.elseBody) def ExprStmt(self, node: ExprStmt): self.visit(node.expr) @@ -430,10 +442,9 @@ def BinaryExpr(self, node: BinaryExpr): return self.builder.icmp_signed(operator, lhs, rhs) elif operator == "is": # pointer comparisons - return self.builder.icmp_unsigned("==", - self.builder.ptrtoint( - lhs, int32_t), - self.builder.ptrtoint(rhs, int32_t)) + lhs_ptr = self.builder.ptrtoint(lhs, int32_t) + rhs_ptr = self.builder.ptrtoint(rhs, int32_t) + return self.builder.icmp_unsigned("==", lhs_ptr, rhs_ptr) # logical operators elif operator == "and": return self.builder.and_(lhs, rhs) @@ -459,19 +470,14 @@ def IndexExpr(self, node: IndexExpr): def listIndex(self, list, index, elemType, check_bounds=False, line: int = 0): # return pointer to list[index] - length = self.list_len(list) if check_bounds: - self.ifHelper( - lambda: self.builder.icmp_signed( - '>', int32_t(0), index), - lambda: self.longJmp(line) - ) - self.ifHelper( - lambda: self.builder.icmp_signed('<=', - length, - index), - lambda: self.longJmp(line) - ) + min_idx = self.builder.icmp_signed('>', int32_t(0), index) + with self.builder.if_then(min_idx): + self.longJmp(line) + length = self.list_len(list) + max_idx = self.builder.icmp_signed('<=', length, index) + with self.builder.if_then(max_idx): + self.longJmp(line) data = self.getListDataPtr(list, elemType) # return pointer to value in array return self.builder.gep(data, [index]) @@ -480,18 +486,13 @@ def strIndex(self, string, index, check_bounds=False, line: int = 0): string = self.toVoidPtr(string) # bounds checks if check_bounds: - self.ifHelper( - lambda: self.builder.icmp_signed( - '>', int32_t(0), index), - lambda: self.longJmp(line) - ) - self.ifHelper( - lambda: self.builder.icmp_signed('<=', - self.builder.call( - self.externs['strlen'], [string]), - index), - lambda: self.longJmp(line) - ) + min_idx = self.builder.icmp_signed('>', int32_t(0), index) + with self.builder.if_then(min_idx): + self.longJmp(line) + length = self.builder.call(self.externs['strlen'], [string]) + max_idx = self.builder.icmp_signed('<=', length, index) + with self.builder.if_then(max_idx): + self.longJmp(line) ptr = self.builder.gep(string, [index]) char = self.builder.load(ptr) addr = self.builder.call(self.externs['malloc'], [ @@ -540,17 +541,10 @@ def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): # unwrap if necessary, re-wrap saved_block = self.builder.block val = self.visit(arg) - # print(val) - addr = self.builder.alloca( - arg.inferredType.getLLVMType()) - # print(addr) - wrapper = self.builder.alloca( - arg.inferredType.getLLVMType().as_pointer(), None, "wrapper") - # print(wrapper) + addr = self.builder.alloca(arg.inferredType.getLLVMType()) self.builder.position_at_end(saved_block) self.builder.store(val, addr) - self.builder.store(addr, wrapper) - return wrapper + return addr else: # non-ref param, maybe unwrap return self.visit(arg) @@ -609,8 +603,7 @@ def forBody(self, node: ForStmt, var, idxFn, idx_var): currIdx = self.builder.load(idx_var) self.builder.store(idxFn(currIdx), var) self.visitStmtList(node.body) - self.builder.store(self.builder.add( - currIdx, int32_t(1)), idx_var) + self.builder.store(self.builder.add(currIdx, int32_t(1)), idx_var) def ListExpr(self, node: ListExpr): n = len(node.elements) @@ -701,44 +694,18 @@ def MemberExpr(self, node: MemberExpr): return self.builder.load(ptr, attr) def IfExpr(self, node: IfExpr): - return self.ifHelper(lambda: self.visit(node.condition), - lambda: self.visit(node.thenExpr), - lambda: self.visit(node.elseExpr), - node.inferredType.getLLVMType()) - - def ifHelper(self, condFn, thenFn, elseFn=None, returnType=None): - cond = condFn() - if returnType is not None: - assert elseFn is not None - - then_block = self.builder.append_basic_block('then') - if elseFn is not None: - else_block = self.builder.append_basic_block('else') - merge_block = self.builder.append_basic_block('merge') - self.builder.cbranch(cond, - then_block, - else_block if elseFn is not None else merge_block) - - self.builder.position_at_start(then_block) - then_val = thenFn() - if not then_block.is_terminated: - self.builder.branch(merge_block) - then_block = self.builder.block - - if elseFn is not None: - self.builder.position_at_start(else_block) - else_val = elseFn() - if not else_block.is_terminated: - self.builder.branch(merge_block) - else_block = self.builder.block - - self.builder.position_at_start(merge_block) - - if returnType is not None: - phi = self.builder.phi(returnType, 'phi') - phi.add_incoming(then_val, then_block) - phi.add_incoming(else_val, else_block) - return phi + cond = self.visit(node.condition) + with self.builder.if_else(cond) as (then, else_): + with then: + then_val = self.visit(node.thenExpr) + then_block = self.builder.block + with else_: + else_val = self.visit(node.elseExpr) + else_block = self.builder.block + phi = self.builder.phi(node.inferredType.getLLVMType(), 'phi') + phi.add_incoming(then_val, then_block) + phi.add_incoming(else_val, else_block) + return phi def MethodCallExpr(self, node: MethodCallExpr): className = node.method.object.inferredType.className @@ -758,7 +725,7 @@ def MethodCallExpr(self, node: MethodCallExpr): call_args = [self.builder.bitcast(obj, voidptr_t)] for i in range(len(node.args)): call_args.append(self.visitArg( - node.method.inferredType, i, node.args[i])) + node.method.inferredType, i + 1, node.args[i])) return self.builder.call(callee_func, call_args, 'callmethodtmp') # LITERALS @@ -793,11 +760,9 @@ def emit_len(self, arg: Expr): def assert_nonnull(self, val, line): val = self.toVoidPtr(val) - self.ifHelper( - lambda: self.builder.icmp_signed( - '==', voidptr_t(None), val), - lambda: self.longJmp(line) - ) + cond = self.builder.icmp_signed('==', voidptr_t(None), val) + with self.builder.if_then(cond): + self.longJmp(line) def list_len(self, arg): val = self.builder.bitcast(arg, int32_t.as_pointer()) @@ -806,27 +771,34 @@ def list_len(self, arg): def emit_assert(self, arg: Expr): line = arg.location[0] arg = self.visit(arg) - return self.ifHelper( - lambda: self.builder.icmp_unsigned( - '==', bool_t(0), arg), - lambda: self.longJmp(line) - ) + cond = self.builder.icmp_unsigned('==', bool_t(0), arg) + with self.builder.if_then(cond): + self.longJmp(line) def longJmp(self, line: int): jmp_buf = self.module.get_global('__jmp_buf') self.builder.call(self.externs['longjmp'], [ jmp_buf, int32_t(line)]) + self.builder.unreachable() def emit_print(self, arg: Expr): if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: raise Exception("Only bool, int, or str may be printed") if arg.inferredType == BoolType(): - text = self.ifHelper( - lambda: self.visit(arg), - lambda: self.toVoidPtr(self.module.get_global('__true')), - lambda: self.toVoidPtr(self.module.get_global('__false')), - voidptr_t) - self.printf(self.module.get_global('__fmt_s'), text) + cond = self.visit(arg) + with self.builder.if_else(cond) as (then, else_): + with then: + then_text = self.toVoidPtr( + self.module.get_global('__true')) + then_block = self.builder.block + with else_: + else_text = self.toVoidPtr( + self.module.get_global('__false')) + else_block = self.builder.block + phi = self.builder.phi(voidptr_t, 'phi') + phi.add_incoming(then_text, then_block) + phi.add_incoming(else_text, else_block) + self.printf(self.module.get_global('__fmt_s'), phi) elif arg.inferredType.className == 'int': self.printf(self.module.get_global('__fmt_i'), self.visit(arg)) else: diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index 4e06f13..b52ac08 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -1,5 +1,5 @@ from .symboltype import SymbolType -from llvmlite import ir +import llvmlite.ir as ir class ValueType(SymbolType): diff --git a/demo_llvm.sh b/demo_llvm.sh new file mode 100755 index 0000000..e4248c1 --- /dev/null +++ b/demo_llvm.sh @@ -0,0 +1,10 @@ +# utility for compiling a Chocopy file to LLVM IR and running it +base_name="$(basename $1 .py)" + +rm -f *.ll +rm -f *.s + +python3 main.py --mode llvm $1 . +lli $base_name.ll + + diff --git a/foobar.py b/foobar.py deleted file mode 100644 index 91ccc78..0000000 --- a/foobar.py +++ /dev/null @@ -1,87 +0,0 @@ -# Binary-search trees -class TreeNode(object): - value: int = 0 - left: "TreeNode" = None - right: "TreeNode" = None - - def insert(self: "TreeNode", x: int) -> bool: - if x < self.value: - if self.left is None: - self.left = makeNode(x) - return True - else: - return self.left.insert(x) - elif x > self.value: - if self.right is None: - self.right = makeNode(x) - return True - else: - return self.right.insert(x) - return False - - def contains(self: "TreeNode", x: int) -> bool: - if x < self.value: - if self.left is None: - return False - else: - return self.left.contains(x) - elif x > self.value: - if self.right is None: - return False - else: - return self.right.contains(x) - else: - return True - - -class Tree(object): - root: TreeNode = None - size: int = 0 - - def insert(self: "Tree", x: int) -> object: - if self.root is None: - self.root = makeNode(x) - self.size = 1 - else: - if self.root.insert(x): - self.size = self.size + 1 - - def contains(self: "Tree", x: int) -> bool: - if self.root is None: - return False - else: - return self.root.contains(x) - - -def makeNode(x: int) -> TreeNode: - b: TreeNode = None - b = TreeNode() - b.value = x - return b - - -# Input parameters -n: int = 100 -c: int = 4 - -# Data -t: Tree = None -i: int = 0 -k: int = 37813 - -# Crunch -t = Tree() -while i < n: - t.insert(k) - k = (k * 37813) % 37831 - if i % c != 0: - t.insert(i) - i = i + 1 - -assert t.size == 175 -assert t.contains(15) -assert t.contains(23) -assert t.contains(42) -assert not t.contains(4) -assert not t.contains(8) -assert not t.contains(16) diff --git a/main.py b/main.py index e699036..8f46ec5 100644 --- a/main.py +++ b/main.py @@ -70,6 +70,11 @@ def main(): outfile = outdir + infile_name + ".j" elif args.mode == "llvm": outfile = outdir + infile_name + ".ll" + elif args.mode == "cil": + outfile = outdir + infile_name + ".cil" + elif args.mode == "wasm": + outfile = outdir + infile_name + ".wat" + assert outfile is not None compiler = Compiler() astparser = compiler.parser @@ -128,27 +133,24 @@ def main(): if args.should_print: print(cil_emitter.emit()) else: - fname = outdir + cil_emitter.name + ".cil" - with open(fname, "w") as f: - out_msg(fname, args.verbose) + with open(outfile, "w") as f: + out_msg(outfile, args.verbose) f.write(cil_emitter.emit()) elif args.mode == "wasm": wat_emitter = compiler.emitWASM(infile_name, tree) if args.should_print: print(wat_emitter.emit()) else: - fname = outdir + wat_emitter.name + ".wat" - with open(fname, "w") as f: - out_msg(fname, args.verbose) + with open(outfile, "w") as f: + out_msg(outfile, args.verbose) f.write(wat_emitter.emit()) elif args.mode == "llvm": - llvm_module = compiler.emitLLVM(infile_name, tree) + llvm_module = compiler.emitLLVM(tree) if args.should_print: print(str(llvm_module)) else: - fname = outdir + llvm_module.name + ".ll" - with open(fname, "w") as f: - out_msg(fname, args.verbose) + with open(outfile, "w") as f: + out_msg(outfile, args.verbose) f.write(str(llvm_module)) diff --git a/test.py b/test.py index 9964139..65426e1 100644 --- a/test.py +++ b/test.py @@ -9,16 +9,14 @@ from compiler.compiler import Compiler import llvmlite.binding as llvm from ctypes import CFUNCTYPE -from typing import List +from typing import List, Optional dump_location = True error_flags = {"error", "Error", "Exception", "exception", "Expected", "expected", "failed"} -disabled_llvm_tests = [ - "/nonlocal.", -] +disabled_llvm_tests = [] disabled_jvm_tests = [] @@ -38,15 +36,15 @@ def should_skip(disabled_tests: List[str], test: Path) -> bool: def run_all_tests(): - # run_parse_tests() - # run_typecheck_tests() - # run_python_backend_tests() - # run_closure_tests() - # run_jvm_tests() - # run_cil_tests() - # run_wasm_tests() - # run_llvm_tests() - test_eval_llvm() + run_parse_tests() + run_typecheck_tests() + run_python_backend_tests() + run_closure_tests() + run_jvm_tests() + run_cil_tests() + run_wasm_tests() + run_llvm_tests() + # run_llvm_test("tests/runtime/nested_list.py", "debug.ll") def run_parse_tests(): @@ -315,18 +313,13 @@ def run_closure_test(test) -> bool: # for valid cases only try: compiler = Compiler() - astparser = compiler.parser - ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - tc = compiler.typechecker - compiler.typecheck(ast) - compiler.closurepass(ast) + chocopy_ast = build_and_check_ast(compiler, test) + compiler.closurepass(chocopy_ast) # clean types to get fresh typecheck - ast.visit(TypeEraser()) + chocopy_ast.visit(TypeEraser()) tc = TypeChecker(TypeSystem()) - tc.visit(ast) - if len(ast.errors.errors) > 0: + tc.visit(chocopy_ast) + if len(chocopy_ast.errors.errors) > 0: for e in ast.errors.errors: print(e.toJSON(dump_location)) return False @@ -343,11 +336,7 @@ def run_closure_runtime_test(test) -> bool: infile_name = str(test)[:-3].split("/")[-1] try: compiler = Compiler() - astparser = compiler.parser - chocopy_ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - compiler.typecheck(chocopy_ast) + chocopy_ast = build_and_check_ast(compiler, test) compiler.closurepass(chocopy_ast) builder = compiler.emitPython(chocopy_ast) name = f"./{infile_name}.test.py" @@ -376,11 +365,7 @@ def run_closure_runtime_test(test) -> bool: def run_python_emit_test(test) -> bool: try: compiler = Compiler() - astparser = compiler.parser - chocopy_ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - compiler.typecheck(chocopy_ast) + chocopy_ast = build_and_check_ast(compiler, test) builder = compiler.emitPython(chocopy_ast) ast.parse(builder.emit()) return True @@ -396,11 +381,7 @@ def run_python_runtime_test(test) -> bool: infile_name = str(test)[:-3].split("/")[-1] try: compiler = Compiler() - astparser = compiler.parser - chocopy_ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - compiler.typecheck(chocopy_ast) + chocopy_ast = build_and_check_ast(compiler, test) builder = compiler.emitPython(chocopy_ast) name = f"./{infile_name}.test.py" with open(name, "w") as f: @@ -431,15 +412,8 @@ def run_jvm_test(test) -> bool: infile_name = str(test)[:-3].split("/")[-1] outdir = "./" compiler = Compiler() - astparser = compiler.parser - ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - compiler.typecheck(ast) - if len(ast.errors.errors) > 0: - print(ast.errors.toJSON(False)) - return False - jvm_emitters = compiler.emitJVM(infile_name, ast) + chocopy_ast = build_and_check_ast(compiler, test) + jvm_emitters = compiler.emitJVM(infile_name, chocopy_ast) for cls in jvm_emitters: jvm_emitter = jvm_emitters[cls] fname = outdir + cls + ".j" @@ -479,15 +453,8 @@ def run_cil_test(test) -> bool: infile_name = name.split("/")[-1] outdir = "./" compiler = Compiler() - astparser = compiler.parser - ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - compiler.typecheck(ast) - if len(ast.errors.errors) > 0: - print(ast.errors.toJSON(False)) - return False - cil_emitter = compiler.emitCIL(infile_name, ast) + chocopy_ast = build_and_check_ast(compiler, test) + cil_emitter = compiler.emitCIL(infile_name, chocopy_ast) fname = outdir + cil_emitter.name + ".cil" with open(fname, "w") as f: f.write(cil_emitter.emit()) @@ -524,15 +491,8 @@ def run_wasm_test(test) -> bool: infile_name = name.split("/")[-1] outdir = "./" compiler = Compiler() - astparser = compiler.parser - ast = compiler.parse(test) - if len(astparser.errors) > 0: - return False - compiler.typecheck(ast) - if len(ast.errors.errors) > 0: - print(ast.errors.toJSON(False)) - return False - wasm_emitter = compiler.emitWASM(infile_name, ast) + chocopy_ast = build_and_check_ast(compiler, test) + wasm_emitter = compiler.emitWASM(infile_name, chocopy_ast) fname = outdir + name + ".wat" with open(fname, "w") as f: f.write(wasm_emitter.emit()) @@ -618,7 +578,7 @@ def run_llvm_tests(): if skip: print("Skipping: " + str(test) + "\n") continue - passed = run_llvm_test(test, False) + passed = run_llvm_test(test) total += 1 if not passed: print("Failed: " + str(test) + "\n") @@ -639,27 +599,16 @@ def eval_llvm(module): llvmmod.verify() with llvm.create_mcjit_compiler(llvmmod, target_machine) as ee: ee.finalize_object() - fptr = CFUNCTYPE(None)(ee.get_function_address("__main__")) + fptr = CFUNCTYPE(None)(ee.get_function_address("main")) fptr() -def test_eval_llvm(): - run_llvm_test("foobar.py", "foobar.ll") - - -def run_llvm_test(test, debug): - print("running test", test) +def run_llvm_test(test: str, debug: Optional[str] = None): + if debug: + print("Running test", test) try: compiler = Compiler() - astparser = compiler.parser - chocopy_ast = compiler.parse(test) - if len(astparser.errors) > 0: - print(astparser.errors) - assert len(astparser.errors) == 0 - compiler.typecheck(chocopy_ast) - if len(compiler.typechecker.errors) > 0: - print(compiler.typechecker.errors) - assert len(compiler.typechecker.errors) == 0 + chocopy_ast = build_and_check_ast(compiler, test) module = compiler.emitLLVM(chocopy_ast) if debug: with open(debug, "w") as f: @@ -672,3 +621,16 @@ def run_llvm_test(test, debug): print(e) print(track) return False + + +def build_and_check_ast(compiler: Compiler, test: str): + astparser = compiler.parser + chocopy_ast = compiler.parse(test) + if len(astparser.errors) > 0: + print(astparser.errors) + assert len(astparser.errors) == 0 + compiler.typecheck(chocopy_ast) + if len(compiler.typechecker.errors) > 0: + print(compiler.typechecker.errors) + assert len(compiler.typechecker.errors) == 0 + return chocopy_ast From 993eca9384e9e273c3096932ff858d7d08ef5a72 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Mon, 19 Jun 2023 23:57:29 -0700 Subject: [PATCH 72/79] update readme, support input and better error messages --- README.md | 19 ++++--- compiler/llvm_backend.py | 111 +++++++++++++++++++++++++++++++++------ 2 files changed, 106 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 6e4938c..83b601f 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,7 @@ The `demo_jvm.sh` script is a useful utility to compile and run files with the J Note that in the above example commands & the `demo_jvm.sh` script all expect the Krakatau directory and this repository's directory to share the same parent - commands will differ if you cloned Krakatau to a different location. -### JVM Backend - Known Issues/Incompatibilities: +### JVM Backend - Incompatibilities: - Since bytecode for each class is stored in a separate file, on operating systems with case-insensitive file names you cannot have 2 classes whose names only differ by case. - Since the main JVM class for a Chocopy program shares the name of the file, do not define other classes with the same name as the source file. @@ -119,6 +119,9 @@ The CIL backend for this compiler outputs CIL bytecode in plaintext formatted fo The `demo_cil.sh` script is a useful utility to compile and run files with the CIL backend with a single command (provide the path to the input source file as an argument). - To run the same example as above, run `./demo_cil.sh tests/runtime/binary_tree.py` +### CIL Backend - Incompatibilities +- Since the main CIL class for a Chocopy program shares the name of the file, do not define other classes with the same name as the source file. + ## WASM Backend Notes: The WASM backend for this compiler outputs WASM in plaintext `.wat` format which can be converted to `.wasm` using `wat2wasm`: @@ -137,8 +140,8 @@ The `demo_wasm.sh` script is a useful utility to compile and run files with the The `wasm.js` file contains all the runtime support needed to run the WASM generated by this compiler. This backend was designed was to minimize runtime JavaScript dependencies, so the only imported functions are for assertions and printing strings/integers/booleans. -### WASM Backend - Unsupported Features: -- `input` stdlib function (node.js does not have synchronous I/O out of the box so this is difficult) +### WASM Backend - Incompatibilities: +- `input` stdlib function is not supported (node.js does not support synchronous I/O) ### WASM Backend - Memory Format, Safety, and Management: @@ -165,10 +168,12 @@ The LLVM backend for this compiler outputs LLVM IR in plaintext `.ll` format whi The `demo_llvm.sh` script is a useful utility to compile and run files with the LLVM backend with a single command (provide the path to the input source file as an argument). - To run the same example as above, run `./demo_llvm.sh tests/runtime/binary_tree.py` -Generated programs should only depend on the C standard library, so there's no custom runtime to link to. +Generated programs should only depend on the C standard library, so there's no custom runtime to link to. -### LLVM Backend - Unsupported Features: -- `input` stdlib function - TODO +### LLVM Backend - Incompatibilities: +- `input` stdlib function - inputs are truncated to 100 characters and newlines are not permitted +- strings are required to be ASCII +- top-level statements are grouped under the `main` function - do not define any functions called `main` in your program, and do not shadow any function names from the C standard library ### LLVM Backend - Memory Format, Safety, and Management: @@ -180,7 +185,7 @@ Generated programs should only depend on the C standard library, so there's no c Memory does not get freed/garbage collected once it is allocated, so large programs may run out of memory. -To provide some memory safety, string/list indexing have bounds checking and list operations have a null-check, which exits the program with a generic error message and line number. +To provide some memory safety, string/list indexing have bounds checking and list operations have a null-check. Unlike the WASM backend, bounds checking and null checks have their own error messages and display a line number similar to assertions. Error handling is done using the `setjmp`/`longjmp` strategy, with the line of the error/assertion used as the argument for `longjmp`. diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index e075879..ca18cff 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -9,12 +9,21 @@ import llvmlite.binding as llvm JMP_BUF_BYTES = 200 +INPUT_CHARS = 100 + + +class ErrorCode: + NULL_PTR = 1 + OUT_OF_BOUNDS = 2 + ASSERT = 3 + bool_t = ir.IntType(1) # for booleans int8_t = ir.IntType(8) # chars, or booleans in arrays int32_t = ir.IntType(32) # for ints voidptr_t = ir.IntType(8).as_pointer() jmp_buf_t = ir.ArrayType(ir.IntType(8), JMP_BUF_BYTES) +input_buf_t = ir.ArrayType(int8_t, INPUT_CHARS + 1) class LlvmBackend(Visitor): @@ -119,6 +128,12 @@ def Program(self, node: Program): self.global_constant('__fmt_s', ir.ArrayType(int8_t, 4), self.make_bytearray('%s\n\00'.encode('ascii'))) + self.global_constant('__fmt_oob', + ir.ArrayType(int8_t, 26), + self.make_bytearray('Out of bounds on line %i\n\00'.encode('ascii'))) + self.global_constant('__fmt_null', + ir.ArrayType(int8_t, 25), + self.make_bytearray('Null pointer on line %i\n\00'.encode('ascii'))) self.global_constant('__fmt_assert', ir.ArrayType(int8_t, 29), self.make_bytearray('Assertion failed on line %i\n\00'.encode('ascii'))) @@ -128,8 +143,20 @@ def Program(self, node: Program): self.global_constant('__fmt_str_concat', ir.ArrayType(int8_t, 5), self.make_bytearray('%s%s\00'.encode('ascii'))) + self.global_constant('__fmt_str', + ir.ArrayType(int8_t, 3), + self.make_bytearray('%s\00'.encode('ascii'))) + self.global_constant('__fmt_input', + ir.ArrayType(int8_t, 9), + self.make_bytearray('%100[^\n]\00'.encode('ascii'))) self.global_constant( "__jmp_buf", jmp_buf_t, ir.Constant(jmp_buf_t, bytearray([0] * JMP_BUF_BYTES))) + self.global_constant( + "__input_buf", input_buf_t, + ir.Constant(input_buf_t, bytearray([0] * (INPUT_CHARS + 1)))) + + error_code = self.global_variable("__error_code", int32_t) + error_line = self.global_variable("__error_line", int32_t) printf_t = ir.FunctionType(int32_t, [voidptr_t], True) self.externs['printf'] = ir.Function(self.module, printf_t, 'printf') @@ -158,6 +185,9 @@ def Program(self, node: Program): memcpy_t = ir.FunctionType(voidptr_t, [voidptr_t, voidptr_t, int32_t]) self.externs['memcpy'] = ir.Function(self.module, memcpy_t, 'memcpy') + scanf_t = ir.FunctionType(int32_t, [voidptr_t], True) + self.externs['scanf'] = ir.Function(self.module, scanf_t, 'scanf') + # begin main function # declare global variables, methods, and functions varDefs = [d for d in node.declarations if isinstance(d, VarDef)] @@ -166,7 +196,9 @@ def Program(self, node: Program): self.global_variable(d.var.name(), t) funcDefs = [d for d in node.declarations if isinstance(d, FuncDef)] for d in funcDefs: - self.declareFunc(d) + funcname = d.name.name + funcType = d.type.getLLVMType() + ir.Function(self.module, funcType, funcname) classDefs = [d for d in node.declarations if isinstance(d, ClassDef)] for cls in classDefs: self.currentClass = cls.name.name @@ -207,7 +239,32 @@ def Program(self, node: Program): program_block) self.builder.position_at_start(error_block) - self.printf(self.module.get_global('__fmt_err'), status) + + error_code = self.builder.load(error_code) + error_line = self.builder.load(error_line) + + assert_cond = self.builder.icmp_signed( + '==', error_code, int32_t(ErrorCode.ASSERT)) + with self.builder.if_else(assert_cond) as (then_assert, else_): + with then_assert: + self.printf(self.module.get_global('__fmt_assert'), error_line) + with else_: + null_cond = self.builder.icmp_signed( + '==', error_code, int32_t(ErrorCode.NULL_PTR)) + with self.builder.if_else(null_cond) as (null_assert, else__): + with null_assert: + self.printf(self.module.get_global( + '__fmt_null'), error_line) + with else__: + oob_cond = self.builder.icmp_signed( + '==', error_code, int32_t(ErrorCode.OUT_OF_BOUNDS)) + with self.builder.if_else(oob_cond) as (oob_assert, else____): + with oob_assert: + self.printf(self.module.get_global( + '__fmt_oob'), error_line) + with else____: + self.printf(self.module.get_global( + '__fmt_err'), error_line) self.builder.branch(end_program) error_block = self.builder.block @@ -257,11 +314,6 @@ def VarDef(self, node: VarDef): def ClassDef(self, node: ClassDef): pass - def declareFunc(self, node: FuncDef): - funcname = node.name.name - funcType = node.type.getLLVMType() - ir.Function(self.module, funcType, funcname) - def FuncDef(self, node: FuncDef): fname = node.getIdentifier().name if node.isMethod: @@ -310,6 +362,7 @@ def AssignStmt(self, node: AssignStmt): cls = var.object.inferredType.className attr = var.member.name obj = self.visit(var.object) + self.assert_nonnull(obj, var.object.location[0]) ptr = self.getAttrPtr(obj, cls, attr) self.builder.store(val, ptr) elif isinstance(var, IndexExpr): @@ -471,13 +524,14 @@ def IndexExpr(self, node: IndexExpr): def listIndex(self, list, index, elemType, check_bounds=False, line: int = 0): # return pointer to list[index] if check_bounds: + assert line != 0 min_idx = self.builder.icmp_signed('>', int32_t(0), index) with self.builder.if_then(min_idx): - self.longJmp(line) + self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) length = self.list_len(list) max_idx = self.builder.icmp_signed('<=', length, index) with self.builder.if_then(max_idx): - self.longJmp(line) + self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) data = self.getListDataPtr(list, elemType) # return pointer to value in array return self.builder.gep(data, [index]) @@ -486,13 +540,14 @@ def strIndex(self, string, index, check_bounds=False, line: int = 0): string = self.toVoidPtr(string) # bounds checks if check_bounds: + assert line != 0 min_idx = self.builder.icmp_signed('>', int32_t(0), index) with self.builder.if_then(min_idx): - self.longJmp(line) + self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) length = self.builder.call(self.externs['strlen'], [string]) max_idx = self.builder.icmp_signed('<=', length, index) with self.builder.if_then(max_idx): - self.longJmp(line) + self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) ptr = self.builder.gep(string, [index]) char = self.builder.load(ptr) addr = self.builder.call(self.externs['malloc'], [ @@ -558,6 +613,8 @@ def CallExpr(self, node: CallExpr): return elif node.function.name == "len": return self.emit_len(node.args[0]) + elif node.function.name == "input": + return self.emit_input() callee_func = self.module.get_global(node.function.name) if callee_func is None or not isinstance(callee_func, ir.Function): raise Exception("unknown function") @@ -583,8 +640,7 @@ def ForStmt(self, node: ForStmt): self.builder.call(self.externs['strlen'], [iterable])), lambda: self.forBody(node, var, - lambda currIdx: self.strIndex( - iterable, currIdx), + lambda currIdx: self.strIndex(iterable, currIdx), idx_var)) else: self.assert_nonnull(iterable, node.iterable.location[0]) @@ -690,6 +746,7 @@ def MemberExpr(self, node: MemberExpr): cls = node.object.inferredType.className attr = node.member.name obj = self.visit(node.object) + self.assert_nonnull(obj, node.object.location[0]) ptr = self.getAttrPtr(obj, cls, attr) return self.builder.load(ptr, attr) @@ -762,7 +819,7 @@ def assert_nonnull(self, val, line): val = self.toVoidPtr(val) cond = self.builder.icmp_signed('==', voidptr_t(None), val) with self.builder.if_then(cond): - self.longJmp(line) + self.longJmp(ErrorCode.NULL_PTR, line) def list_len(self, arg): val = self.builder.bitcast(arg, int32_t.as_pointer()) @@ -773,12 +830,18 @@ def emit_assert(self, arg: Expr): arg = self.visit(arg) cond = self.builder.icmp_unsigned('==', bool_t(0), arg) with self.builder.if_then(cond): - self.longJmp(line) + self.longJmp(ErrorCode.ASSERT, line) + + def longJmp(self, code: int, line: int): + code_addr = self.module.get_global("__error_code") + self.builder.store(int32_t(code), code_addr) + + line_addr = self.module.get_global("__error_line") + self.builder.store(int32_t(line), line_addr) - def longJmp(self, line: int): jmp_buf = self.module.get_global('__jmp_buf') self.builder.call(self.externs['longjmp'], [ - jmp_buf, int32_t(line)]) + jmp_buf, int32_t(1)]) self.builder.unreachable() def emit_print(self, arg: Expr): @@ -805,6 +868,20 @@ def emit_print(self, arg: Expr): self.printf(self.module.get_global('__fmt_s'), self.visit(arg)) return self.NoneLiteral(None) + def emit_input(self): + # get input from user + input_buf = self.toVoidPtr(self.module.get_global("__input_buf")) + fmt = self.toVoidPtr(self.module.get_global('__fmt_input')) + self.builder.call(self.externs['scanf'], [fmt, input_buf]) + + # copy contents into new string so that input buffer can be reused + len = self.builder.call(self.externs['strlen'], [input_buf]) + new_str = self.builder.call( + self.externs['malloc'], [self.builder.add(len, int32_t(1))], 'new_str') + fmt = self.toVoidPtr(self.module.get_global('__fmt_str')) + self.builder.call(self.externs['sprintf'], [new_str, fmt, input_buf]) + return new_str + # UTILS def make_bytearray(self, buf): From 3e31990ff034b8ec7de2df9dd60e633ff5675df4 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 20 Jun 2023 00:04:42 -0700 Subject: [PATCH 73/79] update README --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 83b601f..998334f 100644 --- a/README.md +++ b/README.md @@ -27,10 +27,10 @@ This compiler contains multiple backends not found in the reference implementati The test suite includes both static validation of generated/annotated ASTs, as well as runtime tests that actually execute the output programs to check correctness. Many of the AST validation test cases are taken from test suites included in the release code for Berkeley's CS164, with some additional tests written for more coverage. ## Requirements: -- Python 3.6 - 3.8 +- Python 3.11 - JVM Backend Requirements: - [Krakatau JVM Assembler](https://github.com/Storyyeller/Krakatau) - - Tested with Java 8 + - Tested with Java 8, using Krakatau V1 (the one written in Python, not Rust) - CIL Backend Requirements: - [Mono](https://www.mono-project.com/) - Tested with Mono 6.12 @@ -194,7 +194,7 @@ Error handling is done using the `setjmp`/`longjmp` strategy, with the line of t - What is this for? - The primary goal of the project is for me to practice compiler implementation. The secondary goal is to provide a reference to anyone else who is interested in the topics I explore through working on this project - I go into more detail about each part of the compiler on my blog. - Why Chocopy? - - It has a detailed spec and is a relatively small language while being non-trivial enough to offer interesting compiler implementation problems. + - It has a detailed spec and is a relatively small language while being non-trivial enough to offer interesting compiler implementation problems. For example: arrays, inheritance/dynamic dispatch, nested functions, etc. - Why not design your own language? - This project is focused on compiler implementation. I want to keep the project very focused and make each addition manageable so that I can make progress in my very limited spare time. - Why implement this in Python? From 407e5c847af718a960cc021b8a47d403a3149e98 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Tue, 20 Jun 2023 00:05:54 -0700 Subject: [PATCH 74/79] update README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 998334f..246cd11 100644 --- a/README.md +++ b/README.md @@ -187,7 +187,7 @@ Memory does not get freed/garbage collected once it is allocated, so large progr To provide some memory safety, string/list indexing have bounds checking and list operations have a null-check. Unlike the WASM backend, bounds checking and null checks have their own error messages and display a line number similar to assertions. -Error handling is done using the `setjmp`/`longjmp` strategy, with the line of the error/assertion used as the argument for `longjmp`. +Error handling is done using the `setjmp`/`longjmp` strategy, with the error code and line saved in global variables. ## FAQ From 957dfedceb5c7d28917c301e1b60107b9b517be7 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 18 Jul 2023 01:11:13 -0700 Subject: [PATCH 75/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 246cd11..446be49 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Progress is documented on my [blog](https://yangdanny97.github.io/blog/): - [Part 2: JVM backend](https://yangdanny97.github.io/blog/2021/08/26/chocopy-jvm-backend) - [Part 3: CIL backend](https://yangdanny97.github.io/blog/2022/05/22/chocopy-cil-backend) - [Part 4: WASM backend](https://yangdanny97.github.io/blog/2022/10/11/chocopy-wasm-backend) -- [Part 5: LLVM backend - coming soon!](https://yangdanny97.github.io/blog) +- [Part 5: LLVM backend - coming soon!](https://yangdanny97.github.io/blog/2023/07/18/chocopy-llvm-backend) ## Features From 034b3acd9491700f1eef2b82215e7b173ec45482 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 18 Jul 2023 01:11:22 -0700 Subject: [PATCH 76/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 446be49..991256a 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Progress is documented on my [blog](https://yangdanny97.github.io/blog/): - [Part 2: JVM backend](https://yangdanny97.github.io/blog/2021/08/26/chocopy-jvm-backend) - [Part 3: CIL backend](https://yangdanny97.github.io/blog/2022/05/22/chocopy-cil-backend) - [Part 4: WASM backend](https://yangdanny97.github.io/blog/2022/10/11/chocopy-wasm-backend) -- [Part 5: LLVM backend - coming soon!](https://yangdanny97.github.io/blog/2023/07/18/chocopy-llvm-backend) +- [Part 5: LLVM backend](https://yangdanny97.github.io/blog/2023/07/18/chocopy-llvm-backend) ## Features From 6e0b17c0ce1f5e3b5e43d450c3030fe865afb8a9 Mon Sep 17 00:00:00 2001 From: yangdanny97 Date: Sun, 24 Mar 2024 19:06:38 -0400 Subject: [PATCH 77/79] add type hints w/ pyright --- .gitignore | 1 + compiler/astnodes/classdef.py | 3 +- compiler/astnodes/declaration.py | 4 + compiler/astnodes/errors.py | 2 +- compiler/astnodes/expr.py | 11 +- compiler/astnodes/funcdef.py | 8 +- compiler/astnodes/identifier.py | 8 +- compiler/astnodes/listexpr.py | 4 +- compiler/astnodes/memberexpr.py | 5 +- compiler/astnodes/node.py | 11 +- compiler/astnodes/returnstmt.py | 6 +- compiler/astnodes/typedvar.py | 14 +- compiler/builder.py | 9 +- compiler/cil_backend.py | 115 ++++--- compiler/closuretransformer.py | 14 +- compiler/closurevisitor.py | 1 + compiler/compiler.py | 11 +- compiler/empty_list_typer.py | 22 +- compiler/jvm_backend.py | 149 +++++---- compiler/llvm_backend.py | 553 ++++++++++++++++--------------- compiler/nestedfunchoister.py | 4 +- compiler/parser.py | 7 +- compiler/python_backend.py | 5 +- compiler/typechecker.py | 98 +++--- compiler/types/functype.py | 11 +- compiler/types/listvaluetype.py | 9 +- compiler/types/symboltype.py | 15 +- compiler/types/valuetype.py | 15 +- compiler/typesystem.py | 102 +++--- compiler/varcollector.py | 4 +- compiler/visitor.py | 62 ++-- compiler/wasm_backend.py | 139 ++++---- main.py | 3 +- pyrightconfig.json | 8 + 34 files changed, 774 insertions(+), 659 deletions(-) create mode 100644 pyrightconfig.json diff --git a/.gitignore b/.gitignore index 11331b0..7f86350 100644 --- a/.gitignore +++ b/.gitignore @@ -133,3 +133,4 @@ unused/ *.class *.j **/.DS_Store +.vscode/ diff --git a/compiler/astnodes/classdef.py b/compiler/astnodes/classdef.py index 772f389..1d2c06d 100644 --- a/compiler/astnodes/classdef.py +++ b/compiler/astnodes/classdef.py @@ -50,7 +50,8 @@ def getIdentifier(self) -> Identifier: return self.name def getDefaultConstructor(self) -> FuncDef: - var_decls = [d for d in self.declarations if isinstance(d, VarDef)] + var_decls: List[Declaration] = [ + d for d in self.declarations if isinstance(d, VarDef)] constructor = FuncDef(self.location, Identifier(self.location, "__init__"), [TypedVar(self.location, diff --git a/compiler/astnodes/declaration.py b/compiler/astnodes/declaration.py index 8e95696..0cb7715 100644 --- a/compiler/astnodes/declaration.py +++ b/compiler/astnodes/declaration.py @@ -1,3 +1,4 @@ +from .identifier import Identifier from .node import Node from typing import List @@ -6,3 +7,6 @@ class Declaration(Node): def __init__(self, location: List[int], kind: str): super().__init__(location, kind) + + def getIdentifier(self) -> Identifier: + raise Exception("unimplemented") diff --git a/compiler/astnodes/errors.py b/compiler/astnodes/errors.py index 30491f6..43953c3 100644 --- a/compiler/astnodes/errors.py +++ b/compiler/astnodes/errors.py @@ -10,7 +10,7 @@ def __init__(self, location: List[int], errors: List[CompilerError]): self.errors = errors def visit(self, visitor): - pass + return self def toJSON(self, dump_location=True): d = super().toJSON(dump_location) diff --git a/compiler/astnodes/expr.py b/compiler/astnodes/expr.py index af6a36a..12fcc76 100644 --- a/compiler/astnodes/expr.py +++ b/compiler/astnodes/expr.py @@ -1,10 +1,10 @@ from .node import Node -from typing import List -from ..types import ValueType +from typing import List, Optional, Union +from ..types import ValueType, FuncType class Expr(Node): - inferredType: ValueType + inferredType: Optional[Union[ValueType, FuncType]] def __init__(self, location: List[int], kind: str): super().__init__(location, kind) @@ -16,3 +16,8 @@ def toJSON(self, dump_location=True): if self.inferredType is not None: d['inferredType'] = self.inferredType.toJSON(dump_location) return d + + def inferredValueType(self) -> ValueType: + assert self.inferredType is not None and isinstance( + self.inferredType, ValueType) + return self.inferredType diff --git a/compiler/astnodes/funcdef.py b/compiler/astnodes/funcdef.py index 98cf3fa..ff5b553 100644 --- a/compiler/astnodes/funcdef.py +++ b/compiler/astnodes/funcdef.py @@ -4,12 +4,12 @@ from .typeannotation import TypeAnnotation from .stmt import Stmt from ..types import FuncType -from typing import List +from typing import List, Optional class FuncDef(Declaration): freevars: List[Identifier] # used in AST transformations, not printed out - type: FuncType = None # type signature of function + type: Optional[FuncType] = None # type signature of function # The AST for # def NAME(PARAMS) -> RETURNTYPE: @@ -60,3 +60,7 @@ def toJSON(self, dump_location=True): def getIdentifier(self) -> Identifier: return self.name + + def getTypeX(self) -> FuncType: + assert self.type is not None + return self.type diff --git a/compiler/astnodes/identifier.py b/compiler/astnodes/identifier.py index a5065fa..0816956 100644 --- a/compiler/astnodes/identifier.py +++ b/compiler/astnodes/identifier.py @@ -1,5 +1,5 @@ from .expr import Expr -from typing import List +from typing import List, Optional from ..types import VarInstance CIL_KEYWORDS = set(["char", "value", "int32", "int64", "string", "long", "null"] + @@ -122,7 +122,7 @@ class Identifier(Expr): - varInstance: VarInstance = None + varInstance: Optional[VarInstance] = None def __init__(self, location: List[int], name: str): super().__init__(location, "Identifier") @@ -146,3 +146,7 @@ def getCILName(self): if self.name in CIL_KEYWORDS: return f"'{self.name}'" return self.name + + def varInstanceX(self) -> VarInstance: + assert self.varInstance is not None + return self.varInstance diff --git a/compiler/astnodes/listexpr.py b/compiler/astnodes/listexpr.py index d2af568..539b3fc 100644 --- a/compiler/astnodes/listexpr.py +++ b/compiler/astnodes/listexpr.py @@ -1,8 +1,10 @@ from .expr import Expr -from typing import List +from ..types import ValueType +from typing import List, Optional class ListExpr(Expr): + emptyListType: Optional[ValueType] def __init__(self, location: List[int], elements: List[Expr]): super().__init__(location, "ListExpr") diff --git a/compiler/astnodes/memberexpr.py b/compiler/astnodes/memberexpr.py index 8827dfc..79e1dd9 100644 --- a/compiler/astnodes/memberexpr.py +++ b/compiler/astnodes/memberexpr.py @@ -1,11 +1,10 @@ from .expr import Expr from .identifier import Identifier -from typing import List -from ..types import SymbolType +from typing import List, Union +from ..types import FuncType, ValueType class MemberExpr(Expr): - inferredType: SymbolType def __init__(self, location: List[int], obj: Expr, member: Identifier): super().__init__(location, "MemberExpr") diff --git a/compiler/astnodes/node.py b/compiler/astnodes/node.py index 1a05981..b1c6a9d 100644 --- a/compiler/astnodes/node.py +++ b/compiler/astnodes/node.py @@ -1,7 +1,8 @@ -from typing import List +from typing import List, Self, Optional class Node: + errorMsg: Optional[str] def __init__(self, location: List[int], kind: str): if len(location) != 2: @@ -10,13 +11,13 @@ def __init__(self, location: List[int], kind: str): self.location = location self.errorMsg = None - def visit(self, visitor): - return Exception('operation not supported') + def visit(self, visitor) -> Self: + raise Exception('operation not supported') - def preorder(self, visitor): + def preorder(self, visitor) -> Self: return self.visit(visitor) - def postorder(self, visitor): + def postorder(self, visitor) -> Self: return self.visit(visitor) def toJSON(self, dump_location=True) -> dict: diff --git a/compiler/astnodes/returnstmt.py b/compiler/astnodes/returnstmt.py index d124dfb..5b46043 100644 --- a/compiler/astnodes/returnstmt.py +++ b/compiler/astnodes/returnstmt.py @@ -1,11 +1,13 @@ from .stmt import Stmt from .expr import Expr -from typing import List +from ..types import ValueType +from typing import List, Optional class ReturnStmt(Stmt): + expType: Optional[ValueType] - def __init__(self, location: List[int], value: Expr): + def __init__(self, location: List[int], value: Optional[Expr]): super().__init__(location, "ReturnStmt") self.value = value self.isReturn = True diff --git a/compiler/astnodes/typedvar.py b/compiler/astnodes/typedvar.py index 4903d90..fa9539c 100644 --- a/compiler/astnodes/typedvar.py +++ b/compiler/astnodes/typedvar.py @@ -2,12 +2,12 @@ from .identifier import Identifier from .typeannotation import TypeAnnotation from ..types import ValueType, VarInstance -from typing import List +from typing import List, Optional class TypedVar(Node): - t: ValueType = None # the typechecked type goes here - varInstance: VarInstance = None + t: Optional[ValueType] = None # the typechecked type goes here + varInstance: Optional[VarInstance] = None def __init__(self, location: List[int], identifier: Identifier, typ: TypeAnnotation): super().__init__(location, "TypedVar") @@ -25,3 +25,11 @@ def toJSON(self, dump_location=True): d["identifier"] = self.identifier.toJSON(dump_location) d["type"] = self.type.toJSON(dump_location) return d + + def varInstanceX(self) -> VarInstance: + assert self.varInstance is not None + return self.varInstance + + def getTypeX(self) -> ValueType: + assert self.t is not None + return self.t diff --git a/compiler/builder.py b/compiler/builder.py index 364b33a..a9ca166 100644 --- a/compiler/builder.py +++ b/compiler/builder.py @@ -1,7 +1,10 @@ +from typing import List, Union, cast + + class Builder: def __init__(self, name: str): self.name = name - self.lines = [] # list of strings or children builders + self.lines: List[Union[str, Builder]] = [] # list of strings or children builders self.indentation = 0 def newLine(self, line=""): @@ -16,9 +19,9 @@ def newBlock(self): return child def addText(self, text=""): - if len(self.lines) == 0: + if len(self.lines) == 0 or isinstance(self.lines[-1], Builder): self.newLine() - self.lines[-1] = self.lines[-1] + text + self.lines[-1] = cast(str, self.lines[-1]) + text return self def indent(self): diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index b2c64cb..68760e2 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from typing import List +from typing import List, Optional, cast, Callable import json @@ -44,7 +44,7 @@ def newLabelName(self) -> str: self.counter += 1 return "IL_" + str(self.counter) - def label(self, name: str) -> str: + def label(self, name: str): self.builder.unindent() self.instr(name + ": nop") self.builder.indent() @@ -70,9 +70,9 @@ def load(self, name: str): self.instr(f"ldloc {n.loc}") def loadVarAddr(self, node: Identifier): - if self.defaultToGlobals or node.varInstance.isGlobal: + if self.defaultToGlobals or node.varInstanceX().isGlobal: self.instr( - f"ldsflda {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") + f"ldsflda {node.inferredValueType().getCILName()} {self.main}::{node.getCILName()}") elif self.isFromRefArg(node): self.load(node.name) else: @@ -119,7 +119,7 @@ def newLocalEntry(self, name: str, t: ValueType, isArg: bool = False) -> int: def genLocalName(self, offset: int) -> str: return f"__local__{offset}" - def newLocal(self, name: str, t: ValueType): + def newLocal(self, name: Optional[str], t: ValueType): # store the top of stack as a new local n = len([k for k in self.locals[-1] if not self.locals[-1][k].isArg]) self.instr(f"stloc {n}") @@ -151,7 +151,7 @@ def Program(self, node: Program): # global vars (static members) for v in var_decls: self.instr( - f".field public static {v.var.t.getCILName()} {v.getIdentifier().getCILName()}") + f".field public static {v.var.getTypeX().getCILName()} {v.getIdentifier().getCILName()}") # main method, top level statements self.instr( @@ -164,7 +164,7 @@ def Program(self, node: Program): for v in var_decls: self.visit(v.value) self.instr( - f"stsfld {v.var.t.getCILName()} {self.main}::{v.getIdentifier().getCILName()}") + f"stsfld {v.var.getTypeX().getCILName()} {self.main}::{v.getIdentifier().getCILName()}") self.visitStmtList(node.statements) self.defaultToGlobals = False self.generateLocalsDirective(locals) @@ -182,7 +182,7 @@ def Program(self, node: Program): def ClassDef(self, node: ClassDef): def constructor(superclass: str, func: FuncDef): - func.type = func.type.dropFirstParam() + func.type = func.getTypeX().dropFirstParam() func.name.name = ".ctor" # add call to parent constructor after child field initialization # before other constructor statements @@ -202,7 +202,7 @@ def constructor(superclass: str, func: FuncDef): # field decls for v in var_decls: self.instr( - f".field public {v.var.t.getCILName()} {v.getIdentifier().getCILName()}") + f".field public {v.var.getTypeX().getCILName()} {v.getIdentifier().getCILName()}") for d in func_decls: if d.name.name == "__init__": # constructor @@ -211,7 +211,7 @@ def constructor(superclass: str, func: FuncDef): constructor(superclass, d) else: # method - d.type = d.type.dropFirstParam() + d.type = d.getTypeX().dropFirstParam() self.FuncDef(d, "virtual instance") if constructor_def is None: # give a default constructor if none exists @@ -220,7 +220,7 @@ def constructor(superclass: str, func: FuncDef): self.unindent() # end class - def generateLocalsDirective(self, locals): + def generateLocalsDirective(self, locals: Builder): # defer local declarations until we know what we need locals.newLine(".locals init (").indent() mapping = self.locals[-1] @@ -231,10 +231,10 @@ def generateLocalsDirective(self, locals): locals.newLine(sortedDecls[i].decl() + comma) locals.unindent().newLine(")") - def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None): + def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor: Optional[str] = None): self.instr(f".method public hidebysig {funcType}") self.instr( - f"{node.type.getCILSignature(node.name.getCILName())} cil managed") + f"{node.getTypeX().getCILSignature(node.name.getCILName())} cil managed") self.indent() self.instr(f".maxstack {self.stackLimit}") self.enterScope() @@ -242,11 +242,11 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None # initialize locals locals = self.builder.newBlock() for i in range(len(node.params)): - self.newLocalEntry( - node.params[i].identifier.name, node.params[i].t, True) + param = node.params[i] + self.newLocalEntry(param.identifier.name, param.getTypeX(), True) for d in node.declarations: self.visit(d) - self.returnType = node.type.returnType + self.returnType = node.getTypeX().returnType # handle last return if superConstructor: @@ -266,41 +266,42 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor=None def VarDef(self, node: VarDef): if node.isAttr: # codegen for initialization in constructors + assert node.attrOfClass is not None className = ClassValueType(node.attrOfClass) self.instr("ldarg 0") self.visit(node.value) self.instr( - f"stfld {node.var.t.getCILName()} {className.getCILName()}::{node.getIdentifier().getCILName()}") + f"stfld {node.var.getTypeX().getCILName()} {className.getCILName()}::{node.getIdentifier().getCILName()}") else: self.visit(node.value) - self.newLocal(node.getIdentifier().name, node.var.t) + self.newLocal(node.getIdentifier().name, node.var.getTypeX()) # STATEMENTS def processAssignmentTarget(self, target: Expr): if isinstance(target, Identifier): - if self.defaultToGlobals or target.varInstance.isGlobal: + if self.defaultToGlobals or target.varInstanceX().isGlobal: self.instr( - f"stsfld {target.inferredType.getCILName()} {self.main}::{target.getCILName()}") + f"stsfld {target.inferredValueType().getCILName()} {self.main}::{target.getCILName()}") elif self.isFromRefArg(target): - temp = self.newLocal(None, target.inferredType) + temp = self.newLocal(None, target.inferredValueType()) self.load(target.name) self.load(temp) - self.storeInd(target.inferredType) + self.storeInd(target.inferredValueType()) else: self.store(target.name) elif isinstance(target, IndexExpr): - temp = self.newLocal(None, target.inferredType) + temp = self.newLocal(None, target.inferredValueType()) self.visit(target.list) self.visit(target.index) self.load(temp) - self.arrayStore(target.inferredType) + self.arrayStore(target.inferredValueType()) elif isinstance(target, MemberExpr): - temp = self.newLocal(None, target.inferredType) + temp = self.newLocal(None, target.inferredValueType()) self.visit(target.object) self.load(temp) self.instr( - f"stfld {target.inferredType.getCILName()} {target.object.inferredType.getCILName()}::{target.member.getCILName()}") + f"stfld {target.inferredValueType().getCILName()} {target.object.inferredValueType().getCILName()}::{target.member.getCILName()}") else: raise Exception( "Internal compiler error: unsupported assignment target") @@ -346,8 +347,8 @@ def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) def BinaryExpr(self, node: BinaryExpr): operator = node.operator - leftType = node.left.inferredType - rightType = node.right.inferredType + leftType = node.left.inferredValueType() + rightType = node.right.inferredValueType() shortCircuitOperators = {"and", "or"} if operator not in shortCircuitOperators: self.visit(node.left) @@ -369,9 +370,9 @@ def BinaryExpr(self, node: BinaryExpr): self.instr("ldlen") self.instr("add") self.instr("conv.i4") - merged_t = self.ts.join(leftType, rightType).elementType - self.instr(f"newarr {merged_t.getCILName()}") - merged = self.newLocal(None, ListValueType(merged_t)) + merged_list = cast(ListValueType, self.ts.join(leftType, rightType)) + self.instr(f"newarr {merged_list.elementType.getCILName()}") + merged = self.newLocal(None, merged_list) self.load(l) self.load(merged) self.instr("ldc.i4 0") @@ -455,8 +456,9 @@ def IndexExpr(self, node: IndexExpr): self.visit(node.list) self.visit(node.index) self.instr("conv.i4") - if node.list.inferredType.isListType(): - self.arrayLoad(node.list.inferredType.elementType) + t_list = node.list.inferredValueType() + if isinstance(t_list, ListValueType): + self.arrayLoad(t_list.elementType) else: self.instr( "call instance char [mscorlib]System.String::get_Chars(int32)") @@ -488,6 +490,7 @@ def CallExpr(self, node: CallExpr): else: for i in range(len(node.args)): self.visitArg(node.function.inferredType, i, node.args[i]) + assert isinstance(node.function.inferredType, FuncType) signature = node.function.inferredType.getCILSignature( f"{self.main}::{name}") self.instr(f"call {signature}") @@ -497,7 +500,7 @@ def CallExpr(self, node: CallExpr): def ForStmt(self, node: ForStmt): # itr = {expr}, idx = 0 self.visit(node.iterable) - itr = self.newLocal(None, node.iterable.inferredType) + itr = self.newLocal(None, node.iterable.inferredValueType()) self.instr("ldc.i8 0") idx = self.newLocal(None, IntType()) startLabel = self.newLabelName() @@ -507,7 +510,7 @@ def ForStmt(self, node: ForStmt): self.load(idx) self.instr("conv.i4") self.load(itr) - if node.iterable.inferredType.isListType(): + if isinstance(node.iterable.inferredType, ListValueType): self.instr("ldlen") else: self.instr( @@ -519,7 +522,7 @@ def ForStmt(self, node: ForStmt): self.load(itr) self.load(idx) self.instr("conv.i4") - if node.iterable.inferredType.isListType(): + if isinstance(node.iterable.inferredType, ListValueType): self.arrayLoad(node.iterable.inferredType.elementType) else: self.instr( @@ -549,7 +552,7 @@ def ListExpr(self, node: ListExpr): else: elementType = ClassValueType("object") else: - elementType = t.elementType + elementType = cast(ListValueType, t).elementType self.instr(f"newarr {elementType.getCILName()}") for i in range(len(node.elements)): self.instr("dup") @@ -567,7 +570,7 @@ def WhileStmt(self, node: WhileStmt): self.instr(f"br {startLabel}") self.label(endLabel) - def buildReturn(self, value: Expr): + def buildReturn(self, value: Optional[Expr]): if not self.returnType.isNone(): if value is None: self.NoneLiteral(None) @@ -579,19 +582,19 @@ def ReturnStmt(self, node: ReturnStmt): self.buildReturn(node.value) def Identifier(self, node: Identifier): - if self.defaultToGlobals or node.varInstance.isGlobal: + if self.defaultToGlobals or node.varInstanceX().isGlobal: self.instr( - f"ldsfld {node.inferredType.getCILName()} {self.main}::{node.getCILName()}") + f"ldsfld {node.inferredValueType().getCILName()} {self.main}::{node.getCILName()}") elif self.isFromRefArg(node): self.load(node.name) - self.loadInd(node.inferredType) + self.loadInd(node.inferredValueType()) else: self.load(node.name) def MemberExpr(self, node: MemberExpr): self.visit(node.object) self.instr( - f"ldfld {node.inferredType.getCILName()} {node.object.inferredType.getCILName()}::{node.member.getCILName()}") + f"ldfld {node.inferredValueType().getCILName()} {node.object.inferredValueType().getCILName()}::{node.member.getCILName()}") def IfExpr(self, node: IfExpr): c = lambda: self.visit(node.condition) @@ -599,7 +602,7 @@ def IfExpr(self, node: IfExpr): e = lambda: self.visit(node.elseExpr) self.ternary(c, t, e) - def ternary(self, condFn, thenFn, elseFn): + def ternary(self, condFn: Callable, thenFn: Callable, elseFn: Callable): condFn() l1 = self.newLabelName() l2 = self.newLabelName() @@ -611,6 +614,7 @@ def ternary(self, condFn, thenFn, elseFn): self.label(l2) def MethodCallExpr(self, node: MethodCallExpr): + assert isinstance(node.method.object.inferredType, ClassValueType) className = node.method.object.inferredType.className methodName = node.method.member.getCILName() if methodName == "__init__" and className in {"int", "bool"}: @@ -618,6 +622,7 @@ def MethodCallExpr(self, node: MethodCallExpr): self.visit(node.method.object) for i in range(len(node.args)): self.visitArg(node.method.inferredType, i + 1, node.args[i]) + assert isinstance(node.method.inferredType, FuncType) methodType = node.method.inferredType.dropFirstParam() signature = methodType.getCILSignature( f"{className}::{methodName}") @@ -636,7 +641,7 @@ def BooleanLiteral(self, node: BooleanLiteral): def IntegerLiteral(self, node: IntegerLiteral): self.instr(f"ldc.i8 {node.value}") - def NoneLiteral(self, _: NoneLiteral): + def NoneLiteral(self, node: Optional[NoneLiteral]): self.instr("ldnull") def StringLiteral(self, node: StringLiteral): @@ -662,7 +667,7 @@ def emit_input(self): self.instr("call string [mscorlib]System.Console::ReadLine()") def emit_len(self, arg: Expr): - t = arg.inferredType + t = arg.inferredValueType() is_list = False if t.isListType(): is_list = True @@ -674,8 +679,8 @@ def emit_len(self, arg: Expr): elif t == StrType(): is_list = False else: - self.emit_exn( - f"Built-in function len is unsupported for values of type {arg.inferredType.classname}") + self.emit_exn("Built-in function len is unsupported for values of this type") + return self.visit(arg) if is_list: self.instr("ldlen") @@ -688,16 +693,18 @@ def emit_len(self, arg: Expr): def emit_print(self, arg: Expr): self.visit(arg) self.instr( - f"call void class [mscorlib]System.Console::WriteLine({arg.inferredType.getCILName()})") + f"call void class [mscorlib]System.Console::WriteLine({arg.inferredValueType().getCILName()})") self.NoneLiteral(None) def isFromRefArg(self, arg: Expr): - return self.isFromArg(arg) and arg.varInstance.isNonlocal + if not isinstance(arg, Identifier): + return False + return self.isFromArg(arg) and arg.varInstanceX().isNonlocal def isFromArg(self, arg: Expr): if not isinstance(arg, Identifier): return False - if arg.varInstance.isGlobal: + if arg.varInstanceX().isGlobal: return True n = self.locals[-1][arg.name] if n is None: @@ -708,14 +715,14 @@ def isFromArg(self, arg: Expr): def visitArg(self, funcType, paramIdx: int, arg: Expr): argIsRef = self.isFromRefArg(arg) paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and cast(Identifier, arg).varInstance == funcType.refParams[paramIdx]: # ref -> ref: pass through a ref to a nonlocal - self.load(arg.name) + self.load(cast(Identifier, arg).name) elif paramIsRef and argIsRef: # ref -> ref: # deref, store value in new local, and pass ref to new local self.visit(arg) - temp = self.newLocal(None, arg.inferredType) + temp = self.newLocal(None, arg.inferredValueType()) self.loadAddr(temp) elif paramIsRef: # value -> ref @@ -724,7 +731,7 @@ def visitArg(self, funcType, paramIdx: int, arg: Expr): self.loadVarAddr(arg) else: self.visit(arg) - temp = self.newLocal(None, arg.inferredType) + temp = self.newLocal(None, arg.inferredValueType()) self.loadAddr(temp) else: # value/ref -> value : deref if necessary diff --git a/compiler/closuretransformer.py b/compiler/closuretransformer.py index 0174098..67c10d5 100644 --- a/compiler/closuretransformer.py +++ b/compiler/closuretransformer.py @@ -4,11 +4,13 @@ from .types import * -def typeToAnnotation(t: ValueType) -> SymbolType: +def typeToAnnotation(t: ValueType) -> TypeAnnotation: if isinstance(t, ListValueType): return ListType([0, 0], typeToAnnotation(t.elementType)) elif isinstance(t, ClassValueType): return ClassType([0, 0], t.className) + else: + raise Exception("unexpected type") class ClosureTransformer(TypeChecker): @@ -26,17 +28,18 @@ def getSignature(self, node: FuncDef): t.refParams = {} for i in range(len(node.params)): - if node.params[i].varInstance.isNonlocal: - t.refParams[i] = node.params[i].varInstance + if node.params[i].varInstanceX().isNonlocal: + t.refParams[i] = node.params[i].varInstanceX() for i in range(len(node.freevars)): - if node.freevars[i].varInstance.isNonlocal: + if node.freevars[i].varInstanceX().isNonlocal: t.refParams[len(node.params) + - i] = node.freevars[i].varInstance + i] = node.freevars[i].varInstanceX() return t def funcParams(self, node: FuncDef): for fv in node.freevars: ident = fv.copy() + assert isinstance(ident.inferredType, ValueType) annot = typeToAnnotation(ident.inferredType) tv = TypedVar(node.location, ident, annot) tv.varInstance = ident.varInstance @@ -52,6 +55,7 @@ def callHelper(self, node): else: t = self.getType(fname) if isinstance(node, MethodCallExpr): + assert isinstance(node.method.object.inferredType, ClassValueType) class_name, member_name = node.method.object.inferredType.className, node.method.member.name t = self.ts.getMethod(class_name, member_name) if t is None or len(t.freevars) == 0 or not isinstance(t, FuncType): diff --git a/compiler/closurevisitor.py b/compiler/closurevisitor.py index 33273f2..db5e355 100644 --- a/compiler/closurevisitor.py +++ b/compiler/closurevisitor.py @@ -58,6 +58,7 @@ def Program(self, node: Program): for d in node.declarations: if isinstance(d, VarDef): self.globals[d.getIdentifier().name] = newInstance(d.var) + assert d.var.varInstance is not None d.var.varInstance.isGlobal = True # mark all top-level vars to be global vars = VarCollector().getVarsFromList(node.statements) diff --git a/compiler/compiler.py b/compiler/compiler.py index 64934f0..1e7d74e 100644 --- a/compiler/compiler.py +++ b/compiler/compiler.py @@ -1,10 +1,10 @@ -from compiler.empty_list_typer import EmptyListTyper from .astnodes import * from .types import * from .typechecker import TypeChecker from .parser import Parser, ParseError from .closurevisitor import ClosureVisitor from .closuretransformer import ClosureTransformer +from .empty_list_typer import EmptyListTyper from .nestedfunchoister import NestedFuncHoister from .typesystem import TypeSystem from .jvm_backend import JvmBackend @@ -14,17 +14,18 @@ from .llvm_backend import LlvmBackend import ast from pathlib import Path +from typing import Optional class Compiler: - transformer: ClosureTransformer = None + transformer: Optional[ClosureTransformer] = None def __init__(self): self.ts = TypeSystem() self.parser = Parser() self.typechecker = TypeChecker(self.ts) - def parse(self, infile) -> Program: + def parse(self, infile) -> Optional[Program]: astparser = self.parser # given an input file, parse it into an AST object lines = None @@ -67,6 +68,7 @@ def emitPython(self, ast: Program): def emitJVM(self, main: str, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) + assert self.transformer is not None jvm_backend = JvmBackend(main, self.transformer.ts) jvm_backend.visit(ast) return jvm_backend.classes @@ -74,6 +76,7 @@ def emitJVM(self, main: str, ast: Program): def emitCIL(self, main: str, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) + assert self.transformer is not None cil_backend = CilBackend(main, self.transformer.ts) cil_backend.visit(ast) return cil_backend.builder @@ -81,6 +84,7 @@ def emitCIL(self, main: str, ast: Program): def emitWASM(self, main: str, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) + assert self.transformer is not None wasm_backend = WasmBackend(main, self.transformer.ts) wasm_backend.visit(ast) return wasm_backend.builder @@ -88,6 +92,7 @@ def emitWASM(self, main: str, ast: Program): def emitLLVM(self, ast: Program): self.closurepass(ast) EmptyListTyper().visit(ast) + assert self.transformer is not None llvm_backend = LlvmBackend(self.transformer.ts) llvm_backend.visit(ast) return llvm_backend.module diff --git a/compiler/empty_list_typer.py b/compiler/empty_list_typer.py index bb6a798..9810112 100644 --- a/compiler/empty_list_typer.py +++ b/compiler/empty_list_typer.py @@ -1,7 +1,7 @@ from .astnodes import * from .types import * from .visitor import Visitor -from typing import List +from typing import List, Optional # A visitor to refine the types of empty list literals [] # based on what they are being assigned to @@ -9,13 +9,13 @@ class EmptyListTyper(Visitor): - expectedType: ValueType = None - expReturnType: ValueType = None + expectedType: Optional[ValueType] = None + expReturnType: Optional[ValueType] = None def visit(self, node: Node): return node.preorder(self) - def isEmptyListMultiAssign(self, node: Node): + def isEmptyListMultiAssign(self, node: Node) -> bool: if not isinstance(node, AssignStmt): return False if len(node.targets) == 1 or not isinstance(node.value, ListExpr): @@ -34,35 +34,43 @@ def Program(self, node: Program): statements = [] for s in node.statements: if self.isEmptyListMultiAssign(s): + assert isinstance(s, AssignStmt) statements = statements + self.transformMultiAssign(s) else: statements.append(s) node.statements = statements def FuncDef(self, node: FuncDef): + assert node.type is not None self.expReturnType = node.type.returnType statements = [] for s in node.statements: if self.isEmptyListMultiAssign(s): + assert isinstance(s, AssignStmt) statements = statements + self.transformMultiAssign(s) else: statements.append(s) node.statements = statements def CallExpr(self, node: CallExpr): + assert isinstance(node.function.inferredType, FuncType) for i in range(len(node.args)): self.expectedType = node.function.inferredType.parameters[i] self.visit(node.args[i]) self.expectedType = None def AssignStmt(self, node: AssignStmt): - self.expectedType = node.targets[0].inferredType + target_type = node.targets[0].inferredType + assert isinstance(target_type, ValueType) + self.expectedType = target_type def ListExpr(self, node: ListExpr): if self.expectedType is None: return expType = self.expectedType - if isinstance(self.expectedType, ListValueType) and len(node.elements) == 0: + if not isinstance(expType, ListValueType): + expType = ListValueType(ObjectType()) + if len(node.elements) == 0: node.emptyListType = expType.elementType else: for i in node.elements: @@ -72,6 +80,8 @@ def ListExpr(self, node: ListExpr): def MethodCallExpr(self, node: MethodCallExpr): for i in range(len(node.args)): + assert node.method.inferredType is not None and isinstance( + node.method.inferredType, FuncType) self.expectedType = node.method.inferredType.parameters[i] self.visit(node.args[i]) self.expectedType = None diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index ca83cb1..f3d4de1 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from typing import List, Dict +from typing import List, Dict, Optional, cast, Callable import json @@ -29,7 +29,7 @@ def newLabelName(self) -> str: self.counter += 1 return "L" + str(self.counter) - def label(self, name: str) -> str: + def label(self, name: str): self.currentBuilder().unindent() self.instr(name + ":") self.currentBuilder().indent() @@ -96,7 +96,7 @@ def newLocalEntry(self, name: str) -> int: def genLocalName(self, offset: int) -> str: return f"__local__{offset}" - def newLocal(self, name: str = None, isRef: bool = True) -> int: + def newLocal(self, name: Optional[str] = None, isRef: bool = True) -> int: # store the top of stack as a new local n = len(self.locals[-1]) if isRef: @@ -126,7 +126,7 @@ def Program(self, node: Program): # global decls for v in var_decls: self.instr( - f".field static {v.var.identifier.name} {v.var.t.getJavaSignature()}") + f".field static {v.var.identifier.name} {v.var.getTypeX().getJavaSignature()}") # main self.instr(".method public static main : ([Ljava/lang/String;)V") @@ -148,7 +148,7 @@ def Program(self, node: Program): for v in var_decls: self.visit(v.value) self.instr( - f"putstatic Field {self.main} {v.var.identifier.name} {v.var.t.getJavaSignature()}") + f"putstatic Field {self.main} {v.var.identifier.name} {v.var.getTypeX().getJavaSignature()}") self.instr("return") self.instr(".end code") self.currentBuilder().unindent() @@ -180,7 +180,7 @@ def ClassDef(self, node: ClassDef): # field decls for v in var_decls: self.instr( - f".field {v.var.identifier.name} {v.var.t.getJavaSignature()}") + f".field {v.var.identifier.name} {v.var.getTypeX().getJavaSignature()}") for d in func_decls: if d.name.name == "__init__": constructor_def = d @@ -198,7 +198,7 @@ def funcDefHelper(self, node: FuncDef): self.newLocalEntry(node.params[i].identifier.name) for d in node.declarations: self.visit(d) - self.returnType = node.type.returnType + self.returnType = node.getTypeX().returnType # handle last return self.visitStmtList(node.statements) hasReturn = False @@ -211,7 +211,7 @@ def funcDefHelper(self, node: FuncDef): def constructor(self, superclass: str, node: FuncDef): self.enterScope() - constructorSig = node.type.dropFirstParam() + constructorSig = node.getTypeX().dropFirstParam() self.instr( f".method public : {constructorSig.getJavaSignature()}") self.currentBuilder().indent() @@ -227,7 +227,7 @@ def constructor(self, superclass: str, node: FuncDef): def method(self, node: FuncDef): self.enterScope() - methodSig = node.type.dropFirstParam() + methodSig = node.getTypeX().dropFirstParam() self.instr( f".method public {node.name.name} : {methodSig.getJavaSignature()}") self.currentBuilder().indent() @@ -241,7 +241,7 @@ def method(self, node: FuncDef): def FuncDef(self, node: FuncDef): self.enterScope() self.instr( - f".method public static {node.name.name} : {node.type.getJavaSignature()}") + f".method public static {node.name.name} : {node.getTypeX().getJavaSignature()}") self.currentBuilder().indent() self.instr( f".code stack {self.stackLimit} locals {len(node.declarations) + self.localLimit}") @@ -257,42 +257,42 @@ def VarDef(self, node: VarDef): self.instr("aload 0") self.visit(node.value) self.instr( - f"putfield Field {className.getJavaName()} {varName} {node.var.t.getJavaSignature()}") - elif node.var.varInstance.isNonlocal: - elementType = node.var.t - self.wrap(node.value, elementType) + f"putfield Field {className.getJavaName()} {varName} {node.var.getTypeX().getJavaSignature()}") + elif node.var.varInstanceX().isNonlocal: + self.wrap(node.value, node.var.getTypeX()) self.newLocal(varName, True) else: self.visit(node.value) - self.newLocal(varName, node.value.inferredType.isJavaRef()) + self.newLocal(varName, node.value.inferredValueType().isJavaRef()) # STATEMENTS def processAssignmentTarget(self, target: Expr): if isinstance(target, Identifier): - if self.defaultToGlobals or target.varInstance.isGlobal: + if self.defaultToGlobals or target.varInstanceX().isGlobal: self.instr( - f"putstatic Field {self.main} {target.name} {target.inferredType.getJavaSignature()}") - elif target.varInstance.isNonlocal: - self.load(target.name, ListValueType(target.inferredType)) + f"putstatic Field {self.main} {target.name} {target.inferredValueType().getJavaSignature()}") + elif target.varInstanceX().isNonlocal: + self.load(target.name, ListValueType( + target.inferredValueType())) self.instr("swap") self.loadInt(0) self.instr("swap") - self.arrayStore(target.inferredType) + self.arrayStore(target.inferredValueType()) else: - self.store(target.name, target.inferredType) + self.store(target.name, target.inferredValueType()) elif isinstance(target, IndexExpr): # stack should be array, idx, value self.visit(target.list) self.instr("swap") self.visit(target.index) self.instr("swap") - self.arrayStore(target.inferredType) + self.arrayStore(target.inferredValueType()) elif isinstance(target, MemberExpr): self.visit(target.object) self.instr("swap") self.instr( - f"putfield Field {target.object.inferredType.className} {target.member.name} {target.inferredType.getJavaSignature()}") + f"putfield Field {cast(ClassValueType, target.object.inferredValueType()).className} {target.member.name} {target.inferredValueType().getJavaSignature()}") else: raise Exception( "Internal compiler error: unsupported assignment target") @@ -358,8 +358,8 @@ def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) def BinaryExpr(self, node: BinaryExpr): operator = node.operator - leftType = node.left.inferredType - rightType = node.right.inferredType + leftType = node.left.inferredValueType() + rightType = node.right.inferredValueType() shortCircuitOps = {"and", "or"} if operator not in shortCircuitOps: self.visit(node.left) @@ -378,8 +378,9 @@ def BinaryExpr(self, node: BinaryExpr): self.instr(f"iload {lenR}") self.instr("iadd") # stack is L, total_length + list_t = cast(ListValueType, self.ts.join(leftType, rightType)) self.instr( - f"anewarray {self.ts.join(leftType, rightType).elementType.getJavaName(True)}") + f"anewarray {list_t.elementType.getJavaName(True)}") newArr = self.newLocal(None, True) self.instr("iconst_0") self.instr(f"aload {newArr}") @@ -441,15 +442,15 @@ def BinaryExpr(self, node: BinaryExpr): self.comparator("if_acmpeq") # logical operators elif operator == "and": - condFn = lambda: self.visit(node.left) - thenFn = lambda: self.visit(node.right) - elseFn = lambda: self.instr("iconst_0") - self.ternary(condFn, thenFn, elseFn) + c = lambda: self.visit(node.left) + t = lambda: self.visit(node.right) + e = lambda: self.instr("iconst_0") + self.ternary(c, t, e) elif operator == "or": - condFn = lambda: self.visit(node.left) - thenFn = lambda: self.instr("iconst_1") - elseFn = lambda: self.visit(node.right) - self.ternary(condFn, thenFn, elseFn) + c = lambda: self.visit(node.left) + t = lambda: self.instr("iconst_1") + e = lambda: self.visit(node.right) + self.ternary(c, t, e) else: raise Exception( f"Internal compiler error: unexpected operator {operator}") @@ -457,8 +458,9 @@ def BinaryExpr(self, node: BinaryExpr): def IndexExpr(self, node: IndexExpr): self.visit(node.list) self.visit(node.index) - if node.list.inferredType.isListType(): - self.arrayLoad(node.list.inferredType.elementType) + if node.list.inferredValueType().isListType(): + self.arrayLoad( + cast(ListValueType, node.list.inferredType).elementType) else: self.instr("dup") self.instr("iconst_1") @@ -482,6 +484,7 @@ def buildConstructor(self, node: CallExpr): self.instr(f"invokespecial Method {javaName} ()V") def CallExpr(self, node: CallExpr): + assert isinstance(node.function.inferredType, FuncType) signature = node.function.inferredType.getJavaSignature() name = node.function.name if node.isConstructor: @@ -515,18 +518,19 @@ def ForStmt(self, node: ForStmt): self.label(startLabel) # while idx < len(itr) self.load(self.genLocalName(idx), IntType()) - self.load(self.genLocalName(itr), node.iterable.inferredType) - if node.iterable.inferredType.isListType(): + self.load(self.genLocalName(itr), node.iterable.inferredValueType()) + if node.iterable.inferredValueType().isListType(): self.instr("arraylength") else: self.instr("invokevirtual Method java/lang/String length ()I") self.instr("isub") self.instr(f"ifge {endLabel}") # x = itr[idx] - self.load(self.genLocalName(itr), node.iterable.inferredType) + self.load(self.genLocalName(itr), node.iterable.inferredValueType()) self.load(self.genLocalName(idx), IntType()) - if node.iterable.inferredType.isListType(): - self.arrayLoad(node.iterable.inferredType.elementType) + if node.iterable.inferredValueType().isListType(): + self.arrayLoad( + cast(ListValueType, node.iterable.inferredValueType()).elementType) else: self.instr("dup") self.instr("iconst_1") @@ -556,7 +560,7 @@ def ListExpr(self, node: ListExpr): else: elementType = ClassValueType("object") else: - elementType = t.elementType + elementType = cast(ListValueType, t).elementType self.instr(f"anewarray {elementType.getJavaName(True)}") for i in range(len(node.elements)): self.instr("dup") @@ -575,7 +579,7 @@ def WhileStmt(self, node: WhileStmt): self.label(endLabel) self.instr("nop") - def buildReturn(self, value: Expr): + def buildReturn(self, value: Optional[Expr]): if self.returnType.isNone(): self.instr("return") else: @@ -589,28 +593,28 @@ def ReturnStmt(self, node: ReturnStmt): self.buildReturn(node.value) def Identifier(self, node: Identifier): - if self.defaultToGlobals or node.varInstance.isGlobal: + if self.defaultToGlobals or node.varInstanceX().isGlobal: self.instr( - f"getstatic Field {self.main} {node.name} {node.inferredType.getJavaSignature()}") - elif node.varInstance.isNonlocal: - self.load(node.name, ListValueType(node.inferredType)) + f"getstatic Field {self.main} {node.name} {node.inferredValueType().getJavaSignature()}") + elif node.varInstanceX().isNonlocal: + self.load(node.name, ListValueType(node.inferredValueType())) self.loadInt(0) - self.arrayLoad(node.inferredType) + self.arrayLoad(node.inferredValueType()) else: - self.load(node.name, node.inferredType) + self.load(node.name, node.inferredValueType()) def MemberExpr(self, node: MemberExpr): self.visit(node.object) self.instr( - f"getfield Field {node.object.inferredType.className} {node.member.name} {node.inferredType.getJavaSignature()}") + f"getfield Field {cast(ClassValueType, node.object.inferredValueType()).className} {node.member.name} {node.inferredValueType().getJavaSignature()}") def IfExpr(self, node: IfExpr): - condFn = lambda: self.visit(node.condition) - thenFn = lambda: self.visit(node.thenExpr) - elseFn = lambda: self.visit(node.elseExpr) - self.ternary(condFn, thenFn, elseFn) + c = lambda: self.visit(node.condition) + t = lambda: self.visit(node.thenExpr) + e = lambda: self.visit(node.elseExpr) + self.ternary(c, t, e) - def ternary(self, condFn, thenFn, elseFn): + def ternary(self, condFn: Callable, thenFn: Callable, elseFn: Callable): condFn() l1 = self.newLabelName() l2 = self.newLabelName() @@ -623,18 +627,20 @@ def ternary(self, condFn, thenFn, elseFn): self.instr("nop") def MethodCallExpr(self, node: MethodCallExpr): - className = node.method.object.inferredType.className + className = cast( + ClassValueType, node.method.object.inferredType).className methodName = node.method.member.name if methodName == "__init__" and className in {"int", "bool"}: return self.visit(node.method.object) - for i in range(len(node.args)): - self.visitArg(node.method.inferredType, i + 1, node.args[i]) methodType = node.method.inferredType + assert isinstance(methodType, FuncType) + for i in range(len(node.args)): + self.visitArg(methodType, i + 1, node.args[i]) javaMethodType = methodType.dropFirstParam() self.instr( f"invokevirtual Method {className} {methodName} {javaMethodType.getJavaSignature()}") - if node.method.inferredType.returnType.isNone(): + if methodType.returnType.isNone(): self.NoneLiteral(None) # push null for void return # LITERALS @@ -654,7 +660,7 @@ def loadInt(self, value: int): def IntegerLiteral(self, node: IntegerLiteral): self.loadInt(node.value) - def NoneLiteral(self, node: NoneLiteral): + def NoneLiteral(self, node: Optional[NoneLiteral]): self.instr("aconst_null") def StringLiteral(self, node: StringLiteral): @@ -691,7 +697,7 @@ def emit_input(self): "invokevirtual Method java/util/Scanner nextLine ()Ljava/lang/String;") def emit_len(self, arg: Expr): - t = arg.inferredType + t = arg.inferredValueType() is_list = False if t.isListType(): is_list = True @@ -704,7 +710,8 @@ def emit_len(self, arg: Expr): is_list = False else: self.emit_exn( - f"Built-in function len is unsupported for values of type {arg.inferredType.classname}") + "Built-in function len is unsupported for values of this type") + return self.visit(arg) if is_list: self.instr("arraylength") @@ -712,25 +719,27 @@ def emit_len(self, arg: Expr): self.instr("invokevirtual Method java/lang/String length ()I") def emit_print(self, arg: Expr): - if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: + if isinstance(arg.inferredType, ListValueType) or cast(ClassValueType, arg.inferredType).className not in {"bool", "int", "str"}: self.emit_exn( - f"Built-in function print is unsupported for values of type {arg.inferredType.classname}") - t = arg.inferredType.getJavaSignature() + "Built-in function print is unsupported for values of this type") + t = arg.inferredValueType().getJavaSignature() self.instr("getstatic Field java/lang/System out Ljava/io/PrintStream;") self.visit(arg) self.instr( f"invokevirtual Method java/io/PrintStream println ({t})V") self.NoneLiteral(None) # push None for void return - def visitArg(self, funcType, paramIdx: int, arg: Expr): - argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): + argIsRef = isinstance( + arg, Identifier) and arg.varInstanceX().isNonlocal paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and cast(Identifier, arg).varInstance == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg - self.load(arg.name, ListValueType(arg.inferredType)) + self.load(cast(Identifier, arg).name, + ListValueType(arg.inferredValueType())) elif paramIsRef: # non-ref arg and ref param, or do not pass ref arg # unwrap if necessary, re-wrap - self.wrap(arg, arg.inferredType) + self.wrap(arg, arg.inferredValueType()) else: # non-ref param, maybe unwrap self.visit(arg) diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index ca18cff..3dc2d5d 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -3,7 +3,7 @@ from .typesystem import TypeSystem from .visitor import Visitor from collections import defaultdict -from typing import List, Dict, Tuple +from typing import List, Dict, Tuple, Optional, cast, Any, Callable import llvmlite.ir as ir import llvmlite.binding as llvm @@ -26,8 +26,14 @@ class ErrorCode: input_buf_t = ir.ArrayType(int8_t, INPUT_CHARS + 1) +class LLVMBuilder(ir.IRBuilder): + def cast(self, value, typ, name='') -> ir.Value: + # the bitcast operation should return something, the type is wrong + return ir.IRBuilder.bitcast(self, value, typ, name) # type: ignore + + class LlvmBackend(Visitor): - locals: List[defaultdict] + locals: List[Dict[str, Optional[ir.Value]]] externs: Dict[str, ir.Function] methods: Dict[str, Dict[str, ir.Function]] structs: Dict[str, ir.LiteralStructType] @@ -35,8 +41,8 @@ class LlvmBackend(Visitor): methodOffsets: Dict[Tuple[str, str], Tuple[int, ir.FunctionType]] # idx in struct, initial value attrOffsets: Dict[str, Dict[str, Tuple[int, Expr]]] - builder: ir.IRBuilder = None - currentClass: str = None + builder: Optional[LLVMBuilder] = None + currentClass: Optional[str] = None def __init__(self, ts: TypeSystem): llvm.initialize() @@ -89,7 +95,7 @@ def enterScope(self): def exitScope(self): self.locals.pop() - def visit(self, node: Node): + def visit(self, node: Node) -> Any: return node.visit(self) def visitStmtList(self, stmts: List[Stmt]): @@ -105,7 +111,7 @@ def getClassVtableType(self, cls: str) -> ir.LiteralStructType: return ir.LiteralStructType(elements) def getClassStructType(self, cls: str) -> ir.LiteralStructType: - elements = [self.getClassVtableType( + elements: List[ir.Type] = [self.getClassVtableType( cls).as_pointer()] # pointer to vtable attrs = self.ts.getOrderedAttrs(cls) for attrInfo in attrs: @@ -192,12 +198,12 @@ def Program(self, node: Program): # declare global variables, methods, and functions varDefs = [d for d in node.declarations if isinstance(d, VarDef)] for d in varDefs: - t = d.var.t.getLLVMType() + t = d.var.getTypeX().getLLVMType() self.global_variable(d.var.name(), t) funcDefs = [d for d in node.declarations if isinstance(d, FuncDef)] for d in funcDefs: funcname = d.name.name - funcType = d.type.getLLVMType() + funcType = d.getTypeX().getLLVMType() ir.Function(self.module, funcType, funcname) classDefs = [d for d in node.declarations if isinstance(d, ClassDef)] for cls in classDefs: @@ -213,7 +219,7 @@ def Program(self, node: Program): ctor = self.methods[cls]["__init__"] if len(ctor.blocks) == 0: bb = ctor.append_basic_block('entry') - ir.IRBuilder(bb).ret(voidptr_t(None)) + LLVMBuilder(bb).ret(voidptr_t(None)) # define functions for d in funcDefs: @@ -225,40 +231,40 @@ def Program(self, node: Program): self.enterScope() entry_block = func.append_basic_block('entry') - self.builder = ir.IRBuilder(entry_block) + self.builder = LLVMBuilder(entry_block) - status = self.builder.call(self.externs['setjmp'], [ - self.module.get_global('__jmp_buf')]) - cond = self.builder.icmp_signed("!=", int32_t(0), status) + status = self.getBuilder().call(self.externs['setjmp'], [ + self.module.get_global('__jmp_buf')]) + cond = self.getBuilder().icmp_signed("!=", int32_t(0), status) - error_block = self.builder.append_basic_block('error_handling') - program_block = self.builder.append_basic_block('program_code') - end_program = self.builder.append_basic_block('end_program') - self.builder.cbranch(cond, - error_block, - program_block) + error_block = self.getBuilder().append_basic_block('error_handling') + program_block = self.getBuilder().append_basic_block('program_code') + end_program = self.getBuilder().append_basic_block('end_program') + self.getBuilder().cbranch(cond, + error_block, + program_block) - self.builder.position_at_start(error_block) + self.getBuilder().position_at_start(error_block) - error_code = self.builder.load(error_code) - error_line = self.builder.load(error_line) + error_code = self.getBuilder().load(error_code) + error_line = self.getBuilder().load(error_line) - assert_cond = self.builder.icmp_signed( + assert_cond = self.getBuilder().icmp_signed( '==', error_code, int32_t(ErrorCode.ASSERT)) - with self.builder.if_else(assert_cond) as (then_assert, else_): + with self.getBuilder().if_else(assert_cond) as (then_assert, else_): with then_assert: self.printf(self.module.get_global('__fmt_assert'), error_line) with else_: - null_cond = self.builder.icmp_signed( + null_cond = self.getBuilder().icmp_signed( '==', error_code, int32_t(ErrorCode.NULL_PTR)) - with self.builder.if_else(null_cond) as (null_assert, else__): + with self.getBuilder().if_else(null_cond) as (null_assert, else__): with null_assert: self.printf(self.module.get_global( '__fmt_null'), error_line) with else__: - oob_cond = self.builder.icmp_signed( + oob_cond = self.getBuilder().icmp_signed( '==', error_code, int32_t(ErrorCode.OUT_OF_BOUNDS)) - with self.builder.if_else(oob_cond) as (oob_assert, else____): + with self.getBuilder().if_else(oob_cond) as (oob_assert, else____): with oob_assert: self.printf(self.module.get_global( '__fmt_oob'), error_line) @@ -266,49 +272,49 @@ def Program(self, node: Program): self.printf(self.module.get_global( '__fmt_err'), error_line) - self.builder.branch(end_program) - error_block = self.builder.block + self.getBuilder().branch(end_program) + error_block = self.getBuilder().block - self.builder.position_at_start(program_block) + self.getBuilder().position_at_start(program_block) # initialize globals for d in varDefs: val = self.visit(d.value) addr = self.module.get_global(d.var.name()) assert addr is not None - self.builder.store(val, addr) + self.getBuilder().store(val, addr) self.visitStmtList(node.statements) - self.builder.branch(end_program) - self.builder.position_at_start(end_program) + self.getBuilder().branch(end_program) + self.getBuilder().position_at_start(end_program) assert not end_program.is_terminated - self.builder.ret_void() + self.getBuilder().ret_void() for block in func.blocks: - self.builder.position_at_end(block) + self.getBuilder().position_at_end(block) if not block.is_terminated: - self.builder.unreachable() + self.getBuilder().unreachable() self.exitScope() def VarDef(self, node: VarDef): val = self.visit(node.value) - saved_block = self.builder.block + saved_block = self.getBuilder().block if node.isAttr: raise Exception("this should be handled elsewhere") - elif node.var.varInstance.isNonlocal: - addr = self.builder.alloca( - node.var.t.getLLVMType(), None, node.getName()) - wrapper = self.builder.alloca( - node.var.t.getLLVMType().as_pointer(), None, node.getName() + "_wrapper") - self.builder.position_at_end(saved_block) - self.builder.store(val, addr) - self.builder.store(addr, wrapper) + elif node.var.varInstanceX().isNonlocal: + addr = self.getBuilder().alloca( + node.var.getTypeX().getLLVMType(), None, node.getName()) + wrapper = self.getBuilder().alloca( + node.var.getTypeX().getLLVMType().as_pointer(), None, node.getName() + "_wrapper") + self.getBuilder().position_at_end(saved_block) + self.getBuilder().store(val, addr) + self.getBuilder().store(addr, wrapper) self.locals[-1][node.getName()] = wrapper else: - addr = self.builder.alloca( - node.var.t.getLLVMType(), None, node.getName()) - self.builder.position_at_end(saved_block) - self.builder.store(val, addr) + addr = self.getBuilder().alloca( + node.var.getTypeX().getLLVMType(), None, node.getName()) + self.getBuilder().position_at_end(saved_block) + self.getBuilder().store(val, addr) self.locals[-1][node.getName()] = addr def ClassDef(self, node: ClassDef): @@ -317,74 +323,75 @@ def ClassDef(self, node: ClassDef): def FuncDef(self, node: FuncDef): fname = node.getIdentifier().name if node.isMethod: + assert self.currentClass is not None func = self.module.get_global( self.currentClass + "__" + fname) else: func = self.module.get_global(fname) - self.returnType = node.type.returnType + self.returnType = node.getTypeX().returnType implicitReturn = self.returnType not in { IntType(), BoolType(), StrType()} self.enterScope() bb_entry = func.append_basic_block('entry') - self.builder = ir.IRBuilder(bb_entry) + self.builder = LLVMBuilder(bb_entry) for i, arg in enumerate(func.args): arg.name = node.params[i].name() - alloca = self.builder.alloca( - node.type.getLLVMType().args[i], name=arg.name) - self.builder.store(arg, alloca) + alloca = self.getBuilder().alloca( + node.getTypeX().getLLVMType().args[i], name=arg.name) + self.getBuilder().store(arg, alloca) self.locals[-1][arg.name] = alloca for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) # implicitly return None if needed, close all blocks for block in func.blocks: - self.builder.position_at_end(block) + self.getBuilder().position_at_end(block) if not block.is_terminated: if implicitReturn: - self.builder.ret(self.NoneLiteral(None)) + self.getBuilder().ret(self.NoneLiteral(None)) else: - self.builder.unreachable() + self.getBuilder().unreachable() self.exitScope() return func # STATEMENTS - def getAttrPtr(self, obj, cls: str, attr: str): + def getAttrPtr(self, obj: ir.Value, cls: str, attr: str): offset, _ = self.attrOffsets[cls][attr] - obj = self.builder.bitcast(obj, self.structs[cls].as_pointer()) - attr_ptr = self.builder.gep(obj, [int32_t(0), int32_t(offset)]) + obj = self.getBuilder().cast(obj, self.structs[cls].as_pointer()) + attr_ptr = self.getBuilder().gep(obj, [int32_t(0), int32_t(offset)]) return attr_ptr def AssignStmt(self, node: AssignStmt): val = self.visit(node.value) for var in node.targets[::-1]: if isinstance(var, MemberExpr): - cls = var.object.inferredType.className + cls = cast(ClassValueType, var.object.inferredType).className attr = var.member.name obj = self.visit(var.object) self.assert_nonnull(obj, var.object.location[0]) ptr = self.getAttrPtr(obj, cls, attr) - self.builder.store(val, ptr) + self.getBuilder().store(val, ptr) elif isinstance(var, IndexExpr): lst = self.visit(var.list) idx = self.visit(var.index) self.assert_nonnull(lst, var.list.location[0]) ptr = self.listIndex( - lst, idx, var.inferredType.getLLVMType(), True, var.index.location[0]) - self.builder.store(val, ptr) + lst, idx, var.inferredValueType().getLLVMType(), True, var.index.location[0]) + self.getBuilder().store(val, ptr) elif isinstance(var, Identifier): addr = self.getAddr(var) - self.builder.store(val, addr) + self.getBuilder().store(val, addr) else: raise Exception("Illegal assignment") def IfStmt(self, node: IfStmt): cond = self.visit(node.condition) if len(node.elseBody) == 0: - with self.builder.if_then(cond): + with self.getBuilder().if_then(cond): self.visitStmtList(node.thenBody) else: - with self.builder.if_else(cond) as (then, else_): + with self.getBuilder().if_else(cond) as (then, else_): with then: self.visitStmtList(node.thenBody) with else_: @@ -396,15 +403,15 @@ def ExprStmt(self, node: ExprStmt): def isListConcat(self, operator: str, leftType: ValueType, rightType: ValueType) -> bool: return leftType.isListType() and rightType.isListType() and operator == "+" - def getListDataPtr(self, lst, elemType): - lst = self.builder.bitcast(lst, int32_t.as_pointer()) - lst = self.builder.gep(lst, [int32_t(1)]) - return self.builder.bitcast(lst, elemType.as_pointer()) + def getListDataPtr(self, lst: ir.Value, elemType: ir.Type) -> ir.Value: + lst = self.getBuilder().cast(lst, int32_t.as_pointer()) + lst = self.getBuilder().gep(lst, [int32_t(1)]) + return self.getBuilder().cast(lst, elemType.as_pointer()) def BinaryExpr(self, node: BinaryExpr): operator = node.operator - leftType = node.left.inferredType - rightType = node.right.inferredType + leftType = node.left.inferredValueType() + rightType = node.right.inferredValueType() lhs = self.visit(node.left) rhs = self.visit(node.right) # concatenation and addition @@ -414,95 +421,98 @@ def BinaryExpr(self, node: BinaryExpr): rhs = self.toVoidPtr(rhs) llen = self.list_len(lhs) rlen = self.list_len(rhs) - total_len = self.builder.add(llen, rlen, 'total_len') + total_len = self.getBuilder().add(llen, rlen, 'total_len') if node.inferredType == EmptyType(): elemType = int8_t else: - elemType = node.inferredType.elementType.getLLVMType() + elemType = cast( + ListValueType, node.inferredType).elementType.getLLVMType() assert elemType is not None - size = self.builder.add(int32_t(4), self.builder.mul( + size = self.getBuilder().add(int32_t(4), self.getBuilder().mul( total_len, self.sizeof(elemType)), 'bytes') - new_arr = self.builder.call( + new_arr = self.getBuilder().call( self.externs['malloc'], [size], 'new_list') - size_ptr = self.builder.bitcast(new_arr, int32_t.as_pointer()) - self.builder.store(total_len, size_ptr) + size_ptr = self.getBuilder().cast(new_arr, int32_t.as_pointer()) + self.getBuilder().store(total_len, size_ptr) data_lhs_start = self.getListDataPtr(new_arr, elemType) lhs_data = self.getListDataPtr(lhs, elemType) rhs_data = self.getListDataPtr(rhs, elemType) - lhs_bytes = self.builder.mul(llen, self.sizeof(elemType)) + lhs_bytes = self.getBuilder().mul(llen, self.sizeof(elemType)) - self.builder.call(self.externs['memcpy'], [ + self.getBuilder().call(self.externs['memcpy'], [ self.toVoidPtr(data_lhs_start), self.toVoidPtr(lhs_data), lhs_bytes]) - data_rhs_start = self.builder.gep(data_lhs_start, [llen]) - rhs_bytes = self.builder.mul(rlen, self.sizeof(elemType)) + data_rhs_start = self.getBuilder().gep(data_lhs_start, [llen]) + rhs_bytes = self.getBuilder().mul(rlen, self.sizeof(elemType)) - self.builder.call(self.externs['memcpy'], [ - self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) + self.getBuilder().call(self.externs['memcpy'], [ + self.toVoidPtr(data_rhs_start), self.toVoidPtr(rhs_data), rhs_bytes]) return new_arr elif leftType == StrType(): lhs = self.toVoidPtr(lhs) rhs = self.toVoidPtr(rhs) - llen = self.builder.call(self.externs['strlen'], [lhs]) - rlen = self.builder.call(self.externs['strlen'], [rhs]) - total_len = self.builder.add(self.builder.add( + llen = self.getBuilder().call(self.externs['strlen'], [lhs]) + rlen = self.getBuilder().call(self.externs['strlen'], [rhs]) + total_len = self.getBuilder().add(self.getBuilder().add( llen, rlen), int32_t(1)) - new_str = self.builder.call( + new_str = self.getBuilder().call( self.externs['malloc'], [total_len], 'new_str') fmt = self.toVoidPtr( self.module.get_global('__fmt_str_concat')) - self.builder.call(self.externs['sprintf'], [ - new_str, fmt, lhs, rhs]) + self.getBuilder().call(self.externs['sprintf'], [ + new_str, fmt, lhs, rhs]) return new_str elif leftType == IntType(): - return self.builder.add(lhs, rhs) + return self.getBuilder().add(lhs, rhs) else: raise Exception( "Internal compiler error: unexpected operand types for +") # other arithmetic operators elif operator == "-": - return self.builder.sub(lhs, rhs) + return self.getBuilder().sub(lhs, rhs) elif operator == "*": - return self.builder.mul(lhs, rhs) + return self.getBuilder().mul(lhs, rhs) elif operator == "//": - return self.builder.sdiv(lhs, rhs) + return self.getBuilder().sdiv(lhs, rhs) elif operator == "%": # emulate Python modulo with ((a % b) + b) % b) - val = self.builder.srem(lhs, rhs) - val = self.builder.add(val, rhs) - return self.builder.srem(val, rhs) + val = self.getBuilder().srem(lhs, rhs) + val = self.getBuilder().add(val, rhs) + return self.getBuilder().srem(val, rhs) # relational operators elif operator in {"<", "<=", ">", ">="}: - return self.builder.icmp_signed(operator, lhs, rhs) + return self.getBuilder().icmp_signed(operator, lhs, rhs) elif operator == "==": if leftType == IntType(): - return self.builder.icmp_signed(operator, lhs, rhs) + return self.getBuilder().icmp_signed(operator, lhs, rhs) elif leftType == StrType(): - cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) - return self.builder.icmp_signed("==", cmp, int32_t(0)) + cmp = self.getBuilder().call( + self.externs['strcmp'], [lhs, rhs]) + return self.getBuilder().icmp_signed("==", cmp, int32_t(0)) else: # bool - return self.builder.icmp_signed(operator, lhs, rhs) + return self.getBuilder().icmp_signed(operator, lhs, rhs) elif operator == "!=": if leftType == IntType(): - return self.builder.icmp_signed(operator, lhs, rhs) + return self.getBuilder().icmp_signed(operator, lhs, rhs) elif leftType == StrType(): - cmp = self.builder.call(self.externs['strcmp'], [lhs, rhs]) - return self.builder.icmp_signed("!=", cmp, int32_t(0)) + cmp = self.getBuilder().call( + self.externs['strcmp'], [lhs, rhs]) + return self.getBuilder().icmp_signed("!=", cmp, int32_t(0)) else: # bool - return self.builder.icmp_signed(operator, lhs, rhs) + return self.getBuilder().icmp_signed(operator, lhs, rhs) elif operator == "is": # pointer comparisons - lhs_ptr = self.builder.ptrtoint(lhs, int32_t) - rhs_ptr = self.builder.ptrtoint(rhs, int32_t) - return self.builder.icmp_unsigned("==", lhs_ptr, rhs_ptr) + lhs_ptr = self.getBuilder().ptrtoint(lhs, int32_t) + rhs_ptr = self.getBuilder().ptrtoint(rhs, int32_t) + return self.getBuilder().icmp_unsigned("==", lhs_ptr, rhs_ptr) # logical operators elif operator == "and": - return self.builder.and_(lhs, rhs) + return self.getBuilder().and_(lhs, rhs) elif operator == "or": - return self.builder.or_(lhs, rhs) + return self.getBuilder().or_(lhs, rhs) else: raise Exception( f"Internal compiler error: unexpected operator {operator}") @@ -517,88 +527,90 @@ def IndexExpr(self, node: IndexExpr): idx = self.visit(node.index) self.assert_nonnull(lst, node.list.location[0]) ptr = self.listIndex(lst, idx, - node.inferredType.getLLVMType(), + node.inferredValueType().getLLVMType(), True, node.index.location[0]) - return self.builder.load(ptr) + return self.getBuilder().load(ptr) - def listIndex(self, list, index, elemType, check_bounds=False, line: int = 0): + def listIndex(self, list: ir.Value, index: ir.Value, elemType: ir.Type, check_bounds: bool = False, line: int = 0) -> ir.GEPInstr: # return pointer to list[index] if check_bounds: assert line != 0 - min_idx = self.builder.icmp_signed('>', int32_t(0), index) - with self.builder.if_then(min_idx): + min_idx = self.getBuilder().icmp_signed('>', int32_t(0), index) + with self.getBuilder().if_then(min_idx): self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) length = self.list_len(list) - max_idx = self.builder.icmp_signed('<=', length, index) - with self.builder.if_then(max_idx): + max_idx = self.getBuilder().icmp_signed('<=', length, index) + with self.getBuilder().if_then(max_idx): self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) data = self.getListDataPtr(list, elemType) # return pointer to value in array - return self.builder.gep(data, [index]) + return self.getBuilder().gep(data, [index]) - def strIndex(self, string, index, check_bounds=False, line: int = 0): + def strIndex(self, string: ir.Value, index: ir.Value, check_bounds: bool = False, line: int = 0) -> ir.Value: string = self.toVoidPtr(string) # bounds checks if check_bounds: assert line != 0 - min_idx = self.builder.icmp_signed('>', int32_t(0), index) - with self.builder.if_then(min_idx): + min_idx = self.getBuilder().icmp_signed('>', int32_t(0), index) + with self.getBuilder().if_then(min_idx): self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) - length = self.builder.call(self.externs['strlen'], [string]) - max_idx = self.builder.icmp_signed('<=', length, index) - with self.builder.if_then(max_idx): + length = self.getBuilder().call(self.externs['strlen'], [string]) + max_idx = self.getBuilder().icmp_signed('<=', length, index) + with self.getBuilder().if_then(max_idx): self.longJmp(ErrorCode.OUT_OF_BOUNDS, line) - ptr = self.builder.gep(string, [index]) - char = self.builder.load(ptr) - addr = self.builder.call(self.externs['malloc'], [ - int32_t(2)], 'char') + ptr = self.getBuilder().gep(string, [index]) + char = self.getBuilder().load(ptr) + addr = self.getBuilder().call(self.externs['malloc'], [ + int32_t(2)], 'char') addr = self.toVoidPtr(addr) - char_ptr = self.builder.gep(addr, [int32_t(0)]) - self.builder.store(char, char_ptr, 8) - t_ptr = self.builder.gep(addr, [int32_t(1)]) - self.builder.store(int8_t(0), t_ptr, 8) + char_ptr = self.getBuilder().gep(addr, [int32_t(0)]) + self.getBuilder().store(char, char_ptr, 8) + t_ptr = self.getBuilder().gep(addr, [int32_t(1)]) + self.getBuilder().store(int8_t(0), t_ptr, 8) return addr def UnaryExpr(self, node: UnaryExpr): if node.operator == "-": val = self.visit(node.operand) - return self.builder.neg(val) + return self.getBuilder().neg(val) elif node.operator == "not": val = self.visit(node.operand) - return self.builder.icmp_unsigned('==', bool_t(0), val) + return self.getBuilder().icmp_unsigned('==', bool_t(0), val) - def constructor(self, node: CallExpr): + def constructor(self, node: CallExpr) -> ir.Value: cls = node.function.name size = self.sizeof(self.structs[cls]) - obj = self.builder.call(self.externs['malloc'], [size], 'new_object') + obj = self.getBuilder().call( + self.externs['malloc'], [size], 'new_object') # initialize fields for attr in self.attrOffsets[cls]: _, val = self.attrOffsets[cls][attr] ptr = self.getAttrPtr(obj, cls, attr) - self.builder.store(self.visit(val), ptr) + self.getBuilder().store(self.visit(val), ptr) # set vtable pointer - vtable_ptr = self.builder.bitcast(obj, voidptr_t.as_pointer()) + vtable_ptr = self.getBuilder().cast(obj, voidptr_t.as_pointer()) vtable = self.module.get_global("__" + cls + "__vtable") - vtable = self.builder.bitcast(vtable, voidptr_t) - self.builder.store(vtable, vtable_ptr) + vtable = self.getBuilder().cast(vtable, voidptr_t) + self.getBuilder().store(vtable, vtable_ptr) # call __init__ method - self.builder.call(self.methods[cls]["__init__"], [obj]) + self.getBuilder().call(self.methods[cls]["__init__"], [obj]) return obj - def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): - argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr) -> ir.Value: + argIsRef = isinstance( + arg, Identifier) and arg.varInstanceX().isNonlocal paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and cast(Identifier, arg).varInstanceX() == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg - return self.getAddr(arg) + return self.getAddr(cast(Identifier, arg)) elif paramIsRef: # non-ref arg and ref param, or do not pass ref arg # unwrap if necessary, re-wrap - saved_block = self.builder.block + saved_block = self.getBuilder().block val = self.visit(arg) - addr = self.builder.alloca(arg.inferredType.getLLVMType()) - self.builder.position_at_end(saved_block) - self.builder.store(val, addr) + addr = self.getBuilder().alloca(arg.inferredValueType().getLLVMType()) + self.getBuilder().position_at_end(saved_block) + self.getBuilder().store(val, addr) return addr else: # non-ref param, maybe unwrap return self.visit(arg) @@ -624,43 +636,44 @@ def CallExpr(self, node: CallExpr): call_args = [] for i in range(len(node.args)): call_args.append(self.visitArg( - node.function.inferredType, i, node.args[i])) - return self.builder.call(callee_func, call_args, 'calltmp') + cast(FuncType, node.function.inferredType), i, node.args[i])) + return self.getBuilder().call(callee_func, call_args, 'calltmp') + + def forBody(self, node: ForStmt, var: ir.Value, idxFn: Callable[[ir.Value], ir.Value], idx_var: ir.Value): + currIdx = self.getBuilder().load(idx_var) + self.getBuilder().store(idxFn(currIdx), var) + self.visitStmtList(node.body) + self.getBuilder().store(self.getBuilder().add(currIdx, int32_t(1)), idx_var) def ForStmt(self, node: ForStmt): var = self.getAddr(node.identifier) - idx_var = self.builder.alloca(int32_t, None, 'idx') - self.builder.store(int32_t(0), idx_var) + idx_var = self.getBuilder().alloca(int32_t, None, 'idx') + self.getBuilder().store(int32_t(0), idx_var) iterable = self.visit(node.iterable) if node.iterable.inferredType == StrType(): self.whileHelper( - lambda: self.builder.icmp_signed("<", - self.builder.load(idx_var), - self.builder.call(self.externs['strlen'], [iterable])), + lambda: self.getBuilder().icmp_signed("<", + self.getBuilder().load(idx_var), + self.getBuilder().call(self.externs['strlen'], [iterable])), lambda: self.forBody(node, var, - lambda currIdx: self.strIndex(iterable, currIdx), + lambda currIdx: self.strIndex( + iterable, currIdx), idx_var)) else: self.assert_nonnull(iterable, node.iterable.location[0]) length = self.list_len(iterable) self.whileHelper( - lambda: self.builder.icmp_signed("<", - self.builder.load(idx_var), - length), + lambda: self.getBuilder().icmp_signed("<", + self.getBuilder().load(idx_var), + length), lambda: self.forBody(node, var, - lambda currIdx: self.builder.load(self.listIndex( - iterable, currIdx, node.identifier.inferredType.getLLVMType())), + lambda currIdx: self.getBuilder().load(self.listIndex( + iterable, currIdx, node.identifier.inferredValueType().getLLVMType())), idx_var)) - def forBody(self, node: ForStmt, var, idxFn, idx_var): - currIdx = self.builder.load(idx_var) - self.builder.store(idxFn(currIdx), var) - self.visitStmtList(node.body) - self.builder.store(self.builder.add(currIdx, int32_t(1)), idx_var) - def ListExpr(self, node: ListExpr): n = len(node.elements) if n == 0: @@ -670,21 +683,22 @@ def ListExpr(self, node: ListExpr): # fallback to voidptr elemType = int8_t else: - elemType = node.inferredType.elementType.getLLVMType() + elemType = cast( + ListValueType, node.inferredType).elementType.getLLVMType() assert elemType is not None - size = self.builder.add(int32_t(4), self.builder.mul( + size = self.getBuilder().add(int32_t(4), self.getBuilder().mul( int32_t(n), self.sizeof(elemType))) - addr = self.builder.call(self.externs['malloc'], [ - size], 'list_literal') - addr = self.builder.bitcast(addr, int32_t.as_pointer()) + addr = self.getBuilder().call(self.externs['malloc'], [ + size], 'list_literal') + addr = self.getBuilder().cast(addr, int32_t.as_pointer()) for i in range(n): value = self.visit(node.elements[i]) data = self.getListDataPtr(addr, elemType) - idx_ptr = self.builder.gep(data, [int32_t(i)]) - self.builder.store(value, idx_ptr) - len_ptr = self.builder.gep( + idx_ptr = self.getBuilder().gep(data, [int32_t(i)]) + self.getBuilder().store(value, idx_ptr) + len_ptr = self.getBuilder().gep( addr, [int32_t(0)]) - self.builder.store(int32_t(n), len_ptr) + self.getBuilder().store(int32_t(n), len_ptr) addr = self.toVoidPtr(addr) return addr @@ -693,46 +707,46 @@ def WhileStmt(self, node: WhileStmt): lambda: self.visit(node.condition), lambda: self.visitStmtList(node.body)) - def whileHelper(self, condFn, bodyFn): - while_block = self.builder.append_basic_block('while') - do_block = self.builder.append_basic_block('do') - end_block = self.builder.append_basic_block('end') - self.builder.branch(while_block) + def whileHelper(self, condFn: Callable, bodyFn: Callable): + while_block = self.getBuilder().append_basic_block('while') + do_block = self.getBuilder().append_basic_block('do') + end_block = self.getBuilder().append_basic_block('end') + self.getBuilder().branch(while_block) - self.builder.position_at_start(while_block) + self.getBuilder().position_at_start(while_block) cond = condFn() - self.builder.cbranch(cond, - do_block, - end_block) - while_block = self.builder.block + self.getBuilder().cbranch(cond, do_block, end_block) + while_block = self.getBuilder().block - self.builder.position_at_start(do_block) + self.getBuilder().position_at_start(do_block) bodyFn() - if not self.builder.block.is_terminated: - self.builder.branch(while_block) - do_block = self.builder.block + builder = self.getBuilder() + if builder.block and not builder.block.is_terminated: + self.getBuilder().branch(while_block) + do_block = self.getBuilder().block - self.builder.position_at_start(end_block) + self.getBuilder().position_at_start(end_block) def ReturnStmt(self, node: ReturnStmt): - assert not self.builder.block.is_terminated + builder = self.getBuilder() + assert builder.block and not builder.block.is_terminated if self.returnType.isNone(): - self.builder.ret(self.NoneLiteral(None)) + self.getBuilder().ret(self.NoneLiteral(None)) else: val = None if node.value is None: val = self.NoneLiteral(None) else: val = self.visit(node.value) - self.builder.ret(val) + self.getBuilder().ret(val) - def getAddr(self, node: Identifier): - if node.varInstance.isGlobal: + def getAddr(self, node: Identifier) -> ir.Value: + if node.varInstanceX().isGlobal: return self.module.get_global(node.name) - elif node.varInstance.isNonlocal: + elif node.varInstanceX().isNonlocal: addr = self.locals[-1][node.name] assert addr is not None - return self.builder.load(addr) + return self.getBuilder().load(addr) else: addr = self.locals[-1][node.name] assert addr is not None @@ -740,50 +754,52 @@ def getAddr(self, node: Identifier): def Identifier(self, node: Identifier): addr = self.getAddr(node) - return self.builder.load(addr, node.name) + return self.getBuilder().load(addr, node.name) def MemberExpr(self, node: MemberExpr): - cls = node.object.inferredType.className + cls = cast(ClassValueType, node.object.inferredType).className attr = node.member.name obj = self.visit(node.object) self.assert_nonnull(obj, node.object.location[0]) ptr = self.getAttrPtr(obj, cls, attr) - return self.builder.load(ptr, attr) + return self.getBuilder().load(ptr, attr) def IfExpr(self, node: IfExpr): cond = self.visit(node.condition) - with self.builder.if_else(cond) as (then, else_): + with self.getBuilder().if_else(cond) as (then, else_): with then: then_val = self.visit(node.thenExpr) - then_block = self.builder.block + then_block = self.getBuilder().block with else_: else_val = self.visit(node.elseExpr) - else_block = self.builder.block - phi = self.builder.phi(node.inferredType.getLLVMType(), 'phi') + else_block = self.getBuilder().block + phi = self.getBuilder().phi(node.inferredValueType().getLLVMType(), 'phi') phi.add_incoming(then_val, then_block) phi.add_incoming(else_val, else_block) return phi def MethodCallExpr(self, node: MethodCallExpr): - className = node.method.object.inferredType.className + className = cast( + ClassValueType, node.method.object.inferredType).className obj = self.visit(node.method.object) - obj = self.builder.bitcast(obj, self.structs[className].as_pointer()) + obj = self.getBuilder().cast( + obj, self.structs[className].as_pointer()) methName = node.method.member.name methIdx, _ = self.methodOffsets[(className, methName)] - vtable_ptr = self.builder.gep(obj, [int32_t(0), int32_t(0)]) - vtable = self.builder.load(vtable_ptr) + vtable_ptr = self.getBuilder().gep(obj, [int32_t(0), int32_t(0)]) + vtable = self.getBuilder().load(vtable_ptr) - callee_func_ptr = self.builder.gep(self.builder.gep( + callee_func_ptr = self.getBuilder().gep(self.getBuilder().gep( vtable, [int32_t(0), int32_t(methIdx)]), [int32_t(0)]) - callee_func = self.builder.load(callee_func_ptr) - - call_args = [self.builder.bitcast(obj, voidptr_t)] + callee_func = self.getBuilder().load(callee_func_ptr) + call_args: List[ir.Value] = [ + self.getBuilder().cast(obj, voidptr_t)] for i in range(len(node.args)): call_args.append(self.visitArg( - node.method.inferredType, i + 1, node.args[i])) - return self.builder.call(callee_func, call_args, 'callmethodtmp') + cast(FuncType, node.method.inferredType), i + 1, node.args[i])) + return self.getBuilder().call(callee_func, call_args, 'callmethodtmp') # LITERALS @@ -793,134 +809,139 @@ def BooleanLiteral(self, node: BooleanLiteral): def IntegerLiteral(self, node: IntegerLiteral): return int32_t(node.value) - def NoneLiteral(self, _: NoneLiteral): + def NoneLiteral(self, node: Optional[NoneLiteral]): return voidptr_t(None) def StringLiteral(self, node: StringLiteral): bytes = bytearray((node.value + '\00').encode('ascii')) size = int32_t(1 + len(node.value)) - addr = self.builder.call(self.externs['malloc'], [size], 'str_literal') + addr = self.getBuilder().call( + self.externs['malloc'], [size], 'str_literal') for i in range(len(bytes)): - idx_ptr = self.builder.gep(addr, [int32_t(i)]) - self.builder.store(int8_t(bytes[i]), idx_ptr) + idx_ptr = self.getBuilder().gep(addr, [int32_t(i)]) + self.getBuilder().store(int8_t(bytes[i]), idx_ptr) return addr # BUILT-INS - def emit_len(self, arg: Expr): + def emit_len(self, arg: Expr) -> ir.Value: val = self.visit(arg) if arg.inferredType == StrType(): val = self.toVoidPtr(val) - return self.builder.call(self.externs['strlen'], [val]) + return self.getBuilder().call(self.externs['strlen'], [val]) else: return self.list_len(val) - def assert_nonnull(self, val, line): + def assert_nonnull(self, val: ir.Value, line: int): val = self.toVoidPtr(val) - cond = self.builder.icmp_signed('==', voidptr_t(None), val) - with self.builder.if_then(cond): + cond = self.getBuilder().icmp_signed('==', voidptr_t(None), val) + with self.getBuilder().if_then(cond): self.longJmp(ErrorCode.NULL_PTR, line) - def list_len(self, arg): - val = self.builder.bitcast(arg, int32_t.as_pointer()) - return self.builder.load(val, 'len') + def list_len(self, arg: ir.Value) -> ir.Value: + val = self.getBuilder().cast(arg, int32_t.as_pointer()) + return self.getBuilder().load(val, 'len') def emit_assert(self, arg: Expr): line = arg.location[0] arg = self.visit(arg) - cond = self.builder.icmp_unsigned('==', bool_t(0), arg) - with self.builder.if_then(cond): + cond = self.getBuilder().icmp_unsigned('==', bool_t(0), arg) + with self.getBuilder().if_then(cond): self.longJmp(ErrorCode.ASSERT, line) def longJmp(self, code: int, line: int): code_addr = self.module.get_global("__error_code") - self.builder.store(int32_t(code), code_addr) + self.getBuilder().store(int32_t(code), code_addr) line_addr = self.module.get_global("__error_line") - self.builder.store(int32_t(line), line_addr) + self.getBuilder().store(int32_t(line), line_addr) jmp_buf = self.module.get_global('__jmp_buf') - self.builder.call(self.externs['longjmp'], [ - jmp_buf, int32_t(1)]) - self.builder.unreachable() + self.getBuilder().call(self.externs['longjmp'], [jmp_buf, int32_t(1)]) + self.getBuilder().unreachable() - def emit_print(self, arg: Expr): - if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: + def emit_print(self, arg: Expr) -> ir.Constant: + if isinstance(arg.inferredType, ListValueType) or cast(ClassValueType, arg.inferredType).className not in {"bool", "int", "str"}: raise Exception("Only bool, int, or str may be printed") if arg.inferredType == BoolType(): cond = self.visit(arg) - with self.builder.if_else(cond) as (then, else_): + with self.getBuilder().if_else(cond) as (then, else_): with then: then_text = self.toVoidPtr( self.module.get_global('__true')) - then_block = self.builder.block + then_block = self.getBuilder().block with else_: else_text = self.toVoidPtr( self.module.get_global('__false')) - else_block = self.builder.block - phi = self.builder.phi(voidptr_t, 'phi') + else_block = self.getBuilder().block + phi = self.getBuilder().phi(voidptr_t, 'phi') phi.add_incoming(then_text, then_block) phi.add_incoming(else_text, else_block) self.printf(self.module.get_global('__fmt_s'), phi) - elif arg.inferredType.className == 'int': + elif cast(ClassValueType, arg.inferredType).className == 'int': self.printf(self.module.get_global('__fmt_i'), self.visit(arg)) else: self.printf(self.module.get_global('__fmt_s'), self.visit(arg)) return self.NoneLiteral(None) - def emit_input(self): + def emit_input(self) -> ir.Value: # get input from user input_buf = self.toVoidPtr(self.module.get_global("__input_buf")) fmt = self.toVoidPtr(self.module.get_global('__fmt_input')) - self.builder.call(self.externs['scanf'], [fmt, input_buf]) + self.getBuilder().call(self.externs['scanf'], [fmt, input_buf]) # copy contents into new string so that input buffer can be reused - len = self.builder.call(self.externs['strlen'], [input_buf]) - new_str = self.builder.call( - self.externs['malloc'], [self.builder.add(len, int32_t(1))], 'new_str') + len = self.getBuilder().call(self.externs['strlen'], [input_buf]) + new_str = self.getBuilder().call( + self.externs['malloc'], [self.getBuilder().add(len, int32_t(1))], 'new_str') fmt = self.toVoidPtr(self.module.get_global('__fmt_str')) - self.builder.call(self.externs['sprintf'], [new_str, fmt, input_buf]) + self.getBuilder().call(self.externs['sprintf'], [ + new_str, fmt, input_buf]) return new_str # UTILS - def make_bytearray(self, buf): + def make_bytearray(self, buf: bytes) -> ir.Constant: b = bytearray(buf) n = len(b) return ir.Constant(ir.ArrayType(int8_t, n), b) - def printf(self, format, arg): + def printf(self, format: ir.Value, arg: ir.Value) -> ir.Value: fmt_ptr = self.toVoidPtr(format) - return self.builder.call(self.externs['printf'], [fmt_ptr, arg]) + return self.getBuilder().call(self.externs['printf'], [fmt_ptr, arg]) - def global_constant(self, name, t, value): + def global_constant(self, name: str, t: ir.Type, value: ir.Constant) -> ir.GlobalVariable: module = self.module data = ir.GlobalVariable(module, t, name) data.linkage = 'internal' data.global_constant = True - data.initializer = value + data.initializer = value # type: ignore return data - def global_variable(self, name, t): + def global_variable(self, name: str, t: ir.Type) -> ir.GlobalVariable: module = self.module data = ir.GlobalVariable(module, t, name) data.linkage = 'internal' - data.initializer = t(None) + data.initializer = t(None) # type: ignore data.global_constant = False return data - def sizeof(self, t): + def sizeof(self, t: ir.Type) -> ir.Value: if not (t.is_pointer or isinstance(t, ir.LiteralStructType)): - width = t.width + width = cast(ir.IntType, t).width # each item in array must be at least 1 byte if width < 8: - return int32_t(1) - return int32_t(width // 8) + return int32_t(1) # type: ignore + return int32_t(width // 8) # type: ignore else: null = t.as_pointer()(None) offset = null.gep([int32_t(1)]) - size = self.builder.ptrtoint(offset, int32_t, 'sizeof') - return size + size = self.getBuilder().ptrtoint(offset, int32_t, 'sizeof') + return size # type: ignore + + def toVoidPtr(self, ptr: ir.Value) -> ir.Value: + return self.getBuilder().cast(ptr, voidptr_t) - def toVoidPtr(self, ptr): - return self.builder.bitcast(ptr, voidptr_t) + def getBuilder(self) -> LLVMBuilder: + assert self.builder + return self.builder diff --git a/compiler/nestedfunchoister.py b/compiler/nestedfunchoister.py index 04ea0d5..56fff90 100644 --- a/compiler/nestedfunchoister.py +++ b/compiler/nestedfunchoister.py @@ -1,7 +1,7 @@ from .astnodes import * from .types import * from .visitor import Visitor -from typing import List, Dict +from typing import List, Dict, Optional class HoistedFunctionInfo: @@ -14,7 +14,7 @@ class NestedFuncHoister(Visitor): # hoist all nested funcs to be top level funcs # rename hoisted functions to be unique & rename call sites functionInfo: List[Dict[str, HoistedFunctionInfo]] - currentClass: str + currentClass: Optional[str] nestingNames: List[str] hoisted: List[FuncDef] diff --git a/compiler/parser.py b/compiler/parser.py index 12cd080..3bad1e7 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -32,7 +32,7 @@ def getLocation(self, node: ast.AST) -> typing.List[int]: # make columns 1-indexed return [node.lineno, node.col_offset + 1] - def visit(self, node: ast.AST): + def visit(self, node: ast.AST) -> typing.Any: try: return super().visit(node) except ParseError as e: @@ -242,7 +242,7 @@ def visit_Expr(self, node: ast.Expr) -> ExprStmt: val = self.visit(node.value) return ExprStmt(location, val) - def visit_Pass(self, _: ast.Pass) -> None: + def visit_Pass(self, node: ast.Pass) -> None: # removed by any AST constructors that take in [Stmt] return None @@ -342,9 +342,6 @@ def visit_NameConstant(self, node: ast.NameConstant) -> Expr: else: raise ParseError("Unsupported name constant", node) - def visit_Index(self, node: ast.Index): - return self.visit(node.value) - def visit_arguments(self, node: ast.arguments) -> list: if node.vararg: raise ParseError("Unsupported vararg", node.vararg) diff --git a/compiler/python_backend.py b/compiler/python_backend.py index 4b42db2..f533a9d 100644 --- a/compiler/python_backend.py +++ b/compiler/python_backend.py @@ -7,7 +7,7 @@ class PythonBackend(Visitor): def __init__(self): - self.builder = Builder(None) + self.builder = Builder("") def visit(self, node: Node): return node.visit(self) @@ -141,7 +141,8 @@ def visitArg(self, node, funcType, paramIdx: int, argIdx: int): if isinstance(arg, Identifier) and arg.varInstance is None: self.visit(arg) return - argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + argIsRef = isinstance( + arg, Identifier) and arg.varInstance is not None and arg.varInstance.isNonlocal paramIsRef = paramIdx in funcType.refParams if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg diff --git a/compiler/typechecker.py b/compiler/typechecker.py index 96a2cbe..fd56904 100644 --- a/compiler/typechecker.py +++ b/compiler/typechecker.py @@ -3,15 +3,15 @@ from collections import defaultdict from .typesystem import TypeSystem, ClassInfo from .visitor import Visitor -from typing import List, Optional +from typing import List, Optional, Any, assert_type class TypeChecker(Visitor): symbolTable: List[defaultdict] - currentClass: str + currentClass: Optional[str] errors: List[CompilerError] expReturnType: Optional[ValueType] - program: Program + program: Optional[Program] def __init__(self, ts: TypeSystem): # typechecker attributes and their chocopy typing judgement analogues: @@ -39,7 +39,7 @@ def __init__(self, ts: TypeSystem): self.program = None self.addErrors = True - def visit(self, node: Node): + def visit(self, node: Node) -> Any: if isinstance(node, Program) or isinstance(node, ClassDef) or isinstance(node, FuncDef): return node.visit(self) else: @@ -97,9 +97,10 @@ def addError(self, node: Node, message: str): return message = F"{message}. Line {node.location[0]} Col {node.location[1]}" node.errorMsg = message + assert self.program is not None self.program.errors.errors.append( CompilerError(node.location, message)) - self.errors.append(message) + self.errors.append(CompilerError(node.location, message)) def binopError(self, node): self.addError(node, "Cannot use operator {} on types {} and {}".format( @@ -117,7 +118,7 @@ def Program(self, node: Program): identifier = d.getIdentifier() if self.defInCurrentScope(identifier.name) or self.ts.classExists(identifier.name): self.addError( - identifier, F"Duplicate declaration of identifier: {identifier.name}") + identifier, f"Duplicate declaration of identifier: {identifier.name}") if isinstance(d, ClassDef): className = d.name.name superclass = d.superclass.name @@ -148,7 +149,7 @@ def VarDef(self, node: VarDef): annotationType = self.visit(node.var) if not self.ts.canAssign(node.value.inferredType, annotationType): self.addError( - node, F"Expected {annotationType}, got {node.value.inferredType}") + node, f"Expected {annotationType}, got {node.value.inferredType}") return annotationType def ClassDef(self, node: ClassDef): @@ -202,7 +203,7 @@ def FuncDef(self, node: FuncDef): return if self.defInCurrentScope(funcName): self.addError(node.getIdentifier( - ), F"Duplicate declaration of identifier: {funcName}") + ), f"Duplicate declaration of identifier: {funcName}") return self.addType(funcName, funcType) else: # method decl @@ -210,14 +211,14 @@ def FuncDef(self, node: FuncDef): (not isinstance(funcType.parameters[0], ClassValueType)) or funcType.parameters[0].className != self.currentClass): self.addError( - node.getIdentifier(), F"Missing self param in method: {funcName}") + node.getIdentifier(), f"Missing self param in method: {funcName}") return for p in node.params: t = self.visit(p) pName = p.identifier.name if self.defInCurrentScope(pName) or self.ts.classExists(pName): self.addError( - p.identifier, F"Duplicate parameter name: {pName}") + p.identifier, f"Duplicate parameter name: {pName}") continue if t is not None: self.addType(pName, t) @@ -227,7 +228,7 @@ def FuncDef(self, node: FuncDef): name = identifier.name if self.defInCurrentScope(name) or self.ts.classExists(name): self.addError( - identifier, F"Duplicate declaration of identifier: {name}") + identifier, f"Duplicate declaration of identifier: {name}") continue if isinstance(d, FuncDef): self.funcParams(d) @@ -247,7 +248,7 @@ def FuncDef(self, node: FuncDef): hasReturn = True if (not hasReturn) and (not self.ts.canAssign(NoneType(), self.expReturnType)): self.addError(node.getIdentifier( - ), F"Expected return statement of type {self.expReturnType}") + ), f"Expected return statement of type {self.expReturnType}") self.expReturnType = None self.exitScope() return funcType @@ -263,7 +264,7 @@ def NonLocalDecl(self, node: NonLocalDecl): t = self.getNonLocalType(name) if t is None or not isinstance(t, ValueType): self.addError( - identifier, F"Unknown nonlocal variable: {name}") + identifier, f"Unknown nonlocal variable: {name}") return identifier.inferredType = t return t @@ -277,7 +278,7 @@ def GlobalDecl(self, node: GlobalDecl): t = self.getGlobal(name) if t is None or not isinstance(t, ValueType): self.addError( - identifier, F"Unknown global variable: {name}") + identifier, f"Unknown global variable: {name}") return identifier.inferredType = t return t @@ -294,11 +295,11 @@ def AssignStmt(self, node: AssignStmt): return if isinstance(t, Identifier) and not self.defInCurrentScope(t.name): self.addError( - t, F"Identifier not defined in current scope: {t.name}") + t, f"Identifier not defined in current scope: {t.name}") return - if not self.ts.canAssign(node.value.inferredType, t.inferredType): + if not self.ts.canAssign(node.value.inferredType, t.inferredValueType()): self.addError( - node, F"Expected {t.inferredType}, got {node.value.inferredType}") + node, f"Expected {t.inferredType}, got {node.value.inferredType}") return def IfStmt(self, node: IfStmt): @@ -306,7 +307,7 @@ def IfStmt(self, node: IfStmt): # if a branch is empty, isReturn=False if node.condition.inferredType != BoolType(): self.addError( - node.condition, F"Expected {BoolType()}, got {node.condition.inferredType}") + node.condition, f"Expected {BoolType()}, got {node.condition.inferredType}") return thenBody = False elseBody = False @@ -381,7 +382,7 @@ def BinaryExpr(self, node: BinaryExpr): def IndexExpr(self, node: IndexExpr): if node.index.inferredType != IntType(): self.addError( - node, F"Expected {IntType()} index, got {node.index.inferredType}") + node, f"Expected {IntType()} index, got {node.index.inferredType}") # indexing into a string returns a new string if node.list.inferredType == StrType(): node.inferredType = StrType() @@ -391,7 +392,7 @@ def IndexExpr(self, node: IndexExpr): node.inferredType = node.list.inferredType.elementType return node.inferredType else: - self.addError(node, F"Cannot index into {node.list.inferredType}") + self.addError(node, f"Cannot index into {node.list.inferredType}") node.inferredType = ObjectType() return ObjectType() @@ -402,13 +403,13 @@ def UnaryExpr(self, node: UnaryExpr): node.inferredType = IntType() return IntType() else: - self.addError(node, F"Expected int, got {operandType}") + self.addError(node, f"Expected int, got {operandType}") elif node.operator == "not": if operandType == BoolType(): node.inferredType = BoolType() return BoolType() else: - self.addError(node, F"Expected bool, got {operandType}") + self.addError(node, f"Expected bool, got {operandType}") else: node.inferredType = ObjectType() return ObjectType() @@ -420,30 +421,31 @@ def CallExpr(self, node: CallExpr): # constructor node.isConstructor = True t = self.ts.getMethod(fname, "__init__") + assert t is not None if len(t.parameters) != len(node.args) + 1: self.addError( - node, F"Expected {len(t.parameters) - 1} args, got {len(node.args)}") + node, f"Expected {len(t.parameters) - 1} args, got {len(node.args)}") else: for i in range(len(t.parameters) - 1): if not self.ts.canAssign(node.args[i].inferredType, t.parameters[i + 1]): self.addError( - node, F"Expected {t.parameters[i + 1]}, got {node.args[i].inferredType}") + node, f"Expected {t.parameters[i + 1]}, got {node.args[i].inferredType}") continue node.inferredType = ClassValueType(fname) else: t = self.getType(fname) if not isinstance(t, FuncType): - self.addError(node, F"Not a function: {fname}") + self.addError(node, f"Not a function: {fname}") node.inferredType = ObjectType() return ObjectType() if len(t.parameters) != len(node.args): self.addError( - node, F"Expected {len(t.parameters)} args, got {len(node.args)}") + node, f"Expected {len(t.parameters)} args, got {len(node.args)}") else: for i in range(len(t.parameters)): if not self.ts.canAssign(node.args[i].inferredType, t.parameters[i]): self.addError( - node, F"Expected {t.parameters[i]}, got {node.args[i].inferredType}") + node, f"Expected {t.parameters[i]}, got {node.args[i].inferredType}") continue node.inferredType = t.returnType node.function.inferredType = t @@ -454,21 +456,21 @@ def ForStmt(self, node: ForStmt): iterType = node.iterable.inferredType if not self.defInCurrentScope(node.identifier.name): self.addError( - node.identifier, F"Identifier not mutable in current scope: {node.identifier.name}") + node.identifier, f"Identifier not mutable in current scope: {node.identifier.name}") return if isinstance(iterType, ListValueType): - if not self.ts.canAssign(iterType.elementType, node.identifier.inferredType): + if not self.ts.canAssign(iterType.elementType, node.identifier.inferredValueType()): self.addError( - node.identifier, F"Expected {iterType.elementType}, got {node.identifier.inferredType}") + node.identifier, f"Expected {iterType.elementType}, got {node.identifier.inferredType}") return elif StrType() == iterType: - if not self.ts.canAssign(StrType(), node.identifier.inferredType): + if not self.ts.canAssign(StrType(), node.identifier.inferredValueType()): self.addError( - node.identifier, F"Expected {StrType()}, got {node.identifier.inferredType}") + node.identifier, f"Expected {StrType()}, got {node.identifier.inferredType}") return else: self.addError( - node.iterable, F"Expected iterable, got {node.iterable.inferredType}") + node.iterable, f"Expected iterable, got {node.iterable.inferredType}") return for s in node.body: if s.isReturn: @@ -478,16 +480,16 @@ def ListExpr(self, node: ListExpr): if len(node.elements) == 0: node.inferredType = EmptyType() else: - e_type = node.elements[0].inferredType + e_type = node.elements[0].inferredValueType() for e in node.elements: - e_type = self.ts.join(e_type, e.inferredType) + e_type = self.ts.join(e_type, e.inferredValueType()) node.inferredType = ListValueType(e_type) return node.inferredType def WhileStmt(self, node: WhileStmt): if node.condition.inferredType != BoolType(): self.addError( - node.condition, F"Expected {BoolType()}, got {node.condition.inferredType}") + node.condition, f"Expected {BoolType()}, got {node.condition.inferredType}") return for s in node.body: if s.isReturn: @@ -500,10 +502,10 @@ def ReturnStmt(self, node: ReturnStmt): elif node.value is None: if not self.ts.canAssign(NoneType(), self.expReturnType): self.addError( - node, F"Expected {self.expReturnType}, got {NoneType()}") + node, f"Expected {self.expReturnType}, got {NoneType()}") elif not self.ts.canAssign(node.value.inferredType, self.expReturnType): self.addError( - node, F"Expected {self.expReturnType}, got {node.value.inferredType}") + node, f"Expected {self.expReturnType}, got {node.value.inferredType}") node.expType = self.expReturnType return @@ -516,7 +518,7 @@ def Identifier(self, node: Identifier): if varType is not None and isinstance(varType, ValueType): node.inferredType = varType else: - self.addError(node, F"Unknown identifier: {node.name}") + self.addError(node, f"Unknown identifier: {node.name}") node.inferredType = ObjectType() return node.inferredType @@ -525,12 +527,12 @@ def MemberExpr(self, node: MemberExpr): BoolType(), StrType()} if node.object.inferredType in static_types or not isinstance(node.object.inferredType, ClassValueType): self.addError( - node, F"Expected object, got {node.object.inferredType}") + node, f"Expected object, got {node.object.inferredType}") else: class_name, member_name = node.object.inferredType.className, node.member.name if self.ts.getAttr(class_name, member_name) is None: self.addError( - node, F"Attribute {member_name} doesn't exist for class {class_name}") + node, f"Attribute {member_name} doesn't exist for class {class_name}") node.inferredType = ObjectType() return ObjectType() else: @@ -540,9 +542,9 @@ def MemberExpr(self, node: MemberExpr): def IfExpr(self, node: IfExpr): if node.condition.inferredType != BoolType(): self.addError( - F"Expected boolean, got {node.condition.inferredType}") + node, f"Expected boolean, got {node.condition.inferredType}") node.inferredType = self.ts.join( - node.thenExpr.inferredType, node.elseExpr.inferredType) + node.thenExpr.inferredValueType(), node.elseExpr.inferredValueType()) return node.inferredType def MethodCallExpr(self, node: MethodCallExpr): @@ -552,7 +554,7 @@ def MethodCallExpr(self, node: MethodCallExpr): BoolType(), StrType()} if method_member.object.inferredType in static_types or not isinstance(method_member.object.inferredType, ClassValueType): self.addError( - method_member, F"Expected object, got {method_member.object.inferredType}") + method_member, f"Expected object, got {method_member.object.inferredType}") node.inferredType = ObjectType() return node.inferredType else: @@ -560,18 +562,18 @@ def MethodCallExpr(self, node: MethodCallExpr): t = self.ts.getMethod(class_name, member_name) if t is None: self.addError( - node, F"Method {member_name} doesn't exist for class {class_name}") + node, f"Method {member_name} doesn't exist for class {class_name}") node.inferredType = ObjectType() return node.inferredType # self arguments if len(t.parameters) != len(node.args) + 1: self.addError( - node, F"Expected {len(t.parameters) - 1} args, got {len(node.args)}") + node, f"Expected {len(t.parameters) - 1} args, got {len(node.args)}") else: for i in range(len(t.parameters) - 1): if not self.ts.canAssign(node.args[i].inferredType, t.parameters[i + 1]): self.addError( - node, F"Expected {t.parameters[i + 1]}, got {node.args[i].inferredType}") + node, f"Expected {t.parameters[i + 1]}, got {node.args[i].inferredType}") continue node.method.inferredType = t node.inferredType = t.returnType @@ -607,7 +609,7 @@ def ListType(self, node: ListType): def ClassType(self, node: ClassType): if node.className not in {"", ""} and not self.ts.classExists(node.className): - self.addError(node, F"Unknown class: {node.className}") + self.addError(node, f"Unknown class: {node.className}") return ObjectType() else: return ClassValueType(node.className) diff --git a/compiler/types/functype.py b/compiler/types/functype.py index 0746709..acce9ad 100644 --- a/compiler/types/functype.py +++ b/compiler/types/functype.py @@ -23,10 +23,13 @@ def __eq__(self, other): def dropFirstParam(self): f = FuncType(self.parameters[1:], self.returnType) - f.refParams = [i - 1 for i in self.refParams] + f.refParams = {i - 1: v for i, v in self.refParams.items()} f.freevars = self.freevars return f + def getCILName(self) -> str: + raise Exception("unsupported") + def getCILSignature(self, name: str) -> str: params = [] for i in range(len(self.parameters)): @@ -44,14 +47,14 @@ def getJavaSignature(self) -> str: if self.returnType.isNone(): r = "V" else: - r = self.returnType.getJavaSignature() + r = self.returnType.getJavaSignature(False) params = [] for i in range(len(self.parameters)): p = self.parameters[i] if i in self.refParams and isinstance(p, ClassValueType): sig = '[' + p.getJavaSignature(True) else: - sig = p.getJavaSignature() + sig = p.getJavaSignature(False) params.append(sig) return "({}){}".format("".join(params), r) @@ -75,7 +78,7 @@ def methodEquals(self, other) -> bool: return self.parameters[1:] == other.parameters[1:] and self.returnType == other.returnType return False - def isFuncType() -> bool: + def isFuncType(self) -> bool: return True def __str__(self): diff --git a/compiler/types/listvaluetype.py b/compiler/types/listvaluetype.py index 0a12af8..eb3e4ec 100644 --- a/compiler/types/listvaluetype.py +++ b/compiler/types/listvaluetype.py @@ -3,6 +3,7 @@ class ListValueType(ValueType): + elementType: ValueType def __init__(self, elementType: ValueType): self.elementType = elementType @@ -12,16 +13,16 @@ def __eq__(self, other): return self.elementType == other.elementType return False - def getJavaSignature(self, _=False) -> str: + def getJavaSignature(self, isList=False) -> str: return "[" + self.elementType.getJavaSignature(True) - def getJavaName(self, _=False) -> str: + def getJavaName(self, isList=False) -> str: return "[" + self.elementType.getJavaSignature(True) - def getCILName(self, _=False) -> str: + def getCILName(self) -> str: return self.elementType.getCILName() + "[]" - def getCILSignature(self, _=False) -> str: + def getCILSignature(self) -> str: return self.getCILName() def isListType(self) -> bool: diff --git a/compiler/types/symboltype.py b/compiler/types/symboltype.py index faf77f4..f3a7fcd 100644 --- a/compiler/types/symboltype.py +++ b/compiler/types/symboltype.py @@ -1,23 +1,20 @@ -from typing import Optional +from typing import Optional, Self class SymbolType: # base class for types - def isValueType() -> bool: + def isValueType(self) -> bool: return False - def isListType() -> bool: + def isListType(self) -> bool: return False - def isFuncType() -> bool: + def isFuncType(self) -> bool: return False - def elementType(): - return None - - def isSpecialType() -> bool: + def isSpecialType(self) -> bool: return False - def toJSON(self, dump_location=True): + def toJSON(self, dump_location=True) -> dict: raise Exception("unsupported") diff --git a/compiler/types/valuetype.py b/compiler/types/valuetype.py index b52ac08..c62208b 100644 --- a/compiler/types/valuetype.py +++ b/compiler/types/valuetype.py @@ -3,16 +3,25 @@ class ValueType(SymbolType): - def isValueType() -> bool: + def isValueType(self) -> bool: return True def isNone(self) -> bool: return False - def toJSON(self, dump_location=True): + def toJSON(self, dump_location=True) -> dict: raise Exception("unsupported") - def getJavaSignature(self) -> str: + def getCILName(self) -> str: + raise Exception("unsupported") + + def getCILSignature(self) -> str: + raise Exception("unsupported") + + def getJavaName(self, isList: bool = False) -> str: + raise Exception("unsupported") + + def getJavaSignature(self, isList: bool = False) -> str: raise Exception("unsupported") def isJavaRef(self) -> bool: diff --git a/compiler/typesystem.py b/compiler/typesystem.py index db31ba4..647f79e 100644 --- a/compiler/typesystem.py +++ b/compiler/typesystem.py @@ -1,6 +1,5 @@ from .types import * -from collections import defaultdict -from typing import List, Dict, Tuple, Any +from typing import List, Dict, Tuple, Any, Optional, cast, Union class ClassInfo: @@ -8,11 +7,11 @@ class ClassInfo: attrs: Dict[str, Tuple[ValueType, Any]] methods: Dict[str, FuncType] - def __init__(self, name: str, superclass: str = None): + def __init__(self, name: str, superclass: Optional[str] = None): self.name = name self.superclass = superclass - self.attrs = defaultdict(lambda: None) # (attr type, init value) - self.methods = defaultdict(lambda: None) # type of method + self.attrs = {} # (attr type, init value) + self.methods = {} # type of method self.orderedAttrs = [] def __str__(self): @@ -24,7 +23,7 @@ class TypeSystem: def __init__(self): # information for each class - self.classes = defaultdict(lambda: None) + self.classes = {} objectInfo = ClassInfo("object") objectInfo.methods["__init__"] = FuncType([ObjectType()], NoneType()) @@ -45,15 +44,16 @@ def __init__(self): self.classes[""] = ClassInfo("", "object") self.classes[""] = ClassInfo("", "object") - def getMethodHelper(self, className: str, methodName: str) -> Tuple[FuncType, str]: + def getMethodHelper(self, className: str, methodName: str) -> Tuple[Optional[FuncType], str]: # requires className to be the name of a valid class - if methodName not in self.classes[className].methods: - if self.classes[className].superclass is None: - return (None, None) - return self.getMethodHelper(self.classes[className].superclass, methodName) - return (self.classes[className].methods[methodName], className) - - def getMethod(self, className: str, methodName: str) -> FuncType: + classInfo = self.classes[className] + if methodName not in classInfo.methods: + if classInfo.superclass is None: + return (None, "") + return self.getMethodHelper(classInfo.superclass, methodName) + return (classInfo.methods[methodName], className) + + def getMethod(self, className: str, methodName: str) -> Optional[FuncType]: # requires className to be the name of a valid class return self.getMethodHelper(className, methodName)[0] @@ -62,15 +62,16 @@ def getMethodDefClass(self, className: str, methodName: str) -> str: # requires className to be the name of a valid class return self.getMethodHelper(className, methodName)[1] - def getAttrHelper(self, className: str, attrName: str) -> Tuple[ValueType, Any]: + def getAttrHelper(self, className: str, attrName: str) -> Tuple[Optional[ValueType], Any]: # requires className to be the name of a valid class - if attrName not in self.classes[className].attrs: - if self.classes[className].superclass is None: + classInfo = self.classes[className] + if attrName not in classInfo.attrs: + if classInfo.superclass is None: return (None, None) - return self.getAttrHelper(self.classes[className].superclass, attrName) - return self.classes[className].attrs[attrName] + return self.getAttrHelper(classInfo.superclass, attrName) + return classInfo.attrs[attrName] - def getAttr(self, className: str, attrName: str) -> ValueType: + def getAttr(self, className: str, attrName: str) -> Optional[ValueType]: # returns type of attribute # requires className to be the name of a valid class return self.getAttrHelper(className, attrName)[0] @@ -80,17 +81,18 @@ def getAttrInit(self, className: str, attrName: str) -> Any: # requires className to be the name of a valid class return self.getAttrHelper(className, attrName)[1] - def getAttrOrMethod(self, className: str, name: str) -> SymbolType: + def getAttrOrMethod(self, className: str, name: str) -> Optional[SymbolType]: # returns type of attribute or method # requires className to be the name of a valid class - if name in self.classes[className].methods: - return self.classes[className].methods[name] - elif name in self.classes[className].attrs: - return self.classes[className].attrs[name][0] + classInfo = self.classes[className] + if name in classInfo.methods: + return classInfo.methods[name] + elif name in classInfo.attrs: + return classInfo.attrs[name][0] else: - if self.classes[className].superclass is None: + if classInfo.superclass is None: return None - return self.getAttrOrMethod(self.classes[className].superclass, name) + return self.getAttrOrMethod(classInfo.superclass, name) def classExists(self, className: str) -> bool: # we cannot check for None because it is a defaultdict @@ -107,7 +109,7 @@ def isSubClass(self, a: str, b: str) -> bool: curr = self.classes[curr].superclass return False - def isSubtype(self, a: ValueType, b: ValueType) -> bool: + def isSubtype(self, a: Optional[Union[ValueType, FuncType]], b: ValueType) -> bool: # return if a is a subtype of b if b == ObjectType(): return True @@ -115,7 +117,7 @@ def isSubtype(self, a: ValueType, b: ValueType) -> bool: return self.isSubClass(a.className, b.className) return a == b - def canAssign(self, a: ValueType, b: ValueType) -> bool: + def canAssign(self, a: Optional[Union[ValueType, FuncType]], b: ValueType) -> bool: # return if value of type a can be assigned/passed to type b (ex: b = a) if self.isSubtype(a, b): return True @@ -140,41 +142,46 @@ def join(self, a: ValueType, b: ValueType) -> ValueType: return ObjectType() # for 2 classes that aren't related by subtyping # find paths from A & B to root of typing tree - a, b = a.className, b.className + aCls, bCls = cast(ClassValueType, a).className, cast( + ClassValueType, b).className aAncestors = [] bAncestors = [] - while self.classes[a].superclass is not None: - aAncestors.append(self.classes[a].superclass) - a = self.classes[a].superclass - while self.classes[b].superclass is not None: - aAncestors.append(self.classes[b].superclass) - b = self.classes[b].superclass + curr = aCls + while curr is not None and self.classes[curr].superclass is not None: + if curr != aCls: + aAncestors.append(self.classes[curr].superclass) + curr = self.classes[curr].superclass + curr = bCls + while curr is not None and self.classes[curr].superclass is not None: + if curr != bCls: + bAncestors.append(self.classes[curr].superclass) + curr = self.classes[curr].superclass # reverse lists to find lowest common ancestor aAncestors = aAncestors[::-1] bAncestors = bAncestors[::-1] for i in range(min(len(aAncestors), len(bAncestors))): if aAncestors[i] != bAncestors[i]: - return self.classes[aAncestors[i - 1]] + return ClassValueType(self.classes[aAncestors[i - 1]].name) # this really shouldn't be returned return ObjectType() def getOrderedMethods(self, className: str) -> List[Tuple[str, FuncType, str]]: # (name, signature, defined in class) methods = [] - if self.classes[className].superclass is not None: - methods = self.getOrderedMethods( - self.classes[className].superclass) - for name in self.classes[className].methods: + classInfo = self.classes[className] + if classInfo.superclass is not None: + methods = self.getOrderedMethods(classInfo.superclass) + for name in classInfo.methods: hasExisting = False for i in range(len(methods)): if methods[i][0] == name: methods[i] = ( - name, self.classes[className].methods[name], className) + name, classInfo.methods[name], className) hasExisting = True break if not hasExisting: methods.append( - (name, self.classes[className].methods[name], className)) + (name, classInfo.methods[name], className)) return methods def getMappedMethods(self, className: str) -> Dict[str, Tuple[FuncType, str]]: @@ -185,10 +192,11 @@ def getMappedMethods(self, className: str) -> Dict[str, Tuple[FuncType, str]]: def getOrderedAttrs(self, className: str) -> List[Tuple[str, ValueType, Any]]: # return list of (name, type, init value) triples attrs = [] - if self.classes[className].superclass is not None: - attrs = self.getOrderedAttrs(self.classes[className].superclass) - for attr in self.classes[className].orderedAttrs: - attrType, attrInit = self.classes[className].attrs[attr] + classInfo = self.classes[className] + if classInfo.superclass is not None: + attrs = self.getOrderedAttrs(classInfo.superclass) + for attr in classInfo.orderedAttrs: + attrType, attrInit = classInfo.attrs[attr] attrs.append((attr, attrType, attrInit)) return attrs diff --git a/compiler/varcollector.py b/compiler/varcollector.py index 5c2d21a..1913a47 100644 --- a/compiler/varcollector.py +++ b/compiler/varcollector.py @@ -1,4 +1,4 @@ -from typing import List +from typing import Sequence, List from .astnodes import * from .types import * from .visitor import Visitor @@ -15,7 +15,7 @@ def getVars(self, node: Node): self.visit(node) return self.vars - def getVarsFromList(self, nodes: List[Node]): + def getVarsFromList(self, nodes: Sequence[Node]): for n in nodes: self.visit(n) return self.vars diff --git a/compiler/visitor.py b/compiler/visitor.py index a72ad62..ee530dc 100644 --- a/compiler/visitor.py +++ b/compiler/visitor.py @@ -1,104 +1,104 @@ from .astnodes import * from collections import defaultdict from .builder import Builder -from typing import List +from typing import List, Any class Visitor: - def visit(self, node: Node): + def visit(self, node: Node) -> Any: return node.visit(self) # TOP LEVEL & DECLARATIONS - def Program(self, node: Program): + def Program(self, node: Program) -> Any: pass - def VarDef(self, node: VarDef): + def VarDef(self, node: VarDef) -> Any: pass - def ClassDef(self, node: ClassDef): + def ClassDef(self, node: ClassDef) -> Any: pass - def FuncDef(self, node: FuncDef): + def FuncDef(self, node: FuncDef) -> Any: pass # STATEMENTS - def NonLocalDecl(self, node: NonLocalDecl): + def NonLocalDecl(self, node: NonLocalDecl) -> Any: pass - def GlobalDecl(self, node: GlobalDecl): + def GlobalDecl(self, node: GlobalDecl) -> Any: pass - def AssignStmt(self, node: AssignStmt): + def AssignStmt(self, node: AssignStmt) -> Any: pass - def IfStmt(self, node: IfStmt): + def IfStmt(self, node: IfStmt) -> Any: pass - def ExprStmt(self, node: ExprStmt): + def ExprStmt(self, node: ExprStmt) -> Any: pass - def BinaryExpr(self, node: BinaryExpr): + def BinaryExpr(self, node: BinaryExpr) -> Any: pass - def IndexExpr(self, node: IndexExpr): + def IndexExpr(self, node: IndexExpr) -> Any: pass - def UnaryExpr(self, node: UnaryExpr): + def UnaryExpr(self, node: UnaryExpr) -> Any: pass - def CallExpr(self, node: CallExpr): + def CallExpr(self, node: CallExpr) -> Any: pass - def ForStmt(self, node: ForStmt): + def ForStmt(self, node: ForStmt) -> Any: pass - def ListExpr(self, node: ListExpr): + def ListExpr(self, node: ListExpr) -> Any: pass - def WhileStmt(self, node: WhileStmt): + def WhileStmt(self, node: WhileStmt) -> Any: pass - def ReturnStmt(self, node: ReturnStmt): + def ReturnStmt(self, node: ReturnStmt) -> Any: pass - def Identifier(self, node: Identifier): + def Identifier(self, node: Identifier) -> Any: pass - def MemberExpr(self, node: MemberExpr): + def MemberExpr(self, node: MemberExpr) -> Any: pass - def IfExpr(self, node: IfExpr): + def IfExpr(self, node: IfExpr) -> Any: pass - def MethodCallExpr(self, node: MethodCallExpr): + def MethodCallExpr(self, node: MethodCallExpr) -> Any: pass # LITERALS - def BooleanLiteral(self, node: BooleanLiteral): + def BooleanLiteral(self, node: BooleanLiteral) -> Any: pass - def IntegerLiteral(self, node: IntegerLiteral): + def IntegerLiteral(self, node: IntegerLiteral) -> Any: pass - def NoneLiteral(self, node: NoneLiteral): + def NoneLiteral(self, node: NoneLiteral) -> Any: pass - def StringLiteral(self, node: StringLiteral): + def StringLiteral(self, node: StringLiteral) -> Any: pass # TYPES - def TypedVar(self, node: TypedVar): + def TypedVar(self, node: TypedVar) -> Any: pass - def ListType(self, node: ListType): + def ListType(self, node: ListType) -> Any: pass - def ClassType(self, node: ClassType): + def ClassType(self, node: ClassType) -> Any: pass @@ -107,7 +107,7 @@ class CommonVisitor(Visitor): counter = 0 # for labels # helpers for handling locals - locals: List[defaultdict] = None + locals: List[defaultdict] = [] def enterScope(self): self.locals.append(defaultdict(lambda: None)) diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index f4ac308..6d3c86f 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -3,7 +3,7 @@ from .builder import Builder from .typesystem import TypeSystem from .visitor import CommonVisitor -from typing import List, Dict, Tuple, Set +from typing import List, Dict, Tuple, Set, Optional, cast, Callable class WasmBuilder(Builder): @@ -34,7 +34,7 @@ def _else(self): self.newLine("(else") self.indent() - def func(self, name: str, sig: str = "") -> Builder: + def func(self, name: str, sig: str = "") -> "WasmBuilder": # return new block for declaring extra locals self.newLine(f"(func ${name} {sig}") self.indent() @@ -45,19 +45,18 @@ def end(self): self.newLine(")") def emit(self) -> str: - lines = [] + lines: List[str] = [] for l in self.lines: if isinstance(l, str): - if " drop" in l and " i32.const 0" in lines[-1]: - lines[-1] = None + if " drop" in l and lines[-1] and " i32.const 0" in lines[-1]: + lines.pop() continue lines.append(l) else: lines.append(l.emit()) - lines = [l for l in lines if l is not None] return "\n".join(lines) - def newBlock(self) -> Builder: + def newBlock(self) -> "WasmBuilder": child = WasmBuilder(self.name) child.indentation = self.indentation self.lines.append(child) @@ -70,9 +69,9 @@ class WasmBackend(CommonVisitor): # (class name, method name) -> (class offset, table offset, inherited) methodOffsets: Dict[Tuple[str, str], Tuple[int, int, bool]] # class -> offset of start of vtable - vtables: Dict[str, int] + vtables: Dict[str, List[Tuple[int, int]]] undeclaredFuncs: Set[str] - locals: WasmBuilder = None + localsBuilder: Optional[WasmBuilder] = None def __init__(self, main: str, ts: TypeSystem): self.builder = WasmBuilder(main) @@ -112,9 +111,6 @@ def initializeOffsets(self): self.vtables[cls].append((memOffset + idx * 4, t)) memOffset += (len(methods) * 4) - def currentBuilder(self): - return self.classes[self.currentClass] - def newLabelName(self) -> str: self.counter += 1 return "label_" + str(self.counter) @@ -133,16 +129,17 @@ def teeLocal(self, name: str): def getLocal(self, name: str): self.instr(f"local.get ${name}") - def genLocalName(self, suffix=None) -> str: + def genLocalName(self, suffix: Optional[str] = None) -> str: self.localCounter += 1 suffix = "" if suffix is None else ("_" + suffix) return f"local{suffix}{self.localCounter}" - def newLocal(self, name: str = None, t: str = "i32") -> str: + def newLocal(self, name: Optional[str] = None, t: str = "i32") -> str: # add a new local decl, does not store anything if name is None: name = self.genLocalName() - self.locals.newLine(f"(local ${name} {t})") + assert self.localsBuilder is not None + self.localsBuilder.newLine(f"(local ${name} {t})") return name def visitStmtList(self, stmts: List[Stmt]): @@ -152,7 +149,7 @@ def visitStmtList(self, stmts: List[Stmt]): for s in stmts: self.visit(s) - def alloc(self, local=None): + def alloc(self, local: Optional[str] = None): # consume i32 from top of stack, allocate that many bytes self.instr("call $alloc") if local is not None: @@ -194,7 +191,7 @@ def Program(self, node: Program): # initialize all globals to 0 for now, since we don't statically allocate strings or arrays for v in var_decls: self.instr( - f"(global ${v.var.identifier.name} (mut {v.var.t.getWasmName()}) ({v.var.t.getWasmName()}.const 0))") + f"(global ${v.var.identifier.name} (mut {v.var.getTypeX().getWasmName()}) ({v.var.getTypeX().getWasmName()}.const 0))") for d in func_decls: self.visit(d) for c in cls_decls: @@ -207,7 +204,7 @@ def Program(self, node: Program): module_builder = self.builder self.builder = module_builder.newBlock() - self.locals = self.builder.func("main") + self.localsBuilder = self.builder.func("main") self.defaultToGlobals = True self.initializeVtables() # initialize globals @@ -230,23 +227,12 @@ def initializeVtables(self): self.instr(f"i32.const {funcOffset}") self.instr("i32.store") - def ClassDef(self, node: ClassDef): - self.currentClass = node.name.name - func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] - for func in func_decls: - name = f"{node.name.name}${func.name.name}" - self.undeclaredFuncs.remove("$" + name) - self.funcDefHelper(func, name) - - def FuncDef(self, node: FuncDef): - self.funcDefHelper(node, node.name.name) - def funcDefHelper(self, node: FuncDef, name: str): - self.returnType = node.type.returnType + self.returnType = node.getTypeX().returnType ret = None if self.returnType.isNone() else self.returnType.getWasmName() paramNames = [x.identifier.name for x in node.params] - self.locals = self.builder.func( - name, node.type.getWasmSignature(paramNames)) + self.localsBuilder = self.builder.func( + name, node.getTypeX().getWasmSignature(paramNames)) for d in node.declarations: self.visit(d) self.visitStmtList(node.statements) @@ -258,33 +244,44 @@ def funcDefHelper(self, node: FuncDef, name: str): self.instr("unreachable") self.builder.end() + def ClassDef(self, node: ClassDef): + self.currentClass = node.name.name + func_decls = [d for d in node.declarations if isinstance(d, FuncDef)] + for func in func_decls: + name = f"{node.name.name}${func.name.name}" + self.undeclaredFuncs.remove("$" + name) + self.funcDefHelper(func, name) + + def FuncDef(self, node: FuncDef): + self.funcDefHelper(node, node.name.name) + def VarDef(self, node: VarDef): varName = node.var.identifier.name if node.isAttr: raise Exception("this should be handled elsewhere") - elif node.var.varInstance.isNonlocal: + elif node.var.varInstanceX().isNonlocal: self.instr("i32.const 8") self.instr("call $alloc") addr = self.newLocal(varName) self.teeLocal(addr) self.visit(node.value) - self.instr(f"{node.value.inferredType.getWasmName()}.store") + self.instr(f"{node.value.inferredValueType().getWasmName()}.store") else: self.visit(node.value) - n = self.newLocal(varName, node.value.inferredType.getWasmName()) + n = self.newLocal(varName, node.value.inferredValueType().getWasmName()) self.setLocal(n) # # STATEMENTS def setIdentifier(self, target: Identifier, val: str): # val is the name of the local that the value is stored in - if self.defaultToGlobals or target.varInstance.isGlobal: + if self.defaultToGlobals or target.varInstanceX().isGlobal: self.getLocal(val) self.instr(f"global.set ${target.name}") - elif target.varInstance.isNonlocal: + elif target.varInstanceX().isNonlocal: self.getLocal(target.name) self.getLocal(val) - self.instr(f"{target.inferredType.getWasmName()}.store") + self.instr(f"{target.inferredValueType().getWasmName()}.store") else: self.getLocal(val) self.setLocal(target.name) @@ -307,33 +304,33 @@ def processAssignmentTarget(self, target: Expr, val: str): self.getLocal(iterable) self.instr("i32.add") self.getLocal(val) - self.instr(f"{target.inferredType.getWasmName()}.store") + self.instr(f"{target.inferredValueType().getWasmName()}.store") elif isinstance(target, MemberExpr): - cls = target.object.inferredType.className + cls = cast(ClassValueType, target.object.inferredValueType()).className attr = target.member.name offset = self.attrOffsets[(cls, attr)] self.visit(target.object) self.instr(f"i32.const {offset * 8 + 4}") self.instr("i32.add") self.getLocal(val) - self.instr(f"{target.inferredType.getWasmName()}.store") + self.instr(f"{target.inferredValueType().getWasmName()}.store") else: raise Exception( "Internal compiler error: unsupported assignment target") def MemberExpr(self, node: MemberExpr): - cls = node.object.inferredType.className + cls = cast(ClassValueType, node.object.inferredValueType()).className attr = node.member.name offset = self.attrOffsets[(cls, attr)] self.visit(node.object) self.instr(f"i32.const {offset * 8 + 4}") self.instr("i32.add") - self.instr(f"{node.inferredType.getWasmName()}.load") + self.instr(f"{node.inferredValueType().getWasmName()}.load") def AssignStmt(self, node: AssignStmt): self.visit(node.value) val = self.newLocal(self.genLocalName( - "val"), node.value.inferredType.getWasmName()) + "val"), node.value.inferredValueType().getWasmName()) self.setLocal(val) targets = node.targets[::-1] for t in targets: @@ -373,8 +370,8 @@ def listConcat(self): def BinaryExpr(self, node: BinaryExpr): operator = node.operator - leftType = node.left.inferredType - rightType = node.right.inferredType + leftType = node.left.inferredValueType() + rightType = node.right.inferredValueType() shortCircuitOperators = {"and", "or"} if operator not in shortCircuitOperators: self.visit(node.left) @@ -511,14 +508,14 @@ def CallExpr(self, node: CallExpr): self.emit_assert(node.args[0], node.location[0]) else: for i in range(len(node.args)): - self.visitArg(node.function.inferredType, i, node.args[i]) + self.visitArg(cast(FuncType, node.function.inferredType), i, node.args[i]) self.instr(f"call ${name}") - if node.function.inferredType.returnType.isNone(): + if cast(FuncType, node.function.inferredType).returnType.isNone(): self.NoneLiteral(None) # push null for void return def MethodCallExpr(self, node: MethodCallExpr): - funcType = node.method.inferredType - className = node.method.object.inferredType.className + funcType = cast(FuncType, node.method.inferredType) + className = cast(ClassValueType, node.method.object.inferredType).className methodName = node.method.member.name if methodName == "__init__" and className in {"int", "bool"}: return @@ -579,7 +576,7 @@ def ForStmt(self, node: ForStmt): length = self.newLocal(self.genLocalName("length")) # this temporarily stores the current value temp = self.newLocal(self.genLocalName( - "temp"), node.identifier.inferredType.getWasmName()) + "temp"), node.identifier.inferredValueType().getWasmName()) self.visit(node.iterable) self.teeLocal(iterable) @@ -603,8 +600,8 @@ def ForStmt(self, node: ForStmt): self.instr(f"br_if ${block}") - isList = node.iterable.inferredType.isListType() - contentsType = node.identifier.inferredType.getWasmName() + isList = node.iterable.inferredValueType().isListType() + contentsType = node.identifier.inferredValueType().getWasmName() self.idxHelper(iterable, idx, isList, contentsType) self.setLocal(temp) self.setIdentifier(node.identifier, temp) @@ -622,7 +619,7 @@ def ForStmt(self, node: ForStmt): self.builder.end() self.builder.end() - def buildReturn(self, value: Expr): + def buildReturn(self, value: Optional[Expr]): if self.returnType.isNone(): self.instr("return") else: @@ -636,11 +633,11 @@ def ReturnStmt(self, node: ReturnStmt): self.buildReturn(node.value) def Identifier(self, node: Identifier): - if self.defaultToGlobals or node.varInstance.isGlobal: + if self.defaultToGlobals or node.varInstanceX().isGlobal: self.instr(f"global.get ${node.name}") - elif node.varInstance.isNonlocal: + elif node.varInstanceX().isNonlocal: self.instr(f"local.get ${node.name}") - self.instr(f"{node.inferredType.getWasmName()}.load") + self.instr(f"{node.inferredValueType().getWasmName()}.load") else: self.instr(f"local.get ${node.name}") @@ -648,10 +645,10 @@ def IfExpr(self, node: IfExpr): c = lambda: self.visit(node.condition) t = lambda: self.visit(node.thenExpr) e = lambda: self.visit(node.elseExpr) - resultType = node.inferredType.getWasmName() + resultType = node.inferredValueType().getWasmName() self.ternary(c, t, e, resultType) - def ternary(self, condFn, thenFn, elseFn, resultType): + def ternary(self, condFn: Callable, thenFn: Callable, elseFn: Callable, resultType: str): n = self.newLocal(self.genLocalName("ifexpr_result"), resultType) condFn() self.builder._if() @@ -676,7 +673,7 @@ def ListExpr(self, node: ListExpr): else: elementType = ClassValueType("object") else: - elementType = t.elementType + elementType = cast(ListValueType, t).elementType # 8 bytes per element + 4 for the length, rounded up to nearest 8 increase = (length + 1) * 8 @@ -735,8 +732,8 @@ def IndexExpr(self, node: IndexExpr): iterable = self.newLocal(self.genLocalName("iterable")) self.setLocal(iterable) idx = self.validateIdx(iterable, node) - self.idxHelper(iterable, idx, node.list.inferredType.isListType(), - node.inferredType.getWasmName()) + self.idxHelper(iterable, idx, node.list.inferredValueType().isListType(), + node.inferredValueType().getWasmName()) # # LITERALS @@ -749,7 +746,7 @@ def BooleanLiteral(self, node: BooleanLiteral): def IntegerLiteral(self, node: IntegerLiteral): self.instr(f"i64.const {node.value}") - def NoneLiteral(self, node: NoneLiteral): + def NoneLiteral(self, node: Optional[NoneLiteral]): self.instr("i32.const 0") def StringLiteral(self, node: StringLiteral): @@ -779,12 +776,12 @@ def StringLiteral(self, node: StringLiteral): # load the address the string was stored at to the stack self.getLocal(addr) - def visitArg(self, funcType, paramIdx: int, arg: Expr): - argIsRef = isinstance(arg, Identifier) and arg.varInstance.isNonlocal + def visitArg(self, funcType: FuncType, paramIdx: int, arg: Expr): + argIsRef = isinstance(arg, Identifier) and arg.varInstanceX().isNonlocal paramIsRef = paramIdx in funcType.refParams - if argIsRef and paramIsRef and arg.varInstance == funcType.refParams[paramIdx]: + if argIsRef and paramIsRef and cast(Identifier, arg).varInstanceX() == funcType.refParams[paramIdx]: # ref arg and ref param, pass ref arg - self.getLocal(arg.name) + self.getLocal(cast(Identifier, arg).name) elif paramIsRef: # non-ref arg and ref param, or do not pass ref arg # unwrap if necessary, re-wrap @@ -793,7 +790,7 @@ def visitArg(self, funcType, paramIdx: int, arg: Expr): addr = self.newLocal(self.genLocalName("arg_" + str(paramIdx))) self.teeLocal(addr) self.visit(arg) - self.instr(f"{arg.inferredType.getWasmName()}.store") + self.instr(f"{arg.inferredValueType().getWasmName()}.store") self.getLocal(addr) else: # non-ref param, maybe unwrap @@ -808,11 +805,11 @@ def emit_assert(self, arg: Expr, line: int): self.NoneLiteral(None) def emit_print(self, arg: Expr): - if isinstance(arg.inferredType, ListValueType) or arg.inferredType.className not in {"bool", "int", "str"}: + if isinstance(arg.inferredType, ListValueType) or cast(ClassValueType, arg.inferredType).className not in {"bool", "int", "str"}: raise Exception( - f"Built-in function print is unsupported for values of type {arg.inferredType.classname}") + f"Built-in function print is unsupported for values of type {cast(ClassValueType, arg.inferredType).className}") self.visit(arg) - self.instr(f"call $log_{arg.inferredType.className}") + self.instr(f"call $log_{cast(ClassValueType, arg.inferredType).className}") self.NoneLiteral(None) def emit_len(self, arg: Expr): diff --git a/main.py b/main.py index 8f46ec5..b03cb51 100644 --- a/main.py +++ b/main.py @@ -26,7 +26,8 @@ def main(): parser = argparse.ArgumentParser(description='Chocopy frontend') parser.add_argument('--mode', dest='mode', - choices=["parse", "tc", "python", "jvm", "hoist", "cil", "wasm", "llvm"], + choices=["parse", "tc", "python", "jvm", + "hoist", "cil", "wasm", "llvm"], default="python", help=mode_help) parser.add_argument('--print', dest='should_print', action="iframe.php?url=https%3A%2F%2Fgithub.com%2Fstore_true", diff --git a/pyrightconfig.json b/pyrightconfig.json new file mode 100644 index 0000000..72f63a3 --- /dev/null +++ b/pyrightconfig.json @@ -0,0 +1,8 @@ +{ + "include": [ + "compiler" + ], + "reportMissingImports": true, + "reportMissingTypeStubs": false, + "pythonVersion": "3.11" +} \ No newline at end of file From e18259cbe5fa49e4d537261f550be782654537d0 Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 4 Nov 2025 10:48:50 -0500 Subject: [PATCH 78/79] use pyrefly for type checking --- .github/workflows/pyrefly.yml | 36 +++++++++++++++++++++++++++++ compiler/astnodes/booleanliteral.py | 1 + compiler/astnodes/integerliteral.py | 1 + compiler/astnodes/stringliteral.py | 1 + compiler/cil_backend.py | 2 ++ compiler/jvm_backend.py | 4 ++++ compiler/llvm_backend.py | 20 ++++++++++++++++ compiler/parser.py | 4 ++++ compiler/typesystem.py | 5 ++++ compiler/wasm_backend.py | 6 +++++ pyrefly.toml | 6 +++++ pyrightconfig.json | 8 ------- requirements-dev.txt | 2 ++ requirements.txt | 1 + 14 files changed, 89 insertions(+), 8 deletions(-) create mode 100644 .github/workflows/pyrefly.yml create mode 100644 pyrefly.toml delete mode 100644 pyrightconfig.json create mode 100644 requirements-dev.txt create mode 100644 requirements.txt diff --git a/.github/workflows/pyrefly.yml b/.github/workflows/pyrefly.yml new file mode 100644 index 0000000..b4ee706 --- /dev/null +++ b/.github/workflows/pyrefly.yml @@ -0,0 +1,36 @@ +name: Pyrefly Check + +on: + push: + branches: [ main ] + pull_request: + +jobs: + pyrefly: + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Cache pip + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }} + restore-keys: | + ${{ runner.os }}-pip- + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements-dev.txt + + - name: Run Pyrefly check + run: | + pyrefly check diff --git a/compiler/astnodes/booleanliteral.py b/compiler/astnodes/booleanliteral.py index a704bfb..42413d2 100644 --- a/compiler/astnodes/booleanliteral.py +++ b/compiler/astnodes/booleanliteral.py @@ -6,6 +6,7 @@ class BooleanLiteral(Literal): def __init__(self, location: List[int], value: bool): super().__init__(location, "BooleanLiteral") + # pyrefly: ignore [bad-assignment] self.value = value def visit(self, visitor): diff --git a/compiler/astnodes/integerliteral.py b/compiler/astnodes/integerliteral.py index c7a6427..362ff9d 100644 --- a/compiler/astnodes/integerliteral.py +++ b/compiler/astnodes/integerliteral.py @@ -6,6 +6,7 @@ class IntegerLiteral(Literal): def __init__(self, location: List[int], value: int): super().__init__(location, "IntegerLiteral") + # pyrefly: ignore [bad-assignment] self.value = value def visit(self, visitor): diff --git a/compiler/astnodes/stringliteral.py b/compiler/astnodes/stringliteral.py index 2084813..3358be4 100644 --- a/compiler/astnodes/stringliteral.py +++ b/compiler/astnodes/stringliteral.py @@ -6,6 +6,7 @@ class StringLiteral(Literal): def __init__(self, location: List[int], value: str): super().__init__(location, "StringLiteral") + # pyrefly: ignore [bad-assignment] self.value = value def visit(self, visitor): diff --git a/compiler/cil_backend.py b/compiler/cil_backend.py index 68760e2..c382881 100644 --- a/compiler/cil_backend.py +++ b/compiler/cil_backend.py @@ -246,6 +246,7 @@ def FuncDef(self, node: FuncDef, funcType: str = "static", superConstructor: Opt self.newLocalEntry(param.identifier.name, param.getTypeX(), True) for d in node.declarations: self.visit(d) + # pyrefly: ignore [bad-assignment] self.returnType = node.getTypeX().returnType # handle last return @@ -571,6 +572,7 @@ def WhileStmt(self, node: WhileStmt): self.label(endLabel) def buildReturn(self, value: Optional[Expr]): + # pyrefly: ignore [missing-attribute] if not self.returnType.isNone(): if value is None: self.NoneLiteral(None) diff --git a/compiler/jvm_backend.py b/compiler/jvm_backend.py index f3d4de1..c8acb81 100644 --- a/compiler/jvm_backend.py +++ b/compiler/jvm_backend.py @@ -198,6 +198,7 @@ def funcDefHelper(self, node: FuncDef): self.newLocalEntry(node.params[i].identifier.name) for d in node.declarations: self.visit(d) + # pyrefly: ignore [bad-assignment] self.returnType = node.getTypeX().returnType # handle last return self.visitStmtList(node.statements) @@ -580,6 +581,7 @@ def WhileStmt(self, node: WhileStmt): self.instr("nop") def buildReturn(self, value: Optional[Expr]): + # pyrefly: ignore [missing-attribute] if self.returnType.isNone(): self.instr("return") else: @@ -587,6 +589,7 @@ def buildReturn(self, value: Optional[Expr]): self.NoneLiteral(None) else: self.visit(value) + # pyrefly: ignore [bad-argument-type] self.returnInstr(self.returnType) def ReturnStmt(self, node: ReturnStmt): @@ -658,6 +661,7 @@ def loadInt(self, value: int): self.instr(f"ldc {value}") def IntegerLiteral(self, node: IntegerLiteral): + # pyrefly: ignore [bad-argument-type] self.loadInt(node.value) def NoneLiteral(self, node: Optional[NoneLiteral]): diff --git a/compiler/llvm_backend.py b/compiler/llvm_backend.py index 3dc2d5d..0ba6d05 100644 --- a/compiler/llvm_backend.py +++ b/compiler/llvm_backend.py @@ -428,6 +428,7 @@ def BinaryExpr(self, node: BinaryExpr): elemType = cast( ListValueType, node.inferredType).elementType.getLLVMType() assert elemType is not None + # pyrefly: ignore [missing-argument] size = self.getBuilder().add(int32_t(4), self.getBuilder().mul( total_len, self.sizeof(elemType)), 'bytes') new_arr = self.getBuilder().call( @@ -438,12 +439,14 @@ def BinaryExpr(self, node: BinaryExpr): data_lhs_start = self.getListDataPtr(new_arr, elemType) lhs_data = self.getListDataPtr(lhs, elemType) rhs_data = self.getListDataPtr(rhs, elemType) + # pyrefly: ignore [missing-argument] lhs_bytes = self.getBuilder().mul(llen, self.sizeof(elemType)) self.getBuilder().call(self.externs['memcpy'], [ self.toVoidPtr(data_lhs_start), self.toVoidPtr(lhs_data), lhs_bytes]) data_rhs_start = self.getBuilder().gep(data_lhs_start, [llen]) + # pyrefly: ignore [missing-argument] rhs_bytes = self.getBuilder().mul(rlen, self.sizeof(elemType)) self.getBuilder().call(self.externs['memcpy'], [ @@ -454,6 +457,7 @@ def BinaryExpr(self, node: BinaryExpr): rhs = self.toVoidPtr(rhs) llen = self.getBuilder().call(self.externs['strlen'], [lhs]) rlen = self.getBuilder().call(self.externs['strlen'], [rhs]) + # pyrefly: ignore [missing-argument, missing-argument] total_len = self.getBuilder().add(self.getBuilder().add( llen, rlen), int32_t(1)) new_str = self.getBuilder().call( @@ -464,21 +468,28 @@ def BinaryExpr(self, node: BinaryExpr): new_str, fmt, lhs, rhs]) return new_str elif leftType == IntType(): + # pyrefly: ignore [missing-argument] return self.getBuilder().add(lhs, rhs) else: raise Exception( "Internal compiler error: unexpected operand types for +") # other arithmetic operators elif operator == "-": + # pyrefly: ignore [missing-argument] return self.getBuilder().sub(lhs, rhs) elif operator == "*": + # pyrefly: ignore [missing-argument] return self.getBuilder().mul(lhs, rhs) elif operator == "//": + # pyrefly: ignore [missing-argument] return self.getBuilder().sdiv(lhs, rhs) elif operator == "%": # emulate Python modulo with ((a % b) + b) % b) + # pyrefly: ignore [missing-argument] val = self.getBuilder().srem(lhs, rhs) + # pyrefly: ignore [missing-argument] val = self.getBuilder().add(val, rhs) + # pyrefly: ignore [missing-argument] return self.getBuilder().srem(val, rhs) # relational operators elif operator in {"<", "<=", ">", ">="}: @@ -505,13 +516,17 @@ def BinaryExpr(self, node: BinaryExpr): return self.getBuilder().icmp_signed(operator, lhs, rhs) elif operator == "is": # pointer comparisons + # pyrefly: ignore [missing-argument] lhs_ptr = self.getBuilder().ptrtoint(lhs, int32_t) + # pyrefly: ignore [missing-argument] rhs_ptr = self.getBuilder().ptrtoint(rhs, int32_t) return self.getBuilder().icmp_unsigned("==", lhs_ptr, rhs_ptr) # logical operators elif operator == "and": + # pyrefly: ignore [missing-argument] return self.getBuilder().and_(lhs, rhs) elif operator == "or": + # pyrefly: ignore [missing-argument] return self.getBuilder().or_(lhs, rhs) else: raise Exception( @@ -643,6 +658,7 @@ def forBody(self, node: ForStmt, var: ir.Value, idxFn: Callable[[ir.Value], ir.V currIdx = self.getBuilder().load(idx_var) self.getBuilder().store(idxFn(currIdx), var) self.visitStmtList(node.body) + # pyrefly: ignore [missing-argument] self.getBuilder().store(self.getBuilder().add(currIdx, int32_t(1)), idx_var) def ForStmt(self, node: ForStmt): @@ -686,6 +702,7 @@ def ListExpr(self, node: ListExpr): elemType = cast( ListValueType, node.inferredType).elementType.getLLVMType() assert elemType is not None + # pyrefly: ignore [missing-argument, missing-argument] size = self.getBuilder().add(int32_t(4), self.getBuilder().mul( int32_t(n), self.sizeof(elemType))) addr = self.getBuilder().call(self.externs['malloc'], [ @@ -813,7 +830,9 @@ def NoneLiteral(self, node: Optional[NoneLiteral]): return voidptr_t(None) def StringLiteral(self, node: StringLiteral): + # pyrefly: ignore [unsupported-operation] bytes = bytearray((node.value + '\00').encode('ascii')) + # pyrefly: ignore [bad-argument-type] size = int32_t(1 + len(node.value)) addr = self.getBuilder().call( self.externs['malloc'], [size], 'str_literal') @@ -893,6 +912,7 @@ def emit_input(self) -> ir.Value: # copy contents into new string so that input buffer can be reused len = self.getBuilder().call(self.externs['strlen'], [input_buf]) new_str = self.getBuilder().call( + # pyrefly: ignore [missing-argument] self.externs['malloc'], [self.getBuilder().add(len, int32_t(1))], 'new_str') fmt = self.toVoidPtr(self.module.get_global('__fmt_str')) self.getBuilder().call(self.externs['sprintf'], [ diff --git a/compiler/parser.py b/compiler/parser.py index 3bad1e7..a403ccb 100644 --- a/compiler/parser.py +++ b/compiler/parser.py @@ -30,6 +30,7 @@ def getLocation(self, node: ast.AST) -> typing.List[int]: # input is Python AST node # get 2 item list corresponding to AST node starting location # make columns 1-indexed + # pyrefly: ignore [missing-attribute, missing-attribute] return [node.lineno, node.col_offset + 1] def visit(self, node: ast.AST) -> typing.Any: @@ -49,6 +50,7 @@ def getTypeAnnotation(self, node: ast.expr) -> TypeAnnotation: elif isinstance(node, ast.Name): return ClassType(location, node.id) elif isinstance(node, ast.Str): + # pyrefly: ignore [bad-argument-type, deprecated] return ClassType(location, node.s) else: raise ParseError("Unsupported type annotation", node) @@ -320,12 +322,14 @@ def visit_Name(self, node: ast.Name) -> Identifier: def visit_Num(self, node: ast.Num) -> IntegerLiteral: location = self.getLocation(node) + # pyrefly: ignore [deprecated] if not isinstance(node.n, int): raise ParseError("Only integers are supported", node) return IntegerLiteral(location, node.n) def visit_Str(self, node: ast.Str) -> StringLiteral: location = self.getLocation(node) + # pyrefly: ignore [bad-argument-type, deprecated] return StringLiteral(location, node.s) def visit_List(self, node: ast.List) -> ListExpr: diff --git a/compiler/typesystem.py b/compiler/typesystem.py index 647f79e..566749f 100644 --- a/compiler/typesystem.py +++ b/compiler/typesystem.py @@ -102,6 +102,7 @@ def isSubClass(self, a: str, b: str) -> bool: # requires a and b to be the names of valid classes # return if a is the same class or subclass of b curr = a + # pyrefly: ignore [bad-assignment] while curr is not None: if curr == b: return True @@ -174,14 +175,18 @@ def getOrderedMethods(self, className: str) -> List[Tuple[str, FuncType, str]]: for name in classInfo.methods: hasExisting = False for i in range(len(methods)): + # pyrefly: ignore [bad-index] if methods[i][0] == name: + # pyrefly: ignore [unsupported-operation] methods[i] = ( name, classInfo.methods[name], className) hasExisting = True break if not hasExisting: + # pyrefly: ignore [missing-attribute] methods.append( (name, classInfo.methods[name], className)) + # pyrefly: ignore [bad-return] return methods def getMappedMethods(self, className: str) -> Dict[str, Tuple[FuncType, str]]: diff --git a/compiler/wasm_backend.py b/compiler/wasm_backend.py index 6d3c86f..d3ea00e 100644 --- a/compiler/wasm_backend.py +++ b/compiler/wasm_backend.py @@ -228,7 +228,9 @@ def initializeVtables(self): self.instr("i32.store") def funcDefHelper(self, node: FuncDef, name: str): + # pyrefly: ignore [bad-assignment] self.returnType = node.getTypeX().returnType + # pyrefly: ignore [missing-attribute, missing-attribute] ret = None if self.returnType.isNone() else self.returnType.getWasmName() paramNames = [x.identifier.name for x in node.params] self.localsBuilder = self.builder.func( @@ -238,6 +240,7 @@ def funcDefHelper(self, node: FuncDef, name: str): self.visitStmtList(node.statements) # implicitly return None if possible if ret is not None and not isinstance(node.statements[-1], ReturnStmt): + # pyrefly: ignore [missing-attribute, missing-attribute] if self.returnType.getWasmName() == "i32" and not self.returnType.isSpecialType(): self.instr("i32.const 0") else: @@ -620,6 +623,7 @@ def ForStmt(self, node: ForStmt): self.builder.end() def buildReturn(self, value: Optional[Expr]): + # pyrefly: ignore [missing-attribute] if self.returnType.isNone(): self.instr("return") else: @@ -750,6 +754,7 @@ def NoneLiteral(self, node: Optional[NoneLiteral]): self.instr("i32.const 0") def StringLiteral(self, node: StringLiteral): + # pyrefly: ignore [bad-argument-type] length = len(node.value) memory = length + 4 # 1 byte per char + 4 for length, rounded to nearest 8 @@ -765,6 +770,7 @@ def StringLiteral(self, node: StringLiteral): self.instr("i32.store") for i in range(length): offset = i + 4 + # pyrefly: ignore [unsupported-operation] val = ord(node.value[i]) # addr: mem + 4 + idx self.getLocal(addr) diff --git a/pyrefly.toml b/pyrefly.toml new file mode 100644 index 0000000..894f8c6 --- /dev/null +++ b/pyrefly.toml @@ -0,0 +1,6 @@ +project-includes = ["compiler"] +python-version = "3.11.0" +infer-with-first-use = false + +[errors] +missing-import = "error" diff --git a/pyrightconfig.json b/pyrightconfig.json deleted file mode 100644 index 72f63a3..0000000 --- a/pyrightconfig.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "include": [ - "compiler" - ], - "reportMissingImports": true, - "reportMissingTypeStubs": false, - "pythonVersion": "3.11" -} \ No newline at end of file diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..bbd7932 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,2 @@ +-r requirements.txt +pyrefly diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..454f54e --- /dev/null +++ b/requirements.txt @@ -0,0 +1 @@ +llvmlite From d5906a517cd6236488870fcced54b76cf4de791f Mon Sep 17 00:00:00 2001 From: Danny Yang Date: Tue, 4 Nov 2025 13:46:40 -0500 Subject: [PATCH 79/79] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 991256a..1261d89 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # chocopy-python-compiler +[![pyrefly](https://img.shields.io/endpoint?url=https://pyrefly.org/badge.json)](https://github.com/facebook/pyrefly) + Ahead-of-time compiler for [Chocopy](https://chocopy.org/), a subset of Python 3.6 with type annotations and static type checking. Chocopy is used in compiler courses at several universities. This project has no relation to those courses, and is purely for my own learning/practice/fun.