feat: implement "stateless" modules (#3663)

this commit implements support for "stateless" modules in vyper.

this is the first major step in implementing vyper's module system.
it redesigns the language's import system, allows calling internal
functions from imported modules, allows for limited use of types from
imported modules, and introduces support for `.vyi` interface files.

note that the following features are left for future, follow-up work:
- modules with variables (constants, immutables or storage variables)
- full support for imported events (in that they do not get exported
  in the ABI)
- a system for exporting imported functions in the external interface
  of a contract

this commit first and foremost changes how imports are handled in vyper.

previously, an imported file was assumed to be an interface file. some
very limited validation was performed in `InterfaceT.from_ast`, but not
fully typechecked, and no code was generated for it.

now, when a file is imported, it is checked whether it is
  1. a `.vy` file
  2. a `.vyi` file
  3. a `.json` file

the `.json` pathway remains more or less unchanged. the `.vyi` pathway
is new, but it is fairly straightforward and is basically a "simple"
path through the `.vy` pathway which piggy-backs off the `.vy` analysis
to produce an `InterfaceT` object.

the `.vy` pathway now does full typechecking and analysis of the
imported module. some changes were made to support this:
- a new ImportGraph data structure tracks the position in the import
  graph and detects (and bands) import cycles
- InputBundles now implement a `_normalize_path()` method.
  this method normalizes the path so that source IDs are stable no
  matter how a file is accessed in the filesystem (i.e., no matter
  what the search path was at the time `load_file()` was called).
- CompilerInput now has a distinction between `resolved_path` and `path`
  (the original path that was asked for). this allows us to maintain UX
  considerations (showing unresolved paths etc) while still having a
  1:1:1 correspondence between source id, filepath and filesystem.

these changes were needed in order to stabilize notions like "which file
are we looking at?" no matter the way the file was accessed or how it
was imported. this is important so that types imported transitively
can resolve as expected no matter how they are imported - for instance,
`x.SomeType` and `a.x.SomeType` resolving to the same type.

the other changes needed to support code generation and analysis for
imported functions were fairly simple, and mostly involved generalizing
the analysis/code generation to type-based dispatch instead of AST-based
dispatch.

other changes to the language and compiler API include:
- import restrictions are more relaxed - `import x` is allowed now
  (previously, `import x as x` was required)
- change function labels in IR
  function labels are changed to disambiguate functions of the same name
  (but whose parent module are different). this was done by computing a
  unique function_id for every function and using that function_id when
  constructing its IR identifier.
- add compile_from_file_input which accepts a FileInput instead of a
  string. this is now the new preferred entry point into the compiler.
  its usage simplifies several internal APIs which expect to have
  `source_id` and `path` in addition to the raw source code.
- change compile_code api from contract_name= to contract_path=

additional changes to internal APIs and passes include:
- remove `remove_unused_statements()`
  the "unused statements" are now important to keep around for imports!
  in general, it is planned to remove both the AST expansion and
  constant folding passes as copying around the AST results in both
  performance and correctness problems
- abstract out a common exception rewriting pattern.
  instead of raising `exception.with_annotation(node)` -- just catch-all
  in the parent implementation and then don't have to worry about it at
  the exception site.
- rename "type" metadata key on most top-level declarators to more
  specific names (e.g. "func_type", "getter_type", etc).
- remove dead package pytest-rerunfailures
  use of `--reruns` was removed in c913b2d
- refactor: move `parse_*` functions, remove vyper.ast.annotation
  move `parse_*` functions into new module vyper.ast.parse and merge in
  vyper.ast.annotation
- rename the old `GlobalContext` class to `ModuleT`
- refactor: move InterfaceT into `vyper/semantics/types/module.py`
  it makes more sense here since it is closely coupled with `ModuleT`.

setup.py

@@ -13,7 +13,6 @@
         "pytest-instafail>=0.4,<1.0",
         "pytest-xdist>=2.5,<3.0",
         "pytest-split>=0.7.0,<1.0",
-        "pytest-rerunfailures>=10.2,<11",
         "eth-tester[py-evm]>=0.9.0b1,<0.10",
         "py-evm>=0.7.0a1,<0.8",
         "web3==6.0.0",

tests/conftest.py

@@ -17,7 +17,7 @@
 from vyper import compiler
 from vyper.ast.grammar import parse_vyper_source
 from vyper.codegen.ir_node import IRnode
-from vyper.compiler.input_bundle import FilesystemInputBundle
+from vyper.compiler.input_bundle import FilesystemInputBundle, InputBundle
 from vyper.compiler.settings import OptimizationLevel, Settings, _set_debug_mode
 from vyper.ir import compile_ir, optimizer
 
@@ -103,6 +103,12 @@ def fn(sources_dict):
     return fn
 
 
+# for tests which just need an input bundle, doesn't matter what it is
+@pytest.fixture
+def dummy_input_bundle():
+    return InputBundle([])
+
+
 # TODO: remove me, this is just string.encode("utf-8").ljust()
 # only used in test_logging.py.
 @pytest.fixture
@@ -255,9 +261,11 @@ def ir_compiler(ir, *args, **kwargs):
         ir = IRnode.from_list(ir)
         if optimize != OptimizationLevel.NONE:
             ir = optimizer.optimize(ir)
+
         bytecode, _ = compile_ir.assembly_to_evm(
             compile_ir.compile_to_assembly(ir, optimize=optimize)
         )
+
         abi = kwargs.get("abi") or []
         c = w3.eth.contract(abi=abi, bytecode=bytecode)
         deploy_transaction = c.constructor()

tests/functional/codegen/test_call_graph_stability.py

@@ -55,7 +55,7 @@ def foo():
 
         # check the .called_functions data structure on foo() directly
         foo = t.vyper_module_folded.get_children(vy_ast.FunctionDef, filters={"name": "foo"})[0]
-        foo_t = foo._metadata["type"]
+        foo_t = foo._metadata["func_type"]
         assert [f.name for f in foo_t.called_functions] == func_names
 
         # now for sanity, ensure the order that the function definitions appear

tests/functional/builtins/codegen/test_interfaces.py → tests/functional/codegen/test_interfaces.py

@@ -6,9 +6,9 @@
 from vyper.compiler import compile_code
 from vyper.exceptions import (
     ArgumentException,
+    DuplicateImport,
     InterfaceViolation,
     NamespaceCollision,
-    StructureException,
 )
 
 
@@ -31,7 +31,7 @@ def allowance(_owner: address, _spender: address) -> (uint256, uint256):
 
     out = compile_code(code, output_formats=["interface"])
     out = out["interface"]
-    code_pass = "\n".join(code.split("\n")[:-2] + ["    pass"])  # replace with a pass statement.
+    code_pass = "\n".join(code.split("\n")[:-2] + ["    ..."])  # replace with a pass statement.
 
     assert code_pass.strip() == out.strip()
 
@@ -60,7 +60,7 @@ def allowance(_owner: address, _spender: address) -> (uint256, uint256): view
     def test(_owner: address): nonpayable
     """
 
-    out = compile_code(code, contract_name="One.vy", output_formats=["external_interface"])[
+    out = compile_code(code, contract_path="One.vy", output_formats=["external_interface"])[
         "external_interface"
     ]
 
@@ -85,14 +85,14 @@ def test_external_interface_parsing(make_input_bundle, assert_compile_failed):
     interface_code = """
 @external
 def foo() -> uint256:
-    pass
+    ...
 
 @external
 def bar() -> uint256:
-    pass
+    ...
     """
 
-    input_bundle = make_input_bundle({"a.vy": interface_code})
+    input_bundle = make_input_bundle({"a.vyi": interface_code})
 
     code = """
 import a as FooBarInterface
@@ -121,9 +121,8 @@ def foo() -> uint256:
 
     """
 
-    assert_compile_failed(
-        lambda: compile_code(not_implemented_code, input_bundle=input_bundle), InterfaceViolation
-    )
+    with pytest.raises(InterfaceViolation):
+        compile_code(not_implemented_code, input_bundle=input_bundle)
 
 
 def test_missing_event(make_input_bundle, assert_compile_failed):
@@ -132,7 +131,7 @@ def test_missing_event(make_input_bundle, assert_compile_failed):
     a: uint256
     """
 
-    input_bundle = make_input_bundle({"a.vy": interface_code})
+    input_bundle = make_input_bundle({"a.vyi": interface_code})
 
     not_implemented_code = """
 import a as FooBarInterface
@@ -156,7 +155,7 @@ def test_malformed_event(make_input_bundle, assert_compile_failed):
     a: uint256
     """
 
-    input_bundle = make_input_bundle({"a.vy": interface_code})
+    input_bundle = make_input_bundle({"a.vyi": interface_code})
 
     not_implemented_code = """
 import a as FooBarInterface
@@ -183,7 +182,7 @@ def test_malformed_events_indexed(make_input_bundle, assert_compile_failed):
     a: uint256
     """
 
-    input_bundle = make_input_bundle({"a.vy": interface_code})
+    input_bundle = make_input_bundle({"a.vyi": interface_code})
 
     not_implemented_code = """
 import a as FooBarInterface
@@ -211,7 +210,7 @@ def test_malformed_events_indexed2(make_input_bundle, assert_compile_failed):
     a: indexed(uint256)
     """
 
-    input_bundle = make_input_bundle({"a.vy": interface_code})
+    input_bundle = make_input_bundle({"a.vyi": interface_code})
 
     not_implemented_code = """
 import a as FooBarInterface
@@ -234,13 +233,13 @@ def bar() -> uint256:
 
 VALID_IMPORT_CODE = [
     # import statement, import path without suffix
-    ("import a as Foo", "a.vy"),
-    ("import b.a as Foo", "b/a.vy"),
-    ("import Foo as Foo", "Foo.vy"),
-    ("from a import Foo", "a/Foo.vy"),
-    ("from b.a import Foo", "b/a/Foo.vy"),
-    ("from .a import Foo", "./a/Foo.vy"),
-    ("from ..a import Foo", "../a/Foo.vy"),
+    ("import a as Foo", "a.vyi"),
+    ("import b.a as Foo", "b/a.vyi"),
+    ("import Foo as Foo", "Foo.vyi"),
+    ("from a import Foo", "a/Foo.vyi"),
+    ("from b.a import Foo", "b/a/Foo.vyi"),
+    ("from .a import Foo", "./a/Foo.vyi"),
+    ("from ..a import Foo", "../a/Foo.vyi"),
 ]
 
 
@@ -252,61 +251,79 @@ def test_extract_file_interface_imports(code, filename, make_input_bundle):
 
 
 BAD_IMPORT_CODE = [
-    ("import a", StructureException),  # must alias absolute imports
-    ("import a as A\nimport a as A", NamespaceCollision),
+    ("import a as A\nimport a as A", DuplicateImport),
+    ("import a as A\nimport a as a", DuplicateImport),
+    ("from . import a\nimport a as a", DuplicateImport),
+    ("import a as a\nfrom . import a", DuplicateImport),
     ("from b import a\nfrom . import a", NamespaceCollision),
-    ("from . import a\nimport a as a", NamespaceCollision),
-    ("import a as a\nfrom . import a", NamespaceCollision),
+    ("import a\nimport c as a", NamespaceCollision),
 ]
 
 
 @pytest.mark.parametrize("code,exception_type", BAD_IMPORT_CODE)
 def test_extract_file_interface_imports_raises(
     code, exception_type, assert_compile_failed, make_input_bundle
 ):
-    input_bundle = make_input_bundle({"a.vy": "", "b/a.vy": ""})  # dummy
-    assert_compile_failed(lambda: compile_code(code, input_bundle=input_bundle), exception_type)
+    input_bundle = make_input_bundle({"a.vyi": "", "b/a.vyi": "", "c.vyi": ""})
+    with pytest.raises(exception_type):
+        compile_code(code, input_bundle=input_bundle)
 
 
 def test_external_call_to_interface(w3, get_contract, make_input_bundle):
+    token_interface = """
+@view
+@external
+def balanceOf(addr: address) -> uint256:
+    ...
+
+@external
+def transfer(to: address, amount: uint256):
+    ...
+    """
+
     token_code = """
+import itoken as IToken
+
+implements: IToken
+
 balanceOf: public(HashMap[address, uint256])
 
 @external
-def transfer(to: address, _value: uint256):
-    self.balanceOf[to] += _value
+def transfer(to: address, amount: uint256):
+    self.balanceOf[to] += amount
     """
 
-    input_bundle = make_input_bundle({"one.vy": token_code})
+    input_bundle = make_input_bundle({"token.vy": token_code, "itoken.vyi": token_interface})
 
     code = """
-import one as TokenCode
+import itoken as IToken
 
 interface EPI:
     def test() -> uint256: view
 
 
-token_address: TokenCode
+token_address: IToken
 
 
 @external
 def __init__(_token_address: address):
-    self.token_address = TokenCode(_token_address)
+    self.token_address = IToken(_token_address)
 
 
 @external
 def test():
     self.token_address.transfer(msg.sender, 1000)
     """
 
-    erc20 = get_contract(token_code)
-    test_c = get_contract(code, *[erc20.address], input_bundle=input_bundle)
+    token = get_contract(token_code, input_bundle=input_bundle)
+
+    test_c = get_contract(code, *[token.address], input_bundle=input_bundle)
 
     sender = w3.eth.accounts[0]
-    assert erc20.balanceOf(sender) == 0
+    assert token.balanceOf(sender) == 0
 
     test_c.test(transact={})
-    assert erc20.balanceOf(sender) == 1000
+    assert token.balanceOf(sender) == 1000
 
 
 @pytest.mark.parametrize(
@@ -320,26 +337,36 @@ def test():
     ],
 )
 def test_external_call_to_interface_kwarg(get_contract, kwarg, typ, expected, make_input_bundle):
-    code_a = f"""
+    interface_code = f"""
+@external
+@view
+def foo(_max: {typ} = {kwarg}) -> {typ}:
+    ...
+    """
+    code1 = f"""
+import one as IContract
+
+implements: IContract
+
 @external
 @view
 def foo(_max: {typ} = {kwarg}) -> {typ}:
     return _max
     """
 
-    input_bundle = make_input_bundle({"one.vy": code_a})
+    input_bundle = make_input_bundle({"one.vyi": interface_code})
 
-    code_b = f"""
-import one as ContractA
+    code2 = f"""
+import one as IContract
 
 @external
 @view
 def bar(a_address: address) -> {typ}:
-    return ContractA(a_address).foo()
+    return IContract(a_address).foo()
     """
 
-    contract_a = get_contract(code_a)
-    contract_b = get_contract(code_b, *[contract_a.address], input_bundle=input_bundle)
+    contract_a = get_contract(code1, input_bundle=input_bundle)
+    contract_b = get_contract(code2, *[contract_a.address], input_bundle=input_bundle)
 
     assert contract_b.bar(contract_a.address) == expected
 
@@ -349,8 +376,8 @@ def test_external_call_to_builtin_interface(w3, get_contract):
 balanceOf: public(HashMap[address, uint256])
 
 @external
-def transfer(to: address, _value: uint256) -> bool:
-    self.balanceOf[to] += _value
+def transfer(to: address, amount: uint256) -> bool:
+    self.balanceOf[to] += amount
     return True
     """
 
@@ -510,14 +537,14 @@ def returns_Bytes3() -> Bytes[3]:
     """
 
     should_not_compile = """
-import BadJSONInterface as BadJSONInterface
+import BadJSONInterface
 @external
 def foo(x: BadJSONInterface) -> Bytes[2]:
     return slice(x.returns_Bytes3(), 0, 2)
     """
 
     code = """
-import BadJSONInterface as BadJSONInterface
+import BadJSONInterface
 
 foo: BadJSONInterface
 
@@ -578,10 +605,10 @@ def balanceOf(owner: address) -> uint256:
 @external
 @view
 def balanceOf(owner: address) -> uint256:
-    pass
+    ...
     """
 
-    input_bundle = make_input_bundle({"balanceof.vy": interface_code})
+    input_bundle = make_input_bundle({"balanceof.vyi": interface_code})
 
     c = get_contract(code, input_bundle=input_bundle)
 
@@ -592,7 +619,7 @@ def test_simple_implements(make_input_bundle):
     interface_code = """
 @external
 def foo() -> uint256:
-    pass
+    ...
     """
 
     code = """
@@ -605,7 +632,7 @@ def foo() -> uint256:
     return 1
     """
 
-    input_bundle = make_input_bundle({"a.vy": interface_code})
+    input_bundle = make_input_bundle({"a.vyi": interface_code})
 
     assert compile_code(code, input_bundle=input_bundle) is not None
 

tests/functional/codegen/test_selector_table_stability.py

@@ -14,7 +14,7 @@ def test_dense_jumptable_stability():
 
     # test that the selector table data is stable across different runs
     # (tox should provide different PYTHONHASHSEEDs).
-    expected_asm = """{ DATA _sym_BUCKET_HEADERS b'\\x0bB' _sym_bucket_0 b'\\n' b'+\\x8d' _sym_bucket_1 b'\\x0c' b'\\x00\\x85' _sym_bucket_2 b'\\x08' } { DATA _sym_bucket_1 b'\\xd8\\xee\\xa1\\xe8' _sym_external_foo6___3639517672 b'\\x05' b'\\xd2\\x9e\\xe0\\xf9' _sym_external_foo0___3533627641 b'\\x05' b'\\x05\\xf1\\xe0_' _sym_external_foo2___99737695 b'\\x05' b'\\x91\\t\\xb4{' _sym_external_foo23___2433332347 b'\\x05' b'np3\\x7f' _sym_external_foo11___1852846975 b'\\x05' b'&\\xf5\\x96\\xf9' _sym_external_foo13___653629177 b'\\x05' b'\\x04ga\\xeb' _sym_external_foo14___73884139 b'\\x05' b'\\x89\\x06\\xad\\xc6' _sym_external_foo17___2298916294 b'\\x05' b'\\xe4%\\xac\\xd1' _sym_external_foo4___3827674321 b'\\x05' b'yj\\x01\\xac' _sym_external_foo7___2036990380 b'\\x05' b'\\xf1\\xe6K\\xe5' _sym_external_foo29___4058401765 b'\\x05' b'\\xd2\\x89X\\xb8' _sym_external_foo3___3532216504 b'\\x05' } { DATA _sym_bucket_2 b'\\x06p\\xffj' _sym_external_foo25___108068714 b'\\x05' b'\\x964\\x99I' _sym_external_foo24___2520029513 b'\\x05' b's\\x81\\xe7\\xc1' _sym_external_foo10___1937893313 b'\\x05' b'\\x85\\xad\\xc11' _sym_external_foo28___2242756913 b'\\x05' b'\\xfa"\\xb1\\xed' _sym_external_foo5___4196577773 b'\\x05' b'A\\xe7[\\x05' _sym_external_foo22___1105681157 b'\\x05' b'\\xd3\\x89U\\xe8' _sym_external_foo1___3548993000 b'\\x05' b'hL\\xf8\\xf3' _sym_external_foo20___1749874931 b'\\x05' } { DATA _sym_bucket_0 b'\\xee\\xd9\\x1d\\xe3' _sym_external_foo9___4007206371 b'\\x05' b'a\\xbc\\x1ch' _sym_external_foo16___1639717992 b'\\x05' b'\\xd3*\\xa7\\x0c' _sym_external_foo21___3542787852 b'\\x05' b'\\x18iG\\xd9' _sym_external_foo19___409552857 b'\\x05' b'\\n\\xf1\\xf9\\x7f' _sym_external_foo18___183630207 b'\\x05' b')\\xda\\xd7`' _sym_external_foo27___702207840 b'\\x05' b'2\\xf6\\xaa\\xda' _sym_external_foo12___855026394 b'\\x05' b'\\xbe\\xb5\\x05\\xf5' _sym_external_foo15___3199534581 b'\\x05' b'\\xfc\\xa7_\\xe6' _sym_external_foo8___4238827494 b'\\x05' b'\\x1b\\x12C8' _sym_external_foo26___454181688 b'\\x05' } }"""  # noqa: E501
+    expected_asm = """{ DATA _sym_BUCKET_HEADERS b\'\\x0bB\' _sym_bucket_0 b\'\\n\' b\'+\\x8d\' _sym_bucket_1 b\'\\x0c\' b\'\\x00\\x85\' _sym_bucket_2 b\'\\x08\' } { DATA _sym_bucket_1 b\'\\xd8\\xee\\xa1\\xe8\' _sym_external 6 foo6()3639517672 b\'\\x05\' b\'\\xd2\\x9e\\xe0\\xf9\' _sym_external 0 foo0()3533627641 b\'\\x05\' b\'\\x05\\xf1\\xe0_\' _sym_external 2 foo2()99737695 b\'\\x05\' b\'\\x91\\t\\xb4{\' _sym_external 23 foo23()2433332347 b\'\\x05\' b\'np3\\x7f\' _sym_external 11 foo11()1852846975 b\'\\x05\' b\'&\\xf5\\x96\\xf9\' _sym_external 13 foo13()653629177 b\'\\x05\' b\'\\x04ga\\xeb\' _sym_external 14 foo14()73884139 b\'\\x05\' b\'\\x89\\x06\\xad\\xc6\' _sym_external 17 foo17()2298916294 b\'\\x05\' b\'\\xe4%\\xac\\xd1\' _sym_external 4 foo4()3827674321 b\'\\x05\' b\'yj\\x01\\xac\' _sym_external 7 foo7()2036990380 b\'\\x05\' b\'\\xf1\\xe6K\\xe5\' _sym_external 29 foo29()4058401765 b\'\\x05\' b\'\\xd2\\x89X\\xb8\' _sym_external 3 foo3()3532216504 b\'\\x05\' } { DATA _sym_bucket_2 b\'\\x06p\\xffj\' _sym_external 25 foo25()108068714 b\'\\x05\' b\'\\x964\\x99I\' _sym_external 24 foo24()2520029513 b\'\\x05\' b\'s\\x81\\xe7\\xc1\' _sym_external 10 foo10()1937893313 b\'\\x05\' b\'\\x85\\xad\\xc11\' _sym_external 28 foo28()2242756913 b\'\\x05\' b\'\\xfa"\\xb1\\xed\' _sym_external 5 foo5()4196577773 b\'\\x05\' b\'A\\xe7[\\x05\' _sym_external 22 foo22()1105681157 b\'\\x05\' b\'\\xd3\\x89U\\xe8\' _sym_external 1 foo1()3548993000 b\'\\x05\' b\'hL\\xf8\\xf3\' _sym_external 20 foo20()1749874931 b\'\\x05\' } { DATA _sym_bucket_0 b\'\\xee\\xd9\\x1d\\xe3\' _sym_external 9 foo9()4007206371 b\'\\x05\' b\'a\\xbc\\x1ch\' _sym_external 16 foo16()1639717992 b\'\\x05\' b\'\\xd3*\\xa7\\x0c\' _sym_external 21 foo21()3542787852 b\'\\x05\' b\'\\x18iG\\xd9\' _sym_external 19 foo19()409552857 b\'\\x05\' b\'\\n\\xf1\\xf9\\x7f\' _sym_external 18 foo18()183630207 b\'\\x05\' b\')\\xda\\xd7`\' _sym_external 27 foo27()702207840 b\'\\x05\' b\'2\\xf6\\xaa\\xda\' _sym_external 12 foo12()855026394 b\'\\x05\' b\'\\xbe\\xb5\\x05\\xf5\' _sym_external 15 foo15()3199534581 b\'\\x05\' b\'\\xfc\\xa7_\\xe6\' _sym_external 8 foo8()4238827494 b\'\\x05\' b\'\\x1b\\x12C8\' _sym_external 26 foo26()454181688 b\'\\x05\' } }"""  # noqa: E501
     assert expected_asm in output["asm"]