🪲 Allow comments to be indented #3287 (#5685)

Fixes #3287
The PR addresses multiple small issues related to comments and trailing spaces. Automated tests are added for all scenarios.

**How to test**
1. Indented comments and comments at the end of code blocks should work in levels 9-11 without errors:
```
repeat 3 times
    # comment
    print ‘3’ # another comment
```
2. The `print` command should be able to print a # sign in levels 4-18:
```
print 'comments start with the # sign'
```
3. Equality checks should preserve the original whitespacing of its arguments in levels 8-11. The following program should print the value of `i`.
```
i = this   is
print '[' i ']'
if i = ‘this   is’
  print i
```

grammars/level12-Additions.lark

@@ -5,7 +5,7 @@ assign_list: var (_IS| _EQUALS) (text_in_quotes|NUMBER) (_COMMA (text_in_quotes|
 ?atom: NUMBER | _MINUS NUMBER | var_access | text_in_quotes //unsupported numbers are gone, we can now allow floats with NUMBER
 ?print_expression: var_access_print | error_unsupported_number | NUMBER
 
-equality_check: (list_access | var_access | text_in_quotes | NUMBER) (_IS| _EQUALS) (list_access | var_access | text_in_quotes | NUMBER)
+equality_check: (list_access | var_access | text_in_quotes | NUMBER) (_IS| _EQUALS) (list_access | var_access | text_in_quotes | NUMBER) _SPACE?
 
 in_list_check: (list_access | var_access | text_in_quotes | NUMBER) _IN var_access
 sleep: _SLEEP (NUMBER | list_access | var_access | expression)?

grammars/level14-Additions.lark

@@ -2,12 +2,12 @@
 
 condition:+= equality_check_dequals | smaller | bigger | not_equal | smaller_equal | bigger_equal
 
-equality_check: comparison_arg (_IS | _EQUALS) comparison_arg
-equality_check_dequals: comparison_arg  _DOUBLE_EQUALS  comparison_arg
-smaller: comparison_arg  _SMALLER  comparison_arg
-bigger: comparison_arg  _LARGER  comparison_arg
-smaller_equal: comparison_arg  _SMALLER_EQUALS  comparison_arg
-bigger_equal: comparison_arg  _LARGER_EQUALS  comparison_arg
-not_equal: comparison_arg  _NOT_EQUALS  comparison_arg
+equality_check: comparison_arg (_IS | _EQUALS) comparison_arg _SPACE?
+equality_check_dequals: comparison_arg _DOUBLE_EQUALS comparison_arg _SPACE?
+smaller: comparison_arg _SMALLER comparison_arg _SPACE?
+bigger: comparison_arg _LARGER comparison_arg _SPACE?
+smaller_equal: comparison_arg _SMALLER_EQUALS comparison_arg _SPACE?
+bigger_equal: comparison_arg _LARGER_EQUALS comparison_arg _SPACE?
+not_equal: comparison_arg _NOT_EQUALS comparison_arg _SPACE?
 
 ?comparison_arg: call | var_access | list_access | text_in_quotes | NUMBER

grammars/level16-Additions.lark

@@ -10,7 +10,7 @@ error_assign_list_missing_brackets: var  (_IS | _EQUALS) (_LEFT_SQUARE_BRACKET)?
 
 error_list_access_at<old_rule_to_error list_access>
 
-_print_ask_argument: (_SPACE | quoted_text | list_access | error_list_access_at | expression | print_expression)*
+_print_ask_argument.1: (_SPACE | quoted_text | list_access | error_list_access_at | expression | print_expression)*
 assign: var (_IS| _EQUALS) (list_access | error_list_access_at | boolean | atom | expression)
 error_print_no_quotes: _PRINT (textwithoutspaces | list_access | error_list_access_at | var_access) (_SPACE (textwithoutspaces | list_access | var_access))*  -> error_print_nq
 sleep: _SLEEP (INT | list_access | error_list_access_at | var_access | expression)?

grammars/level4-Additions.lark

@@ -1,7 +1,7 @@
 // redefining it entirely since it has many order-depending rules (e.g ask_no_quotes should be after ask and before assign)
 command: clear | error_non_decimal | print | ask | play | turtle | assign_list | add | remove | error_add_missing_to | error_remove_missing_from | sleep | error_list_access | error_ask_no_quotes| assign | error_invalid_space | error_print_no_quotes | error_print_one_quote_only | error_text_no_print |error_invalid | empty_line
 
-_print_ask_argument: (_SPACE | list_access | error_list_access | quoted_text | var_access_print)*
+_print_ask_argument.1: (_SPACE | list_access | error_list_access | quoted_text | var_access_print)*
 
 error_print_no_quotes: _PRINT text -> error_print_nq
 error_ask_no_quotes: var _IS _ASK text -> error_print_nq

grammars/level5-Additions.lark

@@ -18,7 +18,7 @@ assign_button: NAME _IS _BUTTON
 assign_list: var _IS textwithspaces (_COMMA textwithspaces)+
 assign: var _IS (list_access | textwithspaces)
 
-error_print_no_quotes: _PRINT (textwithoutspaces | list_access | var_access) (_SPACE (textwithoutspaces | list_access | var_access))*  -> error_print_nq
+error_print_no_quotes: _PRINT (textwithoutspaces | list_access | var_access) (_SPACE (textwithoutspaces | list_access | var_access))* -> error_print_nq
 
 // new commands for level 5
 if_pressed_else: _IF (LETTER_OR_NUMERAL | var) _IS _PRESSED _EOL* _if_less_command (_SPACE+ _EOL* | _SPACE* _EOL+) _ELSE (_SPACE+ _EOL* | _SPACE* _EOL+) _if_less_command

grammars/level6-Additions.lark

@@ -1,4 +1,4 @@
-_print_ask_argument: (_SPACE | quoted_text | list_access | expression | print_expression)*
+_print_ask_argument.1: (_SPACE | quoted_text | list_access | expression | print_expression)*
 ?print_expression: var_access_print | error_unsupported_number | INT
 
 // redefining it entirely since it has many order-depending rules

grammars/level8-Additions.lark

@@ -5,7 +5,7 @@ error_repeat_dep_8<old_rule_to_error repeat>
 
 // FH jan 2022 (#1817) we can allow unquoted rhs's again cause there now is a line break after
 // not sure of we want it, but we could
-equality_check: (textwithoutspaces | NUMBER) (_IS | _EQUALS) (textwithoutspaces | NUMBER) (_SPACE | textwithoutspaces | NUMBER)*
+equality_check: (textwithoutspaces | NUMBER) (_IS | _EQUALS) (textwithoutspaces | NUMBER | textwithspaces)
 
 // from level 8 on if_pressed is implemented slightly differently
 if_pressed: _IF (LETTER_OR_NUMERAL | NAME) _IS _PRESSED _EOL (_SPACE command) (_EOL _SPACE command)* _EOL? _END_BLOCK

grammars/terminals.lark

@@ -1,5 +1,8 @@
-// A comment has a negative lookahead to ensure it cannot parse _END_BLOCKS
-COMMENT: /#(?!ENDBLOCK)[^\n]*/
+// If the comment takes up the whole line, it should also encompass its leading newline. In this way the whole
+// comment line can be skipped for the commands containing it. Comments should escape ENDBLOCKS not only in the start
+// but at any point, because if the body of a command (e.g. `if`) ends up with a comment, the ENDBLOCK would appear
+// right after it. Therefore, the ENDBLOCK should not be counted as a part of the comment.
+COMMENT: /[\n ]*\#(?!ENDBLOCK)(?:[^\n#]|#(?!ENDBLOCK))*/
 %ignore COMMENT
 
 // Internal symbol added by the preprocess_blocks function to indicate the end of blocks

hedy.py

@@ -2112,7 +2112,7 @@ def condition_spaces(self, meta, args):
 
     def equality_check(self, meta, args):
         arg0 = self.process_variable(args[0], meta.line)
-        arg1 = ' '.join([self.process_variable(a) for a in args[1:]])
+        arg1 = self.process_variable(str(args[1]).strip(), meta.line)
         return f"{arg0} == {arg1}"
         # TODO, FH 2021: zelfde change moet ik ook nog ff maken voor equal. check in hogere levels
 
@@ -2227,8 +2227,7 @@ def print_ask_args(self, meta, args):
 
     def equality_check(self, meta, args):
         arg0 = self.process_variable(args[0], meta.line)
-        remaining_text = ' '.join(args[1:])
-        arg1 = self.process_variable(remaining_text, meta.line)
+        arg1 = self.process_variable(str(args[1]).strip(), meta.line)
 
         # FH, 2022 this used to be str but convert_numerals in needed to accept non-latin numbers
         # and works exactly as str for latin numbers (i.e. does nothing on str, makes 3 into '3')
@@ -3413,6 +3412,12 @@ def preprocess_blocks(code, level, lang):
             processed_code.append('')
             continue
 
+        # ignore lines that contain only a comment
+        comment_reg_ex = r' *\#[^\n]*'
+        if regex.fullmatch(comment_reg_ex, line):
+            processed_code.append(line)
+            continue
+
         # first encounter sets indent size for this program
         if not indent_size_adapted and leading_spaces > 0:
             indent_size = leading_spaces

tests/test_level/test_level_04.py

@@ -218,6 +218,12 @@ def test_print_asterisk(self):
 
         self.multi_level_tester(code=code, expected=expected, max_level=11)
 
+    def test_print_hash(self):
+        code = "print 'comments start with the # sign'"
+        expected = "print(f'comments start with the # sign')"
+
+        self.multi_level_tester(code=code, expected=expected, max_level=11)
+
     def test_print_without_quotes_gives_error_from_grammar(self):
         # in some cases, there is no variable confusion since 'hedy 123' can't be a variable
         # then we can immediately raise the no quoted exception

tests/test_level/test_level_05.py

@@ -40,9 +40,10 @@ def test_if_equality_linebreak_print(self,  hedy, python):
         self.single_level_tester(code=code, expected=expected, unused_allowed=True)
 
     def test_if_equality_trailing_space_linebreak_print(self):
-        code = textwrap.dedent("""\
+        value = 'trailing_space  '
+        code = textwrap.dedent(f"""\
         naam is James
-        if naam is trailing_space
+        if naam is {value}
         print 'shaken'""")
 
         expected = textwrap.dedent("""\
@@ -66,9 +67,10 @@ def test_if_equality_unquoted_rhs_with_space_linebreak_print(self):
         self.multi_level_tester(code=code, expected=expected, max_level=7)
 
     def test_if_equality_unquoted_rhs_with_space_and_trailing_space_linebreak_print(self):
-        code = textwrap.dedent("""\
+        value = 'trailing space  '
+        code = textwrap.dedent(f"""\
         naam is James
-        if naam is trailing space
+        if naam is {value}
         print 'shaken'""")
 
         expected = textwrap.dedent("""\
@@ -416,10 +418,10 @@ def test_if_else_followed_by_print(self):
         self.single_level_tester(code=code, expected=expected)
 
     def test_if_equality_trailing_space_linebreak_print_else(self):
-        # this code has a space at the end of line 2
-        code = textwrap.dedent("""\
+        value = 'trailing space  '
+        code = textwrap.dedent(f"""\
         naam is James
-        if naam is trailing space
+        if naam is {value}
         print 'shaken' else print 'biertje!'""")
 
         expected = textwrap.dedent("""\

tests/test_level/test_level_06.py

@@ -176,12 +176,53 @@ def test_if_equality_linebreak_print(self):
         if convert_numerals('Latin', naam) == convert_numerals('Latin', 'Hedy'):
           print(f'leuk')""")
 
-        self.multi_level_tester(max_level=7, code=code, expected=expected)
+        self.multi_level_tester(max_level=7, code=code, expected=expected, output='leuk')
 
-    def test_if_equality_trailing_space_linebreak_print(self):
+    def test_if_equality_linebreak_comment_print(self):
+        code = textwrap.dedent("""\
+        naam is Hedy
+        if naam is Hedy
+        # this linebreak is allowed
+        print 'leuk'""")
+
+        expected = textwrap.dedent("""\
+        naam = 'Hedy'
+        if convert_numerals('Latin', naam) == convert_numerals('Latin', 'Hedy'):
+          print(f'leuk')""")
+
+        self.multi_level_tester(max_level=7, code=code, expected=expected, output='leuk')
+
+    def test_if_equality_comment_linebreak_print(self):
+        code = textwrap.dedent("""\
+        naam is Hedy
+        if naam is Hedy  # this linebreak is allowed
+        print 'leuk'""")
+
+        expected = textwrap.dedent("""\
+        naam = 'Hedy'
+        if convert_numerals('Latin', naam) == convert_numerals('Latin', 'Hedy'):
+          print(f'leuk')""")
+
+        self.multi_level_tester(max_level=7, code=code, expected=expected, output='leuk')
+
+    def test_if_equality_linebreak_print_comment(self):
         code = textwrap.dedent("""\
+        naam is Hedy
+        if naam is Hedy
+        print 'leuk'  # this linebreak is allowed""")
+
+        expected = textwrap.dedent("""\
+        naam = 'Hedy'
+        if convert_numerals('Latin', naam) == convert_numerals('Latin', 'Hedy'):
+          print(f'leuk')""")
+
+        self.multi_level_tester(max_level=7, code=code, expected=expected, output='leuk')
+
+    def test_if_equality_trailing_space_linebreak_print(self):
+        value = 'trailing_space  '
+        code = textwrap.dedent(f"""\
         naam is James
-        if naam is trailing_space
+        if naam is {value}
         print 'shaken'""")
 
         expected = textwrap.dedent("""\