Pybricks
While printing a lot of utf-8 characters a memory error occurs on a 20k allocation #1977
Replies: 1 comment 5 replies
-
|
Can you share the full program? |
Beta Was this translation helpful? Give feedback.
-
|
Sure. It's smaller already. UTF_CONVERSION = True # False # THIS switch determines if chr(xx) is used (True) or not (False)from pybricks.tools import wait, StopWatch
import gc
watch = StopWatch()
RANGES = [
(0x000020, 0x00007F), # ONE byte length unicode
(0x000080, 0x0007FF), # TWO byte length unicode
(0x000800, 0x0009FF), # THREE byte length unicode
(0x003400, 0x004DBF), # THREE byte length unicode - (CJK Unified Ideographs Extension A)
(0x004E00, 0x009FFF), # THREE byte length unicode - (CJK Unified Ideographs)
(0x01F680, 0x01F6FF), # FOUR byte length unicode - some nice icons
(0x020000, 0x02A6DF), # FOUR byte length unicode - (CJK Unified Ideographs Extension B)
# (0x02A700, 0x02A7FF), # FOUR byte length unicode - (CJK Unified Ideographs Extension C)
]
UTF_CONVERSION = True # False # THIS switch determines if chr(xx) is used (True) or not (False)
CHARS_PER_LINE = 32
prev_free = 99999
for start, end in RANGES:
for base in range(start, end, CHARS_PER_LINE):
print(f"{base:#08x} | ", end="")
for offset in range(CHARS_PER_LINE):
code = base + offset
try:
if UTF_CONVERSION:
print(chr(code), end=' ')
else:
print(end="😀 ")
# __ = chr(code) # just to convert not to print
except ValueError:
print(end='? ')
print(end='|')
# after each line a garbage collect.
gc.collect()
mem_free = gc.mem_free()
used = mem_free - prev_free
prev_free = mem_free
print(f" Free {mem_free:>6} {used:>6}")
print(f"Ready; duration: {watch.time() / 60000:6.1f} Minutes")Tail of the output: It takes some time to get there. A similar program did something similar running micropyton print_utf8_mre_0_mpy_mem_report.py
import gc
# from pybricks.tools import StopWatch
#watch = StopWatch()
import time
watch = time.localtime()
starttime = time.time()
RANGES = [ # Make sure no invalid utf-8 codepoints in these ranges. No validity check in this code
(0X000021, 0X00007F), # ONE byte length unicode
(0X000080, 0X0007FF), # TWO byte length unicode
(0X000800, 0X0009FF), # THREE byte length unicode
(0X003400, 0X004DBF), # THREE byte length unicode - (CJK Unified Ideographs Extension A)
(0X004E00, 0X009FFF), # THREE byte length unicode - (CJK Unified Ideographs)
(0X01F680, 0X01F6FF), # FOUR byte length unicode - some nice icons
(0X020000, 0X02A6DF), # FOUR byte length unicode - (CJK Unified Ideographs Extension B)
(0X02A700, 0X02A7FF), # FOUR byte length unicode - (CJK Unified Ideographs Extension C)
(0X02A800, 0X10FFFF), # just many to stress python3
]
UTF_CONVERSION = True # THIS switch determines if chr(xx) is used (True) or not (False)
CHARS_PER_LINE = 16
prev_free = 255808 # or whatever . . .
for start, end in RANGES:
for base in range(start, end, CHARS_PER_LINE):
print(end=f"{base:#08x}|")
for offset in range(CHARS_PER_LINE):
code = base + offset
try:
if UTF_CONVERSION:
print(chr(code), end=' ')
else:
print(end="😀 ") # just print (no conversion)
except ValueError:
print(end='? ')
print(end='|')
# after each line a garbage collect.
gc.collect()
mem_free = gc.mem_free()
print(f" {mem_free:>6} dif {mem_free - prev_free:>6}")
prev_free = mem_free
time_diff = time.time() - starttime
print(f"Ready; duration: {time_diff:6.1f} Minutes")Tail of the micropython output: [EDIT} clarification here |
Beta Was this translation helpful? Give feedback.
-
|
Ran the pybricks test on firmware / hardware: It's so easy to forget important info 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I found this bug which looks like it could be the source of the error: micropython/micropython#4422 |
Beta Was this translation helpful? Give feedback.
-
|
This one does not use garbage collect and thus runs into problems earlier ( just over 20 seconds) import gc
RANGES = [ # Make sure no invalid utf-8 codepoints in these ranges. No validity check in this code
(0X000020, 0X0009FF), # ONE and TWO and THREE byte length unicode
(0X003400, 0X004DBF), # THREE byte length unicode - (CJK Unified Ideographs Extension A)
(0X004E00, 0X009FFF), # THREE byte length unicode - (CJK Unified Ideographs)
(0X020000, 0X02A6DF), # FOUR byte length unicode - (CJK Unified Ideographs Extension B)
(0X02A700, 0X02A7FF), # FOUR byte length unicode - (CJK Unified Ideographs Extension C)
(0X02A800, 0X10FFFF), # just many to stress python3
]
UTF_CONVERSION = True # True # THIS switch determines if chr(xx) is used (True) or not (False)
CHARS_PER_LINE = 16
for start, end in RANGES:
print(f'\t\tNew range {start:#08x} end {end:#08x}')
for code_start in range(start, end, CHARS_PER_LINE):
line = [
chr(code) if UTF_CONVERSION else "😀"
for code in range(code_start, min(code_start + CHARS_PER_LINE, end + 1))
]
print(f'{code_start:#08x}', ' '.join(line), '|')Tail of the output: |
Beta Was this translation helpful? Give feedback.
-
I know you guys are fast, but so quick 🚀 |
Beta Was this translation helpful? Give feedback.
-
This is not a question asking to debug my program, you all have plenty other things to do.
If I print a lot of unicode characters (even with extra garbage collect), in the end I run into:
on a simple:
With as an example
ch == 0X28FC0Why is the allocation always as large as 20k and could there be a memory leak in unicode processing?
I tried to make a proof of it ( a long time), but I can not get a solid short program.
Bert
Beta Was this translation helpful? Give feedback.
All reactions