Skip to content

Commit

Permalink
Replaced std::shared_ptr with dsr::Handle and introduced textures as …
Browse files Browse the repository at this point in the history
…separate types.
  • Loading branch information
Dawoodoz committed Jan 25, 2025
1 parent ae1a8bf commit 5a2eee2
Show file tree
Hide file tree
Showing 115 changed files with 7,951 additions and 6,323 deletions.
6 changes: 3 additions & 3 deletions Doc/Buffers.html
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<A href="Manual.html">Back to main page</A>
</P><P>
</P><H1> Buffers</H1><P>Every file that is saved or loaded in the framework will pass through a Buffer.
Buffers can not refer to each other in cycles and are automatically reference counted and deleted, so that you don't have to worry about memory leaks from them unless you explicitly call buffer_replaceDestructor.
Buffers can not refer to each other in cycles and are automatically reference counted and deleted, so that you don't have to worry about memory leaks unless something holding a buffer creates a cycle of handles.
They store a fixed size allocation of memory padded and aligned with DSR_MAXIMUM_ALIGNMENT bytes to work well with the largest SIMD vectors without false sharing of cache lines between threads.

</P><P>
Expand All @@ -41,13 +41,13 @@

</P><P>
If you create a buffer of size zero, it will allocate the head but not the data.
Trying to clone an empty buffer head will just return the same handle without cloning, because empty buffers are immutable.
Trying to clone an empty buffer will just return the same handle without cloning, because empty buffers are immutable.
</P><IMG SRC="Images/Border.png"><P>
</P><H2> Read and write data access</H2><P>
</P><P>
Trying to get the pointer of a non-existing or zero length Buffer will safely return a null pointer, no matter if you use buffer_getSafeData<type>(buffer, "Buffer name") or buffer_dangerous_getUnsafeData(buffer).
You access the data by getting a SafePointer, which can later be sliced into smaller parts.
Sometimes you can't use the SafePointer because an operating system wants a regular C pointer.
Sometimes you can't use the SafePointer because an operating system wants a regular pointer.
</P><IMG SRC="Images/Border.png"><P>
</P>
</BODY> </HTML>
6 changes: 3 additions & 3 deletions Doc/Generator/Input/Buffers.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Title: Buffers
Every file that is saved or loaded in the framework will pass through a Buffer.
Buffers can not refer to each other in cycles and are automatically reference counted and deleted, so that you don't have to worry about memory leaks from them unless you explicitly call buffer_replaceDestructor.
Buffers can not refer to each other in cycles and are automatically reference counted and deleted, so that you don't have to worry about memory leaks unless something holding a buffer creates a cycle of handles.
They store a fixed size allocation of memory padded and aligned with DSR_MAXIMUM_ALIGNMENT bytes to work well with the largest SIMD vectors without false sharing of cache lines between threads.

---
Expand All @@ -15,11 +15,11 @@ To create a buffer that actually stores something, call buffer_create with the n
The memory always start initialized to zero, which prevents random bugs.

If you create a buffer of size zero, it will allocate the head but not the data.
Trying to clone an empty buffer head will just return the same handle without cloning, because empty buffers are immutable.
Trying to clone an empty buffer will just return the same handle without cloning, because empty buffers are immutable.
---
Title2: Read and write data access

Trying to get the pointer of a non-existing or zero length Buffer will safely return a null pointer, no matter if you use buffer_getSafeData<type>(buffer, "Buffer name") or buffer_dangerous_getUnsafeData(buffer).
You access the data by getting a SafePointer, which can later be sliced into smaller parts.
Sometimes you can't use the SafePointer because an operating system wants a regular C pointer.
Sometimes you can't use the SafePointer because an operating system wants a regular pointer.
---
66 changes: 25 additions & 41 deletions Doc/Generator/Input/Troubleshooting.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,31 @@

Title: Troubleshooting

When using a specific framework, the common mistakes and solutions are usually very similar.
This guide explains both the steps for finding your bugs, and reducing the risk of them comming back.
---
Title2: To use or not use an IDE with a built-in debugger

*
If your low-level code is crashing often from advanced optimizations,
you might need an IDE with a built-in debugger to quickly show where the crash happened.
The IDE integration can then directly point to the code instead of showing line numbers from a separate debugger.
If your program crashes with segmentation faults, start by replacing pointers with SafePointer and building the program in debug mode.

*
If your high-level code only crashes rarely but the amount of pixel data is too much for a debugger,
create a debug window showing internal images or debug overlays with coordinates on top of the program's existing graphics.
In the Builder build system, debug mode is activated by writing 'Debug' in the *.DsrProj project file, which declares the Debug variable and assigns it to one.

For other build systems, give -DDEBUG to the compiler to define the DEBUG macro.
Make sure that the release flag -DNDEBUG is not also active.

Then memory.h will enable the SAFE_POINTER_CHECKS macro from detecting debug mode.
Then SafePointer.h will store the permitted region in each SafePointer and perform bound checks when data is accessed using SafePointer.

---
Title2: Finding the cause of bugs takes too long.
Title2: Getting random memory crashes.

*
Unless you are profiling, test in debug mode using the -DDEBUG compiler flag.
This catches bugs earlier with more information about the crash.
Make sure that the release flag -DNDEBUG is not also active.
If your program is getting random memory corruption despite using SafePointer and debug mode, continue by enabling the EXTRA_SAFE_POINTER_CHECKS macro.

In the Builder build system, extra safe memory checks can be enabled for debug mode by writing 'CompilerFlag "-DEXTRA_SAFE_POINTER_CHECKS"' in the *.DsrProj project file.
EXTRA_SAFE_POINTER_CHECKS can also be defined as a macro in settings.h or by giving -DEXTRA_SAFE_POINTER_CHECKS to a different build system.

Then SafePointer will check that the allocation has not been replaced by another allocation in heap.cpp.
SafePointer will also check that no thread is trying to access virtual stack memory allocated by another thread.

*
If using raw pointers, you might want to replace them with the SafePointer class to get tighter bound checks in debug mode.
Debuggers will wait until your bugs write outside of the whole allocation before throwing an error.
If your program does not crash but your image filters do not work as expected, create a debug window showing internal images or debug overlays with coordinates on top of the program's existing graphics.

*
Make sure that all your multi-threading can be turned off easily when finding the root cause.
Expand All @@ -36,11 +36,16 @@ Create a basic reference implementation without dangerous optimizations for ever
Both for finding the cause of instability and being able to remove a feature without sending emergency patches in panic with more bugs.
Image filters are first written using lambdas returning the color of a pixel based on the pixel coordinate and exception-free pixel sampling.
Then one can make an optimized version using SafePointer and SIMD vectors.

---
Title2: Getting memory leaks.

In debug mode, terminating the program should print "All heap memory was freed without leaks.".
If it does not, you might have a memory leak for memory allocated by the framework in heap.cpp.
Due to the non-deterministic release order for global variables in C++, it is not possible to print a warning when there is a memory leak without redefining the _start function.

*
Avoid using manual memory management (malloc, free, new, delete...), because it is a waste of time unless you are writing a new abstraction layer.
Avoid using manual memory management (malloc, free, new, delete...), use dsr::Handle for object handles.

*
Make sure that no reference counted object can create a cycle of reference counted pointers back to itself, because then none of them would be unused according to reference counting.
Expand All @@ -49,40 +54,19 @@ Make sure that no reference counted object can create a cycle of reference count
Use the dsr::Buffer object instead of C allocation calls, to let it automatically free your memory when nobody keeps the reference counted handle.
You can then work on its memory using SafePointer, which provides bound checks in debug mode, but must be kept close to the buffer's reference counted handle to keep the data it points to alive.
<- Buffers.html | Read about the Buffer API
---
Title2: Getting random memory crashes.

*
Check that you are not using raw C pointers by searching for any use of &, *, [] in the code and replacing them with SafePointer to get bound checks.

*
Make sure that you are using debug mode, so that outside access with SafePointer is caught with error messages.

*
Make sure that no SafePointer outlives the parent Buffer, because SafePointer is not reference counting on its own.
If SafePointer would be reference counting, it would not be a zero overhead substitution for raw C pointers in release mode, and nobody would use it for optimizations.

*
Remember that a reference in C++ is a pointer under the C++ syntax, which can also cause crashes if taken from a location in memory that may be freed during the call.
If you passed "const ReadableString &text" from "List<String>" to a function that can reallocate the list that the string is stored in, this can cause a crash by referring to a memory location that got replaced by the list.
If you instead pass "ReadableString text" from "List<String>", no additional heap allocations will be made, but activating reference counting makes sure that the string can be passed around independently from where it came from without causing any crashes.
If you passed "const SomeClass &object" from "List<SomeClass>" to a function that can reallocate the list that the object is stored in, this can cause a crash by referring to a memory location that got replaced by the list.
If you instead pass "SomeClass object" from "List<SomeClass>", the object will be copied in the call instead of referring to freed memory.
Returning a reference to a stack allocated variable, can also cause crashes with references.

*
Assert bounds with assertions in debug mode for fixed size and variable length (VLA) arrays, which have no bound checks in C/C++ but are much faster than heap memory for many small allocations.
Getters and setters can make the bound checks reusable if you only have a few types with fixed size arrays.
---
Title2: The application crashes, but the debugger does not detect it.

Use a debugger directly on the application with debug symbols enabled when compiling.
Connecting Valgrind to the script used to run your application will catch memory leaks, but not invalid memory access.
Without debug symbols, you can see which method crashed, but not the line.

---
Title2: Getting crashes after linking to an external library.

Try disabling the library's memory recycling, by either removing the ReuseMemory flag in your DsrProj build script (cleanest way) or adding the -DDISABLE_ALLOCATOR compiler flag to define the DISABLE_ALLOCATOR macro (works with other build systems).
This will disable the library's memory recycling at DFPSR/base/allocator.cpp in case that another library already has a memory recycler.
---

Title2: Getting linker errors when creating a new project without a window.
Expand Down
77 changes: 32 additions & 45 deletions Doc/Troubleshooting.html
Original file line number Diff line number Diff line change
Expand Up @@ -26,34 +26,38 @@
</P><P>
</P><H1> Troubleshooting</H1><P>
</P><P>
When using a specific framework, the common mistakes and solutions are usually very similar.
This guide explains both the steps for finding your bugs, and reducing the risk of them comming back.
</P><IMG SRC="Images/Border.png"><P>
</P><H2> To use or not use an IDE with a built-in debugger</H2><P>
</P><P>
<IMG SRC="Images/SmallDot.png">
If your low-level code is crashing often from advanced optimizations,
you might need an IDE with a built-in debugger to quickly show where the crash happened.
The IDE integration can then directly point to the code instead of showing line numbers from a separate debugger.
If your program crashes with segmentation faults, start by replacing pointers with SafePointer and building the program in debug mode.

</P><P>
<IMG SRC="Images/SmallDot.png">
If your high-level code only crashes rarely but the amount of pixel data is too much for a debugger,
create a debug window showing internal images or debug overlays with coordinates on top of the program's existing graphics.
In the Builder build system, debug mode is activated by writing 'Debug' in the *.DsrProj project file, which declares the Debug variable and assigns it to one.

</P><P>
For other build systems, give -DDEBUG to the compiler to define the DEBUG macro.
Make sure that the release flag -DNDEBUG is not also active.

</P><P>
Then memory.h will enable the SAFE_POINTER_CHECKS macro from detecting debug mode.
Then SafePointer.h will store the permitted region in each SafePointer and perform bound checks when data is accessed using SafePointer.

</P><P>
</P><IMG SRC="Images/Border.png"><P>
</P><H2> Finding the cause of bugs takes too long.</H2><P>
</P><H2> Getting random memory crashes.</H2><P>
</P><P>
<IMG SRC="Images/SmallDot.png">
Unless you are profiling, test in debug mode using the -DDEBUG compiler flag.
This catches bugs earlier with more information about the crash.
Make sure that the release flag -DNDEBUG is not also active.
If your program is getting random memory corruption despite using SafePointer and debug mode, continue by enabling the EXTRA_SAFE_POINTER_CHECKS macro.

</P><P>
In the Builder build system, extra safe memory checks can be enabled for debug mode by writing 'CompilerFlag "-DEXTRA_SAFE_POINTER_CHECKS"' in the *.DsrProj project file.
EXTRA_SAFE_POINTER_CHECKS can also be defined as a macro in settings.h or by giving -DEXTRA_SAFE_POINTER_CHECKS to a different build system.

</P><P>
Then SafePointer will check that the allocation has not been replaced by another allocation in heap.cpp.
SafePointer will also check that no thread is trying to access virtual stack memory allocated by another thread.

</P><P>
<IMG SRC="Images/SmallDot.png">
If using raw pointers, you might want to replace them with the SafePointer class to get tighter bound checks in debug mode.
Debuggers will wait until your bugs write outside of the whole allocation before throwing an error.
If your program does not crash but your image filters do not work as expected, create a debug window showing internal images or debug overlays with coordinates on top of the program's existing graphics.

</P><P>
<IMG SRC="Images/SmallDot.png">
Expand All @@ -65,11 +69,18 @@
Both for finding the cause of instability and being able to remove a feature without sending emergency patches in panic with more bugs.
Image filters are first written using lambdas returning the color of a pixel based on the pixel coordinate and exception-free pixel sampling.
Then one can make an optimized version using SafePointer and SIMD vectors.

</P><P>
</P><IMG SRC="Images/Border.png"><P>
</P><H2> Getting memory leaks.</H2><P>
</P><P>
In debug mode, terminating the program should print "All heap memory was freed without leaks.".
If it does not, you might have a memory leak for memory allocated by the framework in heap.cpp.
Due to the non-deterministic release order for global variables in C++, it is not possible to print a warning when there is a memory leak without redefining the _start function.

</P><P>
<IMG SRC="Images/SmallDot.png">
Avoid using manual memory management (malloc, free, new, delete...), because it is a waste of time unless you are writing a new abstraction layer.
Avoid using manual memory management (malloc, free, new, delete...), use dsr::Handle for object handles.

</P><P>
<IMG SRC="Images/SmallDot.png">
Expand All @@ -79,45 +90,21 @@
<IMG SRC="Images/SmallDot.png">
Use the dsr::Buffer object instead of C allocation calls, to let it automatically free your memory when nobody keeps the reference counted handle.
You can then work on its memory using SafePointer, which provides bound checks in debug mode, but must be kept close to the buffer's reference counted handle to keep the data it points to alive.
<A href="Buffers.html">Read about the Buffer API</A></P><IMG SRC="Images/Border.png"><P>
</P><H2> Getting random memory crashes.</H2><P>
</P><P>
<IMG SRC="Images/SmallDot.png">
Check that you are not using raw C pointers by searching for any use of &, *, [] in the code and replacing them with SafePointer to get bound checks.

</P><P>
<IMG SRC="Images/SmallDot.png">
Make sure that you are using debug mode, so that outside access with SafePointer is caught with error messages.

</P><P>
<IMG SRC="Images/SmallDot.png">
Make sure that no SafePointer outlives the parent Buffer, because SafePointer is not reference counting on its own.
If SafePointer would be reference counting, it would not be a zero overhead substitution for raw C pointers in release mode, and nobody would use it for optimizations.

<A href="Buffers.html">Read about the Buffer API</A>
</P><P>
<IMG SRC="Images/SmallDot.png">
Remember that a reference in C++ is a pointer under the C++ syntax, which can also cause crashes if taken from a location in memory that may be freed during the call.
If you passed "const ReadableString &text" from "List<String>" to a function that can reallocate the list that the string is stored in, this can cause a crash by referring to a memory location that got replaced by the list.
If you instead pass "ReadableString text" from "List<String>", no additional heap allocations will be made, but activating reference counting makes sure that the string can be passed around independently from where it came from without causing any crashes.
If you passed "const SomeClass &object" from "List<SomeClass>" to a function that can reallocate the list that the object is stored in, this can cause a crash by referring to a memory location that got replaced by the list.
If you instead pass "SomeClass object" from "List<SomeClass>", the object will be copied in the call instead of referring to freed memory.
Returning a reference to a stack allocated variable, can also cause crashes with references.

</P><P>
<IMG SRC="Images/SmallDot.png">
Assert bounds with assertions in debug mode for fixed size and variable length (VLA) arrays, which have no bound checks in C/C++ but are much faster than heap memory for many small allocations.
Getters and setters can make the bound checks reusable if you only have a few types with fixed size arrays.
</P><IMG SRC="Images/Border.png"><P>
</P><H2> The application crashes, but the debugger does not detect it.</H2><P>
</P><P>
Use a debugger directly on the application with debug symbols enabled when compiling.
Connecting Valgrind to the script used to run your application will catch memory leaks, but not invalid memory access.
Without debug symbols, you can see which method crashed, but not the line.

</P><P>
</P><IMG SRC="Images/Border.png"><P>
</P><H2> Getting crashes after linking to an external library.</H2><P>
</P><P>
Try disabling the library's memory recycling, by either removing the ReuseMemory flag in your DsrProj build script (cleanest way) or adding the -DDISABLE_ALLOCATOR compiler flag to define the DISABLE_ALLOCATOR macro (works with other build systems).
This will disable the library's memory recycling at DFPSR/base/allocator.cpp in case that another library already has a memory recycler.
</P><IMG SRC="Images/Border.png"><P>

</P><P>
Expand Down
Loading

0 comments on commit 5a2eee2

Please sign in to comment.