windows: Build separate micropython dll and executable.

To enable building native dll modules export commonly used core
functions and data; functions are exported with a .def file so no code
change is required but exporting data using .def file yields multiple
copies of data so a dllexport/import macro seems the only solution.

MicroPython itself is split in a 2 projects: the original is copied
to micropythoncore.vcxproj and altered to produce a .dll and corresponding
import .lib, and the new micropython.vcxproj is just a stub around main().

Also provide a property sheet with default values for external modules,
and run.h/run.c with extra functions for embedding the interpreter.

.github/workflows/ports_windows.yml

@@ -61,7 +61,7 @@ jobs:
     - name: Build mpy-cross.exe
       run: msbuild mpy-cross\mpy-cross.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }}
     - name: Build micropython.exe
-      run: msbuild ports\windows\micropython.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} -property:PyVariant=${{ matrix.variant }} 
+      run: msbuild ports\windows\micropythoncore.vcxproj;ports\windows\micropython.vcxproj -maxcpucount -property:Configuration=${{ matrix.configuration }} -property:Platform=${{ matrix.platform }} -property:PyVariant=${{ matrix.variant }} 
     - name: Get micropython.exe path
       id: get_path
       run: |

README.md

@@ -1,5 +1,110 @@
 [![Unix CI badge](https://github.com/micropython/micropython/actions/workflows/ports_unix.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![STM32 CI badge](https://github.com/micropython/micropython/actions/workflows/ports_stm32.yml/badge.svg)](https://github.com/micropython/micropython/actions?query=branch%3Amaster+event%3Apush) [![Docs CI badge](https://github.com/micropython/micropython/actions/workflows/docs.yml/badge.svg)](https://docs.micropython.org/) [![codecov](https://codecov.io/gh/micropython/micropython/branch/master/graph/badge.svg?token=I92PfD05sD)](https://codecov.io/gh/micropython/micropython)
 
+The (windows-)pyd branch of MicroPython
+=======================================
+This fork of MicroPython (original readme below) allows building and importing
+of native modules in a CPython-like way: `import <module_name>` will consider
+the dynamic library `<module_name>.pyd` on windows or `<module_name>.so` on unix
+for import (in case `<module_name>.py` is not found), load it, and call it's
+`init_<module_name>()` function which should return a `mp_obj_module_t*`,
+which is then added to the scope.
+
+This fork gets  regularly rebased on the upstream.
+
+**Example for the unix port**
+- create a file mymodule.c with the following content
+``` C
+#include "py/runtime.h"
+
+static mp_obj_t foo() {
+  return MP_OBJ_NEW_QSTR(qstr_from_str("hello from foo"));
+}
+static MP_DEFINE_CONST_FUN_OBJ_0(foo_obj, foo);
+
+mp_obj_module_t *init_mymodule() {
+  mp_obj_t module = mp_obj_new_module(qstr_from_str("mymodule"));
+  mp_store_attr(module, qstr_from_str("foo"), MP_OBJ_FROM_PTR(&foo_obj));
+  return module;
+}
+```
+- set a shell variable UPYDIR to the location of this repository and build MicroPython
+```
+export UPYDIR=/path/to/micropython
+make -C $UPYDIR/ports/unix
+```
+- build the module and place it in the default search path
+```
+gcc -fPIC -I$UPYDIR -I$UPYDIR/ports/unix -I$UPYDIR/ports/unix/build -c mymodule.c -o mymodule.o
+mkdir -p ~/.micropython/lib
+gcc -shared -o ~/.micropython/lib/mymodule.so mymodule.o
+```
+- run micropython and import the module
+```
+$UPYDIR/ports/unix/micropython
+>>> import mymodule
+>>> mymodule.foo()
+'hello from foo'
+```
+
+**Example for the windows port**
+
+When using project files import [extmodule.props](https://github.com/stinos/micropython/blob/windows-pyd/ports/windows/msvc/extmodule.props) to get all options set correctly. Alternatively, here's a commandline sample:
+- create a file mymodule.c with the following content
+``` C
+#include "py/runtime.h"
+
+static mp_obj_t foo(void) {
+  return MP_OBJ_NEW_QSTR(qstr_from_str("hello from foo"));
+}
+
+__declspec(dllexport) mp_obj_module_t *init_mymodule() {
+  mp_obj_t module = mp_obj_new_module(qstr_from_str("mymodule"));
+  //MP_DEFINE_CONST_FUN_OBJ_0 won't compile since mp_type_fun_builtin_0 is
+  //dynamically imported so create the function on the heap instead.
+  mp_obj_fun_builtin_fixed_t *foo_obj = m_new_obj(mp_obj_fun_builtin_fixed_t);
+  foo_obj->base.type = &mp_type_fun_builtin_0;
+  foo_obj->fun._0 = foo;
+  mp_store_attr(module, qstr_from_str("foo"), MP_OBJ_FROM_PTR(foo_obj));
+  return module;
+}
+```
+- set a shell variable UPYDIR to the location of this repository and build MicroPython
+```
+$UPYDIR='/path/to/micropython'
+msbuild $UPYDIR/ports/windows/micropythoncore.vcxproj
+msbuild $UPYDIR/ports/windows/micropython.vcxproj
+```
+- build the module
+```
+cl /c /I$UPYDIR /I$UPYDIR/ports/windows /I$UPYDIR/ports/windows/msvc /I$UPYDIR/ports/windows/build mymodule.c
+link /DLL /OUT:mymodule_d.pyd $UPYDIR/ports/windows/build/Debugx64/micropythoncore.lib mymodule.obj
+```
+- run micropython and import the module
+```
+$UPYDIR/ports/windows/micropython
+>>> import mymodule
+>>> mymodule.foo()
+'hello from foo'
+```
+
+**Other examples**
+
+The [micropython-wrap repository](https://github.com/stinos/micropython-wrap) has a sample module, Makefile and VS/msbuild project files.
+
+**Note on module search path**
+
+On unix `dlopen()` is used to load libraries so it uses a different search path than what MicroPython uses to find modules. This isn't a problem when full paths are passed
+to `dlopen()` which is normally the case, except when MicroPython finds the .so file in
+the current directory for instance: to make that work use `LD_LIBRARY_PATH=$(pwd)` to have the loader
+find the .so file in the current directory.
+
+On windows the current directory is automatically searched for dynamic libraries so no problem there.
+
+**Note on module names**
+
+The CPython naming scheme is followed on windows so when `_DEBUG` is defined MicroPython looks for
+`<module_name>_d.pyd` instead of `<module_name>.pyd`.
+
 The MicroPython project
 =======================
 <p align="center">

mpy-cross/mpy-cross.vcxproj

@@ -94,6 +94,7 @@
     <ClCompile Include="$(PyBaseDir)mpy-cross\gccollect.c"/>
     <ClCompile Include="$(PyBaseDir)mpy-cross\main.c"/>
     <ClCompile Include="$(PyBaseDir)ports\windows\fmode.c" />
+    <ClCompile Remove="$(PyBaseDir)py\run.c" />
   </ItemGroup>
   <Import Project="$(PyMsvcDir)genhdr.targets" />
   <Import Project="$(CustomPropsFile)" Condition="exists('$(CustomPropsFile)')" />

ports/unix/Makefile

@@ -199,6 +199,7 @@ endif
 # source files
 SRC_C += \
 	main.c \
+	pymain.c \
 	loaddynlib.c \
 	gccollect.c \
 	unix_mphal.c \

ports/unix/main.c

@@ -78,6 +78,12 @@ long heap_size = 1024 * 1024 * (sizeof(mp_uint_t) / 4);
 #error "The unix port requires MICROPY_PY_SYS_ARGV=1"
 #endif
 
+#if defined(MICROPY_UNIX_COVERAGE)
+int handle_uncaught_exception(mp_obj_base_t *exc);
+int execute_from_lexer(int source_kind, const void *source, mp_parse_input_kind_t input_kind, bool is_repl);
+int cmain(int argc, char **argv);
+#endif
+
 STATIC void stderr_print_strn(void *env, const char *str, size_t len) {
     (void)env;
     ssize_t ret;
@@ -98,7 +104,7 @@ mp_handle_exception_t mp_handle_exception = {NULL, print_exception};
 // If exc is SystemExit, return value where FORCED_EXIT bit set,
 // and lower 8 bits are SystemExit value. For all other exceptions,
 // return 1.
-STATIC int handle_uncaught_exception(mp_obj_base_t *exc) {
+int handle_uncaught_exception(mp_obj_base_t *exc) {
     // check for SystemExit
     if (mp_obj_is_subclass_fast(MP_OBJ_FROM_PTR(exc->type), MP_OBJ_FROM_PTR(&mp_type_SystemExit))) {
         // None is an exit value of 0; an int is its value; anything else is 1
@@ -125,7 +131,7 @@ STATIC int handle_uncaught_exception(mp_obj_base_t *exc) {
 // Returns standard error codes: 0 for success, 1 for all other errors,
 // except if FORCED_EXIT bit is set then script raised SystemExit and the
 // value of the exit is in the lower 8 bits of the return value
-STATIC int execute_from_lexer(int source_kind, const void *source, mp_parse_input_kind_t input_kind, bool is_repl) {
+int execute_from_lexer(int source_kind, const void *source, mp_parse_input_kind_t input_kind, bool is_repl) {
     mp_hal_set_interrupt_char(CHAR_CTRL_C);
 
     nlr_buf_t nlr;
@@ -474,7 +480,7 @@ STATIC void sys_set_excecutable(char *argv0) {
 
 MP_NOINLINE int main_(int argc, char **argv);
 
-int main(int argc, char **argv) {
+int cmain(int argc, char **argv) {
     #if MICROPY_PY_THREAD
     mp_thread_init();
     #endif

ports/unix/pymain.c

@@ -0,0 +1,5 @@
+int cmain(int argc, char **argv);
+
+int main(int argc, char **argv) {
+    return cmain(argc, argv);
+}

ports/windows/Makefile

@@ -24,6 +24,7 @@ FROZEN_MANIFEST ?= variants/manifest.py
 
 # Define main target
 PROG ?= micropython
+DLL_PROG ?= micropythoncore.dll
 
 # qstr definitions (must come before including py.mk)
 QSTR_DEFS += ../unix/qstrdefsport.h
@@ -55,6 +56,7 @@ SRC_C = \
 	shared/libc/printf.c \
 	shared/runtime/gchelper_generic.c \
 	ports/unix/main.c \
+	ports/unix/pymain.c \
 	ports/unix/input.c \
 	ports/unix/gccollect.c \
 	ports/windows/msvc/loaddynlib.c \
@@ -73,6 +75,9 @@ OBJ += $(addprefix $(BUILD)/, $(SRC_CXX:.cpp=.o))
 OBJ += $(addprefix $(BUILD)/, $(SHARED_SRC_C:.c=.o))
 OBJ += $(addprefix $(BUILD)/, $(LIB_SRC_C:.c=.o))
 
+PROG_DLL_OBJ = $(BUILD)/ports/unix/pymain.o
+DLL_OBJ = $(filter-out $(BPROG_DLL_OBJ), $(OBJ))
+
 ifeq ($(MICROPY_USE_READLINE),1)
 CFLAGS += -DMICROPY_USE_READLINE=1
 SRC_C += shared/readline/readline.c
@@ -102,6 +107,14 @@ include $(TOP)/py/mkrules.mk
 
 .PHONY: test test_full
 
+# Note this one creates both the dll and the exe using it, since the latter target is defined in
+# mkrules.mk already and we want to keep it as default way to build.
+$(BUILD)/$(DLL_PROG): $(DLL_OBJ) $(PROG_DLL_OBJ)
+	$(ECHO) "LINK $@"
+	$(Q)$(CC) -o $@ $^ $(LIB) $(LDFLAGS) -shared
+	$(ECHO) "LINK $(BUILD)/$(PROG)"
+	$(Q)$(CC) -o $(BUILD)/$(PROG) $(PROG_DLL_OBJ) $(LIB) $(LDFLAGS) -L$(BUILD) -l$(subst .dll,,$(DLL_PROG))
+
 RUN_TESTS_SKIP += -e math_fun -e float2int_double -e float_parse -e math_domain_special
 
 test: $(BUILD)/$(PROG) $(TOP)/tests/run-tests.py

ports/windows/micropython.vcxproj

@@ -45,28 +45,32 @@
   </ImportGroup>
   <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="msvc/micropythoncore.props" />
     <Import Project="msvc/common.props" />
     <Import Project="msvc/debug.props" />
+    <Import Project="msvc/application.props" />
   </ImportGroup>
   <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="msvc/micropythoncore.props" />
     <Import Project="msvc/common.props" />
     <Import Project="msvc/release.props" />
+    <Import Project="msvc/application.props" />
   </ImportGroup>
   <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="msvc/micropythoncore.props" />
     <Import Project="msvc/common.props" />
     <Import Project="msvc/debug.props" />
+    <Import Project="msvc/application.props" />
   </ImportGroup>
   <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="msvc/micropythoncore.props" />
     <Import Project="msvc/common.props" />
     <Import Project="msvc/release.props" />
+    <Import Project="msvc/application.props" />
   </ImportGroup>
-  <PropertyGroup Label="UserMacros">
-    <CustomPropsFile Condition="'$(CustomPropsFile)'==''">msvc/user.props</CustomPropsFile>
-    <TargetName>$(PyProg)</TargetName>
-  </PropertyGroup>
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
     <ClCompile />
     <Link />
@@ -83,32 +87,12 @@
     <ClCompile />
     <Link />
   </ItemDefinitionGroup>
-  <Import Project="msvc/sources.props" />
-  <ItemGroup>
-    <ClCompile Include="@(PyCoreSource)" />
-    <ClCompile Include="@(PyExtModSource)" />
-    <ClCompile Include="$(PyBaseDir)shared\readline\*.c" />
-    <ClCompile Include="$(PyBaseDir)shared\runtime\gchelper_generic.c" />
-    <ClCompile Include="$(PyBaseDir)ports\windows\*.c" />
-    <ClCompile Include="$(PyBaseDir)ports\windows\msvc\*.c" />
-    <ClCompile Include="$(PyBaseDir)ports\unix\gccollect.c"/>
-    <ClCompile Include="$(PyBaseDir)ports\unix\input.c"/>
-    <ClCompile Include="$(PyBaseDir)ports\unix\main.c"/>
-    <ClCompile Include="$(PyVariantDir)*.c" />
-  </ItemGroup>
   <ItemGroup>
-    <ClInclude Include="@(PyCoreInclude)" />
-    <ClInclude Include="@(PyExtModInclude)" />
-    <ClInclude Include="$(PyBaseDir)ports\windows\*.h" />
-    <ClInclude Include="$(PyBaseDir)ports\windows\msvc\*.h" />
+    <ClCompile Include="..\unix\pymain.c" />
   </ItemGroup>
   <ItemGroup>
     <ResourceCompile Include="micropython.rc" />
   </ItemGroup>
-  <Import Project="msvc/genhdr.targets" />
-  <Import Project="$(CustomPropsFile)" Condition="exists('$(CustomPropsFile)')" />
-  <Target Name="GenerateMicroPythonSources" BeforeTargets="BuildGenerateSources" DependsOnTargets="GenerateHeaders;FreezeModules">
-  </Target>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>