Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Awesome #1

Open
bamyazi opened this issue Feb 10, 2020 · 14 comments
Open

Awesome #1

bamyazi opened this issue Feb 10, 2020 · 14 comments

Comments

@bamyazi
Copy link

bamyazi commented Feb 10, 2020

This is exactly what i've been hunting for, particularly green threading. I'm also working on a game which features python programming on a virtual computer/network - i've been using IronPython until now, which all works until the player creates a tight loop in a python script and steals all the CPU cycles. I started to build something similar, but this looks way more developed. Look forward to having a play and hopefully contributing.
@rockobonaparte
Copy link
Owner

Thanks!

If you're not really needing the coroutines then you might luck out getting the DLR's insides to just yield every once in awhile. You'd basically just inject some instructions that have it pause for air. The reason I didn't try that myself was because my real killer was scripts that required other subsystems across multiple frames to produce some result to continue. If I wrote a regular IronPython script, that kind of thing would hang since the interpreter runs in the same thread as the game logic. I caused some bad explosions when I tried to use async-await in that kind of environment, and it looked like I was eating up a lot of resources if I wanted to create contexts for every simultaneous script I wanted to run.

I did take other parts of IronPython for granted though. Dropping .NET objects into the interpreter is pretty trivial in IronPython, but it turns out to be a huge chore in reflection. I can't really prove how well this whole thing works in Unity until I'm interacting with some of the stuff in there. Sure, I got a virtual console to run arbitrary code, but that code was pure Python.

You might notice the most recent commits have had to do with operators. I'm at the point of trying to integrate events so I needed += and -=.

Finally, one of the perks of purely interpreting everything is that the script state can theoretically be serialized. I don't think I could do that with IronPython, although IronPython would definitely run scripts much faster.

@bamyazi
Copy link
Author

bamyazi commented Feb 10, 2020

I'm just having a good read through the code at the moment, looks like it will drop in to what i'm doing very easily : video here : www.dropbox.com/s/pph7j5hwmxctwa6/2020-02-09%2017-05-24.flv?dl=0 (warning there is a flashing light in the video). The terminal in the video is a fully emulated ECMA48 device with stdin/out support - i'll have a go later to see if i can get this integrated instead of iron python, my tests are very similar to your GuiDemo.

@rockobonaparte
Copy link
Owner

That's giving me some Minecraft + ComputerCraft vibes right there!

@bamyazi
Copy link
Author

bamyazi commented Feb 10, 2020 •
edited
Loading

Haha very true, i'm terrible at Blender so MagicaVoxel is my goto modelling tool for prototyping things. I hadn't heard of ComputerCraft before - looks interesting.

https://www.dropbox.com/s/lmkzw9zmucd6s2w/success.png?dl=0
Replaced IronPython and succesfully ran a script in game with your engine :) Very cool - I like what you've done a lot. If you're happy for me to get involved i'd like to have a go implementing the PyFile datatype since that would let me implement stdin/out for my terminal. The existing datatypes make it pretty clear what i need to do. I'm thinking it would be a wrapper around a Stream - any guidance how you'd like it implemented ?

@rockobonaparte
Copy link
Owner

Implementing PyFile would be a pretty big undertaking. I haven't looked at the pedantic details of file I/O but I can think of some problems:

  • You'd probably also have to implement open()
  • I don't have context managers so the with statement won't work: with open("blablabla") as xyz: ...
  • The hooks to the .NET code would need custom awaiters using ISubscheduledContinuation so that blocking I/O operations only block the script, not the entire engine.
  • I haven't really thought about it but there are probably newer asynchronous .NET classes to use for that kind of thing anyways.
  • You'd want to make sure that the handles don't get hogged by the interpreter despite going out of context (resource leak).
  • And of course if you want to put in a PR then I'd expect some unit (integration?) test coverage for it. I am figuring it will be easier to test in two steps. First is direct testing of PyFile and second is code tests. I haven't done this with most of the existing data types just because they aren't leaning on things like I/O that are much more complicated.
  • Ultimately, having file I/O at all is probably something I'd have to easily turn on and off or implement with some kind of protections so users (the players) can't arbitrarily open and edit whatever files on the actual disk with injected code.

@bamyazi
Copy link
Author

bamyazi commented Feb 10, 2020

Yeah it's probably not the easiest one to go for, but it's pretty much the core of what i'm doing - i have something similar up and running in the my engine already - there is a virtual file system, and i hijack 'open' with some hacking of the IronPython source to return a custom Stream rather than a file stream so accessing real files isn't possible - i ripped out a lot of stuff from IronPython for exactly the reasons you mention so players couldn't escape the sandbox - it's an issue i'm aware of. I think initially to get things working a simple open method via the Builtin interface would work. I use TDD for pretty much everything these days so Unit Tests are a given - have a look at my Twitter @mutatedmedia for some equally deranged projects ;) I'm up to the task.. I'll get something basic up and running and create a PR just for discussing it until you're happy.

@bamyazi
Copy link
Author

bamyazi commented May 15, 2020

I'm back (whether or not that's a good thing :)...unfortunately real life got in the way there for a bit, but we're back on track and lockdown means i've had some time to spend on this again. Looks like you've been very busy. I got rid of my initial fork and created an new clean one just to start over really, and wow everything seems to be working, i fleshed out my file io classes a bit to test out the inheritance and all seems fine - and a simple program

file = open("test.dat","r")
line = file.readline(1)
print(line)
file.close()

runs 👍 . So now i need to fully implement the python spec for file io stuff.
So i have a couple of question (I know it didn't take me long - sorry, at least it's not problems this time). First up when creating new classes i didn't see a way to specify optional parameters eg. the TextIoWrapper spec defines readline

readline(limit=-1)
Read and return one line from the stream. If limit is specified, at most limit bytes will be read.

I tried implementing this as

[ClassMember]
public static PyString readline(PyIOBase self, PyInteger size = null)

and modifying Injector.cs

             if (in_param_i >= args.Length)
             {
               // We have a missing parameter
               if (paramInfo.HasDefaultValue)
               {
                 // But it has a default
                 outParams[out_param_i] = paramInfo.DefaultValue==null ? null : PyNetConverter.Convert(paramInfo.DefaultValue, paramInfo.ParameterType);
               }
               else
               {
                 throw new Exception("Missing parameter.");
               }
             }
             else
             {
               outParams[out_param_i] = PyNetConverter.Convert(args[in_param_i], paramInfo.ParameterType);
             }

and handling the default in the PyClass

[ClassMember]
public static PyString readline(PyIOBase self, PyInteger size = null)
{
if (size==null) size = PyInteger.Create(-1);
return PyString.Create(self.Readline(size.number));
}

which works but feels a bit awkward but since we can't do

public static PyString readline(PyIOBase self, PyInteger size = PyInteger.Create(-1))

I don't really see any other option ?

@rockobonaparte
Copy link
Owner

rockobonaparte commented May 16, 2020 via email

@rockobonaparte
Copy link
Owner

I was thinking about your injector modifications. I probably just screwed that all up for you with what I just did with extension methods. I was about to consider refactoring that code given what I know now between normal arguments, generic parameters, and the 'this class' that shows up for extension methods. Now I realize I should hold off until we have an understand of optional arguments. Throw injected arguments into this and you might say there's a problem!

Consider that code to have become rather brittle with all the off-by-one shenanigans that can erupt from the different permutations of methods. I really suffered to support, say, generic extension methods that have arguments! I have a bunch of tests added in EmbeddingTests for all of that. I'm also wondering if I want to isolate all the code that's matching and binding everything so I can test it without doing code tests.

@bamyazi
Copy link
Author

bamyazi commented May 19, 2020

Not a problem, i'm currently really just working through the code again to get familiar with it all - and testing out things that i need to get working to have file/stream support - i'll just update as you go along and fit in with any changes. I'm currently trying to wrap by head around the best way to handle the blocking io and understanding the IScheduledAwaiter/FutureAwaiter stuff

@rockobonaparte
Copy link
Owner

rockobonaparte commented May 25, 2020 via email

@rockobonaparte
Copy link
Owner

rockobonaparte commented May 26, 2020 via email

@rockobonaparte
Copy link
Owner

Here's a current, real-life example of the future stuff. The FutureAwaiter wraps a lot of the sausagemaking the scheduler does into something you'd expect from a Future design pattern. You can see an example of that in concurrent.futures in Python, and other languages have a version of it. It's normally a way to have one thread block until a result is available from elsewhere. I have a special version of it because we're not multithreading.

You can think of it as asynchronously getting an IOU from a function call that you'd normally block on when you absolutely, positively need the value to continue doing whatever you have to do. It's been usurped here to have the scheduler take the IOU and ignore the script until it has been fulfilled. That's what puts it in the blocked state.

Here's a script. The prompt() is using it:

import GlobalState, DayPhase
from UnityEngine import Debug

from game import prompt

dayNight = GlobalState.Instance.DayNightTracker

if dayNight.Phase == DayPhase.Day:
    prompt_text = "Take a nap until night time?"
elif dayNight.Phase == DayPhase.Night:
    prompt_text = "Go to bed until morning?"
else:
    raise Exception("BedScript didn't recognize this day/night phase and couldn't change it: " + dayNight.Phase);

selected_idx = prompt(prompt_text, ["No", "Yes"])

if selected_idx == 1:
    if dayNight.Phase == DayPhase.Day:
        dayNight.Phase = DayPhase.Night
    elif dayNight.Phase == DayPhase.Night:
        dayNight.Phase = DayPhase.Day

Here's the implementation of prompt in C#:

    public async Task<FutureAwaiter<PyInteger>> Script_Prompt(IScheduler scheduler, FrameContext context, PyModule ignored, string prompt, PyList choices)
    {
        var choiceCollection = new List<string>();
        foreach (var choice in choices)
        {
            choiceCollection.Add(choice.ToString());
        }
        DialogRequest newRequest = new DialogRequest(prompt, choiceCollection.ToArray());
        SetupForRequest(newRequest);

        var future = new FutureAwaiter<PyInteger>(scheduler, context);
        void fireWhenDialogDone(DialogRequest request)
        {
            request.WhenDialogRequestIsDone -= fireWhenDialogDone;
            future.SetResult(PyInteger.Create(request.ChosenIndex));
        }
        newRequest.WhenDialogRequestIsDone += fireWhenDialogDone;

        scheduler.NotifyBlocked(context, future);
        await future;
        return future;
    }

What you need to do is figure out internally how you're going to asynchronously fulfill the result in your own runtime. In my case, it's an event that's fired by my dialog subsystem when a choice is made. When that fires, I call SetResult() on the future. That kicks it out of the blocked queue and puts it into the active queue.

To complete the contract with the scheduler, you have to do the stuff at the end. That tells the scheduler whatever invoked this--which is why it needs the context, by the way--need to be put in the blocked queue until the given awaiter comes in. Then you await on it to suspend execution. The scheduler will be waiting for the continuation created by the .NET runtime and will file it with the blocked task. You finally return the future. Cloaca will know to open it up and extract the value out to put it on the data stack when things resume.

It's really simple! (no it's not). =D

@bamyazi
Copy link
Author

bamyazi commented May 28, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rockobonaparte @bamyazi