Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add static typing to Python & add type checking CI #546

Closed
wants to merge 27 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ae399b8
remove useless main.py imports
AAGaming00 Aug 5, 2023
ecc5f5c
begin adding static types to backend code
AAGaming00 Aug 27, 2023
75fbc75
type hints on main,plugin,updater,utilites.localsocket
marios8543 Sep 17, 2023
f6401f4
move to module imports
AAGaming00 Sep 25, 2023
5838ddc
add pyright ci
AAGaming00 Sep 25, 2023
300885f
move type checking to other workflow, fix TS errors, add TSC checking
AAGaming00 Sep 25, 2023
e22cc62
make ci title consistent
AAGaming00 Sep 25, 2023
b81c41f
remove quotes on some types
AAGaming00 Sep 25, 2023
2d68809
run lint and typecheck on PRs
AAGaming00 Sep 25, 2023
3960d28
with, not env
AAGaming00 Sep 25, 2023
0c2079f
Moved backend entirely into the backend folder
WerWolv Sep 26, 2023
a0d50ba
Moved main.py
WerWolv Sep 26, 2023
75ae7df
Moved locales folder and requirements.txt
WerWolv Sep 26, 2023
b704365
speed up stupid make
AAGaming00 Sep 30, 2023
00e10be
fix ci (hopefully, because act wont work)
AAGaming00 Sep 30, 2023
4b89fc1
fix broken import
AAGaming00 Sep 30, 2023
e8cbeb1
oops
AAGaming00 Sep 30, 2023
ade7cb7
fix paths
AAGaming00 Sep 30, 2023
4a17474
fix decky_plugin path in pyinstaller
marios8543 Oct 11, 2023
944e0e6
Fix decky_plugin on windows CI
marios8543 Oct 17, 2023
2cdb491
fix logical error when no store was set
PartyWumpus Oct 17, 2023
a231225
fix typo
PartyWumpus Oct 17, 2023
ffbc79d
fix bad type on store.tsx
marios8543 Oct 17, 2023
d454a40
Merge branch 'main' into aa/type-cleanup-py
marios8543 Oct 17, 2023
0736a6f
fix bad type on store.tsx
marios8543 Oct 17, 2023
e0592f0
fix uninstall bug
marios8543 Oct 17, 2023
064c897
Merge branch 'main' into aa/type-cleanup-py
AAGaming00 Oct 22, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/build-win.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ jobs:
python-version: "3.11.4"

- name: Install Python dependencies ⬇️
working-directory: ./backend
run: |
python -m pip install --upgrade pip
pip install pyinstaller==5.13.0
Expand All @@ -43,10 +44,10 @@ jobs:
run: pnpm run build

- name: Build Python Backend 🛠️
run: pyinstaller --noconfirm --onefile --name "PluginLoader" --add-data "./backend/static;/static" --add-data "./backend/locales;/locales" --add-data "./backend/legacy;/legacy" --add-data "./plugin;/plugin" --hidden-import=sqlite3 ./backend/main.py
run: pyinstaller --noconfirm --onefile --name "PluginLoader" --add-data "./backend/static;/static" --add-data "./backend/locales;/locales" --add-data "./backend/src/legacy;/src/legacy" --add-data "./plugin/*;/" --hidden-import=sqlite3 ./backend/main.py

- name: Build Python Backend (noconsole) 🛠️
run: pyinstaller --noconfirm --noconsole --onefile --name "PluginLoader_noconsole" --add-data "./backend/static;/static" --add-data "./backend/locales;/locales" --add-data "./backend/legacy;/legacy" --add-data "./plugin;/plugin" --hidden-import=sqlite3 ./backend/main.py
run: pyinstaller --noconfirm --noconsole --onefile --name "PluginLoader_noconsole" --add-data "./backend/static;/static" --add-data "./backend/locales;/locales" --add-data "./backend/src/legacy;/src/legacy" --add-data "./plugin/*;/" --hidden-import=sqlite3 ./backend/main.py

- name: Upload package artifact ⬆️
uses: actions/upload-artifact@v3
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,18 +62,19 @@ jobs:
-DSQLITE_ENABLE_UNLOCK_NOTIFY -DSQLITE_ENABLE_DBSTAT_VTAB=1 -DSQLITE_ENABLE_FTS3_TOKENIZER=1 \
-DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_SECURE_DELETE -DSQLITE_ENABLE_STMTVTAB -DSQLITE_MAX_VARIABLE_NUMBER=250000 \
-DSQLITE_MAX_EXPR_DEPTH=10000 -DSQLITE_ENABLE_MATH_FUNCTIONS" &&
make &&
make -j$(nproc) &&
sudo make install &&
sudo cp /usr/lib/libsqlite3.so /usr/lib/x86_64-linux-gnu/ &&
sudo cp /usr/lib/libsqlite3.so.0 /usr/lib/x86_64-linux-gnu/ &&
sudo cp /usr/lib/libsqlite3.so.0.8.6 /usr/lib/x86_64-linux-gnu/ &&
rm -r /tmp/sqlite-autoconf-3420000

- name: Install Python dependencies ⬇️
working-directory: ./backend
run: |
python -m pip install --upgrade pip
pip install pyinstaller==5.13.0
[ -f requirements.txt ] && pip install -r requirements.txt
pip install -r requirements.txt

- name: Install JS dependencies ⬇️
working-directory: ./frontend
Expand All @@ -86,7 +87,7 @@ jobs:
run: pnpm run build

- name: Build Python Backend 🛠️
run: pyinstaller --noconfirm --onefile --name "PluginLoader" --add-data ./backend/static:/static --add-data ./backend/locales:/locales --add-data ./backend/legacy:/legacy --add-data ./plugin:/plugin --hidden-import=sqlite3 ./backend/*.py
run: pyinstaller --noconfirm --onefile --name "PluginLoader" --add-data ./backend/static:/static --add-data ./backend/locales:/locales --add-data ./backend/src/legacy:/src/legacy --add-data ./plugin/*:/ --hidden-import=sqlite3 ./backend/main.py

- name: Upload package artifact ⬆️
if: ${{ !env.ACT }}
Expand Down Expand Up @@ -127,7 +128,7 @@ jobs:
- name: Get latest release
uses: rez0n/actions-github-release@main
id: latest_release
env:
with:
token: ${{ secrets.GITHUB_TOKEN }}
repository: "SteamDeckHomebrew/decky-loader"
type: "nodraft"
Expand Down Expand Up @@ -206,7 +207,7 @@ jobs:
- name: Get latest release
uses: rez0n/actions-github-release@main
id: latest_release
env:
with:
token: ${{ secrets.GITHUB_TOKEN }}
repository: "SteamDeckHomebrew/decky-loader"
type: "nodraft"
Expand Down
14 changes: 10 additions & 4 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: Lint

on:
push:
pull_request:

jobs:
lint:
Expand All @@ -10,8 +11,13 @@ jobs:

steps:
- uses: actions/checkout@v3 # Check out the repository first.
- name: Run prettier (JavaScript & TypeScript)

- name: Install TypeScript dependencies
working-directory: frontend
run: |
pushd frontend
npm install
npm run lint
npm i -g pnpm
pnpm i --frozen-lockfile

- name: Run prettier (TypeScript)
working-directory: frontend
run: pnpm run lint
36 changes: 36 additions & 0 deletions .github/workflows/typecheck.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Type Check

on:
push:
pull_request:

jobs:
typecheck:
name: Run type checkers
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v2 # Check out the repository first.

- name: Install Python dependencies
working-directory: backend
run: |
python -m pip install --upgrade pip
[ -f requirements.txt ] && pip install -r requirements.txt

- name: Install TypeScript dependencies
working-directory: frontend
run: |
npm i -g pnpm
pnpm i --frozen-lockfile

- name: Run pyright (Python)
uses: jakebailey/pyright-action@v1
with:
python-version: "3.10.6"
no-comments: true
working-directory: backend

- name: Run tsc (TypeScript)
working-directory: frontend
run: $(pnpm bin)/tsc --noEmit
195 changes: 3 additions & 192 deletions backend/main.py
Original file line number Diff line number Diff line change
@@ -1,193 +1,4 @@
# Change PyInstaller files permissions
import sys
from localplatform import (chmod, chown, service_stop, service_start,
ON_WINDOWS, get_log_level, get_live_reload,
get_server_port, get_server_host, get_chown_plugin_path,
get_unprivileged_user, get_unprivileged_path,
get_privileged_path)
if hasattr(sys, '_MEIPASS'):
chmod(sys._MEIPASS, 755)
# Full imports
from asyncio import new_event_loop, set_event_loop, sleep
from json import dumps, loads
from logging import DEBUG, INFO, basicConfig, getLogger
from os import getenv, path
from traceback import format_exc
import multiprocessing

import aiohttp_cors
# Partial imports
from aiohttp import client_exceptions, WSMsgType
from aiohttp.web import Application, Response, get, run_app, static
from aiohttp_jinja2 import setup as jinja_setup

# local modules
from browser import PluginBrowser
from helpers import (REMOTE_DEBUGGER_UNIT, csrf_middleware, get_csrf_token,
mkdir_as_user, get_system_pythonpaths, get_effective_user_id)

from injector import get_gamepadui_tab, Tab, get_tabs, close_old_tabs
from loader import Loader
from settings import SettingsManager
from updater import Updater
from utilities import Utilities
from customtypes import UserType


basicConfig(
level=get_log_level(),
format="[%(module)s][%(levelname)s]: %(message)s"
)

logger = getLogger("Main")
plugin_path = path.join(get_privileged_path(), "plugins")

def chown_plugin_dir():
if not path.exists(plugin_path): # For safety, create the folder before attempting to do anything with it
mkdir_as_user(plugin_path)

if not chown(plugin_path, UserType.HOST_USER) or not chmod(plugin_path, 555):
logger.error(f"chown/chmod exited with a non-zero exit code")

if get_chown_plugin_path() == True:
chown_plugin_dir()

class PluginManager:
def __init__(self, loop) -> None:
self.loop = loop
self.web_app = Application()
self.web_app.middlewares.append(csrf_middleware)
self.cors = aiohttp_cors.setup(self.web_app, defaults={
"https://steamloopback.host": aiohttp_cors.ResourceOptions(
expose_headers="*",
allow_headers="*",
allow_credentials=True
)
})
self.plugin_loader = Loader(self.web_app, plugin_path, self.loop, get_live_reload())
self.settings = SettingsManager("loader", path.join(get_privileged_path(), "settings"))
self.plugin_browser = PluginBrowser(plugin_path, self.plugin_loader.plugins, self.plugin_loader, self.settings)
self.utilities = Utilities(self)
self.updater = Updater(self)

jinja_setup(self.web_app)

async def startup(_):
if self.settings.getSetting("cef_forward", False):
self.loop.create_task(service_start(REMOTE_DEBUGGER_UNIT))
else:
self.loop.create_task(service_stop(REMOTE_DEBUGGER_UNIT))
self.loop.create_task(self.loader_reinjector())
self.loop.create_task(self.load_plugins())

self.web_app.on_startup.append(startup)

self.loop.set_exception_handler(self.exception_handler)
self.web_app.add_routes([get("/auth/token", self.get_auth_token)])

for route in list(self.web_app.router.routes()):
self.cors.add(route)
self.web_app.add_routes([static("/static", path.join(path.dirname(__file__), 'static'))])
self.web_app.add_routes([static("/legacy", path.join(path.dirname(__file__), 'legacy'))])

def exception_handler(self, loop, context):
if context["message"] == "Unclosed connection":
return
loop.default_exception_handler(context)

async def get_auth_token(self, request):
return Response(text=get_csrf_token())

async def load_plugins(self):
# await self.wait_for_server()
logger.debug("Loading plugins")
self.plugin_loader.import_plugins()
# await inject_to_tab("SP", "window.syncDeckyPlugins();")
if self.settings.getSetting("pluginOrder", None) == None:
self.settings.setSetting("pluginOrder", list(self.plugin_loader.plugins.keys()))
logger.debug("Did not find pluginOrder setting, set it to default")

async def loader_reinjector(self):
while True:
tab = None
nf = False
dc = False
while not tab:
try:
tab = await get_gamepadui_tab()
except (client_exceptions.ClientConnectorError, client_exceptions.ServerDisconnectedError):
if not dc:
logger.debug("Couldn't connect to debugger, waiting...")
dc = True
pass
except ValueError:
if not nf:
logger.debug("Couldn't find GamepadUI tab, waiting...")
nf = True
pass
if not tab:
await sleep(5)
await tab.open_websocket()
await tab.enable()
await self.inject_javascript(tab, True)
try:
async for msg in tab.listen_for_message():
# this gets spammed a lot
if msg.get("method", None) != "Page.navigatedWithinDocument":
logger.debug("Page event: " + str(msg.get("method", None)))
if msg.get("method", None) == "Page.domContentEventFired":
if not await tab.has_global_var("deckyHasLoaded", False):
await self.inject_javascript(tab)
if msg.get("method", None) == "Inspector.detached":
logger.info("CEF has requested that we detach.")
await tab.close_websocket()
break
# If this is a forceful disconnect the loop will just stop without any failure message. In this case, injector.py will handle this for us so we don't need to close the socket.
# This is because of https://github.com/aio-libs/aiohttp/blob/3ee7091b40a1bc58a8d7846e7878a77640e96996/aiohttp/client_ws.py#L321
logger.info("CEF has disconnected...")
# At this point the loop starts again and we connect to the freshly started Steam client once it is ready.
except Exception as e:
logger.error("Exception while reading page events " + format_exc())
await tab.close_websocket()
pass
# while True:
# await sleep(5)
# if not await tab.has_global_var("deckyHasLoaded", False):
# logger.info("Plugin loader isn't present in Steam anymore, reinjecting...")
# await self.inject_javascript(tab)

async def inject_javascript(self, tab: Tab, first=False, request=None):
logger.info("Loading Decky frontend!")
try:
if first:
if await tab.has_global_var("deckyHasLoaded", False):
await close_old_tabs()
await tab.evaluate_js("try{if (window.deckyHasLoaded){setTimeout(() => location.reload(), 100)}else{window.deckyHasLoaded = true;(async()=>{try{while(!window.SP_REACT){await new Promise(r => setTimeout(r, 10))};await import('http://localhost:1337/frontend/index.js')}catch(e){console.error(e)};})();}}catch(e){console.error(e)}", False, False, False)
except:
logger.info("Failed to inject JavaScript into tab\n" + format_exc())
pass

def run(self):
return run_app(self.web_app, host=get_server_host(), port=get_server_port(), loop=self.loop, access_log=None)

# This file is needed to make the relative imports in src/ work properly.
if __name__ == "__main__":
if ON_WINDOWS:
# Fix windows/flask not recognising that .js means 'application/javascript'
import mimetypes
mimetypes.add_type('application/javascript', '.js')

# Required for multiprocessing support in frozen files
multiprocessing.freeze_support()
else:
if get_effective_user_id() != 0:
logger.warning(f"decky is running as an unprivileged user, this is not officially supported and may cause issues")

# Append the loader's plugin path to the recognized python paths
sys.path.append(path.join(path.dirname(__file__), "plugin"))

# Append the system and user python paths
sys.path.extend(get_system_pythonpaths())

loop = new_event_loop()
set_event_loop(loop)
PluginManager(loop).run()
from src.main import main
main()
3 changes: 3 additions & 0 deletions backend/pyrightconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"strict": ["*"]
}
File renamed without changes.
Loading
Loading