v1.5.0

Hello, everyone! v1.5.0 marks the biggest release thus far (and also the 10th release!). This version makes a lot of changes under the hood, but also brings in some pretty substantial new features.

IMPORTANT: Manual Update Action Recommended for Server Owners

Note This isn't strictly necessary, but is a good idea to run regardless.

When updating to v1.5.0, you will have to run Refresh.GameServer (the main server executable) with --import_assets. This is due to new changes to handling game assets. This is a one-time operation, and should only take a few seconds.

This will cause Refresh to read all assets, their types, and import them into the database for API access.

Warning Do not run this command multiple times, as this will cause existing metadata to be deleted.
Refresh from this point forward will track the original author of uploaded assets. This will be useful in the future for moderation.
Running this operation after running it once will make it harder to track down users who have uploaded malicious assets.

We also wrote a guide on setting up a production server. It's for Bunkum, our custom server software, but it applies to Refresh aswell. https://littlebigrefresh.github.io/Docs/setup-bunkum-server

New goodies

To start, we implemented dive-in and room tracking. This is all accessible through the API, meaning you can find a user by their room, track what level they're playing, etc. etc. This works even for players that aren't on the instance, assuming they are a part of a player's lobby.

We've also implemented some score types for score submission - meaning players are now able to properly submit scores for two-player mode, three-player mode, and four-player mode instead of just one-player mode. Why wasn't this part of the original implementation? I'm not quite sure, to be honest.

Lastly, the last major thing in-game we've added is photo uploading support. You can now upload photos, the server will record the photo and its tagged user data, and you can view them on the website and in-game. Once again, tagged users who don't exist on the Refresh instance you're playing will still show up, so there's no data lost when you upload a photo to Refresh.

We've also implemented support for notifications, meaning that you'll be able to hear about new things happening on the server that you'd normally never know about - like when someone you've hearted hearts you back, or when a level fails to upload. You can view them on the website, but they also show on the game's announcement screen on startup.

Naturally, all of these new features are available on the website and the API, so have fun!

I love my website

The website has been entirely re-designed, going from a really ugly blue color to a nice Refresh-themed purple. I spent a few days working around in Figma, and I think what came out looks fantastic.

Lighthouse API Support

We implemented a subset of Project Lighthouse's API (v1). This has been tested with LighthouseRichPresence and PLRPC, LBP Union's applications for bridging Discord to LBP custom servers. Essentially what this means is when running these applications, they'll try to talk to Refresh as if it were a Project Lighthouse instance, and Refresh will try to respond as if it were.

There are caveats of course, Lighthouse and Refresh being based on different architectures. There may be potential for conflict on user ids, as they are based on timestamps. Refresh's API (v2) will not be affected by this. Additionally, the portion of the API implemented is only a subset. Some applications may not work straight away, and may require some patches to Refresh in order to get them working. The two main ones are already fairly well supported, so it shouldn't be too big of a deal. Let us know if you have any issues, though!

Here's a screenshot of PLRPC running on a Refresh instance:

A peek under the hood

We did some work to implement some proper unit testing across Bunkum and Refresh. These have been a great help, I even found a bug or two while working on it.

We also implemented AutoDiscover API v2, letting Refresher (our patching software) know that we're using a custom digest key. Speaking of which, Refresher recently gained the ability to patch RPCS3 titles in just a few clicks. It's pretty sweet. Here's a video by @turecross321 showcasing patching Sound Shapes to his server built on Bunkum:

2023-06-08_16-55-33.mp4.processed.mp4

We wrote a guide on patching RPCS3 titles, which you can find here: https://littlebigrefresh.github.io/Docs/patching/rpcs3

Finally, the last major thing is the ability to convert .TEX to .PNG. It should be pretty efficient (I was able to convert a couple thousand assets in a few seconds with 12 threads) but your mileage may vary, depending on total memory and core count. I tried to cut down on the amount of allocated memory, but due to the nature of the operation it's not possible to remove all allocations. This shouldn't be a problem if you run a small instance with a few dozen levels.

Commits

• b00a0da: Stub legacy API endpoints (jvyden)
• 6a54557: Allow Autodiscover to work with the website changes (jvyden)
• feaac93: Pod (jvyden)
• c5460a4: Parse match into valid JSON (jvyden)
• 20e62da: Cleanup (jvyden)
• 0c6c930: Implement match methods (jvyden)
• fe44dc7: Bump Bunkum to 3.0.5 (jvyden)
• 9b04d0e: Adjust max ratelimit request amount to 200 (jvyden)
• d898ad0: Merge branch 'main' into matching (jvyden)
• 736aa0e: Add testing framework (jvyden)
• 0b43813: Update Bunkum to 3.0.5 (jvyden)
• e5262b5: Rework server initialization code for easier testing (jvyden)
• 83414fe: Update Bunkum to 3.0.6 (jvyden)
• 6146840: Update Bunkum to 3.1.0 (jvyden)
• 750837f: Full test framework (jvyden)
• 611ee70: Put "inmemory" database files into temporary directory (jvyden)
• 6403ca0: CI Debug: Only use in-memory if we are not in CI (jvyden)
• 6cef0bf: CI Debug: Use GameDatabaseProvider for test (jvyden)
• af64e13: Revert CI debug steps (jvyden)
• b730312: Random number for inmemory tests (jvyden)
• e361dd4: Set random database ID only once (jvyden)
• 2a9331b: Update Realm to 11.0.0 (jvyden)
• 14f10a6: ill cry if this works (jvyden)
• e5d0865: Try getting token back from database context, use LBP2 on PS3 when generating game tokens (jvyden)
• cda2ad3: end my suffering (jvyden)
• 3d4da03: Revert "end my suffering" (jvyden)
• 41b77ff: Update Bunkum.RealmDatabase to 3.1.1 (jvyden)
• 84fa241: API authentication test (jvyden)
• d44fb23: Add score submission & result tests (jvyden)
• 9cedea7: Update Bunkum.RealmDatabase to 3.1.2 (jvyden)
• a57162b: More segmenting tests (jvyden)
• 6a5e636: Merge branch 'main' into matching (jvyden)
• 23acb18: Deserialize match object, pass in service and logger to match method (jvyden)
• ff1307d: API, track and store rooms (jvyden)
• b16f39e: Handle Slot and Slots (jvyden)
• 4d31745: Handle host's "mood" (jvyden)
• ed25758: Switch to using Realm's source generators (#43) (Jayden) #43
• 453dd4e: Merge branch 'main' into matching (jvyden)
• 283e74a: Move body deserialization to endpoint (jvyden)
• bb3f558: Refactor GameServerTest (jvyden)
• 70202ae: Basic matching logic (jvyden)
• 98dc94c: Serialize a proper response to dive-in (jvyden)
• eb35632: Add LBP2 US digital to title id list (jvyden)
• a5d8c7b: Update Bunkum to 3.1.3 (jvyden)
• c36eb16: Make PresenceEndpoints use values from MatchService (jvyden)
• edb794e: Allow any user to be stored in rooms (jvyden)
• 3cbd730: Implement storage and retrieval of friends list (jvyden)
• e2a60b4: Update Bunkum to 3.1.3 (jvyden)
• 1c42ef0: Fix NAT type parsing (jvyden)
• a94a5c8: Fix unit test compile errors (jvyden)
• 3523d79: Remove dependency on database for getting rooms (jvyden)
• 3345458: Implement UpdatePlayersInRoom (jvyden)
• b567dd6: Remove the strangest unused import I've ever seen (jvyden)
• a206ad7: Support score submission by type (jvyden)
• c99329b: Merge branch 'main' into legacy-api (jvyden)
• 727f634: Use match service for legacy room endpoint (jvyden)
• c027c58: Support retrieval of levels through legacy api (jvyden)
• 9e05a68: Warmup the database on startup (jvyden)
• dc1987d: Autodiscover API v2 (jvyden)
• c55ff21: Fix trailing slash in URL (jvyden)
• 9cbae06: Don't send HostId in JSON (jvyden)
• 5f66679: API endpoints for room by user uuid, room uuid, and all rooms (jvyden)
• 16d0077: Remove resources from GameLevel (jvyden)
• 27235c0: Statically import HttpStatusCode (jvyden)
• e5080b7: Cleanup usages (jvyden)
• 53c82a7: Add endpoint for tracking server statistics (jvyden)
• a53b496: Show current ingame players count (jvyden)
• c7220a8: Bump NotEnoughLogs from 1.0.3 to 1.0.5 (#47) (dependabot[bot]) #47
• 119a17b: Bump Bunkum from 3.1.3 to 3.1.4 (#46) (dependabot[bot]) #46
• 0407c77: Fix internal server error when retrieving unimplemented slot category (jvyden)
• 0dcd283: Close the security vulnerability I just opened (jvyden)
• eb4d182: Am I OKAY? (jvyden)
• 8313799: Add .svg and .ico to mimetype mapping list (jvyden)
• 50564ad: Upgrade Bunkum to 3.2.0 (jvyden)
• cced95a: Support Bunkum multithreading (jvyden)
• 1b3eb03: Prefix unstored objects with Serialized instead of Game (jvyden)
• a45fe23: Bump Microsoft.CodeAnalysis.CSharp from 4.5.0 to 4.6.0 (#48) (dependabot[bot]) #48
• 3403101: Bump Microsoft.NET.Test.Sdk from 17.5.0 to 17.6.0 (#49) (dependabot[bot]) #49
• c4a47ab: Bump coverlet.collector from 3.2.0 to 6.0.0 (#50) (dependabot[bot]) #50
• 856657f: Allow downloading assets through API (jvyden)
• 6fa6f71: Begin tracking simple metadata for new uploaded assets (jvyden)
• dd757f7: Enable health checks (jvyden)
• 9e505b3: Merge branch 'main' into assets (jvyden)
• 779fbd7: Basic importer (jvyden)
• 3b9d5cc: Fix database not initializing in asset importer (jvyden)
• 148ec30: Implement determining file type (jvyden)
• b74132d: Refactor asset importer to work in server (jvyden)
• 2880a56: Only write after verification (jvyden)
• 627327c: Bump NUnit3TestAdapter from 4.4.2 to 4.5.0 (#52) (dependabot[bot]) #52
• 98e6f71: Bump Microsoft.NET.Test.Sdk from 17.6.0 to 17.6.1 (#53) (dependabot[bot]) #53
• 7223496: Make asset info primary endpoint (jvyden)
• 1b27f69: Merge remote-tracking branch 'upstream/main' (jvyden)
• def46ec: Fix health check endpoints being blocked by website middleware (jvyden)
• edc7a64: Reject asset if it exceeds maximum safety level (jvyden)
• 5bc4568: Add AGPLv3 license to /eula (jvyden)
• 0f1052d: Basic implementation of notifications (jvyden)
• 6eb1a8a: Call to action under notifications in /announce (jvyden)
• 069aa4c: Rename LicenseEndpoints to WelcomeEndpoints (jvyden)
• fc457e1: Add API endpoint for notifications (jvyden)
• 2c0d99d: Ability to get notification by id, delete by id, and delete all (jvyden)
• 14092ce: Add notifications for login failures, publish failures, and hearts (jvyden)
• 3911889: Fix JoinDates never being set (jvyden)
• e01d523: Add checks for a few modded assets (jvyden)
• 66f35d1: Replace autodiscover implementation with Bunkum.AutoDiscover (jvyden)
• dfce061: Stub LastLogin (jvyden)
• dfd0e9c: Create an abstract Importer (jvyden)
• c4037c4: Update Bunkum to 3.3.4 (jvyden)
• d3ed18b: Use new style of retrieving keys from IDataStore (jvyden)
• 6e50f1a: Basic import logic (jvyden)
• 75bbe75: Implement texture to png conversion logic (jvyden)
• e290cb5: Ability to retrieve PNGs from API (jvyden)
• fc4becc: Restore AssetImporter for startup imports (jvyden)
• 286fa11: Pass level's IconHash into LegacyGameLevel (jvyden)
• 2cb4040: Deserialize uploaded photos from game (jvyden)
• d4b57ac: Implement photo uploads (jvyden)
• e2149dc: Fix incorrect timestamps on photos (jvyden)
• 7e22014: Allow reading level from photo (jvyden)
• 1cea798: API Endpoints for photos (jvyden)
• 202dd78: Endpoints for retrieving photos through profile (jvyden)
• b64d72e: Add photo count to profile serialization (jvyden)
• 6ba2220: Default to display name if user in photo is unavailable (jvyden)
• da8209f: API Endpoint for getting recent photos (jvyden)
• 0604985: API endpoint for getting photo by id (jvyden)
• 65ca8d2: Fix serialization of play_level events (jvyden)
• 00748a1: Bully the source generator into working with GameSubmittedScores (jvyden)
• 4c1d3e2: Serialize submitted scores in recent activity (jvyden)
• 82cb537: Log header when asset is unrecognized (jvyden)
• 003a36c: Add MoveRecording and VoiceRecording assets (jvyden)
• 24d4471: Optimize image import memory usage (jvyden)
• ad31080: Multithread image importer (jvyden)
• a8f80c4: Log how long it took for all assets (jvyden)
• 200c8e6: Try to use streams directly instead of through a MemoryBuffer (jvyden)
• 7c6ca41: Oops, remove ReadKey from startup (jvyden)
• a263fab: Support connections via LITTLEBIGPLANETPS3_XML (jvyden)
• 2b89fe0: Fix inability to change profile picture (jvyden)
• 1c43ffb: Tell client to cache resource responses (jvyden)
• d7a703c: Bump Microsoft.NET.Test.Sdk from 17.6.1 to 17.6.2 (#56) (dependabot[bot]) #56
• 10d4752: Do Dependabot's job of updating Microsoft.CodeAnalysis.CSharp to 4.6.0 (jvyden)