[rcore] Initial position reported by GetMouseDelta() incorrect

[alexnivanov]

• I tested it on latest raylib version from master branch
• I checked there is no similar issue already reported
• I checked the documentation on the wiki
• My code has no errors or misuse of raylib

Issue description

I'm having an issue with camera update on MacOS: when the app starts, and I first move the mouse, GetMouseDelta() is very far away. After doing some experiments I figured out that it's assuming that the mouse is initially in the center of the screen. When I start the app having previously moved the mouse to the center of the screen, the behaviour is OK.

I've build from source core_3d_camera_free, and the issue is reproduced as described above.

Environment

MacBook M1 Pro, MacOS Sequoia 15.1.1, OpenGL Version: 4.1 Metal - 89.3.

Issue Screenshot

Just touched my mouse and got this:

Code Example

examples/core_3d_camera_free reproduces the issue.

[CodingMadness]

did you close the issue already, lol?

[alexnivanov]

@asdqwe have you tested the fix? I've tried, and it didn't resolve anything. Also, the issue seems to be unrelated to DisableCursor method: mouse delta should be appropriate when cursor is enabled as well!

[alexnivanov]

Comment line rcore_desktop_glfw.c#L1021 to see the diff.

Yeah well I did exactly that, and there seems to be no difference on MacOS :(. Also as mentioned above, it seems that the fix should be unrelated to DisableCursor method.

[alexnivanov]

@asdqwe I've checked GLFW, it is a well known behaviour, see this example: https://learnopengl.com/Getting-started/Camera

The solution is to ignore first mouse update:

void mouse_callback(GLFWwindow* window, double xpos, double ypos)
{
    if (firstMouse)
    {
        lastX = xpos;
        lastY = ypos;
        firstMouse = false;
    }
   ...
}

However, in rcore_desktop_glfw.c, there is no such logic in MouseCursorPosCallback, hence the result. I'm not sure where to put this temporary firstMouse variable, should be reviewed by somebody familiar with the library architecture.

[alexnivanov]

When cursor is enabled, the mouse delta shouldn't matter

I don’t understand why do you claim that. When you have a cursor and the window appears, mouse delta should be continuous as well.

[ve-nt]

I can confirm I see the same issue on Linux using the GLFW backend.

@asdqwe I've checked GLFW, it is a well known behaviour, see this example: https://learnopengl.com/Getting-started/Camera

The solution is to ignore first mouse update:

void mouse_callback(GLFWwindow* window, double xpos, double ypos)
{
    if (firstMouse)
    {
        lastX = xpos;
        lastY = ypos;
        firstMouse = false;
    }
   ...
}

This appears to be correct. I have implemented this solution in my PR (#4663) and can confirm it fixes the camera jump on my machine.

[alexnivanov]

I have implemented this solution in my PR (#4663) and can confirm it fixes the camera jump on my machine.

I can confirm that this PR fixes the issue for me as well as with enabled and disabled cursor!

[raysan5]

@asdqwe thanks for all the time put on reviewing this issue, definitely the target platform does not deserve it.

@alexnivanov @ve-nt Am I understanding correctly that the solution to the issue is a macOS-platforn-specific-hack for first mouse input read, requiring a global variable to manage it? No way.

[ve-nt]

I don't believe the issue is limited to macOS, as I'm seeing the same issue on Linux. I think the issue is in the GLFW backend in general. I have experienced this same issue when working with GLFW+OpenGL directly, in fact this issue is brought up and fixed as part of the LearnOpenGL book, using the same solution I've implemented in my PR.

See Mouse Input section at: https://learnopengl.com/Getting-started/Camera

Depends on your definition of "global variable" I guess. In my PR its a static local variable inside the callback.

[raysan5]

Depends on your definition of "global variable" I guess. In my PR its a static local variable inside the callback.

Afaik, the lifetime is since first run to the end of the program, so effectively it's a global variable that will require some bytes forever (while the program running).

Still, I think this issue should be addressed at GLFW side, not hacked at raylib side. Thanks for reporting it's not macOS specific, I'm changing the title.

[raysan5]

As per the details of the issue and the further clarification, it seems this issue is related to GLFW, so I'm afraid it should be addressed by GLFW, not hacked at raylib side.

[ve-nt]

I'm not sure that the problem is in the GLFW side. As far as I can tell, its because the CORE.Input.Mouse.previousPosition is zero on the first mouse movement, since MouseCursorPosCallback is only called when the mouse is moved. This means that the first GetMouseDelta after the mouse is moved for the first time will have an accurate currentPosition but a zeroed previousPosition, leading to a huge delta and therefore a camera jolt.

[raysan5]

@ve-nt I see, can it be managed at user side in that case? Actually GetMouseDelta() was added quite recently, before it was up to the user to track mouse delta...

[ve-nt]

I have been trying to fix this on the user side but not had much luck so far. I'm open to any ideas and will keep you posted if I find a fix on the user side.

[raysan5]

I'm reopening for a closer look, in case of a better solution than proposed hack...

[alexnivanov]

@raysan5 I see the root cause in SetMousePosition being called with coordinates of the center of the screen assuming that the cursor will be there by both EnableCursor and DisableCursor which is obviously not true: it doesn't affect actual cursor position hence we get this initial invalid delta.

[alexnivanov]

@alexnivanov @ve-nt @raysan5 Please test #4665. Hopefully it fixes this so we can be done with it.

@asdqwe this is better, but if you call either DisableCursor() or EnableCursor() it breaks anyway :(

[ve-nt]

@alexnivanov @ve-nt @raysan5 Please test #4665. Hopefully it fixes this so we can be done with it.

I'm afraid I've tested it with examples/core/core_3d_camera_free and the camera still jolts on the first mouse movement.

It only jolts when the cursor is positioned inside the window from the start. There's no jolt if the cursor was outside the window when the window opened. Is this the same for you @alexnivanov?

[alexnivanov]

@ve-nt it does jolt for me not depending on initial cursor position being inside/outside of the window. Only removing DisableCursor() call removes the jolt, so it's a good starting point anyway :)

[ve-nt]

So after a bit more digging, this bug is weirder than I thought at first.

It appears that SetMousePosition is not functioning properly...

When my cursor is outside of the created window, it seems to work fine. But when the cursor is over the created window, GLFW does not actually move the cursor.

This means that when MouseCursorPosCallback is called for the first time, GLFW still has the cursor in its original position (i.e. where it was before the window was opened). Meaning the current/previousPosition is out of sync with where the cursor actually is.

This is what is leading to the spike in the mouse delta, and hence why I don't get a camera jolt unless my cursor is over the window when its opened.

Its worth me reiterating that GLFW seems to be setting the cursor position fine if my cursor is not over the window. The coordinates reported from the first callback match up with where DisableCursor is setting the cursor to.

Perhaps this is a GLFW issue after all?

[raysan5]

@asdqwe thanks for the further review. Please, take it easy, no hurries at all.

[ve-nt]

Please see my fix here: ve-nt@28975df

This fixes the issue on my machine, although I don't quite understand why yet, hence why I haven't submitted it as a PR. Though it may help if others want to look.

The thing I don't understand is why GLFW isn't setting its cursor position properly. I believe this is key to why our mouse deltas are wrong. Our SetMousePosition function assumes that GLFW will take our coordinates and use them, and when it doesn't we get out of sync with it.

[ve-nt]

For my 1600x900 screen it's plotting a -400 to 1200 horizontal mouse coord space

This makes sense to me as glfwGetCursorPos returns the coordinates relative to the upper-left corner of the window. So if your window is placed at x400, then the very left side of your screen will be -400, and the very right will be 1200.

[ve-nt]

@ve-nt Ok, ve-nt@28975df didn't work for DisableCursor() + Outside on the LEFT side here:

WARNING: time:0.08   pos: 158.00 x 388.00   delta: 0.00 x 0.00
WARNING: time:0.10   pos: 400.00 x 225.00   delta: 242.00 x -163.00
WARNING: time:0.10   pos: 400.00 x 225.00   delta: 0.00 x 0.00
WARNING: time:0.12   pos: 400.00 x 225.00   delta: 0.00 x 0.00
WARNING: time:0.13   pos: 400.00 x 225.00   delta: 0.00 x 0.00

Strange... It works on the outside left for me.

Could you also print out the x and y arguments that MouseCursorPosCallback is receiving? That's what I've been trying to check. Just so we can compare.

Also, from the GLFW header:

 *  __Do not use this function__ to implement things like camera controls.  GLFW
 *  already provides the `GLFW_CURSOR_DISABLED` cursor mode that hides the
 *  cursor, transparently re-centers it and provides unconstrained cursor
 *  motion.  See @ref glfwSetInputMode for more information.

transparently re-centers it

I get the feeling this might have something to do with it but I'm not sure yet.

[ve-nt]

I'm seeing the same weird results with MouseCursorPosCallback. Although the pos your printing seems off, it should be negative if its to the outside and to the left of your window. Perhaps check it against the straight output of glfwGetCursorPos.

I'm tracing the reported values from glfwGetCursorPos during InitPlatform. I've also added a reread of them at the end of DisableCursor.

Here's what I'm seeing with my cursor OUTSIDE and to the LEFT:

WARNING: InitPlatform: glfwGetCursorPos: -156.000000 282.000000
WARNING: DisableCursor: glfwGetCursorPos: -156.000000 282.000000
WARNING: MouseButtonCallback: 409.000000 225.000000
WARNING: MouseButtonCallback: 408.000000 226.000000

The inputs to MouseButtonCallback seem completely different to glfwGetCursorPos!

Meanwhile, here's the output with my cursor INSIDE:

WARNING: InitPlatform: glfwGetCursorPos: 180.000000 214.000000
WARNING: DisableCursor: glfwGetCursorPos: 180.000000 214.000000
WARNING: MouseButtonCallback: 180.000000 215.000000
WARNING: MouseButtonCallback: 180.000000 216.000000
WARNING: MouseButtonCallback: 180.000000 217.000000

They match up great! Hence no jolt when my cursor is inside, but huge jolt when outside...

I should mention that this is the same whether I SetMousePosition before DISABLE_CURSOR or not. Seems to be the same either way. Perhaps we enter a different coord space once in DISABLE_CURSOR? Although doesn't feel like it. Will have to check the docs tomorrow as its getting late here.

[ve-nt]

Another hint from what I've just noticed:

My MouseButtonCallback x and y always start at 409 and 225 if my cursor started outside the window (no matter left, right, up, or down).

[ve-nt]

From what I'm seeing, it appears that https://github.com/raysan5/raylib/blob/fc29bc27fd1c6904143fc24fbcf5ca629ec491c6/src/external/glfw/src/input.c#L545C1-L553C2 only works when my cursor is outside of the window.

When its inside, the cursor indeed reports that its centered, but the x and y in the mousepos callback still seem to have the old uncentered value.

For context, this is me disabling all the cursor relocations that raylib is doing, and reporting the values from GLFW directly. I'll have a go with your isolated GLFW example when I've got some time later today.

When INSIDE:

>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: WIN SIZE 818 450
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: GLFW NEW CURSOR POS 409.000000 225.000000
INFO: TIMER: Target time per frame: 16.667 milliseconds
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: WIN SIZE 818 450
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: GLFW NEW CURSOR POS 409.000000 225.000000
WARNING: MouseButtonCallback OLD: 400.000000 225.000000   400.000000 225.000000
WARNING: MouseButtonCallback: 186.000000 249.000000   <--- BAD!

When OUTSIDE:

>>>>>>>>>>>>>>>> GLFW_CURSOR_DISABLED
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: WIN SIZE 818 450
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: GLFW NEW CURSOR POS 409.000000 225.000000
INFO: TIMER: Target time per frame: 16.667 milliseconds
WARNING: MouseButtonCallback OLD: 400.000000 225.000000   400.000000 225.000000
WARNING: MouseButtonCallback: 409.000000 225.000000
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: WIN SIZE 818 450
>>>>>>>>>>>>>>>> _glfwCenterCursorInContentArea: GLFW NEW CURSOR POS 409.000000 225.000000
WARNING: MouseButtonCallback OLD: 409.000000 225.000000   409.000000 225.000000
WARNING: MouseButtonCallback: 408.000000 225.000000
WARNING: MouseButtonCallback OLD: 408.000000 225.000000   408.000000 225.000000
WARNING: MouseButtonCallback: 408.000000 226.000000  <--- GOOD!

I'm also going to do a debugging session with GLFW later on to try and track down why this cursor pos isn't setting properly.

Considering Raylib has the option of building its own GLFW, I think that correcting this issue on our end and pushing it upstream would be the best approach if we can manage it. Even if it takes GLFW a long time to take the patch upstream, Raylib will at least benefit from the fix in the meantime (when building GLFW itself at least).

[ve-nt]

@asdqwe Ah sorry, didn't see you had updated your comment. Thats very interesting that the coord space is changing 5-7 frames in. I'm going to try and reproduce that on my machine and see if I get the same results, because this isn't something I've noticed so far (which doesn't mean it isnt happening).

[ve-nt]

It appears I'm seeing the same thing you are @asdqwe

Just from a quick test on the core_3d_camera_free.c example, I have:

• Removed all SetMousePosition calls in DisableCursor
• Skipped the first 8 frames in the example code

With these two, I'm no longer getting any camera jolt at all. Whether my cursor is over the window or outside the window.

Its not an exact science test as I don't have much time to check this right now, but figured I'd update with this for now.

[ve-nt]

Oh! One thing which is making a difference is your code to initialize Raylib's stored mouse positions in InitPlatform.

I.E. your:

    double xpos, ypos;
    glfwGetCursorPos(platform.handle, &xpos, &ypos);
    CORE.Input.Mouse.previousPosition.x = (float)xpos;
    CORE.Input.Mouse.previousPosition.y = (float)ypos;
    CORE.Input.Mouse.currentPosition.x = (float)xpos;
    CORE.Input.Mouse.currentPosition.y = (float)ypos;

With this in place its working.

I'll clean up my code and put together a branch of exact code thats working for me later on. Hopefully I've not missed anything important and you'll be able to reproduce.

[ve-nt]

Okay screw it I'll just do it now. Here's my working code: ve-nt@fcf2a99

[ve-nt]

No sweat, take all the time you need. Thanks for looking into this.

[raysan5]

@asdqwe wow... really? definitely it seems an issue with GLFW or even the OS windowing system... be careful with that rabbit hole...

EDIT: Afaik, @ColleagueRiley has been dealing with it for RGFW, any ideas?

[ColleagueRiley]

GLFW's handling of things tends to be far more complicated than RGFW's in most cases.

RGFW shouldn't have this problem, so it should be easy to compare the two implementations and see what's different. I'm not really what information is relevant, so here are a few things:

I also made an RGFW article on this issue, not sure if the code is outdated or not:

https://github.com/ColleagueRiley/raw-mouse-input
Here's some of the basic information:

X11 and Windows both have a "raw mouse input mode" that must be enabled. When handling raw mouse input, you must handle specific events,

X11:

	case GenericEvent: {			
		XGetEventData(display, &E.xcookie);
		if (E.xcookie.evtype == XI_RawMotion) {

X11 also requires you to handle MotionNotify events, other wise it will be jumpy.

( GLFW should have simular boolean that can be checked)

		if ((win->_winArgs & RGFW_HOLD_MOUSE)) {
   		/* convert E.xmotion to raw input by subtracting the previous point */
   		win->event.point.x = win->_lastMousePoint.x - E.xmotion.x;
   		win->event.point.y = win->_lastMousePoint.y - E.xmotion.y;
   	}

Windows: You only need to handle the WM_INPUT event.

For cocoa you only need to enable raw input, then you can get the raw mouse point from the usual mouse event. deltaX and deltaY are used to get the raw input.

I'm not 100% sure if GLFW handles Cocoa events directly like this.

CGAssociateMouseAndMouseCursorPosition(0);

I'm not sure if any of this information is helpful,

		case NSEventTypeMouseMoved: {
   		NSPoint p;
   		if ((win->_flags & RGFW_HOLD_MOUSE)) {
   			p.x = ((CGFloat(*)(id, SEL))abi_objc_msgSend_fpret)(e, sel_registerName("deltaX"));
   			p.y = ((CGFloat(*)(id, SEL))abi_objc_msgSend_fpret)(e, sel_registerName("deltaY"));
   		}

You could also achieve this with IOKit, but this way integrates better if you already use the Cocoa event system.

I hope something here is helpful.

[ve-nt]

@asdqwe Thank you so much for all the time and effort you've put into investigating this, its really appreciated. I'll see if I can make some time to continue debugging where you left off, in case there's anything I can find.

I've also linked these onto what appears to be the same bug issue on the GLFW repo, in case any GLFW devs can carry the torch as well.

[ve-nt]

@asdqwe That would indeed solve the issue, however it also affects Linux too so we should add that to the #if defined as well.

I don't think anyone has tested this with Windows, have they?

[ve-nt]

With current master branch (62d8969) there's no camera jolt on Linux.

I've just done a build from there, and I'm seeing a camera jolt on the core_3d_camera_free example when my cursor is over the window. If my cursor starts outside the window there's no jolt.

[raysan5]

@asdqwe Just tested current master branch on Windows 10, I have no Linux device ready at the moment.

core_camera_3d_jolt.mp4

Behaviour looks as expected to me...

[ve-nt]

@asdqwe Nope, mine is on my own last-year-ryzen-something-too-much-ram desktop. Perhaps Xorg versions (or maybe even window managers) have something to do with this?

[ve-nt]

I'm using X.Org version 1.21.1.15

[raysan5]

It seems this issue is beyond raylib as per the deep investigation of @asdqwe. I'm closing this issue now, anyone finding it in the future, feel free to continue investigating this rabbit hole.