Input Latency in Window Event

[WenchaoHuang]

Version/Branch of Dear ImGui:

docking

Back-ends:

Win32 + Vulkan

Compiler, OS:

Windows + MSVC

Full config/build information:

io.ConfigFlags |= ImGuiConfigFlags_DockingEnable;         // Enable Docking
io.ConfigFlags |= ImGuiConfigFlags_ViewportsEnable;       // Enable Multi-Viewport / Platform Windows

Details:

When moving a floating windows, I can clearly perceive higher latency compared to many other applications I use (e.g., Chrome, Visual Studio, VS Code). Even with triple buffering enabled in the swap chain (FPS reaching 4000+), there remains latency when compared to other softwares.

In slow-motion video analysis, the relative position between the mouse cursor's clicked point and the window shifts during movement, while other applications maintain consistent alignment. This is because ImGui's event mechanism follows the 'collect -> process + respond' pattern, and ImGui's collection phase is already a response to the Win32 modal loop. A common approach is to notify the OS in the Win32 message callback that the window movement is starting.

Flowing is my experimental codes added in ImGui_ImplWin32_WndProcHandler_PlatformWindow function:

    case WM_NCHITTEST:
        // Let mouse pass-through the window. This will allow the backend to call io.AddMouseViewportEvent() correctly. (which is optional).
        // The ImGuiViewportFlags_NoInputs flag is set while dragging a viewport, as want to detect the window behind the one we are dragging.
        // If you cannot easily access those viewport flags from your windowing/event code: you may manually synchronize its state e.g. in
        // your main loop after calling UpdatePlatformWindows(). Iterate all viewports/platform windows and pass the flag to your windowing system.
        if (viewport->Flags & ImGuiViewportFlags_NoInputs)
            result = HTTRANSPARENT;
        else if (ctx->HoveredWindow)
        {
            //  Should Win32 modal takes over the window move operation, if the window can be dragged?
            if ((!io.ConfigWindowsMoveFromTitleBarOnly && !ImGui::IsAnyItemHovered()) ||
                ( io.ConfigWindowsMoveFromTitleBarOnly && !ImGui::IsAnyItemHovered() && ctx->HoveredWindow->TitleBarRect().Contains(ImGui::GetMousePos())))
            {
                result = DefWindowProc(hWnd, msg, wParam, lParam);
                return (result == HTCLIENT) ? HTCAPTION : result;
            }
        }
        break;
    case WM_ENTERSIZEMOVE:
    case WM_EXITSIZEMOVE:
    {
        ImGui_ImplWin32_Data* bd = ImGui_ImplWin32_GetBackendData(io);
        if (bd) break;

        ImGuiMouseSource mouse_source = ImGui_ImplWin32_GetMouseSourceFromMessageExtraInfo();
        int button = 0;     //  always the left button
        if (::GetCapture() == nullptr)
            ::SetCapture(hWnd); // Allow us to read mouse coordinates when dragging mouse outside of our window bounds.
            else
        ::ReleaseCapture();

        bd->MouseButtonsDown |= 1 << button;
        io.AddMouseSourceEvent(mouse_source);
        io.AddMouseButtonEvent(button, msg == WM_ENTERSIZEMOVE);
        break;
    }
    }

After adding these lines of code, the window no longer feels laggy when moved! However, this introduces another issue: when the Win32 modal loop takes over the window movement event, the window's owning thread gets blocked. If the UI updates run on the same thread, all ImGui windows stop updating. To break this blocking, an alternative update mechanism is needed. I suggest using SetTimer() to make the window's thread trigger a callback function periodically to redraw the UI. With this approach, the window movement experience becomes much smoother.

My experimental code does not take other features into account globally, so I don’t plan to submit a pull request. From a code maintenance perspective, would it be necessary to add an enum value (e.g., ImGuiConfigFlags_OSTakeOverWindowEvent) to enable this feature?

Screenshots/Video:

Desktop.2025.04.01.-.13.56.15.02.mp4

Minimal, Complete and Verifiable Example code:

ImGui::NewFrame();
....
ImGui::Begin("Circle");
if (ImGui::IsMouseDown(0))
ImGui::Text("IsMouseDown(0) = true");
else
ImGui::Text("IsMouseDown(0) = false");
ImDrawList * draw_list = ImGui::GetWindowDrawList();
draw_list->AddCircle(ImGui::GetMousePos(), 5.0f, IM_COL32_WHITE, 200);
draw_list->AddCircle(ImGui::GetMousePos(), 15.0f, IM_COL32_WHITE, 200);
draw_list->AddCircle(ImGui::GetMousePos(), 25.0f, IM_COL32_WHITE, 200);
draw_list->AddCircle(ImGui::GetMousePos(), 35.0f, IM_COL32_WHITE, 200);
ImGui::End();
...
ImGui::Render();

[eXifreXi]

Very interesting. I was one 'enter key' press away from searching about 'Mouse Input Lag' issues when I noticed your post at the top.

I'm currently trying to implement ImGui into Unreal Engine. While there are many existing implementations, they all use the same two options: either require the user to actively enable/disable interaction with ImGui vs. Unreal Engine or use an InputPreProcessor that causes issues when an ImGui widget is ZOrder-wise behind an Unreal Engine widget.

I got it as far as having one Slate Widget per ImGui window, but I have a lot of issues with Positioning and Rendering while dragging said window. I'm using the ImGuiWindow::Pos to position the Slate Widget on Tick and the ImGuiWindow::DrawListInst to push the Vertices etc. to Unreal Engine's RenderThread.

I can somewhat understand that the visuals of the Widget lag behind, due to the RenderThread not being "instant", but the actual ImGuiWindow::Pos property is already causing me a headache because when dragging the window upwards, the mouse manages to "move ahead" and escape the Slate Widget's bounds, causing it to be unfocused.

I could solve some of my problems by still using the InputPreProcessor, to feed ImGui the mouse position at all times, but I would much rather have the ImGuiWindow::Pos be in line with where my mouse cursor moved, so this can't even happen.

What I'm struggling with looks very similar to your attached video and I wasn't sure if it's ImGui being used within UE that naturally causes this, or if I'm doing something wrong, or if the 'issue' is within ImGui. That said, opening the interactive "ImGui Manual" shows the same problem when dragging a window.

Mostly just posting here to share experiences and stay up-to-date with any future responses.

[WenchaoHuang]

Ha, sounds like your latency issue might be even more severe than what I described. Let me try to break this down based on your information and my limited UE knowledge.

The message pipeline appears to be:

1. User actions
2. UE callback or ImGui callback, depending on whether the reciving native window created by ImGui or UE. In any case, ImGui gets notified at this stage.
3. ImGui::NewFrame() + ImGui::Render() (ImGuiWindow::Pos is updated)
4. UE::Tick(): updates widgets (note that this is not real-time)
5. UE render thread submits tasks to GPU, and updates widget positions
6. Monitor refresh (final visual output)

My issue stops at step 3. My solution is to inform the operating system at stage 2 that the window is about to be moved. The Win32 modal loop then takes over and automatically handles the window movement. This ensures that the relative position between the mouse drag point and the window remains consistent on every frame during movement, resulting in extremely low latency. Many applications indeed use this approach.

[eXifreXi]

Hey, thanks for the reply. I didn't mean to pull the problem away towards Unreal Engine-specifics, but I appreciate you trying to re-interpret the general issue.

Throughout yesterday, I managed to resolve a bunch of issues that I had caused, which I had expected anyway.

Let me try to describe what's going on on my end a bit more then. The following wall of text is Unreal Engine-specific. If whoever reads this doesn't care about Unreal Engine, just ignore it. It ultimately touches on the Input Latency problem, but this will be a bit of text before that happens. I packed it into a collapsed section so it doesn't spam too much.

<-- Expand to read about ImGui + Unreal Engine and its reasons for Input Latency.

How ImGui wants the order of actions per "frame" to work (I believe...)

1. Have the backend figure out input.
2. Receive frame-related data via IO, such as ImGuiIO::DeltaTime.
3. Receive the ImGui::NewFrame() call.
4. Receive the ImGui methods to draw UI.
5. Receive the ImGui::Render() call (which internally calls ImGui::EndFrame()).
6. Have the backend draw a given ImDrawData (usually the MainViewport one, aka ImGuiContext::Viewports[0]).

Overview of Unreal Engine's FEngineLoop::Tick()

The following overview leaves a lot of things out (as Unreal Engine does quite a lot of things).
It also lists where the above ImGui actions would happen, marked via (1.).

FEngineLoop::Tick()

1. FCoreDelegates::OnBeginFrame.Broadcast()
   1. (2.) Receive any Frame-related data via IO, such as ImGuiIO::DeltaTime.
   2. (3.) Receive the ImGui::NewFrame() call.
2. FPlatformApplicationMisc::PumpMessages()
   1. (1.) Have the backend figure out input.
3. GEngine::Tick(), ticking the current UEngine implementation.
   1. (4.) Receive the ImGui methods to draw UI by Gameplay objects.
4. FSlateApplication::Get().Tick() -> FSlateApplication::TickAndDrawWidgets() -> SWidget::OnPaint()
   1. (6.) Have the backend draw a given ImDrawData.
5. FTSTicker::GetCoreTicker().Tick()
   1. (4.) Receive the ImGui methods to draw UI by Editor objects.
6. FCoreDelegates::OnEndFrame.Broadcast()
   1. (5.) Receive the ImGui::Render() call (which internally calls ImGui::EndFrame()).

Looks pretty out of order, doesn't it? Let's go over each part that doesn't align.

Input is handled in Unreal Engine after ImGui::Begin() gets called.

ImGui utilizes input inside ImGui::Begin(), but Unreal Engine handles input only after (1.) FCoreDelegates::OnBeginFrame.Broadcast() in step (2.) FPlatformApplicationMisc::PumpMessages().

I tried to find a callback that happens after (2.) FPlatformApplicationMisc::PumpMessages() and before (3.) GEngine::Tick(), but there doesn't seem to be any.

This is the first part of the setup which causes additional Input Latency, unrelated to ImGui, due to having to wait for a full Unreal Engine frame, before being able to utilize the input information within ImGui::NewFrame().

Rendering of the SWidgets happens before ImGui::Render().

This is an annoying thing and caused most of my problems from yesterday. The way one utilizes and renders the ImDrawData is via a custom SWidget (Slate Widget), which has an OnPaint() method. That method provides an FSlateWindowElementList& parameter, into which one would parse the ImDrawData::CmdLists::CmdBuffer data.

The FSlateWindowElementList is then forwarded to an implementation of FSlateRenderer, which will then do the heavy lifting of enqueuing render commands. In our case, that's an RHI Renderer, which will use the underlying renderer API. I assume this will be DirectX11/12 in this case.

Now that's a big problem though because as per the comment on ImDrawData::Valid, one can only utilize the ImDrawData AFTER (5.) Receive the ImGui::Render() call. and before (3.) Receive the ImGui::NewFrame() call..
That's clearly not an option here, as the Slate rendering happens in the middle of the ImGui frame.

The way existing implementations solve that is relatively simple: They copy the ImDrawData after (5.) Receive the ImGui::Render() call. and then use it within SWidget::OnPaint during the next frame.

This is the second part of the setup which causes additional Input Latency, unrelated to ImGui, due to having to wait for yet another full Unreal Engine frame, before being able to utilize the ImDrawData within SWidget::OnPaint. On top of that, of course, comes the delay from the whole async renderer thread.

Some implementation specifics on my end.

I'm actually not using the ImDrawData that comes via the ImGui::GetDrawData() call.

For one, I have the multi-viewport support, so ImGui is actually going to give me an ImGuiViewport to render instead of me calling ImGui::GetDrawData().

And then I'm not using one single SWidget in Unreal Engine to draw the whole ImDrawData, but I collect all ImGuiWindow instances after (5.) Receive the ImGui::Render() call., compare their ImGuiWindow::Viewport against the Viewport ImGui wants to render, and then pass that into a SWidget which will create another SWidget per ImGuiWindow. And that second SWidget then uses ImGuiWindow::DrawListInst to draw just that ImGuiWindow.

Most Unreal Engine implementations will use one single SWidget and draw all ImDrawData in there, but that causes a bunch of issues, which the end-user has to suffer from. That's why I draw one SWidget per ImGuiWindow::DrawListInst.

Now, due to the order of actions being all mixed up, I also had to copy some other properties out of the ImGuiWindow, such as its position, size, if the window is viewport owned, if it's a fallback window, if it's an explicit child, etc.
Most of those properties are used to draw the second SWidget in the right location and size, as well as to offset the ImGui vertices so that the top left corner is always (0, 0) and the position is applied through positioning the SWidget instead.

This is where my additional actual bugs came from because I wasn't copying that data properly, using the ImGuiWindow property after a new frame had already begun.

This is mostly resolved now, and my SWidget + ImGuiWindow setup works quite nicely.

That all said, and all the Unreal Engine introduced Input Latency aside, the default Input Latency of ImGui seems to just add to it, and I'm not convinced that, with 60+FPS in the Editor, it would be that "rubberbandy" with just those 2 frames delay, which is introduced by Unreal Engine.

Here is a video of the delay. Should be at 75 FPS (roughly). Ignore the blue color, I'm overlaying it in Unreal Engine at the moment to see the size of the actual SWidget. The times when I quickly moved the mouse to the right and it stopped moving was when I managed to leave the window with the mouse while dragging. I will fix that it doesn't continue dragging on my end, as that's an implementation issue, but it wouldn't be needed if the window would stick to the mouse.

UnrealEditor-Win64-DebugGame_N2umm88iLQ.mp4

I don't think your suggested solution is something I can apply to the Unreal Engine, so this might generally not be fixable for me.

[WenchaoHuang]

Hey. If the video is in 1:1 real-time and the rendering refresh rate is calculated at 75 FPS, a 2-frame delay would indeed feel "rubberbandy" as you described. You can estimate that if you move the mouse at a speed of about 1000 pixels per second, in 20ms your mouse could move approximately 20 pixels.

From what I understand, if you don’t let the OS take over window movement, it’s impossible to make the window follow the mouse perfectly, and the reason for this is what I mentioned earlier. Enabling triple buffering and increasing the frame rate can significantly alleviate this issue. So, you still need to fix the issue with stopping the drag :D.

Based on my experience, I suspect you might not be using the backend code provided by ImGui but instead modified it yourself. The reason for the drag signal loss is likely because you didn’t tell the OS to use SetCapture() in the mouse click callback and ReleaseCapture() when releasing it.

Although I’m not very familiar with the mechanisms of Unreal Engine, I hope my response can be helpful to you.

[eXifreXi]

From what I understand, if you don’t let the OS take over window movement, it’s impossible to make the window follow the mouse perfectly, and the reason for this is what I mentioned earlier. Enabling triple buffering and increasing the frame rate can significantly alleviate this issue.

Yeah, which I probably won't be able to and won't bother. Maybe in the future.

So, you still need to fix the issue with stopping the drag :D.
Based on my experience, I suspect you might not be using the backend code provided by ImGui but instead modified it yourself. The reason for the drag signal loss is likely because you didn’t tell the OS to use SetCapture() in the mouse click callback and ReleaseCapture() when releasing it.

That sounds reasonable, as I'm indeed not capturing the Mouse at that moment. I was going to resolve it by generally feeding ImGui the Position and MouseButtonUp events, instead of only if the Mouse is above the SWidget. Maybe capturing it will already resolve this. I will play around with both options.

EDIT: For what it's worth, generally feeding ImGui those two events resolves this issue, but I will try with capturing the mouse too.

EDIT2: Good call, using return IO.WantCaptureMouse ? FReply::Handled().CaptureMouse(AsShared()) : FReply::Unhandled(); seems to also resolve the issue. That makes me happy. I'm not sure, however, if I should still have a more "global" setup for some of the Input related ImGuiIO things. Is there anything that always has to be handled, no matter what? I read in the FAQ that ImGui would love to have the MouseButtonDown event either way, to handle focus.

EDIT3: I'll leave your issue post alone then. The Latency Issue is still "the same", and it's good to know that I'm not alone with it, generally, but I also don't think I can figure this out, given Unreal Engine handles a large portion already.