(nut-tree/nut.js/183) Determine scale factor ifor high DPI displays i…

[s1hofmann]

…n Windows

Recent Windows versions allow to change the "Scale" factor of the display. This is particularly useful on high DPI devices, where displaying windows in the regular pixel size would make it really hard to read (think: 3240x2160 resolution on a regular laptop, where the Scale factor would default to 200%). (Basically the same behavior as with retina displays)

To maintain compatibility with scale-unaware software, this is an opt-in feature, where applications have to specify via their manifest that they are aware of it. Otherwise, Windows will pretend that the resolution is actually lower.

However, this leads to a problem where calling screen.capture() thinks that the resolution on above-mentioned laptop is actually 1620x1080, yet when capturing a 1620X1080 rectangle at 0x0, it only captures the upper left quadrant!

This PR calculates the scale factor by comparing desktop sizes to the size of the actual screen bitmap. This way it’s possible to capture the whole screen and report the correct image size so nut.js is able to apply its scale invariant image search.

Please note that this PR does not contain a test. Changing display DPI requires a logout cycle, so I don’t see a way of running it on CI.
A manual test has been carried out successfully by both me and @dscho

[dscho]

Thank you!

[s1hofmann]

@dscho You‘re welcome! Thanks for providing @git-for-windows!

[dscho]

Thank you!!!

[bertyhell]

I tried installing this PR to see if this issue is fixed, but it is still occuring. So maybe i'm doing something wrong with the installation?

install commands:

npm i github:nut-tree/libnut#pull/59/head
npm i @nut-tree/nut-js@next

package json:

    "libnut": "github:nut-tree/libnut#pull/59/head",
    "@nut-tree/nut-js": "^2.0.2-next.20220324193210",

When i run my app and put my cursor all the way at the bottom right of the screen i get:

import { mouse, Point } from '@nut-tree/nut-js'

const mousePosition = await mouse.getPosition()
printProgress('x: ' + mousePosition.x + '\t' + 'y: ' + mousePosition.y)

My windows 11 screen settings (single monitor):

When i change the scaling to 100% instead of 150, the issue no longer occurs:

windows version:

NodeJS version:
v16.13.0

npm version
8.1.0

I'm using this for a test script on my pc, so i can just scale up the mouse position by 1.5 to fix the issue. So this isn't a blocking issue to keep using this package for now.

I love the API btw, so simple and easy to use :D

[s1hofmann]

Hi @bertyhell 👋

First of all: You don’t have to perform any custom install. Every change merged to develop in libnut automatically propagates across repos and ends up triggering a new @next release of nut.js

I love the API btw, so simple and easy to use

Thank you, that’s one of my main goals with nut.js!

Regarding your issue, this is actually expected behavior.
Your native resolution is 3840x2160, but with 150% scaling applied.

This means that the graphics content of your screen is 3840x2160 pixels in size, let’s call that image space.
However, with 150% scaling applied this becomes (3840/1.5)x(2160/1.5) === 2.560x1.440 pixels in screen space.

Screen space is what’s relevant for mouse movement etc., so it’s absolutely correct that your bottom right point is 2.559x1.439.

The main goal I wanted to achieve with this PR was to establish the mapping from image space back to screen space to enable template matching with scaling applied.

I hope this sheds some light on the purpose of this PR!

Best regards

Simon