Windows Scaling issues: screen functions broken

[kgallimore]

Version
1.7.0
Short overview
Windows Scaling is affecting multiple screen functions, not returning right height/width, messing with screen capture, screen find

Issue occurs on

• Virtual machine
• Docker container
• Dev/Host system

Detailed error description
console.log(screen.height()); = Promise { 864 }
Actual height: 2160
It is affected by windows scaling(Set to 250%, 864*2.5 = 2160)

Set scaling to 100%:
console.log(screen.height()); = Promise { 2160 }

screen.find is also broken, it doesn't find matches, and the incorrect guesses highlight boxes are also positioned erratically.

screen.capture only captures a portion of the screen based off the scaling number.

Steps to reproduce error

Additional content

Please provide any (mandatory) additional data to reproduce the error (Dockerfiles etc.)

[kgallimore]

My placeholder workaround in case anyone is coming across the same issue. You need to expose the included opencv installation and then add robotjs and jimp.

const cv = require("opencv4nodejs-prebuilt");
const robot = require("robotjs");
const Jimp = require("jimp");
async function findQuitTemp(templateImage, targetCoefficient = 0.9) {
  let pic = robot.screen.capture();
  const image = new Jimp(pic.width, pic.height);
  let pos = 0;
  image.scan(0, 0, image.bitmap.width, image.bitmap.height, (x, y, idx) => {
    image.bitmap.data[idx + 2] = pic.image.readUInt8(pos++);
    image.bitmap.data[idx + 1] = pic.image.readUInt8(pos++);
    image.bitmap.data[idx + 0] = pic.image.readUInt8(pos++);
    image.bitmap.data[idx + 3] = pic.image.readUInt8(pos++);
  });
  await image.writeAsync(`${__dirname}/tempresolve.png`);
  const target = await cv.imreadAsync(`${__dirname}/tempresolve.png`);
  const template = await cv.imreadAsync(`${__dirname}/${templateImage}`);

  // Match template (the brightest locations indicate the highest match)
  const matched = target.matchTemplate(template, 5);
  return matched.minMaxLoc().maxVal >= targetCoefficient;
}

[s1hofmann]

@kgallimore Just to make sure: I started digging into this issue and gave your workaround script a run.

Are you sure it really solves the problem? It actually captures the exact same portion of the screen as screen.capture() does? As for search input this workaround is just as affected as nut.js.

The tweaked matching coefficient and a single search run might lead to a different result, tho.

[kgallimore]

@kgallimore Just to make sure: I started digging into this issue and gave your workaround script a run.

Are you sure it really solves the problem? It actually captures the exact same portion of the screen as screen.capture() does? As for search input this workaround is just as affected as nut.js.

The tweaked matching coefficient and a single search run might lead to a different result, tho.

So it only captured a portion in a test script, but when I used it within my electron application it found the whole screen? But yea as a standalone my work around seems to not work either. I'm actually not sure what's going on right now. The biggest difference (besides being inside an electron application) I can think of is what node version is being used.

[s1hofmann]

I haven’t checked this yet, but maybe Electron is, in contrast to the plain node executable, scale aware.