`TargetCamera` breaks UI mouse interaction

[musjj]

Bevy version

0.13.1

What you did

This is a combination of 2d/pixel_grid_snap.rs and ui/button.rs from the examples:

use bevy::{
    prelude::*,
    render::{
        camera::RenderTarget,
        render_resource::{
            Extent3d, TextureDescriptor, TextureDimension, TextureFormat, TextureUsages,
        },
        view::RenderLayers,
    },
    window::WindowResized,
};

/// In-game resolution width.
const RES_WIDTH: u32 = 160;

/// In-game resolution height.
const RES_HEIGHT: u32 = 90;

/// Default render layers for pixel-perfect rendering.
/// You can skip adding this component, as this is the default.
const PIXEL_PERFECT_LAYERS: RenderLayers = RenderLayers::layer(0);

/// Render layers for high-resolution rendering.
const HIGH_RES_LAYERS: RenderLayers = RenderLayers::layer(1);

fn main() {
    App::new()
        .add_plugins(DefaultPlugins.set(ImagePlugin::default_nearest()))
        .insert_resource(Msaa::Off)
        .add_systems(Startup, (setup_camera, setup_ui).chain())
        .add_systems(Update, (button_system, fit_canvas))
        .run();
}

/// Low-resolution texture that contains the pixel-perfect world.
/// Canvas itself is rendered to the high-resolution world.
#[derive(Component)]
struct Canvas;

/// Camera that renders the pixel-perfect world to the [`Canvas`].
#[derive(Component)]
struct InGameCamera;

/// Camera that renders the [`Canvas`] (and other graphics on [`HIGH_RES_LAYERS`]) to the screen.
#[derive(Component)]
struct OuterCamera;

/// Our button
#[derive(Component)]
struct MyButton;

fn setup_ui(mut commands: Commands, query: Query<Entity, With<InGameCamera>>) {
    commands
        .spawn((
            NodeBundle {
                style: Style {
                    width: Val::Percent(100.0),
                    height: Val::Percent(100.0),
                    align_items: AlignItems::Center,
                    justify_content: JustifyContent::Center,
                    ..default()
                },
                ..default()
            },
            TargetCamera(query.single()),
        ))
        .with_children(|parent| {
            parent
                .spawn((
                    MyButton,
                    ButtonBundle {
                        style: Style {
                            width: Val::Px(50.0),
                            height: Val::Px(20.0),
                            border: UiRect::all(Val::Px(2.0)),
                            // horizontally center child text
                            justify_content: JustifyContent::Center,
                            // vertically center child text
                            align_items: AlignItems::Center,
                            ..default()
                        },
                        border_color: BorderColor(Color::BLACK),
                        background_color: BackgroundColor(Color::rgb(0.15, 0.15, 0.15)),
                        ..default()
                    },
                ))
                .with_children(|parent| {
                    parent.spawn(TextBundle::from_section(
                        "Button",
                        TextStyle {
                            font_size: 10.0,
                            color: Color::rgb(0.9, 0.9, 0.9),
                            ..default()
                        },
                    ));
                });
        });
}

fn setup_camera(mut commands: Commands, mut images: ResMut<Assets<Image>>) {
    let canvas_size = Extent3d {
        width: RES_WIDTH,
        height: RES_HEIGHT,
        ..default()
    };

    // this Image serves as a canvas representing the low-resolution game screen
    let mut canvas = Image {
        texture_descriptor: TextureDescriptor {
            label: None,
            size: canvas_size,
            dimension: TextureDimension::D2,
            format: TextureFormat::Bgra8UnormSrgb,
            mip_level_count: 1,
            sample_count: 1,
            usage: TextureUsages::TEXTURE_BINDING
                | TextureUsages::COPY_DST
                | TextureUsages::RENDER_ATTACHMENT,
            view_formats: &[],
        },
        ..default()
    };

    // fill image.data with zeroes
    canvas.resize(canvas_size);

    let image_handle = images.add(canvas);

    // this camera renders whatever is on `PIXEL_PERFECT_LAYERS` to the canvas
    commands.spawn((
        Camera2dBundle {
            camera: Camera {
                // render before the "main pass" camera
                order: -1,
                target: RenderTarget::Image(image_handle.clone()),
                ..default()
            },
            ..default()
        },
        InGameCamera,
        PIXEL_PERFECT_LAYERS,
    ));

    // spawn the canvas
    commands.spawn((
        SpriteBundle {
            texture: image_handle,
            ..default()
        },
        Canvas,
        HIGH_RES_LAYERS,
    ));

    // the "outer" camera renders whatever is on `HIGH_RES_LAYERS` to the screen.
    // here, the canvas and one of the sample sprites will be rendered by this camera
    commands.spawn((Camera2dBundle::default(), OuterCamera, HIGH_RES_LAYERS));
}

fn button_system(mut interaction_query: Query<&Interaction, (Changed<Interaction>, With<Button>)>) {
    for interaction in &mut interaction_query {
        match *interaction {
            Interaction::Pressed => {
                println!("pressed!");
            }
            Interaction::Hovered => {
                println!("hovered!");
            }
            Interaction::None => {
                println!("none!");
            }
        }
    }
}

fn fit_canvas(
    mut resize_events: EventReader<WindowResized>,
    mut projections: Query<&mut OrthographicProjection, With<OuterCamera>>,
) {
    for event in resize_events.read() {
        let h_scale = event.width / RES_WIDTH as f32;
        let v_scale = event.height / RES_HEIGHT as f32;
        let mut projection = projections.single_mut();
        projection.scale = 1. / h_scale.min(v_scale).round();
    }
}

What went wrong

Button interaction simply does not work at all. Hovering and clicking the button should cause hovered! and pressed! to be printed to the console, but it does nothing here.

I believe this is a critical issue for pixel-art games as this prevents mouse interaction from working at all. I hope that there is at least a workaround for such a show-stopping bug.

Additional information

Some relevant PRs:

• Camera-driven UI #10559
• Unify RenderLayers and TargetCamera (Part 1: RenderGroups) [ADOPT ME] #12502
• Add interaction for render target with custom size #5815

Relevant issues:

• World-space UI #5476
• Add In-World UI example #5479

[alice-i-cecile]

Can you get this working with bevy_mod_picking for your interactions? I suggest that both as a workaround, and because we're in the process of upstreaming it (and ripping out the existing interaction code to use it).

[musjj]

I just tried using bevy_mod_picking, but alas it still does not work. Without the TargetCamera component everything works just fine, but in case I'm doing something wrong, here's the updated code:

Snippet

use bevy::{
    prelude::*,
    render::{
        camera::RenderTarget,
        render_resource::{
            Extent3d, TextureDescriptor, TextureDimension, TextureFormat, TextureUsages,
        },
        view::RenderLayers,
    },
    window::WindowResized,
};
use bevy_mod_picking::prelude::*;

/// In-game resolution width.
const RES_WIDTH: u32 = 160;

/// In-game resolution height.
const RES_HEIGHT: u32 = 90;

/// Default render layers for pixel-perfect rendering.
/// You can skip adding this component, as this is the default.
const PIXEL_PERFECT_LAYERS: RenderLayers = RenderLayers::layer(0);

/// Render layers for high-resolution rendering.
const HIGH_RES_LAYERS: RenderLayers = RenderLayers::layer(1);

fn main() {
    App::new()
        .add_plugins(DefaultPlugins.set(ImagePlugin::default_nearest()))
        .add_plugins(DefaultPickingPlugins)
        .insert_resource(Msaa::Off)
        .insert_resource(DebugPickingMode::Normal)
        .add_systems(Startup, (setup_camera, setup_ui).chain())
        .add_systems(Update, fit_canvas)
        .run();
}

/// Low-resolution texture that contains the pixel-perfect world.
/// Canvas itself is rendered to the high-resolution world.
#[derive(Component)]
struct Canvas;

/// Camera that renders the pixel-perfect world to the [`Canvas`].
#[derive(Component)]
struct InGameCamera;

/// Camera that renders the [`Canvas`] (and other graphics on [`HIGH_RES_LAYERS`]) to the screen.
#[derive(Component)]
struct OuterCamera;

/// Our button
#[derive(Component)]
struct MyButton;

fn setup_ui(mut commands: Commands, query: Query<Entity, With<InGameCamera>>) {
    commands
        .spawn((
            NodeBundle {
                style: Style {
                    width: Val::Percent(100.0),
                    height: Val::Percent(100.0),
                    align_items: AlignItems::Center,
                    justify_content: JustifyContent::Center,
                    ..default()
                },
                ..default()
            },
            Pickable::IGNORE,
            TargetCamera(query.single()),
        ))
        .with_children(|parent| {
            parent
                .spawn((
                    MyButton,
                    ButtonBundle {
                        style: Style {
                            width: Val::Px(50.0),
                            height: Val::Px(20.0),
                            border: UiRect::all(Val::Px(2.0)),
                            // horizontally center child text
                            justify_content: JustifyContent::Center,
                            // vertically center child text
                            align_items: AlignItems::Center,
                            ..default()
                        },
                        border_color: BorderColor(Color::BLACK),
                        background_color: BackgroundColor(Color::rgb(0.15, 0.15, 0.15)),
                        ..default()
                    },
                    On::<Pointer<Click>>::run(|| info!("pressed!")),
                    On::<Pointer<Over>>::run(|| info!("hovered!")),
                ))
                .with_children(|parent| {
                    parent.spawn((
                        TextBundle::from_section(
                            "Button",
                            TextStyle {
                                font_size: 10.0,
                                color: Color::rgb(0.9, 0.9, 0.9),
                                ..default()
                            },
                        ),
                        Pickable::IGNORE,
                    ));
                });
        });
}

fn setup_camera(mut commands: Commands, mut images: ResMut<Assets<Image>>) {
    let canvas_size = Extent3d {
        width: RES_WIDTH,
        height: RES_HEIGHT,
        ..default()
    };

    // this Image serves as a canvas representing the low-resolution game screen
    let mut canvas = Image {
        texture_descriptor: TextureDescriptor {
            label: None,
            size: canvas_size,
            dimension: TextureDimension::D2,
            format: TextureFormat::Bgra8UnormSrgb,
            mip_level_count: 1,
            sample_count: 1,
            usage: TextureUsages::TEXTURE_BINDING
                | TextureUsages::COPY_DST
                | TextureUsages::RENDER_ATTACHMENT,
            view_formats: &[],
        },
        ..default()
    };

    // fill image.data with zeroes
    canvas.resize(canvas_size);

    let image_handle = images.add(canvas);

    // this camera renders whatever is on `PIXEL_PERFECT_LAYERS` to the canvas
    commands.spawn((
        Camera2dBundle {
            camera: Camera {
                // render before the "main pass" camera
                order: -1,
                target: RenderTarget::Image(image_handle.clone()),
                ..default()
            },
            ..default()
        },
        InGameCamera,
        PIXEL_PERFECT_LAYERS,
    ));

    // spawn the canvas
    commands.spawn((
        SpriteBundle {
            texture: image_handle,
            ..default()
        },
        Canvas,
        Pickable::IGNORE,
        HIGH_RES_LAYERS,
    ));

    // the "outer" camera renders whatever is on `HIGH_RES_LAYERS` to the screen.
    // here, the canvas and one of the sample sprites will be rendered by this camera
    commands.spawn((Camera2dBundle::default(), OuterCamera, HIGH_RES_LAYERS));
}

fn fit_canvas(
    mut resize_events: EventReader<WindowResized>,
    mut projections: Query<&mut OrthographicProjection, With<OuterCamera>>,
) {
    for event in resize_events.read() {
        let h_scale = event.width / RES_WIDTH as f32;
        let v_scale = event.height / RES_HEIGHT as f32;
        let mut projection = projections.single_mut();
        projection.scale = 1. / h_scale.min(v_scale).round();
    }
}

[StrikeForceZero]

So I don't think it's TargetCamera that is at fault here.

Might even be a known limitation when rendering to a non-window target based on this comment here:

bevy/crates/bevy_ui/src/focus.rs

Line 193 in 017ffc5

// Interactions are only supported for cameras rendering to a window.

but if you swap your
With<InGameCamera> in your setup_ui to With<OuterCamera> it will exhibit the same behavior as omitting TargetCamera and that's because without TargetCamera, it will just pick the camera with the highest order that has its render target set to the window

bevy/crates/bevy_ui/src/ui_node.rs

Lines 2180 to 2187 in 017ffc5

	.filter(|(_, c)| match c.target {
	RenderTarget::Window(WindowRef::Primary) => true,
	RenderTarget::Window(WindowRef::Entity(w)) => {
	self.primary_window.get(w).is_ok()
	}
	_ => false,
	})
	.max_by_key(|(e, c)| (c.order, *e))

Edit: If support can't be added, it might be worth having TargetCamera warn when using a camera whose render target is not a window.

[musjj]

Yeah, I think bevy_mod_picking works on the same logic (at least for UI).

I think the best solution here is to provide a way to achieve the same thing without a RenderTarget. Would it be possible to create a projection that retains the original pixel grid, regardless of its zoom level (if that makes sense)?

Currently, if you zoom an OrthographicProjection without RenderTarget-ing a texture you will no longer get pixelation for free: drakmaniso/bevy_pixel_camera#30. In fact, I don't think it even affects the scale of the UI right now.

I would love to hear how other pixel art devs currently do their UI.

Also, does anyone know how other engines like Godot, Love2D or Ebitengine achieves this? I remember having a pretty easy time achieving pixel-art graphics there.

Godot: https://docs.godotengine.org/en/stable/tutorials/rendering/multiple_resolutions.html#desktop-game

Love2D: a327ex/blog#19

Ebitengine: https://ebitengine.org/en/examples/animation.html

[StrikeForceZero]

I haven't tested it yet but I think this might provide some the ground work to get interactions working with UI nodes that specify a TargetCamera rendering to a texture (if you were to use bevy_mod_picking and egui). You can in theory have the replaced InputPlugin set the Interaction component.

https://github.com/aevyrie/bevy_mod_picking/blob/main/examples/render_to_texture.rs

[aevyrie]

I made some noise when target camera was first added because IMO it overcomplicates a bunch of things, picking included. This is solvable, all that is needed is to update the bevy_ui picking backend to look at the target camera render target when testing node hierarchies. It's just going to be some special casing.

I would much prefer we get rid of this weird setup, and just make the ui parented to the camera explicitly.