Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Add texture support for macOS shell. #8507

Merged
merged 40 commits into from
Oct 10, 2019
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
5302cf8
Merge pull request #1 from flutter/master
cloudwebrtc Mar 28, 2019
2953e13
Merge remote-tracking branch 'upstream/master'
cloudwebrtc Apr 9, 2019
4b422b3
Add texture support for macOS shell.
cloudwebrtc Apr 9, 2019
fa3f8b0
code format.
cloudwebrtc Apr 9, 2019
a222a4a
Add missing file.
cloudwebrtc Apr 9, 2019
668e656
Update FLETexture.h
cloudwebrtc Apr 9, 2019
97f1e91
Update FLEPluginRegistrar.h
cloudwebrtc Apr 10, 2019
cd636b4
Restore header file order.
cloudwebrtc Apr 10, 2019
799e9c1
Update code styles and comments.
cloudwebrtc Apr 10, 2019
52b1417
Rename textureId to textureID.
cloudwebrtc Apr 10, 2019
e49c549
Update comment for FLEExternalTextureGL.h.
cloudwebrtc Apr 10, 2019
7589f46
Update comments and partial code.
cloudwebrtc Apr 17, 2019
2c60631
Rename the texture in some methods to disambiguate.
cloudwebrtc Apr 17, 2019
42b7502
Merge remote-tracking branch 'upstream/master'
cloudwebrtc Apr 18, 2019
dbcdf16
Merge branch 'master' into macos_texture
cloudwebrtc Apr 18, 2019
9b15427
Fix errors caused by merge master conflicts.
cloudwebrtc Apr 18, 2019
e4ff652
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc May 19, 2019
ba2efa3
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc Jul 11, 2019
9c19bac
clang format.
cloudwebrtc Jul 16, 2019
6ff0a76
gn format.
cloudwebrtc Jul 16, 2019
ec8083a
Add files to licenses_flutter.
cloudwebrtc Jul 16, 2019
ff49e14
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc Sep 24, 2019
ee09e69
clang-format.
cloudwebrtc Sep 24, 2019
0f06004
Update.
cloudwebrtc Sep 24, 2019
09156bf
Use shared FlutterTexture.h.
cloudwebrtc Sep 25, 2019
235314a
clang-format.
cloudwebrtc Sep 25, 2019
650c00a
Fixed FlutterTexture.h import path for ios.
cloudwebrtc Sep 25, 2019
d4c0be8
Update.
cloudwebrtc Sep 26, 2019
cda947c
clang-format.
cloudwebrtc Sep 26, 2019
e78d2e1
Change _pixelBuffer to a local variable.
cloudwebrtc Sep 27, 2019
eb3bf39
Add missing static on a local constant
stuartmorgan-g Sep 27, 2019
d622f01
Fix variable naming inconsistency
stuartmorgan-g Sep 27, 2019
ec70d4a
Merge branch 'master' into macos_texture
cloudwebrtc Oct 5, 2019
2871956
Use drawImageRect to draw textures of undesired size.
cloudwebrtc Oct 5, 2019
9b63c65
Merge branch 'macos_texture' of https://github.com/cloudwebrtc/flutte…
cloudwebrtc Oct 5, 2019
4962f91
clang-format.
cloudwebrtc Oct 5, 2019
b20be2c
Fixed CI errors.
cloudwebrtc Oct 5, 2019
067e8a1
update.
cloudwebrtc Oct 7, 2019
27c6f94
clang-format.
cloudwebrtc Oct 7, 2019
5f6833f
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc Oct 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions shell/platform/darwin/macos/BUILD.gn
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ _flutter_framework_dir = "$root_out_dir/$_flutter_framework_filename"
# the Flutter engine source root.
_flutter_framework_headers = [
"framework/Headers/FlutterMacOS.h",
"framework/Headers/FLETexture.h",
"framework/Headers/FLEOpenGLContextHandling.h",
"framework/Headers/FLEPlugin.h",
"framework/Headers/FLEPluginRegistrar.h",
Expand All @@ -33,6 +34,8 @@ shared_library("create_flutter_framework_dylib") {
output_name = "$_flutter_framework_name"

sources = [
"framework/Source/FLEExternalTextureGL.h",
"framework/Source/FLEExternalTextureGL.mm",
"framework/Source/FLETextInputModel.h",
"framework/Source/FLETextInputModel.mm",
"framework/Source/FLETextInputPlugin.h",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
#import <Cocoa/Cocoa.h>

#import "FLEPlugin.h"
#import "FLETexture.h"

#if defined(FLUTTER_FRAMEWORK)
#import "flutter/shell/platform/darwin/ios/framework/Headers/FlutterBinaryMessenger.h"
Expand All @@ -31,6 +32,11 @@ FLUTTER_EXPORT
*/
@property(nonnull, readonly) id<FlutterBinaryMessenger> messenger;

/**
* Returns a `FLETextureRegistrar` for registering textures provided by the plugin.
*/
@property(nonnull, readonly) id<FLETextureRegistrar> textures;

/**
* The view displaying Flutter content.
*
Expand Down
43 changes: 43 additions & 0 deletions shell/platform/darwin/macos/framework/Headers/FLETexture.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to be identical to FlutterTexture.h other than the prefixes; now that macOS is in the same repo as iOS the goal is to share code whenever possible (and move away from the FLE prefix). You can just add that file to the frameworkSshared.gni list, and use the existing protocols in the macOS code.

(If you want to add these comments to FlutterTexture.h, ideally as a separate PR, that would be great, since the existing file isn't commented at all.)

Copy link
Contributor Author

@cloudwebrtc cloudwebrtc Apr 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chinmaygarde @stuartmorgan
I found in the macos test that CVOpenGLTextureCacheCreateTextureFromImage creates a GL_TEXTURE_RECTANGLE object instead of GL_TEXTURE_2D, which causes the texture to not be properly scaled to fill the flutter widget when rendering, so I used width/height in texture->copyPixelBuffer when the plugin receives the request, it scales the texture to the corresponding size according to the widget size provided by the engine, and then becomes normal when rendering.
I'm not sure to add width/height to ios/framework/Headers/FlutterTexture.h, or continue to use FLETexture.h, or should report a new issue in flutter engine.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to specify that the texture target is GL_TEXTURE_RECTANGLE by specifying the target property of FlutterOpenGLTexture. All the copying ought to be unnecessarily inefficient.

// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#import <CoreVideo/CoreVideo.h>
#import <Foundation/Foundation.h>

/**
* Implement a texture object for the FLE plugin side.
*/
@protocol FLETexture <NSObject>

/**
* When the texture on the platform side is ready,
* the flutter engine will copy the texture buffer
* using copyPixelBuffer.
*/
- (nullable CVPixelBufferRef)copyPixelBuffer:(size_t)width height:(size_t)height;

@end

/**
* The protocol for an object managing registration for texture.
*/
@protocol FLETextureRegistrar <NSObject>

/**
* Register a |texture| object and return a textureID.
*/
- (int64_t)registerTexture:(nonnull id<FLETexture>)texture;

/**
* Mark a texture buffer is ready.
*/
- (void)textureFrameAvailable:(int64_t)textureID;

/**
* Unregister an existing Texture object.
*/
- (void)unregisterTexture:(int64_t)textureID;

@end
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#import "FLEOpenGLContextHandling.h"
#import "FLEPluginRegistrar.h"
#import "FLEReshapeListener.h"
#import "FLETexture.h"

#if defined(FLUTTER_FRAMEWORK)
#import "flutter/shell/platform/darwin/ios/framework/Headers/FlutterBinaryMessenger.h"
Expand Down Expand Up @@ -35,8 +36,10 @@ typedef NS_ENUM(NSInteger, FlutterMouseTrackingMode) {
* Flutter engine in non-interactive mode, or with a drawable Flutter canvas.
*/
FLUTTER_EXPORT
@interface FLEViewController
: NSViewController <FlutterBinaryMessenger, FLEPluginRegistrar, FLEReshapeListener>
@interface FLEViewController : NSViewController <FlutterBinaryMessenger,
FLEPluginRegistrar,
FLEReshapeListener,
FLETextureRegistrar>

/**
* The view this controller manages when launched in interactive mode (headless set to false). Must
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#import "FLEPlugin.h"
#import "FLEPluginRegistrar.h"
#import "FLEReshapeListener.h"
#import "FLETexture.h"
#import "FLEView.h"
#import "FLEViewController.h"
#import "FlutterBinaryMessenger.h"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#import <Foundation/Foundation.h>

#import "flutter/shell/platform/darwin/macos/framework/Headers/FLETexture.h"
#import "flutter/shell/platform/embedder/embedder.h"

/**
* Used to bridge FLETexture object and handle the texture copy request.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide more context in this comment:

  • Bridge between what and what?
  • What texture copy request?

Providing some high-level explanation of how this class fits into the overall flow of passing textures back and forth wit the engine would be very helpful.

*/
@interface FLEExternalTextureGL : NSObject

/**
* Initializes a texture adapter with |texture| return a FLEExternalTextureGL.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having trouble parsing this comment; what is returning an FLEExternalTextureGL here?

*/
- (nonnull instancetype)initWithFLETexture:(nonnull id<FLETexture>)texture;

/**
* Accept texture buffer copy request from the flutter engine.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add more explanation to this comment:

  • What is the texture being populated from?
  • What is the meaning of the return value?

*/
- (BOOL)populateTextureWithWidth:(size_t)width
height:(size_t)height
texture:(nonnull FlutterOpenGLTexture*)texture;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The uses of the word "texture" in this class (header and implementation) need disambiguation. The class itself is a texture, and accepts two different objects that are also textures. In this method, kind of texture is the parameter? (The type provides the answer here, but not at a call site).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking more closely at the implementation, it looks like the texture being populated is the texture that is the last parameter; the current naming makes it sounds like some other internal texture is being populated from these arguments. As a general rule, you should never be naming the same parameter twice in the signature.

So unless I'm misunderstanding, this should be populateTexture:withWidth:height:.


/**
* The texture ID.
*/
- (int64_t)textureID;

@end
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#import "flutter/shell/platform/darwin/macos/framework/Source/FLEExternalTextureGL.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're moving away from using FLE as a prefix, and this code isn't part of an unstable public API surface, you can use Flutter rather than FLE as the prefix for this class, so that it doesn't need to be renamed later.


#import <AppKit/AppKit.h>
#import <CoreVideo/CoreVideo.h>
#import <OpenGL/gl.h>

@implementation FLEExternalTextureGL {
CVOpenGLTextureCacheRef _textureCache;
CVPixelBufferRef _pixelBuffer;
id<FLETexture> _texture;
}

- (instancetype)initWithFLETexture:(id<FLETexture>)texture {
self = [super init];
if (self) {
_texture = texture;
_pixelBuffer = nil;
_textureCache = nil;
}
return self;
}

static void OnGLTextureRelease(CVPixelBufferRef pixelBuffer) {
CVPixelBufferRelease(pixelBuffer);
}

- (int64_t)textureID {
return reinterpret_cast<int64_t>(self);
}

- (BOOL)populateTextureWithWidth:(size_t)width
height:(size_t)height
texture:(FlutterOpenGLTexture*)texture {
if (!_texture) {
return NO;
}

// Copy image buffer from external texture.
_pixelBuffer = [_texture copyPixelBuffer:width height:height];

if (!_pixelBuffer) {
return NO;
}

// Create the texture cache if necessary.
if (!_textureCache) {
CGLContextObj context = [NSOpenGLContext currentContext].CGLContextObj;
CGLPixelFormatObj format = CGLGetPixelFormat(context);
if (CVOpenGLTextureCacheCreate(kCFAllocatorDefault, NULL, context, format, NULL,
&_textureCache) != kCVReturnSuccess) {
NSLog(@"Could not create texture cache.");
CVPixelBufferRelease(_pixelBuffer);
return NO;
}
}

// Try to clear the cache of OpenGL textures to save memory.
CVOpenGLTextureCacheFlush(_textureCache, 0);

CVOpenGLTextureRef openGLTexture = NULL;
if (CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCache, _pixelBuffer,
NULL, &openGLTexture) != kCVReturnSuccess) {
CVPixelBufferRelease(_pixelBuffer);
return NO;
}

texture->target = static_cast<uint32_t>(CVOpenGLTextureGetTarget(openGLTexture));
texture->name = static_cast<uint32_t>(CVOpenGLTextureGetName(openGLTexture));
texture->format = static_cast<uint32_t>(GL_RGBA8);
texture->destruction_callback = reinterpret_cast<VoidCallback>(&OnGLTextureRelease);
texture->user_data = openGLTexture;

CVPixelBufferRelease(_pixelBuffer);
return YES;
}

- (void)dealloc {
CVOpenGLTextureCacheRelease(_textureCache);
}

@end
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
#import "flutter/shell/platform/darwin/ios/framework/Headers/FlutterCodecs.h"
#import "flutter/shell/platform/darwin/macos/framework/Headers/FLEReshapeListener.h"
#import "flutter/shell/platform/darwin/macos/framework/Headers/FLEView.h"
#import "flutter/shell/platform/darwin/macos/framework/Source/FLEExternalTextureGL.h"
#import "flutter/shell/platform/darwin/macos/framework/Source/FLETextInputPlugin.h"
#import "flutter/shell/platform/embedder/embedder.h"

Expand Down Expand Up @@ -91,6 +92,14 @@ - (void)dispatchMouseEvent:(nonnull NSEvent*)event phase:(FlutterPointerPhase)ph
*/
- (void)dispatchKeyEvent:(NSEvent*)event ofType:(NSString*)type;

/**
* Forwarding texture copy request to the corresponding texture via |textureID|,
* |width| and |height| may be used to get a specific sized texture in the future.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to comment on the meaning of width/height, since this is just forwarding. (Specifically it seems like this comment is talking about implementation details that are in a different class.)

*/
- (BOOL)populateTextureWithIdentifier:(int64_t)textureID
width:(size_t)width
height:(size_t)height
texture:(FlutterOpenGLTexture*)texture;
@end

#pragma mark - Static methods provided to engine configuration
Expand Down Expand Up @@ -142,6 +151,20 @@ static bool OnMakeResourceCurrent(FLEViewController* controller) {
return true;
}

/**
* Dispatching the texture copy request forward to the controller.
*/
static bool OnAcquireExternalTexture(FLEViewController* controller,
int64_t texture_identifier,
size_t width,
size_t height,
FlutterOpenGLTexture* texture) {
return [controller populateTextureWithIdentifier:texture_identifier
width:width
height:height
texture:texture];
}

#pragma mark Static methods provided for headless engine configuration

static bool HeadlessOnMakeCurrent(FLEViewController* controller) {
Expand Down Expand Up @@ -181,6 +204,9 @@ @implementation FLEViewController {
// A message channel for passing key events to the Flutter engine. This should be replaced with
// an embedding API; see Issue #47.
FlutterBasicMessageChannel* _keyEventChannel;

// A mapping of textureID to internal FLEExternalTextureGL adapter.
NSMutableDictionary<NSNumber*, FLEExternalTextureGL*>* _textures;
}

@dynamic view;
Expand All @@ -191,6 +217,7 @@ @implementation FLEViewController {
static void CommonInit(FLEViewController* controller) {
controller->_messageHandlers = [[NSMutableDictionary alloc] init];
controller->_additionalKeyResponders = [[NSMutableOrderedSet alloc] init];
controller->_textures = [[NSMutableDictionary alloc] init];
}

- (instancetype)initWithCoder:(NSCoder*)coder {
Expand Down Expand Up @@ -374,7 +401,9 @@ + (FlutterRendererConfig)createRenderConfigHeadless:(BOOL)headless {
.open_gl.clear_current = (BoolCallback)OnClearCurrent,
.open_gl.present = (BoolCallback)OnPresent,
.open_gl.fbo_callback = (UIntCallback)OnFBO,
.open_gl.make_resource_current = (BoolCallback)OnMakeResourceCurrent};
.open_gl.make_resource_current = (BoolCallback)OnMakeResourceCurrent,
.open_gl.gl_external_texture_frame_callback =
(TextureFrameCallback)OnAcquireExternalTexture};
return config;
}
}
Expand All @@ -389,6 +418,30 @@ - (void)makeResourceContextCurrent {
[_resourceContext makeCurrentContext];
}

- (BOOL)populateTextureWithIdentifier:(int64_t)textureID
width:(size_t)width
height:(size_t)height
texture:(FlutterOpenGLTexture*)texture {
return [_textures[@(textureID)] populateTextureWithWidth:width height:height texture:texture];
}

- (int64_t)registerTexture:(id<FLETexture>)texture {
FLEExternalTextureGL* fleTexture = [[FLEExternalTextureGL alloc] initWithFLETexture:texture];
int64_t textureID = [fleTexture textureID];
FlutterEngineRegisterExternalTexture(_engine, textureID);
_textures[@(textureID)] = fleTexture;
return textureID;
}

- (void)textureFrameAvailable:(int64_t)textureID {
FlutterEngineMarkExternalTextureFrameAvailable(_engine, textureID);
}

- (void)unregisterTexture:(int64_t)textureID {
FlutterEngineUnregisterExternalTexture(_engine, textureID);
[_textures removeObjectForKey:@(textureID)];
}

- (void)handlePlatformMessage:(const FlutterPlatformMessage*)message {
NSData* messageData = [NSData dataWithBytesNoCopy:(void*)message->message
length:message->message_size
Expand Down Expand Up @@ -522,6 +575,10 @@ - (void)setMessageHandlerOnChannel:(nonnull NSString*)channel
return self;
}

- (id<FLETextureRegistrar>)textures {
return self;
}

- (void)addMethodCallDelegate:(nonnull id<FLEPlugin>)delegate
channel:(nonnull FlutterMethodChannel*)channel {
[channel setMethodCallHandler:^(FlutterMethodCall* call, FlutterResult result) {
Expand Down