Change plugin to use new GstGLBaseAudioVisualizer to handle GL context initialization #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tested this on OS X and am getting an error |
* change projectM plugin to use GstGLBaseAudioVisualizer * initialize glew * removed some code that did not seem to be required
|
@nzoschke It's ready for a retest on OS X when you get a chance |
|
Tried again still no luck and getting
More details: % ./test.sh --audio gl-handling
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayCocoa\)\ gldisplaycocoa0";
/GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
0:00:00.303738917 62428 0x600002e6a670 DEBUG projectm plugin.c:275:gst_projectm_setup:<projectm0> Audio Information <Channels: 1, SampleRate: 44100, Description: 16-bit signed PCM audio>
0:00:00.303749584 62428 0x600002e6a670 DEBUG projectm plugin.c:283:gst_projectm_setup:<projectm0> Video Information <Dimensions: 512x512, FPS: 60/1, Depth: 32bit, SamplesPerFrame: 1470>
/GstPipeline:pipeline0/GstProjectM:projectm0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage.GstGhostPad:sink.GstProxyPad:proxypad1: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLUploadElement:gluploadelement0.GstPad:src: caps = video/x-raw(memory:GLMemory), width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR, texture-target=(string)2D
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLColorConvertElement:glcolorconvertelement0.GstPad:src: caps = video/x-raw(memory:GLMemory), width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)RGBA, texture-target=(string)2D
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLColorBalance:glcolorbalance0.GstPad:src: caps = video/x-raw(memory:GLMemory), width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)RGBA, texture-target=(string)2D
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLImageSink:sink.GstPad:sink: caps = video/x-raw(memory:GLMemory), width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)RGBA, texture-target=(string)2D
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLColorBalance:glcolorbalance0.GstPad:sink: caps = video/x-raw(memory:GLMemory), width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)RGBA, texture-target=(string)2D
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLColorConvertElement:glcolorconvertelement0.GstPad:sink: caps = video/x-raw(memory:GLMemory), width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR, texture-target=(string)2D
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLUploadElement:gluploadelement0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage.GstGhostPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
0:00:00.305691334 62428 0x60000035dfe0 DEBUG projectm projectm.c:24:projectm_init:<projectm0> Creating projectM instance..
0:00:00.338407292 62428 0x60000035dfe0 DEBUG projectm projectm.c:34:projectm_init:<projectm0> Created projectM instance!
0:00:00.338429792 62428 0x60000035dfe0 INFO projectm projectm.c:63:projectm_init:<projectm0> Using Properties: preset=(null), texture-dir=(null), beat-sensitivity=1.000000, hard-cut-duration=3.000000, hard-cut-enabled=0, hard-cut-sensitivity=1.000000, soft-cut-duration=3.000000, preset-duration=0.000000, mesh-size=(48, 32)aspect-correction=1, easter-egg=0.000000, preset-locked=1,
/GstPipeline:pipeline0/GstProjectM:projectm0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
0:00:00.340095334 62428 0x600002e6a670 DEBUG projectm plugin.c:275:gst_projectm_setup:<projectm0> Audio Information <Channels: 1, SampleRate: 44100, Description: 16-bit signed PCM audio>
0:00:00.340101750 62428 0x600002e6a670 DEBUG projectm plugin.c:283:gst_projectm_setup:<projectm0> Video Information <Dimensions: 512x512, FPS: 60/1, Depth: 32bit, SamplesPerFrame: 1470>
/GstPipeline:pipeline0/GstProjectM:projectm0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
** (gst-launch-1.0:62428): ERROR **: 08:15:05.205: OpenGL Error: GL_INVALID_ENUM - Enumeration parameter is not legal
./test.sh: line 27: 62428 Trace/BPT trap: 5 GST_DEBUG=projectm:5 gst-launch-1.0 -v audiotestsrc ! queue ! audioconvert ! projectm ! "video/x-raw,width=512,height=512,framerate=60/1" ! videoconvert ! autovideosink sync=falseCrash report |
|
This is officially working on Windows 11! Nice! |
|
@nzoschke Thank you for testing it again! I will try to troubleshoot further, but I don't have a mac to really dig in unfortunately. |
|
No problem, looks like great progress on other platforms. Maybe we should split Mac out to its own issue... |
If you both are good with this, then we can go ahead and go this route! Just give this post a thumbs up if you agree, and I'll handle it when I get home |
|
@nzoschke I'm going to go ahead and push this through! If you'd do me a favor when you have time and create a new issue about this topic with all your logs and any relevant info about your system. |
|
Nice work all, I cleaned up the OS X issue here #1 and will keep hacking there. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I had some issues getting the plugin to work, mostly related to the GLContext management and threading. I took a look at the GStreamer plugins, and they use a quite complex logic to manage the GLContext. I used the GStreamer GstGLBaseSrc as a template to create a similar base plugin that takes care of handling the OpenGL context for a GstAudioVisualizer, and adjusted the projectM plugin implementation accordingly.
This works for me now, there are a few notes:
Video formats between OpenGL and GstVideoFrame show an interesting behavior when frame pixels are copied. Bytes are reversed (e.g. OpenGL ABGR seems to produce the correct result for GStreamer RGBA). This may be expected behavior and I just not know enough about GStreamer to explain it..
Tested on Windows 10 (VS 2022/GStreamer 1.24.3) and Linux (Ubuntu 22.04, GStreamer 1.20.3) with X11 and Wayland. Performance-wise it seems slower (in terms of CPU load) than frontend-SDL2 on Linux, but similar on Windows.