ProjectM Freezes/Deadlocks with Certain Presets

[stevenswart]

22nd of August 2020

I am reporting four separate issues I am experiencing with ProjectM. I started having problems when I installed the massive community preset pack available on the ProjectM GitHub home page. Previously, I was not experiencing any problems. It seems that there are certain presets in there that cause problems. In some cases, I was able to determine a few of these that were causing problems, and reproduce the errors. In other cases, I was not able to do so.

This is the second of these issues – certain presets that cause ProjectM to freeze/deadlock. (See the attached Freezing.Presets.zip.)

I did some fairly extensive investigations into these problems in June this year, and was able to get as far as discovering that there are four separate issues, according to backtraces that I obtained with GDB.

My use case is that I consider myself to be a Kodi power user. I use Kodi to play music on my media server most of the time that I am home. I have the ProjectM addon installed for Kodi, and I love it. I only use the ProjectM PulseAudio executable when, for example, I am streaming music via a browser. But I was able to determine that these problems originate in the ProjectM library itself, not within Kodi, and they seem to affect both Kodi with the ProjectM visualiser installed, as well as the ProjectM PulseAudio executable.

The problems fall into two classes, crashes (three problems found), and freezes/deadlocks (one problem found). As of this writing, with the latest available software, both these problems still exist.

I love the new presets I have installed, and even though I installed them over two months ago, I am still surprised by the occasional preset that I have never seen before. I have been collecting presets for a long time now, but some of the presets in the community pack are stunning!

Nevertheless, it is annoying when my music occasionally stops and I have to kill/restart Kodi and try to figure out where it was before it crashed. Most annoying when I am listening to DJ sets of an hour or longer!

Kodi reports that I now have 40585 presets installed, so, as you can imagine, it would be an extremely painful exercise for me to manually test every single preset individually and remove the ones that cause problems, certainly, I don’t have that kind of time to spend. Also, I imagine that all of these presets used to work at some point in the Milkdrop/ProjectM lifecycle, and ideally, this should be fixed inside the ProjectM library itself, or in cases where there is a bad preset that does not parse, the exception it causes should be handled, and ProjectM should simply move on to the next preset, possibly adding the faulty preset to some kind of banned list. That would be a far more robust way of handling the problem, to my mind.

Hardware:

My media server is based on an HP Proliant MicroServer, Gen10 (WW Entry BTO Model), specs here:

https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=a00008701enw&doctype=quickspecs&doclang=EN_US&searchquery=&cc=us&lc=en

Briefly, the hardware is as follows:

CPU: AMD Opteron(TM) X3216 APU - Dual-core, x86_64
Memory-Main Memory-phys_mem: 8053063680
Monitor: LG TV
Network Cards: NC332i Adapter x2, RTL8192EE PCIe Wireless Network Adapter
Graphics Card: ATI Radeon R5 Graphics
Sound Card: Kabini HDMI/DP Audio (built into graphics card).

The hardware info as reported by YaST may be found here:

https://paste.kodi.tv/lilaqabiqe.kodi

Sample Xorg.0.log may be found here:

https://paste.kodi.tv/serijohada

I am experiencing this problem on OpenSUSE Tumbleweed, latest version, updated today (22nd of August 2020), with Kodi Leia 18.8, installed from the packman repo. (Tumbleweed is OpenSUSE’s rolling release.)

However, I don’t think this is a problem with either the hardware or the operating system.

Sample ProjectM PulseAudio backtrace:

https://paste.kodi.tv/ubecosajew

Sample Kodi backtrace:

https://paste.kodi.tv/texoniquha.kodi

Sample Kodi log:

https://paste.kodi.tv/agasapuyut.kodi

Freezing.Presets.zip

[revmischa]

Thank you for the detailed report.
Looks from the backtrace like the crash is at

#0  0x00007f1e18cd9038 in memset (__len=512, __ch=0, __dest=0x7ffc8eba1340) at /usr/include/bits/string_fortified.h:71
#1  Parser::parseToken (fs=..., string=0x7ffc8eba1340 "") at Parser.cpp:79
#2  0x00007f1e18cd97f8 in Parser::parse_top_comment (fs=...) at Parser.cpp:302
#3  0x00007f1e18ce938d in MilkdropPreset::readIn (this=this@entry=0x50ac100, fs=...) at MilkdropPreset.cpp:467
#4  0x00007f1e18cea9b6 in MilkdropPreset::loadPresetFile (this=0x50ac100, 
    pathname="/usr/share/kodi/addons/visualization.projectm/resources/projectM/presets/presets_bltc201/file_03.milk") at MilkdropPreset.cpp:522
#5  0x00007f1e18ceaaed in MilkdropPreset::initialize (this=this@entry=0x50ac100, 
    pathname="/usr/share/kodi/addons/visualization.projectm/resources/projectM/presets/presets_bltc201/file_03.milk") at MilkdropPreset.cpp:306

Hopefully someone can take a look into this.

[milkdropper]

Hi stevenswart. Thanks for reporting this. The biggest problem with the community preset pack is that here is no quality control. In SDL there is an option to TEST_ALL_PRESETS (by modifying a macro on build, or manually modifying the source). I tested and found 1,000's of presets that don't work or that are duplicates.

Three of the presets you supplied do not have a header comment preset name, which is common in almost all presets. This is typically [preset00]. I found there is a bug in parse_top_comment() where projectM will get stuck in an infinite loop if this header comment can't be found. Once I fixed this infinite loop, it would stop the deadlock. But all three would be a black screen with no visualizations.

I tested the presets with milkdrop and found that 2 / 3 also produce a black screen with no visualization. I think they are incomplete or invalid presets. But one of the presets does work if you manually add [preset00]. Since the preset comment name is not needed by milkdrop or by projectM, we can probably just skip this requirement.

[stevenswart]

Hi milkdropper!

Have been thinking about your comment. It's probably not realistic to implement QC for every preset out there. In my collection, I do have a few presets that just give a black screen. Not a big issue for me.

My main issue is the crashes and freezes. Just when the music is getting good and I am grooving, bang!

So, ideally what I would like out of this, and I am sure most ProjectM users will agree, is a robust parser that won't either crash or deadlock.

I do not have much experience with parsers or graphics applications, but in my time writing server applications, a crash is absolutely the last thing you want. I am not sure how complex it would be to do this for ProjectM and all possible presets.

A relatively simple technique I have used for my work is to catch and handle every exception you can think of, and as a final measure, put a generic exception handler around the main execution loop, that would trap and log any exceptions (with stacktraces) that were not caught by any of the other exception handling code, before it exits. (Ideally logging the current preset as well.) That way, if you have a problem, at least you have some sort of clue as to the cause.

You would be surprised at how, over time, one catches strange and unpredictable exceptions that one would never have dreamed of happening, using this technique. And at how much more robust your app becomes by doing this diligently, over time.

[stevenswart]

Another thought I had - back in the day when I was still working in C++ regularly, we had deadlock detection code in the app. What would happen is that if a deadlock was detected, the app would coredump and exit. Not sure if this would work for simple infinite loops.

Something similar to what is described in this article:

https://www.drdobbs.com/detecting-deadlocks-in-c-using-a-locks-m/184416644

Perhaps you could find some similar FOSS code somewhere on the interwebs that you could adapt and use in ProjectM.

[revmischa]

Theoretically these presets all work fine with Milkdrop, so something is broken with projectM. If we can make a list of presets that are broken, we can dig deeper into why and hopefully fix the issues for all time. See #118

[stevenswart]

Hi revmischa!

I have been thinking about your comment.

It would, of course, be extremely cool if ProjectM were backwards-compatible with every version of Milkdrop and ProjectM, and if every preset that ever worked would still work with the latest version of ProjectM.

However, given that people are still writing presets, and probably always will, it would still be possible to write a non-conformant preset, or a preset that doesn't do anything.

For me personally, this is a separate issue from the stability issues. And, I would guess that the stability issues would be much quicker and easier to fix.

Further, in my career, stability problems have always had a higher priority to fix than adding cool new features, and in my use case with Kodi, getting ProjectM stable is far more important to me personally.

[revmischa]

It is certainly possible to write a broken preset that doesn't work in projectM or milkdrop but I'm assuming most of the ones that are out there are working fine at least in Milkdrop. I consider making Milkdrop presets work with projectM properly a high priority and can be considered stability depending on your definition.

[stevenswart]

Well, it's possible that the issues are related. I am not familiar with the code.

But as I have said before, the occasional black screen, for me, is not a problem, but when the music stops... ;-)

[stevenswart]

I would like to convey my heartfelt gratitude to the ProjectM team!

On Wednesday night I did a distribution update on my media server. I didn't notice it at first, but part of that update must've been a new version of libprojectM. (The .so version I currently have installed is v 3.1.1.) I had a look in my home directory, it seems that there have been no Kodi crash logs since that update, and I also don't recall having any more freezes/deadlocks since then either.

I have been playing music all day with no interruptions! :-)

I am not sure what you guys did, but it seems to have fixed the problem on my side.

Thank you, all! You have removed a major annoyance from my life! ;-)

[stevenswart]

2nd of November 2020

Hi again, all!

It seems I was mistaken, shortly after I posted my previous comment, I started experiencing crashes and freezes/deadlocks again. I suppose, due to the random nature of the preset select, a two-day run without any problems is possible.

Two weekends ago, I did some more investigations. It seems things have changed somewhat.

The created date I currently have for my libprojectM.so.3.1.1 is Wednesday, September 16, 2020 7:43:03 PM SAST. According to zypper, this is the version of the library I now have installed:

steven@linux-ll3r:~> sudo zypper info projectm
Loading repository data...
Reading installed packages...

Information for package projectM:
---------------------------------
Repository     : openSUSE:Factory
Name           : projectM
Version        : 3.1.3-2.4
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 368.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : projectM-3.1.3-2.4.src
Summary        : A Music Visualizer
Description    : 
    projectM is a music visualizer.

I have a version of Kodi 18.8 that I cloned and built on Saturday, October 17, 2020, which I had to do as a first step to rebuild the Kodi ProjectM visualiser addon.

I also discovered something I didn’t know; when running the kodi-x11 binary directly from a terminal, without using a shell-script, it seems to log the presets as they are loaded to stdout or stderr, which is useful. So, now when I have a problem, I can identify and remove the problem preset. So, I am slowly getting my preset collection pruned down to those that work. For anyone else who is having the same problem, take a tip!

I retested the faulty presets for both crashes and freezes/deadlocks, the same presets are still causing problems, although from the backtraces it looks like the internals have changed considerably. Ignore the visualization.projectm.so.2.2.0, in the backtrace that’s a symlink to the latest version. See attached Kodi.Crash.Parser.Backtrace.txt and Kodi.Freeze.Backtrace.file_cohere2.milk.txt.

Kodi.Crash.Parser.Backtrace.txt

Kodi.Freeze.Backtrace.file_cohere2.milk.txt

[capellavacancies]

what about deleting these presets from the folder ?
I've got the same problem, I don't know how to solve it.
using projectm on windows 10 as a visualizations for vlc

[stevenswart]

Good day, capellavacancies!

There is only one way, for now, that I know how to solve it. I am running ProjectM from the Kodi visualisation addon on Linux. I run the Kodi binary from the command-line in a Linux terminal. I noted that, as presets are loaded, it outputs the current preset to the terminal. Every time I have a crash or freeze, I just move the faulty preset into another directory. I hope, that way, to eventually locate and remove all of the problem presets. I have only been doing this for three weeks, but I hope to eventually compile a list of all these presets, and report them here.

The only thing I can suggest, in the meantime, is that you try to do something similar. It's been many years since I last used Windows, but if you try running the VLC .exe from a Windows shell (cmd window), and not the start menu icon, you might find that the same thing happens - you may see a log output in your Windows terminal. Then you can use the same procedure as I do. I have not tried this, however, it may or may not work.

[stevenswart]

About a month ago, I started moving problem presets out of my preset working folder. I am uploading my latest collection of freezing/deadlocking presets here. All of these cause ProjectM freezes. Please see attached freezes.presets.zip.

I am also going to list the filenames below, for anyone else who has the Community Preset Pack installed. This list is probably not complete, and I will probably update it later. Due to the random nature of the preset selection, ProjectM may run for months or years without ever loading a problem preset. But for now, this should solve a big part of the problem.

N.O. phathmolesc.milk
Zylot - Paint Spill (Music Reactive Paint Mix) nz+ infinite broadening of quantum.milk
file_03.milk
file_04.milk
file_05.milk
file_09.milk
file_Custom Wave 1.milk
file_aural3.milk
file_aurals.milk
file_auralw.milk
file_auralw2neg.milk
file_auralwneg.milk
file_ball.milk
file_bird.milk
file_boatsh.milk
file_boatwa1.milk
file_boatwa2.milk
file_camel2.milk
file_cohere1.milk
file_cohere2.milk
file_dotef1.milk
file_dotef2.milk
file_n7772.milk
file_n7774.milk
file_phar.milk
file_pharm.milk
file_roosta1.milk
file_shifter's word wave.milk
file_stal1.milk
file_stal3.milk
file_stal4.milk
file_tlx1.milk
file_tlx2.milk
file_wroam2.milk
file_wstatics1.milk
martin - mandelbox explorer - wreck diver.milk
suksma - fiShbRaiN - witchcraft (phugly shame on remix) - darning kneedless - hot vessel cadavar lexical decoherence.milk
xtramartin (87).milk

freezes.presets.zip

[revmischa]

I loaded up these presets and debugged that Parser::parse_top_comment function. Turns out the parser can freeze or crash if a preset file does not contain the standard [preset00] header. If no [ is found in the file it scans until the buffer is full (which results in an infinite loop) or it overflows something and blows up.
I changed the parser to just proceed normally if there is no header. We don't do anything with the header anyway.

[revmischa]

Fixed in #450