Setting VideoCapture Position Not Working

[RishabhK12]

Trying to extract every 10 frames from a video to then be run through a neural network using the tflite flutter package. Everything works but the detections displayed are in the wrong spot because of the inference time. They get delayed and are displayed late because the video moves on and it keeps getting further back. Trying to set the VideoCaptures position doesn't work.

Using this to extract frames:

double frameNum = 0;
final videoCapture = cv.VideoCapture.create(videoPath);
_videoController!.addListener(() {
            var result = videoCapture.read();
            cv.Mat frame = result.$2;  
              onLatestImageAvailable(frame);
              setState(() {});
            frameNum += 10;
            videoCapture.set(cv.CAP_PROP_POS_FRAMES, frameNum);
            print('frame count: ' + videoCapture.get(cv.CAP_PROP_POS_FRAMES).toString());
          });

Every 10 frames are extracted and sent through the onLatestImageAvailable method to run a neural network on.

Printing out the value it thinks it is at just produces 0.0. I have tried using CAP_PROP_POS_MSEC and CAP_PROP_POS_AVI_RATIO. This works fine in using opencv in python but trying to incorporate it with a mobile app.

[rainyl]

1. Which version?
2. Do you mean that the videoCapture stays on the old frame after calling onLatestImageAvailable(frame);?
3. Based on the above question, do you mean that print('frame count: ' + videoCapture.get(cv.CAP_PROP_POS_FRAMES).toString()); always prints 0.0 even setted manually via videoCapture.set(cv.CAP_PROP_POS_FRAMES, frameNum);?

If so, I tested using the following code and it seems every thing is ok.

import 'dart:io';

import 'package:opencv_dart/opencv_dart.dart' as cv;

void main(List<String> args) {
  void onLatestImageAvailable(cv.Mat frame) {
    print("Working...");
    sleep(Duration(seconds: 1));
  }

  final videoCapture = cv.VideoCapture.create("test/images/small.mp4", apiPreference: cv.CAP_ANY);
  double frameNum = 0;
  const frameStep = 10;
  var (ret, frame) = videoCapture.read();
  final frameCount = videoCapture.get(cv.CAP_PROP_FRAME_COUNT);
  print("Frame Count: $frameCount");
  while (frameNum < frameCount) {
    frameNum += frameStep;
    videoCapture.read(m:frame);
    videoCapture.set(cv.CAP_PROP_POS_FRAMES, frameNum);
    onLatestImageAvailable(frame);
    print('frame pos: ${videoCapture.get(cv.CAP_PROP_POS_FRAMES)}');
  }
  print("Finished");
}

output

Frame Count: 166.0 Working... frame pos: 10.0 Working... frame pos: 20.0 Working... frame pos: 30.0 Working... frame pos: 40.0 Working... frame pos: 50.0 Working... frame pos: 60.0 Working... frame pos: 70.0 Working... frame pos: 80.0 Working... frame pos: 90.0 Working... frame pos: 100.0 Working... frame pos: 110.0 Working... frame pos: 120.0 Working... frame pos: 130.0 Working... frame pos: 140.0 Working... frame pos: 150.0 Working... frame pos: 160.0 Working... frame pos: 166.0 Finished

If onLatestImageAvailable is synchronous, theoretically, it should work as expected.

If the problem still exists, please provide a minimum reproducable example so I can try to fix it.

[RishabhK12]

I am running 1.1.0. The VideoCapture moves on but it moves on frame by frame when I use set and when I don't. When I print it out, it always prints to 0.0, same with the frame count and so the code you provided doesn't work. Below I am using a simple widget that allows the user to choose a video and it extracts frames from it.

import 'package:opencv_dart/opencv_dart.dart' as cv;

class VideoDetectorWidget extends StatefulWidget {
  const VideoDetectorWidget({super.key});

  @override
  State<VideoDetectorWidget> createState() => _VideoDetectorWidgetState();
}

class _VideoDetectorWidgetState extends State<VideoDetectorWidget> {
  @override
  void initState() {
    super.initState();
  }

  void onLatestImageAvailable(cv.Mat frame) {
    print("Working...");
    sleep(Duration(seconds: 1));
  }

  void _extractFrames() async {
    FilePickerResult? video = await FilePicker.platform.pickFiles(
      type: FileType.video,
    );
    String? videoPath = video?.paths.first;
    if (videoPath != null) {
      final videoCapture = cv.VideoCapture.create(videoPath, apiPreference: cv.CAP_ANY);
      double frameNum = 0;
      const frameStep = 10;
      var (ret, frame) = videoCapture.read();
      final frameCount = videoCapture.get(cv.CAP_PROP_FRAME_COUNT);
      print("Frame Count: $frameCount");
      while (frameNum < frameCount) {
        frameNum += frameStep;
        videoCapture.read(m: frame);
        videoCapture.set(cv.CAP_PROP_POS_FRAMES, frameNum);
        onLatestImageAvailable(frame);
        print('frame pos: ${videoCapture.get(cv.CAP_PROP_POS_FRAMES)}');
      }
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Frame Extraction'),
      ),
      body: Center(
        child: ElevatedButton(
          onPressed: _extractFrames,
          child: Text('Extract Frames'),
        ),
      ),
    );
  }
}

[rainyl]

@RishabhK12 Confirmed, if you are running on windows or linux, would you please refer to #160 to setup it manually and check whether it is solved?

If you are not running on the above platforms, I have also tested with your code and it also works on windows.

Code

import 'dart:io';

import 'package:flutter/material.dart';
import 'package:opencv_dart/opencv_dart.dart' as cv;
import 'package:file_picker/file_picker.dart';


void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        // This is the theme of your application.
        //
        // TRY THIS: Try running your application with "flutter run". You'll see
        // the application has a purple toolbar. Then, without quitting the app,
        // try changing the seedColor in the colorScheme below to Colors.green
        // and then invoke "hot reload" (save your changes or press the "hot
        // reload" button in a Flutter-supported IDE, or press "r" if you used
        // the command line to start the app).
        //
        // Notice that the counter didn't reset back to zero; the application
        // state is not lost during the reload. To reset the state, use hot
        // restart instead.
        //
        // This works for code too, not just values: Most code changes can be
        // tested with just a hot reload.
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const VideoDetectorWidget(),
    );
  }
}


class VideoDetectorWidget extends StatefulWidget {
  const VideoDetectorWidget({super.key});

  @override
  State<VideoDetectorWidget> createState() => _VideoDetectorWidgetState();
}

class _VideoDetectorWidgetState extends State<VideoDetectorWidget> {
  @override
  void initState() {
    super.initState();
  }

  void onLatestImageAvailable(cv.Mat frame) {
    print("Working...");
    sleep(Duration(seconds: 1));
  }

  void _extractFrames() async {
    FilePickerResult? video = await FilePicker.platform.pickFiles(
      type: FileType.video,
    );
    String? videoPath = video?.paths.first;
    if (videoPath != null) {
      final videoCapture = cv.VideoCapture.create(videoPath, apiPreference: cv.CAP_ANY);
      double frameNum = 0;
      const frameStep = 10;
      var (ret, frame) = videoCapture.read();
      final frameCount = videoCapture.get(cv.CAP_PROP_FRAME_COUNT);
      print("Frame Count: $frameCount");
      while (frameNum < frameCount) {
        frameNum += frameStep;
        videoCapture.read(m: frame);
        videoCapture.set(cv.CAP_PROP_POS_FRAMES, frameNum);
        onLatestImageAvailable(frame);
        print('frame pos: ${videoCapture.get(cv.CAP_PROP_POS_FRAMES)}');
      }
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Frame Extraction'),
      ),
      body: Center(
        child: ElevatedButton(
          onPressed: _extractFrames,
          child: Text('Extract Frames'),
        ),
      ),
    );
  }
}

---

Output

Launching lib\main.dart on Windows in debug mode...
√  Built build\windows\x64\runner\Debug\flutter_application_1.exe.
Connecting to VM Service at ws://127.0.0.1:62389/K1X0WBvBRkY=/ws
flutter: Frame Count: 166.0
flutter: Working...
flutter: frame pos: 10.0
flutter: Working...
flutter: frame pos: 20.0
flutter: Working...
flutter: frame pos: 30.0
flutter: Working...
flutter: frame pos: 40.0
flutter: Working...
flutter: frame pos: 50.0
flutter: Working...
flutter: frame pos: 60.0
flutter: Working...
flutter: frame pos: 70.0
flutter: Working...
flutter: frame pos: 80.0
flutter: Working...
flutter: frame pos: 90.0
flutter: Working...
flutter: frame pos: 100.0
flutter: Working...
flutter: frame pos: 110.0
flutter: Working...
flutter: frame pos: 120.0

[rainyl]

A hotfix version v1.1.0+1 has been published, please upgrade and try again.

https://pub.dev/packages/opencv_dart/versions/1.1.0+1

[RishabhK12]

@rainyl Thanks for the response. I am running this on android (s23 ultra). The problem is still happening with the same code. I tried both solutions but neither worked. I have found some logs though.

E/cv::error()(15439): OpenCV(4.10.0) Error: Requested object was not found (could not open directory: /data/app/~~huLjj5np-Bk3fmxAWdsKEg==/com.example.lamp-rX3MyqcmJX5DGfbY0Jkrmg==/base.apk!/lib/arm64-v8a) in glob_rec, file /home/runner/work/opencv.full/opencv.full/build/opencv/modules/core/src/glob.cpp, line 279
E/cv::error()(15439): OpenCV(4.10.0) Error: Requested object was not found (could not open directory: /data/app/~~huLjj5np-Bk3fmxAWdsKEg==/com.example.lamp-rX3MyqcmJX5DGfbY0Jkrmg==/base.apk!/lib/arm64-v8a) in glob_rec, file /home/runner/work/opencv.full/opencv.full/build/opencv/modules/core/src/glob.cpp, line 279
E/cv::error()(15439): OpenCV(4.10.0) Error: Requested object was not found (could not open directory: /data/app/~~huLjj5np-Bk3fmxAWdsKEg==/com.example.lamp-rX3MyqcmJX5DGfbY0Jkrmg==/base.apk!/lib/arm64-v8a) in glob_rec, file /home/runner/work/opencv.full/opencv.full/build/opencv/modules/core/src/glob.cpp, line 279
I/flutter (15439): Frame Count: 0.0

[rainyl]

@RishabhK12 Finally I figured it out.

TLDR: OpenCV doesn't support it on android.
Releated Issues:

• cv::VideoCapture set(CAP_PROP_POS_FRAMES, target) does not work in android cpp opencv/opencv#19387
• Android Opencv VideoCapture property all return 0 opencv/opencv#15547

Details:

From the opencv source code (v4.10.0 https://github.com/opencv/opencv/blob/71d3237a093b60a27601c20e9ee6c3e52154e8b1/modules/videoio/src/cap_android_mediandk.cpp#L181-L213), we can see that only several props are supported on android mediandk, which is the default backend on android, and I didn't managed to compile opencv with FFMPEG on android so the mediandk is the only backend to process videos on android.

If you or other contributors can successfully compile opencv with FFMPEG (or gstreamer) backend on android, I am very willing to merge it.

Also, on android API > 29 (android 10), the prop CAP_PROP_FRAME_COUNT was supported, https://github.com/opencv/opencv/blob/71d3237a093b60a27601c20e9ee6c3e52154e8b1/modules/videoio/src/cap_android_mediandk.cpp#L264-L266 , however the opencv used in this project was compile with an API of 24 https://github.com/rainyl/opencv.full/blob/899bc6989070a3cd92eb6bce176aa0fc8e258a9e/profiles/android-armv8#L5 , so CAP_PROP_FRAME_COUNT is also not supported by this package, it's possible to recompile opencv with an API of 29 but I think it's unnecessary, for keeping compatibility.

Allright, now it's the solution: just use (bool, Mat) read(), the first return value indicates whether it's success, will return false if reach to the last frame, and use another loop to skip n steps (maybe I will add a parameter step to read() in the future). e.g.,

Code

import 'dart:io';

import 'package:flutter/material.dart';
import 'package:opencv_dart/opencv_dart.dart' as cv;
import 'package:file_picker/file_picker.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const VideoDetectorWidget(),
    );
  }
}

class VideoDetectorWidget extends StatefulWidget {
  const VideoDetectorWidget({super.key});

  @override
  State<VideoDetectorWidget> createState() => _VideoDetectorWidgetState();
}

class _VideoDetectorWidgetState extends State<VideoDetectorWidget> {
  @override
  void initState() {
    super.initState();
  }

  void onLatestImageAvailable(cv.Mat frame) {
    print("Processing Frame, rows: ${frame.rows}, cols: ${frame.cols}, channels: ${frame.channels}");
    sleep(Duration(seconds: 1));
  }

  void _extractFrames() async {
    FilePickerResult? video = await FilePicker.platform.pickFiles(
      type: FileType.video,
    );
    String? videoPath = video?.paths.first;
    if (videoPath != null) {
      final videoCapture = cv.VideoCapture.fromFile(videoPath);
      double frameNum = 0;
      const frameStep = 10;
      var (success, frame) = videoCapture.read();
      if(success) onLatestImageAvailable(frame);
      while (success) {
        // skip every `frameStep` frames
        for (var i = 0; success && i < frameStep; i++) {
          // mind to pass the `m` parameters or a new Mat will be created every time you call this method, then you may need to manually dispose them.
          (success, _) = videoCapture.read(m: frame);
          frameNum += 1;
        }
        print("frameNum: $frameNum");
        if(success) onLatestImageAvailable(frame);
      }
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Frame Extraction'),
      ),
      body: Center(
        child: ElevatedButton(
          onPressed: _extractFrames,
          child: Text('Extract Frames'),
        ),
      ),
    );
  }
}

And the output:

BTW, the supported video formats depend on mediandk, I didn't test it but you can refer to https://developer.android.com/media/platform/supported-formats#video-codecs

[rainyl]

So if no further problems, I am going to close this issue.

Feel free to reopen it if you still have problems.