fengalin / media-toc Goto Github PK

Enhance waveform rendering by drawing marks at meaningful time positions and a line at amplitude 0

Thanks to the double buffering and incremental update mechanisms, we can now spend more time rendering the waveform. This could be an occasion to revert to displaying the channels separately, possibly using transparency, as one of the issues was that channels overlapped.

When previewing the media, it is sometimes necessary to read multiple frames to use the waveforms display efficiently.

Not only is it necessary to drive the media::Context in order to stop the preview, frames' data must also be kept in the controller for drawing / redrawing.

The requirements for each media controller are different, so a controller might need to keep frames' data eventhough its requirements are already fulfilled. For instance, the VideoController may already have received enough data to display, but the AudioController might need more samples to show a significant amount of data. Since, the packets will be parsed by the media::Context, we should get and keep video frames in the process.

This processing lays the basis for next step: buffering frames for playback / seek.

Integrate GStreamer instead of ffmpeg and see which one is more appropriate.

GStreamer comes with a ready to use way (plugin?) to draw to GtkDrawingArea as shown here .

There is also a dedicated structs for Table Of Content.

There are two Rust bindings:

• arturoc/gstreamer1.0-rs comes with documentation but seems to be handwritten. It doesn't seem to include GstToc.
• sdroege/gstreamer-rs has no documentation (but comes with examples) but seems to be generated: see sdroege/gstreamer-sys. Apparently, gstreamer-sys includes GstToc.

I think sdroege/gstreamer-rs is the way to go, even though it may be less mature than arturoc/gstreamer1.0-rs. Anyway, current requirements for Media-TOC are pretty main stream.

The table widget should display the table of contents of medias which support this feature.

The table could use a separate window in order to leave real estate for the video and audio visualizations.

Allow seeking in media from the time scale and waveform.

The following sequence exhibits incorrect waveform rendering:

1. Play a media.
2. Pause.
3. Zoom in to the maximum resolution possible.
4. Seek forward, then backward in window several times.

Remains from previous waveform images appear. After a window resize, gaps also appear on the waveform image.

Caps may change during playback. See this discussion.

With some files (e.g. sample.mkv - Opus audio codec and some Ogg audio files), the audio buffer are received after the playback. The waveform presentation is not capable of rendering the actual samples being played.

Try adding a tee with a dedicated queue for samples buffers.
If that's not enough, try adjusting the pipeline buffering.

When the application is launched, the video drawing area is painted with the application background. One a regular light theme, this is not consistent with the audio drawing area which is filled with the dark theme background at the moment.

Use black as a background as it is the colour that will be used by the video widget.

There are several possible enhancements:

• Audio processing: a tee can be added to split the audio processing in two queues (which also implies 2 threads). One will handle audio playback just as right now, the other will be in charge of pre-processing the samples to be ready for waveform rendering.
• Video sink construction: it seems that the way the sink is constructed right now is not really thread safe and may not work on other platforms. See this discussion.
• Caps might change during playback, so this should be monitored. See this discussion.

I suspect the tracker (based on gtk::timeout_add) to not being triggered as steadily as one could wish.

gtk_widget_add_tick_callback might help getting a smoother waveform rendering.

The following sequence produces a panic:

1. Open a media with a chapter list.
2. Select the second chapter.
3. Click play.

Message:

thread '<unnamed>' panicked at 'Cairo error "invalid matrix (not invertible)"', /home/francois/.cargo/git/checkouts/cairo-e6055a6d391c3fc9/911c1fe/cairo-sys-rs/src/enums.rs:75:12

Timestamp needs rework in order to use NaiveDateTime properly. NaiveTime is already included in NaiveDateTime, so there should be no reason to handle 2 attributes.

The waveform drawing area uses colours that way not be suitable for certain themes. At the very least, the background should be forced to very dark in order for the waveform and position cursor to be readable in the regular theme.

The tracker and listener continue executing after eos which causes CPU usage.
Note that, they should be reactivated in case of a seek.

Position (time ATM) is queried by the UI on a regular basis. It seems like the query hangs for a variable time, probably waiting to acquire a lock on the pipeline.

This enhancement is an attempt to avoid hanging the UI by updating the position / time in a separate thread.

Displaying the waveforms on a single graph results in a clumsy representation. The first channels rendered are hidden by the last ones if they have larger amplitude.

I don't feel that the channel information is pertinent to the definition of a chapter start or end position. Drawing a mono waveform would reduce the sample buffer, which in turn would reduce the computation needed to draw the waveform and the time spent draining the buffer.

For aesthetic reasons, we could keep the information of which channel has the highest amplitude before mixing down, and draw the sample using a colour matching this channel. It might also be too clumsy though as that could result in a sort of colour patchwork.

When media is paused and the widget is enlarged, the waveform is drawn but the amplitude doesn't follow the window's height.

The application raised:
panicked at 'assertion failed: first >= self.samples_offset', src/media/audio_buffer.rs:158:8

After enlarging / reducing the application window while playing a music file. I think the problem might have appeared after a buffer drain.

Not able to reproduce after. However, the condition should be ease to handle.

Current implementation defines the sample step from a target duration for the full range of the widget (its width). This presents the following drawbacks:

• When the window is resized, the sample step may change, which is counter-intuitive.
• When the zoom level is high, the cursor position is erratic and the samples window displayed might not use the full range of the widget (easily visible after an in-window seek in paused state).

It would be better to define zoom steps as say samples / 1000 px and compute the sample window accordingly.

Current implementation of audio sample drops induces discontinuities while drawing waveforms.
It also takes time and tends to lag the process.
One lead to follow would be to drop samples when no buffer is received, if that happens...

Resizing the application in paused mode makes the waveform shift in an unexpected way and the cursor remain centred.

Some videos start with a black frame, which is not particularly suitable for a preview. In the case of videos, seek within the media file (e.g. 10%) to get the preview frame.

In paused mode, seeking doesn't always show samples even though they are available in the AudioBuffer.

Allow user to click on the +/- button to zoom in the time window around current position.

Assemble the main technology bricks:

• Initialize the UI
• Load a video
• Display a frame and wave form

Media can be discovered before building the pipeline using GstDiscoverer as suggested by @sdroege. See this tutorial.

This could allow displaying (some) media info before the actual pipeline is built and choose the audio stream to connect when multiple audio streams are present.

In WaveformBuffer, the sample range calculation tries to guess the best range for the target image. This is inherited from the time when only playback/pause was possible and the AudioBuffer would received samples to be appended at the end until eos.

Now that seeking and zooming are implemented, a lot of complex situations are handled and a distinction between an image extension following the arrival of new samples or a refresh in paused mode after a seek or a zoom was introduced. Most of the complexity about positioning the first visible sample and the cursor needs to take place when the image is drawn in the drawing area.

It should be possible to simplify the sample range calculation by selecting the range around current sample with margin to handle in window centred seeks. This is the approach used in AudioBuffer::refresh and it could probably be generalized to extract_samples too.

There must be a crate for this.

In paused mode, it makes sense to zoom to the sample level. However in playing mode, the waveform is scrolling to fast for a human observer and for the double buffering mechanism stability.

In undetermined conditions, resuming playback is sometimes not possible after pausing the media.

It is straight forward for audio and video playback.
Waveform display needs more attention:

• Set a max size for the VecDeque of audio buffers (max. nb of samples).
• Drop buffers at the begin only for the moment (will need to be properly handled when seek will be implemented).

While seeking in MVI_1790.AVI, the WaveformImage sample_step is sometimes 0:

thread 'main' panicked at 'attempt to divide by zero', src/ui/waveform_buffer.rs:171:35

It can also appear at line 265 in WaveformImage.

This should reduce computation needs (especially during waveform drawing)

How to reproduce:

1. Open and play an audio file.
2. In the same session, open a video file and play.

After a while, the waveform rendering will start skipping chunks and high CPU usage is observed.

See https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gstreamer-audio.html
And recent evolutions on https://github.com/sdroege/gstreamer-rs

During playback, the listener in the GTK thread is in charge of checking for incoming audio buffer and triggering a waveform redraw when the pipeline position has changed.

The timing for both features might be different. For example, we probably want to drain the buffers as early as possible in order to avoid to pull many of them at once, while redrawing the waveform could occur at a rate of 24 or less per second.

Add hidden columns to chapters list with start and end as nano seconds in order to ease searching and selecting in a single iteration.

On a Windows 64bits box, the application crashes upon media selection in Context on line 177:

 let audio_sink = gst::ElementFactory::make("autoaudiosink", "audio_playback_sink").unwrap();

Waveform rendering sometimes exhibits garbage when zooming out or seeking. Sometimes the end of the waveform doesn't seem to correspond to the actual samples at these positions, sometimes there are gaps between sample chunks.

Build a plugin to handle double buffering and waveform rendering. This should make it easier to control the position using a dedicated clock.

The drawing area could be added to the UI the same way the video sink widget is added to the UI.

Cursor position is erratic when zoomed in in paused mode:

1. Play a media, wait for the cursor to be centred.
2. Pause the stream.
3. Zoom in / out.

The cursor will be rendered at different positions when it should remain at the same position.

For media with multiple audio or video stream, it should be possible to select the stream to be used.

With some files (e.g. sample.mk), the waveform flickers when the mouse hovers over the play/pause button.

Stream position should be visible on the following widgets:

• Chapter ListView: select current chapter if any
• Playback control Scale: set position / duration. It should also be possible to add marks at each chapter's start (and stop if it doesn't correspond to the next chapter's start or end of stream).

Big frames should be resized during conversion to RGB. This would save memory and avoid resizing every time the drawing areas are redrawn.