Q: Getting Line Breaks into class TimeCode about ly2video HOT 15 CLOSED

I think the answer here is to add control change (CC) messages to the midi file for bar, line, and page locations. According to what I've read, 80 to 84 are reserved for General Purpose use. So using mido:

mido.Message('control_change', control=80, time=tick)

Could be used to encode either of the 3 events. I don't know if the midi channel matters. I really just need line breaks.

It looks like any event will be handled by midi rubato properly. So before running midi rubato and before creating miditicks, it would be necessary to run a similar process to add in the line breaks as control messages. This feels ugly.

Better would be if midi rubato was run on miditicks after it had been created. Except that miditicks is no longer midi messages. I'm just rambling now.

Ideally, lilypond would add these directly to the midi. I had a look at the code, but it seems like some c++ code would need to be changed. The control messages are strictly defined so I don't think any scheme code can change them.

from ly2video.

Earlier I posted:

ly2video:    ( 12.1143247086614170, 13.4185367086614170) 0.00000
ly2video:    ( 14.8278751262453050, 16.1320871262453020) 0.25000
ly2video:    ( 17.5414255438291900, 18.8456375438291900) 0.50000
ly2video:    ( 20.2549759614130770, 21.5591879614130770) 0.75000
ly2video:    ( 23.9489604331111930, 25.2531724331111920) 1.00000
ly2videoBar: ( 22.8032482361398220, 22.9932482361398240) 1.00000

Those are quarter note beats, but there are 5 of them before the first bar. The way I interpret the above output is that the beat on moment 1.0000 is in the next bar given the x position. However, the bar and the beat can't share the same position in time. So having that information in the midi would be pointless. Or rather the time in the midi should reflect the distance between the last note and the next note given the bar's x position.

Prior to this 'discovery' I thought all I needed was the moment. That's no longer the case. I should have been paying more attention earlier on.

from ly2video.

I don't understand why you need to care about the timing of a bar grob?

from ly2video.

Because of animations. I think.

For Drumistry type videos, the transition from semi-transparent to opaque begins on beat 2. This requires the number of frames from beat 2 to the barline of the last bar on a line. There's not necessarily going to be a midi tick on beat 2, but it can be derived from the tempo working back from the barline at the next line break. I need to convert that time to a number of frames at a certain spot.

For Drumeo type videos, the page starts moving when the cursor passes the last note in the bar. The page transition seems to be 250ms. So instead of looking for a line break, it's necessary to find the last note before a line break. For this I just need the location in frames. 250ms will be X frames based on the fps.

from ly2video.

Can't you achieve all of this by considering an end-of-line barline and the first beat of the next bar to occur at exactly the same moment, and then all the rest should work by interpolation like it already does?

So for Drumistry, full opacity is reached exactly at the point in time of the end-of-line barline, at which point the next line appears and starts fading in.

For Drumeo the cursor instantaneously jumps from the barline to the start of the first note or rest on the next line. And if you want the scrolling to start 250ms before that transition, you can do that regardless of where the cursor is or where any notes are, so I don't understand why you say it's necessary to find the last note before a line break. 250ms before the end of the line might be many notes before the line break, but that's not relevant because the scrolling is controlled purely by time, and ly2video already knows how to map grob moments to the video timeline and interpolate between those moments.

from ly2video.

Can't you achieve all of this by considering an end-of-line barline and the first beat of the next bar to occur at exactly the same moment, and then all the rest should work by interpolation like it already does?

I don't see how this helps. ly2video doesn't count beats or bars. If the beat is not in midiTicks, ly2video doesn't know about it?

So for Drumistry, full opacity is reached exactly at the point in time of the end-of-line barline, at which point the next line appears and starts fading in.

You could have 1, 2, 3 or even more bars of music on a single line. That's many seconds of audio. The fade in should only start during the last bar of music prior to the line break. So it can't begin fading at the point which it appears.

You could start the transition when the next line appears, but then the transition to opaque won't take exactly 3 beats like I want.

For Drumeo the cursor instantaneously jumps from the barline to the start of the first note or rest on the next line.

Right. It can't do anything else. The start of a line could have many things (clef, key signature, time signature) and it only makes sense to bypass it all.

And if you want the scrolling to start 250ms before that transition, you can do that regardless of where the cursor is or where any notes are, so I don't understand why you say it's necessary to find the last note before a line break. 250ms before the end of the line might be many notes before the line break, but that's not relevant because the scrolling is controlled purely by time, and ly2video already knows how to map grob moments to the video timeline and interpolate between those moments.

Drumeo begins the scrolling on the last note. The scrolling lasts 250ms. This is different than beginning the scrolling at the bar line transition. Scrolling on the last note is a very nice touch. I like it and I want to do it.

Still how does one know it's 250ms before the bar line transition you spoke of?

from ly2video.

For Drumistry type videos, the transition from semi-transparent to opaque begins on beat 2.

For 4/4 songs, the transition to opaque begins at beat 2. On 12/8 songs, the transition begins on beat 4. So a more general formula for this might be:

transition time = 3 * 60/qpm

subtracted from the line break. It also means the transition will start

transition start = 1 * 60/qpm

after the correct bar. The issue here is that we don't have the information of when a bar starts. It's not encoded into the midi file. If it were encoded, it could end up in midiTicks. Actually the time will probably already be in midiTicks because of downbeats, but it's not correct to assume there will always be one.

Ultimately what I need are three pieces of information:

1. number of frames to bar line when there is a line break
2. number of frames to an animation point prior to a line break
3. otherwise number of frames to the next note

So let's assume that I collect all the moments for bars and breaks and I add them to midiTicks if missing. I can now iterate through the list and create animation points in ticks. For drumistry this is easy:

    animationPointsInTicks = {}
    for moment in breakMoments:
        tick = int(round(moment * midiResolution * 4))
        tick -= int(round(3 * midiResolution))
        animationPointsInTicks.append(tick)

For drumeo it's a little harder, but it just means searching notesInTicks. It's doable to find the animation point in ticks. Then add them to the TimeCode class somehow.

The problem is that I can't insert any points into midiTicks without breaking how it works. The TimeCode class just iterates through the midiTicks expecting that they are all notes. The obvious solution is to tag the midiTicks with a type.

What I want is a function like nbFramesToNextNote, perhaps called nbFramesToNext, that returns those three pieces of information.

I'll continue to think about this, but feel free to chime in with comments. I don't really have a great understanding of how everything works. Please challenge anything that sounds stupid.

from ly2video.

You wrote:

ly2video doesn't count beats or bars. If the beat is not in midiTicks, ly2video doesn't know about it?

and later:

The issue here is that we don't have the information of when a bar starts.

I'm confused why you wrote these, because you've already shown in the original issue description above that you were able to obtain Lilypond moments (i.e. timing information) for barlines:

ly2videoBar: ( 22.8032482361398220, 22.9932482361398240) 1.00000

With this technique you know exactly when the bars are. (If you needed it, you could also use this to figure out how many beats there are in any given bar, but I don't see why you should need that anyway.)

For 4/4 songs, the transition to opaque begins at beat 2. On 12/8 songs, the transition begins on beat 4.

That seems really illogical and inconsistent to me. I can't think of any good reason why it would sometimes start 25% of the way through the bar, and other times 75% through, and it seems even more arbitrary that this would be decided by a time signature. I think the most logical way to transition is based on a fixed wallclock duration. Otherwise it's not only inconsistent but also suffers from the problem that if the tempo is high, the transition could happen too quickly, and if it's low then the transition could happen too slowly. Shouldn't the main goal be to give the brain enough time to prepare for the switch to the next line?

from ly2video.

With this technique you know exactly when the bars are. (If you needed it, you could also use this to figure out how many beats there are in any given bar, but I don't see why you should need that anyway.)

ly2video doesn't currently retain that information. Like I said, those moments don't get into midiTicks.

For 4/4 songs, the transition to opaque begins at beat 2. On 12/8 songs, the transition begins on beat 4.

That seems really illogical and inconsistent to me. I can't think of any good reason why it would sometimes start 25% of the way through the bar, and other times 75% through

Erm, maybe I wasn't clear. Let's pretend we are at the last bar of the line. We have time A which represents beat 1 in absolute ticks. We have time B which represents beat 1 of the next line (ie. the end bar) in absolute ticks. If the time of one measure (midiResolution * 4) is C, then the starting tick is either A + 0.25*C or B - 0.75*C.

from ly2video.

I think the most logical way to transition is based on a fixed wallclock duration. Otherwise it's not only inconsistent but also suffers from the problem that if the tempo is high, the transition could happen too quickly, and if it's low then the transition could happen too slowly. Shouldn't the main goal be to give the brain enough time to prepare for the switch to the next line?

The line being played is replaced as soon as it's finished. That means the next line is available in a semi-transparent state immediately. I think the important cue is actually the moment when one line becomes fully opaque and the next line is a semi-opaque. That's a very strong beat 1 cue. The fact that it fades in is more like eye candy. Being sync'd to the music is more eye candy than a fixed duration. That's my take on it anyway.

You make a good point about the tempo. I was lucky enough to find one song done by both drumistry and noisecraft and it's a fast song at 173 bpm. Foo Fighters - The Pretender:

Noisecraft:
https://www.youtube.com/watch?v=3tPrT0UnFS8

Drumistry:
https://www.youtube.com/watch?v=cIgh2OM1AJo

I don't like the noisecraft video. It's better than nothing though.

Drumistry is not being consistent. The song is so fast, the transition to opaque is happening on beat 3 of the first of the two bars shown on a line. I've checked out some other Drumistry videos of various tempi and it seems like the transitions are 'to taste'.

A fixed wall clock time might be okay too. I don't see how it makes anything easier though.

I still have the problem that midiTicks drives everything. Basically I need to convert whatever time it is into midi ticks and figure out how to get the TimeCode class to deal with it. Unless there's another way.

from ly2video.

Here's an idea I've been playing with. This would be easier to read as a diff, but I haven't committed anything yet.

barMoments and breakMoments are populated from the spacetime info.

    measuresXpositions, barMoments = getMeasuresIndices(output, options.dpi,
                                                leftPaperMargin)

    breakMoments = getBreakMoments(output)

There is already a function to extract the x positions of measures, used for the measure cursor. I modified it to populate barMoments. And I added getBreakMoments which looks like getMeasuresIndices except it doesn't extract the x position.

    noteIndices = getNoteIndices(leftmostGrobsByMoment,
                                 midiResolution, midiTicks, notesInTicks,
                                 pitchBends)

    ############ New Code #########################
    # notesInTicks and pitchBends contain midi ticks.
    # getNoteIndicies has a side effect of trimming midiTicks.  So make
    # changes here.

    barsInTicks = []
    for moment in barMoments:
        tick = int(round(moment * midiResolution * 4))
        barsInTicks.append(tick)
        if tick not in midiTicks:
            midiTicks.append(tick)

    breaksInTicks = []
    for moment in breakMoments:
        tick = int(round(moment * midiResolution * 4))
        breaksInTicks.append(tick)
        if tick not in midiTicks:
            midiTicks.append(tick)

    midiTicks.sort()
    # midiTicks now contains extra ticks.  Not really.
    # The bars and breaks will all line up with a beat 1.  It's highly likely that
    # there are already beat 1's in the music at those ticks.

    # create animation point (exact animation details tbd)
    # tag the start point of the animation (length is tbd)
    animationPointsInTicks = []
    for moment in breakMoments:
        tick = int(round(moment * midiResolution * 4))  # location of line break
        tick -= int(round(3 * midiResolution))                 # location of start point
        animationPointsInTicks.append(tick)
    ########## End New Code #######################

    # generate notes
    frameWriter = VideoFrameWriter(
        fps, getCursorLineColor(options),
        midiResolution, midiTicks, temposList)

I think that encodes the details, but now come the problems.

1. midiTicks doesn't contain any info about the ticks. The TimeCode class expects all note events.
2. The TimeCode class doesn't know how to deal with midiTicks unless they are note events.

    def nbFramesToNextNote(self):
        # If we have 1+ tempo changes in between adjacent indices,
        # we need to keep track of how many seconds elapsed since
        # the last one, since this will allow us to calculate how
        # many frames we need in between the current pair of
        # indices.
        secsSinceIndex = self.secsElapsedForTempoChanges(self.__currentTick, self.__nextTick)

        # This is the exact time we are *aiming* for the frameset
        # to finish at (i.e. the start time of the first frame
        # generated after the writeVideoFrames() invocation below
        # has written all the frames for the current frameset).
        # However, since we have less than an infinite number of
        # frames per second, there will typically be a rounding
        # error and we'll miss our target by a small amount.
        targetSecs = self.secs + secsSinceIndex
        debug("    secs at new tick %d: %f" % (self.__nextTick, targetSecs))

        # The ideal duration of the current frameset is the target
        # end time minus the *actual* start time, not the ideal
        # start time.  This is crucially important to avoid
        # rounding errors from accumulating over the course of the
        # video.
        neededFrameSetSecs = targetSecs - float(self.__wroteFrames)/self.fps
        debug("    need next frameset to last %f secs" % neededFrameSetSecs)

        debug("    need %f frames @ %.3f fps" % (neededFrameSetSecs * self.fps, self.fps))
        neededFrames = int(round(neededFrameSetSecs * self.fps))
        self.__wroteFrames += neededFrames
        # Update time in the *ideal* (i.e. not real) world - this
        # is totally independent of fps.
        self.secs = targetSecs

        return neededFrames

This function returns the number of frames to the next tick, but it doesn't know if that tick is a note, an animation point, or a line break. Well it doesn't have to because the assumption is that every midi tick is a note.

So I'm trying to figure out the best way to get access to that information without breaking everything. Ultimately, in the frame writing function, I need to know if neededFrames returns a number to the next note, animation point, or line break.

I guess one way would be to return self.__nextTick with neededFrames. Then the frame/score writer side could look up if that particular tick came from a line break or animation point.

The only problem is that I don't yet know how the note x positions are updated and the midi ticks are consumed. If we did insert extra ticks into midiTicks (because there was a rest on beat1 so no midi tick for a note), it would need to conditionally advance the note index. Because I believe at the moment there is an implied one to one relationship between midiTicks and noteIndices.

Over in ScoreImage:

    def moveToNextNote (self):
        self.__currentNotesIndex += 1
        if self.__measuresXpositions:
            if self.currentXposition > self.__measuresXpositions[self.__currentMeasureIndex+1] :
                self.__currentMeasureIndex += 1

moveToNextNote is only called by this update function in the same class

    def update (self, timecode):
        self.moveToNextNote()

And that happens when the update function is called from the TimeCode class via the notifyObservers() call.

    def goToNextNote (self):
        self.__currentTickIndex += 1
        self.__currentTick = self.__miditicks[self.__currentTickIndex]
        self.__nextTick = self.__miditicks[self.__currentTickIndex+1]
        self.currentOffset = float(self.__currentTick)/self.midiResolution
        self.nextOffset = float(self.__nextTick)/self.midiResolution
        ticks = self.__nextTick - self.__currentTick
        debug("ticks: %d -> %d (%d)" % (self.__currentTick, self.__nextTick, ticks))

        self.notifyObservers()

And back to the VideoFrameWriter class where goToNextNote() is called:

    @property
    def frames (self):
        while not self.__timecode.atEnd() :
            neededFrames = self.__timecode.nbFramesToNextNote()
            for i in range(neededFrames):
                frame = self.__makeFrame(i, neededFrames)
                if not self.firstFrame:
                    self.firstFrame = frame

                self.frameNum += 1
                yield frame
            else:
                self.lastFrame = frame
            self.__timecode.goToNextNote()

So that's the paper trail as I understand it. In order to modify the assumption that each midi tick corresponds to an x-position. there's lots to break...

from ly2video.

I don't know why I didn't see this before. I just need to add a method to get the ticks from TimeCode:

    def getTicks(self):
        return (self.__currentTick, self.__nextTick)

And then I have access to it in VideoFrameWriter:

    @property
    def frames (self):
        print(self.__timecode.getTicks())
        while not self.__timecode.atEnd() :
            neededFrames = self.__timecode.nbFramesToNextNote()

Then there are two ways to do this:

1. implement a setTicks in ScoreImage and call that before calling self__makeFrame
2. pass the ticks in with __makeFrame and then into self.__scoreImage.makeFrame(numFrame, among).

I can pass barTicks, breakTicks, and animationPointsInTicks in with the ScoreImage constructor. Now all the information will be available where it's needed. I think.

from ly2video.

Collecting some thoughts here.

Ultimately, makeFrame is making a single frame which will be a static image or an animation of some kind. Right now animations are limited to either the cursor is scrolling, the notes are scrolling, or the measure cursor is written. The measure cursor changes based on changing note position.

One of the animations I'm planning has a start and end point measured in ticks. So midiTicks could contain the sequence:

note
note and animationStart
note
note and break
note and animationStop and break

or the animation and break events could come without notes. The animationStop may be superfluous if it always stops on a break. I'm just thinking that animations could be more flexible this way. Perhaps it's not needed.

The problem is that I need to know how many frames are between animationStop (or line break) and animationStart with respect to any tempo changes. To do this I would need to precompute the frames from the midi information. Which isn't so bad.

    precomp = TimeCode(midiTicks, temposList, midiResolution, fps)
    frameList = []
    while not precomp.atEnd() :
        ticks = precomp.getTicks()
        frames = precomp.nbFramesToNextNote()
        frameList.append(dict(ticks=ticks, frames=frames))
        precomp.goToNextNote()

I still need to think about how to do stuff with that information. It does mean that the animation points could be a single midi tick with accompanying duration in frames. And this frameList should match what the TimeCode in VideoFrameWriter is iterating.

The other animation is just wall clock and easier. The animation point would be a tick, the duration would just be a number of frames.

Going further, the animation point could be converted to frames and then it's just a matter of keeping track of the frames on the VideoFrameWriter side. In a way this is nicer because the ticks stay related to notes and these other things stay related to frames.

from ly2video.

Using that trick above, I created a dictionary called animationsInFrames and it looks like this when populated:

0 : fadeStart for 90 frames
90 : changePage for 0 frames
120 : fadeStart for 90 frames
210 : changePage for 0 frames
240 : fadeStart for 90 frames
330 : changePage for 0 frames
360 : fadeStart for 90 frames
450 : changePage for 0 frames
472 : fadeStart for 68 frames
540 : changePage for 0 frames
562 : fadeStart for 68 frames
630 : changePage for 0 frames

with <frame number> : <event> for <duration> frames

Then it becomes pretty easy to handle in the ScoreImageTwoUp class. I pass in animationsInFrames during init, count the frames as they're generated, and then there's some logic to change the page when that frame number arrives. I have not implemented fading yet, but I do have the inactive line semi-transparent.

I can create a drumistry style video now, but there are issues. The biggest one is that ffmpeg is not preserving the transparency. I can see that the intermediate slides (using Image.show()) are being created properly, but notes.mpg doesn't have any transparency.

Googling around it seems like you can pass "-vcodec png" to ffmpeg and that should preserve transparency in the video. It's not exactly what I want, but when I try that option the frame generation stops. There's no error messages and ly2video exits. Probably not worth pursuing.

ScoreImageTwoUp produces an image like so:

        # Todo: Make the size configurable and optionally use a background image
        scoreFrame = Image.new('RGBA', (1280,720), color='grey')

        if self.opaque == 0:
            self.pictureA.putalpha(255)
            self.pictureB.putalpha(100)
        else:
            self.pictureA.putalpha(100)
            self.pictureB.putalpha(255)

        # resize before paste.  This way we can apply the cursor writing before
        # resizing.  Copy may be heavy.
        picA = self.pictureA.copy().resize(self.regionSize)
        picB = self.pictureB.copy().resize(self.regionSize)

        # paste each pic to it's own region
        scoreFrame.paste(picA, self.regionA)
        scoreFrame.paste(picB, self.regionB)

        return scoreFrame

And what I need is a function like scoreFrame.compositeSelf() before returning scoreFrame. The existing methods in Pillow seem to take two arguments. I'll keep digging.

The resize and copy is ugly. I did some work to remove it by properly specifying the dpi. I was able to work out how to create an exact width image from liliypond for the standard staff size (20), but I couldn't figure it out how to generalise it for other staff sizes. Eventually it can go away.

from ly2video.

After looking at the documentation for Pillow, I found there is a self compositing function that also takes a region. It's a direct replacement for paste called alpha_composite

        # paste each pic to it's own region
        scoreFrame.alpha_composite(picA, self.regionA)
        scoreFrame.alpha_composite(picB, self.regionB)

So now it's effectively working and I'll close this issue.

from ly2video.