Artefacts described in this ticket are improved in SVP with Rife v4.15 but still present about practical-rife HOT 56 OPEN

4.9 is better. One remainder of an artefact that was improved in 4.7 is now completely gone in 4.9. The Spider-man artefact I show above seems to have improved slightly. I have another similar artefact that has also improved to the point where I can remove it by decreasing the scene detection threshold. It's also a little more efficient than 4.7 too Rife is getting better and better 👍

from practical-rife.

Well, in terms of speed, it's faster than 4.13-lite, the lite models are more for speed than quality. Keep using 4.15 for the best results.

from practical-rife.

I uploaded here some recordings of the display motion testing software Smoothfrog(https://www.aperturegrille.com/software/) at both 30fps and 24fps. Both use a 3840px/sec pan speed(for the 1920 width recording this is 1 screen width per 0.5sec).
I realize that this test is not similar to the training data, and will not reflect all content. However, it can be considered a "torture test" for fast panning. It reflects a part of VFI performance(fast simple motions with fine patterns)

for 24fps(which is a more strict test due to lower base fps) 2x:

4.7

4.8

4.9

4.10

4.11

4.12

4.13

4.14

4.15

4.16 lite

As you can see 4.7 beats all later models on this test. I still think that 4.15 is the best overall performing rife model, but for certain cases it's outperformed by 4.7. In the movie "Spirited Away" the frame slightly jitters up and down on each frame on the original. With all models 4.8 and later this causes all pans to jitter in the rife output. With all models 4.7 and previous this jitter is not there. Also not a problem in softsplat based interpolation GMFSS pg104/UMSS v1/VFSS Anime v1.

Here is a link to a 25 second clip of one of the pans from the intro: https://gofile.io/d/WJDoap

Please eyetrack the stained glass windows, and compare 4.7 to 4.15 on this clip. On 4.15 you should see what looks like a microstutter periodically during the motion, which is not present on 4.7.

If anyone is using scdetect for the verification/testing, please turn it off. It will turn off the interpolation in most fast motion if you have it adjusted to sensitive values(anything lower than like .3). That goes for these clips and also any other verification of interpolation.. Well, unless you're using a transitionchooser list to manually select the correct scene changes.

I hope this can be useful feedback for rife @hzwer, the blendy/blurry appearance in fast motion(and especially the panning jitter on some sources) has bothered me since I first noticed it with later models. Rife 4.15 is very very good for general watching both of live action and anime, but I think it can still be better if the training(and especially verification) includes more extremely fast motions. Especially fast continuous motions without any motion blur on the source, like game footage recorded at 24 or 30fps for example. The biggest weaknesses are the multiple layers, fine patterns, and fast motion.
Anyways, thanks for all the work so far, you're doing a good job and I still eagerly await future iterations of Rife.

from practical-rife.

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

from practical-rife.

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

from practical-rife.

@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.

If you test out all different models and all of them exhibit this error, then this might be a rife problem.

It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.

BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.

from practical-rife.

@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.

If you test out all different models and all of them exhibit this error, then this might be a rife problem.

It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.

BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.

I've tested 4.6 and 4.7 , both have the same issue and since 4.7 > 4.6 > 4.4 I'm guessing that 4.4 will be the same. Maybe I'm missing something here but even the pay version of Flowframes seems to be only using 4.4 which has even more artefacts and errors.

from practical-rife.

FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.

from practical-rife.

FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.

Installed Flowframes. Says GPU: Virtual Desktop Monitor even though I have an Nvidia GPU. Tried changing GPU id makes no difference. When interpolating CPU is basically idling for scene extraction and frame extraction. If it's going to be this slow I really would need a realtime mode.

from practical-rife.

@netExtra OK if all of them have it, then it's probably RIFE Model, probably needs more training.

from practical-rife.

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

Rife.error.mov

cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.

from practical-rife.

cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.

Thanks but there were other ugly errors in Rife 4.6 that have been fixed in 4.7/4.8 so I would hope that this eventually gets fixed soon. Especially since these large and ugly errors don't occur in SVP when not using Rife.

from practical-rife.

Update 04/11/2023 I don't know if this helps but I can remove the Spider-man artefact shown above by significantly reducing rife-scene change threshold in SVP. But it is so low that it cause stuttering issues elsewhere.

from practical-rife.

The 'scene change threshold in SVP' you mentioned is out of rife's control. Because it use a different 'solution' to detect scene changes. Once some frames were marked by this 'solution', rife do nothing on them.

from practical-rife.

Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.

from practical-rife.

Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.

I disabled the sc algorithm. I double checked it was off by setting the sc threshold to 5 which I know fixes the issue in SVP manager GUI. When I run the clip it's broken just like the video attached above. When I re-enable the algorithm the artefact disappears but causes major problems elsewhere so it's not a solution.

I'm not sure how testing a clip without scene-changes would help because there are no issues with those and never have been.

from practical-rife.

choose a clip without scene-changes to test.

It's not talking about the movement of len.

Could you give the raw clip of your test src?

from practical-rife.

choose a clip without scene-changes to test.

It's not talking about the movement of len.

Could you give the raw clip of your test src?

Hi. Sorry I am confused about your request for a clip without scene changes. I don't have any issues with that. What do you mean by "movment of len"?. Also, my raw clip is the one that is attached to this issue. Thanks.

from practical-rife.

I only see the screen recording here. #34 (comment)

I want to test its raw clip on my side.

from practical-rife.

I only see the screen recording here. #34 (comment)

I want to test its raw clip on my side.

OK I'm doing it now. Will upload it in a couple of minutes. Thanks.

from practical-rife.

Attached. I had to convert it to MP4.

from practical-rife.

Thanks. It could be reproduced by inference_video.py with model4.9.
https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687

from practical-rife.

Thanks. It could be reproduced by inference_video.py with model4.9. https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687

Now it can be replicated does that help? BTW the attached video doesn't work :)

from practical-rife.

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

from practical-rife.

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

OK Thanks. I've raised a new issue for Rife 4.11 but I'm not sure if I should be since it probably doesn't affect the model training.

from practical-rife.

OK so this artefact seems to be gone now and I am confused because up until this morning it was still there with Rife 4.9. I have another annoying artefact similar to this which was getting better but seems to have regressed a little. SVP has been updated and new code added so I will have to investigate further.

from practical-rife.

Update:
The problem artefact is back again and the one that had regressed is back to being improved again. Very strange! Not going to reopen at the moment.

from practical-rife.

OK there was a weird issue where the player I was using wasn't registering with SVP after the first file. I'm guessing that is why the artefacts went away because SVP and therefore RIFE was not running.

from practical-rife.

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans

BTW the issue applies to Rife v4.6 as well so it probably goes back some way.

from practical-rife.

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.
I didn't try legacy v3/2 models so I have no ideas more.

I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans

BTW the issue applies to Rife v4.6 as well so it probably goes back some way.

At this point you're just confusing SC thresholds. Setting a scene change threshold literally outright disables RIFE on those frames, so obviously there wouldn't be artifacting, and hence why it will obviously be stuttery at some parts.

from practical-rife.

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.
I didn't try legacy v3/2 models so I have no ideas more.

I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans
BTW the issue applies to Rife v4.6 as well so it probably goes back some way.

At this point you're just confusing SC thresholds. Setting a scene change threshold literally outright disables RIFE on those frames, so obviously there wouldn't be artifacting, and hence why it will obviously be stuttery at some parts.

I have disabled the SC threshold in SVP and the artefact is improved but it is still there. But as I said above, I can fix the artefact by setting SVP SC to below 6 but setting SC this low causes stuttering for slow pans.

Here is a a direct quote from the SVP Rife AI FAQ:
Scene change threshold - Higher values gives more interpolation artifacts when scene change was not detected. Lower values give more "stuttering" (i.e. repeated frames instead of interpolated) when there was a false detection.

FYI with SC disabled, slow pans are almost perfect but fast object movement causes double images in Rife 4.9 and below. But the double images are gone in Rife 4.12, so there is a visible improvement between 4.9 and 4.12.

from practical-rife.

In my experience, Rife 4.1 is the best model for panning. 4.12 is slightly worse than 4.1. And 4.13 is slightly worse than 4.12.
All other v4 models have significantly more artifacts in fast panning scenes.

from practical-rife.

In my experience, Rife 4.1 is the best model for panning. 4.12 is slightly worse than 4.1. And 4.13 is slightly worse than 4.12. All other v4 models have significantly more artifacts in fast panning scenes.

What I was talking about with Rife 4.12/4.13 was fast object movement in live action scenes. Fast panning seems to be identical between the two.

Rife 4.6 has more artefacts than 4.9 so I wouldn't consider using Rife 4.1.

from practical-rife.

OK so I did a bit of reading up on Rife today and from what I read it made sense to increase multi which I think is also known as the implementation factor. Once I did that, the double images went away from the Spider-man clip shown above as well as another movie with a similar issue.
I would like to close this but I'm just wondering else what impact increasing multi will have on anything else. Besides I assume increased resource usage.

Update: It fixes the issue but when run Rife is loaded then after a few seconds the screen goes blank then returns but I don't think Rife is running anymore because the GPU usage goes down a lot.

from practical-rife.

With the current Rife onnx models, the only way to fix the double images (in fast motion) is to increase the fps. At 72fps the issue is reduced a little. At 90 or more fps it disappears. With my GPU I can just about do 72fps but with some ugly stuttering on slow pans.

from practical-rife.

Hello,
I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.
If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.
But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

Rife.error.mov

The artefact seen in the certain types of fast movement like the one shown in Amazing Spider-man clip above has been reduced in v4.15. It's not fixed but it definitely has been improved :)

from practical-rife.

Any way to reduce panning artefacts? My TV motion interpolation never has issues with straight pans like this but rife often has issues with these kind of shots.

from practical-rife.

@carycary246 Maybe I'm missing something but I can't see any major artefacts in that clip. But if it is there, then it looks very minor. Also which version of Rife are you using?

from practical-rife.

@carycary246 Maybe I'm missing something but I can't see any major artefacts in that clip. But if it is there, then it looks very minor. Also which version of Rife are you using?

Look at the right side with the building fences. I think Rife doesn't know how to handle those kind of objects (thin lines?) even when it is a straight pan with no other motion. I am using 4.14 lite now, but all previous versions are the same. Here is another example, in motion it is distracting because it warps a large area over several frames.

from practical-rife.

@carycary246 The artefacts are not being captured clearly in your clips, but they are clear in the still image. I don't watch anime, but in live action this type of horizontal and vertical pan straight line warping is much reduced by Rife v4.9 and mostly gone by Rife v4.13, with only a couple of minor artefacts visible if you look for them. The main issue remaining is fast movement artefacts but v4.15 is an improvement. Sorry I can't be more help.

from practical-rife.

4 frame interpolation model may resolve the problem

from practical-rife.

@carycary246 can you share these two original videos without interpolation?

from practical-rife.

@hexiaoyi95 I've uploaded the two clips without interpolation here.

https://mega.nz/file/9ToS0KgQ#S03DLc5YPMCPnDhPmsjN-bLpb4O3MR2xQjVJpC4HNhg
https://mega.nz/file/Zf5X1IAR#HesJScsyt7iBT0b4zM8Rtp7iwDXvJL-Z8Crc0Bgd0pM

from practical-rife.

@hexiaoyi95 I've uploaded the two clips without interpolation here.

https://mega.nz/file/9ToS0KgQ#S03DLc5YPMCPnDhPmsjN-bLpb4O3MR2xQjVJpC4HNhg https://mega.nz/file/Zf5X1IAR#HesJScsyt7iBT0b4zM8Rtp7iwDXvJL-Z8Crc0Bgd0pM

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

from practical-rife.

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

Your Name is a standard 24fps film, not sure why you wouldn't be able to play it. Panning motion wouldn't be any different from any other kind of 24 fps media.

from practical-rife.

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

Your Name is a standard 24fps film, not sure why you wouldn't be able to play it. Panning motion wouldn't be any different from any other kind of 24 fps media.

SVP doesn't like it and it's default config treats it like HFR but I still got a good idea of the artefacts for both clips.

from practical-rife.

Rife v4.15 was a major improvement for fast motion but Rife v4.16 is a major regression. I hope @hzwer does NOT use this as the backbone for future versions because for live action fast motion, it would be like going back to Rife 4.9 or worse.

from practical-rife.

4.16 is a lite model, so it is similar to 4.9 in terms of quality. I'm assuming he means that it will be the base for future lite models.

from practical-rife.

Maybe. But 4.16 being pretty much like 4.9 for quality is going in the wrong direction. Rife 4.9 is a nice all round model but is pretty bad for live action. Rife 4.15 is the best for quality.

from practical-rife.

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a heavier version to see the effect.

from practical-rife.

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a “heavy” version to see the effect.

I've checked the SVP forums and the consensus is that v4.16 lite is a good all round replacement for v4.9. But v4.15 has the least artefacts, especially for fast motion. Maybe your friends are talking about animation? I'm talking about video and movies.

from practical-rife.

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a “heavy” version to see the effect.

Rife v4.15 lite arrived today and is almost as good as v4.15 (heavy) and is much better than v4.16 for movies with fast motion. Rife v4.15 (heavy) still handles the artefacts I have shown above on Nov. 5th 2023 with Spider-man much better.

from practical-rife.

@DisplayTalk what you say about 4.7 is interesting. I will test with my normal test scenes and compare.

Update: 4.7 is actually very good for fast movement but it's not quite as good as the two v4.15 models. It still suffers a little from similar artefacts to v4.6. But v4.15 lite is just as good for GPU usage (if not a little better) and doesn't suffer from the same artefacts as models pre v4.9.

from practical-rife.

Yes, 4.15 is still the better model generally, but in some situations 4.7 is better. I think this can be fixed by changing training/validation a bit to prioritize simple fast continuous motions, handling of fine details/patterns, and stability of panning.

from practical-rife.

It's been a while since I've used Rife v4.15 v2 so I thought I should have a proper look again. It minimises or removes every fast motion artefact I have come across with the default SCT in SVP. Maybe I'm repeating myself a little here but Rife v4.15 lite is pretty close to 4.15 but uses much less CPU. So if you don't have the same artefacts it makes more sense to use the lite version.

from practical-rife.

I'm not sure if this helps but:
SVP with TensorRT 9.2 - Rife v4.15 (heavy) at 60fps works best with fast motion.
SVP with TensorRT 8.5.1 - Rife v4.15 lite at 72fps and above works best. Rife v4.15 (heavy) artefact handling gets worse.

i.e.
TRT 8.5.1 with Rife v4.15 lite/72fps = TRT 9.2 with Rife 4.15/60fps.

from practical-rife.