Add aspect ratio support about cloud-game HOT 21 CLOSED

It's just a basic nearest neighbor algo (interpolation), same as yours but it works properly (: And it is fast.

Something like that:

func toRgba(fn convert, width int, height int, packedWidth int, data []byte) *image.RGBA {
	xRatio := ((width << 16) / ewidth) + 1
	yRatio := ((height << 16) / eheight) + 1

	for i := 0; i < eheight; i++ {
		for j := 0; j < ewidth; j++ {
			x2 := (j * xRatio) >> 16
			y2 := (i * yRatio) >> 16

			index := (y2 * packedWidth) + x2

			fn(data, index, j, i) 
                        // pixel := (int)(data[index*int(video.bpp)]) + 
                        // ((int)(data[(index*int(video.bpp))+1]) << 8)
		}
	}

	return outputImg
}

Viewport resolution is fixed to the basic core resolution * 2 and it ignores the config. Not sure what resolution will be best or how to handle this stuff, double scaling leads to a noticeable performance hit but, on the other hand, texts in games don't look like garbage.
Speaking about performance bottlenecks, you could try to run your profiler to see that the largest one is frame render time in a core, second -- vpx encoding time, third -- these go RGBA.set parts outputImg.Set(x, y, color.RGBA{byte(r8), byte(g8), byte(b8), 255}).

I guess, it will be much faster with a running x264 (hardware) encoder but I have no idea how to make it work.

from cloud-game.

Oh, I see, leave it to me, then. I thought it's harder to do (:.

from cloud-game.

I decided to use calculated AR from core provided base w and h values (PAR 1:1) because core provided AR in most cases will be wrong. By default, viewport size is set to 320x240 before AR calculations.

from cloud-game.

Nice!!
How you do the upscaling and down scaling. I expect it to be fast so it will not affect streaming performance. I think interpolation scaling might cause it slow.
Does viewport resolution for GBA is still fixed with 240*160 again?

from cloud-game.

I think that scaling algorithm should work in this case. I think it's fast enough. Will be a great improvement!

The legend of Zelda screen is broken because we scaled down the resolution last commit. It was fine with 240*160. The number I got in config is the major resolution collecting from all frames in each emulator. However, that number varies from Game to Game, not emulator. I think we can have a dynamic external config that can be updated per game.

Regarding to performance bottleneck, you are right pointing those three, and all of them are out of my control :(

Btw, I have both VP8 and H264 encoder in the codebase and experimented them. The project is currently using VP8 encoder because I realize the implementation of Go H264 Encoder is not proper.
https://github.com/giongto35/cloud-game/blob/master/pkg/util/util.go#L90

from cloud-game.

The legend of Zelda screen is broken because we scaled down the resolution last commit. It was fine with 240*160.

Not sure about that, I will try to run the master branch with 240x160.

The number I got in config is the major resolution collecting from all frames in each emulator. However, that number varies from Game to Game, not emulator. I think we can have a dynamic external config that can be updated per game.

Ok, if you want a resolution config for each core so be it, in my opinion it's too tedious to hand pick these resolutions and a simple x2 scale works fine. I'll add a new cmd key for scaling factor of resolutions in the config, then.

Btw, I have both VP8 and H264 encoder in the codebase and experimented them. The project is currently using VP8 encoder because I realize the implementation of Go H264 Encoder is not proper.
https://github.com/giongto35/cloud-game/blob/master/pkg/util/util.go#L90

I'm getting some segfault crash when the first frame is being encoded with x264.

from cloud-game.

I just realized that, I handpicked the resolution just because I didn't have upscaling : D . If we have upscaling algorithm work, we can just fix the resolution to be 360p: 480 x 360 or 480p ... (configurable).
I'm also fine with your approach to have a scaling factor of the resolution also. Either way is fine because the scaling algorithm helped.

from cloud-game.

Not related to this issue, but I updated audio streaming by batch. https://github.com/giongto35/cloud-game/pull/106/files . You can review it.

from cloud-game.

I've just pushed commit into my fork. Can you make a dev branch here I will make PR into that instead of
the master branch.

You can try how it works with different settings in the config. Couldn't make it work with cmd flags, don't know how to pass them through the whole app into a room and emulator (:
And one more thing with this nearest interpolation -- without x2 resolution scaling there is too much aliasing.

Not related to this issue, but I updated audio streaming by batch. https://github.com/giongto35/cloud-game/pull/106/files . You can review it.

Seems alright. Not sure what's that all about slice copies but now it's working without crashes. Thanks.

from cloud-game.

Thank you :D, Just take a quick look, It is really promising. I will check it later because I will sleep soon
I pushed dev branch, you can send PR to that first :D
https://github.com/giongto35/cloud-game/tree/dev

from cloud-game.

@sergystepanov : Tested and worked well.
I think Aspect ratio makes more sense, omit hardcoded config.
Aliasing is the unavoidable problem of nearest neighbor algorithm and even x2 resolution I still see some aliasing. I think applying bilinear scaling can resolve that but it's ok to go forward with nearest neighbor algorithm for now.

from cloud-game.

So, I was thinking to make it use by default your old interpolation algo without xN scaling and to add two new cmd params: --use-aspect-ratio, --scaler (old, nearest, skip), and maybe --out-resolution (NxM = 120x200, 320x240 and so on) but I don't know how to pass them down into the room and emulator modules, is it possible without too much mess in the code? And what you think?

from cloud-game.

Let's stick with nearest neighbor all the time. Later if we have bilinear algorithm, we can switch to it.

I think only having a flag for --use-aspect-ratio is valid, can make it default as false. If bilinear algorithm is applied, I think we no longer have that aliasing problem and we can permanently use aspect ratio.

from cloud-game.

Ok, got it. You can tweak those hardcoded params as you see fit (code also) and merge it.
Will try to implement both bilinear and bicubic interpolations later (and next, I guess, going to research proper x264 encoder support and encoding)

from cloud-game.

To pass the config
https://github.com/sergystepanov/cloud-game/blob/master/pkg/worker/overworker.go#L53
can change to pass cfg to Handler
worker := NewHandler(o.cfg)

and
https://github.com/sergystepanov/cloud-game/blob/master/pkg/worker/handlers.go#L138
can change to pass cfg to Room also
room := room.NewRoom(roomID, gameName, videoEncoderType, h.onlineStorage, h.cfg)

You can leave it to me if you feel like not want to do it and just hardcode config first :D

from cloud-game.

The x264 encoding framework I based on https://github.com/gen2brain/x264-go
I test on production with x264 and realized it is slower than VP8. You can try switching to config.H264 and profile, I saw they have some hotspot with high allocation in the implementation.

from cloud-game.

The x264 encoding framework I based on https://github.com/gen2brain/x264-go
I test on production with x264 and realized it is slower than VP8. You can try switching to config.H264 and profile, I saw they have some hotspot with high allocation in the implementation.

Hm, that's strange, I assume, both encode in the software mode without hardware acceleration, with the same settings? then x264 should be faster. Maybe try another implementation, or even ffmpeg (RetroArch uses it for streaming) frontend.
Anyway it crashes on me as I said on the first frame encode call (on Windows).

from cloud-game.

The slow part is the Golang implementation of the library is flawed, not H264 codec performance. I think the creator of Pion (Golang webRTC) recommended to use GStreamer but I haven't tried to use it. =.=
I think it's better if we research on GStreamer, I think it has better and easier to use API https://github.com/pion/example-webrtc-applications/tree/master/gstreamer-send. Both VP8 and H264 library I'm using is non-official and relied on CGO.

btw, H264 is working on my machine. :/ hmm maybe another OS issue. I think it will work on ubuntu VM.

from cloud-game.

Got your point, I will land your change on master after retest and close this ticket. Is it ok ?

from cloud-game.

Sure thing. Also you can remove those hardcoded widths and heights for each core from the config unless you need them for something.

from cloud-game.

I just realize that the images produced by bilinear is a bit blurry and low contrast. I'm experimenting how to deal with it. Maybe I can increase contrast or try increasing resolution.

Bilinear interpolation works by interpolating pixel color values, introducing a continuous transition into the output even where the original material has discrete transitions. Although this is desirable for continuous-tone images, this algorithm reduces contrast (sharp edges) in a way that may be undesirable for line art. Bicubic interpolation yields substantially better results, with only a small increase in computational complexity.

from cloud-game.