owncast / owncast Goto Github PK

I'm not sure if it's in the entire code or just the Configuration.md but if you mean joinmastodon.org with mastadon, then the social should instead be called mastodon.

This is a placeholder thread where I can document experiments and have conversations about performance.

Maybe it would be useful to use the websocket connection to be notified of the status updates? Instead of continuously polling?

It's a purposeful panic, so look into it and handle it.

panic: RequestError: send request failed
caused by: Put "https://s3.us-west-2.amazonaws.com/gabevideo/hls/0/stream-20200617-1592382043.ts": write tcp 74.207.245.84:57926->52.218.222.8:443: use of closed network connection

goroutine 81672 [running]:
main.(*S3Storage).Save(0xc0000e2000, 0xc0005f4000, 0x23, 0x0, 0x0)
        /home/gabek/src/video-stream/s3Storage.go:58 +0x38e
main.monitorVideoContent.func1.1(0x22a6300, 0xc0000e2000, 0xc0000b4900, 0xc000195180, 0xc00026a1e0)
        /home/gabek/src/video-stream/playlistMonitor.go:105 +0xb1
created by main.monitorVideoContent.func1
        /home/gabek/src/video-stream/playlistMonitor.go:104 +0x470
exit status 2

We need a way to distribute the binary + required content (webroot, docs, config).

• Create a dist directory
• Build binary for architecture X
• Copy binary to dist
• Copy webroot to dist
• Copy config to dist
• Copy docs to dist
• Copy any additional media (offline image)
• Zip it up

What about the Dockerfile? That probably has nothing to do with a binary distribution.

Tonight while live streaming something happened that I think made the web server start hanging. I'm guessing it was the web server, anyway.

@gingervitis said that calls to things like /config and stream.m3u8 were hanging, so that's why it seems like the web server was hanging issues. It wasn't serving video, so that has nothing to do with it.

One thing of note, at one point during the stream the viewer count went up to > 50, so I'm assuming that's connected. Somebody was watching with VLC and originally didn't know the URL, so I'm wondering if he tried to guess the URL, and then VLC somehow made the web server go nuts, creating a bunch of invalid "viewers", and causing some kind of issue from there. This is all a guess.

Update, it eventually started working on desktop browser -- took like five minutes to sync up

Add command line flag ust like we already have for the config file path, and add an additional option to provide a path in the config file.

Flag should override the config file, config file should override the default.

Right now there's a monitor on the output directory where ffmpeg is spitting out updated playlists and HLS segments in order to know when new files are available and when the playlists are updated so we can upload to S3. This adds both additional CPU utilization since it's constantly polling the filesystem, and additional delay since the poll is timed.

A better solution would be to somehow grab the status from ffmpeg directly that files have been written. But this would require digging through the verbose output of ffmpeg and pulling out the statuses that we want, and that feels brittle and possibly error-prone. There might be a better way to do this. Maybe there's some flags we can pass ffmpeg to get some usable output in order to replace the existing polling.

As a secondary thought, we could replace this polling with native inotify events, but that's limited to POSIX-based operating systems. It wouldn't support windows.

This is mostly noticeable when in an offline state since the offline video plays over and over. I think it has to do with resetPlayer setting the src and the browser autoplaying it.

I've seen some RTMP broadcasting software be able to set a title for the current broadcast. I'd like to figure out how to get ahold of that if possible and we could display it in the web UI.

Have you thought about using the minio sdk instead of the aws one?

https://github.com/minio/minio-go

Hi, thanks for this awesome project. I was following the QuickStart guide for Docker. While building the container worked, I was unable to start it:

$ docker run -p 8080:8080 -p 1935:1935 -it owncast
FATA[0000] Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub

I'm sure that the CGO_ENABLED=0 is in the Dockerfile for a reason, however the programm does not seem to start?

I'm using Docker for Windows through Hyper-V, in case this is an issue?

It's been a couple days since using goth.land for any testing, but I opened it up on the web today and the web socket never connected. There aren't any server-side errors, and there aren't any client-side errors. It just hangs. The chat API (https://goth.land/chat) is fine, though, so the server is ok and handling the messages. It's just the websocket?

Currently It takes a while for the thumbnail to show up as the player's poster after initially loading the web page, I think as a side effect of polling /status, but I think it takes two polls for it to set properly for some reason.

This is how I think it should work, and kind of does:

• If you go to the page and the stream is offline, then the poster should be the logo, not the thumbnail.
• If you go to the page and the stream is online, then the poster should be the thumbnail.
• If the stream is offline, but changes to online, then the player's poster should change from the logo to the thumbnail.

I'm also thinking that if the stream is online maybe we should reload the poster (thumbnail) every 10 or so seconds to update what's in the player to reflect the stream.

ERRO[0014] Server closed by error: Err = Random echo is not matched
github.com/yutopp/go-rtmp/handshake.HandshakeWithClient
        /home/bradley/go/pkg/mod/github.com/yutopp/[email protected]/handshake/handshake.go:102
github.com/yutopp/go-rtmp.(*serverConn).Serve
        /home/bradley/go/pkg/mod/github.com/yutopp/[email protected]/server_conn.go:28
github.com/yutopp/go-rtmp.(*Server).handleConn
        /home/bradley/go/pkg/mod/github.com/yutopp/[email protected]/server.go:115
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
Failed to handshake
github.com/yutopp/go-rtmp.(*serverConn).Serve
        /home/bradley/go/pkg/mod/github.com/yutopp/[email protected]/server_conn.go:31
github.com/yutopp/go-rtmp.(*Server).handleConn
        /home/bradley/go/pkg/mod/github.com/yutopp/[email protected]/server.go:115
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373

Set this up with restream, in addition to two other streams going out of restream to facebook and youtube.

I see in Apple's WWDC sessions for this week they're starting to push fMP4 within HLS. It's been around for a while, but I think it's finally starting to take hold. I don't know if there's any benefit for owncast to switch, and there might even be drawbacks (older devices and players not yet supporting it). But I'm leaving this as a placeholder issue to research and discuss at some point in the future. I'd like to find specifically if there's any encoding performance wins or losses.

https://hlsbook.net/hls-fragmented-mp4/

For people who only want to use the service as a video source they should be able to turn off the web UI.

ffmpeg has a flag to allow us to go backwards from the end of a segment.

Like the -ss option but relative to the "end of file". That is negative values are earlier in the file, 0 is at EOF.

Some quirk of how video.js and the video element plays together on Webkit browsers. Video is still playing, and you hear audio, but when the poster is updated the poster comes to the forefront and displays over the video. It's fine on non-webkit browsers, though. @gingervitis is looking at this in her updates.

Ran through the Dockerized setup in the quickstart, and I was able to successfully build the container, but starting it I'm encountering this error:

root@ubuntu-s-1vcpu-1gb-sfo2-01:~/owncast# docker run -p 8080:8080 -p 1935:1935 -it owncast
INFO[2020-07-12T19:20:21Z] Owncast v0.0.0-localdev (unknown)
FATA[2020-07-12T19:20:21Z] ERROR: valid config/config.yaml is required.  Copy config-example.yaml to config.yaml and edit

I copied the sample config into config/config.yaml before building the container:

root@ubuntu-s-1vcpu-1gb-sfo2-01:~/owncast# ls -l config
total 16
-rw-r--r-- 1 root root 4941 Jul 12 19:10 config.go
-rw-r--r-- 1 root root 1441 Jul 12 19:18 config.yaml
-rw-r--r-- 1 root root  842 Jul 12 19:10 configUtils.go

The only line I changed was updating streamingKey to a different value. I also tried copying config-example.yaml directly, and that also failed.

I'll try the non-Docker approach next, but would definitely like to have this running in a container as an option!

For context, this is a stock DigitalOcean VPS running Ubuntu 20.04

Just spun up a 3 bitrate instance and only bitrate 0 is getting uploaded to storage. I think the file monitor isn't seeing the variant specific playlists getting generated or something.

Either on RTMP disconnect or RTMP re-connect we should wipe all the segment references and on-disk files related to the previous session, since they no longer matter. We wipe the files on launch if the user were to re-launch the service, but if it stays up that currently doesn't take place.

When running for the first time, I run into the following error:

Enabling passthrough video
/usr/local/bin/ffmpeg -hide_banner -i doc/logo.png -i webroot/thumbnail.png -filter_complex "[0:v]scale=2640:2360[bg];[bg][1:v]overlay=200:250:enable='between(t,0,3)'"  -f hls -hls_time 4 -hls_playlist_type event -master_pl_name stream.m3u8 -strftime 1 -use_localtime 1 -hls_flags temp_file -tune zerolatency -g 50  -keyint_min 50 -framerate 25 -preset superfast -sc_threshold 0 -profile:v main -pix_fmt yuv420p -var_stream_map "v:0" -hls_segment_filename webroot/hls/%v/offline-%s.ts webroot/hls/%v/stream.m3u8
exit status 1
PANI[0000] exit status 1                                
panic: (*logrus.Entry) (0x52af2a0,0xc000422070)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc000142000, 0xc0004c0120, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:259 +0x335
github.com/sirupsen/logrus.(*Entry).Log(0xc000422000, 0xc000000000, 0xc00063fa10, 0x1, 0x1)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Logger).Log(0xc000142000, 0xc000000000, 0xc00063fa10, 0x1, 0x1)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:193 +0x7d
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:234
github.com/sirupsen/logrus.Panic(...)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/exported.go:129
main.verifyError(...)
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/utils.go:38
main.showStreamOfflineState(0x0, 0xc0000471d0, 0xf, 0xc000047020, 0xb, 0xc000047040, 0x3, 0x4, 0xc0000470f0, 0x6, ...)
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/ffmpeg.go:88 +0xd85
main.createInitialOfflineState()
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/utils.go:81 +0x54
main.main()
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/main.go:47 +0x1e9
exit status 2

And when I run the ffmpeg command, it gives me: webroot/thumbnail.png: No such file or directory

Obviously, I can place a file there but figured I would open this issue just to let you know and maybe a file can be placed there for people wanting to download it and just try it out.

Okay, so one thing I don't like about the current restructure. In my opinion, the core package should initiate the chat.Server and maybe the router package should tell it to start listening. However, this causes a circular dependency from the core and the chat package due to the adding/removing of a client.

https://github.com/gabek/owncast/blob/487bd1244497e0ad3a9d793bc6b7ca80bd3e7fb0/core/chat/server.go#L142-L152

Any ideas on how we should proceed with this?

I'm guessing this probably started happening when I refactored how video.js was initialized.

What would be your thoughts on the following project structure:

├── config 
│   └── config.go
├── controllers (http controllers)
│   ├── chat.go
│   └── static.go
├── core (the logic lives here)
│   ├── ffmpeg/
│   │   ├── offline.go
│   │   └── start.go
│   ├── rtmp/
│   │   ├── handler.go
│   │   └── server.go
│   ├── storage/ (implementations)
│   │   ├── ipfs.go
│   │   └── aws.go
│   ├── storage.go (interface)
│   ├── core.go
│   ├── stats.go
├── models (definitions for how data will be handled)
│   ├── stat.go
│   ├── server.go
│   └── etc.go
├── utils
│   └── utils.go
├── static (stays the same)
│   └── logo.png (etc)
├── webroot (stays the same)
│   ├── img/
│   ├── js/
│   ├── static/
│   ├── styles/
│   ├── vendor/
│   └── index.html
├── go.mod
├── go.sum
├── main.go
├── config.sample.yaml
├── README.md
├── Dockerfile
└── .gitignore

This is a massive difference from the current structure of the project and I am willing to start the PR to move it and work on the restructuring. Curious to hear your thoughts

$ go run *.go
INFO[0000] Owncast v0.0.0/localdev (unknown)            
INFO[0000] Resetting file directories to a clean slate. 
----- Stream offline!  Showing offline state!
Enabling passthrough video
/usr/local/bin/ffmpeg -hide_banner -i doc/logo.png -i webroot/thumbnail.png -filter_complex "[0:v]scale=2640:2360[bg];[bg][1:v]overlay=200:250:enable='between(t,0,3)'"  -f hls -hls_time 4 -hls_playlist_type event -master_pl_name stream.m3u8 -strftime 1 -use_localtime 1 -hls_flags temp_file -tune zerolatency -g 50  -keyint_min 50 -framerate 25 -preset superfast -sc_threshold 0 -profile:v main -pix_fmt yuv420p -var_stream_map "v:0" -hls_segment_filename webroot/hls/%v/offline-%s.ts webroot/hls/%v/stream.m3u8
<nil>
INFO[0000] RTMP server is listening for incoming stream on port 1935. 
INFO[0000] Starting public web server on port 8181      
Enabling passthrough video
PANI[0041] exit status 1                                
panic: (*logrus.Entry) (0x52af2a0,0xc000250070)

goroutine 101 [running]:
github.com/sirupsen/logrus.Entry.log(0xc00013a000, 0xc0005c8570, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:259 +0x335
github.com/sirupsen/logrus.(*Entry).Log(0xc000250000, 0xc000000000, 0xc000082ee0, 0x1, 0x1)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Logger).Log(0xc00013a000, 0xc000000000, 0xc000082ee0, 0x1, 0x1)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:193 +0x7d
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:234
github.com/sirupsen/logrus.Panic(...)
        /Users/bradleyhilton/go/pkg/mod/github.com/sirupsen/[email protected]/exported.go:129
main.verifyError(...)
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/utils.go:38
main.fireThumbnailGenerator(0xc00020cff0, 0xb)
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/thumbnailGenerator.go:71 +0x4dd
main.startThumbnailGenerator.func1(0xc000218910, 0xc00020cff0, 0xb, 0xc00027e6c0)
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/thumbnailGenerator.go:21 +0x4f
created by main.startThumbnailGenerator
        /Users/bradleyhilton/go/src/github.com/graywolf336/owncast/thumbnailGenerator.go:17 +0x97
exit status 2

Here is my streaming preferences from ProPresenter 7.1.1:

However, when streaming from OBS at 1920x1080 60fps it doesn't crash.

I'd like to get some different people, from the broadcasting side pointing at an instance with whatever client software they use, and see what happens. Just as one-offs to see how things run in different stream examples, see what crazy stuff they try to do with the chat, see what kind of performance issues we run into, etc. Just having @graywolf336 try it with some different software and environments immediately surfaced issues, so it would be nice to continue this.

What are some major things lacking before we feel comfortable testing it out with people? Not that we'd have many people, but a few friends of mine are willing to stream to it and send a link out for one-off streams. Plus there's always @gingervitis streaming Animal Crossing :P

My plan is to use that $10/mo machine I grabbed from Linode to host three bitrates (passthrough, medium, low) and see what happens with whatever people throw at it.

I'm guessing some HTML metadata tag should resolve this?

This is a placeholder issue for discussing and researching reducing the ~20seconds of delay from live that I'm currently seeing.

There's a lot of pieces at play with owncast, each adding their own delay into the mix, but I'd love to see this closer to 10 seconds.

Videojs hides the progress bar during live content. @gingervitis thinks it would be useful to show it and be able to seek back, so let's enable it. There's probably a way to force Videojs to show it.

Twitch has this and it's interesting information.

The websocket should not be disconnected, but if it's blocked or there's a problem then chat input should be disabled.

I think it would be nice to have a single place to make changes and not have to specify that you're talking about the backend config vs. the frontend config, especially in initial setup and in documentation.

Since we're already pulling in config.json on the frontend asynchronously we could simply create a /config endpoint that returns the same values config.json does based on the values in config.yaml and point there instead.

No rush to do this, since having a standalone frontend config right now allows faster iteration on the web UI, but would this be a positive change eventually? What are your thoughts @gingervitis ?

Adding https://github.com/mattn/go-sqlite3#cross-compile to the project is adding an additional level of complexity due to cgo and cross compiling. It seems like we can cross-compile to eliminate additional dependancies on the target machines with xgo, but I'm not sure yet. @graywolf336 if you have any thoughts around this or have worked through this in the past, let me know!

~/owncast# ./owncast -enableVerboseLogging -enableDebugFeatures
FATA[0000] Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub 

@graywolf336 Added a /chat endpoint, so I think it makes sense to use that to populate the chat history instead of funneling the entire history through the socket every time somebody connects.

For example, @graywolf336 sent me a link on Mastodon, and I received it with the Mastonaut app, but it wasn't pulling the opengraph tags (Name, description, image, etc). The user-agent detection probably wasn't routing correctly. Will need to find a more extensive way to detect that a client is trying to pull embed metadata.

Not pressing, but some options would be nice. Some that come to mind.

• Location of config file (suggested by @graywolf336)
• Log level
• EnableSetReportCaller for debugging.
• Misc config file overrides.

Instead of an array of arbitrary flags as strings messily passed to ffmpeg, create a struct with typed properties that will spit out a usable command. Right now there's duplicated code for simple development purposes, but it's time to remove the magic strings around this.

• It should have a Variant struct for supporting the multiple bitrates.
• Have things like SetBitrate, SetFramerate, SetPreset, SetProfile, SetPlaylistName, SetSegmentName, SetSegmentLength, etc.

This should also allow us to tweak values in continued performance testing and get a reliable output.

Right now it's hardcoded that variant 0 is always used to get a thumbnail, since 0 is always guaranteed to exist. However, 0 may not be the highest quality source to pull a thumbnail from. So instead we should go through the config's quality variants and determine what is the highest quality (video passthrough being the best option if available).

goroutine 19555 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000b0000, 0xc00013dad0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/gabek/src/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:259 +0x335
github.com/sirupsen/logrus.(*Entry).Log(0xc0001a6460, 0xc000000000, 0xc000c65bf0, 0x1, 0x1)
        /home/gabek/src/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Logger).Log(0xc0000b0000, 0xc000000000, 0xc000c65bf0, 0x1, 0x1)
        /home/gabek/src/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:193 +0x7d
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /home/gabek/src/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:234
github.com/sirupsen/logrus.Panic(...)
        /home/gabek/src/go/pkg/mod/github.com/sirupsen/[email protected]/exported.go:129
main.verifyError(...)
        /home/gabek/src/video-stream/utils.go:38
main.startFfmpeg(0x0, 0xc00049a260, 0xf, 0xc00049a0d0, 0xb, 0xc00049a0f8, 0x3, 0x4, 0xc00049a190, 0x6, ...)
        /home/gabek/src/video-stream/ffmpeg.go:178 +0x1242
created by main.(*Handler).OnPublish
        /home/gabek/src/video-stream/handler.go:69 +0x3b0
exit status 2

Should both fix the crash and reject concurrent incoming RTMP connections?

Somehow the chat toggle button disappeared on my iPad.

When testing, it appears that the web player is using the lowest bitrate even though I have ample bandwidth for the high quality video

While in the future I'm sure there will be a major update, I wanted to add an issue for short term things to add.

• Change message IDs from being client generated to server generated. Mentioned again here: #25 (comment)
• Persist the chat. Mentioned here #18 (comment)
• @gingervitis suggested disabling chat when a stream is no longer active. A timeout here would make sense, to allow chat after N minutes of a stream being disconnected, but lock it down when we think a "session" has ended.
• Include timestamps with each message

Github offers a registry for Docker images. It might be nice to leverage this as a fast way for people to spin up an instance of Owncast. There's already a Dockerfile in the project that builds a Owncast image.

https://help.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages

I haven't thought through the configuration story in this scenario.

When the user stops streaming the stream should be replaced with an "offline" visual state. This can be a static image that's transcoded into the server's existing HLS stream(s) or a short video clip that gets added to. Essentially just appending to the HLS playlists after the user drops their RTMP.