The repository of conference talks videos about lmsa HOT 13 OPEN

Hi Victor, I am interested to work on this project. I am a Ruby developer with 1 year of Ruby experience. Am I eligible to work on this project? Please let me know. Thank you.

from lmsa.

Am I eligible to work on this project?

Of course! Do you have any ideas of where to start, or do you want me to give you some initial steps and hints?

from lmsa.

It will be great if you would be kind to give some initial guidance on how to approach it.

from lmsa.

OK then! As I see it, the very first steps could be those:

1. Go to http://rubyconferences.org/past/ and select some conferences that have videos published...
2. For completeness, select:
   • one with videos and slides right on the site, for ex. RubyKaigi: http://rubykaigi.org/2017
   • one with the site containing only talk announces, and videos on YouTube playlist, for ex. EuRuko: https://www.euruko2017.org/schedule/ (videos, could be found through Twitter: https://www.youtube.com/playlist?list=PLCrwBqiOtwpIRdk4VMfD6hbFVHZoKOyMg and https://www.youtube.com/playlist?list=PLCrwBqiOtwpINmaC2eBX8QHuLYngECD19&disable_polymer=true)
   • (for later) say, one with the site preserved only in webarchive: http://web.archive.org/web/20150927215820/http://rubyconf.org/program and videos on Confreaks.tv: http://confreaks.tv/events/rubyconf2015
3. Now, we need from all of the examples above, produce some YAML (one file per conference) with the list of talks, containing, probably:
   • title
   • announce
   • link to slides, if available
   • link to video, wherever it is
   • speaker name
   • speaker bio/about
   • link to speaker's photo if available
   • talk's date/time (for historical reasons)
   • some conference meta information (title, date, place, description)
4. I recommend trying to write conference site scrapers with wombat, it seems the most appropriate tool
   • for YouTube, it probably would be wise to use official YouTube Ruby client library (and you'll need to match titles of talks on YouTube with titles on conference site, be careful)
5. Then, when you have YAML with all the data, you can try writing renderers of all the data into a static site, probably with the help of middleman. For the first iteration, nothing fancy should be done, just pages like /conferences/ (links to all), /conferences/<conferenceid> (metainformation about the conference + list of talks), conferences/<id>/talks/<talkid> (particular talk with info, links to slides, embedded video)
6. Next step would be to cover all conferences in existence with parsers (sometimes it will also require some digging the web for their long-gone sites, but videos are mostly on YouTube and still present) and have a HUGE bunch of parsed YAML files
   • probably, some small DSLs would emerge to DRY this task, like "This conference has a site at that URL, use this Wombat definitions and fetch videos from those YouTube playlists, and match their titles to conference program with this regexp" or something, we'll see.
7. And then "site crafting" stage comes, which we'll probably discuss and experiment more when the time will come.

How it sounds? Let me know if something is unclear in this plan (especially first stages of it, the later stages intentionally left generic, we can discuss them in more details later)

from lmsa.

from lmsa.

Where do I create the repo?

Just do it in your personal GitHub for now. Then, when the work would be close to publishing, we'll create separate GitHub organization and you'll "transit" repo there (GitHub allows this). The separate organization (something like ruby-talks or something) is needed because GitHub will provide nice org-name.github.io domain name.

from lmsa.

Or let's create org immediately, in fact :) Going to it.

from lmsa.

https://github.com/ruby-talks-archive/ruby-talks Here we go. You should have access to it.

from lmsa.

Hi Victor, just pushed my first commit to ruby-talks under repo named "videotalks". The first one i.e videos.rb is working. Please advise why creating class and specifying model Wombat::Crawler is not working. Thank you.

from lmsa.

Hi Victor, has this progressed since the last update? I would be interested to contribute if I can. Thank you for organising these projects!

from lmsa.

@Krass101 Nope, nothing changed for the project since the discussion in this issue 🤷‍♀️

I'd be glad if you'll contribute your work to this project, and I am ready to answer any questions and provide other help whenever you need.

from lmsa.

@zverok thanks for your offer to help! I have started my own repo and will make small updates every day. I am still learning ruby, wombat and other gems. My goal is to use a TDD approach and as a first step I have written the basic code to scrape the speech titles of the RubyKaigi 2018 conference. My repo is here: https://github.com/Krass101/ruby-talks

Do you have a preferred way for me to ask you questions? I can collect them somewhere and either send to you at once or maintain a list in the repo?

from lmsa.

@Krass101 We can discuss things at https://gitter.im/molybdenum-99/rubytalks (it is chat service that is easy to log in from GitHub, and easy to discuss code in), or you can drop me an email at [email protected] (but Gitter is more convenient).

As for starting the work -- I believe the very first steps could be to just save some page(s) of the conference of interest locally and write some specs saying "I can extract this and that data from the page", this would be a good start (and will also be head-first so you'll see if you love to work on problems like this) which is easy to achieve.

from lmsa.