Currently the system requires a network to exist for a user, so we need to create a network on user registration, otherwise:

• model/schema
• parse/format tools
  • parse XML (psc)
  • format XML (psc)
  • parse from JSON
  • format as JSON
  • parse mp4chaps
  • format as mp4chaps
• GraphQL
  • queryable
    • add batching
  • mutatable
    • fix multi-imports
• REST queryable
• psc in feed

Goal: Files can be uploaded and selected from a list in the episode creation screen.

• API: create file / upload
• API: list all (recent) files
• GUI: upload new file
• GUI: select file from recent uploads (not necessary until we implement async uploads)

I prefer distillery releases and a multi-stage Dockerfile for phenomenally lean production nodes. distillery has good guides:

• https://hexdocs.pm/distillery/guides/phoenix_walkthrough.html
• https://hexdocs.pm/distillery/guides/building_in_docker.html

Imported feeds might have their episodes displayed in the wrong order (ORDER BY id desc instead of ordering by publishing date).

I think that users will welcome to have their latest published episodes at the top and this was, how the app was behaving before todays merges. I will soon publish a PR defining the order of the episodes list.

This is a bug with the signup module where incorrect form data saves a dataset, resulting in "Email has already been taken".

Steps to reproduce:

1. Open `/signup` and type in bad form data (e.g. non-matching passwords or an empty username).
2. An exception page will show up.
   Exception from logs

   radiator_1  | [debug] QUERY OK source="networks" db=0.9ms queue=1.2ms
   radiator_1  | SELECT n0."id", n0."image", n0."title", n0."slug", n0."inserted_at", n0."updated_at" FROM "networks" AS n0 LIMIT 1 []
   radiator_1  | [info] Sent 404 in 869ms
   radiator_1  | [debug] ** (Ecto.NoResultsError) expected at least one result but got none in query:
   radiator_1  |
   radiator_1  | from n0 in Radiator.Directory.Network,
   radiator_1  |   limit: 1
   radiator_1  |
   radiator_1  |     (ecto) lib/ecto/repo/queryable.ex:76: Ecto.Repo.Queryable.one!/3
   radiator_1  |     (radiator) lib/radiator_web/controllers/login_controller.ex:70: RadiatorWeb.LoginController.sign_in_valid_user/3
   radiator_1  |     (radiator) lib/radiator_web/controllers/login_controller.ex:1: RadiatorWeb.LoginController.action/2
   radiator_1  |     (radiator) lib/radiator_web/controllers/login_controller.ex:1: RadiatorWeb.LoginController.phoenix_controller_pipeline/2
   radiator_1  |     (radiator) lib/radiator_web/endpoint.ex:1: RadiatorWeb.Endpoint.instrument/4
   radiator_1  |     (phoenix) lib/phoenix/router.ex:275: Phoenix.Router.__call__/1
   radiator_1  |     (radiator) lib/radiator_web/endpoint.ex:1: RadiatorWeb.Endpoint.plug_builder_call/2
   radiator_1  |     (radiator) lib/plug/debugger.ex:122: RadiatorWeb.Endpoint."call (overridable 3)"/2
   radiator_1  |     (radiator) lib/radiator_web/endpoint.ex:1: RadiatorWeb.Endpoint.call/2
   radiator_1  |     (phoenix) lib/phoenix/endpoint/cowboy2_handler.ex:33: Phoenix.Endpoint.Cowboy2Handler.init/2radiator_1  |     (cowboy) /app/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2
   radiator_1  |     (cowboy) /app/deps/cowboy/src/cowboy_stream_h.erl:296: :cowboy_stream_h.execute/3
   radiator_1  |     (cowboy) /app/deps/cowboy/src/cowboy_stream_h.erl:274: :cowboy_stream_h.request_process/3
   radiator_1  |     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
   

3. Open `/signup` again and try to signup with the same email address. Radiator will return an input validation message "*Email has already been taken*".

On application start, verify that ua_inspector is setup correctly by checking that its database is available.

See https://hexdocs.pm/ua_inspector/UAInspector.html#ready?/0

In case of failure, print a helpful message, mainly that it can be fixed by running mix ua_inspector.download --force.

A user needs to know the number of episodes in a podcast.

Not sure if there is a naming convention for this in GraphQL, otherwise I suggest episodes_count.

• network model as collection of podcasts
• one network has many podcasts
• one podcast belongs to one network
• graphql
  • get single network by id
  • get list of networks
  • create network
  • update network
  • when creating a podcast, assign a network

• change password
• add avatar
• change display name
• change email (more elaborate, needs confirmation of new email by sending a mail to the old account before it sticks)

When providing the same name as someone else did before in signup an Ecto.ConstraintError shows up. The same problem exists when using the same email as someone else used before.

A validation message should show up here instead of the server error.

To unblock frontend development

The repo name should be just "radiator", without "spark".

• delete or archive existing podlove/radiator repo
• rename this repo to podlove/radiator
• inform Slack about rename as local git configs need to be updated

When an episode or podcast is created, a "slug" must be generated and saved in the database. Slugs must be simple and URL-safe, so only a-z0-9 and dashes.

slug "Is this the end of the wōrld?"
"is-this-the-end-of-the-world"

Slugs must not be changed on update as this would lead to changing URLs.

Ideally slugs are not written on create but the first publishing of an entity to allow for preliminary draft titles.

• make Audio an attribute of Episode
• make AudioFiles discoverable
• create Audio in Network
• attach Audio to Episode

It is currently possible to publish an episode now but it must also be possible to schedule a publication for the future.

• add Radiator.Directory.Editor.Manager.schedule_episode(episode, datetime)
• add GraphQL endpoint
• validate given date/time is in the future

We have a mutation for login (authenticatedSession) but the initial user registration is missing.

Can be done by adding a lower case index on these columns and querying against that column.

create index(:users, ["(lower(email))"], unique: true)

via https://minhajuddin.com/2019/04/14/how-to-store-username-or-email-with-case-insensitive-search-using-ecto/

It's a temporary function that should not be necessary any more.

Blocker: Mix.Tasks.Radiator.Import currently uses it but should probably take a network id as an argument instead, or prompt with a choice of networks.

• store chapter images in S3/minio
• document storage/access of these images
• adjust importer (use atp.fm, fanboys.fm for testing)
• display in cms
• PWP4 support?

With the users from #9 implement a permission system working for the API and interface for Podcasts and Episodes

On cd assets && npm install, npm informs you of vulnerabilities which are shipped with the current dependencies:

                                                                                
                       === npm audit security report ===                        
                                                                                
# Run  npm install --save-dev [email protected]  to resolve 1 vulnerability
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate      │ Denial of Service                                            │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ js-yaml                                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ css-loader [dev]                                             │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ css-loader > cssnano > postcss-svgo > svgo > js-yaml         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/788                             │
└───────────────┴──────────────────────────────────────────────────────────────┘


# Run  npm install --save-dev [email protected]  to resolve 1 vulnerability
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate      │ Denial of Service                                            │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ js-yaml                                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ optimize-css-assets-webpack-plugin [dev]                     │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ optimize-css-assets-webpack-plugin > cssnano > postcss-svgo  │
│               │ > svgo > js-yaml                                             │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/788                             │
└───────────────┴──────────────────────────────────────────────────────────────┘


# Run  npm install --save-dev [email protected]  to resolve 2 vulnerabilities
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low           │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ braces                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ webpack-cli [dev]                                            │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ webpack-cli > jscodeshift > micromatch > braces              │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/786                             │
└───────────────┴──────────────────────────────────────────────────────────────┘


┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low           │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ braces                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ webpack-cli [dev]                                            │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ webpack-cli > webpack-addons > jscodeshift > micromatch >    │
│               │ braces                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/786                             │
└───────────────┴──────────────────────────────────────────────────────────────┘


# Run  npm update js-yaml --depth 4  to resolve 1 vulnerability
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate      │ Denial of Service                                            │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ js-yaml                                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ postcss-loader [dev]                                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ postcss-loader > postcss-load-config > cosmiconfig > js-yaml │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/788                             │
└───────────────┴──────────────────────────────────────────────────────────────┘


found 5 vulnerabilities (2 low, 3 moderate) in 16007 scanned packages
  run `npm audit fix` to fix 1 of them.
  4 vulnerabilities require semver-major dependency updates.

E.g.:

• show progress while importing
• download and upload all media
• use the chapter parsing to extract chapter information from mp3 id3 if there

Currently only S3 compatible storage works. However, we want to be more flexible. One use case is having external storage where Radiator simply knows the file URL and nothing else.

My first attempt is below. The idea is to still use Arc but add a :storage_type to the audiofile schema and switch to a different url/store implementation when it's "external_url".

Supposedly Arc.Actions.Url.url/3 is overridable, however my url/3 implementation never gets called and dialyzer also complains that it can never be called ¯\(ツ)/¯

diff --git a/lib/radiator/media/audio_file.ex b/lib/radiator/media/audio_file.ex
index 03ee231..e1cb21f 100644
--- a/lib/radiator/media/audio_file.ex
+++ b/lib/radiator/media/audio_file.ex
@@ -8,6 +8,7 @@ defmodule Radiator.Media.AudioFile do
   import Arc.Ecto.Changeset
 
   alias Radiator.Media
+  alias __MODULE__
 
   schema "audio_files" do
     field :file, Media.AudioFile.Type
@@ -15,6 +16,9 @@ defmodule Radiator.Media.AudioFile do
     field :mime_type, :string
     field :byte_length, :integer
 
+    field :storage_type, :string
+    field :external_url, :string
+
     has_many :attachments,
              {"episode_attachments", Media.EpisodeAttachment},
              foreign_key: :audio_id
@@ -39,6 +43,20 @@ defmodule Radiator.Media.AudioFile do
     RadiatorWeb.Router.Helpers.tracking_url(RadiatorWeb.Endpoint, :show, audio.id)
   end
 
+  # arc override
+  def url(
+        {_file, %AudioFile{storage_type: "external_url", external_url: url}},
+        _version,
+        _options
+      ) do
+    url
+  end
+
+  # default implementation
+  def url(file, version, options) do
+    super(file, version, options)
+  end
+
   # arc override
   def storage_dir(_version, {_file, audio}) do
     "audio/#{audio.id}"
diff --git a/priv/repo/migrations/20190420102616_create_audio.exs b/priv/repo/migrations/20190420102616_create_audio.exs
index 9f17d79..827f23b 100644
--- a/priv/repo/migrations/20190420102616_create_audio.exs
+++ b/priv/repo/migrations/20190420102616_create_audio.exs
@@ -8,6 +8,9 @@ defmodule Radiator.Repo.Migrations.CreateAudioFile do
       add :byte_length, :integer
       add :file, :string
 
+      add :storage_type, :string
+      add :external_url, :text
+
       timestamps()
     end
   end

(diff based on the public-file-access branch)

Progress

• Tracking
• Reporting
• API

Tracking

Download tracking works by exposing a tracking URL that goes through Phoenix before redirecting to the actual requested file. Phoenix asynchronously persists the download.

Attributes of the tracking system:

• downloads must still work even if tracking system is slow or down
• bot requests (or in general: not real downloads) are detected and discarded
• duplicate requests are detected and merged

In general, IAB Guidelines should be followed.

Analytics

Analytics provide aggregated views on the raw download data. For good viewing performance independent from amount of downloads, data needs to be pre-calculated. Side benefit of pre-calculating all reports is that old raw tracking data can be discarded.

Podcast Reports

• total downloads
• downloads per month
• downloads per day
• User Agent analytics per month (client, operating system, device type)
• Geo Analytics (countries, cities via maxmind GeoLite2)

Episode Reports

• total downloads
• downloads per month
• downloads per day
• downloads [1d/2d/3d/4d/5d/6d/1w/2w/4w/1q/2q/3q/1y] after release, providing numbers relative to publication date/time (time frames taken from Podlove Publisher; however, how useful are time frames > 1q, really?)

API

All reporting data must be accessible via API.

Boring Tech Stack

We considered using elastic stack for tracking & analytics. However, the longer I look at it the more daunting the infrastructure complexity becomes. Yes, it’s a great stack for wading through log data but it’s also a whole stack that needs to be learned, setup and maintained. And since I know that the boring approach, logging download requests into Postgres, works, I have a hard time making elastic a core dependency at this point.

• generate/set if empty
• do not generate if provided
• add "regenerate" function
• render in feed

Due to password hashing, creating users in tests takes a long time (500ms on my MBP).

Find a way to make this way quicker in test env.

Related to #1

As per discussion here #29 - split up the login controller in multiple parts. One suggestions is to split out the actual login and signup/register actions into their own module.

• add web player to project
• add configuration endpoint for episode player configs

Import podcasts from existing RSS feed - also good for testing.

• Split out metalove from podlove-preview and make it a minimal hex package to include here
• API
• GUI
• mix task

Depends on #45

We need a generic way to paginate lists of documents.

See https://graphql.org/learn/pagination/

High Level Goal: publish an audio without being in an episode or podcast

Currently there is Episode with all its metadata, which has AudioFile attachments.

A new layer of abstractions is required in between those two: Audio, representing an audio object with all its metadata that can stand on its own and be played within a web player. It will take most of the metadata that is now part of Episode. Episode can be seen as the (optional) relation of an audio to a podcast. Episode only keeps metadata that is purely required for an episode but not an audio. There may be fields that both have, and a precedence must be defined.

For example, both Episode and Audio will have a publish date and a title. In the episode context, the episode values take precedence.

If an Audio does not belong to an Episode, it still has to belong to something. I suggest the Network.

    +---------+                 +---------+
    | Episode +------+   +------+ Network |
    +---------+      |   |      +---------+
                     |   |
                   +-v---v-+
       +-----------+ Audio +----------+
       |           +---+---+          |
       |               |              |
       |               v              |
+------v-----+   +-----+-----+  +-----v----+
| Transcript |   | AudioFile |  | Chapters |
+------------+   +-----------+  +----------+

(edit diagram with http://asciiflow.com/)

It feels like GraphQL code should live in its own directory instead of mixed in between all other web concerns, for example RadiatorWeb.GraphQL.*.

via #23 (comment)

Some resources for authentication:

• guardian (based on JWT https://jwt.io)
• comeonin (password hashing)
• I have a demo app for these https://github.com/eteubert/phoenix-auth-demo/ (maybe useful for a first look, however using outdated versions of Phoenix and the libraries above)

Uploading is currently documented in https://github.com/podlove/radiator/blob/master/guides/Files.md This will be out of date once the Audio restructuring is done.

It would be good to include a JS example on how to upload using gql.

Have a look if this is something to pursue:

• https://github.com/jaydenseric/graphql-upload
• https://github.com/jaydenseric/graphql-multipart-request-spec

Since it's downloading episodes and other files it should be something easily identifyable as a bot so tracking software can exclude it. I suggest including "bot" as it's often already filtered out by default.

Maybe "Podlove Radiator Import Bot"? Or something more creative? :)

Looks promising, see https://github.com/podlove/radiator-spark/tree/graphql

• server side with Absinthe is very sexy
• self documenting
• good tooling, for example https://electronjs.org/apps/graphql-playground
• frontend probably with https://www.apollographql.com/ as it does not require react, but haven't got it working yet

Should be used to configure what needs to be configured. E.g. at least:

• main user account
• smtp stuff
• domain
• storage options

also needs a data model for "publishing"

Needs Editor Context to be fully functional and adopted everywhere first.

Remove the following functions:

• Radiator.Directory.get_network!
• Radiator.Directory.get_podcast!
• Radiator.Directory.get_episode!

and replace their usage with non-bang-versions. Especially in controllers that means handling error cases instead of just raising in the user's face.

⚠️ Don't start before #76 is merged

• make import task use the existing infrastructure
• make import in phoenix ui use the task infrastructure
• let imports be aborted
• make imports not published as default, but add an import option to the api for this

As a podcaster
I want to be able to add images to my chapter marks

Acceptance Criteria

1. images need to be uploaded to s3/minio
2. import should also import the images

Currently the upload to the S3 style storage just returns a URL that is in turn used in the metadata. We need a blob abstraction that we use instead, which e.g. has a notion of source, derived data, public urls, etc.

Create step-by-step deployment guide to get from zero to production-ready running system.

Since we're on Docker, Radiator can be hosted on any Docker-compatible architecture. But for the sake of getting started, I think the guide should be very specific, for example using DigitalOcean so each step can be replicated with no prior knowledge.