radiorabe / suisa_sendemeldung Goto Github PK

Overview

The CSV output shall be adapted to match with the SUISA document Gemeinsamer Tarif S 2020 – 2023, as published on the SUISA Radio and TV page.

Background

The following parts are relevant from the Gemeinsamer Tarif S 2020 – 2023 document:

There's an additional template called Vorlage für Sendemeldungen: Gemeinsamer Tarif S:

• This template has the same fields as the Anhang I mentioned within paragraph 33, but includes an additional field named Identifikationsnummer which matches with the mentioned data "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" of paragraph 34, which is missing in Anhang I.
• The template seems to uses a coloring scheme (gray and white), whereas the grey fields match with the required data mentioned in paragraph 34 (except for Label Code, which is probably only a formatting error, as it's not mentioned in paragraph 34 although Label and Label Codeare marked as mandatory, only one is required according to clarifications with SIUSA as of 2023-02)

Hypothesis: The data under paragraph 34 is mandatory, whereas all the other data can be considered optional.

This would imply that at least the following fields with the names below, have to be provided:

• Titel: Titel sind in Orginalsprache gemäss Tonträger, inklusive ggf. Versionsabgaben ("live in London" "dirty remix" etc.) anzugeben
• Komponist: Name des/der Komponisten
• Interpret: Name des (Haupt-) Interpreten, bzw. Gruppenname
• Sendedatum: Datum der Nutzung im Einheitformat, z.B. YYYYMMDD - 20110101
• Sendedauer: Dauer der Nutzung im Einheitsformat, z.B. hh:mm:ss - 00:03:54
• Sendezeit: Startzeitpunkt der Nutzung im Einheitsformat, z.B. hh:mm:ss - 06:09:13
• ISRC: ISRC Code des Quelltonträgers
• Label: Name des Labels
• Identifikationsnummer: Vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer

The header of the current CSV output looks as follows and currently doesn't exactly match with the fields from the template:

Sendedatum,Sendezeit,Sendedauer,Titel,Künstler,Komponist,ISRC,Label

Proposal

To match with the template from the Gemeinsamer Tarif S 2020 – 2023 document the CSV header and field ordering shall be adapted as follows:

Titel,Komponist,Interpret,Sendedatum,Sendedauer,Sendezeit,ISRC,Label,Identifikationsnummer

• Name the fields (within the CSV header row) exactly the same as within the template:
  • Künstler 👉 Interpret
• Use the same field ordering as within the template
• For the Identifikationsnummer the PR #185 could be used
• Ensure that additional fields (with possible empty values) could be introduced (depending on the outcome of the clarifications mentioned below)

Additionally, the date format of the Sendedatum shall be changed from DD/MM/YY to YYYYMMDD in order to match with the example from the template.

The name of the CSV file shall be changed from suisa_sendemeldung_<MONTH>.csv to <RADIO-STATION>_<YYYY>_<MM>.csv (e.g. my-station_2023_01.csv), in order to match with the naming convention of Anhang I.

Preliminary clarifications

Before the change can be made, the following preliminary clarifications should be made with SUISA:

• Is the Gemeinsamer Tarif S 2020 – 2023 document still valid for 2023, or is there a newer superseding one?

  Gültigkeitsdauer (click to expand)

  49 Dieser Tarif ist vom 1. Januar 2020 bis zum 31. Dezember 2022 gültig.
  [...]
  50 Die Gültigkeitsdauer des Tarifs verlängert sich automatisch um jeweils ein Jahr bis längstens 31. Dezember 2025, wenn er nicht von einem der Verhandlungspartner durch schriftliche Anzeige an den anderen ein Jahr vor Ablauf gekündigt wird.

  Feedback from SUISA as of 2023-02:
  Yes the Tarif is still valid for 2023 as confirmed by SUISA on 2023-02-02

• Is the assumption correct, that the fields corresponding to the data mentioned in paragraph 34 are mandatory and everything else is optional?

  Feedback from SUISA as of 2023-02:
  Yes, that's correct. The following note applies to the fields Label and Label Code: Only one of them must be provided, that's why both cells are marked as mandatory (grey background) in Vorlage für Sendemeldungen: Gemeinsamer Tarif S

• Why is the there no field matching the data "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" reference within paragraph 34, in Anhang I?

  Feedback from SUISA as of 2023-02:
  The field has to be provided and is named Identifikationsnummer as assumed

• Does SUISA expect the report to contain all fields from the template or only the mandatory ones?

  Feedback from SUISA as of 2023-02:
  Yes, all fields are expected, but only values for the mandatory ones must be provided, the format and field ordering must be based on the Vorlage für Sendemeldungen: Gemeinsamer Tarif S

• What value is expected in case a certain field is not applicable ("Sofern zutreffend/bekannt"), e.g. empty value or N/A?

  Feedback from SUISA as of 2023-02:
  An empty value is expected.

• Is it still OK to deliver the report as CSV or is an XLSX format (same as the template) required?

  Feedback from SUISA as of 2023-02:
  SUISA expects the report in XLSX format for 2023 based on the Vorlage für Sendemeldungen: Gemeinsamer Tarif S and not CSV anymore, addressed in #203

• Request approved example entries for multiple genres (rock/pop, classic, jazz and electronics as well as jingles)

  Feedback from SUISA as of 2023-02:
  Example entries have been provided.

This would make the integration into a monitoring system easier

SUISA expects the reports to be delivered in the Excel format (2007 - 365, .xlsx), based on Vorlage für Sendemeldungen: Gemeinsamer Tarif S for 2023.

Either the reports must be generated directly in the Excel format using a Python module (such as XlsxWriter or openpyxl) or a CLI post-conversion task (such as csv2xlsx) must be implemented.

This issue is related to #187

Looking at how travis is failing on #48 we should consider bumping to py36.

It should be possible to set cc and bcc adresses

This line evaluates to 0 (12 is intended)

It should be possible to use date strings like %Y, %m, %B.. in the text and body of the email

I did some checking against the ISRC spec using iso3901 and there are several records in last years data with wrong ISRC records.

Examples

time,isrc,artist,title
2022-01-05 21:48:36,ISRCUKW3Z2000162,Razorlight,Burn, Camden, Burn
2022-01-14 03:09:41,ISRCUKW3Z2000162,Razorlight,Burn, Camden, Burn
2022-01-22 07:01:33,ISRCQZ8GX1700810,Jake Isaac,Tomorrow feat. The Kingdom Choir
2022-02-15 00:48:42,QAN65701,Khalid Al Iraqi,Safana
2022-04-03 18:36:22,ISRCGBBLY2000218,Kim Wilde,Cambodia (2020 Remaster)
2022-05-19 06:24:25,3283451081126-1,Hairy Diamond,Givin up
2022-05-20 13:14:40,DEAW1050019,Slackwax,Night Out - Original Mix
2022-12-01 13:17:45,ISRCUKW3Z2200668,LOWLIFE/RAT BOY/MILA,FUCKED UP
2022-12-17 00:46:40,ISRCUKW3Z2200668,LOWLIFE/RAT BOY/MILA,FUCKED UP
2022-12-28 19:08:55,QAN594474,Mohammed Mounir,El Leila Bayet Andena
2022-12-31 18:32:27,3283451081126-2,Bobby Blanco & Miki Moto,Black Sugar

Proposal

I want to implement a workaround for the ISRC prefix case, figure out what is wrong with the others, and also implement not-sending ISRC those records. With the aim being that we rather send an empty string than a clearly invalid isrc (like the 3283451081126-2 case).

Given the examples, i think we have a reasonably low amount of errors (5 for all of 2022 with the ISRC wrokaround).

from datetime import datetime, timedelta

from iso3901 import ISRC
import requests_cache

# python -m pip install git+https://github.com/radiorabe/suisa_sendemeldung.git
from suisa_sendemeldung.acrclient import ACRClient
from suisa_sendemeldung.suisa_sendemeldung import get_artist

access_key = ""
stream_id = ""
year = 2022

requests_cache.install_cache("suisa_sendemeldungen_cache")

print(f"time,isrc,artist,title")
client = ACRClient(access_key)
for month in range(1, 13):
    # get proper start and end date for month (ACRClient takes care of rest)
    start = datetime(year, month, 1)
    end_year = year + 1 if month == 12 else year
    end_month = 1 if month == 12 else month + 1
    end = datetime(end_year, end_month, 1) - timedelta(days=1)

    # calculate total seconds in month based on days in said month
    total_duration = end - start
    total_duration_s = total_duration.days * 86400

    # get dat data from acrcloud
    data = client.get_interval_data(stream_id, start, end)
    for entry in data:
        metadata = entry.get("metadata")
        timestamp = datetime.strptime(metadata.get("timestamp_local"), ACRClient.TS_FMT)

        try:
            music = metadata.get("music")[0]
        except TypeError:
            music = metadata.get("custom_files")[0]
        artist = get_artist(music)
        title = music.get("title")

        if music.get("external_ids") and len(music.get("external_ids")) > 0:
            isrc = music.get("external_ids").get("isrc")
        elif music.get("isrc"):
            isrc = music.get("isrc")
        # validate if not empty
        if isrc and not ISRC.validate(isrc):
            print(f"{timestamp},{isrc},{artist},{title}")

Right now it only supports a single, to, cc, or bcc recipient. This leads to fun situations where configuring multiple cc addresses in the config file leads to only the last address specified being used as a cc recipient instead of it composing a mail with multiple cc recipients.

We should allow specifying the relevant params several times and allow composing mails with several to, cc, and bcc recipients,

I had this stacktrace:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/user/suisa_sendemeldung/suisa_sendemeldung/suisa_sendemeldung.py", line 793, in <module>
    main()
  File "/home/user/suisa_sendemeldung/suisa_sendemeldung/suisa_sendemeldung.py", line 738, in main
    data = get_xlsx(data, station_name=args.station_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/suisa_sendemeldung/suisa_sendemeldung/suisa_sendemeldung.py", line 569, in get_xlsx
    csv = get_csv(data, station_name=station_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/suisa_sendemeldung/suisa_sendemeldung/suisa_sendemeldung.py", line 517, in get_csv
    release_date = funge_release_date(music.get("release_date", ""))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/suisa_sendemeldung/suisa_sendemeldung/suisa_sendemeldung.py", line 379, in funge_release_date
    return datetime.strptime(release_date, "%Y-%m-%d").strftime("%Y%m%d")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '0000-00-00' does not match format '%Y-%m-%d'

Example:

• Use [email protected] as login for the smtp server
• Use [email protected] as actual sender of the email (From: )

This issue provides visibility into Renovate updates and their statuses. Learn more

Repository problems

These problems occurred while renovating this repository.

• WARN: App has not been granted permissions to update Workflows - aborting branch.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

• chore(deps): update crazy-max/ghaction-docker-meta action to v3

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update python:3-alpine docker digest to 78604a2

• Check this box to trigger a request for Renovate to run again on this repository

Got this error on a fairly vanilla Fedora 36 installation:

Traceback (most recent call last):
  File "/usr/lib64/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/user/suisa_sendemeldung/suisa_sendemeldung/suisa_sendemeldung.py", line 24, in <module>
    from dateutil.relativedelta import relativedelta
ModuleNotFoundError: No module named 'dateutil'

I think if we have a config file for pre-commit it also makes sense to add it to the dev dependencies. What do you think @hairmare?

The script fails when cc and bcc adresses are not provided. The following exception is raised:

Traceback (most recent call last):
  File "./suisa_sendemeldung/suisa_sendemeldung.py", line 289, in <module>
    main()
  File "./suisa_sendemeldung/suisa_sendemeldung.py", line 278, in main
    args.email_text, filename, csv, cc=args.email_cc.split(','),
AttributeError: 'NoneType' object has no attribute 'split'

Some of our files are not in the database and require a manual upload to ACRCloud. The recorded metadata are then returned in an array custom_files:

    {
        "status": {
            "msg": "Success",
            "code": 0,
            "version": "1.0"
        },
        "result_type": 0,
        "metadata": {
            "timestamp_utc": "2023-01-31 14:44:51",
            "played_duration": 507,
            "type": "delay",
            "custom_files": [
                {
                    "album": "",
                    "play_offset_ms": 17320,
                    "sample_begin_time_offset_ms": 1140,
                    "score": 89,
                    "title": "Future Shock (Club Mix)",
                    "release_date": "",
                    "Artist": "STARTER",
                    "sample_end_time_offset_ms": 9740,
                    "count": 49,
                    "label": "",
                    "db_end_time_offset_ms": 17320,
                    "bucket_id": "16272",
                    "db_begin_time_offset_ms": 8720,
                    "artists": "STARTER",
                    "duration_ms": 524817,
                    "isrc": "CHAAA2116394",
                    "acrid": "22f5455cd10e288a30a1aabf3144938a"
                }
            ],
            "timestamp_local": "2023-01-31 14:44:51"
        }
    },

These additional metadata should also be recorded by the script.

I've recently shown nowplaying quite some 🐍 love and want to ensure that SUISA sendemeldung also benefits from that.

• configure CI (actions, dependabot) #68
• configure CD (pypi, semantic-release) #169
• implement testing #170
• deprecate python 3.8, switch to 3.11 on ubi9 #306
• switch to poetry instead of soon to be deprecated setuptools #306
• replace twine with poetry for pypi publishing #306
• introduce quality gates (mypy, ...) #395
• harmonize logging with otel (+ further telemetry like traces and metrics)
• use radiorabe/python-minimal container image #118
• cleanup README.md
  • should also dox #289
• helm chart

Basically, what i wat for suisa_sendemeldung is for it to be in a state where consistent and regular rollouts + upgrades are a thing (like with nowplaying which is currently in a quite ok state).

There seems to be a new version of the api and we should porbably stop using the legacy api at some point. The main difference being that we need a modern access token that is transmitted using a bear-token style Authorization header instead of a get parameter.

The data we currently use is available on the new https://api-v2.acrcloud.com/api/bm-cs-projects/:pid/streams/:stream_id/results endpoint. The bearer token also works for a bunch of other data, like track metadata or the modern UCF projects.

As part of this issue it might be time to prepare the current ACRClient class into something that can be used more broadly (i specifically have some ideas wrt scanning our existing archives in mind).

I'm not assuming the v2 api to be more well formed, but some testing might tell us more. Should we consider to build this into a validating and normalising client to ensure we cover the full acr cloud format?

Overview

The mail report template shall be adapted to reference certain paragraphs from the SUISA document Gemeinsamer Tarif S 2020 – 2023, as published on the SUISA Radio and TV page. It shall be clearly stated, that the report confirms to the terms of the tariff.

Relevant paragraphs

The following citations are a summary of relevant paragraphs from the Gemeinsamer Tarif S 2020 – 2023 to be referenced within the mail template:

Gemeinsamer Tarif S 2020 – 2023

G. Verzeichnisse
31 Die Sender stellen der SUISA die in diesem Abschnitt G aufgeführten Angaben zu. [...]

32 Die Sender melden die unter Buchstabe G genannten Programmangaben in elektronischer Form in einem standardisierten importierbaren Format.

34 Die Angaben enthalten [...]

d) Termine
45 Alle Angaben sind der SUISA [...] monatlich jeweils bis zum Ende des folgenden Monats zuzustellen. [...]

46 Erkennen oder vermuten die SUISA und/oder die SWISSPERFORM Fehler oder Lücken in den Angaben des Senders, beanstanden sie diese innerhalb von drei Monaten gegenüber dem Sender und gewähren diesem eine Nachfrist von 45 Tagen zur Behebung der Mängel.

20 Sender, welche ihr System auf die Meldepflichten gemäss Buchstabe G einrichten und korrekt und rechtzeitig nach diesen Meldepflichten melden, erhalten auf der Abrechnung einen Rabatt von 5 %.

Proposal

Proposal from @hairmare

Hallo SUISA

Im Anhang finden Sie die Sendemeldung von Radio Bern RaBe für den Dezember 2022.

Wir senden ihnen diese Sendemeldung da wir bis Ende 2021 nicht informiert
wurden, dass Radio Bern RaBe im laufenden Jahr von der Meldepflicht befreit ist.

In der Sendungsmeldung enthalten sind die unter Buchstaben G im Tarif
"Gemeinsamer Tarif S 2020 - 2023" genannten Programmangaben in elektronischer
Form. Es handelt sich bei der Datei um eine Comma-Separated Values (CSV)
Datei welche dem RFC 4180 Standard folgt.

Diese Sendemeldung enthält unter anderem folgende Angaben:

• Titel des Musikwerks
• Name des Komponisten
• Name des bzw. der Hauptinterpreten
• Label
• ISRC
• vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer
• Sendezeit
• Sendedauer

Die Datei folgt nach bestem Wissen und Gewissen und wo dies technisch möglich
ist der Vorlage in "Gemeinsamer Tarif S 2020 - 2023" S. 15 ff.

Das Feld ISRC ist dabei mindestens in Fällen wo der ISRC zusammen mit der Aufnahme
vom Lieferanten der Aufnahme in irgendeiner Form mitgeteilt bzw. mitgeliefert
wurde angegeben. Wir ergänzen diese Angaben wo möglich aus unseren eigenen
Datenbeständen und arbeiten zu diesem Zweck auch mit externen Partnern zusammen.
Nachmeldungen und Korrekturen von ISRC werden wir selbstverständlich sofort
verarbeiten und ihnen mitteilen.

Wir bitten sie höflich uns den Erhalt wie auch die erfolgreiche, automatisierte
Verarbeitung dieser Sendemeldung zu bestätigen. Bitte kontaktieren sie uns
dringlich und unverzüglich falls diese Sendemeldung Mängel wie Inkorrektheit
oder andere Missstände wie Fehler oder Lücken aufweist. Beanstandungen zu dieser
Sendemeldung lassen sie uns bitte in den nächsten drei Monaten vor dem 31. März 2023
zukommen. Im Falle einer Beanstandung streben wir an etwaige Mängel raschmöglichst
innerhalb der Nachfrist von 45 Tagen zu beheben.

Diese Email wurde automatisch generiert. Bei vertraglichen Fragen erreichen
sie uns an besten zu Bürozeiten unter 012 345 67 89. Technische Anliegen
im Zusammenhang mit der Sendemeldung melden sie bitte ebenda oder bevorzugt
direkt per Email an [email protected].

Freundliche Grüsse,
Radio Bern RaBe