Like script used to update data catalog (from LOD API), we could add a script to update dashboard data (from agg API)

ideally something defined by an external standard.
Low prio, probably improvement post launch.
For now assigning @mbrinkerinkbeng, but Evelien and Jesse might also have input on this.

Vegalite (I think) causes an error on (re)load of dataset page. Can be avoided by navigating from home page

500
structuredClone is not defined

https://github.com/beeldengeluid/labs.beeldengeluid.nl/blob/006e1d7d93a205146cf15a249b5aa1dc823f468d/nuxt.config.js#L20

in the html:

<meta data-n-head="ssr" data-hid="description" name="description" content="">

We could use e.g. the text under 'about' on /.

Sound & Vision LABS represent our Research & Innovation activities. On a wide range of topics, we put media innovation into practice by connecting new technologies and innovative ideas to real data ("big multimedia data") and evaluating them in real-life use scenarios in media, cultural heritage, research, education and journalism, together with real users.

support localization.

Feedback René about the current ontospy solution:

• It would be easier if the menu on the lefthand side is always open
• namespaces are not so interesting
• misses the information about which classes a class is related to (e.g. what classes can be linked to a CreativeWork (Mari: this information is available via the properties list at the bottom, but it would perhaps be clearer if there were also a separate list of the related classes)
• preferably you would have this overview per dataset, as some classes/properties are irrelevant for certain datasets (Mari: I think this is more for the future, if ever)

Tasks:

• Investigate if we can configure ontospy to keep the menu open
• Consider (given that namespaces are not so interesting) whether a different page should be Home, and see if ontospy allows this
• Consider how to make it clearer which classes a class is related to
• Review any changes with users

Width and height are now hard-coded in specification, should scale nicely

Also, it would be nice if the user can zoom in on interesting area's (especially in the map visualisation)

Feedback René: would expect this from Podiumkunsten. RDA data would need improvement first, blank nodes etc are a problem

Mari: I assume not a high priority at present

For some future point. Publishing manually/semi-automatically via the ODL spreadsheet is fine for now.

Possibly a tool such as https://datahub.io/ could be useful to us.

STORY B8 As an innovation manager I want to have a clear set of guidelines for users of the data so I can make an appeal on third parties to frequently update the dataset and tackle privacy issues and such that might arise.

For reference: https://www.europeana.eu/en/rights/usage-guidelines-for-metadata

These pieces of information appear as if they should be links - especially the rights statement - and invite to be clicked on to find out more, but they are not clickable. Would expect to go to the metadata when clicking on the number of records.

Tasks:

• Consider if it is feasible to make (some of) these links clickable
• If not, consider altering their appearance so it is clear that they are static information

Acceptance criteria:

• Either these pieces of information are clickable links, or it is clear that they are static pieces of information

STORY B9: As a third party developer I want to be able to harvest the open metadata using a harvesting protocol set so I can be sure the data is synchronised with the source.

Acceptance criteria

• using OAI-PMH

No context is provided. This is needed because it contains the namespaces etc.

Currently, this is what you get with the download button:

{
   "@id": "http://data.beeldengeluid.nl/id/dataset/0024",
   "@type": "sdo:Dataset",
   "sdo:description": {
      "@language": "nl",
      "@value": "Politiek, cultuur en sport, kortom alle facetten van het leven komen aan bod aan de tafel van Frits Barend en Henk van Dorp, immer bijgestaan door oud-voetballer en columnist Jan Mulder. Naast Jan Mulder zijn schrijver Boudewijn Büch en politiek verslaggever Frits Wester vaste gasten; ze schuiven regelmatig aan de tafel aan. Aanvankelijk wordt het programma in een café/restaurant in Eemnes opgenomen. Later verhuizen de opnamen naar Gooiland en sinds 2002 komen de uitzendingen vanaf het Media Park. In een gezellige sfeer, met nootjes op tafel. Spraakmakende gasten, emotionele onderwerpen en opvallende uitspraken maken dat je Barend & Van Dorp gezien moet hebben. In totaal worden er 960 afleveringen gemaakt. In 2008 dragen Frits Barend en Henk van Dorp alle afleveringen van de talkshow over aan Beeld en Geluid."
   },
   "sdo:distribution": {
      "@id": "http://data.beeldengeluid.nl/id/datadownload/0023"
   },
   "sdo:inLanguage": "nl-NL",
   "sdo:includedInDataCatalog": {
      "@id": "http://data.beeldengeluid.nl/id/datacatalog/0001"
   },
   "sdo:license": {
      "@id": "http://rightsstatements.org/vocab/InC/1.0/"
   },
   "sdo:mainEntityOfPage": {
      "@id": "https://wiki.beeldengeluid.nl/index.php/Barend_%26_Van_Dorp_(praatprogramma)"
   },
   "sdo:name": {
      "@language": "nl",
      "@value": "Barend & van Dorp"
   },
   "sdo:publisher": {
      "@id": "https://www.beeldengeluid.nl"
   },
   "sdo:size": "910",
   "sdo:temporalCoverage": "2002/2006"
}

But, as you can see above, no proper prefixes of namespaces are listed. What should be provided is what the beng-lod-server serves (Please note that we did cut the contents of the @context, because rdflib is providing too many namespaces as of the latest version. We will fix that later):

{
  "@context": {
    "sdo": "https://schema.org/",
  },
  "@graph": [
    {
      "@id": "http://data.beeldengeluid.nl/id/dataset/0024",
      "@type": "sdo:Dataset",
      "sdo:description": {
        "@language": "nl",
        "@value": "Politiek, cultuur en sport, kortom alle facetten van het leven komen aan bod aan de tafel van Frits Barend en Henk van Dorp, immer bijgestaan door oud-voetballer en columnist Jan Mulder. Naast Jan Mulder zijn schrijver Boudewijn Büch en politiek verslaggever Frits Wester vaste gasten; ze schuiven regelmatig aan de tafel aan. Aanvankelijk wordt het programma in een café/restaurant in Eemnes opgenomen. Later verhuizen de opnamen naar Gooiland en sinds 2002 komen de uitzendingen vanaf het Media Park. In een gezellige sfeer, met nootjes op tafel. Spraakmakende gasten, emotionele onderwerpen en opvallende uitspraken maken dat je Barend & Van Dorp gezien moet hebben. In totaal worden er 960 afleveringen gemaakt. In 2008 dragen Frits Barend en Henk van Dorp alle afleveringen van de talkshow over aan Beeld en Geluid."
      },
      "sdo:distribution": {
        "@id": "http://data.beeldengeluid.nl/id/datadownload/0023"
      },
      "sdo:inLanguage": "nl-NL",
      "sdo:includedInDataCatalog": {
        "@id": "http://data.beeldengeluid.nl/id/datacatalog/0001"
      },
      "sdo:license": {
        "@id": "http://rightsstatements.org/vocab/InC/1.0/"
      },
      "sdo:mainEntityOfPage": {
        "@id": "https://wiki.beeldengeluid.nl/index.php/Barend_%26_Van_Dorp_(praatprogramma)"
      },
      "sdo:name": {
        "@language": "nl",
        "@value": "Barend & van Dorp"
      },
      "sdo:publisher": {
        "@id": "https://www.beeldengeluid.nl"
      },
      "sdo:size": "910",
      "sdo:temporalCoverage": "2002/2006"
    },
    {
      "@id": "https://www.beeldengeluid.nl",
      "@type": "sdo:Organization",
      "sdo:alternateName": {
        "@language": "nl",
        "@value": "Beeld en Geluid"
      },
      "sdo:name": {
        "@language": "nl",
        "@value": "Nederlands Instituut voor Beeld en Geluid"
      },
      "sdo:sameAs": [
        {
          "@id": "https://data.overheid.nl/community/organization/beeld-en-geluid"
        },
        {
          "@id": "http://www.openbeelden.nl/users/beeldengeluid"
        },
        {
          "@id": "https://data.netwerkdigitaalerfgoed.nl/beeldengeluid"
        },
        {
          "@id": "https://data.overheid.nl/community/organization/nlpo12345678"
        },
        {
          "@id": "http://www.wikidata.org/entity/Q1131877"
        },
        {
          "@id": "https://grid.ac/institutes/grid.425952.d"
        }
      ]
    },
    {
      "@id": "http://data.beeldengeluid.nl/id/datadownload/0023",
      "@type": "sdo:DataDownload",
      "sdo:contentUrl": {
        "@id": "https://euscreen.eu/series.html?id=EUS_513F04B47411BCDD8C80532EE577FAC4&activeItem=EUS_003399A7DB2D2A6B14313EF963DE4421"
      },
      "sdo:description": {
        "@language": "nl",
        "@value": "Actuele dagelijkse talkshow van Frits Barend en Henk van Dorp. Vaste gast is columnist Jan Mulder. De gasten zitten gezamenlijk aan tafel en reageren op elkaars gespreksonderwerpen."
      },
      "sdo:encodingFormat": "text/html",
      "sdo:inLanguage": "nl-NL",
      "sdo:license": {
        "@id": "http://rightsstatements.org/vocab/InC/1.0/"
      },
      "sdo:name": {
        "@language": "nl",
        "@value": "Barend & van Dorp op EUScreen"
      },
      "sdo:usageInfo": {
        "@id": "https://euscreen.eu/about.html"
      }
    }
  ]
}

Also, datadownloads that are part of the dataset are not listed.

Feedback ophalen van:
Polifonia

• Marilena
• Paul
• Ahsan
  Podiumkunsten
• Monique?
• Eric?
  NDE
• Bob Coret

Currently, I either added all the data inside the vegalite spec (or I used a web URL (github raw)).
It's nicer/ more maintainable to separate the specification from the data.
I think it should be possible to use a file from the repo directly, but I'm not sure how to do this (in the context of Vue/Nuxt)

Produce website Website statististics, like:

• number of requests
• number of unique users
• list of users
• most frequent pages requested
• ...

Use server logs, matomo, or other...

So, additionally, we might generate these statistics also for other data products.

For example... I created a simple shell script that extracts the clients from the server logs:

[wmelder@goldfinger-8 ~]$ cat whousescatapis.sh
for N in `cat /var/log/httpd/cat.apis.beeldengeluid.nl_access.log|cut -d ' ' -f 1 | sort | uniq`
do
        echo -e "$N\t=>\t$(nslookup $N| cut -d '=' -f 2)"
done

[wmelder@goldfinger-8 ~]$ sudo ./whousescatapis.sh
145.126.111.228 =>       wlan111228.wireless.utwente.nl.

Authoritative answers can be found from:
145.126.69.145  =>       wlan069145.wireless.utwente.nl.

Authoritative answers can be found from:
162.240.160.168 =>       server.ewq.pci.mybluehostin.me.

Authoritative answers can be found from:
172.19.53.43    =>       jackspang.beeldengeluid.nl.
31.201.102.24   =>       24-102-201-31.ftth.glasoperator.nl.

Authoritative answers can be found from:
37.161.123.171  =>      ** server can't find 171.123.161.37.in-addr.arpa: NXDOMAIN
52.114.75.216   =>       weu-azsc-urlp.cloudapp.net.

Authoritative answers can be found from:
54.220.158.70   =>       ec2-54-220-158-70.eu-west-1.compute.amazonaws.com.

Authoritative answers can be found from:
87.7.247.26     =>       host-87-7-247-26.retail.telecomitalia.it.

Authoritative answers can be found from:

see e.g. https://developers.google.com/search/docs/advanced/structured-data/generate-structured-data-with-javascript#custom-javascript

Currently:

1. A vega-lite spec is loaded in pages/datasets/[...slug].vue
2. it can then be used to populate a VegaView element in components/dataset/TabDashboard.vue

However, you would like to

1. Load all specs belonging to a specific dataset at once (from all files in one folder, or even all from one file??)
2. and then embed the Vegaview elements in the .md-file for that specific dashboard

Based on the new content from @rozelemarijn.

See 'doelstelling 3': https://docs.google.com/document/d/1SIrozvyKbLV4JAjYBRbzUOzgApkpflB9qYUVllP0Cag

like in PeerTube

The following information should be displayed on the dataset page, where applicable:

• Links to available tools (e.g. Media Suite)
• Dataset description
• Links to available APIs (e.g. Analysis API)
• Links to NDE/Clariah registers
• Links to any data downloads
• Link to the dataset on the SPARQL endpoint
• Link to the Linked Data schema

Comment moved from this issue:
"Consideration: improve presentation of LOD info of dataset on the dataportal page. To make it clear there is LOD for a dataset. E.g. don't put it on a tab, also consider logo/colors for infobox."

Tasks:

• Get user feedback (René) as to the desired location of this information (e.g. Description, Metadata...?)
• Determine where to store the information (spreadsheet, dataset description in register...?)
• Design a new layout for the page
• Add the information for the MOZ
• Display the information on the MOZ page - just in markdown
• Review with team, adjust
• Work out how best to display the information, implement this
• Review with René
• Add the information used in the final version to the MOZ document in the datasets folder, and update the README to encourage adding this information

Acceptance criteria:

• The desired information is shown on the MOZ page

See Publication datasets appendix for more details
Tasks:

• Add the dataset to the spreadsheet
• Add a document describing the dataset to the datasets folder
• Run script to publish the dataset to our catalog
• Run script to publish the dataset to the register

• Remove labs and topics tab
• Archive current projects, remove tab
• Blogs become showcases
• Update logo, labs -> data

In datasetregister naar pagina op dataplatform linken

Feedback René: Access to linked data is currently via the (example queries in the) Comunica endpoint or the resource pages (e.g. https://data.beeldengeluid.nl/id/program/2101608110093854631). The results in the endpoint are not clickable, and the resource pages do not support easy browsing as they have too few links. So if, for example, you see a list of concerts for the MOZ in the endpoint, you can't click on one of the concerts to find out more about it. And if you click through from the resource page for a concert to a person, you can't see the concerts that person is involved in. Whereas it is exactly this sort of clicking through the graph that you would expect for linked data. THIS IS VERY IMPORTANT

Would expect to be able to reach the data easily from the dataset page, but also from the data catalog

Tasks:

• Investigate if Comunica can enable clickable results. If so, implement this. If not, this issue is out of scope for this roadmap block
• Test with René

Out of scope:

• Investigate how we can use the resource pages (e.g. https://data.beeldengeluid.nl/id/program/2101608110093854631) more browsable - inverse relations? Link data to datasets and to datacatalogs?
• If above points are not a sufficient solution, consider if it is worthwhile looking into new software (René is a fan of the Yasgui interface, e.g. data.digitopia.nl/yasgui

Acceptance criteria:

• A user can click through our linked data

• Check current status
• improve specs to match flourish vis.

There is a lot of confusion at present about what is possible regarding working with data. In particular as our rules have become stricter recently, and SANE has created expectations that cannot yet be met. The suggestion is to put this information on data.b&g, possibly in the form of a Q&A. This communicates it clearly to users, and also gives B&G colleagues an authoritative reference if they are asked questions.

Tasks:

• In a Google Doc, clearly state what is and isn't possible for data access
• Consider how we handle 'exceptions' - e.g. projects in which it is our role to deliver data - how do we leave the door open a crack to these without the door appearing to be wide open ;)
• Have the document reviewed by Roosmarijn and Roeland
• Have the document reviewed by the legal department
• Choose an appropriate format for placing this on the website
• Consider how the information will be kept up to date (who is responsible? how can the information be updated?)
• Implement the solution

Acceptance criteria:

• What is - and is not - possible regarding data is clearly stated on data.beeldengeluid.nl

on medium width screens, the language switch is currently not visible:

For example on https://labs.beeldengeluid.nl/datasets/nisv-media-catalog.

They seem to get clipped. Note: they do work on blogs, apis, ... Perhaps the tab component is the issue.

• Number of concerts per year
• Distribution over musical genres
• The top ten creators, based on number of concerts
• The top ten performers, based on number of concerts
• Number of concerts per location
• The top ten production companies, based on number of concerts

STORY B1 As an end user of the data I want to be able to consult the data in a human readable format online so I can find additional information about the resource, and follow links to this resource on platforms that contain additional metadata or the actual video/audio themselves (BOS, OB, Peertube, GPP)

• At least one link back to a NISV platform should be available (preference for GPP)

• The link(s) to other platforms provided in the metadata are persistent (not forever, but for the lifetime of the platform)

• html landingpage must be dynamically generated

Context
Deze Epic beschrijft hoe we vanuit de huidige kennis en infrastructuur komen tot een werkende oplossing die het mogelijk maakt om de vraag naar de data van Beeld en Geluid in een open omgeving.

Hierbij moet rekening gehouden worden met vooral de informatievoorziening richting het netwerk van digitale erfgoedinstellingen waar beeld en Geluid deel van uit maakt. Zie overzichtsplaatje

Verder moet er ook goed gekeken worden naar huidige oplossingen ne de mogelijkheden en beperkingen daarvan.

Taken

DAAN als LD: geschikt voor het open data lab (cc0 & schema.org)

Doel: Perfectioneren van het DAAN LD formaat tbv het ODL en een monster (LD files) op te leveren voor verdere ontwikkelingen.

• perfectioneer de cc0 versie in schema.org formaat middels de LOD server aan te sluiten op de Flexstore API. zie deze issue
• implementeer een simpel scriptje om een LD file dump (monster) te maken op basis van de LOD server. zie issue hier
  COMMENT WM: uitzoeken hdt dump
• publiceer het monster op in de opendatalab GH (ook voor devops)
  COMMENT WM: Het lijkt me niet zinvol om het monster op GH te plaatsen, omdat het data is. En heel veel data. Een link naar een file op een server moet voldoende zijn.
• stel het NOB op de hoogte van het monster
• (optioneel) pas profile negotiation toe om beide schema.org en het B&G schema opvraagbaar te maken

DAAN LD in een triple store (devops)

Doel: automatisch uitrollen van VMs met cliopatria triple stores + het geautomatiseerd importeren van linked data files

• beeldengeluid/labs.beeldengeluid.nl#180
• importeer het monster (via GH) in de cliopatria store
  COMMENT WM: ik zie geen toegevoegde waarde om dit via GH te doen. Een gewone file link is voldoende. Suggestie: publiceer de file link op de CKAN instance van de mediasuite. Dan kunnen we daar ook de updates bijhouden.
• gebruik postman om de SPARQL endpoint te testen (met een query die iets uit het monster opdiept)
• gebruik postman om de service te monitoren

DAAN cc0 collectie gekoppeld aan search API (Elasticsearch)

Doel: in ODL is de DAAN beschikbaar als ES collectie, welke via de search API doorzocht kan worden. De daan-catalogue-aggr index dient als basis (wordt gevuld via de DAAN importer).

• onderzoek of het nodig is daan-catalogue-aggr te filteren of dat het nodig is een alias aan te maken om bepaalde resources uit te sluiten
• implementeer een nieuwe search API mapping (voor een nieuwe indexnaam daan-catalogue-aggr-cc0) die ervoor zorgt dat de geretourneerde resources cc0 zijn (b.v. omschrijvingen inkorten en bepaalde velden uitsluiten)
• reindex de nieuwe alias (of de hele index) naar het AWS ES7 cluster naar het nieuwe daan-catalogue-aggr-cc0

Implementatie DAAN LD importer

Doel: lange termijn voorziening bouwen welke LD kan voorzien welke in sync is met de DAAN catalogus

• implementeer een nieuwe importer die gebruik maakt van dezelfde cc0 mapping welke gebruikt is bij de LOD server koppeling met de flexstore
• zorg dat de importer automatisch gedeployed wordt via GitHub actions
• zorg dat de output eenvoudigweg in LD files wordt opgeslagen OF direct in een AWS triplestore
• verzin een manier om om te gaan met binnenkomende updates en deletes (hoe wijzig je reeds opgeslagen triples in files; bij triplestore wel vrij eenvoudig)

DAAN geaggregeerde index GTAA links fixen

Doel: Op dit moment zijn er geen GTAA URIs aanwezig in de daan-catalogue-aggr index die door de DAAN importer wordt gevuld... Voor een betere link met het LD werk is het wenselijk deze URIs wel te hebben.

• pas de DAAN importer aan, zodat de URIs standaard worden meegenomen
• implementeer een script dat ontbrekende URIs kan bijvoegen in de index
• zorg dat de index_template (mapping) wordt bijgewerkt

B&G LOD server voor algemeen publiek

Doel: Resolven van B&G DAAN URIs voor het algemene publiek. Waarschijnlijk pas op te leveren als de kopie van de Flexstore beschikbaar is

• zorg dat de LOD server is aangesloten op de kopie van de Flexstore
• zorg ervoor dat er een duidelijke afspraak is dat de kopie Flexstore gekoppeld mag worden tbv het ODL
• test het geheel binnen het domein data.rdlabs.beeldengeluid.nl (in AWS in het ODL)
• zorg dat het allemaal draait op HTTPS

Finale versie

• zorg dat het data.beeldengeluid.nl domein gekoppeld is aan de LOD server

If a blog (e.g.) is only available in NL locale, the build will fail.

Depends on #45
Tasks:

• Embed the developed Vegalite visualisations in the MOZ tabs
• Add visualisations for ASR
• Add ASR to the data update
• Test the data is being regularly updated
• Test that the visualisations are regularly updated
• Ensure that last updated date is shown
• any other loose ends

STORY B13: As an aggregator at the European level I want to have access to the data in EDM format so I can publish my data in high quality on Europeana.

In addition to schema.org, EDM is an important format to have available for participation in European projects. Mappings from schema to EDM have already been made, check with Dimitra if these mappings are needed.

WM: think that this is what was meant for this issue

add the information that leads to the HTML lodview page to the dataset page

• use lod-view in labs.beeldengeluid.nl website.

Is it still within scope?

STORY B11 As a data researcher I want to be able to export the results of my queries to a CSV so I can process the data in consumer spreadsheet applications.

Feedback from René: The current descriptions are sometimes confusing. For example, Radio Oranje talks about a 'selection' of the data, but does not explain how this selection came about.

Tasks:

• Review and improve the descriptions

For the dashboard tab.

STORY B5 As a third party developer I want to be able to download a dump of the data and keep it up to date so I can process the data in a way that is most fitting to my application.

• The dumps are updated monthly

Tasks:

• Determine necessary and feasible improvements based on user feedback (below)
• Make the improvements in the tab

User feedback:

• The descriptions for each of the visualizations are a bit padded under with some gaps between them and then the next one starts right off without any gap. At first glance there could be some confusion
• if the font colors can be changed in the graphs, then it would be good to because right now, they are sort-of light shade, and it could be difficult to read.
• include more artists in the bubble visualisation
• consider how to make the bubble sizes show the differences between the values more
• add context to explain what qualifies as an artist and what as a creator (@KleinRana are we going to add more text in general to this tab?)

Tasks:

• Create markdown pages in NL and EN, using the newest format (see MOZ dataset as an example)
• Write descriptions
• Determine which stats/viz should be included and add these
• Add links to the Media Suite and to the Amateurfilm query in the SPARQL endpoint
• Add link to the NDE register entry
• After the site has been updated with the new dataset pages, add a link to the dataset page to the mainEntityOfPAge column in the ODL spreadsheet

Regular updates of stats/viz is out of scope

Feedback René: would expect this by the dataset. Ideally, you would want a custom image per dataset, as some fields are not relevant for all datasets. This should also be included in showcases about datasets.

Mari: perhaps a link to a central version per dataset/showcase would be a good start?

Feedback René: Expects SPARQL/Linked Data to be on the highest level (couldn't find it, as it is a subheading on the B&G Media Catalog APIs page), also expects APIs to be organised according to technical protocol, not according to the content of the datasets. The GTAA page links through to a result page, would expect a query endpoint.

(general comment: APIs are only interesting once you want to get information about a dataset)

Tasks:

• Review the API information, consider how we can reorganise them to make it easier to find what you need. Should the APIs page be pure technical with links to datasets that can be accessed via the API (and then link to the API page from the content datasets)?
• Review the link from the GTAA page, can this be to an endpoint? Or is that not feasible/desirable?
• Test with René, and other users if we can find them

Acceptance criteria:

• A technical user can find the information they need about APIs

Tasks:

• Review of current ontospy solution by René
• Add a link to the schema to the MOZ dataset
• Document steps for updating schema and overview in the publication steps document

Feedback from René is in this issue, as the points are about improvements rather than blockers

NB: adding links to the schema to other pages is outside of the current scope

Acceptance criteria:

• The user can gain a basic understanding of the schema.org entities and properties used for the NISV data

Forestry has been discontinued.