All errors in the module should to be localizable, so we have to move from just throwing text errors to using localizable ids.

I suggest throwing a custom error object (e.g. H5pError) on critical errors that contains a (human-readable) error code an associative array with relevant variables. (see #44)

While implementing the validation of packages in h5p-extractor I've come across the problem of reporting back validation errors to the user. The way the PHP implementation does this is this:

1. setErrorMessage(...) is called by the validation functions when a validation error occurs. This method is implemented by the plugin (e.g. Moodle or WordPress) and stores the message somewhere. It does not immediately output them
2. If there was a validation error, the messages are later pulled out again, e.g. in the function isValidPackage() by calling getMessages('error'). The messages are passed on to H5PCore::ajaxError(), which produces the response to the AJAX call (by printing out to stdout).

So far, I have opted to throw JavaScript errors when the validation produces an error, with the idea of catching these errors at a top level and then producing the corresponding response. However, the problem is that using throw and catch means that the validation process will be interrupted right away when an errors occurs. This is not the way the PHP implementation does it: it continues the validation if the error is not critical and collects all errors to report all of them to the user.

I assume you'll want the NodeJS implementation's behavior to be as close to the PHP implementation as possible. This means there must be a messaging mechanism that stores error (and info) messages and allows later retrieval of them. I've worked with generic loggers like Winston before, but I'm not entirely sure if they are a good fit for our purposes here. Have you had any thoughts of this yet?

I am very interested in this project too !

Currently, packages created with the editor that require other libraries, can't be played because the required library is not added to the dependencies.

Example: create a Course Presentation with an Image. When played, it throws the error Unable to find constructor for: H5P.Image 1.1 which is due to Image not being in preloadedDependencies.

Several (all of the library validation) tests fail in a Windows environment even though they work fine on Linux (and in WSL bash on Windows). We should fix this to encourage development by developers who prefer Windows.

The reason for the errors might be connected to how paths are formed in Windows (using backslashes).

If an H5P-file is reloaded into the Editor, the Editor only shows a blue button instead of the image.

Yauzl is a very good choice as an unzipping library, as adm-zip has some problems. The PackageValidator needs to be moved to yauzl, too.

Downloading the "Twitter User Feed" content types produces a 404 error. It was probably removed.

Hello,

Is it possible to have a look to your work on this project ?

Thanks a lot :)
Théo

At the moment all uploaded files are stored in a temporary directory. We have to move these files into their final destination when an H5P is created or updated.

We should transpile it down to ES5 with Babel.

We need a configuration object, that tells the editor which URLs to use.
The URLs are the following:

• baseUrl
• ajaxPath
• libraryUrl
• filesPath

Files uploaded through the AJAX endpoint (images, video etc.) need to be check whether they are valid:

• the extension must be in H5PEditorConfig.contentWhitelist
• the file must be smaller than H5PEditorConfig.maxFileSize
• the package it is added to must not have a greater total size than H5PEditorConfig.maxTotalSize

There are further checks in h5p-editor-php-library:h5p-editor-file.class.php:

• check if field type & mime-type match
• check if mime-type is allowed
  • if image (check by mime type):
    - only allow png,jpg,jpg,gif
    - read image (& get file size)
  • if audio (check by mime type): only allow mpeg, mp3, m4a, mp4, wav, x-wav, ogg
  • if video (check by mime type): only allow mp4,webm, ogg
• check if mime-type matches extension

Hey @JPSchellenberg and @sr258

I thought about the domain model today and wanted to know what you guys think about it. Sebastian already started with classes for the Library and Content Type and I think if we pushed that further and designed a full domain model, we could reduce complexity in both the interface and the implementation.

Here my proposal. The listed properties are not exhaustive.

Looking forward to your comments.

Entities

Library

Arguably the most important entity. A Library is everything that can be found in the library folders in an H5P file, including the properies stored in library.json, its semantics, hook and upgrade functions, its scripts, styles and other assets.

Library
- title:String
- machineName:String
- version:Version
- coreApi:{version:Version}
- styles:[Asset]
- scripts:[Asset]
- dependencies:[Library]
- editorDependencies:[Library]
- hooks:{preSave:Function,...}
- upgrades:{1:{2:Function}}

Version
- minor:Integer
- major:Integer
- patch:Integer

Asset
- path:String
- data:Buffer
- stream:ReadStream

Content Type

A Content Type is what we get from H5P.org. It mostly describes a Library.

ContentType
- library:Library
- coreApiVersionNeeded:Version
- title:String
- summary:String
...

Package

A Package is everything you find in an H5P file, which is not a Library, meaning the properties of the h5p.json, the content and its assets like images.

Package
- title:String
- language:String
- mainLibrary:Library
- dependencies:[Library]
- content:{}
- assets:[Asset]

Implementation

Regarding the implementation I'm thinking of a mix between Active Record and Repository pattern. Repositories would be used to load entities, and the entities themselves to save changes and access properties. Each entity would be a proxy, loading properties and references lazily. This means that properties could only be access via async methods which might feel a bit awkward but I think it's worth the gain of clearer interaction patterns.

The Ajax-Endpoint is the endpoint that the Editor uses to load and save data from the server. Instead of providing different endpoints the H5P-Team used one endpoint which can be configured via the ajaxPath-field in the Editor-Integration-Object

As far as my research goes, it works like this:

GET or POST /ajaxPath?action=

1. Get Content-Type-Cache

See #5

Method:
GET /ajaxPath?action=content-type-cache

Response:

{
"outdated": boolean,
"libraries": Array<Library>
}

Example

2. Get Libraries

See #6

Method:
GET /ajaxPath?action=libraries?machineName=<machine_name>&majorVersion=<major_version>&minorVersion=<minor_version>

Response:

{
"semantics": LibrarySemantics,
"language": LibraryLanguage,
"javascript": Array<JSDependency>,
"css": Array<CSSDependency>,
"translation": LibraryTranslation
}

3. Post Libraries

See #7

Method:
POST /ajaxPath?action=libraries

Body:

Array<LibraryName>

Example:
["H5P.AdvancedText 1.1", "H5P.AppearIn 1.0"]

Response:

Array<Library>

4. Post files

See #8

Method:
POST /ajaxPath?action=files

Body: file

Response:

{
mime: MimeType,
path: Path
}

I hope this makes it a little bit clearer and helps you understand how the H5P-Editor works. There might be other mechanisms that I am not aware of, so please feel free to contribute and share your knowledge about the H5P-Structure.

We have to delete all uploaded files in the temporary directory when an H5P creation is canceled.

Method:
POST /ajaxPath?action=libraries

Body:

Array<LibraryName>

Example:
["H5P.AdvancedText 1.1", "H5P.AppearIn 1.0"]

Response:

Array<Library>

I need to configure a custom Hub and other options int the config but currently it's hard-coded.

A H5P package doesn't necessarily include all required packages, but might rely on them already being present in the system.

Before the libraries and the content is added to the system, we need to check if all required libraries will be present if the ones in the package are added to it. If not, the PackageManager needs to reject the request to add the package.

We also need to check if the content of a library is usable if the user doesn't have permission to install or update libraries. As the libraries can't be installed by him, the content won't be usable.

We have to decide what to do if the package of a user contains a new patch of a library, but the user doesn't have permission to update libraries. There are two options:

1. reject the package and tell the user to ask for an update by the administrator
2. accept the content without applying the patch (as patches don't change the semantics of a library, the content should be playable)

@rtens @JPSchellenberg Which option would you prefer here?

The more I look at h5p-editor-php-library and h5p-php-library, the less does the distinction between the two make actual sense to me: Many things that are clearly part of the editor (content and package validation, file upload, ...) are part of h5p-php-library. As far as I know the "player" can't be used without also having the editor in current PHP implementations. There are also many dependencies of the editor on the other library, which makes the whole h5p project difficult to understand for new programmers, because you constantly have to switch between projects and lose things like "Go to definition" in IDEs.

So why have to separate projects at all? If there should be a standalone "player" application, you could always start the application with a special flag or change something in the configuration to disable the editor.

This "Endpoint" is used to get data about a library from the server and is called when you select use:

Method:
GET /ajaxPath?action=libraries?machineName=<machine_name>&majorVersion=<major_version>&minorVersion=<minor_version>

Response:

{
"semantics": LibrarySemantics,
"language": LibraryLanguage,
"javascript": Array<JSDependency>,
"css": Array<CSSDependency>,
"translation": LibraryTranslation
}

Method:
POST /ajaxPath?action=files

Body: file

Response:

{
mime: MimeType,
path: Path
}

Hey,

the getLibraryData returns an object containing information about a specific library. Right now semantics and javascript dependencies are included in this object, but we still need to implement the translations field.

While working on the validation part of h5p-extractor I've come across the need to use string literals that are displayed to the user (the validation errors). These literals should obviously be localized to the user's language.

The PHP library does it this way: Whenever it outputs a message to the user, it calls the function t($message, $replacements) with the English message string as first parameter. This function returns the translated string and does some placeholder replacements.

This means that the English message string is put in as a literal value into the actual source code. I think this is bad style (changing English literals means you have to look through the whole source code; creating translations is tedious; translations can't take context into consideration, which means that the same English literal in different places always has to have the same translation) and shouldn't be replicated in the NodeJS library.

What I've done is put the (English) literals into a JSON file as key-value pairs. The JSON file for the user's language is loaded by the framework and the content is passed to the validator as an object. Whenever the validator outputs a literal it looks up the string in the translation object through the key and outputs the value (the translation). If it requires placeholder replacements it does it with a little helper function (see here for an example).

I think my current approach is not the final word on localization of the NodeJS implementation (doesn't cater for single strings that are not translated in a language even though there is a language file for it), but this is something that has to be used project wide, so I would like to check if you've had any thoughts on this.

Hey,

the getLibraryData returns an object containing information about a specific library. Right now semantics and javascript dependencies are included in this object, but we still need to implement the language and languages fields.

(This has also been discussed in #43. #80 is also related, as all currently unlocalizable error messages in the code need to be replaced if we've decided how to manage reporting errors)

The H5P client expects errors (and more detailed success messages) to be a part of the AJAX response in several cases.

Examples:

• it gets a (success) message about how many libraries were added or updates when the user uploads a package
• it gets a (success) message of how many updates were applied when updating content types from the hub
• it gets an (error) message with what exactly is wrong when the validation of an uploaded package fails

The PHP implementation uses a global function to add success and error messages to the AJAX response. I would like to avoid this pattern as it won't work nicely with unit tests. I also think that the domain classes shouldn't know about how they are called, as there are scenarios where they might not be called by an AJAX call at all, but are used to automate things (e.g. in CLI dev tools). Thus, it would be better if these messages were reported back to the top level with another mechanism. Suggestions:

• Option 1: Report messages by sending them as events through a Publish-Subscribe pattern.
• Option 2: Report messages by using a logging solution with log levels (like Winston)
• Option 3: Return success messages as strings and report error messages by throwing error objects.

The top level caller (i.e. the method in the implementation (h5p-demo) dealing with the AJAX request) then adds the success or error message to the AJAX response.

The npm run build command is missing, which is required to run without errors.

Currently contentId is simply generated in the example implementation by adding the query parameter contentId=XXX to every request. At some point, this needs to be replaced with a solution that is closer to the PHP implementation. I've tried to find out how the PHP implementation assigns contentIds and so far I've found out this:

1. contentIds are integers and are not necessarily the same as the id of the CMS/LMS's content node. (Even though this would be possible if the ids of the CMS/LMS are integers too)
2. They seem to be generated by the CMS/LMS (plugin). In the moodle implementation it is simply the auto-generated key of the database entry of a saved piece of content.
3. When editing an existing content piece the contentId is passed to the editor in H5PIntegration.editor.nodeVersionId.
4. When uploading images in already existing content, the contentId is in the property contentId of the request's body (form data). It is 0 if the user is creating new content. In both cases the image is uploaded to a global temporary location, though. I'm not sure how the PHP implementation keeps track of the connection between new uploaded images and the unsaved content (that only exists in the client at this point). It looks like this is connected to the current session and not to the contentId received through the body. I've checked in the Moodle implementation: If I upload an image in an unsaved piece of content and "abort" editing by opening other Moodle pages, I can still access the image by calling the temporary URL (http://moodle/pluginfile.php/XXXXXXX/mod_hvp/editor/images/file-5d61958f640a9.png). I don't have access to it once I've logged out of moodle or from another browser with the same user. So uploaded temporary images seem to exist at session scope (and are deleted through a timeout).
5. When saving a new piece of content, the contentId is generated and there is a HTML redirect to the viewer (or to the course overview, depending on what the user clicked). There it is passed to the JavaScript client as described in 6.
6. When viewing a piece of content, the contentId is passed to the JavaScript viewer by the plugin
   1. by setting an object to H5PIntegration.contents["cid-XXX"] and
   2. by setting a HTML property in

All of this means that we need to finish #61 and introduce sessions to the server (#46 needs to be moved back here) to be able to remove the contentId parameter.

The editor saves state in various places, e.g. when refreshing the Content Type Cache:

• it stores a unique ID assigned to the instance by the H5P Hub, which has to be used in subsequent requests
• it stores information about when the it last downloaded the information from the Hub
• the data of the content cache itself must also be stored somewhere
• There are other uses of persistence such as configuration settings (e.g. if usage statistics should be sent).

In the original H5P implementation this is done by calling interface functions which are implemented by the H5P integrations (into WordPress, Moodle). The Moodle implementation for example stores the data in the SQL database.

Do you want to take the same approach here? If there is an interface to call, people using the library could then employ whatever they want (MongoDB, files, ...).

When selecting Image in the Media section, the h5p.js throws an error, because it can not determine the correct uberName.

This is caused by the ns.libraryFromString function, which receives the argument H5P.Image-1.1. I think the - makes the problem, but I haven't figured out where this value comes from, or how it is set.

This causes the following GET request to fail, because no Library was specified:

Originally posted in #15 . The postinstall script fails and the H5P-scripts are not downloaded, which causes the Editor to not find the necessary scripts.
It looks like the rights are not correct. npm-issue

One option could be to run npm install --unsafe-perm stackoverflow, but I would not want users to use the --unsafe-perm-flag.

Has anyone an idea?

While working on finishing the installation PR I've started working on a proper domain model for Library that follows the proxy pattern proposed by @rtens . This makes sense now as I have to move code from FileStorage to FileLibraryStorage and change method calls, so why not do it now before the code becomes bigger and there will be even more changes?

You can see the current state in the install-proxy-library branch.

Relevant classes where something has happened are Library, LibraryProxy, FileLibraryStorage and LibraryManager. The code is not tested or integrated at all, but it'd like some feedback before I continue down this lane further.

This is possibly a duplicate of #32

The goal of this issue is to edit uploaded content and save any changes made, including uploaded files (eg images).

For the first iteration, we skip access control.

Dependencies should be checked for the following criteria:

• is it a dev-dependency, which accidentally has been installed as dependency?
• is it a hard-required dependency (for example to run the editor)
• is it an optional dependency?
• ... what else?

Example: fs-extra is currently in dependencies but i only used as dev-dependency.

The Editor is missing translations. PR #38 is trying to solve this issue.

The content-type-cache is a list of H5P-content-types that are installed or can be installed.

It is implemented via the ajax-endpoint with the action content_type_cache.
GET /ajax?action=content_type_cache
An example response is here.

Right now the content-type-cache is hard-coded.

Todo:

• Get a list of installable content-types from the H5P Hub.
• Determine local-installed content-types.
• Generate a list and serve it via load_content_type_cache

When you upload packages or get libraries from the H5P Hub, these may contain two kinds of updates:

1. a new revision of the library (e.g. old library version: 1.0, new library version: 1.1) In this case the updates library is stored side-by-side to the old one
2. a new patch of an already installed library (e.g. old library version: 1.0.0, new library version: 1.0.1) In this case the old library version is completely replaces by the new one.

We have to implement the patching process for case 2. Installing the patch should only be allowed for users have the permission to update libraries. The update process should also revert to the original patch version if something goes wrong when installing the new version.

This issue combines #23 and #33.

Currently, when uploading files during editing (eg by adding a picture to a Course Presentation slide), the file is stored in the content folder (under h5p/content/{contentId}/content/{fileName}).

This leads to the problem that if the changes are not saved and the file is not used in the end, it will still be part of the package and not be deleted.

A possible solution would be to store uploaded files in a temporary folder and only move them to the content folder when the changes are saved.

The temporary folders should be regularly purged.

Currently, uploading images doesn't work if the content you're editing is new and hasn't been saved before. The reason is this would require either having a temporary storage place for unsaved content or the concept of preliminary content in storage (see #61). We have to choose one path and implement it to make uploading images for unsaved content work again.

To demonstrate the usage of this library, we'll create an extra repository containing an exemplary project using it.

So all files and code solely used as example is to be moved to that repo.

In some content types like Course Presentation or Interactive Video the h5p-drag-text.js throws an error:

Hi,

I'm interested in your efforts to develop a NodeJS H5P integration. Have you done any research on how the editor works so far?

Best
Sebastian

Considering the fact that we're building a server application, it would make sense to use the AGPL license instead of GPL. That way someone who uses the project to run a server and changes the code has to publish it too. Under the GPL this would not be necessary.

I've looked through the code of the PHP implementation (editor and regular library) and identified these components / this functionality that we haven't put into our implementation yet:

• logging & statistics generation: e.g. use of libraries by author, view of embedded content etc. (see h5p-php-library:h5p-event-base.class.php for a list of events)
• store user state (for continuing later)
• full validation of content against library semantics

A few (less important things maybe) that we haven't done yet:

• turn hub on - off (config)
• presave.js files (see h5p-editor-php-library:h5peditor.class.php->hasPresave(...) and ->addPresaveFile(...)
• prefixing css files (not sure why this is done; see: h5p-editor-php-library:h5peditor.class.php->buildCssPath(...)
• cached assets (aggregate all css and js files into two big files to decrease http requests; done in h5p-php-library:h5p-default-storage.class.php->cacheAssets(...); looks very much like what Webpack does, maybe we can use a library like this for this)
• library devolopment functionality
• siteTypes: local, network, internet (local = localhost; network = internal network); auto-detected in h5p-php-library:h5p.classes.php->detectSiteType()
• check if server can run with the configuration (on startup) (e.g.: are the directories writeable, sanity check if config values)

We are still missing the upgradesScript-field in the getLibraryData-method.

The CI tests of my pull requests don't seem to pass through. One is always yellow and says "Travis CI - Branch Expected — Waiting for status to be reported". The fix might be found here: https://travis-ci.community/t/known-issue-travis-ci-reports-expected-waiting-for-status-to-be-reported-on-the-github-status-api-but-the-status-never-arrives/1154

When I use certain content-types (course-presentation, interactive-video, etc..) , the depedency-order seems to be wrong. It looks a lot like the error from the renderer where the depedency-order was wrong, due to the async-libraryLoader: See here.

We have two options:

1. Use an sync-libraryLoader in the editor as well.
2. Get a async-libraryLoader working. (See #66)

What do you think? @rtens @sr258

Hey, I'm a young developer but I do my best to fully understand the code and to propel the project. I was wondering if you managed to display the editor, even if there're lots of bugs. I tried to import every missing file, then to display the .h5p-editor class but I think this is not a good practice (and it didn't work). Thanks a lot !

Hello,

I'm working on the project and I'm looking at the WP and Moodle modules to understand the operation of the Server-link / AJAX requests. Did you work on that ? Did you succeed ?

Best,
Théo