Test a4.2.05 has the description "There must be a TOC, either in the manifest (linked or embedded) or in the Primary Entry Page".
This is misleading, as the spec does not make the TOC mandatory. There is only a SHOULD if supplemental content is present.
The expected error is therefore wrong IMO ("Validation error on missing TOC")

In the audiobooks testsuite, test a5.6.01 says to return an empty manifest and a fatal errors.

Where in the spec does it indicate that an empty manifest should be returned in this particular case?

One problem I found running the manifest processing tests is that the invalid URLs don't register as invalid per the javascript URL api. I fiddled with the tests to make the URLs fail to ensure the right result, but just looked at the URL standard and a percent in the pathname is not invalid anymore, which is what it is following (browsers still choke on the percent, but that's another story, I guess).

By the percent encoded bytes section, if a percent character isn't followed by two hex characters, the percent is dropped during parsing and the following two characters retained.

If you put a percent in the host, however, you will get an invalid URL.

I'm not sure what can go in a relative path to break the conversion, though. I tried some of the problematic patterns in the standard, but the api just wipes out the invalid parts and returns a URL.

In the publication manifest testsuite, several tests indicate that the offending item should be flagged. I can certainly interpret this loosely but as the processing algorithm is quite specific, down to the different types of errors, it's odd that there is no definition of "flagged". Should it generate its own error that specifically mentions it by, say, URL? Or should the expected error include enough information to identify the offending item?

Here are the relevant tests:

• m4.8.1.1.01
• m4.8.1.2.01
• m4.8.1.3.01
• m4.8.1.3.02

https://www.w3.org/2018/audiobook_examples/flatland.audiopub

• The file extension should be lpf, not audiopub
• There should be a file named publication.json at the root of the zip archive's directory
  (instead, there is a file named manifest.jsonld)

Specification references:
https://www.w3.org/TR/lpf/#sec-structure
https://w3c.github.io/lpf/#sec-structure

Related issue:
#36

In the publication manifest testsuite, test m4.6.03 does not say to expect any errors, but because the test file is an audiobooks manifest, it raises several validation errors related to this requirement about recommended properties

"id": "a8.1.04",
"description": "If the audiobook contains a primary entry page, it is displayed first (or the content within is presented first)",
"actions": "Primary Entry Page is displayed first"

The behavior presented here makes no sense if the audiobook is packaged. UA developers may prefer displaying immediately the cover and audio controls, plus something giving access to the PEP at will, like an info page. And I bet that the few PEP created will be so bizarre or ugly that developers will not create a big button.

This test should therefore be edited as
"If the user agent gains access to the audiobook via its primary entry page, the content of this primary entry page is displayed first, before the user can listen to the audiobook"

or much less intrusive and applicable to LPF files also (I would prefer in fact)
"If the audiobook contains a primary entry page, the user agent has provided a means to display it to the user".

In the publication manifest testsuite, tests:

• m4.2.5.01
• m4.2.5.02
• m4.2.5.03
• m6.01
• m6.02
• m6.03
• m6.04
• m6.06

and in the audiobooks testsuite, tests:

• a4.2.01
• a4.2.02
• a4.2.03
• a4.2.04
• a4.2.05

result in error that the document URL is not in the unique resources, according to this part of the specification. However, this error is not listed as expected by the tests.

For test m4.7.2.3.01, a conformant processor reports a validation error because the link has no rel property. See manifest data validation step 12.c.

Because the error is not related to what the test is testing, I would say the test still passes, but technically, the expected output is different, so the test does seem a bit flawed.

Note that this is also true for several of the other links tests.

Among the three testsuites, "publication manifest processing" and "audiobooks manifest processing" test files are named after the test ID, prefixed with test_, but those in the "toc processing" testsuite are not prefixed.

E.g.

https://github.com/w3c/publ-tests/blob/master/publication_manifest/manifest_processing/tests/index.json#L11

seems to mean that testers should use the file test_m4.01.jsonld

vs

https://github.com/w3c/publ-tests/blob/master/publication_manifest/toc_processing/tests/index.json#L13

points to s4.8.1.3.01.html, not test_s4.8.1.3.01.html.

I care because I have written something to help me step through each testsuite and record the results of my audiobooks parsing library:

https://marisademeglio.github.io/audiobooks-js/official-tests/

It would be nice if the index.json id reference/test file name correlation was consistent.

The setup is (for audiobooks) to locate the TOC in the PEP, because the manifest itself does not have a 'content' structure. This is all right (although see w3c/audiobooks#63 for issues around that).

The ToC in the PEP is:

<nav role="doc-toc">
	<h2>Test Table of Contents</h2>
	<ol>
		<li>
			<a href="#s1">Section 1</a>
		</li>
	</ol>
</nav>

meaning that the link in the ToC goes to .../tests/s4813-02/toc.html#s1. However, this URL does not appear in any resource list of the manifest; according to rule 4.6.2.2. this is an erroneous entry in the TOC.

(if I am not mistaken)

Lang_Crimson_Fairy_Book.lpf
downloaded from:
https://github.com/w3c/publ-tests/tree/master/test_content/audiobooks
=>
/Lang_Crimson_Fairy_Book/publication.json
should be:
/publication.json

Test m4.7.2.1.02 in the publication manifest testsuite says it should cause two errors. Where in the spec/processing algorithm does it indicate that this should raise two errors instead of one?

Two new tests have been added: "Blazing World-Output from Obi" and "Cavendish_Blazing World".

The Obi version is Unpackaged, but is not listed in the Test Audiobook Content "Unpackaged" section below (on "https://github.com/w3c/publ-tests/tree/master/test_content/audiobooks").

The other one ("Cavendish_Blazing World") is packaged as an LPF and is correctly listed in the "Packaged as LPF" section below. However, in that section, it says the test file contains "Manifest with PEP and TOC" -- this is not true -- no PEP and no TOC.

In the publication manifest testsuite:

• m4.7.2.1.04 indicates to warn on duplicate URLs in reading order, but do not remove them.
• m4.7.2.2.03 indicates to warn on duplicate URLs in resources, and then remove them.

Where in the spec does it make this distinction? Both sections say SHOULD NOT include duplicates but I can't find where it says to remove them in the reading order, and leave them in resources.