RFC: Create a repo for Pan Docs as a living document. about pandocs HOT 26 CLOSED

I disagree with @aaaaaa123456789's strong disagreement against converting the format. Even if we ask nocash for the converter, and he obliges, it will most likely be written in x86 assembly, which is not the greatest for the use case. Even so, it's an easy enough format to write a parser for. But if we do so, that's a tool that we have to maintain. I'd much rather use a format that's a widely adopted standard and has maintained tools. The current format is essentially equivalent to markdown or perhaps an early type of wikitext, but different and slightly worse. I don't see much point in preserving that tbh. (The documents that nocash is maintaining himself is a different story of course, since he can run his own tools easily enough.)

As for images, that image from GBdoc is kind of a bad example. It's filled with text with font smoothing, which seems to be the majority of the weight of the file. I was thinking of more conservative use of images. The following page is a good example. With the small image size, it might even be possible to sanely store the images inline as Base64 without too much trouble.

http://www.devrs.com/gb/files/opcodes.html

I think these images on that page are more clear as images than they would be as ASCII art using only regular characters. With that said, now that we're in the age of Unicode, another feasible option might be to draw ASCII art nicely in the way it was originally intended, with DOS style box drawing characters.

Test:

   ┌─────┐
   │     ▼
┌──┴──┬─────┐
│7   4│3   0│
└─────┴──┬──┘
   ▲     │
   └─────┘

from pandocs.

I'm gonna take an hard stance on this: I don't see any point in negating ourselves images, code blocks styling (and possibily syntax highlighting), pagination, responsive layouts, healthy CI/CD tools and web frameworks just to use a plain TXT format for the sake of "we've always did this and there's a similar project doing the same".

We are not picking Markdown because it's trendy. We are choosing it because the number of available (and mature) tools like pandoc makes it an agnostic/meta language with which we can easily mantain the source of the document, while rendering, exporting and transpiling it in any other form we want to present it (including plain txt!) with almost zero effort.

If the diagrams were SVG instead of PNG, the viewer's PC would do all the font smoothing.

What is to SVG as Markdown is to (X)HTML?

We can use something like draw.io to draw the figures and keep them maintanable, while having the possibility to export them in SVG for scalability.

from pandocs.

@avivace
Ah, I see. During my brief research I mostly found references to having to insert an HTML anchor manually to make it work. Apparently, Github's flavor of Markdown supports it natively though.

<a id="anchorname"></a>

from pandocs.

Yeah, but my point was that it gets further and further detached by documentation by Pan, so a rename may make sense. But the name is definitely one that's stuck, so there's weight to maintaining that too...I'm split on this to be honest.

from pandocs.

The name "GBdoc" is already taken by a project of @endrift (and contributors)'s, which is severely unfinished. I think this brings up the choice of making our own "GBdoc", or trying to finish this one by adding all Pan Docs info.

Pros of merging both docs:

• Pan Docs seriously lack structure, a motivation to redo this is really welcome
• Less confusion, point everyone to a single link

Cons of merging:

• Lack of newbie-friendliness, as newcomers may be lost from the details (and they are often told to use Pan Docs as a reference, so this is relevant)
• Possible clutter (see nitro's point 2)
• Losing history of the original Pan Docs

My preference goes towards rebuilding Pan Docs from the ground up, but as a separate document from GBdoc. That's from a point of view of making it a newcomer's entry doc, and leaving the "complete reference for advanced users" role to GBdoc.

Although from a naming point of view, I'd be inclined to naming this "GBdoc" and renaming the original GBdoc to "GBref", as I believe it'd make more sense.

Note that the staleness will not go away. Despite the wiki existing, people still refer to the copy hosted by beware (and sometimes nocash's), I believe for a large part because beware's copy appears first in Google.

1. Agreed. Having different pages has a few props, but browsing fullsnes has proven easier than the wiki, imo.
2. See my thoughts above on why I disagree with the "reference document" part, but I do agree the document should be kept somewhat terse. See also point 6 below.
3. N/A
4. The name should be different, as otherwise people will still refer to other copies, as pointed in the "staleness" paragraph above. See also the first section of this comment.
5. Markdown is also my favorite because it has a low barrier of entry, meaning more people could contribute. The downside of Markdown is being fairly poor in semantics, so we would have to define a strict style guide to be consistent across the entire document. Mandoc, which RGBDS uses for its man pages and HTML documentation, has semantics, but is less beginner-friendly, and probably not adapted to this document, even.
6. Unconditionally, YES, use images. From a newcomer standpoint, it's very difficult to grasp how the hardware works from just the descriptions(I've been there, and my first programs were really broken because I didn't quite understand what I was doing). Images help a lot with understanding, but we don't need more than schematics. Good example imo:

from pandocs.

I cast another vote for using diagrams.

Would it be worthwhile to also maintain a text version in the style of nocash's docs? It'd have to replace the diagrams with ASCII art, such as the following:

Each scanline is 456 dots long, or 114 CPU cycles at single speed.
There are 154 scanlines, or 70224 dots (17556 cycles) per frame.
In the following diagram, each column represents 8 dots or 2 cycles.

     Mode 2    Mode 3                              Mode 0
     OAM       Background and                      Horizontal
     search    sprite rendering                    blanking
     80 dots   172-288 dots                        88-204 dots
    Mode 3 lasts longer for each sprite on the scanline, or
    when fine scrolling (SCX % 8) or the window is in use
              |<-                ->|      - - - ->| VRAM inaccessible
                                                    CGB palettes inaccessible
    |<-                          ->|      - - - ->| OAM inaccessible except by
                                                    DMA (which hides sprites)
    ,---------+--------------------+--------------+----------.
  0 | Fixed   | Shortest           | Longest                 |
  1 |         | Longest                           | Shortest |
  2 |         |                    ::::::::::::::::          |
  3 |         |                    ::::::::::::::::          |
                               . . .
143 |         |                    ::::::::::::::::          |
    +---------+--------------------+--------------+----------+
144 | Mode 1 begins                                          |
145 |                                                        |
                               . . .
153 |                                            Mode 1 ends |
    `--------------------------------------------------------'
     Mode 1: Vertical blanking. Everything is accessible.
     10 lines, 4560 dots, 6.5% of frame time

Duration of transfers to video memory
    ,---.
    |   | Copy CHR with GDMA (GBC only):     32 dots/tile
    +---+-------------------------------.
    |     Copy CHR with CPU:            |   288 dots/tile
    +-----------------------------------+--------------------.
    |     Copy sprite list to OAM with DMA: 640 dots         |
    |     or 1.4 scanlines ,-- or 14% of vblank time --------'
    `----------------------'    

Then when building the plain text doc, the build process would substitute the ASCII version of each diagram instead of the SVG version.

from pandocs.

I'll have more time to go over this later, but I just want to say that I had always intended for my GBdoc to have images. You can only do so much with ASCII art.

from pandocs.

Points 1-3 were really meant as a mission statement, a list of "givens", whereas 4-6 were meant as discussion points. Anything outside of that scope would be a different project by definition. And by that token, it's possible that the correct course of action for what I'm proposing is actually to work on improving endrift's GBdoc.

A point worth discussing is what I mean by reference. It does not mean an exhaustive description of all behavior and corner cases. It does mean including most factual information one might need to look up while developing. The example in the graphic from GBdoc is borderline TMI within the scope of this idea of Pan Docs, even if the graphics themselves are excellent.

As for 2), it should be mentioned that most of the high ranking copies of Pan Docs is within control of the community, so if everyone, including beware, would cooperate, all copies that Google would be likely to link to, would be updated, or contain a notice that an updated version exists.

As for 6) the alternative to images is not no graphics, but ASCII art. It might seem like a no-brainer, but a potential concern is the maintainability of the graphical assets, especially if the assets should be kept consistent. For example, in the image used as an example, how easy is to edit from the (source) file format is being use? If I want to correct a typo, or change the dimensions of the graph, or add another row, how much work is that to do, assuming I want to make sure none of the changes are misaligned? The relative simplicity of ASCII art might be in favor of it, even if it means using slightly less fancy graphics. Maintaining both would be a wasted effort in this day and age though.

from pandocs.

It might be worth having both reference and introductory stuff being in the same doc (or same repo at least), so if a dev wants to dig in deeper they can find a maintained cross-reference.

from pandocs.

Pan Docs need a proper repository

Agree with 1-3.

However, mantaining the entire Pan Docs document as a single file would be a nightmare without any advantage. A simple render script (in whatever tecnology stack we decide) would simply merge the sources and present the document as a whole, allowing crosslinks and CTRL+F navigation.

4. Beware's copy is very high in search results, but this shouldn't be the reason to change the name. Beware can add a link or a redirect to the newly mantained and "living" version we are launching, and we can link somewhere the historical copy. I don't see any reason why the two should coexist, if one is old with wrong information and the other is updated and mantained. I vote to keep the name and continue the legacy of the document. Search results are not an issue: just try to look for "gbdev" in Google now. However, I very much agree with @nitro2k01 and we should understand what are the intentions of beware on this.

5. The image question is very much broader than "images are better than ASCII art" or "images are more descriptive". The points raised by @nitro2k01 make a lot of sense. We should consider:

   • Image responsiveness. We do not want images large 2000px unviewable in anything not 16:9 or breaking responsive versions of the documentation.
   • Image scalability
   • Mantainability. SVG? Sources?

I propose we start by migrating the Wikified version of the Pan Docs to plain markdown, with each article/part being a single file, following and respecting the general Pan Docs structure.

Do keep in mind that markdown is very much extendible and the "pandoc" tool can literally do everything with it, automagically.

from pandocs.

This seems like a good idea, although it's important to maintain the "spirit" so to speak of the original document — it's not an assembly tutorial, for instance, and it should never be. (Most of the document is correct and useful; a good frame of reference is that a full rewrite is not the right thing to do.)

The current format of the document, while terse, is quite functional, and it's something most of us have gotten used to throughout the years. It's also identical to gbatek's (which is a living document, being mostly updated with 3DS-related data), which makes it easier to use for people who develop for both platforms.
The HTML version is clearly autogenerated from a plain text file, and I strongly believe the format should be maintained; we could either ask nocash for the converter or replicate it if that becomes impossible. Maintaining the format also handles the question of what source format to use.

Of course, since concatenating files is trivial, there's no problem in using multiple source files for maintainability. The index (see lines 4 to 33 of the raw text file) could even be used as an input to indicate in which order the files should be concatenated.

Finally, images. The format doesn't allow for images; that should be clear enough. I think that should remain to be the case. ASCII art is for the most part good enough, and images come with a lot of problems: they are hard to edit, they take a long time to load, and they are mostly not relevant to the part of the document you're looking for. The current version of the document is 46K long after compression; the single image above is by itself two and a half times as large as the whole document! Bad connections still exist in many parts of the world; I've had trouble loading gbatek (which is 4 megs) on my phone many times.
For this reason, I'd recommend not using images in the document itself. For the parts where images really help explaining a point and ASCII art is insufficient, a link to an image could be used instead. Not only the minor inconvenience of following links prevents images from being used unnecessarily, but moreover, links only take a few bytes.

One last word: I'm strongly against a conversion to Markdown. The documents have had a stable text format for 20 years; this format indubitably works, and the fact that gbatek is still maintained in it proves it. There's no good reason to change other than "Markdown is trendy"; the existing format doesn't take more than a few minutes to learn.

from pandocs.

If the diagrams were SVG instead of PNG, the viewer's PC would do all the font smoothing.

What is to SVG as Markdown is to (X)HTML?

from pandocs.

I'm definitely not against using Unicode properly; UTF-8 support is basically universal nowadays.

As for the format converter, if we wrote our own, why would it require maintenance? If the format doesn't change, the tool doesn't have to change either. And why is x86 assembly a problem? Are you planning on converting files on non-x86-based machines? Any continuous integration service will offer x86-based machines if you want to run the converter automatically.

What kind of tooling do you need in the first place, anyway?

from pandocs.

A Mac running macOS Catalina (10.15) or later is a non-x86-based machine. So is a Linux box that doesn't have multiarch installed. This is because they run executables in x86-64 mode, not x86 mode. Do common CI services support installing 32-bit Wine?

from pandocs.

@pinobatch Do you really think converting x86 assembly to x86-64 assembly is a hard task? For the most part, it's just changing the names of registers used in addresses...

from pandocs.

For what it's worth, I wrote a tool for converting GBATEK to markdown; it may work on the HTML version of pan docs as well.

from pandocs.

In that case, I'd be inclined to go with Markdown just for the "Yo dawg, I heard you like Pan Docs so I put pandoc in your Pan Docs" factor.

from pandocs.

@aaaaaa123456789

Do you really think converting x86 assembly to x86-64 assembly is a hard task?

It's not a hard task, it's just not a necessary task since writing a parser for the format would probably be easier and faster than dealing with nocash's code.

@avivace
On the other hand, the one thing the current format does have, silly and insignificant as it may seem, is a builtin syntax for anchors and anchor links. The system is simple. Every headline becomes a name attribute with the headline converted to lowercase and with whitespace removed. Same with links, using a corresponding href attribute. Another option would be to use nocash's format, possibly extend it if needed, and transpile it into Markdown and feed that into pandoc. Although looking at GBAtek It may be stored in yet another internal format, considering that the top ToC is divided into columns which don't seem to be encoded in any way in the text file.

from pandocs.

Every headline becomes a name attribute with the headline converted to lowercase and with whitespace removed

This is a very common feature of the web frameworks consuming Markdown or tools exporting Markdown to HTML. Just take a look at the README of this repository and hover your cursor over any heading.

from pandocs.

I talked to beware about the name, and he explained it like this: nocash is the current "owner" of Pan Docs, and our project would be more like a fork and should have a different name for that reason.

I kind of agree with this. If we can agree on a name, I will take the first step and create the repo with an initial version of content. The other aspects can be solved as we go along. Suggestions:

1. Pan Docs (Ie unchanged)
2. New Pan Docs
3. Neo Pan Docs
4. Modern Pan Docs
5. Something else?

Please vote for the name you prefer.

from pandocs.

nocash is the current "owner" of Pan Docs, and our project would be more like a fork

Again, why?

As the header recites:

Pan of -ATX- Document 
Updated by contributions from:      
Marat Fayzullin, Pascal Felber, Paul Robson, Martin Korth
CPU, SGB, CGB, AUX specs by Martin Korth

With a quick search I could find at least 4 big revisions:

Expanded by Jeff Frohwein, 31-Dec-96
Last updated 10/2001 by nocash
updated 4-Mar-98 by kOOPa

The document has been trough a lot of changes and contributions through the years, until the wikification. Then it got more and more easily.

To me, no single contributor owns a particular named version of the document and I don't see any sense in this.

The wikified version, the one we will use as baseline for our source conversion it's always been called and considered Pan Docs (even if a text become mediawiki).

I still can't understand why are we giving relevancy to a particular version dating back to 2001 because someone considers it "official" (while it's itself a 3rd or 4rd big revision).

Keep in mind that we aren't even discussing about actual (or breaking) content changes, but just a re-hosting potentially giving space and tools for a better (and more accessible) contribution process and content presentation.

I'm voting to keep the name Pan Docs as we are simply moving the Pan Docs updated articles from the wiki.

from pandocs.

I propose "gbdev Pan Docs" because it makes a logical contrast with nocash Pan Docs.

from pandocs.

Isn't the name just in reference to the person who originally wrote them? It's changed a lot since then.

from pandocs.

I propose "gbdev Pan Docs" because it makes a logical contrast with nocash Pan Docs.

Yes, and that's exactly our document. Nocash is just hosting a major revision of it

Isn't the name just in reference to the person who originally wrote them? It's changed a lot since then.

Yes but the name stuck. Maybe nocash started calling it "Pan Docs" instead of gbspec.txt? I don't really know but the document remained with that name until today, as you can see on the wiki version, self-defined as Pan Docs.

from pandocs.

Oh, I understand and I agree with you on this.

My point was we can consider a rename, but not because we can't keep the actual name.

The document has been called Pan Docs for over 15 years (at least) and it's pretty popular and recognised in the community. The wiki having taken the legacy of the document while keeping the name was probably a big part of it.

I'm also sure that renaming or not at this point makes little sense, our effort is something else, once it's completed we can propose the rename in any given point time.

from pandocs.

I'm closing this and moving the discussions, RFCs, feedbacks and contributions on the Pan Docs project to the new repo: https://github.com/gbdev/pandocs.

The initial importing into the new format has been completed. The result is here.

Every revision has been ported to the git history, mantaining authors and diffs. We are also mantaning a folder with every historical version we could find.

Looking forward to your contributions and feedbacks there.

from pandocs.