[parse] Extract Links about markdowndb HOT 13 CLOSED

@rufuspollock It was easier for me to start developing this and running tests in Flowershow repo. I've created a draft of the PR so that you can easily see the code related to only this issue. https://github.com/flowershow/flowershow/pull/468/files

There are a bunch of passing tests and currently, I'm trying to extract common mark style embeds.

from markdowndb.

@olayway awesome and i agree re flowershow! i think we could actually port markdowndb right now into a package in flowershow - this is basically datopian/flowershow#465

@rufuspollock It was easier for me to start developing this and running tests in Flowershow repo. I've created a draft of the PR so that you can easily see the code related to only this issue. https://github.com/flowershow/flowershow/pull/468/files

My one thought about this approach is this seems to be part of remark-wiki-link whereas we probably want it as it own separate mini remark plugin (i guess? with a defined way the link data is represented that is then easy for markdowndb pipeline to use and store).

from markdowndb.

This doesn't seem to be working for me in a local test. @mohamedsalem401 can you comment on your experience too. We may need to reopen and fix.

from markdowndb.

@olayway

Links don't work between files when using my local machine.

For example, the following links in an md file don't add any links to the database, although the file markdown.md exists and is correctly added to the files table:

[page](markdowndb.md)
[page](markdowndb)
![page link](markdowndb)

One possible fix for the issue is in the markdown.db file on lines 121 and 122:

const destPath = resolveLinkToUrlPath(linkSrc, url);
const destFile = filesToInsert.find((file) => file.url_path === destPath);
if (!destFile) {
    return;
}

If we console.log the destPath (the path to the link) and url_path of all files:

files.url_path => [
  'data-cli',
  'datapipes',
  'flowershow',
  'giftless',
  'markdowndb',
  'portaljs'
]
destPath => ":\markdowndb"

You can see that, for some reason, the path for links in the file gets preceded by :\ I don't know if this is intentional to match one of our use cases. In this case, we may want to fix it for using our library with local files.

Let me know if I should just delete :\ in destPath or add :\ before url_paths or only add :\ in the comparison (i.e., const destFile = filesToInsert.find((file) => ":\" + file.url_path === destPath);).

from markdowndb.

@mohamedsalem401 i think we can address this fix after #47 and #5

from markdowndb.

Reopening the issue as there is a bug atm.

from markdowndb.

@mohamedsalem401

• finish doing the analysis of the cases with the results in hackmd and then move above ✅2023-11-16 ( see the Design section on the issue description )
• turn this into tests (can take over existing extractWikiLinks (and don't worry if we break existing system) ✅2023-11-16 ( PR )
• get the tests passing by updating the code ✅2023-11-16 ( PR )
• add a simple test (we don't need all the cases) on process.ts

NB: this is also a chance to look at how remark markdown parsing works ...

from markdowndb.

When coding, I like to maintain high-level notes for consideration. I will leave them in a comment if that's acceptable.

1. Default Extension for Paths:
   If a path is presented as [hello](world), should the system default the extension to md, mdx, or not?
2. Handling Embedded External Links:
   How should the system handle embedded external links, such as YouTube iframes or similar elements?
3. Handling Paths Outside the Root Folder:
   When a file (e.g., abc/foobar) contains a link like [hello](../../world), what should the to attribute point to?
4. Absolute vs. Relative Paths:
   How do we want to handle absolute paths like [hello](/world) and [hello](./world)? Should the to attribute reflect the absolute path, or should it be relative to the root?
5. Handling Link Text:
   How should the system handle link text, e.g., should it be trimmed (e.g., text: 'hello ')?
6. Differentiating Internal and External Links:
   How should we differentiate between internal and external links? Should we rely on a specific prefix (e.g., / for internal and https for external), or is there another preferred method?

Advanced Considerations (these were on the back of my mind and should definitely be ignored for now):
7. Links to Sections or Headings:
How should the system handle links to specific sections or headings within a document?
8. Handling Links to Directories:
How should the system handle links that point to directories rather than individual files? Should it default to pointing to index.md?
9. Supporting Link Titles for Accessibility:
Should the system support link titles for improved accessibility, e.g., [hello](./world 'TITLE')?
10. Handling URLs and Email Addresses:
How should the system handle URLs and email addresses enclosed in angle brackets, e.g., <[email protected]>?
11. Linking Images in Markdown:
How would you like the system to handle linking images in Markdown, considering the special syntax for adding links to images?

from markdowndb.

Great set of questions.

When coding, I like to maintain high-level notes for consideration. I will leave them in a comment if that's acceptable.

1. Default Extension for Paths:
   If a path is presented as [hello](world), should the system default the extension to md, mdx, or not?

That isn't a valid link so this situation should never happen.

2. Handling Embedded External Links:
   How should the system handle embedded external links, such as YouTube iframes or similar elements?

Ignore them. Only stuff like ![](youtube link)

3. Handling Paths Outside the Root Folder:
   When a file (e.g., abc/foobar) contains a link like [hello](../../world), what should the to attribute point to?

Allow it for now. It's just a broken link. Bonus would be logging a warning.

4. Absolute vs. Relative Paths:
   How do we want to handle absolute paths like [hello](/world) and [hello](./world)? Should the to attribute reflect the absolute path, or should it be relative to the root?

./world is easy in that it is the same as world and /world should default to the root of the content system.

5. Handling Link Text:
   How should the system handle link text, e.g., should it be trimmed (e.g., text: 'hello ')?

Don't trim the text.

6. Differentiating Internal and External Links:
   How should we differentiate between internal and external links? Should we rely on a specific prefix (e.g., / for internal and https for external), or is there another preferred method?

External links have a scheme e.g. http, https, ftp.

Advanced Considerations (these were on the back of my mind and should definitely be ignored for now):
7. Links to Sections or Headings:
How should the system handle links to specific sections or headings within a document?

This is a great one and i think we need to have a fragment field and extract the fragment attribute on links.

8. Handling Links to Directories:
   How should the system handle links that point to directories rather than individual files? Should it default to pointing to index.md?

Just point to the directory as it would be like a file e.g. if i have the profiles section just have to be to: "profiles/"

9. Supporting Link Titles for Accessibility:
   Should the system support link titles for improved accessibility, e.g., hello?

Not for now.

10. Handling URLs and Email Addresses:
    How should the system handle URLs and email addresses enclosed in angle brackets, e.g., [email protected]?

ignore them

11. Linking Images in Markdown:
    How would you like the system to handle linking images in Markdown, considering the special syntax for adding links to images?

I'm not sure i get this one ...

from markdowndb.

@mohamedsalem401 can we now close this?

from markdowndb.

./world is easy in that it is the same as world and /world should default to the root of the content system

This needs a simple change in the code to be addressed.

from markdowndb.

@mohamedsalem401 to summarize:

• What do we want to use the link database for? Basically to find back links etc
• We want [[xxx] to work and we can default to this being a .md extension
• What does obsidian dataview do?

from markdowndb.

./world is easy in that it is the same as world and /world should default to the root of the content system

This needs a simple change in the code to be addressed.

Did we make this change? If not let's open an inbox issue and have a section called "Link improvements" and put this in there so we don't forget it.

from markdowndb.