Import process must align with adobe/theblog#194: assets must be dowloaded, stored in the proper folder and referenced accordingly.

This requires https://github.com/adobe/helix-word2md/issues/109 to be useful.

this page:

https://theblog.adobe.com/where-is-personalization-missing-the-mark/

has a slideshare embed at the bottom, which gets imported as:

//www.slideshare.net/slideshow/embed_code/key/scA1akaUei9IOH

into the .md, when i prepended https: everything seems to work as expected.

https://theblog-adobe.hlx.page/en/drafts/migrated/2020/04/14/using-creativity-to-honorheroes-doing-extraordinary-things-during-covid-19.html

Many spaces are missing. Source DOM contains spans where space is trimmed.

Assuming it's an importer problem: the avatar URL here has a funny .bin extension and the image is broken on all posts by this author. Haven't seen other broken ones so far...

https://staging--theblog--adobe.hlx.page/en/authors/adobe-communications-team.html

No activity since 2020.

Imported files go into /en/publish

Generated md should look like:

Bla bla bla.

Social:
- [](twitter)
- [](linkedin)
...

Instead of

Bla bla bla.

- [](twitter)
- [](linkedin)
...

cc @rofe

See https://theblog-adobe.hlx.page/en/drafts/migrated/2015/03/12/mobile-reality.html

Images in the content are wrapped with a link to themselves which allows reader to open the image in full size. The link still points to the original image on theblog.adobe.com while it should point to the blobstore image link.

Similar to adobe/helix-theblog-scanner#5

With the new folder structure, the new import destination needs to be (temporarily) changed from en/archive to en/drafts/migrated with yyy/mm/dd sub folders.

PS: the final import destination will be en/publish directly

i seems that depending on the post there is a section about comments / replies at the end of the .md

see:
https://theblog.helix-demo.xyz/en/archive/2020/your-tutorial-resource-guide-to-photography-editing.html

i think this is a section that we probably shouldn't import.

Embeds currently use Image-Style. This should be changed to IA Writer-Style for readability and simplicity. Also we'll have less frictions with (complex) image handling.

See https://github.com/adobe/helix-pipeline/blob/master/docs/markdown.md#ia-writer-style-embeds for definition.

Articles can contain incomplete links (e.g. missing schema prefix) and will therefore get rendered as relative links. I think if a URL starts with www, it is safe to assume that https:// should be added.

An example has been posted here: adobe/theblog#314

/admin/importer/mappings.xlsx contains 2 tables with:

• a mapping between topic names found on theblog and new category names
• a mapping between product names found on theblog and filtered / cleaned up product names

This allows to import topics and products and do a cleanup (on the fly). The importer should consume those data and map accordingly during import.

Files like en/publish/2019/04/01/k%d3%93rcher-cleans-up-software-licensing-with-an-adobe-etla-mobilizing-for-growth.md must be saved as en/publish/2019/04/01/kärcher-cleans-up-software-licensing-with-an-adobe-etla-mobilizing-for-growth.md otherwise the file is not found during rendering (404).

Idea is to have an Excel spreadsheet (we already have urls.xlsx with the url imported and the publication date) with all the data available for a post entry, similar to what is stored in Algolia. The spreadsheet could then become a replacement for Algolia leveraging Helix Data Embed.

cc @davidnuescheler @dominique-pfister

2 types of issues.

Pure import process issues
Errors resulting of import edge cases.

• blob storage image copy fails if image is being a redirect (x-ms-copy-source does not support redirect)
• some images have weird / unsupported urls. Need to examine case by case

**Unsupported md data structure

• <figure> tag transformed to code block: https://theblog--adobe.hlx.page/en/drafts/migrated/2020/04/01/get-to-great-faster.html
• transform to embeds for list of supported embeds - see adobe/theblog#166
  • youtube: https://theblog.adobe.com/so-how-are-you-feeling/ -> https://theblog-adobe.hlx.page/en/drafts/migrated/2020/04/13/so-how-are-you-feeling.html
  • soundcloud: https://theblog.adobe.com/building-a-better-onboarding-experience/ -> https://theblog-adobe.hlx.page/en/drafts/migrated/2020/02/13/building-a-better-onboarding-experience.html
  • slideshare: https://theblog.adobe.com/5-takeaways-from-this-months-digital-price-index/ -> https://theblog-adobe.hlx.page/en/drafts/migrated/2019/04/24/5-takeaways-from-this-months-digital-price-index.html
• transform banners - see adobe/theblog#162

Non exhaustive list - WIP/

https://theblog.adobe.com/a-place-of-belonging

Author name contains a dot (Dr. Eddie Webb). This should just be filtered out.

This page has a large number of emphasis mess: https://blog.adobe.com/en/drafts/migrated/2020/04/30/protect-your-most-sensitive-digital-assets-while-working-from-home.html

Looks quite different from original: https://theblog.adobe.com/protect-your-most-sensitive-digital-assets-while-working-from-home

• spacing before and after emphasis is wrongly handled
• links at wrong place (outside of em)
• same applies to strong

Open https://theblog-adobe.hlx.page/en/drafts/migrated/2015/03/12/mobile-reality.html

Folder: 2015/03/12 (YYYY/MM/DD)
Publication date in the article: 03-11-2015 (MM/DD/YYYY)

Importer takes the property="article:published_time" meta entry in the DOM (basically, the "real" publication date).
It should take the same than the one displayed for consistency.

Remove a line from the urls.xlsx file and run the import for the removed url - while import happens, it fails in the last step:

[ERROR] Duplicate dictionary_item: 'dictionary_id=3iRICZS9xdH9gSxfkWupUT item_key=/10-must-haves-for-your-data-toolkit service_id=6v0sHgrPTGUGS5PHOXZ0H1'
[ERROR] error { params: { url: 'https://theblog.adobe.com/10-must-haves-for-your-data-toolkit/', force: true }, error: { FastlyError: Duplicate dictionary_item: 'dictionary_id=3iRICZS9xdH9gSxfkWupUT item_key=/10-must-haves-for-your-data-toolkit service_id=6v0sHgrPTGUGS5PHOXZ0H1'
       at request.then (/Users/acapt/work/dev/helix/theblog/helix-theblog-importer/node_modules/@adobe/fastly-native-promises/src/httpclient.js:107:17)
       at process._tickCallback (internal/process/next_tick.js:68:7) data: { msg: 'Duplicate record', detail: 'Duplicate dictionary_item: \'dictionary_id=3iRICZS9xdH9gSxfkWupUT item_key=/10-must-haves-for-your-data-toolkit service_id=6v0sHgrPTGUGS5PHOXZ0H1\'' }, status: 409, code: 'Duplicate record', name: 'FastlyError' } }

We should be able to re-import

☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet.
We recommend using:

• CircleCI
• Travis CI
• Buildkite
• CodeShip
• Azure Pipelines
• TeamCity
• Buddy
• AppVeyor
  But Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.

The importer seems to be modifying product (and topic) .md files in SharePoint a lot:

I think we should avoid that, unless we really need to change something.

Recent changes store the topics including the whole parent tree (all based on mappings document). We should change that to store only the leafs.
We'll be able to create queries based on the taxonomy document to find topics / categories based on their parents. Same to display the full tree at the bottom of the posts (list of topics), we can use the taxonomy document to display all if needed.

cc @rofe

Some embeds are not imported properly and even fire an exception during import.

See for the list adobe/theblog#186

Until now, "embeds" were surrounded with a div and a .embed-wrapper class. iframe was in that div. Now, we found a case were only an iframe was in the DOM.
In general, all iframes can be transformed into embeds since we cannot do anything with them in md.

See https://theblog.adobe.com/using-creativity-to-honorheroes-doing-extraordinary-things-during-covid-19/

Import of https://theblog.adobe.com/adobe-withdraws-from-nab-show-2020 leads to broken layout: https://blog.adobe.com/en/drafts/migrated/2020/03/09/adobe-withdraws-from-nab-show-2020.html#gs.8lkhs3

(Premiere Pro link + date on top left).

Follow up of adobe/theblog#330 (comment)

Importer checks if a file (topics, products, authors...) already exists to not override it but it only checks for the .md version. When file requires modification, authors "migrate" the file to .docx and delete the .md version. Which comes back on next import...
Importer should check for .md and .docx version of the file.

According to the link in the top navigation, the category's name is "Trends & Research", not "Trends Research". This error stems from the mapping document where the ampersand has been omitted.

When importing from the original blog, we search iframes for the src:

The problem is these links don't work with our embed services; what we need is link to the original content not the link given in the src attribute of an <iframe>.

For example, these link cannot be embedded properly by Iframely but have been extracted during an import. You can check them at https://iframely.com/embed/:

https://embed.spotify.com/?uri=spotify4gzpq5DPGxSnKTe4SA8HAU

https://www.linkedin.com/video/embed/live/urn:li:ugcPost:6666809526391570432

https://w.soundcloud.com/player/?url=https%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F673363193&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true

@trieloff @kptdobe do you think this should be a helix-embed issue? I was thinking maybe we can make helix-embed more robust to these kinds of links.