Code Monkey home page Code Monkey logo

wordpress-importer's Introduction

Storyblok Logo

Storyblok WordPress importer

A simple script for migrating content from WordPress to Storyblok.

Follow @Storyblok
Follow @Storyblok

🚀 Usage

Prerequisets

This script has been tested on WordPress v5 with API v2. WordPress REST API must be publicly available during the migration process as this script won't handle authentication. On the Storyblok side you just need a space. In case the space is not an empty one, we recommend to test it before with a copy of the original space to make sure the migration process doesn't cause an issue to the existing content.

How to use

To use the script, just import it, initialise a new instance of the Wp2Storyblok class and run the Wp2Storyblok.migrate() method.

import {Wp2Storyblok} from './index.js'

const wp2storyblok = new Wp2Storyblok('http://yoursite.com/wp-json', {
  token: 'storyblok-oauth-token',
  space_id: 110836,
  blocks_mapping: [
    {
      name: 'core/paragraph',
      new_block_name: 'richtext',
      schema_mapping: {
        'attrs.content': 'content'
      }
    },
    {
      name: 'core/image',
      new_block_name: 'image',
      schema_mapping: {
        'attrs.url': 'image'
      }
    }
  ],
  content_types: [
    {
      name: 'pages',
      new_content_type: 'page',
      folder: 'your-custom-folder',
      taxonomies: [
        {
          name: 'categories',
          field: 'categories',
          type: 'value'
        }
      ],
      schema_mapping: {
        title: 'name',
        '_links.wp:featuredmedia.0': 'content.preview_image',
        content: {
          field: 'content.body_items',
          component: 'rich-text',
          component_field: 'content',
          categories: 'content.categories'
        }
      }
    }
  ]
})

wp2storyblok.migrate()

Parameters

  • endpoint String, The main endpoint for the WordPress REST API, without the /wp/v2/ part
  • settings Object
    • (region String, Optional, The region of your Storyblok space. Default is eu)
    • token String, The oauth token for the management API that can be retrieved in the account section of https://app.storyblok.com
    • space_id Integer, The id of your space
    • content_types Array of Objects
      • name String, The name of the content type in WordPress
      • new_content_type String, The name of the content type in Storyblok
      • schema_mapping Object, The mapping of the fields from WordPress to the fields in Storyblok. More info about the mapping here
    • (blocks_mapping Array of Objects, Optional, More info here)
      • name String, The name of the block in WordPress
      • (new_block_name String, Optional, The name of the component in Storyblok. If not set the original name of the component will be used)
      • schema_mapping Object, The mapping of the fields from WordPress to the fields in Storyblok. More info about the mapping here
    • (taxonomies Array of Objects, Optional, The taxonomies of the content type, More info here)
      • name String, The name of the taxonomy in WordPress
      • field String, The name of the source field in WordPress
      • (type String, Set to value to replace the taxonomy id with the slug of the taxonomy value. Set to relationship or leave empty in case you imported also the taxonomy entries as stories and you want to link the taxonomy entry with an option or multi-option field by UUID )
    • (folder String, Optional, The full slug of the destination folder in Storyblok)

Fields Mapping

The fields mapping object requires you to use the name of the field from WordPress as keys of the attributes and the name of the field in Storyblok as its value. You can also target subproperties and array elements using the dot notation.

"schema_mapping": {
  "_links.wp:featuredmedia.0": "content.preview_image"
}

In case you want a field to be migrated as content inside a nested block in a field in Storyblok, you can do that defining the target as an object with the following properties:

  • field String, The name of the field in Storyblok
  • component String, The name of the component you want to store inside the above field
  • component_field String, The name of the field inside the component where you want to migrate the content
"schema_mapping": {
  "content": {
    "field": "content.body_items", 
    "component": "rich-text", 
    "component_field": "content" 
  }
}

WordPress Blocks Mapping

You can import blocks created with Gutenber as components in Storyblok. To achieve this you need to install the REST API blocks plugin and fill out the blocks_mapping property in the migration settings. You need to create an array of objects where you specify the name of the block from Gutenberg (called blockName in the REST API), the name of the component in Storyblok and then the schema mapping in the same format as for the content types. The blocks from Gutenberg are returned by the REST API inside the main object of an entry in a property called blocks.

  {
    "name": "pages",
    "new_content_type": "page",
    "folder": "",
    "schema_mapping": {
      "title": "name",
      "blocks": "content.body"
    },
  },

Importing Taxonomies

Taxonomies can be imported along with the other fields. You need to fill out the taxonomies settings in the settings of your content_type and the script will get the taxonomy value from WordPress instead of the taxonomy id and it will add it to your Stories in the field you chose.

🔗 Related Links

  • How To Migrate From WordPress To A Headless CMS: In this article, we will look at when it makes sense to migrate from a monolithic project to a headless setup and the benefits that come with it. In addition to a step-by-step guide on how to migrate WordPress to Storyblok Headless CMS, the problems that will arise during the process and how to deal with them;
  • Storyblok Technologies Hub: we prepared technology hubs so that you can find selected beginner tutorials, videos, boilerplates, and even cheatsheets all in one place.

ℹ️ More Resources

Support

Contributing

Please see our contributing guidelines and our code of conduct. This project use semantic-release for generate new versions by using commit messages and we use the Angular Convention to naming the commits. Check this question about it in semantic-release FAQ.

License

This repository is published under the MIT license.

wordpress-importer's People

Contributors

christianzoppi avatar michelepeghini-revium avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wordpress-importer's Issues

config.folder does not seem to be used

The readme mentions a top-level folder (in config) but it doesn't seem to work nor does it seem to be used in the code.
There is however a content_type.folder field, so maybe this is just a mistake in the documentation?

Bug in regular expression matching assets

const assets_regex = new RegExp(`(\\"((http)?s?:?(\\/?\\/[^"]*.(${this.settings.import_assets.types.join('|')})))(\\\\)?")`, "g")

If we look at that regular expression, it basically boils down to something like:

<snip> [^"]*.(jpg|png|pdf <snip> ) <snip>

That dot/period looks suspicious - why do we care that there is at least one character before the file extension?
I am guessing that the intention there was to match a literal dot (as in .png) and not "any character" (as in *png), in which case it should have been escaped like \. (but should be escaped twice in that string: \\.).

In general, that code looks like it works, but it also matches things like https://somesite.com/amazing-blog-about-png. This leads to errors later on, in my case something like so (I still don't fully understand the reason behind error, but the regex fix should fix it):
image

Handle authentication when Wordpress REST API is not public

Not really an issue, this is more like an improvement suggestion. It would be helpful to have a piece of authentication handling in cases the Wordpress REST API is not public.

Expected Behavior

Authentication handling would allow people to use the plugin even when the Wordpress REST API is not public.

Current Behavior

Currently the plugin doesn't handle authentication so you're able to use it only with public Wordpress REST APIs.

Steps to Reproduce

Run the migration script and get a 401: Unauthorized when trying to access the Wordpress REST API that isn't publicly available.

Code block language not migrated properly

The turndown module is able to extract code block language defined as a class, but only when codeBlockStyle is "fenced" (which is not the default).

So for example:

const html = '<pre><code class="language-css">h1 { font-size: 20px; }</code></pre>'

(new TurndownService()).turndown(html)
// ❌ results:
//     h1 { font-size: 20px; }

(new TurndownService({ codeBlockStyle: 'fenced' })).turndown(html)
// ✅ results:
// ```css
// h1 { font-size: 20px; }
// ```

Apparently wp2storyblok, handles both cases - so it seems beneficial to use fenced style instead of indented style, to also migrate the language:

const turndownService = new TurndownService()


Side note: in my case I need it to be able to safely migrate content that is not supported in markdown (as components).
For example, say we want to migrates <iframe>s, first I convert all iframes to <pre><code class="language-sb-component-json">{"component":"iframe"....}</code></pre> and then I expand it later to a richtext component.

Of course, this could be improved or automated inside the migration module..

TypeError: Cannot read property 'field' of undefined

I'm getting the following error:

TypeError: Cannot read property 'field' of undefined
    at file:///.../node_modules/wp2storyblok/src/migration.js:348:184
    at Array.map (<anonymous>)
    at file:///.../node_modules/wp2storyblok/src/migration.js:346:52
const field_name = typeof content_type.schema_mapping[taxonomy.field] === 'string' ? content_type.schema_mapping[taxonomy.field] : content_type.schema_mapping[taxonomy.field].field

Maybe the condition should be reversed to check for an object?

const field_name = typeof content_type.schema_mapping[taxonomy.field] !== 'object' ? content_type.schema_mapping[taxonomy.field] : content_type.schema_mapping[taxonomy.field].field

PS: the "submit an issue" link seems to be broken: https://github.com/storyblok/wordpress-importer#support

Can not run

i want to import my live wordpress website at https://www.indomascot.com

i use this code below

import { Wp2Storyblok } from "./index.js";

const wp2storyblok = new Wp2Storyblok("https://www.indomascot.com/wp-json", {
  token: "my-api-key",
  space_id: my-space-id,
  content_types: [
    {
      name: "pages",
      new_content_type: "page",
      folder: "",
      schema_mapping: {
        title: "name",
        "_links.wp:featuredmedia.0": "content.preview_image",
        content: {
          field: "content.body_items",
          component: "rich-text",
          component_field: "content",
        },
      },
    },
  ],
});

wp2storyblok.migrate();

it return error

(node:2635) UnhandledPromiseRejectionWarning: Error: Request failed with status code 401
    at createError (/Volumes/HDD/gangsar/wordpress-importer/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Volumes/HDD/gangsar/wordpress-importer/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/Volumes/HDD/gangsar/wordpress-importer/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (events.js:387:35)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:2635) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:2635) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Error: Request failed with status code 404

Hi! I'm trying to import 342 posts from WP to SB but I get this error when I run the script. I can't find any media or relation to this media 3386 in my WP database. Any ideas?

❯ npm run migrate

> [email protected] migrate
> node ./migrate-wp-to-storyblok.js

Fetched all the entries of posts type
/home/****/wordpress-importer/node_modules/axios/lib/core/createError.js:16
  var error = new Error(message);
              ^

Error: Request failed with status code 404
    at createError (/home/****/wordpress-importer/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/home/****/wordpress-importer/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd /home/****/wordpress-importer/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  config: {
    url: 'https://blog.******/wp-json/wp/v2/media/3386',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'User-Agent': 'axios/0.21.1'
    },
    transformRequest: [ [Function: transformRequest] ],
...
...
...
[Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype] {
        accept: [ 'Accept', 'application/json, text/plain, */*' ],
        'user-agent': [ 'User-Agent', 'axios/0.21.1' ],
        host: [ 'Host', 'blog.****' ]
      }
    },
    data: {
      code: 'rest_post_invalid_id',
      message: 'ID inválido de post.',
      data: { status: 404 }
    }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]

My setup file contains:

content_types: [
    {
      name: 'posts',
      new_content_type: 'post',
      folder: 'blog',
      schema_mapping: {
        title: 'title',
        '_links.wp:featuredmedia.0': 'featured_image',
        content: 'content'
      }
    }
  ]

Empty page (rendered = '') causes error

} else if (typeof field_value === 'object' && field_value.rendered) {

If field.value.rendered is an empty string, that condition is skipped and an object is returned, causing the error below inside turndown:

Uncaught TypeError: [object Object] is not a string, or an element/document/fragment node.

A fix would be to check if rendered is actually undefined, eg:

    } else if (typeof field_value === 'object' && field_value.rendered !== undefined) {

Note that this might also apply to line 64 with href.

Edit: I tested the change locally and it seems to work.

Import Capped

Hello. When using Dawn Traoz's version of the plugin and mapping out the post fields correctly. I do have them move over. However its only randomly 100 items or 150 items. I am never able to go more than that. The site i'm migrating has roughly 1000 posts.

Title with html entities not properly decoded

WordPress titles typically do not contain HTML, as far as I know, but special characters seem to be encoded as html entities (see also: https://stackoverflow.com/q/52229126/314056).

This is my solution for now:

const oldPopulateFields = wp2storyblok.populateFields;
wp2storyblok.populateFields = async function (data, component_name, mapping, taxonomies) {
  const fields = await oldPopulateFields.call(wp2storyblok, data, component_name, mapping, taxonomies);

  // note: textarea is a trick to avoid html injection, see also: https://stackoverflow.com/a/7394787/314056
  fields.name = new JSDOM(
    `<body><textarea>${fields.name}</textarea></body>`
  ).window.document.body.firstChild.value;

  return fields;
};

It uses jsdom, which I'm already using for other purposes.

Since this project uses turndown, (which in turn uses domino), maybe that could be used in place of jsdom.

Add env file to `.gitignore`

Please add:

.env
.env.local
.env.development.local
.env.test.local
.env.production.local

to .gitignore. I accidentally committed my API token in a public repo after forking this. Otherwise great script!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.