Code Monkey home page Code Monkey logo

wikijs-sitemap's Introduction

Wiki.js Sitemap Generator

This software allows you to generate a sitemap for your Wiki.js instance.

Currently, it only supports Postgres, but support for MySQL will be added if requested or if I have the time.
It supports Postgres, MySQL and MariaDB.

You can run it as a standalone Node.js program or within a Docker container.

How it works:

  1. It connects to your Wiki.js database
  2. It generates a sitemap for public posts.

Demo

You can view a demo of the sitemap on the following Wiki.js site: https://testwiki.hostwiki.io
The sitemap can be accessed at: https://testwiki.hostwiki.io/sitemap.xml

Limitations

  • Does not handle splitting the sitemap in the event of exceeding the 50,000 URL limit for sitemaps
  • It regenerates the sitemap every 24 hours (will be configurable in future updates)
  • Only supports postgres (need MYSQL or SQLite support? create an issue)

Requirements

  • Wiki.js (with Postgres, MySQL or MariaDB)
  • Reverse proxy (e.g Nginx, Apache)

To use, you must be serving your Wiki.js instance over a reverse proxy server.

Installation

.env config file content

The service is exposed via port 3012 to avoid any port conflicts with Wiki.js.

Standalone Nodejs

Edit the .env to include your database credential.

git clone https://github.com/hostwiki/wikijs-sitemap
cd wikijs-sitemap
npm ci install

Start the server with

node server

To keep the nodejs program running, you can use pm2 or run it as a service.

Docker

Make sure to pass the correct environment variables.
The DB_TYPE accepts postgres, mysql and mariadb as possible values. It defaults to postgres if not set.
You use DB_PASS or DB_PASS_FILE to set your database password.

-e DB_TYPE=postgres
-e DB_HOST=
-e DB_PORT=
-e DB_PASS=
-e DB_USER=
-e DB_NAME=
Docker Compose

You can find a Docker Compose examples for Postgres and MySQL in the example directory.

Docker create

If you wish to run it via docker create:

docker create --name=wikijs-sitemap -e DB_TYPE=postgres -e DB_HOST=db -e DB_PORT=5432 -e DB_PASS_FILE=/etc/wiki/.db-secret -v /etc/wiki/.db-secret:/etc/wiki/.db-secret:ro -e DB_USER=wiki -e DB_NAME=wiki --restart=unless-stopped --network=wikinet -p 3012:3012 hostwiki/wikijs-sitemap:latest
docker start wikijs-sitemap

Docker run

If you wish to run it via docker run:

docker run --name wikijs-sitemap -e DB_TYPE=postgres -e DB_HOST=db -e DB_PORT=5432 -e DB_PASS_FILE=/etc/wiki/.db-secret -v /etc/wiki/.db-secret:/etc/wiki/.db-secret:ro -e DB_USER=wiki -e DB_NAME=wiki --restart=unless-stopped --network=wikinet -p 3012:3012 -d hostwiki/wikijs-sitemap:latest

After a successful setup, the sitemap will be available at localhost:3012/sitemap.xml. Next, you have to proxy it via a reverse proxy server like Nginx or Apache.

Reverse proxy configuration for Nginx

location /sitemap.xml {
    proxy_pass http://127.0.0.1:3012/sitemap.xml;
    proxy_http_version 1.1;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Reverse proxy configuration for Apache

Make sure to install the necessary Apache mods.

sudo a2enmod proxy proxy_http

Then add this to your Wiki.js apache configuration file.

ProxyPreserveHost On
ProxyPass /sitemap.xml http://localhost:3012/sitemap.xml
ProxyPassReverse /sitemap.xml http://localhost:3012/sitemap.xml

wikijs-sitemap's People

Contributors

philipinho avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

vmario89 whitehua

wikijs-sitemap's Issues

db password

I have successfully started wikijs-sitemap.
Thank you for this!
Only I am too stupid to put the wikijs password in a file, which I try to do, I only get a successful postgres database connect if the password is unencrypted in the docker_run file. So like this:
-e 'DB_PASS=my!password'
Other Methods are all wrong...
What am I doing wrong?
I use Debian Bookworm.

Error: Cannot find module 'node-cron'

Hi! Thanks a lot for your tool. Yesterday was the first time I noticed missing sitemap.xml for wiki.js and first link in google lead to your repository created two days ago :)

Got following issue:

wiki@myhost:/data/wiki/wikijs-sitemap$ node server
node:internal/modules/cjs/loader:998
  throw err;
  ^

Error: Cannot find module 'node-cron'
Require stack:
- /data/wiki/wikijs-sitemap/server.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Function.Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1067:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/data/wiki/wikijs-sitemap/server.js:3:14)
    at Module._compile (node:internal/modules/cjs/loader:1165:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1219:10)
    at Module.load (node:internal/modules/cjs/loader:1043:32)
    at Function.Module._load (node:internal/modules/cjs/loader:878:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/data/wiki/wikijs-sitemap/server.js' ]
}

Resolved it by installing node-cron

wiki@myhost:/data/wiki/wikijs-sitemap$ npm install --save node-cron

Probably it might be added to Installation section

Sitemap mysql undefined domain

Hi!
Thanks for implementing the sitemap for mysql.
I have a problem below in the screenshot.
image

Wikijs in a container according to the examples. Tried to proxy through nginx and haproxy.
Could you suggest what is the problem.
Why is the domain not visible (undefined)?
Maybe I forgot to specify it somewhere?

Sitemap regeneration period

hourly sitemap regeneration is a bit overkill, why not set it to weekly by default? Keep in mind that Google is periodically crawling the sitemap!

Docker image could not be run on linux/amd64 platform

I try to install docker container on linux host but got errors:

The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested

When I specified the platform in yml, now I getting:

image with reference hostwiki/wikijs-sitemap:1 was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64/v8

How can I build it and run on Linux machine?

@Philipinho

Locales missing in urls

When generating the sitemap XML, locales seem to be ignored.

For example I've got

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<!-- Wiki.js sitemap generator by https://hostwiki.com -->
<url>
    <loc>https://domain.tld/new-page</loc>
    <lastmod>2023-11-30T13:49:20.203Z</lastmod>
  </url>
<url>
    <loc>https://domain.tld/new-page</loc>
    <lastmod>2023-11-30T13:49:24.491Z</lastmod>
  </url>
</urlset>

instead of

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<!-- Wiki.js sitemap generator by https://hostwiki.com -->
<url>
    <loc>https://domain.tld/en/new-page</loc>
    <lastmod>2023-11-30T13:49:20.203Z</lastmod>
  </url>
<url>
    <loc>https://domain.tld/fr/new-page</loc>
    <lastmod>2023-11-30T13:49:24.491Z</lastmod>
  </url>
</urlset>

sitemap.xml is not generated because static/sitemap.xml does not exist.

Summary

When trying to generate a sitemap using the Docker version ``wikijs-sitemap'', an error occurs if the file does not exist.

 [[2023-12-19T15:06:59.006Z] Error generating sitemap (attempt 1/15):  Error: Database connection error: ENOENT: no such file or directory, open 'static/sitemap.xml'
     at generateSitemap (/wiki/sitemap/generate-sitemap.js:47:15)
     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
     at async generateSitemapAndLog (/wiki/sitemap/server.js:40:13)
     at async Server.<anonymous> (/wiki/sitemap/server.js:62:5)

Expected Behavior

sitemap.xml with the following content is generated.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<!-- Wiki.js sitemap generator by https://hostwiki.com -->
<url>
    <loc>https://example.com/ja/hoge/fuga</loc>
    <lastmod>2021-09-21T01:20:32.146Z</lastmod>
  </url>
<url>
    <loc>https://example.com/ja/foo</loc>
    <lastmod>2022-02-18T13:39:07.843Z</lastmod>
  </url>
<url>
    <loc>https://example.com/ja/bar/fizz</loc>
    <lastmod>2021-11-03T02:48:14.114Z</lastmod>
  </url>
<url>
    <loc>https://example.com/ja/buzz</loc>
    <lastmod>2023-07-01T12:43:09.629Z</lastmod>
  </url>
<url>

Actual Behavior

sitemap.xml not generated.

Error reading sitemap.xml file

with the error below

Server listening on port 3012
[[2023-12-20T00:09:27.130Z] Error generating sitemap (attempt 1/15):  Error: Database connection error: ENOENT: no such file or directory, open 'static/sitemap.xml'
    at generateSitemap (/wiki/sitemap/generate-sitemap.js:47:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateSitemapAndLog (/wiki/sitemap/server.js:40:13)
    at async Server.<anonymous> (/wiki/sitemap/server.js:62:5)
[[2023-12-20T00:09:27.130Z] Error generating sitemap (attempt 2/15):  Error: Database connection error: ENOENT: no such file or directory, open 'static/sitemap.xml'
    at generateSitemap (/wiki/sitemap/generate-sitemap.js:47:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateSitemapAndLog (/wiki/sitemap/server.js:40:13)
    at async Server.<anonymous> (/wiki/sitemap/server.js:62:5)
[[2023-12-20T00:09:27.130Z] Error generating sitemap (attempt 3/15):  Error: Database connection error: ENOENT: no such file or directory, open 'static/sitemap.xml'
    at generateSitemap (/wiki/sitemap/generate-sitemap.js:47:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateSitemapAndLog (/wiki/sitemap/server.js:40:13)
    at async Server.<anonymous> (/wiki/sitemap/server.js:62:5)
[[2023-12-20T00:09:27.130Z] Error generating sitemap (attempt 4/15):  Error: Database connection error: ENOENT: no such file or directory, open 'static/sitemap.xml'
    at generateSitemap (/wiki/sitemap/generate-sitemap.js:47:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateSitemapAndLog (/wiki/sitemap/server.js:40:13)
    at async Server.<anonymous> (/wiki/sitemap/server.js:62:5)
[Error: ENOENT: no such file or directory, open 'static/sitemap.xml'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'static/sitemap.xml'
}
[[2023-12-20T00:09:27.130Z] Error generating sitemap (attempt 5/15):  Error: Database connection error: ENOENT: no such file or directory, open 'static/sitemap.xml'
    at generateSitemap (/wiki/sitemap/generate-sitemap.js:47:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateSitemapAndLog (/wiki/sitemap/server.js:40:13)
    at async Server.<anonymous> (/wiki/sitemap/server.js:62:5)
[Error: ENOENT: no such file or directory, open 'static/sitemap.xml'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'static/sitemap.xml'
}

Steps to Reproduce

  1. Configure docker-compose.yml according to README
  2. Start Docker

Environment

  • Docker version 24.0.2, build cb74dfc
  • Docker Compose version v2.16.0
  • wikijs v2.5.300
  • hostwiki/wikijs-sitemap:latest (As of 2023/12/20)
  • PostgreSQL 11.22

sitemap is generated at the first start and restart but not while running later

wikijs-sitemap database connection error.
'Error: Database connection error: Unable to acquire a connection' but when I restart the container it works
` # this is an example from today
wikijs-sitemap | at generateSitemap (/wiki/sitemap/generate-sitemap.js:54:15)
wikijs-sitemap | at async Task.generateSitemapAndLog [as _execution] (/wiki/sitemap/server.js:40:13)
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 14/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | at generateSitemap (/wiki/sitemap/generate-sitemap.js:54:15)
wikijs-sitemap | at async Task.generateSitemapAndLog [as _execution] (/wiki/sitemap/server.js:40:13)
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 15/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | at generateSitemap (/wiki/sitemap/generate-sitemap.js:54:15)
wikijs-sitemap | at async Task.generateSitemapAndLog [as _execution] (/wiki/sitemap/server.js:40:13)
wikijs-sitemap | Server listening on port 3012
wikijs-sitemap | [2024-04-11T06:39:53.025Z] sitemap generated successfully.

Container was restarted @6:39

#################### the whole of 15 attempts
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 1/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 2/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 3/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 4/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 5/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 6/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 7/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 8/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 9/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 10/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 11/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 12/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 13/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 14/15): Error: Database connection error: Unable to acquire a connection
wikijs-sitemap | [[2024-04-11T00:00:00.981Z] Error generating sitemap (attempt 15/15): Error: Database connection error: Unable to acquire a connection
`
I find it striking that the time stamp to the microsecond is always exactly the same after 15 attempts.

Are onother users facing this behavior, too?

Feature Request: Blacklists

In wikijs you can set fine grained privileges for sites and paths.
When there are pages that cannot be seen by the public but only by authorized personel, they still appear in the sitemap.xml.
Please offer a environment variable that carries a list of paths that won't appear in the sitemap.

Sitemap mysql

Hello.
Is it possible to make a sitemap for mysql when you have time?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.