Code Monkey home page Code Monkey logo

sitemap's Introduction

Sitemap

Build Status Hex version Hex downloads Inline docs hex.pm

Generating sitemap.xml

Installation

If available in Hex, the package can be installed as:

  1. Add sitemap to your list of dependencies in mix.exs:
def deps do
  [{:sitemap, "~> 1.1"}]
end
  1. Ensure sitemap is started before your application:
def application do
  [extra_applications: [:sitemap]]
end

Usage

sitemap helps you define a module with a generate function which will build a sitemap for your site. You must decide how to call generate - via a manual Mix task, a recurring background job, or whatever you choose.

The resulting sitemap is currently written to a file. Because some web hosts do not support writing to the filesystem, we plan to support uploading to S3 in the future.

You can always specify your own adapter module with a write/2 function and persist the sitemaps wherever you like.

Basic
defmodule Sitemaps do
  use Sitemap

  def generate do
    create do
      # list each URL that should be included
      add "path1", priority: 0.5, changefreq: "hourly", expires: nil, mobile: true
      # ...
    end

    # notify search engines (currently Google and Bing) of the updated sitemap
    ping()
  end
end
With Phoenix
defmodule Sitemaps do
  alias MyAppWeb.{Endpoint, Router.Helpers}

  use Sitemap,
    host: "http://#{Application.get_env(:myapp, Endpoint)[:url][:host]}",
    files_path: "priv/static/sitemaps/",
    public_path: "sitemaps/"

  def generate do
    create do
      # list each URL that should be included, using your application's routes
      add Helpers.entry_path(Endpoint, :index), priority: 0.5, changefreq: "hourly", expires: nil
      add Helpers.entry_path(Endpoint, :about), priority: 0.5, changefreq: "hourly", expires: nil
      # ...
    end

    # notify search engines (currently Google and Bing) of the updated sitemap
    ping()
  end
end

Ways to set sitemap's options

Set options via the use statement
defmodule Sitemaps do
  use Sitemap, compress: false, host: "http://example.com"

  def generate do
    create do
      add "path1", priority: 0.5, changefreq: "hourly"
      add "path2", priority: 0.5, changefreq: "hourly"
    end

    ping()
  end
end
Set options via arguments to create
defmodule Sitemaps do
  use Sitemap

  def generate do
    create compress: false, host: "http://example.com" do
      add "path1", priority: 0.5, changefreq: "hourly"
      add "path2", priority: 0.5, changefreq: "hourly"
    end

    ping()
  end
end
Set options via Mix config
use Mix.Config

config :sitemap, [
  compress: false,
  host: "http://example.com",
]
Set options via environment variables
SITEMAP_COMPRESS=false SITEMAP_HOST=http://example.com mix run ./sitemap.exs

And you guys should follow mix task documents, here:

Available options
Name Default Value Environment -
max_sitemap_files 10000 SITEMAP_MAXFILES Max sitemap links per index file
max_sitemap_links 10000 SITEMAP_MAXLINKS Max links per sitemap
max_sitemap_news 1000 SITEMAP_MAXNEWS Max news sitemap per index_file
max_sitemap_images 1000 SITEMAP_MAXIMAGES Max images per url
max_sitemap_filesize 5000000 SITEMAP_MAXFILESIZE Bytes
host http://www.example.com SITEMAP_HOST Your domain, also host with http scheme.
filename sitemap SITEMAP_FILENAME Name of sitemap file.
files_path sitemap SITEMAP_SITEMAPS_PATH After domain path's location on URL.
public_path sitemap SITEMAP_PUBLIC_PATH Write sitemap files to this local path.
adapter Sitemap.Adapters.File SITEMAP_ADAPTER You'd change to write each filesystem
verbose true SITEMAP_VERBOSE Getting more information in sitemap working.
compress true SITEMAP_COMPRESS Gzip compression.
create_index auto SITEMAP_CREATE_INDEX Generating sitemps to this directory path.

Features

Current Features or To-Do

Supports: generate kind of some sitemaps

News Sitemaps

defmodule Sitemaps do
  use Sitemap, compress: false, host: "http://example.com"

  def generate do
    create do
      add "index.html", news: [
           publication_name: "Example",
           publication_language: "en",
           title: "My Article",
           keywords: "my article, articles about myself",
           stock_tickers: "SAO:PETR3",
           publication_date: "2011-08-22",
           access: "Subscription",
           genres: "PressRelease"
         ]
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/index.html</loc>
 <lastmod>2016-05-30T13:13:12Z</lastmod>
 <news:news>
   <news:publication>
     <news:name>Example</news:name>
     <news:language>en</news:language>
   </news:publication>
   <news:title>My Article</news:title>
   <news:access>Subscription</news:access>
   <news:genres>PressRelease</news:genres>
   <news:keywords>my article, articles about myself</news:keywords>
   <news:stock_tickers>SAO:PETR3</news:stock_tickers>
   <news:publication_date>2011-08-22</news:publication_date>
 </news:news>
</url>

Look at Creating a Google News Sitemap as required.

Image sitemaps

defmodule Sitemaps do
  use Sitemap, compress: false, host: "http://example.com"

  def generate do
    create do
      add "index.html", images: [
           loc: "http://example.com/image.jpg",
           caption: "Caption",
           title: "Title",
           license: "https://github.com/ikeikeikeike/sitemap/blob/master/LICENSE",
           geo_location: "Limerick, Ireland",
         ]
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/image.html</loc>
 <lastmod>2016-05-31T13:32:40Z</lastmod>
 <image:image>
   <image:loc>http://example.com/image.jpg</image:loc>
   <image:caption>Caption</image:caption>
   <image:title>Title</image:title>
   <image:license>https://github.com/ikeikeikeike/sitemap/blob/master/LICENSE</image:license>
   <image:geo_location>Limerick, Ireland</image:geo_location>
 </image:image>
</url>

Look at Image sitemaps as required.

Video sitemaps

defmodule Sitemaps do
  use Sitemap, compress: true, host: "http://example.com"

  def generate do
    create do
      add "index.html", videos: [
           thumbnail_loc: "http://www.example.com/thumbs/123.jpg",
           title: "Grilling steaks for summer",
           description: "Alkis shows you how to get perfectly done steaks every time",
           content_loc: "http://www.example.com/video123.flv",
           player_loc: "http://www.example.com/videoplayer.swf?video=123",
           allow_embed: true,
           autoplay: true,
           duration: 600,
           expiration_date: "2009-11-05T19:20:30+08:00",
           publication_date: "2007-11-05T19:20:30+08:00",
           rating: 0.5,
           view_count: 1000,
           tags: ~w(tag1 tag2 tag3),
           tag: "tag4",
           category: "Category",
           family_friendly: true,
           restriction: "IE GB US CA",
           relationship: true,
           gallery_loc: "http://cooking.example.com",
           gallery_title: "Cooking Videos",
           price: "1.99",
           price_currency: "EUR",
           price_type: "own",
           price_resolution: "HD",
           uploader: "GrillyMcGrillerson",
           uploader_info: "http://www.example.com/users/grillymcgrillerson",
           live: true,
           requires_subscription: false
         ]
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/video.html</loc>
 <lastmod>2016-05-31T12:51:47Z</lastmod>
 <video:video>
   <video:title>Grilling steaks for summer</video:title>
   <video:description>Alkis shows you how to get perfectly done steaks every time</video:description>
   <video:player_loc allow_embed="yes" autoplay="ap=1">http://www.example.com/videoplayer.swf?video=123</video:player_loc>
   <video:content_loc>http://www.example.com/video123.flv</video:content_loc>
   <video:thumbnail_loc>http://www.example.com/thumbs/123.jpg</video:thumbnail_loc>
   <video:duration>600</video:duration>
   <video:gallery_loc title="Cooking Videos">http://cooking.example.com</video:gallery_loc>
   <video:rating>0.5</video:rating>
   <video:view_count>1000</video:view_count>
   <video:expiration_date>2009-11-05T19:20:30+08:00</video:expiration_date>
   <video:publication_date>2007-11-05T19:20:30+08:00</video:publication_date>
   <video:tag>tag1</video:tag>
   <video:tag>tag2</video:tag>
   <video:tag>tag3</video:tag>
   <video:tag>tag4</video:tag>
   <video:category>Category</video:category>
   <video:family_friendly>yes</video:family_friendly>
   <video:restriction relationship="allow">IE GB US CA</video:restriction>
   <video:uploader info="http://www.example.com/users/grillymcgrillerson">GrillyMcGrillerson</video:uploader>
   <video:price currency="EUR" resolution="HD" type="own">1.99</video:price>
   <video:live>yes</video:live>
   <video:requires_subscription>no</video:requires_subscription>
 </video:video>
</url>

Look at Video sitemaps as required.

Alternate Links

defmodule Sitemaps do
  use Sitemap, compress: true, host: "http://example.com"

  def generate do
    create do
      add "index.html", alternates: [
           href: "http://www.example.de/index.html",
           lang: "de",
           nofollow: true,
           media: "only screen and (max-width: 640px)"
         ]
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/video.html</loc>
 <lastmod>2016-06-01T14:05:05Z</lastmod>
 <xhtml:link href="http://www.example.de/index.html" hreflang="de" media="only screen and (max-width: 640px)" rel="alternate nofollow"/>
</url>

Look at Alternate Links as required.

Geo Sitemaps

defmodule Sitemaps do
  use Sitemap, compress: true, host: "http://example.com"

  def generate do
    create do
      add "geo.html", alternates: [
           format: "kml"
         ]
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/geo.html</loc>
 <lastmod>2016-06-01T14:15:25Z</lastmod>
 <geo:geo>
   <geo:format>kml</geo:format>
 </geo:geo>
</url>

Look at Geo Sitemaps as required.

Mobile Sitemaps

defmodule Sitemaps do
  use Sitemap, compress: true, host: "http://example.com"

  def generate do
    create do
      add "mobile.html", priority: 0.5, changefreq: "hourly", mobile: true
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/mobile.html</loc>
 <lastmod>2016-06-01T14:24:44Z</lastmod>
 <changefreq>hourly</changefreq>
 <priority>0.5</priority>
 <mobile:mobile/>
</url>

Look at Mobile Sitemaps as required.

PageMap sitemap

defmodule Sitemaps do
  use Sitemap, compress: true, host: "http://example.com"

  def generate do
    create do
      add "pagemap.html", pagemap: [
        dataobjects: [[
          type: "document",
          id: "hibachi",
          attributes: [
            [name: "name",   value: "Dragon"],
            [name: "review", value: "3.5"],
          ]
        ]]
      ]
    end
  end
end
Generated Result
<url>
 <loc>http://www.example.com/pagemap.html</loc>
 <lastmod>2016-06-02T17:01:17Z</lastmod>
 <PageMap>
   <DataObject id="hibachi" type="document">
     <Attribute name="name">Dragon</Attribute>
     <Attribute name="review">3.5</Attribute>
   </DataObject>
 </PageMap>
</url>

Look at PageMap sitemap as required.

Additional links into the Sitemap Index

create do
  add_to_index "/mysitemap1.xml.gz"
  add_to_index "/alternatemap.xml"
  add_to_index "/changehost.xml.gz", host: "http://something.com"

  add ...
  add ....
end
<?xml version='1.0' encoding='utf-8'?>
<sitemapindex xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
 <sitemap>
   <loc>http://example.com/mysitemap1.xml.gz</loc>
   <lastmod>2017-05-19T11:42:45+09:00</lastmod>
 </sitemap>
 <sitemap>
   <loc>http://example.com/mysitemap2.xml.gz</loc>
   <lastmod>2017-05-19T11:42:45+09:00</lastmod>
 </sitemap>
 <sitemap>
   <loc>http://something.com/changehost.xml.gz</loc>
   <lastmod>2017-05-19T11:42:45+09:00</lastmod>
 </sitemap>
 <sitemap>
   <loc>http://example.com/sitemap1.xml.gz</loc>
   <lastmod>2017-05-19T11:42:45+09:00</lastmod>
 </sitemap>
</sitemapindex>

Known issue

Inspired by

sitemap's People

Contributors

hauntedhost avatar ikeikeikeike avatar josevalim avatar nathanl avatar samhamilton avatar schultzer avatar sheharyarn avatar x-ji avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

sitemap's Issues

Incompatible with Ecto 3

Hit an error when I upgraded a project with sitemap to Ecto 3:

==> sitemap
Compiling 19 files (.ex)

== Compilation error in file lib/sitemap/funcs.ex ==
** (CompileError)  deadlocked waiting on struct Ecto.DateTime


== Compilation error in file lib/sitemap/config.ex ==
** (CompileError)  deadlocked waiting on module Sitemap.Funcs


Compilation failed because of a deadlock between files.
The following files depended on the following modules:

   lib/sitemap/funcs.ex => Ecto.DateTime
  29   end
  lib/sitemap/config.ex => Sitemap.Funcs

Ensure there are no compile-time dependencies between those files and that the modules they reference exist and are correctly named

could not compile dependency :sitemap, "mix compile" failed. You can recompile this dependency with "mix deps.compile sitemap", update it with "mix deps.update sitemap" or clean it with "mix deps.clean sitemap"

Cannot add multiple images to a Sitemap URL

Thanks for awesome package, but i think I've found an issue with the Image Sitemaps.

The Google spec for Image Sitemaps allows adding multiple images to a url in a sitemap, but this project doesn't seem to support that.

I've tried passing a List of Keyword Lists but it doesn't put any of the images in the generated sitemap:

def generate do
  create do
    add "index.html", images: [
      [
        loc:     "http://example.com/image-1.jpg",
        title:   "Title",
        caption: "Image 1",
      ],
      [
        loc:     "http://example.com/image-2.jpg",
        title:   "Title",
        caption: "Image 2",
      ]
    ]
  end
end

I've also tried passing the images Keyword tuple multiple times, but it only puts the first image in the sitemap:

def generate do
  create do
    add "index.html",
      images: [
        loc:     "http://example.com/image-1.jpg",
        title:   "Title",
        caption: "Image 1",
      ],
      images: [
        loc:     "http://example.com/image-2.jpg",
        title:   "Title",
        caption: "Image 2",
      ]
  end
end

How to run this on a production app

I tried to run this in production, but I get

{"Kernel pid terminated",application_controller,"{application_start_failure,scotchy,{{shutdown,{failed_to_start_child,'Elixir.Scotchy.Endpoint',{shutdown,{failed_to_start_child,'Elixir.Phoenix.Endpoint.Server',{shutdown,{failed_to_start_child,{ranch_listener_sup,'Elixir.Scotchy.Endpoint.HTTP'},{shutdown,{failed_to_start_child,ranch_acceptors_sup,{listen_error,'Elixir.Scotchy.Endpoint.HTTP',eaddrinuse}}}}}}}}},{'Elixir.Scotchy',start,[normal,[]]}}}"}

Crash dump is being written to: erl_crash.dump...done
Kernel pid terminated (application_controller) ({application_start_failure,scotchy,{{shutdown,{failed_to_start_child,'Elixir.Scotchy.Endpoint',{shutdown,{failed_to_start_child,'Elixir.Phoenix.Endpoint

Option create_index is always true

I tried an option 'create_index: false' or 'create_index: auto' on mix config, use statement and create but it always generates index files.

Could you check and update it?

How to set custom lastmod value?

I see the sitemap automatically sets the lastmod value to the current time, but for my ecto resources I already know the last time they were modified and want to set that as the lastmod value.

I've tried setting lastmod directly, but it does not work. Here's an example of what I'm trying to do:

def generate do
  create do
    Resource
    |> Repo.all
    |> Enum.each(fn res ->
      add Helper.resource_path(res), lastmod: Helper.format_time(res.updated_at)
    end)
  end
end

Error running ping() in a production release

Running in development mode:

Successful ping of http://www.bing.com.br/webmaster/ping.aspx?sitemap=https://www.myapp.com/sitemaps/sitemaps.xml.gz
Successful ping of http://google.com/ping?sitemap=https://www.myapp.com.br/sitemaps/sitemaps.xml.gz

but running a release in foreground (_build/prod/rel/my_app/bin/my_app foreground) :

Error in process #PID<6602.8547.20> on node :"node@ip_address" with exit value:
{:undef,
 [{:httpc, :request,
   ['http://google.com/ping?sitemap=https://www.myapp.com.br/sitemaps/sitemaps.xml.gz'],
   []},
  {Sitemap.Generator, :"-ping/1-fun-0-", 2,
   [file: 'lib/sitemap/generator.ex', line: 45]}]}

 [error] Error in process #PID<6602.8548.20> on node :"node@ip_address" with exit value:
{:undef,
 [{:httpc, :request,
   ['http://www.bing.com/webmaster/ping.aspx?sitemap=https://www.myapp.com.br/sitemaps/sitemaps.xml.gz'],
   []},
  {Sitemap.Generator, :"-ping/1-fun-0-", 2,
   [file: 'lib/sitemap/generator.ex', line: 45]}]}

Environment:

Elixir: 1.4
distillery: 1.0
sitemap: 0.8.0

mix.exs:

def application do
    [mod: {MyApp, []},
     applications: [..., :sitemap]]
end
defp deps do
    [...,
     {:sitemap, "~> 0.8.0"}]
end

Compilation error with ** (EXIT) no process

I'm using Phoenix, getting this error during compilation:

== Compilation error on file lib/sitemaps.ex ==
** (exit) exited in: GenServer.call(Sitemap.Config, {:update, #Function<2.23984558/1 in Sitemap.Config.set/2>}, 5000)
    ** (EXIT) no process
    (elixir) lib/gen_server.ex:596: GenServer.call/3
    (elixir) lib/enum.ex:651: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:651: Enum.each/2
    lib/sitemaps.ex:10: (module)
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
    (elixir) lib/kernel/parallel_compiler.ex:117: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1

I have verified that :sitemap is already in my :applications list.

Any clues what's missing here? Thanks!

Confusion over the :public_path parameter

In the example in the README, the sitemap is exposed under the folder /sitemaps. This also seems to be the default.

However, according to the information here, the sitemap should be under the root of the website. Otherwise,

A Sitemap file located at
http://example.com/catalog/sitemap.xml can include any URLs starting
with http://example.com/catalog/ but can not include URLs starting
with http://example.com/images/.

I wonder if the intention is to let the user manually specify another sitemap.xml under the website root to point to those files in /sitemaps, otherwise this configuration could be a bit misleading?

Customize Sitemap entries grouping

Hello. First of all, thank you for your work on this library. It makes my life way easier :)

I was wondering if there's any way to customize how sitemap entries are grouped together ?
Is there way to specify that I want 1 sitemap for static pages, 1 for video pages, etc?

I'd love to have more control about how exactly my Sitemap index is generated, and how my entries are separated into different files.

Feature request: Add DateTime support for lastmod

It would be great if the add function supported DateTime or Ecto.DateTime for the lastmod parameter.

add Helpers.item_path(Endpoint, :show, item.slug), priority: 0.5, changefreq: "daily", expires: nil, lastmod: item.updated_at

Concurrent sitemap generation

Hey, i am writing an app that should generate sitemaps for multiple websites.

Currently it doesnt work correctly concurrently because of the shared states, stored in Agents. Looking through Sitemap.State code, i found that there are functions for starting named agents. But all of the agents start as single instances.

Am i missing something, is there a way to store an isolated state for every :generate call? Or is this kind of behaviour not supported?

Thanks for your time in advance!

Any performance improvement?

I have been using this library for more than a year and thank you for maintaining. :)

As my sites are growing, there are a few million entries, so it takes almost 12 hours to generate sitemaps.

I have been using Repo.stream with an option(max_rows: 500) not to exceed max memory limit and Repo.transaction with options(timeout: :infinity, pool_timeout: :infinity) not to be closed before finishing generation.

Do you have any ideas to make it faster or plans to improve the performance?

Thanks for your answer in advance.

Return generated sitemap?

When I follow the docs and make a generate method, it calls write on my adapter from within the create call. But create returns only :ok.

My sitemap is pretty small and fast to generate, and I'm hosting on Heroku, where writing to a file is not an option. The simplest option for me would be to just generate the sitemap synchronously, on request, from a controller. But I don't see a way to do that currently. I may have my adapter write the sitemap to a cache that can be read in-request, but it's not the simplest way to start.

Is there some way to generate and return the sitemap in-memory?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.