Code Monkey home page Code Monkey logo

pguri's Introduction

uri type for PostgreSQL

https://twitter.com/pvh/status/567395527357001728

This is an extension for PostgreSQL that provides a uri data type. Advantages over using plain text for storing URIs include:

  • URI syntax checking
  • functions for extracting URI components
  • human-friendly sorting

The actual URI parsing is provided by the uriparser library, which supports URI syntax as per RFC 3986.

Note that this might not be the right data type to use if you want to store user-provided URI data, such as HTTP referrers, since they might contain arbitrary junk.

Installation

You need to have the above-mentioned uriparser library installed. It is included in many operating system distributions and package management systems. pkg-config will be used to find it. I recommend at least version 0.8.0. Older versions will also work, but they apparently contain some bugs and might fail to correctly accept or reject URI syntax corner cases. This is mainly a problem if your application needs to be robust against junk input.

To build and install this module:

make
make install

or selecting a specific PostgreSQL installation:

make PG_CONFIG=/some/where/bin/pg_config
make PG_CONFIG=/some/where/bin/pg_config install

And finally inside the database:

CREATE EXTENSION uri;

Using

This module provides a data type uri that you can use like a normal type. For example:

CREATE TABLE links (
    id int PRIMARY KEY,
    link uri
);

INSERT INTO links VALUES (1, 'https://github.com/petere/pguri');

A number of functions are provided to extract parts of a URI:

  • uri_scheme(uri) returns text

    Extracts the scheme of a URI, for example http or ftp or mailto.

  • uri_userinfo(uri) returns text

    Extracts the user info part of a URI. This is normally a user name, but could also be of the form username:password. If the URI does not contain a user info part, then this will return null.

  • uri_host(uri) returns text

    Extracts the host of a URI, for example www.example.com or 192.168.0.1. (For IPv6 addresses, the brackets are not included here.) If there is no host, the return value is null.

  • uri_host_inet(uri) returns inet

    If the host is a raw IP address, then this will return it as an inet datum. Otherwise (not an IP address or no host at all), the return value is null.

  • uri_port(uri) returns integer

    Extracts the port of a URI as an integer, for example 5432. If no port is specified, the return value is null.

  • uri_path(uri) returns text

    Extracts the path component of a URI. Logically, a URI always contains a path. The return value can be an empty string but never null.

  • uri_path_array(uri) returns text[]

    Returns the path component of a URI as an array, with the path split at the slash characters. This is probably not as useful as the uri_path function, but it is provided here because the uriparser library exposes it.

  • uri_query(uri) returns text

    Extracts the query part of a URI (roughly speaking, everything after the ?). If there is no query part, returns null.

  • uri_fragment(uri) returns text

    Extracts the fragment part of a URI (roughly speaking, everything after the #). If there is no fragment part, returns null.

Other functions:

  • uri_normalize(uri) returns uri

    Performs syntax-based normalization of the URI. This includes case normalization, percent-encoding normalization, and removing redundant . and .. path segments. See RFC 3986 section 6.2.2 for the full details.

    Note that this module (and similar modules in other programming languages) compares URIs for equality in their original form, without normalization. If you want to consider distinct URIs without regard for mostly irrelevant syntax differences, pass them through this function.

  • uri_escape(text, space_to_plus boolean DEFAULT false, normalize_breaks boolean DEFAULT false) returns text

    Percent-encodes all reserved characters from the text. This can be useful for constructing URIs from strings.

    If space_to_plus is true, then spaces are replaced by plus signs. If normalize_breaks is true, then line breaks are converted to CR LF pairs (and subsequently percent-encoded). Note that these two conversions come from the HTML standard for encoding form data but are not part of the specification for URIs.

  • uri_unescape(text, plus_to_space boolean DEFAULT false, break_conversion boolean DEFAULT false) returns text

    Decodes all percent-encodings in the text.

    If plus_to_space is true, then plus signs are converted to spaces. If break_conversion is true, then CR LF pairs are converted to simple newlines (\n).

pguri's People

Contributors

petere avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pguri's Issues

Missing Dependency in readme:

Hi,

when building on Debian 11 you also need to install postgresql-server-dev-13 otherwise the build will fail due to missing postgres.h:

make
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -fPIC  -I. -I./ -I/usr/include/postgresql/13/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2   -c -o uri.o uri.c
uri.c:1:10: fatal error: postgres.h: Datei oder Verzeichnis nicht gefunden
   1 | #include <postgres.h>
     |          ^~~~~~~~~~~~
compilation terminated.
make: *** [<eingebaut>: uri.o] Fehler 1

pguri extension

Hello,

I have a copy of Postgresql running in a Docker container and would like to be able to compile up anything that is needed for pguri on a development machine after which adding it to the running postgresql server in the container.

Can you please advise on how best to accomplish this?
Thanks,

Invalid input syntax when inserting urls with accents

Hello,

When inserting urls with accents, I run into the following error:

insert into urljson values ('http://www.example.com/évidemment', 200, 'text/html; charset=UTF-8', 33055);
ERROR:  invalid input syntax for type uri at or near "évidemment"
LINE 1: insert into urljson values ('http://www.example.com/évidemme...
                                                     ^

I know that urls should not contain accents ... But they actually do! Is it caused by some kind of encoding problems?

Thanks!

Setting parts of URI?

An interesting and useful thing to do with URIs is setting various parts of the URI, without changing other parts, say changing only the host ot the or query string. This functionality is provided by Perl's URI package, for example.

Also, converters from/to query string part to various key/value types like rowtypes and json can be quite handy. Now one has to assemble and take these apart somewhat manually.
Perhaps something akin to row_to_json, but for building URL query strings.

creating table with uri type to uuid_generate

Hello,

I am trying to create a PG table such that:

CREATE TABLE websites (
id serial primary key,
site_link uri,
site_sha1 uuid NOT NULL DEFAULT uuid_generate_v5(uuid_ns_url(), site_link),
site_md5 uuid NOT NULL DEFAULT uuid_generate_v3(uuid_ns_url(), site_link)
);

but I think that I need to be able to send the site_link as a text input to the "uuid_generate_5/3()"

How can I cast the URI as text?

Thanks,

Add data type for urls

It would be nice to have a data type url, which validates the scheme against http/https. It would be nicer if this extension had support for hostname validation against TLDs.

PostgreSQL 16 support

Hi @petere ,

Latest pguri release fails to build against v16. Some of the errors are probably GCC 13.2 errors as well. All are below. Can you please take a look?

Thanks!

Regards, Devrim

/usr/lib64/ccache/clang -Wno-ignored-attributes -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Xclang -no-opaque-pointers -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -O2 -I. -I./ -I/usr/pgsql-16/include/server -I/usr/pgsql-16/include/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -flto=thin -emit-llvm -c -o uri.bc uri.c uri.c:152:20: error: incompatible integer to pointer conversion passing 'Datum' (aka 'unsigned long') to parameter of type 'const inet *' [-Wint-conversion] PG_RETURN_INET_P(DirectFunctionCall1(inet_in, CStringGetDatum(tmp))); ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/pgsql-16/include/server/fmgr.h:643:2: note: expanded from macro 'DirectFunctionCall1' DirectFunctionCall1Coll(func, InvalidOid, arg1) ^ /usr/pgsql-16/include/server/utils/inet.h:135:50: note: expanded from macro 'PG_RETURN_INET_P' #define PG_RETURN_INET_P(x) return InetPGetDatum(x) ^ /usr/pgsql-16/include/server/utils/inet.h:129:27: note: passing argument to parameter 'X' here InetPGetDatum(const inet *X) ^ uri.c:164:20: error: incompatible integer to pointer conversion passing 'Datum' (aka 'unsigned long') to parameter of type 'const inet *' [-Wint-conversion] PG_RETURN_INET_P(DirectFunctionCall1(inet_in, CStringGetDatum(tmp))); ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/pgsql-16/include/server/fmgr.h:643:2: note: expanded from macro 'DirectFunctionCall1' DirectFunctionCall1Coll(func, InvalidOid, arg1) ^ /usr/pgsql-16/include/server/utils/inet.h:135:50: note: expanded from macro 'PG_RETURN_INET_P' #define PG_RETURN_INET_P(x) return InetPGetDatum(x) ^ /usr/pgsql-16/include/server/utils/inet.h:129:27: note: passing argument to parameter 'X' here InetPGetDatum(const inet *X) ^ uri.c:302:25: error: incompatible integer to pointer conversion passing 'Datum' (aka 'unsigned long') to parameter of type 'const void *' [-Wint-conversion] PG_RETURN_ARRAYTYPE_P(makeArrayResult(astate, CurrentMemoryContext)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/pgsql-16/include/server/utils/array.h:258:54: note: expanded from macro 'PG_RETURN_ARRAYTYPE_P' #define PG_RETURN_ARRAYTYPE_P(x) PG_RETURN_POINTER(x) ^ /usr/pgsql-16/include/server/fmgr.h:361:53: note: expanded from macro 'PG_RETURN_POINTER' #define PG_RETURN_POINTER(x) return PointerGetDatum(x) ^ /usr/pgsql-16/include/server/postgres.h:322:29: note: passing argument to parameter 'X' here PointerGetDatum(const void *X) ^ 3 errors generated. make[1]: *** [/usr/pgsql-16/lib/pgxs/src/makefiles/../../src/Makefile.global:1080: uri.bc] Error 1 make[1]: *** Waiting for unfinished jobs.... In file included from uri.c:8: uri.c: In function 'uri_host_inet': /usr/pgsql-16/include/server/fmgr.h:643:9: warning: passing argument 1 of 'InetPGetDatum' makes pointer from integer without a cast [-Wint-conversion] 643 | DirectFunctionCall1Coll(func, InvalidOid, arg1) | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | | | Datum {aka long unsigned int} /usr/pgsql-16/include/server/utils/inet.h:135:50: note: in definition of macro 'PG_RETURN_INET_P' 135 | #define PG_RETURN_INET_P(x) return InetPGetDatum(x) | ^ uri.c:152:34: note: in expansion of macro 'DirectFunctionCall1' 152 | PG_RETURN_INET_P(DirectFunctionCall1(inet_in, CStringGetDatum(tmp))); | ^~~~~~~~~~~~~~~~~~~ /usr/pgsql-16/include/server/utils/inet.h:129:27: note: expected 'const inet *' but argument is of type 'Datum' {aka 'long unsigned int'} 129 | InetPGetDatum(const inet *X) | ~~~~~~~~~~~~^ /usr/pgsql-16/include/server/fmgr.h:643:9: warning: passing argument 1 of 'InetPGetDatum' makes pointer from integer without a cast [-Wint-conversion] 643 | DirectFunctionCall1Coll(func, InvalidOid, arg1) | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | | | Datum {aka long unsigned int} /usr/pgsql-16/include/server/utils/inet.h:135:50: note: in definition of macro 'PG_RETURN_INET_P' 135 | #define PG_RETURN_INET_P(x) return InetPGetDatum(x) | ^ uri.c:164:34: note: in expansion of macro 'DirectFunctionCall1' 164 | PG_RETURN_INET_P(DirectFunctionCall1(inet_in, CStringGetDatum(tmp))); | ^~~~~~~~~~~~~~~~~~~ /usr/pgsql-16/include/server/utils/inet.h:129:27: note: expected 'const inet *' but argument is of type 'Datum' {aka 'long unsigned int'} 129 | InetPGetDatum(const inet *X) | ~~~~~~~~~~~~^ In file included from /usr/pgsql-16/include/server/access/skey.h:19, from /usr/pgsql-16/include/server/access/genam.h:18, from /usr/pgsql-16/include/server/access/amapi.h:15, from /usr/pgsql-16/include/server/access/hash.h:20, from uri.c:2: uri.c: In function 'uri_path_array': uri.c:302:39: warning: passing argument 1 of 'PointerGetDatum' makes pointer from integer without a cast [-Wint-conversion] 302 | PG_RETURN_ARRAYTYPE_P(makeArrayResult(astate, CurrentMemoryContext)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | | | Datum {aka long unsigned int} /usr/pgsql-16/include/server/fmgr.h:361:53: note: in definition of macro 'PG_RETURN_POINTER' 361 | #define PG_RETURN_POINTER(x) return PointerGetDatum(x) | ^ uri.c:302:17: note: in expansion of macro 'PG_RETURN_ARRAYTYPE_P' 302 | PG_RETURN_ARRAYTYPE_P(makeArrayResult(astate, CurrentMemoryContext)); | ^~~~~~~~~~~~~~~~~~~~~ In file included from uri.c:1: /usr/pgsql-16/include/server/postgres.h:322:29: note: expected 'const void *' but argument is of type 'Datum' {aka 'long unsigned int'} 322 | PointerGetDatum(const void *X) | ~~~~~~~~~~~~^ In file included from /usr/pgsql-16/include/server/postgres.h:46: uri.c: In function 'parse_uri': /usr/pgsql-16/include/server/utils/elog.h:142:12: warning: this statement may fall through [-Wimplicit-fallthrough=] 142 | do { \ | ^ /usr/pgsql-16/include/server/utils/elog.h:164:9: note: in expansion of macro 'ereport_domain' 164 | ereport_domain(elevel, TEXTDOMAIN, __VA_ARGS__) | ^~~~~~~~~~~~~~ uri.c:41:25: note: in expansion of macro 'ereport' 41 | ereport(ERROR, | ^~~~~~~ uri.c:45:17: note: here 45 | default: | ^~~~~~~

License

Hi! This is a great pg extension, would you mind adding a license file to the project? Thanks!

PS created an ebuild for gentoo here.

warning: implicit declaration of function uriIsHostSetA

uri.c: In function ‘uri_path’:
uri.c:238:2: warning: implicit declaration of function ‘uriIsHostSetA’ [-Wimplicit-function-declaration]
    if (uri.absolutePath || (uriIsHostSetA(&uri) && uri.pathHead))
                             ^

uriparser seems to only define uriIsHostSet, without the 'A', typo?

Add a META.json

And release on PGXN?

{
   "name": "uri",
   "abstract": "A URI type for PostgreSQL",
   "description": "This is an extension for PostgreSQL that provides a uri data type. The actual URI parsing is provided by the uriparser library, which supports URI syntax as per RFC 3986.",
   "version": "1.20150415.0",
   "maintainer": "Peter Eisentraut <[email protected]>",
   "license": "postgresql",
   "provides": {
      "uri": {
         "abstract": "A URI type for PostgreSQL",
         "file": "uri.sql",
         "docfile": "README.md",
         "version": "1.20150415.0"
      }
   },
   "prereqs": {
      "runtime": {
         "requires": {
            "PostgreSQL": "9.1.0"
         }
      }
   },
   "resources": {
      "bugtracker": {
         "web": "https://github.com/petere/pguri/issues/"
      },
      "repository": {
        "url":  "git://github.com/petere/pguri.git",
        "web":  "https://github.com/petere/pguri/",
        "type": "git"
      }
   },
   "generated_by": "David E. Wheeler",
   "meta-spec": {
      "version": "1.0.0",
      "url": "http://pgxn.org/meta/spec.txt"
   },
   "tags": [
      "uri",
      "url",
      "data type"
   ]
}

Foreign keys ERROR: could not find hash function for hash operator 929350

We are building a CMS and using the uri type in the database to archive pointers to each article to properly handle redirection. To do so, the article table has a current uri that must be present in the article_url table as well. When I try to apply the DDL file that alters the table to add the foreign key constraint, it pops the following error out:

ALTER TABLE ONLY article_i18n
    ADD CONSTRAINT article_id_supported_culture_uri_fk FOREIGN KEY (article_id, supported_culture_id, uri) REFERENCES article_url(article_id, supported_culture_id, article_url) MATCH FULL;
psql:sources/schema/postgres/post-data.sql:316: ERROR:  could not find hash function for hash operator 929350
CONTEXTE : SQL statement "SELECT fk."article_id", fk."supported_culture_id", fk."uri" FROM ONLY "article"."article_i18n" fk LEFT OUTER JOIN ONLY "article"."article_url" pk ON ( pk."article_id" OPERATOR(pg_catalog.=) fk."article_id" AND pk.
"supported_culture_id" OPERATOR(pg_catalog.=) fk."supported_culture_id" AND pk."article_url" OPERATOR(public.=) fk."uri") WHERE pk."article_id" IS NULL AND (fk."article_id" IS NOT NULL OR fk."supported_culture_id" IS NOT NULL OR fk."uri" I
S NOT NULL)"

When I test for the operator 929350, it says it is the = operator:

select 929350::regoperator;
┌─────────────┐
│ regoperator │
├─────────────┤
│ =(uri,uri)  │
└─────────────┘
(1 row)
PostgreSQL 9.4.3 on x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.9.2-10) 4.9.2, 64-bit

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.