Code Monkey home page Code Monkey logo

hypopg's Introduction

HypoPG

HypoPG is a PostgreSQL extension adding support for hypothetical indexes.

An hypothetical -- or virtual -- index is an index that doesn't really exists, and thus doesn't cost CPU, disk or any resource to create. They're useful to know if specific indexes can increase performance for problematic queries, since you can know if PostgreSQL will use these indexes or not without having to spend resources to create them.

For more thorough informations, please consult the official documentation.

For other general information, you can also consult this blog post.

Installation

  • Compatible with PostgreSQL 9.2 and above
  • Needs PostgreSQL header files
  • Decompress the tarball
  • sudo make install
  • In every needed database: CREATE EXTENSION hypopg;

Updating the extension

Note that hypopg doesn't provide extension upgrade scripts, as there's no data saved in any of the objects created. Therefore, you need to first drop the extension then create it again to get the new version.

Usage

NOTE: The hypothetical indexes are contained in a single backend. Therefore, if you add multiple hypothetical indexes, concurrent connections doing EXPLAIN won't be bothered by your hypothetical indexes.

Assuming a simple test case:

rjuju=# CREATE TABLE hypo AS SELECT id, 'line ' || id AS val FROM generate_series(1,10000) id;
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                      QUERY PLAN
-------------------------------------------------------
 Seq Scan on hypo  (cost=0.00..180.00 rows=1 width=13)
   Filter: (id = 1)
(2 rows)

The easiest way to create an hypothetical index is to use the hypopg_create_index functions with a regular CREATE INDEX statement as arg.

For instance:

rjuju=# SELECT * FROM hypopg_create_index('CREATE INDEX ON hypo (id)');

NOTE: Some information from the CREATE INDEX statement will be ignored, such as the index name if provided. Some of the ignored information will be handled in a future release.

You can check the available hypothetical indexes in your own backend:

rjuju=# SELECT * FROM hypopg_list_indexes ;
 indexrelid |          index_name           | schema_name | table_name | am_name 
------------+-------------------------------+-------------+------------+---------
      50573 | <50573>btree_hypo_id          | public      | hypo       | btree

If you need more technical information on the hypothetical indexes, the hypopg() function will return the hypothetical indexes in a similar way as pg_index system catalog.

And now, let's see if your previous EXPLAIN statement would use such an index:

rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                                     QUERY PLAN
------------------------------------------------------------------------------------
 Index Scan using <41072>hypo_btree_hypo_id on hypo  (cost=0.29..8.30 rows=1 width=13)
   Index Cond: (id = 1)
(2 rows)

Of course, only EXPLAIN without ANALYZE will use hypothetical indexes:

rjuju=# EXPLAIN ANALYZE SELECT * FROM hypo WHERE id = 1;
                                           QUERY PLAN
-------------------------------------------------------------------------------------------------
 Seq Scan on hypo  (cost=0.00..180.00 rows=1 width=13) (actual time=0.036..6.072 rows=1 loops=1)
   Filter: (id = 1)
   Rows Removed by Filter: 9999
 Planning time: 0.109 ms
 Execution time: 6.113 ms
(5 rows)

To remove your backend's hypothetical indexes, you can use the function hypopg_drop_index(indexrelid) with the OID that the hypopg_list_indexes view returns and call hypopg_reset() to remove all at once, or just close your current connection.

Continuing with the above case, you can hide existing indexes, but should be use hypopg_reset() to clear the previous effects of other indexes at first.

Create two real indexes and run EXPLAIN:

rjuju=# SELECT hypopg_reset();
rjuju=# CREATE INDEX ON hypo(id);
rjuju=# CREATE INDEX ON hypo(id, val);
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                                    QUERY PLAN
----------------------------------------------------------------------------------
Index Only Scan using hypo_id_val_idx on hypo  (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)

The query plan is using the hypo_id_val_idx index. Use hypopg_hide_index(oid) to hide one of the indexes:

rjuju=# SELECT hypopg_hide_index('hypo_id_val_idx'::REGCLASS);
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                            QUERY PLAN
-------------------------------------------------------------------------
Index Scan using hypo_id_idx on hypo  (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)

The query plan is using the other index hypo_id_idx now. Use hypopg_hide_index(oid) to hide it:

rjuju=# SELECT hypopg_hide_index('hypo_id_idx'::REGCLASS);
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                    QUERY PLAN
-------------------------------------------------------
Seq Scan on hypo  (cost=0.00..180.00 rows=1 width=13)
Filter: (id = 1)
(2 rows)

And now the query plan changes back to Seq Scan. Use hypopg_unhide_index(oid) to restore index:

rjuju=# SELECT hypopg_unhide_index('hypo_id_idx'::regclass);
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                            QUERY PLAN
-------------------------------------------------------------------------
Index Scan using hypo_id_idx on hypo  (cost=0.29..8.30 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)

Of course, you can also hide hypothetical indexes:

rjuju=# SELECT hypopg_create_index('CREATE INDEX ON hypo(id)');
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                                    QUERY PLAN
------------------------------------------------------------------------------------
Index Scan using "<12659>btree_hypo_id" on hypo  (cost=0.04..8.05 rows=1 width=13)
Index Cond: (id = 1)
(2 rows)

rjuju=# SELECT hypopg_hide_index(12659);
rjuju=# EXPLAIN SELECT * FROM hypo WHERE id = 1;
                    QUERY PLAN
-------------------------------------------------------
Seq Scan on hypo  (cost=0.00..180.00 rows=1 width=13)
Filter: (id = 1)
(2 rows)

You can check which indexes are hidden using hypopg_hidden_indexes() or the hypopg_hidden_indexes view:

rjuju=# SELECT * FROM hypopg_hidden_indexes();
indexid
---------
526604
526603
12659
(3 rows)

rjuju=# SELECT * FROM hypopg_hidden_indexes;
 indexrelid |      index_name      | schema_name | table_name | am_name | is_hypo
------------+----------------------+-------------+------------+---------+---------
      12659 | <12659>btree_hypo_id | public      | hypo       | btree   | t
     526603 | hypo_id_idx          | public      | hypo       | btree   | f
     526604 | hypo_id_val_idx      | public      | hypo       | btree   | f
(3 rows)

To restore all existing indexes, you can use the function hypopg_unhide_all_indexes(). Note that the functionality to hide existing indexes only applies to the EXPLAIN command in the current session and will not affect other sessions.

hypopg's People

Contributors

chrisma avatar df7cb avatar godwottery avatar joelvh avatar kmosolov avatar nutvi avatar pepl avatar rdunklau avatar rekgrpth avatar rjuju avatar szuliq avatar weiwch avatar xzyaoi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hypopg's Issues

A weird index name issue

I'm using the hypopg extension 1.1.3 for a while (installed by source code in ubuntu 20.04). A weird thing happened today, the index created by hypopg extension missed a ">" char in its name (e.g., now the name became <1000000053btree_lineitem_l_orderkey) from <1000000053>btree_lineitem_l_orderkey) ).

it behaves normally before, and I do not recall anything that I do to trigger this abnormal behavior. Have anyone encountered similar things, or what should I do to inspect the issues (reinstallation of the hypopg extension seems not working).

Thanks!

After creating the index, use the explain command to report an error.

-- create test table
create table test_st2(id int);
insert into test_st2 select generate_series(1,1000000);
-- ceate index
select monitor.hypopg_create_index('CREATE INDEX idx_test_st2_id ON public.test_st2 USING btree (id)');
(13741,<13741>btree_test_st2_id)
-- execute explain
explain (FORMAT JSON) select * from test_st2 where id=9280;
ERROR: cache lookup failed for index 13740

Compilation Error Encountered: ERROR: could not access file "$libdir/hypopg": No such file or directory

Hi! When I attempt to create the extension in the psql cmd, The following error appears.

postgres=# create extension hypopg;
ERROR:  could not access file "$libdir/hypopg": No such file or directory

I have checked the file in the $libdir library, it does exist the file hypopg.so.

And the details about when running the make install command is below.

/usr/bin/mkdir -p '/usr/lib64/pgsql'
/usr/bin/mkdir -p '/usr/share/pgsql/extension'
/usr/bin/mkdir -p '/usr/share/pgsql/extension'
/bin/sh /usr/lib64/pgsql/pgxs/src/makefiles/../../config/install-sh -c -m 755  hypopg.so '/usr/lib64/pgsql/hypopg.so'
/bin/sh /usr/lib64/pgsql/pgxs/src/makefiles/../../config/install-sh -c -m 644 ./hypopg.control '/usr/share/pgsql/extension/'
/bin/sh /usr/lib64/pgsql/pgxs/src/makefiles/../../config/install-sh -c -m 644 ./hypopg--2.0.0beta.sql ./hypopg--1.1.2.sql  '/usr/share/pgsql/extension/'

getting the error `undefined symbol: get_am_name`

Thank you very much for hypopg.
I'm trying to run it in the official docker container for Postgres 9.4 but it seems that something is wrong:

root@host:/tmp/hypopg-1.1.0# make install
/bin/mkdir -p '/usr/lib/postgresql/9.4/lib'
/bin/mkdir -p '/usr/share/postgresql/9.4/extension'
/bin/mkdir -p '/usr/share/postgresql/9.4/extension'
/usr/bin/install -c -m 755  hypopg.so '/usr/lib/postgresql/9.4/lib/hypopg.so'
/usr/bin/install -c -m 644 hypopg.control '/usr/share/postgresql/9.4/extension/'
/usr/bin/install -c -m 644 hypopg--1.1.0.sql '/usr/share/postgresql/9.4/extension/'
root@host:/tmp/hypopg-1.1.0# psql postgresql://postgres@localhost:5432/database
psql (9.4.15)
Type "help" for help.
database=# CREATE EXTENSION hypopg;
ERROR:  could not load library "/usr/lib/postgresql/9.4/lib/hypopg.so": /usr/lib/postgresql/9.4/lib/hypopg.so: undefined symbol: get_am_name

Do you have any idea what it could be?

hypo index is not working on partition table

I have created a hypo index on the partition table column, but when I run, explain query it doing seq scan instead hypo index scan.

Is it a bug or expected behavior?

select version(); --> PostgreSQL 10.11, compiled by Visual C++ build 1800, 64-bit

Thank you!

I am sorry for writing this, but this is not an issue.

I just want to say thank you very much for this awesome extension. It is easy to install and easy to use. I wish it supports more index type e.g: gin, gist. But, this works very well and I love it.

Thank you, again.

Compile warning

hypopg.c: In function ‘hypopg_get_indexdef’:
hypopg.c:1601:21: warning: ‘entry’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  if (!entry || entry->oid != indexid)
                     ^

Can this be addressed?

Hypopg is not working properly

Hi Team, first of all thanks a lot to open this channel for outstanding issues.

We have input all data into the POWA repository but unfortunately Hypopg is not suggesting anything on queries. When looking into the postgresql Logs, sounds like it is not parsing the Constant values and we got an sintaxe error message.

We are under 10.1 version and the last POWA + Archivist + Hypopg version. We followed the install guide point to point with no issues.

Would you mind guys helping us on understanding this?

Thanks a lot.
Rafael.

Cannot create extension in psql (Ubuntu)

I use Ubuntu 20.04 (wsl2). I tried sudo make install and it ran perfectly. However, it fails with this error when I create extension:

ERROR: could not load library "/usr/lib/postgresql/13/lib/hypopg.so": /usr/lib/postgresql/13/lib/hypopg.so: undefined symbol: errstart_cold

Other index types support

Hi! Are there any plans for BRIN indices or others? How difficult that would be to implement?
Thanks you for your anwser. Great work by the way.

Add functional dependency between columns from extended statistics

This is a feature suggestion along the lines of functions like hypopg_create_statistics.

Optimization assumes that columns are independent, so a query plan can be enhanced not only by an additional index, but to let optimizer know about functional dependencies (FD) between columns. Real world datasets often have FD between columns because of denormalization or just because the data is like this.

PostgreSQL offers CREATE STATISTICS on FD.

Expected behaviour: Adding hypopg_create_statistics in order to simulate existence of FD.

Example: Looking e.g. at this test database from ATP tennis tour there's at least one FD, namely between tourney_id and tourney_name. So this is what's ususally done:

CREATE STATISTCS atp_matches_2019_tourney_id_tourney_name (DEPENDENCIES) 
	ON tourney_id, tourney_name 
	FROM atp_matches_2019;

ANALYZE atp_matches_2019; -- update extended statistics

SELECT * FROM pg_statistic_ext; -- now shows the entry of "atp_matches_2019_tourney_id_tourney_name"

BTW: CREATE STATISTICS currently also supports ndistinct, which enables n-distinct statistics, and mcv which enables most-common values lists.

ALTER EXTENSION fails

There are no such scripts as ''hypopg--1.1.3-1.1.4.sql'', so an ALTER UPDATE fails:

powa=# ALTER EXTENSION hypopg UPDATE;
ERROR: extension "hypopg" has no update path from version "1.1.3" to version "1.1.4"

The extension can be dropped and reinstalled, but this is disturbing for a user.

Are dummy scripts an option in the future?

Error when creating extension

Hi,

I am getting below error when creating the extension:

-bash-4.2$ psql
psql (11.11)
Type "help" for help.

postgres=# create extension hypopg;
ERROR: could not load library "/usr/pgsql-11/lib/hypopg.so": /usr/pgsql-11/lib/hypopg.so: undefined symbol: pg_leftmost_one_pos32

When I did "sudo make install", I had got one warning related to pg_leftmost_one_pos32

hypopg_index.c:1954:2: warning: implicit declaration of function ‘pg_leftmost_one_pos32’ [-Wimplicit-function-declaration]
num_bitmap = Max(1, num_overflow /
^
hypopg_index.c:1955:7: warning: implicit declaration of function 'pg_leftmost_one_pos32' is invalid in C99 [-Wimplicit-function-declaration]
pg_leftmost_one_pos32(HypoHashGetMaxBitmapSize()));

Install on OSX : could not open extension control file 'hypopg.control'

Hello

I tried installing the extension on osx :

 🚴  hypopg-1.0.0/ » sudo make install
Password:
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument  -I. -I. -I/usr/local/Cellar/postgresql-9.3/9.3.16/include/server -I/usr/local/Cellar/postgresql-9.3/9.3.16/include/internal -I/usr/local/opt/gettext/include -I/usr/local/opt/openldap/include -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/opt/tcl-tk/include  -c -o hypopg.o hypopg.c
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument  -I. -I. -I/usr/local/Cellar/postgresql-9.3/9.3.16/include/server -I/usr/local/Cellar/postgresql-9.3/9.3.16/include/internal -I/usr/local/opt/gettext/include -I/usr/local/opt/openldap/include -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/opt/tcl-tk/include  -c -o hypopg_import.o hypopg_import.c
clang -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument  -bundle -multiply_defined suppress -o hypopg.so hypopg.o hypopg_import.o -L/usr/local/Cellar/postgresql-9.3/9.3.16/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/openldap/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/tcl-tk/lib -Wl,-dead_strip_dylibs   -bundle_loader /usr/local/Cellar/postgresql-9.3/9.3.16/bin/postgres
/bin/sh /usr/local/Cellar/postgresql-9.3/9.3.16/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql-9.3/9.3.16/lib'
/bin/sh /usr/local/Cellar/postgresql-9.3/9.3.16/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql-9.3/9.3.16/share/extension'
/bin/sh /usr/local/Cellar/postgresql-9.3/9.3.16/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql-9.3/9.3.16/share/extension'
/bin/sh /usr/local/Cellar/postgresql-9.3/9.3.16/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql-9.3/9.3.16/share/doc/extension'
/usr/bin/install -c -m 755  hypopg.so '/usr/local/Cellar/postgresql-9.3/9.3.16/lib/hypopg.so'
/usr/bin/install -c -m 644 ./hypopg.control '/usr/local/Cellar/postgresql-9.3/9.3.16/share/extension/'
/usr/bin/install -c -m 644 ./hypopg--1.0.0.sql  '/usr/local/Cellar/postgresql-9.3/9.3.16/share/extension/'
/usr/bin/install -c -m 644 ./README.md '/usr/local/Cellar/postgresql-9.3/9.3.16/share/doc/extension/'
 🚴  hypopg-1.0.0/ » psql
L'affichage de null est « [NULL] ».
L'affichage étendu est utilisé automatiquement.
Chronométrage activé.
psql (9.3.16, serveur 9.4.11)
ATTENTION : psql version majeure 9.3, version majeure du serveur 9.4.
         Certaines fonctionnalités de psql pourraient ne pas fonctionner.
Saisissez « help » pour l'aide.

bti=# CREATE EXTENSION hypopg;
ERROR:  58P01: could not open extension control file "/usr/local/Cellar/[email protected]/9.4.11/share/[email protected]/extension/hypopg.control": No such file or directory
EMPLACEMENT : parse_extension_control_file, extension.c : 474
Temps : 5,305 ms
bti=#

Did I miss something ? Should I have errors with missing PostgreSQL header files on the make install?

Index with sort order not being used

Hey, thanks for this awesome extension!

I think I may have found an issue with using a sort order on Postgres 9.6 (not sure about other versions). Here's code to repro:

CREATE EXTENSION IF NOT EXISTS hypopg;
SELECT hypopg_reset();

-- create posts
DROP TABLE IF EXISTS posts;
CREATE TABLE posts (
  id int,
  blog_id int,
  user_id int
);
INSERT INTO posts (SELECT n AS id, n % 1000 AS blog_id, n % 10 AS user_id FROM generate_series(1, 100000) n);

-- add hypothetical index and explain
-- i'd expect the hypo index to be used, but it's not
SELECT * FROM hypopg_create_index('CREATE INDEX ON posts (user_id DESC, blog_id)');
EXPLAIN SELECT * FROM posts ORDER BY user_id DESC, blog_id LIMIT 10;

-- add real index and explain
-- the real index is used here, as expected
CREATE INDEX ON posts (user_id DESC, blog_id);
EXPLAIN SELECT * FROM posts ORDER BY user_id DESC, blog_id LIMIT 10;

Let me know if anything is unclear.

Server crash

I get a server crash when performing the following steps on current 9.5devel:

# CREATE TABLE test (id serial, sampledate timestamp);
CREATE TABLE

# INSERT INTO test (sampledate) SELECT '1970-01-01
' 00:00:00'::timestamp + (random()*1350561945 || ' seconds')::interval
- FROM generate_series(1,21000000);
INSERT 0 21000000

# VACUUM;
VACUUM

# ANALYSE;
ANALYZE

# CREATE EXTENSION hypopg ;
CREATE EXTENSION

# SELECT * FROM hypopg_create_index('CREATE INDEX idx_test_sampledate_month ON test
' (extract(month FROM sampledate))');
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

Whether support the parallel use when estimated?

Hi, I am going to using this extension for parallel estimation i.e., create the separate hypothetical index and then get the estimated cost in each of my test case and in parallel. But I am not sure whether this extension could support this and whether these parallel test cases would interfere with each other to get the wrong estimation result. Now my method is to create multiple database connect to fulfill this request.

Index size estimations for small indexes seem inaccurate

Hi again,

I am running some experiments with a TPC-DS data set with a scale factor of 10 in PostgreSQL 12.5 (Ubuntu 12.5-0ubuntu0.20.04.1). For some small indexes, the estimated size does not change if I append attributes to a single attribute index, i.e., create a multi-attribute index. I am not sure if this is intended and a correct estimation or actually an issue.

SELECT * FROM hypopg_create_index('CREATE INDEX ON store (s_store_sk)');
SELECT * FROM hypopg_create_index('CREATE INDEX ON store (s_number_employees)');
SELECT * FROM hypopg_create_index('CREATE INDEX ON store (s_store_name)');

SELECT * FROM hypopg_create_index('CREATE INDEX ON store (s_store_sk, s_number_employees)');
SELECT * FROM hypopg_create_index('CREATE INDEX ON store (s_store_sk, s_number_employees, s_store_name)');

SELECT indexname, pg_size_pretty(hypopg_relation_size(indexrelid)) FROM hypopg_list_indexes();
                            indexname                            | pg_size_pretty
-----------------------------------------------------------------+----------------
 <84741773>btree_store_s_store_sk                                | 8192 bytes
 <84741774>btree_store_s_number_employees                        | 8192 bytes
 <84741777>btree_store_s_store_name                              | 8192 bytes
 <84741776>btree_store_s_store_sk_s_number_employees             | 8192 bytes
 <84741775>btree_store_s_store_sk_s_number_employees_s_store_nam | 8192 bytes
select attname, avg_width from pg_stats where tablename = 'store';
      attname       | avg_width
--------------------+-----------
 s_store_sk         |         4
 s_street_number    |         3
 s_street_name      |         9
 s_street_type      |        16
 s_suite_number     |        11
 s_city             |        10
 s_county           |        15
 s_state            |         3
 s_zip              |        11
 s_country          |        14
 s_gmt_offset       |         5
 s_tax_precentage   |         4
 s_hours            |        21
 s_store_id         |        17
 s_rec_start_date   |         4
 s_rec_end_date     |         4
 s_closed_date_sk   |         4
 s_store_name       |         4
 s_number_employees |         4
 s_floor_space      |         4
 s_division_name    |         8
 s_market_id        |         4
 s_geography_class  |         8
 s_company_id       |         4
 s_company_name     |         8
 s_market_desc      |        58
 s_market_manager   |        13
 s_manager          |        13
 s_division_id      |         4
(29 rows)

Thanks for your support!

build failure with PG12: hypopg.c:56:10: fatal error: optimizer/var.h: No such file or directory

Trying to build against 12beta1:

15:30:55 gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -I./ -I/usr/include/postgresql/12/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/mit-krb5  -c -o hypopg.o hypopg.c
15:30:55 hypopg.c:56:10: fatal error: optimizer/var.h: No such file or directory
15:30:55  #include "optimizer/var.h"
15:30:55           ^~~~~~~~~~~~~~~~~
15:30:55 compilation terminated.

[Question] Compilation error

I'm not able to successfully compile the extension.
Any Idea what might be going wrong ? Tips to make it work ?
Below I pasted pg_config and finally the returned warnings and errors from running make

# pg_config
BINDIR = /usr/lib/postgresql/13/bin
DOCDIR = /usr/share/doc/postgresql-doc-13
HTMLDIR = /usr/share/doc/postgresql-doc-13
INCLUDEDIR = /usr/include/postgresql
PKGINCLUDEDIR = /usr/include/postgresql
INCLUDEDIR-SERVER = /usr/include/postgresql/13/server
LIBDIR = /usr/lib/x86_64-linux-gnu
PKGLIBDIR = /usr/lib/postgresql/13/lib
LOCALEDIR = /usr/share/locale
MANDIR = /usr/share/postgresql/13/man
SHAREDIR = /usr/share/postgresql/13
SYSCONFDIR = /etc/postgresql-common
PGXS = /usr/lib/postgresql/13/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE =  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-icu' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python3' '--mandir=/usr/share/postgresql/13/man' '--docdir=/usr/share/doc/postgresql-doc-13' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/13' '--bindir=/usr/lib/postgresql/13/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--with-extra-version= (Debian 13.1-1.pgdg100+1)' '--enable-nls' '--enable-integer-datetimes' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-llvm' 'LLVM_CONFIG=/usr/bin/llvm-config-7' 'CLANG=/usr/bin/clang-7' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'TAR=/bin/tar' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now' '--with-gssapi' '--with-ldap' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security'
CC = gcc
CPPFLAGS = -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer
CFLAGS_SL = -fPIC
LDFLAGS = -Wl,-z,relro -Wl,-z,now -L/usr/lib/llvm-7/lib -Wl,--as-needed
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgcommon -lpgport -lpthread -lselinux -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lrt -ldl -lm 
VERSION = PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1)
# make
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -fPIC -I. -I./ -I/usr/include/postgresql/13/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2   -c -o hypopg.o hypopg.c
hypopg.c: In function ‘_PG_init’:
hypopg.c:131:22: warning: assignment to ‘ProcessUtility_hook_type’ {aka ‘void (*)(struct PlannedStmt *, const char *, enum <anonymous>,  struct ParamListInfoData *, struct QueryEnvironment *, struct _DestReceiver *, struct QueryCompletion *)’} from incompatible pointer type ‘void (*)(PlannedStmt *, const char *, ProcessUtilityContext,  struct ParamListInfoData *, QueryEnvironment *, DestReceiver *, char *)’ {aka ‘void (*)(struct PlannedStmt *, const char *, enum <anonymous>,  struct ParamListInfoData *, struct QueryEnvironment *, struct _DestReceiver *, char *)’} [-Wincompatible-pointer-types]
  ProcessUtility_hook = hypo_utility_hook;
                      ^
hypopg.c: In function ‘hypo_utility_hook’:
hypopg.c:295:15: warning: passing argument 7 of ‘prev_utility_hook’ from incompatible pointer type [-Wincompatible-pointer-types]
         dest, completionTag);
               ^~~~~~~~~~~~~
hypopg.c:295:15: note: expected ‘QueryCompletion *’ {aka ‘struct QueryCompletion *’} but argument is of type ‘char *’
hypopg.c:314:15: warning: passing argument 7 of ‘standard_ProcessUtility’ from incompatible pointer type [-Wincompatible-pointer-types]
         dest, completionTag);
               ^~~~~~~~~~~~~
In file included from include/hypopg_index.h:22,
                 from hypopg.c:39:
/usr/include/postgresql/13/server/tcop/utility.h:85:47: note: expected ‘QueryCompletion *’ {aka ‘struct QueryCompletion *’} but argument is of type ‘char *’
          DestReceiver *dest, QueryCompletion *qc);
                              ~~~~~~~~~~~~~~~~~^~
hypopg.c: In function ‘hypo_get_relation_stats_hook’:
hypopg.c:680:6: warning: implicit declaration of function ‘pg_class_aclcheck’; did you mean ‘pclose_check’? [-Wimplicit-function-declaration]
     (pg_class_aclcheck(rte->relid, GetUserId(),
      ^~~~~~~~~~~~~~~~~
      pclose_check
hypopg.c:681:27: error: ‘ACLCHECK_OK’ undeclared (first use in this function)
            ACL_SELECT) == ACLCHECK_OK) ||
                           ^~~~~~~~~~~
hypopg.c:681:27: note: each undeclared identifier is reported only once for each function it appears in
hypopg.c:682:6: warning: implicit declaration of function ‘pg_attribute_aclchec’; did you mean ‘pg_attribute_packed’? [-Wimplicit-function-declaration]
     (pg_attribute_aclcheck(rte->relid, attnum, GetUserId(),
      ^~~~~~~~~~~~~~~~~~~~~
      pg_attribute_packed
make: *** [<builtin>: hypopg.o] Error 1

undefined symbol: errstart_cold

Hi, I have the same question, did you solve it?
I install the pg_bigm to create index.
It seems that someone install the PG14, but i use
' dpkg -l | grep postgres
sudo apt --purge remove [version-name] ' to remove it .
when i 'sudo pg_config', it seems PG13,but when i create extension.it says 'undefined symbol: errstart_cold'

estimate entry->pages and tuples question

Recently, I want to write a plug about virtual index on pg. And I'm really happy to see this project. I have some questions about the funciton hypo_estimate_index. when you estimate(btree) the entry->tuples=rel->tuples,but there also are some tuples in upper pages you don't consider in. The same as pages.

hypopg is incompatible with PostgreSQL 10

Unfortunately, hypopg is not compatible with the newest version of Postgres.

RhodiumToad from freenode's #postgresql writes:

<RhodiumToad> specifically,  entry->amhasgettuple = OidIsValid(amroutine->amgettuple);   <-- amroutine->amgettuple there is a function pointer, not an oid
<RhodiumToad> so it should be testing it against NULL, not using OidIsValid
<RhodiumToad> the other [issue] is a change in function signature for ProcessUtility

Here is a compilation log demonstrating the latter issue: http://ix.io/AEt. Compiled using gcc 7.2.0 with Postgres 10rc1.

Promote 2.0 release out of beta

Is there any known issue which prevents 2.0 release to come out of beta ?

REL1_STABLE branch had newer commits compared to master branch.
Does this mean REL1_STABLE branch would be the active branch going forward ?

Packages?

For those interested in safely installing and maintaining this in a production environment, will this ever be packaged?

Could not load library on postgresql 12

After pg_upgrade from v11 to v12 I'm getting this error when craeting the extension.
ERROR: could not load library "/usr/lib/postgresql/12/lib/hypopg.so": /usr/lib/postgresql/12/lib/hypopg.so: undefined symbol: AllocSetContextCreateExtended

Index size estimation is sometimes very inaccurate

I have a TPC-H data set with a scale factor of 10 in Postgres 12 (12.1 (Ubuntu 12.1-1.pgdg19.10+1). If I create a multi-column index, it is in certain cases larger than a single-column index:

select * from hypopg_create_index('create index on lineitem (l_partkey,l_extendedprice,l_quantity)');
select * from hypopg_create_index('create index on lineitem (l_partkey)');

tpch___10=# SELECT indexname, pg_size_pretty(hypopg_relation_size(indexrelid)) FROM hypopg_list_indexes() ;
                           indexname                           | pg_size_pretty
---------------------------------------------------------------+----------------
 <15540732>btree_lineitem_l_partkey_l_extendedprice_l_quantity | 164 MB
 <15540733>btree_lineitem_l_partkey                            | 1495 MB
(2 rows)

The results are the same if I exchange l_partkey, with l_orderkey. If I physically create these indexes via CREATE INDEX, the multi-column index has 2.2 and the single-column 1.1 GB.

Some bugs

Hello! Thank you for your extension!
I found some problems with it:

In DOCS you created column "line" in example (CREATE TABLE hypo (id integer, line text) ;), but after you tried to select column "val" (EXPLAIN SELECT val FROM hypo WHERE id = 1;)

And second problem with functions:
\df
Schema | Name | Result data type |
--------+----------------------+------------------+-----------------------------------------------------------
public | hypopg | SETOF record | OUT indexname text, OUT indexrelid oid, OUT indrelid oid,
public | hypopg_create_index | SETOF record | sql_order text, OUT indexrelid oid, OUT indexname text
public | hypopg_drop_index | boolean | indexid oid
public | hypopg_get_indexdef | text | indexid oid
public | hypopg_list_indexes | SETOF record | OUT indexrelid oid, OUT indexname text, OUT nspname name,
public | hypopg_relation_size | bigint | indexid oid
public | hypopg_reset | void |
public | hypopg_reset_index | void |
(8 rows)

I got error when i try to do something like that:
SELECT hypopg_partition_table('hypo_part_range', 'PARTITION BY RANGE(id)');

select * from pg_available_extensions where name = 'hypopg' \gx
-[ RECORD 1 ]-----+------------------------------------
name | hypopg
default_version | 1.1.3
installed_version | 1.1.3
comment | Hypothetical indexes for PostgreSQL

PostgreSQL 11.7

Not working for Pg14

I recently ran brew upgrade which updated postgres to version 14.

Now when I run CREATE EXTENSION hypopg; I get:

ERROR:  incompatible library "/usr/local/lib/postgresql/hypopg.so": version mismatch
DETAIL:  Server is version 14, library is version 13.
SQL state: XX000

hypopg_list_indexes() does not work normally

I follow the steps listed in the readme. Everything works fine, including other examples using hypopg's function with a command inside(). However, I have trouble with this command:

tpch=# SELECT * FROM hypopg_list_indexes();
ERROR: function hypopg_list_indexes() does not exist
LINE 1: SELECT * FROM hypopg_list_indexes();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.

Use hypopg to simulate dropping existing indexes

Would it be possible to have hypopg simulate dropping existing indexes, to see how the query plans change if that is done? It is my understanding that this is not possible right now (or maybe at all?), but I did not find a prior issue discussing this while browsing through the list of closed issues.

inaccurate cost estimation from explain

Very promising project. I'm new here, when I tried out the extension, I found a problem described below
environment: debiain 10, postgres 10

  1. example table hypo like in the README
  2. created a real index on id, and a real index real_index(id)
    image
  3. now whether I drop or not drop the real index, the cost estimation results is like below
    image
    ------->>>>>> the problem is the cost evaluation value don't equal?
    I don't know whether it's bug or something else, plz check it out, thanks!

Test failure with PG11

hypopg compiles against PG11beta1, but doesn't pass the regression tests:

**** regression.diffs ****
*** /tmp/autopkgtest.O5qY5O/tree/expected/hypopg.out	2018-03-28 00:28:29.000000000 +0200
--- /tmp/autopkgtest.O5qY5O/tree/results/hypopg.out	2018-05-26 21:57:21.007399085 +0200
***************
*** 37,43 ****
  WHERE e ~ 'Index.*<\d+>btree_hypo.*';
   count 
  -------
!      1
  (1 row)
  
  -- Should use hypothetical index
--- 37,43 ----
  WHERE e ~ 'Index.*<\d+>btree_hypo.*';
   count 
  -------
!      0
  (1 row)
  
  -- Should use hypothetical index
***************
*** 45,51 ****
  WHERE e ~ 'Index.*<\d+>btree_hypo.*';
   count 
  -------
!      1
  (1 row)
  
  -- Should not use hypothetical index
--- 45,51 ----
  WHERE e ~ 'Index.*<\d+>btree_hypo.*';
   count 
  -------
!      0
  (1 row)
  
  -- Should not use hypothetical index
***************
*** 152,158 ****
  WHERE e ~ 'Index.*<\d+>btree_hypo.*';
   count 
  -------
!      1
  (1 row)
  
  -- Deparse an index DDL, with almost every possible pathcode
--- 152,158 ----
  WHERE e ~ 'Index.*<\d+>btree_hypo.*';
   count 
  -------
!      0
  (1 row)
  
  -- Deparse an index DDL, with almost every possible pathcode

======================================================================

Postgres 12

Hi, thanks for this awesome project 👍

When running make on the master branch against Postgres 12 (on Debian 10), it's erroring with:

hypopg_index.c:47:10: fatal error: nodes/relation.h: No such file or directory
 #include "nodes/relation.h"
          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [<builtin>: hypopg_index.o] Error 1

Support for materialized views

Thanks again for this great project 👍

It looks like hypothetical indexes don't work with materialized views. Here's code to repro:

CREATE EXTENSION IF NOT EXISTS hypopg;
SELECT hypopg_reset();

-- create posts
DROP TABLE IF EXISTS posts CASCADE;
CREATE TABLE posts (
  id int,
  blog_id int,
  user_id int
);
INSERT INTO posts (SELECT n AS id, n % 1000 AS blog_id, n % 10 AS user_id FROM generate_series(1, 100000) n);

-- create materialized view
CREATE MATERIALIZED VIEW posts_materialized AS SELECT * FROM posts;

-- add hypothetical index and explain
-- i'd expect the hypo index to be used, but it's not
SELECT * FROM hypopg_create_index('CREATE INDEX ON posts_materialized (id)');
EXPLAIN SELECT * FROM posts_materialized WHERE id = 1;

-- add real index and explain
-- the real index is used here, as expected
CREATE INDEX ON posts_materialized (id);
EXPLAIN SELECT * FROM posts_materialized WHERE id = 1;

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.