ohmu / pgmemcache Goto Github PK

pgmemcache doen't seems to be able to work with a proxy server nutcracker "twitter twemproxy" !!!
any reason for that ??
os debian 8
latest version of pgmemcache and libmemcached

ERROR: pgmemcache: memcached_fetch: NOT FOUND
os : debian 8.3
postgresql-9.5-pgmemcache : 2.3.0-2.pgdg80+1
libmemcached11 : 1.0.18-4
memcache_get work fine

Hi,

My scenario is as below:
Running 2 memcached servers

select memcache_stats();

memcache_stats

                         +

Server: 192.168.1.33 (11211)+
pid: 2061 +
uptime: 4 +
time: 1426227684 +
version: 1.4.14 +
pointer_size: 64 +
rusage_user: 0.2712 +
rusage_system: 0.4045 +
curr_items: 0 +
total_items: 0 +
bytes: 0 +
curr_connections: 5 +
total_connections: 6 +
connection_structures: 6 +
cmd_get: 0 +
cmd_set: 0 +
get_hits: 0 +
get_misses: 0 +
evictions: 0 +
bytes_read: 8 +
bytes_written: 0 +
limit_maxbytes: 536870912 +
threads: 4 +
Server: 192.168.1.35 (11211)+
pid: 1385 +
uptime: 7 +
time: 1426227685 +
version: 1.4.14 +
pointer_size: 64 +
rusage_user: 0.1417 +
rusage_system: 0.1780 +
curr_items: 0 +
total_items: 0 +
bytes: 0 +
curr_connections: 5 +
total_connections: 6 +
connection_structures: 6 +
cmd_get: 0 +
cmd_set: 0 +
get_hits: 0 +
get_misses: 0 +
evictions: 0 +
bytes_read: 8 +
bytes_written: 0 +
limit_maxbytes: 536870912 +
threads: 4

select memcache_set(id,a||','||b||','||c||','||d) from mydb.test;

memcache_set

t
t
t
t
t

1 memcache server goes down

select memcache_set(id,a||','||b||','||c||','||d) from mydb.test;
WARNING: SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
WARNING: SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
WARNING: SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
WARNING: SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY

memcache_set

f
t
f
f
f

I keep getting this error, how do I delete the server which went down from pgmemcache configuration.
I have used pgmemcache.default_servers in conf file to add the servers at startup. I tried using pgmemcache.default_behavior = 'DEAD_TIMEOUT:2', but seems like this parameter is not supported as it gives an error during postgres startup.
The environment is Postgres 9.3 running on Ubuntu 14.04

Thanks!

I thought that GUC variables can be set at database or role level:

testdb=# ALTER DATABASE testdb SET pgmemcache.default_servers='localhost:11211';
ALTER DATABASE
testdb=# SELECT memcache_get('C22072631');
ERROR:  pgmemcache: memcached_get: NO SERVERS DEFINED

testdb=# ALTER ROLE teo SET pgmemcache.default_servers='localhost:11211';
ALTER ROLE
testdb=# SELECT memcache_get('C22072631');
ERROR:  pgmemcache: memcached_get: NO SERVERS DEFINED

From #11 (comment)

Hi,

the 2.1.1 build throws some strange git errors while doing a make and make install:

# make
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
cp -fp ext/pgmemcache.sql pgmemcache--.sql
fatal: Not a git repository (or any of the parent directories): .git
sed -e 's,__short_ver__,,g' < ext/pgmemcache.control > pgmemcache.control
gcc -g -O2 -fPIC -pie -I/usr/include/mit-krb5 -DLINUX_OOM_ADJ=0 -fPIC -pie -I/usr/include/mit-krb5 -DLINUX_OOM_ADJ=0 -Wall -Wmissing-prototypes                   -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -fpic                   -I. -I. -I/usr/include/postgresql/9.2/server -I/usr/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.5  -c                   -o pgmemcache.o pgmemcache.c
gcc -g -O2 -fPIC -pie -I/usr/include/mit-krb5 -DLINUX_OOM_ADJ=0 -fPIC -pie -I/usr/include/mit-krb5 -DLINUX_OOM_ADJ=0 -Wall -Wmissing-prototypes                   -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -fpic                   -shared -o pgmemcache.so pgmemcache.o -L/usr/lib -Wl,--as-needed -L/usr/lib/mit-krb5 -L/usr/lib//mit-krb5 -Wl,--as-needed -L/usr/lib/mit-krb5 -L                  /usr/lib//mit-krb5  -Wl,--as-needed  -lmemcached -lsasl2

The generated files have errors too:

# CREATE EXTENSION pgmemcache;
ERROR:  invalid extension version name: ""
DETAIL:  Version names must not be empty.

# CREATE EXTENSION pgmemcache WITH VERSION '2.1.1';
ERROR:  could not stat file "/usr/share/postgresql/9.2/extension/pgmemcache--2.1.1.sql": No such file or directory

The extension file is missing the version:

/usr/share/postgresql/9.2/extension/pgmemcache--.sql

Version 2.1.0 works fine.

Cheers,
Patryk

Hello!

I'm trying to setup pgmemcache for use with my Postgres 9.2 database, and am having an issue. Specifically, when I attempt to MAKE the file from source (grabbed from here, so most recent files), I get the following error...

$ make
gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -DUSE_LIBMEMCACHED -I. -I. -I/usr/include/server -I/usr/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o pgmemcache.o pgmemcache.c
pgmemcache.c: In function ‘_PG_init’:
pgmemcache.c:71: warning: passing argument 8 of ‘DefineCustomStringVariable’ from incompatible pointer type
/usr/include/utils/guc.h:225: note: expected ‘GucStringAssignHook’ but argument is of type ‘void (*)(const char *, void *)’
pgmemcache.c:84: warning: passing argument 8 of ‘DefineCustomStringVariable’ from incompatible pointer type
/usr/include/utils/guc.h:225: note: expected ‘GucStringAssignHook’ but argument is of type ‘void (*)(const char *, void *)’
pgmemcache.c: In function ‘pgmemcache_xact_callback’:
pgmemcache.c:165: error: ‘XACT_EVENT_PRE_COMMIT’ undeclared (first use in this function)
pgmemcache.c:165: error: (Each undeclared identifier is reported only once
pgmemcache.c:165: error: for each function it appears in.)
pgmemcache.c: In function ‘memcache_get_multi’:
pgmemcache.c:622: warning: passing argument 1 of ‘get_arg_cstring’ makes pointer from integer without a cast
pgmemcache.c:393: note: expected ‘struct text *’ but argument is of type ‘Datum’
pgmemcache.c:622: warning: assignment discards qualifiers from pointer target type
make: *** [pgmemcache.o] Error 1

I'm guessing there's something wrong with where my pg_config is pointed or something similar, and I'm looking into it. Putting it here on the off chance that it's actually something off with the code and/or someone has some suggestions on what to do that I'm not thinking of.

If there's any information on my end that would help you help me pinpoint the issue, just ask.

Thanks!

ON PG 9.5:
CREATE EXTENSION pgmemcache WITH SCHEMA pgmemcache;
select pgmemcache.memcache_server_add('localhost');
select pgmemcache.memcache_stat();

crush database ;
error message from db log

LOG: server process (PID 2265) was terminated by signal 11: Segmentation fault
DETAIL: Failed process was running: select * from pgmemcache.memcache_stats ();
LOG: terminating any other active server processes

not just memcache_stat function
when i used just two function memcache_set and memcache_get_multi after a while it crush the system with error saying :
*** Error in `postgres: user_name db_name client_ip(41613) SELECT': double free or corruption (!prev): 0x000000000182bcb0 ***
======= Backtrace: ========= ???

A PL/pgSQL script that help to create and mantain a large collection of table partitions o PostgreSQL that use a INTEGER column as a key to split table.

We tried to use the memcache_get_multi function from PL/PGSQL, but the PostgreSQL server randomly crashed. I noticed that in pgmemcache.c, inside the function memcache_get_multi, the following code probably behaves erroneously:

values[0] = (char *) palloc(fctx->key_lens[funcctx->call_cntr] * sizeof(char));
values[1] = (char *) palloc(value_length * sizeof(char));

snprintf(values[0], fctx->key_lens[funcctx->call_cntr] + 1, "%s", ctx->keys[funcctx->call_cntr]);
snprintf(values[1], value_length + 1, "%s", value);

value[1] is allocated value_length characters, but the snprintf can store value_length + 1 characters. Besides, if value contains NUL characters, the function will not store the content after it at all. The approach used in memcache_get utilizing memcpy IMHO is much more appropriate.

Hello,

today we tried to upgrade to pgmemcache 2.1.2 for our PostgreSQL 9.2 installation. We noticed that memcache_get('foobar'); hangs and you have to manuallly cancel the statement. Other commands, like memcache_stats() work fine.

> SELECT memcache_get('foobar');
^CCancel request sent
ERROR:  pgmemcache: memcached_get: FAILURE
Time: 6889.980 ms

In our PostgreSQL 9.3 installation the version 2.1.2 works fine, or version 2.1.0 for PostgreSQL 9.2:

> SELECT memcache_get('foobar');
 memcache_get
--------------
 [null]
(1 row)
Time: 3.192 ms

Cheers,
Patryk

i would like to report an oversight.

The first 2 lines of Makefile are:

short_ver = $(shell git describe --abbrev=0)
long_ver = $(shell git describe --long)

because of the above lines there is an error at make:
fatal: Not a git repository (or any of the parent directories): .git

Is there any reason why there is no memcache_server_delete()? Currently it seems impossible to remove the servers without restarting your server.

Hi,

we ran into a problem with pgmemcache where i can't find much
information on the internet.

First of all, we are running:

Debian 6.0.7
PostgreSQL 9.2.4
pgmemcache 2.0.6
libmemcached-dev 0.40-1

Our postgres server did run for over a week without any problem, but
yesterday suddenly all pgmemcache calls terminated with the following
warning and error message (exception):

WARNING:  invalid value for parameter "pgmemcache.default_servers": "example.org" 
ERROR: NO SERVERS DEFINED

Our configuration is very simple, we have just one server which is
configured in the postgresql.conf:

pgmemcache.default_servers = 'example.org'

And the only pgmemcache function which we use is:

memcache_incr()

After a restart of the postgresql instance the error is gone and
everything works fine again. Unfortunately i'm currently not able to
reproduce the problem.

The memcached server is running fine and serving cache data all the time, there are no errors. We tried to debug this a bit, it seems that the postgresql instance doesn't even try to talk to the memcached server (which makes sense in regards to the warning and error message). A few hours later the persistent database connections of the FCGI processes were WORKING, but i was unable to open a new psql shell and issue a pgmemcache command (new database connection, same error as above).

Can you tell me if you seen this error before, and if it is fixed in one
of the newer pgmemache versions (2.1.0, 2.1.1)?

Thank you very much for your time!

Best regards,
Patryk

Hi,

In http://docs.libmemcached.org/memcached_get.html#memcached_get the description says:

memcached_get() is used to fetch an individual value from the server. 

[..snip..]

The object will be returned upon success and NULL will be returned on
failure.  Any object returned by memcached_get() must be released by
the caller application.

But as far as I can tell, pgmemcache's memcache_get() never releases the value after it has been copied to palloc'd memory.

How is this not a severe memory leak? Am I misunderstanding something?

https://wiki.debian.org/ReleaseGoals/RemoveYada

pgmemcache.c:130:5: error: expected ‘;’ before ‘}’ token

elog(ERROR, "pgmemcache: sasl_client_init failed: %d", rc)

should be:

elog(ERROR, "pgmemcache: sasl_client_init failed: %d", rc);

Have a good day.

See pull requests #31 and #32 : olegdunkan brought up the fact that we use DatumGetCString(DirectFunctionCall1(textout, PointerGetDatum(get_key))) in some places to grab the function argument text and some other places use VARDATA(). It's inconsistent and both of them are probably wrong.

This needs to be fixed in all places handling text fields in pgmemcache and to make sure we properly handle binary strings (including embedded nulls) and toasted and compressed values.

Building fails with latest libmemcached in Debian testing (1.0.4-2):

In file included from pgmemcache.c:11:0:
pgmemcache.h:26:31: fatal error: libmemcached/sasl.h: No such file or directory

They moved the internal header files to a different directory, namely /usr/include/libmemcached-1.0/sasl.h (see http://packages.debian.org/wheezy/i386/libmemcached-dev/filelist). Apparently, you're not supposed to include these directly, since memcached.h ends up including them all anyway. So I suggest just removing the extra includes:

diff --git i/pgmemcache.h w/pgmemcache.h
index 79868ff..d113990 100644
--- i/pgmemcache.h
+++ w/pgmemcache.h
@@ -23,9 +23,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/lsyscache.h"
-#include <libmemcached/sasl.h>
 #include <libmemcached/memcached.h>
-#include <libmemcached/server.h>
 #include <sasl/sasl.h>

 #undef PACKAGE_BUGREPORT

Hi,

Here's situation:

1. One codebase connects to multiple databases;
2. Few cashes can store data from the same table, data is gathered by different sql queries;
3. Having file based cache allows me to create a folder per database and then reset it by tags, having tags allows me to generate keys like md5 of a sql query.

Is something like this possible?

1. memcache_set() - add tags parameter type text[] at the end
2. memcache_flush_tags(tags text[], optional policy smallint) - new function, policy can be ALL or ANY tags.

Thank you in advance.

The make command is failing with below error on Ubuntu 14.04

postgres@ubuntu:~/pgmemcache$ make
gcc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -pie -I/usr/include/mit-krb5 -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -fpic -I. -I./ -I/usr/include/postgresql/9.3/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.6 -c -o pgmemcache.o pgmemcache.c
In file included from pgmemcache.c:11:0:
pgmemcache.h:26:31: fatal error: libmemcached/sasl.h: No such file or directory
#include <libmemcached/sasl.h>
^
compilation terminated.
make: *** [pgmemcache.o] Error 1

postgres@ubuntu:~$ locate sasl.h
/usr/local/include/libmemcached-1.0/sasl.h
/usr/local/include/libmemcached-1.0/struct/sasl.h
/var/lib/postgresql/libmemcached-1.0.18/libmemcached/sasl.hpp
/var/lib/postgresql/libmemcached-1.0.18/libmemcached-1.0/sasl.h
/var/lib/postgresql/libmemcached-1.0.18/libmemcached-1.0/struct/sasl.h

Can you let me know how to point correctly to libmemcached/sasl.h during make process?

Thanks!