Code Monkey home page Code Monkey logo

libvmod-querystring's Introduction

vmod-querystring

Travis CI badge

Codecov badge

Description

The purpose of this module is to give you a fine-grained control over a URL's query-string in Varnish Cache. It's possible to remove the query-string, clean it, sort its parameters or filter it to only keep a subset of them.

This can greatly improve your hit ratio and efficiency with Varnish, because by default two URLs with the same path but different query-strings are also different. This is what the RFCs mandate but probably not what you usually want for your web site or application.

A query-string is just a character string starting after a question mark in a URL. But in a web context, it is usually a structured key/values store encoded with the application/x-www-form-urlencoded media type. This module deals with this kind of query-strings.

Examples

Consider the default hashing in Varnish:

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

Clients requesting /index.html and /index.html? will most likely get the exact same response with most web servers / frameworks / stacks / wossname but Varnish will see two different URLs and end up with two duplicate objects in the cache.

This is a problem hard to solve with Varnish alone because it requires some knowledge of the back-end application but it can usually be mitigated with a couple assumptions:

  • the application doesn't need query-strings
  • except for POST requests that are not cached
  • and for analytics/tracking purposes

In this case it can be solved like this:

import querystring;

sub vcl_hash {
    if (req.method == "GET" || req.method == "HEAD") {
        hash_data(querystring.remove(req.url));
    }
    else {
        hash_data(req.url);
    }
    hash_data(req.http.host);
    return (lookup);
}

This way Varnish will get the same unique hash for both /index.html and /index.html? but the back-end application will receive the original client request. Depending on your requirements/goals, you may also take a different approach.

Surely enough this module can do more than what a simple regular expression substitution (regsub) could do, right? First, readability is improved. It should be obvious what the previous snippet does with no regex to decipher.

Second, it makes more complex operations easier to implement. For instance, you may want to remove Google Analytics parameters from requests because:

  • they could create cache duplicates for every campaigns
  • the application does not need them, only marketing folks
  • the user's browser makes AJAX calls to GA regardless
  • they can be delivered to marketing via varnishncsa

It could be solved like this:

import std;
import querystring;

sub vcl_init {
    new ga = querystring.filter();
    ga.add_regex("^utm_.*");
}

sub vcl_recv {
    std.log("ga:" + ga.extract(req.url, mode = keep));
    set req.url = ga.apply(req.url);
}

This is enough to remove all Analytics parameters you may use (utm_source, utm_medium, utm_campaign etc) and keep the rest of the query-string unless there are no other parameters in which case it's simply removed. The log statement allows you to get those analytics parameters (and only them) in varnishncsa using the format string %{VCL_Log:ga}x.

All functions are documented in the manual page vmod_querystring(3).

Installation

The module relies on the GNU Build System, also known as autotools. To install it, start by grabbing the latest release1 and follow these steps:

# Get to the source tree
tar -xzf vmod-querystring-${VERSION}.tar.gz
cd vmod-querystring-${VERSION}

# Build and install
./configure
make
make check # optional
sudo make install

You only need to have Varnish (at least 6.0.6) and its development files installed on your system. Instead of manually installing the module you can build packages, see below. The configure script also needs pkg-config installed to find Varnish development files.

If your Varnish installation did not use the default /usr prefix, you will likely need to at least set the pkg-config path to find your Varnish installation. For example add this in your environment before running ./configure:

export PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig

Or the approach recommended by autoconf:

./configure PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig ...

The module is then configured for an installation inside ${PREFIX}, unless the --prefix option was used in the configure execution. For more information about what can be configured, run ./configure --help.

Alongside the release archive, you will find a PDF export of the module's manual.

RPM Packaging

Instead of directly installing the package you can build an RPM:

make rpm

The resulting packages can be found in the rpmbuild directory in your build tree.

If you need to build an RPM for a different platform you may use mock(1) with the proper --root option. All you got to do is run make mockbuild and set the desired flags in the MOCK_OPTS variable. For instance, to build RPMs for CentOS 7:

make mockbuild MOCK_OPTS='--root epel-7-x86_64'

The resulting packages can be found in the mockbuild directory in your build tree.

DPKG Packaging

DPKG packaging is also available with dpkg-buildpackage(1), using the deb target:

make deb

It is possible to either redefine the DPKG_BUILDPACKAGE command or simply add options via DPKG_BUILDPACKAGE_OPTS. For example to specify a specific privilege escalation method:

make deb DPKG_BUILDPACKAGE_OPTS=-rfakeroot

The resulting packages can be found in the dpkgbuild directory in your build tree. By default sources and changes are NOT signed, in order to sign packages the DPKG_BUILDPACKAGE variable MUST be redefined.

If you need to build a Deb for a specific platform you may use pdebuild(1) and pbuilder(8) to set up the base tarball and then run make pdebuild and set the desired flags in the PDEBUILD_OPTS variable. For instance to build debs for Debian Sid, assuming your environment is properly configured to switch between distributions:

make pdebuild PDEBUILD_OPTS='-- --distribution sid'

The resulting packages can be found in the pdebuild directory in your build tree.

As an alternative to pdebuild(1) you may prefer sbuild(1) instead. Similarly, you may run make sbuild and set the desired flags in the SBUILD_OPTS variable. For instance to build debs for Debian Sid, assuming your environment is properly configured to switch between distributions:

make sbuild SBUILD_OPTS='--dist sid'

The resulting packages can be found in the sbuild directory in your build tree.

Hacking

When working on the source code, there are additional dependencies:

  • autoconf
  • automake
  • libtool
  • rst2man (python3-docutils)
  • varnish (at least 6.0.6)

You will notice the lack of a configure script, it needs to be generated with the various autotools programs. Instead, you can use the bootstrap script that takes care of both generating and running configure. It also works for VPATH builds.

Arguments to the bootstrap script are passed to the underlying execution of the generated configure script. Once bootstrap is done, you can later run the configure script directly if you need to reconfigure your build tree or use more than one VPATH.

See also

To learn more about query-strings and HTTP caching, you can have a look at the relevant RFCs:

The test suite also shows the differences in cache hits and misses with and without the use of this module.


  1. https://github.com/Dridi/libvmod-querystring/releases/latest

libvmod-querystring's People

Contributors

db-pj avatar dridi avatar gandikun avatar lucasluitjes avatar nigoroll avatar rwoverdijk avatar tjakobsson avatar vdloo avatar zmousm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libvmod-querystring's Issues

Bootstrap fail on CentOS 6.8

Hi,

Im running a CentOS box and the ./bootstrap seems to fail.

configure.ac:31: warning: macro `AM_PROG_AR' not found in library
configure.ac:17: error: Autoconf version 2.68 or higher is required
configure.ac:17: the top level
autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63

I have autoconf268 and autoconf but unsure how to make it use the 268.

v1.0.1 doesn't compile at all

git clone https://github.com/Dridi/libvmod-querystring.git
cd libvmod-querystring
git checkout tags/v1.0.1

Followed the installation steps, doesn't work:

root@varnish1:/tmp/libvmod-querystring# ./bootstrap
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'.
libtoolize: copying file `build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
configure.ac:31: installing 'build-aux/ar-lib'
configure.ac:31: installing 'build-aux/compile'
configure.ac:34: installing 'build-aux/config.guess'
configure.ac:34: installing 'build-aux/config.sub'
configure.ac:29: installing 'build-aux/install-sh'
configure.ac:29: installing 'build-aux/missing'
src/Makefile.am:21: error: CODE_COVERAGE_ENABLED does not appear in AM_CONDITIONAL
src/Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'

root@varnish1:/tmp/libvmod-querystring# make check
make: *** No rule to make target `check'.  Stop.

root@varnish1:/tmp/libvmod-querystring# make
make: *** No targets specified and no makefile found.  Stop.

Symbol not found: 'querystring.regfilter'

Hello
i'm trying to migrate to varnish 5.1.3. So i built lib from vmod-querystring-1.0.3.tar.gz release.

Symbol not found: 'querystring.regfilter' (expected type STRING_LIST): ('/etc/varnish/default.vcl' Line 89 Pos 45) set req.http.locale_name = querystring.regfilter(req.http.url, "[^location|locale]"); ---------------------------#####################--------------------------------------------

how can i use the same functionality as in old libvmod version?

Debian Jessie compatibility

Make fails on Debian Jessie with GCC v4:

make  all-recursive
make[1]: Entering directory '/tmp/vmod-querystring-1.0.2'
Making all in src
make[2]: Entering directory '/tmp/vmod-querystring-1.0.2/src'
  VMODTOOL vmod_querystring.vcc
  CC       vmod_querystring.lo
vmod_querystring.c: In function 'qs_truncate':
vmod_querystring.c:37:7: error: assignment makes pointer from integer without a cast [-Werror]
   tmp = WS_Snapshot(ws); \
       ^
vmod_querystring.c:114:3: note: in expansion of macro 'WS_ClearOverflow'
   WS_ClearOverflow(ws, res);
   ^
vmod_querystring.c:114:24: error: passing argument 2 of 'WS_Reset' makes integer from pointer without a cast [-Werror]
   WS_ClearOverflow(ws, res);
                        ^
vmod_querystring.c:38:16: note: in definition of macro 'WS_ClearOverflow'
   WS_Reset(ws, tmp); \
                ^
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1064:6: note: expected 'uintptr_t' but argument is of type 'char *'
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c: In function 'qs_apply':
vmod_querystring.c:37:7: error: assignment makes pointer from integer without a cast [-Werror]
   tmp = WS_Snapshot(ws); \
       ^
vmod_querystring.c:284:3: note: in expansion of macro 'WS_ClearOverflow'
   WS_ClearOverflow(ctx->ws, res);
   ^
vmod_querystring.c:284:29: error: passing argument 2 of 'WS_Reset' makes integer from pointer without a cast [-Werror]
   WS_ClearOverflow(ctx->ws, res);
                             ^
vmod_querystring.c:38:16: note: in definition of macro 'WS_ClearOverflow'
   WS_Reset(ws, tmp); \
                ^
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1064:6: note: expected 'uintptr_t' but argument is of type 'char *'
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c:347:21: error: passing argument 2 of 'WS_Reset' makes integer from pointer without a cast [-Werror]
   WS_Reset(ctx->ws, res);
                     ^
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1064:6: note: expected 'uintptr_t' but argument is of type 'char *'
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c:362:20: error: passing argument 2 of 'WS_Reset' makes integer from pointer without a cast [-Werror]
  WS_Reset(ctx->ws, cur);
                    ^
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1064:6: note: expected 'uintptr_t' but argument is of type 'char *'
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c: At top level:
cc1: error: unrecognized command line option "-Wno-int-conversion" [-Werror]
cc1: all warnings being treated as errors
Makefile:696: recipe for target 'vmod_querystring.lo' failed
make[2]: Leaving directory '/tmp/vmod-querystring-1.0.2/src'
make[2]: *** [vmod_querystring.lo] Error 1
Makefile:544: recipe for target 'all-recursive' failed
make[1]: Leaving directory '/tmp/vmod-querystring-1.0.2'
Makefile:449: recipe for target 'all' failed
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Is it possible to enable compilation on Jessie with GCC v4? Thanks.

Segmentation fault with sort function

@Dridi, first of all I want to thank you for this awesome vmod. This will help us a lot to improve our hit ratio.

Unfortunately we've found a problem with the sort function.

On my varnish, after the change to regfilter and sort the URL, syslog started to log this:

Mar 23 18:40:48 ca-site-cache-6423 varnishd[2957]: Child (11361) Last panic at: Wed, 23 Mar 2016 21:40:48 GMT#012"Assert error in child_sigsegv_handler(), mgt/mgt_child.c line 282:#012  Condition(Segmentation fault by instruction at (nil)) not true.#012thread = (cache-worker)#012version = varnish-4.1.2 revision 0d7404e#012ident = Linux,4.1.13-19.30.amzn1.x86_64,x86_64,-junix,-sfile,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432d03: varnishd() [0x432d03]#012  0x452659: varnishd() [0x452659]#012  0x7f13bf64f100: libpthread.so.0(+0xf100) [0x7f13bf64f100]#012  0x7f13b1dfadef: libvmod_querystring.so(vmod_sort+0x2ef) [0x7f13b1dfadef]#012  0x7f13b6484242: vgc.so(VGC_function_vcl_recv+0x62) [0x7f13b6484242]#012  0x43f274: varnishd() [0x43f274]#012  0x43f63a: varnishd(VCL_recv_method+0x5a) [0x43f63a]#012  0x437496: varnishd(CNT_Request+0x7a6) [0x437496]#012  0x44e27b: varnishd(HTTP1_Session+0x11b) [0x44e27b]#012  0x43a22d: varnishd(SES_Proto_Req+0x5d) [0x43a22d]#012req = 0x7f13bea51020 {#012  vxid = 557057, step = R_STP_RECV,#012  req_body = R_BODY_NONE,#012  restarts = 0, esi_level = 0,#012  sp = 0x7f13b712d420 {#012    fd = 20, vxid = 1,#012    client = 10.2.5.120 59583,#012    step = S_STP_H1PROC,#012  },#012  worker = 0x7f13c0c9fc00 {#012    stack = {0x7f13c0ca0000 -> 0x7f13c0c94000},#012    ws = 0x7f13c0c9fdf8 {#012      id = \"wrk\",#012      {s,f,r,e} = {0x7f13c0c9f3c0,0x7f13c0c9f3c0,(nil),+2040},#012    },#012    VCL::method = inside RECV,#012    VCL::return = abandon,#012    VCL::methods = {RECV},#012  },#012  ws = 0x7f13bea51200 {#012    id = \"req\",#012    {s,f,r,e} = {0x7f13bea53000,+3040,+57336,+57336},#012  },#012  http_conn = 0x7f13bea51128 {#012    fd = 20,#012    doclose = NULL,#012    ws = 0x7f13bea51200,#012    {rxbuf_b, rxbuf_e} = {0x7f13bea53000, 0x7f13bea538da},#012    {pipeline_b, pipeline_e} = {(nil), (nil)},#012    content_length = -1,#012    body_status = none,#012    first_byte_timeout = 0.000000,#012    between_bytes_timeout = 0.000000,#012  },#012  http[req] = 0x7f13bea51298 {#012    ws[req] = 0x7f13bea51200,#012    hdrs {#012      \"GET\",#012      \"/vivacorretor/wp-admin/admin-ajax.php?callback=po_data&orig_request_uri=%2Fvivacorretor%2Fmelhores-horarios-de-postagem-da-sua-pagina-no-facebook%2F&action=inc_popup&do=get_data&thefrom=%2Fkoobecaf-on-anigap-aus-ad-megatsop-ed-soiraroh-serohlem%2Froterrocaviv%2Frb.moc.laeraviv.www%2F%2F%3Aptth&thereferrer=%2Fmoc.gnib.www%2F%2F%3Asptth&_=1458769246888\",#012      \"HTTP/1.1\",#012      \"host: www.vivareal.com.br\",#012      \"Accept: text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01\",#012      \"Accept-Encoding: gzip, deflate\",#012      \"Accept-Language: pt-BR,pt;q=0.5\",#012      \"Cookie: optimizelyEndUserId=oeu1453926388295r0.7119827927739377; optOutCookie=true; zanox_cookie=other; vivareal_ranking=0; PHPSESSID=h9a2jrdtkboja9bd1jodacebc5; VR_COUNT_ACCESS=91; _ga=GA1.3.1837013940.1453926389; new_vivareal_user_id=6ac1f2c2a2f4e5bf2bb5f3d4829af6ef; vivareal_ranking=0; localytics.js=1; __ll_websdk_csi=14; __ll_websdk_cst=1456886438; __ll_websdk_csu=%22e7f6b30c-da7e-ddc7-f183-9fb4cbaa2cce%22; __ll_websdk_lot=1456886438; __ll_websdk_cd={%220%22:%22Generic%22}; __ll_websdk_lct=1456886703785; __ll_websdk_pa=1453926402; __ll_websdk_csq=518; __ll_websdk_iu=%22b02b4bcb-c2b6-31d1-8e76-848da33631a8%22; __ll_websdk_ids={%22customer_id%22:%226ac1f2c2a2f4e5bf2bb5f3d4829af6ef%22}; __ll_websdk_fl=[%22Search%20Results%22]; __gads=ID=4570b7d6e270b8bb:T=1453926404:S=ALNI_MZnxPmBcpJ7nIZQVwT_AhlsbtG30w; _VR_COOKIE=RmVsaXBlJTIwUGluaGVpcm86ZmVsaXBlLnJhaXpjaWRhZGFuaXN0YUBnbWFpbC5jb206MTE5OTcxMTM1OTE; VR_SIMILAR_OPT_OUT=false; optimizelySegments=%7B%22231107883%22%3A%22false%22%2C%22231408497%22%3A%22edge%22%2C%22231453599%22%3A%22campaign%22%2C%224404411069%22%3A%22none%22%7D; optimizelyBuckets=%7B%224973150094%22%3A%220%22%7D; _ceg.s=o4ff07; _ceg.u=o4ff07; _spl_pv=10; sb_client=56d48439725bce5f86778d69; sb_customer=55s62f809dfdfc9ee4762c5bbb2c2ec1d99hf7dc72c418ec7f21769e4t23fbe3a3e677\",#012      \"Referer: http://www.vivareal.com.br/vivacorretor/melhores-horarios-de-postagem-da-sua-pagina-no-facebook/\",#012      \"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.10586\",#012      \"X-Requested-With: XMLHttpRequest\",#012      \"X-Forwarded-Port: 80\",#012      \"X-Forwarded-Proto: http\",#012      \"Connection: keep-alive\",#012      \"X-Forwarded-For: 187.113.154.125, 10.2.5.120\",#012    },#012  },#012  vcl = {#012    temp = warm#012    srcname = {#012      \"/etc/varnish/site.vcl\",#012      \"Builtin\",#012      \"/etc/varnish/devicedetect.vcl\",#012    },#012  },#012  flags = {#012  },#012},#012#012"
Mar 23 18:40:49 ca-site-cache-6423 varnishd[2957]: Child (11923) said Child starts
Mar 23 18:40:49 ca-site-cache-6423 varnishd[2957]: Child (11923) said SMF.s0 mmap'ed 27917287424 bytes of 27917287424
Mar 23 18:40:53 ca-site-cache-6423 varnishd[2957]: Child (11923) died signal=6
Mar 23 18:40:53 ca-site-cache-6423 varnishd[2957]: Child (11923) Last panic at: Wed, 23 Mar 2016 21:40:53 GMT#012"Assert error in child_sigsegv_handler(), mgt/mgt_child.c line 282:#012  Condition(Segmentation fault by instruction at (nil)) not true.#012thread = (cache-worker)#012version = varnish-4.1.2 revision 0d7404e#012ident = Linux,4.1.13-19.30.amzn1.x86_64,x86_64,-junix,-sfile,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432d03: varnishd() [0x432d03]#012  0x452659: varnishd() [0x452659]#012  0x7f13bf64f100: libpthread.so.0(+0xf100) [0x7f13bf64f100]#012  0x7f13b25fbdef: libvmod_querystring.so(vmod_sort+0x2ef) [0x7f13b25fbdef]#012  0x7f13b6484242: vgc.so(VGC_function_vcl_recv+0x62) [0x7f13b6484242]#012  0x43f274: varnishd() [0x43f274]#012  0x43f63a: varnishd(VCL_recv_method+0x5a) [0x43f63a]#012  0x437496: varnishd(CNT_Request+0x7a6) [0x437496]#012  0x44e27b: varnishd(HTTP1_Session+0x11b) [0x44e27b]#012  0x43a22d: varnishd(SES_Proto_Req+0x5d) [0x43a22d]#012req = 0x7f13b60d1020 {#012  vxid = 196627, step = R_STP_RECV,#012  req_body = R_BODY_NONE,#012  restarts = 0, esi_level = 0,#012  sp = 0x7f13b3820420 {#012    fd = 22, vxid = 32769,#012    client = 10.2.5.120 59671,#012    step = S_STP_H1PROC,#012  },#012  worker = 0x7f13b66b9c00 {#012    stack = {0x7f13b66ba000 -> 0x7f13b66ae000},#012    ws = 0x7f13b66b9df8 {#012      id = \"wrk\",#012      {s,f,r,e} = {0x7f13b66b93c0,0x7f13b66b93c0,(nil),+2040},#012    },#012    VCL::method = inside RECV,#012    VCL::return = abandon,#012    VCL::methods = {RECV},#012  },#012  ws = 0x7f13b60d1200 {#012    id = \"req\",#012    {s,f,r,e} = {0x7f13b60d3000,+2600,+57336,+57336},#012  },#012  http_conn = 0x7f13b60d1128 {#012    fd = 22,#012    doclose = NULL,#012    ws = 0x7f13b60d1200,#012    {rxbuf_b, rxbuf_e} = {0x7f13b60d3000, 0x7f13b60d351e},#012    {pipeline_b, pipeline_e} = {(nil), (nil)},#012    content_length = -1,#012    body_status = none,#012    first_byte_timeout = 0.000000,#012    between_bytes_timeout = 0.000000,#012  },#012  http[req] = 0x7f13b60d1298 {#012    ws[req] = 0x7f13b60d1200,#012    hdrs {#012      \"GET\",#012      \"/vivacorretor/wp-admin/admin-ajax.php?callback=po_data&orig_request_uri=%2Fvivacorretor%2Fdicasvaliosasparacorretoresdeimoveisiniciantes%2F&action=inc_popup&do=get_data&thefrom=%2Fsetnaicinisievomiedseroterrocarapsasoilavsacid%2Froterrocaviv%2Frb.moc.laeraviv.www%2F%2F%3Aptth&thereferrer=QA3H_Hwyn2py2GIVqeGTLsgJJftFNCjQFA%3Dgsu%26F2%25setnaicinisievomiedseroterrocarapsasoilavsacidF2%25roterrocavivF2%25rb.moc.laeraviv.wwwF2%25F2%25A3%25ptth%3Dlru%26UAM6ggFQcBAi2UHKApjMUhALft4AbMsyjwEKUha0%3Ddev%266%3Ddc%26bew%3Decruos%26s%3Dcrse%26%3Dq%26j%3Dtcr%26t%3Das%3Flru%2Frb.moc.elgoog.www%2F%2F%3Aptth&_=1458769251374\",#012      \"HTTP/1.1\",#012      \"host: www.vivareal.com.br\",#012      \"Accept: text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01\",#012      \"Accept-Encoding: gzip, deflate\",#012      \"Accept-Language: pt-br,pt;q=0.8,en-us;q=0.5,en;q=0.3\",#012      \"Cache-Control: max-age=259200\",#012      \"Cookie: PHPSESSID=1oa3l6uvv0labm1anpuebvo0n6\",#012      \"Referer: http://www.vivareal.com.br/vivacorretor/dicasvaliosasparacorretoresdeimoveisiniciantes/\",#012      \"User-Agent: Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0\",#012      \"Via: 1.1 prdproxy51.prd:8080 (squid/2.7.STABLE9)\",#012      \"X-Requested-With: XMLHttpRequest\",#012      \"X-Forwarded-Port: 80\",#012      \"X-Forwarded-Proto: http\",#012      \"Connection: keep-alive\",#012      \"X-Forwarded-For: 10.137.34.2, 200.201.3.51, 10.2.5.120\",#012    },#012  },#012  vcl = {#012    temp = warm#012    srcname = {#012      \"/etc/varnish/site.vcl\",#012      \"Builtin\",#012      \"/etc/varnish/devicedetect.vcl\",#012    },#012  },#012  flags = {#012  },#012},#012#012"
Mar 23 18:40:53 ca-site-cache-6423 varnishd[2957]: Child (12211) said Child starts
Mar 23 18:40:53 ca-site-cache-6423 varnishd[2957]: Child (12211) said SMF.s0 mmap'ed 27917287424 bytes of 27917287424
Mar 23 18:41:20 ca-site-cache-6423 varnishd[2957]: Child (12211) died signal=6
Mar 23 18:41:20 ca-site-cache-6423 varnishd[2957]: Child (12211) Last panic at: Wed, 23 Mar 2016 21:41:20 GMT#012"Assert error in child_sigsegv_handler(), mgt/mgt_child.c line 282:#012  Condition(Segmentation fault by instruction at (nil)) not true.#012thread = (cache-worker)#012version = varnish-4.1.2 revision 0d7404e#012ident = Linux,4.1.13-19.30.amzn1.x86_64,x86_64,-junix,-sfile,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432d03: varnishd() [0x432d03]#012  0x452659: varnishd() [0x452659]#012  0x7f13bf64f100: libpthread.so.0(+0xf100) [0x7f13bf64f100]#012  0x7f13b25fbdef: libvmod_querystring.so(vmod_sort+0x2ef) [0x7f13b25fbdef]#012  0x7f13b6484242: vgc.so(VGC_function_vcl_recv+0x62) [0x7f13b6484242]#012  0x43f274: varnishd() [0x43f274]#012  0x43f63a: varnishd(VCL_recv_method+0x5a) [0x43f63a]#012  0x437496: varnishd(CNT_Request+0x7a6) [0x437496]#012  0x44e27b: varnishd(HTTP1_Session+0x11b) [0x44e27b]#012  0x43a22d: varnishd(SES_Proto_Req+0x5d) [0x43a22d]#012req = 0x7f13beb12020 {#012  vxid = 30, step = R_STP_RECV,#012  req_body = R_BODY_NONE,#012  restarts = 0, esi_level = 0,#012  sp = 0x7f13b712ea20 {#012    fd = 66, vxid = 229413,#012    client = 10.2.5.120 59872,#012    step = S_STP_H1PROC,#012  },#012  worker = 0x7f13bf02cc00 {#012    stack = {0x7f13bf02d000 -> 0x7f13bf021000},#012    ws = 0x7f13bf02cdf8 {#012      id = \"wrk\",#012      {s,f,r,e} = {0x7f13bf02c3c0,0x7f13bf02c3c0,(nil),+2040},#012    },#012    VCL::method = inside RECV,#012    VCL::return = abandon,#012    VCL::methods = {RECV},#012  },#012  ws = 0x7f13beb12200 {#012    id = \"req\",#012    {s,f,r,e} = {0x7f13beb14000,+1664,+57336,+57336},#012  },#012  http_conn = 0x7f13beb12128 {#012    fd = 66,#012    doclose = NULL,#012    ws = 0x7f13beb12200,#012    {rxbuf_b, rxbuf_e} = {0x7f13beb14000, 0x7f13beb143ba},#012    {pipeline_b, pipeline_e} = {(nil), (nil)},#012    content_length = -1,#012    body_status = none,#012    first_byte_timeout = 0.000000,#012    between_bytes_timeout = 0.000000,#012  },#012  http[req] = 0x7f13beb12298 {#012    ws[req] = 0x7f13beb12200,#012    hdrs {#012      \"GET\",#012      \"/vivacorretor/wp-admin/admin-ajax.php?callback=po_data&orig_request_uri=%2Fvivacorretor%2Fsalario-afinal-quanto-ganha-um-corretor-de-imoveis%2F&action=inc_popup&do=get_data&thefrom=%2Fsievomi-ed-roterroc-mu-ahnag-otnauq-lanifa-oiralas%2Froterrocaviv%2Frb.moc.laeraviv.www%2F%2F%3Aptth&thereferrer=&_=1458769278622\",#012      \"HTTP/1.1\",#012      \"host: www.vivareal.com.br\",#012      \"Accept: text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01\",#012      \"Accept-Encoding: gzip,deflate,br\",#012      \"Accept-Language: en\",#012      \"Referer: http://www.vivareal.com.br/vivacorretor/salario-afinal-quanto-ganha-um-corretor-de-imoveis/\",#012      \"User-Agent: Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19\",#012      \"X-Requested-With: XMLHttpRequest\",#012      \"X-Forwarded-Port: 80\",#012      \"X-Forwarded-Proto: http\",#012      \"Connection: keep-alive\",#012      \"X-Forwarded-For: 201.150.48.230, 64.233.172.182, 10.2.5.120\",#012    },#012  },#012  vcl = {#012    temp = warm#012    srcname = {#012      \"/etc/varnish/site.vcl\",#012      \"Builtin\",#012      \"/etc/varnish/devicedetect.vcl\",#012    },#012  },#012  flags = {#012  },#012},#012#012"
Mar 23 18:41:20 ca-site-cache-6423 varnishd[2957]: Child (13527) said Child starts
Mar 23 18:41:20 ca-site-cache-6423 varnishd[2957]: Child (13527) said SMF.s0 mmap'ed 27917287424 bytes of 27917287424
Mar 23 18:51:28 ca-site-cache-6423 varnishd[2957]: Child (13527) died signal=6
Mar 23 18:51:28 ca-site-cache-6423 varnishd[2957]: Child (13527) Last panic at: Wed, 23 Mar 2016 21:51:28 GMT#012"Assert error in child_sigsegv_handler(), mgt/mgt_child.c line 282:#012  Condition(Segmentation fault by instruction at (nil)) not true.#012thread = (cache-worker)#012version = varnish-4.1.2 revision 0d7404e#012ident = Linux,4.1.13-19.30.amzn1.x86_64,x86_64,-junix,-sfile,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432d03: varnishd() [0x432d03]#012  0x452659: varnishd() [0x452659]#012  0x7f13bf64f100: libpthread.so.0(+0xf100) [0x7f13bf64f100]#012  0x7f13b25fbdef: libvmod_querystring.so(vmod_sort+0x2ef) [0x7f13b25fbdef]#012  0x7f13b6484242: vgc.so(VGC_function_vcl_recv+0x62) [0x7f13b6484242]#012  0x43f274: varnishd() [0x43f274]#012  0x43f63a: varnishd(VCL_recv_method+0x5a) [0x43f63a]#012  0x437496: varnishd(CNT_Request+0x7a6) [0x437496]#012  0x44e27b: varnishd(HTTP1_Session+0x11b) [0x44e27b]#012  0x43a22d: varnishd(SES_Proto_Req+0x5d) [0x43a22d]#012req = 0x7f13b6097020 {#012  vxid = 196849, step = R_STP_RECV,#012  req_body = R_BODY_WITH_LEN,#012  restarts = 0, esi_level = 0,#012  sp = 0x7f13b381fc20 {#012    fd = 67, vxid = 196846,#012    client = 10.2.5.120 63095,#012    step = S_STP_H1PROC,#012  },#012  worker = 0x7f13b66c5c00 {#012    stack = {0x7f13b66c6000 -> 0x7f13b66ba000},#012    ws = 0x7f13b66c5df8 {#012      id = \"wrk\",#012      {s,f,r,e} = {0x7f13b66c53c0,0x7f13b66c53c0,(nil),+2040},#012    },#012    VCL::method = inside RECV,#012    VCL::return = abandon,#012    VCL::methods = {RECV},#012  },#012  ws = 0x7f13b6097200 {#012    id = \"req\",#012    {s,f,r,e} = {0x7f13b6099000,+4488,+57336,+57336},#012  },#012  http_conn = 0x7f13b6097128 {#012    fd = 67,#012    doclose = NULL,#012    ws = 0x7f13b6097200,#012    {rxbuf_b, rxbuf_e} = {0x7f13b6099000, 0x7f13b6099dd5},#012    {pipeline_b, pipeline_e} = {0x7f13b6099dd5, 0x7f13b6099e23},#012    content_length = 78,#012    body_status = length,#012    first_byte_timeout = 0.000000,#012    between_bytes_timeout = 0.000000,#012  },#012  http[req] = 0x7f13b6097298 {#012    ws[req] = 0x7f13b6097200,#012    hdrs {#012      \"POST\",#012      \"/searchers/login?apiKey=183d98b9-fc81-4ef1-b841-7432c610b36e&portal=VR_BR&access_token=eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX25hbWUiOiI1NmFhOWY1OWU0YjBjNWM0ZTcxNmZiODUiLCJhdXRob3JpdGllcyI6WyJST0xFX1VTRVIiXSwianRpIjoiYjljNTljYjktZjI3NC00ZTA2LThiNGItYzhhNTYxODk4OWQ1IiwiY2xpZW50X2lkIjoiZDdjNThlYjgtMmJhYS00ZTZjLTgyOTctNjk3ZmNmMGM3ZDM0Iiwic2NvcGUiOlsicmVhZCIsIndyaXRlIl19.t2C5yKsPdkKojeyX61CiRp3bu1UJtiIm9hQRfSQipaI\",#012      \"HTTP/1.1\",#012      \"host: www.vivareal.com.br\",#012      \"Accept: application/json, text/javascript, */*; q=0.01\",#012      \"Accept-Encoding: gzip, deflate\",#012      \"Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4\",#012      \"Content-Type: application/json\",#012      \"Cookie: new_vivareal_user_id=2a9ec957ed64e28ac2b197bb116971ba; __gads=ID=b9129c4992ff0762:T=1457031724:S=ALNI_MYeP7lscSg5hjx-yRQLktG3KEeBhA; optimizelyEndUserId=oeu1457184170267r0.7011718996800482; __ll_websdk_pa=1457184170; __ll_websdk_iu=%22f01e7ac7-218c-2e58-5d74-8c98016ab56c%22; optOutCookie=true; vivareal_ranking=0; VR_SIMILAR_OPT_OUT=false; __ll_websdk_csi=27; __ll_websdk_cst=1458767779; __ll_websdk_csu=%22eb04bce9-aa9f-8bd2-2d9a-5ea3df56ae14%22; __ll_websdk_lot=1458767779; __ll_websdk_fl=[%22Search%20Results%22]; sb_client=56d48439725bce5f86778d69; sb_customer=55s62e8c93fdfc8e14762c5btb2c2ec0d79e176c521498bf7a2772974h633b231376c7; sb_browser=52cdcbc18c913619a4e400096f085850; sb_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJodHRwczpcL1wvYXBpLnNiYWNrLnRlY2giLCJpYXQiOjE0NTg3Njc4MDQsImV4cCI6MTQ1ODg1NDIwNCwiZGF0YSI6eyJwdWJsaWNfaWQiOnsiY2xpZW50IjoiNTZkNDg0Mzk3MjViY2U1Zjg2Nzc4ZDY5IiwiY3VzdG9tZXIiOiI1NXM2MmU4YzkzZmRmYzhlMTQ3NjJjNWJ0YjJjMmVjMGQ3OWUxNzZjNTIxNDk4YmY3YTI3NzI5NzRoNjMzYjIzMTM3NmM3IiwiZG9tYWluIjoidml2YXJlYWwuY29tLmJyIn0sImxldmVsIjoianNzZGsifX0.XwPh2aBjhO44WoG49GWQP1jFjgCDXXCdrWAEouEF_lM.WrWrzRDriYEiuyEiiYqBzR; sb_helper_url=no; sb_helper_cart=no; zanox_cookie=other; localytics.js=1; _vwo_uuid_v2=010D6623DDC69D15562FA2CA0C7B0B12|c3a7f8539bfbba9bdfcd2a28ee6450ed; __ll_websdk_cd={%220%22:%22Generic%22}; __ll_websdk_ids={%22customer_id%22:%222a9ec957ed64e28ac2b197bb116971ba%22}; vrSpToken=eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX25hbWUiOiI1NmFhOWY1OWU0YjBjNWM0ZTcxNmZiODUiLCJhdXRob3JpdGllcyI6WyJST0xFX1VTRVIiXSwianRpIjoiYjljNTljYjktZjI3NC00ZTA2LThiNGItYzhhNTYxODk4OWQ1IiwiY2xpZW50X2lkIjoiZDdjNThlYjgtMmJhYS00ZTZjLTgyOTctNjk3ZmNmMGM3ZDM0Iiwic2NvcGUiOlsicmVhZCIsIndyaXRlIl19.t2C5yKsPdkKojeyX61CiRp3bu1UJtiIm9hQRfSQipaI; _VR_COOKIE=VkFMRE9NSVJPOm1pcm8ucHJhaWFzQGdtYWlsLmNvbTooMTEpOTcwMi0zODgyMg; __ll_websdk_lct=1458769497476; __ll_websdk_csq=645; normalWeb=true; JSESSIONID=34E848ED525EC89FA20C1A0FF6A2CA3E-n1; optimizelySegments=%7B%22231107883%22%3A%22false%22%2C%22231408497%22%3A%22gc%22%2C%22231453599%22%3A%22campaign%22%2C%224404411069%22%3A%22confirmacao-lead-lcm%22%7D; optimizelyBuckets=%7B%225195491626%22%3A%225198741473%22%7D; VR_COUNT_ACCESS=305; _ga=GA1.3.336455948.1457031729; _gat=1; _gat_newTracker=1; _ceg.s=o4ii18; _ceg.u=o4ii18; _st_ses=3126315704991802; _spcid=264; _st_no_script=1; _spl_pv=544; optimizelyPendingLogEvents=%5B%22n%3Dengagement%26u%3Doeu1457184170267r0.7011718996800482%26wxhr%3Dtrue%26time%3D1458769888.739%26f%3D5195491626%2C5377881308%26g%3D29327782%22%5D; _st_id=bWlyby5pbnZlc3RpbW92ZWlzQGdtYWlsLmNvbQ==\",#012      \"Origin: http://www.vivareal.com.br\",#012      \"Referer: http://www.vivareal.com.br/\",#012      \"User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36\",#012      \"X-Requested-With: XMLHttpRequest\",#012      \"X-Forwarded-Port: 80\",#012      \"X-Forwarded-Proto: http\",#012      \"Content-Length: 78\",#012      \"Connection: keep-alive\",#012      \"X-Forwarded-For: 177.18.226.191, 10.2.5.120\",#012    },#012  },#012  vcl = {#012    temp = warm#012    srcname = {#012      \"/etc/varnish/site.vcl\",#012      \"Builtin\",#012      \"/etc/varnish/devicedetect.vcl\",#012    },#012  },#012  flags = {#012  },#012},#012#012"
Mar 23 18:51:28 ca-site-cache-6423 varnishd[2957]: Child (18658) said Child starts
Mar 23 18:51:28 ca-site-cache-6423 varnishd[2957]: Child (18658) said SMF.s0 mmap'ed 27917287424 bytes of 27917287424

My vcl set this req.url:

set req.url = querystring.regfilter(req.url, "(utm_.*)|(img.*)|(gclid.*)");
set req.url = querystring.sort(req.url);

When I removed the sort function, the error disappeared.

Am I using this right?

My varnish version:

varnishd (varnish-4.1.2 revision 0d7404e)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS

Any information that I can provide to help with this problem, feel free to ask me.

Thanks

Request to remove ?ver query string

I am using this with Varnish 3 to prevent bots form using query strings and it's working great (no more weird hit ratio) so than you for that. I do not see it removing ?ver query strings like wp-emoji-release.min.js?ver=4.3.1 though

The code I am using is this

if (req.url ~ "\.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf|cur)") {
if (!req.url ~ "sccss") {
        set req.url = querystring.remove(req.url);
}
}

I tried adding set req.url = querystring.regfilter(req.url, "?ver=+."); after querystring.remove but no luck

regfilter remove parameter regex

how can filter an exactly example: accept param1 and discard param1_fail for example.
because this set req.url = querystring.regfilter(req.url, "^(?!param1|param2)"); don't work.

Regards.

Error: Child (##) died signal=6 during: set req.url = querystring.sort(req.url);

@Dridi, first of all I want to thank you for this awesome vmod!

This issue looks similar to #22.
As you suggested I used set req.url = std.querysort(req.url); as a workaround.

For example this URL:

http://###domain###/led-panels/?actionData=%7B%22added_product%22%3A%2216668%22%2C%22added_item%22%3A%222310773%22%2C%22added_parent_item%22%3Anull%2C%22added_product_qty%22%3A3%2C%22content%22%3Afalse%2C%22mobile%22%3Afalse%7D&block%5B%5D=cart&block%5B%5D=sidebar&block%5B%5D=topLinks&block%5B%5D=wishlist&block%5B%5D=miniWishlist&block%5B%5D=addProductConfirmation&block%5B%5D=removeProductConfirmation&awacp=1&no_cache=1

crashes Varnish when executing:

set req.url = querystring.sort(req.url);

Here is some information I gathered:

varnish version: varnish-4.1.1 revision 66bb824
querystring vmod version da7b5c9 "Bump version to 0.5"
os: Ubuntu 16.04.1 LTS
kernel: Linux ##host## 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)

$ varnishadm panic.show

Last panic at: Fri, 12 Aug 2016 19:15:38 GMT
"Assert error in child_sigsegv_handler(), mgt/mgt_child.c line 282:
  Condition(Segmentation fault by instruction at (nil)) not true.
thread = (cache-worker)
version = varnish-4.1.1 revision 66bb824
ident = Linux,4.4.0-34-generic,x86_64,-jnone,-smalloc,-smalloc,-hcritbit,epoll
Backtrace:
  0x4333b5: varnishd() [0x4333b5]
  0x452853: varnishd() [0x452853]
  0x7fa7c6bc73d0: libpthread.so.0(+0x113d0) [0x7fa7c6bc73d0]
  0x7fa7b65fafe0: libvmod_querystring.so(vmod_sort+0x420) [0x7fa7b65fafe0]
  0x7fa7bb1f2233: vgc.so(VGC_function_normalize_url+0x93) [0x7fa7bb1f2233]
  0x7fa7bb1f2c22: vgc.so(VGC_function_vcl_recv+0xf2) [0x7fa7bb1f2c22]
  0x43edac: varnishd() [0x43edac]
  0x440eda: varnishd(VCL_recv_method+0x5a) [0x440eda]
  0x4367a6: varnishd(CNT_Request+0x996) [0x4367a6]
  0x44e78e: varnishd(HTTP1_Session+0xbe) [0x44e78e]
req = 0x7fa7b5419020 {
  vxid = 2, step = R_STP_RECV,
  req_body = R_BODY_NONE,
  restarts = 0, esi_level = 0,
  sp = 0x7fa7b540e220 {
    fd = 8, vxid = 1,
    client = 86.109.18.96 50960,
    step = S_STP_H1PROC,
  },
  worker = 0x7fa7bea2ec80 {
    stack = {0x7fa7bea2f000 -> 0x7fa7bea23000},
    ws = 0x7fa7bea2ee78 {
      id = \"wrk\",
      {s,f,r,e} = {0x7fa7bea2e420,0x7fa7bea2e420,(nil),+2040},
    },
    VCL::method = inside RECV,
    VCL::return = abandon,
    VCL::methods = {RECV},
  },
  ws = 0x7fa7b5419200 {
    id = \"req\",
    {s,f,r,e} = {0x7fa7b541b000,+3120,+57336,+57336},
  },
  http_conn = 0x7fa7b5419128 {
    fd = 8,
    doclose = NULL,
    ws = 0x7fa7b5419200,
    {rxbuf_b, rxbuf_e} = {0x7fa7b541b000, 0x7fa7b541b658},
    {pipeline_b, pipeline_e} = {(nil), (nil)},
    content_length = -1,
    body_status = none,
    first_byte_timeout = 0.000000,
    between_bytes_timeout = 0.000000,
  },
  http[req] = 0x7fa7b5419298 {
    ws[req] = 0x7fa7b5419200,
    hdrs {
      \"GET\",
      \"/led-panels/?actionData=%7B%22added_product%22%3A%2216668%22%2C%22added_item%22%3A%222310773%22%2C%22added_parent_item%22%3Anull%2C%22added_product_qty%22%3A3%2C%22content%22%3Afalse%2C%22mobile%22%3Afalse%7D&block%5B%5D=cart&block%5B%5D=sidebar&block%5B%5D=topLinks&block%5B%5D=wishlist&block%5B%5D=miniWishlist&block%5B%5D=addProductConfirmation&block%5B%5D=removeProductConfirmation&awacp=1&no_cache=1\",
      \"HTTP/1.1\",
      \"Host: ###domain###\",
      \"Accept: text/javascript, text/html, application/xml, text/xml, */*\",
      \"X-Prototype-Version: 1.7\",
      \"X-Requested-With: XMLHttpRequest\",
      \"User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36\",
      \"Referer: https://###domain###/led-panels/\",
      \"Accept-Encoding: gzip, deflate, sdch, br\",
      \"Accept-Language: en-US,en;q=0.8,nl;q=0.6,de;q=0.4,fr;q=0.2\",
      \"Cookie: __utma=184780153.537474437.1452263132.1460032141.1460032141.1; __utmz=184780153.1460032141.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _ga=GA1.2.537474437.1452263132; varnish_token=CEsbRmy0XFW8BoEk; varnish_token_checksum=4eaae2a0b9f954180d802ebaa5cff3ca; external_no_cache=1; last-visited-homepage=aHR0cHM6Ly9zdGFnaW5nMy5hbnktbGFtcC5jb20vZml4dHVyZXMv; store=###store###; frontend=b68e08ca5408579fa73bc022c0069675; last-visited-homepage-time=1471029337; bmec=95006b633bf89df2fdafefc4e7a101aa; quote_checksum=0f2ff1d8a764bcd64de5a638c44f9a4f; segment_checksum=b7ab7e48dcc687dba7b2eaac86398e78; varnish_messages=checkout\",
      \"X-Real-Ip: ##.##.##.243\",
      \"X-forwarded-Proto: https\",
      \"Ssl-Offloaded: 1\",
      \"Connection: close\",
      \"X-Forwarded-For: ##.##.##.243, ##.##.##.96\",
    },
  },
  vcl = {
    temp = warm
    srcname = {
      \"/etc/varnish/varnish-###name###.vcl\",
      \"Builtin\",
      \"/data/vhosts/###domain###/httpdocs/var/default.vcl\",
    },
  },
  flags = {
  },
},

"

vmod-querystring enterprise edition

Background

This is an approximately 3-years old idea (precisely from around Apr 25th 2013) of taking advantage of a new feature for an unreleased Varnish 4. Namely VMOD objects.

Having supported this VMOD for Varnish 3 until its EOL, I've never done it and in the mean time Varnish 4.1 was released with more cool stuff. So this time I've decided to lower support for Varnish 4.0 down to bug fixes only and do a 4.1-only rewrite of the module.

You can try it in its dedicated branch:
https://github.com/Dridi/libvmod-querystring/tree/feature/obj

Documentation should be up-to-date but I'll describe the main difference here. YMMV and all changes may not appeal to everyone, but I think it's the best compromise. It has more capabilities with a reduced code base and a lower footprint.

Object initialization

The big change is the introduction of VMOD objects instead of plain functions. It has the inconvenience of adding one indirection and the associated overhead.

# before
sub vcl_recv {
    # remove google analytics query params
    set req.url = querystring.regfilter(req.url, "^utm_.*");
}

# after
sub vcl_init {
    # a filter for google analytics query params
    new ga = querystring.filter();
    ga.add_regex("^utm_.*");
}

sub vcl_recv {
    set req.url = ga.apply(req.url);
}

There are however some benefits that deserve to be mentioned. Filters have names, which may improve readability and of something like ga is too short you may pick a less abbreviated name like googanal or analytics. Also if your regex is not valid, it will fail the vcl.load operation instead of logging it every time it is used (who reads the logs anyway?) which also means that the regex is compiled only once.

This looks also better when using an exact-match criteria, you no longer need to clutter your VCL code with querystring.filtersep() which is AFAIK the longest short separator ever. Did you notice how separator is abbreviated to just sep? :trollface:

Composition

If you need to apply more than one filter, and on top of that sort the remaining parameters, you can now do that with a single filter.

# before
sub vcl_recv {
    # removed query params:
    # - google analytics
    # - timestamp-based cache bypass
    # - anti-CSRF token
    set req.url = querystring.globfilter(req.url, "utm_*");
    set req.url = querystring.filter(req.url, "_");
    set req.url = querystring.regfilter(req.url, "sess[0-9]+");

    set req.url = querystring.sort(req.url);
}

# after
sub vcl_init {
    new qf = querystring.filter(sort = true);
    qf.add_glob("utm_*"); # google analytics parameters
    qf.add_string("_"); # timestamp-based cache bypass
    qf.add_regex("sess[0-9]+"); # anti-CSRF token
}

sub vcl_recv {
    set req.url = qf.apply(req.url);
}

Of course the "before" example is exaggerated and could be solved with a single call to regfilter (+ sort) but the point is that the "after" example handles it better and requires a lot less workspace to do so. Enabling sorting in a filter has no space overhead whatsoever, only time.

Better semantics

What would be the *filter without their *filter_except counterparts? With objects it can be solved in different places like the constructor, the add_* methods, or the apply method. I opted for the latter, even though it's not the most flexible of all, it's the best compromise wrt simplicity.

There's no apply_except function but instead an optional mode ENUM (like the optional sort BOOL in the constructor, 4.1-only).

# before
sub vcl_recv {
    set req.url = querystring.filter_except(req.url, "q");
}

# after
sub vcl_init {
    new search = querystring.filter();
    search.add_string("q");
}

sub vcl_recv {
    set req.url = search.apply(req.url, mode = keep);
    # or just
    set req.url = search.apply(req.url, keep);
}

I find it a lot easier to reason about negating filters, especially for regular expressions.

Misc

The clean, sort and remove shorthand functions are still here and rely on filters under the hood.

There's also a new .extract method similar to .apply except that it discards the rest of the URL and only returns the query-string.

The code style evolved from my 2012 self's to Varnish's, making it more maintainable. There's also less code to maintain but more documentation now.

Feedback

This work is already done and testable, including documentation. I still have to cover some cases in the test suite, so until I'm done I'm interested in your feedback.

Am I missing something obvious?
Do you have comments?

Thanks,
Dridi

Documentation

Your module is pretty cool. But it's seriously lacking documentation. Also, it doesn't work on FreeBSD ootb.

Failing when running /usr/share/varnish-plus/vmodtool.py on varnisn-plus-6

config.status: executing libtool commands

	==== libvmod-querystring 1.0.5 ====

	varnish:      6.0.0r2
	prefix:       /usr
	vmoddir:      /usr/lib64/varnish-plus/vmods

	compiler:     gcc -std=gnu99
	cflags:       -g -O2
	ldflags:

	--enable-silent-rules=
	--enable-warnings=no
	--enable-docs=no
	--enable-lcov=no

Making install in src
make[1]: Entering directory `/usr/local/src/vmod-querystring-1.0.5/src'
/usr/bin/python /usr/share/varnish-plus/vmodtool.py -o vcc_querystring_if ./vmod_querystring.vcc
While parsing line:
	 Synopsis manual

WARNING: Unknown stanze $Synopsis manual

Traceback (most recent call last):
  File "/usr/share/varnish-plus/vmodtool.py", line 1089, in <module>
    runmain(i_vcc, opts.rstdir, opts.output)
  File "/usr/share/varnish-plus/vmodtool.py", line 1042, in runmain
    v.parse()
  File "/usr/share/varnish-plus/vmodtool.py", line 857, in parse
    m([c[0], " ".join(c[1:])], ss[i:].split('\n'), self)
TypeError: 'NoneType' object is not callable
make[1]: *** [vcc_querystring_if.c] Error 1
make[1]: Leaving directory `/usr/local/src/vmod-querystring-1.0.5/src'
make: *** [install-recursive] Error 1

error compiling with varnish 3.0.5

Hi,

Im trying to compile libvmod-querystring with the new varnish version. Are there any incompatibility’s?

vmod_querystring.c: In function 'is_param_regfiltered':
vmod_querystring.c:283: warning: passing argument 1 of 'VRT_re_match' from incompatible pointer type
/root/varnish-3.0.5/include/vrt.h:149: note: expected 'const struct sess ' but argument is of type 'char *'
vmod_querystring.c:283: error: too few arguments to function 'VRT_re_match'
make[2]: *
* [vmod_querystring.lo] Error 1
make[2]: Leaving directory /root/libvmod-querystring/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory/root/libvmod-querystring'
make: *** [all] Error 2

failed build w/ varnish 6.6: " 'VRT_re_fini' undeclared"

just starting to look at this myself.

fyi, building with varnish 6.6,

https://copr.fedorainfracloud.org/coprs/pgfed/varnish/builds/

build fails @

...
make[2]: Leaving directory '/builddir/build/BUILD/vmod-querystring-2.0.2/src'
vmod_querystring.c: In function 'vmod_filter_add_regex':
vmod_querystring.c:598:21: error: 'VRT_re_fini' undeclared (first use in this function); did you mean 'VRT_priv_fini'?
  598 |         qsf->free = VRT_re_fini;
      |                     ^~~~~~~~~~~
      |                     VRT_priv_fini
vmod_querystring.c:598:21: note: each undeclared identifier is reported only once for each function it appears in

build log @

https://download.copr.fedorainfracloud.org/results/pgfed/varnish/fedora-34-x86_64/02199579-vmod-querystring/builder-live.log.gz

make install fails on 7.3

----------------------------------------------------------------------
 /bin/mkdir -p '/usr/share/man/man3'
 /usr/bin/install -c -m 644 ./vmod_querystring.3 '/usr/share/man/man3'
/usr/bin/install: cannot stat './vmod_querystring.3': No such file or directory

also, make keeps building stuff, never reaching the usual "nothing to be done" stage:

root@e24fe2d4ec34:/tmp/module_to_build# make
make  all-recursive
make[1]: Entering directory '/tmp/module_to_build'
Making all in .
make[2]: Entering directory '/tmp/module_to_build'
cd src && \
/usr/bin/python3.9 /usr/share/varnish/vmodtool.py -o vcc_querystring_if vmod_querystring.vcc
make[2]: Leaving directory '/tmp/module_to_build'
Making all in src
make[2]: Entering directory '/tmp/module_to_build/src'
/bin/bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I..    -I/usr/include/varnish -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c -o vmod_querystring.lo vmod_querystring.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -I/usr/include/varnish -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c vmod_querystring.c  -fPIC -DPIC -o .libs/vmod_querystring.o
mv -f .deps/vmod_querystring.Tpo .deps/vmod_querystring.Plo
/bin/bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I..    -I/usr/include/varnish -g -O2 -MT vcc_querystring_if.lo -MD -MP -MF .deps/vcc_querystring_if.Tpo -c -o vcc_querystring_if.lo vcc_querystring_if.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -I/usr/include/varnish -g -O2 -MT vcc_querystring_if.lo -MD -MP -MF .deps/vcc_querystring_if.Tpo -c vcc_querystring_if.c  -fPIC -DPIC -o .libs/vcc_querystring_if.o
mv -f .deps/vcc_querystring_if.Tpo .deps/vcc_querystring_if.Plo
/bin/bash ../libtool  --tag=CC   --mode=link gcc -I/usr/include/varnish -g -O2 -module -export-dynamic -avoid-version -shared  -o libvmod_querystring.la -rpath /usr/lib/varnish/vmods vmod_querystring.lo vcc_querystring_if.lo  
libtool: link: rm -fr  .libs/libvmod_querystring.la .libs/libvmod_querystring.lai .libs/libvmod_querystring.so
libtool: link: gcc -shared  -fPIC -DPIC  .libs/vmod_querystring.o .libs/vcc_querystring_if.o    -g -O2   -Wl,-soname -Wl,libvmod_querystring.so -o .libs/libvmod_querystring.so
libtool: link: ( cd ".libs" && rm -f "libvmod_querystring.la" && ln -s "../libvmod_querystring.la" "libvmod_querystring.la" )
: vmod_querystring.man.rst vmod_querystring.3
make[2]: Leaving directory '/tmp/module_to_build/src'
make[1]: Leaving directory '/tmp/module_to_build'

most probably this is because vmodtool doesn't generate the same files as it used to?

Can't build on varnish 4.1.1 (ubuntu linux precise 12.04.5 LTS)

Hi

Great VMOD... I've been able to build this vmod for Varnish 4.1.0 on OSX Yosemite but struggling with Varnish 4.1.1 on Ubuntu LTS Precise

root@mgmt:~/libvmod-querystring# varnishd -V
varnishd (varnish-4.1.1 revision 66bb824)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS

root@mgmt:~/libvmod-querystring# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise

Steps to reproduce:

  1. checkout libvmod-querystring source (on ubuntu server)
  2. switch to v0.4 release tag
  3. run ./autogen.sh
  4. run ./configure
  5. run make check

Get the following error

root@blah:~/libvmod-querystring# make check
Making check in src
make[1]: Entering directory /home/quinton/libvmod-querystring/src' /bin/bash ../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -DQS_ENABLE_LOGGING -I/usr/include/varnish -Wall -Werror -Wextra -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c -o vmod_querystring.lo vmod_querystring.c libtool: compile: gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -DQS_ENABLE_LOGGING -I/usr/include/varnish -Wall -Werror -Wextra -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c vmod_querystring.c -fPIC -DPIC -o .libs/vmod_querystring.o vmod_querystring.c:44:20: fatal error: vcc_if.h: No such file or directory compilation terminated. make[1]: *** [vmod_querystring.lo] Error 1 make[1]: Leaving directory/home/quinton/libvmod-querystring/src'
make: *** [check-recursive] Error 1

Please help!

Does not build on Varnish 4.0.0-tp2

Hi,

I've seen that your VMOD is prepared for Varnish 4.0.0 tp1 but, tp2 VMOD structure has changed a lot and I don't think you'll be able to keep the VMOD compatible with both v3 and v4 in the same source...

I can help converting the VMOD structure for tp2 but v3 and v4-tp1 compatibility would be lost...

1.0.2 using tar-ball does not build on RHEL 7.3

Two points.

When using bootstrap:

  • automake-version required is 1.15, but RHEL 7.3 does only have automake 1.13.x. Is automake 1.15 really needed?

When just doing configure, make:

  • GCC outputs "-Wint-conversion" is an unknown option. So, same thing there. You may require a newer gcc that knows this option, but .. is the option really needed?

RHEL 7.3 is fully updated. Yea, RedHat is slow on advancing versions, but 1.0.1 builds on RHEL 7.3 just fine.

failed build w/ varnish 7.0: "vmod_querystring.c:589:42: error: passing argument 3 of 'VRE_compile' from incompatible pointer type'"

fyi, building with varnish 7.0 (currently src.rpm rebuild from just-fresh v7 pkgs @ Fedora F36/rawhide ...)

https://copr.fedorainfracloud.org/coprs/pgfed/varnish/builds/

build fails @

...
vmod_querystring.c:589:42: error: passing argument 3 of 'VRE_compile' from incompatible pointer type [-Werror=incompatible-pointer-types]
  589 |         qsf->ptr = VRE_compile(regex, 0, &error, &error_offset);
      |                                          ^~~~~~
      |                                          |
      |                                          const char **
In file included from vmod_querystring.c:32:
/usr/include/varnish/vre.h:58:44: note: expected 'int *' but argument is of type 'const char **'
   58 | vre_t *VRE_compile(const char *, unsigned, int *, int *, unsigned);
      |                                            ^~~~~
vmod_querystring.c:589:20: error: too few arguments to function 'VRE_compile'
  589 |         qsf->ptr = VRE_compile(regex, 0, &error, &error_offset);
      |                    ^~~~~~~~~~~
In file included from vmod_querystring.c:32:
/usr/include/varnish/vre.h:58:8: note: declared here
   58 | vre_t *VRE_compile(const char *, unsigned, int *, int *, unsigned);
      |        ^~~~~~~~~~~
cc1: all warnings being treated as errors
make[2]: *** [Makefile:710: vmod_querystring.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Entering directory '/builddir/build/BUILD/libvmod-querystring-master/src'
rst2man vmod_querystring.man.rst vmod_querystring.3
make[2]: Leaving directory '/builddir/build/BUILD/libvmod-querystring-master/src'
make[1]: *** [Makefile:590: all-recursive] Error 1
make: *** [Makefile:492: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.DiWzNs (%build)
...

current full build log here:

https://gist.github.com/pgnd/2cbe7e5e3c34fc6a9522ef8db4584855

It looks like this from @nigoroll

9202445

may be needed?

Compile fails with Varnish 5.2

Compiles and works in the same environment with 4.1 installed,

make
make  all-recursive
make[1]: Entering directory `/etc/varnish/vmods/vmod-querystring-1.0.3'
Making all in src
make[2]: Entering directory `/etc/varnish/vmods/vmod-querystring-1.0.3/src'
  CC       vmod_querystring.lo
vmod_querystring.c: In function ‘qs_truncate’:
vmod_querystring.c:37:7: error: assignment makes pointer from integer without a cast [-Werror]
   tmp = WS_Snapshot(ws); \
       ^
vmod_querystring.c:114:3: note: in expansion of macro ‘WS_ClearOverflow’
   WS_ClearOverflow(ws, res);
   ^
vmod_querystring.c:114:3: error: passing argument 2 of ‘WS_Reset’ makes integer from pointer without a cast [-Werror]
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1057:6: note: expected ‘uintptr_t’ but argument is of type ‘char *’
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c: In function ‘qs_apply’:
vmod_querystring.c:37:7: error: assignment makes pointer from integer without a cast [-Werror]
   tmp = WS_Snapshot(ws); \
       ^
vmod_querystring.c:284:3: note: in expansion of macro ‘WS_ClearOverflow’
   WS_ClearOverflow(ctx->ws, res);
   ^
vmod_querystring.c:284:3: error: passing argument 2 of ‘WS_Reset’ makes integer from pointer without a cast [-Werror]
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1057:6: note: expected ‘uintptr_t’ but argument is of type ‘char *’
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c:347:3: error: passing argument 2 of ‘WS_Reset’ makes integer from pointer without a cast [-Werror]
   WS_Reset(ctx->ws, res);
   ^
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1057:6: note: expected ‘uintptr_t’ but argument is of type ‘char *’
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c:362:2: error: passing argument 2 of ‘WS_Reset’ makes integer from pointer without a cast [-Werror]
  WS_Reset(ctx->ws, cur);
  ^
In file included from vmod_querystring.c:28:0:
/usr/include/varnish/cache/cache.h:1057:6: note: expected ‘uintptr_t’ but argument is of type ‘char *’
 void WS_Reset(struct ws *ws, uintptr_t);
      ^
vmod_querystring.c: At top level:
cc1: error: unrecognized command line option "-Wno-int-conversion" [-Werror]
cc1: all warnings being treated as errors
make[2]: *** [vmod_querystring.lo] Error 1
make[2]: Leaving directory `/etc/varnish/vmods/vmod-querystring-1.0.3/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/etc/varnish/vmods/vmod-querystring-1.0.3'
make: *** [all] Error 2

Release for Varnish 6?

Hi Dridi,

I'm not sure if this is the best place to make this request, and if not, my apologies for the troubles.

I was looking at packaging this for Varnish 6, and it looks like the latest released version (1.0.4) doesn't contain the fixes you'd applied to have the VMOD work with Varnish 6. Would you be willing to cut a minor release that has the match_proto fix in there?

I tested the VMOD using make distcheck on a Varnish 6 host, then running ./configure && make check in VMs containing Varnish 5.2 and 6.0 and it looked to work without issue.

Thanks so much for your work on this, it's been quite helpful!
-Spikes

Add bootstrap script to release tarball

... so when downloading release tarball the ./bootstrap command works as well.

Typically not needed, but useful when patching and you don't want to modify (much) build scripts.

Doesn't compile against varnish-3.0.3

With https://github.com/varnish/Varnish-Cache.git at varnish-3.0.3 and https://github.com/Dridi/querysort at master:

./configure VARNISHSRC=../Varnish-Cache VMODDIR=/usr/lib64/varnish/vmods/ --with-querysort-src=../querysort` && make

eventually bails out with:

/bin/sh ../libtool --tag=CC   --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/home/richardg/workspace/querysort/src -I/home/richardg/workspace/Varnish-Cache/include -I/home/richardg/workspace/Varnish-Cache    -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c -o vmod_querystring.lo vmod_querystring.c
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/home/richardg/workspace/querysort/src -I/home/richardg/workspace/Varnish-Cache/include -I/home/richardg/workspace/Varnish-Cache -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c vmod_querystring.c  -fPIC -DPIC -o .libs/vmod_querystring.o
vmod_querystring.c: In function 'is_param_regfiltered':
vmod_querystring.c:235:2: warning: passing argument 1 of 'VRT_re_match' from incompatible pointer type [enabled by default]
In file included from vmod_querystring.c:38:0:
/home/richardg/workspace/Varnish-Cache/include/vrt.h:148:5: note: expected 'const struct sess *' but argument is of type 'char *'
vmod_querystring.c:235:2: error: too few arguments to function 'VRT_re_match'
In file included from vmod_querystring.c:38:0:
/home/richardg/workspace/Varnish-Cache/include/vrt.h:148:5: note: declared here
make[2]: *** [vmod_querystring.lo] Error 1
make[2]: Leaving directory `/home/richardg/workspace/libvmod-querystring/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/richardg/workspace/libvmod-querystring'
make: *** [all] Error 2

Compile fails varnish 6.2

Compile fails varnish 6.2
I tried 1.0.5, 1.0.6, 2.0.1, master
error in make install

# make
make  all-recursive
make[1]: Entering directory `/root/libvmod-querystring-2.0.1'
Making all in src
make[2]: Entering directory `/root/libvmod-querystring-2.0.1/src'
/usr/bin/python3.6 /usr/share/varnish/vmodtool.py -o vcc_querystring_if ./vmod_querystring.vcc

NOTICE: Please put $Module description in quotes.

/bin/sh ../libtool  --tag=CC   --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I..    -I/usr/include/varnish   -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c -o vmod_querystring.lo vmod_querystring.c
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/varnish -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c vmod_querystring.c  -fPIC -DPIC -o .libs/vmod_querystring.o
mv -f .deps/vmod_querystring.Tpo .deps/vmod_querystring.Plo
/bin/sh ../libtool  --tag=CC   --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I..    -I/usr/include/varnish   -g -O2 -MT vcc_querystring_if.lo -MD -MP -MF .deps/vcc_querystring_if.Tpo -c -o vcc_querystring_if.lo vcc_querystring_if.c
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/usr/include/varnish -g -O2 -MT vcc_querystring_if.lo -MD -MP -MF .deps/vcc_querystring_if.Tpo -c vcc_querystring_if.c  -fPIC -DPIC -o .libs/vcc_querystring_if.o
mv -f .deps/vcc_querystring_if.Tpo .deps/vcc_querystring_if.Plo
/bin/sh ../libtool  --tag=CC   --mode=link gcc -std=gnu99 -I/usr/include/varnish   -g -O2 -module -export-dynamic -avoid-version -shared  -o libvmod_querystring.la -rpath /usr/lib64/varnish/vmods vmod_querystring.lo vcc_querystring_if.lo
libtool: link: gcc -shared  -fPIC -DPIC  .libs/vmod_querystring.o .libs/vcc_querystring_if.o    -O2   -Wl,-soname -Wl,libvmod_querystring.so -o .libs/libvmod_querystring.so
libtool: link: ( cd ".libs" && rm -f "libvmod_querystring.la" && ln -s "../libvmod_querystring.la" "libvmod_querystring.la" )
: vmod_querystring.man.rst vmod_querystring.3
make[2]: Leaving directory `/root/libvmod-querystring-2.0.1/src'
make[2]: Entering directory `/root/libvmod-querystring-2.0.1'
make[2]: Leaving directory `/root/libvmod-querystring-2.0.1'
make[1]: Leaving directory `/root/libvmod-querystring-2.0.1'
[root@kvm19061801 libvmod-querystring-2.0.1]# sudo make install
Making install in src
make[1]: Entering directory `/root/libvmod-querystring-2.0.1/src'
: vmod_querystring.man.rst vmod_querystring.3
make[2]: Entering directory `/root/libvmod-querystring-2.0.1/src'
make[2]: Nothing to be done for `install-exec-am'.
: vmod_querystring.man.rst vmod_querystring.3
 /usr/bin/mkdir -p '/usr/share/man/man3'
 /usr/bin/install -c -m 644 ./vmod_querystring.3 '/usr/share/man/man3'
/usr/bin/install: cannot stat ‘./vmod_querystring.3’: No such file or directory
make[2]: *** [install-man3] Error 1
make[2]: Leaving directory `/root/libvmod-querystring-2.0.1/src'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/root/libvmod-querystring-2.0.1/src'
make: *** [install-recursive] Error 1

Whitelist functionality

Hi,

So I've had a little play around with the plugin, is it possible to set a whitelist of query strings? seems I can only do a blacklist?

For example if I wanted to say to only allow ?v= on assets it seems a little tricky to do, just as an example.

filter_except creates garbage in return string when req.url over a certain size

Compiled and tested against Varnish 4.1.3 installed from the Varnish debian repo:

If req.url is over a certain size, the filter_except method will return a broken string where parts of the url are removed or replaced by parts of the query string.

Example varnish test that fails due to the req.url size:

varnishtest "Test filter_except querystring from req.url in vcl_hash"

server s1 {
        rxreq
        expect req.url ==  "/item/deal/1728366?refcode=xxxxx-xxxx-xxxxxx-xxxxxxx"
        txresp
} -start

varnish v1 -vcl+backend {
        import ${vmod_querystring};

        sub vcl_hash {
                set req.url = querystring.filter_except(req.url,
                                          "q" + querystring.filtersep() + "refcode");
        }
} -start

client c1 {
        txreq -url "/item/deal/1728366?refcode=xxxxx-xxxx-xxxxxx-xxxxxxx"
        rxresp
        expect resp.status == 200

        txreq -url "/item/deal/1728366?refcode=xxxxx-xxxx-xxxxxx-xxxxxxx&_=timestamp"
        rxresp
        expect resp.status == 200
}

varnish v1 -expect n_object == 0
varnish v1 -expect cache_miss == 0
varnish v1 -expect cache_hit == 0

client c1 -run
delay .1

varnish v1 -expect n_object == 1
varnish v1 -expect cache_miss == 1
varnish v1 -expect cache_hit == 1

If you replace the above test url with something smaller like /item/deal/1728366?refcode=xxxxx-xxxx-xxxxxx-xxx, the test will pass.

make install fails after disabling man file generation with ./configure --with-rst2man=true

After running configure with ./configure --with-rst2man=true to disable man file generation, make install still expects the man pages, and will fail.

This is a production image and no man pages are needed, nor do I need python-docutils and all of its dependencies installed for anything else.

If rst2man is missing when compiling official vmods like those at https://github.com/varnish/varnish-modules, the configure script will recognize that and not bother trying to generate, or install the man files.

make install error output:

# ./configure --with-rst2man=true
...
# make
...
# make install
Making install in src
make[1]: Entering directory `/tmp/libvmod-querystring-0.5/src'
true vmod_querystring.man.rst vmod_querystring.3
make[2]: Entering directory `/tmp/libvmod-querystring-0.5/src'
make[2]: Nothing to be done for `install-exec-am'.
true vmod_querystring.man.rst vmod_querystring.3
 /bin/mkdir -p '/usr/share/man/man3'
 /usr/bin/install -c -m 644 ./vmod_querystring.3 '/usr/share/man/man3'
/usr/bin/install: cannot stat './vmod_querystring.3': No such file or directory
make[2]: *** [install-man3] Error 1
make[2]: Leaving directory `/tmp/libvmod-querystring-0.5/src'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/tmp/libvmod-querystring-0.5/src'
make: *** [install-recursive] Error 1

Crash on make install

System:
Vagrant Ubuntu precise32
Varnish 3.0.7 revision f544cd8

root@precise32:~# git https://github.com/Dridi/libvmod-querystring.git ~/libvmod-querystring [Good]
root@precise32:~# cd ~/libvmod-querystring
root@precise32:~/libvmod-querystring# ./autogen.sh # [Good]
root@precise32:~/libvmod-querystring# ./configure VARNISHSRC=/opt/varnish-3.0.7 VMODDIR=/usr/local/lib/varnish/vmods # [Good] 
root@precise32:~/libvmod-querystring# make # [Good]
root@precise32:~/libvmod-querystring# make install # [Bad]
Making install in src
make[1]: Entering directory `/root/libvmod-querystring/src'
/bin/bash ../libtool --tag=CC   --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/opt/varnish-3.0.7/include -I/opt/varnish-3.0.7/bin/varnishd -I/opt/varnish-3.0.7   -DVARNISH_MAJOR=3 -DHAVE_VARNISH_3_0_0 -DQS_ENABLE_LOGGING -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c -o vmod_querystring.lo vmod_querystring.c
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I/opt/varnish-3.0.7/include -I/opt/varnish-3.0.7/bin/varnishd -I/opt/varnish-3.0.7 -DVARNISH_MAJOR=3 -DHAVE_VARNISH_3_0_0 -DQS_ENABLE_LOGGING -g -O2 -MT vmod_querystring.lo -MD -MP -MF .deps/vmod_querystring.Tpo -c vmod_querystring.c  -fPIC -DPIC -o .libs/vmod_querystring.o
vmod_querystring.c: In function 'is_param_regfiltered':
vmod_querystring.c:284:2: warning: passing argument 1 of 'VRT_re_match' from incompatible pointer type [enabled by default]
/opt/varnish-3.0.7/include/vrt.h:149:5: note: expected 'const struct sess *' but argument is of type 'char *'
vmod_querystring.c:284:2: error: too few arguments to function 'VRT_re_match'
/opt/varnish-3.0.7/include/vrt.h:149:5: note: declared here
make[1]: *** [vmod_querystring.lo] Error 1
make[1]: Leaving directory `/root/libvmod-querystring/src'
make: *** [install-recursive] Error 1

Thanks!

Can't build on Varnish 4

I am getting No Varnish source tree specified. I am able to build other vmods like vsthrottle and vshield. Does libvmod-query string not take advantage of the libvarnishapi-dev

src/Makefile.am:26: error: 'vmod_LTLIBRARIES' is used but 'vmoddir' is undefined, varnish 6.6.0

/usr/src/libvmod-querystring# ./bootstrap
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:30: installing 'build-aux/compile'
configure.ac:29: installing 'build-aux/missing'
src/Makefile.am:26: error: 'vmod_LTLIBRARIES' is used but 'vmoddir' is undefined
automake: warnings are treated as errors
src/Makefile.am:30: warning: variable 'libvmod_querystring_la_SOURCES' is defined but no program or
src/Makefile.am:30: library has 'libvmod_querystring_la' as canonical name (possible typo)
src/Makefile.am:33: warning: variable 'nodist_libvmod_querystring_la_SOURCES' is defined but no program or
src/Makefile.am:33: library has 'libvmod_querystring_la' as canonical name (possible typo)

libvmod-querystring:
fffe758
Varnish: 6.6.0
Regards

branch 4.0 filter_except generates garbage URL

Having the problem like in #25 .

Your branch 4.0 with Varnish 4.1 (other combinations didn't compile).

My varnishd is from the following repo:

https://repo.varnish-cache.org/debian/ trusty varnish-4.1

varnishd (varnish-4.1.4 revision 4529ff7)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS

It works with some URLs, but with those URLs like the one in this example, they get mixed up the strangest way possible. Even the parameters from the exception list get mixed in several times completely weird (notice 3 times the ? symbol).
It's not only with long URLs. It happens also with /article/a.html?some=param.

config in recv_pool:

set req.http.X-Test = querystring.filter_except(req.url,
    "otest"
);
set req.http.X-Test = querystring.sort(req.http.X-Test);
set req.url = querystring.clean(req.http.X-Test);

Requested URL:

/artikel/aaaaaaaaaa-2017-aa-aaa-aaa-aaaaaaaaaaaa-1234567.html?otest=HELLO

Returned URL:

/artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO

varnishlog output:

VCL_call       vmod_filter_except("/artikel/aaaaaaaaaa-2017-aa-aaa-aaa-aaaaaaaaaaaa-1234567.html?otest=HELLO", "otest", ...)
VCL_return     "/artikel/aaaaaaaaaa-2017-aa-aaa-aaa-aaaaaaaaaaaa-1234567.html?otest=HELLO"
ReqHeader      X-Test: /artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO
VCL_call       vmod_sort("/artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO")
VCL_return     "/artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO"
ReqUnset       X-Test: /artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO
ReqHeader      X-Test: /artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO
VCL_call       vmod_clean("/artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO")
VCL_return     "/artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO"
ReqURL         /artikel/aa-2017-aaaaaaa-aaaaaaa-html?ot-html?otehtml?otest=HELLest=HELLO

Test-failure: regfilter03_regfilter_empty_parameters.vtc

With little effort I managed to build the module against Varnish-3.0.7 -- the latest release available in the 3.x branch.

Running the check, however, I see the following error:

#     top  TEST tests/regfilter01_regfilter_in_vcl_hash.vtc passed (0.754)
/data/X/packages/varnish/bin/varnishtest -Dvarnishd=/data/X/packages/varnish/sbin/varnishd -Dvmod_topbuild=/data/home/Y/rpmbuild/varnish-querystring/BUILD/Dridi-libvmod-querystring-e3cfe0d tests/regfilter03_regfilter_empty_parameters.vtc
\######## tests/regfilter03_regfilter_empty_parameters.vtc ########
**Assert error in VSL_Open_CallBack(), vsl.c line 366:**
  Condition(sha != NULL) not true.
  errno = 2 (No such file or directory)
**** top   0.0 macro def varnishd=/data/X/packages/varnish/sbin/varnishd
**** top   0.0 macro def vmod_topbuild=/data/home/Y/rpmbuild/varnish-querystring/BUILD/Dridi-libvmod-querystring-e3cfe0d
**** top   0.0 macro def pwd=/data/home/Y/rpmbuild/varnish-querystring/BUILD/Dridi-libvmod-querystring-e3cfe0d/src
**** top   0.0 macro def topbuild=/data/home/Y/rpmbuild/varnish-querystring/BUILD/Dridi-libvmod-querystring-e3cfe0d/src/../..
**** top   0.0 macro def bad_ip=192.0.2.255
**** top   0.0 macro def tmpdir=/tmp/vtc.13957.44d4b9a6
*    top   0.0 TEST tests/regfilter03_regfilter_empty_parameters.vtc starting
***  top   0.0 varnishtest
*    top   0.0 TEST Test regfilter empty parameters
***  top   0.0 server
**   s1    0.0 Starting server
**** s1    0.0 macro def s1_addr=127.0.0.1
**** s1    0.0 macro def s1_port=56435
**** s1    0.0 macro def s1_sock=127.0.0.1 56435
*    s1    0.0 Listen on 127.0.0.1 56435
***  top   0.0 varnish
**   s1    0.0 Started on 127.0.0.1 56435
**   v1    0.0 Launch
***  v1    0.0 CMD: cd ${pwd} && ${varnishd} -d -d -n /tmp/vtc.13957.44d4b9a6/v1 -l 10m,1m,- -p auto_restart=off -p syslog_cli_traffic=off -a '127.0.0.1:0' -S /tmp/vtc.13957.44d4b9a6/v1/_S -M '127.0.0.1 39224' -P /tmp/vtc.13957.44d4b9a6/v1/varnishd.pid -sfile,/tmp/vtc.13957.44d4b9a6/v1,10M 
***  v1    0.0 CMD: cd /data/home/Y/rpmbuild/varnish-querystring/BUILD/Dridi-libvmod-querystring-e3cfe0d/src && /data/X/packages/varnish/sbin/varnishd -d -d -n /tmp/vtc.13957.44d4b9a6/v1 -l 10m,1m,- -p auto_restart=off -p syslog_cli_traffic=off -a '127.0.0.1:0' -S /tmp/vtc.13957.44d4b9a6/v1/_S -M '127.0.0.1 39224' -P /tmp/vtc.13957.44d4b9a6/v1/varnishd.pid -sfile,/tmp/vtc.13957.44d4b9a6/v1,10M 
***  v1    0.0 PID: 13963
***  v1    0.1 debug| Platform: Linux,2.6.32-504.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit\n
***  v1    0.1 debug| 200 278     \n
***  v1    0.1 debug| -----------------------------\n
***  v1    0.1 debug| Varnish Cache CLI 1.0\n
***  v1    0.1 debug| -----------------------------\n
***  v1    0.1 debug| Linux,2.6.32-504.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit\n
***  v1    0.1 debug| varnish-3.0.7 revision f544cd8\n
***  v1    0.1 debug| \n
***  v1    0.1 debug| Type 'help' for command list.\n
***  v1    0.1 debug| Type 'quit' to close CLI session.\n
***  v1    0.1 debug| Type 'start' to launch worker process.\n
***  v1    0.1 debug| \n

#     top  TEST tests/regfilter03_regfilter_empty_parameters.vtc FAILED (0.069) signal=6 exit=0

Please, advise.

Compile time warnings on el7

When doing a mock build against varnish-plus-41 on epel7, I get

/usr/include/varnish-plus/miniobj.h:8:5: warning: "HAVE_EXPLICIT_BZERO" is not defined [-Wundef]
#if HAVE_EXPLICIT_BZERO
^
cc1: error: unrecognized command line option "-Wno-int-conversion" [-Werror]
cc1: all warnings being treated as errors

The fix is probably trivial. I can override by removing "-Werror" Red Hat's build CFLAGS.

Also, the generated specfile could have a macro %varnish_name, to make it compatible with both varnish and varnish-plus.

Ingvar

src build fails with varnish 6.5.0; 'WS_Reserve()' removed

after upgrading 'varnish'

version 6.4.0 -> 6.5.0

vmod querystring src rebuild fails at missing 'WE_Reserve' symbol; e.g. @:

https://download.copr.fedorainfracloud.org/results/pgfed/varnish/fedora-32-x86_64/01678125-vmod-querystring/builder-live.log.gz
...
***  v1   debug|Could not delete 'vcl_vcl1.1600526223.737474/vgc.sym': No such file or directory
***  v1   CLI RX  106
**** v1   CLI RX|Message from VCC-compiler:
**** v1   CLI RX|Could not open VMOD querystring
**** v1   CLI RX|\tFile name: /builddir/build/BUILD/vmod-querystring-2.0.1/src/.libs/libvmod_querystring.so
**** v1   CLI RX|\tdlerror: /builddir/build/BUILD/vmod-querystring-2.0.1/src/.libs/libvmod_querystring.so: undefined symbol: WS_Reserve
**** v1   CLI RX|('<vcl.inline>' Line 5 Pos 16)
**** v1   CLI RX|        import querystring;
**** v1   CLI RX|---------------###########-
**** v1   CLI RX|
**** v1   CLI RX|Running VCC-compiler failed, exited with 2
**** v1   CLI RX|VCL compilation failed
...

checking notes

https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst#user-content-next-scheduled-2020-09-15

	"
	...
	Overhaul of the workspace API
		...
		The previously deprecated WS_Reserve() has been removed
		...
	"

Varnish 5 support

Is Varnish 5 supported? You have a branch for varnish 4.0 but Varnish 5 does not have one.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.