valyala / ybc Goto Github PK
View Code? Open in Web Editor NEWFast in-process BLOB cache with persistence support
Home Page: https://groups.google.com/forum/?fromgroups=#!forum/ybc-main
License: Other
Fast in-process BLOB cache with persistence support
Home Page: https://groups.google.com/forum/?fromgroups=#!forum/ybc-main
License: Other
YBC - Yet another Blob Cache library. This library implements fast in-process blob cache with persistence support. =============================================================================== YBC features. * Huge amounts of data can be cached. Cache size may exceed available RAM size by multiple orders of magnitude. * Very large objects may be cached (up to 2^64 bytes). Think of media files such as videos, audios and images. * Very large cache size is supported (up to 2^64 items and up to 2^64 bytes). In practice cache size is limited by free space on backing store, not by available RAM size. * Persistence support. Cached objects survive process restart if the cache is explicitly backed by files. * Data file layout is optimized for HDD and SSD devices. The code prefers sequential I/O instead of random I/O. If random I/O is inevitable, the code tries hard localizing it in the smallest possible memory region. * Automatic robust recovery from corrupted index files. Corruptions in data files may be left unnoticed due to performance reasons - it may be quite expensive validating multi-GB blobs on every access. * Built-in handling of dogpile effect (aka 'thundering herd'). * Cached objects may be sharded among available backing store devices. Such sharding may linearly increase cache performance if frequently accessed items don't fit available physical RAM. * The library is thread-safe out of the box. * Concurrent atomic updates. It is safe updating an item from multiple threads while other threads are reading old value for the item under the same key. * 'Add transaction' support allows constructing object's value on the fly without serializing it into a temporary buffer (aka 'zero-copy'). Uncommited transaction can be rolled back at any time. Think of videos streamed directly into the cache without usage of temporary buffers or complex objects requiring serialization from multiple distinct places before storing them into the cache. * Readers and writers don't block each other while reading/writing data from/into the cache. The speed is actually limited by hardware memory bandwidth (if frequently accessed items fit RAM) or backing store random I/O bandwidth (if requently accessed items don't fit RAM). * Instant invalidation of all items in the cache irregardless of cache size. * Optimization for multi-tiered memory hierarchy in modern CPUs. The code avoids unnecessary random memory accesses and tightly packs frequently accessed data in order to reduce working set size and increase CPU cache hit ratio. * The code avoids using dynamic memory allocations in frequently executed paths, so cache performance is almost independent of the efficiency of the provided malloc() implementation. * The code avoids using expensive system calls in frequently executed paths. * The code depends only on OS-supplied libraries. * The code can be easily ported to new platforms. All platform-specific functions and types are hidden behind platform-independent wrappers. * Public interface (ybc.h) is designed with future versions' compatibility in mind. It doesn't expose private structures' contents, so they can be freely modified in the future versions of the library without breaking applications dynamically linked against older versions. * Small library size (can be packed to 22Kb with -Os). =============================================================================== Use-cases. * CDN cache. * File hosting cache. * Memcache-like shared cache. * Per-process local cache in web-servers. * Web-proxy or web-accelerator cache. * Web-browser cache. * Out-of-GC blob cache for programming languages with GC support. ================================================================================ Credits. YBC design is inspired by Varnish's "Notes from the Architect" - https://www.varnish-cache.org/trac/wiki/ArchitectNotes . ================================================================================ FAQ. Q: Give me performance numbers! A: Well, YBC achieves 25.8 Mops/s for get() calls and 5.8 Mops/s for set() calls on my not-so-fast laptop with 2.50GHz Intel i5 CPU when running in 4 threads. Q: Does YBC work with /dev/shm/ ? A: Yes. Just create data and index files on /dev/shm . This will completely eliminate page swapping at the cost of potential data loss on power failure or computer reboot. Q: Is this yet another boring cache implementation? A: No. YBC is designed for modern OSes running on modern hardware. It takes advantage of OS's and computer's hardware features, while simultaneouly avoiding their weaknesses. This results in high performance and low resource usage. Read 'Features' chapter for more details. Q: OK, how to use it in my program? A: 1. Investigate API provided by YBC at ybc.h . See tests/ folder for examples on how to properly use the API. 2. Use this API in your program. 3. Build either object file (ybc-[32|64]-[debug|release].o) or library (libybc-[debug|release].so) using the corresponding build target in the provided Makefile. 4. Link the object file or library against your program. 5. ????? 6. PROFIT!!! Take a look also at bindings/ folder. It contains YBC bindings for various programming languages if you don't like programming in C. Though currently only Go is supported :) Other folders also worth investigation: * libs/ folder contains additional libraries built on top of YBC. * apps/ folder contains applications built on top of YBC. Currently there are the following apps here with self-describing names: * cdn-booster - caching HTTP proxy implementation on top of Go bindings. * cdn-booster-bench - benchmark tool for HTTP/1.1-compliant servers. * memcached - memcache server implementation on top of Go bindings for YBC. Unlike the original memcache server, it supports cache sizes exceeding available RAM by multiple orders of magnitude. It also provides cache persistence, so cached data survives server restarts or server crashes. * memcached-bench - benchmark tool for memcached servers. Makefile already contains build targets for all these apps. Q: Why recently added items may disappear from the cache, while their ttl isn't expired yet? A: Because YBC is a cache, not a storage. They serve different pruposes: - Storage is used for items' persistence. It guarantees that added items exist until they are explicitly deleted. - Cache is used as a performance booster. It doesn't guarantee that added items exist until they are explicitly deleted. YBC may delete any item at any time due to various reasons, which depend on implementation details. So always expect that the added item may disappear at any time during subsequent requests. Q: Can YBC cache small items, not large blobs? A: Yes. YBC works well with small items - it even supports zero-byte key and zero-byte value. Q: Why YBC cannot adjust backing files' size on-demand? I.e. automatically shrink files when the number of items in the cache is small and automatically expand files when the number of items exceeds current files' capacity. A: Because this is stupid idea due to the following reasons: - If cache size isn't bound, then backing files may eat all the available space on storage devices. - Automatic file size adjustment may significantly increase file fragmentation, which will lead to performance degradation. - File size adjustment may lead to arbitrary long delays in requests' serving. - This will complicate the implementation, which may result in more bugs. Q: What's the purpose of cache persistence? A: Cache persistence allows skipping cache warm-up step ater the application restart. This step can be very expensive and time-consuming for large caches and slow backends. You can also make a 'golden' copy of a persistent cache and start every node in the application cluster using their own copy of the 'golden' cache, thus avoiding warm-up step on all nodes in the cluster. Q: How well YBC performance scales on multiple CPUs? A: It scales perfectly if specially designed functions are used - ybc_config_disable_overwrite_protection(), ybc_simple_set() and ybc_simple_get(). Q: Are cache files platform-neutral? A: No. Cache files are tightly tied to the platform they were created. Cache files created on one platform (for example, x86) will be completely broken when read on another platform (for example, x64). Though it is likely they will appear as empty. So copy cache files only between machines with identical platforms.
Hello. I get the following trying to build.
GOPATH="/var/www/toxbid/feeds/pgsql/gopath:/var/www/toxbid/feeds/pgsql" go build -ldflags '-L/var/www/toxbid/feeds/pgsql/gopath/src/github.com/valyala/ybc/ -s' -o /var/www/toxbid/feeds/pgsql/feed-server-pgsql /var/www/toxbid/feeds/pgsql/main.go
github.com/valyala/ybc/bindings/go/ybc
In file included from gopath/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1:0:
gopath/src/github.com/valyala/ybc/bindings/go/ybc/platform.h:240:30: fatal error: platform/linux.c: No such file or directory
compilation terminated.
make: *** [feed-server-pgsql] Error 2
I'm working on a deployment model where I spin up a new process before killing off the old process. I understand ybc doesn't support being opened from multiple processes but I'm wondering if there is a way that I can detect if a cache file is currently open before opening it in the new process.
If not, I'll probably just build my own signaling mechanism. I know sqlite has similar restrictions but does manage to avoid corruption.
Thanks!
Not sure what is going on here...
package main
import (
"encoding/json"
"github.com/valyala/ybc/bindings/go/ybc"
"time"
)
func main() {
cfg := &ybc.Config{
DataFileSize: ybc.SizeT(10240),
MaxItemsCount: ybc.SizeT(100),
IndexFile: "foo.idx",
DataFile: "foo.dat",
}
cache, err := cfg.OpenCache(true)
if err != nil {
panic(err)
}
b, err := json.Marshal(123)
if err != nil {
panic(err)
}
if err = cache.Set([]byte("key"), b, time.Hour); err != nil {
panic(err)
}
}
gives
[adam@scopuli ~]$ go run test.go
panic: runtime error: cgo argument has Go pointer to Go pointer
goroutine 1 [running]:
panic(0x5123a0, 0xc82000a430)
/usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/valyala/ybc/bindings/go/ybc.(*Cache).Set(0xc82000e2e0, 0xc82000a410, 0x3, 0x8, 0xc820082024, 0x3, 0x40, 0x34630b8a000, 0x0, 0x0)
/home/adam/go/src/github.com/valyala/ybc/bindings/go/ybc/ybc.go:457 +0x385
main.main()
/home/adam/test.go:26 +0x226
exit status 2
However, this works fine:
cache.Set([]byte("key"), []byte(string(b)), time.Hour);
Running:
go version go1.6 linux/amd64
and ybc 82bc72dc617945514a43056c92c023f65e045ae6
As of go 1.11 the bindings no longer build:
go get -u github.com/valyala/ybc/bindings/go/ybc
go: finding github.com/valyala/ybc/bindings/go/ybc latest
go: finding github.com/valyala/ybc/bindings/go latest
go: finding github.com/valyala/ybc/bindings latest
go: finding github.com/valyala/ybc latest
# github.com/valyala/ybc/bindings/go/ybc
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:663:7: could not determine kind of name for C.YBC_DE_NOTFOUND
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:667:7: could not determine kind of name for C.YBC_DE_SUCCESS
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:659:7: could not determine kind of name for C.YBC_DE_WOULDBLOCK
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:483:5: could not determine kind of name for C.go_item_set
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:326:39: could not determine kind of name for C.uint64_t
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:712:2: could not determine kind of name for C.ybc_clear
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:469:2: could not determine kind of name for C.ybc_close
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:254:8: could not determine kind of name for C.ybc_config_destroy
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:329:3: could not determine kind of name for C.ybc_config_disable_overwrite_protection
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:56:19: could not determine kind of name for C.ybc_config_get_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:296:2: could not determine kind of name for C.ybc_config_init
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:313:3: could not determine kind of name for C.ybc_config_set_data_file
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:302:3: could not determine kind of name for C.ybc_config_set_data_file_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:319:3: could not determine kind of name for C.ybc_config_set_de_hashtable_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:317:2: could not determine kind of name for C.ybc_config_set_hot_data_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:316:2: could not determine kind of name for C.ybc_config_set_hot_items_count
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:307:3: could not determine kind of name for C.ybc_config_set_index_file
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:299:3: could not determine kind of name for C.ybc_config_set_max_items_count
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:326:3: could not determine kind of name for C.ybc_config_set_sync_interval
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:57:19: could not determine kind of name for C.ybc_get_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:59:19: could not determine kind of name for C.ybc_item_get_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:859:2: could not determine kind of name for C.ybc_item_release
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:272:5: could not determine kind of name for C.ybc_open
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:287:2: could not determine kind of name for C.ybc_remove
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:753:2: could not determine kind of name for C.ybc_set_txn_commit
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:58:19: could not determine kind of name for C.ybc_set_txn_get_size
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:832:3: could not determine kind of name for C.ybc_set_txn_get_value
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:761:2: could not determine kind of name for C.ybc_set_txn_rollback
../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:819:2: could not determine kind of name for C.ybc_set_txn_update_value_size
cgo:
gcc errors for preamble:
In file included from ./ybc_go_glue.c:1:0,
from ../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:19:
./ybc.h:1:1: error: expected identifier or '(' before '.' token
../../../ybc.h
^
In file included from ../go/pkg/mod/github.com/valyala/[email protected]/bindings/go/ybc/ybc.go:19:0:
./ybc_go_glue.c:21:38: error: unknown type name 'size_t'
const void *const key_ptr, const size_t key_size)
^
./ybc_go_glue.c: In function 'go_item_remove':
./ybc_go_glue.c:23:16: error: variable 'key' has initializer but incomplete type
const struct ybc_key key = {
^
./ybc_go_glue.c:24:5: error: unknown field 'ptr' specified in initializer
.ptr = key_ptr,
^
...
If I manually disable the module support, it will work:
GO111MODULE=off -u github.com/valyala/ybc/bindings/go/ybc
But this means I cannot use go 1.11 module vendoring support. I believe this has to do with the way source data is cached. The go tool isn't copying across the required c source files for ybc when it uses module logic.
It appears the go ybc bindings break the go1.6 cgo rules.
panic: runtime error: cgo argument has Go pointer to Go pointer
goroutine 1 [running]:
github.com/valyala/ybc/bindings/go/ybc.(*Cache).GetItem(0xc82000e510, 0xc82000a3a0, 0x3, 0x8, 0xc820012300, 0x0, 0x0)
/home/shansen/src/github.com/valyala/ybc/bindings/go/ybc/ybc.go:563 +0x409
github.com/valyala/ybc/bindings/go/ybc.(*Cache).Get(0xc82000e510, 0xc82000a3a0, 0x3, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/shansen/src/github.com/valyala/ybc/bindings/go/ybc/ybc.go:468 +0x6a
main.main()
/home/shansen/$snip/main.go:18 +0x13c
exit status 2
Source:
package main
import (
"github.com/valyala/ybc/bindings/go/ybc"
)
func main() {
cfg := &ybc.Config{
DataFileSize: ybc.SizeT(10240),
MaxItemsCount: ybc.SizeT(100),
IndexFile: "foo.idx",
DataFile: "foo.dat",
}
cache, err := cfg.OpenCache(true)
if err != nil {
panic(err)
}
if _, err = cache.Get([]byte("foo")); err != nil {
panic(err)
}
}
I get the following trying to install/build:
go get -u -a github.com/valyala/ybc/libs/go/memcache
github.com/valyala/ybc/bindings/go/ybc
In file included from /usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform.h:240,
from /usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1:
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform/linux.c: In function ‘p_file_create’:
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform/linux.c:362: error: ‘O_CLOEXEC’ undeclared (first use in this function)
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform/linux.c:362: error: (Each undeclared identifier is reported only once
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform/linux.c:362: error: for each function it appears in.)
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform/linux.c: In function ‘p_file_open’:
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform/linux.c:382: error: ‘O_CLOEXEC’ undeclared (first use in this function)
Hi... I thought I should report that I've experienced what seem like a corner case in the handling of requests blocked by dogpile effect handling.
Due to the polling of C_DE_ITEM_SLEEP_TIME intervals it's possible that the entry being waited on to appear in the cache and get evicted again between two polls - making the blocked request wait for the full grace_ttl.
This of course all depends on load and cache tuning.
When I execute this command go get -u github.com/valyala/ybc/bindings/go/ybc
I get the next result(MAC 10.10.1) :
gcc did not produce error at completed:1
on input:
/* Define intgo when compiling with GCC. */
typedef ptrdiff_t intgo;
typedef struct { char *p; intgo n; } GoString;
typedef struct { char *p; intgo n; intgo c; } GoBytes;
GoString GoString(char *p);
GoString GoStringN(char *p, int l);
GoBytes GoBytes(void *p, int n);
char *CString(GoString);
void *_CMalloc(size_t);
void cgo_f_1_1(void) { __typeof(CString) *cgo_undefined; }
void *cgo_f_1_2(void) { CString __cgo_undefined; }
void cgo_f_1_3(void) { enum { __cgo__undefined = (CString)*1 }; }
void cgo_f_2_1(void) { __typeof(GoBytes) *cgo_undefined; }
void *cgo_f_2_2(void) { GoBytes __cgo_undefined; }
void cgo_f_2_3(void) { enum { __cgo__undefined = (GoBytes)*1 }; }
void cgo_f_3_1(void) { __typeof(YBC_DE_NOTFOUND) *cgo_undefined; }
void *cgo_f_3_2(void) { YBC_DE_NOTFOUND __cgo_undefined; }
void cgo_f_3_3(void) { enum { __cgo__undefined = (YBC_DE_NOTFOUND)*1 }; }
void cgo_f_4_1(void) { __typeof(YBC_DE_SUCCESS) *cgo_undefined; }
void *cgo_f_4_2(void) { YBC_DE_SUCCESS __cgo_undefined; }
void cgo_f_4_3(void) { enum { __cgo__undefined = (YBC_DE_SUCCESS)*1 }; }
void cgo_f_5_1(void) { __typeof(YBC_DE_WOULDBLOCK) *cgo_undefined; }
void *cgo_f_5_2(void) { YBC_DE_WOULDBLOCK __cgo_undefined; }
void cgo_f_5_3(void) { enum { __cgo__undefined = (YBC_DE_WOULDBLOCK)*1 }; }
void cgo_f_6_1(void) { __typeof(free) *cgo_undefined; }
void *cgo_f_6_2(void) { free __cgo_undefined; }
void cgo_f_6_3(void) { enum { __cgo__undefined = (free)*1 }; }
void cgo_f_7_1(void) { __typeof(go_commit_item_and_value) *cgo_undefined; }
void *cgo_f_7_2(void) { go_commit_item_and_value __cgo_undefined; }
void cgo_f_7_3(void) { enum { __cgo__undefined = (go_commit_item_and_value)*1 }; }
void cgo_f_8_1(void) { __typeof(go_get_item_and_value) *cgo_undefined; }
void *cgo_f_8_2(void) { go_get_item_and_value __cgo_undefined; }
void cgo_f_8_3(void) { enum { __cgo__undefined = (go_get_item_and_value)*1 }; }
void cgo_f_9_1(void) { __typeof(go_get_item_and_value_de_async) *cgo_undefined; }
void *cgo_f_9_2(void) { go_get_item_and_value_de_async __cgo_undefined; }
void cgo_f_9_3(void) { enum { __cgo__undefined = (go_get_item_and_value_de_async)*1 }; }
void cgo_f_10_1(void) { __typeof(go_set_item_and_value) *cgo_undefined; }
void *cgo_f_10_2(void) { go_set_item_and_value __cgo_undefined; }
void cgo_f_10_3(void) { enum { __cgo__undefined = (go_set_item_and_value)*1 }; }
void cgo_f_11_1(void) { __typeof(int) *cgo_undefined; }
void *cgo_f_11_2(void) { int __cgo_undefined; }
void cgo_f_11_3(void) { enum { __cgo__undefined = (int)*1 }; }
void cgo_f_12_1(void) { __typeof(size_t) *cgo_undefined; }
void *cgo_f_12_2(void) { size_t __cgo_undefined; }
void cgo_f_12_3(void) { enum { __cgo__undefined = (size_t)*1 }; }
void cgo_f_13_1(void) { __typeof(uint64_t) *cgo_undefined; }
void *cgo_f_13_2(void) { uint64_t __cgo_undefined; }
void cgo_f_13_3(void) { enum { __cgo__undefined = (uint64_t)*1 }; }
void cgo_f_14_1(void) { __typeof(ybc_clear) *cgo_undefined; }
void *cgo_f_14_2(void) { ybc_clear __cgo_undefined; }
void cgo_f_14_3(void) { enum { __cgo__undefined = (ybc_clear)*1 }; }
void cgo_f_15_1(void) { __typeof(ybc_close) *cgo_undefined; }
void *cgo_f_15_2(void) { ybc_close __cgo_undefined; }
void cgo_f_15_3(void) { enum { __cgo__undefined = (ybc_close)*1 }; }
void cgo_f_16_1(void) { __typeof(ybc_config_destroy) *cgo_undefined; }
void *cgo_f_16_2(void) { ybc_config_destroy __cgo_undefined; }
void cgo_f_16_3(void) { enum { __cgo__undefined = (ybc_config_destroy)*1 }; }
void cgo_f_17_1(void) { __typeof(ybc_config_disable_overwrite_protection) *cgo_undefined; }
void *cgo_f_17_2(void) { ybc_config_disable_overwrite_protection __cgo_undefined; }
void cgo_f_17_3(void) { enum { __cgo__undefined = (ybc_config_disable_overwrite_protection)*1 }; }
void cgo_f_18_1(void) { __typeof(ybc_config_get_size) *cgo_undefined; }
void *cgo_f_18_2(void) { ybc_config_get_size __cgo_undefined; }
void cgo_f_18_3(void) { enum { __cgo__undefined = (ybc_config_get_size)*1 }; }
void cgo_f_19_1(void) { __typeof(ybc_config_init) *cgo_undefined; }
void *cgo_f_19_2(void) { ybc_config_init __cgo_undefined; }
void cgo_f_19_3(void) { enum { __cgo__undefined = (ybc_config_init)*1 }; }
void cgo_f_20_1(void) { __typeof(ybc_config_set_data_file) *cgo_undefined; }
void *cgo_f_20_2(void) { ybc_config_set_data_file __cgo_undefined; }
void cgo_f_20_3(void) { enum { __cgo__undefined = (ybc_config_set_data_file)*1 }; }
void cgo_f_21_1(void) { __typeof(ybc_config_set_data_file_size) *cgo_undefined; }
void *cgo_f_21_2(void) { ybc_config_set_data_file_size __cgo_undefined; }
void cgo_f_21_3(void) { enum { __cgo__undefined = (ybc_config_set_data_file_size)*1 }; }
void cgo_f_22_1(void) { __typeof(ybc_config_set_de_hashtable_size) *cgo_undefined; }
void *cgo_f_22_2(void) { ybc_config_set_de_hashtable_size __cgo_undefined; }
void cgo_f_22_3(void) { enum { __cgo__undefined = (ybc_config_set_de_hashtable_size)*1 }; }
void cgo_f_23_1(void) { __typeof(ybc_config_set_hot_data_size) *cgo_undefined; }
void *cgo_f_23_2(void) { ybc_config_set_hot_data_size __cgo_undefined; }
void cgo_f_23_3(void) { enum { __cgo__undefined = (ybc_config_set_hot_data_size)*1 }; }
void cgo_f_24_1(void) { __typeof(ybc_config_set_hot_items_count) *cgo_undefined; }
void *cgo_f_24_2(void) { ybc_config_set_hot_items_count __cgo_undefined; }
void cgo_f_24_3(void) { enum { __cgo__undefined = (ybc_config_set_hot_items_count)*1 }; }
void cgo_f_25_1(void) { __typeof(ybc_config_set_index_file) *cgo_undefined; }
void *cgo_f_25_2(void) { ybc_config_set_index_file __cgo_undefined; }
void cgo_f_25_3(void) { enum { __cgo__undefined = (ybc_config_set_index_file)*1 }; }
void cgo_f_26_1(void) { __typeof(ybc_config_set_max_items_count) *cgo_undefined; }
void *cgo_f_26_2(void) { ybc_config_set_max_items_count __cgo_undefined; }
void cgo_f_26_3(void) { enum { __cgo__undefined = (ybc_config_set_max_items_count)*1 }; }
void cgo_f_27_1(void) { __typeof(ybc_config_set_sync_interval) *cgo_undefined; }
void *cgo_f_27_2(void) { ybc_config_set_sync_interval __cgo_undefined; }
void cgo_f_27_3(void) { enum { __cgo__undefined = (ybc_config_set_sync_interval)*1 }; }
void cgo_f_28_1(void) { __typeof(ybc_get_size) *cgo_undefined; }
void *cgo_f_28_2(void) { ybc_get_size __cgo_undefined; }
void cgo_f_28_3(void) { enum { __cgo__undefined = (ybc_get_size)*1 }; }
void cgo_f_29_1(void) { __typeof(ybc_item_get_size) *cgo_undefined; }
void *cgo_f_29_2(void) { ybc_item_get_size __cgo_undefined; }
void cgo_f_29_3(void) { enum { __cgo__undefined = (ybc_item_get_size)*1 }; }
void cgo_f_30_1(void) { __typeof(ybc_item_release) *cgo_undefined; }
void *cgo_f_30_2(void) { ybc_item_release __cgo_undefined; }
void cgo_f_30_3(void) { enum { __cgo__undefined = (ybc_item_release)*1 }; }
void cgo_f_31_1(void) { __typeof(ybc_item_remove) *cgo_undefined; }
void *cgo_f_31_2(void) { ybc_item_remove __cgo_undefined; }
void cgo_f_31_3(void) { enum { __cgo__undefined = (ybc_item_remove)*1 }; }
void cgo_f_32_1(void) { __typeof(ybc_item_set) *cgo_undefined; }
void *cgo_f_32_2(void) { ybc_item_set __cgo_undefined; }
void cgo_f_32_3(void) { enum { __cgo__undefined = (ybc_item_set)*1 }; }
void cgo_f_33_1(void) { __typeof(ybc_open) *cgo_undefined; }
void *cgo_f_33_2(void) { ybc_open __cgo_undefined; }
void cgo_f_33_3(void) { enum { __cgo__undefined = (ybc_open)*1 }; }
void cgo_f_34_1(void) { __typeof(ybc_remove) *cgo_undefined; }
void *cgo_f_34_2(void) { ybc_remove __cgo_undefined; }
void cgo_f_34_3(void) { enum { __cgo__undefined = (ybc_remove)*1 }; }
void cgo_f_35_1(void) { __typeof(ybc_set_txn_begin) *cgo_undefined; }
void *cgo_f_35_2(void) { ybc_set_txn_begin __cgo_undefined; }
void cgo_f_35_3(void) { enum { __cgo__undefined = (ybc_set_txn_begin)*1 }; }
void cgo_f_36_1(void) { __typeof(ybc_set_txn_commit) *cgo_undefined; }
void *cgo_f_36_2(void) { ybc_set_txn_commit __cgo_undefined; }
void cgo_f_36_3(void) { enum { __cgo__undefined = (ybc_set_txn_commit)*1 }; }
void cgo_f_37_1(void) { __typeof(ybc_set_txn_get_size) *cgo_undefined; }
void *cgo_f_37_2(void) { ybc_set_txn_get_size __cgo_undefined; }
void cgo_f_37_3(void) { enum { __cgo__undefined = (ybc_set_txn_get_size)*1 }; }
void cgo_f_38_1(void) { __typeof(ybc_set_txn_get_value) *cgo_undefined; }
void *cgo_f_38_2(void) { ybc_set_txn_get_value __cgo_undefined; }
void cgo_f_38_3(void) { enum { __cgo__undefined = (ybc_set_txn_get_value)*1 }; }
void cgo_f_39_1(void) { __typeof(ybc_set_txn_rollback) *cgo_undefined; }
void *cgo_f_39_2(void) { ybc_set_txn_rollback __cgo_undefined; }
void cgo_f_39_3(void) { enum { __cgo__undefined = (ybc_set_txn_rollback)*1 }; }
void cgo_f_40_1(void) { __typeof(ybc_set_txn_update_value_size) *cgo_undefined; }
void *cgo_f_40_2(void) { ybc_set_txn_update_value_size __cgo_undefined; }
void cgo_f_40_3(void) { enum { __cgo__undefined = (ybc_set_txn_update_value_size)*1 }; }
void cgo_f_41_1(void) { __typeof(ybc_simple_get) *cgo_undefined; }
void *cgo_f_41_2(void) { ybc_simple_get __cgo_undefined; }
void cgo_f_41_3(void) { enum { __cgo__undefined = (ybc_simple_get)*1 }; }
void cgo_f_42_1(void) { __typeof(ybc_simple_set) *cgo_undefined; }
void *cgo_f_42_2(void) { ybc_simple_set __cgo_undefined; }
void cgo_f_42_3(void) { enum { __cgo__undefined = (ybc_simple_set)*1 }; }
int __cgo__1 = __cgo__2;
full error output:
not-type:1:35: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_1_2(void) { CString __cgo_undefined; }
^
not-const:1:63: error: invalid operands to binary expression ('char ()(GoString)' and 'int')
void cgo_f_1_3(void) { enum { __cgo__undefined = (CString)1 }; }
~~~~~~~~~^~
not-type:2:35: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_2_2(void) { GoBytes __cgo_undefined; }
^
not-const:2:63: error: invalid operands to binary expression ('GoBytes ()(void _, int)' and 'int')
void cgo_f_2_3(void) { enum { __cgo__undefined = (GoBytes)1 }; }
~~~~~~~~~^~
not-type:3:43: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_3_2(void) { YBC_DE_NOTFOUND __cgo_undefined; }
^
not-type:4:42: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_4_2(void) { YBC_DE_SUCCESS __cgo_undefined; }
^
not-type:5:45: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_5_2(void) { YBC_DE_WOULDBLOCK __cgo_undefined; }
^
not-type:6:32: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_6_2(void) { free __cgo_undefined; }
^
not-const:6:60: error: invalid operands to binary expression ('void ()(void *)' and 'int')
void cgo_f_6_3(void) { enum { __cgo__undefined = (free)1 }; }
~~~~~~^~
not-type:7:52: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_7_2(void) { go_commit_item_and_value __cgo_undefined; }
^
not-const:7:80: error: invalid operands to binary expression ('void ()(struct ybc_set_txn *const, struct ybc_item *const, struct ybc_value *const)' and 'int')
void cgo_f_7_3(void) { enum { __cgo__undefined = (go_commit_item_and_value)1 }; }
~~~~~~~~~~~~~~~~~~~~~~~~~~^~
not-type:8:49: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_8_2(void) { go_get_item_and_value __cgo_undefined; }
^
not-const:8:77: error: invalid operands to binary expression ('int ()(struct ybc *const, struct ybc_item *const, struct ybc_value *const, const struct ybc_key *const)' and 'int')
void cgo_f_8_3(void) { enum { __cgo__undefined = (go_get_item_and_value)1 }; }
~~~~~~~~~~~~~~~~~~~~~~~^~
not-type:9:58: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_9_2(void) { go_get_item_and_value_de_async __cgo_undefined; }
^
not-const:9:86: error: invalid operands to binary expression ('enum ybc_de_status ()(struct ybc *const, struct ybc_item *const, struct ybc_value *const, const struct ybc_key *const, const uint64_t)' and 'int')
void cgo_f_9_3(void) { enum { __cgo__undefined = (go_get_item_and_value_de_async)1 }; }
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
not-type:10:50: error: use of undeclared identifier 'cgo_undefined'
void *cgo_f_10_2(void) { go_set_item_and_value __cgo_undefined; }
^
not-const:10:78: error: invalid operands to binary expression ('int ()(struct ybc *const, struct ybc_item *const, const struct ybc_key *const, struct ybc_value *const)' and 'int')
void cgo_f_10_3(void) { enum { __cgo__undefined = (go_set_item_and_value)_1 }; }
~~~~~~~~~~~~~~~~~~~~~~~^~
not-const:11:60: error: indirection requires pointer operand ('int' invalid)
void cgo_f_11_3(void) { enum { __cgo__undefined = (int)_1 }; }
^~
not-const:12:63: error: indirection requires pointer operand ('int' invalid)
void cgo_f_12_3(void) { enum { __cgo__undefined = (size_t)_1 }; }
^~
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
I'm try rebuild pkg with go build -a (rebuild all packages) and get error like:
../go/src/github.com/valyala/ybc/bindings/go/ybc/debugguard_devel.go:81: undefined: errPanic
from the readme..
YBC may delete any item at any time due to various reasons, which depend on implementation details. So always expect that the added item may disappear at any time during subsequent requests.
Can I know which reasons apart from cache size getting large would lead to evictions? Specifically pertaining to the Go binding.
In my usecase, I would be using ybc as a hybrid between a kv store and a cache. i.e.
Currently I use redis, and am looking to use something in process.
Sorry, this is not an issue but a question. I dont know which is the right mailing list to ask.
Just wanted you to know as a side project im trying to build another reverse proxy inspired by cdn-booster
https://github.com/sajal/gohttpcache/blob/master/proxy.go
Im doing it just for fun... Its a very very rough draft at the moment and probably does not perform too well.
Some things I do differently.
Hi,
I recently read the implement of ybc's replacement policy. It seems to me that it will find the item with the smallest expire time as a victim. My question is that if all item is not expired, say I set all the ttl to maxttl, will it has performance issue if the total number of items is large? It seems to me that it will loop through all the items to find the item with the smallest expire time for every set operation.
I want to get all the data stored in ybc. How can I get them without putting key somewhere else?
I've got a server with three instances where on every one is running YBC. the problem that I have is when updated one record on the cache on YBC, only one instance is updated and the other two the data is deleted.
Someone can help me, or explain me what is the reason of this problem.
Thanks in advance
Basically I am curious if anyone is using YBC in production?
in the cdn,i want to clear the a.com cache.
the keys like this, and how to clear?
cache:a.com:get:/index.html
cache:a.com:get:/js/1.js
cache:a.com:get:/js/2.css
...
cache:b.com:get:/index.html
...
/home/kane/mygo/pkg/linux_386/github.com/valyala/ybc/bindings/go/ybc.a(ybc.o)(.text): __udivdi3: not defined
/home/kane/mygo/pkg/linux_386/github.com/valyala/ybc/bindings/go/ybc.a(ybc.o)(.text): __umoddi3: not defined
/home/kane/mygo/pkg/linux_386/github.com/valyala/ybc/bindings/go/ybc.a(ybc.o)(.text): __umoddi3: not defined
/home/kane/mygo/pkg/linux_386/github.com/valyala/ybc/bindings/go/ybc.a(ybc.o)(.text): __umoddi3: not defined
/home/kane/mygo/pkg/linux_386/github.com/valyala/ybc/bindings/go/ybc.a(ybc.o)(.text): __umoddi3: not defined
__umoddi3(0): not defined
__udivdi3(0): not defined
Running Go 1.4.1 on Mac OS 10.7.5.
I get the following error trying to get/build:
go get github.com/valyala/ybc/libs/go/memcache
In file included from /usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1:
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/platform.h:242:4: error: #error "unsupported platform"
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_hash_get':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:57: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_item_skiplist_init':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:292: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_item_skiplist_destroy':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:312: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_item_skiplist_get_prevs':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:339: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_item_skiplist_add':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:356: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_item_skiplist_del':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:380: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_item_skiplist_relocate':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:396: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_map_lookup_slot_index':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1014: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: At top level:
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1408: error: field 'stop_event' has incomplete type
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1413: error: field 'sync_thread' has incomplete type
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_de_init':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1626: error: dereferencing pointer to incomplete type
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1630: error: invalid use of undefined type 'struct p_lock'
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1632: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1633: error: invalid use of undefined type 'struct p_lock'
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_de_destroy':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1655: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1657: error: invalid use of undefined type 'struct p_lock'
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_de_item_register':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1716: error: invalid use of undefined type 'struct p_lock'
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1728: error: invalid use of undefined type 'struct p_lock'
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: At top level:
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1839: error: field 'lock' has incomplete type
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1840: error: field 'index_file' has incomplete type
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:1841: error: field 'storage_file' has incomplete type
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'm_cluster_close_caches':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:2435: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'ybc_cluster_open':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:2468: error: 'for' loop initial declaration used outside C99 mode
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c: In function 'ybc_cluster_clear':
/usr/local/go/bin/src/github.com/valyala/ybc/bindings/go/ybc/ybc.c:2543: error: 'for' loop initial declaration used outside C99 mode
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.