Code Monkey home page Code Monkey logo

limiter's Introduction

Limiter

Documentation License Build Status Go Report Card

Dead simple rate limit middleware for Go.

  • Simple API
  • "Store" approach for backend
  • Redis support (but not tied too)
  • Middlewares: HTTP, FastHTTP and Gin

Installation

Using Go Modules

$ go get github.com/ulule/limiter/[email protected]

Usage

In five steps:

  • Create a limiter.Rate instance (the number of requests per period)
  • Create a limiter.Store instance (see Redis or In-Memory)
  • Create a limiter.Limiter instance that takes store and rate instances as arguments
  • Create a middleware instance using the middleware of your choice
  • Give the limiter instance to your middleware initializer

Example:

// Create a rate with the given limit (number of requests) for the given
// period (a time.Duration of your choice).
import "github.com/ulule/limiter/v3"

rate := limiter.Rate{
    Period: 1 * time.Hour,
    Limit:  1000,
}

// You can also use the simplified format "<limit>-<period>"", with the given
// periods:
//
// * "S": second
// * "M": minute
// * "H": hour
// * "D": day
//
// Examples:
//
// * 5 reqs/second: "5-S"
// * 10 reqs/minute: "10-M"
// * 1000 reqs/hour: "1000-H"
// * 2000 reqs/day: "2000-D"
//
rate, err := limiter.NewRateFromFormatted("1000-H")
if err != nil {
    panic(err)
}

// Then, create a store. Here, we use the bundled Redis store. Any store
// compliant to limiter.Store interface will do the job. The defaults are
// "limiter" as Redis key prefix and a maximum of 3 retries for the key under
// race condition.
import "github.com/ulule/limiter/v3/drivers/store/redis"

store, err := redis.NewStore(client)
if err != nil {
    panic(err)
}

// Alternatively, you can pass options to the store with the "WithOptions"
// function. For example, for Redis store:
import "github.com/ulule/limiter/v3/drivers/store/redis"

store, err := redis.NewStoreWithOptions(pool, limiter.StoreOptions{
    Prefix:   "your_own_prefix",
})
if err != nil {
    panic(err)
}

// Or use a in-memory store with a goroutine which clears expired keys.
import "github.com/ulule/limiter/v3/drivers/store/memory"

store := memory.NewStore()

// Then, create the limiter instance which takes the store and the rate as arguments.
// Now, you can give this instance to any supported middleware.
instance := limiter.New(store, rate)

// Alternatively, you can pass options to the limiter instance with several options.
instance := limiter.New(store, rate, limiter.WithClientIPHeader("True-Client-IP"), limiter.WithIPv6Mask(mask))

// Finally, give the limiter instance to your middleware initializer.
import "github.com/ulule/limiter/v3/drivers/middleware/stdlib"

middleware := stdlib.NewMiddleware(instance)

See middleware examples:

How it works

The ip address of the request is used as a key in the store.

If the key does not exist in the store we set a default value with an expiration period.

You will find two stores:

  • Redis: rely on TTL and incrementing the rate limit on each request.
  • In-Memory: rely on a fork of go-cache with a goroutine to clear expired keys using a default interval.

When the limit is reached, a 429 HTTP status code is sent.

Limiter behind a reverse proxy

Introduction

If your limiter is behind a reverse proxy, it could be difficult to obtain the "real" client IP.

Some reverse proxies, like AWS ALB, lets all header values through that it doesn't set itself. Like for example, True-Client-IP and X-Real-IP. Similarly, X-Forwarded-For is a list of comma-separated IPs that gets appended to by each traversed proxy. The idea is that the first IP (added by the first proxy) is the true client IP. Each subsequent IP is another proxy along the path.

An attacker can spoof either of those headers, which could be reported as a client IP.

By default, limiter doesn't trust any of those headers: you have to explicitly enable them in order to use them. If you enable them, you must always be aware that any header added by any (reverse) proxy not controlled by you are completely unreliable.

X-Forwarded-For

For example, if you make this request to your load balancer:

curl -X POST https://example.com/login -H "X-Forwarded-For: 1.2.3.4, 11.22.33.44"

And your server behind the load balancer obtain this:

X-Forwarded-For: 1.2.3.4, 11.22.33.44, <actual client IP>

That's mean you can't use X-Forwarded-For header, because it's unreliable and untrustworthy. So keep TrustForwardHeader disabled in your limiter option.

However, if you have configured your reverse proxy to always remove/overwrite X-Forwarded-For and/or X-Real-IP headers so that if you execute this (same) request:

curl -X POST https://example.com/login -H "X-Forwarded-For: 1.2.3.4, 11.22.33.44"

And your server behind the load balancer obtain this:

X-Forwarded-For: <actual client IP>

Then, you can enable TrustForwardHeader in your limiter option.

Custom header

Many CDN and Cloud providers add a custom header to define the client IP. Like for example, this non exhaustive list:

  • Fastly-Client-IP from Fastly
  • CF-Connecting-IP from Cloudflare
  • X-Azure-ClientIP from Azure

You can use these headers using ClientIPHeader in your limiter option.

None of the above

If none of the above solution are working, please use a custom KeyGetter in your middleware.

You can use this excellent article to help you define the best strategy depending on your network topology and your security need: https://adam-p.ca/blog/2022/03/x-forwarded-for/

If you have any idea/suggestions on how we could simplify this steps, don't hesitate to raise an issue. We would like some feedback on how we could implement this steps in the Limiter API.

Thank you.

Why Yet Another Package

You could ask us: why yet another rate limit package?

Because existing packages did not suit our needs.

We tried a lot of alternatives:

  1. Throttled. This package uses the generic cell-rate algorithm. To cite the documentation: "The algorithm has been slightly modified from its usual form to support limiting with an additional quantity parameter, such as for limiting the number of bytes uploaded". It is brillant in term of algorithm but documentation is quite unclear at the moment, we don't need burst feature for now, impossible to get a correct After-Retry (when limit exceeds, we can still make a few requests, because of the max burst) and it only supports http.Handler middleware (we use Gin). Currently, we only need to return 429 and X-Ratelimit-* headers for n reqs/duration.

  2. Speedbump. Good package but maybe too lightweight. No Reset support, only one middleware for Gin framework and too Redis-coupled. We rather prefer to use a "store" approach.

  3. Tollbooth. Good one too but does both too much and too little. It limits by remote IP, path, methods, custom headers and basic auth usernames... but does not provide any Redis support (only in-memory) and a ready-to-go middleware that sets X-Ratelimit-* headers. tollbooth.LimitByRequest(limiter, r) only returns an HTTP code.

  4. ratelimit. Probably the closer to our needs but, once again, too lightweight, no middleware available and not active (last commit was in August 2014). Some parts of code (Redis) comes from this project. It should deserve much more love.

There are other many packages on GitHub but most are either too lightweight, too old (only support old Go versions) or unmaintained. So that's why we decided to create yet another one.

Contributing

Don't hesitate ;)

limiter's People

Contributors

akdilsiz avatar andrewhoff avatar dependabot-preview[bot] avatar dependabot[bot] avatar dougnukem avatar gadelkareem avatar gillesfabio avatar git-hulk avatar harry-027 avatar hiyali avatar kindermoumoute avatar kolaente avatar manavo avatar micanzhang avatar novln avatar patwie avatar pyjac avatar radisson avatar shareed2k avatar stphnrdmr avatar thoas avatar twmbx avatar yansal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

limiter's Issues

GetIP TrustForwardHeader varies per request

The recent change in

limiter/network.go

Lines 17 to 39 in 4499266

func (limiter *Limiter) GetIP(r *http.Request) net.IP {
if limiter.Options.TrustForwardHeader {
ip := r.Header.Get("X-Forwarded-For")
if ip != "" {
parts := strings.SplitN(ip, ",", 2)
part := strings.TrimSpace(parts[0])
return net.ParseIP(part)
}
ip = strings.TrimSpace(r.Header.Get("X-Real-IP"))
if ip != "" {
return net.ParseIP(ip)
}
}
remoteAddr := strings.TrimSpace(r.RemoteAddr)
host, _, err := net.SplitHostPort(remoteAddr)
if err != nil {
return net.ParseIP(remoteAddr)
}
return net.ParseIP(host)
}
conflicts with the nature of the limiter. Since the limiter object is instantiated early before the server stats and then shared among different requests whose IPs are different. IMHO the TrustForwardHeader should vary based on how much the server trusts the IP and should not be as a config rule.

Also the change of GetIP is not backward compatible - AKA broke my build :)

The old signature used to be

func GetIP(r *http.Request, trustForwardHeader bool) net.IP 

http.Client rate limited http.Transport

Overview

It'd be nice if this library supported a way for http.Client outgoing requests to utilizing limiter.Limit to rate limit based on configured API rate limits useful for things like:

Features

  • Filter and intercept http.Client requests and map to a limit.Limiter which could then have some policy callback to let the client
    • block and wait until reset time
    • abort request and send error back
      • maybe fake an HTTP rate-limit response (but avoid hitting actual servers)
  • HTTP API's usually return with headers like X-RateLimit-* it'd be useful if there was a way to call limiter.Limiter.Set(key string, c *Context) to attempt to keep in sync with server state

I think you'd want to be able to configure a limit.Limiter per URL request path e.g.:

  • /1.1/users/user_timeline.json

Similar to how server-side HTTP handlers are setup:

mux.HandleFunc("/1.1/statuses/user_timeline.json", func(r *http.Request) error {
    c, err := userTimelineLimiter.Get("user_timeline.json")
    if err != nil {
      return err
    }

    if c.Reached {
      // client policy:
      // 1. wait and retry
      // 2. error rate limit
      // 3. simulate http response error with rate limit
      err := handleRatelimitReached(r)
    }
})

Maybe even use http.NewServeMux to client-side proxy rate limit responses?

There's some example of it here:

// RateLimitedTransport is used to conform to rate limits that are
// communicated through "X-RateLimit-" headers, like GitHub's API. It
// implements http.RoundTripper and can be used for configuring a http.Client.
type RateLimitedTransport struct {
    Base http.RoundTripper
}

func (t *RateLimitedTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    res, err := t.base().RoundTrip(req)
    if err != nil {
        return res, err
    }

    // Fetch headers
    remStr := res.Header.Get("X-RateLimit-Remaining")
    if remStr == "" {
        return res, err
    }
    resetStr := res.Header.Get("X-RateLimit-Reset")
    if resetStr == "" {
        return res, err
    }

    rem, err := strconv.Atoi(remStr)
    if err != nil {
        return res, err
    }
    epoch, err := strconv.ParseInt(resetStr, 10, 64)
    if err != nil {
        return res, err
    }
    reset := time.Unix(epoch, 0)

    // Determine sleep time
    untilReset := reset.Sub(time.Now())
    delay := time.Duration(float64(untilReset) / (float64(rem) + 1))
    time.Sleep(delay)

    return res, err
}

Should it be more convenient to use multiple Rates at the same time?

I want to rate limit my API's login endpoint thusly: max 1 request per second, max 5 requests per hour, max 10 requests per day.

I have implemented it like this using your middleware:

func loginLimiterHandler() http.Handler {
	secondRate := limiter.Rate{Period: 1 * time.Second, Limit: 1}
	hourRate := limiter.Rate{Period: 1 * time.Hour, Limit: 5}
	dayRate := limiter.Rate{Period: 24 * time.Hour, Limit: 10}

	secondLimit := limiter.New(memory.NewStore(), secondRate)
	hourLimit := limiter.New(memory.NewStore(), hourRate)
	dayLimit := limiter.New(memory.NewStore(), dayRate)

	var handler http.Handler = http.HandlerFunc(LoginHandler)

	handler = stdlib.NewMiddleware(dayLimit).Handler(handler)
	handler = stdlib.NewMiddleware(hourLimit).Handler(handler)
	handler = stdlib.NewMiddleware(secondLimit).Handler(handler)

	return handler
}

I think it would be practical to add the function limiter.NewMultipleRates (or similar) to allow this:

func loginLimiterHandler() http.Handler {
	secondRate := limiter.Rate{Period: 1 * time.Second, Limit: 1}
	hourRate := limiter.Rate{Period: 1 * time.Hour, Limit: 5}
	dayRate := limiter.Rate{Period: 24 * time.Hour, Limit: 10}

	limit := limiter.NewMultipleRates(memory.NewStore(), secondRate, hourRate, dayRate)

	return stdlib.NewMiddleware(limit).Handler(http.HandlerFunc(LoginHandler))
}

That way you could also properly display all the right HTTP headers. With my solution, when the second-limit blocks the request, the response doesn't include headers about the hourly or daily limits.

My apologies in advance if I've massively over-complicated my solution.

IP Addresses are not differentiated from one another?

I deployed an API that uses limiter on Google Cloud Run. On separate devices with different IP addresses, when I hit the rate limit on one device, the limit is hit on the other device. I have verified the two devices have different IP addresses by searching "my ip" in a browser on both devices. So it seems that the IP address or the key used for the store is the same.

Check given number of requests instead of one at the same time?

Hi,

Is it simple and necessary to add something like AllowN functionality like the other rate limiter instead of just Get to check a single request?
For example, in my case, I want to put a rate limit on the accumulated batch size of a list inside one request, rather than on the number of requests.

And another suggestion is that since the user can specify their key for this limiter for example API keys instead of IP, it is maybe good to point this out in example code or README.

Thanks

package limiter/drivers/store/common includes test flags in main binary

I'm currently using "limiter/drivers/store/memory" cache independently from also using the Limiter. It seems that because there is a "limiter/drivers/store/common/tests.go" in the common package, the testing package injects its flags into my application.

Could this tests.go file be renamed to "common_test.go"?

Is there any way to skip rate limit and keep service available even store crashed?

I've found the Handle function will always do c.Abort() even go cache or redis server crashed.
Just want to keep service available when store disconnected.
Is it possible to move gin context abort to the default handler? So we can customize our ErrorHandler for our needs.

func (middleware *Middleware) Handle(c *gin.Context) {
	key := middleware.KeyGetter(c)
	if middleware.ExcludedKey != nil && middleware.ExcludedKey(key) {
		c.Next()
		return
	}

	context, err := middleware.Limiter.Get(c, key)
	if err != nil {
		middleware.OnError(c, err)
		c.Abort()
		return
	}
}

Rate Limiting certain methods on a Thrift endpoint

I think its more of a question but was not sure what is the right place to ask this.

I am using thrift with HTTP middleware. Our clients call us using Thrift JSON.

Here is a sample call to the login endpoint:

URL : /auth
Body:
{
  "method" : "login",
  "arguments": {
    "email": "",
    "password": "<PASSWORD>"
  }

I can rate limit the entire /auth endpoint. Is there a way I can rate limit certain methods on the endpoint?

Add Feature to control Redis Storing

Right now it is storing the redis key based on client IP.
Can we modify this to something, let say header or client ID.

Storing only API's is unusable where the use cases is to limit based on request or client

Concurrency issues in memory store

Noticed when reviewing the code in store_memory.go that it retrieved the cache item checked if it expired and then attempted to increment it but there was no mutex or anything to protect that.

So it's possible that we check that it hasn't expired but by the time we IncrementInt64 it has expired and is not in the cache resulting in an error:

item, found := s.Cache.Items()[key]
    ms := int64(time.Millisecond)
    now := time.Now()

       // ** TIME A: HAS NOT EXPIRED YET
    if !found || item.Expired() {
        s.Cache.Set(key, int64(1), rate.Period)

        return Context{
            Limit:     rate.Limit,
            Remaining: rate.Limit - 1,
            Reset:     (now.UnixNano()/ms + int64(rate.Period)/ms) / 1000,
            Reached:   false,
        }, nil
    }

        // ** TIME B: key has expired so this will return an error
    count, err := s.Cache.IncrementInt64(key, 1)
    if err != nil {
        return ctx, err
    }

This is a little bit of an edge case, but it does result in unexpected behavior in a high concurrency environment.

Here's a unit test spawning a bunch of goroutines resulting in an error:

func TestConcurrency(t *testing.T) {
    rate := Rate{Period: time.Nanosecond * 10, Limit: 100000}

    store := NewMemoryStoreWithOptions(StoreOptions{
        Prefix:          "limitertests:memory",
        CleanUpInterval: 1 * time.Nanosecond,
    })

    wg := sync.WaitGroup{}
    limiter := NewLimiter(store, rate)
    for i := 0; i < 1000; i++ {
        wg.Add(1)
        go func(i int) {
            for j := 0; j < 10000; j++ {
                _, err := limiter.Get("boo2")
                assert.NoError(t, err)
            }
            wg.Done()
        }(i)
    }
    wg.Wait()
}
        Error Trace:    2:
    Error:      No error is expected but got Item limitertests:memory:boo2 not found

        Error Trace:    2:
    Error:      No error is expected but got Item limitertests:memory:boo2 not found

rate limiting by IP address prefix

For IPv6, it doesn't make sense to rate limit individual /128 addresses since end hosts often get a huge prefix assigned to them, making it easy to bypass rate limits. Instead, how about adding a normalization prefix (default /32 for IPv4 and /128 IPv6 to keep existing functionality)?

It can be trivially accomplished by masking the desired number of bits for the prefix and using that as the IP key, for example:

IPv4: 192.0.2.123 with a /24 prefix: becomes192.0.2.0
IPv6: 2001:db8:cafe:1234:beef::fafa with /48 prefix becomes 2001:db8:cafe::

Limiter for API Key and/or IP address

Hello,
Nice work!
I was wondering how much work would need to extend the project to not only have limits for IP addresses (as keys) but also for API keys.
The use case is: users might have API keys that could have, e.g., higher API limits than the normal per IP limits.
Is this something you're planning to develop?

README is wrong

Docs are wrong.

// Or use a in-memory store with a goroutine which clears expired keys every 30 seconds
store := limiter.NewMemoryStore("prefix_for_keys", 30*time.Second)

According to the code it doesn't take any arguments, so this should just read:

// Or use a in-memory store
store := limiter.NewMemoryStore()

// Alternatively, use a in-memory store with some extra configuration
store := limiter.NewMemoryStoreWithOptions(???)

Something like that anyway.

v3.0.0 tag?

Does the v3.0.0 tag actually point to the latest stable release? It seems to point to a commit in a development branch...

memory store not updating entries

in my testing, the Redis store is working fine, but the in-memory store doesn't seem to be incrementing counters. for example, if i set the following rate:

rate = limiter.Rate{
    Period: 1 * time.Hour,
    Limit:  1000,
}

yields the same counters every time:

< X-Ratelimit-Limit: 1000
< X-Ratelimit-Remaining: 999

any ideas why?

stateless solution - but how to accomplish for non-logged in users

We can throttle logged-in users in our application code by inspecting the JWT and looking at how many requests they made in the last minute etc. AKA, we decrypt the JWT, add the current request timestamp to a queue property, and see how many requests there were in the tiny queue in the JWT.

So that works for non-logged in users without using Redis etc.
But is there a good methodology to throttle non-logged in users? By IP address or something else?
I don't think using an external data store for every request is optimal in terms of performance or simplicity.

I posted a similar question on SO for reference:
https://stackoverflow.com/questions/60958202/how-to-throttle-non-logged-in-users

Find a solution to reduce dependency leak

Currently, while using github.com/ulule/limiter it doesn't download gin, redis or even fasthttp if you doesn't use it.

It bothers me that these libraries are exported to the go.sum of a project, even though it's not included in it's go.mod.

Maybe there is a solution to reduce this annoyance. I don't know ๐Ÿคทโ€โ™‚๏ธ

Calling Reset() on a non-existing key will throw a "retry limit exceeded" error in Redis

// doResetValue will execute resetValue with a retry mecanism (optimistic locking) until store.MaxRetry is reached.
func (store *Store) doResetValue(rtx *libredis.Tx, key string) error {
	for i := 0; i < store.MaxRetry; i++ {
		err := resetValue(rtx, key)
		if err == nil {
			return nil
		}
	}
	return errors.New("retry limit exceeded")
}

// resetValue will try to reset the counter identified by given key.
func resetValue(rtx *libredis.Tx, key string) error {
	deletion := rtx.Del(key)

	count, err := deletion.Result()
	if err != nil {
		return err
	}
	if count != 1 {
		return errors.New("cannot delete key")   <------------
	}

	return nil

}

Because resetValue assumes a key will be present.

To test you can Get a key, flush redis, then try to delete the key.

I use a Redis LRU so sometimes a limiter will get evicted on its own.

Anyway Resetting an already blank limiter should not throw an error in my opinion.

Thanks

Potential race between Redis key TTL and Context.Reset?

I'm seeing what seems to be a race condition where the limiter key is still present in the Redis store after the code waits for Context.Reset to be reached. This causes the next iteration of the code to also be rate-limited, but Context.Reset is already past, so the sleep duration is negative.

Been seeing this with the following:

  • ulele/limiter v3.5.0
  • redis store (redis 5.0.8)

Here's a snippet of the code which should be able to reproduce it:

for {
    limit, err := r.Limiter.Get(ctx, key)
    if err != nil {
	log.Printf("[ERROR] failed to fetch rate-limit context %s", err)
	return err
    }

    if limit.Reached {
        sleep := time.Until(time.Unix(limit.Reset, 0))

        log.Printf("[ERROR] client has proactively throttled for %s", sleep.String())
        <-time.After(sleep)
        continue
    }
    
    // do stuff 
}

And here are the logs:

2020/04/07 19:41:41 [DEBUG] ratelimit context: {Limit:2 Remaining:0 Reset:1586288531 Reached:true}
2020/04/07 19:41:41 [ERROR] client has proactively throttled for 29.954364124s
2020/04/07 19:42:11 [DEBUG] ratelimit context: {Limit:2 Remaining:0 Reset:1586288531 Reached:true}
2020/04/07 19:42:11 [ERROR] client has proactively throttled for -81.245359ms

Can anyone else reproduce this?

Builds fail when using gomod

Building the gin example when using gomods causes the build to fail.

Here is how to reproduce the error:

git clone https://github.com/ulule/limiter-examples.git
cd limiter-examples/gin
go mod init example.com/m
go mod tidy
go build -o main .

It will result in the following error:

# example.com/m
./main.go:33:42: cannot use client (type *"github.com/go-redis/redis/v7".Client) as type "github.com/ulule/limiter/v3/drivers/store/redis".Client in argument to "github.com/ulule/limiter/v3/drivers/store/redis".NewStoreWithOptions:
	*"github.com/go-redis/redis/v7".Client does not implement "github.com/ulule/limiter/v3/drivers/store/redis".Client (wrong type for Del method)
		have Del(...string) *"github.com/go-redis/redis/v7".IntCmd
		want Del(context.Context, ...string) *"github.com/go-redis/redis/v8".IntCmd

Create new ContextStore interface

Since most "Stores" will require an external request, the interface should implement context.Context so that those requests can respect timeouts and cancellations. I'd expect it to look something like this.

// ContextStore is the common interface for limiter stores.
type ContextStore interface {
	Get(ctx context.Context, key string, rate Rate) (Context, error)
	Peek(ctx context.Context, key string, rate Rate) (Context, error)
}

Honestly, it would be great if the whole package went v2 and implemented context as the first parameter everywhere, including the middleware functions.

limiter.Store.Peek(key string) method

It'd be useful if the limiter.Store had a Peek(key string) method to retrieve the current Context without updating it.

It'd be useful for monitoring and debugging, but also for implementing logic where you might want to not take action that would actually use the resource that's being rate limited.

type Store interface {
    Get(key string, rate Rate) (Context, error)
        // inspect limit context without decrementing remaining
    Peek(key string, rate Rate) (Context, error)
}

V3 with dep

How can I use this package if I'm using dep rather than modules?

When I run dep ensure I get these errors:

Solving failure: No versions of github.com/ulule/limiter met constraints:
	v3.1.0: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v3.0.0: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v2.2.2: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v2.2.1: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v2.2.0: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v2.1.0: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v2.0.0: Could not introduce github.com/ulule/[email protected], as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v1.0.0: Could not introduce github.com/ulule/[email protected] due to multiple problematic subpackages:
	Subpackage github.com/ulule/limiter/drivers/middleware/stdlib is missing. (Package is required by (root).)	Subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	master: Could not introduce github.com/ulule/limiter@master, as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	dev: Could not introduce github.com/ulule/limiter@dev, as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v1: Could not introduce github.com/ulule/limiter@v1 due to multiple problematic subpackages:
	Subpackage github.com/ulule/limiter/drivers/middleware/stdlib is missing. (Package is required by (root).)	Subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)
	v2: Could not introduce github.com/ulule/limiter@v2, as its subpackage github.com/ulule/limiter/v3/drivers/store/redis is missing. (Package is required by (root).)

item.Expiration.Second undefined

After trying to go get the project I got the following:

# github.com/ulule/limiter
../../ulule/limiter/store_memory.go:70: item.Expiration.Second undefined (type int64 has no field or method Second)

Looks like go-cache dependency was changed in a non-compatible way. I noticed that somebody already fixed this problem in the fork: radisson@218c2c9

If the change makes sense, would it be possible to merge it? Thanks.

Dep hell

I'm fairly new to Go and I love it except for the dependency management. Maybe you have some experience with this:

Gin hasn't had a release since July 2017 but they are always updating the master branch with great features. In my project, I'm using "branch=master" and you are using "version=1.2" in Gopkg.toml. Do you know any tricks to make importing your project work with the gin's mismatching dependency versions? I would like to stay on the master branch of gin while using your wonderful library.

Thanks

Memory store clean command is a blocker

Limiter is running on a high traffic server so while debugging some problems I saw high letancy from the store cleaner. I know that we need to remove the expired records but maybe there is a better way to do it?

Screenshot 2020-03-27 at 8 07 56 PM

Change limiter rate on the fly

Can we change the period and limit on the fly?

I have the limit and period stored in DB and want to change it in DB and it should reflect as a new period and limit.

Limiter v4 Checklist

This is a checklist of refactoring, enhancements and/or features that I would like to take in the next major version of limiter:

  • Remove panic in DefaultErrorHandler: people may not have a recover middleware, and I would prefer that they overwrite the default behavior than relying on potential capture.
  • Reset precision: It seems some people use this library as a client-side limiter. The current reset mechanism is not great for burst because you could miss some tick if your rate period is in second. This is caused by an imprecise conversion from a TTL to a Unix timestamp.
  • Change rate limit dynamically: Few peoples have asked how we can change/update the rate limit "on the fly". I think it's a great feature.
  • Enable "client" mode: This limiter was designed for a server with HTTP. It's a shame that we can't use it properly as a "client" flavor.
  • Refactor GetContextFromState: We don't need time.Time for expiration and now is unused.

patch release for redis pttl bug fix

Hey! just noticed that a patch release was never cut for #45 and #46 combined, wondering if that will be something we can expect soon? Or that maybe isn't necessary as people can just constrain to master...

Thanks!

Go mod & v3

Hello,

I'm new to go mod and i'm adding it to a project :
Currently with a go mod init I got this version of the limiter :

  • github.com/ulule/limiter v2.2.2+incompatible

I tried to force ugrade to v3 but I got an error :

$ go get github.com/ulule/[email protected]
go: finding github.com/ulule/limiter v3.1.0
go: github.com/ulule/[email protected]: go.mod has post-v0 module path "github.com/ulule/limiter/v3" at revision 4a9155baecad
go: error loading module requirements

I see this PR #39 and look like supported.
Do you have an idea ?

Thanks !

Unnecessary hashing of IP key

I don't believe there's any value to the sha256 of the IP address keys. Particularly as the hashes are expanded back into hex, a sha256 hash in hex is 64 characters. The longest possible IPv6 address is something like 44 characters or so.

address common concern of multi-node clusters

Most production users will be using multiple golang servers in a cluster like Kubernetes or Mesos. If we rate limit by IP in each individual node it's possible by chance, even with a round robin strategy, that a certain node will get hit by the same IP and get unfairly limited. or the reverse, hit all the node's in the cluster evenly and then slip through.

Perhaps add a caveat that rate limiting by IP should probably be done by load balancers when working with clusters?

setRate function refresh the key expire time when the key exist?

hi,
i have a question, this function below in store_redis.go

  func (s RedisStore) setRate(c redis.Conn, key string, rate Rate) ([]int, error) {
        c.Send("MULTI")
        c.Send("SETNX", key, 1)
        c.Send("EXPIRE", key, rate.Period.Seconds())
        return redis.Ints(c.Do("EXEC"))
 }

if the key exist in redis, the function will refresh the expire time of the key, it's design so or bug?

Using with GraphQL

So I tried using this middleware with a graphQL API and I have been getting back a syntax error after the limit has been reached instead of a limit reached message. I am using gqlgen and based on their documentation, I should have been able to use this middleware without any issues, so I am not certain of what is causing my error of:

{
  "error": "JSON.parse: unexpected character at line 1 column 1 of the JSON data"
}

when the limit is reached.

My code looks like this:

      rate, err := limiter.NewRateFromFormatted("5-S")
	if err != nil {
		logger.Panic("Problem setting rate: ", err)
	}

	store := memory.NewStore()

	instance := limiter.New(store, rate)

	middleware := mhttp.NewMiddleware(instance)

	router := chi.NewRouter()

	router.Use(middleware.Handler)

       router.Handle("/graphql", rootResolver)

What should I do in this case?

Update:

This workaround seems to work:

middleware := mhttp.NewMiddleware(instance, mhttp.WithLimitReachedHandler(limitHandler))

func limitHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.Write([]byte(`{"message": "Limit Reached!"}`))
}

This seems to return the proper response, but sometimes I see a signal: terminated message in my terminal when the limit has been reached, not sure if this is suppose to happen?

Add support for resetting the request count for an IP address

I'm using your middleware to protect our API's login endpoint. When a user successfully logs in, I would like to reset the request counter for that user's IP address. That way, other users on the same network (with the same IP address) isn't punished.

This is particularly important when an API is used by lots of users in an office environment where everybody is logging in from the same IP address.

Currently I have to set the daily attempt limit to the maximum amount of users behind a single IP address times 5 (for example). This makes the login endpoint of the API significantly easier to brute-force using a botnet.

(This is offset by a strong password requirement and a slow server side hashing algorithm, but the high rate limit still makes the API much easier to DDOS with a much smaller botnet)


TLDR: It would be very nice to be able to clear a single key from the storage. This will allow setting much more strict limits.

store_memory.go cache.Items()[key] causes panic in high concurrency situations

While testing #17 I ran into an occasional panic in the store_memory.go accessing the `cache.Items()[key]

=== RUN   TestConcurrency
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x5e304]

goroutine 555 [running]:
github.com/ulule/limiter.(*MemoryStore).Get(0xc82000bba0, 0xc820866920, 0x18, 0x0, 0x0, 0xa, 0x186a0, 0x0, 0x0, 0x0, ...)
    /Users/ddaniels/dev/src/github.com/ulule/limiter/store_memory.go:35 +0x24c
github.com/ulule/limiter.(*Limiter).Get(0xc820017b60, 0x4742b8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/ddaniels/dev/src/github.com/ulule/limiter/limiter.go:35 +0x9f
github.com/ulule/limiter.TestConcurrency.func1(0xc820017b60, 0xc820020750, 0xc8200be4f0, 0x210)
    /Users/ddaniels/dev/src/github.com/ulule/limiter/limiter_test.go:96 +0x50
created by github.com/ulule/limiter.TestConcurrency
    /Users/ddaniels/dev/src/github.com/ulule/limiter/limiter_test.go:100 +0x200

The go-cache documentation says you shouldn't access Items() without synchronizing access (which I don't think is possible because the RWLock is private to the cache)

https://github.com/patrickmn/go-cache/blob/master/cache.go#L1001

// Returns the items in the cache. This may include items that have expired,
// but have not yet been cleaned up. If this is significant, the Expiration
// fields of the items should be checked. Note that explicit synchronization
// is needed to use a cache and its corresponding Items() return value at
// the same time, as the map is shared.
func (c *cache) Items() map[string]Item

I put up a PR to them to ensure that cache.Items() returns a copy of the map using the synchronized lock while doing so.

Also I added a method GetItem to retrieve a cache Item:

https://github.com/dougnukem/go-cache/blob/53c1a5bdb67be8eaf0beee21efdbf7db3bab734a/cache.go#L146

// Get a cache Item from the cache (includes Item.Expiration, Item.Object).
// Returns the item or nil, and a bool indicating
// whether the key was found.
func (c *cache) GetItem(k string) (Item, bool) {

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.