cemerick / url Goto Github PK

clojurescript belongs in the provided profile. I'll try and remember to follow up with more info, but it would prevent clojurescript being pulled in on non-cljs projects.

(str (assoc (cemerick.url/url "http://abc.com:80/haha/") :port 8080))
=> "http://abc.com:8080/haha"
where you can see the ending slash is striped off.

I am using this url library in a reverse proxy. The reverse proxy would replace the "Location" header returned by a web server with its own hostname/port. This means last slash character is stripped off. Imagine below interactions

1. browser request http://www.simpleweb.org/ietf/mibs/modules/html
2. reverse proxy forward it to web server
3. web server returns 301, Location: http://www.simpleweb.org/ietf/mibs/modules/html/
4. reverse proxy replace it with its own hostname/port, using url library and new url is: http://reverseproxy_host:8080/ietf/mibs/modules/html
5. browser request http://reverseproxy_host:8080/ietf/mibs/modules/html
6. reverse proxy forward it to web server
7. web server sees it not ending with '/', returns 301 again
8. repeats
   ...

Hi,

thank you for creating this useful library!

I ran out into a problem, but I'm not sure if it's a bug or a design decision. Here is an example:

(-> "http://myserver.com/path/ url (update :query {:param "value"}) str) ; http://myserver.com/path?param=value

A trailing slash is trimmed from (before ?). What I expect:

(-> "http://myserver.com/path/ url (update :query {:param "value"}) str) ; http://myserver.com/path/?param=value

What do you think?

Y U NO URI?

I wanted to use this to parse a mongodb:// URL, but java.net.URL wont have it. How can we sort this out?

Per RFC, mailto is special and does not use //: https://tools.ietf.org/html/rfc2368

Is really necessary in my opinion something like that:
=>(str (url "http://www.saltycrane.com/blog/2011/11/unique-python-redis-based-queue-delay/ :query {:a 3, :b 4}))
;; "http://www.saltycrane.com/blog/2011/11/unique-python-redis-based-queue-delay/a=3&b=4"
That is different from
=> (str (assoc *2 :query {:a 5 :b 6}))
"https://api.twitter.com/1/users/profile_image/cemerick?a=5&b=6"

Hi @cemerick,

Thanks for this project, it's been very useful! Just asking, do you still have time to maintain it?

Why are paths normalized? I work with quite a few urls that require a trailing /, every time I try to rewrite an url the trailing slash is removed.

(let [test "http://some.domain.com/some/path/"
      test' (cemerick.url/url test)]
  (prn [:test test])
  (prn [:test' (str test')])
  (= test (str test')))

I don't think anything parts of the url should be touched by default, since it my case all resulting URLs are invalid. :(

Any thoughts?

The url is not split in correct parts if "/" is percent encoded (%2F). Would it be nice to support percent encoded slashes?

Hello,

Thank you for that free software :-) I've noticed it doesn't support well nester parameters. However it's not that difficult. Would you accept a PR with a code like this?

(defn- kv-encoder-fn
  "Convert a [key value] pair into an \"k=v\" encoded string. Only recurs over maps, vectors and lists for the value."
  [map->query-string path]
  (fn [[key value]]
    (let [[path-head & path-tail :as nested-path] (map #(cond (keyword? %) (name %)
                                                              :else %)
                                                       (conj (vec path) key))]
      (cond (map? value) (map->query-string nested-path value)
            (sequential? value) (map->query-string nested-path (into {} (map-indexed vector (vec value))))
            :else (let [encoded-key (url-encode (str path-head (str/join (map #(str "[" % "]") path-tail))))
                        encoded-value (url-encode value)]
                    (str encoded-key "=" encoded-value))))))

(defn map->query-string
  "Convert a nested map of parameters into a query string. Only recurs over maps, vectors and lists."
  ([input]
   (map->query-string [] input))
  ([path input]
   (->> input
        sort
        (map (kv-encoder-fn map->query-string path))
        (str/join "&"))))

The library won't load in Windows, just making a require will throw an error.

This is a problem with pathetic, see davidsantiago/pathetic#2

ClojureScript:cljs.user> (str (cemerick.url/url "/asdf"))
":///asdf"

which is not something that a browser can understand.

First I'd like to say awesome project! We have a project using Stripes and we are trying to parse a referrer header using cemerick. The issue here is that this is a valid url and should be parsed as such.

(url "http://localhost/icmobile/actions/User.action?listAllPageShow=")
=> IllegalArgumentException No value supplied for key: listAllPageShow

I think we should adjust the internal query->map function as follows:

(defn- augment 
    [query-vector]
    (if (= 1 (count query-vector))
        [(first query-vector) nil]
        query-vector))

(defn- query->map
  [qstr]
  (when qstr
    (-?>> (.split qstr "&")
      seq
      (mapcat #(.split % "="))
      (map url-decode)
      (augment)
      (apply hash-map))))

such that the output is now:

(url "http://localhost/icmobile/actions/User.action?listAllPageShow=abc")
=> #cemerick.url.URL{:protocol "http", :username nil, :password nil, :host "localhost", :port -1, :path "/icmobile/actions/User.action", :query {"listAllPageShow" "abc"}, :anchor nil}
(url "http://localhost/icmobile/actions/User.action?listAllPageShow=")
=> #cemerick.url.URL{:protocol "http", :username nil, :password nil, :host "localhost", :port -1, :path "/icmobile/actions/User.action", :query {"listAllPageShow" nil}, :anchor nil}

I'm not a Clojure expert which is why I'm not submitting this as a pull request. If you feel this is acceptable, I'd be happy to submit a pull request.

(This is a strawman proposal.)

Query parameters in uris allow keys to be repeated. It would be nice if this were supported as many services make use of this, e.g. default query param parsing in PHP and ring.middleware.params.

I propose that if the value in query map is a seq, repeat the key in the serialized uri for each value in the seq. Likewise on deserialization emit a vector of values for the key.

E.g.

(-> (url "http://example.org?a=foo&a=bar") (:query))
; => {"a" ["foo" "bar"]}
(str (-> (url "http://example.org") (assoc :a ["foo" "bar"]))
; => "http://example.org?a=foo&a=bar"

The problem with this is that the cardinality of query param values in parsed urls will depend on the query string.

Other approaches I've seen paper over this difference by using a custom type that pretends all keys have scalar values unless you do something special to ask for the other values for a key (e.g. many Python libraries use a "MultiDict" with a special .getall(k) method). This could be done in clojure with a custom type for query parameters that returns the last seen value for lookups, but has some extra protocol that returns a vector. (However if you go down this route you no longer have a simple library.)

Another possibility is to force the user to be explicit about which query params are cardinality-many. This could be done with different url functions, examples:

(-> (multi-query-url "http://example?a=foo&a=bar&b=baz") :query)
;=> {"a" ["foo" "bar"] "b" ["baz"]}
(-> (multi-query-url "http://example?a=foo&a=bar&b=baz" #{"a"}) :query)
;=> {"a" ["foo" "bar"] "b" "baz"}
(-> (multi-query-url "http://example?a=foo&a=bar&b=baz" (complement #{"a"})) :query)
;=> {"a" "bar" "b" ["baz"]}
;;;  (A subset of) PHP style:
(-> (multi-query-url "http://example?a[]=foo&a[]=bar&b=baz" #(.endsWith % "[]")) :query)
;=> {"a" ["foo" "bar"] "b" "baz"}

Thoughts?

=> (str (assoc (cemerick.url/url "https://XXX.cloudant.com/") :path "databasename"))
"https://XXX.cloudant.comdatabasename"

LOL

The example is from the clutch README.md, i.e. clutch is broken by extension.

If a map like this:

{:a 1 "b" 2}

Is passed to map->query (or to url as the :query map), the function throws an error:

java.lang.ClassCastException: clojure.lang.Keyword cannot be cast to java.lang.String
    at java.lang.String.compareTo(String.java:111)
    at clojure.lang.Util.compare(Util.java:153)
    at clojure.lang.APersistentVector.compareTo(APersistentVector.java:424)
    at clojure.lang.Util.compare(Util.java:153)
    at clojure.core$compare.invoke(core.clj:805)
    at clojure.lang.AFunction.compare(AFunction.java:47)
    at java.util.TimSort.countRunAndMakeAscending(TimSort.java:351)
    at java.util.TimSort.sort(TimSort.java:216)

This is caused by sort trying to compare string and keyword keys.

I can try to submit a pull request for this - just need to call name on the keys before sorting I think.

Since it's 3-5 years old, are there better alternatives?
Anyone still using?