Hey anyone who happens to still read things around here :)

I'm opening this issue to discuss some of my recent efforts in a new (still unregistered) package HTTP.jl.

The julia webstack has long been the neglected child of the julia ecosystem, hobbling along with just enough attention to still be considered useable packages, but not much else. To be fair, the webstack originally started out as a HackerSchool project that thankfully provided basic functionality, but which unfortunately has atrophied as the original authors have moved on to bigger and better things. We found an unlikely hero in @malmaud when he stepped up and fixed some glaring usability issues with Requests.jl, as well as creating the MbedTLS.jl package to fill the glaring TLS need, but otherwise, the webstack as a whole has received very little attention over the last 3-4 years.

I have some personal and professional interests around building web services in Julia, but which require a solid stack. As I researched and dug into our webstack code, I originally was just aiming to make some improvements to HttpServer, but the more I looked around, the more I realized a complete overhaul was needed. In particular, most of the webstack code used outdated idioms from 3-4 years ago and was failing to utilize some of the great new languages features that have been released recently. There were also glaring issues of rampant code duplication due to the micro-separation of functionality between HttpCommon, HttpParser, URIParser, Requests, HttpServer; as well as a broad lack of testing in most areas.

The approach I've taken to the webstack overhaul has involved:

• Research: in particular, reading through the actual HTTP 1.1 spec to derive necessary functionality and generate tests, as well as other language http implementations. I became particularly interested in the implementation in go, due to the friendly licensing, modern-ness (i.e. implemented in recent years), and general web focus of the language
• Webstack consolidation: most other languages/libraries don't separate webstack functionality as granular as it is in julia; the most separation you generally see is between client vs. server. While the intent is right, there just aren't that many uses of having HttpCommon or HttpParser exist as stand-alone packages. In particular, this has led to a lot of code duplication in the webstack, as well as quite a few cases of "code squatting", i.e. code living in a repo that actually extends/is relevant for a different package entirely, not to mention the problem users have in figuring out where to request functionality or submit issues. In general, I think it's made the webstack less maintainable and harder to develop on/with having everything separated out. My first efforts in overhauling were to combine the existing webstack packages into a single git repo, making sure to preserve the git histories of each.
• Design from the top down: I tried to take what most high-level client/server implementations look like in other libraries/languages and come up with a Julia-version that makes the most sense. Starting at the highest-levels and working down helps drive the right abstractions and fuels testing.
• Overall webstack quality: one of my main goals in this endeavor is to make the webstack code "Base worthy"; meaning it's well-tested, well-documented, and as many warts as possible are excised, so that, as a package, it could be ready to be included in Base. Obviously there are a lot of other design decisions for including something like this in Base (or rather, the Julia standard library), and that's not ultimately my goal, just that it get to and remain in a state that it could.

State of progress:

• I know, I know, I probably should have posted this earlier, but my initial research/code exploration kind of snow-balled into the overhaul and I ended up having the time/motivation to do it, so I just started.
• The overhaul is about 2/3 of the way to a v0.1, IMO. The client-side has been totally revamped and I'm working on wrapping up a few remaining "robustness" features and filling in tests and docs. Server-side has been through a cycle or two of redesign, but there's still some core design left to do around utilizing Julia's built-in parallelism and how that affects the core server definition and event loop.
• In terms of dependencies, I'm aiming for one, julia or shared library otherwise. HTTP.jl will depend on MbedTLS.jl, but won't be hard-coded to it. MbedTLS provides an implementation of the TransportLayerSecurity.jl interface, that other TLS packages could do as well, and swapping out MbedTLS with another TLS package would be a matter of changing a single line. Currently, we're relying on the http-parser library from joyent/node, but the actual library code isn't more than 1K LOC and I have plans to do a pure reimplementation in Julia (shouldn't take more than a week or two); this will leave us with the single dependency of a TLS package.
• All of that said, I'd say the package as a whole is 2-3 months from being registered and tagged for v0.1.

I'd like to welcome anyone and everyone to provide feedback to the discussion here about what they think and ideas for moving the webstack forward. Also feel free to take HTTP.jl for a spin, it should be fairly functional already. I'd of course welcome all feedback and help designing, implementing, and testing the package from as many people as possible. Happy to open up a JuliaWeb gitter if we want a more fast-paced place to discuss if there's interest, otherwise, we can use github issues and the Julia discourse forum to discuss.

Heres how I see the state of JuliaWeb right now: too many packages, not enough developers. This isn't a criticism of anyone, more just a concerning state of affairs that I'd like to find a way out of. A lot of these packages are pretty important.

• I think there are some opportunities for rationalization.
  • For example, Requests.jl and HTTPClient.jl doing the exact same thing, in different ways. Would be best to pick one and concentrate efforts I think, especially as Requests.jl is one of the most depended on packages.
  • URITemplate.jl could possibly be merged into URIParser.jl?
• Morsel/Meddle are neat, but are currently receiving very little love. Morsel is actually one of the top 10 Julia packages by ⭐s, and I'm not sure if it even works properly.
• GnuTLS.jl is a very important package, but it isn't even passing tests right now, and I think there is approximately 1 person who understands the package.

I think it'd be great getting more developers at least getting more familiar with these packages, and giving them a polish.

Side note: I do feel bad about saying "someone should do something!" because I don't believe thats the open source way, but I've got too many other packages that I do use in my work to think about + PkgEval stuff.

I think it would be good if we could move Escher over to the JuliaWeb organization. Escher would fit wonderfully in the Julia Web stack as part of the fronted, this would allow to ease and continue development of the Escher package, nowadays, @shashi is busy at other things, and he hasn't been able to be responsive to the issues at his repository.

Likewise, Escher is known not to work correctly for a long time now and it doesn't work with v0.5 either, the issues and chat room are silent from any response from Shashi, it's main developer, but there is a lot of interest in updating this package to keep it working.

I'm not sure if it's better to have this issue at Escher repo or here, so please tell me what you think, thanks!

Hi all,

I've written a first draft of an implementation of secure session cookies. It is Example 5 of this repo. It is intended as a starting point for developing a separate package (or piece of middleware in a larger package) that provides this functionality. Feel free to use/modify it as you please.

The security protocol is described in docs/security_status.md.

At this stage the most important thing is to recognise that this implementation is based on my own not-so-thorough investigation, and I am not a security professional. Indeed the code hasn't been scrutinised by any security professionals, and therefore is to be used at one's own risk.

Rather than reject the example as insecure, I find it helpful to consider how long it would take a security professional to break the implementation (with a given hardware setup), and to gradually improve (increase) this time, i.e., harden the implementation.

With the current example I'm guessing it wouldn't take long, but it's a start. It's also the limit of my expertise. Suggestions for improving the security of the example and/or separating the code into its own package are most welcome. If you think a radical rethink is in order then I'm all ears.

Thoughts?

Hello!

I am the developer of DandelionWebSockets.jl at https://github.com/dandeliondeathray/DandelionWebSockets.jl. It's a WebSockets client.
I'm wondering if you would be open to moving this project to JuliaWeb? I think it would fit well functionality-wise with the other projects, especially as you already have a WebSockets server project.

Unfortunately I am no longer using Julia, so I'm not actively maintaining or developing it though. I'm having a hard time prioritizing it in my daily life. I put a call out in the Julia Discourse for anyone willing to take over maintenance, but no takers so far (https://discourse.julialang.org/t/request-for-help-maintenance-of-dandelionwebsockets-jl/5572/1).

I don't want to simply dump the responsibility on your organization though, so I think that if we move it to JuliaWeb, we'll wait until a maintainer has be found.

Best regards,
Erik Edin

Inspired by #5 (comment) - A placeholder for discussion of APIs that are abstract enough to be independent of the underlying TLS library (whether it be GnuTLS, OpenSSL, or other).

I want to apply to apply to GSoC. I posted it on Julia-user google group here but didn't get any response from any potential mentor for HTTP2 Implementation project, for other project I did get one.

I have some experience while contributing to Julia for months now.

Please reply soon because 25/03/2016 is the deadline for proposal submission.

Thank you.

GnuTLS.jl is the core of a bunch of other JuliaWeb packages but it's not stable at this point, and this results in the other packages failing intermittently (and without any consistency). Since this is a roadmap issue, I'd like to suggest we set a direction here:

1. Confirm GnuTLS as the appropriate package for TLS / secure communications, and put effort into fixing it (and making sure it keeps in sync with libgnutls); or
2. Deprecate GnuTLS and seek a replacement that will be as easy (or easier) to maintain.

Once SMP multithreading becomes available in julia, I'd like to see HttpServer running on a threadpool (with nr. threads = nr of cores) so that the server can actually use all cores of a machine.

It was proposed to rename some packages in JuliaWeb:

• HttpCommon -> HTTPCommon or HTTPCommons (JuliaWeb/HttpCommon.jl#18)
• HttpServer -> HTTPServer or HTTPServers (JuliaWeb/HttpServer.jl#43)
• HttpParser -> HTTPParser or HTTPParsers (JuliaWeb/HttpParser.jl#29)

Most past discussion happened here: JuliaWeb/HttpCommon.jl#18. As you can see, I am not in favor of this change because I don't think that the gain in consistency is worth the broken code.

Hi all,

I've started work on AccessControl.jl, which is intended to provide authentication and authorization for web apps written in Julia. A few observations below. Thoughts?

Cheers,
Jock

1. You can see what is done and what is intended by looking at docs/Contents. I'll flesh this out over the next few weeks.
2. The API is deliberately very imperative. Package devs can then wrap the API in whatever style turns out to be idiomatic for web dev in Julia.
3. It's monolithic. It is modular and could be split into separate packages but:
   • A lot of packages would result, confusing newcomers...and me :)
   • Most of the time, I suspect all of the time, the functionality of the separate packages would be used in concert anyway, so why not bundle them together in the first place?
4. Further to Point 3, I have included functionality for password hashing and secure cookies in AccessControl.jl. This makes SecureSessions.jl redundant. Should I deprecate SecureSessions.jl in favour of AccessControl.jl (when the latter is built out and tested)?
5. Over time I will harden the security implementation, but at this stage I'm seeking feedback on the API, which needn't change as the security improves.

I helped implement the first generation web code back in 2013 at the Recurse Center (formerly Hacker School) as referenced in #16. I have not subsequently had the pleasure of getting to work with Julia on a regular basis, though I am 👏 🎉 excited by all the great work that you all are doing.

I own juliawebstack.org which was set up back then, and points at the github pages for the JuliaArchive/juliawebstack.org repo. It is the top google hit for "julia web" and right now that information is woefully out of date.

Here are some options:

1. We do nothing, the domain continues to serve the existing content
2. Someone updates the information in the associated repo / GH pages
3. I keep the domain alive as a volunteer and point it somewhere new
4. I donate and transfer the domain to someone else
5. I take it down and let the domain lapse

Hi Eric,

I copied your message (below) into a new issue that focuses on routing - hope you don't mind.

Glad the tutorial appeals. First of all, I'm no web dev guru. I'm just learning, so please take what I'm saying with a grain of salt and compare to other opinions.

That said, while I haven't needed routing so far, my approach to routing in Julia web apps would be to use existing packages to get the desired functionality, then if significant boilerplate code is required create a package that contains this boilerplate. So I'd probably do something like this:

1. Use URIParser.jl to extract the different parts of the URL (which is a type of URI) into separate fields of a URI object.
2. The most relevant parts will probably be the path and query fields.
   a) Compare the path to a regex...see the apps in the tutorial for examples
   b) Extract any path parameters if they exist
   c) Parse the query into a dict using parsequerystring() in HttpCommon.jl

The code would look something like this:

function app(req::Request)
    res = Response()    # Init response

    # Extract information from request
    uri = URI(req.resource)
    path_params = extract_path_params(uri.path)
    q = parsequerystring(uri.query)

    # Dispatch to handler
    if condition1
        handler1        # Modifies response
    elseif condition2
        handler2        # Modifies response
    ...
    end
    res
end

The conditions in the branching logic can be functions of the uri, parameters and/or query as appropriate.

This approach enables one to get into the mechanics of routing and understand what's happening. You may prefer a ready-made solution such as Mux.jl from @one-more-minute.

Hope this helps,
Jock

From @EricForgy

Hi JockLawrie,

I think I am the exact target audience for your tutorial and remember seeing an earlier version of it. I will definitely work through it. I have been meaning to ask a general question though and your third bullet point above pushes me to ask it (before reading the tutorial - dangerous!).

I have 20+ years working with Matlab including some web-based work (taking advantage of the fact you can embed Java in m-files) and now I am hoping to build Julia web apps. I was a little sad to see Morsel.jl abandoned, but really, the feature I used from Morsel was routing and I liked the way it worked. I plan to give Mux.jl a try, but was wondering if I could ask about "best practices" for routing. Is just HttpCommon and HttpServer sufficient?

I need enterprise quality technology suitable for large financial institutions.

submission.docx

Hello,

It will be nice to have an implementation of a joinurl function
See discussion at https://discourse.julialang.org/t/join-parts-of-a-url-joinpath/12441/12
Pinging @sirex @Nosferican @sbromberger
But I don't know exactly in which package of JuliaWeb org such a function could be.

Maybe an other approach could be to have simply a function named join which could take parameters of URI type.

Kind regards

add a HttpSession type/object to HttpServer which would basicallly be like a dict per user to track user data in a http session. See e.g. session in java jsp/servlets.

Proposal:

1. new packages should be as "pure Julia" as possible, minimizing any external code/dependencies, and
2. existing packages that survive "rationalization" should be rewritten/updated to reduce dependencies on external code.

In the near future, I'd like JuliaWeb to only contain working and useful packages. Being listed in JuliaWeb should be an indication that that developers of Julia's web infrastructure stand behind the package - both certifying that the package is functional and advocating for its use.

A corollary of that is that we shouldn't have packages that are almost perfect substitutes for each other without a compelling reason.

I believe the following packages no longer meet that criterion:

• SMTPClient.jl
• LibCURL.jl (this meets the "useful functionality" bar, but is currently broken and needs a maintainer to stay in JuliaWeb).
• HTTPClient.jl
• Meddle.jl
• GnuTLS.jl
• Morsel.jl

They should be moved back to their original developers' private repositories, or perhaps to some kind of "JuliaWebDeprecated" organization.

Meanwhile, depending on @one-more-minute's disposition, Mux.jl could be moved into JuliaWeb.

Thoughts, @IainNZ, @sbromberger, @quinnj?

This would be very handy to easily add images, pdfs etc to dynamically generated webpages (now you have to write a webprog itself that load the content from the filesystem and returns it via http and link to that prog). This would be similar to what Tomcat provides.