Code Monkey home page Code Monkey logo

roadmap's Introduction

JuliaWeb Roadmap

Discussion and planning for JuliaWeb packages.

Please see the Issues page for more.

roadmap's People

Contributors

iainnz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

roadmap's Issues

Webstack overhaul

Hey anyone who happens to still read things around here :)

I'm opening this issue to discuss some of my recent efforts in a new (still unregistered) package HTTP.jl.

The julia webstack has long been the neglected child of the julia ecosystem, hobbling along with just enough attention to still be considered useable packages, but not much else. To be fair, the webstack originally started out as a HackerSchool project that thankfully provided basic functionality, but which unfortunately has atrophied as the original authors have moved on to bigger and better things. We found an unlikely hero in @malmaud when he stepped up and fixed some glaring usability issues with Requests.jl, as well as creating the MbedTLS.jl package to fill the glaring TLS need, but otherwise, the webstack as a whole has received very little attention over the last 3-4 years.

I have some personal and professional interests around building web services in Julia, but which require a solid stack. As I researched and dug into our webstack code, I originally was just aiming to make some improvements to HttpServer, but the more I looked around, the more I realized a complete overhaul was needed. In particular, most of the webstack code used outdated idioms from 3-4 years ago and was failing to utilize some of the great new languages features that have been released recently. There were also glaring issues of rampant code duplication due to the micro-separation of functionality between HttpCommon, HttpParser, URIParser, Requests, HttpServer; as well as a broad lack of testing in most areas.

The approach I've taken to the webstack overhaul has involved:

  • Research: in particular, reading through the actual HTTP 1.1 spec to derive necessary functionality and generate tests, as well as other language http implementations. I became particularly interested in the implementation in go, due to the friendly licensing, modern-ness (i.e. implemented in recent years), and general web focus of the language
  • Webstack consolidation: most other languages/libraries don't separate webstack functionality as granular as it is in julia; the most separation you generally see is between client vs. server. While the intent is right, there just aren't that many uses of having HttpCommon or HttpParser exist as stand-alone packages. In particular, this has led to a lot of code duplication in the webstack, as well as quite a few cases of "code squatting", i.e. code living in a repo that actually extends/is relevant for a different package entirely, not to mention the problem users have in figuring out where to request functionality or submit issues. In general, I think it's made the webstack less maintainable and harder to develop on/with having everything separated out. My first efforts in overhauling were to combine the existing webstack packages into a single git repo, making sure to preserve the git histories of each.
  • Design from the top down: I tried to take what most high-level client/server implementations look like in other libraries/languages and come up with a Julia-version that makes the most sense. Starting at the highest-levels and working down helps drive the right abstractions and fuels testing.
  • Overall webstack quality: one of my main goals in this endeavor is to make the webstack code "Base worthy"; meaning it's well-tested, well-documented, and as many warts as possible are excised, so that, as a package, it could be ready to be included in Base. Obviously there are a lot of other design decisions for including something like this in Base (or rather, the Julia standard library), and that's not ultimately my goal, just that it get to and remain in a state that it could.

State of progress:

  • I know, I know, I probably should have posted this earlier, but my initial research/code exploration kind of snow-balled into the overhaul and I ended up having the time/motivation to do it, so I just started.
  • The overhaul is about 2/3 of the way to a v0.1, IMO. The client-side has been totally revamped and I'm working on wrapping up a few remaining "robustness" features and filling in tests and docs. Server-side has been through a cycle or two of redesign, but there's still some core design left to do around utilizing Julia's built-in parallelism and how that affects the core server definition and event loop.
  • In terms of dependencies, I'm aiming for one, julia or shared library otherwise. HTTP.jl will depend on MbedTLS.jl, but won't be hard-coded to it. MbedTLS provides an implementation of the TransportLayerSecurity.jl interface, that other TLS packages could do as well, and swapping out MbedTLS with another TLS package would be a matter of changing a single line. Currently, we're relying on the http-parser library from joyent/node, but the actual library code isn't more than 1K LOC and I have plans to do a pure reimplementation in Julia (shouldn't take more than a week or two); this will leave us with the single dependency of a TLS package.
  • All of that said, I'd say the package as a whole is 2-3 months from being registered and tagged for v0.1.

I'd like to welcome anyone and everyone to provide feedback to the discussion here about what they think and ideas for moving the webstack forward. Also feel free to take HTTP.jl for a spin, it should be fairly functional already. I'd of course welcome all feedback and help designing, implementing, and testing the package from as many people as possible. Happy to open up a JuliaWeb gitter if we want a more fast-paced place to discuss if there's interest, otherwise, we can use github issues and the Julia discourse forum to discuss.

State of JuliaWeb

Heres how I see the state of JuliaWeb right now: too many packages, not enough developers. This isn't a criticism of anyone, more just a concerning state of affairs that I'd like to find a way out of. A lot of these packages are pretty important.

  • I think there are some opportunities for rationalization.
    • For example, Requests.jl and HTTPClient.jl doing the exact same thing, in different ways. Would be best to pick one and concentrate efforts I think, especially as Requests.jl is one of the most depended on packages.
    • URITemplate.jl could possibly be merged into URIParser.jl?
  • Morsel/Meddle are neat, but are currently receiving very little love. Morsel is actually one of the top 10 Julia packages by ⭐s, and I'm not sure if it even works properly.
  • GnuTLS.jl is a very important package, but it isn't even passing tests right now, and I think there is approximately 1 person who understands the package.

I think it'd be great getting more developers at least getting more familiar with these packages, and giving them a polish.

Side note: I do feel bad about saying "someone should do something!" because I don't believe thats the open source way, but I've got too many other packages that I do use in my work to think about + PkgEval stuff.

Adopt Escher.jl

I think it would be good if we could move Escher over to the JuliaWeb organization. Escher would fit wonderfully in the Julia Web stack as part of the fronted, this would allow to ease and continue development of the Escher package, nowadays, @shashi is busy at other things, and he hasn't been able to be responsive to the issues at his repository.

Likewise, Escher is known not to work correctly for a long time now and it doesn't work with v0.5 either, the issues and chat room are silent from any response from Shashi, it's main developer, but there is a lot of interest in updating this package to keep it working.

I'm not sure if it's better to have this issue at Escher repo or here, so please tell me what you think, thanks!

Secure sessions

Hi all,

I've written a first draft of an implementation of secure session cookies. It is Example 5 of this repo. It is intended as a starting point for developing a separate package (or piece of middleware in a larger package) that provides this functionality. Feel free to use/modify it as you please.

The security protocol is described in docs/security_status.md.

At this stage the most important thing is to recognise that this implementation is based on my own not-so-thorough investigation, and I am not a security professional. Indeed the code hasn't been scrutinised by any security professionals, and therefore is to be used at one's own risk.

Rather than reject the example as insecure, I find it helpful to consider how long it would take a security professional to break the implementation (with a given hardware setup), and to gradually improve (increase) this time, i.e., harden the implementation.

With the current example I'm guessing it wouldn't take long, but it's a start. It's also the limit of my expertise. Suggestions for improving the security of the example and/or separating the code into its own package are most welcome. If you think a radical rethink is in order then I'm all ears.

Thoughts?

Moving DandelionWebSockets.jl to JuliaWeb?

Hello!

I am the developer of DandelionWebSockets.jl at https://github.com/dandeliondeathray/DandelionWebSockets.jl. It's a WebSockets client.
I'm wondering if you would be open to moving this project to JuliaWeb? I think it would fit well functionality-wise with the other projects, especially as you already have a WebSockets server project.

Unfortunately I am no longer using Julia, so I'm not actively maintaining or developing it though. I'm having a hard time prioritizing it in my daily life. I put a call out in the Julia Discourse for anyone willing to take over maintenance, but no takers so far (https://discourse.julialang.org/t/request-for-help-maintenance-of-dandelionwebsockets-jl/5572/1).

I don't want to simply dump the responsibility on your organization though, so I think that if we move it to JuliaWeb, we'll wait until a maintainer has be found.

Best regards,
Erik Edin

API planning for TLS backend

Inspired by #5 (comment) - A placeholder for discussion of APIs that are abstract enough to be independent of the underlying TLS library (whether it be GnuTLS, OpenSSL, or other).

Seeking mentor for GSoC 2016 project HTTP2 Implementation.

I want to apply to apply to GSoC. I posted it on Julia-user google group here but didn't get any response from any potential mentor for HTTP2 Implementation project, for other project I did get one.

I have some experience while contributing to Julia for months now.

Please reply soon because 25/03/2016 is the deadline for proposal submission.

Thank you.

Fix or (deprecate and replace) GnuTLS

GnuTLS.jl is the core of a bunch of other JuliaWeb packages but it's not stable at this point, and this results in the other packages failing intermittently (and without any consistency). Since this is a roadmap issue, I'd like to suggest we set a direction here:

  1. Confirm GnuTLS as the appropriate package for TLS / secure communications, and put effort into fixing it (and making sure it keeps in sync with libgnutls); or
  2. Deprecate GnuTLS and seek a replacement that will be as easy (or easier) to maintain.

multithreaded HttpServer

Once SMP multithreading becomes available in julia, I'd like to see HttpServer running on a threadpool (with nr. threads = nr of cores) so that the server can actually use all cores of a machine.

AccessControl.jl

Hi all,

I've started work on AccessControl.jl, which is intended to provide authentication and authorization for web apps written in Julia. A few observations below. Thoughts?

Cheers,
Jock

  1. You can see what is done and what is intended by looking at docs/Contents. I'll flesh this out over the next few weeks.
  2. The API is deliberately very imperative. Package devs can then wrap the API in whatever style turns out to be idiomatic for web dev in Julia.
  3. It's monolithic. It is modular and could be split into separate packages but:
    • A lot of packages would result, confusing newcomers...and me :)
    • Most of the time, I suspect all of the time, the functionality of the separate packages would be used in concert anyway, so why not bundle them together in the first place?
  4. Further to Point 3, I have included functionality for password hashing and secure cookies in AccessControl.jl. This makes SecureSessions.jl redundant. Should I deprecate SecureSessions.jl in favour of AccessControl.jl (when the latter is built out and tested)?
  5. Over time I will harden the security implementation, but at this stage I'm seeking feedback on the API, which needn't change as the security improves.

Future of juliawebstack.org

I helped implement the first generation web code back in 2013 at the Recurse Center (formerly Hacker School) as referenced in #16. I have not subsequently had the pleasure of getting to work with Julia on a regular basis, though I am 👏 🎉 excited by all the great work that you all are doing.

I own juliawebstack.org which was set up back then, and points at the github pages for the JuliaArchive/juliawebstack.org repo. It is the top google hit for "julia web" and right now that information is woefully out of date.

Here are some options:

  1. We do nothing, the domain continues to serve the existing content
  2. Someone updates the information in the associated repo / GH pages
  3. I keep the domain alive as a volunteer and point it somewhere new
  4. I donate and transfer the domain to someone else
  5. I take it down and let the domain lapse

Routing

Hi Eric,

I copied your message (below) into a new issue that focuses on routing - hope you don't mind.

Glad the tutorial appeals. First of all, I'm no web dev guru. I'm just learning, so please take what I'm saying with a grain of salt and compare to other opinions.

That said, while I haven't needed routing so far, my approach to routing in Julia web apps would be to use existing packages to get the desired functionality, then if significant boilerplate code is required create a package that contains this boilerplate. So I'd probably do something like this:

  1. Use URIParser.jl to extract the different parts of the URL (which is a type of URI) into separate fields of a URI object.
  2. The most relevant parts will probably be the path and query fields.
    a) Compare the path to a regex...see the apps in the tutorial for examples
    b) Extract any path parameters if they exist
    c) Parse the query into a dict using parsequerystring() in HttpCommon.jl

The code would look something like this:

function app(req::Request)
    res = Response()    # Init response

    # Extract information from request
    uri = URI(req.resource)
    path_params = extract_path_params(uri.path)
    q = parsequerystring(uri.query)

    # Dispatch to handler
    if condition1
        handler1        # Modifies response
    elseif condition2
        handler2        # Modifies response
    ...
    end
    res
end

The conditions in the branching logic can be functions of the uri, parameters and/or query as appropriate.

This approach enables one to get into the mechanics of routing and understand what's happening. You may prefer a ready-made solution such as Mux.jl from @one-more-minute.

Hope this helps,
Jock


From @EricForgy

Hi JockLawrie,

I think I am the exact target audience for your tutorial and remember seeing an earlier version of it. I will definitely work through it. I have been meaning to ask a general question though and your third bullet point above pushes me to ask it (before reading the tutorial - dangerous!).

I have 20+ years working with Matlab including some web-based work (taking advantage of the fact you can embed Java in m-files) and now I am hoping to build Julia web apps. I was a little sad to see Morsel.jl abandoned, but really, the feature I used from Morsel was routing and I liked the way it worked. I plan to give Mux.jl a try, but was wondering if I could ask about "best practices" for routing. Is just HttpCommon and HttpServer sufficient?

I need enterprise quality technology suitable for large financial institutions.

add HttpSession

add a HttpSession type/object to HttpServer which would basicallly be like a dict per user to track user data in a http session. See e.g. session in java jsp/servlets.

RFC - Strategic direction - native Julia packages

Proposal:

  1. new packages should be as "pure Julia" as possible, minimizing any external code/dependencies, and
  2. existing packages that survive "rationalization" should be rewritten/updated to reduce dependencies on external code.

Large-scale cleanup of JuliaWeb

In the near future, I'd like JuliaWeb to only contain working and useful packages. Being listed in JuliaWeb should be an indication that that developers of Julia's web infrastructure stand behind the package - both certifying that the package is functional and advocating for its use.

A corollary of that is that we shouldn't have packages that are almost perfect substitutes for each other without a compelling reason.

I believe the following packages no longer meet that criterion:

  • SMTPClient.jl
  • LibCURL.jl (this meets the "useful functionality" bar, but is currently broken and needs a maintainer to stay in JuliaWeb).
  • HTTPClient.jl
  • Meddle.jl
  • GnuTLS.jl
  • Morsel.jl

They should be moved back to their original developers' private repositories, or perhaps to some kind of "JuliaWebDeprecated" organization.

Meanwhile, depending on @one-more-minute's disposition, Mux.jl could be moved into JuliaWeb.

Thoughts, @IainNZ, @sbromberger, @quinnj?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.