This is crossposted at blog.beeminder.com/docdriven.

It seems like every time I talk about principles of software engineering to you all I get jaw-droppingly insightful replies.
No pressure.

Ok, if you google "documentation-driven development", it seems to be a lot of people saying that documentation is so important that you should write it first blah blah blah.
But I think there's a deeper insight here that I want to run by you.

Making design decisions -- like deciding what to build -- is really hard.
(That's not the deep part yet.)
We live and breathe Beeminder and we wallow around in its code all day long.
Even if we were to establish design criteria like "reduce cognitive load" our first-line thoughts on these are different than our users'.

Power users may live and breathe Beeminder all day long, but we know how the literal bits go together, which parts are wonderful, and which are eldritch abominations.
When coming up with ideas, we will naturally discount those which are difficult to implement.
Ease of implementation is good to factor in to the decision-making but not good if it shapes what the decisions even are and what ideas are even thought of.

A trick that helps us decide what to build (we're finally at the deeper insight part now) is to pretend the thing exists and fully write up how you'd describe it to users, both new and old.
I've been surprised how much clarity I can end up with about the right thing to do by doing that.

"Pretend the thing exists and fully write up how you'd describe it to users"

There are downsides to actually publishing the posts.
When I floated the idea of
requiring a credit card in order to create a goal, everyone freaked the frack out and we were too scared to do it for years.
Then finally we did and it was immediately obvious that it was way better and no one (well, maybe literally one person) minded a bit.

But you can always write the documentation or blog post and not actually publish it until the thing it describes exists.
Which is the point: let the documentation guide what you build.
I feel like I need to say it another way to drive home the profundity of it.
Maybe in another language?
So far I'm making it sound too obvious.
I think the key, and the part that you need to try for yourself, is to put yourself in the mindset of the thing already existing and then start writing.
That forces a lot of design decisions that you'd otherwise weasel out of making by writing things like "and then we'd either do X or Y".
Not allowed!
Commit to a decision and see where the document goes.
You'll often back yourself into a corner and have to backtrack.
Which is wonderful, to be doing that before any code is written instead of after.
It's a lot like
writing a spec.

So that was the armchair quarterbacking.
Now let's actually try it.
I'm using a design question we've been
heatedly discussing:
the right way to make do-less goals have teeth.

Fictional Feature Announcement 1: Auto-Derail

This is all in italics and block-quoted so you don't forget that it's all make-believe at this point.
I mean, at least three of these are make-believe.

We've had
Pessimistic Presumptive Reports (PPRs) for seven years and they're so much better than nothing but still are so gross,
magically adding made-up data to your goals.
The problem they solve -- Beeminder goals must derail if you ignore them -- is as critical as ever but today we are announcing a better solution:
Auto-Derail.

 

Auto-Derail is like PPRs but much simpler and much more draconian:
instead of gradually derailing you by eating through your safety buffer if you stick your head in the sand, we just force-derail you immediately if you hit your deadline without having entered data for that day.
It's like an ultra-PPR, but doesn't touch your data.

 

And it doesn't have to be immediate.
Auto-Derail is specified as a number of days you're allowed to go without entering data.
By default this is zero for do-less goals, 30 for do-more goals, and 7 for weight-loss goals.
But you can set it to whatever you like.
You can effectively opt out of the whole feature, just like you can opt out of PPRs, by picking a huge number of days as your auto-derail.

 

I personally (as user-me) am excited to have a zero-day auto-derail on do-less goals like my sugar consumption.
I used to let PPRs come through and then every couple days I'd correct them (if I could remember -- otherwise I'd estimate or let the pessimistic amount stand).
Now I'm grateful for the greater discipline of a red goal screaming at me to enter my number by the deadline.
It's like any other beemergency this way.

 

What about scheduled breaks, you ask?
It's already the case that you need to turn PPRs off before going on vacation.
(Actually there's a trick you can use to give yourself safety buffer for a vacation without turning off PPRs but it's
obscure enough that it might as well not exist.
The right answer to this is a future feature we call True Breaks.)
So with auto-derail you just also have to remember to turn up your auto-derail to be more than the number of days you'll be going offline.
That's bad, having to remember to do that and especially having to remember to undo that when you come back, but since the previous status quo was as bad or worse in that regard we feel good about shipping this feature first and then improving scheduled breaks separately.

 

Bonus: This also solves a potentially frustrating corner case discovered by
Grayson Bray Morris where
a change in slope in your road can mean that PPRs take way too long to derail you. With auto-derail there's no uncertainty about how long you can go without entering data -- on any goal, not just do-less -- before derailing.
It's exactly the number of days you specified.

 

Here's a big question you probably have at this point:
How does your graph indicate how close you are to derailing?
If you're deep in the green based on how far above the the yellow brick road you are on a do-more goal, how does the graph reflect that really you're in the orange because you've almost used up your number of days before auto-derailing?
I guess I better answer it!
The answer is that we've written new code (beautiful, shimmering new code) to make all the countdowns show the minimum of the normal countdown and the auto-derail countdown and we've overlaid a ticking time bomb graphic on any graph where the auto-derail is due to happen before the regular derail.

 

Does that mean that a do-less goal will always have a time bomb on it regardless of the auto-derail setting?
Sigh.
Consistency dictates that it does.
It goes away when you enter a datapoint for today but you generally do that at the end of the day and the time bomb reappears as soon as your deadline is past and it's the new day.

Aaaand, emergency eject.
It was going so well and then things started getting pretty dicey.
If we were dead set on auto-derail we'd have our work cut out for us to flesh out that time bomb idea.
Instead, let's move on to another approach.

Fictional Feature Announcement 2: Customizably Pessimistic Presumptions

What if we doubled down on PPRs and improved them?

Pessimistic Presumptive Reports (PPRs) are critical for do-less goals but they've felt really wrong to a lot of people and for good reason:
they're
magically adding made-up data to your goals.

 

Today we are announcing a simple solution:
When you create a goal to, for example, eat less sugar, we include a step where we say,
"if you don't tell us how much sugar you ate we're sure as heck not presuming zero so tell us what we should presume!"

 

Now PPRs are both less magical and less made-up, since you pick them explicitly.
You can also change them any time in settings.

 

We encourage you -- by which I mean the UI encourages you by defaulting to it -- to pick a PPR of twice your daily rate, same as the previous status quo, because that has a nice property:
Just as with flatlining on a do-more goal, it means that you lose a day of safety buffer with each day that you ignore the goal.

 

If you want to be forced to enter data every day on your do-less goals (as I personally emphatically do) you can just choose a PPR of (in my case of dessert/candy-minding) twice your body weight in sugar or whatever amount is more than any safety buffer you might ever accumulate.

 

And you do have the choice to set your PPR to zero if you really want to.
We don't think you should and mildly discourage that by still putting the ugly PPRs in your list of datapoints regardless.
We consider implicit zeros on a do-less goal to be anathema.
If a zero is being graphed on a do-less goal, it's going to show up as such in your datapoints.
This also means you can no longer delete datapoints on do-less goals -- only edit them.

 

But as always, entering a real datapoint makes the PPR self-destruct.

 

The hard part about this was generalizing all the graphing code to understand arbitrary PPR values, as well as the now special case of "whatever twice your current daily rate is".
It's also not great how many users will likely insist on picking ill-advised PPRs and make themselves infinitely safe, which may look weird or ugly on the graph, not to mention defeating the point of Beeminder.
But we're all adults, right?
The interface makes clear that you shouldn't touch the PPR setting unless you know what you're doing.

 

Bonus: Weight loss goals now work much better!
In the old status quo if you had a very shallow yellow brick road and then fasted or got sick and found yourself well below the road, you'd suddenly have an absurd amount of safety buffer and be able to stick your head in the sand avoiding the scale while actually gaining weight.
You'd back yourself into a corner where, by the time you finally had to weigh in, you'd be hopelessly above your yellow brick road.
(Many of us resorted to meta-goals until now, a separate do-more goal for actually stepping on the scale most days.)
Now, your "max daily fluctuation" is also your PPR.
So your weight-loss goal never lets you go long without weighing in.

That went better!
(The weight loss bonus part applies to auto-derail too, of course.)

Fictional Feature Announcement 3: Meta-Goals Now Come Standard

Here's a very us approach...

We've had
Pessimistic Presumptive Reports (PPRs) for seven years but they're so inelegant.

 

Fundamentally, the problem PPRs solve is to force you to enter data on your do-less goals.
Do-more goals don't have that problem because if you don't report on a do-more goal we presume you did nothing and that eventually derails you.
PPRs were a kludge to make that be true for do-less goals as well.
If you didn't report on a do-less goal, we'd presume you did more than your daily rate.
That would eventually derail you.

 

Well, we have a much better way than that to force you to do things.
Do you know what it is?
It's Beeminder.

 

From now on, when you create a do-less goal, we automatically create a sister goal for it that ensures you enter data.

 

Isn't that overkill?
Too much clutter on your dashboard?
Confusing for newbees?
Yes, yes, and yes.
Which is why we hide the meta goal and automatically populate it.
As part of do-less goal creation we ask how many days per week you want to commit to adding data.
That generates the meta goal -- a do-more goal that gets a +1 whenever you add a datapoint to the do-less goal.

 

Then a miracle occurs.
We show you only your primary goal in most cases but give you access to the meta goal when you need it, without that being bewildering to newbees.

Ouch. Moving on.

Fictional Feature Announcement 4: Hitting Users With Hammers

I think this rounds out all the possible approaches to forcing users to do something.
We know that nagging can never suffice so there have to be some kind of consequences.

Pessimistic Presumptive Reports (PPRs) are no more!

 

How do we ensure that users don't stick their heads in the sand on their do-less goals and accumulate infinite safety buffer, you ask?

 

We simply hide all your other graphs if you end the day without entering data on any do-less goal.
Now, when you look at your dashboard, if you have any do-less goals without a datapoint for yesterday, all your goals except those goals will be blurred out.
So you have no choice but to enter the missing data first.

 

A message at the top of your dashboard, as well as on the graph page of each blurred-out goal, will explain what you need to do and link you to the most urgent do-less goal (ties broken arbitrarily) that's missing data.

 

And what if you only have do-less goals?
There are no other goals to blur out so in that case we hit you with the only other hammer we have:
After 7 days of no data, and lots of warnings, we delete the goal!
Use it or lose it, beetches.

That's sounding extreme enough to abort for now, though I'm not convinced that something like that couldn't work, if done really thoughtfully.

Back to Reality

It felt weird to write those.
I guess I'm not used to writing fiction.
Or it just weirds me out to write things that aren't true.
And at least some of those fictional feature announcements can't ever be true, since they contradict each other.
Which is the other reason it felt weird:
Writing each one I started to feel attached to it, like it was truly the Right Answer and the thing we really should do.
At least at first.
Then as I was forced to think it through well enough to cover everything users would need to know, the cracks started to appear.
Which is how I know that none of these proposals are ready to start building yet!
That seems obvious looking at the proposals now but it genuinely wasn't until I went through this exercise.

I'm now excited to debate with you all, both at the meta level, about blog-post-driven / documentation-driven / spec-driven development, and at the object level, about which of those fictional features we should actually build.

Image credit:
Faire Soule-Reeves. Thanks also to
Faire Soule-Reeves,
Adam Wolf,
Mary Renaud,
Christopher Moravec, and (as always)
Bee Soule
for reading and commenting on drafts of this.
Adam Wolf was especially helpful in thinking through all the fictional feature announcements and even co-authoring part of the intro.
Thanks also to all the
daily beemail subscribers and
forum participants who commented on an
early version (without the fictional feature announcements, though an
early version of one of those was also in the forum).

{"canonical": "https://blog.beeminder.com/docdriven/"}

Sometimes I wish I could just photosynthesize so that just by being ... I could be doing the work of the world." — Robin All Kimmerer

From the latest SWMLC

This site is has been a very incremental process - lots and lots of hard-coding where you'd expect more data-oriented, generalized systems. For example, the post title, recent posts, etc. are all produced in OCaml, rather than liquid. I'd like to change that, and bit by bit I'm getting closer to that.

In fact there's a whole list of things I'd like to change:

• Routing is hard-coded. I want to bring in Opium to be able to use the nice routing syntax, and middleware for auth, etc. However, its dependency on unix means that it can't be used with the Mirage backend. Definitely keeping an eye on the open PRs here.
• Every page is fully re-rendered on each request - Reading the index.html (template file), searching through it for the targets to replace, reading the markdown files, rendering them into html and inserting them into the html, and finally serving the page. For production, this should be memoized.
• Posts can't specify their template file - everything is just inserted into index.html. Should be trivial to change.
• The liquid parser mangles input html to the point where it significantly changes index.html. It needs to be fixed up.
• Similarly, I want to move more (e.g. some) logic into the liquid templates, for things like conditionals, loops, etc.
• Along those lines, the ReactJS bindings are very primitive, I need to come up with a small app in this site (perhaps logging in) to start exercising and building them out (with ppx extensions at some points, etc.)
• An application I'm considering is to first expose an API to update posts in dev-mode, then building a ReactJS-based editor on the frontend (draft.js is obviously a very cool tool that could be used). That way editing is a live, in-app experience, and then rendering is memoized in production. Production could even have a flag to load the dev tools given the right credentials, and allow for a GitHub PR to be created off of the changes.
• Possibly use Irmin as a storage interface for the posts.

Plenty of other things as well. I'll update this as I remember them.

As part of due diligence before introducing OCaml to our company, I've been building this site and exploring what OCaml has to offer on a lot of fronts. Now that I have a basic (sometimes terribly painful) flow in place, I've wanted to move on to slimming it down quite a bit. Especially the Mirage build + deploy process. Right now it looks like this:

1. Dev on OSX (for minutes, hours, days, weeks) until happy with the changes
2. Git push everything to master
3. Start up VirtualBox, ssh in
4. Type history to find the previous incantation
5. Build Xen artifacts
6. scp artifacts to an EC2 build machine
7. ssh into build machine.
8. Run a deploy script to turn the Xen artifacts into a running server
9. Clean up left over EC2 resources

As nice as the idea is that I can "just develop" Mirage apps on OSX, it's actually not quite true. Particularly as a beginner, it's easy to add a package as a dependency, and get stuck in a loop between steps 1 (which could be a long time depending on what I'm hacking on) and 3, as you find out that - aha! - the package isn't compatible with the Mirage stack (usually because of the dreaded unix transitive dependency).

Not only that, but I have quite a few pinned packages at this point, and I build everything in step 3 in a carefully hand-crafted virtualbox machine. The idea of manually keeping my own dev envs in sync (much less coworkers!) sounded tedious in the extreme.

At a friend's insistence I've tried out Docker for OSX. I'm very dubious about this idea, but so far it seems like it could help a bit for providing a stable dev environment for a team.

To that end, I updated to Version 1.10.3-beta5 (build: 5049), and went to work trying random commands. It didn't take too long thanks to a great overview by Amir Chaudry that saved a ton of guesswork (thanks Amir!). I started with a Mirage Docker image, unikernel / mirage, exported the opam switch config from my virtualbox side, imported it in the docker image, installed some system dependencies (openssl, dbm, etc.), and then committed the image. Seems to work a charm, and I'm relatively happy with sharing the file system across Docker/OSX (eliminates step 2 the dev iteration process). I may consider just running the server on the docker instance at this point, though that's sadly losing some of the appeal of the Mirage workflow.

Another problem with this workflow is that mirage configure --xen screws up the same makefile I use for OSX-side dev (due to the shared filesystem). So flipping back and forth isn't as seamless as I want.

So now the process is a bit shorter:

1. Dev on OSX/Docker until happy with the changes
2. Build Xen artifacts
3. scp artifacts to an EC2 build machine
4. ssh into build machine.
5. Run a deploy script to turn the Xen artifacts into a running server
6. Clean up left over EC2 resources

Already slimmed down! I'm in the process of converting the EC2 deploy script from bash to OCaml (via the previous Install OCaml AWS and dbm on OSX), so soon I'd like it to look like:

1. Dev on OSX/Docker until happy with the changes
2. git commit code, push
3. CI system picks up the new code + artifact commit, tests that it boots and binds to a port, then runs the EC2 deploy script.

I'll be pretty close to happy once that's the loop, and the last step can happen within ~20 seconds.

The OCaml web-situation is barren. Really barren.

I'm not sure if it's because the powers-that-be in the OCaml world are simply uninterested in the domain, or if it's looked down upon as "not-real development" by established/current OCaml devs, but it's a pretty dire situation. There's some movement in the right direction between Opium and Ocaml WebMachine, but both are 1.) extremely raw and 2.) pretty much completely incompatible. There's no middleware standard (Rack, Connect, or the one I'm most familiar with, Ring), so it's not easy to layer in orthogonal-but-important pieces like session-management, authentication, authorization, logging, and - relevant for today's post - error reporting.

I've worked over the past few years on ever-increasingly useful error reporting, in part because it was so terrible before, especially compared to error reports from the server-side. A few years ago, you probably wouldn't even know if your users had an error. If you worked hard, you'd get a rollbar notification that "main.js:0:0: undefined is not a function". How do you repro this case? What did the user do? What path through a (for a human) virtually unbounded state-space lead to this error? Well friend, get ready to play computer in your head, because you're on your own. I wanted to make it better, and so I worked on it in various ways, include improved source-map support in the language I was using at the time (ClojureScript), user session replay in development, predictive testing, automated repro cases, etc., until it was so nice that getting server-side errors was a terrible drag because it didn't have any of the pleasantries that I had come to be used to on the frontend.

Fast forward to this week in OCaml, when I was poking around my site, and hit a "Not found" error. The url was correct, I had just previously a top-level error handler in my Mirage code return "Not found" on any error, because I was very new to OCaml in general and that seemed to work to the extend I needed that day. But today I wanted to know what was going on - why did this happen? Googling a bit for "reporting OCaml errors in production" brought back that familiar frustration of working in an environment where devs just care (let's assume they're capable). Not much for the web, to say the least.

So I figured I would cobble together a quick solution. I didn't want to pull in an SMTP library (finding that 1. the namespacing in OCaml is fucking crazy and 2. some OPAM packages don't work with Mirage only when compiling for a non-Unix backend after developing a full feature has led me to be very cautious about any dependency) - but no worries, the ever-excellent Mailgun offers a great service to send emails via HTTP POSTs. Sadly, Cohttp can't handle multipart (e.g. form) posts (another sign of the weakness of OCaml's infrastructure compared to the excellent clj-http), so I had to do that on my own. I ended up copying the curl examples from Mailgun's, but directing the url to an http requestbin, so I could see exactly what the post looked like. Then, it was just matter of building up the examples in a utop with Cohttp bit by bit until I was able to match the exact data sent over by the curl example. From there, the last bit was to generate a random boundary to make sure there would never be a collision between form values. It's been awhile since I had to work at that level (I definitely prefer to just focus on my app and not constantly be sucked down into implementing this kind of thing), but luckily it still proved possible, if unpleasant. Here's the full module in all its glory currently:

(* Renamed from http://www.codecodex.com/wiki/Generate_a_random_password_or_random_string#OCaml *)
let gen_boundary length =
    let gen() = match Random.int(26+26+10) with
        n when n < 26 -> int_of_char 'a' + n
      | n when n < 26 + 26 -> int_of_char 'A' + n - 26
      | n -> int_of_char '0' + n - 26 - 26 in
    let gen _ = String.make 1 (char_of_int(gen())) in
    String.concat "" (Array.to_list (Array.init length gen))

let helper boundary key value =
  Printf.sprintf "%s\r\nContent-Disposition: form-data; name=\"%s\"\r\n\r\n%s\r\n" boundary key value

let send ~domain ~api_key params =
  let authorization = "Basic " ^ (B64.encode ("api:" ^ api_key)) in
  let _boundary = gen_boundary 24 in 
  let header_boundary = "------------------------" ^ _boundary in
  let boundary = "--------------------------" ^ _boundary in
  let content_type = "multipart/form-data; boundary=" ^ header_boundary in
  let form_value = List.fold_left (fun run (key, value) ->
      run ^ helper boundary key value) "" params in
  let headers = Cohttp.Header.of_list [
      ("Content-Type", content_type);
      ("Authorization", authorization)
    ] in
  let uri = (Printf.sprintf "https://api.mailgun.net/v3/%s/messages" domain) in
  let body = Cohttp_lwt_body.of_string (Printf.sprintf "%s\r\n%s--" form_value boundary) in
  Cohttp_mirage.Client.post ~headers ~body (Uri.of_string uri)

Perhaps I should expand it a bit so that it could become an OPAM package?

From there, I changed the error-handler for the site dispatcher to catch the error and send me the top level message. A bit more work, and I had a stack trace. It still wasn't quite right though, because to debug an error like this, you often need to know the context. With some help from @das_cube, I was able to serialize the request, with info like the headers, URI, etc. and send it along with the error report. The final step was to use @Drup's bootvar work (or is it Functoria? I'm not sure what the line is here) to make all of the keys configurable, so that I only send emails in production, and to a comma-separated list of email supplied either at compile- or boot-time:

let report_error exn request =
  let error = Printexc.to_string exn in
  let trace = Printexc.get_backtrace () in
  let body = String.concat "\n" [error; trace] in
  let req_text = Format.asprintf "%a@." Cohttp.Request.pp_hum request in
  ignore(
    let emails = Str.split (Str.regexp ",") (Key_gen.error_report_emails ())
                 |> List.map (fun email -> ("to", email)) in
    let params = List.append emails [
        ("from", "RiseOS (OCaml) <[email protected]>");
        ("subject", (Printf.sprintf "[%s] Exception: %s" site_title error));
        ("text", (Printf.sprintf "%s\n\nRequest:\n\n%s" body req_text))
      ]
    in
    (* TODO: Figure out how to capture context (via
       middleware?) and send as context with error email *)
    ignore(Mailgun.send ~domain:"riseos.com" ~api_key:(Key_gen.mailgun_api_key ()) params))

let dispatcher fs c request uri =
  let open Lwt.Infix in
  Lwt.catch
    (fun () ->
       let (lwt_body, content_type) = get_content c fs request uri in
       lwt_body >>= fun body ->
       S.respond_string
         ~status:`OK
         ~headers: (Cohttp.Header.of_list [("Content-Type", content_type)]) ~body ())
    (fun exn ->
       let status = `Internal_server_error in
       let error = Printexc.to_string exn in
       let trace = Printexc.get_backtrace () in
       let body = String.concat "\n" [error; trace] in
       ignore(match (Key_gen.report_errors ()) with
           | true -> report_error exn request
           | false -> ());
       match (Key_gen.show_errors ()) with
       | true -> S.respond_error ~status ~body ()
       (* If we're not showing a stacktrace, then show a nice html
          page *)
       | false -> read_fs fs "error.html" >>=
         fun body ->
         S.respond_string
           ~headers:(Cohttp.Header.of_list [("Content-Type", Magic_mime.lookup "error.html")])
           ~status
           ~body ())

It's still not anywhere near what you get for free in Rails, Clojure, etc. - and definitely not close to session-replay, predictive testing, etc. - but it's a huge step up from before!

An example error email, in all its glory:

abc do-re-mi

I've been working with Elixir for about 2 months so far, and it's been quite fun. Coming from a background in mostly TypeScript/JavaScript and Ruby, I wasn't sure how approachable I would find it.

A lot of articles I've read say that most Ruby developers would feel comfortable getting started with Elixir, but I'm not sure how much I agree with that. Aside from some superficial similarities, Elixir really forces you to think about solving problems in a slightly different way.

Over the course of my career so far, I've dabbled in programming languages unrelated to the jobs I've been paid for, but this was the first time I really learned a language by jumping right in and attempting to build a full-stack application. I'm a little ashamed to say that I've spent relatively little time going through books on Elixir, and have mostly just gone straight to hacking on our product. That being said, a lot of the opinions below come from the perspective of someone who probably hasn't written much high-quality Elixir code in production. 😬

What I like so far

Here are a few of the things that make me excited about working with Elixir. 😊

The community

This is an easy one. One of the first things I did when I started spinning up on Elixir was joining the Elixir Slack group, and it's been one of the most helpful resources for me as a beginner. The community has been nothing but friendly, patient, and supportive. When I was misusing with statements, they showed me how to refactor it. When I was starting to set up authentication, they pointed me to Pow. When I needed to set up workers, they showed me Oban. People have even been nice enough to review some of my shitty code on Github. It's been amazing.

The extensive built-in functionality

It's kind of nice just having so many useful functions built into the language. Want to flatten an array? Boom, List.flatten(). No need to import {flatten} from 'lodash'. Need to group a list of records by a given key? Boom, Enum.group_by(). I could go on and on!

I especially love that lists, maps, and ranges all implement the Enum protocol. For example, if I wanted to map over an object/map in JavaScript and double each value, I'd have to do something like:

const obj = {a: 1, b: 2, c: 3};

const result = Object.keys(obj).reduce((acc, key) => {
  return {...acc, [key]: obj[key] * 2};
}, {});

// {a: 2, b: 4, c: 6}

Whereas in Elixir, I could just do:

map = %{a: 1, b: 2, c: 3}

result = map |> Enum.map(fn {k, v} -> {k, v * 2} end) |> Map.new()

# %{a: 2, b: 4, c: 6}

Edit: Apparently there's an even easier way to handle this using Map.new/2! (Thanks to /u/metis_seeker on Reddit for the tip 😊)

Map.new(map, fn {k, v} -> {k, v * 2} end)

# %{a: 2, b: 4, c: 6}

Lastly, I love that there are methods like String.jaro_distance/2, which calculates the distance/similarity between two strings. I don't currently use it, but I could see how this might be useful for validating email address domains (e.g. [email protected] -> "Did you mean [email protected]?")

Pattern matching

Pattern matching feels like one of more powerful features Elixir offers as language. While it certainly takes some getting used to, I've found that it forces me to write cleaner code. (It's also caused me to write more case statements and much fewer if clauses than I ever have before!)

For example, if I wanted to write a method in Elixir that determines if a user has a given role (e.g. for the sake of restricting access to certain functionality), I might do something like this:

defp has_role?(nil, _roles), do: false

defp has_role?(user, roles) when is_list(roles),
  do: Enum.any?(roles, &has_role?(user, &1))

defp has_role?(%{role: role}, role), do: true

defp has_role?(_user, _role), do: false

(Note the additional use of pattern matching in the 3rd variant of has_role?/2 to check if the user.role in the 1st parameter is the same as the role provided in the 2nd parameter!)

In TypeScript, the (very rough) equivalent of the above might look something like:

const hasRole = (user: User, roleOrRoles: string | Array<string>) => {
  if (!user) {
    return false;
  }

  // This is probably not the most idiomatic TS/JS code :/
  const roles = Array.isArray(roleOrRoles) ? roleOrRoles : [roleOrRoles];

  return roles.some((role) => user.role === role);
};

Still confused? I don't blame you. Here's the Elixir code again, with some annotations:

# If the user is `nil`, return false
defp has_role?(nil, _roles), do: false

# Allow 2nd argument to be list or string; if it is a list, check
# if any of the values match by applying method recursively to each one
defp has_role?(user, roles) when is_list(roles),
  do: Enum.any?(roles, &has_role?(user, &1))

# Use pattern matching to check if the `user.role` matches the `role`
defp has_role?(%{role: role}, role), do: true

# If none of the patterns match above, fall back to return false
defp has_role?(_user, _role), do: false

This approach has taken some getting used to, but it's definitely growing on me. For example, one pattern I've started using to roll out new features (e.g. Slack notifications) is something like this:

def notify(msg), do: notify(msg, slack_enabled?())

# If Slack is not enabled, do nothing
def notify(msg, false), do: {:ok, nil}

# If it _is_ enabled, send the message
def notify(msg, true), do: Slack.post("/chat.postMessage", msg)

Not sure how idiomatic that is, but it's a nice way to avoid if blocks!

Async handling

A lot of JavaScript is conventionally handled asynchronously (non-blocking) by default. This can be a bit tricky for new programmers, but it can be quite powerful once you get the hang of it (e.g. Promise.all is a nice way to execute a bunch of async processes concurrently).

Elixir is handled synchronously (blocking) by default — which makes things much easier, in my opinion — but Elixir also happens to make it incredibly easy to handle processes asynchronously if you would like to.

As a somewhat naive example, when I was setting up our Messages API, I noticed it slowing down as we added more and more notification side effects (e.g. Slack, Webhooks) whenever a message was created. I loved that I could temporarily fix this issue by simply throwing the logic into an async process with a Task:

Task.start(fn -> Papercups.Webhooks.notify(message))

Now, this is definitely not the most ideal way to handle this. (It would probably make more sense to put it on a queue, e.g. with Oban.) But I loved how easy it was to unblock myself.

If we wanted to implement something similar to JavaScript's Promise.all, Elixir gives us something even better: control over timeouts!

tasks = [
  Task.async(fn -> Process.sleep(1000) end), # Sleep 1s
  Task.async(fn -> Process.sleep(4000) end), # Sleep 4s
  Task.async(fn -> Process.sleep(7000) end)  # Sleep 7s, will timeout
]

tasks
|> Task.yield_many(5000) # Set timeout limit to 5s
|> Enum.map(fn {t, res} -> res || Task.shutdown(t, :brutal_kill) end)

This allows us to shutdown any processes that are taking longer than expected. 🔥

The pipe operator

It's almost as if any blog post introducing Elixir is obligated to mention this, so here we are.

Let's just take an example directly from the Papercups codebase. In one of our modules, we do some email validation by checking the MX records of the given domain. Here's how it looks in Elixir:

defp lookup_all_mx_records(domain_name) do
  domain_name
  |> String.to_charlist()
  |> :inet_res.lookup(:in, :mx, [], max_timeout())
  |> normalize_mx_records_to_string()
end

If I wanted to write this in TypeScript, I would probably do something like:

const lookupAllMxRecords = async (domain: string) => {
  const charlist = domain.split('');
  const records = await InetRes.lookup(charlist, opts);
  const normalized = normalizeMxRecords(records);

  return normalized;
};

There's nothing inherently wrong with that, but pipes save us some unhelpful variable declarations, and produce code that is arguably just as readable!

I think the thing people like most about the pipe operator is that it both looks cool AND improves (or at least doesn't detract from) readability. But mostly it just looks cool. 🤓

Since I wasn't able to write anything particulary intelligent about pipes, I'll leave this section with a quote from Saša Juric's "Elixir in Action":

The pipeline operator highlights the power of functional programming. You treat functions as data transformations and then combine them in different ways to gain the desired effect.

Immutability

I can't tell you how many times I've been writing JavaScript and forgotten that calling .reverse() or .sort() on an array actually mutates the original value. (This almost screwed me over in my last technical interview, embarrassingly enough.)

For example:

> const arr = [1, 6, 2, 5, 3, 4];
> arr.sort().reverse()
[ 6, 5, 4, 3, 2, 1 ]
> arr
[ 6, 5, 4, 3, 2, 1 ] // arr was mutated 👎

I love that in Elixir, everything is immutable by default. So if I define a list, and want to reverse or sort it, the original list never changes:

iex(12)> arr = [1, 6, 2, 5, 3, 4]
[1, 6, 2, 5, 3, 4]
iex(13)> arr |> Enum.sort() |> Enum.reverse()
[6, 5, 4, 3, 2, 1]
iex(14)> arr
[1, 6, 2, 5, 3, 4] # nothing has changed 👌

Hooray! This makes the code much more predictable.

Dealing with strings

I love that there are so many ways to format and interpolate strings in Elixir. This might be a bit of a niche use case, but the triple-quote """ approach has been super useful for email text templates, since it removes all the preceding whitespace from each line:

def welcome_email_text(name) do
  """
  Hi #{name}!

  Thanks for signing up for Papercups :)

  Best,
  Alex
  """
end

If I wanted to do this in TypeScript, I'd have to do something like:

const welcomeEmailText = (name: string) => {
  return `
Hi ${name}!

Thanks for signing up for Papercups :)

Best,
Alex
  `.trim();
};

Which just looks... awkward.

What I'm... still getting used to

I almost called this section, "What I dislike so far", but I thought that would be a little unfair. Just because I'm not accustomed to certain ways of thinking doesn't mean I have to hate on it.

So without further ado, here are some of the things I'm still getting used to with Elixir. 😬

Error handling

One of the first things I noticed when I started to dip my toes in Elixir was the prevalence of methods returning {:ok, result}/{:error, reason} tuples. I didn't give it much thought at first, and found myself writing a lot of code that looked like:

{:ok, foo} = Foo.retrieve(foo_id)
{:ok, bar} = Bar.retrieve(bar_id)
{:ok, baz} = Baz.retrieve(baz_id)

...and then got hit with a bunch of MatchErrors.

As you might have guessed (if you've written any Elixir), this led me to start getting a little overly enthusastic about the with statement. Which if you haven't written any Elixir, looks something like this:

with {:ok, foo} <- Foo.retrieve(foo_id),
     {:ok, bar} <- Bar.retrieve(bar_id),
     {:ok, baz} <- Baz.retrieve(baz_id) do
  # Do whatever, as long as all 3 methods above execute without error
else
  error -> handle_error(error)
end

There's nothing particularly wrong with that, but I've also found myself writing some methods that basically just extract the result portion of the {:ok, result} tuple, which feels a little silly:

case Foo.retrieve(foo_id) do
  {:ok, foo} -> foo
  error -> error
end

(It's very possible that the above code is an antipattern, and I'm simply not handling things correctly.)

Anyway, on one hand, I feel like this convention of the language is good because it forces programmers to be more cognizant of error handling. But on the other hand, it definitely takes some getting used to.

Implicit returns (and no return keyword)

While pattern matching is great and all, the fact that Elixir does not have the ability to break out of a function early can be a bit frustrating as a beginner.

For example, if I wanted to write a function to compute the total cost of a bill in TypeScript, I might do something like:

const calculateTotalPrice = (bill: Bill) => {
  if (!bill) {
    return 0;
  }

  const {prices = []} = bill;

  // This is a little unnecessary, but illustrates the point of
  // a second reason we may want to return early in a function
  if (prices.length === 0) {
    return 0;
  }

  return prices.reduce((total, price) => total + price, 0);
};

The code above allows me to break early and return 0 under certain circumstances (e.g. when bill is null, or prices is an empty list).

Elixir solves this with pattern matching (as we've discussed in more detail above).

def calculate_total_price(nil), do: 0

def calculate_total_price(%{prices: prices}) when is_list(prices),
  do: Enum.sum(prices)

def calculate_total_price(_bill), do: 0

For someone approaching Elixir as a newbie like myself, this can take some getting used to, because it forces you to take a step back and rethink how you would normally design your functions.

Dialyzer and the development experience

There's not much to say here, other than that Dialyzer can be pretty frustrating to deal with at times. Sometimes it's just slow, and warnings take a few seconds to pop up... this is annoying when I: change some code to fix a warning; the warning goes away for a few seconds; I feel good about myself for having fixed it; and then boom, another warning pops up.

Other times, the warnings and just cryptic or confusing:

(I have no idea what this means...)

Debugging macros

When I was starting off with the Pow library to implement auth, I ran into Elixir macros for the first time. I felt like such an idiot trying to figure out where the pow_password_changeset method was defined, until I finally found this piece of code:

@changeset_methods [:user_id_field_changeset, :password_changeset, :current_password_changeset]

# ...

for method <- @changeset_methods do
  pow_method_name = String.to_atom("pow_#{method}")

  quote do
    @spec unquote(pow_method_name)(Ecto.Schema.t() | Changeset.t(), map()) :: Changeset.t()
    def unquote(pow_method_name)(user_or_changeset, attrs) do
      unquote(__MODULE__).Changeset.unquote(method)(user_or_changeset, attrs, @pow_config)
    end
  end
end

It's pretty cool that Elixir supports macros, but the syntax and the idea of dynamically generating methods is not something I've ever had to deal with. But I'm excited to try it out!

Dealing with JSON

Honestly, I feel like this is true for most languages (other than JavaScript/TypeScript). Since most maps in Elixir use atoms for keys, I've found myself accidentally mixing atom/string keys when I'm unknowing working with a map that has been decoded from JSON.

Unclear trajectory of the language

I honestly have no idea whether Elixir is growing, stagnating, or declining in popularity, but so far things seem much more enjoyable and less painful than I expected.

When we first started building Papercups in Elixir, a few people warned us that the lack of libraries and support would make it much harder to move quickly. While it's clear that the amount of open source libraries is much lower compared to languages like JavaScript, Ruby, Python, and Go, so far this hasn't been a huge issue.

As more well-known companies (e.g. WhatsApp, Discord, Brex) begin using Elixir in production, I'm hoping developer adoption continues to grow. I'm optimistic! 😊

That's all for now!

If you're interested in contributing to an open source Elixir project, come check out Papercups on Github!

{"canonical": "https://papercups.io/blog/elixir-noob"}

{"canonical": "https://stopa.io/post/222"}

Many years ago, Peter Norvig wrote a beautiful article about creating a lisp interpreter in Python. It’s the most fun tutorial I’ve seen, not just because it teaches you about my favorite language family (Lisp), but because it cuts through to the essence of interpreters, is fun to follow and quick to finish.

Recently, I had some time and wanted to learn Rust. It’s a beautiful systems language, and I’ve seen some great work come out from those who adopt it. I thought, what better way to learn Rust, than to create a lisp interpreter in it?

Hence, Risp — a lisp in rust — was born. In this essay you and I will follow along with Norvig’s Lispy, but instead of Python, we’ll do it in Rust 🙂.

Syntax, Semantics and Notes on Following Along

If you haven’t heard of lisp, some Paul Graham’s essays (one, two, three), alongside some Rich Hickey talks will get you fired up. In short, everything is a list, everything is an expression, and that makes for a very powerful language.

Our structure will be similar to Norvig’s tutorial, though I depart slightly in two ways:

1. Instead of 2 stopping points (Lispy Calculator and Full Lispy), we have 4 stopping points. This reflects the phases I took to build it in Rust.
2. Norvig’s syntax is based on Scheme. We will base it on Scheme too, but since I’m also a Clojure fan, I sometimes used slightly different naming, and different implementations for a few functions. I will note when I do that in the essay.

Finally, this is the first program I wrote in Rust. I may have misused some things, so if you’re a Rust hacker, I’d love to hear your feedback 🙂.

With the notes out of the way, let’s get into it.

Language 1: Just a Risp calculator

As Norvig suggests, our first goal is to create a subset of lisp, that can do what a basic calculator can do.

To make it as simple as possible to follow, for language 1, we’ll only support addition and subtraction. No variable definitions, no if statements, nada.

This departs a bit from Lispy, but I found this stopping point a lot more convenient when writing it in Rust. So, our goal:

(+ 10 5 2)//=> 17
(- 10 5 2) //=> 3

The important process we need to remember is the flow of an interpreter:

our program ⟶ parse ⟶ abstract syntax tree ⟶ eval ⟶ result

We will need to parse our program and convert it into an abstract syntax tree. After that, we can eval the abstract syntax tree and get our result. (Refer to Norvig’s article for more detailed definitions and explanations).

Type Definitions

Risp can have three kinds of values for now:

#[derive(Clone)]
enum RispExp {
  Symbol(String),
  Number(f64),
  List(Vec<RispExp>),
} 

We’ll also need an error type. We’ll keep this simple, but if you’re curious there is a more robust approach.

#[derive(Debug)]
enum RispErr {
  Reason(String),
}

Finally, we’ll need an environment type. This is where we will store defined variables, built-in functions, and so forth:

#[derive(Clone)]
struct RispEnv {
  data: HashMap<String, RispExp>,
}

Parsing

Our goal is to take our program, and build an abstract syntax tree from it. For us, that is going to be a RispExp. To do this, first we will take our program, and cut it up into a bunch of tokens:

tokenize("(+ 10 5)") //=> ["(", "+", "10", "5", ")"]

Here’s how we can do that in Rust:

fn tokenize(expr: String) -> Vec<String> {
  expr
    .replace("(", " ( ")
    .replace(")", " ) ")
    .split_whitespace()
    .map(|x| x.to_string())
    .collect()
}

Then, we can parse these tokens, into a RispExp:

fn parse<'a>(tokens: &'a [String]) -> Result<(RispExp, &'a [String]), RispErr> {
  let (token, rest) = tokens.split_first()
    .ok_or(
      RispErr::Reason("could not get token".to_string())
    )?;
  match &token[..] {
    "(" => read_seq(rest),
    ")" => Err(RispErr::Reason("unexpected `)`".to_string())),
    _ => Ok((parse_atom(token), rest)),
  }
}

Note: I depart slightly from Norvig’s implementation, by returning the “next” slice. This lets us recurse and parse nested lists, without mutating the original list.

We get the token for the current position. If it’s the beginning of a list “(“, we start reading and parsing the tokens that follow, until we hit a closing parenthesis:

fn read_seq<'a>(tokens: &'a [String]) -> Result<(RispExp, &'a [String]), RispErr> {
  let mut res: Vec<RispExp> = vec![];
  let mut xs = tokens;
  loop {
    let (next_token, rest) = xs
      .split_first()
      .ok_or(RispErr::Reason("could not find closing `)`".to_string()))
      ?;
    if next_token == ")" {
      return Ok((RispExp::List(res), rest)) // skip `)`, head to the token after
    }
    let (exp, new_xs) = parse(&xs)?;
    res.push(exp);
    xs = new_xs;
  }
}

If it’s a closing tag of a list “)”, we return an error, as read_seq should have skipped past it.

Otherwise, it can only be an atom, so we parse that:

fn parse_atom(token: &str) -> RispExp {      
  let potential_float: Result<f64, ParseFloatError> = token.parse();
  match potential_float {
    Ok(v) => RispExp::Number(v),
    Err(_) => RispExp::Symbol(token.to_string().clone())
  }
}

Environment

Let’s go ahead and create the default, global environment. As Norvig explains, environments are where we will store variable definitions and built-in functions.

To implement built-in operations (+, -), we need a way to save rust function references. Let’s update RispExp, so that we can store rust function references:

#[derive(Clone)]
enum RispExp {
  Symbol(String),
  Number(f64),
  List(Vec<RispExp>),
  Func(fn(&[RispExp]) -> Result<RispExp, RispErr>), // bam
}

Then, we can create a default_env function, that returns a RispEnv, which implements +, and -

fn default_env() -> RispEnv {
  let mut data: HashMap<String, RispExp> = HashMap::new();
  data.insert(
    "+".to_string(), 
    RispExp::Func(
      |args: &[RispExp]| -> Result<RispExp, RispErr> {
        let sum = parse_list_of_floats(args)?.iter().fold(0.0, |sum, a| sum + a);
        
        Ok(RispExp::Number(sum))
      }
    )
  );
  data.insert(
    "-".to_string(), 
    RispExp::Func(
      |args: &[RispExp]| -> Result<RispExp, RispErr> {
        let floats = parse_list_of_floats(args)?;
        let first = *floats.first().ok_or(RispErr::Reason("expected at least one number".to_string()))?;
        let sum_of_rest = floats[1..].iter().fold(0.0, |sum, a| sum + a);
        
        Ok(RispExp::Number(first - sum_of_rest))
      }
    )
  );
  
  RispEnv {data}
}

Note: I am following Clojure’s spec for + and -.

To make this simpler, I made a quick helper, which enforces that all RispExp that we receive are floats:

fn parse_list_of_floats(args: &[RispExp]) -> Result<Vec<f64>, RispErr> {
  args
    .iter()
    .map(|x| parse_single_float(x))
    .collect()
}

fn parse_single_float(exp: &RispExp) -> Result<f64, RispErr> {
  match exp {
    RispExp::Number(num) => Ok(*num),
    _ => Err(RispErr::Reason("expected a number".to_string())),
  }
}

Evaluation

Now, time to implement eval.

If it’s a symbol, we’ll query for that symbol in the environment and return it (for now, it should be a RispExp::Func)

If it’s a number, we’ll simply return it.

If it’s a list, we’ll evaluate the first form. It should be a RispExp::Func. Then, we’ll call that function with all the other evaluated forms as the arguments.

fn eval(exp: &RispExp, env: &mut RispEnv) -> Result<RispExp, RispErr> {
  match exp {
    RispExp::Symbol(k) =>
        env.data.get(k)
        .ok_or(
          RispErr::Reason(
            format!("unexpected symbol k='{}'", k)
          )
        )
        .map(|x| x.clone())
    ,
    RispExp::Number(_a) => Ok(exp.clone()),
    RispExp::List(list) => {
      let first_form = list
        .first()
        .ok_or(RispErr::Reason("expected a non-empty list".to_string()))?;
      let arg_forms = &list[1..];
      let first_eval = eval(first_form, env)?;
      match first_eval {
        RispExp::Func(f) => {
          let args_eval = arg_forms
            .iter()
            .map(|x| eval(x, env))
            .collect::<Result<Vec<RispExp>, RispErr>>();
          f(&args_eval?)
        },
        _ => Err(
          RispErr::Reason("first form must be a function".to_string())
        ),
      }
    },
    RispExp::Func(_) => Err(
      RispErr::Reason("unexpected form".to_string())
    ),
  }
}

Aand, bam, we have eval.

Repl

Now, to make this fun and interactive, let’s make a repl.

We first need a way to convert our RispExp to a string. Let’s implement the Display trait

impl fmt::Display for RispExp {
  fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
    let str = match self {
      RispExp::Symbol(s) => s.clone(),
      RispExp::Number(n) => n.to_string(),
      RispExp::List(list) => {
        let xs: Vec<String> = list
          .iter()
          .map(|x| x.to_string())
          .collect();
        format!("({})", xs.join(","))
      },
      RispExp::Func(_) => "Function {}".to_string(),
    };
    
    write!(f, "{}", str)
  }
}

Then, let’s tie the interpreter process into a loop

fn parse_eval(expr: String, env: &mut RispEnv) -> Result<RispExp, RispErr> {
  let (parsed_exp, _) = parse(&tokenize(expr))?;
  let evaled_exp = eval(&parsed_exp, env)?;
  
  Ok(evaled_exp)
}

fn slurp_expr() -> String {
  let mut expr = String::new();
  
  io::stdin().read_line(&mut expr)
    .expect("Failed to read line");
  
  expr
}

fn main() {
  let env = &mut default_env();
  loop {
    println!("risp >");
    let expr = slurp_expr();
    match parse_eval(expr, env) {
      Ok(res) => println!("// 🔥 => {}", res),
      Err(e) => match e {
        RispErr::Reason(msg) => println!("// 🙀 => {}", msg),
      },
    }
  }
}

Aand, voila, language 1.0 is done. Here’s the code so far 🙂

We can now add and subtract!

risp >
(+ 10 5 (- 10 3 3))
// 🔥 => 19

Language 1.1: Risp calculator++

Okay, we have a basic calculator. Now, let’s add support for booleans, and introduce some equality comparators.

To implement bools, let’s include it in our RispExp

#[derive(Clone)]
enum RispExp {
  Bool(bool), // bam
  Symbol(String),
  Number(f64),
  List(Vec<RispExp>),
  Func(fn(&[RispExp]) -> Result<RispExp, RispErr>),
}

Rust will tell us to update Display

impl fmt::Display for RispExp {
  fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
    let str = match self {
      RispExp::Bool(a) => a.to_string(),

Then Rust will tell us we should change eval, to consider bools:

fn eval(exp: &RispExp, env: &mut RispEnv) -> Result<RispExp, RispErr> {
  match exp {
    ...
    RispExp::Bool(_a) => Ok(exp.clone()),

Let’s also update our parse_atom function, to consider bools:

fn parse_atom(token: &str) -> RispExp {
  match token.as_ref() {
    "true" => RispExp::Bool(true),
    "false" => RispExp::Bool(false),
    _ => {
      let potential_float: Result<f64, ParseFloatError> = token.parse();
      match potential_float {
        Ok(v) => RispExp::Number(v),
        Err(_) => RispExp::Symbol(token.to_string().clone())
      }
    }
  }
}

Now, we should be good to go. To really see these in action though, let’s implement =, >, <, >=, <=

Comparison Operators

In clojure, these comparison operators are a bit special. They can take more than 2 args, and return true if they are in a monotonic order that satisfies the operator.

For example (> 6 5 3 2) is true, because 6 > 5 > 3 > 2. Let’s do this for Risp:

fn default_env() -> RispEnv {
  let mut data: HashMap<String, RispExp> = HashMap::new();
  ...
  data.insert(
    "=".to_string(), 
    RispExp::Func(ensure_tonicity!(|a, b| a == b))
  );
  data.insert(
    ">".to_string(), 
    RispExp::Func(ensure_tonicity!(|a, b| a > b))
  );
  data.insert(
    ">=".to_string(), 
    RispExp::Func(ensure_tonicity!(|a, b| a >= b))
  );
  data.insert(
    "<".to_string(), 
    RispExp::Func(ensure_tonicity!(|a, b| a < b))
  );
  data.insert(
    "<=".to_string(), 
    RispExp::Func(ensure_tonicity!(|a, b| a <= b))
  );
  
  RispEnv {data}
}

The key here is our helper macro ensure_tonicty. This takes a checker function, and ensures that the conditional passes in a monotonic way:

macro_rules! ensure_tonicity {
  ($check_fn:expr) => {{
    |args: &[RispExp]| -> Result<RispExp, RispErr> {
      let floats = parse_list_of_floats(args)?;
      let first = floats.first().ok_or(RispErr::Reason("expected at least one number".to_string()))?;
      let rest = &floats[1..];
      fn f (prev: &f64, xs: &[f64]) -> bool {
        match xs.first() {
          Some(x) => $check_fn(prev, x) && f(x, &xs[1..]),
          None => true,
        }
      };
      Ok(RispExp::Bool(f(first, rest)))
    }
  }};
}

Aand, voila, language 1.1 is done. Here’s the code so far 🙂

We can now use comparators, and see booleans!

risp >
(> 6 4 3 1)
// 🔥 => true

Language 1.2: Almost Risp

Okay, now, let’s make this a language. Let’s introduce def and if.

To do this, let’s update eval to deal with built-in operators:

fn eval(exp: &RispExp, env: &mut RispEnv) -> Result<RispExp, RispErr> {
  match exp {
    ...
    RispExp::List(list) => {
      let first_form = list
        .first()
        .ok_or(RispErr::Reason("expected a non-empty list".to_string()))?;
      let arg_forms = &list[1..];
      match eval_built_in_form(first_form, arg_forms, env) {
        Some(res) => res,
        None => {
          let first_eval = eval(first_form, env)?;
          match first_eval {
            RispExp::Func(f) => {
              let args_eval = arg_forms
                .iter()
                .map(|x| eval(x, env))
                .collect::<Result<Vec<RispExp>, RispErr>>();
              return f(&args_eval?);
            },
            _ => Err(
              RispErr::Reason("first form must be a function".to_string())
            ),
          }
        }
      }
    },

We take the first form, and try to eval it as a built-in. If we can, voila, otherwise we evaluate as normal.

Here’s how eval_built_in_form looks:

fn eval_built_in_form(
  exp: &RispExp, arg_forms: &[RispExp], env: &mut RispEnv
) -> Option<Result<RispExp, RispErr>> {
  match exp {
    RispExp::Symbol(s) => 
      match s.as_ref() {
        "if" => Some(eval_if_args(arg_forms, env)),
        "def" => Some(eval_def_args(arg_forms, env)),
        _ => None,
      }
    ,
    _ => None,
  }
}

if

Here’s how we can implement if:

fn eval_if_args(arg_forms: &[RispExp], env: &mut RispEnv) -> Result<RispExp, RispErr> {
  let test_form = arg_forms.first().ok_or(
    RispErr::Reason(
      "expected test form".to_string(),
    )
  )?;
  let test_eval = eval(test_form, env)?;
  match test_eval {
    RispExp::Bool(b) => {
      let form_idx = if b { 1 } else { 2 };
      let res_form = arg_forms.get(form_idx)
        .ok_or(RispErr::Reason(
          format!("expected form idx={}", form_idx)
        ))?;
      let res_eval = eval(res_form, env);
      
      res_eval
    },
    _ => Err(
      RispErr::Reason(format!("unexpected test form='{}'", test_form.to_string()))
    )
  }
}

def

And here’s def:

fn eval_def_args(arg_forms: &[RispExp], env: &mut RispEnv) -> Result<RispExp, RispErr> {
  let first_form = arg_forms.first().ok_or(
    RispErr::Reason(
      "expected first form".to_string(),
    )
  )?;
  let first_str = match first_form {
    RispExp::Symbol(s) => Ok(s.clone()),
    _ => Err(RispErr::Reason(
      "expected first form to be a symbol".to_string(),
    ))
  }?;
  let second_form = arg_forms.get(1).ok_or(
    RispErr::Reason(
      "expected second form".to_string(),
    )
  )?;
  if arg_forms.len() > 2 {
    return Err(
      RispErr::Reason(
        "def can only have two forms ".to_string(),
      )
    )
  } 
  let second_eval = eval(second_form, env)?;
  env.data.insert(first_str, second_eval);
  
  Ok(first_form.clone())
}

Aand bam, language 1.2 is done. Here’s the code so far 🙂

We now have some coool built-in functions.

risp >
(def a 1)
// 🔥 => a
risp >
(+ a 1)
// 🔥 => 2
risp >
(if (> 2 4 6) 1 2)
// 🔥 => 2
risp >
(if (< 2 4 6) 1 2)
// 🔥 => 1

Language 2: Full Risp

Now, let’s make this a full-on language. Let’s implement _lambdas_! Our syntax can look like this:

(def add-one (fn (a) (+ 1 a)))
(add-one 1) // => 2

First, create the lambda expression

First things first, let’s introduce a Lambda type for our RispExp

#[derive(Clone)]
enum RispExp {
  Bool(bool),
  Symbol(String),
  Number(f64),
  List(Vec<RispExp>),
  Func(fn(&[RispExp]) -> Result<RispExp, RispErr>),
  Lambda(RispLambda) // bam
}

#[derive(Clone)]
struct RispLambda {
  params_exp: Rc<RispExp>,
  body_exp: Rc<RispExp>,
}

Rust will tell us to update Display:

impl fmt::Display for RispExp {
  fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
    let str = match self {
      ...
      RispExp::Lambda(_) => "Lambda {}".to_string(),

Then Rust will tell us to update eval:

fn eval(exp: &RispExp, env: &mut RispEnv) -> Result<RispExp, RispErr> {
  match exp {
    ...
    RispExp::Lambda(_) => Err(RispErr::Reason("unexpected form".to_string())),

Then, support the built-in constructor

Now, let’s update eval, to handle fn — this will be the built-in call that creates a Lambda expression:

fn eval_built_in_form(
  exp: &RispExp, arg_forms: &[RispExp], env: &mut RispEnv
        ...
        "fn" => Some(eval_lambda_args(arg_forms)),

eval_lambda_args can look like this:

fn eval_lambda_args(arg_forms: &[RispExp]) -> Result<RispExp, RispErr> {
  let params_exp = arg_forms.first().ok_or(
    RispErr::Reason(
      "expected args form".to_string(),
    )
  )?;
  let body_exp = arg_forms.get(1).ok_or(
    RispErr::Reason(
      "expected second form".to_string(),
    )
  )?;
  if arg_forms.len() > 2 {
    return Err(
      RispErr::Reason(
        "fn definition can only have two forms ".to_string(),
      )
    )
  }
  
  Ok(
    RispExp::Lambda(
      RispLambda {
        body_exp: Rc::new(body_exp.clone()),
        params_exp: Rc::new(params_exp.clone()),
      }
    )
  )
}

Then, let’s support scoped environments

For now we only have a global environment. To support lambdas, we need to introduce the concept of scoped environments. Whenever we call a lambda, we’ll need to instantiate a new environment.

To do this, let’s first update our RispEnv struct, to keep an outer reference:

#[derive(Clone)]
struct RispEnv<'a> {
  data: HashMap<String, RispExp>,
  outer: Option<&'a RispEnv<'a>>,
}

Let’s update default_env, to specify the lifetime and return None as the outer environment:

fn default_env<'a>() -> RispEnv<'a> {
  ... 
  RispEnv {data, outer: None}
}

Then, let’s update eval, to recursively search for symbols in our environment:

fn env_get(k: &str, env: &RispEnv) -> Option<RispExp> {
  match env.data.get(k) {
    Some(exp) => Some(exp.clone()),
    None => {
      match &env.outer {
        Some(outer_env) => env_get(k, &outer_env),
        None => None
      }
    }
  }
}

fn eval(exp: &RispExp, env: &mut RispEnv) -> Result<RispExp, RispErr> {
  match exp {
    RispExp::Symbol(k) =>
      env_get(k, env)
      .ok_or(
        RispErr::Reason(
          format!("unexpected symbol k='{}'", k)
        )
      )
    ,

Finally, let’s support calling lambdas

Let’s update eval, so that we know what to do when the first form in a list is a lambda:

fn eval(exp: &RispExp, env: &mut RispEnv) -> Result<RispExp, RispErr> {
          ...
          let first_eval = eval(first_form, env)?;
          match first_eval {
            RispExp::Func(f) => {
              f(&eval_forms(arg_forms, env)?)
            },
            RispExp::Lambda(lambda) => {
              let new_env = &mut env_for_lambda(lambda.params_exp, arg_forms, env)?;
              eval(&lambda.body_exp, new_env)
            },
            _ => Err(
              RispErr::Reason("first form must be a function".to_string())
            ),
          }

We first have a quick helper function to eval a list of expressions, as we’ll be doing that both for RispExp::Func and RispExp::Lambda

fn eval_forms(arg_forms: &[RispExp], env: &mut RispEnv) -> Result<Vec<RispExp>, RispErr> {
  arg_forms
    .iter()
    .map(|x| eval(x, env))
    .collect()
}

Then, we create a function call env_for_lambda. This will get the params_exp, and create an environment, where each param corresponds to the argument at that index:

fn env_for_lambda<'a>(
  params: Rc<RispExp>, 
  arg_forms: &[RispExp],
  outer_env: &'a mut RispEnv,
) -> Result<RispEnv<'a>, RispErr> {
  let ks = parse_list_of_symbol_strings(params)?;
  if ks.len() != arg_forms.len() {
    return Err(
      RispErr::Reason(
        format!("expected {} arguments, got {}", ks.len(), arg_forms.len())
      )
    );
  }
  let vs = eval_forms(arg_forms, outer_env)?;
  let mut data: HashMap<String, RispExp> = HashMap::new();
  for (k, v) in ks.iter().zip(vs.iter()) {
    data.insert(k.clone(), v.clone());
  }
  Ok(
    RispEnv {
      data,
      outer: Some(outer_env),
    }
  )
}

To do this, we need the helper parse_list_of_symbol_strings, to make sure all of our param definitions are in fact symbols:

fn parse_list_of_symbol_strings(form: Rc<RispExp>) -> Result<Vec<String>, RispErr> {
  let list = match form.as_ref() {
    RispExp::List(s) => Ok(s.clone()),
    _ => Err(RispErr::Reason(
      "expected args form to be a list".to_string(),
    ))
  }?;
  list
    .iter()
    .map(
      |x| {
        match x {
          RispExp::Symbol(s) => Ok(s.clone()),
          _ => Err(RispErr::Reason(
            "expected symbols in the argument list".to_string(),
          ))
        }   
      }
    ).collect()
}

With that, we can eval(lambda.body_exp, new_env), and…

Voila…language 2.0 is done. Take a look at the code so far 🙂

We now support lambdas!

risp >
(def add-one (fn (a) (+ 1 a)))
// 🔥 => add-one
risp >
(add-one 1)
// 🔥 => 2

Fin

And with that, we’ve reached the end of this adventure. I hope it’s been fun!

There’s still a bunch more to implement, and ways we can make this even more elegant. If you get to it, send me your thoughts 🙂.

Finally, I have to say, I loved using Rust. It’s the least mental overhead I’ve had to maintain with a systems language, and it was a blast to use. The community is alive and well, plus — their guides are phenomenal! Give it a shot if you haven’t already.

If you liked this post, please share it. For more posts and thoughts, follow me on twitter 🙂.

Special thanks to Mark Shlick, Taryn Hill, Kaczor Donald, for reviewing this essay.

Thanks to eridius for suggesting a cleaner implementation of parse Thanks to thenewwazoo for suggesting a better way to do error handling Thanks to phil_gk for suggesting the use the Display trait

{"source":"medium","postId":"90a0dad5b116","publishedDate":1556819077154,"url":"https://m.stopa.io/risp-lisp-in-rust-90a0dad5b116"}

The goal is to get a good portion of this down to a static site generator, along the lines of jekyll. I have a super hacky implementation of Liquid Templating working - it mangles input html, but it works enough to write this post!

In fact, this post is powered by:

• OCaml
• Mirage
• Markdown via (the quite nice) Omd
• A custom Liquid templating parser built using Menhir.
• Super unsafe Reactjs js bindings written in OCaml and compiled down via js_of_ocaml
• Xen hypervisor

Example of template being filled in: post.author -> {{post.author}} <- Filled in by the usual liquid templating [[post.author]] (not actually [, but I can not escape the curly brace yet)

The workflow is that context (the data that fills in holds like the above [[post.author]]) is hardcoded in OCaml. While running locally, everything is completely re-rendered on every load. In production (or in unikernel mode), the page is rendered on the first load, and then cached indefinitely for performance. At some point I would like to be able to enumerate over the Opium routes and generate a final html output for each possible route, so that the artifacts themselves could be deployed to e.g. AWS S3. Any non-resource/dynamic routes could still fallback to hitting the unikernel in order to get the best of both worlds for free - as much pre-generated (cacheable) static output as possible, with the ability to make portions dynamic at will, all while writing in the same OCaml way.

I also would like to have a JSON (or perhaps EDN) header so that context could be provided by the post for use elsewhere in the template (e.g. have the sidebar title/tags defined by the blog post markdown file) - move as much out of OCaml as possible, while still keeping type safety, etc.

Still missing:

• Support for liquid control structures, e.g. if, |, etc.
• Full support for existing jekyll templates - importing them should eventually be possible
• HTTP/JSON endpoints. I am hoping to use Opium, but it has some transitive dependency on Unix (through Core), and looks like it may take more effort to port off of (in order to use with a Xen backend).
• Safe and convenient Reactjs bindings - it is hellish writing them right now

Below, you can see the Reactjs app written in OCaml running and creating clickable buttons. I have a custom watcher that recompiles the entire OCaml dependency change into js whenever a relevant file is changed - it happens fairly quickly right now, so it is not too painful, but I certainly hope something like incremental compilation is possible in the near future.

Be sure to subscribe to my blog for more content.

Before we get into the details of what localizer is and how it came to be, it's crucial that we look at what developer environments were and the motivations behind the ones I create.

What is the Goal of a Developer Environment?

Ever since I wrote my first ever developer environment for the now-defunct StayMarta, I've always focused on one thing: ease of use. For a docker-compose based development environment, this was a relatively simple task. Create Docker containers, write a compose file, declare ports, and they'd be available on my local machine. It was as simple as docker-compose up. While this approach didn't necessarily match what production looked like, it was really the best option available for containers at the time. Options that exist now, such as Kubernetes, weren't available, which rendered equality between production and development a work-in-progress dream.

Scaling Up

Fast forward to 2017, at my first larger-scale startup, Azuqua, I had a chance to reimagine what a developer environment looked like under a whole new set of constraints. While Docker Compose works for small teams, it falls apart when you map it to production systems. At the time, our production system was based on Chef. Docker Compose doesn't map to Ruby-based Chef configuration files. When I joined Azuqua, the pain around having separate tooling for infrastructure had become incredibly clear. It wasn't sustainable to have an entire team; write the configuration for our services, communicate with developers why infrastructure can't infinitely scale without good software design, and do it all without blame or single points of failure. Fundamentally this is why I take issue with DevOps teams and prefer promoting the Google SRE Model instead. While at Azuqua, we started a transition to an SRE model and used Kubernetes as a way to help facilitate that.

Introducing Kubernetes

While at Azuqua, I identified the need to run Kubernetes to ease cloud resources' scalability and improve the developer experience. At the same time, this drastically decreased the developer experience. While this may seem contradictory at first, it's essential to consider the multiple ways that developer experience presents itself. While at first, it may seem like it's just tooling to test/write code, it's a combination of testing code, writing code, and deploying that code. With the Docker Compose environment at StayMarta, we made the test/build cycle incredibly simple but shifted the deploy aspects onto a bespoke team, the DevOps model. That approach works for small teams, but as you grow, this quickly doesn't scale. If you can't deploy code efficiently, the developer experience is frustrating and promptly turns to an unhealthy relationship with the team responsible for that cycle.

So, how exactly does Kubernetes make this better then?

Going back to how Kubernetes improved the developer experience, which I assure you it does. The benefit to Kubernetes was that it brought a control plane into the mix and focused on being just a container orchestrator. The DSL it does have is highly specific to orchestrating containers and making them work. The control-plane allows self-healing and building tooling to enable all different types of use-cases consistently. While it lacks abstractions, it brings something to developers they've never had before, the ability to deploy code reproducibly. With the introduction of minikube, KinD, and others, you could now run the same stack you run in production, but locally.

The ability to reproducibly deploy helps both deployment confidence and the amount of time needed to get from nothing to running in production. However, it's not without its faults. Whenever you start moving deployment tooling onto developers, you've decreased the developer experience. It's unavoidable because you're introducing net new materials, DSLs, and more for the developers to learn. While KinD and minikube are great projects, they all suffer from needing developers to understand how to work with Kubernetes. You need to be able to build a Docker image, push the docker image into the cluster, delete the application pod, verify if it's even using the correct tag, wait for Kubernetes to recreate the container, and make sure you've configured an ingress route to access it outside of your cluster or use kubectl port-forward to access it. The second that breaks, you're now required to dig into why service connectivity isn't working, why your Docker image isn't in the containerd cache, or other not so easily solved areas. While to someone who's worked with Kubernetes for years now, this isn't very difficult, this is hardly achieving the "ease of use" goal I have.

How do we make developing on a Kubernetes developer environment easier?

Solving these problems is not easy. Debugging Kubernetes is difficult and requires knowledge that can only be solved with education, more tooling, or a combination of both. Unfortunately, these are not cheap. Education is hard to do correctly and tends to result in lots of time writing quickly out of date material that is hard to maintain. While at Azuqua, we encountered this same problem. Very few engineers wanted to learn Kubernetes or invest time in the technology around it. We decided to go back to the basics and focus on a tooling based approach. How do we get the same developer experience level, at a minimum, of Docker Compose with Kubernetes? Our answer ended up being a tool called Telepresence. Telepresence describes itself as:

[...] an open source tool that lets you run a single service locally, [sic] while connecting that service to a remote Kubernetes cluster.

https://www.telepresence.io/discussion/overview

It seemed perfect, a tool that enabled your local machine to act as if it were running in Kubernetes and allow your services to be targeted from inside of Kubernetes. We rolled it out at Azuqua after initial tests showed it worked well, and we went with it. Unfortunately, it didn't last.

The Problem With Current Local Service Development Tooling

While Telepresence solved the problem of communicating with our local Kubernetes cluster, it spectacularly failed at letting services inside the cluster talk to our local services. When it worked, it worked amazingly, but nine times out of ten, it'd fail in a magnitude of ways. Whether that's failing to clean up after random crashing or slowly taking down the computer's internet connection, it generally wouldn't work. Luckily for us at Azuqua, we had a pretty well-defined system for service discovery and few services that needed to talk to one another directly. The few that needed that could just run outside of the cluster. That allowed us to accept those pains and be successful with Telepresence. To be clear, this is not a hit on Telepresence. It worked very well for us at Azuqua, but it's not a complete solution.

When I started developing a new development environment for Outreach, I again tried to use Telepresence as the solution to bridge the gap between our local cluster and our local machine. Unfortunately, it didn't work. We did not have a well-defined service discovery mechanism to work around this, we had a much larger engineering team, and almost all services needed to talk to each other. We found more and more edge cases with Telepresence, and our developer experience was suffering. Our NPS score for our developer environment was at a low of -26.

It was pretty clear that Telepresence was not going to solve our use-cases. It was time to look into alternatives.

What are other alternatives out there?

Telepresence was interesting since it used a VPN to give you access to your resources. However, it also required a hack DNS injector component, which tended to be the primary source of network problems. There weren't many projects out there that seemed to solve this, but one interesting one was kubefwd. If you're not aware of Kubernetes port-forwarding, there is a command that allows you to bring down a set of ports for a given service and have them be available on your local machine via kubectl port-forward. Think of the -p argument to docker run, but for Kubernetes. Unfortunately, this didn't support FQDNs (.svc.cluster.local), statefulsets, or all namespaces automatically. It also didn't support reverse tunneling. I wanted to keep the spirit of Telepresence by continuing to be a one-stop tool. Aside from kubefwd, there were seemingly no tools that could do this. Reverse tunneling into Kubernetes alone was seemingly an unsolved project. To fill that gap, I decided to write Localizer.

Introducing Localizer

Localizer is a rewrite of Telepresence in Golang, but without the complex surface area. Its goal is to be a simple, no-frills development tool for application developers using Kubernetes. Its goals are to be easy to use, self-explanatory, and easy to debug. Localizer is also a daemon. When trying to expose multiple services, developers generally had to have many terminal windows to run Telepresence or other reverse tunnel solutions. Localizer moves away from that by having all commands send a gRPC message to the daemon. More on that later. At the core of Localizer is three commands.

Default: Creating Tunnels into Kubernetes

When you run localizer without any arguments, it automatically creates port-forwards for all Kubernetes Services (this is important), a loopback-interface powered IP address, and adds them to /etc/hosts. Out of the box, this allows code to communicate with Kubernetes services despite being outside of the cluster.

Expose: Creating a Reverse Tunnel

The expose command takes namespace/service as an argument that allows you to point a Kubernetes service down to your machine's local service instance. When run, it automatically scales down any endpoints that already exist for the service. Localizer then creates an OpenSSH pod with the Kubernetes service's selector, which makes Kubernetes route traffic to the created pod. Localizer then creates an SSH reverse proxy over a Kubernetes port-forward (which enables firewall bypassing for remote clusters) to the OpenSSH pod created, which exposes and routes traffic to your local service. In essence, it creates an SSH reverse tunnel for you.

List: Listing the status of the tunnel(s)

The arguably most useful command that localizer provides is list. It enables developers to view the various tunnels running and provide insight into the health and status.

How Localizer Solved Our Problems (sort of)

Rolling out Localizer was a massive success for our developer environment's stability at Outreach, but it's ultimately not a complete solution. It helps enable the developer experience's build/test aspect, but it doesn't solve the deployment complexity aspect of Kubernetes. Ultimately, it's out of scope for Localizer to fix these problems and is just another tool that helps bridge the gap between Kubernetes and a developer's local machine.

The lack of abstractions is a fundamental problem for Kubernetes right now, and I look forward to writing tooling that can help developers focus on the business problems and not waste time on the details.

Looking to the Future

Localizer has a lot planned for the future! Namely proper daemon support, but also trying to improve the visibility into different failure modes. Stability and recoverability are going to be a constant focus for this project.

Are you interested in Localizer? Check it out on Github!

Special Thanks: Mark Lee for editing!

Original posted on December 4th, 2020 on blog.jaredallard.me

Really an impressive setup, if I'm honest!

This blog is built using https://essay.dev

I didn't write any code, nor I configured/deployed anything to get it up.

I just opened GitHub app on my iPhone and created an issue in essay.dev's repo. That's it. 🚀

No computer

Early this morning I was able to get some very, very simple OCaml code running on my physical iPhone 6+, which was pretty exciting for me.

I had been excited about the idea since seeing a post on Hacker News. Reading through, I actually expected the whole process to be beyond-terrible, difficult, and buggy - to the point where I didn't even want to start on it. Luckily, Edgar Aroutiounian went well beyond the normal open-source author's limits and actually sat down with me and guided me through the process. Being in-person and able to quickly ask questions, explore ideas, and clear up confusion is so strikingly different to chatting over IRC/Slack. I'll write a bit more about the process later, but here's an example of the entire dev flow right now: edit OCaml (upper left), recompile and copy the object file, and hit play in XCode.

The next goal is to incorporate the code into this site's codebase, to build a native iOS app for this site as an example (open source) iOS client with a unikernel backend. I'm very eager to try to use ReactNative, for:

1. The fantastic state models available (just missing a pure-OCaml version of DataScript)
2. Code sharing between the ReactJS and ReactNative portions
3. Hot-code loading
4. Tons of great packages, like ReactMotion that just seem like a blast to play with

Acknowledgements

I'd really like to thank Edgar Aroutiounian and Gina Maini for helping me out, and for being so thoughtful about what's necessary to smooth out the rough (or dangerously sharp) edges in the OCaml world. Given that tooling is a multiplicative force to make devs more productive, I often complain about the lack of thoughtful, long-term investment in it. Edgar (not me!) is stepping up to the challenge and actually making very impressive progress on that front, both in terms of code and in documenting/blogging.

As a side note, he even has an example native OSX app built using OCaml, tallgeese.

Looks pretty neat! Didn't know blogging would be so easy. What's stopping you from blogging now? Just create issues and write out your mind. Put your ideas in public and see them evolve.

Why I write (and why you should too)

The end goal is to think clearly. Writing helps me get the thoughts on to paper and give them a direction.

You don't write when you have something to say. You write to find that out.

And when you have a thought or idea clearly formed, publish it so others get inspired.

More about me - https://aravindballa.com/

Do you already have a blog? Share it below 👇

My last job search lasted 100 days. I went through 11 onsites at companies like Google, Airbnb, Lyft, Uber, Amazon, Square, Stripe — you name it and I did an onsite there. Out of the 11 I landed 10 offers, mostly as Staff Engineer, ultimately choosing Airbnb.

The process changes drastically as you move into the Senior / Staff levels. I wanted to write this essay to draw an outline of the kind of preparation and change in focus that's needed and to do well in these interviews. I've seen little written about this, and I hope it's useful to you. This essay is formatted like a "100 day curriculum" of sorts. I hope you enjoy it!

Day 0: Foundations

To kick off the search, there are a few mindset shifts that if done well, can make everything else simple. In fact, I’ll venture to say that some of these mindset shifts when generalized can change your perspective… on life itself!

Okay okay that’s a serious statement, but give this a read, try it yourself and let me know what you think after 😊.

Foundations I: Attitude

Most people dread the job search. They think it’s a kind of performance, and “they just want it to be over with”. This kind of attitude can shoot you in the foot. If you dread something, you’re less likely to invest the time needed to get great at it. That can produce negative experiences, which reinforces your opinion and causes a spiral of bad outcomes. Plus, if you “just want it be over with” you’re more likely to choose the first few offers and miss out on the opportunity that’s really right for you.

Let’s shift this. Learn to love the job search_._ You can look it as an infrequent, leveraged opportunity for you to grow. This is your chance to select exactly the right next move for your career. It’s one of the best times to negotiate your role and your compensation. Plus, it’s a great opportunity for you to expand and deepen your relationships: you’ll lean on your community throughout the search and meet many talented people at some amazing companies.

Well, that’s a lot of real good reasons to love the job search. Internalize this attitude, and every step that follows turns into a meaningful part of your life story.

Foundations II: Community

Most people treat the job search as a solo activity. They apply for jobs by themselves, they negotiate themselves, and they choose themselves. I think this is because people associate asking for help with dependance, and they’re afraid to make themselves vulnerable.

Now, this is fine, it’s an independent mindset. But we can go further. Let’s shift to an interdependent mindset_._ Think about it, teams achieve great things, not individuals_._

Form a team around your search and involve them in every phase**.** Who are the mentors who can show you what you didn’t know you didn’t know? Who are the peers that can train you and give you advice? Who are friends who can refer you to the right companies? Engage them from the start.

For example, before starting the search I met up with friends and mentors to get their advice on what to do. I can honestly tell you that these conversations opened paths I could never have conceived before. People who champion you can give you ambitious advice that you may be afraid to see, and that changes the game**.** A special shoutout to my previous co-founder and life-long friend Sebastian Marshall, whose strategic advice changed was invaluable to me at this stage.

When I did start the search, I created a document that tracked the companies I was aiming for, alongside todos. This made it easy to share where I was at with my community, and easy for my community to discover ways they can help. Through this I received referrals at all 11 companies I aimed for.

When I began, two friends took the time to write up detailed advice from their last job search. Many, many friends took the time to train me. One of them was 12 timezones away and made time to skype for hours. They showed me where the bar was at, and gave me the feedback I needed to iterate.

Abraham Sorock, Alex Reichert, Andrew Wong, Daniel Woelfel, Giff Huang, Jacky Wang, Kam Leung, Mark Shlick, Robert Honsby, Sean Grove special shoutouts for the support and advice you gave me 😊. There’s many more — if you’re reading this and I haven’t included you, it’s mainly because I thought you preferred privacy — let me know and will update!

You’re starting to get a sense how interdependent this was. If I listed every person, I think it would surpass a hundred people. I truly believe if you embrace this mindset and look at problems as a team, you’ll see a huge change in all you go for, not just the search.

Foundations III: Narrative

We get to the final part of the mindset, how you look at your career. Most people, if you ask them “Why are you looking for a new job”, will give you either a generic answer — “I’m looking for a change_”,_ “I want to be impactful”, or some negative story — “I was bored_”, “_I didn’t like X_”._ This hurts you in at least three ways. First, it’s what everybody says, and it demonstrates that you haven’t thought deeply about your career. Second, it makes it hard for people to help_._ If you tell your mentors “you’re looking to be impactful”, that doesn’t narrow much down, and They can’t just open up their rolodex to email every company they know. Third, it makes it impossible to be a _strong fit_ for any company. What can a recruiter note down if you say “you’re looking for a change”, which will let them think you’re a perfect fit for their company? If you’re looking for any company, then by definition you can’t be a strong fit for a specific company.

Instead, let’s shift your viewpoint and take charge of your career. Look at your career as an adventure. Sit down and reflect: What got you here? What did you love about your last job? What would the most amazing opportunity look like? As you go deeper, you’ll discover reasons that will get your thrilled, and show you how unique and awesome your path has already been. It will make the search a lot more fun, it will make it easier for people to help, and get recruiters very excited about you. Most importantly, it will change how you think about yourself and your career. Let me be clear, this isn’t some trick to change your viewpoint artificially. If you dig deep, you’ll discover real, true reasons, and that will reverberate throughout your life.

Day 1 → 5: Kick Off

Okay, now we’ve learned to love the job search, we see it as a team endeavor, and we have a strong conviction about our career. It’s time to kick off the search. To do that, let’s do two things:

Kick Off I: Communication

In every interview you do, people aim to extract signal and scope. Signal is what demonstrates that you can do your job — from coding to designing systems. Scope is what demonstrates the kinds of problems you can take on — do you lead a team, a team of teams, or the whole org?

The combination of signal and scope is what determines your level_._ Your level, and where you fall on the spectrum is the most leveraged decision for determining your compensation and the kinds expectations you have.

Many people get annoyed about this. It feels very bureaucratic. Why should some number define you? You may also know people who have “high levels” but produce less. Thinking this way can hinder you. Another way you can look at it is this: it’s how things are for now, and it’s the best way we know so far to align your skills with the company. Let’s lean in and work with it.

Your goal for the first 5 days is to form a conviction about your level. Go over expectations, and prove to yourself that you are the level you are. If you want to do go deeper on this, visit our course and see the “Communication” module — it gives you a leveling guide to calibrate. Once you’ve done this, update your resume and LinkedIn to communicate that level. When you speak with recruiters and interviewers, you should be able to freely communicate it as well. Finally, go to levels.fyi, and form a conviction about the kind of compensation ranges you’re looking for too.

Kick Off II: Schedule Screens

From consulting with your community and forming your narrative, you should have about 10 or so companies that you could see yourself being thrilled to work at_._ Now, engage your community and find a referral for each company. You can do this by searching through your LinkedIn, or sharing a document of your top choices with your community and asking them if they know people who can refer. Do the best you can to go through referrals, as having someone to champion you helps in many ways throughout the process. If you can’t find someone, it’s okay to send a direct application, but do that as a very last resort.

With all recruiters, ask to schedule a screen about 20–30 days in advance. They may be surprised, but you can tell them you want to prepare, and that will send positive signal too. With that, you’ll have a deadline, and a real good reason to start studying 😊.

Day 5 → 30: Preparation

Now comes the hard work. It’s time to ramp up for the technical interviews.

Preparation I: Key Mindset

The mindset at this stage I want to stress, is to give it all you’ve got. Sometimes, as a way of coping with rejection, people give themselves an out — “Oh I didn’t really study anyways”. My suggestion is to frame your success on how hard you prepare, not on the results. Aim to remove any reason to say, you didn’t try your absolute best.

Fun story on that — months before the search, I had trip planned with one of my closest friends to go to Israel and Jordan, and it fell right during preparation time…

Well then, we need to travel and study at the same time 🙂. A special shoutout for Jacky Wang for being such an awesome travel partner, and being an encourager for the studies every day. Jacky’s the kind of guy who not only pushes the envelope to explore (just check out his photos), but he’ll indulge and converse with excitement about programming questions for hours on long car rides!

Preparation II: General Advice

There are three pieces of advice that apply generally to all interview types. First, the process to study: I recommend self-study first, then schedule practice interviews, then try the real thing, then calibrate and repeat the process if necessary. Second, approach every interview as if you are solving the problem for real_._ Most people seem to “pretend” when they solve the problem. They may not go deep enough, write code that won’t run, explain things for the sake of explaining, or create a system that won’t scale. Imagine a team at work set up a meeting and asked you to solve the problem, and act accordingly. Third, most people are blissfully unaware of what an excellent performance looks like. They assume the interviewer will read their minds and see how amazing they are. As you can guess, this won’t work. Study each interview type deeply, and get feedback on how you’re doing from your peers. Calibrate, make sure you are aware the bar, and you are well above it.

…Now, for some more specific advice

Preparation III: Algorithm Interviews

Unless you’re a specialist, this is the interview type to master first. No matter how skilled and senior you are, algorithm interviews play a significant role. I suggest you pick one book — Joe and I prefer Elements of Programming Interviews — and work through it linearly every day (personal nit: skip the first chapter in this book on bitwise operators, this is the most annoying part of the book, and isn’t needed for most interviews). If you get stuck on a question, just look at the answer, but make sure to write it with your own code, line by line. By the end of it, you’ll be surprised with all that you learn, and I guarantee you, at least if you go through this book, you’ll rarely be asked a question that you can’t map to something you’ve learned.

One common pitfall here is, people go on leetcode and do a bunch of random questions. I suggest avoiding this — since the questions are random, you won’t see improvement quickly, and you may not learn what’s needed. Leetcode is great though for finding and doing specific questions.

A week or so after you’re into the book, schedule practice interviews. My favorites are https://interviewing.io and https://pramp.com. A special shoutout to the founder of interviewing.io, Aline Lerner. There was an obscure 19 year-old on hackernews who was having trouble coming to the U.S. She took the time to help, and even gave him free access to her lawyers. That 19 year-old was me, and I am still grateful 🙂.

The third or fourth week, schedule practice interviews with your friend group. This is where you’ll really get a sense of where you’re at. If the feedback is positive, you’re ready for the screens. If not, _move them. Y_ou’ll be surprised how many times you can do that. We go quite deep on this, and the kinds of questions you should be able to answer, in the “Algorithm Interview” module of jobsearch.dev.

Preparation IV: UI Interviews

For UI Engineers, I suggest three steps. First, get comfortable with online editors, you’ll be using them all the time, so you should be able to set up an environment quickly. Second, ramp back up on some UI concepts that you may not have worked on recently — things like drag and drop, selection apis, etc. If you’re looking for more specific inspiration, we’ve included some homework in the “UI Interview” module of the course. Third, explore the latest and greatest in JS — look up what interests you — from react hooks to all that’s new in GraphQL. If you’re looking for some courses, Wes Bos has some great content to ramp you up. The key to these interviews is to demonstrate your ability to build UIs and your love of UI engineering. If you follow these steps it’ll come through naturally.

Preparation V: System Design Interviews

For System Design, I also suggest three steps. First, master the structure — scope down, go broad, outline a complete solution, go deep on as many components as you can, then go the extra mile, to share how you would actually launch the system. The “System Design” module in the course does a great job of going deep on this. Next, as you go through the practice interviews, you’ll start to notice places where you can improve. Start noting those down and build a plan to learn. For example, if you don’t know how to scale, you can search “{Company Name} InfoQ” and find some awesome talks. If you’re unsure about concurrency, the book 7 concurrency models in 7 weeks can ramp you up. Don’t know much about databases? 7 databases in 7 weeks. Pick what interests you and start going deep. This is my favorite interview type, because as you go deeper, the skills and knowledge you learn transfers into making you a better architect.

Preparation VI: Experience Interviews

The purpose of the Experience Interview is to understand your scope — the kinds of problems you can solve — and whether you are a culture fit. This interview type largely takes care of itself if you’ve centered on your narrative (you know what you want and where you’re going), and your communication (you know what level you are). The first coneys your culture fit, and the second your scope. You can go deeper on this in the “Experience Interview” module in the course.

Preparation VII: How to Focus for these 20 days

Now you’ve got an outline of each kind of interview, and how to prepare. For the first 20 days, I suggest focusing heavily on Algorithms. If you’re a UI engineer, still do algos, but spend maybe 30–40% of the time building UI components too. You want to focus on this because the screens are mainly this type of interview. As you pass the screens, you’ll have to go deeper on systems before you head into the onsites.

Day 30 → 60: Execution

Okay, we’re now ready to execute. Over the next month, you should be finished with all your screens and on-sites.

Execution I: Key Pieces

Three key pieces of advice here:

Come in to every interview with a positive mindset, and treat everyone with class. These people could become your friends, and it’s fun to learn about how a company works. Come with that mindset, and the confidence will seep through.

Batch all the interviews. Make sure all the screens and onsites are scheduled around the same time. The preparation you can do part time, but this piece Joe and I highly suggest, you figure out how to do full-time. When offers land around the same time, you’ll have a lot more leverage.

Communicate your level and your narrative. Keep signal and scope top of mind, and communicate clearly during every phase — recruiter screens, technical screens, and and onsites. We go deeper on this in the “Interview Phases” module of the course.

Execution II: Dealing with the Unexpected

As you go through the onsites, two things can happen:

You’ll fail at some interview

As you may have noted, I did 11 onsites, and got 10 offers…that means I failed one of em 😄. Even with some of the most intense prep, sh*t happens. When it does, do not over-react, embarrass yourself by being mean, or change your plans and cancel all upcoming interviews_._ Instead, treat everyone with a smile and with class. Then, be kind to yourself, take out what lessons you can, and continue on with the plan. If you see failure on more than one or two interviews, consider re-scheduling and re-calibrating.

You’ll get something unexpected

Life can throw curve ball at anytime, and you’re unlikely to avoid it during the job search. Case in point:

On Christmas break, Joe and I went skiing. I did an unintended backflip, and landed in the ER. When something like this happens, again, avoid over-reacting. Treat surprises as an opportunity to be malleable, and make it part of your story. For example, the day I visited the ER, I had a conversation with one of the best recruiters I’ve ever met, Twitter’s Matt Robbins. I sent him that picture, and I wager that will be the most unique recruiter conversation either of us will have in our career.

Day 60 → 100: Choice

Around Day 60, offers should start to roll in, and it’s time to choose. The first surprise that people may have here, is the number of days. Most people take about a week or two to decide…Day 60–100 is 40 days! Does it really take that long to choose a company, even after you’ve received your offers?

Yes!

Let me be clear, this isn’t about negotiation. You’ve already done 90% of the legwork for that through your preparation and execution — if you communicated your level well, you’ve aced the interviews, and you landed your offers around the same time, you can safely assume every company will be competitive and get you the best offer. There are specific things you can do at this point — the main advice being to use clear with what you would sign right away with. We go deeper on this in the “Negotiation” section of the course. Patio11’s Salary Negotiation and Rands’ The Business are great reading too.

The reason this takes so long is because the purpose for this phase is to choose the company that’s the best fit for you. It’s your turn to interview back. Meet your team, meet your manager, really align on the kinds of work you would do, and let your future team show you how they could be a part of your narrative.

To give you context, Stripe, Airbnb, Square, and Google were my final choices. I ended up meeting 6 managers at google, and visited Stripe, Airbnb and Square offices at least 7 times each. By the end of these meetings, you’ll have formed friends, and have gotten a great sense of what it’s like to work at each company.

The final step before making the decision, is to get feedback from your community. I suggest you write out an essay that goes over your decision, and discuss with your peers and mentors — do they agree with your thinking, what do they see that you don’t, do they have any suggestions?

To get a sense of how this looks, here’s a compressed screenshot of the essay I sent my friends and mentors: “On Choosing”

It’s a bit too personal to share outright, but I wanted to give you a sense of the kind of deliberation that went behind this: thousands of words, and insights from more than a dozen of my closest friends and mentors. These were amazing companies and the decision was close.

If you do this well, you’ll have clear, strong conclusions for exactly why you would choose a certain company. Compensation will be far away from the main reasons, and you’ll be thrilled to sign.

Day 100: Breathe

This will be a day where you breathe and realize how exciting, yet how tiring things were. Take a break and plan a celebration with your community.

With that, we’ve covered the advice 😊.

Final Notes

Want to go deeper?

If you made it this far, and you’re pumped, get ready to go deep into the rabbit hole. Joe, Luba, and I, with the help of Aamir Patel and Jordan Yagiello, expounded on all the lessons learned here, and created a video course that will walk you through it all. From leveling guides, to example narratives, to deep explanations on system design, UI interviews, and more, you can get it there. Best of all, it’s 100% free. We were thinking of making it paid course initially, but after some long talks, we wanted to share this with our community 😊. Go to https://jobsearch.dev and see for yourself.

Onwards from Airbnb

My best friend and co-founder Joe and I left Airbnb, to start a company of our own. We are currently exploring fitness, figuring out what it takes to reach advanced levels of performance. If that kind of stuff interests you, sign up for our waitlist here.

Want to set up GraphQL subscriptions on GitHub with your custom GitHub app?

Here's a video showing how to set that up:

This is also a test of embedding mov files directly in GitHub issues to be displayed in OneBlog.

anything in there

By popular demand... (I.e., thank you to our fantabulous community for the impetus to write this post!) This is crossposted at blog.beeminder.com/upsidedown.

Not to brag but our users are constantly telling us that Beeminder's customer support is shockingly good. The best they've ever seen, even. Long ago we wrote about our secrets of success in "Beehind the Curtain" and that's all still true and it's one of our classic posts that we refer to all the time. Today we're sending our users out of the room to tell our fellow startups and small businesses more about one of those secrets, and, importantly, to give it a concept handle.

Upside-down support is what we call it when we turn a support request from a user upside-down and ask for the user's help to improve the app or website the user needs support with. We've collected various techniques for doing this over the years and it makes so many things in support so much better. We even have a much better way to say no to users!

First, in case any of our users are still in the room¹, we should emphasize how great this is for everyone involved². It turns out that trying to be the diametric opposite of helpful to users in support delights them all the more. Weird, right? But less weird than it sounds. Let us explain, first by quoting ourselves from seven years ago, since this has clearly stood the test of time:

I spotted this in an old email from one of us: "Let me know if there's anything else I can help you with!" Perfectly informal and sounds so friendly and helpful. But I’ve come to believe that it backfires.

First, people are thoroughly cynical about companies wanting to "help" them. Second, psychology! People want to help people. They go out of their way to do so. People are nice, even altruistic. They don't want to take up our time asking questions.

When we were just getting started and willing to do pretty much anything for our initial users, Melanie, our resident fitness expert, would offer advice and coaching and people would think to themselves "well, that’s not fair, I’m not paying for that".

You can bend over backwards offering help and it makes users feel guilty or suspicious and ignore you. If you ask them for help -- or make clear that asking you questions is not a burden but a vital form of feedback that you need for improving the product -- then they respond effusively, and bend over backwards to help you.

It's an empirical fact. People respond better to helping us than us helping them. To be clear, this is no gimmick. Users only need our help in the first place because our website sucks. We're not helping them, we're just making up for the sucking (and learning how to unsuckify it). So no matter how much you help someone, don't talk about it that way. Turn it around and explicitly thank them for helping you suck less.

Since we wrote that, we've really doubled down. How so? I'll start with a fictional example from our job posting for new support workerbees. The scenario is a user who's being kind of obtuse and a little demanding:

Hi! Oops! Sorry! I didn't submit my information on time again and then I didn't notice this until today so the charge has already gone through. This wasn't a legitimate charge because my cat ate my phone and I couldn't enter the data. Afterwards, I hit the wrong button (button A) again instead of... button B, I think? So now my goal is broken again. Can you fix that for me? I know there's a way for me to fix it, but I've been really busy this week and I just can't remember how it works and keep breaking it.
Thanks!
Userperson

One goal is to make them happy, of course, but our primary goal is to turn the interaction upside-down and make it about helping Beeminder. Here was my own off-the-cuff attempt, pretending I was in support replying to that person:

hey userperson! eek, we need your help! can you show us a screenshot and write down some of your internal dialog so we can figure out how to make this button-A-vs-button-B thing easier for people? and then the same for fixing it (button C is probably what you're thinking of). the fact that you can't remember and worry about breaking it is really important for us to understand better. (if you do break it, we'll of course fix it for you; we just really need to understand how that kind of confusion/frustration arises.) thank you so much for helping making beeminder better!

ps, i did the refund for this goal. it really helps us when you can reply quickly when something's not legit.

Here's a non-fictional example where a user called non-legit on a derailment because they didn't get Beeminder's reminders and we replied like so:

Yikes, that's extremely bad that the Beeminder emails stopped coming through for you. Can you go to beeminder.com/reminders and confirm that the checkboxes for email are still checked? You can also go to an individual goal's settings tab and send a test reminder and see if that comes through. Really appreciate your help figuring out what went wrong there! Deliverability problems with email could be devastating for us.

PS, canceled the charge and undid the derailment on your ____ goal.

In both cases above, we ignored the user's actual request (until addressing it in a PS as an afterthought) and instead recruited them to help diagnose what went wrong. Importantly, we include "user was confused" as something going wrong with Beeminder!

(Clarification from Support Czar Nicky: In the case of an actual refund or canceling a charge, it's probably worth making it a prescript rather than a postscript. It can still be written as incidental, just that users can be very anxious until getting the reassurance that their money's safe and sound. After that they can better engage with the rest of the reply!)

I think doing this is harder than it seems. For my cofounder and me, it comes a bit more naturally. Beeminder is our baby and whatever caused this user to email you, it's ultimately our fault and it stings and our thoughts immediately go to how to fix the root problem -- how to make Beeminder better.

But even for us, it's taken training and practice and pushing ourselves out of our comfort zones. Our support workerbees -- Nicky and Simone and Robin -- are extraordinarily good and kind humans and every instinct in their bodies screams at them to leap to your assistance when you email support with a problem. So it takes very conscious control to flip the interactions upside down.

Tips and tricks

Big companies have ruined most variants of "thanks for the feedback". Here's a way to not just express appreciation but demonstrate it:

Keep this kind of feedback coming; hugely helpful!

I like how it's particularly unapologetic about the upside-down-ness. It's literally an imperative: keep doing this (because it goes without saying that the point of support is improving Beeminder). Or here's a less direct version:

Ok, I fixed your thing; thanks for highlighting the confusingness of this; it really helps us figure out how to make this work better for newbees!

Sometimes it seems like there's no way to turn a support interaction upside-down. The user has some problem, only you can fix it, so you do, and now you need to tell them you did. Even then, we reframe it as helping Beeminder. Think about what the ideal version of Beeminder would've done to have avoided the problem in the first place.

Oh no! I think ideally Beeminder should ____ or ____ in a case like this. Or maybe ____ would make it moot? Thanks for the help in thinking this through!

PS: I fixed this for you for this instance.

Next trick, from Support Czar Nicky again: for any support interaction which isn't 100% routine, you want to come away with at least one piece of useful feedback. For instance, we often get people asking us to put in breaks because they won't be able to enter data. It'd be easy to just do that, assuming they forgot. Don't assume! You've got to ask. So here's an example:

Oh no! Normally when you need breaks, you need to put them in a week in advance, because of the akrasia horizon. It would help us a lot to understand better how that can go wrong for people. In your case was it (a) not knowing how to put in breaks or (b) not knowing you needed to do it in advance? I've gone ahead and put in the break for you since it's too late for you to add it yourself, but definitely let us know what tripped you up! It really helps us figure out how to improve the interface and the documentation.

A better way to say no

Suppose a user is all entitled and asks for special treatment in some way, like, "can you short-circuit my pledge to go straight to $810? I don't want to pay for a month of premium to do that for just that one goal?" We're way too fairness-obsessed to say yes to something like that, but upside-down support gives us a much better way to say no than something like "I'm sorry but our policy is...". We can instead do this:

Hi ____, I sure love how you think! And this is also really good feedback that it feels a little ... unfair? money-grubbing? I'm not sure how to put it... that we don't let you set your own pledge level without the fancy premium plan. Some of the history behind our thinking on this is at blog.beeminder.com/shortcircuit.

​Actually it would be really good to have you read that and see if it changes your mind about our stance on this, or if you have ideas for us on either how we could convey this better so that it seems more fair or, if it doesn't convince you, what actually would feel the most fair to you.

​​Thanks so much for all the beeminding and helping us think this stuff through! Can we send you stickers? Tell us a postal address if so!

That's not exactly a no. The door is open a crack to change our minds, but they'd have to make the argument from Beeminder's perspective, not their own. (Also notice that they're getting stickers as a consolation prize.)

Don't be helpful!

In conclusion, don't do things for users that they could do themselves. Point them in the right direction and ask for feedback about how easy it is to find and do. And even when the support request really does require you to do something for the user, always explicitly shift the focus to the user helping you (the business/startup/app), not you helping them.

I also want to re-emphasize the genuineness of all this. We really do need users' help in understanding what Beeminder could've done better to have avoided the problems users email us about -- figuring out the root issues. And asking for help is really hard. But it's worth it and we've gone to great lengths to overcome our fear of doing it and Beeminder support interactions tend to be pretty amazing because of it. Well, that and our amazing support workerbees.

Image credit: Faire Soule-Reeves

{"canonical": "https://blog.beeminder.com/upsidedown/"}

My first

post

This is a test

When it’s released, React 18 will include out-of-the-box improvements including:

• Automatic batching
• New APIs like startTransition
• A new streaming server renderer with built-in support for React.lazy

The React team has also taken a new step by creating the React 18 Working Group to provide feedback, ask questions, and collaborate on the release. The Working Group is hosted on GitHub Discussions and is available for the public to read.

React 18 Working Group

Members of the working group can leave feedback, ask questions, and share ideas. The core team will also use the discussions repo to share their research findings. As the stable release gets closer, any important information will also be posted on the React blog.

Because an initial surge of interest in the Working Group is expected, only invited members will be allowed to create or comment on threads. However, the threads are fully visible to the public, so everyone has access to the same information. The team believes this is a good compromise between creating a productive environment for working group members, while maintaining transparency with the wider community.

No specific release date is scheduled, but the team expects it will take several months of feedback and iteration before React 18 is ready for most production applications.

• Library Alpha: Available today
• Public Beta: At least several months
• Release Candidate (RC): At least several weeks after Beta
• General Availability: At least several weeks after RC

More details about the projected release timeline are available in the Working Group.

Create React App with Vite's React Template

yarn create @vitejs/app ajcwebdev-react18 --template react

Install dependencies

cd ajcwebdev-react18
yarn

Start development server

yarn dev

Install react@alpha and react-dom@alpha

package.json

If we look in our package.json we'll see the following dependencies included from the Vite template.

"dependencies": {
  "react": "^17.0.0",
  "react-dom": "^17.0.0"
},

Install alpha versions.

yarn add react@alpha react-dom@alpha

Check your dependencies for the new versions.

"dependencies": {
  "react": "^18.0.0-alpha-e6be2d531",
  "react-dom": "^18.0.0-alpha-e6be2d531"
},

Use esbuild.jsxInject to automatically inject JSX helper imports for every file transformed by ESBuild:

// vite.config.js

export default defineConfig({
  plugins: [reactRefresh()],
  esbuild: {
    jsxInject: `import React from 'react'`
  }
})

main.jsx

import ReactDOM from 'react-dom'
import './index.css'
import App from './App'

const root = ReactDOM.createRoot(
  document.getElementById('root')
);

root.render(<App />);

App.jsx

import logo from './logo.svg'
import './App.css'

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <img
          src={logo}
          className="App-logo"
          alt="logo"
        />

        <p>
          React 18 Deployed on Netlify with Vite
        </p>
      </header>
    </div>
  )
}

export default App

Deploy to Netlify

touch netlify.toml

[build]
  publish = "dist"
  command = "yarn build"

Create a blank GitHub repository at github.new.

git init
git add README.md
git commit -m "first commit"
git remote add origin https://github.com/ajcwebdev/ajcwebdev-react18.git
git push -u origin main

Connect your GitHub repository to Netlify.

The build commands are included from your netlify.toml.

$ yarn build
yarn run v1.22.4
warning package.json: No license field

$ vite build
vite v2.3.7 building for production...
transforming...
✓ 26 modules transformed.
rendering chunks...

dist/assets/favicon.17e50649.svg   1.49kb
dist/assets/logo.ecc203fb.svg      2.61kb
dist/index.html                    0.51kb
dist/assets/index.7cb030a3.js      0.39kb / brotli: 0.20kb
dist/assets/index.0673ce28.css     0.76kb / brotli: 0.40kb
dist/assets/vendor.9aeda92c.js     134.00kb / brotli: 37.26kb

Done in 4.86s.
​
(build.command completed in 5.1s)

Set a custom domain.

Go to your new domain.

You can find all the code for this article on my GitHub.

limited supply crisis continues to disrupt thousands of mchenry utilities customers in the castro distrcit

Edit this design

testing, 123...

I'm toying with the idea of rewriting the deploy script I cribbed from @yomimono for this blog from bash to OCaml (there are some features I'd like to make more robust to the full deploy is automated and resources are cleaned up), and came across the OCaml AWS library. Unfortunately, installing it was a bit frustrating on OSX, I kept hitting:

NDBM not found, the "camldbm" library cannot be built.

After a bit of googling around, it was fairly simple: Simple install the Command Line Tools, and you should have the right header-files/etc. so that opam install aws or opam install dbm should work. Hope that helps someone who runs into a similar problem!

Happy hacking!

Trying to blow the buzzword meter with that title...

Note of Caution!

This never made it quite 100% of the way, it was blocked largely on
account of me not being able to get the correct version of the
dependencies to install in CI. Bits and pieces of this may still be
useful for others though, so I'm putting this up in case it helps
out.

Also, I really like the PIC bug, it tickles me how far down the stack
that ended up being. It may be the closest I ever come to being
vaguely involved (as in having stumbled across, not having
diagnosed/fixed) in something as interesting as
Dave Baggett's
hardest bug ever

Feel free to ping me on the
OCaml discourse, though I'll likely just
point you at the more experienced and talented people who helped me
put this all together (in particular
Martin Lucin, an absurdly intelligent and
capable OG hacker and a driving force behing
Solo5).

Topics

• What are unikernels?
  • What's MirageOS?
• Public hosting for unikernels
  • AWS
  • GCE
  • DeferPanic
  • Why GCE?
• Problems
  • Xen -> KVM (testing kernel output via QEMU)
  • Bootable disk image
  • Virtio problems
    • DHCP lease
    • TCP/IP stack
    • Crashes
• Deployment
  • Compiling an artifact
  • Initial deploy script
  • Zero-downtime instance updates
  • Scaling based on CPU usage (how cool are the GCE suggestions to downsize an under-used image?)
  • Custom deployment/infrastructure with Jitsu ¹

Continuously Deploying Mirage Unikernels to Google Compute Engine using CircleCI

Or "Launch your unikernel-as-a-site with a zero-downtime rolling
updates, health-check monitors that'll restart an instance if it
crashes every 30 seconds, and a load balancer that'll auto-scale based
on CPU usage with every git push"

This post talks about achieving a production-like deploy pipeline for
a publicly-available service built using Mirage, specifically using
the fairly amazing Google Compute Engine infrastructure. I'll talk a
bit about the progression to the current setup, and some future
platforms that might be usable soon.

What are unikernels?

Unikernels are specialised, single-address-space machine images
constructed by using library operating systems.

Easy! ...right?

The short, high-level idea is that unikernels are the equivalent of
opt-in operating systems, rather than
opt-out-if-you-can-possibly-figure-out-how.

For example, when we build a virtual machine using a unikernel, we
only include the code necessary for our specific application. Don't
use a block-storage device for your Heroku-like application? The code
to interact with block-devices won't be run at all in your app - in
fact, it won't even be included in the final virtual machine image.

And when your app is running, it's the only thing running. No other
processes vying for resources, threatening to push your server over in
the middle of the night even though you didn't know a service was
configured to run by default.

There are a few immediately obvious advantages to this approach:

• Size: Unikernels are typically microscopic as deployable
  artifacts
• Efficiency: When running, unikernels only use the bare minimum
  of what your code needs. Nothing else.
• Security: Removing millions of lines of code and eliminating
  the inter-process protection model from your app drastically
  reduces attack surface
• Simplicity: Knowing exactly what's in your application, and how
  it's all running considerably simplifies the mental model for both
  performance and correctness

What's MirageOS?

MirageOS is a library operating system that constructs unikernels
for secure, high-performance network applications across a variety
of cloud computing and mobile platforms

Mirage (which is a very clever name once you get it) is a library to
build clean-slate unikernels using OCaml. That means to build a Mirage
unikernel, you need to write your entire app (more or less) in
OCaml. I've talked quite a bit now about why
OCaml is pretty solid,
but I understand if some of you run away screaming now. No worries,
there are other approaches to unikernels that may work better for
you². But as for me and my house, we will use Mirage.

There are some great talks that go over some of the cool aspects of
Mirage in much more detail ³⁴, but it's unclear if they're
actually usable in any major way. There are even companies that take
out ads against unikernels, highlighting many of the ways in which
they're (currently) unsuitable for production:

Bit weird, that.

But I suspect that bit by bit this will change, assuming sufficient
elbow grease and determination on our parts. So with that said, let's
roll up our sleeves and figure out one of the biggest hurdles to using
unikernels in production today: deploying them!

Public hosting for unikernels

Having written our app as a unikernel, how do we get it up and running
in a production-like setting? I've used AWS fairly heavily in the
past, so it was my initial go-to for this site.

AWS runs on the Xen hypervisor, which is the main non-unix target
Mirage was developed for. In theory, it should be the smoothest
option. Sadly, the primitives and API that AWS expose just don't match
well. The process is something like
this:

1. Download the AWS command line tools
2. Start an instance
3. Create, attach, and partition an EBS volume (we'll turn this into
   an AMI once we get our unikernel on it)
4. Copy the Xen unikernel over to the volume
5. Create the GRUB entries... blablabla
6. Create a snapshot of the volume ohmygod
7. Register your AMI using the pv-grub kernel id what was I doing again
8. Start a new instance from the AMI

Unfortunately #3 means that we need to have a build machine that's
on the AWS network so that we can attach the volume, and we need to
SSH into the machine to do the heavy lifting. Also, we end up with a
lot of left over detritus - the volume, the snapshot, and the AMI. It
could be scripted at some point though.

GCE to the rescue!

GCE is Google's public computing
offering, and I currently can't recommend it highly enough. The
per-minute pricing model is a much better match for instances that
boot in less than 100ms, the interface is considerably nicer and
offers the equivalent REST API call for most actions you take, and the
primitives exposed in the API mean we can much more easily deploy a
unikernel. Win, win, win!

GCE Challenges

Xen -> KVM

There is a big potential show-stopper though: GCE uses the KVM
hypervisor instead of Xen, which is much, much nicer, but not
supported by Mirage as of the beginning of this year. Luckily, some
fairly crazy heroes (Dan Williams,
Ricardo Koller, and
Martin Lucina, specifically) stepped up and made it
happen with Solo5!

Solo5 Unikernel implements a unikernel base, or the lowest layer of
code inside a unikernel, which interacts with the hardware
abstraction exposed by the hypervisor and forms a platform for
building language runtimes and applications. Solo5 currently
interfaces with the MirageOS ecosystem, enabling Mirage unikernels
to run on either Linux KVM/QEMU

I highly recommend checking out a replay of the great webinar the
authors gave on the topic
https://developer.ibm.com/open/solo5-unikernel/ It'll give you a sense
of how much room for optimization and cleanup there is as our hosting
infrastructure evolves.

Now that we have KVM kernels, we can test them locally fairly easily
using QEMU, which shortens the iterations while we dealt with teething
on the new platform. The

Bootable disk image

This was just on the other side of my experience/abilities,
personally. Constructing a disk image that would boot a custom
(non-Linux) kernel isn't something I've done before, and I struggled
to remember how the pieces fit together. Once again, @mato came to the
rescue with a
lovely little script
that does exactly what we need, no muss, no fuss.

Virtio driver

Initially we had booting unikernels that printed to the serial console
just fine, but didn't seem to get any DHCP lease. The unikernel was
sending
DHCP discover broadcasts,
but not getting anything in return, poor lil' fella. I then tried with
a hard-coded IP literally configured at compile time, and booted an
instance on GCE with a matching IP, and still nothing. Nearly the
entire Mirage stack is in plain OCaml though, including the
TCP/IP stack, so I was able
to add in plenty of debug log statements and see
what
was
happening. Finally
tracked everything down to problems with the Virtio implementation,
quoting @ricarkol:

The issue was that the vring sizes were hardcoded (not the buffer
length as I mentioned above). The issue with the vring sizes is kind
of interesting, the thing is that the virtio spec allows for
different sizes, but every single qemu we tried uses the same 256
len. The QEMU in GCE must be patched as it uses 4096 as the size,
which is pretty big, I guess they do that for performance reasons. -
@ricarkol

I tried out the fixes, and we had a booting, publicly accessible
unikernel! However, it was extremely slow, with no obvious reason
why. Looking at the logs however, I saw that I had forgotten to remove
a ton of
logging per-frame. Careful
what you wish for with accessibility, I guess!

Position-independent Code

This was a deep rabbit hole. The
bug manifested as Fatal error: exception (Invalid_argument "equal: abstract value"), which
seemed strange since the site worked on Unix and Xen backends, so
there shouldn't have been anything logically wrong with the OCaml
types, despite what the exception message hinted at. Read
this comment
for the full, thrilling detective work and explanation, but a
simplified version seems to be that portions of the OCaml/Solo5 code
were placed in between the bootloader and the entry point of the
program, and the bootloader zero'd all the memory in-between (as it
should) before handing control over to our program. So eventually our
program did some comparison of values, and a portion of the value had
at compile/link time been relocated and destroyed, and OCaml threw the
above error.

Crashes

Finally, we have a booting, non-slow, publicly-accessible Mirage
instance running on GCE! Great! However, every ~50 http requests, it
panics and dies:

[11] serving //104.198.15.176/stylesheets/normalize.css.
[12] serving //104.198.15.176/js/client.js.
[13] serving //104.198.15.176/stylesheets/foundation.css.
[10] serving //104.198.15.176/images/sofuji_black_30.png.
[10] serving //104.198.15.176/images/posts/riseos_error_email.png.
PANIC: virtio/virtio.c:369
assertion failed: "e->len <= PKT_BUFFER_LEN"

Oh no! However, being a bit of a kludgy-hacker desperate to get a
stable unikernel I can show to some friends, I figured out a terrible
workaround: GCE offers fantastic health-check monitors that'll restart
an instance if it crashes because of a virtio (or whatever) failure
every 30 seconds. Problem solved, right? At least I don't have restart
the instance personally...

And that was an acceptable temporary fix until @ricarkol was once
again able to track down the cause of the crashes and fix things up
that had to do with some GCE/Virtio IO buffer descriptor wrinkle:

The second issue is that Virtio allows for dividing IO requests in
multiple buffer descriptors. For some reason the QEMU in GCE didn't
like that. While cleaning up stuff I simplified our Virtio layer to
send a single buffer descriptor, and GCE liked it and let our IOs go
through - @ricarkol

So now Solo5 unikernels seem fairly stable on GCE as well! Looks like
it's time to wrap everything up into a nice deploy pipeline.

Deployment

With the help of the GCE support staff and the Solo5 authors, we're
now able to run Mirage apps on GCE. The process in this case looks
like this:

1. Compile our unikernel
2. Create a tar'd and gzipped bootable disk image locally with our unikernel
3. Upload said disk image (should be ~1-10MB, depending on our contents. Right now this site is ~6.6MB)
4. Create an image from the disk image
5. Trigger a rolling update

Importantly, because we can simply upload bootable disk images, we
don't need any specialized build machine, and the entire process can
be automated!

One time setup

We'll create two abstract pieces that'll let us continually deploy and
scale: An instance group, and a load balancer.

Creating the template and instance group

First, two quick definitions...

Managed instance groups:

A managed instance group uses an instance template to create
identical instances. You control a managed instance group as a
single entity. If you wanted to make changes to instances that are
part of a managed instance group, you would apply the change to the
whole instance group.

And templates:

Instance templates define the machine type, image, zone, and other
instance properties for the instances in a managed instance group.

We'll create a template with

FINISH THIS SECTION(FIN)

Setting up the load balancer

Honestly there's not much to say here, GCE makes this trivial. We
simply say what class of instances we want (vCPU, RAM, etc.), what the
trigger/threshold to scale is (CPU usage or request amount), and the
image we want to boot as we scale out.

In this case, I'm using a fairly small instance with the instance
group we just created, and I want another instance whenever we
sustained CPU usage over 60% for more than 30 seconds:

`PUT THE BASH CODE TO CREATE THAT HERE`(FIN)

Subsequent deploys

The actual cli to do everything looks like this:

    mirage configure -t virtio --dhcp=true \
            --show_errors=true --report_errors=true \
            --mailgun_api_key="<>" \
            [email protected]
    make clean
    make
    bin/unikernel-mkimage.sh tmp/disk.raw mir-riseos.virtio
    cd tmp/
    tar -czvf mir-riseos-01.tar.gz disk.raw
    cd ..

    # Upload the file to Google Compute Storage 
    # as the original filename
    gsutil cp tmp/mir-riseos-01.tar.gz  gs://mir-riseos

    # Copy/Alias it as *-latest
    gsutil cp gs://mir-riseos/mir-riseos01.tar.gz \
              gs://mir-riseos/mir-riseos-latest.tar.gz
    
    # Delete the image if it exists
    y | gcloud compute images delete mir-riseos-latest
    
    # Create an image from the new latest file
    gcloud compute images create mir-riseos-latest \
       --source-uri gs://mir-riseos/mir-riseos-latest.tar.gz
    
    # Updating the mir-riseos-latest *image* in place will mutate the
    # *instance-template* that points to it.  To then update all of
    # our instances with zero downtime, we now just have to ask gcloud
    # to do a rolling update to a group using said
    # *instance-template*.

    gcloud alpha compute rolling-updates start \
        --group mir-riseos-group \
        --template mir-riseos-1 \
        --zone us-west1-a

Or, after splitting this up into two scripts:

    export NAME=mir-riseos-1 CANONICAL=mir-riseos GCS_FOLDER=mir-riseos
    bin/build_kvm.sh
    gce_deploy.sh

Not too shabby to - once again - launch your unikernel-as-a-site
with zero-downtime rolling updates, health-check monitors that'll
restart any crashed instance every 30 seconds, and a load balancer
that auto-scales based on CPU usage. The next step is to hook up
CircleCI so we have continuous deploy of our
unikernels on every push to master.

CircleCI

The biggest blocker here, and one I haven't been able to solve yet, is
the OPAM switch setup. My current docker image has (apparently) a
hand-selected list of packages and pins that is nearly impossible to
duplicate elsewhere.

I wanted to show a quick glimpse of the programming dream (or my own, at least) - immediate feedback while programming, validated against a live server. Power, simplicity, reliability, and convenience, I want it all!

Here's an example video with narration on what it's like working with ReasonML + GraphQL in emacs:

https://www.youtube.com/watch?v=yMqE37LqRLA

A few benefits of this approach:

• Full-stack safety: The server presents its entire known schema, so my Reason app won't even compile if it's trying to access a non-existent field (or trying to use it in a type-incorrect way, e.g. mistaking a string for an integer)
• Long-term safety: Because fields are never removed from a GraphQL (only deprecated), I never have to worry about shipping a client that might be broken by future server changes. This goes a long way towards ever-green clients. * No forgotten edge cases - this one kills me continually outside of Reason. I forget to check if the response is still loading, or if it error'ed, or I try to access data on the wrong field. I can easily add a catch-all to throw an error and ignore all the edge cases if I'm prototyping, but once I have my happy-path, I want to make sure things are battened-down tightly.
• In-editor completion: When accessing fields
• Editor-guidance: Along the previous lines, with Reason the data structures guide me to handling each case and field access gracefully as I explore the response structure. As soon as I hit save, I'll know if I have a typo, or if I accessed a nullable field without checking, or if I used the type incorrectly.

Some drawbacks:

The only drawback I can think of is I can't quite see a way to get auto-completion while writing the GraphQL in the PPX. I'd ideally like to have a GraphiQL-like experience with the fields auto-completing, and being able to read docs/types inline. Currently I tend to write the bulk of my queries in our fork of GraphiQL, then paste in the result. It's minor, but would be really nice if there was a way to do this (I know there's a way in Atom for example, but emacs may not make this easy).

Closing notes:

This example is in emacs, but the experience should be the same (or better!) in vim, Atom, and especially vscode, thanks to the great Reason editor integrations there.

I've switched my personal site (riseos.com)
over to Docusaurus from a Mirage unikernel
for a few reasons. First, I was putting off writing blog posts because
of the amount of yak shaving I was doing. Second, the dependency
situation never really got to the point I felt it was worth the
effort. And third, some projects I've been working on have pushed me
to get a lot more familiar with frontend topics, especially
static-sites that are rendered with React.

I've deployed a few sites with Gatsby,
and was looking for something significantly simpler and more
reliable. At the recommendation of the
ReasonML team, I gave docusaurus a shot
on another site, and it worked out nicely. I appreciate that it's
limited enough to encourage you not to yak-shave too much (which is
good from time to time, but not for my personal site at this time).

Anyway, certainly recommend giving
Docusaurus +
netlify a shot, worked like a charm for
me.

I used Let's Encrypt (LE) to get a nice SSL cert for www.riseos.com (and riseos.com, though I really would like that to simply redirect to www. Someday I'll wrap up all the loose ends).

Going through the process wasn't too bad, but unfortunately it was a bit tedious with the current flow. To pass the automated LE checks, you're supposed to place a random string at a random URL (thus demonstrating that you have control over the domain and are therefore the likely owner). I thought I would do this by responding to the url in my existing OCaml app, but

1. The deploy feedback cycle is just too long
2. The SSL cert generated by make secrets doesn't pass work for the check.

In the end I simply switched the DNS records to point to my local machine, opened up my router, and copy/pasted the example python code. Because I use Route53, it was instantaneous. Then after a bit of mucking about with permissions, I copied fullchain1.pem -> secrets/server.pem, and privkey.pem -> secrets/server.key, fixed the dns records, redeployed (now a single script on a local vm + a single script on an EC2 vm), et voila, a working SSL site!

There are some problems with the Let's Encrypt certificate however. The JVM SSL libraries will throw and error when trying to connect to it, saying something like, "unable to find valid certification path to requested target". That transitively affects Apache HttpClient, and therefore clj-http. In the end, I had to pull the cert and insert it into the keystore.

As a side note, the deploy cycle is still too long, and still too involved, but it hugely better than just a week or two ago. I expect to soon be able to remove the EC2 vm entirely, and to be able to run a full, unattended deploy from my VM - or even better, from CircleCI after every push to master. After those sets of paper cuts are healed, I want to do a full deploy on a fresh account, and get the time from initial example-mirage git checkout to running publicly-accesible server (possibly with valid https cert) to under three minutes, on either EC2, Prgmr, or Google Cloud (or Linode/Digital Ocean if anyone knows how to get xen images booting there).

just tryingt
for step in range(10):print(step)

I have a project, source code shots, that turns your source code into a png as you type. It takes a little bit of time to generate the image, so I wanted to indicate to the user that some work was being done to give them a visual cue that the image on the page was going to update.

I made a simple loading spinner component, using a css animation, and rendered it while the image was being fetched.

The code looked something like

/* styles.css */
@keyframes throb {
  0% {
    transform: scale(0.1, 0.1);
    opacity: 0;
  }
  50% {
    opacity: 0.8;
  }
  100% {
    transform: scale(1.2, 1.2);
    opacity: 0;
  }
}
.loading {
  animation: throb 1s ease-out;
  animation-iteration-count: infinite;
  opacity: 0;
  border: 3px solid #999;
  border-radius: 30px;
  height: 16px;
  width: 16px;
}

export default function App() {
  const [isLoading, setIsLoading] = React.useState(false);
  const [code, setCode] = React.useState("");
  React.useEffect(() => {
    let canceled = false;
    setIsLoading(true);
    // Simulate getting the image for the code
    new Promise((resolve) =>
      setTimeout(resolve, Math.random() * 300 + 100)
    ).then(() => {
      if (!canceled) {
        setIsLoading(false);
      }
    });
    return () => {
      canceled = true;
    };
  }, [code, setIsLoading]);

  return (
    <div>
      <textarea value={code} onChange={(e) => setCode(e.target.value)} />
      <div
        className="loading"
        style={{ display: isLoading ? "block" : "none" }}
      />
    </div>
  );
}

I thought I was done, but the spinner was so annoying I almost threw it out. As I typed, the animation would start and stop, creating an annoying jittery effect.

Type a few characters at a time into the textarea below to see what I mean:

What I would like is for the animation to continue to completion, even if we’re finished loading. That way the animation will never flash in and out.

A little google searching reveals a family of animation hooks that look like they could be useful. There is an onanimationiteration event, accessible with the onAnimationIteration React prop on our loading div, that will fire on every round of the animation. In our case, it will fire every second.

Instead of hiding our loading spinner when isLoading switches to false, we can wait for the onAnimationIteration hook to fire and hide the loading spinner in the callback.

The relevant change looks like

      <div
        className="loading"
        onAnimationIteration={() => {
          if (!isLoading) {
            setPlayAnimation(false);
          }
        }}
        style={{ display: playAnimation ? "block" : "none" }}
      />

Try it below:

You can also try it out at Source Code Shots¹.

Do you have a better way to build an elegant loading spinner? Let me know in the comments below.

{"authors": [{"name": "Daniel", "url": "https://twitter.com/danielwoelfel", "avatarUrl": "https://avatars0.githubusercontent.com/u/476818?s=96&u=8902611617e1833d27ce6e32f06f6db5113c60aa&v=4"}]}

So @dwwoelfel and I have been working on a powerful blogging system that keeps all of your data inside of GitHub issues - you can see the result (and post yourself) live on essay.dev - or you can fork the open-source repo and deploy your instance, and all the instructions below will work just fine on your own repo.

Watch me create a blog post from inside magit-forge

https://www.youtube.com/watch?v=VVOd1yOKVqQ

GitHub-issue powered blogging and commenting

The entire site is powered by GitHub issues and next.js (and hosted on Vercel). Any issue with a Publish tag will be made publicly available immediately (and can be similarly unpublished by removing the Publish label).

That's pretty fantastic for lots of reasons - your posts are now in an API that's easy to slice and dice so there's no lock-in to your content or comments, it's a familiar place for devs to work, etc.

There are hundreds of features and polish in essay.dev, but importantly for me, it's compatible with emacs' magit-forge!

magit-forge, I choose you!

magit is the famous git control system for emacs, and it has an equally powerful integration to manage GitHub issues called magit-forge.

You can do all the normal CRUD operations on GitHub issues inside a familiar emacs workflow - which means we can do the same for our posts¹!

Creating a post on essay.dev

First make sure you've installed magit and magit-forge (or for spacemacs users, just add the GitHub layer).

Now, let's clone the essay.dev repo:

git clone https://github.com/OneGraph/essay.dev.git
cd essay.dev
emacs README.md

Next we'll connect forge with our GitHub repository via M-x forge-add-repository - and now we're ready to see a list of all of the posts, so run M-x forge-list-issues:

If we hit Enter on any of the issues, we'll see the content and the comments:

Create a new post

Running M-x forge-create-issue will create a new buffer pre-filled via the default new-post template:

Simply fill out the title and the body, and when you're ready, "commit" the new post via C-c C-c. Forge will commit it to a local database first for safe-keeping, and then create an issue on GitHub! Back in the *forge-issue-list...* buffer, hit g to refresh the lists of posts, with your newest one at the top. Hit Enter on it to view the contents.

Your post is ready!

A few seconds later, run M-x forge-pull to update your local copy - you should find there's a new comment waiting for you from onegraph-bot:

View your post at https://.essay.dev/post//

That's it, your post is available to the world.

What's a post without comments?

You can also leave comments on your posts (and others) with M-x forge-create-post:

It'll show up instantly on your post (both in forge and on the site):

What's next?

Your content belongs to you, and is easily accessible through the GitHub API - here's an example query that'll pull out the posts for you:

query MyPostsOnGitHub(
  $owner: String = "onegraph"
  $name: String = "essay.dev"
  $createdBy: String = "sgrove"
) {
  gitHub {
    repository(name: $name, owner: $owner) {
      issues(
        first: 10
        orderBy: { field: CREATED_AT, direction: DESC }
        filterBy: { createdBy: $createdBy }
      ) {
        edges {
          node {
            body
            number
            title
          }
        }
      }
    }
  }
}

Try it out here

And again, note that this setup will work with any repo, so if you want to self-host your content it's as easy as using the deploy on vercel link.

Today I found this project on HackerNews and decided to give it a try.

The idea of a blog that uses Github issues as posts is intriguing, I'm wondering if it supports Github-flavored Markdown in its entirety.

For instance, should the hash 431af35 should point to a commit of this repository ?

Mirage is going to have a ton of growing pains as it's used for real-world applications. I suspect that most of that will be spent on polish and glue (which is desperately missing right now), because the core is relatively solid (especially compared to e.g. one year ago).

Still, I have tons of Mirage questions, and would like answers/guides to them, or even better - code to completely obsolete them. I'll keep a list here, and update it with links as answers come in.

• How to express pinned dependencies in the the mirage config.ml Apparently this isn't possible right now, which means others are going to have a hard time using my example repository.
• Seamless, continuous, one-click deploy from any platform to AWS, GC, Linode, Digital Ocean, and prgrm
• How to get stack traces from crashes in the unikernel in production (ideally we'd be able to combine with with e.g. bugsnag at some point)
• How to build a xen unikernel image from OSX (likely to be a big requirement)
• If the above isn't feasible, how to tie into e.g. CircleCI to build the xen artifacts and upload them somewhere.
• How to parameterize the ports for development (where I don't want to use sudo to start my binary) and for production (where I don't mind it, of course). Also applies to other things besides just ports (ssl certs, etc.).

This is crossposted at blog.beeminder.com/choices.

What's the most absurdly provocative way I can put this?
Never imagine what your users will want!
Apps must only ever do one single thing!
If-statements considered harmful!

Yes, this is all pretty rich coming from the people who built a goal-tracking app with, if I'm doing this math right, multiplying out all the various settings...
73,728 types of goals.
Not to mention all our reminder settings, pledge settings, ways to schedule breaks, settings for what happens when you derail, weaselproofing, etc etc etc.
We've learned this the hard way, ok?
Also we're exaggerating.
Some settings are great.
And Beeminder is a big beautiful bounteous beast that can do amazing things that we, the creators, have never thought of¹ and that's wonderful and we're very proud.
But there are some important rules of thumb that are counter to a lot of people's -- especially programmers', especially our own -- instincts.
So listen up!

I know it's exciting to imagine far-flung use cases and to try to please as many people as possible but it's like Abraham Lincoln says:
You can't please/fool all of them all the time.
If you're very sure you've identified genuinely divergent use cases (and that you genuinely want to support both use cases, which
as a startup you probably don't!) then maybe it's ok to add the setting.²
My rule of thumb -- and this is a necessary but not sufficient condition -- is to wait for a user to make an impassioned argument for their use case.
Never add a setting speculatively, i.e., by imagining what a user could want.
Default to total paternalism where you decide what the user wants.

Or look at it from a programming perspective.
As you add more settings you increase the number of possible code paths and places for bugs to hide exponentially.

Literally exponentially!
It's 2020 and we're all clear on what exponential means, right?
Say your app has a few simple checkboxes as settings and your code needs to work for every combination of them.
Then you add one more measly checkbox.
You've just doubled the number of combinations.
I mean, not always.
Maybe your app is pristinely designed and your settings are beautifully orthogonal.
But probably not.

When debating a design choice, never say "let's make it a setting!"

Need more reasons?
We have a whole smörgåsbord for you to choose from!
Documentation is much, much easier the fewer settings there are.
Customer support even more so.
And if you add a setting that turns out to be stupid, someone will get attached to it and it will be a
big ordeal to get rid of it.
Worst of all, it's all too easy to add a setting due to uncertainty about what the right design choice is.

"I guess we can ship that as long as we include a setting to opt out of it"
should be a glaring red flag.
We've violated this a lot and I suspect we were always dead wrong to do so.
An especially painful example was when we shipped a now-defining feature of Beeminder that both doubled our revenue and made Beeminder drastically more effective for users.
We called it
precommit-to-recommit at the time.
Now we don't call it anything because there's no such thing as anything but precommit-to-recommit.
But that's my point.
At the time we were too timid to ship that -- goals that automatically continue and recommit you when you derail -- without a setting to opt out of it.
Which was a massive thorn in our side and source of complexity, user consternation³, and technical debt for years.

Here's one more reason to eschew settings: they're a burden on users.
In "Chapter 3: Choices" of Joel Spolsky's
User Interface Design For Programmers he argues hilariously against adding settings for most things, focusing on how baffling or obtrusive excess customizability can be.
In a similar vein, see
Basecamp's exhortation to
"Make Opinionated Software".
Your app should not be flexible and agnostic.
It should take sides and have a cohesive vision.

Anti-Settings and Anti-Magic

If we wanted to get fancy or philosophical we could say that the Anti-Settings Principle and the
Anti-Magic Principle are special cases of a more general principle:
"try not to let the business logic branch".
Of course, most of the time there's no choice (about there being choices).
That's just what the business logic is.
If X then do Y else do Z, etc etc.
But, just... resist it.
Ask yourself if the business logic inherently must bifurcate or if, at least for the initial implementation, you can take just one of the paths.
When you come to a fork in the road, do not take it.
Pick a single tine.
Bifurcation will bite you.
This also relates to software engineering concepts like premature optimization being the root of all evil, and YAGNI.

Don't Take Our Word For It

I think this is all becoming very conventional wisdom despite how hard-won it's been for us.
The huge open source encrypted messaging service, Signal, sums up the Anti-Settings Principle nicely as item number one of their
development ideology:
"The answer is not more options.
If you feel compelled to add a preference that's exposed to the user, it's very possible you've made a wrong turn somewhere."

Image credit:
Faire Soule-Reeves

{"canonical": "https://blog.beeminder.com/choices/"}

This is crossposted at blog.beeminder.com/nerds.

For years we've gotten advice to widen our appeal. We shall now explain why you're all wrong¹.

Let's start with an intuition-shaping factoid: GitHub is focused 100% on developers even though writers and designers and many other categories of people could be -- ought to be! -- using version control. (Additional factoid: GitHub was worth $7.5 billion when Microsoft bought them, with tens of millions of users -- all programmers.)

And Facebook, despite all its mainstream appeal now, started out focused exclusively on college students. Harvard students, specifically. In fact, their early growth strategy was to very carefully and deliberately add a single university at a time and only much later expand beyond university students.

Or take Habitica -- bigger than Beeminder while catering strictly to people so nerdy they're into RPGs. Or Discord, a perfectly general chat app which got huge by focusing on gamers. They have like a quarter billion users and only relatively recently showed any interest in anyone but gamers.

Running with the GitHub-is-for-devs example, you can imagine how they must make all their design and strategic decisions: "What works best for Daphne the Developer?" They're not afraid to scare off novelists who don't know programming lingo. GitHub is brilliant for novelists and there's nothing stopping them using it, but that's not who GitHub is for. There's a lot of value in only having one kind of user to worry about. Product design and making users happy is hard and gets combinatorially harder the more kinds of users and use cases you have to support.

"Beeminder is probably not for you"

For Beeminder, we make all decisions from the perspective of what's best for Quentin the Quantified-Self akratic ambitious self-aware high-integrity lifehacking graph-loving data nerd. Also he loves puns. If none of that speaks to you then, for now, Beeminder is probably not for you. We don't want to actively push you away, we just need to only be thinking about our core demographic.

We should emphasize that there are plenty of issues with Beeminder's usability to this core demographic; more on that below. Also, I'm exaggerating how narrow our core demographic is. "Data nerds" probably suffices.

Isn't that placing an unnecessary fence around expansion?

Counterintuitively, it's the opposite! Paul Graham explains it pretty well in the "Well" section of his essay on startup ideas. He describes two types of startups: those that a large number of people need a little, and those that a small number of people need a lot. (He also explains why startups can't be the best of both, and why he calls the second type "digging a deep well".)

Nearly all good startup ideas are of the second type. Microsoft was a well when they made Altair Basic. There were only a couple thousand Altair owners, but without this software they were programming in machine language. [...]

In practice the link between depth and narrowness is so strong that it's a good sign when you know that an idea will appeal strongly to a specific group or type of user.

Or here's Patrick McKenzie of Stripe in an old Twitter thread explaining why the book "Writing for Software Developers" is focused on software developers when there seems to be no reason for it not to be way more general:

It's an artifact for the consumption of software developers which includes basically no software.

It's a book about writing which intentionally anti-targets 99.99% of writers.

Why? Because constraining the topic to the needs of software developers specifically makes the artifact much better (you know you're writing for someone who cares about e.g. blog posts to get senior developer jobs and not midlist fiction, and can tailor advice appropriately).

And note that you certainly could have hypothetically written Writing For $PROFESSION for literally any profession but choosing Software Developers means you structurally have a client who has effectively infinite budget for books which deliver career effectiveness.

This lets you *clears throat* Charge More.

Or here's a different old Twitter thread where he's talking about businesses rather than books, though it's all the same lesson:

When I was consulting life got better in every way after I said “Eff it: B2B SaaS businesses, $10M to $50M in annual revenue, probably engineer-led” as the target market. Pitches got crisper. Results got better. Rates went up.

What does this mean we should do?

It's mostly about what we shouldn't do. Like if we're, say, explaining a hypothetical new feature to show a non-cumulative version of graphs, we can call it a derivative and not bother to define what a derivative is². In other words, we should do what comes naturally to us. When talking amongst ourselves, it wouldn't occur to us define terms like "derivative". So the main thing to do is not hold back -- let our freak flags fly!

At this point, when this was just an internal memo, I emphasized that I was less certain about this than I sounded. Now it seems almost obvious. But also it matters less than it sounds in terms of Beeminder's roadmap. Regardless of the degree to which we want to double down on nerds, there's a long list of things we need to do to make Beeminder better for nerds and normals alike. Things like lifecycle emails, smoothing onboarding friction, getting confusing advanced shıt out of sight of newbees, fixing the miasma of brokenness with scheduled breaks / ratchets / weekends-off / restarts, interactive graphs and road.beeminder.com/tutorial and road.beeminder.com/sandbox. The list goes on and on and there's a huge newbee focus in most of it.

When normal humans can pass hallway tests -- and it's critical that they be able to! -- only then, at the earliest, should we think about adjusting copy and marketing and UI and everything to target them. In the meantime, almost all our onboarding and newbee-related problems have best-of-both-worlds solutions. We're not backing ourselves into any corners (not more so than we already are, at least) with our nerd focus. The work ahead of us, at least for some months, involves improving onboarding and reducing confusion in ways that applies equally to nerds and normals.

At some point we'll find ourselves deciding between choice A that's better for QS-y Quentin and choice B that's better for Normal Norman and that's when we can revisit this. (But probably the answer will be a resounding "Choice A! Norman can suck it" for the foreseeable future.)

Other benefits of focusing on nerds

First of all, it helps our own focus, our clarity, even our motivation to be helping fellow nerds. Relatedly, our distinctive voice and vibe and culture is pretty key to us having True Fans (which holy cow do we ever have, and it feels amazing). Next, being a bit exclusionary -- even alienating users who aren't quite your target audience -- attracts and endears and generates passion among those who are. They feel part of an in-group.

As a bonus, nerds tend to be easier and more valuable in support. Maybe sometimes they can be a bit much, but very positive on net, we've found.

Troll filtering and expectations management

It's infuriating when people leave low-effort 1-star reviews in the app store. Sometimes they do that without even logging in and there's little we can do about that. But -- and this one is more conjecture so far -- making it obvious that Beeminder is for nerds (equations on the splash screen, we're not sure yet) will tend to make those trolls' eyes glaze over and not bother trolling.

Or imagine someone who's like "Goals? I have goals!" and starts signing up and then sees graphs and numbers and is like "WTF? 1 star." The more front-and-center the graphs and numbers are, the less likely they are to feel frustrated and disappointed.

Anti-example

Take our oldest competitor, StickK.com, targetting anyone with goals. Or you could say they target anyone who's akratic -- those who struggle to stick to their intentions -- but that's still far too broad a swathe of the population. They're clearly trying to have mainstream appeal and they just end up being, well, StickKly.

On that slightly meanspirited note, we rest our case. Broadening our appeal too soon is at least one classic startup mistake we've avoided. We can cross the chasm in due time.

Image credit: Faire Soule-Reeves

{"canonical": "https://blog.beeminder.com/nerds/"}