With the current nginx config, I can't run the app locally. If I comment out the SSL stuff, it's fine.

This is true even when enable_ssl is false in the config.lua

Looks easy:

https://www.cyberciti.biz/faq/linux-unix-nginx-access-control-howto/

I guess we could even do that dynamically. We know the paths these bots are trying to target, so we could automatically ban IPs that try to access things like:

• /manager
• /xmlrpc.php
• /blog/xmlrpc.php

It would be nice to do this at some point.

For logging, it seems from postgres.conf that logging is off. I think we should turn on logging and bump up the level to warning -- at least until we know how stable things are.

Edit: It is logging. :) But I still think we should bump the level a bit.

I'd also like to figure out some nice tool for actually monitoring performance, but I don't have specific ideas yet.

Both these changes will require restarts.

http://snap-cloud.cs10.org/xyz

I'm logging this as something to discuss.

EDC is wondering about us running a CORS proxy, since all the existing ones keep disappearing.

Basically we'd run our own CORS proxy but lock it down so it only works with Snap!. I'm not sure if simply checking for an active session is enough? Would it need monitoring or auditing of users to make sure things are safe?

We could store if an email bounces and warn users.

We could even potentially store the last time an activation email was delivered and update users that way.

This way when users log in the messages will be more informative.

Putting any info here to log the work that's done, and note progress etc.

Useful sites:

• https://calnetweb.berkeley.edu/calnet-technologists/incommon-comodo-certificate-service/extended-incommon-ssl-certificate-example
• https://www.sslshopper.com/ssl-checker.html#hostname=snap.berkeley.edu
• https://www.sslshopper.com/ssl-checker.html#hostname=cloud.snap.berkeley.edu

Easy enough. :)

This will help keep the server in sync and will make dev'ing easier.

Not blocking a release, but could be helpful.
http://leafo.net/lapis/reference/database.html#database-schemas

Using the lapis schema and migrations will be helpful for when things change overtime.
if anyone of us updates the DB structure then we can all just run lapis migrate to only get the new changes.

Instead of sending down base64 data, I think we should try to send URLs down instead of data URLs. Primarily, base64 data is about 1.3x the size of the binary format, and it would avoid repeated encoding (on the server) and decoding on the client.

you'd still get asynchronous loading of thumbnails in the Open dialog in Snap!. I guess the Canvas version is where an actual image file might need to be re-encoded, but I still think this would be faster since you'd be transferring less data.

The main difference would be in the request sent down for a page with a lot of project thumbnails. Instead of 1 giant request, the page would load 1 JSON result, then N small requests directly for each thumbnail file. Given that modern browsers (and nginx) handle http/2 well, I think this should make the index pages load quite a bit more quickly.

This isn't necessary for a demo or anything, but I think it'd help performance (both on the server and in the front end).

This will make things more secure.

To investigate!

There used to be validations on dom_username, but they were removed:

6990be1#diff-bef302d3cdd1ce8c0d4e2e008fa3652a

If there's no validations, then should we not just use the text type? (It would make writing the initial migrations in #10 Easier).

Will come in handy for admin utilities and sorting. :)

All requests should be upgraded to https, except for a specific folder, which would make working with robots easier.
It shouldn't apply to the localhost server. -- easy to do with #68

Why not do it once before any request is processed? Otherwise, we're bound to slip up.
Like here, https://github.com/bromagosa/snapCloud/blob/master/app.lua#L89

It seems that a lot of people end up on the invalid token page because they visit the same URL twice.

I wanted to add something to the Invalid token page that explains the user's verification status. However, without knowing the token, explain the verification status would make it a public value. (Maybe that's really not such a problem, but it's worth thinking about.)

I see two changes we could make:

• not delete tokens when they're used for verification (password tokens need to be deleted, of course)
• keep expired tokens around a bit longer (typically they're deleted fairly quickly).

In the first case, during the window where the token is still valid, we would (in the simplest version), re-apply the verified update. I think that's not the worst thing to do. Most sites don't seem to warn you if you activate your account twice, and if they do it's not an error.
(if it were a big problem, we could track the used as a column.)

When your token is expired but we know it used to exist we can tell users their status, and then direct them to resend a verification email if necessary.

I adjusted the log in nginx.conf to log to logs/error.log -- notice seems to be quite noisy. It should be fine for now but we might want to turn it down. Logging query times is helpful for now. Things are good. :)

Most actions are fairly simple, but it would help, I think to have some tests.

It would be useful to track some login data:

• session count
• last login time
• most recent IPs

The last one isn't super useful right away but it could help later on with detecting anomalies and such.

@cycomachead can help :)

We should probably expose a config that's a list and join by .

I'm mostly done configuring the staging server, but we need to do a few more things:
It's snap-staging.cs10.org for now.

• create a local DB instance on the staging server
• expose a staging config in config.lua
• update the allowed domains
• update snap to point to the staging server
• consider getting a berkeley SSL cert?
• setup http basic auth? So people don't accidentally use the wrong server.

Eventually, I think we should setup a script to extract a subset of data from the production server to the staging server. (Done for admins; it can easily be extended for others.)

Right now, everything returns 200.

Most of these should be straightforward, but I think we should also take case with accessing an existing private project project.
We're should probably say "This project doesn't exist" and return a 404, not a 403. (And definitely not a 200).

It seems like a few of the undelivered emails are probably typos. We should allow users to change their account email, and then reset the verification process.

For example, this can happen:

Michael at Yosemite-Retina in ~/Dropbox/Projects/snap/snapCloud on master [!]
👉 curl -k -X POST https://localhost:8081/users/tester -d password='123456' -d password_repeat='123456' -d email='[email protected]'
<!DOCTYPE HTML><html lang="en"><head><title>Error</title><style type="text/css">        body {
          color: #222;
          background: #ddd;
          font-family: sans-serif;
          margin: 20px;
        }

        h1, h2, pre {
          margin: 20px;
        }

        pre {
          white-space: pre-wrap;
        }

        .box {
          background: white;
          overflow: hidden;
          box-shadow: 1px 1px 8px gray;
          border-radius: 1px;
        }

        .footer {
          text-align: center;
          font-family: serif;
          margin: 10px;
          font-size: 12px;
          color: #A7A7A7;
        }
      </style></head><body><div class="box"><h1>Error</h1><pre>/usr/local/share/lua/5.1/lapis/application.lua:396: /usr/local/share/lua/5.1/lapis/db/postgres.lua:82: INSERT INTO &quot;users&quot; (&quot;created&quot;, &quot;salt&quot;, &quot;joined&quot;, &quot;isadmin&quot;, &quot;username&quot;, &quot;email&quot;, &quot;password&quot;) VALUES (&#039;2018-02-09 07:43:36&#039;, &#039;613a59cebf7a14a385aa1f52707480fa&#039;, &#039;2018-02-09 07:43:36&#039;, FALSE, &#039;tester&#039;, &#039;[email protected]&#039;, &#039;6df585c38ceaf50d05716362029f877b59ad718817e5fb3241b0c4871978f0b689646740eb830dfe06a030e69517d99dfcd0134201b8dd5373a2742d01a08cda&#039;) RETURNING &quot;username&quot;
ERROR: column &quot;joined&quot; of relation &quot;users&quot; does not exist (41)
stack traceback:
	[C]: in function &#039;error&#039;
	/usr/local/share/lua/5.1/lapis/db/postgres.lua:82: in function &#039;insert&#039;
	/usr/local/share/lua/5.1/lapis/db/postgres/model.lua:190: in function &#039;create&#039;
	./api.lua:126: in function &lt;./api.lua:113&gt;</pre><h2>Traceback</h2><pre>
stack traceback:
	[C]: in function &#039;error&#039;
	/usr/local/share/lua/5.1/lapis/application.lua:396: in function &#039;handler&#039;
	/usr/local/share/lua/5.1/lapis/application.lua:130: in function &#039;resolve&#039;
	/usr/local/share/lua/5.1/lapis/application.lua:161: in function &lt;/usr/local/share/lua/5.1/lapis/application.lua:159&gt;
	[C]: in function &#039;xpcall&#039;
	/usr/local/share/lua/5.1/lapis/application.lua:159: in function &#039;dispatch&#039;
	/usr/local/share/lua/5.1/lapis/nginx.lua:214: in function &#039;serve&#039;
	content_by_lua(nginx.conf.compiled:69):1: in function &lt;content_by_lua(n`ginx.conf.compiled:69):1&gt;</pre></div><div class="footer">lapis 1.5.1</div></body></html>
``

Whoops, my fault! Fixing.

Not to publish the package, but I think there's an easy way to install them all?

• using .env (source .env && lapis server ...)
• using a different SSL port on macOS. 443 requires sudo, but anything else doesn't. I'm using 4431.
  • could / should this be the default dev ssl port?
• generating a local SSL cert. (Maybe we can just script this?)

You can't just run lapis server in the current configuration because the auto-ssl library needs access to protected folders. This should be configurable.

(I'm going to try to make this work).

If a user is already verified and asks for a verification link, send an email informing that the account is already verified instead.

https://amplify.nginx.com/docs/guide-metrics-and-metadata.html#additional-nginx-metrics

I set up some monitoring, up to this point, but this involves just tweaking the log format.

The way the current Snap! front end works is that it hashes the passwords once, and then sends them down.

Since that's the case, we can eliminate the prehash step here:
https://github.com/bromagosa/snapCloud/blob/master/validation.lua#L85

BTW, where is the new cloud.js is? Is it still the one in snapSite?

Should detect reasonably unique views.

I don't get why these are in the gitignore (I know lapis adds them by default, but it doesn't make sense. lol).

mime.types is needed even to boot the app -- shouldn't it be included?
And I don't know too much about lapis, but the models seem like the type of thing to include too. :)

(I still haven't fully booted everything, but I think I've got it close now.)

Things we almost certainly should do:

• gzip responses

Things to explore:

• increase the open file limits

Some references:
https://gist.github.com/v0lkan/90fcb83c86918732b894

When I visited /users/c it for my account is says {"isadmin":false,"username":"cycomachead"}

When I do /users/cycomachead I see: {"isadmin":true,"created":"2018-01-25 15:53:39.669+00","username":"cycomachead","email":"[email protected]"}

I'm was also seeing the "Your account is not verified" message.

When I logged out, then logged back in, both the /users/c endpoint and the (absence of) the message in Snap! were correct.

We should make sure that password hashes aren't showing in the logs.

No POST data is being shown it seems, just the query strings. I wonder if the easy solution is to fix the way the params are passed.

https://github.com/bromagosa/snapCloud/blob/master/api.lua#L120

It seems like we should be using something a bit stronger for a salt.
https://github.com/openresty/lua-resty-string seems like a good bet, and the readme has the following example

 local resty_random = require "resty.random"
    local str = require "resty.string"
    local strong_random = resty_random.bytes(16,true)
        -- attempt to generate 16 bytes of
        -- cryptographically strong random data
    while strong_random == nil do
        strong_random = resty_random.bytes(16,true)
    end
    ngx.say("random: ", str.to_hex(strong_random))

It seems like the error log gets clobbered when the server is restarted. I'm not sure if that's b/c the config wasn't saved or not.

This isn't necessary, but Postgres has some improved parallelism which might be useful since we have a large enough server.
https://www.postgresql.org/docs/10/static/release-9-6.html

We should consider providing support at the server level for places where a cookie won't work.

There are two options:

• always generate a session token AND use cookies
• detect if cookies fail and then only generate a session token.

I think this flow might work for Snap!:

• User makes login request
• Server sends back {cookie, token}
• Snap! Detects if cookie works
• If cookie works, Snap! ignores the token and it will silently expire on the server
• If cookie fails, then Snap! Saves the token and sends it w/ each request
• On the server: if there's a cookie: no change
• If there's no cookie, use the session token to perform a user lookup.

There's a slight bit of overhead but in most cases it shouldn't be a big deal. It's only 1 database query to regenerate the session. (In almost all cases there's an existing query, so if we were clever we'd just use an extra JOIN.)

Not sure if there's a good way to do this without a bunch of copy-pasta in the nginx config. That's probably OK for a while.

From @cycomachead on December 28, 2016 6:52

I don't know why I never wrote this down, but it would be awesome if the browser level settings could be saved to my cloud account. I switch browsers and devices frequently and my settings are often out of sync.

Copied from original issue: jmoenig/Snap#1588

Sigh, this one may be annoying, but it's worth exploring as we haven't yet found a long term partner for the cloud. I'll look into this once things settle down.

Brian would like all private projects to be encrypted with a user's password.

This nicely outlines the basic steps:
https://stackoverflow.com/questions/7690339/how-can-i-encrypt-user-content-on-my-site-so-that-not-even-i-can-access-the-cont

Though, we need to think about how/when that password is sent and what parts should(n't) be encrypted.