Handling updates to the server about adventurelookup HOT 29 CLOSED

All PRs should be subject to review before they're merged in anywhere. Is there really a use-case where a change could cause data loss that's unlikely to be caught on review?

Also, configuring simple mysql dumps (for backup purposes) is going to be a good idea no matter what.

from adventurelookup.

Agreed. We need some infrastructure to reduce manual workload. Deployer seems sensible.

from adventurelookup.

I think it'll depend on how many contributors / PRs there end up being. If it's fairly quiet, manual isn't a bad way to go, since it'll give the team a good deal of control over each release.

I imagine at some point they'll come to trust at least one or two people with keys to the kingdom.

from adventurelookup.

Setting up Travis to push any updates on the master branch directly to the live site wouldn't be difficult to setup. Development could happen on a new dev branch and merged into master as frequent as possible.
But ultimately, it's up to @JohnnyFlash how much control over the servers he wants to / can give away.

from adventurelookup.

It would be nice to have a dev server for testing PRs before deploying to the production site. Could we use @cmfcmf's server for this purpose? That way, the dev team can iterate rapidly and make sure things are stable before deploying to production, so @JohnnyFlash's time isn't wasted.

We could set up a more automated deployment system to the dev server as well, since it's more OK if stuff breaks there. That way we can get stuff tested faster, but without losing positive control over production deployments. Thoughts?

from adventurelookup.

+1'd.

from adventurelookup.

I'm ambivalent about this. The prominence of localized VMs should minimize a lot of defects that break based on OS or configuration variances. No amount of stuttering in the development/production paradigm will fix developers that aren't carefully testing changes and understanding the scope of whatever they're changing.

Also, reverting PRs is easy on github, should the production system run into errors during a deployment, its easy to roll that back and re-deploy again. Even if it causes an outage, it should be, what? 30 seconds? a minute of downtime?

If this were a bigger, mission-critical app, I'd say the extra process and effort of running and maintaining two separate servers was worth it, but for AL being what it is, my opinion is that I don't see a compelling argument for that added effort.

from adventurelookup.

I think the primary argument is that we're dealing with data here. Code wise, you're 100% right, a revert is easy enough. But data wise, a small change in code could result in a catastrophic loss of data.

A staging server won't catch all potential data issues, but it would catch more, and give us a way to preflight schema change deployments.

from adventurelookup.

Also it'd open up changes to be reviewed by more users before going live. @MattColville not spinning up a local VM to check our code. Production shouldn't be the first time he sees things.

from adventurelookup.

Could we use @cmfcmf's server for this purpose?

I would like to shutdown my server soon, because I'm currently paying $10/month to keep it running.

I think the primary argument is that we're dealing with data here. Code wise, you're 100% right, a revert is easy enough. But data wise, a small change in code could result in a catastrophic loss of data.

This also is my main concern when it comes to updates.

from adventurelookup.

Ooh, good point @cmfcmf - you shouldn't have to pay to keep a server running.

@etchalon raised another very valid point in favor of a dev/staging server. Matt isn't spinning up a VM and cloning the repo to look at new features. He's also unlikely to review code in a PR. Having a dev server would allow him to see proposed changes to the site, which would let him give us better feedback.

I'm willing to pick up the cost of a dev server, if we think it would be worthwhile. But, I'm happy to go with whatever the group decides. If the extra hassle isn't worth it, no problem.

from adventurelookup.

I'd also be willing to cover the staging server costs if need be.

In terms of the data risk – this might be a situation where my experience is working against me, but I've seen way too many "non-harmful" updates in my day turn into catastrophic OHGODOHGOD moments. Humans are inherently fallible, and eventually a mistake will happen that costs us some data.

from adventurelookup.

Generally speaking, there are two big types of changes that could affect data. One being migrations (we should be running these automatically as part of the deployment) that affect data, the other being code changes that introduce defects which could affect the collection of new data. Both are concerns, obviously, the former being of higher noteworthiness.

Migrations should be pretty obviously caught most of the time before they're even merged in. If they aren't though, and a migration does get out and run, resulting in the loss of data, data recovery is handled by restoring from a properly orchestrated backup scheme including nightly full backups and incremental binary logs. The later case, where a code change breaks data collection, is a simple reversion and re-deploy (or fixing the defect, whichever makes sense). This one is certainly the most common and is also the most easily prevented through more careful contributor work and a PR review process.

The staging server definitely improves on both of these situations, I'm just not sure building an infrastructure and process around these "rarely occurring" and recoverable scenarios.

from adventurelookup.

A catastrophic "rarely occurring" scenario is still catastrophic.

In terms of recoverability, there are so many moving parts there that it should not be treated as a "eh, we'll be fine" situation.

1. When did the last backups run, and how many changes did the community make since then? Yes, we can recover back to a working set, but we will lose data.

2. Where the last backups any good? Automatic backups fail. Scheduled processes fail silently, and no one bothers to check on them until the moment of disaster. And then it's all "running around with chickens on fire and hair as chickens".

3. When was the error caught? A few days might pass before a data-loss bug gets uncovered. Did you end up with 7 days worth of backups of bad data?

Now, staging servers do not mitigate any/all of these issues, but it does point to the idea that a staging server, where you can preflight the deployment of an update in a matching environment, and allow for non-developer testing, is, baring a cost concern, insanely valuable.

from adventurelookup.

Not only do staging servers not mitigate all of those problems, but they're totally different issues.

If your production environment does not have database backups running reliably and tested regularly you have a major failure point regardless of how many barriers exist between developers and getting code onto production.

from adventurelookup.

Bad code can and will make it onto a server. Better it be a staging server than production.

Which doesn't mean it won't happen in production, nor does it mean you need a hundred staging servers "just in case". Only, that it's a sensible, minimal-effort step that often catches problems early.

We've already seen one bad deploy go into production and take the site down for a moment. That would have been caught with a staging server, and the community would have avoided any downtime.

There's a very, very good reason that staging servers are a thing people do.

from adventurelookup.

They're a very frequently used tool in business applications; they're almost unheard of in open source development for good reason, they're often unnecessary.

Ultimately I'm not stringently opposed to it, but a lot of the arguments I'm seeing used have nothing to do with what this project needs, its purpose, and the best way to achieve those purposes pragmatically.

If bad code makes it onto staging, who exactly is going to be testing it all the time to find regressions? Who exactly is going to be allowed to continually merge bad code to the staging server and still be treated seriously by the other contributors? Who is approving PRs that bad code is a continual concern where we need a staging server?

from adventurelookup.

I see your point, @Harkenn, but still see three benefits to a staging server:

• It insulates the production site from catastrophic, site-breaking bugs like the one we had earlier. You are absolutely correct that these shouldn't happen if a) devs are good about testing their code before submitting PRs, and 2) we are careful about reviewing PRs before merging. However, stuff happens. While the community will understand occasional instability why not reduce the risk as much as we can?
• It lets us test and refine new features before they go live. This is especially important for getting Matt's feedback, as he is unlikely to sync the repo and set up a local server. Having a staging site live seems like the quickest way to let him test drive new features. It also seems like the best way for us to collaborate on things like the UX/UI changes that @etchalon is working on.
• It reduces Jerod's workload a bit. With a staging server, we would make fewer PRs to this repository, which means fewer merges that Jerod has to handle. We could do a "batch" PR once or twice a week, which would already have been tested, so all Jerod would have to do is merge and deploy to the production server.

Having said all that, I appreciate that managing a staging server is extra work, and we are all busy. I'm happy to go along with whatever everyone else wants to do.

The issue of making regular backups of the database seems separate to me, and is an absolute no-brainer. The database should definitely be getting backed up nightly!

from adventurelookup.

Again, I'm fine with a staging server, but I don't want there to be the illusion that it provides some protection that it actually doesn't. If we aren't reviewing code carefully and if contributors aren't testing the changes they make before submitting those changes, it won't matter if we have a staging server or not. Having a staging server may sound (or even seem, to those inexperienced in the matter) to provide a protective layer, that shields the production server from bugs....but it doesn't.

Almost no one is going to consistently spend the time fiddling around with the staging server looking for bugs that some other contributor created. So who exactly is going to be using the staging server? Not the contributor themself. If they were going to be testing their own bugs, they would find the database-destroying defect on their local VM during their testing, before they submit it.

Many instances of staging servers actually end up being a bed to smoke-test the system such that it appears ok...and all that ends up doing is promoting lazy developers, because hey..if they break anything, surely it'll show up on staging on a smoke test that they may (or may not) do before it gets rolled out to production. Its just not the "solution" that everyone seems to think it is, in my experience.

from adventurelookup.

While a staging server won't let us catch all, or maybe even most, bugs, I bet it'll catch some! And that's better than none.

Also I really like the ability to check out the staging build and see how everything's working and sign off on it before it goes live.

from adventurelookup.

That's perfectly fine with me.. I want to make sure you guys understand that I'm not trying to be disagreeable; I just want to see AL succeed

from adventurelookup.

The participation of passionate, intelligent people is what will help AL succeed. Sadly, it seems passionate, intelligent people seem to have a whole lot of opinions about everything.

(I didn't think you were trying to be disagreeable at all! You just have a different experience/opinion.)

from adventurelookup.

No worries, @Harkenn!

from adventurelookup.

What's the best way to get a staging server up? @cmfcmf shouldn't be paying for it. I'm happy to pick up or split the cost, but setting the server up is beyond my technical abilities.

from adventurelookup.

I can configure it; a small, cheap host on linode or digital ocean is probably sufficient? Maybe the $5 or $10 ones? Technically I have a server that I'm barely using and still paying for every month. I can put it up on there and see if I can get it working....but I won't get to it until a ~5-6 hours from now.

from adventurelookup.

I would be happy to duplicate the current VM and put it up on dev.adventurelookup.com

I would also be fine with giving you guys ssh access to that server. I would just need public ssh keys for you.

That being said, work has been crazy right now, so it may take me a couple days get it up and running. Also Matt has control of the AL.com DNS and he's off to Comic-Con tomorrow.

from adventurelookup.

Sounds great, @JohnnyFlash! My SSH public key can be found here: https://github.com/cmfcmf.keys (the third one please).

from adventurelookup.

Thanks, @JohnnyFlash! My public key is at https://github.com/jsaugustyn.keys

from adventurelookup.

@JohnnyFlash My key can be found here: https://github.com/etchalon.keys

from adventurelookup.