This would also help with the prereqs of having python, curl, git, svn, and the others listed in the readme. (The readme's dependency list assumes using wattsi-server for remote builds, so it's not complete---for a truly local build you'd also want wattsi.)

from html-build.

This is the first pass at making a base image based on the items I found on the readme of this project. I will first describe how to achieve and end to end build with the basic Dockerfiles I created and then outline some of the things I think could be changed around in the current build.sh setup I encountered while setting up the Dockerfile.

Build

End result is a docker container with nginx where you can see your build artifacts from the html repo.

Build base build image

1. Clone my fork github.com:defunctzombie/html-build.git somewhere (does not matter where).
2. In the cloned directory run docker build -t whatwg-html-build

This build the base image and tags it locally as whatwg-html-build which is important for the next step.

Build the html repo

1. Clone the [email protected]:defunctzombie/html.git repo (again doesn't matter where)
2. In the cloned directory run docker build -t whatwg-html
3. run docker run --rm -it -p 8080:80 whatwg-html

You can open http://localhost:8080 to see your built output files (osx users will need the IP address of their docker-machine vm which you can get with docker-machine env command).

** An alternative approach would be to have a Dockerfile in this build repo and reference it from the html repo to build like so: docker build -f /path/to/html-build/html.Dockerfile whatwg-html. I find this solution more confusing for the end user. With the Dockerfile in html repo solution (as I have setup) the whatwg team can publish the base build image to docker hub and the html repo contributor does not need to ever really know about this repo.

Thoughts

• This was a quick hack based on just running the build.sh script; I believe the real version should have a simpler script which only does the building of the output files. All of the dependencies (wattsi, etc) should be installed in the base image and not part of this build script. I did not go digging around to do this because I am not familiar with the build process to do such without some guidance on preferred approach.
• index file in html repo should be called index.html (I found this trivial and annoying :/)
• nginx was just the first thing that came to mind to serve the built output files, if there is something else that you prefer using that would be just as easy to do. nginx (or whatever) could go into the base image. I left it out of the base image because you could imaging it not being relevant to the build tools image but since this base image is created specifically for the other repo everything can go into it.
• base images should be versioned when actually pushed up to docker hub (good practice to avoid breaking things later)
• Whenever a developer makes a change and wants to see their final build files they can simple run the docker build command followed by the docker run command. This is the most versatile approach as users on windows and osx cannot just "mount" the cloned source tree without some extra pain steps.

Final

Happy to help push this over the line. The biggest point now is to really determine what workflow you are ok with and would like to aim for. IMO (based on quick scan of this and other repo) would be one with a base image full of all the build deps published to docker hub and then a basic Dockerfile for contributors (with instructions) in the other repo.

from html-build.

I guess I should add that the Dockerfile in the html repo could (without much effort) also become the Dockerfile used to deploy the production site but I imagine that is done via other means so didn't really think it important to discuss.

from html-build.

index file in html repo should be called index.html (I found this trivial and annoying :/)

Yep; also see whatwg/html#507

from html-build.

OK, got a chance to try this!

In the cloned directory run docker build -t whatwg-html-build

I think this is supposed to be docker build -t whatwg-html-build .? And for the other repo too.

This was a quick hack based on just running the build.sh script; I believe the real version should have a simpler script which only does the building of the output files. All of the dependencies (wattsi, etc) should be installed in the base image and not part of this build script. I did not go digging around to do this because I am not familiar with the build process to do such without some guidance on preferred approach.

It worked pretty well IMO!!

The biggest thing to add is that we'd like to get Wattsi installed in the base image. Currently the build.sh script will detect if wattsi is in the path, and if it is not, it will call out to a web service to do the build for us. We'd like to avoid that in the docker world, so it's a pre-packaged solution that works for people with spotty internet.

To install wattsi, the process is basically: download the appropriate tar from ftp://freepascal.stack.nl/pub/fpc/dist/3.0.0/, run its install.sh, then make sure the resulting fpc (Free Pascal Compiler) is in your PATH. Now clone whatwg/wattsi, and run ./build.sh. Now put the resulting wattsi binary in your PATH.

nginx was just the first thing that came to mind to serve the built output files, if there is something else that you prefer using that would be just as easy to do.

Ideally we'd use apache, since that's what production uses and we currently have a bunch of .htaccess files to control apache. I think there is definitely interest in moving to nginx eventually, but one thing at a time probably.

If we could get apache bundled in with all of the appropriate apache addons that would be great. I remember getting apache working locally and it kept throwing errors about us using something inside .htaccess that required me to install an addon.

nginx (or whatever) could go into the base image. I left it out of the base image because you could imaging it not being relevant to the build tools image but since this base image is created specifically for the other repo everything can go into it.

The html-build image vs. html image is an interesting distinction. I can kind of see how it works, but would you mind explaining how you see the split, or maybe how such a split is done in more conventional projects?

One thing that might be good is if we can consolidate it so that there's fewer commands to run. But otherwise I am OK with a split.

Whenever a developer makes a change and wants to see their final build files they can simple run the docker build command followed by the docker run command.

Got it, I was wondering about that. It seems reasonable enough, as long as we have a nice "How to use Docker to do all this" guide.

Happy to help push this over the line.

Thank you!!!!

The biggest point now is to really determine what workflow you are ok with and would like to aim for. IMO (based on quick scan of this and other repo) would be one with a base image full of all the build deps published to docker hub and then a basic Dockerfile for contributors (with instructions) in the other repo.

So let's see if I understand. The idea is that we set up the build environment (and maybe the server environment?) in the html-build Dockerfile, and push that to docker hub. Whenever the build repo changes we rev the version and push again.

Then, people who know how to use docker don't have to clone html-build at all. They just clone whatwg/html, and run a couple docker commands each time they want to build/view the server. Sounds pretty good to me!

The only thing that seems a little annoying is the need to stop the server, build, restart the server. If there were a way to have the server run continuously and get fed the result of the build every time you run some build command, that would be ideal. If it helps, build.sh will put its output in $HTML_OUTPUT if that variable is set.

I guess I should add that the Dockerfile in the html repo could (without much effort) also become the Dockerfile used to deploy the production site but I imagine that is done via other means so didn't really think it important to discuss.

It's a pretty attractive idea that I'd like to explore medium-term, maybe if we move our server to nginx. But yeah, let's not worry about that for now.

from html-build.

Ideally we'd use apache, since that's what production uses and we currently have a bunch of .htaccess files to control apache. I think there is definitely interest in moving to nginx eventually, but one thing at a time probably.

Works for me, its been years since I have configured or launched apache so would just need someone to provide the apache config needed to serve the site. I thought it was all static assets but could be wrong.

So let's see if I understand. The idea is that we set up the build environment (and maybe the server environment?) in the html-build Dockerfile, and push that to docker hub. Whenever the build repo changes we rev the version and push again.

Yep. Although upon further thought tho you could do this with just a single Dockerfile in the html repo. The primary difference between this approach and the approach of baseimage in docker hub is that for the first docker build run experience. With the two image approach (the dockerfile in html repo being started off some custom base image) the build will only need to fetch the image from docker hub and then do the actual build. If you use a single dockerfile (no build base image or repo) then the first parts of the dockerfile will be to install the build deps (which would otherwise be present in the base image).

IMO the typical approach you see in most places is the single dockerfile approach whereby there would be no html-build repo image and the dockerfile in the html repo would just perform the steps needed to install the build tools first (easily done by cloning the build tools repo inside the docker image build process). You may think that this will slow down every incremental docker build after the first one but it won't due to the way docker layer caching works. As longs as the dockerfile is written correctly, subsequent docker build commands won't need to re-install the dev tools again. This approach is the typical one cause there is not usually a separate build repo.

That being said, I think there is something to the idea of a build base image and if you had more repos all needing the same base tools, etc I would say it would be worth exploring but if you only have just this one repo to build then it isn't as big a win IMO and a single Dockerfile in the html repo would suffice.

The only thing that seems a little annoying is the need to stop the server, build, restart the server. If there were a way to have the server run continuously and get fed the result of the build every time you run some build command, that would be ideal. If it helps, build.sh will put its output in $HTML_OUTPUT if that variable is set.

I agree. The primary issue here is how to get the source files into the docker container. Unfortunately on windows and osx systems docker runs in a VM which means to "mount" any folder into a docker container (using docker volumes) that folder must first be mounted into the vm environment. While not impossible (the more crafty users certainly will do it if they want) the is a PITA for a small change if you are not day-to-day working on the repo. Fortunately docker layer caching can make the rebuild process faster if the dockerfile is setup correctly.

I will update the dockerfile in the html repo to not require a custom base image so you can get an idea of how that process would look. I think for your current use case it would be better cause you won't need to manage anything in docker hub. If in the future that changes then a base tools image would be simple enough.

from html-build.

Works for me, its been years since I have configured or launched apache so would just need someone to provide the apache config needed to serve the site. I thought it was all static assets but could be wrong.

It's all static assets with some checked-in .htaccess controlling the rest. You might need to install some mods that don't come out of the box though. I apologize for not remembering them but I think there were like 2. You should be able to figure it out pretty quickly when you try to run the server then get errors in the logs saying "did not recognize htaccess directive somethingsomething". If this sounds like a pain let me know and I'll wipe my Apache install and start from scratch recording which mods I install this time.

My apache site config looks like this:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /home/domenic/src/html-build/output

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    <Directory /home/domenic/src/html-build/output>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>

(the log and server admin lines were there by default). I used https://www.digitalocean.com/community/tutorials/how-to-configure-the-apache-web-server-on-an-ubuntu-or-debian-vps to get myself up and running.

Looking forward to seeing the single-file take; sounds pretty good!

from html-build.

I have updated the Dockerfile in the html repo to not have any other image or repo dependencies.

The instructions are now simply:

1. Clone the [email protected]:defunctzombie/html.git repo (again doesn't matter where)
2. In the cloned directory run docker build -t whatwg-html .
3. run docker run --rm -it -p 8080:80 whatwg-html

There is no need to build any base image or use docker hub for the base image.

thoughts

• I found it a bit strange to have a separate html-build repo instead of just putting those scripts in the html repo itself. Usually you do this when your build tools are used across many projects but that doesn't seem to be the case so having two repo just adds extra work since now any tweaks to the building process require changing in html-build first and then making use of the new changes in the html repo. Again depends on your use cases.
• I didn't dig into the build.sh script but it would be nice to have a pre-build or dependencies script and the actual build script. I couldn't quite discern which items in the cache vs html source dir that got installed during the build.sh process were just deps that could be pre-installed before the build and which were tied to the build and needed to be in the source dir. In either case, making that a bit clearer would allow the dep install process to happen before the build.sh process in the docker file and thus not need to re-download the deps on each rebuild as currently happens. I didn't want to mess with the build.sh file or split it up since that would then mean putting from that fork of the repo (which gets back to the above point).
• After doing both the two file and single Dockerfile approach, I think the single file is 100% the way to go for now due to the simplicity.
• Apache is old :-p

from html-build.

OK, this is looking pretty good. Sorry for being away for a week and dropping the ball on this.

Any chance you could get a wattsi binary in there as described in #55 (comment)? I'm actually having trouble testing it since the build server is kind of broken right now...

from html-build.

This can be closed in favor of whatwg/html#612 since the changes are only needed in the html repo. If a separate build image is desired (and published to dockerhub) that work can be based off the PR in the html repo.

from html-build.