Hello! I am unable to run opam init on an ipv6-only host due to a routing issue to opam.ocaml.org. The domain resolves to 2001:bc8:5080:8e02::1 and 2001:bc8:1d80:4600::1. When I try to run tracepath it seems to go into a loop:

$ tracepath  2001:bc8:1d80:4600::1
[...SNIP...]
 9:  2001:1900:5:2:2:0:11c:6b2                            51.250ms asymm  8 
10:  2001:bc8:1d00:1::1                                   51.624ms asymm 11 
11:  2001:bc8:1d10:4::1                                   52.090ms 
12:  2001:bc8:1d10:4::6                                   49.801ms asymm 11 
13:  2001:bc8:1d10:4::1                                   52.756ms asymm 12 
14:  2001:bc8:1d10:4::6                                   51.107ms asymm 11 
15:  2001:bc8:1d10:4::1                                   52.004ms asymm 11 
16:  2001:bc8:1d10:4::6                                   49.985ms asymm 11 
17:  2001:bc8:1d10:4::1                                   52.133ms asymm 12 
18:  2001:bc8:1d10:4::6                                   51.931ms asymm 11 
19:  2001:bc8:1d10:4::1                                   51.175ms asymm 11 
20:  2001:bc8:1d10:4::6                                   50.132ms asymm 11 
21:  2001:bc8:1d10:4::1                                   51.262ms asymm 12 
22:  2001:bc8:1d10:4::6                                   50.977ms asymm 11 
23:  2001:bc8:1d10:4::1                                   51.414ms asymm 11 
24:  2001:bc8:1d10:4::6                                   51.919ms asymm 11 
25:  2001:bc8:1d10:4::1                                   51.293ms asymm 12 
26:  2001:bc8:1d10:4::6                                   51.969ms asymm 11 
27:  2001:bc8:1d10:4::1                                   52.046ms asymm 11 
28:  2001:bc8:1d10:4::6                                   50.070ms asymm 11 
29:  2001:bc8:1d10:4::1                                   53.301ms asymm 12 
30:  2001:bc8:1d10:4::6                                   51.972ms asymm 11 
     Too many hops: pmtu 1500
     Resume: pmtu 1500

The lists.ocaml.org service is temporarily sunsetted, but it would be good to bring it back, especially for infrastructure-related requests.

This is related to #15 and restoring inbox.ocaml.org, which also needs an email service inbound for the archival. It would be nice if these were all one SMTP service.

Follow up requests from #45 .

• Tag individual machines as being monitored in a certain way.
• Have a single page explaining the monitoring methods that the first point can reference.
• Also explain the short-comings of the approaches we currently use.
• Document possible approaches for machines that are not currently monitored.
• Export the raw/summary data from Grafana/Prometheus somewhere publicly accessible -- this includes backing up this data which is currently being rewritten every 2 weeks or so I think.

Hi ocaml/infrastructure,

I would like to publish some of the videos from emelle.tv to whatch.ocaml.org. I read on https://watch.ocaml.org/about/instance#terms about opening an issue here to create a personal account. I would like to have an emelletv account If that's possible.

Thanks

It would be useful to host a mirror of the Cygwin distribution for OCaml CI use that:

• has a preinstalled version that can be unpacked without running a graphical installer (for use with Windows 10 native containers)
• controls the versions of packages such that upstream changes can be staged before they hit our CI (and break it, as in ocaml/ocaml#963).

/cc @dra27

What are the plans for server certificates ? Has auto-renewal been put in place somewhere ?

I just got notification from Letsencrypt about the expiration of the opam.ocaml.org certificate, that I got in a hurry in the middle of ICFP. I am going to renew by hand for now, but:

• the ocaml.org certificate is going to expire soon too: who is handling this ?
• are we getting back to a certificate including subdomains, or should I put automated renewal for opam.ocaml.org separately ?

We currently have the staging docs pipeline running at docs-staging.sw.ocaml.org but are serving the static files generated from docs-data.ocamllabs.io.

Please can we have a DNS CNAME entry in ocaml.org for these files? To be consistent with the current naming, this would be docs-data-staging.sw.ocaml.org.

However, I wonder if we might standardise the staging naming with the other services. We have:

• opam.ocaml.org and staging.opam.ocaml.org, and
• ocaml.org and staging.ocaml.org.

Therefore, can we have:

• staging.docs.ci.ocaml.org for docs.ci.ocaml.org (replacing docs-staging.sw.ocaml.org), and
• staging.docs-data.ocaml.org and for docs-data.ocaml.org.

Although we have support for ARM container builds in the infrastructure (e.g. raspbian-8_ocaml-4.03.0), but builds currently fail as qemu threading emulation is insufficient.

It would be good to offload these builds onto a dedicated ARM box with the appropriate base images on the Hub, so that it can be built natively instead of under emulation.

I suggest a pointer to the wiki. I tried to add it myself and make a PR but all kinds of stuff is broken on GitHub when a repo doesn't have any commits... which is why we should add one. Here's the README.md I wrote:

How ocaml.org runs

Look in the wiki.

Merlin mailing-list (barely ever used) is on OCamlforge, which is expected to disappear in the future.
It would be nice to have the mailing-list hosted with the ocaml infrastructure.

Old-url: https://lists.forge.ocamlcore.org/cgi-bin/listinfo/merlin-discuss

I'm creating a Tarsnap account for any "precious" data on virtual machine services. Primarily, this is to back up watch.ocaml.org which has all the videos uploaded. While in theory we can reconstruct this information from the federated Peertube network, in practise it's easier to just have a backup of the volumes.

There were some bad infrastructure failures in Rackspace that led to ocaml.org logs getting lost. As a backup to the live deployment, I fixed up the Dockerfile and a stateless pure-HTML server container would be useful to rapidly push out ocaml.org updates without depending on a single VM.

Add in a simple Markdown Jekyll blog to the infra.ocaml.org site, so that we can post significant updates here that the OCaml community should be aware of. These include significant base container upgrades, changes to central CI and so on. No need for low level operational things like DNS/machine changes, just user-facing things.

Going through the old list of tasks here. There is one item we talked about, moving the https://hub.docker.com/r/ocurrent/opam.ocaml.org images into official ocaml ops account. If we did this a few other services on the ocaml.org deployer could move too.
Out of the things being deployed by the public ocaml.org deployer the two ocaml.org services, watch.ocaml.org, and opam2web aka opam.ocaml.org, might make sense to move all together. The other things like base-image-builder and docs-ci make less sense to change their published image location. What do people think?

Originally posted by @tmcgilchrist in #19 (comment)

I have a pending PR to add opam package build check results in package documentation ocaml/ocaml.org#977 using https://check.ocamllabs.io

This uses the http rest API at http://check.ocamllabs.io/api/v1/latest/packages which lies outside the ocaml infrastructure. Would it make sense to have it in ocaml.org instead?

CC: @tmattio, @kit-ty-kate

Please can the DNS records for opam.ci.ocaml.org and opam-repo.ci.ocaml.org be created/updated as follows:

opam.ci.ocaml.org: Opam CI Web UI points to 163.172.138.177, which redirects it to opam-ci.ci3.ocamllabs.io. Please can opam.ci.ocaml.org be an A record pointing to 212.47.237.247.

opam-repo.ci.ocaml.org: Opam CI service be created as an A record pointing to 212.47.237.247. Currently, this service is reachable only via opam-repo-ci.ci3.ocamllabs.io

check.ci.ocaml.org: Created as an A record pointing to 51.159.138.208. Currently, this service is available via check.ocamllabs.io

Are there specific reasons to:

1. Use OPAM with a deep clone of opam-repository rather than the default https://opam.ocaml.org, and
2. Set WorkingDir to /home/opam/opam-repository rather than /home/opam?

Migrating issue from the wiki to allow discussion.

What should be on a status.ocaml.org page?

At a minimum we should have operational status of:

• OCaml website @ ocaml.org
• watch.ocaml.org
• Documentation CI pipeline https://docs.ci.ocaml.org
• Deployer https://deploy.ci.ocaml.org

What are the options for hosting? Independent of the current infrastructure.

Dear Madam or Sir,

first of all thanks for running opam.ocaml.org as a community service. :)

I noticed from opam update that the opam.ocaml.org hosts are not getting updates since Sunday June 4th 19:04:41 2023 +0100 (commit 9681b042 according to the repo file of the opam.ocaml.org hosts).

I'm curious how to move here, is the infrastructure and its setup/deployment maybe a bit too involved (in terms of complexity, requiring GitHub, some machines to produce artifacts (docker images), Docker Hub for download and upload, and some other machines to execute things), esp. with the recent issues in this area: IPv6 outage, and failure to update some of the machines that serve the repository (missing ssh key).

Another question is whether you have monitoring of the service opam.ocaml.org (about the key things: online, replies to HTTP requests, serves an up-to-date archive), and if yes, is that online and available somewhere? (I suggest setting up a "status.opam.ocaml.org" with some information, and maybe post-mortens about the issues that happened in recent months.)

I hope this is the right repository to report this issue to, in case you've any questions or want to discuss this topic further, don't hesitate to reach out to me.

Once #19 is completed and the new opam.ocaml.org servers are stable, we need to start on merging opam.ocaml.org and ocaml.org/p and finishing the consolidation. Some low hanging fruits first:

• ocaml.org's SEO is terrible at present since there isn't a sitemap.xml generated. (tracked in ocaml/ocaml.org#754)
• the opam.ocaml.org opam2web could be modified easily to add a link to the ocaml.org/p version, which has the documentation. Right now there's no mention of it at all. /cc @dra27 @rjbou
• the ocaml.org package page isn't at parity in terms of the information it displays vs opam.ocaml.org. The two missing fields are package conflicts, and an edit link to the opam file. (tracked in ocaml/ocaml.org#755 and ocaml/ocaml.org#756)

Once that's done, we could do with a plan for how to migrate.

• all old links to opam.ocaml.org must continue to work, but a redirect is fine.
• integrate the opam documentation and manuals directly into ocaml.org.
• redirect the platform blog over to ocaml.org. Already being worked on in ocaml/ocaml.org#691

Disk sdb has failed on x86-bm-b1.

root@x86-bm-b1:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0 953.9G  0 disk  
├─sda1        8:1    0     1M  0 part  
├─sda2        8:2    0   512M  0 part  [SWAP]
├─sda3        8:3    0   600M  0 part  
│ └─md0       9:0    0   599M  0 raid1 
│   └─md0p1 259:0    0   594M  0 part  /boot
└─sda4        8:4    0 952.2G  0 part  
  └─md1       9:1    0 952.1G  0 raid1 
    └─md1p1 259:1    0 952.1G  0 part  /
sdb           8:16   0 953.9G  0 disk  
├─sdb1        8:17   0     1M  0 part  
├─sdb2        8:18   0   512M  0 part  [SWAP]
├─sdb3        8:19   0   600M  0 part  
│ └─md0       9:0    0   599M  0 raid1 
│   └─md0p1 259:0    0   594M  0 part  /boot
└─sdb4        8:20   0 952.2G  0 part  /var/cache/obuilder

Error message in dmesg

[1558841.377823] blk_update_request: I/O error, dev sdb, sector 1052680 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[1558841.379214] md: super_written gets error=-5
[1558841.380677] md/raid1:md0: Disk failure on sdb3, disabling device.
                 md/raid1:md0: Operation continuing on 1 devices.

CI worker x86-bm-5.ocamllabs.io is no longer responding to SSH. Please can it be reset? I suspect another NMVe failure.

Currently we use GitHub as the primary hosting for OCaml.org git repositories.

We would like to have a backup git mirror independent of GitHub, but using the same conversion scripts so that repositories and branches can be shared between the two services. Ideally we could serve it off git.ocaml.org.

From https://github.com/ocaml/infrastructure/wiki/Git-source-mirroring.

What other requirements or features did you have in mind @avsm ?

Can m4 be added to the ocaml/opam2 images?

Dear Madam or Sir,

with huge interest I read through some of the issues in this repository. Thanks for being open and transparent what you like to achieve.

Every other issue when it comes to migrations, I see that there are issues related to let's encrypt certificates and migration of services. The underlying reason, as far as I can tell, stems from the methodology of retrieving let's encrypt certificates: run a "certbot" locally, which requires (a) a web server on port 80, and (b) some ad-hoc configuration to serve static files, and (c) DNS changes being propagated for the desired hostname(s). This means, only once the actual service is deployed to live it can retrieve its certificate. This also makes moving services hard (without downtime).

Over the years, I worked on (fully open source, fully developed in OCaml as MirageOS unikernels) automation to push the whole let's encrypt interaction into DNS (a secondary server on steroids), and thus decoupling the actual service deployment from the certificate provisioning.

The idea is pretty simple: both certificates and signing requests are public data anyways (they're stored in the certificate transparency log, ...). DNS is a fault-tolerant key-value store. Each CSR and certificate is embedded as TLSA (https://www.rfc-editor.org/rfc/rfc6698.html) record (in DER encoding, i.e. no base64/pem, just the bare minimal stuff). Thanks to DNS TSIG we also have authentication (so not everyone may upload CSR) ;).

The mechanism is as follows: the primary DNS sends out DNS NOTIFY whenever the zone changes. The dns-letsencrypt-secondary observes zone(s), and whenever a fresh CSR is detected (or a soon expiring certificate, or a CSR without matching certificate (i.e. key rollover)), the let's encrypt DNS challenge is used to provision a new certificate.

The services behind just download (dig tlsa _letsencrypt._tcp.robur.coop) the certificate, and only need to have their private key distributed.

The operator can use nsupdate to upload a new certificate signing request.

If you're interested in using such a system (and run your own DNS servers - of course you can keep gandi's as advertised ones / public ones), don't hesitate to reach out. I'm happy to help figuring out how to work in that area. :)

As part of the process to reduce our emissions resulting from the ocaml.org infrastructure, we need to first systematically track and measure each service. This issue tracks progress towards determining our current energy expenditure, and we will then have initiatives to reduce and consolidate as appropriate.

• have a machine readable list of physical/virtual machines we are operating, and their locations. We almost have this now with the information in this repository, @mtelvers?
• obtain more specific information about energy usage of the various data centres we use.
  • @patricoferris and @avsm have requested access to the University of Cambridge's energy monitoring platform, for all the cluster nodes hosted there (~12 or so)
  • we use Scaleway's Paris2 datacentre which has some statistics here
  • Equinix Metal and Works on ARM have sustainability reports but need to find more specific information.
• deploy Clarke against the Prometheus instance so we are tracking each machine's energy. @patricoferris and @mtelvers are handling this. As an aside, do we have an ocaml.org-specific instance of Grafana/Prometheus, or is it still running on status.ci.ocamllabs.io?
• publish the data on ocaml.org, and link to it from https://ocaml.org/policies/carbon-footprint
• publish blog post #45 and cross post to discuss.ocaml.org

Once this is done, we can have a review of the actual services and determine how to reduce the footprint.

The machine toxis has multiple hardware issues. The following services have been affected:

• Opam Repo CI - https://opam-repo.ci.ocaml.org
• OCaml CI - https://ocaml.ci.dev
• Prometheus/Grafana - https://status.ocaml.ci.dev

Issues:

• BTRFS volume has been corrupted and causing it to be marked as read-only.
• Multiple ECC memory errors have been reported.

The machine has a spare spinning disk, which has been brought into service with a copy of /var/lib/docker, but due to size constraints, the job log output, var/job, has not been copied.

The current configuration is 2 x 18 core CPUs giving 72 threads with 512GB RAM and 1.8TB SSD. Historically, toxis also performed the solves locally, but this has recently been migrated to the solver-service; therefore, a smaller machine is required. We know that Opam Repo CI requires > 32GB of RAM, which is why it was migrated to toxis.

The suggested new specification is:

• 64GB RAM
• 4 x vCPU
• ~500GB (or larger depending upon how many historic logs we wish to retain)

Hello !

Following the announcement on Discuss regarding watch.ocaml.org, I open this issue on behalf of the Tarides Tech Talk team to obtain an account in order for us to share with the community the recordings we think are valuable for the community. :-)

Let us know if it is possible for us to obtain an account and what other informations we need to provide.

Thank you very much!

The current linked key expired on 28/08/2022 (see ocaml/opam.ocaml.org#25 (comment)).
For the moment, I extended its expiration date to 11/2023. The file need then to be updated on the website.

For future releases, I plan to set up another (more common) mechanism: long standing primary key and "short" subkeys (1-2 years), all registered on key servers.

The OCaml.org and OCaml Platform teams are working towards making the OCaml Platform governance more transparent (and as a by-product, the structure of the broader ecosystem).

We've recently included the OCaml Platform governance policy (including its lifecycle and the requirements for each stage of the Platform) into the OCaml governance: https://ocaml.org/policies/governance.

As a next step, we're working on a governance page that lists the teams and working groups of the OCaml ecosystem: ocaml/ocaml.org#1239.

The main challenge with this page is to accurately list the members of each team (i.e. the maintainers of each project).

We initially attempted to ask the projects for a list of maintainers, but realised that the outcome had false negatives: the lists were biased towards recently active maintainers.

While our goal was always to move towards using GitHub as a source of truth for the list of maintainers so we don't duplicate the information, we now think that this should be a pre-requisite for the governance page, so as to avoid any bias and use a clear and objective definition of "maintainer": one who has commit access to the repository.

However, GitHub teams are not used uniformly across projects in the OCaml GitHub organisation. The present issue is to request the harmonisation of project permission handling to use GitHub teams for all the projects, so we can use these teams as the source of truth for project maintainers.

At the same time, we'd also like to move a few projects of the OCaml Platform that have reached the Active stage to the OCaml GitHub organisation, as indicated in the Platform governance (namely, utop, ppxlib and opam-publish).

If the request seems sensible, I'll edit the issue to add a TODO list I've prepared for the teams to create and Platform repositories to move to the GH organisation.

I'd like to mirror any videos I post to YouTube (ex. https://www.youtube.com/watch?v=OV19_FqAUCw).

Thanks! Jonah

@sabine has a pending PR to add search to ocaml.org in packages documentation ocaml/ocaml.org#1096 using sherlodoc

This search uses the http rest api at https://doc.sherlocode.com which lies outside the ocaml infrastructure. We would like to deploy the sherlodoc executable closer to ocaml.org to avoid depending on an external service.

The main issue is the ~9go database file that needs to be accessible (otherwise we could just run the sherlodoc executable from the ocaml.org container) We aren't there yet, but we are planning to run the database creation as part of voodoo to keep it up to date. In the mean time, it would be nice if the search UI could be made accessible, for users benefit, to gather their feedback and improve upon in parallel...

Do you have some advice on how to proceed? :)

Dear Madam or Sir,

I'm hapoy that after years of a single service running https://opam.ocaml.org, now it is deployed to multiple services (using DNS for "load balancing).

Unfortunately, I've to admit that the provided index.tar.gz are diverging -- the sha256 of c9e5011660bc1b77cc314d97dbca1eed39aa8252a2aaea2cb46931db5905dace is what I get from 51.158.232.133; the sha256 bd8a07db6c14abfb049caef6b120a92da30ff7c6ad031b53a5198c91b8f9d77d is what I receive from 151.115.76.159

Would it be feasible to get a synchronized deployment (and monitoring thereof)? The index.tar.gz from 151.115.76.159 looks very out of date :(
According to the contained repo file, it is 4ac28b10 -- which refers to an opam-repository commit from 3 days ago.

//cc @mtelvers whom I believe is taking care of this as well (please correct me if I'm wrong) /cc @avsm @dra27

After initial success with the new staging.docs.ci.ocaml.org, we can no longer SSH into the machine. Please can you reset it?

Right now the build failures get sent to the opam-commits mailing list, which it appears most people filter. e.g. ocaml/ocaml.org#853 took a few days to spot.

Ideally we would have the cron job failures highlighted somehow, perhaps via an online service that can alert us more reliably when things go wrong (like PagerDuty?).

This issue is to investigate how to improve the state of affairs for opam.ocaml.org to start with, and then the rest of the ocaml.org infrastructure. /cc @AltGr

Now that we have an infra.ocaml.org blog on this repository, it would be good if significant changes to live CI infrastructure could be occasionally rounded up (say, once a quarter, or when there is a big change) and posted to infra.ocaml.org, with the eventual intention of cross-posting this to discuss.ocaml.org so the community is aware.

Might you look into this for recent base-images changes like ocurrent/ocaml-dockerfile#129, @tmcgilchrist @MisterDA? @mtelvers can answer any questions about the blog setup on this repository, but it should be fairly straightforward to clone and contribute a post.

This is a continuation of #49. I don't believe forgotten password emails are going out; I haven't received an email to jonah +AT diskuv.com after doing Reset a few times.

@sadiqj mentioned the other day that a number of the machines used for benchmarking were configured in their BIOSes to turn off hyperthreading and power throttling and generally made more predictable.

This also makes them slower, so we should undo the changes and go back to BIOS defaults when returning them to the pool. /cc @mtelvers to check (I think winter is one of these in the inventory).

In the context of improving OCaml.org's documentation pipeline, we'd like to apply changes to odoc, voodoos, and other projects that the pipeline uses to generate the documentation.

We test these changes locally, however, it'd be best to test them on a production-like environment before they are pushed in production.

The docs pipeline, which outputs its result to https://docs-data.ocaml.org/live/ currently listens to changes on the main branch of https://github.com/ocurrent/ocaml-docs-ci. We propose to have a staging environment for that pipeline that will listen to the staging branch of that repository and will output its result to https://docs-data-staging.ocaml.org/live/.

OCaml.org's own staging environment can then be updated to set the OCAMLORG_DOC_URL environment variable to https://docs-data-staging.ocaml.org/live/ so changes to the docs pipeline staging environment can be tested visually on https://staging.ocaml.org/.

This mailing list thread started by @bluddy got dropped without resolution:

http://lists.ocaml.org/pipermail/infrastructure/2016-July/thread.html

After surveying the opinions of people on IRC and discussions of advancing the community's interests on a mailing list thread ('How to encourage the adoption of OCaml'), I'd like to propose opening a room on gitter.im for the ocaml/ocaml repository (gitter rooms exist on a per-repository basis).

An example gitter room can be viewed here: https://gitter.im/neovim/neovim

In general, I believe having real-time chat with persistent history and notifications is extremely valuable. It is especially valuable for beginners or people inquiring about OCaml, who are afraid to compose a mailing list post. It is also valuable to developers of all sorts who need full-duplex high-frequency minimally-formal communication, which is often hampered by email.

Gitter chat is readily available to newcomers, unlike IRC. Even the best current IRC viewer (https://vector.im/beta/#/room/#freenode_#ocaml:matrix.org) still requires steps to connect and lacks some of gitter's features. Gitter rooms are read-only before logging in (which is still of value), and read-write once you use a github login, which just about every dev has nowadays. Additionally, gitter allows for easy integration with github issues, comments etc as can be seen in the example above.

The trickier part is connecting gitter to IRC. We want to do this both to preserve history on our own terms - scraper tools exist for gitter but it's nicer to preserve our own history using existing methods - and to prevent fracturing of the community. The neovim project linked-to above does this using a gitter-IRC bridge, which must be run on a community server.

This means that to establish the gitter room + bridge we need to:

1. Create the gitter room on the gitter site. This must be done by someone with OCaml github credentials by going to the gitter.im page and choosing to create a room.
2. Follow the instructions at https://github.com/matrix-org/matrix-appservice-gitter using an OCaml community server. The bridge must have access to github credentials as well.
3. For maximum exposure, it would be nice to add the new room to the ocaml.org community page.

Note that gitter rooms already exists for ocaml/oasis and ocaml/opam. These are obviously project-specific and have use by themselves. I envision ocaml/ocaml being used for the compiler as well as for the language as a whole. It's worth thinking about the idea of creating an ocaml/ocaml_beginners repo just for the sake of opening a gitter room for beginners, similar to the way we have a (non-working) ocaml_beginners list. Additionally, it would be a good idea to open a gitter room for ocamlbuild.

The remote solver service currently uses a base aspcud in Debian 7. Upgrade this to aspcud 1.9.1 so that the latest solver criteria can be used by remote clients.

@mtelvers, might you be able to add dates to the front page of blog posts for http://infra.ocaml.org (the 'recent posts' column). That way they're easy to distinguish from the more wiki-like posts such as the opam.ocaml.org description.

Right now older distributions such as Ubuntu 15.10 are simply not rebuilt, but the relevant tags are not deprecated. The effect is that it is invisibly possible to depend on an old OPAM repository.

I'm not sure what the right thing to do here.

• Delete the tag: this will break any repositories that depend on that for CI.
• Leave as-is: security holes mount up in the base distro and OPAM repo is outdated. Leads to issues like this one in MirageOS.
• Rebuild one last time with a big warning: so that CI systems at least see the problem.

If we do the latter, then we can also document prominently the right aliases to use (e.g. ocaml/opam:alpine instead of ocaml/opam:alpine-3.4) so that the latest version is retrieved rather than pinning to something that will be difficult to rebuild in the future.

IPv6 does not work as expected with docker service deploy.

Using this trivial docker-compose.yml as an example.

version: "3.9"

services:

  caddy:
    image: caddy
    ports:
      - "80:80"
    volumes:
      - /etc/caddy:/etc/caddy:ro

  nginx:
    image: nginx

/etc/caddy/Caddyfile contains a reverse proxy definition like this:

:80 {
        reverse_proxy nginx:80
}

Basic connectivity can be checked using using docker compose up and using curl http://<ipv4>:80 and curl http://<ipv6>:80, which both return the nginx default website.

After docker swarm init, the stack can be deployed using docker stack deploy --compose-file ./docker-compose.yml test. Connectivity tests on IPv6 now fail. IPv4 works as expected.

Updating /etc/docker/daemon.json by adding ipv6 and fixed-cidr-v6 as described here does not help.

The situation can be worked around by deploying the caddy image with host ports like this. Now, IPv4 and IPv6 connectivity work.

version: "3.9"

services:

  caddy:
    image: caddy
    deploy:
      mode: global
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
    volumes:
      - /etc/caddy:/etc/caddy:ro

  nginx:
    image: nginx

watch.ocaml.org has been quite successful, and a great source of permanent storage for OCaml videos. It's time to promote it to a non-beta service:

• switch it to being deployed via docker service / the OCaml deployer service. (@mtelvers this was with you last I think?)
• ensure backups (#14) are active for this VM (after previous is done).
• document the process of adding accounts/videos. It's a little confusing due to the ActivityPub integration meaning that the account appears in Mastodon clients rather than the full domain (@avsm/@patricoferris to discuss this in the new year).
• remove the banner saying 'beta service subject to change' (@avsm)

Reported at https://discuss.ocaml.org/t/manual-server-down/12389.

@nojb has got everything ready (as a Dockerfile) for a inbox.ocaml.org replacement, but it needs provisioning as a Scaleway VM. Depends on having #14 for backups first, so history doesn't repeat itself.

opam-2.ocaml.org and opam-3.ocaml.org are presently running on Amazon EC2 VMs which need to be decommissioned. opam-2 is also deploying manually an older version of the Docker deployment on https://github.com/ocaml-opam/opam2web.

Please could we have two Scaleway VMs, which will then be plumbed into https://deploy.ci.ocaml.org/?repo=ocaml-opam/opam2web and used to replace VMs behind opam-2 and opam-3.

• Provision two Scaleway VMs with 2 vCPU, 4GiB memory and at least 100GiB storage (@avsm)
• Deploy both live and live-staging branches to each of the new VMs from deploy.ci.ocaml.org (@mtelvers)
• Copy SSL certs from running VMs to the new VMs to avoid a service gap (@avsm; @mtelvers)
• Switch DNS for opam-2.ocaml.org and opam-3.ocaml.org over to the new machines (@avsm)
• Deprovision EC2 VMs (@avsm)
• DNS round-robin configuration for opam.ocaml.org and staging.opam.ocaml.org between opam-2 and opam3 (@avsm)

At present:

• opam.ocaml.org is a CNAME for opam-2.ocaml.org (54.146.41.74)
• staging.opam.ocaml.org is a CNAME for opam-3.ocaml.org (54.224.129.120)

staging.docs.ci.ocaml.org is provisioned with 16 x vCPU and 64GB of RAM. With the recent work done by @tmcgilchrist these requirements could easily be halved to save on the hosting costs.

We still need to retain about the same amount of disk space ~200GB.

No migration plan is needed, the current instance can be deleted, and we will rebuild using the Ansible playbook.

Equinix is closing the data centre in Amsterdam, which currently hosts www.ocaml.org and staging.ocaml.org. These websites need to be moved elsewhere.

In the short term, I propose moving these to Equinix Dallas, running on ARM hardware.

To facilitate this move, please can these DNS records be updated:

145.40.81.195 v3a.ocaml.org, v3.ocaml.org, ocaml.org, www.ocaml.org
147.75.47.79 staging.ocaml.org

Test deployments are currently visible on these domains v3a.ocamllabs.io, staging.ocamllabs.io pointing to those IP addresses.