Spago database falling out of sync with registry without explicit `spago fetch` about spago HOT 11 OPEN

Encountered a second issue related to this. This time, installing the spec package results in the same error:

[     328ms] Got metadata from DB: spec
[     328ms] 
❌ Couldn't read metadata, reason:
  Didn't find version in the metadata file

However, running spago fetch doesn't help. I get the same error! So, once again I looked at the global cache for the registry and registry-index repositories and confirm that the version in package set 50.15.0 ― [email protected] — is present in both the registry metadata and registry index.

But then, weirdly:

➜ spago registry info spec
<snip>
  7.5.5:
    bytes: 19330
    hash: sha256-HdyBH7Ys1/m2SdTq3u2u9LdQ4cGeaohWeEMYay2mHdU=
    publishedTime: 2023-09-07T07:15:30.137Z
    ref: v7.5.5
unpublished: {}

The spago registry info command only finds versions up to 7.5.5! Now that is weird, because my package set and checkouts of the registry and registry-index all reflect 7.6.0.

This one is a blocking issue; I can't install spec because I always encounter the "didn't find version in metadata file" error and I haven't found a way around it.

from spago.

This is another downstream consequence of the whole "lockfiles make spago not hit the network": we used to always pull the registry and refresh the database, but we don't do that as consistently anymore and this creates discrepancies in the DB cache - see #1214 for another example of this.

I'll have a look at this, but in the meantime does removing the lockfile help? This should trigger a proper refresh.

from spago.

Had a little digging:

• every time we pull the Registry we update the package set entries in the DB, so - as you verified - these are always up to date
• however, that's not the case for Metadata files: we only insert them in the DB with the insertMetadata function, which is ever only called in Spago.Registry.getMetadata, which we call a few times in the implementation of fetch:
  • when we need to write a new lockfile (and here as well)
  • and when fetching a package from the Registry

Given the last point, the fact that fetch works and install doesn't is weird - they mostly share the same code path, given that install is fetch + build.
The only difference is this case is that we are installing a specific package, and there's a separate code path for that, which is where this must be failing. We don't do anything Registry-related there though. The only other difference is that we invalidate the lockfile when installing new packages, but that should actually help since we'd have to refresh it. Quite puzzling.

from spago.

Is there a reason not to fully update the database (ie. including metadata) every time we pull the registry? I was puzzled to see the registry metadata in git was up to date, but not up to date in the database.

from spago.

I'll have a look at this, but in the meantime does removing the lockfile help? This should trigger a proper refresh.

I’ll have to verify this when I’m back at that machine; if I remember correctly, no, removing the lock file didn’t make a difference.

from spago.

Is there a reason not to fully update the database (ie. including metadata) every time we pull the registry?

"Fully update the database" in this case would mean reading all the metadata files potentially every 15 minutes (which is our cutoff for "the registry is fresh enough to not pull"). It's not necessary because very few of them have changed at any given moment, so we read them up on demand, which worked when the lockfile was not delaying updates.

As a side note, I have never encountered this particular instance of the problem - I would be curious to see if we can repro, e.g. fresh cloning of a repo with a lockfile and no local database

from spago.

Interesting — I came back to the machine I was testing this on and attempted an install of spec and this time the lockfile was detected as out-of-date, it got regenerated, and the package was installed:

➜ spago install spec
Reading Spago workspace configuration...

✅ Selecting package to build: wat

Refreshing the Registry Index...
Adding 1 package to the config in spago.yaml
Downloading dependencies...
Fetching package [email protected]
Lockfile is out of date (installing new packages), generating it...
Lockfile written to spago.lock. Please commit this file

This is odd because I haven't even accessed this machine since I was testing for this issue. Nothing has changed about its state except for more time elapsing, which makes me think there is some interaction with the 15-minute timer too.

from spago.

I wonder if this logic is at fault?

The important bit is that not fetchingFreshRegistry: if we have cached a metadata file, we only go read it if we are outside of that 15 minutes window.
I can now see how the bug happens: if you recently pulled the registry, and then go do spago install spec, and no projects recently refreshed the metadata file for spec, then it won't be found in the DB cache.

Not sure how to go at it - maybe we should stat the metadata file and see if it's newer than the copy we have in cache, and then read it if not?

from spago.

Encountered this again, but this time without even being in a Spago project at all. When running spago registry package-sets the entries I see are way out of date:

(nix:purescript-overlay-env)$ spago registry package-sets
+---------+------------+----------+
| VERSION | DATE       | COMPILER |
+---------+------------+----------+
(...previous entries omitted)
| 48.2.0  | 2024-01-16 | 0.15.14  |
+---------+------------+----------+

This is showing the latest package set is from Jan 16 2024, but of course we've had many releases since then. So I thought — is this a spago fetch issue again?

(nix:purescript-overlay-env)$ mkdir wat && cd wat && spago init
Initializing a new project...
Refreshing the Registry Index...
Found PureScript 0.15.16, will use package set 51.1.0
Set up a new Spago project.
Try running `spago run`

OK, spago init picked up a later package set. What does spago registry return now?

(nix:purescript-overlay-env)$ spago registry package-sets
+---------+------------+----------+
| VERSION | DATE       | COMPILER |
+---------+------------+----------+
(...previous entries omitted)
| 51.1.0  | 2024-06-05 | 0.15.15  |
+---------+------------+----------+

Now it's picking up all the package set entries.

from spago.

Seems like I bumped into this as well. I've just upgraded to spago-next and migrated a project.

First I ran spago migrate, installed spago-next, ran spago test, and then tried to switch to the new Registry package sets (instead of urls), but I don't know how to get a set "number".

➜ spago --version
0.93.30

➜ spago registry package-sets --latest
+---------+------+----------+
| VERSION | DATE | COMPILER |
+---------+------+----------+
+---------+------+----------+

➜ spago registry package-sets
+---------+------+----------+
| VERSION | DATE | COMPILER |
+---------+------+----------+
+---------+------+----------

spago fetch didn't help:

➜ spago fetch
Reading Spago workspace configuration...

✅ Selecting package to build: my-project

Downloading dependencies...
➜ spago registry package-sets --latest
+---------+------+----------+
| VERSION | DATE | COMPILER |
+---------+------+----------+
+---------+------+----------+

After that, in a different random directory

Following @thomashoneyman recommendation, spago init did refresh the index:

➜  test spago init
Initializing a new project...
Refreshing the Registry Index...
Cloning https://github.com/purescript/registry-index.git
Cloning https://github.com/purescript/registry.git
...

➜ spago registry package-sets --latest
+---------+------------+----------+
| VERSION | DATE       | COMPILER |
+---------+------------+----------+
| 10.0.0  | 2023-01-05 | 0.15.4   |
| 20.0.3  | 2023-04-08 | 0.15.7   |
| 27.2.0  | 2023-06-17 | 0.15.8   |
| 29.1.0  | 2023-07-18 | 0.15.9   |
| 43.3.0  | 2023-10-23 | 0.15.10  |
| 46.2.0  | 2023-11-23 | 0.15.12  |
| 47.10.0 | 2023-12-28 | 0.15.13  |
| 50.1.2  | 2024-02-27 | 0.15.14  |
| 51.1.0  | 2024-06-05 | 0.15.15  |
+---------+------------+----------+

from spago.

@Zelenya this is a related issue, but a different bug 🙂
It is the same issue as #1212, and the fix for that (#1214) was too ad-hoc and missed the bigger picture. There is a more proper fix for this in #1228.

This ticket should stay open because I am still of the opinion that the fix will involve taking care of the bit of code that I described in #1221 (comment)

from spago.