Comments (6)
@TheBay0r, when you're testing this, are you modifying the file in any way between uploads? (For example, are you reuploading the exact same file without any changes at all, or are you changing it in some way before reuploading?)
from nexus-repository-composer.
@fjmilens3 I wasn't changing the files in between. Will try to test the case where the content of the zip is slightly changed if that has an impact on the JSON generated.
from nexus-repository-composer.
@fjmilens3 So I tested this case. When the zip file contains a change it seems that the json is rebuilt
from nexus-repository-composer.
So I tested this case. When the zip file contains a change it seems that the json is rebuilt
This is something we're going to have to live with because of other factors related to suppressing unneeded rebuilds, but fortunately shouldn't be that much of a problem.
Explanation
Every time a new upload comes in for an existing artifact, we generate an update event within the data layer:
We then receive this event and if certain conditions hold, we use that to generate a rebuild event for the metadata:
However, since these are database-level events, other events, notably downloads, can also cause the same record to be updated (as we have to increment the last downloaded time, etc.).
We don't want to rebuild metadata in this event, and until we have a better application-level solution, our preferred workaround for this problem is to see whether or not the blob was updated within a short period of time.
If it has, we assume that it was the result of the blob changing, and if has not, we infer that it was the result of a download (or other operation) that touched the asset but should not force a rebuild of any associated metadata:
Along with the above, Nexus Repository Manager tries to deduplicate blobs (for the same asset), such that if we receive a blob for an asset that's identical to the blob we already have for that asset, we don't have any churn at the storage level. As a practical matter, that means that the blob was not updated, so the blob updated timestamp is not updated either:
The end result being that you won't have any metadata rebuilt because the blob has not changed. Of course, there are changes within the repository manager itself that could be used to handle this, or we could broadcast custom events from within the content facet; however, I'm trying to minimize the divergence between the approach we have here and the approach we have in our supported/proprietary format implementations.
Conclusions
Under normal circumstances this won't matter as if the blob hasn't changed then the generated metadata would not change (at least not in any meaningful way) either, as the metadata is extracted from the content of the blob (in our case, the composer.json
file in the archive).
However, under unusual circumstances it can be advantageous to have a scheduled task within Nexus that can rebuild metadata for all or part of a repository's contents. This is typically useful either to mitigate the effects of some breaking change or to recover from some unexpected situation where the generated metadata is inaccurate or incomplete (a special case of which would be the scenario you first encountered, where I'd made breaking changes with the metadata generation in that PR, but you didn't see the metadata regenerate by reuploading the same artifact).
If you have no objections (and are satisfied with the above explanation), I would like to consider this "closed" and implement the aforementioned scheduled task in #21 before we someday promote this to 1.0.0
.
from nexus-repository-composer.
Ah, wow! Thank you for the detailed explanation. My naive approach to this was that it just would be hooked up to the post request and whenever a post request comes in an update is triggered no matter what 🤔
But this approach makes sense of course! 🙂
From my point of view this one can be closed, thanks.
from nexus-repository-composer.
Closing based on conversation with @TheBay0r.
from nexus-repository-composer.
Related Issues (20)
- E401 errors when publishing with scoped :_auth
- Does nexus supports arm64
- Packages do not appear in any search HOT 6
- Nexus and composer integration - failing authentication step
- composer install --prefer-source not working HOT 1
- composer redirect issue
- Composer repository no longer updates composer2 JSON files for new versions in case if redeploys are not allowed HOT 1
- Cleanup policy for composer repositories does not find any packages to remove HOT 1
- Index rebuild won't finish successfully HOT 6
- Plugin not installing on Nexus 3.51 HOT 3
- Nexus fails starting with java.lang.IllegalStateException: Missing recipe: composer-proxy after upgrading to v3.53 (Docker based) HOT 7
- An error occurred loading data. Request failed with status code 500
- package meta data - version_normalized
- Repository Cleanup , S3 storage
- External repo returns 400: Invalid Request resulting in caching 0 byte artifacts HOT 2
- Exception (com.google.inject.CreationException) while activating nexus-repository-composer for Nexus v3.62.0-01 HOT 1
- Cleanup Policy Preview is not working HOT 1
- Clarification in Composer proxy Repository
- Problem in adding the extension to nexus version 3.27 HOT 1
- Composer in proxy mode does not working
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nexus-repository-composer.