Comments (7)
Please consider adding submodule update flag similar to --fetch-opt.
Which submodule options have you already tried to pass to --fetch-opt=
?
The only thing west
does is invoking git
with various options.
from west.
I have not tried to pass any submodule-related options to fetch-opt as I didn't find any useful flags in the git fetch documentation (I might have missed something!).
So, I tried using --depth=1 as fetch-opts, but with this flag, submodules are updated with the full history.
from west.
west
is mostly just a wrapper on top of git and it will rarely ever go beyond what git
itself can do - especially not for a git feature that duplicates what west
does!
So if you can't find a way to optimize submodules with git
alone, then we'll have to close this issue as "wontfix".
If you're really desperate for cloning performance (e.g.: in CI) then you can have some git repos registered as BOTH a git submodule AND in a west manifest. Then you can use west --fetch-opt=...
to initialize git submodules
faster "behind their back". This duplication has some minor inconveniences but it's routine and very well tested in at least this project:
thesofproject/sof@8fd351ea9aa6c3117
To keep such an "submodule override" optimization private you can use a submanifests directory:
https://docs.zephyrproject.org/latest/develop/west/manifest.html#manifest-imports
Working example here: thesofproject/sof@fca84b8773ee4a8e6308a0
For more performance discussions take a look at this label: https://github.com/zephyrproject-rtos/west/issues?q=+label%3Aperformance+
What would probably make the most performance difference is (as often) parallelization:
from west.
Git itself supports additional flags for submodule update (e.g. depth and reference flags mentioned here: https://git-scm.com/docs/git-submodule).
However, inside west, submodule update seems quite fixed. I mean, there isn't an equivalent for e.g. fetch-opts around here: https://github.com/zephyrproject-rtos/west/blob/main/src/west/app/project.py#L1168
Would it make sense to have e.g. submodule-update-opts flag here?
You are correct that this is in CI. We already use west update --path-cache and it works great. However path-cache won't utilize cache for submodules. Maybe path-cache could be extended to support submodules?
Having repos both as git submodules and in the west manifest does not sound very elegant solution. It probably is ok for couple of repos which rarely move but what if submodules look something like this: https://github.com/project-chip/connectedhomeip/blob/master/.gitmodules
from west.
Would it make sense to have e.g. submodule-update-opts flag here?
Maybe. Or simpler: pass --fetch-opt
arguments to git submodule update
too. Could you take a stab at it?
Maybe path-cache could be extended to support submodules?
I don't know that part of the code but it sounds like it would be more complicated.
Having repos both as git submodules and in the west manifest does not sound very elegant solution.
Agreed, I wrote "If you're really desperate for cloning performance". Then this can give CI a boost TODAY without other users noticing (so it's not that ugly).
It probably is ok for couple of repos which rarely move
Agreed, that's what I had in mind. Not that:
but what if submodules look something like this: https://github.com/project-chip/connectedhomeip/blob/master/.gitmodules
If submodules look like this, you have a bigger problem than just performance. Managing multiple git repos is complex and bordering on "package management"
zephyrproject-rtos/zephyr#54276 (comment)
Managing a large number of git repos using TWO different solutions at the same time (west + submodules) seems very problematic long before even looking at performance. You should explore alternative ways not to do that; such a complex combination of project manifests seems like a recipe for various disasters.
Circling back: like it or not west
is "competing" with git submodules with extremely limited development resources. So it's not going to bend over backwards to help manage git submodules. If simple git submodule optimizations can be achieved then great! But if other git submodule work requires a lot of development effort and longer term maintenance then I would strongly advise against it.
@mbolivar-ampere without looking at the technical details can you please confirm, correct or nuance the last, non-technical point?
from west.
Maybe. Or simpler: pass --fetch-opt arguments to git submodule update too. Could you take a stab at it?
I'm slightly worried that this could cause backward compatibility issues in case someone is using the current fetch-opt for purposes other than, for example, depth.
Anyway I conducted some tests locally with --depth 1 for one of the projects that has submodules and compared cloned sizes. The gains with --depth 1 were not substantial IMO (only ~10% smaller size on disk), but this obviously depends on the repos. west update time was fluctuating much more than 10% for me so I would need a lot of samples to see how much time could be saved.
I also tried extending path-cache support for submodules using --reference flag, and it appears more promising to me, at least from a performance standpoint.
from west.
The gains with --depth 1 were not substantial IMO (only ~10% smaller size on disk), but this obviously depends on the repos.
--depth 1
saves time and disk only for repos with large histories. In smaller repos, the entire history takes less space than one checkout (because it's compressed)
Try BOTH --depth 1 and --filter=tree:0
from west.
Related Issues (20)
- [RFC] New `inactive-by-default:` project key HOT 2
- Support `west manifest --resolve` and `--freeze` with `manifest.project-filter` HOT 1
- Follow up work for `manifest.project-filter` implementation
- west init: CLI argument to automatically setup new workspace from template HOT 11
- `already defined as extension command`-error prints wrong spec HOT 1
- Introduce absolute path variant of zephyr.base HOT 13
- west cannot process a git branch name containing a single quote ' HOT 8
- west init access denied on windows HOT 2
- git: Add support for sparse checkout HOT 5
- west update loops infinitely over the first repository HOT 14
- West Re-Implementation - git ws
- west update - AttributeError: 'NoneType' object has no attribute 'err' HOT 1
- menuconfig aborting due to Kconfig warnings HOT 4
- Allow using reference repositories to share objects HOT 8
- Allow import of optional projects from manifest file HOT 7
- Moving from Zephyr 3.2.99 to 3.5.99: `west` cannot find its configuration file HOT 3
- --mr documents argument to be a revision HOT 1
- `pip install` fails on MSYS2 HOT 3
- Infinite loop when building in a moved directory HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from west.