Support Fortran submodules about fpm HOT 15 CLOSED

I think I am against removing the restriction by default. I very strongly believe the default should be setup in a way to guarantee that there cannot be any name collisions.

This restriction is there to prevent name collisions. What we can do is remove the restrictions if the user requests that in fpm.toml --- with the understanding that fpm cannot guarantee there won't be name collisions with user code any more; this option would be used to get existing packages working quickly, but they will not be "conforming", and thus there can be collisions.

from fpm.

Thanks. Yes, fpm must work with submodules and yes, we must be able to build in parallel.

from fpm.

fpm does support parallel builds right now. All modules are compiled in parallel (to the extent they can be of course, given dependencies on the .mod files). Supporting submodules won't be too difficult, but I haven't found a particularly compelling use case.

from fpm.

I've gotten a request from a paying customer to add support for submodules ASAP. The question I have for the rest of the group has to do with the naming convention restrictions we currently have in place for modules.

The dependency detection and build system has to get a lot more rigorous and complex, so I'm thinking of just removing the name-spacing and file name matching requirement we had for modules, since it doesn't quite apply properly to submodules. And while I like it as a convention, it was done partly to avoid doing the harder work of properly looking in the source files.

Do you guys have any thoughts? It won't add any backwards incompatibilities, and it won't add any complexities from a user perspective.

from fpm.

The restrictions currently in place do not actually guarantee there cannot be any name collisions.

That's true but this is not so much about the current guarantees, but about the design we're aiming for.

Do we want to prevent name clashes? (I think it's unanimous yes)

If yes, then what is the optimal solution to do it? Prefixing with the package name is the best I can think of.

The only exception should be single-module libraries, which could have a module with the same name as the package.

I think this would be good to discuss on our call on Thursday.

from fpm.

fpm does support parallel builds right now. All modules are compiled in parallel

This is awesome! I didn't realise. Does it use all available threads by default?

from fpm.

Yep

from fpm.

The dependency detection and build system has to get a lot more rigorous and complex, so I'm thinking of just removing the name-spacing and file name matching requirement we had for modules, since it doesn't quite apply properly to submodules. And while I like it as a convention, it was done partly to avoid doing the harder work of properly looking in the source files.

I have no objection to removing these naming requirements - I actually found it very restrictive when trying out fpm with existing packages which have their own various naming conventions.

I'm excited for submodule support! 🚀

from fpm.

This sentence confuses me:

I'm thinking of just removing the name-spacing and file name matching requirement we had for modules, since it doesn't quite apply properly to submodules.

Do you mean keep the existing requirement for modules, but don't require it for submodules? I think so, because later you wrote "It won't add any backwards incompatibilities".

I don't understand well how submodules are managed by compilers. If there's no possibility of name clash between submodules from different packages, then I don't see the need to enforce the same convention we have for modules. So, this sounds good to me, but let's make sure that that's the case.

from fpm.

I mean remove the existing requirement for modules. Removing that requirement is backwards compatible.

Submodules are a bit convoluted.

Compiling a module produces a .mod and .smod with the name of the module. Compiling a submodule depends on the .mod and .smod of its parent module, and the .smod of any parent submodules, and produces a .smod file named parent_mod@parent_smod1@parent_smod2@[email protected]. Then all of the object files from the whole tree must be linked into the library or executable.

from fpm.

If there's no possibility of name clash between submodules from different packages, then I don't see the need to enforce the same convention we have for modules.

I hadn't realised this convention was for avoiding name clashes but I guess that makes sense - how would name clashes be avoided without this requirement @everythingfunctional ?

Submodules are a bit convoluted.

I remember reading somewhere that submodule file naming conventions vary between compiler vendors - can these dependencies not be tracked without relying on .mod and .smod files?

Within my makefiles I never reference *mod files and simply rely on the corresponding object file for updating dependencies; in this way submodule dependencies can be treated like a normal use dependency where the submodule 'uses' its parent module.

from fpm.

I'd still recommend people follow that naming convention, it just won't be a hard requirement. So you can still have a module named utilities_stats in a file src/utilities/stats.f90 per the current requirement and it will still work just fine. With the change, the module could just be named stats, but if any other package has a module named stats, they'll conflict of course. The possibility of conflict already exists, and I think always will unless we make a breaking change and require all modules in a library be prefixed with the name of the library.

I don't recall exactly where I read that naming convention. It may have just been the gfortran documentation, in which case this may be a bit trickier when we try and support multiple compilers, but hopefully not a big deal.

While you can write your makefiles without mentioning the .mod files, it's not strictly correct. Compiling a source file doesn't actually depend on the object files, just the .mod files. And it sidesteps a potential optimization I wish the compilers would actually implement; don't update the .mod file if none of the interfaces changed. That would actually take care of the "recompilation cascades" that most people give as the motivation for using submodules.

from fpm.

The restrictions currently in place do not actually guarantee there cannot be any name collisions. If you put a module named utils in the root src folder, you're almost guaranteed to have a name collision with some other package and there isn't anything in place preventing that at the moment.

The restriction is really only preventing accidental name collisions within a library, and so far the errors have caused more confusion than they have helped anything.

I can add a check for it back in once this is implemented, but at that point it's just enforcing a convention (albeit a good one), more than any technical requirement, and still not actually guaranteeing no name collisions between packages.

from fpm.

While you can write your makefiles without mentioning the .mod files, it's not strictly correct. Compiling a source file doesn't actually depend on the object files, just the .mod files.

Good point, but I disagree with your assertion of incorrectness; depending on module object files will not result in an incorrect dependency between source files, rather it (theoretically) results in redundant recompilation as you point out. Therefore object file dependencies are correct for modules, but not optimal.

I am genuinely intrigued as to why your proposed optimisation is not implemented anywhere, it seems like a simple fix to a big problem!

That would actually take care of the "recompilation cascades" that most people give as the motivation for using submodules.

My own motivation for submodules is mainly to flatten the dependency tree to just two layers so that each layer can be compiled completely in parallel. Being able to compile in parallel quickly outweighs any redundancy in my dependencie.

I think I am against removing the restriction by default. I very strongly believe the default should be setup in a way to guarantee that there cannot be any name collisions.

My preference would be for an alternative solution to avoiding name collisions, perhaps we should discuss in a dedicated thread?

from fpm.

My preference would be for an alternative solution to avoiding name collisions, perhaps we should discuss in a dedicated thread?

Yes, I'll start a new issue.

from fpm.