rash build result varies from parallelism about racket-rash HOT 15 OPEN

longer diff: http://rb.zq1.de/compare.factory-20190405/rash-compare.out (showing first 200 lines of each file's diff)

from racket-rash.

Thanks! This looks like a problem more generally with Racket's build system, so I've added an issue to the Racket repository as well: racket/racket#2601

from racket-rash.

Or perhaps I'm wrong! I'll have to look through the implementation and see if/where I'm using some non-deterministic hash-table traversal or gensym...

from racket-rash.

I've removed some code that was likely the culprit (gensyms in macros that I was using for some reason). So if this were re-done using the master branch it would perhaps be reproducible now. I can't think of any other macros that could be doing anything non-reproducible.

from racket-rash.

@willghatch usually you can test this by compiling to zo on a few different machines (or with -j options to raco make) and seeing if you get the same contents.

from racket-rash.

@samth Thanks. After trying to think of a more convenient way and failing, I've been doing exactly that.

Which leads me to the bad news that it's still not reproducible. I think I've isolated it to a single file (git-info.rkt). But I haven't figured out what causes it yet. The only macro used there should be deterministic. It uses gensym at runtime to get a singleton value, but that shouldn't affect reproducible compilation (and removing it doesn't fix it). Anyway, I'll look at this some more after I eat some lunch.

from racket-rash.

So my last statement of isolation was way off, that was just an artifact of the way I first started re-kicking builds. At this point it seems that: Rash's main.rkt and its dependencies are reproducible. But any file using main.rkt via #lang rash that also uses begin-for-syntax, define-syntax, or otherwise directly has code at phase 1 is not reproducible (though using pre-defined Rash macros causes no problem). I'm confused as to how this is happening. My best guess is that it has something to do with how my #%module-begin macro is set up. But generally I'm mystified.

from racket-rash.

@samth Thanks for your help on this, by the way. Aside from gensym and hash-table traversals, are there any other features that come to mind? As far as I know, raco setup gets parallelism by using places, not by spawning new processes. If it were spawning new processes, I wouldn't expect gensym or hash-table traversals to cause non-reproducibility, but I'm not really sure how those things work in places where some things are shared. Even with places I find this slightly confusing, since each place has its own garbage collector and everything. But I've never really used places and I'm not sure what is shared besides the ability to send things over channels.

from racket-rash.

Both gensym and hash traversal order are affected by what order things happen in globally (for example, memory addresses or how many gensyms have been created). So lots of things can affect them.

from racket-rash.

Looking at the expansion of a simple rash module, here are a few things that might be an issue.

1. It looks like there's an options hash that gets put into the module -- is the order that those keys appear deterministic?
2. It looks like there are some gensyms in something about current-paramter-environment

I also noticed that a trivial #lang rash program expands to 360+ lines of code. Could some of that be moved into functions where you expand to a reference to that? That makes it less likely that the expansion could change non-deterministically while also making zo sizes smaller.

from racket-rash.

When having non-deterministic hashes in other languages, we usually sort when iterating over them.
https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/hash has 2 examples.

from racket-rash.

from racket-rash.

@samth I really appreciate your help on this. Thanks!

from racket-rash.

Hi, were there patches done that I can test?

from racket-rash.

This is still a problem with
racket-8.11.1
rash-0.2

+++ new//usr/share/racket/pkgs/rash/rash/compiled/experimental_rkt.dep  2022-12-27 00:00:00.000000000 +0000
@@ -1 +1 @@
-("8.11.1" ta6le ("c766a8edc6824eb84469005591ebecfd6a74207e" . "010b0d6788990d0e951970461b12bdfa90d34464") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"escapable-template.rkt") (collects #"rash" #"private" #"lang-funcs.rkt"))
+("8.11.1" ta6le ("c766a8edc6824eb84469005591ebecfd6a74207e" . "0d7d0978458e495d931d91cc1a50fb926f7dd956") (collects #"racket" #"base.rkt") (collects #"racket" #"runtime-config.rkt") (collects #"rash" #"private" #"escapable-template.rkt") (collects #"rash" #"private" #"lang-funcs.rkt"))

from racket-rash.