petvana / autosysimages.jl Goto Github PK

View Code? Open in Web Editor NEW

90.0 90.0 5.0 337 KB

Automate user-specific system images for Julia

License: MIT License

Julia 100.00%

autosysimages.jl's People

Contributors

Stargazers

Watchers

Forkers

dzonimn findmyway ufechner7 curio-sitas touchft

autosysimages.jl's Issues

Version 0.2.5 broken, asks twice to create sysimage

I am asked twice if I want to create a sysimage when leaving the first session after update to new package version. In the end two images are created, but they do not work (are loaded but no decrease in FTTP).

ufechner@TUD277255:~/repos/Plotting$ asysimg --project
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    Default (You may run AutoSysimages.build_sysimage())
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/23ZRuw
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_ABiNGf03c1-snoop.jl
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.1 (2022-09-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

asysimg> @time include("src/plot.jl")
To zoom, draw a rectangle while keeping the right mouse key pressed!
To go back to see the full graph, press <CTRL><f>
 17.227440 seconds (43.38 M allocations: 2.395 GiB, 4.71% gc time, 71.96% compilation time: 12% of which was recompilation)

asysimg> 
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
[ Info: There is no sysimage for this project. Do you want to build one?
 > Yes
   No
[ Info: AutoSysimages: Collecting precompile statements for empty run (-e "")
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    Default (You may run AutoSysimages.build_sysimage())
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/1wTEXm
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/1wTEXm/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_ytapmHunwN-snoop.jl
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/1wTEXm/snoop-file.jl
[ Info: There is no sysimage for this project. Do you want to build one?
 > Yes
   No
[ Info: AutoSysimages: Collecting precompile statements for empty run (-e "")
[ Info: AutoSysimages: Building system image by PackageCompiler.
⠸ [00m:09s] PackageCompiler: compiling incremental system image┌ Warning: There is no statement file
└ @ Main.anonymous /home/ufechner/.julia/packages/AutoSysimages/9SsCR/src/precompile-PackageCompiler.jl:44
⢰ [00m:09s] PackageCompiler: compiling incremental system image
✔ [02m:24s] PackageCompiler: compiling incremental system image
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
[ Info: AutoSysimages: Builded in 146.55096817016602 s
[ Info: AutoSysimages: Building system image by PackageCompiler.
⠇ [00m:08s] PackageCompiler: compiling incremental system image┌ Warning: There is no statement file
└ @ Main.anonymous /home/ufechner/.julia/packages/AutoSysimages/pf4ZI/src/precompile-PackageCompiler.jl:44

✔ [02m:11s] PackageCompiler: compiling incremental system image
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/ufechner/packages/julias/julia-1.8/lib/julia/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
[ Info: AutoSysimages: Builded in 295.65954995155334 s
ufechner@TUD277255:~/repos/Plotting$

Second time starting asysimg, no speed improvement:

ufechner@TUD277255:~/repos/Plotting$ asysimg --project
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/asysimg-2022-09-22T08-19-05.177.so
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/23ZRuw
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_fqH2RYj43C-snoop.jl
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.1 (2022-09-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

asysimg> @time include("src/plot.jl")
To zoom, draw a rectangle while keeping the right mouse key pressed!
To go back to see the full graph, press <CTRL><f>
 17.210109 seconds (43.38 M allocations: 2.396 GiB, 4.81% gc time, 71.98% compilation time: 13% of which was recompilation)

asysimg>

ufechner@TUD277255:~/repos/Plotting$ cat Project.toml 
[deps]
AutoSysimages = "4eb35182-e5c8-446d-8ecc-33e714a26cef"
Gtk = "4c0ca9eb-093a-5379-98c5-f87ac0bbbf44"
InspectDR = "d0351b0e-4b05-5898-87b3-e2a8edfddd1d"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"

[compat]
Gtk = "~1.2"
julia = "1.7, 1.8"

Add function install()

Add a function that copies the required script at the suggested location.

Should be trivial for Linux.

Provide information which packages are loaded from sysimage

See #28

Startup and shutdown overhead

By default we have quite a high startup and shutdown time:

ufechner@ubuntu:~/repos/LogFiles$ time asysimg --project -t auto -e ""
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/asysimg-2022-09-20T10-49-17.212.so
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_9uDsjwxerC-snoop.jl
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/snoop-file.jl

real	0m3.351s
user	0m3.309s
sys	0m1.459s

compared to:

ufechner@ubuntu:~/repos/LogFiles$ time julia --project -t auto -e ""

real	0m0.175s
user	0m0.105s
sys	0m0.105s

Support `install()` for Windows

The install() function is implemented only for Linux now. It would be nice to provide the same functionality for other OS as well.

AutoSysimages.jl/src/AutoSysimages.jl

Lines 388 to 389 in a28a424

    
                   @warn """AutoSysimages: Installation is not yet supported for your OS. 
        
           Feel free to submit a PR."""

Enable building sysimages in background

The idea is to build syimages while you can continue working in the same REPL. The only challenge is how to output information from the compilation process into the REPL asynchronously, see JuliaLang/julia#13356.

I've already tested some ideas, and this is the proof-of-concept I have:

using REPL
p(x) = (print("\u1b[s\u1b[1A\r$x\u1b[u"), REPL.LineEdit.refresh_multi_line(Base.active_repl.mistate))
@async begin for i=1:100; sleep(0.1); p("ID $i"); end end

However, I cannot record the output using asciinema because the result differs.

Unable to save/load sysimage

@ufechner7 reported this issue at forum:

To be honest, I do not see ANY effect from using AutoSysimages.

What I did (on Ubuntu):

install AutoSysimages
change folder of my work project
run asysimg --project
run include(“src/plot.jl”) from the asysimage prompt
close the plot window
press <ctrl>+<d>
I get the message:

asysimg>
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/23RTv7/snoop-file.jl
[ Info: There is no sysimage for this project. Do you want to build one?

Yes
No

I press enter and select No for the next question (do you want to select packages).

Now it builds a system image.

[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
[ Info: There is no sysimage for this project. Do you want to build one?

Yes
No
[ Info: AutoSysimages: Collecting precompile statements for empty run (-e "")
[ Info: AutoSysimages: Building system image by PackageCompiler.
⠇ [00m:10s] PackageCompiler: compiling incremental system image┌ Warning: There is no statement file
└ @ Main.anonymous /home/ufechner/.julia/packages/AutoSysimages/LTaGS/src/precompile-PackageCompiler.jl:44

✔ [02m:25s] PackageCompiler: compiling incremental system image
[ Info: AutoSysimages: Builded in 147.87036108970642 s

If I now launch julia again and run the same include statement the time to first plot is 17s as before:

ufechner@TUD277255:~/repos/Plotting$ asysimg --project
┌ Info: The package AutoSysimages.jl started!
│ Loaded sysimage: Default (You may run AutoSysimages.build_sysimage())
│ Active directory: /home/ufechner/.julia/asysimg/1.8.1/23ZRuw
│ Global snoop file: /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
└ Tmp. snoop file: /tmp/jl_gJmh8ds5yB-snoop.jl
_
_ _ ()_ | Documentation: https://docs.julialang.org
() | () () |
_ _ | | __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ ` | |
| | || | | | (| | | Version 1.8.1 (2022-09-06)
/ |_'|||_'_| | Official https://julialang.org/ release
|__/ |

asysimg> @time include("src/plot.jl")
To zoom, draw a rectangle while keeping the right mouse key pressed!
To go back to see the full graph, press
17.618484 seconds (43.41 M allocations: 2.397 GiB, 4.78% gc time, 71.93% compilation time: 12% of which was recompilation)

You can find and clone the example project I used here: GitHub - ufechner7/Plotting

My impression is that it creates a sysimage, but it does not use it later.

Any idea?

`cleanup` function in the core code

There is a clean.sh file in the repo. As the package creates many files in projects folder and julia bin directory, it would be nice to have a cleanup routine in the code as a functionality of the package.

Example :

julia> using AutoSysimages
julia> AutoSysimages.cleanup()

Should remove all the files and directory created by the package, including the asysimg script

Make it compatible with OhMyREPL

Currently, loading OhMyREPL.jl has no effect because we set magenta prompt asysimg>

AutoSysimages.jl/src/AutoSysimages.jl

Lines 466 to 474 in 47128ab

    
           # TODO - make it compatible with OhMyREPL 
        
           function _update_prompt(isbuilding::Bool = false) 
        
               function _update_prompt(repl::AbstractREPL) 
        
                   mode = repl.interface.modes[1] 
        
                   mode.prompt = "asysimg> " 
        
                   if Base.have_color 
        
                       mode.prompt_prefix = Base.text_colors[isbuilding ? :red : :magenta] 
        
                   end 
        
               end

Script for Windows not working

trying to follow this: https://petvana.github.io/AutoSysimages.jl/stable/install/#Script-for-Windows and getting the below error

C:\Users\aryan\Documents\AutoSysimages.jl\scripts>cd windows

C:\Users\aryan\Documents\AutoSysimages.jl\scripts\windows>./asysimg.bat
'.' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\aryan\Documents\AutoSysimages.jl\scripts\windows>asysimg.bat
'"C:\Users\aryan\Documents\AutoSysimages.jl\scripts\windows\\julia.exe"' is not recognized as an internal or external command,
operable program or batch file.

Doesn't work on second use

It worked on first use (or mostly): JuliaPy/PythonCall.jl#240

But then:

$ PATH=/home/pharaldsson/.local/bin/:$PATH asysimg

asysimg> @time using CairoMakie
 14.293240 seconds (20.02 M allocations: 1.383 GiB, 4.58% gc time, 3.49% compilation time)

asysimg> exit()
[ Info: AutoSysimages: Copy snooped statements to: /home/pharaldsson/.julia/asysimg/1.9.0-DEV.1694/30JkAN/snoop-file.jl

I think it should run PackageCompiler again(?).

Scripts missing?

Hey @petvana,

I love your project, directly tested it somtime last month, when I saw your package annoucement. However, when I now install the package
] add AutoSysimages or directly the github repository
and I use
Autosysimages.start() followed by some commands and then Autosysimages.build_sysimages() it throws the following error.

ERROR: IOError: could not spawn `asysimg -J /opt/julia-1.8.2/lib/julia/sys.so -e '' --trace-compile /tmp/jl_RGLarUfsnM`: no such file or directory (ENOENT)
Stacktrace:
  [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:128
  [2] #725
    @ ./process.jl:139 [inlined]
  [3] setup_stdios(f::Base.var"#725#726"{Cmd}, stdios::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:223
  [4] _spawn
    @ ./process.jl:138 [inlined]
  [5] run(::Cmd; wait::Bool)
    @ Base ./process.jl:479
  [6] run
    @ ./process.jl:477 [inlined]
  [7] _build_system_image()
    @ AutoSysimages ~/.julia/packages/AutoSysimages/CtLWM/src/AutoSysimages.jl:573
  [8] build_sysimage(background::Bool)
    @ AutoSysimages ~/.julia/packages/AutoSysimages/CtLWM/src/AutoSysimages.jl:196
  [9] build_sysimage()
    @ AutoSysimages ~/.julia/packages/AutoSysimages/CtLWM/src/AutoSysimages.jl:166
 [10] top-level scope
    @ REPL[4]:1

The same happens when I exit the REPL (^D) and the interactive promt appears.
It seems the asysimg script is just missing? I could also not see it in the repository, on older commits it is still there.
Am I missing something?

Warning: Function 'pkgversion' is needed to check versions.

When launching asysimg I get the following warning:

ufechner@TUD277255:~/repos/Plotting$ asysimg 
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    /home/ufechner/.julia/asysimg/1.8.1/1wTEXm/asysimg-2022-09-17T12-51-08.314.so
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/1wTEXm
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/1wTEXm/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_fi3s5sV0Nu-snoop.jl
┌ Warning: Function 'pkgversion' is needed to check versions.
└ @ AutoSysimages ~/.julia/packages/AutoSysimages/LTaGS/src/AutoSysimages.jl:375
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.1 (2022-09-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

asysimg>

Project is not propagagted in asysimg.bat script for Windows

This is the same problem as reported in #5 for Linux.

The solution is to propagate arguments, like in 30db564

	@warn """AutoSysimages: Installation is not yet supported for your OS.
	Feel free to submit a PR."""

	# TODO - make it compatible with OhMyREPL
	function _update_prompt(isbuilding::Bool = false)
	function _update_prompt(repl::AbstractREPL)
	mode = repl.interface.modes[1]
	mode.prompt = "asysimg> "
	if Base.have_color
	mode.prompt_prefix = Base.text_colors[isbuilding ? :red : :magenta]
	end
	end