Comments (4)
Here's an idea that might not make sense:
We might need something similar for a "simple rsh" implementation to handle stdout/err.
Imagine if the flux rsh RANKS COMMAND...
frontend worked something like:
- generate a unique ID for the current run
- subscribe to "log" stream for ID
- send rsh.execute or similar command with json decsription of command+environment
- stderr/out "log" messages would be copied back to stderr/out of
flux rsh
command -- other log messages could be optionally displayed based on --verbose. Collapsed lines could optionally be expanded byflux rsh
- exit code(s) could come back as CMB replies, or perhaps specially formatted log messages
Does this make any sense? Maybe it doesn't make sense to derive the flux rsh
protocol from the
log implementation, but instead think of a lower abstraction from which both rsh
and log
services
are derived?
from flux-core.
Could we just use the existing logging interface on the rshd end, e.g.
flux_log_set_facility (h, "rsh-%d", rsh_jobid);
flux_log (h, LOG_INFO, "%s", stdout_line)
flux_log (h, LOG_ERR, "%s", stderr_line)
Then we would just need a way for the rsh end to subscribe to messages sent to that facility. Are we OK with presuming that stdio will be consumed on rank 0? If so maybe part of the log design could be an ipc:// socket that all logs are published to, with PUB-SUB topic string derived from the facility. Then rsh could connect to the socket and subscribe to its particular rsh_jobid.
The flux-snoop
utility works with a "snoop socket" in pretty much this way now.
from flux-core.
With the "reduction handle" improvements in pr #298, I was thinking perhaps this issue should be revisited. Since TIMEDWAIT is the obvious "flush policy" for compressing identical log messages, and flux_reduce_t requires the flux reactor for installing internal timer watchers, the fact that the broker still uses zloop is an impediment.
I've opened #320 to remind us to get off zloop in the broker.
from flux-core.
#320 is no longer a blocker, but this feels to me a bit like premature optimization and furthermore, is a fairly obvious possibility so I don't think needs an issue to remind us. Closing.
from flux-core.
Related Issues (20)
- startlog: need to initialize flag enum to a number HOT 1
- content-sqlite: preallocate database space HOT 26
- some JSON implementations may not be able to handle the full range of job IDs encoded as JSON numbers HOT 2
- shell: increase `output.batch-timeout` default for very large jobs HOT 2
- housekeeping drain message is not informative after timeout HOT 1
- python: need easy way to wait until a subinstance is ready to accept jobs HOT 1
- housekeeping doesn't release resource for a large job HOT 11
- content: flush data if backing store is loaded at a later time HOT 8
- t2406-job-exec-cleanup Racy sub-test HOT 1
- idea: vacuum sqlite on ENOSPC errors during teardown HOT 2
- flux overlay errors does not display version mismatch information
- Suggestion: ability to specify a node range in flux jobs HOT 1
- satisfiability bug: specifying a queue and a constraint matching another queue hangs HOT 1
- throughput.py test: incorrect number of tests submitted in status HOT 10
- job is terminated with `scheduler-restart` exception on Fluxion module reload
- not ok 17 - attach: reports job shell Killed if job shell is killed
- Interested in elastic deployments on Slurm HOT 7
- job-manager: job update feasibility check topic string should be configurable HOT 2
- `flux perilog-run` is not scalable
- job-manager: jobs with alloc-bypass flag appear to still generate a free request to scheduler HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flux-core.