Comments (7)
@fuzzycow Sorry for the late response; I have been trying to juggle a lot of different tasks over the last few weeks, so I am not as fast to respond as I would like to be.
At this time, the internal handling of file handles is left up to the implementation, and not specified in the spec, as it can be language-dependent.
As you said, the tradeoff is generally being safe and slow(er) vs possibly unsafe and fast(er). But the way I see it, there is not always a clear benefit to doing it either way:
- In some languages (such as JavaScript and Node.JS), the user is giving up the execution speed in exchange for a less work-intensive development cycle. Just starting the Node interpreter can take a second on the EV3, and I'm not even sure that you can maintain a file handle in the same way that one would in a language like C++.
- Keeping file handles means that you need to use slightly more memory, and have to check to see if you have a cached copy of the handle before reading from it. You also have to seek to the beginning of the file each time. These effects may be negligible, but it really depends on the language and implementation. I have found that even small things like this can make a huge difference on the EV3's processor.
- If we have a file handle open to a device's property file and that device is unplugged, how would we handle that? We could check every time that we read from a file to make sure that it's still valid, or maybe catch the errors when they are thrown. But that leads to more checks that we have to do.
So in the end, I see your point, but I don't think that we would be able to draw a clear-cut conclusion that every binding must maintain the open file handle. Here's my proposal: we put a note in our spec that asks the maintainers to be cognisant of the tradeoffs between the two options, and advises that they experiment with both to decide how they implement it. Does that sound good?
@fdetro You are more knowledgeable on the subject of performance maximization than I am; jump in if you have anything to add.
from ev3dev-lang.
@WasabiFan I agree with your comments regarding CPU vs. resource / memory usage optimization. As one normally does not read sensor / motor values / attributes continuously from different threads and typically also with a quite low frequency (some 10 Hz?), the small overhead of re-opening the file every time shouldn't really hurt.
On the other side with a larger number of sensors and motors connected one can end up with quite a lot of open file handles in the case of caching, which will use memory and OS resources even if used only once.
So I would vote for not caching the handles in the standard case. If someone has a use case where read performance really matters, he can easily optimize this case with a custom implementation.
from ev3dev-lang.
Thanks for responding!
I agree with your comments - its often best to default to pragmatic approach, and to advise developers on the potential impact and ways for further optimization.
I would still like to share some thoughts om the issue - for those who may want to do alot of sysfs file io on ev3:
(EDIT - correct performance numbers for python)
In my simple tests single-process/single-thread test (python) managed to process on the order of ~600 open/read/close per second.
If we consider a project reading ~5 sensor values, and ~10 motor values (e.g.: datalogging/charting) - you end up with ~15 reads per iteration.
At 10Hz this translates into ~1/4 of max available cpu time being spent on open/close.
Golang-specific notes:
Single coroutine (green/light thread) performance in Golang was actually lower then in python.
I did some basic profiling on the test application, and it seems that a single system call in golang arm file open code (this golang bit is written asm) is "responsible".
Golang concurrent coroutines test without cached filehandles suggests that while one coroutine is doing open another coroutine can not progress. I'm not sure if only I/O operation are blocked process-wide, or the whole process is paused.
x10 performance improvement when using concurrent coroutines and cached filehandles seems to indicate that golang arm concurrency plays well with read/write I/O but not with open/close.
Regarding performance impact:
CPU impact - as benchmarks show CPU impact is lessened when using cached filehandles. Furthermore cached filehandles lower the amount of garbage collection in languages like python and golang.
Memory impact - I couldn't find exact numbers, but I think filehandle objects are small enough and given the choice of "keeping" objects around and gc - the first approach is often better for performance.
I will post my test code snippets and benchmark results as soon as I get a bit of time.
Lastly - I would like to reemphasize that I'm not suggesting that existing code be rewritten to suit my particular needs or opinions. Information is provided as food for thought, with hope that it will assist others implementing heavy sysfs file io.
from ev3dev-lang.
Benchmarks:
Python reopen before each read (with open(filename,"rb") as f) - ~600 read/per second
Python cached filehandle - 1860 to 2300
Golang (reopen file, 5 reader coroutine, 1 consumer coroutine) - ~520
Golang (cached filehandles, 5 reader coroutine, 1 consumer coroutine) - ~7000
from ev3dev-lang.
Looks like this was addressed in #25.
from ev3dev-lang.
That PR only helps with this for C++, the other languages will need their own solution.
from ev3dev-lang.
Python binding is based on C++ one, so that leaves js, lua, and R.
from ev3dev-lang.
Related Issues (20)
- Updating human-readable spec HOT 25
- How to create spec.json HOT 2
- How strictly to follow the language binding specification? HOT 6
- How to implement Button Class HOT 1
- How to use with the autogen system? HOT 26
- Unified language for Java - Library Submission HOT 5
- Having issue to connect Pycharm to the Brick HOT 3
- Standardize "systemDeviceNameConvention"
- Add binding for IR-SEEK to spec HOT 1
- Javascript motor and sensor example HOT 3
- A question about sensor inherits HOT 5
- 0.9.3? HOT 1
- Merge NXT Motor and Large Motor classes
- Some typo on the driver docs HOT 3
- stop lcd going back to menu HOT 2
- Dual motors HOT 2
- UNMET PEER DEPENDENCY bluebird@~3.5.0 HOT 7
- Buttons Error (platform-gpio-keys.0-event not found) HOT 3
- HiTechnic Compass Sensor "Device is not connected" HOT 3
- Status of the project - current ev3dev bindings HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ev3dev-lang.