Comments (11)
In the years since this discussion, the Go garbage collector has gotten so good that just calling runtime.GC() is cheap and quite viable. As time goes by it should become practically as cheap as a manual deallocation.
from imaging.
Currently fixed by running runtime.GC()
at the end of every request - that sounds very inefficient, though.
from imaging.
@sudhirj
Hi,
There is a flag but it's not public currently.
Can you please clone the repository. Edit file parallel.go:
https://github.com/disintegration/imaging/blob/master/parallel.go#L9
Set parallelizationEnabled to false, to see if it solves the problem.
If so I'll try to find what causes the memory leak. I use sync.WaitGroup to ensure that all the goroutines finished. Maybe there is a bug in my code.
Maybe I'll make parallelizationEnabled option public.
from imaging.
Found it and tried that a while ago, but it has no effect on App Engine. I did some basic testing on my dev machine (ran on a continuous load), and I don't think there's actually a memory leak, per se.
I think the problem is more that on GAE, an application is terminated as soon as it starts using about 120MB of memory. This problem also goes away when forcing the GC after every request. It might just be that the algorithm (I'm using Lanczos) uses a lot of memory. It may not leak memory, but it might just be using a lot. Trying to optimize for memory might help, but I don't know if that's a priority.
from imaging.
I think resampling filter doesn't affect a lot, but I'll test it.
Image size is important: 10 megapixel image is 40MB of memory (32bits per pixel). Two-pass resize needs two additional allocations. First - the temp image with changed width. Second - the resulting image.
I'm not sure how Go decides when to run the garbage collection. Maybe forcing runtime.GC is the only way for your application.
Anyway, I'll try to find ways to reduce memory usage by Resize function.
from imaging.
So just for my reference (I'm not familiar with image resizing algorithms), holding and resizing a 10MB image with a two pass filter will require 30MB of RAM altogether? That would definitely explain the App Engine problem. It simply wouldn't have run the GC in time.
One possible way I can see would be to forcibly deallocate the intermediate objects once the operation is finished. I don't know if Go has the equivalent of destroy()
or free()
, though.
from imaging.
So just for my reference (I'm not familiar with image resizing algorithms), holding and resizing a 10MB image with a two pass filter will require 30MB of RAM altogether?
Not quite right. For example: if you want to resize 3000x2000 image to 150x100. In first pass 150x2000 pixels will be allocated. In second pass 150x100 pixels will be allocated (image that is returned from function). Temporary 150x2000 will be freed as soon as the gc collects them.
from imaging.
Ah, right. Got it.
Temporary 150x2000 will be freed as soon as the gc collects them.
Forcing this collection manually, before returning, instead of waiting for GC will automatically solve half the memory problem. I'm trying to see if there a way to do that.
from imaging.
There are no such things as destroy or free in Go. I think runtime.GC() is the only way.
from imaging.
Just for future users who may find this useful, I had also the same issue with imaging.Blur
, and I got better results by running runtime.GC()
, The latency is still volatile though and depends on whether the server handles other requests at the same time. After doing some profiling, I saw that dst := image.NewNRGBA(image.Rect(0, 0, src.w, src.h))
at https://github.com/disintegration/imaging/blob/master/effects.go#L36 eats about 80% of the memory that was allocated (111MB of 138MB). Since there is not much to do besides runtime.GC()
, I am thinking of trying gocv
which is a go wrapper for opencv. It wrapps the C implementation which hopefully frees memory as soon as the process is done.
from imaging.
runtime.GC()
was not enough in my case. For the information purpose: I tried GaussianBlur
from gocv, and it does better in term of memory consumption, but the docker image is now much bigger (more then 10x), I am still using other functionalities like contrast, and brightness without issues so far.
from imaging.
Related Issues (20)
- WebP support HOT 2
- How to get image width from imaging HOT 1
- Cannot apply transparency HOT 1
- imaging.Open throw error: image: unknown format HOT 2
- How to get file from postman HOT 2
- how can we resize image HOT 3
- Does imaging support wartermark text?
- Images not get fully processed even if background is filled with base color? HOT 1
- resize image become bigger HOT 1
- What is the different Resize and Thumbnail functions? HOT 1
- How to convert png-24 to png-8? HOT 1
- Support HEIF (.heic) files with AutoOrientation HOT 1
- Panic in the Rotate90 function.
- Export fixOrientation HOT 1
- Paste function panics
- Rotated image background won't blend according to Alpha when pasting
- Specific image will cause the index of the scan function in scanner.go to go out of bounds
- Snyk reports vulnerability HOT 1
- Maybe it's time to release version 1.6.3 so as not to have to go to the master branch for it
- 5 years doesn't update?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imaging.