grosser / parallel Goto Github PK
View Code? Open in Web Editor NEWRuby: parallel processing made simple and fast
License: MIT License
Ruby: parallel processing made simple and fast
License: MIT License
There's a nifty gem for adding fork()
in Windows: win32-process
When I run
bundle exec ruby spec/cases/map_with_ar_postges.rb
I get back:
making user
making user
making user
making user
making user
making user
making user
making user
/Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:512:in `exec': server closed the connection unexpectedly (PG::UnableToSend)
This probably means the server terminated abnormally
before or while processing the request.
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:512:in `dealloc'
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:495:in `block in clear'
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:494:in `each_value'
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:494:in `clear'
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:557:in `clear_cache!'
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract_adapter.rb:322:in `reconnect!'
from /Users/tyler/.rvm/gems/ruby-1.9.3-p392@parallel/gems/activerecord-4.0.0/lib/active_record/connection_adapters/postgresql_adapter.rb:569:in `reconnect!'
from spec/cases/map_with_ar_postges.rb:39:in `block in <main>'
from /Users/tyler/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/tempfile.rb:320:in `open'
from spec/cases/map_with_ar_postges.rb:4:in `<main>'
It appears that something is going awry with postgres.
Hi,
Many thanks for the cool lib - finding it really handy.
I have been using Parallel (0.5.1) with sqlite for a background job largely successfully, but now I am moving to mysql and seem to be getting issues with parallel db access.
I have tried the mysql, mysql2 and mysqlplus adaptors.
I am using Rails3/ActiveRecord on top of the adaptor - 3.0.3
With mysql, in the console, I do this:
Parallel.each(QuoteResponse.all, :in_processes => 5) do |r|
puts r.quote_site.site_name
end
Which gives this error (the site_name is a method on a related table - so it seems like its not loading the relating table - the non parallel version of above works fine :( ):
NoMethodError: undefined method site_name' for nil:NilClass from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activesupport-3.0.3/lib/active_support/whiny_nil.rb:48:in
method_missing'
from (irb):2
from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/parallel-0.5.1/lib/parallel.rb:250:in call' from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/parallel-0.5.1/lib/parallel.rb:250:in
call_with_index'
from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/parallel-0.5.1/lib/parallel.rb:186:in process_incoming_jobs' from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/parallel-0.5.1/lib/parallel.rb:169:in
worker'
from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/parallel-0.5.1/lib/parallel.rb:164:in `fork'
On Windows (jruby 1.7.3 and MRI Ruby 1.9.3 and 2.0.0) I tried this:
Parallel.each((1..8), :in_threads=>4) {|i|
p i
if (i==4) then
break
end
}
And it results in an exception:
LocalJumpError: break from proc-closure
Is this a bug? Is there (another?) way to stop all parallel threads?
Hi.
Parallel.map(orders, :in_processes => 6) do |order|
ActiveRecord::Base.establish_connection
# some huge calculation
end
And i get this error:
Mysql::Error: Lost connection to MySQL server during query
or this one:
Mysql::Error: MySQL server has gone away
What i did wrong?
process_count method fails. This version seems to work ok tho...at least for jruby on os x.
In general I think one could come up with a more robust way of finding it by using the os gem
if OS.mac?
case RUBY_PLATFORM
...
:)
Cheers!
-roger-
def self.processor_count
case RUBY_PLATFORM
when /darwin9/
hwprefs cpu_count
.to_i
when /darwin10/
(hwprefs_available? ? hwprefs thread_count
: sysctl -n hw.ncpu
).to_i
when /linux/
cat /proc/cpuinfo | grep processor | wc -l
.to_i
when /freebsd/
sysctl -n hw.ncpu
.to_i
else
require 'rbconfig'
if RbConfig::CONFIG['host_os'] =~ /darwin/
(hwprefs_available? ? hwprefs thread_count
: sysctl -n hw.ncpu
).to_i
else
raise 'unable to determine processor count'
end
end
end
Hi!
I'm not sure if it's a bug or a feature, but if I try this and press CTRL-C, it stil waits till the end:
$ ruby -e 'require "parallel"; Parallel.each([1,2,3], :in_threads => 3) {|n| sleep 60; puts n }'
^C
2
3
1
I would like the threads to stop on SIGINT.
Thanks!
Programs that use this parallel gem don't behave as expected running in OpenBSD.
Things were going wrong for no apparent reason, until I noticed the following message:
Unknown architecture ( openbsd5.1 ) assuming one processor.
Since parallel forks for each job, as opposed to forking on each process, when you have many small jobs where the forking overhead is high, Parallel takes a really long time.
Here is my benchmark script with results:
I just created a library playing with using the same fork per process, and using bi-directional pipes to pass jobs back and forth between the master and worker fork.
http://github.com/ngauthier/forque
Then someone pointed out you guys were already doing a similar project.
Think we should merge forque's double-pipe fork conservation into parallel for the in_processes method?
(note: the system used in forque is the core of the Hydra project http://github.com/ngauthier/hydra )
Can you add a feature which allows to immediately start new processes if some processes have finished, rather than waiting for all of the processes to complete in the current batch?
e.g. I have 10 processes and I have set the batch size to 4. From the first 4 processes, if 2 are over, Parallel should start the 5th one immediately before waiting for all 4 processes to finish.
Something like this --
http://parallelforkmgr.rubyforge.org/
But Parallel::ForkManager is ugly and confusing to use.
require 'parallel'
f = File.new('test.dat', 'w')
f.write "#{$$}: Start processing\n"
puts "#{$$}: Start processing"
Parallel.map([1, 2, 3, 4], :in_processes => 2){ |x| x }
f.write "#{$$}: Stop processing\n"
puts "#{$$}: Stop processing"
f.close
STDOUT contains strings as I expected:
32023: Start processing
32023: Stop processing
But the file contains 3 strings"Start processing" with the same PID:
32023: Start processing
32023: Start processing
32023: Start processing
32023: Stop processing
Is this normal behavior?
Hi there,
First and foremost, great job on the easy-to-use API!
I have a little question: Is there a way to kill sleeping child processes once the last iteration is completed? If I run something like this:
Parallel.each(1..1000, :in_processes => 4){|lol| puts lol}
The program goes through all the numbers just fine but it refuses to exit properly and kill all the processes. Even if I ctrl+c, the processes persist.
Cheers, Alejandro.
Trying to run Parallel in rake like this:
desc "Parallel Continuous integration"
task :parallel_check do
require 'parallel'
results = Parallel.map(FileList.new('??')+FileList.new('??-??')) do |lang|
Rake::Task["ci:" +lang+"_check"].execute
end
fail if results.any { |result| result!=0}
end
I get the following backtrace at the end of each subprocess:
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:279:in `process_incoming_jobs'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:257:in `block in worker'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:250:in `fork'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:250:in `worker'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:238:in `block in create_workers'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:237:in `each'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:237:in `create_workers'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:201:in `work_in_processes'
from /home/jnavila/.gems/gems/parallel-0.8.0/lib/parallel.rb:106:in `map'
...
Is there a way to get rid these exeptions?
Thanks for the amazing library!
Part of some code I'm writing features a web spider that crawls pages, with each hit updating an array that functions essentially as a queue. Semi-pseudocode:
@queue = []
@visited = []
def hit(url)
find_links.each { |link|
next if @visited.include? link
@queue << link
}
@visited << url
end
def crawl
Parallel.map(@queue, :in_threads => 4) { |url|
hit(url)
}
end
Using in_threads
it all works perfectly; though the array begins with one element, the first run-through queues up some more and these are gradually processed by the worker until all the unique URLs are hit.
However, there's a slight issue. Because of this line in map
:
size = [array.size, size].min
…since my array starts with only one element, only one worker thread is started, and so my code never uses more than one thread.
I can get around this by queuing the initial URL x
times, where x
is the number of threads I want to use; that fetches the initial URL x
times and everything else approximately once, but that seems awfully hacky and generally suboptimal.
Is there a way to tell Parallel to use more than one worker thread even if the initial workload doesn't have more than one element?
Hey,
would be great if a timeout option could be added to Parallel. Sometime I want to return after X seconds rather than waiting for the slowest thread to come back.
I recently noticed that parallel doesn't loop through all elements of hashes or array, at least in 0.8.4. Here's a simple example:
#!/usr/bin/env ruby
require 'parallel'
array = []
2000.times do
array << rand
end
STDERR.puts "Array Length: #{array.length}"
Parallel.each(array) do |item|
puts "Item: #{item}"
end
Outputs as follows:
$ ./bug.rb > test.log
Array Length: 2000
$ grep Item test.log | wc -l
1798
Or so it seems at a glance.
I started investigating when my script (https://github.com/MatmaRex/bio-index/blob/master/build-index.rb) started eating ungodly amounts of memory after replacing some each
es with Parallel.each
es. Adding nil
at the end of the block to ensure the data downloaded from the API isn't kept around makes it sane again.
Parallel.map([1]){|x| x} == [[1]]
I have a number of files which I need to download and process. To speed up my application I would like to download files during processing of previosly downloaded files.
Can you please provide any suggestion how to do it? My only idea is to use a queue.
I read this pull request but didn't find the queue implementation in the current source code. Is queue supported or not? Or is there any solution how to achieve what I need?
I'm trying to send three parallel request to a soap api. Normally each request takes around 13 seconds to complete so for three requests it takes around 39 seconds. After using parallel gem and sending three parallel requests using 3 threads it takes around 23 seconds to complete all three requests which is really nice but I'm not able to figure out why its not completing it in like 14-15 seconds. Its not an issue actually but as you know alot about parallel processing so I thought maybe you can point me in the right direction.
On my ops server I'm getting a 500 internal server error with only the below output in the server log when trying to run the action calling Parallel.
On my dev environment using plain old "rails server" it works fine though. I'm not really sure what is causing this though.
Completed 500 Internal Server Error in 1332ms
Parallel::DeadWorker (Parallel::DeadWorker):
parallel (0.6.4) lib/parallel.rb:51:in rescue in work' parallel (0.6.4) lib/parallel.rb:48:in
work'
parallel (0.6.4) lib/parallel.rb:210:in block (3 levels) in work_in_processes ' parallel (0.6.4) lib/parallel.rb:331:in
with_instrumentation'
parallel (0.6.4) lib/parallel.rb:209:in block (2 levels) in work_in_processes ' parallel (0.6.4) lib/parallel.rb:204:in
loop'
parallel (0.6.4) lib/parallel.rb:204:in block in work_in_processes' parallel (0.6.4) lib/parallel.rb:65:in
block (2 levels) in in_threads'
Calling it with the following code inside an objets model:
results = Parallel.map((calc_start..calc_stop)) do | start |
... (do some calculations
result = [array]
end
::ActiveRecord::Base.establish_connection
There are no access to any of the object's parameters inside of the loop but I threw in the establish connection command after the loop because during my development I was noticing I was still losing access to the database when running later commands.
Are there any other logs or things I should look at?
Why are you using base64 in the encode and decode methods? Is this needed?
I get the following error when I try to access my DB after Parallel finishes a job:
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
my code is as such:
def first_function
results = Parallel.map(some_array, :in_processes=>3) do |element|
ActiveRecord::Base.connection.reconnect!
do_something
end
return results
end
def second_function
do_something_with_results(results) #works well
ActiveRecord::Base.connection.execute(some_query) #this fails on every query I try
end
I know this is a problem starts with Parallel because when I change the line:
results = Parallel.map(some_array, :in_processes=>3) do |element|
to the line:
results = some_array.collect do |element|
I get the same value in "results" but the queries after don't fail.
please help.
This gem would be severely more useful if it were capable of enumerating over an Enumerator without first converting it to an array. I'm trying to use this to speed up a project that works with Minecraft's .mca format. I have a method that returns an enumerator over all the map chunks - which are read only as they are needed from the file, and doing that at the beginning of the process takes a bit of overhead. I can imagine other scenarios where the enumerator should not be converted to an array before enumeration as well.
I can't seem to find a way to make it work for threads. I think I have a process version working.
My development environment is choking on load errors while running parallel'd code, saying something about having trouble reloading certain classes in my /lib directory.
The classes change from time to time, and I have to reload my server to make it work. It works the first time I run a request that triggers the parallel code, and fails after that.
From my development.rb:
# Explicitly tells ActiveSupport to always unload these constants after
# every reqeust. Allows lib files to look like gems, but still use Rails autoloading.
# http://wondible.com/2011/12/23/give-rails-autoloading-a-boot-to-the-head/
# http://mechanicalbee.com/2011/autoreload-gems-rails-3.html
ActiveSupport::Dependencies.explicitly_unloadable_constants += %w[ Buyads ]
I didn't post app-specific code because I wasn't sure what was relevant.
Am I misconfiguring something?
Not really a code bug but I'm curious how this gem is slicing up the problem set. Maybe updating the readme with this information will be helpful to others later
I.e. if I say
Parallel.map((1..100), :in_processes => 20) do | item |
...
end
In my code I'm dealing with a shifting window problem where i have to also change the size of the window so I'm dealing with nested loops. The last iteration of my top loop has the least possible window sizes so it runs the fastest but the first iteration has the most possible window sizes so it runs the slowest. If you're using the first way of splitting up the problem set then it works fine for my code but the second way would mean I'm not getting quite as much performance.
There could be other problems though where the second method would be faster because of the lower overhead.
Hi, I'm using parallel(0.5.8) to process a regular amount of data with a rake task and parallel is crashing on production box, here is the log: https://gist.github.com/1204831
Ruby version: ruby 1.8.7 (2010-04-19 patchlevel 253) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2010.02
O.S: CentOS release 5.5 (Final)
On my development box the same rake task works fine:
Ruby version: ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10.5.0]
O.S: OS X
I'm trying to use the parallel gem via autotest / parallel_test, but am getting an unimplemented error for the fork in parallel.rb. Is it possible that you might add windows support ?
Parallel.map(accounts, :in_threads => accounts.count) do |account|
start_time = Time.now
begin
timeout(3.5) do
client.get(xxx)
end
rescue Timeout::Error
elapsed = "%.1f" % (Time.now - start_time)
logger.info "Timed out after #{elapsed}s"
next
end
end
Ubuntu 11.04
Apache passenger 3.0.4
Ruby 1.9.2-p180
Hi,
Great little gem but I've got a feature request. Could you make it work on Enumerable instead of Array?
I wanted to use it to split up the work of processing objects from a database. In this case it's from Mongo.
I tried to write something like
Parallel.each(mongo_collection.find) do |obj|
end
but realized that you don't call .each internally but instead use the [index] operator for Array.
Regards,
Although I am a fan of cleaning up things. Can you please put the license back in the gem.
some docs that at least say what functions are defined would be nice ;)
Hi,
I am getting the following errors when parallel processing on an ubuntu 12.04 LTS, using rbenv en ruby 1.9.3-p448.
ruby: pthread_mutex_lock.c:84: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
Parallel::DeadWorker
Tasks: TOP => upgrade
(See full trace by running task with --trace)
/foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:319:in write': Broken pipe (Errno::EPIPE) from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:319:in
dump'
from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:319:in process_incoming_jobs' from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:298:in
block in worker'
from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:291:in fork' from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:291:in
worker'
from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:279:in block in create_workers' from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:278:in
each'
from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:278:in create_workers' from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:242:in
work_in_processes'
from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:114:in map' from /foo/bar/.rbenv/versions/1.9.3-p448/lib/ruby/gems/1.9.1/gems/parallel-0.9.0/lib/parallel.rb:81:in
each'
I googled around and found that
ruby: pthread_mutex_lock.c:84: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
is caused by the fact that a mutex should only be unlocked by the thread that locked it. See http://stackoverflow.com/questions/1105745/pthread-mutex-assertion-error.
I am running in 4 processes. My Ubuntu VM has 2vCPU's .
Any ideas what to do about this?
require "parallel"
class Test1
attr_accessor :count
def initialize
@count = "First"
end
def inc
@count = "Second"
end
end
arr = []
arr.push Test1.new
puts "** Before #{arr[0].count}"
Parallel.map(arr, :in_processes => 8) { |test|
test.inc
}
puts "** After #{arr[0].count}"
I am expecting the second puts to print
** After Second
but it doesn't.
Am I wrong somewhere?
It says "MIT" in a few places (gemspec and readme), but that's a bit thin. There are multiple versions of the MIT license out there (see Wikipedia).
Assuming you mean the original MIT license, could you please include the following in a file LICENSE.txt
in the repository?
This is currently blocking the Wikimedia Foundation (the non-profit behind Wikipedia.org) from using a Ruby application that depends on the parallel
gem.
Copyright (C) 2013 Michael Grosser <[email protected]>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
It appears that "hw.physicalcpu" is missing in my FreeBSD 9.1 VM, so the physical processor count for Parallel is not working properly. I don't know if this is something related to it being a VM, or simply an OID that is not used anymore...?
$ ruby -e 'require "parallel"; puts Parallel.physical_processor_count'
sysctl: unknown oid 'hw.physicalcpu'
0
However, "hw.ncpu" is still available.
Hello ,
Is it possible to detach processes
when I do that
Parallel ( :processes >= X) do
bla bla
end
p "bla"
all processes must be finished before executing the next code
thnx
I have the following code in my rails app
results = Parallel.map(user_ids, :in_processes => 4) do |user_id|
//some calculations
[user_id, result]
end
This works properly in development environment. But when I try to test the method which contains this code, test fails with ActiveRecord::StatementInvalid: Mysql2::Error: MySQL server has gone away
Any help is much appreciated!
I think it is item, i
not i, item
It looks like the block passed to Thread.new ends up writing its output to a shared 'out' array that is a local variable. I think there is a potential race condition here since multiple threads could be concurrently writing their results to the same array, which is not a thread-safe data structure. This is probably more visible in JRuby where more than a single thread can be executing at a given time.
I'm getting this error when using the Mechanize gem:
def get_page(n, agent)
puts "Getting page #{n}..."
return agent.get(url_for_page(n))
end
pages = Parallel.map(page_num_array) do |n|
get_page(n, agent)
end
This is on Mac OS X, using RVM, in both ruby 1.9.3 and 2.0.0.
I'm getting exceptions in parallel.map threads when I save a record to postgresql. Most of the time it works ok, but sometimes I get an exception (not always the same one) such as below. Not sure if it's an issue with parallels, ruby, postgresql or Passenger/REE.
I use Rails 3.1.1rc2, ruby 1.8.7 on Passenger/REE.
uninitialized constant ActiveRecord::Associations::JoinDependency::JoinBase
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/associations/join_dependency.rb:13:in `initialize'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/relation/finder_methods.rb:219:in `new'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/relation/finder_methods.rb:219:in `construct_join_dependency_for_association_find'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/relation/finder_methods.rb:192:in `exists?'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations/uniqueness.rb:33:in `validate_each'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validator.rb:153:in `validate'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validator.rb:150:in `each'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validator.rb:150:in `validate'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:302:in `send'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:302:in `_callback_before_375'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:449:in `_run_validate_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `send'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations.rb:212:in `run_validations!'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:390:in `_run_validation_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `send'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations.rb:179:in `valid?'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations.rb:69:in `valid?'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations/associated.rb:5:in `validate_each'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations/associated.rb:5:in `collect'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations/associated.rb:5:in `validate_each'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validator.rb:153:in `validate'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validator.rb:150:in `each'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validator.rb:150:in `validate'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:302:in `send'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:302:in `_callback_before_411'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:449:in `_run_validate_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `send'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations.rb:212:in `run_validations!'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:390:in `_run_validation_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `send'
/home/aaa/shared/bundle/ruby/1.8/gems/activesupport-3.1.1.rc2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
/home/aaa/shared/bundle/ruby/1.8/gems/activemodel-3.1.1.rc2/lib/active_model/validations.rb:179:in `valid?'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations.rb:69:in `valid?'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations.rb:77:in `perform_validations'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations.rb:56:in `save!'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/transactions.rb:246:in `save!'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/transactions.rb:295:in `with_transaction_returning_status'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/transactions.rb:208:in `transaction'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/transactions.rb:246:in `save!'
/home/aaa/shared/bundle/ruby/1.8/gems/activerecord-3.1.1.rc2/lib/active_record/validations.rb:41:in `create!'
/home/aaa.rb:165:in `save' ...
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:260:in `call'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:260:in `call_with_index'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:114:in `work_in_threads'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:107:in `loop'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:107:in `work_in_threads'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:16:in `in_threads'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:15:in `initialize'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:15:in `new'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:15:in `in_threads'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:14:in `times'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:14:in `in_threads'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:105:in `work_in_threads'
/home/aaa/shared/bundle/ruby/1.8/gems/parallel-0.5.9/lib/parallel.rb:55:in `map'
Does the parallel gem work on Android, via Ruboto/JRuby?
I was having an issue where I was trying to perform too many long running DB connections in parallel. Rails suggested I increase my connection pool size, but I was afraid of having too many connections to mySQL per web worker process.
I realized my error was that I was simply using too many of the available connections, so I wanted to make sure I never exceeded the available pool. My solution was to use something like
:in_threads => [User.all.lenght, ActiveRecord::Base.connection_pool.instance_eval { @size }].min
This will use the number of connections equal to the number of records, OR the connection pool size, whichever is smaller.
Recently I have not been able to get the parallel gem working (in the current version) for the INT signal (e.g. Ctrl+C). For a little background, the machine in question is 32-bit Ubuntu 12.04 with Ruby 1.9.3, and 4 processors detected (4 cores, 1 CPU). The funny thing is that for two items in parallel, the code works just as expected. However, for three items or more, I get Errno::ESRCH (process cannot be found) when doing Ctrl+C.
$ ruby -v
ruby 1.9.3p0 (2011-10-30 revision 33570) [i686-linux]
$ ruby -e 'require "parallel"; Parallel.each([1,2]) {|n| sleep 60; puts n }'
^CParallel execution interrupted, exiting ...
$ ruby -e 'require "parallel"; Parallel.each([1,2,3]) {|n| sleep 60; puts n }'
^CParallel execution interrupted, exiting ...
/var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:267:in `kill': No such process (Errno::ESRCH)
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:267:in `block (2 levels) in kill_on_ctrl_c'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:267:in `each'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:267:in `block in kill_on_ctrl_c'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:237:in `call'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:237:in `join'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:237:in `block in wait_for_threads'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:235:in `each'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:235:in `wait_for_threads'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:18:in `in_threads'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:149:in `work_in_processes'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:55:in `map'
from /var/lib/gems/1.9.1/gems/parallel-0.5.18/lib/parallel.rb:30:in `each'
from -e:1:in `<main>'
I changed the code around to something like....
pids.each do |pid|
if pid
begin
Process.kill(:KILL, pid)
rescue Errno::ESRCH
# Child PID already dead
end
end
end
That seemed to fix the problem. Not sure why children would already be dead, though...?
Why is it necessary to run ActiveRecord::Base.connection.reconnect!
in order to parallelize the processing of ActiveRecord::Base objects? ActiveRecord::Base is thread safe, so what is this actually fixing? Reconnecting every iteration is extremely wasteful.
Hi,
Quick question. Does Parallel process in batches? For example if I have an array of 100 objects that I want to iterate through and I set the :in_threads => 10
or :in_processes => 10
, would it start processing the first 10 jobs (1..10), then wait for all 10 to finish, and then start on the second batch (job 11..20)? Or does it immediately begin processing the next job when 1 of the threads finishes its job? For example if you're using 10 threads, and thread #3 finishes it's job first, would it go ahead and pick up job #11 from the array while the other 9 are still processing their current jobs, or would thread #3 have to wait for all other threads to finish first (the batch) before starting a new job?
Looking at the source it would seem both threads and processes will be processing in batches. Could you confirm whether this is true or not? Thanks! :)
When installing the gem NOT using rvm, via:
$ sudo gem install parallel
Then the following file has read permissions only for root user:
As a result 'require "parallel"' fails
The completed stack trace is:
/usr/local/lib/ruby/gems/1.9.1/gems/parallel-0.5.15/lib/parallel.rb:5:in read': Permission denied - /usr/local/lib/ruby/gems/1.9.1/gems/parallel-0.5.15/lib/../VERSION (Errno::EACCES) from /usr/local/lib/ruby/gems/1.9.1/gems/parallel-0.5.15/lib/parallel.rb:5:in
class:Parallel'
from /usr/local/lib/ruby/gems/1.9.1/gems/parallel-0.5.15/lib/parallel.rb:4:in <top (required)>' from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:68:in
require'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:68:in block (2 levels) in require' from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in
each'
from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:66:in block in require' from /usr/local/lib/ruby/gems/1.9.1/gems/bundler-1.0.21/lib/bundler/runtime.rb:55:in
each'
The permission of the VERSION file in this case is:
$ ls -la /usr/local/lib/ruby/gems/1.9.1/gems/parallel-0.5.15/lib/../VERSION
-rw------- 1 root root 7 Mar 4 14:06 VERSION
I'd appreciate your assistance in fixing this and bumping the gem version.
Thanks,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.