parallelize ( list list , int nbThreads , closure method , int _verbose ) : void
Launch parallel threads which run method on some items of the list. All actions in methods must be independant with elements of the list.
Be careful to use local variables in your method.
To fully understand parallelisation, please read this article.
Returns true when all threads are finished. Note parallelize interrupt the execution of the code in the main tread (synchronous execution). So your code continue its execution when all parallelized threads have finished.
Example
parallelize( [1,2,3,4,5] , 2 , {def param->
print(param)
} )
Same than
parall={def param->
print(param)
}
parallelize([1,2,3,4,5],2, parall )
It is often appropriate to list the tasks to be accomplished by asking yourself the question "what are the variable elements of my tasks?" and then paralleling this list of tasks:
myTasks=[
["id_category":27, "page":1],
["id_category":27, "page":2],
["id_category":27, "page":3],
["id_category":580, "page":1],
["id_category":580, "page":2],
]
parallelize(myTasks,50, { def params->
def id_category = get(params, "id_category")
def page = get(params, "page")
process_category_page(id_category, page)
} )
Note on multi-thread operations
All Grimport variables are global by default, use "def" to make local variables in your method. Use synchronize to manage shared variables between thread (read and write operations).See also
getOrCreateThreadObjectasynchronous
synchronize
waitDelay
Other solution : Gpars
You can use in another way the Gpars features, like that :
import groovyx.gpars.*
list = [ 1, 2, 3, 4, 5 ]
GParsPool.withPool{
result = list.eachParallel {param->
print(param)
}
}
With Gpars the number of optimal threads is calculated automatically. In a withPool block, you can use features each, collect and find like normally, adding the term Parallel at the end.This functionality is taken from Groovy but we note however generally unsatisfactory results in the context of Web-Mining with this method.
Parameters
list
list where are arguments or array of arguments for the method
nbThreads
Number of parallel threads to launch
method
The function which is launched in threads
_verbose (optional)
Default 2. Conditions the number of messages returned by the function.
• 0 = none
• 1 = just 1 at the end
• 2 = more messages
• 0 = none
• 1 = just 1 at the end
• 2 = more messages