oslandia / osm-data-classification Goto Github PK
View Code? Open in Web Editor NEWMigrated to: https://gitlab.com/Oslandia/osm-data-classification
License: MIT License
osm-data-classification's People
Contributors
Stargazers
Watchers
osm-data-classification's Issues
Write the how to extract changesets editors by user
Failed Tasks and dependency
Hi,
Thank you so much for the tool!
I have been following the README but still get failed tasks.I've installed the required dependencies but I still get an output with tasks failed or left pending. I am running this on Windows.
Thanks in advance.
f:\osm-data-classification-master\abc\lib\site-packages\statsmodels\compat\pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
from pandas.core import datetools
2018-07-06 16:50:50,761 :: INFO :: instance : Loaded []
DEBUG: Checking if AutoKMeans(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2018-07-06 16:50:51,870 :: DEBUG :: check_complete : Checking if AutoKMeans(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
DEBUG: Checking if KMeansReport(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2018-07-06 16:50:51,873 :: DEBUG :: check_complete : Checking if KMeansReport(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
DEBUG: Checking if KMeansAnalysis(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2018-07-06 16:50:51,874 :: DEBUG :: check_complete : Checking if KMeansAnalysis(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
INFO: Informed scheduler that task AutoKMeans_data_bordeaux_metropo_user_c76397ef0a has status PENDING
2018-07-06 16:50:51,876 :: INFO :: _add_task : Informed scheduler that task AutoKMeans_data_bordeaux_metropo_user_c76397ef0a has status PENDING
INFO: Informed scheduler that task KMeansAnalysis_data_bordeaux_metropo_user_c76397ef0a has status PENDING
2018-07-06 16:50:51,878 :: INFO :: _add_task : Informed scheduler that task KMeansAnalysis_data_bordeaux_metropo_user_c76397ef0a has status PENDING
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=2) is complete
2018-07-06 16:50:51,882 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=2) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=3) is complete
2018-07-06 16:50:51,883 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=3) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=4) is complete
2018-07-06 16:50:51,884 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=4) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=5) is complete
2018-07-06 16:50:51,885 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=5) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=6) is complete
2018-07-06 16:50:51,887 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=6) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=7) is complete
2018-07-06 16:50:51,888 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=7) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=8) is complete
2018-07-06 16:50:51,889 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=8) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=9) is complete
2018-07-06 16:50:51,893 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, n_components=0, nb_clusters=9) is complete
INFO: Informed scheduler that task KMeansReport_data_bordeaux_metropo_user_c76397ef0a has status PENDING
2018-07-06 16:50:51,895 :: INFO :: _add_task : Informed scheduler that task KMeansReport_data_bordeaux_metropo_user_c76397ef0a has status PENDING
DEBUG: Checking if AutoPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
2018-07-06 16:50:51,898 :: DEBUG :: check_complete : Checking if AutoPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
DEBUG: Checking if PlottingPCAFeatureContributions(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
2018-07-06 16:50:51,899 :: DEBUG :: check_complete : Checking if PlottingPCAFeatureContributions(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
DEBUG: Checking if PlottingPCACorrelationCircle(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
2018-07-06 16:50:51,900 :: DEBUG :: check_complete : Checking if PlottingPCACorrelationCircle(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_460c2d45eb has status PENDING
2018-07-06 16:50:51,905 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_460c2d45eb has status PENDING
DEBUG: Checking if VarianceAnalysisTask(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=) is complete
2018-07-06 16:50:51,907 :: DEBUG :: check_complete : Checking if VarianceAnalysisTask(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=) is complete
INFO: Informed scheduler that task PlottingPCACorrelationCircle_data_bordeaux_metropo_user_3601814570 has status PENDING
2018-07-06 16:50:51,909 :: INFO :: _add_task : Informed scheduler that task PlottingPCACorrelationCircle_data_bordeaux_metropo_user_3601814570 has status PENDING
DEBUG: Checking if MetadataNormalization(datarep=data, dsname=bordeaux-metropole, metadata_type=user) is complete
2018-07-06 16:50:51,910 :: DEBUG :: check_complete : Checking if MetadataNormalization(datarep=data, dsname=bordeaux-metropole, metadata_type=user) is complete
INFO: Informed scheduler that task VarianceAnalysisTask_data_bordeaux_metropo__28294a788a has status PENDING
2018-07-06 16:50:51,911 :: INFO :: _add_task : Informed scheduler that task VarianceAnalysisTask_data_bordeaux_metropo__28294a788a has status PENDING
DEBUG: Checking if OSMElementEnrichment(datarep=data, dsname=bordeaux-metropole) is complete
2018-07-06 16:50:51,915 :: DEBUG :: check_complete : Checking if OSMElementEnrichment(datarep=data, dsname=bordeaux-metropole) is complete
DEBUG: Checking if AddExtraInfoUserMetadata(datarep=data, dsname=bordeaux-metropole, n_top_editor=5) is complete
2018-07-06 16:50:51,916 :: DEBUG :: check_complete : Checking if AddExtraInfoUserMetadata(datarep=data, dsname=bordeaux-metropole, n_top_editor=5) is complete
INFO: Informed scheduler that task MetadataNormalization_data_bordeaux_metropo_user_7662008ac0 has status PENDING
2018-07-06 16:50:51,918 :: INFO :: _add_task : Informed scheduler that task MetadataNormalization_data_bordeaux_metropo_user_7662008ac0 has status PENDING
DEBUG: Checking if EditorCountByUser(datarep=data, n_top_editor=5) is complete
2018-07-06 16:50:51,920 :: DEBUG :: check_complete : Checking if EditorCountByUser(datarep=data, n_top_editor=5) is complete
DEBUG: Checking if UserMetadataExtract(datarep=data, dsname=bordeaux-metropole) is complete
2018-07-06 16:50:51,922 :: DEBUG :: check_complete : Checking if UserMetadataExtract(datarep=data, dsname=bordeaux-metropole) is complete
INFO: Informed scheduler that task AddExtraInfoUserMetadata_data_bordeaux_metropo_5_f27c0fcbd7 has status PENDING
2018-07-06 16:50:51,926 :: INFO :: _add_task : Informed scheduler that task AddExtraInfoUserMetadata_data_bordeaux_metropo_5_f27c0fcbd7 has status PENDING
DEBUG: Checking if ChangeSetMetadataExtract(datarep=data, dsname=bordeaux-metropole) is complete
2018-07-06 16:50:51,927 :: DEBUG :: check_complete : Checking if ChangeSetMetadataExtract(datarep=data, dsname=bordeaux-metropole) is complete
INFO: Informed scheduler that task UserMetadataExtract_data_bordeaux_metropo_ddd7fcc55b has status PENDING
2018-07-06 16:50:51,929 :: INFO :: _add_task : Informed scheduler that task UserMetadataExtract_data_bordeaux_metropo_ddd7fcc55b has status PENDING
INFO: Informed scheduler that task ChangeSetMetadataExtract_data_bordeaux_metropo_ddd7fcc55b has status PENDING
2018-07-06 16:50:51,930 :: INFO :: _add_task : Informed scheduler that task ChangeSetMetadataExtract_data_bordeaux_metropo_ddd7fcc55b has status PENDING
DEBUG: Checking if TopMostUsedEditors(datarep=data) is complete
2018-07-06 16:50:51,931 :: DEBUG :: check_complete : Checking if TopMostUsedEditors(datarep=data) is complete
INFO: Informed scheduler that task EditorCountByUser_data_5_2262508398 has status PENDING
2018-07-06 16:50:51,933 :: INFO :: _add_task : Informed scheduler that task EditorCountByUser_data_5_2262508398 has status PENDING
INFO: Informed scheduler that task TopMostUsedEditors_data_c4ab8ddf6b has status PENDING
2018-07-06 16:50:51,937 :: INFO :: _add_task : Informed scheduler that task TopMostUsedEditors_data_c4ab8ddf6b has status PENDING
DEBUG: Checking if OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole) is complete
2018-07-06 16:50:51,938 :: DEBUG :: check_complete : Checking if OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole) is complete
INFO: Informed scheduler that task OSMElementEnrichment_data_bordeaux_metropo_ddd7fcc55b has status PENDING
2018-07-06 16:50:51,940 :: INFO :: _add_task : Informed scheduler that task OSMElementEnrichment_data_bordeaux_metropo_ddd7fcc55b has status PENDING
INFO: Informed scheduler that task OSMHistoryParsing_data_bordeaux_metropo_ddd7fcc55b has status PENDING
2018-07-06 16:50:51,941 :: INFO :: _add_task : Informed scheduler that task OSMHistoryParsing_data_bordeaux_metropo_ddd7fcc55b has status PENDING
INFO: Informed scheduler that task PlottingPCAFeatureContributions_data_bordeaux_metropo_user_3601814570 has status PENDING
2018-07-06 16:50:51,943 :: INFO :: _add_task : Informed scheduler that task PlottingPCAFeatureContributions_data_bordeaux_metropo_user_3601814570 has status PENDING
DEBUG: Checking if PlottingVarianceAnalysis(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
2018-07-06 16:50:51,948 :: DEBUG :: check_complete : Checking if PlottingVarianceAnalysis(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
INFO: Informed scheduler that task AutoPCA_data_bordeaux_metropo__b79d16b3a5 has status PENDING
2018-07-06 16:50:51,950 :: INFO :: _add_task : Informed scheduler that task AutoPCA_data_bordeaux_metropo__b79d16b3a5 has status PENDING
INFO: Informed scheduler that task PlottingVarianceAnalysis_data_bordeaux_metropo__b79d16b3a5 has status PENDING
2018-07-06 16:50:51,951 :: INFO :: _add_task : Informed scheduler that task PlottingVarianceAnalysis_data_bordeaux_metropo__b79d16b3a5 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_6ca60848b4 has status PENDING
2018-07-06 16:50:51,954 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_6ca60848b4 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_7e2b172958 has status PENDING
2018-07-06 16:50:51,957 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_7e2b172958 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_907bd7349a has status PENDING
2018-07-06 16:50:51,959 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_907bd7349a has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_d310e9eff3 has status PENDING
2018-07-06 16:50:51,961 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_d310e9eff3 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_6cc67586f5 has status PENDING
2018-07-06 16:50:51,963 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_6cc67586f5 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_9581a89357 has status PENDING
2018-07-06 16:50:51,965 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_9581a89357 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_e37b4903dd has status PENDING
2018-07-06 16:50:51,968 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_bordeaux_metropo_user_e37b4903dd has status PENDING
INFO: Done scheduling tasks
2018-07-06 16:50:51,969 :: INFO :: _schedule_and_run : Done scheduling tasks
INFO: Running Worker with 1 processes
2018-07-06 16:50:51,970 :: INFO :: run : Running Worker with 1 processes
DEBUG: Asking scheduler for work...
2018-07-06 16:50:51,971 :: DEBUG :: _get_work : Asking scheduler for work...
2018-07-06 16:50:51,972 :: INFO :: prune : Starting pruning of task graph
2018-07-06 16:50:51,973 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 24
2018-07-06 16:50:51,973 :: DEBUG :: run : Pending tasks: 24
INFO: [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) running TopMostUsedEditors(datarep=data)
2018-07-06 16:50:51,976 :: INFO :: run : [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) running TopMostUsedEditors(datarep=data)
ERROR: [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) failed TopMostUsedEditors(datarep=data)
Traceback (most recent call last):
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 191, in run
new_deps = self._run_get_new_deps()
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 129, in _run_get_new_deps
task_gen = self.task.run()
File "F:\osm-data-classification-master\src\analysis_tasks.py", line 296, in run
user_editor = pd.read_csv(fobj, header=None, names=['uid', 'value', 'num'])
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f
return _read(filepath_or_buffer, kwds)
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 411, in _read
data = parser.read(nrows)
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 1005, in read
ret = self._engine.read(nrows)
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 1748, in read
data = self._reader.read(nrows)
File "pandas\_libs\parsers.pyx", line 890, in pandas._libs.parsers.TextReader.read (pandas\_libs\parsers.c:10862)
File "pandas\_libs\parsers.pyx", line 912, in pandas._libs.parsers.TextReader._read_low_memory (pandas\_libs\parsers.c:11138)
File "pandas\_libs\parsers.pyx", line 966, in pandas._libs.parsers.TextReader._read_rows (pandas\_libs\parsers.c:11884)
File "pandas\_libs\parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas\_libs\parsers.c:11755)
File "pandas\_libs\parsers.pyx", line 2173, in pandas._libs.parsers.raise_parser_error (pandas\_libs\parsers.c:28589)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 111158: character maps to <undefined>
2018-07-06 16:50:53,524 :: ERROR :: run : [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) failed TopMostUsedEditors(datarep=data)
Traceback (most recent call last):
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 191, in run
new_deps = self._run_get_new_deps()
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 129, in _run_get_new_deps
task_gen = self.task.run()
File "F:\osm-data-classification-master\src\analysis_tasks.py", line 296, in run
user_editor = pd.read_csv(fobj, header=None, names=['uid', 'value', 'num'])
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f
return _read(filepath_or_buffer, kwds)
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 411, in _read
data = parser.read(nrows)
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 1005, in read
ret = self._engine.read(nrows)
File "f:\osm-data-classification-master\abc\lib\site-packages\pandas\io\parsers.py", line 1748, in read
data = self._reader.read(nrows)
File "pandas\_libs\parsers.pyx", line 890, in pandas._libs.parsers.TextReader.read (pandas\_libs\parsers.c:10862)
File "pandas\_libs\parsers.pyx", line 912, in pandas._libs.parsers.TextReader._read_low_memory (pandas\_libs\parsers.c:11138)
File "pandas\_libs\parsers.pyx", line 966, in pandas._libs.parsers.TextReader._read_rows (pandas\_libs\parsers.c:11884)
File "pandas\_libs\parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas\_libs\parsers.c:11755)
File "pandas\_libs\parsers.pyx", line 2173, in pandas._libs.parsers.raise_parser_error (pandas\_libs\parsers.c:28589)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 111158: character maps to <undefined>
DEBUG: 1 running tasks, waiting for next task to finish
2018-07-06 16:50:53,545 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task TopMostUsedEditors_data_c4ab8ddf6b has status FAILED
2018-07-06 16:50:53,560 :: INFO :: _add_task : Informed scheduler that task TopMostUsedEditors_data_c4ab8ddf6b has status FAILED
DEBUG: Asking scheduler for work...
2018-07-06 16:50:53,563 :: DEBUG :: _get_work : Asking scheduler for work...
2018-07-06 16:50:53,565 :: INFO :: prune : Starting pruning of task graph
2018-07-06 16:50:53,567 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 24
2018-07-06 16:50:53,570 :: DEBUG :: run : Pending tasks: 24
INFO: [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) running OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole)
2018-07-06 16:50:53,572 :: INFO :: run : [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) running OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole)
<TRACE> Initialization of a TimelineHandler instance !
ERROR: [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) failed OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole)
Traceback (most recent call last):
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 191, in run
new_deps = self._run_get_new_deps()
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 129, in _run_get_new_deps
task_gen = self.task.run()
File "F:\osm-data-classification-master\src\data_preparation_tasks.py", line 62, in run
tlhandler.apply_file(datapath)
RuntimeError: Open failed for 'data\raw\bordeaux-metropole.osh.pbf': The system cannot find the file specified.
2018-07-06 16:50:53,664 :: ERROR :: run : [pid 11560] Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) failed OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole)
Traceback (most recent call last):
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 191, in run
new_deps = self._run_get_new_deps()
File "f:\osm-data-classification-master\abc\lib\site-packages\luigi\worker.py", line 129, in _run_get_new_deps
task_gen = self.task.run()
File "F:\osm-data-classification-master\src\data_preparation_tasks.py", line 62, in run
tlhandler.apply_file(datapath)
RuntimeError: Open failed for 'data\raw\bordeaux-metropole.osh.pbf': The system cannot find the file specified.
DEBUG: 1 running tasks, waiting for next task to finish
2018-07-06 16:50:53,703 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task OSMHistoryParsing_data_bordeaux_metropo_ddd7fcc55b has status FAILED
2018-07-06 16:50:53,707 :: INFO :: _add_task : Informed scheduler that task OSMHistoryParsing_data_bordeaux_metropo_ddd7fcc55b has status FAILED
DEBUG: Asking scheduler for work...
2018-07-06 16:50:53,708 :: DEBUG :: _get_work : Asking scheduler for work...
2018-07-06 16:50:53,712 :: INFO :: prune : Starting pruning of task graph
2018-07-06 16:50:53,712 :: INFO :: prune : Done pruning task graph
DEBUG: Done
2018-07-06 16:50:53,713 :: DEBUG :: _log_remote_tasks : Done
DEBUG: There are no more tasks to run at this time
2018-07-06 16:50:53,714 :: DEBUG :: _log_remote_tasks : There are no more tasks to run at this time
DEBUG: There are 24 pending tasks possibly being run by other workers
2018-07-06 16:50:53,715 :: DEBUG :: _log_remote_tasks : There are 24 pending tasks possibly being run by other workers
DEBUG: There are 24 pending tasks unique to this worker
2018-07-06 16:50:53,716 :: DEBUG :: _log_remote_tasks : There are 24 pending tasks unique to this worker
DEBUG: There are 24 pending tasks last scheduled by this worker
2018-07-06 16:50:53,717 :: DEBUG :: _log_remote_tasks : There are 24 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) was stopped. Shutting down Keep-Alive thread
2018-07-06 16:50:53,718 :: INFO :: run : Worker Worker(salt=377050056, workers=1, host=LAPTOP-CGLPDH3E, username=Gangothri, pid=11560) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====
Scheduled 24 tasks of which:
* 2 failed:
- 1 OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole)
- 1 TopMostUsedEditors(datarep=data)
* 22 were left pending, among these:
* 22 had failed dependencies:
- 1 AddExtraInfoUserMetadata(datarep=data, dsname=bordeaux-metropole, n_top_editor=5)
- 1 AutoKMeans(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
- 1 AutoPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
- 1 ChangeSetMetadataExtract(datarep=data, dsname=bordeaux-metropole)
- 1 EditorCountByUser(datarep=data, n_top_editor=5)
...
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
2018-07-06 16:50:53,739 :: INFO :: _schedule_and_run :
===== Luigi Execution Summary =====
Scheduled 24 tasks of which:
* 2 failed:
- 1 OSMHistoryParsing(datarep=data, dsname=bordeaux-metropole)
- 1 TopMostUsedEditors(datarep=data)
* 22 were left pending, among these:
* 22 had failed dependencies:
- 1 AddExtraInfoUserMetadata(datarep=data, dsname=bordeaux-metropole, n_top_editor=5)
- 1 AutoKMeans(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
- 1 AutoPCA(datarep=data, dsname=bordeaux-metropole, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
- 1 ChangeSetMetadataExtract(datarep=data, dsname=bordeaux-metropole)
- 1 EditorCountByUser(datarep=data, n_top_editor=5)
...
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====```
Clean the project packaging
To avoid misunderstanding and make the project reusability easier, one should package the project in a cleaner way.
Failed dependencies
Hello,
Thank you so much for this tool!
I've been testing it out and I've come across some failed tasks. I've installed the required dependencies but I still get an output with tasks failed or left pending. Here is a screenshot of my console.
Any help is much appreciated!
user@ubuntu:~/osGeo/osm-data-classification$ PYTHONPATH=src python3 -m luigi --local-scheduler --module analysis_tasks AutoKMeans --dsname prince-edward-island
/home/user/.local/lib/python3.5/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
from pandas.core import datetools
2017-10-12 17:11:32,227 :: INFO :: instance : Loaded []
DEBUG: Checking if AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2017-10-12 17:11:32,279 :: DEBUG :: check_complete : Checking if AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
DEBUG: Checking if KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2017-10-12 17:11:32,281 :: DEBUG :: check_complete : Checking if KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
DEBUG: Checking if KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
2017-10-12 17:11:32,282 :: DEBUG :: check_complete : Checking if KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8) is complete
INFO: Informed scheduler that task AutoKMeans_data_prince_edward_is_user_17626ff787 has status PENDING
2017-10-12 17:11:32,283 :: INFO :: _add_task : Informed scheduler that task AutoKMeans_data_prince_edward_is_user_17626ff787 has status PENDING
INFO: Informed scheduler that task KMeansAnalysis_data_prince_edward_is_user_17626ff787 has status PENDING
2017-10-12 17:11:32,283 :: INFO :: _add_task : Informed scheduler that task KMeansAnalysis_data_prince_edward_is_user_17626ff787 has status PENDING
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2) is complete
2017-10-12 17:11:32,286 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3) is complete
2017-10-12 17:11:32,287 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4) is complete
2017-10-12 17:11:32,287 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5) is complete
2017-10-12 17:11:32,288 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6) is complete
2017-10-12 17:11:32,288 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7) is complete
2017-10-12 17:11:32,289 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8) is complete
2017-10-12 17:11:32,289 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8) is complete
DEBUG: Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9) is complete
2017-10-12 17:11:32,289 :: DEBUG :: check_complete : Checking if KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9) is complete
INFO: Informed scheduler that task KMeansReport_data_prince_edward_is_user_17626ff787 has status PENDING
2017-10-12 17:11:32,290 :: INFO :: _add_task : Informed scheduler that task KMeansReport_data_prince_edward_is_user_17626ff787 has status PENDING
DEBUG: Checking if AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
2017-10-12 17:11:32,292 :: DEBUG :: check_complete : Checking if AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
DEBUG: Checking if PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
2017-10-12 17:11:32,293 :: DEBUG :: check_complete : Checking if PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
DEBUG: Checking if PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
2017-10-12 17:11:32,293 :: DEBUG :: check_complete : Checking if PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12) is complete
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd has status PENDING
2017-10-12 17:11:32,294 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd has status PENDING
INFO: Informed scheduler that task PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5 has status PENDING
2017-10-12 17:11:32,295 :: INFO :: _add_task : Informed scheduler that task PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5 has status PENDING
DEBUG: Checking if VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=) is complete
2017-10-12 17:11:32,296 :: DEBUG :: check_complete : Checking if VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=) is complete
INFO: Informed scheduler that task PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5 has status PENDING
2017-10-12 17:11:32,297 :: INFO :: _add_task : Informed scheduler that task PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5 has status PENDING
DEBUG: Checking if MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user) is complete
2017-10-12 17:11:32,298 :: DEBUG :: check_complete : Checking if MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user) is complete
INFO: Informed scheduler that task VarianceAnalysisTask_data_prince_edward_is__de6ea788c3 has status PENDING
2017-10-12 17:11:32,298 :: INFO :: _add_task : Informed scheduler that task VarianceAnalysisTask_data_prince_edward_is__de6ea788c3 has status PENDING
DEBUG: Checking if OSMElementEnrichment(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,299 :: DEBUG :: check_complete : Checking if OSMElementEnrichment(datarep=data, dsname=prince-edward-island) is complete
DEBUG: Checking if AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5) is complete
2017-10-12 17:11:32,300 :: DEBUG :: check_complete : Checking if AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5) is complete
INFO: Informed scheduler that task MetadataNormalization_data_prince_edward_is_user_37dbbaf168 has status PENDING
2017-10-12 17:11:32,301 :: INFO :: _add_task : Informed scheduler that task MetadataNormalization_data_prince_edward_is_user_37dbbaf168 has status PENDING
DEBUG: Checking if EditorCountByUser(datarep=data, n_top_editor=5) is complete
2017-10-12 17:11:32,302 :: DEBUG :: check_complete : Checking if EditorCountByUser(datarep=data, n_top_editor=5) is complete
DEBUG: Checking if UserMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,302 :: DEBUG :: check_complete : Checking if UserMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
INFO: Informed scheduler that task AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60 has status PENDING
2017-10-12 17:11:32,303 :: INFO :: _add_task : Informed scheduler that task AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60 has status PENDING
DEBUG: Checking if ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,304 :: DEBUG :: check_complete : Checking if ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island) is complete
INFO: Informed scheduler that task UserMetadataExtract_data_prince_edward_is_ee0c385c15 has status PENDING
2017-10-12 17:11:32,304 :: INFO :: _add_task : Informed scheduler that task UserMetadataExtract_data_prince_edward_is_ee0c385c15 has status PENDING
INFO: Informed scheduler that task ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15 has status PENDING
2017-10-12 17:11:32,305 :: INFO :: _add_task : Informed scheduler that task ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15 has status PENDING
INFO: Informed scheduler that task EditorCountByUser_data_5_2262508398 has status DONE
2017-10-12 17:11:32,305 :: INFO :: _add_task : Informed scheduler that task EditorCountByUser_data_5_2262508398 has status DONE
DEBUG: Checking if OSMHistoryParsing(datarep=data, dsname=prince-edward-island) is complete
2017-10-12 17:11:32,306 :: DEBUG :: check_complete : Checking if OSMHistoryParsing(datarep=data, dsname=prince-edward-island) is complete
INFO: Informed scheduler that task OSMElementEnrichment_data_prince_edward_is_ee0c385c15 has status PENDING
2017-10-12 17:11:32,306 :: INFO :: _add_task : Informed scheduler that task OSMElementEnrichment_data_prince_edward_is_ee0c385c15 has status PENDING
INFO: Informed scheduler that task OSMHistoryParsing_data_prince_edward_is_ee0c385c15 has status PENDING
2017-10-12 17:11:32,307 :: INFO :: _add_task : Informed scheduler that task OSMHistoryParsing_data_prince_edward_is_ee0c385c15 has status PENDING
DEBUG: Checking if PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
2017-10-12 17:11:32,308 :: DEBUG :: check_complete : Checking if PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=) is complete
INFO: Informed scheduler that task AutoPCA_data_prince_edward_is__dba8e840cb has status PENDING
2017-10-12 17:11:32,308 :: INFO :: _add_task : Informed scheduler that task AutoPCA_data_prince_edward_is__dba8e840cb has status PENDING
INFO: Informed scheduler that task PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb has status PENDING
2017-10-12 17:11:32,309 :: INFO :: _add_task : Informed scheduler that task PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_d62355b835 has status PENDING
2017-10-12 17:11:32,310 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_d62355b835 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f has status PENDING
2017-10-12 17:11:32,311 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_107c39fe48 has status PENDING
2017-10-12 17:11:32,312 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_107c39fe48 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_23ac327dcf has status PENDING
2017-10-12 17:11:32,313 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_23ac327dcf has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_3d109ea54e has status PENDING
2017-10-12 17:11:32,314 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_3d109ea54e has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_2afd311993 has status PENDING
2017-10-12 17:11:32,315 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_2afd311993 has status PENDING
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd has status PENDING
2017-10-12 17:11:32,317 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd has status PENDING
INFO: Done scheduling tasks
2017-10-12 17:11:32,317 :: INFO :: _schedule_and_run : Done scheduling tasks
INFO: Running Worker with 1 processes
2017-10-12 17:11:32,317 :: INFO :: run : Running Worker with 1 processes
DEBUG: Asking scheduler for work...
2017-10-12 17:11:32,317 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:11:32,317 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:11:32,317 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 22
2017-10-12 17:11:32,318 :: DEBUG :: run : Pending tasks: 22
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
2017-10-12 17:11:32,318 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
<TRACE> Initialization of a TimelineHandler instance !
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
2017-10-12 17:12:46,953 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done OSMHistoryParsing(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:12:46,954 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task OSMHistoryParsing_data_prince_edward_is_ee0c385c15 has status DONE
2017-10-12 17:12:46,954 :: INFO :: _add_task : Informed scheduler that task OSMHistoryParsing_data_prince_edward_is_ee0c385c15 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:12:46,954 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:12:46,954 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:12:46,955 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 21
2017-10-12 17:12:46,955 :: DEBUG :: run : Pending tasks: 21
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
2017-10-12 17:12:46,955 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
/home/user/.local/lib/python3.5/site-packages/numpy/lib/arraysetops.py:463: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
mask |= (ar1 == a)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
2017-10-12 17:13:32,723 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done OSMElementEnrichment(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:13:32,723 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task OSMElementEnrichment_data_prince_edward_is_ee0c385c15 has status DONE
2017-10-12 17:13:32,724 :: INFO :: _add_task : Informed scheduler that task OSMElementEnrichment_data_prince_edward_is_ee0c385c15 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:13:32,724 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:13:32,724 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:13:32,724 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 20
2017-10-12 17:13:32,725 :: DEBUG :: run : Pending tasks: 20
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:13:32,725 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:14:06,386 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:06,386 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15 has status DONE
2017-10-12 17:14:06,387 :: INFO :: _add_task : Informed scheduler that task ChangeSetMetadataExtract_data_prince_edward_is_ee0c385c15 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:06,387 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:06,387 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:06,387 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 19
2017-10-12 17:14:06,388 :: DEBUG :: run : Pending tasks: 19
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running UserMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:14:06,388 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running UserMetadataExtract(datarep=data, dsname=prince-edward-island)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done UserMetadataExtract(datarep=data, dsname=prince-edward-island)
2017-10-12 17:14:14,528 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done UserMetadataExtract(datarep=data, dsname=prince-edward-island)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:14,529 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task UserMetadataExtract_data_prince_edward_is_ee0c385c15 has status DONE
2017-10-12 17:14:14,529 :: INFO :: _add_task : Informed scheduler that task UserMetadataExtract_data_prince_edward_is_ee0c385c15 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:14,529 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:14,529 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:14,530 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 18
2017-10-12 17:14:14,530 :: DEBUG :: run : Pending tasks: 18
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
2017-10-12 17:14:14,530 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
2017-10-12 17:14:15,679 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:15,680 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60 has status DONE
2017-10-12 17:14:15,680 :: INFO :: _add_task : Informed scheduler that task AddExtraInfoUserMetadata_data_prince_edward_is_5_ae03abeb60 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:15,681 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:15,681 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:15,681 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 17
2017-10-12 17:14:15,681 :: DEBUG :: run : Pending tasks: 17
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
2017-10-12 17:14:15,681 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
2017-10-12 17:14:22,565 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done MetadataNormalization(datarep=data, dsname=prince-edward-island, metadata_type=user)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:22,566 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task MetadataNormalization_data_prince_edward_is_user_37dbbaf168 has status DONE
2017-10-12 17:14:22,566 :: INFO :: _add_task : Informed scheduler that task MetadataNormalization_data_prince_edward_is_user_37dbbaf168 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:22,566 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:22,567 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:22,567 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 16
2017-10-12 17:14:22,567 :: DEBUG :: run : Pending tasks: 16
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
2017-10-12 17:14:22,567 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
2017-10-12 17:14:22,589 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done VarianceAnalysisTask(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_mindimensions=3, nb_maxdimensions=12, features=)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:22,589 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task VarianceAnalysisTask_data_prince_edward_is__de6ea788c3 has status DONE
2017-10-12 17:14:22,590 :: INFO :: _add_task : Informed scheduler that task VarianceAnalysisTask_data_prince_edward_is__de6ea788c3 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:22,590 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:22,590 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:22,590 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 15
2017-10-12 17:14:22,591 :: DEBUG :: run : Pending tasks: 15
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:22,591 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:23,022 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done PlottingVarianceAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:23,023 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb has status DONE
2017-10-12 17:14:23,023 :: INFO :: _add_task : Informed scheduler that task PlottingVarianceAnalysis_data_prince_edward_is__dba8e840cb has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:23,023 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:23,023 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:23,023 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 14
2017-10-12 17:14:23,024 :: DEBUG :: run : Pending tasks: 14
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:23,024 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
2017-10-12 17:14:23,182 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:23,183 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task AutoPCA_data_prince_edward_is__dba8e840cb has status DONE
2017-10-12 17:14:23,183 :: INFO :: _add_task : Informed scheduler that task AutoPCA_data_prince_edward_is__dba8e840cb has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:23,183 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:23,183 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:23,183 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 13
2017-10-12 17:14:23,184 :: DEBUG :: run : Pending tasks: 13
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:23,184 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:23,906 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done PlottingPCAFeatureContributions(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:23,907 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5 has status DONE
2017-10-12 17:14:23,907 :: INFO :: _add_task : Informed scheduler that task PlottingPCAFeatureContributions_data_prince_edward_is_user_712495f9c5 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:23,908 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:23,908 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:23,908 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 12
2017-10-12 17:14:23,908 :: DEBUG :: run : Pending tasks: 12
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:23,908 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
2017-10-12 17:14:24,874 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done PlottingPCACorrelationCircle(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:24,875 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5 has status DONE
2017-10-12 17:14:24,876 :: INFO :: _add_task : Informed scheduler that task PlottingPCACorrelationCircle_data_prince_edward_is_user_712495f9c5 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:24,876 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:24,876 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:24,876 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 11
2017-10-12 17:14:24,876 :: DEBUG :: run : Pending tasks: 11
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
2017-10-12 17:14:24,876 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
2017-10-12 17:14:25,304 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=5)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:25,304 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_23ac327dcf has status DONE
2017-10-12 17:14:25,305 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_23ac327dcf has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:25,305 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:25,305 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:25,305 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 10
2017-10-12 17:14:25,306 :: DEBUG :: run : Pending tasks: 10
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
2017-10-12 17:14:25,306 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
2017-10-12 17:14:25,764 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=6)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:25,764 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_107c39fe48 has status DONE
2017-10-12 17:14:25,765 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_107c39fe48 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:25,765 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:25,765 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:25,765 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 9
2017-10-12 17:14:25,766 :: DEBUG :: run : Pending tasks: 9
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
2017-10-12 17:14:25,766 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
2017-10-12 17:14:26,272 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=7)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:26,272 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f has status DONE
2017-10-12 17:14:26,273 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_a7a3bb456f has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:26,273 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:26,273 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:26,273 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 8
2017-10-12 17:14:26,274 :: DEBUG :: run : Pending tasks: 8
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
2017-10-12 17:14:26,274 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
2017-10-12 17:14:26,834 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=8)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:26,835 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_d62355b835 has status DONE
2017-10-12 17:14:26,835 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_d62355b835 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:26,835 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:26,836 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:26,836 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 7
2017-10-12 17:14:26,836 :: DEBUG :: run : Pending tasks: 7
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
2017-10-12 17:14:26,836 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
2017-10-12 17:14:27,182 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=3)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:27,182 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_2afd311993 has status DONE
2017-10-12 17:14:27,183 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_2afd311993 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:27,183 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:27,183 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:27,183 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 6
2017-10-12 17:14:27,184 :: DEBUG :: run : Pending tasks: 6
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
2017-10-12 17:14:27,184 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
2017-10-12 17:14:27,579 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=4)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:27,580 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_3d109ea54e has status DONE
2017-10-12 17:14:27,581 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_3d109ea54e has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:27,581 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:27,581 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:27,581 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 5
2017-10-12 17:14:27,581 :: DEBUG :: run : Pending tasks: 5
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
2017-10-12 17:14:27,581 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
2017-10-12 17:14:27,779 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:27,779 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd has status DONE
2017-10-12 17:14:27,780 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_6fba20c9dd has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:27,780 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:27,780 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:27,780 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 4
2017-10-12 17:14:27,781 :: DEBUG :: run : Pending tasks: 4
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
2017-10-12 17:14:27,781 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
2017-10-12 17:14:28,377 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=9)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:28,377 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd has status DONE
2017-10-12 17:14:28,378 :: INFO :: _add_task : Informed scheduler that task KMeansFromPCA_data_prince_edward_is_user_f145ee7fbd has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:28,378 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:28,378 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:28,378 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 3
2017-10-12 17:14:28,378 :: DEBUG :: run : Pending tasks: 3
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
2017-10-12 17:14:28,379 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
2017-10-12 17:14:28,490 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) done KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:28,491 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansReport_data_prince_edward_is_user_17626ff787 has status DONE
2017-10-12 17:14:28,491 :: INFO :: _add_task : Informed scheduler that task KMeansReport_data_prince_edward_is_user_17626ff787 has status DONE
DEBUG: Asking scheduler for work...
2017-10-12 17:14:28,492 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:28,492 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:28,492 :: INFO :: prune : Done pruning task graph
DEBUG: Pending tasks: 2
2017-10-12 17:14:28,493 :: DEBUG :: run : Pending tasks: 2
INFO: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
2017-10-12 17:14:28,493 :: INFO :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) running KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
ERROR: [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) failed KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
Traceback (most recent call last):
File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 194, in run
new_deps = self._run_get_new_deps()
File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 131, in _run_get_new_deps
task_gen = self.task.run()
File "/home/user/osGeo/osm-data-classification/src/analysis_tasks.py", line 910, in run
self.nbmin_clusters, self.nbmax_clusters)
File "/home/user/osGeo/osm-data-classification/src/unsupervised_learning.py", line 451, in kmeans_elbow_silhouette
s = random.sample(range(len(feature)), 2000)
File "/usr/lib/python3.5/random.py", line 315, in sample
raise ValueError("Sample larger than population")
ValueError: Sample larger than population
2017-10-12 17:14:28,608 :: ERROR :: run : [pid 31843] Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) failed KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
Traceback (most recent call last):
File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 194, in run
new_deps = self._run_get_new_deps()
File "/home/user/.local/lib/python3.5/site-packages/luigi/worker.py", line 131, in _run_get_new_deps
task_gen = self.task.run()
File "/home/user/osGeo/osm-data-classification/src/analysis_tasks.py", line 910, in run
self.nbmin_clusters, self.nbmax_clusters)
File "/home/user/osGeo/osm-data-classification/src/unsupervised_learning.py", line 451, in kmeans_elbow_silhouette
s = random.sample(range(len(feature)), 2000)
File "/usr/lib/python3.5/random.py", line 315, in sample
raise ValueError("Sample larger than population")
ValueError: Sample larger than population
DEBUG: 1 running tasks, waiting for next task to finish
2017-10-12 17:14:28,632 :: DEBUG :: run : 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task KMeansAnalysis_data_prince_edward_is_user_17626ff787 has status FAILED
2017-10-12 17:14:28,639 :: INFO :: _add_task : Informed scheduler that task KMeansAnalysis_data_prince_edward_is_user_17626ff787 has status FAILED
DEBUG: Asking scheduler for work...
2017-10-12 17:14:28,639 :: DEBUG :: _get_work : Asking scheduler for work...
2017-10-12 17:14:28,639 :: INFO :: prune : Starting pruning of task graph
2017-10-12 17:14:28,639 :: INFO :: prune : Done pruning task graph
DEBUG: Done
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : Done
DEBUG: There are no more tasks to run at this time
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
2017-10-12 17:14:28,640 :: DEBUG :: _log_remote_tasks : There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) was stopped. Shutting down Keep-Alive thread
2017-10-12 17:14:28,641 :: INFO :: run : Worker Worker(salt=081532708, workers=1, host=ubuntu, username=user, pid=31843) was stopped. Shutting down Keep-Alive thread
INFO:
**===== Luigi Execution Summary =====
Scheduled 23 tasks of which:
* 1 present dependencies were encountered:
- 1 EditorCountByUser(datarep=data, n_top_editor=5)
* 20 ran successfully:
- 1 AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
- 1 AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
- 1 ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
- 8 KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2...9)
- 1 KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
...
* 1 failed:
- 1 KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
* 1 were left pending, among these:
* 1 had failed dependencies:
- 1 AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
2017-10-12 17:14:28,648 :: INFO :: _schedule_and_run :
===== Luigi Execution Summary =====
Scheduled 23 tasks of which:
* 1 present dependencies were encountered:
- 1 EditorCountByUser(datarep=data, n_top_editor=5)
* 20 ran successfully:
- 1 AddExtraInfoUserMetadata(datarep=data, dsname=prince-edward-island, n_top_editor=5)
- 1 AutoPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, nb_min_dim=3, nb_max_dim=12, features=)
- 1 ChangeSetMetadataExtract(datarep=data, dsname=prince-edward-island)
- 8 KMeansFromPCA(datarep=data, dsname=prince-edward-island, metadata_type=user, n_components=0, nb_clusters=2...9)
- 1 KMeansReport(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
...
* 1 failed:
- 1 KMeansAnalysis(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
* 1 were left pending, among these:
* 1 had failed dependencies:
- 1 AutoKMeans(datarep=data, dsname=prince-edward-island, metadata_type=user, nbmin_clusters=3, nbmax_clusters=8)
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====Refactor the OSM data parsing
OSM data are parsed with pyosmium, however we use only very simple features of this tool. Maybe we can dive into the pyosmium API to redesign the way we retrieve OSM metadata.
Acknowledgements
Hello,
I would like to say thank you for this amazing framework that has allowed me to assess contributor characteristics for part of my Master's thesis research. I am currently submitting a draft for publication and I was wondering if I could include thanks to the Oslandia team in the Acknowledgements sections of the paper?
Acknowledgments
Contributions from anyone who does not meet the criteria for authorship should be listed, with permission from the contributor, in an Acknowledgments section.
https://onlinelibrary.wiley.com/page/journal/14679671/homepage/forauthors.html
Regards,
Kent
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.