Support none ASCII characters in create_assignments from csv. about workforce-scripts HOT 7 CLOSED

from workforce-scripts.

Hi @hmelendez

I cannot open the text file in python (defaults to UTF-8 encoding); it seems to be in a different encoding than UTF-8? Do you what encoding it is in?

I added some non-ascii characters to a UTF-8 encoded CSV file and was able to create the assignments as expected. I cannot reproduce your issue.

My simple csv

xField,yField,Type,Location,Dispatcher,Description,Priority,Work Order Id,Due Date,Attachment,Worker
-118.15,33.8,1,123 Street # 765,1,Test áéíóú,4,1,4/28/2016 23:59:59,,,my_username

Output on workforce app:

I'm guessing it's something to do with how your csv file is encoded, but I'm not sure. Can you post a screenshot of what you see after creation and how they are jumbled?

from workforce-scripts.

Thanks @MikeMillerGIS that worked.

@hmelendez I was able to import your text file of assignments by doing:

 with open(csvFile, 'r', encoding='iso-8859-1') as file:
        reader = csv.DictReader(file)
        for row in reader:
            assignments_in_csv.append(row)

Then in workforce they seem correct:

Does that look correct? It seems to match the txt file you provided. Can you try using iso-8859-1 encoding? See here for language coverage.

from workforce-scripts.

I added the enconding in line 93 but the error persists. Here is the screen capture and the csv.

today_assignments.txt

from workforce-scripts.

@hmelendez

Okay so I looked into this a bit more.

The first file you provided appears to be encoded as ANSI (at least that's what notepad++ sniffed from it) so you should probably use iso-8859-1 as the encoding. This will read the the CSV you provided properly and render properly as shown in my first comment.

The second file you provided appears to be encoded as UTF-8 (at least that's what notepad++ sniffed from it) so you should use UTF-8 as the encoding (which is default file io in python). When I do this it reads it as expected and renders accordingly:

Reading today_assignments.txt with UTF-8 encoding

As far as I know there's no reliable way to determine the encoding of the file though there as some heuristics that can be used. Chardet is a library that can detect the encoding using such heuristics. You could try incorporating this library if you have to routinely read CSV files with different encodings. You could also look into how the CSV files are generated and ensure they are the same encoding.

from workforce-scripts.

Hi.

I'm using export_assignments_to_csv and suddenly started getting the below - any advice appreciated.

I can't find the character reported anywhere in my Assignments (was expecting it to be in Notes left by Workers but, alas not).

Thanks in advance!

Jonathan

[2017-06-10 09:14:48,319] [ export_assignments_to_csv.py: 115 - main()] [MainThread] [ root] [ INFO] Authenticating...
[2017-06-10 09:14:48,807] [ export_assignments_to_csv.py: 118 - main()] [MainThread] [ root] [ INFO] Getting assignment
feature layer...
[2017-06-10 09:14:49,403] [ export_assignments_to_csv.py: 121 - main()] [MainThread] [ root] [ INFO] Querying assignmen
ts...
[2017-06-10 09:14:51,005] [ export_assignments_to_csv.py: 124 - main()] [MainThread] [ root] [ INFO] Writing to CSV...
[2017-06-10 09:14:56,757] [ export_assignments_to_csv.py: 149 - ()] [MainThread] [ root] [CRITICAL] Exception detected
, script exiting
[2017-06-10 09:14:56,758] [ export_assignments_to_csv.py: 150 - ()] [MainThread] [ root] [CRITICAL] 'ascii' codec can'
t encode character u'\u013a' in position 21: ordinal not in range(128)
[2017-06-10 09:14:56,760] [ export_assignments_to_csv.py: 151 - ()] [MainThread] [ root] [CRITICAL] Traceback (most re
cent call last): | File "/home/pi/arcgis/export_assignments_to_csv.py", line 147, in | main(args) | File "/home/pi/arcgis/export_assignments_to_c
sv.py", line 125, in main | write_assignments_to_csv(args.outCSV, assignments, args.dateFormat, args.timezone) | File "/home/pi/arcgis/export_assignments_to
_csv.py", line 110, in write_assignments_to_csv | writer.writerows(assignment_attributes) | File "/usr/lib/python2.7/csv.py", line 158, in writerows | r
eturn self.writer.writerows(rows) | UnicodeEncodeError: 'ascii' codec can't encode character u'\u013a' in position 21: ordinal not in range(128) |

^bump

from workforce-scripts.

Closing.

The scripts assume UTF-8 encoding of CSV files. If your csv files use a different encoding then you need to specify that in the script.

from workforce-scripts.