tarsh / pyshp Goto Github PK

What steps will reproduce the problem?
1. Open a shapefile in which its dbf has an overflow in a column (such as [1])
2. Read records

What is the expected output? What do you see instead?
It shouldn't crash, it crashes because it cannot convert asterisks to numbers.

What version of the product are you using? On what operating system?
Trunk

Please provide any additional information below.
According to [2] DBf files can contain numeric fields with asterisks. I suppose 
the best way to handle them is as if they where null. Right now pyshp converts 
those to 0. Attached is a patch that handles this issue.

[1] 
http://intgis.montevideo.gub.uy/sit/php/common/datos/generar_zip2.php?nom_tab=v_
uptu_lsv_destinos&tipo=gis
[2] 
http://connect.microsoft.com/SQLServer/feedback/details/648410/sql-reports-an-er
ror-when-querying-standard-dbf-data-via-vfp-ole-db-provider

Hi,

It's important to make python packages easily installable for users.  I saw one 
comment on the wiki about this and I certainly want to be able to pip install 
or easy_install this package.  It's a real pain when you see you have to 
download a file manually and put it somewhere manually.

Attached is a setup.py file that works with standard python packaging so it can 
be on pypi with easy installation.

Simply create a ~/.pypirc file with:

[distutils]
index-servers =
    pypi

[pypi]
username:yourusername
password:yourpassword

Then in your project dir you can register the pyshp project which only ever 
needs to be done once with:

python setup.py register

From then on when creating a new release you can create a source tarball with:

python setup.py sdist

And once you are happy you can upload with:

python setup.py sdist upload


I tested all these and they work which made pyshp installable with:

pip install pyshp

easy_install pyshp

(I have since deleted the package that I registered and uploaded for testing).


You can also use develop mode when working on the project with:

python setup.py develop

What steps will reproduce the problem?
1. Load the attached file
2. Ask for shapeRecords()

Get this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "shapefile.py", line 430, in shapeRecords
    for rec in zip(self.shapes(), self.records())]
  File "shapefile.py", line 413, in records
    r = self.__record()
  File "shapefile.py", line 378, in __record
    value = int(value)
ValueError: invalid literal for int() with base 10: 
'********************************'

What steps will reproduce the problem?

outShp = shapefile.Writer(shapefile.POLYGONZ)
outShp.field('id','N','10')
poly = [ [(0,0,0), (1,0,0), (1,0,1), (0,0,1), (0,0,0)] ]
outShp.poly(poly, shapeType=15)
outShp.record(1)
outShp.save(outShpPath)

What is the expected output? What do you see instead?

File "C:\Python32\lib\site-packages\shapefile.py", line 863, in save
    self.saveShp(target)
File "C:\Python32\lib\site-packages\shapefile.py", line 831, in saveShp
    self.__shpRecords()
File "C:\Python32\lib\site-packages\shapefile.py", line 678, in __shpRecords
    f.write(pack("<%sd" % len(s.z), *s.z))
AttributeError: '_Shape' object has no attribute 'z'

What version of the product are you using? On what operating system?
1.1.6-beta

Please provide any additional information below.

no error with 1.1.4

The parcel/point shapefile from http://www.ccmap.us/catalog.asp gives:

>>> sf = shapefile.Reader("CAD_AO_ParcelPoint_0711")
>>> sf.fields
[('DeletionFlag', 'C', 1, 0), ['APN', 'C', 9, 0], ['PARC_PY_ID', 'N', 19, 11], 
['REV_DATE', 'D', 8, 0], ['S_STR_NM', 'C', 30, 0], ['S_STR_SUF', 'C', 4, 0], 
['S_STR_NBR', 'C', 5, 0], ['S_FRAC', 'C', 4, 0], ['S_APT_NBR', 'C', 9, 0], 
['S_CTY_ABBR', 'C', 5, 0], ['S_ZIP', 'C', 5, 0], ['S_ZIP_EXT', 'C', 4, 0]]
>>> shapes = sf.shapes()
>>> len(shapes)
370097
>>> records = sf.records()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "shapefile.py", line 315, in records
    r = self.__record()
  File "shapefile.py", line 286, in __record
    y, m, d = int(value[:4]), int(value[4:6]), int(value[6:8])
ValueError: invalid literal for int() with base 10: ''

The constructor for `Editor()` optionally takes a `shapefile` param. As 
currently written,

* If no `shapefile` specified, we call the `Writer()` super constructor as if 
we're creating a new shapefile
* If `shapefile` is specified by name, and it exists, we call the `Writer()` 
super constructor, and populate it with the contents of the shapefile read by a 
`Reader() ` instance
* If `shapefile` is specified by name but it does not exist, the `Writer()` 
super constructor is never called and the `_shapes`, `fields`, and `records` 
members never get initialized, causing things to blow up on the first call to 
`point()`.

Trivial patch attached to call the `Writer()` constructor in this case.

What steps will reproduce the problem?
try to save an empty shape

w = shapefile.Writer(shapefile.POLYGON)
w.field('ID','N','16')
w.field('Name','C','40')
w.save('polygon.shp')


What is the expected output? What do you see instead?
get an empty shapefile
get an error because pyshp is trying to calculate the bbox of nothing

File "shapefile.py", line 529, in __bbox
    return [min(x), min(y), max(x), max(y)]
ValueError: min() arg is an empty sequence

What version of the product are you using? On what operating system?
Windows Vista SP2
Python 2.6.5
Pyshp 1.1.4

I was using pyshp today on a set of test shapefiles.  When I copied my test 
shapefiles into the test folder, I missed one of the .shx files.  I received an 
exception when the code attempted to access one of the shapefiles.  Summarized, 
my code looked like this:

import shapefile
reader = shapefile.Reader(r'D:\work\code\test\streets.shp')
reader.shape(0)

I received the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "shapefile.py", line 220, in shape
    offset = self.__shapeIndex(i)
  File "shapefile.py", line 196, in __shapeIndex
    f = self.__getFileObj(self.shx)
  File "shapefile.py", line 115, in __getFileObj
    raise ShapefileException("Required file not available.")
shapefile.ShapefileException: Required file not available.

Now, technically, there is nothing wrong with this message.  I'm dealing with 
an "exceptional case" and an Exception was thrown in this case.  I just wish it 
was a bit more specific about which file was missing.  So, I patched 
shapefile.py to report the issue more cleanly for the missing shx case.

I have attached a diff with my change.

Joseph Armbruster

What steps will reproduce the problem?
1. Download the dataset from 
http://www1.toronto.ca/wps/portal/open_data/open_data_item_details?vgnextoid=9ec
d5f9cd70bb210VgnVCM1000003dd60f89RCRD&vgnextchannel=6e886aa8cc819210VgnVCM100000
67d60f89RCRD
2. Run this sample program to print all of the records

import shapefile
sf = shapefile.Reader("TCL3_Centreline_OD_bikeways")
shapes = sf.shapes()
records = sf.records()

for index, i in enumerate(shapes):
    print records[index]

3. search through the CP_TYPE column for data. I grepped the output of my 
python program for the work bike
python test.py | grep -i "bike"

What is the expected output? What do you see instead?
The CP_TYPE column should have data like "Bike Lanes", "Signed Routes", 
"Suggested On-Street Routes", ... but it is always empty

What version of the product are you using? On what operating system?
OSX Snow Leopard 
Python 2.6.1
pyshp 1.0.5
Bikeway data was updated March 2011

I get the expected output using other code like this guys 
http://indiemaps.com/blog/2008/03/easy-shapefile-loading-in-python/

What steps will reproduce the problem?
1. Install attached file.
2. Run

from shapefile import Reader
rdr = Reader('brownfield_sites')
for s in rdr.shapes():
   print s.shapeType

What is the expected output? What do you see instead?

Expect to see list of shapeType constants, instead see exception:

Traceback (most recent call last):
  File "LoadGISFeatures.py", line 27, in <module>
    load_file('X:/IT/Florida/brownfield_sites/brownfield_sites')
  File "LoadGISFeatures.py", line 20, in load_file
    for shape in rdr.shapes()[:10]:
  File "C:\Python27\lib\site-packages\pyshp-1.1.4-py2.7.egg\shapefile.py", line
310, in shapes
    shapes.append(self.__shape())
  File "C:\Python27\lib\site-packages\pyshp-1.1.4-py2.7.egg\shapefile.py", line
239, in __shape
    record.parts = _Array('i', unpack("<%si" % nParts, f.read(nParts * 4)))
OverflowError: Python int too large to convert to C long

What version of the product are you using? On what operating system?

v 1.1.4 on Windows 7 w/ Python 2.7

Shapefiles are unprojected data.  For many users this means an extra step after 
pyshp has run.  A few words about common projections and what "good" coordinate 
data looks like will help.

I don't expect pyshp to handle projections, but guiding reads about the issue 
and offering some tips would be nice
http://trac.osgeo.org/proj/
http://wiki.openstreetmap.org/wiki/Ogr2osm

What steps will reproduce the problem?
1. Creating a temporary shapefile using some function in arcpy
2. Use shapefile to read the shapefile
3. Try to delete/overwrite the shapefile with some arcpy function

What is the expected output? What do you see instead?
Need to some clear the shapefile from memory having read in shapes and records 

e.g. 
sfr = shapefile.Reader(some_file_name)
Recs = sfr.records()
Shapes = sfr.shapes()
#insert sfr.clear() or sfr.close() or something similar here

What version of the product are you using? On what operating system?
1.1.6-beta, Windows 7

Please provide any additional information below.

What steps will reproduce the problem?
1. Download a shapefile of 100+ mb
2. Run the .shapes() method

What version of the product are you using? On what operating system?
Mac OSX Lion, pyshp 1.1.6-beta

Please provide any additional information below.

Created an iterator class within the reader class, to allow for large file 
processing. Method to use iterator is Reader.shapes_iter()

What steps will reproduce the problem?
1. Unzip the attached zip file.
2. reader = Reader('bedrock.shp')
3. reader.shape(2)
4. reader.record(2) => generates error

What is the expected output? What do you see instead?
The attached shapefile is missing the last record in the .dbf.  I think the 
code should compare the actual file size against the expected file size
using fields in the .dbf header (fs = headerlen + nrecords*recordlen).  If they 
don't match, throw an exception from the __init__() method.

What version of the product are you using? On what operating system?
1.1.4, on Mac OS 10.7.5

Please provide any additional information below.

What steps will reproduce the problem?
1. Use shapefile.py for parse my shapes.
2. Got an error.
3. Make a small fix in shapefile.py
4. Profit :)

What is the expected output? What do you see instead?
I see
Traceback (most recent call last):
...
  File "shp2sql.py", line 99, in __init__
    self.sfo = VShapeRecords(shpFileName)
  File "shp2sql.py", line 21, in __init__
    self.sfo = shapefile.Reader(filename)
  File "c:\d\code\shape\shapefile.py", line 88, in __init__
    self.load(shapefile)
  File "c:\d\code\shape\shapefile.py", line 108, in load
    self.__dbfHeader()
  File "c:\d\code\shape\shapefile.py", line 250, in __dbfHeader
    fieldDesc[name] = fieldDesc[name][:fieldDesc[name].index("\x00")]
ValueError: substring not found


What version of the product are you using? On what operating system?
Last from CVS.

Please provide any additional information below.

If in file shapefile.py in method
def __dbfHeader(self):
change line (250)
from
fieldDesc[name] = fieldDesc[name][:fieldDesc[name].index("\x00")]
to
fieldDesc[name] = fieldDesc[name].replace('\0', '')

all works fine for me.
Sample shape in attach.

Are island polygons/polygons with holes supported? If so, please add an example 
to the wiki. I've been trying to create such with pyshp, but didn't manage.

Instead of loading the shapefile, an error arises:
Traceback (most recent call last):
  File "/[path]/import_shapefile.py", line 115, in <module>
    loadShapefile()
  File "/[path]/import_shapefile.py", line 19, in loadShapefile
    records = sf.shapeRecords()
  File "/[path]/shapefile.py", line 430, in shapeRecords
    for rec in zip(self.shapes(), self.records())]
  File "/[path]/shapefile.py", line 413, in records
    r = self.__record()
  File "/[path]/shapefile.py", line 389, in __record
    value = u(value)
  File "/[path]/shapefile.py", line 53, in u
    return v.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc6 in position 1: invalid 
continuation byte


Using:
pyshp version: 1.1.4
Ubuntu 12.04
Python 3.2.3 from Ubuntu 12.04 repos

For me it works using the following patch:
==============
@@ -50,7 +50,7 @@
     if PYTHON3:
         if isinstance(v, bytes):
             # For python 3 decode bytes to str.
-            return v.decode('utf-8')
+            return v.decode('iso-8859-1')
         elif isinstance(v, str):
             # Already str.
             return v
==============

A better solution, I think, would be to make it possible for the user to define 
custom encoding when loading shapefile using arguments.

PointZ records are not read in properly. To fix this line 188 in shapefile.py 
should read

185     if shapeType == 11:
186         record.z = unpack("<d", f.read(8))
187     # Read a single M value
188     if shapeType == 11 or shapeType == 21:
189         record.m = unpack("<d", f.read(8))

This is documented on page 15 of the ESRI white paper.

http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf

What steps will reproduce the problem?
1. Load the attached shapefile using the Reader class.
2. Count the number of fields in the Reader and then in any record.
3.

What is the expected output? What do you see instead?
This file contains two fields called "ASIA2000_2".  One of these fields gets 
dropped by the reader.  The ArcGIS "solution" to this problem is to change the 
name of the duplicate field.  Why it lets users create data with duplicate 
fields is beyond me.

What version of the product are you using? On what operating system?
1.1.4, on Mac OS 10.7.5

Please provide any additional information below.
The version of shapefile I am using is embedded in the Basemap library 
(http://matplotlib.org/basemap/).  It might be helpful if the author of that 
library is notified when this package is updated.

I hacked my own solution - basically doing the same thing that ArcGIS is doing. 
 I'd include that but I'm sure you can come up with something a little cleaner.

This code:
import shapefile
sf = shapefile.Editor('/home/jorge/temp/42MUE250GC_SIR.shp')
shapes = sf.shapes()
for s in shapes:
    print( s.shapeType)
    for p in range(1):#len(s.points)):
        s.points[p][0] = -s.points[p][0]
sf.save('/home/jorge/temp/x.shp')

give all shapetype = 25 and a error:
"  File 
"/usr/local/lib/python2.7/site-packages/pyshp-1.1.4-py2.7.egg/shapefile.py", 
line 687, in __shpRecords
    [f.write(pack("<d", p[3])) for p in s.points]
IndexError: array index out of range
"
But my shapefile have only 2D objects, there is no exists p[2] or p[3]

Another code, using gdal/osgeo that read the same shapefile, give me that most 
objects are POLYGON' and a few are MULTIPOLYGON

What is the expected output? What do you see instead?
Why the two libraries have different results for the geometry type for the same 
file?



What version of the product are you using? On what operating system?
pyshape 1.1.4 and openSUSE 12.2 (x86_64)


Please provide any additional information below.
This code is a simple test. I'm trying to convert a 2D file in a 3D file.
I receive a shapefile of highways and need to add the elevation data to all 
points.

What steps will reproduce the problem?
1. Get a really large shapefile.
2. Load it in memory.
3. Now try to write it to a file.

What is the expected output? What do you see instead?
The output is exactly as expected, but the memory requirements of the program 
are very high. 
There is no good way of incrementally appending data to an output file. The 
save method can be provided file handlers instead of file names, but it does a 
seek(0) in the __*Header methods and then does a seek(100) in the __*Records 
method and over-rides any data already written to the output file. An "append" 
type version of the save method would be very useful for dealing with large 
files.

Deleting of the first line does not work. (nothing happend)

e = shapefile.Editor(shapefile="point")
e.delete(0)
e.save('point')


When deleting a line between first and last line, the shape is deleted, but the 
record is still there and so all records are shifted

e = shapefile.Editor(shapefile="point")
e.delete(1)
e.save('point')

( the old shape 2 has attributes of the old shape 1)


Environment:
Windows Vista SP2
Python 2.6.5
Pyshp 1.1.4

What steps will reproduce the problem?
1. Unzip attached file
2. Run:

from shapefile import Reader)
rdr = Reader('dryclean_sp_vw')
for s in rdr.shapes():
   print s.shapeType, s.points

What is the expected output? What do you see instead?

Expect to see valid shapeType and a single point on each output line.

Actually see:

21          [[526856.2550000001, 426634.1260000001]]
1092619666  []
50331648    []
-2104533974 []
1092198249  []
21          [[524895.9480000002, 428961.89000000013]]
1092603877  []
100663296   []
498216209   []
1092281351  []

What version of the product are you using? On what operating system?

V 1.1.4 on Windows 7 w/ python 2.7.

What steps will reproduce the problem?
1.Created keyed dictionary 
sfRecord = {"num1":"bob","num2":"bob2","num3":"bob3"}
2.Add it to the writer
w.field("field1")
w.field("field2")
w.field("field3")
w.record(recordDict=sfRecord)

What is the expected output? What do you see instead?
print sf.records
[]

You should see something.
The issue is in def record(self, *recordList, **recordDict):
...
elif recordDict:
                                for field in self.fields:
                                        if recordDict.get(field[0], None):
                                                record.append(recordDict[field[0]])
Here, recordDict is a keyed object with 'recordDict' as the only keyed object 
in it.
So, replacing recordDict above with recordDict['recordDict'] will get you an 
output.

However, this will still cause an issue where if the key is present but the 
value isn't, the value isn't appended. This causes the later fields to shift.
So an input of this:

sfRecord = {"num1":"bob","num2":"","num3":"bob3"}

will result in an array in record like this:

["bob","bob3"], which is wrong.

Instead we could rewrite the whole block to look something like this (hopefully 
a little more elegant with the names):

recordDict=recordDict['recordDict']
for field in self.fields:
    if field[0] in recordDict:
        record.append(recordDict[field[0]])

Support was added to save to file like objs recently.

This is good, but I need to load not save from file like objs (they are 
compressed, but could also come from network - no file).

It would be nice to have load / save consistent and be simple 1 line.

Suggested solution in attached patch along with doctests for them.

Here's my code:

import shapefile as sf
w = sf.Writer(sf.POINT)
w.point(37.7793, -122.4192)
w.field('FIRST_FLD')
w.record('First','Point')
w.save('test/point')

...it creates point.dbf, point.shp, and point.shx, but it does not create 
point.prj

Since the projection file is not generated I cannot open it in Google Earth Pro

If the Python `zip` function is used to combine X and Y coordinates, the result 
is a list of tuples:

X = [1, 5, 5, 3, 1]
Y = [5, 5, 1, 3, 1]
coors = zip(X, Y) # list of tuples


However, this object cannot be used by the shapefile.py module:

import shapefile
w = shapefile.Writer(shapefile.POLYGON)
w.field('MYFIELD','C')
w.poly([coors])
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "shapefile.py", line 684, in poly
    point.append(0)
AttributeError: 'tuple' object has no attribute 'append'


To fix this, I've added a check to shapefile.py near line 679:
# Ensure point is list
if not isinstance(point, list):
        point = list(point)

See attached patch. I've also removed trailing whitespace on various lines in 
this patch.

What steps will reproduce the problem?
1. Open a shape file with a date field that has an empty string for the day - 
example: geocommunicator.gov township files


I wrote a simple patch to split on a regex and fail silently if the 
day/month/year cannot be coerced to int.

What steps will reproduce the problem?
1. Load example shapefile from 
http://gis.stackexchange.com/questions/50264/how-to-get-text-attributes-from-sha
pefile-with-python-pyshp/
2. r = shapefile.Reader(
3. len(r.records())

What is the expected output? What do you see instead?

len(r.records()) is 0, I expected 59117.

Please provide any additional information below.

As I noted in my answer on 
http://gis.stackexchange.com/questions/50264/how-to-get-text-attributes-from-sha
pefile-with-python-pyshp/ the DBF file is invalid. It uses an illegal value for 
the DBF deleted flag. Pyshp should either ignore this (I've never seen a DBF 
file in Shapefile context with actual deleted fields) or raise an exception 
upon illegal value.

Perhaps raise an exception by default and provide a lenient mode that ignores 
the flag?

With Shapefile r28 (for Python2.x), I cannot seem to write a simple 3D 
Shapefile:

Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import shapefile
>>> sf = shapefile.Writer(shapefile.POINTZ)
>>> sf.field('RecID')
>>> sf.point(100.0, 200.0, 300.0)
>>> sf.record(1)
>>> sf.save('foo.shp')
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "I:\somepath\shapefile.py", line 765, in save
    self.saveShp(target)
  File "I:\somepath\shapefile.py", line 738, in saveShp
    self.__shapefileHeader(self.shp, headerType='shp')
  File "I:\somepath\shapefile.py", line 486, in __shapefileHeader
    f.write(pack(">i", self.__shpFileLength()))
  File "I:\somepath\shapefile.py", line 372, in __shpFileLength
    nParts = len(s.parts)
AttributeError: _Shape instance has no attribute 'parts'

Hi,

With both python 2 / 3 branches separate they have to be maintained and get out 
of sync with the duplication.

Suggest keeping the 1 codebase and make it work with both python 2 and python 
3.  This is becoming quite a common approach and if it is suitable for pyshp 
then it means only 1 codebase and also only 1 package on pypi that works for 
all python versions.

I may be able to help with this if there is interest.

Thoughts?

:)

What steps will reproduce the problem?
1.Add point data in a non-number format (this is not very smart when doing it 
by hand, but easy to do when parsing data from an input file):
sf.point('12.23','14.24') 

2.Attempt to save the shapefile:
sf.save('shapefile')

3.See error that doesn't make a lot of sense:
 File "../../py_libs/pyShp/shapefile.py", line 492, in __shapefileHeader
    f.write(pack("<4d", *self.bbox()))
struct.error: required argument is not a float

It would be nice if when adding a point a legible error was thrown if the data 
was in a format that would work (such as accidentally adding numbers that are 
encoded as string values from a csv file)

Python's rjust and ljust don't seem to truncate, so, around line 735 (and 739) 
in shapefile.py, instead of:
    value = str(value).rjust(size)
I believe you need:
    value = str(value).rjust(size)[:size]

What steps will reproduce the problem?
1. attempt to write floats with higher precision than that alloted in the dbf 
field. 
2.
3.

What is the expected output? A DBF. What do you see instead? An assertion error.

What version of the product are you using? On what operating system?
1.1.4 on Windows XP.

Please provide any additional information below.

Thanks!

This isn't a bug report.  The pyshp library is awesome!  I actually just have a 
few suggestions for the pyshp code.  

The first is to allow a list of coordinates when writing points:

Currently you must do this:

    w = shapefile.Writer(shapefile.POINTZ)
    coordList = [-80, 40, 300]
    x = coordList[0]
    y = coordList[1]
    z = coordList[2]
    w.point(x,y,z)

I made changes to the code so I can do this:

    w = shapefile.Writer(shapefile.POINTZ)
    coordList = [-80, 40, 300]
    w.point(coordList)

Here are the edits to the pyshp code for allowing lists of coordinates:

    def getValue(self, argList, argNum):
        try:
            return argList[argNum]
        except IndexError:
            return 0

    def point(self, x, y=None, z=0, m=0):
        """Creates a point shape."""
        if type(x) is list:
            truex = self.getValue(x, 0)
            y = self.getValue(x, 1)
            z = self.getValue(x, 2)
            m = self.getValue(x, 3)
            x = truex
        elif type(x) is not list and y == None:
            raise ShapefileException('A y-value was not supplied for the point')
        pointShape = _Shape(self.shapeType)
        pointShape.points.append([x, y, z, m])
        self._shapes.append(pointShape)



My second suggestion is to allow a list or dict to be used for writing records:

Currently you must do this:

    w = shapefile.Writer(shapefile.POINTZ)
    fieldNames = ['longitude','latitude','elevation']
    makeFields = [ w.field(field) for field in fieldNames ]
    w.point(-80,40,300)
    fieldValues = ['-80','40','300']
    longitude = fieldValues[0]
    latitude = fieldValues[1]
    elevation = fieldValues[1]
    w.record(longitude,latitude,elevation)

I made changes to the code so I use a list instead of multiple arguments:

    w = shapefile.Writer(shapefile.POINTZ)
    fieldNames = ['longitude','latitude','elevation']
    makeFields = [ w.field(field) for field in fieldNames ]
    w.point(-80,40,300)
    fieldValues = ['-80','40','300']
    w.record(fieldValues)

or use a dict:

    w = shapefile.Writer(shapefile.POINTZ)
    fieldNames = ['longitude','latitude','elevation']
    makeFields = [ w.field(field) for field in fieldNames ]
    w.point(-80,40,300)
    fieldValues = {'longitude': '-80', 'latitude': '45', 'elevation': '300'}
    w.record(fieldValues)


Here are the changes I made to allow for lists or dicts but still allow 
individual args:

    def record(self, *recordList, **recordDict):
        """Creates a dbf attribute record. You can submit either a sequence of
        field values or keyword arguments of field names and values. Before
        adding records you must add fields for the record values using the
        fields() method. If the record values exceed the number of fields the
        extra ones won't be added. In the case of using keyword arguments to specify
        field/value pairs only fields matching the already registered fields
        will be added."""
        record = []
        fieldCount = len(self.fields)
        # Compensate for deletion flag
        if self.fields[0][0].startswith("Deletion"): fieldCount -= 1
        if recordList and type(recordList[0]) is list:
            [record.append(recordList[0][i]) for i in range(fieldCount)]
        elif recordList and type(recordList[0]) is not list and type(recordList[0]) is not dict:
            [record.append(recordList[i]) for i in range(fieldCount)]
        elif recordList and type(recordList[0]) is dict or recordDict:
            recordDict = recordList[0] if recordList and type(recordList[0]) is dict else recordDict
            for field in self.fields:
                if field[0] in recordDict:
                    val = recordDict[field[0]]
                    if val:
                        record.append(val)
                    else:
                        record.append("")
        if record:
            self.records.append(record)



I tested these changes to make sure they don't break the functionality of 
passing in individual arguments, but I was using single file to test and could 
have easily overlooked something.

I used these pyshp changes in a GPX waypoints to SHP script I wrote.  It allows 
me to pass lists to the point and record methods while looping through the 
waypoints.

I think these changes would be beneficial for others and they don't add too 
much to the existing code.

Thanks for the great work.

What steps will reproduce the problem?
1.keep appending features to the shape file

What is the expected output? What do you see instead?
The writing time should not increase that much. 

What version of the product are you using? On what operating system?
1.14. Windows

Please provide any additional information below.

A simple testing example is as following:

----------------------------------------------------------------
def append():
    start = datetime.now()
    w = shapefile.Editor(shapefile=r'E:\data\shape.shp')
    w.point(90.3, 30)
    w.record('First', 'Point', datetime.now(), '123213212.22')
    w.save(r'E:/data/shape')
    print(str(i) + ': ' + str((datetime.now() - start).total_seconds()))

for i in range(10000):
    append()
----------------------------------------------------------------

The writing speed keep declining as more features are written, is there any way 
to improve this? I noted in the code, when opening a Editor:

-----------------------------------------
 elif is_string(shapefile):
            base = os.path.splitext(shapefile)[0]
            if os.path.isfile("%s.shp" % base):
                r = Reader(base)
                Writer.__init__(self, r.shapeType)
                self._shapes = r.shapes()
                self.records = r.records()
                self.fields = r.fields
---------------------------------------------

I assume that the writer basically reads all the data into the cache, and then 
rewrite them together with the appended data when "save" is called. Am I right? 
This makes it a real problem after several thousand of records are stored. 

I expect a way that can use the "append" mode to write new features. Is this 
possible?

What steps will reproduce the problem?
1.read a empty .dbf file


What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
version 1.1.4 in windows xp

Please provide any additional information below.

What steps will reproduce the problem?
s = shapefile.Reader('<none_existing_file>.shp')
shapes = s.shapes()

What is the expected output? What do you see instead?
Expect an exception stating the file does not exists
Get:
    225         def shapes(self):
    226                 """Returns all shapes in a shapefile."""
--> 227                 self.shp.seek(100)
    228                 shapes = []
    229                 while self.shp.tell() < self.shpLength:

AttributeError: 'NoneType' object has no attribute 'seek'

What version of the product are you using? On what operating system?
1.06
Python 2.6
Ubuntu Linux 10.04 64 bit

Please provide any additional information below.

Checking syntax of files using (e.g. in the Python3 directory):
> python -c "import shapefile3"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "shapefile3.py", line 772
    def __init__(self, shapefile=None, shapeType=POINT, autoBalance=1):
                                                                      ^

I've fixed up the indentation to consistently use tabs. I've also removed any 
extra whitespace and the end of lines, and corrected some spelling mistakes.

What steps will reproduce the problem?

1. open (Reader) a shapefile (with N elements)
2. call the "shape" method with the value -N (which is valid !)

The expected value is the first (index 0) shape object but an exception is 
raised in __restricted index after the test at line 194 (version 1.1.4).

The code below fix the bug

def __restrictIndex(self, i):
    """Provides list-like handling of a record index with a clearer
    error message if the index is out of bounds."""
    if self.numRecords:
        if i < -self.numRecords or self.numRecords <= i:
            raise IndexError("Shape or Record index out of range.")
        if i < 0: i += self.numRecords
    return i

What steps will reproduce the problem?
1.Open a file named "a.b.shp" with the editor

What is the expected output? What do you see instead?
The file a.b.shp should be opened. But the editor class
uses:

base = os.path.splitext(shapefile)[0]
            if os.path.isfile("%s.shp" % base):
                r = Reader(base) """here"""
                Writer.__init__(self, r.shapeType)
                self._shapes = r.shapes()
                self.fields = r.fields
                self.records = r.records()

So "a.b" is sent to the reader constructor instead of "a.b.shp".

The reader constructor then uses the splitext function again and sees
it as a file named "a" with a ".b" extension, instead of a file
named "a.b" with a ".shp" extension.

A solution is to replace:
                r = Reader(base)
by
                r = Reader(shapefile)

(line 877)

Handling date fields (type 'D') is currently done awkwardly and incorrectly in 
pyshp.

Try writing a shapefile with a 'D' type field, re-opening it for with an 
`Editor`, and saving the file: the date field becomes corrupted because pyshp 
has written it's value as a string version of [2012,02,18], which is "[2012,02".

Issue 6 <http://code.google.com/p/pyshp/issues/detail?id=6&can=1> suggested 
regex parsing of date fields, falling back to parsing out positions, but the 
fact is that this is unnecessary. The DBF format 'D' field must be exactly 8 
characters, with date stored as YYYYMMDD; no delimiters, no ambiguity with size.

I've attached a patch (unified diff format) against current trunk (r72) which 
does the following:
* overrides column size when 'D' type is specified so that it is always exactly 
8 characters
* marshalls to and from the actual Python datetime.date type; no string 
formatting or arrays-of-integers for dealing with dates

What steps will reproduce the problem?
1. Create a POINTZ shapefile using a `Writer()`
2. Attempt to append to the shapefile using an `Editor()`, specifying the 
filename in Editor constructor
3. Write fails with the following exception:

{{{
Traceback (most recent call last):
  File "/Users/driggs2/workspace/batutils/batutils/transect/tkgui_pro.py", line 270, in go
    transect.main()
  File "/Users/driggs2/workspace/batutils/batutils/transect/app.py", line 281, in main
    self._finalize()
  File "/Users/driggs2/workspace/batutils/batutils/transect/app_pro.py", line 90, in _finalize
    self.w.save(self.outfile)
  File "/Users/driggs2/workspace/batutils/batutils/transect/shapefile.py", line 862, in save
    self.saveShp(target)
  File "/Users/driggs2/workspace/batutils/batutils/transect/shapefile.py", line 829, in saveShp
    self.__shapefileHeader(self.shp, headerType='shp')
  File "/Users/driggs2/workspace/batutils/batutils/transect/shapefile.py", line 596, in __shapefileHeader
    raise ShapefileException("Failed to write shapefile elevation and measure values. Floats required.")
ShapefileException: Failed to write shapefile elevation and measure values. 
Floats required.
}}}

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
  pyshp 1.1.4
  Python 2.6.6 (Apple)
  Mac OS X 10.6.8

Please provide any additional information below.

Writing the shapefile as POINT rather than POINTZ works correctly.

The above exception itself is due to the Editor's Writer trying to write an "m" 
value of None. For some reason, `Editor.point()` has a different method 
signature than `Writer.point()`, and the former assigns a default value of None 
for "m" and "z", while the latter assigns default value of 0. Changing the 
method signature of `Editor.point()` to use default values m=0, z=0 appears to 
solve the above exception, but still fails later with:

{{{
Traceback (most recent call last):
  File "/Users/driggs2/workspace/batutils/batutils/transect/tkgui_pro.py", line 270, in go
    transect.main()
  File "/Users/driggs2/workspace/batutils/batutils/transect/app.py", line 281, in main
    self._finalize()
  File "/Users/driggs2/workspace/batutils/batutils/transect/app_pro.py", line 93, in _finalize
    self.w.save(self.outfile)
  File "/Users/driggs2/workspace/batutils/batutils/transect/shapefile.py", line 862, in save
    self.saveShp(target)
  File "/Users/driggs2/workspace/batutils/batutils/transect/shapefile.py", line 830, in saveShp
    self.__shpRecords()
  File "/Users/driggs2/workspace/batutils/batutils/transect/shapefile.py", line 699, in __shpRecords
    f.write(pack("<1d", s.points[0][2]))
IndexError: array index out of range
}}}

Again, I solved my own immediate problem by using POINT type rather than 
POINTZ, but it looks like pyshp may have some more bugs related to editing 
POINTZ types.

Hi,

I saw the usage.py file used to get installed which was a bit of a generic 
name.  Now that been changed to pyshp_usage.py, so it's more specific which is 
good.

Installed it looks like

ls -1 e1/lib/python2.7/site-packages/

pyshp_usage.py
pyshp_usage.pyc
shapefile.py
shapefile.pyc

The pyshp_usage.py is not really used though since it is just a docstring in 
rst format that has examples run by doctest.

Suggest that pyshp_usage.py could be changed to a README or README.txt file 
with the doc in rst format.  doctest should be able to run on that file too, 
might just have to adjust a bit.

Advantage with this is that the README file will be packaged as part of the 
sdist (source distribution) but won't be installed into 
lib/python2.7/site-packages/.

Thoughts?

:)

The logic of line 808:

                if field[0] in recordDict:
                    val = recordDict[field[0]]
         -->        if val:
                        record.append(val)
                    else:
                        record.append("")

...doesn't allow the writing of zeros to numeric fields.

There may be a more robust way to do things, but 'if val != "":' works for me.

What steps will reproduce the problem?
1. try to write a zero to a numeric field

What version of the product are you using? On what operating system?
1.1.4 on winXP

Please provide any additional information below.
Thanks!

What steps will reproduce the problem?
1. Create a polygon that is not closed.  
'e.poly(parts=[[[5.1,5],[9.9,5],[9.9,1],[7.5,3],[5.1,1]]])'

What is the expected output? What do you see instead?
A complete shapefile.  IMO, the best solution would be for the polygon creation 
to test if last point is coincident with the first, and if it is do nothing, if 
not append the first point to close the polygon.  Another solution would be to 
make sure the examples all close.

What version of the product are you using? On what operating system?
1.1.4 from PyPi.  Linux.

Please provide any additional information below.
The overall appearance in ArcGIS looks correct.  There is a line closing the 
polygon.  The problem though is that the selection of polygons is strange.

By the way - Thanks for a great bit of software!

What steps will reproduce the problem?
1.Run the program enter 100 for total, then 10 for bet
2.Keep hitting enter until error appears
3.

What is the expected output? What do you see instead?
expected is in the slots attachment


What version of the product are you using? On what operating system?
python 3.2, windows 7


Please provide any additional information below.
This is a project due tomorrow, need help asap.

If you have a DBF field that has garbage after the terminating NULL, the record 
will contain the terminating NULL and the garbage, even if it is defined as 
something else than C (F for instance).

The expected behavior is to end the string at the NULL, disregard anything 
after the NULL and do the correct conversion based on field type.

What steps will reproduce the problem?
1. Load shapefile with column name that has maximum allowed number of 
characters (then there is no \x00 behind the column name).
2. .index("\x00") raises ValueError

What is the expected output? What do you see instead?


Expected behaviour: read shapefile with such a column name


What version of the product are you using? On what operating system?


Latest download provided.


Please provide any additional information below.

These lines (around 250) should fix the problem (not thoroughly tested, though):

                        if "\x00" in fieldDesc[name]:
                                indx = fieldDesc[name].index("\x00")
                        else:
                                indx = len(fieldDesc[name]) - 1
                        fieldDesc[name] = fieldDesc[name][:indx]

replacing the line

fieldDesc[name] = fieldDesc[name][:fieldDesc[name].index("\x00")]

What steps will reproduce the problem?
1. >> e = shapefile.Editor(shapefile="shapefiles/test/polygon.shp")
2. >> print "No. of shapes: %i" % len(e.shapes())
3. >> e.delete(0)
4. >> print "No. of shapes: %i" % len(e.shapes())

What is the expected output? What do you see instead?

I expect the number of shapes to be reduced by one after the delete method is 
called. Instead the number of shapes remains the same.

What version of the product are you using? On what operating system?

Using shapefile-1.1.6-b.py under Python 2.7 under linux (Fedora 14).

Please provide any additional information below.

The problem is caused by the way that the delete method determines whether 
points, shapes, or parts are to be deleted.  This logic fails because Python 
evaluates 0 shape index as "False" when it applies the filtering logic.  The 
attached patch fixes the problem.

What steps will reproduce the problem?
1. sf = shapefile.Reader(u'C:\\Test\\MyFile.shp')

What is the expected output? What do you see instead?

Expected to get a new Reader class. Instead get:

ShapefileException("Shapefile Reader requires a shapefile or file-like object.")

What version of the product are you using? On what operating system?

version: 1.1.4
Seems to be the same code in the trunk. 

Please provide any additional information below.

It fails due to the type checking in the __init__ function of the reader, as a 
string and unicode strings are different types. 
Is type checking necessary at all here?

if type(args[0]) is type("stringTest"): 

>>> type("hello")
<type 'str'>
>>> type(u"hello")
<type 'unicode'>