Unknown format code 'd' for object of type 'float'

[etiennethomassen]

Updating pyshp from 1.2.0 tot 1.2.11 results in the following error when trying to write a shapefile: ValueError:Unknown format code 'd' for object of type 'float'

python3.6/site-packages/shapefile.py in __dbfRecords
935.  value = format(value, "d")[:size].rjust(size) # caps the size if exceeds the field size

[karimbahgat]

This seems to happen when you add attempt to add a float value to an integer field (ie decimal=0), so it expects but fails to format it as an integer. In v1.2.10 this went unnoticed since we just forced the value to string.

In one way the error is doing its job, but the lib should also do some autocorrecting, so I think we can add force conversion of floats to ints. Will look at adding a fix.

[jesegal]

I'm experiencing a similar issue, after upgrade from 1.2.10 to 1.2.11, writing a shapefile:

Unknown format code 'd' for object of type 'str'

using python 2.7.13

[jesegal]

pyshp 1.2.11 seems to introduce incompatibility with the prior documented method of specifying the shapefile attributes. While this version does make more sense (using numeric for field length), it is not compatible with the previously documented specification, string for the length, numeric for the precision.

This probably should have been a major release rather than a bug fix.
Or at least maintain compatibility with the previous field specification.

[karimbahgat]

@jesegal, for your first comment could you share the code that produces the error? As in my previous answer, the error you mention seems to be caused by adding a string value to a field defined as "N" with decimal=0. The fact that this went unnoticed in the previous version was a bug, this simply fixes it, and notifies the user as it should. Since several have now raised this issue, do you think perhaps we should explicitly test for the value mismatch and raise a more informative exception?

As for you second comment, you seem to be referring to another issue unrelated to this one: a change in how we define fields, that the README used to specify the "size" arg of the field() method with a string-number, whereas now it is specified with a number, i.e. field("Field1", "C", "30") VS field("Field1", "C", 30)? It's true that the documentation changed, but both in the previous version and the current version it was acceptable to use either a string-number or number for the "size" arg, since internally its value is forced to an int. So there shouldn't be any backwards incompatibility, it's just a change in the docs. Did I understand you correctly on this point?

Regardless, your point about major version incrementing for backwards incompatible changes is well taken, and the upcoming version will indeed increment the major version due to some other changes in the code.

[jesegal]

@karimbahgat, Using shapefiles with field specifications, some of them as shown below.
It seems the Numeric fields are causing the issue, where it was possible to specify the field width using a string until version 1.2.11.

Yes, it makes more sense to specify this as a number, but the documentation clearly specified it as a string, until version 1.2.11.

	('LOCN_TYPE', 'C', '10'),
	('LAT', 'N', '14', 8),
	('LON', 'N', '14', 8),

This field is failing: ('ID', 'N', '8')

It's failing at the following statement:
w.save(sf)
...and stepping into the library, it's failing at:
value = format(value, "d")[:size].rjust(size) # caps the size if exceeds the field size

in my scenario, at this line of code
value is a str: '0'

which causes this to fail.

This worked in 1.2.10

[karimbahgat]

Ok, so it seems you indeed are encountering the same issue as @etiennethomassen. The issue is with the value you are supplying to the field when you call record(*values). The error says it all: You are trying to insert a string value (value is a str: '0') into your ID field which is defined as numeric. Instead of inserting the number 0 you are inserting it as a '0' string.

To make it more concrete, this does not work:

w = shapefile.Writer()
w.field('ID', 'N', '8')
w.record("0")
w.null()
w.save("test.shp")

But changing w.record("0") to w.record(0) works.

So your error has nothing to do with how you define the field. As I said previously, the field width change was only in the docs, the library accepts it as either a string or a number. You can write it like this w.field('ID', 'N', 8), or like this w.field('ID', 'N', '8'), both works.

It worked in 1.2.10, but only because it was a bug. Passing strings to a numeric field really shouldn't be allowed, a bug that was fixed in 1.2.11. I am will probably end up loosening this criteria a little, only throw an error if the string cannot be forced to a number.

[karimbahgat]

Since the issue seems common enough and does break with the more lenient approach of <=1.2.10, it has now been fixed both in the 1.2.x branch and the master branch, and will be included in release 1.2.12 and the upcoming major version 2.0.0.

This should make it compatible with <1.2.10 again and make it easier to write slightly wrong value types, such as float to int, string number to int or float, etc. But attempting to write non-numeric strings to numeric fields will still raise an exception, as it should.

[karimbahgat]

Version 1.2.12 which fixes this issue is now up on PyPI.

[toferkey]

Hi Karimbahgat,

I just wanted to leave a quick comment regarding this issue. I think there is a bug that caused this which still exists. Specifically, with the 'field' method when using floats. For example, in the un-fixed version I create float field like so:

shp.field('lat', 'F')
shp.field('lon', 'F')
lat = 60.47
shp.point(lon,lat)
shp.record(lat, lon)
shp.save('out.shp')

In this example with the previous version I got the same error code as the others mentioned, even though I know this was a float field and float number.

In the new version 1.2.12 I did the same and it writes the field as 60. (Thus dropping the decimal). I believe this is what is causing the error for people. However, of course I can fix the problem like such:

shp.field('lat','F', 5,5)

Thus, it seems users need to explicitly specify the length of floats fields or else it may change it to integer.

[karimbahgat]

Hi klasko2, you raise a good point. This is actually the reason I originally didn't want to force value type conversion to the field type, and instead raise an exception.

As you say, the reason here is that the default decimal length is 0, which is stated in the Readme. This has always been the case, but previous versions of pyshp simply ignored the decimal arg, and wrote the entire floating nr regardless, leading to possibly incorrect files.

So the behavior is correct, but I agree with your point that since the 'F' type is float by definition it should default to using decimals>0. Also with the 'N' numeric type I think a case can be made as you suggest that most users would expect it to default to decimals rather than int. Defaulting to decimals is also less damaging because if people forget to specify field type then at least no information is lost.

I am going to raise this as a separate improvement issue, and we can discuss there whether the default decimal should be changed to >0 and if so how many decimals. Also maybe a discussion of the default field size, and if that should be different for different data types.