-
Notifications
You must be signed in to change notification settings - Fork 30
django: duration arithmetic fails with dates #253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
RemedyJust like @timgraham stated in his issue statement: Test caseGive an existing database, run this program #!/usr/bin/env python3
from spanner.dbapi import connect
def tables_map(cursor):
return {key[0]: True for key in cursor.list_tables()}
def main():
with connect(project='appdev-soda-spanner-staging', instance='django-tests', database='issues-17') as conn:
with conn.cursor() as cursor:
tables = tables_map(cursor)
table_name = 'Experiment'
exists = tables.get(table_name, None)
if False:
cursor.execute('''DROP TABLE Experiment''')
exists = False
conn.commit()
if not exists:
cursor.execute('''CREATE TABLE Experiment (
name STRING(24),
id INT64 NOT NULL,
assigned DATE,
completed DATE,
estimated_time INT64,
start TIMESTAMP,
terminate TIMESTAMP
) PRIMARY KEY(id)''')
conn.commit()
print(tables)
# Clear the table firstly.
cursor.execute('''DELETE FROM Experiment WHERE 1=1''')
# Now insert some data into it.
cursor.execute(
'''INSERT INTO Experiment (id, name, assigned, completed, estimated_time, start, terminate) VALUES (%s, %s, %s, %s, %s, %s, %s)''',
[
2660792221924433694, 'e0', '2010-06-25', '2010-06-25', 0, '2010-06-25T12:15:30.747000Z', '2010-06-25T12:15:30.747000Z',
673743920250785767, 'e1', '2010-06-25', '2010-06-26', 253000, '2010-06-26T12:15:30.747000Z', '2010-06-26T12:15:31.000000Z',
])
cursor.execute('''SELECT
*
FROM
Experiment
WHERE
assigned <
((TIMESTAMP_SUB(completed, INTERVAL %s MICROSECOND)))
ORDER BY
name
ASC''', (345600000000,))
listing = cursor.fetchall()
for item in listing:
print(item)
if __name__ == '__main__':
main() Results...
File "/Users/emmanuelodeke/Library/Python/3.7/lib/python/site-packages/spanner/dbapi/cursor.py", line 112, in execute
raise ProgrammingError(e.details if hasattr(e, 'details') else e)
spanner.dbapi.exceptions.ProgrammingError: 400 No matching signature for function TIMESTAMP_SUB for argument types: DATE, INTERVAL INT64 DATE_TIME_PART. Supported signature: TIMESTAMP_SUB(TIMESTAMP, INTERVAL INT64 DATE_TIME_PART) [at 7:35]\n ((TIMESTAMP_SUB(completed, INTERVAL @a0 MICRO...\n Full investigation:This is what the full statement looks like SELECT
expressions_ExPeRiMeNt.id, expressions_ExPeRiMeNt.name,
expressions_ExPeRiMeNt.assigned, expressions_ExPeRiMeNt.completed,
expressions_ExPeRiMeNt.estimated_time, expressions_ExPeRiMeNt.start,
expressions_ExPeRiMeNt.`end`
FROM
expressions_ExPeRiMeNt
WHERE
expressions_ExPeRiMeNt.assigned <
((TIMESTAMP_SUB(expressions_ExPeRiMeNt.completed, INTERVAL %s MICROSECOND)))
ORDER BY
expressions_ExPeRiMeNt.name
ASC
Args: (345600000000,) and failure now: django.db.utils.ProgrammingError: 400 No matching signature for function
TIMESTAMP_SUB for argument types: DATE, INTERVAL INT64 DATE_TIME_PART. Supported
signature: TIMESTAMP_SUB(TIMESTAMP, INTERVAL INT64 DATE_TIME_PART) [at 1:299]\n
...WHERE expressions_ExPeRiMeNt.assigned < ((TIMESTAMP_SUB(expressions_ExPeRi...\n with relevant Django code at def test_date_minus_duration(self):
more_than_4_days = Experiment.objects.filter(
assigned__lt=F('completed') - Value(datetime.timedelta(days=4), output_field=DurationField())
)
self.assertQuerysetEqual(more_than_4_days, ['e3', 'e4', 'e5'], lambda e: e.name) and the respective model is class Experiment(models.Model):
name = models.CharField(max_length=24)
assigned = models.DateField()
completed = models.DateField()
estimated_time = models.DurationField()
start = models.DateTimeField()
end = models.DateTimeField() which translates into the following DDL CREATE TABLE Experiment (
name STRING(24)
id STRING(32) NOT NULL,
assigned DATE,
completed DATE,
estimated_time INT64,
start TIMESTAMP,
end TIMESTAMP
) PRIMARY KEY(id) and for some population SQL SQL: INSERT INTO expressions_ExPeRiMeNt (id, name, assigned, completed, estimated_time, start, `end`) VALUES (%s, %s, %s, %s, %s, %s, %s)
Args: (2660792221924433694, 'e0', '2010-06-25', '2010-06-25', 0, '2010-06-25T12:15:30.747000Z', '2010-06-25T12:15:30.747000Z')
SQL: INSERT INTO expressions_ExPeRiMeNt (id, name, assigned, completed, estimated_time, start, `end`) VALUES (%s, %s, %s, %s, %s, %s, %s)
Args: (673743920250785767, 'e1', '2010-06-25', '2010-06-26', 253000, '2010-06-26T12:15:30.747000Z', '2010-06-26T12:15:31.000000Z')
SQL: INSERT INTO expressions_ExPeRiMeNt (id, name, assigned, completed, estimated_time, start, `end`) VALUES (%s, %s, %s, %s, %s, %s, %s)
Args: (4474564260460410667, 'e2', '2010-06-22', '2010-06-25', 3600000000, '2010-06-25T12:15:30.747000Z', '2010-06-25T12:16:14.747000Z')
SQL: INSERT INTO expressions_ExPeRiMeNt (id, name, assigned, completed, estimated_time, start, `end`) VALUES (%s, %s, %s, %s, %s, %s, %s)
Args: (2820814987796123967, 'e3', '2010-06-25', '2010-06-30', 76080000000, '2010-06-29T12:15:30.747000Z', '2010-06-30T09:23:30.747000Z')
SQL: INSERT INTO expressions_ExPeRiMeNt (id, name, assigned, completed, estimated_time, start, `end`) VALUES (%s, %s, %s, %s, %s, %s, %s)
Args: (3527083678376937815, 'e4', '2010-06-15', '2010-07-05', 777600000000, '2010-06-25T12:15:30.747000Z', '2010-07-05T12:15:30.747000Z')
SQL: INSERT INTO expressions_ExPeRiMeNt (id, name, assigned, completed, estimated_time, start, `end`) VALUES (%s, %s, %s, %s, %s, %s, %s)
Args: (2875353117097818790, 'e5', '2010-06-25', '2010-10-23', 7776000000000, '2010-07-25T12:15:30.747000Z', '2010-10-23T12:15:30.747000Z') |
Seems like only strings are passed into spanner.django.operations.DatabaseOperations.combine_duration_expression so I can't switch or predict the types of operations thus I shall refer this to you @timgraham, otherwise I'd have sent a PR now that the functions to be used are straightforward. |
When I filed this issue, I didn't have a solution in mind. I'm not sure it's solvable but perhaps Spanner engineers can comment if the limitation has come up before. |
This VERSION (adding CAST ... AS TIMESTAMP) doesn't really make things better: def combine_duration_expression(self, connector, sub_expressions):
if connector == '+':
return 'TIMESTAMP_ADD(CAST(%s AS TIMESTAMP), %s)' % tuple(sub_expressions)
elif connector == '-':
return 'TIMESTAMP_SUB(CAST(%s AS TIMESTAMP), %s)' % tuple(sub_expressions)
else:
raise DatabaseError('Invalid connector for timedelta: %s.' % connector) Usually if the first argument is a date, then a date is expected to be returned, not a timestamp. Example test failure:
There's also the possibility that the first argument is an interval like this:
|
Until (if feasible) Spanner provides a way of adding date or datetime with interval that doesn't require choosing between TIMESTAMP_ADD and DATE_ADD, I've proposed to document this limitation in #446. |
Currently this is a documented know issue as per this: #446 that we don't plan on working, hence closing this issue |
In #209, I made the assumption that
DatabaseOperations.combine_duration_expression()
operates on TIMESTAMP (TIMESTAMP_ADD
andTIMESTAMP_SUB
). However, it can also receive a DATE in which case, Spanner crashes as it requires usingDATE_ADD
/DATE_SUB
instead:The text was updated successfully, but these errors were encountered: