Add doc note on memory usage of read_sql with chunksize

[jorisvandenbossche]

As this typically does not give you much memory usage improvement (which is a bit unexpected from the keyword explanation), this is worth a note in the docs.

From some discussion on gitter: https://gitter.im/pydata/pandas?at=55b61bf952d85d450f404be1 (with @litaotao) and https://gitter.im/pydata/pandas?at=554609295edd84254582fb39 (with @twiecki)

[zirmite]

It should give you memory improvement if the db api/engine is hooked up correctly, right?

Using it with sqlalchemy+psycopg2 it needs to make sure that sqlengine.dialect.server_side_cursors == True for the sqlengine.execute call to be separate from the results.fetchmany call. This seems to be a postgres-only issue and engine option because of how it handles cursors (perhaps?).

However, sqlalchemy's engine.execution_options API provides stream_results which will try to execute without pre-buffering.

stream_results – Available on: Connection, statement. Indicate to the dialect that results should be “streamed” and not pre-buffered, if possible. This is a limitation of many DBAPIs. The flag is currently understood only by the psycopg2 dialect.

Maybe pandas could try and set that flag if chunksize is not None?

[dostabhi]

Any update on this issue?

[MordorianGuy]

Does anyone know which parameters should be passed to iterate over Clickhouse query with the stream?