You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I tried to get data from oracle database and convert it to dataframe.
I've noticed that oracledb's cursor seems consume lot of memory and never get it back.
This is my env. info.
Oracle version : 19c
oracle client version : 19.20
oracledb lib version : 2.1
Table create query for testing
CREATE TABLE test_table (
ID NUMBER,
RandomColumn1 VARCHAR2(50),
RandomColumn2 NUMBER,
RandomColumn3 DATE
);
DECLARE
v_counter NUMBER := 1;
BEGIN
WHILE v_counter <= 10000000 LOOP
INSERT INTO test_table (ID, RandomColumn1, RandomColumn2, RandomColumn3)
VALUES (
v_counter,
DBMS_RANDOM.STRING('X', 10),
ROUND(DBMS_RANDOM.VALUE(-100, 100)),
TRUNC(SYSDATE - DBMS_RANDOM.VALUE(1, 365))
);
v_counter := v_counter + 1;
END LOOP;
COMMIT;
END;
/
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I tried to get data from oracle database and convert it to dataframe.
I've noticed that oracledb's cursor seems consume lot of memory and never get it back.
This is my env. info.
Oracle version : 19c
oracle client version : 19.20
oracledb lib version : 2.1
Table create query for testing
Python code to check memory usage
Result
C:\Users\SDS\AppData\Local\Programs\Python\Python310\python.exe D:\workspace\nscm_pre\z_local_run\test_oracle_memory.py
init memory usage 88.41797 MB
data fetch memory usage 2475.40625 MB
make dataframe memory usage 2781.08594 MB
close connection memory usage 2772.56641 MB
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000000 entries, 0 to 9999999
Data columns (total 4 columns):
Column Dtype
0 ID int64
1 RANDOMCOLUMN1 object
2 RANDOMCOLUMN2 int64
3 RANDOMCOLUMN3 datetime64[ns]
dtypes: datetime64ns, int64(2), object(1)
memory usage: 305.2+ MB
dataframe size : None
Process finished with exit code 0
I'd like to make used memory free after making dataframe.
Could somebody help?
Beta Was this translation helpful? Give feedback.
All reactions