-
Notifications
You must be signed in to change notification settings - Fork 14
Concurrent ChunksDownloader #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
If you do that you have to merge them in the correct order Right now I'm facing an issue with the chunk data being in the incorrect order. Trying to determine if the chunk data is wrong or if the way it's being processed by this library is wrong. |
Yes, you need to merge the result in the correct order, which can be completed easily by marking each task with a number, then ordering by the number before merge the chunks. Something like this:
|
Yep, agreed. Also, updating my client library from .4.0 to .4.3 seems to have solved my issue with data being in the incorrect order. |
@tungda-ccbji, Thanks and sorry for a long waited answer. As for As for concurrent |
That's weird. Because right now it just downloads chunks one-by-one and merge them in the same order as they come in |
@fixer-m Some change between .40 and .43 fixed my issue. I was able to reproduce it easily for basically anything that chunked. The easy check was to use a ROW_NUMBER(). My first rows were in the 300s instead of 1, 2, 3, ... It didn't happen every time, but I'd say once every 3 or 4 queries at least. |
@tungda-ccbji I've implemented concurrent ChunksDownloader in 0.4.4. You can specify threads count using As well as I've added new option |
@grexican Yeah - I used almost the same approach as you suggested for concurrent chunks downloader. After downloading chunks got sorted in the same order as they were received in initial response. I've only added concurrency threshold with SemaphoreSlim. Please let me know if you find any issues with it. Thanks! |
On the method QueryAsync, you do have a check for Chunks > 0, then call ChunksDownloader to get data from all the chunks, however, you missed this on method QueryRawResponseAsync.
Another thing I think that can be improved is the ChunksDownloader, currently, you are downloading multiple chunks 1 by 1. When I tried to make a query that returned around 14000 records, it took about 5 seconds to complete. If you let these chunks run by themselves on multiple tasks, then merge the result once all the tasks are completed, it could reduce the time to under 1 second.
The text was updated successfully, but these errors were encountered: