Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 2.0.5 #148

Merged
merged 1 commit into from
Nov 11, 2021
Merged

Release 2.0.5 #148

merged 1 commit into from
Nov 11, 2021

Conversation

avahuang0429
Copy link
Contributor

@avahuang0429 avahuang0429 commented Nov 11, 2021 •
edited
Loading

Issue #, if available:
Release 2.0.5 with dependencies upgrade
Description of changes:

Release 2.0.5 (November 11, 2021)

  • Upgraded multiple dependencies PR #148
    • Amazon Kinesis Client Library 2.3.8
    • AWS SDK 2.17.52
  • Added dependencies
    • AWS SDK json-utils 2.17.52
    • third-party-jackson-core 2.17.52
    • third-party-jackson-dataformat-cbor 2.17.52
  • Updated samples/sample.properties reflecting support for InitialPositionInStreamExtended
    • Related: #804 Allowing user to specify an initial timestamp in which daemon will process records.
    • Feature released with previous release 2.0.4

Testing
Tested with changed sample.properties, look for ShardFilter=ShardFilter(Type=AT_TIMESTAMP, Timestamp=2021-11-11T05:39:02Z)) in the below output for test result

(21-11-11 6:03:21) <0> [~/workspace/amazon-kinesis-client-python]  
dev-dsk-xinlyh-2a-6cde8fe0 % `amazon_kclpy_helper.py --print_command --java /usr/bin/java --properties samples/sample.properties`
2021-11-11 06:03:23,541 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Using a cached thread pool. 
2021-11-11 06:03:23,545 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Running PythonKCLSample to process stream words with executable sample_kclpy_app.py 
2021-11-11 06:03:23,546 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Using workerId: bb8220a0-6cf6-40e2-849b-88657d7b0179 
2021-11-11 06:03:23,546 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - MultiLangDaemon is adding the following fields to the User Agent: amazon-kinesis-client-library-java amazon-kinesis-multi-lang-daemon/1.0.1 python/3.9 sample_kclpy_app.py 
2021-11-11 06:03:24,005 [main] INFO  s.a.k.l.d.DynamoDBLeaseCoordinator [NONE] - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time. 
2021-11-11 06:03:24,010 [main] INFO  s.a.k.l.d.DynamoDBLeaseCoordinator [NONE] - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time. 
2021-11-11 06:03:24,016 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Initialization attempt 1 
2021-11-11 06:03:24,016 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Initializing LeaseCoordinator 
2021-11-11 06:03:25,087 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-11-11 06:03:28,168 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-11-11 06:03:31,251 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-11-11 06:03:34,334 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Lease table is still empty. Checking again in 3000 ms 
2021-11-11 06:03:37,335 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Worker bb8220a0-6cf6-40e2-849b-88657d7b0179 is initiating the lease sync. 
2021-11-11 06:03:37,335 [multi-lang-daemon-0000] INFO  s.a.k.c.PeriodicShardSyncManager [NONE] - Syncing Kinesis shard info for words 
2021-11-11 06:03:37,430 [multi-lang-daemon-0000] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream words: listing shards with list shards request ListShardsRequest(StreamName=words, ShardFilter=ShardFilter(Type=AT_TIMESTAMP, Timestamp=2021-11-11T05:39:02Z)) 
2021-11-11 06:04:38,382 [multi-lang-daemon-0000] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Starting lease cleanup thread. 
2021-11-11 06:04:38,384 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Starting LeaseCoordinator 
2021-11-11 06:04:38,384 [pool-15-thread-1] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Number of pending leases to clean before the scan : 0 
2021-11-11 06:04:38,694 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Scheduling periodicShardSync 
2021-11-11 06:04:38,695 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Initialization complete. Starting worker loop. 
2021-11-11 06:04:38,775 [LeaseCoordinator-0000] INFO  s.a.k.l.dynamodb.DynamoDBLeaseTaker [NONE] - Taking leases that have been expired for a long time: [Lease(leaseKey=shardId-000000000000, leaseOwner=null, leaseCounter=0, concurrencyToken=null, lastCounterIncrementNanos=0, checkpoint={SequenceNumber: AT_TIMESTAMP,SubsequenceNumber: 0}, pendingCheckpoint=null, pendingCheckpointState=null, ownerSwitchesSinceCheckpoint=0, parentShardIds=[], childShardIds=[], hashKeyRangeForLease=HashKeyRangeForLease(startingHashKey=0, endingHashKey=340282366920938463463374607431768211455))] 
2021-11-11 06:04:38,866 [LeaseCoordinator-0000] INFO  s.a.k.l.dynamodb.DynamoDBLeaseTaker [NONE] - Worker bb8220a0-6cf6-40e2-849b-88657d7b0179 successfully took 1 leases: shardId-000000000000 
2021-11-11 06:04:38,923 [multi-lang-daemon-0000] INFO  s.a.k.c.DeterministicShuffleShardSyncLeaderDecider [NONE] - Elected leaders: bb8220a0-6cf6-40e2-849b-88657d7b0179 
2021-11-11 06:04:38,924 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=0, maximumPoolSize=2147483647) 
2021-11-11 06:04:39,925 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Current stream shard assignments: shardId-000000000000 
2021-11-11 06:04:40,148 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : StreamConsumer not found, need to create it. 
2021-11-11 06:04:40,345 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-11-11 06:04:41,425 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-11-11 06:04:42,503 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-11-11 06:04:43,581 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-11-11 06:04:44,659 [multi-lang-daemon-0000] INFO  s.a.k.r.f.FanOutConsumerRegistration [NONE] - words : Waiting for StreamConsumer PythonKCLSample to have ACTIVE status... 
2021-11-11 06:04:45,676 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Created new shardConsumer for : ShardInfo(streamIdentifierSerOpt=Optional.empty, shardId=shardId-000000000000, concurrencyToken=809f6607-b683-4902-aa21-516426bccbd7, parentShardIds=[], checkpoint={SequenceNumber: AT_TIMESTAMP,SubsequenceNumber: 0}) 
2021-11-11 06:04:45,677 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Sleeping ... 
2021-11-11 06:04:45,678 [ShardRecordProcessor-0000] INFO  s.a.k.l.BlockOnParentShardTask [NONE] - No need to block on parents [] of shard shardId-000000000000 
2021-11-11 06:04:46,766 [multi-lang-daemon-0001] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading STDERR for shardId-000000000000 
2021-11-11 06:04:46,767 [ShardRecordProcessor-0000] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing InitializeMessage to child process for shard shardId-000000000000 
2021-11-11 06:04:46,791 [multi-lang-daemon-0002] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Message size == 110 bytes for shard shardId-000000000000 
2021-11-11 06:04:46,791 [multi-lang-daemon-0003] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000000 
2021-11-11 06:04:46,815 [ShardRecordProcessor-0000] INFO  s.a.k.multilang.MultiLangProtocol [NONE] - Received response {"action":"status","responseFor":"initialize"} from subprocess while waiting for initialize while processing shard shardId-000000000000 
2021-11-11 06:04:48,978 [ShardRecordProcessor-0001] INFO  s.a.k.leases.KinesisShardDetector [NONE] - Stream words: listing shards with list shards request ListShardsRequest(StreamName=words) 
2021-11-11 06:04:49,054 [ShardRecordProcessor-0001] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Writing ProcessRecordsMessage to child process for shard shardId-000000000000 
2021-11-11 06:04:49,065 [multi-lang-daemon-0002] INFO  s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000000 
2021-11-11 06:04:49,065 [multi-lang-daemon-0003] INFO  s.a.kinesis.multilang.MessageWriter [NONE] - Message size == 261 bytes for shard shardId-000000000000 
2021-11-11 06:04:49,066 [ShardRecordProcessor-0001] INFO  s.a.k.multilang.MultiLangProtocol [NONE] - Received response {"action":"status","responseFor":"processRecords"} from subprocess while waiting for processRecords while processing shard shardId-000000000000 
2021-11-11 06:05:09,741 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-11-11 06:05:38,384 [pool-15-thread-1] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Number of pending leases to clean before the scan : 0 
2021-11-11 06:05:38,695 [pool-14-thread-1] INFO  s.a.k.c.PeriodicShardSyncManager [NONE] - WorkerId bb8220a0-6cf6-40e2-849b-88657d7b0179 is leader, running the periodic shard sync task 
2021-11-11 06:05:38,770 [pool-14-thread-1] INFO  s.a.k.c.PeriodicShardSyncManager [NONE] - Skipping shard sync for words due to the reason - Hash Ranges are complete for words 
2021-11-11 06:05:38,998 [pool-13-thread-1] INFO  s.a.k.c.DeterministicShuffleShardSyncLeaderDecider [NONE] - Elected leaders: bb8220a0-6cf6-40e2-849b-88657d7b0179 
2021-11-11 06:05:39,778 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-11-11 06:05:46,781 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Current stream shard assignments: shardId-000000000000 
2021-11-11 06:05:46,781 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Sleeping ... 
2021-11-11 06:06:09,857 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-11-11 06:06:38,384 [pool-15-thread-1] INFO  s.a.k.leases.LeaseCleanupManager [NONE] - Number of pending leases to clean before the scan : 0 
2021-11-11 06:06:39,937 [multi-lang-daemon-0000] INFO  s.a.k.c.DiagnosticEventLogger [NONE] - Current thread pool executor state: ExecutorStateEvent(executorName=SchedulerThreadPoolExecutor, currentQueueSize=0, activeThreads=0, coreThreads=0, leasesOwned=1, largestPoolSize=2, maximumPoolSize=2147483647) 
2021-11-11 06:06:47,940 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Current stream shard assignments: shardId-000000000000 
2021-11-11 06:06:47,940 [multi-lang-daemon-0000] INFO  s.a.kinesis.coordinator.Scheduler [NONE] - Sleeping ...

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@avahuang0429 avahuang0429 force-pushed the 2.0.5 branch 3 times, most recently from 3535a13 to fe5f1b7 Compare November 11, 2021 08:17
rexcsn
rexcsn reviewed Nov 11, 2021
View reviewed changes
setup.py Outdated
('software.amazon.awssdk', 'kinesis', '2.16.98'),
('software.amazon.awssdk', 'aws-cbor-protocol', '2.16.98'),
('software.amazon.kinesis', 'amazon-kinesis-client-multilang', '2.3.7'),
('software.amazon.kinesis', 'amazon-kinesis-client', '2.3.7'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release note in README says 2.3.8, but looks like the version is 2.3.7 here. Should we fix so that the version is consistent?

rexcsn
rexcsn reviewed Nov 11, 2021
View reviewed changes
setup.py Outdated
('software.amazon.awssdk', 'protocol-core', '2.17.52'),
('software.amazon.awssdk', 'profiles', '2.17.52'),
('software.amazon.awssdk', 'sdk-core', '2.17.52'),
('software.amazon.awssdk', 'sdk-core', '2.17.52'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicated sdk-core

README.md Outdated
* Amazon Kinesis Client Library 2.3.8
* AWS SDK 2.17.52
* Added dependencies
* AWS SDK json-utils
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: do we need to specify the version to be consistent?

README.md Outdated
* AWS SDK 2.17.52
* Added dependencies
* AWS SDK json-utils
* third-party-jackson 2.17.52
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: looks like we are adding third-party-jackson-core and third-party-jackson-dataformat-cbor, not sure if they make up third-party-jackson but might be nice to list the specific dependencies we are adding

@avahuang0429 avahuang0429 force-pushed the 2.0.5 branch 3 times, most recently from 96bf74f to 9535214 Compare November 11, 2021 18:38
rexcsn
rexcsn approved these changes Nov 11, 2021
View reviewed changes
Copy link
Contributor

@rexcsn rexcsn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for cleaning up the dependencies!

@avahuang0429 avahuang0429 merged commit 5d08fc3 into master Nov 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rexcsn rexcsn rexcsn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@avahuang0429 @rexcsn