Skip to content

Commit 9d0aa96

Browse files
committed
Add documentation
1 parent 5d6e5b9 commit 9d0aa96

File tree

1 file changed

+67
-0
lines changed

1 file changed

+67
-0
lines changed

README.md

+67
Original file line numberDiff line numberDiff line change
@@ -618,6 +618,73 @@ my_periodic_resque_job:
618618

619619
and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.
620620

621+
## Multisharding
622+
623+
If your application reaches a point where the pressure on the database used for jobs is such that you need to spread the load over multiple databases, then this section is for you.
624+
625+
You can extend the Solid Queue database configuration to use different shards:
626+
627+
```ruby
628+
config.solid_queue.connects_to = {
629+
shards: {
630+
queue_shard_one: { writing: :queue_shard_one },
631+
queue_shard_two: { writing: :queue_shard_two }
632+
}
633+
}
634+
```
635+
636+
Queue database shards will need to have been defined in `config/database.yml` as shown in the installation section. Both shards need to share the same schema, and down the line share the same migration configuration:
637+
638+
```yaml
639+
production:
640+
primary:
641+
<<: *default
642+
database: storage/production.sqlite3
643+
queue_shard_one:
644+
<<: *default
645+
database: storage/production_queue_shard_one.sqlite3
646+
migrations_paths: db/queue_migrate
647+
queue_shard_two:
648+
<<: *default
649+
database: storage/production_queue_shard_two.sqlite3
650+
migrations_paths: db/queue_migrate
651+
```
652+
653+
Simply converting a simpler database configuration such as `config.solid_queue.connects_to = { database: { writing: :queue } }` to `config.solid_queue.connects_to = { shards: { queue: { writing: :queue } } }` will not have any effects on the behavior of Solid Queue.
654+
655+
### Configuration
656+
657+
In `config/environments/production.rb` or for the environment you want to enable Solid Queue in, you can define the following options:
658+
659+
```ruby
660+
config.solid_queue.primary_shard = :queue_shard_one # optional
661+
config.solid_queue.active_shard = ENV["SOLID_QUEUE_ACTIVE_SHARD"]&.to_sym
662+
config.solid_queue.shard_selection_lambda = ->(active_job:, active_jobs:) { nil }
663+
```
664+
665+
- `config.solid_queue.primary_shard` is the shard that will be used to enqueue or schedule jobs without any specific adapter configuration. It defaults to the first shard found in `config.solid_queue.connects_to` (ActiveRecord default)
666+
- `config.solid_queue.active_shard` is the shard that will be used by workers, dispatchers and schedulers to manage and process jobs. It defaults to the `primary_shard`.
667+
With a basic Solid Queue configuration and the option described above you can start a worker and dispatcher working on the `queue_shard_two` shard by running `SOLID_QUEUE_ACTIVE_SHARD=queue_shard_two bin/jobs start`
668+
- `config.solid_queue.shard_selection_lambda` helps you define a custom strategy to determine in which shard a job should be enqueued. It accepts keyword parameters `active_job` when a single job is enqueued or scheduled and `active_jobs` when jobs are bulk enqueued. If the lambda is defined but returns `nil`, Solid Queue will use the adapter defined for the job.
669+
670+
### Enqueueing jobs in different shards
671+
672+
Individual jobs can be assigned to shards by leveraging their `queue_adapter` property:
673+
674+
```ruby
675+
class SomeJob < ApplicationJob
676+
self.queue_adapter = ActiveJob::QueueAdapters::SolidQueueAdapter.new(db_shard: :queue_shard_two)
677+
```
678+
679+
This job will be enqueued in the shard named `queue_shard_two`.
680+
681+
Alternatively you can define a lambda to implement a custom strategy for defining which shard a job will be enqueued to:
682+
683+
```ruby
684+
config.solid_queue.shard_selection_lambda = ->(active_job:, active_jobs:) { SolidQueue.connects_to[:shards].keys.sample } # pick a shard randomly
685+
```
686+
687+
621688
## Inspiration
622689

623690
Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.

0 commit comments

Comments
 (0)