Error during 0024_auto_20191230_2052 migration

[Fogapod]

I tried upgrading my 9.12 installation to 10 and this error happened during database migration:

Applying sentry.0024_auto_20191230_2052...Events to process: 802
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 119, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 200, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 92, in migrate
    self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 121, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 198, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 123, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 62, in backfill_eventstream
    for event in RangeQuerySetWrapper(events, step=100, callbacks=(_attach_related,)):
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/query.py", line 83, in __iter__
    cb(results)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 46, in _attach_related
    event.group = groups[event.group_id]
KeyError: 22L

Looks like my database is a bit broken, one project got deleted, but there are still references to it from some tables. I had an error in previous migration too and had to delete some rows from tables that pointed to deleted project_id. This error is probably related to that too, but I don't understand what exactly causes it and what can I do.

How can I fix this?

I have a backup of my database volume and can try different solutions.

[BYK]

Oh, this is probably something we can protect against but I need our expert on this: @lynnagara - any ideas?

[Fogapod]

I can't reproduce this anymore for some reason. This worked the same way yesterday when I tested this multiple times. Now I always get different error running same commands:

sudo rm -rf /var/lib/docker/volumes/sentry-postgres
sudo cp ~/sentry-postgres /var/lib/docker/volumes/ -r
./install.sh

gives me

Faking migration for contenttypes.0001_initial
Faking migration for contenttypes.0002_remove_content_type_name
Faking migration for auth.0001_initial
Faking migration for auth.0002_alter_permission_name_max_length
Faking migration for auth.0003_alter_user_email_max_length
Faking migration for auth.0004_alter_user_username_opts
Faking migration for auth.0005_alter_user_last_login_null
Faking migration for auth.0006_require_contenttypes_0002
Faking migration for sites.0001_initial
Faking migration for admin.0001_initial
Faking migration for sessions.0001_initial
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 130, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 86, in handle
    executor.loader.check_consistent_history(connection)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 292, in check_consistent_history
    connection.alias,
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration jira_ac.0001_initial is applied before its dependency sentry.0003_auto_20191022_0122 on database 'default'.

every time. Database backup was not changed, sentry 9.12 runs with it without problems.
Should I open a new issue?

[BYK]

@Fogapod, yes, can you file a new issue for this second one. I'm fairly sure it is due to a missing line over at https://github.com/getsentry/sentry/blob/2ee72b121658496633e71d77f34e006de88a220b/src/sentry/conf/server.py#L1739-L1741 and I hope to resolve this one very soon with the help of @wedamija. I'm guessing you had the Jira plugin installed earlier? (or just sentry-plugins?)

[Fogapod]

I'm nots sure, I had github plugin installed before but I'm not using it anymore, it might got uninstalled at some point

[BYK]

All plugins are now baked in so no need to install them anymore and that would explain the issue as at some point you had the Jira plugin installed too (as part of the sentry-plugins package), having the migration applied earlier. Submitted a potential fix and hoping to resolve it by EOD today.

[servomac]

I have fixed first reported error directly deleting a few events referencing not existing groups:

DELETE from sentry_message where id in (select M.id from sentry_message M left join sentry_groupedmessage G on G.id = M.group_id where G.id IS NULL)