Migration 2019-04-28-201506_create_tantivy_index is failing to run #702

New issue

Closed

opened 2019-11-26 10:24:27 +00:00 by rhaamo · 3 comments

rhaamo commented

2019-11-26 10:24:27 +00:00

(Migrated from github.com)

Updated to latest git from an old plume and the migration 2019-04-28-201506_create_tantivy_index doesn't run (postgresql).

plume-dashie@fedistodon:~/Plume$ plm migration run
Configuration read from /home/plume-dashie/Plume/.env
Running migration 20190428201506
thread 'main' panicked at 'Failed to run migrations: Db(DatabaseError(__Unknown, "column users.role does not exist"))', src/libcore/result.rs:997:5

this is probably due to the migration using the latest Database Model, but with an old database, because users.role is created in migration 2019-06-18-152700_moderator_role later.

Updated to latest git from an old plume and the migration `2019-04-28-201506_create_tantivy_index` doesn't run (postgresql). ``` plume-dashie@fedistodon:~/Plume$ plm migration run Configuration read from /home/plume-dashie/Plume/.env Running migration 20190428201506 thread 'main' panicked at 'Failed to run migrations: Db(DatabaseError(__Unknown, "column users.role does not exist"))', src/libcore/result.rs:997:5 ``` this is probably due to the migration using the latest Database Model, but with an old database, because `users.role` is created in migration `2019-06-18-152700_moderator_role` later.

👍 1

trinity-1686a commented

2019-12-09 21:41:35 +00:00

Owner

I can't reproduce but I'm pretty sure I know what the issue is. Could you run the same command with RUST_BACKTRACE=1? I would expect Searcher::add_document to appear probably as the first non std call in the backtrace.

The following is my understanding from your report, but would require a bit more testing to make sure it's actually the issue.

Short story:
#555 introduced arbitrary rust in migration, that is totally broken.

Long story
At compile time, the schema used in rust structures is always the one of latest migration, and it is assumed that migrations will be run before trying to query, which works pretty much always. But in this case, some rust code is run before end of migrations (it is a migration), and so it might query the database (again, it is a migration), but will expect the model to be as it will be after the last migration will be run. This is a big design flaw, and I think the best way to handle it is to remove such migrations for now, and search for better ideas or some other way to do. While that seemed like a good idea at the time, I don't see any clean way of handling this very non rare situation in the future.

I can't reproduce but I'm pretty sure I know what the issue is. Could you run the same command with `RUST_BACKTRACE=1`? I would expect `Searcher::add_document` to appear probably as the first non std call in the backtrace. The following is my understanding from your report, but would require a bit more testing to make sure it's actually the issue. Short story: #555 introduced arbitrary rust in migration, that is totally broken. Long story At compile time, the schema used in rust structures is always the one of latest migration, and it is assumed that migrations will be run before trying to query, which works pretty much always. But in this case, some rust code is run before end of migrations (it _is_ a migration), and so it might query the database (again, it _is_ a migration), but will expect the model to be as it will be after the last migration will be run. This is a big design flaw, and I think the best way to handle it is to remove such migrations for now, and search for better ideas or some other way to do. While that seemed like a good idea at the time, I don't see any clean way of handling this very non rare situation in the future.

rhaamo commented

2019-12-10 07:14:30 +00:00

(Migrated from github.com)

Here is the full backtrace:

plume-dashie@fedistodon:~/Plume$ RUST_BACKTRACE=1 plm migration run
Configuration read from /home/plume-dashie/Plume/.env
Running migration 20190428201506
thread 'main' panicked at 'Failed to run migrations: Db(DatabaseError(__Unknown, "column users.role does not exist"))', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   6: rust_begin_unwind
             at src/libstd/panicking.rs:308
   7: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   8: core::result::unwrap_failed
   9: plm::migration::run
  10: plm::main
  11: std::rt::lang_start::{{closure}}
  12: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:293
  13: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:87
  14: std::rt::lang_start_internal
             at src/libstd/panicking.rs:272
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  15: main
  16: __libc_start_main
  17: _start
plume-dashie@fedistodon:~/Plume$

Yeah I've thought of the same thing, unfortunately there is no magic solution for data migrations mixed with schema ones.

I had issues with data migrations in my schema ones on my projects and either I was able to use the ORM without binding a schema by querying directly the right things, or I ended up moving all the data migrations in separate CLI commands.

Here is the full backtrace: ``` plume-dashie@fedistodon:~/Plume$ RUST_BACKTRACE=1 plm migration run Configuration read from /home/plume-dashie/Plume/.env Running migration 20190428201506 thread 'main' panicked at 'Failed to run migrations: Db(DatabaseError(__Unknown, "column users.role does not exist"))', src/libcore/result.rs:997:5 stack backtrace: 0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39 1: std::sys_common::backtrace::_print at src/libstd/sys_common/backtrace.rs:71 2: std::panicking::default_hook::{{closure}} at src/libstd/sys_common/backtrace.rs:59 at src/libstd/panicking.rs:197 3: std::panicking::default_hook at src/libstd/panicking.rs:211 4: std::panicking::rust_panic_with_hook at src/libstd/panicking.rs:474 5: std::panicking::continue_panic_fmt at src/libstd/panicking.rs:381 6: rust_begin_unwind at src/libstd/panicking.rs:308 7: core::panicking::panic_fmt at src/libcore/panicking.rs:85 8: core::result::unwrap_failed 9: plm::migration::run 10: plm::main 11: std::rt::lang_start::{{closure}} 12: std::panicking::try::do_call at src/libstd/rt.rs:49 at src/libstd/panicking.rs:293 13: __rust_maybe_catch_panic at src/libpanic_unwind/lib.rs:87 14: std::rt::lang_start_internal at src/libstd/panicking.rs:272 at src/libstd/panic.rs:388 at src/libstd/rt.rs:48 15: main 16: __libc_start_main 17: _start plume-dashie@fedistodon:~/Plume$ ``` Yeah I've thought of the same thing, unfortunately there is no magic solution for data migrations mixed with schema ones. I had issues with data migrations in my schema ones on my projects and either I was able to use the ORM without binding a schema by querying directly the right things, or I ended up moving all the data migrations in separate CLI commands.

trinity-1686a commented

2019-12-10 07:49:38 +00:00

Owner

The goal of such migrations was to reduce such separate commands (the exact migration failing for you is strictly equivalent to plm search init, although running it won't mark the migration as run).
The fix will probably be to remove this migration, reintroduce it's command as an installation step, and remove one other, and try to rewrite it in sql

The goal of such migrations was to reduce such separate commands (the exact migration failing for you is strictly equivalent to `plm search init`, although running it won't mark the migration as run). The fix will probably be to remove this migration, reintroduce it's command as an installation step, and remove one other, and try to rewrite it in sql

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Plume/Plume#702

No description provided.

Rows
Columns