Plume/Plume

派生 24

[WIP] [PoC] '&' -> '&mut' #777

草稿

jebrosen 想要将来自 igalic/go/async-all-mut 的 2 笔提交合并到 go/async

jebrosen 评论于

2020-05-25 19:40:35 +00:00

(从 github.com 迁移)

This is a (incomplete) proof of concept of a possible workaround for Sync issues that appear in async code.

Roughly, the issue is this:

Route handler futures must be Send. This requirement comes from Rocket, and would be nontrivial and/or undesirable to change in Rocket.
Handlers have an &PlumeRocket or an &Connection held across an await point
Therefore, &PlumeRocket / &Connection must be Send
&T: Send iff T: Sync, so PlumeRocket / Connection must be Sync
PlumeRocket contains a Connection, and Connection contains a diesel PgConnection, which is not Sync.

The approach demonstrated here is to change every &PlumeRocket or &Connection to an &mut PlumeRocket or &mut Connection. &mut T is Send if T is Send, so the problem is eliminated:

Route handler futures must be Send.
Handlers have an &mut PlumeRocket or an &mut Connection held across an await point
Therefore, &mut PlumeRocket / &mut Connection must be Send
&mut T: Send iff T: Send, so PlumeRocket / Connection must be Send
PlumeRocket contains a Connection, and Connection contains a diesel PgConnection, which is Send.

Downsides

In theory &PlumeRocket could allow more work to be done in parallel, at least in the future. It does not look like that is currently the case, since every call to the database blocks anyway.
This change is pervasive - it reaches all the way to FromId and Inbox. I know relatively little about the overall structure of this code, so this could be incorrect or inconvenient in ways I don't know about!
Many of the remaining errors are caused by or made worse by the & -> &mut change. A different solution that keeps & in more places would be easier to work with overall.
This approach does not address the problem of making blocking database calls inside async fns, which can cause issues ranging from degraded performance to deadlocks.

Alternatives

Put a Mutex around the Connection somewhere. Uncontended mutexes (which this one should be) are not a huge performance concern, but Mutex may be at least as or more unwieldy than this solution throughout the code.
Replace or wrap Connection with an API like conn.run(|c| Post::load(&c)).await, where run handles the synchronization. This has similar tradeoffs to a Mutex, is probably the most inconvenient option in terms of overall code changes, and is also a significant chunk of new code to write and debug. However, it has the advantage of being capable of fixing the blocking-in-async-fn problem.

This is a (incomplete) proof of concept of a possible workaround for `Sync` issues that appear in `async` code. Roughly, the issue is this: * Route handler futures must be `Send`. This requirement comes from Rocket, and would be nontrivial and/or undesirable to change in Rocket. * Handlers have an `&PlumeRocket` or an `&Connection` held across an `await` point * Therefore, `&PlumeRocket` / `&Connection` must be `Send` * `&T: Send` iff `T: Sync`, so `PlumeRocket` / `Connection` must be `Sync` * `PlumeRocket` contains a `Connection`, and `Connection` contains a diesel `PgConnection`, which is not `Sync`. The approach demonstrated here is to change every `&PlumeRocket` or `&Connection` to an `&mut PlumeRocket` or `&mut Connection`. `&mut T` is `Send` if `T` is `Send`, so the problem is eliminated: * Route handler futures must be `Send`. * Handlers have an `&mut PlumeRocket` or an `&mut Connection` held across an `await` point * Therefore, `&mut PlumeRocket` / `&mut Connection` must be `Send` * **`&mut T: Send` iff `T: Send`**, so `PlumeRocket` / `Connection` must be **`Send`** * `PlumeRocket` contains a `Connection`, and `Connection` contains a diesel `PgConnection`, which **is `Send`.** ## Downsides * In theory `&PlumeRocket` *could* allow more work to be done in parallel, at least in the future. It does not look like that is currently the case, since every call to the database blocks anyway. * This change is pervasive - it reaches all the way to `FromId` and `Inbox`. I know relatively little about the overall structure of this code, so this could be incorrect or inconvenient in ways I don't know about! * Many of the remaining errors are caused by or made worse by the `&` -> `&mut` change. A different solution that keeps `&` in more places would be easier to work with overall. * This approach does not address the problem of making blocking database calls inside `async` fns, which can cause issues ranging from degraded performance to deadlocks. ## Alternatives * Put a `Mutex` around the `Connection` somewhere. Uncontended mutexes (which this one should be) are not a *huge* performance concern, but `Mutex` may be at least as or more unwieldy than this solution throughout the code. * Replace or wrap `Connection` with an API like `conn.run(|c| Post::load(&c)).await`, where `run` handles the synchronization. This has similar tradeoffs to a `Mutex`, is probably the most inconvenient option in terms of overall code changes, and is also a significant chunk of new code to write and debug. However, it has the advantage of being capable of fixing the blocking-in-`async`-fn problem.

trinity-1686a 评论于

2020-05-25 19:52:50 +00:00

管理员

I think the best way to handle sql connections would be to have worker threads that are basically dedicated to that, and have a mpsc channel through which requests can be send to them, alongside a one shot channel that allow to return a result.
This is basically how actors work (like in Erlang and derivative, or the Actix lib for Rust), it would allow to keep &, would properly handle blocking operation out of async context, and maybe allow to compile both Postgresql and Sqlite in the same binary (however this would also be a lot of work, not that much new code, but lots of moving things around)

I think the best way to handle sql connections would be to have worker threads that are basically dedicated to that, and have a [mpsc channel](https://docs.rs/async-std/1.6.0/async_std/sync/fn.channel.html) through which requests can be send to them, alongside a [one shot channel](https://docs.rs/tokio/0.1.22/tokio/sync/oneshot/fn.channel.html) that allow to return a result. This is basically how actors work (like in Erlang and derivative, or the Actix lib for Rust), it would allow to keep `&`, would properly handle blocking operation out of async context, and _maybe_ allow to compile both Postgresql and Sqlite in the same binary (however this would also be a lot of work, not that much new code, but lots of moving things around)

jebrosen 评论于

2020-05-25 20:05:49 +00:00

(从 github.com 迁移)

I think the best way to handle sql connections would be to have worker threads that are basically dedicated to that

Yeah, I think that's more or less the direction I was going with "wrap Connection with an API like conn.run(|c| Post::load(&c)).await". I agree that it's a nicer overall solution, with the biggest drawback being:

however this would also be a lot of work, not that much new code, but lots of moving things around

> I think the best way to handle sql connections would be to have worker threads that are basically dedicated to that Yeah, I think that's more or less the direction I was going with "wrap `Connection` with an API like `conn.run(|c| Post::load(&c)).await`". I agree that it's a nicer overall solution, with the biggest drawback being: > however this would also be a lot of work, not that much new code, but lots of moving things around

igalic 评论于

2020-05-25 20:10:29 +00:00

(从 github.com 迁移)

I agree that it's a nicer overall solution, with the biggest drawback being:

however this would also be a lot of work, not that much new code, but lots of moving things around

🤷‍♀️

we have come this far, we might as well do it right.

> I agree that it's a nicer overall solution, with the biggest drawback being: >> however this would also be a lot of work, not that much new code, but lots of moving things around :woman_shrugging: we have come this far, we might as well do it right.

igalic 引用了合并请求

2020-07-14 20:42:13 +00:00

[Refactoring] Move DbConn out PlumeRocket #797

此合并请求被标记为正在进行的工作。

查看命令行说明

检出

从你的仓库中检出一个新的分支并测试变更。

git fetch -u origin igalic/go/async-all-mut:igalic/go/async-all-mut

git checkout igalic/go/async-all-mut

合并

合并更改并在 Forgejo 上更新。

警告：未启用此仓库的“自动检测手动合并”设置，您之后必须将此合并请求标记为手动合并。

git checkout go/async

git merge --no-ff igalic/go/async-all-mut

git checkout igalic/go/async-all-mut

git rebase go/async

git checkout go/async

git merge --ff-only igalic/go/async-all-mut

git checkout igalic/go/async-all-mut

git rebase go/async

git checkout go/async

git merge --no-ff igalic/go/async-all-mut

git checkout go/async

git merge --squash igalic/go/async-all-mut

git checkout go/async

git merge --ff-only igalic/go/async-all-mut

git checkout go/async

git merge igalic/go/async-all-mut

git push origin go/async

登录并参与到对话中。

无评审员

未选择里程碑

暂无项目

未指派成员

2 位参与者

通知

到期时间

到期日期无效或超出范围。请使用“yyyy-mm-dd”格式。

未设置到期时间。

依赖工单

没有设置依赖项。

引用：Plume/Plume#777

没有提供说明。

行数
列数