WIP: Extract DbConn, Searcher & Worker into Riker Actors
#813
Draft
igalic
wants to merge 18 commits from igalic/Plume:refactor/extract-actors
into main
pull from: igalic/Plume:refactor/extract-actors
merge into: Plume:main
Plume:paginate-search-init
Plume:main
Plume:s3
Plume:fix-delete-user
Plume:timeline-cli
Plume:blog-title
Plume:signature
Plume:remove-dup-images
Plume:ldap-non-anon
Plume:drone-ci
Plume:DearRude/force-lang
Plume:igalic/go/async-all-mut
Plume:go/async
Plume:floreal/translations-update
Plume:missing-docs
Plume:RAOF/fix-arm64-build
Plume:epsilon-phase/authorized-fetch
Plume:upgrade
Plume:improve-the-editor-once-again
Plume:igalic/feat/custom-fairing-domains
Plume:feature/ldap
Plume:test/dotenv_error
Plume:fix-mobile-margin
Reviewers
Request review
No reviewers
Labels
Clear labels
Related to the REST API
Code running on the server
Stuff related to Federation
Related to the front-end
Translations, and related code
More about project management or code than the project itself
The building, or installation process of Plume
Something isn't working
We need to talk
New feature or request
This is a new feature
Compatibility with different browsers, readers and OS
Related to an external package that Plume uses
UI/UX related issues and PRs
Good for newcomers
Extra attention is needed
Issues affecting only mobile UX
How elements're rendered out for the end user
Something else needs to be fixed first
This issue or pull request already exists
This PR is not complete yet
Issues concern a limited number of instances
This doesn't seem right
Need to be discussed by the community (on Loomio)
This PR is ready to be reviewed
Proposed ideas worth considering
This is issue has been created after a vote on Loomio
This will not be worked on
Apply labels
A: API
Related to the REST API
A: Backend
Code running on the server
A: Federation
Stuff related to Federation
A: Front-End
Related to the front-end
A: I18N
Translations, and related code
A: Meta
More about project management or code than the project itself
A: Security
Build
The building, or installation process of Plume
C: Bug
Something isn't working
C: Discussion
We need to talk
C: Enhancement
New feature or request
C: Feature
This is a new feature
Compatibility
Compatibility with different browsers, readers and OS
Dependency
Related to an external package that Plume uses
Design
UI/UX related issues and PRs
Documentation
Good first issue
Good for newcomers
Help welcome
Extra attention is needed
Mobile
Issues affecting only mobile UX
Rendering
How elements're rendered out for the end user
S: Blocked
Something else needs to be fixed first
S: Duplicate
This issue or pull request already exists
S: Incomplete
This PR is not complete yet
S: Instance specific
Issues concern a limited number of instances
S: Invalid
This doesn't seem right
S: Needs Voting/Discussion
Need to be discussed by the community (on Loomio)
S: Ready for review
This PR is ready to be reviewed
Suggestion
Proposed ideas worth considering
S: Voted on Loomio
This is issue has been created after a vote on Loomio
S: Wontfix
This will not be worked on
No Label
A: API
A: Backend
A: Federation
A: Front-End
A: I18N
A: Meta
A: Security
Build
C: Bug
C: Discussion
C: Enhancement
C: Feature
Compatibility
Dependency
Design
Documentation
Good first issue
Help welcome
Mobile
Rendering
S: Blocked
S: Duplicate
S: Incomplete
S: Instance specific
S: Invalid
S: Needs Voting/Discussion
S: Ready for review
Suggestion
S: Voted on Loomio
S: Wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Assignees
Assign users
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.
No due date set.
Dependencies
No dependencies set.
Reference: Plume/Plume#813
Reference in New Issue
There is no content yet.
Delete Branch 'igalic/Plume:refactor/extract-actors'
Deleting a branch is permanent. It CANNOT be undone. Continue?
No
Yes
This PR addresses #799 and #797.
In #805 and #809 we have learned that
Searcher
andDbConn
are intimately connected, especially inPost
s.We want to replace
PlumeRocket
as the main context being passed around with a Riker ActorSystem, so we'll be able to access all the systems managed by the actor system. This includes the Context of our federationAsObject
s.Can you compile this branch? I couldn't...
Note: SearcherActor cannot commit indices due to
EnterError
(https://github.com/tantivy-search/tantivy/issues/898). This issue might be related: https://github.com/rust-lang/futures-rs/issues/2090.This occurs because perhaps both Riker and Tantivy use
futures::excutor::LocalPool
and Tantivy'sblock_on
is called in Riker'sLocalPool
.Possible options (based on my poor understanding of async) are:
Note that we cannot search new posts until their documents are commited not just added.
I respect your great research and implementation, and completely agree with introducing Riker. In addition, I make a suggestion: use Riker's Channels as a global variable. There are two points:
What is solved?
At current attempting (its strategy and current implementation:
8aa4ae4302
), we need touch searcher more than ever before. Also, we need flow, some agents (like searcher actor or post ids) through multiple structs.See updating search document procedure. When a user updates a post, a request handler does two (or more) things: it
UpdateDocument
message toSearcherActor
Developers need always do them together. This is needed for not only request handlers but
AsObject
andFromId
implementations for models. As such, developers need write more code than now. And they might have forgotten sending message to the actor.To make matters worse, this introduces more complexity. Assume you're deleting a user.
routes::user::delete()
request handler is called. It needs to know all blogs and posts that the deleting user owns in order to sendDeleteDocument
messeges corresponding to deleted posts to searcher actor. How?One way is:
Post::delete()
returns itself toBlog
,Blog::delete()
returns deletedPost
s toUser
,User::delete()
returns deletedBlog
s' posts to the handler, and fanally the handler sends messeges to the actor. If some posts could not be deleted, the handler need make sure to sendDeleteDocument
for each only deleted post. That's complex.The another is: the handler passes the searcher actor to
User::delete()
as an argument,User::delete()
passes it toBlog::delete()
,Blog::delete()
callsPost::delete()
and send aDeleteDocument
message to the actor several times. This is essentially same to above.Introducing a global variable solves this. Using it,
Post::insert()
,update()
anddelete()
need to neither accept a searcher as an agument nor return back itself (or its ID) to the caller. What it needs to do is getting global searcher (or so) and sending a messege that it is deleted. That's it.There are options what struct should be global like a searcher actor, an actor system or a channel. I suggest Riker's channel because...
More benefit
Use of channel introduces pub/sub model. In the case above,
Post
publishes "delete" event to a channel andSearcherActor
subscribes to the event and deletes the search index corresponding to the post. More than that, we can add other actors which concern post's deletion without touching any code of request handlers and models. Examples are:A proof of concept is here: https://git.joinplu.me/KitaitiMakoto/Plume/compare/igalic/refactor/extract-actors...channel. Pay your attention to
Rebasing
What do you think? I you aggree with me, I'm thiking it might be better to restart from current 0.6.0+ code (of course, I will do it). Any questions and discussions are welcome!
that's a good idea, i hope I'll have time to do that soon.
but i do feel like every time i attempt this i have learned something in between, and so it becomes a little bit easier
Thanks! I will start later.