Plume/Plume
9
18
Fork 24
Code Issues 177 Pull requests 8 Releases 8 Wiki Activity

Paginate the outbox responses. Fixes #669 #681

Merged
epsilon-phase merged 7 commits from Outbox-pagination into master 2019-10-30 10:22:29 +00:00
Conversation 11 Commits 7 Files changed 7 +186 -26
epsilon-phase commented 2019-10-23 20:39:58 +00:00 (Migrated from github.com)
Copy link

This seems to work on our end, so let us know if there's anything else that needs fixing.

This seems to work on our end, so let us know if there's anything else that needs fixing.
👍 1 ❤️ 1
codecov[bot] commented 2019-10-23 21:07:47 +00:00 (Migrated from github.com)
Copy link

Codecov Report

Merging #681 into master will decrease coverage by 0.45%.
The diff coverage is 1.65%.

@@            Coverage Diff             @@
##           master     #681      +/-   ##
==========================================
- Coverage    39.8%   39.35%   -0.46%     
==========================================
  Files          72       72              
  Lines        9383     9494     +111     
  Branches     2219     2263      +44     
==========================================
+ Hits         3735     3736       +1     
- Misses       4592     4702     +110     
  Partials     1056     1056
# [Codecov](https://codecov.io/gh/Plume-org/Plume/pull/681?src=pr&el=h1) Report > Merging [#681](https://codecov.io/gh/Plume-org/Plume/pull/681?src=pr&el=desc) into [master](https://codecov.io/gh/Plume-org/Plume/commit/866465c603cf169d24a65b8f77d4e462c7422f52?src=pr&el=desc) will **decrease** coverage by `0.45%`. > The diff coverage is `1.65%`. ```diff @@ Coverage Diff @@ ## master #681 +/- ## ========================================== - Coverage 39.8% 39.35% -0.46% ========================================== Files 72 72 Lines 9383 9494 +111 Branches 2219 2263 +44 ========================================== + Hits 3735 3736 +1 - Misses 4592 4702 +110 Partials 1056 1056 ```
trinity-1686a commented 2019-10-24 10:44:08 +00:00
Owner
Copy link

Thanks for participating to Plume dev!

There seems to be a few issues with you patch:

  • it returns the oldest posts first, which is the opposite of what other software are doing (but I'm not sure how much of an issue that actually is)
  • it does not link to previous/next page, so only 12 posts may be accessible. I'm not very familiar with the Activitypub crate, but I guess there is something about using OrderedCollectionPage instead/in addition to OrderedCollection
Thanks for participating to Plume dev! There seems to be a few issues with you patch: - it returns the oldest posts first, which is the opposite of what other software are doing (but I'm not sure how much of an issue that actually is) - it does not link to previous/next page, so only 12 posts may be accessible. I'm not very familiar with the Activitypub crate, but I guess there is something about using `OrderedCollectionPage` instead/in addition to `OrderedCollection`
👍 2
elegaanz commented 2019-10-24 14:50:16 +00:00 (Migrated from github.com)
Copy link

For ActivityPub, you need to have both an OrderedCollection on /@/USER/outbox, with a first field, containing a link to the first page of the outbox, and an endpoint like /@/USER/outbox/page=PAGE serving a specific OrderedCollectionPage, that should have the items (an array of activities), prev (a link to the previous page, if any) and next (a link to the next page, if any) fields.

I don't know if that's clear, I can try to explain it in another way if you want.

For ActivityPub, you need to have both an `OrderedCollection` on `/@/USER/outbox`, with a `first` field, containing a link to the first page of the outbox, and an endpoint like `/@/USER/outbox/page=PAGE` serving a specific `OrderedCollectionPage`, that should have the `items` (an array of activities), `prev` (a link to the previous page, if any) and `next` (a link to the next page, if any) fields. I don't know if that's clear, I can try to explain it in another way if you want.
epsilon-phase commented 2019-10-24 17:26:54 +00:00 (Migrated from github.com)
Copy link

That's a good start for sure then.

So it does need to also request that the most recent posts are retrieved in an earlier page too right?

That's a good start for sure then. So it does need to also request that the most recent posts are retrieved in an earlier page too right?
epsilon-phase commented 2019-10-24 18:28:08 +00:00 (Migrated from github.com)
Copy link

Okay, so, we think we understand what we need to do, but we're not sure what the proper kind of Link object is for this.

Okay, so, we think we understand what we need to do, but we're not sure what the proper kind of `Link` object is for this.
epsilon-phase commented 2019-10-24 19:37:31 +00:00 (Migrated from github.com)
Copy link

With any luck that does it

With any luck that does it
igalic (Migrated from github.com) reviewed 2019-10-27 17:37:39 +00:00
igalic (Migrated from github.com) left a comment
Copy link

❓

❓
plume-models/src/users.rs
igalic (Migrated from github.com) commented 2019-10-27 17:35:13 +00:00
Copy link

12 is hard coded. for now.?

12 is hard coded. for now.?
src/routes/user.rs
@ -557,2 +566,4 @@
user.outbox_page(&*rockets.conn, page.limits()).ok()
}
#[post("/@/<name>/inbox", data = "<data>")]
pub fn inbox(
igalic (Migrated from github.com) commented 2019-10-27 17:37:08 +00:00
Copy link

does a blog also have such an outbox that could benefit from such a pagination?

does a blog also have such an outbox that could benefit from such a pagination?
epsilon-phase (Migrated from github.com) reviewed 2019-10-27 19:51:29 +00:00
plume-models/src/users.rs
epsilon-phase (Migrated from github.com) commented 2019-10-27 19:51:29 +00:00
Copy link

Oh, we didn't know about the constant elsewhere... But it's in the routes code rather than the models code.

We'll move it over then.

Oh, we didn't know about the constant elsewhere... But it's in the routes code rather than the models code. We'll move it over then.
epsilon-phase (Migrated from github.com) reviewed 2019-10-27 20:00:44 +00:00
src/routes/user.rs
@ -557,2 +566,4 @@
user.outbox_page(&*rockets.conn, page.limits()).ok()
}
#[post("/@/<name>/inbox", data = "<data>")]
pub fn inbox(
epsilon-phase (Migrated from github.com) commented 2019-10-27 20:00:44 +00:00
Copy link

True enough, we'll get that made then....

... get_activities is a method stub for blogs. Huh.
In any case, we'll write the method stubs for now and see what's up with that.

True enough, we'll get that made then.... ... get_activities is a method stub for blogs. Huh. In any case, we'll write the method stubs for now and see what's up with that.
trinity-1686a reviewed 2019-10-27 21:44:04 +00:00
trinity-1686a left a comment
Owner
Copy link

overall this seems good, but I think there are some issues with page numbers at some places

overall this seems good, but I think there are some issues with page numbers at some places
plume-models/src/blogs.rs
trinity-1686a commented 2019-10-27 21:42:50 +00:00
Owner
Copy link

I think this does not give the right result, divisions are flooring where in this case we need them ceiling. An easy way to achive that is to add ITEMS_PER_PAGE-1 before dividing

                (self.get_activities(conn)?.len() + ITEMS_PER_PAGE - 1) as u64 / ITEMS_PER_PAGE as u64
I think this does not give the right result, divisions are flooring where in this case we need them ceiling. An easy way to achive that is to add ITEMS_PER_PAGE-1 before dividing ```suggestion (self.get_activities(conn)?.len() + ITEMS_PER_PAGE - 1) as u64 / ITEMS_PER_PAGE as u64 ```
plume-models/src/users.rs
trinity-1686a commented 2019-10-27 21:25:29 +00:00
Owner
Copy link

why is it commented? It should probably either stay or be deleted

why is it commented? It should probably either stay or be deleted
trinity-1686a commented 2019-10-27 21:35:36 +00:00
Owner
Copy link

Previous page should not be added if we are already on the first page, and a next page should probable not be added either if get_activities_page returned less entries than it could have.

Also I think there is an out-of-one on the page index, if we are on page n°1 (that is the first with current implementation of outbox above), min is going to be 0, so next page will be 0/12+1 = n°1 again, and previous would be n°-1

Previous page should not be added if we are already on the first page, and a next page should probable not be added either if `get_activities_page` returned less entries than it could have. Also I think there is an out-of-one on the page index, if we are on page n°1 (that is the first with current implementation of `outbox` above), min is going to be 0, so next page will be 0/12+1 = n°1 again, and previous would be n°-1
epsilon-phase (Migrated from github.com) reviewed 2019-10-28 00:03:52 +00:00
plume-models/src/users.rs
epsilon-phase (Migrated from github.com) commented 2019-10-28 00:03:52 +00:00
Copy link

That makes sense. Should be fixed now

That makes sense. Should be fixed now
epsilon-phase (Migrated from github.com) reviewed 2019-10-28 00:04:02 +00:00
plume-models/src/blogs.rs
epsilon-phase (Migrated from github.com) commented 2019-10-28 00:04:02 +00:00
Copy link

Alright, that makes sense too.

Alright, that makes sense too.
epsilon-phase (Migrated from github.com) reviewed 2019-10-28 00:04:35 +00:00
plume-models/src/users.rs
epsilon-phase (Migrated from github.com) commented 2019-10-28 00:04:34 +00:00
Copy link

That makes sense.

That makes sense.
trinity-1686a reviewed 2019-10-28 07:36:51 +00:00
plume-models/src/users.rs
trinity-1686a commented 2019-10-28 07:36:51 +00:00
Owner
Copy link

the part about not including previous/next when unnecessary is fixed, but I don't think you addressed the other half about page index

the part about not including previous/next when unnecessary is fixed, but I don't think you addressed the other half about page index
elegaanz (Migrated from github.com) reviewed 2019-10-28 22:17:33 +00:00
elegaanz (Migrated from github.com) left a comment
Copy link

Globally, what you have done so far seems to work well. I left a few comments.

The only thing that really needs to be fixed is that fetching old posts for a remote user is no longer working.

Globally, what you have done so far seems to work well. I left a few comments. The only thing that really needs to be fixed is that fetching old posts for a remote user is no longer working.
plume-models/src/users.rs
elegaanz (Migrated from github.com) commented 2019-10-28 22:14:21 +00:00
Copy link

It starts at 1 for users, but 0 for blogs. Either are fine, but it would be better if it was consistent.

It starts at 1 for users, but 0 for blogs. Either are fine, but it would be better if it was consistent.
src/routes/blogs.rs
elegaanz (Migrated from github.com) commented 2019-10-28 22:16:13 +00:00
Copy link

Could you please call Blog::outbox_page here, even if it doesn't do anything yet?

Could you please call `Blog::outbox_page` here, even if it doesn't do anything yet?
epsilon-phase (Migrated from github.com) reviewed 2019-10-29 02:22:35 +00:00
src/routes/blogs.rs
epsilon-phase (Migrated from github.com) commented 2019-10-29 02:22:35 +00:00
Copy link

Sure thing.

Sure thing.
epsilon-phase (Migrated from github.com) reviewed 2019-10-29 02:24:35 +00:00
plume-models/src/users.rs
epsilon-phase (Migrated from github.com) commented 2019-10-29 02:24:35 +00:00
Copy link

Looks like it should be one given the current Page code.

Let's make it that then.

Looks like it should be one given the current `Page` code. Let's make it that then.
epsilon-phase commented 2019-10-29 02:31:54 +00:00 (Migrated from github.com)
Copy link

@AnaGelez You wouldn't have any recommendations on fixing the retrieval of older messages would you?

@AnaGelez You wouldn't have any recommendations on fixing the retrieval of older messages would you?
elegaanz commented 2019-10-29 13:26:26 +00:00 (Migrated from github.com)
Copy link

The method that needs changing is this one: https://github.com/Plume-org/Plume/blob/master/plume-models/src/users.rs#L333

It should be modified so that it fetches all the pages, and puts all the activities in the returned Vec.

The method that needs changing is this one: https://github.com/Plume-org/Plume/blob/master/plume-models/src/users.rs#L333 It should be modified so that it fetches all the pages, and puts all the activities in the returned `Vec`.
👍 1
elegaanz commented 2019-10-29 20:38:36 +00:00 (Migrated from github.com)
Copy link

There is an issue with the last/first/prev/next links: the https:// part is doubled (for instance I have https://https://plume.one/@/admin_one/outbox?page=1).

I think it is because User::outbox_url already includes https://, and thus the call to ap_url is superfluous, you can use format! directly.

There is an issue with the `last`/`first`/`prev`/`next` links: the `https://` part is doubled (for instance I have `https://https://plume.one/@/admin_one/outbox?page=1`). I think it is because `User::outbox_url` already includes `https://`, and thus the call to `ap_url` is superfluous, you can use `format!` directly.
epsilon-phase commented 2019-10-29 21:09:13 +00:00 (Migrated from github.com)
Copy link

That's a silly mistake to have made. Should be fixed now.

That's a silly mistake to have made. Should be fixed now.
elegaanz commented 2019-10-29 21:34:00 +00:00 (Migrated from github.com)
Copy link

OK, it now fetches the pages correctly, but it loops infinitely, because on the last page last point the current page (which, indeed, is the last).

OK, it now fetches the pages correctly, but it loops infinitely, because on the last page `last` point the current page (which, indeed, is the last).
elegaanz (Migrated from github.com) approved these changes 2019-10-30 10:22:09 +00:00
elegaanz (Migrated from github.com) left a comment
Copy link

It seems to work fine now! Thank you very much for your help (and your patience) 😊

It seems to work fine now! Thank you very much for your help (and your patience) :blush:
Sign in to join this conversation.
Reviewers
No reviewers
elegaanz
Labels
Clear labels   
A: API

Related to the REST API

  
A: Backend

Code running on the server

  
A: Federation

Stuff related to Federation

  
A: Front-End

Related to the front-end

  
A: I18N

Translations, and related code

  
A: Meta

More about project management or code than the project itself

  
A: Security

   
Build

The building, or installation process of Plume

  
C: Bug

Something isn't working

  
C: Discussion

We need to talk

  
C: Enhancement

New feature or request

  
C: Feature

This is a new feature

  
Compatibility

Compatibility with different browsers, readers and OS

  
Dependency

Related to an external package that Plume uses

  
Design

UI/UX related issues and PRs

  
Documentation

   
Good first issue

Good for newcomers

  
Help welcome

Extra attention is needed

  
Mobile

Issues affecting only mobile UX

  
Rendering

How elements're rendered out for the end user

  
S: Blocked

Something else needs to be fixed first

  
S: Duplicate

This issue or pull request already exists

  
S: Incomplete

This PR is not complete yet

  
S: Instance specific

Issues concern a limited number of instances

  
S: Invalid

This doesn't seem right

  
S: Needs Voting/Discussion

Need to be discussed by the community (on Loomio)

  
S: Ready for review

This PR is ready to be reviewed

  
Suggestion

Proposed ideas worth considering

  
S: Voted on Loomio

This is issue has been created after a vote on Loomio

  
S: Wontfix

This will not be worked on

No labels
A: API
A: Backend
A: Federation
A: Front-End
A: I18N
A: Meta
A: Security
Build
C: Bug
C: Discussion
C: Enhancement
C: Feature
Compatibility
Dependency
Design
Documentation
Good first issue
Help welcome
Mobile
Rendering
S: Blocked
S: Duplicate
S: Incomplete
S: Instance specific
S: Invalid
S: Needs Voting/Discussion
S: Ready for review
Suggestion
S: Voted on Loomio
S: Wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Plume/Plume#681
No description provided.
Delete branch "Outbox-pagination"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?