feat: Add opaque pagination#231
Conversation
…iscussion) Signed-off-by: Paul Horton <phorton@sonatype.com>
Signed-off-by: Paul Horton <phorton@sonatype.com>
`reuslts` is now required in Products Response Signed-off-by: Paul Horton <phorton@sonatype.com>
Signed-off-by: Paul Horton <phorton@sonatype.com>
|
Pagination has been added to the following endpoints:
FYI @ppkarwasz @oej |
taleodor
left a comment
There was a problem hiding this comment.
Hi @madpah , great work there.
I believe we should bring back all instances of id-type and id-value filters as those are not filters and not part of pagination. In relation to pagination, the server would need to take into account whether specific filtering was used. If that gets too complex, maybe we need a separate conversation on this.
Also we may need extra nesting due to behaviour of some generators, I'll try to test this over weekend based on your changes - essentially, at some point of TEA iterations, I was getting things like "inlinePaginationClass" without clear attribution to what class exactly was paginated. I'll get back on this but need a little extra time.
Understood - the purpose of those fields was not immediately clear - so I'll read add and aim to make it clear they are about result filtering and NOT pagination. |
Signed-off-by: Paul Horton <phorton@sonatype.com>
FWIW - the generated libs from this schema seem fine and work for me - if you see issues, let me know - might require a fix upstream in OpenAPI Generator 😉 |
|
So the issue I mentioned is happening here, see below list of classes for Java server generator and notice TeaInlineObject ones: Steps to reproduce: Generally, AI should know how to fix this (it'll create some additional nested structures), but let me know if you need input from me. |
|
What's the issue with the TeaInline* objects being generated @taleodor? This is fine in other languages for sure - so not sure of the issue here. |
Basically, it makes it very hard to produce server code and then makes code unmaintainable because:
|
|
Yeah - I see the inline objects - but don't understand what the issue is here @taleodor - is it because the naming is not deterministic, or something else? |
Yes, essentially we can't build proper server code like that. Like we need to use these classes to build pagination but we can't do it in a meaningful way - it's like working with obfuscated code. |
|
Not sure I agree with the assessment @taleodor - I've used this before in both generated server and client code. The only real issue I've experienced is non-deterministic names of Classes which can be a pain. Do you have a repo or commit somewhere that illustrates your pain that we can reference? |
|
I believe we shouldn't leave it like that in any case since it's public spec and this creates a lot of friction and uncertainty on the implementation side. I'll propose a fix to this PR - should have time over the weekend. |
| name: pageToken | ||
| description: | | ||
| An opaque token used to retrieve the next page of results. | ||
| This should be copied exactly from the `nextPageToken` field of a previous response. |
There was a problem hiding this comment.
Can we define how pageToken interacts with the other query parameters?
For cursor/token pagination, clients need to know whether pageToken already encapsulates the original query state, including filters, sortField, sortOrder, and possibly pageSize.
For example, if the first request uses:
sortField=createdDate&sortOrder=desc&pageSize=25
and the next request sends:
pageToken=...&sortField=version&sortOrder=asc&pageSize=100
what should the server do?
Possible options:
- The server MUST ignore
sortField,sortOrder, filters, andpageSizewhenpageTokenis present. - The server MUST return
400if parameters conflict with the state encoded inpageToken. - The client MUST repeat the same parameters, and mismatch is an invalid request.
I think the spec should choose one behavior before v1.0 to avoid incompatible implementations. Defining this behavior now will ensure forward compatibility and consistent behavior across different TEA server implementations.
| $ref: "#/components/schemas/component" | ||
| It must always be supplied in responses. | ||
| required: | ||
| - hasNext |
There was a problem hiding this comment.
Could we clarify the expected semantics of nextPageToken when hasNext is false?
Right now nextPageToken is required, non-nullable, and described as always supplied. If there is no next page, it is unclear whether the server should return an empty string, a terminal token, or some other value.
Would it be cleaner to make nextPageToken optional and present only when hasNext=true?
For example:
hasNext=true:nextPageTokenMUST be present.hasNext=false:nextPageTokenMUST be absent.
Alternatively, if the intention is to always return a checkpoint token for future polling/sync, maybe that should be modeled separately from a next-page token, because those are slightly different semantics.
| description: | | ||
| A flag (to aid clients) to know whether there is a next page of results to fetch. | ||
|
|
||
| `nextPageToken` will always be supplied, hence this hint is included to aid clients. |
There was a problem hiding this comment.
The hasNext description currently says that nextPageToken will always be supplied.
If we decide that nextPageToken should only be present when hasNext=true, this description should be updated accordingly. Otherwise, the spec should clarify what value nextPageToken has when hasNext=false.
Making nextPageToken optional when hasNext is false would simplify client-side logic, as consumers wouldn't need to handle 'empty' or 'dummy' tokens for terminal pages.
| - $ref: "#/components/parameters/page-size" | ||
| - $ref: "#/components/parameters/page-token" | ||
| - $ref: "#/components/parameters/sort-field-collection" | ||
| - $ref: "#/components/parameters/sort-order" |
There was a problem hiding this comment.
Could we define a deterministic tie-breaker for each sortable resource type?
The addition of sortField is helpful, but fields like createdDate, releaseDate, version, and name are not guaranteed to be unique. Without a stable tie-breaker, cursor pagination can still produce duplicate or missing results when multiple records share the same sort value.
A simple rule could be:
Results MUST be ordered by the selected sortField, then by UUID as a deterministic tie-breaker.
This would make token pagination more reliable and easier to implement consistently.
| - $ref: "#/components/parameters/page-size" | ||
| - $ref: "#/components/parameters/page-token" | ||
| - $ref: "#/components/parameters/sort-field-collection" | ||
| - $ref: "#/components/parameters/sort-order" |
There was a problem hiding this comment.
Could we define a deterministic tie-breaker for each sortable resource type?
The addition of sortField is helpful, but fields like createdDate, releaseDate, version, and name are not guaranteed to be unique. Without a stable tie-breaker, cursor pagination can still produce duplicate or missing results when multiple records share the same sort value.
A simple rule could be:
Results MUST be ordered by the selected sortField, then by UUID as a deterministic tie-breaker.
This would make token pagination more reliable and easier to implement consistently.
| schema: | ||
| type: string | ||
| enum: | ||
| - name |
There was a problem hiding this comment.
sort-field-component:
enum:
- name
and
sort-field-product:
enum:
- name
If name is the default sort field for products and components, do we also need to specify whether names are unique within a TEA server?
If names are not guaranteed to be unique, token pagination still needs a deterministic secondary ordering, such as UUID. If names are intended to be unique, that constraint should probably be documented in the corresponding product/component schema descriptions.
| enum: | ||
| - asc | ||
| - desc | ||
| default: asc |
There was a problem hiding this comment.
I noticed sortOrder defaults to asc globally. Would it be worth considering resource-specific defaults instead?
For security-related resources like product/component releases (often sorted by createdDate) or collections (sorted by version), consumers typically expect the most recent data first (newest-to-oldest).
If a universal asc default is preferred for simplicity, perhaps we could add a small note in the description to help set expectations for implementers/consumers.
This PR (currently) adds Opaque Pagination to the
/productsendpoint as described in #179 - for review and comment before rolling it out to the other endpoints.