Conversation
Phil-V
force-pushed
the
IA-4847-entities-list-performance
branch
from
11d174c to
1d918b7
Compare
| @property | ||
| def pending_duplicate_ids(self): | ||
| """Retrieve the id list of related pending duplicate entities.""" | ||
| results = set() |
There was a problem hiding this comment.
return list(
self.duplicates1.filter(validation_status=ValidationStatus.PENDING)
.values_list("id", flat=True)
.union(
self.duplicates2.filter(validation_status=ValidationStatus.PENDING)
.values_list("id", flat=True)
)
)instead ?
There was a problem hiding this comment.
Sadly, using values_list doesn't seem to work with Prefetch. But, I've refined the code to filter on validation_status in the prefetch itself. Let me know if you can think of better ways to architecture this.
| for instance in self.instances.all() | ||
| if (saved_at := instance.source_created_at or instance.created_at) is not None | ||
| ) | ||
| return max(instance_dates, default=self.created_at) |
There was a problem hiding this comment.
I'd try to turn this into a sql operation (django ORM) as it might be expensive if there are a lot of instances. Moreover if you do that on a list it might trigger N+1 query problem.
Be also aware that, imho, we should avoid putting @property on something that triggers a DB operation. (E.g some debuggers immediately evaluates it and then you don't get why you have more queries than expected)
There was a problem hiding this comment.
Good point on the properties, I've moved them to regular methods.
| # Handle streaming responses | ||
|
|
||
| queryset = self.filter_queryset(self.get_queryset()) | ||
| queryset = self.filter_queryset(self.get_queryset()).order_by("-id") |
There was a problem hiding this comment.
just to be sure, this would erase any other ordering passed, do we want this ?
There was a problem hiding this comment.
Added a comment to clarify that this is for performance reasons (as there are few indexed columns available on the Entity model and the exports iterate over potentially a very high number of records).
4 participants
What problem is this PR solving?
Improve the entity list performance.
Related JIRA tickets
IA-4847
Changes
-idto avoid more expensive ordering if not necessaryHow to test
Visit the entities list page and:
Print screen / video
/
Notes
-idinstead of-created_atas the latter is not indexed (to be potentially addressed in future task)Doc
/