From 14e911a0d8c06956b869a6a6c2afa6cfaadf9edd Mon Sep 17 00:00:00 2001
From: epriestley <git@epriestley.com>
Date: Fri, 22 Jun 2018 07:21:24 -0700
Subject: [PATCH] Index only the first 1,000 comments on any object

Summary:
Depends on D19502. Ref T13151. See PHI719. An install ended up with an object with 111,000+ comments on it because someone wrote a script to treat it like a logfile.

Although we seem to do mostly okay with this (locally, it only takes about 30s to index a similar object) we'll hit a wall somewhere (since we need to hold everything in memory), and it's hard to imagine a legitimate object with more than 1,000 comments. Just ignore comments past the first thousand.

(Conpherence threads may legitimately have more than 1,000 comments, but go through a different indexer.)

Test Plan:
  - Piped some comments into `maniphest.edit` in a loop to create a task with 100K comments.
  - Ran `bin/search index Txxx --force` to reindex it, with `--trace`.
    - Before: task indexed in about 30s.
    - After: script loaded comments with LIMIT 1000 and indexed in a couple seconds.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13151

Differential Revision: https://secure.phabricator.com/D19503
---
 ...torTransactionsFulltextEngineExtension.php | 22 ++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/src/applications/transactions/engineextension/PhabricatorTransactionsFulltextEngineExtension.php b/src/applications/transactions/engineextension/PhabricatorTransactionsFulltextEngineExtension.php
index c9f43f49b2..4eeda3251c 100644
--- a/src/applications/transactions/engineextension/PhabricatorTransactionsFulltextEngineExtension.php
+++ b/src/applications/transactions/engineextension/PhabricatorTransactionsFulltextEngineExtension.php
@@ -22,12 +22,28 @@ final class PhabricatorTransactionsFulltextEngineExtension
       return;
     }
 
-    $xactions = $query
+    $query
       ->setViewer($this->getViewer())
       ->withObjectPHIDs(array($object->getPHID()))
       ->withComments(true)
-      ->needComments(true)
-      ->execute();
+      ->needComments(true);
+
+    // See PHI719. Users occasionally create objects with huge numbers of
+    // comments, which can be slow to index. We handle this with reasonable
+    // grace: at time of writing, we can index a task with 100K comments in
+    // about 30 seconds. However, we do need to hold all the comments in
+    // memory in the AbstractDocument, so there's some practical limit to what
+    // we can realistically index.
+
+    // Since objects with more than 1,000 comments are not likely to be
+    // legitimate objects with actual discussion, index only the first
+    // thousand comments.
+
+    $query
+      ->setOrderVector(array('-id'))
+      ->setLimit(1000);
+
+    $xactions = $query->execute();
 
     foreach ($xactions as $xaction) {
       if (!$xaction->hasComment()) {
-- 
2.51.2