How to View More Than 1,000 Search Results

Overview

An issue exists where searches with more than 1,000 results will not display a "Next Page" option once the thousandth result is reached. The issue can occur when many pages are used to reach the 1,000 threshold or when the 1,000 threshold is reached in one search.

Workaround

As a temporary measure, you can work around this limit by using another cursor in the query to identify where one query ends and the next begins. The steps below outline how to use the createdAt value to accomplish this while a permanent solution is developed.

Perform a search sorting all entries in ascending order by their createdAt value that includes the following parameters:
- "include"=>"details",
- "orderBy"=>"createdAt",
- "direction"=>"ASC",
- "limit" => \<SOME_VALUE>
Use the last value in the search results to start a new search using the parameter below. The results will include the next set of values that match the search critera:
- createdAt >= ”\<Last_Value_In_Previous_Search>”
  Important: The >= operator ensures that if two submissions have the same createdAt value, the search returns both submissions. It also means that each search will contain the same submission across the Previous Search and Next Search results, causing duplicate submissions in our collection. The duplicates will need to be removed after the data is retrieved. You can determine duplicates using their submissionId values.
Continue searching using the createdAt >= ”\<Last_Value_In_Previous_Search>”parameter until no further submissions are returned.

Example

The example below was written in Ruby using the Kinetic SDK. You can extend the sample into other languages and/or the API to fit other use cases.

Note: If using the API instead of the SDK, replace “CORE_SDK.find_form_submissions” with the API call.

# Create an empty array to contain all submissions even if > 1000 are found
COMBINED_SUBMISSIONS = []


# Get submissions using createdAt value as a workaround for the pagination bug affecting > 1000 entries
def get_submissions(query)
  kapp_slug   = query["kapp_slug"]
  form_slug   = query["form_slug"]
                # set limit at 1000
  limit       = 1000
                # Parameters must include details, orderBy, direction, and limit. Other includes and parameters may be added.
  params      = {"include"=>"details", "orderBy"=>"createdAt", "direction"=>"ASC", "limit" => limit}
                # Append createdAt to the Parameters if passed to method. (used by recursion to get next chunk/page)
                # Note: >= is used in order to account for possible submissions made at the same time. Therefore duplicate entries will also be found. Below duplicates are scrubbed.
  params["q"] = "createdAt>=\"#{query["start_date"]}\"" if query["start_date"]


  # Perform the query
  response = CORE_SDK.find_form_submissions(kapp_slug, form_slug, params)
  # Extract the submissions from the response
  submissions = response.content['submissions']
  # Add the submissions from the query to COMBINED_SUBMISSIONS which contains all submissions
  COMBINED_SUBMISSIONS.push(*submissions)


  # If more submissions are found perform another query else return to end queries
  if submissions.length == limit
    # Append the last/greatest createdAt date as a starting date for the next query
    query["start_date"] = (submissions.last)["createdAt"]
    # Recursively run the query
    get_submissions(query)
  else
    # No more entries found return the submissions ensuring only unique submissions are included
    return COMBINED_SUBMISSIONS.uniq!{|submission| [submission["id"]]}
  end


end
# Run the query to get submissions
get_submissions({"kapp_slug" => kapp_slug, "form_slug" => form_slug})
# Output the results to the screen
puts "Results: #{COMBINED_SUBMISSIONS}" 
puts "Results: #{COMBINED_SUBMISSIONS.length}"parameters