SharePoint full crawl never ends

[Fixed]: SharePoint full crawl never ends – Revealed the technical reasons Why SharePoint Search Crawler never ends

No comments

 927 total views,  3 views today

SharePoint full crawl never ends: Revealed the technical reasons Why SharePoint Search Crawler never ends- in this article, we will learn how to fix the SharePoint full crawl never ends, issue. SharePoint Full or Incremental Crawl Stuck? And also we will learn about how to forcibly stop the SharePoint search crawler using the PowerShell script. Many times, what happens is either an incremental crawl or a full crawl – if it is running, it will never stop – it gets stuck – this behavior is seen in SharePoint 2013 and as well as SharePoint 2016.

For this issue, if we try to stop the search crawl manually through the search crawl menu – it will not stop. In order to stop the search crawl, we need to execute the PowerShell script. This will stop search crawlers either incremental or full but it will end up with other issues like automatically it will again start the full crawl which will never stop and will go on infinite.

Key Highlights: SharePoint crawling never stops fix
SharePoint full crawl never ends
SharePoint full crawl never ends

The traditional fix for the SharePoint full crawl never ends – SharePoint 2016/2013

If we google around for the SharePoint full crawl never ends, everywhere we will get the below troubleshooting tips:

If the above solutions do not work, perform the reset search index from the central administration site. If this solution doesn’t bring the full or incremental back to normal, all experts from the community or forum suggest that we have no luck if the search service is corrupted or damaged, we must create a new search service application and delete the faulty one. Creating the new search service application and bringing back the old configuration will take cumbersome efforts.

Here is a new un-revealed solution that is not available anywhere on google where you don’t need to delete your search service application, we can fix the faulty search service application. Before that let’s understand why SharePoint’s full crawl never ends.

Why SharePoint’s full crawl never ends (SharePoint full crawl never ends)?

First, let’s understand the SharePoint Search database. When we create a search service application in the SharePoint application server, in the SQL backend, the below database will be created automatically:

  • Crawl database
  • Link database
  • Analytics reporting database
  • Search administration database

Crawl database: SharePoint full crawl never ends

This database stores tracking information and historical information about crawled items such as documents and URLs. It also stores information such as the last crawl time, the last crawl ID, and the type of update (add, update, delete) during the last crawl.

Link database:

This database stores unprocessed information that is extracted by the content processing component and information about search clicks. The analytics processing component analyzes this information.

Analytics reporting database:

This database stores the results of usage analysis.

Search administration database

This database stores search configuration data.

All the databases will be created in the below format:

  • <Search_Service Application_Name>_Crawl_GUID
  • <Search_Service Application_Name>_Link_GUID
  • <Search_Service Application_Name>_Analytics_GUID
  • <Search_Service Application_Name>_SearchAdministration_GUID

The SharePoint’s full crawl never ends issue lies in the Crawl database. If we open this database from the SQL server, we can see the below table:

  • Table Name: MSSCrawlURL

In this table along with many columns, we have the column “ErrorCount” whose type is “smallint”, and the “smallint” datatype can hold a maximum value of 32767 (smallint range is “-32,768” to “32767”).

The problem is here in this “ErrorCount” column, while full crawl runs and that generates plenty of crawl errors – all details get logged to this MSSCrawlURL table. When the search service application is no more able to add any records in this table while running the full or incremental crawl as the “ErrorCount” value reached the max (32767), in this scenario, the crawler goes on the infinite loop which never ends.

By now, we have known the technical reason for the SharePoint’s crawl never ends issue,  now in the below section we will learn how to fix this issue.

 How to fix SharePoint’s crawl never ends issue (SharePoint full crawl never ends)?

Login to the SQL server and open the crawl database, update the “ErrorCount” column value from 32767 to 1 wherever it reached the max. If you are an expert in SQL database administration you can handle it otherwise take help from the SQL database administration team. Then using the below PowerShell scripts, stop the crawl (Full and Incremental whichever running), and restart the search and timer service in the application server (detailed steps mentioned in the below sections).

Then don’t do anything, leave it as its – from the next craw schedule the SharePoint search crawl will be running as usual without any issues.

Few Suggestions (Disclaimer):

  • You can take this decision at the final when you have decided there is no other go than to delete the old SSA (search service application) and re-creating the new search service application.
  • You can execute this step at your own risk, the above solution worked for us, so shared it.

SharePoint Incremental Crawl Stuck? Force stop search crawl using PowerShell

Using the below PowerShell script, we can force stop search crawl (incremental/full):


####The below script is used to stop the SharePoint Search content source crawler forcibly######.
Add-PSSnapin Microsoft.SharePoint.Powershell -ErrorAction SilentlyContinue

cls

$PSshell = Get-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorVariable err -ErrorAction SilentlyContinue
if($PSshell -eq $null)
{
Add-PSSnapin "Microsoft.SharePoint.PowerShell"
}

$fileName = "StopSearchCrawlerUsingPowerShell"

#'yyyyMMddhhmm yyyyMMdd
$enddate = (Get-Date).tostring("yyyyMMddhhmmss")
#$filename = $enddate + '_VMReport.doc'
$logFileName = $fileName +"_"+ $enddate+"_Log.txt"
$invocation = (Get-Variable MyInvocation).Value
$directoryPath = Split-Path $invocation.MyCommand.Path

$directoryPathForLog=$directoryPath+"\"+"LogFiles"
if(!(Test-Path -path $directoryPathForLog))
{
New-Item -ItemType directory -Path $directoryPathForLog
#Write-Host "Please Provide Proper Log Path" -ForegroundColor Red
}
#$logPath = $directoryPath + "\" + $logFileName

$logPath = $directoryPathForLog + "\" + $logFileName

$isLogFileCreated = $False

function Write-Log([string]$logMsg)
{
if(!$isLogFileCreated){
Write-Host "Creating Log File..."
if(!(Test-Path -path $directoryPath))
{
Write-Host "Please Provide Proper Log Path" -ForegroundColor Red
}
else
{
$script:isLogFileCreated = $True
Write-Host "Log File ($logFileName) Created..."
[string]$logMessage = [System.String]::Format("[$(Get-Date)] - {0}", $logMsg)
Add-Content -Path $logPath -Value $logMessage
}
}
else
{
[string]$logMessage = [System.String]::Format("[$(Get-Date)] - {0}", $logMsg)
Add-Content -Path $logPath -Value $logMessage
}
}

#########The below section code will continue to stop the search crawler for a particular content source until it stops #######

#Here we need to pass the content source name for which we want to stop the crawling.
$contentSourceName= "Your content source Name" #The default content source is "Local SharePoint sites"

#Get the search service application from your application or search server
$SSA = Get-SPEnterpriseSearchServiceApplication -Identity "" #If you have only one SSA in your server, no need to pass the -Identity parameter, automatically connect to default SSA.

#Get the content source
$contentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource -Identity $contentSourceName

#Check whether the crawler is not running and in "Idle" state
if($contentSource.CrawlState -ne "Idle")
{
Write-Host "Stopping the crawl for the $contentSourceName...."

$contentSource.StopCrawl()

#The below code will continue to execute till its stopped
while($contentSource.CrawlState -ne "Idle")
{
Write-Host "Waiting to $contentSourceName crawl to be stopped......." -f Yellow
sleep 10
}
write-host "The $contentSourceName crawl has been stopped Successfully!" -f Green
}
else
{
Write-Host "The search crawl is not running! for the $contentSourceName" -f Red
}

#########The content sourcer stop crawler code ends here #######

Notes:

  • The above script will continue to execute the code until the particular search crawl doesn’t stop.
  • Sometimes the code execution takes a longer time, for this while the code is running try to restart the SharePoint Search and Timer Job service from the server services console (service.msc).
  • The code must be run inside the SharePoint server.

SharePoint Incremental Crawl Stuck? Disable all continuous crawl in SharePoint using PowerShell

Using the below PowerShell script, we can disable the continuous crawl:


#########The below code will disable all continuous content source crawl #####################

$SSA = Get-SPEnterpriseSearchServiceApplication -Identity "" #If you have only one SSA in your server, no need to pass the -Identity parameter, automatically connect to default SSA.
$contentSourceColl = $SSA | Get-SPEnterpriseSearchCrawlContentSource | Where{$_.Type -eq "SharePoint"}
foreach ($oneContentSource in $contentSourceColl)
{
$oneContentSource.EnableContinuousCrawls = $false
$oneContentSource.Update()
}

#########The below code will disable all continuous content source crawl - ends here ###########

SharePoint Incremental Crawl Stuck? Stop search crawl for All Content sources using the PowerShell Script

Using the below PowerShell script we can stop the search crawl for all running content sources:


###############The below code will stop all content sources crawl forcibly ######################################

#Get the search service application from your application or search server
$SSA = Get-SPEnterpriseSearchServiceApplication -Identity "" #If you have only one SSA in your server, no need to pass the -Identity parameter, automatically connect to default SSA.

#Get the content source collections
$contentSourceCollections = $SSA | Get-SPEnterpriseSearchCrawlContentSource

foreach ($oneContentSource in $contentSourceCollections)
{

#Check whether the crawler is not running and in "Idle" state for one content source
if($oneContentSource.CrawlState -ne "Idle")
{
Write-Host "Stopping the crawl ......"

$oneContentSource.StopCrawl()

#The below code will continue to execute till its stopped
while($oneContentSource.CrawlState -ne "Idle")
{
Write-Host "Waiting to crawl to be stopped......." -f Yellow
sleep 10
}
write-host "The crawl has been stopped Successfully!" -f Green
}
else
{
Write-Host "The search crawl is not running!" -f Red
}

}

###############The below code will stop all content sources crawl forcibly - code ends here ######################

Summary: SharePoint full crawl never ends

Thus, in this article, we have learned about how to fix SharePoint’s search full crawl never ends issue and how to stop the SharePoint search crawl (Incremental/Full) by force using the PowerShell script and also learned the hidden technical reasons for the search full crawl never end issue.

See Also: SharePoint PowerShell Tutorial

You may also like the following SharePoint PowerShell tutorials:

 

 

 

 

About Post Author

Do you have a better solution or question on this topic? Please leave a comment