Wednesday, July 6, 2016

Running Solr 6.1 as a Windows Service

Goal

Get Solr 6.1 running as a Windows Service and have it recover on OutOfMemory Exceptions.

Install Solr 6.1 as a Service

  1. Download NSSM
  2. Extract nssm.exe somewhere
  3. Create solr_start_6.1.0.cmd (this is a Windows Command file that does all of my configuration--yours will definitely be different):
    C:\apache\solr-6.1.0\bin\solr start -f -h node3 -c -p 8983 -z "zk1:2181,zk2:2181,zk3:2181" -m 128m -s C:\apache\solr-6-cores
    NOTE: The -f to run the script in the foreground.  I set the JVM heap size to 128 MB (we want this thing to crash and burn to test out OutOfMemoryException restart!).
  4. Test your script to make sure it starts solr and you can access your Solr Admin UI in a web browser
  5. Open a command window and navigate to your nssm.exe directory
  6. nssm.exe install "Apache - Solr 6.1"
    1. Application Tab
      1. Path: Select your solr_start_6.1.0.cmd from earlier
      2. Startup Directory: set it to the directory containing your script (should populate by default)
    2. Details Tab
      1. Display Name: Apache - Solr 6.1
      2. Startup type: Automatic
    3. Log on Tab
      1. Make sure you specify an account that has administrator-level permissions (Use your account if you're stuck here--but make sure to set it to something production-worthy later)
    4. I/O Tab
      1. I/O Redirection
        1. Output (stdout): Set this to something like path\to\my\solr\cmd\script\dir\solr-6.1.0-out.txt
        2. Error (stderr): path\to\my\solr\cmd\script\dir\solr-6.1.0-error.txt
      2. File rotation
        1. Check Rotate files
        2. Check Rotate while service is running
        3. Restrict rotation to files bigger than: (use common sense here, I did 5 MB, so 5242880 went into the box)
    5. Click Install Service
  7. Open Component Services and select Apache - Solr 6.1
  8. Start the service
  9. Validate that it came up by going to your Admin UI webpage

Make Solr Service respond to Out Of Memory Exceptions

  1. Navigate to this JIRA ticket
  2. Download oom_win.cmd and place it in your solr\bin directory next to solr.cmd
  3. Open solr.cmd in a text editor
  4. Find all the places where the script starts the server:
    1. Search for /solr_gc.log
  5. Immediately after /solr_gc.log, paste the following:
    -XX:OnOutOfMemoryError="%SOLR_SERVER_DIR%\..\bin\oom_win.cmd %SOLR_PORT% !SOLR_LOGS_DIR!"
    1. I had to replace two lines.  NOTE that this is just the manual way of applying the patch file associated with the JIRA ticket above.  If you want, apply it however you want.
  6. Now that we've made our changes, go ahead and restart our new Solr 6.1 service so it knows to kill the process on OutOfMemory errors.
  7. To force an OutOfMemoryError, query *:* and return 1000000 rows
    1. If you have a decent amount of content, this should force an OutOfMemory exception.  If you don't have a lot of content, do whatever you can to make it do a lot of memory-intensive work.  Perhaps consider lowering the JVM memory, too.
    2. You should see the web server go offline temporarily and then come back online
  8. Now that you've seen it restart and come back online, let's give the JVM a good amount of RAM so that it doesn't run out of memory every other request.  Just edit your solr_start_6.1.0.cmd file and change the -m 128m to -m 4g (128 MB to 4 GB)
  9. Save and restart the service
  10. Confirm that you have the new amount of RAM for the JVM by visiting the Dashboard tab in the Admin UI

Logs

When the OutOfMemory Killer runs, it generates a log file in the normal log directory.  Navigate to that directory and you should see a file that looks something like: solr_oom_killer-9000-2016-07-06_13_59_39.  Now you can know when this script runs and hopefully anticipate it in the future or make changes to not get it.

Final step (important!)

Do a happy dance!

Tuesday, May 24, 2016

Apache Solr - RangeQuery

RangeQuery appears to be a range query that operates on strings.  I'm not sure if it only works on strings or if the strings are a way for it to auto-determine what field type it's going to work with.
If you know how to properly use this, please leave a comment below.  I'm stumped on this one.

Fields

  • fieldName (required in hierarchy here or a parent)
  • lowerTerm - string value
  • upperTerm - string value
  • includeLower (optional, default true)
  • includeUpper (optional, default true)

Examples

Simply paste the following into the q= field in the Admin UI.

Here's a query that I came up with, but it doesn't work as I'd expect.

{!xmlparser}
<RangeQuery
 fieldName="price"
 lowerTerm="1.00"
 upperTerm="3.00"
 includeLower="true"
 includeUpper="true">
</RangeQuery>

The above query returns no results, which isn't what I was expecting.


{!xmlparser}
<RangeQuery
 fieldName="price"
 lowerTerm="0"
 upperTerm="10.00"
 includeLower="true"
 includeUpper="true">
</RangeQuery>

The above query returns 30 documents, of which includes prices > 10.00, which isn't what I expected.

Apache Solr - MatchAllDocsQuery

MatchAllDocsQuery is probably the simplest query there is.  All it does is quickly match all documents in an index.

Example

Simply paste the following into the q= field in the Admin UI.

{!xmlparser}
<MatchAllDocsQuery></MatchAllDocsQuery>


Or, using the standard lucene syntax: *:*

Apache Solr - LegacyNumericRangeQuery

NumericRangeQuery was renamed to LegacyNumericRangeQuery about lucene 6.0.0 and marked deprecated. If you're on an older system, you will find it using NumericRangeQuery. Afterwards, LegacyNumericRangeQuery will be what you will want to use.

It looks like NumericRangeQuery has been replaced with PointRangeQuery.

Fields

  • fieldName (required here or in a parent node)
  • lowerTerm (optional, default null)
  • upperTerm (optional, default null)
  • includeLower (optional, default true)
  • includeUpper (optional, default true)
  • precisionStep (optional, default 16)
  • type (optional, default "int") - long | int | double | float

Examples

Simply paste the following into the q= field in the Admin UI.
NOTE: These examples will not work in Solr since Solr does not support Point types.

Lucene 6.0.0+:
{!xmlparser}
<LegacyNumericRangeQuery
 fieldName="price"
 lowerTerm="0"
 upperTerm="10"
 includeLower="true"
 includeUpper="true"
 precisionStep="16"
 type="float">
</LegacyNumericRangeQuery>

Before Lucene 6.0.0:
{!xmlparser}
<NumericRangeQuery
 fieldName="price"
 lowerTerm="0"
 upperTerm="10"
 includeLower="true"
 includeUpper="true"
 precisionStep="16"
 type="float">
</NumericRangeQuery>

Apache Solr - PointRangeQuery

PointRangeQuery is new as of about Apache Solr 6.0 and is meant to replace the, now deprecated, NumericRangeQuery (which got renamed to LegacyNumericRangeQuery).

NOTE: The following may be confusing, and here's why.  The PointRangeQueryBuilder, used by XmlQueryParser, requires that lowerTerm and upperTerm are specified, which means that neither of them may default.  However, in the code, it covers its bases in case lowerTerm or upperTerm was not specified.  In these cases, it defaults to the MIN_VALUE and MAX_VALUE for the respective types.  Until the code is updated after 6.0.0, the defaults cannot be reached.  If you are reading this after they become optional, the following may be beneficial to you.  Until then, you will be required to enter a value for both.

Fields

  • fieldName (required in hierarchy here or in a parent node)
  • lowerTerm (required)
  • upperTerm (required)
  • type (optional, default "int") - long | int | double | float
    • long
      • If lowerTerm is not specified, default to Long.MIN_VALUE (-9223372036854775808)
      • If upperTerm is not specified, default to Long.MAX_VALUE (9223372036854775807)
    • int
      • If lowerTerm is not specified, default to Integer.MIN_VALUE (-2147483648)
      • If upperTerm is not specified, default to Integer.MAX_VALUE (2147483647)
    • double
      • If lowerTerm is not specified, default to Double.NEGATIVE_INFINITY
      • If upperTerm is not specified, default to Double.POSITIVE_INFINITY
    • float
      • If lowerTerm is not specified, default to Float.NEGATIVE_INFINITY
      • If upperTerm is not specified, default to Float.POSITIVE_INFINITY

Example

Simply paste the following into the q= field in the Admin UI.
NOTE: This example should work, but it doesn't in my Solr 6.0.0 build because Point types aren't supported in Solr yet.  I do know that the PointRangeQuery code is running when I submit this query, because it gives me error messages when I don't specify required parameters.  However, somewhere in between, something isn't supported and no documents are returned.  Please leave a comment if you know more.

{!xmlparser}
<PointRangeQuery
 fieldName="price"
 lowerTerm="0.00"
 upperTerm="100.00"
 type="float">
</PointRangeQuery>

Apache Solr - TermsQuery

TermsQuery is essentially a list of TermQuery with a couple of extra options.
It allows you to specify a group of terms on a field and require a minimum number of matches.

Fields

  • fieldName (required here or in a parent node)
  • disableCoord (optional, default false)
  • minimumNumberShouldMatch (optional, default 0)
  • boost (optional, default 1.0)
  • Value: list of tokens (space-delimited will do it)

Example

Simply paste the following into the q= field in the Admin UI.
{!xmlparser}
<TermsQuery
 fieldName="series_t"
 disableCoord="true"
 boost="1.2"
 minimumNumberShouldMatch="2">song ice fire ender black company</TermsQuery>

Monday, May 23, 2016

Apache Solr - TermQuery

TermQuery is a very simple query that matches documents containing a term.
A term represents a word found in a field.  A term has a field property and a text property.

Fields


Examples

Simply paste the following into the q= field in the Admin UI.
Query ParserSyntax
XmlQueryParser
{!xmlparser}
<TermQuery fieldName="_text_" boost="1.3">test</TermQuery>
Lucene _text_:test^1.3