Solr and Load Balancer Log Files in SearchStax Cloud

---

SearchStax Cloud log files help you tune Solr performance and isolate potential bugs in your application. The SearchStax Help Center references many typical errors and other log entries very specifically.

The Solr log files may be viewed in a limited way from the Solr dashboard. Recent solr.log files may be downloaded from SearchStax Cloud. There is a special log in SearchStax Cloud that records your team’s actions in the SearchStax Cloud Dashboard. Some other logs are available through the agency of SearchStax support engineers.

Contents:

1. Solr Server Logs
   • Log Retention
   • Logging Levels
2. Load Balancer Logs
3. Nginx Logs
4. SearchStax Cloud Activity Log

Solr Server Logs

Each Solr instance maintains a solr.log file, managed by Apache Log4j.

This file accumulates log messages at the INFO, WARN, and ERROR levels until the size of the file reaches 33.55 MB or until Solr is restarted. At that point, Solr closes the file, renames it solr.log.N, and opens a new solr.log file. You can see the files in the SearchStax Cloud dashboard, Dedicated Infrastructure > Deployment > Deployment Details > Logs screen.

Solr.log is the live file, containing the most-recent log entries. The rollover files are numbered sequentially, so the one with the highest number is the most-recent one.

The files in this list are a snapshot of the Solr log files on the server. This snapshot is updated at the top of every hour. This means that when you experiment with your system, you must wait until the top of the hour to see changes in the logs. (If you are in India, the logs become available at the bottom of every hour.)

Log Retention

The topic of “log retention” using Log4j can be a bit complicated. Please see Configuring Logging for all the details.

As expressed in SearchStax Cloud, a new solr.log file is initiated each time a Solr instance is started. That file continues to record log messages until either Solr stops, or until the size of solr.log exceeds 33.55 MB. The solr.log file of a small, stable system can accumulate log messages for weeks or months without rolling over.

When solr.log reaches 33.55 MB, it rolls over.

• The original solr.log file is automatically renamed solr.log.1.
• A new solr.log file is created and begins to record messages.

The next time the log rolls over:

• The current solr.log file is renamed solr.log.2.
• A new solr.log file is initiated.

Note that the solr.log contains the current log messages. The most recent archived messages are in the file with the largest extension number.

This process continues until there is one solr.log file and ten numbered rollover files:

solr.log	Current entries. “Current” might mean hours to months, depending on log message volume.
solr.log.1	Oldest entries still retained by the system.
solr.log.2	A little newer than solr.log.1.
solr.log.3	Newer than solr.log.2.
solr.log.4	Newer than solr.log.3.
solr.log.5	Newer than solr.log.4.
solr.log.6	Newer than solr.log.5.
solr.log.7	Newer than solr.log.6.
solr.log.8	Newer than solr.log.7.
solr.log.9	Newer than solr.log.8.
solr.log.10	Newest rollover entries from solr.log.

SearchStax Cloud retains a maximum of ten rollover files. Therefore, when solr.log rolls over for the eleventh time, something has to give.

• Solr.log.1, the oldest rollover file, is deleted. This is automatic and is not recoverable.
• All of the remaining rollover files are decremented, so solr.log.2 becomes the new solr.log.1 and so forth. At the bottom of the list, solr.log.10 is renamed to be solr.log.9.
• The current solr.log file is saved as the new solr.log.10.

This process is dynamic, so note that the log messages you saw in solr.log.6 last week might have migrated to solr.log.5 by now. If you need to reference messages in a log file, you should save a physical copy of the file.

When debugging a troubled Solr deployment, there isn’t much point in looking back more than a few days. Therefore, one downloads the solr.log file (current) and sometimes also the rollover file with the largest number (which is often solr.log.10). These two files contain a hefty sample of recent log messages.

Note that you must examine the logs from each of the Solr nodes in your cluster. Solr problems sometimes occur on only one node at a time.

Logging Levels

The remarks in this section assume that you want to modify the default logging behavior of Log4j in SearchStax Cloud.

You can adjust Solr logging in the Solr Dashboard:

You can monitor significant Solr events on an adjacent screen in the Solr dashboard:

See Configuring Logging in the Apache Solr documentation.

Load Balancer Logs

All traffic to a SearchStax cluster passes through a Load Balancer that distributes requests across the available servers.

Due to technical considerations, query logs are most naturally obtained from the Solr cluster’s load balancer. SearchStax Cloud makes the load balancer logs available through the deployment dashboard.

These are access logs that record the incoming queries, like this one:

{"httpRequest":{"remoteIp":"76.9.66.238","requestMethod":"GET","requestSize":"118","requestUrl":"https://ss123456-us-west2-gcp.searchstax.com/solr/filmcollection/select?q=director_name:%22Steven%20Spielberg%22&wt=json&indent=true&fl=movie_title","responseSize":"956","serverIp":"10.76.0.2","status":200,"userAgent":"curl/7.58.0"},"insertId":"1sxeg25ff8ohxo","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry","enforcedSecurityPolicy":{"configuredAction":"ACCEPT","name":"ss123456-armor","outcome":"ACCEPT","priority":2147483647},"statusDetails":"response_sent_by_backend"},"logName":"projects/searchstax-production/logs/requests","receiveTimestamp":"2019-01-09T15:00:29.930210771Z","resource":{"labels":{"backend_service_name":"","forwarding_rule_name":"ss123456-https-forwarding","project_id":"searchstax-production","target_proxy_name":"ss123456-https-proxy","url_map_name":"ss123456-loadbalancer","zone":"global"},"type":"http_load_balancer"},"severity":"INFO","spanId":"e549479c9a69ea78","timestamp":"2019-01-09T15:00:29.513737511Z","trace":"projects/searchstax-production/traces/b8e7795aa135c44964cd1a8343a2f34c"}

SearchStax updates these logs once per hour.

Nginx Logs

SearchStax Cloud single-node deployments don’t have Load Balancers, but do use Nginx to handle incoming requests.

SearchStax premium clients may ask our Support Desk for help in obtaining the Nginx logs when necessary.

SearchStax Cloud Activity Log

SearchStax Cloud automatically records all significant user actions that occur in the SearchStax Cloud Dashboard. To view them, click Activity Log in the left-side navigation panel.

Activity Log entries are retained indefinitely.

Help with Log Files

Premium customers may ask SearchStax support for help tracking down memory and performance issues. This often involves enabling logging on servers that the SearchStax end-user cannot reach.

For this type of help please reach out to the SearchStax Support Desk.