error launching Solr 9.7+, looks for config directory in wrong place, one dir up

[jrochkind]

It appears to me that solr_wrapper cannot be run with solr 9.7+. At first I thought my problem might be #155, but the error message is different -- and #154, suggested to fix $155, was indeed merged on Sep 24, and seems to be included on solr_wrapper 4.1.0+.

However, when I try to use solr_wrapper to start solr latest 9.8.1, I get this:

Failed to execute solr create: Option '-url': Deprecated for removal since 9.7: Use -s instead (RuntimeError)
Option '-url': Deprecated for removal since 9.7: Use -s instead
Specified configuration directory /Users/jrochkind/code/solr/conf not found!
. Further information may be available in /var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/logs

I'm not sure if the -url stuff is a misidrection, that's just a warning about deprecation (although prob should also be fixed), but the real failure is /Users/jrochkind/code/solr/conf not found!?

• That is the wrong place to be looking for ./solr/conf, it's going up a directory
• .solr_wrapper is at /Users/jrochkind/code/blacklight_8_11_app/.solr_wrapper.yml, and specifies collection: dir: solr/conf/
• I am running bundle exec solr_wrapper from /Users/jrochkind/code/blacklight_8_11_app/
• So it ought to be looking at /Users/jrochkind/code/blacklight_8_11_app/solr/conf (a directory that is present), not at /Users/jrochkind/code/solr/conf

Not sure if this is a separate/new problem?

Let's see the last solr I can install succesfully

• 9.7.1 -- can't install with solr_wrapper, due to fingerprint mismatch, different problem
• 9.7.0 -- reproduces problem, can't start
• 9.6.1 -- does not reproduce, installs and boots fine

So this does seem to be related to something different that changed in 9.7.0.

Has anyone else succesfully run solr_wrapper with solr 9.7+? @ewlarson any experiences to report, are you having the same probelm?

[jrochkind]

Having trouble figuring out where/how solr_wrapper or anything else is telling solr to look at solr/conf at all, doesn't seem to be a command line arg bieng sent to solr start, like I expected it to be.

[jrochkind]

OK I think I'm getting further.

This line:

solr_wrapper/lib/solr_wrapper/instance.rb

Line 167 in 46e7d42

create_options[:d] = File.expand_path(options[:dir], '..')

I think should actually be . instead of ...

create_options[:d] = File.expand_path(options[:dir], '.')

But if i do that, I'm off to a different weird error:

ERROR:
Core 'blacklight-core' already exists!
Checked core existence using Core API command

But there's no blacklight-core already there in the instance directory as far as I can tell.... I think this is some kind of weird race condition, it reminds me of something I've seen before?

I don't understand what is going on well enough to be sure if I should PR the change from .. to . on it's own, without a working example yet? Any advice?

@ewlarson, curious if you've managed to use solr_wrapper with solr 9.7+ yet.

[jrochkind]

I am thinking the "core already exists message" may be misleading/masking what's really going on... but I'm not finding anything in the solr logs it points to explain what's really going on, they are pretty empty.

solr-8990-console.log

OpenJDK 64-Bit Server VM warning: -XX:+UseLargePages not supported in this VM
CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future release
2025-07-22 13:42:59.787 INFO  (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.22; built: 2024-06-27T16:03:51.502Z; git: 5c8471e852d377fd726ad9b1692c35ffc5febb09; jvm 17.0.2+0
2025-07-22 13:43:00.116 WARN  (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2025-07-22 13:43:00.135 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.8.1
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port 8990
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2025-07-22T13:43:00.139440Z
2025-07-22 13:43:00.141 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/logs/jvm_crash_97444.log

solr.log

2025-07-22 13:42:59.787 INFO  (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.22; built: 2024-06-27T16:03:51.502Z; git: 5c8471e852d377fd726ad9b1692c35ffc5febb09; jvm 17.0.2+0
2025-07-22 13:43:00.116 WARN  (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2025-07-22 13:43:00.135 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.8.1
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port 8990
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1
2025-07-22 13:43:00.139 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2025-07-22T13:43:00.139440Z
2025-07-22 13:43:00.141 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/logs/jvm_crash_97444.log
2025-07-22 13:43:00.222 INFO  (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr (source: system property: solr.solr.home)
2025-07-22 13:43:00.247 INFO  (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr/solr.xml
2025-07-22 13:43:00.270 INFO  (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/lib]
2025-07-22 13:43:00.736 INFO  (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2025-07-22 13:43:00.935 WARN  (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2025-07-22 13:43:00.937 WARN  (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
2025-07-22 13:43:01.040 INFO  (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr
2025-07-22 13:43:01.157 WARN  (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2025-07-22 13:43:01.284 INFO  (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2025-07-22 13:43:01.351 INFO  (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@2e185cd7{solr-jetty-context.xml,/solr,file:///private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr-webapp/webapp/,AVAILABLE}{/private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr-webapp/webapp}
2025-07-22 13:43:01.359 INFO  (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/logs/2025_07_22.request.log
2025-07-22 13:43:01.361 INFO  (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@12bcd0c0{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:8990}
2025-07-22 13:43:01.362 INFO  (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@20b5f2ac{STARTING}[10.0.22,sto=0] @2566ms
2025-07-22 13:43:02.177 INFO  (qtp1361817590-19-localhost-1) [c: s: r: x: t:localhost-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=20
2025-07-22 13:43:02.840 INFO  (qtp1361817590-24-localhost-2) [c: s: r: x: t:localhost-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=0
2025-07-22 13:43:03.419 INFO  (qtp1361817590-27-localhost-3) [c: s: r: x: t:localhost-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=0
2025-07-22 13:43:03.425 INFO  (qtp1361817590-19-localhost-4) [c: s: r: x: t:localhost-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=0
2025-07-22 13:43:04.180 INFO  (qtp1361817590-27-localhost-5) [c: s: r: x: t:localhost-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=0
2025-07-22 13:43:04.393 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.e.j.s.Server Stopped Server@20b5f2ac{STOPPING}[10.0.22,sto=0]
2025-07-22 13:43:04.394 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.e.j.s.AbstractConnector Stopped ServerConnector@12bcd0c0{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:8990}
2025-07-22 13:43:04.400 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.a.s.c.CoreContainer Shutting down CoreContainer instance=585496942
2025-07-22 13:43:04.401 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
2025-07-22 13:43:04.406 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
2025-07-22 13:43:04.409 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
2025-07-22 13:43:04.414 INFO  (ShutdownMonitor) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Stopped o.e.j.w.WebAppContext@2e185cd7{solr-jetty-context.xml,/solr,file:///private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr-webapp/webapp/,STOPPED}{/private/var/folders/_1/89lqv5550mx2tggl22z27_p18516fz/T/solr-9.8.1/server/solr-webapp/webapp}

[jrochkind]

@cjcolvar suggested taking a look at projectblacklight/blacklight#3626

I am using Blacklight 8.11, which may not have that change to .solr_wrapper.yml backported yet... but making that change manually to my generated .solr_wrapper.yml does not seem to have changed things for me, still same error messages.