Re: [HTCondor-users] condor_adstash missing package dependencies

[Jason Patton <jpatton@xxxxxxxxxxx>]

Hi Stuart,

Not listing "python3-elasticsearch" as a dependency was intentional as there is a strong dependencyÂbetweenÂthe versions of the Elasticsearch client library and the Elasticsearch server. For example, the minimum wire compatibility version is 7.17.0 for a server running the latest Elasticsearch release (8.6.1). EPEL tends to ship pretty old versions of the Elasticsearch Python library... I just launched a CentOS Stream 8 container and installed epel-release, and "yum install python3-elasticsearch" got me version 7.0.5 of the Python library, which is incompatible with the latest ES server.

Unless you're willing to run pip as root to install a newer version of the Elasticsearch library to your system Python 3 libraries (and I wouldn't recommend doing so), running adstash as a condor daemon instead of as a standalone application becomes rather difficult because, as you've pointed out, the #! line of the entry script to adstash points to the system. If, for example, you were to install newer ES libraries in a virtualenv, this line would need to change to point to the virtualenv'sÂpython3 path.

I'm definitely open to suggestions on how to improve or workaround this limited Elasticsearch backwards compatibility situation, because it would be wonderful for adstash to just work by installing the condor-adstash RPM and setting "use feature: adstash" in the condor config. For what it's worth, I run adstash as standalone script inside a conda environment where it's easy to bump the version of the ES library as needed.

Jason Patton

On Wed, Feb 8, 2023 at 12:56 PM Anderson, Stuart B. <sba@xxxxxxxxxxx> wrote:

Tim, Jason,
    On an EL8.7 system I am finding missing package dependencies for condor_adstash. Is it possible to add these to the rpm or is condor not compatible with the versions available from EL8?


[root@ldas-pcdev8 ~]# uname -a
Linux ldas-pcdev8 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Thu Jan 12 16:32:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[root@ldas-pcdev8 ~]# condor_version
$CondorVersion: 10.0.1 2023-01-05 BuildID: 621397 PackageID: 10.0.1-1 $
$CondorPlatform: x86_64_AlmaLinux8 $

[root@ldas-pcdev8 ~]# head -1 /usr/bin/condor_adstash
#!/usr/libexec/platform-python

[root@ldas-pcdev8 ~]# ls -l /usr/libexec/platform-python
lrwxrwxrwx 1 root root 20 Nov 8 03:34 /usr/libexec/platform-python -> ./platform-python3.6

[root@ldas-pcdev8 ~]# condor_adstash
02/08/23 10:38:47 ******************************************************
02/08/23 10:38:47 ** /usr/bin/condor_adstash (CONDOR_ADSTASH) STARTING UP
02/08/23 10:38:47 ** /usr/bin/condor_adstash
02/08/23 10:38:47 ** Configuration: subsystem:ADSTASH type:ADSTASH class:DAEMON
02/08/23 10:38:47 ** $CondorVersion: 10.0.1 2023-01-05 BuildID: 621397 PackageID: 10.0.1-1 $
02/08/23 10:38:47 ** $CondorPlatform: x86_64_AlmaLinux8 $
02/08/23 10:38:47 ** PID: 1850361
02/08/23 10:38:47 ******************************************************
02/08/23 10:38:47 Could not send DC_SET_READY message to condor_master
02/08/23 10:38:47 Starting adstash loop
Traceback (most recent call last):
 File "/usr/bin/condor_adstash", line 143, in <module>
  main()
 File "/usr/bin/condor_adstash", line 113, in main
  last_loop_time = adstash(args)
 File "/usr/libexec/condor/adstash/adstash.py", line 51, in adstash
  interface = interface_info["class"]()(**iface_kwargs)
 File "/usr/libexec/condor/adstash/interfaces/registry.py", line 21, in elasticsearch_interface
  from adstash.interfaces.elasticsearch import ElasticsearchInterface
 File "/usr/libexec/condor/adstash/interfaces/elasticsearch.py", line 25, in <module>
  import elasticsearch
ModuleNotFoundError: No module named 'elasticsearch'

[root@ldas-pcdev8 ~]# yum install python3-elasticsearch
Last metadata expiration check: 1:53:44 ago on Wed 08 Feb 2023 08:45:13 AM PST.
Dependencies resolved.
================================================================================
ÂPackage          ÂArchitecture Version       Repository Size
================================================================================
Installing:
Âpython3-elasticsearch   Ânoarch   Â7.0.5-2.el8     epel   Â126 k

Transaction Summary
================================================================================
Install 1 Package

[root@ldas-pcdev8 ~]# condor_adstash
02/08/23 10:39:05 ******************************************************
02/08/23 10:39:05 ** /usr/bin/condor_adstash (CONDOR_ADSTASH) STARTING UP
02/08/23 10:39:05 ** /usr/bin/condor_adstash
02/08/23 10:39:05 ** Configuration: subsystem:ADSTASH type:ADSTASH class:DAEMON
02/08/23 10:39:05 ** $CondorVersion: 10.0.1 2023-01-05 BuildID: 621397 PackageID: 10.0.1-1 $
02/08/23 10:39:05 ** $CondorPlatform: x86_64_AlmaLinux8 $
02/08/23 10:39:05 ** PID: 1850584
02/08/23 10:39:05 ******************************************************
02/08/23 10:39:05 Could not send DC_SET_READY message to condor_master
02/08/23 10:39:05 Starting adstash loop
Traceback (most recent call last):
 File "/usr/bin/condor_adstash", line 143, in <module>
  main()
 File "/usr/bin/condor_adstash", line 113, in main
  last_loop_time = adstash(args)
 File "/usr/libexec/condor/adstash/adstash.py", line 51, in adstash
  interface = interface_info["class"]()(**iface_kwargs)
 File "/usr/libexec/condor/adstash/interfaces/registry.py", line 21, in elasticsearch_interface
  from adstash.interfaces.elasticsearch import ElasticsearchInterface
 File "/usr/libexec/condor/adstash/interfaces/elasticsearch.py", line 26, in <module>
  from elasticsearch._version import __versionstr__ as ES_VERSIONSTR
ModuleNotFoundError: No module named 'elasticsearch._version'

Thanks.

--
Stuart Anderson
sba@xxxxxxxxxxx




_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/