Re: [HTCondor-users] Grafana + HTCondor Job Metrics

[Kevin Retzke <kretzke@xxxxxxxx>]

Ok, some things to check on the graphite side:

* Are there errors in /var/log/carbon/listener.log?

* Are the new metrics listed in /var/log/carbon/creates.log? 

* Are there files in LOCAL_DATA_DIR (defined in /etc/carbon/carbon.conf)?

* Do you have a whitelist set up? (/etc/carbon/whitelist.conf) Does it allow "^clusters\." and "^probes\."?

Try sending a metric with netcat, and then check the above, e.g.

echo "test.foo.bar 10 $(date +%s)" | nc MY_GRAPHITE_HOST 2003

Kevin

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Wednesday, March 1, 2017 3:07 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Yeah I'm not seeing data in graphite. I imported the batch jobs dashboard and changed the graphite datasource using sed but still not seeing data.

Thanks,

Uchenna

On Wed, Mar 1, 2017 at 3:59 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

That means it's working! You'll see [ERROR] or [WARNING] messages if there's a problem. Are you not seeing any data in graphite?

Kevin

---

From: HTCondor-users <htcondor-users-bounces@cs.wisc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Wednesday, March 1, 2017 2:41 PM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

I'm getting this repeatedly in stdout. Could be because my graphite server wasn't specified properly?

[root@xxxxxxx bin]# ./condor_probe.py /root/probes/etc/condor-probe.cfg
2017-03-01 14:58:10,416 [INFO] __main__ - Probe configuraion:
{'delay': 30,
 'graphite_host': 'xxxxxx.xxxxxx.xxxxx.xxxx.xxxx',
 'graphite_pickle_port': 2004,
 'influxdb_db': 'xxxxx',
 'influxdb_host': 'xxxxx.xxxxx.xxxxx.xxxxx.xxx',
 'influxdb_port': '8086',
 'influxdb_tags': {'foo': 'bar'},
 'interval': 240,
 'meta_namespace': 'probes.condor-mypool',
 'namespace': 'clusters.mypool',
 'once': False,
 'pool': 'xxxx.xxxx.xxxxx.xxxxx.xxxxx',
 'post_pool_glideins': False,
 'post_pool_jobs': True,
 'post_pool_prio': False,
 'post_pool_slots': True,
 'post_pool_status': True,
 'retries': 10,
 'test': False,
 'use_graphite': True,
 'use_gsi_auth': False,
 'use_influxdb': False,
 'x509_user_cert': '',
 'x509_user_key': ''}
2017-03-01 14:58:10,417 [INFO] __main__ - querying pool xxxx.xxxxxx.xxxxxx.xxxxx.xxxxx status
2017-03-01 14:58:10,446 [INFO] __main__ - querying pool xxxxxx.xxxxx.xxxxxxx.xxxxx.xxxx slots
2017-03-01 14:58:10,473 [INFO] __main__ - querying pool xxxxxx.xxxxx.xxxxxxx.xxxxxx.xxx jobs
2017-03-01 14:58:10,476 [INFO] fifemon.probe - (clusters.mypool) posted data in 0.059867143631 s
2017-03-01 14:58:10,477 [INFO] fifemon.probe - (clusters.mypool) sleeping 229.940132856 s

Thanks,

Uchenna

On Wed, Mar 1, 2017 at 2:25 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

Great, it worked! It's just showing you all the metrics it will publish. Take off the "--test" flag and it will publish the metrics to graphite and/or influxdb (and not dump them all to stdout).  Grab the Grafana dashboards from https://github.com/fifemon/dashboards [1]and you should see data!

Regards,

Kevin

[1] note that the dashboards are currently hard-coded to use a graphite datasource named "fifemon-graphite", they were created before Grafana would template data sources on export. Pretty easy to change in Grafana, or with sed or editor of your choice (s/fifemon-graphite/my-grapite-datasource/).

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Wednesday, March 1, 2017 12:25 PM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

The result was quite long. Here's the head and tail of the result.

(venv2.0)[xxxxxx@xxxxxxxxxxx bin]# ./condor_probe.py --test /root/probes/etc/condor-probe.cfg
2017-03-01 11:54:57,882 [INFO] __main__ - Probe configuraion:
{'delay': 30,
 'graphite_host': 'xxxxxxxxx.xxxxxx.xxxxxxxxxx.xxxxxxxxxxxx',
 'graphite_pickle_port': 2004,
 'influxdb_db': 'xxxxx',
 'influxdb_host': 'xxxxxxxxxxxxxxxxxx.xxxxxxx.xxxxxxxxx',
 'influxdb_port': '8086',
 'influxdb_tags': {'foo': 'bar'},
 'interval': 240,
 'meta_namespace': 'probes.condor-mypool',
 'namespace': 'clusters.mypool',
 'once': False,
 'pool': 'xxxxxxxxxxx.xxxxxx.xxxxxxx.xxxxxxxxxxxxx',
 'post_pool_glideins': False,
 'post_pool_jobs': True,
 'post_pool_prio': False,
 'post_pool_slots': True,
 'post_pool_status': True,
 'retries': 10,
 'test': True,
 'use_graphite': True,
 'use_gsi_auth': False,
 'use_influxdb': False,
 'x509_user_cert': '',
 'x509_user_key': ''}
2017-03-01 11:54:57,883 [INFO] __main__ - querying pool xxxxxxx.xxxxxx.xxxxxx.xxxx status
2017-03-01 11:54:57,905 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_-_xxxxxx_xxxxxx_xxxxxxx_noaa_gov-xxxxx_xxxx_xxxxx_xxxx_xxxx.RecentStatsLifetime', (1488387297.9056799, 1200L))
2017-03-01 11:54:57,905 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.MonitorSelfTime', (1488387297.9056799, 1488387088L))
2017-03-01 11:54:57,905 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.CurrentJobsRunningStandard', (1488387297.9056799, 0L))
2017-03-01 11:54:57,905 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.MonitorSelfRegisteredSocketCount', (1488387297.9056799, 15L))
2017-03-01 11:54:57,905 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.UpdatesInitial_Master', (1488387297.9056799, 5L))
2017-03-01 11:54:57,905 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.RecentUpdatesTotal_StartdPvt', (1488387297.9056799, 18L))
2017-03-01 11:54:57,906 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.LastHeardFrom', (1488387297.9056799, 1488387097L))
2017-03-01 11:54:57,906 [DEBUG] fifemon.graphite - ('clusters.mypool.schedds.xxxxx_xxxx_xxxxx_xxxx_xxxx.StatsLifetime', (1488387297.9056799, 3533605L))
2017-03-01 11:54:57,906 [DEBUG] fifemon.graphite - ('clusters.mypool.schedds.xxxxx_xxxx_xxxxx_xxxx_xxxx.TransferQueueNumWaitingToUpload', (1488387297.9056799, 0L))
2017-03-01 11:54:57,906 [DEBUG] fifemon.graphite - ('clusters.mypool.collectors.My_Pool_xxxxx_xxxx_xxxxx_xxxx_xxxx-xxxxx_xxxx_xxxxx_xxxx_xxxxx.MaxJobsRunningPipe', (1488387297.9056799, 0L))

TAIL END:


2017-03-01 11:54:57,920 [DEBUG] fifemon.graphite - ('clusters.mypool.negotiators.xxxxx_xxxx_xxxxx_xxxx_xxxx.LastNegotiationCycleMatchRate0', (1488387297.9056799, 0.0))
2017-03-01 11:54:57,920 [DEBUG] fifemon.graphite - ('clusters.mypool.negotiators.xxxxx_xxxx_xxxxx_xxxx_xxxx.LastNegotiationCycleMatchRate1', (1488387297.9056799, 0.0))
2017-03-01 11:54:57,920 [INFO] __main__ - querying pool xxxxx.xxxx.xxxxx.xxxx.xxxxslots
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalSlotDisk', (1488387297.9352701, 15220332.0))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.jobs.totals.DiskUsage', (1488387297.9352701, 0))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalSlotCpus', (1488387297.9352701, 112L))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalSlotMemory', (1488387297.9352701, 1258795L))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.Memory', (1488387297.9352701, 1258795L))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalDisk', (1488387297.9352701, 14885972L))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalCondorLoadAvg', (1488387297.9352701, 0.0))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalCpus', (1488387297.9352701, 120.0))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.Cpus', (1488387297.9352701, 112L))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalLoadAvg', (1488387297.9352701, 0.59000000000000008))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.Disk', (1488387297.9352701, 14885972L))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.LoadAvg', (1488387297.9352701, 0.59000000000000008))
2017-03-01 11:54:57,935 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.jobs.totals.MemoryUsage', (1488387297.9352701, 0))
2017-03-01 11:54:57,936 [DEBUG] fifemon.graphite - ('clusters.mypool.slots.Partitionable.totals.TotalMemory', (1488387297.9352701, 1258795L))
2017-03-01 11:54:57,936 [INFO] __main__ - querying pool xxxxxx.xxxxxx.xxxxx.xxxxxx.xxxxxx jobs
2017-03-01 11:54:57,939 [INFO] fifemon.probe - (clusters.mypool) posted data in 0.055379152298 s
2017-03-01 11:54:57,939 [DEBUG] fifemon.graphite - ('probes.condor-mypool.update_time', (1488387297.939198, 0.055379152297973633))
2017-03-01 11:54:57,939 [INFO] fifemon.probe - (clusters.mypool) sleeping 229.944620848 s

Thanks,

Uchenna

On Wed, Mar 1, 2017 at 11:51 AM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

Ok, great, the probe is working, it's just not finding the collector. Try setting the "pool" option in the "[condor]" section to your pool's central manager/collector host.

See output of "condor_status -collector". Maybe setting 'pool=""' will also work if you're running on a machine within the pool; I've never tried.

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Wednesday, March 1, 2017 10:30 AM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Here's the rest:

2017-03-01 11:09:21,507 [INFO] __main__ - querying pool localhost status
2017-03-01 11:09:21,513 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:09:51,546 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:10:21,589 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:10:51,620 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:11:21,654 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:11:51,686 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:12:21,725 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:12:51,757 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:13:21,802 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:13:51,834 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:14:21,864 [ERROR] condor.status - trouble getting pool localhost negotiators status, giving up.
2017-03-01 11:14:21,870 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:14:51,901 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:15:21,949 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:15:51,981 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:16:22,023 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:16:52,055 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:17:22,099 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:17:52,137 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:18:22,173 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:18:52,205 [WARNING] condor.status - trouble getting pool localhost schedds status, retrying in 30s.
2017-03-01 11:19:22,235 [ERROR] condor.status - trouble getting pool localhost schedds status, giving up.
2017-03-01 11:19:22,249 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:19:52,284 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:20:22,327 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:20:52,360 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:21:22,403 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:21:52,443 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:22:22,482 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:22:52,521 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:23:22,570 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:23:52,605 [WARNING] condor.status - trouble getting pool localhost collectors status, retrying in 30s.
2017-03-01 11:24:22,636 [ERROR] condor.status - trouble getting pool localhost collectors status, giving up.
2017-03-01 11:24:22,636 [INFO] __main__ - querying pool localhost slots
2017-03-01 11:24:22,653 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:24:52,685 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:25:22,725 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:25:52,759 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:26:22,798 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:26:52,835 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:27:22,871 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:27:52,904 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:28:22,943 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:28:52,975 [WARNING] condor.slots - trouble getting pool localhost startds, retrying in 30s.
2017-03-01 11:29:23,005 [ERROR] condor.slots - trouble getting pool localhost startds, giving up.
2017-03-01 11:29:23,005 [INFO] __main__ - querying pool localhost jobs
2017-03-01 11:29:23,020 [ERROR] root - Trouble getting pool localhost schedds.
2017-03-01 11:29:23,020 [WARNING] fifemon.graphite - send_dict called with no data
2017-03-01 11:29:23,020 [INFO] fifemon.probe - (clusters.mypool) posted data in 1201.51364803 s
2017-03-01 11:29:23,020 [DEBUG] fifemon.graphite - ('probes.condor-mypool.update_time', (1488385763.020906, 1201.5136480331421))
2017-03-01 11:29:23,020 [INFO] fifemon.probe - (clusters.mypool) sleeping 0 s

Thanks,

Uchenna

On Wed, Mar 1, 2017 at 11:07 AM, Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx> wrote:

Hi Kevin,

The same warning keeps repeating. This result is with influxdb and Graphite output.

[xxxxxxxxxx@xxxxxxxxxxxxxxxxx bin]# ./condor_probe.py --test /root/probes/etc/condor-probe.cfg
2017-03-01 11:01:16,784 [INFO] __main__ - Probe configuraion:
{'delay': 30,
 'graphite_host': 'xxxxxxxxxxx.xxxxxxxxx.xxxxxxx.xxxxxxxxxx',
 'graphite_pickle_port': 2004,
 'influxdb_db': 'xxxxxx',
 'influxdb_host': 'xxxxxxxxx.xxxxxxxxx.xxxxxx.xxxxxxxxx',
 'influxdb_port': '8086',
 'influxdb_tags': {'foo': 'bar'},
 'interval': 240,
 'meta_namespace': 'probes.condor-mypool',
 'namespace': 'clusters.mypool',
 'once': False,
 'pool': 'localhost',
 'post_pool_glideins': False,
 'post_pool_jobs': True,
 'post_pool_prio': False,
 'post_pool_slots': True,
 'post_pool_status': True,
 'retries': 10,
 'test': True,
 'use_graphite': True,
 'use_gsi_auth': False,
 'use_influxdb': False,
 'x509_user_cert': '',
 'x509_user_key': ''}
2017-03-01 11:01:16,786 [INFO] __main__ - querying pool localhost status
2017-03-01 11:01:16,826 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:01:46,858 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:02:16,895 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.
2017-03-01 11:02:46,931 [WARNING] condor.status - trouble getting pool localhost negotiators status, retrying in 30s.

Thanks,

Uche

On Wed, Mar 1, 2017 at 10:31 AM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

Hi Uchenna,

The exact same error? The last error was because you didn't have a db specified in the InfluxDB section, so if you added that... Try using just the Graphite output?

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Tuesday, February 28, 2017 9:46 AM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

I'm still getting the same error after I made those changes.

Thanks,

Uchenna

On Mon, Feb 27, 2017 at 4:05 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

Ah, I didn't see you had InfluxDB output enabled (note that the InfluxDB output is incomplete and only includes daemon statistics). Your config is not right, please see the required parameters in the docs:

[influxdb] enable = false # enable output to influxdb (not fully supported) host = localhost # influxdb host port = 8086 # influxdb api port db = test # influxdb database tags = foo:bar # extra tags to include with all metrics (comma-separated key:value)

Username and password should be set in environment variables:

export INFLUXDB_USERNAME=<username> export INFLUXDB_PASSWORD=<password>

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Monday, February 27, 2017 2:17 PM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

I keep getting the same issue.


(venv2.0)[xxxxxxxx@xxxxxxxx bin]# ./condor_probe.py --test /root/probes/etc/condor-probe.cfg
Traceback (most recent call last):
  File "./condor_probe.py", line 147, in <module>
    opts = get_options()
  File "./condor_probe.py", line 135, in get_options
    'influxdb_db':       config.get("influxdb", "db"),
  File "/usr/lib64/python2.6/ConfigParser.py", line 565, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'db' in section: 'influxdb'

Thanks,

Uchenna

On Mon, Feb 27, 2017 at 2:03 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

The config file looks fine, you just need to specify it as an argument to the probe on the command line, e.g.

./condor_probe.py --test ../etc/condor-probe.cfg

(I included the --test flag, which will cause it to run once and output extra debug information, without sending the data to graphite)

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Monday, February 27, 2017 12:11 PM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

I modified the etc/condor-probe.cfg file and the error was the result I got. Is how I modified the file the issue?

[probe]
interval = 240
retries = 10
delay = 30
test = false
>
[graphite]
enable = true
host = xxxxxxxxxxxxxxx
port = 2004
namespace = clusters.mypool
meta_namespace = probes.condor-mypool

[influxdb]
enable = true
host = xxxxxxxxxxxxxxxxxxxxxxxx
port = 8086
username = xxxxxxxxxxxxxxxxx
password = xxxxxxxxxxxxxxxxxxx

[condor]
pool = localhost
post_pool_status = true
post_pool_slots = true
post_pool_glideins = false
post_pool_prio = false
post_pool_jobs = true
use_gsi_auth = false
X509_USER_CERT = ""
X509_USER_KEY = ""

Thanks,

Uchenna

On Mon, Feb 27, 2017 at 12:53 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

You'll need to pass it a config file. See etc/condor-probe.cfg for an example, and documentation in the README or at https://fifemon.github.io/probes/. 

Sorry for the terrible error message, I opened an issue to fix that.

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Monday, February 27, 2017 11:19 AM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Thanks Luke.

Kevin, After installing condor-python successfully and updating the condor-probes config file I did:

(venv2.0)[xxxxxxxxxxxxxx@xxxxxxxxxx bin]$ ./condor_probe.py
Traceback (most recent call last):
  File "./condor_probe.py", line 147, in <module>
    opts = get_options()
  File "./condor_probe.py", line 118, in get_options
    'pool':              config.get("condor", "pool"),
  File "/usr/lib64/python2.6/ConfigParser.py", line 556, in get
    raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'condor'

What could be the issue?

Thanks,

Uchenna

On Mon, Feb 27, 2017 at 11:04 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Hi Uchenna,

I see you are already using virtualenv for some isolation.

If you would be happy to move to miniconda instead, you can try [1].

The recipe uses python 2.7 within the condor environment as well as the 8.4.2 python bindings.

The channel has also the version 8.4.3, 8.4.10, 8.4.11 and 8.6.

Cheers,

Luke

[1]

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh

# follow instructions https://conda.io/docs/install/quick.html

bash miniconda.sh -b -p <where you want your conda installation to be>

export PATH=<path to miniconda>/bin:$PATH

# it is important that you use conda-forge for the boost package

# as the default does not come with libboost_python

conda create -n condor python=2.7

source activate condor

conda install -c kreczko -c conda-forge htcondor-python=8.4.2

On 27 February 2017 at 14:40, Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx> wrote:

Hi Kevin,

I am only able to use the machines that have condor-8.4.2 installed. The machines with later versions of condor are my personal machines. Do you have another suggestion?

Thanks,

Uchenna

On Sat, Feb 25, 2017 at 9:55 AM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

You can run the probe on any machine that can talk to your central manager (it doesn't have to be part of of the pool itself), e.g. we monitor six or so pools from a dedicated VM. The version of condor installed on that machine doesn't have to be the same as what your pool is running (there's probably some caveats). So my suggestion is to run a separate "monitoring" host (this can be a VM at first, at minimum I'd recommend 2 cores, 4 GB RAM, and 200 GB disk, SSD if possible). Unless you keep it hidden, you'll probably soon find people saying "hey that's cool, can I send telemetry for my service x there?" so plan to scale up

A quick test to see if you can monitor a pool from a host would be to run 'condor_status -pool <cm host>'. If that works the probe should work.

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Friday, February 24, 2017 9:37 PM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

I'm currently using condor-8.4.2 el6, I see condor-python is for condor-8.4.9. On my other machines, I am able to install condor-python because they have a later version of condor. What do you suggest I do for my machines that have condor-8.4.2 installed?

On Fri, Feb 24, 2017 at 9:40 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

Where did you install condor from? It's in the official YUM repos (http://research.cs.wisc.edu/htcondor/yum/).

It's available for EL6 and 7 (we are currently running on SL6).

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Friday, February 24, 2017 8:09 PM

To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

I tried installing that initially. YUM could not find any package by that name "condor-python". Is that package for Centos6 or is it only for Centos7?

Thanks,

Uchenna

On Fri, Feb 24, 2017 at 4:57 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

You need to have the HTCondor Python bindings installed ("condor-python" package).

Regards,

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Friday, February 24, 2017 3:49 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hi Kevin,

I cannot use the systemd unit file because I am using Centos6 and not Centos7. I tried running the condor_probe.py script and this is the error I got. In fact, throughout my logs this is what I keep getting.

"Traceback (most recent call last):
  File "./condor_probe.py", line 12, in <module>
    import condor
  File "/directory/homedir/name/probes/bin/condor/__init__.py", line 4, in <module>
    from .status import get_pool_status
  File "/directory/homedir/name/probes/bin/condor/status.py", line 5, in <module>
    import classad
ImportError: No module named classad"

I'm not sure what you mean by the "--once" flag, but I still wasn't able to run the script doing './condor_probe.py'

Thanks,
Uchenna

On Fri, Feb 24, 2017 at 2:51 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

Hi, author of Fifemon here.  Do you really want to run Supervisor and the probes as root? We run as much as possible in user space. 

See http://supervisord.org/running.html#runtime-security.

If you don't want to deal with supervisor, you can instead run the probe with cron by passing the '--once' flag to 'condor_probe.py'.

Or, here's a simple systemd unit file you could use (untested; change user, group, and command as appropriate): https://gist.github.com/retzkek/0dcd7b19548de96531bd5362c51ee2a6

Init script is left as an exercise

Please let me know if you have any other questions or issues!

Regards,

Kevin Retzke

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxc.edu> on behalf of Uchenna Ojiaku - NOAA Affiliate <uchenna.ojiaku@xxxxxxxx>
Sent: Friday, February 24, 2017 1:20 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Grafana + HTCondor Job Metrics

 

Hello,

Has anyone configured grafana to display htcondor job/slot metrics? I found this online. But my supervisord isn't working. I get this is error "Supervisord is running as root and it is searching" So I try killing any running supervisord processes but I still get the same error. Then I tried unlinking with 'sudo unlink /tmp/supervisor.sock' but then I get this notice "unlink: cannot unlink '/tmp/supervisor.sock': No such file or directory"

If you know of a better software/method, I will be happy to look into it.





Reference:


Fifemon

Collect HTCondor statistics and report into time-series database. All modules support Graphite, and there is some support for InfluxDB.

Additionally report select job and slot Classads into Elasticsearch via Logstash.

Note: this is a fork of the scripts used for monitoring the HTCondor pools at Fermilab, and while generally intended to be "generic" for any pool still may require some tweaking to work well for your pool.

Copyright Fermi National Accelerator Laboratory (FNAL/Fermilab). See LICENSE.txt.

Requirements

• Python 2.6 or greater recommended.
• HTCondor libraries and Python bindings
  • https://research.cs.wisc.edu/htcondor/downloads/
• A running Graphite server (available in EPEL or via PIP)
  • http://graphite.readthedocs.org/en/latest/

For current job and slot state:

• A running Elasticsearch cluster
  • https://www.elastic.co/products/elasticsearch
• Logstash (tested with v2.0.0+)
  • https://www.elastic.co/downloads/logstash

Installation

Assuming HTCondor and Python virtualenv packages are already installed:

cd $INSTALLDIR
git clone https://github.com/fifemon/probes
cd probes
virtualenv --system-site-packages venv
source venv/bin/activate
pip install supervisor influxdb

Optionally, for crash mails:

pip install superlance

Configuration

Condor metrics probe

Example probe config is in etc/condor-probe.cfg:

[probe]
interval = 240   # how often to send data in seconds
retries = 10     # how many times to retry condor queries
delay = 30       # seconds to wait beteeen retries
test = false     # if true, data is output to stdout and not sent downstream
     # run one time and exit, i.e. for running wtih cron (not recommended)

[graphite]
enable = true                           # enable output to graphite
host = localhost                        # graphite host
port = 2004                             # graphite pickle port
namespace = clusters.mypool             # base namespace for metrics
meta_namespace = probes.condor-mypool   # namespace for probe metrics

[influxdb]
enable = false       # enable output to influxdb (not fully supported)
host = localhost     # influxdb host
port = 8086          # influxdb api port
db = test            # influxdb database
tags = foo:bar       # extra tags to include with all metrics (comma-separated key:value)

[condor]
pool = localhost            # condor pool (collector) to query
post_pool_status = true     # collect basic daemon metrics
post_pool_slots = true      # collect slot metrics
post_pool_glideins = false  # collect glidein-specific metrics
post_pool_prio = false      # collect user priorities
post_pool_jobs = false      # collect job metrics
use_gsi_auth = false        # set true if collector requires authentication
X509_USER_CERT = ""         # location of X.509 certificate to authenticate to condor with
X509_USER_KEY = ""          # private key

Supervisor

Example supervisor config is in etc/supervisord.conf, it can be used as-is for basic usage. Reuqires some modification to enable crashmails or to report job and slot details to elsaticsearch (via logstash).

Job and slot state

The scripts that collect raw job and slot records into elasticsearch are much simpler than the metrics probe - simply point at your pool with --pool and JSON records are output to stdout. We use logstash to pipe the output to Elasticsearch; see etc/logstash-fifemon.conf.

Running

Using supervisor:

cd $INSTALLDIR/probes
source venv/bin/activate

If using influxdb:

export INFLUXDB_USERNAME=<username>
export INFLUXDB_PASSWORD=<password>

Start supervisor:

supervisord

Thanks,

Uchenna 

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

*********************************************************
  Dr Lukasz Kreczko           
  Research Associate

  Department of Physics

  Particle Physics Group

  University of Bristol
  HH Wills Physics Lab
  University of Bristol
  Tyndall Avenue
  Bristol
  BS8 1TL

  +44 (0)117 928 8724 

  L.Kreczko@xxxxxxxxxxxxx

  

  A top 5 UK university with leading employers (2015)

  A top 5 UK university for research (2014 REF)

  A world top 40 university (QS Ranking 2015)

*********************************************************

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/