<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Just for Records sake problem resolved, I added: <tt>CONDOR_Q_ONLY_MY_JOBS = FALSE</tt> <tt> </tt><tt>to my config files, and it resolved the issue.</tt> <tt>Thanks for the help on the other errors!</tt> <tt>-Brandon</tt> <div class="moz-cite-prefix">On 1/8/18 3:09 PM, Brandon Graves wrote: </div> <blockquote type="cite" cite="mid:d98e95a3-fd5c-8b5e-5fa2-6dc14aaf8fbc@uw.edu"> <pre wrap="">Alright, I now Have Central Manager = MASTER, COLLECTOR, NEGOTIATOR Submit Nodes = MASTER, SCHEDD Execute nodes = MASTER, SCHEDD, STARTD That has fixed the error in the logs, but "condor_q -global" still presents: </pre> <blockquote type="cite"> <pre wrap="">Failed to fetch ads from: </pre> </blockquote> <pre wrap=""><xxx.xxx.xxx.xxx:9618?addrs=xxx.xxx.xxx.xxx-9618&noUDP&sock=3573919_bf24_3> : host.my.domain.com </pre> <blockquote type="cite"> <pre wrap="">AUTHENTICATE:1003:Failed to authenticate with any method AUTHENTICATE:1004:Failed to authenticate using GSI GSI:5003:Failed to authenticate. Globus is reporting error </pre> </blockquote> <pre wrap="">(851968:50). There is probably a problem with your credentials. (Did you run grid-proxy-init?) </pre> <blockquote type="cite"> <pre wrap="">AUTHENTICATE:1004:Failed to authenticate using KERBEROS AUTHENTICATE:1004:Failed to authenticate using FS </pre> </blockquote> <pre wrap=""> Thank's again for your help, if you have any more idea's I'd be very appreciative --Brandon On 1/8/18 1:16 PM, John M Knoeller wrote: </pre> <blockquote type="cite"> <pre wrap="">This message. </pre> <blockquote type="cite"> <pre wrap="">01/08/18 09:59:56 DaemonCore: Can't receive command request from </pre> </blockquote> <pre wrap="">xxx.xxx.xxx.105 (perhaps a timeout?) Is generally not a problem. You will see it in working pools once per negotiation cycle, it happens because the negotiator hangs up after updating accounting ads, but the collector assumes that any socket opened for updates will never be closed, so it looks for a second command after the first and we get a warning when It doesn't find one. This message. </pre> <blockquote type="cite"> <pre wrap="">01/08/18 10:03:14 PERMISSION DENIED to <a class="moz-txt-link-abbreviated" href="mailto:condor_pool@my.domain.com">condor_pool@my.domain.com</a> from </pre> </blockquote> <pre wrap="">host xxx.xxx.xxx.60 for command 10 (QUERY_STARTD_PVT_ADS), access level NEGOTIATOR: reason: cached result for NEGOTIATOR; see first case for the full reason Is a problem. It indicates that host xxx.xxx.xxx.60 is running a NEGOTIATOR, and that negotiator is unable to negotiate because the COLLECTOR will not send it the information it needs to do so. if xxx.xxx.xxx.60 is not your central manager, then you probably just need to remove NEGOATIATOR from the DAEMON_LIST in the configuration of host xxx.xxx.xxx.60 try running condor_config_val -verbose DAEMON_LIST on each of your submit nodes. the result should not have either COLLECTOR or NEGOTIATOR. -tj -----Original Message----- From: HTCondor-users [<a class="moz-txt-link-freetext" href="mailto:htcondor-users-bounces@cs.wisc.edu">mailto:htcondor-users-bounces@cs.wisc.edu</a>] On Behalf Of Brandon Graves Sent: Monday, January 8, 2018 12:18 PM To: <a class="moz-txt-link-abbreviated" href="mailto:htcondor-users@cs.wisc.edu">htcondor-users@cs.wisc.edu</a> Subject: Re: [HTCondor-users] Error with global Queue Both commands yield valid results without errors. in the Central Managers CollectorLog I have: </pre> <blockquote type="cite"> <pre wrap="">01/08/18 10:03:14 PERMISSION DENIED to <a class="moz-txt-link-abbreviated" href="mailto:condor_pool@my.domain.com">condor_pool@my.domain.com</a> from </pre> </blockquote> <pre wrap="">host xxx.xxx.xxx.60 for command 10 (QUERY_STARTD_PVT_ADS), access level NEGOTIATOR: reason: cached result for NEGOTIATOR; see first case for the full reason </pre> <blockquote type="cite"> <pre wrap="">01/08/18 10:03:14 DC_AUTHENTICATE: Command not authorized, done! 01/08/18 10:03:20 Got QUERY_STARTD_ADS 01/08/18 10:03:20 Number of Active Workers 0 01/08/18 10:03:20 Got QUERY_STARTD_ADS 01/08/18 10:03:20 Number of Active Workers 0 01/08/18 10:03:26 Got QUERY_STARTD_PVT_ADS 01/08/18 10:03:26 Number of Active Workers 0 01/08/18 10:03:26 Number of Active Workers 0 01/08/18 10:03:26 DaemonCore: Can't receive command request from </pre> </blockquote> <pre wrap="">xxx.xxx.xxx.105 (perhaps a timeout?) xxx.60 is one of my submit nodes, and xxx.105 is the central manager. There is also a similar entry for other nodes. I looked through for logs with a bit more detail and got: </pre> <blockquote type="cite"> <pre wrap="">01/08/18 09:59:56 DaemonCore: Can't receive command request from </pre> </blockquote> <pre wrap="">xxx.xxx.xxx.105 (perhaps a timeout?) </pre> <blockquote type="cite"> <pre wrap="">01/08/18 09:59:56 PERMISSION DENIED to <a class="moz-txt-link-abbreviated" href="mailto:condor_pool@ling.washington.edu">condor_pool@ling.washington.edu</a> </pre> </blockquote> <pre wrap="">from host xxx.xxx.xxx.52 for command 10 (QUERY_STARTD_PVT_ADS), access level NEGOTIATOR: reason: cached result for NEGOTIATOR; see first case for the full reason </pre> <blockquote type="cite"> <pre wrap="">01/08/18 09:59:56 DC_AUTHENTICATE: Command not authorized, done! </pre> </blockquote> <pre wrap="">Thank you for any further insight you can provide! -Brandon On 1/8/18 9:46 AM, John M Knoeller wrote: </pre> <blockquote type="cite"> <pre wrap="">I think this means that condor_q is unable to fetch schedd ads from the collector. Try running condor_status -schedd do you get the same error? does a simple condor_status work? If you look in the CollectorLog on the central manager, do you see any messages about the rejected query? -----Original Message----- From: HTCondor-users [<a class="moz-txt-link-freetext" href="mailto:htcondor-users-bounces@cs.wisc.edu">mailto:htcondor-users-bounces@cs.wisc.edu</a>] On Behalf Of Brandon Graves Sent: Monday, January 8, 2018 11:26 AM To: HTCondor-Users Mail List <a class="moz-txt-link-rfc2396E" href="mailto:htcondor-users@cs.wisc.edu"><htcondor-users@cs.wisc.edu></a> Subject: [HTCondor-users] Error with global Queue Hello All, I recently replaced my Central Manager, and a few odd things have come up. The only definite error message I can find though happens when "condor_q -global" is run: </pre> <blockquote type="cite"> <pre wrap="">-- Failed to fetch ads from: </pre> </blockquote> <pre wrap=""><xxx.xxx.xxx.49:9618?addrs=xxx.xxx.xxx.49-9618+[> : server1.my.domain.com </pre> <blockquote type="cite"> <pre wrap="">AUTHENTICATE:1003:Failed to authenticate with any method AUTHENTICATE:1004:Failed to authenticate using GSI GSI:5003:Failed to authenticate. Globus is reporting error </pre> </blockquote> <pre wrap="">(851968:50). There is probably a problem with your credentials. (Did you run grid-proxy-init?) </pre> <blockquote type="cite"> <pre wrap="">AUTHENTICATE:1004:Failed to authenticate using KERBEROS AUTHENTICATE:1004:Failed to authenticate using FS </pre> </blockquote> <pre wrap="">My basic configuration is Central manager, connected to 2 submit nodes. Each submit node seems to be able to see it's own queue, one of the submit nodes off and on seems to be having trouble running jobs, but I can't seem to find any errors that make sense. For now I'd like to figure out the global queue error as I suspect they are related. My config file as far as authentication goes looks like this: </pre> <blockquote type="cite"> <pre wrap="">SEC_PASSWORD_FILE = /etc/condor/pool_password SEC_DAEMON_AUTHENTICATION = REQUIRED SEC_DAEMON_INTEGRITY = REQUIRED SEC_DAEMON_AUTHENTICATION_METHODS = PASSWORD SEC_NEGOTIATOR_AUTHENTICATION = REQUIRED SEC_NEGOTIATOR_INTEGRITY = REQUIRED SEC_NEGOTIATOR_AUTHENTICATION_METHODS = PASSWORD SEC_CLIENT_AUTHENTICATION_METHODS = FS, PASSWORD, KERBEROS, GSI </pre> </blockquote> <pre wrap="">( I didn't do the initial install/configuration of HTcondor on these systems, I'm just the new admin for them, and still getting my footing) I've looked through some of the logs, but I can't seem to find any specific error messages that point me in a new direction. Any tips/tricks/idea's would be appreciated --Brandon _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to <a class="moz-txt-link-abbreviated" href="mailto:htcondor-users-request@cs.wisc.edu">htcondor-users-request@cs.wisc.edu</a> with a subject: Unsubscribe You can also unsubscribe by visiting <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a> The archives can be found at: <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/archive/htcondor-users/">https://lists.cs.wisc.edu/archive/htcondor-users/</a> _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to <a class="moz-txt-link-abbreviated" href="mailto:htcondor-users-request@cs.wisc.edu">htcondor-users-request@cs.wisc.edu</a> with a subject: Unsubscribe You can also unsubscribe by visiting <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a> The archives can be found at: <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/archive/htcondor-users/">https://lists.cs.wisc.edu/archive/htcondor-users/</a> </pre> </blockquote> <pre wrap="">_______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to <a class="moz-txt-link-abbreviated" href="mailto:htcondor-users-request@cs.wisc.edu">htcondor-users-request@cs.wisc.edu</a> with a subject: Unsubscribe You can also unsubscribe by visiting <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a> The archives can be found at: <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/archive/htcondor-users/">https://lists.cs.wisc.edu/archive/htcondor-users/</a> _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to <a class="moz-txt-link-abbreviated" href="mailto:htcondor-users-request@cs.wisc.edu">htcondor-users-request@cs.wisc.edu</a> with a subject: Unsubscribe You can also unsubscribe by visiting <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a> The archives can be found at: <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/archive/htcondor-users/">https://lists.cs.wisc.edu/archive/htcondor-users/</a> </pre> </blockquote> <pre wrap=""> _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to <a class="moz-txt-link-abbreviated" href="mailto:htcondor-users-request@cs.wisc.edu">htcondor-users-request@cs.wisc.edu</a> with a subject: Unsubscribe You can also unsubscribe by visiting <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users">https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users</a> The archives can be found at: <a class="moz-txt-link-freetext" href="https://lists.cs.wisc.edu/archive/htcondor-users/">https://lists.cs.wisc.edu/archive/htcondor-users/</a></pre> </blockquote> </body> </html>