Mailing List Archives Authenticated access	UW Madison Computer Sciences Department Computer Systems Lab

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Condor-users] Delegated CredentialError : Submitting jobs from Condor to GT4 host

Date: Thu, 26 Mar 2009 11:05:11 +0530
From: Petri Nieminen <petriniem@xxxxxxxxx>
Subject: [Condor-users] Delegated CredentialError : Submitting jobs from Condor to GT4 host

I have a simple setup of two machines theRock1(GT4 host) and tripleH1(central manager and GT4 host).
I can successfully do this :
1) submit a grid universe job with condor_submit at tripleH1 to the GT4 container running at tripleH1 itself.
2) submit a Staging job from tripleH1 to theRock1 using globusrun-ws

However, when I try to submit with condor_submit at tripleH1 to theRock1, the job fails.

The job.cmd GridManagerLog.globus and the container.log files are as below:

The job.cmd file I am using is:
executable = /bin/hostname
transfer_executable = false
universe = grid
grid_resource = gt4 https://theRock1/wsrf/services/ManagedJobFactoryService Fork
output = test.out
log = test.log
queue

The GridManagerLog.globus at tripleH1 shows the following log

03/26 10:57:25 ******************************************************
03/26 10:57:25 ** condor_gridmanager (CONDOR_GRIDMANAGER) STARTING UP
03/26 10:57:25 ** /usr/local/condor/sbin/condor_gridmanager
03/26 10:57:25 ** SubsystemInfo: name=GRIDMANAGER type=DAEMON(10) class=DAEMON(1)
03/26 10:57:25 ** Configuration: subsystem:GRIDMANAGER local:<NONE> class:DAEMON
03/26 10:57:25 ** $CondorVersion: 7.3.0 Feb 16 2009 BuildID: 132922 $
03/26 10:57:25 ** $CondorPlatform: I386-LINUX_DEBIAN50 $
03/26 10:57:25 ** PID = 4282
03/26 10:57:25 ** Log last touched 3/26 10:50:01
03/26 10:57:25 ******************************************************
03/26 10:57:25 Using config source: /usr/local/condor/etc/condor_config
03/26 10:57:25 Using local config sources:
03/26 10:57:25    /usr/local/condor//condor_config.local
03/26 10:57:25 DaemonCore: Command Socket at <172.27.20.169:35687>
03/26 10:57:28 [4282] JEF: ConfigureGahp()
03/26 10:57:28 [4282] Found job 73.0 --- inserting
03/26 10:57:28 [4282] gahp server not up yet, delaying ping
03/26 10:57:28 [4282] gahp server not up yet, delaying checkDelegation
03/26 10:57:28 [4282] (73.0) doEvaluateState called: gmState GM_INIT, globusState
03/26 10:57:28 [4282] GAHP server pid = 4341
03/26 10:57:35 [4282] resource https://theRock1/wsrf/services/ManagedJobFactoryService is now up
03/26 10:57:35 [4282] (73.0) doEvaluateState called: gmState GM_DELEGATE_PROXY, globusState
03/26 10:57:36 [4282] (73.0) doEvaluateState called: gmState GM_DELEGATE_PROXY, globusState
03/26 10:57:36 [4282] (73.0) doEvaluateState called: gmState GM_GENERATE_ID, globusState
03/26 10:57:36 [4282] (73.0) doEvaluateState called: gmState GM_SUBMIT_ID_SAVE, globusState
03/26 10:57:37 [4282] (73.0) doEvaluateState called: gmState GM_SUBMIT, globusState
03/26 10:57:38 [4282] (73.0) gram callback: state StageIn, fault (null), exit code 0
03/26 10:57:38 [4282] (73.0) doEvaluateState called: gmState GM_SUBMIT_SAVE, globusState
03/26 10:57:38 [4282] (73.0) gram callback: state Failed, fault Staging error for RSL element fileStageIn., exit code 0
03/26 10:57:38 [4282] (73.0) doEvaluateState called: gmState GM_SUBMIT_SAVE, globusState
03/26 10:57:41 [4282] (73.0) doEvaluateState called: gmState GM_SUBMIT_SAVE, globusState
03/26 10:57:41 [4282] (73.0) doEvaluateState called: gmState GM_FAILED, globusState Failed
03/26 10:57:41 [4282] (73.0) doEvaluateState called: gmState GM_FAILED, globusState Failed
03/26 10:57:46 [4282] No jobs left, shutting down
03/26 10:57:46 [4282] Got SIGTERM. Performing graceful shutdown.
03/26 10:57:46 [4282] **** condor_gridmanager (condor_GRIDMANAGER) pid 4282 EXITING WITH STATUS 0

The container.log at theRock1 shows this

----------------------------------------------------------------------------------------------------------------------------------------------------------------

2009-03-26 11:00:57,872 INFO exec.StateMachine [RunQueueThread_12,logJobAccepted:3424] Job 48afa330-19c7-11de-88f1-bf8c99b20c15 accepted for local user 'globus'
2009-03-26 11:00:58,318 ERROR exec.StateMachine [RunQueueThread_4,fileCleanUp:2785] A secondary fault occured while trying to gracefully fail.
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.rmi.RemoteException: Unable to create RFT resource; nested exception is:
        org.globus.transfer.reliable.service.exception.RftException: Error processing delegated credentialError getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException]]
faultActor:
faultNode:
faultDetail:
        {http://xml.apache.org/axis/}stackTrace:java.rmi.RemoteException: Unable to create RFT resource; nested exception is:
        org.globus.transfer.reliable.service.exception.RftException: Error processing delegated credentialError getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException]]
        at org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:247)
        at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
        at org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:107)
        at org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:42)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:55)
        at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:90)
        at org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:97)
        at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
        at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
        at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:664)
        at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:382)
        at org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:153)
        at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
Caused by: org.globus.transfer.reliable.service.exception.RftException: Error processing delegated credentialError getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException]]
        at org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:392)
        at org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:354)
        at org.globus.transfer.reliable.service.ReliableFileTransferHome.create(ReliableFileTransferHome.java:134)
        at org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:237)
        ... 22 more

        {http://xml.apache.org/axis/}hostname:tripleH1

java.rmi.RemoteException: Unable to create RFT resource; nested exception is:
        org.globus.transfer.reliable.service.exception.RftException: Error processing delegated credentialError getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException]]
        at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
        at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
        at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
        at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
        at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
        at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:645)
        at org.apache.axis.Message.getSOAPEnvelope(Message.java:424)
        at org.apache.axis.message.addressing.handler.AddressingHandler.processClientResponse(AddressingHandler.java:305)
        at org.apache.axis.message.addressing.handler.AddressingHandler.invoke(AddressingHandler.java:110)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.client.AxisClient.invoke(AxisClient.java:190)
        at org.apache.axis.client.Call.invokeEngine(Call.java:2745)
        at org.apache.axis.client.Call.invoke(Call.java:2728)
        at org.apache.axis.client.Call.invoke(Call.java:2405)
        at org.apache.axis.client.Call.invoke(Call.java:2327)
        at org.apache.axis.client.Call.invoke(Call.java:1767)
        at org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
        at org.globus.exec.service.exec.utils.WSStagingHelper.submitStagingRequest(WSStagingHelper.java:123)
        at org.globus.exec.service.exec.StateMachine.fileCleanUp(StateMachine.java:2767)
        at org.globus.exec.service.exec.StateMachine.processFailureFileCleanUpState(StateMachine.java:2142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:329)
        at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
2009-03-26 11:00:58,597 INFO exec.StateMachine [RunQueueThread_0,logJobFailed:3454] Job 48afa330-19c7-11de-88f1-bf8c99b20c15 failed. Description: Staging error for RSL element fileStageIn. Cause: org.globus.exec.generated.StagingFaultType: Staging error for RSL element fileStageIn. caused by [0: org.oasis.wsrf.faults.BaseFaultType: java.rmi.RemoteException: Unable to create RFT resource; nested exception is:
        org.globus.transfer.reliable.service.exception.RftException: Error processing delegated credentialError getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error getting delegation resource [Caused by: org.globus.wsrf.NoSuchResourceException]]]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Prev by Date: Re: [Condor-users] about from idle to running
Next by Date: [Condor-users] Living w/o a shared filesystem
Previous by thread: Re: [Condor-users] Jobs fetched with a hook being killed after 20 minutes
Next by thread: [Condor-users] Living w/o a shared filesystem
Index(es):
- Date
- Thread

Mailing List Archives

Authenticated access

[Condor-users] Delegated CredentialError : Submitting jobs from Condor to GT4 host