A possible reason for "Error while accessing backend services for API key validation" in WSO2 API manager

Problem

When try to validate a token in WSO2 API manager if it returns the error,
 

<ams:fault xmlns:ams="http://wso2.org/apimanager/security">
<ams:code>900900</ams:code>
<ams:message>Unclassified Authentication Failure</ams:message> 
<ams:description>Error while accessing backend services for API key validation</ams:description>
</ams:fault> 

Most likely cause of this problem is an error with Key Manager. This error means that it could not validate the tokens because it could not access the back-end or in other words, the  the Key Manager.

I had a distributed API manager 1.6 deployment and when I tried to generate a token for a user this error was returned. I went and had a look on the Key Manager's wso2carbon.log since it indicates an error in Key Manager. In the log file I noticed the following log But there was nothing wrong in Key Manager,

{org.wso2.carbon.identity.thrift.authentication.ThriftAuthenticatorServiceImpl} - Authentication failed for user: admin Hence, returning null for session id. {org.wso2.carbon.identity.thrift.authentication.ThriftAuthenticatorServiceImpl} 

 And In the API Gateway's log file following error was logged,

TID: [0] [AM] [2015-04-06 21:08:15,918] ERROR {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} -  API authentication failure {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Error while accessing backend services for API key validation
        at org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftAPIDataStore.getAllURITemplates(ThriftAPIDataStore.java:97)
        at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getAllURITemplates(APIKeyValidator.java:385)
        at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.doGetAPIInfo(APIKeyValidator.java:240)
        at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getResourceAuthenticationScheme(APIKeyValidator.java:153)
        at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:85)
        at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:92)
        at org.apache.synapse.rest.API.process(API.java:284)
        at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:76)
        at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:63)
        at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:220)
        at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
        at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
        at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:336)
        at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:377)
        at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183)
        at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient.<init>(ThriftKeyValidatorClient.java:45)
        at org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClientPool$1.makeObject(ThriftKeyValidatorClientPool.java:40)
        at org.apache.commons.pool.impl.StackObjectPool.borrowObject(StackObjectPool.java:170)
        at org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClientPool.get(ThriftKeyValidatorClientPool.java:50)
        at org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftAPIDataStore.getAllURITemplates(ThriftAPIDataStore.java:94)

Solution

When I investigated the problem further I realized that I have NOT put the correct super user name and password in /repository/conf/api-manager.xml  in the gateway (or the user name and password used to log into the management console). When I used the correct user name and password the problem was solved. 

This error occurs because the Gateway could not connect to Key Manager validation service due to invalid credentials. 

In api-manager.xml following 3 sections contians <Username> and <Password> and make sure thy are correct,

1) <AuthManager> 

2)<APIGateway> 

3)<APIKeyManager>

Note

This is not the only possible reason for the above mentioned error. Some other common causes are(but not limited to),

- Mis-configured master-datasources.xml file of Key manager
- Connectivity issue between Gateway and Key Manager

- Connectivity issues between Database and Key Manager

- Key manager is not reachable

- etc .....

I suggest you should have a look at the Key manager log file when you investigate this error and it's very likely you would find a clue