I've 8 wso2 apim 4.1.0 services running in a distributed deployment (4 control-planes + KM and 4 gateways). Like described on .1.0/install-and-setup/setup/distributed-deployment/deploying-wso2-api-m-in-a-distributed-setup/#configure-the-control-plane-nodes.

3 of the 4 control-planes are running without problems and serving the Token / Key manager information as expected. Still, one of them constantly gives the erros below when using the devportal and requesting tokens:

Caused by: .wso2.carbon.apimgt.impl.kmclient.KeyManagerClientException: Received status code: 401 Reason:

and

Polling tasks on destination : keyManager of type topic for listener Siddhi-JMS-Consumer#keyManager have not yet started after 3 seconds (and incrementing the time)

Where is a full exception:

ERROR {.wso2.carbon.apimgt.impl.UserAwareAPIConsumer} - Error while retrieving Application Information .wso2.carbon.apimgt.api.APIManagementException: Cannot retrieve service provider for the given consumer key : ...
        at .wso2.carbon.apimgt.impl.AbstractKeyManager.handleException_aroundBody12(AbstractKeyManager.java:274)
        at .wso2.carbon.apimgt.impl.AbstractKeyManager.handleException(AbstractKeyManager.java:272)
        at .wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.retrieveApplication_aroundBody10(AMDefaultKeyManagerImpl.java:437)
        at .wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.retrieveApplication(AMDefaultKeyManagerImpl.java:423)
        at .wso2.carbon.apimgt.impl.AbstractAPIManager.getApplicationKeys_aroundBody216(AbstractAPIManager.java:3077)
        at .wso2.carbon.apimgt.impl.AbstractAPIManager.getApplicationKeys(AbstractAPIManager.java:3028)
        at .wso2.carbon.apimgt.impl.APIConsumerImpl.getApplicationKeysOfApplication_aroundBody206(APIConsumerImpl.java:5044)
        at .wso2.carbon.apimgt.impl.APIConsumerImpl.getApplicationKeysOfApplication(APIConsumerImpl.java:5042)
        at .wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl.getApplicationKeys(ApplicationsApiServiceImpl.java:824)
        at .wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl.applicationsApplicationIdOauthKeysGet(ApplicationsApiServiceImpl.java:1121)
        at .wso2.carbon.apimgt.rest.api.store.v1.ApplicationsApi.applicationsApplicationIdOauthKeysGet(ApplicationsApi.java:300)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at .apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
        at .apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
        at .apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
        at .apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
        at .apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
        at .apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
        at .apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
        at .apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
        at .apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
        at .apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
        at .apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
        at .apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
        at .apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
        at .apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304)
        at .apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:222)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
        at .apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:279)
        at .apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at .apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at .apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at .apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at .apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at .apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at .apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at .apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
        at .apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at .apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at .wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:107)
        at .wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110)
        at .wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:102)
        at .wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:101)
        at .wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
        at .wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
        at .wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:146)
        at .apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
        at .wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:58)
        at .wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:126)
        at .apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
        at .apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at .apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at .apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
        at .apache.tomcat.util.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
        at .apache.tomcat.util.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at .apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at .apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at .apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: .wso2.carbon.apimgt.impl.kmclient.KeyManagerClientException: Received status code: 401 Reason:
        at .wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode_aroundBody0(KMClientErrorDecoder.java:42)
        at .wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode(KMClientErrorDecoder.java:35)
        at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
        at com.sun.proxy.$Proxy431.getApplication(Unknown Source)
        at .wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.retrieveApplication_aroundBody10(AMDefaultKeyManagerImpl.java:430)
        ... 60 more

The error indicates that the problem has something to do with HTTP 401 when trying to retrieve the information about that consumer key/client id, but the other 3 services use the same configuration, user (is the super_user of the deployment.toml) and key-manager parameters on the deployment.toml and don't get a 401 which is strange. I only have this error when i try to see the subscritption details of the user logged in or try out an API subscribed, other than that the service seam fine but the tokens are not working well.

Image of the devportal:

Having this, I have 2 questions:

How can I log this request on the wso2carbon.log tha originates the 401? It's possible to log this request? I've tried the synapse-wire logger on log4j file without success.

Does anyone come across a similar problem? Any suggestion on how to fix this?