Can’t talk to HBase from different kubernetes namespace: java.net.UnknownHostException: hregion-0.hregion

I am using kubernetes, where I have a Hadoop cluster running in namespace ‘platform’. I have an example application running in namespace ‘example’ The example application needs to talk to HBase. When it does so, we see the following error:

java.net.UnknownHostException: hregion-0.hregion at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at java.net.InetAddress.getByName(InetAddress.java:1076) at org.apache.hadoop.hbase.client.ConnectionUtils.getStubKey(ConnectionUtils.java:233) at org.apache.hadoop.hbase.client.ConnectionImplementation.getClient(ConnectionImplementation.java:1192) at org.apache.hadoop.hbase.client.ClientServiceCallable.setStubByServiceName(ClientServiceCallable.java:44) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:229) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360) at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1078) at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:403) at org.apache.hadoop.hbase.client.HBaseAdmin$  6.rpcCall(HBaseAdmin.java:445) at org.apache.hadoop.hbase.client.HBaseAdmin$  6.rpcCall(HBaseAdmin.java:442) at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3084) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3076) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:442) 

The command

> nslookup hregion-0.hregion 

on the client machine fails, because the hregion service is in the platform namespace (where that command will succeed).

We suspected that the HBase region server has registered itself with zookeeper using an incomplete name, and verified by connecting to the zookeeper server:

[zk: localhost:2181(CONNECTED) 8] ls /hbase/rs [hregion-0.hregion,16020,1560851357442] 

The ConnectionUtils.getStubKey method simply uses java.net.InetAddress.getByName(hostname) and it is this method which fails.

How do I fix this? I assume I need to tell HBase to register its nodes in zookeeper using their fully qualified domain name – how?

Netstat, lsof, etc. are limited to one namespace, what else connection is hidden from them?

On Linux, network monitoring tools are limited to the current namespace and does not show the whole machine connections (like lsof -i, netstat -ltup)

I have to loop on all namespaces to see the whole machine connections with sudo ip -all netns exec lsof -i

Is there other connections that are also hidden from lsof/netstat?

Coredns only resolving cluster address for default namespace


Summary

I can only resolve hostnames from and for the default namespace. Requests from and for other namespaces, like kube-system, return NXDOMAIN from coredns. Is there some authorisation that needs to be done?

Details

I have set up a learning cluster by following Kubernetes the Hard Way except using VirtualBox instead of GCP. All else appears to work except DNS queries in non-default namespaces. More concretely:

  • Works (run from default namespace): kubernetes.default
  • Does Not work (run from kube-system namespace, get NXDOMAIN): kubernetes.kube-system

Configuration

I have upgraded to Coredns 1.12 because 1.2.2 failed to resolve anything around 50% of the time. With 1.12 requests resolve 100% of the time (if in the default namespace).

I had to edit the config (in the configmap) by removing the “loop” and “proxy” (replace proxy with forward) plugins to get it working. My config is as follows:

    .:53 {         forward . 192.168.99.254:53 # dnsmasq on host machine         log         errors         health         kubernetes cluster.local in-addr.arpa ip6.arpa {           pods insecure           upstream           fallthrough in-addr.arpa ip6.arpa         }         prometheus :9153         cache 30         reload         loadbalance 

Troubleshooting

Following this guide I see:

The pods are running OK

$   kubectl get pods --namespace=kube-system -l k8s-app=kube-dns NAME                       READY   STATUS    RESTARTS   AGE coredns-6f8fb48d88-dz9cd   1/1     Running   0          14m coredns-6f8fb48d88-s5rvp   1/1     Running   0          14m 

Checking the console I see healthy logging

2019-05-24T04:41:13.990Z [INFO] plugin/reload: Running configuration MD5 = 7c717fabd578b2514cf60fee426fabc6 2019-05-24T04:41:13.992Z [INFO] CoreDNS-1.5.0 2019-05-24T04:41:13.992Z [INFO] linux/amd64, go1.12.2, e3f9a80 

The DNS service is available:

 $   kubectl get svc --namespace=kube-system NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE kube-dns         ClusterIP   10.32.0.10   <none>        53/UDP,53/TCP   18d 

The endpoints are exposed:

$   kubectl get ep kube-dns --namespace=kube-system NAME       ENDPOINTS                                                     AGE kube-dns   10.200.21.33:53,10.200.22.17:53,10.200.21.33:53 + 1 more...   18d 

As mentioned above, DNS works but only from the default namespace.

Avoid duplicate name in namespace?

I have different cluster of models for a registering process.

One cluster is the set of all models belong to a group-upload. All models to the group-upload are inside the folder

'App\Register\GroupUpload` 

Here is a short overview of the tables:

enter image description here

The actual model names are

App\Register\GroupUpload\GroupUpload  App\Register\GroupUpload\User App\Register\GroupUpload\Mode App\Register\GroupUpload\Issue App\Register\GroupUpload\UserUpdate App\Register\GroupUpload\Level 

bascially I just removed the GroupUpload prefix and put them in the GroupUpload folder. So the model of GroupUploadUser table is just called User.php inside the GroupUpload folder.

I am not happy with the outcome, that I have the word GroupUpload twice in the core model of the cluster, namly App\Register\GroupUpload\GroupUpload. Of course I have this issue for each cluster

\App\Register\Celebration\Celebration  \App\Register\Activation\Activation 

and so on.

Any recommendations how I may avoid this type of situation?

Получить доступ к определениям типов Node.js-зависимости (Cannot find namespace “XXX”)

Ниже приведены определения TypeScirpt-типов библиотеки arg, парсера консольных команд:

declare const flagSymbol: unique symbol;  declare function arg<T extends arg.Spec>(spec: T, options?: arg.Options): arg.Result<T>;  declare namespace arg {     export function flag<T>(fn: T): T & { [flagSymbol]: true };      export const COUNT: Handler<number> & { [flagSymbol]: true };      export type Handler <T = any> = (value: string, name: string, previousValue?: T) => T;      export interface Spec {         [key: string]: string | Handler | [Handler];     }      export type Result<T extends Spec> = { _: string[] } & {         [K in keyof T]?: T[K] extends Handler             ? ReturnType<T[K]>             : T[K] extends [Handler]             ? Array<ReturnType<T[K][0]>>             : never     };      export interface Options {         argv?: string[];         permissive?: boolean;         stopAtPositional?: boolean;     } }  export = arg; 

Нужно аннотировать результаты парсинга:

import parseConsoleArgument from 'arg';  export function cli(rawConsoleCommandData: Array<string>): void {   const consoleCommandArguments: /* ??? */ = parseConsoleArgument(       {}, { argv: rawConsoleCommandData.slice(2)}   ); } 

Логически правильный ответ – arg.Result. Однако, в моём файле нет доступа к этому пространству имён. Приведённый ниже код (то же, что и до этого, только с аннотацией типа)

import parseConsoleArgument from 'arg';  export function cli(rawConsoleCommandData: Array<string>): void {   const consoleCommandArguments: arg.Result = parseConsoleArgument(       {}, { argv: rawConsoleCommandData.slice(2)}   ); } 

даёт ошибку TS2503: Cannot find namespace "arg".

Is it a good thing to create a namespace for global functions in C++?

I’m making a C++ OpenSource project and I want it the more understandable and clean, so I need some point of views about doing things like this :

Server.cpp:

void Server::setOption(type foo, type bar) {/* ... */} void Server::start() {/* ... */} 

App.cpp:

namespace App {   void startServer() {     Server s;     s.setOption(o1, o2);     s.start();   } } 

Then in main function:

App::startServer(); 

Or should I simply write the whole code in main ?

JAX-WS – Error during call because a namespace has no prefix – How to add it?

I am facing an issue when I try to call a WS using JAX-WS. All my java classes are generated with wsdl-2-java maven plugin (cxf).

Below my maven wsdlOptions:

<wsdlOption> <wsdl>https://mywsdl</wsdl> <wsdlLocation>https://mywsdl</wsdlLocation> <extendedSoapHeaders>true</extendedSoapHeaders> <extraargs>     <extraarg>-client</extraarg>     <extraarg>-autoNameResolution</extraarg>     <extraarg>-p</extraarg>     <extraarg>http://myws=com.ws.myws</extraarg>     <extraarg>-p</extraarg>     <extraarg>http://namespace1=com.xsd.namespace1</extraarg>     <extraarg>-p</extraarg>     <extraarg>http://namespace2=com.xsd.namespace2</extraarg>     <extraarg>-p</extraarg>     <extraarg>http://namespace3=com.xsd.namespace3</extraarg> </extraargs> 

Here is the generated SOAP call :

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Header>     <ns2:Options      xmlns="http://namespace1"     xmlns:ns2="http://namespace2"      xmlns:ns3="http://namespace3" /> </soap:Header> <soap:Body>     <ns2:GetRequest      xmlns="http://namespace1"     xmlns:ns2="http://namespace2"      xmlns:ns3="http://namespace3">         <ns2:Key>             <ns3:Code>AAA</ns3:Code>             <ns3:CmpCode>BBB</ns3:CmpCode>             <ns3:Level>CCC</ns3:Level>         </ns2:Key>     </ns2:GetRequest> </soap:Body> 

Here is the call that works with SOAP UI :

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Header>     <ns2:Options      xmlns:ns1="http://namespace1"     xmlns:ns2="http://namespace2"      xmlns:ns3="http://namespace3" /> </soap:Header> <soap:Body>     <ns2:GetRequest      xmlns:ns1="http://namespace1"     xmlns:ns2="http://namespace2"      xmlns:ns3="http://namespace3">         <ns2:Key>             <ns3:Code>AAA</ns3:Code>             <ns3:CmpCode>BBB</ns3:CmpCode>             <ns3:Level>CCC</ns3:Level>         </ns2:Key>     </ns2:GetRequest> </soap:Body> 

The only difference is that the namespace xmlns (related to namespace1) has no prefix.

<ns2:Options xmlns="http://namespace1" [...] /> 

The error returned says that I try to call a method on the “namespace1”, which I don’t.

So I have tried to add a SOAPHandler to be able to parse the SOAP Body and do some modifications. But when I open all the variables that I can access at runtime, I don’t even see this “namespace1” so I can’t add/remove it. Maybe I should continue on this way.

I have also tried to change the order in my wsdloptions but it’s always “namespace1” that is ko.

I am looking for a global/elegant solution because I have nearly 20 others ws to generate for the same provider, so I am pretty sure that I will have the same issue.

Any idea ?

Linux NFS (v4) user namespace file permissions

I am trying to set up a rootless container setup using podman and am having a problem with file permissions on NFS (running NFSv4).

Within the user namespace, I am able to create files as root on any filesystem, and to change ownership on any local filesystem, but not on a NFS mounted filesystem. Outside of the user namespace, as real root, I am able to change ownership of the same file to either a real or mapped UID.

Using buildah unshare to enter the user namespace:

> cat /etc/subuid bruce:1065536:65536  > date > /local/testfile > buildah unshare chown 250:250 /local/testfile > ls -l /local/testfile -rw-rw-r-- 1 1065785 1065785 29 May  6 11:51 /local/testfile  > date > /nfs/testfile > buildah unshare chown 250:250 /nfs/testfile chown: changing ownership of '/nfs/testfile': Operation not permitted ERRO[0000] exit status 1                                 ERRO[0000] exit status 1                                 > sudo chown 1065785:1065785 /nfs/testfile Password:  > ls -l /nfs/testfile -rw-rw-r-- 1 1065785 1065785 29 May  6 11:51 /nfs/testfile 

What might I be missing here?

How can I reference Windows.System namespace in a Xamarin project?

I am trying to open a pdf file on the browser, acrobat or any other default pdf program. For this I need to call LaunchFileAsync() and I need to reference Windows.System to be able to use it. However, this is not working and I can’t call any reference.

I tried looking for a specific library or nuget package but I couldn’t find anything. The documentation just tells me to use that namespace but it doesn’t say anything about how to add the reference. I also tried using a different approach by calling System.Diagnostics.Process.Start(path) and some variations but it doesnt’t work using Xamarin apparently.

This is basically what I am trying to implement but there’s no namespace found event though I did try implementing it.

string uriToLaunch = @”file://” + @file;

var uri = new Uri(uriToLaunch);

var success = await Windows.System.Launcher.LaunchUriAsync(uri);

Any help would be appreciated.