Add serial console support (Feature #2217)


Added by Lance Albertson over 6 years ago. Updated about 1 year ago.


Status:In Progress Start date:12/17/2010
Priority:High Due date:
Assignee:Jack Twilley % Done:

0%

Category:VirtualMachines Spent time: -
Target version:-
Difficulty:Difficult Needs Tests:No
Tags:

Description

We need to eventually support connecting to the serial console on a VM. Its going to be tricky but doable with another tool. I recommend we look at Ajaxterm as a possible solution for this feature. We will need to directly connect to the ganeti cluster outside of the RAPI for this feature to work unfortunately. Additionally there will probably be differences between xen/kvm in how we implement it.

Perhaps the best route to go is to somehow run or tie into doing "gnt-instance console $vname" which will simplify dealing with different hypervisors. I will take a look at this from the backend point of view and report back.

NOTE: this is in addition to the VNC console.

Related to Ajaxterm: WebShell


History

Updated by Lance Albertson over 6 years ago

I did a simple test by doing the following:

./ajaxterm.py -c "gnt-instance console gimager"

I was able to connect to the console without any issues. So the basic test works for now. We'll have to figure out doing it dynamically and integrating it into the site.

Updated by Peter Krenesky over 6 years ago

  • Target version changed from 0.6 to 0.7

Updated by Piotr Banaszkiewicz over 6 years ago

Ajaxterm is probably too old. And it supports only latin1, no unicode. I'm currently looking over WebShell. This is probably the right thing to use.

Updated by Piotr Banaszkiewicz over 6 years ago

Okay, using WebShell is fine, but there are some issues, like no backspace, no ctrl-c, ctrl-d shurtcuts [at least for Chromium].

Updated by Piotr Banaszkiewicz about 6 years ago

Okay, Peter, you decide what to do. Here are options:
1) Change WebShell to use command defined on demand for connecting to the VM
2) Rewrite it and use DB as a store for sessions
3) Rewrite it to use Canvas and WebSockets

And some issue: this is the command used by gnt-instance console VMname

[...]
/usr/bin/socat STDIO,raw,echo=0,escape=0x1d UNIX-CONNECT:/var/run/ganeti/kvm-hypervisor/ctrl/pbnan2.gwm.osuosl.org.serial

My guess is that serial console is not accessible from outside of the node.
Some lsof:
gwm1 ~ # lsof | grep serial
qemu-syst  8016    root   13u     unix 0xffff8800786580c0         0t0   14921230 /var/run/ganeti/kvm-hypervisor/ctrl/pbnan2.gwm.osuosl.org.serial

Updated by Lance Albertson about 6 years ago

Piotr Banaszkiewicz wrote:

And some issue: this is the command used by gnt-instance console VMname
[...]
My guess is that serial console is not accessible from outside of the node.
Some lsof:
[...]

This is correct which is why using gnt-instance console $vmname should be used. The master node will send a remove ssh command to the node which has access to the console. Additionally, the method that ganeti uses in the backend to connect to the console will differ between the hypervisors so its not a good idea to hardcode each backend method that ganeti itself uses (i.e. xen has its own method for connecting to the serial console). From a maintainability POV its much better to use gnt-instance console $vmname.

Finally, whatever daemon you end up using (assuming you need one) will need to run on every node to account for moving which node acts as the master in the cluster. The master always points to the cluster dns name (i.e. gwm.osuosl.org). The daemons running on the other nodes will just remain idle until their nodes becomes a master.

I'm not sure if I've explained how ganeti's master node will do the work of connecting to a remove node for commands it does before in case that helps this make more sense.

Updated by Lance Albertson about 6 years ago

I talked with Peter directly about this and we decided to try and get something implemented in Ganeti proper. The basic idea is adding a feature to the RAPI which will open an INET socket on the node where the VM lives. I've started a discussion on the Ganeti mailing list which we can follow.

  • Status changed from New to Feedback

Updated by Lance Albertson about 6 years ago

We're going to take a completely different approach to this. Following the thread on the Ganeti mailing list, we'll take the approach of implementing most of this inside of the vncauthproxy. The auth proxy will connect to the ganeti cluster via ssh and an ssh key specifically for this use. It will do a remote call to gnt-instance console $vm. On the other side webshell should just use the information passed from vncauthproxy in a format it can understand.

It seems that Google uses a similar approach internally but using an ssh only client to do simple commands.

Updated by Lance Albertson about 6 years ago

Looks like we may want to wait until Ganeti 2.4 is released. Looking at the 2.4 RAPI docs, I found they added support for providing detailed information on how to connect to the console.

Updated by Lance Albertson almost 6 years ago

  • Target version changed from 0.7 to 0.8

Updated by Heiko Baumann almost 6 years ago

just want to mention http://anyterm.org/

it works great for me without issues like (ctr-c, tab completion etc.)

Updated by Peter Krenesky almost 6 years ago

Heiko Baumann wrote:

just want to mention http://anyterm.org/

it works great for me without issues like (ctr-c, tab completion etc.)

Thanks that looks good. I'm not really familiar with ajaxterm either so we'll need to evaluate both and determine which makes the most sense

Updated by Peter Krenesky almost 6 years ago

  • Priority changed from Normal to High

Updated by Corbin Simpson almost 6 years ago

After talking with Peter, I agree that we shouldn't need any proxying support as long as Ganeti's backend can serve SSH. If not, we can quite easily serve SSH in the proxy.

Updated by Lance Albertson almost 6 years ago

Corbin Simpson wrote:

After talking with Peter, I agree that we shouldn't need any proxying support as long as Ganeti's backend can serve SSH. If not, we can quite easily serve SSH in the proxy.

What do you mean by "Ganeti's backend serving ssh"? AFAIK there is no ssh service that ganeti provides.

Updated by Peter Krenesky almost 6 years ago

Corbin Simpson wrote:

After talking with Peter, I agree that we shouldn't need any proxying support as long as Ganeti's backend can serve SSH. If not, we can quite easily serve SSH in the proxy.

Had a long conversation with lance about how to make this work:

VNC or Serial

We should have a single tab for console. This will display either VNC or Serial Console (SSH). Serial has preference if both are available. The reasoning here is that not all instances will support VNC and vice versa.

Proxy

We will need a proxy due to:
  • The backend may be behind a VPN or firewall. The proxy will act as a gateway into the secure network.
  • The backend commands required for serial console requires and account that can ssh to the Node and then run a command with sudo. This obviously can't be given to users.

The best solution is to create a MITM auth proxy for SSH and use the same workflow that VNC connections use:

  1. connect button triggers port request with proxy
  2. proxy opens ssh server
  3. proxy client executes ssh command to backend server (e.g. ssh node1.osuosl.org sudo socat ....)
  4. javascript client uses temp password to login

Updated by Corbin Simpson almost 6 years ago

Assigning myself to this. I presume we'll be reusing lots of TVAP code and building this into TVAP, right?

  • Assignee set to Corbin Simpson

Updated by Peter Krenesky almost 6 years ago

Corbin Simpson wrote:

Assigning myself to this. I presume we'll be reusing lots of TVAP code and building this into TVAP, right?

I think building it into the TVAP is the best choice. No need to run multiple different proxies

Updated by Corbin Simpson over 5 years ago

I have an SSH proxy working. It can successfully connect to nodes and run the invocation to access the console of a given instance.

I have started looking into shells. Web-shell extends Anyterm, which extends Ajaxterm. All of them use AJAX to communicate with the backend, instead of WebSockets.

  • Status changed from Feedback to In Progress

Updated by Corbin Simpson over 5 years ago

We are going to be using jsTerm (https://github.com/pnitsch/jsTerm) for the terminal; it's the only one which really does what we want using WebSockets. I will have to modify it slightly, but upstream claims to be alright with accepting and maintaining changes, so it shouldn't be a problem.

Updated by Peter Krenesky over 5 years ago

  • Target version changed from 0.8 to 0.9

Updated by Lance Albertson over 5 years ago

Where are we at with this feature? I know Corbin was working on it but I never heard what was left to do to make this work.

Updated by Lance Albertson about 5 years ago

  • Category set to VirtualMachines

Updated by Lance Albertson about 5 years ago

Where are you at with this feature? Is it at a point where it can make it into 0.9 or should we push it back to 0.9.1?

Updated by Kenneth Lett almost 5 years ago

  • Needs Tests set to No
  • Target version changed from 0.9 to 0.9.1

Updated by Kenneth Lett almost 5 years ago

  • Target version changed from 0.9.1 to 0.10

Updated by Kenneth Lett over 4 years ago

  • Target version deleted (0.10)

Updated by Chance Zibolski almost 4 years ago

  • Assignee changed from Corbin Simpson to Jack Twilley

Updated by Jack Twilley over 3 years ago

There have been no significant changes to the RAPI in this area since Lance first mentioned the 2.4 changes, so that's good.

I spoke with Corbin and now understand where the SSH proxy code is (it's in the twisted VNC auth proxy at https://github.com/osuosl/twisted_vncauthproxy/blob/master/vncap/ssh/protocol.py) and how the console is bound.

I did some additional research on alternatives to jsTerm and found tty.js which relies on node.js but is actively used for this exact functionality with Raspberry Pi devices so getting it to work should be straightforward.

Also available in: Atom PDF