내부 소개

====================================
  
1          

2 KVM       
  2.1 KVM     
  2.2 KVM vcpu
  2.3 KVM IO   
      2.3.1 IO    
      2.3.2 VirtIO

3 KVM-IO      
  3.1 Virt-IO     
  3.2          (Direct Assign)
  3.3        

===================================

1          

	               、MMU(Memory Map Unit)  IO            。

	               (Ring   )  ,         ,       ,         。    Ring        ,                      :Guest  [1],          。    Guest   ,               , Guest                     。Guest Host                     。  ,      (Virtual Machine Monitor)             (Intel  VMCS, AMD  VMCB)   Guest   Host     ,           。

	           MMU              。       ,Guest              ,         :

    Guest     -> Guest     -> Host    
                 ^               ^
                 |               |
                MMU1            MMU2

  MMU1       (Shadow paging  vTLB)      (Intel EPT、AMD NPT)。MMU2     。

	   IO     ,   VMM  Guest IO  ,                      。              ,      ( Intel VT-d),         Guest    ,      。

[1]X86              ,      Guest    non-root operation,      root operation,    host  。


2 KVM       

	KVM Linux       ,           VMM   。                                MMU    ,  Guest IO  (  Port IO mmap IO)         。  IO       ,       QEMU    。QEMU    IO   ,                  Host    , Host          IO  。            :

    +--------------+               +--------+    
    | Qemu         |               |        |   
    |              |               |        |               
    | +---+  +----+|               | Guest  |               
    | |vHD|  |vNIC||<-----+        |        |               
    | +---+  +----+|      |        |        |               
    +--------------+      |        +--------+               
         ^                |            ^                    
         | syscall        |IO stream   |                    
         | via FDs        |            |                      
    +----|----------------|------------|--------+             
    |    |                |            v        |             
    |    v                |       +----------+  |   
    |  +--------+         +------>|          |  |
    |  |drivers |<--+             |  kvm.ko  |  |
    |  +--------+   |             +----------+  |
    |    ^          |   Host kernel             |   
    +----|----------|---------------------------+
         v          v                      
    +--------+    +---+                    
    | HDD    |    |NIC|                    
    +--------+    +---+    
                             1

	 Host         ,KVM Guest                 ,      kill、top、taskset       Guest。            Guest     ,         Guest    。

	 Qemu     ,KVM          ,      KVM  ,Guest      (VM)        (VCPU)。                 ,Qemu            IOCTL        , KVM  。  ,Qemu       ,    CPU MMU       kvm.ko   。

2.1 KVM     

	KVM     (Qemu)      /dev/kvm,  open  ioctl           KVM         ,Guest      (fd_vcpu[N]), Guest     (fd_vm), KVM  (fd_kvm)。     Guest      vcpu,   vcpu   Host        。

	Qemu  Guest   ,  /dev/kvm  fd_kvm fd_vm,    fd_vm Guest “    ”mmap Qemu       ,         vcpu[N]  ,  fd_vcpu[N]。  Qemu   fd_vcpu         mmap  KVM       。     (    shared)   kvm.ko  ,  Guest IO  ,    ,    ,     。Qemu      ,                     ,  Guest IO           Host  。  Guest          Qemu       ,Qemu               Guest         。              :
                            
    +----------+            |         +--------+
    | Qemu     | Host user  |         |        |
    |          |            |         |        |
    |          |            |         | Guest  |
    |  +------+|            |         | user   |
    |  |shared||            |         |        |
    |  +------+|            |         |        |
    |       ^  |            |         |        |
    +-------|--+            |         |        |
        |   |               |         |        |
     fds|   |               |         |        |
  ------|---|---------------|         |--------|
        |   |               |         |        |
        v   v   Host kernel |         | Guest  |
     +---------+            |         | kernel |
     |         |            |         |        |
     |  kvm.ko |----+       |         |        |
     |         |    |fd_kvm |         |        |
     +---------+    |       |         +--------+
                    v                     ^
                  +----+       fd_vm      |
                  |vmcs|----+--------------
      +------+    +----+    |          +------+
      | host |              |          | Guest|
      | mode |              |fd_vcpu   | mode |       
      +------+              |          +------+
          ^                 v             ^
          |             +-------+         |
          | vm exit     |  Phy  | vm entry|
          +-------------|  CPU  |---------+
                        +-------+
                             2

	  vm-exit       host  ,  kvm Qemu   。vm-entry       Guest  ,    Guest     。    ,Qemu          kvm.ko  ,  kvm.ko  vmcs            ,      Guest     。  fd_kvm    Qemu KVM     ,    KVM    ,      、vcpu 。fd_vcpu              ,    Guest mode         (      ,    、     、     ),  Qemu    fd_vcpu mmap  KVM       。fd_vm    Qemu  Guest     , Guest       。

2.2 KVM vcpu
	
	     ,KVM vcpu   host        。 Qemu     ,      loop :

	for (;;) {
		kvm_run(vcpu);
		switch (shared_data->exit_reason) {
		...
		case KVM_IO: 
			handle_io(vcpu);
			break;
		case KVM_MMIO:
			handle_mmio(vcpu);
			break;
		...
		}
	}

    Guest vcpu    。            Guest vcpu    ,   Host     , vcpu            :Host user/Host kernel/Guest,             。 vcpu        :

      Host user   |  Host kernel  | Guest mode   |
                  |               |              |
                  |               |              |
 +->kvm_run(vcpu)-------+         |              | 
 |                |     v         |              |
 |                | +->vm entry----------+       |     
 |                | |             |      v       |
 |                | |             |   Execute    |
 |                | |             |   Natively   |
 |                | |             |      |       |
 |                | |  vm exit<----------+       |  
 |                | |    |        |              |  
 |                | |    |        |              |
 |            Yes | |    v        |              |   
 |     +----------------I/O ?     |              |    
 |     |          | |    | No     |              |
 |     |          | |    |        |              |      
 |     |          | |    v        |              |
 |     v      Yes | |  Signal     |              |
 +--Handle IO<---------Pending?   |              |
                  | |    | No     |              |
                  | +----+        |              |
                             3

   , host   ps       vcpu                    。

2.3 KVM IO   

2.3.1 IO    
	
	     ,           PCI   。PCI    PCI                          ,              。                   Port             ,           VMM    Guest PIO MMIO    ,                   。           VMM           Guest。     :

        +-----------------------------------+
        | +--------------+                  |   
        | | PCI config   |     +----------+ |
        | +--------------+<--->| driver   | | 
        | +--------------+<--->|          | | 
        | | Device memory|     +----------+ |
        | +--------------+           ^      |   
        |       ^                    |      |   
        +-------|--------------------|------+  
                |                    | vINTR via VMM    
PIO/MMIO via VMM|         +----------+           
                v         |                       
         +------------------------+    
         | +--------+  +--------+ |    
         | |  PCI   |  | Device | |    
         | | config |  | memory | |  Virtual Device    
         | +--------+  +--------+ |              
         +------------------------+    
                      |   
                      v   
                +------------+
                |host driver |
                +------------+
                               4

	                      。         ,     Guest          。            cycle     。          Guest     ,            IO                           ,     :

        +----------------+
        |                |
        | +-----------+  |
        | |para-driver|  |
        | +-----------+  |
        +-------^--------+
                |         
                | new I/O interface via VMM
                v
            +---------+                      
            |Virtual  |                       
            |device   |                       
            +---------+                       
                |                             
                v
           +------------+
           |host driver |
           +------------+
                            5

KVM virtio             IO  。

	      ,                                 Guest。              (    remmappable MSI    ),                    CPU  ,      VT-d  。                  ,   DMA Remapping, DMA    Guest        Host     ,      Remapping,  remappable        VMM  ,     IRT(Interrupt Remapping Table)      vcpu 。

	PC   ,    (       root-complex)   CPU、      。  DMA Remapping   Remapping          。    :

      +-------------+
      |cpu0, cpu1...|              
      +-------------+    
            ^    
            |        <-- System Bus  
            |                |   
            v                v   
   +---------------------+  
   |  North Bridge       |   
   |                     |       +--------+
   |    +--------+       |<----->| Memory |
   |    |  vt-d  |       |       +--------+
   |    +--------+       |   
   +---------------------+   
         ^            ^          
         |            |   
         v            v    
    +--------+    +--------+
    | PCI-e  |    | South  |<-----> PCI legacy devices...
    | device |    | Bridge |
    +--------+    +--------+
                              6

  ,    MSI PCI/PCI-e         Guest。  PCI-e               ,         Guest。           PCI               Guest。KVM            PCI-e/PCI       。

2.3.2 VirtIO

	VirtIO Guest Qemu      IO  。Guest          VirtIO        。  Guest     mmap Qemu      ,VirtIO                  (para-virtualized)   Guest            IO  。  ,KVM VirtIO   Guest VirtIO          ,     :

     +---------------+
     |  Qemu         |
     |    +--------+ |        +-------------------+
     |    | VirtIO | |        | +---------+       |
     |    | Device | |        | | VirtIO  | Guest |
     |    +--------+ |        | | Driver  |       |  
     +------|--^-----+        | +---------+       |  
            |  |              +---|---^-----------+  
      irqfd |  |              PIO |   |               
      fd_vm |  |ioeventfd         |   |vInterrupt             
   ---------|--|------------------|---|------------
            v  |                  v   |
        +----------+         +--------------+ Host
        | eventfd  |<------->|  KVM.ko      | kernel
        | core     |         |              |
        +----------+         +--------------+
                             7

    ,Guest VirtIO      port   Qemu VirtIO    IO    。       irqfd  IOCTL fd_vm  Guest  IO    。irqfd ioeventfd KVM         eventfd         ,      IO  (    IO   vcpu     )。     PIO   MMIO,   
KVM  PIO     MMIO。


3 KVM-IO      

3.1 Virt-IO     

	  1     ,Guest IO      Qemu              Host IO     Host   。                      IO      ,  Qemu                       。
	  Linux      vhost       Virt-IO          ,                    IO              。     :

                             +-------------------+
                             | +---------+       |
                             | | VirtIO  | Guest |
                             | | Driver  |       |  
                             | +-----+---+       |  
                             +---|---^-----------+  
                             PIO |   |               
                                 |   | vInterrupt             
   ------------------------------|---|--------------
                                 v   |
        +----------+         +--------------+  Host
        | Vhost    |<------->|  KVM.ko      |  kernel
        | net      |         |              |
        +----^-----+         +--------------+
             |               
             |               
         +---v----+          
         | NIC    |          
         | Driver |          
         +--------+      
                               8

	  KVM              Qemu    。      vhost                      。

3.2          (Direct Assign)

	     ,       PCI            Guest, VMM-bypass,         。     KVM    DMA     remapping  ,               Guest。     :

               +----------------+
               |  Guest         |
               |  +---------+   |
     +-------->|  | Driver  |   |
     |         |  +---------+   |
     |         +------------^---+
   D |              |       |    
   M |      DMA Req.|       | vINTR
   A |              |       |
     |      +-------|-------|----------+   
   O |      |       v KVM   |          |  
   p |      |   +------------------+   |  
   e |      |   | DMA remmapping   |   |              
   r |      |   |                  |   |  
   a |      |   | INTR remmapping  |   |              
   t |      |   +-----------^------+   |  
   i |      +-------|-------|----------+  
   o |              |       | INTR        
   n |              v       |             
     |              +---------+           
     +------------->| Deivce  |           
                    +---------+
                               9

      Guest            (   KVM   ),            ,  IO         KVM            。  ,IO      Qemu/KVM       ,  IO    、          KVM remapping    。

3.3        

	              ,         Guest      ,  m     n       (n < m)。            ,                 Guest;     n      m Guest(n < m)。       ,   KVM          ,            :KVM              (             Guest),                        。     :
                                                 
                        |                  KVM   |
                        |  Device context        |
                        |  queue                 |
           +------+     |     +-+                |
           |Guest |---------->| |                |
           -------+     |     +-+                |
                        |      |                 |
           +------+     |     +-+                |
           |Guest |---------->| |   +----------+ |
           +------+     |     +-+   | Device   | |
                        |      |    | Scheduler| |
           +------+     |     +-+   +----------+ |
           |Guest |---------->| |-----+          |
           +------+     |     +-+     |          |
                        |          +--v--------+ |
                        | Current--->+--+  DM  | |     +-----+
                        | Context  | +--+------------->| NIC |
                        |          +-----------+ |     +-----+
                        |                        |

                                 10

  ,Device Modle(DM)       remapping  ,Device Scheduler                     。             ,            ,      、  、   IO  。      ,       SR-IOV   ,          ,    (  ,         )   PCI    ,    ,               (      ,        ,          )。

  :
1[  ] qemu-kvm. git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
2[  ] Linux-2.6{/virt/kvm/*, arch/x86/kvm/*, drivers/virtio/*, drivers/block/virtio_blk.c, drivers/vhost/*}
3[  ] Intel® Virtualization Technology for Directed I/O
4[  ] Intel® 64 and IA-32 Architectures Software Developer’s Manual 3B.
5[  ] Towards Virtual Passthrough I/O on Commodity Devices. 2008.
6[  ] kvm: the Linux Virtual Machine Monitor. 2007.
7[  ] virtio: Towards a De-Facto Standard For Virtual I/O Devices. 2008
8[  ] High Performance Network Virtualization with SR-IOV. 2010.
9[  ] QEMU, a Fast and Portable Dynamic Translator. 2005.

좋은 웹페이지 즐겨찾기