PIC


Virtual I/O Device (VIRTIO) Version 1.3

Committee Specification Draft 01

06 October 2023

This stage:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/tex/ (Authoritative)
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.pdf
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html

Previous stage:
N/A

Latest stage:
https://docs.oasis-open.org/virtio/virtio/v1.3/virtio-v1.3.pdf
https://docs.oasis-open.org/virtio/virtio/v1.3/virtio-v1.3.html

Technical Committee:
OASIS Virtual I/O Device (VIRTIO) TC

Chairs:
Michael S. Tsirkin (mst@redhat.com), Red Hat
Cornelia Huck (cohuck@redhat.com), Red Hat

Editors:
Michael S. Tsirkin (mst@redhat.com), Red Hat
Cornelia Huck (cohuck@redhat.com), Red Hat

Additional artifacts:
This prose specification is one component of a Work Product that also includes:

Related work:
This specification replaces or supersedes:

Abstract:
This document describes the specifications of the “virtio” family of devices. These devices are found in virtual environments, yet by design they look like physical devices to the guest within the virtual machine - and this document treats them as such. This similarity allows the guest to use standard drivers and discovery mechanisms.

The purpose of virtio and this specification is that virtual environments and guests should have a straightforward, efficient, standard and extensible mechanism for virtual devices, rather than boutique per-environment or per-OS mechanisms.

Status:
This document was last revised or approved by the Virtual I/O Device (VIRTIO) TC on the above date. The level of approval is also listed above. Check the “Latest stage” location noted above for possible later revisions of this document. Any other numbered Versions and other technical work produced by the Technical Committee (TC) are listed at https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=virtio#technical.

Technical Committee members should send comments on this specification to the Technical Committee’s email list. Others should send comments to the Technical Committee by using the “Send A Comment” button on the Technical Committee’s web page at https://www.oasis-open.org/committees/virtio/.

This specification is provided under the Non-Assertion Mode of the OASIS IPR Policy, the mode chosen when the Technical Committee was established. For information on whether any patents have been disclosed that may be essential to implementing this specification, and any offers of patent licensing terms, please refer to the Intellectual Property Rights page in the TC’s GitHub repository (https://github.com/oasis-tcs/virtio-admin/blob/master/IPR.md).

Note that any machine-readable content (Computer Language Definitions) declared Normative for this Work Product is provided in separate plain text files. In the event of a discrepancy between any such plain text file and display content in the Work Product’s prose narrative document(s), the content in the separate plain text file prevails.

Citation format:
When referencing this specification the following citation format should be used:

[VIRTIO-v1.3]
Virtual I/O Device (VIRTIO) Version 1.3. Edited by Michael S. Tsirkin and Cornelia Huck. 06 October 2023. OASIS Committee Specification Draft 01. https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html. Latest stage: https://docs.oasis-open.org/virtio/virtio/v1.3/virtio-v1.3.html.

__________________________________________________________________

Notices

Copyright © OASIS Open 2023. All Rights Reserved.

All capitalized terms in the following text have the meanings assigned to them in the OASIS Intellectual Property Rights Policy (the "OASIS IPR Policy"). The full Policy may be found at the OASIS website.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published, and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this section are included on all such copies and derivative works. However, this document itself may not be modified in any way, including by removing the copyright notice or references to OASIS, except as needed for the purpose of developing any document or deliverable produced by an OASIS Technical Committee (in which case the rules applicable to copyrights, as set forth in the OASIS IPR Policy, must be followed) or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by OASIS or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and OASIS DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY OWNERSHIP RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. OASIS AND ITS MEMBERS WILL NOT BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF ANY USE OF THIS DOCUMENT OR ANY PART THEREOF.

As stated in the OASIS IPR Policy, the following three paragraphs in brackets apply to OASIS Standards Final Deliverable documents (Committee Specifications, OASIS Standards, or Approved Errata).

[OASIS requests that any OASIS Party or any other party that believes it has patent claims that would necessarily be infringed by implementations of this OASIS Committee Specification or OASIS Standard, to notify OASIS TC Administrator and provide an indication of its willingness to grant patent licenses to such patent claims in a manner consistent with the IPR Mode of the OASIS Technical Committee that produced this specification.]

[OASIS invites any party to contact the OASIS TC Administrator if it is aware of a claim of ownership of any patent claims that would necessarily be infringed by implementations of this specification by a patent holder that is not willing to provide a license to such patent claims in a manner consistent with the IPR Mode of the OASIS Technical Committee that produced this specification. OASIS may include such claims on its website, but disclaims any obligation to do so.]

[OASIS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on OASIS’ procedures with respect to rights in any document or deliverable produced by an OASIS Technical Committee can be found on the OASIS website. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this OASIS Committee Specification or OASIS Standard, can be obtained from the OASIS TC Administrator. OASIS makes no representation that any information or list of intellectual property rights will at any time be complete, or that any claims in such list are, in fact, Essential Claims.]

The name "OASIS" is a trademark of OASIS, the owner and developer of this specification, and should be used only to refer to the organization and its official outputs. OASIS welcomes reference to, and implementation and use of, specifications, while reserving the right to enforce its marks against misleading uses. Please see https://www.oasis-open.org/policies-guidelines/trademark/ for above guidance.

__________________________________________________________________

Table of Contents

1 Introduction
 1.1 Normative References
 1.2 Non-Normative References
 1.3 Terminology
  1.3.1 Legacy Interface: Terminology
  1.3.2 Transition from earlier specification drafts
 1.4 Structure Specifications
 1.5 Constant Specifications
2 Basic Facilities of a Virtio Device
 2.1 Device Status Field
  2.1.1 Driver Requirements: Device Status Field
  2.1.2 Device Requirements: Device Status Field
 2.2 Feature Bits
  2.2.1 Driver Requirements: Feature Bits
  2.2.2 Device Requirements: Feature Bits
  2.2.3 Legacy Interface: A Note on Feature Bits
 2.3 Notifications
 2.4 Device Reset
  2.4.1 Device Requirements: Device Reset
  2.4.2 Driver Requirements: Device Reset
 2.5 Device Configuration Space
  2.5.1 Driver Requirements: Device Configuration Space
  2.5.2 Device Requirements: Device Configuration Space
  2.5.3 Legacy Interface: A Note on Device Configuration Space endian-ness
  2.5.4 Legacy Interface: Device Configuration Space
 2.6 Virtqueues
  2.6.1 Virtqueue Reset
 2.7 Split Virtqueues
  2.7.1 Driver Requirements: Virtqueues
  2.7.2 Legacy Interfaces: A Note on Virtqueue Layout
  2.7.3 Legacy Interfaces: A Note on Virtqueue Endianness
  2.7.4 Message Framing
  2.7.5 The Virtqueue Descriptor Table
  2.7.6 The Virtqueue Available Ring
  2.7.7 Used Buffer Notification Suppression
  2.7.8 The Virtqueue Used Ring
  2.7.9 In-order use of descriptors
  2.7.10 Available Buffer Notification Suppression
  2.7.11 Helpers for Operating Virtqueues
  2.7.12 Virtqueue Operation
  2.7.13 Supplying Buffers to The Device
  2.7.14 Receiving Used Buffers From The Device
 2.8 Packed Virtqueues
  2.8.1 Driver and Device Ring Wrap Counters
  2.8.2 Polling of available and used descriptors
  2.8.3 Write Flag
  2.8.4 Element Address and Length
  2.8.5 Scatter-Gather Support
  2.8.6 Next Flag: Descriptor Chaining
  2.8.7 Indirect Flag: Scatter-Gather Support
  2.8.8 In-order use of descriptors
  2.8.9 Multi-buffer requests
  2.8.10 Driver and Device Event Suppression
  2.8.11 Driver Requirements: Virtqueues
  2.8.12 Device Requirements: Virtqueues
  2.8.13 The Virtqueue Descriptor Format
  2.8.14 Event Suppression Structure Format
  2.8.15 Device Requirements: The Virtqueue Descriptor Table
  2.8.16 Driver Requirements: The Virtqueue Descriptor Table
  2.8.17 Driver Requirements: Scatter-Gather Support
  2.8.18 Device Requirements: Scatter-Gather Support
  2.8.19 Driver Requirements: Indirect Descriptors
  2.8.20 Virtqueue Operation
  2.8.21 Supplying Buffers to The Device
  2.8.22 Receiving Used Buffers From The Device
 2.9 Driver Notifications
 2.10 Shared Memory Regions
  2.10.1 Addressing within regions
  2.10.2 Device Requirements: Shared Memory Regions
 2.11 Exporting Objects
 2.12 Device groups
  2.12.1 Group administration commands
 2.13 Administration Virtqueues
  2.13.1 Device Requirements: Group administration commands
  2.13.2 Driver Requirements: Group administration commands
3 General Initialization And Device Operation
 3.1 Device Initialization
  3.1.1 Driver Requirements: Device Initialization
  3.1.2 Legacy Interface: Device Initialization
 3.2 Device Operation
  3.2.1 Notification of Device Configuration Changes
 3.3 Device Cleanup
  3.3.1 Driver Requirements: Device Cleanup
4 Virtio Transport Options
 4.1 Virtio Over PCI Bus
  4.1.1 Device Requirements: Virtio Over PCI Bus
  4.1.2 PCI Device Discovery
  4.1.3 PCI Device Layout
  4.1.4 Virtio Structure PCI Capabilities
  4.1.5 PCI-specific Initialization And Device Operation
 4.2 Virtio Over MMIO
  4.2.1 MMIO Device Discovery
  4.2.2 MMIO Device Register Layout
  4.2.3 MMIO-specific Initialization And Device Operation
  4.2.4 Legacy interface
  4.2.5 Features reserved for future use
 4.3 Virtio Over Channel I/O
  4.3.1 Basic Concepts
  4.3.2 Device Initialization
  4.3.3 Device Operation
  4.3.4 Features reserved for future use
5 Device Types
 5.1 Network Device
  5.1.1 Device ID
  5.1.2 Virtqueues
  5.1.3 Feature bits
  5.1.4 Device configuration layout
  5.1.5 Device Initialization
  5.1.6 Device Operation
 5.2 Block Device
  5.2.1 Device ID
  5.2.2 Virtqueues
  5.2.3 Feature bits
  5.2.4 Device configuration layout
  5.2.5 Device Initialization
  5.2.6 Device Operation
 5.3 Console Device
  5.3.1 Device ID
  5.3.2 Virtqueues
  5.3.3 Feature bits
  5.3.4 Device configuration layout
  5.3.5 Device Initialization
  5.3.6 Device Operation
 5.4 Entropy Device
  5.4.1 Device ID
  5.4.2 Virtqueues
  5.4.3 Feature bits
  5.4.4 Device configuration layout
  5.4.5 Device Initialization
  5.4.6 Device Operation
 5.5 Traditional Memory Balloon Device
  5.5.1 Device ID
  5.5.2 Virtqueues
  5.5.3 Feature bits
  5.5.4 Device configuration layout
  5.5.5 Device Initialization
  5.5.6 Device Operation
 5.6 SCSI Host Device
  5.6.1 Device ID
  5.6.2 Virtqueues
  5.6.3 Feature bits
  5.6.4 Device configuration layout
  5.6.5 Device Requirements: Device Initialization
  5.6.6 Device Operation
 5.7 GPU Device
  5.7.1 Device ID
  5.7.2 Virtqueues
  5.7.3 Feature bits
  5.7.4 Device configuration layout
  5.7.5 Device Requirements: Device Initialization
  5.7.6 Device Operation
  5.7.7 VGA Compatibility
 5.8 Input Device
  5.8.1 Device ID
  5.8.2 Virtqueues
  5.8.3 Feature bits
  5.8.4 Device configuration layout
  5.8.5 Device Initialization
  5.8.6 Device Operation
 5.9 Crypto Device
  5.9.1 Device ID
  5.9.2 Virtqueues
  5.9.3 Feature bits
  5.9.4 Supported crypto services
  5.9.5 Device configuration layout
  5.9.6 Device Initialization
  5.9.7 Device Operation
 5.10 Socket Device
  5.10.1 Device ID
  5.10.2 Virtqueues
  5.10.3 Feature bits
  5.10.4 Device configuration layout
  5.10.5 Device Initialization
  5.10.6 Device Operation
 5.11 File System Device
  5.11.1 Device ID
  5.11.2 Virtqueues
  5.11.3 Feature bits
  5.11.4 Device configuration layout
  5.11.5 Device Initialization
  5.11.6 Device Operation
 5.12 RPMB Device
  5.12.1 Device ID
  5.12.2 Virtqueues
  5.12.3 Feature bits
  5.12.4 Device configuration layout
  5.12.5 Device Requirements: Device Initialization
  5.12.6 Device Operation
 5.13 IOMMU device
  5.13.1 Device ID
  5.13.2 Virtqueues
  5.13.3 Feature bits
  5.13.4 Device configuration layout
  5.13.5 Device initialization
  5.13.6 Device operations
 5.14 Sound Device
  5.14.1 Device ID
  5.14.2 Virtqueues
  5.14.3 Feature Bits
  5.14.4 Device Configuration Layout
  5.14.5 Device Initialization
  5.14.6 Device Operation
 5.15 Memory Device
  5.15.1 Device ID
  5.15.2 Virtqueues
  5.15.3 Feature bits
  5.15.4 Device configuration layout
  5.15.5 Device Initialization
  5.15.6 Device Operation
 5.16 I2C Adapter Device
  5.16.1 Device ID
  5.16.2 Virtqueues
  5.16.3 Feature bits
  5.16.4 Device configuration layout
  5.16.5 Device Initialization
  5.16.6 Device Operation
 5.17 SCMI Device
  5.17.1 Device ID
  5.17.2 Virtqueues
  5.17.3 Feature bits
  5.17.4 Device configuration layout
  5.17.5 Device Initialization
  5.17.6 Device Operation
 5.18 GPIO Device
  5.18.1 Device ID
  5.18.2 Virtqueues
  5.18.3 Feature bits
  5.18.4 Device configuration layout
  5.18.5 Device Initialization
  5.18.6 Device Operation: requestq
  5.18.7 Device Operation: eventq
 5.19 PMEM Device
  5.19.1 Device ID
  5.19.2 Virtqueues
  5.19.3 Feature bits
  5.19.4 Device configuration layout
  5.19.5 Device Initialization
  5.19.6 Driver Operations
  5.19.7 Device Operations
  5.19.8 Possible security implications
  5.19.9 Countermeasures
6 Reserved Feature Bits
 6.1 Driver Requirements: Reserved Feature Bits
 6.2 Device Requirements: Reserved Feature Bits
 6.3 Legacy Interface: Reserved Feature Bits
7 Conformance
 7.1 Conformance Targets
 7.2 Clause 1: Driver Conformance
  7.2.1 Clause 2: PCI Driver Conformance
  7.2.2 Clause 3: MMIO Driver Conformance
  7.2.3 Clause 4: Channel I/O Driver Conformance
  7.2.4 Clause 5: Network Driver Conformance
  7.2.5 Clause 6: Block Driver Conformance
  7.2.6 Clause 7: Console Driver Conformance
  7.2.7 Clause 8: Entropy Driver Conformance
  7.2.8 Clause 9: Traditional Memory Balloon Driver Conformance
  7.2.9 Clause 10: SCSI Host Driver Conformance
  7.2.10 Clause 11: Input Driver Conformance
  7.2.11 Clause 12: Crypto Driver Conformance
  7.2.12 Clause 13: Socket Driver Conformance
  7.2.13 Clause 14: File System Driver Conformance
  7.2.14 Clause 15: RPMB Driver Conformance
  7.2.15 Clause 16: IOMMU Driver Conformance
  7.2.16 Clause 17: Sound Driver Conformance
  7.2.17 Clause 18: Memory Driver Conformance
  7.2.18 Clause 19: I2C Adapter Driver Conformance
  7.2.19 Clause 20: SCMI Driver Conformance
  7.2.20 Clause 21: GPIO Driver Conformance
  7.2.21 Clause 22: PMEM Driver Conformance
 7.3 Clause 23: Device Conformance
  7.3.1 Clause 24: PCI Device Conformance
  7.3.2 Clause 25: MMIO Device Conformance
  7.3.3 Clause 26: Channel I/O Device Conformance
  7.3.4 Clause 27: Network Device Conformance
  7.3.5 Clause 28: Block Device Conformance
  7.3.6 Clause 29: Console Device Conformance
  7.3.7 Clause 30: Entropy Device Conformance
  7.3.8 Clause 31: Traditional Memory Balloon Device Conformance
  7.3.9 Clause 32: SCSI Host Device Conformance
  7.3.10 Clause 33: GPU Device Conformance
  7.3.11 Clause 34: Input Device Conformance
  7.3.12 Clause 35: Crypto Device Conformance
  7.3.13 Clause 36: Socket Device Conformance
  7.3.14 Clause 37: File System Device Conformance
  7.3.15 Clause 38: RPMB Device Conformance
  7.3.16 Clause 39: IOMMU Device Conformance
  7.3.17 Clause 40: Sound Device Conformance
  7.3.18 Clause 41: Memory Device Conformance
  7.3.19 Clause 42: I2C Adapter Device Conformance
  7.3.20 Clause 43: SCMI Device Conformance
  7.3.21 Clause 44: GPIO Device Conformance
  7.3.22 Clause 45: PMEM Device Conformance
 7.4 Clause 46: Legacy Interface: Transitional Device and Transitional Driver Conformance
A virtio_queue.h
B Creating New Device Types
 B.1 How Many Virtqueues?
 B.2 What Device Configuration Space Layout?
 B.3 What Device Number?
 B.4 How many MSI-X vectors? (for PCI)
 B.5 Device Improvements
C Acknowledgements
D Revision History


1 Introduction

This document describes the specifications of the “virtio” family of devices. These devices are found in virtual environments, yet by design they look like physical devices to the guest within the virtual machine - and this document treats them as such. This similarity allows the guest to use standard drivers and discovery mechanisms.

The purpose of virtio and this specification is that virtual environments and guests should have a straightforward, efficient, standard and extensible mechanism for virtual devices, rather than boutique per-environment or per-OS mechanisms.

Straightforward:
Virtio devices use normal bus mechanisms of interrupts and DMA which should be familiar to any device driver author. There is no exotic page-flipping or COW mechanism: it’s just a normal device.1
Efficient:
Virtio devices consist of rings of descriptors for both input and output, which are neatly laid out to avoid cache effects from both driver and device writing to the same cache lines.
Standard:
Virtio makes no assumptions about the environment in which it operates, beyond supporting the bus to which device is attached. In this specification, virtio devices are implemented over MMIO, Channel I/O and PCI bus transports 2, earlier drafts have been implemented on other buses not included here.
Extensible:
Virtio devices contain feature bits which are acknowledged by the guest operating system during device setup. This allows forwards and backwards compatibility: the device offers all the features it knows about, and the driver acknowledges those it understands and wishes to use.

1.1 Normative References

[RFC2119]

Bradner S., “Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997.
http://www.ietf.org/rfc/rfc2119.txt

[RFC4122]

Leach, P., Mealling, M., and R. Salz, “A Universally Unique IDentifier (UUID) URN Namespace”, RFC 4122, DOI 10.17487/RFC4122, July 2005.
http://www.ietf.org/rfc/rfc4122.txt

[S390 PoP]

z/Architecture Principles of Operation, IBM Publication SA22-7832,
https://www.ibm.com/docs/en/SSQ2R2_15.0.0/com.ibm.tpf.toolkit.hlasm.doc/dz9zr006.pdf, and any future revisions

[S390 Common I/O]

ESA/390 Common I/O-Device and Self-Description, IBM Publication SA22-7204,
https://www.ibm.com/resources/publications/OutputPubsDetails?PubID=SA22720401, and any future revisions

[PCI]

Conventional PCI Specifications,
http://www.pcisig.com/specifications/conventional/, PCI-SIG

[PCIe]

PCI Express Specifications
http://www.pcisig.com/specifications/pciexpress/, PCI-SIG

[IEEE 802]

IEEE Standard for Local and Metropolitan Area Networks: Overview and Architecture,
http://www.ieee802.org/, IEEE

[SAM]

SCSI Architectural Model,
http://www.t10.org/cgi-bin/ac.pl?t=f&f=sam4r05.pdf

[SCSI MMC]

SCSI Multimedia Commands,
http://www.t10.org/cgi-bin/ac.pl?t=f&f=mmc6r00.pdf

[FUSE]

Linux FUSE interface,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h

[errno]

Linux error names and numbers,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/asm-generic/errno-base.h

[eMMC]

eMMC Electrical Standard (5.1), JESD84-B51,
http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf

[HDA]

High Definition Audio Specification,
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf

[I2C]

I2C-bus specification and user manual,
https://www.nxp.com/docs/en/user-guide/UM10204.pdf

[SCMI]

Arm System Control and Management Interface, DEN0056,
https://developer.arm.com/docs/den0056/c, version C and any future revisions

[RFC3447]

J. Jonsson.,“Public-Key Cryptography Standards (PKCS) #1: RSA Cryptography”, February 2003.
https://www.ietf.org/rfc/rfc3447.txt

[FIPS186-3]

National Institute of Standards and Technology (NIST), FIPS Publication 180-3: Secure Hash Standard, October 2008.
https://csrc.nist.gov/csrc/media/publications/fips/186/3/archive/2009-06-25/documents/fips_186-3.pdf

[RFC5915]

“Elliptic Curve Private Key Structure”, June 2010.
https://www.rfc-editor.org/rfc/rfc5915

[RFC6025]

C.Wallace., “ASN.1 Translation”, October 2010.
https://www.ietf.org/rfc/rfc6025.txt

[RFC3279]

W.Polk., “Algorithms and Identifiers for the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile”, April 2002.
https://www.ietf.org/rfc/rfc3279.txt

[SEC1]

Standards for Efficient Cryptography Group(SECG), “SEC1: Elliptic Cureve Cryptography”, Version 1.0, September 2000.
https://www.secg.org/sec1-v2.pdf

[RFC2784]

Generic Routing Encapsulation. This protocol is only specified for IPv4 and used as either the payload or delivery protocol.
https://datatracker.ietf.org/doc/rfc2784/

[RFC2890]

Key and Sequence Number Extensions to GRE. This protocol describes extensions by which two fields, Key and Sequence Number, can be optionally carried in the GRE Header.
https://www.rfc-editor.org/rfc/rfc2890

[RFC7676]

IPv6 Support for Generic Routing Encapsulation (GRE). This protocol is specified for IPv6 and used as either the payload or delivery protocol. Note that this does not change the GRE header format or any behaviors specified by RFC 2784 or RFC 2890.
https://datatracker.ietf.org/doc/rfc7676/

[GRE-in-UDP]

GRE-in-UDP Encapsulation. This specifies a method of encapsulating network protocol packets within GRE and UDP headers. This protocol is specified for IPv4 and IPv6, and used as either the payload or delivery protocol.
https://www.rfc-editor.org/rfc/rfc8086

[VXLAN]

Virtual eXtensible Local Area Network.
https://datatracker.ietf.org/doc/rfc7348/

[VXLAN-GPE]

Generic Protocol Extension for VXLAN. This protocol describes extending Virtual eXtensible Local Area Network (VXLAN) via changes to the VXLAN header.
https://www.ietf.org/archive/id/draft-ietf-nvo3-vxlan-gpe-12.txt

[GENEVE]

Generic Network Virtualization Encapsulation.
https://datatracker.ietf.org/doc/rfc8926/

[IPIP]

IP Encapsulation within IP.
https://www.rfc-editor.org/rfc/rfc2003

[NVGRE]

NVGRE: Network Virtualization Using Generic Routing Encapsulation
https://www.rfc-editor.org/rfc/rfc7637.html

[IP]

INTERNET PROTOCOL
https://www.rfc-editor.org/rfc/rfc791

[UDP]

User Datagram Protocol
https://www.rfc-editor.org/rfc/rfc768

[TCP]

TRANSMISSION CONTROL PROTOCOL
https://www.rfc-editor.org/rfc/rfc793

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017
http://www.ietf.org/rfc/rfc8174.txt

1.2 Non-Normative References

[Virtio PCI Draft]

Virtio PCI Draft Specification
http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf

1.3 Terminology

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals, as shown here.

1.3.1 Legacy Interface: Terminology

Specification drafts preceding version 1.0 of this specification (e.g. see [Virtio PCI Draft]) defined a similar, but different interface between the driver and the device. Since these are widely deployed, this specification accommodates OPTIONAL features to simplify transition from these earlier draft interfaces.

Specifically devices and drivers MAY support:

Legacy Interface
is an interface specified by an earlier draft of this specification (before 1.0)
Legacy Device
is a device implemented before this specification was released, and implementing a legacy interface on the host side
Legacy Driver
is a driver implemented before this specification was released, and implementing a legacy interface on the guest side

Legacy devices and legacy drivers are not compliant with this specification.

To simplify transition from these earlier draft interfaces, a device MAY implement:

Transitional Device
a device supporting both drivers conforming to this specification, and allowing legacy drivers.

Similarly, a driver MAY implement:

Transitional Driver
a driver supporting both devices conforming to this specification, and legacy devices.
Note: Legacy interfaces are not required; ie. don’t implement them unless you have a need for backwards compatibility!

Devices or drivers with no legacy compatibility are referred to as non-transitional devices and drivers, respectively.

1.3.2 Transition from earlier specification drafts

For devices and drivers already implementing the legacy interface, some changes will have to be made to support this specification.

In this case, it might be beneficial for the reader to focus on sections tagged "Legacy Interface" in the section title. These highlight the changes made since the earlier drafts.

1.4 Structure Specifications

Many device and driver in-memory structure layouts are documented using the C struct syntax. All structures are assumed to be without additional padding. To stress this, cases where common C compilers are known to insert extra padding within structures are tagged using the GNU C __attribute__((packed)) syntax.

For the integer data types used in the structure definitions, the following conventions are used:

u8, u16, u32, u64
An unsigned integer of the specified length in bits.
le16, le32, le64
An unsigned integer of the specified length in bits, in little-endian byte order.
be16, be32, be64
An unsigned integer of the specified length in bits, in big-endian byte order.

Some of the fields to be defined in this specification don’t start or don’t end on a byte boundary. Such fields are called bit-fields. A set of bit-fields is always a sub-division of an integer typed field.

Bit-fields within integer fields are always listed in order, from the least significant to the most significant bit. The bit-fields are considered unsigned integers of the specified width with the next in significance relationship of the bits preserved.

For example:

struct S { 
        be16 { 
                A : 15; 
                B : 1; 
        } x; 
        be16 y; 
};

documents the value A stored in the low 15 bit of x and the value B stored in the high bit of x, the 16-bit integer x in turn stored using the big-endian byte order at the beginning of the structure S, and being followed immediately by an unsigned integer y stored in big-endian byte order at an offset of 2 bytes (16 bits) from the beginning of the structure.

Note that this notation somewhat resembles the C bitfield syntax but should not be naively converted to a bitfield notation for portable code: it matches the way bitfields are packed by C compilers on little-endian architectures but not the way bitfields are packed by C compilers on big-endian architectures.

Assuming that CPU_TO_BE16 converts a 16-bit integer from a native CPU to the big-endian byte order, the following is the equivalent portable C code to generate a value to be stored into x:

CPU_TO_BE16(B << 15 | A)

1.5 Constant Specifications

In many cases, numeric values used in the interface between the device and the driver are documented using the C #define and /* */ comment syntax. Multiple related values are grouped together with a common name as a prefix, using _ as a separator. Using _XXX as a suffix refers to all values in a group. For example:

/* Field Fld value A description */ 
#define VIRTIO_FLD_A        (1 << 0) 
/* Field Fld value B description */ 
#define VIRTIO_FLD_B        (1 << 1)

documents two numeric values for a field Fld, with Fld having value 1 referring to A and Fld having value 2 referring to B. Note that << refers to the shift-left operation.

Further, in this case VIRTIO_FLD_A and VIRTIO_FLD_B refer to values 1 and 2 of Fld respectively. Further, VIRTIO_FLD_XXX refers to either VIRTIO_FLD_A or VIRTIO_FLD_B.


2 Basic Facilities of a Virtio Device

A virtio device is discovered and identified by a bus-specific method (see the bus specific sections: 4.1 Virtio Over PCI Bus, 4.2 Virtio Over MMIO and 4.3 Virtio Over Channel I/O). Each device consists of the following parts:

2.1 Device Status Field

During device initialization by a driver, the driver follows the sequence of steps specified in 3.1.

The device status field provides a simple low-level indication of the completed steps of this sequence. It’s most useful to imagine it hooked up to traffic lights on the console indicating the status of each device. The following bits are defined (listed below in the order in which they would be typically set):

ACKNOWLEDGE (1)
Indicates that the guest OS has found the device and recognized it as a valid virtio device.
DRIVER (2)
Indicates that the guest OS knows how to drive the device. Note: There could be a significant (or infinite) delay before setting this bit. For example, under Linux, drivers can be loadable modules.
FAILED (128)
Indicates that something went wrong in the guest, and it has given up on the device. This could be an internal error, or the driver didn’t like the device for some reason, or even a fatal error during device operation.
FEATURES_OK (8)
Indicates that the driver has acknowledged all the features it understands, and feature negotiation is complete.
DRIVER_OK (4)
Indicates that the driver is set up and ready to drive the device.
DEVICE_NEEDS_RESET (64)
Indicates that the device has experienced an error from which it can’t recover.

The device status field starts out as 0, and is reinitialized to 0 by the device during reset.

2.1.1 Driver Requirements: Device Status Field

The driver MUST update device status, setting bits to indicate the completed steps of the driver initialization sequence specified in 3.1. The driver MUST NOT clear a device status bit. If the driver sets the FAILED bit, the driver MUST later reset the device before attempting to re-initialize.

The driver SHOULD NOT rely on completion of operations of a device if DEVICE_NEEDS_RESET is set. Note: For example, the driver can’t assume requests in flight will be completed if DEVICE_NEEDS_RESET is set, nor can it assume that they have not been completed. A good implementation will try to recover by issuing a reset.

2.1.2 Device Requirements: Device Status Field

The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK.

The device SHOULD set DEVICE_NEEDS_RESET when it enters an error state that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver.

2.2 Feature Bits

Each virtio device offers all the features it understands. During device initialization, the driver reads this and tells the device the subset that it accepts. The only way to renegotiate is to reset the device.

This allows for forwards and backwards compatibility: if the device is enhanced with a new feature bit, older drivers will not write that feature bit back to the device. Similarly, if a driver is enhanced with a feature that the device doesn’t support, it see the new feature is not offered.

Feature bits are allocated as follows:

0 to 23, and 50 to 127
Feature bits for the specific device type
24 to 41
Feature bits reserved for extensions to the queue and feature negotiation mechanisms
42 to 49, and 128 and above
Feature bits reserved for future extensions.
Note: For example, feature bit 0 for a network device (i.e. Device ID 1) indicates that the device supports checksumming of packets.

In particular, new fields in the device configuration space are indicated by offering a new feature bit.

To keep the feature negotiation mechanism extensible, it is important that devices do not offer any feature bits that they would not be able to handle if the driver accepted them (even though drivers are not supposed to accept any unspecified, reserved, or unsupported features even if offered, according to the specification.) Likewise, it is important that drivers do not accept feature bits they do not know how to handle (even though devices are not supposed to offer any unspecified, reserved, or unsupported features in the first place, according to the specification.) The preferred way for handling reserved and unexpected features is that the driver ignores them.

In particular, this is especially important for features limited to specific transports, as enabling these for more transports in future versions of the specification is highly likely to require changing the behaviour from drivers and devices. Drivers and devices supporting multiple transports need to carefully maintain per-transport lists of allowed features.

2.2.1 Driver Requirements: Feature Bits

The driver MUST NOT accept a feature which the device did not offer, and MUST NOT accept a feature which requires another feature which was not accepted.

The driver MUST validate the feature bits offered by the device. The driver MUST ignore and MUST NOT accept any feature bit that is

The driver SHOULD go into backwards compatibility mode if the device does not offer a feature it understands, otherwise MUST set the FAILED device status bit and cease initialization.

By contrast, the driver MUST NOT fail solely because a feature it does not understand has been offered by the device.

2.2.2 Device Requirements: Feature Bits

The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK device status bit when the driver writes it.

The device MUST NOT offer feature bits corresponding to features it would not support if accepted by the driver (even if the driver is prohibited from accepting the feature bits by the specification); for the sake of clarity, this refers to feature bits not described in this specification, reserved feature bits and feature bits reserved or not supported for the specific transport or the specific device type, but this does not preclude devices written to a future version of this specification from offering such feature bits should such a specification have a provision for devices to support the corresponding features.

If a device has successfully negotiated a set of features at least once (by accepting the FEATURES_OK device status bit during device initialization), then it SHOULD NOT fail re-negotiation of the same set of features after a device or system reset. Failure to do so would interfere with resuming from suspend and error recovery.

2.2.3 Legacy Interface: A Note on Feature Bits

Transitional Drivers MUST detect Legacy Devices by detecting that the feature bit VIRTIO_F_VERSION_1 is not offered. Transitional devices MUST detect Legacy drivers by detecting that VIRTIO_F_VERSION_1 has not been acknowledged by the driver.

In this case device is used through the legacy interface.

Legacy interface support is OPTIONAL. Thus, both transitional and non-transitional devices and drivers are compliant with this specification.

Requirements pertaining to transitional devices and drivers is contained in sections named ’Legacy Interface’ like this one.

When device is used through the legacy interface, transitional devices and transitional drivers MUST operate according to the requirements documented within these legacy interface sections. Specification text within these sections generally does not apply to non-transitional devices.

2.3 Notifications

The notion of sending a notification (driver to device or device to driver) plays an important role in this specification. The modus operandi of the notifications is transport specific.

There are three types of notifications:

Configuration change notifications and used buffer notifications are sent by the device, the recipient is the driver. A configuration change notification indicates that the device configuration space has changed; a used buffer notification indicates that a buffer may have been made used on the virtqueue designated by the notification.

Available buffer notifications are sent by the driver, the recipient is the device. This type of notification indicates that a buffer may have been made available on the virtqueue designated by the notification.

The semantics, the transport-specific implementations, and other important aspects of the different notifications are specified in detail in the following chapters.

Most transports implement notifications sent by the device to the driver using interrupts. Therefore, in previous versions of this specification, these notifications were often called interrupts. Some names defined in this specification still retain this interrupt terminology. Occasionally, the term event is used to refer to a notification or a receipt of a notification.

2.4 Device Reset

The driver may want to initiate a device reset at various times; notably, it is required to do so during device initialization and device cleanup.

The mechanism used by the driver to initiate the reset is transport specific.

2.4.1 Device Requirements: Device Reset

A device MUST reinitialize device status to 0 after receiving a reset.

A device MUST NOT send notifications or interact with the queues after indicating completion of the reset by reinitializing device status to 0, until the driver re-initializes the device.

2.4.2 Driver Requirements: Device Reset

The driver SHOULD consider a driver-initiated reset complete when it reads device status as 0.

2.5 Device Configuration Space

Device configuration space is generally used for rarely-changing or initialization-time parameters. Where configuration fields are optional, their existence is indicated by feature bits: Future versions of this specification will likely extend the device configuration space by adding extra fields at the tail. Note: The device configuration space uses the little-endian format for multi-byte fields.

Each transport also provides a generation count for the device configuration space, which will change whenever there is a possibility that two accesses to the device configuration space can see different versions of that space.

2.5.1 Driver Requirements: Device Configuration Space

Drivers MUST NOT assume reads from fields greater than 32 bits wide are atomic, nor are reads from multiple fields: drivers SHOULD read device configuration space fields like so:

u32 before, after; 
do { 
        before = get_config_generation(device); 
        // read config entry/entries. 
        after = get_config_generation(device); 
} while (after != before);

For optional configuration space fields, the driver MUST check that the corresponding feature is offered before accessing that part of the configuration space. Note: See section 3.1 for details on feature negotiation.

Drivers MUST NOT limit structure size and device configuration space size. Instead, drivers SHOULD only check that device configuration space is large enough to contain the fields necessary for device operation. Note: For example, if the specification states that device configuration space ’includes a single 8-bit field’ drivers should understand this to mean that the device configuration space might also include an arbitrary amount of tail padding, and accept any device configuration space size equal to or greater than the specified 8-bit size.

2.5.2 Device Requirements: Device Configuration Space

The device MUST allow reading of any device-specific configuration field before FEATURES_OK is set by the driver. This includes fields which are conditional on feature bits, as long as those feature bits are offered by the device.

2.5.3 Legacy Interface: A Note on Device Configuration Space endian-ness

Note that for legacy interfaces, device configuration space is generally the guest’s native endian, rather than PCI’s little-endian. The correct endian-ness is documented for each device.

2.5.4 Legacy Interface: Device Configuration Space

Legacy devices did not have a configuration generation field, thus are susceptible to race conditions if configuration is updated. This affects the block capacity (see 5.2.4) and network mac (see 5.1.4) fields; when using the legacy interface, drivers SHOULD read these fields multiple times until two reads generate a consistent result.

2.6 Virtqueues

The mechanism for bulk data transport on virtio devices is pretentiously called a virtqueue. Each device can have zero or more virtqueues3.

A virtio device can have maximum of 65536 virtqueues. Each virtqueue is identified by a virtqueue index. A virtqueue index has a value in the range of 0 to 65535.

Driver makes requests available to device by adding an available buffer to the queue, i.e., adding a buffer describing the request to a virtqueue, and optionally triggering a driver event, i.e., sending an available buffer notification to the device.

Device executes the requests and - when complete - adds a used buffer to the queue, i.e., lets the driver know by marking the buffer as used. Device can then trigger a device event, i.e., send a used buffer notification to the driver.

Device reports the number of bytes it has written to memory for each buffer it uses. This is referred to as “used length”.

Device is not generally required to use buffers in the same order in which they have been made available by the driver.

Some devices always use descriptors in the same order in which they have been made available. These devices can offer the VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge might allow optimizations or simplify driver and/or device code.

Each virtqueue can consist of up to 3 parts:

Note: Note that previous versions of this spec used different names for these parts (following 2.7):

Two formats are supported: Split Virtqueues (see 2.7 Split Virtqueues) and Packed Virtqueues (see 2.8 Packed Virtqueues).

Every driver and device supports either the Packed or the Split Virtqueue format, or both.

2.6.1 Virtqueue Reset

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue individually. The way to reset the virtqueue is transport specific.

Virtqueue reset is divided into two parts. The driver first resets a queue and can afterwards optionally re-enable it.

2.6.1.1 Virtqueue Reset

2.6.1.1.1 Device Requirements: Virtqueue Reset
After a queue has been reset by the driver, the device MUST NOT execute any requests from that virtqueue, or notify the driver for it.

The device MUST reset any state of a virtqueue to the default state, including the available state and the used state.

2.6.1.1.2 Driver Requirements: Virtqueue Reset
After the driver tells the device to reset a queue, the driver MUST verify that the queue has actually been reset.

After the queue has been successfully reset, the driver MAY release any resource associated with that virtqueue.

2.6.1.2 Virtqueue Re-enable

This process is the same as the initialization process of a single queue during the initialization of the entire device.

2.6.1.2.1 Device Requirements: Virtqueue Re-enable
The device MUST observe any queue configuration that may have been changed by the driver, like the maximum queue size.

2.6.1.2.2 Driver Requirements: Virtqueue Re-enable
When re-enabling a queue, the driver MUST configure the queue resources as during initial virtqueue discovery, but optionally with different parameters.

2.7 Split Virtqueues

The split virtqueue format was the only format supported by the version 1.0 (and earlier) of this standard.

The split virtqueue format separates the virtqueue into several parts, where each part is write-able by either the driver or the device, but not both. Multiple parts and/or locations within a part need to be updated when making a buffer available and when marking it as used.

Each queue has a 16-bit queue size parameter, which sets the number of entries and implies the total size of the queue.

Each virtqueue consists of three parts:

where each part is physically-contiguous in guest memory, and has different alignment requirements.

The memory alignment and size requirements, in bytes, of each part of the virtqueue are summarized in the following table:




Virtqueue Part Alignment Size






Descriptor Table 16 16(Queue Size)



Available Ring 2 6 + 2(Queue Size)



Used Ring 4 6 + 8(Queue Size)



The Alignment column gives the minimum alignment for each part of the virtqueue.

The Size column gives the total number of bytes for each part of the virtqueue.

Queue Size corresponds to the maximum number of buffers in the virtqueue4. Queue Size value is always a power of 2. The maximum Queue Size value is 32768. This value is specified in a bus-specific way.

When the driver wants to send a buffer to the device, it fills in a slot in the descriptor table (or chains several together), and writes the descriptor index into the available ring. It then notifies the device. When the device has finished a buffer, it writes the descriptor index into the used ring, and sends a used buffer notification.

2.7.1 Driver Requirements: Virtqueues

The driver MUST ensure that the physical address of the first byte of each virtqueue part is a multiple of the specified alignment value in the above table.

2.7.2 Legacy Interfaces: A Note on Virtqueue Layout

For Legacy Interfaces, several additional restrictions are placed on the virtqueue layout:

Each virtqueue occupies two or more physically-contiguous pages (usually defined as 4096 bytes, but depending on the transport; henceforth referred to as Queue Align) and consists of three parts:




Descriptor Table Available Ring (…padding…) Used Ring



The bus-specific Queue Size field controls the total number of bytes for the virtqueue. When using the legacy interface, the transitional driver MUST retrieve the Queue Size field from the device and MUST allocate the total number of bytes for the virtqueue according to the following formula (Queue Align given in qalign and Queue Size given in qsz):

#define ALIGN(x) (((x) + qalign) & "qalign) 
static inline unsigned virtq_size(unsigned int qsz) 
{ 
     return ALIGN(sizeof(struct virtq_desc)*qsz + sizeof(u16)*(3 + qsz)) 
          + ALIGN(sizeof(u16)*3 + sizeof(struct virtq_used_elem)*qsz); 
}

This wastes some space with padding. When using the legacy interface, both transitional devices and drivers MUST use the following virtqueue layout structure to locate elements of the virtqueue:

struct virtq { 
        // The actual descriptors (16 bytes each) 
        struct virtq_desc desc[ Queue Size ]; 
 
        // A ring of available descriptor heads with free-running index. 
        struct virtq_avail avail; 
 
        // Padding to the next Queue Align boundary. 
        u8 pad[ Padding ]; 
 
        // A ring of used descriptor heads with free-running index. 
        struct virtq_used used; 
};

2.7.3 Legacy Interfaces: A Note on Virtqueue Endianness

Note that when using the legacy interface, transitional devices and drivers MUST use the native endian of the guest as the endian of fields and in the virtqueue. This is opposed to little-endian for non-legacy interface as specified by this standard. It is assumed that the host is already aware of the guest endian.

2.7.4 Message Framing

The framing of messages with descriptors is independent of the contents of the buffers. For example, a network transmit buffer consists of a 12 byte header followed by the network packet. This could be most simply placed in the descriptor table as a 12 byte output descriptor followed by a 1514 byte output descriptor, but it could also consist of a single 1526 byte output descriptor in the case where the header and packet are adjacent, or even three or more descriptors (possibly with loss of efficiency in that case).

Note that, some device implementations have large-but-reasonable restrictions on total descriptor size (such as based on IOV_MAX in the host OS). This has not been a problem in practice: little sympathy will be given to drivers which create unreasonably-sized descriptors such as by dividing a network packet into 1500 single-byte descriptors!

2.7.4.1 Device Requirements: Message Framing

The device MUST NOT make assumptions about the particular arrangement of descriptors. The device MAY have a reasonable limit of descriptors it will allow in a chain.

2.7.4.2 Driver Requirements: Message Framing

The driver MUST place any device-writable descriptor elements after any device-readable descriptor elements.

The driver SHOULD NOT use an excessive number of descriptors to describe a buffer.

2.7.4.3 Legacy Interface: Message Framing

Regrettably, initial driver implementations used simple layouts, and devices came to rely on it, despite this specification wording. In addition, the specification for virtio_blk SCSI commands required intuiting field lengths from frame boundaries (see 5.2.6.3 Legacy Interface: Device Operation)

Thus when using the legacy interface, the VIRTIO_F_ANY_LAYOUT feature indicates to both the device and the driver that no assumptions were made about framing. Requirements for transitional drivers when this is not negotiated are included in each device section.

2.7.5 The Virtqueue Descriptor Table

The descriptor table refers to the buffers the driver is using for the device. addr is a physical address, and the buffers can be chained via next. Each descriptor describes a buffer which is read-only for the device (“device-readable”) or write-only for the device (“device-writable”), but a chain of descriptors can contain both device-readable and device-writable buffers.

The actual contents of the memory offered to the device depends on the device type. Most common is to begin the data with a header (containing little-endian fields) for the device to read, and postfix it with a status tailer for the device to write.

struct virtq_desc { 
        /* Address (guest-physical). */ 
        le64 addr; 
        /* Length. */ 
        le32 len; 
 
/* This marks a buffer as continuing via the next field. */ 
#define VIRTQ_DESC_F_NEXT   1 
/* This marks a buffer as device write-only (otherwise device read-only). */ 
#define VIRTQ_DESC_F_WRITE     2 
/* This means the buffer contains a list of buffer descriptors. */ 
#define VIRTQ_DESC_F_INDIRECT   4 
        /* The flags as indicated above. */ 
        le16 flags; 
        /* Next field if flags & NEXT */ 
        le16 next; 
};

The number of descriptors in the table is defined by the queue size for this virtqueue: this is the maximum possible descriptor chain length.

If VIRTIO_F_IN_ORDER has been negotiated, driver uses descriptors in ring order: starting from offset 0 in the table, and wrapping around at the end of the table. Note: The legacy [Virtio PCI Draft] referred to this structure as vring_desc, and the constants as VRING_DESC_F_NEXT, etc, but the layout and values were identical.

2.7.5.1 Device Requirements: The Virtqueue Descriptor Table

A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT read a device-writable buffer (it MAY do so for debugging or diagnostic purposes). A device MUST NOT write to any descriptor table entry.

2.7.5.2 Driver Requirements: The Virtqueue Descriptor Table

Drivers MUST NOT add a descriptor chain longer than 232 bytes in total; this implies that loops in the descriptor chain are forbidden!

If VIRTIO_F_IN_ORDER has been negotiated, and when making a descriptor with VRING_DESC_F_NEXT set in flags at offset x in the table available to the device, driver MUST set next to 0 for the last descriptor in the table (where x = queue_size 1) and to x + 1 for the rest of the descriptors.

2.7.5.3 Indirect Descriptors

Some devices benefit by concurrently dispatching a large number of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this (see A virtio_queue.h). To increase ring capacity the driver can store a table of indirect descriptors anywhere in memory, and insert a descriptor in main virtqueue (with flags&VIRTQ_DESC_F_INDIRECT on) that refers to memory buffer containing this indirect descriptor table; addr and len refer to the indirect table address and length in bytes, respectively.

The indirect table layout structure looks like this (len is the length of the descriptor that refers to this table, which is a variable, so this code won’t compile):

struct indirect_descriptor_table { 
        /* The actual descriptors (16 bytes each) */ 
        struct virtq_desc desc[len / 16]; 
};

The first indirect descriptor is located at start of the indirect descriptor table (index 0), additional indirect descriptors are chained by next. An indirect descriptor without a valid next (with flags&VIRTQ_DESC_F_NEXT off) signals the end of the descriptor. A single indirect descriptor table can include both device-readable and device-writable descriptors.

If VIRTIO_F_IN_ORDER has been negotiated, indirect descriptors use sequential indices, in-order: index 0 followed by index 1 followed by index 2, etc.

2.7.5.3.1 Driver Requirements: Indirect Descriptors
The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the VIRTIO_F_INDIRECT_DESC feature was negotiated. The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag within an indirect descriptor (ie. only one table per descriptor).

A driver MUST NOT create a descriptor chain longer than the Queue Size of the device.

A driver MUST NOT set both VIRTQ_DESC_F_INDIRECT and VIRTQ_DESC_F_NEXT in flags.

If VIRTIO_F_IN_ORDER has been negotiated, indirect descriptors MUST appear sequentially, with next taking the value of 1 for the 1st descriptor, 2 for the 2nd one, etc.

2.7.5.3.2 Device Requirements: Indirect Descriptors
The device MUST ignore the write-only flag (flags&VIRTQ_DESC_F_WRITE) in the descriptor that refers to an indirect table.

The device MUST handle the case of zero or more normal chained descriptors followed by a single descriptor with flags&VIRTQ_DESC_F_INDIRECT. Note: While unusual (most implementations either create a chain solely using non-indirect descriptors, or use a single indirect element), such a layout is valid.

2.7.6 The Virtqueue Available Ring

The available ring has the following layout structure:

struct virtq_avail { 
#define VIRTQ_AVAIL_F_NO_INTERRUPT      1 
        le16 flags; 
        le16 idx; 
        le16 ring[ /* Queue Size */ ]; 
        le16 used_event; /* Only if VIRTIO_F_EVENT_IDX */ 
};

The driver uses the available ring to offer buffers to the device: each ring entry refers to the head of a descriptor chain. It is only written by the driver and read by the device.

idx field indicates where the driver would put the next descriptor entry in the ring (modulo the queue size). This starts at 0, and increases. Note: The legacy [Virtio PCI Draft] referred to this structure as vring_avail, and the constant as VRING_AVAIL_F_NO_INTERRUPT, but the layout and value were identical.

2.7.6.1 Driver Requirements: The Virtqueue Available Ring

A driver MUST NOT decrement the available idx on a virtqueue (ie. there is no way to “unexpose” buffers).

2.7.7 Used Buffer Notification Suppression

If the VIRTIO_F_EVENT_IDX feature bit is not negotiated, the flags field in the available ring offers a crude mechanism for the driver to inform the device that it doesn’t want notifications when buffers are used. Otherwise used_event is a more performant alternative where the driver specifies how far the device can progress before a notification is required.

Neither of these notification suppression methods are reliable, as they are not synchronized with the device, but they serve as useful optimizations.

2.7.7.1 Driver Requirements: Used Buffer Notification Suppression

If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:

The driver MUST handle spurious notifications from the device.

2.7.7.2 Device Requirements: Used Buffer Notification Suppression

If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:

Note: For example, if used_event is 0, then a device using

VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).

2.7.8 The Virtqueue Used Ring

The used ring has the following layout structure:

struct virtq_used { 
#define VIRTQ_USED_F_NO_NOTIFY  1 
        le16 flags; 
        le16 idx; 
        struct virtq_used_elem ring[ /* Queue Size */]; 
        le16 avail_event; /* Only if VIRTIO_F_EVENT_IDX */ 
}; 
 
/* le32 is used here for ids for padding reasons. */ 
struct virtq_used_elem { 
        /* Index of start of used descriptor chain. */ 
        le32 id; 
        /* 
         * The number of bytes written into the device writable portion of 
         * the buffer described by the descriptor chain. 
         */ 
        le32 len; 
};

The used ring is where the device returns buffers once it is done with them: it is only written to by the device, and read by the driver.

Each entry in the ring is a pair: id indicates the head entry of the descriptor chain describing the buffer (this matches an entry placed in the available ring by the guest earlier), and len the total of bytes written into the buffer. Note: len is particularly useful for drivers using untrusted buffers: if a driver does not know exactly how much has been written by the device, the driver would have to zero the buffer in advance to ensure no data leakage occurs.

For example, a network driver may hand a received buffer directly to an unprivileged userspace application. If the network device has not overwritten the bytes which were in that buffer, this could leak the contents of freed memory from other processes to the application.

idx field indicates where the device would put the next descriptor entry in the ring (modulo the queue size). This starts at 0, and increases. Note: The legacy [Virtio PCI Draft] referred to these structures as vring_used and vring_used_elem, and the constant as VRING_USED_F_NO_NOTIFY, but the layout and value were identical.

2.7.8.1 Legacy Interface: The Virtqueue Used Ring

Historically, many drivers ignored the len value, as a result, many devices set len incorrectly. Thus, when using the legacy interface, it is generally a good idea to ignore the len value in used ring entries if possible. Specific known issues are listed per device type.

2.7.8.2 Device Requirements: The Virtqueue Used Ring

The device MUST set len prior to updating the used idx.

The device MUST write at least len bytes to descriptor, beginning at the first device-writable buffer, prior to updating the used idx.

The device MAY write more than len bytes to descriptor. Note: There are potential error cases where a device might not know what parts of the buffers have been written. This is why len is permitted to be an underestimate: that’s preferable to the driver believing that uninitialized memory has been overwritten when it has not.

2.7.8.3 Driver Requirements: The Virtqueue Used Ring

The driver MUST NOT make assumptions about data in device-writable buffers beyond the first len bytes, and SHOULD ignore this data.

2.7.9 In-order use of descriptors

Some devices always use descriptors in the same order in which they have been made available. These devices can offer the VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge allows devices to notify the use of a batch of buffers to the driver by only writing out a single used ring entry with the id corresponding to the head entry of the descriptor chain describing the last buffer in the batch.

The device then skips forward in the ring according to the size of the batch. Accordingly, it increments the used idx by the size of the batch.

The driver needs to look up the used id and calculate the batch size to be able to advance to where the next used ring entry will be written by the device.

This will result in the used ring entry at an offset matching the first available ring entry in the batch, the used ring entry for the next batch at an offset matching the first available ring entry in the next batch, etc.

The skipped buffers (for which no used ring entry was written) are assumed to have been used (read or written) by the device completely.

2.7.10 Available Buffer Notification Suppression

The device can suppress available buffer notifications in a manner analogous to the way drivers can suppress used buffer notifications as detailed in section 2.7.7. The device manipulates flags or avail_event in the used ring the same way the driver manipulates flags or used_event in the available ring.

2.7.10.1 Driver Requirements: Available Buffer Notification Suppression

The driver MUST initialize flags in the used ring to 0 when allocating the used ring.

If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:

2.7.10.2 Device Requirements: Available Buffer Notification Suppression

If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:

The device MUST handle spurious notifications from the driver.

2.7.11 Helpers for Operating Virtqueues

The Linux Kernel Source code contains the definitions above and helper routines in a more usable form, in include/uapi/linux/virtio_ring.h. This was explicitly licensed by IBM and Red Hat under the (3-clause) BSD license so that it can be freely used by all other projects, and is reproduced (with slight variation) in A virtio_queue.h.

2.7.12 Virtqueue Operation

There are two parts to virtqueue operation: supplying new available buffers to the device, and processing used buffers from the device. Note: As an example, the simplest virtio network device has two virtqueues: the transmit virtqueue and the receive virtqueue. The driver adds outgoing (device-readable) packets to the transmit virtqueue, and then frees them after they are used. Similarly, incoming (device-writable) buffers are added to the receive virtqueue, and processed after they are used.

What follows is the requirements of each of these two parts when using the split virtqueue format in more detail.

2.7.13 Supplying Buffers to The Device

The driver offers buffers to one of the device’s virtqueues as follows:

1.
The driver places the buffer into free descriptor(s) in the descriptor table, chaining as necessary (see 2.7.5 The Virtqueue Descriptor Table).
2.
The driver places the index of the head of the descriptor chain into the next ring entry of the available ring.
3.
Steps 1 and 2 MAY be performed repeatedly if batching is possible.
4.
The driver performs a suitable memory barrier to ensure the device sees the updated descriptor table and available ring before the next step.
5.
The available idx is increased by the number of descriptor chain heads added to the available ring.
6.
The driver performs a suitable memory barrier to ensure that it updates the idx field before checking for notification suppression.
7.
The driver sends an available buffer notification to the device if such notifications are not suppressed.

Note that the above code does not take precautions against the available ring buffer wrapping around: this is not possible since the ring buffer is the same size as the descriptor table, so step (1) will prevent such a condition.

In addition, the maximum queue size is 32768 (the highest power of 2 which fits in 16 bits), so the 16-bit idx value can always distinguish between a full and empty buffer.

What follows is the requirements of each stage in more detail.

2.7.13.1 Placing Buffers Into The Descriptor Table

A buffer consists of zero or more device-readable physically-contiguous elements followed by zero or more physically-contiguous device-writable elements (each has at least one element). This algorithm maps it into the descriptor table to form a descriptor chain:

for each buffer element, b:

1.
Get the next free descriptor table entry, d
2.
Set d.addr to the physical address of the start of b
3.
Set d.len to the length of b.
4.
If b is device-writable, set d.flags to VIRTQ_DESC_F_WRITE, otherwise 0.
5.
If there is a buffer element after this:
(a)
Set d.next to the index of the next free descriptor element.
(b)
Set the VIRTQ_DESC_F_NEXT bit in d.flags.

In practice, d.next is usually used to chain free descriptors, and a separate count kept to check there are enough free descriptors before beginning the mappings.

2.7.13.2 Updating The Available Ring

The descriptor chain head is the first d in the algorithm above, ie. the index of the descriptor table entry referring to the first part of the buffer. A naive driver implementation MAY do the following (with the appropriate conversion to-and-from little-endian assumed):

avail->ring[avail->idx % qsz] = head;

However, in general the driver MAY add many descriptor chains before it updates idx (at which point they become visible to the device), so it is common to keep a counter of how many the driver has added:

avail->ring[(avail->idx + added++) % qsz] = head;
2.7.13.3 Updating idx

idx always increments, and wraps naturally at 65536:

avail->idx += added;

Once available idx is updated by the driver, this exposes the descriptor and its contents. The device MAY access the descriptor chains the driver created and the memory they refer to immediately.

2.7.13.3.1 Driver Requirements: Updating idx
The driver MUST perform a suitable memory barrier before the idx update, to ensure the device sees the most up-to-date copy.
2.7.13.4 Notifying The Device

The actual method of device notification is bus-specific, but generally it can be expensive. So the device MAY suppress such notifications if it doesn’t need them, as detailed in section 2.7.10.

The driver has to be careful to expose the new idx value before checking if notifications are suppressed.

2.7.13.4.1 Driver Requirements: Notifying The Device
The driver MUST perform a suitable memory barrier before reading flags or avail_event, to avoid missing a notification.

2.7.14 Receiving Used Buffers From The Device

Once the device has used buffers referred to by a descriptor (read from or written to them, or parts of both, depending on the nature of the virtqueue and the device), it sends a used buffer notification to the driver as detailed in section 2.7.7. Note:

For optimal performance, a driver MAY disable used buffer notifications while processing the used ring, but beware the problem of missing notifications between emptying the ring and reenabling notifications. This is usually handled by re-checking for more used buffers after notifications are re-enabled:

virtq_disable_used_buffer_notifications(vq); 
 
for (;;) { 
        if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) { 
                virtq_enable_used_buffer_notifications(vq); 
                mb(); 
 
                if (vq->last_seen_used != le16_to_cpu(virtq->used.idx)) 
                        break; 
 
                virtq_disable_used_buffer_notifications(vq); 
        } 
 
        struct virtq_used_elem *e = virtq.used->ring[vq->last_seen_used%vsz]; 
        process_buffer(e); 
        vq->last_seen_used++; 
}

2.8 Packed Virtqueues

Packed virtqueues is an alternative compact virtqueue layout using read-write memory, that is memory that is both read and written by both host and guest.

Use of packed virtqueues is negotiated by the VIRTIO_F_RING_PACKED feature bit.

Packed virtqueues support up to 215 entries each.

With current transports, virtqueues are located in guest memory allocated by the driver. Each packed virtqueue consists of three parts:

Where the Descriptor Ring in turn consists of descriptors, and where each descriptor can contain the following parts:

A buffer consists of zero or more device-readable physically-contiguous elements followed by zero or more physically-contiguous device-writable elements (each buffer has at least one element).

When the driver wants to send such a buffer to the device, it writes at least one available descriptor describing elements of the buffer into the Descriptor Ring. The descriptor(s) are associated with a buffer by means of a Buffer ID stored within the descriptor.

The driver then notifies the device. When the device has finished processing the buffer, it writes a used device descriptor including the Buffer ID into the Descriptor Ring (overwriting a driver descriptor previously made available), and sends a used event notification.

The Descriptor Ring is used in a circular manner: the driver writes descriptors into the ring in order. After reaching the end of the ring, the next descriptor is placed at the head of the ring. Once the ring is full of driver descriptors, the driver stops sending new requests and waits for the device to start processing descriptors and to write out some used descriptors before making new driver descriptors available.

Similarly, the device reads descriptors from the ring in order and detects that a driver descriptor has been made available. As processing of descriptors is completed, used descriptors are written by the device back into the ring.

Note: after reading driver descriptors and starting their processing in order, the device might complete their processing out of order. Used device descriptors are written in the order in which their processing is complete.

The Device Event Suppression data structure is write-only by the device. It includes information for reducing the number of device events, i.e., sending fewer available buffer notifications to the device.

The Driver Event Suppression data structure is read-only by the device. It includes information for reducing the number of driver events, i.e., sending fewer used buffer notifications to the driver.

2.8.1 Driver and Device Ring Wrap Counters

Each of the driver and the device are expected to maintain, internally, a single-bit ring wrap counter initialized to 1.

The counter maintained by the driver is called the Driver Ring Wrap Counter. The driver changes the value of this counter each time it makes available the last descriptor in the ring (after making the last descriptor available).

The counter maintained by the device is called the Device Ring Wrap Counter. The device changes the value of this counter each time it uses the last descriptor in the ring (after marking the last descriptor used).

It is easy to see that the Driver Ring Wrap Counter in the driver matches the Device Ring Wrap Counter in the device when both are processing the same descriptor, or when all available descriptors have been used.

To mark a descriptor as available and used, both the driver and the device use the following two flags:

#define VIRTQ_DESC_F_AVAIL     (1 << 7) 
#define VIRTQ_DESC_F_USED      (1 << 15)

To mark a descriptor as available, the driver sets the VIRTQ_DESC_F_AVAIL bit in Flags to match the internal Driver Ring Wrap Counter. It also sets the VIRTQ_DESC_F_USED bit to match the inverse value (i.e. to not match the internal Driver Ring Wrap Counter).

To mark a descriptor as used, the device sets the VIRTQ_DESC_F_USED bit in Flags to match the internal Device Ring Wrap Counter. It also sets the VIRTQ_DESC_F_AVAIL bit to match the same value.

Thus VIRTQ_DESC_F_AVAIL and VIRTQ_DESC_F_USED bits are different for an available descriptor and equal for a used descriptor.

Note that this observation is mostly useful for sanity-checking as these are necessary but not sufficient conditions - for example, all descriptors are zero-initialized. To detect used and available descriptors it is possible for drivers and devices to keep track of the last observed value of VIRTQ_DESC_F_USED/VIRTQ_DESC_F_AVAIL. Other techniques to detect VIRTQ_DESC_F_AVAIL/VIRTQ_DESC_F_USED bit changes might also be possible.

2.8.2 Polling of available and used descriptors

Writes of device and driver descriptors can generally be reordered, but each side (driver and device) are only required to poll (or test) a single location in memory: the next device descriptor after the one they processed previously, in circular order.

Sometimes the device needs to only write out a single used descriptor after processing a batch of multiple available descriptors. As described in more detail below, this can happen when using descriptor chaining or with in-order use of descriptors. In this case, the device writes out a used descriptor with the buffer id of the last descriptor in the group. After processing the used descriptor, both device and driver then skip forward in the ring the number of the remaining descriptors in the group until processing (reading for the driver and writing for the device) the next used descriptor.

2.8.3 Write Flag

In an available descriptor, the VIRTQ_DESC_F_WRITE bit within Flags is used to mark a descriptor as corresponding to a write-only or read-only element of a buffer.

/* This marks a descriptor as device write-only (otherwise device read-only). */ 
#define VIRTQ_DESC_F_WRITE     2

In a used descriptor, this bit is used to specify whether any data has been written by the device into any parts of the buffer.

2.8.4 Element Address and Length

In an available descriptor, Element Address corresponds to the physical address of the buffer element. The length of the element assumed to be physically contiguous is stored in Element Length.

In a used descriptor, Element Address is unused. Element Length specifies the length of the buffer that has been initialized (written to) by the device.

Element Length is reserved for used descriptors without the VIRTQ_DESC_F_WRITE flag, and is ignored by drivers.

2.8.5 Scatter-Gather Support

Some drivers need an ability to supply a list of multiple buffer elements (also known as a scatter/gather list) with a request. Two features support this: descriptor chaining and indirect descriptors.

If neither feature is in use by the driver, each buffer is physically-contiguous, either read-only or write-only and is described completely by a single descriptor.

While unusual (most implementations either create all lists solely using non-indirect descriptors, or always use a single indirect element), if both features have been negotiated, mixing indirect and non-indirect descriptors in a ring is valid, as long as each list only contains descriptors of a given type.

Scatter/gather lists only apply to available descriptors. A single used descriptor corresponds to the whole list.

The device limits the number of descriptors in a list through a transport-specific and/or device-specific value. If not limited, the maximum number of descriptors in a list is the virt queue size.

2.8.6 Next Flag: Descriptor Chaining

The packed ring format allows the driver to supply a scatter/gather list to the device by using multiple descriptors, and setting the VIRTQ_DESC_F_NEXT bit in Flags for all but the last available descriptor.

/* This marks a buffer as continuing. */ 
#define VIRTQ_DESC_F_NEXT   1

Buffer ID is included in the last descriptor in the list.

The driver always makes the first descriptor in the list available after the rest of the list has been written out into the ring. This guarantees that the device will never observe a partial scatter/gather list in the ring.

Note: all flags, including VIRTQ_DESC_F_AVAIL, VIRTQ_DESC_F_USED, VIRTQ_DESC_F_WRITE must be set/cleared correctly in all descriptors in the list, not just the first one.

The device only writes out a single used descriptor for the whole list. It then skips forward according to the number of descriptors in the list. The driver needs to keep track of the size of the list corresponding to each buffer ID, to be able to skip to where the next used descriptor is written by the device.

For example, if descriptors are used in the same order in which they are made available, this will result in the used descriptor overwriting the first available descriptor in the list, the used descriptor for the next list overwriting the first available descriptor in the next list, etc.

VIRTQ_DESC_F_NEXT is reserved in used descriptors, and should be ignored by drivers.

2.8.7 Indirect Flag: Scatter-Gather Support

Some devices benefit by concurrently dispatching a large number of large requests. The VIRTIO_F_INDIRECT_DESC feature allows this. To increase ring capacity the driver can store a (read-only by the device) table of indirect descriptors anywhere in memory, and insert a descriptor in the main virtqueue (with Flags bit VIRTQ_DESC_F_INDIRECT on) that refers to a buffer element containing this indirect descriptor table; addr and len refer to the indirect table address and length in bytes, respectively.

/* This means the element contains a table of descriptors. */ 
#define VIRTQ_DESC_F_INDIRECT   4

The indirect table layout structure looks like this (len is the Buffer Length of the descriptor that refers to this table, which is a variable):

struct pvirtq_indirect_descriptor_table { 
        /* The actual descriptor structures (struct pvirtq_desc each) */ 
        struct pvirtq_desc desc[len / sizeof(struct pvirtq_desc)]; 
};

The first descriptor is located at the start of the indirect descriptor table, additional indirect descriptors come immediately afterwards. The VIRTQ_DESC_F_WRITE flags bit is the only valid flag for descriptors in the indirect table. Others are reserved and are ignored by the device. Buffer ID is also reserved and is ignored by the device.

In descriptors with VIRTQ_DESC_F_INDIRECT set VIRTQ_DESC_F_WRITE is reserved and is ignored by the device.

2.8.8 In-order use of descriptors

Some devices always use descriptors in the same order in which they have been made available. These devices can offer the VIRTIO_F_IN_ORDER feature. If negotiated, this knowledge allows devices to notify the use of a batch of buffers to the driver by only writing out a single used descriptor with the Buffer ID corresponding to the last descriptor in the batch.

The device then skips forward in the ring according to the size of the batch. The driver needs to look up the used Buffer ID and calculate the batch size to be able to advance to where the next used descriptor will be written by the device.

This will result in the used descriptor overwriting the first available descriptor in the batch, the used descriptor for the next batch overwriting the first available descriptor in the next batch, etc.

The skipped buffers (for which no used descriptor was written) are assumed to have been used (read or written) by the device completely.

2.8.9 Multi-buffer requests

Some devices combine multiple buffers as part of processing of a single request. These devices always mark the descriptor corresponding to the first buffer in the request used after the rest of the descriptors (corresponding to rest of the buffers) in the request - which follow the first descriptor in ring order - has been marked used and written out into the ring. This guarantees that the driver will never observe a partial request in the ring.

2.8.10 Driver and Device Event Suppression

In many systems used and available buffer notifications involve significant overhead. To mitigate this overhead, each virtqueue includes two identical structures used for controlling notifications between the device and the driver.

The Driver Event Suppression structure is read-only by the device and controls the used buffer notifications sent by the device to the driver.

The Device Event Suppression structure is read-only by the driver and controls the available buffer notifications sent by the driver to the device.

Each of these Event Suppression structures includes the following fields:

Descriptor Ring Change Event Flags
Takes values:
/* Enable events */ 
#define RING_EVENT_FLAGS_ENABLE 0x0 
/* Disable events */ 
#define RING_EVENT_FLAGS_DISABLE 0x1 
/* 
 * Enable events for a specific descriptor 
 * (as specified by Descriptor Ring Change Event Offset/Wrap Counter). 
 * Only valid if VIRTIO_F_EVENT_IDX has been negotiated. 
 */ 
#define RING_EVENT_FLAGS_DESC 0x2 
/* The value 0x3 is reserved */
Descriptor Ring Change Event Offset
If Event Flags set to descriptor specific event: offset within the ring (in units of descriptor size). Event will only trigger when this descriptor is made available/used respectively.
Descriptor Ring Change Event Wrap Counter
If Event Flags set to descriptor specific event: offset within the ring (in units of descriptor size). Event will only trigger when Ring Wrap Counter matches this value and a descriptor is made available/used respectively.

After writing out some descriptors, both the device and the driver are expected to consult the relevant structure to find out whether a used respectively an available buffer notification should be sent.

2.8.10.1 Structure Size and Alignment

Each part of the virtqueue is physically-contiguous in guest memory, and has different alignment requirements.

The memory alignment and size requirements, in bytes, of each part of the virtqueue are summarized in the following table:




Virtqueue Part Alignment Size






Descriptor Ring 16 16(Queue Size)



Device Event Suppression 4 4



Driver Event Suppression 4 4



The Alignment column gives the minimum alignment for each part of the virtqueue.

The Size column gives the total number of bytes for each part of the virtqueue.

Queue Size corresponds to the maximum number of descriptors in the virtqueue5. The Queue Size value does not have to be a power of 2.

2.8.11 Driver Requirements: Virtqueues

The driver MUST ensure that the physical address of the first byte of each virtqueue part is a multiple of the specified alignment value in the above table.

2.8.12 Device Requirements: Virtqueues

The device MUST start processing driver descriptors in the order in which they appear in the ring. The device MUST start writing device descriptors into the ring in the order in which they complete. The device MAY reorder descriptor writes once they are started.

2.8.13 The Virtqueue Descriptor Format

The available descriptor refers to the buffers the driver is sending to the device. addr is a physical address, and the descriptor is identified with a buffer using the id field.

struct pvirtq_desc { 
        /* Buffer Address. */ 
        le64 addr; 
        /* Buffer Length. */ 
        le32 len; 
        /* Buffer ID. */ 
        le16 id; 
        /* The flags depending on descriptor type. */ 
        le16 flags; 
};

The descriptor ring is zero-initialized.

2.8.14 Event Suppression Structure Format

The following structure is used to reduce the number of notifications sent between driver and device.

struct pvirtq_event_suppress { 
        le16 { 
             desc_event_off : 15; /* Descriptor Ring Change Event Offset */ 
             desc_event_wrap : 1; /* Descriptor Ring Change Event Wrap Counter */ 
        } desc; /* If desc_event_flags set to RING_EVENT_FLAGS_DESC */ 
        le16 { 
             desc_event_flags : 2, /* Descriptor Ring Change Event Flags */ 
             reserved : 14; /* Reserved, set to 0 */ 
        } flags; 
};

2.8.15 Device Requirements: The Virtqueue Descriptor Table

A device MUST NOT write to a device-readable buffer, and a device SHOULD NOT read a device-writable buffer. A device MUST NOT use a descriptor unless it observes the VIRTQ_DESC_F_AVAIL bit in its flags being changed (e.g. as compared to the initial zero value). A device MUST NOT change a descriptor after changing it’s the VIRTQ_DESC_F_USED bit in its flags.

2.8.16 Driver Requirements: The Virtqueue Descriptor Table

A driver MUST NOT change a descriptor unless it observes the VIRTQ_DESC_F_USED bit in its flags being changed. A driver MUST NOT change a descriptor after changing the VIRTQ_DESC_F_AVAIL bit in its flags. When notifying the device, driver MUST set next_off and next_wrap to match the next descriptor not yet made available to the device. A driver MAY send multiple available buffer notifications without making any new descriptors available to the device.

2.8.17 Driver Requirements: Scatter-Gather Support

A driver MUST NOT create a descriptor list longer than allowed by the device.

A driver MUST NOT create a descriptor list longer than the Queue Size.

This implies that loops in the descriptor list are forbidden!

The driver MUST place any device-writable descriptor elements after any device-readable descriptor elements.

A driver MUST NOT depend on the device to use more descriptors to be able to write out all descriptors in a list. A driver MUST make sure there’s enough space in the ring for the whole list before making the first descriptor in the list available to the device.

A driver MUST NOT make the first descriptor in the list available before all subsequent descriptors comprising the list are made available.

2.8.18 Device Requirements: Scatter-Gather Support

The device MUST use descriptors in a list chained by the VIRTQ_DESC_F_NEXT flag in the same order that they were made available by the driver.

The device MAY limit the number of buffers it will allow in a list.

2.8.19 Driver Requirements: Indirect Descriptors

The driver MUST NOT set the VIRTQ_DESC_F_INDIRECT flag unless the VIRTIO_F_INDIRECT_DESC feature was negotiated. The driver MUST NOT set any flags except DESC_F_WRITE within an indirect descriptor.

A driver MUST NOT create a descriptor chain longer than allowed by the device.

A driver MUST NOT write direct descriptors with VIRTQ_DESC_F_INDIRECT set in a scatter-gather list linked by VIRTQ_DESC_F_NEXT. flags.

2.8.20 Virtqueue Operation

There are two parts to virtqueue operation: supplying new available buffers to the device, and processing used buffers from the device.

What follows is the requirements of each of these two parts when using the packed virtqueue format in more detail.

2.8.21 Supplying Buffers to The Device

The driver offers buffers to one of the device’s virtqueues as follows:

1.
The driver places the buffer into free descriptor(s) in the Descriptor Ring.
2.
The driver performs a suitable memory barrier to ensure that it updates the descriptor(s) before checking for notification suppression.
3.
If notifications are not suppressed, the driver notifies the device of the new available buffers.

What follows are the requirements of each stage in more detail.

2.8.21.1 Placing Available Buffers Into The Descriptor Ring

For each buffer element, b:

1.
Get the next descriptor table entry, d
2.
Get the next free buffer id value
3.
Set d.addr to the physical address of the start of b
4.
Set d.len to the length of b.
5.
Set d.id to the buffer id
6.
Calculate the flags as follows:
(a)
If b is device-writable, set the VIRTQ_DESC_F_WRITE bit to 1, otherwise 0
(b)
Set the VIRTQ_DESC_F_AVAIL bit to the current value of the Driver Ring Wrap Counter
(c)
Set the VIRTQ_DESC_F_USED bit to inverse value
7.
Perform a memory barrier to ensure that the descriptor has been initialized
8.
Set d.flags to the calculated flags value
9.
If d is the last descriptor in the ring, toggle the Driver Ring Wrap Counter
10.
Otherwise, increment d to point at the next descriptor

This makes a single descriptor buffer available. However, in general the driver MAY make use of a batch of descriptors as part of a single request. In that case, it defers updating the descriptor flags for the first descriptor (and the previous memory barrier) until after the rest of the descriptors have been initialized.

Once the descriptor flags field is updated by the driver, this exposes the descriptor and its contents. The device MAY access the descriptor and any following descriptors the driver created and the memory they refer to immediately.

2.8.21.1.1 Driver Requirements: Updating flags
The driver MUST perform a suitable memory barrier before the flags update, to ensure the device sees the most up-to-date copy.
2.8.21.2 Sending Available Buffer Notifications

The actual method of device notification is bus-specific, but generally it can be expensive. So the device MAY suppress such notifications if it doesn’t need them, using the Event Suppression structure comprising the Device Area as detailed in section 2.8.14.

The driver has to be careful to expose the new flags value before checking if notifications are suppressed.

2.8.21.3 Implementation Example

Below is a driver code example. It does not attempt to reduce the number of available buffer notifications, neither does it support the VIRTIO_F_EVENT_IDX feature.

/* Note: vq->avail_wrap_count is initialized to 1 */ 
/* Note: vq->sgs is an array same size as the ring */ 
 
id = alloc_id(vq); 
 
first = vq->next_avail; 
sgs = 0; 
for (each buffer element b) { 
        sgs++; 
 
        vq->ids[vq->next_avail] = -1; 
        vq->desc[vq->next_avail].address = get_addr(b); 
        vq->desc[vq->next_avail].len = get_len(b); 
 
        avail = vq->avail_wrap_count ? VIRTQ_DESC_F_AVAIL : 0; 
        used = !vq->avail_wrap_count ? VIRTQ_DESC_F_USED : 0; 
        f = get_flags(b) | avail | used; 
        if (b is not the last buffer element) { 
                f |= VIRTQ_DESC_F_NEXT; 
        } 
 
        /* Dont mark the 1st descriptor available until all of them are ready. */ 
        if (vq->next_avail == first) { 
                flags = f; 
        } else { 
                vq->desc[vq->next_avail].flags = f; 
        } 
 
        last = vq->next_avail; 
 
        vq->next_avail++; 
 
        if (vq->next_avail >= vq->size) { 
                vq->next_avail = 0; 
                vq->avail_wrap_count ^= 1; 
        } 
} 
vq->sgs[id] = sgs; 
/* ID included in the last descriptor in the list */ 
vq->desc[last].id = id; 
write_memory_barrier(); 
vq->desc[first].flags = flags; 
 
memory_barrier(); 
 
if (vq->device_event.flags != RING_EVENT_FLAGS_DISABLE) { 
        notify_device(vq); 
}

2.8.21.3.1 Driver Requirements: Sending Available Buffer Notifications
The driver MUST perform a suitable memory barrier before reading the Event Suppression structure occupying the Device Area. Failing to do so could result in mandatory available buffer notifications not being sent.

2.8.22 Receiving Used Buffers From The Device

Once the device has used buffers referred to by a descriptor (read from or written to them, or parts of both, depending on the nature of the virtqueue and the device), it sends a used buffer notification to the driver as detailed in section 2.8.14. Note:

For optimal performance, a driver MAY disable used buffer notifications while processing the used buffers, but beware the problem of missing notifications between emptying the ring and reenabling used buffer notifications. This is usually handled by re-checking for more used buffers after notifications are re-enabled:

/* Note: vq->used_wrap_count is initialized to 1 */ 
 
vq->driver_event.flags = RING_EVENT_FLAGS_DISABLE; 
 
for (;;) { 
        struct pvirtq_desc *d = vq->desc[vq->next_used]; 
 
        /* 
         * Check that 
         * 1. Descriptor has been made available. This check is necessary 
         *    if the driver is making new descriptors available in parallel 
         *    with this processing of used descriptors (e.g. from another thread). 
         *    Note: there are many other ways to check this, e.g. 
         *    track the number of outstanding available descriptors or buffers 
         *    and check that its not 0. 
         * 2. Descriptor has been used by the device. 
         */ 
        flags = d->flags; 
        bool avail = flags & VIRTQ_DESC_F_AVAIL; 
        bool used = flags & VIRTQ_DESC_F_USED; 
        if (avail != vq->used_wrap_count || used != vq->used_wrap_count) { 
                vq->driver_event.flags = RING_EVENT_FLAGS_ENABLE; 
                memory_barrier(); 
 
                /* 
                 * Re-test in case the driver made more descriptors available in 
                 * parallel with the used descriptor processing (e.g. from another 
                 * thread) and/or the device used more descriptors before the driver 
                 * enabled events. 
                 */ 
                flags = d->flags; 
                bool avail = flags & VIRTQ_DESC_F_AVAIL; 
                bool used = flags & VIRTQ_DESC_F_USED; 
                if (avail != vq->used_wrap_count || used != vq->used_wrap_count) { 
                        break; 
                } 
 
                vq->driver_event.flags = RING_EVENT_FLAGS_DISABLE; 
        } 
 
        read_memory_barrier(); 
 
        /* skip descriptors until the next buffer */ 
        id = d->id; 
        assert(id < vq->size); 
        sgs = vq->sgs[id]; 
        vq->next_used += sgs; 
        if (vq->next_used >= vq->size) { 
                vq->next_used -= vq->size; 
                vq->used_wrap_count ^= 1; 
        } 
 
        free_id(vq, id); 
 
        process_buffer(d); 
}

2.9 Driver Notifications

The driver is sometimes required to send an available buffer notification to the device.

When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, this notification contains either a virtqueue index if VIRTIO_F_NOTIF_CONFIG_DATA is not negotiated or device supplied virtqueue notification config data if VIRTIO_F_NOTIF_CONFIG_DATA is negotiated.

The notification method and supplying any such virtqueue notification config data is transport specific.

However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in memory: for efficiency or as a debugging aid.

To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has been negotiated, driver notifications to the device include the following information:

vq_index or vq_notif_config_data
Either virtqueue index or device supplied queue notification config data corresponding to a virtqueue.
next_off
Offset within the ring where the next available ring entry will be written. When VIRTIO_F_RING_PACKED has not been negotiated this refers to the 15 least significant bits of the available index. When VIRTIO_F_RING_PACKED has been negotiated this refers to the offset (in units of descriptor entries) within the descriptor ring where the next available descriptor will be written.
next_wrap
Wrap Counter. With VIRTIO_F_RING_PACKED this is the wrap counter referring to the next available descriptor. Without VIRTIO_F_RING_PACKED this is the most significant bit (bit 15) of the available index.

Note that the driver can send multiple notifications even without making any more buffers available. When VIRTIO_F_NOTIFICATION_DATA has been negotiated, these notifications would then have identical next_off and next_wrap values.

2.10 Shared Memory Regions

Shared memory regions are an additional facility available to devices that need a region of memory that’s continuously shared between the device and the driver, rather than passed between them in the way virtqueue elements are.

Example uses include shared caches and version pools for versioned data structures.

The memory region is allocated by the device and presented to the driver. Where the device is implemented in software on a host, this arrangement allows the memory region to be allocated by a library on the host, which the device may not have full control over.

A device may have multiple shared memory regions associated with it. Each region has a shmid to identify it, the meaning of which is device-specific.

Enumeration and location of shared memory regions is performed in a transport-specific way.

Memory consistency rules vary depending on the region and the device and they will be specified as required by each device.

2.10.1 Addressing within regions

References into shared memory regions are represented as offsets from the beginning of the region instead of absolute memory addresses. Offsets are used both for references between structures stored within shared memory and for requests placed in virtqueues that refer to shared memory. The shmid may be explicit or may be inferred from the context of the reference.

2.10.2 Device Requirements: Shared Memory Regions

Shared memory regions MUST NOT expose shared memory regions which are used to control the operation of the device, nor to stream data.

2.11 Exporting Objects

When an object created by one virtio device needs to be shared with a seperate virtio device, the first device can export the object by generating a UUID which can then be passed to the second device to identify the object.

What constitutes an object, how to export objects, and how to import objects are defined by the individual device types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by [RFC4122].

2.12 Device groups

It is occasionally useful to have a device control a group of other devices. Terminology used in such cases:

Device group
or just group, includes zero or more devices.
Owner device
or owner, the device controlling the group.
Member device
a device within a group. The owner device itself is not a member of the group.
Member identifier
each member has this identifier, unique within the group and used to address it through the owner device.
Group type identifier
specifies what kind of member devices there are in a group, how the member identifier is interpreted and what kind of control the owner has. A given owner can control multiple groups of different types but only a single group of a given type, thus the type and the owner together identify the group. 6
Note: Each device only has a single driver, thus for the purposes of this section, "the driver" is usually unambiguous and refers to the driver of the owner device. When there’s ambiguity, "owner driver" refers to the driver of the owner device, while "member driver" refers to the driver of a member device.

The following group types, and their identifiers, are currently specified:

SR-IOV group type (0x1)
This device group has a PCI Single Root I/O Virtualization (SR-IOV) physical function (PF) device as the owner and includes all its SR-IOV virtual functions (VFs) as members (see [PCIe]).

The PF device itself is not a member of the group.

The group type identifier for this group is 0x1.

A member identifier for this group can have a value from 0x1 to NumVFs as specified in the SR-IOV Extended Capability of the owner device and equals the SR-IOV VF number of the member device; the group only exists when the VF Enable bit in the SR-IOV Control Register within the SR-IOV Extended Capability of the owner device is set (see [PCIe]).

Both owner and member devices for this group type use the Virtio PCI transport (see 4.1).

2.12.1 Group administration commands

The driver sends group administration commands to the owner device of a group to control member devices of the group. This mechanism can be used, for example, to configure a member device before it is initialized by its driver. 7

All the group administration commands are of the following form:

struct virtio_admin_cmd { 
        /* Device-readable part */ 
        le16 opcode; 
        /* 
         * 1       - SR-IOV 
         * 2-65535 - reserved 
         */ 
        le16 group_type; 
        /* unused, reserved for future extensions */ 
        u8 reserved1[12]; 
        le64 group_member_id; 
        le64 command_specific_data[]; 
 
        /* Device-writable part */ 
        le16 status; 
        le16 status_qualifier; 
        /* unused, reserved for future extensions */ 
        u8 reserved2[4]; 
        u8 command_specific_result[]; 
};

For all commands, opcode, group_type and if necessary group_member_id and command_specific_data are set by the driver, and the owner device sets status and if needed status_qualifier and command_specific_result.

Generally, any unused device-readable fields are set to zero by the driver and ignored by the device. Any unused device-writeable fields are set to zero by the device and ignored by the driver.

opcode specifies the command. The valid values for opcode can be found in the following table:




opcode Name Command Description






0x0000 VIRTIO_ADMIN_CMD_LIST_QUERY Provides to driver list of commands supported for this group type



0x0001 VIRTIO_ADMIN_CMD_LIST_USE Provides to device list of commands used for this group type



0x0002 VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE Writes into the legacy common configuration structure



0x0003 VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ Reads from the legacy common configuration structure



0x0004 VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE Writes into the legacy device configuration structure



0x0005 VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ Reads into the legacy device configuration structure



0x0006 VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO Query the notification region information



0x0007 - 0x7FFF - Commands using struct virtio_admin_cmd



0x8000 - 0xFFFF - Reserved for future commands (possibly using a different structure)



The group_type specifies the group type identifier. The group_member_id specifies the member identifier within the group. See section 2.12 for the definition of the group type identifier and group member identifier.

The status describes the command result and possibly failure reason at an abstract level, this is appropriate for forwarding to applications. The status_qualifier describes failures at a low virtio specific level, as appropriate for debugging. The following table describes possible status values; to simplify common implementations, they are intentionally matching common Linux error names and numbers:




Status (decimal) Name Description






00 VIRTIO_ADMIN_STATUS_OK successful completion



11 VIRTIO_ADMIN_STATUS_EAGAIN try again



12 VIRTIO_ADMIN_STATUS_ENOMEM insufficient resources



22 VIRTIO_ADMIN_STATUS_EINVAL invalid command



other - group administration command error



When status is VIRTIO_ADMIN_STATUS_OK, status_qualifier is reserved and set to zero by the device.

The following table describes possible status_qualifier values:




Status Name Description






0x00 VIRTIO_ADMIN_STATUS_Q_OK used with VIRTIO_ADMIN_STATUS_OK



0x01 VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND command error: no additional information



0x02 VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE unsupported or invalid opcode



0x03 VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD unsupported or invalid field within command_specific_data



0x04 VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP unsupported or invalid group_type



0x05 VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER unsupported or invalid group_member_id



0x06 VIRTIO_ADMIN_STATUS_Q_NORESOURCE out of internal resources: ok to retry



0x07 VIRTIO_ADMIN_STATUS_Q_TRYAGAIN command blocks for too long: should retry



0x08-0xFFFF - reserved for future use



Each command uses a different command_specific_data and command_specific_result structures and the length of command_specific_data and command_specific_result depends on these structures and is described separately or is implicit in the structure description.

Before sending any group administration commands to the device, the driver needs to communicate to the device which commands it is going to use. Initially (after reset), only two commands are assumed to be used: VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.

Before sending any other commands for any member of a specific group to the device, the driver queries the supported commands via VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it is capable of using via VIRTIO_ADMIN_CMD_LIST_USE.

Commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE both use the following structure describing the command opcodes:

struct virtio_admin_cmd_list { 
       /* Indicates which of the below fields were returned 
       le64 device_admin_cmd_opcodes[]; 
};

This structure is an array of 64 bit values in little-endian byte order, in which a bit is set if the specific command opcode is supported. Thus, device_admin_cmd_opcodes[0] refers to the first 64-bit value in this array corresponding to opcodes 0 to 63, device_admin_cmd_opcodes[1] is the second 64-bit value corresponding to opcodes 64 to 127, etc. For example, the array of size 2 including the values 0x3 in device_admin_cmd_opcodes[0] and 0x1 in device_admin_cmd_opcodes[1] indicates that only opcodes 0, 1 and 64 are supported. The length of the array depends on the supported opcodes - it is large enough to include bits set for all supported opcodes, that is the length can be calculated by starting with the largest supported opcode adding one, dividing by 64 and rounding up. In other words, for VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the length of command_specific_result and command_specific_data respectively will be DIV ROUNDUP(maxcmd,64) 8 where DIV_ROUND_UP is integer division with round up and max_cmd is the largest available command opcode.

The array is also allowed to be larger and to additionally include an arbitrary number of all-zero entries.

Accordingly, bits 0 and 1 corresponding to opcode 0 (VIRTIO_ADMIN_CMD_LIST_QUERY) and 1 (VIRTIO_ADMIN_CMD_LIST_USE) are always set in device_admin_cmd_opcodes[0] returned by VIRTIO_ADMIN_CMD_LIST_QUERY.

For the command VIRTIO_ADMIN_CMD_LIST_QUERY, opcode is set to 0x0. The group_member_id is unused. It is set to zero by driver. This command has no command specific data. The device, upon success, returns a result in command_specific_result in the format struct virtio_admin_cmd_list describing the list of group administration commands supported for the group type specified by group_type.

For the command VIRTIO_ADMIN_CMD_LIST_USE, opcode is set to 0x1. The group_member_id is unused. It is set to zero by driver. The command_specific_data is in the format struct virtio_admin_cmd_list describing the list of group administration commands used by the driver with the group type specified by group_type.

This command has no command specific result.

The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to query the list of commands valid for this group and before sending any commands for any member of a group.

The driver then enables use of some of the opcodes by sending to the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is both understood and used by the driver.

If the device supports the command list used by the driver, the device completes the command with status VIRTIO_ADMIN_STATUS_OK. If the device does not support the command list (for example, if the driver is not capable to use some required commands), the device completes the command with status VIRTIO_ADMIN_STATUS_INVALID_FIELD.

Note: the driver is assumed not to set bits in device_admin_cmd_opcodes if it is not familiar with how the command opcode is used, since the device could have dependencies between command opcodes.

It is assumed that all members in a group support and are used with the same list of commands. However, for owner devices supporting multiple group types, the list of supported commands might differ between different group types.

2.12.1.1 Legacy Interfaces

In some systems, there is a need to support utilizing a legacy driver with a device that does not directly support the legacy interface. In such scenarios, a group owner device can provide the legacy interface functionality for the group member devices. The driver of the owner device can then access the legacy interface of a member device on behalf of the legacy member device driver.

For example, with the SR-IOV group type, group members (VFs) can not present the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver. If the legacy driver is running inside a virtual machine, the hypervisor executing the virtual machine can present a virtual device with an I/O BAR in BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and forwards them to the group owner device (PF) using group administration commands.

The following commands support such a legacy interface functionality:

1.
Legacy Common Configuration Write Command
2.
Legacy Common Configuration Read Command
3.
Legacy Device Configuration Write Command
4.
Legacy Device Configuration Read Command

These commands are currently only defined for the SR-IOV group type and have, generally, the same effect as member device accesses through a legacy interface listed in section 4.1.4.10 except that little-endian format is assumed unconditionally.

2.12.1.1.1 Legacy Common Configuration Write Command
This command has the same effect as writing into the virtio common configuration structure through the legacy interface. The command_specific_data is in the format struct virtio_admin_cmd_legacy_common_cfg_wr_data describing the access to be performed.
struct virtio_admin_cmd_legacy_common_cfg_wr_data { 
        u8 offset; /* Starting byte offset within the common configuration structure to write */ 
        u8 reserved[7]; 
        u8 data[]; 
};

For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, opcode is set to 0x2. The group_member_id refers to the member device to be accessed. The offset refers to the offset for the write within the virtio common configuration structure, and excluding the device-specific configuration. The length of the data to write is simply the length of data.

No length or alignment restrictions are placed on the value of the offset and the length of the data, except that the resulting access refers to a single field and is completely within the virtio common configuration structure, excluding the device-specific configuration.

This command has no command specific result.

2.12.1.1.2 Legacy Common Configuration Read Command
This command has the same effect as reading from the virtio common configuration structure through the legacy interface. The command_specific_data is in the format struct virtio_admin_cmd_legacy_common_cfg_rd_data describing the access to be performed.
struct virtio_admin_cmd_legacy_common_cfg_rd_data { 
        u8 offset; /* Starting byte offset within the common configuration structure to read */ 
};

For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, opcode is set to 0x3. The group_member_id refers to the member device to be accessed. The offset refers to the offset for the read from the virtio common configuration structure, and excluding the device-specific configuration.

struct virtio_admin_cmd_legacy_common_cfg_rd_result { 
        u8 data[]; 
};

No length or alignment restrictions are placed on the value of the offset and the length of the data, except that the resulting access refers to a single field and is completely within the virtio common configuration structure, excluding the device-specific configuration.

When the command completes successfully, command_specific_result is in the format struct virtio_admin_cmd_legacy_common_cfg_rd_result returned by the device. The length of the data read is simply the length of data.

2.12.1.1.3 Legacy Device Configuration Write Command
This command has the same effect as writing into the virtio device-specific configuration through the legacy interface. The command_specific_data is in the format struct virtio_admin_cmd_legacy_dev_reg_wr_data describing the access to be performed.
struct virtio_admin_cmd_legacy_dev_reg_wr_data { 
        u8 offset; /* Starting byte offset within the device-specific configuration to write */ 
        u8 reserved[7]; 
        u8 data[]; 
};

For the command VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, opcode is set to 0x4. The group_member_id refers to the member device to be accessed. The offset refers to the offset for the write within the virtio device-specific configuration. The length of the data to write is simply the length of data.

No length or alignment restrictions are placed on the value of the offset and the length of the data, except that the resulting access refers to a single field and is completely within the device-specific configuration.

This command has no command specific result.

2.12.1.1.4 Legacy Device Configuration Read Command
This command has the same effect as reading from the virtio device-specific configuration through the legacy interface. The command_specific_data is in the format struct virtio_admin_cmd_legacy_common_cfg_rd_data describing the access to be performed.
struct virtio_admin_cmd_legacy_dev_cfg_rd_data { 
        u8 offset; /* Starting byte offset within the device-specific configuration to read */ 
};

For the command VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ, opcode is set to 0x5. The group_member_id refers to the member device to be accessed. The offset refers to the offset for the read from the virtio device-specific configuration.

struct virtio_admin_cmd_legacy_dev_reg_rd_result { 
        u8 data[]; 
};

No length or alignment restrictions are placed on the value of the offset and the length of the data, except that the resulting access refers to a single field and is completely within the device-specific configuration.

When the command completes successfully, command_specific_result is in the format struct virtio_admin_cmd_legacy_dev_reg_rd_result returned by the device.

The length of the data read is simply the length of data.

2.12.1.1.5 Legacy Driver Notification
The driver of the owner device can send a driver notification to the member device operated using the legacy interface by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE with the offset matching Queue Notify and the data containing a 16-bit virtqueue index to be notified.

However, as VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE is also used for slow path configuration a separate dedicated mechanism for sending such driver notifications to the member device can be made available by the owner device. For the SR-IOV group type, the optional command VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO addresses this need by returning to the driver one or more addresses which can be used to send such driver notifications. The notification address returned can be in the device memory (PCI BAR or VF BAR) of the device.

In this alternative approach, driver notifications are sent by writing a 16-bit virtqueue index to be notified, in the little-endian format, to the notification address returned by the VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO command.

Any driver notification sent through the notification address has the same effect as if it was sent using the VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE command with the offset matching Queue Notify.

This command is only defined for the SR-IOV group type.

For the command VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO, opcode is set to 0x6. The group_member_id refers to the member device to be accessed. This command does not use command_specific_data.

When the device supports the VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO command, the group owner device hardwires VF BAR0 to zero in the SR-IOV Extended capability.

struct virtio_pci_legacy_notify_info { 
        u8 flags;  /* 0 = end of list, 1 = owner device, 2 = member device */ 
        u8 bar;    /* BAR of the member or the owner device */ 
        u8 padding[6]; 
        le64 offset; /* Offset within bar. */ 
}; 
 
struct virtio_admin_cmd_legacy_notify_info_result { 
        struct virtio_pci_legacy_notify_info entries[4]; 
};

A flags value of 0x1 indicates that the notification address is of the owner device, the value of 0x2 indicates that the notification address is of the member device and the value of 0x0 indicates that all the entries starting from that entry are invalid entries in entries. All other values in flags are reserved.

The bar values 0x1 to 0x5 specify BAR1 to BAR5 respectively: when the flags is 0x1 this is specified by the Base Address Registers in the PCI header of the device, when the flags is 0x2 this is specified by the VF BARn registers in the SR-IOV Extended Capability of the device.

The offset indicates the notification address relative to BAR indicated in bar. This value is 2-byte aligned.

When the command completes successfully, command_specific_result is in the format struct virtio_admin_cmd_legacy_notify_info_result. The device can supply up to 4 entries each with a different notification address. In this case, any of the entries can be used by the driver. The order of the entries serves as a preference hint to the driver. The driver is expected to utilize the entries placed earlier in the array in preference to the later ones. The driver is also expected to ignore any invalid entries, as well as the end of list entry if present and any entries following the end of list.

2.12.1.1.6 Device Requirements: Legacy Interface
A device MUST either support all of, or none of VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands.

For VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands, the device MUST decode and encode (respectively) the value of the data using the little-endian format.

For the VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands, the device MUST fail the command when the value of the offset and the length of the data do not refer to a single field or are not completely within the virtio common configuration excluding the device-specific configuration.

For the VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands, the device MUST fail the command when the value of the offset and the length of the data do not refer to a single field or are not completely within the virtio device-specific configuration.

The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST have the same effect as writing into the virtio common configuration structure through the legacy interface.

The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ MUST have the same effect as reading from the virtio common configuration structure through the legacy interface.

The command VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE MUST have the same effect as writing into the virtio device-specific configuration through the legacy interface.

The command VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ MUST have the same effect as reading from the virtio device-specific configuration through the legacy interface.

For the SR-IOV group type, when the owner device supports VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO commands,

If the device supports the VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO command,

2.12.1.1.7 Driver Requirements: Legacy Interface
For VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands, the driver MUST encode and decode (respectively) the value of the data using the little-endian format.

For the VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands, the driver SHOULD set offset and the length of the data to refer to a single field within the virtio common configuration structure excluding the device-specific configuration.

For the VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands, the driver SHOULD set offset and the length of the data to refer to a single field within device specific configuration.

If VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO command is supported, the driver SHOULD use the notification address to send all driver notifications to the device.

If within struct virtio_admin_cmd_legacy_notify_info_result returned by VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO, the flags value for a specific struct virtio_pci_legacy_notify_info entry is 0x0, the driver MUST ignore this entry and all the following entries. Additionally, for all other entries, the driver MUST validate that

, any entry which does not meet these constraints MUST be ignored by the driver.

2.12.1.2 Device Requirements: Group administration commands

The device MUST validate opcode, group_type and group_member_id, and if any of these has an invalid or unsupported value, set status to VIRTIO_ADMIN_STATUS_EINVAL and set status_qualifier accordingly:

If a command completes successfully, the device MUST set status to VIRTIO_ADMIN_STATUS_OK.

If a command fails, the device MUST set status to a value different from VIRTIO_ADMIN_STATUS_OK.

If status is set to VIRTIO_ADMIN_STATUS_EINVAL, the device state MUST NOT change, that is the command MUST NOT have any side effects on the device, in particular the device MUST NOT enter an error state as a result of this command.

If a command fails, the device state generally SHOULD NOT change, as far as possible.

The device MAY enforce additional restrictions and dependencies on opcodes used by the driver and MAY fail the command VIRTIO_ADMIN_CMD_LIST_USE with status set to VIRTIO_ADMIN_STATUS_EINVAL and status_qualifier set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD if the list of commands used violate internal device dependencies.

If the device supports multiple group types, commands for each group type MUST operate independently of each other, in particular, the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY for different group types.

After reset, if the device supports a given group type and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type the device MUST assume that the list of legal commands used by the driver consists of the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.

After completing VIRTIO_ADMIN_CMD_LIST_USE successfully, the device MUST set the list of legal commands used by the driver to the one supplied in command_specific_data.

The device MUST validate commands against the list used by the driver and MUST fail any commands not in the list with status set to VIRTIO_ADMIN_STATUS_EINVAL and status_qualifier set to VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.

The list of supported commands reported by the device MUST NOT shrink (but MAY expand): after reporting a given command as supported through VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it as unsupported. Further, after a given set of commands has been used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a device or system reset the device SHOULD complete successfully any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails after a device or system reset, the device MUST not fail it solely because of the command list used. Failure to do so would interfere with resuming from suspend and error recovery. Exceptions MAY apply if the system configuration assures, in some way, that the driver does not cache the previous value of VIRTIO_ADMIN_CMD_LIST_USE, such as in the case of a firmware upgrade or downgrade.

When processing a command with the SR-IOV group type, if the device does not have an SR-IOV Extended Capability or if VF Enable is clear then the device MUST fail all commands with status set to VIRTIO_ADMIN_STATUS_EINVAL and status_qualifier set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; otherwise, if group_member_id is not between 1 and NumVFs inclusive, the device MUST fail all commands with status set to VIRTIO_ADMIN_STATUS_EINVAL and status_qualifier set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER; NumVFs, VF Migration Capable and VF Enable refer to registers within the SR-IOV Extended Capability as specified by [PCIe].

2.12.1.3 Driver Requirements: Group administration commands

The driver MAY discover whether device supports a specific group type by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching group_type.

The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE and wait for it to be completed with status VIRTIO_ADMIN_STATUS_OK before issuing any commands (except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE).

The driver MAY issue VIRTIO_ADMIN_CMD_LIST_USE any number of times but MUST NOT issue VIRTIO_ADMIN_CMD_LIST_USE commands if any other command has been submitted to the device and has not yet completed processing by the device.

The driver SHOULD NOT set bits in device_admin_cmd_opcodes if it is not familiar with how the command opcode is used, as dependencies between command opcodes might exist.

The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE) the use of any commands not previously reported as supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.

The driver MUST NOT use any commands for a given group type before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct list of command opcodes and group type.

The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE with respective bits cleared in command_specific_data.

The driver MUST handle a command error with a reserved status value in the same way as status set to VIRTIO_ADMIN_STATUS_EINVAL (except possibly for different error reporting/diagnostic messages).

The driver MUST handle a command error with a reserved status_qualifier value in the same way as status_qualifier set to VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for different error reporting/diagnostic messages).

When sending commands with the SR-IOV group type, the driver specify a value for group_member_id between 1 and NumVFs inclusive, the driver MUST also make sure that as long as any such command is outstanding, VF Migration Capable is clear and VF Enable is set; NumVFs, VF Migration Capable and VF Enable refer to registers within the SR-IOV Extended Capability as specified by [PCIe].

2.13 Administration Virtqueues

An administration virtqueue of an owner device is used to submit group administration commands. An owner device can have more than one administration virtqueue.

If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one or more adminstration virtqueues. The number and locations of the administration virtqueues are exposed by the owner device in a transport specific manner.

The driver enqueues requests to an arbitrary administration virtqueue, and they are used by the device on that same virtqueue. It is the responsibility of the driver to ensure strict request ordering for commands, because they will be consumed with no order constraints. For example, if consistency is required then the driver can wait for the processing of a first command by the device to be completed before submitting another command depending on the first one.

Administration virtqueues are used as follows:

For each command, this specification describes a distinct format structure used for command_specific_data and command_specific_result, the length of these fields depends on the command.

However, to ensure forward compatibility

The device compares the length of each part (device-readable and device-writeable) of the buffer as submitted by driver to what it expects and then silently truncates the structures to either the length submitted by the driver, or the length described in this specification, whichever is shorter. The device silently ignores any data falling outside the shorter of the two lengths. Any missing fields are interpreted as set to zero.

Similarly, the driver compares the used buffer length of the buffer to what it expects and then silently truncates the structure to the used buffer length. The driver silently ignores any data falling outside the used buffer length reported by the device. Any missing fields are interpreted as set to zero.

This simplifies driver and device implementations since the driver/device can simply maintain a single large structure (such as a C structure) for a command and its result. As new versions of the specification are designed, new fields can be added to the tail of a structure, with the driver/device using the full structure without concern for versioning.

2.13.1 Device Requirements: Group administration commands

The device MUST support device-readable and device-writeable buffers shorter than described in this specification, by

1.
acting as if any data that would be read outside the device-readable buffers is set to zero, and
2.
discarding data that would be written outside the specified device-writeable buffers.

The device MUST support device-readable and device-writeable buffers longer than described in this specification, by

1.
ignoring any data in device-readable buffers outside the expected length, and
2.
only writing the expected structure to the device-writeable buffers, ignoring any extra buffers, and reporting the actual length of data written, in bytes, as buffer used length.

The device SHOULD initialize the device-writeable buffer up to the length of the structure described by this specification or the length of the buffer supplied by the driver (even if the buffer is all set to zero), whichever is shorter.

The device MUST NOT fail a command solely because the buffers provided are shorter or longer than described in this specification.

The device MUST initialize the device-writeable part of struct virtio_admin_cmd that is a multiple of 64 bit in size.

The device MUST initialize status and status_qualifier in struct virtio_admin_cmd.

The device MUST process commands on a given administration virtqueue in the order in which they are queued.

If multiple administration virtqueues have been configured, device MAY process commands on distinct virtqueues with no order constraints.

If the device sets status to either VIRTIO_ADMIN_STATUS_EAGAIN or VIRTIO_ADMIN_STATUS_ENOMEM, then the command MUST NOT have any side effects, making it safe to retry.

2.13.2 Driver Requirements: Group administration commands

The driver MAY supply device-readable or device-writeable parts of struct virtio_admin_cmd that are longer than described in this specification.

The driver SHOULD supply device-readable part of struct virtio_admin_cmd that is at least as large as the structure described by this specification (even if the structure is all set to zero).

The driver MUST supply both device-readable or device-writeable parts of struct virtio_admin_cmd that are a multiple of 64 bit in length.

The device MUST supply both device-readable or device-writeable parts of struct virtio_admin_cmd that are larger than zero in length. However, command_specific_data and command_specific_result MAY be zero in length, unless specified otherwise for the command.

The driver MUST NOT assume that the device will initialize the whole device-writeable part of struct virtio_admin_cmd as described in the specification; instead, the driver MUST act as if the structure outside the part of the buffer used by the device is set to zero.

If multiple administration virtqueues have been configured, the driver MUST ensure ordering for commands placed on different administration virtqueues.

The driver SHOULD retry a command that completed with status VIRTIO_ADMIN_STATUS_EAGAIN.


3 General Initialization And Device Operation

We start with an overview of device initialization, then expand on the details of the device and how each step is preformed. This section is best read along with the bus-specific section which describes how to communicate with the specific device.

3.1 Device Initialization

3.1.1 Driver Requirements: Device Initialization

The driver MUST follow this sequence to initialize a device:

1.
Reset the device.
2.
Set the ACKNOWLEDGE status bit: the guest OS has noticed the device.
3.
Set the DRIVER status bit: the guest OS knows how to drive the device.
4.
Read device feature bits, and write the subset of feature bits understood by the OS and driver to the device. During this step the driver MAY read (but MUST NOT write) the device-specific configuration fields to check that it can support the device before accepting it.
5.
Set the FEATURES_OK status bit. The driver MUST NOT accept new feature bits after this step.
6.
Re-read device status to ensure the FEATURES_OK bit is still set: otherwise, the device does not support our subset of features and the device is unusable.
7.
Perform device-specific setup, including discovery of virtqueues for the device, optional per-bus setup, reading and possibly writing the device’s virtio configuration space, and population of virtqueues.
8.
Set the DRIVER_OK status bit. At this point the device is “live”.

If any of these steps go irrecoverably wrong, the driver SHOULD set the FAILED status bit to indicate that it has given up on the device (it can reset the device later to restart if desired). The driver MUST NOT continue initialization in that case.

The driver MUST NOT send any buffer available notifications to the device before setting DRIVER_OK.

3.1.2 Legacy Interface: Device Initialization

Legacy devices did not support the FEATURES_OK status bit, and thus did not have a graceful way for the device to indicate unsupported feature combinations. They also did not provide a clear mechanism to end feature negotiation, which meant that devices finalized features on first-use, and no features could be introduced which radically changed the initial operation of the device.

Legacy driver implementations often used the device before setting the DRIVER_OK bit, and sometimes even before writing the feature bits to the device.

The result was the steps 5 and 6 were omitted, and steps 4, 7 and 8 were conflated.

Therefore, when using the legacy interface:

3.2 Device Operation

When operating the device, each field in the device configuration space can be changed by either the driver or the device.

Whenever such a configuration change is triggered by the device, driver is notified. This makes it possible for drivers to cache device configuration, avoiding expensive configuration reads unless notified.

3.2.1 Notification of Device Configuration Changes

For devices where the device-specific configuration information can be changed, a configuration change notification is sent when a device-specific configuration change occurs.

In addition, this notification is triggered by the device setting DEVICE_NEEDS_RESET (see 2.1.2).

3.3 Device Cleanup

Once the driver has set the DRIVER_OK status bit, all the configured virtqueue of the device are considered live. None of the virtqueues of a device are live once the device has been reset.

3.3.1 Driver Requirements: Device Cleanup

A driver MUST NOT alter virtqueue entries for exposed buffers, i.e., buffers which have been made available to the device (and not been used by the device) of a live virtqueue.

Thus a driver MUST ensure a virtqueue isn’t live (by device reset) before removing exposed buffers.


4 Virtio Transport Options

Virtio can use various different buses, thus the standard is split into virtio general and bus-specific sections.

4.1 Virtio Over PCI Bus

Virtio devices are commonly implemented as PCI devices.

A Virtio device can be implemented as any kind of PCI device: a Conventional PCI device or a PCI Express device. To assure designs meet the latest level requirements, see the PCI-SIG home page at http://www.pcisig.com for any approved changes.

4.1.1 Device Requirements: Virtio Over PCI Bus

A Virtio device using Virtio Over PCI Bus MUST expose to guest an interface that meets the specification requirements of the appropriate PCI specification: [PCI] and [PCIe] respectively.

4.1.2 PCI Device Discovery

Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device ID 0x1000 through 0x107F inclusive is a virtio device. The actual value within this range indicates which virtio device is supported by the device. The PCI Device ID is calculated by adding 0x1040 to the Virtio Device ID, as indicated in section 5. Additionally, devices MAY utilize a Transitional PCI Device ID range, 0x1000 to 0x103F depending on the device type.

4.1.2.1 Device Requirements: PCI Device Discovery

Devices MUST have the PCI Vendor ID 0x1AF4. Devices MUST either have the PCI Device ID calculated by adding 0x1040 to the Virtio Device ID, as indicated in section 5 or have the Transitional PCI Device ID depending on the device type, as follows:



Transitional PCI Device ID Virtio Device




0x1000 network device


0x1001 block device


0x1002 memory ballooning (traditional)


0x1003 console


0x1004 SCSI host


0x1005 entropy source


0x1009 9P transport


For example, the network device with the Virtio Device ID 1 has the PCI Device ID 0x1041 or the Transitional PCI Device ID 0x1000.

The PCI Subsystem Vendor ID and the PCI Subsystem Device ID MAY reflect the PCI Vendor and Device ID of the environment (for informational purposes by the driver).

Non-transitional devices SHOULD have a PCI Device ID in the range 0x1040 to 0x107f. Non-transitional devices SHOULD have a PCI Revision ID of 1 or higher. Non-transitional devices SHOULD have a PCI Subsystem Device ID of 0x40 or higher.

This is to reduce the chance of a legacy driver attempting to drive the device.

4.1.2.2 Driver Requirements: PCI Device Discovery

Drivers MUST match devices with the PCI Vendor ID 0x1AF4 and the PCI Device ID in the range 0x1040 to 0x107f, calculated by adding 0x1040 to the Virtio Device ID, as indicated in section 5. Drivers for device types listed in section 4.1.2 MUST match devices with the PCI Vendor ID 0x1AF4 and the Transitional PCI Device ID indicated in section 4.1.2.

Drivers MUST match any PCI Revision ID value. Drivers MAY match any PCI Subsystem Vendor ID and any PCI Subsystem Device ID value.

4.1.2.3 Legacy Interfaces: A Note on PCI Device Discovery

Transitional devices MUST have a PCI Revision ID of 0. Transitional devices MUST have the PCI Subsystem Device ID matching the Virtio Device ID, as indicated in section 5. Transitional devices MUST have the Transitional PCI Device ID in the range 0x1000 to 0x103f.

This is to match legacy drivers.

4.1.3 PCI Device Layout

The device is configured via I/O and/or memory regions (though see 4.1.4.9 for access via the PCI configuration space), as specified by Virtio Structure PCI Capabilities.

Fields of different sizes are present in the device configuration regions. All 64-bit, 32-bit and 16-bit fields are little-endian. 64-bit fields are to be treated as two 32-bit fields, with low 32 bit part followed by the high 32 bit part.

4.1.3.1 Driver Requirements: PCI Device Layout

For device configuration access, the driver MUST use 8-bit wide accesses for 8-bit wide fields, 16-bit wide and aligned accesses for 16-bit wide fields and 32-bit wide and aligned accesses for 32-bit and 64-bit wide fields. For 64-bit fields, the driver MAY access each of the high and low 32-bit parts of the field independently.

4.1.3.2 Device Requirements: PCI Device Layout

For 64-bit device configuration fields, the device MUST allow driver independent access to high and low 32-bit parts of the field.

4.1.4 Virtio Structure PCI Capabilities

The virtio device configuration layout includes several structures:

Each structure can be mapped by a Base Address register (BAR) belonging to the function, or accessed via the special VIRTIO_PCI_CAP_PCI_CFG field in the PCI configuration space.

The location of each structure is specified using a vendor-specific PCI capability located on the capability list in PCI configuration space of the device. This virtio structure capability uses little-endian format; all fields are read-only for the driver unless stated otherwise:

struct virtio_pci_cap { 
        u8 cap_vndr;    /* Generic PCI field: PCI_CAP_ID_VNDR */ 
        u8 cap_next;    /* Generic PCI field: next ptr. */ 
        u8 cap_len;     /* Generic PCI field: capability length */ 
        u8 cfg_type;    /* Identifies the structure. */ 
        u8 bar;         /* Where to find it. */ 
        u8 id;          /* Multiple capabilities of the same type */ 
        u8 padding[2];  /* Pad to full dword. */ 
        le32 offset;    /* Offset within bar. */ 
        le32 length;    /* Length of the structure, in bytes. */ 
};

This structure can be followed by extra data, depending on cfg_type, as documented below.

The fields are interpreted as follows:

cap_vndr
0x09; Identifies a vendor-specific capability.
cap_next
Link to next capability in the capability list in the PCI configuration space.
cap_len
Length of this capability structure, including the whole of struct virtio_pci_cap, and extra data if any. This length MAY include padding, or fields unused by the driver.
cfg_type
identifies the structure, according to the following table:
/* Common configuration */ 
#define VIRTIO_PCI_CAP_COMMON_CFG        1 
/* Notifications */ 
#define VIRTIO_PCI_CAP_NOTIFY_CFG        2 
/* ISR Status */ 
#define VIRTIO_PCI_CAP_ISR_CFG           3 
/* Device specific configuration */ 
#define VIRTIO_PCI_CAP_DEVICE_CFG        4 
/* PCI configuration access */ 
#define VIRTIO_PCI_CAP_PCI_CFG           5 
/* Shared memory region */ 
#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8 
/* Vendor-specific data */ 
#define VIRTIO_PCI_CAP_VENDOR_CFG        9

Any other value is reserved for future use.

Each structure is detailed individually below.

The device MAY offer more than one structure of any type - this makes it possible for the device to expose multiple interfaces to drivers. The order of the capabilities in the capability list specifies the order of preference suggested by the device. A device may specify that this ordering mechanism be overridden by the use of the id field. Note: For example, on some hypervisors, notifications using IO accesses are faster than memory accesses. In this case, the device would expose two capabilities with cfg_type set to VIRTIO_PCI_CAP_NOTIFY_CFG: the first one addressing an I/O BAR, the second one addressing a memory BAR. In this example, the driver would use the I/O BAR if I/O resources are available, and fall back on memory BAR when I/O resources are unavailable.

bar
values 0x0 to 0x5 specify a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space and used to map the structure into Memory or I/O Space. The BAR is permitted to be either 32-bit or 64-bit, it can map Memory Space or I/O Space.

Any other value is reserved for future use.

id
Used by some device types to uniquely identify multiple capabilities of a certain type. If the device type does not specify the meaning of this field, its contents are undefined.
offset
indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of offset are indicated in each structure-specific section below.
length
indicates the length of the structure.

length MAY include padding, or fields unused by the driver, or future extensions. Note: For example, a future device might present a large structure size of several MBytes. As current devices never utilize structures larger than 4KBytes in size, driver MAY limit the mapped structure size to e.g. 4KBytes (thus ignoring parts of structure after the first 4KBytes) to allow forward compatibility with such devices without loss of functionality and without wasting resources.

A variant of this type, struct virtio_pci_cap64, is defined for those capabilities that require offsets or lengths larger than 4GiB:

struct virtio_pci_cap64 { 
        struct virtio_pci_cap cap; 
        u32 offset_hi; 
        u32 length_hi; 
};

Given that the cap.length and cap.offset fields are only 32 bit, the additional offset_hi and length_hi fields provide the most significant 32 bits of a total 64 bit offset and length within the BAR specified by cap.bar.

4.1.4.1 Driver Requirements: Virtio Structure PCI Capabilities

The driver MUST ignore any vendor-specific capability structure which has a reserved cfg_type value.

The driver SHOULD use the first instance of each virtio structure type they can support.

The driver MUST accept a cap_len value which is larger than specified here.

The driver MUST ignore any vendor-specific capability structure which has a reserved bar value.

The drivers SHOULD only map part of configuration structure large enough for device operation. The drivers MUST handle an unexpectedly large length, but MAY check that length is large enough for device operation.

The driver MUST NOT write into any field of the capability structure, with the exception of those with cap_type VIRTIO_PCI_CAP_PCI_CFG as detailed in 4.1.4.9.2.

4.1.4.2 Device Requirements: Virtio Structure PCI Capabilities

The device MUST include any extra data (from the beginning of the cap_vndr field through end of the extra data fields if any) in cap_len. The device MAY append extra data or padding to any structure beyond that.

If the device presents multiple structures of the same type, it SHOULD order them from optimal (first) to least-optimal (last).

4.1.4.3 Common configuration structure layout

The common configuration structure is found at the bar and offset within the VIRTIO_PCI_CAP_COMMON_CFG capability; its layout is below.

struct virtio_pci_common_cfg { 
        /* About the whole device. */ 
        le32 device_feature_select;     /* read-write */ 
        le32 device_feature;            /* read-only for driver */ 
        le32 driver_feature_select;     /* read-write */ 
        le32 driver_feature;            /* read-write */ 
        le16 config_msix_vector;        /* read-write */ 
        le16 num_queues;                /* read-only for driver */ 
        u8 device_status;               /* read-write */ 
        u8 config_generation;           /* read-only for driver */ 
 
        /* About a specific virtqueue. */ 
        le16 queue_select;              /* read-write */ 
        le16 queue_size;                /* read-write */ 
        le16 queue_msix_vector;         /* read-write */ 
        le16 queue_enable;              /* read-write */ 
        le16 queue_notify_off;          /* read-only for driver */ 
        le64 queue_desc;                /* read-write */ 
        le64 queue_driver;              /* read-write */ 
        le64 queue_device;              /* read-write */ 
        le16 queue_notif_config_data;   /* read-only for driver */ 
        le16 queue_reset;               /* read-write */ 
 
        /* About the administration virtqueue. */ 
        le16 admin_queue_index;         /* read-only for driver */ 
        le16 admin_queue_num;         /* read-only for driver */ 
};
device_feature_select
The driver uses this to select which feature bits device_feature shows. Value 0x0 selects Feature Bits 0 to 31, 0x1 selects Feature Bits 32 to 63, etc.
device_feature
The device uses this to report which feature bits it is offering to the driver: the driver writes to device_feature_select to select which feature bits are presented.
driver_feature_select
The driver uses this to select which feature bits driver_feature shows. Value 0x0 selects Feature Bits 0 to 31, 0x1 selects Feature Bits 32 to 63, etc.
driver_feature
The driver writes this to accept feature bits offered by the device. Driver Feature Bits selected by driver_feature_select.
config_msix_vector
Set by the driver to the MSI-X vector for configuration change notifications.
num_queues
The device specifies the maximum number of virtqueues supported here. This excludes administration virtqueues if any are supported.
device_status
The driver writes the device status here (see 2.1). Writing 0 into this field resets the device.
config_generation
Configuration atomicity value. The device changes this every time the configuration noticeably changes.
queue_select
Queue Select. The driver selects which virtqueue the following fields refer to.
queue_size
Queue Size. On reset, specifies the maximum queue size supported by the device. This can be modified by the driver to reduce memory requirements. A 0 means the queue is unavailable.
queue_msix_vector
Set by the driver to the MSI-X vector for virtqueue notifications.
queue_enable
The driver uses this to selectively prevent the device from executing requests from this virtqueue. 1 - enabled; 0 - disabled.
queue_notify_off
The driver reads this to calculate the offset from start of Notification structure at which this virtqueue is located. Note: this is not an offset in bytes. See 4.1.4.4 below.
queue_desc
The driver writes the physical address of Descriptor Area here. See section 2.6.
queue_driver
The driver writes the physical address of Driver Area here. See section 2.6.
queue_device
The driver writes the physical address of Device Area here. See section 2.6.
queue_notif_config_data
This field exists only if VIRTIO_F_NOTIF_CONFIG_DATA has been negotiated. The driver will use this value when driver sends available buffer notification to the device. See section 4.1.5.2. Note: This field provides the device with flexibility to determine how virtqueues will be referred to in available buffer notifications. In a trivial case the device can set queue_notif_config_data to the virtqueue index. Some devices may benefit from providing another value, for example an internal virtqueue identifier, or an internal offset related to the virtqueue index. Note: This field was previously known as queue_notify_data.
queue_reset
The driver uses this to selectively reset the queue. This field exists only if VIRTIO_F_RING_RESET has been negotiated. (see 2.6.1).
admin_queue_index
The device uses this to report the index of the first administration virtqueue. This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.
admin_queue_num
The device uses this to report the number of the supported administration virtqueues. Virtqueues with index between admin_queue_index and (admin_queue_index + admin_queue_num - 1) inclusive serve as administration virtqueues. The value 0 indicates no supported administration virtqueues. This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated.

4.1.4.3.1 Device Requirements: Common configuration structure layout
offset MUST be 4-byte aligned.

The device MUST present at least one common configuration capability.

The device MUST present the feature bits it is offering in device_feature, starting at bit device_feature_select 32 for any device_feature_select written by the driver. Note: This means that it will present 0 for any device_feature_select other than 0 or 1, since no feature defined here exceeds 63.

The device MUST present any valid feature bits the driver has written in driver_feature, starting at bit driver_feature_select 32 for any driver_feature_select written by the driver. Valid feature bits are those which are subset of the corresponding device_feature bits. The device MAY present invalid bits written by the driver. Note: This means that a device can ignore writes for feature bits it never offers, and simply present 0 on reads. Or it can just mirror what the driver wrote (but it will still have to check them when the driver sets FEATURES_OK). Note: A driver shouldn’t write invalid bits anyway, as per 3.1.1, but this attempts to handle it.

The device MUST present a changed config_generation after the driver has read a device-specific configuration value which has changed since any part of the device-specific configuration was last read. Note: As config_generation is an 8-bit value, simply incrementing it on every configuration change could violate this requirement due to wrap. Better would be to set an internal flag when it has changed, and if that flag is set when the driver reads from the device-specific configuration, increment config_generation and clear the flag.

The device MUST reset when 0 is written to device_status, and present a 0 in device_status once that is done.

The device MUST present a 0 in queue_enable on reset.

If VIRTIO_F_RING_RESET has been negotiated, the device MUST present a 0 in queue_reset on reset.

If VIRTIO_F_RING_RESET has been negotiated, the device MUST present a 0 in queue_reset after the virtqueue is enabled with queue_enable.

The device MUST reset the queue when 1 is written to queue_reset. The device MUST continue to present 1 in queue_reset as long as the queue reset is ongoing. The device MUST present 0 in both queue_reset and queue_enable when queue reset has completed. (see 2.6.1).

The device MUST present a 0 in queue_size if the virtqueue corresponding to the current queue_select is unavailable.

If VIRTIO_F_RING_PACKED has not been negotiated, the device MUST present either a value of 0 or a power of 2 in queue_size.

If VIRTIO_F_ADMIN_VQ has been negotiated, the value admin_queue_index MUST be equal to, or bigger than num_queues; also, admin_queue_num MUST be smaller than, or equal to 0x10000 - admin_queue_index, to ensure that indices of valid admin queues fit into a 16 bit range beyond all other virtqueues.

4.1.4.3.2 Driver Requirements: Common configuration structure layout
The driver MUST NOT write to device_feature, num_queues, config_generation, queue_notify_off or queue_notif_config_data.

If VIRTIO_F_RING_PACKED has been negotiated, the driver MUST NOT write the value 0 to queue_size. If VIRTIO_F_RING_PACKED has not been negotiated, the driver MUST NOT write a value which is not a power of 2 to queue_size.

The driver MUST configure the other virtqueue fields before enabling the virtqueue with queue_enable.

After writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device.

The driver MUST NOT write a 0 to queue_enable.

If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to queue_reset to reset the queue, the driver MUST NOT consider queue reset to be complete until it reads back 0 in queue_reset. The driver MAY re-enable the queue by writing 1 to queue_enable after ensuring that other virtqueue fields have been set up correctly. The driver MAY set driver-writeable queue configuration values to different values than those that were used before the queue reset. (see 2.6.1).

If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver configures any administration virtqueues, the driver MUST configure the administration virtqueues using the index in the range admin_queue_index to admin_queue_index + admin_queue_num - 1 inclusive. The driver MAY configure fewer administration virtqueues than supported by the device.

4.1.4.4 Notification structure layout

The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG capability. This capability is immediately followed by an additional field, like so:

struct virtio_pci_notify_cap { 
        struct virtio_pci_cap cap; 
        le32 notify_off_multiplier; /* Multiplier for queue_notify_off. */ 
};

notify_off_multiplier is combined with the queue_notify_off to derive the Queue Notify address within a BAR for a virtqueue:

        cap.offset + queue_notify_off * notify_off_multiplier

The cap.offset and notify_off_multiplier are taken from the notification capability structure above, and the queue_notify_off is taken from the common configuration structure. Note: For example, if notifier_off_multiplier is 0, the device uses the same Queue Notify address for all queues.

4.1.4.4.1 Device Requirements: Notification capability
The device MUST present at least one notification capability.

For devices not offering VIRTIO_F_NOTIFICATION_DATA:

The cap.offset MUST be 2-byte aligned.

The device MUST either present notify_off_multiplier as an even power of 2, or present notify_off_multiplier as 0.

The value cap.length presented by the device MUST be at least 2 and MUST be large enough to support queue notification offsets for all supported queues in all possible configurations.

For all queues, the value cap.length presented by the device MUST satisfy:

cap.length >= queue_notify_off * notify_off_multiplier + 2

For devices offering VIRTIO_F_NOTIFICATION_DATA:

The device MUST either present notify_off_multiplier as a number that is a power of 2 that is also a multiple 4, or present notify_off_multiplier as 0.

The cap.offset MUST be 4-byte aligned.

The value cap.length presented by the device MUST be at least 4 and MUST be large enough to support queue notification offsets for all supported queues in all possible configurations.

For all queues, the value cap.length presented by the device MUST satisfy:

cap.length >= queue_notify_off * notify_off_multiplier + 4
4.1.4.5 ISR status capability

The VIRTIO_PCI_CAP_ISR_CFG capability refers to at least a single byte, which contains the 8-bit ISR status field to be used for INT#x interrupt handling.

The offset for the ISR status has no alignment requirements.

The ISR bits allow the driver to distinguish between device-specific configuration change interrupts and normal virtqueue interrupts:





Bits 0 1 2 to 31




Purpose Queue Interrupt Device Configuration Interrupt Reserved




To avoid an extra access, simply reading this register resets it to 0 and causes the device to de-assert the interrupt.

In this way, driver read of ISR status causes the device to de-assert an interrupt.

See sections 4.1.5.3 and 4.1.5.4 for how this is used.

4.1.4.5.1 Device Requirements: ISR status capability
The device MUST present at least one VIRTIO_PCI_CAP_ISR_CFG capability.

The device MUST set the Device Configuration Interrupt bit in ISR status before sending a device configuration change notification to the driver.

If MSI-X capability is disabled, the device MUST set the Queue Interrupt bit in ISR status before sending a virtqueue notification to the driver.

If MSI-X capability is disabled, the device MUST set the Interrupt Status bit in the PCI Status register in the PCI Configuration Header of the device to the logical OR of all bits in ISR status of the device. The device then asserts/deasserts INT#x interrupts unless masked according to standard PCI rules [PCI].

The device MUST reset ISR status to 0 on driver read.

4.1.4.5.2 Driver Requirements: ISR status capability
If MSI-X capability is enabled, the driver SHOULD NOT access ISR status upon detecting a Queue Interrupt.
4.1.4.6 Device-specific configuration

The device MUST present at least one VIRTIO_PCI_CAP_DEVICE_CFG capability for any device type which has a device-specific configuration.

4.1.4.6.1 Device Requirements: Device-specific configuration
The offset for the device-specific configuration MUST be 4-byte aligned.
4.1.4.7 Shared memory capability

Shared memory regions 2.10 are enumerated on the PCI transport as a sequence of VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capabilities, one per region.

The capability is defined by a struct virtio_pci_cap64 and utilises the cap.id to allow multiple shared memory regions per device. The identifier in cap.id does not denote a certain order of preference; it is only used to uniquely identify a region.

4.1.4.7.1 Device Requirements: Shared memory capability
The region defined by the combination of the cap.offset, offset_hi, and cap.length, length_hi fields MUST be contained within the BAR specified by cap.bar.

The cap.id MUST be unique for any one device instance.

4.1.4.8 Vendor data capability

The optional Vendor data capability allows the device to present vendor-specific data to the driver, without conflicts, for debugging and/or reporting purposes, and without conflicting with standard functionality.

This capability augments but does not replace the standard subsystem ID and subsystem vendor ID fields (offsets 0x2C and 0x2E in the PCI configuration space header) as specified by [PCI].

Vendor data capability is enumerated on the PCI transport as a VIRTIO_PCI_CAP_VENDOR_CFG capability.

The capability has the following structure:

struct virtio_pci_vndr_data { 
        u8 cap_vndr;    /* Generic PCI field: PCI_CAP_ID_VNDR */ 
        u8 cap_next;    /* Generic PCI field: next ptr. */ 
        u8 cap_len;     /* Generic PCI field: capability length */ 
        u8 cfg_type;    /* Identifies the structure. */ 
        u16 vendor_id;  /* Identifies the vendor-specific format. */ 
  /* For Vendor Definition */ 
  /* Pads structure to a multiple of 4 bytes */ 
  /* Reads must not have side effects */ 
};

Where vendor_id identifies the PCI-SIG assigned Vendor ID as specified by [PCI].

Note that the capability size is required to be a multiple of 4.

To make it safe for a generic driver to access the capability, reads from this capability MUST NOT have any side effects.

4.1.4.8.1 Device Requirements: Vendor data capability
Devices CAN present vendor_id that does not match either the PCI Vendor ID or the PCI Subsystem Vendor ID.

Devices CAN present multiple Vendor data capabilities with either different or identical vendor_id values.

The value vendor_id MUST NOT equal 0x1AF4.

The size of the Vendor data capability MUST be a multiple of 4 bytes.

Reads of the Vendor data capability by the driver MUST NOT have any side effects.

4.1.4.8.2 Driver Requirements: Vendor data capability
The driver SHOULD NOT use the Vendor data capability except for debugging and reporting purposes.

The driver MUST qualify the vendor_id before interpreting or writing into the Vendor data capability.

4.1.4.9 PCI configuration access capability

The VIRTIO_PCI_CAP_PCI_CFG capability creates an alternative (and likely suboptimal) access method to the common configuration, notification, ISR and device-specific configuration regions.

The capability is immediately followed by an additional field like so:

struct virtio_pci_cfg_cap { 
        struct virtio_pci_cap cap; 
        u8 pci_cfg_data[4]; /* Data for BAR access. */ 
};

The fields cap.bar, cap.length, cap.offset and pci_cfg_data are read-write (RW) for the driver.

To access a device region, the driver writes into the capability structure (ie. within the PCI configuration space) as follows:

At that point, pci_cfg_data will provide a window of size cap.length into the given cap.bar at offset cap.offset.

4.1.4.9.1 Device Requirements: PCI configuration access capability
The device MUST present at least one VIRTIO_PCI_CAP_PCI_CFG capability.

Upon detecting driver write access to pci_cfg_data, the device MUST execute a write access at offset cap.offset at BAR selected by cap.bar using the first cap.length bytes from pci_cfg_data.

Upon detecting driver read access to pci_cfg_data, the device MUST execute a read access of length cap.length at offset cap.offset at BAR selected by cap.bar and store the first cap.length bytes in pci_cfg_data.

4.1.4.9.2 Driver Requirements: PCI configuration access capability
The driver MUST NOT write a cap.offset which is not a multiple of cap.length (ie. all accesses MUST be aligned).

The driver MUST NOT read or write pci_cfg_data unless cap.bar, cap.length and cap.offset address cap.length bytes within a BAR range specified by some other Virtio Structure PCI Capability of type other than VIRTIO_PCI_CAP_PCI_CFG.

4.1.4.10 Legacy Interfaces: A Note on PCI Device Layout

Transitional devices MUST present part of configuration registers in a legacy configuration structure in BAR0 in the first I/O region of the PCI device, as documented below. When using the legacy interface, transitional drivers MUST use the legacy configuration structure in BAR0 in the first I/O region of the PCI device, as documented below.

When using the legacy interface the driver MAY access the device-specific configuration region using any width accesses, and a transitional device MUST present driver with the same results as when accessed using the “natural” access method (i.e. 32-bit accesses for 32-bit fields, etc).

Note that this is possible because while the virtio common configuration structure is PCI (i.e. little) endian, when using the legacy interface the device-specific configuration region is encoded in the native endian of the guest (where such distinction is applicable).

When used through the legacy interface, the virtio common configuration structure looks as follows:










Bits 32 32 32 16 16 16 8 8









Read / Write R R+W R+W R R+W R+W R+W R









Purpose Device Features bits 0:31 Driver Features bits 0:31 Queue Address queue_sizequeue_selectQueue Notify Device Status ISR
Status









If MSI-X is enabled for the device, two additional fields immediately follow this header:




Bits 16 16



Read/Write R+W R+W



Purpose (MSI-X) config_msix_vector queue_msix_vector



Note: When MSI-X capability is enabled, device-specific configuration starts at byte offset 24 in virtio common configuration structure. When MSI-X capability is not enabled, device-specific configuration starts at byte offset 20 in virtio header. ie. once you enable MSI-X on the device, the other fields move. If you turn it off again, they move back!

Any device-specific configuration space immediately follows these general headers:




Bits Device Specific


Read / Write Device Specific


Purpose Device Specific



When accessing the device-specific configuration space using the legacy interface, transitional drivers MUST access the device-specific configuration space at an offset immediately following the general headers.

When using the legacy interface, transitional devices MUST present the device-specific configuration space if any at an offset immediately following the general headers.

Note that only Feature Bits 0 to 31 are accessible through the Legacy Interface. When used through the Legacy Interface, Transitional Devices MUST assume that Feature Bits 32 to 63 are not acknowledged by Driver.

As legacy devices had no config_generation field, see 2.5.4 Legacy Interface: Device Configuration Space for workarounds.

4.1.4.11 Non-transitional Device With Legacy Driver: A Note on PCI Device Layout

All known legacy drivers check either the PCI Revision or the Device and Vendor IDs, and thus won’t attempt to drive a non-transitional device.

A buggy legacy driver might mistakenly attempt to drive a non-transitional device. If support for such drivers is required (as opposed to fixing the bug), the following would be the recommended way to detect and handle them. Note: Such buggy drivers are not currently known to be used in production.

4.1.4.11.0.1 Device Requirements: Non-transitional Device With Legacy Driver
Non-transitional devices, on a platform where a legacy driver for a legacy device with the same ID (including PCI Revision, Device and Vendor IDs) is known to have previously existed, SHOULD take the following steps to cause the legacy driver to fail gracefully when it attempts to drive them:
1.
Present an I/O BAR in BAR0, and
2.
Respond to a single-byte zero write to offset 18 (corresponding to Device Status register in the legacy layout) of BAR0 by presenting zeroes on every BAR and ignoring writes.

4.1.5 PCI-specific Initialization And Device Operation

4.1.5.1 Device Initialization

This documents PCI-specific steps executed during Device Initialization.

4.1.5.1.1 Virtio Device Configuration Layout Detection
As a prerequisite to device initialization, the driver scans the PCI capability list, detecting virtio configuration layout using Virtio Structure PCI capabilities as detailed in 4.1.4

4.1.5.1.1.1 Legacy Interface: A Note on Device Layout Detection
Legacy drivers skipped the Device Layout Detection step, assuming legacy device configuration space in BAR0 in I/O space unconditionally.

Legacy devices did not have the Virtio PCI Capability in their capability list.

Therefore:

Transitional devices MUST expose the Legacy Interface in I/O space in BAR0.

Transitional drivers MUST look for the Virtio PCI Capabilities on the capability list. If these are not present, driver MUST assume a legacy device, and use it through the legacy interface.

Non-transitional drivers MUST look for the Virtio PCI Capabilities on the capability list. If these are not present, driver MUST assume a legacy device, and fail gracefully.

4.1.5.1.2 MSI-X Vector Configuration
When MSI-X capability is present and enabled in the device (through standard PCI configuration space) config_msix_vector and queue_msix_vector are used to map configuration change and queue interrupts to MSI-X vectors. In this case, the ISR Status is unused.

Writing a valid MSI-X Table entry number, 0 to 0x7FF, to config_msix_vector/queue_msix_vector maps interrupts triggered by the configuration change/selected queue events respectively to the corresponding MSI-X vector. To disable interrupts for an event type, the driver unmaps this event by writing a special NO_VECTOR value:

/* Vector value used to disable MSI for queue */ 
#define VIRTIO_MSI_NO_VECTOR            0xffff

Note that mapping an event to vector might require device to allocate internal device resources, and thus could fail.

4.1.5.1.2.1 Device Requirements: MSI-X Vector Configuration
A device that has an MSI-X capability SHOULD support at least 2 and at most 0x800 MSI-X vectors. Device MUST report the number of vectors supported in Table Size in the MSI-X Capability as specified in [PCI]. The device SHOULD restrict the reported MSI-X Table Size field to a value that might benefit system performance. Note: For example, a device which does not expect to send interrupts at a high rate might only specify 2 MSI-X vectors.

Device MUST support mapping any event type to any valid vector 0 to MSI-X Table Size. Device MUST support unmapping any event type.

The device MUST return vector mapped to a given event, (NO_VECTOR if unmapped) on read of config_msix_vector/queue_msix_vector. The device MUST have all queue and configuration change events are unmapped upon reset.

Devices SHOULD NOT cause mapping an event to vector to fail unless it is impossible for the device to satisfy the mapping request. Devices MUST report mapping failures by returning the NO_VECTOR value when the relevant config_msix_vector/queue_msix_vector field is read.

4.1.5.1.2.2 Driver Requirements: MSI-X Vector Configuration
Driver MUST support device with any MSI-X Table Size 0 to 0x7FF. Driver MAY fall back on using INT#x interrupts for a device which only supports one MSI-X vector (MSI-X Table Size = 0).

Driver MAY interpret the Table Size as a hint from the device for the suggested number of MSI-X vectors to use.

Driver MUST NOT attempt to map an event to a vector outside the MSI-X Table supported by the device, as reported by Table Size in the MSI-X Capability.

After mapping an event to vector, the driver MUST verify success by reading the Vector field value: on success, the previously written value is returned, and on failure, NO_VECTOR is returned. If a mapping failure is detected, the driver MAY retry mapping with fewer vectors, disable MSI-X or report device failure.

4.1.5.1.3 Virtqueue Configuration
As a device can have zero or more virtqueues for bulk data transport8, the driver needs to configure them as part of the device-specific configuration.

The driver typically does this as follows, for each virtqueue a device has:

1.
Write the virtqueue index to queue_select.
2.
Read the virtqueue size from queue_size. This controls how big the virtqueue is (see 2.6 Virtqueues). If this field is 0, the virtqueue does not exist.
3.
Optionally, select a smaller virtqueue size and write it to queue_size.
4.
Allocate and zero Descriptor Table, Available and Used rings for the virtqueue in contiguous physical memory.
5.
Optionally, if MSI-X capability is present and enabled on the device, select a vector to use to request interrupts triggered by virtqueue events. Write the MSI-X Table entry number corresponding to this vector into queue_msix_vector. Read queue_msix_vector: on success, previously written value is returned; on failure, NO_VECTOR value is returned.

4.1.5.1.3.1 Legacy Interface: A Note on Virtqueue Configuration
When using the legacy interface, the queue layout follows 2.7.2 Legacy Interfaces: A Note on Virtqueue Layout with an alignment of 4096. Driver writes the physical address, divided by 4096 to the Queue Address field9. There was no mechanism to negotiate the queue size.
4.1.5.2 Available Buffer Notifications

When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, the driver sends an available buffer notification to the device by writing only the 16-bit notification value to the Queue Notify address of the virtqueue. A notification value depends on the negotiation of VIRTIO_F_NOTIF_CONFIG_DATA.

If VIRTIO_F_NOTIFICATION_DATA has been negotiated, the driver sends an available buffer notification to the device by writing the following 32-bit value to the Queue Notify address:

 
le32 { 
  union { 
    vq_index: 16; /* Used if VIRTIO_F_NOTIF_CONFIG_DATA not negotiated */ 
    vq_notif_config_data: 16; /* Used if VIRTIO_F_NOTIF_CONFIG_DATA negotiated */ 
  }; 
  next_off : 15; 
  next_wrap : 1; 
};

See 2.9 Driver Notifications for the definition of the components.

See 4.1.4.4 for how to calculate the Queue Notify address.

4.1.5.2.1 Driver Requirements: Available Buffer Notifications
If VIRTIO_F_NOTIFICATION_DATA is not negotiated, the driver notification MUST be a 16-bit notification.

If VIRTIO_F_NOTIFICATION_DATA is negotiated, the driver notification MUST be a 32-bit notification.

If VIRTIO_F_NOTIF_CONFIG_DATA is not negotiated:

If VIRTIO_F_NOTIF_CONFIG_DATA is negotiated:

4.1.5.3 Used Buffer Notifications

If a used buffer notification is necessary for a virtqueue, the device would typically act as follows:

4.1.5.3.1 Device Requirements: Used Buffer Notifications
If MSI-X capability is enabled and queue_msix_vector is NO_VECTOR for a virtqueue, the device MUST NOT deliver an interrupt for that virtqueue.
4.1.5.4 Notification of Device Configuration Changes

Some virtio PCI devices can change the device configuration state, as reflected in the device-specific configuration region of the device. In this case:

A single interrupt MAY indicate both that one or more virtqueue has been used and that the configuration space has changed.

4.1.5.4.1 Device Requirements: Notification of Device Configuration Changes
If MSI-X capability is enabled and config_msix_vector is NO_VECTOR, the device MUST NOT deliver an interrupt for device configuration space changes.

4.1.5.4.2 Driver Requirements: Notification of Device Configuration Changes
A driver MUST handle the case where the same interrupt is used to indicate both device configuration space change and one or more virtqueues being used.
4.1.5.5 Driver Handling Interrupts

The driver interrupt handler would typically:

4.2 Virtio Over MMIO

Virtual environments without PCI support (a common situation in embedded devices models) might use simple memory mapped device (“virtio-mmio”) instead of the PCI device.

The memory mapped virtio device behaviour is based on the PCI device specification. Therefore most operations including device initialization, queues configuration and buffer transfers are nearly identical. Existing differences are described in the following sections.

4.2.1 MMIO Device Discovery

Unlike PCI, MMIO provides no generic device discovery mechanism. For each device, the guest OS will need to know the location of the registers and interrupt(s) used. The suggested binding for systems using flattened device trees is shown in this example:

// EXAMPLE: virtio_block device taking 512 bytes at 0x1e000, interrupt 42. 
virtio_block@1e000 { 
        compatible = "virtio,mmio"; 
        reg = <0x1e000 0x200>; 
        interrupts = <42>; 
}

4.2.2 MMIO Device Register Layout

MMIO virtio devices provide a set of memory mapped control registers followed by a device-specific configuration space, described in the table 4.1.

All register values are organized as Little Endian.

Table 4.1: MMIO Device Register Layout


Name
Offset from base
Direction

Function
Description



MagicValue
0x000
R

Magic value
0x74726976 (a Little Endian equivalent of the “virt” string).



Version
0x004
R

Device version number
0x2. Note: Legacy devices (see 4.2.4 Legacy interface) used 0x1.



DeviceID
0x008
R

Virtio Subsystem Device ID
See 5 Device Types for possible values. Value zero (0x0) is used to define a system memory map with placeholder devices at static, well known addresses, assigning functions to them depending on user’s needs.



VendorID
0x00c
R

Virtio Subsystem Vendor ID



DeviceFeatures
0x010
R

Flags representing features the device supports
Reading from this register returns 32 consecutive flag bits, the least significant bit depending on the last value written to DeviceFeaturesSel. Access to this register returns bits DeviceFeaturesSel 32 to (DeviceFeaturesSel 32) + 31, eg. feature bits 0 to 31 if DeviceFeaturesSel is set to 0 and features bits 32 to 63 if DeviceFeaturesSel is set to 1. Also see 2.2 Feature Bits.



DeviceFeaturesSel
0x014
W

Device (host) features word selection.
Writing to this register selects a set of 32 device feature bits accessible by reading from DeviceFeatures.



DriverFeatures
0x020
W

Flags representing device features understood and activated by the driver
Writing to this register sets 32 consecutive flag bits, the least significant bit depending on the last value written to DriverFeaturesSel. Access to this register sets bits DriverFeaturesSel 32 to (DriverFeaturesSel 32) + 31, eg. feature bits 0 to 31 if DriverFeaturesSel is set to 0 and features bits 32 to 63 if DriverFeaturesSel is set to 1. Also see 2.2 Feature Bits.



DriverFeaturesSel
0x024
W

Activated (guest) features word selection
Writing to this register selects a set of 32 activated feature bits accessible by writing to DriverFeatures.



QueueSel
0x030
W

Virtqueue index
Writing to this register selects the virtqueue that the following operations on QueueSizeMax, QueueSize, QueueReady, QueueDescLow, QueueDescHigh, QueueDriverlLow, QueueDriverHigh, QueueDeviceLow, QueueDeviceHigh and QueueReset apply to.



QueueSizeMax
0x034
R

Maximum virtqueue size
Reading from the register returns the maximum size (number of elements) of the queue the device is ready to process or zero (0x0) if the queue is not available. This applies to the queue selected by writing to QueueSel. Note: QueueSizeMax was previously known as QueueNumMax.



QueueSize
0x038
W

Virtqueue size
Queue size is the number of elements in the queue. Writing to this register notifies the device what size of the queue the driver will use. This applies to the queue selected by writing to QueueSel. Note: QueueSize was previously known as QueueNum.



QueueReady
0x044
RW

Virtqueue ready bit
Writing one (0x1) to this register notifies the device that it can execute requests from this virtqueue. Reading from this register returns the last value written to it. Both read and write accesses apply to the queue selected by writing to QueueSel.



QueueNotify
0x050
W

Queue notifier
Writing a value to this register notifies the device that there are new buffers to process in a queue.

When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, the value written is the queue index.

When VIRTIO_F_NOTIFICATION_DATA has been negotiated, the Notification data value has the following format:

 
le32 { 
  vq_index: 16; /* previously known as vqn */ 
  next_off : 15; 
  next_wrap : 1; 
};

See 2.9 Driver Notifications for the definition of the components.



InterruptStatus
0x60
R

Interrupt status
Reading from this register returns a bit mask of events that caused the device interrupt to be asserted. The following events are possible:

Used Buffer Notification
- bit 0 - the interrupt was asserted because the device has used a buffer in at least one of the active virtqueues.
Configuration Change Notification
- bit 1 - the interrupt was asserted because the configuration of the device has changed.



InterruptACK
0x064
W

Interrupt acknowledge
Writing a value with bits set as defined in InterruptStatus to this register notifies the device that events causing the interrupt have been handled.



Status
0x070
RW

Device status
Reading from this register returns the current device status flags. Writing non-zero values to this register sets the status flags, indicating the driver progress. Writing zero (0x0) to this register triggers a device reset. See also p. 4.2.3.1 Device Initialization.



QueueDescLow
0x080
QueueDescHigh
0x084
W

Virtqueue’s Descriptor Area 64 bit long physical address
Writing to these two registers (lower 32 bits of the address to QueueDescLow, higher 32 bits to QueueDescHigh) notifies the device about location of the Descriptor Area of the queue selected by writing to QueueSel register.



QueueDriverLow
0x090
QueueDriverHigh
0x094
W

Virtqueue’s Driver Area 64 bit long physical address
Writing to these two registers (lower 32 bits of the address to QueueDriverLow, higher 32 bits to QueueDriverHigh) notifies the device about location of the Driver Area of the queue selected by writing to QueueSel.



QueueDeviceLow
0x0a0
QueueDeviceHigh
0x0a4
W

Virtqueue’s Device Area 64 bit long physical address
Writing to these two registers (lower 32 bits of the address to QueueDeviceLow, higher 32 bits to QueueDeviceHigh) notifies the device about location of the Device Area of the queue selected by writing to QueueSel.



SHMSel
0x0ac
W

Shared memory id
Writing to this register selects the shared memory region 2.10 following operations on SHMLenLow, SHMLenHigh, SHMBaseLow and SHMBaseHigh apply to.



SHMLenLow
0x0b0
SHMLenHigh
0x0b4
R

Shared memory region 64 bit long length
These registers return the length of the shared memory region in bytes, as defined by the device for the region selected by the SHMSel register. The lower 32 bits of the length are read from SHMLenLow and the higher 32 bits from SHMLenHigh. Reading from a non-existent region (i.e. where the ID written to SHMSel is unused) results in a length of -1.



SHMBaseLow
0x0b8
SHMBaseHigh
0x0bc
R

Shared memory region 64 bit long physical address
The driver reads these registers to discover the base address of the region in physical address space. This address is chosen by the device (or other part of the VMM). The lower 32 bits of the address are read from SHMBaseLow with the higher 32 bits from SHMBaseHigh. Reading from a non-existent region (i.e. where the ID written to SHMSel is unused) results in a base address of 0xffffffffffffffff.



QueueReset
0x0c0
RW

Virtqueue reset bit
If VIRTIO_F_RING_RESET has been negotiated, writing one (0x1) to this register selectively resets the queue. Both read and write accesses apply to the queue selected by writing to QueueSel.



ConfigGeneration
0x0fc
R

Configuration atomicity value
Reading from this register returns a value describing a version of the device-specific configuration space (see Config). The driver can then access the configuration space and, when finished, read ConfigGeneration again. If no part of the configuration space has changed between these two ConfigGeneration reads, the returned values are identical. If the values are different, the configuration space accesses were not atomic and the driver has to perform the operations again. See also 2.5.



Config
0x100+
RW

Configuration space
Device-specific configuration space starts at the offset 0x100 and is accessed with byte alignment. Its meaning and size depend on the device and the driver.



4.2.2.1 Device Requirements: MMIO Device Register Layout

The device MUST return 0x74726976 in MagicValue.

The device MUST return value 0x2 in Version.

The device MUST present each event by setting the corresponding bit in InterruptStatus from the moment it takes place, until the driver acknowledges the interrupt by writing a corresponding bit mask to the InterruptACK register. Bits which do not represent events which took place MUST be zero.

Upon reset, the device MUST clear all bits in InterruptStatus and ready bits in the QueueReady register for all queues in the device.

The device MUST change value returned in ConfigGeneration if there is any risk of a driver seeing an inconsistent configuration state.

The device MUST NOT access virtqueue contents when QueueReady is zero (0x0).

If VIRTIO_F_RING_RESET has been negotiated, the device MUST present a 0 in QueueReset on reset.

If VIRTIO_F_RING_RESET has been negotiated, The device MUST present a 0 in QueueReset after the virtqueue is enabled with QueueReady.

The device MUST reset the queue when 1 is written to QueueReset. The device MUST continue to present 1 in QueueReset as long as the queue reset is ongoing. The device MUST present 0 in both QueueReset and QueueReady when queue reset has completed. (see 2.6.1).

4.2.2.2 Driver Requirements: MMIO Device Register Layout

The driver MUST NOT access memory locations not described in the table 4.1 (or, in case of the configuration space, described in the device specification), MUST NOT write to the read-only registers (direction R) and MUST NOT read from the write-only registers (direction W).

The driver MUST only use 32 bit wide and aligned reads and writes to access the control registers described in table 4.1. For the device-specific configuration space, the driver MUST use 8 bit wide accesses for 8 bit wide fields, 16 bit wide and aligned accesses for 16 bit wide fields and 32 bit wide and aligned accesses for 32 and 64 bit wide fields.

The driver MUST ignore a device with MagicValue which is not 0x74726976, although it MAY report an error.

The driver MUST ignore a device with Version which is not 0x2, although it MAY report an error.

The driver MUST ignore a device with DeviceID 0x0, but MUST NOT report any error.

Before reading from DeviceFeatures, the driver MUST write a value to DeviceFeaturesSel.

Before writing to the DriverFeatures register, the driver MUST write a value to the DriverFeaturesSel register.

The driver MUST write a value to QueueSize which is less than or equal to the value presented by the device in QueueSizeMax.

When QueueReady is not zero, the driver MUST NOT access QueueSize, QueueDescLow, QueueDescHigh, QueueDriverLow, QueueDriverHigh, QueueDeviceLow, QueueDeviceHigh.

To stop using the queue the driver MUST write zero (0x0) to this QueueReady and MUST read the value back to ensure synchronization.

The driver MUST ignore undefined bits in InterruptStatus.

The driver MUST write a value with a bit mask describing events it handled into InterruptACK when it finishes handling an interrupt and MUST NOT set any of the undefined bits in the value.

If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to QueueReset to reset the queue, the driver MUST NOT consider queue reset to be complete until it reads back 0 in QueueReset. The driver MAY re-enable the queue by writing 1 to QueueReady after ensuring that other virtqueue fields have been set up correctly. The driver MAY set driver-writeable queue configuration values to different values than those that were used before the queue reset. (see 2.6.1).

4.2.3 MMIO-specific Initialization And Device Operation

4.2.3.1 Device Initialization

4.2.3.1.1 Driver Requirements: Device Initialization
The driver MUST start the device initialization by reading and checking values from MagicValue and Version. If both values are valid, it MUST read DeviceID and if its value is zero (0x0) MUST abort initialization and MUST NOT access any other register.

Drivers not expecting shared memory MUST NOT use the shared memory registers.

Further initialization MUST follow the procedure described in 3.1 Device Initialization.

4.2.3.2 Virtqueue Configuration

The driver will typically initialize the virtqueue in the following way:

1.
Select the queue by writing its index to QueueSel.
2.
Check if the queue is not already in use: read QueueReady, and expect a returned value of zero (0x0).
3.
Read maximum queue size (number of elements) from QueueSizeMax. If the returned value is zero (0x0) the queue is not available.
4.
Allocate and zero the queue memory, making sure the memory is physically contiguous.
5.
Notify the device about the queue size by writing the size to QueueSize.
6.
Write physical addresses of the queue’s Descriptor Area, Driver Area and Device Area to (respectively) the QueueDescLow/QueueDescHigh, QueueDriverLow/QueueDriverHigh and QueueDeviceLow/QueueDeviceHigh register pairs.
7.
Write 0x1 to QueueReady.
4.2.3.3 Available Buffer Notifications

When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, the driver sends an available buffer notification to the device by writing the 16-bit virtqueue index of the queue to be notified to QueueNotify.

When VIRTIO_F_NOTIFICATION_DATA has been negotiated, the driver sends an available buffer notification to the device by writing the following 32-bit value to QueueNotify:

 
le32 { 
  vq_index: 16; /* previously known as vqn */ 
  next_off : 15; 
  next_wrap : 1; 
};

See 2.9 Driver Notifications for the definition of the components.

4.2.3.4 Notifications From The Device

The memory mapped virtio device is using a single, dedicated interrupt signal, which is asserted when at least one of the bits described in the description of InterruptStatus is set. This is how the device sends a used buffer notification or a configuration change notification to the device.

4.2.3.4.1 Driver Requirements: Notifications From The Device
After receiving an interrupt, the driver MUST read InterruptStatus to check what caused the interrupt (see the register description). The used buffer notification bit being set SHOULD be interpreted as a used buffer notification for each active virtqueue. After the interrupt is handled, the driver MUST acknowledge it by writing a bit mask corresponding to the handled events to the InterruptACK register.

4.2.4 Legacy interface

The legacy MMIO transport used page-based addressing, resulting in a slightly different control register layout, the device initialization and the virtqueue configuration procedure.

Table 4.2 presents control registers layout, omitting descriptions of registers which did not change their function nor behaviour:

Table 4.2: MMIO Device Legacy Register Layout


Name
Offset from base
Direction

Function
Description



MagicValue
0x000
R

Magic value



Version
0x004
R

Device version number
Legacy device returns value 0x1.



DeviceID
0x008
R

Virtio Subsystem Device ID



VendorID
0x00c
R

Virtio Subsystem Vendor ID



HostFeatures
0x010
R

Flags representing features the device supports



HostFeaturesSel
0x014
W

Device (host) features word selection.



GuestFeatures
0x020
W

Flags representing device features understood and activated by the driver



GuestFeaturesSel
0x024
W

Activated (guest) features word selection



GuestPageSize
0x028
W

Guest page size
The driver writes the guest page size in bytes to the register during initialization, before any queues are used. This value should be a power of 2 and is used by the device to calculate the Guest address of the first queue page (see QueuePFN).



QueueSel
0x030
W

Virtqueue index
Writing to this register selects the virtqueue that the following operations on the QueueSizeMax, QueueSize, QueueAlign and QueuePFN registers apply to.



QueueSizeMax
0x034
R

Maximum virtqueue size
Reading from the register returns the maximum size of the queue the device is ready to process or zero (0x0) if the queue is not available. This applies to the queue selected by writing to QueueSel and is allowed only when QueuePFN is set to zero (0x0), so when the queue is not actively used. Note: QueueSizeMax was previously known as QueueNumMax.



QueueSize
0x038
W

Virtqueue size
Queue size is the number of elements in the queue. Writing to this register notifies the device what size of the queue the driver will use. This applies to the queue selected by writing to QueueSel. Note: QueueSize was previously known as QueueNum.



QueueAlign
0x03c
W

Used Ring alignment in the virtqueue
Writing to this register notifies the device about alignment boundary of the Used Ring in bytes. This value should be a power of 2 and applies to the queue selected by writing to QueueSel.



QueuePFN
0x040
RW

Guest physical page number of the virtqueue
Writing to this register notifies the device about location of the virtqueue in the Guest’s physical address space. This value is the index number of a page starting with the queue Descriptor Table. Value zero (0x0) means physical address zero (0x00000000) and is illegal. When the driver stops using the queue it writes zero (0x0) to this register. Reading from this register returns the currently used page number of the queue, therefore a value other than zero (0x0) means that the queue is in use. Both read and write accesses apply to the queue selected by writing to QueueSel.



QueueNotify
0x050
W

Queue notifier



InterruptStatus
0x60
R

Interrupt status



InterruptACK
0x064
W

Interrupt acknowledge



Status
0x070
RW

Device status
Reading from this register returns the current device status flags. Writing non-zero values to this register sets the status flags, indicating the OS/driver progress. Writing zero (0x0) to this register triggers a device reset. The device sets QueuePFN to zero (0x0) for all queues in the device. Also see 3.1 Device Initialization.



Config
0x100+
RW

Configuration space



The virtqueue page size is defined by writing to GuestPageSize, as written by the guest. The driver does this before the virtqueues are configured.

The virtqueue layout follows p. 2.7.2 Legacy Interfaces: A Note on Virtqueue Layout, with the alignment defined in QueueAlign.

The virtqueue is configured as follows:

1.
Select the queue by writing its index to QueueSel.
2.
Check if the queue is not already in use: read QueuePFN, expecting a returned value of zero (0x0).
3.
Read maximum queue size (number of elements) from QueueSizeMax. If the returned value is zero (0x0) the queue is not available.
4.
Allocate and zero the queue pages in contiguous virtual memory, aligning the Used Ring to an optimal boundary (usually page size). The driver should choose a queue size smaller than or equal to QueueSizeMax.
5.
Notify the device about the queue size by writing the size to QueueSize.
6.
Notify the device about the used alignment by writing its value in bytes to QueueAlign.
7.
Write the physical number of the first page of the queue to the QueuePFN register.

Notification mechanisms did not change.

4.2.5 Features reserved for future use

Devices and drivers utilizing Virtio Over MMIO do not support the following features:

These features are reserved for future use.

4.3 Virtio Over Channel I/O

S/390 based virtual machines support neither PCI nor MMIO, so a different transport is needed there.

virtio-ccw uses the standard channel I/O based mechanism used for the majority of devices on S/390. A virtual channel device with a special control unit type acts as proxy to the virtio device (similar to the way virtio-pci uses a PCI device) and configuration and operation of the virtio device is accomplished (mostly) via channel commands. This means virtio devices are discoverable via standard operating system algorithms, and adding virtio support is mainly a question of supporting a new control unit type.

As the S/390 is a big endian machine, the data structures transmitted via channel commands are big-endian: this is made clear by use of the types be16, be32 and be64.

4.3.1 Basic Concepts

As a proxy device, virtio-ccw uses a channel-attached I/O control unit with a special control unit type (0x3832) and a control unit model corresponding to the attached virtio device’s subsystem device ID, accessed via a virtual I/O subchannel and a virtual channel path of type 0x32. This proxy device is discoverable via normal channel subsystem device discovery (usually a STORE SUBCHANNEL loop) and answers to the basic channel commands:

For a virtio-ccw proxy device, SENSE ID will return the following information:




Bytes Description Contents






0 reserved 0xff



1-2 control unit type 0x3832



3 control unit model



4-5 device type zeroes (unset)



6 device model zeroes (unset)



7-255 extended SenseId data zeroes (unset)



A virtio-ccw proxy device facilitates:

4.3.1.1 Channel Commands for Virtio

In addition to the basic channel commands, virtio-ccw defines a set of channel commands related to configuration and operation of virtio:

#define CCW_CMD_SET_VQ 0x13 
#define CCW_CMD_VDEV_RESET 0x33 
#define CCW_CMD_SET_IND 0x43 
#define CCW_CMD_SET_CONF_IND 0x53 
#define CCW_CMD_SET_IND_ADAPTER 0x73 
#define CCW_CMD_READ_FEAT 0x12 
#define CCW_CMD_WRITE_FEAT 0x11 
#define CCW_CMD_READ_CONF 0x22 
#define CCW_CMD_WRITE_CONF 0x21 
#define CCW_CMD_WRITE_STATUS 0x31 
#define CCW_CMD_READ_VQ_CONF 0x32 
#define CCW_CMD_SET_VIRTIO_REV 0x83 
#define CCW_CMD_READ_STATUS 0x72
4.3.1.2 Notifications

Available buffer notifications are realized as a hypercall. No additional setup by the driver is needed. The operation of available buffer notifications is described in section 4.3.3.2.

Used buffer notifications are realized either as so-called classic or adapter I/O interrupts depending on a transport level negotiation. The initialization is described in sections 4.3.2.6.1 and 4.3.2.6.3 respectively. The operation of each flavor is described in sections 4.3.3.1.1 and 4.3.3.1.2 respectively.

Configuration change notifications are done using so-called classic I/O interrupts. The initialization is described in section 4.3.2.6.2 and the operation in section 4.3.3.1.1.

4.3.1.3 Device Requirements: Basic Concepts

The virtio-ccw device acts like a normal channel device, as specified in [S390 PoP] and [S390 Common I/O]. In particular:

4.3.1.4 Driver Requirements: Basic Concepts

A driver for virtio-ccw devices MUST check for a control unit type of 0x3832 and MUST ignore the device type and model.

A driver SHOULD attempt to provide the correct length in a channel command even if it suppresses length checks for that command.

4.3.2 Device Initialization

virtio-ccw uses several channel commands to set up a device.

4.3.2.1 Setting the Virtio Revision

CCW_CMD_SET_VIRTIO_REV is issued by the driver to set the revision of the virtio-ccw transport it intends to drive the device with. It uses the following communication structure:

struct virtio_rev_info { 
        be16 revision; 
        be16 length; 
        u8 data[]; 
};

revision contains the desired revision id, length the length of the data portion and data revision-dependent additional desired options.

The following values are supported:





revision length data remarks








0 0 legacy interface; transitional devices only




1 0 Virtio 1




2 0 CCW_CMD_READ_STATUS support




3-n reserved for later revisions




Note that a change in the virtio standard does not necessarily correspond to a change in the virtio-ccw revision.

4.3.2.1.1 Device Requirements: Setting the Virtio Revision
A device MUST post a unit check with command reject for any revision it does not support. For any invalid combination of revision, length and data, it MUST post a unit check with command reject as well. A non-transitional device MUST reject revision id 0.

A device SHOULD answer with command reject to any virtio-ccw specific channel command that is not contained in the revision selected by the driver.

A device MUST answer with command reject to any attempt to select a different revision after a revision has been successfully selected by the driver.

A device MUST treat the revision as unset from the time the associated subchannel has been enabled until a revision has been successfully set by the driver. This implies that revisions are not persistent across disabling and enabling of the associated subchannel.

4.3.2.1.2 Driver Requirements: Setting the Virtio Revision
A driver SHOULD start with trying to set the highest revision it supports and continue with lower revisions if it gets a command reject.

A driver MUST NOT issue any other virtio-ccw specific channel commands prior to setting the revision.

After a revision has been successfully selected by the driver, it MUST NOT attempt to select a different revision.

4.3.2.1.3 Legacy Interfaces: A Note on Setting the Virtio Revision
A legacy device will not support the CCW_CMD_SET_VIRTIO_REV and answer with a command reject. A non-transitional driver MUST stop trying to operate this device in that case. A transitional driver MUST operate the device as if it had been able to set revision 0.

A legacy driver will not issue the CCW_CMD_SET_VIRTIO_REV prior to issuing other virtio-ccw specific channel commands. A non-transitional device therefore MUST answer any such attempts with a command reject. A transitional device MUST assume in this case that the driver is a legacy driver and continue as if the driver selected revision 0. This implies that the device MUST reject any command not valid for revision 0, including a subsequent CCW_CMD_SET_VIRTIO_REV.

4.3.2.2 Configuring a Virtqueue

CCW_CMD_READ_VQ_CONF is issued by the driver to obtain information about a queue. It uses the following structure for communicating:

struct vq_config_block { 
        be16 index; 
        be16 max_queue_size; /* previously known as max_num */ 
};

The requested number of buffers for queue index is returned in max_queue_size.

Afterwards, CCW_CMD_SET_VQ is issued by the driver to inform the device about the location used for its queue. The transmitted structure is

struct vq_info_block { 
        be64 desc; 
        be32 res0; 
        be16 index; 
        be16 size; /* previously known as num */ 
        be64 driver; 
        be64 device; 
};

desc, driver and device contain the guest addresses for the descriptor area, available area and used area for queue index, respectively. The actual virtqueue size (number of allocated buffers) is transmitted in size.

4.3.2.2.1 Device Requirements: Configuring a Virtqueue
res0 is reserved and MUST be ignored by the device.

4.3.2.2.2 Legacy Interface: A Note on Configuring a Virtqueue
For a legacy driver or for a driver that selected revision 0, CCW_CMD_SET_VQ uses the following communication block:
struct vq_info_block_legacy { 
        be64 queue; 
        be32 align; 
        be16 index; 
        be16 size; /* previously known as num */ 
};

queue contains the guest address for queue index, size the number of buffers and align the alignment. The queue layout follows 2.7.2 Legacy Interfaces: A Note on Virtqueue Layout.

4.3.2.3 Communicating Status Information

The driver changes the status of a device via the CCW_CMD_WRITE_STATUS command, which transmits an 8 bit status value.

As described in 2.2.2, a device sometimes fails to set the device status field: For example, it might fail to accept the FEATURES_OK status bit during device initialization.

With revision 2, CCW_CMD_READ_STATUS is defined: It reads an 8 bit status value from the device and acts as a reverse operation to CCW_CMD_WRITE_STATUS.

4.3.2.3.1 Driver Requirements: Communicating Status Information
If the device posts a unit check with command reject in response to the CCW_CMD_WRITE_STATUS command, the driver MUST assume that the device failed to set the status and the device status field retained its previous value.

If at least revision 2 has been negotiated, the driver SHOULD use the CCW_CMD_READ_STATUS command to retrieve the device status field after a configuration change has been detected.

If not at least revision 2 has been negotiated, the driver MUST NOT attempt to issue the CCW_CMD_READ_STATUS command.

4.3.2.3.2 Device Requirements: Communicating Status Information
If the device fails to set the device status field to the value written by the driver, the device MUST assure that the device status field is left unchanged and MUST post a unit check with command reject.

If at least revision 2 has been negotiated, the device MUST return the current device status field if the CCW_CMD_READ_STATUS command is issued.

4.3.2.4 Handling Device Features

Feature bits are arranged in an array of 32 bit values, making for a total of 8192 feature bits. Feature bits are in little-endian byte order.

The CCW commands dealing with features use the following communication block:

struct virtio_feature_desc { 
        le32 features; 
        u8 index; 
};

features are the 32 bits of features currently accessed, while index describes which of the feature bit values is to be accessed. No padding is added at the end of the structure, it is exactly 5 bytes in length.

The guest obtains the device’s device feature set via the CCW_CMD_READ_FEAT command. The device stores the features at index to features.

For communicating its supported features to the device, the driver uses the CCW_CMD_WRITE_FEAT command, denoting a features/index combination.

4.3.2.5 Device Configuration

The device’s configuration space is located in host memory.

To obtain information from the configuration space, the driver uses CCW_CMD_READ_CONF, specifying the guest memory for the device to write to.

For changing configuration information, the driver uses CCW_CMD_WRITE_CONF, specifying the guest memory for the device to read from.

In both cases, the complete configuration space is transmitted. This allows the driver to compare the new configuration space with the old version, and keep a generation count internally whenever it changes.

4.3.2.6 Setting Up Indicators

In order to set up the indicator bits for host->guest notification, the driver uses different channel commands depending on whether it wishes to use traditional I/O interrupts tied to a subchannel or adapter I/O interrupts for virtqueue notifications. For any given device, the two mechanisms are mutually exclusive.

For the configuration change indicators, only a mechanism using traditional I/O interrupts is provided, regardless of whether traditional or adapter I/O interrupts are used for virtqueue notifications.

4.3.2.6.1 Setting Up Classic Queue Indicators
Indicators for notification via classic I/O interrupts are contained in a 64 bit value per virtio-ccw proxy device.

To communicate the location of the indicator bits for host->guest notification, the driver uses the CCW_CMD_SET_IND command, pointing to a location containing the guest address of the indicators in a 64 bit value.

If the driver has already set up two-staged queue indicators via the CCW_CMD_SET_IND_ADAPTER command, the device MUST post a unit check with command reject to any subsequent CCW_CMD_SET_IND command.

4.3.2.6.2 Setting Up Configuration Change Indicators
Indicators for configuration change host->guest notification are contained in a 64 bit value per virtio-ccw proxy device.

To communicate the location of the indicator bits used in the configuration change host->guest notification, the driver issues the CCW_CMD_SET_CONF_IND command, pointing to a location containing the guest address of the indicators in a 64 bit value.

4.3.2.6.3 Setting Up Two-Stage Queue Indicators
Indicators for notification via adapter I/O interrupts consist of two stages:

To communicate the location of the summary and queue indicator bits, the driver uses the CCW_CMD_SET_IND_ADAPTER command with the following payload:

struct virtio_thinint_area { 
        be64 summary_indicator; 
        be64 indicator; 
        be64 bit_nr; 
        u8 isc; 
} __attribute__ ((packed));

summary_indicator contains the guest address of the 8 bit summary indicator. indicator contains the guest address of an area wherein the indicators for the devices are contained, starting at bit_nr, one bit per virtqueue of the device. Bit numbers start at the left, i.e. the most significant bit in the first byte is assigned the bit number 0. isc contains the I/O interruption subclass to be used for the adapter I/O interrupt. It MAY be different from the isc used by the proxy virtio-ccw device’s subchannel. No padding is added at the end of the structure, it is exactly 25 bytes in length.

4.3.2.6.3.1 Device Requirements: Setting Up Two-Stage Queue Indicators
If the driver has already set up classic queue indicators via the CCW_CMD_SET_IND command, the device MUST post a unit check with command reject to any subsequent CCW_CMD_SET_IND_ADAPTER command.

4.3.2.6.4 Legacy Interfaces: A Note on Setting Up Indicators
In some cases, legacy devices will only support classic queue indicators; in that case, they will reject CCW_CMD_SET_IND_ADAPTER as they don’t know that command. Some legacy devices will support two-stage queue indicators, though, and a driver will be able to successfully use CCW_CMD_SET_IND_ADAPTER to set them up.

4.3.3 Device Operation

4.3.3.1 Host->Guest Notification

There are two modes of operation regarding host->guest notification, classic I/O interrupts and adapter I/O interrupts. The mode to be used is determined by the driver by using CCW_CMD_SET_IND respectively CCW_CMD_SET_IND_ADAPTER to set up queue indicators.

For configuration changes, the driver always uses classic I/O interrupts.

4.3.3.1.1 Notification via Classic I/O Interrupts
If the driver used the CCW_CMD_SET_IND command to set up queue indicators, the device will use classic I/O interrupts for host->guest notification about virtqueue activity.

For notifying the driver of virtqueue buffers, the device sets the corresponding bit in the guest-provided indicators. If an interrupt is not already pending for the subchannel, the device generates an unsolicited I/O interrupt.

If the device wants to notify the driver about configuration changes, it sets bit 0 in the configuration indicators and generates an unsolicited I/O interrupt, if needed. This also applies if adapter I/O interrupts are used for queue notifications.

4.3.3.1.2 Notification via Adapter I/O Interrupts
If the driver used the CCW_CMD_SET_IND_ADAPTER command to set up queue indicators, the device will use adapter I/O interrupts for host->guest notification about virtqueue activity.

For notifying the driver of virtqueue buffers, the device sets the bit in the guest-provided indicator area at the corresponding offset. The guest-provided summary indicator is set to 0x01. An adapter I/O interrupt for the corresponding interruption subclass is generated.

The recommended way to process an adapter I/O interrupt by the driver is as follows:

4.3.3.1.2.1 Device Requirements: Notification via Adapter I/O Interrupts
The device SHOULD only generate an adapter I/O interrupt if the summary indicator had not been set prior to notification.

4.3.3.1.2.2 Driver Requirements: Notification via Adapter I/O Interrupts
The driver MUST clear the summary indicator after receiving an adapter I/O interrupt before it processes the queue indicators.

4.3.3.1.3 Legacy Interfaces: A Note on Host->Guest Notification
As legacy devices and drivers support only classic queue indicators, host->guest notification will always be done via classic I/O interrupts.
4.3.3.2 Guest->Host Notification

For notifying the device of virtqueue buffers, the driver unfortunately can’t use a channel command (the asynchronous characteristics of channel I/O interact badly with the host block I/O backend). Instead, it uses a diagnose 0x500 call with subcode 3 specifying the queue, as follows:




GPR Input Value Output Value






1 0x3



2 Subchannel ID Host Cookie



3 Notification data



4 Host Cookie



When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, the Notification data contains the virtqueue index.

When VIRTIO_F_NOTIFICATION_DATA has been negotiated, the value has the following format:

 
be32 { 
  vq_index: 16; /* previously known as vqn */ 
  next_off : 15; 
  next_wrap : 1; 
};

See 2.9 Driver Notifications for the definition of the components.

4.3.3.2.1 Device Requirements: Guest->Host Notification
The device MUST ignore bits 0-31 (counting from the left) of GPR2. This aligns passing the subchannel ID with the way it is passed for the existing I/O instructions.

The device MAY return a 64-bit host cookie in GPR2 to speed up the notification execution.

4.3.3.2.2 Driver Requirements: Guest->Host Notification
For each notification, the driver SHOULD use GPR4 to pass the host cookie received in GPR2 from the previous notification. Note: For example:
info->cookie = do_notify(schid, 
                         virtqueue_get_queue_index(vq), 
                         info->cookie);
4.3.3.3 Resetting Devices

In order to reset a device, a driver sends the CCW_CMD_VDEV_RESET command. This command does not carry any payload.

The device signals completion of the virtio reset operation through successful conclusion of the CCW_CMD_VDEV_RESET channel command. In particular, the command not only triggers the reset operation, but the reset operation is already completed when the operation concludes successfully.

4.3.3.3.1 Device Requirements: Resetting Devices
The device MUST finish the virtio reset operation and reinitialize device status to zero before it concludes the CCW_CMD_VDEV_RESET command successfully.

The device MUST NOT send notifications or interact with the queues after it signaled successful conclusion of the CCW_CMD_VDEV_RESET command.

4.3.3.3.2 Driver Requirements: Resetting Devices
The driver MAY consider the virtio reset operation to be complete already after successful conclusion of the CCW_CMD_VDEV_RESET channel command, although it MAY also choose to verify reset completion by reading device status via CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.

4.3.4 Features reserved for future use

Devices and drivers utilizing Virtio over channel I/O do not support the following features:

These features are reserved for future use.


5 Device Types

On top of the queues, config space and feature negotiation facilities built into virtio, several devices are defined.

The following device IDs are used to identify different types of virtio devices. Some device IDs are reserved for devices which are not currently defined in this standard.

Discovering what devices are available and their type is bus-dependent.



Device ID Virtio Device




0 reserved (invalid)


1 network device


2 block device


3 console


4 entropy source


5 memory ballooning (traditional)


6 ioMemory


7 rpmsg


8 SCSI host


9 9P transport


10 mac80211 wlan


11 rproc serial


12 virtio CAIF


13 memory balloon


16 GPU device


17 Timer/Clock device


18 Input device


19 Socket device


20 Crypto device


21 Signal Distribution Module


22 pstore device


23 IOMMU device


24 Memory device


25 Sound device


26 file system device


27 PMEM device


28 RPMB device


29 mac80211 hwsim wireless simulation device


30 Video encoder device


31 Video decoder device


32 SCMI device


33 NitroSecureModule


34 I2C adapter


35 Watchdog


36 CAN device


38 Parameter Server


39 Audio policy device


40 Bluetooth device


41 GPIO device


42 RDMA device


43 Camera device


44 ISM device


45 SPI master


Some of the devices above are unspecified by this document, because they are seen as immature or especially niche. Be warned that some are only specified by the sole existing implementation; they could become part of a future specification, be abandoned entirely, or live on outside this standard. We shall speak of them no further.

5.1 Network Device

The virtio network device is a virtual network interface controller. It consists of a virtual Ethernet link which connects the device to the Ethernet network. The device has transmit and receive queues. The driver adds empty buffers to the receive virtqueue. The device receives incoming packets from the link; the device places these incoming packets in the receive virtqueue buffers. The driver adds outgoing packets to the transmit virtqueue. The device removes these packets from the transmit virtqueue and sends them to the link. The device may have a control virtqueue. The driver uses the control virtqueue to dynamically manipulate various features of the initialized device.

5.1.1 Device ID

1

5.1.2 Virtqueues

0
receiveq1
1
transmitq1
2(N-1)
receiveqN
2(N-1)+1
transmitqN
2N
controlq

N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by max_virtqueue_pairs.

controlq is optional; it only exists if VIRTIO_NET_F_CTRL_VQ is negotiated.

5.1.3 Feature bits

VIRTIO_NET_F_CSUM (0)
Device handles packets with partial checksum offload.
VIRTIO_NET_F_GUEST_CSUM (1)
Driver handles packets with partial checksum.
VIRTIO_NET_F_CTRL_GUEST_OFFLOADS (2)
Control channel offloads reconfiguration support.
VIRTIO_NET_F_MTU(3)
Device maximum MTU reporting is supported. If offered by the device, device advises driver about the value of its maximum MTU. If negotiated, the driver uses mtu as the maximum MTU value.
VIRTIO_NET_F_MAC (5)
Device has given MAC address.
VIRTIO_NET_F_GUEST_TSO4 (7)
Driver can receive TSOv4.
VIRTIO_NET_F_GUEST_TSO6 (8)
Driver can receive TSOv6.
VIRTIO_NET_F_GUEST_ECN (9)
Driver can receive TSO with ECN.
VIRTIO_NET_F_GUEST_UFO (10)
Driver can receive UFO.
VIRTIO_NET_F_HOST_TSO4 (11)
Device can receive TSOv4.
VIRTIO_NET_F_HOST_TSO6 (12)
Device can receive TSOv6.
VIRTIO_NET_F_HOST_ECN (13)
Device can receive TSO with ECN.
VIRTIO_NET_F_HOST_UFO (14)
Device can receive UFO.
VIRTIO_NET_F_MRG_RXBUF (15)
Driver can merge receive buffers.
VIRTIO_NET_F_STATUS (16)
Configuration status field is available.
VIRTIO_NET_F_CTRL_VQ (17)
Control channel is available.
VIRTIO_NET_F_CTRL_RX (18)
Control channel RX mode support.
VIRTIO_NET_F_CTRL_VLAN (19)
Control channel VLAN filtering.
VIRTIO_NET_F_CTRL_RX_EXTRA (20)
Control channel RX extra mode support.
VIRTIO_NET_F_GUEST_ANNOUNCE(21)
Driver can send gratuitous packets.
VIRTIO_NET_F_MQ(22)
Device supports multiqueue with automatic receive steering.
VIRTIO_NET_F_CTRL_MAC_ADDR(23)
Set MAC address through control channel.
VIRTIO_NET_F_HASH_TUNNEL(51)
Device supports inner header hash for encapsulated packets.
VIRTIO_NET_F_VQ_NOTF_COAL(52)
Device supports virtqueue notification coalescing.
VIRTIO_NET_F_NOTF_COAL(53)
Device supports notifications coalescing.
VIRTIO_NET_F_GUEST_USO4 (54)
Driver can receive USOv4 packets.
VIRTIO_NET_F_GUEST_USO6 (55)
Driver can receive USOv6 packets.
VIRTIO_NET_F_HOST_USO (56)
Device can receive USO packets. Unlike UFO (fragmenting the packet) the USO splits large UDP packet to several segments when each of these smaller packets has UDP header.
VIRTIO_NET_F_HASH_REPORT(57)
Device can report per-packet hash value and a type of calculated hash.
VIRTIO_NET_F_GUEST_HDRLEN(59)
Driver can provide the exact hdr_len value. Device benefits from knowing the exact header length.
VIRTIO_NET_F_RSS(60)
Device supports RSS (receive-side scaling) with Toeplitz hash calculation and configurable hash parameters for receive steering.
VIRTIO_NET_F_RSC_EXT(61)
Device can process duplicated ACKs and report number of coalesced segments and duplicated ACKs.
VIRTIO_NET_F_STANDBY(62)
Device may act as a standby for a primary device with the same MAC address.
VIRTIO_NET_F_SPEED_DUPLEX(63)
Device reports speed and duplex.
5.1.3.1 Feature bit requirements

Some networking feature bits require other networking feature bits (see 2.2.1):

VIRTIO_NET_F_GUEST_TSO4
Requires VIRTIO_NET_F_GUEST_CSUM.
VIRTIO_NET_F_GUEST_TSO6
Requires VIRTIO_NET_F_GUEST_CSUM.
VIRTIO_NET_F_GUEST_ECN
Requires VIRTIO_NET_F_GUEST_TSO4 or VIRTIO_NET_F_GUEST_TSO6.
VIRTIO_NET_F_GUEST_UFO
Requires VIRTIO_NET_F_GUEST_CSUM.
VIRTIO_NET_F_GUEST_USO4
Requires VIRTIO_NET_F_GUEST_CSUM.
VIRTIO_NET_F_GUEST_USO6
Requires VIRTIO_NET_F_GUEST_CSUM.
VIRTIO_NET_F_HOST_TSO4
Requires VIRTIO_NET_F_CSUM.
VIRTIO_NET_F_HOST_TSO6
Requires VIRTIO_NET_F_CSUM.
VIRTIO_NET_F_HOST_ECN
Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
VIRTIO_NET_F_HOST_UFO
Requires VIRTIO_NET_F_CSUM.
VIRTIO_NET_F_HOST_USO
Requires VIRTIO_NET_F_CSUM.
VIRTIO_NET_F_CTRL_RX
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_CTRL_VLAN
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_GUEST_ANNOUNCE
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_MQ
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_CTRL_MAC_ADDR
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_NOTF_COAL
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_RSC_EXT
Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
VIRTIO_NET_F_RSS
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_VQ_NOTF_COAL
Requires VIRTIO_NET_F_CTRL_VQ.
VIRTIO_NET_F_HASH_TUNNEL
Requires VIRTIO_NET_F_CTRL_VQ along with VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
5.1.3.2 Legacy Interface: Feature bits
VIRTIO_NET_F_GSO (6)
Device handles packets with any GSO type. This was supposed to indicate segmentation offload support, but upon further investigation it became clear that multiple bits were needed.
VIRTIO_NET_F_GUEST_RSC4 (41)
Device coalesces TCPIP v4 packets. This was implemented by hypervisor patch for certification purposes and current Windows driver depends on it. It will not function if virtio-net device reports this feature.
VIRTIO_NET_F_GUEST_RSC6 (42)
Device coalesces TCPIP v6 packets. Similar to VIRTIO_NET_F_GUEST_RSC4.

5.1.4 Device configuration layout

The network device has the following device configuration layout. All of the device configuration fields are read-only for the driver.

struct virtio_net_config { 
        u8 mac[6]; 
        le16 status; 
        le16 max_virtqueue_pairs; 
        le16 mtu; 
        le32 speed; 
        u8 duplex; 
        u8 rss_max_key_size; 
        le16 rss_max_indirection_table_length; 
        le32 supported_hash_types; 
        le32 supported_tunnel_types; 
};

The mac address field always exists (although it is only valid if VIRTIO_NET_F_MAC is set).

The status only exists if VIRTIO_NET_F_STATUS is set. Two bits are currently defined for the status field: VIRTIO_NET_S_LINK_UP and VIRTIO_NET_S_ANNOUNCE.

#define VIRTIO_NET_S_LINK_UP     1 
#define VIRTIO_NET_S_ANNOUNCE    2

The following field, max_virtqueue_pairs only exists if VIRTIO_NET_F_MQ or VIRTIO_NET_F_RSS is set. This field specifies the maximum number of each of transmit and receive virtqueues (receiveq1…receiveqN and transmitq1…transmitqN respectively) that can be configured once at least one of these features is negotiated.

The following field, mtu only exists if VIRTIO_NET_F_MTU is set. This field specifies the maximum MTU for the driver to use.

The following two fields, speed and duplex, only exist if VIRTIO_NET_F_SPEED_DUPLEX is set.

speed contains the device speed, in units of 1 MBit per second, 0 to 0x7fffffff, or 0xffffffff for unknown speed.

duplex has the values of 0x01 for full duplex, 0x00 for half duplex and 0xff for unknown duplex state.

Both speed and duplex can change, thus the driver is expected to re-read these values after receiving a configuration change notification.

The following field, rss_max_key_size only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set. It specifies the maximum supported length of RSS key in bytes.

The following field, rss_max_indirection_table_length only exists if VIRTIO_NET_F_RSS is set. It specifies the maximum number of 16-bit entries in RSS indirection table.

The next field, supported_hash_types only exists if the device supports hash calculation, i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.

Field supported_hash_types contains the bitmask of supported hash types. See 5.1.6.4.3.1 for details of supported hash types.

Field supported_tunnel_types only exists if the device supports inner header hash, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.

Field supported_tunnel_types contains the bitmask of encapsulation types supported by the device for inner header hash. Encapsulation types are defined in 5.1.6.4.4.2.

5.1.4.1 Device Requirements: Device configuration layout

The device MUST set max_virtqueue_pairs to between 1 and 0x8000 inclusive, if it offers VIRTIO_NET_F_MQ.

The device MUST set mtu to between 68 and 65535 inclusive, if it offers VIRTIO_NET_F_MTU.

The device SHOULD set mtu to at least 1280, if it offers VIRTIO_NET_F_MTU.

The device MUST NOT modify mtu once it has been set.

The device MUST NOT pass received packets that exceed mtu (plus low level ethernet header length) size with gso_type NONE or ECN after VIRTIO_NET_F_MTU has been successfully negotiated.

The device MUST forward transmitted packets of up to mtu (plus low level ethernet header length) size with gso_type NONE or ECN, and do so without fragmentation, after VIRTIO_NET_F_MTU has been successfully negotiated.

The device MUST set rss_max_key_size to at least 40, if it offers VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.

The device MUST set rss_max_indirection_table_length to at least 128, if it offers VIRTIO_NET_F_RSS.

If the driver negotiates the VIRTIO_NET_F_STANDBY feature, the device MAY act as a standby device for a primary device with the same MAC address.

If VIRTIO_NET_F_SPEED_DUPLEX has been negotiated, speed MUST contain the device speed, in units of 1 MBit per second, 0 to 0x7ffffffff, or 0xfffffffff for unknown.

If VIRTIO_NET_F_SPEED_DUPLEX has been negotiated, duplex MUST have the values of 0x00 for full duplex, 0x01 for half duplex, or 0xff for unknown.

If VIRTIO_NET_F_SPEED_DUPLEX and VIRTIO_NET_F_STATUS have both been negotiated, the device SHOULD NOT change the speed and duplex fields as long as VIRTIO_NET_S_LINK_UP is set in the status.

The device SHOULD NOT offer VIRTIO_NET_F_HASH_REPORT if it does not offer VIRTIO_NET_F_CTRL_VQ.

The device SHOULD NOT offer VIRTIO_NET_F_CTRL_RX_EXTRA if it does not offer VIRTIO_NET_F_CTRL_VQ.

5.1.4.2 Driver Requirements: Device configuration layout

The driver MUST NOT write to any of the device configuration fields.

A driver SHOULD negotiate VIRTIO_NET_F_MAC if the device offers it. If the driver negotiates the VIRTIO_NET_F_MAC feature, the driver MUST set the physical address of the NIC to mac. Otherwise, it SHOULD use a locally-administered MAC address (see IEEE 802, “9.2 48-bit universal LAN MAC addresses”).

If the driver does not negotiate the VIRTIO_NET_F_STATUS feature, it SHOULD assume the link is active, otherwise it SHOULD read the link status from the bottom bit of status.

A driver SHOULD negotiate VIRTIO_NET_F_MTU if the device offers it.

If the driver negotiates VIRTIO_NET_F_MTU, it MUST supply enough receive buffers to receive at least one receive packet of size mtu (plus low level ethernet header length) with gso_type NONE or ECN.

If the driver negotiates VIRTIO_NET_F_MTU, it MUST NOT transmit packets of size exceeding the value of mtu (plus low level ethernet header length) with gso_type NONE or ECN.

A driver SHOULD negotiate the VIRTIO_NET_F_STANDBY feature if the device offers it.

If VIRTIO_NET_F_SPEED_DUPLEX has been negotiated, the driver MUST treat any value of speed above 0x7fffffff as well as any value of duplex not matching 0x00 or 0x01 as an unknown value.

If VIRTIO_NET_F_SPEED_DUPLEX has been negotiated, the driver SHOULD re-read speed and duplex after a configuration change notification.

A driver SHOULD NOT negotiate VIRTIO_NET_F_HASH_REPORT if it does not negotiate VIRTIO_NET_F_CTRL_VQ.

A driver SHOULD NOT negotiate VIRTIO_NET_F_CTRL_RX_EXTRA if it does not negotiate VIRTIO_NET_F_CTRL_VQ.

5.1.4.3 Legacy Interface: Device configuration layout

When using the legacy interface, transitional devices and drivers MUST format status and max_virtqueue_pairs in struct virtio_net_config according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.

When using the legacy interface, mac is driver-writable which provided a way for drivers to update the MAC without negotiating VIRTIO_NET_F_CTRL_MAC_ADDR.

5.1.5 Device Initialization

A driver would perform a typical initialization routine like so:

1.
Identify and initialize the receive and transmission virtqueues, up to N of each kind. If VIRTIO_NET_F_MQ feature bit is negotiated, N=max_virtqueue_pairs, otherwise identify N=1.
2.
If the VIRTIO_NET_F_CTRL_VQ feature bit is negotiated, identify the control virtqueue.
3.
Fill the receive queues with buffers: see 5.1.6.3.
4.
Even with VIRTIO_NET_F_MQ, only receiveq1, transmitq1 and controlq are used by default. The driver would send the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying the number of the transmit and receive queues to use.
5.
If the VIRTIO_NET_F_MAC feature bit is set, the configuration space mac entry indicates the “physical” address of the device, otherwise the driver would typically generate a random local MAC address.
6.
If the VIRTIO_NET_F_STATUS feature bit is negotiated, the link status comes from the bottom bit of status. Otherwise, the driver assumes it’s active.
7.
A performant driver would indicate that it will generate checksumless packets by negotiating the VIRTIO_NET_F_CSUM feature.
8.
If that feature is negotiated, a driver can use TCP segmentation or UDP segmentation/fragmentation offload by negotiating the VIRTIO_NET_F_HOST_TSO4 (IPv4 TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP), VIRTIO_NET_F_HOST_UFO (UDP fragmentation) and VIRTIO_NET_F_HOST_USO (UDP segmentation) features.
9.
The converse features are also available: a driver can save the virtual device some work by negotiating these features. Note: For example, a network packet transported between two guests on the same system might not need checksumming at all, nor segmentation, if both guests are amenable. The VIRTIO_NET_F_GUEST_CSUM feature indicates that partially checksummed packets can be received, and if it can do that then the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4 and VIRTIO_NET_F_GUEST_USO6 are the input equivalents of the features described above. See 5.1.6.3 Setting Up Receive Buffers and 5.1.6.4 Processing of Incoming Packets below.

A truly minimal driver would only accept VIRTIO_NET_F_MAC and ignore everything else.

5.1.6 Device Operation

Packets are transmitted by placing them in the transmitq1…transmitqN, and buffers for incoming packets are placed in the receiveq1…receiveqN. In each case, the packet itself is preceded by a header:

struct virtio_net_hdr { 
#define VIRTIO_NET_HDR_F_NEEDS_CSUM    1 
#define VIRTIO_NET_HDR_F_DATA_VALID    2 
#define VIRTIO_NET_HDR_F_RSC_INFO      4 
        u8 flags; 
#define VIRTIO_NET_HDR_GSO_NONE        0 
#define VIRTIO_NET_HDR_GSO_TCPV4       1 
#define VIRTIO_NET_HDR_GSO_UDP         3 
#define VIRTIO_NET_HDR_GSO_TCPV6       4 
#define VIRTIO_NET_HDR_GSO_UDP_L4      5 
#define VIRTIO_NET_HDR_GSO_ECN      0x80 
        u8 gso_type; 
        le16 hdr_len; 
        le16 gso_size; 
        le16 csum_start; 
        le16 csum_offset; 
        le16 num_buffers; 
        le32 hash_value;        (Only if VIRTIO_NET_F_HASH_REPORT negotiated) 
        le16 hash_report;       (Only if VIRTIO_NET_F_HASH_REPORT negotiated) 
        le16 padding_reserved;  (Only if VIRTIO_NET_F_HASH_REPORT negotiated) 
};

The controlq is used to control various device features described further in section 5.1.6.5.

5.1.6.1 Legacy Interface: Device Operation

When using the legacy interface, transitional devices and drivers MUST format the fields in struct virtio_net_hdr according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.

The legacy driver only presented num_buffers in the struct virtio_net_hdr when VIRTIO_NET_F_MRG_RXBUF was negotiated; without that feature the structure was 2 bytes shorter.

When using the legacy interface, the driver SHOULD ignore the used length for the transmit queues and the controlq queue. Note: Historically, some devices put the total descriptor length there, even though no data was actually written.

5.1.6.2 Packet Transmission

Transmitting a single packet is simple, but varies depending on the different features the driver negotiated.

1.
The driver can send a completely checksummed packet. In this case, flags will be zero, and gso_type will be VIRTIO_NET_HDR_GSO_NONE.
2.
If the driver negotiated VIRTIO_NET_F_CSUM, it can skip checksumming the packet:
  • flags has the VIRTIO_NET_HDR_F_NEEDS_CSUM set,
  • csum_start is set to the offset within the packet to begin checksumming, and
  • csum_offset indicates how many bytes after the csum_start the new (16 bit ones’ complement) checksum is placed by the device.
  • The TCP checksum field in the packet is set to the sum of the TCP pseudo header, so that replacing it by the ones’ complement checksum of the TCP header and body will give the correct result.
Note: For example, consider a partially checksummed TCP (IPv4) packet. It will have a 14 byte ethernet header and 20 byte IP header followed by the TCP header (with the TCP checksum field 16 bytes into that header). csum_start will be 14+20 = 34 (the TCP checksum includes the header), and csum_offset will be 16.
3.
If the driver negotiated VIRTIO_NET_F_HOST_TSO4, TSO6, USO or UFO, and the packet requires TCP segmentation, UDP segmentation or fragmentation, then gso_type is set to VIRTIO_NET_HDR_GSO_TCPV4, TCPV6, UDP_L4 or UDP. (Otherwise, it is set to VIRTIO_NET_HDR_GSO_NONE). In this case, packets larger than 1514 bytes can be transmitted: the metadata indicates how to replicate the packet header to cut it into smaller packets. The other gso fields are set:
  • If the VIRTIO_NET_F_GUEST_HDRLEN feature has been negotiated, hdr_len indicates the header length that needs to be replicated for each packet. It’s the number of bytes from the beginning of the packet to the beginning of the transport payload. Otherwise, if the VIRTIO_NET_F_GUEST_HDRLEN feature has not been negotiated, hdr_len is a hint to the device as to how much of the header needs to be kept to copy into each packet, usually set to the length of the headers, including the transport header10. Note: Some devices benefit from knowledge of the exact header length.
  • gso_size is the maximum size of each packet beyond that header (ie. MSS).
  • If the driver negotiated the VIRTIO_NET_F_HOST_ECN feature, the VIRTIO_NET_HDR_GSO_ECN bit in gso_type indicates that the TCP packet has the ECN bit set11.
4.
num_buffers is set to zero. This field is unused on transmitted packets.
5.
The header and packet are added as one output descriptor to the transmitq, and the device is notified of the new entry (see 5.1.5 Device Initialization).

5.1.6.2.1 Driver Requirements: Packet Transmission
The driver MUST set num_buffers to zero.

If VIRTIO_NET_F_CSUM is not negotiated, the driver MUST set flags to zero and SHOULD supply a fully checksummed packet to the device.

If VIRTIO_NET_F_HOST_TSO4 is negotiated, the driver MAY set gso_type to VIRTIO_NET_HDR_GSO_TCPV4 to request TCPv4 segmentation, otherwise the driver MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_TCPV4.

If VIRTIO_NET_F_HOST_TSO6 is negotiated, the driver MAY set gso_type to VIRTIO_NET_HDR_GSO_TCPV6 to request TCPv6 segmentation, otherwise the driver MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_TCPV6.

If VIRTIO_NET_F_HOST_UFO is negotiated, the driver MAY set gso_type to VIRTIO_NET_HDR_GSO_UDP to request UDP fragmentation, otherwise the driver MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_UDP.

If VIRTIO_NET_F_HOST_USO is negotiated, the driver MAY set gso_type to VIRTIO_NET_HDR_GSO_UDP_L4 to request UDP segmentation, otherwise the driver MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_UDP_L4.

The driver SHOULD NOT send to the device TCP packets requiring segmentation offload which have the Explicit Congestion Notification bit set, unless the VIRTIO_NET_F_HOST_ECN feature is negotiated, in which case the driver MUST set the VIRTIO_NET_HDR_GSO_ECN bit in gso_type.

If the VIRTIO_NET_F_CSUM feature has been negotiated, the driver MAY set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags, if so:

1.
the driver MUST validate the packet checksum at offset csum_offset from csum_start as well as all preceding offsets;
2.
the driver MUST set the packet checksum stored in the buffer to the TCP/UDP pseudo header;
3.
the driver MUST set csum_start and csum_offset such that calculating a ones’ complement checksum from csum_start up until the end of the packet and storing the result at offset csum_offset from csum_start will result in a fully checksummed packet;

If none of the VIRTIO_NET_F_HOST_TSO4, TSO6, USO or UFO options have been negotiated, the driver MUST set gso_type to VIRTIO_NET_HDR_GSO_NONE.

If gso_type differs from VIRTIO_NET_HDR_GSO_NONE, then the driver MUST also set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags and MUST set gso_size to indicate the desired MSS.

If one of the VIRTIO_NET_F_HOST_TSO4, TSO6, USO or UFO options have been negotiated:

The driver SHOULD accept the VIRTIO_NET_F_GUEST_HDRLEN feature if it has been offered, and if it’s able to provide the exact header length.

The driver MUST NOT set the VIRTIO_NET_HDR_F_DATA_VALID and VIRTIO_NET_HDR_F_RSC_INFO bits in flags.

5.1.6.2.2 Device Requirements: Packet Transmission
The device MUST ignore flag bits that it does not recognize.

If VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags is not set, the device MUST NOT use the csum_start and csum_offset.

If one of the VIRTIO_NET_F_HOST_TSO4, TSO6, USO or UFO options have been negotiated:

If VIRTIO_NET_HDR_F_NEEDS_CSUM is not set, the device MUST NOT rely on the packet checksum being correct.

5.1.6.2.3 Packet Transmission Interrupt
Often a driver will suppress transmission virtqueue interrupts and check for used packets in the transmit path of following packets.

The normal behavior in this interrupt handler is to retrieve used buffers from the virtqueue and free the corresponding headers and packets.

5.1.6.3 Setting Up Receive Buffers

It is generally a good idea to keep the receive virtqueue as fully populated as possible: if it runs out, network performance will suffer.

If the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6 features are used, the maximum incoming packet will be to 65550 bytes long (the maximum size of a TCP or UDP packet, plus the 14 byte ethernet header), otherwise 1514 bytes. The 12-byte struct virtio_net_hdr is prepended to this, making for 65562 or 1526 bytes.

5.1.6.3.1 Driver Requirements: Setting Up Receive Buffers
Note: Obviously each buffer can be split across multiple descriptor elements.

If VIRTIO_NET_F_MQ is negotiated, each of receiveq1…receiveqN that will be used SHOULD be populated with receive buffers.

5.1.6.3.2 Device Requirements: Setting Up Receive Buffers
The device MUST set num_buffers to the number of descriptors used to hold the incoming packet.

The device MUST use only a single descriptor if VIRTIO_NET_F_MRG_RXBUF was not negotiated. Note: This means that num_buffers will always be 1 if VIRTIO_NET_F_MRG_RXBUF is not negotiated.

5.1.6.4 Processing of Incoming Packets

When a packet is copied into a buffer in the receiveq, the optimal path is to disable further used buffer notifications for the receiveq and process packets until no more are found, then re-enable them.

Processing incoming packets involves:

1.
num_buffers indicates how many descriptors this packet is spread over (including this one): this will always be 1 if VIRTIO_NET_F_MRG_RXBUF was not negotiated. This allows receipt of large packets without having to allocate large buffers: a packet that does not fit in a single buffer can flow over to the next buffer, and so on. In this case, there will be at least num_buffers used buffers in the virtqueue, and the device chains them together to form a single packet in a way similar to how it would store it in a single buffer spread over multiple descriptors. The other buffers will not begin with a struct virtio_net_hdr.
2.
If num_buffers is one, then the entire packet will be contained within this buffer, immediately following the struct virtio_net_hdr.
3.
If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has validated the packet checksum. In case of multiple encapsulated protocols, one level of checksums has been validated.

Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features enable receive checksum, large receive offload and ECN support which are the input equivalents of the transmit checksum, transmit segmentation offloading and ECN features, as described in 5.1.6.2:

1.
If the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options were negotiated, then gso_type MAY be something other than VIRTIO_NET_HDR_GSO_NONE, and gso_size field indicates the desired MSS (see Packet Transmission point 2).
2.
If the VIRTIO_NET_F_RSC_EXT option was negotiated (this implies one of VIRTIO_NET_F_GUEST_TSO4, TSO6), the device processes also duplicated ACK segments, reports number of coalesced TCP segments in csum_start field and number of duplicated ACK segments in csum_offset field and sets bit VIRTIO_NET_HDR_F_RSC_INFO in flags.
3.
If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags can be set: if so, the packet checksum at offset csum_offset from csum_start and any preceding checksums have been validated. The checksum on the packet is incomplete and if bit VIRTIO_NET_HDR_F_RSC_INFO is not set in flags, then csum_start and csum_offset indicate how to calculate it (see Packet Transmission point 1).

If applicable, the device calculates per-packet hash for incoming packets as defined in 5.1.6.4.3.

If applicable, the device reports hash information for incoming packets as defined in 5.1.6.4.5.

5.1.6.4.1 Device Requirements: Processing of Incoming Packets
If VIRTIO_NET_F_MRG_RXBUF has not been negotiated, the device MUST set num_buffers to 1.

If VIRTIO_NET_F_MRG_RXBUF has been negotiated, the device MUST set num_buffers to indicate the number of buffers the packet (including the header) is spread over.

If a receive packet is spread over multiple buffers, the device MUST use all buffers but the last (i.e. the first num_buffers - 1 buffers) completely up to the full length of each buffer supplied by the driver.

The device MUST use all buffers used by a single receive packet together, such that at least num_buffers are observed by driver as used.

If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device MUST set flags to zero and SHOULD supply a fully checksummed packet to the driver.

If VIRTIO_NET_F_GUEST_TSO4 is not negotiated, the device MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_TCPV4.

If VIRTIO_NET_F_GUEST_UDP is not negotiated, the device MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_UDP.

If VIRTIO_NET_F_GUEST_TSO6 is not negotiated, the device MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_TCPV6.

If none of VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6 have been negotiated, the device MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_UDP_L4.

The device SHOULD NOT send to the driver TCP packets requiring segmentation offload which have the Explicit Congestion Notification bit set, unless the VIRTIO_NET_F_GUEST_ECN feature is negotiated, in which case the device MUST set the VIRTIO_NET_HDR_GSO_ECN bit in gso_type.

If the VIRTIO_NET_F_GUEST_CSUM feature has been negotiated, the device MAY set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags, if so:

1.
the device MUST validate the packet checksum at offset csum_offset from csum_start as well as all preceding offsets;
2.
the device MUST set the packet checksum stored in the receive buffer to the TCP/UDP pseudo header;
3.
the device MUST set csum_start and csum_offset such that calculating a ones’ complement checksum from csum_start up until the end of the packet and storing the result at offset csum_offset from csum_start will result in a fully checksummed packet;

If none of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have been negotiated, the device MUST set gso_type to VIRTIO_NET_HDR_GSO_NONE.

If gso_type differs from VIRTIO_NET_HDR_GSO_NONE, then the device MUST also set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags MUST set gso_size to indicate the desired MSS. If VIRTIO_NET_F_RSC_EXT was negotiated, the device MUST also set VIRTIO_NET_HDR_F_RSC_INFO bit in flags, set csum_start to number of coalesced TCP segments and set csum_offset to number of received duplicated ACK segments.

If VIRTIO_NET_F_RSC_EXT was not negotiated, the device MUST not set VIRTIO_NET_HDR_F_RSC_INFO bit in flags.

If one of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have been negotiated, the device SHOULD set hdr_len to a value not less than the length of the headers, including the transport header.

If the VIRTIO_NET_F_GUEST_CSUM feature has been negotiated, the device MAY set the VIRTIO_NET_HDR_F_DATA_VALID bit in flags, if so, the device MUST validate the packet checksum (in case of multiple encapsulated protocols, one level of checksums is validated).

5.1.6.4.2 Driver Requirements: Processing of Incoming Packets
The driver MUST ignore flag bits that it does not recognize.

If VIRTIO_NET_HDR_F_NEEDS_CSUM bit in flags is not set or if VIRTIO_NET_HDR_F_RSC_INFO bit flags is set, the driver MUST NOT use the csum_start and csum_offset.

If one of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have been negotiated, the driver MAY use hdr_len only as a hint about the transport header size. The driver MUST NOT rely on hdr_len to be correct. Note: This is due to various bugs in implementations.

If neither VIRTIO_NET_HDR_F_NEEDS_CSUM nor VIRTIO_NET_HDR_F_DATA_VALID is set, the driver MUST NOT rely on the packet checksum being correct.

5.1.6.4.3 Hash calculation for incoming packets
A device attempts to calculate a per-packet hash in the following cases:

If the feature VIRTIO_NET_F_RSS was negotiated:

If the feature VIRTIO_NET_F_RSS was not negotiated:

Note that if the device offers VIRTIO_NET_F_HASH_REPORT, even if it supports only one pair of virtqueues, it MUST support at least one of commands of VIRTIO_NET_CTRL_MQ class to configure reported hash parameters:

The per-packet hash calculation can depend on the IP packet type. See [IP], [UDP] and [TCP].

5.1.6.4.3.1 Supported/enabled hash types
Hash types applicable for IPv4 packets:
#define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0) 
#define VIRTIO_NET_HASH_TYPE_TCPv4             (1 << 1) 
#define VIRTIO_NET_HASH_TYPE_UDPv4             (1 << 2)

Hash types applicable for IPv6 packets without extension headers

#define VIRTIO_NET_HASH_TYPE_IPv6              (1 << 3) 
#define VIRTIO_NET_HASH_TYPE_TCPv6             (1 << 4) 
#define VIRTIO_NET_HASH_TYPE_UDPv6             (1 << 5)

Hash types applicable for IPv6 packets with extension headers

#define VIRTIO_NET_HASH_TYPE_IP_EX             (1 << 6) 
#define VIRTIO_NET_HASH_TYPE_TCP_EX            (1 << 7) 
#define VIRTIO_NET_HASH_TYPE_UDP_EX            (1 << 8)

5.1.6.4.3.2 IPv4 packets
The device calculates the hash on IPv4 packets according to ’Enabled hash types’ bitmask as follows:

5.1.6.4.3.3 IPv6 packets without extension header
The device calculates the hash on IPv6 packets without extension headers according to ’Enabled hash types’ bitmask as follows:

5.1.6.4.3.4 IPv6 packets with extension header
The device calculates the hash on IPv6 packets with extension headers according to ’Enabled hash types’ bitmask as follows:

5.1.6.4.4 Inner Header Hash
If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the driver can send the command VIRTIO_NET_CTRL_HASH_TUNNEL_SET to configure the calculation of the inner header hash.
struct virtnet_hash_tunnel { 
    le32 enabled_tunnel_types; 
}; 
 
#define VIRTIO_NET_CTRL_HASH_TUNNEL 7 
 #define VIRTIO_NET_CTRL_HASH_TUNNEL_SET 0

Field enabled_tunnel_types contains the bitmask of encapsulation types enabled for inner header hash. See 5.1.6.4.4.2.

The class VIRTIO_NET_CTRL_HASH_TUNNEL has one command: VIRTIO_NET_CTRL_HASH_TUNNEL_SET sets enabled_tunnel_types for the device using the virtnet_hash_tunnel structure, which is read-only for the device.

Inner header hash is disabled by VIRTIO_NET_CTRL_HASH_TUNNEL_SET with enabled_tunnel_types set to 0.

Initially (before the driver sends any VIRTIO_NET_CTRL_HASH_TUNNEL_SET command) all encapsulation types are disabled for inner header hash.

5.1.6.4.4.1 Encapsulated packet
Multiple tunneling protocols allow encapsulating an inner, payload packet in an outer, encapsulated packet. The encapsulated packet thus contains an outer header and an inner header, and the device calculates the hash over either the inner header or the outer header.

If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated packet’s outer header matches one of the encapsulation types enabled in enabled_tunnel_types, then the device uses the inner header for hash calculations (only a single level of encapsulation is currently supported).

If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received packet’s (outer) header does not match any encapsulation types enabled in enabled_tunnel_types, then the device uses the outer header for hash calculations.

5.1.6.4.4.2 Encapsulation types supported/enabled for inner header hash
Encapsulation types applicable for inner header hash:
#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784    (1 << 0) /* [RFC2784] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890    (1 << 1) /* [RFC2890] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676    (1 << 2) /* [RFC7676] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP     (1 << 3) /* [GRE-in-UDP] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 4) /* [VXLAN] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 5) /* [VXLAN-GPE] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 6) /* [GENEVE] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 7) /* [IPIP] */ 
#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 8) /* [NVGRE] */

5.1.6.4.4.3 Advice
Example uses of the inner header hash:

As using the inner header hash completely discards the outer header entropy, care must be taken if the inner header is controlled by an adversary, as the adversary can then intentionally create configurations with insufficient entropy.

Besides disabling the inner header hash, mitigations would depend on how the hash is used. When the hash use is limited to the RSS queue selection, the inner header hash may have quality of service (QoS) limitations.

5.1.6.4.4.4 Device Requirements: Inner Header Hash
If the (outer) header of the received packet does not match any encapsulation types enabled in enabled_tunnel_types, the device MUST calculate the hash on the outer header.

If the device receives any bits in enabled_tunnel_types which are not set in supported_tunnel_types, it SHOULD respond to the VIRTIO_NET_CTRL_HASH_TUNNEL_SET command with VIRTIO_NET_ERR.

If the driver sets enabled_tunnel_types to 0 through VIRTIO_NET_CTRL_HASH_TUNNEL_SET or upon the device reset, the device MUST disable the inner header hash for all encapsulation types.

5.1.6.4.4.5 Driver Requirements: Inner Header Hash
The driver MUST have negotiated the VIRTIO_NET_F_HASH_TUNNEL feature when issuing the VIRTIO_NET_CTRL_HASH_TUNNEL_SET command.

The driver MUST NOT set any bits in enabled_tunnel_types which are not set in supported_tunnel_types.

The driver MUST ignore bits in supported_tunnel_types which are not documented in this specification.

5.1.6.4.5 Hash reporting for incoming packets
If VIRTIO_NET_F_HASH_REPORT was negotiated and the device has calculated the hash for the packet, the device fills hash_report with the report type of calculated hash and hash_value with the value of calculated hash.

If VIRTIO_NET_F_HASH_REPORT was negotiated but due to any reason the hash was not calculated, the device sets hash_report to VIRTIO_NET_HASH_REPORT_NONE.

Possible values that the device can report in hash_report are defined below. They correspond to supported hash types defined in 5.1.6.4.3.1 as follows:

VIRTIO_NET_HASH_TYPE_XXX = 1 « (VIRTIO_NET_HASH_REPORT_XXX - 1)

#define VIRTIO_NET_HASH_REPORT_NONE            0 
#define VIRTIO_NET_HASH_REPORT_IPv4            1 
#define VIRTIO_NET_HASH_REPORT_TCPv4           2 
#define VIRTIO_NET_HASH_REPORT_UDPv4           3 
#define VIRTIO_NET_HASH_REPORT_IPv6            4 
#define VIRTIO_NET_HASH_REPORT_TCPv6           5 
#define VIRTIO_NET_HASH_REPORT_UDPv6           6 
#define VIRTIO_NET_HASH_REPORT_IPv6_EX         7 
#define VIRTIO_NET_HASH_REPORT_TCPv6_EX        8 
#define VIRTIO_NET_HASH_REPORT_UDPv6_EX        9
5.1.6.5 Control Virtqueue

The driver uses the control virtqueue (if VIRTIO_NET_F_CTRL_VQ is negotiated) to send commands to manipulate various features of the device which would not easily map into the configuration space.

All commands are of the following form:

struct virtio_net_ctrl { 
        u8 class; 
        u8 command; 
        u8 command-specific-data[]; 
        u8 ack; 
}; 
 
/* ack values */ 
#define VIRTIO_NET_OK     0 
#define VIRTIO_NET_ERR    1

The class, command and command-specific-data are set by the driver, and the device sets the ack byte. There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK.

5.1.6.5.1 Packet Receive Filtering
If the VIRTIO_NET_F_CTRL_RX and VIRTIO_NET_F_CTRL_RX_EXTRA features are negotiated, the driver can send control commands for promiscuous mode, multicast, unicast and broadcast receiving. Note: In general, these commands are best-effort: unwanted packets could still arrive.
#define VIRTIO_NET_CTRL_RX    0 
 #define VIRTIO_NET_CTRL_RX_PROMISC      0 
 #define VIRTIO_NET_CTRL_RX_ALLMULTI     1 
 #define VIRTIO_NET_CTRL_RX_ALLUNI       2 
 #define VIRTIO_NET_CTRL_RX_NOMULTI      3 
 #define VIRTIO_NET_CTRL_RX_NOUNI        4 
 #define VIRTIO_NET_CTRL_RX_NOBCAST      5

5.1.6.5.1.1 Device Requirements: Packet Receive Filtering
If the VIRTIO_NET_F_CTRL_RX feature has been negotiated, the device MUST support the following VIRTIO_NET_CTRL_RX class commands:

If the VIRTIO_NET_F_CTRL_RX_EXTRA feature has been negotiated, the device MUST support the following VIRTIO_NET_CTRL_RX class commands:

5.1.6.5.1.2 Driver Requirements: Packet Receive Filtering
If the VIRTIO_NET_F_CTRL_RX feature has not been negotiated, the driver MUST NOT issue commands VIRTIO_NET_CTRL_RX_PROMISC or VIRTIO_NET_CTRL_RX_ALLMULTI.

If the VIRTIO_NET_F_CTRL_RX_EXTRA feature has not been negotiated, the driver MUST NOT issue commands VIRTIO_NET_CTRL_RX_ALLUNI, VIRTIO_NET_CTRL_RX_NOMULTI, VIRTIO_NET_CTRL_RX_NOUNI or VIRTIO_NET_CTRL_RX_NOBCAST.

5.1.6.5.2 Setting MAC Address Filtering
If the VIRTIO_NET_F_CTRL_RX feature is negotiated, the driver can send control commands for MAC address filtering.
struct virtio_net_ctrl_mac { 
        le32 entries; 
        u8 macs[entries][6]; 
}; 
 
#define VIRTIO_NET_CTRL_MAC    1 
 #define VIRTIO_NET_CTRL_MAC_TABLE_SET        0 
 #define VIRTIO_NET_CTRL_MAC_ADDR_SET         1

The device can filter incoming packets by any number of destination MAC addresses12. This table is set using the class VIRTIO_NET_CTRL_MAC and the command VIRTIO_NET_CTRL_MAC_TABLE_SET. The command-specific-data is two variable length tables of 6-byte MAC addresses (as described in struct virtio_net_ctrl_mac). The first table contains unicast addresses, and the second contains multicast addresses.

The VIRTIO_NET_CTRL_MAC_ADDR_SET command is used to set the default MAC address which rx filtering accepts (and if VIRTIO_NET_F_MAC has been negotiated, this will be reflected in mac in config space).

The command-specific-data for VIRTIO_NET_CTRL_MAC_ADDR_SET is the 6-byte MAC address.

5.1.6.5.2.1 Device Requirements: Setting MAC Address Filtering
The device MUST have an empty MAC filtering table on reset.

The device MUST update the MAC filtering table before it consumes the VIRTIO_NET_CTRL_MAC_TABLE_SET command.

The device MUST update mac in config space before it consumes the VIRTIO_NET_CTRL_MAC_ADDR_SET command, if VIRTIO_NET_F_MAC has been negotiated.

The device SHOULD drop incoming packets which have a destination MAC which matches neither the mac (or that set with VIRTIO_NET_CTRL_MAC_ADDR_SET) nor the MAC filtering table.

5.1.6.5.2.2 Driver Requirements: Setting MAC Address Filtering
If VIRTIO_NET_F_CTRL_RX has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_MAC class commands.

If VIRTIO_NET_F_CTRL_RX has been negotiated, the driver SHOULD issue VIRTIO_NET_CTRL_MAC_ADDR_SET to set the default mac if it is different from mac.

The driver MUST follow the VIRTIO_NET_CTRL_MAC_TABLE_SET command by a le32 number, followed by that number of non-multicast MAC addresses, followed by another le32 number, followed by that number of multicast addresses. Either number MAY be 0.

5.1.6.5.2.3 Legacy Interface: Setting MAC Address Filtering
When using the legacy interface, transitional devices and drivers MUST format entries in struct virtio_net_ctrl_mac according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.

Legacy drivers that didn’t negotiate VIRTIO_NET_F_CTRL_MAC_ADDR changed mac in config space when NIC is accepting incoming packets. These drivers always wrote the mac value from first to last byte, therefore after detecting such drivers, a transitional device MAY defer MAC update, or MAY defer processing incoming packets until driver writes the last byte of mac in the config space.

5.1.6.5.3 VLAN Filtering
If the driver negotiates the VIRTIO_NET_F_CTRL_VLAN feature, it can control a VLAN filter table in the device. The VLAN filter table applies only to VLAN tagged packets.

When VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with an empty VLAN filter table. Note: Similar to the MAC address based filtering, the VLAN filtering is also best-effort: unwanted packets could still arrive.

#define VIRTIO_NET_CTRL_VLAN       2 
 #define VIRTIO_NET_CTRL_VLAN_ADD             0 
 #define VIRTIO_NET_CTRL_VLAN_DEL             1

Both the VIRTIO_NET_CTRL_VLAN_ADD and VIRTIO_NET_CTRL_VLAN_DEL command take a little-endian 16-bit VLAN id as the command-specific-data.

VIRTIO_NET_CTRL_VLAN_ADD command adds the specified VLAN to the VLAN filter table.

VIRTIO_NET_CTRL_VLAN_DEL command removes the specified VLAN from the VLAN filter table.

5.1.6.5.3.1 Device Requirements: VLAN Filtering
When VIRTIO_NET_F_CTRL_VLAN is not negotiated, the device MUST accept all VLAN tagged packets.

When VIRTIO_NET_F_CTRL_VLAN is negotiated, the device MUST accept all VLAN tagged packets whose VLAN tag is present in the VLAN filter table and SHOULD drop all VLAN tagged packets whose VLAN tag is absent in the VLAN filter table.

5.1.6.5.3.2 Legacy Interface: VLAN Filtering
When using the legacy interface, transitional devices and drivers MUST format the VLAN id according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.

5.1.6.5.4 Gratuitous Packet Sending
If the driver negotiates the VIRTIO_NET_F_GUEST_ANNOUNCE (depends on VIRTIO_NET_F_CTRL_VQ), the device can ask the driver to send gratuitous packets; this is usually done after the guest has been physically migrated, and needs to announce its presence on the new network links. (As hypervisor does not have the knowledge of guest network configuration (eg. tagged vlan) it is simplest to prod the guest in this way).
#define VIRTIO_NET_CTRL_ANNOUNCE       3 
 #define VIRTIO_NET_CTRL_ANNOUNCE_ACK             0

The driver checks VIRTIO_NET_S_ANNOUNCE bit in the device configuration status field when it notices the changes of device configuration. The command VIRTIO_NET_CTRL_ANNOUNCE_ACK is used to indicate that driver has received the notification and device clears the VIRTIO_NET_S_ANNOUNCE bit in status.

Processing this notification involves:

1.
Sending the gratuitous packets (eg. ARP) or marking there are pending gratuitous packets to be sent and letting deferred routine to send them.
2.
Sending VIRTIO_NET_CTRL_ANNOUNCE_ACK command through control vq.

5.1.6.5.4.1 Driver Requirements: Gratuitous Packet Sending
If the driver negotiates VIRTIO_NET_F_GUEST_ANNOUNCE, it SHOULD notify network peers of its new location after it sees the VIRTIO_NET_S_ANNOUNCE bit in status. The driver MUST send a command on the command queue with class VIRTIO_NET_CTRL_ANNOUNCE and command VIRTIO_NET_CTRL_ANNOUNCE_ACK.

5.1.6.5.4.2 Device Requirements: Gratuitous Packet Sending
If VIRTIO_NET_F_GUEST_ANNOUNCE is negotiated, the device MUST clear the VIRTIO_NET_S_ANNOUNCE bit in status upon receipt of a command buffer with class VIRTIO_NET_CTRL_ANNOUNCE and command VIRTIO_NET_CTRL_ANNOUNCE_ACK before marking the buffer as used.

5.1.6.5.5 Device operation in multiqueue mode
This specification defines the following modes that a device MAY implement for operation with multiple transmit/receive virtqueues:

A device MAY support one of these features or both. The driver MAY negotiate any set of these features that the device supports.

Multiqueue is disabled by default.

The driver enables multiqueue by sending a command using class VIRTIO_NET_CTRL_MQ. The command selects the mode of multiqueue operation, as follows:

#define VIRTIO_NET_CTRL_MQ    4 
 #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET        0 (for automatic receive steering) 
 #define VIRTIO_NET_CTRL_MQ_RSS_CONFIG          1 (for configurable receive steering) 
 #define VIRTIO_NET_CTRL_MQ_HASH_CONFIG         2 (for configurable hash calculation)

If more than one multiqueue mode is negotiated, the resulting device configuration is defined by the last command sent by the driver.

5.1.6.5.6 Automatic receive steering in multiqueue mode
If the driver negotiates the VIRTIO_NET_F_MQ feature bit (depends on VIRTIO_NET_F_CTRL_VQ), it MAY transmit outgoing packets on one of the multiple transmitq1…transmitqN and ask the device to queue incoming packets into one of the multiple receiveq1…receiveqN depending on the packet flow.

The driver enables multiqueue by sending the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command, specifying the number of the transmit and receive queues to be used up to max_virtqueue_pairs; subsequently, transmitq1…transmitqn and receiveq1…receiveqn where n=virtqueue_pairs MAY be used.

struct virtio_net_ctrl_mq_pairs_set { 
       le16 virtqueue_pairs; 
}; 
#define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MIN        1 
#define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX        0x8000

When multiqueue is enabled by VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command, the device MUST use automatic receive steering based on packet flow. Programming of the receive steering classificator is implicit. After the driver transmitted a packet of a flow on transmitqX, the device SHOULD cause incoming packets for that flow to be steered to receiveqX. For uni-directional protocols, or where no packets have been transmitted yet, the device MAY steer a packet to a random queue out of the specified receiveq1…receiveqn.

Multiqueue is disabled by VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET with virtqueue_pairs to 1 (this is the default) and waiting for the device to use the command buffer.

5.1.6.5.6.1 Driver Requirements: Automatic receive steering in multiqueue mode
The driver MUST configure the virtqueues before enabling them with the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command.

The driver MUST NOT request a virtqueue_pairs of 0 or greater than max_virtqueue_pairs in the device configuration space.

The driver MUST queue packets only on any transmitq1 before the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command.

The driver MUST NOT queue packets on transmit queues greater than virtqueue_pairs once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in the available ring.

5.1.6.5.6.2 Device Requirements: Automatic receive steering in multiqueue mode
After initialization of reset, the device MUST queue packets only on receiveq1.

The device MUST NOT queue packets on receive queues greater than virtqueue_pairs once it has placed the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command in a used buffer.

If the destination receive queue is being reset (See 2.6.1), the device SHOULD re-select another random queue. If all receive queues are being reset, the device MUST drop the packet.

5.1.6.5.6.3 Legacy Interface: Automatic receive steering in multiqueue mode
When using the legacy interface, transitional devices and drivers MUST format virtqueue_pairs according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.

5.1.6.5.6.4 Hash calculation
If VIRTIO_NET_F_HASH_REPORT was negotiated and the device uses automatic receive steering, the device MUST support a command to configure hash calculation parameters.

The driver provides parameters for hash calculation as follows:

class VIRTIO_NET_CTRL_MQ, command VIRTIO_NET_CTRL_MQ_HASH_CONFIG.

The command-specific-data has following format:

struct virtio_net_hash_config { 
    le32 hash_types; 
    le16 reserved[4]; 
    u8 hash_key_length; 
    u8 hash_key_data[hash_key_length]; 
};

Field hash_types contains a bitmask of allowed hash types as defined in 5.1.6.4.3.1. Initially the device has all hash types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.

Field reserved MUST contain zeroes. It is defined to make the structure to match the layout of virtio_net_rss_config structure, defined in 5.1.6.5.7.

Fields hash_key_length and hash_key_data define the key to be used in hash calculation.

5.1.6.5.7 Receive-side scaling (RSS)
A device offers the feature VIRTIO_NET_F_RSS if it supports RSS receive steering with Toeplitz hash calculation and configurable parameters.

A driver queries RSS capabilities of the device by reading device configuration as defined in 5.1.4

5.1.6.5.7.1 Setting RSS parameters
Driver sends a VIRTIO_NET_CTRL_MQ_RSS_CONFIG command using the following format for command-specific-data:
struct rss_rq_id { 
   le16 vq_index_1_16: 15; /* Bits 1 to 16 of the virtqueue index */ 
   le16 reserved: 1; /* Set to zero */ 
}; 
 
struct virtio_net_rss_config { 
    le32 hash_types; 
    le16 indirection_table_mask; 
    struct rss_rq_id unclassified_queue; 
    struct rss_rq_id indirection_table[indirection_table_length]; 
    le16 max_tx_vq; 
    u8 hash_key_length; 
    u8 hash_key_data[hash_key_length]; 
};

Field hash_types contains a bitmask of allowed hash types as defined in 5.1.6.4.3.1.

Field indirection_table_mask is a mask to be applied to the calculated hash to produce an index in the indirection_table array. Number of entries in indirection_table is (indirection_table_mask + 1).

rss_rq_id is a receive virtqueue id. vq_index_1_16 consists of bits 1 to 16 of a virtqueue index. For example, a vq_index_1_16 value of 3 corresponds to virtqueue index 6, which maps to receiveq4.

Field unclassified_queue specifies the receive virtqueue id in which to place unclassified packets.

Field indirection_table is an array of receive virtqueues ids.

A driver sets max_tx_vq to inform a device how many transmit virtqueues it may use (transmitq1…transmitq max_tx_vq).

Fields hash_key_length and hash_key_data define the key to be used in hash calculation.

5.1.6.5.7.2 Driver Requirements: Setting RSS parameters
A driver MUST NOT send the VIRTIO_NET_CTRL_MQ_RSS_CONFIG command if the feature VIRTIO_NET_F_RSS has not been negotiated.

A driver MUST fill the indirection_table array only with enabled receive virtqueues ids.

The number of entries in indirection_table (indirection_table_mask + 1) MUST be a power of two.

A driver MUST use indirection_table_mask values that are less than rss_max_indirection_table_length reported by a device.

A driver MUST NOT set any VIRTIO_NET_HASH_TYPE_ flags that are not supported by a device.

5.1.6.5.7.3 Device Requirements: RSS processing
The device MUST determine the destination queue for a network packet as follows:

5.1.6.5.8 Offloads State Configuration
If the VIRTIO_NET_F_CTRL_GUEST_OFFLOADS feature is negotiated, the driver can send control commands for dynamic offloads state configuration.

5.1.6.5.8.1 Setting Offloads State
To configure the offloads, the following layout structure and definitions are used:
le64 offloads; 
 
#define VIRTIO_NET_F_GUEST_CSUM       1 
#define VIRTIO_NET_F_GUEST_TSO4       7 
#define VIRTIO_NET_F_GUEST_TSO6       8 
#define VIRTIO_NET_F_GUEST_ECN        9 
#define VIRTIO_NET_F_GUEST_UFO        10 
#define VIRTIO_NET_F_GUEST_USO4       54 
#define VIRTIO_NET_F_GUEST_USO6       55 
 
#define VIRTIO_NET_CTRL_GUEST_OFFLOADS       5 
 #define VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET   0

The class VIRTIO_NET_CTRL_GUEST_OFFLOADS has one command: VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET applies the new offloads configuration.

le64 value passed as command data is a bitmask, bits set define offloads to be enabled, bits cleared - offloads to be disabled.

There is a corresponding device feature for each offload. Upon feature negotiation corresponding offload gets enabled to preserve backward compatibility.

5.1.6.5.8.2 Driver Requirements: Setting Offloads State
A driver MUST NOT enable an offload for which the appropriate feature has not been negotiated.

5.1.6.5.8.3 Legacy Interface: Setting Offloads State
When using the legacy interface, transitional devices and drivers MUST format offloads according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.

5.1.6.5.9 Notifications Coalescing
If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET for notification coalescing.

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

struct virtio_net_ctrl_coal { 
    le32 max_packets; 
    le32 max_usecs; 
}; 
 
struct virtio_net_ctrl_coal_vq { 
    le16 vq_index; 
    le16 reserved; 
    struct virtio_net_ctrl_coal coal; 
}; 
 
#define VIRTIO_NET_CTRL_NOTF_COAL 6 
 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0 
 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 
 #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 
 #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3

Coalescing parameters:

reserved is reserved and it is ignored by the device.

Read/Write attributes for coalescing parameters:

The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:

1.
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: use the structure virtio_net_ctrl_coal to set the max_usecs and max_packets parameters for all transmit virtqueues.
2.
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: use the structure virtio_net_ctrl_coal to set the max_usecs and max_packets parameters for all receive virtqueues.
3.
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: use the structure virtio_net_ctrl_coal_vq to set the max_usecs and max_packets parameters for an enabled transmit/receive virtqueue whose index is vq_index.
4.
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the max_usecs and max_packets parameters for an enabled transmit/receive virtqueue whose index is vq_index.

The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

If coalescing parameters are being set, the device applies the last coalescing parameters set for a virtqueue, regardless of the command used to set the parameters. Use the following command sequence with two pairs of virtqueues as an example: Each of the following commands sets max_usecs and max_packets parameters for virtqueues.

5.1.6.5.9.1 Operation
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.

When the device has non-zero max_usecs and non-zero max_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met when max_usecs microseconds elapse, or upon sending/receiving max_packets packets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.

When the device has max_usecs = 0 or max_packets = 0, the notification conditions are met after every packet received/sent.

5.1.6.5.9.2 RX Example
If, for example:

then each receive virtqueue of a device will operate as follows:

5.1.6.5.9.3 TX Example
If, for example:

then each transmit virtqueue of a device will operate as follows:

5.1.6.5.9.4 Notifications When Coalescing Parameters Change
When the coalescing parameters of a device change, the device needs to check if the new notification conditions are met and send a used buffer notification if so.

For example, max_packets = 15 for a device with a single transmit virtqueue: if the device sends 10 packets and afterwards receives a VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command with max_packets = 8, then the notification condition is immediately considered to be met; the device needs to immediately send a used buffer notification, if the notifications are not suppressed by the driver.

5.1.6.5.9.5 Driver Requirements: Notifications Coalescing
The driver MUST set vq_index to the virtqueue index of an enabled transmit or receive virtqueue.

The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.

5.1.6.5.9.6 Device Requirements: Notifications Coalescing
The device MUST ignore reserved.

The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.

The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.

Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.

The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort: the device MAY generate notifications more or less frequently than specified.

A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.

Upon reset, a device MUST initialize all coalescing parameters to 0.

5.1.6.6 Legacy Interface: Framing Requirements

When using legacy interfaces, transitional drivers which have not negotiated VIRTIO_F_ANY_LAYOUT MUST use a single descriptor for the struct virtio_net_hdr on both transmit and receive, with the network data in the following descriptors.

Additionally, when using the control virtqueue (see 5.1.6.5) , transitional drivers which have not negotiated VIRTIO_F_ANY_LAYOUT MUST:

See 2.7.4.

5.2 Block Device

The virtio block device is a simple virtual block device (ie. disk). Read and write requests (and other exotic requests) are placed in one of its queues, and serviced (probably out of order) by the device except where noted.

5.2.1 Device ID

2

5.2.2 Virtqueues

0
requestq1
N-1
requestqN

N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by num_queues.

5.2.3 Feature bits

VIRTIO_BLK_F_SIZE_MAX (1)
Maximum size of any single segment is in size_max.
VIRTIO_BLK_F_SEG_MAX (2)
Maximum number of segments in a request is in seg_max.
VIRTIO_BLK_F_GEOMETRY (4)
Disk-style geometry specified in geometry.
VIRTIO_BLK_F_RO (5)
Device is read-only.
VIRTIO_BLK_F_BLK_SIZE (6)
Block size of disk is in blk_size.
VIRTIO_BLK_F_FLUSH (9)
Cache flush command support.
VIRTIO_BLK_F_TOPOLOGY (10)
Device exports information on optimal I/O alignment.
VIRTIO_BLK_F_CONFIG_WCE (11)
Device can toggle its cache between writeback and writethrough modes.
VIRTIO_BLK_F_MQ (12)
Device supports multiqueue.
VIRTIO_BLK_F_DISCARD (13)
Device can support discard command, maximum discard sectors size in max_discard_sectors and maximum discard segment number in max_discard_seg.
VIRTIO_BLK_F_WRITE_ZEROES (14)
Device can support write zeroes command, maximum write zeroes sectors size in max_write_zeroes_sectors and maximum write zeroes segment number in max_write_zeroes_seg.
VIRTIO_BLK_F_LIFETIME (15)
Device supports providing storage lifetime information.
VIRTIO_BLK_F_SECURE_ERASE (16)
Device supports secure erase command, maximum erase sectors count in max_secure_erase_sectors and maximum erase segment number in max_secure_erase_seg.
VIRTIO_BLK_F_ZONED(17)
Device is a Zoned Block Device, that is, a device that follows the zoned storage device behavior that is also supported by industry standards such as the T10 Zoned Block Command standard (ZBC r05) or the NVMe(TM) NVM Express Zoned Namespace Command Set Specification 1.1b (ZNS). For brevity, these standard documents are referred as "ZBD standards" from this point on in the text.
5.2.3.1 Legacy Interface: Feature bits
VIRTIO_BLK_F_BARRIER (0)
Device supports request barriers.
VIRTIO_BLK_F_SCSI (7)
Device supports scsi packet commands.
Note: In the legacy interface, VIRTIO_BLK_F_FLUSH was also called VIRTIO_BLK_F_WCE.

5.2.4 Device configuration layout

The block device has the following device configuration layout.

struct virtio_blk_config { 
        le64 capacity; 
        le32 size_max; 
        le32 seg_max; 
        struct virtio_blk_geometry { 
                le16 cylinders; 
                u8 heads; 
                u8 sectors; 
        } geometry; 
        le32 blk_size; 
        struct virtio_blk_topology { 
                // # of logical blocks per physical block (log2) 
                u8 physical_block_exp; 
                // offset of first aligned logical block 
                u8 alignment_offset; 
                // suggested minimum I/O size in blocks 
                le16 min_io_size; 
                // optimal (suggested maximum) I/O size in blocks 
                le32 opt_io_size; 
        } topology; 
        u8 writeback; 
        u8 unused0; 
        u16 num_queues; 
        le32 max_discard_sectors; 
        le32 max_discard_seg; 
        le32 discard_sector_alignment; 
        le32 max_write_zeroes_sectors; 
        le32 max_write_zeroes_seg; 
        u8 write_zeroes_may_unmap; 
        u8 unused1[3]; 
        le32 max_secure_erase_sectors; 
        le32 max_secure_erase_seg; 
        le32 secure_erase_sector_alignment; 
        struct virtio_blk_zoned_characteristics { 
                le32 zone_sectors; 
                le32 max_open_zones; 
                le32 max_active_zones; 
                le32 max_append_sectors; 
                le32 write_granularity; 
                u8 model; 
                u8 unused2[3]; 
        } zoned; 
};

The capacity of the device (expressed in 512-byte sectors) is always present. The availability of the others all depend on various feature bits as indicated above.

The field num_queues only exists if VIRTIO_BLK_F_MQ is set. This field specifies the number of queues.

The parameters in the configuration space of the device max_discard_sectors discard_sector_alignment are expressed in 512-byte units if the VIRTIO_BLK_F_DISCARD feature bit is negotiated. The max_write_zeroes_sectors is expressed in 512-byte units if the VIRTIO_BLK_F_WRITE_ZEROES feature bit is negotiated. The parameters in the configuration space of the device max_secure_erase_sectors secure_erase_sector_alignment are expressed in 512-byte units if the VIRTIO_BLK_F_SECURE_ERASE feature bit is negotiated.

If the VIRTIO_BLK_F_ZONED feature is negotiated, then in