Merge tag 'kvm-s390-next-4.20-1' of git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux into HEAD

KVM: s390: Features for 4.20 - Initial version of AP crypto virtualization via vfio-mdev - Set the host program identifier - Optimize page table locking

Paolo Bonzini committed 8 years ago

Commit dd5bd0a2 parents 7e71268 55d09dd16 files · +2397 −63Browse files

Unified · Split

16 files

Documentation/s390/vfio-ap.txt +837 −0
MAINTAINERS +12 −0
arch/s390/Kconfig +11 −0
arch/s390/include/asm/kvm_host.h +12 −1
arch/s390/include/uapi/asm/kvm.h +2 −0
arch/s390/kvm/kvm-s390.c +93 −42
arch/s390/kvm/kvm-s390.h +1 −0
arch/s390/kvm/vsie.c +192 −18
arch/s390/mm/gmap.c +8 −2
arch/s390/tools/gen_facilities.c +2 −0
drivers/iommu/Kconfig +8 −0
drivers/s390/crypto/Makefile +4 −0
drivers/s390/crypto/vfio_ap_drv.c +157 −0
drivers/s390/crypto/vfio_ap_ops.c +968 −0
drivers/s390/crypto/vfio_ap_private.h +88 −0
include/uapi/linux/vfio.h +2 −0

Documentation/s390/vfio-ap.txt+837−0

@@ -0,0 +1,837 @@
	1	Introduction:
	2	============
	3	The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
	4	of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
	5	The AP devices provide cryptographic functions to all CPUs assigned to a
	6	linux system running in an IBM Z system LPAR.
	7
	8	The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
	9	is to make AP cards available to KVM guests using the VFIO mediated device
	10	framework. This implementation relies considerably on the s390 virtualization
	11	facilities which do most of the hard work of providing direct access to AP
	12	devices.
	13
	14	AP Architectural Overview:
	15	=========================
	16	To facilitate the comprehension of the design, let's start with some
	17	definitions:
	18
	19	* AP adapter
	20
	21	An AP adapter is an IBM Z adapter card that can perform cryptographic
	22	functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
	23	assigned to the LPAR in which a linux host is running will be available to
	24	the linux host. Each adapter is identified by a number from 0 to 255; however,
	25	the maximum adapter number is determined by machine model and/or adapter type.
	26	When installed, an AP adapter is accessed by AP instructions executed by any
	27	CPU.
	28
	29	The AP adapter cards are assigned to a given LPAR via the system's Activation
	30	Profile which can be edited via the HMC. When the linux host system is IPL'd
	31	in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
	32	creates a sysfs device for each assigned adapter. For example, if AP adapters
	33	4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
	34	sysfs device entries:
	35
	36	/sys/devices/ap/card04
	37	/sys/devices/ap/card0a
	38
	39	Symbolic links to these devices will also be created in the AP bus devices
	40	sub-directory:
	41
	42	/sys/bus/ap/devices/[card04]
	43	/sys/bus/ap/devices/[card04]
	44
	45	* AP domain
	46
	47	An adapter is partitioned into domains. An adapter can hold up to 256 domains
	48	depending upon the adapter type and hardware configuration. A domain is
	49	identified by a number from 0 to 255; however, the maximum domain number is
	50	determined by machine model and/or adapter type.. A domain can be thought of
	51	as a set of hardware registers and memory used for processing AP commands. A
	52	domain can be configured with a secure private key used for clear key
	53	encryption. A domain is classified in one of two ways depending upon how it
	54	may be accessed:
	55
	56	* Usage domains are domains that are targeted by an AP instruction to
	57	process an AP command.
	58
	59	* Control domains are domains that are changed by an AP command sent to a
	60	usage domain; for example, to set the secure private key for the control
	61	domain.
	62
	63	The AP usage and control domains are assigned to a given LPAR via the system's
	64	Activation Profile which can be edited via the HMC. When a linux host system
	65	is IPL'd in the LPAR, the AP bus module detects the AP usage and control
	66	domains assigned to the LPAR. The domain number of each usage domain and
	67	adapter number of each AP adapter are combined to create AP queue devices
	68	(see AP Queue section below). The domain number of each control domain will be
	69	represented in a bitmask and stored in a sysfs file
	70	/sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least
	71	significant bit, correspond to domains 0-255.
	72
	73	* AP Queue
	74
	75	An AP queue is the means by which an AP command is sent to a usage domain
	76	inside a specific adapter. An AP queue is identified by a tuple
	77	comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
	78	APQI corresponds to a given usage domain number within the adapter. This tuple
	79	forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
	80	instructions include a field containing the APQN to identify the AP queue to
	81	which the AP command is to be sent for processing.
	82
	83	The AP bus will create a sysfs device for each APQN that can be derived from
	84	the cross product of the AP adapter and usage domain numbers detected when the
	85	AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
	86	domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
	87	following sysfs entries:
	88
	89	/sys/devices/ap/card04/04.0006
	90	/sys/devices/ap/card04/04.0047
	91	/sys/devices/ap/card0a/0a.0006
	92	/sys/devices/ap/card0a/0a.0047
	93
	94	The following symbolic links to these devices will be created in the AP bus
	95	devices subdirectory:
	96
	97	/sys/bus/ap/devices/[04.0006]
	98	/sys/bus/ap/devices/[04.0047]
	99	/sys/bus/ap/devices/[0a.0006]
	100	/sys/bus/ap/devices/[0a.0047]
	101
	102	* AP Instructions:
	103
	104	There are three AP instructions:
	105
	106	* NQAP: to enqueue an AP command-request message to a queue
	107	* DQAP: to dequeue an AP command-reply message from a queue
	108	* PQAP: to administer the queues
	109
	110	AP instructions identify the domain that is targeted to process the AP
	111	command; this must be one of the usage domains. An AP command may modify a
	112	domain that is not one of the usage domains, but the modified domain
	113	must be one of the control domains.
	114
	115	AP and SIE:
	116	==========
	117	Let's now take a look at how AP instructions executed on a guest are interpreted
	118	by the hardware.
	119
	120	A satellite control block called the Crypto Control Block (CRYCB) is attached to
	121	our main hardware virtualization control block. The CRYCB contains three fields
	122	to identify the adapters, usage domains and control domains assigned to the KVM
	123	guest:
	124
	125	* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
	126	to the KVM guest. Each bit in the mask, from left to right (i.e. from most
	127	significant to least significant bit in big endian order), corresponds to
	128	an APID from 0-255. If a bit is set, the corresponding adapter is valid for
	129	use by the KVM guest.
	130
	131	* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
	132	assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from
	133	most significant to least significant bit in big endian order), corresponds to
	134	an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue
	135	is valid for use by the KVM guest.
	136
	137	* The AP Domain Mask field is a bit mask that identifies the AP control domains
	138	assigned to the KVM guest. The ADM bit mask controls which domains can be
	139	changed by an AP command-request message sent to a usage domain from the
	140	guest. Each bit in the mask, from left to right (i.e. from most significant to
	141	least significant bit in big endian order), corresponds to a domain from
	142	0-255. If a bit is set, the corresponding domain can be modified by an AP
	143	command-request message sent to a usage domain.
	144
	145	If you recall from the description of an AP Queue, AP instructions include
	146	an APQN to identify the AP queue to which an AP command-request message is to be
	147	sent (NQAP and PQAP instructions), or from which a command-reply message is to
	148	be received (DQAP instruction). The validity of an APQN is defined by the matrix
	149	calculated from the APM and AQM; it is the cross product of all assigned adapter
	150	numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1
	151	and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6),
	152	(2,5) and (2,6) will be valid for the guest.
	153
	154	The APQNs can provide secure key functionality - i.e., a private key is stored
	155	on the adapter card for each of its domains - so each APQN must be assigned to
	156	at most one guest or to the linux host.
	157
	158	Example 1: Valid configuration:
	159	------------------------------
	160	Guest1: adapters 1,2 domains 5,6
	161	Guest2: adapter 1,2 domain 7
	162
	163	This is valid because both guests have a unique set of APQNs:
	164	Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
	165	Guest2 has APQNs (1,7), (2,7)
	166
	167	Example 2: Valid configuration:
	168	------------------------------
	169	Guest1: adapters 1,2 domains 5,6
	170	Guest2: adapters 3,4 domains 5,6
	171
	172	This is also valid because both guests have a unique set of APQNs:
	173	Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
	174	Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
	175
	176	Example 3: Invalid configuration:
	177	--------------------------------
	178	Guest1: adapters 1,2 domains 5,6
	179	Guest2: adapter 1 domains 6,7
	180
	181	This is an invalid configuration because both guests have access to
	182	APQN (1,6).
	183
	184	The Design:
	185	===========
	186	The design introduces three new objects:
	187
	188	1. AP matrix device
	189	2. VFIO AP device driver (vfio_ap.ko)
	190	3. VFIO AP mediated matrix pass-through device
	191
	192	The VFIO AP device driver
	193	-------------------------
	194	The VFIO AP (vfio_ap) device driver serves the following purposes:
	195
	196	1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.
	197
	198	2. Sets up the VFIO mediated device interfaces to manage a mediated matrix
	199	device and creates the sysfs interfaces for assigning adapters, usage
	200	domains, and control domains comprising the matrix for a KVM guest.
	201
	202	3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's
	203	SIE state description to grant the guest access to a matrix of AP devices
	204
	205	Reserve APQNs for exclusive use of KVM guests
	206	---------------------------------------------
	207	The following block diagram illustrates the mechanism by which APQNs are
	208	reserved:
	209
	210	+------------------+
	211	7 remove \| \|
	212	+--------------------> cex4queue driver \|
	213	\| \| \|
	214	\| +------------------+
	215	\|
	216	\|
	217	\| +------------------+ +-----------------+
	218	\| 5 register driver \| \| 3 create \| \|
	219	\| +----------------> Device core +----------> matrix device \|
	220	\| \| \| \| \| \|
	221	\| \| +--------^---------+ +-----------------+
	222	\| \| \|
	223	\| \| +-------------------+
	224	\| \| +-----------------------------------+ \|
	225	\| \| \| 4 register AP driver \| \| 2 register device
	226	\| \| \| \| \|
	227	+--------+---+-v---+ +--------+-------+-+
	228	\| \| \| \|
	229	\| ap_bus +--------------------- > vfio_ap driver \|
	230	\| \| 8 probe \| \|
	231	+--------^---------+ +--^--^------------+
	232	6 edit \| \| \|
	233	apmask \| +-----------------------------+ \| 9 mdev create
	234	aqmask \| \| 1 modprobe \|
	235	+--------+-----+---+ +----------------+-+ +------------------+
	236	\| \| \| \|8 create \| mediated \|
	237	\| admin \| \| VFIO device core \|---------> matrix \|
	238	\| + \| \| \| device \|
	239	+------+-+---------+ +--------^---------+ +--------^---------+
	240	\| \| \| \|
	241	\| \| 9 create vfio_ap-passthrough \| \|
	242	\| +------------------------------+ \|
	243	+-------------------------------------------------------------+
	244	10 assign adapter/domain/control domain
	245
	246	The process for reserving an AP queue for use by a KVM guest is:
	247
	248	1. The administrator loads the vfio_ap device driver
	249	2. The vfio-ap driver during its initialization will register a single 'matrix'
	250	device with the device core. This will serve as the parent device for
	251	all mediated matrix devices used to configure an AP matrix for a guest.
	252	3. The /sys/devices/vfio_ap/matrix device is created by the device core
	253	4 The vfio_ap device driver will register with the AP bus for AP queue devices
	254	of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
	255	driver's probe and remove callback interfaces. Devices older than CEX4 queues
	256	are not supported to simplify the implementation by not needlessly
	257	complicating the design by supporting older devices that will go out of
	258	service in the relatively near future, and for which there are few older
	259	systems around on which to test.
	260	5. The AP bus registers the vfio_ap device driver with the device core
	261	6. The administrator edits the AP adapter and queue masks to reserve AP queues
	262	for use by the vfio_ap device driver.
	263	7. The AP bus removes the AP queues reserved for the vfio_ap driver from the
	264	default zcrypt cex4queue driver.
	265	8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
	266	it.
	267	9. The administrator creates a passthrough type mediated matrix device to be
	268	used by a guest
	269	10 The administrator assigns the adapters, usage domains and control domains
	270	to be exclusively used by a guest.
	271
	272	Set up the VFIO mediated device interfaces
	273	------------------------------------------
	274	The VFIO AP device driver utilizes the common interface of the VFIO mediated
	275	device core driver to:
	276	* Register an AP mediated bus driver to add a mediated matrix device to and
	277	remove it from a VFIO group.
	278	* Create and destroy a mediated matrix device
	279	* Add a mediated matrix device to and remove it from the AP mediated bus driver
	280	* Add a mediated matrix device to and remove it from an IOMMU group
	281
	282	The following high-level block diagram shows the main components and interfaces
	283	of the VFIO AP mediated matrix device driver:
	284
	285	+-------------+
	286	\| \|
	287	\| +---------+ \| mdev_register_driver() +--------------+
	288	\| \| Mdev \| +<-----------------------+ \|
	289	\| \| bus \| \| \| vfio_mdev.ko \|
	290	\| \| driver \| +----------------------->+ \|<-> VFIO user
	291	\| +---------+ \| probe()/remove() +--------------+ APIs
	292	\| \|
	293	\| MDEV CORE \|
	294	\| MODULE \|
	295	\| mdev.ko \|
	296	\| +---------+ \| mdev_register_device() +--------------+
	297	\| \|Physical \| +<-----------------------+ \|
	298	\| \| device \| \| \| vfio_ap.ko \|<-> matrix
	299	\| \|interface\| +----------------------->+ \| device
	300	\| +---------+ \| callback +--------------+
	301	+-------------+
	302
	303	During initialization of the vfio_ap module, the matrix device is registered
	304	with an 'mdev_parent_ops' structure that provides the sysfs attribute
	305	structures, mdev functions and callback interfaces for managing the mediated
	306	matrix device.
	307
	308	* sysfs attribute structures:
	309	* supported_type_groups
	310	The VFIO mediated device framework supports creation of user-defined
	311	mediated device types. These mediated device types are specified
	312	via the 'supported_type_groups' structure when a device is registered
	313	with the mediated device framework. The registration process creates the
	314	sysfs structures for each mediated device type specified in the
	315	'mdev_supported_types' sub-directory of the device being registered. Along
	316	with the device type, the sysfs attributes of the mediated device type are
	317	provided.
	318
	319	The VFIO AP device driver will register one mediated device type for
	320	passthrough devices:
	321	/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough
	322	Only the read-only attributes required by the VFIO mdev framework will
	323	be provided:
	324	... name
	325	... device_api
	326	... available_instances
	327	... device_api
	328	Where:
	329	* name: specifies the name of the mediated device type
	330	* device_api: the mediated device type's API
	331	* available_instances: the number of mediated matrix passthrough devices
	332	that can be created
	333	* device_api: specifies the VFIO API
	334	* mdev_attr_groups
	335	This attribute group identifies the user-defined sysfs attributes of the
	336	mediated device. When a device is registered with the VFIO mediated device
	337	framework, the sysfs attribute files identified in the 'mdev_attr_groups'
	338	structure will be created in the mediated matrix device's directory. The
	339	sysfs attributes for a mediated matrix device are:
	340	* assign_adapter:
	341	* unassign_adapter:
	342	Write-only attributes for assigning/unassigning an AP adapter to/from the
	343	mediated matrix device. To assign/unassign an adapter, the APID of the
	344	adapter is echoed to the respective attribute file.
	345	* assign_domain:
	346	* unassign_domain:
	347	Write-only attributes for assigning/unassigning an AP usage domain to/from
	348	the mediated matrix device. To assign/unassign a domain, the domain
	349	number of the the usage domain is echoed to the respective attribute
	350	file.
	351	* matrix:
	352	A read-only file for displaying the APQNs derived from the cross product
	353	of the adapter and domain numbers assigned to the mediated matrix device.
	354	* assign_control_domain:
	355	* unassign_control_domain:
	356	Write-only attributes for assigning/unassigning an AP control domain
	357	to/from the mediated matrix device. To assign/unassign a control domain,
	358	the ID of the domain to be assigned/unassigned is echoed to the respective
	359	attribute file.
	360	* control_domains:
	361	A read-only file for displaying the control domain numbers assigned to the
	362	mediated matrix device.
	363
	364	* functions:
	365	* create:
	366	allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
	367	* Store the reference to the KVM structure for the guest using the mdev
	368	* Store the AP matrix configuration for the adapters, domains, and control
	369	domains assigned via the corresponding sysfs attributes files
	370	* remove:
	371	deallocates the mediated matrix device's ap_matrix_mdev structure. This will
	372	be allowed only if a running guest is not using the mdev.
	373
	374	* callback interfaces
	375	* open:
	376	The vfio_ap driver uses this callback to register a
	377	VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix
	378	device. The open is invoked when QEMU connects the VFIO iommu group
	379	for the mdev matrix device to the MDEV bus. Access to the KVM structure used
	380	to configure the KVM guest is provided via this callback. The KVM structure,
	381	is used to configure the guest's access to the AP matrix defined via the
	382	mediated matrix device's sysfs attribute files.
	383	* release:
	384	unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
	385	mdev matrix device and deconfigures the guest's AP matrix.
	386
	387	Configure the APM, AQM and ADM in the CRYCB:
	388	-------------------------------------------
	389	Configuring the AP matrix for a KVM guest will be performed when the
	390	VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
	391	function is called when QEMU connects to KVM. The guest's AP matrix is
	392	configured via it's CRYCB by:
	393	* Setting the bits in the APM corresponding to the APIDs assigned to the
	394	mediated matrix device via its 'assign_adapter' interface.
	395	* Setting the bits in the AQM corresponding to the domains assigned to the
	396	mediated matrix device via its 'assign_domain' interface.
	397	* Setting the bits in the ADM corresponding to the domain dIDs assigned to the
	398	mediated matrix device via its 'assign_control_domains' interface.
	399
	400	The CPU model features for AP
	401	-----------------------------
	402	The AP stack relies on the presence of the AP instructions as well as two
	403	facilities: The AP Facilities Test (APFT) facility; and the AP Query
	404	Configuration Information (QCI) facility. These features/facilities are made
	405	available to a KVM guest via the following CPU model features:
	406
	407	1. ap: Indicates whether the AP instructions are installed on the guest. This
	408	feature will be enabled by KVM only if the AP instructions are installed
	409	on the host.
	410
	411	2. apft: Indicates the APFT facility is available on the guest. This facility
	412	can be made available to the guest only if it is available on the host (i.e.,
	413	facility bit 15 is set).
	414
	415	3. apqci: Indicates the AP QCI facility is available on the guest. This facility
	416	can be made available to the guest only if it is available on the host (i.e.,
	417	facility bit 12 is set).
	418
	419	Note: If the user chooses to specify a CPU model different than the 'host'
	420	model to QEMU, the CPU model features and facilities need to be turned on
	421	explicitly; for example:
	422
	423	/usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on
	424
	425	A guest can be precluded from using AP features/facilities by turning them off
	426	explicitly; for example:
	427
	428	/usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off
	429
	430	Note: If the APFT facility is turned off (apft=off) for the guest, the guest
	431	will not see any AP devices. The zcrypt device drivers that register for type 10
	432	and newer AP devices - i.e., the cex4card and cex4queue device drivers - need
	433	the APFT facility to ascertain the facilities installed on a given AP device. If
	434	the APFT facility is not installed on the guest, then the probe of device
	435	drivers will fail since only type 10 and newer devices can be configured for
	436	guest use.
	437
	438	Example:
	439	=======
	440	Let's now provide an example to illustrate how KVM guests may be given
	441	access to AP facilities. For this example, we will show how to configure
	442	three guests such that executing the lszcrypt command on the guests would
	443	look like this:
	444
	445	Guest1
	446	------
	447	CARD.DOMAIN TYPE MODE
	448	------------------------------
	449	05 CEX5C CCA-Coproc
	450	05.0004 CEX5C CCA-Coproc
	451	05.00ab CEX5C CCA-Coproc
	452	06 CEX5A Accelerator
	453	06.0004 CEX5A Accelerator
	454	06.00ab CEX5C CCA-Coproc
	455
	456	Guest2
	457	------
	458	CARD.DOMAIN TYPE MODE
	459	------------------------------
	460	05 CEX5A Accelerator
	461	05.0047 CEX5A Accelerator
	462	05.00ff CEX5A Accelerator
	463
	464	Guest2
	465	------
	466	CARD.DOMAIN TYPE MODE
	467	------------------------------
	468	06 CEX5A Accelerator
	469	06.0047 CEX5A Accelerator
	470	06.00ff CEX5A Accelerator
	471
	472	These are the steps:
	473
	474	1. Install the vfio_ap module on the linux host. The dependency chain for the
	475	vfio_ap module is:
	476	* iommu
	477	* s390
	478	* zcrypt
	479	* vfio
	480	* vfio_mdev
	481	* vfio_mdev_device
	482	* KVM
	483
	484	To build the vfio_ap module, the kernel build must be configured with the
	485	following Kconfig elements selected:
	486	* IOMMU_SUPPORT
	487	* S390
	488	* ZCRYPT
	489	* S390_AP_IOMMU
	490	* VFIO
	491	* VFIO_MDEV
	492	* VFIO_MDEV_DEVICE
	493	* KVM
	494
	495	If using make menuconfig select the following to build the vfio_ap module:
	496	-> Device Drivers
	497	-> IOMMU Hardware Support
	498	select S390 AP IOMMU Support
	499	-> VFIO Non-Privileged userspace driver framework
	500	-> Mediated device driver frramework
	501	-> VFIO driver for Mediated devices
	502	-> I/O subsystem
	503	-> VFIO support for AP devices
	504
	505	2. Secure the AP queues to be used by the three guests so that the host can not
	506	access them. To secure them, there are two sysfs files that specify
	507	bitmasks marking a subset of the APQN range as 'usable by the default AP
	508	queue device drivers' or 'not usable by the default device drivers' and thus
	509	available for use by the vfio_ap device driver'. The location of the sysfs
	510	files containing the masks are:
	511
	512	/sys/bus/ap/apmask
	513	/sys/bus/ap/aqmask
	514
	515	The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
	516	(APID). Each bit in the mask, from left to right (i.e., from most significant
	517	to least significant bit in big endian order), corresponds to an APID from
	518	0-255. If a bit is set, the APID is marked as usable only by the default AP
	519	queue device drivers; otherwise, the APID is usable by the vfio_ap
	520	device driver.
	521
	522	The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
	523	(APQI). Each bit in the mask, from left to right (i.e., from most significant
	524	to least significant bit in big endian order), corresponds to an APQI from
	525	0-255. If a bit is set, the APQI is marked as usable only by the default AP
	526	queue device drivers; otherwise, the APQI is usable by the vfio_ap device
	527	driver.
	528
	529	Take, for example, the following mask:
	530
	531	0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
	532
	533	It indicates:
	534
	535	1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
	536	belong to the vfio_ap device driver's pool.
	537
	538	The APQN of each AP queue device assigned to the linux host is checked by the
	539	AP bus against the set of APQNs derived from the cross product of APIDs
	540	and APQIs marked as usable only by the default AP queue device drivers. If a
	541	match is detected, only the default AP queue device drivers will be probed;
	542	otherwise, the vfio_ap device driver will be probed.
	543
	544	By default, the two masks are set to reserve all APQNs for use by the default
	545	AP queue device drivers. There are two ways the default masks can be changed:
	546
	547	1. The sysfs mask files can be edited by echoing a string into the
	548	respective sysfs mask file in one of two formats:
	549
	550	* An absolute hex string starting with 0x - like "0x12345678" - sets
	551	the mask. If the given string is shorter than the mask, it is padded
	552	with 0s on the right; for example, specifying a mask value of 0x41 is
	553	the same as specifying:
	554
	555	0x4100000000000000000000000000000000000000000000000000000000000000
	556
	557	Keep in mind that the mask reads from left to right (i.e., most
	558	significant to least significant bit in big endian order), so the mask
	559	above identifies device numbers 1 and 7 (01000001).
	560
	561	If the string is longer than the mask, the operation is terminated with
	562	an error (EINVAL).
	563
	564	* Individual bits in the mask can be switched on and off by specifying
	565	each bit number to be switched in a comma separated list. Each bit
	566	number string must be prepended with a ('+') or minus ('-') to indicate
	567	the corresponding bit is to be switched on ('+') or off ('-'). Some
	568	valid values are:
	569
	570	"+0" switches bit 0 on
	571	"-13" switches bit 13 off
	572	"+0x41" switches bit 65 on
	573	"-0xff" switches bit 255 off
	574
	575	The following example:
	576	+0,-6,+0x47,-0xf0
	577
	578	Switches bits 0 and 71 (0x47) on
	579	Switches bits 6 and 240 (0xf0) off
	580
	581	Note that the bits not specified in the list remain as they were before
	582	the operation.
	583
	584	2. The masks can also be changed at boot time via parameters on the kernel
	585	command line like this:
	586
	587	ap.apmask=0xffff ap.aqmask=0x40
	588
	589	This would create the following masks:
	590
	591	apmask:
	592	0xffff000000000000000000000000000000000000000000000000000000000000
	593
	594	aqmask:
	595	0x4000000000000000000000000000000000000000000000000000000000000000
	596
	597	Resulting in these two pools:
	598
	599	default drivers pool: adapter 0-15, domain 1
	600	alternate drivers pool: adapter 16-255, domains 0, 2-255
	601
	602	Securing the APQNs for our example:
	603	----------------------------------
	604	To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
	605	06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
	606	APQNs can either be removed from the default masks:
	607
	608	echo -5,-6 > /sys/bus/ap/apmask
	609
	610	echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
	611
	612	Or the masks can be set as follows:
	613
	614	echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
	615	> apmask
	616
	617	echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
	618	> aqmask
	619
	620	This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
	621	06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
	622	sysfs directory for the vfio_ap device driver will now contain symbolic links
	623	to the AP queue devices bound to it:
	624
	625	/sys/bus/ap
	626	... [drivers]
	627	...... [vfio_ap]
	628	......... [05.0004]
	629	......... [05.0047]
	630	......... [05.00ab]
	631	......... [05.00ff]
	632	......... [06.0004]
	633	......... [06.0047]
	634	......... [06.00ab]
	635	......... [06.00ff]
	636
	637	Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
	638	can be bound to the vfio_ap device driver. The reason for this is to
	639	simplify the implementation by not needlessly complicating the design by
	640	supporting older devices that will go out of service in the relatively near
	641	future and for which there are few older systems on which to test.
	642
	643	The administrator, therefore, must take care to secure only AP queues that
	644	can be bound to the vfio_ap device driver. The device type for a given AP
	645	queue device can be read from the parent card's sysfs directory. For example,
	646	to see the hardware type of the queue 05.0004:
	647
	648	cat /sys/bus/ap/devices/card05/hwtype
	649
	650	The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
	651	vfio_ap device driver.
	652
	653	3. Create the mediated devices needed to configure the AP matrixes for the
	654	three guests and to provide an interface to the vfio_ap driver for
	655	use by the guests:
	656
	657	/sys/devices/vfio_ap/matrix/
	658	--- [mdev_supported_types]
	659	------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
	660	--------- create
	661	--------- [devices]
	662
	663	To create the mediated devices for the three guests:
	664
	665	uuidgen > create
	666	uuidgen > create
	667	uuidgen > create
	668
	669	or
	670
	671	echo $uuid1 > create
	672	echo $uuid2 > create
	673	echo $uuid3 > create
	674
	675	This will create three mediated devices in the [devices] subdirectory named
	676	after the UUID written to the create attribute file. We call them $uuid1,
	677	$uuid2 and $uuid3 and this is the sysfs directory structure after creation:
	678
	679	/sys/devices/vfio_ap/matrix/
	680	--- [mdev_supported_types]
	681	------ [vfio_ap-passthrough]
	682	--------- [devices]
	683	------------ [$uuid1]
	684	--------------- assign_adapter
	685	--------------- assign_control_domain
	686	--------------- assign_domain
	687	--------------- matrix
	688	--------------- unassign_adapter
	689	--------------- unassign_control_domain
	690	--------------- unassign_domain
	691
	692	------------ [$uuid2]
	693	--------------- assign_adapter
	694	--------------- assign_control_domain
	695	--------------- assign_domain
	696	--------------- matrix
	697	--------------- unassign_adapter
	698	----------------unassign_control_domain
	699	----------------unassign_domain
	700
	701	------------ [$uuid3]
	702	--------------- assign_adapter
	703	--------------- assign_control_domain
	704	--------------- assign_domain
	705	--------------- matrix
	706	--------------- unassign_adapter
	707	----------------unassign_control_domain
	708	----------------unassign_domain
	709
	710	4. The administrator now needs to configure the matrixes for the mediated
	711	devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
	712
	713	This is how the matrix is configured for Guest1:
	714
	715	echo 5 > assign_adapter
	716	echo 6 > assign_adapter
	717	echo 4 > assign_domain
	718	echo 0xab > assign_domain
	719
	720	Control domains can similarly be assigned using the assign_control_domain
	721	sysfs file.
	722
	723	If a mistake is made configuring an adapter, domain or control domain,
	724	you can use the unassign_xxx files to unassign the adapter, domain or
	725	control domain.
	726
	727	To display the matrix configuration for Guest1:
	728
	729	cat matrix
	730
	731	This is how the matrix is configured for Guest2:
	732
	733	echo 5 > assign_adapter
	734	echo 0x47 > assign_domain
	735	echo 0xff > assign_domain
	736
	737	This is how the matrix is configured for Guest3:
	738
	739	echo 6 > assign_adapter
	740	echo 0x47 > assign_domain
	741	echo 0xff > assign_domain
	742
	743	In order to successfully assign an adapter:
	744
	745	* The adapter number specified must represent a value from 0 up to the
	746	maximum adapter number configured for the system. If an adapter number
	747	higher than the maximum is specified, the operation will terminate with
	748	an error (ENODEV).
	749
	750	* All APQNs that can be derived from the adapter ID and the IDs of
	751	the previously assigned domains must be bound to the vfio_ap device
	752	driver. If no domains have yet been assigned, then there must be at least
	753	one APQN with the specified APID bound to the vfio_ap driver. If no such
	754	APQNs are bound to the driver, the operation will terminate with an
	755	error (EADDRNOTAVAIL).
	756
	757	No APQN that can be derived from the adapter ID and the IDs of the
	758	previously assigned domains can be assigned to another mediated matrix
	759	device. If an APQN is assigned to another mediated matrix device, the
	760	operation will terminate with an error (EADDRINUSE).
	761
	762	In order to successfully assign a domain:
	763
	764	* The domain number specified must represent a value from 0 up to the
	765	maximum domain number configured for the system. If a domain number
	766	higher than the maximum is specified, the operation will terminate with
	767	an error (ENODEV).
	768
	769	* All APQNs that can be derived from the domain ID and the IDs of
	770	the previously assigned adapters must be bound to the vfio_ap device
	771	driver. If no domains have yet been assigned, then there must be at least
	772	one APQN with the specified APQI bound to the vfio_ap driver. If no such
	773	APQNs are bound to the driver, the operation will terminate with an
	774	error (EADDRNOTAVAIL).
	775
	776	No APQN that can be derived from the domain ID and the IDs of the
	777	previously assigned adapters can be assigned to another mediated matrix
	778	device. If an APQN is assigned to another mediated matrix device, the
	779	operation will terminate with an error (EADDRINUSE).
	780
	781	In order to successfully assign a control domain, the domain number
	782	specified must represent a value from 0 up to the maximum domain number
	783	configured for the system. If a control domain number higher than the maximum
	784	is specified, the operation will terminate with an error (ENODEV).
	785
	786	5. Start Guest1:
	787
	788	/usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
	789	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
	790
	791	7. Start Guest2:
	792
	793	/usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
	794	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
	795
	796	7. Start Guest3:
	797
	798	/usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
	799	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
	800
	801	When the guest is shut down, the mediated matrix devices may be removed.
	802
	803	Using our example again, to remove the mediated matrix device $uuid1:
	804
	805	/sys/devices/vfio_ap/matrix/
	806	--- [mdev_supported_types]
	807	------ [vfio_ap-passthrough]
	808	--------- [devices]
	809	------------ [$uuid1]
	810	--------------- remove
	811
	812
	813	echo 1 > remove
	814
	815	This will remove all of the mdev matrix device's sysfs structures including
	816	the mdev device itself. To recreate and reconfigure the mdev matrix device,
	817	all of the steps starting with step 3 will have to be performed again. Note
	818	that the remove will fail if a guest using the mdev is still running.
	819
	820	It is not necessary to remove an mdev matrix device, but one may want to
	821	remove it if no guest will use it during the remaining lifetime of the linux
	822	host. If the mdev matrix device is removed, one may want to also reconfigure
	823	the pool of adapters and queues reserved for use by the default drivers.
	824
	825	Limitations
	826	===========
	827	* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
	828	to the default drivers pool of a queue that is still assigned to a mediated
	829	device in use by a guest. It is incumbent upon the administrator to
	830	ensure there is no mediated device in use by a guest to which the APQN is
	831	assigned lest the host be given access to the private data of the AP queue
	832	device such as a private key configured specifically for the guest.
	833
	834	* Dynamically modifying the AP matrix for a running guest (which would amount to
	835	hot(un)plug of AP devices for the guest) is currently not supported
	836
	837	* Live guest migration is not supported for guests using AP devices.

MAINTAINERS+12−0

@@ -12668,6 +12668,18 @@ W: http://www.ibm.com/developerworks/linux/linux390/ · 12667 unmodified lines
12668	S: Supported	12668	S: Supported
12669	F: drivers/s390/crypto/	12669	F: drivers/s390/crypto/
12670		12670
		12671	S390 VFIO AP DRIVER
		12672	M: Tony Krowiak <akrowiak@linux.ibm.com>
		12673	M: Pierre Morel <pmorel@linux.ibm.com>
		12674	M: Halil Pasic <pasic@linux.ibm.com>
		12675	L: linux-s390@vger.kernel.org
		12676	W: http://www.ibm.com/developerworks/linux/linux390/
		12677	S: Supported
		12678	F: drivers/s390/crypto/vfio_ap_drv.c
		12679	F: drivers/s390/crypto/vfio_ap_private.h
		12680	F: drivers/s390/crypto/vfio_ap_ops.c
		12681	F: Documentation/s390/vfio-ap.txt
		12682
12671	S390 ZFCP DRIVER	12683	S390 ZFCP DRIVER
12672	M: Steffen Maier <maier@linux.ibm.com>	12684	M: Steffen Maier <maier@linux.ibm.com>
12673	M: Benjamin Block <bblock@linux.ibm.com>	12685	M: Benjamin Block <bblock@linux.ibm.com>

arch/s390/Kconfig+11−0

@@ -773,6 +773,17 @@ config VFIO_CCW · 772 unmodified lines
773	To compile this driver as a module, choose M here: the	773	To compile this driver as a module, choose M here: the
774	module will be called vfio_ccw.	774	module will be called vfio_ccw.
775		775
		776	config VFIO_AP
		777	def_tristate n
		778	prompt "VFIO support for AP devices"
		779	depends on S390_AP_IOMMU && VFIO_MDEV_DEVICE && KVM
		780	help
		781	This driver grants access to Adjunct Processor (AP) devices
		782	via the VFIO mediated device interface.
		783
		784	To compile this driver as a module, choose M here: the module
		785	will be called vfio_ap.
		786
776	endmenu	787	endmenu
777		788
778	menu "Dump support"	789	menu "Dump support"

arch/s390/include/asm/kvm_host.h+12−1

 #define KVM_REQ_ICPT_OPEREXC	KVM_ARCH_REQ(2)
 #define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3)
 #define KVM_REQ_STOP_MIGRATION  KVM_ARCH_REQ(4)
+#define KVM_REQ_VSIE_RESTART	KVM_ARCH_REQ(5)
 #define SIGP_CTRL_C		0x80
 #define SIGP_CTRL_SCN_MASK	0x3f
 #define ECA_AIV		0x00200000
 #define ECA_VX		0x00020000
 #define ECA_PROTEXCI	0x00002000
+#define ECA_APIE	0x00000008
 #define ECA_SII		0x00000001
 	__u32	eca;			/* 0x004c */
 #define ICPT_INST	0x04
 	psw_t	gpsw;			/* 0x0090 */
 	__u64	gg14;			/* 0x00a0 */
 	__u64	gg15;			/* 0x00a8 */
-	__u8	reservedb0[20];		/* 0x00b0 */
+	__u8	reservedb0[8];		/* 0x00b0 */
+#define HPID_KVM	0x4
+#define HPID_VSIE	0x5
+	__u8	hpid;			/* 0x00b8 */
+	__u8	reservedb9[11];		/* 0x00b9 */
 	__u16	extcpuaddr;		/* 0x00c4 */
 	__u16	eic;			/* 0x00c6 */
 	__u32	reservedc8;		/* 0x00c8 */
 	__u8	reservede4[4];		/* 0x00e4 */
 	__u64	tecmc;			/* 0x00e8 */
 	__u8	reservedf0[12];		/* 0x00f0 */
+#define CRYCB_FORMAT_MASK 0x00000003
+#define CRYCB_FORMAT0 0x00000000
 #define CRYCB_FORMAT1 0x00000001
 #define CRYCB_FORMAT2 0x00000003
 	__u32	crycbd;			/* 0x00fc */
 	__u32 crycbd;
 	__u8 aes_kw;
 	__u8 dea_kw;
+	__u8 apie;
 };
 #define APCB0_MASK_SIZE 1
 void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
 				 struct kvm_async_pf *work);
+void kvm_arch_crypto_clear_masks(struct kvm *kvm);
 extern int sie64a(struct kvm_s390_sie_block *, u64 *);
 extern char sie_exit;

arch/s390/include/uapi/asm/kvm.h+2−0

@@ -160,6 +160,8 @@ struct kvm_s390_vm_cpu_subfunc { · 159 unmodified lines
160	#define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1	160	#define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
161	#define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2	161	#define KVM_S390_VM_CRYPTO_DISABLE_AES_KW 2
162	#define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3	162	#define KVM_S390_VM_CRYPTO_DISABLE_DEA_KW 3
		163	#define KVM_S390_VM_CRYPTO_ENABLE_APIE 4
		164	#define KVM_S390_VM_CRYPTO_DISABLE_APIE 5
163		165
164	/* kvm attributes for migration mode */	166	/* kvm attributes for migration mode */
165	#define KVM_S390_VM_MIGRATION_STOP 0	167	#define KVM_S390_VM_MIGRATION_STOP 0

arch/s390/kvm/kvm-s390.c+93−42

 #include <asm/sclp.h>
 #include <asm/cpacf.h>
 #include <asm/timex.h>
+#include <asm/ap.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
 	kvm_s390_vcpu_block_all(kvm);
-	kvm_for_each_vcpu(i, vcpu, kvm)
+	kvm_for_each_vcpu(i, vcpu, kvm) {
 		kvm_s390_vcpu_crypto_setup(vcpu);
+		/* recreate the shadow crycb by leaving the VSIE handler */
+		kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
+	}
 	kvm_s390_vcpu_unblock_all(kvm);
 }
 static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
 {
-	if (!test_kvm_facility(kvm, 76))
-		return -EINVAL;
 	mutex_lock(&kvm->lock);
 	switch (attr->attr) {
 	case KVM_S390_VM_CRYPTO_ENABLE_AES_KW:
+		if (!test_kvm_facility(kvm, 76))
+			return -EINVAL;
 		get_random_bytes(
 			kvm->arch.crypto.crycb->aes_wrapping_key_mask,
 			sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
 		VM_EVENT(kvm, 3, "%s", "ENABLE: AES keywrapping support");
 		break;
 	case KVM_S390_VM_CRYPTO_ENABLE_DEA_KW:
+		if (!test_kvm_facility(kvm, 76))
+			return -EINVAL;
 		get_random_bytes(
 			kvm->arch.crypto.crycb->dea_wrapping_key_mask,
 			sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
 		VM_EVENT(kvm, 3, "%s", "ENABLE: DEA keywrapping support");
 		break;
 	case KVM_S390_VM_CRYPTO_DISABLE_AES_KW:
+		if (!test_kvm_facility(kvm, 76))
+			return -EINVAL;
 		kvm->arch.crypto.aes_kw = 0;
 		memset(kvm->arch.crypto.crycb->aes_wrapping_key_mask, 0,
 			sizeof(kvm->arch.crypto.crycb->aes_wrapping_key_mask));
 		VM_EVENT(kvm, 3, "%s", "DISABLE: AES keywrapping support");
 		break;
 	case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
+		if (!test_kvm_facility(kvm, 76))
+			return -EINVAL;
 		kvm->arch.crypto.dea_kw = 0;
 		memset(kvm->arch.crypto.crycb->dea_wrapping_key_mask, 0,
 			sizeof(kvm->arch.crypto.crycb->dea_wrapping_key_mask));
 		VM_EVENT(kvm, 3, "%s", "DISABLE: DEA keywrapping support");
 		break;
+	case KVM_S390_VM_CRYPTO_ENABLE_APIE:
+		if (!ap_instructions_available()) {
+			mutex_unlock(&kvm->lock);
+			return -EOPNOTSUPP;
+		}
+		kvm->arch.crypto.apie = 1;
+		break;
+	case KVM_S390_VM_CRYPTO_DISABLE_APIE:
+		if (!ap_instructions_available()) {
+			mutex_unlock(&kvm->lock);
+			return -EOPNOTSUPP;
+		}
+		kvm->arch.crypto.apie = 0;
+		break;
 	default:
 		mutex_unlock(&kvm->lock);
 		return -ENXIO;
 		case KVM_S390_VM_CRYPTO_DISABLE_DEA_KW:
 			ret = 0;
 			break;
+		case KVM_S390_VM_CRYPTO_ENABLE_APIE:
+		case KVM_S390_VM_CRYPTO_DISABLE_APIE:
+			ret = ap_instructions_available() ? 0 : -ENXIO;
+			break;
 		default:
 			ret = -ENXIO;
 			break;
 	return r;
 }
-static int kvm_s390_query_ap_config(u8 *config)
-{
-	u32 fcn_code = 0x04000000UL;
-	u32 cc = 0;
-	memset(config, 0, 128);
-	asm volatile(
-		"lgr 0,%1\n"
-		"lgr 2,%2\n"
-		".long 0xb2af0000\n"		/* PQAP(QCI) */
-		"0: ipm %0\n"
-		"srl %0,28\n"
-		"1:\n"
-		EX_TABLE(0b, 1b)
-		: "+r" (cc)
-		: "r" (fcn_code), "r" (config)
-		: "cc", "0", "2", "memory"
-	);
-	return cc;
-}
 static int kvm_s390_apxa_installed(void)
 {
-	u8 config[128];
+	struct ap_config_info info;
-	int cc;
-	if (test_facility(12)) {
-		cc = kvm_s390_query_ap_config(config);
-		if (cc)
+	if (ap_instructions_available()) {
-			pr_err("PQAP(QCI) failed with cc=%d", cc);
+		if (ap_qci(&info) == 0)
-		else
+			return info.apxa;
-			return config[0] & 0x40;
 	}
 	return 0;
 }
+/*
+ * The format of the crypto control block (CRYCB) is specified in the 3 low
+ * order bits of the CRYCB designation (CRYCBD) field as follows:
+ * Format 0: Neither the message security assist extension 3 (MSAX3) nor the
+ *	     AP extended addressing (APXA) facility are installed.
+ * Format 1: The APXA facility is not installed but the MSAX3 facility is.
+ * Format 2: Both the APXA and MSAX3 facilities are installed
+ */
 static void kvm_s390_set_crycb_format(struct kvm *kvm)
 {
 	kvm->arch.crypto.crycbd = (__u32)(unsigned long) kvm->arch.crypto.crycb;
+	/* Clear the CRYCB format bits - i.e., set format 0 by default */
+	kvm->arch.crypto.crycbd &= ~(CRYCB_FORMAT_MASK);
+	/* Check whether MSAX3 is installed */
+	if (!test_kvm_facility(kvm, 76))
+		return;
 	if (kvm_s390_apxa_installed())
 		kvm->arch.crypto.crycbd |= CRYCB_FORMAT2;
 	else
 		kvm->arch.crypto.crycbd |= CRYCB_FORMAT1;
 }
+void kvm_arch_crypto_clear_masks(struct kvm *kvm)
+{
+	mutex_lock(&kvm->lock);
+	kvm_s390_vcpu_block_all(kvm);
+	memset(&kvm->arch.crypto.crycb->apcb0, 0,
+	       sizeof(kvm->arch.crypto.crycb->apcb0));
+	memset(&kvm->arch.crypto.crycb->apcb1, 0,
+	       sizeof(kvm->arch.crypto.crycb->apcb1));
+	/* recreate the shadow crycb for each vcpu */
+	kvm_s390_sync_request_broadcast(kvm, KVM_REQ_VSIE_RESTART);
+	kvm_s390_vcpu_unblock_all(kvm);
+	mutex_unlock(&kvm->lock);
+}
+EXPORT_SYMBOL_GPL(kvm_arch_crypto_clear_masks);
 static u64 kvm_s390_get_initial_cpuid(void)
 {
 	struct cpuid cpuid;
 static void kvm_s390_crypto_init(struct kvm *kvm)
 {
-	if (!test_kvm_facility(kvm, 76))
-		return;
 	kvm->arch.crypto.crycb = &kvm->arch.sie_page2->crycb;
 	kvm_s390_set_crycb_format(kvm);
+	if (!test_kvm_facility(kvm, 76))
+		return;
 	/* Enable AES/DEA protected key functions by default */
 	kvm->arch.crypto.aes_kw = 1;
 	kvm->arch.crypto.dea_kw = 1;
 static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu)
 {
-	if (!test_kvm_facility(vcpu->kvm, 76))
+	/*
+	 * If the AP instructions are not being interpreted and the MSAX3
+	 * facility is not configured for the guest, there is nothing to set up.
+	 */
+	if (!vcpu->kvm->arch.crypto.apie && !test_kvm_facility(vcpu->kvm, 76))
 		return;
+	vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
 	vcpu->arch.sie_block->ecb3 &= ~(ECB3_AES | ECB3_DEA);
+	vcpu->arch.sie_block->eca &= ~ECA_APIE;
+	if (vcpu->kvm->arch.crypto.apie)
+		vcpu->arch.sie_block->eca |= ECA_APIE;
+	/* Set up protected key support */
 	if (vcpu->kvm->arch.crypto.aes_kw)
 		vcpu->arch.sie_block->ecb3 |= ECB3_AES;
 	if (vcpu->kvm->arch.crypto.dea_kw)
 		vcpu->arch.sie_block->ecb3 |= ECB3_DEA;
-	vcpu->arch.sie_block->crycbd = vcpu->kvm->arch.crypto.crycbd;
 }
 void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu)
 	hrtimer_init(&vcpu->arch.ckc_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
 	vcpu->arch.ckc_timer.function = kvm_s390_idle_wakeup;
+	vcpu->arch.sie_block->hpid = HPID_KVM;
 	kvm_s390_vcpu_crypto_setup(vcpu);
 	return rc;
 	exit_sie(vcpu);
 }
+bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu)
+{
+	return atomic_read(&vcpu->arch.sie_block->prog20) &
+	       (PROG_BLOCK_SIE | PROG_REQUEST);
+}
 static void kvm_s390_vcpu_request_handled(struct kvm_vcpu *vcpu)
 {
 	atomic_andnot(PROG_REQUEST, &vcpu->arch.sie_block->prog20);
 }
 /*
- * Kick a guest cpu out of SIE and wait until SIE is not running.
+ * Kick a guest cpu out of (v)SIE and wait until (v)SIE is not running.
  * If the CPU is not running (e.g. waiting as idle) the function will
  * return immediately. */
 void exit_sie(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_set_cpuflags(vcpu, CPUSTAT_STOP_INT);
+	kvm_s390_vsie_kick(vcpu);
 	while (vcpu->arch.sie_block->prog0c & PROG_IN_SIE)
 		cpu_relax();
 }
 	/* nothing to do, just clear the request */
 	kvm_clear_request(KVM_REQ_UNHALT, vcpu);
+	/* we left the vsie handler, nothing to do, just clear the request */
+	kvm_clear_request(KVM_REQ_VSIE_RESTART, vcpu);
 	return 0;
 }

arch/s390/kvm/kvm-s390.h+1−0

@@ -290,6 +290,7 @@ void kvm_s390_vcpu_start(struct kvm_vcpu *vcpu); · 289 unmodified lines
290	void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu);	290	void kvm_s390_vcpu_stop(struct kvm_vcpu *vcpu);
291	void kvm_s390_vcpu_block(struct kvm_vcpu *vcpu);	291	void kvm_s390_vcpu_block(struct kvm_vcpu *vcpu);
292	void kvm_s390_vcpu_unblock(struct kvm_vcpu *vcpu);	292	void kvm_s390_vcpu_unblock(struct kvm_vcpu *vcpu);
		293	bool kvm_s390_vcpu_sie_inhibited(struct kvm_vcpu *vcpu);
293	void exit_sie(struct kvm_vcpu *vcpu);	294	void exit_sie(struct kvm_vcpu *vcpu);
294	void kvm_s390_sync_request(int req, struct kvm_vcpu *vcpu);	295	void kvm_s390_sync_request(int req, struct kvm_vcpu *vcpu);
295	int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);	296	int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);

arch/s390/kvm/vsie.c+192−18

@@ -135,14 +135,148 @@ static int prepare_cpuflags(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 134 unmodified lines
135	atomic_set(&scb_s->cpuflags, newflags);	135	atomic_set(&scb_s->cpuflags, newflags);
136	return 0;	136	return 0;
137	}	137	}
		138	/* Copy to APCB FORMAT1 from APCB FORMAT0 */
		139	static int setup_apcb10(struct kvm_vcpu vcpu, struct kvm_s390_apcb1 apcb_s,
		140	unsigned long apcb_o, struct kvm_s390_apcb1 *apcb_h)
		141	{
		142	struct kvm_s390_apcb0 tmp;
138		143
139	/*	144	if (read_guest_real(vcpu, apcb_o, &tmp, sizeof(struct kvm_s390_apcb0)))
		145	return -EFAULT;
		146
		147	apcb_s->apm[0] = apcb_h->apm[0] & tmp.apm[0];
		148	apcb_s->aqm[0] = apcb_h->aqm[0] & tmp.aqm[0] & 0xffff000000000000UL;
		149	apcb_s->adm[0] = apcb_h->adm[0] & tmp.adm[0] & 0xffff000000000000UL;
		150
		151	return 0;
		152
		153	}
		154
		155	/**
		156	* setup_apcb00 - Copy to APCB FORMAT0 from APCB FORMAT0
		157	* @vcpu: pointer to the virtual CPU
		158	* @apcb_s: pointer to start of apcb in the shadow crycb
		159	* @apcb_o: pointer to start of original apcb in the guest2
		160	* @apcb_h: pointer to start of apcb in the guest1
		161	*
		162	* Returns 0 and -EFAULT on error reading guest apcb
		163	*/
		164	static int setup_apcb00(struct kvm_vcpu vcpu, unsigned long apcb_s,
		165	unsigned long apcb_o, unsigned long *apcb_h)
		166	{
		167	if (read_guest_real(vcpu, apcb_o, apcb_s,
		168	sizeof(struct kvm_s390_apcb0)))
		169	return -EFAULT;
		170
		171	bitmap_and(apcb_s, apcb_s, apcb_h, sizeof(struct kvm_s390_apcb0));
		172
		173	return 0;
		174	}
		175
		176	/**
		177	* setup_apcb11 - Copy the FORMAT1 APCB from the guest to the shadow CRYCB
		178	* @vcpu: pointer to the virtual CPU
		179	* @apcb_s: pointer to start of apcb in the shadow crycb
		180	* @apcb_o: pointer to start of original guest apcb
		181	* @apcb_h: pointer to start of apcb in the host
		182	*
		183	* Returns 0 and -EFAULT on error reading guest apcb
		184	*/
		185	static int setup_apcb11(struct kvm_vcpu vcpu, unsigned long apcb_s,
		186	unsigned long apcb_o,
		187	unsigned long *apcb_h)
		188	{
		189	if (read_guest_real(vcpu, apcb_o, apcb_s,
		190	sizeof(struct kvm_s390_apcb1)))
		191	return -EFAULT;
		192
		193	bitmap_and(apcb_s, apcb_s, apcb_h, sizeof(struct kvm_s390_apcb1));
		194
		195	return 0;
		196	}
		197
		198	/**
		199	* setup_apcb - Create a shadow copy of the apcb.
		200	* @vcpu: pointer to the virtual CPU
		201	* @crycb_s: pointer to shadow crycb
		202	* @crycb_o: pointer to original guest crycb
		203	* @crycb_h: pointer to the host crycb
		204	* @fmt_o: format of the original guest crycb.
		205	* @fmt_h: format of the host crycb.
		206	*
		207	* Checks the compatibility between the guest and host crycb and calls the
		208	* appropriate copy function.
		209	*
		210	* Return 0 or an error number if the guest and host crycb are incompatible.
		211	*/
		212	static int setup_apcb(struct kvm_vcpu vcpu, struct kvm_s390_crypto_cb crycb_s,
		213	const u32 crycb_o,
		214	struct kvm_s390_crypto_cb *crycb_h,
		215	int fmt_o, int fmt_h)
		216	{
		217	struct kvm_s390_crypto_cb *crycb;
		218
		219	crycb = (struct kvm_s390_crypto_cb *) (unsigned long)crycb_o;
		220
		221	switch (fmt_o) {
		222	case CRYCB_FORMAT2:
		223	if ((crycb_o & PAGE_MASK) != ((crycb_o + 256) & PAGE_MASK))
		224	return -EACCES;
		225	if (fmt_h != CRYCB_FORMAT2)
		226	return -EINVAL;
		227	return setup_apcb11(vcpu, (unsigned long *)&crycb_s->apcb1,
		228	(unsigned long) &crycb->apcb1,
		229	(unsigned long *)&crycb_h->apcb1);
		230	case CRYCB_FORMAT1:
		231	switch (fmt_h) {
		232	case CRYCB_FORMAT2:
		233	return setup_apcb10(vcpu, &crycb_s->apcb1,
		234	(unsigned long) &crycb->apcb0,
		235	&crycb_h->apcb1);
		236	case CRYCB_FORMAT1:
		237	return setup_apcb00(vcpu,
		238	(unsigned long *) &crycb_s->apcb0,
		239	(unsigned long) &crycb->apcb0,
		240	(unsigned long *) &crycb_h->apcb0);
		241	}
		242	break;
		243	case CRYCB_FORMAT0:
		244	if ((crycb_o & PAGE_MASK) != ((crycb_o + 32) & PAGE_MASK))
		245	return -EACCES;
		246
		247	switch (fmt_h) {
		248	case CRYCB_FORMAT2:
		249	return setup_apcb10(vcpu, &crycb_s->apcb1,
		250	(unsigned long) &crycb->apcb0,
		251	&crycb_h->apcb1);
		252	case CRYCB_FORMAT1:
		253	case CRYCB_FORMAT0:
		254	return setup_apcb00(vcpu,
		255	(unsigned long *) &crycb_s->apcb0,
		256	(unsigned long) &crycb->apcb0,
		257	(unsigned long *) &crycb_h->apcb0);
		258	}
		259	}
		260	return -EINVAL;
		261	}
		262
		263	/**
		264	* shadow_crycb - Create a shadow copy of the crycb block
		265	* @vcpu: a pointer to the virtual CPU
		266	* @vsie_page: a pointer to internal date used for the vSIE
		267	*
140	* Create a shadow copy of the crycb block and setup key wrapping, if	268	* Create a shadow copy of the crycb block and setup key wrapping, if
141	* requested for guest 3 and enabled for guest 2.	269	* requested for guest 3 and enabled for guest 2.
142	*	270	*
143	* We only accept format-1 (no AP in g2), but convert it into format-2	271	* We accept format-1 or format-2, but we convert format-1 into format-2
		272	* in the shadow CRYCB.
		273	* Using format-2 enables the firmware to choose the right format when
		274	* scheduling the SIE.
144	* There is nothing to do for format-0.	275	* There is nothing to do for format-0.
145	*	276	*
		277	* This function centralize the issuing of set_validity_icpt() for all
		278	* the subfunctions working on the crycb.
		279	*
146	* Returns: - 0 if shadowed or nothing to do	280	* Returns: - 0 if shadowed or nothing to do
147	* - > 0 if control has to be given to guest 2	281	* - > 0 if control has to be given to guest 2
148	*/	282	*/
@@ -154,23 +288,40 @@ static int shadow_crycb(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 5 unmodified lines
154	const u32 crycb_addr = crycbd_o & 0x7ffffff8U;	288	const u32 crycb_addr = crycbd_o & 0x7ffffff8U;
155	unsigned long b1, b2;	289	unsigned long b1, b2;
156	u8 ecb3_flags;	290	u8 ecb3_flags;
		291	int apie_h;
		292	int key_msk = test_kvm_facility(vcpu->kvm, 76);
		293	int fmt_o = crycbd_o & CRYCB_FORMAT_MASK;
		294	int fmt_h = vcpu->arch.sie_block->crycbd & CRYCB_FORMAT_MASK;
		295	int ret = 0;
157		296
158	scb_s->crycbd = 0;	297	scb_s->crycbd = 0;
159	if (!(crycbd_o & vcpu->arch.sie_block->crycbd & CRYCB_FORMAT1))	298
160	return 0;	299	apie_h = vcpu->arch.sie_block->eca & ECA_APIE;
161	/* format-1 is supported with message-security-assist extension 3 */	300	if (!apie_h && !key_msk)
162	if (!test_kvm_facility(vcpu->kvm, 76))
163	return 0;	301	return 0;
		302
		303	if (!crycb_addr)
		304	return set_validity_icpt(scb_s, 0x0039U);
		305
		306	if (fmt_o == CRYCB_FORMAT1)
		307	if ((crycb_addr & PAGE_MASK) !=
		308	((crycb_addr + 128) & PAGE_MASK))
		309	return set_validity_icpt(scb_s, 0x003CU);
		310
		311	if (apie_h && (scb_o->eca & ECA_APIE)) {
		312	ret = setup_apcb(vcpu, &vsie_page->crycb, crycb_addr,
		313	vcpu->kvm->arch.crypto.crycb,
		314	fmt_o, fmt_h);
		315	if (ret)
		316	goto end;
		317	scb_s->eca \|= scb_o->eca & ECA_APIE;
		318	}
		319
164	/* we may only allow it if enabled for guest 2 */	320	/* we may only allow it if enabled for guest 2 */
165	ecb3_flags = scb_o->ecb3 & vcpu->arch.sie_block->ecb3 &	321	ecb3_flags = scb_o->ecb3 & vcpu->arch.sie_block->ecb3 &
166	(ECB3_AES \| ECB3_DEA);	322	(ECB3_AES \| ECB3_DEA);
167	if (!ecb3_flags)	323	if (!ecb3_flags)
168	return 0;	324	goto end;
169
170	if ((crycb_addr & PAGE_MASK) != ((crycb_addr + 128) & PAGE_MASK))
171	return set_validity_icpt(scb_s, 0x003CU);
172	else if (!crycb_addr)
173	return set_validity_icpt(scb_s, 0x0039U);
174		325
175	/* copy only the wrapping keys */	326	/* copy only the wrapping keys */
176	if (read_guest_real(vcpu, crycb_addr + 72,	327	if (read_guest_real(vcpu, crycb_addr + 72,
@@ -178,8 +329,6 @@ static int shadow_crycb(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 1 unmodified line
178	return set_validity_icpt(scb_s, 0x0035U);	329	return set_validity_icpt(scb_s, 0x0035U);
179		330
180	scb_s->ecb3 \|= ecb3_flags;	331	scb_s->ecb3 \|= ecb3_flags;
181	scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) \| CRYCB_FORMAT1 \|
182	CRYCB_FORMAT2;
183		332
184	/* xor both blocks in one run */	333	/* xor both blocks in one run */
185	b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;	334	b1 = (unsigned long *) vsie_page->crycb.dea_wrapping_key_mask;
@@ -187,6 +336,16 @@ static int shadow_crycb(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 1 unmodified line
187	vcpu->kvm->arch.crypto.crycb->dea_wrapping_key_mask;	336	vcpu->kvm->arch.crypto.crycb->dea_wrapping_key_mask;
188	/* as 56%8 == 0, bitmap_xor won't overwrite any data */	337	/* as 56%8 == 0, bitmap_xor won't overwrite any data */
189	bitmap_xor(b1, b1, b2, BITS_PER_BYTE * 56);	338	bitmap_xor(b1, b1, b2, BITS_PER_BYTE * 56);
		339	end:
		340	switch (ret) {
		341	case -EINVAL:
		342	return set_validity_icpt(scb_s, 0x0020U);
		343	case -EFAULT:
		344	return set_validity_icpt(scb_s, 0x0035U);
		345	case -EACCES:
		346	return set_validity_icpt(scb_s, 0x003CU);
		347	}
		348	scb_s->crycbd = ((__u32)(__u64) &vsie_page->crycb) \| CRYCB_FORMAT2;
190	return 0;	349	return 0;
191	}	350	}
192		351
@@ -383,6 +542,8 @@ static int shadow_scb(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 190 unmodified lines
383	if (test_kvm_facility(vcpu->kvm, 156))	542	if (test_kvm_facility(vcpu->kvm, 156))
384	scb_s->ecd \|= scb_o->ecd & ECD_ETOKENF;	543	scb_s->ecd \|= scb_o->ecd & ECD_ETOKENF;
385		544
		545	scb_s->hpid = HPID_VSIE;
		546
386	prepare_ibc(vcpu, vsie_page);	547	prepare_ibc(vcpu, vsie_page);
387	rc = shadow_crycb(vcpu, vsie_page);	548	rc = shadow_crycb(vcpu, vsie_page);
388	out:	549	out:
@@ -830,7 +991,7 @@ static int do_vsie_run(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 441 unmodified lines
830	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;	991	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
831	struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;	992	struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
832	int guest_bp_isolation;	993	int guest_bp_isolation;
833	int rc;	994	int rc = 0;
834		995
835	handle_last_fault(vcpu, vsie_page);	996	handle_last_fault(vcpu, vsie_page);
836		997
@@ -858,7 +1019,18 @@ static int do_vsie_run(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 21 unmodified lines
858	guest_enter_irqoff();	1019	guest_enter_irqoff();
859	local_irq_enable();	1020	local_irq_enable();
860		1021
861	rc = sie64a(scb_s, vcpu->run->s.regs.gprs);	1022	/*
		1023	* Simulate a SIE entry of the VCPU (see sie64a), so VCPU blocking
		1024	* and VCPU requests also hinder the vSIE from running and lead
		1025	* to an immediate exit. kvm_s390_vsie_kick() has to be used to
		1026	* also kick the vSIE.
		1027	*/
		1028	vcpu->arch.sie_block->prog0c \|= PROG_IN_SIE;
		1029	barrier();
		1030	if (!kvm_s390_vcpu_sie_inhibited(vcpu))
		1031	rc = sie64a(scb_s, vcpu->run->s.regs.gprs);
		1032	barrier();
		1033	vcpu->arch.sie_block->prog0c &= ~PROG_IN_SIE;
862		1034
863	local_irq_disable();	1035	local_irq_disable();
864	guest_exit_irqoff();	1036	guest_exit_irqoff();
@@ -1005,7 +1177,8 @@ static int vsie_run(struct kvm_vcpu vcpu, struct vsie_page vsie_page) · 140 unmodified lines
1005	if (rc == -EAGAIN)	1177	if (rc == -EAGAIN)
1006	rc = 0;	1178	rc = 0;
1007	if (rc \|\| scb_s->icptcode \|\| signal_pending(current) \|\|	1179	if (rc \|\| scb_s->icptcode \|\| signal_pending(current) \|\|
1008	kvm_s390_vcpu_has_irq(vcpu, 0))	1180	kvm_s390_vcpu_has_irq(vcpu, 0) \|\|
		1181	kvm_s390_vcpu_sie_inhibited(vcpu))
1009	break;	1182	break;
1010	}	1183	}
1011		1184
@@ -1122,7 +1295,8 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu) · 110 unmodified lines
1122	if (unlikely(scb_addr & 0x1ffUL))	1295	if (unlikely(scb_addr & 0x1ffUL))
1123	return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);	1296	return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
1124		1297
1125	if (signal_pending(current) \|\| kvm_s390_vcpu_has_irq(vcpu, 0))	1298	if (signal_pending(current) \|\| kvm_s390_vcpu_has_irq(vcpu, 0) \|\|
		1299	kvm_s390_vcpu_sie_inhibited(vcpu))
1126	return 0;	1300	return 0;
1127		1301
1128	vsie_page = get_vsie_page(vcpu->kvm, scb_addr);	1302	vsie_page = get_vsie_page(vcpu->kvm, scb_addr);

arch/s390/mm/gmap.c+8−2

 	pmd_t *pmdp;
 	BUG_ON(gmap_is_shadow(gmap));
-	spin_lock(&gmap->guest_table_lock);
 	pmdp = (pmd_t *) gmap_table_walk(gmap, gaddr, 1);
+	if (!pmdp)
+		return NULL;
-	if (!pmdp || pmd_none(*pmdp)) {
+	/* without huge pages, there is no need to take the table lock */
+	if (!gmap->mm->context.allow_gmap_hpage_1m)
+		return pmd_none(*pmdp) ? NULL : pmdp;
+	spin_lock(&gmap->guest_table_lock);
+	if (pmd_none(*pmdp)) {
 		spin_unlock(&gmap->guest_table_lock);
 		return NULL;
 	}

arch/s390/tools/gen_facilities.c+2−0

@@ -106,6 +106,8 @@ static struct facility_def facility_defs[] = { · 105 unmodified lines
106		106
107	.name = "FACILITIES_KVM_CPUMODEL",	107	.name = "FACILITIES_KVM_CPUMODEL",
108	.bits = (int[]){	108	.bits = (int[]){
		109	12, /* AP Query Configuration Information */
		110	15, /* AP Facilities Test */
109	156, /* etoken facility */	111	156, /* etoken facility */
110	-1 /* END */	112	-1 /* END */
111	}	113	}

drivers/iommu/Kconfig+8−0

@@ -372,6 +372,14 @@ config S390_CCW_IOMMU · 371 unmodified lines
372	Enables bits of IOMMU API required by VFIO. The iommu_ops	372	Enables bits of IOMMU API required by VFIO. The iommu_ops
373	is not implemented as it is not necessary for VFIO.	373	is not implemented as it is not necessary for VFIO.
374		374
		375	config S390_AP_IOMMU
		376	bool "S390 AP IOMMU Support"
		377	depends on S390 && ZCRYPT
		378	select IOMMU_API
		379	help
		380	Enables bits of IOMMU API required by VFIO. The iommu_ops
		381	is not implemented as it is not necessary for VFIO.
		382
375	config MTK_IOMMU	383	config MTK_IOMMU
376	bool "MTK IOMMU Support"	384	bool "MTK IOMMU Support"
377	depends on ARM \|\| ARM64	385	depends on ARM \|\| ARM64

drivers/s390/crypto/Makefile+4−0

@@ -15,3 +15,7 @@ obj-$(CONFIG_ZCRYPT) += zcrypt_pcixcc.o zcrypt_cex2a.o zcrypt_cex4.o · 14 unmodified lines
15	# pkey kernel module	15	# pkey kernel module
16	pkey-objs := pkey_api.o	16	pkey-objs := pkey_api.o
17	obj-$(CONFIG_PKEY) += pkey.o	17	obj-$(CONFIG_PKEY) += pkey.o
		18
		19	# adjunct processor matrix
		20	vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o
		21	obj-$(CONFIG_VFIO_AP) += vfio_ap.o

drivers/s390/crypto/vfio_ap_drv.c+157−0

@@ -0,0 +1,157 @@
	1	// SPDX-License-Identifier: GPL-2.0+
	2	/*
	3	* VFIO based AP device driver
	4	*
	5	* Copyright IBM Corp. 2018
	6	*
	7	* Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
	8	*/
	9
	10	#include <linux/module.h>
	11	#include <linux/mod_devicetable.h>
	12	#include <linux/slab.h>
	13	#include <linux/string.h>
	14	#include "vfio_ap_private.h"
	15
	16	#define VFIO_AP_ROOT_NAME "vfio_ap"
	17	#define VFIO_AP_DEV_TYPE_NAME "ap_matrix"
	18	#define VFIO_AP_DEV_NAME "matrix"
	19
	20	MODULE_AUTHOR("IBM Corporation");
	21	MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2018");
	22	MODULE_LICENSE("GPL v2");
	23
	24	static struct ap_driver vfio_ap_drv;
	25
	26	static struct device_type vfio_ap_dev_type = {
	27	.name = VFIO_AP_DEV_TYPE_NAME,
	28	};
	29
	30	struct ap_matrix_dev *matrix_dev;
	31
	32	/* Only type 10 adapters (CEX4 and later) are supported
	33	* by the AP matrix device driver
	34	*/
	35	static struct ap_device_id ap_queue_ids[] = {
	36	{ .dev_type = AP_DEVICE_TYPE_CEX4,
	37	.match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
	38	{ .dev_type = AP_DEVICE_TYPE_CEX5,
	39	.match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
	40	{ .dev_type = AP_DEVICE_TYPE_CEX6,
	41	.match_flags = AP_DEVICE_ID_MATCH_QUEUE_TYPE },
	42	{ /* end of sibling */ },
	43	};
	44
	45	MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
	46
	47	static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
	48	{
	49	return 0;
	50	}
	51
	52	static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
	53	{
	54	/* Nothing to do yet */
	55	}
	56
	57	static void vfio_ap_matrix_dev_release(struct device *dev)
	58	{
	59	struct ap_matrix_dev *matrix_dev = dev_get_drvdata(dev);
	60
	61	kfree(matrix_dev);
	62	}
	63
	64	static int vfio_ap_matrix_dev_create(void)
	65	{
	66	int ret;
	67	struct device *root_device;
	68
	69	root_device = root_device_register(VFIO_AP_ROOT_NAME);
	70	if (IS_ERR(root_device))
	71	return PTR_ERR(root_device);
	72
	73	matrix_dev = kzalloc(sizeof(*matrix_dev), GFP_KERNEL);
	74	if (!matrix_dev) {
	75	ret = -ENOMEM;
	76	goto matrix_alloc_err;
	77	}
	78
	79	/* Fill in config info via PQAP(QCI), if available */
	80	if (test_facility(12)) {
	81	ret = ap_qci(&matrix_dev->info);
	82	if (ret)
	83	goto matrix_alloc_err;
	84	}
	85
	86	mutex_init(&matrix_dev->lock);
	87	INIT_LIST_HEAD(&matrix_dev->mdev_list);
	88
	89	matrix_dev->device.type = &vfio_ap_dev_type;
	90	dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME);
	91	matrix_dev->device.parent = root_device;
	92	matrix_dev->device.release = vfio_ap_matrix_dev_release;
	93	matrix_dev->device.driver = &vfio_ap_drv.driver;
	94
	95	ret = device_register(&matrix_dev->device);
	96	if (ret)
	97	goto matrix_reg_err;
	98
	99	return 0;
	100
	101	matrix_reg_err:
	102	put_device(&matrix_dev->device);
	103	matrix_alloc_err:
	104	root_device_unregister(root_device);
	105
	106	return ret;
	107	}
	108
	109	static void vfio_ap_matrix_dev_destroy(void)
	110	{
	111	device_unregister(&matrix_dev->device);
	112	root_device_unregister(matrix_dev->device.parent);
	113	}
	114
	115	int __init vfio_ap_init(void)
	116	{
	117	int ret;
	118
	119	/* If there are no AP instructions, there is nothing to pass through. */
	120	if (!ap_instructions_available())
	121	return -ENODEV;
	122
	123	ret = vfio_ap_matrix_dev_create();
	124	if (ret)
	125	return ret;
	126
	127	memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv));
	128	vfio_ap_drv.probe = vfio_ap_queue_dev_probe;
	129	vfio_ap_drv.remove = vfio_ap_queue_dev_remove;
	130	vfio_ap_drv.ids = ap_queue_ids;
	131
	132	ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME);
	133	if (ret) {
	134	vfio_ap_matrix_dev_destroy();
	135	return ret;
	136	}
	137
	138	ret = vfio_ap_mdev_register();
	139	if (ret) {
	140	ap_driver_unregister(&vfio_ap_drv);
	141	vfio_ap_matrix_dev_destroy();
	142
	143	return ret;
	144	}
	145
	146	return 0;
	147	}
	148
	149	void __exit vfio_ap_exit(void)
	150	{
	151	vfio_ap_mdev_unregister();
	152	ap_driver_unregister(&vfio_ap_drv);
	153	vfio_ap_matrix_dev_destroy();
	154	}
	155
	156	module_init(vfio_ap_init);
	157	module_exit(vfio_ap_exit);

drivers/s390/crypto/vfio_ap_ops.c+968−0

@@ -0,0 +1,968 @@
	1	// SPDX-License-Identifier: GPL-2.0+
	2	/*
	3	* Adjunct processor matrix VFIO device driver callbacks.
	4	*
	5	* Copyright IBM Corp. 2018
	6	*
	7	* Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
	8	* Halil Pasic <pasic@linux.ibm.com>
	9	* Pierre Morel <pmorel@linux.ibm.com>
	10	*/
	11	#include <linux/string.h>
	12	#include <linux/vfio.h>
	13	#include <linux/device.h>
	14	#include <linux/list.h>
	15	#include <linux/ctype.h>
	16	#include <linux/bitops.h>
	17	#include <linux/kvm_host.h>
	18	#include <linux/module.h>
	19	#include <asm/kvm.h>
	20	#include <asm/zcrypt.h>
	21
	22	#include "vfio_ap_private.h"
	23
	24	#define VFIO_AP_MDEV_TYPE_HWVIRT "passthrough"
	25	#define VFIO_AP_MDEV_NAME_HWVIRT "VFIO AP Passthrough Device"
	26
	27	static void vfio_ap_matrix_init(struct ap_config_info *info,
	28	struct ap_matrix *matrix)
	29	{
	30	matrix->apm_max = info->apxa ? info->Na : 63;
	31	matrix->aqm_max = info->apxa ? info->Nd : 15;
	32	matrix->adm_max = info->apxa ? info->Nd : 15;
	33	}
	34
	35	static int vfio_ap_mdev_create(struct kobject kobj, struct mdev_device mdev)
	36	{
	37	struct ap_matrix_mdev *matrix_mdev;
	38
	39	if ((atomic_dec_if_positive(&matrix_dev->available_instances) < 0))
	40	return -EPERM;
	41
	42	matrix_mdev = kzalloc(sizeof(*matrix_mdev), GFP_KERNEL);
	43	if (!matrix_mdev) {
	44	atomic_inc(&matrix_dev->available_instances);
	45	return -ENOMEM;
	46	}
	47
	48	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
	49	mdev_set_drvdata(mdev, matrix_mdev);
	50	mutex_lock(&matrix_dev->lock);
	51	list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
	52	mutex_unlock(&matrix_dev->lock);
	53
	54	return 0;
	55	}
	56
	57	static int vfio_ap_mdev_remove(struct mdev_device *mdev)
	58	{
	59	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	60
	61	if (matrix_mdev->kvm)
	62	return -EBUSY;
	63
	64	mutex_lock(&matrix_dev->lock);
	65	list_del(&matrix_mdev->node);
	66	mutex_unlock(&matrix_dev->lock);
	67
	68	kfree(matrix_mdev);
	69	mdev_set_drvdata(mdev, NULL);
	70	atomic_inc(&matrix_dev->available_instances);
	71
	72	return 0;
	73	}
	74
	75	static ssize_t name_show(struct kobject kobj, struct device dev, char *buf)
	76	{
	77	return sprintf(buf, "%s\n", VFIO_AP_MDEV_NAME_HWVIRT);
	78	}
	79
	80	MDEV_TYPE_ATTR_RO(name);
	81
	82	static ssize_t available_instances_show(struct kobject *kobj,
	83	struct device dev, char buf)
	84	{
	85	return sprintf(buf, "%d\n",
	86	atomic_read(&matrix_dev->available_instances));
	87	}
	88
	89	MDEV_TYPE_ATTR_RO(available_instances);
	90
	91	static ssize_t device_api_show(struct kobject kobj, struct device dev,
	92	char *buf)
	93	{
	94	return sprintf(buf, "%s\n", VFIO_DEVICE_API_AP_STRING);
	95	}
	96
	97	MDEV_TYPE_ATTR_RO(device_api);
	98
	99	static struct attribute *vfio_ap_mdev_type_attrs[] = {
	100	&mdev_type_attr_name.attr,
	101	&mdev_type_attr_device_api.attr,
	102	&mdev_type_attr_available_instances.attr,
	103	NULL,
	104	};
	105
	106	static struct attribute_group vfio_ap_mdev_hwvirt_type_group = {
	107	.name = VFIO_AP_MDEV_TYPE_HWVIRT,
	108	.attrs = vfio_ap_mdev_type_attrs,
	109	};
	110
	111	static struct attribute_group *vfio_ap_mdev_type_groups[] = {
	112	&vfio_ap_mdev_hwvirt_type_group,
	113	NULL,
	114	};
	115
	116	struct vfio_ap_queue_reserved {
	117	unsigned long *apid;
	118	unsigned long *apqi;
	119	bool reserved;
	120	};
	121
	122	/**
	123	* vfio_ap_has_queue
	124	*
	125	* @dev: an AP queue device
	126	* @data: a struct vfio_ap_queue_reserved reference
	127	*
	128	* Flags whether the AP queue device (@dev) has a queue ID containing the APQN,
	129	* apid or apqi specified in @data:
	130	*
	131	* - If @data contains both an apid and apqi value, then @data will be flagged
	132	* as reserved if the APID and APQI fields for the AP queue device matches
	133	*
	134	* - If @data contains only an apid value, @data will be flagged as
	135	* reserved if the APID field in the AP queue device matches
	136	*
	137	* - If @data contains only an apqi value, @data will be flagged as
	138	* reserved if the APQI field in the AP queue device matches
	139	*
	140	* Returns 0 to indicate the input to function succeeded. Returns -EINVAL if
	141	* @data does not contain either an apid or apqi.
	142	*/
	143	static int vfio_ap_has_queue(struct device dev, void data)
	144	{
	145	struct vfio_ap_queue_reserved *qres = data;
	146	struct ap_queue *ap_queue = to_ap_queue(dev);
	147	ap_qid_t qid;
	148	unsigned long id;
	149
	150	if (qres->apid && qres->apqi) {
	151	qid = AP_MKQID(qres->apid, qres->apqi);
	152	if (qid == ap_queue->qid)
	153	qres->reserved = true;
	154	} else if (qres->apid && !qres->apqi) {
	155	id = AP_QID_CARD(ap_queue->qid);
	156	if (id == *qres->apid)
	157	qres->reserved = true;
	158	} else if (!qres->apid && qres->apqi) {
	159	id = AP_QID_QUEUE(ap_queue->qid);
	160	if (id == *qres->apqi)
	161	qres->reserved = true;
	162	} else {
	163	return -EINVAL;
	164	}
	165
	166	return 0;
	167	}
	168
	169	/**
	170	* vfio_ap_verify_queue_reserved
	171	*
	172	* @matrix_dev: a mediated matrix device
	173	* @apid: an AP adapter ID
	174	* @apqi: an AP queue index
	175	*
	176	* Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device
	177	* driver according to the following rules:
	178	*
	179	* - If both @apid and @apqi are not NULL, then there must be an AP queue
	180	* device bound to the vfio_ap driver with the APQN identified by @apid and
	181	* @apqi
	182	*
	183	* - If only @apid is not NULL, then there must be an AP queue device bound
	184	* to the vfio_ap driver with an APQN containing @apid
	185	*
	186	* - If only @apqi is not NULL, then there must be an AP queue device bound
	187	* to the vfio_ap driver with an APQN containing @apqi
	188	*
	189	* Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL.
	190	*/
	191	static int vfio_ap_verify_queue_reserved(unsigned long *apid,
	192	unsigned long *apqi)
	193	{
	194	int ret;
	195	struct vfio_ap_queue_reserved qres;
	196
	197	qres.apid = apid;
	198	qres.apqi = apqi;
	199	qres.reserved = false;
	200
	201	ret = driver_for_each_device(matrix_dev->device.driver, NULL, &qres,
	202	vfio_ap_has_queue);
	203	if (ret)
	204	return ret;
	205
	206	if (qres.reserved)
	207	return 0;
	208
	209	return -EADDRNOTAVAIL;
	210	}
	211
	212	static int
	213	vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev,
	214	unsigned long apid)
	215	{
	216	int ret;
	217	unsigned long apqi;
	218	unsigned long nbits = matrix_mdev->matrix.aqm_max + 1;
	219
	220	if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits)
	221	return vfio_ap_verify_queue_reserved(&apid, NULL);
	222
	223	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) {
	224	ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
	225	if (ret)
	226	return ret;
	227	}
	228
	229	return 0;
	230	}
	231
	232	/**
	233	* vfio_ap_mdev_verify_no_sharing
	234	*
	235	* Verifies that the APQNs derived from the cross product of the AP adapter IDs
	236	* and AP queue indexes comprising the AP matrix are not configured for another
	237	* mediated device. AP queue sharing is not allowed.
	238	*
	239	* @matrix_mdev: the mediated matrix device
	240	*
	241	* Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE.
	242	*/
	243	static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
	244	{
	245	struct ap_matrix_mdev *lstdev;
	246	DECLARE_BITMAP(apm, AP_DEVICES);
	247	DECLARE_BITMAP(aqm, AP_DOMAINS);
	248
	249	list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
	250	if (matrix_mdev == lstdev)
	251	continue;
	252
	253	memset(apm, 0, sizeof(apm));
	254	memset(aqm, 0, sizeof(aqm));
	255
	256	/*
	257	* We work on full longs, as we can only exclude the leftover
	258	* bits in non-inverse order. The leftover is all zeros.
	259	*/
	260	if (!bitmap_and(apm, matrix_mdev->matrix.apm,
	261	lstdev->matrix.apm, AP_DEVICES))
	262	continue;
	263
	264	if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
	265	lstdev->matrix.aqm, AP_DOMAINS))
	266	continue;
	267
	268	return -EADDRINUSE;
	269	}
	270
	271	return 0;
	272	}
	273
	274	/**
	275	* assign_adapter_store
	276	*
	277	* @dev: the matrix device
	278	* @attr: the mediated matrix device's assign_adapter attribute
	279	* @buf: a buffer containing the AP adapter number (APID) to
	280	* be assigned
	281	* @count: the number of bytes in @buf
	282	*
	283	* Parses the APID from @buf and sets the corresponding bit in the mediated
	284	* matrix device's APM.
	285	*
	286	* Returns the number of bytes processed if the APID is valid; otherwise,
	287	* returns one of the following errors:
	288	*
	289	* 1. -EINVAL
	290	* The APID is not a valid number
	291	*
	292	* 2. -ENODEV
	293	* The APID exceeds the maximum value configured for the system
	294	*
	295	* 3. -EADDRNOTAVAIL
	296	* An APQN derived from the cross product of the APID being assigned
	297	* and the APQIs previously assigned is not bound to the vfio_ap device
	298	* driver; or, if no APQIs have yet been assigned, the APID is not
	299	* contained in an APQN bound to the vfio_ap device driver.
	300	*
	301	* 4. -EADDRINUSE
	302	* An APQN derived from the cross product of the APID being assigned
	303	* and the APQIs previously assigned is being used by another mediated
	304	* matrix device
	305	*/
	306	static ssize_t assign_adapter_store(struct device *dev,
	307	struct device_attribute *attr,
	308	const char *buf, size_t count)
	309	{
	310	int ret;
	311	unsigned long apid;
	312	struct mdev_device *mdev = mdev_from_dev(dev);
	313	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	314
	315	/* If the guest is running, disallow assignment of adapter */
	316	if (matrix_mdev->kvm)
	317	return -EBUSY;
	318
	319	ret = kstrtoul(buf, 0, &apid);
	320	if (ret)
	321	return ret;
	322
	323	if (apid > matrix_mdev->matrix.apm_max)
	324	return -ENODEV;
	325
	326	/*
	327	* Set the bit in the AP mask (APM) corresponding to the AP adapter
	328	* number (APID). The bits in the mask, from most significant to least
	329	* significant bit, correspond to APIDs 0-255.
	330	*/
	331	mutex_lock(&matrix_dev->lock);
	332
	333	ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid);
	334	if (ret)
	335	goto done;
	336
	337	set_bit_inv(apid, matrix_mdev->matrix.apm);
	338
	339	ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
	340	if (ret)
	341	goto share_err;
	342
	343	ret = count;
	344	goto done;
	345
	346	share_err:
	347	clear_bit_inv(apid, matrix_mdev->matrix.apm);
	348	done:
	349	mutex_unlock(&matrix_dev->lock);
	350
	351	return ret;
	352	}
	353	static DEVICE_ATTR_WO(assign_adapter);
	354
	355	/**
	356	* unassign_adapter_store
	357	*
	358	* @dev: the matrix device
	359	* @attr: the mediated matrix device's unassign_adapter attribute
	360	* @buf: a buffer containing the adapter number (APID) to be unassigned
	361	* @count: the number of bytes in @buf
	362	*
	363	* Parses the APID from @buf and clears the corresponding bit in the mediated
	364	* matrix device's APM.
	365	*
	366	* Returns the number of bytes processed if the APID is valid; otherwise,
	367	* returns one of the following errors:
	368	* -EINVAL if the APID is not a number
	369	* -ENODEV if the APID it exceeds the maximum value configured for the
	370	* system
	371	*/
	372	static ssize_t unassign_adapter_store(struct device *dev,
	373	struct device_attribute *attr,
	374	const char *buf, size_t count)
	375	{
	376	int ret;
	377	unsigned long apid;
	378	struct mdev_device *mdev = mdev_from_dev(dev);
	379	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	380
	381	/* If the guest is running, disallow un-assignment of adapter */
	382	if (matrix_mdev->kvm)
	383	return -EBUSY;
	384
	385	ret = kstrtoul(buf, 0, &apid);
	386	if (ret)
	387	return ret;
	388
	389	if (apid > matrix_mdev->matrix.apm_max)
	390	return -ENODEV;
	391
	392	mutex_lock(&matrix_dev->lock);
	393	clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
	394	mutex_unlock(&matrix_dev->lock);
	395
	396	return count;
	397	}
	398	DEVICE_ATTR_WO(unassign_adapter);
	399
	400	static int
	401	vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev,
	402	unsigned long apqi)
	403	{
	404	int ret;
	405	unsigned long apid;
	406	unsigned long nbits = matrix_mdev->matrix.apm_max + 1;
	407
	408	if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits)
	409	return vfio_ap_verify_queue_reserved(NULL, &apqi);
	410
	411	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) {
	412	ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
	413	if (ret)
	414	return ret;
	415	}
	416
	417	return 0;
	418	}
	419
	420	/**
	421	* assign_domain_store
	422	*
	423	* @dev: the matrix device
	424	* @attr: the mediated matrix device's assign_domain attribute
	425	* @buf: a buffer containing the AP queue index (APQI) of the domain to
	426	* be assigned
	427	* @count: the number of bytes in @buf
	428	*
	429	* Parses the APQI from @buf and sets the corresponding bit in the mediated
	430	* matrix device's AQM.
	431	*
	432	* Returns the number of bytes processed if the APQI is valid; otherwise returns
	433	* one of the following errors:
	434	*
	435	* 1. -EINVAL
	436	* The APQI is not a valid number
	437	*
	438	* 2. -ENODEV
	439	* The APQI exceeds the maximum value configured for the system
	440	*
	441	* 3. -EADDRNOTAVAIL
	442	* An APQN derived from the cross product of the APQI being assigned
	443	* and the APIDs previously assigned is not bound to the vfio_ap device
	444	* driver; or, if no APIDs have yet been assigned, the APQI is not
	445	* contained in an APQN bound to the vfio_ap device driver.
	446	*
	447	* 4. -EADDRINUSE
	448	* An APQN derived from the cross product of the APQI being assigned
	449	* and the APIDs previously assigned is being used by another mediated
	450	* matrix device
	451	*/
	452	static ssize_t assign_domain_store(struct device *dev,
	453	struct device_attribute *attr,
	454	const char *buf, size_t count)
	455	{
	456	int ret;
	457	unsigned long apqi;
	458	struct mdev_device *mdev = mdev_from_dev(dev);
	459	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	460	unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
	461
	462	/* If the guest is running, disallow assignment of domain */
	463	if (matrix_mdev->kvm)
	464	return -EBUSY;
	465
	466	ret = kstrtoul(buf, 0, &apqi);
	467	if (ret)
	468	return ret;
	469	if (apqi > max_apqi)
	470	return -ENODEV;
	471
	472	mutex_lock(&matrix_dev->lock);
	473
	474	ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi);
	475	if (ret)
	476	goto done;
	477
	478	set_bit_inv(apqi, matrix_mdev->matrix.aqm);
	479
	480	ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
	481	if (ret)
	482	goto share_err;
	483
	484	ret = count;
	485	goto done;
	486
	487	share_err:
	488	clear_bit_inv(apqi, matrix_mdev->matrix.aqm);
	489	done:
	490	mutex_unlock(&matrix_dev->lock);
	491
	492	return ret;
	493	}
	494	DEVICE_ATTR_WO(assign_domain);
	495
	496
	497	/**
	498	* unassign_domain_store
	499	*
	500	* @dev: the matrix device
	501	* @attr: the mediated matrix device's unassign_domain attribute
	502	* @buf: a buffer containing the AP queue index (APQI) of the domain to
	503	* be unassigned
	504	* @count: the number of bytes in @buf
	505	*
	506	* Parses the APQI from @buf and clears the corresponding bit in the
	507	* mediated matrix device's AQM.
	508	*
	509	* Returns the number of bytes processed if the APQI is valid; otherwise,
	510	* returns one of the following errors:
	511	* -EINVAL if the APQI is not a number
	512	* -ENODEV if the APQI exceeds the maximum value configured for the system
	513	*/
	514	static ssize_t unassign_domain_store(struct device *dev,
	515	struct device_attribute *attr,
	516	const char *buf, size_t count)
	517	{
	518	int ret;
	519	unsigned long apqi;
	520	struct mdev_device *mdev = mdev_from_dev(dev);
	521	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	522
	523	/* If the guest is running, disallow un-assignment of domain */
	524	if (matrix_mdev->kvm)
	525	return -EBUSY;
	526
	527	ret = kstrtoul(buf, 0, &apqi);
	528	if (ret)
	529	return ret;
	530
	531	if (apqi > matrix_mdev->matrix.aqm_max)
	532	return -ENODEV;
	533
	534	mutex_lock(&matrix_dev->lock);
	535	clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
	536	mutex_unlock(&matrix_dev->lock);
	537
	538	return count;
	539	}
	540	DEVICE_ATTR_WO(unassign_domain);
	541
	542	/**
	543	* assign_control_domain_store
	544	*
	545	* @dev: the matrix device
	546	* @attr: the mediated matrix device's assign_control_domain attribute
	547	* @buf: a buffer containing the domain ID to be assigned
	548	* @count: the number of bytes in @buf
	549	*
	550	* Parses the domain ID from @buf and sets the corresponding bit in the mediated
	551	* matrix device's ADM.
	552	*
	553	* Returns the number of bytes processed if the domain ID is valid; otherwise,
	554	* returns one of the following errors:
	555	* -EINVAL if the ID is not a number
	556	* -ENODEV if the ID exceeds the maximum value configured for the system
	557	*/
	558	static ssize_t assign_control_domain_store(struct device *dev,
	559	struct device_attribute *attr,
	560	const char *buf, size_t count)
	561	{
	562	int ret;
	563	unsigned long id;
	564	struct mdev_device *mdev = mdev_from_dev(dev);
	565	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	566
	567	/* If the guest is running, disallow assignment of control domain */
	568	if (matrix_mdev->kvm)
	569	return -EBUSY;
	570
	571	ret = kstrtoul(buf, 0, &id);
	572	if (ret)
	573	return ret;
	574
	575	if (id > matrix_mdev->matrix.adm_max)
	576	return -ENODEV;
	577
	578	/* Set the bit in the ADM (bitmask) corresponding to the AP control
	579	* domain number (id). The bits in the mask, from most significant to
	580	* least significant, correspond to IDs 0 up to the one less than the
	581	* number of control domains that can be assigned.
	582	*/
	583	mutex_lock(&matrix_dev->lock);
	584	set_bit_inv(id, matrix_mdev->matrix.adm);
	585	mutex_unlock(&matrix_dev->lock);
	586
	587	return count;
	588	}
	589	DEVICE_ATTR_WO(assign_control_domain);
	590
	591	/**
	592	* unassign_control_domain_store
	593	*
	594	* @dev: the matrix device
	595	* @attr: the mediated matrix device's unassign_control_domain attribute
	596	* @buf: a buffer containing the domain ID to be unassigned
	597	* @count: the number of bytes in @buf
	598	*
	599	* Parses the domain ID from @buf and clears the corresponding bit in the
	600	* mediated matrix device's ADM.
	601	*
	602	* Returns the number of bytes processed if the domain ID is valid; otherwise,
	603	* returns one of the following errors:
	604	* -EINVAL if the ID is not a number
	605	* -ENODEV if the ID exceeds the maximum value configured for the system
	606	*/
	607	static ssize_t unassign_control_domain_store(struct device *dev,
	608	struct device_attribute *attr,
	609	const char *buf, size_t count)
	610	{
	611	int ret;
	612	unsigned long domid;
	613	struct mdev_device *mdev = mdev_from_dev(dev);
	614	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	615	unsigned long max_domid = matrix_mdev->matrix.adm_max;
	616
	617	/* If the guest is running, disallow un-assignment of control domain */
	618	if (matrix_mdev->kvm)
	619	return -EBUSY;
	620
	621	ret = kstrtoul(buf, 0, &domid);
	622	if (ret)
	623	return ret;
	624	if (domid > max_domid)
	625	return -ENODEV;
	626
	627	mutex_lock(&matrix_dev->lock);
	628	clear_bit_inv(domid, matrix_mdev->matrix.adm);
	629	mutex_unlock(&matrix_dev->lock);
	630
	631	return count;
	632	}
	633	DEVICE_ATTR_WO(unassign_control_domain);
	634
	635	static ssize_t control_domains_show(struct device *dev,
	636	struct device_attribute *dev_attr,
	637	char *buf)
	638	{
	639	unsigned long id;
	640	int nchars = 0;
	641	int n;
	642	char *bufpos = buf;
	643	struct mdev_device *mdev = mdev_from_dev(dev);
	644	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	645	unsigned long max_domid = matrix_mdev->matrix.adm_max;
	646
	647	mutex_lock(&matrix_dev->lock);
	648	for_each_set_bit_inv(id, matrix_mdev->matrix.adm, max_domid + 1) {
	649	n = sprintf(bufpos, "%04lx\n", id);
	650	bufpos += n;
	651	nchars += n;
	652	}
	653	mutex_unlock(&matrix_dev->lock);
	654
	655	return nchars;
	656	}
	657	DEVICE_ATTR_RO(control_domains);
	658
	659	static ssize_t matrix_show(struct device dev, struct device_attribute attr,
	660	char *buf)
	661	{
	662	struct mdev_device *mdev = mdev_from_dev(dev);
	663	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	664	char *bufpos = buf;
	665	unsigned long apid;
	666	unsigned long apqi;
	667	unsigned long apid1;
	668	unsigned long apqi1;
	669	unsigned long napm_bits = matrix_mdev->matrix.apm_max + 1;
	670	unsigned long naqm_bits = matrix_mdev->matrix.aqm_max + 1;
	671	int nchars = 0;
	672	int n;
	673
	674	apid1 = find_first_bit_inv(matrix_mdev->matrix.apm, napm_bits);
	675	apqi1 = find_first_bit_inv(matrix_mdev->matrix.aqm, naqm_bits);
	676
	677	mutex_lock(&matrix_dev->lock);
	678
	679	if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) {
	680	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
	681	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
	682	naqm_bits) {
	683	n = sprintf(bufpos, "%02lx.%04lx\n", apid,
	684	apqi);
	685	bufpos += n;
	686	nchars += n;
	687	}
	688	}
	689	} else if (apid1 < napm_bits) {
	690	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
	691	n = sprintf(bufpos, "%02lx.\n", apid);
	692	bufpos += n;
	693	nchars += n;
	694	}
	695	} else if (apqi1 < naqm_bits) {
	696	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, naqm_bits) {
	697	n = sprintf(bufpos, ".%04lx\n", apqi);
	698	bufpos += n;
	699	nchars += n;
	700	}
	701	}
	702
	703	mutex_unlock(&matrix_dev->lock);
	704
	705	return nchars;
	706	}
	707	DEVICE_ATTR_RO(matrix);
	708
	709	static struct attribute *vfio_ap_mdev_attrs[] = {
	710	&dev_attr_assign_adapter.attr,
	711	&dev_attr_unassign_adapter.attr,
	712	&dev_attr_assign_domain.attr,
	713	&dev_attr_unassign_domain.attr,
	714	&dev_attr_assign_control_domain.attr,
	715	&dev_attr_unassign_control_domain.attr,
	716	&dev_attr_control_domains.attr,
	717	&dev_attr_matrix.attr,
	718	NULL,
	719	};
	720
	721	static struct attribute_group vfio_ap_mdev_attr_group = {
	722	.attrs = vfio_ap_mdev_attrs
	723	};
	724
	725	static const struct attribute_group *vfio_ap_mdev_attr_groups[] = {
	726	&vfio_ap_mdev_attr_group,
	727	NULL
	728	};
	729
	730	static void vfio_ap_mdev_copy_masks(struct ap_matrix_mdev *matrix_mdev)
	731	{
	732	int nbytes;
	733	unsigned long apm, aqm, *adm;
	734	struct kvm_s390_crypto_cb *crycb = matrix_mdev->kvm->arch.crypto.crycb;
	735
	736	switch (matrix_mdev->kvm->arch.crypto.crycbd & CRYCB_FORMAT_MASK) {
	737	case CRYCB_FORMAT2:
	738	apm = (unsigned long *)crycb->apcb1.apm;
	739	aqm = (unsigned long *)crycb->apcb1.aqm;
	740	adm = (unsigned long *)crycb->apcb1.adm;
	741	break;
	742	case CRYCB_FORMAT1:
	743	case CRYCB_FORMAT0:
	744	apm = (unsigned long *)crycb->apcb0.apm;
	745	aqm = (unsigned long *)crycb->apcb0.aqm;
	746	adm = (unsigned long *)crycb->apcb0.adm;
	747	break;
	748	default:
	749	/* cannot happen */
	750	return;
	751	}
	752
	753	nbytes = DIV_ROUND_UP(matrix_mdev->matrix.apm_max + 1, BITS_PER_BYTE);
	754	memcpy(apm, matrix_mdev->matrix.apm, nbytes);
	755	nbytes = DIV_ROUND_UP(matrix_mdev->matrix.aqm_max + 1, BITS_PER_BYTE);
	756	memcpy(aqm, matrix_mdev->matrix.aqm, nbytes);
	757	nbytes = DIV_ROUND_UP(matrix_mdev->matrix.adm_max + 1, BITS_PER_BYTE);
	758	memcpy(adm, matrix_mdev->matrix.adm, nbytes);
	759	}
	760
	761	/**
	762	* vfio_ap_mdev_set_kvm
	763	*
	764	* @matrix_mdev: a mediated matrix device
	765	* @kvm: reference to KVM instance
	766	*
	767	* Verifies no other mediated matrix device has @kvm and sets a reference to
	768	* it in @matrix_mdev->kvm.
	769	*
	770	* Return 0 if no other mediated matrix device has a reference to @kvm;
	771	* otherwise, returns an -EPERM.
	772	*/
	773	static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,
	774	struct kvm *kvm)
	775	{
	776	struct ap_matrix_mdev *m;
	777
	778	mutex_lock(&matrix_dev->lock);
	779
	780	list_for_each_entry(m, &matrix_dev->mdev_list, node) {
	781	if ((m != matrix_mdev) && (m->kvm == kvm)) {
	782	mutex_unlock(&matrix_dev->lock);
	783	return -EPERM;
	784	}
	785	}
	786
	787	matrix_mdev->kvm = kvm;
	788	mutex_unlock(&matrix_dev->lock);
	789
	790	return 0;
	791	}
	792
	793	static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
	794	unsigned long action, void *data)
	795	{
	796	int ret;
	797	struct ap_matrix_mdev *matrix_mdev;
	798
	799	if (action != VFIO_GROUP_NOTIFY_SET_KVM)
	800	return NOTIFY_OK;
	801
	802	matrix_mdev = container_of(nb, struct ap_matrix_mdev, group_notifier);
	803
	804	if (!data) {
	805	matrix_mdev->kvm = NULL;
	806	return NOTIFY_OK;
	807	}
	808
	809	ret = vfio_ap_mdev_set_kvm(matrix_mdev, data);
	810	if (ret)
	811	return NOTIFY_DONE;
	812
	813	/* If there is no CRYCB pointer, then we can't copy the masks */
	814	if (!matrix_mdev->kvm->arch.crypto.crycbd)
	815	return NOTIFY_DONE;
	816
	817	vfio_ap_mdev_copy_masks(matrix_mdev);
	818
	819	return NOTIFY_OK;
	820	}
	821
	822	static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi,
	823	unsigned int retry)
	824	{
	825	struct ap_queue_status status;
	826
	827	do {
	828	status = ap_zapq(AP_MKQID(apid, apqi));
	829	switch (status.response_code) {
	830	case AP_RESPONSE_NORMAL:
	831	return 0;
	832	case AP_RESPONSE_RESET_IN_PROGRESS:
	833	case AP_RESPONSE_BUSY:
	834	msleep(20);
	835	break;
	836	default:
	837	/* things are really broken, give up */
	838	return -EIO;
	839	}
	840	} while (retry--);
	841
	842	return -EBUSY;
	843	}
	844
	845	static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev)
	846	{
	847	int ret;
	848	int rc = 0;
	849	unsigned long apid, apqi;
	850	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	851
	852	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
	853	matrix_mdev->matrix.apm_max + 1) {
	854	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
	855	matrix_mdev->matrix.aqm_max + 1) {
	856	ret = vfio_ap_mdev_reset_queue(apid, apqi, 1);
	857	/*
	858	* Regardless whether a queue turns out to be busy, or
	859	* is not operational, we need to continue resetting
	860	* the remaining queues.
	861	*/
	862	if (ret)
	863	rc = ret;
	864	}
	865	}
	866
	867	return rc;
	868	}
	869
	870	static int vfio_ap_mdev_open(struct mdev_device *mdev)
	871	{
	872	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	873	unsigned long events;
	874	int ret;
	875
	876
	877	if (!try_module_get(THIS_MODULE))
	878	return -ENODEV;
	879
	880	matrix_mdev->group_notifier.notifier_call = vfio_ap_mdev_group_notifier;
	881	events = VFIO_GROUP_NOTIFY_SET_KVM;
	882
	883	ret = vfio_register_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
	884	&events, &matrix_mdev->group_notifier);
	885	if (ret) {
	886	module_put(THIS_MODULE);
	887	return ret;
	888	}
	889
	890	return 0;
	891	}
	892
	893	static void vfio_ap_mdev_release(struct mdev_device *mdev)
	894	{
	895	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
	896
	897	if (matrix_mdev->kvm)
	898	kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
	899
	900	vfio_ap_mdev_reset_queues(mdev);
	901	vfio_unregister_notifier(mdev_dev(mdev), VFIO_GROUP_NOTIFY,
	902	&matrix_mdev->group_notifier);
	903	matrix_mdev->kvm = NULL;
	904	module_put(THIS_MODULE);
	905	}
	906
	907	static int vfio_ap_mdev_get_device_info(unsigned long arg)
	908	{
	909	unsigned long minsz;
	910	struct vfio_device_info info;
	911
	912	minsz = offsetofend(struct vfio_device_info, num_irqs);
	913
	914	if (copy_from_user(&info, (void __user *)arg, minsz))
	915	return -EFAULT;
	916
	917	if (info.argsz < minsz)
	918	return -EINVAL;
	919
	920	info.flags = VFIO_DEVICE_FLAGS_AP \| VFIO_DEVICE_FLAGS_RESET;
	921	info.num_regions = 0;
	922	info.num_irqs = 0;
	923
	924	return copy_to_user((void __user *)arg, &info, minsz);
	925	}
	926
	927	static ssize_t vfio_ap_mdev_ioctl(struct mdev_device *mdev,
	928	unsigned int cmd, unsigned long arg)
	929	{
	930	int ret;
	931
	932	switch (cmd) {
	933	case VFIO_DEVICE_GET_INFO:
	934	ret = vfio_ap_mdev_get_device_info(arg);
	935	break;
	936	case VFIO_DEVICE_RESET:
	937	ret = vfio_ap_mdev_reset_queues(mdev);
	938	break;
	939	default:
	940	ret = -EOPNOTSUPP;
	941	break;
	942	}
	943
	944	return ret;
	945	}
	946
	947	static const struct mdev_parent_ops vfio_ap_matrix_ops = {
	948	.owner = THIS_MODULE,
	949	.supported_type_groups = vfio_ap_mdev_type_groups,
	950	.mdev_attr_groups = vfio_ap_mdev_attr_groups,
	951	.create = vfio_ap_mdev_create,
	952	.remove = vfio_ap_mdev_remove,
	953	.open = vfio_ap_mdev_open,
	954	.release = vfio_ap_mdev_release,
	955	.ioctl = vfio_ap_mdev_ioctl,
	956	};
	957
	958	int vfio_ap_mdev_register(void)
	959	{
	960	atomic_set(&matrix_dev->available_instances, MAX_ZDEV_ENTRIES_EXT);
	961
	962	return mdev_register_device(&matrix_dev->device, &vfio_ap_matrix_ops);
	963	}
	964
	965	void vfio_ap_mdev_unregister(void)
	966	{
	967	mdev_unregister_device(&matrix_dev->device);
	968	}

drivers/s390/crypto/vfio_ap_private.h+88−0

@@ -0,0 +1,88 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* Private data and functions for adjunct processor VFIO matrix driver.
	4	*
	5	* Author(s): Tony Krowiak <akrowiak@linux.ibm.com>
	6	* Halil Pasic <pasic@linux.ibm.com>
	7	*
	8	* Copyright IBM Corp. 2018
	9	*/
	10
	11	#ifndef _VFIO_AP_PRIVATE_H_
	12	#define _VFIO_AP_PRIVATE_H_
	13
	14	#include <linux/types.h>
	15	#include <linux/device.h>
	16	#include <linux/mdev.h>
	17	#include <linux/delay.h>
	18	#include <linux/mutex.h>
	19
	20	#include "ap_bus.h"
	21
	22	#define VFIO_AP_MODULE_NAME "vfio_ap"
	23	#define VFIO_AP_DRV_NAME "vfio_ap"
	24
	25	/**
	26	* ap_matrix_dev - the AP matrix device structure
	27	* @device: generic device structure associated with the AP matrix device
	28	* @available_instances: number of mediated matrix devices that can be created
	29	* @info: the struct containing the output from the PQAP(QCI) instruction
	30	* mdev_list: the list of mediated matrix devices created
	31	* lock: mutex for locking the AP matrix device. This lock will be
	32	* taken every time we fiddle with state managed by the vfio_ap
	33	* driver, be it using @mdev_list or writing the state of a
	34	* single ap_matrix_mdev device. It's quite coarse but we don't
	35	* expect much contention.
	36	*/
	37	struct ap_matrix_dev {
	38	struct device device;
	39	atomic_t available_instances;
	40	struct ap_config_info info;
	41	struct list_head mdev_list;
	42	struct mutex lock;
	43	};
	44
	45	extern struct ap_matrix_dev *matrix_dev;
	46
	47	/**
	48	* The AP matrix is comprised of three bit masks identifying the adapters,
	49	* queues (domains) and control domains that belong to an AP matrix. The bits i
	50	* each mask, from least significant to most significant bit, correspond to IDs
	51	* 0 to 255. When a bit is set, the corresponding ID belongs to the matrix.
	52	*
	53	* @apm_max: max adapter number in @apm
	54	* @apm identifies the AP adapters in the matrix
	55	* @aqm_max: max domain number in @aqm
	56	* @aqm identifies the AP queues (domains) in the matrix
	57	* @adm_max: max domain number in @adm
	58	* @adm identifies the AP control domains in the matrix
	59	*/
	60	struct ap_matrix {
	61	unsigned long apm_max;
	62	DECLARE_BITMAP(apm, 256);
	63	unsigned long aqm_max;
	64	DECLARE_BITMAP(aqm, 256);
	65	unsigned long adm_max;
	66	DECLARE_BITMAP(adm, 256);
	67	};
	68
	69	/**
	70	* struct ap_matrix_mdev - the mediated matrix device structure
	71	* @list: allows the ap_matrix_mdev struct to be added to a list
	72	* @matrix: the adapters, usage domains and control domains assigned to the
	73	* mediated matrix device.
	74	* @group_notifier: notifier block used for specifying callback function for
	75	* handling the VFIO_GROUP_NOTIFY_SET_KVM event
	76	* @kvm: the struct holding guest's state
	77	*/
	78	struct ap_matrix_mdev {
	79	struct list_head node;
	80	struct ap_matrix matrix;
	81	struct notifier_block group_notifier;
	82	struct kvm *kvm;
	83	};
	84
	85	extern int vfio_ap_mdev_register(void);
	86	extern void vfio_ap_mdev_unregister(void);
	87
	88	#endif /* _VFIO_AP_PRIVATE_H_ */

include/uapi/linux/vfio.h+2−0

 #define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2)	/* vfio-platform device */
 #define VFIO_DEVICE_FLAGS_AMBA  (1 << 3)	/* vfio-amba device */
 #define VFIO_DEVICE_FLAGS_CCW	(1 << 4)	/* vfio-ccw device */
+#define VFIO_DEVICE_FLAGS_AP	(1 << 5)	/* vfio-ap device */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 };
 #define VFIO_DEVICE_API_PLATFORM_STRING		"vfio-platform"
 #define VFIO_DEVICE_API_AMBA_STRING		"vfio-amba"
 #define VFIO_DEVICE_API_CCW_STRING		"vfio-ccw"
+#define VFIO_DEVICE_API_AP_STRING		"vfio-ap"
 /**
  * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,