Backmerging to pick up a fix from drm-misc-next-fixes.
Unified · Split
| @@ -127,3 +127,7 @@ all.config · 126 unmodified lines | |||
| 127 | 127 | ||
| 128 | # Kdevelop4 | 128 | # Kdevelop4 |
| 129 | *.kdev4 | 129 | *.kdev4 |
| 130 | |||
| 131 | #Automatically generated by ASN.1 compiler | ||
| 132 | net/ipv4/netfilter/nf_nat_snmp_basic-asn1.c | ||
| 133 | net/ipv4/netfilter/nf_nat_snmp_basic-asn1.h | ||
| @@ -1,110 +1,139 @@ | |||
| 1 | What: /sys/class/ata_... | 1 | What: /sys/class/ata_... |
| 2 | Date: August 2008 | ||
| 3 | Contact: Gwendal Grignou<gwendal@google.com> | ||
| 4 | Description: | 2 | Description: |
| 5 | 3 | Provide a place in sysfs for storing the ATA topology of the | |
| 6 | Provide a place in sysfs for storing the ATA topology of the system. This allows | 4 | system. This allows retrieving various information about ATA |
| 7 | retrieving various information about ATA objects. | 5 | objects. |
| 8 | 6 | ||
| 9 | Files under /sys/class/ata_port | 7 | Files under /sys/class/ata_port |
| 10 | ------------------------------- | 8 | ------------------------------- |
| 11 | 9 | ||
| 12 | For each port, a directory ataX is created where X is the ata_port_id of | 10 | For each port, a directory ataX is created where X is the ata_port_id of the |
| 13 | the port. The device parent is the ata host device. | 11 | port. The device parent is the ata host device. |
| 14 | 12 | ||
| 15 | idle_irq (read) | ||
| 16 | 13 | ||
| 17 | Number of IRQ received by the port while idle [some ata HBA only]. | 14 | What: /sys/class/ata_port/ataX/nr_pmp_links |
| 15 | What: /sys/class/ata_port/ataX/idle_irq | ||
| 16 | Date: May, 2010 | ||
| 17 | KernelVersion: v2.6.37 | ||
| 18 | Contact: Gwendal Grignou <gwendal@chromium.org> | ||
| 19 | Description: | ||
| 20 | nr_pmp_links: (RO) If a SATA Port Multiplier (PM) is | ||
| 21 | connected, the number of links behind it. | ||
| 18 | 22 | ||
| 19 | nr_pmp_links (read) | 23 | idle_irq: (RO) Number of IRQ received by the port while |
| 24 | idle [some ata HBA only]. | ||
| 20 | 25 | ||
| 21 | If a SATA Port Multiplier (PM) is connected, number of link behind it. | 26 | |
| 27 | What: /sys/class/ata_port/ataX/port_no | ||
| 28 | Date: May, 2013 | ||
| 29 | KernelVersion: v3.11 | ||
| 30 | Contact: Gwendal Grignou <gwendal@chromium.org> | ||
| 31 | Description: | ||
| 32 | (RO) Host local port number. While registering host controller, | ||
| 33 | port numbers are tracked based upon number of ports available on | ||
| 34 | the controller. This attribute is needed by udev for composing | ||
| 35 | persistent links in /dev/disk/by-path. | ||
| 22 | 36 | ||
| 23 | Files under /sys/class/ata_link | 37 | Files under /sys/class/ata_link |
| 24 | ------------------------------- | 38 | ------------------------------- |
| 25 | 39 | ||
| 26 | Behind each port, there is a ata_link. If there is a SATA PM in the | 40 | Behind each port, there is a ata_link. If there is a SATA PM in the topology, 15 |
| 27 | topology, 15 ata_link objects are created. | 41 | ata_link objects are created. |
| 28 | |||
| 29 | If a link is behind a port, the directory name is linkX, where X is | ||
| 30 | ata_port_id of the port. | ||
| 31 | If a link is behind a PM, its name is linkX.Y where X is ata_port_id | ||
| 32 | of the parent port and Y the PM port. | ||
| 33 | 42 | ||
| 34 | hw_sata_spd_limit | 43 | If a link is behind a port, the directory name is linkX, where X is ata_port_id |
| 44 | of the port. If a link is behind a PM, its name is linkX.Y where X is | ||
| 45 | ata_port_id of the parent port and Y the PM port. | ||
| 35 | 46 | ||
| 36 | Maximum speed supported by the connected SATA device. | ||
| 37 | 47 | ||
| 38 | sata_spd_limit | 48 | What: /sys/class/ata_link/linkX[.Y]/hw_sata_spd_limit |
| 49 | What: /sys/class/ata_link/linkX[.Y]/sata_spd_limit | ||
| 50 | What: /sys/class/ata_link/linkX[.Y]/sata_spd | ||
| 51 | Date: May, 2010 | ||
| 52 | KernelVersion: v2.6.37 | ||
| 53 | Contact: Gwendal Grignou <gwendal@chromium.org> | ||
| 54 | Description: | ||
| 55 | hw_sata_spd_limit: (RO) Maximum speed supported by the | ||
| 56 | connected SATA device. | ||
| 39 | 57 | ||
| 40 | Maximum speed imposed by libata. | 58 | sata_spd_limit: (RO) Maximum speed imposed by libata. |
| 41 | 59 | ||
| 42 | sata_spd | 60 | sata_spd: (RO) Current speed of the link |
| 61 | eg. 1.5, 3 Gbps etc. | ||
| 43 | 62 | ||
| 44 | Current speed of the link [1.5, 3Gps,...]. | ||
| 45 | 63 | ||
| 46 | Files under /sys/class/ata_device | 64 | Files under /sys/class/ata_device |
| 47 | --------------------------------- | 65 | --------------------------------- |
| 48 | 66 | ||
| 49 | Behind each link, up to two ata device are created. | 67 | Behind each link, up to two ata devices are created. |
| 50 | The name of the directory is devX[.Y].Z where: | 68 | The name of the directory is devX[.Y].Z where: |
| 51 | - X is ata_port_id of the port where the device is connected, | 69 | - X is ata_port_id of the port where the device is connected, |
| 52 | - Y the port of the PM if any, and | 70 | - Y the port of the PM if any, and |
| 53 | - Z the device id: for PATA, there is usually 2 devices [0,1], | 71 | - Z the device id: for PATA, there is usually 2 devices [0,1], only 1 for SATA. |
| 54 | only 1 for SATA. | 72 | |
| 55 | 73 | ||
| 56 | class | 74 | What: /sys/class/ata_device/devX[.Y].Z/spdn_cnt |
| 57 | Device class. Can be "ata" for disk, "atapi" for packet device, | 75 | What: /sys/class/ata_device/devX[.Y].Z/gscr |
| 58 | "pmp" for PM, or "none" if no device was found behind the link. | 76 | What: /sys/class/ata_device/devX[.Y].Z/ering |
| 59 | 77 | What: /sys/class/ata_device/devX[.Y].Z/id | |
| 60 | dma_mode | 78 | What: /sys/class/ata_device/devX[.Y].Z/pio_mode |
| 79 | What: /sys/class/ata_device/devX[.Y].Z/xfer_mode | ||
| 80 | What: /sys/class/ata_device/devX[.Y].Z/dma_mode | ||
| 81 | What: /sys/class/ata_device/devX[.Y].Z/class | ||
| 82 | Date: May, 2010 | ||
| 83 | KernelVersion: v2.6.37 | ||
| 84 | Contact: Gwendal Grignou <gwendal@chromium.org> | ||
| 85 | Description: | ||
| 86 | spdn_cnt: (RO) Number of times libata decided to lower the | ||
| 87 | speed of link due to errors. | ||
| 61 | 88 | ||
| 62 | Transfer modes supported by the device when in DMA mode. | 89 | gscr: (RO) Cached result of the dump of PM GSCR |
| 63 | Mostly used by PATA device. | 90 | register. Valid registers are: |
| 64 | 91 | ||
| 65 | pio_mode | 92 | 0: SATA_PMP_GSCR_PROD_ID, |
| 93 | 1: SATA_PMP_GSCR_REV, | ||
| 94 | 2: SATA_PMP_GSCR_PORT_INFO, | ||
| 95 | 32: SATA_PMP_GSCR_ERROR, | ||
| 96 | 33: SATA_PMP_GSCR_ERROR_EN, | ||
| 97 | 64: SATA_PMP_GSCR_FEAT, | ||
| 98 | 96: SATA_PMP_GSCR_FEAT_EN, | ||
| 99 | 130: SATA_PMP_GSCR_SII_GPIO | ||
| 66 | 100 | ||
| 67 | Transfer modes supported by the device when in PIO mode. | 101 | Only valid if the device is a PM. |
| 68 | Mostly used by PATA device. | ||
| 69 | 102 | ||
| 70 | xfer_mode | 103 | ering: (RO) Formatted output of the error ring of the |
| 104 | device. | ||
| 71 | 105 | ||
| 72 | Current transfer mode. | 106 | id: (RO) Cached result of IDENTIFY command, as |
| 107 | described in ATA8 7.16 and 7.17. Only valid if | ||
| 108 | the device is not a PM. | ||
| 73 | 109 | ||
| 74 | id | 110 | pio_mode: (RO) Transfer modes supported by the device when |
| 111 | in PIO mode. Mostly used by PATA device. | ||
| 75 | 112 | ||
| 76 | Cached result of IDENTIFY command, as described in ATA8 7.16 and 7.17. | 113 | xfer_mode: (RO) Current transfer mode |
| 77 | Only valid if the device is not a PM. | ||
| 78 | 114 | ||
| 79 | gscr | 115 | dma_mode: (RO) Transfer modes supported by the device when |
| 116 | in DMA mode. Mostly used by PATA device. | ||
| 80 | 117 | ||
| 81 | Cached result of the dump of PM GSCR register. | 118 | class: (RO) Device class. Can be "ata" for disk, |
| 82 | Valid registers are: | 119 | "atapi" for packet device, "pmp" for PM, or |
| 83 | 0: SATA_PMP_GSCR_PROD_ID, | 120 | "none" if no device was found behind the link. |
| 84 | 1: SATA_PMP_GSCR_REV, | ||
| 85 | 2: SATA_PMP_GSCR_PORT_INFO, | ||
| 86 | 32: SATA_PMP_GSCR_ERROR, | ||
| 87 | 33: SATA_PMP_GSCR_ERROR_EN, | ||
| 88 | 64: SATA_PMP_GSCR_FEAT, | ||
| 89 | 96: SATA_PMP_GSCR_FEAT_EN, | ||
| 90 | 130: SATA_PMP_GSCR_SII_GPIO | ||
| 91 | Only valid if the device is a PM. | ||
| 92 | 121 | ||
| 93 | trim | ||
| 94 | 122 | ||
| 95 | Shows the DSM TRIM mode currently used by the device. Valid | 123 | What: /sys/class/ata_device/devX[.Y].Z/trim |
| 96 | values are: | 124 | Date: May, 2015 |
| 97 | unsupported: Drive does not support DSM TRIM | 125 | KernelVersion: v4.10 |
| 98 | unqueued: Drive supports unqueued DSM TRIM only | 126 | Contact: Gwendal Grignou <gwendal@chromium.org> |
| 99 | queued: Drive supports queued DSM TRIM | 127 | Description: |
| 100 | forced_unqueued: Drive's queued DSM support is known to be | 128 | (RO) Shows the DSM TRIM mode currently used by the device. Valid |
| 101 | buggy and only unqueued TRIM commands | 129 | values are: |
| 102 | are sent | ||
| 103 | 130 | ||
| 104 | spdn_cnt | 131 | unsupported: Drive does not support DSM TRIM |
| 105 | 132 | ||
| 106 | Number of time libata decided to lower the speed of link due to errors. | 133 | unqueued: Drive supports unqueued DSM TRIM only |
| 107 | 134 | ||
| 108 | ering | 135 | queued: Drive supports queued DSM TRIM |
| 109 | 136 | ||
| 110 | Formatted output of the error ring of the device. | 137 | forced_unqueued: Drive's queued DSM support is known to |
| 138 | be buggy and only unqueued TRIM commands | ||
| 139 | are sent | ||
| @@ -0,0 +1,58 @@ | |||
| 1 | What: /sys/block/*/device/sw_activity | ||
| 2 | Date: Jun, 2008 | ||
| 3 | KernelVersion: v2.6.27 | ||
| 4 | Contact: linux-ide@vger.kernel.org | ||
| 5 | Description: | ||
| 6 | (RW) Used by drivers which support software controlled activity | ||
| 7 | LEDs. | ||
| 8 | |||
| 9 | It has the following valid values: | ||
| 10 | |||
| 11 | 0 OFF - the LED is not activated on activity | ||
| 12 | 1 BLINK_ON - the LED blinks on every 10ms when activity is | ||
| 13 | detected. | ||
| 14 | 2 BLINK_OFF - the LED is on when idle, and blinks off | ||
| 15 | every 10ms when activity is detected. | ||
| 16 | |||
| 17 | Note that the user must turn sw_activity OFF it they wish to | ||
| 18 | control the activity LED via the em_message file. | ||
| 19 | |||
| 20 | |||
| 21 | What: /sys/block/*/device/unload_heads | ||
| 22 | Date: Sep, 2008 | ||
| 23 | KernelVersion: v2.6.28 | ||
| 24 | Contact: linux-ide@vger.kernel.org | ||
| 25 | Description: | ||
| 26 | (RW) Hard disk shock protection | ||
| 27 | |||
| 28 | Writing an integer value to this file will take the heads of the | ||
| 29 | respective drive off the platter and block all I/O operations | ||
| 30 | for the specified number of milliseconds. | ||
| 31 | |||
| 32 | - If the device does not support the unload heads feature, | ||
| 33 | access is denied with -EOPNOTSUPP. | ||
| 34 | - The maximal value accepted for a timeout is 30000 | ||
| 35 | milliseconds. | ||
| 36 | - A previously set timeout can be cancelled and disk can resume | ||
| 37 | normal operation immediately by specifying a timeout of 0. | ||
| 38 | - Some hard drives only comply with an earlier version of the | ||
| 39 | ATA standard, but support the unload feature nonetheless. | ||
| 40 | There is no safe way Linux can detect these devices, so this | ||
| 41 | is not enabled by default. If it is known that your device | ||
| 42 | does support the unload feature, then you can tell the kernel | ||
| 43 | to enable it by writing -1. It can be disabled again by | ||
| 44 | writing -2. | ||
| 45 | - Values below -2 are rejected with -EINVAL | ||
| 46 | |||
| 47 | For more information, see | ||
| 48 | Documentation/laptops/disk-shock-protection.txt | ||
| 49 | |||
| 50 | |||
| 51 | What: /sys/block/*/device/ncq_prio_enable | ||
| 52 | Date: Oct, 2016 | ||
| 53 | KernelVersion: v4.10 | ||
| 54 | Contact: linux-ide@vger.kernel.org | ||
| 55 | Description: | ||
| 56 | (RW) Write to the file to turn on or off the SATA ncq (native | ||
| 57 | command queueing) support. By default this feature is turned | ||
| 58 | off. | ||
| @@ -27,3 +27,92 @@ Description: This file contains the current status of the "SSD Smart Path" · 26 unmodified lines | |||
| 27 | the direct i/o path to physical devices. This setting is | 27 | the direct i/o path to physical devices. This setting is |
| 28 | controller wide, affecting all configured logical drives on the | 28 | controller wide, affecting all configured logical drives on the |
| 29 | controller. This file is readable and writable. | 29 | controller. This file is readable and writable. |
| 30 | |||
| 31 | What: /sys/class/scsi_host/hostX/link_power_management_policy | ||
| 32 | Date: Oct, 2007 | ||
| 33 | KernelVersion: v2.6.24 | ||
| 34 | Contact: linux-ide@vger.kernel.org | ||
| 35 | Description: | ||
| 36 | (RW) This parameter allows the user to read and set the link | ||
| 37 | (interface) power management. | ||
| 38 | |||
| 39 | There are four possible options: | ||
| 40 | |||
| 41 | min_power: Tell the controller to try to make the link use the | ||
| 42 | least possible power when possible. This may sacrifice some | ||
| 43 | performance due to increased latency when coming out of lower | ||
| 44 | power states. | ||
| 45 | |||
| 46 | max_performance: Generally, this means no power management. | ||
| 47 | Tell the controller to have performance be a priority over power | ||
| 48 | management. | ||
| 49 | |||
| 50 | medium_power: Tell the controller to enter a lower power state | ||
| 51 | when possible, but do not enter the lowest power state, thus | ||
| 52 | improving latency over min_power setting. | ||
| 53 | |||
| 54 | med_power_with_dipm: Identical to the existing medium_power | ||
| 55 | setting except that it enables dipm (device initiated power | ||
| 56 | management) on top, which makes it match the Windows IRST (Intel | ||
| 57 | Rapid Storage Technology) driver settings. This setting is also | ||
| 58 | close to min_power, except that: | ||
| 59 | a) It does not use host-initiated slumber mode, but it does | ||
| 60 | allow device-initiated slumber | ||
| 61 | b) It does not enable low power device sleep mode (DevSlp). | ||
| 62 | |||
| 63 | What: /sys/class/scsi_host/hostX/em_message | ||
| 64 | What: /sys/class/scsi_host/hostX/em_message_type | ||
| 65 | Date: Jun, 2008 | ||
| 66 | KernelVersion: v2.6.27 | ||
| 67 | Contact: linux-ide@vger.kernel.org | ||
| 68 | Description: | ||
| 69 | em_message: (RW) Enclosure management support. For the LED | ||
| 70 | protocol, writes and reads correspond to the LED message format | ||
| 71 | as defined in the AHCI spec. | ||
| 72 | |||
| 73 | The user must turn sw_activity (under /sys/block/*/device/) OFF | ||
| 74 | it they wish to control the activity LED via the em_message | ||
| 75 | file. | ||
| 76 | |||
| 77 | em_message_type: (RO) Displays the current enclosure management | ||
| 78 | protocol that is being used by the driver (for eg. LED, SAF-TE, | ||
| 79 | SES-2, SGPIO etc). | ||
| 80 | |||
| 81 | What: /sys/class/scsi_host/hostX/ahci_port_cmd | ||
| 82 | What: /sys/class/scsi_host/hostX/ahci_host_caps | ||
| 83 | What: /sys/class/scsi_host/hostX/ahci_host_cap2 | ||
| 84 | Date: Mar, 2010 | ||
| 85 | KernelVersion: v2.6.35 | ||
| 86 | Contact: linux-ide@vger.kernel.org | ||
| 87 | Description: | ||
| 88 | [to be documented] | ||
| 89 | |||
| 90 | What: /sys/class/scsi_host/hostX/ahci_host_version | ||
| 91 | Date: Mar, 2010 | ||
| 92 | KernelVersion: v2.6.35 | ||
| 93 | Contact: linux-ide@vger.kernel.org | ||
| 94 | Description: | ||
| 95 | (RO) Display the version of the AHCI spec implemented by the | ||
| 96 | host. | ||
| 97 | |||
| 98 | What: /sys/class/scsi_host/hostX/em_buffer | ||
| 99 | Date: Apr, 2010 | ||
| 100 | KernelVersion: v2.6.35 | ||
| 101 | Contact: linux-ide@vger.kernel.org | ||
| 102 | Description: | ||
| 103 | (RW) Allows access to AHCI EM (enclosure management) buffer | ||
| 104 | directly if the host supports EM. | ||
| 105 | |||
| 106 | For eg. the AHCI driver supports SGPIO EM messages but the | ||
| 107 | SATA/AHCI specs do not define the SGPIO message format of the EM | ||
| 108 | buffer. Different hardware(HW) vendors may have different | ||
| 109 | definitions. With the em_buffer attribute, this issue can be | ||
| 110 | solved by allowing HW vendors to provide userland drivers and | ||
| 111 | tools for their SGPIO initiators. | ||
| 112 | |||
| 113 | What: /sys/class/scsi_host/hostX/em_message_supported | ||
| 114 | Date: Oct, 2009 | ||
| 115 | KernelVersion: v2.6.39 | ||
| 116 | Contact: linux-ide@vger.kernel.org | ||
| 117 | Description: | ||
| 118 | (RO) Displays supported enclosure management message types. | ||
| @@ -0,0 +1,39 @@ | |||
| 1 | What: /sys/devices/platform/dock.N/docked | ||
| 2 | Date: Dec, 2006 | ||
| 3 | KernelVersion: 2.6.19 | ||
| 4 | Contact: linux-acpi@vger.kernel.org | ||
| 5 | Description: | ||
| 6 | (RO) Value 1 or 0 indicates whether the software believes the | ||
| 7 | laptop is docked in a docking station. | ||
| 8 | |||
| 9 | What: /sys/devices/platform/dock.N/undock | ||
| 10 | Date: Dec, 2006 | ||
| 11 | KernelVersion: 2.6.19 | ||
| 12 | Contact: linux-acpi@vger.kernel.org | ||
| 13 | Description: | ||
| 14 | (WO) Writing to this file causes the software to initiate an | ||
| 15 | undock request to the firmware. | ||
| 16 | |||
| 17 | What: /sys/devices/platform/dock.N/uid | ||
| 18 | Date: Feb, 2007 | ||
| 19 | KernelVersion: v2.6.21 | ||
| 20 | Contact: linux-acpi@vger.kernel.org | ||
| 21 | Description: | ||
| 22 | (RO) Displays the docking station the laptop is docked to. | ||
| 23 | |||
| 24 | What: /sys/devices/platform/dock.N/flags | ||
| 25 | Date: May, 2007 | ||
| 26 | KernelVersion: v2.6.21 | ||
| 27 | Contact: linux-acpi@vger.kernel.org | ||
| 28 | Description: | ||
| 29 | (RO) Show dock station flags, useful for checking if undock | ||
| 30 | request has been made by the user (from the immediate_undock | ||
| 31 | option). | ||
| 32 | |||
| 33 | What: /sys/devices/platform/dock.N/type | ||
| 34 | Date: Aug, 2008 | ||
| 35 | KernelVersion: v2.6.27 | ||
| 36 | Contact: linux-acpi@vger.kernel.org | ||
| 37 | Description: | ||
| 38 | (RO) Display the dock station type- dock_station, ata_bay or | ||
| 39 | battery_bay. | ||
| @@ -108,6 +108,8 @@ Description: CPU topology files that describe a logical CPU's relationship · 107 unmodified lines | |||
| 108 | 108 | ||
| 109 | What: /sys/devices/system/cpu/cpuidle/current_driver | 109 | What: /sys/devices/system/cpu/cpuidle/current_driver |
| 110 | /sys/devices/system/cpu/cpuidle/current_governer_ro | 110 | /sys/devices/system/cpu/cpuidle/current_governer_ro |
| 111 | /sys/devices/system/cpu/cpuidle/available_governors | ||
| 112 | /sys/devices/system/cpu/cpuidle/current_governor | ||
| 111 | Date: September 2007 | 113 | Date: September 2007 |
| 112 | Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org> | 114 | Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org> |
| 113 | Description: Discover cpuidle policy and mechanism | 115 | Description: Discover cpuidle policy and mechanism |
| @@ -119,13 +121,84 @@ Description: Discover cpuidle policy and mechanism · 5 unmodified lines | |||
| 119 | Idle policy (governor) is differentiated from idle mechanism | 121 | Idle policy (governor) is differentiated from idle mechanism |
| 120 | (driver) | 122 | (driver) |
| 121 | 123 | ||
| 122 | current_driver: displays current idle mechanism | 124 | current_driver: (RO) displays current idle mechanism |
| 123 | 125 | ||
| 124 | current_governor_ro: displays current idle policy | 126 | current_governor_ro: (RO) displays current idle policy |
| 127 | |||
| 128 | With the cpuidle_sysfs_switch boot option enabled (meant for | ||
| 129 | developer testing), the following three attributes are visible | ||
| 130 | instead: | ||
| 131 | |||
| 132 | current_driver: same as described above | ||
| 133 | |||
| 134 | available_governors: (RO) displays a space separated list of | ||
| 135 | available governors | ||
| 136 | |||
| 137 | current_governor: (RW) displays current idle policy. Users can | ||
| 138 | switch the governor at runtime by writing to this file. | ||
| 125 | 139 | ||
| 126 | See files in Documentation/cpuidle/ for more information. | 140 | See files in Documentation/cpuidle/ for more information. |
| 127 | 141 | ||
| 128 | 142 | ||
| 143 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/name | ||
| 144 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/latency | ||
| 145 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/power | ||
| 146 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/time | ||
| 147 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/usage | ||
| 148 | Date: September 2007 | ||
| 149 | KernelVersion: v2.6.24 | ||
| 150 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 151 | Description: | ||
| 152 | The directory /sys/devices/system/cpu/cpuX/cpuidle contains per | ||
| 153 | logical CPU specific cpuidle information for each online cpu X. | ||
| 154 | The processor idle states which are available for use have the | ||
| 155 | following attributes: | ||
| 156 | |||
| 157 | name: (RO) Name of the idle state (string). | ||
| 158 | |||
| 159 | latency: (RO) The latency to exit out of this idle state (in | ||
| 160 | microseconds). | ||
| 161 | |||
| 162 | power: (RO) The power consumed while in this idle state (in | ||
| 163 | milliwatts). | ||
| 164 | |||
| 165 | time: (RO) The total time spent in this idle state (in microseconds). | ||
| 166 | |||
| 167 | usage: (RO) Number of times this state was entered (a count). | ||
| 168 | |||
| 169 | |||
| 170 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/desc | ||
| 171 | Date: February 2008 | ||
| 172 | KernelVersion: v2.6.25 | ||
| 173 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 174 | Description: | ||
| 175 | (RO) A small description about the idle state (string). | ||
| 176 | |||
| 177 | |||
| 178 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/disable | ||
| 179 | Date: March 2012 | ||
| 180 | KernelVersion: v3.10 | ||
| 181 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 182 | Description: | ||
| 183 | (RW) Option to disable this idle state (bool). The behavior and | ||
| 184 | the effect of the disable variable depends on the implementation | ||
| 185 | of a particular governor. In the ladder governor, for example, | ||
| 186 | it is not coherent, i.e. if one is disabling a light state, then | ||
| 187 | all deeper states are disabled as well, but the disable variable | ||
| 188 | does not reflect it. Likewise, if one enables a deep state but a | ||
| 189 | lighter state still is disabled, then this has no effect. | ||
| 190 | |||
| 191 | |||
| 192 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/residency | ||
| 193 | Date: March 2014 | ||
| 194 | KernelVersion: v3.15 | ||
| 195 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 196 | Description: | ||
| 197 | (RO) Display the target residency i.e. the minimum amount of | ||
| 198 | time (in microseconds) this cpu should spend in this idle state | ||
| 199 | to make the transition worth the effort. | ||
| 200 | |||
| 201 | |||
| 129 | What: /sys/devices/system/cpu/cpu#/cpufreq/* | 202 | What: /sys/devices/system/cpu/cpu#/cpufreq/* |
| 130 | Date: pre-git history | 203 | Date: pre-git history |
| 131 | Contact: linux-pm@vger.kernel.org | 204 | Contact: linux-pm@vger.kernel.org |
| @@ -0,0 +1,40 @@ | |||
| 1 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/charger_type | ||
| 2 | Date: Jul, 2016 | ||
| 3 | KernelVersion: v4.10 | ||
| 4 | Contact: linux-acpi@vger.kernel.org | ||
| 5 | Description: | ||
| 6 | (RO) The charger type - Traditional, Hybrid or NVDC. | ||
| 7 | |||
| 8 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/adapter_rating_mw | ||
| 9 | Date: Jul, 2016 | ||
| 10 | KernelVersion: v4.10 | ||
| 11 | Contact: linux-acpi@vger.kernel.org | ||
| 12 | Description: | ||
| 13 | (RO) Adapter rating in milliwatts (the maximum Adapter power). | ||
| 14 | Must be 0 if no AC Adaptor is plugged in. | ||
| 15 | |||
| 16 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/max_platform_power_mw | ||
| 17 | Date: Jul, 2016 | ||
| 18 | KernelVersion: v4.10 | ||
| 19 | Contact: linux-acpi@vger.kernel.org | ||
| 20 | Description: | ||
| 21 | (RO) Maximum platform power that can be supported by the battery | ||
| 22 | in milliwatts. | ||
| 23 | |||
| 24 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/platform_power_source | ||
| 25 | Date: Jul, 2016 | ||
| 26 | KernelVersion: v4.10 | ||
| 27 | Contact: linux-acpi@vger.kernel.org | ||
| 28 | Description: | ||
| 29 | (RO) Display the platform power source | ||
| 30 | 0x00 = DC | ||
| 31 | 0x01 = AC | ||
| 32 | 0x02 = USB | ||
| 33 | 0x03 = Wireless Charger | ||
| 34 | |||
| 35 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/battery_steady_power | ||
| 36 | Date: Jul, 2016 | ||
| 37 | KernelVersion: v4.10 | ||
| 38 | Contact: linux-acpi@vger.kernel.org | ||
| 39 | Description: | ||
| 40 | (RO) The maximum sustained power for battery in milliwatts. | ||
| @@ -570,7 +570,9 @@ your driver if they're helpful, or just use plain hex constants. · 569 unmodified lines | |||
| 570 | The device IDs are arbitrary hex numbers (vendor controlled) and normally used | 570 | The device IDs are arbitrary hex numbers (vendor controlled) and normally used |
| 571 | only in a single location, the pci_device_id table. | 571 | only in a single location, the pci_device_id table. |
| 572 | 572 | ||
| 573 | Please DO submit new vendor/device IDs to http://pciids.sourceforge.net/. | 573 | Please DO submit new vendor/device IDs to http://pci-ids.ucw.cz/. |
| 574 | There are mirrors of the pci.ids file at http://pciids.sourceforge.net/ | ||
| 575 | and https://github.com/pciutils/pciids. | ||
| 574 | 576 | ||
| 575 | 577 | ||
| 576 | 578 | ||
| @@ -152,6 +152,11 @@ OCXL_IOCTL_IRQ_SET_FD: · 151 unmodified lines | |||
| 152 | Associate an event fd to an AFU interrupt so that the user process | 152 | Associate an event fd to an AFU interrupt so that the user process |
| 153 | can be notified when the AFU sends an interrupt. | 153 | can be notified when the AFU sends an interrupt. |
| 154 | 154 | ||
| 155 | OCXL_IOCTL_GET_METADATA: | ||
| 156 | |||
| 157 | Obtains configuration information from the card, such at the size of | ||
| 158 | MMIO areas, the AFU version, and the PASID for the current context. | ||
| 159 | |||
| 155 | 160 | ||
| 156 | mmap | 161 | mmap |
| 157 | ---- | 162 | ---- |
| @@ -58,7 +58,12 @@ Like with atomic_t, the rule of thumb is: · 57 unmodified lines | |||
| 58 | 58 | ||
| 59 | - RMW operations that have a return value are fully ordered. | 59 | - RMW operations that have a return value are fully ordered. |
| 60 | 60 | ||
| 61 | Except for test_and_set_bit_lock() which has ACQUIRE semantics and | 61 | - RMW operations that are conditional are unordered on FAILURE, |
| 62 | otherwise the above rules apply. In the case of test_and_{}_bit() operations, | ||
| 63 | if the bit in memory is unchanged by the operation then it is deemed to have | ||
| 64 | failed. | ||
| 65 | |||
| 66 | Except for a successful test_and_set_bit_lock() which has ACQUIRE semantics and | ||
| 62 | clear_bit_unlock() which has RELEASE semantics. | 67 | clear_bit_unlock() which has RELEASE semantics. |
| 63 | 68 | ||
| 64 | Since a platform only has a single means of achieving atomic operations | 69 | Since a platform only has a single means of achieving atomic operations |
File renamed without changes.
| @@ -1,23 +1,3 @@ | |||
| 1 | Etnaviv DRM master device | ||
| 2 | ========================= | ||
| 3 | |||
| 4 | The Etnaviv DRM master device is a virtual device needed to list all | ||
| 5 | Vivante GPU cores that comprise the GPU subsystem. | ||
| 6 | |||
| 7 | Required properties: | ||
| 8 | - compatible: Should be one of | ||
| 9 | "fsl,imx-gpu-subsystem" | ||
| 10 | "marvell,dove-gpu-subsystem" | ||
| 11 | - cores: Should contain a list of phandles pointing to Vivante GPU devices | ||
| 12 | |||
| 13 | example: | ||
| 14 | |||
| 15 | gpu-subsystem { | ||
| 16 | compatible = "fsl,imx-gpu-subsystem"; | ||
| 17 | cores = <&gpu_2d>, <&gpu_3d>; | ||
| 18 | }; | ||
| 19 | |||
| 20 | |||
| 21 | Vivante GPU core devices | 1 | Vivante GPU core devices |
| 22 | ======================== | 2 | ======================== |
| 23 | 3 | ||
| @@ -32,7 +12,9 @@ Required properties: · 8 unmodified lines | |||
| 32 | - clocks: should contain one clock for entry in clock-names | 12 | - clocks: should contain one clock for entry in clock-names |
| 33 | see Documentation/devicetree/bindings/clock/clock-bindings.txt | 13 | see Documentation/devicetree/bindings/clock/clock-bindings.txt |
| 34 | - clock-names: | 14 | - clock-names: |
| 35 | - "bus": AXI/register clock | 15 | - "bus": AXI/master interface clock |
| 16 | - "reg": AHB/slave interface clock | ||
| 17 | (only required if GPU can gate slave interface independently) | ||
| 36 | - "core": GPU core clock | 18 | - "core": GPU core clock |
| 37 | - "shader": Shader clock (only required if GPU has feature PIPE_3D) | 19 | - "shader": Shader clock (only required if GPU has feature PIPE_3D) |
| 38 | 20 | ||
| @@ -16,6 +16,7 @@ Required properties: · 15 unmodified lines | |||
| 16 | - ddc: phandle to the hdmi ddc node | 16 | - ddc: phandle to the hdmi ddc node |
| 17 | - phy: phandle to the hdmi phy node | 17 | - phy: phandle to the hdmi phy node |
| 18 | - samsung,syscon-phandle: phandle for system controller node for PMU. | 18 | - samsung,syscon-phandle: phandle for system controller node for PMU. |
| 19 | - #sound-dai-cells: should be 0. | ||
| 19 | 20 | ||
| 20 | Required properties for Exynos 4210, 4212, 5420 and 5433: | 21 | Required properties for Exynos 4210, 4212, 5420 and 5433: |
| 21 | - clocks: list of clock IDs from SoC clock driver. | 22 | - clocks: list of clock IDs from SoC clock driver. |
| @@ -11,7 +11,11 @@ Required properties: · 10 unmodified lines | |||
| 11 | interrupts. | 11 | interrupts. |
| 12 | 12 | ||
| 13 | Optional properties: | 13 | Optional properties: |
| 14 | - clocks: Optional reference to the clock used by the XOR engine. | 14 | - clocks: Optional reference to the clocks used by the XOR engine. |
| 15 | - clock-names: mandatory if there is a second clock, in this case the | ||
| 16 | name must be "core" for the first clock and "reg" for the second | ||
| 17 | one | ||
| 18 | |||
| 15 | 19 | ||
| 16 | Example: | 20 | Example: |
| 17 | 21 | ||
| @@ -38,9 +38,9 @@ Required properties: · 37 unmodified lines | |||
| 38 | 38 | ||
| 39 | "catalyst", | 39 | "catalyst", |
| 40 | "microchip", | 40 | "microchip", |
| 41 | "nxp", | ||
| 41 | "ramtron", | 42 | "ramtron", |
| 42 | "renesas", | 43 | "renesas", |
| 43 | "nxp", | ||
| 44 | "st", | 44 | "st", |
| 45 | 45 | ||
| 46 | Some vendors use different model names for chips which are just | 46 | Some vendors use different model names for chips which are just |
| @@ -3,11 +3,11 @@ Device-Tree bindings for sigma delta modulator · 2 unmodified lines | |||
| 3 | Required properties: | 3 | Required properties: |
| 4 | - compatible: should be "ads1201", "sd-modulator". "sd-modulator" can be use | 4 | - compatible: should be "ads1201", "sd-modulator". "sd-modulator" can be use |
| 5 | as a generic SD modulator if modulator not specified in compatible list. | 5 | as a generic SD modulator if modulator not specified in compatible list. |
| 6 | - #io-channel-cells = <1>: See the IIO bindings section "IIO consumers". | 6 | - #io-channel-cells = <0>: See the IIO bindings section "IIO consumers". |
| 7 | 7 | ||
| 8 | Example node: | 8 | Example node: |
| 9 | 9 | ||
| 10 | ads1202: adc@0 { | 10 | ads1202: adc@0 { |
| 11 | compatible = "sd-modulator"; | 11 | compatible = "sd-modulator"; |
| 12 | #io-channel-cells = <1>; | 12 | #io-channel-cells = <0>; |
| 13 | }; | 13 | }; |
| @@ -14,6 +14,7 @@ Required properties: · 13 unmodified lines | |||
| 14 | - "renesas,irqc-r8a7794" (R-Car E2) | 14 | - "renesas,irqc-r8a7794" (R-Car E2) |
| 15 | - "renesas,intc-ex-r8a7795" (R-Car H3) | 15 | - "renesas,intc-ex-r8a7795" (R-Car H3) |
| 16 | - "renesas,intc-ex-r8a7796" (R-Car M3-W) | 16 | - "renesas,intc-ex-r8a7796" (R-Car M3-W) |
| 17 | - "renesas,intc-ex-r8a77965" (R-Car M3-N) | ||
| 17 | - "renesas,intc-ex-r8a77970" (R-Car V3M) | 18 | - "renesas,intc-ex-r8a77970" (R-Car V3M) |
| 18 | - "renesas,intc-ex-r8a77995" (R-Car D3) | 19 | - "renesas,intc-ex-r8a77995" (R-Car D3) |
| 19 | - #interrupt-cells: has to be <2>: an interrupt index and flags, as defined in | 20 | - #interrupt-cells: has to be <2>: an interrupt index and flags, as defined in |
| @@ -50,14 +50,15 @@ Example: · 49 unmodified lines | |||
| 50 | compatible = "marvell,mv88e6085"; | 50 | compatible = "marvell,mv88e6085"; |
| 51 | reg = <0>; | 51 | reg = <0>; |
| 52 | reset-gpios = <&gpio5 1 GPIO_ACTIVE_LOW>; | 52 | reset-gpios = <&gpio5 1 GPIO_ACTIVE_LOW>; |
| 53 | }; | 53 | |
| 54 | mdio { | 54 | mdio { |
| 55 | #address-cells = <1>; | 55 | #address-cells = <1>; |
| 56 | #size-cells = <0>; | 56 | #size-cells = <0>; |
| 57 | switch1phy0: switch1phy0@0 { | 57 | switch1phy0: switch1phy0@0 { |
| 58 | reg = <0>; | 58 | reg = <0>; |
| 59 | interrupt-parent = <&switch0>; | 59 | interrupt-parent = <&switch0>; |
| 60 | interrupts = <0 IRQ_TYPE_LEVEL_HIGH>; | 60 | interrupts = <0 IRQ_TYPE_LEVEL_HIGH>; |
| 61 | }; | ||
| 61 | }; | 62 | }; |
| 62 | }; | 63 | }; |
| 63 | }; | 64 | }; |
| @@ -74,23 +75,24 @@ Example: · 10 unmodified lines | |||
| 74 | compatible = "marvell,mv88e6390"; | 75 | compatible = "marvell,mv88e6390"; |
| 75 | reg = <0>; | 76 | reg = <0>; |
| 76 | reset-gpios = <&gpio5 1 GPIO_ACTIVE_LOW>; | 77 | reset-gpios = <&gpio5 1 GPIO_ACTIVE_LOW>; |
| 77 | }; | 78 | |
| 78 | mdio { | 79 | mdio { |
| 79 | #address-cells = <1>; | 80 | #address-cells = <1>; |
| 80 | #size-cells = <0>; | 81 | #size-cells = <0>; |
| 81 | switch1phy0: switch1phy0@0 { | 82 | switch1phy0: switch1phy0@0 { |
| 82 | reg = <0>; | 83 | reg = <0>; |
| 83 | interrupt-parent = <&switch0>; | 84 | interrupt-parent = <&switch0>; |
| 84 | interrupts = <0 IRQ_TYPE_LEVEL_HIGH>; | 85 | interrupts = <0 IRQ_TYPE_LEVEL_HIGH>; |
| 86 | }; | ||
| 85 | }; | 87 | }; |
| 86 | }; | ||
| 87 | 88 | ||
| 88 | mdio1 { | 89 | mdio1 { |
| 89 | compatible = "marvell,mv88e6xxx-mdio-external"; | 90 | compatible = "marvell,mv88e6xxx-mdio-external"; |
| 90 | #address-cells = <1>; | 91 | #address-cells = <1>; |
| 91 | #size-cells = <0>; | 92 | #size-cells = <0>; |
| 92 | switch1phy9: switch1phy0@9 { | 93 | switch1phy9: switch1phy0@9 { |
| 93 | reg = <9>; | 94 | reg = <9>; |
| 95 | }; | ||
| 94 | }; | 96 | }; |
| 95 | }; | 97 | }; |
| 96 | }; | 98 | }; |
| @@ -18,6 +18,7 @@ Required properties: · 17 unmodified lines | |||
| 18 | - "renesas,etheravb-r8a7795" for the R8A7795 SoC. | 18 | - "renesas,etheravb-r8a7795" for the R8A7795 SoC. |
| 19 | - "renesas,etheravb-r8a7796" for the R8A7796 SoC. | 19 | - "renesas,etheravb-r8a7796" for the R8A7796 SoC. |
| 20 | - "renesas,etheravb-r8a77970" for the R8A77970 SoC. | 20 | - "renesas,etheravb-r8a77970" for the R8A77970 SoC. |
| 21 | - "renesas,etheravb-r8a77980" for the R8A77980 SoC. | ||
| 21 | - "renesas,etheravb-r8a77995" for the R8A77995 SoC. | 22 | - "renesas,etheravb-r8a77995" for the R8A77995 SoC. |
| 22 | - "renesas,etheravb-rcar-gen3" as a fallback for the above | 23 | - "renesas,etheravb-rcar-gen3" as a fallback for the above |
| 23 | R-Car Gen3 devices. | 24 | R-Car Gen3 devices. |
| @@ -26,7 +27,11 @@ Required properties: · 2 unmodified lines | |||
| 26 | SoC-specific version corresponding to the platform first followed by | 27 | SoC-specific version corresponding to the platform first followed by |
| 27 | the generic version. | 28 | the generic version. |
| 28 | 29 | ||
| 29 | - reg: offset and length of (1) the register block and (2) the stream buffer. | 30 | - reg: Offset and length of (1) the register block and (2) the stream buffer. |
| 31 | The region for the register block is mandatory. | ||
| 32 | The region for the stream buffer is optional, as it is only present on | ||
| 33 | R-Car Gen2 and RZ/G1 SoCs, and on R-Car H3 (R8A7795), M3-W (R8A7796), | ||
| 34 | and M3-N (R8A77965). | ||
| 30 | - interrupts: A list of interrupt-specifiers, one for each entry in | 35 | - interrupts: A list of interrupt-specifiers, one for each entry in |
| 31 | interrupt-names. | 36 | interrupt-names. |
| 32 | If interrupt-names is not present, an interrupt specifier | 37 | If interrupt-names is not present, an interrupt specifier |
| @@ -0,0 +1,8 @@ | |||
| 1 | Binding for MIPS Cluster Power Controller (CPC). | ||
| 2 | |||
| 3 | This binding allows a system to specify where the CPC registers are | ||
| 4 | located. | ||
| 5 | |||
| 6 | Required properties: | ||
| 7 | compatible : Should be "mti,mips-cpc". | ||
| 8 | regs: Should describe the address & size of the CPC register region. | ||
| @@ -60,7 +60,7 @@ Examples · 59 unmodified lines | |||
| 60 | #size-cells = <0>; | 60 | #size-cells = <0>; |
| 61 | 61 | ||
| 62 | button@1 { | 62 | button@1 { |
| 63 | debounce_interval = <50>; | 63 | debounce-interval = <50>; |
| 64 | wakeup-source; | 64 | wakeup-source; |
| 65 | linux,code = <116>; | 65 | linux,code = <116>; |
| 66 | label = "POWER"; | 66 | label = "POWER"; |
| @@ -22,7 +22,32 @@ Optional properties: · 21 unmodified lines | |||
| 22 | - clocks : thermal sensor's clock source. | 22 | - clocks : thermal sensor's clock source. |
| 23 | 23 | ||
| 24 | Example: | 24 | Example: |
| 25 | ocotp: ocotp@21bc000 { | ||
| 26 | #address-cells = <1>; | ||
| 27 | #size-cells = <1>; | ||
| 28 | compatible = "fsl,imx6sx-ocotp", "syscon"; | ||
| 29 | reg = <0x021bc000 0x4000>; | ||
| 30 | clocks = <&clks IMX6SX_CLK_OCOTP>; | ||
| 25 | 31 | ||
| 32 | tempmon_calib: calib@38 { | ||
| 33 | reg = <0x38 4>; | ||
| 34 | }; | ||
| 35 | |||
| 36 | tempmon_temp_grade: temp-grade@20 { | ||
| 37 | reg = <0x20 4>; | ||
| 38 | }; | ||
| 39 | }; | ||
| 40 | |||
| 41 | tempmon: tempmon { | ||
| 42 | compatible = "fsl,imx6sx-tempmon", "fsl,imx6q-tempmon"; | ||
| 43 | interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>; | ||
| 44 | fsl,tempmon = <&anatop>; | ||
| 45 | nvmem-cells = <&tempmon_calib>, <&tempmon_temp_grade>; | ||
| 46 | nvmem-cell-names = "calib", "temp_grade"; | ||
| 47 | clocks = <&clks IMX6SX_CLK_PLL3_USB_OTG>; | ||
| 48 | }; | ||
| 49 | |||
| 50 | Legacy method (Deprecated): | ||
| 26 | tempmon { | 51 | tempmon { |
| 27 | compatible = "fsl,imx6q-tempmon"; | 52 | compatible = "fsl,imx6q-tempmon"; |
| 28 | fsl,tempmon = <&anatop>; | 53 | fsl,tempmon = <&anatop>; |
| @@ -19,7 +19,7 @@ Required properties: · 18 unmodified lines | |||
| 19 | configured in FS mode; | 19 | configured in FS mode; |
| 20 | - "st,stm32f4x9-hsotg": The DWC2 USB HS controller instance in STM32F4x9 SoCs | 20 | - "st,stm32f4x9-hsotg": The DWC2 USB HS controller instance in STM32F4x9 SoCs |
| 21 | configured in HS mode; | 21 | configured in HS mode; |
| 22 | - "st,stm32f7xx-hsotg": The DWC2 USB HS controller instance in STM32F7xx SoCs | 22 | - "st,stm32f7-hsotg": The DWC2 USB HS controller instance in STM32F7 SoCs |
| 23 | configured in HS mode; | 23 | configured in HS mode; |
| 24 | - reg : Should contain 1 register range (address and length) | 24 | - reg : Should contain 1 register range (address and length) |
| 25 | - interrupts : Should contain 1 interrupt | 25 | - interrupts : Should contain 1 interrupt |
| @@ -4,6 +4,7 @@ Required properties: · 3 unmodified lines | |||
| 4 | - compatible: Must contain one of the following: | 4 | - compatible: Must contain one of the following: |
| 5 | - "renesas,r8a7795-usb3-peri" | 5 | - "renesas,r8a7795-usb3-peri" |
| 6 | - "renesas,r8a7796-usb3-peri" | 6 | - "renesas,r8a7796-usb3-peri" |
| 7 | - "renesas,r8a77965-usb3-peri" | ||
| 7 | - "renesas,rcar-gen3-usb3-peri" for a generic R-Car Gen3 compatible | 8 | - "renesas,rcar-gen3-usb3-peri" for a generic R-Car Gen3 compatible |
| 8 | device | 9 | device |
| 9 | 10 | ||
| @@ -12,6 +12,7 @@ Required properties: · 11 unmodified lines | |||
| 12 | - "renesas,usbhs-r8a7794" for r8a7794 (R-Car E2) compatible device | 12 | - "renesas,usbhs-r8a7794" for r8a7794 (R-Car E2) compatible device |
| 13 | - "renesas,usbhs-r8a7795" for r8a7795 (R-Car H3) compatible device | 13 | - "renesas,usbhs-r8a7795" for r8a7795 (R-Car H3) compatible device |
| 14 | - "renesas,usbhs-r8a7796" for r8a7796 (R-Car M3-W) compatible device | 14 | - "renesas,usbhs-r8a7796" for r8a7796 (R-Car M3-W) compatible device |
| 15 | - "renesas,usbhs-r8a77965" for r8a77965 (R-Car M3-N) compatible device | ||
| 15 | - "renesas,usbhs-r8a77995" for r8a77995 (R-Car D3) compatible device | 16 | - "renesas,usbhs-r8a77995" for r8a77995 (R-Car D3) compatible device |
| 16 | - "renesas,usbhs-r7s72100" for r7s72100 (RZ/A1) compatible device | 17 | - "renesas,usbhs-r7s72100" for r7s72100 (RZ/A1) compatible device |
| 17 | - "renesas,rcar-gen2-usbhs" for R-Car Gen2 or RZ/G1 compatible devices | 18 | - "renesas,rcar-gen2-usbhs" for R-Car Gen2 or RZ/G1 compatible devices |
| @@ -13,6 +13,7 @@ Required properties: · 12 unmodified lines | |||
| 13 | - "renesas,xhci-r8a7793" for r8a7793 SoC | 13 | - "renesas,xhci-r8a7793" for r8a7793 SoC |
| 14 | - "renesas,xhci-r8a7795" for r8a7795 SoC | 14 | - "renesas,xhci-r8a7795" for r8a7795 SoC |
| 15 | - "renesas,xhci-r8a7796" for r8a7796 SoC | 15 | - "renesas,xhci-r8a7796" for r8a7796 SoC |
| 16 | - "renesas,xhci-r8a77965" for r8a77965 SoC | ||
| 16 | - "renesas,rcar-gen2-xhci" for a generic R-Car Gen2 or RZ/G1 compatible | 17 | - "renesas,rcar-gen2-xhci" for a generic R-Car Gen2 or RZ/G1 compatible |
| 17 | device | 18 | device |
| 18 | - "renesas,rcar-gen3-xhci" for a generic R-Car Gen3 compatible device | 19 | - "renesas,rcar-gen3-xhci" for a generic R-Car Gen3 compatible device |
| @@ -0,0 +1,62 @@ | |||
| 1 | # | ||
| 2 | # Feature name: membarrier-sync-core | ||
| 3 | # Kconfig: ARCH_HAS_MEMBARRIER_SYNC_CORE | ||
| 4 | # description: arch supports core serializing membarrier | ||
| 5 | # | ||
| 6 | # Architecture requirements | ||
| 7 | # | ||
| 8 | # * arm64 | ||
| 9 | # | ||
| 10 | # Rely on eret context synchronization when returning from IPI handler, and | ||
| 11 | # when returning to user-space. | ||
| 12 | # | ||
| 13 | # * x86 | ||
| 14 | # | ||
| 15 | # x86-32 uses IRET as return from interrupt, which takes care of the IPI. | ||
| 16 | # However, it uses both IRET and SYSEXIT to go back to user-space. The IRET | ||
| 17 | # instruction is core serializing, but not SYSEXIT. | ||
| 18 | # | ||
| 19 | # x86-64 uses IRET as return from interrupt, which takes care of the IPI. | ||
| 20 | # However, it can return to user-space through either SYSRETL (compat code), | ||
| 21 | # SYSRETQ, or IRET. | ||
| 22 | # | ||
| 23 | # Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely | ||
| 24 | # instead on write_cr3() performed by switch_mm() to provide core serialization | ||
| 25 | # after changing the current mm, and deal with the special case of kthread -> | ||
| 26 | # uthread (temporarily keeping current mm into active_mm) by issuing a | ||
| 27 | # sync_core_before_usermode() in that specific case. | ||
| 28 | # | ||
| 29 | ----------------------- | ||
| 30 | | arch |status| | ||
| 31 | ----------------------- | ||
| 32 | | alpha: | TODO | | ||
| 33 | | arc: | TODO | | ||
| 34 | | arm: | TODO | | ||
| 35 | | arm64: | ok | | ||
| 36 | | blackfin: | TODO | | ||
| 37 | | c6x: | TODO | | ||
| 38 | | cris: | TODO | | ||
| 39 | | frv: | TODO | | ||
| 40 | | h8300: | TODO | | ||
| 41 | | hexagon: | TODO | | ||
| 42 | | ia64: | TODO | | ||
| 43 | | m32r: | TODO | | ||
| 44 | | m68k: | TODO | | ||
| 45 | | metag: | TODO | | ||
| 46 | | microblaze: | TODO | | ||
| 47 | | mips: | TODO | | ||
| 48 | | mn10300: | TODO | | ||
| 49 | | nios2: | TODO | | ||
| 50 | | openrisc: | TODO | | ||
| 51 | | parisc: | TODO | | ||
| 52 | | powerpc: | TODO | | ||
| 53 | | s390: | TODO | | ||
| 54 | | score: | TODO | | ||
| 55 | | sh: | TODO | | ||
| 56 | | sparc: | TODO | | ||
| 57 | | tile: | TODO | | ||
| 58 | | um: | TODO | | ||
| 59 | | unicore32: | TODO | | ||
| 60 | | x86: | ok | | ||
| 61 | | xtensa: | TODO | | ||
| 62 | ----------------------- | ||
| @@ -3,4 +3,4 @@ · 2 unmodified lines | |||
| 3 | ================================== | 3 | ================================== |
| 4 | 4 | ||
| 5 | .. kernel-doc:: drivers/gpu/drm/tve200/tve200_drv.c | 5 | .. kernel-doc:: drivers/gpu/drm/tve200/tve200_drv.c |
| 6 | :doc: Faraday TV Encoder 200 | 6 | :doc: Faraday TV Encoder TVE200 DRM Driver |
| @@ -28,8 +28,10 @@ Supported adapters: · 27 unmodified lines | |||
| 28 | * Intel Wildcat Point (PCH) | 28 | * Intel Wildcat Point (PCH) |
| 29 | * Intel Wildcat Point-LP (PCH) | 29 | * Intel Wildcat Point-LP (PCH) |
| 30 | * Intel BayTrail (SOC) | 30 | * Intel BayTrail (SOC) |
| 31 | * Intel Braswell (SOC) | ||
| 31 | * Intel Sunrise Point-H (PCH) | 32 | * Intel Sunrise Point-H (PCH) |
| 32 | * Intel Sunrise Point-LP (PCH) | 33 | * Intel Sunrise Point-LP (PCH) |
| 34 | * Intel Kaby Lake-H (PCH) | ||
| 33 | * Intel DNV (SOC) | 35 | * Intel DNV (SOC) |
| 34 | * Intel Broxton (SOC) | 36 | * Intel Broxton (SOC) |
| 35 | * Intel Lewisburg (PCH) | 37 | * Intel Lewisburg (PCH) |
| @@ -111,7 +111,7 @@ TROUBLESHOOTING SERIAL CONSOLE PROBLEMS · 110 unmodified lines | |||
| 111 | 111 | ||
| 112 | - If you don't have an HCDP, the kernel doesn't know where | 112 | - If you don't have an HCDP, the kernel doesn't know where |
| 113 | your console lives until the driver discovers serial | 113 | your console lives until the driver discovers serial |
| 114 | devices. Use "console=uart, io,0x3f8" (or appropriate | 114 | devices. Use "console=uart,io,0x3f8" (or appropriate |
| 115 | address for your machine). | 115 | address for your machine). |
| 116 | 116 | ||
| 117 | Kernel and init script output works fine, but no "login:" prompt: | 117 | Kernel and init script output works fine, but no "login:" prompt: |
| @@ -21,37 +21,23 @@ Implementation · 20 unmodified lines | |||
| 21 | -------------- | 21 | -------------- |
| 22 | 22 | ||
| 23 | Mutexes are represented by 'struct mutex', defined in include/linux/mutex.h | 23 | Mutexes are represented by 'struct mutex', defined in include/linux/mutex.h |
| 24 | and implemented in kernel/locking/mutex.c. These locks use a three | 24 | and implemented in kernel/locking/mutex.c. These locks use an atomic variable |
| 25 | state atomic counter (->count) to represent the different possible | 25 | (->owner) to keep track of the lock state during its lifetime. Field owner |
| 26 | transitions that can occur during the lifetime of a lock: | 26 | actually contains 'struct task_struct *' to the current lock owner and it is |
| 27 | 27 | therefore NULL if not currently owned. Since task_struct pointers are aligned | |
| 28 | 1: unlocked | 28 | at at least L1_CACHE_BYTES, low bits (3) are used to store extra state (e.g., |
| 29 | 0: locked, no waiters | 29 | if waiter list is non-empty). In its most basic form it also includes a |
| 30 | negative: locked, with potential waiters | 30 | wait-queue and a spinlock that serializes access to it. Furthermore, |
| 31 | 31 | CONFIG_MUTEX_SPIN_ON_OWNER=y systems use a spinner MCS lock (->osq), described | |
| 32 | In its most basic form it also includes a wait-queue and a spinlock | 32 | below in (ii). |
| 33 | that serializes access to it. CONFIG_SMP systems can also include | ||
| 34 | a pointer to the lock task owner (->owner) as well as a spinner MCS | ||
| 35 | lock (->osq), both described below in (ii). | ||
| 36 | 33 | ||
| 37 | When acquiring a mutex, there are three possible paths that can be | 34 | When acquiring a mutex, there are three possible paths that can be |
| 38 | taken, depending on the state of the lock: | 35 | taken, depending on the state of the lock: |
| 39 | 36 | ||
| 40 | (i) fastpath: tries to atomically acquire the lock by decrementing the | 37 | (i) fastpath: tries to atomically acquire the lock by cmpxchg()ing the owner with |
| 41 | counter. If it was already taken by another task it goes to the next | 38 | the current task. This only works in the uncontended case (cmpxchg() checks |
| 42 | possible path. This logic is architecture specific. On x86-64, the | 39 | against 0UL, so all 3 state bits above have to be 0). If the lock is |
| 43 | locking fastpath is 2 instructions: | 40 | contended it goes to the next possible path. |
| 44 | |||
| 45 | 0000000000000e10 <mutex_lock>: | ||
| 46 | e21: f0 ff 0b lock decl (%rbx) | ||
| 47 | e24: 79 08 jns e2e <mutex_lock+0x1e> | ||
| 48 | |||
| 49 | the unlocking fastpath is equally tight: | ||
| 50 | |||
| 51 | 0000000000000bc0 <mutex_unlock>: | ||
| 52 | bc8: f0 ff 07 lock incl (%rdi) | ||
| 53 | bcb: 7f 0a jg bd7 <mutex_unlock+0x17> | ||
| 54 | |||
| 55 | 41 | ||
| 56 | (ii) midpath: aka optimistic spinning, tries to spin for acquisition | 42 | (ii) midpath: aka optimistic spinning, tries to spin for acquisition |
| 57 | while the lock owner is running and there are no other tasks ready | 43 | while the lock owner is running and there are no other tasks ready |
| @@ -143,11 +129,10 @@ Test if the mutex is taken: · 85 unmodified lines | |||
| 143 | Disadvantages | 129 | Disadvantages |
| 144 | ------------- | 130 | ------------- |
| 145 | 131 | ||
| 146 | Unlike its original design and purpose, 'struct mutex' is larger than | 132 | Unlike its original design and purpose, 'struct mutex' is among the largest |
| 147 | most locks in the kernel. E.g: on x86-64 it is 40 bytes, almost twice | 133 | locks in the kernel. E.g: on x86-64 it is 32 bytes, where 'struct semaphore' |
| 148 | as large as 'struct semaphore' (24 bytes) and tied, along with rwsems, | 134 | is 24 bytes and rw_semaphore is 40 bytes. Larger structure sizes mean more CPU |
| 149 | for the largest lock in the kernel. Larger structure sizes mean more | 135 | cache and memory footprint. |
| 150 | CPU cache and memory footprint. | ||
| 151 | 136 | ||
| 152 | When to use mutexes | 137 | When to use mutexes |
| 153 | ------------------- | 138 | ------------------- |
| @@ -50,9 +50,15 @@ replace typedef dmx_filter_t :c:type:`dmx_filter` · 49 unmodified lines | |||
| 50 | replace typedef dmx_pes_type_t :c:type:`dmx_pes_type` | 50 | replace typedef dmx_pes_type_t :c:type:`dmx_pes_type` |
| 51 | replace typedef dmx_input_t :c:type:`dmx_input` | 51 | replace typedef dmx_input_t :c:type:`dmx_input` |
| 52 | 52 | ||
| 53 | ignore symbol DMX_OUT_DECODER | 53 | replace symbol DMX_BUFFER_FLAG_HAD_CRC32_DISCARD :c:type:`dmx_buffer_flags` |
| 54 | ignore symbol DMX_OUT_TAP | 54 | replace symbol DMX_BUFFER_FLAG_TEI :c:type:`dmx_buffer_flags` |
| 55 | ignore symbol DMX_OUT_TS_TAP | 55 | replace symbol DMX_BUFFER_PKT_COUNTER_MISMATCH :c:type:`dmx_buffer_flags` |
| 56 | ignore symbol DMX_OUT_TSDEMUX_TAP | 56 | replace symbol DMX_BUFFER_FLAG_DISCONTINUITY_DETECTED :c:type:`dmx_buffer_flags` |
| 57 | replace symbol DMX_BUFFER_FLAG_DISCONTINUITY_INDICATOR :c:type:`dmx_buffer_flags` | ||
| 58 | |||
| 59 | replace symbol DMX_OUT_DECODER :c:type:`dmx_output` | ||
| 60 | replace symbol DMX_OUT_TAP :c:type:`dmx_output` | ||
| 61 | replace symbol DMX_OUT_TS_TAP :c:type:`dmx_output` | ||
| 62 | replace symbol DMX_OUT_TSDEMUX_TAP :c:type:`dmx_output` | ||
| 57 | 63 | ||
| 58 | replace ioctl DMX_DQBUF dmx_qbuf | 64 | replace ioctl DMX_DQBUF dmx_qbuf |
| @@ -51,9 +51,10 @@ out to disk. Buffers remain locked until dequeued, until the · 50 unmodified lines | |||
| 51 | the device is closed. | 51 | the device is closed. |
| 52 | 52 | ||
| 53 | Applications call the ``DMX_DQBUF`` ioctl to dequeue a filled | 53 | Applications call the ``DMX_DQBUF`` ioctl to dequeue a filled |
| 54 | (capturing) buffer from the driver's outgoing queue. They just set the ``reserved`` field array to zero. When ``DMX_DQBUF`` is called with a | 54 | (capturing) buffer from the driver's outgoing queue. |
| 55 | pointer to this structure, the driver fills the remaining fields or | 55 | They just set the ``index`` field withe the buffer ID to be queued. |
| 56 | returns an error code. | 56 | When ``DMX_DQBUF`` is called with a pointer to struct :c:type:`dmx_buffer`, |
| 57 | the driver fills the remaining fields or returns an error code. | ||
| 57 | 58 | ||
| 58 | By default ``DMX_DQBUF`` blocks when no buffer is in the outgoing | 59 | By default ``DMX_DQBUF`` blocks when no buffer is in the outgoing |
| 59 | queue. When the ``O_NONBLOCK`` flag was given to the | 60 | queue. When the ``O_NONBLOCK`` flag was given to the |
| @@ -13,14 +13,15 @@ The following technologies are described: · 12 unmodified lines | |||
| 13 | * Generic Segmentation Offload - GSO | 13 | * Generic Segmentation Offload - GSO |
| 14 | * Generic Receive Offload - GRO | 14 | * Generic Receive Offload - GRO |
| 15 | * Partial Generic Segmentation Offload - GSO_PARTIAL | 15 | * Partial Generic Segmentation Offload - GSO_PARTIAL |
| 16 | * SCTP accelleration with GSO - GSO_BY_FRAGS | ||
| 16 | 17 | ||
| 17 | TCP Segmentation Offload | 18 | TCP Segmentation Offload |
| 18 | ======================== | 19 | ======================== |
| 19 | 20 | ||
| 20 | TCP segmentation allows a device to segment a single frame into multiple | 21 | TCP segmentation allows a device to segment a single frame into multiple |
| 21 | frames with a data payload size specified in skb_shinfo()->gso_size. | 22 | frames with a data payload size specified in skb_shinfo()->gso_size. |
| 22 | When TCP segmentation requested the bit for either SKB_GSO_TCP or | 23 | When TCP segmentation requested the bit for either SKB_GSO_TCPV4 or |
| 23 | SKB_GSO_TCP6 should be set in skb_shinfo()->gso_type and | 24 | SKB_GSO_TCPV6 should be set in skb_shinfo()->gso_type and |
| 24 | skb_shinfo()->gso_size should be set to a non-zero value. | 25 | skb_shinfo()->gso_size should be set to a non-zero value. |
| 25 | 26 | ||
| 26 | TCP segmentation is dependent on support for the use of partial checksum | 27 | TCP segmentation is dependent on support for the use of partial checksum |
| @@ -49,6 +50,10 @@ datagram into multiple IPv4 fragments. Many of the requirements for UDP · 22 unmodified lines | |||
| 49 | fragmentation offload are the same as TSO. However the IPv4 ID for | 50 | fragmentation offload are the same as TSO. However the IPv4 ID for |
| 50 | fragments should not increment as a single IPv4 datagram is fragmented. | 51 | fragments should not increment as a single IPv4 datagram is fragmented. |
| 51 | 52 | ||
| 53 | UFO is deprecated: modern kernels will no longer generate UFO skbs, but can | ||
| 54 | still receive them from tuntap and similar devices. Offload of UDP-based | ||
| 55 | tunnel protocols is still supported. | ||
| 56 | |||
| 52 | IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads | 57 | IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads |
| 53 | ======================================================== | 58 | ======================================================== |
| 54 | 59 | ||
| @@ -83,10 +88,10 @@ SKB_GSO_UDP_TUNNEL_CSUM. These two additional tunnel types reflect the · 28 unmodified lines | |||
| 83 | fact that the outer header also requests to have a non-zero checksum | 88 | fact that the outer header also requests to have a non-zero checksum |
| 84 | included in the outer header. | 89 | included in the outer header. |
| 85 | 90 | ||
| 86 | Finally there is SKB_GSO_REMCSUM which indicates that a given tunnel header | 91 | Finally there is SKB_GSO_TUNNEL_REMCSUM which indicates that a given tunnel |
| 87 | has requested a remote checksum offload. In this case the inner headers | 92 | header has requested a remote checksum offload. In this case the inner |
| 88 | will be left with a partial checksum and only the outer header checksum | 93 | headers will be left with a partial checksum and only the outer header |
| 89 | will be computed. | 94 | checksum will be computed. |
| 90 | 95 | ||
| 91 | Generic Segmentation Offload | 96 | Generic Segmentation Offload |
| 92 | ============================ | 97 | ============================ |
| @@ -128,3 +133,38 @@ values for if the header was simply duplicated. The one exception to this · 35 unmodified lines | |||
| 128 | is the outer IPv4 ID field. It is up to the device drivers to guarantee | 133 | is the outer IPv4 ID field. It is up to the device drivers to guarantee |
| 129 | that the IPv4 ID field is incremented in the case that a given header does | 134 | that the IPv4 ID field is incremented in the case that a given header does |
| 130 | not have the DF bit set. | 135 | not have the DF bit set. |
| 136 | |||
| 137 | SCTP accelleration with GSO | ||
| 138 | =========================== | ||
| 139 | |||
| 140 | SCTP - despite the lack of hardware support - can still take advantage of | ||
| 141 | GSO to pass one large packet through the network stack, rather than | ||
| 142 | multiple small packets. | ||
| 143 | |||
| 144 | This requires a different approach to other offloads, as SCTP packets | ||
| 145 | cannot be just segmented to (P)MTU. Rather, the chunks must be contained in | ||
| 146 | IP segments, padding respected. So unlike regular GSO, SCTP can't just | ||
| 147 | generate a big skb, set gso_size to the fragmentation point and deliver it | ||
| 148 | to IP layer. | ||
| 149 | |||
| 150 | Instead, the SCTP protocol layer builds an skb with the segments correctly | ||
| 151 | padded and stored as chained skbs, and skb_segment() splits based on those. | ||
| 152 | To signal this, gso_size is set to the special value GSO_BY_FRAGS. | ||
| 153 | |||
| 154 | Therefore, any code in the core networking stack must be aware of the | ||
| 155 | possibility that gso_size will be GSO_BY_FRAGS and handle that case | ||
| 156 | appropriately. | ||
| 157 | |||
| 158 | There are some helpers to make this easier: | ||
| 159 | |||
| 160 | - skb_is_gso(skb) && skb_is_gso_sctp(skb) is the best way to see if | ||
| 161 | an skb is an SCTP GSO skb. | ||
| 162 | |||
| 163 | - For size checks, the skb_gso_validate_*_len family of helpers correctly | ||
| 164 | considers GSO_BY_FRAGS. | ||
| 165 | |||
| 166 | - For manipulating packets, skb_increase_gso_size and skb_decrease_gso_size | ||
| 167 | will check for GSO_BY_FRAGS and WARN if asked to manipulate these skbs. | ||
| 168 | |||
| 169 | This also affects drivers with the NETIF_F_FRAGLIST & NETIF_F_GSO_SCTP bits | ||
| 170 | set. Note also that NETIF_F_GSO_SCTP is included in NETIF_F_GSO_SOFTWARE. | ||
| @@ -36,8 +36,7 @@ import glob · 35 unmodified lines | |||
| 36 | 36 | ||
| 37 | from docutils import nodes, statemachine | 37 | from docutils import nodes, statemachine |
| 38 | from docutils.statemachine import ViewList | 38 | from docutils.statemachine import ViewList |
| 39 | from docutils.parsers.rst import directives | 39 | from docutils.parsers.rst import directives, Directive |
| 40 | from sphinx.util.compat import Directive | ||
| 41 | from sphinx.ext.autodoc import AutodocReporter | 40 | from sphinx.ext.autodoc import AutodocReporter |
| 42 | 41 | ||
| 43 | __version__ = '1.0' | 42 | __version__ = '1.0' |
| @@ -123,14 +123,15 @@ memory layout to fit in user mode), check KVM_CAP_MIPS_VZ and use the · 122 unmodified lines | |||
| 123 | flag KVM_VM_MIPS_VZ. | 123 | flag KVM_VM_MIPS_VZ. |
| 124 | 124 | ||
| 125 | 125 | ||
| 126 | 4.3 KVM_GET_MSR_INDEX_LIST | 126 | 4.3 KVM_GET_MSR_INDEX_LIST, KVM_GET_MSR_FEATURE_INDEX_LIST |
| 127 | 127 | ||
| 128 | Capability: basic | 128 | Capability: basic, KVM_CAP_GET_MSR_FEATURES for KVM_GET_MSR_FEATURE_INDEX_LIST |
| 129 | Architectures: x86 | 129 | Architectures: x86 |
| 130 | Type: system | 130 | Type: system ioctl |
| 131 | Parameters: struct kvm_msr_list (in/out) | 131 | Parameters: struct kvm_msr_list (in/out) |
| 132 | Returns: 0 on success; -1 on error | 132 | Returns: 0 on success; -1 on error |
| 133 | Errors: | 133 | Errors: |
| 134 | EFAULT: the msr index list cannot be read from or written to | ||
| 134 | E2BIG: the msr index list is to be to fit in the array specified by | 135 | E2BIG: the msr index list is to be to fit in the array specified by |
| 135 | the user. | 136 | the user. |
| 136 | 137 | ||
| @@ -139,16 +140,23 @@ struct kvm_msr_list { · 2 unmodified lines | |||
| 139 | __u32 indices[0]; | 140 | __u32 indices[0]; |
| 140 | }; | 141 | }; |
| 141 | 142 | ||
| 142 | This ioctl returns the guest msrs that are supported. The list varies | 143 | The user fills in the size of the indices array in nmsrs, and in return |
| 143 | by kvm version and host processor, but does not change otherwise. The | 144 | kvm adjusts nmsrs to reflect the actual number of msrs and fills in the |
| 144 | user fills in the size of the indices array in nmsrs, and in return | 145 | indices array with their numbers. |
| 145 | kvm adjusts nmsrs to reflect the actual number of msrs and fills in | 146 | |
| 146 | the indices array with their numbers. | 147 | KVM_GET_MSR_INDEX_LIST returns the guest msrs that are supported. The list |
| 148 | varies by kvm version and host processor, but does not change otherwise. | ||
| 147 | 149 | ||
| 148 | Note: if kvm indicates supports MCE (KVM_CAP_MCE), then the MCE bank MSRs are | 150 | Note: if kvm indicates supports MCE (KVM_CAP_MCE), then the MCE bank MSRs are |
| 149 | not returned in the MSR list, as different vcpus can have a different number | 151 | not returned in the MSR list, as different vcpus can have a different number |
| 150 | of banks, as set via the KVM_X86_SETUP_MCE ioctl. | 152 | of banks, as set via the KVM_X86_SETUP_MCE ioctl. |
| 151 | 153 | ||
| 154 | KVM_GET_MSR_FEATURE_INDEX_LIST returns the list of MSRs that can be passed | ||
| 155 | to the KVM_GET_MSRS system ioctl. This lets userspace probe host capabilities | ||
| 156 | and processor features that are exposed via MSRs (e.g., VMX capabilities). | ||
| 157 | This list also varies by kvm version and host processor, but does not change | ||
| 158 | otherwise. | ||
| 159 | |||
| 152 | 160 | ||
| 153 | 4.4 KVM_CHECK_EXTENSION | 161 | 4.4 KVM_CHECK_EXTENSION |
| 154 | 162 | ||
| @@ -475,14 +483,22 @@ Support for this has been removed. Use KVM_SET_GUEST_DEBUG instead. · 320 unmodified lines | |||
| 475 | 483 | ||
| 476 | 4.18 KVM_GET_MSRS | 484 | 4.18 KVM_GET_MSRS |
| 477 | 485 | ||
| 478 | Capability: basic | 486 | Capability: basic (vcpu), KVM_CAP_GET_MSR_FEATURES (system) |
| 479 | Architectures: x86 | 487 | Architectures: x86 |
| 480 | Type: vcpu ioctl | 488 | Type: system ioctl, vcpu ioctl |
| 481 | Parameters: struct kvm_msrs (in/out) | 489 | Parameters: struct kvm_msrs (in/out) |
| 482 | Returns: 0 on success, -1 on error | 490 | Returns: number of msrs successfully returned; |
| 491 | -1 on error | ||
| 492 | |||
| 493 | When used as a system ioctl: | ||
| 494 | Reads the values of MSR-based features that are available for the VM. This | ||
| 495 | is similar to KVM_GET_SUPPORTED_CPUID, but it returns MSR indices and values. | ||
| 496 | The list of msr-based features can be obtained using KVM_GET_MSR_FEATURE_INDEX_LIST | ||
| 497 | in a system ioctl. | ||
| 483 | 498 | ||
| 499 | When used as a vcpu ioctl: | ||
| 484 | Reads model-specific registers from the vcpu. Supported msr indices can | 500 | Reads model-specific registers from the vcpu. Supported msr indices can |
| 485 | be obtained using KVM_GET_MSR_INDEX_LIST. | 501 | be obtained using KVM_GET_MSR_INDEX_LIST in a system ioctl. |
| 486 | 502 | ||
| 487 | struct kvm_msrs { | 503 | struct kvm_msrs { |
| 488 | __u32 nmsrs; /* number of msrs in entries */ | 504 | __u32 nmsrs; /* number of msrs in entries */ |
| @@ -58,6 +58,10 @@ KVM_FEATURE_PV_TLB_FLUSH || 9 || guest checks this feature bit · 57 unmodified lines | |||
| 58 | || || before enabling paravirtualized | 58 | || || before enabling paravirtualized |
| 59 | || || tlb flush. | 59 | || || tlb flush. |
| 60 | ------------------------------------------------------------------------------ | 60 | ------------------------------------------------------------------------------ |
| 61 | KVM_FEATURE_ASYNC_PF_VMEXIT || 10 || paravirtualized async PF VM exit | ||
| 62 | || || can be enabled by setting bit 2 | ||
| 63 | || || when writing to msr 0x4b564d02 | ||
| 64 | ------------------------------------------------------------------------------ | ||
| 61 | KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side | 65 | KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side |
| 62 | || || per-cpu warps are expected in | 66 | || || per-cpu warps are expected in |
| 63 | || || kvmclock. | 67 | || || kvmclock. |
| @@ -170,7 +170,8 @@ MSR_KVM_ASYNC_PF_EN: 0x4b564d02 · 169 unmodified lines | |||
| 170 | when asynchronous page faults are enabled on the vcpu 0 when | 170 | when asynchronous page faults are enabled on the vcpu 0 when |
| 171 | disabled. Bit 1 is 1 if asynchronous page faults can be injected | 171 | disabled. Bit 1 is 1 if asynchronous page faults can be injected |
| 172 | when vcpu is in cpl == 0. Bit 2 is 1 if asynchronous page faults | 172 | when vcpu is in cpl == 0. Bit 2 is 1 if asynchronous page faults |
| 173 | are delivered to L1 as #PF vmexits. | 173 | are delivered to L1 as #PF vmexits. Bit 2 can be set only if |
| 174 | KVM_FEATURE_ASYNC_PF_VMEXIT is present in CPUID. | ||
| 174 | 175 | ||
| 175 | First 4 byte of 64 byte memory location will be written to by | 176 | First 4 byte of 64 byte memory location will be written to by |
| 176 | the hypervisor at the time of asynchronous page fault (APF) | 177 | the hypervisor at the time of asynchronous page fault (APF) |
| @@ -671,7 +671,7 @@ occupancy of the real time threads on these cores. · 670 unmodified lines | |||
| 671 | # mkdir p1 | 671 | # mkdir p1 |
| 672 | 672 | ||
| 673 | Move the cpus 4-7 over to p1 | 673 | Move the cpus 4-7 over to p1 |
| 674 | # echo f0 > p0/cpus | 674 | # echo f0 > p1/cpus |
| 675 | 675 | ||
| 676 | View the llc occupancy snapshot | 676 | View the llc occupancy snapshot |
| 677 | 677 | ||
| @@ -108,7 +108,7 @@ The topology of a system is described in the units of: · 107 unmodified lines | |||
| 108 | 108 | ||
| 109 | The number of online threads is also printed in /proc/cpuinfo "siblings." | 109 | The number of online threads is also printed in /proc/cpuinfo "siblings." |
| 110 | 110 | ||
| 111 | - topology_sibling_mask(): | 111 | - topology_sibling_cpumask(): |
| 112 | 112 | ||
| 113 | The cpumask contains all online threads in the core to which a thread | 113 | The cpumask contains all online threads in the core to which a thread |
| 114 | belongs. | 114 | belongs. |
| @@ -1240,7 +1240,7 @@ F: drivers/clk/at91 · 1239 unmodified lines | |||
| 1240 | 1240 | ||
| 1241 | ARM/ATMEL AT91RM9200, AT91SAM9 AND SAMA5 SOC SUPPORT | 1241 | ARM/ATMEL AT91RM9200, AT91SAM9 AND SAMA5 SOC SUPPORT |
| 1242 | M: Nicolas Ferre <nicolas.ferre@microchip.com> | 1242 | M: Nicolas Ferre <nicolas.ferre@microchip.com> |
| 1243 | M: Alexandre Belloni <alexandre.belloni@free-electrons.com> | 1243 | M: Alexandre Belloni <alexandre.belloni@bootlin.com> |
| 1244 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 1244 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 1245 | W: http://www.linux4sam.org | 1245 | W: http://www.linux4sam.org |
| 1246 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/nferre/linux-at91.git | 1246 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/nferre/linux-at91.git |
| @@ -1592,7 +1592,7 @@ ARM/Marvell Dove/MV78xx0/Orion SOC support · 345 unmodified lines | |||
| 1592 | M: Jason Cooper <jason@lakedaemon.net> | 1592 | M: Jason Cooper <jason@lakedaemon.net> |
| 1593 | M: Andrew Lunn <andrew@lunn.ch> | 1593 | M: Andrew Lunn <andrew@lunn.ch> |
| 1594 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> | 1594 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> |
| 1595 | M: Gregory Clement <gregory.clement@free-electrons.com> | 1595 | M: Gregory Clement <gregory.clement@bootlin.com> |
| 1596 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 1596 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 1597 | S: Maintained | 1597 | S: Maintained |
| 1598 | F: Documentation/devicetree/bindings/soc/dove/ | 1598 | F: Documentation/devicetree/bindings/soc/dove/ |
| @@ -1606,7 +1606,7 @@ F: arch/arm/boot/dts/orion5x* · 7 unmodified lines | |||
| 1606 | ARM/Marvell Kirkwood and Armada 370, 375, 38x, 39x, XP, 3700, 7K/8K SOC support | 1606 | ARM/Marvell Kirkwood and Armada 370, 375, 38x, 39x, XP, 3700, 7K/8K SOC support |
| 1607 | M: Jason Cooper <jason@lakedaemon.net> | 1607 | M: Jason Cooper <jason@lakedaemon.net> |
| 1608 | M: Andrew Lunn <andrew@lunn.ch> | 1608 | M: Andrew Lunn <andrew@lunn.ch> |
| 1609 | M: Gregory Clement <gregory.clement@free-electrons.com> | 1609 | M: Gregory Clement <gregory.clement@bootlin.com> |
| 1610 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> | 1610 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> |
| 1611 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 1611 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 1612 | S: Maintained | 1612 | S: Maintained |
| @@ -2001,8 +2001,10 @@ M: Maxime Coquelin <mcoquelin.stm32@gmail.com> · 388 unmodified lines | |||
| 2001 | M: Alexandre Torgue <alexandre.torgue@st.com> | 2001 | M: Alexandre Torgue <alexandre.torgue@st.com> |
| 2002 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 2002 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 2003 | S: Maintained | 2003 | S: Maintained |
| 2004 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/mcoquelin/stm32.git | 2004 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git stm32-next |
| 2005 | N: stm32 | 2005 | N: stm32 |
| 2006 | F: arch/arm/boot/dts/stm32* | ||
| 2007 | F: arch/arm/mach-stm32/ | ||
| 2006 | F: drivers/clocksource/armv7m_systick.c | 2008 | F: drivers/clocksource/armv7m_systick.c |
| 2007 | 2009 | ||
| 2008 | ARM/TANGO ARCHITECTURE | 2010 | ARM/TANGO ARCHITECTURE |
| @@ -7610,8 +7612,10 @@ F: mm/kasan/ · 5601 unmodified lines | |||
| 7610 | F: scripts/Makefile.kasan | 7612 | F: scripts/Makefile.kasan |
| 7611 | 7613 | ||
| 7612 | KCONFIG | 7614 | KCONFIG |
| 7615 | M: Masahiro Yamada <yamada.masahiro@socionext.com> | ||
| 7616 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig | ||
| 7613 | L: linux-kbuild@vger.kernel.org | 7617 | L: linux-kbuild@vger.kernel.org |
| 7614 | S: Orphan | 7618 | S: Maintained |
| 7615 | F: Documentation/kbuild/kconfig-language.txt | 7619 | F: Documentation/kbuild/kconfig-language.txt |
| 7616 | F: scripts/kconfig/ | 7620 | F: scripts/kconfig/ |
| 7617 | 7621 | ||
| @@ -7919,7 +7923,6 @@ S: Maintained · 301 unmodified lines | |||
| 7919 | F: scripts/leaking_addresses.pl | 7923 | F: scripts/leaking_addresses.pl |
| 7920 | 7924 | ||
| 7921 | LED SUBSYSTEM | 7925 | LED SUBSYSTEM |
| 7922 | M: Richard Purdie <rpurdie@rpsys.net> | ||
| 7923 | M: Jacek Anaszewski <jacek.anaszewski@gmail.com> | 7926 | M: Jacek Anaszewski <jacek.anaszewski@gmail.com> |
| 7924 | M: Pavel Machek <pavel@ucw.cz> | 7927 | M: Pavel Machek <pavel@ucw.cz> |
| 7925 | L: linux-leds@vger.kernel.org | 7928 | L: linux-leds@vger.kernel.org |
| @@ -9216,6 +9219,7 @@ MIPS GENERIC PLATFORM · 1290 unmodified lines | |||
| 9216 | M: Paul Burton <paul.burton@mips.com> | 9219 | M: Paul Burton <paul.burton@mips.com> |
| 9217 | L: linux-mips@linux-mips.org | 9220 | L: linux-mips@linux-mips.org |
| 9218 | S: Supported | 9221 | S: Supported |
| 9222 | F: Documentation/devicetree/bindings/power/mti,mips-cpc.txt | ||
| 9219 | F: arch/mips/generic/ | 9223 | F: arch/mips/generic/ |
| 9220 | F: arch/mips/tools/generic-board-config.sh | 9224 | F: arch/mips/tools/generic-board-config.sh |
| 9221 | 9225 | ||
| @@ -9931,6 +9935,13 @@ F: Documentation/ABI/stable/sysfs-bus-nvmem · 709 unmodified lines | |||
| 9931 | F: include/linux/nvmem-consumer.h | 9935 | F: include/linux/nvmem-consumer.h |
| 9932 | F: include/linux/nvmem-provider.h | 9936 | F: include/linux/nvmem-provider.h |
| 9933 | 9937 | ||
| 9938 | NXP SGTL5000 DRIVER | ||
| 9939 | M: Fabio Estevam <fabio.estevam@nxp.com> | ||
| 9940 | L: alsa-devel@alsa-project.org (moderated for non-subscribers) | ||
| 9941 | S: Maintained | ||
| 9942 | F: Documentation/devicetree/bindings/sound/sgtl5000.txt | ||
| 9943 | F: sound/soc/codecs/sgtl5000* | ||
| 9944 | |||
| 9934 | NXP TDA998X DRM DRIVER | 9945 | NXP TDA998X DRM DRIVER |
| 9935 | M: Russell King <linux@armlinux.org.uk> | 9946 | M: Russell King <linux@armlinux.org.uk> |
| 9936 | S: Supported | 9947 | S: Supported |
| @@ -9955,6 +9966,7 @@ F: drivers/nfc/nxp-nci · 18 unmodified lines | |||
| 9955 | 9966 | ||
| 9956 | OBJTOOL | 9967 | OBJTOOL |
| 9957 | M: Josh Poimboeuf <jpoimboe@redhat.com> | 9968 | M: Josh Poimboeuf <jpoimboe@redhat.com> |
| 9969 | M: Peter Zijlstra <peterz@infradead.org> | ||
| 9958 | S: Supported | 9970 | S: Supported |
| 9959 | F: tools/objtool/ | 9971 | F: tools/objtool/ |
| 9960 | 9972 | ||
| @@ -10332,7 +10344,7 @@ F: drivers/oprofile/ · 371 unmodified lines | |||
| 10332 | F: include/linux/oprofile.h | 10344 | F: include/linux/oprofile.h |
| 10333 | 10345 | ||
| 10334 | ORACLE CLUSTER FILESYSTEM 2 (OCFS2) | 10346 | ORACLE CLUSTER FILESYSTEM 2 (OCFS2) |
| 10335 | M: Mark Fasheh <mfasheh@versity.com> | 10347 | M: Mark Fasheh <mark@fasheh.com> |
| 10336 | M: Joel Becker <jlbec@evilplan.org> | 10348 | M: Joel Becker <jlbec@evilplan.org> |
| 10337 | L: ocfs2-devel@oss.oracle.com (moderated for non-subscribers) | 10349 | L: ocfs2-devel@oss.oracle.com (moderated for non-subscribers) |
| 10338 | W: http://ocfs2.wiki.kernel.org | 10350 | W: http://ocfs2.wiki.kernel.org |
| @@ -10842,6 +10854,7 @@ F: drivers/platform/x86/peaq-wmi.c · 503 unmodified lines | |||
| 10842 | PER-CPU MEMORY ALLOCATOR | 10854 | PER-CPU MEMORY ALLOCATOR |
| 10843 | M: Tejun Heo <tj@kernel.org> | 10855 | M: Tejun Heo <tj@kernel.org> |
| 10844 | M: Christoph Lameter <cl@linux.com> | 10856 | M: Christoph Lameter <cl@linux.com> |
| 10857 | M: Dennis Zhou <dennisszhou@gmail.com> | ||
| 10845 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git | 10858 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git |
| 10846 | S: Maintained | 10859 | S: Maintained |
| 10847 | F: include/linux/percpu*.h | 10860 | F: include/linux/percpu*.h |
| @@ -10935,6 +10948,17 @@ L: linux-gpio@vger.kernel.org · 87 unmodified lines | |||
| 10935 | S: Supported | 10948 | S: Supported |
| 10936 | F: drivers/pinctrl/pinctrl-at91-pio4.* | 10949 | F: drivers/pinctrl/pinctrl-at91-pio4.* |
| 10937 | 10950 | ||
| 10951 | PIN CONTROLLER - FREESCALE | ||
| 10952 | M: Dong Aisheng <aisheng.dong@nxp.com> | ||
| 10953 | M: Fabio Estevam <festevam@gmail.com> | ||
| 10954 | M: Shawn Guo <shawnguo@kernel.org> | ||
| 10955 | M: Stefan Agner <stefan@agner.ch> | ||
| 10956 | R: Pengutronix Kernel Team <kernel@pengutronix.de> | ||
| 10957 | L: linux-gpio@vger.kernel.org | ||
| 10958 | S: Maintained | ||
| 10959 | F: drivers/pinctrl/freescale/ | ||
| 10960 | F: Documentation/devicetree/bindings/pinctrl/fsl,* | ||
| 10961 | |||
| 10938 | PIN CONTROLLER - INTEL | 10962 | PIN CONTROLLER - INTEL |
| 10939 | M: Mika Westerberg <mika.westerberg@linux.intel.com> | 10963 | M: Mika Westerberg <mika.westerberg@linux.intel.com> |
| 10940 | M: Heikki Krogerus <heikki.krogerus@linux.intel.com> | 10964 | M: Heikki Krogerus <heikki.krogerus@linux.intel.com> |
| @@ -12101,6 +12125,7 @@ M: Sylwester Nawrocki <s.nawrocki@samsung.com> · 1160 unmodified lines | |||
| 12101 | L: alsa-devel@alsa-project.org (moderated for non-subscribers) | 12125 | L: alsa-devel@alsa-project.org (moderated for non-subscribers) |
| 12102 | S: Supported | 12126 | S: Supported |
| 12103 | F: sound/soc/samsung/ | 12127 | F: sound/soc/samsung/ |
| 12128 | F: Documentation/devicetree/bindings/sound/samsung* | ||
| 12104 | 12129 | ||
| 12105 | SAMSUNG EXYNOS PSEUDO RANDOM NUMBER GENERATOR (RNG) DRIVER | 12130 | SAMSUNG EXYNOS PSEUDO RANDOM NUMBER GENERATOR (RNG) DRIVER |
| 12106 | M: Krzysztof Kozlowski <krzk@kernel.org> | 12131 | M: Krzysztof Kozlowski <krzk@kernel.org> |
| @@ -2,7 +2,7 @@ · 1 unmodified line | |||
| 2 | VERSION = 4 | 2 | VERSION = 4 |
| 3 | PATCHLEVEL = 16 | 3 | PATCHLEVEL = 16 |
| 4 | SUBLEVEL = 0 | 4 | SUBLEVEL = 0 |
| 5 | EXTRAVERSION = -rc1 | 5 | EXTRAVERSION = -rc7 |
| 6 | NAME = Fearless Coyote | 6 | NAME = Fearless Coyote |
| 7 | 7 | ||
| 8 | # *DOCUMENTATION* | 8 | # *DOCUMENTATION* |
| @@ -388,7 +388,7 @@ PYTHON = python · 379 unmodified lines | |||
| 388 | CHECK = sparse | 388 | CHECK = sparse |
| 389 | 389 | ||
| 390 | CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ | 390 | CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ |
| 391 | -Wbitwise -Wno-return-void $(CF) | 391 | -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) |
| 392 | NOSTDINC_FLAGS = | 392 | NOSTDINC_FLAGS = |
| 393 | CFLAGS_MODULE = | 393 | CFLAGS_MODULE = |
| 394 | AFLAGS_MODULE = | 394 | AFLAGS_MODULE = |
| @@ -489,6 +489,11 @@ KBUILD_CFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) · 94 unmodified lines | |||
| 489 | KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) | 489 | KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) |
| 490 | endif | 490 | endif |
| 491 | 491 | ||
| 492 | RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register | ||
| 493 | RETPOLINE_CFLAGS_CLANG := -mretpoline-external-thunk | ||
| 494 | RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(RETPOLINE_CFLAGS_CLANG))) | ||
| 495 | export RETPOLINE_CFLAGS | ||
| 496 | |||
| 492 | ifeq ($(config-targets),1) | 497 | ifeq ($(config-targets),1) |
| 493 | # =========================================================================== | 498 | # =========================================================================== |
| 494 | # *config targets only - make sure prerequisites are updated, and descend | 499 | # *config targets only - make sure prerequisites are updated, and descend |
| @@ -579,10 +584,9 @@ ifeq ($(KBUILD_EXTMOD),) · 84 unmodified lines | |||
| 579 | # To avoid any implicit rule to kick in, define an empty command | 584 | # To avoid any implicit rule to kick in, define an empty command |
| 580 | $(KCONFIG_CONFIG) include/config/auto.conf.cmd: ; | 585 | $(KCONFIG_CONFIG) include/config/auto.conf.cmd: ; |
| 581 | 586 | ||
| 582 | # If .config is newer than include/config/auto.conf, someone tinkered | 587 | # The actual configuration files used during the build are stored in |
| 583 | # with it and forgot to run make oldconfig. | 588 | # include/generated/ and include/config/. Update them if .config is newer than |
| 584 | # if auto.conf.cmd is missing then we are probably in a cleaned tree so | 589 | # include/config/auto.conf (which mirrors .config). |
| 585 | # we execute the config step to be sure to catch updated Kconfig files | ||
| 586 | include/config/%.conf: $(KCONFIG_CONFIG) include/config/auto.conf.cmd | 590 | include/config/%.conf: $(KCONFIG_CONFIG) include/config/auto.conf.cmd |
| 587 | $(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig | 591 | $(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig |
| 588 | else | 592 | else |
| @@ -822,6 +826,15 @@ KBUILD_CFLAGS += $(call cc-disable-warning, pointer-sign) · 233 unmodified lines | |||
| 822 | # disable invalid "can't wrap" optimizations for signed / pointers | 826 | # disable invalid "can't wrap" optimizations for signed / pointers |
| 823 | KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow) | 827 | KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow) |
| 824 | 828 | ||
| 829 | # clang sets -fmerge-all-constants by default as optimization, but this | ||
| 830 | # is non-conforming behavior for C and in fact breaks the kernel, so we | ||
| 831 | # need to disable it here generally. | ||
| 832 | KBUILD_CFLAGS += $(call cc-option,-fno-merge-all-constants) | ||
| 833 | |||
| 834 | # for gcc -fno-merge-all-constants disables everything, but it is fine | ||
| 835 | # to have actual conforming behavior enabled. | ||
| 836 | KBUILD_CFLAGS += $(call cc-option,-fmerge-constants) | ||
| 837 | |||
| 825 | # Make sure -fstack-check isn't enabled (like gentoo apparently did) | 838 | # Make sure -fstack-check isn't enabled (like gentoo apparently did) |
| 826 | KBUILD_CFLAGS += $(call cc-option,-fno-stack-check,) | 839 | KBUILD_CFLAGS += $(call cc-option,-fno-stack-check,) |
| 827 | 840 | ||
| @@ -857,8 +870,7 @@ KBUILD_AFLAGS += $(ARCH_AFLAGS) $(KAFLAGS) · 29 unmodified lines | |||
| 857 | KBUILD_CFLAGS += $(ARCH_CFLAGS) $(KCFLAGS) | 870 | KBUILD_CFLAGS += $(ARCH_CFLAGS) $(KCFLAGS) |
| 858 | 871 | ||
| 859 | # Use --build-id when available. | 872 | # Use --build-id when available. |
| 860 | LDFLAGS_BUILD_ID := $(patsubst -Wl$(comma)%,%,\ | 873 | LDFLAGS_BUILD_ID := $(call ld-option, --build-id) |
| 861 | $(call cc-ldoption, -Wl$(comma)--build-id,)) | ||
| 862 | KBUILD_LDFLAGS_MODULE += $(LDFLAGS_BUILD_ID) | 874 | KBUILD_LDFLAGS_MODULE += $(LDFLAGS_BUILD_ID) |
| 863 | LDFLAGS_vmlinux += $(LDFLAGS_BUILD_ID) | 875 | LDFLAGS_vmlinux += $(LDFLAGS_BUILD_ID) |
| 864 | 876 | ||
| @@ -6,7 +6,6 @@ · 5 unmodified lines | |||
| 6 | * Atomic exchange routines. | 6 | * Atomic exchange routines. |
| 7 | */ | 7 | */ |
| 8 | 8 | ||
| 9 | #define __ASM__MB | ||
| 10 | #define ____xchg(type, args...) __xchg ## type ## _local(args) | 9 | #define ____xchg(type, args...) __xchg ## type ## _local(args) |
| 11 | #define ____cmpxchg(type, args...) __cmpxchg ## type ## _local(args) | 10 | #define ____cmpxchg(type, args...) __cmpxchg ## type ## _local(args) |
| 12 | #include <asm/xchg.h> | 11 | #include <asm/xchg.h> |
| @@ -33,10 +32,6 @@ · 20 unmodified lines | |||
| 33 | cmpxchg_local((ptr), (o), (n)); \ | 32 | cmpxchg_local((ptr), (o), (n)); \ |
| 34 | }) | 33 | }) |
| 35 | 34 | ||
| 36 | #ifdef CONFIG_SMP | ||
| 37 | #undef __ASM__MB | ||
| 38 | #define __ASM__MB "\tmb\n" | ||
| 39 | #endif | ||
| 40 | #undef ____xchg | 35 | #undef ____xchg |
| 41 | #undef ____cmpxchg | 36 | #undef ____cmpxchg |
| 42 | #define ____xchg(type, args...) __xchg ##type(args) | 37 | #define ____xchg(type, args...) __xchg ##type(args) |
| @@ -64,7 +59,6 @@ · 21 unmodified lines | |||
| 64 | cmpxchg((ptr), (o), (n)); \ | 59 | cmpxchg((ptr), (o), (n)); \ |
| 65 | }) | 60 | }) |
| 66 | 61 | ||
| 67 | #undef __ASM__MB | ||
| 68 | #undef ____cmpxchg | 62 | #undef ____cmpxchg |
| 69 | 63 | ||
| 70 | #endif /* _ALPHA_CMPXCHG_H */ | 64 | #endif /* _ALPHA_CMPXCHG_H */ |
| @@ -12,6 +12,10 @@ · 11 unmodified lines | |||
| 12 | * Atomic exchange. | 12 | * Atomic exchange. |
| 13 | * Since it can be used to implement critical sections | 13 | * Since it can be used to implement critical sections |
| 14 | * it must clobber "memory" (also for interrupts in UP). | 14 | * it must clobber "memory" (also for interrupts in UP). |
| 15 | * | ||
| 16 | * The leading and the trailing memory barriers guarantee that these | ||
| 17 | * operations are fully ordered. | ||
| 18 | * | ||
| 15 | */ | 19 | */ |
| 16 | 20 | ||
| 17 | static inline unsigned long | 21 | static inline unsigned long |
| @@ -19,6 +23,7 @@ ____xchg(_u8, volatile char *m, unsigned long val) · 1 unmodified line | |||
| 19 | { | 23 | { |
| 20 | unsigned long ret, tmp, addr64; | 24 | unsigned long ret, tmp, addr64; |
| 21 | 25 | ||
| 26 | smp_mb(); | ||
| 22 | __asm__ __volatile__( | 27 | __asm__ __volatile__( |
| 23 | " andnot %4,7,%3\n" | 28 | " andnot %4,7,%3\n" |
| 24 | " insbl %1,%4,%1\n" | 29 | " insbl %1,%4,%1\n" |
| @@ -28,12 +33,12 @@ ____xchg(_u8, volatile char *m, unsigned long val) · 3 unmodified lines | |||
| 28 | " or %1,%2,%2\n" | 33 | " or %1,%2,%2\n" |
| 29 | " stq_c %2,0(%3)\n" | 34 | " stq_c %2,0(%3)\n" |
| 30 | " beq %2,2f\n" | 35 | " beq %2,2f\n" |
| 31 | __ASM__MB | ||
| 32 | ".subsection 2\n" | 36 | ".subsection 2\n" |
| 33 | "2: br 1b\n" | 37 | "2: br 1b\n" |
| 34 | ".previous" | 38 | ".previous" |
| 35 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) | 39 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) |
| 36 | : "r" ((long)m), "1" (val) : "memory"); | 40 | : "r" ((long)m), "1" (val) : "memory"); |
| 41 | smp_mb(); | ||
| 37 | 42 | ||
| 38 | return ret; | 43 | return ret; |
| 39 | } | 44 | } |
| @@ -43,6 +48,7 @@ ____xchg(_u16, volatile short *m, unsigned long val) · 3 unmodified lines | |||
| 43 | { | 48 | { |
| 44 | unsigned long ret, tmp, addr64; | 49 | unsigned long ret, tmp, addr64; |
| 45 | 50 | ||
| 51 | smp_mb(); | ||
| 46 | __asm__ __volatile__( | 52 | __asm__ __volatile__( |
| 47 | " andnot %4,7,%3\n" | 53 | " andnot %4,7,%3\n" |
| 48 | " inswl %1,%4,%1\n" | 54 | " inswl %1,%4,%1\n" |
| @@ -52,12 +58,12 @@ ____xchg(_u16, volatile short *m, unsigned long val) · 3 unmodified lines | |||
| 52 | " or %1,%2,%2\n" | 58 | " or %1,%2,%2\n" |
| 53 | " stq_c %2,0(%3)\n" | 59 | " stq_c %2,0(%3)\n" |
| 54 | " beq %2,2f\n" | 60 | " beq %2,2f\n" |
| 55 | __ASM__MB | ||
| 56 | ".subsection 2\n" | 61 | ".subsection 2\n" |
| 57 | "2: br 1b\n" | 62 | "2: br 1b\n" |
| 58 | ".previous" | 63 | ".previous" |
| 59 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) | 64 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) |
| 60 | : "r" ((long)m), "1" (val) : "memory"); | 65 | : "r" ((long)m), "1" (val) : "memory"); |
| 66 | smp_mb(); | ||
| 61 | 67 | ||
| 62 | return ret; | 68 | return ret; |
| 63 | } | 69 | } |
| @@ -67,17 +73,18 @@ ____xchg(_u32, volatile int *m, unsigned long val) · 3 unmodified lines | |||
| 67 | { | 73 | { |
| 68 | unsigned long dummy; | 74 | unsigned long dummy; |
| 69 | 75 | ||
| 76 | smp_mb(); | ||
| 70 | __asm__ __volatile__( | 77 | __asm__ __volatile__( |
| 71 | "1: ldl_l %0,%4\n" | 78 | "1: ldl_l %0,%4\n" |
| 72 | " bis $31,%3,%1\n" | 79 | " bis $31,%3,%1\n" |
| 73 | " stl_c %1,%2\n" | 80 | " stl_c %1,%2\n" |
| 74 | " beq %1,2f\n" | 81 | " beq %1,2f\n" |
| 75 | __ASM__MB | ||
| 76 | ".subsection 2\n" | 82 | ".subsection 2\n" |
| 77 | "2: br 1b\n" | 83 | "2: br 1b\n" |
| 78 | ".previous" | 84 | ".previous" |
| 79 | : "=&r" (val), "=&r" (dummy), "=m" (*m) | 85 | : "=&r" (val), "=&r" (dummy), "=m" (*m) |
| 80 | : "rI" (val), "m" (*m) : "memory"); | 86 | : "rI" (val), "m" (*m) : "memory"); |
| 87 | smp_mb(); | ||
| 81 | 88 | ||
| 82 | return val; | 89 | return val; |
| 83 | } | 90 | } |
| @@ -87,17 +94,18 @@ ____xchg(_u64, volatile long *m, unsigned long val) · 3 unmodified lines | |||
| 87 | { | 94 | { |
| 88 | unsigned long dummy; | 95 | unsigned long dummy; |
| 89 | 96 | ||
| 97 | smp_mb(); | ||
| 90 | __asm__ __volatile__( | 98 | __asm__ __volatile__( |
| 91 | "1: ldq_l %0,%4\n" | 99 | "1: ldq_l %0,%4\n" |
| 92 | " bis $31,%3,%1\n" | 100 | " bis $31,%3,%1\n" |
| 93 | " stq_c %1,%2\n" | 101 | " stq_c %1,%2\n" |
| 94 | " beq %1,2f\n" | 102 | " beq %1,2f\n" |
| 95 | __ASM__MB | ||
| 96 | ".subsection 2\n" | 103 | ".subsection 2\n" |
| 97 | "2: br 1b\n" | 104 | "2: br 1b\n" |
| 98 | ".previous" | 105 | ".previous" |
| 99 | : "=&r" (val), "=&r" (dummy), "=m" (*m) | 106 | : "=&r" (val), "=&r" (dummy), "=m" (*m) |
| 100 | : "rI" (val), "m" (*m) : "memory"); | 107 | : "rI" (val), "m" (*m) : "memory"); |
| 108 | smp_mb(); | ||
| 101 | 109 | ||
| 102 | return val; | 110 | return val; |
| 103 | } | 111 | } |
| @@ -128,10 +136,12 @@ ____xchg(, volatile void *ptr, unsigned long x, int size) · 24 unmodified lines | |||
| 128 | * store NEW in MEM. Return the initial value in MEM. Success is | 136 | * store NEW in MEM. Return the initial value in MEM. Success is |
| 129 | * indicated by comparing RETURN with OLD. | 137 | * indicated by comparing RETURN with OLD. |
| 130 | * | 138 | * |
| 131 | * The memory barrier should be placed in SMP only when we actually | 139 | * The leading and the trailing memory barriers guarantee that these |
| 132 | * make the change. If we don't change anything (so if the returned | 140 | * operations are fully ordered. |
| 133 | * prev is equal to old) then we aren't acquiring anything new and | 141 | * |
| 134 | * we don't need any memory barrier as far I can tell. | 142 | * The trailing memory barrier is placed in SMP unconditionally, in |
| 143 | * order to guarantee that dependency ordering is preserved when a | ||
| 144 | * dependency is headed by an unsuccessful operation. | ||
| 135 | */ | 145 | */ |
| 136 | 146 | ||
| 137 | static inline unsigned long | 147 | static inline unsigned long |
| @@ -139,6 +149,7 @@ ____cmpxchg(_u8, volatile char *m, unsigned char old, unsigned char new) · 1 unmodified line | |||
| 139 | { | 149 | { |
| 140 | unsigned long prev, tmp, cmp, addr64; | 150 | unsigned long prev, tmp, cmp, addr64; |
| 141 | 151 | ||
| 152 | smp_mb(); | ||
| 142 | __asm__ __volatile__( | 153 | __asm__ __volatile__( |
| 143 | " andnot %5,7,%4\n" | 154 | " andnot %5,7,%4\n" |
| 144 | " insbl %1,%5,%1\n" | 155 | " insbl %1,%5,%1\n" |
| @@ -150,13 +161,13 @@ ____cmpxchg(_u8, volatile char *m, unsigned char old, unsigned char new) · 5 unmodified lines | |||
| 150 | " or %1,%2,%2\n" | 161 | " or %1,%2,%2\n" |
| 151 | " stq_c %2,0(%4)\n" | 162 | " stq_c %2,0(%4)\n" |
| 152 | " beq %2,3f\n" | 163 | " beq %2,3f\n" |
| 153 | __ASM__MB | ||
| 154 | "2:\n" | 164 | "2:\n" |
| 155 | ".subsection 2\n" | 165 | ".subsection 2\n" |
| 156 | "3: br 1b\n" | 166 | "3: br 1b\n" |
| 157 | ".previous" | 167 | ".previous" |
| 158 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) | 168 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) |
| 159 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); | 169 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); |
| 170 | smp_mb(); | ||
| 160 | 171 | ||
| 161 | return prev; | 172 | return prev; |
| 162 | } | 173 | } |
| @@ -166,6 +177,7 @@ ____cmpxchg(_u16, volatile short *m, unsigned short old, unsigned short new) · 3 unmodified lines | |||
| 166 | { | 177 | { |
| 167 | unsigned long prev, tmp, cmp, addr64; | 178 | unsigned long prev, tmp, cmp, addr64; |
| 168 | 179 | ||
| 180 | smp_mb(); | ||
| 169 | __asm__ __volatile__( | 181 | __asm__ __volatile__( |
| 170 | " andnot %5,7,%4\n" | 182 | " andnot %5,7,%4\n" |
| 171 | " inswl %1,%5,%1\n" | 183 | " inswl %1,%5,%1\n" |
| @@ -177,13 +189,13 @@ ____cmpxchg(_u16, volatile short *m, unsigned short old, unsigned short new) · 5 unmodified lines | |||
| 177 | " or %1,%2,%2\n" | 189 | " or %1,%2,%2\n" |
| 178 | " stq_c %2,0(%4)\n" | 190 | " stq_c %2,0(%4)\n" |
| 179 | " beq %2,3f\n" | 191 | " beq %2,3f\n" |
| 180 | __ASM__MB | ||
| 181 | "2:\n" | 192 | "2:\n" |
| 182 | ".subsection 2\n" | 193 | ".subsection 2\n" |
| 183 | "3: br 1b\n" | 194 | "3: br 1b\n" |
| 184 | ".previous" | 195 | ".previous" |
| 185 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) | 196 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) |
| 186 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); | 197 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); |
| 198 | smp_mb(); | ||
| 187 | 199 | ||
| 188 | return prev; | 200 | return prev; |
| 189 | } | 201 | } |
| @@ -193,6 +205,7 @@ ____cmpxchg(_u32, volatile int *m, int old, int new) · 3 unmodified lines | |||
| 193 | { | 205 | { |
| 194 | unsigned long prev, cmp; | 206 | unsigned long prev, cmp; |
| 195 | 207 | ||
| 208 | smp_mb(); | ||
| 196 | __asm__ __volatile__( | 209 | __asm__ __volatile__( |
| 197 | "1: ldl_l %0,%5\n" | 210 | "1: ldl_l %0,%5\n" |
| 198 | " cmpeq %0,%3,%1\n" | 211 | " cmpeq %0,%3,%1\n" |
| @@ -200,13 +213,13 @@ ____cmpxchg(_u32, volatile int *m, int old, int new) · 1 unmodified line | |||
| 200 | " mov %4,%1\n" | 213 | " mov %4,%1\n" |
| 201 | " stl_c %1,%2\n" | 214 | " stl_c %1,%2\n" |
| 202 | " beq %1,3f\n" | 215 | " beq %1,3f\n" |
| 203 | __ASM__MB | ||
| 204 | "2:\n" | 216 | "2:\n" |
| 205 | ".subsection 2\n" | 217 | ".subsection 2\n" |
| 206 | "3: br 1b\n" | 218 | "3: br 1b\n" |
| 207 | ".previous" | 219 | ".previous" |
| 208 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) | 220 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) |
| 209 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); | 221 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); |
| 222 | smp_mb(); | ||
| 210 | 223 | ||
| 211 | return prev; | 224 | return prev; |
| 212 | } | 225 | } |
| @@ -216,6 +229,7 @@ ____cmpxchg(_u64, volatile long *m, unsigned long old, unsigned long new) · 3 unmodified lines | |||
| 216 | { | 229 | { |
| 217 | unsigned long prev, cmp; | 230 | unsigned long prev, cmp; |
| 218 | 231 | ||
| 232 | smp_mb(); | ||
| 219 | __asm__ __volatile__( | 233 | __asm__ __volatile__( |
| 220 | "1: ldq_l %0,%5\n" | 234 | "1: ldq_l %0,%5\n" |
| 221 | " cmpeq %0,%3,%1\n" | 235 | " cmpeq %0,%3,%1\n" |
| @@ -223,13 +237,13 @@ ____cmpxchg(_u64, volatile long *m, unsigned long old, unsigned long new) · 1 unmodified line | |||
| 223 | " mov %4,%1\n" | 237 | " mov %4,%1\n" |
| 224 | " stq_c %1,%2\n" | 238 | " stq_c %1,%2\n" |
| 225 | " beq %1,3f\n" | 239 | " beq %1,3f\n" |
| 226 | __ASM__MB | ||
| 227 | "2:\n" | 240 | "2:\n" |
| 228 | ".subsection 2\n" | 241 | ".subsection 2\n" |
| 229 | "3: br 1b\n" | 242 | "3: br 1b\n" |
| 230 | ".previous" | 243 | ".previous" |
| 231 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) | 244 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) |
| 232 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); | 245 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); |
| 246 | smp_mb(); | ||
| 233 | 247 | ||
| 234 | return prev; | 248 | return prev; |
| 235 | } | 249 | } |
| @@ -484,7 +484,6 @@ config ARC_CURR_IN_REG · 483 unmodified lines | |||
| 484 | 484 | ||
| 485 | config ARC_EMUL_UNALIGNED | 485 | config ARC_EMUL_UNALIGNED |
| 486 | bool "Emulate unaligned memory access (userspace only)" | 486 | bool "Emulate unaligned memory access (userspace only)" |
| 487 | default N | ||
| 488 | select SYSCTL_ARCH_UNALIGN_NO_WARN | 487 | select SYSCTL_ARCH_UNALIGN_NO_WARN |
| 489 | select SYSCTL_ARCH_UNALIGN_ALLOW | 488 | select SYSCTL_ARCH_UNALIGN_ALLOW |
| 490 | depends on ISA_ARCOMPACT | 489 | depends on ISA_ARCOMPACT |
| @@ -17,6 +17,6 @@ · 16 unmodified lines | |||
| 17 | compatible = "snps,axs101", "snps,arc-sdp"; | 17 | compatible = "snps,axs101", "snps,arc-sdp"; |
| 18 | 18 | ||
| 19 | chosen { | 19 | chosen { |
| 20 | bootargs = "earlycon=uart8250,mmio32,0xe0022000,115200n8 console=tty0 console=ttyS3,115200n8 consoleblank=0 video=1280x720@60"; | 20 | bootargs = "earlycon=uart8250,mmio32,0xe0022000,115200n8 console=tty0 console=ttyS3,115200n8 consoleblank=0 video=1280x720@60 print-fatal-signals=1"; |
| 21 | }; | 21 | }; |
| 22 | }; | 22 | }; |
| @@ -214,13 +214,13 @@ · 213 unmodified lines | |||
| 214 | }; | 214 | }; |
| 215 | 215 | ||
| 216 | eeprom@0x54{ | 216 | eeprom@0x54{ |
| 217 | compatible = "24c01"; | 217 | compatible = "atmel,24c01"; |
| 218 | reg = <0x54>; | 218 | reg = <0x54>; |
| 219 | pagesize = <0x8>; | 219 | pagesize = <0x8>; |
| 220 | }; | 220 | }; |
| 221 | 221 | ||
| 222 | eeprom@0x57{ | 222 | eeprom@0x57{ |
| 223 | compatible = "24c04"; | 223 | compatible = "atmel,24c04"; |
| 224 | reg = <0x57>; | 224 | reg = <0x57>; |
| 225 | pagesize = <0x8>; | 225 | pagesize = <0x8>; |
| 226 | }; | 226 | }; |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | |||
| 22 | }; | 22 | }; |
| 23 | 23 | ||
| 24 | chosen { | 24 | chosen { |
| 25 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=ttyS0,115200n8 debug"; | 25 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=ttyS0,115200n8 debug print-fatal-signals=1"; |
| 26 | }; | 26 | }; |
| 27 | 27 | ||
| 28 | aliases { | 28 | aliases { |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | |||
| 17 | interrupt-parent = <&core_intc>; | 17 | interrupt-parent = <&core_intc>; |
| 18 | 18 | ||
| 19 | chosen { | 19 | chosen { |
| 20 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8"; | 20 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8 print-fatal-signals=1"; |
| 21 | }; | 21 | }; |
| 22 | 22 | ||
| 23 | aliases { | 23 | aliases { |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | |||
| 24 | }; | 24 | }; |
| 25 | 25 | ||
| 26 | chosen { | 26 | chosen { |
| 27 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8"; | 27 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8 print-fatal-signals=1"; |
| 28 | }; | 28 | }; |
| 29 | 29 | ||
| 30 | aliases { | 30 | aliases { |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | |||
| 15 | interrupt-parent = <&core_intc>; | 15 | interrupt-parent = <&core_intc>; |
| 16 | 16 | ||
| 17 | chosen { | 17 | chosen { |
| 18 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8"; | 18 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8 print-fatal-signals=1"; |
| 19 | }; | 19 | }; |
| 20 | 20 | ||
| 21 | aliases { | 21 | aliases { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | |||
| 20 | /* this is for console on PGU */ | 20 | /* this is for console on PGU */ |
| 21 | /* bootargs = "console=tty0 consoleblank=0"; */ | 21 | /* bootargs = "console=tty0 consoleblank=0"; */ |
| 22 | /* this is for console on serial */ | 22 | /* this is for console on serial */ |
| 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24"; | 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24 print-fatal-signals=1"; |
| 24 | }; | 24 | }; |
| 25 | 25 | ||
| 26 | aliases { | 26 | aliases { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | |||
| 20 | /* this is for console on PGU */ | 20 | /* this is for console on PGU */ |
| 21 | /* bootargs = "console=tty0 consoleblank=0"; */ | 21 | /* bootargs = "console=tty0 consoleblank=0"; */ |
| 22 | /* this is for console on serial */ | 22 | /* this is for console on serial */ |
| 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24"; | 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24 print-fatal-signals=1"; |
| 24 | }; | 24 | }; |
| 25 | 25 | ||
| 26 | aliases { | 26 | aliases { |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | |||
| 18 | 18 | ||
| 19 | chosen { | 19 | chosen { |
| 20 | /* this is for console on serial */ | 20 | /* this is for console on serial */ |
| 21 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblan=0 debug video=640x480-24"; | 21 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblan=0 debug video=640x480-24 print-fatal-signals=1"; |
| 22 | }; | 22 | }; |
| 23 | 23 | ||
| 24 | aliases { | 24 | aliases { |
| @@ -23,7 +23,8 @@ void die(const char *str, struct pt_regs *regs, unsigned long address); · 22 unmodified lines | |||
| 23 | 23 | ||
| 24 | #define BUG() do { \ | 24 | #define BUG() do { \ |
| 25 | pr_warn("BUG: failure at %s:%d/%s()!\n", __FILE__, __LINE__, __func__); \ | 25 | pr_warn("BUG: failure at %s:%d/%s()!\n", __FILE__, __LINE__, __func__); \ |
| 26 | dump_stack(); \ | 26 | barrier_before_unreachable(); \ |
| 27 | __builtin_trap(); \ | ||
| 27 | } while (0) | 28 | } while (0) |
| 28 | 29 | ||
| 29 | #define HAVE_ARCH_BUG | 30 | #define HAVE_ARCH_BUG |
| @@ -184,7 +184,7 @@ · 183 unmodified lines | |||
| 184 | .macro FAKE_RET_FROM_EXCPN | 184 | .macro FAKE_RET_FROM_EXCPN |
| 185 | lr r9, [status32] | 185 | lr r9, [status32] |
| 186 | bic r9, r9, (STATUS_U_MASK|STATUS_DE_MASK|STATUS_AE_MASK) | 186 | bic r9, r9, (STATUS_U_MASK|STATUS_DE_MASK|STATUS_AE_MASK) |
| 187 | or r9, r9, (STATUS_L_MASK|STATUS_IE_MASK) | 187 | or r9, r9, STATUS_IE_MASK |
| 188 | kflag r9 | 188 | kflag r9 |
| 189 | .endm | 189 | .endm |
| 190 | 190 | ||
| @@ -22,10 +22,79 @@ static DEFINE_RAW_SPINLOCK(mcip_lock); · 21 unmodified lines | |||
| 22 | 22 | ||
| 23 | static char smp_cpuinfo_buf[128]; | 23 | static char smp_cpuinfo_buf[128]; |
| 24 | 24 | ||
| 25 | /* | ||
| 26 | * Set mask to halt GFRC if any online core in SMP cluster is halted. | ||
| 27 | * Only works for ARC HS v3.0+, on earlier versions has no effect. | ||
| 28 | */ | ||
| 29 | static void mcip_update_gfrc_halt_mask(int cpu) | ||
| 30 | { | ||
| 31 | struct bcr_generic gfrc; | ||
| 32 | unsigned long flags; | ||
| 33 | u32 gfrc_halt_mask; | ||
| 34 | |||
| 35 | READ_BCR(ARC_REG_GFRC_BUILD, gfrc); | ||
| 36 | |||
| 37 | /* | ||
| 38 | * CMD_GFRC_SET_CORE and CMD_GFRC_READ_CORE commands were added in | ||
| 39 | * GFRC 0x3 version. | ||
| 40 | */ | ||
| 41 | if (gfrc.ver < 0x3) | ||
| 42 | return; | ||
| 43 | |||
| 44 | raw_spin_lock_irqsave(&mcip_lock, flags); | ||
| 45 | |||
| 46 | __mcip_cmd(CMD_GFRC_READ_CORE, 0); | ||
| 47 | gfrc_halt_mask = read_aux_reg(ARC_REG_MCIP_READBACK); | ||
| 48 | gfrc_halt_mask |= BIT(cpu); | ||
| 49 | __mcip_cmd_data(CMD_GFRC_SET_CORE, 0, gfrc_halt_mask); | ||
| 50 | |||
| 51 | raw_spin_unlock_irqrestore(&mcip_lock, flags); | ||
| 52 | } | ||
| 53 | |||
| 54 | static void mcip_update_debug_halt_mask(int cpu) | ||
| 55 | { | ||
| 56 | u32 mcip_mask = 0; | ||
| 57 | unsigned long flags; | ||
| 58 | |||
| 59 | raw_spin_lock_irqsave(&mcip_lock, flags); | ||
| 60 | |||
| 61 | /* | ||
| 62 | * mcip_mask is same for CMD_DEBUG_SET_SELECT and CMD_DEBUG_SET_MASK | ||
| 63 | * commands. So read it once instead of reading both CMD_DEBUG_READ_MASK | ||
| 64 | * and CMD_DEBUG_READ_SELECT. | ||
| 65 | */ | ||
| 66 | __mcip_cmd(CMD_DEBUG_READ_SELECT, 0); | ||
| 67 | mcip_mask = read_aux_reg(ARC_REG_MCIP_READBACK); | ||
| 68 | |||
| 69 | mcip_mask |= BIT(cpu); | ||
| 70 | |||
| 71 | __mcip_cmd_data(CMD_DEBUG_SET_SELECT, 0, mcip_mask); | ||
| 72 | /* | ||
| 73 | * Parameter specified halt cause: | ||
| 74 | * STATUS32[H]/actionpoint/breakpoint/self-halt | ||
| 75 | * We choose all of them (0xF). | ||
| 76 | */ | ||
| 77 | __mcip_cmd_data(CMD_DEBUG_SET_MASK, 0xF, mcip_mask); | ||
| 78 | |||
| 79 | raw_spin_unlock_irqrestore(&mcip_lock, flags); | ||
| 80 | } | ||
| 81 | |||
| 25 | static void mcip_setup_per_cpu(int cpu) | 82 | static void mcip_setup_per_cpu(int cpu) |
| 26 | { | 83 | { |
| 84 | struct mcip_bcr mp; | ||
| 85 | |||
| 86 | READ_BCR(ARC_REG_MCIP_BCR, mp); | ||
| 87 | |||
| 27 | smp_ipi_irq_setup(cpu, IPI_IRQ); | 88 | smp_ipi_irq_setup(cpu, IPI_IRQ); |
| 28 | smp_ipi_irq_setup(cpu, SOFTIRQ_IRQ); | 89 | smp_ipi_irq_setup(cpu, SOFTIRQ_IRQ); |
| 90 | |||
| 91 | /* Update GFRC halt mask as new CPU came online */ | ||
| 92 | if (mp.gfrc) | ||
| 93 | mcip_update_gfrc_halt_mask(cpu); | ||
| 94 | |||
| 95 | /* Update MCIP debug mask as new CPU came online */ | ||
| 96 | if (mp.dbg) | ||
| 97 | mcip_update_debug_halt_mask(cpu); | ||
| 29 | } | 98 | } |
| 30 | 99 | ||
| 31 | static void mcip_ipi_send(int cpu) | 100 | static void mcip_ipi_send(int cpu) |
| @@ -101,11 +170,6 @@ static void mcip_probe_n_setup(void) · 69 unmodified lines | |||
| 101 | IS_AVAIL1(mp.gfrc, "GFRC")); | 170 | IS_AVAIL1(mp.gfrc, "GFRC")); |
| 102 | 171 | ||
| 103 | cpuinfo_arc700[0].extn.gfrc = mp.gfrc; | 172 | cpuinfo_arc700[0].extn.gfrc = mp.gfrc; |
| 104 | |||
| 105 | if (mp.dbg) { | ||
| 106 | __mcip_cmd_data(CMD_DEBUG_SET_SELECT, 0, 0xf); | ||
| 107 | __mcip_cmd_data(CMD_DEBUG_SET_MASK, 0xf, 0xf); | ||
| 108 | } | ||
| 109 | } | 173 | } |
| 110 | 174 | ||
| 111 | struct plat_smp_ops plat_smp_ops = { | 175 | struct plat_smp_ops plat_smp_ops = { |
| @@ -51,7 +51,7 @@ static const struct id_to_str arc_cpu_rel[] = { · 50 unmodified lines | |||
| 51 | { 0x51, "R2.0" }, | 51 | { 0x51, "R2.0" }, |
| 52 | { 0x52, "R2.1" }, | 52 | { 0x52, "R2.1" }, |
| 53 | { 0x53, "R3.0" }, | 53 | { 0x53, "R3.0" }, |
| 54 | { 0x54, "R4.0" }, | 54 | { 0x54, "R3.10a" }, |
| 55 | #endif | 55 | #endif |
| 56 | { 0x00, NULL } | 56 | { 0x00, NULL } |
| 57 | }; | 57 | }; |
| @@ -373,7 +373,7 @@ static void arc_chk_core_config(void) · 315 unmodified lines | |||
| 373 | { | 373 | { |
| 374 | struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()]; | 374 | struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()]; |
| 375 | int saved = 0, present = 0; | 375 | int saved = 0, present = 0; |
| 376 | char *opt_nm = NULL;; | 376 | char *opt_nm = NULL; |
| 377 | 377 | ||
| 378 | if (!cpu->extn.timer0) | 378 | if (!cpu->extn.timer0) |
| 379 | panic("Timer0 is not present!\n"); | 379 | panic("Timer0 is not present!\n"); |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | |||
| 24 | #include <linux/reboot.h> | 24 | #include <linux/reboot.h> |
| 25 | #include <linux/irqdomain.h> | 25 | #include <linux/irqdomain.h> |
| 26 | #include <linux/export.h> | 26 | #include <linux/export.h> |
| 27 | #include <linux/of_fdt.h> | ||
| 27 | 28 | ||
| 28 | #include <asm/processor.h> | 29 | #include <asm/processor.h> |
| 29 | #include <asm/setup.h> | 30 | #include <asm/setup.h> |
| @@ -47,6 +48,42 @@ void __init smp_prepare_boot_cpu(void) · 17 unmodified lines | |||
| 47 | { | 48 | { |
| 48 | } | 49 | } |
| 49 | 50 | ||
| 51 | static int __init arc_get_cpu_map(const char *name, struct cpumask *cpumask) | ||
| 52 | { | ||
| 53 | unsigned long dt_root = of_get_flat_dt_root(); | ||
| 54 | const char *buf; | ||
| 55 | |||
| 56 | buf = of_get_flat_dt_prop(dt_root, name, NULL); | ||
| 57 | if (!buf) | ||
| 58 | return -EINVAL; | ||
| 59 | |||
| 60 | if (cpulist_parse(buf, cpumask)) | ||
| 61 | return -EINVAL; | ||
| 62 | |||
| 63 | return 0; | ||
| 64 | } | ||
| 65 | |||
| 66 | /* | ||
| 67 | * Read from DeviceTree and setup cpu possible mask. If there is no | ||
| 68 | * "possible-cpus" property in DeviceTree pretend all [0..NR_CPUS-1] exist. | ||
| 69 | */ | ||
| 70 | static void __init arc_init_cpu_possible(void) | ||
| 71 | { | ||
| 72 | struct cpumask cpumask; | ||
| 73 | |||
| 74 | if (arc_get_cpu_map("possible-cpus", &cpumask)) { | ||
| 75 | pr_warn("Failed to get possible-cpus from dtb, pretending all %u cpus exist\n", | ||
| 76 | NR_CPUS); | ||
| 77 | |||
| 78 | cpumask_setall(&cpumask); | ||
| 79 | } | ||
| 80 | |||
| 81 | if (!cpumask_test_cpu(0, &cpumask)) | ||
| 82 | panic("Master cpu (cpu[0]) is missed in cpu possible mask!"); | ||
| 83 | |||
| 84 | init_cpu_possible(&cpumask); | ||
| 85 | } | ||
| 86 | |||
| 50 | /* | 87 | /* |
| 51 | * Called from setup_arch() before calling setup_processor() | 88 | * Called from setup_arch() before calling setup_processor() |
| 52 | * | 89 | * |
| @@ -58,10 +95,7 @@ void __init smp_prepare_boot_cpu(void) · 5 unmodified lines | |||
| 58 | */ | 95 | */ |
| 59 | void __init smp_init_cpus(void) | 96 | void __init smp_init_cpus(void) |
| 60 | { | 97 | { |
| 61 | unsigned int i; | 98 | arc_init_cpu_possible(); |
| 62 | |||
| 63 | for (i = 0; i < NR_CPUS; i++) | ||
| 64 | set_cpu_possible(i, true); | ||
| 65 | 99 | ||
| 66 | if (plat_smp_ops.init_early_smp) | 100 | if (plat_smp_ops.init_early_smp) |
| 67 | plat_smp_ops.init_early_smp(); | 101 | plat_smp_ops.init_early_smp(); |
| @@ -70,16 +104,12 @@ void __init smp_init_cpus(void) · 2 unmodified lines | |||
| 70 | /* called from init ( ) => process 1 */ | 104 | /* called from init ( ) => process 1 */ |
| 71 | void __init smp_prepare_cpus(unsigned int max_cpus) | 105 | void __init smp_prepare_cpus(unsigned int max_cpus) |
| 72 | { | 106 | { |
| 73 | int i; | ||
| 74 | |||
| 75 | /* | 107 | /* |
| 76 | * if platform didn't set the present map already, do it now | 108 | * if platform didn't set the present map already, do it now |
| 77 | * boot cpu is set to present already by init/main.c | 109 | * boot cpu is set to present already by init/main.c |
| 78 | */ | 110 | */ |
| 79 | if (num_present_cpus() <= 1) { | 111 | if (num_present_cpus() <= 1) |
| 80 | for (i = 0; i < max_cpus; i++) | 112 | init_cpu_present(cpu_possible_mask); |
| 81 | set_cpu_present(i, true); | ||
| 82 | } | ||
| 83 | } | 113 | } |
| 84 | 114 | ||
| 85 | void __init smp_cpus_done(unsigned int max_cpus) | 115 | void __init smp_cpus_done(unsigned int max_cpus) |
| @@ -366,7 +366,7 @@ static void init_unwind_hdr(struct unwind_table *table, · 365 unmodified lines | |||
| 366 | return; | 366 | return; |
| 367 | 367 | ||
| 368 | ret_err: | 368 | ret_err: |
| 369 | panic("Attention !!! Dwarf FDE parsing errors\n");; | 369 | panic("Attention !!! Dwarf FDE parsing errors\n"); |
| 370 | } | 370 | } |
| 371 | 371 | ||
| 372 | #ifdef CONFIG_MODULES | 372 | #ifdef CONFIG_MODULES |
| @@ -780,7 +780,10 @@ noinline static void slc_entire_op(const int op) · 779 unmodified lines | |||
| 780 | 780 | ||
| 781 | write_aux_reg(r, ctrl); | 781 | write_aux_reg(r, ctrl); |
| 782 | 782 | ||
| 783 | write_aux_reg(ARC_REG_SLC_INVALIDATE, 1); | 783 | if (op & OP_INV) /* Inv or flush-n-inv use same cmd reg */ |
| 784 | write_aux_reg(ARC_REG_SLC_INVALIDATE, 0x1); | ||
| 785 | else | ||
| 786 | write_aux_reg(ARC_REG_SLC_FLUSH, 0x1); | ||
| 784 | 787 | ||
| 785 | /* Make sure "busy" bit reports correct stataus, see STAR 9001165532 */ | 788 | /* Make sure "busy" bit reports correct stataus, see STAR 9001165532 */ |
| 786 | read_aux_reg(r); | 789 | read_aux_reg(r); |
| @@ -55,7 +55,7 @@ · 54 unmodified lines | |||
| 55 | <0x3ff00100 0x100>; | 55 | <0x3ff00100 0x100>; |
| 56 | }; | 56 | }; |
| 57 | 57 | ||
| 58 | smc@0x3404c000 { | 58 | smc@3404c000 { |
| 59 | compatible = "brcm,bcm11351-smc", "brcm,kona-smc"; | 59 | compatible = "brcm,bcm11351-smc", "brcm,kona-smc"; |
| 60 | reg = <0x3404c000 0x400>; /* 1 KiB in SRAM */ | 60 | reg = <0x3404c000 0x400>; /* 1 KiB in SRAM */ |
| 61 | }; | 61 | }; |
| @@ -55,7 +55,7 @@ · 54 unmodified lines | |||
| 55 | <0x3ff00100 0x100>; | 55 | <0x3ff00100 0x100>; |
| 56 | }; | 56 | }; |
| 57 | 57 | ||
| 58 | smc@0x3404e000 { | 58 | smc@3404e000 { |
| 59 | compatible = "brcm,bcm21664-smc", "brcm,kona-smc"; | 59 | compatible = "brcm,bcm21664-smc", "brcm,kona-smc"; |
| 60 | reg = <0x3404e000 0x400>; /* 1 KiB in SRAM */ | 60 | reg = <0x3404e000 0x400>; /* 1 KiB in SRAM */ |
| 61 | }; | 61 | }; |
| @@ -18,10 +18,10 @@ · 17 unmodified lines | |||
| 18 | soc { | 18 | soc { |
| 19 | ranges = <0x7e000000 0x20000000 0x02000000>; | 19 | ranges = <0x7e000000 0x20000000 0x02000000>; |
| 20 | dma-ranges = <0x40000000 0x00000000 0x20000000>; | 20 | dma-ranges = <0x40000000 0x00000000 0x20000000>; |
| 21 | }; | ||
| 21 | 22 | ||
| 22 | arm-pmu { | 23 | arm-pmu { |
| 23 | compatible = "arm,arm1176-pmu"; | 24 | compatible = "arm,arm1176-pmu"; |
| 24 | }; | ||
| 25 | }; | 25 | }; |
| 26 | }; | 26 | }; |
| 27 | 27 | ||
| @@ -9,19 +9,19 @@ · 8 unmodified lines | |||
| 9 | <0x40000000 0x40000000 0x00001000>; | 9 | <0x40000000 0x40000000 0x00001000>; |
| 10 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; | 10 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; |
| 11 | 11 | ||
| 12 | local_intc: local_intc { | 12 | local_intc: local_intc@40000000 { |
| 13 | compatible = "brcm,bcm2836-l1-intc"; | 13 | compatible = "brcm,bcm2836-l1-intc"; |
| 14 | reg = <0x40000000 0x100>; | 14 | reg = <0x40000000 0x100>; |
| 15 | interrupt-controller; | 15 | interrupt-controller; |
| 16 | #interrupt-cells = <2>; | 16 | #interrupt-cells = <2>; |
| 17 | interrupt-parent = <&local_intc>; | 17 | interrupt-parent = <&local_intc>; |
| 18 | }; | 18 | }; |
| 19 | }; | ||
| 19 | 20 | ||
| 20 | arm-pmu { | 21 | arm-pmu { |
| 21 | compatible = "arm,cortex-a7-pmu"; | 22 | compatible = "arm,cortex-a7-pmu"; |
| 22 | interrupt-parent = <&local_intc>; | 23 | interrupt-parent = <&local_intc>; |
| 23 | interrupts = <9 IRQ_TYPE_LEVEL_HIGH>; | 24 | interrupts = <9 IRQ_TYPE_LEVEL_HIGH>; |
| 24 | }; | ||
| 25 | }; | 25 | }; |
| 26 | 26 | ||
| 27 | timer { | 27 | timer { |
| @@ -8,7 +8,7 @@ · 7 unmodified lines | |||
| 8 | <0x40000000 0x40000000 0x00001000>; | 8 | <0x40000000 0x40000000 0x00001000>; |
| 9 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; | 9 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; |
| 10 | 10 | ||
| 11 | local_intc: local_intc { | 11 | local_intc: local_intc@40000000 { |
| 12 | compatible = "brcm,bcm2836-l1-intc"; | 12 | compatible = "brcm,bcm2836-l1-intc"; |
| 13 | reg = <0x40000000 0x100>; | 13 | reg = <0x40000000 0x100>; |
| 14 | interrupt-controller; | 14 | interrupt-controller; |
| @@ -465,7 +465,7 @@ · 464 unmodified lines | |||
| 465 | status = "disabled"; | 465 | status = "disabled"; |
| 466 | }; | 466 | }; |
| 467 | 467 | ||
| 468 | aux: aux@0x7e215000 { | 468 | aux: aux@7e215000 { |
| 469 | compatible = "brcm,bcm2835-aux"; | 469 | compatible = "brcm,bcm2835-aux"; |
| 470 | #clock-cells = <1>; | 470 | #clock-cells = <1>; |
| 471 | reg = <0x7e215000 0x8>; | 471 | reg = <0x7e215000 0x8>; |
| @@ -49,7 +49,7 @@ · 48 unmodified lines | |||
| 49 | 49 | ||
| 50 | memory { | 50 | memory { |
| 51 | device_type = "memory"; | 51 | device_type = "memory"; |
| 52 | reg = <0x60000000 0x80000000>; | 52 | reg = <0x60000000 0x20000000>; |
| 53 | }; | 53 | }; |
| 54 | 54 | ||
| 55 | gpio-restart { | 55 | gpio-restart { |
| @@ -269,7 +269,7 @@ · 268 unmodified lines | |||
| 269 | 269 | ||
| 270 | sata: sata@46000000 { | 270 | sata: sata@46000000 { |
| 271 | /* The ROM uses this muxmode */ | 271 | /* The ROM uses this muxmode */ |
| 272 | cortina,gemini-ata-muxmode = <3>; | 272 | cortina,gemini-ata-muxmode = <0>; |
| 273 | cortina,gemini-enable-sata-bridge; | 273 | cortina,gemini-enable-sata-bridge; |
| 274 | status = "okay"; | 274 | status = "okay"; |
| 275 | }; | 275 | }; |
| @@ -42,7 +42,7 @@ · 41 unmodified lines | |||
| 42 | 42 | ||
| 43 | /dts-v1/; | 43 | /dts-v1/; |
| 44 | 44 | ||
| 45 | #include "imx6q.dtsi" | 45 | #include "imx6dl.dtsi" |
| 46 | #include "imx6qdl-icore-rqs.dtsi" | 46 | #include "imx6qdl-icore-rqs.dtsi" |
| 47 | 47 | ||
| 48 | / { | 48 | / { |
| @@ -71,6 +71,8 @@ · 70 unmodified lines | |||
| 71 | }; | 71 | }; |
| 72 | 72 | ||
| 73 | &i2c1 { | 73 | &i2c1 { |
| 74 | pinctrl-names = "default"; | ||
| 75 | pinctrl-0 = <&i2c1_pins>; | ||
| 74 | clock-frequency = <2600000>; | 76 | clock-frequency = <2600000>; |
| 75 | 77 | ||
| 76 | twl: twl@48 { | 78 | twl: twl@48 { |
| @@ -189,7 +191,12 @@ · 112 unmodified lines | |||
| 189 | >; | 191 | >; |
| 190 | }; | 192 | }; |
| 191 | 193 | ||
| 192 | 194 | i2c1_pins: pinmux_i2c1_pins { | |
| 195 | pinctrl-single,pins = < | ||
| 196 | OMAP3_CORE1_IOPAD(0x21ba, PIN_INPUT | MUX_MODE0) /* i2c1_scl.i2c1_scl */ | ||
| 197 | OMAP3_CORE1_IOPAD(0x21bc, PIN_INPUT | MUX_MODE0) /* i2c1_sda.i2c1_sda */ | ||
| 198 | >; | ||
| 199 | }; | ||
| 193 | }; | 200 | }; |
| 194 | 201 | ||
| 195 | &omap3_pmx_wkup { | 202 | &omap3_pmx_wkup { |
| @@ -66,6 +66,8 @@ · 65 unmodified lines | |||
| 66 | }; | 66 | }; |
| 67 | 67 | ||
| 68 | &i2c1 { | 68 | &i2c1 { |
| 69 | pinctrl-names = "default"; | ||
| 70 | pinctrl-0 = <&i2c1_pins>; | ||
| 69 | clock-frequency = <2600000>; | 71 | clock-frequency = <2600000>; |
| 70 | 72 | ||
| 71 | twl: twl@48 { | 73 | twl: twl@48 { |
| @@ -136,6 +138,12 @@ · 64 unmodified lines | |||
| 136 | OMAP3_CORE1_IOPAD(0x21b8, PIN_INPUT | MUX_MODE0) /* hsusb0_data7.hsusb0_data7 */ | 138 | OMAP3_CORE1_IOPAD(0x21b8, PIN_INPUT | MUX_MODE0) /* hsusb0_data7.hsusb0_data7 */ |
| 137 | >; | 139 | >; |
| 138 | }; | 140 | }; |
| 141 | i2c1_pins: pinmux_i2c1_pins { | ||
| 142 | pinctrl-single,pins = < | ||
| 143 | OMAP3_CORE1_IOPAD(0x21ba, PIN_INPUT | MUX_MODE0) /* i2c1_scl.i2c1_scl */ | ||
| 144 | OMAP3_CORE1_IOPAD(0x21bc, PIN_INPUT | MUX_MODE0) /* i2c1_sda.i2c1_sda */ | ||
| 145 | >; | ||
| 146 | }; | ||
| 139 | }; | 147 | }; |
| 140 | 148 | ||
| 141 | &uart2 { | 149 | &uart2 { |
| @@ -47,7 +47,7 @@ · 46 unmodified lines | |||
| 47 | gpios = <&gpio3 19 GPIO_ACTIVE_LOW>; /* gpio3_83 */ | 47 | gpios = <&gpio3 19 GPIO_ACTIVE_LOW>; /* gpio3_83 */ |
| 48 | wakeup-source; | 48 | wakeup-source; |
| 49 | autorepeat; | 49 | autorepeat; |
| 50 | debounce_interval = <50>; | 50 | debounce-interval = <50>; |
| 51 | }; | 51 | }; |
| 52 | }; | 52 | }; |
| 53 | 53 | ||
| @@ -280,7 +280,7 @@ · 279 unmodified lines | |||
| 280 | max-frequency = <37500000>; | 280 | max-frequency = <37500000>; |
| 281 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, | 281 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, |
| 282 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; | 282 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; |
| 283 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 283 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 284 | fifo-depth = <0x100>; | 284 | fifo-depth = <0x100>; |
| 285 | interrupts = <GIC_SPI 15 IRQ_TYPE_LEVEL_HIGH>; | 285 | interrupts = <GIC_SPI 15 IRQ_TYPE_LEVEL_HIGH>; |
| 286 | resets = <&cru SRST_SDIO>; | 286 | resets = <&cru SRST_SDIO>; |
| @@ -298,7 +298,7 @@ · 11 unmodified lines | |||
| 298 | max-frequency = <37500000>; | 298 | max-frequency = <37500000>; |
| 299 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, | 299 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, |
| 300 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; | 300 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; |
| 301 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 301 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 302 | default-sample-phase = <158>; | 302 | default-sample-phase = <158>; |
| 303 | disable-wp; | 303 | disable-wp; |
| 304 | dmas = <&pdma 12>; | 304 | dmas = <&pdma 12>; |
| @@ -621,7 +621,7 @@ · 620 unmodified lines | |||
| 621 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; | 621 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; |
| 622 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, | 622 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, |
| 623 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; | 623 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; |
| 624 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 624 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 625 | fifo-depth = <0x100>; | 625 | fifo-depth = <0x100>; |
| 626 | pinctrl-names = "default"; | 626 | pinctrl-names = "default"; |
| 627 | pinctrl-0 = <&sdmmc_clk &sdmmc_cmd &sdmmc_bus4>; | 627 | pinctrl-0 = <&sdmmc_clk &sdmmc_cmd &sdmmc_bus4>; |
| @@ -634,7 +634,7 @@ · 6 unmodified lines | |||
| 634 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; | 634 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; |
| 635 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, | 635 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, |
| 636 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; | 636 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; |
| 637 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 637 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 638 | fifo-depth = <0x100>; | 638 | fifo-depth = <0x100>; |
| 639 | pinctrl-names = "default"; | 639 | pinctrl-names = "default"; |
| 640 | pinctrl-0 = <&sdio_clk &sdio_cmd &sdio_bus4>; | 640 | pinctrl-0 = <&sdio_clk &sdio_cmd &sdio_bus4>; |
| @@ -649,7 +649,7 @@ · 8 unmodified lines | |||
| 649 | max-frequency = <37500000>; | 649 | max-frequency = <37500000>; |
| 650 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, | 650 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, |
| 651 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; | 651 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; |
| 652 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 652 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 653 | bus-width = <8>; | 653 | bus-width = <8>; |
| 654 | default-sample-phase = <158>; | 654 | default-sample-phase = <158>; |
| 655 | fifo-depth = <0x100>; | 655 | fifo-depth = <0x100>; |
| @@ -110,26 +110,6 @@ · 109 unmodified lines | |||
| 110 | }; | 110 | }; |
| 111 | }; | 111 | }; |
| 112 | 112 | ||
| 113 | &cpu0 { | ||
| 114 | cpu0-supply = <&vdd_cpu>; | ||
| 115 | operating-points = < | ||
| 116 | /* KHz uV */ | ||
| 117 | 1800000 1400000 | ||
| 118 | 1608000 1350000 | ||
| 119 | 1512000 1300000 | ||
| 120 | 1416000 1200000 | ||
| 121 | 1200000 1100000 | ||
| 122 | 1008000 1050000 | ||
| 123 | 816000 1000000 | ||
| 124 | 696000 950000 | ||
| 125 | 600000 900000 | ||
| 126 | 408000 900000 | ||
| 127 | 312000 900000 | ||
| 128 | 216000 900000 | ||
| 129 | 126000 900000 | ||
| 130 | >; | ||
| 131 | }; | ||
| 132 | |||
| 133 | &emmc { | 113 | &emmc { |
| 134 | status = "okay"; | 114 | status = "okay"; |
| 135 | bus-width = <8>; | 115 | bus-width = <8>; |
| @@ -56,7 +56,7 @@ · 55 unmodified lines | |||
| 56 | clocks = <&topclk ZX296702_A9_PERIPHCLK>; | 56 | clocks = <&topclk ZX296702_A9_PERIPHCLK>; |
| 57 | }; | 57 | }; |
| 58 | 58 | ||
| 59 | l2cc: l2-cache-controller@0x00c00000 { | 59 | l2cc: l2-cache-controller@c00000 { |
| 60 | compatible = "arm,pl310-cache"; | 60 | compatible = "arm,pl310-cache"; |
| 61 | reg = <0x00c00000 0x1000>; | 61 | reg = <0x00c00000 0x1000>; |
| 62 | cache-unified; | 62 | cache-unified; |
| @@ -67,30 +67,30 @@ · 4 unmodified lines | |||
| 67 | arm,double-linefill-incr = <0>; | 67 | arm,double-linefill-incr = <0>; |
| 68 | }; | 68 | }; |
| 69 | 69 | ||
| 70 | pcu: pcu@0xa0008000 { | 70 | pcu: pcu@a0008000 { |
| 71 | compatible = "zte,zx296702-pcu"; | 71 | compatible = "zte,zx296702-pcu"; |
| 72 | reg = <0xa0008000 0x1000>; | 72 | reg = <0xa0008000 0x1000>; |
| 73 | }; | 73 | }; |
| 74 | 74 | ||
| 75 | topclk: topclk@0x09800000 { | 75 | topclk: topclk@9800000 { |
| 76 | compatible = "zte,zx296702-topcrm-clk"; | 76 | compatible = "zte,zx296702-topcrm-clk"; |
| 77 | reg = <0x09800000 0x1000>; | 77 | reg = <0x09800000 0x1000>; |
| 78 | #clock-cells = <1>; | 78 | #clock-cells = <1>; |
| 79 | }; | 79 | }; |
| 80 | 80 | ||
| 81 | lsp1clk: lsp1clk@0x09400000 { | 81 | lsp1clk: lsp1clk@9400000 { |
| 82 | compatible = "zte,zx296702-lsp1crpm-clk"; | 82 | compatible = "zte,zx296702-lsp1crpm-clk"; |
| 83 | reg = <0x09400000 0x1000>; | 83 | reg = <0x09400000 0x1000>; |
| 84 | #clock-cells = <1>; | 84 | #clock-cells = <1>; |
| 85 | }; | 85 | }; |
| 86 | 86 | ||
| 87 | lsp0clk: lsp0clk@0x0b000000 { | 87 | lsp0clk: lsp0clk@b000000 { |
| 88 | compatible = "zte,zx296702-lsp0crpm-clk"; | 88 | compatible = "zte,zx296702-lsp0crpm-clk"; |
| 89 | reg = <0x0b000000 0x1000>; | 89 | reg = <0x0b000000 0x1000>; |
| 90 | #clock-cells = <1>; | 90 | #clock-cells = <1>; |
| 91 | }; | 91 | }; |
| 92 | 92 | ||
| 93 | uart0: serial@0x09405000 { | 93 | uart0: serial@9405000 { |
| 94 | compatible = "zte,zx296702-uart"; | 94 | compatible = "zte,zx296702-uart"; |
| 95 | reg = <0x09405000 0x1000>; | 95 | reg = <0x09405000 0x1000>; |
| 96 | interrupts = <GIC_SPI 37 IRQ_TYPE_LEVEL_HIGH>; | 96 | interrupts = <GIC_SPI 37 IRQ_TYPE_LEVEL_HIGH>; |
| @@ -98,7 +98,7 @@ · 1 unmodified line | |||
| 98 | status = "disabled"; | 98 | status = "disabled"; |
| 99 | }; | 99 | }; |
| 100 | 100 | ||
| 101 | uart1: serial@0x09406000 { | 101 | uart1: serial@9406000 { |
| 102 | compatible = "zte,zx296702-uart"; | 102 | compatible = "zte,zx296702-uart"; |
| 103 | reg = <0x09406000 0x1000>; | 103 | reg = <0x09406000 0x1000>; |
| 104 | interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>; | 104 | interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>; |
| @@ -106,7 +106,7 @@ · 1 unmodified line | |||
| 106 | status = "disabled"; | 106 | status = "disabled"; |
| 107 | }; | 107 | }; |
| 108 | 108 | ||
| 109 | mmc0: mmc@0x09408000 { | 109 | mmc0: mmc@9408000 { |
| 110 | compatible = "snps,dw-mshc"; | 110 | compatible = "snps,dw-mshc"; |
| 111 | #address-cells = <1>; | 111 | #address-cells = <1>; |
| 112 | #size-cells = <0>; | 112 | #size-cells = <0>; |
| @@ -119,7 +119,7 @@ · 6 unmodified lines | |||
| 119 | status = "disabled"; | 119 | status = "disabled"; |
| 120 | }; | 120 | }; |
| 121 | 121 | ||
| 122 | mmc1: mmc@0x0b003000 { | 122 | mmc1: mmc@b003000 { |
| 123 | compatible = "snps,dw-mshc"; | 123 | compatible = "snps,dw-mshc"; |
| 124 | #address-cells = <1>; | 124 | #address-cells = <1>; |
| 125 | #size-cells = <0>; | 125 | #size-cells = <0>; |
| @@ -132,7 +132,7 @@ · 6 unmodified lines | |||
| 132 | status = "disabled"; | 132 | status = "disabled"; |
| 133 | }; | 133 | }; |
| 134 | 134 | ||
| 135 | sysctrl: sysctrl@0xa0007000 { | 135 | sysctrl: sysctrl@a0007000 { |
| 136 | compatible = "zte,sysctrl", "syscon"; | 136 | compatible = "zte,sysctrl", "syscon"; |
| 137 | reg = <0xa0007000 0x1000>; | 137 | reg = <0xa0007000 0x1000>; |
| 138 | }; | 138 | }; |
| @@ -319,7 +319,7 @@ CONFIG_MEDIA_CAMERA_SUPPORT=y · 318 unmodified lines | |||
| 319 | CONFIG_RC_CORE=m | 319 | CONFIG_RC_CORE=m |
| 320 | CONFIG_MEDIA_CONTROLLER=y | 320 | CONFIG_MEDIA_CONTROLLER=y |
| 321 | CONFIG_VIDEO_V4L2_SUBDEV_API=y | 321 | CONFIG_VIDEO_V4L2_SUBDEV_API=y |
| 322 | CONFIG_LIRC=m | 322 | CONFIG_LIRC=y |
| 323 | CONFIG_RC_DEVICES=y | 323 | CONFIG_RC_DEVICES=y |
| 324 | CONFIG_IR_RX51=m | 324 | CONFIG_IR_RX51=m |
| 325 | CONFIG_V4L_PLATFORM_DRIVERS=y | 325 | CONFIG_V4L_PLATFORM_DRIVERS=y |
| @@ -83,7 +83,7 @@ static void dummy_clock_access(struct timespec64 *ts) · 82 unmodified lines | |||
| 83 | } | 83 | } |
| 84 | 84 | ||
| 85 | static clock_access_fn __read_persistent_clock = dummy_clock_access; | 85 | static clock_access_fn __read_persistent_clock = dummy_clock_access; |
| 86 | static clock_access_fn __read_boot_clock = dummy_clock_access;; | 86 | static clock_access_fn __read_boot_clock = dummy_clock_access; |
| 87 | 87 | ||
| 88 | void read_persistent_clock64(struct timespec64 *ts) | 88 | void read_persistent_clock64(struct timespec64 *ts) |
| 89 | { | 89 | { |
| @@ -7,6 +7,8 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING · 6 unmodified lines | |||
| 7 | 7 | ||
| 8 | KVM=../../../../virt/kvm | 8 | KVM=../../../../virt/kvm |
| 9 | 9 | ||
| 10 | CFLAGS_ARMV7VE :=$(call cc-option, -march=armv7ve) | ||
| 11 | |||
| 10 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o | 12 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o |
| 11 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o | 13 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o |
| 12 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o | 14 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o |
| @@ -15,7 +17,10 @@ obj-$(CONFIG_KVM_ARM_HOST) += tlb.o · 2 unmodified lines | |||
| 15 | obj-$(CONFIG_KVM_ARM_HOST) += cp15-sr.o | 17 | obj-$(CONFIG_KVM_ARM_HOST) += cp15-sr.o |
| 16 | obj-$(CONFIG_KVM_ARM_HOST) += vfp.o | 18 | obj-$(CONFIG_KVM_ARM_HOST) += vfp.o |
| 17 | obj-$(CONFIG_KVM_ARM_HOST) += banked-sr.o | 19 | obj-$(CONFIG_KVM_ARM_HOST) += banked-sr.o |
| 20 | CFLAGS_banked-sr.o += $(CFLAGS_ARMV7VE) | ||
| 21 | |||
| 18 | obj-$(CONFIG_KVM_ARM_HOST) += entry.o | 22 | obj-$(CONFIG_KVM_ARM_HOST) += entry.o |
| 19 | obj-$(CONFIG_KVM_ARM_HOST) += hyp-entry.o | 23 | obj-$(CONFIG_KVM_ARM_HOST) += hyp-entry.o |
| 20 | obj-$(CONFIG_KVM_ARM_HOST) += switch.o | 24 | obj-$(CONFIG_KVM_ARM_HOST) += switch.o |
| 25 | CFLAGS_switch.o += $(CFLAGS_ARMV7VE) | ||
| 21 | obj-$(CONFIG_KVM_ARM_HOST) += s2-setup.o | 26 | obj-$(CONFIG_KVM_ARM_HOST) += s2-setup.o |
| @@ -20,6 +20,10 @@ · 19 unmodified lines | |||
| 20 | 20 | ||
| 21 | #include <asm/kvm_hyp.h> | 21 | #include <asm/kvm_hyp.h> |
| 22 | 22 | ||
| 23 | /* | ||
| 24 | * gcc before 4.9 doesn't understand -march=armv7ve, so we have to | ||
| 25 | * trick the assembler. | ||
| 26 | */ | ||
| 23 | __asm__(".arch_extension virt"); | 27 | __asm__(".arch_extension virt"); |
| 24 | 28 | ||
| 25 | void __hyp_text __banked_save_state(struct kvm_cpu_context *ctxt) | 29 | void __hyp_text __banked_save_state(struct kvm_cpu_context *ctxt) |
| @@ -69,7 +69,7 @@ static void clps711x_restart(enum reboot_mode mode, const char *cmd) · 68 unmodified lines | |||
| 69 | soft_restart(0); | 69 | soft_restart(0); |
| 70 | } | 70 | } |
| 71 | 71 | ||
| 72 | static const char *clps711x_compat[] __initconst = { | 72 | static const char *const clps711x_compat[] __initconst = { |
| 73 | "cirrus,ep7209", | 73 | "cirrus,ep7209", |
| 74 | NULL | 74 | NULL |
| 75 | }; | 75 | }; |
| @@ -368,7 +368,7 @@ static struct spi_eeprom at25640a = { · 367 unmodified lines | |||
| 368 | .flags = EE_ADDR2, | 368 | .flags = EE_ADDR2, |
| 369 | }; | 369 | }; |
| 370 | 370 | ||
| 371 | static struct spi_board_info dm355_evm_spi_info[] __initconst = { | 371 | static const struct spi_board_info dm355_evm_spi_info[] __initconst = { |
| 372 | { | 372 | { |
| 373 | .modalias = "at25", | 373 | .modalias = "at25", |
| 374 | .platform_data = &at25640a, | 374 | .platform_data = &at25640a, |
| @@ -217,7 +217,7 @@ static struct spi_eeprom at25640a = { · 216 unmodified lines | |||
| 217 | .flags = EE_ADDR2, | 217 | .flags = EE_ADDR2, |
| 218 | }; | 218 | }; |
| 219 | 219 | ||
| 220 | static struct spi_board_info dm355_leopard_spi_info[] __initconst = { | 220 | static const struct spi_board_info dm355_leopard_spi_info[] __initconst = { |
| 221 | { | 221 | { |
| 222 | .modalias = "at25", | 222 | .modalias = "at25", |
| 223 | .platform_data = &at25640a, | 223 | .platform_data = &at25640a, |
| @@ -726,7 +726,7 @@ static struct spi_eeprom at25640 = { · 725 unmodified lines | |||
| 726 | .flags = EE_ADDR2, | 726 | .flags = EE_ADDR2, |
| 727 | }; | 727 | }; |
| 728 | 728 | ||
| 729 | static struct spi_board_info dm365_evm_spi_info[] __initconst = { | 729 | static const struct spi_board_info dm365_evm_spi_info[] __initconst = { |
| 730 | { | 730 | { |
| 731 | .modalias = "at25", | 731 | .modalias = "at25", |
| 732 | .platform_data = &at25640, | 732 | .platform_data = &at25640, |
| @@ -41,7 +41,7 @@ config MACH_ARMADA_375 · 40 unmodified lines | |||
| 41 | depends on ARCH_MULTI_V7 | 41 | depends on ARCH_MULTI_V7 |
| 42 | select ARMADA_370_XP_IRQ | 42 | select ARMADA_370_XP_IRQ |
| 43 | select ARM_ERRATA_720789 | 43 | select ARM_ERRATA_720789 |
| 44 | select ARM_ERRATA_753970 | 44 | select PL310_ERRATA_753970 |
| 45 | select ARM_GIC | 45 | select ARM_GIC |
| 46 | select ARMADA_375_CLK | 46 | select ARMADA_375_CLK |
| 47 | select HAVE_ARM_SCU | 47 | select HAVE_ARM_SCU |
| @@ -57,7 +57,7 @@ config MACH_ARMADA_38X · 9 unmodified lines | |||
| 57 | bool "Marvell Armada 380/385 boards" | 57 | bool "Marvell Armada 380/385 boards" |
| 58 | depends on ARCH_MULTI_V7 | 58 | depends on ARCH_MULTI_V7 |
| 59 | select ARM_ERRATA_720789 | 59 | select ARM_ERRATA_720789 |
| 60 | select ARM_ERRATA_753970 | 60 | select PL310_ERRATA_753970 |
| 61 | select ARM_GIC | 61 | select ARM_GIC |
| 62 | select ARM_GLOBAL_TIMER | 62 | select ARM_GLOBAL_TIMER |
| 63 | select CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK | 63 | select CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK |
| @@ -1011,17 +1011,17 @@ static int clk_debugfs_register_one(struct clk *c) · 1010 unmodified lines | |||
| 1011 | return -ENOMEM; | 1011 | return -ENOMEM; |
| 1012 | c->dent = d; | 1012 | c->dent = d; |
| 1013 | 1013 | ||
| 1014 | d = debugfs_create_u8("usecount", S_IRUGO, c->dent, (u8 *)&c->usecount); | 1014 | d = debugfs_create_u8("usecount", S_IRUGO, c->dent, &c->usecount); |
| 1015 | if (!d) { | 1015 | if (!d) { |
| 1016 | err = -ENOMEM; | 1016 | err = -ENOMEM; |
| 1017 | goto err_out; | 1017 | goto err_out; |
| 1018 | } | 1018 | } |
| 1019 | d = debugfs_create_u32("rate", S_IRUGO, c->dent, (u32 *)&c->rate); | 1019 | d = debugfs_create_ulong("rate", S_IRUGO, c->dent, &c->rate); |
| 1020 | if (!d) { | 1020 | if (!d) { |
| 1021 | err = -ENOMEM; | 1021 | err = -ENOMEM; |
| 1022 | goto err_out; | 1022 | goto err_out; |
| 1023 | } | 1023 | } |
| 1024 | d = debugfs_create_x32("flags", S_IRUGO, c->dent, (u32 *)&c->flags); | 1024 | d = debugfs_create_x8("flags", S_IRUGO, c->dent, &c->flags); |
| 1025 | if (!d) { | 1025 | if (!d) { |
| 1026 | err = -ENOMEM; | 1026 | err = -ENOMEM; |
| 1027 | goto err_out; | 1027 | goto err_out; |
| @@ -299,8 +299,6 @@ static void irq_save_context(void) · 298 unmodified lines | |||
| 299 | if (soc_is_dra7xx()) | 299 | if (soc_is_dra7xx()) |
| 300 | return; | 300 | return; |
| 301 | 301 | ||
| 302 | if (!sar_base) | ||
| 303 | sar_base = omap4_get_sar_ram_base(); | ||
| 304 | if (wakeupgen_ops && wakeupgen_ops->save_context) | 302 | if (wakeupgen_ops && wakeupgen_ops->save_context) |
| 305 | wakeupgen_ops->save_context(); | 303 | wakeupgen_ops->save_context(); |
| 306 | } | 304 | } |
| @@ -598,6 +596,8 @@ static int __init wakeupgen_init(struct device_node *node, · 291 unmodified lines | |||
| 598 | irq_hotplug_init(); | 596 | irq_hotplug_init(); |
| 599 | irq_pm_init(); | 597 | irq_pm_init(); |
| 600 | 598 | ||
| 599 | sar_base = omap4_get_sar_ram_base(); | ||
| 600 | |||
| 601 | return 0; | 601 | return 0; |
| 602 | } | 602 | } |
| 603 | IRQCHIP_DECLARE(ti_wakeupgen, "ti,omap4-wugen-mpu", wakeupgen_init); | 603 | IRQCHIP_DECLARE(ti_wakeupgen, "ti,omap4-wugen-mpu", wakeupgen_init); |
| @@ -977,6 +977,9 @@ static int _enable_clocks(struct omap_hwmod *oh) · 976 unmodified lines | |||
| 977 | 977 | ||
| 978 | pr_debug("omap_hwmod: %s: enabling clocks\n", oh->name); | 978 | pr_debug("omap_hwmod: %s: enabling clocks\n", oh->name); |
| 979 | 979 | ||
| 980 | if (oh->flags & HWMOD_OPT_CLKS_NEEDED) | ||
| 981 | _enable_optional_clocks(oh); | ||
| 982 | |||
| 980 | if (oh->_clk) | 983 | if (oh->_clk) |
| 981 | clk_enable(oh->_clk); | 984 | clk_enable(oh->_clk); |
| 982 | 985 | ||
| @@ -985,9 +988,6 @@ static int _enable_clocks(struct omap_hwmod *oh) · 2 unmodified lines | |||
| 985 | clk_enable(os->_clk); | 988 | clk_enable(os->_clk); |
| 986 | } | 989 | } |
| 987 | 990 | ||
| 988 | if (oh->flags & HWMOD_OPT_CLKS_NEEDED) | ||
| 989 | _enable_optional_clocks(oh); | ||
| 990 | |||
| 991 | /* The opt clocks are controlled by the device driver. */ | 991 | /* The opt clocks are controlled by the device driver. */ |
| 992 | 992 | ||
| 993 | return 0; | 993 | return 0; |
| @@ -186,7 +186,7 @@ static void omap_pm_end(void) · 185 unmodified lines | |||
| 186 | cpu_idle_poll_ctrl(false); | 186 | cpu_idle_poll_ctrl(false); |
| 187 | } | 187 | } |
| 188 | 188 | ||
| 189 | static void omap_pm_finish(void) | 189 | static void omap_pm_wake(void) |
| 190 | { | 190 | { |
| 191 | if (soc_is_omap34xx()) | 191 | if (soc_is_omap34xx()) |
| 192 | omap_prcm_irq_complete(); | 192 | omap_prcm_irq_complete(); |
| @@ -196,7 +196,7 @@ static const struct platform_suspend_ops omap_pm_ops = { · 3 unmodified lines | |||
| 196 | .begin = omap_pm_begin, | 196 | .begin = omap_pm_begin, |
| 197 | .end = omap_pm_end, | 197 | .end = omap_pm_end, |
| 198 | .enter = omap_pm_enter, | 198 | .enter = omap_pm_enter, |
| 199 | .finish = omap_pm_finish, | 199 | .wake = omap_pm_wake, |
| 200 | .valid = suspend_valid_only_mem, | 200 | .valid = suspend_valid_only_mem, |
| 201 | }; | 201 | }; |
| 202 | 202 | ||
| @@ -156,12 +156,6 @@ static struct clock_event_device clockevent_gpt = { · 155 unmodified lines | |||
| 156 | .tick_resume = omap2_gp_timer_shutdown, | 156 | .tick_resume = omap2_gp_timer_shutdown, |
| 157 | }; | 157 | }; |
| 158 | 158 | ||
| 159 | static struct property device_disabled = { | ||
| 160 | .name = "status", | ||
| 161 | .length = sizeof("disabled"), | ||
| 162 | .value = "disabled", | ||
| 163 | }; | ||
| 164 | |||
| 165 | static const struct of_device_id omap_timer_match[] __initconst = { | 159 | static const struct of_device_id omap_timer_match[] __initconst = { |
| 166 | { .compatible = "ti,omap2420-timer", }, | 160 | { .compatible = "ti,omap2420-timer", }, |
| 167 | { .compatible = "ti,omap3430-timer", }, | 161 | { .compatible = "ti,omap3430-timer", }, |
| @@ -203,8 +197,17 @@ static struct device_node * __init omap_get_timer_dt(const struct of_device_id * · 35 unmodified lines | |||
| 203 | of_get_property(np, "ti,timer-secure", NULL))) | 197 | of_get_property(np, "ti,timer-secure", NULL))) |
| 204 | continue; | 198 | continue; |
| 205 | 199 | ||
| 206 | if (!of_device_is_compatible(np, "ti,omap-counter32k")) | 200 | if (!of_device_is_compatible(np, "ti,omap-counter32k")) { |
| 207 | of_add_property(np, &device_disabled); | 201 | struct property *prop; |
| 202 | |||
| 203 | prop = kzalloc(sizeof(*prop), GFP_KERNEL); | ||
| 204 | if (!prop) | ||
| 205 | return NULL; | ||
| 206 | prop->name = "status"; | ||
| 207 | prop->value = "disabled"; | ||
| 208 | prop->length = strlen(prop->value); | ||
| 209 | of_add_property(np, prop); | ||
| 210 | } | ||
| 208 | return np; | 211 | return np; |
| 209 | } | 212 | } |
| 210 | 213 | ||
| @@ -58,7 +58,6 @@ config MACH_KUROBOX_PRO · 57 unmodified lines | |||
| 58 | 58 | ||
| 59 | config MACH_DNS323 | 59 | config MACH_DNS323 |
| 60 | bool "D-Link DNS-323" | 60 | bool "D-Link DNS-323" |
| 61 | select GENERIC_NET_UTILS | ||
| 62 | select I2C_BOARDINFO if I2C | 61 | select I2C_BOARDINFO if I2C |
| 63 | help | 62 | help |
| 64 | Say 'Y' here if you want your kernel to support the | 63 | Say 'Y' here if you want your kernel to support the |
| @@ -66,7 +65,6 @@ config MACH_DNS323 · 1 unmodified line | |||
| 66 | 65 | ||
| 67 | config MACH_TS209 | 66 | config MACH_TS209 |
| 68 | bool "QNAP TS-109/TS-209" | 67 | bool "QNAP TS-109/TS-209" |
| 69 | select GENERIC_NET_UTILS | ||
| 70 | help | 68 | help |
| 71 | Say 'Y' here if you want your kernel to support the | 69 | Say 'Y' here if you want your kernel to support the |
| 72 | QNAP TS-109/TS-209 platform. | 70 | QNAP TS-109/TS-209 platform. |
| @@ -101,7 +99,6 @@ config MACH_LINKSTATION_LS_HGL · 28 unmodified lines | |||
| 101 | 99 | ||
| 102 | config MACH_TS409 | 100 | config MACH_TS409 |
| 103 | bool "QNAP TS-409" | 101 | bool "QNAP TS-409" |
| 104 | select GENERIC_NET_UTILS | ||
| 105 | help | 102 | help |
| 106 | Say 'Y' here if you want your kernel to support the | 103 | Say 'Y' here if you want your kernel to support the |
| 107 | QNAP TS-409 platform. | 104 | QNAP TS-409 platform. |
| @@ -173,10 +173,42 @@ static struct mv643xx_eth_platform_data dns323_eth_data = { · 172 unmodified lines | |||
| 173 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), | 173 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), |
| 174 | }; | 174 | }; |
| 175 | 175 | ||
| 176 | /* dns323_parse_hex_*() taken from tsx09-common.c; should a common copy of these | ||
| 177 | * functions be kept somewhere? | ||
| 178 | */ | ||
| 179 | static int __init dns323_parse_hex_nibble(char n) | ||
| 180 | { | ||
| 181 | if (n >= '0' && n <= '9') | ||
| 182 | return n - '0'; | ||
| 183 | |||
| 184 | if (n >= 'A' && n <= 'F') | ||
| 185 | return n - 'A' + 10; | ||
| 186 | |||
| 187 | if (n >= 'a' && n <= 'f') | ||
| 188 | return n - 'a' + 10; | ||
| 189 | |||
| 190 | return -1; | ||
| 191 | } | ||
| 192 | |||
| 193 | static int __init dns323_parse_hex_byte(const char *b) | ||
| 194 | { | ||
| 195 | int hi; | ||
| 196 | int lo; | ||
| 197 | |||
| 198 | hi = dns323_parse_hex_nibble(b[0]); | ||
| 199 | lo = dns323_parse_hex_nibble(b[1]); | ||
| 200 | |||
| 201 | if (hi < 0 || lo < 0) | ||
| 202 | return -1; | ||
| 203 | |||
| 204 | return (hi << 4) | lo; | ||
| 205 | } | ||
| 206 | |||
| 176 | static int __init dns323_read_mac_addr(void) | 207 | static int __init dns323_read_mac_addr(void) |
| 177 | { | 208 | { |
| 178 | u_int8_t addr[6]; | 209 | u_int8_t addr[6]; |
| 179 | void __iomem *mac_page; | 210 | int i; |
| 211 | char *mac_page; | ||
| 180 | 212 | ||
| 181 | /* MAC address is stored as a regular ol' string in /dev/mtdblock4 | 213 | /* MAC address is stored as a regular ol' string in /dev/mtdblock4 |
| 182 | * (0x007d0000-0x00800000) starting at offset 196480 (0x2ff80). | 214 | * (0x007d0000-0x00800000) starting at offset 196480 (0x2ff80). |
| @@ -185,8 +217,23 @@ static int __init dns323_read_mac_addr(void) · 2 unmodified lines | |||
| 185 | if (!mac_page) | 217 | if (!mac_page) |
| 186 | return -ENOMEM; | 218 | return -ENOMEM; |
| 187 | 219 | ||
| 188 | if (!mac_pton((__force const char *) mac_page, addr)) | 220 | /* Sanity check the string we're looking at */ |
| 189 | goto error_fail; | 221 | for (i = 0; i < 5; i++) { |
| 222 | if (*(mac_page + (i * 3) + 2) != ':') { | ||
| 223 | goto error_fail; | ||
| 224 | } | ||
| 225 | } | ||
| 226 | |||
| 227 | for (i = 0; i < 6; i++) { | ||
| 228 | int byte; | ||
| 229 | |||
| 230 | byte = dns323_parse_hex_byte(mac_page + (i * 3)); | ||
| 231 | if (byte < 0) { | ||
| 232 | goto error_fail; | ||
| 233 | } | ||
| 234 | |||
| 235 | addr[i] = byte; | ||
| 236 | } | ||
| 190 | 237 | ||
| 191 | iounmap(mac_page); | 238 | iounmap(mac_page); |
| 192 | printk("DNS-323: Found ethernet MAC address: %pM\n", addr); | 239 | printk("DNS-323: Found ethernet MAC address: %pM\n", addr); |
| @@ -53,12 +53,53 @@ struct mv643xx_eth_platform_data qnap_tsx09_eth_data = { · 52 unmodified lines | |||
| 53 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), | 53 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), |
| 54 | }; | 54 | }; |
| 55 | 55 | ||
| 56 | static int __init qnap_tsx09_parse_hex_nibble(char n) | ||
| 57 | { | ||
| 58 | if (n >= '0' && n <= '9') | ||
| 59 | return n - '0'; | ||
| 60 | |||
| 61 | if (n >= 'A' && n <= 'F') | ||
| 62 | return n - 'A' + 10; | ||
| 63 | |||
| 64 | if (n >= 'a' && n <= 'f') | ||
| 65 | return n - 'a' + 10; | ||
| 66 | |||
| 67 | return -1; | ||
| 68 | } | ||
| 69 | |||
| 70 | static int __init qnap_tsx09_parse_hex_byte(const char *b) | ||
| 71 | { | ||
| 72 | int hi; | ||
| 73 | int lo; | ||
| 74 | |||
| 75 | hi = qnap_tsx09_parse_hex_nibble(b[0]); | ||
| 76 | lo = qnap_tsx09_parse_hex_nibble(b[1]); | ||
| 77 | |||
| 78 | if (hi < 0 || lo < 0) | ||
| 79 | return -1; | ||
| 80 | |||
| 81 | return (hi << 4) | lo; | ||
| 82 | } | ||
| 83 | |||
| 56 | static int __init qnap_tsx09_check_mac_addr(const char *addr_str) | 84 | static int __init qnap_tsx09_check_mac_addr(const char *addr_str) |
| 57 | { | 85 | { |
| 58 | u_int8_t addr[6]; | 86 | u_int8_t addr[6]; |
| 87 | int i; | ||
| 59 | 88 | ||
| 60 | if (!mac_pton(addr_str, addr)) | 89 | for (i = 0; i < 6; i++) { |
| 61 | return -1; | 90 | int byte; |
| 91 | |||
| 92 | /* | ||
| 93 | * Enforce "xx:xx:xx:xx:xx:xx\n" format. | ||
| 94 | */ | ||
| 95 | if (addr_str[(i * 3) + 2] != ((i < 5) ? ':' : '\n')) | ||
| 96 | return -1; | ||
| 97 | |||
| 98 | byte = qnap_tsx09_parse_hex_byte(addr_str + (i * 3)); | ||
| 99 | if (byte < 0) | ||
| 100 | return -1; | ||
| 101 | addr[i] = byte; | ||
| 102 | } | ||
| 62 | 103 | ||
| 63 | printk(KERN_INFO "tsx09: found ethernet mac address %pM\n", addr); | 104 | printk(KERN_INFO "tsx09: found ethernet mac address %pM\n", addr); |
| 64 | 105 | ||
| @@ -77,12 +118,12 @@ void __init qnap_tsx09_find_mac_addr(u32 mem_base, u32 size) · 12 unmodified lines | |||
| 77 | unsigned long addr; | 118 | unsigned long addr; |
| 78 | 119 | ||
| 79 | for (addr = mem_base; addr < (mem_base + size); addr += 1024) { | 120 | for (addr = mem_base; addr < (mem_base + size); addr += 1024) { |
| 80 | void __iomem *nor_page; | 121 | char *nor_page; |
| 81 | int ret = 0; | 122 | int ret = 0; |
| 82 | 123 | ||
| 83 | nor_page = ioremap(addr, 1024); | 124 | nor_page = ioremap(addr, 1024); |
| 84 | if (nor_page != NULL) { | 125 | if (nor_page != NULL) { |
| 85 | ret = qnap_tsx09_check_mac_addr((__force const char *)nor_page); | 126 | ret = qnap_tsx09_check_mac_addr(nor_page); |
| 86 | iounmap(nor_page); | 127 | iounmap(nor_page); |
| 87 | } | 128 | } |
| 88 | 129 | ||
| @@ -23,7 +23,6 @@ · 22 unmodified lines | |||
| 23 | #include <linux/of.h> | 23 | #include <linux/of.h> |
| 24 | #include <linux/of_address.h> | 24 | #include <linux/of_address.h> |
| 25 | #include <linux/of_platform.h> | 25 | #include <linux/of_platform.h> |
| 26 | #include <linux/perf/arm_pmu.h> | ||
| 27 | #include <linux/regulator/machine.h> | 26 | #include <linux/regulator/machine.h> |
| 28 | 27 | ||
| 29 | #include <asm/outercache.h> | 28 | #include <asm/outercache.h> |
| @@ -112,37 +111,6 @@ static void ux500_restart(enum reboot_mode mode, const char *cmd) · 82 unmodified lines | |||
| 112 | prcmu_system_reset(0); | 111 | prcmu_system_reset(0); |
| 113 | } | 112 | } |
| 114 | 113 | ||
| 115 | /* | ||
| 116 | * The PMU IRQ lines of two cores are wired together into a single interrupt. | ||
| 117 | * Bounce the interrupt to the other core if it's not ours. | ||
| 118 | */ | ||
| 119 | static irqreturn_t db8500_pmu_handler(int irq, void *dev, irq_handler_t handler) | ||
| 120 | { | ||
| 121 | irqreturn_t ret = handler(irq, dev); | ||
| 122 | int other = !smp_processor_id(); | ||
| 123 | |||
| 124 | if (ret == IRQ_NONE && cpu_online(other)) | ||
| 125 | irq_set_affinity(irq, cpumask_of(other)); | ||
| 126 | |||
| 127 | /* | ||
| 128 | * We should be able to get away with the amount of IRQ_NONEs we give, | ||
| 129 | * while still having the spurious IRQ detection code kick in if the | ||
| 130 | * interrupt really starts hitting spuriously. | ||
| 131 | */ | ||
| 132 | return ret; | ||
| 133 | } | ||
| 134 | |||
| 135 | static struct arm_pmu_platdata db8500_pmu_platdata = { | ||
| 136 | .handle_irq = db8500_pmu_handler, | ||
| 137 | .irq_flags = IRQF_NOBALANCING | IRQF_NO_THREAD, | ||
| 138 | }; | ||
| 139 | |||
| 140 | static struct of_dev_auxdata u8500_auxdata_lookup[] __initdata = { | ||
| 141 | /* Requires call-back bindings. */ | ||
| 142 | OF_DEV_AUXDATA("arm,cortex-a9-pmu", 0, "arm-pmu", &db8500_pmu_platdata), | ||
| 143 | {}, | ||
| 144 | }; | ||
| 145 | |||
| 146 | static struct of_dev_auxdata u8540_auxdata_lookup[] __initdata = { | 114 | static struct of_dev_auxdata u8540_auxdata_lookup[] __initdata = { |
| 147 | OF_DEV_AUXDATA("stericsson,db8500-prcmu", 0x80157000, "db8500-prcmu", NULL), | 115 | OF_DEV_AUXDATA("stericsson,db8500-prcmu", 0x80157000, "db8500-prcmu", NULL), |
| 148 | {}, | 116 | {}, |
| @@ -165,9 +133,6 @@ static void __init u8500_init_machine(void) · 16 unmodified lines | |||
| 165 | if (of_machine_is_compatible("st-ericsson,u8540")) | 133 | if (of_machine_is_compatible("st-ericsson,u8540")) |
| 166 | of_platform_populate(NULL, u8500_local_bus_nodes, | 134 | of_platform_populate(NULL, u8500_local_bus_nodes, |
| 167 | u8540_auxdata_lookup, NULL); | 135 | u8540_auxdata_lookup, NULL); |
| 168 | else | ||
| 169 | of_platform_populate(NULL, u8500_local_bus_nodes, | ||
| 170 | u8500_auxdata_lookup, NULL); | ||
| 171 | } | 136 | } |
| 172 | 137 | ||
| 173 | static const char * stericsson_dt_platform_compat[] = { | 138 | static const char * stericsson_dt_platform_compat[] = { |
| @@ -472,28 +472,27 @@ void __init orion_ge11_init(struct mv643xx_eth_platform_data *eth_data, · 471 unmodified lines | |||
| 472 | /***************************************************************************** | 472 | /***************************************************************************** |
| 473 | * Ethernet switch | 473 | * Ethernet switch |
| 474 | ****************************************************************************/ | 474 | ****************************************************************************/ |
| 475 | static __initconst const char *orion_ge00_mvmdio_bus_name = "orion-mii"; | 475 | static __initdata struct mdio_board_info orion_ge00_switch_board_info = { |
| 476 | static __initdata struct mdio_board_info | 476 | .bus_id = "orion-mii", |
| 477 | orion_ge00_switch_board_info; | 477 | .modalias = "mv88e6085", |
| 478 | }; | ||
| 478 | 479 | ||
| 479 | void __init orion_ge00_switch_init(struct dsa_chip_data *d) | 480 | void __init orion_ge00_switch_init(struct dsa_chip_data *d) |
| 480 | { | 481 | { |
| 481 | struct mdio_board_info *bd; | ||
| 482 | unsigned int i; | 482 | unsigned int i; |
| 483 | 483 | ||
| 484 | if (!IS_BUILTIN(CONFIG_PHYLIB)) | 484 | if (!IS_BUILTIN(CONFIG_PHYLIB)) |
| 485 | return; | 485 | return; |
| 486 | 486 | ||
| 487 | for (i = 0; i < ARRAY_SIZE(d->port_names); i++) | 487 | for (i = 0; i < ARRAY_SIZE(d->port_names); i++) { |
| 488 | if (!strcmp(d->port_names[i], "cpu")) | 488 | if (!strcmp(d->port_names[i], "cpu")) { |
| 489 | d->netdev[i] = &orion_ge00.dev; | ||
| 489 | break; | 490 | break; |
| 491 | } | ||
| 492 | } | ||
| 490 | 493 | ||
| 491 | bd = &orion_ge00_switch_board_info; | 494 | orion_ge00_switch_board_info.mdio_addr = d->sw_addr; |
| 492 | bd->bus_id = orion_ge00_mvmdio_bus_name; | 495 | orion_ge00_switch_board_info.platform_data = d; |
| 493 | bd->mdio_addr = d->sw_addr; | ||
| 494 | d->netdev[i] = &orion_ge00.dev; | ||
| 495 | strcpy(bd->modalias, "mv88e6085"); | ||
| 496 | bd->platform_data = d; | ||
| 497 | 496 | ||
| 498 | mdiobus_register_board_info(&orion_ge00_switch_board_info, 1); | 497 | mdiobus_register_board_info(&orion_ge00_switch_board_info, 1); |
| 499 | } | 498 | } |
| @@ -165,14 +165,14 @@ · 164 unmodified lines | |||
| 165 | 165 | ||
| 166 | uart_A: serial@24000 { | 166 | uart_A: serial@24000 { |
| 167 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; | 167 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; |
| 168 | reg = <0x0 0x24000 0x0 0x14>; | 168 | reg = <0x0 0x24000 0x0 0x18>; |
| 169 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; | 169 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; |
| 170 | status = "disabled"; | 170 | status = "disabled"; |
| 171 | }; | 171 | }; |
| 172 | 172 | ||
| 173 | uart_B: serial@23000 { | 173 | uart_B: serial@23000 { |
| 174 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; | 174 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; |
| 175 | reg = <0x0 0x23000 0x0 0x14>; | 175 | reg = <0x0 0x23000 0x0 0x18>; |
| 176 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; | 176 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; |
| 177 | status = "disabled"; | 177 | status = "disabled"; |
| 178 | }; | 178 | }; |
| @@ -235,14 +235,14 @@ · 234 unmodified lines | |||
| 235 | 235 | ||
| 236 | uart_A: serial@84c0 { | 236 | uart_A: serial@84c0 { |
| 237 | compatible = "amlogic,meson-gx-uart"; | 237 | compatible = "amlogic,meson-gx-uart"; |
| 238 | reg = <0x0 0x84c0 0x0 0x14>; | 238 | reg = <0x0 0x84c0 0x0 0x18>; |
| 239 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; | 239 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; |
| 240 | status = "disabled"; | 240 | status = "disabled"; |
| 241 | }; | 241 | }; |
| 242 | 242 | ||
| 243 | uart_B: serial@84dc { | 243 | uart_B: serial@84dc { |
| 244 | compatible = "amlogic,meson-gx-uart"; | 244 | compatible = "amlogic,meson-gx-uart"; |
| 245 | reg = <0x0 0x84dc 0x0 0x14>; | 245 | reg = <0x0 0x84dc 0x0 0x18>; |
| 246 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; | 246 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; |
| 247 | status = "disabled"; | 247 | status = "disabled"; |
| 248 | }; | 248 | }; |
| @@ -287,7 +287,7 @@ · 38 unmodified lines | |||
| 287 | 287 | ||
| 288 | uart_C: serial@8700 { | 288 | uart_C: serial@8700 { |
| 289 | compatible = "amlogic,meson-gx-uart"; | 289 | compatible = "amlogic,meson-gx-uart"; |
| 290 | reg = <0x0 0x8700 0x0 0x14>; | 290 | reg = <0x0 0x8700 0x0 0x18>; |
| 291 | interrupts = <GIC_SPI 93 IRQ_TYPE_EDGE_RISING>; | 291 | interrupts = <GIC_SPI 93 IRQ_TYPE_EDGE_RISING>; |
| 292 | status = "disabled"; | 292 | status = "disabled"; |
| 293 | }; | 293 | }; |
| @@ -404,14 +404,14 @@ · 110 unmodified lines | |||
| 404 | 404 | ||
| 405 | uart_AO: serial@4c0 { | 405 | uart_AO: serial@4c0 { |
| 406 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; | 406 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; |
| 407 | reg = <0x0 0x004c0 0x0 0x14>; | 407 | reg = <0x0 0x004c0 0x0 0x18>; |
| 408 | interrupts = <GIC_SPI 193 IRQ_TYPE_EDGE_RISING>; | 408 | interrupts = <GIC_SPI 193 IRQ_TYPE_EDGE_RISING>; |
| 409 | status = "disabled"; | 409 | status = "disabled"; |
| 410 | }; | 410 | }; |
| 411 | 411 | ||
| 412 | uart_AO_B: serial@4e0 { | 412 | uart_AO_B: serial@4e0 { |
| 413 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; | 413 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; |
| 414 | reg = <0x0 0x004e0 0x0 0x14>; | 414 | reg = <0x0 0x004e0 0x0 0x18>; |
| 415 | interrupts = <GIC_SPI 197 IRQ_TYPE_EDGE_RISING>; | 415 | interrupts = <GIC_SPI 197 IRQ_TYPE_EDGE_RISING>; |
| 416 | status = "disabled"; | 416 | status = "disabled"; |
| 417 | }; | 417 | }; |
| @@ -631,6 +631,7 @@ · 630 unmodified lines | |||
| 631 | 631 | ||
| 632 | internal_phy: ethernet-phy@8 { | 632 | internal_phy: ethernet-phy@8 { |
| 633 | compatible = "ethernet-phy-id0181.4400", "ethernet-phy-ieee802.3-c22"; | 633 | compatible = "ethernet-phy-id0181.4400", "ethernet-phy-ieee802.3-c22"; |
| 634 | interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH>; | ||
| 634 | reg = <8>; | 635 | reg = <8>; |
| 635 | max-speed = <100>; | 636 | max-speed = <100>; |
| 636 | }; | 637 | }; |
| @@ -98,7 +98,7 @@ · 97 unmodified lines | |||
| 98 | clock-output-names = "clk125mhz"; | 98 | clock-output-names = "clk125mhz"; |
| 99 | }; | 99 | }; |
| 100 | 100 | ||
| 101 | pci { | 101 | pcie@30000000 { |
| 102 | compatible = "pci-host-ecam-generic"; | 102 | compatible = "pci-host-ecam-generic"; |
| 103 | device_type = "pci"; | 103 | device_type = "pci"; |
| 104 | #interrupt-cells = <1>; | 104 | #interrupt-cells = <1>; |
| @@ -118,6 +118,7 @@ · 13 unmodified lines | |||
| 118 | ranges = | 118 | ranges = |
| 119 | <0x02000000 0 0x40000000 0 0x40000000 0 0x20000000 | 119 | <0x02000000 0 0x40000000 0 0x40000000 0 0x20000000 |
| 120 | 0x43000000 0x40 0x00000000 0x40 0x00000000 0x20 0x00000000>; | 120 | 0x43000000 0x40 0x00000000 0x40 0x00000000 0x20 0x00000000>; |
| 121 | bus-range = <0 0xff>; | ||
| 121 | interrupt-map-mask = <0 0 0 7>; | 122 | interrupt-map-mask = <0 0 0 7>; |
| 122 | interrupt-map = | 123 | interrupt-map = |
| 123 | /* addr pin ic icaddr icintr */ | 124 | /* addr pin ic icaddr icintr */ |
| @@ -51,7 +51,7 @@ · 50 unmodified lines | |||
| 51 | #size-cells = <2>; | 51 | #size-cells = <2>; |
| 52 | ranges; | 52 | ranges; |
| 53 | 53 | ||
| 54 | ramoops@0x21f00000 { | 54 | ramoops@21f00000 { |
| 55 | compatible = "ramoops"; | 55 | compatible = "ramoops"; |
| 56 | reg = <0x0 0x21f00000 0x0 0x00100000>; | 56 | reg = <0x0 0x21f00000 0x0 0x00100000>; |
| 57 | record-size = <0x00020000>; | 57 | record-size = <0x00020000>; |
| @@ -341,7 +341,7 @@ · 340 unmodified lines | |||
| 341 | reg = <0 0x10005000 0 0x1000>; | 341 | reg = <0 0x10005000 0 0x1000>; |
| 342 | }; | 342 | }; |
| 343 | 343 | ||
| 344 | pio: pinctrl@0x10005000 { | 344 | pio: pinctrl@10005000 { |
| 345 | compatible = "mediatek,mt8173-pinctrl"; | 345 | compatible = "mediatek,mt8173-pinctrl"; |
| 346 | reg = <0 0x1000b000 0 0x1000>; | 346 | reg = <0 0x1000b000 0 0x1000>; |
| 347 | mediatek,pctl-regmap = <&syscfg_pctl_a>; | 347 | mediatek,pctl-regmap = <&syscfg_pctl_a>; |
| @@ -140,16 +140,16 @@ · 139 unmodified lines | |||
| 140 | }; | 140 | }; |
| 141 | 141 | ||
| 142 | agnoc@0 { | 142 | agnoc@0 { |
| 143 | qcom,pcie@00600000 { | 143 | qcom,pcie@600000 { |
| 144 | perst-gpio = <&msmgpio 35 GPIO_ACTIVE_LOW>; | 144 | perst-gpio = <&msmgpio 35 GPIO_ACTIVE_LOW>; |
| 145 | }; | 145 | }; |
| 146 | 146 | ||
| 147 | qcom,pcie@00608000 { | 147 | qcom,pcie@608000 { |
| 148 | status = "okay"; | 148 | status = "okay"; |
| 149 | perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; | 149 | perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; |
| 150 | }; | 150 | }; |
| 151 | 151 | ||
| 152 | qcom,pcie@00610000 { | 152 | qcom,pcie@610000 { |
| 153 | status = "okay"; | 153 | status = "okay"; |
| 154 | perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; | 154 | perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; |
| 155 | }; | 155 | }; |
| @@ -840,7 +840,7 @@ · 839 unmodified lines | |||
| 840 | #size-cells = <1>; | 840 | #size-cells = <1>; |
| 841 | ranges; | 841 | ranges; |
| 842 | 842 | ||
| 843 | pcie0: qcom,pcie@00600000 { | 843 | pcie0: qcom,pcie@600000 { |
| 844 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; | 844 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; |
| 845 | status = "disabled"; | 845 | status = "disabled"; |
| 846 | power-domains = <&gcc PCIE0_GDSC>; | 846 | power-domains = <&gcc PCIE0_GDSC>; |
| @@ -893,7 +893,7 @@ · 46 unmodified lines | |||
| 893 | 893 | ||
| 894 | }; | 894 | }; |
| 895 | 895 | ||
| 896 | pcie1: qcom,pcie@00608000 { | 896 | pcie1: qcom,pcie@608000 { |
| 897 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; | 897 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; |
| 898 | power-domains = <&gcc PCIE1_GDSC>; | 898 | power-domains = <&gcc PCIE1_GDSC>; |
| 899 | bus-range = <0x00 0xff>; | 899 | bus-range = <0x00 0xff>; |
| @@ -946,7 +946,7 @@ · 46 unmodified lines | |||
| 946 | "bus_slave"; | 946 | "bus_slave"; |
| 947 | }; | 947 | }; |
| 948 | 948 | ||
| 949 | pcie2: qcom,pcie@00610000 { | 949 | pcie2: qcom,pcie@610000 { |
| 950 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; | 950 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; |
| 951 | power-domains = <&gcc PCIE2_GDSC>; | 951 | power-domains = <&gcc PCIE2_GDSC>; |
| 952 | bus-range = <0x00 0xff>; | 952 | bus-range = <0x00 0xff>; |
| @@ -132,17 +132,16 @@ · 131 unmodified lines | |||
| 132 | assigned-clocks = <&cru SCLK_MAC2IO>, <&cru SCLK_MAC2IO_EXT>; | 132 | assigned-clocks = <&cru SCLK_MAC2IO>, <&cru SCLK_MAC2IO_EXT>; |
| 133 | assigned-clock-parents = <&gmac_clkin>, <&gmac_clkin>; | 133 | assigned-clock-parents = <&gmac_clkin>, <&gmac_clkin>; |
| 134 | clock_in_out = "input"; | 134 | clock_in_out = "input"; |
| 135 | /* shows instability at 1GBit right now */ | ||
| 136 | max-speed = <100>; | ||
| 137 | phy-supply = <&vcc_io>; | 135 | phy-supply = <&vcc_io>; |
| 138 | phy-mode = "rgmii"; | 136 | phy-mode = "rgmii"; |
| 139 | pinctrl-names = "default"; | 137 | pinctrl-names = "default"; |
| 140 | pinctrl-0 = <&rgmiim1_pins>; | 138 | pinctrl-0 = <&rgmiim1_pins>; |
| 139 | snps,force_thresh_dma_mode; | ||
| 141 | snps,reset-gpio = <&gpio1 RK_PC2 GPIO_ACTIVE_LOW>; | 140 | snps,reset-gpio = <&gpio1 RK_PC2 GPIO_ACTIVE_LOW>; |
| 142 | snps,reset-active-low; | 141 | snps,reset-active-low; |
| 143 | snps,reset-delays-us = <0 10000 50000>; | 142 | snps,reset-delays-us = <0 10000 50000>; |
| 144 | tx_delay = <0x26>; | 143 | tx_delay = <0x24>; |
| 145 | rx_delay = <0x11>; | 144 | rx_delay = <0x18>; |
| 146 | status = "okay"; | 145 | status = "okay"; |
| 147 | }; | 146 | }; |
| 148 | 147 | ||
| @@ -730,7 +730,7 @@ · 729 unmodified lines | |||
| 730 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; | 730 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; |
| 731 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, | 731 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, |
| 732 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; | 732 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; |
| 733 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 733 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 734 | fifo-depth = <0x100>; | 734 | fifo-depth = <0x100>; |
| 735 | status = "disabled"; | 735 | status = "disabled"; |
| 736 | }; | 736 | }; |
| @@ -741,7 +741,7 @@ · 4 unmodified lines | |||
| 741 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; | 741 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; |
| 742 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, | 742 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, |
| 743 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; | 743 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; |
| 744 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 744 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 745 | fifo-depth = <0x100>; | 745 | fifo-depth = <0x100>; |
| 746 | status = "disabled"; | 746 | status = "disabled"; |
| 747 | }; | 747 | }; |
| @@ -752,7 +752,7 @@ · 4 unmodified lines | |||
| 752 | interrupts = <GIC_SPI 14 IRQ_TYPE_LEVEL_HIGH>; | 752 | interrupts = <GIC_SPI 14 IRQ_TYPE_LEVEL_HIGH>; |
| 753 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, | 753 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, |
| 754 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; | 754 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; |
| 755 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 755 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 756 | fifo-depth = <0x100>; | 756 | fifo-depth = <0x100>; |
| 757 | status = "disabled"; | 757 | status = "disabled"; |
| 758 | }; | 758 | }; |
| @@ -257,7 +257,7 @@ · 256 unmodified lines | |||
| 257 | max-frequency = <150000000>; | 257 | max-frequency = <150000000>; |
| 258 | clocks = <&cru HCLK_SDIO0>, <&cru SCLK_SDIO0>, | 258 | clocks = <&cru HCLK_SDIO0>, <&cru SCLK_SDIO0>, |
| 259 | <&cru SCLK_SDIO0_DRV>, <&cru SCLK_SDIO0_SAMPLE>; | 259 | <&cru SCLK_SDIO0_DRV>, <&cru SCLK_SDIO0_SAMPLE>; |
| 260 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 260 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 261 | fifo-depth = <0x100>; | 261 | fifo-depth = <0x100>; |
| 262 | interrupts = <GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>; | 262 | interrupts = <GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>; |
| 263 | resets = <&cru SRST_SDIO0>; | 263 | resets = <&cru SRST_SDIO0>; |
| @@ -457,7 +457,7 @@ · 456 unmodified lines | |||
| 457 | assigned-clocks = <&cru SCLK_PCIEPHY_REF>; | 457 | assigned-clocks = <&cru SCLK_PCIEPHY_REF>; |
| 458 | assigned-clock-parents = <&cru SCLK_PCIEPHY_REF100M>; | 458 | assigned-clock-parents = <&cru SCLK_PCIEPHY_REF100M>; |
| 459 | assigned-clock-rates = <100000000>; | 459 | assigned-clock-rates = <100000000>; |
| 460 | ep-gpios = <&gpio3 RK_PB5 GPIO_ACTIVE_HIGH>; | 460 | ep-gpios = <&gpio2 RK_PA4 GPIO_ACTIVE_HIGH>; |
| 461 | num-lanes = <4>; | 461 | num-lanes = <4>; |
| 462 | pinctrl-names = "default"; | 462 | pinctrl-names = "default"; |
| 463 | pinctrl-0 = <&pcie_clkreqn_cpm>; | 463 | pinctrl-0 = <&pcie_clkreqn_cpm>; |
| @@ -1739,8 +1739,8 @@ · 1738 unmodified lines | |||
| 1739 | compatible = "rockchip,rk3399-edp"; | 1739 | compatible = "rockchip,rk3399-edp"; |
| 1740 | reg = <0x0 0xff970000 0x0 0x8000>; | 1740 | reg = <0x0 0xff970000 0x0 0x8000>; |
| 1741 | interrupts = <GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH 0>; | 1741 | interrupts = <GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH 0>; |
| 1742 | clocks = <&cru PCLK_EDP>, <&cru PCLK_EDP_CTRL>; | 1742 | clocks = <&cru PCLK_EDP>, <&cru PCLK_EDP_CTRL>, <&cru PCLK_VIO_GRF>; |
| 1743 | clock-names = "dp", "pclk"; | 1743 | clock-names = "dp", "pclk", "grf"; |
| 1744 | pinctrl-names = "default"; | 1744 | pinctrl-names = "default"; |
| 1745 | pinctrl-0 = <&edp_hpd>; | 1745 | pinctrl-0 = <&edp_hpd>; |
| 1746 | power-domains = <&power RK3399_PD_EDP>; | 1746 | power-domains = <&power RK3399_PD_EDP>; |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | |||
| 20 | 20 | ||
| 21 | #define MPIDR_UP_BITMASK (0x1 << 30) | 21 | #define MPIDR_UP_BITMASK (0x1 << 30) |
| 22 | #define MPIDR_MT_BITMASK (0x1 << 24) | 22 | #define MPIDR_MT_BITMASK (0x1 << 24) |
| 23 | #define MPIDR_HWID_BITMASK 0xff00ffffff | 23 | #define MPIDR_HWID_BITMASK UL(0xff00ffffff) |
| 24 | 24 | ||
| 25 | #define MPIDR_LEVEL_BITS_SHIFT 3 | 25 | #define MPIDR_LEVEL_BITS_SHIFT 3 |
| 26 | #define MPIDR_LEVEL_BITS (1 << MPIDR_LEVEL_BITS_SHIFT) | 26 | #define MPIDR_LEVEL_BITS (1 << MPIDR_LEVEL_BITS_SHIFT) |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | |||
| 22 | 22 | ||
| 23 | static inline pte_t huge_ptep_get(pte_t *ptep) | 23 | static inline pte_t huge_ptep_get(pte_t *ptep) |
| 24 | { | 24 | { |
| 25 | return *ptep; | 25 | return READ_ONCE(*ptep); |
| 26 | } | 26 | } |
| 27 | 27 | ||
| 28 | 28 | ||
| @@ -185,42 +185,42 @@ static inline pmd_t kvm_s2pmd_mkexec(pmd_t pmd) · 184 unmodified lines | |||
| 185 | return pmd; | 185 | return pmd; |
| 186 | } | 186 | } |
| 187 | 187 | ||
| 188 | static inline void kvm_set_s2pte_readonly(pte_t *pte) | 188 | static inline void kvm_set_s2pte_readonly(pte_t *ptep) |
| 189 | { | 189 | { |
| 190 | pteval_t old_pteval, pteval; | 190 | pteval_t old_pteval, pteval; |
| 191 | 191 | ||
| 192 | pteval = READ_ONCE(pte_val(*pte)); | 192 | pteval = READ_ONCE(pte_val(*ptep)); |
| 193 | do { | 193 | do { |
| 194 | old_pteval = pteval; | 194 | old_pteval = pteval; |
| 195 | pteval &= ~PTE_S2_RDWR; | 195 | pteval &= ~PTE_S2_RDWR; |
| 196 | pteval |= PTE_S2_RDONLY; | 196 | pteval |= PTE_S2_RDONLY; |
| 197 | pteval = cmpxchg_relaxed(&pte_val(*pte), old_pteval, pteval); | 197 | pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval); |
| 198 | } while (pteval != old_pteval); | 198 | } while (pteval != old_pteval); |
| 199 | } | 199 | } |
| 200 | 200 | ||
| 201 | static inline bool kvm_s2pte_readonly(pte_t *pte) | 201 | static inline bool kvm_s2pte_readonly(pte_t *ptep) |
| 202 | { | 202 | { |
| 203 | return (pte_val(*pte) & PTE_S2_RDWR) == PTE_S2_RDONLY; | 203 | return (READ_ONCE(pte_val(*ptep)) & PTE_S2_RDWR) == PTE_S2_RDONLY; |
| 204 | } | 204 | } |
| 205 | 205 | ||
| 206 | static inline bool kvm_s2pte_exec(pte_t *pte) | 206 | static inline bool kvm_s2pte_exec(pte_t *ptep) |
| 207 | { | 207 | { |
| 208 | return !(pte_val(*pte) & PTE_S2_XN); | 208 | return !(READ_ONCE(pte_val(*ptep)) & PTE_S2_XN); |
| 209 | } | 209 | } |
| 210 | 210 | ||
| 211 | static inline void kvm_set_s2pmd_readonly(pmd_t *pmd) | 211 | static inline void kvm_set_s2pmd_readonly(pmd_t *pmdp) |
| 212 | { | 212 | { |
| 213 | kvm_set_s2pte_readonly((pte_t *)pmd); | 213 | kvm_set_s2pte_readonly((pte_t *)pmdp); |
| 214 | } | 214 | } |
| 215 | 215 | ||
| 216 | static inline bool kvm_s2pmd_readonly(pmd_t *pmd) | 216 | static inline bool kvm_s2pmd_readonly(pmd_t *pmdp) |
| 217 | { | 217 | { |
| 218 | return kvm_s2pte_readonly((pte_t *)pmd); | 218 | return kvm_s2pte_readonly((pte_t *)pmdp); |
| 219 | } | 219 | } |
| 220 | 220 | ||
| 221 | static inline bool kvm_s2pmd_exec(pmd_t *pmd) | 221 | static inline bool kvm_s2pmd_exec(pmd_t *pmdp) |
| 222 | { | 222 | { |
| 223 | return !(pmd_val(*pmd) & PMD_S2_XN); | 223 | return !(READ_ONCE(pmd_val(*pmdp)) & PMD_S2_XN); |
| 224 | } | 224 | } |
| 225 | 225 | ||
| 226 | static inline bool kvm_page_empty(void *ptr) | 226 | static inline bool kvm_page_empty(void *ptr) |
| @@ -141,13 +141,13 @@ static inline void cpu_install_idmap(void) · 140 unmodified lines | |||
| 141 | * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, | 141 | * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, |
| 142 | * avoiding the possibility of conflicting TLB entries being allocated. | 142 | * avoiding the possibility of conflicting TLB entries being allocated. |
| 143 | */ | 143 | */ |
| 144 | static inline void cpu_replace_ttbr1(pgd_t *pgd) | 144 | static inline void cpu_replace_ttbr1(pgd_t *pgdp) |
| 145 | { | 145 | { |
| 146 | typedef void (ttbr_replace_func)(phys_addr_t); | 146 | typedef void (ttbr_replace_func)(phys_addr_t); |
| 147 | extern ttbr_replace_func idmap_cpu_replace_ttbr1; | 147 | extern ttbr_replace_func idmap_cpu_replace_ttbr1; |
| 148 | ttbr_replace_func *replace_phys; | 148 | ttbr_replace_func *replace_phys; |
| 149 | 149 | ||
| 150 | phys_addr_t pgd_phys = virt_to_phys(pgd); | 150 | phys_addr_t pgd_phys = virt_to_phys(pgdp); |
| 151 | 151 | ||
| 152 | replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); | 152 | replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); |
| 153 | 153 | ||
| @@ -36,23 +36,23 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) · 35 unmodified lines | |||
| 36 | return (pmd_t *)__get_free_page(PGALLOC_GFP); | 36 | return (pmd_t *)__get_free_page(PGALLOC_GFP); |
| 37 | } | 37 | } |
| 38 | 38 | ||
| 39 | static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) | 39 | static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) |
| 40 | { | 40 | { |
| 41 | BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); | 41 | BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); |
| 42 | free_page((unsigned long)pmd); | 42 | free_page((unsigned long)pmdp); |
| 43 | } | 43 | } |
| 44 | 44 | ||
| 45 | static inline void __pud_populate(pud_t *pud, phys_addr_t pmd, pudval_t prot) | 45 | static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) |
| 46 | { | 46 | { |
| 47 | set_pud(pud, __pud(__phys_to_pud_val(pmd) | prot)); | 47 | set_pud(pudp, __pud(__phys_to_pud_val(pmdp) | prot)); |
| 48 | } | 48 | } |
| 49 | 49 | ||
| 50 | static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) | 50 | static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) |
| 51 | { | 51 | { |
| 52 | __pud_populate(pud, __pa(pmd), PMD_TYPE_TABLE); | 52 | __pud_populate(pudp, __pa(pmdp), PMD_TYPE_TABLE); |
| 53 | } | 53 | } |
| 54 | #else | 54 | #else |
| 55 | static inline void __pud_populate(pud_t *pud, phys_addr_t pmd, pudval_t prot) | 55 | static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) |
| 56 | { | 56 | { |
| 57 | BUILD_BUG(); | 57 | BUILD_BUG(); |
| 58 | } | 58 | } |
| @@ -65,30 +65,30 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 6 unmodified lines | |||
| 65 | return (pud_t *)__get_free_page(PGALLOC_GFP); | 65 | return (pud_t *)__get_free_page(PGALLOC_GFP); |
| 66 | } | 66 | } |
| 67 | 67 | ||
| 68 | static inline void pud_free(struct mm_struct *mm, pud_t *pud) | 68 | static inline void pud_free(struct mm_struct *mm, pud_t *pudp) |
| 69 | { | 69 | { |
| 70 | BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); | 70 | BUG_ON((unsigned long)pudp & (PAGE_SIZE-1)); |
| 71 | free_page((unsigned long)pud); | 71 | free_page((unsigned long)pudp); |
| 72 | } | 72 | } |
| 73 | 73 | ||
| 74 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pud, pgdval_t prot) | 74 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) |
| 75 | { | 75 | { |
| 76 | set_pgd(pgdp, __pgd(__phys_to_pgd_val(pud) | prot)); | 76 | set_pgd(pgdp, __pgd(__phys_to_pgd_val(pudp) | prot)); |
| 77 | } | 77 | } |
| 78 | 78 | ||
| 79 | static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud) | 79 | static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, pud_t *pudp) |
| 80 | { | 80 | { |
| 81 | __pgd_populate(pgd, __pa(pud), PUD_TYPE_TABLE); | 81 | __pgd_populate(pgdp, __pa(pudp), PUD_TYPE_TABLE); |
| 82 | } | 82 | } |
| 83 | #else | 83 | #else |
| 84 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pud, pgdval_t prot) | 84 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) |
| 85 | { | 85 | { |
| 86 | BUILD_BUG(); | 86 | BUILD_BUG(); |
| 87 | } | 87 | } |
| 88 | #endif /* CONFIG_PGTABLE_LEVELS > 3 */ | 88 | #endif /* CONFIG_PGTABLE_LEVELS > 3 */ |
| 89 | 89 | ||
| 90 | extern pgd_t *pgd_alloc(struct mm_struct *mm); | 90 | extern pgd_t *pgd_alloc(struct mm_struct *mm); |
| 91 | extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); | 91 | extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); |
| 92 | 92 | ||
| 93 | static inline pte_t * | 93 | static inline pte_t * |
| 94 | pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) | 94 | pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) |
| @@ -114,10 +114,10 @@ pte_alloc_one(struct mm_struct *mm, unsigned long addr) · 19 unmodified lines | |||
| 114 | /* | 114 | /* |
| 115 | * Free a PTE table. | 115 | * Free a PTE table. |
| 116 | */ | 116 | */ |
| 117 | static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) | 117 | static inline void pte_free_kernel(struct mm_struct *mm, pte_t *ptep) |
| 118 | { | 118 | { |
| 119 | if (pte) | 119 | if (ptep) |
| 120 | free_page((unsigned long)pte); | 120 | free_page((unsigned long)ptep); |
| 121 | } | 121 | } |
| 122 | 122 | ||
| 123 | static inline void pte_free(struct mm_struct *mm, pgtable_t pte) | 123 | static inline void pte_free(struct mm_struct *mm, pgtable_t pte) |
| @@ -126,10 +126,10 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pte) · 2 unmodified lines | |||
| 126 | __free_page(pte); | 126 | __free_page(pte); |
| 127 | } | 127 | } |
| 128 | 128 | ||
| 129 | static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t pte, | 129 | static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t ptep, |
| 130 | pmdval_t prot) | 130 | pmdval_t prot) |
| 131 | { | 131 | { |
| 132 | set_pmd(pmdp, __pmd(__phys_to_pmd_val(pte) | prot)); | 132 | set_pmd(pmdp, __pmd(__phys_to_pmd_val(ptep) | prot)); |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | /* | 135 | /* |
| @@ -218,7 +218,7 @@ static inline pmd_t pmd_mkcont(pmd_t pmd) · 217 unmodified lines | |||
| 218 | 218 | ||
| 219 | static inline void set_pte(pte_t *ptep, pte_t pte) | 219 | static inline void set_pte(pte_t *ptep, pte_t pte) |
| 220 | { | 220 | { |
| 221 | *ptep = pte; | 221 | WRITE_ONCE(*ptep, pte); |
| 222 | 222 | ||
| 223 | /* | 223 | /* |
| 224 | * Only if the new pte is valid and kernel, otherwise TLB maintenance | 224 | * Only if the new pte is valid and kernel, otherwise TLB maintenance |
| @@ -250,6 +250,8 @@ extern void __sync_icache_dcache(pte_t pteval, unsigned long addr); · 25 unmodified lines | |||
| 250 | static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, | 250 | static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, |
| 251 | pte_t *ptep, pte_t pte) | 251 | pte_t *ptep, pte_t pte) |
| 252 | { | 252 | { |
| 253 | pte_t old_pte; | ||
| 254 | |||
| 253 | if (pte_present(pte) && pte_user_exec(pte) && !pte_special(pte)) | 255 | if (pte_present(pte) && pte_user_exec(pte) && !pte_special(pte)) |
| 254 | __sync_icache_dcache(pte, addr); | 256 | __sync_icache_dcache(pte, addr); |
| 255 | 257 | ||
| @@ -258,14 +260,15 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines | |||
| 258 | * hardware updates of the pte (ptep_set_access_flags safely changes | 260 | * hardware updates of the pte (ptep_set_access_flags safely changes |
| 259 | * valid ptes without going through an invalid entry). | 261 | * valid ptes without going through an invalid entry). |
| 260 | */ | 262 | */ |
| 261 | if (IS_ENABLED(CONFIG_DEBUG_VM) && pte_valid(*ptep) && pte_valid(pte) && | 263 | old_pte = READ_ONCE(*ptep); |
| 264 | if (IS_ENABLED(CONFIG_DEBUG_VM) && pte_valid(old_pte) && pte_valid(pte) && | ||
| 262 | (mm == current->active_mm || atomic_read(&mm->mm_users) > 1)) { | 265 | (mm == current->active_mm || atomic_read(&mm->mm_users) > 1)) { |
| 263 | VM_WARN_ONCE(!pte_young(pte), | 266 | VM_WARN_ONCE(!pte_young(pte), |
| 264 | "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", | 267 | "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", |
| 265 | __func__, pte_val(*ptep), pte_val(pte)); | 268 | __func__, pte_val(old_pte), pte_val(pte)); |
| 266 | VM_WARN_ONCE(pte_write(*ptep) && !pte_dirty(pte), | 269 | VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), |
| 267 | "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", | 270 | "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", |
| 268 | __func__, pte_val(*ptep), pte_val(pte)); | 271 | __func__, pte_val(old_pte), pte_val(pte)); |
| 269 | } | 272 | } |
| 270 | 273 | ||
| 271 | set_pte(ptep, pte); | 274 | set_pte(ptep, pte); |
| @@ -431,7 +434,7 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, · 159 unmodified lines | |||
| 431 | 434 | ||
| 432 | static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) | 435 | static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) |
| 433 | { | 436 | { |
| 434 | *pmdp = pmd; | 437 | WRITE_ONCE(*pmdp, pmd); |
| 435 | dsb(ishst); | 438 | dsb(ishst); |
| 436 | isb(); | 439 | isb(); |
| 437 | } | 440 | } |
| @@ -482,7 +485,7 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd) · 44 unmodified lines | |||
| 482 | 485 | ||
| 483 | static inline void set_pud(pud_t *pudp, pud_t pud) | 486 | static inline void set_pud(pud_t *pudp, pud_t pud) |
| 484 | { | 487 | { |
| 485 | *pudp = pud; | 488 | WRITE_ONCE(*pudp, pud); |
| 486 | dsb(ishst); | 489 | dsb(ishst); |
| 487 | isb(); | 490 | isb(); |
| 488 | } | 491 | } |
| @@ -500,7 +503,7 @@ static inline phys_addr_t pud_page_paddr(pud_t pud) · 11 unmodified lines | |||
| 500 | /* Find an entry in the second-level page table. */ | 503 | /* Find an entry in the second-level page table. */ |
| 501 | #define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1)) | 504 | #define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1)) |
| 502 | 505 | ||
| 503 | #define pmd_offset_phys(dir, addr) (pud_page_paddr(*(dir)) + pmd_index(addr) * sizeof(pmd_t)) | 506 | #define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t)) |
| 504 | #define pmd_offset(dir, addr) ((pmd_t *)__va(pmd_offset_phys((dir), (addr)))) | 507 | #define pmd_offset(dir, addr) ((pmd_t *)__va(pmd_offset_phys((dir), (addr)))) |
| 505 | 508 | ||
| 506 | #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) | 509 | #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) |
| @@ -535,7 +538,7 @@ static inline phys_addr_t pud_page_paddr(pud_t pud) · 28 unmodified lines | |||
| 535 | 538 | ||
| 536 | static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) | 539 | static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) |
| 537 | { | 540 | { |
| 538 | *pgdp = pgd; | 541 | WRITE_ONCE(*pgdp, pgd); |
| 539 | dsb(ishst); | 542 | dsb(ishst); |
| 540 | } | 543 | } |
| 541 | 544 | ||
| @@ -552,7 +555,7 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd) · 10 unmodified lines | |||
| 552 | /* Find an entry in the frst-level page table. */ | 555 | /* Find an entry in the frst-level page table. */ |
| 553 | #define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) | 556 | #define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) |
| 554 | 557 | ||
| 555 | #define pud_offset_phys(dir, addr) (pgd_page_paddr(*(dir)) + pud_index(addr) * sizeof(pud_t)) | 558 | #define pud_offset_phys(dir, addr) (pgd_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t)) |
| 556 | #define pud_offset(dir, addr) ((pud_t *)__va(pud_offset_phys((dir), (addr)))) | 559 | #define pud_offset(dir, addr) ((pud_t *)__va(pud_offset_phys((dir), (addr)))) |
| 557 | 560 | ||
| 558 | #define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) | 561 | #define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) |
| @@ -28,7 +28,7 @@ struct stackframe { · 27 unmodified lines | |||
| 28 | unsigned long fp; | 28 | unsigned long fp; |
| 29 | unsigned long pc; | 29 | unsigned long pc; |
| 30 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER | 30 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER |
| 31 | unsigned int graph; | 31 | int graph; |
| 32 | #endif | 32 | #endif |
| 33 | }; | 33 | }; |
| 34 | 34 | ||
| @@ -72,15 +72,15 @@ static inline void set_fs(mm_segment_t fs) · 71 unmodified lines | |||
| 72 | * This is equivalent to the following test: | 72 | * This is equivalent to the following test: |
| 73 | * (u65)addr + (u65)size <= (u65)current->addr_limit + 1 | 73 | * (u65)addr + (u65)size <= (u65)current->addr_limit + 1 |
| 74 | */ | 74 | */ |
| 75 | static inline unsigned long __range_ok(unsigned long addr, unsigned long size) | 75 | static inline unsigned long __range_ok(const void __user *addr, unsigned long size) |
| 76 | { | 76 | { |
| 77 | unsigned long limit = current_thread_info()->addr_limit; | 77 | unsigned long ret, limit = current_thread_info()->addr_limit; |
| 78 | 78 | ||
| 79 | __chk_user_ptr(addr); | 79 | __chk_user_ptr(addr); |
| 80 | asm volatile( | 80 | asm volatile( |
| 81 | // A + B <= C + 1 for all A,B,C, in four easy steps: | 81 | // A + B <= C + 1 for all A,B,C, in four easy steps: |
| 82 | // 1: X = A + B; X' = X % 2^64 | 82 | // 1: X = A + B; X' = X % 2^64 |
| 83 | " adds %0, %0, %2\n" | 83 | " adds %0, %3, %2\n" |
| 84 | // 2: Set C = 0 if X > 2^64, to guarantee X' > C in step 4 | 84 | // 2: Set C = 0 if X > 2^64, to guarantee X' > C in step 4 |
| 85 | " csel %1, xzr, %1, hi\n" | 85 | " csel %1, xzr, %1, hi\n" |
| 86 | // 3: Set X' = ~0 if X >= 2^64. For X == 2^64, this decrements X' | 86 | // 3: Set X' = ~0 if X >= 2^64. For X == 2^64, this decrements X' |
| @@ -92,9 +92,9 @@ static inline unsigned long __range_ok(unsigned long addr, unsigned long size) · 5 unmodified lines | |||
| 92 | // testing X' - C == 0, subject to the previous adjustments. | 92 | // testing X' - C == 0, subject to the previous adjustments. |
| 93 | " sbcs xzr, %0, %1\n" | 93 | " sbcs xzr, %0, %1\n" |
| 94 | " cset %0, ls\n" | 94 | " cset %0, ls\n" |
| 95 | : "+r" (addr), "+r" (limit) : "Ir" (size) : "cc"); | 95 | : "=&r" (ret), "+r" (limit) : "Ir" (size), "0" (addr) : "cc"); |
| 96 | 96 | ||
| 97 | return addr; | 97 | return ret; |
| 98 | } | 98 | } |
| 99 | 99 | ||
| 100 | /* | 100 | /* |
| @@ -104,7 +104,7 @@ static inline unsigned long __range_ok(unsigned long addr, unsigned long size) · 3 unmodified lines | |||
| 104 | */ | 104 | */ |
| 105 | #define untagged_addr(addr) sign_extend64(addr, 55) | 105 | #define untagged_addr(addr) sign_extend64(addr, 55) |
| 106 | 106 | ||
| 107 | #define access_ok(type, addr, size) __range_ok((unsigned long)(addr), size) | 107 | #define access_ok(type, addr, size) __range_ok(addr, size) |
| 108 | #define user_addr_max get_fs | 108 | #define user_addr_max get_fs |
| 109 | 109 | ||
| 110 | #define _ASM_EXTABLE(from, to) \ | 110 | #define _ASM_EXTABLE(from, to) \ |
| @@ -370,6 +370,7 @@ static unsigned int __kprobes aarch32_check_condition(u32 opcode, u32 psr) · 369 unmodified lines | |||
| 370 | static int swp_handler(struct pt_regs *regs, u32 instr) | 370 | static int swp_handler(struct pt_regs *regs, u32 instr) |
| 371 | { | 371 | { |
| 372 | u32 destreg, data, type, address = 0; | 372 | u32 destreg, data, type, address = 0; |
| 373 | const void __user *user_ptr; | ||
| 373 | int rn, rt2, res = 0; | 374 | int rn, rt2, res = 0; |
| 374 | 375 | ||
| 375 | perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS, 1, regs, regs->pc); | 376 | perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS, 1, regs, regs->pc); |
| @@ -401,7 +402,8 @@ static int swp_handler(struct pt_regs *regs, u32 instr) · 25 unmodified lines | |||
| 401 | aarch32_insn_extract_reg_num(instr, A32_RT2_OFFSET), data); | 402 | aarch32_insn_extract_reg_num(instr, A32_RT2_OFFSET), data); |
| 402 | 403 | ||
| 403 | /* Check access in reasonable access range for both SWP and SWPB */ | 404 | /* Check access in reasonable access range for both SWP and SWPB */ |
| 404 | if (!access_ok(VERIFY_WRITE, (address & ~3), 4)) { | 405 | user_ptr = (const void __user *)(unsigned long)(address & ~3); |
| 406 | if (!access_ok(VERIFY_WRITE, user_ptr, 4)) { | ||
| 405 | pr_debug("SWP{B} emulation: access to 0x%08x not allowed!\n", | 407 | pr_debug("SWP{B} emulation: access to 0x%08x not allowed!\n", |
| 406 | address); | 408 | address); |
| 407 | goto fault; | 409 | goto fault; |
| @@ -178,7 +178,7 @@ static int enable_smccc_arch_workaround_1(void *data) · 177 unmodified lines | |||
| 178 | case PSCI_CONDUIT_HVC: | 178 | case PSCI_CONDUIT_HVC: |
| 179 | arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, | 179 | arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, |
| 180 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); | 180 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); |
| 181 | if (res.a0) | 181 | if ((int)res.a0 < 0) |
| 182 | return 0; | 182 | return 0; |
| 183 | cb = call_hvc_arch_workaround_1; | 183 | cb = call_hvc_arch_workaround_1; |
| 184 | smccc_start = __smccc_workaround_1_hvc_start; | 184 | smccc_start = __smccc_workaround_1_hvc_start; |
| @@ -188,7 +188,7 @@ static int enable_smccc_arch_workaround_1(void *data) · 3 unmodified lines | |||
| 188 | case PSCI_CONDUIT_SMC: | 188 | case PSCI_CONDUIT_SMC: |
| 189 | arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, | 189 | arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, |
| 190 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); | 190 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); |
| 191 | if (res.a0) | 191 | if ((int)res.a0 < 0) |
| 192 | return 0; | 192 | return 0; |
| 193 | cb = call_smc_arch_workaround_1; | 193 | cb = call_smc_arch_workaround_1; |
| 194 | smccc_start = __smccc_workaround_1_smc_start; | 194 | smccc_start = __smccc_workaround_1_smc_start; |
| @@ -406,6 +406,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = { · 211 unmodified lines | |||
| 406 | .capability = ARM64_HARDEN_BP_POST_GUEST_EXIT, | 406 | .capability = ARM64_HARDEN_BP_POST_GUEST_EXIT, |
| 407 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR_V1), | 407 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR_V1), |
| 408 | }, | 408 | }, |
| 409 | { | ||
| 410 | .capability = ARM64_HARDEN_BRANCH_PREDICTOR, | ||
| 411 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR), | ||
| 412 | .enable = qcom_enable_link_stack_sanitization, | ||
| 413 | }, | ||
| 414 | { | ||
| 415 | .capability = ARM64_HARDEN_BP_POST_GUEST_EXIT, | ||
| 416 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR), | ||
| 417 | }, | ||
| 409 | { | 418 | { |
| 410 | .capability = ARM64_HARDEN_BRANCH_PREDICTOR, | 419 | .capability = ARM64_HARDEN_BRANCH_PREDICTOR, |
| 411 | MIDR_ALL_VERSIONS(MIDR_BRCM_VULCAN), | 420 | MIDR_ALL_VERSIONS(MIDR_BRCM_VULCAN), |
| @@ -199,9 +199,11 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr2[] = { · 198 unmodified lines | |||
| 199 | }; | 199 | }; |
| 200 | 200 | ||
| 201 | static const struct arm64_ftr_bits ftr_ctr[] = { | 201 | static const struct arm64_ftr_bits ftr_ctr[] = { |
| 202 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RAO */ | 202 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RES1 */ |
| 203 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 29, 1, 1), /* DIC */ | ||
| 204 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 28, 1, 1), /* IDC */ | ||
| 203 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */ | 205 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */ |
| 204 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 20, 4, 0), /* ERG */ | 206 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_SAFE, 20, 4, 0), /* ERG */ |
| 205 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */ | 207 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */ |
| 206 | /* | 208 | /* |
| 207 | * Linux can handle differing I-cache policies. Userspace JITs will | 209 | * Linux can handle differing I-cache policies. Userspace JITs will |
| @@ -90,7 +90,7 @@ static int __init set_permissions(pte_t *ptep, pgtable_t token, · 89 unmodified lines | |||
| 90 | unsigned long addr, void *data) | 90 | unsigned long addr, void *data) |
| 91 | { | 91 | { |
| 92 | efi_memory_desc_t *md = data; | 92 | efi_memory_desc_t *md = data; |
| 93 | pte_t pte = *ptep; | 93 | pte_t pte = READ_ONCE(*ptep); |
| 94 | 94 | ||
| 95 | if (md->attribute & EFI_MEMORY_RO) | 95 | if (md->attribute & EFI_MEMORY_RO) |
| 96 | pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); | 96 | pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); |
| @@ -202,10 +202,10 @@ static int create_safe_exec_page(void *src_start, size_t length, · 201 unmodified lines | |||
| 202 | gfp_t mask) | 202 | gfp_t mask) |
| 203 | { | 203 | { |
| 204 | int rc = 0; | 204 | int rc = 0; |
| 205 | pgd_t *pgd; | 205 | pgd_t *pgdp; |
| 206 | pud_t *pud; | 206 | pud_t *pudp; |
| 207 | pmd_t *pmd; | 207 | pmd_t *pmdp; |
| 208 | pte_t *pte; | 208 | pte_t *ptep; |
| 209 | unsigned long dst = (unsigned long)allocator(mask); | 209 | unsigned long dst = (unsigned long)allocator(mask); |
| 210 | 210 | ||
| 211 | if (!dst) { | 211 | if (!dst) { |
| @@ -216,38 +216,38 @@ static int create_safe_exec_page(void *src_start, size_t length, · 4 unmodified lines | |||
| 216 | memcpy((void *)dst, src_start, length); | 216 | memcpy((void *)dst, src_start, length); |
| 217 | flush_icache_range(dst, dst + length); | 217 | flush_icache_range(dst, dst + length); |
| 218 | 218 | ||
| 219 | pgd = pgd_offset_raw(allocator(mask), dst_addr); | 219 | pgdp = pgd_offset_raw(allocator(mask), dst_addr); |
| 220 | if (pgd_none(*pgd)) { | 220 | if (pgd_none(READ_ONCE(*pgdp))) { |
| 221 | pud = allocator(mask); | 221 | pudp = allocator(mask); |
| 222 | if (!pud) { | 222 | if (!pudp) { |
| 223 | rc = -ENOMEM; | 223 | rc = -ENOMEM; |
| 224 | goto out; | 224 | goto out; |
| 225 | } | 225 | } |
| 226 | pgd_populate(&init_mm, pgd, pud); | 226 | pgd_populate(&init_mm, pgdp, pudp); |
| 227 | } | 227 | } |
| 228 | 228 | ||
| 229 | pud = pud_offset(pgd, dst_addr); | 229 | pudp = pud_offset(pgdp, dst_addr); |
| 230 | if (pud_none(*pud)) { | 230 | if (pud_none(READ_ONCE(*pudp))) { |
| 231 | pmd = allocator(mask); | 231 | pmdp = allocator(mask); |
| 232 | if (!pmd) { | 232 | if (!pmdp) { |
| 233 | rc = -ENOMEM; | 233 | rc = -ENOMEM; |
| 234 | goto out; | 234 | goto out; |
| 235 | } | 235 | } |
| 236 | pud_populate(&init_mm, pud, pmd); | 236 | pud_populate(&init_mm, pudp, pmdp); |
| 237 | } | 237 | } |
| 238 | 238 | ||
| 239 | pmd = pmd_offset(pud, dst_addr); | 239 | pmdp = pmd_offset(pudp, dst_addr); |
| 240 | if (pmd_none(*pmd)) { | 240 | if (pmd_none(READ_ONCE(*pmdp))) { |
| 241 | pte = allocator(mask); | 241 | ptep = allocator(mask); |
| 242 | if (!pte) { | 242 | if (!ptep) { |
| 243 | rc = -ENOMEM; | 243 | rc = -ENOMEM; |
| 244 | goto out; | 244 | goto out; |
| 245 | } | 245 | } |
| 246 | pmd_populate_kernel(&init_mm, pmd, pte); | 246 | pmd_populate_kernel(&init_mm, pmdp, ptep); |
| 247 | } | 247 | } |
| 248 | 248 | ||
| 249 | pte = pte_offset_kernel(pmd, dst_addr); | 249 | ptep = pte_offset_kernel(pmdp, dst_addr); |
| 250 | set_pte(pte, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC)); | 250 | set_pte(ptep, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC)); |
| 251 | 251 | ||
| 252 | /* | 252 | /* |
| 253 | * Load our new page tables. A strict BBM approach requires that we | 253 | * Load our new page tables. A strict BBM approach requires that we |
| @@ -263,7 +263,7 @@ static int create_safe_exec_page(void *src_start, size_t length, · 9 unmodified lines | |||
| 263 | */ | 263 | */ |
| 264 | cpu_set_reserved_ttbr0(); | 264 | cpu_set_reserved_ttbr0(); |
| 265 | local_flush_tlb_all(); | 265 | local_flush_tlb_all(); |
| 266 | write_sysreg(phys_to_ttbr(virt_to_phys(pgd)), ttbr0_el1); | 266 | write_sysreg(phys_to_ttbr(virt_to_phys(pgdp)), ttbr0_el1); |
| 267 | isb(); | 267 | isb(); |
| 268 | 268 | ||
| 269 | *phys_dst_addr = virt_to_phys((void *)dst); | 269 | *phys_dst_addr = virt_to_phys((void *)dst); |
| @@ -320,9 +320,9 @@ int swsusp_arch_suspend(void) · 50 unmodified lines | |||
| 320 | return ret; | 320 | return ret; |
| 321 | } | 321 | } |
| 322 | 322 | ||
| 323 | static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr) | 323 | static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) |
| 324 | { | 324 | { |
| 325 | pte_t pte = *src_pte; | 325 | pte_t pte = READ_ONCE(*src_ptep); |
| 326 | 326 | ||
| 327 | if (pte_valid(pte)) { | 327 | if (pte_valid(pte)) { |
| 328 | /* | 328 | /* |
| @@ -330,7 +330,7 @@ static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr) · 1 unmodified line | |||
| 330 | * read only (code, rodata). Clear the RDONLY bit from | 330 | * read only (code, rodata). Clear the RDONLY bit from |
| 331 | * the temporary mappings we use during restore. | 331 | * the temporary mappings we use during restore. |
| 332 | */ | 332 | */ |
| 333 | set_pte(dst_pte, pte_mkwrite(pte)); | 333 | set_pte(dst_ptep, pte_mkwrite(pte)); |
| 334 | } else if (debug_pagealloc_enabled() && !pte_none(pte)) { | 334 | } else if (debug_pagealloc_enabled() && !pte_none(pte)) { |
| 335 | /* | 335 | /* |
| 336 | * debug_pagealloc will removed the PTE_VALID bit if | 336 | * debug_pagealloc will removed the PTE_VALID bit if |
| @@ -343,112 +343,116 @@ static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr) · 6 unmodified lines | |||
| 343 | */ | 343 | */ |
| 344 | BUG_ON(!pfn_valid(pte_pfn(pte))); | 344 | BUG_ON(!pfn_valid(pte_pfn(pte))); |
| 345 | 345 | ||
| 346 | set_pte(dst_pte, pte_mkpresent(pte_mkwrite(pte))); | 346 | set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte))); |
| 347 | } | 347 | } |
| 348 | } | 348 | } |
| 349 | 349 | ||
| 350 | static int copy_pte(pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long start, | 350 | static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start, |
| 351 | unsigned long end) | 351 | unsigned long end) |
| 352 | { | 352 | { |
| 353 | pte_t *src_pte; | 353 | pte_t *src_ptep; |
| 354 | pte_t *dst_pte; | 354 | pte_t *dst_ptep; |
| 355 | unsigned long addr = start; | 355 | unsigned long addr = start; |
| 356 | 356 | ||
| 357 | dst_pte = (pte_t *)get_safe_page(GFP_ATOMIC); | 357 | dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); |
| 358 | if (!dst_pte) | 358 | if (!dst_ptep) |
| 359 | return -ENOMEM; | 359 | return -ENOMEM; |
| 360 | pmd_populate_kernel(&init_mm, dst_pmd, dst_pte); | 360 | pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep); |
| 361 | dst_pte = pte_offset_kernel(dst_pmd, start); | 361 | dst_ptep = pte_offset_kernel(dst_pmdp, start); |
| 362 | 362 | ||
| 363 | src_pte = pte_offset_kernel(src_pmd, start); | 363 | src_ptep = pte_offset_kernel(src_pmdp, start); |
| 364 | do { | 364 | do { |
| 365 | _copy_pte(dst_pte, src_pte, addr); | 365 | _copy_pte(dst_ptep, src_ptep, addr); |
| 366 | } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); | 366 | } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end); |
| 367 | 367 | ||
| 368 | return 0; | 368 | return 0; |
| 369 | } | 369 | } |
| 370 | 370 | ||
| 371 | static int copy_pmd(pud_t *dst_pud, pud_t *src_pud, unsigned long start, | 371 | static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start, |
| 372 | unsigned long end) | 372 | unsigned long end) |
| 373 | { | 373 | { |
| 374 | pmd_t *src_pmd; | 374 | pmd_t *src_pmdp; |
| 375 | pmd_t *dst_pmd; | 375 | pmd_t *dst_pmdp; |
| 376 | unsigned long next; | 376 | unsigned long next; |
| 377 | unsigned long addr = start; | 377 | unsigned long addr = start; |
| 378 | 378 | ||
| 379 | if (pud_none(*dst_pud)) { | 379 | if (pud_none(READ_ONCE(*dst_pudp))) { |
| 380 | dst_pmd = (pmd_t *)get_safe_page(GFP_ATOMIC); | 380 | dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); |
| 381 | if (!dst_pmd) | 381 | if (!dst_pmdp) |
| 382 | return -ENOMEM; | 382 | return -ENOMEM; |
| 383 | pud_populate(&init_mm, dst_pud, dst_pmd); | 383 | pud_populate(&init_mm, dst_pudp, dst_pmdp); |
| 384 | } | 384 | } |
| 385 | dst_pmd = pmd_offset(dst_pud, start); | 385 | dst_pmdp = pmd_offset(dst_pudp, start); |
| 386 | 386 | ||
| 387 | src_pmd = pmd_offset(src_pud, start); | 387 | src_pmdp = pmd_offset(src_pudp, start); |
| 388 | do { | 388 | do { |
| 389 | pmd_t pmd = READ_ONCE(*src_pmdp); | ||
| 390 | |||
| 389 | next = pmd_addr_end(addr, end); | 391 | next = pmd_addr_end(addr, end); |
| 390 | if (pmd_none(*src_pmd)) | 392 | if (pmd_none(pmd)) |
| 391 | continue; | 393 | continue; |
| 392 | if (pmd_table(*src_pmd)) { | 394 | if (pmd_table(pmd)) { |
| 393 | if (copy_pte(dst_pmd, src_pmd, addr, next)) | 395 | if (copy_pte(dst_pmdp, src_pmdp, addr, next)) |
| 394 | return -ENOMEM; | 396 | return -ENOMEM; |
| 395 | } else { | 397 | } else { |
| 396 | set_pmd(dst_pmd, | 398 | set_pmd(dst_pmdp, |
| 397 | __pmd(pmd_val(*src_pmd) & ~PMD_SECT_RDONLY)); | 399 | __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY)); |
| 398 | } | 400 | } |
| 399 | } while (dst_pmd++, src_pmd++, addr = next, addr != end); | 401 | } while (dst_pmdp++, src_pmdp++, addr = next, addr != end); |
| 400 | 402 | ||
| 401 | return 0; | 403 | return 0; |
| 402 | } | 404 | } |
| 403 | 405 | ||
| 404 | static int copy_pud(pgd_t *dst_pgd, pgd_t *src_pgd, unsigned long start, | 406 | static int copy_pud(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start, |
| 405 | unsigned long end) | 407 | unsigned long end) |
| 406 | { | 408 | { |
| 407 | pud_t *dst_pud; | 409 | pud_t *dst_pudp; |
| 408 | pud_t *src_pud; | 410 | pud_t *src_pudp; |
| 409 | unsigned long next; | 411 | unsigned long next; |
| 410 | unsigned long addr = start; | 412 | unsigned long addr = start; |
| 411 | 413 | ||
| 412 | if (pgd_none(*dst_pgd)) { | 414 | if (pgd_none(READ_ONCE(*dst_pgdp))) { |
| 413 | dst_pud = (pud_t *)get_safe_page(GFP_ATOMIC); | 415 | dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); |
| 414 | if (!dst_pud) | 416 | if (!dst_pudp) |
| 415 | return -ENOMEM; | 417 | return -ENOMEM; |
| 416 | pgd_populate(&init_mm, dst_pgd, dst_pud); | 418 | pgd_populate(&init_mm, dst_pgdp, dst_pudp); |
| 417 | } | 419 | } |
| 418 | dst_pud = pud_offset(dst_pgd, start); | 420 | dst_pudp = pud_offset(dst_pgdp, start); |
| 419 | 421 | ||
| 420 | src_pud = pud_offset(src_pgd, start); | 422 | src_pudp = pud_offset(src_pgdp, start); |
| 421 | do { | 423 | do { |
| 424 | pud_t pud = READ_ONCE(*src_pudp); | ||
| 425 | |||
| 422 | next = pud_addr_end(addr, end); | 426 | next = pud_addr_end(addr, end); |
| 423 | if (pud_none(*src_pud)) | 427 | if (pud_none(pud)) |
| 424 | continue; | 428 | continue; |
| 425 | if (pud_table(*(src_pud))) { | 429 | if (pud_table(pud)) { |
| 426 | if (copy_pmd(dst_pud, src_pud, addr, next)) | 430 | if (copy_pmd(dst_pudp, src_pudp, addr, next)) |
| 427 | return -ENOMEM; | 431 | return -ENOMEM; |
| 428 | } else { | 432 | } else { |
| 429 | set_pud(dst_pud, | 433 | set_pud(dst_pudp, |
| 430 | __pud(pud_val(*src_pud) & ~PMD_SECT_RDONLY)); | 434 | __pud(pud_val(pud) & ~PMD_SECT_RDONLY)); |
| 431 | } | 435 | } |
| 432 | } while (dst_pud++, src_pud++, addr = next, addr != end); | 436 | } while (dst_pudp++, src_pudp++, addr = next, addr != end); |
| 433 | 437 | ||
| 434 | return 0; | 438 | return 0; |
| 435 | } | 439 | } |
| 436 | 440 | ||
| 437 | static int copy_page_tables(pgd_t *dst_pgd, unsigned long start, | 441 | static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start, |
| 438 | unsigned long end) | 442 | unsigned long end) |
| 439 | { | 443 | { |
| 440 | unsigned long next; | 444 | unsigned long next; |
| 441 | unsigned long addr = start; | 445 | unsigned long addr = start; |
| 442 | pgd_t *src_pgd = pgd_offset_k(start); | 446 | pgd_t *src_pgdp = pgd_offset_k(start); |
| 443 | 447 | ||
| 444 | dst_pgd = pgd_offset_raw(dst_pgd, start); | 448 | dst_pgdp = pgd_offset_raw(dst_pgdp, start); |
| 445 | do { | 449 | do { |
| 446 | next = pgd_addr_end(addr, end); | 450 | next = pgd_addr_end(addr, end); |
| 447 | if (pgd_none(*src_pgd)) | 451 | if (pgd_none(READ_ONCE(*src_pgdp))) |
| 448 | continue; | 452 | continue; |
| 449 | if (copy_pud(dst_pgd, src_pgd, addr, next)) | 453 | if (copy_pud(dst_pgdp, src_pgdp, addr, next)) |
| 450 | return -ENOMEM; | 454 | return -ENOMEM; |
| 451 | } while (dst_pgd++, src_pgd++, addr = next, addr != end); | 455 | } while (dst_pgdp++, src_pgdp++, addr = next, addr != end); |
| 452 | 456 | ||
| 453 | return 0; | 457 | return 0; |
| 454 | } | 458 | } |
| @@ -908,9 +908,9 @@ static void __armv8pmu_probe_pmu(void *info) · 907 unmodified lines | |||
| 908 | int pmuver; | 908 | int pmuver; |
| 909 | 909 | ||
| 910 | dfr0 = read_sysreg(id_aa64dfr0_el1); | 910 | dfr0 = read_sysreg(id_aa64dfr0_el1); |
| 911 | pmuver = cpuid_feature_extract_signed_field(dfr0, | 911 | pmuver = cpuid_feature_extract_unsigned_field(dfr0, |
| 912 | ID_AA64DFR0_PMUVER_SHIFT); | 912 | ID_AA64DFR0_PMUVER_SHIFT); |
| 913 | if (pmuver < 1) | 913 | if (pmuver == 0xf || pmuver == 0) |
| 914 | return; | 914 | return; |
| 915 | 915 | ||
| 916 | probe->present = true; | 916 | probe->present = true; |
| @@ -220,8 +220,15 @@ void __show_regs(struct pt_regs *regs) · 219 unmodified lines | |||
| 220 | 220 | ||
| 221 | show_regs_print_info(KERN_DEFAULT); | 221 | show_regs_print_info(KERN_DEFAULT); |
| 222 | print_pstate(regs); | 222 | print_pstate(regs); |
| 223 | printk("pc : %pS\n", (void *)regs->pc); | 223 | |
| 224 | printk("lr : %pS\n", (void *)lr); | 224 | if (!user_mode(regs)) { |
| 225 | printk("pc : %pS\n", (void *)regs->pc); | ||
| 226 | printk("lr : %pS\n", (void *)lr); | ||
| 227 | } else { | ||
| 228 | printk("pc : %016llx\n", regs->pc); | ||
| 229 | printk("lr : %016llx\n", lr); | ||
| 230 | } | ||
| 231 | |||
| 225 | printk("sp : %016llx\n", sp); | 232 | printk("sp : %016llx\n", sp); |
| 226 | 233 | ||
| 227 | i = top_reg; | 234 | i = top_reg; |
| @@ -1419,7 +1419,7 @@ static int compat_ptrace_hbp_get(unsigned int note_type, · 1418 unmodified lines | |||
| 1419 | u64 addr = 0; | 1419 | u64 addr = 0; |
| 1420 | u32 ctrl = 0; | 1420 | u32 ctrl = 0; |
| 1421 | 1421 | ||
| 1422 | int err, idx = compat_ptrace_hbp_num_to_idx(num);; | 1422 | int err, idx = compat_ptrace_hbp_num_to_idx(num); |
| 1423 | 1423 | ||
| 1424 | if (num & 1) { | 1424 | if (num & 1) { |
| 1425 | err = ptrace_hbp_get_addr(note_type, tsk, idx, &addr); | 1425 | err = ptrace_hbp_get_addr(note_type, tsk, idx, &addr); |
| @@ -59,6 +59,11 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) · 58 unmodified lines | |||
| 59 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER | 59 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER |
| 60 | if (tsk->ret_stack && | 60 | if (tsk->ret_stack && |
| 61 | (frame->pc == (unsigned long)return_to_handler)) { | 61 | (frame->pc == (unsigned long)return_to_handler)) { |
| 62 | if (WARN_ON_ONCE(frame->graph == -1)) | ||
| 63 | return -EINVAL; | ||
| 64 | if (frame->graph < -1) | ||
| 65 | frame->graph += FTRACE_NOTRACE_DEPTH; | ||
| 66 | |||
| 62 | /* | 67 | /* |
| 63 | * This is a case where function graph tracer has | 68 | * This is a case where function graph tracer has |
| 64 | * modified a return address (LR) in a stack frame | 69 | * modified a return address (LR) in a stack frame |
| @@ -57,7 +57,7 @@ do_compat_cache_op(unsigned long start, unsigned long end, int flags) · 56 unmodified lines | |||
| 57 | if (end < start || flags) | 57 | if (end < start || flags) |
| 58 | return -EINVAL; | 58 | return -EINVAL; |
| 59 | 59 | ||
| 60 | if (!access_ok(VERIFY_READ, start, end - start)) | 60 | if (!access_ok(VERIFY_READ, (const void __user *)start, end - start)) |
| 61 | return -EFAULT; | 61 | return -EFAULT; |
| 62 | 62 | ||
| 63 | return __do_compat_cache_op(start, end); | 63 | return __do_compat_cache_op(start, end); |
| @@ -52,7 +52,7 @@ unsigned long profile_pc(struct pt_regs *regs) · 51 unmodified lines | |||
| 52 | frame.fp = regs->regs[29]; | 52 | frame.fp = regs->regs[29]; |
| 53 | frame.pc = regs->pc; | 53 | frame.pc = regs->pc; |
| 54 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER | 54 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER |
| 55 | frame.graph = -1; /* no task info */ | 55 | frame.graph = current->curr_ret_stack; |
| 56 | #endif | 56 | #endif |
| 57 | do { | 57 | do { |
| 58 | int ret = unwind_frame(NULL, &frame); | 58 | int ret = unwind_frame(NULL, &frame); |
| @@ -57,7 +57,7 @@ static const char *handler[]= { · 56 unmodified lines | |||
| 57 | "Error" | 57 | "Error" |
| 58 | }; | 58 | }; |
| 59 | 59 | ||
| 60 | int show_unhandled_signals = 1; | 60 | int show_unhandled_signals = 0; |
| 61 | 61 | ||
| 62 | static void dump_backtrace_entry(unsigned long where) | 62 | static void dump_backtrace_entry(unsigned long where) |
| 63 | { | 63 | { |
| @@ -526,14 +526,6 @@ asmlinkage long do_ni_syscall(struct pt_regs *regs) · 462 unmodified lines | |||
| 526 | } | 526 | } |
| 527 | #endif | 527 | #endif |
| 528 | 528 | ||
| 529 | if (show_unhandled_signals_ratelimited()) { | ||
| 530 | pr_info("%s[%d]: syscall %d\n", current->comm, | ||
| 531 | task_pid_nr(current), regs->syscallno); | ||
| 532 | dump_instr("", regs); | ||
| 533 | if (user_mode(regs)) | ||
| 534 | __show_regs(regs); | ||
| 535 | } | ||
| 536 | |||
| 537 | return sys_ni_syscall(); | 529 | return sys_ni_syscall(); |
| 538 | } | 530 | } |
| 539 | 531 | ||
| @@ -363,8 +363,6 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, · 362 unmodified lines | |||
| 363 | { | 363 | { |
| 364 | int ret = 0; | 364 | int ret = 0; |
| 365 | 365 | ||
| 366 | vcpu_load(vcpu); | ||
| 367 | |||
| 368 | trace_kvm_set_guest_debug(vcpu, dbg->control); | 366 | trace_kvm_set_guest_debug(vcpu, dbg->control); |
| 369 | 367 | ||
| 370 | if (dbg->control & ~KVM_GUESTDBG_VALID_MASK) { | 368 | if (dbg->control & ~KVM_GUESTDBG_VALID_MASK) { |
| @@ -386,7 +384,6 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, · 15 unmodified lines | |||
| 386 | } | 384 | } |
| 387 | 385 | ||
| 388 | out: | 386 | out: |
| 389 | vcpu_put(vcpu); | ||
| 390 | return ret; | 387 | return ret; |
| 391 | } | 388 | } |
| 392 | 389 | ||
| @@ -407,8 +407,10 @@ again: · 406 unmodified lines | |||
| 407 | u32 midr = read_cpuid_id(); | 407 | u32 midr = read_cpuid_id(); |
| 408 | 408 | ||
| 409 | /* Apply BTAC predictors mitigation to all Falkor chips */ | 409 | /* Apply BTAC predictors mitigation to all Falkor chips */ |
| 410 | if ((midr & MIDR_CPU_MODEL_MASK) == MIDR_QCOM_FALKOR_V1) | 410 | if (((midr & MIDR_CPU_MODEL_MASK) == MIDR_QCOM_FALKOR) || |
| 411 | ((midr & MIDR_CPU_MODEL_MASK) == MIDR_QCOM_FALKOR_V1)) { | ||
| 411 | __qcom_hyp_sanitize_btac_predictors(); | 412 | __qcom_hyp_sanitize_btac_predictors(); |
| 413 | } | ||
| 412 | } | 414 | } |
| 413 | 415 | ||
| 414 | fp_enabled = __fpsimd_enabled(); | 416 | fp_enabled = __fpsimd_enabled(); |
| @@ -286,48 +286,52 @@ static void note_page(struct pg_state *st, unsigned long addr, unsigned level, · 285 unmodified lines | |||
| 286 | 286 | ||
| 287 | } | 287 | } |
| 288 | 288 | ||
| 289 | static void walk_pte(struct pg_state *st, pmd_t *pmd, unsigned long start) | 289 | static void walk_pte(struct pg_state *st, pmd_t *pmdp, unsigned long start) |
| 290 | { | 290 | { |
| 291 | pte_t *pte = pte_offset_kernel(pmd, 0UL); | 291 | pte_t *ptep = pte_offset_kernel(pmdp, 0UL); |
| 292 | unsigned long addr; | 292 | unsigned long addr; |
| 293 | unsigned i; | 293 | unsigned i; |
| 294 | 294 | ||
| 295 | for (i = 0; i < PTRS_PER_PTE; i++, pte++) { | 295 | for (i = 0; i < PTRS_PER_PTE; i++, ptep++) { |
| 296 | addr = start + i * PAGE_SIZE; | 296 | addr = start + i * PAGE_SIZE; |
| 297 | note_page(st, addr, 4, pte_val(*pte)); | 297 | note_page(st, addr, 4, READ_ONCE(pte_val(*ptep))); |
| 298 | } | 298 | } |
| 299 | } | 299 | } |
| 300 | 300 | ||
| 301 | static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) | 301 | static void walk_pmd(struct pg_state *st, pud_t *pudp, unsigned long start) |
| 302 | { | 302 | { |
| 303 | pmd_t *pmd = pmd_offset(pud, 0UL); | 303 | pmd_t *pmdp = pmd_offset(pudp, 0UL); |
| 304 | unsigned long addr; | 304 | unsigned long addr; |
| 305 | unsigned i; | 305 | unsigned i; |
| 306 | 306 | ||
| 307 | for (i = 0; i < PTRS_PER_PMD; i++, pmd++) { | 307 | for (i = 0; i < PTRS_PER_PMD; i++, pmdp++) { |
| 308 | pmd_t pmd = READ_ONCE(*pmdp); | ||
| 309 | |||
| 308 | addr = start + i * PMD_SIZE; | 310 | addr = start + i * PMD_SIZE; |
| 309 | if (pmd_none(*pmd) || pmd_sect(*pmd)) { | 311 | if (pmd_none(pmd) || pmd_sect(pmd)) { |
| 310 | note_page(st, addr, 3, pmd_val(*pmd)); | 312 | note_page(st, addr, 3, pmd_val(pmd)); |
| 311 | } else { | 313 | } else { |
| 312 | BUG_ON(pmd_bad(*pmd)); | 314 | BUG_ON(pmd_bad(pmd)); |
| 313 | walk_pte(st, pmd, addr); | 315 | walk_pte(st, pmdp, addr); |
| 314 | } | 316 | } |
| 315 | } | 317 | } |
| 316 | } | 318 | } |
| 317 | 319 | ||
| 318 | static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) | 320 | static void walk_pud(struct pg_state *st, pgd_t *pgdp, unsigned long start) |
| 319 | { | 321 | { |
| 320 | pud_t *pud = pud_offset(pgd, 0UL); | 322 | pud_t *pudp = pud_offset(pgdp, 0UL); |
| 321 | unsigned long addr; | 323 | unsigned long addr; |
| 322 | unsigned i; | 324 | unsigned i; |
| 323 | 325 | ||
| 324 | for (i = 0; i < PTRS_PER_PUD; i++, pud++) { | 326 | for (i = 0; i < PTRS_PER_PUD; i++, pudp++) { |
| 327 | pud_t pud = READ_ONCE(*pudp); | ||
| 328 | |||
| 325 | addr = start + i * PUD_SIZE; | 329 | addr = start + i * PUD_SIZE; |
| 326 | if (pud_none(*pud) || pud_sect(*pud)) { | 330 | if (pud_none(pud) || pud_sect(pud)) { |
| 327 | note_page(st, addr, 2, pud_val(*pud)); | 331 | note_page(st, addr, 2, pud_val(pud)); |
| 328 | } else { | 332 | } else { |
| 329 | BUG_ON(pud_bad(*pud)); | 333 | BUG_ON(pud_bad(pud)); |
| 330 | walk_pmd(st, pud, addr); | 334 | walk_pmd(st, pudp, addr); |
| 331 | } | 335 | } |
| 332 | } | 336 | } |
| 333 | } | 337 | } |
| @@ -335,17 +339,19 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) · 1 unmodified line | |||
| 335 | static void walk_pgd(struct pg_state *st, struct mm_struct *mm, | 339 | static void walk_pgd(struct pg_state *st, struct mm_struct *mm, |
| 336 | unsigned long start) | 340 | unsigned long start) |
| 337 | { | 341 | { |
| 338 | pgd_t *pgd = pgd_offset(mm, 0UL); | 342 | pgd_t *pgdp = pgd_offset(mm, 0UL); |
| 339 | unsigned i; | 343 | unsigned i; |
| 340 | unsigned long addr; | 344 | unsigned long addr; |
| 341 | 345 | ||
| 342 | for (i = 0; i < PTRS_PER_PGD; i++, pgd++) { | 346 | for (i = 0; i < PTRS_PER_PGD; i++, pgdp++) { |
| 347 | pgd_t pgd = READ_ONCE(*pgdp); | ||
| 348 | |||
| 343 | addr = start + i * PGDIR_SIZE; | 349 | addr = start + i * PGDIR_SIZE; |
| 344 | if (pgd_none(*pgd)) { | 350 | if (pgd_none(pgd)) { |
| 345 | note_page(st, addr, 1, pgd_val(*pgd)); | 351 | note_page(st, addr, 1, pgd_val(pgd)); |
| 346 | } else { | 352 | } else { |
| 347 | BUG_ON(pgd_bad(*pgd)); | 353 | BUG_ON(pgd_bad(pgd)); |
| 348 | walk_pud(st, pgd, addr); | 354 | walk_pud(st, pgdp, addr); |
| 349 | } | 355 | } |
| 350 | } | 356 | } |
| 351 | } | 357 | } |
| @@ -130,7 +130,8 @@ static void mem_abort_decode(unsigned int esr) · 129 unmodified lines | |||
| 130 | void show_pte(unsigned long addr) | 130 | void show_pte(unsigned long addr) |
| 131 | { | 131 | { |
| 132 | struct mm_struct *mm; | 132 | struct mm_struct *mm; |
| 133 | pgd_t *pgd; | 133 | pgd_t *pgdp; |
| 134 | pgd_t pgd; | ||
| 134 | 135 | ||
| 135 | if (addr < TASK_SIZE) { | 136 | if (addr < TASK_SIZE) { |
| 136 | /* TTBR0 */ | 137 | /* TTBR0 */ |
| @@ -149,33 +150,37 @@ void show_pte(unsigned long addr) · 12 unmodified lines | |||
| 149 | return; | 150 | return; |
| 150 | } | 151 | } |
| 151 | 152 | ||
| 152 | pr_alert("%s pgtable: %luk pages, %u-bit VAs, pgd = %p\n", | 153 | pr_alert("%s pgtable: %luk pages, %u-bit VAs, pgdp = %p\n", |
| 153 | mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K, | 154 | mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K, |
| 154 | VA_BITS, mm->pgd); | 155 | VA_BITS, mm->pgd); |
| 155 | pgd = pgd_offset(mm, addr); | 156 | pgdp = pgd_offset(mm, addr); |
| 156 | pr_alert("[%016lx] *pgd=%016llx", addr, pgd_val(*pgd)); | 157 | pgd = READ_ONCE(*pgdp); |
| 158 | pr_alert("[%016lx] pgd=%016llx", addr, pgd_val(pgd)); | ||
| 157 | 159 | ||
| 158 | do { | 160 | do { |
| 159 | pud_t *pud; | 161 | pud_t *pudp, pud; |
| 160 | pmd_t *pmd; | 162 | pmd_t *pmdp, pmd; |
| 161 | pte_t *pte; | 163 | pte_t *ptep, pte; |
| 162 | 164 | ||
| 163 | if (pgd_none(*pgd) || pgd_bad(*pgd)) | 165 | if (pgd_none(pgd) || pgd_bad(pgd)) |
| 164 | break; | 166 | break; |
| 165 | 167 | ||
| 166 | pud = pud_offset(pgd, addr); | 168 | pudp = pud_offset(pgdp, addr); |
| 167 | pr_cont(", *pud=%016llx", pud_val(*pud)); | 169 | pud = READ_ONCE(*pudp); |
| 168 | if (pud_none(*pud) || pud_bad(*pud)) | 170 | pr_cont(", pud=%016llx", pud_val(pud)); |
| 171 | if (pud_none(pud) || pud_bad(pud)) | ||
| 169 | break; | 172 | break; |
| 170 | 173 | ||
| 171 | pmd = pmd_offset(pud, addr); | 174 | pmdp = pmd_offset(pudp, addr); |
| 172 | pr_cont(", *pmd=%016llx", pmd_val(*pmd)); | 175 | pmd = READ_ONCE(*pmdp); |
| 173 | if (pmd_none(*pmd) || pmd_bad(*pmd)) | 176 | pr_cont(", pmd=%016llx", pmd_val(pmd)); |
| 177 | if (pmd_none(pmd) || pmd_bad(pmd)) | ||
| 174 | break; | 178 | break; |
| 175 | 179 | ||
| 176 | pte = pte_offset_map(pmd, addr); | 180 | ptep = pte_offset_map(pmdp, addr); |
| 177 | pr_cont(", *pte=%016llx", pte_val(*pte)); | 181 | pte = READ_ONCE(*ptep); |
| 178 | pte_unmap(pte); | 182 | pr_cont(", pte=%016llx", pte_val(pte)); |
| 183 | pte_unmap(ptep); | ||
| 179 | } while(0); | 184 | } while(0); |
| 180 | 185 | ||
| 181 | pr_cont("\n"); | 186 | pr_cont("\n"); |
| @@ -196,8 +201,9 @@ int ptep_set_access_flags(struct vm_area_struct *vma, · 14 unmodified lines | |||
| 196 | pte_t entry, int dirty) | 201 | pte_t entry, int dirty) |
| 197 | { | 202 | { |
| 198 | pteval_t old_pteval, pteval; | 203 | pteval_t old_pteval, pteval; |
| 204 | pte_t pte = READ_ONCE(*ptep); | ||
| 199 | 205 | ||
| 200 | if (pte_same(*ptep, entry)) | 206 | if (pte_same(pte, entry)) |
| 201 | return 0; | 207 | return 0; |
| 202 | 208 | ||
| 203 | /* only preserve the access flags and write permission */ | 209 | /* only preserve the access flags and write permission */ |
| @@ -210,7 +216,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, · 6 unmodified lines | |||
| 210 | * (calculated as: a & b == ~(~a | ~b)). | 216 | * (calculated as: a & b == ~(~a | ~b)). |
| 211 | */ | 217 | */ |
| 212 | pte_val(entry) ^= PTE_RDONLY; | 218 | pte_val(entry) ^= PTE_RDONLY; |
| 213 | pteval = READ_ONCE(pte_val(*ptep)); | 219 | pteval = pte_val(pte); |
| 214 | do { | 220 | do { |
| 215 | old_pteval = pteval; | 221 | old_pteval = pteval; |
| 216 | pteval ^= PTE_RDONLY; | 222 | pteval ^= PTE_RDONLY; |
| @@ -54,14 +54,14 @@ static inline pgprot_t pte_pgprot(pte_t pte) · 53 unmodified lines | |||
| 54 | static int find_num_contig(struct mm_struct *mm, unsigned long addr, | 54 | static int find_num_contig(struct mm_struct *mm, unsigned long addr, |
| 55 | pte_t *ptep, size_t *pgsize) | 55 | pte_t *ptep, size_t *pgsize) |
| 56 | { | 56 | { |
| 57 | pgd_t *pgd = pgd_offset(mm, addr); | 57 | pgd_t *pgdp = pgd_offset(mm, addr); |
| 58 | pud_t *pud; | 58 | pud_t *pudp; |
| 59 | pmd_t *pmd; | 59 | pmd_t *pmdp; |
| 60 | 60 | ||
| 61 | *pgsize = PAGE_SIZE; | 61 | *pgsize = PAGE_SIZE; |
| 62 | pud = pud_offset(pgd, addr); | 62 | pudp = pud_offset(pgdp, addr); |
| 63 | pmd = pmd_offset(pud, addr); | 63 | pmdp = pmd_offset(pudp, addr); |
| 64 | if ((pte_t *)pmd == ptep) { | 64 | if ((pte_t *)pmdp == ptep) { |
| 65 | *pgsize = PMD_SIZE; | 65 | *pgsize = PMD_SIZE; |
| 66 | return CONT_PMDS; | 66 | return CONT_PMDS; |
| 67 | } | 67 | } |
| @@ -181,11 +181,8 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 113 unmodified lines | |||
| 181 | 181 | ||
| 182 | clear_flush(mm, addr, ptep, pgsize, ncontig); | 182 | clear_flush(mm, addr, ptep, pgsize, ncontig); |
| 183 | 183 | ||
| 184 | for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) { | 184 | for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) |
| 185 | pr_debug("%s: set pte %p to 0x%llx\n", __func__, ptep, | ||
| 186 | pte_val(pfn_pte(pfn, hugeprot))); | ||
| 187 | set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); | 185 | set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); |
| 188 | } | ||
| 189 | } | 186 | } |
| 190 | 187 | ||
| 191 | void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, | 188 | void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, |
| @@ -203,20 +200,20 @@ void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, · 11 unmodified lines | |||
| 203 | pte_t *huge_pte_alloc(struct mm_struct *mm, | 200 | pte_t *huge_pte_alloc(struct mm_struct *mm, |
| 204 | unsigned long addr, unsigned long sz) | 201 | unsigned long addr, unsigned long sz) |
| 205 | { | 202 | { |
| 206 | pgd_t *pgd; | 203 | pgd_t *pgdp; |
| 207 | pud_t *pud; | 204 | pud_t *pudp; |
| 208 | pte_t *pte = NULL; | 205 | pmd_t *pmdp; |
| 209 | 206 | pte_t *ptep = NULL; | |
| 210 | pr_debug("%s: addr:0x%lx sz:0x%lx\n", __func__, addr, sz); | 207 | |
| 211 | pgd = pgd_offset(mm, addr); | 208 | pgdp = pgd_offset(mm, addr); |
| 212 | pud = pud_alloc(mm, pgd, addr); | 209 | pudp = pud_alloc(mm, pgdp, addr); |
| 213 | if (!pud) | 210 | if (!pudp) |
| 214 | return NULL; | 211 | return NULL; |
| 215 | 212 | ||
| 216 | if (sz == PUD_SIZE) { | 213 | if (sz == PUD_SIZE) { |
| 217 | pte = (pte_t *)pud; | 214 | ptep = (pte_t *)pudp; |
| 218 | } else if (sz == (PAGE_SIZE * CONT_PTES)) { | 215 | } else if (sz == (PAGE_SIZE * CONT_PTES)) { |
| 219 | pmd_t *pmd = pmd_alloc(mm, pud, addr); | 216 | pmdp = pmd_alloc(mm, pudp, addr); |
| 220 | 217 | ||
| 221 | WARN_ON(addr & (sz - 1)); | 218 | WARN_ON(addr & (sz - 1)); |
| 222 | /* | 219 | /* |
| @@ -226,60 +223,55 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, · 3 unmodified lines | |||
| 226 | * will be no pte_unmap() to correspond with this | 223 | * will be no pte_unmap() to correspond with this |
| 227 | * pte_alloc_map(). | 224 | * pte_alloc_map(). |
| 228 | */ | 225 | */ |
| 229 | pte = pte_alloc_map(mm, pmd, addr); | 226 | ptep = pte_alloc_map(mm, pmdp, addr); |
| 230 | } else if (sz == PMD_SIZE) { | 227 | } else if (sz == PMD_SIZE) { |
| 231 | if (IS_ENABLED(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && | 228 | if (IS_ENABLED(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && |
| 232 | pud_none(*pud)) | 229 | pud_none(READ_ONCE(*pudp))) |
| 233 | pte = huge_pmd_share(mm, addr, pud); | 230 | ptep = huge_pmd_share(mm, addr, pudp); |
| 234 | else | 231 | else |
| 235 | pte = (pte_t *)pmd_alloc(mm, pud, addr); | 232 | ptep = (pte_t *)pmd_alloc(mm, pudp, addr); |
| 236 | } else if (sz == (PMD_SIZE * CONT_PMDS)) { | 233 | } else if (sz == (PMD_SIZE * CONT_PMDS)) { |
| 237 | pmd_t *pmd; | 234 | pmdp = pmd_alloc(mm, pudp, addr); |
| 238 | |||
| 239 | pmd = pmd_alloc(mm, pud, addr); | ||
| 240 | WARN_ON(addr & (sz - 1)); | 235 | WARN_ON(addr & (sz - 1)); |
| 241 | return (pte_t *)pmd; | 236 | return (pte_t *)pmdp; |
| 242 | } | 237 | } |
| 243 | 238 | ||
| 244 | pr_debug("%s: addr:0x%lx sz:0x%lx ret pte=%p/0x%llx\n", __func__, addr, | 239 | return ptep; |
| 245 | sz, pte, pte_val(*pte)); | ||
| 246 | return pte; | ||
| 247 | } | 240 | } |
| 248 | 241 | ||
| 249 | pte_t *huge_pte_offset(struct mm_struct *mm, | 242 | pte_t *huge_pte_offset(struct mm_struct *mm, |
| 250 | unsigned long addr, unsigned long sz) | 243 | unsigned long addr, unsigned long sz) |
| 251 | { | 244 | { |
| 252 | pgd_t *pgd; | 245 | pgd_t *pgdp; |
| 253 | pud_t *pud; | 246 | pud_t *pudp, pud; |
| 254 | pmd_t *pmd; | 247 | pmd_t *pmdp, pmd; |
| 255 | 248 | ||
| 256 | pgd = pgd_offset(mm, addr); | 249 | pgdp = pgd_offset(mm, addr); |
| 257 | pr_debug("%s: addr:0x%lx pgd:%p\n", __func__, addr, pgd); | 250 | if (!pgd_present(READ_ONCE(*pgdp))) |
| 258 | if (!pgd_present(*pgd)) | ||
| 259 | return NULL; | 251 | return NULL; |
| 260 | 252 | ||
| 261 | pud = pud_offset(pgd, addr); | 253 | pudp = pud_offset(pgdp, addr); |
| 262 | if (sz != PUD_SIZE && pud_none(*pud)) | 254 | pud = READ_ONCE(*pudp); |
| 255 | if (sz != PUD_SIZE && pud_none(pud)) | ||
| 263 | return NULL; | 256 | return NULL; |
| 264 | /* hugepage or swap? */ | 257 | /* hugepage or swap? */ |
| 265 | if (pud_huge(*pud) || !pud_present(*pud)) | 258 | if (pud_huge(pud) || !pud_present(pud)) |
| 266 | return (pte_t *)pud; | 259 | return (pte_t *)pudp; |
| 267 | /* table; check the next level */ | 260 | /* table; check the next level */ |
| 268 | 261 | ||
| 269 | if (sz == CONT_PMD_SIZE) | 262 | if (sz == CONT_PMD_SIZE) |
| 270 | addr &= CONT_PMD_MASK; | 263 | addr &= CONT_PMD_MASK; |
| 271 | 264 | ||
| 272 | pmd = pmd_offset(pud, addr); | 265 | pmdp = pmd_offset(pudp, addr); |
| 266 | pmd = READ_ONCE(*pmdp); | ||
| 273 | if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) && | 267 | if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) && |
| 274 | pmd_none(*pmd)) | 268 | pmd_none(pmd)) |
| 275 | return NULL; | 269 | return NULL; |
| 276 | if (pmd_huge(*pmd) || !pmd_present(*pmd)) | 270 | if (pmd_huge(pmd) || !pmd_present(pmd)) |
| 277 | return (pte_t *)pmd; | 271 | return (pte_t *)pmdp; |
| 278 | 272 | ||
| 279 | if (sz == CONT_PTE_SIZE) { | 273 | if (sz == CONT_PTE_SIZE) |
| 280 | pte_t *pte = pte_offset_kernel(pmd, (addr & CONT_PTE_MASK)); | 274 | return pte_offset_kernel(pmdp, (addr & CONT_PTE_MASK)); |
| 281 | return pte; | ||
| 282 | } | ||
| 283 | 275 | ||
| 284 | return NULL; | 276 | return NULL; |
| 285 | } | 277 | } |
| @@ -367,7 +359,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, · 81 unmodified lines | |||
| 367 | size_t pgsize; | 359 | size_t pgsize; |
| 368 | pte_t pte; | 360 | pte_t pte; |
| 369 | 361 | ||
| 370 | if (!pte_cont(*ptep)) { | 362 | if (!pte_cont(READ_ONCE(*ptep))) { |
| 371 | ptep_set_wrprotect(mm, addr, ptep); | 363 | ptep_set_wrprotect(mm, addr, ptep); |
| 372 | return; | 364 | return; |
| 373 | } | 365 | } |
| @@ -391,7 +383,7 @@ void huge_ptep_clear_flush(struct vm_area_struct *vma, · 17 unmodified lines | |||
| 391 | size_t pgsize; | 383 | size_t pgsize; |
| 392 | int ncontig; | 384 | int ncontig; |
| 393 | 385 | ||
| 394 | if (!pte_cont(*ptep)) { | 386 | if (!pte_cont(READ_ONCE(*ptep))) { |
| 395 | ptep_clear_flush(vma, addr, ptep); | 387 | ptep_clear_flush(vma, addr, ptep); |
| 396 | return; | 388 | return; |
| 397 | } | 389 | } |
| @@ -44,92 +44,92 @@ static phys_addr_t __init kasan_alloc_zeroed_page(int node) · 43 unmodified lines | |||
| 44 | return __pa(p); | 44 | return __pa(p); |
| 45 | } | 45 | } |
| 46 | 46 | ||
| 47 | static pte_t *__init kasan_pte_offset(pmd_t *pmd, unsigned long addr, int node, | 47 | static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, |
| 48 | bool early) | 48 | bool early) |
| 49 | { | 49 | { |
| 50 | if (pmd_none(*pmd)) { | 50 | if (pmd_none(READ_ONCE(*pmdp))) { |
| 51 | phys_addr_t pte_phys = early ? __pa_symbol(kasan_zero_pte) | 51 | phys_addr_t pte_phys = early ? __pa_symbol(kasan_zero_pte) |
| 52 | : kasan_alloc_zeroed_page(node); | 52 | : kasan_alloc_zeroed_page(node); |
| 53 | __pmd_populate(pmd, pte_phys, PMD_TYPE_TABLE); | 53 | __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); |
| 54 | } | 54 | } |
| 55 | 55 | ||
| 56 | return early ? pte_offset_kimg(pmd, addr) | 56 | return early ? pte_offset_kimg(pmdp, addr) |
| 57 | : pte_offset_kernel(pmd, addr); | 57 | : pte_offset_kernel(pmdp, addr); |
| 58 | } | 58 | } |
| 59 | 59 | ||
| 60 | static pmd_t *__init kasan_pmd_offset(pud_t *pud, unsigned long addr, int node, | 60 | static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, |
| 61 | bool early) | 61 | bool early) |
| 62 | { | 62 | { |
| 63 | if (pud_none(*pud)) { | 63 | if (pud_none(READ_ONCE(*pudp))) { |
| 64 | phys_addr_t pmd_phys = early ? __pa_symbol(kasan_zero_pmd) | 64 | phys_addr_t pmd_phys = early ? __pa_symbol(kasan_zero_pmd) |
| 65 | : kasan_alloc_zeroed_page(node); | 65 | : kasan_alloc_zeroed_page(node); |
| 66 | __pud_populate(pud, pmd_phys, PMD_TYPE_TABLE); | 66 | __pud_populate(pudp, pmd_phys, PMD_TYPE_TABLE); |
| 67 | } | 67 | } |
| 68 | 68 | ||
| 69 | return early ? pmd_offset_kimg(pud, addr) : pmd_offset(pud, addr); | 69 | return early ? pmd_offset_kimg(pudp, addr) : pmd_offset(pudp, addr); |
| 70 | } | 70 | } |
| 71 | 71 | ||
| 72 | static pud_t *__init kasan_pud_offset(pgd_t *pgd, unsigned long addr, int node, | 72 | static pud_t *__init kasan_pud_offset(pgd_t *pgdp, unsigned long addr, int node, |
| 73 | bool early) | 73 | bool early) |
| 74 | { | 74 | { |
| 75 | if (pgd_none(*pgd)) { | 75 | if (pgd_none(READ_ONCE(*pgdp))) { |
| 76 | phys_addr_t pud_phys = early ? __pa_symbol(kasan_zero_pud) | 76 | phys_addr_t pud_phys = early ? __pa_symbol(kasan_zero_pud) |
| 77 | : kasan_alloc_zeroed_page(node); | 77 | : kasan_alloc_zeroed_page(node); |
| 78 | __pgd_populate(pgd, pud_phys, PMD_TYPE_TABLE); | 78 | __pgd_populate(pgdp, pud_phys, PMD_TYPE_TABLE); |
| 79 | } | 79 | } |
| 80 | 80 | ||
| 81 | return early ? pud_offset_kimg(pgd, addr) : pud_offset(pgd, addr); | 81 | return early ? pud_offset_kimg(pgdp, addr) : pud_offset(pgdp, addr); |
| 82 | } | 82 | } |
| 83 | 83 | ||
| 84 | static void __init kasan_pte_populate(pmd_t *pmd, unsigned long addr, | 84 | static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, |
| 85 | unsigned long end, int node, bool early) | 85 | unsigned long end, int node, bool early) |
| 86 | { | 86 | { |
| 87 | unsigned long next; | 87 | unsigned long next; |
| 88 | pte_t *pte = kasan_pte_offset(pmd, addr, node, early); | 88 | pte_t *ptep = kasan_pte_offset(pmdp, addr, node, early); |
| 89 | 89 | ||
| 90 | do { | 90 | do { |
| 91 | phys_addr_t page_phys = early ? __pa_symbol(kasan_zero_page) | 91 | phys_addr_t page_phys = early ? __pa_symbol(kasan_zero_page) |
| 92 | : kasan_alloc_zeroed_page(node); | 92 | : kasan_alloc_zeroed_page(node); |
| 93 | next = addr + PAGE_SIZE; | 93 | next = addr + PAGE_SIZE; |
| 94 | set_pte(pte, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); | 94 | set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); |
| 95 | } while (pte++, addr = next, addr != end && pte_none(*pte)); | 95 | } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); |
| 96 | } | 96 | } |
| 97 | 97 | ||
| 98 | static void __init kasan_pmd_populate(pud_t *pud, unsigned long addr, | 98 | static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, |
| 99 | unsigned long end, int node, bool early) | 99 | unsigned long end, int node, bool early) |
| 100 | { | 100 | { |
| 101 | unsigned long next; | 101 | unsigned long next; |
| 102 | pmd_t *pmd = kasan_pmd_offset(pud, addr, node, early); | 102 | pmd_t *pmdp = kasan_pmd_offset(pudp, addr, node, early); |
| 103 | 103 | ||
| 104 | do { | 104 | do { |
| 105 | next = pmd_addr_end(addr, end); | 105 | next = pmd_addr_end(addr, end); |
| 106 | kasan_pte_populate(pmd, addr, next, node, early); | 106 | kasan_pte_populate(pmdp, addr, next, node, early); |
| 107 | } while (pmd++, addr = next, addr != end && pmd_none(*pmd)); | 107 | } while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp))); |
| 108 | } | 108 | } |
| 109 | 109 | ||
| 110 | static void __init kasan_pud_populate(pgd_t *pgd, unsigned long addr, | 110 | static void __init kasan_pud_populate(pgd_t *pgdp, unsigned long addr, |
| 111 | unsigned long end, int node, bool early) | 111 | unsigned long end, int node, bool early) |
| 112 | { | 112 | { |
| 113 | unsigned long next; | 113 | unsigned long next; |
| 114 | pud_t *pud = kasan_pud_offset(pgd, addr, node, early); | 114 | pud_t *pudp = kasan_pud_offset(pgdp, addr, node, early); |
| 115 | 115 | ||
| 116 | do { | 116 | do { |
| 117 | next = pud_addr_end(addr, end); | 117 | next = pud_addr_end(addr, end); |
| 118 | kasan_pmd_populate(pud, addr, next, node, early); | 118 | kasan_pmd_populate(pudp, addr, next, node, early); |
| 119 | } while (pud++, addr = next, addr != end && pud_none(*pud)); | 119 | } while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp))); |
| 120 | } | 120 | } |
| 121 | 121 | ||
| 122 | static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, | 122 | static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, |
| 123 | int node, bool early) | 123 | int node, bool early) |
| 124 | { | 124 | { |
| 125 | unsigned long next; | 125 | unsigned long next; |
| 126 | pgd_t *pgd; | 126 | pgd_t *pgdp; |
| 127 | 127 | ||
| 128 | pgd = pgd_offset_k(addr); | 128 | pgdp = pgd_offset_k(addr); |
| 129 | do { | 129 | do { |
| 130 | next = pgd_addr_end(addr, end); | 130 | next = pgd_addr_end(addr, end); |
| 131 | kasan_pud_populate(pgd, addr, next, node, early); | 131 | kasan_pud_populate(pgdp, addr, next, node, early); |
| 132 | } while (pgd++, addr = next, addr != end); | 132 | } while (pgdp++, addr = next, addr != end); |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | /* The early shadow maps everything to a single page of zeroes */ | 135 | /* The early shadow maps everything to a single page of zeroes */ |
| @@ -155,14 +155,14 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, · 19 unmodified lines | |||
| 155 | */ | 155 | */ |
| 156 | void __init kasan_copy_shadow(pgd_t *pgdir) | 156 | void __init kasan_copy_shadow(pgd_t *pgdir) |
| 157 | { | 157 | { |
| 158 | pgd_t *pgd, *pgd_new, *pgd_end; | 158 | pgd_t *pgdp, *pgdp_new, *pgdp_end; |
| 159 | 159 | ||
| 160 | pgd = pgd_offset_k(KASAN_SHADOW_START); | 160 | pgdp = pgd_offset_k(KASAN_SHADOW_START); |
| 161 | pgd_end = pgd_offset_k(KASAN_SHADOW_END); | 161 | pgdp_end = pgd_offset_k(KASAN_SHADOW_END); |
| 162 | pgd_new = pgd_offset_raw(pgdir, KASAN_SHADOW_START); | 162 | pgdp_new = pgd_offset_raw(pgdir, KASAN_SHADOW_START); |
| 163 | do { | 163 | do { |
| 164 | set_pgd(pgd_new, *pgd); | 164 | set_pgd(pgdp_new, READ_ONCE(*pgdp)); |
| 165 | } while (pgd++, pgd_new++, pgd != pgd_end); | 165 | } while (pgdp++, pgdp_new++, pgdp != pgdp_end); |
| 166 | } | 166 | } |
| 167 | 167 | ||
| 168 | static void __init clear_pgds(unsigned long start, | 168 | static void __init clear_pgds(unsigned long start, |
| @@ -108,7 +108,7 @@ static bool pgattr_change_is_safe(u64 old, u64 new) · 107 unmodified lines | |||
| 108 | * The following mapping attributes may be updated in live | 108 | * The following mapping attributes may be updated in live |
| 109 | * kernel mappings without the need for break-before-make. | 109 | * kernel mappings without the need for break-before-make. |
| 110 | */ | 110 | */ |
| 111 | static const pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE; | 111 | static const pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; |
| 112 | 112 | ||
| 113 | /* creating or taking down mappings is always safe */ | 113 | /* creating or taking down mappings is always safe */ |
| 114 | if (old == 0 || new == 0) | 114 | if (old == 0 || new == 0) |
| @@ -118,52 +118,55 @@ static bool pgattr_change_is_safe(u64 old, u64 new) · 3 unmodified lines | |||
| 118 | if ((old | new) & PTE_CONT) | 118 | if ((old | new) & PTE_CONT) |
| 119 | return false; | 119 | return false; |
| 120 | 120 | ||
| 121 | /* Transitioning from Global to Non-Global is safe */ | 121 | /* Transitioning from Non-Global to Global is unsafe */ |
| 122 | if (((old ^ new) == PTE_NG) && (new & PTE_NG)) | 122 | if (old & ~new & PTE_NG) |
| 123 | return true; | 123 | return false; |
| 124 | 124 | ||
| 125 | return ((old ^ new) & ~mask) == 0; | 125 | return ((old ^ new) & ~mask) == 0; |
| 126 | } | 126 | } |
| 127 | 127 | ||
| 128 | static void init_pte(pmd_t *pmd, unsigned long addr, unsigned long end, | 128 | static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, |
| 129 | phys_addr_t phys, pgprot_t prot) | 129 | phys_addr_t phys, pgprot_t prot) |
| 130 | { | 130 | { |
| 131 | pte_t *pte; | 131 | pte_t *ptep; |
| 132 | 132 | ||
| 133 | pte = pte_set_fixmap_offset(pmd, addr); | 133 | ptep = pte_set_fixmap_offset(pmdp, addr); |
| 134 | do { | 134 | do { |
| 135 | pte_t old_pte = *pte; | 135 | pte_t old_pte = READ_ONCE(*ptep); |
| 136 | 136 | ||
| 137 | set_pte(pte, pfn_pte(__phys_to_pfn(phys), prot)); | 137 | set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); |
| 138 | 138 | ||
| 139 | /* | 139 | /* |
| 140 | * After the PTE entry has been populated once, we | 140 | * After the PTE entry has been populated once, we |
| 141 | * only allow updates to the permission attributes. | 141 | * only allow updates to the permission attributes. |
| 142 | */ | 142 | */ |
| 143 | BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), pte_val(*pte))); | 143 | BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), |
| 144 | READ_ONCE(pte_val(*ptep)))); | ||
| 144 | 145 | ||
| 145 | phys += PAGE_SIZE; | 146 | phys += PAGE_SIZE; |
| 146 | } while (pte++, addr += PAGE_SIZE, addr != end); | 147 | } while (ptep++, addr += PAGE_SIZE, addr != end); |
| 147 | 148 | ||
| 148 | pte_clear_fixmap(); | 149 | pte_clear_fixmap(); |
| 149 | } | 150 | } |
| 150 | 151 | ||
| 151 | static void alloc_init_cont_pte(pmd_t *pmd, unsigned long addr, | 152 | static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, |
| 152 | unsigned long end, phys_addr_t phys, | 153 | unsigned long end, phys_addr_t phys, |
| 153 | pgprot_t prot, | 154 | pgprot_t prot, |
| 154 | phys_addr_t (*pgtable_alloc)(void), | 155 | phys_addr_t (*pgtable_alloc)(void), |
| 155 | int flags) | 156 | int flags) |
| 156 | { | 157 | { |
| 157 | unsigned long next; | 158 | unsigned long next; |
| 159 | pmd_t pmd = READ_ONCE(*pmdp); | ||
| 158 | 160 | ||
| 159 | BUG_ON(pmd_sect(*pmd)); | 161 | BUG_ON(pmd_sect(pmd)); |
| 160 | if (pmd_none(*pmd)) { | 162 | if (pmd_none(pmd)) { |
| 161 | phys_addr_t pte_phys; | 163 | phys_addr_t pte_phys; |
| 162 | BUG_ON(!pgtable_alloc); | 164 | BUG_ON(!pgtable_alloc); |
| 163 | pte_phys = pgtable_alloc(); | 165 | pte_phys = pgtable_alloc(); |
| 164 | __pmd_populate(pmd, pte_phys, PMD_TYPE_TABLE); | 166 | __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); |
| 167 | pmd = READ_ONCE(*pmdp); | ||
| 165 | } | 168 | } |
| 166 | BUG_ON(pmd_bad(*pmd)); | 169 | BUG_ON(pmd_bad(pmd)); |
| 167 | 170 | ||
| 168 | do { | 171 | do { |
| 169 | pgprot_t __prot = prot; | 172 | pgprot_t __prot = prot; |
| @@ -175,67 +178,69 @@ static void alloc_init_cont_pte(pmd_t *pmd, unsigned long addr, · 5 unmodified lines | |||
| 175 | (flags & NO_CONT_MAPPINGS) == 0) | 178 | (flags & NO_CONT_MAPPINGS) == 0) |
| 176 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); | 179 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); |
| 177 | 180 | ||
| 178 | init_pte(pmd, addr, next, phys, __prot); | 181 | init_pte(pmdp, addr, next, phys, __prot); |
| 179 | 182 | ||
| 180 | phys += next - addr; | 183 | phys += next - addr; |
| 181 | } while (addr = next, addr != end); | 184 | } while (addr = next, addr != end); |
| 182 | } | 185 | } |
| 183 | 186 | ||
| 184 | static void init_pmd(pud_t *pud, unsigned long addr, unsigned long end, | 187 | static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, |
| 185 | phys_addr_t phys, pgprot_t prot, | 188 | phys_addr_t phys, pgprot_t prot, |
| 186 | phys_addr_t (*pgtable_alloc)(void), int flags) | 189 | phys_addr_t (*pgtable_alloc)(void), int flags) |
| 187 | { | 190 | { |
| 188 | unsigned long next; | 191 | unsigned long next; |
| 189 | pmd_t *pmd; | 192 | pmd_t *pmdp; |
| 190 | 193 | ||
| 191 | pmd = pmd_set_fixmap_offset(pud, addr); | 194 | pmdp = pmd_set_fixmap_offset(pudp, addr); |
| 192 | do { | 195 | do { |
| 193 | pmd_t old_pmd = *pmd; | 196 | pmd_t old_pmd = READ_ONCE(*pmdp); |
| 194 | 197 | ||
| 195 | next = pmd_addr_end(addr, end); | 198 | next = pmd_addr_end(addr, end); |
| 196 | 199 | ||
| 197 | /* try section mapping first */ | 200 | /* try section mapping first */ |
| 198 | if (((addr | next | phys) & ~SECTION_MASK) == 0 && | 201 | if (((addr | next | phys) & ~SECTION_MASK) == 0 && |
| 199 | (flags & NO_BLOCK_MAPPINGS) == 0) { | 202 | (flags & NO_BLOCK_MAPPINGS) == 0) { |
| 200 | pmd_set_huge(pmd, phys, prot); | 203 | pmd_set_huge(pmdp, phys, prot); |
| 201 | 204 | ||
| 202 | /* | 205 | /* |
| 203 | * After the PMD entry has been populated once, we | 206 | * After the PMD entry has been populated once, we |
| 204 | * only allow updates to the permission attributes. | 207 | * only allow updates to the permission attributes. |
| 205 | */ | 208 | */ |
| 206 | BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), | 209 | BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), |
| 207 | pmd_val(*pmd))); | 210 | READ_ONCE(pmd_val(*pmdp)))); |
| 208 | } else { | 211 | } else { |
| 209 | alloc_init_cont_pte(pmd, addr, next, phys, prot, | 212 | alloc_init_cont_pte(pmdp, addr, next, phys, prot, |
| 210 | pgtable_alloc, flags); | 213 | pgtable_alloc, flags); |
| 211 | 214 | ||
| 212 | BUG_ON(pmd_val(old_pmd) != 0 && | 215 | BUG_ON(pmd_val(old_pmd) != 0 && |
| 213 | pmd_val(old_pmd) != pmd_val(*pmd)); | 216 | pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); |
| 214 | } | 217 | } |
| 215 | phys += next - addr; | 218 | phys += next - addr; |
| 216 | } while (pmd++, addr = next, addr != end); | 219 | } while (pmdp++, addr = next, addr != end); |
| 217 | 220 | ||
| 218 | pmd_clear_fixmap(); | 221 | pmd_clear_fixmap(); |
| 219 | } | 222 | } |
| 220 | 223 | ||
| 221 | static void alloc_init_cont_pmd(pud_t *pud, unsigned long addr, | 224 | static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, |
| 222 | unsigned long end, phys_addr_t phys, | 225 | unsigned long end, phys_addr_t phys, |
| 223 | pgprot_t prot, | 226 | pgprot_t prot, |
| 224 | phys_addr_t (*pgtable_alloc)(void), int flags) | 227 | phys_addr_t (*pgtable_alloc)(void), int flags) |
| 225 | { | 228 | { |
| 226 | unsigned long next; | 229 | unsigned long next; |
| 230 | pud_t pud = READ_ONCE(*pudp); | ||
| 227 | 231 | ||
| 228 | /* | 232 | /* |
| 229 | * Check for initial section mappings in the pgd/pud. | 233 | * Check for initial section mappings in the pgd/pud. |
| 230 | */ | 234 | */ |
| 231 | BUG_ON(pud_sect(*pud)); | 235 | BUG_ON(pud_sect(pud)); |
| 232 | if (pud_none(*pud)) { | 236 | if (pud_none(pud)) { |
| 233 | phys_addr_t pmd_phys; | 237 | phys_addr_t pmd_phys; |
| 234 | BUG_ON(!pgtable_alloc); | 238 | BUG_ON(!pgtable_alloc); |
| 235 | pmd_phys = pgtable_alloc(); | 239 | pmd_phys = pgtable_alloc(); |
| 236 | __pud_populate(pud, pmd_phys, PUD_TYPE_TABLE); | 240 | __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); |
| 241 | pud = READ_ONCE(*pudp); | ||
| 237 | } | 242 | } |
| 238 | BUG_ON(pud_bad(*pud)); | 243 | BUG_ON(pud_bad(pud)); |
| 239 | 244 | ||
| 240 | do { | 245 | do { |
| 241 | pgprot_t __prot = prot; | 246 | pgprot_t __prot = prot; |
| @@ -247,7 +252,7 @@ static void alloc_init_cont_pmd(pud_t *pud, unsigned long addr, · 5 unmodified lines | |||
| 247 | (flags & NO_CONT_MAPPINGS) == 0) | 252 | (flags & NO_CONT_MAPPINGS) == 0) |
| 248 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); | 253 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); |
| 249 | 254 | ||
| 250 | init_pmd(pud, addr, next, phys, __prot, pgtable_alloc, flags); | 255 | init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); |
| 251 | 256 | ||
| 252 | phys += next - addr; | 257 | phys += next - addr; |
| 253 | } while (addr = next, addr != end); | 258 | } while (addr = next, addr != end); |
| @@ -265,25 +270,27 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, · 11 unmodified lines | |||
| 265 | return true; | 270 | return true; |
| 266 | } | 271 | } |
| 267 | 272 | ||
| 268 | static void alloc_init_pud(pgd_t *pgd, unsigned long addr, unsigned long end, | 273 | static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, |
| 269 | phys_addr_t phys, pgprot_t prot, | 274 | phys_addr_t phys, pgprot_t prot, |
| 270 | phys_addr_t (*pgtable_alloc)(void), | 275 | phys_addr_t (*pgtable_alloc)(void), |
| 271 | int flags) | 276 | int flags) |
| 272 | { | 277 | { |
| 273 | pud_t *pud; | ||
| 274 | unsigned long next; | 278 | unsigned long next; |
| 279 | pud_t *pudp; | ||
| 280 | pgd_t pgd = READ_ONCE(*pgdp); | ||
| 275 | 281 | ||
| 276 | if (pgd_none(*pgd)) { | 282 | if (pgd_none(pgd)) { |
| 277 | phys_addr_t pud_phys; | 283 | phys_addr_t pud_phys; |
| 278 | BUG_ON(!pgtable_alloc); | 284 | BUG_ON(!pgtable_alloc); |
| 279 | pud_phys = pgtable_alloc(); | 285 | pud_phys = pgtable_alloc(); |
| 280 | __pgd_populate(pgd, pud_phys, PUD_TYPE_TABLE); | 286 | __pgd_populate(pgdp, pud_phys, PUD_TYPE_TABLE); |
| 287 | pgd = READ_ONCE(*pgdp); | ||
| 281 | } | 288 | } |
| 282 | BUG_ON(pgd_bad(*pgd)); | 289 | BUG_ON(pgd_bad(pgd)); |
| 283 | 290 | ||
| 284 | pud = pud_set_fixmap_offset(pgd, addr); | 291 | pudp = pud_set_fixmap_offset(pgdp, addr); |
| 285 | do { | 292 | do { |
| 286 | pud_t old_pud = *pud; | 293 | pud_t old_pud = READ_ONCE(*pudp); |
| 287 | 294 | ||
| 288 | next = pud_addr_end(addr, end); | 295 | next = pud_addr_end(addr, end); |
| 289 | 296 | ||
| @@ -292,23 +299,23 @@ static void alloc_init_pud(pgd_t *pgd, unsigned long addr, unsigned long end, · 2 unmodified lines | |||
| 292 | */ | 299 | */ |
| 293 | if (use_1G_block(addr, next, phys) && | 300 | if (use_1G_block(addr, next, phys) && |
| 294 | (flags & NO_BLOCK_MAPPINGS) == 0) { | 301 | (flags & NO_BLOCK_MAPPINGS) == 0) { |
| 295 | pud_set_huge(pud, phys, prot); | 302 | pud_set_huge(pudp, phys, prot); |
| 296 | 303 | ||
| 297 | /* | 304 | /* |
| 298 | * After the PUD entry has been populated once, we | 305 | * After the PUD entry has been populated once, we |
| 299 | * only allow updates to the permission attributes. | 306 | * only allow updates to the permission attributes. |
| 300 | */ | 307 | */ |
| 301 | BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), | 308 | BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), |
| 302 | pud_val(*pud))); | 309 | READ_ONCE(pud_val(*pudp)))); |
| 303 | } else { | 310 | } else { |
| 304 | alloc_init_cont_pmd(pud, addr, next, phys, prot, | 311 | alloc_init_cont_pmd(pudp, addr, next, phys, prot, |
| 305 | pgtable_alloc, flags); | 312 | pgtable_alloc, flags); |
| 306 | 313 | ||
| 307 | BUG_ON(pud_val(old_pud) != 0 && | 314 | BUG_ON(pud_val(old_pud) != 0 && |
| 308 | pud_val(old_pud) != pud_val(*pud)); | 315 | pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); |
| 309 | } | 316 | } |
| 310 | phys += next - addr; | 317 | phys += next - addr; |
| 311 | } while (pud++, addr = next, addr != end); | 318 | } while (pudp++, addr = next, addr != end); |
| 312 | 319 | ||
| 313 | pud_clear_fixmap(); | 320 | pud_clear_fixmap(); |
| 314 | } | 321 | } |
| @@ -320,7 +327,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, · 5 unmodified lines | |||
| 320 | int flags) | 327 | int flags) |
| 321 | { | 328 | { |
| 322 | unsigned long addr, length, end, next; | 329 | unsigned long addr, length, end, next; |
| 323 | pgd_t *pgd = pgd_offset_raw(pgdir, virt); | 330 | pgd_t *pgdp = pgd_offset_raw(pgdir, virt); |
| 324 | 331 | ||
| 325 | /* | 332 | /* |
| 326 | * If the virtual and physical address don't have the same offset | 333 | * If the virtual and physical address don't have the same offset |
| @@ -336,10 +343,10 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, · 9 unmodified lines | |||
| 336 | end = addr + length; | 343 | end = addr + length; |
| 337 | do { | 344 | do { |
| 338 | next = pgd_addr_end(addr, end); | 345 | next = pgd_addr_end(addr, end); |
| 339 | alloc_init_pud(pgd, addr, next, phys, prot, pgtable_alloc, | 346 | alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, |
| 340 | flags); | 347 | flags); |
| 341 | phys += next - addr; | 348 | phys += next - addr; |
| 342 | } while (pgd++, addr = next, addr != end); | 349 | } while (pgdp++, addr = next, addr != end); |
| 343 | } | 350 | } |
| 344 | 351 | ||
| 345 | static phys_addr_t pgd_pgtable_alloc(void) | 352 | static phys_addr_t pgd_pgtable_alloc(void) |
| @@ -401,10 +408,10 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, · 55 unmodified lines | |||
| 401 | flush_tlb_kernel_range(virt, virt + size); | 408 | flush_tlb_kernel_range(virt, virt + size); |
| 402 | } | 409 | } |
| 403 | 410 | ||
| 404 | static void __init __map_memblock(pgd_t *pgd, phys_addr_t start, | 411 | static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, |
| 405 | phys_addr_t end, pgprot_t prot, int flags) | 412 | phys_addr_t end, pgprot_t prot, int flags) |
| 406 | { | 413 | { |
| 407 | __create_pgd_mapping(pgd, start, __phys_to_virt(start), end - start, | 414 | __create_pgd_mapping(pgdp, start, __phys_to_virt(start), end - start, |
| 408 | prot, early_pgtable_alloc, flags); | 415 | prot, early_pgtable_alloc, flags); |
| 409 | } | 416 | } |
| 410 | 417 | ||
| @@ -418,7 +425,7 @@ void __init mark_linear_text_alias_ro(void) · 7 unmodified lines | |||
| 418 | PAGE_KERNEL_RO); | 425 | PAGE_KERNEL_RO); |
| 419 | } | 426 | } |
| 420 | 427 | ||
| 421 | static void __init map_mem(pgd_t *pgd) | 428 | static void __init map_mem(pgd_t *pgdp) |
| 422 | { | 429 | { |
| 423 | phys_addr_t kernel_start = __pa_symbol(_text); | 430 | phys_addr_t kernel_start = __pa_symbol(_text); |
| 424 | phys_addr_t kernel_end = __pa_symbol(__init_begin); | 431 | phys_addr_t kernel_end = __pa_symbol(__init_begin); |
| @@ -451,7 +458,7 @@ static void __init map_mem(pgd_t *pgd) · 26 unmodified lines | |||
| 451 | if (memblock_is_nomap(reg)) | 458 | if (memblock_is_nomap(reg)) |
| 452 | continue; | 459 | continue; |
| 453 | 460 | ||
| 454 | __map_memblock(pgd, start, end, PAGE_KERNEL, flags); | 461 | __map_memblock(pgdp, start, end, PAGE_KERNEL, flags); |
| 455 | } | 462 | } |
| 456 | 463 | ||
| 457 | /* | 464 | /* |
| @@ -464,7 +471,7 @@ static void __init map_mem(pgd_t *pgd) · 6 unmodified lines | |||
| 464 | * Note that contiguous mappings cannot be remapped in this way, | 471 | * Note that contiguous mappings cannot be remapped in this way, |
| 465 | * so we should avoid them here. | 472 | * so we should avoid them here. |
| 466 | */ | 473 | */ |
| 467 | __map_memblock(pgd, kernel_start, kernel_end, | 474 | __map_memblock(pgdp, kernel_start, kernel_end, |
| 468 | PAGE_KERNEL, NO_CONT_MAPPINGS); | 475 | PAGE_KERNEL, NO_CONT_MAPPINGS); |
| 469 | memblock_clear_nomap(kernel_start, kernel_end - kernel_start); | 476 | memblock_clear_nomap(kernel_start, kernel_end - kernel_start); |
| 470 | 477 | ||
| @@ -475,7 +482,7 @@ static void __init map_mem(pgd_t *pgd) · 4 unmodified lines | |||
| 475 | * through /sys/kernel/kexec_crash_size interface. | 482 | * through /sys/kernel/kexec_crash_size interface. |
| 476 | */ | 483 | */ |
| 477 | if (crashk_res.end) { | 484 | if (crashk_res.end) { |
| 478 | __map_memblock(pgd, crashk_res.start, crashk_res.end + 1, | 485 | __map_memblock(pgdp, crashk_res.start, crashk_res.end + 1, |
| 479 | PAGE_KERNEL, | 486 | PAGE_KERNEL, |
| 480 | NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); | 487 | NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); |
| 481 | memblock_clear_nomap(crashk_res.start, | 488 | memblock_clear_nomap(crashk_res.start, |
| @@ -499,7 +506,7 @@ void mark_rodata_ro(void) · 17 unmodified lines | |||
| 499 | debug_checkwx(); | 506 | debug_checkwx(); |
| 500 | } | 507 | } |
| 501 | 508 | ||
| 502 | static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end, | 509 | static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, |
| 503 | pgprot_t prot, struct vm_struct *vma, | 510 | pgprot_t prot, struct vm_struct *vma, |
| 504 | int flags, unsigned long vm_flags) | 511 | int flags, unsigned long vm_flags) |
| 505 | { | 512 | { |
| @@ -509,7 +516,7 @@ static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end, · 3 unmodified lines | |||
| 509 | BUG_ON(!PAGE_ALIGNED(pa_start)); | 516 | BUG_ON(!PAGE_ALIGNED(pa_start)); |
| 510 | BUG_ON(!PAGE_ALIGNED(size)); | 517 | BUG_ON(!PAGE_ALIGNED(size)); |
| 511 | 518 | ||
| 512 | __create_pgd_mapping(pgd, pa_start, (unsigned long)va_start, size, prot, | 519 | __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, |
| 513 | early_pgtable_alloc, flags); | 520 | early_pgtable_alloc, flags); |
| 514 | 521 | ||
| 515 | if (!(vm_flags & VM_NO_GUARD)) | 522 | if (!(vm_flags & VM_NO_GUARD)) |
| @@ -562,7 +569,7 @@ core_initcall(map_entry_trampoline); · 46 unmodified lines | |||
| 562 | /* | 569 | /* |
| 563 | * Create fine-grained mappings for the kernel. | 570 | * Create fine-grained mappings for the kernel. |
| 564 | */ | 571 | */ |
| 565 | static void __init map_kernel(pgd_t *pgd) | 572 | static void __init map_kernel(pgd_t *pgdp) |
| 566 | { | 573 | { |
| 567 | static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, | 574 | static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, |
| 568 | vmlinux_initdata, vmlinux_data; | 575 | vmlinux_initdata, vmlinux_data; |
| @@ -578,24 +585,24 @@ static void __init map_kernel(pgd_t *pgd) · 9 unmodified lines | |||
| 578 | * Only rodata will be remapped with different permissions later on, | 585 | * Only rodata will be remapped with different permissions later on, |
| 579 | * all other segments are allowed to use contiguous mappings. | 586 | * all other segments are allowed to use contiguous mappings. |
| 580 | */ | 587 | */ |
| 581 | map_kernel_segment(pgd, _text, _etext, text_prot, &vmlinux_text, 0, | 588 | map_kernel_segment(pgdp, _text, _etext, text_prot, &vmlinux_text, 0, |
| 582 | VM_NO_GUARD); | 589 | VM_NO_GUARD); |
| 583 | map_kernel_segment(pgd, __start_rodata, __inittext_begin, PAGE_KERNEL, | 590 | map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, |
| 584 | &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); | 591 | &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); |
| 585 | map_kernel_segment(pgd, __inittext_begin, __inittext_end, text_prot, | 592 | map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, |
| 586 | &vmlinux_inittext, 0, VM_NO_GUARD); | 593 | &vmlinux_inittext, 0, VM_NO_GUARD); |
| 587 | map_kernel_segment(pgd, __initdata_begin, __initdata_end, PAGE_KERNEL, | 594 | map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, |
| 588 | &vmlinux_initdata, 0, VM_NO_GUARD); | 595 | &vmlinux_initdata, 0, VM_NO_GUARD); |
| 589 | map_kernel_segment(pgd, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); | 596 | map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); |
| 590 | 597 | ||
| 591 | if (!pgd_val(*pgd_offset_raw(pgd, FIXADDR_START))) { | 598 | if (!READ_ONCE(pgd_val(*pgd_offset_raw(pgdp, FIXADDR_START)))) { |
| 592 | /* | 599 | /* |
| 593 | * The fixmap falls in a separate pgd to the kernel, and doesn't | 600 | * The fixmap falls in a separate pgd to the kernel, and doesn't |
| 594 | * live in the carveout for the swapper_pg_dir. We can simply | 601 | * live in the carveout for the swapper_pg_dir. We can simply |
| 595 | * re-use the existing dir for the fixmap. | 602 | * re-use the existing dir for the fixmap. |
| 596 | */ | 603 | */ |
| 597 | set_pgd(pgd_offset_raw(pgd, FIXADDR_START), | 604 | set_pgd(pgd_offset_raw(pgdp, FIXADDR_START), |
| 598 | *pgd_offset_k(FIXADDR_START)); | 605 | READ_ONCE(*pgd_offset_k(FIXADDR_START))); |
| 599 | } else if (CONFIG_PGTABLE_LEVELS > 3) { | 606 | } else if (CONFIG_PGTABLE_LEVELS > 3) { |
| 600 | /* | 607 | /* |
| 601 | * The fixmap shares its top level pgd entry with the kernel | 608 | * The fixmap shares its top level pgd entry with the kernel |
| @@ -604,14 +611,15 @@ static void __init map_kernel(pgd_t *pgd) · 2 unmodified lines | |||
| 604 | * entry instead. | 611 | * entry instead. |
| 605 | */ | 612 | */ |
| 606 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); | 613 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); |
| 607 | pud_populate(&init_mm, pud_set_fixmap_offset(pgd, FIXADDR_START), | 614 | pud_populate(&init_mm, |
| 615 | pud_set_fixmap_offset(pgdp, FIXADDR_START), | ||
| 608 | lm_alias(bm_pmd)); | 616 | lm_alias(bm_pmd)); |
| 609 | pud_clear_fixmap(); | 617 | pud_clear_fixmap(); |
| 610 | } else { | 618 | } else { |
| 611 | BUG(); | 619 | BUG(); |
| 612 | } | 620 | } |
| 613 | 621 | ||
| 614 | kasan_copy_shadow(pgd); | 622 | kasan_copy_shadow(pgdp); |
| 615 | } | 623 | } |
| 616 | 624 | ||
| 617 | /* | 625 | /* |
| @@ -621,10 +629,10 @@ static void __init map_kernel(pgd_t *pgd) · 3 unmodified lines | |||
| 621 | void __init paging_init(void) | 629 | void __init paging_init(void) |
| 622 | { | 630 | { |
| 623 | phys_addr_t pgd_phys = early_pgtable_alloc(); | 631 | phys_addr_t pgd_phys = early_pgtable_alloc(); |
| 624 | pgd_t *pgd = pgd_set_fixmap(pgd_phys); | 632 | pgd_t *pgdp = pgd_set_fixmap(pgd_phys); |
| 625 | 633 | ||
| 626 | map_kernel(pgd); | 634 | map_kernel(pgdp); |
| 627 | map_mem(pgd); | 635 | map_mem(pgdp); |
| 628 | 636 | ||
| 629 | /* | 637 | /* |
| 630 | * We want to reuse the original swapper_pg_dir so we don't have to | 638 | * We want to reuse the original swapper_pg_dir so we don't have to |
| @@ -635,7 +643,7 @@ void __init paging_init(void) · 4 unmodified lines | |||
| 635 | * To do this we need to go via a temporary pgd. | 643 | * To do this we need to go via a temporary pgd. |
| 636 | */ | 644 | */ |
| 637 | cpu_replace_ttbr1(__va(pgd_phys)); | 645 | cpu_replace_ttbr1(__va(pgd_phys)); |
| 638 | memcpy(swapper_pg_dir, pgd, PGD_SIZE); | 646 | memcpy(swapper_pg_dir, pgdp, PGD_SIZE); |
| 639 | cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); | 647 | cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); |
| 640 | 648 | ||
| 641 | pgd_clear_fixmap(); | 649 | pgd_clear_fixmap(); |
| @@ -655,37 +663,40 @@ void __init paging_init(void) · 13 unmodified lines | |||
| 655 | */ | 663 | */ |
| 656 | int kern_addr_valid(unsigned long addr) | 664 | int kern_addr_valid(unsigned long addr) |
| 657 | { | 665 | { |
| 658 | pgd_t *pgd; | 666 | pgd_t *pgdp; |
| 659 | pud_t *pud; | 667 | pud_t *pudp, pud; |
| 660 | pmd_t *pmd; | 668 | pmd_t *pmdp, pmd; |
| 661 | pte_t *pte; | 669 | pte_t *ptep, pte; |
| 662 | 670 | ||
| 663 | if ((((long)addr) >> VA_BITS) != -1UL) | 671 | if ((((long)addr) >> VA_BITS) != -1UL) |
| 664 | return 0; | 672 | return 0; |
| 665 | 673 | ||
| 666 | pgd = pgd_offset_k(addr); | 674 | pgdp = pgd_offset_k(addr); |
| 667 | if (pgd_none(*pgd)) | 675 | if (pgd_none(READ_ONCE(*pgdp))) |
| 668 | return 0; | 676 | return 0; |
| 669 | 677 | ||
| 670 | pud = pud_offset(pgd, addr); | 678 | pudp = pud_offset(pgdp, addr); |
| 671 | if (pud_none(*pud)) | 679 | pud = READ_ONCE(*pudp); |
| 680 | if (pud_none(pud)) | ||
| 672 | return 0; | 681 | return 0; |
| 673 | 682 | ||
| 674 | if (pud_sect(*pud)) | 683 | if (pud_sect(pud)) |
| 675 | return pfn_valid(pud_pfn(*pud)); | 684 | return pfn_valid(pud_pfn(pud)); |
| 676 | 685 | ||
| 677 | pmd = pmd_offset(pud, addr); | 686 | pmdp = pmd_offset(pudp, addr); |
| 678 | if (pmd_none(*pmd)) | 687 | pmd = READ_ONCE(*pmdp); |
| 688 | if (pmd_none(pmd)) | ||
| 679 | return 0; | 689 | return 0; |
| 680 | 690 | ||
| 681 | if (pmd_sect(*pmd)) | 691 | if (pmd_sect(pmd)) |
| 682 | return pfn_valid(pmd_pfn(*pmd)); | 692 | return pfn_valid(pmd_pfn(pmd)); |
| 683 | 693 | ||
| 684 | pte = pte_offset_kernel(pmd, addr); | 694 | ptep = pte_offset_kernel(pmdp, addr); |
| 685 | if (pte_none(*pte)) | 695 | pte = READ_ONCE(*ptep); |
| 696 | if (pte_none(pte)) | ||
| 686 | return 0; | 697 | return 0; |
| 687 | 698 | ||
| 688 | return pfn_valid(pte_pfn(*pte)); | 699 | return pfn_valid(pte_pfn(pte)); |
| 689 | } | 700 | } |
| 690 | #ifdef CONFIG_SPARSEMEM_VMEMMAP | 701 | #ifdef CONFIG_SPARSEMEM_VMEMMAP |
| 691 | #if !ARM64_SWAPPER_USES_SECTION_MAPS | 702 | #if !ARM64_SWAPPER_USES_SECTION_MAPS |
| @@ -700,32 +711,32 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, · 8 unmodified lines | |||
| 700 | { | 711 | { |
| 701 | unsigned long addr = start; | 712 | unsigned long addr = start; |
| 702 | unsigned long next; | 713 | unsigned long next; |
| 703 | pgd_t *pgd; | 714 | pgd_t *pgdp; |
| 704 | pud_t *pud; | 715 | pud_t *pudp; |
| 705 | pmd_t *pmd; | 716 | pmd_t *pmdp; |
| 706 | 717 | ||
| 707 | do { | 718 | do { |
| 708 | next = pmd_addr_end(addr, end); | 719 | next = pmd_addr_end(addr, end); |
| 709 | 720 | ||
| 710 | pgd = vmemmap_pgd_populate(addr, node); | 721 | pgdp = vmemmap_pgd_populate(addr, node); |
| 711 | if (!pgd) | 722 | if (!pgdp) |
| 712 | return -ENOMEM; | 723 | return -ENOMEM; |
| 713 | 724 | ||
| 714 | pud = vmemmap_pud_populate(pgd, addr, node); | 725 | pudp = vmemmap_pud_populate(pgdp, addr, node); |
| 715 | if (!pud) | 726 | if (!pudp) |
| 716 | return -ENOMEM; | 727 | return -ENOMEM; |
| 717 | 728 | ||
| 718 | pmd = pmd_offset(pud, addr); | 729 | pmdp = pmd_offset(pudp, addr); |
| 719 | if (pmd_none(*pmd)) { | 730 | if (pmd_none(READ_ONCE(*pmdp))) { |
| 720 | void *p = NULL; | 731 | void *p = NULL; |
| 721 | 732 | ||
| 722 | p = vmemmap_alloc_block_buf(PMD_SIZE, node); | 733 | p = vmemmap_alloc_block_buf(PMD_SIZE, node); |
| 723 | if (!p) | 734 | if (!p) |
| 724 | return -ENOMEM; | 735 | return -ENOMEM; |
| 725 | 736 | ||
| 726 | pmd_set_huge(pmd, __pa(p), __pgprot(PROT_SECT_NORMAL)); | 737 | pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL)); |
| 727 | } else | 738 | } else |
| 728 | vmemmap_verify((pte_t *)pmd, node, addr, next); | 739 | vmemmap_verify((pte_t *)pmdp, node, addr, next); |
| 729 | } while (addr = next, addr != end); | 740 | } while (addr = next, addr != end); |
| 730 | 741 | ||
| 731 | return 0; | 742 | return 0; |
| @@ -739,20 +750,22 @@ void vmemmap_free(unsigned long start, unsigned long end, · 7 unmodified lines | |||
| 739 | 750 | ||
| 740 | static inline pud_t * fixmap_pud(unsigned long addr) | 751 | static inline pud_t * fixmap_pud(unsigned long addr) |
| 741 | { | 752 | { |
| 742 | pgd_t *pgd = pgd_offset_k(addr); | 753 | pgd_t *pgdp = pgd_offset_k(addr); |
| 754 | pgd_t pgd = READ_ONCE(*pgdp); | ||
| 743 | 755 | ||
| 744 | BUG_ON(pgd_none(*pgd) || pgd_bad(*pgd)); | 756 | BUG_ON(pgd_none(pgd) || pgd_bad(pgd)); |
| 745 | 757 | ||
| 746 | return pud_offset_kimg(pgd, addr); | 758 | return pud_offset_kimg(pgdp, addr); |
| 747 | } | 759 | } |
| 748 | 760 | ||
| 749 | static inline pmd_t * fixmap_pmd(unsigned long addr) | 761 | static inline pmd_t * fixmap_pmd(unsigned long addr) |
| 750 | { | 762 | { |
| 751 | pud_t *pud = fixmap_pud(addr); | 763 | pud_t *pudp = fixmap_pud(addr); |
| 764 | pud_t pud = READ_ONCE(*pudp); | ||
| 752 | 765 | ||
| 753 | BUG_ON(pud_none(*pud) || pud_bad(*pud)); | 766 | BUG_ON(pud_none(pud) || pud_bad(pud)); |
| 754 | 767 | ||
| 755 | return pmd_offset_kimg(pud, addr); | 768 | return pmd_offset_kimg(pudp, addr); |
| 756 | } | 769 | } |
| 757 | 770 | ||
| 758 | static inline pte_t * fixmap_pte(unsigned long addr) | 771 | static inline pte_t * fixmap_pte(unsigned long addr) |
| @@ -768,30 +781,31 @@ static inline pte_t * fixmap_pte(unsigned long addr) · 9 unmodified lines | |||
| 768 | */ | 781 | */ |
| 769 | void __init early_fixmap_init(void) | 782 | void __init early_fixmap_init(void) |
| 770 | { | 783 | { |
| 771 | pgd_t *pgd; | 784 | pgd_t *pgdp, pgd; |
| 772 | pud_t *pud; | 785 | pud_t *pudp; |
| 773 | pmd_t *pmd; | 786 | pmd_t *pmdp; |
| 774 | unsigned long addr = FIXADDR_START; | 787 | unsigned long addr = FIXADDR_START; |
| 775 | 788 | ||
| 776 | pgd = pgd_offset_k(addr); | 789 | pgdp = pgd_offset_k(addr); |
| 790 | pgd = READ_ONCE(*pgdp); | ||
| 777 | if (CONFIG_PGTABLE_LEVELS > 3 && | 791 | if (CONFIG_PGTABLE_LEVELS > 3 && |
| 778 | !(pgd_none(*pgd) || pgd_page_paddr(*pgd) == __pa_symbol(bm_pud))) { | 792 | !(pgd_none(pgd) || pgd_page_paddr(pgd) == __pa_symbol(bm_pud))) { |
| 779 | /* | 793 | /* |
| 780 | * We only end up here if the kernel mapping and the fixmap | 794 | * We only end up here if the kernel mapping and the fixmap |
| 781 | * share the top level pgd entry, which should only happen on | 795 | * share the top level pgd entry, which should only happen on |
| 782 | * 16k/4 levels configurations. | 796 | * 16k/4 levels configurations. |
| 783 | */ | 797 | */ |
| 784 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); | 798 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); |
| 785 | pud = pud_offset_kimg(pgd, addr); | 799 | pudp = pud_offset_kimg(pgdp, addr); |
| 786 | } else { | 800 | } else { |
| 787 | if (pgd_none(*pgd)) | 801 | if (pgd_none(pgd)) |
| 788 | __pgd_populate(pgd, __pa_symbol(bm_pud), PUD_TYPE_TABLE); | 802 | __pgd_populate(pgdp, __pa_symbol(bm_pud), PUD_TYPE_TABLE); |
| 789 | pud = fixmap_pud(addr); | 803 | pudp = fixmap_pud(addr); |
| 790 | } | 804 | } |
| 791 | if (pud_none(*pud)) | 805 | if (pud_none(READ_ONCE(*pudp))) |
| 792 | __pud_populate(pud, __pa_symbol(bm_pmd), PMD_TYPE_TABLE); | 806 | __pud_populate(pudp, __pa_symbol(bm_pmd), PMD_TYPE_TABLE); |
| 793 | pmd = fixmap_pmd(addr); | 807 | pmdp = fixmap_pmd(addr); |
| 794 | __pmd_populate(pmd, __pa_symbol(bm_pte), PMD_TYPE_TABLE); | 808 | __pmd_populate(pmdp, __pa_symbol(bm_pte), PMD_TYPE_TABLE); |
| 795 | 809 | ||
| 796 | /* | 810 | /* |
| 797 | * The boot-ioremap range spans multiple pmds, for which | 811 | * The boot-ioremap range spans multiple pmds, for which |
| @@ -800,11 +814,11 @@ void __init early_fixmap_init(void) · 2 unmodified lines | |||
| 800 | BUILD_BUG_ON((__fix_to_virt(FIX_BTMAP_BEGIN) >> PMD_SHIFT) | 814 | BUILD_BUG_ON((__fix_to_virt(FIX_BTMAP_BEGIN) >> PMD_SHIFT) |
| 801 | != (__fix_to_virt(FIX_BTMAP_END) >> PMD_SHIFT)); | 815 | != (__fix_to_virt(FIX_BTMAP_END) >> PMD_SHIFT)); |
| 802 | 816 | ||
| 803 | if ((pmd != fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN))) | 817 | if ((pmdp != fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN))) |
| 804 | || pmd != fixmap_pmd(fix_to_virt(FIX_BTMAP_END))) { | 818 | || pmdp != fixmap_pmd(fix_to_virt(FIX_BTMAP_END))) { |
| 805 | WARN_ON(1); | 819 | WARN_ON(1); |
| 806 | pr_warn("pmd %p != %p, %p\n", | 820 | pr_warn("pmdp %p != %p, %p\n", |
| 807 | pmd, fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN)), | 821 | pmdp, fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN)), |
| 808 | fixmap_pmd(fix_to_virt(FIX_BTMAP_END))); | 822 | fixmap_pmd(fix_to_virt(FIX_BTMAP_END))); |
| 809 | pr_warn("fix_to_virt(FIX_BTMAP_BEGIN): %08lx\n", | 823 | pr_warn("fix_to_virt(FIX_BTMAP_BEGIN): %08lx\n", |
| 810 | fix_to_virt(FIX_BTMAP_BEGIN)); | 824 | fix_to_virt(FIX_BTMAP_BEGIN)); |
| @@ -824,16 +838,16 @@ void __set_fixmap(enum fixed_addresses idx, · 13 unmodified lines | |||
| 824 | phys_addr_t phys, pgprot_t flags) | 838 | phys_addr_t phys, pgprot_t flags) |
| 825 | { | 839 | { |
| 826 | unsigned long addr = __fix_to_virt(idx); | 840 | unsigned long addr = __fix_to_virt(idx); |
| 827 | pte_t *pte; | 841 | pte_t *ptep; |
| 828 | 842 | ||
| 829 | BUG_ON(idx <= FIX_HOLE || idx >= __end_of_fixed_addresses); | 843 | BUG_ON(idx <= FIX_HOLE || idx >= __end_of_fixed_addresses); |
| 830 | 844 | ||
| 831 | pte = fixmap_pte(addr); | 845 | ptep = fixmap_pte(addr); |
| 832 | 846 | ||
| 833 | if (pgprot_val(flags)) { | 847 | if (pgprot_val(flags)) { |
| 834 | set_pte(pte, pfn_pte(phys >> PAGE_SHIFT, flags)); | 848 | set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); |
| 835 | } else { | 849 | } else { |
| 836 | pte_clear(&init_mm, addr, pte); | 850 | pte_clear(&init_mm, addr, ptep); |
| 837 | flush_tlb_kernel_range(addr, addr+PAGE_SIZE); | 851 | flush_tlb_kernel_range(addr, addr+PAGE_SIZE); |
| 838 | } | 852 | } |
| 839 | } | 853 | } |
| @@ -915,36 +929,56 @@ int __init arch_ioremap_pmd_supported(void) · 75 unmodified lines | |||
| 915 | return 1; | 929 | return 1; |
| 916 | } | 930 | } |
| 917 | 931 | ||
| 918 | int pud_set_huge(pud_t *pud, phys_addr_t phys, pgprot_t prot) | 932 | int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) |
| 919 | { | 933 | { |
| 920 | pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT | | 934 | pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT | |
| 921 | pgprot_val(mk_sect_prot(prot))); | 935 | pgprot_val(mk_sect_prot(prot))); |
| 936 | |||
| 937 | /* ioremap_page_range doesn't honour BBM */ | ||
| 938 | if (pud_present(READ_ONCE(*pudp))) | ||
| 939 | return 0; | ||
| 940 | |||
| 922 | BUG_ON(phys & ~PUD_MASK); | 941 | BUG_ON(phys & ~PUD_MASK); |
| 923 | set_pud(pud, pfn_pud(__phys_to_pfn(phys), sect_prot)); | 942 | set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot)); |
| 924 | return 1; | 943 | return 1; |
| 925 | } | 944 | } |
| 926 | 945 | ||
| 927 | int pmd_set_huge(pmd_t *pmd, phys_addr_t phys, pgprot_t prot) | 946 | int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) |
| 928 | { | 947 | { |
| 929 | pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT | | 948 | pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT | |
| 930 | pgprot_val(mk_sect_prot(prot))); | 949 | pgprot_val(mk_sect_prot(prot))); |
| 950 | |||
| 951 | /* ioremap_page_range doesn't honour BBM */ | ||
| 952 | if (pmd_present(READ_ONCE(*pmdp))) | ||
| 953 | return 0; | ||
| 954 | |||
| 931 | BUG_ON(phys & ~PMD_MASK); | 955 | BUG_ON(phys & ~PMD_MASK); |
| 932 | set_pmd(pmd, pfn_pmd(__phys_to_pfn(phys), sect_prot)); | 956 | set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot)); |
| 933 | return 1; | 957 | return 1; |
| 934 | } | 958 | } |
| 935 | 959 | ||
| 936 | int pud_clear_huge(pud_t *pud) | 960 | int pud_clear_huge(pud_t *pudp) |
| 937 | { | 961 | { |
| 938 | if (!pud_sect(*pud)) | 962 | if (!pud_sect(READ_ONCE(*pudp))) |
| 939 | return 0; | 963 | return 0; |
| 940 | pud_clear(pud); | 964 | pud_clear(pudp); |
| 941 | return 1; | 965 | return 1; |
| 942 | } | 966 | } |
| 943 | 967 | ||
| 944 | int pmd_clear_huge(pmd_t *pmd) | 968 | int pmd_clear_huge(pmd_t *pmdp) |
| 945 | { | 969 | { |
| 946 | if (!pmd_sect(*pmd)) | 970 | if (!pmd_sect(READ_ONCE(*pmdp))) |
| 947 | return 0; | 971 | return 0; |
| 948 | pmd_clear(pmd); | 972 | pmd_clear(pmdp); |
| 949 | return 1; | 973 | return 1; |
| 950 | } | 974 | } |
| 975 | |||
| 976 | int pud_free_pmd_page(pud_t *pud) | ||
| 977 | { | ||
| 978 | return pud_none(*pud); | ||
| 979 | } | ||
| 980 | |||
| 981 | int pmd_free_pte_page(pmd_t *pmd) | ||
| 982 | { | ||
| 983 | return pmd_none(*pmd); | ||
| 984 | } | ||
| @@ -29,7 +29,7 @@ static int change_page_range(pte_t *ptep, pgtable_t token, unsigned long addr, · 28 unmodified lines | |||
| 29 | void *data) | 29 | void *data) |
| 30 | { | 30 | { |
| 31 | struct page_change_data *cdata = data; | 31 | struct page_change_data *cdata = data; |
| 32 | pte_t pte = *ptep; | 32 | pte_t pte = READ_ONCE(*ptep); |
| 33 | 33 | ||
| 34 | pte = clear_pte_bit(pte, cdata->clear_mask); | 34 | pte = clear_pte_bit(pte, cdata->clear_mask); |
| 35 | pte = set_pte_bit(pte, cdata->set_mask); | 35 | pte = set_pte_bit(pte, cdata->set_mask); |
| @@ -156,30 +156,32 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) · 120 unmodified lines | |||
| 156 | */ | 156 | */ |
| 157 | bool kernel_page_present(struct page *page) | 157 | bool kernel_page_present(struct page *page) |
| 158 | { | 158 | { |
| 159 | pgd_t *pgd; | 159 | pgd_t *pgdp; |
| 160 | pud_t *pud; | 160 | pud_t *pudp, pud; |
| 161 | pmd_t *pmd; | 161 | pmd_t *pmdp, pmd; |
| 162 | pte_t *pte; | 162 | pte_t *ptep; |
| 163 | unsigned long addr = (unsigned long)page_address(page); | 163 | unsigned long addr = (unsigned long)page_address(page); |
| 164 | 164 | ||
| 165 | pgd = pgd_offset_k(addr); | 165 | pgdp = pgd_offset_k(addr); |
| 166 | if (pgd_none(*pgd)) | 166 | if (pgd_none(READ_ONCE(*pgdp))) |
| 167 | return false; | 167 | return false; |
| 168 | 168 | ||
| 169 | pud = pud_offset(pgd, addr); | 169 | pudp = pud_offset(pgdp, addr); |
| 170 | if (pud_none(*pud)) | 170 | pud = READ_ONCE(*pudp); |
| 171 | if (pud_none(pud)) | ||
| 171 | return false; | 172 | return false; |
| 172 | if (pud_sect(*pud)) | 173 | if (pud_sect(pud)) |
| 173 | return true; | 174 | return true; |
| 174 | 175 | ||
| 175 | pmd = pmd_offset(pud, addr); | 176 | pmdp = pmd_offset(pudp, addr); |
| 176 | if (pmd_none(*pmd)) | 177 | pmd = READ_ONCE(*pmdp); |
| 178 | if (pmd_none(pmd)) | ||
| 177 | return false; | 179 | return false; |
| 178 | if (pmd_sect(*pmd)) | 180 | if (pmd_sect(pmd)) |
| 179 | return true; | 181 | return true; |
| 180 | 182 | ||
| 181 | pte = pte_offset_kernel(pmd, addr); | 183 | ptep = pte_offset_kernel(pmdp, addr); |
| 182 | return pte_valid(*pte); | 184 | return pte_valid(READ_ONCE(*ptep)); |
| 183 | } | 185 | } |
| 184 | #endif /* CONFIG_HIBERNATION */ | 186 | #endif /* CONFIG_HIBERNATION */ |
| 185 | #endif /* CONFIG_DEBUG_PAGEALLOC */ | 187 | #endif /* CONFIG_DEBUG_PAGEALLOC */ |
| @@ -205,7 +205,8 @@ ENDPROC(idmap_cpu_replace_ttbr1) · 204 unmodified lines | |||
| 205 | dc cvac, cur_\()\type\()p // Ensure any existing dirty | 205 | dc cvac, cur_\()\type\()p // Ensure any existing dirty |
| 206 | dmb sy // lines are written back before | 206 | dmb sy // lines are written back before |
| 207 | ldr \type, [cur_\()\type\()p] // loading the entry | 207 | ldr \type, [cur_\()\type\()p] // loading the entry |
| 208 | tbz \type, #0, next_\()\type // Skip invalid entries | 208 | tbz \type, #0, skip_\()\type // Skip invalid and |
| 209 | tbnz \type, #11, skip_\()\type // non-global entries | ||
| 209 | .endm | 210 | .endm |
| 210 | 211 | ||
| 211 | .macro __idmap_kpti_put_pgtable_ent_ng, type | 212 | .macro __idmap_kpti_put_pgtable_ent_ng, type |
| @@ -265,8 +266,9 @@ ENTRY(idmap_kpti_install_ng_mappings) · 53 unmodified lines | |||
| 265 | add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) | 266 | add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) |
| 266 | do_pgd: __idmap_kpti_get_pgtable_ent pgd | 267 | do_pgd: __idmap_kpti_get_pgtable_ent pgd |
| 267 | tbnz pgd, #1, walk_puds | 268 | tbnz pgd, #1, walk_puds |
| 268 | __idmap_kpti_put_pgtable_ent_ng pgd | ||
| 269 | next_pgd: | 269 | next_pgd: |
| 270 | __idmap_kpti_put_pgtable_ent_ng pgd | ||
| 271 | skip_pgd: | ||
| 270 | add cur_pgdp, cur_pgdp, #8 | 272 | add cur_pgdp, cur_pgdp, #8 |
| 271 | cmp cur_pgdp, end_pgdp | 273 | cmp cur_pgdp, end_pgdp |
| 272 | b.ne do_pgd | 274 | b.ne do_pgd |
| @@ -294,8 +296,9 @@ walk_puds: · 21 unmodified lines | |||
| 294 | add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) | 296 | add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) |
| 295 | do_pud: __idmap_kpti_get_pgtable_ent pud | 297 | do_pud: __idmap_kpti_get_pgtable_ent pud |
| 296 | tbnz pud, #1, walk_pmds | 298 | tbnz pud, #1, walk_pmds |
| 297 | __idmap_kpti_put_pgtable_ent_ng pud | ||
| 298 | next_pud: | 299 | next_pud: |
| 300 | __idmap_kpti_put_pgtable_ent_ng pud | ||
| 301 | skip_pud: | ||
| 299 | add cur_pudp, cur_pudp, 8 | 302 | add cur_pudp, cur_pudp, 8 |
| 300 | cmp cur_pudp, end_pudp | 303 | cmp cur_pudp, end_pudp |
| 301 | b.ne do_pud | 304 | b.ne do_pud |
| @@ -314,8 +317,9 @@ walk_pmds: · 12 unmodified lines | |||
| 314 | add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) | 317 | add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) |
| 315 | do_pmd: __idmap_kpti_get_pgtable_ent pmd | 318 | do_pmd: __idmap_kpti_get_pgtable_ent pmd |
| 316 | tbnz pmd, #1, walk_ptes | 319 | tbnz pmd, #1, walk_ptes |
| 317 | __idmap_kpti_put_pgtable_ent_ng pmd | ||
| 318 | next_pmd: | 320 | next_pmd: |
| 321 | __idmap_kpti_put_pgtable_ent_ng pmd | ||
| 322 | skip_pmd: | ||
| 319 | add cur_pmdp, cur_pmdp, #8 | 323 | add cur_pmdp, cur_pmdp, #8 |
| 320 | cmp cur_pmdp, end_pmdp | 324 | cmp cur_pmdp, end_pmdp |
| 321 | b.ne do_pmd | 325 | b.ne do_pmd |
| @@ -333,7 +337,7 @@ walk_ptes: · 11 unmodified lines | |||
| 333 | add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) | 337 | add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) |
| 334 | do_pte: __idmap_kpti_get_pgtable_ent pte | 338 | do_pte: __idmap_kpti_get_pgtable_ent pte |
| 335 | __idmap_kpti_put_pgtable_ent_ng pte | 339 | __idmap_kpti_put_pgtable_ent_ng pte |
| 336 | next_pte: | 340 | skip_pte: |
| 337 | add cur_ptep, cur_ptep, #8 | 341 | add cur_ptep, cur_ptep, #8 |
| 338 | cmp cur_ptep, end_ptep | 342 | cmp cur_ptep, end_ptep |
| 339 | b.ne do_pte | 343 | b.ne do_pte |
| @@ -250,8 +250,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 249 unmodified lines | |||
| 250 | off = offsetof(struct bpf_array, map.max_entries); | 250 | off = offsetof(struct bpf_array, map.max_entries); |
| 251 | emit_a64_mov_i64(tmp, off, ctx); | 251 | emit_a64_mov_i64(tmp, off, ctx); |
| 252 | emit(A64_LDR32(tmp, r2, tmp), ctx); | 252 | emit(A64_LDR32(tmp, r2, tmp), ctx); |
| 253 | emit(A64_MOV(0, r3, r3), ctx); | ||
| 253 | emit(A64_CMP(0, r3, tmp), ctx); | 254 | emit(A64_CMP(0, r3, tmp), ctx); |
| 254 | emit(A64_B_(A64_COND_GE, jmp_offset), ctx); | 255 | emit(A64_B_(A64_COND_CS, jmp_offset), ctx); |
| 255 | 256 | ||
| 256 | /* if (tail_call_cnt > MAX_TAIL_CALL_CNT) | 257 | /* if (tail_call_cnt > MAX_TAIL_CALL_CNT) |
| 257 | * goto out; | 258 | * goto out; |
| @@ -259,7 +260,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 1 unmodified line | |||
| 259 | */ | 260 | */ |
| 260 | emit_a64_mov_i64(tmp, MAX_TAIL_CALL_CNT, ctx); | 261 | emit_a64_mov_i64(tmp, MAX_TAIL_CALL_CNT, ctx); |
| 261 | emit(A64_CMP(1, tcc, tmp), ctx); | 262 | emit(A64_CMP(1, tcc, tmp), ctx); |
| 262 | emit(A64_B_(A64_COND_GT, jmp_offset), ctx); | 263 | emit(A64_B_(A64_COND_HI, jmp_offset), ctx); |
| 263 | emit(A64_ADD_I(1, tcc, tcc, 1), ctx); | 264 | emit(A64_ADD_I(1, tcc, tcc, 1), ctx); |
| 264 | 265 | ||
| 265 | /* prog = array->ptrs[index]; | 266 | /* prog = array->ptrs[index]; |
| @@ -44,18 +44,25 @@ struct bug_frame { · 43 unmodified lines | |||
| 44 | * not be used like this with newer versions of gcc. | 44 | * not be used like this with newer versions of gcc. |
| 45 | */ | 45 | */ |
| 46 | #define BUG() \ | 46 | #define BUG() \ |
| 47 | do { \ | ||
| 47 | __asm__ __volatile__ ("clear.d [" __stringify(BUG_MAGIC) "]\n\t"\ | 48 | __asm__ __volatile__ ("clear.d [" __stringify(BUG_MAGIC) "]\n\t"\ |
| 48 | "movu.w " __stringify(__LINE__) ",$r0\n\t"\ | 49 | "movu.w " __stringify(__LINE__) ",$r0\n\t"\ |
| 49 | "jump 0f\n\t" \ | 50 | "jump 0f\n\t" \ |
| 50 | ".section .rodata\n" \ | 51 | ".section .rodata\n" \ |
| 51 | "0:\t.string \"" __FILE__ "\"\n\t" \ | 52 | "0:\t.string \"" __FILE__ "\"\n\t" \ |
| 52 | ".previous") | 53 | ".previous"); \ |
| 54 | unreachable(); \ | ||
| 55 | } while (0) | ||
| 53 | #endif | 56 | #endif |
| 54 | 57 | ||
| 55 | #else | 58 | #else |
| 56 | 59 | ||
| 57 | /* This just causes an oops. */ | 60 | /* This just causes an oops. */ |
| 58 | #define BUG() (*(int *)0 = 0) | 61 | #define BUG() \ |
| 62 | do { \ | ||
| 63 | barrier_before_unreachable(); \ | ||
| 64 | __builtin_trap(); \ | ||
| 65 | } while (0) | ||
| 59 | 66 | ||
| 60 | #endif | 67 | #endif |
| 61 | 68 | ||
| @@ -2,7 +2,6 @@ · 1 unmodified line | |||
| 2 | #ifndef __H8300_BYTEORDER_H__ | 2 | #ifndef __H8300_BYTEORDER_H__ |
| 3 | #define __H8300_BYTEORDER_H__ | 3 | #define __H8300_BYTEORDER_H__ |
| 4 | 4 | ||
| 5 | #define __BIG_ENDIAN __ORDER_BIG_ENDIAN__ | ||
| 6 | #include <linux/byteorder/big_endian.h> | 5 | #include <linux/byteorder/big_endian.h> |
| 7 | 6 | ||
| 8 | #endif | 7 | #endif |
| @@ -66,38 +66,35 @@ ATOMIC_OPS(add, +) · 65 unmodified lines | |||
| 66 | ATOMIC_OPS(sub, -) | 66 | ATOMIC_OPS(sub, -) |
| 67 | 67 | ||
| 68 | #ifdef __OPTIMIZE__ | 68 | #ifdef __OPTIMIZE__ |
| 69 | #define __ia64_atomic_const(i) __builtin_constant_p(i) ? \ | 69 | #define __ia64_atomic_const(i) \ |
| 70 | static const int __ia64_atomic_p = __builtin_constant_p(i) ? \ | ||
| 70 | ((i) == 1 || (i) == 4 || (i) == 8 || (i) == 16 || \ | 71 | ((i) == 1 || (i) == 4 || (i) == 8 || (i) == 16 || \ |
| 71 | (i) == -1 || (i) == -4 || (i) == -8 || (i) == -16) : 0 | 72 | (i) == -1 || (i) == -4 || (i) == -8 || (i) == -16) : 0;\ |
| 73 | __ia64_atomic_p | ||
| 74 | #else | ||
| 75 | #define __ia64_atomic_const(i) 0 | ||
| 76 | #endif | ||
| 72 | 77 | ||
| 73 | #define atomic_add_return(i, v) \ | 78 | #define atomic_add_return(i,v) \ |
| 74 | ({ \ | 79 | ({ \ |
| 75 | int __i = (i); \ | 80 | int __ia64_aar_i = (i); \ |
| 76 | static const int __ia64_atomic_p = __ia64_atomic_const(i); \ | 81 | __ia64_atomic_const(i) \ |
| 77 | __ia64_atomic_p ? ia64_fetch_and_add(__i, &(v)->counter) : \ | 82 | ? ia64_fetch_and_add(__ia64_aar_i, &(v)->counter) \ |
| 78 | ia64_atomic_add(__i, v); \ | 83 | : ia64_atomic_add(__ia64_aar_i, v); \ |
| 79 | }) | 84 | }) |
| 80 | 85 | ||
| 81 | #define atomic_sub_return(i, v) \ | 86 | #define atomic_sub_return(i,v) \ |
| 82 | ({ \ | 87 | ({ \ |
| 83 | int __i = (i); \ | 88 | int __ia64_asr_i = (i); \ |
| 84 | static const int __ia64_atomic_p = __ia64_atomic_const(i); \ | 89 | __ia64_atomic_const(i) \ |
| 85 | __ia64_atomic_p ? ia64_fetch_and_add(-__i, &(v)->counter) : \ | 90 | ? ia64_fetch_and_add(-__ia64_asr_i, &(v)->counter) \ |
| 86 | ia64_atomic_sub(__i, v); \ | 91 | : ia64_atomic_sub(__ia64_asr_i, v); \ |
| 87 | }) | 92 | }) |
| 88 | #else | ||
| 89 | #define atomic_add_return(i, v) ia64_atomic_add(i, v) | ||
| 90 | #define atomic_sub_return(i, v) ia64_atomic_sub(i, v) | ||
| 91 | #endif | ||
| 92 | 93 | ||
| 93 | #define atomic_fetch_add(i,v) \ | 94 | #define atomic_fetch_add(i,v) \ |
| 94 | ({ \ | 95 | ({ \ |
| 95 | int __ia64_aar_i = (i); \ | 96 | int __ia64_aar_i = (i); \ |
| 96 | (__builtin_constant_p(i) \ | 97 | __ia64_atomic_const(i) \ |
| 97 | && ( (__ia64_aar_i == 1) || (__ia64_aar_i == 4) \ | ||
| 98 | || (__ia64_aar_i == 8) || (__ia64_aar_i == 16) \ | ||
| 99 | || (__ia64_aar_i == -1) || (__ia64_aar_i == -4) \ | ||
| 100 | || (__ia64_aar_i == -8) || (__ia64_aar_i == -16))) \ | ||
| 101 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ | 98 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ |
| 102 | : ia64_atomic_fetch_add(__ia64_aar_i, v); \ | 99 | : ia64_atomic_fetch_add(__ia64_aar_i, v); \ |
| 103 | }) | 100 | }) |
| @@ -105,11 +102,7 @@ ATOMIC_OPS(sub, -) · 1 unmodified line | |||
| 105 | #define atomic_fetch_sub(i,v) \ | 102 | #define atomic_fetch_sub(i,v) \ |
| 106 | ({ \ | 103 | ({ \ |
| 107 | int __ia64_asr_i = (i); \ | 104 | int __ia64_asr_i = (i); \ |
| 108 | (__builtin_constant_p(i) \ | 105 | __ia64_atomic_const(i) \ |
| 109 | && ( (__ia64_asr_i == 1) || (__ia64_asr_i == 4) \ | ||
| 110 | || (__ia64_asr_i == 8) || (__ia64_asr_i == 16) \ | ||
| 111 | || (__ia64_asr_i == -1) || (__ia64_asr_i == -4) \ | ||
| 112 | || (__ia64_asr_i == -8) || (__ia64_asr_i == -16))) \ | ||
| 113 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ | 106 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ |
| 114 | : ia64_atomic_fetch_sub(__ia64_asr_i, v); \ | 107 | : ia64_atomic_fetch_sub(__ia64_asr_i, v); \ |
| 115 | }) | 108 | }) |
| @@ -170,11 +163,7 @@ ATOMIC64_OPS(sub, -) · 54 unmodified lines | |||
| 170 | #define atomic64_add_return(i,v) \ | 163 | #define atomic64_add_return(i,v) \ |
| 171 | ({ \ | 164 | ({ \ |
| 172 | long __ia64_aar_i = (i); \ | 165 | long __ia64_aar_i = (i); \ |
| 173 | (__builtin_constant_p(i) \ | 166 | __ia64_atomic_const(i) \ |
| 174 | && ( (__ia64_aar_i == 1) || (__ia64_aar_i == 4) \ | ||
| 175 | || (__ia64_aar_i == 8) || (__ia64_aar_i == 16) \ | ||
| 176 | || (__ia64_aar_i == -1) || (__ia64_aar_i == -4) \ | ||
| 177 | || (__ia64_aar_i == -8) || (__ia64_aar_i == -16))) \ | ||
| 178 | ? ia64_fetch_and_add(__ia64_aar_i, &(v)->counter) \ | 167 | ? ia64_fetch_and_add(__ia64_aar_i, &(v)->counter) \ |
| 179 | : ia64_atomic64_add(__ia64_aar_i, v); \ | 168 | : ia64_atomic64_add(__ia64_aar_i, v); \ |
| 180 | }) | 169 | }) |
| @@ -182,11 +171,7 @@ ATOMIC64_OPS(sub, -) · 1 unmodified line | |||
| 182 | #define atomic64_sub_return(i,v) \ | 171 | #define atomic64_sub_return(i,v) \ |
| 183 | ({ \ | 172 | ({ \ |
| 184 | long __ia64_asr_i = (i); \ | 173 | long __ia64_asr_i = (i); \ |
| 185 | (__builtin_constant_p(i) \ | 174 | __ia64_atomic_const(i) \ |
| 186 | && ( (__ia64_asr_i == 1) || (__ia64_asr_i == 4) \ | ||
| 187 | || (__ia64_asr_i == 8) || (__ia64_asr_i == 16) \ | ||
| 188 | || (__ia64_asr_i == -1) || (__ia64_asr_i == -4) \ | ||
| 189 | || (__ia64_asr_i == -8) || (__ia64_asr_i == -16))) \ | ||
| 190 | ? ia64_fetch_and_add(-__ia64_asr_i, &(v)->counter) \ | 175 | ? ia64_fetch_and_add(-__ia64_asr_i, &(v)->counter) \ |
| 191 | : ia64_atomic64_sub(__ia64_asr_i, v); \ | 176 | : ia64_atomic64_sub(__ia64_asr_i, v); \ |
| 192 | }) | 177 | }) |
| @@ -194,11 +179,7 @@ ATOMIC64_OPS(sub, -) · 1 unmodified line | |||
| 194 | #define atomic64_fetch_add(i,v) \ | 179 | #define atomic64_fetch_add(i,v) \ |
| 195 | ({ \ | 180 | ({ \ |
| 196 | long __ia64_aar_i = (i); \ | 181 | long __ia64_aar_i = (i); \ |
| 197 | (__builtin_constant_p(i) \ | 182 | __ia64_atomic_const(i) \ |
| 198 | && ( (__ia64_aar_i == 1) || (__ia64_aar_i == 4) \ | ||
| 199 | || (__ia64_aar_i == 8) || (__ia64_aar_i == 16) \ | ||
| 200 | || (__ia64_aar_i == -1) || (__ia64_aar_i == -4) \ | ||
| 201 | || (__ia64_aar_i == -8) || (__ia64_aar_i == -16))) \ | ||
| 202 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ | 183 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ |
| 203 | : ia64_atomic64_fetch_add(__ia64_aar_i, v); \ | 184 | : ia64_atomic64_fetch_add(__ia64_aar_i, v); \ |
| 204 | }) | 185 | }) |
| @@ -206,11 +187,7 @@ ATOMIC64_OPS(sub, -) · 1 unmodified line | |||
| 206 | #define atomic64_fetch_sub(i,v) \ | 187 | #define atomic64_fetch_sub(i,v) \ |
| 207 | ({ \ | 188 | ({ \ |
| 208 | long __ia64_asr_i = (i); \ | 189 | long __ia64_asr_i = (i); \ |
| 209 | (__builtin_constant_p(i) \ | 190 | __ia64_atomic_const(i) \ |
| 210 | && ( (__ia64_asr_i == 1) || (__ia64_asr_i == 4) \ | ||
| 211 | || (__ia64_asr_i == 8) || (__ia64_asr_i == 16) \ | ||
| 212 | || (__ia64_asr_i == -1) || (__ia64_asr_i == -4) \ | ||
| 213 | || (__ia64_asr_i == -8) || (__ia64_asr_i == -16))) \ | ||
| 214 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ | 191 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ |
| 215 | : ia64_atomic64_fetch_sub(__ia64_asr_i, v); \ | 192 | : ia64_atomic64_fetch_sub(__ia64_asr_i, v); \ |
| 216 | }) | 193 | }) |
| @@ -4,7 +4,11 @@ · 3 unmodified lines | |||
| 4 | 4 | ||
| 5 | #ifdef CONFIG_BUG | 5 | #ifdef CONFIG_BUG |
| 6 | #define ia64_abort() __builtin_trap() | 6 | #define ia64_abort() __builtin_trap() |
| 7 | #define BUG() do { printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); ia64_abort(); } while (0) | 7 | #define BUG() do { \ |
| 8 | printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ | ||
| 9 | barrier_before_unreachable(); \ | ||
| 10 | ia64_abort(); \ | ||
| 11 | } while (0) | ||
| 8 | 12 | ||
| 9 | /* should this BUG be made generic? */ | 13 | /* should this BUG be made generic? */ |
| 10 | #define HAVE_ARCH_BUG | 14 | #define HAVE_ARCH_BUG |
| @@ -41,7 +41,6 @@ ifneq ($(CONFIG_IA64_ESI),) · 40 unmodified lines | |||
| 41 | obj-y += esi_stub.o # must be in kernel proper | 41 | obj-y += esi_stub.o # must be in kernel proper |
| 42 | endif | 42 | endif |
| 43 | obj-$(CONFIG_INTEL_IOMMU) += pci-dma.o | 43 | obj-$(CONFIG_INTEL_IOMMU) += pci-dma.o |
| 44 | obj-$(CONFIG_SWIOTLB) += pci-swiotlb.o | ||
| 45 | 44 | ||
| 46 | obj-$(CONFIG_BINFMT_ELF) += elfcore.o | 45 | obj-$(CONFIG_BINFMT_ELF) += elfcore.o |
| 47 | 46 | ||
| @@ -117,7 +117,7 @@ store_call_start(struct device *dev, struct device_attribute *attr, · 116 unmodified lines | |||
| 117 | 117 | ||
| 118 | #ifdef ERR_INJ_DEBUG | 118 | #ifdef ERR_INJ_DEBUG |
| 119 | printk(KERN_DEBUG "Returns: status=%d,\n", (int)status[cpu]); | 119 | printk(KERN_DEBUG "Returns: status=%d,\n", (int)status[cpu]); |
| 120 | printk(KERN_DEBUG "capapbilities=%lx,\n", capabilities[cpu]); | 120 | printk(KERN_DEBUG "capabilities=%lx,\n", capabilities[cpu]); |
| 121 | printk(KERN_DEBUG "resources=%lx\n", resources[cpu]); | 121 | printk(KERN_DEBUG "resources=%lx\n", resources[cpu]); |
| 122 | #endif | 122 | #endif |
| 123 | return size; | 123 | return size; |
| @@ -142,7 +142,7 @@ store_virtual_to_phys(struct device *dev, struct device_attribute *attr, · 18 unmodified lines | |||
| 142 | u64 virt_addr=simple_strtoull(buf, NULL, 16); | 142 | u64 virt_addr=simple_strtoull(buf, NULL, 16); |
| 143 | int ret; | 143 | int ret; |
| 144 | 144 | ||
| 145 | ret = get_user_pages(virt_addr, 1, FOLL_WRITE, NULL, NULL); | 145 | ret = get_user_pages_fast(virt_addr, 1, FOLL_WRITE, NULL); |
| 146 | if (ret<=0) { | 146 | if (ret<=0) { |
| 147 | #ifdef ERR_INJ_DEBUG | 147 | #ifdef ERR_INJ_DEBUG |
| 148 | printk("Virtual address %lx is not existing.\n",virt_addr); | 148 | printk("Virtual address %lx is not existing.\n",virt_addr); |
| @@ -16,7 +16,7 @@ import re · 15 unmodified lines | |||
| 16 | import sys | 16 | import sys |
| 17 | 17 | ||
| 18 | if len(sys.argv) != 2: | 18 | if len(sys.argv) != 2: |
| 19 | print "Usage: %s FILE" % sys.argv[0] | 19 | print("Usage: %s FILE" % sys.argv[0]) |
| 20 | sys.exit(2) | 20 | sys.exit(2) |
| 21 | 21 | ||
| 22 | readelf = os.getenv("READELF", "readelf") | 22 | readelf = os.getenv("READELF", "readelf") |
| @@ -29,7 +29,7 @@ def check_func (func, slots, rlen_sum): · 6 unmodified lines | |||
| 29 | global num_errors | 29 | global num_errors |
| 30 | num_errors += 1 | 30 | num_errors += 1 |
| 31 | if not func: func = "[%#x-%#x]" % (start, end) | 31 | if not func: func = "[%#x-%#x]" % (start, end) |
| 32 | print "ERROR: %s: %lu slots, total region length = %lu" % (func, slots, rlen_sum) | 32 | print("ERROR: %s: %lu slots, total region length = %lu" % (func, slots, rlen_sum)) |
| 33 | return | 33 | return |
| 34 | 34 | ||
| 35 | num_funcs = 0 | 35 | num_funcs = 0 |
| @@ -43,23 +43,23 @@ for line in os.popen("%s -u %s" % (readelf, sys.argv[1])): · 7 unmodified lines | |||
| 43 | check_func(func, slots, rlen_sum) | 43 | check_func(func, slots, rlen_sum) |
| 44 | 44 | ||
| 45 | func = m.group(1) | 45 | func = m.group(1) |
| 46 | start = long(m.group(2), 16) | 46 | start = int(m.group(2), 16) |
| 47 | end = long(m.group(3), 16) | 47 | end = int(m.group(3), 16) |
| 48 | slots = 3 * (end - start) / 16 | 48 | slots = 3 * (end - start) / 16 |
| 49 | rlen_sum = 0L | 49 | rlen_sum = 0 |
| 50 | num_funcs += 1 | 50 | num_funcs += 1 |
| 51 | else: | 51 | else: |
| 52 | m = rlen_pattern.match(line) | 52 | m = rlen_pattern.match(line) |
| 53 | if m: | 53 | if m: |
| 54 | rlen_sum += long(m.group(1)) | 54 | rlen_sum += int(m.group(1)) |
| 55 | check_func(func, slots, rlen_sum) | 55 | check_func(func, slots, rlen_sum) |
| 56 | 56 | ||
| 57 | if num_errors == 0: | 57 | if num_errors == 0: |
| 58 | print "No errors detected in %u functions." % num_funcs | 58 | print("No errors detected in %u functions." % num_funcs) |
| 59 | else: | 59 | else: |
| 60 | if num_errors > 1: | 60 | if num_errors > 1: |
| 61 | err="errors" | 61 | err="errors" |
| 62 | else: | 62 | else: |
| 63 | err="error" | 63 | err="error" |
| 64 | print "%u %s detected in %u functions." % (num_errors, err, num_funcs) | 64 | print("%u %s detected in %u functions." % (num_errors, err, num_funcs)) |
| 65 | sys.exit(1) | 65 | sys.exit(1) |
| @@ -8,16 +8,19 @@ · 7 unmodified lines | |||
| 8 | #ifndef CONFIG_SUN3 | 8 | #ifndef CONFIG_SUN3 |
| 9 | #define BUG() do { \ | 9 | #define BUG() do { \ |
| 10 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ | 10 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ |
| 11 | barrier_before_unreachable(); \ | ||
| 11 | __builtin_trap(); \ | 12 | __builtin_trap(); \ |
| 12 | } while (0) | 13 | } while (0) |
| 13 | #else | 14 | #else |
| 14 | #define BUG() do { \ | 15 | #define BUG() do { \ |
| 15 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ | 16 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ |
| 17 | barrier_before_unreachable(); \ | ||
| 16 | panic("BUG!"); \ | 18 | panic("BUG!"); \ |
| 17 | } while (0) | 19 | } while (0) |
| 18 | #endif | 20 | #endif |
| 19 | #else | 21 | #else |
| 20 | #define BUG() do { \ | 22 | #define BUG() do { \ |
| 23 | barrier_before_unreachable(); \ | ||
| 21 | __builtin_trap(); \ | 24 | __builtin_trap(); \ |
| 22 | } while (0) | 25 | } while (0) |
| 23 | #endif | 26 | #endif |
| @@ -24,6 +24,7 @@ config MICROBLAZE · 23 unmodified lines | |||
| 24 | select HAVE_FTRACE_MCOUNT_RECORD | 24 | select HAVE_FTRACE_MCOUNT_RECORD |
| 25 | select HAVE_FUNCTION_GRAPH_TRACER | 25 | select HAVE_FUNCTION_GRAPH_TRACER |
| 26 | select HAVE_FUNCTION_TRACER | 26 | select HAVE_FUNCTION_TRACER |
| 27 | select NO_BOOTMEM | ||
| 27 | select HAVE_MEMBLOCK | 28 | select HAVE_MEMBLOCK |
| 28 | select HAVE_MEMBLOCK_NODE_MAP | 29 | select HAVE_MEMBLOCK_NODE_MAP |
| 29 | select HAVE_OPROFILE | 30 | select HAVE_OPROFILE |
| @@ -8,7 +8,6 @@ menu "Platform options" · 7 unmodified lines | |||
| 8 | 8 | ||
| 9 | config OPT_LIB_FUNCTION | 9 | config OPT_LIB_FUNCTION |
| 10 | bool "Optimalized lib function" | 10 | bool "Optimalized lib function" |
| 11 | depends on CPU_LITTLE_ENDIAN | ||
| 12 | default y | 11 | default y |
| 13 | help | 12 | help |
| 14 | Allows turn on optimalized library function (memcpy and memmove). | 13 | Allows turn on optimalized library function (memcpy and memmove). |
| @@ -21,6 +20,7 @@ config OPT_LIB_FUNCTION · 6 unmodified lines | |||
| 21 | config OPT_LIB_ASM | 20 | config OPT_LIB_ASM |
| 22 | bool "Optimalized lib function ASM" | 21 | bool "Optimalized lib function ASM" |
| 23 | depends on OPT_LIB_FUNCTION && (XILINX_MICROBLAZE0_USE_BARREL = 1) | 22 | depends on OPT_LIB_FUNCTION && (XILINX_MICROBLAZE0_USE_BARREL = 1) |
| 23 | depends on CPU_BIG_ENDIAN | ||
| 24 | default n | 24 | default n |
| 25 | help | 25 | help |
| 26 | Allows turn on optimalized library function (memcpy and memmove). | 26 | Allows turn on optimalized library function (memcpy and memmove). |
| @@ -44,7 +44,6 @@ void machine_shutdown(void); · 43 unmodified lines | |||
| 44 | void machine_halt(void); | 44 | void machine_halt(void); |
| 45 | void machine_power_off(void); | 45 | void machine_power_off(void); |
| 46 | 46 | ||
| 47 | extern void *alloc_maybe_bootmem(size_t size, gfp_t mask); | ||
| 48 | extern void *zalloc_maybe_bootmem(size_t size, gfp_t mask); | 47 | extern void *zalloc_maybe_bootmem(size_t size, gfp_t mask); |
| 49 | 48 | ||
| 50 | # endif /* __ASSEMBLY__ */ | 49 | # endif /* __ASSEMBLY__ */ |
| @@ -29,10 +29,6 @@ · 28 unmodified lines | |||
| 29 | * between mem locations with size of xfer spec'd in bytes | 29 | * between mem locations with size of xfer spec'd in bytes |
| 30 | */ | 30 | */ |
| 31 | 31 | ||
| 32 | #ifdef __MICROBLAZEEL__ | ||
| 33 | #error Microblaze LE not support ASM optimized lib func. Disable OPT_LIB_ASM. | ||
| 34 | #endif | ||
| 35 | |||
| 36 | #include <linux/linkage.h> | 32 | #include <linux/linkage.h> |
| 37 | .text | 33 | .text |
| 38 | .globl memcpy | 34 | .globl memcpy |
| @@ -32,9 +32,6 @@ int mem_init_done; · 31 unmodified lines | |||
| 32 | #ifndef CONFIG_MMU | 32 | #ifndef CONFIG_MMU |
| 33 | unsigned int __page_offset; | 33 | unsigned int __page_offset; |
| 34 | EXPORT_SYMBOL(__page_offset); | 34 | EXPORT_SYMBOL(__page_offset); |
| 35 | |||
| 36 | #else | ||
| 37 | static int init_bootmem_done; | ||
| 38 | #endif /* CONFIG_MMU */ | 35 | #endif /* CONFIG_MMU */ |
| 39 | 36 | ||
| 40 | char *klimit = _end; | 37 | char *klimit = _end; |
| @@ -117,7 +114,6 @@ static void __init paging_init(void) · 76 unmodified lines | |||
| 117 | 114 | ||
| 118 | void __init setup_memory(void) | 115 | void __init setup_memory(void) |
| 119 | { | 116 | { |
| 120 | unsigned long map_size; | ||
| 121 | struct memblock_region *reg; | 117 | struct memblock_region *reg; |
| 122 | 118 | ||
| 123 | #ifndef CONFIG_MMU | 119 | #ifndef CONFIG_MMU |
| @@ -174,17 +170,6 @@ void __init setup_memory(void) · 50 unmodified lines | |||
| 174 | pr_info("%s: max_low_pfn: %#lx\n", __func__, max_low_pfn); | 170 | pr_info("%s: max_low_pfn: %#lx\n", __func__, max_low_pfn); |
| 175 | pr_info("%s: max_pfn: %#lx\n", __func__, max_pfn); | 171 | pr_info("%s: max_pfn: %#lx\n", __func__, max_pfn); |
| 176 | 172 | ||
| 177 | /* | ||
| 178 | * Find an area to use for the bootmem bitmap. | ||
| 179 | * We look for the first area which is at least | ||
| 180 | * 128kB in length (128kB is enough for a bitmap | ||
| 181 | * for 4GB of memory, using 4kB pages), plus 1 page | ||
| 182 | * (in case the address isn't page-aligned). | ||
| 183 | */ | ||
| 184 | map_size = init_bootmem_node(NODE_DATA(0), | ||
| 185 | PFN_UP(TOPHYS((u32)klimit)), min_low_pfn, max_low_pfn); | ||
| 186 | memblock_reserve(PFN_UP(TOPHYS((u32)klimit)) << PAGE_SHIFT, map_size); | ||
| 187 | |||
| 188 | /* Add active regions with valid PFNs */ | 173 | /* Add active regions with valid PFNs */ |
| 189 | for_each_memblock(memory, reg) { | 174 | for_each_memblock(memory, reg) { |
| 190 | unsigned long start_pfn, end_pfn; | 175 | unsigned long start_pfn, end_pfn; |
| @@ -196,32 +181,9 @@ void __init setup_memory(void) · 5 unmodified lines | |||
| 196 | &memblock.memory, 0); | 181 | &memblock.memory, 0); |
| 197 | } | 182 | } |
| 198 | 183 | ||
| 199 | /* free bootmem is whole main memory */ | ||
| 200 | free_bootmem_with_active_regions(0, max_low_pfn); | ||
| 201 | |||
| 202 | /* reserve allocate blocks */ | ||
| 203 | for_each_memblock(reserved, reg) { | ||
| 204 | unsigned long top = reg->base + reg->size - 1; | ||
| 205 | |||
| 206 | pr_debug("reserved - 0x%08x-0x%08x, %lx, %lx\n", | ||
| 207 | (u32) reg->base, (u32) reg->size, top, | ||
| 208 | memory_start + lowmem_size - 1); | ||
| 209 | |||
| 210 | if (top <= (memory_start + lowmem_size - 1)) { | ||
| 211 | reserve_bootmem(reg->base, reg->size, BOOTMEM_DEFAULT); | ||
| 212 | } else if (reg->base < (memory_start + lowmem_size - 1)) { | ||
| 213 | unsigned long trunc_size = memory_start + lowmem_size - | ||
| 214 | reg->base; | ||
| 215 | reserve_bootmem(reg->base, trunc_size, BOOTMEM_DEFAULT); | ||
| 216 | } | ||
| 217 | } | ||
| 218 | |||
| 219 | /* XXX need to clip this if using highmem? */ | 184 | /* XXX need to clip this if using highmem? */ |
| 220 | sparse_memory_present_with_active_regions(0); | 185 | sparse_memory_present_with_active_regions(0); |
| 221 | 186 | ||
| 222 | #ifdef CONFIG_MMU | ||
| 223 | init_bootmem_done = 1; | ||
| 224 | #endif | ||
| 225 | paging_init(); | 187 | paging_init(); |
| 226 | } | 188 | } |
| 227 | 189 | ||
| @@ -398,30 +360,16 @@ asmlinkage void __init mmu_init(void) · 170 unmodified lines | |||
| 398 | /* This is only called until mem_init is done. */ | 360 | /* This is only called until mem_init is done. */ |
| 399 | void __init *early_get_page(void) | 361 | void __init *early_get_page(void) |
| 400 | { | 362 | { |
| 401 | void *p; | 363 | /* |
| 402 | if (init_bootmem_done) { | 364 | * Mem start + kernel_tlb -> here is limit |
| 403 | p = alloc_bootmem_pages(PAGE_SIZE); | 365 | * because of mem mapping from head.S |
| 404 | } else { | 366 | */ |
| 405 | /* | 367 | return __va(memblock_alloc_base(PAGE_SIZE, PAGE_SIZE, |
| 406 | * Mem start + kernel_tlb -> here is limit | 368 | memory_start + kernel_tlb)); |
| 407 | * because of mem mapping from head.S | ||
| 408 | */ | ||
| 409 | p = __va(memblock_alloc_base(PAGE_SIZE, PAGE_SIZE, | ||
| 410 | memory_start + kernel_tlb)); | ||
| 411 | } | ||
| 412 | return p; | ||
| 413 | } | 369 | } |
| 414 | 370 | ||
| 415 | #endif /* CONFIG_MMU */ | 371 | #endif /* CONFIG_MMU */ |
| 416 | 372 | ||
| 417 | void * __ref alloc_maybe_bootmem(size_t size, gfp_t mask) | ||
| 418 | { | ||
| 419 | if (mem_init_done) | ||
| 420 | return kmalloc(size, mask); | ||
| 421 | else | ||
| 422 | return alloc_bootmem(size); | ||
| 423 | } | ||
| 424 | |||
| 425 | void * __ref zalloc_maybe_bootmem(size_t size, gfp_t mask) | 373 | void * __ref zalloc_maybe_bootmem(size_t size, gfp_t mask) |
| 426 | { | 374 | { |
| 427 | void *p; | 375 | void *p; |
| @@ -135,6 +135,8 @@ int __init ath25_find_config(phys_addr_t base, unsigned long size) · 134 unmodified lines | |||
| 135 | } | 135 | } |
| 136 | 136 | ||
| 137 | board_data = kzalloc(BOARD_CONFIG_BUFSZ, GFP_KERNEL); | 137 | board_data = kzalloc(BOARD_CONFIG_BUFSZ, GFP_KERNEL); |
| 138 | if (!board_data) | ||
| 139 | goto error; | ||
| 138 | ath25_board.config = (struct ath25_boarddata *)board_data; | 140 | ath25_board.config = (struct ath25_boarddata *)board_data; |
| 139 | memcpy_fromio(board_data, bcfg, 0x100); | 141 | memcpy_fromio(board_data, bcfg, 0x100); |
| 140 | if (broken_boarddata) { | 142 | if (broken_boarddata) { |
| @@ -126,6 +126,7 @@ $(obj)/vmlinux.its.S: $(addprefix $(srctree)/arch/mips/$(PLATFORM)/,$(ITS_INPUTS · 125 unmodified lines | |||
| 126 | 126 | ||
| 127 | quiet_cmd_cpp_its_S = ITS $@ | 127 | quiet_cmd_cpp_its_S = ITS $@ |
| 128 | cmd_cpp_its_S = $(CPP) $(cpp_flags) -P -C -o $@ $< \ | 128 | cmd_cpp_its_S = $(CPP) $(cpp_flags) -P -C -o $@ $< \ |
| 129 | -D__ASSEMBLY__ \ | ||
| 129 | -DKERNEL_NAME="\"Linux $(KERNELRELEASE)\"" \ | 130 | -DKERNEL_NAME="\"Linux $(KERNELRELEASE)\"" \ |
| 130 | -DVMLINUX_BINARY="\"$(3)\"" \ | 131 | -DVMLINUX_BINARY="\"$(3)\"" \ |
| 131 | -DVMLINUX_COMPRESSION="\"$(2)\"" \ | 132 | -DVMLINUX_COMPRESSION="\"$(2)\"" \ |
| @@ -2277,6 +2277,8 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node, · 2276 unmodified lines | |||
| 2277 | } | 2277 | } |
| 2278 | 2278 | ||
| 2279 | host_data = kzalloc(sizeof(*host_data), GFP_KERNEL); | 2279 | host_data = kzalloc(sizeof(*host_data), GFP_KERNEL); |
| 2280 | if (!host_data) | ||
| 2281 | return -ENOMEM; | ||
| 2280 | raw_spin_lock_init(&host_data->lock); | 2282 | raw_spin_lock_init(&host_data->lock); |
| 2281 | 2283 | ||
| 2282 | addr = of_get_address(ciu_node, 0, NULL, NULL); | 2284 | addr = of_get_address(ciu_node, 0, NULL, NULL); |
| @@ -86,7 +86,6 @@ struct compat_flock { · 85 unmodified lines | |||
| 86 | compat_off_t l_len; | 86 | compat_off_t l_len; |
| 87 | s32 l_sysid; | 87 | s32 l_sysid; |
| 88 | compat_pid_t l_pid; | 88 | compat_pid_t l_pid; |
| 89 | short __unused; | ||
| 90 | s32 pad[4]; | 89 | s32 pad[4]; |
| 91 | }; | 90 | }; |
| 92 | 91 | ||
| @@ -10,6 +10,8 @@ · 9 unmodified lines | |||
| 10 | 10 | ||
| 11 | #include <linux/errno.h> | 11 | #include <linux/errno.h> |
| 12 | #include <linux/percpu.h> | 12 | #include <linux/percpu.h> |
| 13 | #include <linux/of.h> | ||
| 14 | #include <linux/of_address.h> | ||
| 13 | #include <linux/spinlock.h> | 15 | #include <linux/spinlock.h> |
| 14 | 16 | ||
| 15 | #include <asm/mips-cps.h> | 17 | #include <asm/mips-cps.h> |
| @@ -22,6 +24,17 @@ static DEFINE_PER_CPU_ALIGNED(unsigned long, cpc_core_lock_flags); · 6 unmodified lines | |||
| 22 | 24 | ||
| 23 | phys_addr_t __weak mips_cpc_default_phys_base(void) | 25 | phys_addr_t __weak mips_cpc_default_phys_base(void) |
| 24 | { | 26 | { |
| 27 | struct device_node *cpc_node; | ||
| 28 | struct resource res; | ||
| 29 | int err; | ||
| 30 | |||
| 31 | cpc_node = of_find_compatible_node(of_root, NULL, "mti,mips-cpc"); | ||
| 32 | if (cpc_node) { | ||
| 33 | err = of_address_to_resource(cpc_node, 0, &res); | ||
| 34 | if (!err) | ||
| 35 | return res.start; | ||
| 36 | } | ||
| 37 | |||
| 25 | return 0; | 38 | return 0; |
| 26 | } | 39 | } |
| 27 | 40 | ||
| @@ -375,6 +375,7 @@ static void __init bootmem_init(void) · 374 unmodified lines | |||
| 375 | unsigned long reserved_end; | 375 | unsigned long reserved_end; |
| 376 | unsigned long mapstart = ~0UL; | 376 | unsigned long mapstart = ~0UL; |
| 377 | unsigned long bootmap_size; | 377 | unsigned long bootmap_size; |
| 378 | phys_addr_t ramstart = (phys_addr_t)ULLONG_MAX; | ||
| 378 | bool bootmap_valid = false; | 379 | bool bootmap_valid = false; |
| 379 | int i; | 380 | int i; |
| 380 | 381 | ||
| @@ -395,7 +396,8 @@ static void __init bootmem_init(void) · 14 unmodified lines | |||
| 395 | max_low_pfn = 0; | 396 | max_low_pfn = 0; |
| 396 | 397 | ||
| 397 | /* | 398 | /* |
| 398 | * Find the highest page frame number we have available. | 399 | * Find the highest page frame number we have available |
| 400 | * and the lowest used RAM address | ||
| 399 | */ | 401 | */ |
| 400 | for (i = 0; i < boot_mem_map.nr_map; i++) { | 402 | for (i = 0; i < boot_mem_map.nr_map; i++) { |
| 401 | unsigned long start, end; | 403 | unsigned long start, end; |
| @@ -407,6 +409,8 @@ static void __init bootmem_init(void) · 5 unmodified lines | |||
| 407 | end = PFN_DOWN(boot_mem_map.map[i].addr | 409 | end = PFN_DOWN(boot_mem_map.map[i].addr |
| 408 | + boot_mem_map.map[i].size); | 410 | + boot_mem_map.map[i].size); |
| 409 | 411 | ||
| 412 | ramstart = min(ramstart, boot_mem_map.map[i].addr); | ||
| 413 | |||
| 410 | #ifndef CONFIG_HIGHMEM | 414 | #ifndef CONFIG_HIGHMEM |
| 411 | /* | 415 | /* |
| 412 | * Skip highmem here so we get an accurate max_low_pfn if low | 416 | * Skip highmem here so we get an accurate max_low_pfn if low |
| @@ -436,6 +440,13 @@ static void __init bootmem_init(void) · 23 unmodified lines | |||
| 436 | mapstart = max(reserved_end, start); | 440 | mapstart = max(reserved_end, start); |
| 437 | } | 441 | } |
| 438 | 442 | ||
| 443 | /* | ||
| 444 | * Reserve any memory between the start of RAM and PHYS_OFFSET | ||
| 445 | */ | ||
| 446 | if (ramstart > PHYS_OFFSET) | ||
| 447 | add_memory_region(PHYS_OFFSET, ramstart - PHYS_OFFSET, | ||
| 448 | BOOT_MEM_RESERVED); | ||
| 449 | |||
| 439 | if (min_low_pfn >= max_low_pfn) | 450 | if (min_low_pfn >= max_low_pfn) |
| 440 | panic("Incorrect memory mapping !!!"); | 451 | panic("Incorrect memory mapping !!!"); |
| 441 | if (min_low_pfn > ARCH_PFN_OFFSET) { | 452 | if (min_low_pfn > ARCH_PFN_OFFSET) { |
| @@ -664,9 +675,6 @@ static int __init early_parse_mem(char *p) · 222 unmodified lines | |||
| 664 | 675 | ||
| 665 | add_memory_region(start, size, BOOT_MEM_RAM); | 676 | add_memory_region(start, size, BOOT_MEM_RAM); |
| 666 | 677 | ||
| 667 | if (start && start > PHYS_OFFSET) | ||
| 668 | add_memory_region(PHYS_OFFSET, start - PHYS_OFFSET, | ||
| 669 | BOOT_MEM_RESERVED); | ||
| 670 | return 0; | 678 | return 0; |
| 671 | } | 679 | } |
| 672 | early_param("mem", early_parse_mem); | 680 | early_param("mem", early_parse_mem); |
| @@ -168,11 +168,11 @@ static void bmips_prepare_cpus(unsigned int max_cpus) · 167 unmodified lines | |||
| 168 | return; | 168 | return; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | if (request_irq(IPI0_IRQ, bmips_ipi_interrupt, IRQF_PERCPU, | 171 | if (request_irq(IPI0_IRQ, bmips_ipi_interrupt, |
| 172 | "smp_ipi0", NULL)) | 172 | IRQF_PERCPU | IRQF_NO_SUSPEND, "smp_ipi0", NULL)) |
| 173 | panic("Can't request IPI0 interrupt"); | 173 | panic("Can't request IPI0 interrupt"); |
| 174 | if (request_irq(IPI1_IRQ, bmips_ipi_interrupt, IRQF_PERCPU, | 174 | if (request_irq(IPI1_IRQ, bmips_ipi_interrupt, |
| 175 | "smp_ipi1", NULL)) | 175 | IRQF_PERCPU | IRQF_NO_SUSPEND, "smp_ipi1", NULL)) |
| 176 | panic("Can't request IPI1 interrupt"); | 176 | panic("Can't request IPI1 interrupt"); |
| 177 | } | 177 | } |
| 178 | 178 | ||
| @@ -572,7 +572,7 @@ asmlinkage void __weak plat_wired_tlb_setup(void) · 393 unmodified lines | |||
| 572 | */ | 572 | */ |
| 573 | } | 573 | } |
| 574 | 574 | ||
| 575 | void __init bmips_cpu_setup(void) | 575 | void bmips_cpu_setup(void) |
| 576 | { | 576 | { |
| 577 | void __iomem __maybe_unused *cbr = BMIPS_GET_CBR(); | 577 | void __iomem __maybe_unused *cbr = BMIPS_GET_CBR(); |
| 578 | u32 __maybe_unused cfg; | 578 | u32 __maybe_unused cfg; |
| @@ -13,6 +13,8 @@ choice · 12 unmodified lines | |||
| 13 | config SOC_AMAZON_SE | 13 | config SOC_AMAZON_SE |
| 14 | bool "Amazon SE" | 14 | bool "Amazon SE" |
| 15 | select SOC_TYPE_XWAY | 15 | select SOC_TYPE_XWAY |
| 16 | select MFD_SYSCON | ||
| 17 | select MFD_CORE | ||
| 16 | 18 | ||
| 17 | config SOC_XWAY | 19 | config SOC_XWAY |
| 18 | bool "XWAY" | 20 | bool "XWAY" |
| @@ -549,9 +549,9 @@ void __init ltq_soc_init(void) · 548 unmodified lines | |||
| 549 | clkdev_add_static(ltq_ar9_cpu_hz(), ltq_ar9_fpi_hz(), | 549 | clkdev_add_static(ltq_ar9_cpu_hz(), ltq_ar9_fpi_hz(), |
| 550 | ltq_ar9_fpi_hz(), CLOCK_250M); | 550 | ltq_ar9_fpi_hz(), CLOCK_250M); |
| 551 | clkdev_add_pmu("1f203018.usb2-phy", "phy", 1, 0, PMU_USB0_P); | 551 | clkdev_add_pmu("1f203018.usb2-phy", "phy", 1, 0, PMU_USB0_P); |
| 552 | clkdev_add_pmu("1e101000.usb", "otg", 1, 0, PMU_USB0); | 552 | clkdev_add_pmu("1e101000.usb", "otg", 1, 0, PMU_USB0 | PMU_AHBM); |
| 553 | clkdev_add_pmu("1f203034.usb2-phy", "phy", 1, 0, PMU_USB1_P); | 553 | clkdev_add_pmu("1f203034.usb2-phy", "phy", 1, 0, PMU_USB1_P); |
| 554 | clkdev_add_pmu("1e106000.usb", "otg", 1, 0, PMU_USB1); | 554 | clkdev_add_pmu("1e106000.usb", "otg", 1, 0, PMU_USB1 | PMU_AHBM); |
| 555 | clkdev_add_pmu("1e180000.etop", "switch", 1, 0, PMU_SWITCH); | 555 | clkdev_add_pmu("1e180000.etop", "switch", 1, 0, PMU_SWITCH); |
| 556 | clkdev_add_pmu("1e103000.sdio", NULL, 1, 0, PMU_SDIO); | 556 | clkdev_add_pmu("1e103000.sdio", NULL, 1, 0, PMU_SDIO); |
| 557 | clkdev_add_pmu("1e103100.deu", NULL, 1, 0, PMU_DEU); | 557 | clkdev_add_pmu("1e103100.deu", NULL, 1, 0, PMU_DEU); |
| @@ -560,7 +560,7 @@ void __init ltq_soc_init(void) · 2 unmodified lines | |||
| 560 | } else { | 560 | } else { |
| 561 | clkdev_add_static(ltq_danube_cpu_hz(), ltq_danube_fpi_hz(), | 561 | clkdev_add_static(ltq_danube_cpu_hz(), ltq_danube_fpi_hz(), |
| 562 | ltq_danube_fpi_hz(), ltq_danube_pp32_hz()); | 562 | ltq_danube_fpi_hz(), ltq_danube_pp32_hz()); |
| 563 | clkdev_add_pmu("1f203018.usb2-phy", "ctrl", 1, 0, PMU_USB0); | 563 | clkdev_add_pmu("1e101000.usb", "otg", 1, 0, PMU_USB0 | PMU_AHBM); |
| 564 | clkdev_add_pmu("1f203018.usb2-phy", "phy", 1, 0, PMU_USB0_P); | 564 | clkdev_add_pmu("1f203018.usb2-phy", "phy", 1, 0, PMU_USB0_P); |
| 565 | clkdev_add_pmu("1e103000.sdio", NULL, 1, 0, PMU_SDIO); | 565 | clkdev_add_pmu("1e103000.sdio", NULL, 1, 0, PMU_SDIO); |
| 566 | clkdev_add_pmu("1e103100.deu", NULL, 1, 0, PMU_DEU); | 566 | clkdev_add_pmu("1e103100.deu", NULL, 1, 0, PMU_DEU); |
| @@ -7,6 +7,8 @@ choice · 6 unmodified lines | |||
| 7 | config LEMOTE_FULOONG2E | 7 | config LEMOTE_FULOONG2E |
| 8 | bool "Lemote Fuloong(2e) mini-PC" | 8 | bool "Lemote Fuloong(2e) mini-PC" |
| 9 | select ARCH_SPARSEMEM_ENABLE | 9 | select ARCH_SPARSEMEM_ENABLE |
| 10 | select ARCH_MIGHT_HAVE_PC_PARPORT | ||
| 11 | select ARCH_MIGHT_HAVE_PC_SERIO | ||
| 10 | select CEVT_R4K | 12 | select CEVT_R4K |
| 11 | select CSRC_R4K | 13 | select CSRC_R4K |
| 12 | select SYS_HAS_CPU_LOONGSON2E | 14 | select SYS_HAS_CPU_LOONGSON2E |
| @@ -33,6 +35,8 @@ config LEMOTE_FULOONG2E · 20 unmodified lines | |||
| 33 | config LEMOTE_MACH2F | 35 | config LEMOTE_MACH2F |
| 34 | bool "Lemote Loongson 2F family machines" | 36 | bool "Lemote Loongson 2F family machines" |
| 35 | select ARCH_SPARSEMEM_ENABLE | 37 | select ARCH_SPARSEMEM_ENABLE |
| 38 | select ARCH_MIGHT_HAVE_PC_PARPORT | ||
| 39 | select ARCH_MIGHT_HAVE_PC_SERIO | ||
| 36 | select BOARD_SCACHE | 40 | select BOARD_SCACHE |
| 37 | select BOOT_ELF32 | 41 | select BOOT_ELF32 |
| 38 | select CEVT_R4K if ! MIPS_EXTERNAL_TIMER | 42 | select CEVT_R4K if ! MIPS_EXTERNAL_TIMER |
| @@ -62,6 +66,8 @@ config LEMOTE_MACH2F · 23 unmodified lines | |||
| 62 | config LOONGSON_MACH3X | 66 | config LOONGSON_MACH3X |
| 63 | bool "Generic Loongson 3 family machines" | 67 | bool "Generic Loongson 3 family machines" |
| 64 | select ARCH_SPARSEMEM_ENABLE | 68 | select ARCH_SPARSEMEM_ENABLE |
| 69 | select ARCH_MIGHT_HAVE_PC_PARPORT | ||
| 70 | select ARCH_MIGHT_HAVE_PC_SERIO | ||
| 65 | select GENERIC_ISA_DMA_SUPPORT_BROKEN | 71 | select GENERIC_ISA_DMA_SUPPORT_BROKEN |
| 66 | select BOOT_ELF32 | 72 | select BOOT_ELF32 |
| 67 | select BOARD_SCACHE | 73 | select BOARD_SCACHE |
| @@ -170,6 +170,28 @@ void prom_soc_init(struct ralink_soc_info *soc_info) · 169 unmodified lines | |||
| 170 | u32 n1; | 170 | u32 n1; |
| 171 | u32 rev; | 171 | u32 rev; |
| 172 | 172 | ||
| 173 | /* Early detection of CMP support */ | ||
| 174 | mips_cm_probe(); | ||
| 175 | mips_cpc_probe(); | ||
| 176 | |||
| 177 | if (mips_cps_numiocu(0)) { | ||
| 178 | /* | ||
| 179 | * mips_cm_probe() wipes out bootloader | ||
| 180 | * config for CM regions and we have to configure them | ||
| 181 | * again. This SoC cannot talk to pamlbus devices | ||
| 182 | * witout proper iocu region set up. | ||
| 183 | * | ||
| 184 | * FIXME: it would be better to do this with values | ||
| 185 | * from DT, but we need this very early because | ||
| 186 | * without this we cannot talk to pretty much anything | ||
| 187 | * including serial. | ||
| 188 | */ | ||
| 189 | write_gcr_reg0_base(MT7621_PALMBUS_BASE); | ||
| 190 | write_gcr_reg0_mask(~MT7621_PALMBUS_SIZE | | ||
| 191 | CM_GCR_REGn_MASK_CMTGT_IOCU0); | ||
| 192 | __sync(); | ||
| 193 | } | ||
| 194 | |||
| 173 | n0 = __raw_readl(sysc + SYSC_REG_CHIP_NAME0); | 195 | n0 = __raw_readl(sysc + SYSC_REG_CHIP_NAME0); |
| 174 | n1 = __raw_readl(sysc + SYSC_REG_CHIP_NAME1); | 196 | n1 = __raw_readl(sysc + SYSC_REG_CHIP_NAME1); |
| 175 | 197 | ||
| @@ -194,26 +216,6 @@ void prom_soc_init(struct ralink_soc_info *soc_info) · 18 unmodified lines | |||
| 194 | 216 | ||
| 195 | rt2880_pinmux_data = mt7621_pinmux_data; | 217 | rt2880_pinmux_data = mt7621_pinmux_data; |
| 196 | 218 | ||
| 197 | /* Early detection of CMP support */ | ||
| 198 | mips_cm_probe(); | ||
| 199 | mips_cpc_probe(); | ||
| 200 | |||
| 201 | if (mips_cps_numiocu(0)) { | ||
| 202 | /* | ||
| 203 | * mips_cm_probe() wipes out bootloader | ||
| 204 | * config for CM regions and we have to configure them | ||
| 205 | * again. This SoC cannot talk to pamlbus devices | ||
| 206 | * witout proper iocu region set up. | ||
| 207 | * | ||
| 208 | * FIXME: it would be better to do this with values | ||
| 209 | * from DT, but we need this very early because | ||
| 210 | * without this we cannot talk to pretty much anything | ||
| 211 | * including serial. | ||
| 212 | */ | ||
| 213 | write_gcr_reg0_base(MT7621_PALMBUS_BASE); | ||
| 214 | write_gcr_reg0_mask(~MT7621_PALMBUS_SIZE | | ||
| 215 | CM_GCR_REGn_MASK_CMTGT_IOCU0); | ||
| 216 | } | ||
| 217 | 219 | ||
| 218 | if (!register_cps_smp_ops()) | 220 | if (!register_cps_smp_ops()) |
| 219 | return; | 221 | return; |
| @@ -96,16 +96,9 @@ static void ralink_restart(char *command) · 95 unmodified lines | |||
| 96 | unreachable(); | 96 | unreachable(); |
| 97 | } | 97 | } |
| 98 | 98 | ||
| 99 | static void ralink_halt(void) | ||
| 100 | { | ||
| 101 | local_irq_disable(); | ||
| 102 | unreachable(); | ||
| 103 | } | ||
| 104 | |||
| 105 | static int __init mips_reboot_setup(void) | 99 | static int __init mips_reboot_setup(void) |
| 106 | { | 100 | { |
| 107 | _machine_restart = ralink_restart; | 101 | _machine_restart = ralink_restart; |
| 108 | _machine_halt = ralink_halt; | ||
| 109 | 102 | ||
| 110 | return 0; | 103 | return 0; |
| 111 | } | 104 | } |
| @@ -26,6 +26,7 @@ void flush_user_icache_range_asm(unsigned long, unsigned long); · 25 unmodified lines | |||
| 26 | void flush_kernel_icache_range_asm(unsigned long, unsigned long); | 26 | void flush_kernel_icache_range_asm(unsigned long, unsigned long); |
| 27 | void flush_user_dcache_range_asm(unsigned long, unsigned long); | 27 | void flush_user_dcache_range_asm(unsigned long, unsigned long); |
| 28 | void flush_kernel_dcache_range_asm(unsigned long, unsigned long); | 28 | void flush_kernel_dcache_range_asm(unsigned long, unsigned long); |
| 29 | void purge_kernel_dcache_range_asm(unsigned long, unsigned long); | ||
| 29 | void flush_kernel_dcache_page_asm(void *); | 30 | void flush_kernel_dcache_page_asm(void *); |
| 30 | void flush_kernel_icache_page(void *); | 31 | void flush_kernel_icache_page(void *); |
| 31 | 32 | ||
| @@ -316,6 +316,8 @@ extern int _parisc_requires_coherency; · 315 unmodified lines | |||
| 316 | #define parisc_requires_coherency() (0) | 316 | #define parisc_requires_coherency() (0) |
| 317 | #endif | 317 | #endif |
| 318 | 318 | ||
| 319 | extern int running_on_qemu; | ||
| 320 | |||
| 319 | #endif /* __ASSEMBLY__ */ | 321 | #endif /* __ASSEMBLY__ */ |
| 320 | 322 | ||
| 321 | #endif /* __ASM_PARISC_PROCESSOR_H */ | 323 | #endif /* __ASM_PARISC_PROCESSOR_H */ |
| @@ -465,10 +465,10 @@ EXPORT_SYMBOL(copy_user_page); · 464 unmodified lines | |||
| 465 | int __flush_tlb_range(unsigned long sid, unsigned long start, | 465 | int __flush_tlb_range(unsigned long sid, unsigned long start, |
| 466 | unsigned long end) | 466 | unsigned long end) |
| 467 | { | 467 | { |
| 468 | unsigned long flags, size; | 468 | unsigned long flags; |
| 469 | 469 | ||
| 470 | size = (end - start); | 470 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 471 | if (size >= parisc_tlb_flush_threshold) { | 471 | end - start >= parisc_tlb_flush_threshold) { |
| 472 | flush_tlb_all(); | 472 | flush_tlb_all(); |
| 473 | return 1; | 473 | return 1; |
| 474 | } | 474 | } |
| @@ -539,13 +539,12 @@ void flush_cache_mm(struct mm_struct *mm) · 64 unmodified lines | |||
| 539 | struct vm_area_struct *vma; | 539 | struct vm_area_struct *vma; |
| 540 | pgd_t *pgd; | 540 | pgd_t *pgd; |
| 541 | 541 | ||
| 542 | /* Flush the TLB to avoid speculation if coherency is required. */ | ||
| 543 | if (parisc_requires_coherency()) | ||
| 544 | flush_tlb_all(); | ||
| 545 | |||
| 546 | /* Flushing the whole cache on each cpu takes forever on | 542 | /* Flushing the whole cache on each cpu takes forever on |
| 547 | rp3440, etc. So, avoid it if the mm isn't too big. */ | 543 | rp3440, etc. So, avoid it if the mm isn't too big. */ |
| 548 | if (mm_total_size(mm) >= parisc_cache_flush_threshold) { | 544 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 545 | mm_total_size(mm) >= parisc_cache_flush_threshold) { | ||
| 546 | if (mm->context) | ||
| 547 | flush_tlb_all(); | ||
| 549 | flush_cache_all(); | 548 | flush_cache_all(); |
| 550 | return; | 549 | return; |
| 551 | } | 550 | } |
| @@ -553,9 +552,9 @@ void flush_cache_mm(struct mm_struct *mm) · 1 unmodified line | |||
| 553 | if (mm->context == mfsp(3)) { | 552 | if (mm->context == mfsp(3)) { |
| 554 | for (vma = mm->mmap; vma; vma = vma->vm_next) { | 553 | for (vma = mm->mmap; vma; vma = vma->vm_next) { |
| 555 | flush_user_dcache_range_asm(vma->vm_start, vma->vm_end); | 554 | flush_user_dcache_range_asm(vma->vm_start, vma->vm_end); |
| 556 | if ((vma->vm_flags & VM_EXEC) == 0) | 555 | if (vma->vm_flags & VM_EXEC) |
| 557 | continue; | 556 | flush_user_icache_range_asm(vma->vm_start, vma->vm_end); |
| 558 | flush_user_icache_range_asm(vma->vm_start, vma->vm_end); | 557 | flush_tlb_range(vma, vma->vm_start, vma->vm_end); |
| 559 | } | 558 | } |
| 560 | return; | 559 | return; |
| 561 | } | 560 | } |
| @@ -573,6 +572,8 @@ void flush_cache_mm(struct mm_struct *mm) · 11 unmodified lines | |||
| 573 | pfn = pte_pfn(*ptep); | 572 | pfn = pte_pfn(*ptep); |
| 574 | if (!pfn_valid(pfn)) | 573 | if (!pfn_valid(pfn)) |
| 575 | continue; | 574 | continue; |
| 575 | if (unlikely(mm->context)) | ||
| 576 | flush_tlb_page(vma, addr); | ||
| 576 | __flush_cache_page(vma, addr, PFN_PHYS(pfn)); | 577 | __flush_cache_page(vma, addr, PFN_PHYS(pfn)); |
| 577 | } | 578 | } |
| 578 | } | 579 | } |
| @@ -581,30 +582,45 @@ void flush_cache_mm(struct mm_struct *mm) · 2 unmodified lines | |||
| 581 | void flush_cache_range(struct vm_area_struct *vma, | 582 | void flush_cache_range(struct vm_area_struct *vma, |
| 582 | unsigned long start, unsigned long end) | 583 | unsigned long start, unsigned long end) |
| 583 | { | 584 | { |
| 584 | BUG_ON(!vma->vm_mm->context); | 585 | pgd_t *pgd; |
| 585 | 586 | unsigned long addr; | |
| 586 | /* Flush the TLB to avoid speculation if coherency is required. */ | ||
| 587 | if (parisc_requires_coherency()) | ||
| 588 | flush_tlb_range(vma, start, end); | ||
| 589 | 587 | ||
| 590 | if ((end - start) >= parisc_cache_flush_threshold | 588 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 591 | || vma->vm_mm->context != mfsp(3)) { | 589 | end - start >= parisc_cache_flush_threshold) { |
| 590 | if (vma->vm_mm->context) | ||
| 591 | flush_tlb_range(vma, start, end); | ||
| 592 | flush_cache_all(); | 592 | flush_cache_all(); |
| 593 | return; | 593 | return; |
| 594 | } | 594 | } |
| 595 | 595 | ||
| 596 | flush_user_dcache_range_asm(start, end); | 596 | if (vma->vm_mm->context == mfsp(3)) { |
| 597 | if (vma->vm_flags & VM_EXEC) | 597 | flush_user_dcache_range_asm(start, end); |
| 598 | flush_user_icache_range_asm(start, end); | 598 | if (vma->vm_flags & VM_EXEC) |
| 599 | flush_user_icache_range_asm(start, end); | ||
| 600 | flush_tlb_range(vma, start, end); | ||
| 601 | return; | ||
| 602 | } | ||
| 603 | |||
| 604 | pgd = vma->vm_mm->pgd; | ||
| 605 | for (addr = vma->vm_start; addr < vma->vm_end; addr += PAGE_SIZE) { | ||
| 606 | unsigned long pfn; | ||
| 607 | pte_t *ptep = get_ptep(pgd, addr); | ||
| 608 | if (!ptep) | ||
| 609 | continue; | ||
| 610 | pfn = pte_pfn(*ptep); | ||
| 611 | if (pfn_valid(pfn)) { | ||
| 612 | if (unlikely(vma->vm_mm->context)) | ||
| 613 | flush_tlb_page(vma, addr); | ||
| 614 | __flush_cache_page(vma, addr, PFN_PHYS(pfn)); | ||
| 615 | } | ||
| 616 | } | ||
| 599 | } | 617 | } |
| 600 | 618 | ||
| 601 | void | 619 | void |
| 602 | flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long pfn) | 620 | flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long pfn) |
| 603 | { | 621 | { |
| 604 | BUG_ON(!vma->vm_mm->context); | ||
| 605 | |||
| 606 | if (pfn_valid(pfn)) { | 622 | if (pfn_valid(pfn)) { |
| 607 | if (parisc_requires_coherency()) | 623 | if (likely(vma->vm_mm->context)) |
| 608 | flush_tlb_page(vma, vmaddr); | 624 | flush_tlb_page(vma, vmaddr); |
| 609 | __flush_cache_page(vma, vmaddr, PFN_PHYS(pfn)); | 625 | __flush_cache_page(vma, vmaddr, PFN_PHYS(pfn)); |
| 610 | } | 626 | } |
| @@ -613,21 +629,33 @@ flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long · 2 unmodified lines | |||
| 613 | void flush_kernel_vmap_range(void *vaddr, int size) | 629 | void flush_kernel_vmap_range(void *vaddr, int size) |
| 614 | { | 630 | { |
| 615 | unsigned long start = (unsigned long)vaddr; | 631 | unsigned long start = (unsigned long)vaddr; |
| 632 | unsigned long end = start + size; | ||
| 616 | 633 | ||
| 617 | if ((unsigned long)size > parisc_cache_flush_threshold) | 634 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 635 | (unsigned long)size >= parisc_cache_flush_threshold) { | ||
| 636 | flush_tlb_kernel_range(start, end); | ||
| 618 | flush_data_cache(); | 637 | flush_data_cache(); |
| 619 | else | 638 | return; |
| 620 | flush_kernel_dcache_range_asm(start, start + size); | 639 | } |
| 640 | |||
| 641 | flush_kernel_dcache_range_asm(start, end); | ||
| 642 | flush_tlb_kernel_range(start, end); | ||
| 621 | } | 643 | } |
| 622 | EXPORT_SYMBOL(flush_kernel_vmap_range); | 644 | EXPORT_SYMBOL(flush_kernel_vmap_range); |
| 623 | 645 | ||
| 624 | void invalidate_kernel_vmap_range(void *vaddr, int size) | 646 | void invalidate_kernel_vmap_range(void *vaddr, int size) |
| 625 | { | 647 | { |
| 626 | unsigned long start = (unsigned long)vaddr; | 648 | unsigned long start = (unsigned long)vaddr; |
| 649 | unsigned long end = start + size; | ||
| 627 | 650 | ||
| 628 | if ((unsigned long)size > parisc_cache_flush_threshold) | 651 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 652 | (unsigned long)size >= parisc_cache_flush_threshold) { | ||
| 653 | flush_tlb_kernel_range(start, end); | ||
| 629 | flush_data_cache(); | 654 | flush_data_cache(); |
| 630 | else | 655 | return; |
| 631 | flush_kernel_dcache_range_asm(start, start + size); | 656 | } |
| 657 | |||
| 658 | purge_kernel_dcache_range_asm(start, end); | ||
| 659 | flush_tlb_kernel_range(start, end); | ||
| 632 | } | 660 | } |
| 633 | EXPORT_SYMBOL(invalidate_kernel_vmap_range); | 661 | EXPORT_SYMBOL(invalidate_kernel_vmap_range); |
| @@ -138,6 +138,16 @@ $pgt_fill_loop: · 137 unmodified lines | |||
| 138 | std %dp,0x18(%r10) | 138 | std %dp,0x18(%r10) |
| 139 | #endif | 139 | #endif |
| 140 | 140 | ||
| 141 | #ifdef CONFIG_64BIT | ||
| 142 | /* Get PDCE_PROC for monarch CPU. */ | ||
| 143 | #define MEM_PDC_LO 0x388 | ||
| 144 | #define MEM_PDC_HI 0x35C | ||
| 145 | ldw MEM_PDC_LO(%r0),%r3 | ||
| 146 | ldw MEM_PDC_HI(%r0),%r10 | ||
| 147 | depd %r10, 31, 32, %r3 /* move to upper word */ | ||
| 148 | #endif | ||
| 149 | |||
| 150 | |||
| 141 | #ifdef CONFIG_SMP | 151 | #ifdef CONFIG_SMP |
| 142 | /* Set the smp rendezvous address into page zero. | 152 | /* Set the smp rendezvous address into page zero. |
| 143 | ** It would be safer to do this in init_smp_config() but | 153 | ** It would be safer to do this in init_smp_config() but |
| @@ -196,12 +206,6 @@ common_stext: · 52 unmodified lines | |||
| 196 | ** Someday, palo might not do this for the Monarch either. | 206 | ** Someday, palo might not do this for the Monarch either. |
| 197 | */ | 207 | */ |
| 198 | 2: | 208 | 2: |
| 199 | #define MEM_PDC_LO 0x388 | ||
| 200 | #define MEM_PDC_HI 0x35C | ||
| 201 | ldw MEM_PDC_LO(%r0),%r3 | ||
| 202 | ldw MEM_PDC_HI(%r0),%r6 | ||
| 203 | depd %r6, 31, 32, %r3 /* move to upper word */ | ||
| 204 | |||
| 205 | mfctl %cr30,%r6 /* PCX-W2 firmware bug */ | 209 | mfctl %cr30,%r6 /* PCX-W2 firmware bug */ |
| 206 | 210 | ||
| 207 | ldo PDC_PSW(%r0),%arg0 /* 21 */ | 211 | ldo PDC_PSW(%r0),%arg0 /* 21 */ |
| @@ -268,6 +272,8 @@ $install_iva: · 60 unmodified lines | |||
| 268 | aligned_rfi: | 272 | aligned_rfi: |
| 269 | pcxt_ssm_bug | 273 | pcxt_ssm_bug |
| 270 | 274 | ||
| 275 | copy %r3, %arg0 /* PDCE_PROC for smp_callin() */ | ||
| 276 | |||
| 271 | rsm PSW_SM_QUIET,%r0 /* off troublesome PSW bits */ | 277 | rsm PSW_SM_QUIET,%r0 /* off troublesome PSW bits */ |
| 272 | /* Don't need NOPs, have 8 compliant insn before rfi */ | 278 | /* Don't need NOPs, have 8 compliant insn before rfi */ |
| 273 | 279 | ||
| @@ -1110,6 +1110,28 @@ ENTRY_CFI(flush_kernel_dcache_range_asm) · 1109 unmodified lines | |||
| 1110 | .procend | 1110 | .procend |
| 1111 | ENDPROC_CFI(flush_kernel_dcache_range_asm) | 1111 | ENDPROC_CFI(flush_kernel_dcache_range_asm) |
| 1112 | 1112 | ||
| 1113 | ENTRY_CFI(purge_kernel_dcache_range_asm) | ||
| 1114 | .proc | ||
| 1115 | .callinfo NO_CALLS | ||
| 1116 | .entry | ||
| 1117 | |||
| 1118 | ldil L%dcache_stride, %r1 | ||
| 1119 | ldw R%dcache_stride(%r1), %r23 | ||
| 1120 | ldo -1(%r23), %r21 | ||
| 1121 | ANDCM %r26, %r21, %r26 | ||
| 1122 | |||
| 1123 | 1: cmpb,COND(<<),n %r26, %r25,1b | ||
| 1124 | pdc,m %r23(%r26) | ||
| 1125 | |||
| 1126 | sync | ||
| 1127 | syncdma | ||
| 1128 | bv %r0(%r2) | ||
| 1129 | nop | ||
| 1130 | .exit | ||
| 1131 | |||
| 1132 | .procend | ||
| 1133 | ENDPROC_CFI(purge_kernel_dcache_range_asm) | ||
| 1134 | |||
| 1113 | ENTRY_CFI(flush_user_icache_range_asm) | 1135 | ENTRY_CFI(flush_user_icache_range_asm) |
| 1114 | .proc | 1136 | .proc |
| 1115 | .callinfo NO_CALLS | 1137 | .callinfo NO_CALLS |
| @@ -292,10 +292,15 @@ smp_cpu_init(int cpunum) · 291 unmodified lines | |||
| 292 | * Slaves start using C here. Indirectly called from smp_slave_stext. | 292 | * Slaves start using C here. Indirectly called from smp_slave_stext. |
| 293 | * Do what start_kernel() and main() do for boot strap processor (aka monarch) | 293 | * Do what start_kernel() and main() do for boot strap processor (aka monarch) |
| 294 | */ | 294 | */ |
| 295 | void __init smp_callin(void) | 295 | void __init smp_callin(unsigned long pdce_proc) |
| 296 | { | 296 | { |
| 297 | int slave_id = cpu_now_booting; | 297 | int slave_id = cpu_now_booting; |
| 298 | 298 | ||
| 299 | #ifdef CONFIG_64BIT | ||
| 300 | WARN_ON(((unsigned long)(PAGE0->mem_pdc_hi) << 32 | ||
| 301 | | PAGE0->mem_pdc) != pdce_proc); | ||
| 302 | #endif | ||
| 303 | |||
| 299 | smp_cpu_init(slave_id); | 304 | smp_cpu_init(slave_id); |
| 300 | preempt_disable(); | 305 | preempt_disable(); |
| 301 | 306 | ||
| @@ -76,10 +76,10 @@ irqreturn_t __irq_entry timer_interrupt(int irq, void *dev_id) · 75 unmodified lines | |||
| 76 | next_tick = cpuinfo->it_value; | 76 | next_tick = cpuinfo->it_value; |
| 77 | 77 | ||
| 78 | /* Calculate how many ticks have elapsed. */ | 78 | /* Calculate how many ticks have elapsed. */ |
| 79 | now = mfctl(16); | ||
| 79 | do { | 80 | do { |
| 80 | ++ticks_elapsed; | 81 | ++ticks_elapsed; |
| 81 | next_tick += cpt; | 82 | next_tick += cpt; |
| 82 | now = mfctl(16); | ||
| 83 | } while (next_tick - now > cpt); | 83 | } while (next_tick - now > cpt); |
| 84 | 84 | ||
| 85 | /* Store (in CR16 cycles) up to when we are accounting right now. */ | 85 | /* Store (in CR16 cycles) up to when we are accounting right now. */ |
| @@ -103,16 +103,17 @@ irqreturn_t __irq_entry timer_interrupt(int irq, void *dev_id) · 17 unmodified lines | |||
| 103 | * if one or the other wrapped. If "now" is "bigger" we'll end up | 103 | * if one or the other wrapped. If "now" is "bigger" we'll end up |
| 104 | * with a very large unsigned number. | 104 | * with a very large unsigned number. |
| 105 | */ | 105 | */ |
| 106 | while (next_tick - mfctl(16) > cpt) | 106 | now = mfctl(16); |
| 107 | while (next_tick - now > cpt) | ||
| 107 | next_tick += cpt; | 108 | next_tick += cpt; |
| 108 | 109 | ||
| 109 | /* Program the IT when to deliver the next interrupt. | 110 | /* Program the IT when to deliver the next interrupt. |
| 110 | * Only bottom 32-bits of next_tick are writable in CR16! | 111 | * Only bottom 32-bits of next_tick are writable in CR16! |
| 111 | * Timer interrupt will be delivered at least a few hundred cycles | 112 | * Timer interrupt will be delivered at least a few hundred cycles |
| 112 | * after the IT fires, so if we are too close (<= 500 cycles) to the | 113 | * after the IT fires, so if we are too close (<= 8000 cycles) to the |
| 113 | * next cycle, simply skip it. | 114 | * next cycle, simply skip it. |
| 114 | */ | 115 | */ |
| 115 | if (next_tick - mfctl(16) <= 500) | 116 | if (next_tick - now <= 8000) |
| 116 | next_tick += cpt; | 117 | next_tick += cpt; |
| 117 | mtctl(next_tick, 16); | 118 | mtctl(next_tick, 16); |
| 118 | 119 | ||
| @@ -248,7 +249,7 @@ static int __init init_cr16_clocksource(void) · 129 unmodified lines | |||
| 248 | * different sockets, so mark them unstable and lower rating on | 249 | * different sockets, so mark them unstable and lower rating on |
| 249 | * multi-socket SMP systems. | 250 | * multi-socket SMP systems. |
| 250 | */ | 251 | */ |
| 251 | if (num_online_cpus() > 1) { | 252 | if (num_online_cpus() > 1 && !running_on_qemu) { |
| 252 | int cpu; | 253 | int cpu; |
| 253 | unsigned long cpu0_loc; | 254 | unsigned long cpu0_loc; |
| 254 | cpu0_loc = per_cpu(cpu_data, 0).cpu_loc; | 255 | cpu0_loc = per_cpu(cpu_data, 0).cpu_loc; |
| @@ -629,7 +629,12 @@ void __init mem_init(void) · 628 unmodified lines | |||
| 629 | #endif | 629 | #endif |
| 630 | 630 | ||
| 631 | mem_init_print_info(NULL); | 631 | mem_init_print_info(NULL); |
| 632 | #ifdef CONFIG_DEBUG_KERNEL /* double-sanity-check paranoia */ | 632 | |
| 633 | #if 0 | ||
| 634 | /* | ||
| 635 | * Do not expose the virtual kernel memory layout to userspace. | ||
| 636 | * But keep code for debugging purposes. | ||
| 637 | */ | ||
| 633 | printk("virtual kernel memory layout:\n" | 638 | printk("virtual kernel memory layout:\n" |
| 634 | " vmalloc : 0x%px - 0x%px (%4ld MB)\n" | 639 | " vmalloc : 0x%px - 0x%px (%4ld MB)\n" |
| 635 | " memory : 0x%px - 0x%px (%4ld MB)\n" | 640 | " memory : 0x%px - 0x%px (%4ld MB)\n" |
| @@ -101,7 +101,8 @@ $(addprefix $(obj)/,$(zlib-y)): \ · 100 unmodified lines | |||
| 101 | libfdt := fdt.c fdt_ro.c fdt_wip.c fdt_sw.c fdt_rw.c fdt_strerror.c | 101 | libfdt := fdt.c fdt_ro.c fdt_wip.c fdt_sw.c fdt_rw.c fdt_strerror.c |
| 102 | libfdtheader := fdt.h libfdt.h libfdt_internal.h | 102 | libfdtheader := fdt.h libfdt.h libfdt_internal.h |
| 103 | 103 | ||
| 104 | $(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o): \ | 104 | $(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o \ |
| 105 | treeboot-akebono.o treeboot-currituck.o treeboot-iss4xx.o): \ | ||
| 105 | $(addprefix $(obj)/,$(libfdtheader)) | 106 | $(addprefix $(obj)/,$(libfdtheader)) |
| 106 | 107 | ||
| 107 | src-wlib-y := string.S crt0.S stdio.c decompress.c main.c \ | 108 | src-wlib-y := string.S crt0.S stdio.c decompress.c main.c \ |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | |||
| 16 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) | 16 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) |
| 17 | 17 | ||
| 18 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE | 18 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE |
| 19 | #define PUD_CACHE_INDEX PUD_INDEX_SIZE | ||
| 19 | 20 | ||
| 20 | #ifndef __ASSEMBLY__ | 21 | #ifndef __ASSEMBLY__ |
| 21 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) | 22 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) |
| @@ -63,7 +63,8 @@ static inline int hash__hugepd_ok(hugepd_t hpd) · 62 unmodified lines | |||
| 63 | * keeping the prototype consistent across the two formats. | 63 | * keeping the prototype consistent across the two formats. |
| 64 | */ | 64 | */ |
| 65 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, | 65 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, |
| 66 | unsigned int subpg_index, unsigned long hidx) | 66 | unsigned int subpg_index, unsigned long hidx, |
| 67 | int offset) | ||
| 67 | { | 68 | { |
| 68 | return (hidx << H_PAGE_F_GIX_SHIFT) & | 69 | return (hidx << H_PAGE_F_GIX_SHIFT) & |
| 69 | (H_PAGE_F_SECOND | H_PAGE_F_GIX); | 70 | (H_PAGE_F_SECOND | H_PAGE_F_GIX); |
| @@ -45,7 +45,7 @@ · 44 unmodified lines | |||
| 45 | * generic accessors and iterators here | 45 | * generic accessors and iterators here |
| 46 | */ | 46 | */ |
| 47 | #define __real_pte __real_pte | 47 | #define __real_pte __real_pte |
| 48 | static inline real_pte_t __real_pte(pte_t pte, pte_t *ptep) | 48 | static inline real_pte_t __real_pte(pte_t pte, pte_t *ptep, int offset) |
| 49 | { | 49 | { |
| 50 | real_pte_t rpte; | 50 | real_pte_t rpte; |
| 51 | unsigned long *hidxp; | 51 | unsigned long *hidxp; |
| @@ -59,7 +59,7 @@ static inline real_pte_t __real_pte(pte_t pte, pte_t *ptep) · 7 unmodified lines | |||
| 59 | */ | 59 | */ |
| 60 | smp_rmb(); | 60 | smp_rmb(); |
| 61 | 61 | ||
| 62 | hidxp = (unsigned long *)(ptep + PTRS_PER_PTE); | 62 | hidxp = (unsigned long *)(ptep + offset); |
| 63 | rpte.hidx = *hidxp; | 63 | rpte.hidx = *hidxp; |
| 64 | return rpte; | 64 | return rpte; |
| 65 | } | 65 | } |
| @@ -86,9 +86,10 @@ static inline unsigned long __rpte_to_hidx(real_pte_t rpte, unsigned long index) · 20 unmodified lines | |||
| 86 | * expected to modify the PTE bits accordingly and commit the PTE to memory. | 86 | * expected to modify the PTE bits accordingly and commit the PTE to memory. |
| 87 | */ | 87 | */ |
| 88 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, | 88 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, |
| 89 | unsigned int subpg_index, unsigned long hidx) | 89 | unsigned int subpg_index, |
| 90 | unsigned long hidx, int offset) | ||
| 90 | { | 91 | { |
| 91 | unsigned long *hidxp = (unsigned long *)(ptep + PTRS_PER_PTE); | 92 | unsigned long *hidxp = (unsigned long *)(ptep + offset); |
| 92 | 93 | ||
| 93 | rpte.hidx &= ~HIDX_BITS(0xfUL, subpg_index); | 94 | rpte.hidx &= ~HIDX_BITS(0xfUL, subpg_index); |
| 94 | *hidxp = rpte.hidx | HIDX_BITS(HIDX_SHIFT_BY_ONE(hidx), subpg_index); | 95 | *hidxp = rpte.hidx | HIDX_BITS(HIDX_SHIFT_BY_ONE(hidx), subpg_index); |
| @@ -140,13 +141,18 @@ static inline int hash__remap_4k_pfn(struct vm_area_struct *vma, unsigned long a · 45 unmodified lines | |||
| 140 | } | 141 | } |
| 141 | 142 | ||
| 142 | #define H_PTE_TABLE_SIZE PTE_FRAG_SIZE | 143 | #define H_PTE_TABLE_SIZE PTE_FRAG_SIZE |
| 143 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE | 144 | #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined (CONFIG_HUGETLB_PAGE) |
| 144 | #define H_PMD_TABLE_SIZE ((sizeof(pmd_t) << PMD_INDEX_SIZE) + \ | 145 | #define H_PMD_TABLE_SIZE ((sizeof(pmd_t) << PMD_INDEX_SIZE) + \ |
| 145 | (sizeof(unsigned long) << PMD_INDEX_SIZE)) | 146 | (sizeof(unsigned long) << PMD_INDEX_SIZE)) |
| 146 | #else | 147 | #else |
| 147 | #define H_PMD_TABLE_SIZE (sizeof(pmd_t) << PMD_INDEX_SIZE) | 148 | #define H_PMD_TABLE_SIZE (sizeof(pmd_t) << PMD_INDEX_SIZE) |
| 148 | #endif | 149 | #endif |
| 150 | #ifdef CONFIG_HUGETLB_PAGE | ||
| 151 | #define H_PUD_TABLE_SIZE ((sizeof(pud_t) << PUD_INDEX_SIZE) + \ | ||
| 152 | (sizeof(unsigned long) << PUD_INDEX_SIZE)) | ||
| 153 | #else | ||
| 149 | #define H_PUD_TABLE_SIZE (sizeof(pud_t) << PUD_INDEX_SIZE) | 154 | #define H_PUD_TABLE_SIZE (sizeof(pud_t) << PUD_INDEX_SIZE) |
| 155 | #endif | ||
| 150 | #define H_PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE) | 156 | #define H_PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE) |
| 151 | 157 | ||
| 152 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE | 158 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| @@ -23,7 +23,8 @@ · 22 unmodified lines | |||
| 23 | H_PUD_INDEX_SIZE + H_PGD_INDEX_SIZE + PAGE_SHIFT) | 23 | H_PUD_INDEX_SIZE + H_PGD_INDEX_SIZE + PAGE_SHIFT) |
| 24 | #define H_PGTABLE_RANGE (ASM_CONST(1) << H_PGTABLE_EADDR_SIZE) | 24 | #define H_PGTABLE_RANGE (ASM_CONST(1) << H_PGTABLE_EADDR_SIZE) |
| 25 | 25 | ||
| 26 | #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_PPC_64K_PAGES) | 26 | #if (defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE)) && \ |
| 27 | defined(CONFIG_PPC_64K_PAGES) | ||
| 27 | /* | 28 | /* |
| 28 | * only with hash 64k we need to use the second half of pmd page table | 29 | * only with hash 64k we need to use the second half of pmd page table |
| 29 | * to store pointer to deposited pgtable_t | 30 | * to store pointer to deposited pgtable_t |
| @@ -32,6 +33,16 @@ · 2 unmodified lines | |||
| 32 | #else | 33 | #else |
| 33 | #define H_PMD_CACHE_INDEX H_PMD_INDEX_SIZE | 34 | #define H_PMD_CACHE_INDEX H_PMD_INDEX_SIZE |
| 34 | #endif | 35 | #endif |
| 36 | /* | ||
| 37 | * We store the slot details in the second half of page table. | ||
| 38 | * Increase the pud level table so that hugetlb ptes can be stored | ||
| 39 | * at pud level. | ||
| 40 | */ | ||
| 41 | #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_PPC_64K_PAGES) | ||
| 42 | #define H_PUD_CACHE_INDEX (H_PUD_INDEX_SIZE + 1) | ||
| 43 | #else | ||
| 44 | #define H_PUD_CACHE_INDEX (H_PUD_INDEX_SIZE) | ||
| 45 | #endif | ||
| 35 | /* | 46 | /* |
| 36 | * Define the address range of the kernel non-linear virtual area | 47 | * Define the address range of the kernel non-linear virtual area |
| 37 | */ | 48 | */ |
| @@ -73,10 +73,16 @@ static inline void radix__pgd_free(struct mm_struct *mm, pgd_t *pgd) · 72 unmodified lines | |||
| 73 | 73 | ||
| 74 | static inline pgd_t *pgd_alloc(struct mm_struct *mm) | 74 | static inline pgd_t *pgd_alloc(struct mm_struct *mm) |
| 75 | { | 75 | { |
| 76 | pgd_t *pgd; | ||
| 77 | |||
| 76 | if (radix_enabled()) | 78 | if (radix_enabled()) |
| 77 | return radix__pgd_alloc(mm); | 79 | return radix__pgd_alloc(mm); |
| 78 | return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), | 80 | |
| 79 | pgtable_gfp_flags(mm, GFP_KERNEL)); | 81 | pgd = kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), |
| 82 | pgtable_gfp_flags(mm, GFP_KERNEL)); | ||
| 83 | memset(pgd, 0, PGD_TABLE_SIZE); | ||
| 84 | |||
| 85 | return pgd; | ||
| 80 | } | 86 | } |
| 81 | 87 | ||
| 82 | static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) | 88 | static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) |
| @@ -93,13 +99,13 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud) · 10 unmodified lines | |||
| 93 | 99 | ||
| 94 | static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) | 100 | static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) |
| 95 | { | 101 | { |
| 96 | return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), | 102 | return kmem_cache_alloc(PGT_CACHE(PUD_CACHE_INDEX), |
| 97 | pgtable_gfp_flags(mm, GFP_KERNEL)); | 103 | pgtable_gfp_flags(mm, GFP_KERNEL)); |
| 98 | } | 104 | } |
| 99 | 105 | ||
| 100 | static inline void pud_free(struct mm_struct *mm, pud_t *pud) | 106 | static inline void pud_free(struct mm_struct *mm, pud_t *pud) |
| 101 | { | 107 | { |
| 102 | kmem_cache_free(PGT_CACHE(PUD_INDEX_SIZE), pud); | 108 | kmem_cache_free(PGT_CACHE(PUD_CACHE_INDEX), pud); |
| 103 | } | 109 | } |
| 104 | 110 | ||
| 105 | static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) | 111 | static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) |
| @@ -115,7 +121,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, · 9 unmodified lines | |||
| 115 | * ahead and flush the page walk cache | 121 | * ahead and flush the page walk cache |
| 116 | */ | 122 | */ |
| 117 | flush_tlb_pgtable(tlb, address); | 123 | flush_tlb_pgtable(tlb, address); |
| 118 | pgtable_free_tlb(tlb, pud, PUD_INDEX_SIZE); | 124 | pgtable_free_tlb(tlb, pud, PUD_CACHE_INDEX); |
| 119 | } | 125 | } |
| 120 | 126 | ||
| 121 | static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) | 127 | static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) |
| @@ -232,11 +232,13 @@ extern unsigned long __pmd_index_size; · 231 unmodified lines | |||
| 232 | extern unsigned long __pud_index_size; | 232 | extern unsigned long __pud_index_size; |
| 233 | extern unsigned long __pgd_index_size; | 233 | extern unsigned long __pgd_index_size; |
| 234 | extern unsigned long __pmd_cache_index; | 234 | extern unsigned long __pmd_cache_index; |
| 235 | extern unsigned long __pud_cache_index; | ||
| 235 | #define PTE_INDEX_SIZE __pte_index_size | 236 | #define PTE_INDEX_SIZE __pte_index_size |
| 236 | #define PMD_INDEX_SIZE __pmd_index_size | 237 | #define PMD_INDEX_SIZE __pmd_index_size |
| 237 | #define PUD_INDEX_SIZE __pud_index_size | 238 | #define PUD_INDEX_SIZE __pud_index_size |
| 238 | #define PGD_INDEX_SIZE __pgd_index_size | 239 | #define PGD_INDEX_SIZE __pgd_index_size |
| 239 | #define PMD_CACHE_INDEX __pmd_cache_index | 240 | #define PMD_CACHE_INDEX __pmd_cache_index |
| 241 | #define PUD_CACHE_INDEX __pud_cache_index | ||
| 240 | /* | 242 | /* |
| 241 | * Because of use of pte fragments and THP, size of page table | 243 | * Because of use of pte fragments and THP, size of page table |
| 242 | * are not always derived out of index size above. | 244 | * are not always derived out of index size above. |
| @@ -348,7 +350,7 @@ extern unsigned long pci_io_base; · 105 unmodified lines | |||
| 348 | */ | 350 | */ |
| 349 | #ifndef __real_pte | 351 | #ifndef __real_pte |
| 350 | 352 | ||
| 351 | #define __real_pte(e,p) ((real_pte_t){(e)}) | 353 | #define __real_pte(e, p, o) ((real_pte_t){(e)}) |
| 352 | #define __rpte_to_pte(r) ((r).pte) | 354 | #define __rpte_to_pte(r) ((r).pte) |
| 353 | #define __rpte_to_hidx(r,index) (pte_val(__rpte_to_pte(r)) >> H_PAGE_F_GIX_SHIFT) | 355 | #define __rpte_to_hidx(r,index) (pte_val(__rpte_to_pte(r)) >> H_PAGE_F_GIX_SHIFT) |
| 354 | 356 | ||
| @@ -645,7 +645,7 @@ END_FTR_SECTION_NESTED(ftr,ftr,943) · 644 unmodified lines | |||
| 645 | EXC_HV, SOFTEN_TEST_HV, bitmask) | 645 | EXC_HV, SOFTEN_TEST_HV, bitmask) |
| 646 | 646 | ||
| 647 | #define MASKABLE_RELON_EXCEPTION_HV_OOL(vec, label, bitmask) \ | 647 | #define MASKABLE_RELON_EXCEPTION_HV_OOL(vec, label, bitmask) \ |
| 648 | MASKABLE_EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_NOTEST_HV, vec, bitmask);\ | 648 | MASKABLE_EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_TEST_HV, vec, bitmask);\ |
| 649 | EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_HV) | 649 | EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_HV) |
| 650 | 650 | ||
| 651 | /* | 651 | /* |
| @@ -52,7 +52,7 @@ · 51 unmodified lines | |||
| 52 | #define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000) | 52 | #define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000) |
| 53 | #define FW_FEATURE_PRRN ASM_CONST(0x0000000200000000) | 53 | #define FW_FEATURE_PRRN ASM_CONST(0x0000000200000000) |
| 54 | #define FW_FEATURE_DRMEM_V2 ASM_CONST(0x0000000400000000) | 54 | #define FW_FEATURE_DRMEM_V2 ASM_CONST(0x0000000400000000) |
| 55 | #define FW_FEATURE_DRC_INFO ASM_CONST(0x0000000400000000) | 55 | #define FW_FEATURE_DRC_INFO ASM_CONST(0x0000000800000000) |
| 56 | 56 | ||
| 57 | #ifndef __ASSEMBLY__ | 57 | #ifndef __ASSEMBLY__ |
| 58 | 58 | ||
| @@ -29,6 +29,16 @@ · 28 unmodified lines | |||
| 29 | #define PACA_IRQ_HMI 0x20 | 29 | #define PACA_IRQ_HMI 0x20 |
| 30 | #define PACA_IRQ_PMI 0x40 | 30 | #define PACA_IRQ_PMI 0x40 |
| 31 | 31 | ||
| 32 | /* | ||
| 33 | * Some soft-masked interrupts must be hard masked until they are replayed | ||
| 34 | * (e.g., because the soft-masked handler does not clear the exception). | ||
| 35 | */ | ||
| 36 | #ifdef CONFIG_PPC_BOOK3S | ||
| 37 | #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE|PACA_IRQ_PMI) | ||
| 38 | #else | ||
| 39 | #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE) | ||
| 40 | #endif | ||
| 41 | |||
| 32 | /* | 42 | /* |
| 33 | * flags for paca->irq_soft_mask | 43 | * flags for paca->irq_soft_mask |
| 34 | */ | 44 | */ |
| @@ -244,7 +254,7 @@ static inline bool lazy_irq_pending(void) · 209 unmodified lines | |||
| 244 | static inline void may_hard_irq_enable(void) | 254 | static inline void may_hard_irq_enable(void) |
| 245 | { | 255 | { |
| 246 | get_paca()->irq_happened &= ~PACA_IRQ_HARD_DIS; | 256 | get_paca()->irq_happened &= ~PACA_IRQ_HARD_DIS; |
| 247 | if (!(get_paca()->irq_happened & PACA_IRQ_EE)) | 257 | if (!(get_paca()->irq_happened & PACA_IRQ_MUST_HARD_MASK)) |
| 248 | __hard_irq_enable(); | 258 | __hard_irq_enable(); |
| 249 | } | 259 | } |
| 250 | 260 | ||
| @@ -140,6 +140,12 @@ static inline bool kdump_in_progress(void) · 139 unmodified lines | |||
| 140 | return false; | 140 | return false; |
| 141 | } | 141 | } |
| 142 | 142 | ||
| 143 | static inline void crash_ipi_callback(struct pt_regs *regs) { } | ||
| 144 | |||
| 145 | static inline void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *)) | ||
| 146 | { | ||
| 147 | } | ||
| 148 | |||
| 143 | #endif /* CONFIG_KEXEC_CORE */ | 149 | #endif /* CONFIG_KEXEC_CORE */ |
| 144 | #endif /* ! __ASSEMBLY__ */ | 150 | #endif /* ! __ASSEMBLY__ */ |
| 145 | #endif /* __KERNEL__ */ | 151 | #endif /* __KERNEL__ */ |
| @@ -24,6 +24,7 @@ extern int icache_44x_need_flush; · 23 unmodified lines | |||
| 24 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) | 24 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) |
| 25 | 25 | ||
| 26 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE | 26 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE |
| 27 | #define PUD_CACHE_INDEX PUD_INDEX_SIZE | ||
| 27 | 28 | ||
| 28 | #ifndef __ASSEMBLY__ | 29 | #ifndef __ASSEMBLY__ |
| 29 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) | 30 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | |||
| 27 | #else | 27 | #else |
| 28 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE | 28 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE |
| 29 | #endif | 29 | #endif |
| 30 | #define PUD_CACHE_INDEX PUD_INDEX_SIZE | ||
| 30 | 31 | ||
| 31 | /* | 32 | /* |
| 32 | * Define the address range of the kernel non-linear virtual area | 33 | * Define the address range of the kernel non-linear virtual area |
| @@ -44,6 +44,11 @@ extern int sysfs_add_device_to_node(struct device *dev, int nid); · 43 unmodified lines | |||
| 44 | extern void sysfs_remove_device_from_node(struct device *dev, int nid); | 44 | extern void sysfs_remove_device_from_node(struct device *dev, int nid); |
| 45 | extern int numa_update_cpu_topology(bool cpus_locked); | 45 | extern int numa_update_cpu_topology(bool cpus_locked); |
| 46 | 46 | ||
| 47 | static inline void update_numa_cpu_lookup_table(unsigned int cpu, int node) | ||
| 48 | { | ||
| 49 | numa_cpu_lookup_table[cpu] = node; | ||
| 50 | } | ||
| 51 | |||
| 47 | static inline int early_cpu_to_node(int cpu) | 52 | static inline int early_cpu_to_node(int cpu) |
| 48 | { | 53 | { |
| 49 | int nid; | 54 | int nid; |
| @@ -76,12 +81,16 @@ static inline int numa_update_cpu_topology(bool cpus_locked) · 26 unmodified lines | |||
| 76 | { | 81 | { |
| 77 | return 0; | 82 | return 0; |
| 78 | } | 83 | } |
| 84 | |||
| 85 | static inline void update_numa_cpu_lookup_table(unsigned int cpu, int node) {} | ||
| 86 | |||
| 79 | #endif /* CONFIG_NUMA */ | 87 | #endif /* CONFIG_NUMA */ |
| 80 | 88 | ||
| 81 | #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR) | 89 | #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR) |
| 82 | extern int start_topology_update(void); | 90 | extern int start_topology_update(void); |
| 83 | extern int stop_topology_update(void); | 91 | extern int stop_topology_update(void); |
| 84 | extern int prrn_is_enabled(void); | 92 | extern int prrn_is_enabled(void); |
| 93 | extern int find_and_online_cpu_nid(int cpu); | ||
| 85 | #else | 94 | #else |
| 86 | static inline int start_topology_update(void) | 95 | static inline int start_topology_update(void) |
| 87 | { | 96 | { |
| @@ -95,6 +104,10 @@ static inline int prrn_is_enabled(void) · 7 unmodified lines | |||
| 95 | { | 104 | { |
| 96 | return 0; | 105 | return 0; |
| 97 | } | 106 | } |
| 107 | static inline int find_and_online_cpu_nid(int cpu) | ||
| 108 | { | ||
| 109 | return 0; | ||
| 110 | } | ||
| 98 | #endif /* CONFIG_NUMA && CONFIG_PPC_SPLPAR */ | 111 | #endif /* CONFIG_NUMA && CONFIG_PPC_SPLPAR */ |
| 99 | 112 | ||
| 100 | #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_NEED_MULTIPLE_NODES) | 113 | #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_NEED_MULTIPLE_NODES) |
| @@ -384,7 +384,8 @@ static void *eeh_report_resume(void *data, void *userdata) · 383 unmodified lines | |||
| 384 | eeh_pcid_put(dev); | 384 | eeh_pcid_put(dev); |
| 385 | pci_uevent_ers(dev, PCI_ERS_RESULT_RECOVERED); | 385 | pci_uevent_ers(dev, PCI_ERS_RESULT_RECOVERED); |
| 386 | #ifdef CONFIG_PCI_IOV | 386 | #ifdef CONFIG_PCI_IOV |
| 387 | eeh_ops->notify_resume(eeh_dev_to_pdn(edev)); | 387 | if (eeh_ops->notify_resume && eeh_dev_to_pdn(edev)) |
| 388 | eeh_ops->notify_resume(eeh_dev_to_pdn(edev)); | ||
| 388 | #endif | 389 | #endif |
| 389 | return NULL; | 390 | return NULL; |
| 390 | } | 391 | } |
| @@ -943,6 +943,8 @@ kernel_dbg_exc: · 942 unmodified lines | |||
| 943 | /* | 943 | /* |
| 944 | * An interrupt came in while soft-disabled; We mark paca->irq_happened | 944 | * An interrupt came in while soft-disabled; We mark paca->irq_happened |
| 945 | * accordingly and if the interrupt is level sensitive, we hard disable | 945 | * accordingly and if the interrupt is level sensitive, we hard disable |
| 946 | * hard disable (full_mask) corresponds to PACA_IRQ_MUST_HARD_MASK, so | ||
| 947 | * keep these in synch. | ||
| 946 | */ | 948 | */ |
| 947 | 949 | ||
| 948 | .macro masked_interrupt_book3e paca_irq full_mask | 950 | .macro masked_interrupt_book3e paca_irq full_mask |
| @@ -1426,7 +1426,7 @@ EXC_COMMON_BEGIN(soft_nmi_common) · 1425 unmodified lines | |||
| 1426 | * triggered and won't automatically refire. | 1426 | * triggered and won't automatically refire. |
| 1427 | * - If it was a HMI we return immediately since we handled it in realmode | 1427 | * - If it was a HMI we return immediately since we handled it in realmode |
| 1428 | * and it won't refire. | 1428 | * and it won't refire. |
| 1429 | * - else we hard disable and return. | 1429 | * - Else it is one of PACA_IRQ_MUST_HARD_MASK, so hard disable and return. |
| 1430 | * This is called with r10 containing the value to OR to the paca field. | 1430 | * This is called with r10 containing the value to OR to the paca field. |
| 1431 | */ | 1431 | */ |
| 1432 | #define MASKED_INTERRUPT(_H) \ | 1432 | #define MASKED_INTERRUPT(_H) \ |
| @@ -1441,8 +1441,8 @@ masked_##_H##interrupt: \ · 8 unmodified lines | |||
| 1441 | ori r10,r10,0xffff; \ | 1441 | ori r10,r10,0xffff; \ |
| 1442 | mtspr SPRN_DEC,r10; \ | 1442 | mtspr SPRN_DEC,r10; \ |
| 1443 | b MASKED_DEC_HANDLER_LABEL; \ | 1443 | b MASKED_DEC_HANDLER_LABEL; \ |
| 1444 | 1: andi. r10,r10,(PACA_IRQ_DBELL|PACA_IRQ_HMI); \ | 1444 | 1: andi. r10,r10,PACA_IRQ_MUST_HARD_MASK; \ |
| 1445 | bne 2f; \ | 1445 | beq 2f; \ |
| 1446 | mfspr r10,SPRN_##_H##SRR1; \ | 1446 | mfspr r10,SPRN_##_H##SRR1; \ |
| 1447 | xori r10,r10,MSR_EE; /* clear MSR_EE */ \ | 1447 | xori r10,r10,MSR_EE; /* clear MSR_EE */ \ |
| 1448 | mtspr SPRN_##_H##SRR1,r10; \ | 1448 | mtspr SPRN_##_H##SRR1,r10; \ |
| @@ -874,7 +874,6 @@ struct ibm_arch_vec __cacheline_aligned ibm_architecture_vec = { · 873 unmodified lines | |||
| 874 | .mmu = 0, | 874 | .mmu = 0, |
| 875 | .hash_ext = 0, | 875 | .hash_ext = 0, |
| 876 | .radix_ext = 0, | 876 | .radix_ext = 0, |
| 877 | .byte22 = OV5_FEAT(OV5_DRC_INFO), | ||
| 878 | }, | 877 | }, |
| 879 | 878 | ||
| 880 | /* option vector 6: IBM PAPR hints */ | 879 | /* option vector 6: IBM PAPR hints */ |
| @@ -788,7 +788,8 @@ static int register_cpu_online(unsigned int cpu) · 787 unmodified lines | |||
| 788 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) | 788 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) |
| 789 | device_create_file(s, &dev_attr_pir); | 789 | device_create_file(s, &dev_attr_pir); |
| 790 | 790 | ||
| 791 | if (cpu_has_feature(CPU_FTR_ARCH_206)) | 791 | if (cpu_has_feature(CPU_FTR_ARCH_206) && |
| 792 | !firmware_has_feature(FW_FEATURE_LPAR)) | ||
| 792 | device_create_file(s, &dev_attr_tscr); | 793 | device_create_file(s, &dev_attr_tscr); |
| 793 | #endif /* CONFIG_PPC64 */ | 794 | #endif /* CONFIG_PPC64 */ |
| 794 | 795 | ||
| @@ -873,7 +874,8 @@ static int unregister_cpu_online(unsigned int cpu) · 78 unmodified lines | |||
| 873 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) | 874 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) |
| 874 | device_remove_file(s, &dev_attr_pir); | 875 | device_remove_file(s, &dev_attr_pir); |
| 875 | 876 | ||
| 876 | if (cpu_has_feature(CPU_FTR_ARCH_206)) | 877 | if (cpu_has_feature(CPU_FTR_ARCH_206) && |
| 878 | !firmware_has_feature(FW_FEATURE_LPAR)) | ||
| 877 | device_remove_file(s, &dev_attr_tscr); | 879 | device_remove_file(s, &dev_attr_tscr); |
| 878 | #endif /* CONFIG_PPC64 */ | 880 | #endif /* CONFIG_PPC64 */ |
| 879 | 881 | ||
| @@ -195,6 +195,12 @@ static void kvmppc_pte_free(pte_t *ptep) · 194 unmodified lines | |||
| 195 | kmem_cache_free(kvm_pte_cache, ptep); | 195 | kmem_cache_free(kvm_pte_cache, ptep); |
| 196 | } | 196 | } |
| 197 | 197 | ||
| 198 | /* Like pmd_huge() and pmd_large(), but works regardless of config options */ | ||
| 199 | static inline int pmd_is_leaf(pmd_t pmd) | ||
| 200 | { | ||
| 201 | return !!(pmd_val(pmd) & _PAGE_PTE); | ||
| 202 | } | ||
| 203 | |||
| 198 | static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, | 204 | static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, |
| 199 | unsigned int level, unsigned long mmu_seq) | 205 | unsigned int level, unsigned long mmu_seq) |
| 200 | { | 206 | { |
| @@ -219,7 +225,7 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, · 18 unmodified lines | |||
| 219 | else | 225 | else |
| 220 | new_pmd = pmd_alloc_one(kvm->mm, gpa); | 226 | new_pmd = pmd_alloc_one(kvm->mm, gpa); |
| 221 | 227 | ||
| 222 | if (level == 0 && !(pmd && pmd_present(*pmd))) | 228 | if (level == 0 && !(pmd && pmd_present(*pmd) && !pmd_is_leaf(*pmd))) |
| 223 | new_ptep = kvmppc_pte_alloc(); | 229 | new_ptep = kvmppc_pte_alloc(); |
| 224 | 230 | ||
| 225 | /* Check if we might have been invalidated; let the guest retry if so */ | 231 | /* Check if we might have been invalidated; let the guest retry if so */ |
| @@ -244,12 +250,30 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, · 18 unmodified lines | |||
| 244 | new_pmd = NULL; | 250 | new_pmd = NULL; |
| 245 | } | 251 | } |
| 246 | pmd = pmd_offset(pud, gpa); | 252 | pmd = pmd_offset(pud, gpa); |
| 247 | if (pmd_large(*pmd)) { | 253 | if (pmd_is_leaf(*pmd)) { |
| 248 | /* Someone else has instantiated a large page here; retry */ | 254 | unsigned long lgpa = gpa & PMD_MASK; |
| 249 | ret = -EAGAIN; | 255 | |
| 250 | goto out_unlock; | 256 | /* |
| 251 | } | 257 | * If we raced with another CPU which has just put |
| 252 | if (level == 1 && !pmd_none(*pmd)) { | 258 | * a 2MB pte in after we saw a pte page, try again. |
| 259 | */ | ||
| 260 | if (level == 0 && !new_ptep) { | ||
| 261 | ret = -EAGAIN; | ||
| 262 | goto out_unlock; | ||
| 263 | } | ||
| 264 | /* Valid 2MB page here already, remove it */ | ||
| 265 | old = kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), | ||
| 266 | ~0UL, 0, lgpa, PMD_SHIFT); | ||
| 267 | kvmppc_radix_tlbie_page(kvm, lgpa, PMD_SHIFT); | ||
| 268 | if (old & _PAGE_DIRTY) { | ||
| 269 | unsigned long gfn = lgpa >> PAGE_SHIFT; | ||
| 270 | struct kvm_memory_slot *memslot; | ||
| 271 | memslot = gfn_to_memslot(kvm, gfn); | ||
| 272 | if (memslot && memslot->dirty_bitmap) | ||
| 273 | kvmppc_update_dirty_map(memslot, | ||
| 274 | gfn, PMD_SIZE); | ||
| 275 | } | ||
| 276 | } else if (level == 1 && !pmd_none(*pmd)) { | ||
| 253 | /* | 277 | /* |
| 254 | * There's a page table page here, but we wanted | 278 | * There's a page table page here, but we wanted |
| 255 | * to install a large page. Tell the caller and let | 279 | * to install a large page. Tell the caller and let |
| @@ -412,28 +436,24 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, · 156 unmodified lines | |||
| 412 | } else { | 436 | } else { |
| 413 | page = pages[0]; | 437 | page = pages[0]; |
| 414 | pfn = page_to_pfn(page); | 438 | pfn = page_to_pfn(page); |
| 415 | if (PageHuge(page)) { | 439 | if (PageCompound(page)) { |
| 416 | page = compound_head(page); | 440 | pte_size <<= compound_order(compound_head(page)); |
| 417 | pte_size <<= compound_order(page); | ||
| 418 | /* See if we can insert a 2MB large-page PTE here */ | 441 | /* See if we can insert a 2MB large-page PTE here */ |
| 419 | if (pte_size >= PMD_SIZE && | 442 | if (pte_size >= PMD_SIZE && |
| 420 | (gpa & PMD_MASK & PAGE_MASK) == | 443 | (gpa & (PMD_SIZE - PAGE_SIZE)) == |
| 421 | (hva & PMD_MASK & PAGE_MASK)) { | 444 | (hva & (PMD_SIZE - PAGE_SIZE))) { |
| 422 | level = 1; | 445 | level = 1; |
| 423 | pfn &= ~((PMD_SIZE >> PAGE_SHIFT) - 1); | 446 | pfn &= ~((PMD_SIZE >> PAGE_SHIFT) - 1); |
| 424 | } | 447 | } |
| 425 | } | 448 | } |
| 426 | /* See if we can provide write access */ | 449 | /* See if we can provide write access */ |
| 427 | if (writing) { | 450 | if (writing) { |
| 428 | /* | ||
| 429 | * We assume gup_fast has set dirty on the host PTE. | ||
| 430 | */ | ||
| 431 | pgflags |= _PAGE_WRITE; | 451 | pgflags |= _PAGE_WRITE; |
| 432 | } else { | 452 | } else { |
| 433 | local_irq_save(flags); | 453 | local_irq_save(flags); |
| 434 | ptep = find_current_mm_pte(current->mm->pgd, | 454 | ptep = find_current_mm_pte(current->mm->pgd, |
| 435 | hva, NULL, NULL); | 455 | hva, NULL, NULL); |
| 436 | if (ptep && pte_write(*ptep) && pte_dirty(*ptep)) | 456 | if (ptep && pte_write(*ptep)) |
| 437 | pgflags |= _PAGE_WRITE; | 457 | pgflags |= _PAGE_WRITE; |
| 438 | local_irq_restore(flags); | 458 | local_irq_restore(flags); |
| 439 | } | 459 | } |
| @@ -459,18 +479,15 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, · 19 unmodified lines | |||
| 459 | pte = pfn_pte(pfn, __pgprot(pgflags)); | 479 | pte = pfn_pte(pfn, __pgprot(pgflags)); |
| 460 | ret = kvmppc_create_pte(kvm, pte, gpa, level, mmu_seq); | 480 | ret = kvmppc_create_pte(kvm, pte, gpa, level, mmu_seq); |
| 461 | } | 481 | } |
| 462 | if (ret == 0 || ret == -EAGAIN) | ||
| 463 | ret = RESUME_GUEST; | ||
| 464 | 482 | ||
| 465 | if (page) { | 483 | if (page) { |
| 466 | /* | 484 | if (!ret && (pgflags & _PAGE_WRITE)) |
| 467 | * We drop pages[0] here, not page because page might | 485 | set_page_dirty_lock(page); |
| 468 | * have been set to the head page of a compound, but | 486 | put_page(page); |
| 469 | * we have to drop the reference on the correct tail | ||
| 470 | * page to match the get inside gup() | ||
| 471 | */ | ||
| 472 | put_page(pages[0]); | ||
| 473 | } | 487 | } |
| 488 | |||
| 489 | if (ret == 0 || ret == -EAGAIN) | ||
| 490 | ret = RESUME_GUEST; | ||
| 474 | return ret; | 491 | return ret; |
| 475 | } | 492 | } |
| 476 | 493 | ||
| @@ -644,7 +661,7 @@ void kvmppc_free_radix(struct kvm *kvm) · 167 unmodified lines | |||
| 644 | continue; | 661 | continue; |
| 645 | pmd = pmd_offset(pud, 0); | 662 | pmd = pmd_offset(pud, 0); |
| 646 | for (im = 0; im < PTRS_PER_PMD; ++im, ++pmd) { | 663 | for (im = 0; im < PTRS_PER_PMD; ++im, ++pmd) { |
| 647 | if (pmd_huge(*pmd)) { | 664 | if (pmd_is_leaf(*pmd)) { |
| 648 | pmd_clear(pmd); | 665 | pmd_clear(pmd); |
| 649 | continue; | 666 | continue; |
| 650 | } | 667 | } |
| @@ -2885,7 +2885,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) · 2884 unmodified lines | |||
| 2885 | */ | 2885 | */ |
| 2886 | trace_hardirqs_on(); | 2886 | trace_hardirqs_on(); |
| 2887 | 2887 | ||
| 2888 | guest_enter(); | 2888 | guest_enter_irqoff(); |
| 2889 | 2889 | ||
| 2890 | srcu_idx = srcu_read_lock(&vc->kvm->srcu); | 2890 | srcu_idx = srcu_read_lock(&vc->kvm->srcu); |
| 2891 | 2891 | ||
| @@ -2893,8 +2893,6 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) · 1 unmodified line | |||
| 2893 | 2893 | ||
| 2894 | srcu_read_unlock(&vc->kvm->srcu, srcu_idx); | 2894 | srcu_read_unlock(&vc->kvm->srcu, srcu_idx); |
| 2895 | 2895 | ||
| 2896 | guest_exit(); | ||
| 2897 | |||
| 2898 | trace_hardirqs_off(); | 2896 | trace_hardirqs_off(); |
| 2899 | set_irq_happened(trap); | 2897 | set_irq_happened(trap); |
| 2900 | 2898 | ||
| @@ -2937,6 +2935,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) · 36 unmodified lines | |||
| 2937 | kvmppc_set_host_core(pcpu); | 2935 | kvmppc_set_host_core(pcpu); |
| 2938 | 2936 | ||
| 2939 | local_irq_enable(); | 2937 | local_irq_enable(); |
| 2938 | guest_exit(); | ||
| 2940 | 2939 | ||
| 2941 | /* Let secondaries go back to the offline loop */ | 2940 | /* Let secondaries go back to the offline loop */ |
| 2942 | for (i = 0; i < controlled_threads; ++i) { | 2941 | for (i = 0; i < controlled_threads; ++i) { |
| @@ -3656,15 +3655,17 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu) · 713 unmodified lines | |||
| 3656 | goto up_out; | 3655 | goto up_out; |
| 3657 | 3656 | ||
| 3658 | psize = vma_kernel_pagesize(vma); | 3657 | psize = vma_kernel_pagesize(vma); |
| 3659 | porder = __ilog2(psize); | ||
| 3660 | 3658 | ||
| 3661 | up_read(¤t->mm->mmap_sem); | 3659 | up_read(¤t->mm->mmap_sem); |
| 3662 | 3660 | ||
| 3663 | /* We can handle 4k, 64k or 16M pages in the VRMA */ | 3661 | /* We can handle 4k, 64k or 16M pages in the VRMA */ |
| 3664 | err = -EINVAL; | 3662 | if (psize >= 0x1000000) |
| 3665 | if (!(psize == 0x1000 || psize == 0x10000 || | 3663 | psize = 0x1000000; |
| 3666 | psize == 0x1000000)) | 3664 | else if (psize >= 0x10000) |
| 3667 | goto out_srcu; | 3665 | psize = 0x10000; |
| 3666 | else | ||
| 3667 | psize = 0x1000; | ||
| 3668 | porder = __ilog2(psize); | ||
| 3668 | 3669 | ||
| 3669 | senc = slb_pgsize_encoding(psize); | 3670 | senc = slb_pgsize_encoding(psize); |
| 3670 | kvm->arch.vrma_slb_v = senc | SLB_VSID_B_1T | | 3671 | kvm->arch.vrma_slb_v = senc | SLB_VSID_B_1T | |
| @@ -320,7 +320,6 @@ kvm_novcpu_exit: · 319 unmodified lines | |||
| 320 | stw r12, STACK_SLOT_TRAP(r1) | 320 | stw r12, STACK_SLOT_TRAP(r1) |
| 321 | bl kvmhv_commence_exit | 321 | bl kvmhv_commence_exit |
| 322 | nop | 322 | nop |
| 323 | lwz r12, STACK_SLOT_TRAP(r1) | ||
| 324 | b kvmhv_switch_to_host | 323 | b kvmhv_switch_to_host |
| 325 | 324 | ||
| 326 | /* | 325 | /* |
| @@ -1220,6 +1219,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) · 893 unmodified lines | |||
| 1220 | 1219 | ||
| 1221 | secondary_too_late: | 1220 | secondary_too_late: |
| 1222 | li r12, 0 | 1221 | li r12, 0 |
| 1222 | stw r12, STACK_SLOT_TRAP(r1) | ||
| 1223 | cmpdi r4, 0 | 1223 | cmpdi r4, 0 |
| 1224 | beq 11f | 1224 | beq 11f |
| 1225 | stw r12, VCPU_TRAP(r4) | 1225 | stw r12, VCPU_TRAP(r4) |
| @@ -1558,12 +1558,12 @@ mc_cont: · 332 unmodified lines | |||
| 1558 | 3: stw r5,VCPU_SLB_MAX(r9) | 1558 | 3: stw r5,VCPU_SLB_MAX(r9) |
| 1559 | 1559 | ||
| 1560 | guest_bypass: | 1560 | guest_bypass: |
| 1561 | stw r12, STACK_SLOT_TRAP(r1) | ||
| 1561 | mr r3, r12 | 1562 | mr r3, r12 |
| 1562 | /* Increment exit count, poke other threads to exit */ | 1563 | /* Increment exit count, poke other threads to exit */ |
| 1563 | bl kvmhv_commence_exit | 1564 | bl kvmhv_commence_exit |
| 1564 | nop | 1565 | nop |
| 1565 | ld r9, HSTATE_KVM_VCPU(r13) | 1566 | ld r9, HSTATE_KVM_VCPU(r13) |
| 1566 | lwz r12, VCPU_TRAP(r9) | ||
| 1567 | 1567 | ||
| 1568 | /* Stop others sending VCPU interrupts to this physical CPU */ | 1568 | /* Stop others sending VCPU interrupts to this physical CPU */ |
| 1569 | li r0, -1 | 1569 | li r0, -1 |
| @@ -1898,6 +1898,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_POWER9_DD1) · 328 unmodified lines | |||
| 1898 | * POWER7/POWER8 guest -> host partition switch code. | 1898 | * POWER7/POWER8 guest -> host partition switch code. |
| 1899 | * We don't have to lock against tlbies but we do | 1899 | * We don't have to lock against tlbies but we do |
| 1900 | * have to coordinate the hardware threads. | 1900 | * have to coordinate the hardware threads. |
| 1901 | * Here STACK_SLOT_TRAP(r1) contains the trap number. | ||
| 1901 | */ | 1902 | */ |
| 1902 | kvmhv_switch_to_host: | 1903 | kvmhv_switch_to_host: |
| 1903 | /* Secondary threads wait for primary to do partition switch */ | 1904 | /* Secondary threads wait for primary to do partition switch */ |
| @@ -1950,12 +1951,12 @@ BEGIN_FTR_SECTION · 46 unmodified lines | |||
| 1950 | END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S) | 1951 | END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S) |
| 1951 | 1952 | ||
| 1952 | /* If HMI, call kvmppc_realmode_hmi_handler() */ | 1953 | /* If HMI, call kvmppc_realmode_hmi_handler() */ |
| 1954 | lwz r12, STACK_SLOT_TRAP(r1) | ||
| 1953 | cmpwi r12, BOOK3S_INTERRUPT_HMI | 1955 | cmpwi r12, BOOK3S_INTERRUPT_HMI |
| 1954 | bne 27f | 1956 | bne 27f |
| 1955 | bl kvmppc_realmode_hmi_handler | 1957 | bl kvmppc_realmode_hmi_handler |
| 1956 | nop | 1958 | nop |
| 1957 | cmpdi r3, 0 | 1959 | cmpdi r3, 0 |
| 1958 | li r12, BOOK3S_INTERRUPT_HMI | ||
| 1959 | /* | 1960 | /* |
| 1960 | * At this point kvmppc_realmode_hmi_handler may have resync-ed | 1961 | * At this point kvmppc_realmode_hmi_handler may have resync-ed |
| 1961 | * the TB, and if it has, we must not subtract the guest timebase | 1962 | * the TB, and if it has, we must not subtract the guest timebase |
| @@ -2008,10 +2009,8 @@ BEGIN_FTR_SECTION · 46 unmodified lines | |||
| 2008 | lwz r8, KVM_SPLIT_DO_RESTORE(r3) | 2009 | lwz r8, KVM_SPLIT_DO_RESTORE(r3) |
| 2009 | cmpwi r8, 0 | 2010 | cmpwi r8, 0 |
| 2010 | beq 47f | 2011 | beq 47f |
| 2011 | stw r12, STACK_SLOT_TRAP(r1) | ||
| 2012 | bl kvmhv_p9_restore_lpcr | 2012 | bl kvmhv_p9_restore_lpcr |
| 2013 | nop | 2013 | nop |
| 2014 | lwz r12, STACK_SLOT_TRAP(r1) | ||
| 2015 | b 48f | 2014 | b 48f |
| 2016 | 47: | 2015 | 47: |
| 2017 | END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) | 2016 | END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) |
| @@ -2049,6 +2048,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX) · 31 unmodified lines | |||
| 2049 | li r0, KVM_GUEST_MODE_NONE | 2048 | li r0, KVM_GUEST_MODE_NONE |
| 2050 | stb r0, HSTATE_IN_GUEST(r13) | 2049 | stb r0, HSTATE_IN_GUEST(r13) |
| 2051 | 2050 | ||
| 2051 | lwz r12, STACK_SLOT_TRAP(r1) /* return trap # in r12 */ | ||
| 2052 | ld r0, SFS+PPC_LR_STKOFF(r1) | 2052 | ld r0, SFS+PPC_LR_STKOFF(r1) |
| 2053 | addi r1, r1, SFS | 2053 | addi r1, r1, SFS |
| 2054 | mtlr r0 | 2054 | mtlr r0 |
| @@ -188,7 +188,7 @@ static int xive_provision_queue(struct kvm_vcpu *vcpu, u8 prio) · 187 unmodified lines | |||
| 188 | if (!qpage) { | 188 | if (!qpage) { |
| 189 | pr_err("Failed to allocate queue %d for VCPU %d\n", | 189 | pr_err("Failed to allocate queue %d for VCPU %d\n", |
| 190 | prio, xc->server_num); | 190 | prio, xc->server_num); |
| 191 | return -ENOMEM;; | 191 | return -ENOMEM; |
| 192 | } | 192 | } |
| 193 | memset(qpage, 0, 1 << xive->q_order); | 193 | memset(qpage, 0, 1 << xive->q_order); |
| 194 | 194 | ||
| @@ -1345,7 +1345,7 @@ static int kvmppc_emulate_mmio_vsx_loadstore(struct kvm_vcpu *vcpu, · 1344 unmodified lines | |||
| 1345 | int kvmppc_handle_load128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, | 1345 | int kvmppc_handle_load128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, |
| 1346 | unsigned int rt, int is_default_endian) | 1346 | unsigned int rt, int is_default_endian) |
| 1347 | { | 1347 | { |
| 1348 | enum emulation_result emulated; | 1348 | enum emulation_result emulated = EMULATE_DONE; |
| 1349 | 1349 | ||
| 1350 | while (vcpu->arch.mmio_vmx_copy_nums) { | 1350 | while (vcpu->arch.mmio_vmx_copy_nums) { |
| 1351 | emulated = __kvmppc_handle_load(run, vcpu, rt, 8, | 1351 | emulated = __kvmppc_handle_load(run, vcpu, rt, 8, |
| @@ -1608,7 +1608,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · 256 unmodified lines | |||
| 1608 | 1608 | ||
| 1609 | kvm_sigset_deactivate(vcpu); | 1609 | kvm_sigset_deactivate(vcpu); |
| 1610 | 1610 | ||
| 1611 | #ifdef CONFIG_ALTIVEC | ||
| 1611 | out: | 1612 | out: |
| 1613 | #endif | ||
| 1612 | vcpu_put(vcpu); | 1614 | vcpu_put(vcpu); |
| 1613 | return r; | 1615 | return r; |
| 1614 | } | 1616 | } |
| @@ -98,7 +98,7 @@ static void init_drconf_v2_cell(struct of_drconf_cell_v2 *dr_cell, · 97 unmodified lines | |||
| 98 | dr_cell->base_addr = cpu_to_be64(lmb->base_addr); | 98 | dr_cell->base_addr = cpu_to_be64(lmb->base_addr); |
| 99 | dr_cell->drc_index = cpu_to_be32(lmb->drc_index); | 99 | dr_cell->drc_index = cpu_to_be32(lmb->drc_index); |
| 100 | dr_cell->aa_index = cpu_to_be32(lmb->aa_index); | 100 | dr_cell->aa_index = cpu_to_be32(lmb->aa_index); |
| 101 | dr_cell->flags = cpu_to_be32(lmb->flags); | 101 | dr_cell->flags = cpu_to_be32(drmem_lmb_flags(lmb)); |
| 102 | } | 102 | } |
| 103 | 103 | ||
| 104 | static int drmem_update_dt_v2(struct device_node *memory, | 104 | static int drmem_update_dt_v2(struct device_node *memory, |
| @@ -121,7 +121,7 @@ static int drmem_update_dt_v2(struct device_node *memory, · 16 unmodified lines | |||
| 121 | } | 121 | } |
| 122 | 122 | ||
| 123 | if (prev_lmb->aa_index != lmb->aa_index || | 123 | if (prev_lmb->aa_index != lmb->aa_index || |
| 124 | prev_lmb->flags != lmb->flags) | 124 | drmem_lmb_flags(prev_lmb) != drmem_lmb_flags(lmb)) |
| 125 | lmb_sets++; | 125 | lmb_sets++; |
| 126 | 126 | ||
| 127 | prev_lmb = lmb; | 127 | prev_lmb = lmb; |
| @@ -150,7 +150,7 @@ static int drmem_update_dt_v2(struct device_node *memory, · 22 unmodified lines | |||
| 150 | } | 150 | } |
| 151 | 151 | ||
| 152 | if (prev_lmb->aa_index != lmb->aa_index || | 152 | if (prev_lmb->aa_index != lmb->aa_index || |
| 153 | prev_lmb->flags != lmb->flags) { | 153 | drmem_lmb_flags(prev_lmb) != drmem_lmb_flags(lmb)) { |
| 154 | /* end of one set, start of another */ | 154 | /* end of one set, start of another */ |
| 155 | dr_cell->seq_lmbs = cpu_to_be32(seq_lmbs); | 155 | dr_cell->seq_lmbs = cpu_to_be32(seq_lmbs); |
| 156 | dr_cell++; | 156 | dr_cell++; |
| @@ -216,6 +216,8 @@ static void __init __walk_drmem_v1_lmbs(const __be32 *prop, const __be32 *usm, · 59 unmodified lines | |||
| 216 | u32 i, n_lmbs; | 216 | u32 i, n_lmbs; |
| 217 | 217 | ||
| 218 | n_lmbs = of_read_number(prop++, 1); | 218 | n_lmbs = of_read_number(prop++, 1); |
| 219 | if (n_lmbs == 0) | ||
| 220 | return; | ||
| 219 | 221 | ||
| 220 | for (i = 0; i < n_lmbs; i++) { | 222 | for (i = 0; i < n_lmbs; i++) { |
| 221 | read_drconf_v1_cell(&lmb, &prop); | 223 | read_drconf_v1_cell(&lmb, &prop); |
| @@ -245,6 +247,8 @@ static void __init __walk_drmem_v2_lmbs(const __be32 *prop, const __be32 *usm, · 23 unmodified lines | |||
| 245 | u32 i, j, lmb_sets; | 247 | u32 i, j, lmb_sets; |
| 246 | 248 | ||
| 247 | lmb_sets = of_read_number(prop++, 1); | 249 | lmb_sets = of_read_number(prop++, 1); |
| 250 | if (lmb_sets == 0) | ||
| 251 | return; | ||
| 248 | 252 | ||
| 249 | for (i = 0; i < lmb_sets; i++) { | 253 | for (i = 0; i < lmb_sets; i++) { |
| 250 | read_drconf_v2_cell(&dr_cell, &prop); | 254 | read_drconf_v2_cell(&dr_cell, &prop); |
| @@ -354,6 +358,8 @@ static void __init init_drmem_v1_lmbs(const __be32 *prop) · 103 unmodified lines | |||
| 354 | struct drmem_lmb *lmb; | 358 | struct drmem_lmb *lmb; |
| 355 | 359 | ||
| 356 | drmem_info->n_lmbs = of_read_number(prop++, 1); | 360 | drmem_info->n_lmbs = of_read_number(prop++, 1); |
| 361 | if (drmem_info->n_lmbs == 0) | ||
| 362 | return; | ||
| 357 | 363 | ||
| 358 | drmem_info->lmbs = kcalloc(drmem_info->n_lmbs, sizeof(*lmb), | 364 | drmem_info->lmbs = kcalloc(drmem_info->n_lmbs, sizeof(*lmb), |
| 359 | GFP_KERNEL); | 365 | GFP_KERNEL); |
| @@ -373,6 +379,8 @@ static void __init init_drmem_v2_lmbs(const __be32 *prop) · 13 unmodified lines | |||
| 373 | int lmb_index; | 379 | int lmb_index; |
| 374 | 380 | ||
| 375 | lmb_sets = of_read_number(prop++, 1); | 381 | lmb_sets = of_read_number(prop++, 1); |
| 382 | if (lmb_sets == 0) | ||
| 383 | return; | ||
| 376 | 384 | ||
| 377 | /* first pass, calculate the number of LMBs */ | 385 | /* first pass, calculate the number of LMBs */ |
| 378 | p = prop; | 386 | p = prop; |
| @@ -55,7 +55,7 @@ int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, · 54 unmodified lines | |||
| 55 | * need to add in 0x1 if it's a read-only user page | 55 | * need to add in 0x1 if it's a read-only user page |
| 56 | */ | 56 | */ |
| 57 | rflags = htab_convert_pte_flags(new_pte); | 57 | rflags = htab_convert_pte_flags(new_pte); |
| 58 | rpte = __real_pte(__pte(old_pte), ptep); | 58 | rpte = __real_pte(__pte(old_pte), ptep, PTRS_PER_PTE); |
| 59 | 59 | ||
| 60 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && | 60 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && |
| 61 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) | 61 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) |
| @@ -117,7 +117,7 @@ repeat: · 55 unmodified lines | |||
| 117 | return -1; | 117 | return -1; |
| 118 | } | 118 | } |
| 119 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; | 119 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; |
| 120 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot); | 120 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot, PTRS_PER_PTE); |
| 121 | } | 121 | } |
| 122 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); | 122 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); |
| 123 | return 0; | 123 | return 0; |
| @@ -86,7 +86,7 @@ int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, · 85 unmodified lines | |||
| 86 | 86 | ||
| 87 | subpg_index = (ea & (PAGE_SIZE - 1)) >> shift; | 87 | subpg_index = (ea & (PAGE_SIZE - 1)) >> shift; |
| 88 | vpn = hpt_vpn(ea, vsid, ssize); | 88 | vpn = hpt_vpn(ea, vsid, ssize); |
| 89 | rpte = __real_pte(__pte(old_pte), ptep); | 89 | rpte = __real_pte(__pte(old_pte), ptep, PTRS_PER_PTE); |
| 90 | /* | 90 | /* |
| 91 | *None of the sub 4k page is hashed | 91 | *None of the sub 4k page is hashed |
| 92 | */ | 92 | */ |
| @@ -214,7 +214,7 @@ repeat: · 121 unmodified lines | |||
| 214 | return -1; | 214 | return -1; |
| 215 | } | 215 | } |
| 216 | 216 | ||
| 217 | new_pte |= pte_set_hidx(ptep, rpte, subpg_index, slot); | 217 | new_pte |= pte_set_hidx(ptep, rpte, subpg_index, slot, PTRS_PER_PTE); |
| 218 | new_pte |= H_PAGE_HASHPTE; | 218 | new_pte |= H_PAGE_HASHPTE; |
| 219 | 219 | ||
| 220 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); | 220 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); |
| @@ -262,7 +262,7 @@ int __hash_page_64K(unsigned long ea, unsigned long access, · 41 unmodified lines | |||
| 262 | } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); | 262 | } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); |
| 263 | 263 | ||
| 264 | rflags = htab_convert_pte_flags(new_pte); | 264 | rflags = htab_convert_pte_flags(new_pte); |
| 265 | rpte = __real_pte(__pte(old_pte), ptep); | 265 | rpte = __real_pte(__pte(old_pte), ptep, PTRS_PER_PTE); |
| 266 | 266 | ||
| 267 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && | 267 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && |
| 268 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) | 268 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) |
| @@ -327,7 +327,7 @@ repeat: · 58 unmodified lines | |||
| 327 | } | 327 | } |
| 328 | 328 | ||
| 329 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; | 329 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; |
| 330 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot); | 330 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot, PTRS_PER_PTE); |
| 331 | } | 331 | } |
| 332 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); | 332 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); |
| 333 | return 0; | 333 | return 0; |
| @@ -1008,6 +1008,7 @@ void __init hash__early_init_mmu(void) · 1007 unmodified lines | |||
| 1008 | __pmd_index_size = H_PMD_INDEX_SIZE; | 1008 | __pmd_index_size = H_PMD_INDEX_SIZE; |
| 1009 | __pud_index_size = H_PUD_INDEX_SIZE; | 1009 | __pud_index_size = H_PUD_INDEX_SIZE; |
| 1010 | __pgd_index_size = H_PGD_INDEX_SIZE; | 1010 | __pgd_index_size = H_PGD_INDEX_SIZE; |
| 1011 | __pud_cache_index = H_PUD_CACHE_INDEX; | ||
| 1011 | __pmd_cache_index = H_PMD_CACHE_INDEX; | 1012 | __pmd_cache_index = H_PMD_CACHE_INDEX; |
| 1012 | __pte_table_size = H_PTE_TABLE_SIZE; | 1013 | __pte_table_size = H_PTE_TABLE_SIZE; |
| 1013 | __pmd_table_size = H_PMD_TABLE_SIZE; | 1014 | __pmd_table_size = H_PMD_TABLE_SIZE; |
| @@ -27,7 +27,7 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 26 unmodified lines | |||
| 27 | unsigned long vpn; | 27 | unsigned long vpn; |
| 28 | unsigned long old_pte, new_pte; | 28 | unsigned long old_pte, new_pte; |
| 29 | unsigned long rflags, pa, sz; | 29 | unsigned long rflags, pa, sz; |
| 30 | long slot; | 30 | long slot, offset; |
| 31 | 31 | ||
| 32 | BUG_ON(shift != mmu_psize_defs[mmu_psize].shift); | 32 | BUG_ON(shift != mmu_psize_defs[mmu_psize].shift); |
| 33 | 33 | ||
| @@ -63,7 +63,11 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 29 unmodified lines | |||
| 63 | } while(!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); | 63 | } while(!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); |
| 64 | 64 | ||
| 65 | rflags = htab_convert_pte_flags(new_pte); | 65 | rflags = htab_convert_pte_flags(new_pte); |
| 66 | rpte = __real_pte(__pte(old_pte), ptep); | 66 | if (unlikely(mmu_psize == MMU_PAGE_16G)) |
| 67 | offset = PTRS_PER_PUD; | ||
| 68 | else | ||
| 69 | offset = PTRS_PER_PMD; | ||
| 70 | rpte = __real_pte(__pte(old_pte), ptep, offset); | ||
| 67 | 71 | ||
| 68 | sz = ((1UL) << shift); | 72 | sz = ((1UL) << shift); |
| 69 | if (!cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) | 73 | if (!cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) |
| @@ -104,7 +108,7 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 34 unmodified lines | |||
| 104 | return -1; | 108 | return -1; |
| 105 | } | 109 | } |
| 106 | 110 | ||
| 107 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot); | 111 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot, offset); |
| 108 | } | 112 | } |
| 109 | 113 | ||
| 110 | /* | 114 | /* |
| @@ -100,6 +100,6 @@ void pgtable_cache_init(void) · 99 unmodified lines | |||
| 100 | * same size as either the pgd or pmd index except with THP enabled | 100 | * same size as either the pgd or pmd index except with THP enabled |
| 101 | * on book3s 64 | 101 | * on book3s 64 |
| 102 | */ | 102 | */ |
| 103 | if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE)) | 103 | if (PUD_CACHE_INDEX && !PGT_CACHE(PUD_CACHE_INDEX)) |
| 104 | pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor); | 104 | pgtable_cache_add(PUD_CACHE_INDEX, pud_ctor); |
| 105 | } | 105 | } |
| @@ -143,11 +143,6 @@ static void reset_numa_cpu_lookup_table(void) · 142 unmodified lines | |||
| 143 | numa_cpu_lookup_table[cpu] = -1; | 143 | numa_cpu_lookup_table[cpu] = -1; |
| 144 | } | 144 | } |
| 145 | 145 | ||
| 146 | static void update_numa_cpu_lookup_table(unsigned int cpu, int node) | ||
| 147 | { | ||
| 148 | numa_cpu_lookup_table[cpu] = node; | ||
| 149 | } | ||
| 150 | |||
| 151 | static void map_cpu_to_node(int cpu, int node) | 146 | static void map_cpu_to_node(int cpu, int node) |
| 152 | { | 147 | { |
| 153 | update_numa_cpu_lookup_table(cpu, node); | 148 | update_numa_cpu_lookup_table(cpu, node); |
| @@ -17,9 +17,11 @@ · 16 unmodified lines | |||
| 17 | #include <linux/of_fdt.h> | 17 | #include <linux/of_fdt.h> |
| 18 | #include <linux/mm.h> | 18 | #include <linux/mm.h> |
| 19 | #include <linux/string_helpers.h> | 19 | #include <linux/string_helpers.h> |
| 20 | #include <linux/stop_machine.h> | ||
| 20 | 21 | ||
| 21 | #include <asm/pgtable.h> | 22 | #include <asm/pgtable.h> |
| 22 | #include <asm/pgalloc.h> | 23 | #include <asm/pgalloc.h> |
| 24 | #include <asm/mmu_context.h> | ||
| 23 | #include <asm/dma.h> | 25 | #include <asm/dma.h> |
| 24 | #include <asm/machdep.h> | 26 | #include <asm/machdep.h> |
| 25 | #include <asm/mmu.h> | 27 | #include <asm/mmu.h> |
| @@ -333,6 +335,22 @@ static void __init radix_init_pgtable(void) · 307 unmodified lines | |||
| 333 | "r" (TLBIEL_INVAL_SET_LPID), "r" (0)); | 335 | "r" (TLBIEL_INVAL_SET_LPID), "r" (0)); |
| 334 | asm volatile("eieio; tlbsync; ptesync" : : : "memory"); | 336 | asm volatile("eieio; tlbsync; ptesync" : : : "memory"); |
| 335 | trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1); | 337 | trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1); |
| 338 | |||
| 339 | /* | ||
| 340 | * The init_mm context is given the first available (non-zero) PID, | ||
| 341 | * which is the "guard PID" and contains no page table. PIDR should | ||
| 342 | * never be set to zero because that duplicates the kernel address | ||
| 343 | * space at the 0x0... offset (quadrant 0)! | ||
| 344 | * | ||
| 345 | * An arbitrary PID that may later be allocated by the PID allocator | ||
| 346 | * for userspace processes must not be used either, because that | ||
| 347 | * would cause stale user mappings for that PID on CPUs outside of | ||
| 348 | * the TLB invalidation scheme (because it won't be in mm_cpumask). | ||
| 349 | * | ||
| 350 | * So permanently carve out one PID for the purpose of a guard PID. | ||
| 351 | */ | ||
| 352 | init_mm.context.id = mmu_base_pid; | ||
| 353 | mmu_base_pid++; | ||
| 336 | } | 354 | } |
| 337 | 355 | ||
| 338 | static void __init radix_init_partition_table(void) | 356 | static void __init radix_init_partition_table(void) |
| @@ -535,6 +553,7 @@ void __init radix__early_init_mmu(void) · 196 unmodified lines | |||
| 535 | __pmd_index_size = RADIX_PMD_INDEX_SIZE; | 553 | __pmd_index_size = RADIX_PMD_INDEX_SIZE; |
| 536 | __pud_index_size = RADIX_PUD_INDEX_SIZE; | 554 | __pud_index_size = RADIX_PUD_INDEX_SIZE; |
| 537 | __pgd_index_size = RADIX_PGD_INDEX_SIZE; | 555 | __pgd_index_size = RADIX_PGD_INDEX_SIZE; |
| 556 | __pud_cache_index = RADIX_PUD_INDEX_SIZE; | ||
| 538 | __pmd_cache_index = RADIX_PMD_INDEX_SIZE; | 557 | __pmd_cache_index = RADIX_PMD_INDEX_SIZE; |
| 539 | __pte_table_size = RADIX_PTE_TABLE_SIZE; | 558 | __pte_table_size = RADIX_PTE_TABLE_SIZE; |
| 540 | __pmd_table_size = RADIX_PMD_TABLE_SIZE; | 559 | __pmd_table_size = RADIX_PMD_TABLE_SIZE; |
| @@ -579,7 +598,8 @@ void __init radix__early_init_mmu(void) · 38 unmodified lines | |||
| 579 | 598 | ||
| 580 | radix_init_iamr(); | 599 | radix_init_iamr(); |
| 581 | radix_init_pgtable(); | 600 | radix_init_pgtable(); |
| 582 | 601 | /* Switch to the guard PID before turning on MMU */ | |
| 602 | radix__switch_mmu_context(NULL, &init_mm); | ||
| 583 | if (cpu_has_feature(CPU_FTR_HVMODE)) | 603 | if (cpu_has_feature(CPU_FTR_HVMODE)) |
| 584 | tlbiel_all(); | 604 | tlbiel_all(); |
| 585 | } | 605 | } |
| @@ -604,6 +624,7 @@ void radix__early_init_mmu_secondary(void) · 18 unmodified lines | |||
| 604 | } | 624 | } |
| 605 | radix_init_iamr(); | 625 | radix_init_iamr(); |
| 606 | 626 | ||
| 627 | radix__switch_mmu_context(NULL, &init_mm); | ||
| 607 | if (cpu_has_feature(CPU_FTR_HVMODE)) | 628 | if (cpu_has_feature(CPU_FTR_HVMODE)) |
| 608 | tlbiel_all(); | 629 | tlbiel_all(); |
| 609 | } | 630 | } |
| @@ -666,6 +687,30 @@ static void free_pmd_table(pmd_t *pmd_start, pud_t *pud) · 56 unmodified lines | |||
| 666 | pud_clear(pud); | 687 | pud_clear(pud); |
| 667 | } | 688 | } |
| 668 | 689 | ||
| 690 | struct change_mapping_params { | ||
| 691 | pte_t *pte; | ||
| 692 | unsigned long start; | ||
| 693 | unsigned long end; | ||
| 694 | unsigned long aligned_start; | ||
| 695 | unsigned long aligned_end; | ||
| 696 | }; | ||
| 697 | |||
| 698 | static int stop_machine_change_mapping(void *data) | ||
| 699 | { | ||
| 700 | struct change_mapping_params *params = | ||
| 701 | (struct change_mapping_params *)data; | ||
| 702 | |||
| 703 | if (!data) | ||
| 704 | return -1; | ||
| 705 | |||
| 706 | spin_unlock(&init_mm.page_table_lock); | ||
| 707 | pte_clear(&init_mm, params->aligned_start, params->pte); | ||
| 708 | create_physical_mapping(params->aligned_start, params->start); | ||
| 709 | create_physical_mapping(params->end, params->aligned_end); | ||
| 710 | spin_lock(&init_mm.page_table_lock); | ||
| 711 | return 0; | ||
| 712 | } | ||
| 713 | |||
| 669 | static void remove_pte_table(pte_t *pte_start, unsigned long addr, | 714 | static void remove_pte_table(pte_t *pte_start, unsigned long addr, |
| 670 | unsigned long end) | 715 | unsigned long end) |
| 671 | { | 716 | { |
| @@ -694,6 +739,52 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr, · 22 unmodified lines | |||
| 694 | } | 739 | } |
| 695 | } | 740 | } |
| 696 | 741 | ||
| 742 | /* | ||
| 743 | * clear the pte and potentially split the mapping helper | ||
| 744 | */ | ||
| 745 | static void split_kernel_mapping(unsigned long addr, unsigned long end, | ||
| 746 | unsigned long size, pte_t *pte) | ||
| 747 | { | ||
| 748 | unsigned long mask = ~(size - 1); | ||
| 749 | unsigned long aligned_start = addr & mask; | ||
| 750 | unsigned long aligned_end = addr + size; | ||
| 751 | struct change_mapping_params params; | ||
| 752 | bool split_region = false; | ||
| 753 | |||
| 754 | if ((end - addr) < size) { | ||
| 755 | /* | ||
| 756 | * We're going to clear the PTE, but not flushed | ||
| 757 | * the mapping, time to remap and flush. The | ||
| 758 | * effects if visible outside the processor or | ||
| 759 | * if we are running in code close to the | ||
| 760 | * mapping we cleared, we are in trouble. | ||
| 761 | */ | ||
| 762 | if (overlaps_kernel_text(aligned_start, addr) || | ||
| 763 | overlaps_kernel_text(end, aligned_end)) { | ||
| 764 | /* | ||
| 765 | * Hack, just return, don't pte_clear | ||
| 766 | */ | ||
| 767 | WARN_ONCE(1, "Linear mapping %lx->%lx overlaps kernel " | ||
| 768 | "text, not splitting\n", addr, end); | ||
| 769 | return; | ||
| 770 | } | ||
| 771 | split_region = true; | ||
| 772 | } | ||
| 773 | |||
| 774 | if (split_region) { | ||
| 775 | params.pte = pte; | ||
| 776 | params.start = addr; | ||
| 777 | params.end = end; | ||
| 778 | params.aligned_start = addr & ~(size - 1); | ||
| 779 | params.aligned_end = min_t(unsigned long, aligned_end, | ||
| 780 | (unsigned long)__va(memblock_end_of_DRAM())); | ||
| 781 | stop_machine(stop_machine_change_mapping, ¶ms, NULL); | ||
| 782 | return; | ||
| 783 | } | ||
| 784 | |||
| 785 | pte_clear(&init_mm, addr, pte); | ||
| 786 | } | ||
| 787 | |||
| 697 | static void remove_pmd_table(pmd_t *pmd_start, unsigned long addr, | 788 | static void remove_pmd_table(pmd_t *pmd_start, unsigned long addr, |
| 698 | unsigned long end) | 789 | unsigned long end) |
| 699 | { | 790 | { |
| @@ -709,13 +800,7 @@ static void remove_pmd_table(pmd_t *pmd_start, unsigned long addr, · 9 unmodified lines | |||
| 709 | continue; | 800 | continue; |
| 710 | 801 | ||
| 711 | if (pmd_huge(*pmd)) { | 802 | if (pmd_huge(*pmd)) { |
| 712 | if (!IS_ALIGNED(addr, PMD_SIZE) || | 803 | split_kernel_mapping(addr, end, PMD_SIZE, (pte_t *)pmd); |
| 713 | !IS_ALIGNED(next, PMD_SIZE)) { | ||
| 714 | WARN_ONCE(1, "%s: unaligned range\n", __func__); | ||
| 715 | continue; | ||
| 716 | } | ||
| 717 | |||
| 718 | pte_clear(&init_mm, addr, (pte_t *)pmd); | ||
| 719 | continue; | 804 | continue; |
| 720 | } | 805 | } |
| 721 | 806 | ||
| @@ -740,13 +825,7 @@ static void remove_pud_table(pud_t *pud_start, unsigned long addr, · 18 unmodified lines | |||
| 740 | continue; | 825 | continue; |
| 741 | 826 | ||
| 742 | if (pud_huge(*pud)) { | 827 | if (pud_huge(*pud)) { |
| 743 | if (!IS_ALIGNED(addr, PUD_SIZE) || | 828 | split_kernel_mapping(addr, end, PUD_SIZE, (pte_t *)pud); |
| 744 | !IS_ALIGNED(next, PUD_SIZE)) { | ||
| 745 | WARN_ONCE(1, "%s: unaligned range\n", __func__); | ||
| 746 | continue; | ||
| 747 | } | ||
| 748 | |||
| 749 | pte_clear(&init_mm, addr, (pte_t *)pud); | ||
| 750 | continue; | 829 | continue; |
| 751 | } | 830 | } |
| 752 | 831 | ||
| @@ -772,13 +851,7 @@ static void remove_pagetable(unsigned long start, unsigned long end) · 19 unmodified lines | |||
| 772 | continue; | 851 | continue; |
| 773 | 852 | ||
| 774 | if (pgd_huge(*pgd)) { | 853 | if (pgd_huge(*pgd)) { |
| 775 | if (!IS_ALIGNED(addr, PGDIR_SIZE) || | 854 | split_kernel_mapping(addr, end, PGDIR_SIZE, (pte_t *)pgd); |
| 776 | !IS_ALIGNED(next, PGDIR_SIZE)) { | ||
| 777 | WARN_ONCE(1, "%s: unaligned range\n", __func__); | ||
| 778 | continue; | ||
| 779 | } | ||
| 780 | |||
| 781 | pte_clear(&init_mm, addr, (pte_t *)pgd); | ||
| 782 | continue; | 855 | continue; |
| 783 | } | 856 | } |
| 784 | 857 | ||
| @@ -82,6 +82,8 @@ unsigned long __pgd_index_size; · 81 unmodified lines | |||
| 82 | EXPORT_SYMBOL(__pgd_index_size); | 82 | EXPORT_SYMBOL(__pgd_index_size); |
| 83 | unsigned long __pmd_cache_index; | 83 | unsigned long __pmd_cache_index; |
| 84 | EXPORT_SYMBOL(__pmd_cache_index); | 84 | EXPORT_SYMBOL(__pmd_cache_index); |
| 85 | unsigned long __pud_cache_index; | ||
| 86 | EXPORT_SYMBOL(__pud_cache_index); | ||
| 85 | unsigned long __pte_table_size; | 87 | unsigned long __pte_table_size; |
| 86 | EXPORT_SYMBOL(__pte_table_size); | 88 | EXPORT_SYMBOL(__pte_table_size); |
| 87 | unsigned long __pmd_table_size; | 89 | unsigned long __pmd_table_size; |
| @@ -471,6 +473,8 @@ void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0, · 383 unmodified lines | |||
| 471 | if (old & PATB_HR) { | 473 | if (old & PATB_HR) { |
| 472 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : : | 474 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : : |
| 473 | "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); | 475 | "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); |
| 476 | asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : : | ||
| 477 | "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); | ||
| 474 | trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1); | 478 | trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1); |
| 475 | } else { | 479 | } else { |
| 476 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : : | 480 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : : |
| @@ -51,7 +51,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 50 unmodified lines | |||
| 51 | unsigned int psize; | 51 | unsigned int psize; |
| 52 | int ssize; | 52 | int ssize; |
| 53 | real_pte_t rpte; | 53 | real_pte_t rpte; |
| 54 | int i; | 54 | int i, offset; |
| 55 | 55 | ||
| 56 | i = batch->index; | 56 | i = batch->index; |
| 57 | 57 | ||
| @@ -67,6 +67,10 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 9 unmodified lines | |||
| 67 | psize = get_slice_psize(mm, addr); | 67 | psize = get_slice_psize(mm, addr); |
| 68 | /* Mask the address for the correct page size */ | 68 | /* Mask the address for the correct page size */ |
| 69 | addr &= ~((1UL << mmu_psize_defs[psize].shift) - 1); | 69 | addr &= ~((1UL << mmu_psize_defs[psize].shift) - 1); |
| 70 | if (unlikely(psize == MMU_PAGE_16G)) | ||
| 71 | offset = PTRS_PER_PUD; | ||
| 72 | else | ||
| 73 | offset = PTRS_PER_PMD; | ||
| 70 | #else | 74 | #else |
| 71 | BUG(); | 75 | BUG(); |
| 72 | psize = pte_pagesize_index(mm, addr, pte); /* shutup gcc */ | 76 | psize = pte_pagesize_index(mm, addr, pte); /* shutup gcc */ |
| @@ -78,6 +82,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 5 unmodified lines | |||
| 78 | * support 64k pages, this might be different from the | 82 | * support 64k pages, this might be different from the |
| 79 | * hardware page size encoded in the slice table. */ | 83 | * hardware page size encoded in the slice table. */ |
| 80 | addr &= PAGE_MASK; | 84 | addr &= PAGE_MASK; |
| 85 | offset = PTRS_PER_PTE; | ||
| 81 | } | 86 | } |
| 82 | 87 | ||
| 83 | 88 | ||
| @@ -91,7 +96,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 7 unmodified lines | |||
| 91 | } | 96 | } |
| 92 | WARN_ON(vsid == 0); | 97 | WARN_ON(vsid == 0); |
| 93 | vpn = hpt_vpn(addr, vsid, ssize); | 98 | vpn = hpt_vpn(addr, vsid, ssize); |
| 94 | rpte = __real_pte(__pte(pte), ptep); | 99 | rpte = __real_pte(__pte(pte), ptep, offset); |
| 95 | 100 | ||
| 96 | /* | 101 | /* |
| 97 | * Check if we have an active batch on this CPU. If not, just | 102 | * Check if we have an active batch on this CPU. If not, just |
| @@ -327,6 +327,9 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, · 326 unmodified lines | |||
| 327 | BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4); | 327 | BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4); |
| 328 | PPC_LWZ_OFFS(r_A, r_skb, offsetof(struct sk_buff, len)); | 328 | PPC_LWZ_OFFS(r_A, r_skb, offsetof(struct sk_buff, len)); |
| 329 | break; | 329 | break; |
| 330 | case BPF_LDX | BPF_W | BPF_ABS: /* A = *((u32 *)(seccomp_data + K)); */ | ||
| 331 | PPC_LWZ_OFFS(r_A, r_skb, K); | ||
| 332 | break; | ||
| 330 | case BPF_LDX | BPF_W | BPF_LEN: /* X = skb->len; */ | 333 | case BPF_LDX | BPF_W | BPF_LEN: /* X = skb->len; */ |
| 331 | PPC_LWZ_OFFS(r_X, r_skb, offsetof(struct sk_buff, len)); | 334 | PPC_LWZ_OFFS(r_X, r_skb, offsetof(struct sk_buff, len)); |
| 332 | break; | 335 | break; |
| @@ -240,6 +240,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 · 239 unmodified lines | |||
| 240 | * goto out; | 240 | * goto out; |
| 241 | */ | 241 | */ |
| 242 | PPC_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)); | 242 | PPC_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)); |
| 243 | PPC_RLWINM(b2p_index, b2p_index, 0, 0, 31); | ||
| 243 | PPC_CMPLW(b2p_index, b2p[TMP_REG_1]); | 244 | PPC_CMPLW(b2p_index, b2p[TMP_REG_1]); |
| 244 | PPC_BCC(COND_GE, out); | 245 | PPC_BCC(COND_GE, out); |
| 245 | 246 | ||
| @@ -199,9 +199,11 @@ static void disable_nest_pmu_counters(void) · 198 unmodified lines | |||
| 199 | const struct cpumask *l_cpumask; | 199 | const struct cpumask *l_cpumask; |
| 200 | 200 | ||
| 201 | get_online_cpus(); | 201 | get_online_cpus(); |
| 202 | for_each_online_node(nid) { | 202 | for_each_node_with_cpus(nid) { |
| 203 | l_cpumask = cpumask_of_node(nid); | 203 | l_cpumask = cpumask_of_node(nid); |
| 204 | cpu = cpumask_first(l_cpumask); | 204 | cpu = cpumask_first_and(l_cpumask, cpu_online_mask); |
| 205 | if (cpu >= nr_cpu_ids) | ||
| 206 | continue; | ||
| 205 | opal_imc_counters_stop(OPAL_IMC_COUNTERS_NEST, | 207 | opal_imc_counters_stop(OPAL_IMC_COUNTERS_NEST, |
| 206 | get_hard_smp_processor_id(cpu)); | 208 | get_hard_smp_processor_id(cpu)); |
| 207 | } | 209 | } |
| @@ -1854,7 +1854,7 @@ static int pnv_pci_ioda_dma_set_mask(struct pci_dev *pdev, u64 dma_mask) · 1853 unmodified lines | |||
| 1854 | s64 rc; | 1854 | s64 rc; |
| 1855 | 1855 | ||
| 1856 | if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE)) | 1856 | if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE)) |
| 1857 | return -ENODEV;; | 1857 | return -ENODEV; |
| 1858 | 1858 | ||
| 1859 | pe = &phb->ioda.pe_array[pdn->pe_number]; | 1859 | pe = &phb->ioda.pe_array[pdn->pe_number]; |
| 1860 | if (pe->tce_bypass_enabled) { | 1860 | if (pe->tce_bypass_enabled) { |
| @@ -80,6 +80,10 @@ static void pnv_setup_rfi_flush(void) · 79 unmodified lines | |||
| 80 | if (np && of_property_read_bool(np, "disabled")) | 80 | if (np && of_property_read_bool(np, "disabled")) |
| 81 | enable--; | 81 | enable--; |
| 82 | 82 | ||
| 83 | np = of_get_child_by_name(fw_features, "speculation-policy-favor-security"); | ||
| 84 | if (np && of_property_read_bool(np, "disabled")) | ||
| 85 | enable = 0; | ||
| 86 | |||
| 83 | of_node_put(np); | 87 | of_node_put(np); |
| 84 | of_node_put(fw_features); | 88 | of_node_put(fw_features); |
| 85 | } | 89 | } |
| @@ -1063,16 +1063,16 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop, · 1062 unmodified lines | |||
| 1063 | rc = PTR_ERR(txwin->paste_kaddr); | 1063 | rc = PTR_ERR(txwin->paste_kaddr); |
| 1064 | goto free_window; | 1064 | goto free_window; |
| 1065 | } | 1065 | } |
| 1066 | } else { | ||
| 1067 | /* | ||
| 1068 | * A user mapping must ensure that context switch issues | ||
| 1069 | * CP_ABORT for this thread. | ||
| 1070 | */ | ||
| 1071 | rc = set_thread_uses_vas(); | ||
| 1072 | if (rc) | ||
| 1073 | goto free_window; | ||
| 1066 | } | 1074 | } |
| 1067 | 1075 | ||
| 1068 | /* | ||
| 1069 | * Now that we have a send window, ensure context switch issues | ||
| 1070 | * CP_ABORT for this thread. | ||
| 1071 | */ | ||
| 1072 | rc = -EINVAL; | ||
| 1073 | if (set_thread_uses_vas() < 0) | ||
| 1074 | goto free_window; | ||
| 1075 | |||
| 1076 | set_vinst_win(vinst, txwin); | 1076 | set_vinst_win(vinst, txwin); |
| 1077 | 1077 | ||
| 1078 | return txwin; | 1078 | return txwin; |
| @@ -36,6 +36,7 @@ · 35 unmodified lines | |||
| 36 | #include <asm/xics.h> | 36 | #include <asm/xics.h> |
| 37 | #include <asm/xive.h> | 37 | #include <asm/xive.h> |
| 38 | #include <asm/plpar_wrappers.h> | 38 | #include <asm/plpar_wrappers.h> |
| 39 | #include <asm/topology.h> | ||
| 39 | 40 | ||
| 40 | #include "pseries.h" | 41 | #include "pseries.h" |
| 41 | #include "offline_states.h" | 42 | #include "offline_states.h" |
| @@ -331,6 +332,7 @@ static void pseries_remove_processor(struct device_node *np) · 289 unmodified lines | |||
| 331 | BUG_ON(cpu_online(cpu)); | 332 | BUG_ON(cpu_online(cpu)); |
| 332 | set_cpu_present(cpu, false); | 333 | set_cpu_present(cpu, false); |
| 333 | set_hard_smp_processor_id(cpu, -1); | 334 | set_hard_smp_processor_id(cpu, -1); |
| 335 | update_numa_cpu_lookup_table(cpu, -1); | ||
| 334 | break; | 336 | break; |
| 335 | } | 337 | } |
| 336 | if (cpu >= nr_cpu_ids) | 338 | if (cpu >= nr_cpu_ids) |
| @@ -340,8 +342,6 @@ static void pseries_remove_processor(struct device_node *np) · 3 unmodified lines | |||
| 340 | cpu_maps_update_done(); | 342 | cpu_maps_update_done(); |
| 341 | } | 343 | } |
| 342 | 344 | ||
| 343 | extern int find_and_online_cpu_nid(int cpu); | ||
| 344 | |||
| 345 | static int dlpar_online_cpu(struct device_node *dn) | 345 | static int dlpar_online_cpu(struct device_node *dn) |
| 346 | { | 346 | { |
| 347 | int rc = 0; | 347 | int rc = 0; |
| @@ -48,6 +48,28 @@ static irqreturn_t ras_epow_interrupt(int irq, void *dev_id); · 47 unmodified lines | |||
| 48 | static irqreturn_t ras_error_interrupt(int irq, void *dev_id); | 48 | static irqreturn_t ras_error_interrupt(int irq, void *dev_id); |
| 49 | 49 | ||
| 50 | 50 | ||
| 51 | /* | ||
| 52 | * Enable the hotplug interrupt late because processing them may touch other | ||
| 53 | * devices or systems (e.g. hugepages) that have not been initialized at the | ||
| 54 | * subsys stage. | ||
| 55 | */ | ||
| 56 | int __init init_ras_hotplug_IRQ(void) | ||
| 57 | { | ||
| 58 | struct device_node *np; | ||
| 59 | |||
| 60 | /* Hotplug Events */ | ||
| 61 | np = of_find_node_by_path("/event-sources/hot-plug-events"); | ||
| 62 | if (np != NULL) { | ||
| 63 | if (dlpar_workqueue_init() == 0) | ||
| 64 | request_event_sources_irqs(np, ras_hotplug_interrupt, | ||
| 65 | "RAS_HOTPLUG"); | ||
| 66 | of_node_put(np); | ||
| 67 | } | ||
| 68 | |||
| 69 | return 0; | ||
| 70 | } | ||
| 71 | machine_late_initcall(pseries, init_ras_hotplug_IRQ); | ||
| 72 | |||
| 51 | /* | 73 | /* |
| 52 | * Initialize handlers for the set of interrupts caused by hardware errors | 74 | * Initialize handlers for the set of interrupts caused by hardware errors |
| 53 | * and power system events. | 75 | * and power system events. |
| @@ -66,15 +88,6 @@ static int __init init_ras_IRQ(void) · 12 unmodified lines | |||
| 66 | of_node_put(np); | 88 | of_node_put(np); |
| 67 | } | 89 | } |
| 68 | 90 | ||
| 69 | /* Hotplug Events */ | ||
| 70 | np = of_find_node_by_path("/event-sources/hot-plug-events"); | ||
| 71 | if (np != NULL) { | ||
| 72 | if (dlpar_workqueue_init() == 0) | ||
| 73 | request_event_sources_irqs(np, ras_hotplug_interrupt, | ||
| 74 | "RAS_HOTPLUG"); | ||
| 75 | of_node_put(np); | ||
| 76 | } | ||
| 77 | |||
| 78 | /* EPOW Events */ | 91 | /* EPOW Events */ |
| 79 | np = of_find_node_by_path("/event-sources/epow-events"); | 92 | np = of_find_node_by_path("/event-sources/epow-events"); |
| 80 | if (np != NULL) { | 93 | if (np != NULL) { |
| @@ -482,7 +482,8 @@ static void pseries_setup_rfi_flush(void) · 481 unmodified lines | |||
| 482 | if (types == L1D_FLUSH_NONE) | 482 | if (types == L1D_FLUSH_NONE) |
| 483 | types = L1D_FLUSH_FALLBACK; | 483 | types = L1D_FLUSH_FALLBACK; |
| 484 | 484 | ||
| 485 | if (!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) | 485 | if ((!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) || |
| 486 | (!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY))) | ||
| 486 | enable = false; | 487 | enable = false; |
| 487 | } else { | 488 | } else { |
| 488 | /* Default to fallback if case hcall is not available */ | 489 | /* Default to fallback if case hcall is not available */ |
| @@ -356,7 +356,8 @@ static int xive_spapr_configure_queue(u32 target, struct xive_q *q, u8 prio, · 355 unmodified lines | |||
| 356 | 356 | ||
| 357 | rc = plpar_int_get_queue_info(0, target, prio, &esn_page, &esn_size); | 357 | rc = plpar_int_get_queue_info(0, target, prio, &esn_page, &esn_size); |
| 358 | if (rc) { | 358 | if (rc) { |
| 359 | pr_err("Error %lld getting queue info prio %d\n", rc, prio); | 359 | pr_err("Error %lld getting queue info CPU %d prio %d\n", rc, |
| 360 | target, prio); | ||
| 360 | rc = -EIO; | 361 | rc = -EIO; |
| 361 | goto fail; | 362 | goto fail; |
| 362 | } | 363 | } |
| @@ -370,7 +371,8 @@ static int xive_spapr_configure_queue(u32 target, struct xive_q *q, u8 prio, · 7 unmodified lines | |||
| 370 | /* Configure and enable the queue in HW */ | 371 | /* Configure and enable the queue in HW */ |
| 371 | rc = plpar_int_set_queue_config(flags, target, prio, qpage_phys, order); | 372 | rc = plpar_int_set_queue_config(flags, target, prio, qpage_phys, order); |
| 372 | if (rc) { | 373 | if (rc) { |
| 373 | pr_err("Error %lld setting queue for prio %d\n", rc, prio); | 374 | pr_err("Error %lld setting queue for CPU %d prio %d\n", rc, |
| 375 | target, prio); | ||
| 374 | rc = -EIO; | 376 | rc = -EIO; |
| 375 | } else { | 377 | } else { |
| 376 | q->qpage = qpage; | 378 | q->qpage = qpage; |
| @@ -389,8 +391,8 @@ static int xive_spapr_setup_queue(unsigned int cpu, struct xive_cpu *xc, · 12 unmodified lines | |||
| 389 | if (IS_ERR(qpage)) | 391 | if (IS_ERR(qpage)) |
| 390 | return PTR_ERR(qpage); | 392 | return PTR_ERR(qpage); |
| 391 | 393 | ||
| 392 | return xive_spapr_configure_queue(cpu, q, prio, qpage, | 394 | return xive_spapr_configure_queue(get_hard_smp_processor_id(cpu), |
| 393 | xive_queue_shift); | 395 | q, prio, qpage, xive_queue_shift); |
| 394 | } | 396 | } |
| 395 | 397 | ||
| 396 | static void xive_spapr_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, | 398 | static void xive_spapr_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, |
| @@ -399,10 +401,12 @@ static void xive_spapr_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, · 2 unmodified lines | |||
| 399 | struct xive_q *q = &xc->queue[prio]; | 401 | struct xive_q *q = &xc->queue[prio]; |
| 400 | unsigned int alloc_order; | 402 | unsigned int alloc_order; |
| 401 | long rc; | 403 | long rc; |
| 404 | int hw_cpu = get_hard_smp_processor_id(cpu); | ||
| 402 | 405 | ||
| 403 | rc = plpar_int_set_queue_config(0, cpu, prio, 0, 0); | 406 | rc = plpar_int_set_queue_config(0, hw_cpu, prio, 0, 0); |
| 404 | if (rc) | 407 | if (rc) |
| 405 | pr_err("Error %ld setting queue for prio %d\n", rc, prio); | 408 | pr_err("Error %ld setting queue for CPU %d prio %d\n", rc, |
| 409 | hw_cpu, prio); | ||
| 406 | 410 | ||
| 407 | alloc_order = xive_alloc_order(xive_queue_shift); | 411 | alloc_order = xive_alloc_order(xive_queue_shift); |
| 408 | free_pages((unsigned long)q->qpage, alloc_order); | 412 | free_pages((unsigned long)q->qpage, alloc_order); |
| @@ -8,7 +8,6 @@ config RISCV · 7 unmodified lines | |||
| 8 | select OF | 8 | select OF |
| 9 | select OF_EARLY_FLATTREE | 9 | select OF_EARLY_FLATTREE |
| 10 | select OF_IRQ | 10 | select OF_IRQ |
| 11 | select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE | ||
| 12 | select ARCH_WANT_FRAME_POINTERS | 11 | select ARCH_WANT_FRAME_POINTERS |
| 13 | select CLONE_BACKWARDS | 12 | select CLONE_BACKWARDS |
| 14 | select COMMON_CLK | 13 | select COMMON_CLK |
| @@ -20,7 +19,6 @@ config RISCV · 5 unmodified lines | |||
| 20 | select GENERIC_STRNLEN_USER | 19 | select GENERIC_STRNLEN_USER |
| 21 | select GENERIC_SMP_IDLE_THREAD | 20 | select GENERIC_SMP_IDLE_THREAD |
| 22 | select GENERIC_ATOMIC64 if !64BIT || !RISCV_ISA_A | 21 | select GENERIC_ATOMIC64 if !64BIT || !RISCV_ISA_A |
| 23 | select ARCH_WANT_OPTIONAL_GPIOLIB | ||
| 24 | select HAVE_MEMBLOCK | 22 | select HAVE_MEMBLOCK |
| 25 | select HAVE_MEMBLOCK_NODE_MAP | 23 | select HAVE_MEMBLOCK_NODE_MAP |
| 26 | select HAVE_DMA_API_DEBUG | 24 | select HAVE_DMA_API_DEBUG |
| @@ -34,7 +32,6 @@ config RISCV · 7 unmodified lines | |||
| 34 | select HAVE_ARCH_TRACEHOOK | 32 | select HAVE_ARCH_TRACEHOOK |
| 35 | select MODULES_USE_ELF_RELA if MODULES | 33 | select MODULES_USE_ELF_RELA if MODULES |
| 36 | select THREAD_INFO_IN_TASK | 34 | select THREAD_INFO_IN_TASK |
| 37 | select RISCV_IRQ_INTC | ||
| 38 | select RISCV_TIMER | 35 | select RISCV_TIMER |
| 39 | 36 | ||
| 40 | config MMU | 37 | config MMU |
| @@ -34,9 +34,9 @@ · 33 unmodified lines | |||
| 34 | #define wmb() RISCV_FENCE(ow,ow) | 34 | #define wmb() RISCV_FENCE(ow,ow) |
| 35 | 35 | ||
| 36 | /* These barriers do not need to enforce ordering on devices, just memory. */ | 36 | /* These barriers do not need to enforce ordering on devices, just memory. */ |
| 37 | #define smp_mb() RISCV_FENCE(rw,rw) | 37 | #define __smp_mb() RISCV_FENCE(rw,rw) |
| 38 | #define smp_rmb() RISCV_FENCE(r,r) | 38 | #define __smp_rmb() RISCV_FENCE(r,r) |
| 39 | #define smp_wmb() RISCV_FENCE(w,w) | 39 | #define __smp_wmb() RISCV_FENCE(w,w) |
| 40 | 40 | ||
| 41 | /* | 41 | /* |
| 42 | * This is a very specific barrier: it's currently only used in two places in | 42 | * This is a very specific barrier: it's currently only used in two places in |
| @@ -172,6 +172,9 @@ ENTRY(handle_exception) · 171 unmodified lines | |||
| 172 | move a1, sp /* pt_regs */ | 172 | move a1, sp /* pt_regs */ |
| 173 | tail do_IRQ | 173 | tail do_IRQ |
| 174 | 1: | 174 | 1: |
| 175 | /* Exceptions run with interrupts enabled */ | ||
| 176 | csrs sstatus, SR_SIE | ||
| 177 | |||
| 175 | /* Handle syscalls */ | 178 | /* Handle syscalls */ |
| 176 | li t0, EXC_SYSCALL | 179 | li t0, EXC_SYSCALL |
| 177 | beq s4, t0, handle_syscall | 180 | beq s4, t0, handle_syscall |
| @@ -198,8 +201,6 @@ handle_syscall: · 20 unmodified lines | |||
| 198 | */ | 201 | */ |
| 199 | addi s2, s2, 0x4 | 202 | addi s2, s2, 0x4 |
| 200 | REG_S s2, PT_SEPC(sp) | 203 | REG_S s2, PT_SEPC(sp) |
| 201 | /* System calls run with interrupts enabled */ | ||
| 202 | csrs sstatus, SR_SIE | ||
| 203 | /* Trace syscalls, but only if requested by the user. */ | 204 | /* Trace syscalls, but only if requested by the user. */ |
| 204 | REG_L t0, TASK_TI_FLAGS(tp) | 205 | REG_L t0, TASK_TI_FLAGS(tp) |
| 205 | andi t0, t0, _TIF_SYSCALL_TRACE | 206 | andi t0, t0, _TIF_SYSCALL_TRACE |
| @@ -64,7 +64,7 @@ ENTRY(_start) · 63 unmodified lines | |||
| 64 | /* Start the kernel */ | 64 | /* Start the kernel */ |
| 65 | mv a0, s0 | 65 | mv a0, s0 |
| 66 | mv a1, s1 | 66 | mv a1, s1 |
| 67 | call sbi_save | 67 | call parse_dtb |
| 68 | tail start_kernel | 68 | tail start_kernel |
| 69 | 69 | ||
| 70 | relocate: | 70 | relocate: |
| @@ -144,7 +144,7 @@ asmlinkage void __init setup_vm(void) · 143 unmodified lines | |||
| 144 | #endif | 144 | #endif |
| 145 | } | 145 | } |
| 146 | 146 | ||
| 147 | void __init sbi_save(unsigned int hartid, void *dtb) | 147 | void __init parse_dtb(unsigned int hartid, void *dtb) |
| 148 | { | 148 | { |
| 149 | early_init_dt_scan(__va(dtb)); | 149 | early_init_dt_scan(__va(dtb)); |
| 150 | } | 150 | } |
| @@ -63,6 +63,7 @@ static inline int init_new_context(struct task_struct *tsk, · 62 unmodified lines | |||
| 63 | _ASCE_USER_BITS | _ASCE_TYPE_SEGMENT; | 63 | _ASCE_USER_BITS | _ASCE_TYPE_SEGMENT; |
| 64 | /* pgd_alloc() did not account this pmd */ | 64 | /* pgd_alloc() did not account this pmd */ |
| 65 | mm_inc_nr_pmds(mm); | 65 | mm_inc_nr_pmds(mm); |
| 66 | mm_inc_nr_puds(mm); | ||
| 66 | } | 67 | } |
| 67 | crst_table_init((unsigned long *) mm->pgd, pgd_entry_type(mm)); | 68 | crst_table_init((unsigned long *) mm->pgd, pgd_entry_type(mm)); |
| 68 | return 0; | 69 | return 0; |
| @@ -14,6 +14,7 @@ · 13 unmodified lines | |||
| 14 | #include <asm/processor.h> | 14 | #include <asm/processor.h> |
| 15 | #include <asm/cache.h> | 15 | #include <asm/cache.h> |
| 16 | #include <asm/ctl_reg.h> | 16 | #include <asm/ctl_reg.h> |
| 17 | #include <asm/dwarf.h> | ||
| 17 | #include <asm/errno.h> | 18 | #include <asm/errno.h> |
| 18 | #include <asm/ptrace.h> | 19 | #include <asm/ptrace.h> |
| 19 | #include <asm/thread_info.h> | 20 | #include <asm/thread_info.h> |
| @@ -230,7 +231,7 @@ _PIF_WORK = (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART) · 210 unmodified lines | |||
| 230 | .hidden \name | 231 | .hidden \name |
| 231 | .type \name,@function | 232 | .type \name,@function |
| 232 | \name: | 233 | \name: |
| 233 | .cfi_startproc | 234 | CFI_STARTPROC |
| 234 | #ifdef CONFIG_HAVE_MARCH_Z10_FEATURES | 235 | #ifdef CONFIG_HAVE_MARCH_Z10_FEATURES |
| 235 | exrl 0,0f | 236 | exrl 0,0f |
| 236 | #else | 237 | #else |
| @@ -239,7 +240,7 @@ _PIF_WORK = (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART) · 2 unmodified lines | |||
| 239 | #endif | 240 | #endif |
| 240 | j . | 241 | j . |
| 241 | 0: br \reg | 242 | 0: br \reg |
| 242 | .cfi_endproc | 243 | CFI_ENDPROC |
| 243 | .endm | 244 | .endm |
| 244 | 245 | ||
| 245 | GEN_BR_THUNK __s390x_indirect_jump_r1use_r9,%r9,%r1 | 246 | GEN_BR_THUNK __s390x_indirect_jump_r1use_r9,%r9,%r1 |
| @@ -426,13 +427,13 @@ ENTRY(system_call) · 180 unmodified lines | |||
| 426 | UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER | 427 | UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER |
| 427 | BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP | 428 | BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP |
| 428 | stmg %r0,%r7,__PT_R0(%r11) | 429 | stmg %r0,%r7,__PT_R0(%r11) |
| 429 | # clear user controlled register to prevent speculative use | ||
| 430 | xgr %r0,%r0 | ||
| 431 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC | 430 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC |
| 432 | mvc __PT_PSW(16,%r11),__LC_SVC_OLD_PSW | 431 | mvc __PT_PSW(16,%r11),__LC_SVC_OLD_PSW |
| 433 | mvc __PT_INT_CODE(4,%r11),__LC_SVC_ILC | 432 | mvc __PT_INT_CODE(4,%r11),__LC_SVC_ILC |
| 434 | stg %r14,__PT_FLAGS(%r11) | 433 | stg %r14,__PT_FLAGS(%r11) |
| 435 | .Lsysc_do_svc: | 434 | .Lsysc_do_svc: |
| 435 | # clear user controlled register to prevent speculative use | ||
| 436 | xgr %r0,%r0 | ||
| 436 | # load address of system call table | 437 | # load address of system call table |
| 437 | lg %r10,__THREAD_sysc_table(%r13,%r12) | 438 | lg %r10,__THREAD_sysc_table(%r13,%r12) |
| 438 | llgh %r8,__PT_INT_CODE+2(%r11) | 439 | llgh %r8,__PT_INT_CODE+2(%r11) |
| @@ -1439,6 +1440,7 @@ cleanup_critical: · 1000 unmodified lines | |||
| 1439 | stg %r15,__LC_SYSTEM_TIMER | 1440 | stg %r15,__LC_SYSTEM_TIMER |
| 1440 | 0: # update accounting time stamp | 1441 | 0: # update accounting time stamp |
| 1441 | mvc __LC_LAST_UPDATE_TIMER(8),__LC_SYNC_ENTER_TIMER | 1442 | mvc __LC_LAST_UPDATE_TIMER(8),__LC_SYNC_ENTER_TIMER |
| 1443 | BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP | ||
| 1442 | # set up saved register r11 | 1444 | # set up saved register r11 |
| 1443 | lg %r15,__LC_KERNEL_STACK | 1445 | lg %r15,__LC_KERNEL_STACK |
| 1444 | la %r9,STACK_FRAME_OVERHEAD(%r15) | 1446 | la %r9,STACK_FRAME_OVERHEAD(%r15) |
| @@ -2,8 +2,8 @@ · 1 unmodified line | |||
| 2 | #include <linux/module.h> | 2 | #include <linux/module.h> |
| 3 | #include <asm/nospec-branch.h> | 3 | #include <asm/nospec-branch.h> |
| 4 | 4 | ||
| 5 | int nospec_call_disable = IS_ENABLED(EXPOLINE_OFF); | 5 | int nospec_call_disable = IS_ENABLED(CONFIG_EXPOLINE_OFF); |
| 6 | int nospec_return_disable = !IS_ENABLED(EXPOLINE_FULL); | 6 | int nospec_return_disable = !IS_ENABLED(CONFIG_EXPOLINE_FULL); |
| 7 | 7 | ||
| 8 | static int __init nospectre_v2_setup_early(char *str) | 8 | static int __init nospectre_v2_setup_early(char *str) |
| 9 | { | 9 | { |
| @@ -22,22 +22,6 @@ · 21 unmodified lines | |||
| 22 | #include "trace.h" | 22 | #include "trace.h" |
| 23 | #include "trace-s390.h" | 23 | #include "trace-s390.h" |
| 24 | 24 | ||
| 25 | |||
| 26 | static const intercept_handler_t instruction_handlers[256] = { | ||
| 27 | [0x01] = kvm_s390_handle_01, | ||
| 28 | [0x82] = kvm_s390_handle_lpsw, | ||
| 29 | [0x83] = kvm_s390_handle_diag, | ||
| 30 | [0xaa] = kvm_s390_handle_aa, | ||
| 31 | [0xae] = kvm_s390_handle_sigp, | ||
| 32 | [0xb2] = kvm_s390_handle_b2, | ||
| 33 | [0xb6] = kvm_s390_handle_stctl, | ||
| 34 | [0xb7] = kvm_s390_handle_lctl, | ||
| 35 | [0xb9] = kvm_s390_handle_b9, | ||
| 36 | [0xe3] = kvm_s390_handle_e3, | ||
| 37 | [0xe5] = kvm_s390_handle_e5, | ||
| 38 | [0xeb] = kvm_s390_handle_eb, | ||
| 39 | }; | ||
| 40 | |||
| 41 | u8 kvm_s390_get_ilen(struct kvm_vcpu *vcpu) | 25 | u8 kvm_s390_get_ilen(struct kvm_vcpu *vcpu) |
| 42 | { | 26 | { |
| 43 | struct kvm_s390_sie_block *sie_block = vcpu->arch.sie_block; | 27 | struct kvm_s390_sie_block *sie_block = vcpu->arch.sie_block; |
| @@ -129,16 +113,39 @@ static int handle_validity(struct kvm_vcpu *vcpu) · 85 unmodified lines | |||
| 129 | 113 | ||
| 130 | static int handle_instruction(struct kvm_vcpu *vcpu) | 114 | static int handle_instruction(struct kvm_vcpu *vcpu) |
| 131 | { | 115 | { |
| 132 | intercept_handler_t handler; | ||
| 133 | |||
| 134 | vcpu->stat.exit_instruction++; | 116 | vcpu->stat.exit_instruction++; |
| 135 | trace_kvm_s390_intercept_instruction(vcpu, | 117 | trace_kvm_s390_intercept_instruction(vcpu, |
| 136 | vcpu->arch.sie_block->ipa, | 118 | vcpu->arch.sie_block->ipa, |
| 137 | vcpu->arch.sie_block->ipb); | 119 | vcpu->arch.sie_block->ipb); |
| 138 | handler = instruction_handlers[vcpu->arch.sie_block->ipa >> 8]; | 120 | |
| 139 | if (handler) | 121 | switch (vcpu->arch.sie_block->ipa >> 8) { |
| 140 | return handler(vcpu); | 122 | case 0x01: |
| 141 | return -EOPNOTSUPP; | 123 | return kvm_s390_handle_01(vcpu); |
| 124 | case 0x82: | ||
| 125 | return kvm_s390_handle_lpsw(vcpu); | ||
| 126 | case 0x83: | ||
| 127 | return kvm_s390_handle_diag(vcpu); | ||
| 128 | case 0xaa: | ||
| 129 | return kvm_s390_handle_aa(vcpu); | ||
| 130 | case 0xae: | ||
| 131 | return kvm_s390_handle_sigp(vcpu); | ||
| 132 | case 0xb2: | ||
| 133 | return kvm_s390_handle_b2(vcpu); | ||
| 134 | case 0xb6: | ||
| 135 | return kvm_s390_handle_stctl(vcpu); | ||
| 136 | case 0xb7: | ||
| 137 | return kvm_s390_handle_lctl(vcpu); | ||
| 138 | case 0xb9: | ||
| 139 | return kvm_s390_handle_b9(vcpu); | ||
| 140 | case 0xe3: | ||
| 141 | return kvm_s390_handle_e3(vcpu); | ||
| 142 | case 0xe5: | ||
| 143 | return kvm_s390_handle_e5(vcpu); | ||
| 144 | case 0xeb: | ||
| 145 | return kvm_s390_handle_eb(vcpu); | ||
| 146 | default: | ||
| 147 | return -EOPNOTSUPP; | ||
| 148 | } | ||
| 142 | } | 149 | } |
| 143 | 150 | ||
| 144 | static int inject_prog_on_prog_intercept(struct kvm_vcpu *vcpu) | 151 | static int inject_prog_on_prog_intercept(struct kvm_vcpu *vcpu) |
| @@ -169,8 +169,15 @@ static int ckc_interrupts_enabled(struct kvm_vcpu *vcpu) · 168 unmodified lines | |||
| 169 | 169 | ||
| 170 | static int ckc_irq_pending(struct kvm_vcpu *vcpu) | 170 | static int ckc_irq_pending(struct kvm_vcpu *vcpu) |
| 171 | { | 171 | { |
| 172 | if (vcpu->arch.sie_block->ckc >= kvm_s390_get_tod_clock_fast(vcpu->kvm)) | 172 | const u64 now = kvm_s390_get_tod_clock_fast(vcpu->kvm); |
| 173 | const u64 ckc = vcpu->arch.sie_block->ckc; | ||
| 174 | |||
| 175 | if (vcpu->arch.sie_block->gcr[0] & 0x0020000000000000ul) { | ||
| 176 | if ((s64)ckc >= (s64)now) | ||
| 177 | return 0; | ||
| 178 | } else if (ckc >= now) { | ||
| 173 | return 0; | 179 | return 0; |
| 180 | } | ||
| 174 | return ckc_interrupts_enabled(vcpu); | 181 | return ckc_interrupts_enabled(vcpu); |
| 175 | } | 182 | } |
| 176 | 183 | ||
| @@ -187,12 +194,6 @@ static int cpu_timer_irq_pending(struct kvm_vcpu *vcpu) · 10 unmodified lines | |||
| 187 | return kvm_s390_get_cpu_timer(vcpu) >> 63; | 194 | return kvm_s390_get_cpu_timer(vcpu) >> 63; |
| 188 | } | 195 | } |
| 189 | 196 | ||
| 190 | static inline int is_ioirq(unsigned long irq_type) | ||
| 191 | { | ||
| 192 | return ((irq_type >= IRQ_PEND_IO_ISC_7) && | ||
| 193 | (irq_type <= IRQ_PEND_IO_ISC_0)); | ||
| 194 | } | ||
| 195 | |||
| 196 | static uint64_t isc_to_isc_bits(int isc) | 197 | static uint64_t isc_to_isc_bits(int isc) |
| 197 | { | 198 | { |
| 198 | return (0x80 >> isc) << 24; | 199 | return (0x80 >> isc) << 24; |
| @@ -236,10 +237,15 @@ static inline int kvm_s390_gisa_tac_ipm_gisc(struct kvm_s390_gisa *gisa, u32 gis · 37 unmodified lines | |||
| 236 | return test_and_clear_bit_inv(IPM_BIT_OFFSET + gisc, (unsigned long *) gisa); | 237 | return test_and_clear_bit_inv(IPM_BIT_OFFSET + gisc, (unsigned long *) gisa); |
| 237 | } | 238 | } |
| 238 | 239 | ||
| 239 | static inline unsigned long pending_irqs(struct kvm_vcpu *vcpu) | 240 | static inline unsigned long pending_irqs_no_gisa(struct kvm_vcpu *vcpu) |
| 240 | { | 241 | { |
| 241 | return vcpu->kvm->arch.float_int.pending_irqs | | 242 | return vcpu->kvm->arch.float_int.pending_irqs | |
| 242 | vcpu->arch.local_int.pending_irqs | | 243 | vcpu->arch.local_int.pending_irqs; |
| 244 | } | ||
| 245 | |||
| 246 | static inline unsigned long pending_irqs(struct kvm_vcpu *vcpu) | ||
| 247 | { | ||
| 248 | return pending_irqs_no_gisa(vcpu) | | ||
| 243 | kvm_s390_gisa_get_ipm(vcpu->kvm->arch.gisa) << IRQ_PEND_IO_ISC_7; | 249 | kvm_s390_gisa_get_ipm(vcpu->kvm->arch.gisa) << IRQ_PEND_IO_ISC_7; |
| 244 | } | 250 | } |
| 245 | 251 | ||
| @@ -337,7 +343,7 @@ static void __reset_intercept_indicators(struct kvm_vcpu *vcpu) · 91 unmodified lines | |||
| 337 | 343 | ||
| 338 | static void set_intercept_indicators_io(struct kvm_vcpu *vcpu) | 344 | static void set_intercept_indicators_io(struct kvm_vcpu *vcpu) |
| 339 | { | 345 | { |
| 340 | if (!(pending_irqs(vcpu) & IRQ_PEND_IO_MASK)) | 346 | if (!(pending_irqs_no_gisa(vcpu) & IRQ_PEND_IO_MASK)) |
| 341 | return; | 347 | return; |
| 342 | else if (psw_ioint_disabled(vcpu)) | 348 | else if (psw_ioint_disabled(vcpu)) |
| 343 | kvm_s390_set_cpuflags(vcpu, CPUSTAT_IO_INT); | 349 | kvm_s390_set_cpuflags(vcpu, CPUSTAT_IO_INT); |
| @@ -1011,24 +1017,6 @@ out: · 667 unmodified lines | |||
| 1011 | return rc; | 1017 | return rc; |
| 1012 | } | 1018 | } |
| 1013 | 1019 | ||
| 1014 | typedef int (*deliver_irq_t)(struct kvm_vcpu *vcpu); | ||
| 1015 | |||
| 1016 | static const deliver_irq_t deliver_irq_funcs[] = { | ||
| 1017 | [IRQ_PEND_MCHK_EX] = __deliver_machine_check, | ||
| 1018 | [IRQ_PEND_MCHK_REP] = __deliver_machine_check, | ||
| 1019 | [IRQ_PEND_PROG] = __deliver_prog, | ||
| 1020 | [IRQ_PEND_EXT_EMERGENCY] = __deliver_emergency_signal, | ||
| 1021 | [IRQ_PEND_EXT_EXTERNAL] = __deliver_external_call, | ||
| 1022 | [IRQ_PEND_EXT_CLOCK_COMP] = __deliver_ckc, | ||
| 1023 | [IRQ_PEND_EXT_CPU_TIMER] = __deliver_cpu_timer, | ||
| 1024 | [IRQ_PEND_RESTART] = __deliver_restart, | ||
| 1025 | [IRQ_PEND_SET_PREFIX] = __deliver_set_prefix, | ||
| 1026 | [IRQ_PEND_PFAULT_INIT] = __deliver_pfault_init, | ||
| 1027 | [IRQ_PEND_EXT_SERVICE] = __deliver_service, | ||
| 1028 | [IRQ_PEND_PFAULT_DONE] = __deliver_pfault_done, | ||
| 1029 | [IRQ_PEND_VIRTIO] = __deliver_virtio, | ||
| 1030 | }; | ||
| 1031 | |||
| 1032 | /* Check whether an external call is pending (deliverable or not) */ | 1020 | /* Check whether an external call is pending (deliverable or not) */ |
| 1033 | int kvm_s390_ext_call_pending(struct kvm_vcpu *vcpu) | 1021 | int kvm_s390_ext_call_pending(struct kvm_vcpu *vcpu) |
| 1034 | { | 1022 | { |
| @@ -1066,13 +1054,19 @@ int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu) · 31 unmodified lines | |||
| 1066 | 1054 | ||
| 1067 | static u64 __calculate_sltime(struct kvm_vcpu *vcpu) | 1055 | static u64 __calculate_sltime(struct kvm_vcpu *vcpu) |
| 1068 | { | 1056 | { |
| 1069 | u64 now, cputm, sltime = 0; | 1057 | const u64 now = kvm_s390_get_tod_clock_fast(vcpu->kvm); |
| 1058 | const u64 ckc = vcpu->arch.sie_block->ckc; | ||
| 1059 | u64 cputm, sltime = 0; | ||
| 1070 | 1060 | ||
| 1071 | if (ckc_interrupts_enabled(vcpu)) { | 1061 | if (ckc_interrupts_enabled(vcpu)) { |
| 1072 | now = kvm_s390_get_tod_clock_fast(vcpu->kvm); | 1062 | if (vcpu->arch.sie_block->gcr[0] & 0x0020000000000000ul) { |
| 1073 | sltime = tod_to_ns(vcpu->arch.sie_block->ckc - now); | 1063 | if ((s64)now < (s64)ckc) |
| 1074 | /* already expired or overflow? */ | 1064 | sltime = tod_to_ns((s64)ckc - (s64)now); |
| 1075 | if (!sltime || vcpu->arch.sie_block->ckc <= now) | 1065 | } else if (now < ckc) { |
| 1066 | sltime = tod_to_ns(ckc - now); | ||
| 1067 | } | ||
| 1068 | /* already expired */ | ||
| 1069 | if (!sltime) | ||
| 1076 | return 0; | 1070 | return 0; |
| 1077 | if (cpu_timer_interrupts_enabled(vcpu)) { | 1071 | if (cpu_timer_interrupts_enabled(vcpu)) { |
| 1078 | cputm = kvm_s390_get_cpu_timer(vcpu); | 1072 | cputm = kvm_s390_get_cpu_timer(vcpu); |
| @@ -1192,7 +1186,6 @@ void kvm_s390_clear_local_irqs(struct kvm_vcpu *vcpu) · 113 unmodified lines | |||
| 1192 | int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) | 1186 | int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) |
| 1193 | { | 1187 | { |
| 1194 | struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; | 1188 | struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; |
| 1195 | deliver_irq_t func; | ||
| 1196 | int rc = 0; | 1189 | int rc = 0; |
| 1197 | unsigned long irq_type; | 1190 | unsigned long irq_type; |
| 1198 | unsigned long irqs; | 1191 | unsigned long irqs; |
| @@ -1212,16 +1205,57 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) · 13 unmodified lines | |||
| 1212 | while ((irqs = deliverable_irqs(vcpu)) && !rc) { | 1205 | while ((irqs = deliverable_irqs(vcpu)) && !rc) { |
| 1213 | /* bits are in the reverse order of interrupt priority */ | 1206 | /* bits are in the reverse order of interrupt priority */ |
| 1214 | irq_type = find_last_bit(&irqs, IRQ_PEND_COUNT); | 1207 | irq_type = find_last_bit(&irqs, IRQ_PEND_COUNT); |
| 1215 | if (is_ioirq(irq_type)) { | 1208 | switch (irq_type) { |
| 1209 | case IRQ_PEND_IO_ISC_0: | ||
| 1210 | case IRQ_PEND_IO_ISC_1: | ||
| 1211 | case IRQ_PEND_IO_ISC_2: | ||
| 1212 | case IRQ_PEND_IO_ISC_3: | ||
| 1213 | case IRQ_PEND_IO_ISC_4: | ||
| 1214 | case IRQ_PEND_IO_ISC_5: | ||
| 1215 | case IRQ_PEND_IO_ISC_6: | ||
| 1216 | case IRQ_PEND_IO_ISC_7: | ||
| 1216 | rc = __deliver_io(vcpu, irq_type); | 1217 | rc = __deliver_io(vcpu, irq_type); |
| 1217 | } else { | 1218 | break; |
| 1218 | func = deliver_irq_funcs[irq_type]; | 1219 | case IRQ_PEND_MCHK_EX: |
| 1219 | if (!func) { | 1220 | case IRQ_PEND_MCHK_REP: |
| 1220 | WARN_ON_ONCE(func == NULL); | 1221 | rc = __deliver_machine_check(vcpu); |
| 1221 | clear_bit(irq_type, &li->pending_irqs); | 1222 | break; |
| 1222 | continue; | 1223 | case IRQ_PEND_PROG: |
| 1223 | } | 1224 | rc = __deliver_prog(vcpu); |
| 1224 | rc = func(vcpu); | 1225 | break; |
| 1226 | case IRQ_PEND_EXT_EMERGENCY: | ||
| 1227 | rc = __deliver_emergency_signal(vcpu); | ||
| 1228 | break; | ||
| 1229 | case IRQ_PEND_EXT_EXTERNAL: | ||
| 1230 | rc = __deliver_external_call(vcpu); | ||
| 1231 | break; | ||
| 1232 | case IRQ_PEND_EXT_CLOCK_COMP: | ||
| 1233 | rc = __deliver_ckc(vcpu); | ||
| 1234 | break; | ||
| 1235 | case IRQ_PEND_EXT_CPU_TIMER: | ||
| 1236 | rc = __deliver_cpu_timer(vcpu); | ||
| 1237 | break; | ||
| 1238 | case IRQ_PEND_RESTART: | ||
| 1239 | rc = __deliver_restart(vcpu); | ||
| 1240 | break; | ||
| 1241 | case IRQ_PEND_SET_PREFIX: | ||
| 1242 | rc = __deliver_set_prefix(vcpu); | ||
| 1243 | break; | ||
| 1244 | case IRQ_PEND_PFAULT_INIT: | ||
| 1245 | rc = __deliver_pfault_init(vcpu); | ||
| 1246 | break; | ||
| 1247 | case IRQ_PEND_EXT_SERVICE: | ||
| 1248 | rc = __deliver_service(vcpu); | ||
| 1249 | break; | ||
| 1250 | case IRQ_PEND_PFAULT_DONE: | ||
| 1251 | rc = __deliver_pfault_done(vcpu); | ||
| 1252 | break; | ||
| 1253 | case IRQ_PEND_VIRTIO: | ||
| 1254 | rc = __deliver_virtio(vcpu); | ||
| 1255 | break; | ||
| 1256 | default: | ||
| 1257 | WARN_ONCE(1, "Unknown pending irq type %ld", irq_type); | ||
| 1258 | clear_bit(irq_type, &li->pending_irqs); | ||
| 1225 | } | 1259 | } |
| 1226 | } | 1260 | } |
| 1227 | 1261 | ||
| @@ -1701,7 +1735,8 @@ static void __floating_irq_kick(struct kvm *kvm, u64 type) · 473 unmodified lines | |||
| 1701 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_STOP_INT); | 1735 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_STOP_INT); |
| 1702 | break; | 1736 | break; |
| 1703 | case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX: | 1737 | case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX: |
| 1704 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT); | 1738 | if (!(type & KVM_S390_INT_IO_AI_MASK && kvm->arch.gisa)) |
| 1739 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT); | ||
| 1705 | break; | 1740 | break; |
| 1706 | default: | 1741 | default: |
| 1707 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_EXT_INT); | 1742 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_EXT_INT); |
| @@ -86,6 +86,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { · 85 unmodified lines | |||
| 86 | { "deliver_prefix_signal", VCPU_STAT(deliver_prefix_signal) }, | 86 | { "deliver_prefix_signal", VCPU_STAT(deliver_prefix_signal) }, |
| 87 | { "deliver_restart_signal", VCPU_STAT(deliver_restart_signal) }, | 87 | { "deliver_restart_signal", VCPU_STAT(deliver_restart_signal) }, |
| 88 | { "deliver_program_interruption", VCPU_STAT(deliver_program_int) }, | 88 | { "deliver_program_interruption", VCPU_STAT(deliver_program_int) }, |
| 89 | { "deliver_io_interrupt", VCPU_STAT(deliver_io_int) }, | ||
| 89 | { "exit_wait_state", VCPU_STAT(exit_wait_state) }, | 90 | { "exit_wait_state", VCPU_STAT(exit_wait_state) }, |
| 90 | { "instruction_epsw", VCPU_STAT(instruction_epsw) }, | 91 | { "instruction_epsw", VCPU_STAT(instruction_epsw) }, |
| 91 | { "instruction_gs", VCPU_STAT(instruction_gs) }, | 92 | { "instruction_gs", VCPU_STAT(instruction_gs) }, |
| @@ -179,6 +180,28 @@ int kvm_arch_hardware_enable(void) · 87 unmodified lines | |||
| 179 | static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start, | 180 | static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start, |
| 180 | unsigned long end); | 181 | unsigned long end); |
| 181 | 182 | ||
| 183 | static void kvm_clock_sync_scb(struct kvm_s390_sie_block *scb, u64 delta) | ||
| 184 | { | ||
| 185 | u8 delta_idx = 0; | ||
| 186 | |||
| 187 | /* | ||
| 188 | * The TOD jumps by delta, we have to compensate this by adding | ||
| 189 | * -delta to the epoch. | ||
| 190 | */ | ||
| 191 | delta = -delta; | ||
| 192 | |||
| 193 | /* sign-extension - we're adding to signed values below */ | ||
| 194 | if ((s64)delta < 0) | ||
| 195 | delta_idx = -1; | ||
| 196 | |||
| 197 | scb->epoch += delta; | ||
| 198 | if (scb->ecd & ECD_MEF) { | ||
| 199 | scb->epdx += delta_idx; | ||
| 200 | if (scb->epoch < delta) | ||
| 201 | scb->epdx += 1; | ||
| 202 | } | ||
| 203 | } | ||
| 204 | |||
| 182 | /* | 205 | /* |
| 183 | * This callback is executed during stop_machine(). All CPUs are therefore | 206 | * This callback is executed during stop_machine(). All CPUs are therefore |
| 184 | * temporarily stopped. In order not to change guest behavior, we have to | 207 | * temporarily stopped. In order not to change guest behavior, we have to |
| @@ -194,13 +217,17 @@ static int kvm_clock_sync(struct notifier_block *notifier, unsigned long val, · 9 unmodified lines | |||
| 194 | unsigned long long *delta = v; | 217 | unsigned long long *delta = v; |
| 195 | 218 | ||
| 196 | list_for_each_entry(kvm, &vm_list, vm_list) { | 219 | list_for_each_entry(kvm, &vm_list, vm_list) { |
| 197 | kvm->arch.epoch -= *delta; | ||
| 198 | kvm_for_each_vcpu(i, vcpu, kvm) { | 220 | kvm_for_each_vcpu(i, vcpu, kvm) { |
| 199 | vcpu->arch.sie_block->epoch -= *delta; | 221 | kvm_clock_sync_scb(vcpu->arch.sie_block, *delta); |
| 222 | if (i == 0) { | ||
| 223 | kvm->arch.epoch = vcpu->arch.sie_block->epoch; | ||
| 224 | kvm->arch.epdx = vcpu->arch.sie_block->epdx; | ||
| 225 | } | ||
| 200 | if (vcpu->arch.cputm_enabled) | 226 | if (vcpu->arch.cputm_enabled) |
| 201 | vcpu->arch.cputm_start += *delta; | 227 | vcpu->arch.cputm_start += *delta; |
| 202 | if (vcpu->arch.vsie_block) | 228 | if (vcpu->arch.vsie_block) |
| 203 | vcpu->arch.vsie_block->epoch -= *delta; | 229 | kvm_clock_sync_scb(vcpu->arch.vsie_block, |
| 230 | *delta); | ||
| 204 | } | 231 | } |
| 205 | } | 232 | } |
| 206 | return NOTIFY_OK; | 233 | return NOTIFY_OK; |
| @@ -902,12 +929,9 @@ static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr) · 695 unmodified lines | |||
| 902 | if (copy_from_user(>od, (void __user *)attr->addr, sizeof(gtod))) | 929 | if (copy_from_user(>od, (void __user *)attr->addr, sizeof(gtod))) |
| 903 | return -EFAULT; | 930 | return -EFAULT; |
| 904 | 931 | ||
| 905 | if (test_kvm_facility(kvm, 139)) | 932 | if (!test_kvm_facility(kvm, 139) && gtod.epoch_idx) |
| 906 | kvm_s390_set_tod_clock_ext(kvm, >od); | ||
| 907 | else if (gtod.epoch_idx == 0) | ||
| 908 | kvm_s390_set_tod_clock(kvm, gtod.tod); | ||
| 909 | else | ||
| 910 | return -EINVAL; | 933 | return -EINVAL; |
| 934 | kvm_s390_set_tod_clock(kvm, >od); | ||
| 911 | 935 | ||
| 912 | VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx", | 936 | VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx", |
| 913 | gtod.epoch_idx, gtod.tod); | 937 | gtod.epoch_idx, gtod.tod); |
| @@ -932,13 +956,14 @@ static int kvm_s390_set_tod_high(struct kvm *kvm, struct kvm_device_attr *attr) · 18 unmodified lines | |||
| 932 | 956 | ||
| 933 | static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr) | 957 | static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr) |
| 934 | { | 958 | { |
| 935 | u64 gtod; | 959 | struct kvm_s390_vm_tod_clock gtod = { 0 }; |
| 936 | 960 | ||
| 937 | if (copy_from_user(>od, (void __user *)attr->addr, sizeof(gtod))) | 961 | if (copy_from_user(>od.tod, (void __user *)attr->addr, |
| 962 | sizeof(gtod.tod))) | ||
| 938 | return -EFAULT; | 963 | return -EFAULT; |
| 939 | 964 | ||
| 940 | kvm_s390_set_tod_clock(kvm, gtod); | 965 | kvm_s390_set_tod_clock(kvm, >od); |
| 941 | VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod); | 966 | VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod.tod); |
| 942 | return 0; | 967 | return 0; |
| 943 | } | 968 | } |
| 944 | 969 | ||
| @@ -2122,6 +2147,7 @@ static void sca_add_vcpu(struct kvm_vcpu *vcpu) · 1177 unmodified lines | |||
| 2122 | /* we still need the basic sca for the ipte control */ | 2147 | /* we still need the basic sca for the ipte control */ |
| 2123 | vcpu->arch.sie_block->scaoh = (__u32)(((__u64)sca) >> 32); | 2148 | vcpu->arch.sie_block->scaoh = (__u32)(((__u64)sca) >> 32); |
| 2124 | vcpu->arch.sie_block->scaol = (__u32)(__u64)sca; | 2149 | vcpu->arch.sie_block->scaol = (__u32)(__u64)sca; |
| 2150 | return; | ||
| 2125 | } | 2151 | } |
| 2126 | read_lock(&vcpu->kvm->arch.sca_lock); | 2152 | read_lock(&vcpu->kvm->arch.sca_lock); |
| 2127 | if (vcpu->kvm->arch.use_esca) { | 2153 | if (vcpu->kvm->arch.use_esca) { |
| @@ -2389,6 +2415,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) · 261 unmodified lines | |||
| 2389 | mutex_lock(&vcpu->kvm->lock); | 2415 | mutex_lock(&vcpu->kvm->lock); |
| 2390 | preempt_disable(); | 2416 | preempt_disable(); |
| 2391 | vcpu->arch.sie_block->epoch = vcpu->kvm->arch.epoch; | 2417 | vcpu->arch.sie_block->epoch = vcpu->kvm->arch.epoch; |
| 2418 | vcpu->arch.sie_block->epdx = vcpu->kvm->arch.epdx; | ||
| 2392 | preempt_enable(); | 2419 | preempt_enable(); |
| 2393 | mutex_unlock(&vcpu->kvm->lock); | 2420 | mutex_unlock(&vcpu->kvm->lock); |
| 2394 | if (!kvm_is_ucontrol(vcpu->kvm)) { | 2421 | if (!kvm_is_ucontrol(vcpu->kvm)) { |
| @@ -3021,8 +3048,8 @@ retry: · 626 unmodified lines | |||
| 3021 | return 0; | 3048 | return 0; |
| 3022 | } | 3049 | } |
| 3023 | 3050 | ||
| 3024 | void kvm_s390_set_tod_clock_ext(struct kvm *kvm, | 3051 | void kvm_s390_set_tod_clock(struct kvm *kvm, |
| 3025 | const struct kvm_s390_vm_tod_clock *gtod) | 3052 | const struct kvm_s390_vm_tod_clock *gtod) |
| 3026 | { | 3053 | { |
| 3027 | struct kvm_vcpu *vcpu; | 3054 | struct kvm_vcpu *vcpu; |
| 3028 | struct kvm_s390_tod_clock_ext htod; | 3055 | struct kvm_s390_tod_clock_ext htod; |
| @@ -3034,10 +3061,12 @@ void kvm_s390_set_tod_clock_ext(struct kvm *kvm, · 5 unmodified lines | |||
| 3034 | get_tod_clock_ext((char *)&htod); | 3061 | get_tod_clock_ext((char *)&htod); |
| 3035 | 3062 | ||
| 3036 | kvm->arch.epoch = gtod->tod - htod.tod; | 3063 | kvm->arch.epoch = gtod->tod - htod.tod; |
| 3037 | kvm->arch.epdx = gtod->epoch_idx - htod.epoch_idx; | 3064 | kvm->arch.epdx = 0; |
| 3038 | 3065 | if (test_kvm_facility(kvm, 139)) { | |
| 3039 | if (kvm->arch.epoch > gtod->tod) | 3066 | kvm->arch.epdx = gtod->epoch_idx - htod.epoch_idx; |
| 3040 | kvm->arch.epdx -= 1; | 3067 | if (kvm->arch.epoch > gtod->tod) |
| 3068 | kvm->arch.epdx -= 1; | ||
| 3069 | } | ||
| 3041 | 3070 | ||
| 3042 | kvm_s390_vcpu_block_all(kvm); | 3071 | kvm_s390_vcpu_block_all(kvm); |
| 3043 | kvm_for_each_vcpu(i, vcpu, kvm) { | 3072 | kvm_for_each_vcpu(i, vcpu, kvm) { |
| @@ -3050,22 +3079,6 @@ void kvm_s390_set_tod_clock_ext(struct kvm *kvm, · 6 unmodified lines | |||
| 3050 | mutex_unlock(&kvm->lock); | 3079 | mutex_unlock(&kvm->lock); |
| 3051 | } | 3080 | } |
| 3052 | 3081 | ||
| 3053 | void kvm_s390_set_tod_clock(struct kvm *kvm, u64 tod) | ||
| 3054 | { | ||
| 3055 | struct kvm_vcpu *vcpu; | ||
| 3056 | int i; | ||
| 3057 | |||
| 3058 | mutex_lock(&kvm->lock); | ||
| 3059 | preempt_disable(); | ||
| 3060 | kvm->arch.epoch = tod - get_tod_clock(); | ||
| 3061 | kvm_s390_vcpu_block_all(kvm); | ||
| 3062 | kvm_for_each_vcpu(i, vcpu, kvm) | ||
| 3063 | vcpu->arch.sie_block->epoch = kvm->arch.epoch; | ||
| 3064 | kvm_s390_vcpu_unblock_all(kvm); | ||
| 3065 | preempt_enable(); | ||
| 3066 | mutex_unlock(&kvm->lock); | ||
| 3067 | } | ||
| 3068 | |||
| 3069 | /** | 3082 | /** |
| 3070 | * kvm_arch_fault_in_page - fault-in guest page if necessary | 3083 | * kvm_arch_fault_in_page - fault-in guest page if necessary |
| 3071 | * @vcpu: The corresponding virtual cpu | 3084 | * @vcpu: The corresponding virtual cpu |
| @@ -19,8 +19,6 @@ · 18 unmodified lines | |||
| 19 | #include <asm/processor.h> | 19 | #include <asm/processor.h> |
| 20 | #include <asm/sclp.h> | 20 | #include <asm/sclp.h> |
| 21 | 21 | ||
| 22 | typedef int (*intercept_handler_t)(struct kvm_vcpu *vcpu); | ||
| 23 | |||
| 24 | /* Transactional Memory Execution related macros */ | 22 | /* Transactional Memory Execution related macros */ |
| 25 | #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE)) | 23 | #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE)) |
| 26 | #define TDB_FORMAT1 1 | 24 | #define TDB_FORMAT1 1 |
| @@ -283,9 +281,8 @@ int kvm_s390_handle_sigp(struct kvm_vcpu *vcpu); · 256 unmodified lines | |||
| 283 | int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu); | 281 | int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu); |
| 284 | 282 | ||
| 285 | /* implemented in kvm-s390.c */ | 283 | /* implemented in kvm-s390.c */ |
| 286 | void kvm_s390_set_tod_clock_ext(struct kvm *kvm, | 284 | void kvm_s390_set_tod_clock(struct kvm *kvm, |
| 287 | const struct kvm_s390_vm_tod_clock *gtod); | 285 | const struct kvm_s390_vm_tod_clock *gtod); |
| 288 | void kvm_s390_set_tod_clock(struct kvm *kvm, u64 tod); | ||
| 289 | long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable); | 286 | long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable); |
| 290 | int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr); | 287 | int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr); |
| 291 | int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr); | 288 | int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr); |
| @@ -85,9 +85,10 @@ int kvm_s390_handle_e3(struct kvm_vcpu *vcpu) · 84 unmodified lines | |||
| 85 | /* Handle SCK (SET CLOCK) interception */ | 85 | /* Handle SCK (SET CLOCK) interception */ |
| 86 | static int handle_set_clock(struct kvm_vcpu *vcpu) | 86 | static int handle_set_clock(struct kvm_vcpu *vcpu) |
| 87 | { | 87 | { |
| 88 | struct kvm_s390_vm_tod_clock gtod = { 0 }; | ||
| 88 | int rc; | 89 | int rc; |
| 89 | u8 ar; | 90 | u8 ar; |
| 90 | u64 op2, val; | 91 | u64 op2; |
| 91 | 92 | ||
| 92 | vcpu->stat.instruction_sck++; | 93 | vcpu->stat.instruction_sck++; |
| 93 | 94 | ||
| @@ -97,12 +98,12 @@ static int handle_set_clock(struct kvm_vcpu *vcpu) · 3 unmodified lines | |||
| 97 | op2 = kvm_s390_get_base_disp_s(vcpu, &ar); | 98 | op2 = kvm_s390_get_base_disp_s(vcpu, &ar); |
| 98 | if (op2 & 7) /* Operand must be on a doubleword boundary */ | 99 | if (op2 & 7) /* Operand must be on a doubleword boundary */ |
| 99 | return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION); | 100 | return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION); |
| 100 | rc = read_guest(vcpu, op2, ar, &val, sizeof(val)); | 101 | rc = read_guest(vcpu, op2, ar, >od.tod, sizeof(gtod.tod)); |
| 101 | if (rc) | 102 | if (rc) |
| 102 | return kvm_s390_inject_prog_cond(vcpu, rc); | 103 | return kvm_s390_inject_prog_cond(vcpu, rc); |
| 103 | 104 | ||
| 104 | VCPU_EVENT(vcpu, 3, "SCK: setting guest TOD to 0x%llx", val); | 105 | VCPU_EVENT(vcpu, 3, "SCK: setting guest TOD to 0x%llx", gtod.tod); |
| 105 | kvm_s390_set_tod_clock(vcpu->kvm, val); | 106 | kvm_s390_set_tod_clock(vcpu->kvm, >od); |
| 106 | 107 | ||
| 107 | kvm_s390_set_psw_cc(vcpu, 0); | 108 | kvm_s390_set_psw_cc(vcpu, 0); |
| 108 | return 0; | 109 | return 0; |
| @@ -795,55 +796,60 @@ out: · 686 unmodified lines | |||
| 795 | return rc; | 796 | return rc; |
| 796 | } | 797 | } |
| 797 | 798 | ||
| 798 | static const intercept_handler_t b2_handlers[256] = { | ||
| 799 | [0x02] = handle_stidp, | ||
| 800 | [0x04] = handle_set_clock, | ||
| 801 | [0x10] = handle_set_prefix, | ||
| 802 | [0x11] = handle_store_prefix, | ||
| 803 | [0x12] = handle_store_cpu_address, | ||
| 804 | [0x14] = kvm_s390_handle_vsie, | ||
| 805 | [0x21] = handle_ipte_interlock, | ||
| 806 | [0x29] = handle_iske, | ||
| 807 | [0x2a] = handle_rrbe, | ||
| 808 | [0x2b] = handle_sske, | ||
| 809 | [0x2c] = handle_test_block, | ||
| 810 | [0x30] = handle_io_inst, | ||
| 811 | [0x31] = handle_io_inst, | ||
| 812 | [0x32] = handle_io_inst, | ||
| 813 | [0x33] = handle_io_inst, | ||
| 814 | [0x34] = handle_io_inst, | ||
| 815 | [0x35] = handle_io_inst, | ||
| 816 | [0x36] = handle_io_inst, | ||
| 817 | [0x37] = handle_io_inst, | ||
| 818 | [0x38] = handle_io_inst, | ||
| 819 | [0x39] = handle_io_inst, | ||
| 820 | [0x3a] = handle_io_inst, | ||
| 821 | [0x3b] = handle_io_inst, | ||
| 822 | [0x3c] = handle_io_inst, | ||
| 823 | [0x50] = handle_ipte_interlock, | ||
| 824 | [0x56] = handle_sthyi, | ||
| 825 | [0x5f] = handle_io_inst, | ||
| 826 | [0x74] = handle_io_inst, | ||
| 827 | [0x76] = handle_io_inst, | ||
| 828 | [0x7d] = handle_stsi, | ||
| 829 | [0xb1] = handle_stfl, | ||
| 830 | [0xb2] = handle_lpswe, | ||
| 831 | }; | ||
| 832 | |||
| 833 | int kvm_s390_handle_b2(struct kvm_vcpu *vcpu) | 799 | int kvm_s390_handle_b2(struct kvm_vcpu *vcpu) |
| 834 | { | 800 | { |
| 835 | intercept_handler_t handler; | 801 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 836 | 802 | case 0x02: | |
| 837 | /* | 803 | return handle_stidp(vcpu); |
| 838 | * A lot of B2 instructions are priviledged. Here we check for | 804 | case 0x04: |
| 839 | * the privileged ones, that we can handle in the kernel. | 805 | return handle_set_clock(vcpu); |
| 840 | * Anything else goes to userspace. | 806 | case 0x10: |
| 841 | */ | 807 | return handle_set_prefix(vcpu); |
| 842 | handler = b2_handlers[vcpu->arch.sie_block->ipa & 0x00ff]; | 808 | case 0x11: |
| 843 | if (handler) | 809 | return handle_store_prefix(vcpu); |
| 844 | return handler(vcpu); | 810 | case 0x12: |
| 845 | 811 | return handle_store_cpu_address(vcpu); | |
| 846 | return -EOPNOTSUPP; | 812 | case 0x14: |
| 813 | return kvm_s390_handle_vsie(vcpu); | ||
| 814 | case 0x21: | ||
| 815 | case 0x50: | ||
| 816 | return handle_ipte_interlock(vcpu); | ||
| 817 | case 0x29: | ||
| 818 | return handle_iske(vcpu); | ||
| 819 | case 0x2a: | ||
| 820 | return handle_rrbe(vcpu); | ||
| 821 | case 0x2b: | ||
| 822 | return handle_sske(vcpu); | ||
| 823 | case 0x2c: | ||
| 824 | return handle_test_block(vcpu); | ||
| 825 | case 0x30: | ||
| 826 | case 0x31: | ||
| 827 | case 0x32: | ||
| 828 | case 0x33: | ||
| 829 | case 0x34: | ||
| 830 | case 0x35: | ||
| 831 | case 0x36: | ||
| 832 | case 0x37: | ||
| 833 | case 0x38: | ||
| 834 | case 0x39: | ||
| 835 | case 0x3a: | ||
| 836 | case 0x3b: | ||
| 837 | case 0x3c: | ||
| 838 | case 0x5f: | ||
| 839 | case 0x74: | ||
| 840 | case 0x76: | ||
| 841 | return handle_io_inst(vcpu); | ||
| 842 | case 0x56: | ||
| 843 | return handle_sthyi(vcpu); | ||
| 844 | case 0x7d: | ||
| 845 | return handle_stsi(vcpu); | ||
| 846 | case 0xb1: | ||
| 847 | return handle_stfl(vcpu); | ||
| 848 | case 0xb2: | ||
| 849 | return handle_lpswe(vcpu); | ||
| 850 | default: | ||
| 851 | return -EOPNOTSUPP; | ||
| 852 | } | ||
| 847 | } | 853 | } |
| 848 | 854 | ||
| 849 | static int handle_epsw(struct kvm_vcpu *vcpu) | 855 | static int handle_epsw(struct kvm_vcpu *vcpu) |
| @@ -1105,25 +1111,22 @@ static int handle_essa(struct kvm_vcpu *vcpu) · 255 unmodified lines | |||
| 1105 | return 0; | 1111 | return 0; |
| 1106 | } | 1112 | } |
| 1107 | 1113 | ||
| 1108 | static const intercept_handler_t b9_handlers[256] = { | ||
| 1109 | [0x8a] = handle_ipte_interlock, | ||
| 1110 | [0x8d] = handle_epsw, | ||
| 1111 | [0x8e] = handle_ipte_interlock, | ||
| 1112 | [0x8f] = handle_ipte_interlock, | ||
| 1113 | [0xab] = handle_essa, | ||
| 1114 | [0xaf] = handle_pfmf, | ||
| 1115 | }; | ||
| 1116 | |||
| 1117 | int kvm_s390_handle_b9(struct kvm_vcpu *vcpu) | 1114 | int kvm_s390_handle_b9(struct kvm_vcpu *vcpu) |
| 1118 | { | 1115 | { |
| 1119 | intercept_handler_t handler; | 1116 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 1120 | 1117 | case 0x8a: | |
| 1121 | /* This is handled just as for the B2 instructions. */ | 1118 | case 0x8e: |
| 1122 | handler = b9_handlers[vcpu->arch.sie_block->ipa & 0x00ff]; | 1119 | case 0x8f: |
| 1123 | if (handler) | 1120 | return handle_ipte_interlock(vcpu); |
| 1124 | return handler(vcpu); | 1121 | case 0x8d: |
| 1125 | 1122 | return handle_epsw(vcpu); | |
| 1126 | return -EOPNOTSUPP; | 1123 | case 0xab: |
| 1124 | return handle_essa(vcpu); | ||
| 1125 | case 0xaf: | ||
| 1126 | return handle_pfmf(vcpu); | ||
| 1127 | default: | ||
| 1128 | return -EOPNOTSUPP; | ||
| 1129 | } | ||
| 1127 | } | 1130 | } |
| 1128 | 1131 | ||
| 1129 | int kvm_s390_handle_lctl(struct kvm_vcpu *vcpu) | 1132 | int kvm_s390_handle_lctl(struct kvm_vcpu *vcpu) |
| @@ -1271,22 +1274,20 @@ static int handle_stctg(struct kvm_vcpu *vcpu) · 141 unmodified lines | |||
| 1271 | return rc ? kvm_s390_inject_prog_cond(vcpu, rc) : 0; | 1274 | return rc ? kvm_s390_inject_prog_cond(vcpu, rc) : 0; |
| 1272 | } | 1275 | } |
| 1273 | 1276 | ||
| 1274 | static const intercept_handler_t eb_handlers[256] = { | ||
| 1275 | [0x2f] = handle_lctlg, | ||
| 1276 | [0x25] = handle_stctg, | ||
| 1277 | [0x60] = handle_ri, | ||
| 1278 | [0x61] = handle_ri, | ||
| 1279 | [0x62] = handle_ri, | ||
| 1280 | }; | ||
| 1281 | |||
| 1282 | int kvm_s390_handle_eb(struct kvm_vcpu *vcpu) | 1277 | int kvm_s390_handle_eb(struct kvm_vcpu *vcpu) |
| 1283 | { | 1278 | { |
| 1284 | intercept_handler_t handler; | 1279 | switch (vcpu->arch.sie_block->ipb & 0x000000ff) { |
| 1285 | 1280 | case 0x25: | |
| 1286 | handler = eb_handlers[vcpu->arch.sie_block->ipb & 0xff]; | 1281 | return handle_stctg(vcpu); |
| 1287 | if (handler) | 1282 | case 0x2f: |
| 1288 | return handler(vcpu); | 1283 | return handle_lctlg(vcpu); |
| 1289 | return -EOPNOTSUPP; | 1284 | case 0x60: |
| 1285 | case 0x61: | ||
| 1286 | case 0x62: | ||
| 1287 | return handle_ri(vcpu); | ||
| 1288 | default: | ||
| 1289 | return -EOPNOTSUPP; | ||
| 1290 | } | ||
| 1290 | } | 1291 | } |
| 1291 | 1292 | ||
| 1292 | static int handle_tprot(struct kvm_vcpu *vcpu) | 1293 | static int handle_tprot(struct kvm_vcpu *vcpu) |
| @@ -1346,10 +1347,12 @@ out_unlock: · 53 unmodified lines | |||
| 1346 | 1347 | ||
| 1347 | int kvm_s390_handle_e5(struct kvm_vcpu *vcpu) | 1348 | int kvm_s390_handle_e5(struct kvm_vcpu *vcpu) |
| 1348 | { | 1349 | { |
| 1349 | /* For e5xx... instructions we only handle TPROT */ | 1350 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 1350 | if ((vcpu->arch.sie_block->ipa & 0x00ff) == 0x01) | 1351 | case 0x01: |
| 1351 | return handle_tprot(vcpu); | 1352 | return handle_tprot(vcpu); |
| 1352 | return -EOPNOTSUPP; | 1353 | default: |
| 1354 | return -EOPNOTSUPP; | ||
| 1355 | } | ||
| 1353 | } | 1356 | } |
| 1354 | 1357 | ||
| 1355 | static int handle_sckpf(struct kvm_vcpu *vcpu) | 1358 | static int handle_sckpf(struct kvm_vcpu *vcpu) |
| @@ -1380,17 +1383,14 @@ static int handle_ptff(struct kvm_vcpu *vcpu) · 24 unmodified lines | |||
| 1380 | return 0; | 1383 | return 0; |
| 1381 | } | 1384 | } |
| 1382 | 1385 | ||
| 1383 | static const intercept_handler_t x01_handlers[256] = { | ||
| 1384 | [0x04] = handle_ptff, | ||
| 1385 | [0x07] = handle_sckpf, | ||
| 1386 | }; | ||
| 1387 | |||
| 1388 | int kvm_s390_handle_01(struct kvm_vcpu *vcpu) | 1386 | int kvm_s390_handle_01(struct kvm_vcpu *vcpu) |
| 1389 | { | 1387 | { |
| 1390 | intercept_handler_t handler; | 1388 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 1391 | 1389 | case 0x04: | |
| 1392 | handler = x01_handlers[vcpu->arch.sie_block->ipa & 0x00ff]; | 1390 | return handle_ptff(vcpu); |
| 1393 | if (handler) | 1391 | case 0x07: |
| 1394 | return handler(vcpu); | 1392 | return handle_sckpf(vcpu); |
| 1395 | return -EOPNOTSUPP; | 1393 | default: |
| 1394 | return -EOPNOTSUPP; | ||
| 1395 | } | ||
| 1396 | } | 1396 | } |
| @@ -821,6 +821,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 820 unmodified lines | |||
| 821 | { | 821 | { |
| 822 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; | 822 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; |
| 823 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; | 823 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; |
| 824 | int guest_bp_isolation; | ||
| 824 | int rc; | 825 | int rc; |
| 825 | 826 | ||
| 826 | handle_last_fault(vcpu, vsie_page); | 827 | handle_last_fault(vcpu, vsie_page); |
| @@ -831,6 +832,20 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 4 unmodified lines | |||
| 831 | s390_handle_mcck(); | 832 | s390_handle_mcck(); |
| 832 | 833 | ||
| 833 | srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); | 834 | srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); |
| 835 | |||
| 836 | /* save current guest state of bp isolation override */ | ||
| 837 | guest_bp_isolation = test_thread_flag(TIF_ISOLATE_BP_GUEST); | ||
| 838 | |||
| 839 | /* | ||
| 840 | * The guest is running with BPBC, so we have to force it on for our | ||
| 841 | * nested guest. This is done by enabling BPBC globally, so the BPBC | ||
| 842 | * control in the SCB (which the nested guest can modify) is simply | ||
| 843 | * ignored. | ||
| 844 | */ | ||
| 845 | if (test_kvm_facility(vcpu->kvm, 82) && | ||
| 846 | vcpu->arch.sie_block->fpf & FPF_BPBC) | ||
| 847 | set_thread_flag(TIF_ISOLATE_BP_GUEST); | ||
| 848 | |||
| 834 | local_irq_disable(); | 849 | local_irq_disable(); |
| 835 | guest_enter_irqoff(); | 850 | guest_enter_irqoff(); |
| 836 | local_irq_enable(); | 851 | local_irq_enable(); |
| @@ -840,6 +855,11 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 3 unmodified lines | |||
| 840 | local_irq_disable(); | 855 | local_irq_disable(); |
| 841 | guest_exit_irqoff(); | 856 | guest_exit_irqoff(); |
| 842 | local_irq_enable(); | 857 | local_irq_enable(); |
| 858 | |||
| 859 | /* restore guest state for bp isolation override */ | ||
| 860 | if (!guest_bp_isolation) | ||
| 861 | clear_thread_flag(TIF_ISOLATE_BP_GUEST); | ||
| 862 | |||
| 843 | vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu); | 863 | vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu); |
| 844 | 864 | ||
| 845 | if (rc == -EINTR) { | 865 | if (rc == -EINTR) { |
| @@ -1 +1,3 @@ | |||
| 1 | obj-$(CONFIG_USE_BUILTIN_DTB) += $(patsubst "%",%,$(CONFIG_BUILTIN_DTB_SOURCE)).dtb.o | 1 | ifneq ($(CONFIG_BUILTIN_DTB_SOURCE),"") |
| 2 | obj-y += $(patsubst "%",%,$(CONFIG_BUILTIN_DTB_SOURCE)).dtb.o | ||
| 3 | endif | ||
| @@ -430,6 +430,8 @@ config SPARC_LEON · 429 unmodified lines | |||
| 430 | depends on SPARC32 | 430 | depends on SPARC32 |
| 431 | select USB_EHCI_BIG_ENDIAN_MMIO | 431 | select USB_EHCI_BIG_ENDIAN_MMIO |
| 432 | select USB_EHCI_BIG_ENDIAN_DESC | 432 | select USB_EHCI_BIG_ENDIAN_DESC |
| 433 | select USB_UHCI_BIG_ENDIAN_MMIO | ||
| 434 | select USB_UHCI_BIG_ENDIAN_DESC | ||
| 433 | ---help--- | 435 | ---help--- |
| 434 | If you say Y here if you are running on a SPARC-LEON processor. | 436 | If you say Y here if you are running on a SPARC-LEON processor. |
| 435 | The LEON processor is a synthesizable VHDL model of the | 437 | The LEON processor is a synthesizable VHDL model of the |
| @@ -9,10 +9,14 @@ · 8 unmodified lines | |||
| 9 | void do_BUG(const char *file, int line); | 9 | void do_BUG(const char *file, int line); |
| 10 | #define BUG() do { \ | 10 | #define BUG() do { \ |
| 11 | do_BUG(__FILE__, __LINE__); \ | 11 | do_BUG(__FILE__, __LINE__); \ |
| 12 | barrier_before_unreachable(); \ | ||
| 12 | __builtin_trap(); \ | 13 | __builtin_trap(); \ |
| 13 | } while (0) | 14 | } while (0) |
| 14 | #else | 15 | #else |
| 15 | #define BUG() __builtin_trap() | 16 | #define BUG() do { \ |
| 17 | barrier_before_unreachable(); \ | ||
| 18 | __builtin_trap(); \ | ||
| 19 | } while (0) | ||
| 16 | #endif | 20 | #endif |
| 17 | 21 | ||
| 18 | #define HAVE_ARCH_BUG | 22 | #define HAVE_ARCH_BUG |
| @@ -163,13 +163,10 @@ static void tlb_batch_pmd_scan(struct mm_struct *mm, unsigned long vaddr, · 162 unmodified lines | |||
| 163 | pte_unmap(pte); | 163 | pte_unmap(pte); |
| 164 | } | 164 | } |
| 165 | 165 | ||
| 166 | void set_pmd_at(struct mm_struct *mm, unsigned long addr, | ||
| 167 | pmd_t *pmdp, pmd_t pmd) | ||
| 168 | { | ||
| 169 | pmd_t orig = *pmdp; | ||
| 170 | |||
| 171 | *pmdp = pmd; | ||
| 172 | 166 | ||
| 167 | static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr, | ||
| 168 | pmd_t orig, pmd_t pmd) | ||
| 169 | { | ||
| 173 | if (mm == &init_mm) | 170 | if (mm == &init_mm) |
| 174 | return; | 171 | return; |
| 175 | 172 | ||
| @@ -219,6 +216,15 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, · 43 unmodified lines | |||
| 219 | } | 216 | } |
| 220 | } | 217 | } |
| 221 | 218 | ||
| 219 | void set_pmd_at(struct mm_struct *mm, unsigned long addr, | ||
| 220 | pmd_t *pmdp, pmd_t pmd) | ||
| 221 | { | ||
| 222 | pmd_t orig = *pmdp; | ||
| 223 | |||
| 224 | *pmdp = pmd; | ||
| 225 | __set_pmd_acct(mm, addr, orig, pmd); | ||
| 226 | } | ||
| 227 | |||
| 222 | static inline pmd_t pmdp_establish(struct vm_area_struct *vma, | 228 | static inline pmd_t pmdp_establish(struct vm_area_struct *vma, |
| 223 | unsigned long address, pmd_t *pmdp, pmd_t pmd) | 229 | unsigned long address, pmd_t *pmdp, pmd_t pmd) |
| 224 | { | 230 | { |
| @@ -227,6 +233,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · 2 unmodified lines | |||
| 227 | do { | 233 | do { |
| 228 | old = *pmdp; | 234 | old = *pmdp; |
| 229 | } while (cmpxchg64(&pmdp->pmd, old.pmd, pmd.pmd) != old.pmd); | 235 | } while (cmpxchg64(&pmdp->pmd, old.pmd, pmd.pmd) != old.pmd); |
| 236 | __set_pmd_acct(vma->vm_mm, address, old, pmd); | ||
| 230 | 237 | ||
| 231 | return old; | 238 | return old; |
| 232 | } | 239 | } |
| @@ -1,6 +1,7 @@ | |||
| 1 | boot/compressed/vmlinux | 1 | boot/compressed/vmlinux |
| 2 | tools/test_get_len | 2 | tools/test_get_len |
| 3 | tools/insn_sanity | 3 | tools/insn_sanity |
| 4 | tools/insn_decoder_test | ||
| 4 | purgatory/kexec-purgatory.c | 5 | purgatory/kexec-purgatory.c |
| 5 | purgatory/purgatory.ro | 6 | purgatory/purgatory.ro |
| 6 | 7 | ||
| @@ -423,12 +423,6 @@ config X86_MPPARSE · 422 unmodified lines | |||
| 423 | For old smp systems that do not have proper acpi support. Newer systems | 423 | For old smp systems that do not have proper acpi support. Newer systems |
| 424 | (esp with 64bit cpus) with acpi support, MADT and DSDT will override it | 424 | (esp with 64bit cpus) with acpi support, MADT and DSDT will override it |
| 425 | 425 | ||
| 426 | config X86_BIGSMP | ||
| 427 | bool "Support for big SMP systems with more than 8 CPUs" | ||
| 428 | depends on X86_32 && SMP | ||
| 429 | ---help--- | ||
| 430 | This option is needed for the systems that have more than 8 CPUs | ||
| 431 | |||
| 432 | config GOLDFISH | 426 | config GOLDFISH |
| 433 | def_bool y | 427 | def_bool y |
| 434 | depends on X86_GOLDFISH | 428 | depends on X86_GOLDFISH |
| @@ -436,6 +430,7 @@ config GOLDFISH · 1 unmodified line | |||
| 436 | config RETPOLINE | 430 | config RETPOLINE |
| 437 | bool "Avoid speculative indirect branches in kernel" | 431 | bool "Avoid speculative indirect branches in kernel" |
| 438 | default y | 432 | default y |
| 433 | select STACK_VALIDATION if HAVE_STACK_VALIDATION | ||
| 439 | help | 434 | help |
| 440 | Compile kernel with the retpoline compiler options to guard against | 435 | Compile kernel with the retpoline compiler options to guard against |
| 441 | kernel-to-user data leaks by avoiding speculative indirect | 436 | kernel-to-user data leaks by avoiding speculative indirect |
| @@ -460,6 +455,12 @@ config INTEL_RDT · 18 unmodified lines | |||
| 460 | Say N if unsure. | 455 | Say N if unsure. |
| 461 | 456 | ||
| 462 | if X86_32 | 457 | if X86_32 |
| 458 | config X86_BIGSMP | ||
| 459 | bool "Support for big SMP systems with more than 8 CPUs" | ||
| 460 | depends on SMP | ||
| 461 | ---help--- | ||
| 462 | This option is needed for the systems that have more than 8 CPUs | ||
| 463 | |||
| 463 | config X86_EXTENDED_PLATFORM | 464 | config X86_EXTENDED_PLATFORM |
| 464 | bool "Support for extended (non-PC) x86 platforms" | 465 | bool "Support for extended (non-PC) x86 platforms" |
| 465 | default y | 466 | default y |
| @@ -949,25 +950,66 @@ config MAXSMP · 483 unmodified lines | |||
| 949 | Enable maximum number of CPUS and NUMA Nodes for this architecture. | 950 | Enable maximum number of CPUS and NUMA Nodes for this architecture. |
| 950 | If unsure, say N. | 951 | If unsure, say N. |
| 951 | 952 | ||
| 953 | # | ||
| 954 | # The maximum number of CPUs supported: | ||
| 955 | # | ||
| 956 | # The main config value is NR_CPUS, which defaults to NR_CPUS_DEFAULT, | ||
| 957 | # and which can be configured interactively in the | ||
| 958 | # [NR_CPUS_RANGE_BEGIN ... NR_CPUS_RANGE_END] range. | ||
| 959 | # | ||
| 960 | # The ranges are different on 32-bit and 64-bit kernels, depending on | ||
| 961 | # hardware capabilities and scalability features of the kernel. | ||
| 962 | # | ||
| 963 | # ( If MAXSMP is enabled we just use the highest possible value and disable | ||
| 964 | # interactive configuration. ) | ||
| 965 | # | ||
| 966 | |||
| 967 | config NR_CPUS_RANGE_BEGIN | ||
| 968 | int | ||
| 969 | default NR_CPUS_RANGE_END if MAXSMP | ||
| 970 | default 1 if !SMP | ||
| 971 | default 2 | ||
| 972 | |||
| 973 | config NR_CPUS_RANGE_END | ||
| 974 | int | ||
| 975 | depends on X86_32 | ||
| 976 | default 64 if SMP && X86_BIGSMP | ||
| 977 | default 8 if SMP && !X86_BIGSMP | ||
| 978 | default 1 if !SMP | ||
| 979 | |||
| 980 | config NR_CPUS_RANGE_END | ||
| 981 | int | ||
| 982 | depends on X86_64 | ||
| 983 | default 8192 if SMP && ( MAXSMP || CPUMASK_OFFSTACK) | ||
| 984 | default 512 if SMP && (!MAXSMP && !CPUMASK_OFFSTACK) | ||
| 985 | default 1 if !SMP | ||
| 986 | |||
| 987 | config NR_CPUS_DEFAULT | ||
| 988 | int | ||
| 989 | depends on X86_32 | ||
| 990 | default 32 if X86_BIGSMP | ||
| 991 | default 8 if SMP | ||
| 992 | default 1 if !SMP | ||
| 993 | |||
| 994 | config NR_CPUS_DEFAULT | ||
| 995 | int | ||
| 996 | depends on X86_64 | ||
| 997 | default 8192 if MAXSMP | ||
| 998 | default 64 if SMP | ||
| 999 | default 1 if !SMP | ||
| 1000 | |||
| 952 | config NR_CPUS | 1001 | config NR_CPUS |
| 953 | int "Maximum number of CPUs" if SMP && !MAXSMP | 1002 | int "Maximum number of CPUs" if SMP && !MAXSMP |
| 954 | range 2 8 if SMP && X86_32 && !X86_BIGSMP | 1003 | range NR_CPUS_RANGE_BEGIN NR_CPUS_RANGE_END |
| 955 | range 2 64 if SMP && X86_32 && X86_BIGSMP | 1004 | default NR_CPUS_DEFAULT |
| 956 | range 2 512 if SMP && !MAXSMP && !CPUMASK_OFFSTACK && X86_64 | ||
| 957 | range 2 8192 if SMP && !MAXSMP && CPUMASK_OFFSTACK && X86_64 | ||
| 958 | default "1" if !SMP | ||
| 959 | default "8192" if MAXSMP | ||
| 960 | default "32" if SMP && X86_BIGSMP | ||
| 961 | default "8" if SMP && X86_32 | ||
| 962 | default "64" if SMP | ||
| 963 | ---help--- | 1005 | ---help--- |
| 964 | This allows you to specify the maximum number of CPUs which this | 1006 | This allows you to specify the maximum number of CPUs which this |
| 965 | kernel will support. If CPUMASK_OFFSTACK is enabled, the maximum | 1007 | kernel will support. If CPUMASK_OFFSTACK is enabled, the maximum |
| 966 | supported value is 8192, otherwise the maximum value is 512. The | 1008 | supported value is 8192, otherwise the maximum value is 512. The |
| 967 | minimum value which makes sense is 2. | 1009 | minimum value which makes sense is 2. |
| 968 | 1010 | ||
| 969 | This is purely to save memory - each supported CPU adds | 1011 | This is purely to save memory: each supported CPU adds about 8KB |
| 970 | approximately eight kilobytes to the kernel image. | 1012 | to the kernel image. |
| 971 | 1013 | ||
| 972 | config SCHED_SMT | 1014 | config SCHED_SMT |
| 973 | bool "SMT (Hyperthreading) scheduler support" | 1015 | bool "SMT (Hyperthreading) scheduler support" |
| @@ -1363,7 +1405,7 @@ config HIGHMEM4G · 389 unmodified lines | |||
| 1363 | 1405 | ||
| 1364 | config HIGHMEM64G | 1406 | config HIGHMEM64G |
| 1365 | bool "64GB" | 1407 | bool "64GB" |
| 1366 | depends on !M486 | 1408 | depends on !M486 && !M586 && !M586TSC && !M586MMX && !MGEODE_LX && !MGEODEGX1 && !MCYRIXIII && !MELAN && !MWINCHIPC6 && !WINCHIP3D && !MK6 |
| 1367 | select X86_PAE | 1409 | select X86_PAE |
| 1368 | ---help--- | 1410 | ---help--- |
| 1369 | Select this if you have a 32-bit processor and more than 4 | 1411 | Select this if you have a 32-bit processor and more than 4 |
| @@ -2265,7 +2307,7 @@ choice · 895 unmodified lines | |||
| 2265 | it can be used to assist security vulnerability exploitation. | 2307 | it can be used to assist security vulnerability exploitation. |
| 2266 | 2308 | ||
| 2267 | This setting can be changed at boot time via the kernel command | 2309 | This setting can be changed at boot time via the kernel command |
| 2268 | line parameter vsyscall=[native|emulate|none]. | 2310 | line parameter vsyscall=[emulate|none]. |
| 2269 | 2311 | ||
| 2270 | On a system with recent enough glibc (2.14 or newer) and no | 2312 | On a system with recent enough glibc (2.14 or newer) and no |
| 2271 | static binaries, you can say None without a performance penalty | 2313 | static binaries, you can say None without a performance penalty |
| @@ -2273,15 +2315,6 @@ choice · 1 unmodified line | |||
| 2273 | 2315 | ||
| 2274 | If unsure, select "Emulate". | 2316 | If unsure, select "Emulate". |
| 2275 | 2317 | ||
| 2276 | config LEGACY_VSYSCALL_NATIVE | ||
| 2277 | bool "Native" | ||
| 2278 | help | ||
| 2279 | Actual executable code is located in the fixed vsyscall | ||
| 2280 | address mapping, implementing time() efficiently. Since | ||
| 2281 | this makes the mapping executable, it can be used during | ||
| 2282 | security vulnerability exploitation (traditionally as | ||
| 2283 | ROP gadgets). This configuration is not recommended. | ||
| 2284 | |||
| 2285 | config LEGACY_VSYSCALL_EMULATE | 2318 | config LEGACY_VSYSCALL_EMULATE |
| 2286 | bool "Emulate" | 2319 | bool "Emulate" |
| 2287 | help | 2320 | help |
| @@ -315,19 +315,6 @@ config X86_L1_CACHE_SHIFT · 314 unmodified lines | |||
| 315 | default "4" if MELAN || M486 || MGEODEGX1 | 315 | default "4" if MELAN || M486 || MGEODEGX1 |
| 316 | default "5" if MWINCHIP3D || MWINCHIPC6 || MCRUSOE || MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX | 316 | default "5" if MWINCHIP3D || MWINCHIPC6 || MCRUSOE || MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX |
| 317 | 317 | ||
| 318 | config X86_PPRO_FENCE | ||
| 319 | bool "PentiumPro memory ordering errata workaround" | ||
| 320 | depends on M686 || M586MMX || M586TSC || M586 || M486 || MGEODEGX1 | ||
| 321 | ---help--- | ||
| 322 | Old PentiumPro multiprocessor systems had errata that could cause | ||
| 323 | memory operations to violate the x86 ordering standard in rare cases. | ||
| 324 | Enabling this option will attempt to work around some (but not all) | ||
| 325 | occurrences of this problem, at the cost of much heavier spinlock and | ||
| 326 | memory barrier operations. | ||
| 327 | |||
| 328 | If unsure, say n here. Even distro kernels should think twice before | ||
| 329 | enabling this: there are few systems, and an unlikely bug. | ||
| 330 | |||
| 331 | config X86_F00F_BUG | 318 | config X86_F00F_BUG |
| 332 | def_bool y | 319 | def_bool y |
| 333 | depends on M586MMX || M586TSC || M586 || M486 | 320 | depends on M586MMX || M586TSC || M586 || M486 |
| @@ -374,7 +361,7 @@ config X86_TSC · 40 unmodified lines | |||
| 374 | 361 | ||
| 375 | config X86_CMPXCHG64 | 362 | config X86_CMPXCHG64 |
| 376 | def_bool y | 363 | def_bool y |
| 377 | depends on X86_PAE || X86_64 || MCORE2 || MPENTIUM4 || MPENTIUMM || MPENTIUMIII || MPENTIUMII || M686 || MATOM | 364 | depends on X86_PAE || X86_64 || MCORE2 || MPENTIUM4 || MPENTIUMM || MPENTIUMIII || MPENTIUMII || M686 || M586TSC || M586MMX || MATOM || MGEODE_LX || MGEODEGX1 || MK6 || MK7 || MK8 |
| 378 | 365 | ||
| 379 | # this should be set for all -march=.. options where the compiler | 366 | # this should be set for all -march=.. options where the compiler |
| 380 | # generates cmov. | 367 | # generates cmov. |
| @@ -385,7 +372,7 @@ config X86_CMOV · 4 unmodified lines | |||
| 385 | config X86_MINIMUM_CPU_FAMILY | 372 | config X86_MINIMUM_CPU_FAMILY |
| 386 | int | 373 | int |
| 387 | default "64" if X86_64 | 374 | default "64" if X86_64 |
| 388 | default "6" if X86_32 && X86_P6_NOP | 375 | default "6" if X86_32 && (MPENTIUM4 || MPENTIUMM || MPENTIUMIII || MPENTIUMII || M686 || MVIAC3_2 || MVIAC7 || MEFFICEON || MATOM || MCRUSOE || MCORE2 || MK7 || MK8) |
| 389 | default "5" if X86_32 && X86_CMPXCHG64 | 376 | default "5" if X86_32 && X86_CMPXCHG64 |
| 390 | default "4" | 377 | default "4" |
| 391 | 378 | ||
| @@ -223,6 +223,15 @@ KBUILD_CFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) · 222 unmodified lines | |||
| 223 | 223 | ||
| 224 | LDFLAGS := -m elf_$(UTS_MACHINE) | 224 | LDFLAGS := -m elf_$(UTS_MACHINE) |
| 225 | 225 | ||
| 226 | # | ||
| 227 | # The 64-bit kernel must be aligned to 2MB. Pass -z max-page-size=0x200000 to | ||
| 228 | # the linker to force 2MB page size regardless of the default page size used | ||
| 229 | # by the linker. | ||
| 230 | # | ||
| 231 | ifdef CONFIG_X86_64 | ||
| 232 | LDFLAGS += $(call ld-option, -z max-page-size=0x200000) | ||
| 233 | endif | ||
| 234 | |||
| 226 | # Speed up the build | 235 | # Speed up the build |
| 227 | KBUILD_CFLAGS += -pipe | 236 | KBUILD_CFLAGS += -pipe |
| 228 | # Workaround for a gcc prelease that unfortunately was shipped in a suse release | 237 | # Workaround for a gcc prelease that unfortunately was shipped in a suse release |
| @@ -232,10 +241,9 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables · 3 unmodified lines | |||
| 232 | 241 | ||
| 233 | # Avoid indirect branches in kernel to deal with Spectre | 242 | # Avoid indirect branches in kernel to deal with Spectre |
| 234 | ifdef CONFIG_RETPOLINE | 243 | ifdef CONFIG_RETPOLINE |
| 235 | RETPOLINE_CFLAGS += $(call cc-option,-mindirect-branch=thunk-extern -mindirect-branch-register) | 244 | ifneq ($(RETPOLINE_CFLAGS),) |
| 236 | ifneq ($(RETPOLINE_CFLAGS),) | 245 | KBUILD_CFLAGS += $(RETPOLINE_CFLAGS) -DRETPOLINE |
| 237 | KBUILD_CFLAGS += $(RETPOLINE_CFLAGS) -DRETPOLINE | 246 | endif |
| 238 | endif | ||
| 239 | endif | 247 | endif |
| 240 | 248 | ||
| 241 | archscripts: scripts_basic | 249 | archscripts: scripts_basic |
| @@ -439,7 +439,7 @@ setup_uga32(void **uga_handle, unsigned long size, u32 *width, u32 *height) · 438 unmodified lines | |||
| 439 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; | 439 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; |
| 440 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; | 440 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; |
| 441 | unsigned long nr_ugas; | 441 | unsigned long nr_ugas; |
| 442 | u32 *handles = (u32 *)uga_handle;; | 442 | u32 *handles = (u32 *)uga_handle; |
| 443 | efi_status_t status = EFI_INVALID_PARAMETER; | 443 | efi_status_t status = EFI_INVALID_PARAMETER; |
| 444 | int i; | 444 | int i; |
| 445 | 445 | ||
| @@ -484,7 +484,7 @@ setup_uga64(void **uga_handle, unsigned long size, u32 *width, u32 *height) · 38 unmodified lines | |||
| 484 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; | 484 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; |
| 485 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; | 485 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; |
| 486 | unsigned long nr_ugas; | 486 | unsigned long nr_ugas; |
| 487 | u64 *handles = (u64 *)uga_handle;; | 487 | u64 *handles = (u64 *)uga_handle; |
| 488 | efi_status_t status = EFI_INVALID_PARAMETER; | 488 | efi_status_t status = EFI_INVALID_PARAMETER; |
| 489 | int i; | 489 | int i; |
| 490 | 490 | ||
| @@ -309,6 +309,10 @@ static void parse_elf(void *output) · 308 unmodified lines | |||
| 309 | 309 | ||
| 310 | switch (phdr->p_type) { | 310 | switch (phdr->p_type) { |
| 311 | case PT_LOAD: | 311 | case PT_LOAD: |
| 312 | #ifdef CONFIG_X86_64 | ||
| 313 | if ((phdr->p_align % 0x200000) != 0) | ||
| 314 | error("Alignment of LOAD segment isn't multiple of 2MB"); | ||
| 315 | #endif | ||
| 312 | #ifdef CONFIG_RELOCATABLE | 316 | #ifdef CONFIG_RELOCATABLE |
| 313 | dest = output; | 317 | dest = output; |
| 314 | dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR); | 318 | dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR); |
| @@ -57,10 +57,12 @@ void sha512_mb_mgr_init_avx2(struct sha512_mb_mgr *state) · 56 unmodified lines | |||
| 57 | { | 57 | { |
| 58 | unsigned int j; | 58 | unsigned int j; |
| 59 | 59 | ||
| 60 | state->lens[0] = 0; | 60 | /* initially all lanes are unused */ |
| 61 | state->lens[1] = 1; | 61 | state->lens[0] = 0xFFFFFFFF00000000; |
| 62 | state->lens[2] = 2; | 62 | state->lens[1] = 0xFFFFFFFF00000001; |
| 63 | state->lens[3] = 3; | 63 | state->lens[2] = 0xFFFFFFFF00000002; |
| 64 | state->lens[3] = 0xFFFFFFFF00000003; | ||
| 65 | |||
| 64 | state->unused_lanes = 0xFF03020100; | 66 | state->unused_lanes = 0xFF03020100; |
| 65 | for (j = 0; j < 4; j++) | 67 | for (j = 0; j < 4; j++) |
| 66 | state->ldata[j].job_in_lane = NULL; | 68 | state->ldata[j].job_in_lane = NULL; |
| @@ -97,80 +97,78 @@ For 32-bit we have the following conventions - kernel is built with · 96 unmodified lines | |||
| 97 | 97 | ||
| 98 | #define SIZEOF_PTREGS 21*8 | 98 | #define SIZEOF_PTREGS 21*8 |
| 99 | 99 | ||
| 100 | .macro ALLOC_PT_GPREGS_ON_STACK | 100 | .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax save_ret=0 |
| 101 | addq $-(15*8), %rsp | 101 | /* |
| 102 | .endm | 102 | * Push registers and sanitize registers of values that a |
| 103 | 103 | * speculation attack might otherwise want to exploit. The | |
| 104 | .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8910=1 r11=1 | 104 | * lower registers are likely clobbered well before they |
| 105 | .if \r11 | 105 | * could be put to use in a speculative execution gadget. |
| 106 | movq %r11, 6*8+\offset(%rsp) | 106 | * Interleave XOR with PUSH for better uop scheduling: |
| 107 | .endif | 107 | */ |
| 108 | .if \r8910 | 108 | .if \save_ret |
| 109 | movq %r10, 7*8+\offset(%rsp) | 109 | pushq %rsi /* pt_regs->si */ |
| 110 | movq %r9, 8*8+\offset(%rsp) | 110 | movq 8(%rsp), %rsi /* temporarily store the return address in %rsi */ |
| 111 | movq %r8, 9*8+\offset(%rsp) | 111 | movq %rdi, 8(%rsp) /* pt_regs->di (overwriting original return address) */ |
| 112 | .endif | 112 | .else |
| 113 | .if \rax | 113 | pushq %rdi /* pt_regs->di */ |
| 114 | movq %rax, 10*8+\offset(%rsp) | 114 | pushq %rsi /* pt_regs->si */ |
| 115 | .endif | 115 | .endif |
| 116 | .if \rcx | 116 | pushq \rdx /* pt_regs->dx */ |
| 117 | movq %rcx, 11*8+\offset(%rsp) | 117 | pushq %rcx /* pt_regs->cx */ |
| 118 | pushq \rax /* pt_regs->ax */ | ||
| 119 | pushq %r8 /* pt_regs->r8 */ | ||
| 120 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 121 | pushq %r9 /* pt_regs->r9 */ | ||
| 122 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 123 | pushq %r10 /* pt_regs->r10 */ | ||
| 124 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 125 | pushq %r11 /* pt_regs->r11 */ | ||
| 126 | xorl %r11d, %r11d /* nospec r11*/ | ||
| 127 | pushq %rbx /* pt_regs->rbx */ | ||
| 128 | xorl %ebx, %ebx /* nospec rbx*/ | ||
| 129 | pushq %rbp /* pt_regs->rbp */ | ||
| 130 | xorl %ebp, %ebp /* nospec rbp*/ | ||
| 131 | pushq %r12 /* pt_regs->r12 */ | ||
| 132 | xorl %r12d, %r12d /* nospec r12*/ | ||
| 133 | pushq %r13 /* pt_regs->r13 */ | ||
| 134 | xorl %r13d, %r13d /* nospec r13*/ | ||
| 135 | pushq %r14 /* pt_regs->r14 */ | ||
| 136 | xorl %r14d, %r14d /* nospec r14*/ | ||
| 137 | pushq %r15 /* pt_regs->r15 */ | ||
| 138 | xorl %r15d, %r15d /* nospec r15*/ | ||
| 139 | UNWIND_HINT_REGS | ||
| 140 | .if \save_ret | ||
| 141 | pushq %rsi /* return address on top of stack */ | ||
| 118 | .endif | 142 | .endif |
| 119 | movq %rdx, 12*8+\offset(%rsp) | 143 | .endm |
| 120 | movq %rsi, 13*8+\offset(%rsp) | 144 | |
| 121 | movq %rdi, 14*8+\offset(%rsp) | 145 | .macro POP_REGS pop_rdi=1 skip_r11rcx=0 |
| 122 | UNWIND_HINT_REGS offset=\offset extra=0 | ||
| 123 | .endm | ||
| 124 | .macro SAVE_C_REGS offset=0 | ||
| 125 | SAVE_C_REGS_HELPER \offset, 1, 1, 1, 1 | ||
| 126 | .endm | ||
| 127 | .macro SAVE_C_REGS_EXCEPT_RAX_RCX offset=0 | ||
| 128 | SAVE_C_REGS_HELPER \offset, 0, 0, 1, 1 | ||
| 129 | .endm | ||
| 130 | .macro SAVE_C_REGS_EXCEPT_R891011 | ||
| 131 | SAVE_C_REGS_HELPER 0, 1, 1, 0, 0 | ||
| 132 | .endm | ||
| 133 | .macro SAVE_C_REGS_EXCEPT_RCX_R891011 | ||
| 134 | SAVE_C_REGS_HELPER 0, 1, 0, 0, 0 | ||
| 135 | .endm | ||
| 136 | .macro SAVE_C_REGS_EXCEPT_RAX_RCX_R11 | ||
| 137 | SAVE_C_REGS_HELPER 0, 0, 0, 1, 0 | ||
| 138 | .endm | ||
| 139 | |||
| 140 | .macro SAVE_EXTRA_REGS offset=0 | ||
| 141 | movq %r15, 0*8+\offset(%rsp) | ||
| 142 | movq %r14, 1*8+\offset(%rsp) | ||
| 143 | movq %r13, 2*8+\offset(%rsp) | ||
| 144 | movq %r12, 3*8+\offset(%rsp) | ||
| 145 | movq %rbp, 4*8+\offset(%rsp) | ||
| 146 | movq %rbx, 5*8+\offset(%rsp) | ||
| 147 | UNWIND_HINT_REGS offset=\offset | ||
| 148 | .endm | ||
| 149 | |||
| 150 | .macro POP_EXTRA_REGS | ||
| 151 | popq %r15 | 146 | popq %r15 |
| 152 | popq %r14 | 147 | popq %r14 |
| 153 | popq %r13 | 148 | popq %r13 |
| 154 | popq %r12 | 149 | popq %r12 |
| 155 | popq %rbp | 150 | popq %rbp |
| 156 | popq %rbx | 151 | popq %rbx |
| 157 | .endm | 152 | .if \skip_r11rcx |
| 158 | 153 | popq %rsi | |
| 159 | .macro POP_C_REGS | 154 | .else |
| 160 | popq %r11 | 155 | popq %r11 |
| 156 | .endif | ||
| 161 | popq %r10 | 157 | popq %r10 |
| 162 | popq %r9 | 158 | popq %r9 |
| 163 | popq %r8 | 159 | popq %r8 |
| 164 | popq %rax | 160 | popq %rax |
| 161 | .if \skip_r11rcx | ||
| 162 | popq %rsi | ||
| 163 | .else | ||
| 165 | popq %rcx | 164 | popq %rcx |
| 165 | .endif | ||
| 166 | popq %rdx | 166 | popq %rdx |
| 167 | popq %rsi | 167 | popq %rsi |
| 168 | .if \pop_rdi | ||
| 168 | popq %rdi | 169 | popq %rdi |
| 169 | .endm | 170 | .endif |
| 170 | 171 | .endm | |
| 171 | .macro icebp | ||
| 172 | .byte 0xf1 | ||
| 173 | .endm | ||
| 174 | 172 | ||
| 175 | /* | 173 | /* |
| 176 | * This is a sneaky trick to help the unwinder find pt_regs on the stack. The | 174 | * This is a sneaky trick to help the unwinder find pt_regs on the stack. The |
| @@ -178,17 +176,12 @@ For 32-bit we have the following conventions - kernel is built with · 1 unmodified line | |||
| 178 | * is just setting the LSB, which makes it an invalid stack address and is also | 176 | * is just setting the LSB, which makes it an invalid stack address and is also |
| 179 | * a signal to the unwinder that it's a pt_regs pointer in disguise. | 177 | * a signal to the unwinder that it's a pt_regs pointer in disguise. |
| 180 | * | 178 | * |
| 181 | * NOTE: This macro must be used *after* SAVE_EXTRA_REGS because it corrupts | 179 | * NOTE: This macro must be used *after* PUSH_AND_CLEAR_REGS because it corrupts |
| 182 | * the original rbp. | 180 | * the original rbp. |
| 183 | */ | 181 | */ |
| 184 | .macro ENCODE_FRAME_POINTER ptregs_offset=0 | 182 | .macro ENCODE_FRAME_POINTER ptregs_offset=0 |
| 185 | #ifdef CONFIG_FRAME_POINTER | 183 | #ifdef CONFIG_FRAME_POINTER |
| 186 | .if \ptregs_offset | 184 | leaq 1+\ptregs_offset(%rsp), %rbp |
| 187 | leaq \ptregs_offset(%rsp), %rbp | ||
| 188 | .else | ||
| 189 | mov %rsp, %rbp | ||
| 190 | .endif | ||
| 191 | orq $0x1, %rbp | ||
| 192 | #endif | 185 | #endif |
| 193 | .endm | 186 | .endm |
| 194 | 187 | ||
| @@ -252,8 +252,7 @@ ENTRY(__switch_to_asm) · 251 unmodified lines | |||
| 252 | * exist, overwrite the RSB with entries which capture | 252 | * exist, overwrite the RSB with entries which capture |
| 253 | * speculative execution to prevent attack. | 253 | * speculative execution to prevent attack. |
| 254 | */ | 254 | */ |
| 255 | /* Clobbers %ebx */ | 255 | FILL_RETURN_BUFFER %ebx, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW |
| 256 | FILL_RETURN_BUFFER RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW | ||
| 257 | #endif | 256 | #endif |
| 258 | 257 | ||
| 259 | /* restore callee-saved registers */ | 258 | /* restore callee-saved registers */ |
| @@ -55,7 +55,7 @@ END(native_usergs_sysret64) · 54 unmodified lines | |||
| 55 | 55 | ||
| 56 | .macro TRACE_IRQS_FLAGS flags:req | 56 | .macro TRACE_IRQS_FLAGS flags:req |
| 57 | #ifdef CONFIG_TRACE_IRQFLAGS | 57 | #ifdef CONFIG_TRACE_IRQFLAGS |
| 58 | bt $9, \flags /* interrupts off? */ | 58 | btl $9, \flags /* interrupts off? */ |
| 59 | jnc 1f | 59 | jnc 1f |
| 60 | TRACE_IRQS_ON | 60 | TRACE_IRQS_ON |
| 61 | 1: | 61 | 1: |
| @@ -213,7 +213,7 @@ ENTRY(entry_SYSCALL_64) · 151 unmodified lines | |||
| 213 | 213 | ||
| 214 | swapgs | 214 | swapgs |
| 215 | /* | 215 | /* |
| 216 | * This path is not taken when PAGE_TABLE_ISOLATION is disabled so it | 216 | * This path is only taken when PAGE_TABLE_ISOLATION is disabled so it |
| 217 | * is not required to switch CR3. | 217 | * is not required to switch CR3. |
| 218 | */ | 218 | */ |
| 219 | movq %rsp, PER_CPU_VAR(rsp_scratch) | 219 | movq %rsp, PER_CPU_VAR(rsp_scratch) |
| @@ -227,22 +227,8 @@ ENTRY(entry_SYSCALL_64) · 7 unmodified lines | |||
| 227 | pushq %rcx /* pt_regs->ip */ | 227 | pushq %rcx /* pt_regs->ip */ |
| 228 | GLOBAL(entry_SYSCALL_64_after_hwframe) | 228 | GLOBAL(entry_SYSCALL_64_after_hwframe) |
| 229 | pushq %rax /* pt_regs->orig_ax */ | 229 | pushq %rax /* pt_regs->orig_ax */ |
| 230 | pushq %rdi /* pt_regs->di */ | 230 | |
| 231 | pushq %rsi /* pt_regs->si */ | 231 | PUSH_AND_CLEAR_REGS rax=$-ENOSYS |
| 232 | pushq %rdx /* pt_regs->dx */ | ||
| 233 | pushq %rcx /* pt_regs->cx */ | ||
| 234 | pushq $-ENOSYS /* pt_regs->ax */ | ||
| 235 | pushq %r8 /* pt_regs->r8 */ | ||
| 236 | pushq %r9 /* pt_regs->r9 */ | ||
| 237 | pushq %r10 /* pt_regs->r10 */ | ||
| 238 | pushq %r11 /* pt_regs->r11 */ | ||
| 239 | pushq %rbx /* pt_regs->rbx */ | ||
| 240 | pushq %rbp /* pt_regs->rbp */ | ||
| 241 | pushq %r12 /* pt_regs->r12 */ | ||
| 242 | pushq %r13 /* pt_regs->r13 */ | ||
| 243 | pushq %r14 /* pt_regs->r14 */ | ||
| 244 | pushq %r15 /* pt_regs->r15 */ | ||
| 245 | UNWIND_HINT_REGS | ||
| 246 | 232 | ||
| 247 | TRACE_IRQS_OFF | 233 | TRACE_IRQS_OFF |
| 248 | 234 | ||
| @@ -321,15 +307,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe) · 72 unmodified lines | |||
| 321 | syscall_return_via_sysret: | 307 | syscall_return_via_sysret: |
| 322 | /* rcx and r11 are already restored (see code above) */ | 308 | /* rcx and r11 are already restored (see code above) */ |
| 323 | UNWIND_HINT_EMPTY | 309 | UNWIND_HINT_EMPTY |
| 324 | POP_EXTRA_REGS | 310 | POP_REGS pop_rdi=0 skip_r11rcx=1 |
| 325 | popq %rsi /* skip r11 */ | ||
| 326 | popq %r10 | ||
| 327 | popq %r9 | ||
| 328 | popq %r8 | ||
| 329 | popq %rax | ||
| 330 | popq %rsi /* skip rcx */ | ||
| 331 | popq %rdx | ||
| 332 | popq %rsi | ||
| 333 | 311 | ||
| 334 | /* | 312 | /* |
| 335 | * Now all regs are restored except RSP and RDI. | 313 | * Now all regs are restored except RSP and RDI. |
| @@ -386,8 +364,7 @@ ENTRY(__switch_to_asm) · 50 unmodified lines | |||
| 386 | * exist, overwrite the RSB with entries which capture | 364 | * exist, overwrite the RSB with entries which capture |
| 387 | * speculative execution to prevent attack. | 365 | * speculative execution to prevent attack. |
| 388 | */ | 366 | */ |
| 389 | /* Clobbers %rbx */ | 367 | FILL_RETURN_BUFFER %r12, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW |
| 390 | FILL_RETURN_BUFFER RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW | ||
| 391 | #endif | 368 | #endif |
| 392 | 369 | ||
| 393 | /* restore callee-saved registers */ | 370 | /* restore callee-saved registers */ |
| @@ -471,9 +448,19 @@ END(irq_entries_start) · 77 unmodified lines | |||
| 471 | * | 448 | * |
| 472 | * The invariant is that, if irq_count != -1, then the IRQ stack is in use. | 449 | * The invariant is that, if irq_count != -1, then the IRQ stack is in use. |
| 473 | */ | 450 | */ |
| 474 | .macro ENTER_IRQ_STACK regs=1 old_rsp | 451 | .macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0 |
| 475 | DEBUG_ENTRY_ASSERT_IRQS_OFF | 452 | DEBUG_ENTRY_ASSERT_IRQS_OFF |
| 453 | |||
| 454 | .if \save_ret | ||
| 455 | /* | ||
| 456 | * If save_ret is set, the original stack contains one additional | ||
| 457 | * entry -- the return address. Therefore, move the address one | ||
| 458 | * entry below %rsp to \old_rsp. | ||
| 459 | */ | ||
| 460 | leaq 8(%rsp), \old_rsp | ||
| 461 | .else | ||
| 476 | movq %rsp, \old_rsp | 462 | movq %rsp, \old_rsp |
| 463 | .endif | ||
| 477 | 464 | ||
| 478 | .if \regs | 465 | .if \regs |
| 479 | UNWIND_HINT_REGS base=\old_rsp | 466 | UNWIND_HINT_REGS base=\old_rsp |
| @@ -519,6 +506,15 @@ END(irq_entries_start) · 39 unmodified lines | |||
| 519 | .if \regs | 506 | .if \regs |
| 520 | UNWIND_HINT_REGS indirect=1 | 507 | UNWIND_HINT_REGS indirect=1 |
| 521 | .endif | 508 | .endif |
| 509 | |||
| 510 | .if \save_ret | ||
| 511 | /* | ||
| 512 | * Push the return address to the stack. This return address can | ||
| 513 | * be found at the "real" original RSP, which was offset by 8 at | ||
| 514 | * the beginning of this macro. | ||
| 515 | */ | ||
| 516 | pushq -8(\old_rsp) | ||
| 517 | .endif | ||
| 522 | .endm | 518 | .endm |
| 523 | 519 | ||
| 524 | /* | 520 | /* |
| @@ -542,29 +538,65 @@ END(irq_entries_start) · 17 unmodified lines | |||
| 542 | .endm | 538 | .endm |
| 543 | 539 | ||
| 544 | /* | 540 | /* |
| 545 | * Interrupt entry/exit. | 541 | * Interrupt entry helper function. |
| 546 | * | ||
| 547 | * Interrupt entry points save only callee clobbered registers in fast path. | ||
| 548 | * | 542 | * |
| 549 | * Entry runs with interrupts off. | 543 | * Entry runs with interrupts off. Stack layout at entry: |
| 544 | * +----------------------------------------------------+ | ||
| 545 | * | regs->ss | | ||
| 546 | * | regs->rsp | | ||
| 547 | * | regs->eflags | | ||
| 548 | * | regs->cs | | ||
| 549 | * | regs->ip | | ||
| 550 | * +----------------------------------------------------+ | ||
| 551 | * | regs->orig_ax = ~(interrupt number) | | ||
| 552 | * +----------------------------------------------------+ | ||
| 553 | * | return address | | ||
| 554 | * +----------------------------------------------------+ | ||
| 550 | */ | 555 | */ |
| 551 | 556 | ENTRY(interrupt_entry) | |
| 552 | /* 0(%rsp): ~(interrupt number) */ | 557 | UNWIND_HINT_FUNC |
| 553 | .macro interrupt func | 558 | ASM_CLAC |
| 554 | cld | 559 | cld |
| 555 | 560 | ||
| 556 | testb $3, CS-ORIG_RAX(%rsp) | 561 | testb $3, CS-ORIG_RAX+8(%rsp) |
| 557 | jz 1f | 562 | jz 1f |
| 558 | SWAPGS | 563 | SWAPGS |
| 559 | call switch_to_thread_stack | 564 | |
| 565 | /* | ||
| 566 | * Switch to the thread stack. The IRET frame and orig_ax are | ||
| 567 | * on the stack, as well as the return address. RDI..R12 are | ||
| 568 | * not (yet) on the stack and space has not (yet) been | ||
| 569 | * allocated for them. | ||
| 570 | */ | ||
| 571 | pushq %rdi | ||
| 572 | |||
| 573 | /* Need to switch before accessing the thread stack. */ | ||
| 574 | SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi | ||
| 575 | movq %rsp, %rdi | ||
| 576 | movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp | ||
| 577 | |||
| 578 | /* | ||
| 579 | * We have RDI, return address, and orig_ax on the stack on | ||
| 580 | * top of the IRET frame. That means offset=24 | ||
| 581 | */ | ||
| 582 | UNWIND_HINT_IRET_REGS base=%rdi offset=24 | ||
| 583 | |||
| 584 | pushq 7*8(%rdi) /* regs->ss */ | ||
| 585 | pushq 6*8(%rdi) /* regs->rsp */ | ||
| 586 | pushq 5*8(%rdi) /* regs->eflags */ | ||
| 587 | pushq 4*8(%rdi) /* regs->cs */ | ||
| 588 | pushq 3*8(%rdi) /* regs->ip */ | ||
| 589 | pushq 2*8(%rdi) /* regs->orig_ax */ | ||
| 590 | pushq 8(%rdi) /* return address */ | ||
| 591 | UNWIND_HINT_FUNC | ||
| 592 | |||
| 593 | movq (%rdi), %rdi | ||
| 560 | 1: | 594 | 1: |
| 561 | 595 | ||
| 562 | ALLOC_PT_GPREGS_ON_STACK | 596 | PUSH_AND_CLEAR_REGS save_ret=1 |
| 563 | SAVE_C_REGS | 597 | ENCODE_FRAME_POINTER 8 |
| 564 | SAVE_EXTRA_REGS | ||
| 565 | ENCODE_FRAME_POINTER | ||
| 566 | 598 | ||
| 567 | testb $3, CS(%rsp) | 599 | testb $3, CS+8(%rsp) |
| 568 | jz 1f | 600 | jz 1f |
| 569 | 601 | ||
| 570 | /* | 602 | /* |
| @@ -572,7 +604,7 @@ END(irq_entries_start) · 1 unmodified line | |||
| 572 | * | 604 | * |
| 573 | * We need to tell lockdep that IRQs are off. We can't do this until | 605 | * We need to tell lockdep that IRQs are off. We can't do this until |
| 574 | * we fix gsbase, and we should do it before enter_from_user_mode | 606 | * we fix gsbase, and we should do it before enter_from_user_mode |
| 575 | * (which can take locks). Since TRACE_IRQS_OFF idempotent, | 607 | * (which can take locks). Since TRACE_IRQS_OFF is idempotent, |
| 576 | * the simplest way to handle it is to just call it twice if | 608 | * the simplest way to handle it is to just call it twice if |
| 577 | * we enter from user mode. There's no reason to optimize this since | 609 | * we enter from user mode. There's no reason to optimize this since |
| 578 | * TRACE_IRQS_OFF is a no-op if lockdep is off. | 610 | * TRACE_IRQS_OFF is a no-op if lockdep is off. |
| @@ -582,12 +614,15 @@ END(irq_entries_start) · 3 unmodified lines | |||
| 582 | CALL_enter_from_user_mode | 614 | CALL_enter_from_user_mode |
| 583 | 615 | ||
| 584 | 1: | 616 | 1: |
| 585 | ENTER_IRQ_STACK old_rsp=%rdi | 617 | ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 |
| 586 | /* We entered an interrupt context - irqs are off: */ | 618 | /* We entered an interrupt context - irqs are off: */ |
| 587 | TRACE_IRQS_OFF | 619 | TRACE_IRQS_OFF |
| 588 | 620 | ||
| 589 | call \func /* rdi points to pt_regs */ | 621 | ret |
| 590 | .endm | 622 | END(interrupt_entry) |
| 623 | |||
| 624 | |||
| 625 | /* Interrupt entry/exit. */ | ||
| 591 | 626 | ||
| 592 | /* | 627 | /* |
| 593 | * The interrupt stubs push (~vector+0x80) onto the stack and | 628 | * The interrupt stubs push (~vector+0x80) onto the stack and |
| @@ -595,9 +630,10 @@ END(irq_entries_start) · 1 unmodified line | |||
| 595 | */ | 630 | */ |
| 596 | .p2align CONFIG_X86_L1_CACHE_SHIFT | 631 | .p2align CONFIG_X86_L1_CACHE_SHIFT |
| 597 | common_interrupt: | 632 | common_interrupt: |
| 598 | ASM_CLAC | ||
| 599 | addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ | 633 | addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ |
| 600 | interrupt do_IRQ | 634 | call interrupt_entry |
| 635 | UNWIND_HINT_REGS indirect=1 | ||
| 636 | call do_IRQ /* rdi points to pt_regs */ | ||
| 601 | /* 0(%rsp): old RSP */ | 637 | /* 0(%rsp): old RSP */ |
| 602 | ret_from_intr: | 638 | ret_from_intr: |
| 603 | DISABLE_INTERRUPTS(CLBR_ANY) | 639 | DISABLE_INTERRUPTS(CLBR_ANY) |
| @@ -622,15 +658,7 @@ GLOBAL(swapgs_restore_regs_and_return_to_usermode) · 18 unmodified lines | |||
| 622 | ud2 | 658 | ud2 |
| 623 | 1: | 659 | 1: |
| 624 | #endif | 660 | #endif |
| 625 | POP_EXTRA_REGS | 661 | POP_REGS pop_rdi=0 |
| 626 | popq %r11 | ||
| 627 | popq %r10 | ||
| 628 | popq %r9 | ||
| 629 | popq %r8 | ||
| 630 | popq %rax | ||
| 631 | popq %rcx | ||
| 632 | popq %rdx | ||
| 633 | popq %rsi | ||
| 634 | 662 | ||
| 635 | /* | 663 | /* |
| 636 | * The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS. | 664 | * The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS. |
| @@ -688,8 +716,7 @@ GLOBAL(restore_regs_and_return_to_kernel) · 51 unmodified lines | |||
| 688 | ud2 | 716 | ud2 |
| 689 | 1: | 717 | 1: |
| 690 | #endif | 718 | #endif |
| 691 | POP_EXTRA_REGS | 719 | POP_REGS |
| 692 | POP_C_REGS | ||
| 693 | addq $8, %rsp /* skip regs->orig_ax */ | 720 | addq $8, %rsp /* skip regs->orig_ax */ |
| 694 | /* | 721 | /* |
| 695 | * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization | 722 | * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization |
| @@ -799,10 +826,11 @@ END(common_interrupt) · 103 unmodified lines | |||
| 799 | .macro apicinterrupt3 num sym do_sym | 826 | .macro apicinterrupt3 num sym do_sym |
| 800 | ENTRY(\sym) | 827 | ENTRY(\sym) |
| 801 | UNWIND_HINT_IRET_REGS | 828 | UNWIND_HINT_IRET_REGS |
| 802 | ASM_CLAC | ||
| 803 | pushq $~(\num) | 829 | pushq $~(\num) |
| 804 | .Lcommon_\sym: | 830 | .Lcommon_\sym: |
| 805 | interrupt \do_sym | 831 | call interrupt_entry |
| 832 | UNWIND_HINT_REGS indirect=1 | ||
| 833 | call \do_sym /* rdi points to pt_regs */ | ||
| 806 | jmp ret_from_intr | 834 | jmp ret_from_intr |
| 807 | END(\sym) | 835 | END(\sym) |
| 808 | .endm | 836 | .endm |
| @@ -865,34 +893,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt · 56 unmodified lines | |||
| 865 | */ | 893 | */ |
| 866 | #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) | 894 | #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) |
| 867 | 895 | ||
| 868 | /* | ||
| 869 | * Switch to the thread stack. This is called with the IRET frame and | ||
| 870 | * orig_ax on the stack. (That is, RDI..R12 are not on the stack and | ||
| 871 | * space has not been allocated for them.) | ||
| 872 | */ | ||
| 873 | ENTRY(switch_to_thread_stack) | ||
| 874 | UNWIND_HINT_FUNC | ||
| 875 | |||
| 876 | pushq %rdi | ||
| 877 | /* Need to switch before accessing the thread stack. */ | ||
| 878 | SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi | ||
| 879 | movq %rsp, %rdi | ||
| 880 | movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp | ||
| 881 | UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI | ||
| 882 | |||
| 883 | pushq 7*8(%rdi) /* regs->ss */ | ||
| 884 | pushq 6*8(%rdi) /* regs->rsp */ | ||
| 885 | pushq 5*8(%rdi) /* regs->eflags */ | ||
| 886 | pushq 4*8(%rdi) /* regs->cs */ | ||
| 887 | pushq 3*8(%rdi) /* regs->ip */ | ||
| 888 | pushq 2*8(%rdi) /* regs->orig_ax */ | ||
| 889 | pushq 8(%rdi) /* return address */ | ||
| 890 | UNWIND_HINT_FUNC | ||
| 891 | |||
| 892 | movq (%rdi), %rdi | ||
| 893 | ret | ||
| 894 | END(switch_to_thread_stack) | ||
| 895 | |||
| 896 | .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 | 896 | .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 |
| 897 | ENTRY(\sym) | 897 | ENTRY(\sym) |
| 898 | UNWIND_HINT_IRET_REGS offset=\has_error_code*8 | 898 | UNWIND_HINT_IRET_REGS offset=\has_error_code*8 |
| @@ -908,10 +908,8 @@ ENTRY(\sym) · 9 unmodified lines | |||
| 908 | pushq $-1 /* ORIG_RAX: no syscall to restart */ | 908 | pushq $-1 /* ORIG_RAX: no syscall to restart */ |
| 909 | .endif | 909 | .endif |
| 910 | 910 | ||
| 911 | ALLOC_PT_GPREGS_ON_STACK | ||
| 912 | |||
| 913 | .if \paranoid < 2 | 911 | .if \paranoid < 2 |
| 914 | testb $3, CS(%rsp) /* If coming from userspace, switch stacks */ | 912 | testb $3, CS-ORIG_RAX(%rsp) /* If coming from userspace, switch stacks */ |
| 915 | jnz .Lfrom_usermode_switch_stack_\@ | 913 | jnz .Lfrom_usermode_switch_stack_\@ |
| 916 | .endif | 914 | .endif |
| 917 | 915 | ||
| @@ -1121,9 +1119,7 @@ ENTRY(xen_failsafe_callback) · 203 unmodified lines | |||
| 1121 | addq $0x30, %rsp | 1119 | addq $0x30, %rsp |
| 1122 | UNWIND_HINT_IRET_REGS | 1120 | UNWIND_HINT_IRET_REGS |
| 1123 | pushq $-1 /* orig_ax = -1 => not a system call */ | 1121 | pushq $-1 /* orig_ax = -1 => not a system call */ |
| 1124 | ALLOC_PT_GPREGS_ON_STACK | 1122 | PUSH_AND_CLEAR_REGS |
| 1125 | SAVE_C_REGS | ||
| 1126 | SAVE_EXTRA_REGS | ||
| 1127 | ENCODE_FRAME_POINTER | 1123 | ENCODE_FRAME_POINTER |
| 1128 | jmp error_exit | 1124 | jmp error_exit |
| 1129 | END(xen_failsafe_callback) | 1125 | END(xen_failsafe_callback) |
| @@ -1142,7 +1138,7 @@ apicinterrupt3 HYPERV_REENLIGHTENMENT_VECTOR \ · 12 unmodified lines | |||
| 1142 | #endif /* CONFIG_HYPERV */ | 1138 | #endif /* CONFIG_HYPERV */ |
| 1143 | 1139 | ||
| 1144 | idtentry debug do_debug has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK | 1140 | idtentry debug do_debug has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK |
| 1145 | idtentry int3 do_int3 has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK | 1141 | idtentry int3 do_int3 has_error_code=0 |
| 1146 | idtentry stack_segment do_stack_segment has_error_code=1 | 1142 | idtentry stack_segment do_stack_segment has_error_code=1 |
| 1147 | 1143 | ||
| 1148 | #ifdef CONFIG_XEN | 1144 | #ifdef CONFIG_XEN |
| @@ -1170,8 +1166,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1 · 21 unmodified lines | |||
| 1170 | ENTRY(paranoid_entry) | 1166 | ENTRY(paranoid_entry) |
| 1171 | UNWIND_HINT_FUNC | 1167 | UNWIND_HINT_FUNC |
| 1172 | cld | 1168 | cld |
| 1173 | SAVE_C_REGS 8 | 1169 | PUSH_AND_CLEAR_REGS save_ret=1 |
| 1174 | SAVE_EXTRA_REGS 8 | ||
| 1175 | ENCODE_FRAME_POINTER 8 | 1170 | ENCODE_FRAME_POINTER 8 |
| 1176 | movl $1, %ebx | 1171 | movl $1, %ebx |
| 1177 | movl $MSR_GS_BASE, %ecx | 1172 | movl $MSR_GS_BASE, %ecx |
| @@ -1211,21 +1206,20 @@ ENTRY(paranoid_exit) · 33 unmodified lines | |||
| 1211 | jmp .Lparanoid_exit_restore | 1206 | jmp .Lparanoid_exit_restore |
| 1212 | .Lparanoid_exit_no_swapgs: | 1207 | .Lparanoid_exit_no_swapgs: |
| 1213 | TRACE_IRQS_IRETQ_DEBUG | 1208 | TRACE_IRQS_IRETQ_DEBUG |
| 1209 | RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 | ||
| 1214 | .Lparanoid_exit_restore: | 1210 | .Lparanoid_exit_restore: |
| 1215 | jmp restore_regs_and_return_to_kernel | 1211 | jmp restore_regs_and_return_to_kernel |
| 1216 | END(paranoid_exit) | 1212 | END(paranoid_exit) |
| 1217 | 1213 | ||
| 1218 | /* | 1214 | /* |
| 1219 | * Save all registers in pt_regs, and switch gs if needed. | 1215 | * Save all registers in pt_regs, and switch GS if needed. |
| 1220 | * Return: EBX=0: came from user mode; EBX=1: otherwise | 1216 | * Return: EBX=0: came from user mode; EBX=1: otherwise |
| 1221 | */ | 1217 | */ |
| 1222 | ENTRY(error_entry) | 1218 | ENTRY(error_entry) |
| 1223 | UNWIND_HINT_FUNC | 1219 | UNWIND_HINT_FUNC |
| 1224 | cld | 1220 | cld |
| 1225 | SAVE_C_REGS 8 | 1221 | PUSH_AND_CLEAR_REGS save_ret=1 |
| 1226 | SAVE_EXTRA_REGS 8 | ||
| 1227 | ENCODE_FRAME_POINTER 8 | 1222 | ENCODE_FRAME_POINTER 8 |
| 1228 | xorl %ebx, %ebx | ||
| 1229 | testb $3, CS+8(%rsp) | 1223 | testb $3, CS+8(%rsp) |
| 1230 | jz .Lerror_kernelspace | 1224 | jz .Lerror_kernelspace |
| 1231 | 1225 | ||
| @@ -1406,22 +1400,7 @@ ENTRY(nmi) · 174 unmodified lines | |||
| 1406 | pushq 1*8(%rdx) /* pt_regs->rip */ | 1400 | pushq 1*8(%rdx) /* pt_regs->rip */ |
| 1407 | UNWIND_HINT_IRET_REGS | 1401 | UNWIND_HINT_IRET_REGS |
| 1408 | pushq $-1 /* pt_regs->orig_ax */ | 1402 | pushq $-1 /* pt_regs->orig_ax */ |
| 1409 | pushq %rdi /* pt_regs->di */ | 1403 | PUSH_AND_CLEAR_REGS rdx=(%rdx) |
| 1410 | pushq %rsi /* pt_regs->si */ | ||
| 1411 | pushq (%rdx) /* pt_regs->dx */ | ||
| 1412 | pushq %rcx /* pt_regs->cx */ | ||
| 1413 | pushq %rax /* pt_regs->ax */ | ||
| 1414 | pushq %r8 /* pt_regs->r8 */ | ||
| 1415 | pushq %r9 /* pt_regs->r9 */ | ||
| 1416 | pushq %r10 /* pt_regs->r10 */ | ||
| 1417 | pushq %r11 /* pt_regs->r11 */ | ||
| 1418 | pushq %rbx /* pt_regs->rbx */ | ||
| 1419 | pushq %rbp /* pt_regs->rbp */ | ||
| 1420 | pushq %r12 /* pt_regs->r12 */ | ||
| 1421 | pushq %r13 /* pt_regs->r13 */ | ||
| 1422 | pushq %r14 /* pt_regs->r14 */ | ||
| 1423 | pushq %r15 /* pt_regs->r15 */ | ||
| 1424 | UNWIND_HINT_REGS | ||
| 1425 | ENCODE_FRAME_POINTER | 1404 | ENCODE_FRAME_POINTER |
| 1426 | 1405 | ||
| 1427 | /* | 1406 | /* |
| @@ -1631,7 +1610,6 @@ end_repeat_nmi: · 203 unmodified lines | |||
| 1631 | * frame to point back to repeat_nmi. | 1610 | * frame to point back to repeat_nmi. |
| 1632 | */ | 1611 | */ |
| 1633 | pushq $-1 /* ORIG_RAX: no syscall to restart */ | 1612 | pushq $-1 /* ORIG_RAX: no syscall to restart */ |
| 1634 | ALLOC_PT_GPREGS_ON_STACK | ||
| 1635 | 1613 | ||
| 1636 | /* | 1614 | /* |
| 1637 | * Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit | 1615 | * Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit |
| @@ -1655,8 +1633,7 @@ end_repeat_nmi: · 17 unmodified lines | |||
| 1655 | nmi_swapgs: | 1633 | nmi_swapgs: |
| 1656 | SWAPGS_UNSAFE_STACK | 1634 | SWAPGS_UNSAFE_STACK |
| 1657 | nmi_restore: | 1635 | nmi_restore: |
| 1658 | POP_EXTRA_REGS | 1636 | POP_REGS |
| 1659 | POP_C_REGS | ||
| 1660 | 1637 | ||
| 1661 | /* | 1638 | /* |
| 1662 | * Skip orig_ax and the "outermost" frame to point RSP at the "iret" | 1639 | * Skip orig_ax and the "outermost" frame to point RSP at the "iret" |
| @@ -85,15 +85,25 @@ ENTRY(entry_SYSENTER_compat) · 84 unmodified lines | |||
| 85 | pushq %rcx /* pt_regs->cx */ | 85 | pushq %rcx /* pt_regs->cx */ |
| 86 | pushq $-ENOSYS /* pt_regs->ax */ | 86 | pushq $-ENOSYS /* pt_regs->ax */ |
| 87 | pushq $0 /* pt_regs->r8 = 0 */ | 87 | pushq $0 /* pt_regs->r8 = 0 */ |
| 88 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 88 | pushq $0 /* pt_regs->r9 = 0 */ | 89 | pushq $0 /* pt_regs->r9 = 0 */ |
| 90 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 89 | pushq $0 /* pt_regs->r10 = 0 */ | 91 | pushq $0 /* pt_regs->r10 = 0 */ |
| 92 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 90 | pushq $0 /* pt_regs->r11 = 0 */ | 93 | pushq $0 /* pt_regs->r11 = 0 */ |
| 94 | xorl %r11d, %r11d /* nospec r11 */ | ||
| 91 | pushq %rbx /* pt_regs->rbx */ | 95 | pushq %rbx /* pt_regs->rbx */ |
| 96 | xorl %ebx, %ebx /* nospec rbx */ | ||
| 92 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ | 97 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ |
| 98 | xorl %ebp, %ebp /* nospec rbp */ | ||
| 93 | pushq $0 /* pt_regs->r12 = 0 */ | 99 | pushq $0 /* pt_regs->r12 = 0 */ |
| 100 | xorl %r12d, %r12d /* nospec r12 */ | ||
| 94 | pushq $0 /* pt_regs->r13 = 0 */ | 101 | pushq $0 /* pt_regs->r13 = 0 */ |
| 102 | xorl %r13d, %r13d /* nospec r13 */ | ||
| 95 | pushq $0 /* pt_regs->r14 = 0 */ | 103 | pushq $0 /* pt_regs->r14 = 0 */ |
| 104 | xorl %r14d, %r14d /* nospec r14 */ | ||
| 96 | pushq $0 /* pt_regs->r15 = 0 */ | 105 | pushq $0 /* pt_regs->r15 = 0 */ |
| 106 | xorl %r15d, %r15d /* nospec r15 */ | ||
| 97 | cld | 107 | cld |
| 98 | 108 | ||
| 99 | /* | 109 | /* |
| @@ -214,15 +224,25 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe) · 114 unmodified lines | |||
| 214 | pushq %rbp /* pt_regs->cx (stashed in bp) */ | 224 | pushq %rbp /* pt_regs->cx (stashed in bp) */ |
| 215 | pushq $-ENOSYS /* pt_regs->ax */ | 225 | pushq $-ENOSYS /* pt_regs->ax */ |
| 216 | pushq $0 /* pt_regs->r8 = 0 */ | 226 | pushq $0 /* pt_regs->r8 = 0 */ |
| 227 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 217 | pushq $0 /* pt_regs->r9 = 0 */ | 228 | pushq $0 /* pt_regs->r9 = 0 */ |
| 229 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 218 | pushq $0 /* pt_regs->r10 = 0 */ | 230 | pushq $0 /* pt_regs->r10 = 0 */ |
| 231 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 219 | pushq $0 /* pt_regs->r11 = 0 */ | 232 | pushq $0 /* pt_regs->r11 = 0 */ |
| 233 | xorl %r11d, %r11d /* nospec r11 */ | ||
| 220 | pushq %rbx /* pt_regs->rbx */ | 234 | pushq %rbx /* pt_regs->rbx */ |
| 235 | xorl %ebx, %ebx /* nospec rbx */ | ||
| 221 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ | 236 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ |
| 237 | xorl %ebp, %ebp /* nospec rbp */ | ||
| 222 | pushq $0 /* pt_regs->r12 = 0 */ | 238 | pushq $0 /* pt_regs->r12 = 0 */ |
| 239 | xorl %r12d, %r12d /* nospec r12 */ | ||
| 223 | pushq $0 /* pt_regs->r13 = 0 */ | 240 | pushq $0 /* pt_regs->r13 = 0 */ |
| 241 | xorl %r13d, %r13d /* nospec r13 */ | ||
| 224 | pushq $0 /* pt_regs->r14 = 0 */ | 242 | pushq $0 /* pt_regs->r14 = 0 */ |
| 243 | xorl %r14d, %r14d /* nospec r14 */ | ||
| 225 | pushq $0 /* pt_regs->r15 = 0 */ | 244 | pushq $0 /* pt_regs->r15 = 0 */ |
| 245 | xorl %r15d, %r15d /* nospec r15 */ | ||
| 226 | 246 | ||
| 227 | /* | 247 | /* |
| 228 | * User mode is traced as though IRQs are on, and SYSENTER | 248 | * User mode is traced as though IRQs are on, and SYSENTER |
| @@ -278,9 +298,9 @@ sysret32_from_system_call: · 49 unmodified lines | |||
| 278 | */ | 298 | */ |
| 279 | SWITCH_TO_USER_CR3_NOSTACK scratch_reg=%r8 scratch_reg2=%r9 | 299 | SWITCH_TO_USER_CR3_NOSTACK scratch_reg=%r8 scratch_reg2=%r9 |
| 280 | 300 | ||
| 281 | xorq %r8, %r8 | 301 | xorl %r8d, %r8d |
| 282 | xorq %r9, %r9 | 302 | xorl %r9d, %r9d |
| 283 | xorq %r10, %r10 | 303 | xorl %r10d, %r10d |
| 284 | swapgs | 304 | swapgs |
| 285 | sysretl | 305 | sysretl |
| 286 | END(entry_SYSCALL_compat) | 306 | END(entry_SYSCALL_compat) |
| @@ -327,26 +347,47 @@ ENTRY(entry_INT80_compat) · 40 unmodified lines | |||
| 327 | */ | 347 | */ |
| 328 | movl %eax, %eax | 348 | movl %eax, %eax |
| 329 | 349 | ||
| 350 | /* switch to thread stack expects orig_ax and rdi to be pushed */ | ||
| 330 | pushq %rax /* pt_regs->orig_ax */ | 351 | pushq %rax /* pt_regs->orig_ax */ |
| 352 | pushq %rdi /* pt_regs->di */ | ||
| 331 | 353 | ||
| 332 | /* switch to thread stack expects orig_ax to be pushed */ | 354 | /* Need to switch before accessing the thread stack. */ |
| 333 | call switch_to_thread_stack | 355 | SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi |
| 356 | movq %rsp, %rdi | ||
| 357 | movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp | ||
| 334 | 358 | ||
| 335 | pushq %rdi /* pt_regs->di */ | 359 | pushq 6*8(%rdi) /* regs->ss */ |
| 360 | pushq 5*8(%rdi) /* regs->rsp */ | ||
| 361 | pushq 4*8(%rdi) /* regs->eflags */ | ||
| 362 | pushq 3*8(%rdi) /* regs->cs */ | ||
| 363 | pushq 2*8(%rdi) /* regs->ip */ | ||
| 364 | pushq 1*8(%rdi) /* regs->orig_ax */ | ||
| 365 | |||
| 366 | pushq (%rdi) /* pt_regs->di */ | ||
| 336 | pushq %rsi /* pt_regs->si */ | 367 | pushq %rsi /* pt_regs->si */ |
| 337 | pushq %rdx /* pt_regs->dx */ | 368 | pushq %rdx /* pt_regs->dx */ |
| 338 | pushq %rcx /* pt_regs->cx */ | 369 | pushq %rcx /* pt_regs->cx */ |
| 339 | pushq $-ENOSYS /* pt_regs->ax */ | 370 | pushq $-ENOSYS /* pt_regs->ax */ |
| 340 | pushq $0 /* pt_regs->r8 = 0 */ | 371 | pushq $0 /* pt_regs->r8 = 0 */ |
| 372 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 341 | pushq $0 /* pt_regs->r9 = 0 */ | 373 | pushq $0 /* pt_regs->r9 = 0 */ |
| 374 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 342 | pushq $0 /* pt_regs->r10 = 0 */ | 375 | pushq $0 /* pt_regs->r10 = 0 */ |
| 376 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 343 | pushq $0 /* pt_regs->r11 = 0 */ | 377 | pushq $0 /* pt_regs->r11 = 0 */ |
| 378 | xorl %r11d, %r11d /* nospec r11 */ | ||
| 344 | pushq %rbx /* pt_regs->rbx */ | 379 | pushq %rbx /* pt_regs->rbx */ |
| 380 | xorl %ebx, %ebx /* nospec rbx */ | ||
| 345 | pushq %rbp /* pt_regs->rbp */ | 381 | pushq %rbp /* pt_regs->rbp */ |
| 382 | xorl %ebp, %ebp /* nospec rbp */ | ||
| 346 | pushq %r12 /* pt_regs->r12 */ | 383 | pushq %r12 /* pt_regs->r12 */ |
| 384 | xorl %r12d, %r12d /* nospec r12 */ | ||
| 347 | pushq %r13 /* pt_regs->r13 */ | 385 | pushq %r13 /* pt_regs->r13 */ |
| 386 | xorl %r13d, %r13d /* nospec r13 */ | ||
| 348 | pushq %r14 /* pt_regs->r14 */ | 387 | pushq %r14 /* pt_regs->r14 */ |
| 388 | xorl %r14d, %r14d /* nospec r14 */ | ||
| 349 | pushq %r15 /* pt_regs->r15 */ | 389 | pushq %r15 /* pt_regs->r15 */ |
| 390 | xorl %r15d, %r15d /* nospec r15 */ | ||
| 350 | cld | 391 | cld |
| 351 | 392 | ||
| 352 | /* | 393 | /* |
| @@ -363,15 +404,3 @@ ENTRY(entry_INT80_compat) · 10 unmodified lines | |||
| 363 | TRACE_IRQS_ON | 404 | TRACE_IRQS_ON |
| 364 | jmp swapgs_restore_regs_and_return_to_usermode | 405 | jmp swapgs_restore_regs_and_return_to_usermode |
| 365 | END(entry_INT80_compat) | 406 | END(entry_INT80_compat) |
| 366 | |||
| 367 | ENTRY(stub32_clone) | ||
| 368 | /* | ||
| 369 | * The 32-bit clone ABI is: clone(..., int tls_val, int *child_tidptr). | ||
| 370 | * The 64-bit clone ABI is: clone(..., int *child_tidptr, int tls_val). | ||
| 371 | * | ||
| 372 | * The native 64-bit kernel's sys_clone() implements the latter, | ||
| 373 | * so we need to swap arguments here before calling it: | ||
| 374 | */ | ||
| 375 | xchg %r8, %rcx | ||
| 376 | jmp sys_clone | ||
| 377 | ENDPROC(stub32_clone) | ||
| @@ -8,12 +8,12 @@ · 7 unmodified lines | |||
| 8 | # | 8 | # |
| 9 | 0 i386 restart_syscall sys_restart_syscall | 9 | 0 i386 restart_syscall sys_restart_syscall |
| 10 | 1 i386 exit sys_exit | 10 | 1 i386 exit sys_exit |
| 11 | 2 i386 fork sys_fork sys_fork | 11 | 2 i386 fork sys_fork |
| 12 | 3 i386 read sys_read | 12 | 3 i386 read sys_read |
| 13 | 4 i386 write sys_write | 13 | 4 i386 write sys_write |
| 14 | 5 i386 open sys_open compat_sys_open | 14 | 5 i386 open sys_open compat_sys_open |
| 15 | 6 i386 close sys_close | 15 | 6 i386 close sys_close |
| 16 | 7 i386 waitpid sys_waitpid sys32_waitpid | 16 | 7 i386 waitpid sys_waitpid compat_sys_x86_waitpid |
| 17 | 8 i386 creat sys_creat | 17 | 8 i386 creat sys_creat |
| 18 | 9 i386 link sys_link | 18 | 9 i386 link sys_link |
| 19 | 10 i386 unlink sys_unlink | 19 | 10 i386 unlink sys_unlink |
| @@ -78,7 +78,7 @@ · 58 unmodified lines | |||
| 78 | 69 i386 ssetmask sys_ssetmask | 78 | 69 i386 ssetmask sys_ssetmask |
| 79 | 70 i386 setreuid sys_setreuid16 | 79 | 70 i386 setreuid sys_setreuid16 |
| 80 | 71 i386 setregid sys_setregid16 | 80 | 71 i386 setregid sys_setregid16 |
| 81 | 72 i386 sigsuspend sys_sigsuspend sys_sigsuspend | 81 | 72 i386 sigsuspend sys_sigsuspend |
| 82 | 73 i386 sigpending sys_sigpending compat_sys_sigpending | 82 | 73 i386 sigpending sys_sigpending compat_sys_sigpending |
| 83 | 74 i386 sethostname sys_sethostname | 83 | 74 i386 sethostname sys_sethostname |
| 84 | 75 i386 setrlimit sys_setrlimit compat_sys_setrlimit | 84 | 75 i386 setrlimit sys_setrlimit compat_sys_setrlimit |
| @@ -96,7 +96,7 @@ · 11 unmodified lines | |||
| 96 | 87 i386 swapon sys_swapon | 96 | 87 i386 swapon sys_swapon |
| 97 | 88 i386 reboot sys_reboot | 97 | 88 i386 reboot sys_reboot |
| 98 | 89 i386 readdir sys_old_readdir compat_sys_old_readdir | 98 | 89 i386 readdir sys_old_readdir compat_sys_old_readdir |
| 99 | 90 i386 mmap sys_old_mmap sys32_mmap | 99 | 90 i386 mmap sys_old_mmap compat_sys_x86_mmap |
| 100 | 91 i386 munmap sys_munmap | 100 | 91 i386 munmap sys_munmap |
| 101 | 92 i386 truncate sys_truncate compat_sys_truncate | 101 | 92 i386 truncate sys_truncate compat_sys_truncate |
| 102 | 93 i386 ftruncate sys_ftruncate compat_sys_ftruncate | 102 | 93 i386 ftruncate sys_ftruncate compat_sys_ftruncate |
| @@ -126,7 +126,7 @@ · 23 unmodified lines | |||
| 126 | 117 i386 ipc sys_ipc compat_sys_ipc | 126 | 117 i386 ipc sys_ipc compat_sys_ipc |
| 127 | 118 i386 fsync sys_fsync | 127 | 118 i386 fsync sys_fsync |
| 128 | 119 i386 sigreturn sys_sigreturn sys32_sigreturn | 128 | 119 i386 sigreturn sys_sigreturn sys32_sigreturn |
| 129 | 120 i386 clone sys_clone stub32_clone | 129 | 120 i386 clone sys_clone compat_sys_x86_clone |
| 130 | 121 i386 setdomainname sys_setdomainname | 130 | 121 i386 setdomainname sys_setdomainname |
| 131 | 122 i386 uname sys_newuname | 131 | 122 i386 uname sys_newuname |
| 132 | 123 i386 modify_ldt sys_modify_ldt | 132 | 123 i386 modify_ldt sys_modify_ldt |
| @@ -186,8 +186,8 @@ · 53 unmodified lines | |||
| 186 | 177 i386 rt_sigtimedwait sys_rt_sigtimedwait compat_sys_rt_sigtimedwait | 186 | 177 i386 rt_sigtimedwait sys_rt_sigtimedwait compat_sys_rt_sigtimedwait |
| 187 | 178 i386 rt_sigqueueinfo sys_rt_sigqueueinfo compat_sys_rt_sigqueueinfo | 187 | 178 i386 rt_sigqueueinfo sys_rt_sigqueueinfo compat_sys_rt_sigqueueinfo |
| 188 | 179 i386 rt_sigsuspend sys_rt_sigsuspend | 188 | 179 i386 rt_sigsuspend sys_rt_sigsuspend |
| 189 | 180 i386 pread64 sys_pread64 sys32_pread | 189 | 180 i386 pread64 sys_pread64 compat_sys_x86_pread |
| 190 | 181 i386 pwrite64 sys_pwrite64 sys32_pwrite | 190 | 181 i386 pwrite64 sys_pwrite64 compat_sys_x86_pwrite |
| 191 | 182 i386 chown sys_chown16 | 191 | 182 i386 chown sys_chown16 |
| 192 | 183 i386 getcwd sys_getcwd | 192 | 183 i386 getcwd sys_getcwd |
| 193 | 184 i386 capget sys_capget | 193 | 184 i386 capget sys_capget |
| @@ -196,14 +196,14 @@ · 2 unmodified lines | |||
| 196 | 187 i386 sendfile sys_sendfile compat_sys_sendfile | 196 | 187 i386 sendfile sys_sendfile compat_sys_sendfile |
| 197 | 188 i386 getpmsg | 197 | 188 i386 getpmsg |
| 198 | 189 i386 putpmsg | 198 | 189 i386 putpmsg |
| 199 | 190 i386 vfork sys_vfork sys_vfork | 199 | 190 i386 vfork sys_vfork |
| 200 | 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit | 200 | 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit |
| 201 | 192 i386 mmap2 sys_mmap_pgoff | 201 | 192 i386 mmap2 sys_mmap_pgoff |
| 202 | 193 i386 truncate64 sys_truncate64 sys32_truncate64 | 202 | 193 i386 truncate64 sys_truncate64 compat_sys_x86_truncate64 |
| 203 | 194 i386 ftruncate64 sys_ftruncate64 sys32_ftruncate64 | 203 | 194 i386 ftruncate64 sys_ftruncate64 compat_sys_x86_ftruncate64 |
| 204 | 195 i386 stat64 sys_stat64 sys32_stat64 | 204 | 195 i386 stat64 sys_stat64 compat_sys_x86_stat64 |
| 205 | 196 i386 lstat64 sys_lstat64 sys32_lstat64 | 205 | 196 i386 lstat64 sys_lstat64 compat_sys_x86_lstat64 |
| 206 | 197 i386 fstat64 sys_fstat64 sys32_fstat64 | 206 | 197 i386 fstat64 sys_fstat64 compat_sys_x86_fstat64 |
| 207 | 198 i386 lchown32 sys_lchown | 207 | 198 i386 lchown32 sys_lchown |
| 208 | 199 i386 getuid32 sys_getuid | 208 | 199 i386 getuid32 sys_getuid |
| 209 | 200 i386 getgid32 sys_getgid | 209 | 200 i386 getgid32 sys_getgid |
| @@ -231,7 +231,7 @@ · 21 unmodified lines | |||
| 231 | # 222 is unused | 231 | # 222 is unused |
| 232 | # 223 is unused | 232 | # 223 is unused |
| 233 | 224 i386 gettid sys_gettid | 233 | 224 i386 gettid sys_gettid |
| 234 | 225 i386 readahead sys_readahead sys32_readahead | 234 | 225 i386 readahead sys_readahead compat_sys_x86_readahead |
| 235 | 226 i386 setxattr sys_setxattr | 235 | 226 i386 setxattr sys_setxattr |
| 236 | 227 i386 lsetxattr sys_lsetxattr | 236 | 227 i386 lsetxattr sys_lsetxattr |
| 237 | 228 i386 fsetxattr sys_fsetxattr | 237 | 228 i386 fsetxattr sys_fsetxattr |
| @@ -256,7 +256,7 @@ · 18 unmodified lines | |||
| 256 | 247 i386 io_getevents sys_io_getevents compat_sys_io_getevents | 256 | 247 i386 io_getevents sys_io_getevents compat_sys_io_getevents |
| 257 | 248 i386 io_submit sys_io_submit compat_sys_io_submit | 257 | 248 i386 io_submit sys_io_submit compat_sys_io_submit |
| 258 | 249 i386 io_cancel sys_io_cancel | 258 | 249 i386 io_cancel sys_io_cancel |
| 259 | 250 i386 fadvise64 sys_fadvise64 sys32_fadvise64 | 259 | 250 i386 fadvise64 sys_fadvise64 compat_sys_x86_fadvise64 |
| 260 | # 251 is available for reuse (was briefly sys_set_zone_reclaim) | 260 | # 251 is available for reuse (was briefly sys_set_zone_reclaim) |
| 261 | 252 i386 exit_group sys_exit_group | 261 | 252 i386 exit_group sys_exit_group |
| 262 | 253 i386 lookup_dcookie sys_lookup_dcookie compat_sys_lookup_dcookie | 262 | 253 i386 lookup_dcookie sys_lookup_dcookie compat_sys_lookup_dcookie |
| @@ -278,7 +278,7 @@ · 15 unmodified lines | |||
| 278 | 269 i386 fstatfs64 sys_fstatfs64 compat_sys_fstatfs64 | 278 | 269 i386 fstatfs64 sys_fstatfs64 compat_sys_fstatfs64 |
| 279 | 270 i386 tgkill sys_tgkill | 279 | 270 i386 tgkill sys_tgkill |
| 280 | 271 i386 utimes sys_utimes compat_sys_utimes | 280 | 271 i386 utimes sys_utimes compat_sys_utimes |
| 281 | 272 i386 fadvise64_64 sys_fadvise64_64 sys32_fadvise64_64 | 281 | 272 i386 fadvise64_64 sys_fadvise64_64 compat_sys_x86_fadvise64_64 |
| 282 | 273 i386 vserver | 282 | 273 i386 vserver |
| 283 | 274 i386 mbind sys_mbind | 283 | 274 i386 mbind sys_mbind |
| 284 | 275 i386 get_mempolicy sys_get_mempolicy compat_sys_get_mempolicy | 284 | 275 i386 get_mempolicy sys_get_mempolicy compat_sys_get_mempolicy |
| @@ -306,7 +306,7 @@ · 21 unmodified lines | |||
| 306 | 297 i386 mknodat sys_mknodat | 306 | 297 i386 mknodat sys_mknodat |
| 307 | 298 i386 fchownat sys_fchownat | 307 | 298 i386 fchownat sys_fchownat |
| 308 | 299 i386 futimesat sys_futimesat compat_sys_futimesat | 308 | 299 i386 futimesat sys_futimesat compat_sys_futimesat |
| 309 | 300 i386 fstatat64 sys_fstatat64 sys32_fstatat | 309 | 300 i386 fstatat64 sys_fstatat64 compat_sys_x86_fstatat |
| 310 | 301 i386 unlinkat sys_unlinkat | 310 | 301 i386 unlinkat sys_unlinkat |
| 311 | 302 i386 renameat sys_renameat | 311 | 302 i386 renameat sys_renameat |
| 312 | 303 i386 linkat sys_linkat | 312 | 303 i386 linkat sys_linkat |
| @@ -320,7 +320,7 @@ · 7 unmodified lines | |||
| 320 | 311 i386 set_robust_list sys_set_robust_list compat_sys_set_robust_list | 320 | 311 i386 set_robust_list sys_set_robust_list compat_sys_set_robust_list |
| 321 | 312 i386 get_robust_list sys_get_robust_list compat_sys_get_robust_list | 321 | 312 i386 get_robust_list sys_get_robust_list compat_sys_get_robust_list |
| 322 | 313 i386 splice sys_splice | 322 | 313 i386 splice sys_splice |
| 323 | 314 i386 sync_file_range sys_sync_file_range sys32_sync_file_range | 323 | 314 i386 sync_file_range sys_sync_file_range compat_sys_x86_sync_file_range |
| 324 | 315 i386 tee sys_tee | 324 | 315 i386 tee sys_tee |
| 325 | 316 i386 vmsplice sys_vmsplice compat_sys_vmsplice | 325 | 316 i386 vmsplice sys_vmsplice compat_sys_vmsplice |
| 326 | 317 i386 move_pages sys_move_pages compat_sys_move_pages | 326 | 317 i386 move_pages sys_move_pages compat_sys_move_pages |
| @@ -330,7 +330,7 @@ · 3 unmodified lines | |||
| 330 | 321 i386 signalfd sys_signalfd compat_sys_signalfd | 330 | 321 i386 signalfd sys_signalfd compat_sys_signalfd |
| 331 | 322 i386 timerfd_create sys_timerfd_create | 331 | 322 i386 timerfd_create sys_timerfd_create |
| 332 | 323 i386 eventfd sys_eventfd | 332 | 323 i386 eventfd sys_eventfd |
| 333 | 324 i386 fallocate sys_fallocate sys32_fallocate | 333 | 324 i386 fallocate sys_fallocate compat_sys_x86_fallocate |
| 334 | 325 i386 timerfd_settime sys_timerfd_settime compat_sys_timerfd_settime | 334 | 325 i386 timerfd_settime sys_timerfd_settime compat_sys_timerfd_settime |
| 335 | 326 i386 timerfd_gettime sys_timerfd_gettime compat_sys_timerfd_gettime | 335 | 326 i386 timerfd_gettime sys_timerfd_gettime compat_sys_timerfd_gettime |
| 336 | 327 i386 signalfd4 sys_signalfd4 compat_sys_signalfd4 | 336 | 327 i386 signalfd4 sys_signalfd4 compat_sys_signalfd4 |
| @@ -5,8 +5,6 @@ · 4 unmodified lines | |||
| 5 | #undef CONFIG_OPTIMIZE_INLINING | 5 | #undef CONFIG_OPTIMIZE_INLINING |
| 6 | #endif | 6 | #endif |
| 7 | 7 | ||
| 8 | #undef CONFIG_X86_PPRO_FENCE | ||
| 9 | |||
| 10 | #ifdef CONFIG_X86_64 | 8 | #ifdef CONFIG_X86_64 |
| 11 | 9 | ||
| 12 | /* | 10 | /* |
| @@ -42,10 +42,8 @@ · 41 unmodified lines | |||
| 42 | #define CREATE_TRACE_POINTS | 42 | #define CREATE_TRACE_POINTS |
| 43 | #include "vsyscall_trace.h" | 43 | #include "vsyscall_trace.h" |
| 44 | 44 | ||
| 45 | static enum { EMULATE, NATIVE, NONE } vsyscall_mode = | 45 | static enum { EMULATE, NONE } vsyscall_mode = |
| 46 | #if defined(CONFIG_LEGACY_VSYSCALL_NATIVE) | 46 | #ifdef CONFIG_LEGACY_VSYSCALL_NONE |
| 47 | NATIVE; | ||
| 48 | #elif defined(CONFIG_LEGACY_VSYSCALL_NONE) | ||
| 49 | NONE; | 47 | NONE; |
| 50 | #else | 48 | #else |
| 51 | EMULATE; | 49 | EMULATE; |
| @@ -56,8 +54,6 @@ static int __init vsyscall_setup(char *str) · 4 unmodified lines | |||
| 56 | if (str) { | 54 | if (str) { |
| 57 | if (!strcmp("emulate", str)) | 55 | if (!strcmp("emulate", str)) |
| 58 | vsyscall_mode = EMULATE; | 56 | vsyscall_mode = EMULATE; |
| 59 | else if (!strcmp("native", str)) | ||
| 60 | vsyscall_mode = NATIVE; | ||
| 61 | else if (!strcmp("none", str)) | 57 | else if (!strcmp("none", str)) |
| 62 | vsyscall_mode = NONE; | 58 | vsyscall_mode = NONE; |
| 63 | else | 59 | else |
| @@ -139,10 +135,6 @@ bool emulate_vsyscall(struct pt_regs *regs, unsigned long address) · 75 unmodified lines | |||
| 139 | 135 | ||
| 140 | WARN_ON_ONCE(address != regs->ip); | 136 | WARN_ON_ONCE(address != regs->ip); |
| 141 | 137 | ||
| 142 | /* This should be unreachable in NATIVE mode. */ | ||
| 143 | if (WARN_ON(vsyscall_mode == NATIVE)) | ||
| 144 | return false; | ||
| 145 | |||
| 146 | if (vsyscall_mode == NONE) { | 138 | if (vsyscall_mode == NONE) { |
| 147 | warn_bad_vsyscall(KERN_INFO, regs, | 139 | warn_bad_vsyscall(KERN_INFO, regs, |
| 148 | "vsyscall attempted with vsyscall=none"); | 140 | "vsyscall attempted with vsyscall=none"); |
| @@ -355,7 +347,7 @@ void __init set_vsyscall_pgtable_user_bits(pgd_t *root) · 206 unmodified lines | |||
| 355 | set_pgd(pgd, __pgd(pgd_val(*pgd) | _PAGE_USER)); | 347 | set_pgd(pgd, __pgd(pgd_val(*pgd) | _PAGE_USER)); |
| 356 | p4d = p4d_offset(pgd, VSYSCALL_ADDR); | 348 | p4d = p4d_offset(pgd, VSYSCALL_ADDR); |
| 357 | #if CONFIG_PGTABLE_LEVELS >= 5 | 349 | #if CONFIG_PGTABLE_LEVELS >= 5 |
| 358 | p4d->p4d |= _PAGE_USER; | 350 | set_p4d(p4d, __p4d(p4d_val(*p4d) | _PAGE_USER)); |
| 359 | #endif | 351 | #endif |
| 360 | pud = pud_offset(p4d, VSYSCALL_ADDR); | 352 | pud = pud_offset(p4d, VSYSCALL_ADDR); |
| 361 | set_pud(pud, __pud(pud_val(*pud) | _PAGE_USER)); | 353 | set_pud(pud, __pud(pud_val(*pud) | _PAGE_USER)); |
| @@ -370,9 +362,7 @@ void __init map_vsyscall(void) · 8 unmodified lines | |||
| 370 | 362 | ||
| 371 | if (vsyscall_mode != NONE) { | 363 | if (vsyscall_mode != NONE) { |
| 372 | __set_fixmap(VSYSCALL_PAGE, physaddr_vsyscall, | 364 | __set_fixmap(VSYSCALL_PAGE, physaddr_vsyscall, |
| 373 | vsyscall_mode == NATIVE | 365 | PAGE_KERNEL_VVAR); |
| 374 | ? PAGE_KERNEL_VSYSCALL | ||
| 375 | : PAGE_KERNEL_VVAR); | ||
| 376 | set_vsyscall_pgtable_user_bits(swapper_pg_dir); | 366 | set_vsyscall_pgtable_user_bits(swapper_pg_dir); |
| 377 | } | 367 | } |
| 378 | 368 | ||
| @@ -2118,7 +2118,8 @@ static int x86_pmu_event_init(struct perf_event *event) · 2117 unmodified lines | |||
| 2118 | event->destroy(event); | 2118 | event->destroy(event); |
| 2119 | } | 2119 | } |
| 2120 | 2120 | ||
| 2121 | if (READ_ONCE(x86_pmu.attr_rdpmc)) | 2121 | if (READ_ONCE(x86_pmu.attr_rdpmc) && |
| 2122 | !(event->hw.flags & PERF_X86_EVENT_LARGE_PEBS)) | ||
| 2122 | event->hw.flags |= PERF_X86_EVENT_RDPMC_ALLOWED; | 2123 | event->hw.flags |= PERF_X86_EVENT_RDPMC_ALLOWED; |
| 2123 | 2124 | ||
| 2124 | return err; | 2125 | return err; |
| @@ -2952,9 +2952,9 @@ static void intel_pebs_aliases_skl(struct perf_event *event) · 2951 unmodified lines | |||
| 2952 | return intel_pebs_aliases_precdist(event); | 2952 | return intel_pebs_aliases_precdist(event); |
| 2953 | } | 2953 | } |
| 2954 | 2954 | ||
| 2955 | static unsigned long intel_pmu_free_running_flags(struct perf_event *event) | 2955 | static unsigned long intel_pmu_large_pebs_flags(struct perf_event *event) |
| 2956 | { | 2956 | { |
| 2957 | unsigned long flags = x86_pmu.free_running_flags; | 2957 | unsigned long flags = x86_pmu.large_pebs_flags; |
| 2958 | 2958 | ||
| 2959 | if (event->attr.use_clockid) | 2959 | if (event->attr.use_clockid) |
| 2960 | flags &= ~PERF_SAMPLE_TIME; | 2960 | flags &= ~PERF_SAMPLE_TIME; |
| @@ -2976,8 +2976,8 @@ static int intel_pmu_hw_config(struct perf_event *event) · 15 unmodified lines | |||
| 2976 | if (!event->attr.freq) { | 2976 | if (!event->attr.freq) { |
| 2977 | event->hw.flags |= PERF_X86_EVENT_AUTO_RELOAD; | 2977 | event->hw.flags |= PERF_X86_EVENT_AUTO_RELOAD; |
| 2978 | if (!(event->attr.sample_type & | 2978 | if (!(event->attr.sample_type & |
| 2979 | ~intel_pmu_free_running_flags(event))) | 2979 | ~intel_pmu_large_pebs_flags(event))) |
| 2980 | event->hw.flags |= PERF_X86_EVENT_FREERUNNING; | 2980 | event->hw.flags |= PERF_X86_EVENT_LARGE_PEBS; |
| 2981 | } | 2981 | } |
| 2982 | if (x86_pmu.pebs_aliases) | 2982 | if (x86_pmu.pebs_aliases) |
| 2983 | x86_pmu.pebs_aliases(event); | 2983 | x86_pmu.pebs_aliases(event); |
| @@ -3194,7 +3194,7 @@ static unsigned bdw_limit_period(struct perf_event *event, unsigned left) · 210 unmodified lines | |||
| 3194 | X86_CONFIG(.event=0xc0, .umask=0x01)) { | 3194 | X86_CONFIG(.event=0xc0, .umask=0x01)) { |
| 3195 | if (left < 128) | 3195 | if (left < 128) |
| 3196 | left = 128; | 3196 | left = 128; |
| 3197 | left &= ~0x3fu; | 3197 | left &= ~0x3fULL; |
| 3198 | } | 3198 | } |
| 3199 | return left; | 3199 | return left; |
| 3200 | } | 3200 | } |
| @@ -3460,7 +3460,7 @@ static __initconst const struct x86_pmu core_pmu = { · 259 unmodified lines | |||
| 3460 | .event_map = intel_pmu_event_map, | 3460 | .event_map = intel_pmu_event_map, |
| 3461 | .max_events = ARRAY_SIZE(intel_perfmon_event_map), | 3461 | .max_events = ARRAY_SIZE(intel_perfmon_event_map), |
| 3462 | .apic = 1, | 3462 | .apic = 1, |
| 3463 | .free_running_flags = PEBS_FREERUNNING_FLAGS, | 3463 | .large_pebs_flags = LARGE_PEBS_FLAGS, |
| 3464 | 3464 | ||
| 3465 | /* | 3465 | /* |
| 3466 | * Intel PMCs cannot be accessed sanely above 32-bit width, | 3466 | * Intel PMCs cannot be accessed sanely above 32-bit width, |
| @@ -3502,7 +3502,7 @@ static __initconst const struct x86_pmu intel_pmu = { · 35 unmodified lines | |||
| 3502 | .event_map = intel_pmu_event_map, | 3502 | .event_map = intel_pmu_event_map, |
| 3503 | .max_events = ARRAY_SIZE(intel_perfmon_event_map), | 3503 | .max_events = ARRAY_SIZE(intel_perfmon_event_map), |
| 3504 | .apic = 1, | 3504 | .apic = 1, |
| 3505 | .free_running_flags = PEBS_FREERUNNING_FLAGS, | 3505 | .large_pebs_flags = LARGE_PEBS_FLAGS, |
| 3506 | /* | 3506 | /* |
| 3507 | * Intel PMCs cannot be accessed sanely above 32 bit width, | 3507 | * Intel PMCs cannot be accessed sanely above 32 bit width, |
| 3508 | * so we install an artificial 1<<31 period regardless of | 3508 | * so we install an artificial 1<<31 period regardless of |
| @@ -3559,7 +3559,7 @@ static int intel_snb_pebs_broken(int cpu) · 50 unmodified lines | |||
| 3559 | break; | 3559 | break; |
| 3560 | 3560 | ||
| 3561 | case INTEL_FAM6_SANDYBRIDGE_X: | 3561 | case INTEL_FAM6_SANDYBRIDGE_X: |
| 3562 | switch (cpu_data(cpu).x86_mask) { | 3562 | switch (cpu_data(cpu).x86_stepping) { |
| 3563 | case 6: rev = 0x618; break; | 3563 | case 6: rev = 0x618; break; |
| 3564 | case 7: rev = 0x70c; break; | 3564 | case 7: rev = 0x70c; break; |
| 3565 | } | 3565 | } |
| @@ -935,7 +935,7 @@ void intel_pmu_pebs_add(struct perf_event *event) · 934 unmodified lines | |||
| 935 | bool needed_cb = pebs_needs_sched_cb(cpuc); | 935 | bool needed_cb = pebs_needs_sched_cb(cpuc); |
| 936 | 936 | ||
| 937 | cpuc->n_pebs++; | 937 | cpuc->n_pebs++; |
| 938 | if (hwc->flags & PERF_X86_EVENT_FREERUNNING) | 938 | if (hwc->flags & PERF_X86_EVENT_LARGE_PEBS) |
| 939 | cpuc->n_large_pebs++; | 939 | cpuc->n_large_pebs++; |
| 940 | 940 | ||
| 941 | pebs_update_state(needed_cb, cpuc, event->ctx->pmu); | 941 | pebs_update_state(needed_cb, cpuc, event->ctx->pmu); |
| @@ -975,7 +975,7 @@ void intel_pmu_pebs_del(struct perf_event *event) · 33 unmodified lines | |||
| 975 | bool needed_cb = pebs_needs_sched_cb(cpuc); | 975 | bool needed_cb = pebs_needs_sched_cb(cpuc); |
| 976 | 976 | ||
| 977 | cpuc->n_pebs--; | 977 | cpuc->n_pebs--; |
| 978 | if (hwc->flags & PERF_X86_EVENT_FREERUNNING) | 978 | if (hwc->flags & PERF_X86_EVENT_LARGE_PEBS) |
| 979 | cpuc->n_large_pebs--; | 979 | cpuc->n_large_pebs--; |
| 980 | 980 | ||
| 981 | pebs_update_state(needed_cb, cpuc, event->ctx->pmu); | 981 | pebs_update_state(needed_cb, cpuc, event->ctx->pmu); |
| @@ -1530,7 +1530,7 @@ void __init intel_ds_init(void) · 548 unmodified lines | |||
| 1530 | x86_pmu.pebs_record_size = | 1530 | x86_pmu.pebs_record_size = |
| 1531 | sizeof(struct pebs_record_skl); | 1531 | sizeof(struct pebs_record_skl); |
| 1532 | x86_pmu.drain_pebs = intel_pmu_drain_pebs_nhm; | 1532 | x86_pmu.drain_pebs = intel_pmu_drain_pebs_nhm; |
| 1533 | x86_pmu.free_running_flags |= PERF_SAMPLE_TIME; | 1533 | x86_pmu.large_pebs_flags |= PERF_SAMPLE_TIME; |
| 1534 | break; | 1534 | break; |
| 1535 | 1535 | ||
| 1536 | default: | 1536 | default: |
| @@ -1186,7 +1186,7 @@ void __init intel_pmu_lbr_init_atom(void) · 1185 unmodified lines | |||
| 1186 | * on PMU interrupt | 1186 | * on PMU interrupt |
| 1187 | */ | 1187 | */ |
| 1188 | if (boot_cpu_data.x86_model == 28 | 1188 | if (boot_cpu_data.x86_model == 28 |
| 1189 | && boot_cpu_data.x86_mask < 10) { | 1189 | && boot_cpu_data.x86_stepping < 10) { |
| 1190 | pr_cont("LBR disabled due to erratum"); | 1190 | pr_cont("LBR disabled due to erratum"); |
| 1191 | return; | 1191 | return; |
| 1192 | } | 1192 | } |
| @@ -234,7 +234,7 @@ static __initconst const struct x86_pmu p6_pmu = { · 233 unmodified lines | |||
| 234 | 234 | ||
| 235 | static __init void p6_pmu_rdpmc_quirk(void) | 235 | static __init void p6_pmu_rdpmc_quirk(void) |
| 236 | { | 236 | { |
| 237 | if (boot_cpu_data.x86_mask < 9) { | 237 | if (boot_cpu_data.x86_stepping < 9) { |
| 238 | /* | 238 | /* |
| 239 | * PPro erratum 26; fixed in stepping 9 and above. | 239 | * PPro erratum 26; fixed in stepping 9 and above. |
| 240 | */ | 240 | */ |
| @@ -3343,6 +3343,7 @@ static struct extra_reg skx_uncore_cha_extra_regs[] = { · 3342 unmodified lines | |||
| 3343 | SNBEP_CBO_EVENT_EXTRA_REG(0x9134, 0xffff, 0x4), | 3343 | SNBEP_CBO_EVENT_EXTRA_REG(0x9134, 0xffff, 0x4), |
| 3344 | SNBEP_CBO_EVENT_EXTRA_REG(0x35, 0xff, 0x8), | 3344 | SNBEP_CBO_EVENT_EXTRA_REG(0x35, 0xff, 0x8), |
| 3345 | SNBEP_CBO_EVENT_EXTRA_REG(0x36, 0xff, 0x8), | 3345 | SNBEP_CBO_EVENT_EXTRA_REG(0x36, 0xff, 0x8), |
| 3346 | SNBEP_CBO_EVENT_EXTRA_REG(0x38, 0xff, 0x3), | ||
| 3346 | EVENT_EXTRA_END | 3347 | EVENT_EXTRA_END |
| 3347 | }; | 3348 | }; |
| 3348 | 3349 | ||
| @@ -3562,24 +3563,27 @@ static struct intel_uncore_type *skx_msr_uncores[] = { · 213 unmodified lines | |||
| 3562 | NULL, | 3563 | NULL, |
| 3563 | }; | 3564 | }; |
| 3564 | 3565 | ||
| 3566 | /* | ||
| 3567 | * To determine the number of CHAs, it should read bits 27:0 in the CAPID6 | ||
| 3568 | * register which located at Device 30, Function 3, Offset 0x9C. PCI ID 0x2083. | ||
| 3569 | */ | ||
| 3570 | #define SKX_CAPID6 0x9c | ||
| 3571 | #define SKX_CHA_BIT_MASK GENMASK(27, 0) | ||
| 3572 | |||
| 3565 | static int skx_count_chabox(void) | 3573 | static int skx_count_chabox(void) |
| 3566 | { | 3574 | { |
| 3567 | struct pci_dev *chabox_dev = NULL; | 3575 | struct pci_dev *dev = NULL; |
| 3568 | int bus, count = 0; | 3576 | u32 val = 0; |
| 3569 | 3577 | ||
| 3570 | while (1) { | 3578 | dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x2083, dev); |
| 3571 | chabox_dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x208d, chabox_dev); | 3579 | if (!dev) |
| 3572 | if (!chabox_dev) | 3580 | goto out; |
| 3573 | break; | ||
| 3574 | if (count == 0) | ||
| 3575 | bus = chabox_dev->bus->number; | ||
| 3576 | if (bus != chabox_dev->bus->number) | ||
| 3577 | break; | ||
| 3578 | count++; | ||
| 3579 | } | ||
| 3580 | 3581 | ||
| 3581 | pci_dev_put(chabox_dev); | 3582 | pci_read_config_dword(dev, SKX_CAPID6, &val); |
| 3582 | return count; | 3583 | val &= SKX_CHA_BIT_MASK; |
| 3584 | out: | ||
| 3585 | pci_dev_put(dev); | ||
| 3586 | return hweight32(val); | ||
| 3583 | } | 3587 | } |
| 3584 | 3588 | ||
| 3585 | void skx_uncore_cpu_init(void) | 3589 | void skx_uncore_cpu_init(void) |
| @@ -3606,7 +3610,7 @@ static struct intel_uncore_type skx_uncore_imc = { · 20 unmodified lines | |||
| 3606 | }; | 3610 | }; |
| 3607 | 3611 | ||
| 3608 | static struct attribute *skx_upi_uncore_formats_attr[] = { | 3612 | static struct attribute *skx_upi_uncore_formats_attr[] = { |
| 3609 | &format_attr_event_ext.attr, | 3613 | &format_attr_event.attr, |
| 3610 | &format_attr_umask_ext.attr, | 3614 | &format_attr_umask_ext.attr, |
| 3611 | &format_attr_edge.attr, | 3615 | &format_attr_edge.attr, |
| 3612 | &format_attr_inv.attr, | 3616 | &format_attr_inv.attr, |
| @@ -69,7 +69,7 @@ struct event_constraint { · 68 unmodified lines | |||
| 69 | #define PERF_X86_EVENT_RDPMC_ALLOWED 0x0100 /* grant rdpmc permission */ | 69 | #define PERF_X86_EVENT_RDPMC_ALLOWED 0x0100 /* grant rdpmc permission */ |
| 70 | #define PERF_X86_EVENT_EXCL_ACCT 0x0200 /* accounted EXCL event */ | 70 | #define PERF_X86_EVENT_EXCL_ACCT 0x0200 /* accounted EXCL event */ |
| 71 | #define PERF_X86_EVENT_AUTO_RELOAD 0x0400 /* use PEBS auto-reload */ | 71 | #define PERF_X86_EVENT_AUTO_RELOAD 0x0400 /* use PEBS auto-reload */ |
| 72 | #define PERF_X86_EVENT_FREERUNNING 0x0800 /* use freerunning PEBS */ | 72 | #define PERF_X86_EVENT_LARGE_PEBS 0x0800 /* use large PEBS */ |
| 73 | 73 | ||
| 74 | 74 | ||
| 75 | struct amd_nb { | 75 | struct amd_nb { |
| @@ -88,7 +88,7 @@ struct amd_nb { · 12 unmodified lines | |||
| 88 | * REGS_USER can be handled for events limited to ring 3. | 88 | * REGS_USER can be handled for events limited to ring 3. |
| 89 | * | 89 | * |
| 90 | */ | 90 | */ |
| 91 | #define PEBS_FREERUNNING_FLAGS \ | 91 | #define LARGE_PEBS_FLAGS \ |
| 92 | (PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_ADDR | \ | 92 | (PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_ADDR | \ |
| 93 | PERF_SAMPLE_ID | PERF_SAMPLE_CPU | PERF_SAMPLE_STREAM_ID | \ | 93 | PERF_SAMPLE_ID | PERF_SAMPLE_CPU | PERF_SAMPLE_STREAM_ID | \ |
| 94 | PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | \ | 94 | PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | \ |
| @@ -608,7 +608,7 @@ struct x86_pmu { · 513 unmodified lines | |||
| 608 | struct event_constraint *pebs_constraints; | 608 | struct event_constraint *pebs_constraints; |
| 609 | void (*pebs_aliases)(struct perf_event *event); | 609 | void (*pebs_aliases)(struct perf_event *event); |
| 610 | int max_pebs_events; | 610 | int max_pebs_events; |
| 611 | unsigned long free_running_flags; | 611 | unsigned long large_pebs_flags; |
| 612 | 612 | ||
| 613 | /* | 613 | /* |
| 614 | * Intel LBR | 614 | * Intel LBR |
| @@ -51,15 +51,14 @@ · 50 unmodified lines | |||
| 51 | #define AA(__x) ((unsigned long)(__x)) | 51 | #define AA(__x) ((unsigned long)(__x)) |
| 52 | 52 | ||
| 53 | 53 | ||
| 54 | asmlinkage long sys32_truncate64(const char __user *filename, | 54 | COMPAT_SYSCALL_DEFINE3(x86_truncate64, const char __user *, filename, |
| 55 | unsigned long offset_low, | 55 | unsigned long, offset_low, unsigned long, offset_high) |
| 56 | unsigned long offset_high) | ||
| 57 | { | 56 | { |
| 58 | return sys_truncate(filename, ((loff_t) offset_high << 32) | offset_low); | 57 | return sys_truncate(filename, ((loff_t) offset_high << 32) | offset_low); |
| 59 | } | 58 | } |
| 60 | 59 | ||
| 61 | asmlinkage long sys32_ftruncate64(unsigned int fd, unsigned long offset_low, | 60 | COMPAT_SYSCALL_DEFINE3(x86_ftruncate64, unsigned int, fd, |
| 62 | unsigned long offset_high) | 61 | unsigned long, offset_low, unsigned long, offset_high) |
| 63 | { | 62 | { |
| 64 | return sys_ftruncate(fd, ((loff_t) offset_high << 32) | offset_low); | 63 | return sys_ftruncate(fd, ((loff_t) offset_high << 32) | offset_low); |
| 65 | } | 64 | } |
| @@ -96,8 +95,8 @@ static int cp_stat64(struct stat64 __user *ubuf, struct kstat *stat) · 30 unmodified lines | |||
| 96 | return 0; | 95 | return 0; |
| 97 | } | 96 | } |
| 98 | 97 | ||
| 99 | asmlinkage long sys32_stat64(const char __user *filename, | 98 | COMPAT_SYSCALL_DEFINE2(x86_stat64, const char __user *, filename, |
| 100 | struct stat64 __user *statbuf) | 99 | struct stat64 __user *, statbuf) |
| 101 | { | 100 | { |
| 102 | struct kstat stat; | 101 | struct kstat stat; |
| 103 | int ret = vfs_stat(filename, &stat); | 102 | int ret = vfs_stat(filename, &stat); |
| @@ -107,8 +106,8 @@ asmlinkage long sys32_stat64(const char __user *filename, · 3 unmodified lines | |||
| 107 | return ret; | 106 | return ret; |
| 108 | } | 107 | } |
| 109 | 108 | ||
| 110 | asmlinkage long sys32_lstat64(const char __user *filename, | 109 | COMPAT_SYSCALL_DEFINE2(x86_lstat64, const char __user *, filename, |
| 111 | struct stat64 __user *statbuf) | 110 | struct stat64 __user *, statbuf) |
| 112 | { | 111 | { |
| 113 | struct kstat stat; | 112 | struct kstat stat; |
| 114 | int ret = vfs_lstat(filename, &stat); | 113 | int ret = vfs_lstat(filename, &stat); |
| @@ -117,7 +116,8 @@ asmlinkage long sys32_lstat64(const char __user *filename, · 2 unmodified lines | |||
| 117 | return ret; | 116 | return ret; |
| 118 | } | 117 | } |
| 119 | 118 | ||
| 120 | asmlinkage long sys32_fstat64(unsigned int fd, struct stat64 __user *statbuf) | 119 | COMPAT_SYSCALL_DEFINE2(x86_fstat64, unsigned int, fd, |
| 120 | struct stat64 __user *, statbuf) | ||
| 121 | { | 121 | { |
| 122 | struct kstat stat; | 122 | struct kstat stat; |
| 123 | int ret = vfs_fstat(fd, &stat); | 123 | int ret = vfs_fstat(fd, &stat); |
| @@ -126,8 +126,9 @@ asmlinkage long sys32_fstat64(unsigned int fd, struct stat64 __user *statbuf) · 2 unmodified lines | |||
| 126 | return ret; | 126 | return ret; |
| 127 | } | 127 | } |
| 128 | 128 | ||
| 129 | asmlinkage long sys32_fstatat(unsigned int dfd, const char __user *filename, | 129 | COMPAT_SYSCALL_DEFINE4(x86_fstatat, unsigned int, dfd, |
| 130 | struct stat64 __user *statbuf, int flag) | 130 | const char __user *, filename, |
| 131 | struct stat64 __user *, statbuf, int, flag) | ||
| 131 | { | 132 | { |
| 132 | struct kstat stat; | 133 | struct kstat stat; |
| 133 | int error; | 134 | int error; |
| @@ -153,7 +154,7 @@ struct mmap_arg_struct32 { · 19 unmodified lines | |||
| 153 | unsigned int offset; | 154 | unsigned int offset; |
| 154 | }; | 155 | }; |
| 155 | 156 | ||
| 156 | asmlinkage long sys32_mmap(struct mmap_arg_struct32 __user *arg) | 157 | COMPAT_SYSCALL_DEFINE1(x86_mmap, struct mmap_arg_struct32 __user *, arg) |
| 157 | { | 158 | { |
| 158 | struct mmap_arg_struct32 a; | 159 | struct mmap_arg_struct32 a; |
| 159 | 160 | ||
| @@ -167,22 +168,22 @@ asmlinkage long sys32_mmap(struct mmap_arg_struct32 __user *arg) · 7 unmodified lines | |||
| 167 | a.offset>>PAGE_SHIFT); | 168 | a.offset>>PAGE_SHIFT); |
| 168 | } | 169 | } |
| 169 | 170 | ||
| 170 | asmlinkage long sys32_waitpid(compat_pid_t pid, unsigned int __user *stat_addr, | 171 | COMPAT_SYSCALL_DEFINE3(x86_waitpid, compat_pid_t, pid, unsigned int __user *, |
| 171 | int options) | 172 | stat_addr, int, options) |
| 172 | { | 173 | { |
| 173 | return compat_sys_wait4(pid, stat_addr, options, NULL); | 174 | return compat_sys_wait4(pid, stat_addr, options, NULL); |
| 174 | } | 175 | } |
| 175 | 176 | ||
| 176 | /* warning: next two assume little endian */ | 177 | /* warning: next two assume little endian */ |
| 177 | asmlinkage long sys32_pread(unsigned int fd, char __user *ubuf, u32 count, | 178 | COMPAT_SYSCALL_DEFINE5(x86_pread, unsigned int, fd, char __user *, ubuf, |
| 178 | u32 poslo, u32 poshi) | 179 | u32, count, u32, poslo, u32, poshi) |
| 179 | { | 180 | { |
| 180 | return sys_pread64(fd, ubuf, count, | 181 | return sys_pread64(fd, ubuf, count, |
| 181 | ((loff_t)AA(poshi) << 32) | AA(poslo)); | 182 | ((loff_t)AA(poshi) << 32) | AA(poslo)); |
| 182 | } | 183 | } |
| 183 | 184 | ||
| 184 | asmlinkage long sys32_pwrite(unsigned int fd, const char __user *ubuf, | 185 | COMPAT_SYSCALL_DEFINE5(x86_pwrite, unsigned int, fd, const char __user *, ubuf, |
| 185 | u32 count, u32 poslo, u32 poshi) | 186 | u32, count, u32, poslo, u32, poshi) |
| 186 | { | 187 | { |
| 187 | return sys_pwrite64(fd, ubuf, count, | 188 | return sys_pwrite64(fd, ubuf, count, |
| 188 | ((loff_t)AA(poshi) << 32) | AA(poslo)); | 189 | ((loff_t)AA(poshi) << 32) | AA(poslo)); |
| @@ -193,8 +194,9 @@ asmlinkage long sys32_pwrite(unsigned int fd, const char __user *ubuf, · 4 unmodified lines | |||
| 193 | * Some system calls that need sign extended arguments. This could be | 194 | * Some system calls that need sign extended arguments. This could be |
| 194 | * done by a generic wrapper. | 195 | * done by a generic wrapper. |
| 195 | */ | 196 | */ |
| 196 | long sys32_fadvise64_64(int fd, __u32 offset_low, __u32 offset_high, | 197 | COMPAT_SYSCALL_DEFINE6(x86_fadvise64_64, int, fd, __u32, offset_low, |
| 197 | __u32 len_low, __u32 len_high, int advice) | 198 | __u32, offset_high, __u32, len_low, __u32, len_high, |
| 199 | int, advice) | ||
| 198 | { | 200 | { |
| 199 | return sys_fadvise64_64(fd, | 201 | return sys_fadvise64_64(fd, |
| 200 | (((u64)offset_high)<<32) | offset_low, | 202 | (((u64)offset_high)<<32) | offset_low, |
| @@ -202,31 +204,43 @@ long sys32_fadvise64_64(int fd, __u32 offset_low, __u32 offset_high, · 1 unmodified line | |||
| 202 | advice); | 204 | advice); |
| 203 | } | 205 | } |
| 204 | 206 | ||
| 205 | asmlinkage ssize_t sys32_readahead(int fd, unsigned off_lo, unsigned off_hi, | 207 | COMPAT_SYSCALL_DEFINE4(x86_readahead, int, fd, unsigned int, off_lo, |
| 206 | size_t count) | 208 | unsigned int, off_hi, size_t, count) |
| 207 | { | 209 | { |
| 208 | return sys_readahead(fd, ((u64)off_hi << 32) | off_lo, count); | 210 | return sys_readahead(fd, ((u64)off_hi << 32) | off_lo, count); |
| 209 | } | 211 | } |
| 210 | 212 | ||
| 211 | asmlinkage long sys32_sync_file_range(int fd, unsigned off_low, unsigned off_hi, | 213 | COMPAT_SYSCALL_DEFINE6(x86_sync_file_range, int, fd, unsigned int, off_low, |
| 212 | unsigned n_low, unsigned n_hi, int flags) | 214 | unsigned int, off_hi, unsigned int, n_low, |
| 215 | unsigned int, n_hi, int, flags) | ||
| 213 | { | 216 | { |
| 214 | return sys_sync_file_range(fd, | 217 | return sys_sync_file_range(fd, |
| 215 | ((u64)off_hi << 32) | off_low, | 218 | ((u64)off_hi << 32) | off_low, |
| 216 | ((u64)n_hi << 32) | n_low, flags); | 219 | ((u64)n_hi << 32) | n_low, flags); |
| 217 | } | 220 | } |
| 218 | 221 | ||
| 219 | asmlinkage long sys32_fadvise64(int fd, unsigned offset_lo, unsigned offset_hi, | 222 | COMPAT_SYSCALL_DEFINE5(x86_fadvise64, int, fd, unsigned int, offset_lo, |
| 220 | size_t len, int advice) | 223 | unsigned int, offset_hi, size_t, len, int, advice) |
| 221 | { | 224 | { |
| 222 | return sys_fadvise64_64(fd, ((u64)offset_hi << 32) | offset_lo, | 225 | return sys_fadvise64_64(fd, ((u64)offset_hi << 32) | offset_lo, |
| 223 | len, advice); | 226 | len, advice); |
| 224 | } | 227 | } |
| 225 | 228 | ||
| 226 | asmlinkage long sys32_fallocate(int fd, int mode, unsigned offset_lo, | 229 | COMPAT_SYSCALL_DEFINE6(x86_fallocate, int, fd, int, mode, |
| 227 | unsigned offset_hi, unsigned len_lo, | 230 | unsigned int, offset_lo, unsigned int, offset_hi, |
| 228 | unsigned len_hi) | 231 | unsigned int, len_lo, unsigned int, len_hi) |
| 229 | { | 232 | { |
| 230 | return sys_fallocate(fd, mode, ((u64)offset_hi << 32) | offset_lo, | 233 | return sys_fallocate(fd, mode, ((u64)offset_hi << 32) | offset_lo, |
| 231 | ((u64)len_hi << 32) | len_lo); | 234 | ((u64)len_hi << 32) | len_lo); |
| 232 | } | 235 | } |
| 236 | |||
| 237 | /* | ||
| 238 | * The 32-bit clone ABI is CONFIG_CLONE_BACKWARDS | ||
| 239 | */ | ||
| 240 | COMPAT_SYSCALL_DEFINE5(x86_clone, unsigned long, clone_flags, | ||
| 241 | unsigned long, newsp, int __user *, parent_tidptr, | ||
| 242 | unsigned long, tls_val, int __user *, child_tidptr) | ||
| 243 | { | ||
| 244 | return sys_clone(clone_flags, newsp, parent_tidptr, child_tidptr, | ||
| 245 | tls_val); | ||
| 246 | } | ||
| @@ -94,7 +94,7 @@ static inline unsigned int acpi_processor_cstate_check(unsigned int max_cstate) · 93 unmodified lines | |||
| 94 | if (boot_cpu_data.x86 == 0x0F && | 94 | if (boot_cpu_data.x86 == 0x0F && |
| 95 | boot_cpu_data.x86_vendor == X86_VENDOR_AMD && | 95 | boot_cpu_data.x86_vendor == X86_VENDOR_AMD && |
| 96 | boot_cpu_data.x86_model <= 0x05 && | 96 | boot_cpu_data.x86_model <= 0x05 && |
| 97 | boot_cpu_data.x86_mask < 0x0A) | 97 | boot_cpu_data.x86_stepping < 0x0A) |
| 98 | return 1; | 98 | return 1; |
| 99 | else if (boot_cpu_has(X86_BUG_AMD_APIC_C1E)) | 99 | else if (boot_cpu_has(X86_BUG_AMD_APIC_C1E)) |
| 100 | return 1; | 100 | return 1; |
| @@ -7,6 +7,8 @@ · 6 unmodified lines | |||
| 7 | #ifndef _ASM_X86_MACH_DEFAULT_APM_H | 7 | #ifndef _ASM_X86_MACH_DEFAULT_APM_H |
| 8 | #define _ASM_X86_MACH_DEFAULT_APM_H | 8 | #define _ASM_X86_MACH_DEFAULT_APM_H |
| 9 | 9 | ||
| 10 | #include <asm/nospec-branch.h> | ||
| 11 | |||
| 10 | #ifdef APM_ZERO_SEGS | 12 | #ifdef APM_ZERO_SEGS |
| 11 | # define APM_DO_ZERO_SEGS \ | 13 | # define APM_DO_ZERO_SEGS \ |
| 12 | "pushl %%ds\n\t" \ | 14 | "pushl %%ds\n\t" \ |
| @@ -32,6 +34,7 @@ static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in, · 19 unmodified lines | |||
| 32 | * N.B. We do NOT need a cld after the BIOS call | 34 | * N.B. We do NOT need a cld after the BIOS call |
| 33 | * because we always save and restore the flags. | 35 | * because we always save and restore the flags. |
| 34 | */ | 36 | */ |
| 37 | firmware_restrict_branch_speculation_start(); | ||
| 35 | __asm__ __volatile__(APM_DO_ZERO_SEGS | 38 | __asm__ __volatile__(APM_DO_ZERO_SEGS |
| 36 | "pushl %%edi\n\t" | 39 | "pushl %%edi\n\t" |
| 37 | "pushl %%ebp\n\t" | 40 | "pushl %%ebp\n\t" |
| @@ -44,6 +47,7 @@ static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in, · 6 unmodified lines | |||
| 44 | "=S" (*esi) | 47 | "=S" (*esi) |
| 45 | : "a" (func), "b" (ebx_in), "c" (ecx_in) | 48 | : "a" (func), "b" (ebx_in), "c" (ecx_in) |
| 46 | : "memory", "cc"); | 49 | : "memory", "cc"); |
| 50 | firmware_restrict_branch_speculation_end(); | ||
| 47 | } | 51 | } |
| 48 | 52 | ||
| 49 | static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, | 53 | static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, |
| @@ -56,6 +60,7 @@ static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, · 6 unmodified lines | |||
| 56 | * N.B. We do NOT need a cld after the BIOS call | 60 | * N.B. We do NOT need a cld after the BIOS call |
| 57 | * because we always save and restore the flags. | 61 | * because we always save and restore the flags. |
| 58 | */ | 62 | */ |
| 63 | firmware_restrict_branch_speculation_start(); | ||
| 59 | __asm__ __volatile__(APM_DO_ZERO_SEGS | 64 | __asm__ __volatile__(APM_DO_ZERO_SEGS |
| 60 | "pushl %%edi\n\t" | 65 | "pushl %%edi\n\t" |
| 61 | "pushl %%ebp\n\t" | 66 | "pushl %%ebp\n\t" |
| @@ -68,6 +73,7 @@ static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, · 6 unmodified lines | |||
| 68 | "=S" (si) | 73 | "=S" (si) |
| 69 | : "a" (func), "b" (ebx_in), "c" (ecx_in) | 74 | : "a" (func), "b" (ebx_in), "c" (ecx_in) |
| 70 | : "memory", "cc"); | 75 | : "memory", "cc"); |
| 76 | firmware_restrict_branch_speculation_end(); | ||
| 71 | return error; | 77 | return error; |
| 72 | } | 78 | } |
| 73 | 79 | ||
| @@ -38,7 +38,4 @@ INDIRECT_THUNK(dx) · 37 unmodified lines | |||
| 38 | INDIRECT_THUNK(si) | 38 | INDIRECT_THUNK(si) |
| 39 | INDIRECT_THUNK(di) | 39 | INDIRECT_THUNK(di) |
| 40 | INDIRECT_THUNK(bp) | 40 | INDIRECT_THUNK(bp) |
| 41 | asmlinkage void __fill_rsb(void); | ||
| 42 | asmlinkage void __clear_rsb(void); | ||
| 43 | |||
| 44 | #endif /* CONFIG_RETPOLINE */ | 41 | #endif /* CONFIG_RETPOLINE */ |
| @@ -40,7 +40,7 @@ static inline unsigned long array_index_mask_nospec(unsigned long index, · 39 unmodified lines | |||
| 40 | 40 | ||
| 41 | asm ("cmp %1,%2; sbb %0,%0;" | 41 | asm ("cmp %1,%2; sbb %0,%0;" |
| 42 | :"=r" (mask) | 42 | :"=r" (mask) |
| 43 | :"r"(size),"r" (index) | 43 | :"g"(size),"r" (index) |
| 44 | :"cc"); | 44 | :"cc"); |
| 45 | return mask; | 45 | return mask; |
| 46 | } | 46 | } |
| @@ -52,11 +52,7 @@ static inline unsigned long array_index_mask_nospec(unsigned long index, · 5 unmodified lines | |||
| 52 | #define barrier_nospec() alternative_2("", "mfence", X86_FEATURE_MFENCE_RDTSC, \ | 52 | #define barrier_nospec() alternative_2("", "mfence", X86_FEATURE_MFENCE_RDTSC, \ |
| 53 | "lfence", X86_FEATURE_LFENCE_RDTSC) | 53 | "lfence", X86_FEATURE_LFENCE_RDTSC) |
| 54 | 54 | ||
| 55 | #ifdef CONFIG_X86_PPRO_FENCE | ||
| 56 | #define dma_rmb() rmb() | ||
| 57 | #else | ||
| 58 | #define dma_rmb() barrier() | 55 | #define dma_rmb() barrier() |
| 59 | #endif | ||
| 60 | #define dma_wmb() barrier() | 56 | #define dma_wmb() barrier() |
| 61 | 57 | ||
| 62 | #ifdef CONFIG_X86_32 | 58 | #ifdef CONFIG_X86_32 |
| @@ -68,30 +64,6 @@ static inline unsigned long array_index_mask_nospec(unsigned long index, · 5 unmodified lines | |||
| 68 | #define __smp_wmb() barrier() | 64 | #define __smp_wmb() barrier() |
| 69 | #define __smp_store_mb(var, value) do { (void)xchg(&var, value); } while (0) | 65 | #define __smp_store_mb(var, value) do { (void)xchg(&var, value); } while (0) |
| 70 | 66 | ||
| 71 | #if defined(CONFIG_X86_PPRO_FENCE) | ||
| 72 | |||
| 73 | /* | ||
| 74 | * For this option x86 doesn't have a strong TSO memory | ||
| 75 | * model and we should fall back to full barriers. | ||
| 76 | */ | ||
| 77 | |||
| 78 | #define __smp_store_release(p, v) \ | ||
| 79 | do { \ | ||
| 80 | compiletime_assert_atomic_type(*p); \ | ||
| 81 | __smp_mb(); \ | ||
| 82 | WRITE_ONCE(*p, v); \ | ||
| 83 | } while (0) | ||
| 84 | |||
| 85 | #define __smp_load_acquire(p) \ | ||
| 86 | ({ \ | ||
| 87 | typeof(*p) ___p1 = READ_ONCE(*p); \ | ||
| 88 | compiletime_assert_atomic_type(*p); \ | ||
| 89 | __smp_mb(); \ | ||
| 90 | ___p1; \ | ||
| 91 | }) | ||
| 92 | |||
| 93 | #else /* regular x86 TSO memory ordering */ | ||
| 94 | |||
| 95 | #define __smp_store_release(p, v) \ | 67 | #define __smp_store_release(p, v) \ |
| 96 | do { \ | 68 | do { \ |
| 97 | compiletime_assert_atomic_type(*p); \ | 69 | compiletime_assert_atomic_type(*p); \ |
| @@ -107,8 +79,6 @@ do { \ · 9 unmodified lines | |||
| 107 | ___p1; \ | 79 | ___p1; \ |
| 108 | }) | 80 | }) |
| 109 | 81 | ||
| 110 | #endif | ||
| 111 | |||
| 112 | /* Atomic operations are already serializing on x86 */ | 82 | /* Atomic operations are already serializing on x86 */ |
| 113 | #define __smp_mb__before_atomic() barrier() | 83 | #define __smp_mb__before_atomic() barrier() |
| 114 | #define __smp_mb__after_atomic() barrier() | 84 | #define __smp_mb__after_atomic() barrier() |
| @@ -78,7 +78,7 @@ set_bit(long nr, volatile unsigned long *addr) · 77 unmodified lines | |||
| 78 | : "iq" ((u8)CONST_MASK(nr)) | 78 | : "iq" ((u8)CONST_MASK(nr)) |
| 79 | : "memory"); | 79 | : "memory"); |
| 80 | } else { | 80 | } else { |
| 81 | asm volatile(LOCK_PREFIX "bts %1,%0" | 81 | asm volatile(LOCK_PREFIX __ASM_SIZE(bts) " %1,%0" |
| 82 | : BITOP_ADDR(addr) : "Ir" (nr) : "memory"); | 82 | : BITOP_ADDR(addr) : "Ir" (nr) : "memory"); |
| 83 | } | 83 | } |
| 84 | } | 84 | } |
| @@ -94,7 +94,7 @@ set_bit(long nr, volatile unsigned long *addr) · 9 unmodified lines | |||
| 94 | */ | 94 | */ |
| 95 | static __always_inline void __set_bit(long nr, volatile unsigned long *addr) | 95 | static __always_inline void __set_bit(long nr, volatile unsigned long *addr) |
| 96 | { | 96 | { |
| 97 | asm volatile("bts %1,%0" : ADDR : "Ir" (nr) : "memory"); | 97 | asm volatile(__ASM_SIZE(bts) " %1,%0" : ADDR : "Ir" (nr) : "memory"); |
| 98 | } | 98 | } |
| 99 | 99 | ||
| 100 | /** | 100 | /** |
| @@ -115,7 +115,7 @@ clear_bit(long nr, volatile unsigned long *addr) · 14 unmodified lines | |||
| 115 | : CONST_MASK_ADDR(nr, addr) | 115 | : CONST_MASK_ADDR(nr, addr) |
| 116 | : "iq" ((u8)~CONST_MASK(nr))); | 116 | : "iq" ((u8)~CONST_MASK(nr))); |
| 117 | } else { | 117 | } else { |
| 118 | asm volatile(LOCK_PREFIX "btr %1,%0" | 118 | asm volatile(LOCK_PREFIX __ASM_SIZE(btr) " %1,%0" |
| 119 | : BITOP_ADDR(addr) | 119 | : BITOP_ADDR(addr) |
| 120 | : "Ir" (nr)); | 120 | : "Ir" (nr)); |
| 121 | } | 121 | } |
| @@ -137,7 +137,7 @@ static __always_inline void clear_bit_unlock(long nr, volatile unsigned long *ad · 15 unmodified lines | |||
| 137 | 137 | ||
| 138 | static __always_inline void __clear_bit(long nr, volatile unsigned long *addr) | 138 | static __always_inline void __clear_bit(long nr, volatile unsigned long *addr) |
| 139 | { | 139 | { |
| 140 | asm volatile("btr %1,%0" : ADDR : "Ir" (nr)); | 140 | asm volatile(__ASM_SIZE(btr) " %1,%0" : ADDR : "Ir" (nr)); |
| 141 | } | 141 | } |
| 142 | 142 | ||
| 143 | static __always_inline bool clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) | 143 | static __always_inline bool clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) |
| @@ -182,7 +182,7 @@ static __always_inline void __clear_bit_unlock(long nr, volatile unsigned long * · 38 unmodified lines | |||
| 182 | */ | 182 | */ |
| 183 | static __always_inline void __change_bit(long nr, volatile unsigned long *addr) | 183 | static __always_inline void __change_bit(long nr, volatile unsigned long *addr) |
| 184 | { | 184 | { |
| 185 | asm volatile("btc %1,%0" : ADDR : "Ir" (nr)); | 185 | asm volatile(__ASM_SIZE(btc) " %1,%0" : ADDR : "Ir" (nr)); |
| 186 | } | 186 | } |
| 187 | 187 | ||
| 188 | /** | 188 | /** |
| @@ -201,7 +201,7 @@ static __always_inline void change_bit(long nr, volatile unsigned long *addr) · 12 unmodified lines | |||
| 201 | : CONST_MASK_ADDR(nr, addr) | 201 | : CONST_MASK_ADDR(nr, addr) |
| 202 | : "iq" ((u8)CONST_MASK(nr))); | 202 | : "iq" ((u8)CONST_MASK(nr))); |
| 203 | } else { | 203 | } else { |
| 204 | asm volatile(LOCK_PREFIX "btc %1,%0" | 204 | asm volatile(LOCK_PREFIX __ASM_SIZE(btc) " %1,%0" |
| 205 | : BITOP_ADDR(addr) | 205 | : BITOP_ADDR(addr) |
| 206 | : "Ir" (nr)); | 206 | : "Ir" (nr)); |
| 207 | } | 207 | } |
| @@ -217,7 +217,8 @@ static __always_inline void change_bit(long nr, volatile unsigned long *addr) · 9 unmodified lines | |||
| 217 | */ | 217 | */ |
| 218 | static __always_inline bool test_and_set_bit(long nr, volatile unsigned long *addr) | 218 | static __always_inline bool test_and_set_bit(long nr, volatile unsigned long *addr) |
| 219 | { | 219 | { |
| 220 | GEN_BINARY_RMWcc(LOCK_PREFIX "bts", *addr, "Ir", nr, "%0", c); | 220 | GEN_BINARY_RMWcc(LOCK_PREFIX __ASM_SIZE(bts), |
| 221 | *addr, "Ir", nr, "%0", c); | ||
| 221 | } | 222 | } |
| 222 | 223 | ||
| 223 | /** | 224 | /** |
| @@ -246,7 +247,7 @@ static __always_inline bool __test_and_set_bit(long nr, volatile unsigned long * · 22 unmodified lines | |||
| 246 | { | 247 | { |
| 247 | bool oldbit; | 248 | bool oldbit; |
| 248 | 249 | ||
| 249 | asm("bts %2,%1" | 250 | asm(__ASM_SIZE(bts) " %2,%1" |
| 250 | CC_SET(c) | 251 | CC_SET(c) |
| 251 | : CC_OUT(c) (oldbit), ADDR | 252 | : CC_OUT(c) (oldbit), ADDR |
| 252 | : "Ir" (nr)); | 253 | : "Ir" (nr)); |
| @@ -263,7 +264,8 @@ static __always_inline bool __test_and_set_bit(long nr, volatile unsigned long * · 10 unmodified lines | |||
| 263 | */ | 264 | */ |
| 264 | static __always_inline bool test_and_clear_bit(long nr, volatile unsigned long *addr) | 265 | static __always_inline bool test_and_clear_bit(long nr, volatile unsigned long *addr) |
| 265 | { | 266 | { |
| 266 | GEN_BINARY_RMWcc(LOCK_PREFIX "btr", *addr, "Ir", nr, "%0", c); | 267 | GEN_BINARY_RMWcc(LOCK_PREFIX __ASM_SIZE(btr), |
| 268 | *addr, "Ir", nr, "%0", c); | ||
| 267 | } | 269 | } |
| 268 | 270 | ||
| 269 | /** | 271 | /** |
| @@ -286,7 +288,7 @@ static __always_inline bool __test_and_clear_bit(long nr, volatile unsigned long · 16 unmodified lines | |||
| 286 | { | 288 | { |
| 287 | bool oldbit; | 289 | bool oldbit; |
| 288 | 290 | ||
| 289 | asm volatile("btr %2,%1" | 291 | asm volatile(__ASM_SIZE(btr) " %2,%1" |
| 290 | CC_SET(c) | 292 | CC_SET(c) |
| 291 | : CC_OUT(c) (oldbit), ADDR | 293 | : CC_OUT(c) (oldbit), ADDR |
| 292 | : "Ir" (nr)); | 294 | : "Ir" (nr)); |
| @@ -298,7 +300,7 @@ static __always_inline bool __test_and_change_bit(long nr, volatile unsigned lon · 5 unmodified lines | |||
| 298 | { | 300 | { |
| 299 | bool oldbit; | 301 | bool oldbit; |
| 300 | 302 | ||
| 301 | asm volatile("btc %2,%1" | 303 | asm volatile(__ASM_SIZE(btc) " %2,%1" |
| 302 | CC_SET(c) | 304 | CC_SET(c) |
| 303 | : CC_OUT(c) (oldbit), ADDR | 305 | : CC_OUT(c) (oldbit), ADDR |
| 304 | : "Ir" (nr) : "memory"); | 306 | : "Ir" (nr) : "memory"); |
| @@ -316,7 +318,8 @@ static __always_inline bool __test_and_change_bit(long nr, volatile unsigned lon · 11 unmodified lines | |||
| 316 | */ | 318 | */ |
| 317 | static __always_inline bool test_and_change_bit(long nr, volatile unsigned long *addr) | 319 | static __always_inline bool test_and_change_bit(long nr, volatile unsigned long *addr) |
| 318 | { | 320 | { |
| 319 | GEN_BINARY_RMWcc(LOCK_PREFIX "btc", *addr, "Ir", nr, "%0", c); | 321 | GEN_BINARY_RMWcc(LOCK_PREFIX __ASM_SIZE(btc), |
| 322 | *addr, "Ir", nr, "%0", c); | ||
| 320 | } | 323 | } |
| 321 | 324 | ||
| 322 | static __always_inline bool constant_test_bit(long nr, const volatile unsigned long *addr) | 325 | static __always_inline bool constant_test_bit(long nr, const volatile unsigned long *addr) |
| @@ -329,7 +332,7 @@ static __always_inline bool variable_test_bit(long nr, volatile const unsigned l · 6 unmodified lines | |||
| 329 | { | 332 | { |
| 330 | bool oldbit; | 333 | bool oldbit; |
| 331 | 334 | ||
| 332 | asm volatile("bt %2,%1" | 335 | asm volatile(__ASM_SIZE(bt) " %2,%1" |
| 333 | CC_SET(c) | 336 | CC_SET(c) |
| 334 | : CC_OUT(c) (oldbit) | 337 | : CC_OUT(c) (oldbit) |
| 335 | : "m" (*(unsigned long *)addr), "Ir" (nr)); | 338 | : "m" (*(unsigned long *)addr), "Ir" (nr)); |
| @@ -5,23 +5,20 @@ · 4 unmodified lines | |||
| 5 | #include <linux/stringify.h> | 5 | #include <linux/stringify.h> |
| 6 | 6 | ||
| 7 | /* | 7 | /* |
| 8 | * Since some emulators terminate on UD2, we cannot use it for WARN. | 8 | * Despite that some emulators terminate on UD2, we use it for WARN(). |
| 9 | * Since various instruction decoders disagree on the length of UD1, | ||
| 10 | * we cannot use it either. So use UD0 for WARN. | ||
| 11 | * | 9 | * |
| 12 | * (binutils knows about "ud1" but {en,de}codes it as 2 bytes, whereas | 10 | * Since various instruction decoders/specs disagree on the encoding of |
| 13 | * our kernel decoder thinks it takes a ModRM byte, which seems consistent | 11 | * UD0/UD1. |
| 14 | * with various things like the Intel SDM instruction encoding rules) | ||
| 15 | */ | 12 | */ |
| 16 | 13 | ||
| 17 | #define ASM_UD0 ".byte 0x0f, 0xff" | 14 | #define ASM_UD0 ".byte 0x0f, 0xff" /* + ModRM (for Intel) */ |
| 18 | #define ASM_UD1 ".byte 0x0f, 0xb9" /* + ModRM */ | 15 | #define ASM_UD1 ".byte 0x0f, 0xb9" /* + ModRM */ |
| 19 | #define ASM_UD2 ".byte 0x0f, 0x0b" | 16 | #define ASM_UD2 ".byte 0x0f, 0x0b" |
| 20 | 17 | ||
| 21 | #define INSN_UD0 0xff0f | 18 | #define INSN_UD0 0xff0f |
| 22 | #define INSN_UD2 0x0b0f | 19 | #define INSN_UD2 0x0b0f |
| 23 | 20 | ||
| 24 | #define LEN_UD0 2 | 21 | #define LEN_UD2 2 |
| 25 | 22 | ||
| 26 | #ifdef CONFIG_GENERIC_BUG | 23 | #ifdef CONFIG_GENERIC_BUG |
| 27 | 24 | ||
| @@ -77,7 +74,11 @@ do { \ · 49 unmodified lines | |||
| 77 | unreachable(); \ | 74 | unreachable(); \ |
| 78 | } while (0) | 75 | } while (0) |
| 79 | 76 | ||
| 80 | #define __WARN_FLAGS(flags) _BUG_FLAGS(ASM_UD0, BUGFLAG_WARNING|(flags)) | 77 | #define __WARN_FLAGS(flags) \ |
| 78 | do { \ | ||
| 79 | _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags)); \ | ||
| 80 | annotate_reachable(); \ | ||
| 81 | } while (0) | ||
| 81 | 82 | ||
| 82 | #include <asm-generic/bug.h> | 83 | #include <asm-generic/bug.h> |
| 83 | 84 | ||
| @@ -148,45 +148,46 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit); · 147 unmodified lines | |||
| 148 | */ | 148 | */ |
| 149 | static __always_inline __pure bool _static_cpu_has(u16 bit) | 149 | static __always_inline __pure bool _static_cpu_has(u16 bit) |
| 150 | { | 150 | { |
| 151 | asm_volatile_goto("1: jmp 6f\n" | 151 | asm_volatile_goto("1: jmp 6f\n" |
| 152 | "2:\n" | 152 | "2:\n" |
| 153 | ".skip -(((5f-4f) - (2b-1b)) > 0) * " | 153 | ".skip -(((5f-4f) - (2b-1b)) > 0) * " |
| 154 | "((5f-4f) - (2b-1b)),0x90\n" | 154 | "((5f-4f) - (2b-1b)),0x90\n" |
| 155 | "3:\n" | 155 | "3:\n" |
| 156 | ".section .altinstructions,\"a\"\n" | 156 | ".section .altinstructions,\"a\"\n" |
| 157 | " .long 1b - .\n" /* src offset */ | 157 | " .long 1b - .\n" /* src offset */ |
| 158 | " .long 4f - .\n" /* repl offset */ | 158 | " .long 4f - .\n" /* repl offset */ |
| 159 | " .word %P1\n" /* always replace */ | 159 | " .word %P[always]\n" /* always replace */ |
| 160 | " .byte 3b - 1b\n" /* src len */ | 160 | " .byte 3b - 1b\n" /* src len */ |
| 161 | " .byte 5f - 4f\n" /* repl len */ | 161 | " .byte 5f - 4f\n" /* repl len */ |
| 162 | " .byte 3b - 2b\n" /* pad len */ | 162 | " .byte 3b - 2b\n" /* pad len */ |
| 163 | ".previous\n" | 163 | ".previous\n" |
| 164 | ".section .altinstr_replacement,\"ax\"\n" | 164 | ".section .altinstr_replacement,\"ax\"\n" |
| 165 | "4: jmp %l[t_no]\n" | 165 | "4: jmp %l[t_no]\n" |
| 166 | "5:\n" | 166 | "5:\n" |
| 167 | ".previous\n" | 167 | ".previous\n" |
| 168 | ".section .altinstructions,\"a\"\n" | 168 | ".section .altinstructions,\"a\"\n" |
| 169 | " .long 1b - .\n" /* src offset */ | 169 | " .long 1b - .\n" /* src offset */ |
| 170 | " .long 0\n" /* no replacement */ | 170 | " .long 0\n" /* no replacement */ |
| 171 | " .word %P0\n" /* feature bit */ | 171 | " .word %P[feature]\n" /* feature bit */ |
| 172 | " .byte 3b - 1b\n" /* src len */ | 172 | " .byte 3b - 1b\n" /* src len */ |
| 173 | " .byte 0\n" /* repl len */ | 173 | " .byte 0\n" /* repl len */ |
| 174 | " .byte 0\n" /* pad len */ | 174 | " .byte 0\n" /* pad len */ |
| 175 | ".previous\n" | 175 | ".previous\n" |
| 176 | ".section .altinstr_aux,\"ax\"\n" | 176 | ".section .altinstr_aux,\"ax\"\n" |
| 177 | "6:\n" | 177 | "6:\n" |
| 178 | " testb %[bitnum],%[cap_byte]\n" | 178 | " testb %[bitnum],%[cap_byte]\n" |
| 179 | " jnz %l[t_yes]\n" | 179 | " jnz %l[t_yes]\n" |
| 180 | " jmp %l[t_no]\n" | 180 | " jmp %l[t_no]\n" |
| 181 | ".previous\n" | 181 | ".previous\n" |
| 182 | : : "i" (bit), "i" (X86_FEATURE_ALWAYS), | 182 | : : [feature] "i" (bit), |
| 183 | [bitnum] "i" (1 << (bit & 7)), | 183 | [always] "i" (X86_FEATURE_ALWAYS), |
| 184 | [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >> 3]) | 184 | [bitnum] "i" (1 << (bit & 7)), |
| 185 | : : t_yes, t_no); | 185 | [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >> 3]) |
| 186 | t_yes: | 186 | : : t_yes, t_no); |
| 187 | return true; | 187 | t_yes: |
| 188 | t_no: | 188 | return true; |
| 189 | return false; | 189 | t_no: |
| 190 | return false; | ||
| 190 | } | 191 | } |
| 191 | 192 | ||
| 192 | #define static_cpu_has(bit) \ | 193 | #define static_cpu_has(bit) \ |
| @@ -213,6 +213,7 @@ · 212 unmodified lines | |||
| 213 | #define X86_FEATURE_SEV ( 7*32+20) /* AMD Secure Encrypted Virtualization */ | 213 | #define X86_FEATURE_SEV ( 7*32+20) /* AMD Secure Encrypted Virtualization */ |
| 214 | 214 | ||
| 215 | #define X86_FEATURE_USE_IBPB ( 7*32+21) /* "" Indirect Branch Prediction Barrier enabled */ | 215 | #define X86_FEATURE_USE_IBPB ( 7*32+21) /* "" Indirect Branch Prediction Barrier enabled */ |
| 216 | #define X86_FEATURE_USE_IBRS_FW ( 7*32+22) /* "" Use IBRS during runtime firmware calls */ | ||
| 216 | 217 | ||
| 217 | /* Virtualization flags: Linux defined, word 8 */ | 218 | /* Virtualization flags: Linux defined, word 8 */ |
| 218 | #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */ | 219 | #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */ |
| @@ -315,6 +316,7 @@ · 96 unmodified lines | |||
| 315 | #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ | 316 | #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ |
| 316 | #define X86_FEATURE_AVX512_VNNI (16*32+11) /* Vector Neural Network Instructions */ | 317 | #define X86_FEATURE_AVX512_VNNI (16*32+11) /* Vector Neural Network Instructions */ |
| 317 | #define X86_FEATURE_AVX512_BITALG (16*32+12) /* Support for VPOPCNT[B,W] and VPSHUF-BITQMB instructions */ | 318 | #define X86_FEATURE_AVX512_BITALG (16*32+12) /* Support for VPOPCNT[B,W] and VPSHUF-BITQMB instructions */ |
| 319 | #define X86_FEATURE_TME (16*32+13) /* Intel Total Memory Encryption */ | ||
| 318 | #define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */ | 320 | #define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */ |
| 319 | #define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */ | 321 | #define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */ |
| 320 | #define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */ | 322 | #define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */ |
| @@ -327,6 +329,7 @@ · 6 unmodified lines | |||
| 327 | /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */ | 329 | /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */ |
| 328 | #define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* AVX-512 Neural Network Instructions */ | 330 | #define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* AVX-512 Neural Network Instructions */ |
| 329 | #define X86_FEATURE_AVX512_4FMAPS (18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */ | 331 | #define X86_FEATURE_AVX512_4FMAPS (18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */ |
| 332 | #define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */ | ||
| 330 | #define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */ | 333 | #define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */ |
| 331 | #define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */ | 334 | #define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */ |
| 332 | #define X86_FEATURE_ARCH_CAPABILITIES (18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */ | 335 | #define X86_FEATURE_ARCH_CAPABILITIES (18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */ |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | |||
| 6 | #include <asm/pgtable.h> | 6 | #include <asm/pgtable.h> |
| 7 | #include <asm/processor-flags.h> | 7 | #include <asm/processor-flags.h> |
| 8 | #include <asm/tlb.h> | 8 | #include <asm/tlb.h> |
| 9 | #include <asm/nospec-branch.h> | ||
| 9 | 10 | ||
| 10 | /* | 11 | /* |
| 11 | * We map the EFI regions needed for runtime services non-contiguously, | 12 | * We map the EFI regions needed for runtime services non-contiguously, |
| @@ -36,8 +37,18 @@ · 24 unmodified lines | |||
| 36 | 37 | ||
| 37 | extern asmlinkage unsigned long efi_call_phys(void *, ...); | 38 | extern asmlinkage unsigned long efi_call_phys(void *, ...); |
| 38 | 39 | ||
| 39 | #define arch_efi_call_virt_setup() kernel_fpu_begin() | 40 | #define arch_efi_call_virt_setup() \ |
| 40 | #define arch_efi_call_virt_teardown() kernel_fpu_end() | 41 | ({ \ |
| 42 | kernel_fpu_begin(); \ | ||
| 43 | firmware_restrict_branch_speculation_start(); \ | ||
| 44 | }) | ||
| 45 | |||
| 46 | #define arch_efi_call_virt_teardown() \ | ||
| 47 | ({ \ | ||
| 48 | firmware_restrict_branch_speculation_end(); \ | ||
| 49 | kernel_fpu_end(); \ | ||
| 50 | }) | ||
| 51 | |||
| 41 | 52 | ||
| 42 | /* | 53 | /* |
| 43 | * Wrap all the virtual calls in a way that forces the parameters on the stack. | 54 | * Wrap all the virtual calls in a way that forces the parameters on the stack. |
| @@ -73,6 +84,7 @@ struct efi_scratch { · 29 unmodified lines | |||
| 73 | efi_sync_low_kernel_mappings(); \ | 84 | efi_sync_low_kernel_mappings(); \ |
| 74 | preempt_disable(); \ | 85 | preempt_disable(); \ |
| 75 | __kernel_fpu_begin(); \ | 86 | __kernel_fpu_begin(); \ |
| 87 | firmware_restrict_branch_speculation_start(); \ | ||
| 76 | \ | 88 | \ |
| 77 | if (efi_scratch.use_pgd) { \ | 89 | if (efi_scratch.use_pgd) { \ |
| 78 | efi_scratch.prev_cr3 = __read_cr3(); \ | 90 | efi_scratch.prev_cr3 = __read_cr3(); \ |
| @@ -91,6 +103,7 @@ struct efi_scratch { · 12 unmodified lines | |||
| 91 | __flush_tlb_all(); \ | 103 | __flush_tlb_all(); \ |
| 92 | } \ | 104 | } \ |
| 93 | \ | 105 | \ |
| 106 | firmware_restrict_branch_speculation_end(); \ | ||
| 94 | __kernel_fpu_end(); \ | 107 | __kernel_fpu_end(); \ |
| 95 | preempt_enable(); \ | 108 | preempt_enable(); \ |
| 96 | }) | 109 | }) |
| @@ -232,21 +232,6 @@ extern void set_iounmap_nonlazy(void); · 231 unmodified lines | |||
| 232 | */ | 232 | */ |
| 233 | #define __ISA_IO_base ((char __iomem *)(PAGE_OFFSET)) | 233 | #define __ISA_IO_base ((char __iomem *)(PAGE_OFFSET)) |
| 234 | 234 | ||
| 235 | /* | ||
| 236 | * Cache management | ||
| 237 | * | ||
| 238 | * This needed for two cases | ||
| 239 | * 1. Out of order aware processors | ||
| 240 | * 2. Accidentally out of order processors (PPro errata #51) | ||
| 241 | */ | ||
| 242 | |||
| 243 | static inline void flush_write_buffers(void) | ||
| 244 | { | ||
| 245 | #if defined(CONFIG_X86_PPRO_FENCE) | ||
| 246 | asm volatile("lock; addl $0,0(%%esp)": : :"memory"); | ||
| 247 | #endif | ||
| 248 | } | ||
| 249 | |||
| 250 | #endif /* __KERNEL__ */ | 235 | #endif /* __KERNEL__ */ |
| 251 | 236 | ||
| 252 | extern void native_io_delay(void); | 237 | extern void native_io_delay(void); |
| @@ -507,6 +507,7 @@ struct kvm_vcpu_arch { · 506 unmodified lines | |||
| 507 | u64 smi_count; | 507 | u64 smi_count; |
| 508 | bool tpr_access_reporting; | 508 | bool tpr_access_reporting; |
| 509 | u64 ia32_xss; | 509 | u64 ia32_xss; |
| 510 | u64 microcode_version; | ||
| 510 | 511 | ||
| 511 | /* | 512 | /* |
| 512 | * Paging state of the vcpu | 513 | * Paging state of the vcpu |
| @@ -1095,6 +1096,8 @@ struct kvm_x86_ops { · 582 unmodified lines | |||
| 1095 | int (*mem_enc_op)(struct kvm *kvm, void __user *argp); | 1096 | int (*mem_enc_op)(struct kvm *kvm, void __user *argp); |
| 1096 | int (*mem_enc_reg_region)(struct kvm *kvm, struct kvm_enc_region *argp); | 1097 | int (*mem_enc_reg_region)(struct kvm *kvm, struct kvm_enc_region *argp); |
| 1097 | int (*mem_enc_unreg_region)(struct kvm *kvm, struct kvm_enc_region *argp); | 1098 | int (*mem_enc_unreg_region)(struct kvm *kvm, struct kvm_enc_region *argp); |
| 1099 | |||
| 1100 | int (*get_msr_feature)(struct kvm_msr_entry *entry); | ||
| 1098 | }; | 1101 | }; |
| 1099 | 1102 | ||
| 1100 | struct kvm_arch_async_pf { | 1103 | struct kvm_arch_async_pf { |
| @@ -1464,7 +1467,4 @@ static inline int kvm_cpu_get_apicid(int mps_cpu) · 363 unmodified lines | |||
| 1464 | #define put_smstate(type, buf, offset, val) \ | 1467 | #define put_smstate(type, buf, offset, val) \ |
| 1465 | *(type *)((buf) + (offset) - 0x7e00) = val | 1468 | *(type *)((buf) + (offset) - 0x7e00) = val |
| 1466 | 1469 | ||
| 1467 | void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, | ||
| 1468 | unsigned long start, unsigned long end); | ||
| 1469 | |||
| 1470 | #endif /* _ASM_X86_KVM_HOST_H */ | 1470 | #endif /* _ASM_X86_KVM_HOST_H */ |
| @@ -37,7 +37,13 @@ struct cpu_signature { · 36 unmodified lines | |||
| 37 | 37 | ||
| 38 | struct device; | 38 | struct device; |
| 39 | 39 | ||
| 40 | enum ucode_state { UCODE_ERROR, UCODE_OK, UCODE_NFOUND }; | 40 | enum ucode_state { |
| 41 | UCODE_OK = 0, | ||
| 42 | UCODE_NEW, | ||
| 43 | UCODE_UPDATED, | ||
| 44 | UCODE_NFOUND, | ||
| 45 | UCODE_ERROR, | ||
| 46 | }; | ||
| 41 | 47 | ||
| 42 | struct microcode_ops { | 48 | struct microcode_ops { |
| 43 | enum ucode_state (*request_microcode_user) (int cpu, | 49 | enum ucode_state (*request_microcode_user) (int cpu, |
| @@ -54,7 +60,7 @@ struct microcode_ops { · 10 unmodified lines | |||
| 54 | * are being called. | 60 | * are being called. |
| 55 | * See also the "Synchronization" section in microcode_core.c. | 61 | * See also the "Synchronization" section in microcode_core.c. |
| 56 | */ | 62 | */ |
| 57 | int (*apply_microcode) (int cpu); | 63 | enum ucode_state (*apply_microcode) (int cpu); |
| 58 | int (*collect_cpu_info) (int cpu, struct cpu_signature *csig); | 64 | int (*collect_cpu_info) (int cpu, struct cpu_signature *csig); |
| 59 | }; | 65 | }; |
| 60 | 66 | ||
| @@ -74,6 +74,7 @@ static inline void *ldt_slot_va(int slot) · 73 unmodified lines | |||
| 74 | return (void *)(LDT_BASE_ADDR + LDT_SLOT_STRIDE * slot); | 74 | return (void *)(LDT_BASE_ADDR + LDT_SLOT_STRIDE * slot); |
| 75 | #else | 75 | #else |
| 76 | BUG(); | 76 | BUG(); |
| 77 | return (void *)fix_to_virt(FIX_HOLE); | ||
| 77 | #endif | 78 | #endif |
| 78 | } | 79 | } |
| 79 | 80 | ||
| @@ -6,6 +6,51 @@ · 5 unmodified lines | |||
| 6 | #include <asm/alternative.h> | 6 | #include <asm/alternative.h> |
| 7 | #include <asm/alternative-asm.h> | 7 | #include <asm/alternative-asm.h> |
| 8 | #include <asm/cpufeatures.h> | 8 | #include <asm/cpufeatures.h> |
| 9 | #include <asm/msr-index.h> | ||
| 10 | |||
| 11 | /* | ||
| 12 | * Fill the CPU return stack buffer. | ||
| 13 | * | ||
| 14 | * Each entry in the RSB, if used for a speculative 'ret', contains an | ||
| 15 | * infinite 'pause; lfence; jmp' loop to capture speculative execution. | ||
| 16 | * | ||
| 17 | * This is required in various cases for retpoline and IBRS-based | ||
| 18 | * mitigations for the Spectre variant 2 vulnerability. Sometimes to | ||
| 19 | * eliminate potentially bogus entries from the RSB, and sometimes | ||
| 20 | * purely to ensure that it doesn't get empty, which on some CPUs would | ||
| 21 | * allow predictions from other (unwanted!) sources to be used. | ||
| 22 | * | ||
| 23 | * We define a CPP macro such that it can be used from both .S files and | ||
| 24 | * inline assembly. It's possible to do a .macro and then include that | ||
| 25 | * from C via asm(".include <asm/nospec-branch.h>") but let's not go there. | ||
| 26 | */ | ||
| 27 | |||
| 28 | #define RSB_CLEAR_LOOPS 32 /* To forcibly overwrite all entries */ | ||
| 29 | #define RSB_FILL_LOOPS 16 /* To avoid underflow */ | ||
| 30 | |||
| 31 | /* | ||
| 32 | * Google experimented with loop-unrolling and this turned out to be | ||
| 33 | * the optimal version — two calls, each with their own speculation | ||
| 34 | * trap should their return address end up getting used, in a loop. | ||
| 35 | */ | ||
| 36 | #define __FILL_RETURN_BUFFER(reg, nr, sp) \ | ||
| 37 | mov $(nr/2), reg; \ | ||
| 38 | 771: \ | ||
| 39 | call 772f; \ | ||
| 40 | 773: /* speculation trap */ \ | ||
| 41 | pause; \ | ||
| 42 | lfence; \ | ||
| 43 | jmp 773b; \ | ||
| 44 | 772: \ | ||
| 45 | call 774f; \ | ||
| 46 | 775: /* speculation trap */ \ | ||
| 47 | pause; \ | ||
| 48 | lfence; \ | ||
| 49 | jmp 775b; \ | ||
| 50 | 774: \ | ||
| 51 | dec reg; \ | ||
| 52 | jnz 771b; \ | ||
| 53 | add $(BITS_PER_LONG/8) * nr, sp; | ||
| 9 | 54 | ||
| 10 | #ifdef __ASSEMBLY__ | 55 | #ifdef __ASSEMBLY__ |
| 11 | 56 | ||
| @@ -22,6 +67,18 @@ · 10 unmodified lines | |||
| 22 | .popsection | 67 | .popsection |
| 23 | .endm | 68 | .endm |
| 24 | 69 | ||
| 70 | /* | ||
| 71 | * This should be used immediately before an indirect jump/call. It tells | ||
| 72 | * objtool the subsequent indirect jump/call is vouched safe for retpoline | ||
| 73 | * builds. | ||
| 74 | */ | ||
| 75 | .macro ANNOTATE_RETPOLINE_SAFE | ||
| 76 | .Lannotate_\@: | ||
| 77 | .pushsection .discard.retpoline_safe | ||
| 78 | _ASM_PTR .Lannotate_\@ | ||
| 79 | .popsection | ||
| 80 | .endm | ||
| 81 | |||
| 25 | /* | 82 | /* |
| 26 | * These are the bare retpoline primitives for indirect jmp and call. | 83 | * These are the bare retpoline primitives for indirect jmp and call. |
| 27 | * Do not use these directly; they only exist to make the ALTERNATIVE | 84 | * Do not use these directly; they only exist to make the ALTERNATIVE |
| @@ -58,9 +115,9 @@ · 30 unmodified lines | |||
| 58 | .macro JMP_NOSPEC reg:req | 115 | .macro JMP_NOSPEC reg:req |
| 59 | #ifdef CONFIG_RETPOLINE | 116 | #ifdef CONFIG_RETPOLINE |
| 60 | ANNOTATE_NOSPEC_ALTERNATIVE | 117 | ANNOTATE_NOSPEC_ALTERNATIVE |
| 61 | ALTERNATIVE_2 __stringify(jmp *\reg), \ | 118 | ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *\reg), \ |
| 62 | __stringify(RETPOLINE_JMP \reg), X86_FEATURE_RETPOLINE, \ | 119 | __stringify(RETPOLINE_JMP \reg), X86_FEATURE_RETPOLINE, \ |
| 63 | __stringify(lfence; jmp *\reg), X86_FEATURE_RETPOLINE_AMD | 120 | __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *\reg), X86_FEATURE_RETPOLINE_AMD |
| 64 | #else | 121 | #else |
| 65 | jmp *\reg | 122 | jmp *\reg |
| 66 | #endif | 123 | #endif |
| @@ -69,18 +126,25 @@ · 2 unmodified lines | |||
| 69 | .macro CALL_NOSPEC reg:req | 126 | .macro CALL_NOSPEC reg:req |
| 70 | #ifdef CONFIG_RETPOLINE | 127 | #ifdef CONFIG_RETPOLINE |
| 71 | ANNOTATE_NOSPEC_ALTERNATIVE | 128 | ANNOTATE_NOSPEC_ALTERNATIVE |
| 72 | ALTERNATIVE_2 __stringify(call *\reg), \ | 129 | ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; call *\reg), \ |
| 73 | __stringify(RETPOLINE_CALL \reg), X86_FEATURE_RETPOLINE,\ | 130 | __stringify(RETPOLINE_CALL \reg), X86_FEATURE_RETPOLINE,\ |
| 74 | __stringify(lfence; call *\reg), X86_FEATURE_RETPOLINE_AMD | 131 | __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; call *\reg), X86_FEATURE_RETPOLINE_AMD |
| 75 | #else | 132 | #else |
| 76 | call *\reg | 133 | call *\reg |
| 77 | #endif | 134 | #endif |
| 78 | .endm | 135 | .endm |
| 79 | 136 | ||
| 80 | /* This clobbers the BX register */ | 137 | /* |
| 81 | .macro FILL_RETURN_BUFFER nr:req ftr:req | 138 | * A simpler FILL_RETURN_BUFFER macro. Don't make people use the CPP |
| 139 | * monstrosity above, manually. | ||
| 140 | */ | ||
| 141 | .macro FILL_RETURN_BUFFER reg:req nr:req ftr:req | ||
| 82 | #ifdef CONFIG_RETPOLINE | 142 | #ifdef CONFIG_RETPOLINE |
| 83 | ALTERNATIVE "", "call __clear_rsb", \ftr | 143 | ANNOTATE_NOSPEC_ALTERNATIVE |
| 144 | ALTERNATIVE "jmp .Lskip_rsb_\@", \ | ||
| 145 | __stringify(__FILL_RETURN_BUFFER(\reg,\nr,%_ASM_SP)) \ | ||
| 146 | \ftr | ||
| 147 | .Lskip_rsb_\@: | ||
| 84 | #endif | 148 | #endif |
| 85 | .endm | 149 | .endm |
| 86 | 150 | ||
| @@ -92,6 +156,12 @@ · 5 unmodified lines | |||
| 92 | ".long 999b - .\n\t" \ | 156 | ".long 999b - .\n\t" \ |
| 93 | ".popsection\n\t" | 157 | ".popsection\n\t" |
| 94 | 158 | ||
| 159 | #define ANNOTATE_RETPOLINE_SAFE \ | ||
| 160 | "999:\n\t" \ | ||
| 161 | ".pushsection .discard.retpoline_safe\n\t" \ | ||
| 162 | _ASM_PTR " 999b\n\t" \ | ||
| 163 | ".popsection\n\t" | ||
| 164 | |||
| 95 | #if defined(CONFIG_X86_64) && defined(RETPOLINE) | 165 | #if defined(CONFIG_X86_64) && defined(RETPOLINE) |
| 96 | 166 | ||
| 97 | /* | 167 | /* |
| @@ -101,6 +171,7 @@ · 3 unmodified lines | |||
| 101 | # define CALL_NOSPEC \ | 171 | # define CALL_NOSPEC \ |
| 102 | ANNOTATE_NOSPEC_ALTERNATIVE \ | 172 | ANNOTATE_NOSPEC_ALTERNATIVE \ |
| 103 | ALTERNATIVE( \ | 173 | ALTERNATIVE( \ |
| 174 | ANNOTATE_RETPOLINE_SAFE \ | ||
| 104 | "call *%[thunk_target]\n", \ | 175 | "call *%[thunk_target]\n", \ |
| 105 | "call __x86_indirect_thunk_%V[thunk_target]\n", \ | 176 | "call __x86_indirect_thunk_%V[thunk_target]\n", \ |
| 106 | X86_FEATURE_RETPOLINE) | 177 | X86_FEATURE_RETPOLINE) |
| @@ -112,7 +183,10 @@ · 5 unmodified lines | |||
| 112 | * otherwise we'll run out of registers. We don't care about CET | 183 | * otherwise we'll run out of registers. We don't care about CET |
| 113 | * here, anyway. | 184 | * here, anyway. |
| 114 | */ | 185 | */ |
| 115 | # define CALL_NOSPEC ALTERNATIVE("call *%[thunk_target]\n", \ | 186 | # define CALL_NOSPEC \ |
| 187 | ALTERNATIVE( \ | ||
| 188 | ANNOTATE_RETPOLINE_SAFE \ | ||
| 189 | "call *%[thunk_target]\n", \ | ||
| 116 | " jmp 904f;\n" \ | 190 | " jmp 904f;\n" \ |
| 117 | " .align 16\n" \ | 191 | " .align 16\n" \ |
| 118 | "901: call 903f;\n" \ | 192 | "901: call 903f;\n" \ |
| @@ -155,20 +229,90 @@ extern char __indirect_thunk_end[]; · 36 unmodified lines | |||
| 155 | static inline void vmexit_fill_RSB(void) | 229 | static inline void vmexit_fill_RSB(void) |
| 156 | { | 230 | { |
| 157 | #ifdef CONFIG_RETPOLINE | 231 | #ifdef CONFIG_RETPOLINE |
| 158 | alternative_input("", | 232 | unsigned long loops; |
| 159 | "call __fill_rsb", | 233 | |
| 160 | X86_FEATURE_RETPOLINE, | 234 | asm volatile (ANNOTATE_NOSPEC_ALTERNATIVE |
| 161 | ASM_NO_INPUT_CLOBBER(_ASM_BX, "memory")); | 235 | ALTERNATIVE("jmp 910f", |
| 236 | __stringify(__FILL_RETURN_BUFFER(%0, RSB_CLEAR_LOOPS, %1)), | ||
| 237 | X86_FEATURE_RETPOLINE) | ||
| 238 | "910:" | ||
| 239 | : "=r" (loops), ASM_CALL_CONSTRAINT | ||
| 240 | : : "memory" ); | ||
| 162 | #endif | 241 | #endif |
| 163 | } | 242 | } |
| 164 | 243 | ||
| 244 | #define alternative_msr_write(_msr, _val, _feature) \ | ||
| 245 | asm volatile(ALTERNATIVE("", \ | ||
| 246 | "movl %[msr], %%ecx\n\t" \ | ||
| 247 | "movl %[val], %%eax\n\t" \ | ||
| 248 | "movl $0, %%edx\n\t" \ | ||
| 249 | "wrmsr", \ | ||
| 250 | _feature) \ | ||
| 251 | : : [msr] "i" (_msr), [val] "i" (_val) \ | ||
| 252 | : "eax", "ecx", "edx", "memory") | ||
| 253 | |||
| 165 | static inline void indirect_branch_prediction_barrier(void) | 254 | static inline void indirect_branch_prediction_barrier(void) |
| 166 | { | 255 | { |
| 167 | alternative_input("", | 256 | alternative_msr_write(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, |
| 168 | "call __ibp_barrier", | 257 | X86_FEATURE_USE_IBPB); |
| 169 | X86_FEATURE_USE_IBPB, | ||
| 170 | ASM_NO_INPUT_CLOBBER("eax", "ecx", "edx", "memory")); | ||
| 171 | } | 258 | } |
| 172 | 259 | ||
| 260 | /* | ||
| 261 | * With retpoline, we must use IBRS to restrict branch prediction | ||
| 262 | * before calling into firmware. | ||
| 263 | * | ||
| 264 | * (Implemented as CPP macros due to header hell.) | ||
| 265 | */ | ||
| 266 | #define firmware_restrict_branch_speculation_start() \ | ||
| 267 | do { \ | ||
| 268 | preempt_disable(); \ | ||
| 269 | alternative_msr_write(MSR_IA32_SPEC_CTRL, SPEC_CTRL_IBRS, \ | ||
| 270 | X86_FEATURE_USE_IBRS_FW); \ | ||
| 271 | } while (0) | ||
| 272 | |||
| 273 | #define firmware_restrict_branch_speculation_end() \ | ||
| 274 | do { \ | ||
| 275 | alternative_msr_write(MSR_IA32_SPEC_CTRL, 0, \ | ||
| 276 | X86_FEATURE_USE_IBRS_FW); \ | ||
| 277 | preempt_enable(); \ | ||
| 278 | } while (0) | ||
| 279 | |||
| 173 | #endif /* __ASSEMBLY__ */ | 280 | #endif /* __ASSEMBLY__ */ |
| 281 | |||
| 282 | /* | ||
| 283 | * Below is used in the eBPF JIT compiler and emits the byte sequence | ||
| 284 | * for the following assembly: | ||
| 285 | * | ||
| 286 | * With retpolines configured: | ||
| 287 | * | ||
| 288 | * callq do_rop | ||
| 289 | * spec_trap: | ||
| 290 | * pause | ||
| 291 | * lfence | ||
| 292 | * jmp spec_trap | ||
| 293 | * do_rop: | ||
| 294 | * mov %rax,(%rsp) | ||
| 295 | * retq | ||
| 296 | * | ||
| 297 | * Without retpolines configured: | ||
| 298 | * | ||
| 299 | * jmp *%rax | ||
| 300 | */ | ||
| 301 | #ifdef CONFIG_RETPOLINE | ||
| 302 | # define RETPOLINE_RAX_BPF_JIT_SIZE 17 | ||
| 303 | # define RETPOLINE_RAX_BPF_JIT() \ | ||
| 304 | EMIT1_off32(0xE8, 7); /* callq do_rop */ \ | ||
| 305 | /* spec_trap: */ \ | ||
| 306 | EMIT2(0xF3, 0x90); /* pause */ \ | ||
| 307 | EMIT3(0x0F, 0xAE, 0xE8); /* lfence */ \ | ||
| 308 | EMIT2(0xEB, 0xF9); /* jmp spec_trap */ \ | ||
| 309 | /* do_rop: */ \ | ||
| 310 | EMIT4(0x48, 0x89, 0x04, 0x24); /* mov %rax,(%rsp) */ \ | ||
| 311 | EMIT1(0xC3); /* retq */ | ||
| 312 | #else | ||
| 313 | # define RETPOLINE_RAX_BPF_JIT_SIZE 2 | ||
| 314 | # define RETPOLINE_RAX_BPF_JIT() \ | ||
| 315 | EMIT2(0xFF, 0xE0); /* jmp *%rax */ | ||
| 316 | #endif | ||
| 317 | |||
| 174 | #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ | 318 | #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ |
| @@ -52,10 +52,6 @@ static inline void clear_page(void *page) · 51 unmodified lines | |||
| 52 | 52 | ||
| 53 | void copy_page(void *to, void *from); | 53 | void copy_page(void *to, void *from); |
| 54 | 54 | ||
| 55 | #ifdef CONFIG_X86_MCE | ||
| 56 | #define arch_unmap_kpfn arch_unmap_kpfn | ||
| 57 | #endif | ||
| 58 | |||
| 59 | #endif /* !__ASSEMBLY__ */ | 55 | #endif /* !__ASSEMBLY__ */ |
| 60 | 56 | ||
| 61 | #ifdef CONFIG_X86_VSYSCALL_EMULATION | 57 | #ifdef CONFIG_X86_VSYSCALL_EMULATION |
| @@ -7,6 +7,7 @@ · 6 unmodified lines | |||
| 7 | #ifdef CONFIG_PARAVIRT | 7 | #ifdef CONFIG_PARAVIRT |
| 8 | #include <asm/pgtable_types.h> | 8 | #include <asm/pgtable_types.h> |
| 9 | #include <asm/asm.h> | 9 | #include <asm/asm.h> |
| 10 | #include <asm/nospec-branch.h> | ||
| 10 | 11 | ||
| 11 | #include <asm/paravirt_types.h> | 12 | #include <asm/paravirt_types.h> |
| 12 | 13 | ||
| @@ -297,9 +298,9 @@ static inline void __flush_tlb_global(void) · 284 unmodified lines | |||
| 297 | { | 298 | { |
| 298 | PVOP_VCALL0(pv_mmu_ops.flush_tlb_kernel); | 299 | PVOP_VCALL0(pv_mmu_ops.flush_tlb_kernel); |
| 299 | } | 300 | } |
| 300 | static inline void __flush_tlb_single(unsigned long addr) | 301 | static inline void __flush_tlb_one_user(unsigned long addr) |
| 301 | { | 302 | { |
| 302 | PVOP_VCALL1(pv_mmu_ops.flush_tlb_single, addr); | 303 | PVOP_VCALL1(pv_mmu_ops.flush_tlb_one_user, addr); |
| 303 | } | 304 | } |
| 304 | 305 | ||
| 305 | static inline void flush_tlb_others(const struct cpumask *cpumask, | 306 | static inline void flush_tlb_others(const struct cpumask *cpumask, |
| @@ -879,23 +880,27 @@ extern void default_banner(void); · 573 unmodified lines | |||
| 879 | 880 | ||
| 880 | #define INTERRUPT_RETURN \ | 881 | #define INTERRUPT_RETURN \ |
| 881 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_iret), CLBR_NONE, \ | 882 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_iret), CLBR_NONE, \ |
| 882 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_iret)) | 883 | ANNOTATE_RETPOLINE_SAFE; \ |
| 884 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_iret);) | ||
| 883 | 885 | ||
| 884 | #define DISABLE_INTERRUPTS(clobbers) \ | 886 | #define DISABLE_INTERRUPTS(clobbers) \ |
| 885 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_disable), clobbers, \ | 887 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_disable), clobbers, \ |
| 886 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ | 888 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ |
| 889 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 887 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_disable); \ | 890 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_disable); \ |
| 888 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) | 891 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) |
| 889 | 892 | ||
| 890 | #define ENABLE_INTERRUPTS(clobbers) \ | 893 | #define ENABLE_INTERRUPTS(clobbers) \ |
| 891 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_enable), clobbers, \ | 894 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_enable), clobbers, \ |
| 892 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ | 895 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ |
| 896 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 893 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_enable); \ | 897 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_enable); \ |
| 894 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) | 898 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) |
| 895 | 899 | ||
| 896 | #ifdef CONFIG_X86_32 | 900 | #ifdef CONFIG_X86_32 |
| 897 | #define GET_CR0_INTO_EAX \ | 901 | #define GET_CR0_INTO_EAX \ |
| 898 | push %ecx; push %edx; \ | 902 | push %ecx; push %edx; \ |
| 903 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 899 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_read_cr0); \ | 904 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_read_cr0); \ |
| 900 | pop %edx; pop %ecx | 905 | pop %edx; pop %ecx |
| 901 | #else /* !CONFIG_X86_32 */ | 906 | #else /* !CONFIG_X86_32 */ |
| @@ -917,21 +922,25 @@ extern void default_banner(void); · 15 unmodified lines | |||
| 917 | */ | 922 | */ |
| 918 | #define SWAPGS \ | 923 | #define SWAPGS \ |
| 919 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_swapgs), CLBR_NONE, \ | 924 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_swapgs), CLBR_NONE, \ |
| 920 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_swapgs) \ | 925 | ANNOTATE_RETPOLINE_SAFE; \ |
| 926 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_swapgs); \ | ||
| 921 | ) | 927 | ) |
| 922 | 928 | ||
| 923 | #define GET_CR2_INTO_RAX \ | 929 | #define GET_CR2_INTO_RAX \ |
| 924 | call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2) | 930 | ANNOTATE_RETPOLINE_SAFE; \ |
| 931 | call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2); | ||
| 925 | 932 | ||
| 926 | #define USERGS_SYSRET64 \ | 933 | #define USERGS_SYSRET64 \ |
| 927 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_usergs_sysret64), \ | 934 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_usergs_sysret64), \ |
| 928 | CLBR_NONE, \ | 935 | CLBR_NONE, \ |
| 929 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_usergs_sysret64)) | 936 | ANNOTATE_RETPOLINE_SAFE; \ |
| 937 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_usergs_sysret64);) | ||
| 930 | 938 | ||
| 931 | #ifdef CONFIG_DEBUG_ENTRY | 939 | #ifdef CONFIG_DEBUG_ENTRY |
| 932 | #define SAVE_FLAGS(clobbers) \ | 940 | #define SAVE_FLAGS(clobbers) \ |
| 933 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_save_fl), clobbers, \ | 941 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_save_fl), clobbers, \ |
| 934 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ | 942 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ |
| 943 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 935 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_save_fl); \ | 944 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_save_fl); \ |
| 936 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) | 945 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) |
| 937 | #endif | 946 | #endif |
| @@ -43,6 +43,7 @@ · 42 unmodified lines | |||
| 43 | #include <asm/desc_defs.h> | 43 | #include <asm/desc_defs.h> |
| 44 | #include <asm/kmap_types.h> | 44 | #include <asm/kmap_types.h> |
| 45 | #include <asm/pgtable_types.h> | 45 | #include <asm/pgtable_types.h> |
| 46 | #include <asm/nospec-branch.h> | ||
| 46 | 47 | ||
| 47 | struct page; | 48 | struct page; |
| 48 | struct thread_struct; | 49 | struct thread_struct; |
| @@ -217,7 +218,7 @@ struct pv_mmu_ops { · 168 unmodified lines | |||
| 217 | /* TLB operations */ | 218 | /* TLB operations */ |
| 218 | void (*flush_tlb_user)(void); | 219 | void (*flush_tlb_user)(void); |
| 219 | void (*flush_tlb_kernel)(void); | 220 | void (*flush_tlb_kernel)(void); |
| 220 | void (*flush_tlb_single)(unsigned long addr); | 221 | void (*flush_tlb_one_user)(unsigned long addr); |
| 221 | void (*flush_tlb_others)(const struct cpumask *cpus, | 222 | void (*flush_tlb_others)(const struct cpumask *cpus, |
| 222 | const struct flush_tlb_info *info); | 223 | const struct flush_tlb_info *info); |
| 223 | 224 | ||
| @@ -392,7 +393,9 @@ int paravirt_disable_iospace(void); · 168 unmodified lines | |||
| 392 | * offset into the paravirt_patch_template structure, and can therefore be | 393 | * offset into the paravirt_patch_template structure, and can therefore be |
| 393 | * freely converted back into a structure offset. | 394 | * freely converted back into a structure offset. |
| 394 | */ | 395 | */ |
| 395 | #define PARAVIRT_CALL "call *%c[paravirt_opptr];" | 396 | #define PARAVIRT_CALL \ |
| 397 | ANNOTATE_RETPOLINE_SAFE \ | ||
| 398 | "call *%c[paravirt_opptr];" | ||
| 396 | 399 | ||
| 397 | /* | 400 | /* |
| 398 | * These macros are intended to wrap calls through one of the paravirt | 401 | * These macros are intended to wrap calls through one of the paravirt |
| @@ -526,7 +526,7 @@ static inline bool x86_this_cpu_variable_test_bit(int nr, · 525 unmodified lines | |||
| 526 | { | 526 | { |
| 527 | bool oldbit; | 527 | bool oldbit; |
| 528 | 528 | ||
| 529 | asm volatile("bt "__percpu_arg(2)",%1" | 529 | asm volatile("btl "__percpu_arg(2)",%1" |
| 530 | CC_SET(c) | 530 | CC_SET(c) |
| 531 | : CC_OUT(c) (oldbit) | 531 | : CC_OUT(c) (oldbit) |
| 532 | : "m" (*(unsigned long __percpu *)addr), "Ir" (nr)); | 532 | : "m" (*(unsigned long __percpu *)addr), "Ir" (nr)); |
| @@ -350,14 +350,14 @@ static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) · 349 unmodified lines | |||
| 350 | { | 350 | { |
| 351 | pmdval_t v = native_pmd_val(pmd); | 351 | pmdval_t v = native_pmd_val(pmd); |
| 352 | 352 | ||
| 353 | return __pmd(v | set); | 353 | return native_make_pmd(v | set); |
| 354 | } | 354 | } |
| 355 | 355 | ||
| 356 | static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) | 356 | static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) |
| 357 | { | 357 | { |
| 358 | pmdval_t v = native_pmd_val(pmd); | 358 | pmdval_t v = native_pmd_val(pmd); |
| 359 | 359 | ||
| 360 | return __pmd(v & ~clear); | 360 | return native_make_pmd(v & ~clear); |
| 361 | } | 361 | } |
| 362 | 362 | ||
| 363 | static inline pmd_t pmd_mkold(pmd_t pmd) | 363 | static inline pmd_t pmd_mkold(pmd_t pmd) |
| @@ -409,14 +409,14 @@ static inline pud_t pud_set_flags(pud_t pud, pudval_t set) · 45 unmodified lines | |||
| 409 | { | 409 | { |
| 410 | pudval_t v = native_pud_val(pud); | 410 | pudval_t v = native_pud_val(pud); |
| 411 | 411 | ||
| 412 | return __pud(v | set); | 412 | return native_make_pud(v | set); |
| 413 | } | 413 | } |
| 414 | 414 | ||
| 415 | static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) | 415 | static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) |
| 416 | { | 416 | { |
| 417 | pudval_t v = native_pud_val(pud); | 417 | pudval_t v = native_pud_val(pud); |
| 418 | 418 | ||
| 419 | return __pud(v & ~clear); | 419 | return native_make_pud(v & ~clear); |
| 420 | } | 420 | } |
| 421 | 421 | ||
| 422 | static inline pud_t pud_mkold(pud_t pud) | 422 | static inline pud_t pud_mkold(pud_t pud) |
| @@ -32,6 +32,7 @@ extern pmd_t initial_pg_pmd[]; · 31 unmodified lines | |||
| 32 | static inline void pgtable_cache_init(void) { } | 32 | static inline void pgtable_cache_init(void) { } |
| 33 | static inline void check_pgt_cache(void) { } | 33 | static inline void check_pgt_cache(void) { } |
| 34 | void paging_init(void); | 34 | void paging_init(void); |
| 35 | void sync_initial_page_table(void); | ||
| 35 | 36 | ||
| 36 | /* | 37 | /* |
| 37 | * Define this if things work differently on an i386 and an i486: | 38 | * Define this if things work differently on an i386 and an i486: |
| @@ -61,7 +62,7 @@ void paging_init(void); · 23 unmodified lines | |||
| 61 | #define kpte_clear_flush(ptep, vaddr) \ | 62 | #define kpte_clear_flush(ptep, vaddr) \ |
| 62 | do { \ | 63 | do { \ |
| 63 | pte_clear(&init_mm, (vaddr), (ptep)); \ | 64 | pte_clear(&init_mm, (vaddr), (ptep)); \ |
| 64 | __flush_tlb_one((vaddr)); \ | 65 | __flush_tlb_one_kernel((vaddr)); \ |
| 65 | } while (0) | 66 | } while (0) |
| 66 | 67 | ||
| 67 | #endif /* !__ASSEMBLY__ */ | 68 | #endif /* !__ASSEMBLY__ */ |
| @@ -28,6 +28,7 @@ extern pgd_t init_top_pgt[]; · 27 unmodified lines | |||
| 28 | #define swapper_pg_dir init_top_pgt | 28 | #define swapper_pg_dir init_top_pgt |
| 29 | 29 | ||
| 30 | extern void paging_init(void); | 30 | extern void paging_init(void); |
| 31 | static inline void sync_initial_page_table(void) { } | ||
| 31 | 32 | ||
| 32 | #define pte_ERROR(e) \ | 33 | #define pte_ERROR(e) \ |
| 33 | pr_err("%s:%d: bad pte %p(%016lx)\n", \ | 34 | pr_err("%s:%d: bad pte %p(%016lx)\n", \ |
| @@ -174,7 +174,6 @@ enum page_cache_mode { · 173 unmodified lines | |||
| 174 | #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) | 174 | #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) |
| 175 | #define __PAGE_KERNEL_RX (__PAGE_KERNEL_EXEC & ~_PAGE_RW) | 175 | #define __PAGE_KERNEL_RX (__PAGE_KERNEL_EXEC & ~_PAGE_RW) |
| 176 | #define __PAGE_KERNEL_NOCACHE (__PAGE_KERNEL | _PAGE_NOCACHE) | 176 | #define __PAGE_KERNEL_NOCACHE (__PAGE_KERNEL | _PAGE_NOCACHE) |
| 177 | #define __PAGE_KERNEL_VSYSCALL (__PAGE_KERNEL_RX | _PAGE_USER) | ||
| 178 | #define __PAGE_KERNEL_VVAR (__PAGE_KERNEL_RO | _PAGE_USER) | 177 | #define __PAGE_KERNEL_VVAR (__PAGE_KERNEL_RO | _PAGE_USER) |
| 179 | #define __PAGE_KERNEL_LARGE (__PAGE_KERNEL | _PAGE_PSE) | 178 | #define __PAGE_KERNEL_LARGE (__PAGE_KERNEL | _PAGE_PSE) |
| 180 | #define __PAGE_KERNEL_LARGE_EXEC (__PAGE_KERNEL_EXEC | _PAGE_PSE) | 179 | #define __PAGE_KERNEL_LARGE_EXEC (__PAGE_KERNEL_EXEC | _PAGE_PSE) |
| @@ -206,7 +205,6 @@ enum page_cache_mode { · 25 unmodified lines | |||
| 206 | #define PAGE_KERNEL_NOCACHE __pgprot(__PAGE_KERNEL_NOCACHE | _PAGE_ENC) | 205 | #define PAGE_KERNEL_NOCACHE __pgprot(__PAGE_KERNEL_NOCACHE | _PAGE_ENC) |
| 207 | #define PAGE_KERNEL_LARGE __pgprot(__PAGE_KERNEL_LARGE | _PAGE_ENC) | 206 | #define PAGE_KERNEL_LARGE __pgprot(__PAGE_KERNEL_LARGE | _PAGE_ENC) |
| 208 | #define PAGE_KERNEL_LARGE_EXEC __pgprot(__PAGE_KERNEL_LARGE_EXEC | _PAGE_ENC) | 207 | #define PAGE_KERNEL_LARGE_EXEC __pgprot(__PAGE_KERNEL_LARGE_EXEC | _PAGE_ENC) |
| 209 | #define PAGE_KERNEL_VSYSCALL __pgprot(__PAGE_KERNEL_VSYSCALL | _PAGE_ENC) | ||
| 210 | #define PAGE_KERNEL_VVAR __pgprot(__PAGE_KERNEL_VVAR | _PAGE_ENC) | 208 | #define PAGE_KERNEL_VVAR __pgprot(__PAGE_KERNEL_VVAR | _PAGE_ENC) |
| 211 | 209 | ||
| 212 | #define PAGE_KERNEL_IO __pgprot(__PAGE_KERNEL_IO) | 210 | #define PAGE_KERNEL_IO __pgprot(__PAGE_KERNEL_IO) |
| @@ -323,6 +321,11 @@ static inline pudval_t native_pud_val(pud_t pud) · 110 unmodified lines | |||
| 323 | #else | 321 | #else |
| 324 | #include <asm-generic/pgtable-nopud.h> | 322 | #include <asm-generic/pgtable-nopud.h> |
| 325 | 323 | ||
| 324 | static inline pud_t native_make_pud(pudval_t val) | ||
| 325 | { | ||
| 326 | return (pud_t) { .p4d.pgd = native_make_pgd(val) }; | ||
| 327 | } | ||
| 328 | |||
| 326 | static inline pudval_t native_pud_val(pud_t pud) | 329 | static inline pudval_t native_pud_val(pud_t pud) |
| 327 | { | 330 | { |
| 328 | return native_pgd_val(pud.p4d.pgd); | 331 | return native_pgd_val(pud.p4d.pgd); |
| @@ -344,6 +347,11 @@ static inline pmdval_t native_pmd_val(pmd_t pmd) · 15 unmodified lines | |||
| 344 | #else | 347 | #else |
| 345 | #include <asm-generic/pgtable-nopmd.h> | 348 | #include <asm-generic/pgtable-nopmd.h> |
| 346 | 349 | ||
| 350 | static inline pmd_t native_make_pmd(pmdval_t val) | ||
| 351 | { | ||
| 352 | return (pmd_t) { .pud.p4d.pgd = native_make_pgd(val) }; | ||
| 353 | } | ||
| 354 | |||
| 347 | static inline pmdval_t native_pmd_val(pmd_t pmd) | 355 | static inline pmdval_t native_pmd_val(pmd_t pmd) |
| 348 | { | 356 | { |
| 349 | return native_pgd_val(pmd.pud.p4d.pgd); | 357 | return native_pgd_val(pmd.pud.p4d.pgd); |
| @@ -91,7 +91,7 @@ struct cpuinfo_x86 { · 90 unmodified lines | |||
| 91 | __u8 x86; /* CPU family */ | 91 | __u8 x86; /* CPU family */ |
| 92 | __u8 x86_vendor; /* CPU vendor */ | 92 | __u8 x86_vendor; /* CPU vendor */ |
| 93 | __u8 x86_model; | 93 | __u8 x86_model; |
| 94 | __u8 x86_mask; | 94 | __u8 x86_stepping; |
| 95 | #ifdef CONFIG_X86_64 | 95 | #ifdef CONFIG_X86_64 |
| 96 | /* Number of 4K pages in DTLB/ITLB combined(in pages): */ | 96 | /* Number of 4K pages in DTLB/ITLB combined(in pages): */ |
| 97 | int x86_tlbsize; | 97 | int x86_tlbsize; |
| @@ -109,7 +109,7 @@ struct cpuinfo_x86 { · 11 unmodified lines | |||
| 109 | char x86_vendor_id[16]; | 109 | char x86_vendor_id[16]; |
| 110 | char x86_model_id[64]; | 110 | char x86_model_id[64]; |
| 111 | /* in KB - valid for CPUS which support this call: */ | 111 | /* in KB - valid for CPUS which support this call: */ |
| 112 | int x86_cache_size; | 112 | unsigned int x86_cache_size; |
| 113 | int x86_cache_alignment; /* In bytes */ | 113 | int x86_cache_alignment; /* In bytes */ |
| 114 | /* Cache QoS architectural values: */ | 114 | /* Cache QoS architectural values: */ |
| 115 | int x86_cache_max_rmid; /* max index */ | 115 | int x86_cache_max_rmid; /* max index */ |
| @@ -977,7 +977,5 @@ bool xen_set_default_idle(void); · 861 unmodified lines | |||
| 977 | 977 | ||
| 978 | void stop_this_cpu(void *dummy); | 978 | void stop_this_cpu(void *dummy); |
| 979 | void df_debug(struct pt_regs *regs, long error_code); | 979 | void df_debug(struct pt_regs *regs, long error_code); |
| 980 | 980 | void microcode_check(void); | |
| 981 | void __ibp_barrier(void); | ||
| 982 | |||
| 983 | #endif /* _ASM_X86_PROCESSOR_H */ | 981 | #endif /* _ASM_X86_PROCESSOR_H */ |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | |||
| 17 | #define _REFCOUNT_EXCEPTION \ | 17 | #define _REFCOUNT_EXCEPTION \ |
| 18 | ".pushsection .text..refcount\n" \ | 18 | ".pushsection .text..refcount\n" \ |
| 19 | "111:\tlea %[counter], %%" _ASM_CX "\n" \ | 19 | "111:\tlea %[counter], %%" _ASM_CX "\n" \ |
| 20 | "112:\t" ASM_UD0 "\n" \ | 20 | "112:\t" ASM_UD2 "\n" \ |
| 21 | ASM_UNREACHABLE \ | 21 | ASM_UNREACHABLE \ |
| 22 | ".popsection\n" \ | 22 | ".popsection\n" \ |
| 23 | "113:\n" \ | 23 | "113:\n" \ |
| @@ -67,13 +67,13 @@ static __always_inline __must_check · 43 unmodified lines | |||
| 67 | bool refcount_sub_and_test(unsigned int i, refcount_t *r) | 67 | bool refcount_sub_and_test(unsigned int i, refcount_t *r) |
| 68 | { | 68 | { |
| 69 | GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", REFCOUNT_CHECK_LT_ZERO, | 69 | GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", REFCOUNT_CHECK_LT_ZERO, |
| 70 | r->refs.counter, "er", i, "%0", e); | 70 | r->refs.counter, "er", i, "%0", e, "cx"); |
| 71 | } | 71 | } |
| 72 | 72 | ||
| 73 | static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r) | 73 | static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r) |
| 74 | { | 74 | { |
| 75 | GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", REFCOUNT_CHECK_LT_ZERO, | 75 | GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", REFCOUNT_CHECK_LT_ZERO, |
| 76 | r->refs.counter, "%0", e); | 76 | r->refs.counter, "%0", e, "cx"); |
| 77 | } | 77 | } |
| 78 | 78 | ||
| 79 | static __always_inline __must_check | 79 | static __always_inline __must_check |
| @@ -2,8 +2,7 @@ · 1 unmodified line | |||
| 2 | #ifndef _ASM_X86_RMWcc | 2 | #ifndef _ASM_X86_RMWcc |
| 3 | #define _ASM_X86_RMWcc | 3 | #define _ASM_X86_RMWcc |
| 4 | 4 | ||
| 5 | #define __CLOBBERS_MEM "memory" | 5 | #define __CLOBBERS_MEM(clb...) "memory", ## clb |
| 6 | #define __CLOBBERS_MEM_CC_CX "memory", "cc", "cx" | ||
| 7 | 6 | ||
| 8 | #if !defined(__GCC_ASM_FLAG_OUTPUTS__) && defined(CC_HAVE_ASM_GOTO) | 7 | #if !defined(__GCC_ASM_FLAG_OUTPUTS__) && defined(CC_HAVE_ASM_GOTO) |
| 9 | 8 | ||
| @@ -40,18 +39,19 @@ do { \ · 30 unmodified lines | |||
| 40 | #endif /* defined(__GCC_ASM_FLAG_OUTPUTS__) || !defined(CC_HAVE_ASM_GOTO) */ | 39 | #endif /* defined(__GCC_ASM_FLAG_OUTPUTS__) || !defined(CC_HAVE_ASM_GOTO) */ |
| 41 | 40 | ||
| 42 | #define GEN_UNARY_RMWcc(op, var, arg0, cc) \ | 41 | #define GEN_UNARY_RMWcc(op, var, arg0, cc) \ |
| 43 | __GEN_RMWcc(op " " arg0, var, cc, __CLOBBERS_MEM) | 42 | __GEN_RMWcc(op " " arg0, var, cc, __CLOBBERS_MEM()) |
| 44 | 43 | ||
| 45 | #define GEN_UNARY_SUFFIXED_RMWcc(op, suffix, var, arg0, cc) \ | 44 | #define GEN_UNARY_SUFFIXED_RMWcc(op, suffix, var, arg0, cc, clobbers...)\ |
| 46 | __GEN_RMWcc(op " " arg0 "\n\t" suffix, var, cc, \ | 45 | __GEN_RMWcc(op " " arg0 "\n\t" suffix, var, cc, \ |
| 47 | __CLOBBERS_MEM_CC_CX) | 46 | __CLOBBERS_MEM(clobbers)) |
| 48 | 47 | ||
| 49 | #define GEN_BINARY_RMWcc(op, var, vcon, val, arg0, cc) \ | 48 | #define GEN_BINARY_RMWcc(op, var, vcon, val, arg0, cc) \ |
| 50 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0, var, cc, \ | 49 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0, var, cc, \ |
| 51 | __CLOBBERS_MEM, vcon (val)) | 50 | __CLOBBERS_MEM(), vcon (val)) |
| 52 | 51 | ||
| 53 | #define GEN_BINARY_SUFFIXED_RMWcc(op, suffix, var, vcon, val, arg0, cc) \ | 52 | #define GEN_BINARY_SUFFIXED_RMWcc(op, suffix, var, vcon, val, arg0, cc, \ |
| 53 | clobbers...) \ | ||
| 54 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0 "\n\t" suffix, var, cc, \ | 54 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0 "\n\t" suffix, var, cc, \ |
| 55 | __CLOBBERS_MEM_CC_CX, vcon (val)) | 55 | __CLOBBERS_MEM(clobbers), vcon (val)) |
| 56 | 56 | ||
| 57 | #endif /* _ASM_X86_RMWcc */ | 57 | #endif /* _ASM_X86_RMWcc */ |
| @@ -10,6 +10,7 @@ extern struct exception_table_entry __stop___ex_table[]; · 9 unmodified lines | |||
| 10 | 10 | ||
| 11 | #if defined(CONFIG_X86_64) | 11 | #if defined(CONFIG_X86_64) |
| 12 | extern char __end_rodata_hpage_align[]; | 12 | extern char __end_rodata_hpage_align[]; |
| 13 | extern char __entry_trampoline_start[], __entry_trampoline_end[]; | ||
| 13 | #endif | 14 | #endif |
| 14 | 15 | ||
| 15 | #endif /* _ASM_X86_SECTIONS_H */ | 16 | #endif /* _ASM_X86_SECTIONS_H */ |
| @@ -129,6 +129,7 @@ static inline void arch_send_call_function_ipi_mask(const struct cpumask *mask) · 128 unmodified lines | |||
| 129 | void cpu_disable_common(void); | 129 | void cpu_disable_common(void); |
| 130 | void native_smp_prepare_boot_cpu(void); | 130 | void native_smp_prepare_boot_cpu(void); |
| 131 | void native_smp_prepare_cpus(unsigned int max_cpus); | 131 | void native_smp_prepare_cpus(unsigned int max_cpus); |
| 132 | void calculate_max_logical_packages(void); | ||
| 132 | void native_smp_cpus_done(unsigned int max_cpus); | 133 | void native_smp_cpus_done(unsigned int max_cpus); |
| 133 | void common_cpu_up(unsigned int cpunum, struct task_struct *tidle); | 134 | void common_cpu_up(unsigned int cpunum, struct task_struct *tidle); |
| 134 | int native_cpu_up(unsigned int cpunum, struct task_struct *tidle); | 135 | int native_cpu_up(unsigned int cpunum, struct task_struct *tidle); |
| @@ -20,31 +20,43 @@ · 19 unmodified lines | |||
| 20 | #include <asm/ia32.h> | 20 | #include <asm/ia32.h> |
| 21 | 21 | ||
| 22 | /* ia32/sys_ia32.c */ | 22 | /* ia32/sys_ia32.c */ |
| 23 | asmlinkage long sys32_truncate64(const char __user *, unsigned long, unsigned long); | 23 | asmlinkage long compat_sys_x86_truncate64(const char __user *, unsigned long, |
| 24 | asmlinkage long sys32_ftruncate64(unsigned int, unsigned long, unsigned long); | 24 | unsigned long); |
| 25 | asmlinkage long compat_sys_x86_ftruncate64(unsigned int, unsigned long, | ||
| 26 | unsigned long); | ||
| 25 | 27 | ||
| 26 | asmlinkage long sys32_stat64(const char __user *, struct stat64 __user *); | 28 | asmlinkage long compat_sys_x86_stat64(const char __user *, |
| 27 | asmlinkage long sys32_lstat64(const char __user *, struct stat64 __user *); | 29 | struct stat64 __user *); |
| 28 | asmlinkage long sys32_fstat64(unsigned int, struct stat64 __user *); | 30 | asmlinkage long compat_sys_x86_lstat64(const char __user *, |
| 29 | asmlinkage long sys32_fstatat(unsigned int, const char __user *, | 31 | struct stat64 __user *); |
| 32 | asmlinkage long compat_sys_x86_fstat64(unsigned int, struct stat64 __user *); | ||
| 33 | asmlinkage long compat_sys_x86_fstatat(unsigned int, const char __user *, | ||
| 30 | struct stat64 __user *, int); | 34 | struct stat64 __user *, int); |
| 31 | struct mmap_arg_struct32; | 35 | struct mmap_arg_struct32; |
| 32 | asmlinkage long sys32_mmap(struct mmap_arg_struct32 __user *); | 36 | asmlinkage long compat_sys_x86_mmap(struct mmap_arg_struct32 __user *); |
| 33 | 37 | ||
| 34 | asmlinkage long sys32_waitpid(compat_pid_t, unsigned int __user *, int); | 38 | asmlinkage long compat_sys_x86_waitpid(compat_pid_t, unsigned int __user *, |
| 39 | int); | ||
| 35 | 40 | ||
| 36 | asmlinkage long sys32_pread(unsigned int, char __user *, u32, u32, u32); | 41 | asmlinkage long compat_sys_x86_pread(unsigned int, char __user *, u32, u32, |
| 37 | asmlinkage long sys32_pwrite(unsigned int, const char __user *, u32, u32, u32); | 42 | u32); |
| 43 | asmlinkage long compat_sys_x86_pwrite(unsigned int, const char __user *, u32, | ||
| 44 | u32, u32); | ||
| 38 | 45 | ||
| 39 | long sys32_fadvise64_64(int, __u32, __u32, __u32, __u32, int); | 46 | asmlinkage long compat_sys_x86_fadvise64_64(int, __u32, __u32, __u32, __u32, |
| 40 | long sys32_vm86_warning(void); | 47 | int); |
| 41 | 48 | ||
| 42 | asmlinkage ssize_t sys32_readahead(int, unsigned, unsigned, size_t); | 49 | asmlinkage ssize_t compat_sys_x86_readahead(int, unsigned int, unsigned int, |
| 43 | asmlinkage long sys32_sync_file_range(int, unsigned, unsigned, | 50 | size_t); |
| 44 | unsigned, unsigned, int); | 51 | asmlinkage long compat_sys_x86_sync_file_range(int, unsigned int, unsigned int, |
| 45 | asmlinkage long sys32_fadvise64(int, unsigned, unsigned, size_t, int); | 52 | unsigned int, unsigned int, |
| 46 | asmlinkage long sys32_fallocate(int, int, unsigned, | 53 | int); |
| 47 | unsigned, unsigned, unsigned); | 54 | asmlinkage long compat_sys_x86_fadvise64(int, unsigned int, unsigned int, |
| 55 | size_t, int); | ||
| 56 | asmlinkage long compat_sys_x86_fallocate(int, int, unsigned int, unsigned int, | ||
| 57 | unsigned int, unsigned int); | ||
| 58 | asmlinkage long compat_sys_x86_clone(unsigned long, unsigned long, int __user *, | ||
| 59 | unsigned long, int __user *); | ||
| 48 | 60 | ||
| 49 | /* ia32/ia32_signal.c */ | 61 | /* ia32/ia32_signal.c */ |
| 50 | asmlinkage long sys32_sigreturn(void); | 62 | asmlinkage long sys32_sigreturn(void); |
| @@ -140,7 +140,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) · 139 unmodified lines | |||
| 140 | #else | 140 | #else |
| 141 | #define __flush_tlb() __native_flush_tlb() | 141 | #define __flush_tlb() __native_flush_tlb() |
| 142 | #define __flush_tlb_global() __native_flush_tlb_global() | 142 | #define __flush_tlb_global() __native_flush_tlb_global() |
| 143 | #define __flush_tlb_single(addr) __native_flush_tlb_single(addr) | 143 | #define __flush_tlb_one_user(addr) __native_flush_tlb_one_user(addr) |
| 144 | #endif | 144 | #endif |
| 145 | 145 | ||
| 146 | static inline bool tlb_defer_switch_to_init_mm(void) | 146 | static inline bool tlb_defer_switch_to_init_mm(void) |
| @@ -400,7 +400,7 @@ static inline void __native_flush_tlb_global(void) · 253 unmodified lines | |||
| 400 | /* | 400 | /* |
| 401 | * flush one page in the user mapping | 401 | * flush one page in the user mapping |
| 402 | */ | 402 | */ |
| 403 | static inline void __native_flush_tlb_single(unsigned long addr) | 403 | static inline void __native_flush_tlb_one_user(unsigned long addr) |
| 404 | { | 404 | { |
| 405 | u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); | 405 | u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); |
| 406 | 406 | ||
| @@ -437,18 +437,31 @@ static inline void __flush_tlb_all(void) · 30 unmodified lines | |||
| 437 | /* | 437 | /* |
| 438 | * flush one page in the kernel mapping | 438 | * flush one page in the kernel mapping |
| 439 | */ | 439 | */ |
| 440 | static inline void __flush_tlb_one(unsigned long addr) | 440 | static inline void __flush_tlb_one_kernel(unsigned long addr) |
| 441 | { | 441 | { |
| 442 | count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ONE); | 442 | count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ONE); |
| 443 | __flush_tlb_single(addr); | 443 | |
| 444 | /* | ||
| 445 | * If PTI is off, then __flush_tlb_one_user() is just INVLPG or its | ||
| 446 | * paravirt equivalent. Even with PCID, this is sufficient: we only | ||
| 447 | * use PCID if we also use global PTEs for the kernel mapping, and | ||
| 448 | * INVLPG flushes global translations across all address spaces. | ||
| 449 | * | ||
| 450 | * If PTI is on, then the kernel is mapped with non-global PTEs, and | ||
| 451 | * __flush_tlb_one_user() will flush the given address for the current | ||
| 452 | * kernel address space and for its usermode counterpart, but it does | ||
| 453 | * not flush it for other address spaces. | ||
| 454 | */ | ||
| 455 | __flush_tlb_one_user(addr); | ||
| 444 | 456 | ||
| 445 | if (!static_cpu_has(X86_FEATURE_PTI)) | 457 | if (!static_cpu_has(X86_FEATURE_PTI)) |
| 446 | return; | 458 | return; |
| 447 | 459 | ||
| 448 | /* | 460 | /* |
| 449 | * __flush_tlb_single() will have cleared the TLB entry for this ASID, | 461 | * See above. We need to propagate the flush to all other address |
| 450 | * but since kernel space is replicated across all, we must also | 462 | * spaces. In principle, we only need to propagate it to kernelmode |
| 451 | * invalidate all others. | 463 | * address spaces, but the extra bookkeeping we would need is not |
| 464 | * worth it. | ||
| 452 | */ | 465 | */ |
| 453 | invalidate_other_asid(); | 466 | invalidate_other_asid(); |
| 454 | } | 467 | } |
| @@ -352,6 +352,7 @@ enum vmcs_field { · 351 unmodified lines | |||
| 352 | #define INTR_TYPE_NMI_INTR (2 << 8) /* NMI */ | 352 | #define INTR_TYPE_NMI_INTR (2 << 8) /* NMI */ |
| 353 | #define INTR_TYPE_HARD_EXCEPTION (3 << 8) /* processor exception */ | 353 | #define INTR_TYPE_HARD_EXCEPTION (3 << 8) /* processor exception */ |
| 354 | #define INTR_TYPE_SOFT_INTR (4 << 8) /* software interrupt */ | 354 | #define INTR_TYPE_SOFT_INTR (4 << 8) /* software interrupt */ |
| 355 | #define INTR_TYPE_PRIV_SW_EXCEPTION (5 << 8) /* ICE breakpoint - undocumented */ | ||
| 355 | #define INTR_TYPE_SOFT_EXCEPTION (6 << 8) /* software exception */ | 356 | #define INTR_TYPE_SOFT_EXCEPTION (6 << 8) /* software exception */ |
| 356 | 357 | ||
| 357 | /* GUEST_INTERRUPTIBILITY_INFO flags. */ | 358 | /* GUEST_INTERRUPTIBILITY_INFO flags. */ |
| @@ -241,24 +241,24 @@ · 240 unmodified lines | |||
| 241 | #define HV_X64_MSR_REENLIGHTENMENT_CONTROL 0x40000106 | 241 | #define HV_X64_MSR_REENLIGHTENMENT_CONTROL 0x40000106 |
| 242 | 242 | ||
| 243 | struct hv_reenlightenment_control { | 243 | struct hv_reenlightenment_control { |
| 244 | u64 vector:8; | 244 | __u64 vector:8; |
| 245 | u64 reserved1:8; | 245 | __u64 reserved1:8; |
| 246 | u64 enabled:1; | 246 | __u64 enabled:1; |
| 247 | u64 reserved2:15; | 247 | __u64 reserved2:15; |
| 248 | u64 target_vp:32; | 248 | __u64 target_vp:32; |
| 249 | }; | 249 | }; |
| 250 | 250 | ||
| 251 | #define HV_X64_MSR_TSC_EMULATION_CONTROL 0x40000107 | 251 | #define HV_X64_MSR_TSC_EMULATION_CONTROL 0x40000107 |
| 252 | #define HV_X64_MSR_TSC_EMULATION_STATUS 0x40000108 | 252 | #define HV_X64_MSR_TSC_EMULATION_STATUS 0x40000108 |
| 253 | 253 | ||
| 254 | struct hv_tsc_emulation_control { | 254 | struct hv_tsc_emulation_control { |
| 255 | u64 enabled:1; | 255 | __u64 enabled:1; |
| 256 | u64 reserved:63; | 256 | __u64 reserved:63; |
| 257 | }; | 257 | }; |
| 258 | 258 | ||
| 259 | struct hv_tsc_emulation_status { | 259 | struct hv_tsc_emulation_status { |
| 260 | u64 inprogress:1; | 260 | __u64 inprogress:1; |
| 261 | u64 reserved:63; | 261 | __u64 reserved:63; |
| 262 | }; | 262 | }; |
| 263 | 263 | ||
| 264 | #define HV_X64_MSR_HYPERCALL_ENABLE 0x00000001 | 264 | #define HV_X64_MSR_HYPERCALL_ENABLE 0x00000001 |
| @@ -26,6 +26,7 @@ · 25 unmodified lines | |||
| 26 | #define KVM_FEATURE_PV_EOI 6 | 26 | #define KVM_FEATURE_PV_EOI 6 |
| 27 | #define KVM_FEATURE_PV_UNHALT 7 | 27 | #define KVM_FEATURE_PV_UNHALT 7 |
| 28 | #define KVM_FEATURE_PV_TLB_FLUSH 9 | 28 | #define KVM_FEATURE_PV_TLB_FLUSH 9 |
| 29 | #define KVM_FEATURE_ASYNC_PF_VMEXIT 10 | ||
| 29 | 30 | ||
| 30 | /* The last 8 bits are used to indicate how to interpret the flags field | 31 | /* The last 8 bits are used to indicate how to interpret the flags field |
| 31 | * in pvclock structure. If no bits are set, all flags are ignored. | 32 | * in pvclock structure. If no bits are set, all flags are ignored. |
| @@ -30,6 +30,7 @@ struct mce { · 29 unmodified lines | |||
| 30 | __u64 synd; /* MCA_SYND MSR: only valid on SMCA systems */ | 30 | __u64 synd; /* MCA_SYND MSR: only valid on SMCA systems */ |
| 31 | __u64 ipid; /* MCA_IPID MSR: only valid on SMCA systems */ | 31 | __u64 ipid; /* MCA_IPID MSR: only valid on SMCA systems */ |
| 32 | __u64 ppin; /* Protected Processor Inventory Number */ | 32 | __u64 ppin; /* Protected Processor Inventory Number */ |
| 33 | __u32 microcode;/* Microcode revision */ | ||
| 33 | }; | 34 | }; |
| 34 | 35 | ||
| 35 | #define MCE_GET_RECORD_LEN _IOR('M', 1, int) | 36 | #define MCE_GET_RECORD_LEN _IOR('M', 1, int) |
| @@ -235,7 +235,7 @@ int amd_cache_northbridges(void) · 234 unmodified lines | |||
| 235 | if (boot_cpu_data.x86 == 0x10 && | 235 | if (boot_cpu_data.x86 == 0x10 && |
| 236 | boot_cpu_data.x86_model >= 0x8 && | 236 | boot_cpu_data.x86_model >= 0x8 && |
| 237 | (boot_cpu_data.x86_model > 0x9 || | 237 | (boot_cpu_data.x86_model > 0x9 || |
| 238 | boot_cpu_data.x86_mask >= 0x1)) | 238 | boot_cpu_data.x86_stepping >= 0x1)) |
| 239 | amd_northbridges.flags |= AMD_NB_L3_INDEX_DISABLE; | 239 | amd_northbridges.flags |= AMD_NB_L3_INDEX_DISABLE; |
| 240 | 240 | ||
| 241 | if (boot_cpu_data.x86 == 0x15) | 241 | if (boot_cpu_data.x86 == 0x15) |
| @@ -546,7 +546,7 @@ static DEFINE_PER_CPU(struct clock_event_device, lapic_events); · 545 unmodified lines | |||
| 546 | 546 | ||
| 547 | static u32 hsx_deadline_rev(void) | 547 | static u32 hsx_deadline_rev(void) |
| 548 | { | 548 | { |
| 549 | switch (boot_cpu_data.x86_mask) { | 549 | switch (boot_cpu_data.x86_stepping) { |
| 550 | case 0x02: return 0x3a; /* EP */ | 550 | case 0x02: return 0x3a; /* EP */ |
| 551 | case 0x04: return 0x0f; /* EX */ | 551 | case 0x04: return 0x0f; /* EX */ |
| 552 | } | 552 | } |
| @@ -556,7 +556,7 @@ static u32 hsx_deadline_rev(void) · 3 unmodified lines | |||
| 556 | 556 | ||
| 557 | static u32 bdx_deadline_rev(void) | 557 | static u32 bdx_deadline_rev(void) |
| 558 | { | 558 | { |
| 559 | switch (boot_cpu_data.x86_mask) { | 559 | switch (boot_cpu_data.x86_stepping) { |
| 560 | case 0x02: return 0x00000011; | 560 | case 0x02: return 0x00000011; |
| 561 | case 0x03: return 0x0700000e; | 561 | case 0x03: return 0x0700000e; |
| 562 | case 0x04: return 0x0f00000c; | 562 | case 0x04: return 0x0f00000c; |
| @@ -568,7 +568,7 @@ static u32 bdx_deadline_rev(void) · 5 unmodified lines | |||
| 568 | 568 | ||
| 569 | static u32 skx_deadline_rev(void) | 569 | static u32 skx_deadline_rev(void) |
| 570 | { | 570 | { |
| 571 | switch (boot_cpu_data.x86_mask) { | 571 | switch (boot_cpu_data.x86_stepping) { |
| 572 | case 0x03: return 0x01000136; | 572 | case 0x03: return 0x01000136; |
| 573 | case 0x04: return 0x02000014; | 573 | case 0x04: return 0x02000014; |
| 574 | } | 574 | } |
| @@ -1603,7 +1603,7 @@ static void __init delay_with_tsc(void) · 1602 unmodified lines | |||
| 1603 | do { | 1603 | do { |
| 1604 | rep_nop(); | 1604 | rep_nop(); |
| 1605 | now = rdtsc(); | 1605 | now = rdtsc(); |
| 1606 | } while ((now - start) < 40000000000UL / HZ && | 1606 | } while ((now - start) < 40000000000ULL / HZ && |
| 1607 | time_before_eq(jiffies, end)); | 1607 | time_before_eq(jiffies, end)); |
| 1608 | } | 1608 | } |
| 1609 | 1609 | ||
| @@ -134,21 +134,40 @@ static void apic_update_vector(struct irq_data *irqd, unsigned int newvec, · 133 unmodified lines | |||
| 134 | { | 134 | { |
| 135 | struct apic_chip_data *apicd = apic_chip_data(irqd); | 135 | struct apic_chip_data *apicd = apic_chip_data(irqd); |
| 136 | struct irq_desc *desc = irq_data_to_desc(irqd); | 136 | struct irq_desc *desc = irq_data_to_desc(irqd); |
| 137 | bool managed = irqd_affinity_is_managed(irqd); | ||
| 137 | 138 | ||
| 138 | lockdep_assert_held(&vector_lock); | 139 | lockdep_assert_held(&vector_lock); |
| 139 | 140 | ||
| 140 | trace_vector_update(irqd->irq, newvec, newcpu, apicd->vector, | 141 | trace_vector_update(irqd->irq, newvec, newcpu, apicd->vector, |
| 141 | apicd->cpu); | 142 | apicd->cpu); |
| 142 | 143 | ||
| 143 | /* Setup the vector move, if required */ | 144 | /* |
| 144 | if (apicd->vector && cpu_online(apicd->cpu)) { | 145 | * If there is no vector associated or if the associated vector is |
| 146 | * the shutdown vector, which is associated to make PCI/MSI | ||
| 147 | * shutdown mode work, then there is nothing to release. Clear out | ||
| 148 | * prev_vector for this and the offlined target case. | ||
| 149 | */ | ||
| 150 | apicd->prev_vector = 0; | ||
| 151 | if (!apicd->vector || apicd->vector == MANAGED_IRQ_SHUTDOWN_VECTOR) | ||
| 152 | goto setnew; | ||
| 153 | /* | ||
| 154 | * If the target CPU of the previous vector is online, then mark | ||
| 155 | * the vector as move in progress and store it for cleanup when the | ||
| 156 | * first interrupt on the new vector arrives. If the target CPU is | ||
| 157 | * offline then the regular release mechanism via the cleanup | ||
| 158 | * vector is not possible and the vector can be immediately freed | ||
| 159 | * in the underlying matrix allocator. | ||
| 160 | */ | ||
| 161 | if (cpu_online(apicd->cpu)) { | ||
| 145 | apicd->move_in_progress = true; | 162 | apicd->move_in_progress = true; |
| 146 | apicd->prev_vector = apicd->vector; | 163 | apicd->prev_vector = apicd->vector; |
| 147 | apicd->prev_cpu = apicd->cpu; | 164 | apicd->prev_cpu = apicd->cpu; |
| 148 | } else { | 165 | } else { |
| 149 | apicd->prev_vector = 0; | 166 | irq_matrix_free(vector_matrix, apicd->cpu, apicd->vector, |
| 167 | managed); | ||
| 150 | } | 168 | } |
| 151 | 169 | ||
| 170 | setnew: | ||
| 152 | apicd->vector = newvec; | 171 | apicd->vector = newvec; |
| 153 | apicd->cpu = newcpu; | 172 | apicd->cpu = newcpu; |
| 154 | BUG_ON(!IS_ERR_OR_NULL(per_cpu(vector_irq, newcpu)[newvec])); | 173 | BUG_ON(!IS_ERR_OR_NULL(per_cpu(vector_irq, newcpu)[newvec])); |
| @@ -1176,16 +1176,25 @@ static void __init decode_gam_rng_tbl(unsigned long ptr) · 1175 unmodified lines | |||
| 1176 | 1176 | ||
| 1177 | uv_gre_table = gre; | 1177 | uv_gre_table = gre; |
| 1178 | for (; gre->type != UV_GAM_RANGE_TYPE_UNUSED; gre++) { | 1178 | for (; gre->type != UV_GAM_RANGE_TYPE_UNUSED; gre++) { |
| 1179 | unsigned long size = ((unsigned long)(gre->limit - lgre) | ||
| 1180 | << UV_GAM_RANGE_SHFT); | ||
| 1181 | int order = 0; | ||
| 1182 | char suffix[] = " KMGTPE"; | ||
| 1183 | |||
| 1184 | while (size > 9999 && order < sizeof(suffix)) { | ||
| 1185 | size /= 1024; | ||
| 1186 | order++; | ||
| 1187 | } | ||
| 1188 | |||
| 1179 | if (!index) { | 1189 | if (!index) { |
| 1180 | pr_info("UV: GAM Range Table...\n"); | 1190 | pr_info("UV: GAM Range Table...\n"); |
| 1181 | pr_info("UV: # %20s %14s %5s %4s %5s %3s %2s\n", "Range", "", "Size", "Type", "NASID", "SID", "PN"); | 1191 | pr_info("UV: # %20s %14s %5s %4s %5s %3s %2s\n", "Range", "", "Size", "Type", "NASID", "SID", "PN"); |
| 1182 | } | 1192 | } |
| 1183 | pr_info("UV: %2d: 0x%014lx-0x%014lx %5luG %3d %04x %02x %02x\n", | 1193 | pr_info("UV: %2d: 0x%014lx-0x%014lx %5lu%c %3d %04x %02x %02x\n", |
| 1184 | index++, | 1194 | index++, |
| 1185 | (unsigned long)lgre << UV_GAM_RANGE_SHFT, | 1195 | (unsigned long)lgre << UV_GAM_RANGE_SHFT, |
| 1186 | (unsigned long)gre->limit << UV_GAM_RANGE_SHFT, | 1196 | (unsigned long)gre->limit << UV_GAM_RANGE_SHFT, |
| 1187 | ((unsigned long)(gre->limit - lgre)) >> | 1197 | size, suffix[order], |
| 1188 | (30 - UV_GAM_RANGE_SHFT), /* 64M -> 1G */ | ||
| 1189 | gre->type, gre->nasid, gre->sockid, gre->pnode); | 1198 | gre->type, gre->nasid, gre->sockid, gre->pnode); |
| 1190 | 1199 | ||
| 1191 | lgre = gre->limit; | 1200 | lgre = gre->limit; |
| @@ -18,7 +18,7 @@ void foo(void) · 17 unmodified lines | |||
| 18 | OFFSET(CPUINFO_x86, cpuinfo_x86, x86); | 18 | OFFSET(CPUINFO_x86, cpuinfo_x86, x86); |
| 19 | OFFSET(CPUINFO_x86_vendor, cpuinfo_x86, x86_vendor); | 19 | OFFSET(CPUINFO_x86_vendor, cpuinfo_x86, x86_vendor); |
| 20 | OFFSET(CPUINFO_x86_model, cpuinfo_x86, x86_model); | 20 | OFFSET(CPUINFO_x86_model, cpuinfo_x86, x86_model); |
| 21 | OFFSET(CPUINFO_x86_mask, cpuinfo_x86, x86_mask); | 21 | OFFSET(CPUINFO_x86_stepping, cpuinfo_x86, x86_stepping); |
| 22 | OFFSET(CPUINFO_cpuid_level, cpuinfo_x86, cpuid_level); | 22 | OFFSET(CPUINFO_cpuid_level, cpuinfo_x86, cpuid_level); |
| 23 | OFFSET(CPUINFO_x86_capability, cpuinfo_x86, x86_capability); | 23 | OFFSET(CPUINFO_x86_capability, cpuinfo_x86, x86_capability); |
| 24 | OFFSET(CPUINFO_x86_vendor_id, cpuinfo_x86, x86_vendor_id); | 24 | OFFSET(CPUINFO_x86_vendor_id, cpuinfo_x86, x86_vendor_id); |
| @@ -119,7 +119,7 @@ static void init_amd_k6(struct cpuinfo_x86 *c) · 118 unmodified lines | |||
| 119 | return; | 119 | return; |
| 120 | } | 120 | } |
| 121 | 121 | ||
| 122 | if (c->x86_model == 6 && c->x86_mask == 1) { | 122 | if (c->x86_model == 6 && c->x86_stepping == 1) { |
| 123 | const int K6_BUG_LOOP = 1000000; | 123 | const int K6_BUG_LOOP = 1000000; |
| 124 | int n; | 124 | int n; |
| 125 | void (*f_vide)(void); | 125 | void (*f_vide)(void); |
| @@ -149,7 +149,7 @@ static void init_amd_k6(struct cpuinfo_x86 *c) · 23 unmodified lines | |||
| 149 | 149 | ||
| 150 | /* K6 with old style WHCR */ | 150 | /* K6 with old style WHCR */ |
| 151 | if (c->x86_model < 8 || | 151 | if (c->x86_model < 8 || |
| 152 | (c->x86_model == 8 && c->x86_mask < 8)) { | 152 | (c->x86_model == 8 && c->x86_stepping < 8)) { |
| 153 | /* We can only write allocate on the low 508Mb */ | 153 | /* We can only write allocate on the low 508Mb */ |
| 154 | if (mbytes > 508) | 154 | if (mbytes > 508) |
| 155 | mbytes = 508; | 155 | mbytes = 508; |
| @@ -168,7 +168,7 @@ static void init_amd_k6(struct cpuinfo_x86 *c) · 12 unmodified lines | |||
| 168 | return; | 168 | return; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | if ((c->x86_model == 8 && c->x86_mask > 7) || | 171 | if ((c->x86_model == 8 && c->x86_stepping > 7) || |
| 172 | c->x86_model == 9 || c->x86_model == 13) { | 172 | c->x86_model == 9 || c->x86_model == 13) { |
| 173 | /* The more serious chips .. */ | 173 | /* The more serious chips .. */ |
| 174 | 174 | ||
| @@ -221,7 +221,7 @@ static void init_amd_k7(struct cpuinfo_x86 *c) · 46 unmodified lines | |||
| 221 | * are more robust with CLK_CTL set to 200xxxxx instead of 600xxxxx | 221 | * are more robust with CLK_CTL set to 200xxxxx instead of 600xxxxx |
| 222 | * As per AMD technical note 27212 0.2 | 222 | * As per AMD technical note 27212 0.2 |
| 223 | */ | 223 | */ |
| 224 | if ((c->x86_model == 8 && c->x86_mask >= 1) || (c->x86_model > 8)) { | 224 | if ((c->x86_model == 8 && c->x86_stepping >= 1) || (c->x86_model > 8)) { |
| 225 | rdmsr(MSR_K7_CLK_CTL, l, h); | 225 | rdmsr(MSR_K7_CLK_CTL, l, h); |
| 226 | if ((l & 0xfff00000) != 0x20000000) { | 226 | if ((l & 0xfff00000) != 0x20000000) { |
| 227 | pr_info("CPU: CLK_CTL MSR was %x. Reprogramming to %x\n", | 227 | pr_info("CPU: CLK_CTL MSR was %x. Reprogramming to %x\n", |
| @@ -241,12 +241,12 @@ static void init_amd_k7(struct cpuinfo_x86 *c) · 13 unmodified lines | |||
| 241 | * but they are not certified as MP capable. | 241 | * but they are not certified as MP capable. |
| 242 | */ | 242 | */ |
| 243 | /* Athlon 660/661 is valid. */ | 243 | /* Athlon 660/661 is valid. */ |
| 244 | if ((c->x86_model == 6) && ((c->x86_mask == 0) || | 244 | if ((c->x86_model == 6) && ((c->x86_stepping == 0) || |
| 245 | (c->x86_mask == 1))) | 245 | (c->x86_stepping == 1))) |
| 246 | return; | 246 | return; |
| 247 | 247 | ||
| 248 | /* Duron 670 is valid */ | 248 | /* Duron 670 is valid */ |
| 249 | if ((c->x86_model == 7) && (c->x86_mask == 0)) | 249 | if ((c->x86_model == 7) && (c->x86_stepping == 0)) |
| 250 | return; | 250 | return; |
| 251 | 251 | ||
| 252 | /* | 252 | /* |
| @@ -256,8 +256,8 @@ static void init_amd_k7(struct cpuinfo_x86 *c) · 3 unmodified lines | |||
| 256 | * See http://www.heise.de/newsticker/data/jow-18.10.01-000 for | 256 | * See http://www.heise.de/newsticker/data/jow-18.10.01-000 for |
| 257 | * more. | 257 | * more. |
| 258 | */ | 258 | */ |
| 259 | if (((c->x86_model == 6) && (c->x86_mask >= 2)) || | 259 | if (((c->x86_model == 6) && (c->x86_stepping >= 2)) || |
| 260 | ((c->x86_model == 7) && (c->x86_mask >= 1)) || | 260 | ((c->x86_model == 7) && (c->x86_stepping >= 1)) || |
| 261 | (c->x86_model > 7)) | 261 | (c->x86_model > 7)) |
| 262 | if (cpu_has(c, X86_FEATURE_MP)) | 262 | if (cpu_has(c, X86_FEATURE_MP)) |
| 263 | return; | 263 | return; |
| @@ -628,7 +628,7 @@ static void early_init_amd(struct cpuinfo_x86 *c) · 364 unmodified lines | |||
| 628 | /* Set MTRR capability flag if appropriate */ | 628 | /* Set MTRR capability flag if appropriate */ |
| 629 | if (c->x86 == 5) | 629 | if (c->x86 == 5) |
| 630 | if (c->x86_model == 13 || c->x86_model == 9 || | 630 | if (c->x86_model == 13 || c->x86_model == 9 || |
| 631 | (c->x86_model == 8 && c->x86_mask >= 8)) | 631 | (c->x86_model == 8 && c->x86_stepping >= 8)) |
| 632 | set_cpu_cap(c, X86_FEATURE_K6_MTRR); | 632 | set_cpu_cap(c, X86_FEATURE_K6_MTRR); |
| 633 | #endif | 633 | #endif |
| 634 | #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_PCI) | 634 | #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_PCI) |
| @@ -795,7 +795,7 @@ static void init_amd_zn(struct cpuinfo_x86 *c) · 160 unmodified lines | |||
| 795 | * Fix erratum 1076: CPB feature bit not being set in CPUID. It affects | 795 | * Fix erratum 1076: CPB feature bit not being set in CPUID. It affects |
| 796 | * all up to and including B1. | 796 | * all up to and including B1. |
| 797 | */ | 797 | */ |
| 798 | if (c->x86_model <= 1 && c->x86_mask <= 1) | 798 | if (c->x86_model <= 1 && c->x86_stepping <= 1) |
| 799 | set_cpu_cap(c, X86_FEATURE_CPB); | 799 | set_cpu_cap(c, X86_FEATURE_CPB); |
| 800 | } | 800 | } |
| 801 | 801 | ||
| @@ -906,11 +906,11 @@ static unsigned int amd_size_cache(struct cpuinfo_x86 *c, unsigned int size) · 104 unmodified lines | |||
| 906 | /* AMD errata T13 (order #21922) */ | 906 | /* AMD errata T13 (order #21922) */ |
| 907 | if ((c->x86 == 6)) { | 907 | if ((c->x86 == 6)) { |
| 908 | /* Duron Rev A0 */ | 908 | /* Duron Rev A0 */ |
| 909 | if (c->x86_model == 3 && c->x86_mask == 0) | 909 | if (c->x86_model == 3 && c->x86_stepping == 0) |
| 910 | size = 64; | 910 | size = 64; |
| 911 | /* Tbird rev A1/A2 */ | 911 | /* Tbird rev A1/A2 */ |
| 912 | if (c->x86_model == 4 && | 912 | if (c->x86_model == 4 && |
| 913 | (c->x86_mask == 0 || c->x86_mask == 1)) | 913 | (c->x86_stepping == 0 || c->x86_stepping == 1)) |
| 914 | size = 256; | 914 | size = 256; |
| 915 | } | 915 | } |
| 916 | return size; | 916 | return size; |
| @@ -1047,7 +1047,7 @@ static bool cpu_has_amd_erratum(struct cpuinfo_x86 *cpu, const int *erratum) · 130 unmodified lines | |||
| 1047 | } | 1047 | } |
| 1048 | 1048 | ||
| 1049 | /* OSVW unavailable or ID unknown, match family-model-stepping range */ | 1049 | /* OSVW unavailable or ID unknown, match family-model-stepping range */ |
| 1050 | ms = (cpu->x86_model << 4) | cpu->x86_mask; | 1050 | ms = (cpu->x86_model << 4) | cpu->x86_stepping; |
| 1051 | while ((range = *erratum++)) | 1051 | while ((range = *erratum++)) |
| 1052 | if ((cpu->x86 == AMD_MODEL_RANGE_FAMILY(range)) && | 1052 | if ((cpu->x86 == AMD_MODEL_RANGE_FAMILY(range)) && |
| 1053 | (ms >= AMD_MODEL_RANGE_START(range)) && | 1053 | (ms >= AMD_MODEL_RANGE_START(range)) && |
| @@ -162,8 +162,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) · 161 unmodified lines | |||
| 162 | if (cmdline_find_option_bool(boot_command_line, "nospectre_v2")) | 162 | if (cmdline_find_option_bool(boot_command_line, "nospectre_v2")) |
| 163 | return SPECTRE_V2_CMD_NONE; | 163 | return SPECTRE_V2_CMD_NONE; |
| 164 | else { | 164 | else { |
| 165 | ret = cmdline_find_option(boot_command_line, "spectre_v2", arg, | 165 | ret = cmdline_find_option(boot_command_line, "spectre_v2", arg, sizeof(arg)); |
| 166 | sizeof(arg)); | ||
| 167 | if (ret < 0) | 166 | if (ret < 0) |
| 168 | return SPECTRE_V2_CMD_AUTO; | 167 | return SPECTRE_V2_CMD_AUTO; |
| 169 | 168 | ||
| @@ -175,8 +174,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) · 5 unmodified lines | |||
| 175 | } | 174 | } |
| 176 | 175 | ||
| 177 | if (i >= ARRAY_SIZE(mitigation_options)) { | 176 | if (i >= ARRAY_SIZE(mitigation_options)) { |
| 178 | pr_err("unknown option (%s). Switching to AUTO select\n", | 177 | pr_err("unknown option (%s). Switching to AUTO select\n", arg); |
| 179 | mitigation_options[i].option); | ||
| 180 | return SPECTRE_V2_CMD_AUTO; | 178 | return SPECTRE_V2_CMD_AUTO; |
| 181 | } | 179 | } |
| 182 | } | 180 | } |
| @@ -185,8 +183,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) · 2 unmodified lines | |||
| 185 | cmd == SPECTRE_V2_CMD_RETPOLINE_AMD || | 183 | cmd == SPECTRE_V2_CMD_RETPOLINE_AMD || |
| 186 | cmd == SPECTRE_V2_CMD_RETPOLINE_GENERIC) && | 184 | cmd == SPECTRE_V2_CMD_RETPOLINE_GENERIC) && |
| 187 | !IS_ENABLED(CONFIG_RETPOLINE)) { | 185 | !IS_ENABLED(CONFIG_RETPOLINE)) { |
| 188 | pr_err("%s selected but not compiled in. Switching to AUTO select\n", | 186 | pr_err("%s selected but not compiled in. Switching to AUTO select\n", mitigation_options[i].option); |
| 189 | mitigation_options[i].option); | ||
| 190 | return SPECTRE_V2_CMD_AUTO; | 187 | return SPECTRE_V2_CMD_AUTO; |
| 191 | } | 188 | } |
| 192 | 189 | ||
| @@ -256,14 +253,14 @@ static void __init spectre_v2_select_mitigation(void) · 63 unmodified lines | |||
| 256 | goto retpoline_auto; | 253 | goto retpoline_auto; |
| 257 | break; | 254 | break; |
| 258 | } | 255 | } |
| 259 | pr_err("kernel not compiled with retpoline; no mitigation available!"); | 256 | pr_err("Spectre mitigation: kernel not compiled with retpoline; no mitigation available!"); |
| 260 | return; | 257 | return; |
| 261 | 258 | ||
| 262 | retpoline_auto: | 259 | retpoline_auto: |
| 263 | if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) { | 260 | if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) { |
| 264 | retpoline_amd: | 261 | retpoline_amd: |
| 265 | if (!boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) { | 262 | if (!boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) { |
| 266 | pr_err("LFENCE not serializing. Switching to generic retpoline\n"); | 263 | pr_err("Spectre mitigation: LFENCE not serializing, switching to generic retpoline\n"); |
| 267 | goto retpoline_generic; | 264 | goto retpoline_generic; |
| 268 | } | 265 | } |
| 269 | mode = retp_compiler() ? SPECTRE_V2_RETPOLINE_AMD : | 266 | mode = retp_compiler() ? SPECTRE_V2_RETPOLINE_AMD : |
| @@ -281,7 +278,7 @@ retpoline_auto: · 11 unmodified lines | |||
| 281 | pr_info("%s\n", spectre_v2_strings[mode]); | 278 | pr_info("%s\n", spectre_v2_strings[mode]); |
| 282 | 279 | ||
| 283 | /* | 280 | /* |
| 284 | * If neither SMEP or KPTI are available, there is a risk of | 281 | * If neither SMEP nor PTI are available, there is a risk of |
| 285 | * hitting userspace addresses in the RSB after a context switch | 282 | * hitting userspace addresses in the RSB after a context switch |
| 286 | * from a shallow call stack to a deeper one. To prevent this fill | 283 | * from a shallow call stack to a deeper one. To prevent this fill |
| 287 | * the entire RSB, even when using IBRS. | 284 | * the entire RSB, even when using IBRS. |
| @@ -295,21 +292,29 @@ retpoline_auto: · 7 unmodified lines | |||
| 295 | if ((!boot_cpu_has(X86_FEATURE_PTI) && | 292 | if ((!boot_cpu_has(X86_FEATURE_PTI) && |
| 296 | !boot_cpu_has(X86_FEATURE_SMEP)) || is_skylake_era()) { | 293 | !boot_cpu_has(X86_FEATURE_SMEP)) || is_skylake_era()) { |
| 297 | setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); | 294 | setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); |
| 298 | pr_info("Filling RSB on context switch\n"); | 295 | pr_info("Spectre v2 mitigation: Filling RSB on context switch\n"); |
| 299 | } | 296 | } |
| 300 | 297 | ||
| 301 | /* Initialize Indirect Branch Prediction Barrier if supported */ | 298 | /* Initialize Indirect Branch Prediction Barrier if supported */ |
| 302 | if (boot_cpu_has(X86_FEATURE_IBPB)) { | 299 | if (boot_cpu_has(X86_FEATURE_IBPB)) { |
| 303 | setup_force_cpu_cap(X86_FEATURE_USE_IBPB); | 300 | setup_force_cpu_cap(X86_FEATURE_USE_IBPB); |
| 304 | pr_info("Enabling Indirect Branch Prediction Barrier\n"); | 301 | pr_info("Spectre v2 mitigation: Enabling Indirect Branch Prediction Barrier\n"); |
| 302 | } | ||
| 303 | |||
| 304 | /* | ||
| 305 | * Retpoline means the kernel is safe because it has no indirect | ||
| 306 | * branches. But firmware isn't, so use IBRS to protect that. | ||
| 307 | */ | ||
| 308 | if (boot_cpu_has(X86_FEATURE_IBRS)) { | ||
| 309 | setup_force_cpu_cap(X86_FEATURE_USE_IBRS_FW); | ||
| 310 | pr_info("Enabling Restricted Speculation for firmware calls\n"); | ||
| 305 | } | 311 | } |
| 306 | } | 312 | } |
| 307 | 313 | ||
| 308 | #undef pr_fmt | 314 | #undef pr_fmt |
| 309 | 315 | ||
| 310 | #ifdef CONFIG_SYSFS | 316 | #ifdef CONFIG_SYSFS |
| 311 | ssize_t cpu_show_meltdown(struct device *dev, | 317 | ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf) |
| 312 | struct device_attribute *attr, char *buf) | ||
| 313 | { | 318 | { |
| 314 | if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) | 319 | if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) |
| 315 | return sprintf(buf, "Not affected\n"); | 320 | return sprintf(buf, "Not affected\n"); |
| @@ -318,28 +323,21 @@ ssize_t cpu_show_meltdown(struct device *dev, · 2 unmodified lines | |||
| 318 | return sprintf(buf, "Vulnerable\n"); | 323 | return sprintf(buf, "Vulnerable\n"); |
| 319 | } | 324 | } |
| 320 | 325 | ||
| 321 | ssize_t cpu_show_spectre_v1(struct device *dev, | 326 | ssize_t cpu_show_spectre_v1(struct device *dev, struct device_attribute *attr, char *buf) |
| 322 | struct device_attribute *attr, char *buf) | ||
| 323 | { | 327 | { |
| 324 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1)) | 328 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1)) |
| 325 | return sprintf(buf, "Not affected\n"); | 329 | return sprintf(buf, "Not affected\n"); |
| 326 | return sprintf(buf, "Mitigation: __user pointer sanitization\n"); | 330 | return sprintf(buf, "Mitigation: __user pointer sanitization\n"); |
| 327 | } | 331 | } |
| 328 | 332 | ||
| 329 | ssize_t cpu_show_spectre_v2(struct device *dev, | 333 | ssize_t cpu_show_spectre_v2(struct device *dev, struct device_attribute *attr, char *buf) |
| 330 | struct device_attribute *attr, char *buf) | ||
| 331 | { | 334 | { |
| 332 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V2)) | 335 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V2)) |
| 333 | return sprintf(buf, "Not affected\n"); | 336 | return sprintf(buf, "Not affected\n"); |
| 334 | 337 | ||
| 335 | return sprintf(buf, "%s%s%s\n", spectre_v2_strings[spectre_v2_enabled], | 338 | return sprintf(buf, "%s%s%s%s\n", spectre_v2_strings[spectre_v2_enabled], |
| 336 | boot_cpu_has(X86_FEATURE_USE_IBPB) ? ", IBPB" : "", | 339 | boot_cpu_has(X86_FEATURE_USE_IBPB) ? ", IBPB" : "", |
| 340 | boot_cpu_has(X86_FEATURE_USE_IBRS_FW) ? ", IBRS_FW" : "", | ||
| 337 | spectre_v2_module_string()); | 341 | spectre_v2_module_string()); |
| 338 | } | 342 | } |
| 339 | #endif | 343 | #endif |
| 340 | |||
| 341 | void __ibp_barrier(void) | ||
| 342 | { | ||
| 343 | __wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0); | ||
| 344 | } | ||
| 345 | EXPORT_SYMBOL_GPL(__ibp_barrier); | ||
| @@ -140,7 +140,7 @@ static void init_centaur(struct cpuinfo_x86 *c) · 139 unmodified lines | |||
| 140 | clear_cpu_cap(c, X86_FEATURE_TSC); | 140 | clear_cpu_cap(c, X86_FEATURE_TSC); |
| 141 | break; | 141 | break; |
| 142 | case 8: | 142 | case 8: |
| 143 | switch (c->x86_mask) { | 143 | switch (c->x86_stepping) { |
| 144 | default: | 144 | default: |
| 145 | name = "2"; | 145 | name = "2"; |
| 146 | break; | 146 | break; |
| @@ -215,7 +215,7 @@ centaur_size_cache(struct cpuinfo_x86 *c, unsigned int size) · 68 unmodified lines | |||
| 215 | * - Note, it seems this may only be in engineering samples. | 215 | * - Note, it seems this may only be in engineering samples. |
| 216 | */ | 216 | */ |
| 217 | if ((c->x86 == 6) && (c->x86_model == 9) && | 217 | if ((c->x86 == 6) && (c->x86_model == 9) && |
| 218 | (c->x86_mask == 1) && (size == 65)) | 218 | (c->x86_stepping == 1) && (size == 65)) |
| 219 | size -= 1; | 219 | size -= 1; |
| 220 | return size; | 220 | return size; |
| 221 | } | 221 | } |
| @@ -731,7 +731,7 @@ void cpu_detect(struct cpuinfo_x86 *c) · 730 unmodified lines | |||
| 731 | cpuid(0x00000001, &tfms, &misc, &junk, &cap0); | 731 | cpuid(0x00000001, &tfms, &misc, &junk, &cap0); |
| 732 | c->x86 = x86_family(tfms); | 732 | c->x86 = x86_family(tfms); |
| 733 | c->x86_model = x86_model(tfms); | 733 | c->x86_model = x86_model(tfms); |
| 734 | c->x86_mask = x86_stepping(tfms); | 734 | c->x86_stepping = x86_stepping(tfms); |
| 735 | 735 | ||
| 736 | if (cap0 & (1<<19)) { | 736 | if (cap0 & (1<<19)) { |
| 737 | c->x86_clflush_size = ((misc >> 8) & 0xff) * 8; | 737 | c->x86_clflush_size = ((misc >> 8) & 0xff) * 8; |
| @@ -1184,9 +1184,9 @@ static void identify_cpu(struct cpuinfo_x86 *c) · 446 unmodified lines | |||
| 1184 | int i; | 1184 | int i; |
| 1185 | 1185 | ||
| 1186 | c->loops_per_jiffy = loops_per_jiffy; | 1186 | c->loops_per_jiffy = loops_per_jiffy; |
| 1187 | c->x86_cache_size = -1; | 1187 | c->x86_cache_size = 0; |
| 1188 | c->x86_vendor = X86_VENDOR_UNKNOWN; | 1188 | c->x86_vendor = X86_VENDOR_UNKNOWN; |
| 1189 | c->x86_model = c->x86_mask = 0; /* So far unknown... */ | 1189 | c->x86_model = c->x86_stepping = 0; /* So far unknown... */ |
| 1190 | c->x86_vendor_id[0] = '\0'; /* Unset */ | 1190 | c->x86_vendor_id[0] = '\0'; /* Unset */ |
| 1191 | c->x86_model_id[0] = '\0'; /* Unset */ | 1191 | c->x86_model_id[0] = '\0'; /* Unset */ |
| 1192 | c->x86_max_cores = 1; | 1192 | c->x86_max_cores = 1; |
| @@ -1378,8 +1378,8 @@ void print_cpu_info(struct cpuinfo_x86 *c) · 185 unmodified lines | |||
| 1378 | 1378 | ||
| 1379 | pr_cont(" (family: 0x%x, model: 0x%x", c->x86, c->x86_model); | 1379 | pr_cont(" (family: 0x%x, model: 0x%x", c->x86, c->x86_model); |
| 1380 | 1380 | ||
| 1381 | if (c->x86_mask || c->cpuid_level >= 0) | 1381 | if (c->x86_stepping || c->cpuid_level >= 0) |
| 1382 | pr_cont(", stepping: 0x%x)\n", c->x86_mask); | 1382 | pr_cont(", stepping: 0x%x)\n", c->x86_stepping); |
| 1383 | else | 1383 | else |
| 1384 | pr_cont(")\n"); | 1384 | pr_cont(")\n"); |
| 1385 | } | 1385 | } |
| @@ -1749,3 +1749,33 @@ static int __init init_cpu_syscore(void) · 363 unmodified lines | |||
| 1749 | return 0; | 1749 | return 0; |
| 1750 | } | 1750 | } |
| 1751 | core_initcall(init_cpu_syscore); | 1751 | core_initcall(init_cpu_syscore); |
| 1752 | |||
| 1753 | /* | ||
| 1754 | * The microcode loader calls this upon late microcode load to recheck features, | ||
| 1755 | * only when microcode has been updated. Caller holds microcode_mutex and CPU | ||
| 1756 | * hotplug lock. | ||
| 1757 | */ | ||
| 1758 | void microcode_check(void) | ||
| 1759 | { | ||
| 1760 | struct cpuinfo_x86 info; | ||
| 1761 | |||
| 1762 | perf_check_microcode(); | ||
| 1763 | |||
| 1764 | /* Reload CPUID max function as it might've changed. */ | ||
| 1765 | info.cpuid_level = cpuid_eax(0); | ||
| 1766 | |||
| 1767 | /* | ||
| 1768 | * Copy all capability leafs to pick up the synthetic ones so that | ||
| 1769 | * memcmp() below doesn't fail on that. The ones coming from CPUID will | ||
| 1770 | * get overwritten in get_cpu_cap(). | ||
| 1771 | */ | ||
| 1772 | memcpy(&info.x86_capability, &boot_cpu_data.x86_capability, sizeof(info.x86_capability)); | ||
| 1773 | |||
| 1774 | get_cpu_cap(&info); | ||
| 1775 | |||
| 1776 | if (!memcmp(&info.x86_capability, &boot_cpu_data.x86_capability, sizeof(info.x86_capability))) | ||
| 1777 | return; | ||
| 1778 | |||
| 1779 | pr_warn("x86/CPU: CPU features have changed after loading microcode, but might not take effect.\n"); | ||
| 1780 | pr_warn("x86/CPU: Please consider either early loading through initrd/built-in or a potential BIOS update.\n"); | ||
| 1781 | } | ||
| @@ -215,7 +215,7 @@ static void init_cyrix(struct cpuinfo_x86 *c) · 214 unmodified lines | |||
| 215 | 215 | ||
| 216 | /* common case step number/rev -- exceptions handled below */ | 216 | /* common case step number/rev -- exceptions handled below */ |
| 217 | c->x86_model = (dir1 >> 4) + 1; | 217 | c->x86_model = (dir1 >> 4) + 1; |
| 218 | c->x86_mask = dir1 & 0xf; | 218 | c->x86_stepping = dir1 & 0xf; |
| 219 | 219 | ||
| 220 | /* Now cook; the original recipe is by Channing Corn, from Cyrix. | 220 | /* Now cook; the original recipe is by Channing Corn, from Cyrix. |
| 221 | * We do the same thing for each generation: we work out | 221 | * We do the same thing for each generation: we work out |
| @@ -105,7 +105,7 @@ static void probe_xeon_phi_r3mwait(struct cpuinfo_x86 *c) · 104 unmodified lines | |||
| 105 | /* | 105 | /* |
| 106 | * Early microcode releases for the Spectre v2 mitigation were broken. | 106 | * Early microcode releases for the Spectre v2 mitigation were broken. |
| 107 | * Information taken from; | 107 | * Information taken from; |
| 108 | * - https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/microcode-update-guidance.pdf | 108 | * - https://newsroom.intel.com/wp-content/uploads/sites/11/2018/03/microcode-update-guidance.pdf |
| 109 | * - https://kb.vmware.com/s/article/52345 | 109 | * - https://kb.vmware.com/s/article/52345 |
| 110 | * - Microcode revisions observed in the wild | 110 | * - Microcode revisions observed in the wild |
| 111 | * - Release note from 20180108 microcode release | 111 | * - Release note from 20180108 microcode release |
| @@ -116,15 +116,13 @@ struct sku_microcode { · 4 unmodified lines | |||
| 116 | u32 microcode; | 116 | u32 microcode; |
| 117 | }; | 117 | }; |
| 118 | static const struct sku_microcode spectre_bad_microcodes[] = { | 118 | static const struct sku_microcode spectre_bad_microcodes[] = { |
| 119 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0B, 0x84 }, | 119 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0B, 0x80 }, |
| 120 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0A, 0x84 }, | 120 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0A, 0x80 }, |
| 121 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x09, 0x84 }, | 121 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x09, 0x80 }, |
| 122 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x0A, 0x84 }, | 122 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x0A, 0x80 }, |
| 123 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x09, 0x84 }, | 123 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x09, 0x80 }, |
| 124 | { INTEL_FAM6_SKYLAKE_X, 0x03, 0x0100013e }, | 124 | { INTEL_FAM6_SKYLAKE_X, 0x03, 0x0100013e }, |
| 125 | { INTEL_FAM6_SKYLAKE_X, 0x04, 0x0200003c }, | 125 | { INTEL_FAM6_SKYLAKE_X, 0x04, 0x0200003c }, |
| 126 | { INTEL_FAM6_SKYLAKE_MOBILE, 0x03, 0xc2 }, | ||
| 127 | { INTEL_FAM6_SKYLAKE_DESKTOP, 0x03, 0xc2 }, | ||
| 128 | { INTEL_FAM6_BROADWELL_CORE, 0x04, 0x28 }, | 126 | { INTEL_FAM6_BROADWELL_CORE, 0x04, 0x28 }, |
| 129 | { INTEL_FAM6_BROADWELL_GT3E, 0x01, 0x1b }, | 127 | { INTEL_FAM6_BROADWELL_GT3E, 0x01, 0x1b }, |
| 130 | { INTEL_FAM6_BROADWELL_XEON_D, 0x02, 0x14 }, | 128 | { INTEL_FAM6_BROADWELL_XEON_D, 0x02, 0x14 }, |
| @@ -136,8 +134,6 @@ static const struct sku_microcode spectre_bad_microcodes[] = { · 5 unmodified lines | |||
| 136 | { INTEL_FAM6_HASWELL_X, 0x02, 0x3b }, | 134 | { INTEL_FAM6_HASWELL_X, 0x02, 0x3b }, |
| 137 | { INTEL_FAM6_HASWELL_X, 0x04, 0x10 }, | 135 | { INTEL_FAM6_HASWELL_X, 0x04, 0x10 }, |
| 138 | { INTEL_FAM6_IVYBRIDGE_X, 0x04, 0x42a }, | 136 | { INTEL_FAM6_IVYBRIDGE_X, 0x04, 0x42a }, |
| 139 | /* Updated in the 20180108 release; blacklist until we know otherwise */ | ||
| 140 | { INTEL_FAM6_ATOM_GEMINI_LAKE, 0x01, 0x22 }, | ||
| 141 | /* Observed in the wild */ | 137 | /* Observed in the wild */ |
| 142 | { INTEL_FAM6_SANDYBRIDGE_X, 0x06, 0x61b }, | 138 | { INTEL_FAM6_SANDYBRIDGE_X, 0x06, 0x61b }, |
| 143 | { INTEL_FAM6_SANDYBRIDGE_X, 0x07, 0x712 }, | 139 | { INTEL_FAM6_SANDYBRIDGE_X, 0x07, 0x712 }, |
| @@ -147,9 +143,16 @@ static bool bad_spectre_microcode(struct cpuinfo_x86 *c) · 3 unmodified lines | |||
| 147 | { | 143 | { |
| 148 | int i; | 144 | int i; |
| 149 | 145 | ||
| 146 | /* | ||
| 147 | * We know that the hypervisor lie to us on the microcode version so | ||
| 148 | * we may as well hope that it is running the correct version. | ||
| 149 | */ | ||
| 150 | if (cpu_has(c, X86_FEATURE_HYPERVISOR)) | ||
| 151 | return false; | ||
| 152 | |||
| 150 | for (i = 0; i < ARRAY_SIZE(spectre_bad_microcodes); i++) { | 153 | for (i = 0; i < ARRAY_SIZE(spectre_bad_microcodes); i++) { |
| 151 | if (c->x86_model == spectre_bad_microcodes[i].model && | 154 | if (c->x86_model == spectre_bad_microcodes[i].model && |
| 152 | c->x86_mask == spectre_bad_microcodes[i].stepping) | 155 | c->x86_stepping == spectre_bad_microcodes[i].stepping) |
| 153 | return (c->microcode <= spectre_bad_microcodes[i].microcode); | 156 | return (c->microcode <= spectre_bad_microcodes[i].microcode); |
| 154 | } | 157 | } |
| 155 | return false; | 158 | return false; |
| @@ -196,7 +199,7 @@ static void early_init_intel(struct cpuinfo_x86 *c) · 40 unmodified lines | |||
| 196 | * need the microcode to have already been loaded... so if it is | 199 | * need the microcode to have already been loaded... so if it is |
| 197 | * not, recommend a BIOS update and disable large pages. | 200 | * not, recommend a BIOS update and disable large pages. |
| 198 | */ | 201 | */ |
| 199 | if (c->x86 == 6 && c->x86_model == 0x1c && c->x86_mask <= 2 && | 202 | if (c->x86 == 6 && c->x86_model == 0x1c && c->x86_stepping <= 2 && |
| 200 | c->microcode < 0x20e) { | 203 | c->microcode < 0x20e) { |
| 201 | pr_warn("Atom PSE erratum detected, BIOS microcode update recommended\n"); | 204 | pr_warn("Atom PSE erratum detected, BIOS microcode update recommended\n"); |
| 202 | clear_cpu_cap(c, X86_FEATURE_PSE); | 205 | clear_cpu_cap(c, X86_FEATURE_PSE); |
| @@ -212,7 +215,7 @@ static void early_init_intel(struct cpuinfo_x86 *c) · 9 unmodified lines | |||
| 212 | 215 | ||
| 213 | /* CPUID workaround for 0F33/0F34 CPU */ | 216 | /* CPUID workaround for 0F33/0F34 CPU */ |
| 214 | if (c->x86 == 0xF && c->x86_model == 0x3 | 217 | if (c->x86 == 0xF && c->x86_model == 0x3 |
| 215 | && (c->x86_mask == 0x3 || c->x86_mask == 0x4)) | 218 | && (c->x86_stepping == 0x3 || c->x86_stepping == 0x4)) |
| 216 | c->x86_phys_bits = 36; | 219 | c->x86_phys_bits = 36; |
| 217 | 220 | ||
| 218 | /* | 221 | /* |
| @@ -310,7 +313,7 @@ int ppro_with_ram_bug(void) · 91 unmodified lines | |||
| 310 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && | 313 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && |
| 311 | boot_cpu_data.x86 == 6 && | 314 | boot_cpu_data.x86 == 6 && |
| 312 | boot_cpu_data.x86_model == 1 && | 315 | boot_cpu_data.x86_model == 1 && |
| 313 | boot_cpu_data.x86_mask < 8) { | 316 | boot_cpu_data.x86_stepping < 8) { |
| 314 | pr_info("Pentium Pro with Errata#50 detected. Taking evasive action.\n"); | 317 | pr_info("Pentium Pro with Errata#50 detected. Taking evasive action.\n"); |
| 315 | return 1; | 318 | return 1; |
| 316 | } | 319 | } |
| @@ -327,7 +330,7 @@ static void intel_smp_check(struct cpuinfo_x86 *c) · 10 unmodified lines | |||
| 327 | * Mask B, Pentium, but not Pentium MMX | 330 | * Mask B, Pentium, but not Pentium MMX |
| 328 | */ | 331 | */ |
| 329 | if (c->x86 == 5 && | 332 | if (c->x86 == 5 && |
| 330 | c->x86_mask >= 1 && c->x86_mask <= 4 && | 333 | c->x86_stepping >= 1 && c->x86_stepping <= 4 && |
| 331 | c->x86_model <= 3) { | 334 | c->x86_model <= 3) { |
| 332 | /* | 335 | /* |
| 333 | * Remember we have B step Pentia with bugs | 336 | * Remember we have B step Pentia with bugs |
| @@ -370,7 +373,7 @@ static void intel_workarounds(struct cpuinfo_x86 *c) · 36 unmodified lines | |||
| 370 | * SEP CPUID bug: Pentium Pro reports SEP but doesn't have it until | 373 | * SEP CPUID bug: Pentium Pro reports SEP but doesn't have it until |
| 371 | * model 3 mask 3 | 374 | * model 3 mask 3 |
| 372 | */ | 375 | */ |
| 373 | if ((c->x86<<8 | c->x86_model<<4 | c->x86_mask) < 0x633) | 376 | if ((c->x86<<8 | c->x86_model<<4 | c->x86_stepping) < 0x633) |
| 374 | clear_cpu_cap(c, X86_FEATURE_SEP); | 377 | clear_cpu_cap(c, X86_FEATURE_SEP); |
| 375 | 378 | ||
| 376 | /* | 379 | /* |
| @@ -388,7 +391,7 @@ static void intel_workarounds(struct cpuinfo_x86 *c) · 11 unmodified lines | |||
| 388 | * P4 Xeon erratum 037 workaround. | 391 | * P4 Xeon erratum 037 workaround. |
| 389 | * Hardware prefetcher may cause stale data to be loaded into the cache. | 392 | * Hardware prefetcher may cause stale data to be loaded into the cache. |
| 390 | */ | 393 | */ |
| 391 | if ((c->x86 == 15) && (c->x86_model == 1) && (c->x86_mask == 1)) { | 394 | if ((c->x86 == 15) && (c->x86_model == 1) && (c->x86_stepping == 1)) { |
| 392 | if (msr_set_bit(MSR_IA32_MISC_ENABLE, | 395 | if (msr_set_bit(MSR_IA32_MISC_ENABLE, |
| 393 | MSR_IA32_MISC_ENABLE_PREFETCH_DISABLE_BIT) > 0) { | 396 | MSR_IA32_MISC_ENABLE_PREFETCH_DISABLE_BIT) > 0) { |
| 394 | pr_info("CPU: C0 stepping P4 Xeon detected.\n"); | 397 | pr_info("CPU: C0 stepping P4 Xeon detected.\n"); |
| @@ -403,7 +406,7 @@ static void intel_workarounds(struct cpuinfo_x86 *c) · 8 unmodified lines | |||
| 403 | * Specification Update"). | 406 | * Specification Update"). |
| 404 | */ | 407 | */ |
| 405 | if (boot_cpu_has(X86_FEATURE_APIC) && (c->x86<<8 | c->x86_model<<4) == 0x520 && | 408 | if (boot_cpu_has(X86_FEATURE_APIC) && (c->x86<<8 | c->x86_model<<4) == 0x520 && |
| 406 | (c->x86_mask < 0x6 || c->x86_mask == 0xb)) | 409 | (c->x86_stepping < 0x6 || c->x86_stepping == 0xb)) |
| 407 | set_cpu_bug(c, X86_BUG_11AP); | 410 | set_cpu_bug(c, X86_BUG_11AP); |
| 408 | 411 | ||
| 409 | 412 | ||
| @@ -650,7 +653,7 @@ static void init_intel(struct cpuinfo_x86 *c) · 240 unmodified lines | |||
| 650 | case 6: | 653 | case 6: |
| 651 | if (l2 == 128) | 654 | if (l2 == 128) |
| 652 | p = "Celeron (Mendocino)"; | 655 | p = "Celeron (Mendocino)"; |
| 653 | else if (c->x86_mask == 0 || c->x86_mask == 5) | 656 | else if (c->x86_stepping == 0 || c->x86_stepping == 5) |
| 654 | p = "Celeron-A"; | 657 | p = "Celeron-A"; |
| 655 | break; | 658 | break; |
| 656 | 659 | ||
| @@ -819,7 +819,7 @@ static __init void rdt_quirks(void) · 818 unmodified lines | |||
| 819 | cache_alloc_hsw_probe(); | 819 | cache_alloc_hsw_probe(); |
| 820 | break; | 820 | break; |
| 821 | case INTEL_FAM6_SKYLAKE_X: | 821 | case INTEL_FAM6_SKYLAKE_X: |
| 822 | if (boot_cpu_data.x86_mask <= 4) | 822 | if (boot_cpu_data.x86_stepping <= 4) |
| 823 | set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat"); | 823 | set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat"); |
| 824 | } | 824 | } |
| 825 | } | 825 | } |
| @@ -1804,6 +1804,7 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, · 1803 unmodified lines | |||
| 1804 | goto out_common_fail; | 1804 | goto out_common_fail; |
| 1805 | } | 1805 | } |
| 1806 | closid = ret; | 1806 | closid = ret; |
| 1807 | ret = 0; | ||
| 1807 | 1808 | ||
| 1808 | rdtgrp->closid = closid; | 1809 | rdtgrp->closid = closid; |
| 1809 | list_add(&rdtgrp->rdtgroup_list, &rdt_all_groups); | 1810 | list_add(&rdtgrp->rdtgroup_list, &rdt_all_groups); |
| @@ -115,4 +115,19 @@ static inline void mce_unregister_injector_chain(struct notifier_block *nb) { } · 114 unmodified lines | |||
| 115 | 115 | ||
| 116 | extern struct mca_config mca_cfg; | 116 | extern struct mca_config mca_cfg; |
| 117 | 117 | ||
| 118 | #ifndef CONFIG_X86_64 | ||
| 119 | /* | ||
| 120 | * On 32-bit systems it would be difficult to safely unmap a poison page | ||
| 121 | * from the kernel 1:1 map because there are no non-canonical addresses that | ||
| 122 | * we can use to refer to the address without risking a speculative access. | ||
| 123 | * However, this isn't much of an issue because: | ||
| 124 | * 1) Few unmappable pages are in the 1:1 map. Most are in HIGHMEM which | ||
| 125 | * are only mapped into the kernel as needed | ||
| 126 | * 2) Few people would run a 32-bit kernel on a machine that supports | ||
| 127 | * recoverable errors because they have too much memory to boot 32-bit. | ||
| 128 | */ | ||
| 129 | static inline void mce_unmap_kpfn(unsigned long pfn) {} | ||
| 130 | #define mce_unmap_kpfn mce_unmap_kpfn | ||
| 131 | #endif | ||
| 132 | |||
| 118 | #endif /* __X86_MCE_INTERNAL_H__ */ | 133 | #endif /* __X86_MCE_INTERNAL_H__ */ |
| @@ -56,6 +56,9 @@ · 55 unmodified lines | |||
| 56 | 56 | ||
| 57 | static DEFINE_MUTEX(mce_log_mutex); | 57 | static DEFINE_MUTEX(mce_log_mutex); |
| 58 | 58 | ||
| 59 | /* sysfs synchronization */ | ||
| 60 | static DEFINE_MUTEX(mce_sysfs_mutex); | ||
| 61 | |||
| 59 | #define CREATE_TRACE_POINTS | 62 | #define CREATE_TRACE_POINTS |
| 60 | #include <trace/events/mce.h> | 63 | #include <trace/events/mce.h> |
| 61 | 64 | ||
| @@ -105,6 +108,10 @@ static struct irq_work mce_irq_work; · 43 unmodified lines | |||
| 105 | 108 | ||
| 106 | static void (*quirk_no_way_out)(int bank, struct mce *m, struct pt_regs *regs); | 109 | static void (*quirk_no_way_out)(int bank, struct mce *m, struct pt_regs *regs); |
| 107 | 110 | ||
| 111 | #ifndef mce_unmap_kpfn | ||
| 112 | static void mce_unmap_kpfn(unsigned long pfn); | ||
| 113 | #endif | ||
| 114 | |||
| 108 | /* | 115 | /* |
| 109 | * CPU/chipset specific EDAC code can register a notifier call here to print | 116 | * CPU/chipset specific EDAC code can register a notifier call here to print |
| 110 | * MCE errors in a human-readable form. | 117 | * MCE errors in a human-readable form. |
| @@ -126,6 +133,8 @@ void mce_setup(struct mce *m) · 15 unmodified lines | |||
| 126 | 133 | ||
| 127 | if (this_cpu_has(X86_FEATURE_INTEL_PPIN)) | 134 | if (this_cpu_has(X86_FEATURE_INTEL_PPIN)) |
| 128 | rdmsrl(MSR_PPIN, m->ppin); | 135 | rdmsrl(MSR_PPIN, m->ppin); |
| 136 | |||
| 137 | m->microcode = boot_cpu_data.microcode; | ||
| 129 | } | 138 | } |
| 130 | 139 | ||
| 131 | DEFINE_PER_CPU(struct mce, injectm); | 140 | DEFINE_PER_CPU(struct mce, injectm); |
| @@ -234,7 +243,7 @@ static void __print_mce(struct mce *m) · 102 unmodified lines | |||
| 234 | m->cs, m->ip); | 243 | m->cs, m->ip); |
| 235 | 244 | ||
| 236 | if (m->cs == __KERNEL_CS) | 245 | if (m->cs == __KERNEL_CS) |
| 237 | pr_cont("{%pS}", (void *)m->ip); | 246 | pr_cont("{%pS}", (void *)(unsigned long)m->ip); |
| 238 | pr_cont("\n"); | 247 | pr_cont("\n"); |
| 239 | } | 248 | } |
| 240 | 249 | ||
| @@ -258,7 +267,7 @@ static void __print_mce(struct mce *m) · 17 unmodified lines | |||
| 258 | */ | 267 | */ |
| 259 | pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x microcode %x\n", | 268 | pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x microcode %x\n", |
| 260 | m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid, | 269 | m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid, |
| 261 | cpu_data(m->extcpu).microcode); | 270 | m->microcode); |
| 262 | } | 271 | } |
| 263 | 272 | ||
| 264 | static void print_mce(struct mce *m) | 273 | static void print_mce(struct mce *m) |
| @@ -590,7 +599,8 @@ static int srao_decode_notifier(struct notifier_block *nb, unsigned long val, · 325 unmodified lines | |||
| 590 | 599 | ||
| 591 | if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { | 600 | if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { |
| 592 | pfn = mce->addr >> PAGE_SHIFT; | 601 | pfn = mce->addr >> PAGE_SHIFT; |
| 593 | memory_failure(pfn, 0); | 602 | if (!memory_failure(pfn, 0)) |
| 603 | mce_unmap_kpfn(pfn); | ||
| 594 | } | 604 | } |
| 595 | 605 | ||
| 596 | return NOTIFY_OK; | 606 | return NOTIFY_OK; |
| @@ -1057,12 +1067,13 @@ static int do_memory_failure(struct mce *m) · 460 unmodified lines | |||
| 1057 | ret = memory_failure(m->addr >> PAGE_SHIFT, flags); | 1067 | ret = memory_failure(m->addr >> PAGE_SHIFT, flags); |
| 1058 | if (ret) | 1068 | if (ret) |
| 1059 | pr_err("Memory error not recovered"); | 1069 | pr_err("Memory error not recovered"); |
| 1070 | else | ||
| 1071 | mce_unmap_kpfn(m->addr >> PAGE_SHIFT); | ||
| 1060 | return ret; | 1072 | return ret; |
| 1061 | } | 1073 | } |
| 1062 | 1074 | ||
| 1063 | #if defined(arch_unmap_kpfn) && defined(CONFIG_MEMORY_FAILURE) | 1075 | #ifndef mce_unmap_kpfn |
| 1064 | 1076 | static void mce_unmap_kpfn(unsigned long pfn) | |
| 1065 | void arch_unmap_kpfn(unsigned long pfn) | ||
| 1066 | { | 1077 | { |
| 1067 | unsigned long decoy_addr; | 1078 | unsigned long decoy_addr; |
| 1068 | 1079 | ||
| @@ -1073,7 +1084,7 @@ void arch_unmap_kpfn(unsigned long pfn) · 4 unmodified lines | |||
| 1073 | * We would like to just call: | 1084 | * We would like to just call: |
| 1074 | * set_memory_np((unsigned long)pfn_to_kaddr(pfn), 1); | 1085 | * set_memory_np((unsigned long)pfn_to_kaddr(pfn), 1); |
| 1075 | * but doing that would radically increase the odds of a | 1086 | * but doing that would radically increase the odds of a |
| 1076 | * speculative access to the posion page because we'd have | 1087 | * speculative access to the poison page because we'd have |
| 1077 | * the virtual address of the kernel 1:1 mapping sitting | 1088 | * the virtual address of the kernel 1:1 mapping sitting |
| 1078 | * around in registers. | 1089 | * around in registers. |
| 1079 | * Instead we get tricky. We create a non-canonical address | 1090 | * Instead we get tricky. We create a non-canonical address |
| @@ -1098,7 +1109,6 @@ void arch_unmap_kpfn(unsigned long pfn) · 18 unmodified lines | |||
| 1098 | 1109 | ||
| 1099 | if (set_memory_np(decoy_addr, 1)) | 1110 | if (set_memory_np(decoy_addr, 1)) |
| 1100 | pr_warn("Could not invalidate pfn=0x%lx from 1:1 map\n", pfn); | 1111 | pr_warn("Could not invalidate pfn=0x%lx from 1:1 map\n", pfn); |
| 1101 | |||
| 1102 | } | 1112 | } |
| 1103 | #endif | 1113 | #endif |
| 1104 | 1114 | ||
| @@ -2081,6 +2091,7 @@ static ssize_t set_ignore_ce(struct device *s, · 976 unmodified lines | |||
| 2081 | if (kstrtou64(buf, 0, &new) < 0) | 2091 | if (kstrtou64(buf, 0, &new) < 0) |
| 2082 | return -EINVAL; | 2092 | return -EINVAL; |
| 2083 | 2093 | ||
| 2094 | mutex_lock(&mce_sysfs_mutex); | ||
| 2084 | if (mca_cfg.ignore_ce ^ !!new) { | 2095 | if (mca_cfg.ignore_ce ^ !!new) { |
| 2085 | if (new) { | 2096 | if (new) { |
| 2086 | /* disable ce features */ | 2097 | /* disable ce features */ |
| @@ -2093,6 +2104,8 @@ static ssize_t set_ignore_ce(struct device *s, · 6 unmodified lines | |||
| 2093 | on_each_cpu(mce_enable_ce, (void *)1, 1); | 2104 | on_each_cpu(mce_enable_ce, (void *)1, 1); |
| 2094 | } | 2105 | } |
| 2095 | } | 2106 | } |
| 2107 | mutex_unlock(&mce_sysfs_mutex); | ||
| 2108 | |||
| 2096 | return size; | 2109 | return size; |
| 2097 | } | 2110 | } |
| 2098 | 2111 | ||
| @@ -2105,6 +2118,7 @@ static ssize_t set_cmci_disabled(struct device *s, · 6 unmodified lines | |||
| 2105 | if (kstrtou64(buf, 0, &new) < 0) | 2118 | if (kstrtou64(buf, 0, &new) < 0) |
| 2106 | return -EINVAL; | 2119 | return -EINVAL; |
| 2107 | 2120 | ||
| 2121 | mutex_lock(&mce_sysfs_mutex); | ||
| 2108 | if (mca_cfg.cmci_disabled ^ !!new) { | 2122 | if (mca_cfg.cmci_disabled ^ !!new) { |
| 2109 | if (new) { | 2123 | if (new) { |
| 2110 | /* disable cmci */ | 2124 | /* disable cmci */ |
| @@ -2116,6 +2130,8 @@ static ssize_t set_cmci_disabled(struct device *s, · 5 unmodified lines | |||
| 2116 | on_each_cpu(mce_enable_ce, NULL, 1); | 2130 | on_each_cpu(mce_enable_ce, NULL, 1); |
| 2117 | } | 2131 | } |
| 2118 | } | 2132 | } |
| 2133 | mutex_unlock(&mce_sysfs_mutex); | ||
| 2134 | |||
| 2119 | return size; | 2135 | return size; |
| 2120 | } | 2136 | } |
| 2121 | 2137 | ||
| @@ -2123,8 +2139,19 @@ static ssize_t store_int_with_restart(struct device *s, · 1 unmodified line | |||
| 2123 | struct device_attribute *attr, | 2139 | struct device_attribute *attr, |
| 2124 | const char *buf, size_t size) | 2140 | const char *buf, size_t size) |
| 2125 | { | 2141 | { |
| 2126 | ssize_t ret = device_store_int(s, attr, buf, size); | 2142 | unsigned long old_check_interval = check_interval; |
| 2143 | ssize_t ret = device_store_ulong(s, attr, buf, size); | ||
| 2144 | |||
| 2145 | if (check_interval == old_check_interval) | ||
| 2146 | return ret; | ||
| 2147 | |||
| 2148 | if (check_interval < 1) | ||
| 2149 | check_interval = 1; | ||
| 2150 | |||
| 2151 | mutex_lock(&mce_sysfs_mutex); | ||
| 2127 | mce_restart(); | 2152 | mce_restart(); |
| 2153 | mutex_unlock(&mce_sysfs_mutex); | ||
| 2154 | |||
| 2128 | return ret; | 2155 | return ret; |
| 2129 | } | 2156 | } |
| 2130 | 2157 | ||
| @@ -339,7 +339,7 @@ int __init save_microcode_in_initrd_amd(unsigned int cpuid_1_eax) · 338 unmodified lines | |||
| 339 | return -EINVAL; | 339 | return -EINVAL; |
| 340 | 340 | ||
| 341 | ret = load_microcode_amd(true, x86_family(cpuid_1_eax), desc.data, desc.size); | 341 | ret = load_microcode_amd(true, x86_family(cpuid_1_eax), desc.data, desc.size); |
| 342 | if (ret != UCODE_OK) | 342 | if (ret > UCODE_UPDATED) |
| 343 | return -EINVAL; | 343 | return -EINVAL; |
| 344 | 344 | ||
| 345 | return 0; | 345 | return 0; |
| @@ -498,7 +498,7 @@ static unsigned int verify_patch_size(u8 family, u32 patch_size, · 152 unmodified lines | |||
| 498 | return patch_size; | 498 | return patch_size; |
| 499 | } | 499 | } |
| 500 | 500 | ||
| 501 | static int apply_microcode_amd(int cpu) | 501 | static enum ucode_state apply_microcode_amd(int cpu) |
| 502 | { | 502 | { |
| 503 | struct cpuinfo_x86 *c = &cpu_data(cpu); | 503 | struct cpuinfo_x86 *c = &cpu_data(cpu); |
| 504 | struct microcode_amd *mc_amd; | 504 | struct microcode_amd *mc_amd; |
| @@ -512,7 +512,7 @@ static int apply_microcode_amd(int cpu) · 7 unmodified lines | |||
| 512 | 512 | ||
| 513 | p = find_patch(cpu); | 513 | p = find_patch(cpu); |
| 514 | if (!p) | 514 | if (!p) |
| 515 | return 0; | 515 | return UCODE_NFOUND; |
| 516 | 516 | ||
| 517 | mc_amd = p->data; | 517 | mc_amd = p->data; |
| 518 | uci->mc = p->data; | 518 | uci->mc = p->data; |
| @@ -523,13 +523,13 @@ static int apply_microcode_amd(int cpu) · 4 unmodified lines | |||
| 523 | if (rev >= mc_amd->hdr.patch_id) { | 523 | if (rev >= mc_amd->hdr.patch_id) { |
| 524 | c->microcode = rev; | 524 | c->microcode = rev; |
| 525 | uci->cpu_sig.rev = rev; | 525 | uci->cpu_sig.rev = rev; |
| 526 | return 0; | 526 | return UCODE_OK; |
| 527 | } | 527 | } |
| 528 | 528 | ||
| 529 | if (__apply_microcode_amd(mc_amd)) { | 529 | if (__apply_microcode_amd(mc_amd)) { |
| 530 | pr_err("CPU%d: update failed for patch_level=0x%08x\n", | 530 | pr_err("CPU%d: update failed for patch_level=0x%08x\n", |
| 531 | cpu, mc_amd->hdr.patch_id); | 531 | cpu, mc_amd->hdr.patch_id); |
| 532 | return -1; | 532 | return UCODE_ERROR; |
| 533 | } | 533 | } |
| 534 | pr_info("CPU%d: new patch_level=0x%08x\n", cpu, | 534 | pr_info("CPU%d: new patch_level=0x%08x\n", cpu, |
| 535 | mc_amd->hdr.patch_id); | 535 | mc_amd->hdr.patch_id); |
| @@ -537,7 +537,7 @@ static int apply_microcode_amd(int cpu) · 1 unmodified line | |||
| 537 | uci->cpu_sig.rev = mc_amd->hdr.patch_id; | 537 | uci->cpu_sig.rev = mc_amd->hdr.patch_id; |
| 538 | c->microcode = mc_amd->hdr.patch_id; | 538 | c->microcode = mc_amd->hdr.patch_id; |
| 539 | 539 | ||
| 540 | return 0; | 540 | return UCODE_UPDATED; |
| 541 | } | 541 | } |
| 542 | 542 | ||
| 543 | static int install_equiv_cpu_table(const u8 *buf) | 543 | static int install_equiv_cpu_table(const u8 *buf) |
| @@ -683,27 +683,35 @@ static enum ucode_state __load_microcode_amd(u8 family, const u8 *data, · 139 unmodified lines | |||
| 683 | static enum ucode_state | 683 | static enum ucode_state |
| 684 | load_microcode_amd(bool save, u8 family, const u8 *data, size_t size) | 684 | load_microcode_amd(bool save, u8 family, const u8 *data, size_t size) |
| 685 | { | 685 | { |
| 686 | struct ucode_patch *p; | ||
| 686 | enum ucode_state ret; | 687 | enum ucode_state ret; |
| 687 | 688 | ||
| 688 | /* free old equiv table */ | 689 | /* free old equiv table */ |
| 689 | free_equiv_cpu_table(); | 690 | free_equiv_cpu_table(); |
| 690 | 691 | ||
| 691 | ret = __load_microcode_amd(family, data, size); | 692 | ret = __load_microcode_amd(family, data, size); |
| 692 | 693 | if (ret != UCODE_OK) { | |
| 693 | if (ret != UCODE_OK) | ||
| 694 | cleanup(); | 694 | cleanup(); |
| 695 | return ret; | ||
| 696 | } | ||
| 695 | 697 | ||
| 696 | #ifdef CONFIG_X86_32 | 698 | p = find_patch(0); |
| 697 | /* save BSP's matching patch for early load */ | 699 | if (!p) { |
| 698 | if (save) { | 700 | return ret; |
| 699 | struct ucode_patch *p = find_patch(0); | 701 | } else { |
| 700 | if (p) { | 702 | if (boot_cpu_data.microcode == p->patch_id) |
| 701 | memset(amd_ucode_patch, 0, PATCH_MAX_SIZE); | 703 | return ret; |
| 702 | memcpy(amd_ucode_patch, p->data, min_t(u32, ksize(p->data), | 704 | |
| 703 | PATCH_MAX_SIZE)); | 705 | ret = UCODE_NEW; |
| 704 | } | ||
| 705 | } | 706 | } |
| 706 | #endif | 707 | |
| 708 | /* save BSP's matching patch for early load */ | ||
| 709 | if (!save) | ||
| 710 | return ret; | ||
| 711 | |||
| 712 | memset(amd_ucode_patch, 0, PATCH_MAX_SIZE); | ||
| 713 | memcpy(amd_ucode_patch, p->data, min_t(u32, ksize(p->data), PATCH_MAX_SIZE)); | ||
| 714 | |||
| 707 | return ret; | 715 | return ret; |
| 708 | } | 716 | } |
| 709 | 717 | ||
| @@ -22,13 +22,16 @@ · 21 unmodified lines | |||
| 22 | #define pr_fmt(fmt) "microcode: " fmt | 22 | #define pr_fmt(fmt) "microcode: " fmt |
| 23 | 23 | ||
| 24 | #include <linux/platform_device.h> | 24 | #include <linux/platform_device.h> |
| 25 | #include <linux/stop_machine.h> | ||
| 25 | #include <linux/syscore_ops.h> | 26 | #include <linux/syscore_ops.h> |
| 26 | #include <linux/miscdevice.h> | 27 | #include <linux/miscdevice.h> |
| 27 | #include <linux/capability.h> | 28 | #include <linux/capability.h> |
| 28 | #include <linux/firmware.h> | 29 | #include <linux/firmware.h> |
| 29 | #include <linux/kernel.h> | 30 | #include <linux/kernel.h> |
| 31 | #include <linux/delay.h> | ||
| 30 | #include <linux/mutex.h> | 32 | #include <linux/mutex.h> |
| 31 | #include <linux/cpu.h> | 33 | #include <linux/cpu.h> |
| 34 | #include <linux/nmi.h> | ||
| 32 | #include <linux/fs.h> | 35 | #include <linux/fs.h> |
| 33 | #include <linux/mm.h> | 36 | #include <linux/mm.h> |
| 34 | 37 | ||
| @@ -64,6 +67,11 @@ LIST_HEAD(microcode_cache); · 29 unmodified lines | |||
| 64 | */ | 67 | */ |
| 65 | static DEFINE_MUTEX(microcode_mutex); | 68 | static DEFINE_MUTEX(microcode_mutex); |
| 66 | 69 | ||
| 70 | /* | ||
| 71 | * Serialize late loading so that CPUs get updated one-by-one. | ||
| 72 | */ | ||
| 73 | static DEFINE_SPINLOCK(update_lock); | ||
| 74 | |||
| 67 | struct ucode_cpu_info ucode_cpu_info[NR_CPUS]; | 75 | struct ucode_cpu_info ucode_cpu_info[NR_CPUS]; |
| 68 | 76 | ||
| 69 | struct cpu_info_ctx { | 77 | struct cpu_info_ctx { |
| @@ -373,26 +381,23 @@ static int collect_cpu_info(int cpu) · 303 unmodified lines | |||
| 373 | return ret; | 381 | return ret; |
| 374 | } | 382 | } |
| 375 | 383 | ||
| 376 | struct apply_microcode_ctx { | ||
| 377 | int err; | ||
| 378 | }; | ||
| 379 | |||
| 380 | static void apply_microcode_local(void *arg) | 384 | static void apply_microcode_local(void *arg) |
| 381 | { | 385 | { |
| 382 | struct apply_microcode_ctx *ctx = arg; | 386 | enum ucode_state *err = arg; |
| 383 | 387 | ||
| 384 | ctx->err = microcode_ops->apply_microcode(smp_processor_id()); | 388 | *err = microcode_ops->apply_microcode(smp_processor_id()); |
| 385 | } | 389 | } |
| 386 | 390 | ||
| 387 | static int apply_microcode_on_target(int cpu) | 391 | static int apply_microcode_on_target(int cpu) |
| 388 | { | 392 | { |
| 389 | struct apply_microcode_ctx ctx = { .err = 0 }; | 393 | enum ucode_state err; |
| 390 | int ret; | 394 | int ret; |
| 391 | 395 | ||
| 392 | ret = smp_call_function_single(cpu, apply_microcode_local, &ctx, 1); | 396 | ret = smp_call_function_single(cpu, apply_microcode_local, &err, 1); |
| 393 | if (!ret) | 397 | if (!ret) { |
| 394 | ret = ctx.err; | 398 | if (err == UCODE_ERROR) |
| 395 | 399 | ret = 1; | |
| 400 | } | ||
| 396 | return ret; | 401 | return ret; |
| 397 | } | 402 | } |
| 398 | 403 | ||
| @@ -489,31 +494,124 @@ static void __exit microcode_dev_exit(void) · 90 unmodified lines | |||
| 489 | /* fake device for request_firmware */ | 494 | /* fake device for request_firmware */ |
| 490 | static struct platform_device *microcode_pdev; | 495 | static struct platform_device *microcode_pdev; |
| 491 | 496 | ||
| 492 | static int reload_for_cpu(int cpu) | 497 | /* |
| 498 | * Late loading dance. Why the heavy-handed stomp_machine effort? | ||
| 499 | * | ||
| 500 | * - HT siblings must be idle and not execute other code while the other sibling | ||
| 501 | * is loading microcode in order to avoid any negative interactions caused by | ||
| 502 | * the loading. | ||
| 503 | * | ||
| 504 | * - In addition, microcode update on the cores must be serialized until this | ||
| 505 | * requirement can be relaxed in the future. Right now, this is conservative | ||
| 506 | * and good. | ||
| 507 | */ | ||
| 508 | #define SPINUNIT 100 /* 100 nsec */ | ||
| 509 | |||
| 510 | static int check_online_cpus(void) | ||
| 493 | { | 511 | { |
| 494 | struct ucode_cpu_info *uci = ucode_cpu_info + cpu; | 512 | if (num_online_cpus() == num_present_cpus()) |
| 495 | enum ucode_state ustate; | 513 | return 0; |
| 496 | int err = 0; | ||
| 497 | 514 | ||
| 498 | if (!uci->valid) | 515 | pr_err("Not all CPUs online, aborting microcode update.\n"); |
| 499 | return err; | ||
| 500 | 516 | ||
| 501 | ustate = microcode_ops->request_microcode_fw(cpu, µcode_pdev->dev, true); | 517 | return -EINVAL; |
| 502 | if (ustate == UCODE_OK) | 518 | } |
| 503 | apply_microcode_on_target(cpu); | 519 | |
| 504 | else | 520 | static atomic_t late_cpus_in; |
| 505 | if (ustate == UCODE_ERROR) | 521 | static atomic_t late_cpus_out; |
| 506 | err = -EINVAL; | 522 | |
| 507 | return err; | 523 | static int __wait_for_cpus(atomic_t *t, long long timeout) |
| 524 | { | ||
| 525 | int all_cpus = num_online_cpus(); | ||
| 526 | |||
| 527 | atomic_inc(t); | ||
| 528 | |||
| 529 | while (atomic_read(t) < all_cpus) { | ||
| 530 | if (timeout < SPINUNIT) { | ||
| 531 | pr_err("Timeout while waiting for CPUs rendezvous, remaining: %d\n", | ||
| 532 | all_cpus - atomic_read(t)); | ||
| 533 | return 1; | ||
| 534 | } | ||
| 535 | |||
| 536 | ndelay(SPINUNIT); | ||
| 537 | timeout -= SPINUNIT; | ||
| 538 | |||
| 539 | touch_nmi_watchdog(); | ||
| 540 | } | ||
| 541 | return 0; | ||
| 542 | } | ||
| 543 | |||
| 544 | /* | ||
| 545 | * Returns: | ||
| 546 | * < 0 - on error | ||
| 547 | * 0 - no update done | ||
| 548 | * 1 - microcode was updated | ||
| 549 | */ | ||
| 550 | static int __reload_late(void *info) | ||
| 551 | { | ||
| 552 | int cpu = smp_processor_id(); | ||
| 553 | enum ucode_state err; | ||
| 554 | int ret = 0; | ||
| 555 | |||
| 556 | /* | ||
| 557 | * Wait for all CPUs to arrive. A load will not be attempted unless all | ||
| 558 | * CPUs show up. | ||
| 559 | * */ | ||
| 560 | if (__wait_for_cpus(&late_cpus_in, NSEC_PER_SEC)) | ||
| 561 | return -1; | ||
| 562 | |||
| 563 | spin_lock(&update_lock); | ||
| 564 | apply_microcode_local(&err); | ||
| 565 | spin_unlock(&update_lock); | ||
| 566 | |||
| 567 | if (err > UCODE_NFOUND) { | ||
| 568 | pr_warn("Error reloading microcode on CPU %d\n", cpu); | ||
| 569 | return -1; | ||
| 570 | /* siblings return UCODE_OK because their engine got updated already */ | ||
| 571 | } else if (err == UCODE_UPDATED || err == UCODE_OK) { | ||
| 572 | ret = 1; | ||
| 573 | } else { | ||
| 574 | return ret; | ||
| 575 | } | ||
| 576 | |||
| 577 | /* | ||
| 578 | * Increase the wait timeout to a safe value here since we're | ||
| 579 | * serializing the microcode update and that could take a while on a | ||
| 580 | * large number of CPUs. And that is fine as the *actual* timeout will | ||
| 581 | * be determined by the last CPU finished updating and thus cut short. | ||
| 582 | */ | ||
| 583 | if (__wait_for_cpus(&late_cpus_out, NSEC_PER_SEC * num_online_cpus())) | ||
| 584 | panic("Timeout during microcode update!\n"); | ||
| 585 | |||
| 586 | return ret; | ||
| 587 | } | ||
| 588 | |||
| 589 | /* | ||
| 590 | * Reload microcode late on all CPUs. Wait for a sec until they | ||
| 591 | * all gather together. | ||
| 592 | */ | ||
| 593 | static int microcode_reload_late(void) | ||
| 594 | { | ||
| 595 | int ret; | ||
| 596 | |||
| 597 | atomic_set(&late_cpus_in, 0); | ||
| 598 | atomic_set(&late_cpus_out, 0); | ||
| 599 | |||
| 600 | ret = stop_machine_cpuslocked(__reload_late, NULL, cpu_online_mask); | ||
| 601 | if (ret > 0) | ||
| 602 | microcode_check(); | ||
| 603 | |||
| 604 | return ret; | ||
| 508 | } | 605 | } |
| 509 | 606 | ||
| 510 | static ssize_t reload_store(struct device *dev, | 607 | static ssize_t reload_store(struct device *dev, |
| 511 | struct device_attribute *attr, | 608 | struct device_attribute *attr, |
| 512 | const char *buf, size_t size) | 609 | const char *buf, size_t size) |
| 513 | { | 610 | { |
| 611 | enum ucode_state tmp_ret = UCODE_OK; | ||
| 612 | int bsp = boot_cpu_data.cpu_index; | ||
| 514 | unsigned long val; | 613 | unsigned long val; |
| 515 | int cpu; | 614 | ssize_t ret = 0; |
| 516 | ssize_t ret = 0, tmp_ret; | ||
| 517 | 615 | ||
| 518 | ret = kstrtoul(buf, 0, &val); | 616 | ret = kstrtoul(buf, 0, &val); |
| 519 | if (ret) | 617 | if (ret) |
| @@ -522,23 +620,24 @@ static ssize_t reload_store(struct device *dev, · 2 unmodified lines | |||
| 522 | if (val != 1) | 620 | if (val != 1) |
| 523 | return size; | 621 | return size; |
| 524 | 622 | ||
| 623 | tmp_ret = microcode_ops->request_microcode_fw(bsp, µcode_pdev->dev, true); | ||
| 624 | if (tmp_ret != UCODE_NEW) | ||
| 625 | return size; | ||
| 626 | |||
| 525 | get_online_cpus(); | 627 | get_online_cpus(); |
| 526 | mutex_lock(µcode_mutex); | ||
| 527 | for_each_online_cpu(cpu) { | ||
| 528 | tmp_ret = reload_for_cpu(cpu); | ||
| 529 | if (tmp_ret != 0) | ||
| 530 | pr_warn("Error reloading microcode on CPU %d\n", cpu); | ||
| 531 | 628 | ||
| 532 | /* save retval of the first encountered reload error */ | 629 | ret = check_online_cpus(); |
| 533 | if (!ret) | 630 | if (ret) |
| 534 | ret = tmp_ret; | 631 | goto put; |
| 535 | } | 632 | |
| 536 | if (!ret) | 633 | mutex_lock(µcode_mutex); |
| 537 | perf_check_microcode(); | 634 | ret = microcode_reload_late(); |
| 538 | mutex_unlock(µcode_mutex); | 635 | mutex_unlock(µcode_mutex); |
| 636 | |||
| 637 | put: | ||
| 539 | put_online_cpus(); | 638 | put_online_cpus(); |
| 540 | 639 | ||
| 541 | if (!ret) | 640 | if (ret >= 0) |
| 542 | ret = size; | 641 | ret = size; |
| 543 | 642 | ||
| 544 | return ret; | 643 | return ret; |
| @@ -606,10 +705,8 @@ static enum ucode_state microcode_init_cpu(int cpu, bool refresh_fw) · 61 unmodified lines | |||
| 606 | if (system_state != SYSTEM_RUNNING) | 705 | if (system_state != SYSTEM_RUNNING) |
| 607 | return UCODE_NFOUND; | 706 | return UCODE_NFOUND; |
| 608 | 707 | ||
| 609 | ustate = microcode_ops->request_microcode_fw(cpu, µcode_pdev->dev, | 708 | ustate = microcode_ops->request_microcode_fw(cpu, µcode_pdev->dev, refresh_fw); |
| 610 | refresh_fw); | 709 | if (ustate == UCODE_NEW) { |
| 611 | |||
| 612 | if (ustate == UCODE_OK) { | ||
| 613 | pr_debug("CPU%d updated upon init\n", cpu); | 710 | pr_debug("CPU%d updated upon init\n", cpu); |
| 614 | apply_microcode_on_target(cpu); | 711 | apply_microcode_on_target(cpu); |
| 615 | } | 712 | } |
| @@ -589,6 +589,23 @@ static int apply_microcode_early(struct ucode_cpu_info *uci, bool early) · 588 unmodified lines | |||
| 589 | if (!mc) | 589 | if (!mc) |
| 590 | return 0; | 590 | return 0; |
| 591 | 591 | ||
| 592 | /* | ||
| 593 | * Save us the MSR write below - which is a particular expensive | ||
| 594 | * operation - when the other hyperthread has updated the microcode | ||
| 595 | * already. | ||
| 596 | */ | ||
| 597 | rev = intel_get_microcode_revision(); | ||
| 598 | if (rev >= mc->hdr.rev) { | ||
| 599 | uci->cpu_sig.rev = rev; | ||
| 600 | return UCODE_OK; | ||
| 601 | } | ||
| 602 | |||
| 603 | /* | ||
| 604 | * Writeback and invalidate caches before updating microcode to avoid | ||
| 605 | * internal issues depending on what the microcode is updating. | ||
| 606 | */ | ||
| 607 | native_wbinvd(); | ||
| 608 | |||
| 592 | /* write microcode via MSR 0x79 */ | 609 | /* write microcode via MSR 0x79 */ |
| 593 | native_wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); | 610 | native_wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); |
| 594 | 611 | ||
| @@ -772,27 +789,44 @@ static int collect_cpu_info(int cpu_num, struct cpu_signature *csig) · 177 unmodified lines | |||
| 772 | return 0; | 789 | return 0; |
| 773 | } | 790 | } |
| 774 | 791 | ||
| 775 | static int apply_microcode_intel(int cpu) | 792 | static enum ucode_state apply_microcode_intel(int cpu) |
| 776 | { | 793 | { |
| 794 | struct ucode_cpu_info *uci = ucode_cpu_info + cpu; | ||
| 795 | struct cpuinfo_x86 *c = &cpu_data(cpu); | ||
| 777 | struct microcode_intel *mc; | 796 | struct microcode_intel *mc; |
| 778 | struct ucode_cpu_info *uci; | ||
| 779 | struct cpuinfo_x86 *c; | ||
| 780 | static int prev_rev; | 797 | static int prev_rev; |
| 781 | u32 rev; | 798 | u32 rev; |
| 782 | 799 | ||
| 783 | /* We should bind the task to the CPU */ | 800 | /* We should bind the task to the CPU */ |
| 784 | if (WARN_ON(raw_smp_processor_id() != cpu)) | 801 | if (WARN_ON(raw_smp_processor_id() != cpu)) |
| 785 | return -1; | 802 | return UCODE_ERROR; |
| 786 | 803 | ||
| 787 | uci = ucode_cpu_info + cpu; | 804 | /* Look for a newer patch in our cache: */ |
| 788 | mc = uci->mc; | 805 | mc = find_patch(uci); |
| 789 | if (!mc) { | 806 | if (!mc) { |
| 790 | /* Look for a newer patch in our cache: */ | 807 | mc = uci->mc; |
| 791 | mc = find_patch(uci); | ||
| 792 | if (!mc) | 808 | if (!mc) |
| 793 | return 0; | 809 | return UCODE_NFOUND; |
| 794 | } | 810 | } |
| 795 | 811 | ||
| 812 | /* | ||
| 813 | * Save us the MSR write below - which is a particular expensive | ||
| 814 | * operation - when the other hyperthread has updated the microcode | ||
| 815 | * already. | ||
| 816 | */ | ||
| 817 | rev = intel_get_microcode_revision(); | ||
| 818 | if (rev >= mc->hdr.rev) { | ||
| 819 | uci->cpu_sig.rev = rev; | ||
| 820 | c->microcode = rev; | ||
| 821 | return UCODE_OK; | ||
| 822 | } | ||
| 823 | |||
| 824 | /* | ||
| 825 | * Writeback and invalidate caches before updating microcode to avoid | ||
| 826 | * internal issues depending on what the microcode is updating. | ||
| 827 | */ | ||
| 828 | native_wbinvd(); | ||
| 829 | |||
| 796 | /* write microcode via MSR 0x79 */ | 830 | /* write microcode via MSR 0x79 */ |
| 797 | wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); | 831 | wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); |
| 798 | 832 | ||
| @@ -801,7 +835,7 @@ static int apply_microcode_intel(int cpu) · 2 unmodified lines | |||
| 801 | if (rev != mc->hdr.rev) { | 835 | if (rev != mc->hdr.rev) { |
| 802 | pr_err("CPU%d update to revision 0x%x failed\n", | 836 | pr_err("CPU%d update to revision 0x%x failed\n", |
| 803 | cpu, mc->hdr.rev); | 837 | cpu, mc->hdr.rev); |
| 804 | return -1; | 838 | return UCODE_ERROR; |
| 805 | } | 839 | } |
| 806 | 840 | ||
| 807 | if (rev != prev_rev) { | 841 | if (rev != prev_rev) { |
| @@ -813,12 +847,10 @@ static int apply_microcode_intel(int cpu) · 5 unmodified lines | |||
| 813 | prev_rev = rev; | 847 | prev_rev = rev; |
| 814 | } | 848 | } |
| 815 | 849 | ||
| 816 | c = &cpu_data(cpu); | ||
| 817 | |||
| 818 | uci->cpu_sig.rev = rev; | 850 | uci->cpu_sig.rev = rev; |
| 819 | c->microcode = rev; | 851 | c->microcode = rev; |
| 820 | 852 | ||
| 821 | return 0; | 853 | return UCODE_UPDATED; |
| 822 | } | 854 | } |
| 823 | 855 | ||
| 824 | static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, | 856 | static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, |
| @@ -830,6 +862,7 @@ static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, · 5 unmodified lines | |||
| 830 | unsigned int leftover = size; | 862 | unsigned int leftover = size; |
| 831 | unsigned int curr_mc_size = 0, new_mc_size = 0; | 863 | unsigned int curr_mc_size = 0, new_mc_size = 0; |
| 832 | unsigned int csig, cpf; | 864 | unsigned int csig, cpf; |
| 865 | enum ucode_state ret = UCODE_OK; | ||
| 833 | 866 | ||
| 834 | while (leftover) { | 867 | while (leftover) { |
| 835 | struct microcode_header_intel mc_header; | 868 | struct microcode_header_intel mc_header; |
| @@ -871,6 +904,7 @@ static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, · 35 unmodified lines | |||
| 871 | new_mc = mc; | 904 | new_mc = mc; |
| 872 | new_mc_size = mc_size; | 905 | new_mc_size = mc_size; |
| 873 | mc = NULL; /* trigger new vmalloc */ | 906 | mc = NULL; /* trigger new vmalloc */ |
| 907 | ret = UCODE_NEW; | ||
| 874 | } | 908 | } |
| 875 | 909 | ||
| 876 | ucode_ptr += mc_size; | 910 | ucode_ptr += mc_size; |
| @@ -900,7 +934,7 @@ static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, · 23 unmodified lines | |||
| 900 | pr_debug("CPU%d found a matching microcode update with version 0x%x (current=0x%x)\n", | 934 | pr_debug("CPU%d found a matching microcode update with version 0x%x (current=0x%x)\n", |
| 901 | cpu, new_rev, uci->cpu_sig.rev); | 935 | cpu, new_rev, uci->cpu_sig.rev); |
| 902 | 936 | ||
| 903 | return UCODE_OK; | 937 | return ret; |
| 904 | } | 938 | } |
| 905 | 939 | ||
| 906 | static int get_ucode_fw(void *to, const void *from, size_t n) | 940 | static int get_ucode_fw(void *to, const void *from, size_t n) |
| @@ -921,7 +955,7 @@ static bool is_blacklisted(unsigned int cpu) · 14 unmodified lines | |||
| 921 | */ | 955 | */ |
| 922 | if (c->x86 == 6 && | 956 | if (c->x86 == 6 && |
| 923 | c->x86_model == INTEL_FAM6_BROADWELL_X && | 957 | c->x86_model == INTEL_FAM6_BROADWELL_X && |
| 924 | c->x86_mask == 0x01 && | 958 | c->x86_stepping == 0x01 && |
| 925 | llc_size_per_core > 2621440 && | 959 | llc_size_per_core > 2621440 && |
| 926 | c->microcode < 0x0b000021) { | 960 | c->microcode < 0x0b000021) { |
| 927 | pr_err_once("Erratum BDF90: late loading with revision < 0x0b000021 (0x%x) disabled.\n", c->microcode); | 961 | pr_err_once("Erratum BDF90: late loading with revision < 0x0b000021 (0x%x) disabled.\n", c->microcode); |
| @@ -944,7 +978,7 @@ static enum ucode_state request_microcode_fw(int cpu, struct device *device, · 16 unmodified lines | |||
| 944 | return UCODE_NFOUND; | 978 | return UCODE_NFOUND; |
| 945 | 979 | ||
| 946 | sprintf(name, "intel-ucode/%02x-%02x-%02x", | 980 | sprintf(name, "intel-ucode/%02x-%02x-%02x", |
| 947 | c->x86, c->x86_model, c->x86_mask); | 981 | c->x86, c->x86_model, c->x86_stepping); |
| 948 | 982 | ||
| 949 | if (request_firmware_direct(&firmware, name, device)) { | 983 | if (request_firmware_direct(&firmware, name, device)) { |
| 950 | pr_debug("data file %s load failed\n", name); | 984 | pr_debug("data file %s load failed\n", name); |
| @@ -982,7 +1016,7 @@ static struct microcode_ops microcode_intel_ops = { · 31 unmodified lines | |||
| 982 | 1016 | ||
| 983 | static int __init calc_llc_size_per_core(struct cpuinfo_x86 *c) | 1017 | static int __init calc_llc_size_per_core(struct cpuinfo_x86 *c) |
| 984 | { | 1018 | { |
| 985 | u64 llc_size = c->x86_cache_size * 1024; | 1019 | u64 llc_size = c->x86_cache_size * 1024ULL; |
| 986 | 1020 | ||
| 987 | do_div(llc_size, c->x86_max_cores); | 1021 | do_div(llc_size, c->x86_max_cores); |
| 988 | 1022 | ||
| @@ -859,7 +859,7 @@ int generic_validate_add_page(unsigned long base, unsigned long size, · 858 unmodified lines | |||
| 859 | */ | 859 | */ |
| 860 | if (is_cpu(INTEL) && boot_cpu_data.x86 == 6 && | 860 | if (is_cpu(INTEL) && boot_cpu_data.x86 == 6 && |
| 861 | boot_cpu_data.x86_model == 1 && | 861 | boot_cpu_data.x86_model == 1 && |
| 862 | boot_cpu_data.x86_mask <= 7) { | 862 | boot_cpu_data.x86_stepping <= 7) { |
| 863 | if (base & ((1 << (22 - PAGE_SHIFT)) - 1)) { | 863 | if (base & ((1 << (22 - PAGE_SHIFT)) - 1)) { |
| 864 | pr_warn("mtrr: base(0x%lx000) is not 4 MiB aligned\n", base); | 864 | pr_warn("mtrr: base(0x%lx000) is not 4 MiB aligned\n", base); |
| 865 | return -EINVAL; | 865 | return -EINVAL; |
| @@ -711,8 +711,8 @@ void __init mtrr_bp_init(void) · 710 unmodified lines | |||
| 711 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && | 711 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && |
| 712 | boot_cpu_data.x86 == 0xF && | 712 | boot_cpu_data.x86 == 0xF && |
| 713 | boot_cpu_data.x86_model == 0x3 && | 713 | boot_cpu_data.x86_model == 0x3 && |
| 714 | (boot_cpu_data.x86_mask == 0x3 || | 714 | (boot_cpu_data.x86_stepping == 0x3 || |
| 715 | boot_cpu_data.x86_mask == 0x4)) | 715 | boot_cpu_data.x86_stepping == 0x4)) |
| 716 | phys_addr = 36; | 716 | phys_addr = 36; |
| 717 | 717 | ||
| 718 | size_or_mask = SIZE_OR_MASK_BITS(phys_addr); | 718 | size_or_mask = SIZE_OR_MASK_BITS(phys_addr); |
| @@ -72,8 +72,8 @@ static int show_cpuinfo(struct seq_file *m, void *v) · 71 unmodified lines | |||
| 72 | c->x86_model, | 72 | c->x86_model, |
| 73 | c->x86_model_id[0] ? c->x86_model_id : "unknown"); | 73 | c->x86_model_id[0] ? c->x86_model_id : "unknown"); |
| 74 | 74 | ||
| 75 | if (c->x86_mask || c->cpuid_level >= 0) | 75 | if (c->x86_stepping || c->cpuid_level >= 0) |
| 76 | seq_printf(m, "stepping\t: %d\n", c->x86_mask); | 76 | seq_printf(m, "stepping\t: %d\n", c->x86_stepping); |
| 77 | else | 77 | else |
| 78 | seq_puts(m, "stepping\t: unknown\n"); | 78 | seq_puts(m, "stepping\t: unknown\n"); |
| 79 | if (c->microcode) | 79 | if (c->microcode) |
| @@ -91,8 +91,8 @@ static int show_cpuinfo(struct seq_file *m, void *v) · 11 unmodified lines | |||
| 91 | } | 91 | } |
| 92 | 92 | ||
| 93 | /* Cache size */ | 93 | /* Cache size */ |
| 94 | if (c->x86_cache_size >= 0) | 94 | if (c->x86_cache_size) |
| 95 | seq_printf(m, "cache size\t: %d KB\n", c->x86_cache_size); | 95 | seq_printf(m, "cache size\t: %u KB\n", c->x86_cache_size); |
| 96 | 96 | ||
| 97 | show_cpuinfo_core(m, c, cpu); | 97 | show_cpuinfo_core(m, c, cpu); |
| 98 | show_cpuinfo_misc(m, c); | 98 | show_cpuinfo_misc(m, c); |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | |||
| 37 | #define X86 new_cpu_data+CPUINFO_x86 | 37 | #define X86 new_cpu_data+CPUINFO_x86 |
| 38 | #define X86_VENDOR new_cpu_data+CPUINFO_x86_vendor | 38 | #define X86_VENDOR new_cpu_data+CPUINFO_x86_vendor |
| 39 | #define X86_MODEL new_cpu_data+CPUINFO_x86_model | 39 | #define X86_MODEL new_cpu_data+CPUINFO_x86_model |
| 40 | #define X86_MASK new_cpu_data+CPUINFO_x86_mask | 40 | #define X86_STEPPING new_cpu_data+CPUINFO_x86_stepping |
| 41 | #define X86_HARD_MATH new_cpu_data+CPUINFO_hard_math | 41 | #define X86_HARD_MATH new_cpu_data+CPUINFO_hard_math |
| 42 | #define X86_CPUID new_cpu_data+CPUINFO_cpuid_level | 42 | #define X86_CPUID new_cpu_data+CPUINFO_cpuid_level |
| 43 | #define X86_CAPABILITY new_cpu_data+CPUINFO_x86_capability | 43 | #define X86_CAPABILITY new_cpu_data+CPUINFO_x86_capability |
| @@ -332,7 +332,7 @@ ENTRY(startup_32_smp) · 288 unmodified lines | |||
| 332 | shrb $4,%al | 332 | shrb $4,%al |
| 333 | movb %al,X86_MODEL | 333 | movb %al,X86_MODEL |
| 334 | andb $0x0f,%cl # mask mask revision | 334 | andb $0x0f,%cl # mask mask revision |
| 335 | movb %cl,X86_MASK | 335 | movb %cl,X86_STEPPING |
| 336 | movl %edx,X86_CAPABILITY | 336 | movl %edx,X86_CAPABILITY |
| 337 | 337 | ||
| 338 | .Lis486: | 338 | .Lis486: |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | |||
| 23 | #include <asm/nops.h> | 23 | #include <asm/nops.h> |
| 24 | #include "../entry/calling.h" | 24 | #include "../entry/calling.h" |
| 25 | #include <asm/export.h> | 25 | #include <asm/export.h> |
| 26 | #include <asm/nospec-branch.h> | ||
| 26 | 27 | ||
| 27 | #ifdef CONFIG_PARAVIRT | 28 | #ifdef CONFIG_PARAVIRT |
| 28 | #include <asm/asm-offsets.h> | 29 | #include <asm/asm-offsets.h> |
| @@ -134,6 +135,7 @@ ENTRY(secondary_startup_64) · 105 unmodified lines | |||
| 134 | 135 | ||
| 135 | /* Ensure I am executing from virtual addresses */ | 136 | /* Ensure I am executing from virtual addresses */ |
| 136 | movq $1f, %rax | 137 | movq $1f, %rax |
| 138 | ANNOTATE_RETPOLINE_SAFE | ||
| 137 | jmp *%rax | 139 | jmp *%rax |
| 138 | 1: | 140 | 1: |
| 139 | UNWIND_HINT_EMPTY | 141 | UNWIND_HINT_EMPTY |
| @@ -160,7 +160,6 @@ static const __initconst struct idt_data early_pf_idts[] = { · 159 unmodified lines | |||
| 160 | */ | 160 | */ |
| 161 | static const __initconst struct idt_data dbg_idts[] = { | 161 | static const __initconst struct idt_data dbg_idts[] = { |
| 162 | INTG(X86_TRAP_DB, debug), | 162 | INTG(X86_TRAP_DB, debug), |
| 163 | INTG(X86_TRAP_BP, int3), | ||
| 164 | }; | 163 | }; |
| 165 | #endif | 164 | #endif |
| 166 | 165 | ||
| @@ -183,7 +182,6 @@ gate_desc debug_idt_table[IDT_ENTRIES] __page_aligned_bss; · 16 unmodified lines | |||
| 183 | static const __initconst struct idt_data ist_idts[] = { | 182 | static const __initconst struct idt_data ist_idts[] = { |
| 184 | ISTG(X86_TRAP_DB, debug, DEBUG_STACK), | 183 | ISTG(X86_TRAP_DB, debug, DEBUG_STACK), |
| 185 | ISTG(X86_TRAP_NMI, nmi, NMI_STACK), | 184 | ISTG(X86_TRAP_NMI, nmi, NMI_STACK), |
| 186 | SISTG(X86_TRAP_BP, int3, DEBUG_STACK), | ||
| 187 | ISTG(X86_TRAP_DF, double_fault, DOUBLEFAULT_STACK), | 185 | ISTG(X86_TRAP_DF, double_fault, DOUBLEFAULT_STACK), |
| 188 | #ifdef CONFIG_X86_MCE | 186 | #ifdef CONFIG_X86_MCE |
| 189 | ISTG(X86_TRAP_MC, &machine_check, MCE_STACK), | 187 | ISTG(X86_TRAP_MC, &machine_check, MCE_STACK), |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | |||
| 23 | /* | 23 | /* |
| 24 | * this changes the io permissions bitmap in the current task. | 24 | * this changes the io permissions bitmap in the current task. |
| 25 | */ | 25 | */ |
| 26 | asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int turn_on) | 26 | SYSCALL_DEFINE3(ioperm, unsigned long, from, unsigned long, num, int, turn_on) |
| 27 | { | 27 | { |
| 28 | struct thread_struct *t = ¤t->thread; | 28 | struct thread_struct *t = ¤t->thread; |
| 29 | struct tss_struct *tss; | 29 | struct tss_struct *tss; |
| @@ -1168,10 +1168,18 @@ NOKPROBE_SYMBOL(longjmp_break_handler); · 1167 unmodified lines | |||
| 1168 | 1168 | ||
| 1169 | bool arch_within_kprobe_blacklist(unsigned long addr) | 1169 | bool arch_within_kprobe_blacklist(unsigned long addr) |
| 1170 | { | 1170 | { |
| 1171 | bool is_in_entry_trampoline_section = false; | ||
| 1172 | |||
| 1173 | #ifdef CONFIG_X86_64 | ||
| 1174 | is_in_entry_trampoline_section = | ||
| 1175 | (addr >= (unsigned long)__entry_trampoline_start && | ||
| 1176 | addr < (unsigned long)__entry_trampoline_end); | ||
| 1177 | #endif | ||
| 1171 | return (addr >= (unsigned long)__kprobes_text_start && | 1178 | return (addr >= (unsigned long)__kprobes_text_start && |
| 1172 | addr < (unsigned long)__kprobes_text_end) || | 1179 | addr < (unsigned long)__kprobes_text_end) || |
| 1173 | (addr >= (unsigned long)__entry_text_start && | 1180 | (addr >= (unsigned long)__entry_text_start && |
| 1174 | addr < (unsigned long)__entry_text_end); | 1181 | addr < (unsigned long)__entry_text_end) || |
| 1182 | is_in_entry_trampoline_section; | ||
| 1175 | } | 1183 | } |
| 1176 | 1184 | ||
| 1177 | int __init arch_init_kprobes(void) | 1185 | int __init arch_init_kprobes(void) |
| @@ -49,7 +49,7 @@ · 48 unmodified lines | |||
| 49 | 49 | ||
| 50 | static int kvmapf = 1; | 50 | static int kvmapf = 1; |
| 51 | 51 | ||
| 52 | static int parse_no_kvmapf(char *arg) | 52 | static int __init parse_no_kvmapf(char *arg) |
| 53 | { | 53 | { |
| 54 | kvmapf = 0; | 54 | kvmapf = 0; |
| 55 | return 0; | 55 | return 0; |
| @@ -58,7 +58,7 @@ static int parse_no_kvmapf(char *arg) · 2 unmodified lines | |||
| 58 | early_param("no-kvmapf", parse_no_kvmapf); | 58 | early_param("no-kvmapf", parse_no_kvmapf); |
| 59 | 59 | ||
| 60 | static int steal_acc = 1; | 60 | static int steal_acc = 1; |
| 61 | static int parse_no_stealacc(char *arg) | 61 | static int __init parse_no_stealacc(char *arg) |
| 62 | { | 62 | { |
| 63 | steal_acc = 0; | 63 | steal_acc = 0; |
| 64 | return 0; | 64 | return 0; |
| @@ -67,7 +67,7 @@ static int parse_no_stealacc(char *arg) · 2 unmodified lines | |||
| 67 | early_param("no-steal-acc", parse_no_stealacc); | 67 | early_param("no-steal-acc", parse_no_stealacc); |
| 68 | 68 | ||
| 69 | static int kvmclock_vsyscall = 1; | 69 | static int kvmclock_vsyscall = 1; |
| 70 | static int parse_no_kvmclock_vsyscall(char *arg) | 70 | static int __init parse_no_kvmclock_vsyscall(char *arg) |
| 71 | { | 71 | { |
| 72 | kvmclock_vsyscall = 0; | 72 | kvmclock_vsyscall = 0; |
| 73 | return 0; | 73 | return 0; |
| @@ -341,10 +341,10 @@ static void kvm_guest_cpu_init(void) · 267 unmodified lines | |||
| 341 | #endif | 341 | #endif |
| 342 | pa |= KVM_ASYNC_PF_ENABLED; | 342 | pa |= KVM_ASYNC_PF_ENABLED; |
| 343 | 343 | ||
| 344 | /* Async page fault support for L1 hypervisor is optional */ | 344 | if (kvm_para_has_feature(KVM_FEATURE_ASYNC_PF_VMEXIT)) |
| 345 | if (wrmsr_safe(MSR_KVM_ASYNC_PF_EN, | 345 | pa |= KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT; |
| 346 | (pa | KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT) & 0xffffffff, pa >> 32) < 0) | 346 | |
| 347 | wrmsrl(MSR_KVM_ASYNC_PF_EN, pa); | 347 | wrmsrl(MSR_KVM_ASYNC_PF_EN, pa); |
| 348 | __this_cpu_write(apf_reason.enabled, 1); | 348 | __this_cpu_write(apf_reason.enabled, 1); |
| 349 | printk(KERN_INFO"KVM setup async PF for cpu %d\n", | 349 | printk(KERN_INFO"KVM setup async PF for cpu %d\n", |
| 350 | smp_processor_id()); | 350 | smp_processor_id()); |
| @@ -545,7 +545,8 @@ static void __init kvm_guest_init(void) · 194 unmodified lines | |||
| 545 | pv_time_ops.steal_clock = kvm_steal_clock; | 545 | pv_time_ops.steal_clock = kvm_steal_clock; |
| 546 | } | 546 | } |
| 547 | 547 | ||
| 548 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH)) | 548 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && |
| 549 | !kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) | ||
| 549 | pv_mmu_ops.flush_tlb_others = kvm_flush_tlb_others; | 550 | pv_mmu_ops.flush_tlb_others = kvm_flush_tlb_others; |
| 550 | 551 | ||
| 551 | if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) | 552 | if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) |
| @@ -633,7 +634,8 @@ static __init int kvm_setup_pv_tlb_flush(void) · 81 unmodified lines | |||
| 633 | { | 634 | { |
| 634 | int cpu; | 635 | int cpu; |
| 635 | 636 | ||
| 636 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH)) { | 637 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && |
| 638 | !kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { | ||
| 637 | for_each_possible_cpu(cpu) { | 639 | for_each_possible_cpu(cpu) { |
| 638 | zalloc_cpumask_var_node(per_cpu_ptr(&__pv_tlb_mask, cpu), | 640 | zalloc_cpumask_var_node(per_cpu_ptr(&__pv_tlb_mask, cpu), |
| 639 | GFP_KERNEL, cpu_to_node(cpu)); | 641 | GFP_KERNEL, cpu_to_node(cpu)); |
| @@ -542,6 +542,7 @@ int arch_kexec_apply_relocations_add(const Elf64_Ehdr *ehdr, · 541 unmodified lines | |||
| 542 | goto overflow; | 542 | goto overflow; |
| 543 | break; | 543 | break; |
| 544 | case R_X86_64_PC32: | 544 | case R_X86_64_PC32: |
| 545 | case R_X86_64_PLT32: | ||
| 545 | value -= (u64)address; | 546 | value -= (u64)address; |
| 546 | *(u32 *)location = value; | 547 | *(u32 *)location = value; |
| 547 | break; | 548 | break; |
| @@ -191,6 +191,7 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, · 190 unmodified lines | |||
| 191 | goto overflow; | 191 | goto overflow; |
| 192 | break; | 192 | break; |
| 193 | case R_X86_64_PC32: | 193 | case R_X86_64_PC32: |
| 194 | case R_X86_64_PLT32: | ||
| 194 | if (*(u32 *)loc != 0) | 195 | if (*(u32 *)loc != 0) |
| 195 | goto invalid_relocation; | 196 | goto invalid_relocation; |
| 196 | val -= (u64)loc; | 197 | val -= (u64)loc; |
| @@ -410,7 +410,7 @@ static inline void __init construct_default_ISA_mptable(int mpc_default_type) · 409 unmodified lines | |||
| 410 | processor.apicver = mpc_default_type > 4 ? 0x10 : 0x01; | 410 | processor.apicver = mpc_default_type > 4 ? 0x10 : 0x01; |
| 411 | processor.cpuflag = CPU_ENABLED; | 411 | processor.cpuflag = CPU_ENABLED; |
| 412 | processor.cpufeature = (boot_cpu_data.x86 << 8) | | 412 | processor.cpufeature = (boot_cpu_data.x86 << 8) | |
| 413 | (boot_cpu_data.x86_model << 4) | boot_cpu_data.x86_mask; | 413 | (boot_cpu_data.x86_model << 4) | boot_cpu_data.x86_stepping; |
| 414 | processor.featureflag = boot_cpu_data.x86_capability[CPUID_1_EDX]; | 414 | processor.featureflag = boot_cpu_data.x86_capability[CPUID_1_EDX]; |
| 415 | processor.reserved[0] = 0; | 415 | processor.reserved[0] = 0; |
| 416 | processor.reserved[1] = 0; | 416 | processor.reserved[1] = 0; |
| @@ -200,9 +200,9 @@ static void native_flush_tlb_global(void) · 199 unmodified lines | |||
| 200 | __native_flush_tlb_global(); | 200 | __native_flush_tlb_global(); |
| 201 | } | 201 | } |
| 202 | 202 | ||
| 203 | static void native_flush_tlb_single(unsigned long addr) | 203 | static void native_flush_tlb_one_user(unsigned long addr) |
| 204 | { | 204 | { |
| 205 | __native_flush_tlb_single(addr); | 205 | __native_flush_tlb_one_user(addr); |
| 206 | } | 206 | } |
| 207 | 207 | ||
| 208 | struct static_key paravirt_steal_enabled; | 208 | struct static_key paravirt_steal_enabled; |
| @@ -401,7 +401,7 @@ struct pv_mmu_ops pv_mmu_ops __ro_after_init = { · 192 unmodified lines | |||
| 401 | 401 | ||
| 402 | .flush_tlb_user = native_flush_tlb, | 402 | .flush_tlb_user = native_flush_tlb, |
| 403 | .flush_tlb_kernel = native_flush_tlb_global, | 403 | .flush_tlb_kernel = native_flush_tlb_global, |
| 404 | .flush_tlb_single = native_flush_tlb_single, | 404 | .flush_tlb_one_user = native_flush_tlb_one_user, |
| 405 | .flush_tlb_others = native_flush_tlb_others, | 405 | .flush_tlb_others = native_flush_tlb_others, |
| 406 | 406 | ||
| 407 | .pgd_alloc = __paravirt_pgd_alloc, | 407 | .pgd_alloc = __paravirt_pgd_alloc, |
| @@ -37,7 +37,6 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, · 36 unmodified lines | |||
| 37 | WARN_ON(size == 0); | 37 | WARN_ON(size == 0); |
| 38 | if (!check_addr("map_single", dev, bus, size)) | 38 | if (!check_addr("map_single", dev, bus, size)) |
| 39 | return NOMMU_MAPPING_ERROR; | 39 | return NOMMU_MAPPING_ERROR; |
| 40 | flush_write_buffers(); | ||
| 41 | return bus; | 40 | return bus; |
| 42 | } | 41 | } |
| 43 | 42 | ||
| @@ -72,25 +71,9 @@ static int nommu_map_sg(struct device *hwdev, struct scatterlist *sg, · 28 unmodified lines | |||
| 72 | return 0; | 71 | return 0; |
| 73 | s->dma_length = s->length; | 72 | s->dma_length = s->length; |
| 74 | } | 73 | } |
| 75 | flush_write_buffers(); | ||
| 76 | return nents; | 74 | return nents; |
| 77 | } | 75 | } |
| 78 | 76 | ||
| 79 | static void nommu_sync_single_for_device(struct device *dev, | ||
| 80 | dma_addr_t addr, size_t size, | ||
| 81 | enum dma_data_direction dir) | ||
| 82 | { | ||
| 83 | flush_write_buffers(); | ||
| 84 | } | ||
| 85 | |||
| 86 | |||
| 87 | static void nommu_sync_sg_for_device(struct device *dev, | ||
| 88 | struct scatterlist *sg, int nelems, | ||
| 89 | enum dma_data_direction dir) | ||
| 90 | { | ||
| 91 | flush_write_buffers(); | ||
| 92 | } | ||
| 93 | |||
| 94 | static int nommu_mapping_error(struct device *dev, dma_addr_t dma_addr) | 77 | static int nommu_mapping_error(struct device *dev, dma_addr_t dma_addr) |
| 95 | { | 78 | { |
| 96 | return dma_addr == NOMMU_MAPPING_ERROR; | 79 | return dma_addr == NOMMU_MAPPING_ERROR; |
| @@ -101,8 +84,6 @@ const struct dma_map_ops nommu_dma_ops = { · 4 unmodified lines | |||
| 101 | .free = dma_generic_free_coherent, | 84 | .free = dma_generic_free_coherent, |
| 102 | .map_sg = nommu_map_sg, | 85 | .map_sg = nommu_map_sg, |
| 103 | .map_page = nommu_map_page, | 86 | .map_page = nommu_map_page, |
| 104 | .sync_single_for_device = nommu_sync_single_for_device, | ||
| 105 | .sync_sg_for_device = nommu_sync_sg_for_device, | ||
| 106 | .is_phys = 1, | 87 | .is_phys = 1, |
| 107 | .mapping_error = nommu_mapping_error, | 88 | .mapping_error = nommu_mapping_error, |
| 108 | .dma_supported = x86_dma_supported, | 89 | .dma_supported = x86_dma_supported, |
| @@ -1204,20 +1204,13 @@ void __init setup_arch(char **cmdline_p) · 1203 unmodified lines | |||
| 1204 | 1204 | ||
| 1205 | kasan_init(); | 1205 | kasan_init(); |
| 1206 | 1206 | ||
| 1207 | #ifdef CONFIG_X86_32 | ||
| 1208 | /* sync back kernel address range */ | ||
| 1209 | clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY, | ||
| 1210 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 1211 | KERNEL_PGD_PTRS); | ||
| 1212 | |||
| 1213 | /* | 1207 | /* |
| 1214 | * sync back low identity map too. It is used for example | 1208 | * Sync back kernel address range. |
| 1215 | * in the 32-bit EFI stub. | 1209 | * |
| 1210 | * FIXME: Can the later sync in setup_cpu_entry_areas() replace | ||
| 1211 | * this call? | ||
| 1216 | */ | 1212 | */ |
| 1217 | clone_pgd_range(initial_page_table, | 1213 | sync_initial_page_table(); |
| 1218 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 1219 | min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY)); | ||
| 1220 | #endif | ||
| 1221 | 1214 | ||
| 1222 | tboot_probe(); | 1215 | tboot_probe(); |
| 1223 | 1216 | ||
| @@ -287,24 +287,15 @@ void __init setup_per_cpu_areas(void) · 286 unmodified lines | |||
| 287 | /* Setup cpu initialized, callin, callout masks */ | 287 | /* Setup cpu initialized, callin, callout masks */ |
| 288 | setup_cpu_local_masks(); | 288 | setup_cpu_local_masks(); |
| 289 | 289 | ||
| 290 | #ifdef CONFIG_X86_32 | ||
| 291 | /* | 290 | /* |
| 292 | * Sync back kernel address range again. We already did this in | 291 | * Sync back kernel address range again. We already did this in |
| 293 | * setup_arch(), but percpu data also needs to be available in | 292 | * setup_arch(), but percpu data also needs to be available in |
| 294 | * the smpboot asm. We can't reliably pick up percpu mappings | 293 | * the smpboot asm. We can't reliably pick up percpu mappings |
| 295 | * using vmalloc_fault(), because exception dispatch needs | 294 | * using vmalloc_fault(), because exception dispatch needs |
| 296 | * percpu data. | 295 | * percpu data. |
| 296 | * | ||
| 297 | * FIXME: Can the later sync in setup_cpu_entry_areas() replace | ||
| 298 | * this call? | ||
| 297 | */ | 299 | */ |
| 298 | clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY, | 300 | sync_initial_page_table(); |
| 299 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 300 | KERNEL_PGD_PTRS); | ||
| 301 | |||
| 302 | /* | ||
| 303 | * sync back low identity map too. It is used for example | ||
| 304 | * in the 32-bit EFI stub. | ||
| 305 | */ | ||
| 306 | clone_pgd_range(initial_page_table, | ||
| 307 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 308 | min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY)); | ||
| 309 | #endif | ||
| 310 | } | 301 | } |
| @@ -43,6 +43,13 @@ static inline void signal_compat_build_tests(void) · 42 unmodified lines | |||
| 43 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields) != 3 * sizeof(int)); | 43 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields) != 3 * sizeof(int)); |
| 44 | #define CHECK_CSI_OFFSET(name) BUILD_BUG_ON(_sifields_offset != offsetof(compat_siginfo_t, _sifields.name)) | 44 | #define CHECK_CSI_OFFSET(name) BUILD_BUG_ON(_sifields_offset != offsetof(compat_siginfo_t, _sifields.name)) |
| 45 | 45 | ||
| 46 | BUILD_BUG_ON(offsetof(siginfo_t, si_signo) != 0); | ||
| 47 | BUILD_BUG_ON(offsetof(siginfo_t, si_errno) != 4); | ||
| 48 | BUILD_BUG_ON(offsetof(siginfo_t, si_code) != 8); | ||
| 49 | |||
| 50 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_signo) != 0); | ||
| 51 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_errno) != 4); | ||
| 52 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_code) != 8); | ||
| 46 | /* | 53 | /* |
| 47 | * Ensure that the size of each si_field never changes. | 54 | * Ensure that the size of each si_field never changes. |
| 48 | * If it does, it is a sign that the | 55 | * If it does, it is a sign that the |
| @@ -63,36 +70,94 @@ static inline void signal_compat_build_tests(void) · 14 unmodified lines | |||
| 63 | CHECK_CSI_SIZE (_kill, 2*sizeof(int)); | 70 | CHECK_CSI_SIZE (_kill, 2*sizeof(int)); |
| 64 | CHECK_SI_SIZE (_kill, 2*sizeof(int)); | 71 | CHECK_SI_SIZE (_kill, 2*sizeof(int)); |
| 65 | 72 | ||
| 73 | BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x10); | ||
| 74 | BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x14); | ||
| 75 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pid) != 0xC); | ||
| 76 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_uid) != 0x10); | ||
| 77 | |||
| 66 | CHECK_CSI_OFFSET(_timer); | 78 | CHECK_CSI_OFFSET(_timer); |
| 67 | CHECK_CSI_SIZE (_timer, 3*sizeof(int)); | 79 | CHECK_CSI_SIZE (_timer, 3*sizeof(int)); |
| 68 | CHECK_SI_SIZE (_timer, 6*sizeof(int)); | 80 | CHECK_SI_SIZE (_timer, 6*sizeof(int)); |
| 69 | 81 | ||
| 82 | BUILD_BUG_ON(offsetof(siginfo_t, si_tid) != 0x10); | ||
| 83 | BUILD_BUG_ON(offsetof(siginfo_t, si_overrun) != 0x14); | ||
| 84 | BUILD_BUG_ON(offsetof(siginfo_t, si_value) != 0x18); | ||
| 85 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_tid) != 0x0C); | ||
| 86 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_overrun) != 0x10); | ||
| 87 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_value) != 0x14); | ||
| 88 | |||
| 70 | CHECK_CSI_OFFSET(_rt); | 89 | CHECK_CSI_OFFSET(_rt); |
| 71 | CHECK_CSI_SIZE (_rt, 3*sizeof(int)); | 90 | CHECK_CSI_SIZE (_rt, 3*sizeof(int)); |
| 72 | CHECK_SI_SIZE (_rt, 4*sizeof(int)); | 91 | CHECK_SI_SIZE (_rt, 4*sizeof(int)); |
| 73 | 92 | ||
| 93 | BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x10); | ||
| 94 | BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x14); | ||
| 95 | BUILD_BUG_ON(offsetof(siginfo_t, si_value) != 0x18); | ||
| 96 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pid) != 0x0C); | ||
| 97 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_uid) != 0x10); | ||
| 98 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_value) != 0x14); | ||
| 99 | |||
| 74 | CHECK_CSI_OFFSET(_sigchld); | 100 | CHECK_CSI_OFFSET(_sigchld); |
| 75 | CHECK_CSI_SIZE (_sigchld, 5*sizeof(int)); | 101 | CHECK_CSI_SIZE (_sigchld, 5*sizeof(int)); |
| 76 | CHECK_SI_SIZE (_sigchld, 8*sizeof(int)); | 102 | CHECK_SI_SIZE (_sigchld, 8*sizeof(int)); |
| 77 | 103 | ||
| 104 | BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x10); | ||
| 105 | BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x14); | ||
| 106 | BUILD_BUG_ON(offsetof(siginfo_t, si_status) != 0x18); | ||
| 107 | BUILD_BUG_ON(offsetof(siginfo_t, si_utime) != 0x20); | ||
| 108 | BUILD_BUG_ON(offsetof(siginfo_t, si_stime) != 0x28); | ||
| 109 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pid) != 0x0C); | ||
| 110 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_uid) != 0x10); | ||
| 111 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_status) != 0x14); | ||
| 112 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_utime) != 0x18); | ||
| 113 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_stime) != 0x1C); | ||
| 114 | |||
| 78 | #ifdef CONFIG_X86_X32_ABI | 115 | #ifdef CONFIG_X86_X32_ABI |
| 79 | CHECK_CSI_OFFSET(_sigchld_x32); | 116 | CHECK_CSI_OFFSET(_sigchld_x32); |
| 80 | CHECK_CSI_SIZE (_sigchld_x32, 7*sizeof(int)); | 117 | CHECK_CSI_SIZE (_sigchld_x32, 7*sizeof(int)); |
| 81 | /* no _sigchld_x32 in the generic siginfo_t */ | 118 | /* no _sigchld_x32 in the generic siginfo_t */ |
| 119 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields._sigchld_x32._utime) != 0x18); | ||
| 120 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields._sigchld_x32._stime) != 0x20); | ||
| 82 | #endif | 121 | #endif |
| 83 | 122 | ||
| 84 | CHECK_CSI_OFFSET(_sigfault); | 123 | CHECK_CSI_OFFSET(_sigfault); |
| 85 | CHECK_CSI_SIZE (_sigfault, 4*sizeof(int)); | 124 | CHECK_CSI_SIZE (_sigfault, 4*sizeof(int)); |
| 86 | CHECK_SI_SIZE (_sigfault, 8*sizeof(int)); | 125 | CHECK_SI_SIZE (_sigfault, 8*sizeof(int)); |
| 87 | 126 | ||
| 127 | BUILD_BUG_ON(offsetof(siginfo_t, si_addr) != 0x10); | ||
| 128 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_addr) != 0x0C); | ||
| 129 | |||
| 130 | BUILD_BUG_ON(offsetof(siginfo_t, si_addr_lsb) != 0x18); | ||
| 131 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_addr_lsb) != 0x10); | ||
| 132 | |||
| 133 | BUILD_BUG_ON(offsetof(siginfo_t, si_lower) != 0x20); | ||
| 134 | BUILD_BUG_ON(offsetof(siginfo_t, si_upper) != 0x28); | ||
| 135 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_lower) != 0x14); | ||
| 136 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_upper) != 0x18); | ||
| 137 | |||
| 138 | BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x20); | ||
| 139 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pkey) != 0x14); | ||
| 140 | |||
| 88 | CHECK_CSI_OFFSET(_sigpoll); | 141 | CHECK_CSI_OFFSET(_sigpoll); |
| 89 | CHECK_CSI_SIZE (_sigpoll, 2*sizeof(int)); | 142 | CHECK_CSI_SIZE (_sigpoll, 2*sizeof(int)); |
| 90 | CHECK_SI_SIZE (_sigpoll, 4*sizeof(int)); | 143 | CHECK_SI_SIZE (_sigpoll, 4*sizeof(int)); |
| 91 | 144 | ||
| 145 | BUILD_BUG_ON(offsetof(siginfo_t, si_band) != 0x10); | ||
| 146 | BUILD_BUG_ON(offsetof(siginfo_t, si_fd) != 0x18); | ||
| 147 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_band) != 0x0C); | ||
| 148 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_fd) != 0x10); | ||
| 149 | |||
| 92 | CHECK_CSI_OFFSET(_sigsys); | 150 | CHECK_CSI_OFFSET(_sigsys); |
| 93 | CHECK_CSI_SIZE (_sigsys, 3*sizeof(int)); | 151 | CHECK_CSI_SIZE (_sigsys, 3*sizeof(int)); |
| 94 | CHECK_SI_SIZE (_sigsys, 4*sizeof(int)); | 152 | CHECK_SI_SIZE (_sigsys, 4*sizeof(int)); |
| 95 | 153 | ||
| 154 | BUILD_BUG_ON(offsetof(siginfo_t, si_call_addr) != 0x10); | ||
| 155 | BUILD_BUG_ON(offsetof(siginfo_t, si_syscall) != 0x18); | ||
| 156 | BUILD_BUG_ON(offsetof(siginfo_t, si_arch) != 0x1C); | ||
| 157 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_call_addr) != 0x0C); | ||
| 158 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_syscall) != 0x10); | ||
| 159 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_arch) != 0x14); | ||
| 160 | |||
| 96 | /* any new si_fields should be added here */ | 161 | /* any new si_fields should be added here */ |
| 97 | } | 162 | } |
| 98 | 163 | ||
| @@ -1281,11 +1281,10 @@ void __init native_smp_prepare_boot_cpu(void) · 1280 unmodified lines | |||
| 1281 | cpu_set_state_online(me); | 1281 | cpu_set_state_online(me); |
| 1282 | } | 1282 | } |
| 1283 | 1283 | ||
| 1284 | void __init native_smp_cpus_done(unsigned int max_cpus) | 1284 | void __init calculate_max_logical_packages(void) |
| 1285 | { | 1285 | { |
| 1286 | int ncpus; | 1286 | int ncpus; |
| 1287 | 1287 | ||
| 1288 | pr_debug("Boot done\n"); | ||
| 1289 | /* | 1288 | /* |
| 1290 | * Today neither Intel nor AMD support heterogenous systems so | 1289 | * Today neither Intel nor AMD support heterogenous systems so |
| 1291 | * extrapolate the boot cpu's data to all packages. | 1290 | * extrapolate the boot cpu's data to all packages. |
| @@ -1293,6 +1292,13 @@ void __init native_smp_cpus_done(unsigned int max_cpus) · 1 unmodified line | |||
| 1293 | ncpus = cpu_data(0).booted_cores * topology_max_smt_threads(); | 1292 | ncpus = cpu_data(0).booted_cores * topology_max_smt_threads(); |
| 1294 | __max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus); | 1293 | __max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus); |
| 1295 | pr_info("Max logical packages: %u\n", __max_logical_packages); | 1294 | pr_info("Max logical packages: %u\n", __max_logical_packages); |
| 1295 | } | ||
| 1296 | |||
| 1297 | void __init native_smp_cpus_done(unsigned int max_cpus) | ||
| 1298 | { | ||
| 1299 | pr_debug("Boot done\n"); | ||
| 1300 | |||
| 1301 | calculate_max_logical_packages(); | ||
| 1296 | 1302 | ||
| 1297 | if (x86_has_numa_in_package) | 1303 | if (x86_has_numa_in_package) |
| 1298 | set_sched_topology(x86_numa_in_package_topology); | 1304 | set_sched_topology(x86_numa_in_package_topology); |
| @@ -1430,8 +1436,8 @@ static void remove_siblinginfo(int cpu) · 131 unmodified lines | |||
| 1430 | cpumask_clear(cpu_llc_shared_mask(cpu)); | 1436 | cpumask_clear(cpu_llc_shared_mask(cpu)); |
| 1431 | cpumask_clear(topology_sibling_cpumask(cpu)); | 1437 | cpumask_clear(topology_sibling_cpumask(cpu)); |
| 1432 | cpumask_clear(topology_core_cpumask(cpu)); | 1438 | cpumask_clear(topology_core_cpumask(cpu)); |
| 1433 | c->phys_proc_id = 0; | ||
| 1434 | c->cpu_core_id = 0; | 1439 | c->cpu_core_id = 0; |
| 1440 | c->booted_cores = 0; | ||
| 1435 | cpumask_clear_cpu(cpu, cpu_sibling_setup_mask); | 1441 | cpumask_clear_cpu(cpu, cpu_sibling_setup_mask); |
| 1436 | recompute_smt_state(); | 1442 | recompute_smt_state(); |
| 1437 | } | 1443 | } |
| @@ -181,7 +181,7 @@ int fixup_bug(struct pt_regs *regs, int trapnr) · 180 unmodified lines | |||
| 181 | break; | 181 | break; |
| 182 | 182 | ||
| 183 | case BUG_TRAP_TYPE_WARN: | 183 | case BUG_TRAP_TYPE_WARN: |
| 184 | regs->ip += LEN_UD0; | 184 | regs->ip += LEN_UD2; |
| 185 | return 1; | 185 | return 1; |
| 186 | } | 186 | } |
| 187 | 187 | ||
| @@ -577,7 +577,6 @@ do_general_protection(struct pt_regs *regs, long error_code) · 389 unmodified lines | |||
| 577 | } | 577 | } |
| 578 | NOKPROBE_SYMBOL(do_general_protection); | 578 | NOKPROBE_SYMBOL(do_general_protection); |
| 579 | 579 | ||
| 580 | /* May run on IST stack. */ | ||
| 581 | dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) | 580 | dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) |
| 582 | { | 581 | { |
| 583 | #ifdef CONFIG_DYNAMIC_FTRACE | 582 | #ifdef CONFIG_DYNAMIC_FTRACE |
| @@ -592,6 +591,13 @@ dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) · 8 unmodified lines | |||
| 592 | if (poke_int3_handler(regs)) | 591 | if (poke_int3_handler(regs)) |
| 593 | return; | 592 | return; |
| 594 | 593 | ||
| 594 | /* | ||
| 595 | * Use ist_enter despite the fact that we don't use an IST stack. | ||
| 596 | * We can be called from a kprobe in non-CONTEXT_KERNEL kernel | ||
| 597 | * mode or even during context tracking state changes. | ||
| 598 | * | ||
| 599 | * This means that we can't schedule. That's okay. | ||
| 600 | */ | ||
| 595 | ist_enter(regs); | 601 | ist_enter(regs); |
| 596 | RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); | 602 | RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); |
| 597 | #ifdef CONFIG_KGDB_LOW_LEVEL_TRAP | 603 | #ifdef CONFIG_KGDB_LOW_LEVEL_TRAP |
| @@ -609,15 +615,10 @@ dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) · 11 unmodified lines | |||
| 609 | SIGTRAP) == NOTIFY_STOP) | 615 | SIGTRAP) == NOTIFY_STOP) |
| 610 | goto exit; | 616 | goto exit; |
| 611 | 617 | ||
| 612 | /* | ||
| 613 | * Let others (NMI) know that the debug stack is in use | ||
| 614 | * as we may switch to the interrupt stack. | ||
| 615 | */ | ||
| 616 | debug_stack_usage_inc(); | ||
| 617 | cond_local_irq_enable(regs); | 618 | cond_local_irq_enable(regs); |
| 618 | do_trap(X86_TRAP_BP, SIGTRAP, "int3", regs, error_code, NULL); | 619 | do_trap(X86_TRAP_BP, SIGTRAP, "int3", regs, error_code, NULL); |
| 619 | cond_local_irq_disable(regs); | 620 | cond_local_irq_disable(regs); |
| 620 | debug_stack_usage_dec(); | 621 | |
| 621 | exit: | 622 | exit: |
| 622 | ist_exit(regs); | 623 | ist_exit(regs); |
| 623 | } | 624 | } |
| @@ -5,7 +5,6 @@ · 4 unmodified lines | |||
| 5 | #include <asm/unwind.h> | 5 | #include <asm/unwind.h> |
| 6 | #include <asm/orc_types.h> | 6 | #include <asm/orc_types.h> |
| 7 | #include <asm/orc_lookup.h> | 7 | #include <asm/orc_lookup.h> |
| 8 | #include <asm/sections.h> | ||
| 9 | 8 | ||
| 10 | #define orc_warn(fmt, ...) \ | 9 | #define orc_warn(fmt, ...) \ |
| 11 | printk_deferred_once(KERN_WARNING pr_fmt("WARNING: " fmt), ##__VA_ARGS__) | 10 | printk_deferred_once(KERN_WARNING pr_fmt("WARNING: " fmt), ##__VA_ARGS__) |
| @@ -148,7 +147,7 @@ static struct orc_entry *orc_find(unsigned long ip) · 136 unmodified lines | |||
| 148 | } | 147 | } |
| 149 | 148 | ||
| 150 | /* vmlinux .init slow lookup: */ | 149 | /* vmlinux .init slow lookup: */ |
| 151 | if (ip >= (unsigned long)_sinittext && ip < (unsigned long)_einittext) | 150 | if (init_kernel_text(ip)) |
| 152 | return __orc_find(__start_orc_unwind_ip, __start_orc_unwind, | 151 | return __orc_find(__start_orc_unwind_ip, __start_orc_unwind, |
| 153 | __stop_orc_unwind_ip - __start_orc_unwind_ip, ip); | 152 | __stop_orc_unwind_ip - __start_orc_unwind_ip, ip); |
| 154 | 153 | ||
| @@ -727,7 +727,8 @@ void handle_vm86_fault(struct kernel_vm86_regs *regs, long error_code) · 726 unmodified lines | |||
| 727 | return; | 727 | return; |
| 728 | 728 | ||
| 729 | check_vip: | 729 | check_vip: |
| 730 | if (VEFLAGS & X86_EFLAGS_VIP) { | 730 | if ((VEFLAGS & (X86_EFLAGS_VIP | X86_EFLAGS_VIF)) == |
| 731 | (X86_EFLAGS_VIP | X86_EFLAGS_VIF)) { | ||
| 731 | save_v86_state(regs, VM86_STI); | 732 | save_v86_state(regs, VM86_STI); |
| 732 | return; | 733 | return; |
| 733 | } | 734 | } |
| @@ -118,9 +118,11 @@ SECTIONS · 117 unmodified lines | |||
| 118 | 118 | ||
| 119 | #ifdef CONFIG_X86_64 | 119 | #ifdef CONFIG_X86_64 |
| 120 | . = ALIGN(PAGE_SIZE); | 120 | . = ALIGN(PAGE_SIZE); |
| 121 | VMLINUX_SYMBOL(__entry_trampoline_start) = .; | ||
| 121 | _entry_trampoline = .; | 122 | _entry_trampoline = .; |
| 122 | *(.entry_trampoline) | 123 | *(.entry_trampoline) |
| 123 | . = ALIGN(PAGE_SIZE); | 124 | . = ALIGN(PAGE_SIZE); |
| 125 | VMLINUX_SYMBOL(__entry_trampoline_end) = .; | ||
| 124 | ASSERT(. - _entry_trampoline == PAGE_SIZE, "entry trampoline is too big"); | 126 | ASSERT(. - _entry_trampoline == PAGE_SIZE, "entry trampoline is too big"); |
| 125 | #endif | 127 | #endif |
| 126 | 128 | ||
| @@ -607,7 +607,8 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, · 606 unmodified lines | |||
| 607 | (1 << KVM_FEATURE_PV_EOI) | | 607 | (1 << KVM_FEATURE_PV_EOI) | |
| 608 | (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT) | | 608 | (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT) | |
| 609 | (1 << KVM_FEATURE_PV_UNHALT) | | 609 | (1 << KVM_FEATURE_PV_UNHALT) | |
| 610 | (1 << KVM_FEATURE_PV_TLB_FLUSH); | 610 | (1 << KVM_FEATURE_PV_TLB_FLUSH) | |
| 611 | (1 << KVM_FEATURE_ASYNC_PF_VMEXIT); | ||
| 611 | 612 | ||
| 612 | if (sched_info_on()) | 613 | if (sched_info_on()) |
| 613 | entry->eax |= (1 << KVM_FEATURE_STEAL_TIME); | 614 | entry->eax |= (1 << KVM_FEATURE_STEAL_TIME); |
| @@ -2002,14 +2002,13 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value) · 2001 unmodified lines | |||
| 2002 | 2002 | ||
| 2003 | void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) | 2003 | void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) |
| 2004 | { | 2004 | { |
| 2005 | struct kvm_lapic *apic; | 2005 | struct kvm_lapic *apic = vcpu->arch.apic; |
| 2006 | int i; | 2006 | int i; |
| 2007 | 2007 | ||
| 2008 | apic_debug("%s\n", __func__); | 2008 | if (!apic) |
| 2009 | return; | ||
| 2009 | 2010 | ||
| 2010 | ASSERT(vcpu); | 2011 | apic_debug("%s\n", __func__); |
| 2011 | apic = vcpu->arch.apic; | ||
| 2012 | ASSERT(apic != NULL); | ||
| 2013 | 2012 | ||
| 2014 | /* Stop the timer in case it's a reset to an active apic */ | 2013 | /* Stop the timer in case it's a reset to an active apic */ |
| 2015 | hrtimer_cancel(&apic->lapic_timer.timer); | 2014 | hrtimer_cancel(&apic->lapic_timer.timer); |
| @@ -2165,7 +2164,6 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu) · 149 unmodified lines | |||
| 2165 | */ | 2164 | */ |
| 2166 | vcpu->arch.apic_base = MSR_IA32_APICBASE_ENABLE; | 2165 | vcpu->arch.apic_base = MSR_IA32_APICBASE_ENABLE; |
| 2167 | static_key_slow_inc(&apic_sw_disabled.key); /* sw disabled at reset */ | 2166 | static_key_slow_inc(&apic_sw_disabled.key); /* sw disabled at reset */ |
| 2168 | kvm_lapic_reset(vcpu, false); | ||
| 2169 | kvm_iodevice_init(&apic->dev, &apic_mmio_ops); | 2167 | kvm_iodevice_init(&apic->dev, &apic_mmio_ops); |
| 2170 | 2168 | ||
| 2171 | return 0; | 2169 | return 0; |
| @@ -2569,7 +2567,6 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) · 397 unmodified lines | |||
| 2569 | 2567 | ||
| 2570 | pe = xchg(&apic->pending_events, 0); | 2568 | pe = xchg(&apic->pending_events, 0); |
| 2571 | if (test_bit(KVM_APIC_INIT, &pe)) { | 2569 | if (test_bit(KVM_APIC_INIT, &pe)) { |
| 2572 | kvm_lapic_reset(vcpu, true); | ||
| 2573 | kvm_vcpu_reset(vcpu, true); | 2570 | kvm_vcpu_reset(vcpu, true); |
| 2574 | if (kvm_vcpu_is_bsp(apic->vcpu)) | 2571 | if (kvm_vcpu_is_bsp(apic->vcpu)) |
| 2575 | vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; | 2572 | vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; |
| @@ -2770,8 +2770,10 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep, · 2769 unmodified lines | |||
| 2770 | else | 2770 | else |
| 2771 | pte_access &= ~ACC_WRITE_MASK; | 2771 | pte_access &= ~ACC_WRITE_MASK; |
| 2772 | 2772 | ||
| 2773 | if (!kvm_is_mmio_pfn(pfn)) | ||
| 2774 | spte |= shadow_me_mask; | ||
| 2775 | |||
| 2773 | spte |= (u64)pfn << PAGE_SHIFT; | 2776 | spte |= (u64)pfn << PAGE_SHIFT; |
| 2774 | spte |= shadow_me_mask; | ||
| 2775 | 2777 | ||
| 2776 | if (pte_access & ACC_WRITE_MASK) { | 2778 | if (pte_access & ACC_WRITE_MASK) { |
| 2777 | 2779 | ||
| @@ -3029,7 +3031,7 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · 251 unmodified lines | |||
| 3029 | return RET_PF_RETRY; | 3031 | return RET_PF_RETRY; |
| 3030 | } | 3032 | } |
| 3031 | 3033 | ||
| 3032 | return -EFAULT; | 3034 | return RET_PF_EMULATE; |
| 3033 | } | 3035 | } |
| 3034 | 3036 | ||
| 3035 | static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, | 3037 | static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, |
| @@ -5080,7 +5082,7 @@ void kvm_mmu_uninit_vm(struct kvm *kvm) · 2044 unmodified lines | |||
| 5080 | typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); | 5082 | typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); |
| 5081 | 5083 | ||
| 5082 | /* The caller should hold mmu-lock before calling this function. */ | 5084 | /* The caller should hold mmu-lock before calling this function. */ |
| 5083 | static bool | 5085 | static __always_inline bool |
| 5084 | slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5086 | slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5085 | slot_level_handler fn, int start_level, int end_level, | 5087 | slot_level_handler fn, int start_level, int end_level, |
| 5086 | gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb) | 5088 | gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb) |
| @@ -5110,7 +5112,7 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, · 23 unmodified lines | |||
| 5110 | return flush; | 5112 | return flush; |
| 5111 | } | 5113 | } |
| 5112 | 5114 | ||
| 5113 | static bool | 5115 | static __always_inline bool |
| 5114 | slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5116 | slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5115 | slot_level_handler fn, int start_level, int end_level, | 5117 | slot_level_handler fn, int start_level, int end_level, |
| 5116 | bool lock_flush_tlb) | 5118 | bool lock_flush_tlb) |
| @@ -5121,7 +5123,7 @@ slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, · 4 unmodified lines | |||
| 5121 | lock_flush_tlb); | 5123 | lock_flush_tlb); |
| 5122 | } | 5124 | } |
| 5123 | 5125 | ||
| 5124 | static bool | 5126 | static __always_inline bool |
| 5125 | slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5127 | slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5126 | slot_level_handler fn, bool lock_flush_tlb) | 5128 | slot_level_handler fn, bool lock_flush_tlb) |
| 5127 | { | 5129 | { |
| @@ -5129,7 +5131,7 @@ slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, · 1 unmodified line | |||
| 5129 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); | 5131 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); |
| 5130 | } | 5132 | } |
| 5131 | 5133 | ||
| 5132 | static bool | 5134 | static __always_inline bool |
| 5133 | slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5135 | slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5134 | slot_level_handler fn, bool lock_flush_tlb) | 5136 | slot_level_handler fn, bool lock_flush_tlb) |
| 5135 | { | 5137 | { |
| @@ -5137,7 +5139,7 @@ slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, · 1 unmodified line | |||
| 5137 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); | 5139 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); |
| 5138 | } | 5140 | } |
| 5139 | 5141 | ||
| 5140 | static bool | 5142 | static __always_inline bool |
| 5141 | slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5143 | slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5142 | slot_level_handler fn, bool lock_flush_tlb) | 5144 | slot_level_handler fn, bool lock_flush_tlb) |
| 5143 | { | 5145 | { |
| @@ -49,6 +49,7 @@ · 48 unmodified lines | |||
| 49 | #include <asm/debugreg.h> | 49 | #include <asm/debugreg.h> |
| 50 | #include <asm/kvm_para.h> | 50 | #include <asm/kvm_para.h> |
| 51 | #include <asm/irq_remapping.h> | 51 | #include <asm/irq_remapping.h> |
| 52 | #include <asm/microcode.h> | ||
| 52 | #include <asm/nospec-branch.h> | 53 | #include <asm/nospec-branch.h> |
| 53 | 54 | ||
| 54 | #include <asm/virtext.h> | 55 | #include <asm/virtext.h> |
| @@ -178,6 +179,8 @@ struct vcpu_svm { · 123 unmodified lines | |||
| 178 | uint64_t sysenter_eip; | 179 | uint64_t sysenter_eip; |
| 179 | uint64_t tsc_aux; | 180 | uint64_t tsc_aux; |
| 180 | 181 | ||
| 182 | u64 msr_decfg; | ||
| 183 | |||
| 181 | u64 next_rip; | 184 | u64 next_rip; |
| 182 | 185 | ||
| 183 | u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS]; | 186 | u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS]; |
| @@ -300,6 +303,8 @@ module_param(vgif, int, 0444); · 116 unmodified lines | |||
| 300 | static int sev = IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT); | 303 | static int sev = IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT); |
| 301 | module_param(sev, int, 0444); | 304 | module_param(sev, int, 0444); |
| 302 | 305 | ||
| 306 | static u8 rsm_ins_bytes[] = "\x0f\xaa"; | ||
| 307 | |||
| 303 | static void svm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); | 308 | static void svm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); |
| 304 | static void svm_flush_tlb(struct kvm_vcpu *vcpu, bool invalidate_gpa); | 309 | static void svm_flush_tlb(struct kvm_vcpu *vcpu, bool invalidate_gpa); |
| 305 | static void svm_complete_interrupts(struct vcpu_svm *svm); | 310 | static void svm_complete_interrupts(struct vcpu_svm *svm); |
| @@ -1383,6 +1388,7 @@ static void init_vmcb(struct vcpu_svm *svm) · 1077 unmodified lines | |||
| 1383 | set_intercept(svm, INTERCEPT_SKINIT); | 1388 | set_intercept(svm, INTERCEPT_SKINIT); |
| 1384 | set_intercept(svm, INTERCEPT_WBINVD); | 1389 | set_intercept(svm, INTERCEPT_WBINVD); |
| 1385 | set_intercept(svm, INTERCEPT_XSETBV); | 1390 | set_intercept(svm, INTERCEPT_XSETBV); |
| 1391 | set_intercept(svm, INTERCEPT_RSM); | ||
| 1386 | 1392 | ||
| 1387 | if (!kvm_mwait_in_guest()) { | 1393 | if (!kvm_mwait_in_guest()) { |
| 1388 | set_intercept(svm, INTERCEPT_MONITOR); | 1394 | set_intercept(svm, INTERCEPT_MONITOR); |
| @@ -1902,6 +1908,7 @@ static void svm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 513 unmodified lines | |||
| 1902 | u32 dummy; | 1908 | u32 dummy; |
| 1903 | u32 eax = 1; | 1909 | u32 eax = 1; |
| 1904 | 1910 | ||
| 1911 | vcpu->arch.microcode_version = 0x01000065; | ||
| 1905 | svm->spec_ctrl = 0; | 1912 | svm->spec_ctrl = 0; |
| 1906 | 1913 | ||
| 1907 | if (!init_event) { | 1914 | if (!init_event) { |
| @@ -3699,6 +3706,12 @@ static int emulate_on_interception(struct vcpu_svm *svm) · 1791 unmodified lines | |||
| 3699 | return emulate_instruction(&svm->vcpu, 0) == EMULATE_DONE; | 3706 | return emulate_instruction(&svm->vcpu, 0) == EMULATE_DONE; |
| 3700 | } | 3707 | } |
| 3701 | 3708 | ||
| 3709 | static int rsm_interception(struct vcpu_svm *svm) | ||
| 3710 | { | ||
| 3711 | return x86_emulate_instruction(&svm->vcpu, 0, 0, | ||
| 3712 | rsm_ins_bytes, 2) == EMULATE_DONE; | ||
| 3713 | } | ||
| 3714 | |||
| 3702 | static int rdpmc_interception(struct vcpu_svm *svm) | 3715 | static int rdpmc_interception(struct vcpu_svm *svm) |
| 3703 | { | 3716 | { |
| 3704 | int err; | 3717 | int err; |
| @@ -3860,6 +3873,22 @@ static int cr8_write_interception(struct vcpu_svm *svm) · 155 unmodified lines | |||
| 3860 | return 0; | 3873 | return 0; |
| 3861 | } | 3874 | } |
| 3862 | 3875 | ||
| 3876 | static int svm_get_msr_feature(struct kvm_msr_entry *msr) | ||
| 3877 | { | ||
| 3878 | msr->data = 0; | ||
| 3879 | |||
| 3880 | switch (msr->index) { | ||
| 3881 | case MSR_F10H_DECFG: | ||
| 3882 | if (boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) | ||
| 3883 | msr->data |= MSR_F10H_DECFG_LFENCE_SERIALIZE; | ||
| 3884 | break; | ||
| 3885 | default: | ||
| 3886 | return 1; | ||
| 3887 | } | ||
| 3888 | |||
| 3889 | return 0; | ||
| 3890 | } | ||
| 3891 | |||
| 3863 | static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) | 3892 | static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) |
| 3864 | { | 3893 | { |
| 3865 | struct vcpu_svm *svm = to_svm(vcpu); | 3894 | struct vcpu_svm *svm = to_svm(vcpu); |
| @@ -3935,9 +3964,6 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 69 unmodified lines | |||
| 3935 | 3964 | ||
| 3936 | msr_info->data = svm->spec_ctrl; | 3965 | msr_info->data = svm->spec_ctrl; |
| 3937 | break; | 3966 | break; |
| 3938 | case MSR_IA32_UCODE_REV: | ||
| 3939 | msr_info->data = 0x01000065; | ||
| 3940 | break; | ||
| 3941 | case MSR_F15H_IC_CFG: { | 3967 | case MSR_F15H_IC_CFG: { |
| 3942 | 3968 | ||
| 3943 | int family, model; | 3969 | int family, model; |
| @@ -3955,6 +3981,9 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 11 unmodified lines | |||
| 3955 | msr_info->data = 0x1E; | 3981 | msr_info->data = 0x1E; |
| 3956 | } | 3982 | } |
| 3957 | break; | 3983 | break; |
| 3984 | case MSR_F10H_DECFG: | ||
| 3985 | msr_info->data = svm->msr_decfg; | ||
| 3986 | break; | ||
| 3958 | default: | 3987 | default: |
| 3959 | return kvm_get_msr_common(vcpu, msr_info); | 3988 | return kvm_get_msr_common(vcpu, msr_info); |
| 3960 | } | 3989 | } |
| @@ -4133,6 +4162,24 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) · 172 unmodified lines | |||
| 4133 | case MSR_VM_IGNNE: | 4162 | case MSR_VM_IGNNE: |
| 4134 | vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data); | 4163 | vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data); |
| 4135 | break; | 4164 | break; |
| 4165 | case MSR_F10H_DECFG: { | ||
| 4166 | struct kvm_msr_entry msr_entry; | ||
| 4167 | |||
| 4168 | msr_entry.index = msr->index; | ||
| 4169 | if (svm_get_msr_feature(&msr_entry)) | ||
| 4170 | return 1; | ||
| 4171 | |||
| 4172 | /* Check the supported bits */ | ||
| 4173 | if (data & ~msr_entry.data) | ||
| 4174 | return 1; | ||
| 4175 | |||
| 4176 | /* Don't allow the guest to change a bit, #GP */ | ||
| 4177 | if (!msr->host_initiated && (data ^ msr_entry.data)) | ||
| 4178 | return 1; | ||
| 4179 | |||
| 4180 | svm->msr_decfg = data; | ||
| 4181 | break; | ||
| 4182 | } | ||
| 4136 | case MSR_IA32_APICBASE: | 4183 | case MSR_IA32_APICBASE: |
| 4137 | if (kvm_vcpu_apicv_active(vcpu)) | 4184 | if (kvm_vcpu_apicv_active(vcpu)) |
| 4138 | avic_update_vapic_bar(to_svm(vcpu), data); | 4185 | avic_update_vapic_bar(to_svm(vcpu), data); |
| @@ -4541,7 +4588,7 @@ static int (*const svm_exit_handlers[])(struct vcpu_svm *svm) = { · 402 unmodified lines | |||
| 4541 | [SVM_EXIT_MWAIT] = mwait_interception, | 4588 | [SVM_EXIT_MWAIT] = mwait_interception, |
| 4542 | [SVM_EXIT_XSETBV] = xsetbv_interception, | 4589 | [SVM_EXIT_XSETBV] = xsetbv_interception, |
| 4543 | [SVM_EXIT_NPF] = npf_interception, | 4590 | [SVM_EXIT_NPF] = npf_interception, |
| 4544 | [SVM_EXIT_RSM] = emulate_on_interception, | 4591 | [SVM_EXIT_RSM] = rsm_interception, |
| 4545 | [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception, | 4592 | [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception, |
| 4546 | [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception, | 4593 | [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception, |
| 4547 | }; | 4594 | }; |
| @@ -5355,7 +5402,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu) · 807 unmodified lines | |||
| 5355 | * being speculatively taken. | 5402 | * being speculatively taken. |
| 5356 | */ | 5403 | */ |
| 5357 | if (svm->spec_ctrl) | 5404 | if (svm->spec_ctrl) |
| 5358 | wrmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl); | 5405 | native_wrmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl); |
| 5359 | 5406 | ||
| 5360 | asm volatile ( | 5407 | asm volatile ( |
| 5361 | "push %%" _ASM_BP "; \n\t" | 5408 | "push %%" _ASM_BP "; \n\t" |
| @@ -5464,11 +5511,11 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu) · 102 unmodified lines | |||
| 5464 | * If the L02 MSR bitmap does not intercept the MSR, then we need to | 5511 | * If the L02 MSR bitmap does not intercept the MSR, then we need to |
| 5465 | * save it. | 5512 | * save it. |
| 5466 | */ | 5513 | */ |
| 5467 | if (!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)) | 5514 | if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL))) |
| 5468 | rdmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl); | 5515 | svm->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL); |
| 5469 | 5516 | ||
| 5470 | if (svm->spec_ctrl) | 5517 | if (svm->spec_ctrl) |
| 5471 | wrmsrl(MSR_IA32_SPEC_CTRL, 0); | 5518 | native_wrmsrl(MSR_IA32_SPEC_CTRL, 0); |
| 5472 | 5519 | ||
| 5473 | /* Eliminate branch target predictions from guest mode */ | 5520 | /* Eliminate branch target predictions from guest mode */ |
| 5474 | vmexit_fill_RSB(); | 5521 | vmexit_fill_RSB(); |
| @@ -6236,16 +6283,18 @@ e_free: · 761 unmodified lines | |||
| 6236 | 6283 | ||
| 6237 | static int sev_launch_measure(struct kvm *kvm, struct kvm_sev_cmd *argp) | 6284 | static int sev_launch_measure(struct kvm *kvm, struct kvm_sev_cmd *argp) |
| 6238 | { | 6285 | { |
| 6286 | void __user *measure = (void __user *)(uintptr_t)argp->data; | ||
| 6239 | struct kvm_sev_info *sev = &kvm->arch.sev_info; | 6287 | struct kvm_sev_info *sev = &kvm->arch.sev_info; |
| 6240 | struct sev_data_launch_measure *data; | 6288 | struct sev_data_launch_measure *data; |
| 6241 | struct kvm_sev_launch_measure params; | 6289 | struct kvm_sev_launch_measure params; |
| 6290 | void __user *p = NULL; | ||
| 6242 | void *blob = NULL; | 6291 | void *blob = NULL; |
| 6243 | int ret; | 6292 | int ret; |
| 6244 | 6293 | ||
| 6245 | if (!sev_guest(kvm)) | 6294 | if (!sev_guest(kvm)) |
| 6246 | return -ENOTTY; | 6295 | return -ENOTTY; |
| 6247 | 6296 | ||
| 6248 | if (copy_from_user(¶ms, (void __user *)(uintptr_t)argp->data, sizeof(params))) | 6297 | if (copy_from_user(¶ms, measure, sizeof(params))) |
| 6249 | return -EFAULT; | 6298 | return -EFAULT; |
| 6250 | 6299 | ||
| 6251 | data = kzalloc(sizeof(*data), GFP_KERNEL); | 6300 | data = kzalloc(sizeof(*data), GFP_KERNEL); |
| @@ -6256,17 +6305,13 @@ static int sev_launch_measure(struct kvm *kvm, struct kvm_sev_cmd *argp) · 4 unmodified lines | |||
| 6256 | if (!params.len) | 6305 | if (!params.len) |
| 6257 | goto cmd; | 6306 | goto cmd; |
| 6258 | 6307 | ||
| 6259 | if (params.uaddr) { | 6308 | p = (void __user *)(uintptr_t)params.uaddr; |
| 6309 | if (p) { | ||
| 6260 | if (params.len > SEV_FW_BLOB_MAX_SIZE) { | 6310 | if (params.len > SEV_FW_BLOB_MAX_SIZE) { |
| 6261 | ret = -EINVAL; | 6311 | ret = -EINVAL; |
| 6262 | goto e_free; | 6312 | goto e_free; |
| 6263 | } | 6313 | } |
| 6264 | 6314 | ||
| 6265 | if (!access_ok(VERIFY_WRITE, params.uaddr, params.len)) { | ||
| 6266 | ret = -EFAULT; | ||
| 6267 | goto e_free; | ||
| 6268 | } | ||
| 6269 | |||
| 6270 | ret = -ENOMEM; | 6315 | ret = -ENOMEM; |
| 6271 | blob = kmalloc(params.len, GFP_KERNEL); | 6316 | blob = kmalloc(params.len, GFP_KERNEL); |
| 6272 | if (!blob) | 6317 | if (!blob) |
| @@ -6290,13 +6335,13 @@ cmd: · 17 unmodified lines | |||
| 6290 | goto e_free_blob; | 6335 | goto e_free_blob; |
| 6291 | 6336 | ||
| 6292 | if (blob) { | 6337 | if (blob) { |
| 6293 | if (copy_to_user((void __user *)(uintptr_t)params.uaddr, blob, params.len)) | 6338 | if (copy_to_user(p, blob, params.len)) |
| 6294 | ret = -EFAULT; | 6339 | ret = -EFAULT; |
| 6295 | } | 6340 | } |
| 6296 | 6341 | ||
| 6297 | done: | 6342 | done: |
| 6298 | params.len = data->len; | 6343 | params.len = data->len; |
| 6299 | if (copy_to_user((void __user *)(uintptr_t)argp->data, ¶ms, sizeof(params))) | 6344 | if (copy_to_user(measure, ¶ms, sizeof(params))) |
| 6300 | ret = -EFAULT; | 6345 | ret = -EFAULT; |
| 6301 | e_free_blob: | 6346 | e_free_blob: |
| 6302 | kfree(blob); | 6347 | kfree(blob); |
| @@ -6597,7 +6642,7 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) · 294 unmodified lines | |||
| 6597 | struct page **pages; | 6642 | struct page **pages; |
| 6598 | void *blob, *hdr; | 6643 | void *blob, *hdr; |
| 6599 | unsigned long n; | 6644 | unsigned long n; |
| 6600 | int ret; | 6645 | int ret, offset; |
| 6601 | 6646 | ||
| 6602 | if (!sev_guest(kvm)) | 6647 | if (!sev_guest(kvm)) |
| 6603 | return -ENOTTY; | 6648 | return -ENOTTY; |
| @@ -6623,6 +6668,10 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) · 19 unmodified lines | |||
| 6623 | if (!data) | 6668 | if (!data) |
| 6624 | goto e_unpin_memory; | 6669 | goto e_unpin_memory; |
| 6625 | 6670 | ||
| 6671 | offset = params.guest_uaddr & (PAGE_SIZE - 1); | ||
| 6672 | data->guest_address = __sme_page_pa(pages[0]) + offset; | ||
| 6673 | data->guest_len = params.guest_len; | ||
| 6674 | |||
| 6626 | blob = psp_copy_user_blob(params.trans_uaddr, params.trans_len); | 6675 | blob = psp_copy_user_blob(params.trans_uaddr, params.trans_len); |
| 6627 | if (IS_ERR(blob)) { | 6676 | if (IS_ERR(blob)) { |
| 6628 | ret = PTR_ERR(blob); | 6677 | ret = PTR_ERR(blob); |
| @@ -6637,8 +6686,8 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) · 8 unmodified lines | |||
| 6637 | ret = PTR_ERR(hdr); | 6686 | ret = PTR_ERR(hdr); |
| 6638 | goto e_free_blob; | 6687 | goto e_free_blob; |
| 6639 | } | 6688 | } |
| 6640 | data->trans_address = __psp_pa(blob); | 6689 | data->hdr_address = __psp_pa(hdr); |
| 6641 | data->trans_len = params.trans_len; | 6690 | data->hdr_len = params.hdr_len; |
| 6642 | 6691 | ||
| 6643 | data->handle = sev->handle; | 6692 | data->handle = sev->handle; |
| 6644 | ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_SECRET, data, &argp->error); | 6693 | ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_SECRET, data, &argp->error); |
| @@ -6821,6 +6870,7 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = { · 176 unmodified lines | |||
| 6821 | .vcpu_unblocking = svm_vcpu_unblocking, | 6870 | .vcpu_unblocking = svm_vcpu_unblocking, |
| 6822 | 6871 | ||
| 6823 | .update_bp_intercept = update_bp_intercept, | 6872 | .update_bp_intercept = update_bp_intercept, |
| 6873 | .get_msr_feature = svm_get_msr_feature, | ||
| 6824 | .get_msr = svm_get_msr, | 6874 | .get_msr = svm_get_msr, |
| 6825 | .set_msr = svm_set_msr, | 6875 | .set_msr = svm_set_msr, |
| 6826 | .get_segment_base = svm_get_segment_base, | 6876 | .get_segment_base = svm_get_segment_base, |
| @@ -51,6 +51,7 @@ · 50 unmodified lines | |||
| 51 | #include <asm/apic.h> | 51 | #include <asm/apic.h> |
| 52 | #include <asm/irq_remapping.h> | 52 | #include <asm/irq_remapping.h> |
| 53 | #include <asm/mmu_context.h> | 53 | #include <asm/mmu_context.h> |
| 54 | #include <asm/microcode.h> | ||
| 54 | #include <asm/nospec-branch.h> | 55 | #include <asm/nospec-branch.h> |
| 55 | 56 | ||
| 56 | #include "trace.h" | 57 | #include "trace.h" |
| @@ -1044,6 +1045,13 @@ static inline bool is_machine_check(u32 intr_info) · 987 unmodified lines | |||
| 1044 | (INTR_TYPE_HARD_EXCEPTION | MC_VECTOR | INTR_INFO_VALID_MASK); | 1045 | (INTR_TYPE_HARD_EXCEPTION | MC_VECTOR | INTR_INFO_VALID_MASK); |
| 1045 | } | 1046 | } |
| 1046 | 1047 | ||
| 1048 | /* Undocumented: icebp/int1 */ | ||
| 1049 | static inline bool is_icebp(u32 intr_info) | ||
| 1050 | { | ||
| 1051 | return (intr_info & (INTR_INFO_INTR_TYPE_MASK | INTR_INFO_VALID_MASK)) | ||
| 1052 | == (INTR_TYPE_PRIV_SW_EXCEPTION | INTR_INFO_VALID_MASK); | ||
| 1053 | } | ||
| 1054 | |||
| 1047 | static inline bool cpu_has_vmx_msr_bitmap(void) | 1055 | static inline bool cpu_has_vmx_msr_bitmap(void) |
| 1048 | { | 1056 | { |
| 1049 | return vmcs_config.cpu_based_exec_ctrl & CPU_BASED_USE_MSR_BITMAPS; | 1057 | return vmcs_config.cpu_based_exec_ctrl & CPU_BASED_USE_MSR_BITMAPS; |
| @@ -3226,6 +3234,11 @@ static inline bool vmx_feature_control_msr_valid(struct kvm_vcpu *vcpu, · 2176 unmodified lines | |||
| 3226 | return !(val & ~valid_bits); | 3234 | return !(val & ~valid_bits); |
| 3227 | } | 3235 | } |
| 3228 | 3236 | ||
| 3237 | static int vmx_get_msr_feature(struct kvm_msr_entry *msr) | ||
| 3238 | { | ||
| 3239 | return 1; | ||
| 3240 | } | ||
| 3241 | |||
| 3229 | /* | 3242 | /* |
| 3230 | * Reads an msr value (of 'msr_index') into 'pdata'. | 3243 | * Reads an msr value (of 'msr_index') into 'pdata'. |
| 3231 | * Returns 0 on success, non-0 otherwise. | 3244 | * Returns 0 on success, non-0 otherwise. |
| @@ -4485,7 +4498,8 @@ static int vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) · 1253 unmodified lines | |||
| 4485 | vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL, | 4498 | vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL, |
| 4486 | SECONDARY_EXEC_DESC); | 4499 | SECONDARY_EXEC_DESC); |
| 4487 | hw_cr4 &= ~X86_CR4_UMIP; | 4500 | hw_cr4 &= ~X86_CR4_UMIP; |
| 4488 | } else | 4501 | } else if (!is_guest_mode(vcpu) || |
| 4502 | !nested_cpu_has2(get_vmcs12(vcpu), SECONDARY_EXEC_DESC)) | ||
| 4489 | vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL, | 4503 | vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL, |
| 4490 | SECONDARY_EXEC_DESC); | 4504 | SECONDARY_EXEC_DESC); |
| 4491 | 4505 | ||
| @@ -5765,6 +5779,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 1273 unmodified lines | |||
| 5765 | vmx->rmode.vm86_active = 0; | 5779 | vmx->rmode.vm86_active = 0; |
| 5766 | vmx->spec_ctrl = 0; | 5780 | vmx->spec_ctrl = 0; |
| 5767 | 5781 | ||
| 5782 | vcpu->arch.microcode_version = 0x100000000ULL; | ||
| 5768 | vmx->vcpu.arch.regs[VCPU_REGS_RDX] = get_rdx_init_val(); | 5783 | vmx->vcpu.arch.regs[VCPU_REGS_RDX] = get_rdx_init_val(); |
| 5769 | kvm_set_cr8(vcpu, 0); | 5784 | kvm_set_cr8(vcpu, 0); |
| 5770 | 5785 | ||
| @@ -6171,7 +6186,7 @@ static int handle_exception(struct kvm_vcpu *vcpu) · 400 unmodified lines | |||
| 6171 | (KVM_GUESTDBG_SINGLESTEP | KVM_GUESTDBG_USE_HW_BP))) { | 6186 | (KVM_GUESTDBG_SINGLESTEP | KVM_GUESTDBG_USE_HW_BP))) { |
| 6172 | vcpu->arch.dr6 &= ~15; | 6187 | vcpu->arch.dr6 &= ~15; |
| 6173 | vcpu->arch.dr6 |= dr6 | DR6_RTM; | 6188 | vcpu->arch.dr6 |= dr6 | DR6_RTM; |
| 6174 | if (!(dr6 & ~DR6_RESERVED)) /* icebp */ | 6189 | if (is_icebp(intr_info)) |
| 6175 | skip_emulated_instruction(vcpu); | 6190 | skip_emulated_instruction(vcpu); |
| 6176 | 6191 | ||
| 6177 | kvm_queue_exception(vcpu, DB_VECTOR); | 6192 | kvm_queue_exception(vcpu, DB_VECTOR); |
| @@ -9452,7 +9467,7 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) · 3274 unmodified lines | |||
| 9452 | * being speculatively taken. | 9467 | * being speculatively taken. |
| 9453 | */ | 9468 | */ |
| 9454 | if (vmx->spec_ctrl) | 9469 | if (vmx->spec_ctrl) |
| 9455 | wrmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl); | 9470 | native_wrmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl); |
| 9456 | 9471 | ||
| 9457 | vmx->__launched = vmx->loaded_vmcs->launched; | 9472 | vmx->__launched = vmx->loaded_vmcs->launched; |
| 9458 | asm( | 9473 | asm( |
| @@ -9587,11 +9602,11 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) · 128 unmodified lines | |||
| 9587 | * If the L02 MSR bitmap does not intercept the MSR, then we need to | 9602 | * If the L02 MSR bitmap does not intercept the MSR, then we need to |
| 9588 | * save it. | 9603 | * save it. |
| 9589 | */ | 9604 | */ |
| 9590 | if (!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)) | 9605 | if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL))) |
| 9591 | rdmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl); | 9606 | vmx->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL); |
| 9592 | 9607 | ||
| 9593 | if (vmx->spec_ctrl) | 9608 | if (vmx->spec_ctrl) |
| 9594 | wrmsrl(MSR_IA32_SPEC_CTRL, 0); | 9609 | native_wrmsrl(MSR_IA32_SPEC_CTRL, 0); |
| 9595 | 9610 | ||
| 9596 | /* Eliminate branch target predictions from guest mode */ | 9611 | /* Eliminate branch target predictions from guest mode */ |
| 9597 | vmexit_fill_RSB(); | 9612 | vmexit_fill_RSB(); |
| @@ -10136,7 +10151,10 @@ static void nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, · 538 unmodified lines | |||
| 10136 | (unsigned long)(vmcs12->posted_intr_desc_addr & | 10151 | (unsigned long)(vmcs12->posted_intr_desc_addr & |
| 10137 | (PAGE_SIZE - 1))); | 10152 | (PAGE_SIZE - 1))); |
| 10138 | } | 10153 | } |
| 10139 | if (!nested_vmx_prepare_msr_bitmap(vcpu, vmcs12)) | 10154 | if (nested_vmx_prepare_msr_bitmap(vcpu, vmcs12)) |
| 10155 | vmcs_set_bits(CPU_BASED_VM_EXEC_CONTROL, | ||
| 10156 | CPU_BASED_USE_MSR_BITMAPS); | ||
| 10157 | else | ||
| 10140 | vmcs_clear_bits(CPU_BASED_VM_EXEC_CONTROL, | 10158 | vmcs_clear_bits(CPU_BASED_VM_EXEC_CONTROL, |
| 10141 | CPU_BASED_USE_MSR_BITMAPS); | 10159 | CPU_BASED_USE_MSR_BITMAPS); |
| 10142 | } | 10160 | } |
| @@ -10224,8 +10242,8 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, · 81 unmodified lines | |||
| 10224 | * updated to reflect this when L1 (or its L2s) actually write to | 10242 | * updated to reflect this when L1 (or its L2s) actually write to |
| 10225 | * the MSR. | 10243 | * the MSR. |
| 10226 | */ | 10244 | */ |
| 10227 | bool pred_cmd = msr_write_intercepted_l01(vcpu, MSR_IA32_PRED_CMD); | 10245 | bool pred_cmd = !msr_write_intercepted_l01(vcpu, MSR_IA32_PRED_CMD); |
| 10228 | bool spec_ctrl = msr_write_intercepted_l01(vcpu, MSR_IA32_SPEC_CTRL); | 10246 | bool spec_ctrl = !msr_write_intercepted_l01(vcpu, MSR_IA32_SPEC_CTRL); |
| 10229 | 10247 | ||
| 10230 | /* Nothing to do if the MSR bitmap is not in use. */ | 10248 | /* Nothing to do if the MSR bitmap is not in use. */ |
| 10231 | if (!cpu_has_vmx_msr_bitmap() || | 10249 | if (!cpu_has_vmx_msr_bitmap() || |
| @@ -11196,7 +11214,12 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) · 964 unmodified lines | |||
| 11196 | if (ret) | 11214 | if (ret) |
| 11197 | return ret; | 11215 | return ret; |
| 11198 | 11216 | ||
| 11199 | if (vmcs12->guest_activity_state == GUEST_ACTIVITY_HLT) | 11217 | /* |
| 11218 | * If we're entering a halted L2 vcpu and the L2 vcpu won't be woken | ||
| 11219 | * by event injection, halt vcpu. | ||
| 11220 | */ | ||
| 11221 | if ((vmcs12->guest_activity_state == GUEST_ACTIVITY_HLT) && | ||
| 11222 | !(vmcs12->vm_entry_intr_info_field & INTR_INFO_VALID_MASK)) | ||
| 11200 | return kvm_vcpu_halt(vcpu); | 11223 | return kvm_vcpu_halt(vcpu); |
| 11201 | 11224 | ||
| 11202 | vmx->nested.nested_run_pending = 1; | 11225 | vmx->nested.nested_run_pending = 1; |
| @@ -12287,6 +12310,7 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { · 1084 unmodified lines | |||
| 12287 | .vcpu_put = vmx_vcpu_put, | 12310 | .vcpu_put = vmx_vcpu_put, |
| 12288 | 12311 | ||
| 12289 | .update_bp_intercept = update_exception_bitmap, | 12312 | .update_bp_intercept = update_exception_bitmap, |
| 12313 | .get_msr_feature = vmx_get_msr_feature, | ||
| 12290 | .get_msr = vmx_get_msr, | 12314 | .get_msr = vmx_get_msr, |
| 12291 | .set_msr = vmx_set_msr, | 12315 | .set_msr = vmx_set_msr, |
| 12292 | .get_segment_base = vmx_get_segment_base, | 12316 | .get_segment_base = vmx_get_segment_base, |
| @@ -1049,6 +1049,45 @@ static u32 emulated_msrs[] = { · 1048 unmodified lines | |||
| 1049 | 1049 | ||
| 1050 | static unsigned num_emulated_msrs; | 1050 | static unsigned num_emulated_msrs; |
| 1051 | 1051 | ||
| 1052 | /* | ||
| 1053 | * List of msr numbers which are used to expose MSR-based features that | ||
| 1054 | * can be used by a hypervisor to validate requested CPU features. | ||
| 1055 | */ | ||
| 1056 | static u32 msr_based_features[] = { | ||
| 1057 | MSR_F10H_DECFG, | ||
| 1058 | MSR_IA32_UCODE_REV, | ||
| 1059 | }; | ||
| 1060 | |||
| 1061 | static unsigned int num_msr_based_features; | ||
| 1062 | |||
| 1063 | static int kvm_get_msr_feature(struct kvm_msr_entry *msr) | ||
| 1064 | { | ||
| 1065 | switch (msr->index) { | ||
| 1066 | case MSR_IA32_UCODE_REV: | ||
| 1067 | rdmsrl(msr->index, msr->data); | ||
| 1068 | break; | ||
| 1069 | default: | ||
| 1070 | if (kvm_x86_ops->get_msr_feature(msr)) | ||
| 1071 | return 1; | ||
| 1072 | } | ||
| 1073 | return 0; | ||
| 1074 | } | ||
| 1075 | |||
| 1076 | static int do_get_msr_feature(struct kvm_vcpu *vcpu, unsigned index, u64 *data) | ||
| 1077 | { | ||
| 1078 | struct kvm_msr_entry msr; | ||
| 1079 | int r; | ||
| 1080 | |||
| 1081 | msr.index = index; | ||
| 1082 | r = kvm_get_msr_feature(&msr); | ||
| 1083 | if (r) | ||
| 1084 | return r; | ||
| 1085 | |||
| 1086 | *data = msr.data; | ||
| 1087 | |||
| 1088 | return 0; | ||
| 1089 | } | ||
| 1090 | |||
| 1052 | bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) | 1091 | bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) |
| 1053 | { | 1092 | { |
| 1054 | if (efer & efer_reserved_bits) | 1093 | if (efer & efer_reserved_bits) |
| @@ -2222,7 +2261,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 1167 unmodified lines | |||
| 2222 | 2261 | ||
| 2223 | switch (msr) { | 2262 | switch (msr) { |
| 2224 | case MSR_AMD64_NB_CFG: | 2263 | case MSR_AMD64_NB_CFG: |
| 2225 | case MSR_IA32_UCODE_REV: | ||
| 2226 | case MSR_IA32_UCODE_WRITE: | 2264 | case MSR_IA32_UCODE_WRITE: |
| 2227 | case MSR_VM_HSAVE_PA: | 2265 | case MSR_VM_HSAVE_PA: |
| 2228 | case MSR_AMD64_PATCH_LOADER: | 2266 | case MSR_AMD64_PATCH_LOADER: |
| @@ -2230,6 +2268,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 1 unmodified line | |||
| 2230 | case MSR_AMD64_DC_CFG: | 2268 | case MSR_AMD64_DC_CFG: |
| 2231 | break; | 2269 | break; |
| 2232 | 2270 | ||
| 2271 | case MSR_IA32_UCODE_REV: | ||
| 2272 | if (msr_info->host_initiated) | ||
| 2273 | vcpu->arch.microcode_version = data; | ||
| 2274 | break; | ||
| 2233 | case MSR_EFER: | 2275 | case MSR_EFER: |
| 2234 | return set_efer(vcpu, data); | 2276 | return set_efer(vcpu, data); |
| 2235 | case MSR_K7_HWCR: | 2277 | case MSR_K7_HWCR: |
| @@ -2525,7 +2567,7 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 289 unmodified lines | |||
| 2525 | msr_info->data = 0; | 2567 | msr_info->data = 0; |
| 2526 | break; | 2568 | break; |
| 2527 | case MSR_IA32_UCODE_REV: | 2569 | case MSR_IA32_UCODE_REV: |
| 2528 | msr_info->data = 0x100000000ULL; | 2570 | msr_info->data = vcpu->arch.microcode_version; |
| 2529 | break; | 2571 | break; |
| 2530 | case MSR_MTRRcap: | 2572 | case MSR_MTRRcap: |
| 2531 | case 0x200 ... 0x2ff: | 2573 | case 0x200 ... 0x2ff: |
| @@ -2680,13 +2722,11 @@ static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, · 148 unmodified lines | |||
| 2680 | int (*do_msr)(struct kvm_vcpu *vcpu, | 2722 | int (*do_msr)(struct kvm_vcpu *vcpu, |
| 2681 | unsigned index, u64 *data)) | 2723 | unsigned index, u64 *data)) |
| 2682 | { | 2724 | { |
| 2683 | int i, idx; | 2725 | int i; |
| 2684 | 2726 | ||
| 2685 | idx = srcu_read_lock(&vcpu->kvm->srcu); | ||
| 2686 | for (i = 0; i < msrs->nmsrs; ++i) | 2727 | for (i = 0; i < msrs->nmsrs; ++i) |
| 2687 | if (do_msr(vcpu, entries[i].index, &entries[i].data)) | 2728 | if (do_msr(vcpu, entries[i].index, &entries[i].data)) |
| 2688 | break; | 2729 | break; |
| 2689 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | ||
| 2690 | 2730 | ||
| 2691 | return i; | 2731 | return i; |
| 2692 | } | 2732 | } |
| @@ -2785,6 +2825,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 92 unmodified lines | |||
| 2785 | case KVM_CAP_SET_BOOT_CPU_ID: | 2825 | case KVM_CAP_SET_BOOT_CPU_ID: |
| 2786 | case KVM_CAP_SPLIT_IRQCHIP: | 2826 | case KVM_CAP_SPLIT_IRQCHIP: |
| 2787 | case KVM_CAP_IMMEDIATE_EXIT: | 2827 | case KVM_CAP_IMMEDIATE_EXIT: |
| 2828 | case KVM_CAP_GET_MSR_FEATURES: | ||
| 2788 | r = 1; | 2829 | r = 1; |
| 2789 | break; | 2830 | break; |
| 2790 | case KVM_CAP_ADJUST_CLOCK: | 2831 | case KVM_CAP_ADJUST_CLOCK: |
| @@ -2899,6 +2940,31 @@ long kvm_arch_dev_ioctl(struct file *filp, · 108 unmodified lines | |||
| 2899 | goto out; | 2940 | goto out; |
| 2900 | r = 0; | 2941 | r = 0; |
| 2901 | break; | 2942 | break; |
| 2943 | case KVM_GET_MSR_FEATURE_INDEX_LIST: { | ||
| 2944 | struct kvm_msr_list __user *user_msr_list = argp; | ||
| 2945 | struct kvm_msr_list msr_list; | ||
| 2946 | unsigned int n; | ||
| 2947 | |||
| 2948 | r = -EFAULT; | ||
| 2949 | if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) | ||
| 2950 | goto out; | ||
| 2951 | n = msr_list.nmsrs; | ||
| 2952 | msr_list.nmsrs = num_msr_based_features; | ||
| 2953 | if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) | ||
| 2954 | goto out; | ||
| 2955 | r = -E2BIG; | ||
| 2956 | if (n < msr_list.nmsrs) | ||
| 2957 | goto out; | ||
| 2958 | r = -EFAULT; | ||
| 2959 | if (copy_to_user(user_msr_list->indices, &msr_based_features, | ||
| 2960 | num_msr_based_features * sizeof(u32))) | ||
| 2961 | goto out; | ||
| 2962 | r = 0; | ||
| 2963 | break; | ||
| 2964 | } | ||
| 2965 | case KVM_GET_MSRS: | ||
| 2966 | r = msr_io(NULL, argp, do_get_msr_feature, 1); | ||
| 2967 | break; | ||
| 2902 | } | 2968 | } |
| 2903 | default: | 2969 | default: |
| 2904 | r = -EINVAL; | 2970 | r = -EINVAL; |
| @@ -3636,12 +3702,18 @@ long kvm_arch_vcpu_ioctl(struct file *filp, · 731 unmodified lines | |||
| 3636 | r = 0; | 3702 | r = 0; |
| 3637 | break; | 3703 | break; |
| 3638 | } | 3704 | } |
| 3639 | case KVM_GET_MSRS: | 3705 | case KVM_GET_MSRS: { |
| 3706 | int idx = srcu_read_lock(&vcpu->kvm->srcu); | ||
| 3640 | r = msr_io(vcpu, argp, do_get_msr, 1); | 3707 | r = msr_io(vcpu, argp, do_get_msr, 1); |
| 3708 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | ||
| 3641 | break; | 3709 | break; |
| 3642 | case KVM_SET_MSRS: | 3710 | } |
| 3711 | case KVM_SET_MSRS: { | ||
| 3712 | int idx = srcu_read_lock(&vcpu->kvm->srcu); | ||
| 3643 | r = msr_io(vcpu, argp, do_set_msr, 0); | 3713 | r = msr_io(vcpu, argp, do_set_msr, 0); |
| 3714 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | ||
| 3644 | break; | 3715 | break; |
| 3716 | } | ||
| 3645 | case KVM_TPR_ACCESS_REPORTING: { | 3717 | case KVM_TPR_ACCESS_REPORTING: { |
| 3646 | struct kvm_tpr_access_ctl tac; | 3718 | struct kvm_tpr_access_ctl tac; |
| 3647 | 3719 | ||
| @@ -4464,6 +4536,19 @@ static void kvm_init_msr_list(void) · 816 unmodified lines | |||
| 4464 | j++; | 4536 | j++; |
| 4465 | } | 4537 | } |
| 4466 | num_emulated_msrs = j; | 4538 | num_emulated_msrs = j; |
| 4539 | |||
| 4540 | for (i = j = 0; i < ARRAY_SIZE(msr_based_features); i++) { | ||
| 4541 | struct kvm_msr_entry msr; | ||
| 4542 | |||
| 4543 | msr.index = msr_based_features[i]; | ||
| 4544 | if (kvm_get_msr_feature(&msr)) | ||
| 4545 | continue; | ||
| 4546 | |||
| 4547 | if (j < i) | ||
| 4548 | msr_based_features[j] = msr_based_features[i]; | ||
| 4549 | j++; | ||
| 4550 | } | ||
| 4551 | num_msr_based_features = j; | ||
| 4467 | } | 4552 | } |
| 4468 | 4553 | ||
| 4469 | static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, | 4554 | static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, |
| @@ -8017,6 +8102,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) · 3547 unmodified lines | |||
| 8017 | 8102 | ||
| 8018 | void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) | 8103 | void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) |
| 8019 | { | 8104 | { |
| 8105 | kvm_lapic_reset(vcpu, init_event); | ||
| 8106 | |||
| 8020 | vcpu->arch.hflags = 0; | 8107 | vcpu->arch.hflags = 0; |
| 8021 | 8108 | ||
| 8022 | vcpu->arch.smi_pending = 0; | 8109 | vcpu->arch.smi_pending = 0; |
| @@ -8460,10 +8547,8 @@ int __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size) · 437 unmodified lines | |||
| 8460 | return r; | 8547 | return r; |
| 8461 | } | 8548 | } |
| 8462 | 8549 | ||
| 8463 | if (!size) { | 8550 | if (!size) |
| 8464 | r = vm_munmap(old.userspace_addr, old.npages * PAGE_SIZE); | 8551 | vm_munmap(old.userspace_addr, old.npages * PAGE_SIZE); |
| 8465 | WARN_ON(r < 0); | ||
| 8466 | } | ||
| 8467 | 8552 | ||
| 8468 | return 0; | 8553 | return 0; |
| 8469 | } | 8554 | } |
| @@ -28,7 +28,6 @@ lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o · 27 unmodified lines | |||
| 28 | lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o | 28 | lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o |
| 29 | lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o | 29 | lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o |
| 30 | lib-$(CONFIG_RETPOLINE) += retpoline.o | 30 | lib-$(CONFIG_RETPOLINE) += retpoline.o |
| 31 | OBJECT_FILES_NON_STANDARD_retpoline.o :=y | ||
| 32 | 31 | ||
| 33 | obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o | 32 | obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o |
| 34 | 33 | ||
| @@ -18,7 +18,7 @@ unsigned int x86_model(unsigned int sig) · 17 unmodified lines | |||
| 18 | { | 18 | { |
| 19 | unsigned int fam, model; | 19 | unsigned int fam, model; |
| 20 | 20 | ||
| 21 | fam = x86_family(sig); | 21 | fam = x86_family(sig); |
| 22 | 22 | ||
| 23 | model = (sig >> 4) & 0xf; | 23 | model = (sig >> 4) & 0xf; |
| 24 | 24 | ||
| @@ -7,6 +7,7 @@ asmlinkage void just_return_func(void); · 6 unmodified lines | |||
| 7 | 7 | ||
| 8 | asm( | 8 | asm( |
| 9 | ".type just_return_func, @function\n" | 9 | ".type just_return_func, @function\n" |
| 10 | ".globl just_return_func\n" | ||
| 10 | "just_return_func:\n" | 11 | "just_return_func:\n" |
| 11 | " ret\n" | 12 | " ret\n" |
| 12 | ".size just_return_func, .-just_return_func\n" | 13 | ".size just_return_func, .-just_return_func\n" |
| @@ -7,7 +7,6 @@ · 6 unmodified lines | |||
| 7 | #include <asm/alternative-asm.h> | 7 | #include <asm/alternative-asm.h> |
| 8 | #include <asm/export.h> | 8 | #include <asm/export.h> |
| 9 | #include <asm/nospec-branch.h> | 9 | #include <asm/nospec-branch.h> |
| 10 | #include <asm/bitsperlong.h> | ||
| 11 | 10 | ||
| 12 | .macro THUNK reg | 11 | .macro THUNK reg |
| 13 | .section .text.__x86.indirect_thunk | 12 | .section .text.__x86.indirect_thunk |
| @@ -47,58 +46,3 @@ GENERATE_THUNK(r13) · 33 unmodified lines | |||
| 47 | GENERATE_THUNK(r14) | 46 | GENERATE_THUNK(r14) |
| 48 | GENERATE_THUNK(r15) | 47 | GENERATE_THUNK(r15) |
| 49 | #endif | 48 | #endif |
| 50 | |||
| 51 | /* | ||
| 52 | * Fill the CPU return stack buffer. | ||
| 53 | * | ||
| 54 | * Each entry in the RSB, if used for a speculative 'ret', contains an | ||
| 55 | * infinite 'pause; lfence; jmp' loop to capture speculative execution. | ||
| 56 | * | ||
| 57 | * This is required in various cases for retpoline and IBRS-based | ||
| 58 | * mitigations for the Spectre variant 2 vulnerability. Sometimes to | ||
| 59 | * eliminate potentially bogus entries from the RSB, and sometimes | ||
| 60 | * purely to ensure that it doesn't get empty, which on some CPUs would | ||
| 61 | * allow predictions from other (unwanted!) sources to be used. | ||
| 62 | * | ||
| 63 | * Google experimented with loop-unrolling and this turned out to be | ||
| 64 | * the optimal version - two calls, each with their own speculation | ||
| 65 | * trap should their return address end up getting used, in a loop. | ||
| 66 | */ | ||
| 67 | .macro STUFF_RSB nr:req sp:req | ||
| 68 | mov $(\nr / 2), %_ASM_BX | ||
| 69 | .align 16 | ||
| 70 | 771: | ||
| 71 | call 772f | ||
| 72 | 773: /* speculation trap */ | ||
| 73 | pause | ||
| 74 | lfence | ||
| 75 | jmp 773b | ||
| 76 | .align 16 | ||
| 77 | 772: | ||
| 78 | call 774f | ||
| 79 | 775: /* speculation trap */ | ||
| 80 | pause | ||
| 81 | lfence | ||
| 82 | jmp 775b | ||
| 83 | .align 16 | ||
| 84 | 774: | ||
| 85 | dec %_ASM_BX | ||
| 86 | jnz 771b | ||
| 87 | add $((BITS_PER_LONG/8) * \nr), \sp | ||
| 88 | .endm | ||
| 89 | |||
| 90 | #define RSB_FILL_LOOPS 16 /* To avoid underflow */ | ||
| 91 | |||
| 92 | ENTRY(__fill_rsb) | ||
| 93 | STUFF_RSB RSB_FILL_LOOPS, %_ASM_SP | ||
| 94 | ret | ||
| 95 | END(__fill_rsb) | ||
| 96 | EXPORT_SYMBOL_GPL(__fill_rsb) | ||
| 97 | |||
| 98 | #define RSB_CLEAR_LOOPS 32 /* To forcibly overwrite all entries */ | ||
| 99 | |||
| 100 | ENTRY(__clear_rsb) | ||
| 101 | STUFF_RSB RSB_CLEAR_LOOPS, %_ASM_SP | ||
| 102 | ret | ||
| 103 | END(__clear_rsb) | ||
| 104 | EXPORT_SYMBOL_GPL(__clear_rsb) | ||
| @@ -163,4 +163,10 @@ void __init setup_cpu_entry_areas(void) · 162 unmodified lines | |||
| 163 | 163 | ||
| 164 | for_each_possible_cpu(cpu) | 164 | for_each_possible_cpu(cpu) |
| 165 | setup_cpu_entry_area(cpu); | 165 | setup_cpu_entry_area(cpu); |
| 166 | |||
| 167 | /* | ||
| 168 | * This is the last essential update to swapper_pgdir which needs | ||
| 169 | * to be synchronized to initial_page_table on 32bit. | ||
| 170 | */ | ||
| 171 | sync_initial_page_table(); | ||
| 166 | } | 172 | } |
| @@ -330,7 +330,7 @@ static noinline int vmalloc_fault(unsigned long address) · 329 unmodified lines | |||
| 330 | if (!pmd_k) | 330 | if (!pmd_k) |
| 331 | return -1; | 331 | return -1; |
| 332 | 332 | ||
| 333 | if (pmd_huge(*pmd_k)) | 333 | if (pmd_large(*pmd_k)) |
| 334 | return 0; | 334 | return 0; |
| 335 | 335 | ||
| 336 | pte_k = pte_offset_kernel(pmd_k, address); | 336 | pte_k = pte_offset_kernel(pmd_k, address); |
| @@ -475,7 +475,7 @@ static noinline int vmalloc_fault(unsigned long address) · 138 unmodified lines | |||
| 475 | if (pud_none(*pud) || pud_pfn(*pud) != pud_pfn(*pud_ref)) | 475 | if (pud_none(*pud) || pud_pfn(*pud) != pud_pfn(*pud_ref)) |
| 476 | BUG(); | 476 | BUG(); |
| 477 | 477 | ||
| 478 | if (pud_huge(*pud)) | 478 | if (pud_large(*pud)) |
| 479 | return 0; | 479 | return 0; |
| 480 | 480 | ||
| 481 | pmd = pmd_offset(pud, address); | 481 | pmd = pmd_offset(pud, address); |
| @@ -486,7 +486,7 @@ static noinline int vmalloc_fault(unsigned long address) · 4 unmodified lines | |||
| 486 | if (pmd_none(*pmd) || pmd_pfn(*pmd) != pmd_pfn(*pmd_ref)) | 486 | if (pmd_none(*pmd) || pmd_pfn(*pmd) != pmd_pfn(*pmd_ref)) |
| 487 | BUG(); | 487 | BUG(); |
| 488 | 488 | ||
| 489 | if (pmd_huge(*pmd)) | 489 | if (pmd_large(*pmd)) |
| 490 | return 0; | 490 | return 0; |
| 491 | 491 | ||
| 492 | pte_ref = pte_offset_kernel(pmd_ref, address); | 492 | pte_ref = pte_offset_kernel(pmd_ref, address); |
| @@ -1248,10 +1248,6 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, · 755 unmodified lines | |||
| 1248 | tsk = current; | 1248 | tsk = current; |
| 1249 | mm = tsk->mm; | 1249 | mm = tsk->mm; |
| 1250 | 1250 | ||
| 1251 | /* | ||
| 1252 | * Detect and handle instructions that would cause a page fault for | ||
| 1253 | * both a tracked kernel page and a userspace page. | ||
| 1254 | */ | ||
| 1255 | prefetchw(&mm->mmap_sem); | 1251 | prefetchw(&mm->mmap_sem); |
| 1256 | 1252 | ||
| 1257 | if (unlikely(kmmio_fault(regs, address))) | 1253 | if (unlikely(kmmio_fault(regs, address))) |
| @@ -453,6 +453,21 @@ static inline void permanent_kmaps_init(pgd_t *pgd_base) · 452 unmodified lines | |||
| 453 | } | 453 | } |
| 454 | #endif /* CONFIG_HIGHMEM */ | 454 | #endif /* CONFIG_HIGHMEM */ |
| 455 | 455 | ||
| 456 | void __init sync_initial_page_table(void) | ||
| 457 | { | ||
| 458 | clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY, | ||
| 459 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 460 | KERNEL_PGD_PTRS); | ||
| 461 | |||
| 462 | /* | ||
| 463 | * sync back low identity map too. It is used for example | ||
| 464 | * in the 32-bit EFI stub. | ||
| 465 | */ | ||
| 466 | clone_pgd_range(initial_page_table, | ||
| 467 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 468 | min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY)); | ||
| 469 | } | ||
| 470 | |||
| 456 | void __init native_pagetable_init(void) | 471 | void __init native_pagetable_init(void) |
| 457 | { | 472 | { |
| 458 | unsigned long pfn, va; | 473 | unsigned long pfn, va; |
| @@ -256,7 +256,7 @@ static void __set_pte_vaddr(pud_t *pud, unsigned long vaddr, pte_t new_pte) · 255 unmodified lines | |||
| 256 | * It's enough to flush this one mapping. | 256 | * It's enough to flush this one mapping. |
| 257 | * (PGE mappings get flushed as well) | 257 | * (PGE mappings get flushed as well) |
| 258 | */ | 258 | */ |
| 259 | __flush_tlb_one(vaddr); | 259 | __flush_tlb_one_kernel(vaddr); |
| 260 | } | 260 | } |
| 261 | 261 | ||
| 262 | void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte) | 262 | void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte) |
| @@ -800,17 +800,11 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, · 537 unmodified lines | |||
| 800 | 800 | ||
| 801 | #define PAGE_INUSE 0xFD | 801 | #define PAGE_INUSE 0xFD |
| 802 | 802 | ||
| 803 | static void __meminit free_pagetable(struct page *page, int order, | 803 | static void __meminit free_pagetable(struct page *page, int order) |
| 804 | struct vmem_altmap *altmap) | ||
| 805 | { | 804 | { |
| 806 | unsigned long magic; | 805 | unsigned long magic; |
| 807 | unsigned int nr_pages = 1 << order; | 806 | unsigned int nr_pages = 1 << order; |
| 808 | 807 | ||
| 809 | if (altmap) { | ||
| 810 | vmem_altmap_free(altmap, nr_pages); | ||
| 811 | return; | ||
| 812 | } | ||
| 813 | |||
| 814 | /* bootmem page has reserved flag */ | 808 | /* bootmem page has reserved flag */ |
| 815 | if (PageReserved(page)) { | 809 | if (PageReserved(page)) { |
| 816 | __ClearPageReserved(page); | 810 | __ClearPageReserved(page); |
| @@ -826,8 +820,16 @@ static void __meminit free_pagetable(struct page *page, int order, · 9 unmodified lines | |||
| 826 | free_pages((unsigned long)page_address(page), order); | 820 | free_pages((unsigned long)page_address(page), order); |
| 827 | } | 821 | } |
| 828 | 822 | ||
| 829 | static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd, | 823 | static void __meminit free_hugepage_table(struct page *page, |
| 830 | struct vmem_altmap *altmap) | 824 | struct vmem_altmap *altmap) |
| 825 | { | ||
| 826 | if (altmap) | ||
| 827 | vmem_altmap_free(altmap, PMD_SIZE / PAGE_SIZE); | ||
| 828 | else | ||
| 829 | free_pagetable(page, get_order(PMD_SIZE)); | ||
| 830 | } | ||
| 831 | |||
| 832 | static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) | ||
| 831 | { | 833 | { |
| 832 | pte_t *pte; | 834 | pte_t *pte; |
| 833 | int i; | 835 | int i; |
| @@ -839,14 +841,13 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd, · 5 unmodified lines | |||
| 839 | } | 841 | } |
| 840 | 842 | ||
| 841 | /* free a pte talbe */ | 843 | /* free a pte talbe */ |
| 842 | free_pagetable(pmd_page(*pmd), 0, altmap); | 844 | free_pagetable(pmd_page(*pmd), 0); |
| 843 | spin_lock(&init_mm.page_table_lock); | 845 | spin_lock(&init_mm.page_table_lock); |
| 844 | pmd_clear(pmd); | 846 | pmd_clear(pmd); |
| 845 | spin_unlock(&init_mm.page_table_lock); | 847 | spin_unlock(&init_mm.page_table_lock); |
| 846 | } | 848 | } |
| 847 | 849 | ||
| 848 | static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, | 850 | static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud) |
| 849 | struct vmem_altmap *altmap) | ||
| 850 | { | 851 | { |
| 851 | pmd_t *pmd; | 852 | pmd_t *pmd; |
| 852 | int i; | 853 | int i; |
| @@ -858,14 +859,13 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, · 5 unmodified lines | |||
| 858 | } | 859 | } |
| 859 | 860 | ||
| 860 | /* free a pmd talbe */ | 861 | /* free a pmd talbe */ |
| 861 | free_pagetable(pud_page(*pud), 0, altmap); | 862 | free_pagetable(pud_page(*pud), 0); |
| 862 | spin_lock(&init_mm.page_table_lock); | 863 | spin_lock(&init_mm.page_table_lock); |
| 863 | pud_clear(pud); | 864 | pud_clear(pud); |
| 864 | spin_unlock(&init_mm.page_table_lock); | 865 | spin_unlock(&init_mm.page_table_lock); |
| 865 | } | 866 | } |
| 866 | 867 | ||
| 867 | static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d, | 868 | static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d) |
| 868 | struct vmem_altmap *altmap) | ||
| 869 | { | 869 | { |
| 870 | pud_t *pud; | 870 | pud_t *pud; |
| 871 | int i; | 871 | int i; |
| @@ -877,7 +877,7 @@ static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d, · 5 unmodified lines | |||
| 877 | } | 877 | } |
| 878 | 878 | ||
| 879 | /* free a pud talbe */ | 879 | /* free a pud talbe */ |
| 880 | free_pagetable(p4d_page(*p4d), 0, altmap); | 880 | free_pagetable(p4d_page(*p4d), 0); |
| 881 | spin_lock(&init_mm.page_table_lock); | 881 | spin_lock(&init_mm.page_table_lock); |
| 882 | p4d_clear(p4d); | 882 | p4d_clear(p4d); |
| 883 | spin_unlock(&init_mm.page_table_lock); | 883 | spin_unlock(&init_mm.page_table_lock); |
| @@ -885,7 +885,7 @@ static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d, · 1 unmodified line | |||
| 885 | 885 | ||
| 886 | static void __meminit | 886 | static void __meminit |
| 887 | remove_pte_table(pte_t *pte_start, unsigned long addr, unsigned long end, | 887 | remove_pte_table(pte_t *pte_start, unsigned long addr, unsigned long end, |
| 888 | struct vmem_altmap *altmap, bool direct) | 888 | bool direct) |
| 889 | { | 889 | { |
| 890 | unsigned long next, pages = 0; | 890 | unsigned long next, pages = 0; |
| 891 | pte_t *pte; | 891 | pte_t *pte; |
| @@ -916,7 +916,7 @@ remove_pte_table(pte_t *pte_start, unsigned long addr, unsigned long end, · 24 unmodified lines | |||
| 916 | * freed when offlining, or simplely not in use. | 916 | * freed when offlining, or simplely not in use. |
| 917 | */ | 917 | */ |
| 918 | if (!direct) | 918 | if (!direct) |
| 919 | free_pagetable(pte_page(*pte), 0, altmap); | 919 | free_pagetable(pte_page(*pte), 0); |
| 920 | 920 | ||
| 921 | spin_lock(&init_mm.page_table_lock); | 921 | spin_lock(&init_mm.page_table_lock); |
| 922 | pte_clear(&init_mm, addr, pte); | 922 | pte_clear(&init_mm, addr, pte); |
| @@ -939,7 +939,7 @@ remove_pte_table(pte_t *pte_start, unsigned long addr, unsigned long end, · 16 unmodified lines | |||
| 939 | 939 | ||
| 940 | page_addr = page_address(pte_page(*pte)); | 940 | page_addr = page_address(pte_page(*pte)); |
| 941 | if (!memchr_inv(page_addr, PAGE_INUSE, PAGE_SIZE)) { | 941 | if (!memchr_inv(page_addr, PAGE_INUSE, PAGE_SIZE)) { |
| 942 | free_pagetable(pte_page(*pte), 0, altmap); | 942 | free_pagetable(pte_page(*pte), 0); |
| 943 | 943 | ||
| 944 | spin_lock(&init_mm.page_table_lock); | 944 | spin_lock(&init_mm.page_table_lock); |
| 945 | pte_clear(&init_mm, addr, pte); | 945 | pte_clear(&init_mm, addr, pte); |
| @@ -974,9 +974,8 @@ remove_pmd_table(pmd_t *pmd_start, unsigned long addr, unsigned long end, · 28 unmodified lines | |||
| 974 | if (IS_ALIGNED(addr, PMD_SIZE) && | 974 | if (IS_ALIGNED(addr, PMD_SIZE) && |
| 975 | IS_ALIGNED(next, PMD_SIZE)) { | 975 | IS_ALIGNED(next, PMD_SIZE)) { |
| 976 | if (!direct) | 976 | if (!direct) |
| 977 | free_pagetable(pmd_page(*pmd), | 977 | free_hugepage_table(pmd_page(*pmd), |
| 978 | get_order(PMD_SIZE), | 978 | altmap); |
| 979 | altmap); | ||
| 980 | 979 | ||
| 981 | spin_lock(&init_mm.page_table_lock); | 980 | spin_lock(&init_mm.page_table_lock); |
| 982 | pmd_clear(pmd); | 981 | pmd_clear(pmd); |
| @@ -989,9 +988,8 @@ remove_pmd_table(pmd_t *pmd_start, unsigned long addr, unsigned long end, · 6 unmodified lines | |||
| 989 | page_addr = page_address(pmd_page(*pmd)); | 988 | page_addr = page_address(pmd_page(*pmd)); |
| 990 | if (!memchr_inv(page_addr, PAGE_INUSE, | 989 | if (!memchr_inv(page_addr, PAGE_INUSE, |
| 991 | PMD_SIZE)) { | 990 | PMD_SIZE)) { |
| 992 | free_pagetable(pmd_page(*pmd), | 991 | free_hugepage_table(pmd_page(*pmd), |
| 993 | get_order(PMD_SIZE), | 992 | altmap); |
| 994 | altmap); | ||
| 995 | 993 | ||
| 996 | spin_lock(&init_mm.page_table_lock); | 994 | spin_lock(&init_mm.page_table_lock); |
| 997 | pmd_clear(pmd); | 995 | pmd_clear(pmd); |
| @@ -1003,8 +1001,8 @@ remove_pmd_table(pmd_t *pmd_start, unsigned long addr, unsigned long end, · 5 unmodified lines | |||
| 1003 | } | 1001 | } |
| 1004 | 1002 | ||
| 1005 | pte_base = (pte_t *)pmd_page_vaddr(*pmd); | 1003 | pte_base = (pte_t *)pmd_page_vaddr(*pmd); |
| 1006 | remove_pte_table(pte_base, addr, next, altmap, direct); | 1004 | remove_pte_table(pte_base, addr, next, direct); |
| 1007 | free_pte_table(pte_base, pmd, altmap); | 1005 | free_pte_table(pte_base, pmd); |
| 1008 | } | 1006 | } |
| 1009 | 1007 | ||
| 1010 | /* Call free_pmd_table() in remove_pud_table(). */ | 1008 | /* Call free_pmd_table() in remove_pud_table(). */ |
| @@ -1033,8 +1031,7 @@ remove_pud_table(pud_t *pud_start, unsigned long addr, unsigned long end, · 22 unmodified lines | |||
| 1033 | IS_ALIGNED(next, PUD_SIZE)) { | 1031 | IS_ALIGNED(next, PUD_SIZE)) { |
| 1034 | if (!direct) | 1032 | if (!direct) |
| 1035 | free_pagetable(pud_page(*pud), | 1033 | free_pagetable(pud_page(*pud), |
| 1036 | get_order(PUD_SIZE), | 1034 | get_order(PUD_SIZE)); |
| 1037 | altmap); | ||
| 1038 | 1035 | ||
| 1039 | spin_lock(&init_mm.page_table_lock); | 1036 | spin_lock(&init_mm.page_table_lock); |
| 1040 | pud_clear(pud); | 1037 | pud_clear(pud); |
| @@ -1048,8 +1045,7 @@ remove_pud_table(pud_t *pud_start, unsigned long addr, unsigned long end, · 7 unmodified lines | |||
| 1048 | if (!memchr_inv(page_addr, PAGE_INUSE, | 1045 | if (!memchr_inv(page_addr, PAGE_INUSE, |
| 1049 | PUD_SIZE)) { | 1046 | PUD_SIZE)) { |
| 1050 | free_pagetable(pud_page(*pud), | 1047 | free_pagetable(pud_page(*pud), |
| 1051 | get_order(PUD_SIZE), | 1048 | get_order(PUD_SIZE)); |
| 1052 | altmap); | ||
| 1053 | 1049 | ||
| 1054 | spin_lock(&init_mm.page_table_lock); | 1050 | spin_lock(&init_mm.page_table_lock); |
| 1055 | pud_clear(pud); | 1051 | pud_clear(pud); |
| @@ -1062,7 +1058,7 @@ remove_pud_table(pud_t *pud_start, unsigned long addr, unsigned long end, · 6 unmodified lines | |||
| 1062 | 1058 | ||
| 1063 | pmd_base = pmd_offset(pud, 0); | 1059 | pmd_base = pmd_offset(pud, 0); |
| 1064 | remove_pmd_table(pmd_base, addr, next, direct, altmap); | 1060 | remove_pmd_table(pmd_base, addr, next, direct, altmap); |
| 1065 | free_pmd_table(pmd_base, pud, altmap); | 1061 | free_pmd_table(pmd_base, pud); |
| 1066 | } | 1062 | } |
| 1067 | 1063 | ||
| 1068 | if (direct) | 1064 | if (direct) |
| @@ -1094,7 +1090,7 @@ remove_p4d_table(p4d_t *p4d_start, unsigned long addr, unsigned long end, · 25 unmodified lines | |||
| 1094 | * to adapt for boot-time switching between 4 and 5 level page tables. | 1090 | * to adapt for boot-time switching between 4 and 5 level page tables. |
| 1095 | */ | 1091 | */ |
| 1096 | if (CONFIG_PGTABLE_LEVELS == 5) | 1092 | if (CONFIG_PGTABLE_LEVELS == 5) |
| 1097 | free_pud_table(pud_base, p4d, altmap); | 1093 | free_pud_table(pud_base, p4d); |
| 1098 | } | 1094 | } |
| 1099 | 1095 | ||
| 1100 | if (direct) | 1096 | if (direct) |
| @@ -1193,8 +1189,8 @@ void __init mem_init(void) · 92 unmodified lines | |||
| 1193 | register_page_bootmem_info(); | 1189 | register_page_bootmem_info(); |
| 1194 | 1190 | ||
| 1195 | /* Register memory areas for /proc/kcore */ | 1191 | /* Register memory areas for /proc/kcore */ |
| 1196 | kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, | 1192 | if (get_gate_vma(&init_mm)) |
| 1197 | PAGE_SIZE, KCORE_OTHER); | 1193 | kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, PAGE_SIZE, KCORE_USER); |
| 1198 | 1194 | ||
| 1199 | mem_init_print_info(NULL); | 1195 | mem_init_print_info(NULL); |
| 1200 | } | 1196 | } |
| @@ -820,5 +820,5 @@ void __init __early_set_fixmap(enum fixed_addresses idx, · 819 unmodified lines | |||
| 820 | set_pte(pte, pfn_pte(phys >> PAGE_SHIFT, flags)); | 820 | set_pte(pte, pfn_pte(phys >> PAGE_SHIFT, flags)); |
| 821 | else | 821 | else |
| 822 | pte_clear(&init_mm, addr, pte); | 822 | pte_clear(&init_mm, addr, pte); |
| 823 | __flush_tlb_one(addr); | 823 | __flush_tlb_one_kernel(addr); |
| 824 | } | 824 | } |
| @@ -168,7 +168,7 @@ static int clear_page_presence(struct kmmio_fault_page *f, bool clear) · 167 unmodified lines | |||
| 168 | return -1; | 168 | return -1; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | __flush_tlb_one(f->addr); | 171 | __flush_tlb_one_kernel(f->addr); |
| 172 | return 0; | 172 | return 0; |
| 173 | } | 173 | } |
| 174 | 174 | ||
| @@ -15,6 +15,7 @@ · 14 unmodified lines | |||
| 15 | #include <asm/page.h> | 15 | #include <asm/page.h> |
| 16 | #include <asm/processor-flags.h> | 16 | #include <asm/processor-flags.h> |
| 17 | #include <asm/msr-index.h> | 17 | #include <asm/msr-index.h> |
| 18 | #include <asm/nospec-branch.h> | ||
| 18 | 19 | ||
| 19 | .text | 20 | .text |
| 20 | .code64 | 21 | .code64 |
| @@ -59,6 +60,7 @@ ENTRY(sme_encrypt_execute) · 38 unmodified lines | |||
| 59 | movq %rax, %r8 /* Workarea encryption routine */ | 60 | movq %rax, %r8 /* Workarea encryption routine */ |
| 60 | addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ | 61 | addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ |
| 61 | 62 | ||
| 63 | ANNOTATE_RETPOLINE_SAFE | ||
| 62 | call *%rax /* Call the encryption routine */ | 64 | call *%rax /* Call the encryption routine */ |
| 63 | 65 | ||
| 64 | pop %r12 | 66 | pop %r12 |
| @@ -702,4 +702,52 @@ int pmd_clear_huge(pmd_t *pmd) · 701 unmodified lines | |||
| 702 | 702 | ||
| 703 | return 0; | 703 | return 0; |
| 704 | } | 704 | } |
| 705 | |||
| 706 | /** | ||
| 707 | * pud_free_pmd_page - Clear pud entry and free pmd page. | ||
| 708 | * @pud: Pointer to a PUD. | ||
| 709 | * | ||
| 710 | * Context: The pud range has been unmaped and TLB purged. | ||
| 711 | * Return: 1 if clearing the entry succeeded. 0 otherwise. | ||
| 712 | */ | ||
| 713 | int pud_free_pmd_page(pud_t *pud) | ||
| 714 | { | ||
| 715 | pmd_t *pmd; | ||
| 716 | int i; | ||
| 717 | |||
| 718 | if (pud_none(*pud)) | ||
| 719 | return 1; | ||
| 720 | |||
| 721 | pmd = (pmd_t *)pud_page_vaddr(*pud); | ||
| 722 | |||
| 723 | for (i = 0; i < PTRS_PER_PMD; i++) | ||
| 724 | if (!pmd_free_pte_page(&pmd[i])) | ||
| 725 | return 0; | ||
| 726 | |||
| 727 | pud_clear(pud); | ||
| 728 | free_page((unsigned long)pmd); | ||
| 729 | |||
| 730 | return 1; | ||
| 731 | } | ||
| 732 | |||
| 733 | /** | ||
| 734 | * pmd_free_pte_page - Clear pmd entry and free pte page. | ||
| 735 | * @pmd: Pointer to a PMD. | ||
| 736 | * | ||
| 737 | * Context: The pmd range has been unmaped and TLB purged. | ||
| 738 | * Return: 1 if clearing the entry succeeded. 0 otherwise. | ||
| 739 | */ | ||
| 740 | int pmd_free_pte_page(pmd_t *pmd) | ||
| 741 | { | ||
| 742 | pte_t *pte; | ||
| 743 | |||
| 744 | if (pmd_none(*pmd)) | ||
| 745 | return 1; | ||
| 746 | |||
| 747 | pte = (pte_t *)pmd_page_vaddr(*pmd); | ||
| 748 | pmd_clear(pmd); | ||
| 749 | free_page((unsigned long)pte); | ||
| 750 | |||
| 751 | return 1; | ||
| 752 | } | ||
| 705 | #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ | 753 | #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ |
| @@ -63,7 +63,7 @@ void set_pte_vaddr(unsigned long vaddr, pte_t pteval) · 62 unmodified lines | |||
| 63 | * It's enough to flush this one mapping. | 63 | * It's enough to flush this one mapping. |
| 64 | * (PGE mappings get flushed as well) | 64 | * (PGE mappings get flushed as well) |
| 65 | */ | 65 | */ |
| 66 | __flush_tlb_one(vaddr); | 66 | __flush_tlb_one_kernel(vaddr); |
| 67 | } | 67 | } |
| 68 | 68 | ||
| 69 | unsigned long __FIXADDR_TOP = 0xfffff000; | 69 | unsigned long __FIXADDR_TOP = 0xfffff000; |
| @@ -332,7 +332,7 @@ static void __init pti_clone_user_shared(void) · 331 unmodified lines | |||
| 332 | } | 332 | } |
| 333 | 333 | ||
| 334 | /* | 334 | /* |
| 335 | * Clone the ESPFIX P4D into the user space visinble page table | 335 | * Clone the ESPFIX P4D into the user space visible page table |
| 336 | */ | 336 | */ |
| 337 | static void __init pti_setup_espfix64(void) | 337 | static void __init pti_setup_espfix64(void) |
| 338 | { | 338 | { |
| @@ -498,7 +498,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, · 497 unmodified lines | |||
| 498 | * flush that changes context.tlb_gen from 2 to 3. If they get | 498 | * flush that changes context.tlb_gen from 2 to 3. If they get |
| 499 | * processed on this CPU in reverse order, we'll see | 499 | * processed on this CPU in reverse order, we'll see |
| 500 | * local_tlb_gen == 1, mm_tlb_gen == 3, and end != TLB_FLUSH_ALL. | 500 | * local_tlb_gen == 1, mm_tlb_gen == 3, and end != TLB_FLUSH_ALL. |
| 501 | * If we were to use __flush_tlb_single() and set local_tlb_gen to | 501 | * If we were to use __flush_tlb_one_user() and set local_tlb_gen to |
| 502 | * 3, we'd be break the invariant: we'd update local_tlb_gen above | 502 | * 3, we'd be break the invariant: we'd update local_tlb_gen above |
| 503 | * 1 without the full flush that's needed for tlb_gen 2. | 503 | * 1 without the full flush that's needed for tlb_gen 2. |
| 504 | * | 504 | * |
| @@ -519,7 +519,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, · 14 unmodified lines | |||
| 519 | 519 | ||
| 520 | addr = f->start; | 520 | addr = f->start; |
| 521 | while (addr < f->end) { | 521 | while (addr < f->end) { |
| 522 | __flush_tlb_single(addr); | 522 | __flush_tlb_one_user(addr); |
| 523 | addr += PAGE_SIZE; | 523 | addr += PAGE_SIZE; |
| 524 | } | 524 | } |
| 525 | if (local) | 525 | if (local) |
| @@ -666,7 +666,7 @@ static void do_kernel_range_flush(void *info) · 140 unmodified lines | |||
| 666 | 666 | ||
| 667 | /* flush range by one by one 'invlpg' */ | 667 | /* flush range by one by one 'invlpg' */ |
| 668 | for (addr = f->start; addr < f->end; addr += PAGE_SIZE) | 668 | for (addr = f->start; addr < f->end; addr += PAGE_SIZE) |
| 669 | __flush_tlb_one(addr); | 669 | __flush_tlb_one_kernel(addr); |
| 670 | } | 670 | } |
| 671 | 671 | ||
| 672 | void flush_tlb_kernel_range(unsigned long start, unsigned long end) | 672 | void flush_tlb_kernel_range(unsigned long start, unsigned long end) |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | |||
| 13 | #include <linux/if_vlan.h> | 13 | #include <linux/if_vlan.h> |
| 14 | #include <asm/cacheflush.h> | 14 | #include <asm/cacheflush.h> |
| 15 | #include <asm/set_memory.h> | 15 | #include <asm/set_memory.h> |
| 16 | #include <asm/nospec-branch.h> | ||
| 16 | #include <linux/bpf.h> | 17 | #include <linux/bpf.h> |
| 17 | 18 | ||
| 18 | /* | 19 | /* |
| @@ -290,7 +291,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 271 unmodified lines | |||
| 290 | EMIT2(0x89, 0xD2); /* mov edx, edx */ | 291 | EMIT2(0x89, 0xD2); /* mov edx, edx */ |
| 291 | EMIT3(0x39, 0x56, /* cmp dword ptr [rsi + 16], edx */ | 292 | EMIT3(0x39, 0x56, /* cmp dword ptr [rsi + 16], edx */ |
| 292 | offsetof(struct bpf_array, map.max_entries)); | 293 | offsetof(struct bpf_array, map.max_entries)); |
| 293 | #define OFFSET1 43 /* number of bytes to jump */ | 294 | #define OFFSET1 (41 + RETPOLINE_RAX_BPF_JIT_SIZE) /* number of bytes to jump */ |
| 294 | EMIT2(X86_JBE, OFFSET1); /* jbe out */ | 295 | EMIT2(X86_JBE, OFFSET1); /* jbe out */ |
| 295 | label1 = cnt; | 296 | label1 = cnt; |
| 296 | 297 | ||
| @@ -299,7 +300,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 2 unmodified lines | |||
| 299 | */ | 300 | */ |
| 300 | EMIT2_off32(0x8B, 0x85, 36); /* mov eax, dword ptr [rbp + 36] */ | 301 | EMIT2_off32(0x8B, 0x85, 36); /* mov eax, dword ptr [rbp + 36] */ |
| 301 | EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_CNT */ | 302 | EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_CNT */ |
| 302 | #define OFFSET2 32 | 303 | #define OFFSET2 (30 + RETPOLINE_RAX_BPF_JIT_SIZE) |
| 303 | EMIT2(X86_JA, OFFSET2); /* ja out */ | 304 | EMIT2(X86_JA, OFFSET2); /* ja out */ |
| 304 | label2 = cnt; | 305 | label2 = cnt; |
| 305 | EMIT3(0x83, 0xC0, 0x01); /* add eax, 1 */ | 306 | EMIT3(0x83, 0xC0, 0x01); /* add eax, 1 */ |
| @@ -313,7 +314,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 7 unmodified lines | |||
| 313 | * goto out; | 314 | * goto out; |
| 314 | */ | 315 | */ |
| 315 | EMIT3(0x48, 0x85, 0xC0); /* test rax,rax */ | 316 | EMIT3(0x48, 0x85, 0xC0); /* test rax,rax */ |
| 316 | #define OFFSET3 10 | 317 | #define OFFSET3 (8 + RETPOLINE_RAX_BPF_JIT_SIZE) |
| 317 | EMIT2(X86_JE, OFFSET3); /* je out */ | 318 | EMIT2(X86_JE, OFFSET3); /* je out */ |
| 318 | label3 = cnt; | 319 | label3 = cnt; |
| 319 | 320 | ||
| @@ -326,7 +327,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 6 unmodified lines | |||
| 326 | * rdi == ctx (1st arg) | 327 | * rdi == ctx (1st arg) |
| 327 | * rax == prog->bpf_func + prologue_size | 328 | * rax == prog->bpf_func + prologue_size |
| 328 | */ | 329 | */ |
| 329 | EMIT2(0xFF, 0xE0); /* jmp rax */ | 330 | RETPOLINE_RAX_BPF_JIT(); |
| 330 | 331 | ||
| 331 | /* out: */ | 332 | /* out: */ |
| 332 | BUILD_BUG_ON(cnt - label1 != OFFSET1); | 333 | BUILD_BUG_ON(cnt - label1 != OFFSET1); |
| @@ -1187,7 +1188,7 @@ skip_init_addrs: · 854 unmodified lines | |||
| 1187 | * may converge on the last pass. In such case do one more | 1188 | * may converge on the last pass. In such case do one more |
| 1188 | * pass to emit the final image | 1189 | * pass to emit the final image |
| 1189 | */ | 1190 | */ |
| 1190 | for (pass = 0; pass < 10 || image; pass++) { | 1191 | for (pass = 0; pass < 20 || image; pass++) { |
| 1191 | proglen = do_jit(prog, addrs, image, oldproglen, &ctx); | 1192 | proglen = do_jit(prog, addrs, image, oldproglen, &ctx); |
| 1192 | if (proglen <= 0) { | 1193 | if (proglen <= 0) { |
| 1193 | image = NULL; | 1194 | image = NULL; |
| @@ -1214,6 +1215,7 @@ skip_init_addrs: · 20 unmodified lines | |||
| 1214 | } | 1215 | } |
| 1215 | } | 1216 | } |
| 1216 | oldproglen = proglen; | 1217 | oldproglen = proglen; |
| 1218 | cond_resched(); | ||
| 1217 | } | 1219 | } |
| 1218 | 1220 | ||
| 1219 | if (bpf_jit_enable > 1) | 1221 | if (bpf_jit_enable > 1) |
| @@ -460,7 +460,7 @@ static int nmi_setup(void) · 459 unmodified lines | |||
| 460 | goto fail; | 460 | goto fail; |
| 461 | 461 | ||
| 462 | for_each_possible_cpu(cpu) { | 462 | for_each_possible_cpu(cpu) { |
| 463 | if (!cpu) | 463 | if (!IS_ENABLED(CONFIG_SMP) || !cpu) |
| 464 | continue; | 464 | continue; |
| 465 | 465 | ||
| 466 | memcpy(per_cpu(cpu_msrs, cpu).counters, | 466 | memcpy(per_cpu(cpu_msrs, cpu).counters, |
| @@ -227,7 +227,7 @@ int __init efi_alloc_page_tables(void) · 226 unmodified lines | |||
| 227 | if (!pud) { | 227 | if (!pud) { |
| 228 | if (CONFIG_PGTABLE_LEVELS > 4) | 228 | if (CONFIG_PGTABLE_LEVELS > 4) |
| 229 | free_page((unsigned long) pgd_page_vaddr(*pgd)); | 229 | free_page((unsigned long) pgd_page_vaddr(*pgd)); |
| 230 | free_page((unsigned long)efi_pgd); | 230 | free_pages((unsigned long)efi_pgd, PGD_ALLOCATION_ORDER); |
| 231 | return -ENOMEM; | 231 | return -ENOMEM; |
| 232 | } | 232 | } |
| 233 | 233 | ||
| @@ -79,7 +79,7 @@ static void intel_mid_power_off(void) · 78 unmodified lines | |||
| 79 | 79 | ||
| 80 | static void intel_mid_reboot(void) | 80 | static void intel_mid_reboot(void) |
| 81 | { | 81 | { |
| 82 | intel_scu_ipc_simple_command(IPCMSG_COLD_BOOT, 0); | 82 | intel_scu_ipc_simple_command(IPCMSG_COLD_RESET, 0); |
| 83 | } | 83 | } |
| 84 | 84 | ||
| 85 | static unsigned long __init intel_mid_calibrate_tsc(void) | 85 | static unsigned long __init intel_mid_calibrate_tsc(void) |
| @@ -299,7 +299,7 @@ static void bau_process_message(struct msg_desc *mdp, struct bau_control *bcp, · 298 unmodified lines | |||
| 299 | local_flush_tlb(); | 299 | local_flush_tlb(); |
| 300 | stat->d_alltlb++; | 300 | stat->d_alltlb++; |
| 301 | } else { | 301 | } else { |
| 302 | __flush_tlb_single(msg->address); | 302 | __flush_tlb_one_user(msg->address); |
| 303 | stat->d_onetlb++; | 303 | stat->d_onetlb++; |
| 304 | } | 304 | } |
| 305 | stat->d_requestee++; | 305 | stat->d_requestee++; |
| @@ -102,7 +102,7 @@ ENTRY(startup_32) · 101 unmodified lines | |||
| 102 | * don't we'll eventually crash trying to execute encrypted | 102 | * don't we'll eventually crash trying to execute encrypted |
| 103 | * instructions. | 103 | * instructions. |
| 104 | */ | 104 | */ |
| 105 | bt $TH_FLAGS_SME_ACTIVE_BIT, pa_tr_flags | 105 | btl $TH_FLAGS_SME_ACTIVE_BIT, pa_tr_flags |
| 106 | jnc .Ldone | 106 | jnc .Ldone |
| 107 | movl $MSR_K8_SYSCFG, %ecx | 107 | movl $MSR_K8_SYSCFG, %ecx |
| 108 | rdmsr | 108 | rdmsr |
| @@ -770,9 +770,12 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, · 769 unmodified lines | |||
| 770 | break; | 770 | break; |
| 771 | 771 | ||
| 772 | case R_X86_64_PC32: | 772 | case R_X86_64_PC32: |
| 773 | case R_X86_64_PLT32: | ||
| 773 | /* | 774 | /* |
| 774 | * PC relative relocations don't need to be adjusted unless | 775 | * PC relative relocations don't need to be adjusted unless |
| 775 | * referencing a percpu symbol. | 776 | * referencing a percpu symbol. |
| 777 | * | ||
| 778 | * NB: R_X86_64_PLT32 can be treated as R_X86_64_PC32. | ||
| 776 | */ | 779 | */ |
| 777 | if (is_percpu_sym(sym, symname)) | 780 | if (is_percpu_sym(sym, symname)) |
| 778 | add_reloc(&relocs32neg, offset); | 781 | add_reloc(&relocs32neg, offset); |
| @@ -30,11 +30,7 @@ · 29 unmodified lines | |||
| 30 | 30 | ||
| 31 | #endif /* CONFIG_X86_32 */ | 31 | #endif /* CONFIG_X86_32 */ |
| 32 | 32 | ||
| 33 | #ifdef CONFIG_X86_PPRO_FENCE | ||
| 34 | #define dma_rmb() rmb() | ||
| 35 | #else /* CONFIG_X86_PPRO_FENCE */ | ||
| 36 | #define dma_rmb() barrier() | 33 | #define dma_rmb() barrier() |
| 37 | #endif /* CONFIG_X86_PPRO_FENCE */ | ||
| 38 | #define dma_wmb() barrier() | 34 | #define dma_wmb() barrier() |
| 39 | 35 | ||
| 40 | #include <asm-generic/barrier.h> | 36 | #include <asm-generic/barrier.h> |
| @@ -1376,8 +1376,6 @@ asmlinkage __visible void __init xen_start_kernel(void) · 1375 unmodified lines | |||
| 1376 | 1376 | ||
| 1377 | if (!xen_initial_domain()) { | 1377 | if (!xen_initial_domain()) { |
| 1378 | add_preferred_console("xenboot", 0, NULL); | 1378 | add_preferred_console("xenboot", 0, NULL); |
| 1379 | add_preferred_console("tty", 0, NULL); | ||
| 1380 | add_preferred_console("hvc", 0, NULL); | ||
| 1381 | if (pci_xen) | 1379 | if (pci_xen) |
| 1382 | x86_init.pci.arch_init = pci_xen_init; | 1380 | x86_init.pci.arch_init = pci_xen_init; |
| 1383 | } else { | 1381 | } else { |
| @@ -1410,6 +1408,10 @@ asmlinkage __visible void __init xen_start_kernel(void) · 26 unmodified lines | |||
| 1410 | 1408 | ||
| 1411 | xen_boot_params_init_edd(); | 1409 | xen_boot_params_init_edd(); |
| 1412 | } | 1410 | } |
| 1411 | |||
| 1412 | add_preferred_console("tty", 0, NULL); | ||
| 1413 | add_preferred_console("hvc", 0, NULL); | ||
| 1414 | |||
| 1413 | #ifdef CONFIG_PCI | 1415 | #ifdef CONFIG_PCI |
| 1414 | /* PCI BIOS service won't work from a PV guest. */ | 1416 | /* PCI BIOS service won't work from a PV guest. */ |
| 1415 | pci_probe &= ~PCI_PROBE_BIOS; | 1417 | pci_probe &= ~PCI_PROBE_BIOS; |
| @@ -1300,12 +1300,12 @@ static void xen_flush_tlb(void) · 1299 unmodified lines | |||
| 1300 | preempt_enable(); | 1300 | preempt_enable(); |
| 1301 | } | 1301 | } |
| 1302 | 1302 | ||
| 1303 | static void xen_flush_tlb_single(unsigned long addr) | 1303 | static void xen_flush_tlb_one_user(unsigned long addr) |
| 1304 | { | 1304 | { |
| 1305 | struct mmuext_op *op; | 1305 | struct mmuext_op *op; |
| 1306 | struct multicall_space mcs; | 1306 | struct multicall_space mcs; |
| 1307 | 1307 | ||
| 1308 | trace_xen_mmu_flush_tlb_single(addr); | 1308 | trace_xen_mmu_flush_tlb_one_user(addr); |
| 1309 | 1309 | ||
| 1310 | preempt_disable(); | 1310 | preempt_disable(); |
| 1311 | 1311 | ||
| @@ -2370,7 +2370,7 @@ static const struct pv_mmu_ops xen_mmu_ops __initconst = { · 1058 unmodified lines | |||
| 2370 | 2370 | ||
| 2371 | .flush_tlb_user = xen_flush_tlb, | 2371 | .flush_tlb_user = xen_flush_tlb, |
| 2372 | .flush_tlb_kernel = xen_flush_tlb, | 2372 | .flush_tlb_kernel = xen_flush_tlb, |
| 2373 | .flush_tlb_single = xen_flush_tlb_single, | 2373 | .flush_tlb_one_user = xen_flush_tlb_one_user, |
| 2374 | .flush_tlb_others = xen_flush_tlb_others, | 2374 | .flush_tlb_others = xen_flush_tlb_others, |
| 2375 | 2375 | ||
| 2376 | .pgd_alloc = xen_pgd_alloc, | 2376 | .pgd_alloc = xen_pgd_alloc, |
| @@ -122,6 +122,8 @@ void __init xen_smp_cpus_done(unsigned int max_cpus) · 121 unmodified lines | |||
| 122 | 122 | ||
| 123 | if (xen_hvm_domain()) | 123 | if (xen_hvm_domain()) |
| 124 | native_smp_cpus_done(max_cpus); | 124 | native_smp_cpus_done(max_cpus); |
| 125 | else | ||
| 126 | calculate_max_logical_packages(); | ||
| 125 | 127 | ||
| 126 | if (xen_have_vcpu_info_placement) | 128 | if (xen_have_vcpu_info_placement) |
| 127 | return; | 129 | return; |
| @@ -1,12 +1,15 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | #include <linux/types.h> | 2 | #include <linux/types.h> |
| 3 | #include <linux/tick.h> | 3 | #include <linux/tick.h> |
| 4 | #include <linux/percpu-defs.h> | ||
| 4 | 5 | ||
| 5 | #include <xen/xen.h> | 6 | #include <xen/xen.h> |
| 6 | #include <xen/interface/xen.h> | 7 | #include <xen/interface/xen.h> |
| 7 | #include <xen/grant_table.h> | 8 | #include <xen/grant_table.h> |
| 8 | #include <xen/events.h> | 9 | #include <xen/events.h> |
| 9 | 10 | ||
| 11 | #include <asm/cpufeatures.h> | ||
| 12 | #include <asm/msr-index.h> | ||
| 10 | #include <asm/xen/hypercall.h> | 13 | #include <asm/xen/hypercall.h> |
| 11 | #include <asm/xen/page.h> | 14 | #include <asm/xen/page.h> |
| 12 | #include <asm/fixmap.h> | 15 | #include <asm/fixmap.h> |
| @@ -15,6 +18,8 @@ · 2 unmodified lines | |||
| 15 | #include "mmu.h" | 18 | #include "mmu.h" |
| 16 | #include "pmu.h" | 19 | #include "pmu.h" |
| 17 | 20 | ||
| 21 | static DEFINE_PER_CPU(u64, spec_ctrl); | ||
| 22 | |||
| 18 | void xen_arch_pre_suspend(void) | 23 | void xen_arch_pre_suspend(void) |
| 19 | { | 24 | { |
| 20 | xen_save_time_memory_area(); | 25 | xen_save_time_memory_area(); |
| @@ -35,6 +40,9 @@ void xen_arch_post_suspend(int cancelled) · 14 unmodified lines | |||
| 35 | 40 | ||
| 36 | static void xen_vcpu_notify_restore(void *data) | 41 | static void xen_vcpu_notify_restore(void *data) |
| 37 | { | 42 | { |
| 43 | if (xen_pv_domain() && boot_cpu_has(X86_FEATURE_SPEC_CTRL)) | ||
| 44 | wrmsrl(MSR_IA32_SPEC_CTRL, this_cpu_read(spec_ctrl)); | ||
| 45 | |||
| 38 | /* Boot processor notified via generic timekeeping_resume() */ | 46 | /* Boot processor notified via generic timekeeping_resume() */ |
| 39 | if (smp_processor_id() == 0) | 47 | if (smp_processor_id() == 0) |
| 40 | return; | 48 | return; |
| @@ -44,7 +52,15 @@ static void xen_vcpu_notify_restore(void *data) · 3 unmodified lines | |||
| 44 | 52 | ||
| 45 | static void xen_vcpu_notify_suspend(void *data) | 53 | static void xen_vcpu_notify_suspend(void *data) |
| 46 | { | 54 | { |
| 55 | u64 tmp; | ||
| 56 | |||
| 47 | tick_suspend_local(); | 57 | tick_suspend_local(); |
| 58 | |||
| 59 | if (xen_pv_domain() && boot_cpu_has(X86_FEATURE_SPEC_CTRL)) { | ||
| 60 | rdmsrl(MSR_IA32_SPEC_CTRL, tmp); | ||
| 61 | this_cpu_write(spec_ctrl, tmp); | ||
| 62 | wrmsrl(MSR_IA32_SPEC_CTRL, 0); | ||
| 63 | } | ||
| 48 | } | 64 | } |
| 49 | 65 | ||
| 50 | void xen_arch_resume(void) | 66 | void xen_arch_resume(void) |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | |||
| 16 | */ | 16 | */ |
| 17 | 17 | ||
| 18 | #include <linux/dma-contiguous.h> | 18 | #include <linux/dma-contiguous.h> |
| 19 | #include <linux/dma-direct.h> | ||
| 19 | #include <linux/gfp.h> | 20 | #include <linux/gfp.h> |
| 20 | #include <linux/highmem.h> | 21 | #include <linux/highmem.h> |
| 21 | #include <linux/mm.h> | 22 | #include <linux/mm.h> |
| @@ -123,7 +124,7 @@ static void *xtensa_dma_alloc(struct device *dev, size_t size, · 101 unmodified lines | |||
| 123 | unsigned long attrs) | 124 | unsigned long attrs) |
| 124 | { | 125 | { |
| 125 | unsigned long ret; | 126 | unsigned long ret; |
| 126 | unsigned long uncached = 0; | 127 | unsigned long uncached; |
| 127 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; | 128 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; |
| 128 | struct page *page = NULL; | 129 | struct page *page = NULL; |
| 129 | 130 | ||
| @@ -144,15 +145,27 @@ static void *xtensa_dma_alloc(struct device *dev, size_t size, · 14 unmodified lines | |||
| 144 | if (!page) | 145 | if (!page) |
| 145 | return NULL; | 146 | return NULL; |
| 146 | 147 | ||
| 147 | ret = (unsigned long)page_address(page); | 148 | *handle = phys_to_dma(dev, page_to_phys(page)); |
| 148 | 149 | ||
| 149 | /* We currently don't support coherent memory outside KSEG */ | 150 | #ifdef CONFIG_MMU |
| 151 | if (PageHighMem(page)) { | ||
| 152 | void *p; | ||
| 150 | 153 | ||
| 154 | p = dma_common_contiguous_remap(page, size, VM_MAP, | ||
| 155 | pgprot_noncached(PAGE_KERNEL), | ||
| 156 | __builtin_return_address(0)); | ||
| 157 | if (!p) { | ||
| 158 | if (!dma_release_from_contiguous(dev, page, count)) | ||
| 159 | __free_pages(page, get_order(size)); | ||
| 160 | } | ||
| 161 | return p; | ||
| 162 | } | ||
| 163 | #endif | ||
| 164 | ret = (unsigned long)page_address(page); | ||
| 151 | BUG_ON(ret < XCHAL_KSEG_CACHED_VADDR || | 165 | BUG_ON(ret < XCHAL_KSEG_CACHED_VADDR || |
| 152 | ret > XCHAL_KSEG_CACHED_VADDR + XCHAL_KSEG_SIZE - 1); | 166 | ret > XCHAL_KSEG_CACHED_VADDR + XCHAL_KSEG_SIZE - 1); |
| 153 | 167 | ||
| 154 | uncached = ret + XCHAL_KSEG_BYPASS_VADDR - XCHAL_KSEG_CACHED_VADDR; | 168 | uncached = ret + XCHAL_KSEG_BYPASS_VADDR - XCHAL_KSEG_CACHED_VADDR; |
| 155 | *handle = virt_to_bus((void *)ret); | ||
| 156 | __invalidate_dcache_range(ret, size); | 169 | __invalidate_dcache_range(ret, size); |
| 157 | 170 | ||
| 158 | return (void *)uncached; | 171 | return (void *)uncached; |
| @@ -161,13 +174,20 @@ static void *xtensa_dma_alloc(struct device *dev, size_t size, · 2 unmodified lines | |||
| 161 | static void xtensa_dma_free(struct device *dev, size_t size, void *vaddr, | 174 | static void xtensa_dma_free(struct device *dev, size_t size, void *vaddr, |
| 162 | dma_addr_t dma_handle, unsigned long attrs) | 175 | dma_addr_t dma_handle, unsigned long attrs) |
| 163 | { | 176 | { |
| 164 | unsigned long addr = (unsigned long)vaddr + | ||
| 165 | XCHAL_KSEG_CACHED_VADDR - XCHAL_KSEG_BYPASS_VADDR; | ||
| 166 | struct page *page = virt_to_page(addr); | ||
| 167 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; | 177 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; |
| 168 | 178 | unsigned long addr = (unsigned long)vaddr; | |
| 169 | BUG_ON(addr < XCHAL_KSEG_CACHED_VADDR || | 179 | struct page *page; |
| 170 | addr > XCHAL_KSEG_CACHED_VADDR + XCHAL_KSEG_SIZE - 1); | 180 | |
| 181 | if (addr >= XCHAL_KSEG_BYPASS_VADDR && | ||
| 182 | addr - XCHAL_KSEG_BYPASS_VADDR < XCHAL_KSEG_SIZE) { | ||
| 183 | addr += XCHAL_KSEG_CACHED_VADDR - XCHAL_KSEG_BYPASS_VADDR; | ||
| 184 | page = virt_to_page(addr); | ||
| 185 | } else { | ||
| 186 | #ifdef CONFIG_MMU | ||
| 187 | dma_common_free_remap(vaddr, size, VM_MAP); | ||
| 188 | #endif | ||
| 189 | page = pfn_to_page(PHYS_PFN(dma_to_phys(dev, dma_handle))); | ||
| 190 | } | ||
| 171 | 191 | ||
| 172 | if (!dma_release_from_contiguous(dev, page, count)) | 192 | if (!dma_release_from_contiguous(dev, page, count)) |
| 173 | __free_pages(page, get_order(size)); | 193 | __free_pages(page, get_order(size)); |
| @@ -79,19 +79,75 @@ void __init zones_init(void) · 78 unmodified lines | |||
| 79 | free_area_init_node(0, zones_size, ARCH_PFN_OFFSET, NULL); | 79 | free_area_init_node(0, zones_size, ARCH_PFN_OFFSET, NULL); |
| 80 | } | 80 | } |
| 81 | 81 | ||
| 82 | #ifdef CONFIG_HIGHMEM | ||
| 83 | static void __init free_area_high(unsigned long pfn, unsigned long end) | ||
| 84 | { | ||
| 85 | for (; pfn < end; pfn++) | ||
| 86 | free_highmem_page(pfn_to_page(pfn)); | ||
| 87 | } | ||
| 88 | |||
| 89 | static void __init free_highpages(void) | ||
| 90 | { | ||
| 91 | unsigned long max_low = max_low_pfn; | ||
| 92 | struct memblock_region *mem, *res; | ||
| 93 | |||
| 94 | reset_all_zones_managed_pages(); | ||
| 95 | /* set highmem page free */ | ||
| 96 | for_each_memblock(memory, mem) { | ||
| 97 | unsigned long start = memblock_region_memory_base_pfn(mem); | ||
| 98 | unsigned long end = memblock_region_memory_end_pfn(mem); | ||
| 99 | |||
| 100 | /* Ignore complete lowmem entries */ | ||
| 101 | if (end <= max_low) | ||
| 102 | continue; | ||
| 103 | |||
| 104 | if (memblock_is_nomap(mem)) | ||
| 105 | continue; | ||
| 106 | |||
| 107 | /* Truncate partial highmem entries */ | ||
| 108 | if (start < max_low) | ||
| 109 | start = max_low; | ||
| 110 | |||
| 111 | /* Find and exclude any reserved regions */ | ||
| 112 | for_each_memblock(reserved, res) { | ||
| 113 | unsigned long res_start, res_end; | ||
| 114 | |||
| 115 | res_start = memblock_region_reserved_base_pfn(res); | ||
| 116 | res_end = memblock_region_reserved_end_pfn(res); | ||
| 117 | |||
| 118 | if (res_end < start) | ||
| 119 | continue; | ||
| 120 | if (res_start < start) | ||
| 121 | res_start = start; | ||
| 122 | if (res_start > end) | ||
| 123 | res_start = end; | ||
| 124 | if (res_end > end) | ||
| 125 | res_end = end; | ||
| 126 | if (res_start != start) | ||
| 127 | free_area_high(start, res_start); | ||
| 128 | start = res_end; | ||
| 129 | if (start == end) | ||
| 130 | break; | ||
| 131 | } | ||
| 132 | |||
| 133 | /* And now free anything which remains */ | ||
| 134 | if (start < end) | ||
| 135 | free_area_high(start, end); | ||
| 136 | } | ||
| 137 | } | ||
| 138 | #else | ||
| 139 | static void __init free_highpages(void) | ||
| 140 | { | ||
| 141 | } | ||
| 142 | #endif | ||
| 143 | |||
| 82 | /* | 144 | /* |
| 83 | * Initialize memory pages. | 145 | * Initialize memory pages. |
| 84 | */ | 146 | */ |
| 85 | 147 | ||
| 86 | void __init mem_init(void) | 148 | void __init mem_init(void) |
| 87 | { | 149 | { |
| 88 | #ifdef CONFIG_HIGHMEM | 150 | free_highpages(); |
| 89 | unsigned long tmp; | ||
| 90 | |||
| 91 | reset_all_zones_managed_pages(); | ||
| 92 | for (tmp = max_low_pfn; tmp < max_pfn; tmp++) | ||
| 93 | free_highmem_page(pfn_to_page(tmp)); | ||
| 94 | #endif | ||
| 95 | 151 | ||
| 96 | max_mapnr = max_pfn - ARCH_PFN_OFFSET; | 152 | max_mapnr = max_pfn - ARCH_PFN_OFFSET; |
| 97 | high_memory = (void *)__va(max_low_pfn << PAGE_SHIFT); | 153 | high_memory = (void *)__va(max_low_pfn << PAGE_SHIFT); |
| @@ -812,7 +812,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, · 811 unmodified lines | |||
| 812 | struct gendisk *disk; | 812 | struct gendisk *disk; |
| 813 | struct request_queue *q; | 813 | struct request_queue *q; |
| 814 | struct blkcg_gq *blkg; | 814 | struct blkcg_gq *blkg; |
| 815 | struct module *owner; | ||
| 816 | unsigned int major, minor; | 815 | unsigned int major, minor; |
| 817 | int key_len, part, ret; | 816 | int key_len, part, ret; |
| 818 | char *body; | 817 | char *body; |
| @@ -904,9 +903,7 @@ fail_unlock: · 85 unmodified lines | |||
| 904 | spin_unlock_irq(q->queue_lock); | 903 | spin_unlock_irq(q->queue_lock); |
| 905 | rcu_read_unlock(); | 904 | rcu_read_unlock(); |
| 906 | fail: | 905 | fail: |
| 907 | owner = disk->fops->owner; | 906 | put_disk_and_module(disk); |
| 908 | put_disk(disk); | ||
| 909 | module_put(owner); | ||
| 910 | /* | 907 | /* |
| 911 | * If queue was bypassing, we should retry. Do so after a | 908 | * If queue was bypassing, we should retry. Do so after a |
| 912 | * short msleep(). It isn't strictly necessary but queue | 909 | * short msleep(). It isn't strictly necessary but queue |
| @@ -931,13 +928,9 @@ EXPORT_SYMBOL_GPL(blkg_conf_prep); · 18 unmodified lines | |||
| 931 | void blkg_conf_finish(struct blkg_conf_ctx *ctx) | 928 | void blkg_conf_finish(struct blkg_conf_ctx *ctx) |
| 932 | __releases(ctx->disk->queue->queue_lock) __releases(rcu) | 929 | __releases(ctx->disk->queue->queue_lock) __releases(rcu) |
| 933 | { | 930 | { |
| 934 | struct module *owner; | ||
| 935 | |||
| 936 | spin_unlock_irq(ctx->disk->queue->queue_lock); | 931 | spin_unlock_irq(ctx->disk->queue->queue_lock); |
| 937 | rcu_read_unlock(); | 932 | rcu_read_unlock(); |
| 938 | owner = ctx->disk->fops->owner; | 933 | put_disk_and_module(ctx->disk); |
| 939 | put_disk(ctx->disk); | ||
| 940 | module_put(owner); | ||
| 941 | } | 934 | } |
| 942 | EXPORT_SYMBOL_GPL(blkg_conf_finish); | 935 | EXPORT_SYMBOL_GPL(blkg_conf_finish); |
| 943 | 936 | ||
| @@ -2434,7 +2434,7 @@ blk_qc_t submit_bio(struct bio *bio) · 2433 unmodified lines | |||
| 2434 | unsigned int count; | 2434 | unsigned int count; |
| 2435 | 2435 | ||
| 2436 | if (unlikely(bio_op(bio) == REQ_OP_WRITE_SAME)) | 2436 | if (unlikely(bio_op(bio) == REQ_OP_WRITE_SAME)) |
| 2437 | count = queue_logical_block_size(bio->bi_disk->queue); | 2437 | count = queue_logical_block_size(bio->bi_disk->queue) >> 9; |
| 2438 | else | 2438 | else |
| 2439 | count = bio_sectors(bio); | 2439 | count = bio_sectors(bio); |
| 2440 | 2440 | ||
| @@ -712,7 +712,6 @@ static void __blk_mq_requeue_request(struct request *rq) · 711 unmodified lines | |||
| 712 | 712 | ||
| 713 | trace_block_rq_requeue(q, rq); | 713 | trace_block_rq_requeue(q, rq); |
| 714 | wbt_requeue(q->rq_wb, &rq->issue_stat); | 714 | wbt_requeue(q->rq_wb, &rq->issue_stat); |
| 715 | blk_mq_sched_requeue_request(rq); | ||
| 716 | 715 | ||
| 717 | if (blk_mq_rq_state(rq) != MQ_RQ_IDLE) { | 716 | if (blk_mq_rq_state(rq) != MQ_RQ_IDLE) { |
| 718 | blk_mq_rq_update_state(rq, MQ_RQ_IDLE); | 717 | blk_mq_rq_update_state(rq, MQ_RQ_IDLE); |
| @@ -725,6 +724,9 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list) · 6 unmodified lines | |||
| 725 | { | 724 | { |
| 726 | __blk_mq_requeue_request(rq); | 725 | __blk_mq_requeue_request(rq); |
| 727 | 726 | ||
| 727 | /* this request will be re-inserted to io scheduler queue */ | ||
| 728 | blk_mq_sched_requeue_request(rq); | ||
| 729 | |||
| 728 | BUG_ON(blk_queued_rq(rq)); | 730 | BUG_ON(blk_queued_rq(rq)); |
| 729 | blk_mq_add_to_requeue_list(rq, true, kick_requeue_list); | 731 | blk_mq_add_to_requeue_list(rq, true, kick_requeue_list); |
| 730 | } | 732 | } |
| @@ -3164,6 +3166,7 @@ static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) · 2433 unmodified lines | |||
| 3164 | cpu_relax(); | 3166 | cpu_relax(); |
| 3165 | } | 3167 | } |
| 3166 | 3168 | ||
| 3169 | __set_current_state(TASK_RUNNING); | ||
| 3167 | return false; | 3170 | return false; |
| 3168 | } | 3171 | } |
| 3169 | 3172 | ||
| @@ -547,7 +547,7 @@ static int exact_lock(dev_t devt, void *data) · 546 unmodified lines | |||
| 547 | { | 547 | { |
| 548 | struct gendisk *p = data; | 548 | struct gendisk *p = data; |
| 549 | 549 | ||
| 550 | if (!get_disk(p)) | 550 | if (!get_disk_and_module(p)) |
| 551 | return -1; | 551 | return -1; |
| 552 | return 0; | 552 | return 0; |
| 553 | } | 553 | } |
| @@ -717,6 +717,11 @@ void del_gendisk(struct gendisk *disk) · 163 unmodified lines | |||
| 717 | blk_integrity_del(disk); | 717 | blk_integrity_del(disk); |
| 718 | disk_del_events(disk); | 718 | disk_del_events(disk); |
| 719 | 719 | ||
| 720 | /* | ||
| 721 | * Block lookups of the disk until all bdevs are unhashed and the | ||
| 722 | * disk is marked as dead (GENHD_FL_UP cleared). | ||
| 723 | */ | ||
| 724 | down_write(&disk->lookup_sem); | ||
| 720 | /* invalidate stuff */ | 725 | /* invalidate stuff */ |
| 721 | disk_part_iter_init(&piter, disk, | 726 | disk_part_iter_init(&piter, disk, |
| 722 | DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); | 727 | DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); |
| @@ -731,6 +736,7 @@ void del_gendisk(struct gendisk *disk) · 8 unmodified lines | |||
| 731 | bdev_unhash_inode(disk_devt(disk)); | 736 | bdev_unhash_inode(disk_devt(disk)); |
| 732 | set_capacity(disk, 0); | 737 | set_capacity(disk, 0); |
| 733 | disk->flags &= ~GENHD_FL_UP; | 738 | disk->flags &= ~GENHD_FL_UP; |
| 739 | up_write(&disk->lookup_sem); | ||
| 734 | 740 | ||
| 735 | if (!(disk->flags & GENHD_FL_HIDDEN)) | 741 | if (!(disk->flags & GENHD_FL_HIDDEN)) |
| 736 | sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi"); | 742 | sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi"); |
| @@ -809,16 +815,28 @@ struct gendisk *get_gendisk(dev_t devt, int *partno) · 72 unmodified lines | |||
| 809 | 815 | ||
| 810 | spin_lock_bh(&ext_devt_lock); | 816 | spin_lock_bh(&ext_devt_lock); |
| 811 | part = idr_find(&ext_devt_idr, blk_mangle_minor(MINOR(devt))); | 817 | part = idr_find(&ext_devt_idr, blk_mangle_minor(MINOR(devt))); |
| 812 | if (part && get_disk(part_to_disk(part))) { | 818 | if (part && get_disk_and_module(part_to_disk(part))) { |
| 813 | *partno = part->partno; | 819 | *partno = part->partno; |
| 814 | disk = part_to_disk(part); | 820 | disk = part_to_disk(part); |
| 815 | } | 821 | } |
| 816 | spin_unlock_bh(&ext_devt_lock); | 822 | spin_unlock_bh(&ext_devt_lock); |
| 817 | } | 823 | } |
| 818 | 824 | ||
| 819 | if (disk && unlikely(disk->flags & GENHD_FL_HIDDEN)) { | 825 | if (!disk) |
| 820 | put_disk(disk); | 826 | return NULL; |
| 827 | |||
| 828 | /* | ||
| 829 | * Synchronize with del_gendisk() to not return disk that is being | ||
| 830 | * destroyed. | ||
| 831 | */ | ||
| 832 | down_read(&disk->lookup_sem); | ||
| 833 | if (unlikely((disk->flags & GENHD_FL_HIDDEN) || | ||
| 834 | !(disk->flags & GENHD_FL_UP))) { | ||
| 835 | up_read(&disk->lookup_sem); | ||
| 836 | put_disk_and_module(disk); | ||
| 821 | disk = NULL; | 837 | disk = NULL; |
| 838 | } else { | ||
| 839 | up_read(&disk->lookup_sem); | ||
| 822 | } | 840 | } |
| 823 | return disk; | 841 | return disk; |
| 824 | } | 842 | } |
| @@ -1418,6 +1436,7 @@ struct gendisk *__alloc_disk_node(int minors, int node_id) · 593 unmodified lines | |||
| 1418 | kfree(disk); | 1436 | kfree(disk); |
| 1419 | return NULL; | 1437 | return NULL; |
| 1420 | } | 1438 | } |
| 1439 | init_rwsem(&disk->lookup_sem); | ||
| 1421 | disk->node_id = node_id; | 1440 | disk->node_id = node_id; |
| 1422 | if (disk_expand_part_tbl(disk, 0)) { | 1441 | if (disk_expand_part_tbl(disk, 0)) { |
| 1423 | free_part_stats(&disk->part0); | 1442 | free_part_stats(&disk->part0); |
| @@ -1453,7 +1472,7 @@ struct gendisk *__alloc_disk_node(int minors, int node_id) · 29 unmodified lines | |||
| 1453 | } | 1472 | } |
| 1454 | EXPORT_SYMBOL(__alloc_disk_node); | 1473 | EXPORT_SYMBOL(__alloc_disk_node); |
| 1455 | 1474 | ||
| 1456 | struct kobject *get_disk(struct gendisk *disk) | 1475 | struct kobject *get_disk_and_module(struct gendisk *disk) |
| 1457 | { | 1476 | { |
| 1458 | struct module *owner; | 1477 | struct module *owner; |
| 1459 | struct kobject *kobj; | 1478 | struct kobject *kobj; |
| @@ -1471,17 +1490,30 @@ struct kobject *get_disk(struct gendisk *disk) · 11 unmodified lines | |||
| 1471 | return kobj; | 1490 | return kobj; |
| 1472 | 1491 | ||
| 1473 | } | 1492 | } |
| 1474 | 1493 | EXPORT_SYMBOL(get_disk_and_module); | |
| 1475 | EXPORT_SYMBOL(get_disk); | ||
| 1476 | 1494 | ||
| 1477 | void put_disk(struct gendisk *disk) | 1495 | void put_disk(struct gendisk *disk) |
| 1478 | { | 1496 | { |
| 1479 | if (disk) | 1497 | if (disk) |
| 1480 | kobject_put(&disk_to_dev(disk)->kobj); | 1498 | kobject_put(&disk_to_dev(disk)->kobj); |
| 1481 | } | 1499 | } |
| 1482 | |||
| 1483 | EXPORT_SYMBOL(put_disk); | 1500 | EXPORT_SYMBOL(put_disk); |
| 1484 | 1501 | ||
| 1502 | /* | ||
| 1503 | * This is a counterpart of get_disk_and_module() and thus also of | ||
| 1504 | * get_gendisk(). | ||
| 1505 | */ | ||
| 1506 | void put_disk_and_module(struct gendisk *disk) | ||
| 1507 | { | ||
| 1508 | if (disk) { | ||
| 1509 | struct module *owner = disk->fops->owner; | ||
| 1510 | |||
| 1511 | put_disk(disk); | ||
| 1512 | module_put(owner); | ||
| 1513 | } | ||
| 1514 | } | ||
| 1515 | EXPORT_SYMBOL(put_disk_and_module); | ||
| 1516 | |||
| 1485 | static void set_disk_ro_uevent(struct gendisk *gd, int ro) | 1517 | static void set_disk_ro_uevent(struct gendisk *gd, int ro) |
| 1486 | { | 1518 | { |
| 1487 | char event[] = "DISK_RO=1"; | 1519 | char event[] = "DISK_RO=1"; |
| @@ -225,7 +225,7 @@ static int blk_ioctl_discard(struct block_device *bdev, fmode_t mode, · 224 unmodified lines | |||
| 225 | 225 | ||
| 226 | if (start + len > i_size_read(bdev->bd_inode)) | 226 | if (start + len > i_size_read(bdev->bd_inode)) |
| 227 | return -EINVAL; | 227 | return -EINVAL; |
| 228 | truncate_inode_pages_range(mapping, start, start + len); | 228 | truncate_inode_pages_range(mapping, start, start + len - 1); |
| 229 | return blkdev_issue_discard(bdev, start >> 9, len >> 9, | 229 | return blkdev_issue_discard(bdev, start >> 9, len >> 9, |
| 230 | GFP_KERNEL, flags); | 230 | GFP_KERNEL, flags); |
| 231 | } | 231 | } |
| @@ -833,6 +833,7 @@ static struct elevator_type kyber_sched = { · 832 unmodified lines | |||
| 833 | .limit_depth = kyber_limit_depth, | 833 | .limit_depth = kyber_limit_depth, |
| 834 | .prepare_request = kyber_prepare_request, | 834 | .prepare_request = kyber_prepare_request, |
| 835 | .finish_request = kyber_finish_request, | 835 | .finish_request = kyber_finish_request, |
| 836 | .requeue_request = kyber_finish_request, | ||
| 836 | .completed_request = kyber_completed_request, | 837 | .completed_request = kyber_completed_request, |
| 837 | .dispatch_request = kyber_dispatch_request, | 838 | .dispatch_request = kyber_dispatch_request, |
| 838 | .has_work = kyber_has_work, | 839 | .has_work = kyber_has_work, |
| @@ -535,13 +535,22 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, · 534 unmodified lines | |||
| 535 | spin_unlock(&dd->lock); | 535 | spin_unlock(&dd->lock); |
| 536 | } | 536 | } |
| 537 | 537 | ||
| 538 | /* | ||
| 539 | * Nothing to do here. This is defined only to ensure that .finish_request | ||
| 540 | * method is called upon request completion. | ||
| 541 | */ | ||
| 542 | static void dd_prepare_request(struct request *rq, struct bio *bio) | ||
| 543 | { | ||
| 544 | } | ||
| 545 | |||
| 538 | /* | 546 | /* |
| 539 | * For zoned block devices, write unlock the target zone of | 547 | * For zoned block devices, write unlock the target zone of |
| 540 | * completed write requests. Do this while holding the zone lock | 548 | * completed write requests. Do this while holding the zone lock |
| 541 | * spinlock so that the zone is never unlocked while deadline_fifo_request() | 549 | * spinlock so that the zone is never unlocked while deadline_fifo_request() |
| 542 | * while deadline_next_request() are executing. | 550 | * or deadline_next_request() are executing. This function is called for |
| 551 | * all requests, whether or not these requests complete successfully. | ||
| 543 | */ | 552 | */ |
| 544 | static void dd_completed_request(struct request *rq) | 553 | static void dd_finish_request(struct request *rq) |
| 545 | { | 554 | { |
| 546 | struct request_queue *q = rq->q; | 555 | struct request_queue *q = rq->q; |
| 547 | 556 | ||
| @@ -756,7 +765,8 @@ static struct elevator_type mq_deadline = { · 208 unmodified lines | |||
| 756 | .ops.mq = { | 765 | .ops.mq = { |
| 757 | .insert_requests = dd_insert_requests, | 766 | .insert_requests = dd_insert_requests, |
| 758 | .dispatch_request = dd_dispatch_request, | 767 | .dispatch_request = dd_dispatch_request, |
| 759 | .completed_request = dd_completed_request, | 768 | .prepare_request = dd_prepare_request, |
| 769 | .finish_request = dd_finish_request, | ||
| 760 | .next_request = elv_rb_latter_request, | 770 | .next_request = elv_rb_latter_request, |
| 761 | .former_request = elv_rb_former_request, | 771 | .former_request = elv_rb_former_request, |
| 762 | .bio_merge = dd_bio_merge, | 772 | .bio_merge = dd_bio_merge, |
| @@ -51,6 +51,12 @@ const char *bdevname(struct block_device *bdev, char *buf) · 50 unmodified lines | |||
| 51 | 51 | ||
| 52 | EXPORT_SYMBOL(bdevname); | 52 | EXPORT_SYMBOL(bdevname); |
| 53 | 53 | ||
| 54 | const char *bio_devname(struct bio *bio, char *buf) | ||
| 55 | { | ||
| 56 | return disk_name(bio->bi_disk, bio->bi_partno, buf); | ||
| 57 | } | ||
| 58 | EXPORT_SYMBOL(bio_devname); | ||
| 59 | |||
| 54 | /* | 60 | /* |
| 55 | * There's very little reason to use this, you should really | 61 | * There's very little reason to use this, you should really |
| 56 | * have a struct block_device just about everywhere and use | 62 | * have a struct block_device just about everywhere and use |
| @@ -490,7 +490,7 @@ static int opal_discovery0_end(struct opal_dev *dev) · 489 unmodified lines | |||
| 490 | 490 | ||
| 491 | if (!found_com_id) { | 491 | if (!found_com_id) { |
| 492 | pr_debug("Could not find OPAL comid for device. Returning early\n"); | 492 | pr_debug("Could not find OPAL comid for device. Returning early\n"); |
| 493 | return -EOPNOTSUPP;; | 493 | return -EOPNOTSUPP; |
| 494 | } | 494 | } |
| 495 | 495 | ||
| 496 | dev->comid = comid; | 496 | dev->comid = comid; |
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | #include "blacklist.h" | 2 | #include "blacklist.h" |
| 3 | 3 | ||
| 4 | const char __initdata *const blacklist_hashes[] = { | 4 | const char __initconst *const blacklist_hashes[] = { |
| 5 | NULL | 5 | NULL |
| 6 | }; | 6 | }; |
| @@ -106,6 +106,7 @@ static int pkcs7_validate_trust_one(struct pkcs7_message *pkcs7, · 105 unmodified lines | |||
| 106 | pr_devel("sinfo %u: Direct signer is key %x\n", | 106 | pr_devel("sinfo %u: Direct signer is key %x\n", |
| 107 | sinfo->index, key_serial(key)); | 107 | sinfo->index, key_serial(key)); |
| 108 | x509 = NULL; | 108 | x509 = NULL; |
| 109 | sig = sinfo->sig; | ||
| 109 | goto matched; | 110 | goto matched; |
| 110 | } | 111 | } |
| 111 | if (PTR_ERR(key) != -ENOKEY) | 112 | if (PTR_ERR(key) != -ENOKEY) |
| @@ -270,7 +270,7 @@ static int pkcs7_verify_sig_chain(struct pkcs7_message *pkcs7, · 269 unmodified lines | |||
| 270 | sinfo->index); | 270 | sinfo->index); |
| 271 | return 0; | 271 | return 0; |
| 272 | } | 272 | } |
| 273 | ret = public_key_verify_signature(p->pub, p->sig); | 273 | ret = public_key_verify_signature(p->pub, x509->sig); |
| 274 | if (ret < 0) | 274 | if (ret < 0) |
| 275 | return ret; | 275 | return ret; |
| 276 | x509->signer = p; | 276 | x509->signer = p; |
| @@ -366,8 +366,7 @@ static int pkcs7_verify_one(struct pkcs7_message *pkcs7, · 89 unmodified lines | |||
| 366 | * | 366 | * |
| 367 | * (*) -EBADMSG if some part of the message was invalid, or: | 367 | * (*) -EBADMSG if some part of the message was invalid, or: |
| 368 | * | 368 | * |
| 369 | * (*) 0 if no signature chains were found to be blacklisted or to contain | 369 | * (*) 0 if a signature chain passed verification, or: |
| 370 | * unsupported crypto, or: | ||
| 371 | * | 370 | * |
| 372 | * (*) -EKEYREJECTED if a blacklisted key was encountered, or: | 371 | * (*) -EKEYREJECTED if a blacklisted key was encountered, or: |
| 373 | * | 372 | * |
| @@ -423,8 +422,11 @@ int pkcs7_verify(struct pkcs7_message *pkcs7, · 49 unmodified lines | |||
| 423 | 422 | ||
| 424 | for (sinfo = pkcs7->signed_infos; sinfo; sinfo = sinfo->next) { | 423 | for (sinfo = pkcs7->signed_infos; sinfo; sinfo = sinfo->next) { |
| 425 | ret = pkcs7_verify_one(pkcs7, sinfo); | 424 | ret = pkcs7_verify_one(pkcs7, sinfo); |
| 426 | if (sinfo->blacklisted && actual_ret == -ENOPKG) | 425 | if (sinfo->blacklisted) { |
| 427 | actual_ret = -EKEYREJECTED; | 426 | if (actual_ret == -ENOPKG) |
| 427 | actual_ret = -EKEYREJECTED; | ||
| 428 | continue; | ||
| 429 | } | ||
| 428 | if (ret < 0) { | 430 | if (ret < 0) { |
| 429 | if (ret == -ENOPKG) { | 431 | if (ret == -ENOPKG) { |
| 430 | sinfo->unsupported_crypto = true; | 432 | sinfo->unsupported_crypto = true; |
| @@ -79,9 +79,11 @@ int public_key_verify_signature(const struct public_key *pkey, · 78 unmodified lines | |||
| 79 | 79 | ||
| 80 | BUG_ON(!pkey); | 80 | BUG_ON(!pkey); |
| 81 | BUG_ON(!sig); | 81 | BUG_ON(!sig); |
| 82 | BUG_ON(!sig->digest); | ||
| 83 | BUG_ON(!sig->s); | 82 | BUG_ON(!sig->s); |
| 84 | 83 | ||
| 84 | if (!sig->digest) | ||
| 85 | return -ENOPKG; | ||
| 86 | |||
| 85 | alg_name = sig->pkey_algo; | 87 | alg_name = sig->pkey_algo; |
| 86 | if (strcmp(sig->pkey_algo, "rsa") == 0) { | 88 | if (strcmp(sig->pkey_algo, "rsa") == 0) { |
| 87 | /* The data wangled by the RSA algorithm is typically padded | 89 | /* The data wangled by the RSA algorithm is typically padded |
| @@ -67,8 +67,9 @@ __setup("ca_keys=", ca_keys_setup); · 66 unmodified lines | |||
| 67 | * | 67 | * |
| 68 | * Returns 0 if the new certificate was accepted, -ENOKEY if we couldn't find a | 68 | * Returns 0 if the new certificate was accepted, -ENOKEY if we couldn't find a |
| 69 | * matching parent certificate in the trusted list, -EKEYREJECTED if the | 69 | * matching parent certificate in the trusted list, -EKEYREJECTED if the |
| 70 | * signature check fails or the key is blacklisted and some other error if | 70 | * signature check fails or the key is blacklisted, -ENOPKG if the signature |
| 71 | * there is a matching certificate but the signature check cannot be performed. | 71 | * uses unsupported crypto, or some other error if there is a matching |
| 72 | * certificate but the signature check cannot be performed. | ||
| 72 | */ | 73 | */ |
| 73 | int restrict_link_by_signature(struct key *dest_keyring, | 74 | int restrict_link_by_signature(struct key *dest_keyring, |
| 74 | const struct key_type *type, | 75 | const struct key_type *type, |
| @@ -88,6 +89,8 @@ int restrict_link_by_signature(struct key *dest_keyring, · 13 unmodified lines | |||
| 88 | return -EOPNOTSUPP; | 89 | return -EOPNOTSUPP; |
| 89 | 90 | ||
| 90 | sig = payload->data[asym_auth]; | 91 | sig = payload->data[asym_auth]; |
| 92 | if (!sig) | ||
| 93 | return -ENOPKG; | ||
| 91 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) | 94 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) |
| 92 | return -ENOKEY; | 95 | return -ENOKEY; |
| 93 | 96 | ||
| @@ -139,6 +142,8 @@ static int key_or_keyring_common(struct key *dest_keyring, · 45 unmodified lines | |||
| 139 | return -EOPNOTSUPP; | 142 | return -EOPNOTSUPP; |
| 140 | 143 | ||
| 141 | sig = payload->data[asym_auth]; | 144 | sig = payload->data[asym_auth]; |
| 145 | if (!sig) | ||
| 146 | return -ENOPKG; | ||
| 142 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) | 147 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) |
| 143 | return -ENOKEY; | 148 | return -ENOKEY; |
| 144 | 149 | ||
| @@ -222,9 +227,9 @@ static int key_or_keyring_common(struct key *dest_keyring, · 77 unmodified lines | |||
| 222 | * | 227 | * |
| 223 | * Returns 0 if the new certificate was accepted, -ENOKEY if we | 228 | * Returns 0 if the new certificate was accepted, -ENOKEY if we |
| 224 | * couldn't find a matching parent certificate in the trusted list, | 229 | * couldn't find a matching parent certificate in the trusted list, |
| 225 | * -EKEYREJECTED if the signature check fails, and some other error if | 230 | * -EKEYREJECTED if the signature check fails, -ENOPKG if the signature uses |
| 226 | * there is a matching certificate but the signature check cannot be | 231 | * unsupported crypto, or some other error if there is a matching certificate |
| 227 | * performed. | 232 | * but the signature check cannot be performed. |
| 228 | */ | 233 | */ |
| 229 | int restrict_link_by_key_or_keyring(struct key *dest_keyring, | 234 | int restrict_link_by_key_or_keyring(struct key *dest_keyring, |
| 230 | const struct key_type *type, | 235 | const struct key_type *type, |
| @@ -249,9 +254,9 @@ int restrict_link_by_key_or_keyring(struct key *dest_keyring, · 18 unmodified lines | |||
| 249 | * | 254 | * |
| 250 | * Returns 0 if the new certificate was accepted, -ENOKEY if we | 255 | * Returns 0 if the new certificate was accepted, -ENOKEY if we |
| 251 | * couldn't find a matching parent certificate in the trusted list, | 256 | * couldn't find a matching parent certificate in the trusted list, |
| 252 | * -EKEYREJECTED if the signature check fails, and some other error if | 257 | * -EKEYREJECTED if the signature check fails, -ENOPKG if the signature uses |
| 253 | * there is a matching certificate but the signature check cannot be | 258 | * unsupported crypto, or some other error if there is a matching certificate |
| 254 | * performed. | 259 | * but the signature check cannot be performed. |
| 255 | */ | 260 | */ |
| 256 | int restrict_link_by_key_or_keyring_chain(struct key *dest_keyring, | 261 | int restrict_link_by_key_or_keyring_chain(struct key *dest_keyring, |
| 257 | const struct key_type *type, | 262 | const struct key_type *type, |
| @@ -20,6 +20,20 @@ · 19 unmodified lines | |||
| 20 | #include <crypto/sha3.h> | 20 | #include <crypto/sha3.h> |
| 21 | #include <asm/unaligned.h> | 21 | #include <asm/unaligned.h> |
| 22 | 22 | ||
| 23 | /* | ||
| 24 | * On some 32-bit architectures (mn10300 and h8300), GCC ends up using | ||
| 25 | * over 1 KB of stack if we inline the round calculation into the loop | ||
| 26 | * in keccakf(). On the other hand, on 64-bit architectures with plenty | ||
| 27 | * of [64-bit wide] general purpose registers, not inlining it severely | ||
| 28 | * hurts performance. So let's use 64-bitness as a heuristic to decide | ||
| 29 | * whether to inline or not. | ||
| 30 | */ | ||
| 31 | #ifdef CONFIG_64BIT | ||
| 32 | #define SHA3_INLINE inline | ||
| 33 | #else | ||
| 34 | #define SHA3_INLINE noinline | ||
| 35 | #endif | ||
| 36 | |||
| 23 | #define KECCAK_ROUNDS 24 | 37 | #define KECCAK_ROUNDS 24 |
| 24 | 38 | ||
| 25 | static const u64 keccakf_rndc[24] = { | 39 | static const u64 keccakf_rndc[24] = { |
| @@ -35,111 +49,115 @@ static const u64 keccakf_rndc[24] = { · 9 unmodified lines | |||
| 35 | 49 | ||
| 36 | /* update the state with given number of rounds */ | 50 | /* update the state with given number of rounds */ |
| 37 | 51 | ||
| 38 | static void __attribute__((__optimize__("O3"))) keccakf(u64 st[25]) | 52 | static SHA3_INLINE void keccakf_round(u64 st[25]) |
| 39 | { | 53 | { |
| 40 | u64 t[5], tt, bc[5]; | 54 | u64 t[5], tt, bc[5]; |
| 41 | int round; | ||
| 42 | 55 | ||
| 43 | for (round = 0; round < KECCAK_ROUNDS; round++) { | 56 | /* Theta */ |
| 57 | bc[0] = st[0] ^ st[5] ^ st[10] ^ st[15] ^ st[20]; | ||
| 58 | bc[1] = st[1] ^ st[6] ^ st[11] ^ st[16] ^ st[21]; | ||
| 59 | bc[2] = st[2] ^ st[7] ^ st[12] ^ st[17] ^ st[22]; | ||
| 60 | bc[3] = st[3] ^ st[8] ^ st[13] ^ st[18] ^ st[23]; | ||
| 61 | bc[4] = st[4] ^ st[9] ^ st[14] ^ st[19] ^ st[24]; | ||
| 62 | |||
| 63 | t[0] = bc[4] ^ rol64(bc[1], 1); | ||
| 64 | t[1] = bc[0] ^ rol64(bc[2], 1); | ||
| 65 | t[2] = bc[1] ^ rol64(bc[3], 1); | ||
| 66 | t[3] = bc[2] ^ rol64(bc[4], 1); | ||
| 67 | t[4] = bc[3] ^ rol64(bc[0], 1); | ||
| 68 | |||
| 69 | st[0] ^= t[0]; | ||
| 70 | |||
| 71 | /* Rho Pi */ | ||
| 72 | tt = st[1]; | ||
| 73 | st[ 1] = rol64(st[ 6] ^ t[1], 44); | ||
| 74 | st[ 6] = rol64(st[ 9] ^ t[4], 20); | ||
| 75 | st[ 9] = rol64(st[22] ^ t[2], 61); | ||
| 76 | st[22] = rol64(st[14] ^ t[4], 39); | ||
| 77 | st[14] = rol64(st[20] ^ t[0], 18); | ||
| 78 | st[20] = rol64(st[ 2] ^ t[2], 62); | ||
| 79 | st[ 2] = rol64(st[12] ^ t[2], 43); | ||
| 80 | st[12] = rol64(st[13] ^ t[3], 25); | ||
| 81 | st[13] = rol64(st[19] ^ t[4], 8); | ||
| 82 | st[19] = rol64(st[23] ^ t[3], 56); | ||
| 83 | st[23] = rol64(st[15] ^ t[0], 41); | ||
| 84 | st[15] = rol64(st[ 4] ^ t[4], 27); | ||
| 85 | st[ 4] = rol64(st[24] ^ t[4], 14); | ||
| 86 | st[24] = rol64(st[21] ^ t[1], 2); | ||
| 87 | st[21] = rol64(st[ 8] ^ t[3], 55); | ||
| 88 | st[ 8] = rol64(st[16] ^ t[1], 45); | ||
| 89 | st[16] = rol64(st[ 5] ^ t[0], 36); | ||
| 90 | st[ 5] = rol64(st[ 3] ^ t[3], 28); | ||
| 91 | st[ 3] = rol64(st[18] ^ t[3], 21); | ||
| 92 | st[18] = rol64(st[17] ^ t[2], 15); | ||
| 93 | st[17] = rol64(st[11] ^ t[1], 10); | ||
| 94 | st[11] = rol64(st[ 7] ^ t[2], 6); | ||
| 95 | st[ 7] = rol64(st[10] ^ t[0], 3); | ||
| 96 | st[10] = rol64( tt ^ t[1], 1); | ||
| 97 | |||
| 98 | /* Chi */ | ||
| 99 | bc[ 0] = ~st[ 1] & st[ 2]; | ||
| 100 | bc[ 1] = ~st[ 2] & st[ 3]; | ||
| 101 | bc[ 2] = ~st[ 3] & st[ 4]; | ||
| 102 | bc[ 3] = ~st[ 4] & st[ 0]; | ||
| 103 | bc[ 4] = ~st[ 0] & st[ 1]; | ||
| 104 | st[ 0] ^= bc[ 0]; | ||
| 105 | st[ 1] ^= bc[ 1]; | ||
| 106 | st[ 2] ^= bc[ 2]; | ||
| 107 | st[ 3] ^= bc[ 3]; | ||
| 108 | st[ 4] ^= bc[ 4]; | ||
| 109 | |||
| 110 | bc[ 0] = ~st[ 6] & st[ 7]; | ||
| 111 | bc[ 1] = ~st[ 7] & st[ 8]; | ||
| 112 | bc[ 2] = ~st[ 8] & st[ 9]; | ||
| 113 | bc[ 3] = ~st[ 9] & st[ 5]; | ||
| 114 | bc[ 4] = ~st[ 5] & st[ 6]; | ||
| 115 | st[ 5] ^= bc[ 0]; | ||
| 116 | st[ 6] ^= bc[ 1]; | ||
| 117 | st[ 7] ^= bc[ 2]; | ||
| 118 | st[ 8] ^= bc[ 3]; | ||
| 119 | st[ 9] ^= bc[ 4]; | ||
| 120 | |||
| 121 | bc[ 0] = ~st[11] & st[12]; | ||
| 122 | bc[ 1] = ~st[12] & st[13]; | ||
| 123 | bc[ 2] = ~st[13] & st[14]; | ||
| 124 | bc[ 3] = ~st[14] & st[10]; | ||
| 125 | bc[ 4] = ~st[10] & st[11]; | ||
| 126 | st[10] ^= bc[ 0]; | ||
| 127 | st[11] ^= bc[ 1]; | ||
| 128 | st[12] ^= bc[ 2]; | ||
| 129 | st[13] ^= bc[ 3]; | ||
| 130 | st[14] ^= bc[ 4]; | ||
| 131 | |||
| 132 | bc[ 0] = ~st[16] & st[17]; | ||
| 133 | bc[ 1] = ~st[17] & st[18]; | ||
| 134 | bc[ 2] = ~st[18] & st[19]; | ||
| 135 | bc[ 3] = ~st[19] & st[15]; | ||
| 136 | bc[ 4] = ~st[15] & st[16]; | ||
| 137 | st[15] ^= bc[ 0]; | ||
| 138 | st[16] ^= bc[ 1]; | ||
| 139 | st[17] ^= bc[ 2]; | ||
| 140 | st[18] ^= bc[ 3]; | ||
| 141 | st[19] ^= bc[ 4]; | ||
| 142 | |||
| 143 | bc[ 0] = ~st[21] & st[22]; | ||
| 144 | bc[ 1] = ~st[22] & st[23]; | ||
| 145 | bc[ 2] = ~st[23] & st[24]; | ||
| 146 | bc[ 3] = ~st[24] & st[20]; | ||
| 147 | bc[ 4] = ~st[20] & st[21]; | ||
| 148 | st[20] ^= bc[ 0]; | ||
| 149 | st[21] ^= bc[ 1]; | ||
| 150 | st[22] ^= bc[ 2]; | ||
| 151 | st[23] ^= bc[ 3]; | ||
| 152 | st[24] ^= bc[ 4]; | ||
| 153 | } | ||
| 44 | 154 | ||
| 45 | /* Theta */ | 155 | static void __optimize("O3") keccakf(u64 st[25]) |
| 46 | bc[0] = st[0] ^ st[5] ^ st[10] ^ st[15] ^ st[20]; | 156 | { |
| 47 | bc[1] = st[1] ^ st[6] ^ st[11] ^ st[16] ^ st[21]; | 157 | int round; |
| 48 | bc[2] = st[2] ^ st[7] ^ st[12] ^ st[17] ^ st[22]; | ||
| 49 | bc[3] = st[3] ^ st[8] ^ st[13] ^ st[18] ^ st[23]; | ||
| 50 | bc[4] = st[4] ^ st[9] ^ st[14] ^ st[19] ^ st[24]; | ||
| 51 | |||
| 52 | t[0] = bc[4] ^ rol64(bc[1], 1); | ||
| 53 | t[1] = bc[0] ^ rol64(bc[2], 1); | ||
| 54 | t[2] = bc[1] ^ rol64(bc[3], 1); | ||
| 55 | t[3] = bc[2] ^ rol64(bc[4], 1); | ||
| 56 | t[4] = bc[3] ^ rol64(bc[0], 1); | ||
| 57 | |||
| 58 | st[0] ^= t[0]; | ||
| 59 | |||
| 60 | /* Rho Pi */ | ||
| 61 | tt = st[1]; | ||
| 62 | st[ 1] = rol64(st[ 6] ^ t[1], 44); | ||
| 63 | st[ 6] = rol64(st[ 9] ^ t[4], 20); | ||
| 64 | st[ 9] = rol64(st[22] ^ t[2], 61); | ||
| 65 | st[22] = rol64(st[14] ^ t[4], 39); | ||
| 66 | st[14] = rol64(st[20] ^ t[0], 18); | ||
| 67 | st[20] = rol64(st[ 2] ^ t[2], 62); | ||
| 68 | st[ 2] = rol64(st[12] ^ t[2], 43); | ||
| 69 | st[12] = rol64(st[13] ^ t[3], 25); | ||
| 70 | st[13] = rol64(st[19] ^ t[4], 8); | ||
| 71 | st[19] = rol64(st[23] ^ t[3], 56); | ||
| 72 | st[23] = rol64(st[15] ^ t[0], 41); | ||
| 73 | st[15] = rol64(st[ 4] ^ t[4], 27); | ||
| 74 | st[ 4] = rol64(st[24] ^ t[4], 14); | ||
| 75 | st[24] = rol64(st[21] ^ t[1], 2); | ||
| 76 | st[21] = rol64(st[ 8] ^ t[3], 55); | ||
| 77 | st[ 8] = rol64(st[16] ^ t[1], 45); | ||
| 78 | st[16] = rol64(st[ 5] ^ t[0], 36); | ||
| 79 | st[ 5] = rol64(st[ 3] ^ t[3], 28); | ||
| 80 | st[ 3] = rol64(st[18] ^ t[3], 21); | ||
| 81 | st[18] = rol64(st[17] ^ t[2], 15); | ||
| 82 | st[17] = rol64(st[11] ^ t[1], 10); | ||
| 83 | st[11] = rol64(st[ 7] ^ t[2], 6); | ||
| 84 | st[ 7] = rol64(st[10] ^ t[0], 3); | ||
| 85 | st[10] = rol64( tt ^ t[1], 1); | ||
| 86 | |||
| 87 | /* Chi */ | ||
| 88 | bc[ 0] = ~st[ 1] & st[ 2]; | ||
| 89 | bc[ 1] = ~st[ 2] & st[ 3]; | ||
| 90 | bc[ 2] = ~st[ 3] & st[ 4]; | ||
| 91 | bc[ 3] = ~st[ 4] & st[ 0]; | ||
| 92 | bc[ 4] = ~st[ 0] & st[ 1]; | ||
| 93 | st[ 0] ^= bc[ 0]; | ||
| 94 | st[ 1] ^= bc[ 1]; | ||
| 95 | st[ 2] ^= bc[ 2]; | ||
| 96 | st[ 3] ^= bc[ 3]; | ||
| 97 | st[ 4] ^= bc[ 4]; | ||
| 98 | |||
| 99 | bc[ 0] = ~st[ 6] & st[ 7]; | ||
| 100 | bc[ 1] = ~st[ 7] & st[ 8]; | ||
| 101 | bc[ 2] = ~st[ 8] & st[ 9]; | ||
| 102 | bc[ 3] = ~st[ 9] & st[ 5]; | ||
| 103 | bc[ 4] = ~st[ 5] & st[ 6]; | ||
| 104 | st[ 5] ^= bc[ 0]; | ||
| 105 | st[ 6] ^= bc[ 1]; | ||
| 106 | st[ 7] ^= bc[ 2]; | ||
| 107 | st[ 8] ^= bc[ 3]; | ||
| 108 | st[ 9] ^= bc[ 4]; | ||
| 109 | |||
| 110 | bc[ 0] = ~st[11] & st[12]; | ||
| 111 | bc[ 1] = ~st[12] & st[13]; | ||
| 112 | bc[ 2] = ~st[13] & st[14]; | ||
| 113 | bc[ 3] = ~st[14] & st[10]; | ||
| 114 | bc[ 4] = ~st[10] & st[11]; | ||
| 115 | st[10] ^= bc[ 0]; | ||
| 116 | st[11] ^= bc[ 1]; | ||
| 117 | st[12] ^= bc[ 2]; | ||
| 118 | st[13] ^= bc[ 3]; | ||
| 119 | st[14] ^= bc[ 4]; | ||
| 120 | |||
| 121 | bc[ 0] = ~st[16] & st[17]; | ||
| 122 | bc[ 1] = ~st[17] & st[18]; | ||
| 123 | bc[ 2] = ~st[18] & st[19]; | ||
| 124 | bc[ 3] = ~st[19] & st[15]; | ||
| 125 | bc[ 4] = ~st[15] & st[16]; | ||
| 126 | st[15] ^= bc[ 0]; | ||
| 127 | st[16] ^= bc[ 1]; | ||
| 128 | st[17] ^= bc[ 2]; | ||
| 129 | st[18] ^= bc[ 3]; | ||
| 130 | st[19] ^= bc[ 4]; | ||
| 131 | |||
| 132 | bc[ 0] = ~st[21] & st[22]; | ||
| 133 | bc[ 1] = ~st[22] & st[23]; | ||
| 134 | bc[ 2] = ~st[23] & st[24]; | ||
| 135 | bc[ 3] = ~st[24] & st[20]; | ||
| 136 | bc[ 4] = ~st[20] & st[21]; | ||
| 137 | st[20] ^= bc[ 0]; | ||
| 138 | st[21] ^= bc[ 1]; | ||
| 139 | st[22] ^= bc[ 2]; | ||
| 140 | st[23] ^= bc[ 3]; | ||
| 141 | st[24] ^= bc[ 4]; | ||
| 142 | 158 | ||
| 159 | for (round = 0; round < KECCAK_ROUNDS; round++) { | ||
| 160 | keccakf_round(st); | ||
| 143 | /* Iota */ | 161 | /* Iota */ |
| 144 | st[0] ^= keccakf_rndc[round]; | 162 | st[0] ^= keccakf_rndc[round]; |
| 145 | } | 163 | } |
| @@ -74,10 +74,10 @@ void __init acpi_watchdog_init(void) · 73 unmodified lines | |||
| 74 | res.start = gas->address; | 74 | res.start = gas->address; |
| 75 | if (gas->space_id == ACPI_ADR_SPACE_SYSTEM_MEMORY) { | 75 | if (gas->space_id == ACPI_ADR_SPACE_SYSTEM_MEMORY) { |
| 76 | res.flags = IORESOURCE_MEM; | 76 | res.flags = IORESOURCE_MEM; |
| 77 | res.end = res.start + ALIGN(gas->access_width, 4); | 77 | res.end = res.start + ALIGN(gas->access_width, 4) - 1; |
| 78 | } else if (gas->space_id == ACPI_ADR_SPACE_SYSTEM_IO) { | 78 | } else if (gas->space_id == ACPI_ADR_SPACE_SYSTEM_IO) { |
| 79 | res.flags = IORESOURCE_IO; | 79 | res.flags = IORESOURCE_IO; |
| 80 | res.end = res.start + gas->access_width; | 80 | res.end = res.start + gas->access_width - 1; |
| 81 | } else { | 81 | } else { |
| 82 | pr_warn("Unsupported address space: %u\n", | 82 | pr_warn("Unsupported address space: %u\n", |
| 83 | gas->space_id); | 83 | gas->space_id); |
| @@ -70,7 +70,6 @@ static async_cookie_t async_cookie; · 69 unmodified lines | |||
| 70 | static bool battery_driver_registered; | 70 | static bool battery_driver_registered; |
| 71 | static int battery_bix_broken_package; | 71 | static int battery_bix_broken_package; |
| 72 | static int battery_notification_delay_ms; | 72 | static int battery_notification_delay_ms; |
| 73 | static int battery_full_discharging; | ||
| 74 | static unsigned int cache_time = 1000; | 73 | static unsigned int cache_time = 1000; |
| 75 | module_param(cache_time, uint, 0644); | 74 | module_param(cache_time, uint, 0644); |
| 76 | MODULE_PARM_DESC(cache_time, "cache time in milliseconds"); | 75 | MODULE_PARM_DESC(cache_time, "cache time in milliseconds"); |
| @@ -215,12 +214,9 @@ static int acpi_battery_get_property(struct power_supply *psy, · 138 unmodified lines | |||
| 215 | return -ENODEV; | 214 | return -ENODEV; |
| 216 | switch (psp) { | 215 | switch (psp) { |
| 217 | case POWER_SUPPLY_PROP_STATUS: | 216 | case POWER_SUPPLY_PROP_STATUS: |
| 218 | if (battery->state & ACPI_BATTERY_STATE_DISCHARGING) { | 217 | if (battery->state & ACPI_BATTERY_STATE_DISCHARGING) |
| 219 | if (battery_full_discharging && battery->rate_now == 0) | 218 | val->intval = POWER_SUPPLY_STATUS_DISCHARGING; |
| 220 | val->intval = POWER_SUPPLY_STATUS_FULL; | 219 | else if (battery->state & ACPI_BATTERY_STATE_CHARGING) |
| 221 | else | ||
| 222 | val->intval = POWER_SUPPLY_STATUS_DISCHARGING; | ||
| 223 | } else if (battery->state & ACPI_BATTERY_STATE_CHARGING) | ||
| 224 | val->intval = POWER_SUPPLY_STATUS_CHARGING; | 220 | val->intval = POWER_SUPPLY_STATUS_CHARGING; |
| 225 | else if (acpi_battery_is_charged(battery)) | 221 | else if (acpi_battery_is_charged(battery)) |
| 226 | val->intval = POWER_SUPPLY_STATUS_FULL; | 222 | val->intval = POWER_SUPPLY_STATUS_FULL; |
| @@ -1170,12 +1166,6 @@ battery_notification_delay_quirk(const struct dmi_system_id *d) · 943 unmodified lines | |||
| 1170 | return 0; | 1166 | return 0; |
| 1171 | } | 1167 | } |
| 1172 | 1168 | ||
| 1173 | static int __init battery_full_discharging_quirk(const struct dmi_system_id *d) | ||
| 1174 | { | ||
| 1175 | battery_full_discharging = 1; | ||
| 1176 | return 0; | ||
| 1177 | } | ||
| 1178 | |||
| 1179 | static const struct dmi_system_id bat_dmi_table[] __initconst = { | 1169 | static const struct dmi_system_id bat_dmi_table[] __initconst = { |
| 1180 | { | 1170 | { |
| 1181 | .callback = battery_bix_broken_package_quirk, | 1171 | .callback = battery_bix_broken_package_quirk, |
| @@ -1193,38 +1183,6 @@ static const struct dmi_system_id bat_dmi_table[] __initconst = { · 11 unmodified lines | |||
| 1193 | DMI_MATCH(DMI_PRODUCT_NAME, "Aspire V5-573G"), | 1183 | DMI_MATCH(DMI_PRODUCT_NAME, "Aspire V5-573G"), |
| 1194 | }, | 1184 | }, |
| 1195 | }, | 1185 | }, |
| 1196 | { | ||
| 1197 | .callback = battery_full_discharging_quirk, | ||
| 1198 | .ident = "ASUS GL502VSK", | ||
| 1199 | .matches = { | ||
| 1200 | DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), | ||
| 1201 | DMI_MATCH(DMI_PRODUCT_NAME, "GL502VSK"), | ||
| 1202 | }, | ||
| 1203 | }, | ||
| 1204 | { | ||
| 1205 | .callback = battery_full_discharging_quirk, | ||
| 1206 | .ident = "ASUS UX305LA", | ||
| 1207 | .matches = { | ||
| 1208 | DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), | ||
| 1209 | DMI_MATCH(DMI_PRODUCT_NAME, "UX305LA"), | ||
| 1210 | }, | ||
| 1211 | }, | ||
| 1212 | { | ||
| 1213 | .callback = battery_full_discharging_quirk, | ||
| 1214 | .ident = "ASUS UX360UA", | ||
| 1215 | .matches = { | ||
| 1216 | DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), | ||
| 1217 | DMI_MATCH(DMI_PRODUCT_NAME, "UX360UA"), | ||
| 1218 | }, | ||
| 1219 | }, | ||
| 1220 | { | ||
| 1221 | .callback = battery_full_discharging_quirk, | ||
| 1222 | .ident = "ASUS UX410UAK", | ||
| 1223 | .matches = { | ||
| 1224 | DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), | ||
| 1225 | DMI_MATCH(DMI_PRODUCT_NAME, "UX410UAK"), | ||
| 1226 | }, | ||
| 1227 | }, | ||
| 1228 | {}, | 1186 | {}, |
| 1229 | }; | 1187 | }; |
| 1230 | 1188 | ||
| @@ -660,13 +660,15 @@ struct acpi_device *acpi_companion_match(const struct device *dev) · 659 unmodified lines | |||
| 660 | * acpi_of_match_device - Match device object using the "compatible" property. | 660 | * acpi_of_match_device - Match device object using the "compatible" property. |
| 661 | * @adev: ACPI device object to match. | 661 | * @adev: ACPI device object to match. |
| 662 | * @of_match_table: List of device IDs to match against. | 662 | * @of_match_table: List of device IDs to match against. |
| 663 | * @of_id: OF ID if matched | ||
| 663 | * | 664 | * |
| 664 | * If @dev has an ACPI companion which has ACPI_DT_NAMESPACE_HID in its list of | 665 | * If @dev has an ACPI companion which has ACPI_DT_NAMESPACE_HID in its list of |
| 665 | * identifiers and a _DSD object with the "compatible" property, use that | 666 | * identifiers and a _DSD object with the "compatible" property, use that |
| 666 | * property to match against the given list of identifiers. | 667 | * property to match against the given list of identifiers. |
| 667 | */ | 668 | */ |
| 668 | static bool acpi_of_match_device(struct acpi_device *adev, | 669 | static bool acpi_of_match_device(struct acpi_device *adev, |
| 669 | const struct of_device_id *of_match_table) | 670 | const struct of_device_id *of_match_table, |
| 671 | const struct of_device_id **of_id) | ||
| 670 | { | 672 | { |
| 671 | const union acpi_object *of_compatible, *obj; | 673 | const union acpi_object *of_compatible, *obj; |
| 672 | int i, nval; | 674 | int i, nval; |
| @@ -690,8 +692,11 @@ static bool acpi_of_match_device(struct acpi_device *adev, · 17 unmodified lines | |||
| 690 | const struct of_device_id *id; | 692 | const struct of_device_id *id; |
| 691 | 693 | ||
| 692 | for (id = of_match_table; id->compatible[0]; id++) | 694 | for (id = of_match_table; id->compatible[0]; id++) |
| 693 | if (!strcasecmp(obj->string.pointer, id->compatible)) | 695 | if (!strcasecmp(obj->string.pointer, id->compatible)) { |
| 696 | if (of_id) | ||
| 697 | *of_id = id; | ||
| 694 | return true; | 698 | return true; |
| 699 | } | ||
| 695 | } | 700 | } |
| 696 | 701 | ||
| 697 | return false; | 702 | return false; |
| @@ -762,10 +767,11 @@ static bool __acpi_match_device_cls(const struct acpi_device_id *id, · 64 unmodified lines | |||
| 762 | return true; | 767 | return true; |
| 763 | } | 768 | } |
| 764 | 769 | ||
| 765 | static const struct acpi_device_id *__acpi_match_device( | 770 | static bool __acpi_match_device(struct acpi_device *device, |
| 766 | struct acpi_device *device, | 771 | const struct acpi_device_id *acpi_ids, |
| 767 | const struct acpi_device_id *ids, | 772 | const struct of_device_id *of_ids, |
| 768 | const struct of_device_id *of_ids) | 773 | const struct acpi_device_id **acpi_id, |
| 774 | const struct of_device_id **of_id) | ||
| 769 | { | 775 | { |
| 770 | const struct acpi_device_id *id; | 776 | const struct acpi_device_id *id; |
| 771 | struct acpi_hardware_id *hwid; | 777 | struct acpi_hardware_id *hwid; |
| @@ -775,30 +781,32 @@ static const struct acpi_device_id *__acpi_match_device( · 3 unmodified lines | |||
| 775 | * driver for it. | 781 | * driver for it. |
| 776 | */ | 782 | */ |
| 777 | if (!device || !device->status.present) | 783 | if (!device || !device->status.present) |
| 778 | return NULL; | 784 | return false; |
| 779 | 785 | ||
| 780 | list_for_each_entry(hwid, &device->pnp.ids, list) { | 786 | list_for_each_entry(hwid, &device->pnp.ids, list) { |
| 781 | /* First, check the ACPI/PNP IDs provided by the caller. */ | 787 | /* First, check the ACPI/PNP IDs provided by the caller. */ |
| 782 | for (id = ids; id->id[0] || id->cls; id++) { | 788 | if (acpi_ids) { |
| 783 | if (id->id[0] && !strcmp((char *) id->id, hwid->id)) | 789 | for (id = acpi_ids; id->id[0] || id->cls; id++) { |
| 784 | return id; | 790 | if (id->id[0] && !strcmp((char *)id->id, hwid->id)) |
| 785 | else if (id->cls && __acpi_match_device_cls(id, hwid)) | 791 | goto out_acpi_match; |
| 786 | return id; | 792 | if (id->cls && __acpi_match_device_cls(id, hwid)) |
| 793 | goto out_acpi_match; | ||
| 794 | } | ||
| 787 | } | 795 | } |
| 788 | 796 | ||
| 789 | /* | 797 | /* |
| 790 | * Next, check ACPI_DT_NAMESPACE_HID and try to match the | 798 | * Next, check ACPI_DT_NAMESPACE_HID and try to match the |
| 791 | * "compatible" property if found. | 799 | * "compatible" property if found. |
| 792 | * | ||
| 793 | * The id returned by the below is not valid, but the only | ||
| 794 | * caller passing non-NULL of_ids here is only interested in | ||
| 795 | * whether or not the return value is NULL. | ||
| 796 | */ | 800 | */ |
| 797 | if (!strcmp(ACPI_DT_NAMESPACE_HID, hwid->id) | 801 | if (!strcmp(ACPI_DT_NAMESPACE_HID, hwid->id)) |
| 798 | && acpi_of_match_device(device, of_ids)) | 802 | return acpi_of_match_device(device, of_ids, of_id); |
| 799 | return id; | ||
| 800 | } | 803 | } |
| 801 | return NULL; | 804 | return false; |
| 805 | |||
| 806 | out_acpi_match: | ||
| 807 | if (acpi_id) | ||
| 808 | *acpi_id = id; | ||
| 809 | return true; | ||
| 802 | } | 810 | } |
| 803 | 811 | ||
| 804 | /** | 812 | /** |
| @@ -815,32 +823,29 @@ static const struct acpi_device_id *__acpi_match_device( · 10 unmodified lines | |||
| 815 | const struct acpi_device_id *acpi_match_device(const struct acpi_device_id *ids, | 823 | const struct acpi_device_id *acpi_match_device(const struct acpi_device_id *ids, |
| 816 | const struct device *dev) | 824 | const struct device *dev) |
| 817 | { | 825 | { |
| 818 | return __acpi_match_device(acpi_companion_match(dev), ids, NULL); | 826 | const struct acpi_device_id *id = NULL; |
| 827 | |||
| 828 | __acpi_match_device(acpi_companion_match(dev), ids, NULL, &id, NULL); | ||
| 829 | return id; | ||
| 819 | } | 830 | } |
| 820 | EXPORT_SYMBOL_GPL(acpi_match_device); | 831 | EXPORT_SYMBOL_GPL(acpi_match_device); |
| 821 | 832 | ||
| 822 | void *acpi_get_match_data(const struct device *dev) | 833 | const void *acpi_device_get_match_data(const struct device *dev) |
| 823 | { | 834 | { |
| 824 | const struct acpi_device_id *match; | 835 | const struct acpi_device_id *match; |
| 825 | 836 | ||
| 826 | if (!dev->driver) | ||
| 827 | return NULL; | ||
| 828 | |||
| 829 | if (!dev->driver->acpi_match_table) | ||
| 830 | return NULL; | ||
| 831 | |||
| 832 | match = acpi_match_device(dev->driver->acpi_match_table, dev); | 837 | match = acpi_match_device(dev->driver->acpi_match_table, dev); |
| 833 | if (!match) | 838 | if (!match) |
| 834 | return NULL; | 839 | return NULL; |
| 835 | 840 | ||
| 836 | return (void *)match->driver_data; | 841 | return (const void *)match->driver_data; |
| 837 | } | 842 | } |
| 838 | EXPORT_SYMBOL_GPL(acpi_get_match_data); | 843 | EXPORT_SYMBOL_GPL(acpi_device_get_match_data); |
| 839 | 844 | ||
| 840 | int acpi_match_device_ids(struct acpi_device *device, | 845 | int acpi_match_device_ids(struct acpi_device *device, |
| 841 | const struct acpi_device_id *ids) | 846 | const struct acpi_device_id *ids) |
| 842 | { | 847 | { |
| 843 | return __acpi_match_device(device, ids, NULL) ? 0 : -ENOENT; | 848 | return __acpi_match_device(device, ids, NULL, NULL, NULL) ? 0 : -ENOENT; |
| 844 | } | 849 | } |
| 845 | EXPORT_SYMBOL(acpi_match_device_ids); | 850 | EXPORT_SYMBOL(acpi_match_device_ids); |
| 846 | 851 | ||
| @@ -849,10 +854,12 @@ bool acpi_driver_match_device(struct device *dev, · 2 unmodified lines | |||
| 849 | { | 854 | { |
| 850 | if (!drv->acpi_match_table) | 855 | if (!drv->acpi_match_table) |
| 851 | return acpi_of_match_device(ACPI_COMPANION(dev), | 856 | return acpi_of_match_device(ACPI_COMPANION(dev), |
| 852 | drv->of_match_table); | 857 | drv->of_match_table, |
| 858 | NULL); | ||
| 853 | 859 | ||
| 854 | return !!__acpi_match_device(acpi_companion_match(dev), | 860 | return __acpi_match_device(acpi_companion_match(dev), |
| 855 | drv->acpi_match_table, drv->of_match_table); | 861 | drv->acpi_match_table, drv->of_match_table, |
| 862 | NULL, NULL); | ||
| 856 | } | 863 | } |
| 857 | EXPORT_SYMBOL_GPL(acpi_driver_match_device); | 864 | EXPORT_SYMBOL_GPL(acpi_driver_match_device); |
| 858 | 865 | ||
| @@ -1927,6 +1927,9 @@ static int acpi_ec_suspend_noirq(struct device *dev) · 1926 unmodified lines | |||
| 1927 | ec->reference_count >= 1) | 1927 | ec->reference_count >= 1) |
| 1928 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_DISABLE); | 1928 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_DISABLE); |
| 1929 | 1929 | ||
| 1930 | if (acpi_sleep_no_ec_events()) | ||
| 1931 | acpi_ec_enter_noirq(ec); | ||
| 1932 | |||
| 1930 | return 0; | 1933 | return 0; |
| 1931 | } | 1934 | } |
| 1932 | 1935 | ||
| @@ -1934,6 +1937,9 @@ static int acpi_ec_resume_noirq(struct device *dev) · 1 unmodified line | |||
| 1934 | { | 1937 | { |
| 1935 | struct acpi_ec *ec = acpi_driver_data(to_acpi_device(dev)); | 1938 | struct acpi_ec *ec = acpi_driver_data(to_acpi_device(dev)); |
| 1936 | 1939 | ||
| 1940 | if (acpi_sleep_no_ec_events()) | ||
| 1941 | acpi_ec_leave_noirq(ec); | ||
| 1942 | |||
| 1937 | if (ec_no_wakeup && test_bit(EC_FLAGS_STARTED, &ec->flags) && | 1943 | if (ec_no_wakeup && test_bit(EC_FLAGS_STARTED, &ec->flags) && |
| 1938 | ec->reference_count >= 1) | 1944 | ec->reference_count >= 1) |
| 1939 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_ENABLE); | 1945 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_ENABLE); |
| @@ -2675,10 +2675,14 @@ static int acpi_nfit_register_region(struct acpi_nfit_desc *acpi_desc, · 2674 unmodified lines | |||
| 2675 | else | 2675 | else |
| 2676 | ndr_desc->numa_node = NUMA_NO_NODE; | 2676 | ndr_desc->numa_node = NUMA_NO_NODE; |
| 2677 | 2677 | ||
| 2678 | if(acpi_desc->platform_cap & ACPI_NFIT_CAPABILITY_CACHE_FLUSH) | 2678 | /* |
| 2679 | * Persistence domain bits are hierarchical, if | ||
| 2680 | * ACPI_NFIT_CAPABILITY_CACHE_FLUSH is set then | ||
| 2681 | * ACPI_NFIT_CAPABILITY_MEM_FLUSH is implied. | ||
| 2682 | */ | ||
| 2683 | if (acpi_desc->platform_cap & ACPI_NFIT_CAPABILITY_CACHE_FLUSH) | ||
| 2679 | set_bit(ND_REGION_PERSIST_CACHE, &ndr_desc->flags); | 2684 | set_bit(ND_REGION_PERSIST_CACHE, &ndr_desc->flags); |
| 2680 | 2685 | else if (acpi_desc->platform_cap & ACPI_NFIT_CAPABILITY_MEM_FLUSH) | |
| 2681 | if (acpi_desc->platform_cap & ACPI_NFIT_CAPABILITY_MEM_FLUSH) | ||
| 2682 | set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc->flags); | 2686 | set_bit(ND_REGION_PERSIST_MEMCTRL, &ndr_desc->flags); |
| 2683 | 2687 | ||
| 2684 | list_for_each_entry(nfit_memdev, &acpi_desc->memdevs, list) { | 2688 | list_for_each_entry(nfit_memdev, &acpi_desc->memdevs, list) { |
| @@ -103,25 +103,27 @@ int acpi_map_pxm_to_node(int pxm) · 102 unmodified lines | |||
| 103 | */ | 103 | */ |
| 104 | int acpi_map_pxm_to_online_node(int pxm) | 104 | int acpi_map_pxm_to_online_node(int pxm) |
| 105 | { | 105 | { |
| 106 | int node, n, dist, min_dist; | 106 | int node, min_node; |
| 107 | 107 | ||
| 108 | node = acpi_map_pxm_to_node(pxm); | 108 | node = acpi_map_pxm_to_node(pxm); |
| 109 | 109 | ||
| 110 | if (node == NUMA_NO_NODE) | 110 | if (node == NUMA_NO_NODE) |
| 111 | node = 0; | 111 | node = 0; |
| 112 | 112 | ||
| 113 | min_node = node; | ||
| 113 | if (!node_online(node)) { | 114 | if (!node_online(node)) { |
| 114 | min_dist = INT_MAX; | 115 | int min_dist = INT_MAX, dist, n; |
| 116 | |||
| 115 | for_each_online_node(n) { | 117 | for_each_online_node(n) { |
| 116 | dist = node_distance(node, n); | 118 | dist = node_distance(node, n); |
| 117 | if (dist < min_dist) { | 119 | if (dist < min_dist) { |
| 118 | min_dist = dist; | 120 | min_dist = dist; |
| 119 | node = n; | 121 | min_node = n; |
| 120 | } | 122 | } |
| 121 | } | 123 | } |
| 122 | } | 124 | } |
| 123 | 125 | ||
| 124 | return node; | 126 | return min_node; |
| 125 | } | 127 | } |
| 126 | EXPORT_SYMBOL(acpi_map_pxm_to_online_node); | 128 | EXPORT_SYMBOL(acpi_map_pxm_to_online_node); |
| 127 | 129 | ||
| @@ -1271,11 +1271,11 @@ static int acpi_fwnode_graph_parse_endpoint(const struct fwnode_handle *fwnode, · 1270 unmodified lines | |||
| 1271 | return 0; | 1271 | return 0; |
| 1272 | } | 1272 | } |
| 1273 | 1273 | ||
| 1274 | static void * | 1274 | static const void * |
| 1275 | acpi_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, | 1275 | acpi_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, |
| 1276 | const struct device *dev) | 1276 | const struct device *dev) |
| 1277 | { | 1277 | { |
| 1278 | return acpi_get_match_data(dev); | 1278 | return acpi_device_get_match_data(dev); |
| 1279 | } | 1279 | } |
| 1280 | 1280 | ||
| 1281 | #define DECLARE_ACPI_FWNODE_OPS(ops) \ | 1281 | #define DECLARE_ACPI_FWNODE_OPS(ops) \ |
| @@ -115,6 +115,7 @@ int __init acpi_parse_spcr(bool enable_earlycon, bool enable_console) · 114 unmodified lines | |||
| 115 | table->serial_port.access_width))) { | 115 | table->serial_port.access_width))) { |
| 116 | default: | 116 | default: |
| 117 | pr_err("Unexpected SPCR Access Width. Defaulting to byte size\n"); | 117 | pr_err("Unexpected SPCR Access Width. Defaulting to byte size\n"); |
| 118 | /* fall through */ | ||
| 118 | case 8: | 119 | case 8: |
| 119 | iotype = "mmio"; | 120 | iotype = "mmio"; |
| 120 | break; | 121 | break; |
| @@ -1991,8 +1991,14 @@ static void binder_send_failed_reply(struct binder_transaction *t, · 1990 unmodified lines | |||
| 1991 | &target_thread->reply_error.work); | 1991 | &target_thread->reply_error.work); |
| 1992 | wake_up_interruptible(&target_thread->wait); | 1992 | wake_up_interruptible(&target_thread->wait); |
| 1993 | } else { | 1993 | } else { |
| 1994 | WARN(1, "Unexpected reply error: %u\n", | 1994 | /* |
| 1995 | target_thread->reply_error.cmd); | 1995 | * Cannot get here for normal operation, but |
| 1996 | * we can if multiple synchronous transactions | ||
| 1997 | * are sent without blocking for responses. | ||
| 1998 | * Just ignore the 2nd error in this case. | ||
| 1999 | */ | ||
| 2000 | pr_warn("Unexpected reply error: %u\n", | ||
| 2001 | target_thread->reply_error.cmd); | ||
| 1996 | } | 2002 | } |
| 1997 | binder_inner_proc_unlock(target_thread->proc); | 2003 | binder_inner_proc_unlock(target_thread->proc); |
| 1998 | binder_thread_dec_tmpref(target_thread); | 2004 | binder_thread_dec_tmpref(target_thread); |
| @@ -2193,7 +2199,7 @@ static void binder_transaction_buffer_release(struct binder_proc *proc, · 194 unmodified lines | |||
| 2193 | int debug_id = buffer->debug_id; | 2199 | int debug_id = buffer->debug_id; |
| 2194 | 2200 | ||
| 2195 | binder_debug(BINDER_DEBUG_TRANSACTION, | 2201 | binder_debug(BINDER_DEBUG_TRANSACTION, |
| 2196 | "%d buffer release %d, size %zd-%zd, failed at %p\n", | 2202 | "%d buffer release %d, size %zd-%zd, failed at %pK\n", |
| 2197 | proc->pid, buffer->debug_id, | 2203 | proc->pid, buffer->debug_id, |
| 2198 | buffer->data_size, buffer->offsets_size, failed_at); | 2204 | buffer->data_size, buffer->offsets_size, failed_at); |
| 2199 | 2205 | ||
| @@ -3705,7 +3711,7 @@ static int binder_thread_write(struct binder_proc *proc, · 1505 unmodified lines | |||
| 3705 | } | 3711 | } |
| 3706 | } | 3712 | } |
| 3707 | binder_debug(BINDER_DEBUG_DEAD_BINDER, | 3713 | binder_debug(BINDER_DEBUG_DEAD_BINDER, |
| 3708 | "%d:%d BC_DEAD_BINDER_DONE %016llx found %p\n", | 3714 | "%d:%d BC_DEAD_BINDER_DONE %016llx found %pK\n", |
| 3709 | proc->pid, thread->pid, (u64)cookie, | 3715 | proc->pid, thread->pid, (u64)cookie, |
| 3710 | death); | 3716 | death); |
| 3711 | if (death == NULL) { | 3717 | if (death == NULL) { |
| @@ -4376,6 +4382,15 @@ static int binder_thread_release(struct binder_proc *proc, · 664 unmodified lines | |||
| 4376 | 4382 | ||
| 4377 | binder_inner_proc_unlock(thread->proc); | 4383 | binder_inner_proc_unlock(thread->proc); |
| 4378 | 4384 | ||
| 4385 | /* | ||
| 4386 | * This is needed to avoid races between wake_up_poll() above and | ||
| 4387 | * and ep_remove_waitqueue() called for other reasons (eg the epoll file | ||
| 4388 | * descriptor being closed); ep_remove_waitqueue() holds an RCU read | ||
| 4389 | * lock, so we can be sure it's done after calling synchronize_rcu(). | ||
| 4390 | */ | ||
| 4391 | if (thread->looper & BINDER_LOOPER_STATE_POLL) | ||
| 4392 | synchronize_rcu(); | ||
| 4393 | |||
| 4379 | if (send_reply) | 4394 | if (send_reply) |
| 4380 | binder_send_failed_reply(send_reply, BR_DEAD_REPLY); | 4395 | binder_send_failed_reply(send_reply, BR_DEAD_REPLY); |
| 4381 | binder_release_work(proc, &thread->todo); | 4396 | binder_release_work(proc, &thread->todo); |
| @@ -4391,6 +4406,8 @@ static __poll_t binder_poll(struct file *filp, · 9 unmodified lines | |||
| 4391 | bool wait_for_proc_work; | 4406 | bool wait_for_proc_work; |
| 4392 | 4407 | ||
| 4393 | thread = binder_get_thread(proc); | 4408 | thread = binder_get_thread(proc); |
| 4409 | if (!thread) | ||
| 4410 | return POLLERR; | ||
| 4394 | 4411 | ||
| 4395 | binder_inner_proc_lock(thread->proc); | 4412 | binder_inner_proc_lock(thread->proc); |
| 4396 | thread->looper |= BINDER_LOOPER_STATE_POLL; | 4413 | thread->looper |= BINDER_LOOPER_STATE_POLL; |
| @@ -5034,7 +5051,7 @@ static void print_binder_transaction_ilocked(struct seq_file *m, · 637 unmodified lines | |||
| 5034 | spin_lock(&t->lock); | 5051 | spin_lock(&t->lock); |
| 5035 | to_proc = t->to_proc; | 5052 | to_proc = t->to_proc; |
| 5036 | seq_printf(m, | 5053 | seq_printf(m, |
| 5037 | "%s %d: %p from %d:%d to %d:%d code %x flags %x pri %ld r%d", | 5054 | "%s %d: %pK from %d:%d to %d:%d code %x flags %x pri %ld r%d", |
| 5038 | prefix, t->debug_id, t, | 5055 | prefix, t->debug_id, t, |
| 5039 | t->from ? t->from->proc->pid : 0, | 5056 | t->from ? t->from->proc->pid : 0, |
| 5040 | t->from ? t->from->pid : 0, | 5057 | t->from ? t->from->pid : 0, |
| @@ -5058,7 +5075,7 @@ static void print_binder_transaction_ilocked(struct seq_file *m, · 17 unmodified lines | |||
| 5058 | } | 5075 | } |
| 5059 | if (buffer->target_node) | 5076 | if (buffer->target_node) |
| 5060 | seq_printf(m, " node %d", buffer->target_node->debug_id); | 5077 | seq_printf(m, " node %d", buffer->target_node->debug_id); |
| 5061 | seq_printf(m, " size %zd:%zd data %p\n", | 5078 | seq_printf(m, " size %zd:%zd data %pK\n", |
| 5062 | buffer->data_size, buffer->offsets_size, | 5079 | buffer->data_size, buffer->offsets_size, |
| 5063 | buffer->data); | 5080 | buffer->data); |
| 5064 | } | 5081 | } |
| @@ -550,7 +550,9 @@ static const struct pci_device_id ahci_pci_tbl[] = { · 549 unmodified lines | |||
| 550 | .driver_data = board_ahci_yes_fbs }, | 550 | .driver_data = board_ahci_yes_fbs }, |
| 551 | { PCI_DEVICE(PCI_VENDOR_ID_MARVELL_EXT, 0x9230), | 551 | { PCI_DEVICE(PCI_VENDOR_ID_MARVELL_EXT, 0x9230), |
| 552 | .driver_data = board_ahci_yes_fbs }, | 552 | .driver_data = board_ahci_yes_fbs }, |
| 553 | { PCI_DEVICE(PCI_VENDOR_ID_TTI, 0x0642), | 553 | { PCI_DEVICE(PCI_VENDOR_ID_TTI, 0x0642), /* highpoint rocketraid 642L */ |
| 554 | .driver_data = board_ahci_yes_fbs }, | ||
| 555 | { PCI_DEVICE(PCI_VENDOR_ID_TTI, 0x0645), /* highpoint rocketraid 644L */ | ||
| 554 | .driver_data = board_ahci_yes_fbs }, | 556 | .driver_data = board_ahci_yes_fbs }, |
| 555 | 557 | ||
| 556 | /* Promise */ | 558 | /* Promise */ |
| @@ -665,6 +665,16 @@ int ahci_stop_engine(struct ata_port *ap) · 664 unmodified lines | |||
| 665 | if ((tmp & (PORT_CMD_START | PORT_CMD_LIST_ON)) == 0) | 665 | if ((tmp & (PORT_CMD_START | PORT_CMD_LIST_ON)) == 0) |
| 666 | return 0; | 666 | return 0; |
| 667 | 667 | ||
| 668 | /* | ||
| 669 | * Don't try to issue commands but return with ENODEV if the | ||
| 670 | * AHCI controller not available anymore (e.g. due to PCIe hot | ||
| 671 | * unplugging). Otherwise a 500ms delay for each port is added. | ||
| 672 | */ | ||
| 673 | if (tmp == 0xffffffff) { | ||
| 674 | dev_err(ap->host->dev, "AHCI controller unavailable!\n"); | ||
| 675 | return -ENODEV; | ||
| 676 | } | ||
| 677 | |||
| 668 | /* setting HBA to idle */ | 678 | /* setting HBA to idle */ |
| 669 | tmp &= ~PORT_CMD_START; | 679 | tmp &= ~PORT_CMD_START; |
| 670 | writel(tmp, port_mmio + PORT_CMD); | 680 | writel(tmp, port_mmio + PORT_CMD); |
| @@ -340,7 +340,7 @@ static int ahci_platform_get_regulator(struct ahci_host_priv *hpriv, u32 port, · 339 unmodified lines | |||
| 340 | * 2) regulator for controlling the targets power (optional) | 340 | * 2) regulator for controlling the targets power (optional) |
| 341 | * 3) 0 - AHCI_MAX_CLKS clocks, as specified in the devs devicetree node, | 341 | * 3) 0 - AHCI_MAX_CLKS clocks, as specified in the devs devicetree node, |
| 342 | * or for non devicetree enabled platforms a single clock | 342 | * or for non devicetree enabled platforms a single clock |
| 343 | * 4) phys (optional) | 343 | * 4) phys (optional) |
| 344 | * | 344 | * |
| 345 | * RETURNS: | 345 | * RETURNS: |
| 346 | * The allocated ahci_host_priv on success, otherwise an ERR_PTR value | 346 | * The allocated ahci_host_priv on success, otherwise an ERR_PTR value |
| @@ -4530,6 +4530,25 @@ static const struct ata_blacklist_entry ata_device_blacklist [] = { · 4529 unmodified lines | |||
| 4530 | { "PIONEER DVD-RW DVR-212D", NULL, ATA_HORKAGE_NOSETXFER }, | 4530 | { "PIONEER DVD-RW DVR-212D", NULL, ATA_HORKAGE_NOSETXFER }, |
| 4531 | { "PIONEER DVD-RW DVR-216D", NULL, ATA_HORKAGE_NOSETXFER }, | 4531 | { "PIONEER DVD-RW DVR-216D", NULL, ATA_HORKAGE_NOSETXFER }, |
| 4532 | 4532 | ||
| 4533 | /* Crucial BX100 SSD 500GB has broken LPM support */ | ||
| 4534 | { "CT500BX100SSD1", NULL, ATA_HORKAGE_NOLPM }, | ||
| 4535 | |||
| 4536 | /* 512GB MX100 with MU01 firmware has both queued TRIM and LPM issues */ | ||
| 4537 | { "Crucial_CT512MX100*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | | ||
| 4538 | ATA_HORKAGE_ZERO_AFTER_TRIM | | ||
| 4539 | ATA_HORKAGE_NOLPM, }, | ||
| 4540 | /* 512GB MX100 with newer firmware has only LPM issues */ | ||
| 4541 | { "Crucial_CT512MX100*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM | | ||
| 4542 | ATA_HORKAGE_NOLPM, }, | ||
| 4543 | |||
| 4544 | /* 480GB+ M500 SSDs have both queued TRIM and LPM issues */ | ||
| 4545 | { "Crucial_CT480M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | | ||
| 4546 | ATA_HORKAGE_ZERO_AFTER_TRIM | | ||
| 4547 | ATA_HORKAGE_NOLPM, }, | ||
| 4548 | { "Crucial_CT960M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | | ||
| 4549 | ATA_HORKAGE_ZERO_AFTER_TRIM | | ||
| 4550 | ATA_HORKAGE_NOLPM, }, | ||
| 4551 | |||
| 4533 | /* devices that don't properly handle queued TRIM commands */ | 4552 | /* devices that don't properly handle queued TRIM commands */ |
| 4534 | { "Micron_M500_*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | | 4553 | { "Micron_M500_*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | |
| 4535 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, | 4554 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, |
| @@ -4541,7 +4560,9 @@ static const struct ata_blacklist_entry ata_device_blacklist [] = { · 5 unmodified lines | |||
| 4541 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, | 4560 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, |
| 4542 | { "Crucial_CT*MX100*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | | 4561 | { "Crucial_CT*MX100*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | |
| 4543 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, | 4562 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, |
| 4544 | { "Samsung SSD 8*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | | 4563 | { "Samsung SSD 840*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | |
| 4564 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, | ||
| 4565 | { "Samsung SSD 850*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | | ||
| 4545 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, | 4566 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, |
| 4546 | { "FCCT*M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | | 4567 | { "FCCT*M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | |
| 4547 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, | 4568 | ATA_HORKAGE_ZERO_AFTER_TRIM, }, |
| @@ -5401,8 +5422,7 @@ void ata_qc_issue(struct ata_queued_cmd *qc) · 853 unmodified lines | |||
| 5401 | * We guarantee to LLDs that they will have at least one | 5422 | * We guarantee to LLDs that they will have at least one |
| 5402 | * non-zero sg if the command is a data command. | 5423 | * non-zero sg if the command is a data command. |
| 5403 | */ | 5424 | */ |
| 5404 | if (WARN_ON_ONCE(ata_is_data(prot) && | 5425 | if (ata_is_data(prot) && (!qc->sg || !qc->n_elem || !qc->nbytes)) |
| 5405 | (!qc->sg || !qc->n_elem || !qc->nbytes))) | ||
| 5406 | goto sys_err; | 5426 | goto sys_err; |
| 5407 | 5427 | ||
| 5408 | if (ata_is_dma(prot) || (ata_is_pio(prot) && | 5428 | if (ata_is_dma(prot) || (ata_is_pio(prot) && |
| @@ -815,7 +815,8 @@ void ata_scsi_port_error_handler(struct Scsi_Host *host, struct ata_port *ap) · 814 unmodified lines | |||
| 815 | 815 | ||
| 816 | if (ap->pflags & ATA_PFLAG_LOADING) | 816 | if (ap->pflags & ATA_PFLAG_LOADING) |
| 817 | ap->pflags &= ~ATA_PFLAG_LOADING; | 817 | ap->pflags &= ~ATA_PFLAG_LOADING; |
| 818 | else if (ap->pflags & ATA_PFLAG_SCSI_HOTPLUG) | 818 | else if ((ap->pflags & ATA_PFLAG_SCSI_HOTPLUG) && |
| 819 | !(ap->flags & ATA_FLAG_SAS_HOST)) | ||
| 819 | schedule_delayed_work(&ap->hotplug_task, 0); | 820 | schedule_delayed_work(&ap->hotplug_task, 0); |
| 820 | 821 | ||
| 821 | if (ap->pflags & ATA_PFLAG_RECOVERED) | 822 | if (ap->pflags & ATA_PFLAG_RECOVERED) |
| @@ -3316,6 +3316,12 @@ static unsigned int ata_scsi_pass_thru(struct ata_queued_cmd *qc) · 3315 unmodified lines | |||
| 3316 | goto invalid_fld; | 3316 | goto invalid_fld; |
| 3317 | } | 3317 | } |
| 3318 | 3318 | ||
| 3319 | /* We may not issue NCQ commands to devices not supporting NCQ */ | ||
| 3320 | if (ata_is_ncq(tf->protocol) && !ata_ncq_enabled(dev)) { | ||
| 3321 | fp = 1; | ||
| 3322 | goto invalid_fld; | ||
| 3323 | } | ||
| 3324 | |||
| 3319 | /* sanity check for pio multi commands */ | 3325 | /* sanity check for pio multi commands */ |
| 3320 | if ((cdb[1] & 0xe0) && !is_multi_taskfile(tf)) { | 3326 | if ((cdb[1] & 0xe0) && !is_multi_taskfile(tf)) { |
| 3321 | fp = 1; | 3327 | fp = 1; |
| @@ -4282,7 +4288,7 @@ static inline void ata_scsi_dump_cdb(struct ata_port *ap, · 960 unmodified lines | |||
| 4282 | #ifdef ATA_DEBUG | 4288 | #ifdef ATA_DEBUG |
| 4283 | struct scsi_device *scsidev = cmd->device; | 4289 | struct scsi_device *scsidev = cmd->device; |
| 4284 | 4290 | ||
| 4285 | DPRINTK("CDB (%u:%d,%d,%d) %9ph\n", | 4291 | DPRINTK("CDB (%u:%d,%d,%lld) %9ph\n", |
| 4286 | ap->print_id, | 4292 | ap->print_id, |
| 4287 | scsidev->channel, scsidev->id, scsidev->lun, | 4293 | scsidev->channel, scsidev->id, scsidev->lun, |
| 4288 | cmd->cmnd); | 4294 | cmd->cmnd); |
| @@ -4309,7 +4315,9 @@ static inline int __ata_scsi_queuecmd(struct scsi_cmnd *scmd, · 20 unmodified lines | |||
| 4309 | if (likely((scsi_op != ATA_16) || !atapi_passthru16)) { | 4315 | if (likely((scsi_op != ATA_16) || !atapi_passthru16)) { |
| 4310 | /* relay SCSI command to ATAPI device */ | 4316 | /* relay SCSI command to ATAPI device */ |
| 4311 | int len = COMMAND_SIZE(scsi_op); | 4317 | int len = COMMAND_SIZE(scsi_op); |
| 4312 | if (unlikely(len > scmd->cmd_len || len > dev->cdb_len)) | 4318 | if (unlikely(len > scmd->cmd_len || |
| 4319 | len > dev->cdb_len || | ||
| 4320 | scmd->cmd_len > ATAPI_CDB_LEN)) | ||
| 4313 | goto bad_cdb_len; | 4321 | goto bad_cdb_len; |
| 4314 | 4322 | ||
| 4315 | xlat_func = atapi_xlat; | 4323 | xlat_func = atapi_xlat; |
| @@ -146,6 +146,7 @@ · 145 unmodified lines | |||
| 146 | enum sata_rcar_type { | 146 | enum sata_rcar_type { |
| 147 | RCAR_GEN1_SATA, | 147 | RCAR_GEN1_SATA, |
| 148 | RCAR_GEN2_SATA, | 148 | RCAR_GEN2_SATA, |
| 149 | RCAR_GEN3_SATA, | ||
| 149 | RCAR_R8A7790_ES1_SATA, | 150 | RCAR_R8A7790_ES1_SATA, |
| 150 | }; | 151 | }; |
| 151 | 152 | ||
| @@ -784,26 +785,11 @@ static void sata_rcar_setup_port(struct ata_host *host) · 632 unmodified lines | |||
| 784 | ioaddr->command_addr = ioaddr->cmd_addr + (ATA_REG_CMD << 2); | 785 | ioaddr->command_addr = ioaddr->cmd_addr + (ATA_REG_CMD << 2); |
| 785 | } | 786 | } |
| 786 | 787 | ||
| 787 | static void sata_rcar_init_controller(struct ata_host *host) | 788 | static void sata_rcar_init_module(struct sata_rcar_priv *priv) |
| 788 | { | 789 | { |
| 789 | struct sata_rcar_priv *priv = host->private_data; | ||
| 790 | void __iomem *base = priv->base; | 790 | void __iomem *base = priv->base; |
| 791 | u32 val; | 791 | u32 val; |
| 792 | 792 | ||
| 793 | /* reset and setup phy */ | ||
| 794 | switch (priv->type) { | ||
| 795 | case RCAR_GEN1_SATA: | ||
| 796 | sata_rcar_gen1_phy_init(priv); | ||
| 797 | break; | ||
| 798 | case RCAR_GEN2_SATA: | ||
| 799 | case RCAR_R8A7790_ES1_SATA: | ||
| 800 | sata_rcar_gen2_phy_init(priv); | ||
| 801 | break; | ||
| 802 | default: | ||
| 803 | dev_warn(host->dev, "SATA phy is not initialized\n"); | ||
| 804 | break; | ||
| 805 | } | ||
| 806 | |||
| 807 | /* SATA-IP reset state */ | 793 | /* SATA-IP reset state */ |
| 808 | val = ioread32(base + ATAPI_CONTROL1_REG); | 794 | val = ioread32(base + ATAPI_CONTROL1_REG); |
| 809 | val |= ATAPI_CONTROL1_RESET; | 795 | val |= ATAPI_CONTROL1_RESET; |
| @@ -824,10 +810,33 @@ static void sata_rcar_init_controller(struct ata_host *host) · 14 unmodified lines | |||
| 824 | /* ack and mask */ | 810 | /* ack and mask */ |
| 825 | iowrite32(0, base + SATAINTSTAT_REG); | 811 | iowrite32(0, base + SATAINTSTAT_REG); |
| 826 | iowrite32(0x7ff, base + SATAINTMASK_REG); | 812 | iowrite32(0x7ff, base + SATAINTMASK_REG); |
| 813 | |||
| 827 | /* enable interrupts */ | 814 | /* enable interrupts */ |
| 828 | iowrite32(ATAPI_INT_ENABLE_SATAINT, base + ATAPI_INT_ENABLE_REG); | 815 | iowrite32(ATAPI_INT_ENABLE_SATAINT, base + ATAPI_INT_ENABLE_REG); |
| 829 | } | 816 | } |
| 830 | 817 | ||
| 818 | static void sata_rcar_init_controller(struct ata_host *host) | ||
| 819 | { | ||
| 820 | struct sata_rcar_priv *priv = host->private_data; | ||
| 821 | |||
| 822 | /* reset and setup phy */ | ||
| 823 | switch (priv->type) { | ||
| 824 | case RCAR_GEN1_SATA: | ||
| 825 | sata_rcar_gen1_phy_init(priv); | ||
| 826 | break; | ||
| 827 | case RCAR_GEN2_SATA: | ||
| 828 | case RCAR_GEN3_SATA: | ||
| 829 | case RCAR_R8A7790_ES1_SATA: | ||
| 830 | sata_rcar_gen2_phy_init(priv); | ||
| 831 | break; | ||
| 832 | default: | ||
| 833 | dev_warn(host->dev, "SATA phy is not initialized\n"); | ||
| 834 | break; | ||
| 835 | } | ||
| 836 | |||
| 837 | sata_rcar_init_module(priv); | ||
| 838 | } | ||
| 839 | |||
| 831 | static const struct of_device_id sata_rcar_match[] = { | 840 | static const struct of_device_id sata_rcar_match[] = { |
| 832 | { | 841 | { |
| 833 | /* Deprecated by "renesas,sata-r8a7779" */ | 842 | /* Deprecated by "renesas,sata-r8a7779" */ |
| @@ -856,7 +865,7 @@ static const struct of_device_id sata_rcar_match[] = { · 22 unmodified lines | |||
| 856 | }, | 865 | }, |
| 857 | { | 866 | { |
| 858 | .compatible = "renesas,sata-r8a7795", | 867 | .compatible = "renesas,sata-r8a7795", |
| 859 | .data = (void *)RCAR_GEN2_SATA | 868 | .data = (void *)RCAR_GEN3_SATA |
| 860 | }, | 869 | }, |
| 861 | { | 870 | { |
| 862 | .compatible = "renesas,rcar-gen2-sata", | 871 | .compatible = "renesas,rcar-gen2-sata", |
| @@ -864,7 +873,7 @@ static const struct of_device_id sata_rcar_match[] = { · 1 unmodified line | |||
| 864 | }, | 873 | }, |
| 865 | { | 874 | { |
| 866 | .compatible = "renesas,rcar-gen3-sata", | 875 | .compatible = "renesas,rcar-gen3-sata", |
| 867 | .data = (void *)RCAR_GEN2_SATA | 876 | .data = (void *)RCAR_GEN3_SATA |
| 868 | }, | 877 | }, |
| 869 | { }, | 878 | { }, |
| 870 | }; | 879 | }; |
| @@ -982,11 +991,18 @@ static int sata_rcar_resume(struct device *dev) · 111 unmodified lines | |||
| 982 | if (ret) | 991 | if (ret) |
| 983 | return ret; | 992 | return ret; |
| 984 | 993 | ||
| 985 | /* ack and mask */ | 994 | if (priv->type == RCAR_GEN3_SATA) { |
| 986 | iowrite32(0, base + SATAINTSTAT_REG); | 995 | sata_rcar_gen2_phy_init(priv); |
| 987 | iowrite32(0x7ff, base + SATAINTMASK_REG); | 996 | sata_rcar_init_module(priv); |
| 988 | /* enable interrupts */ | 997 | } else { |
| 989 | iowrite32(ATAPI_INT_ENABLE_SATAINT, base + ATAPI_INT_ENABLE_REG); | 998 | /* ack and mask */ |
| 999 | iowrite32(0, base + SATAINTSTAT_REG); | ||
| 1000 | iowrite32(0x7ff, base + SATAINTMASK_REG); | ||
| 1001 | |||
| 1002 | /* enable interrupts */ | ||
| 1003 | iowrite32(ATAPI_INT_ENABLE_SATAINT, | ||
| 1004 | base + ATAPI_INT_ENABLE_REG); | ||
| 1005 | } | ||
| 990 | 1006 | ||
| 991 | ata_host_resume(host); | 1007 | ata_host_resume(host); |
| 992 | 1008 | ||
| @@ -97,7 +97,7 @@ static struct img_ascii_lcd_config boston_config = { · 96 unmodified lines | |||
| 97 | static void malta_update(struct img_ascii_lcd_ctx *ctx) | 97 | static void malta_update(struct img_ascii_lcd_ctx *ctx) |
| 98 | { | 98 | { |
| 99 | unsigned int i; | 99 | unsigned int i; |
| 100 | int err; | 100 | int err = 0; |
| 101 | 101 | ||
| 102 | for (i = 0; i < ctx->cfg->num_chars; i++) { | 102 | for (i = 0; i < ctx->cfg->num_chars; i++) { |
| 103 | err = regmap_write(ctx->regmap, | 103 | err = regmap_write(ctx->regmap, |
| @@ -180,7 +180,7 @@ static int sead3_wait_lcd_idle(struct img_ascii_lcd_ctx *ctx) · 76 unmodified lines | |||
| 180 | static void sead3_update(struct img_ascii_lcd_ctx *ctx) | 180 | static void sead3_update(struct img_ascii_lcd_ctx *ctx) |
| 181 | { | 181 | { |
| 182 | unsigned int i; | 182 | unsigned int i; |
| 183 | int err; | 183 | int err = 0; |
| 184 | 184 | ||
| 185 | for (i = 0; i < ctx->cfg->num_chars; i++) { | 185 | for (i = 0; i < ctx->cfg->num_chars; i++) { |
| 186 | err = sead3_wait_lcd_idle(ctx); | 186 | err = sead3_wait_lcd_idle(ctx); |
| @@ -224,7 +224,7 @@ MODULE_DEVICE_TABLE(of, img_ascii_lcd_matches); · 37 unmodified lines | |||
| 224 | 224 | ||
| 225 | /** | 225 | /** |
| 226 | * img_ascii_lcd_scroll() - scroll the display by a character | 226 | * img_ascii_lcd_scroll() - scroll the display by a character |
| 227 | * @arg: really a pointer to the private data structure | 227 | * @t: really a pointer to the private data structure |
| 228 | * | 228 | * |
| 229 | * Scroll the current message along the LCD by one character, rearming the | 229 | * Scroll the current message along the LCD by one character, rearming the |
| 230 | * timer if required. | 230 | * timer if required. |
| @@ -1372,7 +1372,7 @@ static void panel_process_inputs(void) · 1371 unmodified lines | |||
| 1372 | break; | 1372 | break; |
| 1373 | input->rise_timer = 0; | 1373 | input->rise_timer = 0; |
| 1374 | input->state = INPUT_ST_RISING; | 1374 | input->state = INPUT_ST_RISING; |
| 1375 | /* no break here, fall through */ | 1375 | /* fall through */ |
| 1376 | case INPUT_ST_RISING: | 1376 | case INPUT_ST_RISING: |
| 1377 | if ((phys_curr & input->mask) != input->value) { | 1377 | if ((phys_curr & input->mask) != input->value) { |
| 1378 | input->state = INPUT_ST_LOW; | 1378 | input->state = INPUT_ST_LOW; |
| @@ -1385,11 +1385,11 @@ static void panel_process_inputs(void) · 6 unmodified lines | |||
| 1385 | } | 1385 | } |
| 1386 | input->high_timer = 0; | 1386 | input->high_timer = 0; |
| 1387 | input->state = INPUT_ST_HIGH; | 1387 | input->state = INPUT_ST_HIGH; |
| 1388 | /* no break here, fall through */ | 1388 | /* fall through */ |
| 1389 | case INPUT_ST_HIGH: | 1389 | case INPUT_ST_HIGH: |
| 1390 | if (input_state_high(input)) | 1390 | if (input_state_high(input)) |
| 1391 | break; | 1391 | break; |
| 1392 | /* no break here, fall through */ | 1392 | /* fall through */ |
| 1393 | case INPUT_ST_FALLING: | 1393 | case INPUT_ST_FALLING: |
| 1394 | input_state_falling(input); | 1394 | input_state_falling(input); |
| 1395 | } | 1395 | } |
| @@ -310,6 +310,9 @@ static void __device_link_del(struct device_link *link) · 309 unmodified lines | |||
| 310 | dev_info(link->consumer, "Dropping the link to %s\n", | 310 | dev_info(link->consumer, "Dropping the link to %s\n", |
| 311 | dev_name(link->supplier)); | 311 | dev_name(link->supplier)); |
| 312 | 312 | ||
| 313 | if (link->flags & DL_FLAG_PM_RUNTIME) | ||
| 314 | pm_runtime_drop_link(link->consumer); | ||
| 315 | |||
| 313 | list_del(&link->s_node); | 316 | list_del(&link->s_node); |
| 314 | list_del(&link->c_node); | 317 | list_del(&link->c_node); |
| 315 | device_link_free(link); | 318 | device_link_free(link); |
| @@ -321,7 +321,8 @@ void dev_pm_arm_wake_irq(struct wake_irq *wirq) · 320 unmodified lines | |||
| 321 | return; | 321 | return; |
| 322 | 322 | ||
| 323 | if (device_may_wakeup(wirq->dev)) { | 323 | if (device_may_wakeup(wirq->dev)) { |
| 324 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED) | 324 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED && |
| 325 | !pm_runtime_status_suspended(wirq->dev)) | ||
| 325 | enable_irq(wirq->irq); | 326 | enable_irq(wirq->irq); |
| 326 | 327 | ||
| 327 | enable_irq_wake(wirq->irq); | 328 | enable_irq_wake(wirq->irq); |
| @@ -343,7 +344,8 @@ void dev_pm_disarm_wake_irq(struct wake_irq *wirq) · 15 unmodified lines | |||
| 343 | if (device_may_wakeup(wirq->dev)) { | 344 | if (device_may_wakeup(wirq->dev)) { |
| 344 | disable_irq_wake(wirq->irq); | 345 | disable_irq_wake(wirq->irq); |
| 345 | 346 | ||
| 346 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED) | 347 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED && |
| 348 | !pm_runtime_status_suspended(wirq->dev)) | ||
| 347 | disable_irq_nosync(wirq->irq); | 349 | disable_irq_nosync(wirq->irq); |
| 348 | } | 350 | } |
| 349 | } | 351 | } |
| @@ -1410,9 +1410,8 @@ int fwnode_graph_parse_endpoint(const struct fwnode_handle *fwnode, · 1409 unmodified lines | |||
| 1410 | } | 1410 | } |
| 1411 | EXPORT_SYMBOL(fwnode_graph_parse_endpoint); | 1411 | EXPORT_SYMBOL(fwnode_graph_parse_endpoint); |
| 1412 | 1412 | ||
| 1413 | void *device_get_match_data(struct device *dev) | 1413 | const void *device_get_match_data(struct device *dev) |
| 1414 | { | 1414 | { |
| 1415 | return fwnode_call_ptr_op(dev_fwnode(dev), device_get_match_data, | 1415 | return fwnode_call_ptr_op(dev_fwnode(dev), device_get_match_data, dev); |
| 1416 | dev); | ||
| 1417 | } | 1416 | } |
| 1418 | EXPORT_SYMBOL_GPL(device_get_match_data); | 1417 | EXPORT_SYMBOL_GPL(device_get_match_data); |
| @@ -1758,7 +1758,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 1757 unmodified lines | |||
| 1758 | if (unit[drive].type->code == FD_NODRIVE) | 1758 | if (unit[drive].type->code == FD_NODRIVE) |
| 1759 | return NULL; | 1759 | return NULL; |
| 1760 | *part = 0; | 1760 | *part = 0; |
| 1761 | return get_disk(unit[drive].gendisk); | 1761 | return get_disk_and_module(unit[drive].gendisk); |
| 1762 | } | 1762 | } |
| 1763 | 1763 | ||
| 1764 | static int __init amiga_floppy_probe(struct platform_device *pdev) | 1764 | static int __init amiga_floppy_probe(struct platform_device *pdev) |
| @@ -1917,7 +1917,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 1916 unmodified lines | |||
| 1917 | if (drive >= FD_MAX_UNITS || type > NUM_DISK_MINORS) | 1917 | if (drive >= FD_MAX_UNITS || type > NUM_DISK_MINORS) |
| 1918 | return NULL; | 1918 | return NULL; |
| 1919 | *part = 0; | 1919 | *part = 0; |
| 1920 | return get_disk(unit[drive].disk); | 1920 | return get_disk_and_module(unit[drive].disk); |
| 1921 | } | 1921 | } |
| 1922 | 1922 | ||
| 1923 | static int __init atari_floppy_init (void) | 1923 | static int __init atari_floppy_init (void) |
| @@ -456,7 +456,7 @@ static struct kobject *brd_probe(dev_t dev, int *part, void *data) · 455 unmodified lines | |||
| 456 | 456 | ||
| 457 | mutex_lock(&brd_devices_mutex); | 457 | mutex_lock(&brd_devices_mutex); |
| 458 | brd = brd_init_one(MINOR(dev) / max_part, &new); | 458 | brd = brd_init_one(MINOR(dev) / max_part, &new); |
| 459 | kobj = brd ? get_disk(brd->brd_disk) : NULL; | 459 | kobj = brd ? get_disk_and_module(brd->brd_disk) : NULL; |
| 460 | mutex_unlock(&brd_devices_mutex); | 460 | mutex_unlock(&brd_devices_mutex); |
| 461 | 461 | ||
| 462 | if (new) | 462 | if (new) |
| @@ -4505,7 +4505,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 4504 unmodified lines | |||
| 4505 | if (((*part >> 2) & 0x1f) >= ARRAY_SIZE(floppy_type)) | 4505 | if (((*part >> 2) & 0x1f) >= ARRAY_SIZE(floppy_type)) |
| 4506 | return NULL; | 4506 | return NULL; |
| 4507 | *part = 0; | 4507 | *part = 0; |
| 4508 | return get_disk(disks[drive]); | 4508 | return get_disk_and_module(disks[drive]); |
| 4509 | } | 4509 | } |
| 4510 | 4510 | ||
| 4511 | static int __init do_floppy_init(void) | 4511 | static int __init do_floppy_init(void) |
| @@ -266,7 +266,7 @@ static int lo_write_bvec(struct file *file, struct bio_vec *bvec, loff_t *ppos) · 265 unmodified lines | |||
| 266 | struct iov_iter i; | 266 | struct iov_iter i; |
| 267 | ssize_t bw; | 267 | ssize_t bw; |
| 268 | 268 | ||
| 269 | iov_iter_bvec(&i, ITER_BVEC, bvec, 1, bvec->bv_len); | 269 | iov_iter_bvec(&i, ITER_BVEC | WRITE, bvec, 1, bvec->bv_len); |
| 270 | 270 | ||
| 271 | file_start_write(file); | 271 | file_start_write(file); |
| 272 | bw = vfs_iter_write(file, &i, ppos, 0); | 272 | bw = vfs_iter_write(file, &i, ppos, 0); |
| @@ -1922,7 +1922,7 @@ static struct kobject *loop_probe(dev_t dev, int *part, void *data) · 1649 unmodified lines | |||
| 1922 | if (err < 0) | 1922 | if (err < 0) |
| 1923 | kobj = NULL; | 1923 | kobj = NULL; |
| 1924 | else | 1924 | else |
| 1925 | kobj = get_disk(lo->lo_disk); | 1925 | kobj = get_disk_and_module(lo->lo_disk); |
| 1926 | mutex_unlock(&loop_index_mutex); | 1926 | mutex_unlock(&loop_index_mutex); |
| 1927 | 1927 | ||
| 1928 | *part = 0; | 1928 | *part = 0; |
| @@ -1591,7 +1591,7 @@ again: · 1590 unmodified lines | |||
| 1591 | if (new_index < 0) { | 1591 | if (new_index < 0) { |
| 1592 | mutex_unlock(&nbd_index_mutex); | 1592 | mutex_unlock(&nbd_index_mutex); |
| 1593 | printk(KERN_ERR "nbd: failed to add new device\n"); | 1593 | printk(KERN_ERR "nbd: failed to add new device\n"); |
| 1594 | return ret; | 1594 | return new_index; |
| 1595 | } | 1595 | } |
| 1596 | nbd = idr_find(&nbd_index_idr, new_index); | 1596 | nbd = idr_find(&nbd_index_idr, new_index); |
| 1597 | } | 1597 | } |
| @@ -1122,7 +1122,7 @@ static int pkt_start_recovery(struct packet_data *pkt) · 1121 unmodified lines | |||
| 1122 | pkt->sector = new_sector; | 1122 | pkt->sector = new_sector; |
| 1123 | 1123 | ||
| 1124 | bio_reset(pkt->bio); | 1124 | bio_reset(pkt->bio); |
| 1125 | bio_set_set(pkt->bio, pd->bdev); | 1125 | bio_set_dev(pkt->bio, pd->bdev); |
| 1126 | bio_set_op_attrs(pkt->bio, REQ_OP_WRITE, 0); | 1126 | bio_set_op_attrs(pkt->bio, REQ_OP_WRITE, 0); |
| 1127 | pkt->bio->bi_iter.bi_sector = new_sector; | 1127 | pkt->bio->bi_iter.bi_sector = new_sector; |
| 1128 | pkt->bio->bi_iter.bi_size = pkt->frames * CD_FRAMESIZE; | 1128 | pkt->bio->bi_iter.bi_size = pkt->frames * CD_FRAMESIZE; |
| @@ -799,7 +799,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 798 unmodified lines | |||
| 799 | return NULL; | 799 | return NULL; |
| 800 | 800 | ||
| 801 | *part = 0; | 801 | *part = 0; |
| 802 | return get_disk(swd->unit[drive].disk); | 802 | return get_disk_and_module(swd->unit[drive].disk); |
| 803 | } | 803 | } |
| 804 | 804 | ||
| 805 | static int swim_add_floppy(struct swim_priv *swd, enum drive_location location) | 805 | static int swim_add_floppy(struct swim_priv *swd, enum drive_location location) |
| @@ -262,6 +262,7 @@ static DEFINE_SPINLOCK(minor_lock); · 261 unmodified lines | |||
| 262 | 262 | ||
| 263 | static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo); | 263 | static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo); |
| 264 | static void blkfront_gather_backend_features(struct blkfront_info *info); | 264 | static void blkfront_gather_backend_features(struct blkfront_info *info); |
| 265 | static int negotiate_mq(struct blkfront_info *info); | ||
| 265 | 266 | ||
| 266 | static int get_id_from_freelist(struct blkfront_ring_info *rinfo) | 267 | static int get_id_from_freelist(struct blkfront_ring_info *rinfo) |
| 267 | { | 268 | { |
| @@ -1774,11 +1775,18 @@ static int talk_to_blkback(struct xenbus_device *dev, · 1506 unmodified lines | |||
| 1774 | unsigned int i, max_page_order; | 1775 | unsigned int i, max_page_order; |
| 1775 | unsigned int ring_page_order; | 1776 | unsigned int ring_page_order; |
| 1776 | 1777 | ||
| 1778 | if (!info) | ||
| 1779 | return -ENODEV; | ||
| 1780 | |||
| 1777 | max_page_order = xenbus_read_unsigned(info->xbdev->otherend, | 1781 | max_page_order = xenbus_read_unsigned(info->xbdev->otherend, |
| 1778 | "max-ring-page-order", 0); | 1782 | "max-ring-page-order", 0); |
| 1779 | ring_page_order = min(xen_blkif_max_ring_order, max_page_order); | 1783 | ring_page_order = min(xen_blkif_max_ring_order, max_page_order); |
| 1780 | info->nr_ring_pages = 1 << ring_page_order; | 1784 | info->nr_ring_pages = 1 << ring_page_order; |
| 1781 | 1785 | ||
| 1786 | err = negotiate_mq(info); | ||
| 1787 | if (err) | ||
| 1788 | goto destroy_blkring; | ||
| 1789 | |||
| 1782 | for (i = 0; i < info->nr_rings; i++) { | 1790 | for (i = 0; i < info->nr_rings; i++) { |
| 1783 | struct blkfront_ring_info *rinfo = &info->rinfo[i]; | 1791 | struct blkfront_ring_info *rinfo = &info->rinfo[i]; |
| 1784 | 1792 | ||
| @@ -1978,11 +1986,6 @@ static int blkfront_probe(struct xenbus_device *dev, · 193 unmodified lines | |||
| 1978 | } | 1986 | } |
| 1979 | 1987 | ||
| 1980 | info->xbdev = dev; | 1988 | info->xbdev = dev; |
| 1981 | err = negotiate_mq(info); | ||
| 1982 | if (err) { | ||
| 1983 | kfree(info); | ||
| 1984 | return err; | ||
| 1985 | } | ||
| 1986 | 1989 | ||
| 1987 | mutex_init(&info->mutex); | 1990 | mutex_init(&info->mutex); |
| 1988 | info->vdevice = vdevice; | 1991 | info->vdevice = vdevice; |
| @@ -2099,10 +2102,6 @@ static int blkfront_resume(struct xenbus_device *dev) · 110 unmodified lines | |||
| 2099 | 2102 | ||
| 2100 | blkif_free(info, info->connected == BLKIF_STATE_CONNECTED); | 2103 | blkif_free(info, info->connected == BLKIF_STATE_CONNECTED); |
| 2101 | 2104 | ||
| 2102 | err = negotiate_mq(info); | ||
| 2103 | if (err) | ||
| 2104 | return err; | ||
| 2105 | |||
| 2106 | err = talk_to_blkback(dev, info); | 2105 | err = talk_to_blkback(dev, info); |
| 2107 | if (!err) | 2106 | if (!err) |
| 2108 | blk_mq_update_nr_hw_queues(&info->tag_set, info->nr_rings); | 2107 | blk_mq_update_nr_hw_queues(&info->tag_set, info->nr_rings); |
| @@ -332,7 +332,7 @@ static const struct block_device_operations z2_fops = · 331 unmodified lines | |||
| 332 | static struct kobject *z2_find(dev_t dev, int *part, void *data) | 332 | static struct kobject *z2_find(dev_t dev, int *part, void *data) |
| 333 | { | 333 | { |
| 334 | *part = 0; | 334 | *part = 0; |
| 335 | return get_disk(z2ram_gendisk); | 335 | return get_disk_and_module(z2ram_gendisk); |
| 336 | } | 336 | } |
| 337 | 337 | ||
| 338 | static struct request_queue *z2_queue; | 338 | static struct request_queue *z2_queue; |
| @@ -21,6 +21,7 @@ · 20 unmodified lines | |||
| 21 | * | 21 | * |
| 22 | */ | 22 | */ |
| 23 | 23 | ||
| 24 | #include <linux/dmi.h> | ||
| 24 | #include <linux/module.h> | 25 | #include <linux/module.h> |
| 25 | #include <linux/usb.h> | 26 | #include <linux/usb.h> |
| 26 | #include <linux/usb/quirks.h> | 27 | #include <linux/usb/quirks.h> |
| @@ -230,7 +231,6 @@ static const struct usb_device_id blacklist_table[] = { · 203 unmodified lines | |||
| 230 | { USB_DEVICE(0x0930, 0x0227), .driver_info = BTUSB_ATH3012 }, | 231 | { USB_DEVICE(0x0930, 0x0227), .driver_info = BTUSB_ATH3012 }, |
| 231 | { USB_DEVICE(0x0b05, 0x17d0), .driver_info = BTUSB_ATH3012 }, | 232 | { USB_DEVICE(0x0b05, 0x17d0), .driver_info = BTUSB_ATH3012 }, |
| 232 | { USB_DEVICE(0x0cf3, 0x0036), .driver_info = BTUSB_ATH3012 }, | 233 | { USB_DEVICE(0x0cf3, 0x0036), .driver_info = BTUSB_ATH3012 }, |
| 233 | { USB_DEVICE(0x0cf3, 0x3004), .driver_info = BTUSB_ATH3012 }, | ||
| 234 | { USB_DEVICE(0x0cf3, 0x3008), .driver_info = BTUSB_ATH3012 }, | 234 | { USB_DEVICE(0x0cf3, 0x3008), .driver_info = BTUSB_ATH3012 }, |
| 235 | { USB_DEVICE(0x0cf3, 0x311d), .driver_info = BTUSB_ATH3012 }, | 235 | { USB_DEVICE(0x0cf3, 0x311d), .driver_info = BTUSB_ATH3012 }, |
| 236 | { USB_DEVICE(0x0cf3, 0x311e), .driver_info = BTUSB_ATH3012 }, | 236 | { USB_DEVICE(0x0cf3, 0x311e), .driver_info = BTUSB_ATH3012 }, |
| @@ -263,6 +263,7 @@ static const struct usb_device_id blacklist_table[] = { · 26 unmodified lines | |||
| 263 | { USB_DEVICE(0x0489, 0xe03c), .driver_info = BTUSB_ATH3012 }, | 263 | { USB_DEVICE(0x0489, 0xe03c), .driver_info = BTUSB_ATH3012 }, |
| 264 | 264 | ||
| 265 | /* QCA ROME chipset */ | 265 | /* QCA ROME chipset */ |
| 266 | { USB_DEVICE(0x0cf3, 0x3004), .driver_info = BTUSB_QCA_ROME }, | ||
| 266 | { USB_DEVICE(0x0cf3, 0xe007), .driver_info = BTUSB_QCA_ROME }, | 267 | { USB_DEVICE(0x0cf3, 0xe007), .driver_info = BTUSB_QCA_ROME }, |
| 267 | { USB_DEVICE(0x0cf3, 0xe009), .driver_info = BTUSB_QCA_ROME }, | 268 | { USB_DEVICE(0x0cf3, 0xe009), .driver_info = BTUSB_QCA_ROME }, |
| 268 | { USB_DEVICE(0x0cf3, 0xe010), .driver_info = BTUSB_QCA_ROME }, | 269 | { USB_DEVICE(0x0cf3, 0xe010), .driver_info = BTUSB_QCA_ROME }, |
| @@ -379,6 +380,21 @@ static const struct usb_device_id blacklist_table[] = { · 110 unmodified lines | |||
| 379 | { } /* Terminating entry */ | 380 | { } /* Terminating entry */ |
| 380 | }; | 381 | }; |
| 381 | 382 | ||
| 383 | /* The Bluetooth USB module build into some devices needs to be reset on resume, | ||
| 384 | * this is a problem with the platform (likely shutting off all power) not with | ||
| 385 | * the module itself. So we use a DMI list to match known broken platforms. | ||
| 386 | */ | ||
| 387 | static const struct dmi_system_id btusb_needs_reset_resume_table[] = { | ||
| 388 | { | ||
| 389 | /* Dell OptiPlex 3060 (QCA ROME device 0cf3:e007) */ | ||
| 390 | .matches = { | ||
| 391 | DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."), | ||
| 392 | DMI_MATCH(DMI_PRODUCT_NAME, "OptiPlex 3060"), | ||
| 393 | }, | ||
| 394 | }, | ||
| 395 | {} | ||
| 396 | }; | ||
| 397 | |||
| 382 | #define BTUSB_MAX_ISOC_FRAMES 10 | 398 | #define BTUSB_MAX_ISOC_FRAMES 10 |
| 383 | 399 | ||
| 384 | #define BTUSB_INTR_RUNNING 0 | 400 | #define BTUSB_INTR_RUNNING 0 |
| @@ -2945,6 +2961,9 @@ static int btusb_probe(struct usb_interface *intf, · 2560 unmodified lines | |||
| 2945 | hdev->send = btusb_send_frame; | 2961 | hdev->send = btusb_send_frame; |
| 2946 | hdev->notify = btusb_notify; | 2962 | hdev->notify = btusb_notify; |
| 2947 | 2963 | ||
| 2964 | if (dmi_check_system(btusb_needs_reset_resume_table)) | ||
| 2965 | interface_to_usbdev(intf)->quirks |= USB_QUIRK_RESET_RESUME; | ||
| 2966 | |||
| 2948 | #ifdef CONFIG_PM | 2967 | #ifdef CONFIG_PM |
| 2949 | err = btusb_config_oob_wake(hdev); | 2968 | err = btusb_config_oob_wake(hdev); |
| 2950 | if (err) | 2969 | if (err) |
| @@ -3031,12 +3050,6 @@ static int btusb_probe(struct usb_interface *intf, · 80 unmodified lines | |||
| 3031 | if (id->driver_info & BTUSB_QCA_ROME) { | 3050 | if (id->driver_info & BTUSB_QCA_ROME) { |
| 3032 | data->setup_on_usb = btusb_setup_qca; | 3051 | data->setup_on_usb = btusb_setup_qca; |
| 3033 | hdev->set_bdaddr = btusb_set_bdaddr_ath3012; | 3052 | hdev->set_bdaddr = btusb_set_bdaddr_ath3012; |
| 3034 | |||
| 3035 | /* QCA Rome devices lose their updated firmware over suspend, | ||
| 3036 | * but the USB hub doesn't notice any status change. | ||
| 3037 | * explicitly request a device reset on resume. | ||
| 3038 | */ | ||
| 3039 | interface_to_usbdev(intf)->quirks |= USB_QUIRK_RESET_RESUME; | ||
| 3040 | } | 3053 | } |
| 3041 | 3054 | ||
| 3042 | #ifdef CONFIG_BT_HCIBTUSB_RTL | 3055 | #ifdef CONFIG_BT_HCIBTUSB_RTL |
| @@ -244,7 +244,9 @@ static irqreturn_t bcm_host_wake(int irq, void *data) · 243 unmodified lines | |||
| 244 | 244 | ||
| 245 | bt_dev_dbg(bdev, "Host wake IRQ"); | 245 | bt_dev_dbg(bdev, "Host wake IRQ"); |
| 246 | 246 | ||
| 247 | pm_request_resume(bdev->dev); | 247 | pm_runtime_get(bdev->dev); |
| 248 | pm_runtime_mark_last_busy(bdev->dev); | ||
| 249 | pm_runtime_put_autosuspend(bdev->dev); | ||
| 248 | 250 | ||
| 249 | return IRQ_HANDLED; | 251 | return IRQ_HANDLED; |
| 250 | } | 252 | } |
| @@ -301,7 +303,7 @@ static const struct bcm_set_sleep_mode default_sleep_params = { · 50 unmodified lines | |||
| 301 | .usb_auto_sleep = 0, | 303 | .usb_auto_sleep = 0, |
| 302 | .usb_resume_timeout = 0, | 304 | .usb_resume_timeout = 0, |
| 303 | .break_to_host = 0, | 305 | .break_to_host = 0, |
| 304 | .pulsed_host_wake = 0, | 306 | .pulsed_host_wake = 1, |
| 305 | }; | 307 | }; |
| 306 | 308 | ||
| 307 | static int bcm_setup_sleep(struct hci_uart *hu) | 309 | static int bcm_setup_sleep(struct hci_uart *hu) |
| @@ -586,8 +588,11 @@ static int bcm_recv(struct hci_uart *hu, const void *data, int count) · 278 unmodified lines | |||
| 586 | } else if (!bcm->rx_skb) { | 588 | } else if (!bcm->rx_skb) { |
| 587 | /* Delay auto-suspend when receiving completed packet */ | 589 | /* Delay auto-suspend when receiving completed packet */ |
| 588 | mutex_lock(&bcm_device_lock); | 590 | mutex_lock(&bcm_device_lock); |
| 589 | if (bcm->dev && bcm_device_exists(bcm->dev)) | 591 | if (bcm->dev && bcm_device_exists(bcm->dev)) { |
| 590 | pm_request_resume(bcm->dev->dev); | 592 | pm_runtime_get(bcm->dev->dev); |
| 593 | pm_runtime_mark_last_busy(bcm->dev->dev); | ||
| 594 | pm_runtime_put_autosuspend(bcm->dev->dev); | ||
| 595 | } | ||
| 591 | mutex_unlock(&bcm_device_lock); | 596 | mutex_unlock(&bcm_device_lock); |
| 592 | } | 597 | } |
| 593 | 598 | ||
| @@ -922,12 +927,13 @@ static int bcm_get_resources(struct bcm_device *dev) · 328 unmodified lines | |||
| 922 | 927 | ||
| 923 | dev->clk = devm_clk_get(dev->dev, NULL); | 928 | dev->clk = devm_clk_get(dev->dev, NULL); |
| 924 | 929 | ||
| 925 | dev->device_wakeup = devm_gpiod_get(dev->dev, "device-wakeup", | 930 | dev->device_wakeup = devm_gpiod_get_optional(dev->dev, "device-wakeup", |
| 926 | GPIOD_OUT_LOW); | 931 | GPIOD_OUT_LOW); |
| 927 | if (IS_ERR(dev->device_wakeup)) | 932 | if (IS_ERR(dev->device_wakeup)) |
| 928 | return PTR_ERR(dev->device_wakeup); | 933 | return PTR_ERR(dev->device_wakeup); |
| 929 | 934 | ||
| 930 | dev->shutdown = devm_gpiod_get(dev->dev, "shutdown", GPIOD_OUT_LOW); | 935 | dev->shutdown = devm_gpiod_get_optional(dev->dev, "shutdown", |
| 936 | GPIOD_OUT_LOW); | ||
| 931 | if (IS_ERR(dev->shutdown)) | 937 | if (IS_ERR(dev->shutdown)) |
| 932 | return PTR_ERR(dev->shutdown); | 938 | return PTR_ERR(dev->shutdown); |
| 933 | 939 | ||
| @@ -630,7 +630,7 @@ static int sysc_init_dts_quirks(struct sysc *ddata) · 629 unmodified lines | |||
| 630 | for (i = 0; i < ARRAY_SIZE(sysc_dts_quirks); i++) { | 630 | for (i = 0; i < ARRAY_SIZE(sysc_dts_quirks); i++) { |
| 631 | prop = of_get_property(np, sysc_dts_quirks[i].name, &len); | 631 | prop = of_get_property(np, sysc_dts_quirks[i].name, &len); |
| 632 | if (!prop) | 632 | if (!prop) |
| 633 | break; | 633 | continue; |
| 634 | 634 | ||
| 635 | ddata->cfg.quirks |= sysc_dts_quirks[i].mask; | 635 | ddata->cfg.quirks |= sysc_dts_quirks[i].mask; |
| 636 | } | 636 | } |
| @@ -162,7 +162,7 @@ static int via_rng_init(struct hwrng *rng) · 161 unmodified lines | |||
| 162 | /* Enable secondary noise source on CPUs where it is present. */ | 162 | /* Enable secondary noise source on CPUs where it is present. */ |
| 163 | 163 | ||
| 164 | /* Nehemiah stepping 8 and higher */ | 164 | /* Nehemiah stepping 8 and higher */ |
| 165 | if ((c->x86_model == 9) && (c->x86_mask > 7)) | 165 | if ((c->x86_model == 9) && (c->x86_stepping > 7)) |
| 166 | lo |= VIA_NOISESRC2; | 166 | lo |= VIA_NOISESRC2; |
| 167 | 167 | ||
| 168 | /* Esther */ | 168 | /* Esther */ |
| @@ -457,7 +457,7 @@ static int st33zp24_recv(struct tpm_chip *chip, unsigned char *buf, · 456 unmodified lines | |||
| 457 | size_t count) | 457 | size_t count) |
| 458 | { | 458 | { |
| 459 | int size = 0; | 459 | int size = 0; |
| 460 | int expected; | 460 | u32 expected; |
| 461 | 461 | ||
| 462 | if (!chip) | 462 | if (!chip) |
| 463 | return -EBUSY; | 463 | return -EBUSY; |
| @@ -474,7 +474,7 @@ static int st33zp24_recv(struct tpm_chip *chip, unsigned char *buf, · 10 unmodified lines | |||
| 474 | } | 474 | } |
| 475 | 475 | ||
| 476 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); | 476 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); |
| 477 | if (expected > count) { | 477 | if (expected > count || expected < TPM_HEADER_SIZE) { |
| 478 | size = -EIO; | 478 | size = -EIO; |
| 479 | goto out; | 479 | goto out; |
| 480 | } | 480 | } |
| @@ -1190,6 +1190,10 @@ int tpm_get_random(struct tpm_chip *chip, u8 *out, size_t max) · 1189 unmodified lines | |||
| 1190 | break; | 1190 | break; |
| 1191 | 1191 | ||
| 1192 | recd = be32_to_cpu(tpm_cmd.params.getrandom_out.rng_data_len); | 1192 | recd = be32_to_cpu(tpm_cmd.params.getrandom_out.rng_data_len); |
| 1193 | if (recd > num_bytes) { | ||
| 1194 | total = -EFAULT; | ||
| 1195 | break; | ||
| 1196 | } | ||
| 1193 | 1197 | ||
| 1194 | rlength = be32_to_cpu(tpm_cmd.header.out.length); | 1198 | rlength = be32_to_cpu(tpm_cmd.header.out.length); |
| 1195 | if (rlength < offsetof(struct tpm_getrandom_out, rng_data) + | 1199 | if (rlength < offsetof(struct tpm_getrandom_out, rng_data) + |
| @@ -683,6 +683,10 @@ static int tpm2_unseal_cmd(struct tpm_chip *chip, · 682 unmodified lines | |||
| 683 | if (!rc) { | 683 | if (!rc) { |
| 684 | data_len = be16_to_cpup( | 684 | data_len = be16_to_cpup( |
| 685 | (__be16 *) &buf.data[TPM_HEADER_SIZE + 4]); | 685 | (__be16 *) &buf.data[TPM_HEADER_SIZE + 4]); |
| 686 | if (data_len < MIN_KEY_SIZE || data_len > MAX_KEY_SIZE + 1) { | ||
| 687 | rc = -EFAULT; | ||
| 688 | goto out; | ||
| 689 | } | ||
| 686 | 690 | ||
| 687 | rlength = be32_to_cpu(((struct tpm2_cmd *)&buf) | 691 | rlength = be32_to_cpu(((struct tpm2_cmd *)&buf) |
| 688 | ->header.out.length); | 692 | ->header.out.length); |
| @@ -473,7 +473,8 @@ static int recv_data(struct tpm_chip *chip, u8 *buf, size_t count) · 472 unmodified lines | |||
| 473 | static int tpm_tis_i2c_recv(struct tpm_chip *chip, u8 *buf, size_t count) | 473 | static int tpm_tis_i2c_recv(struct tpm_chip *chip, u8 *buf, size_t count) |
| 474 | { | 474 | { |
| 475 | int size = 0; | 475 | int size = 0; |
| 476 | int expected, status; | 476 | int status; |
| 477 | u32 expected; | ||
| 477 | 478 | ||
| 478 | if (count < TPM_HEADER_SIZE) { | 479 | if (count < TPM_HEADER_SIZE) { |
| 479 | size = -EIO; | 480 | size = -EIO; |
| @@ -488,7 +489,7 @@ static int tpm_tis_i2c_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 8 unmodified lines | |||
| 488 | } | 489 | } |
| 489 | 490 | ||
| 490 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); | 491 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); |
| 491 | if ((size_t) expected > count) { | 492 | if (((size_t) expected > count) || (expected < TPM_HEADER_SIZE)) { |
| 492 | size = -EIO; | 493 | size = -EIO; |
| 493 | goto out; | 494 | goto out; |
| 494 | } | 495 | } |
| @@ -281,7 +281,11 @@ static int i2c_nuvoton_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 280 unmodified lines | |||
| 281 | struct device *dev = chip->dev.parent; | 281 | struct device *dev = chip->dev.parent; |
| 282 | struct i2c_client *client = to_i2c_client(dev); | 282 | struct i2c_client *client = to_i2c_client(dev); |
| 283 | s32 rc; | 283 | s32 rc; |
| 284 | int expected, status, burst_count, retries, size = 0; | 284 | int status; |
| 285 | int burst_count; | ||
| 286 | int retries; | ||
| 287 | int size = 0; | ||
| 288 | u32 expected; | ||
| 285 | 289 | ||
| 286 | if (count < TPM_HEADER_SIZE) { | 290 | if (count < TPM_HEADER_SIZE) { |
| 287 | i2c_nuvoton_ready(chip); /* return to idle */ | 291 | i2c_nuvoton_ready(chip); /* return to idle */ |
| @@ -323,7 +327,7 @@ static int i2c_nuvoton_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 35 unmodified lines | |||
| 323 | * to machine native | 327 | * to machine native |
| 324 | */ | 328 | */ |
| 325 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); | 329 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); |
| 326 | if (expected > count) { | 330 | if (expected > count || expected < size) { |
| 327 | dev_err(dev, "%s() expected > count\n", __func__); | 331 | dev_err(dev, "%s() expected > count\n", __func__); |
| 328 | size = -EIO; | 332 | size = -EIO; |
| 329 | continue; | 333 | continue; |
| @@ -270,7 +270,8 @@ static int tpm_tis_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 269 unmodified lines | |||
| 270 | { | 270 | { |
| 271 | struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); | 271 | struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); |
| 272 | int size = 0; | 272 | int size = 0; |
| 273 | int expected, status; | 273 | int status; |
| 274 | u32 expected; | ||
| 274 | 275 | ||
| 275 | if (count < TPM_HEADER_SIZE) { | 276 | if (count < TPM_HEADER_SIZE) { |
| 276 | size = -EIO; | 277 | size = -EIO; |
| @@ -285,7 +286,7 @@ static int tpm_tis_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 8 unmodified lines | |||
| 285 | } | 286 | } |
| 286 | 287 | ||
| 287 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); | 288 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); |
| 288 | if (expected > count) { | 289 | if (expected > count || expected < TPM_HEADER_SIZE) { |
| 289 | size = -EIO; | 290 | size = -EIO; |
| 290 | goto out; | 291 | goto out; |
| 291 | } | 292 | } |
| @@ -449,17 +449,17 @@ struct bcm2835_pll_ana_bits { · 448 unmodified lines | |||
| 449 | static const struct bcm2835_pll_ana_bits bcm2835_ana_default = { | 449 | static const struct bcm2835_pll_ana_bits bcm2835_ana_default = { |
| 450 | .mask0 = 0, | 450 | .mask0 = 0, |
| 451 | .set0 = 0, | 451 | .set0 = 0, |
| 452 | .mask1 = (u32)~(A2W_PLL_KI_MASK | A2W_PLL_KP_MASK), | 452 | .mask1 = A2W_PLL_KI_MASK | A2W_PLL_KP_MASK, |
| 453 | .set1 = (2 << A2W_PLL_KI_SHIFT) | (8 << A2W_PLL_KP_SHIFT), | 453 | .set1 = (2 << A2W_PLL_KI_SHIFT) | (8 << A2W_PLL_KP_SHIFT), |
| 454 | .mask3 = (u32)~A2W_PLL_KA_MASK, | 454 | .mask3 = A2W_PLL_KA_MASK, |
| 455 | .set3 = (2 << A2W_PLL_KA_SHIFT), | 455 | .set3 = (2 << A2W_PLL_KA_SHIFT), |
| 456 | .fb_prediv_mask = BIT(14), | 456 | .fb_prediv_mask = BIT(14), |
| 457 | }; | 457 | }; |
| 458 | 458 | ||
| 459 | static const struct bcm2835_pll_ana_bits bcm2835_ana_pllh = { | 459 | static const struct bcm2835_pll_ana_bits bcm2835_ana_pllh = { |
| 460 | .mask0 = (u32)~(A2W_PLLH_KA_MASK | A2W_PLLH_KI_LOW_MASK), | 460 | .mask0 = A2W_PLLH_KA_MASK | A2W_PLLH_KI_LOW_MASK, |
| 461 | .set0 = (2 << A2W_PLLH_KA_SHIFT) | (2 << A2W_PLLH_KI_LOW_SHIFT), | 461 | .set0 = (2 << A2W_PLLH_KA_SHIFT) | (2 << A2W_PLLH_KI_LOW_SHIFT), |
| 462 | .mask1 = (u32)~(A2W_PLLH_KI_HIGH_MASK | A2W_PLLH_KP_MASK), | 462 | .mask1 = A2W_PLLH_KI_HIGH_MASK | A2W_PLLH_KP_MASK, |
| 463 | .set1 = (6 << A2W_PLLH_KP_SHIFT), | 463 | .set1 = (6 << A2W_PLLH_KP_SHIFT), |
| 464 | .mask3 = 0, | 464 | .mask3 = 0, |
| 465 | .set3 = 0, | 465 | .set3 = 0, |
| @@ -623,8 +623,10 @@ static int bcm2835_pll_on(struct clk_hw *hw) · 157 unmodified lines | |||
| 623 | ~A2W_PLL_CTRL_PWRDN); | 623 | ~A2W_PLL_CTRL_PWRDN); |
| 624 | 624 | ||
| 625 | /* Take the PLL out of reset. */ | 625 | /* Take the PLL out of reset. */ |
| 626 | spin_lock(&cprman->regs_lock); | ||
| 626 | cprman_write(cprman, data->cm_ctrl_reg, | 627 | cprman_write(cprman, data->cm_ctrl_reg, |
| 627 | cprman_read(cprman, data->cm_ctrl_reg) & ~CM_PLL_ANARST); | 628 | cprman_read(cprman, data->cm_ctrl_reg) & ~CM_PLL_ANARST); |
| 629 | spin_unlock(&cprman->regs_lock); | ||
| 628 | 630 | ||
| 629 | /* Wait for the PLL to lock. */ | 631 | /* Wait for the PLL to lock. */ |
| 630 | timeout = ktime_add_ns(ktime_get(), LOCK_TIMEOUT_NS); | 632 | timeout = ktime_add_ns(ktime_get(), LOCK_TIMEOUT_NS); |
| @@ -701,9 +703,11 @@ static int bcm2835_pll_set_rate(struct clk_hw *hw, · 70 unmodified lines | |||
| 701 | } | 703 | } |
| 702 | 704 | ||
| 703 | /* Unmask the reference clock from the oscillator. */ | 705 | /* Unmask the reference clock from the oscillator. */ |
| 706 | spin_lock(&cprman->regs_lock); | ||
| 704 | cprman_write(cprman, A2W_XOSC_CTRL, | 707 | cprman_write(cprman, A2W_XOSC_CTRL, |
| 705 | cprman_read(cprman, A2W_XOSC_CTRL) | | 708 | cprman_read(cprman, A2W_XOSC_CTRL) | |
| 706 | data->reference_enable_mask); | 709 | data->reference_enable_mask); |
| 710 | spin_unlock(&cprman->regs_lock); | ||
| 707 | 711 | ||
| 708 | if (do_ana_setup_first) | 712 | if (do_ana_setup_first) |
| 709 | bcm2835_pll_write_ana(cprman, data->ana_reg_base, ana); | 713 | bcm2835_pll_write_ana(cprman, data->ana_reg_base, ana); |
| @@ -205,6 +205,18 @@ static const struct aspeed_clk_soc_data ast2400_data = { · 204 unmodified lines | |||
| 205 | .calc_pll = aspeed_ast2400_calc_pll, | 205 | .calc_pll = aspeed_ast2400_calc_pll, |
| 206 | }; | 206 | }; |
| 207 | 207 | ||
| 208 | static int aspeed_clk_is_enabled(struct clk_hw *hw) | ||
| 209 | { | ||
| 210 | struct aspeed_clk_gate *gate = to_aspeed_clk_gate(hw); | ||
| 211 | u32 clk = BIT(gate->clock_idx); | ||
| 212 | u32 enval = (gate->flags & CLK_GATE_SET_TO_DISABLE) ? 0 : clk; | ||
| 213 | u32 reg; | ||
| 214 | |||
| 215 | regmap_read(gate->map, ASPEED_CLK_STOP_CTRL, ®); | ||
| 216 | |||
| 217 | return ((reg & clk) == enval) ? 1 : 0; | ||
| 218 | } | ||
| 219 | |||
| 208 | static int aspeed_clk_enable(struct clk_hw *hw) | 220 | static int aspeed_clk_enable(struct clk_hw *hw) |
| 209 | { | 221 | { |
| 210 | struct aspeed_clk_gate *gate = to_aspeed_clk_gate(hw); | 222 | struct aspeed_clk_gate *gate = to_aspeed_clk_gate(hw); |
| @@ -215,6 +227,11 @@ static int aspeed_clk_enable(struct clk_hw *hw) · 4 unmodified lines | |||
| 215 | 227 | ||
| 216 | spin_lock_irqsave(gate->lock, flags); | 228 | spin_lock_irqsave(gate->lock, flags); |
| 217 | 229 | ||
| 230 | if (aspeed_clk_is_enabled(hw)) { | ||
| 231 | spin_unlock_irqrestore(gate->lock, flags); | ||
| 232 | return 0; | ||
| 233 | } | ||
| 234 | |||
| 218 | if (gate->reset_idx >= 0) { | 235 | if (gate->reset_idx >= 0) { |
| 219 | /* Put IP in reset */ | 236 | /* Put IP in reset */ |
| 220 | regmap_update_bits(gate->map, ASPEED_RESET_CTRL, rst, rst); | 237 | regmap_update_bits(gate->map, ASPEED_RESET_CTRL, rst, rst); |
| @@ -255,17 +272,6 @@ static void aspeed_clk_disable(struct clk_hw *hw) · 34 unmodified lines | |||
| 255 | spin_unlock_irqrestore(gate->lock, flags); | 272 | spin_unlock_irqrestore(gate->lock, flags); |
| 256 | } | 273 | } |
| 257 | 274 | ||
| 258 | static int aspeed_clk_is_enabled(struct clk_hw *hw) | ||
| 259 | { | ||
| 260 | struct aspeed_clk_gate *gate = to_aspeed_clk_gate(hw); | ||
| 261 | u32 clk = BIT(gate->clock_idx); | ||
| 262 | u32 reg; | ||
| 263 | |||
| 264 | regmap_read(gate->map, ASPEED_CLK_STOP_CTRL, ®); | ||
| 265 | |||
| 266 | return (reg & clk) ? 0 : 1; | ||
| 267 | } | ||
| 268 | |||
| 269 | static const struct clk_ops aspeed_clk_gate_ops = { | 275 | static const struct clk_ops aspeed_clk_gate_ops = { |
| 270 | .enable = aspeed_clk_enable, | 276 | .enable = aspeed_clk_enable, |
| 271 | .disable = aspeed_clk_disable, | 277 | .disable = aspeed_clk_disable, |
| @@ -1125,8 +1125,10 @@ static int clk_core_round_rate_nolock(struct clk_core *core, · 1124 unmodified lines | |||
| 1125 | { | 1125 | { |
| 1126 | lockdep_assert_held(&prepare_lock); | 1126 | lockdep_assert_held(&prepare_lock); |
| 1127 | 1127 | ||
| 1128 | if (!core) | 1128 | if (!core) { |
| 1129 | req->rate = 0; | ||
| 1129 | return 0; | 1130 | return 0; |
| 1131 | } | ||
| 1130 | 1132 | ||
| 1131 | clk_core_init_rate_req(core, req); | 1133 | clk_core_init_rate_req(core, req); |
| 1132 | 1134 | ||
| @@ -2309,8 +2311,11 @@ static int clk_core_set_phase_nolock(struct clk_core *core, int degrees) · 1176 unmodified lines | |||
| 2309 | 2311 | ||
| 2310 | trace_clk_set_phase(core, degrees); | 2312 | trace_clk_set_phase(core, degrees); |
| 2311 | 2313 | ||
| 2312 | if (core->ops->set_phase) | 2314 | if (core->ops->set_phase) { |
| 2313 | ret = core->ops->set_phase(core->hw, degrees); | 2315 | ret = core->ops->set_phase(core->hw, degrees); |
| 2316 | if (!ret) | ||
| 2317 | core->phase = degrees; | ||
| 2318 | } | ||
| 2314 | 2319 | ||
| 2315 | trace_clk_set_phase_complete(core, degrees); | 2320 | trace_clk_set_phase_complete(core, degrees); |
| 2316 | 2321 | ||
| @@ -2967,23 +2972,38 @@ static int __clk_core_init(struct clk_core *core) · 650 unmodified lines | |||
| 2967 | rate = 0; | 2972 | rate = 0; |
| 2968 | core->rate = core->req_rate = rate; | 2973 | core->rate = core->req_rate = rate; |
| 2969 | 2974 | ||
| 2975 | /* | ||
| 2976 | * Enable CLK_IS_CRITICAL clocks so newly added critical clocks | ||
| 2977 | * don't get accidentally disabled when walking the orphan tree and | ||
| 2978 | * reparenting clocks | ||
| 2979 | */ | ||
| 2980 | if (core->flags & CLK_IS_CRITICAL) { | ||
| 2981 | unsigned long flags; | ||
| 2982 | |||
| 2983 | clk_core_prepare(core); | ||
| 2984 | |||
| 2985 | flags = clk_enable_lock(); | ||
| 2986 | clk_core_enable(core); | ||
| 2987 | clk_enable_unlock(flags); | ||
| 2988 | } | ||
| 2989 | |||
| 2970 | /* | 2990 | /* |
| 2971 | * walk the list of orphan clocks and reparent any that newly finds a | 2991 | * walk the list of orphan clocks and reparent any that newly finds a |
| 2972 | * parent. | 2992 | * parent. |
| 2973 | */ | 2993 | */ |
| 2974 | hlist_for_each_entry_safe(orphan, tmp2, &clk_orphan_list, child_node) { | 2994 | hlist_for_each_entry_safe(orphan, tmp2, &clk_orphan_list, child_node) { |
| 2975 | struct clk_core *parent = __clk_init_parent(orphan); | 2995 | struct clk_core *parent = __clk_init_parent(orphan); |
| 2976 | unsigned long flags; | ||
| 2977 | 2996 | ||
| 2978 | /* | 2997 | /* |
| 2979 | * we could call __clk_set_parent, but that would result in a | 2998 | * We need to use __clk_set_parent_before() and _after() to |
| 2980 | * redundant call to the .set_rate op, if it exists | 2999 | * to properly migrate any prepare/enable count of the orphan |
| 3000 | * clock. This is important for CLK_IS_CRITICAL clocks, which | ||
| 3001 | * are enabled during init but might not have a parent yet. | ||
| 2981 | */ | 3002 | */ |
| 2982 | if (parent) { | 3003 | if (parent) { |
| 2983 | /* update the clk tree topology */ | 3004 | /* update the clk tree topology */ |
| 2984 | flags = clk_enable_lock(); | 3005 | __clk_set_parent_before(orphan, parent); |
| 2985 | clk_reparent(orphan, parent); | 3006 | __clk_set_parent_after(orphan, parent, NULL); |
| 2986 | clk_enable_unlock(flags); | ||
| 2987 | __clk_recalc_accuracies(orphan); | 3007 | __clk_recalc_accuracies(orphan); |
| 2988 | __clk_recalc_rates(orphan, 0); | 3008 | __clk_recalc_rates(orphan, 0); |
| 2989 | } | 3009 | } |
| @@ -3000,16 +3020,6 @@ static int __clk_core_init(struct clk_core *core) · 10 unmodified lines | |||
| 3000 | if (core->ops->init) | 3020 | if (core->ops->init) |
| 3001 | core->ops->init(core->hw); | 3021 | core->ops->init(core->hw); |
| 3002 | 3022 | ||
| 3003 | if (core->flags & CLK_IS_CRITICAL) { | ||
| 3004 | unsigned long flags; | ||
| 3005 | |||
| 3006 | clk_core_prepare(core); | ||
| 3007 | |||
| 3008 | flags = clk_enable_lock(); | ||
| 3009 | clk_core_enable(core); | ||
| 3010 | clk_enable_unlock(flags); | ||
| 3011 | } | ||
| 3012 | |||
| 3013 | kref_init(&core->ref); | 3023 | kref_init(&core->ref); |
| 3014 | out: | 3024 | out: |
| 3015 | clk_pm_runtime_put(core); | 3025 | clk_pm_runtime_put(core); |
| @@ -149,6 +149,8 @@ static int hi3660_stub_clk_probe(struct platform_device *pdev) · 148 unmodified lines | |||
| 149 | return PTR_ERR(stub_clk_chan.mbox); | 149 | return PTR_ERR(stub_clk_chan.mbox); |
| 150 | 150 | ||
| 151 | res = platform_get_resource(pdev, IORESOURCE_MEM, 0); | 151 | res = platform_get_resource(pdev, IORESOURCE_MEM, 0); |
| 152 | if (!res) | ||
| 153 | return -EINVAL; | ||
| 152 | freq_reg = devm_ioremap(dev, res->start, resource_size(res)); | 154 | freq_reg = devm_ioremap(dev, res->start, resource_size(res)); |
| 153 | if (!freq_reg) | 155 | if (!freq_reg) |
| 154 | return -ENOMEM; | 156 | return -ENOMEM; |
| @@ -131,7 +131,17 @@ static const char *ieee1588_sels[] = { "pll3_sw", "pll4_sw", "dummy" /* usbphy2_ · 130 unmodified lines | |||
| 131 | static struct clk *clk[IMX5_CLK_END]; | 131 | static struct clk *clk[IMX5_CLK_END]; |
| 132 | static struct clk_onecell_data clk_data; | 132 | static struct clk_onecell_data clk_data; |
| 133 | 133 | ||
| 134 | static struct clk ** const uart_clks[] __initconst = { | 134 | static struct clk ** const uart_clks_mx51[] __initconst = { |
| 135 | &clk[IMX5_CLK_UART1_IPG_GATE], | ||
| 136 | &clk[IMX5_CLK_UART1_PER_GATE], | ||
| 137 | &clk[IMX5_CLK_UART2_IPG_GATE], | ||
| 138 | &clk[IMX5_CLK_UART2_PER_GATE], | ||
| 139 | &clk[IMX5_CLK_UART3_IPG_GATE], | ||
| 140 | &clk[IMX5_CLK_UART3_PER_GATE], | ||
| 141 | NULL | ||
| 142 | }; | ||
| 143 | |||
| 144 | static struct clk ** const uart_clks_mx50_mx53[] __initconst = { | ||
| 135 | &clk[IMX5_CLK_UART1_IPG_GATE], | 145 | &clk[IMX5_CLK_UART1_IPG_GATE], |
| 136 | &clk[IMX5_CLK_UART1_PER_GATE], | 146 | &clk[IMX5_CLK_UART1_PER_GATE], |
| 137 | &clk[IMX5_CLK_UART2_IPG_GATE], | 147 | &clk[IMX5_CLK_UART2_IPG_GATE], |
| @@ -321,8 +331,6 @@ static void __init mx5_clocks_common_init(void __iomem *ccm_base) · 183 unmodified lines | |||
| 321 | clk_prepare_enable(clk[IMX5_CLK_TMAX1]); | 331 | clk_prepare_enable(clk[IMX5_CLK_TMAX1]); |
| 322 | clk_prepare_enable(clk[IMX5_CLK_TMAX2]); /* esdhc2, fec */ | 332 | clk_prepare_enable(clk[IMX5_CLK_TMAX2]); /* esdhc2, fec */ |
| 323 | clk_prepare_enable(clk[IMX5_CLK_TMAX3]); /* esdhc1, esdhc4 */ | 333 | clk_prepare_enable(clk[IMX5_CLK_TMAX3]); /* esdhc1, esdhc4 */ |
| 324 | |||
| 325 | imx_register_uart_clocks(uart_clks); | ||
| 326 | } | 334 | } |
| 327 | 335 | ||
| 328 | static void __init mx50_clocks_init(struct device_node *np) | 336 | static void __init mx50_clocks_init(struct device_node *np) |
| @@ -388,6 +396,8 @@ static void __init mx50_clocks_init(struct device_node *np) · 59 unmodified lines | |||
| 388 | 396 | ||
| 389 | r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000); | 397 | r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000); |
| 390 | clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r); | 398 | clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r); |
| 399 | |||
| 400 | imx_register_uart_clocks(uart_clks_mx50_mx53); | ||
| 391 | } | 401 | } |
| 392 | CLK_OF_DECLARE(imx50_ccm, "fsl,imx50-ccm", mx50_clocks_init); | 402 | CLK_OF_DECLARE(imx50_ccm, "fsl,imx50-ccm", mx50_clocks_init); |
| 393 | 403 | ||
| @@ -477,6 +487,8 @@ static void __init mx51_clocks_init(struct device_node *np) · 83 unmodified lines | |||
| 477 | val = readl(MXC_CCM_CLPCR); | 487 | val = readl(MXC_CCM_CLPCR); |
| 478 | val |= 1 << 23; | 488 | val |= 1 << 23; |
| 479 | writel(val, MXC_CCM_CLPCR); | 489 | writel(val, MXC_CCM_CLPCR); |
| 490 | |||
| 491 | imx_register_uart_clocks(uart_clks_mx51); | ||
| 480 | } | 492 | } |
| 481 | CLK_OF_DECLARE(imx51_ccm, "fsl,imx51-ccm", mx51_clocks_init); | 493 | CLK_OF_DECLARE(imx51_ccm, "fsl,imx51-ccm", mx51_clocks_init); |
| 482 | 494 | ||
| @@ -606,5 +618,7 @@ static void __init mx53_clocks_init(struct device_node *np) · 123 unmodified lines | |||
| 606 | 618 | ||
| 607 | r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000); | 619 | r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000); |
| 608 | clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r); | 620 | clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r); |
| 621 | |||
| 622 | imx_register_uart_clocks(uart_clks_mx50_mx53); | ||
| 609 | } | 623 | } |
| 610 | CLK_OF_DECLARE(imx53_ccm, "fsl,imx53-ccm", mx53_clocks_init); | 624 | CLK_OF_DECLARE(imx53_ccm, "fsl,imx53-ccm", mx53_clocks_init); |
| @@ -49,11 +49,10 @@ static int qcom_apcs_msm8916_clk_probe(struct platform_device *pdev) · 48 unmodified lines | |||
| 49 | struct clk_regmap_mux_div *a53cc; | 49 | struct clk_regmap_mux_div *a53cc; |
| 50 | struct regmap *regmap; | 50 | struct regmap *regmap; |
| 51 | struct clk_init_data init = { }; | 51 | struct clk_init_data init = { }; |
| 52 | int ret; | 52 | int ret = -ENODEV; |
| 53 | 53 | ||
| 54 | regmap = dev_get_regmap(parent, NULL); | 54 | regmap = dev_get_regmap(parent, NULL); |
| 55 | if (IS_ERR(regmap)) { | 55 | if (!regmap) { |
| 56 | ret = PTR_ERR(regmap); | ||
| 57 | dev_err(dev, "failed to get regmap: %d\n", ret); | 56 | dev_err(dev, "failed to get regmap: %d\n", ret); |
| 58 | return ret; | 57 | return ret; |
| 59 | } | 58 | } |
| @@ -762,7 +762,7 @@ static struct ccu_mp out_a_clk = { · 761 unmodified lines | |||
| 762 | .features = CCU_FEATURE_FIXED_PREDIV, | 762 | .features = CCU_FEATURE_FIXED_PREDIV, |
| 763 | .hw.init = CLK_HW_INIT_PARENTS("out-a", | 763 | .hw.init = CLK_HW_INIT_PARENTS("out-a", |
| 764 | clk_out_parents, | 764 | clk_out_parents, |
| 765 | &ccu_div_ops, | 765 | &ccu_mp_ops, |
| 766 | 0), | 766 | 0), |
| 767 | }, | 767 | }, |
| 768 | }; | 768 | }; |
| @@ -783,7 +783,7 @@ static struct ccu_mp out_b_clk = { · 14 unmodified lines | |||
| 783 | .features = CCU_FEATURE_FIXED_PREDIV, | 783 | .features = CCU_FEATURE_FIXED_PREDIV, |
| 784 | .hw.init = CLK_HW_INIT_PARENTS("out-b", | 784 | .hw.init = CLK_HW_INIT_PARENTS("out-b", |
| 785 | clk_out_parents, | 785 | clk_out_parents, |
| 786 | &ccu_div_ops, | 786 | &ccu_mp_ops, |
| 787 | 0), | 787 | 0), |
| 788 | }, | 788 | }, |
| 789 | }; | 789 | }; |
| @@ -804,7 +804,7 @@ static struct ccu_mp out_c_clk = { · 14 unmodified lines | |||
| 804 | .features = CCU_FEATURE_FIXED_PREDIV, | 804 | .features = CCU_FEATURE_FIXED_PREDIV, |
| 805 | .hw.init = CLK_HW_INIT_PARENTS("out-c", | 805 | .hw.init = CLK_HW_INIT_PARENTS("out-c", |
| 806 | clk_out_parents, | 806 | clk_out_parents, |
| 807 | &ccu_div_ops, | 807 | &ccu_mp_ops, |
| 808 | 0), | 808 | 0), |
| 809 | }, | 809 | }, |
| 810 | }; | 810 | }; |
| @@ -45,7 +45,7 @@ static const struct omap_clkctrl_bit_data am3_gpio4_bit_data[] __initconst = { · 44 unmodified lines | |||
| 45 | 45 | ||
| 46 | static const struct omap_clkctrl_reg_data am3_l4_per_clkctrl_regs[] __initconst = { | 46 | static const struct omap_clkctrl_reg_data am3_l4_per_clkctrl_regs[] __initconst = { |
| 47 | { AM3_CPGMAC0_CLKCTRL, NULL, CLKF_SW_SUP, "cpsw_125mhz_gclk", "cpsw_125mhz_clkdm" }, | 47 | { AM3_CPGMAC0_CLKCTRL, NULL, CLKF_SW_SUP, "cpsw_125mhz_gclk", "cpsw_125mhz_clkdm" }, |
| 48 | { AM3_LCDC_CLKCTRL, NULL, CLKF_SW_SUP, "lcd_gclk", "lcdc_clkdm" }, | 48 | { AM3_LCDC_CLKCTRL, NULL, CLKF_SW_SUP | CLKF_SET_RATE_PARENT, "lcd_gclk", "lcdc_clkdm" }, |
| 49 | { AM3_USB_OTG_HS_CLKCTRL, NULL, CLKF_SW_SUP, "usbotg_fck", "l3s_clkdm" }, | 49 | { AM3_USB_OTG_HS_CLKCTRL, NULL, CLKF_SW_SUP, "usbotg_fck", "l3s_clkdm" }, |
| 50 | { AM3_TPTC0_CLKCTRL, NULL, CLKF_SW_SUP, "l3_gclk", "l3_clkdm" }, | 50 | { AM3_TPTC0_CLKCTRL, NULL, CLKF_SW_SUP, "l3_gclk", "l3_clkdm" }, |
| 51 | { AM3_EMIF_CLKCTRL, NULL, CLKF_SW_SUP, "dpll_ddr_m2_div2_ck", "l3_clkdm" }, | 51 | { AM3_EMIF_CLKCTRL, NULL, CLKF_SW_SUP, "dpll_ddr_m2_div2_ck", "l3_clkdm" }, |
| @@ -187,7 +187,7 @@ static const struct omap_clkctrl_reg_data am4_l4_per_clkctrl_regs[] __initconst · 186 unmodified lines | |||
| 187 | { AM4_OCP2SCP0_CLKCTRL, NULL, CLKF_SW_SUP, "l4ls_gclk" }, | 187 | { AM4_OCP2SCP0_CLKCTRL, NULL, CLKF_SW_SUP, "l4ls_gclk" }, |
| 188 | { AM4_OCP2SCP1_CLKCTRL, NULL, CLKF_SW_SUP, "l4ls_gclk" }, | 188 | { AM4_OCP2SCP1_CLKCTRL, NULL, CLKF_SW_SUP, "l4ls_gclk" }, |
| 189 | { AM4_EMIF_CLKCTRL, NULL, CLKF_SW_SUP, "dpll_ddr_m2_ck", "emif_clkdm" }, | 189 | { AM4_EMIF_CLKCTRL, NULL, CLKF_SW_SUP, "dpll_ddr_m2_ck", "emif_clkdm" }, |
| 190 | { AM4_DSS_CORE_CLKCTRL, NULL, CLKF_SW_SUP, "disp_clk", "dss_clkdm" }, | 190 | { AM4_DSS_CORE_CLKCTRL, NULL, CLKF_SW_SUP | CLKF_SET_RATE_PARENT, "disp_clk", "dss_clkdm" }, |
| 191 | { AM4_CPGMAC0_CLKCTRL, NULL, CLKF_SW_SUP, "cpsw_125mhz_gclk", "cpsw_125mhz_clkdm" }, | 191 | { AM4_CPGMAC0_CLKCTRL, NULL, CLKF_SW_SUP, "cpsw_125mhz_gclk", "cpsw_125mhz_clkdm" }, |
| 192 | { 0 }, | 192 | { 0 }, |
| 193 | }; | 193 | }; |
| @@ -537,6 +537,8 @@ static void __init _ti_omap4_clkctrl_setup(struct device_node *node) · 536 unmodified lines | |||
| 537 | init.parent_names = ®_data->parent; | 537 | init.parent_names = ®_data->parent; |
| 538 | init.num_parents = 1; | 538 | init.num_parents = 1; |
| 539 | init.flags = 0; | 539 | init.flags = 0; |
| 540 | if (reg_data->flags & CLKF_SET_RATE_PARENT) | ||
| 541 | init.flags |= CLK_SET_RATE_PARENT; | ||
| 540 | init.name = kasprintf(GFP_KERNEL, "%s:%s:%04x:%d", | 542 | init.name = kasprintf(GFP_KERNEL, "%s:%s:%04x:%d", |
| 541 | node->parent->name, node->name, | 543 | node->parent->name, node->name, |
| 542 | reg_data->offset, 0); | 544 | reg_data->offset, 0); |
| @@ -386,6 +386,7 @@ config ATMEL_PIT · 385 unmodified lines | |||
| 386 | 386 | ||
| 387 | config ATMEL_ST | 387 | config ATMEL_ST |
| 388 | bool "Atmel ST timer support" if COMPILE_TEST | 388 | bool "Atmel ST timer support" if COMPILE_TEST |
| 389 | depends on HAS_IOMEM | ||
| 389 | select TIMER_OF | 390 | select TIMER_OF |
| 390 | select MFD_SYSCON | 391 | select MFD_SYSCON |
| 391 | help | 392 | help |
| @@ -251,9 +251,14 @@ static irqreturn_t timer_irq_handler(int irq, void *dev_id) · 250 unmodified lines | |||
| 251 | int irq_reenable = clockevent_state_periodic(evt); | 251 | int irq_reenable = clockevent_state_periodic(evt); |
| 252 | 252 | ||
| 253 | /* | 253 | /* |
| 254 | * Any write to CTRL reg ACks the interrupt, we rewrite the | 254 | * 1. ACK the interrupt |
| 255 | * Count when [N]ot [H]alted bit. | 255 | * - For ARC700, any write to CTRL reg ACKs it, so just rewrite |
| 256 | * And re-arm it if perioid by [I]nterrupt [E]nable bit | 256 | * Count when [N]ot [H]alted bit. |
| 257 | * - For HS3x, it is a bit subtle. On taken count-down interrupt, | ||
| 258 | * IP bit [3] is set, which needs to be cleared for ACK'ing. | ||
| 259 | * The write below can only update the other two bits, hence | ||
| 260 | * explicitly clears IP bit | ||
| 261 | * 2. Re-arm interrupt if periodic by writing to IE bit [0] | ||
| 257 | */ | 262 | */ |
| 258 | write_aux_reg(ARC_REG_TIMER0_CTRL, irq_reenable | TIMER_CTRL_NH); | 263 | write_aux_reg(ARC_REG_TIMER0_CTRL, irq_reenable | TIMER_CTRL_NH); |
| 259 | 264 | ||
| @@ -281,7 +281,7 @@ static int __init __ftm_clk_init(struct device_node *np, char *cnt_name, · 280 unmodified lines | |||
| 281 | 281 | ||
| 282 | static unsigned long __init ftm_clk_init(struct device_node *np) | 282 | static unsigned long __init ftm_clk_init(struct device_node *np) |
| 283 | { | 283 | { |
| 284 | unsigned long freq; | 284 | long freq; |
| 285 | 285 | ||
| 286 | freq = __ftm_clk_init(np, "ftm-evt-counter-en", "ftm-evt"); | 286 | freq = __ftm_clk_init(np, "ftm-evt-counter-en", "ftm-evt"); |
| 287 | if (freq <= 0) | 287 | if (freq <= 0) |
| @@ -166,7 +166,7 @@ static int __init __gic_clocksource_init(void) · 165 unmodified lines | |||
| 166 | 166 | ||
| 167 | /* Set clocksource mask. */ | 167 | /* Set clocksource mask. */ |
| 168 | count_width = read_gic_config() & GIC_CONFIG_COUNTBITS; | 168 | count_width = read_gic_config() & GIC_CONFIG_COUNTBITS; |
| 169 | count_width >>= __fls(GIC_CONFIG_COUNTBITS); | 169 | count_width >>= __ffs(GIC_CONFIG_COUNTBITS); |
| 170 | count_width *= 4; | 170 | count_width *= 4; |
| 171 | count_width += 32; | 171 | count_width += 32; |
| 172 | gic_clocksource.mask = CLOCKSOURCE_MASK(count_width); | 172 | gic_clocksource.mask = CLOCKSOURCE_MASK(count_width); |
| @@ -205,12 +205,12 @@ static int __init gic_clocksource_of_init(struct device_node *node) · 32 unmodified lines | |||
| 205 | } else if (of_property_read_u32(node, "clock-frequency", | 205 | } else if (of_property_read_u32(node, "clock-frequency", |
| 206 | &gic_frequency)) { | 206 | &gic_frequency)) { |
| 207 | pr_err("GIC frequency not specified.\n"); | 207 | pr_err("GIC frequency not specified.\n"); |
| 208 | return -EINVAL;; | 208 | return -EINVAL; |
| 209 | } | 209 | } |
| 210 | gic_timer_irq = irq_of_parse_and_map(node, 0); | 210 | gic_timer_irq = irq_of_parse_and_map(node, 0); |
| 211 | if (!gic_timer_irq) { | 211 | if (!gic_timer_irq) { |
| 212 | pr_err("GIC timer IRQ not specified.\n"); | 212 | pr_err("GIC timer IRQ not specified.\n"); |
| 213 | return -EINVAL;; | 213 | return -EINVAL; |
| 214 | } | 214 | } |
| 215 | 215 | ||
| 216 | ret = __gic_clocksource_init(); | 216 | ret = __gic_clocksource_init(); |
| @@ -334,7 +334,7 @@ static int __init sun5i_timer_init(struct device_node *node) · 333 unmodified lines | |||
| 334 | timer_base = of_io_request_and_map(node, 0, of_node_full_name(node)); | 334 | timer_base = of_io_request_and_map(node, 0, of_node_full_name(node)); |
| 335 | if (IS_ERR(timer_base)) { | 335 | if (IS_ERR(timer_base)) { |
| 336 | pr_err("Can't map registers\n"); | 336 | pr_err("Can't map registers\n"); |
| 337 | return PTR_ERR(timer_base);; | 337 | return PTR_ERR(timer_base); |
| 338 | } | 338 | } |
| 339 | 339 | ||
| 340 | irq = irq_of_parse_and_map(node, 0); | 340 | irq = irq_of_parse_and_map(node, 0); |
| @@ -44,10 +44,10 @@ config ARM_DT_BL_CPUFREQ · 43 unmodified lines | |||
| 44 | 44 | ||
| 45 | config ARM_SCPI_CPUFREQ | 45 | config ARM_SCPI_CPUFREQ |
| 46 | tristate "SCPI based CPUfreq driver" | 46 | tristate "SCPI based CPUfreq driver" |
| 47 | depends on ARM_BIG_LITTLE_CPUFREQ && ARM_SCPI_PROTOCOL && COMMON_CLK_SCPI | 47 | depends on ARM_SCPI_PROTOCOL && COMMON_CLK_SCPI |
| 48 | help | 48 | help |
| 49 | This adds the CPUfreq driver support for ARM big.LITTLE platforms | 49 | This adds the CPUfreq driver support for ARM platforms using SCPI |
| 50 | using SCPI protocol for CPU power management. | 50 | protocol for CPU power management. |
| 51 | 51 | ||
| 52 | This driver uses SCPI Message Protocol driver to interact with the | 52 | This driver uses SCPI Message Protocol driver to interact with the |
| 53 | firmware providing the CPU DVFS functionality. | 53 | firmware providing the CPU DVFS functionality. |
| @@ -629,7 +629,7 @@ static int acpi_cpufreq_blacklist(struct cpuinfo_x86 *c) · 628 unmodified lines | |||
| 629 | if (c->x86_vendor == X86_VENDOR_INTEL) { | 629 | if (c->x86_vendor == X86_VENDOR_INTEL) { |
| 630 | if ((c->x86 == 15) && | 630 | if ((c->x86 == 15) && |
| 631 | (c->x86_model == 6) && | 631 | (c->x86_model == 6) && |
| 632 | (c->x86_mask == 8)) { | 632 | (c->x86_stepping == 8)) { |
| 633 | pr_info("Intel(R) Xeon(R) 7100 Errata AL30, processors may lock up on frequency changes: disabling acpi-cpufreq\n"); | 633 | pr_info("Intel(R) Xeon(R) 7100 Errata AL30, processors may lock up on frequency changes: disabling acpi-cpufreq\n"); |
| 634 | return -ENODEV; | 634 | return -ENODEV; |
| 635 | } | 635 | } |
| @@ -775,7 +775,7 @@ static int longhaul_cpu_init(struct cpufreq_policy *policy) · 774 unmodified lines | |||
| 775 | break; | 775 | break; |
| 776 | 776 | ||
| 777 | case 7: | 777 | case 7: |
| 778 | switch (c->x86_mask) { | 778 | switch (c->x86_stepping) { |
| 779 | case 0: | 779 | case 0: |
| 780 | longhaul_version = TYPE_LONGHAUL_V1; | 780 | longhaul_version = TYPE_LONGHAUL_V1; |
| 781 | cpu_model = CPU_SAMUEL2; | 781 | cpu_model = CPU_SAMUEL2; |
| @@ -787,7 +787,7 @@ static int longhaul_cpu_init(struct cpufreq_policy *policy) · 5 unmodified lines | |||
| 787 | break; | 787 | break; |
| 788 | case 1 ... 15: | 788 | case 1 ... 15: |
| 789 | longhaul_version = TYPE_LONGHAUL_V2; | 789 | longhaul_version = TYPE_LONGHAUL_V2; |
| 790 | if (c->x86_mask < 8) { | 790 | if (c->x86_stepping < 8) { |
| 791 | cpu_model = CPU_SAMUEL2; | 791 | cpu_model = CPU_SAMUEL2; |
| 792 | cpuname = "C3 'Samuel 2' [C5B]"; | 792 | cpuname = "C3 'Samuel 2' [C5B]"; |
| 793 | } else { | 793 | } else { |
| @@ -814,7 +814,7 @@ static int longhaul_cpu_init(struct cpufreq_policy *policy) · 20 unmodified lines | |||
| 814 | numscales = 32; | 814 | numscales = 32; |
| 815 | memcpy(mults, nehemiah_mults, sizeof(nehemiah_mults)); | 815 | memcpy(mults, nehemiah_mults, sizeof(nehemiah_mults)); |
| 816 | memcpy(eblcr, nehemiah_eblcr, sizeof(nehemiah_eblcr)); | 816 | memcpy(eblcr, nehemiah_eblcr, sizeof(nehemiah_eblcr)); |
| 817 | switch (c->x86_mask) { | 817 | switch (c->x86_stepping) { |
| 818 | case 0 ... 1: | 818 | case 0 ... 1: |
| 819 | cpu_model = CPU_NEHEMIAH; | 819 | cpu_model = CPU_NEHEMIAH; |
| 820 | cpuname = "C3 'Nehemiah A' [C5XLOE]"; | 820 | cpuname = "C3 'Nehemiah A' [C5XLOE]"; |
| @@ -168,7 +168,7 @@ static int cpufreq_p4_cpu_init(struct cpufreq_policy *policy) · 167 unmodified lines | |||
| 168 | #endif | 168 | #endif |
| 169 | 169 | ||
| 170 | /* Errata workaround */ | 170 | /* Errata workaround */ |
| 171 | cpuid = (c->x86 << 8) | (c->x86_model << 4) | c->x86_mask; | 171 | cpuid = (c->x86 << 8) | (c->x86_model << 4) | c->x86_stepping; |
| 172 | switch (cpuid) { | 172 | switch (cpuid) { |
| 173 | case 0x0f07: | 173 | case 0x0f07: |
| 174 | case 0x0f0a: | 174 | case 0x0f0a: |
| @@ -131,7 +131,7 @@ static int check_powernow(void) · 130 unmodified lines | |||
| 131 | return 0; | 131 | return 0; |
| 132 | } | 132 | } |
| 133 | 133 | ||
| 134 | if ((c->x86_model == 6) && (c->x86_mask == 0)) { | 134 | if ((c->x86_model == 6) && (c->x86_stepping == 0)) { |
| 135 | pr_info("K7 660[A0] core detected, enabling errata workarounds\n"); | 135 | pr_info("K7 660[A0] core detected, enabling errata workarounds\n"); |
| 136 | have_a0 = 1; | 136 | have_a0 = 1; |
| 137 | } | 137 | } |
| @@ -351,7 +351,13 @@ struct clk *s3c_cpufreq_clk_get(struct device *dev, const char *name) · 350 unmodified lines | |||
| 351 | static int s3c_cpufreq_init(struct cpufreq_policy *policy) | 351 | static int s3c_cpufreq_init(struct cpufreq_policy *policy) |
| 352 | { | 352 | { |
| 353 | policy->clk = clk_arm; | 353 | policy->clk = clk_arm; |
| 354 | return cpufreq_generic_init(policy, ftab, cpu_cur.info->latency); | 354 | |
| 355 | policy->cpuinfo.transition_latency = cpu_cur.info->latency; | ||
| 356 | |||
| 357 | if (ftab) | ||
| 358 | return cpufreq_table_validate_and_show(policy, ftab); | ||
| 359 | |||
| 360 | return 0; | ||
| 355 | } | 361 | } |
| 356 | 362 | ||
| 357 | static int __init s3c_cpufreq_initclks(void) | 363 | static int __init s3c_cpufreq_initclks(void) |
| @@ -51,15 +51,23 @@ static unsigned int scpi_cpufreq_get_rate(unsigned int cpu) · 50 unmodified lines | |||
| 51 | static int | 51 | static int |
| 52 | scpi_cpufreq_set_target(struct cpufreq_policy *policy, unsigned int index) | 52 | scpi_cpufreq_set_target(struct cpufreq_policy *policy, unsigned int index) |
| 53 | { | 53 | { |
| 54 | unsigned long freq = policy->freq_table[index].frequency; | ||
| 54 | struct scpi_data *priv = policy->driver_data; | 55 | struct scpi_data *priv = policy->driver_data; |
| 55 | u64 rate = policy->freq_table[index].frequency * 1000; | 56 | u64 rate = freq * 1000; |
| 56 | int ret; | 57 | int ret; |
| 57 | 58 | ||
| 58 | ret = clk_set_rate(priv->clk, rate); | 59 | ret = clk_set_rate(priv->clk, rate); |
| 59 | if (!ret && (clk_get_rate(priv->clk) != rate)) | ||
| 60 | ret = -EIO; | ||
| 61 | 60 | ||
| 62 | return ret; | 61 | if (ret) |
| 62 | return ret; | ||
| 63 | |||
| 64 | if (clk_get_rate(priv->clk) != rate) | ||
| 65 | return -EIO; | ||
| 66 | |||
| 67 | arch_set_freq_scale(policy->related_cpus, freq, | ||
| 68 | policy->cpuinfo.max_freq); | ||
| 69 | |||
| 70 | return 0; | ||
| 63 | } | 71 | } |
| 64 | 72 | ||
| 65 | static int | 73 | static int |
| @@ -37,7 +37,7 @@ struct cpu_id · 36 unmodified lines | |||
| 37 | { | 37 | { |
| 38 | __u8 x86; /* CPU family */ | 38 | __u8 x86; /* CPU family */ |
| 39 | __u8 x86_model; /* model */ | 39 | __u8 x86_model; /* model */ |
| 40 | __u8 x86_mask; /* stepping */ | 40 | __u8 x86_stepping; /* stepping */ |
| 41 | }; | 41 | }; |
| 42 | 42 | ||
| 43 | enum { | 43 | enum { |
| @@ -277,7 +277,7 @@ static int centrino_verify_cpu_id(const struct cpuinfo_x86 *c, · 233 unmodified lines | |||
| 277 | { | 277 | { |
| 278 | if ((c->x86 == x->x86) && | 278 | if ((c->x86 == x->x86) && |
| 279 | (c->x86_model == x->x86_model) && | 279 | (c->x86_model == x->x86_model) && |
| 280 | (c->x86_mask == x->x86_mask)) | 280 | (c->x86_stepping == x->x86_stepping)) |
| 281 | return 1; | 281 | return 1; |
| 282 | return 0; | 282 | return 0; |
| 283 | } | 283 | } |
| @@ -272,9 +272,9 @@ unsigned int speedstep_detect_processor(void) · 271 unmodified lines | |||
| 272 | ebx = cpuid_ebx(0x00000001); | 272 | ebx = cpuid_ebx(0x00000001); |
| 273 | ebx &= 0x000000FF; | 273 | ebx &= 0x000000FF; |
| 274 | 274 | ||
| 275 | pr_debug("ebx value is %x, x86_mask is %x\n", ebx, c->x86_mask); | 275 | pr_debug("ebx value is %x, x86_stepping is %x\n", ebx, c->x86_stepping); |
| 276 | 276 | ||
| 277 | switch (c->x86_mask) { | 277 | switch (c->x86_stepping) { |
| 278 | case 4: | 278 | case 4: |
| 279 | /* | 279 | /* |
| 280 | * B-stepping [M-P4-M] | 280 | * B-stepping [M-P4-M] |
| @@ -361,7 +361,7 @@ unsigned int speedstep_detect_processor(void) · 80 unmodified lines | |||
| 361 | msr_lo, msr_hi); | 361 | msr_lo, msr_hi); |
| 362 | if ((msr_hi & (1<<18)) && | 362 | if ((msr_hi & (1<<18)) && |
| 363 | (relaxed_check ? 1 : (msr_hi & (3<<24)))) { | 363 | (relaxed_check ? 1 : (msr_hi & (3<<24)))) { |
| 364 | if (c->x86_mask == 0x01) { | 364 | if (c->x86_stepping == 0x01) { |
| 365 | pr_debug("early PIII version\n"); | 365 | pr_debug("early PIII version\n"); |
| 366 | return SPEEDSTEP_CPU_PIII_C_EARLY; | 366 | return SPEEDSTEP_CPU_PIII_C_EARLY; |
| 367 | } else | 367 | } else |
| @@ -228,12 +228,16 @@ static int instantiate_rng(struct device *ctrldev, int state_handle_mask, · 227 unmodified lines | |||
| 228 | * without any error (HW optimizations for later | 228 | * without any error (HW optimizations for later |
| 229 | * CAAM eras), then try again. | 229 | * CAAM eras), then try again. |
| 230 | */ | 230 | */ |
| 231 | if (ret) | ||
| 232 | break; | ||
| 233 | |||
| 231 | rdsta_val = rd_reg32(&ctrl->r4tst[0].rdsta) & RDSTA_IFMASK; | 234 | rdsta_val = rd_reg32(&ctrl->r4tst[0].rdsta) & RDSTA_IFMASK; |
| 232 | if ((status && status != JRSTA_SSRC_JUMP_HALT_CC) || | 235 | if ((status && status != JRSTA_SSRC_JUMP_HALT_CC) || |
| 233 | !(rdsta_val & (1 << sh_idx))) | 236 | !(rdsta_val & (1 << sh_idx))) { |
| 234 | ret = -EAGAIN; | 237 | ret = -EAGAIN; |
| 235 | if (ret) | ||
| 236 | break; | 238 | break; |
| 239 | } | ||
| 240 | |||
| 237 | dev_info(ctrldev, "Instantiated RNG4 SH%d\n", sh_idx); | 241 | dev_info(ctrldev, "Instantiated RNG4 SH%d\n", sh_idx); |
| 238 | /* Clear the contents before recreating the descriptor */ | 242 | /* Clear the contents before recreating the descriptor */ |
| 239 | memset(desc, 0x00, CAAM_CMD_SZ * 7); | 243 | memset(desc, 0x00, CAAM_CMD_SZ * 7); |
| @@ -211,7 +211,7 @@ static int __sev_platform_shutdown_locked(int *error) · 210 unmodified lines | |||
| 211 | { | 211 | { |
| 212 | int ret; | 212 | int ret; |
| 213 | 213 | ||
| 214 | ret = __sev_do_cmd_locked(SEV_CMD_SHUTDOWN, 0, error); | 214 | ret = __sev_do_cmd_locked(SEV_CMD_SHUTDOWN, NULL, error); |
| 215 | if (ret) | 215 | if (ret) |
| 216 | return ret; | 216 | return ret; |
| 217 | 217 | ||
| @@ -271,7 +271,7 @@ static int sev_ioctl_do_reset(struct sev_issue_cmd *argp) · 53 unmodified lines | |||
| 271 | return rc; | 271 | return rc; |
| 272 | } | 272 | } |
| 273 | 273 | ||
| 274 | return __sev_do_cmd_locked(SEV_CMD_FACTORY_RESET, 0, &argp->error); | 274 | return __sev_do_cmd_locked(SEV_CMD_FACTORY_RESET, NULL, &argp->error); |
| 275 | } | 275 | } |
| 276 | 276 | ||
| 277 | static int sev_ioctl_do_platform_status(struct sev_issue_cmd *argp) | 277 | static int sev_ioctl_do_platform_status(struct sev_issue_cmd *argp) |
| @@ -299,7 +299,7 @@ static int sev_ioctl_do_pek_pdh_gen(int cmd, struct sev_issue_cmd *argp) · 21 unmodified lines | |||
| 299 | return rc; | 299 | return rc; |
| 300 | } | 300 | } |
| 301 | 301 | ||
| 302 | return __sev_do_cmd_locked(cmd, 0, &argp->error); | 302 | return __sev_do_cmd_locked(cmd, NULL, &argp->error); |
| 303 | } | 303 | } |
| 304 | 304 | ||
| 305 | static int sev_ioctl_do_pek_csr(struct sev_issue_cmd *argp) | 305 | static int sev_ioctl_do_pek_csr(struct sev_issue_cmd *argp) |
| @@ -624,7 +624,7 @@ EXPORT_SYMBOL_GPL(sev_guest_decommission); · 318 unmodified lines | |||
| 624 | 624 | ||
| 625 | int sev_guest_df_flush(int *error) | 625 | int sev_guest_df_flush(int *error) |
| 626 | { | 626 | { |
| 627 | return sev_do_cmd(SEV_CMD_DF_FLUSH, 0, error); | 627 | return sev_do_cmd(SEV_CMD_DF_FLUSH, NULL, error); |
| 628 | } | 628 | } |
| 629 | EXPORT_SYMBOL_GPL(sev_guest_df_flush); | 629 | EXPORT_SYMBOL_GPL(sev_guest_df_flush); |
| 630 | 630 | ||
| @@ -512,7 +512,7 @@ static int __init padlock_init(void) · 511 unmodified lines | |||
| 512 | 512 | ||
| 513 | printk(KERN_NOTICE PFX "Using VIA PadLock ACE for AES algorithm.\n"); | 513 | printk(KERN_NOTICE PFX "Using VIA PadLock ACE for AES algorithm.\n"); |
| 514 | 514 | ||
| 515 | if (c->x86 == 6 && c->x86_model == 15 && c->x86_mask == 2) { | 515 | if (c->x86 == 6 && c->x86_model == 15 && c->x86_stepping == 2) { |
| 516 | ecb_fetch_blocks = MAX_ECB_FETCH_BLOCKS; | 516 | ecb_fetch_blocks = MAX_ECB_FETCH_BLOCKS; |
| 517 | cbc_fetch_blocks = MAX_CBC_FETCH_BLOCKS; | 517 | cbc_fetch_blocks = MAX_CBC_FETCH_BLOCKS; |
| 518 | printk(KERN_NOTICE PFX "VIA Nano stepping 2 detected: enabling workaround.\n"); | 518 | printk(KERN_NOTICE PFX "VIA Nano stepping 2 detected: enabling workaround.\n"); |
| @@ -1922,15 +1922,21 @@ static void s5p_aes_crypt_start(struct s5p_aes_dev *dev, unsigned long mode) · 1921 unmodified lines | |||
| 1922 | uint32_t aes_control; | 1922 | uint32_t aes_control; |
| 1923 | unsigned long flags; | 1923 | unsigned long flags; |
| 1924 | int err; | 1924 | int err; |
| 1925 | u8 *iv; | ||
| 1925 | 1926 | ||
| 1926 | aes_control = SSS_AES_KEY_CHANGE_MODE; | 1927 | aes_control = SSS_AES_KEY_CHANGE_MODE; |
| 1927 | if (mode & FLAGS_AES_DECRYPT) | 1928 | if (mode & FLAGS_AES_DECRYPT) |
| 1928 | aes_control |= SSS_AES_MODE_DECRYPT; | 1929 | aes_control |= SSS_AES_MODE_DECRYPT; |
| 1929 | 1930 | ||
| 1930 | if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CBC) | 1931 | if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CBC) { |
| 1931 | aes_control |= SSS_AES_CHAIN_MODE_CBC; | 1932 | aes_control |= SSS_AES_CHAIN_MODE_CBC; |
| 1932 | else if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CTR) | 1933 | iv = req->info; |
| 1934 | } else if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CTR) { | ||
| 1933 | aes_control |= SSS_AES_CHAIN_MODE_CTR; | 1935 | aes_control |= SSS_AES_CHAIN_MODE_CTR; |
| 1936 | iv = req->info; | ||
| 1937 | } else { | ||
| 1938 | iv = NULL; /* AES_ECB */ | ||
| 1939 | } | ||
| 1934 | 1940 | ||
| 1935 | if (dev->ctx->keylen == AES_KEYSIZE_192) | 1941 | if (dev->ctx->keylen == AES_KEYSIZE_192) |
| 1936 | aes_control |= SSS_AES_KEY_SIZE_192; | 1942 | aes_control |= SSS_AES_KEY_SIZE_192; |
| @@ -1961,7 +1967,7 @@ static void s5p_aes_crypt_start(struct s5p_aes_dev *dev, unsigned long mode) · 24 unmodified lines | |||
| 1961 | goto outdata_error; | 1967 | goto outdata_error; |
| 1962 | 1968 | ||
| 1963 | SSS_AES_WRITE(dev, AES_CONTROL, aes_control); | 1969 | SSS_AES_WRITE(dev, AES_CONTROL, aes_control); |
| 1964 | s5p_set_aes(dev, dev->ctx->aes_key, req->info, dev->ctx->keylen); | 1970 | s5p_set_aes(dev, dev->ctx->aes_key, iv, dev->ctx->keylen); |
| 1965 | 1971 | ||
| 1966 | s5p_set_dma_indata(dev, dev->sg_src); | 1972 | s5p_set_dma_indata(dev, dev->sg_src); |
| 1967 | s5p_set_dma_outdata(dev, dev->sg_dst); | 1973 | s5p_set_dma_outdata(dev, dev->sg_dst); |
| @@ -28,7 +28,7 @@ int sun4i_ss_prng_generate(struct crypto_rng *tfm, const u8 *src, · 27 unmodified lines | |||
| 28 | algt = container_of(alg, struct sun4i_ss_alg_template, alg.rng); | 28 | algt = container_of(alg, struct sun4i_ss_alg_template, alg.rng); |
| 29 | ss = algt->ss; | 29 | ss = algt->ss; |
| 30 | 30 | ||
| 31 | spin_lock(&ss->slock); | 31 | spin_lock_bh(&ss->slock); |
| 32 | 32 | ||
| 33 | writel(mode, ss->base + SS_CTL); | 33 | writel(mode, ss->base + SS_CTL); |
| 34 | 34 | ||
| @@ -51,6 +51,6 @@ int sun4i_ss_prng_generate(struct crypto_rng *tfm, const u8 *src, · 16 unmodified lines | |||
| 51 | } | 51 | } |
| 52 | 52 | ||
| 53 | writel(0, ss->base + SS_CTL); | 53 | writel(0, ss->base + SS_CTL); |
| 54 | spin_unlock(&ss->slock); | 54 | spin_unlock_bh(&ss->slock); |
| 55 | return dlen; | 55 | return 0; |
| 56 | } | 56 | } |
| @@ -1138,6 +1138,10 @@ static int talitos_sg_map(struct device *dev, struct scatterlist *src, · 1137 unmodified lines | |||
| 1138 | struct talitos_private *priv = dev_get_drvdata(dev); | 1138 | struct talitos_private *priv = dev_get_drvdata(dev); |
| 1139 | bool is_sec1 = has_ftr_sec1(priv); | 1139 | bool is_sec1 = has_ftr_sec1(priv); |
| 1140 | 1140 | ||
| 1141 | if (!src) { | ||
| 1142 | to_talitos_ptr(ptr, 0, 0, is_sec1); | ||
| 1143 | return 1; | ||
| 1144 | } | ||
| 1141 | if (sg_count == 1) { | 1145 | if (sg_count == 1) { |
| 1142 | to_talitos_ptr(ptr, sg_dma_address(src) + offset, len, is_sec1); | 1146 | to_talitos_ptr(ptr, sg_dma_address(src) + offset, len, is_sec1); |
| 1143 | return sg_count; | 1147 | return sg_count; |
| @@ -246,12 +246,6 @@ long dax_direct_access(struct dax_device *dax_dev, pgoff_t pgoff, long nr_pages, · 245 unmodified lines | |||
| 246 | { | 246 | { |
| 247 | long avail; | 247 | long avail; |
| 248 | 248 | ||
| 249 | /* | ||
| 250 | * The device driver is allowed to sleep, in order to make the | ||
| 251 | * memory directly accessible. | ||
| 252 | */ | ||
| 253 | might_sleep(); | ||
| 254 | |||
| 255 | if (!dax_dev) | 249 | if (!dax_dev) |
| 256 | return -EOPNOTSUPP; | 250 | return -EOPNOTSUPP; |
| 257 | 251 | ||
| @@ -163,6 +163,7 @@ struct mv_xor_v2_device { · 162 unmodified lines | |||
| 163 | void __iomem *dma_base; | 163 | void __iomem *dma_base; |
| 164 | void __iomem *glob_base; | 164 | void __iomem *glob_base; |
| 165 | struct clk *clk; | 165 | struct clk *clk; |
| 166 | struct clk *reg_clk; | ||
| 166 | struct tasklet_struct irq_tasklet; | 167 | struct tasklet_struct irq_tasklet; |
| 167 | struct list_head free_sw_desc; | 168 | struct list_head free_sw_desc; |
| 168 | struct dma_device dmadev; | 169 | struct dma_device dmadev; |
| @@ -749,13 +750,26 @@ static int mv_xor_v2_probe(struct platform_device *pdev) · 580 unmodified lines | |||
| 749 | if (ret) | 750 | if (ret) |
| 750 | return ret; | 751 | return ret; |
| 751 | 752 | ||
| 753 | xor_dev->reg_clk = devm_clk_get(&pdev->dev, "reg"); | ||
| 754 | if (PTR_ERR(xor_dev->reg_clk) != -ENOENT) { | ||
| 755 | if (!IS_ERR(xor_dev->reg_clk)) { | ||
| 756 | ret = clk_prepare_enable(xor_dev->reg_clk); | ||
| 757 | if (ret) | ||
| 758 | return ret; | ||
| 759 | } else { | ||
| 760 | return PTR_ERR(xor_dev->reg_clk); | ||
| 761 | } | ||
| 762 | } | ||
| 763 | |||
| 752 | xor_dev->clk = devm_clk_get(&pdev->dev, NULL); | 764 | xor_dev->clk = devm_clk_get(&pdev->dev, NULL); |
| 753 | if (IS_ERR(xor_dev->clk) && PTR_ERR(xor_dev->clk) == -EPROBE_DEFER) | 765 | if (IS_ERR(xor_dev->clk) && PTR_ERR(xor_dev->clk) == -EPROBE_DEFER) { |
| 754 | return -EPROBE_DEFER; | 766 | ret = EPROBE_DEFER; |
| 767 | goto disable_reg_clk; | ||
| 768 | } | ||
| 755 | if (!IS_ERR(xor_dev->clk)) { | 769 | if (!IS_ERR(xor_dev->clk)) { |
| 756 | ret = clk_prepare_enable(xor_dev->clk); | 770 | ret = clk_prepare_enable(xor_dev->clk); |
| 757 | if (ret) | 771 | if (ret) |
| 758 | return ret; | 772 | goto disable_reg_clk; |
| 759 | } | 773 | } |
| 760 | 774 | ||
| 761 | ret = platform_msi_domain_alloc_irqs(&pdev->dev, 1, | 775 | ret = platform_msi_domain_alloc_irqs(&pdev->dev, 1, |
| @@ -866,8 +880,9 @@ free_hw_desq: · 104 unmodified lines | |||
| 866 | free_msi_irqs: | 880 | free_msi_irqs: |
| 867 | platform_msi_domain_free_irqs(&pdev->dev); | 881 | platform_msi_domain_free_irqs(&pdev->dev); |
| 868 | disable_clk: | 882 | disable_clk: |
| 869 | if (!IS_ERR(xor_dev->clk)) | 883 | clk_disable_unprepare(xor_dev->clk); |
| 870 | clk_disable_unprepare(xor_dev->clk); | 884 | disable_reg_clk: |
| 885 | clk_disable_unprepare(xor_dev->reg_clk); | ||
| 871 | return ret; | 886 | return ret; |
| 872 | } | 887 | } |
| 873 | 888 | ||
| @@ -917,7 +917,7 @@ rcar_dmac_chan_prep_sg(struct rcar_dmac_chan *chan, struct scatterlist *sgl, · 916 unmodified lines | |||
| 917 | 917 | ||
| 918 | rcar_dmac_chan_configure_desc(chan, desc); | 918 | rcar_dmac_chan_configure_desc(chan, desc); |
| 919 | 919 | ||
| 920 | max_chunk_size = (RCAR_DMATCR_MASK + 1) << desc->xfer_shift; | 920 | max_chunk_size = RCAR_DMATCR_MASK << desc->xfer_shift; |
| 921 | 921 | ||
| 922 | /* | 922 | /* |
| 923 | * Allocate and fill the transfer chunk descriptors. We own the only | 923 | * Allocate and fill the transfer chunk descriptors. We own the only |
| @@ -118,14 +118,15 @@ static void *stm32_dmamux_route_allocate(struct of_phandle_args *dma_spec, · 117 unmodified lines | |||
| 118 | spin_lock_irqsave(&dmamux->lock, flags); | 118 | spin_lock_irqsave(&dmamux->lock, flags); |
| 119 | mux->chan_id = find_first_zero_bit(dmamux->dma_inuse, | 119 | mux->chan_id = find_first_zero_bit(dmamux->dma_inuse, |
| 120 | dmamux->dma_requests); | 120 | dmamux->dma_requests); |
| 121 | set_bit(mux->chan_id, dmamux->dma_inuse); | ||
| 122 | spin_unlock_irqrestore(&dmamux->lock, flags); | ||
| 123 | 121 | ||
| 124 | if (mux->chan_id == dmamux->dma_requests) { | 122 | if (mux->chan_id == dmamux->dma_requests) { |
| 123 | spin_unlock_irqrestore(&dmamux->lock, flags); | ||
| 125 | dev_err(&pdev->dev, "Run out of free DMA requests\n"); | 124 | dev_err(&pdev->dev, "Run out of free DMA requests\n"); |
| 126 | ret = -ENOMEM; | 125 | ret = -ENOMEM; |
| 127 | goto error; | 126 | goto error_chan_id; |
| 128 | } | 127 | } |
| 128 | set_bit(mux->chan_id, dmamux->dma_inuse); | ||
| 129 | spin_unlock_irqrestore(&dmamux->lock, flags); | ||
| 129 | 130 | ||
| 130 | /* Look for DMA Master */ | 131 | /* Look for DMA Master */ |
| 131 | for (i = 1, min = 0, max = dmamux->dma_reqs[i]; | 132 | for (i = 1, min = 0, max = dmamux->dma_reqs[i]; |
| @@ -173,6 +174,8 @@ static void *stm32_dmamux_route_allocate(struct of_phandle_args *dma_spec, · 41 unmodified lines | |||
| 173 | 174 | ||
| 174 | error: | 175 | error: |
| 175 | clear_bit(mux->chan_id, dmamux->dma_inuse); | 176 | clear_bit(mux->chan_id, dmamux->dma_inuse); |
| 177 | |||
| 178 | error_chan_id: | ||
| 176 | kfree(mux); | 179 | kfree(mux); |
| 177 | return ERR_PTR(ret); | 180 | return ERR_PTR(ret); |
| 178 | } | 181 | } |
| @@ -3147,7 +3147,7 @@ static struct amd64_family_type *per_family_init(struct amd64_pvt *pvt) · 3146 unmodified lines | |||
| 3147 | struct amd64_family_type *fam_type = NULL; | 3147 | struct amd64_family_type *fam_type = NULL; |
| 3148 | 3148 | ||
| 3149 | pvt->ext_model = boot_cpu_data.x86_model >> 4; | 3149 | pvt->ext_model = boot_cpu_data.x86_model >> 4; |
| 3150 | pvt->stepping = boot_cpu_data.x86_mask; | 3150 | pvt->stepping = boot_cpu_data.x86_stepping; |
| 3151 | pvt->model = boot_cpu_data.x86_model; | 3151 | pvt->model = boot_cpu_data.x86_model; |
| 3152 | pvt->fam = boot_cpu_data.x86; | 3152 | pvt->fam = boot_cpu_data.x86; |
| 3153 | 3153 | ||
| @@ -279,7 +279,7 @@ static const u32 correrrthrsld[] = { · 278 unmodified lines | |||
| 279 | * sbridge structs | 279 | * sbridge structs |
| 280 | */ | 280 | */ |
| 281 | 281 | ||
| 282 | #define NUM_CHANNELS 4 /* Max channels per MC */ | 282 | #define NUM_CHANNELS 6 /* Max channels per MC */ |
| 283 | #define MAX_DIMMS 3 /* Max DIMMS per channel */ | 283 | #define MAX_DIMMS 3 /* Max DIMMS per channel */ |
| 284 | #define KNL_MAX_CHAS 38 /* KNL max num. of Cache Home Agents */ | 284 | #define KNL_MAX_CHAS 38 /* KNL max num. of Cache Home Agents */ |
| 285 | #define KNL_MAX_CHANNELS 6 /* KNL max num. of PCI channels */ | 285 | #define KNL_MAX_CHANNELS 6 /* KNL max num. of PCI channels */ |
| @@ -1,7 +1,6 @@ | |||
| 1 | /* | 1 | /* |
| 2 | * extcon-axp288.c - X-Power AXP288 PMIC extcon cable detection driver | 2 | * extcon-axp288.c - X-Power AXP288 PMIC extcon cable detection driver |
| 3 | * | 3 | * |
| 4 | * Copyright (C) 2016-2017 Hans de Goede <hdegoede@redhat.com> | ||
| 5 | * Copyright (C) 2015 Intel Corporation | 4 | * Copyright (C) 2015 Intel Corporation |
| 6 | * Author: Ramakrishna Pallala <ramakrishna.pallala@intel.com> | 5 | * Author: Ramakrishna Pallala <ramakrishna.pallala@intel.com> |
| 7 | * | 6 | * |
| @@ -98,15 +97,13 @@ struct axp288_extcon_info { · 90 unmodified lines | |||
| 98 | struct device *dev; | 97 | struct device *dev; |
| 99 | struct regmap *regmap; | 98 | struct regmap *regmap; |
| 100 | struct regmap_irq_chip_data *regmap_irqc; | 99 | struct regmap_irq_chip_data *regmap_irqc; |
| 101 | struct delayed_work det_work; | ||
| 102 | int irq[EXTCON_IRQ_END]; | 100 | int irq[EXTCON_IRQ_END]; |
| 103 | struct extcon_dev *edev; | 101 | struct extcon_dev *edev; |
| 104 | unsigned int previous_cable; | 102 | unsigned int previous_cable; |
| 105 | bool first_detect_done; | ||
| 106 | }; | 103 | }; |
| 107 | 104 | ||
| 108 | /* Power up/down reason string array */ | 105 | /* Power up/down reason string array */ |
| 109 | static char *axp288_pwr_up_down_info[] = { | 106 | static const char * const axp288_pwr_up_down_info[] = { |
| 110 | "Last wake caused by user pressing the power button", | 107 | "Last wake caused by user pressing the power button", |
| 111 | "Last wake caused by a charger insertion", | 108 | "Last wake caused by a charger insertion", |
| 112 | "Last wake caused by a battery insertion", | 109 | "Last wake caused by a battery insertion", |
| @@ -124,7 +121,7 @@ static char *axp288_pwr_up_down_info[] = { · 11 unmodified lines | |||
| 124 | */ | 121 | */ |
| 125 | static void axp288_extcon_log_rsi(struct axp288_extcon_info *info) | 122 | static void axp288_extcon_log_rsi(struct axp288_extcon_info *info) |
| 126 | { | 123 | { |
| 127 | char **rsi; | 124 | const char * const *rsi; |
| 128 | unsigned int val, i, clear_mask = 0; | 125 | unsigned int val, i, clear_mask = 0; |
| 129 | int ret; | 126 | int ret; |
| 130 | 127 | ||
| @@ -140,25 +137,6 @@ static void axp288_extcon_log_rsi(struct axp288_extcon_info *info) · 9 unmodified lines | |||
| 140 | regmap_write(info->regmap, AXP288_PS_BOOT_REASON_REG, clear_mask); | 137 | regmap_write(info->regmap, AXP288_PS_BOOT_REASON_REG, clear_mask); |
| 141 | } | 138 | } |
| 142 | 139 | ||
| 143 | static void axp288_chrg_detect_complete(struct axp288_extcon_info *info) | ||
| 144 | { | ||
| 145 | /* | ||
| 146 | * We depend on other drivers to do things like mux the data lines, | ||
| 147 | * enable/disable vbus based on the id-pin, etc. Sometimes the BIOS has | ||
| 148 | * not set these things up correctly resulting in the initial charger | ||
| 149 | * cable type detection giving a wrong result and we end up not charging | ||
| 150 | * or charging at only 0.5A. | ||
| 151 | * | ||
| 152 | * So we schedule a second cable type detection after 2 seconds to | ||
| 153 | * give the other drivers time to load and do their thing. | ||
| 154 | */ | ||
| 155 | if (!info->first_detect_done) { | ||
| 156 | queue_delayed_work(system_wq, &info->det_work, | ||
| 157 | msecs_to_jiffies(2000)); | ||
| 158 | info->first_detect_done = true; | ||
| 159 | } | ||
| 160 | } | ||
| 161 | |||
| 162 | static int axp288_handle_chrg_det_event(struct axp288_extcon_info *info) | 140 | static int axp288_handle_chrg_det_event(struct axp288_extcon_info *info) |
| 163 | { | 141 | { |
| 164 | int ret, stat, cfg, pwr_stat; | 142 | int ret, stat, cfg, pwr_stat; |
| @@ -223,8 +201,6 @@ no_vbus: · 58 unmodified lines | |||
| 223 | info->previous_cable = cable; | 201 | info->previous_cable = cable; |
| 224 | } | 202 | } |
| 225 | 203 | ||
| 226 | axp288_chrg_detect_complete(info); | ||
| 227 | |||
| 228 | return 0; | 204 | return 0; |
| 229 | 205 | ||
| 230 | dev_det_ret: | 206 | dev_det_ret: |
| @@ -246,11 +222,8 @@ static irqreturn_t axp288_extcon_isr(int irq, void *data) · 15 unmodified lines | |||
| 246 | return IRQ_HANDLED; | 222 | return IRQ_HANDLED; |
| 247 | } | 223 | } |
| 248 | 224 | ||
| 249 | static void axp288_extcon_det_work(struct work_struct *work) | 225 | static void axp288_extcon_enable(struct axp288_extcon_info *info) |
| 250 | { | 226 | { |
| 251 | struct axp288_extcon_info *info = | ||
| 252 | container_of(work, struct axp288_extcon_info, det_work.work); | ||
| 253 | |||
| 254 | regmap_update_bits(info->regmap, AXP288_BC_GLOBAL_REG, | 227 | regmap_update_bits(info->regmap, AXP288_BC_GLOBAL_REG, |
| 255 | BC_GLOBAL_RUN, 0); | 228 | BC_GLOBAL_RUN, 0); |
| 256 | /* Enable the charger detection logic */ | 229 | /* Enable the charger detection logic */ |
| @@ -272,7 +245,6 @@ static int axp288_extcon_probe(struct platform_device *pdev) · 15 unmodified lines | |||
| 272 | info->regmap = axp20x->regmap; | 245 | info->regmap = axp20x->regmap; |
| 273 | info->regmap_irqc = axp20x->regmap_irqc; | 246 | info->regmap_irqc = axp20x->regmap_irqc; |
| 274 | info->previous_cable = EXTCON_NONE; | 247 | info->previous_cable = EXTCON_NONE; |
| 275 | INIT_DELAYED_WORK(&info->det_work, axp288_extcon_det_work); | ||
| 276 | 248 | ||
| 277 | platform_set_drvdata(pdev, info); | 249 | platform_set_drvdata(pdev, info); |
| 278 | 250 | ||
| @@ -318,7 +290,7 @@ static int axp288_extcon_probe(struct platform_device *pdev) · 39 unmodified lines | |||
| 318 | } | 290 | } |
| 319 | 291 | ||
| 320 | /* Start charger cable type detection */ | 292 | /* Start charger cable type detection */ |
| 321 | queue_delayed_work(system_wq, &info->det_work, 0); | 293 | axp288_extcon_enable(info); |
| 322 | 294 | ||
| 323 | return 0; | 295 | return 0; |
| 324 | } | 296 | } |
| @@ -153,8 +153,9 @@ static int int3496_probe(struct platform_device *pdev) · 152 unmodified lines | |||
| 153 | return ret; | 153 | return ret; |
| 154 | } | 154 | } |
| 155 | 155 | ||
| 156 | /* queue initial processing of id-pin */ | 156 | /* process id-pin so that we start with the right status */ |
| 157 | queue_delayed_work(system_wq, &data->work, 0); | 157 | queue_delayed_work(system_wq, &data->work, 0); |
| 158 | flush_delayed_work(&data->work); | ||
| 158 | 159 | ||
| 159 | platform_set_drvdata(pdev, data); | 160 | platform_set_drvdata(pdev, data); |
| 160 | 161 | ||
| @@ -639,7 +639,7 @@ static void __exit dcdbas_exit(void) · 638 unmodified lines | |||
| 639 | platform_driver_unregister(&dcdbas_driver); | 639 | platform_driver_unregister(&dcdbas_driver); |
| 640 | } | 640 | } |
| 641 | 641 | ||
| 642 | module_init(dcdbas_init); | 642 | subsys_initcall_sync(dcdbas_init); |
| 643 | module_exit(dcdbas_exit); | 643 | module_exit(dcdbas_exit); |
| 644 | 644 | ||
| 645 | MODULE_DESCRIPTION(DRIVER_DESCRIPTION " (version " DRIVER_VERSION ")"); | 645 | MODULE_DESCRIPTION(DRIVER_DESCRIPTION " (version " DRIVER_VERSION ")"); |
| @@ -68,11 +68,11 @@ void efi_retrieve_tpm2_eventlog_1_2(efi_system_table_t *sys_table_arg) · 67 unmodified lines | |||
| 68 | efi_guid_t linux_eventlog_guid = LINUX_EFI_TPM_EVENT_LOG_GUID; | 68 | efi_guid_t linux_eventlog_guid = LINUX_EFI_TPM_EVENT_LOG_GUID; |
| 69 | efi_status_t status; | 69 | efi_status_t status; |
| 70 | efi_physical_addr_t log_location, log_last_entry; | 70 | efi_physical_addr_t log_location, log_last_entry; |
| 71 | struct linux_efi_tpm_eventlog *log_tbl; | 71 | struct linux_efi_tpm_eventlog *log_tbl = NULL; |
| 72 | unsigned long first_entry_addr, last_entry_addr; | 72 | unsigned long first_entry_addr, last_entry_addr; |
| 73 | size_t log_size, last_entry_size; | 73 | size_t log_size, last_entry_size; |
| 74 | efi_bool_t truncated; | 74 | efi_bool_t truncated; |
| 75 | void *tcg2_protocol; | 75 | void *tcg2_protocol = NULL; |
| 76 | 76 | ||
| 77 | status = efi_call_early(locate_protocol, &tcg2_guid, NULL, | 77 | status = efi_call_early(locate_protocol, &tcg2_guid, NULL, |
| 78 | &tcg2_protocol); | 78 | &tcg2_protocol); |
| @@ -14,7 +14,6 @@ · 13 unmodified lines | |||
| 14 | * GNU General Public License for more details. | 14 | * GNU General Public License for more details. |
| 15 | */ | 15 | */ |
| 16 | 16 | ||
| 17 | #include <linux/clk.h> | ||
| 18 | #include <linux/err.h> | 17 | #include <linux/err.h> |
| 19 | #include <linux/gpio.h> | 18 | #include <linux/gpio.h> |
| 20 | #include <linux/init.h> | 19 | #include <linux/init.h> |
| @@ -37,10 +36,9 @@ struct gpio_rcar_priv { · 16 unmodified lines | |||
| 37 | struct platform_device *pdev; | 36 | struct platform_device *pdev; |
| 38 | struct gpio_chip gpio_chip; | 37 | struct gpio_chip gpio_chip; |
| 39 | struct irq_chip irq_chip; | 38 | struct irq_chip irq_chip; |
| 40 | struct clk *clk; | ||
| 41 | unsigned int irq_parent; | 39 | unsigned int irq_parent; |
| 40 | atomic_t wakeup_path; | ||
| 42 | bool has_both_edge_trigger; | 41 | bool has_both_edge_trigger; |
| 43 | bool needs_clk; | ||
| 44 | }; | 42 | }; |
| 45 | 43 | ||
| 46 | #define IOINTSEL 0x00 /* General IO/Interrupt Switching Register */ | 44 | #define IOINTSEL 0x00 /* General IO/Interrupt Switching Register */ |
| @@ -186,13 +184,10 @@ static int gpio_rcar_irq_set_wake(struct irq_data *d, unsigned int on) · 139 unmodified lines | |||
| 186 | } | 184 | } |
| 187 | } | 185 | } |
| 188 | 186 | ||
| 189 | if (!p->clk) | ||
| 190 | return 0; | ||
| 191 | |||
| 192 | if (on) | 187 | if (on) |
| 193 | clk_enable(p->clk); | 188 | atomic_inc(&p->wakeup_path); |
| 194 | else | 189 | else |
| 195 | clk_disable(p->clk); | 190 | atomic_dec(&p->wakeup_path); |
| 196 | 191 | ||
| 197 | return 0; | 192 | return 0; |
| 198 | } | 193 | } |
| @@ -330,17 +325,14 @@ static int gpio_rcar_direction_output(struct gpio_chip *chip, unsigned offset, · 131 unmodified lines | |||
| 330 | 325 | ||
| 331 | struct gpio_rcar_info { | 326 | struct gpio_rcar_info { |
| 332 | bool has_both_edge_trigger; | 327 | bool has_both_edge_trigger; |
| 333 | bool needs_clk; | ||
| 334 | }; | 328 | }; |
| 335 | 329 | ||
| 336 | static const struct gpio_rcar_info gpio_rcar_info_gen1 = { | 330 | static const struct gpio_rcar_info gpio_rcar_info_gen1 = { |
| 337 | .has_both_edge_trigger = false, | 331 | .has_both_edge_trigger = false, |
| 338 | .needs_clk = false, | ||
| 339 | }; | 332 | }; |
| 340 | 333 | ||
| 341 | static const struct gpio_rcar_info gpio_rcar_info_gen2 = { | 334 | static const struct gpio_rcar_info gpio_rcar_info_gen2 = { |
| 342 | .has_both_edge_trigger = true, | 335 | .has_both_edge_trigger = true, |
| 343 | .needs_clk = true, | ||
| 344 | }; | 336 | }; |
| 345 | 337 | ||
| 346 | static const struct of_device_id gpio_rcar_of_table[] = { | 338 | static const struct of_device_id gpio_rcar_of_table[] = { |
| @@ -403,7 +395,6 @@ static int gpio_rcar_parse_dt(struct gpio_rcar_priv *p, unsigned int *npins) · 56 unmodified lines | |||
| 403 | ret = of_parse_phandle_with_fixed_args(np, "gpio-ranges", 3, 0, &args); | 395 | ret = of_parse_phandle_with_fixed_args(np, "gpio-ranges", 3, 0, &args); |
| 404 | *npins = ret == 0 ? args.args[2] : RCAR_MAX_GPIO_PER_BANK; | 396 | *npins = ret == 0 ? args.args[2] : RCAR_MAX_GPIO_PER_BANK; |
| 405 | p->has_both_edge_trigger = info->has_both_edge_trigger; | 397 | p->has_both_edge_trigger = info->has_both_edge_trigger; |
| 406 | p->needs_clk = info->needs_clk; | ||
| 407 | 398 | ||
| 408 | if (*npins == 0 || *npins > RCAR_MAX_GPIO_PER_BANK) { | 399 | if (*npins == 0 || *npins > RCAR_MAX_GPIO_PER_BANK) { |
| 409 | dev_warn(&p->pdev->dev, | 400 | dev_warn(&p->pdev->dev, |
| @@ -440,16 +431,6 @@ static int gpio_rcar_probe(struct platform_device *pdev) · 30 unmodified lines | |||
| 440 | 431 | ||
| 441 | platform_set_drvdata(pdev, p); | 432 | platform_set_drvdata(pdev, p); |
| 442 | 433 | ||
| 443 | p->clk = devm_clk_get(dev, NULL); | ||
| 444 | if (IS_ERR(p->clk)) { | ||
| 445 | if (p->needs_clk) { | ||
| 446 | dev_err(dev, "unable to get clock\n"); | ||
| 447 | ret = PTR_ERR(p->clk); | ||
| 448 | goto err0; | ||
| 449 | } | ||
| 450 | p->clk = NULL; | ||
| 451 | } | ||
| 452 | |||
| 453 | pm_runtime_enable(dev); | 434 | pm_runtime_enable(dev); |
| 454 | 435 | ||
| 455 | irq = platform_get_resource(pdev, IORESOURCE_IRQ, 0); | 436 | irq = platform_get_resource(pdev, IORESOURCE_IRQ, 0); |
| @@ -531,11 +512,24 @@ static int gpio_rcar_remove(struct platform_device *pdev) · 75 unmodified lines | |||
| 531 | return 0; | 512 | return 0; |
| 532 | } | 513 | } |
| 533 | 514 | ||
| 515 | static int __maybe_unused gpio_rcar_suspend(struct device *dev) | ||
| 516 | { | ||
| 517 | struct gpio_rcar_priv *p = dev_get_drvdata(dev); | ||
| 518 | |||
| 519 | if (atomic_read(&p->wakeup_path)) | ||
| 520 | device_set_wakeup_path(dev); | ||
| 521 | |||
| 522 | return 0; | ||
| 523 | } | ||
| 524 | |||
| 525 | static SIMPLE_DEV_PM_OPS(gpio_rcar_pm_ops, gpio_rcar_suspend, NULL); | ||
| 526 | |||
| 534 | static struct platform_driver gpio_rcar_device_driver = { | 527 | static struct platform_driver gpio_rcar_device_driver = { |
| 535 | .probe = gpio_rcar_probe, | 528 | .probe = gpio_rcar_probe, |
| 536 | .remove = gpio_rcar_remove, | 529 | .remove = gpio_rcar_remove, |
| 537 | .driver = { | 530 | .driver = { |
| 538 | .name = "gpio_rcar", | 531 | .name = "gpio_rcar", |
| 532 | .pm = &gpio_rcar_pm_ops, | ||
| 539 | .of_match_table = of_match_ptr(gpio_rcar_of_table), | 533 | .of_match_table = of_match_ptr(gpio_rcar_of_table), |
| 540 | } | 534 | } |
| 541 | }; | 535 | }; |
| @@ -241,6 +241,19 @@ struct gpio_desc *of_find_gpio(struct device *dev, const char *con_id, · 240 unmodified lines | |||
| 241 | 241 | ||
| 242 | desc = of_get_named_gpiod_flags(dev->of_node, prop_name, idx, | 242 | desc = of_get_named_gpiod_flags(dev->of_node, prop_name, idx, |
| 243 | &of_flags); | 243 | &of_flags); |
| 244 | /* | ||
| 245 | * -EPROBE_DEFER in our case means that we found a | ||
| 246 | * valid GPIO property, but no controller has been | ||
| 247 | * registered so far. | ||
| 248 | * | ||
| 249 | * This means we don't need to look any further for | ||
| 250 | * alternate name conventions, and we should really | ||
| 251 | * preserve the return code for our user to be able to | ||
| 252 | * retry probing later. | ||
| 253 | */ | ||
| 254 | if (IS_ERR(desc) && PTR_ERR(desc) == -EPROBE_DEFER) | ||
| 255 | return desc; | ||
| 256 | |||
| 244 | if (!IS_ERR(desc) || (PTR_ERR(desc) != -ENOENT)) | 257 | if (!IS_ERR(desc) || (PTR_ERR(desc) != -ENOENT)) |
| 245 | break; | 258 | break; |
| 246 | } | 259 | } |
| @@ -250,7 +263,7 @@ struct gpio_desc *of_find_gpio(struct device *dev, const char *con_id, · 3 unmodified lines | |||
| 250 | desc = of_find_spi_gpio(dev, con_id, &of_flags); | 263 | desc = of_find_spi_gpio(dev, con_id, &of_flags); |
| 251 | 264 | ||
| 252 | /* Special handling for regulator GPIOs if used */ | 265 | /* Special handling for regulator GPIOs if used */ |
| 253 | if (IS_ERR(desc)) | 266 | if (IS_ERR(desc) && PTR_ERR(desc) != -EPROBE_DEFER) |
| 254 | desc = of_find_regulator_gpio(dev, con_id, &of_flags); | 267 | desc = of_find_regulator_gpio(dev, con_id, &of_flags); |
| 255 | 268 | ||
| 256 | if (IS_ERR(desc)) | 269 | if (IS_ERR(desc)) |
| @@ -25,7 +25,6 @@ · 24 unmodified lines | |||
| 25 | #define _ACP_GFX_IF_H | 25 | #define _ACP_GFX_IF_H |
| 26 | 26 | ||
| 27 | #include <linux/types.h> | 27 | #include <linux/types.h> |
| 28 | #include "cgs_linux.h" | ||
| 29 | #include "cgs_common.h" | 28 | #include "cgs_common.h" |
| 30 | 29 | ||
| 31 | int amd_acp_hw_init(struct cgs_device *cgs_device, | 30 | int amd_acp_hw_init(struct cgs_device *cgs_device, |
| @@ -26,6 +26,7 @@ · 25 unmodified lines | |||
| 26 | #define AMDGPU_AMDKFD_H_INCLUDED | 26 | #define AMDGPU_AMDKFD_H_INCLUDED |
| 27 | 27 | ||
| 28 | #include <linux/types.h> | 28 | #include <linux/types.h> |
| 29 | #include <linux/mm.h> | ||
| 29 | #include <linux/mmu_context.h> | 30 | #include <linux/mmu_context.h> |
| 30 | #include <kgd_kfd_interface.h> | 31 | #include <kgd_kfd_interface.h> |
| 31 | #include <drm/ttm/ttm_execbuf_util.h> | 32 | #include <drm/ttm/ttm_execbuf_util.h> |
| @@ -92,27 +93,6 @@ struct amdkfd_process_info { · 60 unmodified lines | |||
| 92 | struct amdgpu_amdkfd_fence *eviction_fence; | 93 | struct amdgpu_amdkfd_fence *eviction_fence; |
| 93 | }; | 94 | }; |
| 94 | 95 | ||
| 95 | /* struct amdkfd_vm - | ||
| 96 | * For Memory Eviction KGD requires a mechanism to keep track of all KFD BOs | ||
| 97 | * belonging to a KFD process. All the VMs belonging to the same process point | ||
| 98 | * to the same amdkfd_process_info. | ||
| 99 | */ | ||
| 100 | struct amdkfd_vm { | ||
| 101 | /* Keep base as the first parameter for pointer compatibility between | ||
| 102 | * amdkfd_vm and amdgpu_vm. | ||
| 103 | */ | ||
| 104 | struct amdgpu_vm base; | ||
| 105 | |||
| 106 | /* List node in amdkfd_process_info.vm_list_head*/ | ||
| 107 | struct list_head vm_list_node; | ||
| 108 | |||
| 109 | struct amdgpu_device *adev; | ||
| 110 | /* Points to the KFD process VM info*/ | ||
| 111 | struct amdkfd_process_info *process_info; | ||
| 112 | |||
| 113 | uint64_t pd_phys_addr; | ||
| 114 | }; | ||
| 115 | |||
| 116 | int amdgpu_amdkfd_init(void); | 96 | int amdgpu_amdkfd_init(void); |
| 117 | void amdgpu_amdkfd_fini(void); | 97 | void amdgpu_amdkfd_fini(void); |
| 118 | 98 | ||
| @@ -165,6 +145,12 @@ uint64_t amdgpu_amdkfd_get_vram_usage(struct kgd_dev *kgd); · 46 unmodified lines | |||
| 165 | int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, | 145 | int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, |
| 166 | void **process_info, | 146 | void **process_info, |
| 167 | struct dma_fence **ef); | 147 | struct dma_fence **ef); |
| 148 | int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd, | ||
| 149 | struct file *filp, | ||
| 150 | void **vm, void **process_info, | ||
| 151 | struct dma_fence **ef); | ||
| 152 | void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev, | ||
| 153 | struct amdgpu_vm *vm); | ||
| 168 | void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm); | 154 | void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm); |
| 169 | uint32_t amdgpu_amdkfd_gpuvm_get_process_page_dir(void *vm); | 155 | uint32_t amdgpu_amdkfd_gpuvm_get_process_page_dir(void *vm); |
| 170 | int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( | 156 | int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( |
| @@ -205,6 +205,7 @@ static const struct kfd2kgd_calls kfd2kgd = { · 204 unmodified lines | |||
| 205 | .get_cu_info = get_cu_info, | 205 | .get_cu_info = get_cu_info, |
| 206 | .get_vram_usage = amdgpu_amdkfd_get_vram_usage, | 206 | .get_vram_usage = amdgpu_amdkfd_get_vram_usage, |
| 207 | .create_process_vm = amdgpu_amdkfd_gpuvm_create_process_vm, | 207 | .create_process_vm = amdgpu_amdkfd_gpuvm_create_process_vm, |
| 208 | .acquire_process_vm = amdgpu_amdkfd_gpuvm_acquire_process_vm, | ||
| 208 | .destroy_process_vm = amdgpu_amdkfd_gpuvm_destroy_process_vm, | 209 | .destroy_process_vm = amdgpu_amdkfd_gpuvm_destroy_process_vm, |
| 209 | .get_process_page_dir = amdgpu_amdkfd_gpuvm_get_process_page_dir, | 210 | .get_process_page_dir = amdgpu_amdkfd_gpuvm_get_process_page_dir, |
| 210 | .set_vm_context_page_table_base = set_vm_context_page_table_base, | 211 | .set_vm_context_page_table_base = set_vm_context_page_table_base, |
| @@ -165,6 +165,7 @@ static const struct kfd2kgd_calls kfd2kgd = { · 164 unmodified lines | |||
| 165 | .get_cu_info = get_cu_info, | 165 | .get_cu_info = get_cu_info, |
| 166 | .get_vram_usage = amdgpu_amdkfd_get_vram_usage, | 166 | .get_vram_usage = amdgpu_amdkfd_get_vram_usage, |
| 167 | .create_process_vm = amdgpu_amdkfd_gpuvm_create_process_vm, | 167 | .create_process_vm = amdgpu_amdkfd_gpuvm_create_process_vm, |
| 168 | .acquire_process_vm = amdgpu_amdkfd_gpuvm_acquire_process_vm, | ||
| 168 | .destroy_process_vm = amdgpu_amdkfd_gpuvm_destroy_process_vm, | 169 | .destroy_process_vm = amdgpu_amdkfd_gpuvm_destroy_process_vm, |
| 169 | .get_process_page_dir = amdgpu_amdkfd_gpuvm_get_process_page_dir, | 170 | .get_process_page_dir = amdgpu_amdkfd_gpuvm_get_process_page_dir, |
| 170 | .set_vm_context_page_table_base = set_vm_context_page_table_base, | 171 | .set_vm_context_page_table_base = set_vm_context_page_table_base, |
| @@ -333,9 +333,9 @@ static int amdgpu_amdkfd_validate(void *param, struct amdgpu_bo *bo) · 332 unmodified lines | |||
| 333 | * again. Page directories are only updated after updating page | 333 | * again. Page directories are only updated after updating page |
| 334 | * tables. | 334 | * tables. |
| 335 | */ | 335 | */ |
| 336 | static int vm_validate_pt_pd_bos(struct amdkfd_vm *vm) | 336 | static int vm_validate_pt_pd_bos(struct amdgpu_vm *vm) |
| 337 | { | 337 | { |
| 338 | struct amdgpu_bo *pd = vm->base.root.base.bo; | 338 | struct amdgpu_bo *pd = vm->root.base.bo; |
| 339 | struct amdgpu_device *adev = amdgpu_ttm_adev(pd->tbo.bdev); | 339 | struct amdgpu_device *adev = amdgpu_ttm_adev(pd->tbo.bdev); |
| 340 | struct amdgpu_vm_parser param; | 340 | struct amdgpu_vm_parser param; |
| 341 | uint64_t addr, flags = AMDGPU_PTE_VALID; | 341 | uint64_t addr, flags = AMDGPU_PTE_VALID; |
| @@ -344,7 +344,7 @@ static int vm_validate_pt_pd_bos(struct amdkfd_vm *vm) · 2 unmodified lines | |||
| 344 | param.domain = AMDGPU_GEM_DOMAIN_VRAM; | 344 | param.domain = AMDGPU_GEM_DOMAIN_VRAM; |
| 345 | param.wait = false; | 345 | param.wait = false; |
| 346 | 346 | ||
| 347 | ret = amdgpu_vm_validate_pt_bos(adev, &vm->base, amdgpu_amdkfd_validate, | 347 | ret = amdgpu_vm_validate_pt_bos(adev, vm, amdgpu_amdkfd_validate, |
| 348 | ¶m); | 348 | ¶m); |
| 349 | if (ret) { | 349 | if (ret) { |
| 350 | pr_err("amdgpu: failed to validate PT BOs\n"); | 350 | pr_err("amdgpu: failed to validate PT BOs\n"); |
| @@ -357,11 +357,11 @@ static int vm_validate_pt_pd_bos(struct amdkfd_vm *vm) · 6 unmodified lines | |||
| 357 | return ret; | 357 | return ret; |
| 358 | } | 358 | } |
| 359 | 359 | ||
| 360 | addr = amdgpu_bo_gpu_offset(vm->base.root.base.bo); | 360 | addr = amdgpu_bo_gpu_offset(vm->root.base.bo); |
| 361 | amdgpu_gmc_get_vm_pde(adev, -1, &addr, &flags); | 361 | amdgpu_gmc_get_vm_pde(adev, -1, &addr, &flags); |
| 362 | vm->pd_phys_addr = addr; | 362 | vm->pd_phys_addr = addr; |
| 363 | 363 | ||
| 364 | if (vm->base.use_cpu_for_update) { | 364 | if (vm->use_cpu_for_update) { |
| 365 | ret = amdgpu_bo_kmap(pd, NULL); | 365 | ret = amdgpu_bo_kmap(pd, NULL); |
| 366 | if (ret) { | 366 | if (ret) { |
| 367 | pr_err("amdgpu: failed to kmap PD, ret=%d\n", ret); | 367 | pr_err("amdgpu: failed to kmap PD, ret=%d\n", ret); |
| @@ -415,14 +415,12 @@ static int vm_update_pds(struct amdgpu_vm *vm, struct amdgpu_sync *sync) · 47 unmodified lines | |||
| 415 | * 4a. Validate new page tables and directories | 415 | * 4a. Validate new page tables and directories |
| 416 | */ | 416 | */ |
| 417 | static int add_bo_to_vm(struct amdgpu_device *adev, struct kgd_mem *mem, | 417 | static int add_bo_to_vm(struct amdgpu_device *adev, struct kgd_mem *mem, |
| 418 | struct amdgpu_vm *avm, bool is_aql, | 418 | struct amdgpu_vm *vm, bool is_aql, |
| 419 | struct kfd_bo_va_list **p_bo_va_entry) | 419 | struct kfd_bo_va_list **p_bo_va_entry) |
| 420 | { | 420 | { |
| 421 | int ret; | 421 | int ret; |
| 422 | struct kfd_bo_va_list *bo_va_entry; | 422 | struct kfd_bo_va_list *bo_va_entry; |
| 423 | struct amdkfd_vm *kvm = container_of(avm, | 423 | struct amdgpu_bo *pd = vm->root.base.bo; |
| 424 | struct amdkfd_vm, base); | ||
| 425 | struct amdgpu_bo *pd = avm->root.base.bo; | ||
| 426 | struct amdgpu_bo *bo = mem->bo; | 424 | struct amdgpu_bo *bo = mem->bo; |
| 427 | uint64_t va = mem->va; | 425 | uint64_t va = mem->va; |
| 428 | struct list_head *list_bo_va = &mem->bo_va_list; | 426 | struct list_head *list_bo_va = &mem->bo_va_list; |
| @@ -441,10 +439,10 @@ static int add_bo_to_vm(struct amdgpu_device *adev, struct kgd_mem *mem, · 12 unmodified lines | |||
| 441 | return -ENOMEM; | 439 | return -ENOMEM; |
| 442 | 440 | ||
| 443 | pr_debug("\t add VA 0x%llx - 0x%llx to vm %p\n", va, | 441 | pr_debug("\t add VA 0x%llx - 0x%llx to vm %p\n", va, |
| 444 | va + bo_size, avm); | 442 | va + bo_size, vm); |
| 445 | 443 | ||
| 446 | /* Add BO to VM internal data structures*/ | 444 | /* Add BO to VM internal data structures*/ |
| 447 | bo_va_entry->bo_va = amdgpu_vm_bo_add(adev, avm, bo); | 445 | bo_va_entry->bo_va = amdgpu_vm_bo_add(adev, vm, bo); |
| 448 | if (!bo_va_entry->bo_va) { | 446 | if (!bo_va_entry->bo_va) { |
| 449 | ret = -EINVAL; | 447 | ret = -EINVAL; |
| 450 | pr_err("Failed to add BO object to VM. ret == %d\n", | 448 | pr_err("Failed to add BO object to VM. ret == %d\n", |
| @@ -467,28 +465,28 @@ static int add_bo_to_vm(struct amdgpu_device *adev, struct kgd_mem *mem, · 16 unmodified lines | |||
| 467 | * fence, so remove it temporarily. | 465 | * fence, so remove it temporarily. |
| 468 | */ | 466 | */ |
| 469 | amdgpu_amdkfd_remove_eviction_fence(pd, | 467 | amdgpu_amdkfd_remove_eviction_fence(pd, |
| 470 | kvm->process_info->eviction_fence, | 468 | vm->process_info->eviction_fence, |
| 471 | NULL, NULL); | 469 | NULL, NULL); |
| 472 | 470 | ||
| 473 | ret = amdgpu_vm_alloc_pts(adev, avm, va, amdgpu_bo_size(bo)); | 471 | ret = amdgpu_vm_alloc_pts(adev, vm, va, amdgpu_bo_size(bo)); |
| 474 | if (ret) { | 472 | if (ret) { |
| 475 | pr_err("Failed to allocate pts, err=%d\n", ret); | 473 | pr_err("Failed to allocate pts, err=%d\n", ret); |
| 476 | goto err_alloc_pts; | 474 | goto err_alloc_pts; |
| 477 | } | 475 | } |
| 478 | 476 | ||
| 479 | ret = vm_validate_pt_pd_bos(kvm); | 477 | ret = vm_validate_pt_pd_bos(vm); |
| 480 | if (ret) { | 478 | if (ret) { |
| 481 | pr_err("validate_pt_pd_bos() failed\n"); | 479 | pr_err("validate_pt_pd_bos() failed\n"); |
| 482 | goto err_alloc_pts; | 480 | goto err_alloc_pts; |
| 483 | } | 481 | } |
| 484 | 482 | ||
| 485 | /* Add the eviction fence back */ | 483 | /* Add the eviction fence back */ |
| 486 | amdgpu_bo_fence(pd, &kvm->process_info->eviction_fence->base, true); | 484 | amdgpu_bo_fence(pd, &vm->process_info->eviction_fence->base, true); |
| 487 | 485 | ||
| 488 | return 0; | 486 | return 0; |
| 489 | 487 | ||
| 490 | err_alloc_pts: | 488 | err_alloc_pts: |
| 491 | amdgpu_bo_fence(pd, &kvm->process_info->eviction_fence->base, true); | 489 | amdgpu_bo_fence(pd, &vm->process_info->eviction_fence->base, true); |
| 492 | amdgpu_vm_bo_rmv(adev, bo_va_entry->bo_va); | 490 | amdgpu_vm_bo_rmv(adev, bo_va_entry->bo_va); |
| 493 | list_del(&bo_va_entry->bo_list); | 491 | list_del(&bo_va_entry->bo_list); |
| 494 | err_vmadd: | 492 | err_vmadd: |
| @@ -703,7 +701,6 @@ static int unmap_bo_from_gpuvm(struct amdgpu_device *adev, · 208 unmodified lines | |||
| 703 | { | 701 | { |
| 704 | struct amdgpu_bo_va *bo_va = entry->bo_va; | 702 | struct amdgpu_bo_va *bo_va = entry->bo_va; |
| 705 | struct amdgpu_vm *vm = bo_va->base.vm; | 703 | struct amdgpu_vm *vm = bo_va->base.vm; |
| 706 | struct amdkfd_vm *kvm = container_of(vm, struct amdkfd_vm, base); | ||
| 707 | struct amdgpu_bo *pd = vm->root.base.bo; | 704 | struct amdgpu_bo *pd = vm->root.base.bo; |
| 708 | 705 | ||
| 709 | /* Remove eviction fence from PD (and thereby from PTs too as | 706 | /* Remove eviction fence from PD (and thereby from PTs too as |
| @@ -713,14 +710,14 @@ static int unmap_bo_from_gpuvm(struct amdgpu_device *adev, · 3 unmodified lines | |||
| 713 | * trigger the eviction fence. | 710 | * trigger the eviction fence. |
| 714 | */ | 711 | */ |
| 715 | amdgpu_amdkfd_remove_eviction_fence(pd, | 712 | amdgpu_amdkfd_remove_eviction_fence(pd, |
| 716 | kvm->process_info->eviction_fence, | 713 | vm->process_info->eviction_fence, |
| 717 | NULL, NULL); | 714 | NULL, NULL); |
| 718 | amdgpu_vm_bo_unmap(adev, bo_va, entry->va); | 715 | amdgpu_vm_bo_unmap(adev, bo_va, entry->va); |
| 719 | 716 | ||
| 720 | amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update); | 717 | amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update); |
| 721 | 718 | ||
| 722 | /* Add the eviction fence back */ | 719 | /* Add the eviction fence back */ |
| 723 | amdgpu_bo_fence(pd, &kvm->process_info->eviction_fence->base, true); | 720 | amdgpu_bo_fence(pd, &vm->process_info->eviction_fence->base, true); |
| 724 | 721 | ||
| 725 | sync_vm_fence(adev, sync, bo_va->last_pt_update); | 722 | sync_vm_fence(adev, sync, bo_va->last_pt_update); |
| 726 | 723 | ||
| @@ -780,7 +777,7 @@ update_gpuvm_pte_failed: · 53 unmodified lines | |||
| 780 | 777 | ||
| 781 | static int process_validate_vms(struct amdkfd_process_info *process_info) | 778 | static int process_validate_vms(struct amdkfd_process_info *process_info) |
| 782 | { | 779 | { |
| 783 | struct amdkfd_vm *peer_vm; | 780 | struct amdgpu_vm *peer_vm; |
| 784 | int ret; | 781 | int ret; |
| 785 | 782 | ||
| 786 | list_for_each_entry(peer_vm, &process_info->vm_list_head, | 783 | list_for_each_entry(peer_vm, &process_info->vm_list_head, |
| @@ -796,12 +793,12 @@ static int process_validate_vms(struct amdkfd_process_info *process_info) · 9 unmodified lines | |||
| 796 | static int process_update_pds(struct amdkfd_process_info *process_info, | 793 | static int process_update_pds(struct amdkfd_process_info *process_info, |
| 797 | struct amdgpu_sync *sync) | 794 | struct amdgpu_sync *sync) |
| 798 | { | 795 | { |
| 799 | struct amdkfd_vm *peer_vm; | 796 | struct amdgpu_vm *peer_vm; |
| 800 | int ret; | 797 | int ret; |
| 801 | 798 | ||
| 802 | list_for_each_entry(peer_vm, &process_info->vm_list_head, | 799 | list_for_each_entry(peer_vm, &process_info->vm_list_head, |
| 803 | vm_list_node) { | 800 | vm_list_node) { |
| 804 | ret = vm_update_pds(&peer_vm->base, sync); | 801 | ret = vm_update_pds(peer_vm, sync); |
| 805 | if (ret) | 802 | if (ret) |
| 806 | return ret; | 803 | return ret; |
| 807 | } | 804 | } |
| @@ -809,33 +806,16 @@ static int process_update_pds(struct amdkfd_process_info *process_info, · 1 unmodified line | |||
| 809 | return 0; | 806 | return 0; |
| 810 | } | 807 | } |
| 811 | 808 | ||
| 812 | int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, | 809 | static int init_kfd_vm(struct amdgpu_vm *vm, void **process_info, |
| 813 | void **process_info, | 810 | struct dma_fence **ef) |
| 814 | struct dma_fence **ef) | ||
| 815 | { | 811 | { |
| 812 | struct amdkfd_process_info *info = NULL; | ||
| 816 | int ret; | 813 | int ret; |
| 817 | struct amdkfd_vm *new_vm; | ||
| 818 | struct amdkfd_process_info *info; | ||
| 819 | struct amdgpu_device *adev = get_amdgpu_device(kgd); | ||
| 820 | |||
| 821 | new_vm = kzalloc(sizeof(*new_vm), GFP_KERNEL); | ||
| 822 | if (!new_vm) | ||
| 823 | return -ENOMEM; | ||
| 824 | |||
| 825 | /* Initialize the VM context, allocate the page directory and zero it */ | ||
| 826 | ret = amdgpu_vm_init(adev, &new_vm->base, AMDGPU_VM_CONTEXT_COMPUTE, 0); | ||
| 827 | if (ret) { | ||
| 828 | pr_err("Failed init vm ret %d\n", ret); | ||
| 829 | goto vm_init_fail; | ||
| 830 | } | ||
| 831 | new_vm->adev = adev; | ||
| 832 | 814 | ||
| 833 | if (!*process_info) { | 815 | if (!*process_info) { |
| 834 | info = kzalloc(sizeof(*info), GFP_KERNEL); | 816 | info = kzalloc(sizeof(*info), GFP_KERNEL); |
| 835 | if (!info) { | 817 | if (!info) |
| 836 | ret = -ENOMEM; | 818 | return -ENOMEM; |
| 837 | goto alloc_process_info_fail; | ||
| 838 | } | ||
| 839 | 819 | ||
| 840 | mutex_init(&info->lock); | 820 | mutex_init(&info->lock); |
| 841 | INIT_LIST_HEAD(&info->vm_list_head); | 821 | INIT_LIST_HEAD(&info->vm_list_head); |
| @@ -846,6 +826,7 @@ int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, · 4 unmodified lines | |||
| 846 | current->mm); | 826 | current->mm); |
| 847 | if (!info->eviction_fence) { | 827 | if (!info->eviction_fence) { |
| 848 | pr_err("Failed to create eviction fence\n"); | 828 | pr_err("Failed to create eviction fence\n"); |
| 829 | ret = -ENOMEM; | ||
| 849 | goto create_evict_fence_fail; | 830 | goto create_evict_fence_fail; |
| 850 | } | 831 | } |
| 851 | 832 | ||
| @@ -853,57 +834,137 @@ int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, · 1 unmodified line | |||
| 853 | *ef = dma_fence_get(&info->eviction_fence->base); | 834 | *ef = dma_fence_get(&info->eviction_fence->base); |
| 854 | } | 835 | } |
| 855 | 836 | ||
| 856 | new_vm->process_info = *process_info; | 837 | vm->process_info = *process_info; |
| 857 | |||
| 858 | mutex_lock(&new_vm->process_info->lock); | ||
| 859 | list_add_tail(&new_vm->vm_list_node, | ||
| 860 | &(new_vm->process_info->vm_list_head)); | ||
| 861 | new_vm->process_info->n_vms++; | ||
| 862 | mutex_unlock(&new_vm->process_info->lock); | ||
| 863 | 838 | ||
| 864 | *vm = (void *) new_vm; | 839 | /* Validate page directory and attach eviction fence */ |
| 840 | ret = amdgpu_bo_reserve(vm->root.base.bo, true); | ||
| 841 | if (ret) | ||
| 842 | goto reserve_pd_fail; | ||
| 843 | ret = vm_validate_pt_pd_bos(vm); | ||
| 844 | if (ret) { | ||
| 845 | pr_err("validate_pt_pd_bos() failed\n"); | ||
| 846 | goto validate_pd_fail; | ||
| 847 | } | ||
| 848 | ret = ttm_bo_wait(&vm->root.base.bo->tbo, false, false); | ||
| 849 | if (ret) | ||
| 850 | goto wait_pd_fail; | ||
| 851 | amdgpu_bo_fence(vm->root.base.bo, | ||
| 852 | &vm->process_info->eviction_fence->base, true); | ||
| 853 | amdgpu_bo_unreserve(vm->root.base.bo); | ||
| 854 | |||
| 855 | /* Update process info */ | ||
| 856 | mutex_lock(&vm->process_info->lock); | ||
| 857 | list_add_tail(&vm->vm_list_node, | ||
| 858 | &(vm->process_info->vm_list_head)); | ||
| 859 | vm->process_info->n_vms++; | ||
| 860 | mutex_unlock(&vm->process_info->lock); | ||
| 865 | 861 | ||
| 866 | pr_debug("Created process vm %p\n", *vm); | 862 | return 0; |
| 867 | 863 | ||
| 864 | wait_pd_fail: | ||
| 865 | validate_pd_fail: | ||
| 866 | amdgpu_bo_unreserve(vm->root.base.bo); | ||
| 867 | reserve_pd_fail: | ||
| 868 | vm->process_info = NULL; | ||
| 869 | if (info) { | ||
| 870 | /* Two fence references: one in info and one in *ef */ | ||
| 871 | dma_fence_put(&info->eviction_fence->base); | ||
| 872 | dma_fence_put(*ef); | ||
| 873 | *ef = NULL; | ||
| 874 | *process_info = NULL; | ||
| 875 | create_evict_fence_fail: | ||
| 876 | mutex_destroy(&info->lock); | ||
| 877 | kfree(info); | ||
| 878 | } | ||
| 868 | return ret; | 879 | return ret; |
| 880 | } | ||
| 869 | 881 | ||
| 870 | create_evict_fence_fail: | 882 | int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, |
| 871 | mutex_destroy(&info->lock); | 883 | void **process_info, |
| 872 | kfree(info); | 884 | struct dma_fence **ef) |
| 873 | alloc_process_info_fail: | 885 | { |
| 874 | amdgpu_vm_fini(adev, &new_vm->base); | 886 | struct amdgpu_device *adev = get_amdgpu_device(kgd); |
| 875 | vm_init_fail: | 887 | struct amdgpu_vm *new_vm; |
| 888 | int ret; | ||
| 889 | |||
| 890 | new_vm = kzalloc(sizeof(*new_vm), GFP_KERNEL); | ||
| 891 | if (!new_vm) | ||
| 892 | return -ENOMEM; | ||
| 893 | |||
| 894 | /* Initialize AMDGPU part of the VM */ | ||
| 895 | ret = amdgpu_vm_init(adev, new_vm, AMDGPU_VM_CONTEXT_COMPUTE, 0); | ||
| 896 | if (ret) { | ||
| 897 | pr_err("Failed init vm ret %d\n", ret); | ||
| 898 | goto amdgpu_vm_init_fail; | ||
| 899 | } | ||
| 900 | |||
| 901 | /* Initialize KFD part of the VM and process info */ | ||
| 902 | ret = init_kfd_vm(new_vm, process_info, ef); | ||
| 903 | if (ret) | ||
| 904 | goto init_kfd_vm_fail; | ||
| 905 | |||
| 906 | *vm = (void *) new_vm; | ||
| 907 | |||
| 908 | return 0; | ||
| 909 | |||
| 910 | init_kfd_vm_fail: | ||
| 911 | amdgpu_vm_fini(adev, new_vm); | ||
| 912 | amdgpu_vm_init_fail: | ||
| 876 | kfree(new_vm); | 913 | kfree(new_vm); |
| 877 | return ret; | 914 | return ret; |
| 878 | |||
| 879 | } | 915 | } |
| 880 | 916 | ||
| 881 | void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm) | 917 | int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd, |
| 918 | struct file *filp, | ||
| 919 | void **vm, void **process_info, | ||
| 920 | struct dma_fence **ef) | ||
| 882 | { | 921 | { |
| 883 | struct amdgpu_device *adev = get_amdgpu_device(kgd); | 922 | struct amdgpu_device *adev = get_amdgpu_device(kgd); |
| 884 | struct amdkfd_vm *kfd_vm = (struct amdkfd_vm *) vm; | 923 | struct drm_file *drm_priv = filp->private_data; |
| 885 | struct amdgpu_vm *avm = &kfd_vm->base; | 924 | struct amdgpu_fpriv *drv_priv = drm_priv->driver_priv; |
| 886 | struct amdgpu_bo *pd; | 925 | struct amdgpu_vm *avm = &drv_priv->vm; |
| 887 | struct amdkfd_process_info *process_info; | 926 | int ret; |
| 888 | 927 | ||
| 889 | if (WARN_ON(!kgd || !vm)) | 928 | /* Already a compute VM? */ |
| 929 | if (avm->process_info) | ||
| 930 | return -EINVAL; | ||
| 931 | |||
| 932 | /* Convert VM into a compute VM */ | ||
| 933 | ret = amdgpu_vm_make_compute(adev, avm); | ||
| 934 | if (ret) | ||
| 935 | return ret; | ||
| 936 | |||
| 937 | /* Initialize KFD part of the VM and process info */ | ||
| 938 | ret = init_kfd_vm(avm, process_info, ef); | ||
| 939 | if (ret) | ||
| 940 | return ret; | ||
| 941 | |||
| 942 | *vm = (void *)avm; | ||
| 943 | |||
| 944 | return 0; | ||
| 945 | } | ||
| 946 | |||
| 947 | void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev, | ||
| 948 | struct amdgpu_vm *vm) | ||
| 949 | { | ||
| 950 | struct amdkfd_process_info *process_info = vm->process_info; | ||
| 951 | struct amdgpu_bo *pd = vm->root.base.bo; | ||
| 952 | |||
| 953 | if (!process_info) | ||
| 890 | return; | 954 | return; |
| 891 | 955 | ||
| 892 | pr_debug("Destroying process vm %p\n", vm); | ||
| 893 | /* Release eviction fence from PD */ | 956 | /* Release eviction fence from PD */ |
| 894 | pd = avm->root.base.bo; | ||
| 895 | amdgpu_bo_reserve(pd, false); | 957 | amdgpu_bo_reserve(pd, false); |
| 896 | amdgpu_bo_fence(pd, NULL, false); | 958 | amdgpu_bo_fence(pd, NULL, false); |
| 897 | amdgpu_bo_unreserve(pd); | 959 | amdgpu_bo_unreserve(pd); |
| 898 | 960 | ||
| 899 | process_info = kfd_vm->process_info; | 961 | /* Update process info */ |
| 900 | |||
| 901 | mutex_lock(&process_info->lock); | 962 | mutex_lock(&process_info->lock); |
| 902 | process_info->n_vms--; | 963 | process_info->n_vms--; |
| 903 | list_del(&kfd_vm->vm_list_node); | 964 | list_del(&vm->vm_list_node); |
| 904 | mutex_unlock(&process_info->lock); | 965 | mutex_unlock(&process_info->lock); |
| 905 | 966 | ||
| 906 | /* Release per-process resources */ | 967 | /* Release per-process resources when last compute VM is destroyed */ |
| 907 | if (!process_info->n_vms) { | 968 | if (!process_info->n_vms) { |
| 908 | WARN_ON(!list_empty(&process_info->kfd_bo_list)); | 969 | WARN_ON(!list_empty(&process_info->kfd_bo_list)); |
| 909 | 970 | ||
| @@ -911,6 +972,17 @@ void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm) · 1 unmodified line | |||
| 911 | mutex_destroy(&process_info->lock); | 972 | mutex_destroy(&process_info->lock); |
| 912 | kfree(process_info); | 973 | kfree(process_info); |
| 913 | } | 974 | } |
| 975 | } | ||
| 976 | |||
| 977 | void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm) | ||
| 978 | { | ||
| 979 | struct amdgpu_device *adev = get_amdgpu_device(kgd); | ||
| 980 | struct amdgpu_vm *avm = (struct amdgpu_vm *)vm; | ||
| 981 | |||
| 982 | if (WARN_ON(!kgd || !vm)) | ||
| 983 | return; | ||
| 984 | |||
| 985 | pr_debug("Destroying process vm %p\n", vm); | ||
| 914 | 986 | ||
| 915 | /* Release the VM context */ | 987 | /* Release the VM context */ |
| 916 | amdgpu_vm_fini(adev, avm); | 988 | amdgpu_vm_fini(adev, avm); |
| @@ -919,7 +991,7 @@ void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm) · 2 unmodified lines | |||
| 919 | 991 | ||
| 920 | uint32_t amdgpu_amdkfd_gpuvm_get_process_page_dir(void *vm) | 992 | uint32_t amdgpu_amdkfd_gpuvm_get_process_page_dir(void *vm) |
| 921 | { | 993 | { |
| 922 | struct amdkfd_vm *avm = (struct amdkfd_vm *)vm; | 994 | struct amdgpu_vm *avm = (struct amdgpu_vm *)vm; |
| 923 | 995 | ||
| 924 | return avm->pd_phys_addr >> AMDGPU_GPU_PAGE_SHIFT; | 996 | return avm->pd_phys_addr >> AMDGPU_GPU_PAGE_SHIFT; |
| 925 | } | 997 | } |
| @@ -930,7 +1002,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( · 4 unmodified lines | |||
| 930 | uint64_t *offset, uint32_t flags) | 1002 | uint64_t *offset, uint32_t flags) |
| 931 | { | 1003 | { |
| 932 | struct amdgpu_device *adev = get_amdgpu_device(kgd); | 1004 | struct amdgpu_device *adev = get_amdgpu_device(kgd); |
| 933 | struct amdkfd_vm *kfd_vm = (struct amdkfd_vm *)vm; | 1005 | struct amdgpu_vm *avm = (struct amdgpu_vm *)vm; |
| 934 | struct amdgpu_bo *bo; | 1006 | struct amdgpu_bo *bo; |
| 935 | int byte_align; | 1007 | int byte_align; |
| 936 | u32 alloc_domain; | 1008 | u32 alloc_domain; |
| @@ -1010,8 +1082,8 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( · 73 unmodified lines | |||
| 1010 | (*mem)->va = va; | 1082 | (*mem)->va = va; |
| 1011 | (*mem)->domain = alloc_domain; | 1083 | (*mem)->domain = alloc_domain; |
| 1012 | (*mem)->mapped_to_gpu_memory = 0; | 1084 | (*mem)->mapped_to_gpu_memory = 0; |
| 1013 | (*mem)->process_info = kfd_vm->process_info; | 1085 | (*mem)->process_info = avm->process_info; |
| 1014 | add_kgd_mem_to_kfd_bo_list(*mem, kfd_vm->process_info); | 1086 | add_kgd_mem_to_kfd_bo_list(*mem, avm->process_info); |
| 1015 | 1087 | ||
| 1016 | if (offset) | 1088 | if (offset) |
| 1017 | *offset = amdgpu_bo_mmap_offset(bo); | 1089 | *offset = amdgpu_bo_mmap_offset(bo); |
| @@ -1092,7 +1164,7 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( · 74 unmodified lines | |||
| 1092 | struct kgd_dev *kgd, struct kgd_mem *mem, void *vm) | 1164 | struct kgd_dev *kgd, struct kgd_mem *mem, void *vm) |
| 1093 | { | 1165 | { |
| 1094 | struct amdgpu_device *adev = get_amdgpu_device(kgd); | 1166 | struct amdgpu_device *adev = get_amdgpu_device(kgd); |
| 1095 | struct amdkfd_vm *kfd_vm = (struct amdkfd_vm *)vm; | 1167 | struct amdgpu_vm *avm = (struct amdgpu_vm *)vm; |
| 1096 | int ret; | 1168 | int ret; |
| 1097 | struct amdgpu_bo *bo; | 1169 | struct amdgpu_bo *bo; |
| 1098 | uint32_t domain; | 1170 | uint32_t domain; |
| @@ -1128,19 +1200,19 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( · 29 unmodified lines | |||
| 1128 | if (unlikely(ret)) | 1200 | if (unlikely(ret)) |
| 1129 | goto out; | 1201 | goto out; |
| 1130 | 1202 | ||
| 1131 | if (check_if_add_bo_to_vm((struct amdgpu_vm *)vm, mem)) { | 1203 | if (check_if_add_bo_to_vm(avm, mem)) { |
| 1132 | ret = add_bo_to_vm(adev, mem, (struct amdgpu_vm *)vm, false, | 1204 | ret = add_bo_to_vm(adev, mem, avm, false, |
| 1133 | &bo_va_entry); | 1205 | &bo_va_entry); |
| 1134 | if (ret) | 1206 | if (ret) |
| 1135 | goto add_bo_to_vm_failed; | 1207 | goto add_bo_to_vm_failed; |
| 1136 | if (mem->aql_queue) { | 1208 | if (mem->aql_queue) { |
| 1137 | ret = add_bo_to_vm(adev, mem, (struct amdgpu_vm *)vm, | 1209 | ret = add_bo_to_vm(adev, mem, avm, |
| 1138 | true, &bo_va_entry_aql); | 1210 | true, &bo_va_entry_aql); |
| 1139 | if (ret) | 1211 | if (ret) |
| 1140 | goto add_bo_to_vm_failed_aql; | 1212 | goto add_bo_to_vm_failed_aql; |
| 1141 | } | 1213 | } |
| 1142 | } else { | 1214 | } else { |
| 1143 | ret = vm_validate_pt_pd_bos((struct amdkfd_vm *)vm); | 1215 | ret = vm_validate_pt_pd_bos(avm); |
| 1144 | if (unlikely(ret)) | 1216 | if (unlikely(ret)) |
| 1145 | goto add_bo_to_vm_failed; | 1217 | goto add_bo_to_vm_failed; |
| 1146 | } | 1218 | } |
| @@ -1184,7 +1256,7 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( · 37 unmodified lines | |||
| 1184 | 1256 | ||
| 1185 | if (!amdgpu_ttm_tt_get_usermm(bo->tbo.ttm) && !bo->pin_count) | 1257 | if (!amdgpu_ttm_tt_get_usermm(bo->tbo.ttm) && !bo->pin_count) |
| 1186 | amdgpu_bo_fence(bo, | 1258 | amdgpu_bo_fence(bo, |
| 1187 | &kfd_vm->process_info->eviction_fence->base, | 1259 | &avm->process_info->eviction_fence->base, |
| 1188 | true); | 1260 | true); |
| 1189 | ret = unreserve_bo_and_vms(&ctx, false, false); | 1261 | ret = unreserve_bo_and_vms(&ctx, false, false); |
| 1190 | 1262 | ||
| @@ -1209,7 +1281,7 @@ int amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu( · 18 unmodified lines | |||
| 1209 | { | 1281 | { |
| 1210 | struct amdgpu_device *adev = get_amdgpu_device(kgd); | 1282 | struct amdgpu_device *adev = get_amdgpu_device(kgd); |
| 1211 | struct amdkfd_process_info *process_info = | 1283 | struct amdkfd_process_info *process_info = |
| 1212 | ((struct amdkfd_vm *)vm)->process_info; | 1284 | ((struct amdgpu_vm *)vm)->process_info; |
| 1213 | unsigned long bo_size = mem->bo->tbo.mem.size; | 1285 | unsigned long bo_size = mem->bo->tbo.mem.size; |
| 1214 | struct kfd_bo_va_list *entry; | 1286 | struct kfd_bo_va_list *entry; |
| 1215 | struct bo_vm_reservation_context ctx; | 1287 | struct bo_vm_reservation_context ctx; |
| @@ -1226,7 +1298,7 @@ int amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu( · 10 unmodified lines | |||
| 1226 | goto unreserve_out; | 1298 | goto unreserve_out; |
| 1227 | } | 1299 | } |
| 1228 | 1300 | ||
| 1229 | ret = vm_validate_pt_pd_bos((struct amdkfd_vm *)vm); | 1301 | ret = vm_validate_pt_pd_bos((struct amdgpu_vm *)vm); |
| 1230 | if (unlikely(ret)) | 1302 | if (unlikely(ret)) |
| 1231 | goto unreserve_out; | 1303 | goto unreserve_out; |
| 1232 | 1304 | ||
| @@ -1368,7 +1440,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence **ef) · 135 unmodified lines | |||
| 1368 | { | 1440 | { |
| 1369 | struct amdgpu_bo_list_entry *pd_bo_list; | 1441 | struct amdgpu_bo_list_entry *pd_bo_list; |
| 1370 | struct amdkfd_process_info *process_info = info; | 1442 | struct amdkfd_process_info *process_info = info; |
| 1371 | struct amdkfd_vm *peer_vm; | 1443 | struct amdgpu_vm *peer_vm; |
| 1372 | struct kgd_mem *mem; | 1444 | struct kgd_mem *mem; |
| 1373 | struct bo_vm_reservation_context ctx; | 1445 | struct bo_vm_reservation_context ctx; |
| 1374 | struct amdgpu_amdkfd_fence *new_fence; | 1446 | struct amdgpu_amdkfd_fence *new_fence; |
| @@ -1390,8 +1462,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence **ef) · 15 unmodified lines | |||
| 1390 | mutex_lock(&process_info->lock); | 1462 | mutex_lock(&process_info->lock); |
| 1391 | list_for_each_entry(peer_vm, &process_info->vm_list_head, | 1463 | list_for_each_entry(peer_vm, &process_info->vm_list_head, |
| 1392 | vm_list_node) | 1464 | vm_list_node) |
| 1393 | amdgpu_vm_get_pd_bo(&peer_vm->base, &ctx.list, | 1465 | amdgpu_vm_get_pd_bo(peer_vm, &ctx.list, &pd_bo_list[i++]); |
| 1394 | &pd_bo_list[i++]); | ||
| 1395 | 1466 | ||
| 1396 | /* Reserve all BOs and page tables/directory. Add all BOs from | 1467 | /* Reserve all BOs and page tables/directory. Add all BOs from |
| 1397 | * kfd_bo_list to ctx.list | 1468 | * kfd_bo_list to ctx.list |
| @@ -1422,7 +1493,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence **ef) · 24 unmodified lines | |||
| 1422 | /* FIXME: I think this isn't needed */ | 1493 | /* FIXME: I think this isn't needed */ |
| 1423 | list_for_each_entry(peer_vm, &process_info->vm_list_head, | 1494 | list_for_each_entry(peer_vm, &process_info->vm_list_head, |
| 1424 | vm_list_node) { | 1495 | vm_list_node) { |
| 1425 | struct amdgpu_bo *bo = peer_vm->base.root.base.bo; | 1496 | struct amdgpu_bo *bo = peer_vm->root.base.bo; |
| 1426 | 1497 | ||
| 1427 | ttm_bo_wait(&bo->tbo, false, false); | 1498 | ttm_bo_wait(&bo->tbo, false, false); |
| 1428 | } | 1499 | } |
| @@ -1491,7 +1562,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence **ef) · 62 unmodified lines | |||
| 1491 | /* Attach eviction fence to PD / PT BOs */ | 1562 | /* Attach eviction fence to PD / PT BOs */ |
| 1492 | list_for_each_entry(peer_vm, &process_info->vm_list_head, | 1563 | list_for_each_entry(peer_vm, &process_info->vm_list_head, |
| 1493 | vm_list_node) { | 1564 | vm_list_node) { |
| 1494 | struct amdgpu_bo *bo = peer_vm->base.root.base.bo; | 1565 | struct amdgpu_bo *bo = peer_vm->root.base.bo; |
| 1495 | 1566 | ||
| 1496 | amdgpu_bo_fence(bo, &process_info->eviction_fence->base, true); | 1567 | amdgpu_bo_fence(bo, &process_info->eviction_fence->base, true); |
| 1497 | } | 1568 | } |
| @@ -569,6 +569,7 @@ static const struct amdgpu_px_quirk amdgpu_px_quirk_list[] = { · 568 unmodified lines | |||
| 569 | { 0x1002, 0x6900, 0x1002, 0x0124, AMDGPU_PX_QUIRK_FORCE_ATPX }, | 569 | { 0x1002, 0x6900, 0x1002, 0x0124, AMDGPU_PX_QUIRK_FORCE_ATPX }, |
| 570 | { 0x1002, 0x6900, 0x1028, 0x0812, AMDGPU_PX_QUIRK_FORCE_ATPX }, | 570 | { 0x1002, 0x6900, 0x1028, 0x0812, AMDGPU_PX_QUIRK_FORCE_ATPX }, |
| 571 | { 0x1002, 0x6900, 0x1028, 0x0813, AMDGPU_PX_QUIRK_FORCE_ATPX }, | 571 | { 0x1002, 0x6900, 0x1028, 0x0813, AMDGPU_PX_QUIRK_FORCE_ATPX }, |
| 572 | { 0x1002, 0x67DF, 0x1028, 0x0774, AMDGPU_PX_QUIRK_FORCE_ATPX }, | ||
| 572 | { 0, 0, 0, 0, 0 }, | 573 | { 0, 0, 0, 0, 0 }, |
| 573 | }; | 574 | }; |
| 574 | 575 | ||
| @@ -28,7 +28,6 @@ · 27 unmodified lines | |||
| 28 | #include <linux/firmware.h> | 28 | #include <linux/firmware.h> |
| 29 | #include <drm/amdgpu_drm.h> | 29 | #include <drm/amdgpu_drm.h> |
| 30 | #include "amdgpu.h" | 30 | #include "amdgpu.h" |
| 31 | #include "cgs_linux.h" | ||
| 32 | #include "atom.h" | 31 | #include "atom.h" |
| 33 | #include "amdgpu_ucode.h" | 32 | #include "amdgpu_ucode.h" |
| 34 | 33 | ||
| @@ -182,109 +181,6 @@ static int amdgpu_cgs_atom_exec_cmd_table(struct cgs_device *cgs_device, unsigne · 147 unmodified lines | |||
| 182 | adev->mode_info.atom_context, table, args); | 181 | adev->mode_info.atom_context, table, args); |
| 183 | } | 182 | } |
| 184 | 183 | ||
| 185 | struct cgs_irq_params { | ||
| 186 | unsigned src_id; | ||
| 187 | cgs_irq_source_set_func_t set; | ||
| 188 | cgs_irq_handler_func_t handler; | ||
| 189 | void *private_data; | ||
| 190 | }; | ||
| 191 | |||
| 192 | static int cgs_set_irq_state(struct amdgpu_device *adev, | ||
| 193 | struct amdgpu_irq_src *src, | ||
| 194 | unsigned type, | ||
| 195 | enum amdgpu_interrupt_state state) | ||
| 196 | { | ||
| 197 | struct cgs_irq_params *irq_params = | ||
| 198 | (struct cgs_irq_params *)src->data; | ||
| 199 | if (!irq_params) | ||
| 200 | return -EINVAL; | ||
| 201 | if (!irq_params->set) | ||
| 202 | return -EINVAL; | ||
| 203 | return irq_params->set(irq_params->private_data, | ||
| 204 | irq_params->src_id, | ||
| 205 | type, | ||
| 206 | (int)state); | ||
| 207 | } | ||
| 208 | |||
| 209 | static int cgs_process_irq(struct amdgpu_device *adev, | ||
| 210 | struct amdgpu_irq_src *source, | ||
| 211 | struct amdgpu_iv_entry *entry) | ||
| 212 | { | ||
| 213 | struct cgs_irq_params *irq_params = | ||
| 214 | (struct cgs_irq_params *)source->data; | ||
| 215 | if (!irq_params) | ||
| 216 | return -EINVAL; | ||
| 217 | if (!irq_params->handler) | ||
| 218 | return -EINVAL; | ||
| 219 | return irq_params->handler(irq_params->private_data, | ||
| 220 | irq_params->src_id, | ||
| 221 | entry->iv_entry); | ||
| 222 | } | ||
| 223 | |||
| 224 | static const struct amdgpu_irq_src_funcs cgs_irq_funcs = { | ||
| 225 | .set = cgs_set_irq_state, | ||
| 226 | .process = cgs_process_irq, | ||
| 227 | }; | ||
| 228 | |||
| 229 | static int amdgpu_cgs_add_irq_source(void *cgs_device, | ||
| 230 | unsigned client_id, | ||
| 231 | unsigned src_id, | ||
| 232 | unsigned num_types, | ||
| 233 | cgs_irq_source_set_func_t set, | ||
| 234 | cgs_irq_handler_func_t handler, | ||
| 235 | void *private_data) | ||
| 236 | { | ||
| 237 | CGS_FUNC_ADEV; | ||
| 238 | int ret = 0; | ||
| 239 | struct cgs_irq_params *irq_params; | ||
| 240 | struct amdgpu_irq_src *source = | ||
| 241 | kzalloc(sizeof(struct amdgpu_irq_src), GFP_KERNEL); | ||
| 242 | if (!source) | ||
| 243 | return -ENOMEM; | ||
| 244 | irq_params = | ||
| 245 | kzalloc(sizeof(struct cgs_irq_params), GFP_KERNEL); | ||
| 246 | if (!irq_params) { | ||
| 247 | kfree(source); | ||
| 248 | return -ENOMEM; | ||
| 249 | } | ||
| 250 | source->num_types = num_types; | ||
| 251 | source->funcs = &cgs_irq_funcs; | ||
| 252 | irq_params->src_id = src_id; | ||
| 253 | irq_params->set = set; | ||
| 254 | irq_params->handler = handler; | ||
| 255 | irq_params->private_data = private_data; | ||
| 256 | source->data = (void *)irq_params; | ||
| 257 | ret = amdgpu_irq_add_id(adev, client_id, src_id, source); | ||
| 258 | if (ret) { | ||
| 259 | kfree(irq_params); | ||
| 260 | kfree(source); | ||
| 261 | } | ||
| 262 | |||
| 263 | return ret; | ||
| 264 | } | ||
| 265 | |||
| 266 | static int amdgpu_cgs_irq_get(void *cgs_device, unsigned client_id, | ||
| 267 | unsigned src_id, unsigned type) | ||
| 268 | { | ||
| 269 | CGS_FUNC_ADEV; | ||
| 270 | |||
| 271 | if (!adev->irq.client[client_id].sources) | ||
| 272 | return -EINVAL; | ||
| 273 | |||
| 274 | return amdgpu_irq_get(adev, adev->irq.client[client_id].sources[src_id], type); | ||
| 275 | } | ||
| 276 | |||
| 277 | static int amdgpu_cgs_irq_put(void *cgs_device, unsigned client_id, | ||
| 278 | unsigned src_id, unsigned type) | ||
| 279 | { | ||
| 280 | CGS_FUNC_ADEV; | ||
| 281 | |||
| 282 | if (!adev->irq.client[client_id].sources) | ||
| 283 | return -EINVAL; | ||
| 284 | |||
| 285 | return amdgpu_irq_put(adev, adev->irq.client[client_id].sources[src_id], type); | ||
| 286 | } | ||
| 287 | |||
| 288 | static int amdgpu_cgs_set_clockgating_state(struct cgs_device *cgs_device, | 184 | static int amdgpu_cgs_set_clockgating_state(struct cgs_device *cgs_device, |
| 289 | enum amd_ip_block_type block_type, | 185 | enum amd_ip_block_type block_type, |
| 290 | enum amd_clockgating_state state) | 186 | enum amd_clockgating_state state) |
| @@ -654,6 +550,9 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 363 unmodified lines | |||
| 654 | else | 550 | else |
| 655 | strcpy(fw_name, "amdgpu/vega10_smc.bin"); | 551 | strcpy(fw_name, "amdgpu/vega10_smc.bin"); |
| 656 | break; | 552 | break; |
| 553 | case CHIP_VEGA12: | ||
| 554 | strcpy(fw_name, "amdgpu/vega12_smc.bin"); | ||
| 555 | break; | ||
| 657 | default: | 556 | default: |
| 658 | DRM_ERROR("SMC firmware not supported\n"); | 557 | DRM_ERROR("SMC firmware not supported\n"); |
| 659 | return -EINVAL; | 558 | return -EINVAL; |
| @@ -715,12 +614,9 @@ static int amdgpu_cgs_get_active_displays_info(struct cgs_device *cgs_device, · 55 unmodified lines | |||
| 715 | return -EINVAL; | 614 | return -EINVAL; |
| 716 | 615 | ||
| 717 | mode_info = info->mode_info; | 616 | mode_info = info->mode_info; |
| 718 | if (mode_info) { | 617 | if (mode_info) |
| 719 | /* if the displays are off, vblank time is max */ | 618 | /* if the displays are off, vblank time is max */ |
| 720 | mode_info->vblank_time_us = 0xffffffff; | 619 | mode_info->vblank_time_us = 0xffffffff; |
| 721 | /* always set the reference clock */ | ||
| 722 | mode_info->ref_clock = adev->clock.spll.reference_freq; | ||
| 723 | } | ||
| 724 | 620 | ||
| 725 | if (!amdgpu_device_has_dc_support(adev)) { | 621 | if (!amdgpu_device_has_dc_support(adev)) { |
| 726 | struct amdgpu_crtc *amdgpu_crtc; | 622 | struct amdgpu_crtc *amdgpu_crtc; |
| @@ -795,12 +691,6 @@ static const struct cgs_ops amdgpu_cgs_ops = { · 68 unmodified lines | |||
| 795 | .lock_grbm_idx = amdgpu_cgs_lock_grbm_idx, | 691 | .lock_grbm_idx = amdgpu_cgs_lock_grbm_idx, |
| 796 | }; | 692 | }; |
| 797 | 693 | ||
| 798 | static const struct cgs_os_ops amdgpu_cgs_os_ops = { | ||
| 799 | .add_irq_source = amdgpu_cgs_add_irq_source, | ||
| 800 | .irq_get = amdgpu_cgs_irq_get, | ||
| 801 | .irq_put = amdgpu_cgs_irq_put | ||
| 802 | }; | ||
| 803 | |||
| 804 | struct cgs_device *amdgpu_cgs_create_device(struct amdgpu_device *adev) | 694 | struct cgs_device *amdgpu_cgs_create_device(struct amdgpu_device *adev) |
| 805 | { | 695 | { |
| 806 | struct amdgpu_cgs_device *cgs_device = | 696 | struct amdgpu_cgs_device *cgs_device = |
| @@ -812,7 +702,6 @@ struct cgs_device *amdgpu_cgs_create_device(struct amdgpu_device *adev) · 5 unmodified lines | |||
| 812 | } | 702 | } |
| 813 | 703 | ||
| 814 | cgs_device->base.ops = &amdgpu_cgs_ops; | 704 | cgs_device->base.ops = &amdgpu_cgs_ops; |
| 815 | cgs_device->base.os_ops = &amdgpu_cgs_os_ops; | ||
| 816 | cgs_device->adev = adev; | 705 | cgs_device->adev = adev; |
| 817 | 706 | ||
| 818 | return (struct cgs_device *)cgs_device; | 707 | return (struct cgs_device *)cgs_device; |
| @@ -729,9 +729,11 @@ amdgpu_connector_lvds_detect(struct drm_connector *connector, bool force) · 728 unmodified lines | |||
| 729 | enum drm_connector_status ret = connector_status_disconnected; | 729 | enum drm_connector_status ret = connector_status_disconnected; |
| 730 | int r; | 730 | int r; |
| 731 | 731 | ||
| 732 | r = pm_runtime_get_sync(connector->dev->dev); | 732 | if (!drm_kms_helper_is_poll_worker()) { |
| 733 | if (r < 0) | 733 | r = pm_runtime_get_sync(connector->dev->dev); |
| 734 | return connector_status_disconnected; | 734 | if (r < 0) |
| 735 | return connector_status_disconnected; | ||
| 736 | } | ||
| 735 | 737 | ||
| 736 | if (encoder) { | 738 | if (encoder) { |
| 737 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); | 739 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); |
| @@ -750,8 +752,12 @@ amdgpu_connector_lvds_detect(struct drm_connector *connector, bool force) · 12 unmodified lines | |||
| 750 | /* check acpi lid status ??? */ | 752 | /* check acpi lid status ??? */ |
| 751 | 753 | ||
| 752 | amdgpu_connector_update_scratch_regs(connector, ret); | 754 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 753 | pm_runtime_mark_last_busy(connector->dev->dev); | 755 | |
| 754 | pm_runtime_put_autosuspend(connector->dev->dev); | 756 | if (!drm_kms_helper_is_poll_worker()) { |
| 757 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 758 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 759 | } | ||
| 760 | |||
| 755 | return ret; | 761 | return ret; |
| 756 | } | 762 | } |
| 757 | 763 | ||
| @@ -861,9 +867,11 @@ amdgpu_connector_vga_detect(struct drm_connector *connector, bool force) · 103 unmodified lines | |||
| 861 | enum drm_connector_status ret = connector_status_disconnected; | 867 | enum drm_connector_status ret = connector_status_disconnected; |
| 862 | int r; | 868 | int r; |
| 863 | 869 | ||
| 864 | r = pm_runtime_get_sync(connector->dev->dev); | 870 | if (!drm_kms_helper_is_poll_worker()) { |
| 865 | if (r < 0) | 871 | r = pm_runtime_get_sync(connector->dev->dev); |
| 866 | return connector_status_disconnected; | 872 | if (r < 0) |
| 873 | return connector_status_disconnected; | ||
| 874 | } | ||
| 867 | 875 | ||
| 868 | encoder = amdgpu_connector_best_single_encoder(connector); | 876 | encoder = amdgpu_connector_best_single_encoder(connector); |
| 869 | if (!encoder) | 877 | if (!encoder) |
| @@ -917,8 +925,10 @@ amdgpu_connector_vga_detect(struct drm_connector *connector, bool force) · 47 unmodified lines | |||
| 917 | amdgpu_connector_update_scratch_regs(connector, ret); | 925 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 918 | 926 | ||
| 919 | out: | 927 | out: |
| 920 | pm_runtime_mark_last_busy(connector->dev->dev); | 928 | if (!drm_kms_helper_is_poll_worker()) { |
| 921 | pm_runtime_put_autosuspend(connector->dev->dev); | 929 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 930 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 931 | } | ||
| 922 | 932 | ||
| 923 | return ret; | 933 | return ret; |
| 924 | } | 934 | } |
| @@ -981,9 +991,11 @@ amdgpu_connector_dvi_detect(struct drm_connector *connector, bool force) · 56 unmodified lines | |||
| 981 | enum drm_connector_status ret = connector_status_disconnected; | 991 | enum drm_connector_status ret = connector_status_disconnected; |
| 982 | bool dret = false, broken_edid = false; | 992 | bool dret = false, broken_edid = false; |
| 983 | 993 | ||
| 984 | r = pm_runtime_get_sync(connector->dev->dev); | 994 | if (!drm_kms_helper_is_poll_worker()) { |
| 985 | if (r < 0) | 995 | r = pm_runtime_get_sync(connector->dev->dev); |
| 986 | return connector_status_disconnected; | 996 | if (r < 0) |
| 997 | return connector_status_disconnected; | ||
| 998 | } | ||
| 987 | 999 | ||
| 988 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { | 1000 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { |
| 989 | ret = connector->status; | 1001 | ret = connector->status; |
| @@ -1108,8 +1120,10 @@ out: · 118 unmodified lines | |||
| 1108 | amdgpu_connector_update_scratch_regs(connector, ret); | 1120 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 1109 | 1121 | ||
| 1110 | exit: | 1122 | exit: |
| 1111 | pm_runtime_mark_last_busy(connector->dev->dev); | 1123 | if (!drm_kms_helper_is_poll_worker()) { |
| 1112 | pm_runtime_put_autosuspend(connector->dev->dev); | 1124 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1125 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1126 | } | ||
| 1113 | 1127 | ||
| 1114 | return ret; | 1128 | return ret; |
| 1115 | } | 1129 | } |
| @@ -1352,9 +1366,11 @@ amdgpu_connector_dp_detect(struct drm_connector *connector, bool force) · 236 unmodified lines | |||
| 1352 | struct drm_encoder *encoder = amdgpu_connector_best_single_encoder(connector); | 1366 | struct drm_encoder *encoder = amdgpu_connector_best_single_encoder(connector); |
| 1353 | int r; | 1367 | int r; |
| 1354 | 1368 | ||
| 1355 | r = pm_runtime_get_sync(connector->dev->dev); | 1369 | if (!drm_kms_helper_is_poll_worker()) { |
| 1356 | if (r < 0) | 1370 | r = pm_runtime_get_sync(connector->dev->dev); |
| 1357 | return connector_status_disconnected; | 1371 | if (r < 0) |
| 1372 | return connector_status_disconnected; | ||
| 1373 | } | ||
| 1358 | 1374 | ||
| 1359 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { | 1375 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { |
| 1360 | ret = connector->status; | 1376 | ret = connector->status; |
| @@ -1424,8 +1440,10 @@ amdgpu_connector_dp_detect(struct drm_connector *connector, bool force) · 63 unmodified lines | |||
| 1424 | 1440 | ||
| 1425 | amdgpu_connector_update_scratch_regs(connector, ret); | 1441 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 1426 | out: | 1442 | out: |
| 1427 | pm_runtime_mark_last_busy(connector->dev->dev); | 1443 | if (!drm_kms_helper_is_poll_worker()) { |
| 1428 | pm_runtime_put_autosuspend(connector->dev->dev); | 1444 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1445 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1446 | } | ||
| 1429 | 1447 | ||
| 1430 | return ret; | 1448 | return ret; |
| 1431 | } | 1449 | } |
| @@ -59,6 +59,7 @@ · 58 unmodified lines | |||
| 59 | #include "amdgpu_pm.h" | 59 | #include "amdgpu_pm.h" |
| 60 | 60 | ||
| 61 | MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin"); | 61 | MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin"); |
| 62 | MODULE_FIRMWARE("amdgpu/vega12_gpu_info.bin"); | ||
| 62 | MODULE_FIRMWARE("amdgpu/raven_gpu_info.bin"); | 63 | MODULE_FIRMWARE("amdgpu/raven_gpu_info.bin"); |
| 63 | 64 | ||
| 64 | #define AMDGPU_RESUME_MS 2000 | 65 | #define AMDGPU_RESUME_MS 2000 |
| @@ -83,12 +84,21 @@ static const char *amdgpu_asic_name[] = { · 18 unmodified lines | |||
| 83 | "POLARIS11", | 84 | "POLARIS11", |
| 84 | "POLARIS12", | 85 | "POLARIS12", |
| 85 | "VEGA10", | 86 | "VEGA10", |
| 87 | "VEGA12", | ||
| 86 | "RAVEN", | 88 | "RAVEN", |
| 87 | "LAST", | 89 | "LAST", |
| 88 | }; | 90 | }; |
| 89 | 91 | ||
| 90 | static void amdgpu_device_get_pcie_info(struct amdgpu_device *adev); | 92 | static void amdgpu_device_get_pcie_info(struct amdgpu_device *adev); |
| 91 | 93 | ||
| 94 | /** | ||
| 95 | * amdgpu_device_is_px - Is the device is a dGPU with HG/PX power control | ||
| 96 | * | ||
| 97 | * @dev: drm_device pointer | ||
| 98 | * | ||
| 99 | * Returns true if the device is a dGPU with HG/PX power control, | ||
| 100 | * otherwise return false. | ||
| 101 | */ | ||
| 92 | bool amdgpu_device_is_px(struct drm_device *dev) | 102 | bool amdgpu_device_is_px(struct drm_device *dev) |
| 93 | { | 103 | { |
| 94 | struct amdgpu_device *adev = dev->dev_private; | 104 | struct amdgpu_device *adev = dev->dev_private; |
| @@ -101,6 +111,15 @@ bool amdgpu_device_is_px(struct drm_device *dev) · 6 unmodified lines | |||
| 101 | /* | 111 | /* |
| 102 | * MMIO register access helper functions. | 112 | * MMIO register access helper functions. |
| 103 | */ | 113 | */ |
| 114 | /** | ||
| 115 | * amdgpu_mm_rreg - read a memory mapped IO register | ||
| 116 | * | ||
| 117 | * @adev: amdgpu_device pointer | ||
| 118 | * @reg: dword aligned register offset | ||
| 119 | * @acc_flags: access flags which require special behavior | ||
| 120 | * | ||
| 121 | * Returns the 32 bit value from the offset specified. | ||
| 122 | */ | ||
| 104 | uint32_t amdgpu_mm_rreg(struct amdgpu_device *adev, uint32_t reg, | 123 | uint32_t amdgpu_mm_rreg(struct amdgpu_device *adev, uint32_t reg, |
| 105 | uint32_t acc_flags) | 124 | uint32_t acc_flags) |
| 106 | { | 125 | { |
| @@ -129,6 +148,14 @@ uint32_t amdgpu_mm_rreg(struct amdgpu_device *adev, uint32_t reg, · 22 unmodified lines | |||
| 129 | * | 148 | * |
| 130 | */ | 149 | */ |
| 131 | 150 | ||
| 151 | /** | ||
| 152 | * amdgpu_mm_rreg8 - read a memory mapped IO register | ||
| 153 | * | ||
| 154 | * @adev: amdgpu_device pointer | ||
| 155 | * @offset: byte aligned register offset | ||
| 156 | * | ||
| 157 | * Returns the 8 bit value from the offset specified. | ||
| 158 | */ | ||
| 132 | uint8_t amdgpu_mm_rreg8(struct amdgpu_device *adev, uint32_t offset) { | 159 | uint8_t amdgpu_mm_rreg8(struct amdgpu_device *adev, uint32_t offset) { |
| 133 | if (offset < adev->rmmio_size) | 160 | if (offset < adev->rmmio_size) |
| 134 | return (readb(adev->rmmio + offset)); | 161 | return (readb(adev->rmmio + offset)); |
| @@ -141,6 +168,15 @@ uint8_t amdgpu_mm_rreg8(struct amdgpu_device *adev, uint32_t offset) { · 6 unmodified lines | |||
| 141 | * @value: the value want to be written to the register | 168 | * @value: the value want to be written to the register |
| 142 | * | 169 | * |
| 143 | */ | 170 | */ |
| 171 | /** | ||
| 172 | * amdgpu_mm_wreg8 - read a memory mapped IO register | ||
| 173 | * | ||
| 174 | * @adev: amdgpu_device pointer | ||
| 175 | * @offset: byte aligned register offset | ||
| 176 | * @value: 8 bit value to write | ||
| 177 | * | ||
| 178 | * Writes the value specified to the offset specified. | ||
| 179 | */ | ||
| 144 | void amdgpu_mm_wreg8(struct amdgpu_device *adev, uint32_t offset, uint8_t value) { | 180 | void amdgpu_mm_wreg8(struct amdgpu_device *adev, uint32_t offset, uint8_t value) { |
| 145 | if (offset < adev->rmmio_size) | 181 | if (offset < adev->rmmio_size) |
| 146 | writeb(value, adev->rmmio + offset); | 182 | writeb(value, adev->rmmio + offset); |
| @@ -148,7 +184,16 @@ void amdgpu_mm_wreg8(struct amdgpu_device *adev, uint32_t offset, uint8_t value) · 1 unmodified line | |||
| 148 | BUG(); | 184 | BUG(); |
| 149 | } | 185 | } |
| 150 | 186 | ||
| 151 | 187 | /** | |
| 188 | * amdgpu_mm_wreg - write to a memory mapped IO register | ||
| 189 | * | ||
| 190 | * @adev: amdgpu_device pointer | ||
| 191 | * @reg: dword aligned register offset | ||
| 192 | * @v: 32 bit value to write to the register | ||
| 193 | * @acc_flags: access flags which require special behavior | ||
| 194 | * | ||
| 195 | * Writes the value specified to the offset specified. | ||
| 196 | */ | ||
| 152 | void amdgpu_mm_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v, | 197 | void amdgpu_mm_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v, |
| 153 | uint32_t acc_flags) | 198 | uint32_t acc_flags) |
| 154 | { | 199 | { |
| @@ -177,6 +222,14 @@ void amdgpu_mm_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v, · 22 unmodified lines | |||
| 177 | } | 222 | } |
| 178 | } | 223 | } |
| 179 | 224 | ||
| 225 | /** | ||
| 226 | * amdgpu_io_rreg - read an IO register | ||
| 227 | * | ||
| 228 | * @adev: amdgpu_device pointer | ||
| 229 | * @reg: dword aligned register offset | ||
| 230 | * | ||
| 231 | * Returns the 32 bit value from the offset specified. | ||
| 232 | */ | ||
| 180 | u32 amdgpu_io_rreg(struct amdgpu_device *adev, u32 reg) | 233 | u32 amdgpu_io_rreg(struct amdgpu_device *adev, u32 reg) |
| 181 | { | 234 | { |
| 182 | if ((reg * 4) < adev->rio_mem_size) | 235 | if ((reg * 4) < adev->rio_mem_size) |
| @@ -187,6 +240,15 @@ u32 amdgpu_io_rreg(struct amdgpu_device *adev, u32 reg) · 4 unmodified lines | |||
| 187 | } | 240 | } |
| 188 | } | 241 | } |
| 189 | 242 | ||
| 243 | /** | ||
| 244 | * amdgpu_io_wreg - write to an IO register | ||
| 245 | * | ||
| 246 | * @adev: amdgpu_device pointer | ||
| 247 | * @reg: dword aligned register offset | ||
| 248 | * @v: 32 bit value to write to the register | ||
| 249 | * | ||
| 250 | * Writes the value specified to the offset specified. | ||
| 251 | */ | ||
| 190 | void amdgpu_io_wreg(struct amdgpu_device *adev, u32 reg, u32 v) | 252 | void amdgpu_io_wreg(struct amdgpu_device *adev, u32 reg, u32 v) |
| 191 | { | 253 | { |
| 192 | if (adev->asic_type >= CHIP_VEGA10 && reg == 0) { | 254 | if (adev->asic_type >= CHIP_VEGA10 && reg == 0) { |
| @@ -355,6 +417,14 @@ static void amdgpu_block_invalid_wreg(struct amdgpu_device *adev, · 162 unmodified lines | |||
| 355 | BUG(); | 417 | BUG(); |
| 356 | } | 418 | } |
| 357 | 419 | ||
| 420 | /** | ||
| 421 | * amdgpu_device_vram_scratch_init - allocate the VRAM scratch page | ||
| 422 | * | ||
| 423 | * @adev: amdgpu device pointer | ||
| 424 | * | ||
| 425 | * Allocates a scratch page of VRAM for use by various things in the | ||
| 426 | * driver. | ||
| 427 | */ | ||
| 358 | static int amdgpu_device_vram_scratch_init(struct amdgpu_device *adev) | 428 | static int amdgpu_device_vram_scratch_init(struct amdgpu_device *adev) |
| 359 | { | 429 | { |
| 360 | return amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, | 430 | return amdgpu_bo_create_kernel(adev, AMDGPU_GPU_PAGE_SIZE, |
| @@ -364,6 +434,13 @@ static int amdgpu_device_vram_scratch_init(struct amdgpu_device *adev) · 3 unmodified lines | |||
| 364 | (void **)&adev->vram_scratch.ptr); | 434 | (void **)&adev->vram_scratch.ptr); |
| 365 | } | 435 | } |
| 366 | 436 | ||
| 437 | /** | ||
| 438 | * amdgpu_device_vram_scratch_fini - Free the VRAM scratch page | ||
| 439 | * | ||
| 440 | * @adev: amdgpu device pointer | ||
| 441 | * | ||
| 442 | * Frees the VRAM scratch page. | ||
| 443 | */ | ||
| 367 | static void amdgpu_device_vram_scratch_fini(struct amdgpu_device *adev) | 444 | static void amdgpu_device_vram_scratch_fini(struct amdgpu_device *adev) |
| 368 | { | 445 | { |
| 369 | amdgpu_bo_free_kernel(&adev->vram_scratch.robj, NULL, NULL); | 446 | amdgpu_bo_free_kernel(&adev->vram_scratch.robj, NULL, NULL); |
| @@ -405,6 +482,14 @@ void amdgpu_device_program_register_sequence(struct amdgpu_device *adev, · 35 unmodified lines | |||
| 405 | } | 482 | } |
| 406 | } | 483 | } |
| 407 | 484 | ||
| 485 | /** | ||
| 486 | * amdgpu_device_pci_config_reset - reset the GPU | ||
| 487 | * | ||
| 488 | * @adev: amdgpu_device pointer | ||
| 489 | * | ||
| 490 | * Resets the GPU using the pci config reset sequence. | ||
| 491 | * Only applicable to asics prior to vega10. | ||
| 492 | */ | ||
| 408 | void amdgpu_device_pci_config_reset(struct amdgpu_device *adev) | 493 | void amdgpu_device_pci_config_reset(struct amdgpu_device *adev) |
| 409 | { | 494 | { |
| 410 | pci_write_config_dword(adev->pdev, 0x7c, AMDGPU_ASIC_RESET_DATA); | 495 | pci_write_config_dword(adev->pdev, 0x7c, AMDGPU_ASIC_RESET_DATA); |
| @@ -565,6 +650,7 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb) · 154 unmodified lines | |||
| 565 | 650 | ||
| 566 | /** | 651 | /** |
| 567 | * amdgpu_device_vram_location - try to find VRAM location | 652 | * amdgpu_device_vram_location - try to find VRAM location |
| 653 | * | ||
| 568 | * @adev: amdgpu device structure holding all necessary informations | 654 | * @adev: amdgpu device structure holding all necessary informations |
| 569 | * @mc: memory controller structure holding memory informations | 655 | * @mc: memory controller structure holding memory informations |
| 570 | * @base: base address at which to put VRAM | 656 | * @base: base address at which to put VRAM |
| @@ -588,6 +674,7 @@ void amdgpu_device_vram_location(struct amdgpu_device *adev, · 17 unmodified lines | |||
| 588 | 674 | ||
| 589 | /** | 675 | /** |
| 590 | * amdgpu_device_gart_location - try to find GTT location | 676 | * amdgpu_device_gart_location - try to find GTT location |
| 677 | * | ||
| 591 | * @adev: amdgpu device structure holding all necessary informations | 678 | * @adev: amdgpu device structure holding all necessary informations |
| 592 | * @mc: memory controller structure holding memory informations | 679 | * @mc: memory controller structure holding memory informations |
| 593 | * | 680 | * |
| @@ -774,6 +861,16 @@ static unsigned int amdgpu_device_vga_set_decode(void *cookie, bool state) · 180 unmodified lines | |||
| 774 | return VGA_RSRC_NORMAL_IO | VGA_RSRC_NORMAL_MEM; | 861 | return VGA_RSRC_NORMAL_IO | VGA_RSRC_NORMAL_MEM; |
| 775 | } | 862 | } |
| 776 | 863 | ||
| 864 | /** | ||
| 865 | * amdgpu_device_check_block_size - validate the vm block size | ||
| 866 | * | ||
| 867 | * @adev: amdgpu_device pointer | ||
| 868 | * | ||
| 869 | * Validates the vm block size specified via module parameter. | ||
| 870 | * The vm block size defines number of bits in page table versus page directory, | ||
| 871 | * a page is 4KB so we have 12 bits offset, minimum 9 bits in the | ||
| 872 | * page table and the remaining bits are in the page directory. | ||
| 873 | */ | ||
| 777 | static void amdgpu_device_check_block_size(struct amdgpu_device *adev) | 874 | static void amdgpu_device_check_block_size(struct amdgpu_device *adev) |
| 778 | { | 875 | { |
| 779 | /* defines number of bits in page table versus page directory, | 876 | /* defines number of bits in page table versus page directory, |
| @@ -789,6 +886,14 @@ static void amdgpu_device_check_block_size(struct amdgpu_device *adev) · 9 unmodified lines | |||
| 789 | } | 886 | } |
| 790 | } | 887 | } |
| 791 | 888 | ||
| 889 | /** | ||
| 890 | * amdgpu_device_check_vm_size - validate the vm size | ||
| 891 | * | ||
| 892 | * @adev: amdgpu_device pointer | ||
| 893 | * | ||
| 894 | * Validates the vm size in GB specified via module parameter. | ||
| 895 | * The VM size is the size of the GPU virtual memory space in GB. | ||
| 896 | */ | ||
| 792 | static void amdgpu_device_check_vm_size(struct amdgpu_device *adev) | 897 | static void amdgpu_device_check_vm_size(struct amdgpu_device *adev) |
| 793 | { | 898 | { |
| 794 | /* no need to check the default value */ | 899 | /* no need to check the default value */ |
| @@ -923,6 +1028,17 @@ static const struct vga_switcheroo_client_ops amdgpu_switcheroo_ops = { · 128 unmodified lines | |||
| 923 | .can_switch = amdgpu_switcheroo_can_switch, | 1028 | .can_switch = amdgpu_switcheroo_can_switch, |
| 924 | }; | 1029 | }; |
| 925 | 1030 | ||
| 1031 | /** | ||
| 1032 | * amdgpu_device_ip_set_clockgating_state - set the CG state | ||
| 1033 | * | ||
| 1034 | * @adev: amdgpu_device pointer | ||
| 1035 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | ||
| 1036 | * @state: clockgating state (gate or ungate) | ||
| 1037 | * | ||
| 1038 | * Sets the requested clockgating state for all instances of | ||
| 1039 | * the hardware IP specified. | ||
| 1040 | * Returns the error code from the last instance. | ||
| 1041 | */ | ||
| 926 | int amdgpu_device_ip_set_clockgating_state(struct amdgpu_device *adev, | 1042 | int amdgpu_device_ip_set_clockgating_state(struct amdgpu_device *adev, |
| 927 | enum amd_ip_block_type block_type, | 1043 | enum amd_ip_block_type block_type, |
| 928 | enum amd_clockgating_state state) | 1044 | enum amd_clockgating_state state) |
| @@ -945,6 +1061,17 @@ int amdgpu_device_ip_set_clockgating_state(struct amdgpu_device *adev, · 16 unmodified lines | |||
| 945 | return r; | 1061 | return r; |
| 946 | } | 1062 | } |
| 947 | 1063 | ||
| 1064 | /** | ||
| 1065 | * amdgpu_device_ip_set_powergating_state - set the PG state | ||
| 1066 | * | ||
| 1067 | * @adev: amdgpu_device pointer | ||
| 1068 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | ||
| 1069 | * @state: powergating state (gate or ungate) | ||
| 1070 | * | ||
| 1071 | * Sets the requested powergating state for all instances of | ||
| 1072 | * the hardware IP specified. | ||
| 1073 | * Returns the error code from the last instance. | ||
| 1074 | */ | ||
| 948 | int amdgpu_device_ip_set_powergating_state(struct amdgpu_device *adev, | 1075 | int amdgpu_device_ip_set_powergating_state(struct amdgpu_device *adev, |
| 949 | enum amd_ip_block_type block_type, | 1076 | enum amd_ip_block_type block_type, |
| 950 | enum amd_powergating_state state) | 1077 | enum amd_powergating_state state) |
| @@ -967,6 +1094,17 @@ int amdgpu_device_ip_set_powergating_state(struct amdgpu_device *adev, · 16 unmodified lines | |||
| 967 | return r; | 1094 | return r; |
| 968 | } | 1095 | } |
| 969 | 1096 | ||
| 1097 | /** | ||
| 1098 | * amdgpu_device_ip_get_clockgating_state - get the CG state | ||
| 1099 | * | ||
| 1100 | * @adev: amdgpu_device pointer | ||
| 1101 | * @flags: clockgating feature flags | ||
| 1102 | * | ||
| 1103 | * Walks the list of IPs on the device and updates the clockgating | ||
| 1104 | * flags for each IP. | ||
| 1105 | * Updates @flags with the feature flags for each hardware IP where | ||
| 1106 | * clockgating is enabled. | ||
| 1107 | */ | ||
| 970 | void amdgpu_device_ip_get_clockgating_state(struct amdgpu_device *adev, | 1108 | void amdgpu_device_ip_get_clockgating_state(struct amdgpu_device *adev, |
| 971 | u32 *flags) | 1109 | u32 *flags) |
| 972 | { | 1110 | { |
| @@ -980,6 +1118,15 @@ void amdgpu_device_ip_get_clockgating_state(struct amdgpu_device *adev, · 7 unmodified lines | |||
| 980 | } | 1118 | } |
| 981 | } | 1119 | } |
| 982 | 1120 | ||
| 1121 | /** | ||
| 1122 | * amdgpu_device_ip_wait_for_idle - wait for idle | ||
| 1123 | * | ||
| 1124 | * @adev: amdgpu_device pointer | ||
| 1125 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | ||
| 1126 | * | ||
| 1127 | * Waits for the request hardware IP to be idle. | ||
| 1128 | * Returns 0 for success or a negative error code on failure. | ||
| 1129 | */ | ||
| 983 | int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, | 1130 | int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, |
| 984 | enum amd_ip_block_type block_type) | 1131 | enum amd_ip_block_type block_type) |
| 985 | { | 1132 | { |
| @@ -999,6 +1146,15 @@ int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, · 13 unmodified lines | |||
| 999 | 1146 | ||
| 1000 | } | 1147 | } |
| 1001 | 1148 | ||
| 1149 | /** | ||
| 1150 | * amdgpu_device_ip_is_idle - is the hardware IP idle | ||
| 1151 | * | ||
| 1152 | * @adev: amdgpu_device pointer | ||
| 1153 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | ||
| 1154 | * | ||
| 1155 | * Check if the hardware IP is idle or not. | ||
| 1156 | * Returns true if it the IP is idle, false if not. | ||
| 1157 | */ | ||
| 1002 | bool amdgpu_device_ip_is_idle(struct amdgpu_device *adev, | 1158 | bool amdgpu_device_ip_is_idle(struct amdgpu_device *adev, |
| 1003 | enum amd_ip_block_type block_type) | 1159 | enum amd_ip_block_type block_type) |
| 1004 | { | 1160 | { |
| @@ -1014,6 +1170,15 @@ bool amdgpu_device_ip_is_idle(struct amdgpu_device *adev, · 9 unmodified lines | |||
| 1014 | 1170 | ||
| 1015 | } | 1171 | } |
| 1016 | 1172 | ||
| 1173 | /** | ||
| 1174 | * amdgpu_device_ip_get_ip_block - get a hw IP pointer | ||
| 1175 | * | ||
| 1176 | * @adev: amdgpu_device pointer | ||
| 1177 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | ||
| 1178 | * | ||
| 1179 | * Returns a pointer to the hardware IP block structure | ||
| 1180 | * if it exists for the asic, otherwise NULL. | ||
| 1181 | */ | ||
| 1017 | struct amdgpu_ip_block * | 1182 | struct amdgpu_ip_block * |
| 1018 | amdgpu_device_ip_get_ip_block(struct amdgpu_device *adev, | 1183 | amdgpu_device_ip_get_ip_block(struct amdgpu_device *adev, |
| 1019 | enum amd_ip_block_type type) | 1184 | enum amd_ip_block_type type) |
| @@ -1075,6 +1240,18 @@ int amdgpu_device_ip_block_add(struct amdgpu_device *adev, · 55 unmodified lines | |||
| 1075 | return 0; | 1240 | return 0; |
| 1076 | } | 1241 | } |
| 1077 | 1242 | ||
| 1243 | /** | ||
| 1244 | * amdgpu_device_enable_virtual_display - enable virtual display feature | ||
| 1245 | * | ||
| 1246 | * @adev: amdgpu_device pointer | ||
| 1247 | * | ||
| 1248 | * Enabled the virtual display feature if the user has enabled it via | ||
| 1249 | * the module parameter virtual_display. This feature provides a virtual | ||
| 1250 | * display hardware on headless boards or in virtualized environments. | ||
| 1251 | * This function parses and validates the configuration string specified by | ||
| 1252 | * the user and configues the virtual display configuration (number of | ||
| 1253 | * virtual connectors, crtcs, etc.) specified. | ||
| 1254 | */ | ||
| 1078 | static void amdgpu_device_enable_virtual_display(struct amdgpu_device *adev) | 1255 | static void amdgpu_device_enable_virtual_display(struct amdgpu_device *adev) |
| 1079 | { | 1256 | { |
| 1080 | adev->enable_virtual_display = false; | 1257 | adev->enable_virtual_display = false; |
| @@ -1120,6 +1297,16 @@ static void amdgpu_device_enable_virtual_display(struct amdgpu_device *adev) · 39 unmodified lines | |||
| 1120 | } | 1297 | } |
| 1121 | } | 1298 | } |
| 1122 | 1299 | ||
| 1300 | /** | ||
| 1301 | * amdgpu_device_parse_gpu_info_fw - parse gpu info firmware | ||
| 1302 | * | ||
| 1303 | * @adev: amdgpu_device pointer | ||
| 1304 | * | ||
| 1305 | * Parses the asic configuration parameters specified in the gpu info | ||
| 1306 | * firmware and makes them availale to the driver for use in configuring | ||
| 1307 | * the asic. | ||
| 1308 | * Returns 0 on success, -EINVAL on failure. | ||
| 1309 | */ | ||
| 1123 | static int amdgpu_device_parse_gpu_info_fw(struct amdgpu_device *adev) | 1310 | static int amdgpu_device_parse_gpu_info_fw(struct amdgpu_device *adev) |
| 1124 | { | 1311 | { |
| 1125 | const char *chip_name; | 1312 | const char *chip_name; |
| @@ -1157,6 +1344,9 @@ static int amdgpu_device_parse_gpu_info_fw(struct amdgpu_device *adev) · 31 unmodified lines | |||
| 1157 | case CHIP_VEGA10: | 1344 | case CHIP_VEGA10: |
| 1158 | chip_name = "vega10"; | 1345 | chip_name = "vega10"; |
| 1159 | break; | 1346 | break; |
| 1347 | case CHIP_VEGA12: | ||
| 1348 | chip_name = "vega12"; | ||
| 1349 | break; | ||
| 1160 | case CHIP_RAVEN: | 1350 | case CHIP_RAVEN: |
| 1161 | chip_name = "raven"; | 1351 | chip_name = "raven"; |
| 1162 | break; | 1352 | break; |
| @@ -1218,6 +1408,16 @@ out: · 55 unmodified lines | |||
| 1218 | return err; | 1408 | return err; |
| 1219 | } | 1409 | } |
| 1220 | 1410 | ||
| 1411 | /** | ||
| 1412 | * amdgpu_device_ip_early_init - run early init for hardware IPs | ||
| 1413 | * | ||
| 1414 | * @adev: amdgpu_device pointer | ||
| 1415 | * | ||
| 1416 | * Early initialization pass for hardware IPs. The hardware IPs that make | ||
| 1417 | * up each asic are discovered each IP's early_init callback is run. This | ||
| 1418 | * is the first stage in initializing the asic. | ||
| 1419 | * Returns 0 on success, negative error code on failure. | ||
| 1420 | */ | ||
| 1221 | static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) | 1421 | static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) |
| 1222 | { | 1422 | { |
| 1223 | int i, r; | 1423 | int i, r; |
| @@ -1270,8 +1470,9 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 46 unmodified lines | |||
| 1270 | return r; | 1470 | return r; |
| 1271 | break; | 1471 | break; |
| 1272 | #endif | 1472 | #endif |
| 1273 | case CHIP_VEGA10: | 1473 | case CHIP_VEGA10: |
| 1274 | case CHIP_RAVEN: | 1474 | case CHIP_VEGA12: |
| 1475 | case CHIP_RAVEN: | ||
| 1275 | if (adev->asic_type == CHIP_RAVEN) | 1476 | if (adev->asic_type == CHIP_RAVEN) |
| 1276 | adev->family = AMDGPU_FAMILY_RV; | 1477 | adev->family = AMDGPU_FAMILY_RV; |
| 1277 | else | 1478 | else |
| @@ -1327,6 +1528,17 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 49 unmodified lines | |||
| 1327 | return 0; | 1528 | return 0; |
| 1328 | } | 1529 | } |
| 1329 | 1530 | ||
| 1531 | /** | ||
| 1532 | * amdgpu_device_ip_init - run init for hardware IPs | ||
| 1533 | * | ||
| 1534 | * @adev: amdgpu_device pointer | ||
| 1535 | * | ||
| 1536 | * Main initialization pass for hardware IPs. The list of all the hardware | ||
| 1537 | * IPs that make up the asic is walked and the sw_init and hw_init callbacks | ||
| 1538 | * are run. sw_init initializes the software state associated with each IP | ||
| 1539 | * and hw_init initializes the hardware associated with each IP. | ||
| 1540 | * Returns 0 on success, negative error code on failure. | ||
| 1541 | */ | ||
| 1330 | static int amdgpu_device_ip_init(struct amdgpu_device *adev) | 1542 | static int amdgpu_device_ip_init(struct amdgpu_device *adev) |
| 1331 | { | 1543 | { |
| 1332 | int i, r; | 1544 | int i, r; |
| @@ -1394,17 +1606,47 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) · 61 unmodified lines | |||
| 1394 | return 0; | 1606 | return 0; |
| 1395 | } | 1607 | } |
| 1396 | 1608 | ||
| 1609 | /** | ||
| 1610 | * amdgpu_device_fill_reset_magic - writes reset magic to gart pointer | ||
| 1611 | * | ||
| 1612 | * @adev: amdgpu_device pointer | ||
| 1613 | * | ||
| 1614 | * Writes a reset magic value to the gart pointer in VRAM. The driver calls | ||
| 1615 | * this function before a GPU reset. If the value is retained after a | ||
| 1616 | * GPU reset, VRAM has not been lost. Some GPU resets may destry VRAM contents. | ||
| 1617 | */ | ||
| 1397 | static void amdgpu_device_fill_reset_magic(struct amdgpu_device *adev) | 1618 | static void amdgpu_device_fill_reset_magic(struct amdgpu_device *adev) |
| 1398 | { | 1619 | { |
| 1399 | memcpy(adev->reset_magic, adev->gart.ptr, AMDGPU_RESET_MAGIC_NUM); | 1620 | memcpy(adev->reset_magic, adev->gart.ptr, AMDGPU_RESET_MAGIC_NUM); |
| 1400 | } | 1621 | } |
| 1401 | 1622 | ||
| 1623 | /** | ||
| 1624 | * amdgpu_device_check_vram_lost - check if vram is valid | ||
| 1625 | * | ||
| 1626 | * @adev: amdgpu_device pointer | ||
| 1627 | * | ||
| 1628 | * Checks the reset magic value written to the gart pointer in VRAM. | ||
| 1629 | * The driver calls this after a GPU reset to see if the contents of | ||
| 1630 | * VRAM is lost or now. | ||
| 1631 | * returns true if vram is lost, false if not. | ||
| 1632 | */ | ||
| 1402 | static bool amdgpu_device_check_vram_lost(struct amdgpu_device *adev) | 1633 | static bool amdgpu_device_check_vram_lost(struct amdgpu_device *adev) |
| 1403 | { | 1634 | { |
| 1404 | return !!memcmp(adev->gart.ptr, adev->reset_magic, | 1635 | return !!memcmp(adev->gart.ptr, adev->reset_magic, |
| 1405 | AMDGPU_RESET_MAGIC_NUM); | 1636 | AMDGPU_RESET_MAGIC_NUM); |
| 1406 | } | 1637 | } |
| 1407 | 1638 | ||
| 1639 | /** | ||
| 1640 | * amdgpu_device_ip_late_set_cg_state - late init for clockgating | ||
| 1641 | * | ||
| 1642 | * @adev: amdgpu_device pointer | ||
| 1643 | * | ||
| 1644 | * Late initialization pass enabling clockgating for hardware IPs. | ||
| 1645 | * The list of all the hardware IPs that make up the asic is walked and the | ||
| 1646 | * set_clockgating_state callbacks are run. This stage is run late | ||
| 1647 | * in the init process. | ||
| 1648 | * Returns 0 on success, negative error code on failure. | ||
| 1649 | */ | ||
| 1408 | static int amdgpu_device_ip_late_set_cg_state(struct amdgpu_device *adev) | 1650 | static int amdgpu_device_ip_late_set_cg_state(struct amdgpu_device *adev) |
| 1409 | { | 1651 | { |
| 1410 | int i = 0, r; | 1652 | int i = 0, r; |
| @@ -1432,6 +1674,18 @@ static int amdgpu_device_ip_late_set_cg_state(struct amdgpu_device *adev) · 21 unmodified lines | |||
| 1432 | return 0; | 1674 | return 0; |
| 1433 | } | 1675 | } |
| 1434 | 1676 | ||
| 1677 | /** | ||
| 1678 | * amdgpu_device_ip_late_init - run late init for hardware IPs | ||
| 1679 | * | ||
| 1680 | * @adev: amdgpu_device pointer | ||
| 1681 | * | ||
| 1682 | * Late initialization pass for hardware IPs. The list of all the hardware | ||
| 1683 | * IPs that make up the asic is walked and the late_init callbacks are run. | ||
| 1684 | * late_init covers any special initialization that an IP requires | ||
| 1685 | * after all of the have been initialized or something that needs to happen | ||
| 1686 | * late in the init process. | ||
| 1687 | * Returns 0 on success, negative error code on failure. | ||
| 1688 | */ | ||
| 1435 | static int amdgpu_device_ip_late_init(struct amdgpu_device *adev) | 1689 | static int amdgpu_device_ip_late_init(struct amdgpu_device *adev) |
| 1436 | { | 1690 | { |
| 1437 | int i = 0, r; | 1691 | int i = 0, r; |
| @@ -1458,6 +1712,17 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev) · 20 unmodified lines | |||
| 1458 | return 0; | 1712 | return 0; |
| 1459 | } | 1713 | } |
| 1460 | 1714 | ||
| 1715 | /** | ||
| 1716 | * amdgpu_device_ip_fini - run fini for hardware IPs | ||
| 1717 | * | ||
| 1718 | * @adev: amdgpu_device pointer | ||
| 1719 | * | ||
| 1720 | * Main teardown pass for hardware IPs. The list of all the hardware | ||
| 1721 | * IPs that make up the asic is walked and the hw_fini and sw_fini callbacks | ||
| 1722 | * are run. hw_fini tears down the hardware associated with each IP | ||
| 1723 | * and sw_fini tears down any software state associated with each IP. | ||
| 1724 | * Returns 0 on success, negative error code on failure. | ||
| 1725 | */ | ||
| 1461 | static int amdgpu_device_ip_fini(struct amdgpu_device *adev) | 1726 | static int amdgpu_device_ip_fini(struct amdgpu_device *adev) |
| 1462 | { | 1727 | { |
| 1463 | int i, r; | 1728 | int i, r; |
| @@ -1493,7 +1758,8 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev) · 29 unmodified lines | |||
| 1493 | continue; | 1758 | continue; |
| 1494 | 1759 | ||
| 1495 | if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_UVD && | 1760 | if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_UVD && |
| 1496 | adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCE) { | 1761 | adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCE && |
| 1762 | adev->ip_blocks[i].version->funcs->set_clockgating_state) { | ||
| 1497 | /* ungate blocks before hw fini so that we can shutdown the blocks safely */ | 1763 | /* ungate blocks before hw fini so that we can shutdown the blocks safely */ |
| 1498 | r = adev->ip_blocks[i].version->funcs->set_clockgating_state((void *)adev, | 1764 | r = adev->ip_blocks[i].version->funcs->set_clockgating_state((void *)adev, |
| 1499 | AMD_CG_STATE_UNGATE); | 1765 | AMD_CG_STATE_UNGATE); |
| @@ -1514,8 +1780,6 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev) · 14 unmodified lines | |||
| 1514 | adev->ip_blocks[i].status.hw = false; | 1780 | adev->ip_blocks[i].status.hw = false; |
| 1515 | } | 1781 | } |
| 1516 | 1782 | ||
| 1517 | /* disable all interrupts */ | ||
| 1518 | amdgpu_irq_disable_all(adev); | ||
| 1519 | 1783 | ||
| 1520 | for (i = adev->num_ip_blocks - 1; i >= 0; i--) { | 1784 | for (i = adev->num_ip_blocks - 1; i >= 0; i--) { |
| 1521 | if (!adev->ip_blocks[i].status.sw) | 1785 | if (!adev->ip_blocks[i].status.sw) |
| @@ -1552,6 +1816,15 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev) · 30 unmodified lines | |||
| 1552 | return 0; | 1816 | return 0; |
| 1553 | } | 1817 | } |
| 1554 | 1818 | ||
| 1819 | /** | ||
| 1820 | * amdgpu_device_ip_late_init_func_handler - work handler for clockgating | ||
| 1821 | * | ||
| 1822 | * @work: work_struct | ||
| 1823 | * | ||
| 1824 | * Work handler for amdgpu_device_ip_late_set_cg_state. We put the | ||
| 1825 | * clockgating setup into a worker thread to speed up driver init and | ||
| 1826 | * resume from suspend. | ||
| 1827 | */ | ||
| 1555 | static void amdgpu_device_ip_late_init_func_handler(struct work_struct *work) | 1828 | static void amdgpu_device_ip_late_init_func_handler(struct work_struct *work) |
| 1556 | { | 1829 | { |
| 1557 | struct amdgpu_device *adev = | 1830 | struct amdgpu_device *adev = |
| @@ -1559,6 +1832,17 @@ static void amdgpu_device_ip_late_init_func_handler(struct work_struct *work) · 1 unmodified line | |||
| 1559 | amdgpu_device_ip_late_set_cg_state(adev); | 1832 | amdgpu_device_ip_late_set_cg_state(adev); |
| 1560 | } | 1833 | } |
| 1561 | 1834 | ||
| 1835 | /** | ||
| 1836 | * amdgpu_device_ip_suspend - run suspend for hardware IPs | ||
| 1837 | * | ||
| 1838 | * @adev: amdgpu_device pointer | ||
| 1839 | * | ||
| 1840 | * Main suspend function for hardware IPs. The list of all the hardware | ||
| 1841 | * IPs that make up the asic is walked, clockgating is disabled and the | ||
| 1842 | * suspend callbacks are run. suspend puts the hardware and software state | ||
| 1843 | * in each IP into a state suitable for suspend. | ||
| 1844 | * Returns 0 on success, negative error code on failure. | ||
| 1845 | */ | ||
| 1562 | int amdgpu_device_ip_suspend(struct amdgpu_device *adev) | 1846 | int amdgpu_device_ip_suspend(struct amdgpu_device *adev) |
| 1563 | { | 1847 | { |
| 1564 | int i, r; | 1848 | int i, r; |
| @@ -1667,6 +1951,18 @@ static int amdgpu_device_ip_reinit_late_sriov(struct amdgpu_device *adev) · 102 unmodified lines | |||
| 1667 | return 0; | 1951 | return 0; |
| 1668 | } | 1952 | } |
| 1669 | 1953 | ||
| 1954 | /** | ||
| 1955 | * amdgpu_device_ip_resume_phase1 - run resume for hardware IPs | ||
| 1956 | * | ||
| 1957 | * @adev: amdgpu_device pointer | ||
| 1958 | * | ||
| 1959 | * First resume function for hardware IPs. The list of all the hardware | ||
| 1960 | * IPs that make up the asic is walked and the resume callbacks are run for | ||
| 1961 | * COMMON, GMC, and IH. resume puts the hardware into a functional state | ||
| 1962 | * after a suspend and updates the software state as necessary. This | ||
| 1963 | * function is also used for restoring the GPU after a GPU reset. | ||
| 1964 | * Returns 0 on success, negative error code on failure. | ||
| 1965 | */ | ||
| 1670 | static int amdgpu_device_ip_resume_phase1(struct amdgpu_device *adev) | 1966 | static int amdgpu_device_ip_resume_phase1(struct amdgpu_device *adev) |
| 1671 | { | 1967 | { |
| 1672 | int i, r; | 1968 | int i, r; |
| @@ -1675,9 +1971,8 @@ static int amdgpu_device_ip_resume_phase1(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 1675 | if (!adev->ip_blocks[i].status.valid) | 1971 | if (!adev->ip_blocks[i].status.valid) |
| 1676 | continue; | 1972 | continue; |
| 1677 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_COMMON || | 1973 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_COMMON || |
| 1678 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC || | 1974 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC || |
| 1679 | adev->ip_blocks[i].version->type == | 1975 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_IH) { |
| 1680 | AMD_IP_BLOCK_TYPE_IH) { | ||
| 1681 | r = adev->ip_blocks[i].version->funcs->resume(adev); | 1976 | r = adev->ip_blocks[i].version->funcs->resume(adev); |
| 1682 | if (r) { | 1977 | if (r) { |
| 1683 | DRM_ERROR("resume of IP block <%s> failed %d\n", | 1978 | DRM_ERROR("resume of IP block <%s> failed %d\n", |
| @@ -1690,6 +1985,19 @@ static int amdgpu_device_ip_resume_phase1(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 1690 | return 0; | 1985 | return 0; |
| 1691 | } | 1986 | } |
| 1692 | 1987 | ||
| 1988 | /** | ||
| 1989 | * amdgpu_device_ip_resume_phase2 - run resume for hardware IPs | ||
| 1990 | * | ||
| 1991 | * @adev: amdgpu_device pointer | ||
| 1992 | * | ||
| 1993 | * First resume function for hardware IPs. The list of all the hardware | ||
| 1994 | * IPs that make up the asic is walked and the resume callbacks are run for | ||
| 1995 | * all blocks except COMMON, GMC, and IH. resume puts the hardware into a | ||
| 1996 | * functional state after a suspend and updates the software state as | ||
| 1997 | * necessary. This function is also used for restoring the GPU after a GPU | ||
| 1998 | * reset. | ||
| 1999 | * Returns 0 on success, negative error code on failure. | ||
| 2000 | */ | ||
| 1693 | static int amdgpu_device_ip_resume_phase2(struct amdgpu_device *adev) | 2001 | static int amdgpu_device_ip_resume_phase2(struct amdgpu_device *adev) |
| 1694 | { | 2002 | { |
| 1695 | int i, r; | 2003 | int i, r; |
| @@ -1698,8 +2006,8 @@ static int amdgpu_device_ip_resume_phase2(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 1698 | if (!adev->ip_blocks[i].status.valid) | 2006 | if (!adev->ip_blocks[i].status.valid) |
| 1699 | continue; | 2007 | continue; |
| 1700 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_COMMON || | 2008 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_COMMON || |
| 1701 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC || | 2009 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC || |
| 1702 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_IH ) | 2010 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_IH) |
| 1703 | continue; | 2011 | continue; |
| 1704 | r = adev->ip_blocks[i].version->funcs->resume(adev); | 2012 | r = adev->ip_blocks[i].version->funcs->resume(adev); |
| 1705 | if (r) { | 2013 | if (r) { |
| @@ -1712,6 +2020,18 @@ static int amdgpu_device_ip_resume_phase2(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 1712 | return 0; | 2020 | return 0; |
| 1713 | } | 2021 | } |
| 1714 | 2022 | ||
| 2023 | /** | ||
| 2024 | * amdgpu_device_ip_resume - run resume for hardware IPs | ||
| 2025 | * | ||
| 2026 | * @adev: amdgpu_device pointer | ||
| 2027 | * | ||
| 2028 | * Main resume function for hardware IPs. The hardware IPs | ||
| 2029 | * are split into two resume functions because they are | ||
| 2030 | * are also used in in recovering from a GPU reset and some additional | ||
| 2031 | * steps need to be take between them. In this case (S3/S4) they are | ||
| 2032 | * run sequentially. | ||
| 2033 | * Returns 0 on success, negative error code on failure. | ||
| 2034 | */ | ||
| 1715 | static int amdgpu_device_ip_resume(struct amdgpu_device *adev) | 2035 | static int amdgpu_device_ip_resume(struct amdgpu_device *adev) |
| 1716 | { | 2036 | { |
| 1717 | int r; | 2037 | int r; |
| @@ -1724,6 +2044,13 @@ static int amdgpu_device_ip_resume(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 1724 | return r; | 2044 | return r; |
| 1725 | } | 2045 | } |
| 1726 | 2046 | ||
| 2047 | /** | ||
| 2048 | * amdgpu_device_detect_sriov_bios - determine if the board supports SR-IOV | ||
| 2049 | * | ||
| 2050 | * @adev: amdgpu_device pointer | ||
| 2051 | * | ||
| 2052 | * Query the VBIOS data tables to determine if the board supports SR-IOV. | ||
| 2053 | */ | ||
| 1727 | static void amdgpu_device_detect_sriov_bios(struct amdgpu_device *adev) | 2054 | static void amdgpu_device_detect_sriov_bios(struct amdgpu_device *adev) |
| 1728 | { | 2055 | { |
| 1729 | if (amdgpu_sriov_vf(adev)) { | 2056 | if (amdgpu_sriov_vf(adev)) { |
| @@ -1740,6 +2067,14 @@ static void amdgpu_device_detect_sriov_bios(struct amdgpu_device *adev) · 10 unmodified lines | |||
| 1740 | } | 2067 | } |
| 1741 | } | 2068 | } |
| 1742 | 2069 | ||
| 2070 | /** | ||
| 2071 | * amdgpu_device_asic_has_dc_support - determine if DC supports the asic | ||
| 2072 | * | ||
| 2073 | * @asic_type: AMD asic type | ||
| 2074 | * | ||
| 2075 | * Check if there is DC (new modesetting infrastructre) support for an asic. | ||
| 2076 | * returns true if DC has support, false if not. | ||
| 2077 | */ | ||
| 1743 | bool amdgpu_device_asic_has_dc_support(enum amd_asic_type asic_type) | 2078 | bool amdgpu_device_asic_has_dc_support(enum amd_asic_type asic_type) |
| 1744 | { | 2079 | { |
| 1745 | switch (asic_type) { | 2080 | switch (asic_type) { |
| @@ -1760,6 +2095,7 @@ bool amdgpu_device_asic_has_dc_support(enum amd_asic_type asic_type) · 14 unmodified lines | |||
| 1760 | return amdgpu_dc != 0; | 2095 | return amdgpu_dc != 0; |
| 1761 | #endif | 2096 | #endif |
| 1762 | case CHIP_VEGA10: | 2097 | case CHIP_VEGA10: |
| 2098 | case CHIP_VEGA12: | ||
| 1763 | #if defined(CONFIG_DRM_AMD_DC_DCN1_0) | 2099 | #if defined(CONFIG_DRM_AMD_DC_DCN1_0) |
| 1764 | case CHIP_RAVEN: | 2100 | case CHIP_RAVEN: |
| 1765 | #endif | 2101 | #endif |
| @@ -2017,7 +2353,6 @@ fence_driver_init: · 251 unmodified lines | |||
| 2017 | } | 2353 | } |
| 2018 | dev_err(adev->dev, "amdgpu_device_ip_init failed\n"); | 2354 | dev_err(adev->dev, "amdgpu_device_ip_init failed\n"); |
| 2019 | amdgpu_vf_error_put(adev, AMDGIM_ERROR_VF_AMDGPU_INIT_FAIL, 0, 0); | 2355 | amdgpu_vf_error_put(adev, AMDGIM_ERROR_VF_AMDGPU_INIT_FAIL, 0, 0); |
| 2020 | amdgpu_device_ip_fini(adev); | ||
| 2021 | goto failed; | 2356 | goto failed; |
| 2022 | } | 2357 | } |
| 2023 | 2358 | ||
| @@ -2116,9 +2451,14 @@ void amdgpu_device_fini(struct amdgpu_device *adev) · 92 unmodified lines | |||
| 2116 | 2451 | ||
| 2117 | DRM_INFO("amdgpu: finishing device.\n"); | 2452 | DRM_INFO("amdgpu: finishing device.\n"); |
| 2118 | adev->shutdown = true; | 2453 | adev->shutdown = true; |
| 2119 | if (adev->mode_info.mode_config_initialized) | 2454 | /* disable all interrupts */ |
| 2120 | drm_crtc_force_disable_all(adev->ddev); | 2455 | amdgpu_irq_disable_all(adev); |
| 2121 | 2456 | if (adev->mode_info.mode_config_initialized){ | |
| 2457 | if (!amdgpu_device_has_dc_support(adev)) | ||
| 2458 | drm_crtc_force_disable_all(adev->ddev); | ||
| 2459 | else | ||
| 2460 | drm_atomic_helper_shutdown(adev->ddev); | ||
| 2461 | } | ||
| 2122 | amdgpu_ib_pool_fini(adev); | 2462 | amdgpu_ib_pool_fini(adev); |
| 2123 | amdgpu_fence_driver_fini(adev); | 2463 | amdgpu_fence_driver_fini(adev); |
| 2124 | amdgpu_pm_sysfs_fini(adev); | 2464 | amdgpu_pm_sysfs_fini(adev); |
| @@ -2378,6 +2718,16 @@ unlock: · 253 unmodified lines | |||
| 2378 | return r; | 2718 | return r; |
| 2379 | } | 2719 | } |
| 2380 | 2720 | ||
| 2721 | /** | ||
| 2722 | * amdgpu_device_ip_check_soft_reset - did soft reset succeed | ||
| 2723 | * | ||
| 2724 | * @adev: amdgpu_device pointer | ||
| 2725 | * | ||
| 2726 | * The list of all the hardware IPs that make up the asic is walked and | ||
| 2727 | * the check_soft_reset callbacks are run. check_soft_reset determines | ||
| 2728 | * if the asic is still hung or not. | ||
| 2729 | * Returns true if any of the IPs are still in a hung state, false if not. | ||
| 2730 | */ | ||
| 2381 | static bool amdgpu_device_ip_check_soft_reset(struct amdgpu_device *adev) | 2731 | static bool amdgpu_device_ip_check_soft_reset(struct amdgpu_device *adev) |
| 2382 | { | 2732 | { |
| 2383 | int i; | 2733 | int i; |
| @@ -2400,6 +2750,17 @@ static bool amdgpu_device_ip_check_soft_reset(struct amdgpu_device *adev) · 16 unmodified lines | |||
| 2400 | return asic_hang; | 2750 | return asic_hang; |
| 2401 | } | 2751 | } |
| 2402 | 2752 | ||
| 2753 | /** | ||
| 2754 | * amdgpu_device_ip_pre_soft_reset - prepare for soft reset | ||
| 2755 | * | ||
| 2756 | * @adev: amdgpu_device pointer | ||
| 2757 | * | ||
| 2758 | * The list of all the hardware IPs that make up the asic is walked and the | ||
| 2759 | * pre_soft_reset callbacks are run if the block is hung. pre_soft_reset | ||
| 2760 | * handles any IP specific hardware or software state changes that are | ||
| 2761 | * necessary for a soft reset to succeed. | ||
| 2762 | * Returns 0 on success, negative error code on failure. | ||
| 2763 | */ | ||
| 2403 | static int amdgpu_device_ip_pre_soft_reset(struct amdgpu_device *adev) | 2764 | static int amdgpu_device_ip_pre_soft_reset(struct amdgpu_device *adev) |
| 2404 | { | 2765 | { |
| 2405 | int i, r = 0; | 2766 | int i, r = 0; |
| @@ -2418,6 +2779,15 @@ static int amdgpu_device_ip_pre_soft_reset(struct amdgpu_device *adev) · 12 unmodified lines | |||
| 2418 | return 0; | 2779 | return 0; |
| 2419 | } | 2780 | } |
| 2420 | 2781 | ||
| 2782 | /** | ||
| 2783 | * amdgpu_device_ip_need_full_reset - check if a full asic reset is needed | ||
| 2784 | * | ||
| 2785 | * @adev: amdgpu_device pointer | ||
| 2786 | * | ||
| 2787 | * Some hardware IPs cannot be soft reset. If they are hung, a full gpu | ||
| 2788 | * reset is necessary to recover. | ||
| 2789 | * Returns true if a full asic reset is required, false if not. | ||
| 2790 | */ | ||
| 2421 | static bool amdgpu_device_ip_need_full_reset(struct amdgpu_device *adev) | 2791 | static bool amdgpu_device_ip_need_full_reset(struct amdgpu_device *adev) |
| 2422 | { | 2792 | { |
| 2423 | int i; | 2793 | int i; |
| @@ -2439,6 +2809,17 @@ static bool amdgpu_device_ip_need_full_reset(struct amdgpu_device *adev) · 15 unmodified lines | |||
| 2439 | return false; | 2809 | return false; |
| 2440 | } | 2810 | } |
| 2441 | 2811 | ||
| 2812 | /** | ||
| 2813 | * amdgpu_device_ip_soft_reset - do a soft reset | ||
| 2814 | * | ||
| 2815 | * @adev: amdgpu_device pointer | ||
| 2816 | * | ||
| 2817 | * The list of all the hardware IPs that make up the asic is walked and the | ||
| 2818 | * soft_reset callbacks are run if the block is hung. soft_reset handles any | ||
| 2819 | * IP specific hardware or software state changes that are necessary to soft | ||
| 2820 | * reset the IP. | ||
| 2821 | * Returns 0 on success, negative error code on failure. | ||
| 2822 | */ | ||
| 2442 | static int amdgpu_device_ip_soft_reset(struct amdgpu_device *adev) | 2823 | static int amdgpu_device_ip_soft_reset(struct amdgpu_device *adev) |
| 2443 | { | 2824 | { |
| 2444 | int i, r = 0; | 2825 | int i, r = 0; |
| @@ -2457,6 +2838,17 @@ static int amdgpu_device_ip_soft_reset(struct amdgpu_device *adev) · 12 unmodified lines | |||
| 2457 | return 0; | 2838 | return 0; |
| 2458 | } | 2839 | } |
| 2459 | 2840 | ||
| 2841 | /** | ||
| 2842 | * amdgpu_device_ip_post_soft_reset - clean up from soft reset | ||
| 2843 | * | ||
| 2844 | * @adev: amdgpu_device pointer | ||
| 2845 | * | ||
| 2846 | * The list of all the hardware IPs that make up the asic is walked and the | ||
| 2847 | * post_soft_reset callbacks are run if the asic was hung. post_soft_reset | ||
| 2848 | * handles any IP specific hardware or software state changes that are | ||
| 2849 | * necessary after the IP has been soft reset. | ||
| 2850 | * Returns 0 on success, negative error code on failure. | ||
| 2851 | */ | ||
| 2460 | static int amdgpu_device_ip_post_soft_reset(struct amdgpu_device *adev) | 2852 | static int amdgpu_device_ip_post_soft_reset(struct amdgpu_device *adev) |
| 2461 | { | 2853 | { |
| 2462 | int i, r = 0; | 2854 | int i, r = 0; |
| @@ -2474,6 +2866,19 @@ static int amdgpu_device_ip_post_soft_reset(struct amdgpu_device *adev) · 11 unmodified lines | |||
| 2474 | return 0; | 2866 | return 0; |
| 2475 | } | 2867 | } |
| 2476 | 2868 | ||
| 2869 | /** | ||
| 2870 | * amdgpu_device_recover_vram_from_shadow - restore shadowed VRAM buffers | ||
| 2871 | * | ||
| 2872 | * @adev: amdgpu_device pointer | ||
| 2873 | * @ring: amdgpu_ring for the engine handling the buffer operations | ||
| 2874 | * @bo: amdgpu_bo buffer whose shadow is being restored | ||
| 2875 | * @fence: dma_fence associated with the operation | ||
| 2876 | * | ||
| 2877 | * Restores the VRAM buffer contents from the shadow in GTT. Used to | ||
| 2878 | * restore things like GPUVM page tables after a GPU reset where | ||
| 2879 | * the contents of VRAM might be lost. | ||
| 2880 | * Returns 0 on success, negative error code on failure. | ||
| 2881 | */ | ||
| 2477 | static int amdgpu_device_recover_vram_from_shadow(struct amdgpu_device *adev, | 2882 | static int amdgpu_device_recover_vram_from_shadow(struct amdgpu_device *adev, |
| 2478 | struct amdgpu_ring *ring, | 2883 | struct amdgpu_ring *ring, |
| 2479 | struct amdgpu_bo *bo, | 2884 | struct amdgpu_bo *bo, |
| @@ -2509,6 +2914,16 @@ err: · 29 unmodified lines | |||
| 2509 | return r; | 2914 | return r; |
| 2510 | } | 2915 | } |
| 2511 | 2916 | ||
| 2917 | /** | ||
| 2918 | * amdgpu_device_handle_vram_lost - Handle the loss of VRAM contents | ||
| 2919 | * | ||
| 2920 | * @adev: amdgpu_device pointer | ||
| 2921 | * | ||
| 2922 | * Restores the contents of VRAM buffers from the shadows in GTT. Used to | ||
| 2923 | * restore things like GPUVM page tables after a GPU reset where | ||
| 2924 | * the contents of VRAM might be lost. | ||
| 2925 | * Returns 0 on success, 1 on failure. | ||
| 2926 | */ | ||
| 2512 | static int amdgpu_device_handle_vram_lost(struct amdgpu_device *adev) | 2927 | static int amdgpu_device_handle_vram_lost(struct amdgpu_device *adev) |
| 2513 | { | 2928 | { |
| 2514 | struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; | 2929 | struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; |
| @@ -2562,17 +2977,17 @@ static int amdgpu_device_handle_vram_lost(struct amdgpu_device *adev) · 47 unmodified lines | |||
| 2562 | else | 2977 | else |
| 2563 | DRM_ERROR("recover vram bo from shadow failed\n"); | 2978 | DRM_ERROR("recover vram bo from shadow failed\n"); |
| 2564 | 2979 | ||
| 2565 | return (r > 0?0:1); | 2980 | return (r > 0) ? 0 : 1; |
| 2566 | } | 2981 | } |
| 2567 | 2982 | ||
| 2568 | /* | 2983 | /** |
| 2569 | * amdgpu_device_reset - reset ASIC/GPU for bare-metal or passthrough | 2984 | * amdgpu_device_reset - reset ASIC/GPU for bare-metal or passthrough |
| 2570 | * | 2985 | * |
| 2571 | * @adev: amdgpu device pointer | 2986 | * @adev: amdgpu device pointer |
| 2572 | * | 2987 | * |
| 2573 | * attempt to do soft-reset or full-reset and reinitialize Asic | 2988 | * attempt to do soft-reset or full-reset and reinitialize Asic |
| 2574 | * return 0 means successed otherwise failed | 2989 | * return 0 means successed otherwise failed |
| 2575 | */ | 2990 | */ |
| 2576 | static int amdgpu_device_reset(struct amdgpu_device *adev) | 2991 | static int amdgpu_device_reset(struct amdgpu_device *adev) |
| 2577 | { | 2992 | { |
| 2578 | bool need_full_reset, vram_lost = 0; | 2993 | bool need_full_reset, vram_lost = 0; |
| @@ -2642,15 +3057,16 @@ out: · 63 unmodified lines | |||
| 2642 | return r; | 3057 | return r; |
| 2643 | } | 3058 | } |
| 2644 | 3059 | ||
| 2645 | /* | 3060 | /** |
| 2646 | * amdgpu_device_reset_sriov - reset ASIC for SR-IOV vf | 3061 | * amdgpu_device_reset_sriov - reset ASIC for SR-IOV vf |
| 2647 | * | 3062 | * |
| 2648 | * @adev: amdgpu device pointer | 3063 | * @adev: amdgpu device pointer |
| 2649 | * | 3064 | * |
| 2650 | * do VF FLR and reinitialize Asic | 3065 | * do VF FLR and reinitialize Asic |
| 2651 | * return 0 means successed otherwise failed | 3066 | * return 0 means successed otherwise failed |
| 2652 | */ | 3067 | */ |
| 2653 | static int amdgpu_device_reset_sriov(struct amdgpu_device *adev, bool from_hypervisor) | 3068 | static int amdgpu_device_reset_sriov(struct amdgpu_device *adev, |
| 3069 | bool from_hypervisor) | ||
| 2654 | { | 3070 | { |
| 2655 | int r; | 3071 | int r; |
| 2656 | 3072 | ||
| @@ -2790,6 +3206,15 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 133 unmodified lines | |||
| 2790 | return r; | 3206 | return r; |
| 2791 | } | 3207 | } |
| 2792 | 3208 | ||
| 3209 | /** | ||
| 3210 | * amdgpu_device_get_pcie_info - fence pcie info about the PCIE slot | ||
| 3211 | * | ||
| 3212 | * @adev: amdgpu_device pointer | ||
| 3213 | * | ||
| 3214 | * Fetchs and stores in the driver the PCIE capabilities (gen speed | ||
| 3215 | * and lanes) of the slot the device is in. Handles APUs and | ||
| 3216 | * virtualized environments where PCIE config space may not be available. | ||
| 3217 | */ | ||
| 2793 | static void amdgpu_device_get_pcie_info(struct amdgpu_device *adev) | 3218 | static void amdgpu_device_get_pcie_info(struct amdgpu_device *adev) |
| 2794 | { | 3219 | { |
| 2795 | u32 mask; | 3220 | u32 mask; |
| @@ -544,6 +544,12 @@ static const struct pci_device_id pciidlist[] = { · 543 unmodified lines | |||
| 544 | {0x1002, 0x6868, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, | 544 | {0x1002, 0x6868, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, |
| 545 | {0x1002, 0x686c, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, | 545 | {0x1002, 0x686c, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, |
| 546 | {0x1002, 0x687f, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, | 546 | {0x1002, 0x687f, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, |
| 547 | /* Vega 12 */ | ||
| 548 | {0x1002, 0x69A0, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, | ||
| 549 | {0x1002, 0x69A1, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, | ||
| 550 | {0x1002, 0x69A2, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, | ||
| 551 | {0x1002, 0x69A3, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, | ||
| 552 | {0x1002, 0x69AF, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, | ||
| 547 | /* Raven */ | 553 | /* Raven */ |
| 548 | {0x1002, 0x15dd, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_RAVEN|AMD_IS_APU}, | 554 | {0x1002, 0x15dd, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_RAVEN|AMD_IS_APU}, |
| 549 | 555 | ||
| @@ -435,7 +435,9 @@ int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring, · 434 unmodified lines | |||
| 435 | if (ring->funcs->type != AMDGPU_RING_TYPE_KIQ) { | 435 | if (ring->funcs->type != AMDGPU_RING_TYPE_KIQ) { |
| 436 | r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, | 436 | r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, |
| 437 | num_hw_submission, amdgpu_job_hang_limit, | 437 | num_hw_submission, amdgpu_job_hang_limit, |
| 438 | msecs_to_jiffies(amdgpu_lockup_timeout), ring->name); | 438 | (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) ? |
| 439 | MAX_SCHEDULE_TIMEOUT : msecs_to_jiffies(amdgpu_lockup_timeout), | ||
| 440 | ring->name); | ||
| 439 | if (r) { | 441 | if (r) { |
| 440 | DRM_ERROR("Failed to create scheduler on ring %s.\n", | 442 | DRM_ERROR("Failed to create scheduler on ring %s.\n", |
| 441 | ring->name); | 443 | ring->name); |
| @@ -56,23 +56,11 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size, · 55 unmodified lines | |||
| 56 | alignment = PAGE_SIZE; | 56 | alignment = PAGE_SIZE; |
| 57 | } | 57 | } |
| 58 | 58 | ||
| 59 | retry: | ||
| 60 | r = amdgpu_bo_create(adev, size, alignment, initial_domain, | 59 | r = amdgpu_bo_create(adev, size, alignment, initial_domain, |
| 61 | flags, type, resv, &bo); | 60 | flags, type, resv, &bo); |
| 62 | if (r) { | 61 | if (r) { |
| 63 | if (r != -ERESTARTSYS) { | 62 | DRM_DEBUG("Failed to allocate GEM object (%ld, %d, %u, %d)\n", |
| 64 | if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { | 63 | size, initial_domain, alignment, r); |
| 65 | flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; | ||
| 66 | goto retry; | ||
| 67 | } | ||
| 68 | |||
| 69 | if (initial_domain == AMDGPU_GEM_DOMAIN_VRAM) { | ||
| 70 | initial_domain |= AMDGPU_GEM_DOMAIN_GTT; | ||
| 71 | goto retry; | ||
| 72 | } | ||
| 73 | DRM_DEBUG("Failed to allocate GEM object (%ld, %d, %u, %d)\n", | ||
| 74 | size, initial_domain, alignment, r); | ||
| 75 | } | ||
| 76 | return r; | 64 | return r; |
| 77 | } | 65 | } |
| 78 | *obj = &bo->gem_base; | 66 | *obj = &bo->gem_base; |
| @@ -259,6 +259,7 @@ void amdgpu_irq_fini(struct amdgpu_device *adev) · 258 unmodified lines | |||
| 259 | } | 259 | } |
| 260 | } | 260 | } |
| 261 | kfree(adev->irq.client[i].sources); | 261 | kfree(adev->irq.client[i].sources); |
| 262 | adev->irq.client[i].sources = NULL; | ||
| 262 | } | 263 | } |
| 263 | } | 264 | } |
| 264 | 265 | ||
| @@ -190,6 +190,10 @@ static int amdgpu_firmware_info(struct drm_amdgpu_info_firmware *fw_info, · 189 unmodified lines | |||
| 190 | fw_info->ver = adev->uvd.fw_version; | 190 | fw_info->ver = adev->uvd.fw_version; |
| 191 | fw_info->feature = 0; | 191 | fw_info->feature = 0; |
| 192 | break; | 192 | break; |
| 193 | case AMDGPU_INFO_FW_VCN: | ||
| 194 | fw_info->ver = adev->vcn.fw_version; | ||
| 195 | fw_info->feature = 0; | ||
| 196 | break; | ||
| 193 | case AMDGPU_INFO_FW_GMC: | 197 | case AMDGPU_INFO_FW_GMC: |
| 194 | fw_info->ver = adev->gmc.fw_version; | 198 | fw_info->ver = adev->gmc.fw_version; |
| 195 | fw_info->feature = 0; | 199 | fw_info->feature = 0; |
| @@ -1198,6 +1202,14 @@ static int amdgpu_debugfs_firmware_info(struct seq_file *m, void *data) · 1002 unmodified lines | |||
| 1198 | i, fw_info.feature, fw_info.ver); | 1202 | i, fw_info.feature, fw_info.ver); |
| 1199 | } | 1203 | } |
| 1200 | 1204 | ||
| 1205 | /* VCN */ | ||
| 1206 | query_fw.fw_type = AMDGPU_INFO_FW_VCN; | ||
| 1207 | ret = amdgpu_firmware_info(&fw_info, &query_fw, adev); | ||
| 1208 | if (ret) | ||
| 1209 | return ret; | ||
| 1210 | seq_printf(m, "VCN feature version: %u, firmware version: 0x%08x\n", | ||
| 1211 | fw_info.feature, fw_info.ver); | ||
| 1212 | |||
| 1201 | return 0; | 1213 | return 0; |
| 1202 | } | 1214 | } |
| 1203 | 1215 | ||
| @@ -356,6 +356,7 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, · 355 unmodified lines | |||
| 356 | struct amdgpu_bo *bo; | 356 | struct amdgpu_bo *bo; |
| 357 | unsigned long page_align; | 357 | unsigned long page_align; |
| 358 | size_t acc_size; | 358 | size_t acc_size; |
| 359 | u32 domains; | ||
| 359 | int r; | 360 | int r; |
| 360 | 361 | ||
| 361 | page_align = roundup(byte_align, PAGE_SIZE) >> PAGE_SHIFT; | 362 | page_align = roundup(byte_align, PAGE_SIZE) >> PAGE_SHIFT; |
| @@ -417,12 +418,23 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, · 55 unmodified lines | |||
| 417 | #endif | 418 | #endif |
| 418 | 419 | ||
| 419 | bo->tbo.bdev = &adev->mman.bdev; | 420 | bo->tbo.bdev = &adev->mman.bdev; |
| 420 | amdgpu_ttm_placement_from_domain(bo, domain); | 421 | domains = bo->preferred_domains; |
| 421 | 422 | retry: | |
| 423 | amdgpu_ttm_placement_from_domain(bo, domains); | ||
| 422 | r = ttm_bo_init_reserved(&adev->mman.bdev, &bo->tbo, size, type, | 424 | r = ttm_bo_init_reserved(&adev->mman.bdev, &bo->tbo, size, type, |
| 423 | &bo->placement, page_align, &ctx, acc_size, | 425 | &bo->placement, page_align, &ctx, acc_size, |
| 424 | NULL, resv, &amdgpu_ttm_bo_destroy); | 426 | NULL, resv, &amdgpu_ttm_bo_destroy); |
| 425 | if (unlikely(r != 0)) | 427 | |
| 428 | if (unlikely(r && r != -ERESTARTSYS)) { | ||
| 429 | if (bo->flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { | ||
| 430 | bo->flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; | ||
| 431 | goto retry; | ||
| 432 | } else if (domains != bo->preferred_domains) { | ||
| 433 | domains = bo->allowed_domains; | ||
| 434 | goto retry; | ||
| 435 | } | ||
| 436 | } | ||
| 437 | if (unlikely(r)) | ||
| 426 | return r; | 438 | return r; |
| 427 | 439 | ||
| 428 | if (adev->gmc.visible_vram_size < adev->gmc.real_vram_size && | 440 | if (adev->gmc.visible_vram_size < adev->gmc.real_vram_size && |
| @@ -132,6 +132,7 @@ static int amdgpu_gem_map_attach(struct dma_buf *dma_buf, · 131 unmodified lines | |||
| 132 | { | 132 | { |
| 133 | struct drm_gem_object *obj = dma_buf->priv; | 133 | struct drm_gem_object *obj = dma_buf->priv; |
| 134 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | 134 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); |
| 135 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); | ||
| 135 | long r; | 136 | long r; |
| 136 | 137 | ||
| 137 | r = drm_gem_map_attach(dma_buf, target_dev, attach); | 138 | r = drm_gem_map_attach(dma_buf, target_dev, attach); |
| @@ -143,7 +144,7 @@ static int amdgpu_gem_map_attach(struct dma_buf *dma_buf, · 5 unmodified lines | |||
| 143 | goto error_detach; | 144 | goto error_detach; |
| 144 | 145 | ||
| 145 | 146 | ||
| 146 | if (dma_buf->ops != &amdgpu_dmabuf_ops) { | 147 | if (attach->dev->driver != adev->dev->driver) { |
| 147 | /* | 148 | /* |
| 148 | * Wait for all shared fences to complete before we switch to future | 149 | * Wait for all shared fences to complete before we switch to future |
| 149 | * use of exclusive fence on this prime shared bo. | 150 | * use of exclusive fence on this prime shared bo. |
| @@ -162,7 +163,7 @@ static int amdgpu_gem_map_attach(struct dma_buf *dma_buf, · 12 unmodified lines | |||
| 162 | if (r) | 163 | if (r) |
| 163 | goto error_unreserve; | 164 | goto error_unreserve; |
| 164 | 165 | ||
| 165 | if (dma_buf->ops != &amdgpu_dmabuf_ops) | 166 | if (attach->dev->driver != adev->dev->driver) |
| 166 | bo->prime_shared_count++; | 167 | bo->prime_shared_count++; |
| 167 | 168 | ||
| 168 | error_unreserve: | 169 | error_unreserve: |
| @@ -179,6 +180,7 @@ static void amdgpu_gem_map_detach(struct dma_buf *dma_buf, · 10 unmodified lines | |||
| 179 | { | 180 | { |
| 180 | struct drm_gem_object *obj = dma_buf->priv; | 181 | struct drm_gem_object *obj = dma_buf->priv; |
| 181 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | 182 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); |
| 183 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); | ||
| 182 | int ret = 0; | 184 | int ret = 0; |
| 183 | 185 | ||
| 184 | ret = amdgpu_bo_reserve(bo, true); | 186 | ret = amdgpu_bo_reserve(bo, true); |
| @@ -186,7 +188,7 @@ static void amdgpu_gem_map_detach(struct dma_buf *dma_buf, · 1 unmodified line | |||
| 186 | goto error; | 188 | goto error; |
| 187 | 189 | ||
| 188 | amdgpu_bo_unpin(bo); | 190 | amdgpu_bo_unpin(bo); |
| 189 | if (dma_buf->ops != &amdgpu_dmabuf_ops && bo->prime_shared_count) | 191 | if (attach->dev->driver != adev->dev->driver && bo->prime_shared_count) |
| 190 | bo->prime_shared_count--; | 192 | bo->prime_shared_count--; |
| 191 | amdgpu_bo_unreserve(bo); | 193 | amdgpu_bo_unreserve(bo); |
| 192 | 194 | ||
| @@ -51,6 +51,7 @@ static int psp_sw_init(void *handle) · 50 unmodified lines | |||
| 51 | 51 | ||
| 52 | switch (adev->asic_type) { | 52 | switch (adev->asic_type) { |
| 53 | case CHIP_VEGA10: | 53 | case CHIP_VEGA10: |
| 54 | case CHIP_VEGA12: | ||
| 54 | psp_v3_1_set_psp_funcs(psp); | 55 | psp_v3_1_set_psp_funcs(psp); |
| 55 | break; | 56 | break; |
| 56 | case CHIP_RAVEN: | 57 | case CHIP_RAVEN: |
| @@ -2021,7 +2021,7 @@ static ssize_t amdgpu_iomem_read(struct file *f, char __user *buf, · 2020 unmodified lines | |||
| 2021 | return -EPERM; | 2021 | return -EPERM; |
| 2022 | 2022 | ||
| 2023 | ptr = kmap(p); | 2023 | ptr = kmap(p); |
| 2024 | r = copy_to_user(buf, ptr, bytes); | 2024 | r = copy_to_user(buf, ptr + off, bytes); |
| 2025 | kunmap(p); | 2025 | kunmap(p); |
| 2026 | if (r) | 2026 | if (r) |
| 2027 | return -EFAULT; | 2027 | return -EFAULT; |
| @@ -2065,7 +2065,7 @@ static ssize_t amdgpu_iomem_write(struct file *f, const char __user *buf, · 37 unmodified lines | |||
| 2065 | return -EPERM; | 2065 | return -EPERM; |
| 2066 | 2066 | ||
| 2067 | ptr = kmap(p); | 2067 | ptr = kmap(p); |
| 2068 | r = copy_from_user(ptr, buf, bytes); | 2068 | r = copy_from_user(ptr + off, buf, bytes); |
| 2069 | kunmap(p); | 2069 | kunmap(p); |
| 2070 | if (r) | 2070 | if (r) |
| 2071 | return -EFAULT; | 2071 | return -EFAULT; |
| @@ -271,6 +271,7 @@ amdgpu_ucode_get_load_type(struct amdgpu_device *adev, int load_type) · 270 unmodified lines | |||
| 271 | return AMDGPU_FW_LOAD_SMU; | 271 | return AMDGPU_FW_LOAD_SMU; |
| 272 | case CHIP_VEGA10: | 272 | case CHIP_VEGA10: |
| 273 | case CHIP_RAVEN: | 273 | case CHIP_RAVEN: |
| 274 | case CHIP_VEGA12: | ||
| 274 | if (!load_type) | 275 | if (!load_type) |
| 275 | return AMDGPU_FW_LOAD_DIRECT; | 276 | return AMDGPU_FW_LOAD_DIRECT; |
| 276 | else | 277 | else |
| @@ -68,6 +68,7 @@ · 67 unmodified lines | |||
| 68 | #define FIRMWARE_POLARIS12 "amdgpu/polaris12_uvd.bin" | 68 | #define FIRMWARE_POLARIS12 "amdgpu/polaris12_uvd.bin" |
| 69 | 69 | ||
| 70 | #define FIRMWARE_VEGA10 "amdgpu/vega10_uvd.bin" | 70 | #define FIRMWARE_VEGA10 "amdgpu/vega10_uvd.bin" |
| 71 | #define FIRMWARE_VEGA12 "amdgpu/vega12_uvd.bin" | ||
| 71 | 72 | ||
| 72 | #define mmUVD_GPCOM_VCPU_DATA0_VEGA10 (0x03c4 + 0x7e00) | 73 | #define mmUVD_GPCOM_VCPU_DATA0_VEGA10 (0x03c4 + 0x7e00) |
| 73 | #define mmUVD_GPCOM_VCPU_DATA1_VEGA10 (0x03c5 + 0x7e00) | 74 | #define mmUVD_GPCOM_VCPU_DATA1_VEGA10 (0x03c5 + 0x7e00) |
| @@ -110,6 +111,7 @@ MODULE_FIRMWARE(FIRMWARE_POLARIS11); · 36 unmodified lines | |||
| 110 | MODULE_FIRMWARE(FIRMWARE_POLARIS12); | 111 | MODULE_FIRMWARE(FIRMWARE_POLARIS12); |
| 111 | 112 | ||
| 112 | MODULE_FIRMWARE(FIRMWARE_VEGA10); | 113 | MODULE_FIRMWARE(FIRMWARE_VEGA10); |
| 114 | MODULE_FIRMWARE(FIRMWARE_VEGA12); | ||
| 113 | 115 | ||
| 114 | static void amdgpu_uvd_idle_work_handler(struct work_struct *work); | 116 | static void amdgpu_uvd_idle_work_handler(struct work_struct *work); |
| 115 | 117 | ||
| @@ -161,11 +163,14 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · 45 unmodified lines | |||
| 161 | case CHIP_POLARIS11: | 163 | case CHIP_POLARIS11: |
| 162 | fw_name = FIRMWARE_POLARIS11; | 164 | fw_name = FIRMWARE_POLARIS11; |
| 163 | break; | 165 | break; |
| 166 | case CHIP_POLARIS12: | ||
| 167 | fw_name = FIRMWARE_POLARIS12; | ||
| 168 | break; | ||
| 164 | case CHIP_VEGA10: | 169 | case CHIP_VEGA10: |
| 165 | fw_name = FIRMWARE_VEGA10; | 170 | fw_name = FIRMWARE_VEGA10; |
| 166 | break; | 171 | break; |
| 167 | case CHIP_POLARIS12: | 172 | case CHIP_VEGA12: |
| 168 | fw_name = FIRMWARE_POLARIS12; | 173 | fw_name = FIRMWARE_VEGA12; |
| 169 | break; | 174 | break; |
| 170 | default: | 175 | default: |
| 171 | return -EINVAL; | 176 | return -EINVAL; |
| @@ -55,6 +55,7 @@ · 54 unmodified lines | |||
| 55 | #define FIRMWARE_POLARIS12 "amdgpu/polaris12_vce.bin" | 55 | #define FIRMWARE_POLARIS12 "amdgpu/polaris12_vce.bin" |
| 56 | 56 | ||
| 57 | #define FIRMWARE_VEGA10 "amdgpu/vega10_vce.bin" | 57 | #define FIRMWARE_VEGA10 "amdgpu/vega10_vce.bin" |
| 58 | #define FIRMWARE_VEGA12 "amdgpu/vega12_vce.bin" | ||
| 58 | 59 | ||
| 59 | #ifdef CONFIG_DRM_AMDGPU_CIK | 60 | #ifdef CONFIG_DRM_AMDGPU_CIK |
| 60 | MODULE_FIRMWARE(FIRMWARE_BONAIRE); | 61 | MODULE_FIRMWARE(FIRMWARE_BONAIRE); |
| @@ -72,6 +73,7 @@ MODULE_FIRMWARE(FIRMWARE_POLARIS11); · 11 unmodified lines | |||
| 72 | MODULE_FIRMWARE(FIRMWARE_POLARIS12); | 73 | MODULE_FIRMWARE(FIRMWARE_POLARIS12); |
| 73 | 74 | ||
| 74 | MODULE_FIRMWARE(FIRMWARE_VEGA10); | 75 | MODULE_FIRMWARE(FIRMWARE_VEGA10); |
| 76 | MODULE_FIRMWARE(FIRMWARE_VEGA12); | ||
| 75 | 77 | ||
| 76 | static void amdgpu_vce_idle_work_handler(struct work_struct *work); | 78 | static void amdgpu_vce_idle_work_handler(struct work_struct *work); |
| 77 | 79 | ||
| @@ -127,11 +129,14 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size) · 49 unmodified lines | |||
| 127 | case CHIP_POLARIS11: | 129 | case CHIP_POLARIS11: |
| 128 | fw_name = FIRMWARE_POLARIS11; | 130 | fw_name = FIRMWARE_POLARIS11; |
| 129 | break; | 131 | break; |
| 132 | case CHIP_POLARIS12: | ||
| 133 | fw_name = FIRMWARE_POLARIS12; | ||
| 134 | break; | ||
| 130 | case CHIP_VEGA10: | 135 | case CHIP_VEGA10: |
| 131 | fw_name = FIRMWARE_VEGA10; | 136 | fw_name = FIRMWARE_VEGA10; |
| 132 | break; | 137 | break; |
| 133 | case CHIP_POLARIS12: | 138 | case CHIP_VEGA12: |
| 134 | fw_name = FIRMWARE_POLARIS12; | 139 | fw_name = FIRMWARE_VEGA12; |
| 135 | break; | 140 | break; |
| 136 | 141 | ||
| 137 | default: | 142 | default: |
| @@ -32,6 +32,7 @@ · 31 unmodified lines | |||
| 32 | #include <drm/amdgpu_drm.h> | 32 | #include <drm/amdgpu_drm.h> |
| 33 | #include "amdgpu.h" | 33 | #include "amdgpu.h" |
| 34 | #include "amdgpu_trace.h" | 34 | #include "amdgpu_trace.h" |
| 35 | #include "amdgpu_amdkfd.h" | ||
| 35 | 36 | ||
| 36 | /* | 37 | /* |
| 37 | * GPUVM | 38 | * GPUVM |
| @@ -2405,8 +2406,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 2367 unmodified lines | |||
| 2405 | if (vm->use_cpu_for_update) | 2406 | if (vm->use_cpu_for_update) |
| 2406 | flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; | 2407 | flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; |
| 2407 | else | 2408 | else |
| 2408 | flags |= (AMDGPU_GEM_CREATE_NO_CPU_ACCESS | | 2409 | flags |= AMDGPU_GEM_CREATE_SHADOW; |
| 2409 | AMDGPU_GEM_CREATE_SHADOW); | ||
| 2410 | 2410 | ||
| 2411 | size = amdgpu_vm_bo_size(adev, adev->vm_manager.root_level); | 2411 | size = amdgpu_vm_bo_size(adev, adev->vm_manager.root_level); |
| 2412 | r = amdgpu_bo_create(adev, size, align, AMDGPU_GEM_DOMAIN_VRAM, flags, | 2412 | r = amdgpu_bo_create(adev, size, align, AMDGPU_GEM_DOMAIN_VRAM, flags, |
| @@ -2461,6 +2461,73 @@ error_free_sched_entity: · 48 unmodified lines | |||
| 2461 | return r; | 2461 | return r; |
| 2462 | } | 2462 | } |
| 2463 | 2463 | ||
| 2464 | /** | ||
| 2465 | * amdgpu_vm_make_compute - Turn a GFX VM into a compute VM | ||
| 2466 | * | ||
| 2467 | * This only works on GFX VMs that don't have any BOs added and no | ||
| 2468 | * page tables allocated yet. | ||
| 2469 | * | ||
| 2470 | * Changes the following VM parameters: | ||
| 2471 | * - use_cpu_for_update | ||
| 2472 | * - pte_supports_ats | ||
| 2473 | * - pasid (old PASID is released, because compute manages its own PASIDs) | ||
| 2474 | * | ||
| 2475 | * Reinitializes the page directory to reflect the changed ATS | ||
| 2476 | * setting. May leave behind an unused shadow BO for the page | ||
| 2477 | * directory when switching from SDMA updates to CPU updates. | ||
| 2478 | * | ||
| 2479 | * Returns 0 for success, -errno for errors. | ||
| 2480 | */ | ||
| 2481 | int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm) | ||
| 2482 | { | ||
| 2483 | bool pte_support_ats = (adev->asic_type == CHIP_RAVEN); | ||
| 2484 | int r; | ||
| 2485 | |||
| 2486 | r = amdgpu_bo_reserve(vm->root.base.bo, true); | ||
| 2487 | if (r) | ||
| 2488 | return r; | ||
| 2489 | |||
| 2490 | /* Sanity checks */ | ||
| 2491 | if (!RB_EMPTY_ROOT(&vm->va.rb_root) || vm->root.entries) { | ||
| 2492 | r = -EINVAL; | ||
| 2493 | goto error; | ||
| 2494 | } | ||
| 2495 | |||
| 2496 | /* Check if PD needs to be reinitialized and do it before | ||
| 2497 | * changing any other state, in case it fails. | ||
| 2498 | */ | ||
| 2499 | if (pte_support_ats != vm->pte_support_ats) { | ||
| 2500 | r = amdgpu_vm_clear_bo(adev, vm, vm->root.base.bo, | ||
| 2501 | adev->vm_manager.root_level, | ||
| 2502 | pte_support_ats); | ||
| 2503 | if (r) | ||
| 2504 | goto error; | ||
| 2505 | } | ||
| 2506 | |||
| 2507 | /* Update VM state */ | ||
| 2508 | vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode & | ||
| 2509 | AMDGPU_VM_USE_CPU_FOR_COMPUTE); | ||
| 2510 | vm->pte_support_ats = pte_support_ats; | ||
| 2511 | DRM_DEBUG_DRIVER("VM update mode is %s\n", | ||
| 2512 | vm->use_cpu_for_update ? "CPU" : "SDMA"); | ||
| 2513 | WARN_ONCE((vm->use_cpu_for_update & !amdgpu_vm_is_large_bar(adev)), | ||
| 2514 | "CPU update of VM recommended only for large BAR system\n"); | ||
| 2515 | |||
| 2516 | if (vm->pasid) { | ||
| 2517 | unsigned long flags; | ||
| 2518 | |||
| 2519 | spin_lock_irqsave(&adev->vm_manager.pasid_lock, flags); | ||
| 2520 | idr_remove(&adev->vm_manager.pasid_idr, vm->pasid); | ||
| 2521 | spin_unlock_irqrestore(&adev->vm_manager.pasid_lock, flags); | ||
| 2522 | |||
| 2523 | vm->pasid = 0; | ||
| 2524 | } | ||
| 2525 | |||
| 2526 | error: | ||
| 2527 | amdgpu_bo_unreserve(vm->root.base.bo); | ||
| 2528 | return r; | ||
| 2529 | } | ||
| 2530 | |||
| 2464 | /** | 2531 | /** |
| 2465 | * amdgpu_vm_free_levels - free PD/PT levels | 2532 | * amdgpu_vm_free_levels - free PD/PT levels |
| 2466 | * | 2533 | * |
| @@ -2508,6 +2575,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm) · 41 unmodified lines | |||
| 2508 | u64 fault; | 2575 | u64 fault; |
| 2509 | int i, r; | 2576 | int i, r; |
| 2510 | 2577 | ||
| 2578 | amdgpu_amdkfd_gpuvm_destroy_cb(adev, vm); | ||
| 2579 | |||
| 2511 | /* Clear pending page faults from IH when the VM is destroyed */ | 2580 | /* Clear pending page faults from IH when the VM is destroyed */ |
| 2512 | while (kfifo_get(&vm->faults, &fault)) | 2581 | while (kfifo_get(&vm->faults, &fault)) |
| 2513 | amdgpu_ih_clear_fault(adev, fault); | 2582 | amdgpu_ih_clear_fault(adev, fault); |
| @@ -207,6 +207,15 @@ struct amdgpu_vm { · 206 unmodified lines | |||
| 207 | 207 | ||
| 208 | /* Limit non-retry fault storms */ | 208 | /* Limit non-retry fault storms */ |
| 209 | unsigned int fault_credit; | 209 | unsigned int fault_credit; |
| 210 | |||
| 211 | /* Points to the KFD process VM info */ | ||
| 212 | struct amdkfd_process_info *process_info; | ||
| 213 | |||
| 214 | /* List node in amdkfd_process_info.vm_list_head */ | ||
| 215 | struct list_head vm_list_node; | ||
| 216 | |||
| 217 | /* Valid while the PD is reserved or fenced */ | ||
| 218 | uint64_t pd_phys_addr; | ||
| 210 | }; | 219 | }; |
| 211 | 220 | ||
| 212 | struct amdgpu_vm_manager { | 221 | struct amdgpu_vm_manager { |
| @@ -251,6 +260,7 @@ void amdgpu_vm_manager_init(struct amdgpu_device *adev); · 38 unmodified lines | |||
| 251 | void amdgpu_vm_manager_fini(struct amdgpu_device *adev); | 260 | void amdgpu_vm_manager_fini(struct amdgpu_device *adev); |
| 252 | int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, | 261 | int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, |
| 253 | int vm_context, unsigned int pasid); | 262 | int vm_context, unsigned int pasid); |
| 263 | int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm); | ||
| 254 | void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm); | 264 | void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm); |
| 255 | bool amdgpu_vm_pasid_fault_credit(struct amdgpu_device *adev, | 265 | bool amdgpu_vm_pasid_fault_credit(struct amdgpu_device *adev, |
| 256 | unsigned int pasid); | 266 | unsigned int pasid); |
| @@ -6244,6 +6244,7 @@ static int ci_dpm_early_init(void *handle) · 6243 unmodified lines | |||
| 6244 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; | 6244 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; |
| 6245 | 6245 | ||
| 6246 | adev->powerplay.pp_funcs = &ci_dpm_funcs; | 6246 | adev->powerplay.pp_funcs = &ci_dpm_funcs; |
| 6247 | adev->powerplay.pp_handle = adev; | ||
| 6247 | ci_dpm_set_irq_funcs(adev); | 6248 | ci_dpm_set_irq_funcs(adev); |
| 6248 | 6249 | ||
| 6249 | return 0; | 6250 | return 0; |
| @@ -473,6 +473,7 @@ static int dce_virtual_hw_init(void *handle) · 472 unmodified lines | |||
| 473 | /* no DCE */ | 473 | /* no DCE */ |
| 474 | break; | 474 | break; |
| 475 | case CHIP_VEGA10: | 475 | case CHIP_VEGA10: |
| 476 | case CHIP_VEGA12: | ||
| 476 | break; | 477 | break; |
| 477 | default: | 478 | default: |
| 478 | DRM_ERROR("Virtual display unsupported ASIC type: 0x%X\n", adev->asic_type); | 479 | DRM_ERROR("Virtual display unsupported ASIC type: 0x%X\n", adev->asic_type); |
| @@ -57,6 +57,13 @@ MODULE_FIRMWARE("amdgpu/vega10_mec.bin"); · 56 unmodified lines | |||
| 57 | MODULE_FIRMWARE("amdgpu/vega10_mec2.bin"); | 57 | MODULE_FIRMWARE("amdgpu/vega10_mec2.bin"); |
| 58 | MODULE_FIRMWARE("amdgpu/vega10_rlc.bin"); | 58 | MODULE_FIRMWARE("amdgpu/vega10_rlc.bin"); |
| 59 | 59 | ||
| 60 | MODULE_FIRMWARE("amdgpu/vega12_ce.bin"); | ||
| 61 | MODULE_FIRMWARE("amdgpu/vega12_pfp.bin"); | ||
| 62 | MODULE_FIRMWARE("amdgpu/vega12_me.bin"); | ||
| 63 | MODULE_FIRMWARE("amdgpu/vega12_mec.bin"); | ||
| 64 | MODULE_FIRMWARE("amdgpu/vega12_mec2.bin"); | ||
| 65 | MODULE_FIRMWARE("amdgpu/vega12_rlc.bin"); | ||
| 66 | |||
| 60 | MODULE_FIRMWARE("amdgpu/raven_ce.bin"); | 67 | MODULE_FIRMWARE("amdgpu/raven_ce.bin"); |
| 61 | MODULE_FIRMWARE("amdgpu/raven_pfp.bin"); | 68 | MODULE_FIRMWARE("amdgpu/raven_pfp.bin"); |
| 62 | MODULE_FIRMWARE("amdgpu/raven_me.bin"); | 69 | MODULE_FIRMWARE("amdgpu/raven_me.bin"); |
| @@ -144,7 +151,42 @@ static const struct soc15_reg_golden golden_settings_gc_9_x_common[] = · 81 unmodified lines | |||
| 144 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmGRBM_CAM_DATA, 0xffffffff, 0x2544c382) | 151 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmGRBM_CAM_DATA, 0xffffffff, 0x2544c382) |
| 145 | }; | 152 | }; |
| 146 | 153 | ||
| 154 | static const struct soc15_reg_golden golden_settings_gc_9_2_1[] = | ||
| 155 | { | ||
| 156 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG2, 0xf00fffff, 0x00000420), | ||
| 157 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_GPU_ID, 0x0000000f, 0x00000000), | ||
| 158 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_BINNER_EVENT_CNTL_3, 0x00000003, 0x82400024), | ||
| 159 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_ENHANCE, 0x3fffffff, 0x00000001), | ||
| 160 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000), | ||
| 161 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSH_MEM_CONFIG, 0x00001000, 0x00001000), | ||
| 162 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_CU_0, 0x0007ffff, 0x00000800), | ||
| 163 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_CU_1, 0x0007ffff, 0x00000800), | ||
| 164 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_EN_CU_0, 0x01ffffff, 0x0000ff87), | ||
| 165 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_EN_CU_1, 0x01ffffff, 0x0000ff8f), | ||
| 166 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSQC_CONFIG, 0x03000000, 0x020a2000), | ||
| 167 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmTA_CNTL_AUX, 0xfffffeef, 0x010b0000), | ||
| 168 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CHAN_STEER_HI, 0xffffffff, 0x4a2c0e68), | ||
| 169 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CHAN_STEER_LO, 0xffffffff, 0xb5d3f197), | ||
| 170 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmVGT_CACHE_INVALIDATION, 0x3fff3af3, 0x19200000), | ||
| 171 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmVGT_GS_MAX_WAVE_ID, 0x00000fff, 0x000003ff) | ||
| 172 | }; | ||
| 173 | |||
| 174 | static const struct soc15_reg_golden golden_settings_gc_9_2_1_vg12[] = | ||
| 175 | { | ||
| 176 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_DCC_CONFIG, 0x00000080, 0x04000080), | ||
| 177 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_HW_CONTROL, 0xfffdf3cf, 0x00014104), | ||
| 178 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_HW_CONTROL_2, 0x0f000000, 0x0a000000), | ||
| 179 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_ADDR_CONFIG, 0xffff77ff, 0x24104041), | ||
| 180 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_ADDR_CONFIG_READ, 0xffff77ff, 0x24104041), | ||
| 181 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_ENHANCE_1, 0xffffffff, 0x04040000), | ||
| 182 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_CONFIG_CNTL_1, 0xffff03ff, 0x01000107), | ||
| 183 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CHAN_STEER_HI, 0xffffffff, 0x00000000), | ||
| 184 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CHAN_STEER_LO, 0xffffffff, 0x76325410), | ||
| 185 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmTD_CNTL, 0x01bd9f33, 0x01000000) | ||
| 186 | }; | ||
| 187 | |||
| 147 | #define VEGA10_GB_ADDR_CONFIG_GOLDEN 0x2a114042 | 188 | #define VEGA10_GB_ADDR_CONFIG_GOLDEN 0x2a114042 |
| 189 | #define VEGA12_GB_ADDR_CONFIG_GOLDEN 0x24104041 | ||
| 148 | #define RAVEN_GB_ADDR_CONFIG_GOLDEN 0x24000042 | 190 | #define RAVEN_GB_ADDR_CONFIG_GOLDEN 0x24000042 |
| 149 | 191 | ||
| 150 | static void gfx_v9_0_set_ring_funcs(struct amdgpu_device *adev); | 192 | static void gfx_v9_0_set_ring_funcs(struct amdgpu_device *adev); |
| @@ -168,6 +210,14 @@ static void gfx_v9_0_init_golden_registers(struct amdgpu_device *adev) · 17 unmodified lines | |||
| 168 | golden_settings_gc_9_0_vg10, | 210 | golden_settings_gc_9_0_vg10, |
| 169 | ARRAY_SIZE(golden_settings_gc_9_0_vg10)); | 211 | ARRAY_SIZE(golden_settings_gc_9_0_vg10)); |
| 170 | break; | 212 | break; |
| 213 | case CHIP_VEGA12: | ||
| 214 | soc15_program_register_sequence(adev, | ||
| 215 | golden_settings_gc_9_2_1, | ||
| 216 | ARRAY_SIZE(golden_settings_gc_9_2_1)); | ||
| 217 | soc15_program_register_sequence(adev, | ||
| 218 | golden_settings_gc_9_2_1_vg12, | ||
| 219 | ARRAY_SIZE(golden_settings_gc_9_2_1_vg12)); | ||
| 220 | break; | ||
| 171 | case CHIP_RAVEN: | 221 | case CHIP_RAVEN: |
| 172 | soc15_program_register_sequence(adev, | 222 | soc15_program_register_sequence(adev, |
| 173 | golden_settings_gc_9_1, | 223 | golden_settings_gc_9_1, |
| @@ -369,6 +419,9 @@ static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) · 195 unmodified lines | |||
| 369 | case CHIP_VEGA10: | 419 | case CHIP_VEGA10: |
| 370 | chip_name = "vega10"; | 420 | chip_name = "vega10"; |
| 371 | break; | 421 | break; |
| 422 | case CHIP_VEGA12: | ||
| 423 | chip_name = "vega12"; | ||
| 424 | break; | ||
| 372 | case CHIP_RAVEN: | 425 | case CHIP_RAVEN: |
| 373 | chip_name = "raven"; | 426 | chip_name = "raven"; |
| 374 | break; | 427 | break; |
| @@ -968,6 +1021,15 @@ static void gfx_v9_0_gpu_early_init(struct amdgpu_device *adev) · 593 unmodified lines | |||
| 968 | adev->gfx.config.sc_earlyz_tile_fifo_size = 0x4C0; | 1021 | adev->gfx.config.sc_earlyz_tile_fifo_size = 0x4C0; |
| 969 | gb_addr_config = VEGA10_GB_ADDR_CONFIG_GOLDEN; | 1022 | gb_addr_config = VEGA10_GB_ADDR_CONFIG_GOLDEN; |
| 970 | break; | 1023 | break; |
| 1024 | case CHIP_VEGA12: | ||
| 1025 | adev->gfx.config.max_hw_contexts = 8; | ||
| 1026 | adev->gfx.config.sc_prim_fifo_size_frontend = 0x20; | ||
| 1027 | adev->gfx.config.sc_prim_fifo_size_backend = 0x100; | ||
| 1028 | adev->gfx.config.sc_hiz_tile_fifo_size = 0x30; | ||
| 1029 | adev->gfx.config.sc_earlyz_tile_fifo_size = 0x4C0; | ||
| 1030 | gb_addr_config = VEGA12_GB_ADDR_CONFIG_GOLDEN; | ||
| 1031 | DRM_INFO("fix gfx.config for vega12\n"); | ||
| 1032 | break; | ||
| 971 | case CHIP_RAVEN: | 1033 | case CHIP_RAVEN: |
| 972 | adev->gfx.config.max_hw_contexts = 8; | 1034 | adev->gfx.config.max_hw_contexts = 8; |
| 973 | adev->gfx.config.sc_prim_fifo_size_frontend = 0x20; | 1035 | adev->gfx.config.sc_prim_fifo_size_frontend = 0x20; |
| @@ -1249,6 +1311,7 @@ static int gfx_v9_0_sw_init(void *handle) · 275 unmodified lines | |||
| 1249 | 1311 | ||
| 1250 | switch (adev->asic_type) { | 1312 | switch (adev->asic_type) { |
| 1251 | case CHIP_VEGA10: | 1313 | case CHIP_VEGA10: |
| 1314 | case CHIP_VEGA12: | ||
| 1252 | case CHIP_RAVEN: | 1315 | case CHIP_RAVEN: |
| 1253 | adev->gfx.mec.num_mec = 2; | 1316 | adev->gfx.mec.num_mec = 2; |
| 1254 | break; | 1317 | break; |
| @@ -3482,6 +3545,7 @@ static int gfx_v9_0_set_clockgating_state(void *handle, · 2227 unmodified lines | |||
| 3482 | 3545 | ||
| 3483 | switch (adev->asic_type) { | 3546 | switch (adev->asic_type) { |
| 3484 | case CHIP_VEGA10: | 3547 | case CHIP_VEGA10: |
| 3548 | case CHIP_VEGA12: | ||
| 3485 | case CHIP_RAVEN: | 3549 | case CHIP_RAVEN: |
| 3486 | gfx_v9_0_update_gfx_clock_gating(adev, | 3550 | gfx_v9_0_update_gfx_clock_gating(adev, |
| 3487 | state == AMD_CG_STATE_GATE ? true : false); | 3551 | state == AMD_CG_STATE_GATE ? true : false); |
| @@ -4453,6 +4517,7 @@ static void gfx_v9_0_set_rlc_funcs(struct amdgpu_device *adev) · 965 unmodified lines | |||
| 4453 | { | 4517 | { |
| 4454 | switch (adev->asic_type) { | 4518 | switch (adev->asic_type) { |
| 4455 | case CHIP_VEGA10: | 4519 | case CHIP_VEGA10: |
| 4520 | case CHIP_VEGA12: | ||
| 4456 | case CHIP_RAVEN: | 4521 | case CHIP_RAVEN: |
| 4457 | adev->gfx.rlc.funcs = &gfx_v9_0_rlc_funcs; | 4522 | adev->gfx.rlc.funcs = &gfx_v9_0_rlc_funcs; |
| 4458 | break; | 4523 | break; |
| @@ -791,6 +791,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev) · 790 unmodified lines | |||
| 791 | if (amdgpu_gart_size == -1) { | 791 | if (amdgpu_gart_size == -1) { |
| 792 | switch (adev->asic_type) { | 792 | switch (adev->asic_type) { |
| 793 | case CHIP_VEGA10: /* all engines support GPUVM */ | 793 | case CHIP_VEGA10: /* all engines support GPUVM */ |
| 794 | case CHIP_VEGA12: /* all engines support GPUVM */ | ||
| 794 | default: | 795 | default: |
| 795 | adev->gmc.gart_size = 512ULL << 20; | 796 | adev->gmc.gart_size = 512ULL << 20; |
| 796 | break; | 797 | break; |
| @@ -849,6 +850,7 @@ static int gmc_v9_0_sw_init(void *handle) · 52 unmodified lines | |||
| 849 | } | 850 | } |
| 850 | break; | 851 | break; |
| 851 | case CHIP_VEGA10: | 852 | case CHIP_VEGA10: |
| 853 | case CHIP_VEGA12: | ||
| 852 | /* | 854 | /* |
| 853 | * To fulfill 4-level page support, | 855 | * To fulfill 4-level page support, |
| 854 | * vm size is 256TB (48bit), maximum size of Vega10, | 856 | * vm size is 256TB (48bit), maximum size of Vega10, |
| @@ -965,6 +967,8 @@ static void gmc_v9_0_init_golden_registers(struct amdgpu_device *adev) · 110 unmodified lines | |||
| 965 | golden_settings_athub_1_0_0, | 967 | golden_settings_athub_1_0_0, |
| 966 | ARRAY_SIZE(golden_settings_athub_1_0_0)); | 968 | ARRAY_SIZE(golden_settings_athub_1_0_0)); |
| 967 | break; | 969 | break; |
| 970 | case CHIP_VEGA12: | ||
| 971 | break; | ||
| 968 | case CHIP_RAVEN: | 972 | case CHIP_RAVEN: |
| 969 | soc15_program_register_sequence(adev, | 973 | soc15_program_register_sequence(adev, |
| 970 | golden_settings_athub_1_0_0, | 974 | golden_settings_athub_1_0_0, |
| @@ -2963,6 +2963,7 @@ static int kv_dpm_early_init(void *handle) · 2962 unmodified lines | |||
| 2963 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; | 2963 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; |
| 2964 | 2964 | ||
| 2965 | adev->powerplay.pp_funcs = &kv_dpm_funcs; | 2965 | adev->powerplay.pp_funcs = &kv_dpm_funcs; |
| 2966 | adev->powerplay.pp_handle = adev; | ||
| 2966 | kv_dpm_set_irq_funcs(adev); | 2967 | kv_dpm_set_irq_funcs(adev); |
| 2967 | 2968 | ||
| 2968 | return 0; | 2969 | return 0; |
| @@ -733,6 +733,7 @@ int mmhub_v1_0_set_clockgating(struct amdgpu_device *adev, · 732 unmodified lines | |||
| 733 | 733 | ||
| 734 | switch (adev->asic_type) { | 734 | switch (adev->asic_type) { |
| 735 | case CHIP_VEGA10: | 735 | case CHIP_VEGA10: |
| 736 | case CHIP_VEGA12: | ||
| 736 | case CHIP_RAVEN: | 737 | case CHIP_RAVEN: |
| 737 | mmhub_v1_0_update_medium_grain_clock_gating(adev, | 738 | mmhub_v1_0_update_medium_grain_clock_gating(adev, |
| 738 | state == AMD_CG_STATE_GATE ? true : false); | 739 | state == AMD_CG_STATE_GATE ? true : false); |
| @@ -129,7 +129,7 @@ static void xgpu_ai_mailbox_trans_msg (struct amdgpu_device *adev, · 128 unmodified lines | |||
| 129 | xgpu_ai_mailbox_set_valid(adev, false); | 129 | xgpu_ai_mailbox_set_valid(adev, false); |
| 130 | trn = xgpu_ai_peek_ack(adev); | 130 | trn = xgpu_ai_peek_ack(adev); |
| 131 | if (trn) { | 131 | if (trn) { |
| 132 | pr_err("trn=%x ACK should not asssert! wait again !\n", trn); | 132 | pr_err("trn=%x ACK should not assert! wait again !\n", trn); |
| 133 | msleep(1); | 133 | msleep(1); |
| 134 | } | 134 | } |
| 135 | } while(trn); | 135 | } while(trn); |
| @@ -220,12 +220,12 @@ static u32 nbio_v6_1_get_hdp_flush_done_offset(struct amdgpu_device *adev) · 219 unmodified lines | |||
| 220 | 220 | ||
| 221 | static u32 nbio_v6_1_get_pcie_index_offset(struct amdgpu_device *adev) | 221 | static u32 nbio_v6_1_get_pcie_index_offset(struct amdgpu_device *adev) |
| 222 | { | 222 | { |
| 223 | return SOC15_REG_OFFSET(NBIO, 0, mmPCIE_INDEX); | 223 | return SOC15_REG_OFFSET(NBIO, 0, mmPCIE_INDEX2); |
| 224 | } | 224 | } |
| 225 | 225 | ||
| 226 | static u32 nbio_v6_1_get_pcie_data_offset(struct amdgpu_device *adev) | 226 | static u32 nbio_v6_1_get_pcie_data_offset(struct amdgpu_device *adev) |
| 227 | { | 227 | { |
| 228 | return SOC15_REG_OFFSET(NBIO, 0, mmPCIE_DATA); | 228 | return SOC15_REG_OFFSET(NBIO, 0, mmPCIE_DATA2); |
| 229 | } | 229 | } |
| 230 | 230 | ||
| 231 | static const struct nbio_hdp_flush_reg nbio_v6_1_hdp_flush_reg = { | 231 | static const struct nbio_hdp_flush_reg nbio_v6_1_hdp_flush_reg = { |
| @@ -39,6 +39,8 @@ · 38 unmodified lines | |||
| 39 | 39 | ||
| 40 | MODULE_FIRMWARE("amdgpu/vega10_sos.bin"); | 40 | MODULE_FIRMWARE("amdgpu/vega10_sos.bin"); |
| 41 | MODULE_FIRMWARE("amdgpu/vega10_asd.bin"); | 41 | MODULE_FIRMWARE("amdgpu/vega10_asd.bin"); |
| 42 | MODULE_FIRMWARE("amdgpu/vega12_sos.bin"); | ||
| 43 | MODULE_FIRMWARE("amdgpu/vega12_asd.bin"); | ||
| 42 | 44 | ||
| 43 | #define smnMP1_FIRMWARE_FLAGS 0x3010028 | 45 | #define smnMP1_FIRMWARE_FLAGS 0x3010028 |
| 44 | 46 | ||
| @@ -107,6 +109,9 @@ static int psp_v3_1_init_microcode(struct psp_context *psp) · 62 unmodified lines | |||
| 107 | case CHIP_VEGA10: | 109 | case CHIP_VEGA10: |
| 108 | chip_name = "vega10"; | 110 | chip_name = "vega10"; |
| 109 | break; | 111 | break; |
| 112 | case CHIP_VEGA12: | ||
| 113 | chip_name = "vega12"; | ||
| 114 | break; | ||
| 110 | default: BUG(); | 115 | default: BUG(); |
| 111 | } | 116 | } |
| 112 | 117 | ||
| @@ -40,6 +40,8 @@ · 39 unmodified lines | |||
| 40 | 40 | ||
| 41 | MODULE_FIRMWARE("amdgpu/vega10_sdma.bin"); | 41 | MODULE_FIRMWARE("amdgpu/vega10_sdma.bin"); |
| 42 | MODULE_FIRMWARE("amdgpu/vega10_sdma1.bin"); | 42 | MODULE_FIRMWARE("amdgpu/vega10_sdma1.bin"); |
| 43 | MODULE_FIRMWARE("amdgpu/vega12_sdma.bin"); | ||
| 44 | MODULE_FIRMWARE("amdgpu/vega12_sdma1.bin"); | ||
| 43 | MODULE_FIRMWARE("amdgpu/raven_sdma.bin"); | 45 | MODULE_FIRMWARE("amdgpu/raven_sdma.bin"); |
| 44 | 46 | ||
| 45 | #define SDMA0_POWER_CNTL__ON_OFF_CONDITION_HOLD_TIME_MASK 0x000000F8L | 47 | #define SDMA0_POWER_CNTL__ON_OFF_CONDITION_HOLD_TIME_MASK 0x000000F8L |
| @@ -84,6 +86,13 @@ static const struct soc15_reg_golden golden_settings_sdma_vg10[] = { · 38 unmodified lines | |||
| 84 | SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GB_ADDR_CONFIG_READ, 0x0018773f, 0x00104002) | 86 | SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GB_ADDR_CONFIG_READ, 0x0018773f, 0x00104002) |
| 85 | }; | 87 | }; |
| 86 | 88 | ||
| 89 | static const struct soc15_reg_golden golden_settings_sdma_vg12[] = { | ||
| 90 | SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_GB_ADDR_CONFIG, 0x0018773f, 0x00104001), | ||
| 91 | SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_GB_ADDR_CONFIG_READ, 0x0018773f, 0x00104001), | ||
| 92 | SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GB_ADDR_CONFIG, 0x0018773f, 0x00104001), | ||
| 93 | SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GB_ADDR_CONFIG_READ, 0x0018773f, 0x00104001) | ||
| 94 | }; | ||
| 95 | |||
| 87 | static const struct soc15_reg_golden golden_settings_sdma_4_1[] = | 96 | static const struct soc15_reg_golden golden_settings_sdma_4_1[] = |
| 88 | { | 97 | { |
| 89 | SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_CHICKEN_BITS, 0xfe931f07, 0x02831d07), | 98 | SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_CHICKEN_BITS, 0xfe931f07, 0x02831d07), |
| @@ -122,6 +131,14 @@ static void sdma_v4_0_init_golden_registers(struct amdgpu_device *adev) · 32 unmodified lines | |||
| 122 | golden_settings_sdma_vg10, | 131 | golden_settings_sdma_vg10, |
| 123 | ARRAY_SIZE(golden_settings_sdma_vg10)); | 132 | ARRAY_SIZE(golden_settings_sdma_vg10)); |
| 124 | break; | 133 | break; |
| 134 | case CHIP_VEGA12: | ||
| 135 | soc15_program_register_sequence(adev, | ||
| 136 | golden_settings_sdma_4, | ||
| 137 | ARRAY_SIZE(golden_settings_sdma_4)); | ||
| 138 | soc15_program_register_sequence(adev, | ||
| 139 | golden_settings_sdma_vg12, | ||
| 140 | ARRAY_SIZE(golden_settings_sdma_vg12)); | ||
| 141 | break; | ||
| 125 | case CHIP_RAVEN: | 142 | case CHIP_RAVEN: |
| 126 | soc15_program_register_sequence(adev, | 143 | soc15_program_register_sequence(adev, |
| 127 | golden_settings_sdma_4_1, | 144 | golden_settings_sdma_4_1, |
| @@ -162,6 +179,9 @@ static int sdma_v4_0_init_microcode(struct amdgpu_device *adev) · 34 unmodified lines | |||
| 162 | case CHIP_VEGA10: | 179 | case CHIP_VEGA10: |
| 163 | chip_name = "vega10"; | 180 | chip_name = "vega10"; |
| 164 | break; | 181 | break; |
| 182 | case CHIP_VEGA12: | ||
| 183 | chip_name = "vega12"; | ||
| 184 | break; | ||
| 165 | case CHIP_RAVEN: | 185 | case CHIP_RAVEN: |
| 166 | chip_name = "raven"; | 186 | chip_name = "raven"; |
| 167 | break; | 187 | break; |
| @@ -1489,6 +1509,7 @@ static int sdma_v4_0_set_clockgating_state(void *handle, · 1321 unmodified lines | |||
| 1489 | 1509 | ||
| 1490 | switch (adev->asic_type) { | 1510 | switch (adev->asic_type) { |
| 1491 | case CHIP_VEGA10: | 1511 | case CHIP_VEGA10: |
| 1512 | case CHIP_VEGA12: | ||
| 1492 | case CHIP_RAVEN: | 1513 | case CHIP_RAVEN: |
| 1493 | sdma_v4_0_update_medium_grain_clock_gating(adev, | 1514 | sdma_v4_0_update_medium_grain_clock_gating(adev, |
| 1494 | state == AMD_CG_STATE_GATE ? true : false); | 1515 | state == AMD_CG_STATE_GATE ? true : false); |
| @@ -1618,7 +1639,7 @@ static void sdma_v4_0_set_irq_funcs(struct amdgpu_device *adev) · 123 unmodified lines | |||
| 1618 | * @dst_offset: dst GPU address | 1639 | * @dst_offset: dst GPU address |
| 1619 | * @byte_count: number of bytes to xfer | 1640 | * @byte_count: number of bytes to xfer |
| 1620 | * | 1641 | * |
| 1621 | * Copy GPU buffers using the DMA engine (VEGA10). | 1642 | * Copy GPU buffers using the DMA engine (VEGA10/12). |
| 1622 | * Used by the amdgpu ttm implementation to move pages if | 1643 | * Used by the amdgpu ttm implementation to move pages if |
| 1623 | * registered as the asic copy callback. | 1644 | * registered as the asic copy callback. |
| 1624 | */ | 1645 | */ |
| @@ -1645,7 +1666,7 @@ static void sdma_v4_0_emit_copy_buffer(struct amdgpu_ib *ib, · 20 unmodified lines | |||
| 1645 | * @dst_offset: dst GPU address | 1666 | * @dst_offset: dst GPU address |
| 1646 | * @byte_count: number of bytes to xfer | 1667 | * @byte_count: number of bytes to xfer |
| 1647 | * | 1668 | * |
| 1648 | * Fill GPU buffers using the DMA engine (VEGA10). | 1669 | * Fill GPU buffers using the DMA engine (VEGA10/12). |
| 1649 | */ | 1670 | */ |
| 1650 | static void sdma_v4_0_emit_fill_buffer(struct amdgpu_ib *ib, | 1671 | static void sdma_v4_0_emit_fill_buffer(struct amdgpu_ib *ib, |
| 1651 | uint32_t src_data, | 1672 | uint32_t src_data, |
| @@ -7917,6 +7917,7 @@ static int si_dpm_early_init(void *handle) · 7916 unmodified lines | |||
| 7917 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; | 7917 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; |
| 7918 | 7918 | ||
| 7919 | adev->powerplay.pp_funcs = &si_dpm_funcs; | 7919 | adev->powerplay.pp_funcs = &si_dpm_funcs; |
| 7920 | adev->powerplay.pp_handle = adev; | ||
| 7920 | si_dpm_set_irq_funcs(adev); | 7921 | si_dpm_set_irq_funcs(adev); |
| 7921 | return 0; | 7922 | return 0; |
| 7922 | } | 7923 | } |
| @@ -508,6 +508,7 @@ int soc15_set_ip_blocks(struct amdgpu_device *adev) · 507 unmodified lines | |||
| 508 | /* Set IP register base before any HW register access */ | 508 | /* Set IP register base before any HW register access */ |
| 509 | switch (adev->asic_type) { | 509 | switch (adev->asic_type) { |
| 510 | case CHIP_VEGA10: | 510 | case CHIP_VEGA10: |
| 511 | case CHIP_VEGA12: | ||
| 511 | case CHIP_RAVEN: | 512 | case CHIP_RAVEN: |
| 512 | vega10_reg_base_init(adev); | 513 | vega10_reg_base_init(adev); |
| 513 | break; | 514 | break; |
| @@ -527,6 +528,7 @@ int soc15_set_ip_blocks(struct amdgpu_device *adev) · 13 unmodified lines | |||
| 527 | 528 | ||
| 528 | switch (adev->asic_type) { | 529 | switch (adev->asic_type) { |
| 529 | case CHIP_VEGA10: | 530 | case CHIP_VEGA10: |
| 531 | case CHIP_VEGA12: | ||
| 530 | amdgpu_device_ip_block_add(adev, &vega10_common_ip_block); | 532 | amdgpu_device_ip_block_add(adev, &vega10_common_ip_block); |
| 531 | amdgpu_device_ip_block_add(adev, &gmc_v9_0_ip_block); | 533 | amdgpu_device_ip_block_add(adev, &gmc_v9_0_ip_block); |
| 532 | amdgpu_device_ip_block_add(adev, &vega10_ih_ip_block); | 534 | amdgpu_device_ip_block_add(adev, &vega10_ih_ip_block); |
| @@ -608,7 +610,6 @@ static const struct amdgpu_asic_funcs soc15_asic_funcs = · 75 unmodified lines | |||
| 608 | 610 | ||
| 609 | static int soc15_common_early_init(void *handle) | 611 | static int soc15_common_early_init(void *handle) |
| 610 | { | 612 | { |
| 611 | bool psp_enabled = false; | ||
| 612 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; | 613 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; |
| 613 | 614 | ||
| 614 | adev->smc_rreg = NULL; | 615 | adev->smc_rreg = NULL; |
| @@ -626,10 +627,6 @@ static int soc15_common_early_init(void *handle) · 11 unmodified lines | |||
| 626 | 627 | ||
| 627 | adev->asic_funcs = &soc15_asic_funcs; | 628 | adev->asic_funcs = &soc15_asic_funcs; |
| 628 | 629 | ||
| 629 | if (amdgpu_device_ip_get_ip_block(adev, AMD_IP_BLOCK_TYPE_PSP) && | ||
| 630 | (amdgpu_ip_block_mask & (1 << AMD_IP_BLOCK_TYPE_PSP))) | ||
| 631 | psp_enabled = true; | ||
| 632 | |||
| 633 | adev->rev_id = soc15_get_rev_id(adev); | 630 | adev->rev_id = soc15_get_rev_id(adev); |
| 634 | adev->external_rev_id = 0xFF; | 631 | adev->external_rev_id = 0xFF; |
| 635 | switch (adev->asic_type) { | 632 | switch (adev->asic_type) { |
| @@ -656,6 +653,28 @@ static int soc15_common_early_init(void *handle) · 20 unmodified lines | |||
| 656 | adev->pg_flags = 0; | 653 | adev->pg_flags = 0; |
| 657 | adev->external_rev_id = 0x1; | 654 | adev->external_rev_id = 0x1; |
| 658 | break; | 655 | break; |
| 656 | case CHIP_VEGA12: | ||
| 657 | adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG | | ||
| 658 | AMD_CG_SUPPORT_GFX_MGLS | | ||
| 659 | AMD_CG_SUPPORT_GFX_CGCG | | ||
| 660 | AMD_CG_SUPPORT_GFX_CGLS | | ||
| 661 | AMD_CG_SUPPORT_GFX_3D_CGCG | | ||
| 662 | AMD_CG_SUPPORT_GFX_3D_CGLS | | ||
| 663 | AMD_CG_SUPPORT_GFX_CP_LS | | ||
| 664 | AMD_CG_SUPPORT_MC_LS | | ||
| 665 | AMD_CG_SUPPORT_MC_MGCG | | ||
| 666 | AMD_CG_SUPPORT_SDMA_MGCG | | ||
| 667 | AMD_CG_SUPPORT_SDMA_LS | | ||
| 668 | AMD_CG_SUPPORT_BIF_MGCG | | ||
| 669 | AMD_CG_SUPPORT_BIF_LS | | ||
| 670 | AMD_CG_SUPPORT_HDP_MGCG | | ||
| 671 | AMD_CG_SUPPORT_HDP_LS | | ||
| 672 | AMD_CG_SUPPORT_ROM_MGCG | | ||
| 673 | AMD_CG_SUPPORT_VCE_MGCG | | ||
| 674 | AMD_CG_SUPPORT_UVD_MGCG; | ||
| 675 | adev->pg_flags = 0; | ||
| 676 | adev->external_rev_id = adev->rev_id + 0x14; | ||
| 677 | break; | ||
| 659 | case CHIP_RAVEN: | 678 | case CHIP_RAVEN: |
| 660 | adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG | | 679 | adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG | |
| 661 | AMD_CG_SUPPORT_GFX_MGLS | | 680 | AMD_CG_SUPPORT_GFX_MGLS | |
| @@ -888,6 +907,7 @@ static int soc15_common_set_clockgating_state(void *handle, · 226 unmodified lines | |||
| 888 | 907 | ||
| 889 | switch (adev->asic_type) { | 908 | switch (adev->asic_type) { |
| 890 | case CHIP_VEGA10: | 909 | case CHIP_VEGA10: |
| 910 | case CHIP_VEGA12: | ||
| 891 | adev->nbio_funcs->update_medium_grain_clock_gating(adev, | 911 | adev->nbio_funcs->update_medium_grain_clock_gating(adev, |
| 892 | state == AMD_CG_STATE_GATE ? true : false); | 912 | state == AMD_CG_STATE_GATE ? true : false); |
| 893 | adev->nbio_funcs->update_medium_grain_light_sleep(adev, | 913 | adev->nbio_funcs->update_medium_grain_light_sleep(adev, |
| @@ -896,7 +896,6 @@ static const struct amdgpu_asic_funcs vi_asic_funcs = · 895 unmodified lines | |||
| 896 | 896 | ||
| 897 | static int vi_common_early_init(void *handle) | 897 | static int vi_common_early_init(void *handle) |
| 898 | { | 898 | { |
| 899 | bool smc_enabled = false; | ||
| 900 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; | 899 | struct amdgpu_device *adev = (struct amdgpu_device *)handle; |
| 901 | 900 | ||
| 902 | if (adev->flags & AMD_IS_APU) { | 901 | if (adev->flags & AMD_IS_APU) { |
| @@ -917,10 +916,6 @@ static int vi_common_early_init(void *handle) · 14 unmodified lines | |||
| 917 | 916 | ||
| 918 | adev->asic_funcs = &vi_asic_funcs; | 917 | adev->asic_funcs = &vi_asic_funcs; |
| 919 | 918 | ||
| 920 | if (amdgpu_device_ip_get_ip_block(adev, AMD_IP_BLOCK_TYPE_SMC) && | ||
| 921 | (amdgpu_ip_block_mask & (1 << AMD_IP_BLOCK_TYPE_SMC))) | ||
| 922 | smc_enabled = true; | ||
| 923 | |||
| 924 | adev->rev_id = vi_get_rev_id(adev); | 919 | adev->rev_id = vi_get_rev_id(adev); |
| 925 | adev->external_rev_id = 0xFF; | 920 | adev->external_rev_id = 0xFF; |
| 926 | switch (adev->asic_type) { | 921 | switch (adev->asic_type) { |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | |||
| 24 | #include <linux/export.h> | 24 | #include <linux/export.h> |
| 25 | #include <linux/err.h> | 25 | #include <linux/err.h> |
| 26 | #include <linux/fs.h> | 26 | #include <linux/fs.h> |
| 27 | #include <linux/file.h> | ||
| 27 | #include <linux/sched.h> | 28 | #include <linux/sched.h> |
| 28 | #include <linux/slab.h> | 29 | #include <linux/slab.h> |
| 29 | #include <linux/uaccess.h> | 30 | #include <linux/uaccess.h> |
| @@ -825,12 +826,155 @@ static int kfd_ioctl_get_process_apertures(struct file *filp, · 795 unmodified lines | |||
| 825 | return 0; | 826 | return 0; |
| 826 | } | 827 | } |
| 827 | 828 | ||
| 829 | static int kfd_ioctl_get_process_apertures_new(struct file *filp, | ||
| 830 | struct kfd_process *p, void *data) | ||
| 831 | { | ||
| 832 | struct kfd_ioctl_get_process_apertures_new_args *args = data; | ||
| 833 | struct kfd_process_device_apertures *pa; | ||
| 834 | struct kfd_process_device *pdd; | ||
| 835 | uint32_t nodes = 0; | ||
| 836 | int ret; | ||
| 837 | |||
| 838 | dev_dbg(kfd_device, "get apertures for PASID %d", p->pasid); | ||
| 839 | |||
| 840 | if (args->num_of_nodes == 0) { | ||
| 841 | /* Return number of nodes, so that user space can alloacate | ||
| 842 | * sufficient memory | ||
| 843 | */ | ||
| 844 | mutex_lock(&p->mutex); | ||
| 845 | |||
| 846 | if (!kfd_has_process_device_data(p)) | ||
| 847 | goto out_unlock; | ||
| 848 | |||
| 849 | /* Run over all pdd of the process */ | ||
| 850 | pdd = kfd_get_first_process_device_data(p); | ||
| 851 | do { | ||
| 852 | args->num_of_nodes++; | ||
| 853 | pdd = kfd_get_next_process_device_data(p, pdd); | ||
| 854 | } while (pdd); | ||
| 855 | |||
| 856 | goto out_unlock; | ||
| 857 | } | ||
| 858 | |||
| 859 | /* Fill in process-aperture information for all available | ||
| 860 | * nodes, but not more than args->num_of_nodes as that is | ||
| 861 | * the amount of memory allocated by user | ||
| 862 | */ | ||
| 863 | pa = kzalloc((sizeof(struct kfd_process_device_apertures) * | ||
| 864 | args->num_of_nodes), GFP_KERNEL); | ||
| 865 | if (!pa) | ||
| 866 | return -ENOMEM; | ||
| 867 | |||
| 868 | mutex_lock(&p->mutex); | ||
| 869 | |||
| 870 | if (!kfd_has_process_device_data(p)) { | ||
| 871 | args->num_of_nodes = 0; | ||
| 872 | kfree(pa); | ||
| 873 | goto out_unlock; | ||
| 874 | } | ||
| 875 | |||
| 876 | /* Run over all pdd of the process */ | ||
| 877 | pdd = kfd_get_first_process_device_data(p); | ||
| 878 | do { | ||
| 879 | pa[nodes].gpu_id = pdd->dev->id; | ||
| 880 | pa[nodes].lds_base = pdd->lds_base; | ||
| 881 | pa[nodes].lds_limit = pdd->lds_limit; | ||
| 882 | pa[nodes].gpuvm_base = pdd->gpuvm_base; | ||
| 883 | pa[nodes].gpuvm_limit = pdd->gpuvm_limit; | ||
| 884 | pa[nodes].scratch_base = pdd->scratch_base; | ||
| 885 | pa[nodes].scratch_limit = pdd->scratch_limit; | ||
| 886 | |||
| 887 | dev_dbg(kfd_device, | ||
| 888 | "gpu id %u\n", pdd->dev->id); | ||
| 889 | dev_dbg(kfd_device, | ||
| 890 | "lds_base %llX\n", pdd->lds_base); | ||
| 891 | dev_dbg(kfd_device, | ||
| 892 | "lds_limit %llX\n", pdd->lds_limit); | ||
| 893 | dev_dbg(kfd_device, | ||
| 894 | "gpuvm_base %llX\n", pdd->gpuvm_base); | ||
| 895 | dev_dbg(kfd_device, | ||
| 896 | "gpuvm_limit %llX\n", pdd->gpuvm_limit); | ||
| 897 | dev_dbg(kfd_device, | ||
| 898 | "scratch_base %llX\n", pdd->scratch_base); | ||
| 899 | dev_dbg(kfd_device, | ||
| 900 | "scratch_limit %llX\n", pdd->scratch_limit); | ||
| 901 | nodes++; | ||
| 902 | |||
| 903 | pdd = kfd_get_next_process_device_data(p, pdd); | ||
| 904 | } while (pdd && (nodes < args->num_of_nodes)); | ||
| 905 | mutex_unlock(&p->mutex); | ||
| 906 | |||
| 907 | args->num_of_nodes = nodes; | ||
| 908 | ret = copy_to_user( | ||
| 909 | (void __user *)args->kfd_process_device_apertures_ptr, | ||
| 910 | pa, | ||
| 911 | (nodes * sizeof(struct kfd_process_device_apertures))); | ||
| 912 | kfree(pa); | ||
| 913 | return ret ? -EFAULT : 0; | ||
| 914 | |||
| 915 | out_unlock: | ||
| 916 | mutex_unlock(&p->mutex); | ||
| 917 | return 0; | ||
| 918 | } | ||
| 919 | |||
| 828 | static int kfd_ioctl_create_event(struct file *filp, struct kfd_process *p, | 920 | static int kfd_ioctl_create_event(struct file *filp, struct kfd_process *p, |
| 829 | void *data) | 921 | void *data) |
| 830 | { | 922 | { |
| 831 | struct kfd_ioctl_create_event_args *args = data; | 923 | struct kfd_ioctl_create_event_args *args = data; |
| 832 | int err; | 924 | int err; |
| 833 | 925 | ||
| 926 | /* For dGPUs the event page is allocated in user mode. The | ||
| 927 | * handle is passed to KFD with the first call to this IOCTL | ||
| 928 | * through the event_page_offset field. | ||
| 929 | */ | ||
| 930 | if (args->event_page_offset) { | ||
| 931 | struct kfd_dev *kfd; | ||
| 932 | struct kfd_process_device *pdd; | ||
| 933 | void *mem, *kern_addr; | ||
| 934 | uint64_t size; | ||
| 935 | |||
| 936 | if (p->signal_page) { | ||
| 937 | pr_err("Event page is already set\n"); | ||
| 938 | return -EINVAL; | ||
| 939 | } | ||
| 940 | |||
| 941 | kfd = kfd_device_by_id(GET_GPU_ID(args->event_page_offset)); | ||
| 942 | if (!kfd) { | ||
| 943 | pr_err("Getting device by id failed in %s\n", __func__); | ||
| 944 | return -EINVAL; | ||
| 945 | } | ||
| 946 | |||
| 947 | mutex_lock(&p->mutex); | ||
| 948 | pdd = kfd_bind_process_to_device(kfd, p); | ||
| 949 | if (IS_ERR(pdd)) { | ||
| 950 | err = PTR_ERR(pdd); | ||
| 951 | goto out_unlock; | ||
| 952 | } | ||
| 953 | |||
| 954 | mem = kfd_process_device_translate_handle(pdd, | ||
| 955 | GET_IDR_HANDLE(args->event_page_offset)); | ||
| 956 | if (!mem) { | ||
| 957 | pr_err("Can't find BO, offset is 0x%llx\n", | ||
| 958 | args->event_page_offset); | ||
| 959 | err = -EINVAL; | ||
| 960 | goto out_unlock; | ||
| 961 | } | ||
| 962 | mutex_unlock(&p->mutex); | ||
| 963 | |||
| 964 | err = kfd->kfd2kgd->map_gtt_bo_to_kernel(kfd->kgd, | ||
| 965 | mem, &kern_addr, &size); | ||
| 966 | if (err) { | ||
| 967 | pr_err("Failed to map event page to kernel\n"); | ||
| 968 | return err; | ||
| 969 | } | ||
| 970 | |||
| 971 | err = kfd_event_page_set(p, kern_addr, size); | ||
| 972 | if (err) { | ||
| 973 | pr_err("Failed to set event page\n"); | ||
| 974 | return err; | ||
| 975 | } | ||
| 976 | } | ||
| 977 | |||
| 834 | err = kfd_event_create(filp, p, args->event_type, | 978 | err = kfd_event_create(filp, p, args->event_type, |
| 835 | args->auto_reset != 0, args->node_id, | 979 | args->auto_reset != 0, args->node_id, |
| 836 | &args->event_id, &args->event_trigger_data, | 980 | &args->event_id, &args->event_trigger_data, |
| @@ -838,6 +982,10 @@ static int kfd_ioctl_create_event(struct file *filp, struct kfd_process *p, · 1 unmodified line | |||
| 838 | &args->event_slot_index); | 982 | &args->event_slot_index); |
| 839 | 983 | ||
| 840 | return err; | 984 | return err; |
| 985 | |||
| 986 | out_unlock: | ||
| 987 | mutex_unlock(&p->mutex); | ||
| 988 | return err; | ||
| 841 | } | 989 | } |
| 842 | 990 | ||
| 843 | static int kfd_ioctl_destroy_event(struct file *filp, struct kfd_process *p, | 991 | static int kfd_ioctl_destroy_event(struct file *filp, struct kfd_process *p, |
| @@ -955,6 +1103,371 @@ static int kfd_ioctl_get_tile_config(struct file *filep, · 111 unmodified lines | |||
| 955 | return 0; | 1103 | return 0; |
| 956 | } | 1104 | } |
| 957 | 1105 | ||
| 1106 | static int kfd_ioctl_acquire_vm(struct file *filep, struct kfd_process *p, | ||
| 1107 | void *data) | ||
| 1108 | { | ||
| 1109 | struct kfd_ioctl_acquire_vm_args *args = data; | ||
| 1110 | struct kfd_process_device *pdd; | ||
| 1111 | struct kfd_dev *dev; | ||
| 1112 | struct file *drm_file; | ||
| 1113 | int ret; | ||
| 1114 | |||
| 1115 | dev = kfd_device_by_id(args->gpu_id); | ||
| 1116 | if (!dev) | ||
| 1117 | return -EINVAL; | ||
| 1118 | |||
| 1119 | drm_file = fget(args->drm_fd); | ||
| 1120 | if (!drm_file) | ||
| 1121 | return -EINVAL; | ||
| 1122 | |||
| 1123 | mutex_lock(&p->mutex); | ||
| 1124 | |||
| 1125 | pdd = kfd_get_process_device_data(dev, p); | ||
| 1126 | if (!pdd) { | ||
| 1127 | ret = -EINVAL; | ||
| 1128 | goto err_unlock; | ||
| 1129 | } | ||
| 1130 | |||
| 1131 | if (pdd->drm_file) { | ||
| 1132 | ret = pdd->drm_file == drm_file ? 0 : -EBUSY; | ||
| 1133 | goto err_unlock; | ||
| 1134 | } | ||
| 1135 | |||
| 1136 | ret = kfd_process_device_init_vm(pdd, drm_file); | ||
| 1137 | if (ret) | ||
| 1138 | goto err_unlock; | ||
361 more lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 475 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 7498 lines aren’t shown here.
This diff is large: its 31161 lines aren’t shown here.
This diff is large: its 1992 lines aren’t shown here.
This diff is large: its 10266 lines aren’t shown here.
This diff is large: its 338 lines aren’t shown here.
This diff is large: its 1250 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 881 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 2091 lines aren’t shown here.
This diff is large: its 439 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 1365 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 431 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 325 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 759 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 217 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 562 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 353 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 698 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 171 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 358 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 256 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 161 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 507 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 153 lines aren’t shown here.
This diff is large: its 143 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 409 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 323 lines aren’t shown here.
This diff is large: its 452 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 161 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 309 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 142 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
File renamed without changes.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 291 lines aren’t shown here.
This diff is large: its 265 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 250 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 170 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 149 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 190 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 338 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 173 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 220 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 166 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 198 lines aren’t shown here.
This diff is large: its 523 lines aren’t shown here.
This diff is large: its 186 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 257 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 382 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 255 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 165 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 233 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 248 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 257 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 161 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 240 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 575 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 454 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 134 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 232 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 267 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 157 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 147 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 350 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 162 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 185 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 112 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 153 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 219 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 354 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 156 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 178 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 226 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 92 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is larger than a page shows: the files after these aren’t listed. Clone the repository to see all of it.