ASoC: Fixes for v4.16 This is a fairly standard collection of fixes, there's no changes to the core here just a bunch of small device specific changes for single drivers plus an update to the MAINTAINERS file for the sgl5000.
Unified · Split
| @@ -127,3 +127,7 @@ all.config · 126 unmodified lines | |||
| 127 | 127 | ||
| 128 | # Kdevelop4 | 128 | # Kdevelop4 |
| 129 | *.kdev4 | 129 | *.kdev4 |
| 130 | |||
| 131 | #Automatically generated by ASN.1 compiler | ||
| 132 | net/ipv4/netfilter/nf_nat_snmp_basic-asn1.c | ||
| 133 | net/ipv4/netfilter/nf_nat_snmp_basic-asn1.h | ||
| @@ -0,0 +1,39 @@ | |||
| 1 | What: /sys/devices/platform/dock.N/docked | ||
| 2 | Date: Dec, 2006 | ||
| 3 | KernelVersion: 2.6.19 | ||
| 4 | Contact: linux-acpi@vger.kernel.org | ||
| 5 | Description: | ||
| 6 | (RO) Value 1 or 0 indicates whether the software believes the | ||
| 7 | laptop is docked in a docking station. | ||
| 8 | |||
| 9 | What: /sys/devices/platform/dock.N/undock | ||
| 10 | Date: Dec, 2006 | ||
| 11 | KernelVersion: 2.6.19 | ||
| 12 | Contact: linux-acpi@vger.kernel.org | ||
| 13 | Description: | ||
| 14 | (WO) Writing to this file causes the software to initiate an | ||
| 15 | undock request to the firmware. | ||
| 16 | |||
| 17 | What: /sys/devices/platform/dock.N/uid | ||
| 18 | Date: Feb, 2007 | ||
| 19 | KernelVersion: v2.6.21 | ||
| 20 | Contact: linux-acpi@vger.kernel.org | ||
| 21 | Description: | ||
| 22 | (RO) Displays the docking station the laptop is docked to. | ||
| 23 | |||
| 24 | What: /sys/devices/platform/dock.N/flags | ||
| 25 | Date: May, 2007 | ||
| 26 | KernelVersion: v2.6.21 | ||
| 27 | Contact: linux-acpi@vger.kernel.org | ||
| 28 | Description: | ||
| 29 | (RO) Show dock station flags, useful for checking if undock | ||
| 30 | request has been made by the user (from the immediate_undock | ||
| 31 | option). | ||
| 32 | |||
| 33 | What: /sys/devices/platform/dock.N/type | ||
| 34 | Date: Aug, 2008 | ||
| 35 | KernelVersion: v2.6.27 | ||
| 36 | Contact: linux-acpi@vger.kernel.org | ||
| 37 | Description: | ||
| 38 | (RO) Display the dock station type- dock_station, ata_bay or | ||
| 39 | battery_bay. | ||
| @@ -108,6 +108,8 @@ Description: CPU topology files that describe a logical CPU's relationship · 107 unmodified lines | |||
| 108 | 108 | ||
| 109 | What: /sys/devices/system/cpu/cpuidle/current_driver | 109 | What: /sys/devices/system/cpu/cpuidle/current_driver |
| 110 | /sys/devices/system/cpu/cpuidle/current_governer_ro | 110 | /sys/devices/system/cpu/cpuidle/current_governer_ro |
| 111 | /sys/devices/system/cpu/cpuidle/available_governors | ||
| 112 | /sys/devices/system/cpu/cpuidle/current_governor | ||
| 111 | Date: September 2007 | 113 | Date: September 2007 |
| 112 | Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org> | 114 | Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org> |
| 113 | Description: Discover cpuidle policy and mechanism | 115 | Description: Discover cpuidle policy and mechanism |
| @@ -119,13 +121,84 @@ Description: Discover cpuidle policy and mechanism · 5 unmodified lines | |||
| 119 | Idle policy (governor) is differentiated from idle mechanism | 121 | Idle policy (governor) is differentiated from idle mechanism |
| 120 | (driver) | 122 | (driver) |
| 121 | 123 | ||
| 122 | current_driver: displays current idle mechanism | 124 | current_driver: (RO) displays current idle mechanism |
| 123 | 125 | ||
| 124 | current_governor_ro: displays current idle policy | 126 | current_governor_ro: (RO) displays current idle policy |
| 127 | |||
| 128 | With the cpuidle_sysfs_switch boot option enabled (meant for | ||
| 129 | developer testing), the following three attributes are visible | ||
| 130 | instead: | ||
| 131 | |||
| 132 | current_driver: same as described above | ||
| 133 | |||
| 134 | available_governors: (RO) displays a space separated list of | ||
| 135 | available governors | ||
| 136 | |||
| 137 | current_governor: (RW) displays current idle policy. Users can | ||
| 138 | switch the governor at runtime by writing to this file. | ||
| 125 | 139 | ||
| 126 | See files in Documentation/cpuidle/ for more information. | 140 | See files in Documentation/cpuidle/ for more information. |
| 127 | 141 | ||
| 128 | 142 | ||
| 143 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/name | ||
| 144 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/latency | ||
| 145 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/power | ||
| 146 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/time | ||
| 147 | /sys/devices/system/cpu/cpuX/cpuidle/stateN/usage | ||
| 148 | Date: September 2007 | ||
| 149 | KernelVersion: v2.6.24 | ||
| 150 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 151 | Description: | ||
| 152 | The directory /sys/devices/system/cpu/cpuX/cpuidle contains per | ||
| 153 | logical CPU specific cpuidle information for each online cpu X. | ||
| 154 | The processor idle states which are available for use have the | ||
| 155 | following attributes: | ||
| 156 | |||
| 157 | name: (RO) Name of the idle state (string). | ||
| 158 | |||
| 159 | latency: (RO) The latency to exit out of this idle state (in | ||
| 160 | microseconds). | ||
| 161 | |||
| 162 | power: (RO) The power consumed while in this idle state (in | ||
| 163 | milliwatts). | ||
| 164 | |||
| 165 | time: (RO) The total time spent in this idle state (in microseconds). | ||
| 166 | |||
| 167 | usage: (RO) Number of times this state was entered (a count). | ||
| 168 | |||
| 169 | |||
| 170 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/desc | ||
| 171 | Date: February 2008 | ||
| 172 | KernelVersion: v2.6.25 | ||
| 173 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 174 | Description: | ||
| 175 | (RO) A small description about the idle state (string). | ||
| 176 | |||
| 177 | |||
| 178 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/disable | ||
| 179 | Date: March 2012 | ||
| 180 | KernelVersion: v3.10 | ||
| 181 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 182 | Description: | ||
| 183 | (RW) Option to disable this idle state (bool). The behavior and | ||
| 184 | the effect of the disable variable depends on the implementation | ||
| 185 | of a particular governor. In the ladder governor, for example, | ||
| 186 | it is not coherent, i.e. if one is disabling a light state, then | ||
| 187 | all deeper states are disabled as well, but the disable variable | ||
| 188 | does not reflect it. Likewise, if one enables a deep state but a | ||
| 189 | lighter state still is disabled, then this has no effect. | ||
| 190 | |||
| 191 | |||
| 192 | What: /sys/devices/system/cpu/cpuX/cpuidle/stateN/residency | ||
| 193 | Date: March 2014 | ||
| 194 | KernelVersion: v3.15 | ||
| 195 | Contact: Linux power management list <linux-pm@vger.kernel.org> | ||
| 196 | Description: | ||
| 197 | (RO) Display the target residency i.e. the minimum amount of | ||
| 198 | time (in microseconds) this cpu should spend in this idle state | ||
| 199 | to make the transition worth the effort. | ||
| 200 | |||
| 201 | |||
| 129 | What: /sys/devices/system/cpu/cpu#/cpufreq/* | 202 | What: /sys/devices/system/cpu/cpu#/cpufreq/* |
| 130 | Date: pre-git history | 203 | Date: pre-git history |
| 131 | Contact: linux-pm@vger.kernel.org | 204 | Contact: linux-pm@vger.kernel.org |
| @@ -0,0 +1,40 @@ | |||
| 1 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/charger_type | ||
| 2 | Date: Jul, 2016 | ||
| 3 | KernelVersion: v4.10 | ||
| 4 | Contact: linux-acpi@vger.kernel.org | ||
| 5 | Description: | ||
| 6 | (RO) The charger type - Traditional, Hybrid or NVDC. | ||
| 7 | |||
| 8 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/adapter_rating_mw | ||
| 9 | Date: Jul, 2016 | ||
| 10 | KernelVersion: v4.10 | ||
| 11 | Contact: linux-acpi@vger.kernel.org | ||
| 12 | Description: | ||
| 13 | (RO) Adapter rating in milliwatts (the maximum Adapter power). | ||
| 14 | Must be 0 if no AC Adaptor is plugged in. | ||
| 15 | |||
| 16 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/max_platform_power_mw | ||
| 17 | Date: Jul, 2016 | ||
| 18 | KernelVersion: v4.10 | ||
| 19 | Contact: linux-acpi@vger.kernel.org | ||
| 20 | Description: | ||
| 21 | (RO) Maximum platform power that can be supported by the battery | ||
| 22 | in milliwatts. | ||
| 23 | |||
| 24 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/platform_power_source | ||
| 25 | Date: Jul, 2016 | ||
| 26 | KernelVersion: v4.10 | ||
| 27 | Contact: linux-acpi@vger.kernel.org | ||
| 28 | Description: | ||
| 29 | (RO) Display the platform power source | ||
| 30 | 0x00 = DC | ||
| 31 | 0x01 = AC | ||
| 32 | 0x02 = USB | ||
| 33 | 0x03 = Wireless Charger | ||
| 34 | |||
| 35 | What: /sys/bus/platform/devices/INT3407:00/dptf_power/battery_steady_power | ||
| 36 | Date: Jul, 2016 | ||
| 37 | KernelVersion: v4.10 | ||
| 38 | Contact: linux-acpi@vger.kernel.org | ||
| 39 | Description: | ||
| 40 | (RO) The maximum sustained power for battery in milliwatts. | ||
| @@ -570,7 +570,9 @@ your driver if they're helpful, or just use plain hex constants. · 569 unmodified lines | |||
| 570 | The device IDs are arbitrary hex numbers (vendor controlled) and normally used | 570 | The device IDs are arbitrary hex numbers (vendor controlled) and normally used |
| 571 | only in a single location, the pci_device_id table. | 571 | only in a single location, the pci_device_id table. |
| 572 | 572 | ||
| 573 | Please DO submit new vendor/device IDs to http://pciids.sourceforge.net/. | 573 | Please DO submit new vendor/device IDs to http://pci-ids.ucw.cz/. |
| 574 | There are mirrors of the pci.ids file at http://pciids.sourceforge.net/ | ||
| 575 | and https://github.com/pciutils/pciids. | ||
| 574 | 576 | ||
| 575 | 577 | ||
| 576 | 578 | ||
| @@ -152,6 +152,11 @@ OCXL_IOCTL_IRQ_SET_FD: · 151 unmodified lines | |||
| 152 | Associate an event fd to an AFU interrupt so that the user process | 152 | Associate an event fd to an AFU interrupt so that the user process |
| 153 | can be notified when the AFU sends an interrupt. | 153 | can be notified when the AFU sends an interrupt. |
| 154 | 154 | ||
| 155 | OCXL_IOCTL_GET_METADATA: | ||
| 156 | |||
| 157 | Obtains configuration information from the card, such at the size of | ||
| 158 | MMIO areas, the AFU version, and the PASID for the current context. | ||
| 159 | |||
| 155 | 160 | ||
| 156 | mmap | 161 | mmap |
| 157 | ---- | 162 | ---- |
| @@ -58,7 +58,12 @@ Like with atomic_t, the rule of thumb is: · 57 unmodified lines | |||
| 58 | 58 | ||
| 59 | - RMW operations that have a return value are fully ordered. | 59 | - RMW operations that have a return value are fully ordered. |
| 60 | 60 | ||
| 61 | Except for test_and_set_bit_lock() which has ACQUIRE semantics and | 61 | - RMW operations that are conditional are unordered on FAILURE, |
| 62 | otherwise the above rules apply. In the case of test_and_{}_bit() operations, | ||
| 63 | if the bit in memory is unchanged by the operation then it is deemed to have | ||
| 64 | failed. | ||
| 65 | |||
| 66 | Except for a successful test_and_set_bit_lock() which has ACQUIRE semantics and | ||
| 62 | clear_bit_unlock() which has RELEASE semantics. | 67 | clear_bit_unlock() which has RELEASE semantics. |
| 63 | 68 | ||
| 64 | Since a platform only has a single means of achieving atomic operations | 69 | Since a platform only has a single means of achieving atomic operations |
File renamed without changes.
| @@ -11,7 +11,11 @@ Required properties: · 10 unmodified lines | |||
| 11 | interrupts. | 11 | interrupts. |
| 12 | 12 | ||
| 13 | Optional properties: | 13 | Optional properties: |
| 14 | - clocks: Optional reference to the clock used by the XOR engine. | 14 | - clocks: Optional reference to the clocks used by the XOR engine. |
| 15 | - clock-names: mandatory if there is a second clock, in this case the | ||
| 16 | name must be "core" for the first clock and "reg" for the second | ||
| 17 | one | ||
| 18 | |||
| 15 | 19 | ||
| 16 | Example: | 20 | Example: |
| 17 | 21 | ||
| @@ -38,9 +38,9 @@ Required properties: · 37 unmodified lines | |||
| 38 | 38 | ||
| 39 | "catalyst", | 39 | "catalyst", |
| 40 | "microchip", | 40 | "microchip", |
| 41 | "nxp", | ||
| 41 | "ramtron", | 42 | "ramtron", |
| 42 | "renesas", | 43 | "renesas", |
| 43 | "nxp", | ||
| 44 | "st", | 44 | "st", |
| 45 | 45 | ||
| 46 | Some vendors use different model names for chips which are just | 46 | Some vendors use different model names for chips which are just |
| @@ -14,6 +14,7 @@ Required properties: · 13 unmodified lines | |||
| 14 | - "renesas,irqc-r8a7794" (R-Car E2) | 14 | - "renesas,irqc-r8a7794" (R-Car E2) |
| 15 | - "renesas,intc-ex-r8a7795" (R-Car H3) | 15 | - "renesas,intc-ex-r8a7795" (R-Car H3) |
| 16 | - "renesas,intc-ex-r8a7796" (R-Car M3-W) | 16 | - "renesas,intc-ex-r8a7796" (R-Car M3-W) |
| 17 | - "renesas,intc-ex-r8a77965" (R-Car M3-N) | ||
| 17 | - "renesas,intc-ex-r8a77970" (R-Car V3M) | 18 | - "renesas,intc-ex-r8a77970" (R-Car V3M) |
| 18 | - "renesas,intc-ex-r8a77995" (R-Car D3) | 19 | - "renesas,intc-ex-r8a77995" (R-Car D3) |
| 19 | - #interrupt-cells: has to be <2>: an interrupt index and flags, as defined in | 20 | - #interrupt-cells: has to be <2>: an interrupt index and flags, as defined in |
| @@ -18,6 +18,7 @@ Required properties: · 17 unmodified lines | |||
| 18 | - "renesas,etheravb-r8a7795" for the R8A7795 SoC. | 18 | - "renesas,etheravb-r8a7795" for the R8A7795 SoC. |
| 19 | - "renesas,etheravb-r8a7796" for the R8A7796 SoC. | 19 | - "renesas,etheravb-r8a7796" for the R8A7796 SoC. |
| 20 | - "renesas,etheravb-r8a77970" for the R8A77970 SoC. | 20 | - "renesas,etheravb-r8a77970" for the R8A77970 SoC. |
| 21 | - "renesas,etheravb-r8a77980" for the R8A77980 SoC. | ||
| 21 | - "renesas,etheravb-r8a77995" for the R8A77995 SoC. | 22 | - "renesas,etheravb-r8a77995" for the R8A77995 SoC. |
| 22 | - "renesas,etheravb-rcar-gen3" as a fallback for the above | 23 | - "renesas,etheravb-rcar-gen3" as a fallback for the above |
| 23 | R-Car Gen3 devices. | 24 | R-Car Gen3 devices. |
| @@ -0,0 +1,8 @@ | |||
| 1 | Binding for MIPS Cluster Power Controller (CPC). | ||
| 2 | |||
| 3 | This binding allows a system to specify where the CPC registers are | ||
| 4 | located. | ||
| 5 | |||
| 6 | Required properties: | ||
| 7 | compatible : Should be "mti,mips-cpc". | ||
| 8 | regs: Should describe the address & size of the CPC register region. | ||
| @@ -60,7 +60,7 @@ Examples · 59 unmodified lines | |||
| 60 | #size-cells = <0>; | 60 | #size-cells = <0>; |
| 61 | 61 | ||
| 62 | button@1 { | 62 | button@1 { |
| 63 | debounce_interval = <50>; | 63 | debounce-interval = <50>; |
| 64 | wakeup-source; | 64 | wakeup-source; |
| 65 | linux,code = <116>; | 65 | linux,code = <116>; |
| 66 | label = "POWER"; | 66 | label = "POWER"; |
| @@ -22,7 +22,32 @@ Optional properties: · 21 unmodified lines | |||
| 22 | - clocks : thermal sensor's clock source. | 22 | - clocks : thermal sensor's clock source. |
| 23 | 23 | ||
| 24 | Example: | 24 | Example: |
| 25 | ocotp: ocotp@21bc000 { | ||
| 26 | #address-cells = <1>; | ||
| 27 | #size-cells = <1>; | ||
| 28 | compatible = "fsl,imx6sx-ocotp", "syscon"; | ||
| 29 | reg = <0x021bc000 0x4000>; | ||
| 30 | clocks = <&clks IMX6SX_CLK_OCOTP>; | ||
| 25 | 31 | ||
| 32 | tempmon_calib: calib@38 { | ||
| 33 | reg = <0x38 4>; | ||
| 34 | }; | ||
| 35 | |||
| 36 | tempmon_temp_grade: temp-grade@20 { | ||
| 37 | reg = <0x20 4>; | ||
| 38 | }; | ||
| 39 | }; | ||
| 40 | |||
| 41 | tempmon: tempmon { | ||
| 42 | compatible = "fsl,imx6sx-tempmon", "fsl,imx6q-tempmon"; | ||
| 43 | interrupts = <GIC_SPI 49 IRQ_TYPE_LEVEL_HIGH>; | ||
| 44 | fsl,tempmon = <&anatop>; | ||
| 45 | nvmem-cells = <&tempmon_calib>, <&tempmon_temp_grade>; | ||
| 46 | nvmem-cell-names = "calib", "temp_grade"; | ||
| 47 | clocks = <&clks IMX6SX_CLK_PLL3_USB_OTG>; | ||
| 48 | }; | ||
| 49 | |||
| 50 | Legacy method (Deprecated): | ||
| 26 | tempmon { | 51 | tempmon { |
| 27 | compatible = "fsl,imx6q-tempmon"; | 52 | compatible = "fsl,imx6q-tempmon"; |
| 28 | fsl,tempmon = <&anatop>; | 53 | fsl,tempmon = <&anatop>; |
| @@ -0,0 +1,62 @@ | |||
| 1 | # | ||
| 2 | # Feature name: membarrier-sync-core | ||
| 3 | # Kconfig: ARCH_HAS_MEMBARRIER_SYNC_CORE | ||
| 4 | # description: arch supports core serializing membarrier | ||
| 5 | # | ||
| 6 | # Architecture requirements | ||
| 7 | # | ||
| 8 | # * arm64 | ||
| 9 | # | ||
| 10 | # Rely on eret context synchronization when returning from IPI handler, and | ||
| 11 | # when returning to user-space. | ||
| 12 | # | ||
| 13 | # * x86 | ||
| 14 | # | ||
| 15 | # x86-32 uses IRET as return from interrupt, which takes care of the IPI. | ||
| 16 | # However, it uses both IRET and SYSEXIT to go back to user-space. The IRET | ||
| 17 | # instruction is core serializing, but not SYSEXIT. | ||
| 18 | # | ||
| 19 | # x86-64 uses IRET as return from interrupt, which takes care of the IPI. | ||
| 20 | # However, it can return to user-space through either SYSRETL (compat code), | ||
| 21 | # SYSRETQ, or IRET. | ||
| 22 | # | ||
| 23 | # Given that neither SYSRET{L,Q}, nor SYSEXIT, are core serializing, we rely | ||
| 24 | # instead on write_cr3() performed by switch_mm() to provide core serialization | ||
| 25 | # after changing the current mm, and deal with the special case of kthread -> | ||
| 26 | # uthread (temporarily keeping current mm into active_mm) by issuing a | ||
| 27 | # sync_core_before_usermode() in that specific case. | ||
| 28 | # | ||
| 29 | ----------------------- | ||
| 30 | | arch |status| | ||
| 31 | ----------------------- | ||
| 32 | | alpha: | TODO | | ||
| 33 | | arc: | TODO | | ||
| 34 | | arm: | TODO | | ||
| 35 | | arm64: | ok | | ||
| 36 | | blackfin: | TODO | | ||
| 37 | | c6x: | TODO | | ||
| 38 | | cris: | TODO | | ||
| 39 | | frv: | TODO | | ||
| 40 | | h8300: | TODO | | ||
| 41 | | hexagon: | TODO | | ||
| 42 | | ia64: | TODO | | ||
| 43 | | m32r: | TODO | | ||
| 44 | | m68k: | TODO | | ||
| 45 | | metag: | TODO | | ||
| 46 | | microblaze: | TODO | | ||
| 47 | | mips: | TODO | | ||
| 48 | | mn10300: | TODO | | ||
| 49 | | nios2: | TODO | | ||
| 50 | | openrisc: | TODO | | ||
| 51 | | parisc: | TODO | | ||
| 52 | | powerpc: | TODO | | ||
| 53 | | s390: | TODO | | ||
| 54 | | score: | TODO | | ||
| 55 | | sh: | TODO | | ||
| 56 | | sparc: | TODO | | ||
| 57 | | tile: | TODO | | ||
| 58 | | um: | TODO | | ||
| 59 | | unicore32: | TODO | | ||
| 60 | | x86: | ok | | ||
| 61 | | xtensa: | TODO | | ||
| 62 | ----------------------- | ||
| @@ -3,4 +3,4 @@ · 2 unmodified lines | |||
| 3 | ================================== | 3 | ================================== |
| 4 | 4 | ||
| 5 | .. kernel-doc:: drivers/gpu/drm/tve200/tve200_drv.c | 5 | .. kernel-doc:: drivers/gpu/drm/tve200/tve200_drv.c |
| 6 | :doc: Faraday TV Encoder 200 | 6 | :doc: Faraday TV Encoder TVE200 DRM Driver |
| @@ -28,8 +28,10 @@ Supported adapters: · 27 unmodified lines | |||
| 28 | * Intel Wildcat Point (PCH) | 28 | * Intel Wildcat Point (PCH) |
| 29 | * Intel Wildcat Point-LP (PCH) | 29 | * Intel Wildcat Point-LP (PCH) |
| 30 | * Intel BayTrail (SOC) | 30 | * Intel BayTrail (SOC) |
| 31 | * Intel Braswell (SOC) | ||
| 31 | * Intel Sunrise Point-H (PCH) | 32 | * Intel Sunrise Point-H (PCH) |
| 32 | * Intel Sunrise Point-LP (PCH) | 33 | * Intel Sunrise Point-LP (PCH) |
| 34 | * Intel Kaby Lake-H (PCH) | ||
| 33 | * Intel DNV (SOC) | 35 | * Intel DNV (SOC) |
| 34 | * Intel Broxton (SOC) | 36 | * Intel Broxton (SOC) |
| 35 | * Intel Lewisburg (PCH) | 37 | * Intel Lewisburg (PCH) |
| @@ -111,7 +111,7 @@ TROUBLESHOOTING SERIAL CONSOLE PROBLEMS · 110 unmodified lines | |||
| 111 | 111 | ||
| 112 | - If you don't have an HCDP, the kernel doesn't know where | 112 | - If you don't have an HCDP, the kernel doesn't know where |
| 113 | your console lives until the driver discovers serial | 113 | your console lives until the driver discovers serial |
| 114 | devices. Use "console=uart, io,0x3f8" (or appropriate | 114 | devices. Use "console=uart,io,0x3f8" (or appropriate |
| 115 | address for your machine). | 115 | address for your machine). |
| 116 | 116 | ||
| 117 | Kernel and init script output works fine, but no "login:" prompt: | 117 | Kernel and init script output works fine, but no "login:" prompt: |
| @@ -21,37 +21,23 @@ Implementation · 20 unmodified lines | |||
| 21 | -------------- | 21 | -------------- |
| 22 | 22 | ||
| 23 | Mutexes are represented by 'struct mutex', defined in include/linux/mutex.h | 23 | Mutexes are represented by 'struct mutex', defined in include/linux/mutex.h |
| 24 | and implemented in kernel/locking/mutex.c. These locks use a three | 24 | and implemented in kernel/locking/mutex.c. These locks use an atomic variable |
| 25 | state atomic counter (->count) to represent the different possible | 25 | (->owner) to keep track of the lock state during its lifetime. Field owner |
| 26 | transitions that can occur during the lifetime of a lock: | 26 | actually contains 'struct task_struct *' to the current lock owner and it is |
| 27 | 27 | therefore NULL if not currently owned. Since task_struct pointers are aligned | |
| 28 | 1: unlocked | 28 | at at least L1_CACHE_BYTES, low bits (3) are used to store extra state (e.g., |
| 29 | 0: locked, no waiters | 29 | if waiter list is non-empty). In its most basic form it also includes a |
| 30 | negative: locked, with potential waiters | 30 | wait-queue and a spinlock that serializes access to it. Furthermore, |
| 31 | 31 | CONFIG_MUTEX_SPIN_ON_OWNER=y systems use a spinner MCS lock (->osq), described | |
| 32 | In its most basic form it also includes a wait-queue and a spinlock | 32 | below in (ii). |
| 33 | that serializes access to it. CONFIG_SMP systems can also include | ||
| 34 | a pointer to the lock task owner (->owner) as well as a spinner MCS | ||
| 35 | lock (->osq), both described below in (ii). | ||
| 36 | 33 | ||
| 37 | When acquiring a mutex, there are three possible paths that can be | 34 | When acquiring a mutex, there are three possible paths that can be |
| 38 | taken, depending on the state of the lock: | 35 | taken, depending on the state of the lock: |
| 39 | 36 | ||
| 40 | (i) fastpath: tries to atomically acquire the lock by decrementing the | 37 | (i) fastpath: tries to atomically acquire the lock by cmpxchg()ing the owner with |
| 41 | counter. If it was already taken by another task it goes to the next | 38 | the current task. This only works in the uncontended case (cmpxchg() checks |
| 42 | possible path. This logic is architecture specific. On x86-64, the | 39 | against 0UL, so all 3 state bits above have to be 0). If the lock is |
| 43 | locking fastpath is 2 instructions: | 40 | contended it goes to the next possible path. |
| 44 | |||
| 45 | 0000000000000e10 <mutex_lock>: | ||
| 46 | e21: f0 ff 0b lock decl (%rbx) | ||
| 47 | e24: 79 08 jns e2e <mutex_lock+0x1e> | ||
| 48 | |||
| 49 | the unlocking fastpath is equally tight: | ||
| 50 | |||
| 51 | 0000000000000bc0 <mutex_unlock>: | ||
| 52 | bc8: f0 ff 07 lock incl (%rdi) | ||
| 53 | bcb: 7f 0a jg bd7 <mutex_unlock+0x17> | ||
| 54 | |||
| 55 | 41 | ||
| 56 | (ii) midpath: aka optimistic spinning, tries to spin for acquisition | 42 | (ii) midpath: aka optimistic spinning, tries to spin for acquisition |
| 57 | while the lock owner is running and there are no other tasks ready | 43 | while the lock owner is running and there are no other tasks ready |
| @@ -143,11 +129,10 @@ Test if the mutex is taken: · 85 unmodified lines | |||
| 143 | Disadvantages | 129 | Disadvantages |
| 144 | ------------- | 130 | ------------- |
| 145 | 131 | ||
| 146 | Unlike its original design and purpose, 'struct mutex' is larger than | 132 | Unlike its original design and purpose, 'struct mutex' is among the largest |
| 147 | most locks in the kernel. E.g: on x86-64 it is 40 bytes, almost twice | 133 | locks in the kernel. E.g: on x86-64 it is 32 bytes, where 'struct semaphore' |
| 148 | as large as 'struct semaphore' (24 bytes) and tied, along with rwsems, | 134 | is 24 bytes and rw_semaphore is 40 bytes. Larger structure sizes mean more CPU |
| 149 | for the largest lock in the kernel. Larger structure sizes mean more | 135 | cache and memory footprint. |
| 150 | CPU cache and memory footprint. | ||
| 151 | 136 | ||
| 152 | When to use mutexes | 137 | When to use mutexes |
| 153 | ------------------- | 138 | ------------------- |
| @@ -50,9 +50,15 @@ replace typedef dmx_filter_t :c:type:`dmx_filter` · 49 unmodified lines | |||
| 50 | replace typedef dmx_pes_type_t :c:type:`dmx_pes_type` | 50 | replace typedef dmx_pes_type_t :c:type:`dmx_pes_type` |
| 51 | replace typedef dmx_input_t :c:type:`dmx_input` | 51 | replace typedef dmx_input_t :c:type:`dmx_input` |
| 52 | 52 | ||
| 53 | ignore symbol DMX_OUT_DECODER | 53 | replace symbol DMX_BUFFER_FLAG_HAD_CRC32_DISCARD :c:type:`dmx_buffer_flags` |
| 54 | ignore symbol DMX_OUT_TAP | 54 | replace symbol DMX_BUFFER_FLAG_TEI :c:type:`dmx_buffer_flags` |
| 55 | ignore symbol DMX_OUT_TS_TAP | 55 | replace symbol DMX_BUFFER_PKT_COUNTER_MISMATCH :c:type:`dmx_buffer_flags` |
| 56 | ignore symbol DMX_OUT_TSDEMUX_TAP | 56 | replace symbol DMX_BUFFER_FLAG_DISCONTINUITY_DETECTED :c:type:`dmx_buffer_flags` |
| 57 | replace symbol DMX_BUFFER_FLAG_DISCONTINUITY_INDICATOR :c:type:`dmx_buffer_flags` | ||
| 58 | |||
| 59 | replace symbol DMX_OUT_DECODER :c:type:`dmx_output` | ||
| 60 | replace symbol DMX_OUT_TAP :c:type:`dmx_output` | ||
| 61 | replace symbol DMX_OUT_TS_TAP :c:type:`dmx_output` | ||
| 62 | replace symbol DMX_OUT_TSDEMUX_TAP :c:type:`dmx_output` | ||
| 57 | 63 | ||
| 58 | replace ioctl DMX_DQBUF dmx_qbuf | 64 | replace ioctl DMX_DQBUF dmx_qbuf |
| @@ -51,9 +51,10 @@ out to disk. Buffers remain locked until dequeued, until the · 50 unmodified lines | |||
| 51 | the device is closed. | 51 | the device is closed. |
| 52 | 52 | ||
| 53 | Applications call the ``DMX_DQBUF`` ioctl to dequeue a filled | 53 | Applications call the ``DMX_DQBUF`` ioctl to dequeue a filled |
| 54 | (capturing) buffer from the driver's outgoing queue. They just set the ``reserved`` field array to zero. When ``DMX_DQBUF`` is called with a | 54 | (capturing) buffer from the driver's outgoing queue. |
| 55 | pointer to this structure, the driver fills the remaining fields or | 55 | They just set the ``index`` field withe the buffer ID to be queued. |
| 56 | returns an error code. | 56 | When ``DMX_DQBUF`` is called with a pointer to struct :c:type:`dmx_buffer`, |
| 57 | the driver fills the remaining fields or returns an error code. | ||
| 57 | 58 | ||
| 58 | By default ``DMX_DQBUF`` blocks when no buffer is in the outgoing | 59 | By default ``DMX_DQBUF`` blocks when no buffer is in the outgoing |
| 59 | queue. When the ``O_NONBLOCK`` flag was given to the | 60 | queue. When the ``O_NONBLOCK`` flag was given to the |
| @@ -13,6 +13,7 @@ The following technologies are described: · 12 unmodified lines | |||
| 13 | * Generic Segmentation Offload - GSO | 13 | * Generic Segmentation Offload - GSO |
| 14 | * Generic Receive Offload - GRO | 14 | * Generic Receive Offload - GRO |
| 15 | * Partial Generic Segmentation Offload - GSO_PARTIAL | 15 | * Partial Generic Segmentation Offload - GSO_PARTIAL |
| 16 | * SCTP accelleration with GSO - GSO_BY_FRAGS | ||
| 16 | 17 | ||
| 17 | TCP Segmentation Offload | 18 | TCP Segmentation Offload |
| 18 | ======================== | 19 | ======================== |
| @@ -49,6 +50,10 @@ datagram into multiple IPv4 fragments. Many of the requirements for UDP · 30 unmodified lines | |||
| 49 | fragmentation offload are the same as TSO. However the IPv4 ID for | 50 | fragmentation offload are the same as TSO. However the IPv4 ID for |
| 50 | fragments should not increment as a single IPv4 datagram is fragmented. | 51 | fragments should not increment as a single IPv4 datagram is fragmented. |
| 51 | 52 | ||
| 53 | UFO is deprecated: modern kernels will no longer generate UFO skbs, but can | ||
| 54 | still receive them from tuntap and similar devices. Offload of UDP-based | ||
| 55 | tunnel protocols is still supported. | ||
| 56 | |||
| 52 | IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads | 57 | IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads |
| 53 | ======================================================== | 58 | ======================================================== |
| 54 | 59 | ||
| @@ -83,10 +88,10 @@ SKB_GSO_UDP_TUNNEL_CSUM. These two additional tunnel types reflect the · 28 unmodified lines | |||
| 83 | fact that the outer header also requests to have a non-zero checksum | 88 | fact that the outer header also requests to have a non-zero checksum |
| 84 | included in the outer header. | 89 | included in the outer header. |
| 85 | 90 | ||
| 86 | Finally there is SKB_GSO_REMCSUM which indicates that a given tunnel header | 91 | Finally there is SKB_GSO_TUNNEL_REMCSUM which indicates that a given tunnel |
| 87 | has requested a remote checksum offload. In this case the inner headers | 92 | header has requested a remote checksum offload. In this case the inner |
| 88 | will be left with a partial checksum and only the outer header checksum | 93 | headers will be left with a partial checksum and only the outer header |
| 89 | will be computed. | 94 | checksum will be computed. |
| 90 | 95 | ||
| 91 | Generic Segmentation Offload | 96 | Generic Segmentation Offload |
| 92 | ============================ | 97 | ============================ |
| @@ -128,3 +133,28 @@ values for if the header was simply duplicated. The one exception to this · 35 unmodified lines | |||
| 128 | is the outer IPv4 ID field. It is up to the device drivers to guarantee | 133 | is the outer IPv4 ID field. It is up to the device drivers to guarantee |
| 129 | that the IPv4 ID field is incremented in the case that a given header does | 134 | that the IPv4 ID field is incremented in the case that a given header does |
| 130 | not have the DF bit set. | 135 | not have the DF bit set. |
| 136 | |||
| 137 | SCTP accelleration with GSO | ||
| 138 | =========================== | ||
| 139 | |||
| 140 | SCTP - despite the lack of hardware support - can still take advantage of | ||
| 141 | GSO to pass one large packet through the network stack, rather than | ||
| 142 | multiple small packets. | ||
| 143 | |||
| 144 | This requires a different approach to other offloads, as SCTP packets | ||
| 145 | cannot be just segmented to (P)MTU. Rather, the chunks must be contained in | ||
| 146 | IP segments, padding respected. So unlike regular GSO, SCTP can't just | ||
| 147 | generate a big skb, set gso_size to the fragmentation point and deliver it | ||
| 148 | to IP layer. | ||
| 149 | |||
| 150 | Instead, the SCTP protocol layer builds an skb with the segments correctly | ||
| 151 | padded and stored as chained skbs, and skb_segment() splits based on those. | ||
| 152 | To signal this, gso_size is set to the special value GSO_BY_FRAGS. | ||
| 153 | |||
| 154 | Therefore, any code in the core networking stack must be aware of the | ||
| 155 | possibility that gso_size will be GSO_BY_FRAGS and handle that case | ||
| 156 | appropriately. (For size checks, the skb_gso_validate_*_len family of | ||
| 157 | helpers do this automatically.) | ||
| 158 | |||
| 159 | This also affects drivers with the NETIF_F_FRAGLIST & NETIF_F_GSO_SCTP bits | ||
| 160 | set. Note also that NETIF_F_GSO_SCTP is included in NETIF_F_GSO_SOFTWARE. | ||
| @@ -36,8 +36,7 @@ import glob · 35 unmodified lines | |||
| 36 | 36 | ||
| 37 | from docutils import nodes, statemachine | 37 | from docutils import nodes, statemachine |
| 38 | from docutils.statemachine import ViewList | 38 | from docutils.statemachine import ViewList |
| 39 | from docutils.parsers.rst import directives | 39 | from docutils.parsers.rst import directives, Directive |
| 40 | from sphinx.util.compat import Directive | ||
| 41 | from sphinx.ext.autodoc import AutodocReporter | 40 | from sphinx.ext.autodoc import AutodocReporter |
| 42 | 41 | ||
| 43 | __version__ = '1.0' | 42 | __version__ = '1.0' |
| @@ -123,14 +123,15 @@ memory layout to fit in user mode), check KVM_CAP_MIPS_VZ and use the · 122 unmodified lines | |||
| 123 | flag KVM_VM_MIPS_VZ. | 123 | flag KVM_VM_MIPS_VZ. |
| 124 | 124 | ||
| 125 | 125 | ||
| 126 | 4.3 KVM_GET_MSR_INDEX_LIST | 126 | 4.3 KVM_GET_MSR_INDEX_LIST, KVM_GET_MSR_FEATURE_INDEX_LIST |
| 127 | 127 | ||
| 128 | Capability: basic | 128 | Capability: basic, KVM_CAP_GET_MSR_FEATURES for KVM_GET_MSR_FEATURE_INDEX_LIST |
| 129 | Architectures: x86 | 129 | Architectures: x86 |
| 130 | Type: system | 130 | Type: system ioctl |
| 131 | Parameters: struct kvm_msr_list (in/out) | 131 | Parameters: struct kvm_msr_list (in/out) |
| 132 | Returns: 0 on success; -1 on error | 132 | Returns: 0 on success; -1 on error |
| 133 | Errors: | 133 | Errors: |
| 134 | EFAULT: the msr index list cannot be read from or written to | ||
| 134 | E2BIG: the msr index list is to be to fit in the array specified by | 135 | E2BIG: the msr index list is to be to fit in the array specified by |
| 135 | the user. | 136 | the user. |
| 136 | 137 | ||
| @@ -139,16 +140,23 @@ struct kvm_msr_list { · 2 unmodified lines | |||
| 139 | __u32 indices[0]; | 140 | __u32 indices[0]; |
| 140 | }; | 141 | }; |
| 141 | 142 | ||
| 142 | This ioctl returns the guest msrs that are supported. The list varies | 143 | The user fills in the size of the indices array in nmsrs, and in return |
| 143 | by kvm version and host processor, but does not change otherwise. The | 144 | kvm adjusts nmsrs to reflect the actual number of msrs and fills in the |
| 144 | user fills in the size of the indices array in nmsrs, and in return | 145 | indices array with their numbers. |
| 145 | kvm adjusts nmsrs to reflect the actual number of msrs and fills in | 146 | |
| 146 | the indices array with their numbers. | 147 | KVM_GET_MSR_INDEX_LIST returns the guest msrs that are supported. The list |
| 148 | varies by kvm version and host processor, but does not change otherwise. | ||
| 147 | 149 | ||
| 148 | Note: if kvm indicates supports MCE (KVM_CAP_MCE), then the MCE bank MSRs are | 150 | Note: if kvm indicates supports MCE (KVM_CAP_MCE), then the MCE bank MSRs are |
| 149 | not returned in the MSR list, as different vcpus can have a different number | 151 | not returned in the MSR list, as different vcpus can have a different number |
| 150 | of banks, as set via the KVM_X86_SETUP_MCE ioctl. | 152 | of banks, as set via the KVM_X86_SETUP_MCE ioctl. |
| 151 | 153 | ||
| 154 | KVM_GET_MSR_FEATURE_INDEX_LIST returns the list of MSRs that can be passed | ||
| 155 | to the KVM_GET_MSRS system ioctl. This lets userspace probe host capabilities | ||
| 156 | and processor features that are exposed via MSRs (e.g., VMX capabilities). | ||
| 157 | This list also varies by kvm version and host processor, but does not change | ||
| 158 | otherwise. | ||
| 159 | |||
| 152 | 160 | ||
| 153 | 4.4 KVM_CHECK_EXTENSION | 161 | 4.4 KVM_CHECK_EXTENSION |
| 154 | 162 | ||
| @@ -475,14 +483,22 @@ Support for this has been removed. Use KVM_SET_GUEST_DEBUG instead. · 320 unmodified lines | |||
| 475 | 483 | ||
| 476 | 4.18 KVM_GET_MSRS | 484 | 4.18 KVM_GET_MSRS |
| 477 | 485 | ||
| 478 | Capability: basic | 486 | Capability: basic (vcpu), KVM_CAP_GET_MSR_FEATURES (system) |
| 479 | Architectures: x86 | 487 | Architectures: x86 |
| 480 | Type: vcpu ioctl | 488 | Type: system ioctl, vcpu ioctl |
| 481 | Parameters: struct kvm_msrs (in/out) | 489 | Parameters: struct kvm_msrs (in/out) |
| 482 | Returns: 0 on success, -1 on error | 490 | Returns: number of msrs successfully returned; |
| 491 | -1 on error | ||
| 492 | |||
| 493 | When used as a system ioctl: | ||
| 494 | Reads the values of MSR-based features that are available for the VM. This | ||
| 495 | is similar to KVM_GET_SUPPORTED_CPUID, but it returns MSR indices and values. | ||
| 496 | The list of msr-based features can be obtained using KVM_GET_MSR_FEATURE_INDEX_LIST | ||
| 497 | in a system ioctl. | ||
| 483 | 498 | ||
| 499 | When used as a vcpu ioctl: | ||
| 484 | Reads model-specific registers from the vcpu. Supported msr indices can | 500 | Reads model-specific registers from the vcpu. Supported msr indices can |
| 485 | be obtained using KVM_GET_MSR_INDEX_LIST. | 501 | be obtained using KVM_GET_MSR_INDEX_LIST in a system ioctl. |
| 486 | 502 | ||
| 487 | struct kvm_msrs { | 503 | struct kvm_msrs { |
| 488 | __u32 nmsrs; /* number of msrs in entries */ | 504 | __u32 nmsrs; /* number of msrs in entries */ |
| @@ -58,6 +58,10 @@ KVM_FEATURE_PV_TLB_FLUSH || 9 || guest checks this feature bit · 57 unmodified lines | |||
| 58 | || || before enabling paravirtualized | 58 | || || before enabling paravirtualized |
| 59 | || || tlb flush. | 59 | || || tlb flush. |
| 60 | ------------------------------------------------------------------------------ | 60 | ------------------------------------------------------------------------------ |
| 61 | KVM_FEATURE_ASYNC_PF_VMEXIT || 10 || paravirtualized async PF VM exit | ||
| 62 | || || can be enabled by setting bit 2 | ||
| 63 | || || when writing to msr 0x4b564d02 | ||
| 64 | ------------------------------------------------------------------------------ | ||
| 61 | KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side | 65 | KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side |
| 62 | || || per-cpu warps are expected in | 66 | || || per-cpu warps are expected in |
| 63 | || || kvmclock. | 67 | || || kvmclock. |
| @@ -170,7 +170,8 @@ MSR_KVM_ASYNC_PF_EN: 0x4b564d02 · 169 unmodified lines | |||
| 170 | when asynchronous page faults are enabled on the vcpu 0 when | 170 | when asynchronous page faults are enabled on the vcpu 0 when |
| 171 | disabled. Bit 1 is 1 if asynchronous page faults can be injected | 171 | disabled. Bit 1 is 1 if asynchronous page faults can be injected |
| 172 | when vcpu is in cpl == 0. Bit 2 is 1 if asynchronous page faults | 172 | when vcpu is in cpl == 0. Bit 2 is 1 if asynchronous page faults |
| 173 | are delivered to L1 as #PF vmexits. | 173 | are delivered to L1 as #PF vmexits. Bit 2 can be set only if |
| 174 | KVM_FEATURE_ASYNC_PF_VMEXIT is present in CPUID. | ||
| 174 | 175 | ||
| 175 | First 4 byte of 64 byte memory location will be written to by | 176 | First 4 byte of 64 byte memory location will be written to by |
| 176 | the hypervisor at the time of asynchronous page fault (APF) | 177 | the hypervisor at the time of asynchronous page fault (APF) |
| @@ -671,7 +671,7 @@ occupancy of the real time threads on these cores. · 670 unmodified lines | |||
| 671 | # mkdir p1 | 671 | # mkdir p1 |
| 672 | 672 | ||
| 673 | Move the cpus 4-7 over to p1 | 673 | Move the cpus 4-7 over to p1 |
| 674 | # echo f0 > p0/cpus | 674 | # echo f0 > p1/cpus |
| 675 | 675 | ||
| 676 | View the llc occupancy snapshot | 676 | View the llc occupancy snapshot |
| 677 | 677 | ||
| @@ -108,7 +108,7 @@ The topology of a system is described in the units of: · 107 unmodified lines | |||
| 108 | 108 | ||
| 109 | The number of online threads is also printed in /proc/cpuinfo "siblings." | 109 | The number of online threads is also printed in /proc/cpuinfo "siblings." |
| 110 | 110 | ||
| 111 | - topology_sibling_mask(): | 111 | - topology_sibling_cpumask(): |
| 112 | 112 | ||
| 113 | The cpumask contains all online threads in the core to which a thread | 113 | The cpumask contains all online threads in the core to which a thread |
| 114 | belongs. | 114 | belongs. |
| @@ -1238,7 +1238,7 @@ F: drivers/clk/at91 · 1237 unmodified lines | |||
| 1238 | 1238 | ||
| 1239 | ARM/ATMEL AT91RM9200, AT91SAM9 AND SAMA5 SOC SUPPORT | 1239 | ARM/ATMEL AT91RM9200, AT91SAM9 AND SAMA5 SOC SUPPORT |
| 1240 | M: Nicolas Ferre <nicolas.ferre@microchip.com> | 1240 | M: Nicolas Ferre <nicolas.ferre@microchip.com> |
| 1241 | M: Alexandre Belloni <alexandre.belloni@free-electrons.com> | 1241 | M: Alexandre Belloni <alexandre.belloni@bootlin.com> |
| 1242 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 1242 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 1243 | W: http://www.linux4sam.org | 1243 | W: http://www.linux4sam.org |
| 1244 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/nferre/linux-at91.git | 1244 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/nferre/linux-at91.git |
| @@ -1590,7 +1590,7 @@ ARM/Marvell Dove/MV78xx0/Orion SOC support · 345 unmodified lines | |||
| 1590 | M: Jason Cooper <jason@lakedaemon.net> | 1590 | M: Jason Cooper <jason@lakedaemon.net> |
| 1591 | M: Andrew Lunn <andrew@lunn.ch> | 1591 | M: Andrew Lunn <andrew@lunn.ch> |
| 1592 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> | 1592 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> |
| 1593 | M: Gregory Clement <gregory.clement@free-electrons.com> | 1593 | M: Gregory Clement <gregory.clement@bootlin.com> |
| 1594 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 1594 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 1595 | S: Maintained | 1595 | S: Maintained |
| 1596 | F: Documentation/devicetree/bindings/soc/dove/ | 1596 | F: Documentation/devicetree/bindings/soc/dove/ |
| @@ -1604,7 +1604,7 @@ F: arch/arm/boot/dts/orion5x* · 7 unmodified lines | |||
| 1604 | ARM/Marvell Kirkwood and Armada 370, 375, 38x, 39x, XP, 3700, 7K/8K SOC support | 1604 | ARM/Marvell Kirkwood and Armada 370, 375, 38x, 39x, XP, 3700, 7K/8K SOC support |
| 1605 | M: Jason Cooper <jason@lakedaemon.net> | 1605 | M: Jason Cooper <jason@lakedaemon.net> |
| 1606 | M: Andrew Lunn <andrew@lunn.ch> | 1606 | M: Andrew Lunn <andrew@lunn.ch> |
| 1607 | M: Gregory Clement <gregory.clement@free-electrons.com> | 1607 | M: Gregory Clement <gregory.clement@bootlin.com> |
| 1608 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> | 1608 | M: Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com> |
| 1609 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 1609 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 1610 | S: Maintained | 1610 | S: Maintained |
| @@ -1999,8 +1999,10 @@ M: Maxime Coquelin <mcoquelin.stm32@gmail.com> · 388 unmodified lines | |||
| 1999 | M: Alexandre Torgue <alexandre.torgue@st.com> | 1999 | M: Alexandre Torgue <alexandre.torgue@st.com> |
| 2000 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 2000 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 2001 | S: Maintained | 2001 | S: Maintained |
| 2002 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/mcoquelin/stm32.git | 2002 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git stm32-next |
| 2003 | N: stm32 | 2003 | N: stm32 |
| 2004 | F: arch/arm/boot/dts/stm32* | ||
| 2005 | F: arch/arm/mach-stm32/ | ||
| 2004 | F: drivers/clocksource/armv7m_systick.c | 2006 | F: drivers/clocksource/armv7m_systick.c |
| 2005 | 2007 | ||
| 2006 | ARM/TANGO ARCHITECTURE | 2008 | ARM/TANGO ARCHITECTURE |
| @@ -7600,8 +7602,10 @@ F: mm/kasan/ · 5593 unmodified lines | |||
| 7600 | F: scripts/Makefile.kasan | 7602 | F: scripts/Makefile.kasan |
| 7601 | 7603 | ||
| 7602 | KCONFIG | 7604 | KCONFIG |
| 7605 | M: Masahiro Yamada <yamada.masahiro@socionext.com> | ||
| 7606 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig | ||
| 7603 | L: linux-kbuild@vger.kernel.org | 7607 | L: linux-kbuild@vger.kernel.org |
| 7604 | S: Orphan | 7608 | S: Maintained |
| 7605 | F: Documentation/kbuild/kconfig-language.txt | 7609 | F: Documentation/kbuild/kconfig-language.txt |
| 7606 | F: scripts/kconfig/ | 7610 | F: scripts/kconfig/ |
| 7607 | 7611 | ||
| @@ -7909,7 +7913,6 @@ S: Maintained · 301 unmodified lines | |||
| 7909 | F: scripts/leaking_addresses.pl | 7913 | F: scripts/leaking_addresses.pl |
| 7910 | 7914 | ||
| 7911 | LED SUBSYSTEM | 7915 | LED SUBSYSTEM |
| 7912 | M: Richard Purdie <rpurdie@rpsys.net> | ||
| 7913 | M: Jacek Anaszewski <jacek.anaszewski@gmail.com> | 7916 | M: Jacek Anaszewski <jacek.anaszewski@gmail.com> |
| 7914 | M: Pavel Machek <pavel@ucw.cz> | 7917 | M: Pavel Machek <pavel@ucw.cz> |
| 7915 | L: linux-leds@vger.kernel.org | 7918 | L: linux-leds@vger.kernel.org |
| @@ -9206,6 +9209,7 @@ MIPS GENERIC PLATFORM · 1290 unmodified lines | |||
| 9206 | M: Paul Burton <paul.burton@mips.com> | 9209 | M: Paul Burton <paul.burton@mips.com> |
| 9207 | L: linux-mips@linux-mips.org | 9210 | L: linux-mips@linux-mips.org |
| 9208 | S: Supported | 9211 | S: Supported |
| 9212 | F: Documentation/devicetree/bindings/power/mti,mips-cpc.txt | ||
| 9209 | F: arch/mips/generic/ | 9213 | F: arch/mips/generic/ |
| 9210 | F: arch/mips/tools/generic-board-config.sh | 9214 | F: arch/mips/tools/generic-board-config.sh |
| 9211 | 9215 | ||
| @@ -9921,6 +9925,13 @@ F: Documentation/ABI/stable/sysfs-bus-nvmem · 709 unmodified lines | |||
| 9921 | F: include/linux/nvmem-consumer.h | 9925 | F: include/linux/nvmem-consumer.h |
| 9922 | F: include/linux/nvmem-provider.h | 9926 | F: include/linux/nvmem-provider.h |
| 9923 | 9927 | ||
| 9928 | NXP SGTL5000 DRIVER | ||
| 9929 | M: Fabio Estevam <fabio.estevam@nxp.com> | ||
| 9930 | L: alsa-devel@alsa-project.org (moderated for non-subscribers) | ||
| 9931 | S: Maintained | ||
| 9932 | F: Documentation/devicetree/bindings/sound/sgtl5000.txt | ||
| 9933 | F: sound/soc/codecs/sgtl5000* | ||
| 9934 | |||
| 9924 | NXP TDA998X DRM DRIVER | 9935 | NXP TDA998X DRM DRIVER |
| 9925 | M: Russell King <linux@armlinux.org.uk> | 9936 | M: Russell King <linux@armlinux.org.uk> |
| 9926 | S: Supported | 9937 | S: Supported |
| @@ -9945,6 +9956,7 @@ F: drivers/nfc/nxp-nci · 18 unmodified lines | |||
| 9945 | 9956 | ||
| 9946 | OBJTOOL | 9957 | OBJTOOL |
| 9947 | M: Josh Poimboeuf <jpoimboe@redhat.com> | 9958 | M: Josh Poimboeuf <jpoimboe@redhat.com> |
| 9959 | M: Peter Zijlstra <peterz@infradead.org> | ||
| 9948 | S: Supported | 9960 | S: Supported |
| 9949 | F: tools/objtool/ | 9961 | F: tools/objtool/ |
| 9950 | 9962 | ||
| @@ -10925,6 +10937,17 @@ L: linux-gpio@vger.kernel.org · 974 unmodified lines | |||
| 10925 | S: Supported | 10937 | S: Supported |
| 10926 | F: drivers/pinctrl/pinctrl-at91-pio4.* | 10938 | F: drivers/pinctrl/pinctrl-at91-pio4.* |
| 10927 | 10939 | ||
| 10940 | PIN CONTROLLER - FREESCALE | ||
| 10941 | M: Dong Aisheng <aisheng.dong@nxp.com> | ||
| 10942 | M: Fabio Estevam <festevam@gmail.com> | ||
| 10943 | M: Shawn Guo <shawnguo@kernel.org> | ||
| 10944 | M: Stefan Agner <stefan@agner.ch> | ||
| 10945 | R: Pengutronix Kernel Team <kernel@pengutronix.de> | ||
| 10946 | L: linux-gpio@vger.kernel.org | ||
| 10947 | S: Maintained | ||
| 10948 | F: drivers/pinctrl/freescale/ | ||
| 10949 | F: Documentation/devicetree/bindings/pinctrl/fsl,* | ||
| 10950 | |||
| 10928 | PIN CONTROLLER - INTEL | 10951 | PIN CONTROLLER - INTEL |
| 10929 | M: Mika Westerberg <mika.westerberg@linux.intel.com> | 10952 | M: Mika Westerberg <mika.westerberg@linux.intel.com> |
| 10930 | M: Heikki Krogerus <heikki.krogerus@linux.intel.com> | 10953 | M: Heikki Krogerus <heikki.krogerus@linux.intel.com> |
| @@ -12091,6 +12114,7 @@ M: Sylwester Nawrocki <s.nawrocki@samsung.com> · 1160 unmodified lines | |||
| 12091 | L: alsa-devel@alsa-project.org (moderated for non-subscribers) | 12114 | L: alsa-devel@alsa-project.org (moderated for non-subscribers) |
| 12092 | S: Supported | 12115 | S: Supported |
| 12093 | F: sound/soc/samsung/ | 12116 | F: sound/soc/samsung/ |
| 12117 | F: Documentation/devicetree/bindings/sound/samsung* | ||
| 12094 | 12118 | ||
| 12095 | SAMSUNG EXYNOS PSEUDO RANDOM NUMBER GENERATOR (RNG) DRIVER | 12119 | SAMSUNG EXYNOS PSEUDO RANDOM NUMBER GENERATOR (RNG) DRIVER |
| 12096 | M: Krzysztof Kozlowski <krzk@kernel.org> | 12120 | M: Krzysztof Kozlowski <krzk@kernel.org> |
| @@ -2,7 +2,7 @@ · 1 unmodified line | |||
| 2 | VERSION = 4 | 2 | VERSION = 4 |
| 3 | PATCHLEVEL = 16 | 3 | PATCHLEVEL = 16 |
| 4 | SUBLEVEL = 0 | 4 | SUBLEVEL = 0 |
| 5 | EXTRAVERSION = -rc1 | 5 | EXTRAVERSION = -rc5 |
| 6 | NAME = Fearless Coyote | 6 | NAME = Fearless Coyote |
| 7 | 7 | ||
| 8 | # *DOCUMENTATION* | 8 | # *DOCUMENTATION* |
| @@ -388,7 +388,7 @@ PYTHON = python · 379 unmodified lines | |||
| 388 | CHECK = sparse | 388 | CHECK = sparse |
| 389 | 389 | ||
| 390 | CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ | 390 | CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ |
| 391 | -Wbitwise -Wno-return-void $(CF) | 391 | -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) |
| 392 | NOSTDINC_FLAGS = | 392 | NOSTDINC_FLAGS = |
| 393 | CFLAGS_MODULE = | 393 | CFLAGS_MODULE = |
| 394 | AFLAGS_MODULE = | 394 | AFLAGS_MODULE = |
| @@ -489,6 +489,11 @@ KBUILD_CFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) · 94 unmodified lines | |||
| 489 | KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) | 489 | KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) |
| 490 | endif | 490 | endif |
| 491 | 491 | ||
| 492 | RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register | ||
| 493 | RETPOLINE_CFLAGS_CLANG := -mretpoline-external-thunk | ||
| 494 | RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(RETPOLINE_CFLAGS_CLANG))) | ||
| 495 | export RETPOLINE_CFLAGS | ||
| 496 | |||
| 492 | ifeq ($(config-targets),1) | 497 | ifeq ($(config-targets),1) |
| 493 | # =========================================================================== | 498 | # =========================================================================== |
| 494 | # *config targets only - make sure prerequisites are updated, and descend | 499 | # *config targets only - make sure prerequisites are updated, and descend |
| @@ -579,10 +584,9 @@ ifeq ($(KBUILD_EXTMOD),) · 84 unmodified lines | |||
| 579 | # To avoid any implicit rule to kick in, define an empty command | 584 | # To avoid any implicit rule to kick in, define an empty command |
| 580 | $(KCONFIG_CONFIG) include/config/auto.conf.cmd: ; | 585 | $(KCONFIG_CONFIG) include/config/auto.conf.cmd: ; |
| 581 | 586 | ||
| 582 | # If .config is newer than include/config/auto.conf, someone tinkered | 587 | # The actual configuration files used during the build are stored in |
| 583 | # with it and forgot to run make oldconfig. | 588 | # include/generated/ and include/config/. Update them if .config is newer than |
| 584 | # if auto.conf.cmd is missing then we are probably in a cleaned tree so | 589 | # include/config/auto.conf (which mirrors .config). |
| 585 | # we execute the config step to be sure to catch updated Kconfig files | ||
| 586 | include/config/%.conf: $(KCONFIG_CONFIG) include/config/auto.conf.cmd | 590 | include/config/%.conf: $(KCONFIG_CONFIG) include/config/auto.conf.cmd |
| 587 | $(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig | 591 | $(Q)$(MAKE) -f $(srctree)/Makefile silentoldconfig |
| 588 | else | 592 | else |
| @@ -857,8 +861,7 @@ KBUILD_AFLAGS += $(ARCH_AFLAGS) $(KAFLAGS) · 268 unmodified lines | |||
| 857 | KBUILD_CFLAGS += $(ARCH_CFLAGS) $(KCFLAGS) | 861 | KBUILD_CFLAGS += $(ARCH_CFLAGS) $(KCFLAGS) |
| 858 | 862 | ||
| 859 | # Use --build-id when available. | 863 | # Use --build-id when available. |
| 860 | LDFLAGS_BUILD_ID := $(patsubst -Wl$(comma)%,%,\ | 864 | LDFLAGS_BUILD_ID := $(call ld-option, --build-id) |
| 861 | $(call cc-ldoption, -Wl$(comma)--build-id,)) | ||
| 862 | KBUILD_LDFLAGS_MODULE += $(LDFLAGS_BUILD_ID) | 865 | KBUILD_LDFLAGS_MODULE += $(LDFLAGS_BUILD_ID) |
| 863 | LDFLAGS_vmlinux += $(LDFLAGS_BUILD_ID) | 866 | LDFLAGS_vmlinux += $(LDFLAGS_BUILD_ID) |
| 864 | 867 | ||
| @@ -6,7 +6,6 @@ · 5 unmodified lines | |||
| 6 | * Atomic exchange routines. | 6 | * Atomic exchange routines. |
| 7 | */ | 7 | */ |
| 8 | 8 | ||
| 9 | #define __ASM__MB | ||
| 10 | #define ____xchg(type, args...) __xchg ## type ## _local(args) | 9 | #define ____xchg(type, args...) __xchg ## type ## _local(args) |
| 11 | #define ____cmpxchg(type, args...) __cmpxchg ## type ## _local(args) | 10 | #define ____cmpxchg(type, args...) __cmpxchg ## type ## _local(args) |
| 12 | #include <asm/xchg.h> | 11 | #include <asm/xchg.h> |
| @@ -33,10 +32,6 @@ · 20 unmodified lines | |||
| 33 | cmpxchg_local((ptr), (o), (n)); \ | 32 | cmpxchg_local((ptr), (o), (n)); \ |
| 34 | }) | 33 | }) |
| 35 | 34 | ||
| 36 | #ifdef CONFIG_SMP | ||
| 37 | #undef __ASM__MB | ||
| 38 | #define __ASM__MB "\tmb\n" | ||
| 39 | #endif | ||
| 40 | #undef ____xchg | 35 | #undef ____xchg |
| 41 | #undef ____cmpxchg | 36 | #undef ____cmpxchg |
| 42 | #define ____xchg(type, args...) __xchg ##type(args) | 37 | #define ____xchg(type, args...) __xchg ##type(args) |
| @@ -64,7 +59,6 @@ · 21 unmodified lines | |||
| 64 | cmpxchg((ptr), (o), (n)); \ | 59 | cmpxchg((ptr), (o), (n)); \ |
| 65 | }) | 60 | }) |
| 66 | 61 | ||
| 67 | #undef __ASM__MB | ||
| 68 | #undef ____cmpxchg | 62 | #undef ____cmpxchg |
| 69 | 63 | ||
| 70 | #endif /* _ALPHA_CMPXCHG_H */ | 64 | #endif /* _ALPHA_CMPXCHG_H */ |
| @@ -12,6 +12,10 @@ · 11 unmodified lines | |||
| 12 | * Atomic exchange. | 12 | * Atomic exchange. |
| 13 | * Since it can be used to implement critical sections | 13 | * Since it can be used to implement critical sections |
| 14 | * it must clobber "memory" (also for interrupts in UP). | 14 | * it must clobber "memory" (also for interrupts in UP). |
| 15 | * | ||
| 16 | * The leading and the trailing memory barriers guarantee that these | ||
| 17 | * operations are fully ordered. | ||
| 18 | * | ||
| 15 | */ | 19 | */ |
| 16 | 20 | ||
| 17 | static inline unsigned long | 21 | static inline unsigned long |
| @@ -19,6 +23,7 @@ ____xchg(_u8, volatile char *m, unsigned long val) · 1 unmodified line | |||
| 19 | { | 23 | { |
| 20 | unsigned long ret, tmp, addr64; | 24 | unsigned long ret, tmp, addr64; |
| 21 | 25 | ||
| 26 | smp_mb(); | ||
| 22 | __asm__ __volatile__( | 27 | __asm__ __volatile__( |
| 23 | " andnot %4,7,%3\n" | 28 | " andnot %4,7,%3\n" |
| 24 | " insbl %1,%4,%1\n" | 29 | " insbl %1,%4,%1\n" |
| @@ -28,12 +33,12 @@ ____xchg(_u8, volatile char *m, unsigned long val) · 3 unmodified lines | |||
| 28 | " or %1,%2,%2\n" | 33 | " or %1,%2,%2\n" |
| 29 | " stq_c %2,0(%3)\n" | 34 | " stq_c %2,0(%3)\n" |
| 30 | " beq %2,2f\n" | 35 | " beq %2,2f\n" |
| 31 | __ASM__MB | ||
| 32 | ".subsection 2\n" | 36 | ".subsection 2\n" |
| 33 | "2: br 1b\n" | 37 | "2: br 1b\n" |
| 34 | ".previous" | 38 | ".previous" |
| 35 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) | 39 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) |
| 36 | : "r" ((long)m), "1" (val) : "memory"); | 40 | : "r" ((long)m), "1" (val) : "memory"); |
| 41 | smp_mb(); | ||
| 37 | 42 | ||
| 38 | return ret; | 43 | return ret; |
| 39 | } | 44 | } |
| @@ -43,6 +48,7 @@ ____xchg(_u16, volatile short *m, unsigned long val) · 3 unmodified lines | |||
| 43 | { | 48 | { |
| 44 | unsigned long ret, tmp, addr64; | 49 | unsigned long ret, tmp, addr64; |
| 45 | 50 | ||
| 51 | smp_mb(); | ||
| 46 | __asm__ __volatile__( | 52 | __asm__ __volatile__( |
| 47 | " andnot %4,7,%3\n" | 53 | " andnot %4,7,%3\n" |
| 48 | " inswl %1,%4,%1\n" | 54 | " inswl %1,%4,%1\n" |
| @@ -52,12 +58,12 @@ ____xchg(_u16, volatile short *m, unsigned long val) · 3 unmodified lines | |||
| 52 | " or %1,%2,%2\n" | 58 | " or %1,%2,%2\n" |
| 53 | " stq_c %2,0(%3)\n" | 59 | " stq_c %2,0(%3)\n" |
| 54 | " beq %2,2f\n" | 60 | " beq %2,2f\n" |
| 55 | __ASM__MB | ||
| 56 | ".subsection 2\n" | 61 | ".subsection 2\n" |
| 57 | "2: br 1b\n" | 62 | "2: br 1b\n" |
| 58 | ".previous" | 63 | ".previous" |
| 59 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) | 64 | : "=&r" (ret), "=&r" (val), "=&r" (tmp), "=&r" (addr64) |
| 60 | : "r" ((long)m), "1" (val) : "memory"); | 65 | : "r" ((long)m), "1" (val) : "memory"); |
| 66 | smp_mb(); | ||
| 61 | 67 | ||
| 62 | return ret; | 68 | return ret; |
| 63 | } | 69 | } |
| @@ -67,17 +73,18 @@ ____xchg(_u32, volatile int *m, unsigned long val) · 3 unmodified lines | |||
| 67 | { | 73 | { |
| 68 | unsigned long dummy; | 74 | unsigned long dummy; |
| 69 | 75 | ||
| 76 | smp_mb(); | ||
| 70 | __asm__ __volatile__( | 77 | __asm__ __volatile__( |
| 71 | "1: ldl_l %0,%4\n" | 78 | "1: ldl_l %0,%4\n" |
| 72 | " bis $31,%3,%1\n" | 79 | " bis $31,%3,%1\n" |
| 73 | " stl_c %1,%2\n" | 80 | " stl_c %1,%2\n" |
| 74 | " beq %1,2f\n" | 81 | " beq %1,2f\n" |
| 75 | __ASM__MB | ||
| 76 | ".subsection 2\n" | 82 | ".subsection 2\n" |
| 77 | "2: br 1b\n" | 83 | "2: br 1b\n" |
| 78 | ".previous" | 84 | ".previous" |
| 79 | : "=&r" (val), "=&r" (dummy), "=m" (*m) | 85 | : "=&r" (val), "=&r" (dummy), "=m" (*m) |
| 80 | : "rI" (val), "m" (*m) : "memory"); | 86 | : "rI" (val), "m" (*m) : "memory"); |
| 87 | smp_mb(); | ||
| 81 | 88 | ||
| 82 | return val; | 89 | return val; |
| 83 | } | 90 | } |
| @@ -87,17 +94,18 @@ ____xchg(_u64, volatile long *m, unsigned long val) · 3 unmodified lines | |||
| 87 | { | 94 | { |
| 88 | unsigned long dummy; | 95 | unsigned long dummy; |
| 89 | 96 | ||
| 97 | smp_mb(); | ||
| 90 | __asm__ __volatile__( | 98 | __asm__ __volatile__( |
| 91 | "1: ldq_l %0,%4\n" | 99 | "1: ldq_l %0,%4\n" |
| 92 | " bis $31,%3,%1\n" | 100 | " bis $31,%3,%1\n" |
| 93 | " stq_c %1,%2\n" | 101 | " stq_c %1,%2\n" |
| 94 | " beq %1,2f\n" | 102 | " beq %1,2f\n" |
| 95 | __ASM__MB | ||
| 96 | ".subsection 2\n" | 103 | ".subsection 2\n" |
| 97 | "2: br 1b\n" | 104 | "2: br 1b\n" |
| 98 | ".previous" | 105 | ".previous" |
| 99 | : "=&r" (val), "=&r" (dummy), "=m" (*m) | 106 | : "=&r" (val), "=&r" (dummy), "=m" (*m) |
| 100 | : "rI" (val), "m" (*m) : "memory"); | 107 | : "rI" (val), "m" (*m) : "memory"); |
| 108 | smp_mb(); | ||
| 101 | 109 | ||
| 102 | return val; | 110 | return val; |
| 103 | } | 111 | } |
| @@ -128,10 +136,12 @@ ____xchg(, volatile void *ptr, unsigned long x, int size) · 24 unmodified lines | |||
| 128 | * store NEW in MEM. Return the initial value in MEM. Success is | 136 | * store NEW in MEM. Return the initial value in MEM. Success is |
| 129 | * indicated by comparing RETURN with OLD. | 137 | * indicated by comparing RETURN with OLD. |
| 130 | * | 138 | * |
| 131 | * The memory barrier should be placed in SMP only when we actually | 139 | * The leading and the trailing memory barriers guarantee that these |
| 132 | * make the change. If we don't change anything (so if the returned | 140 | * operations are fully ordered. |
| 133 | * prev is equal to old) then we aren't acquiring anything new and | 141 | * |
| 134 | * we don't need any memory barrier as far I can tell. | 142 | * The trailing memory barrier is placed in SMP unconditionally, in |
| 143 | * order to guarantee that dependency ordering is preserved when a | ||
| 144 | * dependency is headed by an unsuccessful operation. | ||
| 135 | */ | 145 | */ |
| 136 | 146 | ||
| 137 | static inline unsigned long | 147 | static inline unsigned long |
| @@ -139,6 +149,7 @@ ____cmpxchg(_u8, volatile char *m, unsigned char old, unsigned char new) · 1 unmodified line | |||
| 139 | { | 149 | { |
| 140 | unsigned long prev, tmp, cmp, addr64; | 150 | unsigned long prev, tmp, cmp, addr64; |
| 141 | 151 | ||
| 152 | smp_mb(); | ||
| 142 | __asm__ __volatile__( | 153 | __asm__ __volatile__( |
| 143 | " andnot %5,7,%4\n" | 154 | " andnot %5,7,%4\n" |
| 144 | " insbl %1,%5,%1\n" | 155 | " insbl %1,%5,%1\n" |
| @@ -150,13 +161,13 @@ ____cmpxchg(_u8, volatile char *m, unsigned char old, unsigned char new) · 5 unmodified lines | |||
| 150 | " or %1,%2,%2\n" | 161 | " or %1,%2,%2\n" |
| 151 | " stq_c %2,0(%4)\n" | 162 | " stq_c %2,0(%4)\n" |
| 152 | " beq %2,3f\n" | 163 | " beq %2,3f\n" |
| 153 | __ASM__MB | ||
| 154 | "2:\n" | 164 | "2:\n" |
| 155 | ".subsection 2\n" | 165 | ".subsection 2\n" |
| 156 | "3: br 1b\n" | 166 | "3: br 1b\n" |
| 157 | ".previous" | 167 | ".previous" |
| 158 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) | 168 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) |
| 159 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); | 169 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); |
| 170 | smp_mb(); | ||
| 160 | 171 | ||
| 161 | return prev; | 172 | return prev; |
| 162 | } | 173 | } |
| @@ -166,6 +177,7 @@ ____cmpxchg(_u16, volatile short *m, unsigned short old, unsigned short new) · 3 unmodified lines | |||
| 166 | { | 177 | { |
| 167 | unsigned long prev, tmp, cmp, addr64; | 178 | unsigned long prev, tmp, cmp, addr64; |
| 168 | 179 | ||
| 180 | smp_mb(); | ||
| 169 | __asm__ __volatile__( | 181 | __asm__ __volatile__( |
| 170 | " andnot %5,7,%4\n" | 182 | " andnot %5,7,%4\n" |
| 171 | " inswl %1,%5,%1\n" | 183 | " inswl %1,%5,%1\n" |
| @@ -177,13 +189,13 @@ ____cmpxchg(_u16, volatile short *m, unsigned short old, unsigned short new) · 5 unmodified lines | |||
| 177 | " or %1,%2,%2\n" | 189 | " or %1,%2,%2\n" |
| 178 | " stq_c %2,0(%4)\n" | 190 | " stq_c %2,0(%4)\n" |
| 179 | " beq %2,3f\n" | 191 | " beq %2,3f\n" |
| 180 | __ASM__MB | ||
| 181 | "2:\n" | 192 | "2:\n" |
| 182 | ".subsection 2\n" | 193 | ".subsection 2\n" |
| 183 | "3: br 1b\n" | 194 | "3: br 1b\n" |
| 184 | ".previous" | 195 | ".previous" |
| 185 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) | 196 | : "=&r" (prev), "=&r" (new), "=&r" (tmp), "=&r" (cmp), "=&r" (addr64) |
| 186 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); | 197 | : "r" ((long)m), "Ir" (old), "1" (new) : "memory"); |
| 198 | smp_mb(); | ||
| 187 | 199 | ||
| 188 | return prev; | 200 | return prev; |
| 189 | } | 201 | } |
| @@ -193,6 +205,7 @@ ____cmpxchg(_u32, volatile int *m, int old, int new) · 3 unmodified lines | |||
| 193 | { | 205 | { |
| 194 | unsigned long prev, cmp; | 206 | unsigned long prev, cmp; |
| 195 | 207 | ||
| 208 | smp_mb(); | ||
| 196 | __asm__ __volatile__( | 209 | __asm__ __volatile__( |
| 197 | "1: ldl_l %0,%5\n" | 210 | "1: ldl_l %0,%5\n" |
| 198 | " cmpeq %0,%3,%1\n" | 211 | " cmpeq %0,%3,%1\n" |
| @@ -200,13 +213,13 @@ ____cmpxchg(_u32, volatile int *m, int old, int new) · 1 unmodified line | |||
| 200 | " mov %4,%1\n" | 213 | " mov %4,%1\n" |
| 201 | " stl_c %1,%2\n" | 214 | " stl_c %1,%2\n" |
| 202 | " beq %1,3f\n" | 215 | " beq %1,3f\n" |
| 203 | __ASM__MB | ||
| 204 | "2:\n" | 216 | "2:\n" |
| 205 | ".subsection 2\n" | 217 | ".subsection 2\n" |
| 206 | "3: br 1b\n" | 218 | "3: br 1b\n" |
| 207 | ".previous" | 219 | ".previous" |
| 208 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) | 220 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) |
| 209 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); | 221 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); |
| 222 | smp_mb(); | ||
| 210 | 223 | ||
| 211 | return prev; | 224 | return prev; |
| 212 | } | 225 | } |
| @@ -216,6 +229,7 @@ ____cmpxchg(_u64, volatile long *m, unsigned long old, unsigned long new) · 3 unmodified lines | |||
| 216 | { | 229 | { |
| 217 | unsigned long prev, cmp; | 230 | unsigned long prev, cmp; |
| 218 | 231 | ||
| 232 | smp_mb(); | ||
| 219 | __asm__ __volatile__( | 233 | __asm__ __volatile__( |
| 220 | "1: ldq_l %0,%5\n" | 234 | "1: ldq_l %0,%5\n" |
| 221 | " cmpeq %0,%3,%1\n" | 235 | " cmpeq %0,%3,%1\n" |
| @@ -223,13 +237,13 @@ ____cmpxchg(_u64, volatile long *m, unsigned long old, unsigned long new) · 1 unmodified line | |||
| 223 | " mov %4,%1\n" | 237 | " mov %4,%1\n" |
| 224 | " stq_c %1,%2\n" | 238 | " stq_c %1,%2\n" |
| 225 | " beq %1,3f\n" | 239 | " beq %1,3f\n" |
| 226 | __ASM__MB | ||
| 227 | "2:\n" | 240 | "2:\n" |
| 228 | ".subsection 2\n" | 241 | ".subsection 2\n" |
| 229 | "3: br 1b\n" | 242 | "3: br 1b\n" |
| 230 | ".previous" | 243 | ".previous" |
| 231 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) | 244 | : "=&r"(prev), "=&r"(cmp), "=m"(*m) |
| 232 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); | 245 | : "r"((long) old), "r"(new), "m"(*m) : "memory"); |
| 246 | smp_mb(); | ||
| 233 | 247 | ||
| 234 | return prev; | 248 | return prev; |
| 235 | } | 249 | } |
| @@ -484,7 +484,6 @@ config ARC_CURR_IN_REG · 483 unmodified lines | |||
| 484 | 484 | ||
| 485 | config ARC_EMUL_UNALIGNED | 485 | config ARC_EMUL_UNALIGNED |
| 486 | bool "Emulate unaligned memory access (userspace only)" | 486 | bool "Emulate unaligned memory access (userspace only)" |
| 487 | default N | ||
| 488 | select SYSCTL_ARCH_UNALIGN_NO_WARN | 487 | select SYSCTL_ARCH_UNALIGN_NO_WARN |
| 489 | select SYSCTL_ARCH_UNALIGN_ALLOW | 488 | select SYSCTL_ARCH_UNALIGN_ALLOW |
| 490 | depends on ISA_ARCOMPACT | 489 | depends on ISA_ARCOMPACT |
| @@ -17,6 +17,6 @@ · 16 unmodified lines | |||
| 17 | compatible = "snps,axs101", "snps,arc-sdp"; | 17 | compatible = "snps,axs101", "snps,arc-sdp"; |
| 18 | 18 | ||
| 19 | chosen { | 19 | chosen { |
| 20 | bootargs = "earlycon=uart8250,mmio32,0xe0022000,115200n8 console=tty0 console=ttyS3,115200n8 consoleblank=0 video=1280x720@60"; | 20 | bootargs = "earlycon=uart8250,mmio32,0xe0022000,115200n8 console=tty0 console=ttyS3,115200n8 consoleblank=0 video=1280x720@60 print-fatal-signals=1"; |
| 21 | }; | 21 | }; |
| 22 | }; | 22 | }; |
| @@ -214,13 +214,13 @@ · 213 unmodified lines | |||
| 214 | }; | 214 | }; |
| 215 | 215 | ||
| 216 | eeprom@0x54{ | 216 | eeprom@0x54{ |
| 217 | compatible = "24c01"; | 217 | compatible = "atmel,24c01"; |
| 218 | reg = <0x54>; | 218 | reg = <0x54>; |
| 219 | pagesize = <0x8>; | 219 | pagesize = <0x8>; |
| 220 | }; | 220 | }; |
| 221 | 221 | ||
| 222 | eeprom@0x57{ | 222 | eeprom@0x57{ |
| 223 | compatible = "24c04"; | 223 | compatible = "atmel,24c04"; |
| 224 | reg = <0x57>; | 224 | reg = <0x57>; |
| 225 | pagesize = <0x8>; | 225 | pagesize = <0x8>; |
| 226 | }; | 226 | }; |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | |||
| 22 | }; | 22 | }; |
| 23 | 23 | ||
| 24 | chosen { | 24 | chosen { |
| 25 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=ttyS0,115200n8 debug"; | 25 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=ttyS0,115200n8 debug print-fatal-signals=1"; |
| 26 | }; | 26 | }; |
| 27 | 27 | ||
| 28 | aliases { | 28 | aliases { |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | |||
| 17 | interrupt-parent = <&core_intc>; | 17 | interrupt-parent = <&core_intc>; |
| 18 | 18 | ||
| 19 | chosen { | 19 | chosen { |
| 20 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8"; | 20 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8 print-fatal-signals=1"; |
| 21 | }; | 21 | }; |
| 22 | 22 | ||
| 23 | aliases { | 23 | aliases { |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | |||
| 24 | }; | 24 | }; |
| 25 | 25 | ||
| 26 | chosen { | 26 | chosen { |
| 27 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8"; | 27 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8 print-fatal-signals=1"; |
| 28 | }; | 28 | }; |
| 29 | 29 | ||
| 30 | aliases { | 30 | aliases { |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | |||
| 15 | interrupt-parent = <&core_intc>; | 15 | interrupt-parent = <&core_intc>; |
| 16 | 16 | ||
| 17 | chosen { | 17 | chosen { |
| 18 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8"; | 18 | bootargs = "earlycon=arc_uart,mmio32,0xc0fc1000,115200n8 console=ttyARC0,115200n8 print-fatal-signals=1"; |
| 19 | }; | 19 | }; |
| 20 | 20 | ||
| 21 | aliases { | 21 | aliases { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | |||
| 20 | /* this is for console on PGU */ | 20 | /* this is for console on PGU */ |
| 21 | /* bootargs = "console=tty0 consoleblank=0"; */ | 21 | /* bootargs = "console=tty0 consoleblank=0"; */ |
| 22 | /* this is for console on serial */ | 22 | /* this is for console on serial */ |
| 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24"; | 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24 print-fatal-signals=1"; |
| 24 | }; | 24 | }; |
| 25 | 25 | ||
| 26 | aliases { | 26 | aliases { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | |||
| 20 | /* this is for console on PGU */ | 20 | /* this is for console on PGU */ |
| 21 | /* bootargs = "console=tty0 consoleblank=0"; */ | 21 | /* bootargs = "console=tty0 consoleblank=0"; */ |
| 22 | /* this is for console on serial */ | 22 | /* this is for console on serial */ |
| 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24"; | 23 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblank=0 debug video=640x480-24 print-fatal-signals=1"; |
| 24 | }; | 24 | }; |
| 25 | 25 | ||
| 26 | aliases { | 26 | aliases { |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | |||
| 18 | 18 | ||
| 19 | chosen { | 19 | chosen { |
| 20 | /* this is for console on serial */ | 20 | /* this is for console on serial */ |
| 21 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblan=0 debug video=640x480-24"; | 21 | bootargs = "earlycon=uart8250,mmio32,0xf0000000,115200n8 console=tty0 console=ttyS0,115200n8 consoleblan=0 debug video=640x480-24 print-fatal-signals=1"; |
| 22 | }; | 22 | }; |
| 23 | 23 | ||
| 24 | aliases { | 24 | aliases { |
| @@ -23,7 +23,8 @@ void die(const char *str, struct pt_regs *regs, unsigned long address); · 22 unmodified lines | |||
| 23 | 23 | ||
| 24 | #define BUG() do { \ | 24 | #define BUG() do { \ |
| 25 | pr_warn("BUG: failure at %s:%d/%s()!\n", __FILE__, __LINE__, __func__); \ | 25 | pr_warn("BUG: failure at %s:%d/%s()!\n", __FILE__, __LINE__, __func__); \ |
| 26 | dump_stack(); \ | 26 | barrier_before_unreachable(); \ |
| 27 | __builtin_trap(); \ | ||
| 27 | } while (0) | 28 | } while (0) |
| 28 | 29 | ||
| 29 | #define HAVE_ARCH_BUG | 30 | #define HAVE_ARCH_BUG |
| @@ -184,7 +184,7 @@ · 183 unmodified lines | |||
| 184 | .macro FAKE_RET_FROM_EXCPN | 184 | .macro FAKE_RET_FROM_EXCPN |
| 185 | lr r9, [status32] | 185 | lr r9, [status32] |
| 186 | bic r9, r9, (STATUS_U_MASK|STATUS_DE_MASK|STATUS_AE_MASK) | 186 | bic r9, r9, (STATUS_U_MASK|STATUS_DE_MASK|STATUS_AE_MASK) |
| 187 | or r9, r9, (STATUS_L_MASK|STATUS_IE_MASK) | 187 | or r9, r9, STATUS_IE_MASK |
| 188 | kflag r9 | 188 | kflag r9 |
| 189 | .endm | 189 | .endm |
| 190 | 190 | ||
| @@ -22,10 +22,79 @@ static DEFINE_RAW_SPINLOCK(mcip_lock); · 21 unmodified lines | |||
| 22 | 22 | ||
| 23 | static char smp_cpuinfo_buf[128]; | 23 | static char smp_cpuinfo_buf[128]; |
| 24 | 24 | ||
| 25 | /* | ||
| 26 | * Set mask to halt GFRC if any online core in SMP cluster is halted. | ||
| 27 | * Only works for ARC HS v3.0+, on earlier versions has no effect. | ||
| 28 | */ | ||
| 29 | static void mcip_update_gfrc_halt_mask(int cpu) | ||
| 30 | { | ||
| 31 | struct bcr_generic gfrc; | ||
| 32 | unsigned long flags; | ||
| 33 | u32 gfrc_halt_mask; | ||
| 34 | |||
| 35 | READ_BCR(ARC_REG_GFRC_BUILD, gfrc); | ||
| 36 | |||
| 37 | /* | ||
| 38 | * CMD_GFRC_SET_CORE and CMD_GFRC_READ_CORE commands were added in | ||
| 39 | * GFRC 0x3 version. | ||
| 40 | */ | ||
| 41 | if (gfrc.ver < 0x3) | ||
| 42 | return; | ||
| 43 | |||
| 44 | raw_spin_lock_irqsave(&mcip_lock, flags); | ||
| 45 | |||
| 46 | __mcip_cmd(CMD_GFRC_READ_CORE, 0); | ||
| 47 | gfrc_halt_mask = read_aux_reg(ARC_REG_MCIP_READBACK); | ||
| 48 | gfrc_halt_mask |= BIT(cpu); | ||
| 49 | __mcip_cmd_data(CMD_GFRC_SET_CORE, 0, gfrc_halt_mask); | ||
| 50 | |||
| 51 | raw_spin_unlock_irqrestore(&mcip_lock, flags); | ||
| 52 | } | ||
| 53 | |||
| 54 | static void mcip_update_debug_halt_mask(int cpu) | ||
| 55 | { | ||
| 56 | u32 mcip_mask = 0; | ||
| 57 | unsigned long flags; | ||
| 58 | |||
| 59 | raw_spin_lock_irqsave(&mcip_lock, flags); | ||
| 60 | |||
| 61 | /* | ||
| 62 | * mcip_mask is same for CMD_DEBUG_SET_SELECT and CMD_DEBUG_SET_MASK | ||
| 63 | * commands. So read it once instead of reading both CMD_DEBUG_READ_MASK | ||
| 64 | * and CMD_DEBUG_READ_SELECT. | ||
| 65 | */ | ||
| 66 | __mcip_cmd(CMD_DEBUG_READ_SELECT, 0); | ||
| 67 | mcip_mask = read_aux_reg(ARC_REG_MCIP_READBACK); | ||
| 68 | |||
| 69 | mcip_mask |= BIT(cpu); | ||
| 70 | |||
| 71 | __mcip_cmd_data(CMD_DEBUG_SET_SELECT, 0, mcip_mask); | ||
| 72 | /* | ||
| 73 | * Parameter specified halt cause: | ||
| 74 | * STATUS32[H]/actionpoint/breakpoint/self-halt | ||
| 75 | * We choose all of them (0xF). | ||
| 76 | */ | ||
| 77 | __mcip_cmd_data(CMD_DEBUG_SET_MASK, 0xF, mcip_mask); | ||
| 78 | |||
| 79 | raw_spin_unlock_irqrestore(&mcip_lock, flags); | ||
| 80 | } | ||
| 81 | |||
| 25 | static void mcip_setup_per_cpu(int cpu) | 82 | static void mcip_setup_per_cpu(int cpu) |
| 26 | { | 83 | { |
| 84 | struct mcip_bcr mp; | ||
| 85 | |||
| 86 | READ_BCR(ARC_REG_MCIP_BCR, mp); | ||
| 87 | |||
| 27 | smp_ipi_irq_setup(cpu, IPI_IRQ); | 88 | smp_ipi_irq_setup(cpu, IPI_IRQ); |
| 28 | smp_ipi_irq_setup(cpu, SOFTIRQ_IRQ); | 89 | smp_ipi_irq_setup(cpu, SOFTIRQ_IRQ); |
| 90 | |||
| 91 | /* Update GFRC halt mask as new CPU came online */ | ||
| 92 | if (mp.gfrc) | ||
| 93 | mcip_update_gfrc_halt_mask(cpu); | ||
| 94 | |||
| 95 | /* Update MCIP debug mask as new CPU came online */ | ||
| 96 | if (mp.dbg) | ||
| 97 | mcip_update_debug_halt_mask(cpu); | ||
| 29 | } | 98 | } |
| 30 | 99 | ||
| 31 | static void mcip_ipi_send(int cpu) | 100 | static void mcip_ipi_send(int cpu) |
| @@ -101,11 +170,6 @@ static void mcip_probe_n_setup(void) · 69 unmodified lines | |||
| 101 | IS_AVAIL1(mp.gfrc, "GFRC")); | 170 | IS_AVAIL1(mp.gfrc, "GFRC")); |
| 102 | 171 | ||
| 103 | cpuinfo_arc700[0].extn.gfrc = mp.gfrc; | 172 | cpuinfo_arc700[0].extn.gfrc = mp.gfrc; |
| 104 | |||
| 105 | if (mp.dbg) { | ||
| 106 | __mcip_cmd_data(CMD_DEBUG_SET_SELECT, 0, 0xf); | ||
| 107 | __mcip_cmd_data(CMD_DEBUG_SET_MASK, 0xf, 0xf); | ||
| 108 | } | ||
| 109 | } | 173 | } |
| 110 | 174 | ||
| 111 | struct plat_smp_ops plat_smp_ops = { | 175 | struct plat_smp_ops plat_smp_ops = { |
| @@ -51,7 +51,7 @@ static const struct id_to_str arc_cpu_rel[] = { · 50 unmodified lines | |||
| 51 | { 0x51, "R2.0" }, | 51 | { 0x51, "R2.0" }, |
| 52 | { 0x52, "R2.1" }, | 52 | { 0x52, "R2.1" }, |
| 53 | { 0x53, "R3.0" }, | 53 | { 0x53, "R3.0" }, |
| 54 | { 0x54, "R4.0" }, | 54 | { 0x54, "R3.10a" }, |
| 55 | #endif | 55 | #endif |
| 56 | { 0x00, NULL } | 56 | { 0x00, NULL } |
| 57 | }; | 57 | }; |
| @@ -373,7 +373,7 @@ static void arc_chk_core_config(void) · 315 unmodified lines | |||
| 373 | { | 373 | { |
| 374 | struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()]; | 374 | struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()]; |
| 375 | int saved = 0, present = 0; | 375 | int saved = 0, present = 0; |
| 376 | char *opt_nm = NULL;; | 376 | char *opt_nm = NULL; |
| 377 | 377 | ||
| 378 | if (!cpu->extn.timer0) | 378 | if (!cpu->extn.timer0) |
| 379 | panic("Timer0 is not present!\n"); | 379 | panic("Timer0 is not present!\n"); |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | |||
| 24 | #include <linux/reboot.h> | 24 | #include <linux/reboot.h> |
| 25 | #include <linux/irqdomain.h> | 25 | #include <linux/irqdomain.h> |
| 26 | #include <linux/export.h> | 26 | #include <linux/export.h> |
| 27 | #include <linux/of_fdt.h> | ||
| 27 | 28 | ||
| 28 | #include <asm/processor.h> | 29 | #include <asm/processor.h> |
| 29 | #include <asm/setup.h> | 30 | #include <asm/setup.h> |
| @@ -47,6 +48,42 @@ void __init smp_prepare_boot_cpu(void) · 17 unmodified lines | |||
| 47 | { | 48 | { |
| 48 | } | 49 | } |
| 49 | 50 | ||
| 51 | static int __init arc_get_cpu_map(const char *name, struct cpumask *cpumask) | ||
| 52 | { | ||
| 53 | unsigned long dt_root = of_get_flat_dt_root(); | ||
| 54 | const char *buf; | ||
| 55 | |||
| 56 | buf = of_get_flat_dt_prop(dt_root, name, NULL); | ||
| 57 | if (!buf) | ||
| 58 | return -EINVAL; | ||
| 59 | |||
| 60 | if (cpulist_parse(buf, cpumask)) | ||
| 61 | return -EINVAL; | ||
| 62 | |||
| 63 | return 0; | ||
| 64 | } | ||
| 65 | |||
| 66 | /* | ||
| 67 | * Read from DeviceTree and setup cpu possible mask. If there is no | ||
| 68 | * "possible-cpus" property in DeviceTree pretend all [0..NR_CPUS-1] exist. | ||
| 69 | */ | ||
| 70 | static void __init arc_init_cpu_possible(void) | ||
| 71 | { | ||
| 72 | struct cpumask cpumask; | ||
| 73 | |||
| 74 | if (arc_get_cpu_map("possible-cpus", &cpumask)) { | ||
| 75 | pr_warn("Failed to get possible-cpus from dtb, pretending all %u cpus exist\n", | ||
| 76 | NR_CPUS); | ||
| 77 | |||
| 78 | cpumask_setall(&cpumask); | ||
| 79 | } | ||
| 80 | |||
| 81 | if (!cpumask_test_cpu(0, &cpumask)) | ||
| 82 | panic("Master cpu (cpu[0]) is missed in cpu possible mask!"); | ||
| 83 | |||
| 84 | init_cpu_possible(&cpumask); | ||
| 85 | } | ||
| 86 | |||
| 50 | /* | 87 | /* |
| 51 | * Called from setup_arch() before calling setup_processor() | 88 | * Called from setup_arch() before calling setup_processor() |
| 52 | * | 89 | * |
| @@ -58,10 +95,7 @@ void __init smp_prepare_boot_cpu(void) · 5 unmodified lines | |||
| 58 | */ | 95 | */ |
| 59 | void __init smp_init_cpus(void) | 96 | void __init smp_init_cpus(void) |
| 60 | { | 97 | { |
| 61 | unsigned int i; | 98 | arc_init_cpu_possible(); |
| 62 | |||
| 63 | for (i = 0; i < NR_CPUS; i++) | ||
| 64 | set_cpu_possible(i, true); | ||
| 65 | 99 | ||
| 66 | if (plat_smp_ops.init_early_smp) | 100 | if (plat_smp_ops.init_early_smp) |
| 67 | plat_smp_ops.init_early_smp(); | 101 | plat_smp_ops.init_early_smp(); |
| @@ -70,16 +104,12 @@ void __init smp_init_cpus(void) · 2 unmodified lines | |||
| 70 | /* called from init ( ) => process 1 */ | 104 | /* called from init ( ) => process 1 */ |
| 71 | void __init smp_prepare_cpus(unsigned int max_cpus) | 105 | void __init smp_prepare_cpus(unsigned int max_cpus) |
| 72 | { | 106 | { |
| 73 | int i; | ||
| 74 | |||
| 75 | /* | 107 | /* |
| 76 | * if platform didn't set the present map already, do it now | 108 | * if platform didn't set the present map already, do it now |
| 77 | * boot cpu is set to present already by init/main.c | 109 | * boot cpu is set to present already by init/main.c |
| 78 | */ | 110 | */ |
| 79 | if (num_present_cpus() <= 1) { | 111 | if (num_present_cpus() <= 1) |
| 80 | for (i = 0; i < max_cpus; i++) | 112 | init_cpu_present(cpu_possible_mask); |
| 81 | set_cpu_present(i, true); | ||
| 82 | } | ||
| 83 | } | 113 | } |
| 84 | 114 | ||
| 85 | void __init smp_cpus_done(unsigned int max_cpus) | 115 | void __init smp_cpus_done(unsigned int max_cpus) |
| @@ -366,7 +366,7 @@ static void init_unwind_hdr(struct unwind_table *table, · 365 unmodified lines | |||
| 366 | return; | 366 | return; |
| 367 | 367 | ||
| 368 | ret_err: | 368 | ret_err: |
| 369 | panic("Attention !!! Dwarf FDE parsing errors\n");; | 369 | panic("Attention !!! Dwarf FDE parsing errors\n"); |
| 370 | } | 370 | } |
| 371 | 371 | ||
| 372 | #ifdef CONFIG_MODULES | 372 | #ifdef CONFIG_MODULES |
| @@ -780,7 +780,10 @@ noinline static void slc_entire_op(const int op) · 779 unmodified lines | |||
| 780 | 780 | ||
| 781 | write_aux_reg(r, ctrl); | 781 | write_aux_reg(r, ctrl); |
| 782 | 782 | ||
| 783 | write_aux_reg(ARC_REG_SLC_INVALIDATE, 1); | 783 | if (op & OP_INV) /* Inv or flush-n-inv use same cmd reg */ |
| 784 | write_aux_reg(ARC_REG_SLC_INVALIDATE, 0x1); | ||
| 785 | else | ||
| 786 | write_aux_reg(ARC_REG_SLC_FLUSH, 0x1); | ||
| 784 | 787 | ||
| 785 | /* Make sure "busy" bit reports correct stataus, see STAR 9001165532 */ | 788 | /* Make sure "busy" bit reports correct stataus, see STAR 9001165532 */ |
| 786 | read_aux_reg(r); | 789 | read_aux_reg(r); |
| @@ -55,7 +55,7 @@ · 54 unmodified lines | |||
| 55 | <0x3ff00100 0x100>; | 55 | <0x3ff00100 0x100>; |
| 56 | }; | 56 | }; |
| 57 | 57 | ||
| 58 | smc@0x3404c000 { | 58 | smc@3404c000 { |
| 59 | compatible = "brcm,bcm11351-smc", "brcm,kona-smc"; | 59 | compatible = "brcm,bcm11351-smc", "brcm,kona-smc"; |
| 60 | reg = <0x3404c000 0x400>; /* 1 KiB in SRAM */ | 60 | reg = <0x3404c000 0x400>; /* 1 KiB in SRAM */ |
| 61 | }; | 61 | }; |
| @@ -55,7 +55,7 @@ · 54 unmodified lines | |||
| 55 | <0x3ff00100 0x100>; | 55 | <0x3ff00100 0x100>; |
| 56 | }; | 56 | }; |
| 57 | 57 | ||
| 58 | smc@0x3404e000 { | 58 | smc@3404e000 { |
| 59 | compatible = "brcm,bcm21664-smc", "brcm,kona-smc"; | 59 | compatible = "brcm,bcm21664-smc", "brcm,kona-smc"; |
| 60 | reg = <0x3404e000 0x400>; /* 1 KiB in SRAM */ | 60 | reg = <0x3404e000 0x400>; /* 1 KiB in SRAM */ |
| 61 | }; | 61 | }; |
| @@ -18,10 +18,10 @@ · 17 unmodified lines | |||
| 18 | soc { | 18 | soc { |
| 19 | ranges = <0x7e000000 0x20000000 0x02000000>; | 19 | ranges = <0x7e000000 0x20000000 0x02000000>; |
| 20 | dma-ranges = <0x40000000 0x00000000 0x20000000>; | 20 | dma-ranges = <0x40000000 0x00000000 0x20000000>; |
| 21 | }; | ||
| 21 | 22 | ||
| 22 | arm-pmu { | 23 | arm-pmu { |
| 23 | compatible = "arm,arm1176-pmu"; | 24 | compatible = "arm,arm1176-pmu"; |
| 24 | }; | ||
| 25 | }; | 25 | }; |
| 26 | }; | 26 | }; |
| 27 | 27 | ||
| @@ -9,19 +9,19 @@ · 8 unmodified lines | |||
| 9 | <0x40000000 0x40000000 0x00001000>; | 9 | <0x40000000 0x40000000 0x00001000>; |
| 10 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; | 10 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; |
| 11 | 11 | ||
| 12 | local_intc: local_intc { | 12 | local_intc: local_intc@40000000 { |
| 13 | compatible = "brcm,bcm2836-l1-intc"; | 13 | compatible = "brcm,bcm2836-l1-intc"; |
| 14 | reg = <0x40000000 0x100>; | 14 | reg = <0x40000000 0x100>; |
| 15 | interrupt-controller; | 15 | interrupt-controller; |
| 16 | #interrupt-cells = <2>; | 16 | #interrupt-cells = <2>; |
| 17 | interrupt-parent = <&local_intc>; | 17 | interrupt-parent = <&local_intc>; |
| 18 | }; | 18 | }; |
| 19 | }; | ||
| 19 | 20 | ||
| 20 | arm-pmu { | 21 | arm-pmu { |
| 21 | compatible = "arm,cortex-a7-pmu"; | 22 | compatible = "arm,cortex-a7-pmu"; |
| 22 | interrupt-parent = <&local_intc>; | 23 | interrupt-parent = <&local_intc>; |
| 23 | interrupts = <9 IRQ_TYPE_LEVEL_HIGH>; | 24 | interrupts = <9 IRQ_TYPE_LEVEL_HIGH>; |
| 24 | }; | ||
| 25 | }; | 25 | }; |
| 26 | 26 | ||
| 27 | timer { | 27 | timer { |
| @@ -8,7 +8,7 @@ · 7 unmodified lines | |||
| 8 | <0x40000000 0x40000000 0x00001000>; | 8 | <0x40000000 0x40000000 0x00001000>; |
| 9 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; | 9 | dma-ranges = <0xc0000000 0x00000000 0x3f000000>; |
| 10 | 10 | ||
| 11 | local_intc: local_intc { | 11 | local_intc: local_intc@40000000 { |
| 12 | compatible = "brcm,bcm2836-l1-intc"; | 12 | compatible = "brcm,bcm2836-l1-intc"; |
| 13 | reg = <0x40000000 0x100>; | 13 | reg = <0x40000000 0x100>; |
| 14 | interrupt-controller; | 14 | interrupt-controller; |
| @@ -465,7 +465,7 @@ · 464 unmodified lines | |||
| 465 | status = "disabled"; | 465 | status = "disabled"; |
| 466 | }; | 466 | }; |
| 467 | 467 | ||
| 468 | aux: aux@0x7e215000 { | 468 | aux: aux@7e215000 { |
| 469 | compatible = "brcm,bcm2835-aux"; | 469 | compatible = "brcm,bcm2835-aux"; |
| 470 | #clock-cells = <1>; | 470 | #clock-cells = <1>; |
| 471 | reg = <0x7e215000 0x8>; | 471 | reg = <0x7e215000 0x8>; |
| @@ -49,7 +49,7 @@ · 48 unmodified lines | |||
| 49 | 49 | ||
| 50 | memory { | 50 | memory { |
| 51 | device_type = "memory"; | 51 | device_type = "memory"; |
| 52 | reg = <0x60000000 0x80000000>; | 52 | reg = <0x60000000 0x20000000>; |
| 53 | }; | 53 | }; |
| 54 | 54 | ||
| 55 | gpio-restart { | 55 | gpio-restart { |
| @@ -269,7 +269,7 @@ · 268 unmodified lines | |||
| 269 | 269 | ||
| 270 | sata: sata@46000000 { | 270 | sata: sata@46000000 { |
| 271 | /* The ROM uses this muxmode */ | 271 | /* The ROM uses this muxmode */ |
| 272 | cortina,gemini-ata-muxmode = <3>; | 272 | cortina,gemini-ata-muxmode = <0>; |
| 273 | cortina,gemini-enable-sata-bridge; | 273 | cortina,gemini-enable-sata-bridge; |
| 274 | status = "okay"; | 274 | status = "okay"; |
| 275 | }; | 275 | }; |
| @@ -42,7 +42,7 @@ · 41 unmodified lines | |||
| 42 | 42 | ||
| 43 | /dts-v1/; | 43 | /dts-v1/; |
| 44 | 44 | ||
| 45 | #include "imx6q.dtsi" | 45 | #include "imx6dl.dtsi" |
| 46 | #include "imx6qdl-icore-rqs.dtsi" | 46 | #include "imx6qdl-icore-rqs.dtsi" |
| 47 | 47 | ||
| 48 | / { | 48 | / { |
| @@ -71,6 +71,8 @@ · 70 unmodified lines | |||
| 71 | }; | 71 | }; |
| 72 | 72 | ||
| 73 | &i2c1 { | 73 | &i2c1 { |
| 74 | pinctrl-names = "default"; | ||
| 75 | pinctrl-0 = <&i2c1_pins>; | ||
| 74 | clock-frequency = <2600000>; | 76 | clock-frequency = <2600000>; |
| 75 | 77 | ||
| 76 | twl: twl@48 { | 78 | twl: twl@48 { |
| @@ -189,7 +191,12 @@ · 112 unmodified lines | |||
| 189 | >; | 191 | >; |
| 190 | }; | 192 | }; |
| 191 | 193 | ||
| 192 | 194 | i2c1_pins: pinmux_i2c1_pins { | |
| 195 | pinctrl-single,pins = < | ||
| 196 | OMAP3_CORE1_IOPAD(0x21ba, PIN_INPUT | MUX_MODE0) /* i2c1_scl.i2c1_scl */ | ||
| 197 | OMAP3_CORE1_IOPAD(0x21bc, PIN_INPUT | MUX_MODE0) /* i2c1_sda.i2c1_sda */ | ||
| 198 | >; | ||
| 199 | }; | ||
| 193 | }; | 200 | }; |
| 194 | 201 | ||
| 195 | &omap3_pmx_wkup { | 202 | &omap3_pmx_wkup { |
| @@ -66,6 +66,8 @@ · 65 unmodified lines | |||
| 66 | }; | 66 | }; |
| 67 | 67 | ||
| 68 | &i2c1 { | 68 | &i2c1 { |
| 69 | pinctrl-names = "default"; | ||
| 70 | pinctrl-0 = <&i2c1_pins>; | ||
| 69 | clock-frequency = <2600000>; | 71 | clock-frequency = <2600000>; |
| 70 | 72 | ||
| 71 | twl: twl@48 { | 73 | twl: twl@48 { |
| @@ -136,6 +138,12 @@ · 64 unmodified lines | |||
| 136 | OMAP3_CORE1_IOPAD(0x21b8, PIN_INPUT | MUX_MODE0) /* hsusb0_data7.hsusb0_data7 */ | 138 | OMAP3_CORE1_IOPAD(0x21b8, PIN_INPUT | MUX_MODE0) /* hsusb0_data7.hsusb0_data7 */ |
| 137 | >; | 139 | >; |
| 138 | }; | 140 | }; |
| 141 | i2c1_pins: pinmux_i2c1_pins { | ||
| 142 | pinctrl-single,pins = < | ||
| 143 | OMAP3_CORE1_IOPAD(0x21ba, PIN_INPUT | MUX_MODE0) /* i2c1_scl.i2c1_scl */ | ||
| 144 | OMAP3_CORE1_IOPAD(0x21bc, PIN_INPUT | MUX_MODE0) /* i2c1_sda.i2c1_sda */ | ||
| 145 | >; | ||
| 146 | }; | ||
| 139 | }; | 147 | }; |
| 140 | 148 | ||
| 141 | &uart2 { | 149 | &uart2 { |
| @@ -47,7 +47,7 @@ · 46 unmodified lines | |||
| 47 | gpios = <&gpio3 19 GPIO_ACTIVE_LOW>; /* gpio3_83 */ | 47 | gpios = <&gpio3 19 GPIO_ACTIVE_LOW>; /* gpio3_83 */ |
| 48 | wakeup-source; | 48 | wakeup-source; |
| 49 | autorepeat; | 49 | autorepeat; |
| 50 | debounce_interval = <50>; | 50 | debounce-interval = <50>; |
| 51 | }; | 51 | }; |
| 52 | }; | 52 | }; |
| 53 | 53 | ||
| @@ -280,7 +280,7 @@ · 279 unmodified lines | |||
| 280 | max-frequency = <37500000>; | 280 | max-frequency = <37500000>; |
| 281 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, | 281 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, |
| 282 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; | 282 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; |
| 283 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 283 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 284 | fifo-depth = <0x100>; | 284 | fifo-depth = <0x100>; |
| 285 | interrupts = <GIC_SPI 15 IRQ_TYPE_LEVEL_HIGH>; | 285 | interrupts = <GIC_SPI 15 IRQ_TYPE_LEVEL_HIGH>; |
| 286 | resets = <&cru SRST_SDIO>; | 286 | resets = <&cru SRST_SDIO>; |
| @@ -298,7 +298,7 @@ · 11 unmodified lines | |||
| 298 | max-frequency = <37500000>; | 298 | max-frequency = <37500000>; |
| 299 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, | 299 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, |
| 300 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; | 300 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; |
| 301 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 301 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 302 | default-sample-phase = <158>; | 302 | default-sample-phase = <158>; |
| 303 | disable-wp; | 303 | disable-wp; |
| 304 | dmas = <&pdma 12>; | 304 | dmas = <&pdma 12>; |
| @@ -621,7 +621,7 @@ · 620 unmodified lines | |||
| 621 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; | 621 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; |
| 622 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, | 622 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, |
| 623 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; | 623 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; |
| 624 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 624 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 625 | fifo-depth = <0x100>; | 625 | fifo-depth = <0x100>; |
| 626 | pinctrl-names = "default"; | 626 | pinctrl-names = "default"; |
| 627 | pinctrl-0 = <&sdmmc_clk &sdmmc_cmd &sdmmc_bus4>; | 627 | pinctrl-0 = <&sdmmc_clk &sdmmc_cmd &sdmmc_bus4>; |
| @@ -634,7 +634,7 @@ · 6 unmodified lines | |||
| 634 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; | 634 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; |
| 635 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, | 635 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, |
| 636 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; | 636 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; |
| 637 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 637 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 638 | fifo-depth = <0x100>; | 638 | fifo-depth = <0x100>; |
| 639 | pinctrl-names = "default"; | 639 | pinctrl-names = "default"; |
| 640 | pinctrl-0 = <&sdio_clk &sdio_cmd &sdio_bus4>; | 640 | pinctrl-0 = <&sdio_clk &sdio_cmd &sdio_bus4>; |
| @@ -649,7 +649,7 @@ · 8 unmodified lines | |||
| 649 | max-frequency = <37500000>; | 649 | max-frequency = <37500000>; |
| 650 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, | 650 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, |
| 651 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; | 651 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; |
| 652 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 652 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 653 | bus-width = <8>; | 653 | bus-width = <8>; |
| 654 | default-sample-phase = <158>; | 654 | default-sample-phase = <158>; |
| 655 | fifo-depth = <0x100>; | 655 | fifo-depth = <0x100>; |
| @@ -110,26 +110,6 @@ · 109 unmodified lines | |||
| 110 | }; | 110 | }; |
| 111 | }; | 111 | }; |
| 112 | 112 | ||
| 113 | &cpu0 { | ||
| 114 | cpu0-supply = <&vdd_cpu>; | ||
| 115 | operating-points = < | ||
| 116 | /* KHz uV */ | ||
| 117 | 1800000 1400000 | ||
| 118 | 1608000 1350000 | ||
| 119 | 1512000 1300000 | ||
| 120 | 1416000 1200000 | ||
| 121 | 1200000 1100000 | ||
| 122 | 1008000 1050000 | ||
| 123 | 816000 1000000 | ||
| 124 | 696000 950000 | ||
| 125 | 600000 900000 | ||
| 126 | 408000 900000 | ||
| 127 | 312000 900000 | ||
| 128 | 216000 900000 | ||
| 129 | 126000 900000 | ||
| 130 | >; | ||
| 131 | }; | ||
| 132 | |||
| 133 | &emmc { | 113 | &emmc { |
| 134 | status = "okay"; | 114 | status = "okay"; |
| 135 | bus-width = <8>; | 115 | bus-width = <8>; |
| @@ -56,7 +56,7 @@ · 55 unmodified lines | |||
| 56 | clocks = <&topclk ZX296702_A9_PERIPHCLK>; | 56 | clocks = <&topclk ZX296702_A9_PERIPHCLK>; |
| 57 | }; | 57 | }; |
| 58 | 58 | ||
| 59 | l2cc: l2-cache-controller@0x00c00000 { | 59 | l2cc: l2-cache-controller@c00000 { |
| 60 | compatible = "arm,pl310-cache"; | 60 | compatible = "arm,pl310-cache"; |
| 61 | reg = <0x00c00000 0x1000>; | 61 | reg = <0x00c00000 0x1000>; |
| 62 | cache-unified; | 62 | cache-unified; |
| @@ -67,30 +67,30 @@ · 4 unmodified lines | |||
| 67 | arm,double-linefill-incr = <0>; | 67 | arm,double-linefill-incr = <0>; |
| 68 | }; | 68 | }; |
| 69 | 69 | ||
| 70 | pcu: pcu@0xa0008000 { | 70 | pcu: pcu@a0008000 { |
| 71 | compatible = "zte,zx296702-pcu"; | 71 | compatible = "zte,zx296702-pcu"; |
| 72 | reg = <0xa0008000 0x1000>; | 72 | reg = <0xa0008000 0x1000>; |
| 73 | }; | 73 | }; |
| 74 | 74 | ||
| 75 | topclk: topclk@0x09800000 { | 75 | topclk: topclk@9800000 { |
| 76 | compatible = "zte,zx296702-topcrm-clk"; | 76 | compatible = "zte,zx296702-topcrm-clk"; |
| 77 | reg = <0x09800000 0x1000>; | 77 | reg = <0x09800000 0x1000>; |
| 78 | #clock-cells = <1>; | 78 | #clock-cells = <1>; |
| 79 | }; | 79 | }; |
| 80 | 80 | ||
| 81 | lsp1clk: lsp1clk@0x09400000 { | 81 | lsp1clk: lsp1clk@9400000 { |
| 82 | compatible = "zte,zx296702-lsp1crpm-clk"; | 82 | compatible = "zte,zx296702-lsp1crpm-clk"; |
| 83 | reg = <0x09400000 0x1000>; | 83 | reg = <0x09400000 0x1000>; |
| 84 | #clock-cells = <1>; | 84 | #clock-cells = <1>; |
| 85 | }; | 85 | }; |
| 86 | 86 | ||
| 87 | lsp0clk: lsp0clk@0x0b000000 { | 87 | lsp0clk: lsp0clk@b000000 { |
| 88 | compatible = "zte,zx296702-lsp0crpm-clk"; | 88 | compatible = "zte,zx296702-lsp0crpm-clk"; |
| 89 | reg = <0x0b000000 0x1000>; | 89 | reg = <0x0b000000 0x1000>; |
| 90 | #clock-cells = <1>; | 90 | #clock-cells = <1>; |
| 91 | }; | 91 | }; |
| 92 | 92 | ||
| 93 | uart0: serial@0x09405000 { | 93 | uart0: serial@9405000 { |
| 94 | compatible = "zte,zx296702-uart"; | 94 | compatible = "zte,zx296702-uart"; |
| 95 | reg = <0x09405000 0x1000>; | 95 | reg = <0x09405000 0x1000>; |
| 96 | interrupts = <GIC_SPI 37 IRQ_TYPE_LEVEL_HIGH>; | 96 | interrupts = <GIC_SPI 37 IRQ_TYPE_LEVEL_HIGH>; |
| @@ -98,7 +98,7 @@ · 1 unmodified line | |||
| 98 | status = "disabled"; | 98 | status = "disabled"; |
| 99 | }; | 99 | }; |
| 100 | 100 | ||
| 101 | uart1: serial@0x09406000 { | 101 | uart1: serial@9406000 { |
| 102 | compatible = "zte,zx296702-uart"; | 102 | compatible = "zte,zx296702-uart"; |
| 103 | reg = <0x09406000 0x1000>; | 103 | reg = <0x09406000 0x1000>; |
| 104 | interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>; | 104 | interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>; |
| @@ -106,7 +106,7 @@ · 1 unmodified line | |||
| 106 | status = "disabled"; | 106 | status = "disabled"; |
| 107 | }; | 107 | }; |
| 108 | 108 | ||
| 109 | mmc0: mmc@0x09408000 { | 109 | mmc0: mmc@9408000 { |
| 110 | compatible = "snps,dw-mshc"; | 110 | compatible = "snps,dw-mshc"; |
| 111 | #address-cells = <1>; | 111 | #address-cells = <1>; |
| 112 | #size-cells = <0>; | 112 | #size-cells = <0>; |
| @@ -119,7 +119,7 @@ · 6 unmodified lines | |||
| 119 | status = "disabled"; | 119 | status = "disabled"; |
| 120 | }; | 120 | }; |
| 121 | 121 | ||
| 122 | mmc1: mmc@0x0b003000 { | 122 | mmc1: mmc@b003000 { |
| 123 | compatible = "snps,dw-mshc"; | 123 | compatible = "snps,dw-mshc"; |
| 124 | #address-cells = <1>; | 124 | #address-cells = <1>; |
| 125 | #size-cells = <0>; | 125 | #size-cells = <0>; |
| @@ -132,7 +132,7 @@ · 6 unmodified lines | |||
| 132 | status = "disabled"; | 132 | status = "disabled"; |
| 133 | }; | 133 | }; |
| 134 | 134 | ||
| 135 | sysctrl: sysctrl@0xa0007000 { | 135 | sysctrl: sysctrl@a0007000 { |
| 136 | compatible = "zte,sysctrl", "syscon"; | 136 | compatible = "zte,sysctrl", "syscon"; |
| 137 | reg = <0xa0007000 0x1000>; | 137 | reg = <0xa0007000 0x1000>; |
| 138 | }; | 138 | }; |
| @@ -319,7 +319,7 @@ CONFIG_MEDIA_CAMERA_SUPPORT=y · 318 unmodified lines | |||
| 319 | CONFIG_RC_CORE=m | 319 | CONFIG_RC_CORE=m |
| 320 | CONFIG_MEDIA_CONTROLLER=y | 320 | CONFIG_MEDIA_CONTROLLER=y |
| 321 | CONFIG_VIDEO_V4L2_SUBDEV_API=y | 321 | CONFIG_VIDEO_V4L2_SUBDEV_API=y |
| 322 | CONFIG_LIRC=m | 322 | CONFIG_LIRC=y |
| 323 | CONFIG_RC_DEVICES=y | 323 | CONFIG_RC_DEVICES=y |
| 324 | CONFIG_IR_RX51=m | 324 | CONFIG_IR_RX51=m |
| 325 | CONFIG_V4L_PLATFORM_DRIVERS=y | 325 | CONFIG_V4L_PLATFORM_DRIVERS=y |
| @@ -83,7 +83,7 @@ static void dummy_clock_access(struct timespec64 *ts) · 82 unmodified lines | |||
| 83 | } | 83 | } |
| 84 | 84 | ||
| 85 | static clock_access_fn __read_persistent_clock = dummy_clock_access; | 85 | static clock_access_fn __read_persistent_clock = dummy_clock_access; |
| 86 | static clock_access_fn __read_boot_clock = dummy_clock_access;; | 86 | static clock_access_fn __read_boot_clock = dummy_clock_access; |
| 87 | 87 | ||
| 88 | void read_persistent_clock64(struct timespec64 *ts) | 88 | void read_persistent_clock64(struct timespec64 *ts) |
| 89 | { | 89 | { |
| @@ -7,6 +7,8 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING · 6 unmodified lines | |||
| 7 | 7 | ||
| 8 | KVM=../../../../virt/kvm | 8 | KVM=../../../../virt/kvm |
| 9 | 9 | ||
| 10 | CFLAGS_ARMV7VE :=$(call cc-option, -march=armv7ve) | ||
| 11 | |||
| 10 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o | 12 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o |
| 11 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o | 13 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v3-sr.o |
| 12 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o | 14 | obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/timer-sr.o |
| @@ -15,7 +17,10 @@ obj-$(CONFIG_KVM_ARM_HOST) += tlb.o · 2 unmodified lines | |||
| 15 | obj-$(CONFIG_KVM_ARM_HOST) += cp15-sr.o | 17 | obj-$(CONFIG_KVM_ARM_HOST) += cp15-sr.o |
| 16 | obj-$(CONFIG_KVM_ARM_HOST) += vfp.o | 18 | obj-$(CONFIG_KVM_ARM_HOST) += vfp.o |
| 17 | obj-$(CONFIG_KVM_ARM_HOST) += banked-sr.o | 19 | obj-$(CONFIG_KVM_ARM_HOST) += banked-sr.o |
| 20 | CFLAGS_banked-sr.o += $(CFLAGS_ARMV7VE) | ||
| 21 | |||
| 18 | obj-$(CONFIG_KVM_ARM_HOST) += entry.o | 22 | obj-$(CONFIG_KVM_ARM_HOST) += entry.o |
| 19 | obj-$(CONFIG_KVM_ARM_HOST) += hyp-entry.o | 23 | obj-$(CONFIG_KVM_ARM_HOST) += hyp-entry.o |
| 20 | obj-$(CONFIG_KVM_ARM_HOST) += switch.o | 24 | obj-$(CONFIG_KVM_ARM_HOST) += switch.o |
| 25 | CFLAGS_switch.o += $(CFLAGS_ARMV7VE) | ||
| 21 | obj-$(CONFIG_KVM_ARM_HOST) += s2-setup.o | 26 | obj-$(CONFIG_KVM_ARM_HOST) += s2-setup.o |
| @@ -20,6 +20,10 @@ · 19 unmodified lines | |||
| 20 | 20 | ||
| 21 | #include <asm/kvm_hyp.h> | 21 | #include <asm/kvm_hyp.h> |
| 22 | 22 | ||
| 23 | /* | ||
| 24 | * gcc before 4.9 doesn't understand -march=armv7ve, so we have to | ||
| 25 | * trick the assembler. | ||
| 26 | */ | ||
| 23 | __asm__(".arch_extension virt"); | 27 | __asm__(".arch_extension virt"); |
| 24 | 28 | ||
| 25 | void __hyp_text __banked_save_state(struct kvm_cpu_context *ctxt) | 29 | void __hyp_text __banked_save_state(struct kvm_cpu_context *ctxt) |
| @@ -69,7 +69,7 @@ static void clps711x_restart(enum reboot_mode mode, const char *cmd) · 68 unmodified lines | |||
| 69 | soft_restart(0); | 69 | soft_restart(0); |
| 70 | } | 70 | } |
| 71 | 71 | ||
| 72 | static const char *clps711x_compat[] __initconst = { | 72 | static const char *const clps711x_compat[] __initconst = { |
| 73 | "cirrus,ep7209", | 73 | "cirrus,ep7209", |
| 74 | NULL | 74 | NULL |
| 75 | }; | 75 | }; |
| @@ -368,7 +368,7 @@ static struct spi_eeprom at25640a = { · 367 unmodified lines | |||
| 368 | .flags = EE_ADDR2, | 368 | .flags = EE_ADDR2, |
| 369 | }; | 369 | }; |
| 370 | 370 | ||
| 371 | static struct spi_board_info dm355_evm_spi_info[] __initconst = { | 371 | static const struct spi_board_info dm355_evm_spi_info[] __initconst = { |
| 372 | { | 372 | { |
| 373 | .modalias = "at25", | 373 | .modalias = "at25", |
| 374 | .platform_data = &at25640a, | 374 | .platform_data = &at25640a, |
| @@ -217,7 +217,7 @@ static struct spi_eeprom at25640a = { · 216 unmodified lines | |||
| 217 | .flags = EE_ADDR2, | 217 | .flags = EE_ADDR2, |
| 218 | }; | 218 | }; |
| 219 | 219 | ||
| 220 | static struct spi_board_info dm355_leopard_spi_info[] __initconst = { | 220 | static const struct spi_board_info dm355_leopard_spi_info[] __initconst = { |
| 221 | { | 221 | { |
| 222 | .modalias = "at25", | 222 | .modalias = "at25", |
| 223 | .platform_data = &at25640a, | 223 | .platform_data = &at25640a, |
| @@ -726,7 +726,7 @@ static struct spi_eeprom at25640 = { · 725 unmodified lines | |||
| 726 | .flags = EE_ADDR2, | 726 | .flags = EE_ADDR2, |
| 727 | }; | 727 | }; |
| 728 | 728 | ||
| 729 | static struct spi_board_info dm365_evm_spi_info[] __initconst = { | 729 | static const struct spi_board_info dm365_evm_spi_info[] __initconst = { |
| 730 | { | 730 | { |
| 731 | .modalias = "at25", | 731 | .modalias = "at25", |
| 732 | .platform_data = &at25640, | 732 | .platform_data = &at25640, |
| @@ -41,7 +41,7 @@ config MACH_ARMADA_375 · 40 unmodified lines | |||
| 41 | depends on ARCH_MULTI_V7 | 41 | depends on ARCH_MULTI_V7 |
| 42 | select ARMADA_370_XP_IRQ | 42 | select ARMADA_370_XP_IRQ |
| 43 | select ARM_ERRATA_720789 | 43 | select ARM_ERRATA_720789 |
| 44 | select ARM_ERRATA_753970 | 44 | select PL310_ERRATA_753970 |
| 45 | select ARM_GIC | 45 | select ARM_GIC |
| 46 | select ARMADA_375_CLK | 46 | select ARMADA_375_CLK |
| 47 | select HAVE_ARM_SCU | 47 | select HAVE_ARM_SCU |
| @@ -57,7 +57,7 @@ config MACH_ARMADA_38X · 9 unmodified lines | |||
| 57 | bool "Marvell Armada 380/385 boards" | 57 | bool "Marvell Armada 380/385 boards" |
| 58 | depends on ARCH_MULTI_V7 | 58 | depends on ARCH_MULTI_V7 |
| 59 | select ARM_ERRATA_720789 | 59 | select ARM_ERRATA_720789 |
| 60 | select ARM_ERRATA_753970 | 60 | select PL310_ERRATA_753970 |
| 61 | select ARM_GIC | 61 | select ARM_GIC |
| 62 | select ARM_GLOBAL_TIMER | 62 | select ARM_GLOBAL_TIMER |
| 63 | select CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK | 63 | select CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK |
| @@ -1011,17 +1011,17 @@ static int clk_debugfs_register_one(struct clk *c) · 1010 unmodified lines | |||
| 1011 | return -ENOMEM; | 1011 | return -ENOMEM; |
| 1012 | c->dent = d; | 1012 | c->dent = d; |
| 1013 | 1013 | ||
| 1014 | d = debugfs_create_u8("usecount", S_IRUGO, c->dent, (u8 *)&c->usecount); | 1014 | d = debugfs_create_u8("usecount", S_IRUGO, c->dent, &c->usecount); |
| 1015 | if (!d) { | 1015 | if (!d) { |
| 1016 | err = -ENOMEM; | 1016 | err = -ENOMEM; |
| 1017 | goto err_out; | 1017 | goto err_out; |
| 1018 | } | 1018 | } |
| 1019 | d = debugfs_create_u32("rate", S_IRUGO, c->dent, (u32 *)&c->rate); | 1019 | d = debugfs_create_ulong("rate", S_IRUGO, c->dent, &c->rate); |
| 1020 | if (!d) { | 1020 | if (!d) { |
| 1021 | err = -ENOMEM; | 1021 | err = -ENOMEM; |
| 1022 | goto err_out; | 1022 | goto err_out; |
| 1023 | } | 1023 | } |
| 1024 | d = debugfs_create_x32("flags", S_IRUGO, c->dent, (u32 *)&c->flags); | 1024 | d = debugfs_create_x8("flags", S_IRUGO, c->dent, &c->flags); |
| 1025 | if (!d) { | 1025 | if (!d) { |
| 1026 | err = -ENOMEM; | 1026 | err = -ENOMEM; |
| 1027 | goto err_out; | 1027 | goto err_out; |
| @@ -299,8 +299,6 @@ static void irq_save_context(void) · 298 unmodified lines | |||
| 299 | if (soc_is_dra7xx()) | 299 | if (soc_is_dra7xx()) |
| 300 | return; | 300 | return; |
| 301 | 301 | ||
| 302 | if (!sar_base) | ||
| 303 | sar_base = omap4_get_sar_ram_base(); | ||
| 304 | if (wakeupgen_ops && wakeupgen_ops->save_context) | 302 | if (wakeupgen_ops && wakeupgen_ops->save_context) |
| 305 | wakeupgen_ops->save_context(); | 303 | wakeupgen_ops->save_context(); |
| 306 | } | 304 | } |
| @@ -598,6 +596,8 @@ static int __init wakeupgen_init(struct device_node *node, · 291 unmodified lines | |||
| 598 | irq_hotplug_init(); | 596 | irq_hotplug_init(); |
| 599 | irq_pm_init(); | 597 | irq_pm_init(); |
| 600 | 598 | ||
| 599 | sar_base = omap4_get_sar_ram_base(); | ||
| 600 | |||
| 601 | return 0; | 601 | return 0; |
| 602 | } | 602 | } |
| 603 | IRQCHIP_DECLARE(ti_wakeupgen, "ti,omap4-wugen-mpu", wakeupgen_init); | 603 | IRQCHIP_DECLARE(ti_wakeupgen, "ti,omap4-wugen-mpu", wakeupgen_init); |
| @@ -977,6 +977,9 @@ static int _enable_clocks(struct omap_hwmod *oh) · 976 unmodified lines | |||
| 977 | 977 | ||
| 978 | pr_debug("omap_hwmod: %s: enabling clocks\n", oh->name); | 978 | pr_debug("omap_hwmod: %s: enabling clocks\n", oh->name); |
| 979 | 979 | ||
| 980 | if (oh->flags & HWMOD_OPT_CLKS_NEEDED) | ||
| 981 | _enable_optional_clocks(oh); | ||
| 982 | |||
| 980 | if (oh->_clk) | 983 | if (oh->_clk) |
| 981 | clk_enable(oh->_clk); | 984 | clk_enable(oh->_clk); |
| 982 | 985 | ||
| @@ -985,9 +988,6 @@ static int _enable_clocks(struct omap_hwmod *oh) · 2 unmodified lines | |||
| 985 | clk_enable(os->_clk); | 988 | clk_enable(os->_clk); |
| 986 | } | 989 | } |
| 987 | 990 | ||
| 988 | if (oh->flags & HWMOD_OPT_CLKS_NEEDED) | ||
| 989 | _enable_optional_clocks(oh); | ||
| 990 | |||
| 991 | /* The opt clocks are controlled by the device driver. */ | 991 | /* The opt clocks are controlled by the device driver. */ |
| 992 | 992 | ||
| 993 | return 0; | 993 | return 0; |
| @@ -186,7 +186,7 @@ static void omap_pm_end(void) · 185 unmodified lines | |||
| 186 | cpu_idle_poll_ctrl(false); | 186 | cpu_idle_poll_ctrl(false); |
| 187 | } | 187 | } |
| 188 | 188 | ||
| 189 | static void omap_pm_finish(void) | 189 | static void omap_pm_wake(void) |
| 190 | { | 190 | { |
| 191 | if (soc_is_omap34xx()) | 191 | if (soc_is_omap34xx()) |
| 192 | omap_prcm_irq_complete(); | 192 | omap_prcm_irq_complete(); |
| @@ -196,7 +196,7 @@ static const struct platform_suspend_ops omap_pm_ops = { · 3 unmodified lines | |||
| 196 | .begin = omap_pm_begin, | 196 | .begin = omap_pm_begin, |
| 197 | .end = omap_pm_end, | 197 | .end = omap_pm_end, |
| 198 | .enter = omap_pm_enter, | 198 | .enter = omap_pm_enter, |
| 199 | .finish = omap_pm_finish, | 199 | .wake = omap_pm_wake, |
| 200 | .valid = suspend_valid_only_mem, | 200 | .valid = suspend_valid_only_mem, |
| 201 | }; | 201 | }; |
| 202 | 202 | ||
| @@ -156,12 +156,6 @@ static struct clock_event_device clockevent_gpt = { · 155 unmodified lines | |||
| 156 | .tick_resume = omap2_gp_timer_shutdown, | 156 | .tick_resume = omap2_gp_timer_shutdown, |
| 157 | }; | 157 | }; |
| 158 | 158 | ||
| 159 | static struct property device_disabled = { | ||
| 160 | .name = "status", | ||
| 161 | .length = sizeof("disabled"), | ||
| 162 | .value = "disabled", | ||
| 163 | }; | ||
| 164 | |||
| 165 | static const struct of_device_id omap_timer_match[] __initconst = { | 159 | static const struct of_device_id omap_timer_match[] __initconst = { |
| 166 | { .compatible = "ti,omap2420-timer", }, | 160 | { .compatible = "ti,omap2420-timer", }, |
| 167 | { .compatible = "ti,omap3430-timer", }, | 161 | { .compatible = "ti,omap3430-timer", }, |
| @@ -203,8 +197,17 @@ static struct device_node * __init omap_get_timer_dt(const struct of_device_id * · 35 unmodified lines | |||
| 203 | of_get_property(np, "ti,timer-secure", NULL))) | 197 | of_get_property(np, "ti,timer-secure", NULL))) |
| 204 | continue; | 198 | continue; |
| 205 | 199 | ||
| 206 | if (!of_device_is_compatible(np, "ti,omap-counter32k")) | 200 | if (!of_device_is_compatible(np, "ti,omap-counter32k")) { |
| 207 | of_add_property(np, &device_disabled); | 201 | struct property *prop; |
| 202 | |||
| 203 | prop = kzalloc(sizeof(*prop), GFP_KERNEL); | ||
| 204 | if (!prop) | ||
| 205 | return NULL; | ||
| 206 | prop->name = "status"; | ||
| 207 | prop->value = "disabled"; | ||
| 208 | prop->length = strlen(prop->value); | ||
| 209 | of_add_property(np, prop); | ||
| 210 | } | ||
| 208 | return np; | 211 | return np; |
| 209 | } | 212 | } |
| 210 | 213 | ||
| @@ -58,7 +58,6 @@ config MACH_KUROBOX_PRO · 57 unmodified lines | |||
| 58 | 58 | ||
| 59 | config MACH_DNS323 | 59 | config MACH_DNS323 |
| 60 | bool "D-Link DNS-323" | 60 | bool "D-Link DNS-323" |
| 61 | select GENERIC_NET_UTILS | ||
| 62 | select I2C_BOARDINFO if I2C | 61 | select I2C_BOARDINFO if I2C |
| 63 | help | 62 | help |
| 64 | Say 'Y' here if you want your kernel to support the | 63 | Say 'Y' here if you want your kernel to support the |
| @@ -66,7 +65,6 @@ config MACH_DNS323 · 1 unmodified line | |||
| 66 | 65 | ||
| 67 | config MACH_TS209 | 66 | config MACH_TS209 |
| 68 | bool "QNAP TS-109/TS-209" | 67 | bool "QNAP TS-109/TS-209" |
| 69 | select GENERIC_NET_UTILS | ||
| 70 | help | 68 | help |
| 71 | Say 'Y' here if you want your kernel to support the | 69 | Say 'Y' here if you want your kernel to support the |
| 72 | QNAP TS-109/TS-209 platform. | 70 | QNAP TS-109/TS-209 platform. |
| @@ -101,7 +99,6 @@ config MACH_LINKSTATION_LS_HGL · 28 unmodified lines | |||
| 101 | 99 | ||
| 102 | config MACH_TS409 | 100 | config MACH_TS409 |
| 103 | bool "QNAP TS-409" | 101 | bool "QNAP TS-409" |
| 104 | select GENERIC_NET_UTILS | ||
| 105 | help | 102 | help |
| 106 | Say 'Y' here if you want your kernel to support the | 103 | Say 'Y' here if you want your kernel to support the |
| 107 | QNAP TS-409 platform. | 104 | QNAP TS-409 platform. |
| @@ -173,10 +173,42 @@ static struct mv643xx_eth_platform_data dns323_eth_data = { · 172 unmodified lines | |||
| 173 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), | 173 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), |
| 174 | }; | 174 | }; |
| 175 | 175 | ||
| 176 | /* dns323_parse_hex_*() taken from tsx09-common.c; should a common copy of these | ||
| 177 | * functions be kept somewhere? | ||
| 178 | */ | ||
| 179 | static int __init dns323_parse_hex_nibble(char n) | ||
| 180 | { | ||
| 181 | if (n >= '0' && n <= '9') | ||
| 182 | return n - '0'; | ||
| 183 | |||
| 184 | if (n >= 'A' && n <= 'F') | ||
| 185 | return n - 'A' + 10; | ||
| 186 | |||
| 187 | if (n >= 'a' && n <= 'f') | ||
| 188 | return n - 'a' + 10; | ||
| 189 | |||
| 190 | return -1; | ||
| 191 | } | ||
| 192 | |||
| 193 | static int __init dns323_parse_hex_byte(const char *b) | ||
| 194 | { | ||
| 195 | int hi; | ||
| 196 | int lo; | ||
| 197 | |||
| 198 | hi = dns323_parse_hex_nibble(b[0]); | ||
| 199 | lo = dns323_parse_hex_nibble(b[1]); | ||
| 200 | |||
| 201 | if (hi < 0 || lo < 0) | ||
| 202 | return -1; | ||
| 203 | |||
| 204 | return (hi << 4) | lo; | ||
| 205 | } | ||
| 206 | |||
| 176 | static int __init dns323_read_mac_addr(void) | 207 | static int __init dns323_read_mac_addr(void) |
| 177 | { | 208 | { |
| 178 | u_int8_t addr[6]; | 209 | u_int8_t addr[6]; |
| 179 | void __iomem *mac_page; | 210 | int i; |
| 211 | char *mac_page; | ||
| 180 | 212 | ||
| 181 | /* MAC address is stored as a regular ol' string in /dev/mtdblock4 | 213 | /* MAC address is stored as a regular ol' string in /dev/mtdblock4 |
| 182 | * (0x007d0000-0x00800000) starting at offset 196480 (0x2ff80). | 214 | * (0x007d0000-0x00800000) starting at offset 196480 (0x2ff80). |
| @@ -185,8 +217,23 @@ static int __init dns323_read_mac_addr(void) · 2 unmodified lines | |||
| 185 | if (!mac_page) | 217 | if (!mac_page) |
| 186 | return -ENOMEM; | 218 | return -ENOMEM; |
| 187 | 219 | ||
| 188 | if (!mac_pton((__force const char *) mac_page, addr)) | 220 | /* Sanity check the string we're looking at */ |
| 189 | goto error_fail; | 221 | for (i = 0; i < 5; i++) { |
| 222 | if (*(mac_page + (i * 3) + 2) != ':') { | ||
| 223 | goto error_fail; | ||
| 224 | } | ||
| 225 | } | ||
| 226 | |||
| 227 | for (i = 0; i < 6; i++) { | ||
| 228 | int byte; | ||
| 229 | |||
| 230 | byte = dns323_parse_hex_byte(mac_page + (i * 3)); | ||
| 231 | if (byte < 0) { | ||
| 232 | goto error_fail; | ||
| 233 | } | ||
| 234 | |||
| 235 | addr[i] = byte; | ||
| 236 | } | ||
| 190 | 237 | ||
| 191 | iounmap(mac_page); | 238 | iounmap(mac_page); |
| 192 | printk("DNS-323: Found ethernet MAC address: %pM\n", addr); | 239 | printk("DNS-323: Found ethernet MAC address: %pM\n", addr); |
| @@ -53,12 +53,53 @@ struct mv643xx_eth_platform_data qnap_tsx09_eth_data = { · 52 unmodified lines | |||
| 53 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), | 53 | .phy_addr = MV643XX_ETH_PHY_ADDR(8), |
| 54 | }; | 54 | }; |
| 55 | 55 | ||
| 56 | static int __init qnap_tsx09_parse_hex_nibble(char n) | ||
| 57 | { | ||
| 58 | if (n >= '0' && n <= '9') | ||
| 59 | return n - '0'; | ||
| 60 | |||
| 61 | if (n >= 'A' && n <= 'F') | ||
| 62 | return n - 'A' + 10; | ||
| 63 | |||
| 64 | if (n >= 'a' && n <= 'f') | ||
| 65 | return n - 'a' + 10; | ||
| 66 | |||
| 67 | return -1; | ||
| 68 | } | ||
| 69 | |||
| 70 | static int __init qnap_tsx09_parse_hex_byte(const char *b) | ||
| 71 | { | ||
| 72 | int hi; | ||
| 73 | int lo; | ||
| 74 | |||
| 75 | hi = qnap_tsx09_parse_hex_nibble(b[0]); | ||
| 76 | lo = qnap_tsx09_parse_hex_nibble(b[1]); | ||
| 77 | |||
| 78 | if (hi < 0 || lo < 0) | ||
| 79 | return -1; | ||
| 80 | |||
| 81 | return (hi << 4) | lo; | ||
| 82 | } | ||
| 83 | |||
| 56 | static int __init qnap_tsx09_check_mac_addr(const char *addr_str) | 84 | static int __init qnap_tsx09_check_mac_addr(const char *addr_str) |
| 57 | { | 85 | { |
| 58 | u_int8_t addr[6]; | 86 | u_int8_t addr[6]; |
| 87 | int i; | ||
| 59 | 88 | ||
| 60 | if (!mac_pton(addr_str, addr)) | 89 | for (i = 0; i < 6; i++) { |
| 61 | return -1; | 90 | int byte; |
| 91 | |||
| 92 | /* | ||
| 93 | * Enforce "xx:xx:xx:xx:xx:xx\n" format. | ||
| 94 | */ | ||
| 95 | if (addr_str[(i * 3) + 2] != ((i < 5) ? ':' : '\n')) | ||
| 96 | return -1; | ||
| 97 | |||
| 98 | byte = qnap_tsx09_parse_hex_byte(addr_str + (i * 3)); | ||
| 99 | if (byte < 0) | ||
| 100 | return -1; | ||
| 101 | addr[i] = byte; | ||
| 102 | } | ||
| 62 | 103 | ||
| 63 | printk(KERN_INFO "tsx09: found ethernet mac address %pM\n", addr); | 104 | printk(KERN_INFO "tsx09: found ethernet mac address %pM\n", addr); |
| 64 | 105 | ||
| @@ -77,12 +118,12 @@ void __init qnap_tsx09_find_mac_addr(u32 mem_base, u32 size) · 12 unmodified lines | |||
| 77 | unsigned long addr; | 118 | unsigned long addr; |
| 78 | 119 | ||
| 79 | for (addr = mem_base; addr < (mem_base + size); addr += 1024) { | 120 | for (addr = mem_base; addr < (mem_base + size); addr += 1024) { |
| 80 | void __iomem *nor_page; | 121 | char *nor_page; |
| 81 | int ret = 0; | 122 | int ret = 0; |
| 82 | 123 | ||
| 83 | nor_page = ioremap(addr, 1024); | 124 | nor_page = ioremap(addr, 1024); |
| 84 | if (nor_page != NULL) { | 125 | if (nor_page != NULL) { |
| 85 | ret = qnap_tsx09_check_mac_addr((__force const char *)nor_page); | 126 | ret = qnap_tsx09_check_mac_addr(nor_page); |
| 86 | iounmap(nor_page); | 127 | iounmap(nor_page); |
| 87 | } | 128 | } |
| 88 | 129 | ||
| @@ -23,7 +23,6 @@ · 22 unmodified lines | |||
| 23 | #include <linux/of.h> | 23 | #include <linux/of.h> |
| 24 | #include <linux/of_address.h> | 24 | #include <linux/of_address.h> |
| 25 | #include <linux/of_platform.h> | 25 | #include <linux/of_platform.h> |
| 26 | #include <linux/perf/arm_pmu.h> | ||
| 27 | #include <linux/regulator/machine.h> | 26 | #include <linux/regulator/machine.h> |
| 28 | 27 | ||
| 29 | #include <asm/outercache.h> | 28 | #include <asm/outercache.h> |
| @@ -112,37 +111,6 @@ static void ux500_restart(enum reboot_mode mode, const char *cmd) · 82 unmodified lines | |||
| 112 | prcmu_system_reset(0); | 111 | prcmu_system_reset(0); |
| 113 | } | 112 | } |
| 114 | 113 | ||
| 115 | /* | ||
| 116 | * The PMU IRQ lines of two cores are wired together into a single interrupt. | ||
| 117 | * Bounce the interrupt to the other core if it's not ours. | ||
| 118 | */ | ||
| 119 | static irqreturn_t db8500_pmu_handler(int irq, void *dev, irq_handler_t handler) | ||
| 120 | { | ||
| 121 | irqreturn_t ret = handler(irq, dev); | ||
| 122 | int other = !smp_processor_id(); | ||
| 123 | |||
| 124 | if (ret == IRQ_NONE && cpu_online(other)) | ||
| 125 | irq_set_affinity(irq, cpumask_of(other)); | ||
| 126 | |||
| 127 | /* | ||
| 128 | * We should be able to get away with the amount of IRQ_NONEs we give, | ||
| 129 | * while still having the spurious IRQ detection code kick in if the | ||
| 130 | * interrupt really starts hitting spuriously. | ||
| 131 | */ | ||
| 132 | return ret; | ||
| 133 | } | ||
| 134 | |||
| 135 | static struct arm_pmu_platdata db8500_pmu_platdata = { | ||
| 136 | .handle_irq = db8500_pmu_handler, | ||
| 137 | .irq_flags = IRQF_NOBALANCING | IRQF_NO_THREAD, | ||
| 138 | }; | ||
| 139 | |||
| 140 | static struct of_dev_auxdata u8500_auxdata_lookup[] __initdata = { | ||
| 141 | /* Requires call-back bindings. */ | ||
| 142 | OF_DEV_AUXDATA("arm,cortex-a9-pmu", 0, "arm-pmu", &db8500_pmu_platdata), | ||
| 143 | {}, | ||
| 144 | }; | ||
| 145 | |||
| 146 | static struct of_dev_auxdata u8540_auxdata_lookup[] __initdata = { | 114 | static struct of_dev_auxdata u8540_auxdata_lookup[] __initdata = { |
| 147 | OF_DEV_AUXDATA("stericsson,db8500-prcmu", 0x80157000, "db8500-prcmu", NULL), | 115 | OF_DEV_AUXDATA("stericsson,db8500-prcmu", 0x80157000, "db8500-prcmu", NULL), |
| 148 | {}, | 116 | {}, |
| @@ -165,9 +133,6 @@ static void __init u8500_init_machine(void) · 16 unmodified lines | |||
| 165 | if (of_machine_is_compatible("st-ericsson,u8540")) | 133 | if (of_machine_is_compatible("st-ericsson,u8540")) |
| 166 | of_platform_populate(NULL, u8500_local_bus_nodes, | 134 | of_platform_populate(NULL, u8500_local_bus_nodes, |
| 167 | u8540_auxdata_lookup, NULL); | 135 | u8540_auxdata_lookup, NULL); |
| 168 | else | ||
| 169 | of_platform_populate(NULL, u8500_local_bus_nodes, | ||
| 170 | u8500_auxdata_lookup, NULL); | ||
| 171 | } | 136 | } |
| 172 | 137 | ||
| 173 | static const char * stericsson_dt_platform_compat[] = { | 138 | static const char * stericsson_dt_platform_compat[] = { |
| @@ -472,28 +472,27 @@ void __init orion_ge11_init(struct mv643xx_eth_platform_data *eth_data, · 471 unmodified lines | |||
| 472 | /***************************************************************************** | 472 | /***************************************************************************** |
| 473 | * Ethernet switch | 473 | * Ethernet switch |
| 474 | ****************************************************************************/ | 474 | ****************************************************************************/ |
| 475 | static __initconst const char *orion_ge00_mvmdio_bus_name = "orion-mii"; | 475 | static __initdata struct mdio_board_info orion_ge00_switch_board_info = { |
| 476 | static __initdata struct mdio_board_info | 476 | .bus_id = "orion-mii", |
| 477 | orion_ge00_switch_board_info; | 477 | .modalias = "mv88e6085", |
| 478 | }; | ||
| 478 | 479 | ||
| 479 | void __init orion_ge00_switch_init(struct dsa_chip_data *d) | 480 | void __init orion_ge00_switch_init(struct dsa_chip_data *d) |
| 480 | { | 481 | { |
| 481 | struct mdio_board_info *bd; | ||
| 482 | unsigned int i; | 482 | unsigned int i; |
| 483 | 483 | ||
| 484 | if (!IS_BUILTIN(CONFIG_PHYLIB)) | 484 | if (!IS_BUILTIN(CONFIG_PHYLIB)) |
| 485 | return; | 485 | return; |
| 486 | 486 | ||
| 487 | for (i = 0; i < ARRAY_SIZE(d->port_names); i++) | 487 | for (i = 0; i < ARRAY_SIZE(d->port_names); i++) { |
| 488 | if (!strcmp(d->port_names[i], "cpu")) | 488 | if (!strcmp(d->port_names[i], "cpu")) { |
| 489 | d->netdev[i] = &orion_ge00.dev; | ||
| 489 | break; | 490 | break; |
| 491 | } | ||
| 492 | } | ||
| 490 | 493 | ||
| 491 | bd = &orion_ge00_switch_board_info; | 494 | orion_ge00_switch_board_info.mdio_addr = d->sw_addr; |
| 492 | bd->bus_id = orion_ge00_mvmdio_bus_name; | 495 | orion_ge00_switch_board_info.platform_data = d; |
| 493 | bd->mdio_addr = d->sw_addr; | ||
| 494 | d->netdev[i] = &orion_ge00.dev; | ||
| 495 | strcpy(bd->modalias, "mv88e6085"); | ||
| 496 | bd->platform_data = d; | ||
| 497 | 496 | ||
| 498 | mdiobus_register_board_info(&orion_ge00_switch_board_info, 1); | 497 | mdiobus_register_board_info(&orion_ge00_switch_board_info, 1); |
| 499 | } | 498 | } |
| @@ -165,14 +165,14 @@ · 164 unmodified lines | |||
| 165 | 165 | ||
| 166 | uart_A: serial@24000 { | 166 | uart_A: serial@24000 { |
| 167 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; | 167 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; |
| 168 | reg = <0x0 0x24000 0x0 0x14>; | 168 | reg = <0x0 0x24000 0x0 0x18>; |
| 169 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; | 169 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; |
| 170 | status = "disabled"; | 170 | status = "disabled"; |
| 171 | }; | 171 | }; |
| 172 | 172 | ||
| 173 | uart_B: serial@23000 { | 173 | uart_B: serial@23000 { |
| 174 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; | 174 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-uart"; |
| 175 | reg = <0x0 0x23000 0x0 0x14>; | 175 | reg = <0x0 0x23000 0x0 0x18>; |
| 176 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; | 176 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; |
| 177 | status = "disabled"; | 177 | status = "disabled"; |
| 178 | }; | 178 | }; |
| @@ -235,14 +235,14 @@ · 234 unmodified lines | |||
| 235 | 235 | ||
| 236 | uart_A: serial@84c0 { | 236 | uart_A: serial@84c0 { |
| 237 | compatible = "amlogic,meson-gx-uart"; | 237 | compatible = "amlogic,meson-gx-uart"; |
| 238 | reg = <0x0 0x84c0 0x0 0x14>; | 238 | reg = <0x0 0x84c0 0x0 0x18>; |
| 239 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; | 239 | interrupts = <GIC_SPI 26 IRQ_TYPE_EDGE_RISING>; |
| 240 | status = "disabled"; | 240 | status = "disabled"; |
| 241 | }; | 241 | }; |
| 242 | 242 | ||
| 243 | uart_B: serial@84dc { | 243 | uart_B: serial@84dc { |
| 244 | compatible = "amlogic,meson-gx-uart"; | 244 | compatible = "amlogic,meson-gx-uart"; |
| 245 | reg = <0x0 0x84dc 0x0 0x14>; | 245 | reg = <0x0 0x84dc 0x0 0x18>; |
| 246 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; | 246 | interrupts = <GIC_SPI 75 IRQ_TYPE_EDGE_RISING>; |
| 247 | status = "disabled"; | 247 | status = "disabled"; |
| 248 | }; | 248 | }; |
| @@ -287,7 +287,7 @@ · 38 unmodified lines | |||
| 287 | 287 | ||
| 288 | uart_C: serial@8700 { | 288 | uart_C: serial@8700 { |
| 289 | compatible = "amlogic,meson-gx-uart"; | 289 | compatible = "amlogic,meson-gx-uart"; |
| 290 | reg = <0x0 0x8700 0x0 0x14>; | 290 | reg = <0x0 0x8700 0x0 0x18>; |
| 291 | interrupts = <GIC_SPI 93 IRQ_TYPE_EDGE_RISING>; | 291 | interrupts = <GIC_SPI 93 IRQ_TYPE_EDGE_RISING>; |
| 292 | status = "disabled"; | 292 | status = "disabled"; |
| 293 | }; | 293 | }; |
| @@ -404,14 +404,14 @@ · 110 unmodified lines | |||
| 404 | 404 | ||
| 405 | uart_AO: serial@4c0 { | 405 | uart_AO: serial@4c0 { |
| 406 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; | 406 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; |
| 407 | reg = <0x0 0x004c0 0x0 0x14>; | 407 | reg = <0x0 0x004c0 0x0 0x18>; |
| 408 | interrupts = <GIC_SPI 193 IRQ_TYPE_EDGE_RISING>; | 408 | interrupts = <GIC_SPI 193 IRQ_TYPE_EDGE_RISING>; |
| 409 | status = "disabled"; | 409 | status = "disabled"; |
| 410 | }; | 410 | }; |
| 411 | 411 | ||
| 412 | uart_AO_B: serial@4e0 { | 412 | uart_AO_B: serial@4e0 { |
| 413 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; | 413 | compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; |
| 414 | reg = <0x0 0x004e0 0x0 0x14>; | 414 | reg = <0x0 0x004e0 0x0 0x18>; |
| 415 | interrupts = <GIC_SPI 197 IRQ_TYPE_EDGE_RISING>; | 415 | interrupts = <GIC_SPI 197 IRQ_TYPE_EDGE_RISING>; |
| 416 | status = "disabled"; | 416 | status = "disabled"; |
| 417 | }; | 417 | }; |
| @@ -631,6 +631,7 @@ · 630 unmodified lines | |||
| 631 | 631 | ||
| 632 | internal_phy: ethernet-phy@8 { | 632 | internal_phy: ethernet-phy@8 { |
| 633 | compatible = "ethernet-phy-id0181.4400", "ethernet-phy-ieee802.3-c22"; | 633 | compatible = "ethernet-phy-id0181.4400", "ethernet-phy-ieee802.3-c22"; |
| 634 | interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH>; | ||
| 634 | reg = <8>; | 635 | reg = <8>; |
| 635 | max-speed = <100>; | 636 | max-speed = <100>; |
| 636 | }; | 637 | }; |
| @@ -98,7 +98,7 @@ · 97 unmodified lines | |||
| 98 | clock-output-names = "clk125mhz"; | 98 | clock-output-names = "clk125mhz"; |
| 99 | }; | 99 | }; |
| 100 | 100 | ||
| 101 | pci { | 101 | pcie@30000000 { |
| 102 | compatible = "pci-host-ecam-generic"; | 102 | compatible = "pci-host-ecam-generic"; |
| 103 | device_type = "pci"; | 103 | device_type = "pci"; |
| 104 | #interrupt-cells = <1>; | 104 | #interrupt-cells = <1>; |
| @@ -118,6 +118,7 @@ · 13 unmodified lines | |||
| 118 | ranges = | 118 | ranges = |
| 119 | <0x02000000 0 0x40000000 0 0x40000000 0 0x20000000 | 119 | <0x02000000 0 0x40000000 0 0x40000000 0 0x20000000 |
| 120 | 0x43000000 0x40 0x00000000 0x40 0x00000000 0x20 0x00000000>; | 120 | 0x43000000 0x40 0x00000000 0x40 0x00000000 0x20 0x00000000>; |
| 121 | bus-range = <0 0xff>; | ||
| 121 | interrupt-map-mask = <0 0 0 7>; | 122 | interrupt-map-mask = <0 0 0 7>; |
| 122 | interrupt-map = | 123 | interrupt-map = |
| 123 | /* addr pin ic icaddr icintr */ | 124 | /* addr pin ic icaddr icintr */ |
| @@ -51,7 +51,7 @@ · 50 unmodified lines | |||
| 51 | #size-cells = <2>; | 51 | #size-cells = <2>; |
| 52 | ranges; | 52 | ranges; |
| 53 | 53 | ||
| 54 | ramoops@0x21f00000 { | 54 | ramoops@21f00000 { |
| 55 | compatible = "ramoops"; | 55 | compatible = "ramoops"; |
| 56 | reg = <0x0 0x21f00000 0x0 0x00100000>; | 56 | reg = <0x0 0x21f00000 0x0 0x00100000>; |
| 57 | record-size = <0x00020000>; | 57 | record-size = <0x00020000>; |
| @@ -341,7 +341,7 @@ · 340 unmodified lines | |||
| 341 | reg = <0 0x10005000 0 0x1000>; | 341 | reg = <0 0x10005000 0 0x1000>; |
| 342 | }; | 342 | }; |
| 343 | 343 | ||
| 344 | pio: pinctrl@0x10005000 { | 344 | pio: pinctrl@10005000 { |
| 345 | compatible = "mediatek,mt8173-pinctrl"; | 345 | compatible = "mediatek,mt8173-pinctrl"; |
| 346 | reg = <0 0x1000b000 0 0x1000>; | 346 | reg = <0 0x1000b000 0 0x1000>; |
| 347 | mediatek,pctl-regmap = <&syscfg_pctl_a>; | 347 | mediatek,pctl-regmap = <&syscfg_pctl_a>; |
| @@ -140,16 +140,16 @@ · 139 unmodified lines | |||
| 140 | }; | 140 | }; |
| 141 | 141 | ||
| 142 | agnoc@0 { | 142 | agnoc@0 { |
| 143 | qcom,pcie@00600000 { | 143 | qcom,pcie@600000 { |
| 144 | perst-gpio = <&msmgpio 35 GPIO_ACTIVE_LOW>; | 144 | perst-gpio = <&msmgpio 35 GPIO_ACTIVE_LOW>; |
| 145 | }; | 145 | }; |
| 146 | 146 | ||
| 147 | qcom,pcie@00608000 { | 147 | qcom,pcie@608000 { |
| 148 | status = "okay"; | 148 | status = "okay"; |
| 149 | perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; | 149 | perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; |
| 150 | }; | 150 | }; |
| 151 | 151 | ||
| 152 | qcom,pcie@00610000 { | 152 | qcom,pcie@610000 { |
| 153 | status = "okay"; | 153 | status = "okay"; |
| 154 | perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; | 154 | perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; |
| 155 | }; | 155 | }; |
| @@ -840,7 +840,7 @@ · 839 unmodified lines | |||
| 840 | #size-cells = <1>; | 840 | #size-cells = <1>; |
| 841 | ranges; | 841 | ranges; |
| 842 | 842 | ||
| 843 | pcie0: qcom,pcie@00600000 { | 843 | pcie0: qcom,pcie@600000 { |
| 844 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; | 844 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; |
| 845 | status = "disabled"; | 845 | status = "disabled"; |
| 846 | power-domains = <&gcc PCIE0_GDSC>; | 846 | power-domains = <&gcc PCIE0_GDSC>; |
| @@ -893,7 +893,7 @@ · 46 unmodified lines | |||
| 893 | 893 | ||
| 894 | }; | 894 | }; |
| 895 | 895 | ||
| 896 | pcie1: qcom,pcie@00608000 { | 896 | pcie1: qcom,pcie@608000 { |
| 897 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; | 897 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; |
| 898 | power-domains = <&gcc PCIE1_GDSC>; | 898 | power-domains = <&gcc PCIE1_GDSC>; |
| 899 | bus-range = <0x00 0xff>; | 899 | bus-range = <0x00 0xff>; |
| @@ -946,7 +946,7 @@ · 46 unmodified lines | |||
| 946 | "bus_slave"; | 946 | "bus_slave"; |
| 947 | }; | 947 | }; |
| 948 | 948 | ||
| 949 | pcie2: qcom,pcie@00610000 { | 949 | pcie2: qcom,pcie@610000 { |
| 950 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; | 950 | compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; |
| 951 | power-domains = <&gcc PCIE2_GDSC>; | 951 | power-domains = <&gcc PCIE2_GDSC>; |
| 952 | bus-range = <0x00 0xff>; | 952 | bus-range = <0x00 0xff>; |
| @@ -132,17 +132,16 @@ · 131 unmodified lines | |||
| 132 | assigned-clocks = <&cru SCLK_MAC2IO>, <&cru SCLK_MAC2IO_EXT>; | 132 | assigned-clocks = <&cru SCLK_MAC2IO>, <&cru SCLK_MAC2IO_EXT>; |
| 133 | assigned-clock-parents = <&gmac_clkin>, <&gmac_clkin>; | 133 | assigned-clock-parents = <&gmac_clkin>, <&gmac_clkin>; |
| 134 | clock_in_out = "input"; | 134 | clock_in_out = "input"; |
| 135 | /* shows instability at 1GBit right now */ | ||
| 136 | max-speed = <100>; | ||
| 137 | phy-supply = <&vcc_io>; | 135 | phy-supply = <&vcc_io>; |
| 138 | phy-mode = "rgmii"; | 136 | phy-mode = "rgmii"; |
| 139 | pinctrl-names = "default"; | 137 | pinctrl-names = "default"; |
| 140 | pinctrl-0 = <&rgmiim1_pins>; | 138 | pinctrl-0 = <&rgmiim1_pins>; |
| 139 | snps,force_thresh_dma_mode; | ||
| 141 | snps,reset-gpio = <&gpio1 RK_PC2 GPIO_ACTIVE_LOW>; | 140 | snps,reset-gpio = <&gpio1 RK_PC2 GPIO_ACTIVE_LOW>; |
| 142 | snps,reset-active-low; | 141 | snps,reset-active-low; |
| 143 | snps,reset-delays-us = <0 10000 50000>; | 142 | snps,reset-delays-us = <0 10000 50000>; |
| 144 | tx_delay = <0x26>; | 143 | tx_delay = <0x24>; |
| 145 | rx_delay = <0x11>; | 144 | rx_delay = <0x18>; |
| 146 | status = "okay"; | 145 | status = "okay"; |
| 147 | }; | 146 | }; |
| 148 | 147 | ||
| @@ -730,7 +730,7 @@ · 729 unmodified lines | |||
| 730 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; | 730 | interrupts = <GIC_SPI 12 IRQ_TYPE_LEVEL_HIGH>; |
| 731 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, | 731 | clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, |
| 732 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; | 732 | <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; |
| 733 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 733 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 734 | fifo-depth = <0x100>; | 734 | fifo-depth = <0x100>; |
| 735 | status = "disabled"; | 735 | status = "disabled"; |
| 736 | }; | 736 | }; |
| @@ -741,7 +741,7 @@ · 4 unmodified lines | |||
| 741 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; | 741 | interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>; |
| 742 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, | 742 | clocks = <&cru HCLK_SDIO>, <&cru SCLK_SDIO>, |
| 743 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; | 743 | <&cru SCLK_SDIO_DRV>, <&cru SCLK_SDIO_SAMPLE>; |
| 744 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 744 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 745 | fifo-depth = <0x100>; | 745 | fifo-depth = <0x100>; |
| 746 | status = "disabled"; | 746 | status = "disabled"; |
| 747 | }; | 747 | }; |
| @@ -752,7 +752,7 @@ · 4 unmodified lines | |||
| 752 | interrupts = <GIC_SPI 14 IRQ_TYPE_LEVEL_HIGH>; | 752 | interrupts = <GIC_SPI 14 IRQ_TYPE_LEVEL_HIGH>; |
| 753 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, | 753 | clocks = <&cru HCLK_EMMC>, <&cru SCLK_EMMC>, |
| 754 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; | 754 | <&cru SCLK_EMMC_DRV>, <&cru SCLK_EMMC_SAMPLE>; |
| 755 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 755 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 756 | fifo-depth = <0x100>; | 756 | fifo-depth = <0x100>; |
| 757 | status = "disabled"; | 757 | status = "disabled"; |
| 758 | }; | 758 | }; |
| @@ -257,7 +257,7 @@ · 256 unmodified lines | |||
| 257 | max-frequency = <150000000>; | 257 | max-frequency = <150000000>; |
| 258 | clocks = <&cru HCLK_SDIO0>, <&cru SCLK_SDIO0>, | 258 | clocks = <&cru HCLK_SDIO0>, <&cru SCLK_SDIO0>, |
| 259 | <&cru SCLK_SDIO0_DRV>, <&cru SCLK_SDIO0_SAMPLE>; | 259 | <&cru SCLK_SDIO0_DRV>, <&cru SCLK_SDIO0_SAMPLE>; |
| 260 | clock-names = "biu", "ciu", "ciu_drv", "ciu_sample"; | 260 | clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; |
| 261 | fifo-depth = <0x100>; | 261 | fifo-depth = <0x100>; |
| 262 | interrupts = <GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>; | 262 | interrupts = <GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>; |
| 263 | resets = <&cru SRST_SDIO0>; | 263 | resets = <&cru SRST_SDIO0>; |
| @@ -457,7 +457,7 @@ · 456 unmodified lines | |||
| 457 | assigned-clocks = <&cru SCLK_PCIEPHY_REF>; | 457 | assigned-clocks = <&cru SCLK_PCIEPHY_REF>; |
| 458 | assigned-clock-parents = <&cru SCLK_PCIEPHY_REF100M>; | 458 | assigned-clock-parents = <&cru SCLK_PCIEPHY_REF100M>; |
| 459 | assigned-clock-rates = <100000000>; | 459 | assigned-clock-rates = <100000000>; |
| 460 | ep-gpios = <&gpio3 RK_PB5 GPIO_ACTIVE_HIGH>; | 460 | ep-gpios = <&gpio2 RK_PA4 GPIO_ACTIVE_HIGH>; |
| 461 | num-lanes = <4>; | 461 | num-lanes = <4>; |
| 462 | pinctrl-names = "default"; | 462 | pinctrl-names = "default"; |
| 463 | pinctrl-0 = <&pcie_clkreqn_cpm>; | 463 | pinctrl-0 = <&pcie_clkreqn_cpm>; |
| @@ -1739,8 +1739,8 @@ · 1738 unmodified lines | |||
| 1739 | compatible = "rockchip,rk3399-edp"; | 1739 | compatible = "rockchip,rk3399-edp"; |
| 1740 | reg = <0x0 0xff970000 0x0 0x8000>; | 1740 | reg = <0x0 0xff970000 0x0 0x8000>; |
| 1741 | interrupts = <GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH 0>; | 1741 | interrupts = <GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH 0>; |
| 1742 | clocks = <&cru PCLK_EDP>, <&cru PCLK_EDP_CTRL>; | 1742 | clocks = <&cru PCLK_EDP>, <&cru PCLK_EDP_CTRL>, <&cru PCLK_VIO_GRF>; |
| 1743 | clock-names = "dp", "pclk"; | 1743 | clock-names = "dp", "pclk", "grf"; |
| 1744 | pinctrl-names = "default"; | 1744 | pinctrl-names = "default"; |
| 1745 | pinctrl-0 = <&edp_hpd>; | 1745 | pinctrl-0 = <&edp_hpd>; |
| 1746 | power-domains = <&power RK3399_PD_EDP>; | 1746 | power-domains = <&power RK3399_PD_EDP>; |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | |||
| 20 | 20 | ||
| 21 | #define MPIDR_UP_BITMASK (0x1 << 30) | 21 | #define MPIDR_UP_BITMASK (0x1 << 30) |
| 22 | #define MPIDR_MT_BITMASK (0x1 << 24) | 22 | #define MPIDR_MT_BITMASK (0x1 << 24) |
| 23 | #define MPIDR_HWID_BITMASK 0xff00ffffff | 23 | #define MPIDR_HWID_BITMASK UL(0xff00ffffff) |
| 24 | 24 | ||
| 25 | #define MPIDR_LEVEL_BITS_SHIFT 3 | 25 | #define MPIDR_LEVEL_BITS_SHIFT 3 |
| 26 | #define MPIDR_LEVEL_BITS (1 << MPIDR_LEVEL_BITS_SHIFT) | 26 | #define MPIDR_LEVEL_BITS (1 << MPIDR_LEVEL_BITS_SHIFT) |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | |||
| 22 | 22 | ||
| 23 | static inline pte_t huge_ptep_get(pte_t *ptep) | 23 | static inline pte_t huge_ptep_get(pte_t *ptep) |
| 24 | { | 24 | { |
| 25 | return *ptep; | 25 | return READ_ONCE(*ptep); |
| 26 | } | 26 | } |
| 27 | 27 | ||
| 28 | 28 | ||
| @@ -185,42 +185,42 @@ static inline pmd_t kvm_s2pmd_mkexec(pmd_t pmd) · 184 unmodified lines | |||
| 185 | return pmd; | 185 | return pmd; |
| 186 | } | 186 | } |
| 187 | 187 | ||
| 188 | static inline void kvm_set_s2pte_readonly(pte_t *pte) | 188 | static inline void kvm_set_s2pte_readonly(pte_t *ptep) |
| 189 | { | 189 | { |
| 190 | pteval_t old_pteval, pteval; | 190 | pteval_t old_pteval, pteval; |
| 191 | 191 | ||
| 192 | pteval = READ_ONCE(pte_val(*pte)); | 192 | pteval = READ_ONCE(pte_val(*ptep)); |
| 193 | do { | 193 | do { |
| 194 | old_pteval = pteval; | 194 | old_pteval = pteval; |
| 195 | pteval &= ~PTE_S2_RDWR; | 195 | pteval &= ~PTE_S2_RDWR; |
| 196 | pteval |= PTE_S2_RDONLY; | 196 | pteval |= PTE_S2_RDONLY; |
| 197 | pteval = cmpxchg_relaxed(&pte_val(*pte), old_pteval, pteval); | 197 | pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval); |
| 198 | } while (pteval != old_pteval); | 198 | } while (pteval != old_pteval); |
| 199 | } | 199 | } |
| 200 | 200 | ||
| 201 | static inline bool kvm_s2pte_readonly(pte_t *pte) | 201 | static inline bool kvm_s2pte_readonly(pte_t *ptep) |
| 202 | { | 202 | { |
| 203 | return (pte_val(*pte) & PTE_S2_RDWR) == PTE_S2_RDONLY; | 203 | return (READ_ONCE(pte_val(*ptep)) & PTE_S2_RDWR) == PTE_S2_RDONLY; |
| 204 | } | 204 | } |
| 205 | 205 | ||
| 206 | static inline bool kvm_s2pte_exec(pte_t *pte) | 206 | static inline bool kvm_s2pte_exec(pte_t *ptep) |
| 207 | { | 207 | { |
| 208 | return !(pte_val(*pte) & PTE_S2_XN); | 208 | return !(READ_ONCE(pte_val(*ptep)) & PTE_S2_XN); |
| 209 | } | 209 | } |
| 210 | 210 | ||
| 211 | static inline void kvm_set_s2pmd_readonly(pmd_t *pmd) | 211 | static inline void kvm_set_s2pmd_readonly(pmd_t *pmdp) |
| 212 | { | 212 | { |
| 213 | kvm_set_s2pte_readonly((pte_t *)pmd); | 213 | kvm_set_s2pte_readonly((pte_t *)pmdp); |
| 214 | } | 214 | } |
| 215 | 215 | ||
| 216 | static inline bool kvm_s2pmd_readonly(pmd_t *pmd) | 216 | static inline bool kvm_s2pmd_readonly(pmd_t *pmdp) |
| 217 | { | 217 | { |
| 218 | return kvm_s2pte_readonly((pte_t *)pmd); | 218 | return kvm_s2pte_readonly((pte_t *)pmdp); |
| 219 | } | 219 | } |
| 220 | 220 | ||
| 221 | static inline bool kvm_s2pmd_exec(pmd_t *pmd) | 221 | static inline bool kvm_s2pmd_exec(pmd_t *pmdp) |
| 222 | { | 222 | { |
| 223 | return !(pmd_val(*pmd) & PMD_S2_XN); | 223 | return !(READ_ONCE(pmd_val(*pmdp)) & PMD_S2_XN); |
| 224 | } | 224 | } |
| 225 | 225 | ||
| 226 | static inline bool kvm_page_empty(void *ptr) | 226 | static inline bool kvm_page_empty(void *ptr) |
| @@ -141,13 +141,13 @@ static inline void cpu_install_idmap(void) · 140 unmodified lines | |||
| 141 | * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, | 141 | * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, |
| 142 | * avoiding the possibility of conflicting TLB entries being allocated. | 142 | * avoiding the possibility of conflicting TLB entries being allocated. |
| 143 | */ | 143 | */ |
| 144 | static inline void cpu_replace_ttbr1(pgd_t *pgd) | 144 | static inline void cpu_replace_ttbr1(pgd_t *pgdp) |
| 145 | { | 145 | { |
| 146 | typedef void (ttbr_replace_func)(phys_addr_t); | 146 | typedef void (ttbr_replace_func)(phys_addr_t); |
| 147 | extern ttbr_replace_func idmap_cpu_replace_ttbr1; | 147 | extern ttbr_replace_func idmap_cpu_replace_ttbr1; |
| 148 | ttbr_replace_func *replace_phys; | 148 | ttbr_replace_func *replace_phys; |
| 149 | 149 | ||
| 150 | phys_addr_t pgd_phys = virt_to_phys(pgd); | 150 | phys_addr_t pgd_phys = virt_to_phys(pgdp); |
| 151 | 151 | ||
| 152 | replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); | 152 | replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); |
| 153 | 153 | ||
| @@ -36,23 +36,23 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) · 35 unmodified lines | |||
| 36 | return (pmd_t *)__get_free_page(PGALLOC_GFP); | 36 | return (pmd_t *)__get_free_page(PGALLOC_GFP); |
| 37 | } | 37 | } |
| 38 | 38 | ||
| 39 | static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) | 39 | static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) |
| 40 | { | 40 | { |
| 41 | BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); | 41 | BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); |
| 42 | free_page((unsigned long)pmd); | 42 | free_page((unsigned long)pmdp); |
| 43 | } | 43 | } |
| 44 | 44 | ||
| 45 | static inline void __pud_populate(pud_t *pud, phys_addr_t pmd, pudval_t prot) | 45 | static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) |
| 46 | { | 46 | { |
| 47 | set_pud(pud, __pud(__phys_to_pud_val(pmd) | prot)); | 47 | set_pud(pudp, __pud(__phys_to_pud_val(pmdp) | prot)); |
| 48 | } | 48 | } |
| 49 | 49 | ||
| 50 | static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) | 50 | static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) |
| 51 | { | 51 | { |
| 52 | __pud_populate(pud, __pa(pmd), PMD_TYPE_TABLE); | 52 | __pud_populate(pudp, __pa(pmdp), PMD_TYPE_TABLE); |
| 53 | } | 53 | } |
| 54 | #else | 54 | #else |
| 55 | static inline void __pud_populate(pud_t *pud, phys_addr_t pmd, pudval_t prot) | 55 | static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) |
| 56 | { | 56 | { |
| 57 | BUILD_BUG(); | 57 | BUILD_BUG(); |
| 58 | } | 58 | } |
| @@ -65,30 +65,30 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 6 unmodified lines | |||
| 65 | return (pud_t *)__get_free_page(PGALLOC_GFP); | 65 | return (pud_t *)__get_free_page(PGALLOC_GFP); |
| 66 | } | 66 | } |
| 67 | 67 | ||
| 68 | static inline void pud_free(struct mm_struct *mm, pud_t *pud) | 68 | static inline void pud_free(struct mm_struct *mm, pud_t *pudp) |
| 69 | { | 69 | { |
| 70 | BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); | 70 | BUG_ON((unsigned long)pudp & (PAGE_SIZE-1)); |
| 71 | free_page((unsigned long)pud); | 71 | free_page((unsigned long)pudp); |
| 72 | } | 72 | } |
| 73 | 73 | ||
| 74 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pud, pgdval_t prot) | 74 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) |
| 75 | { | 75 | { |
| 76 | set_pgd(pgdp, __pgd(__phys_to_pgd_val(pud) | prot)); | 76 | set_pgd(pgdp, __pgd(__phys_to_pgd_val(pudp) | prot)); |
| 77 | } | 77 | } |
| 78 | 78 | ||
| 79 | static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud) | 79 | static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, pud_t *pudp) |
| 80 | { | 80 | { |
| 81 | __pgd_populate(pgd, __pa(pud), PUD_TYPE_TABLE); | 81 | __pgd_populate(pgdp, __pa(pudp), PUD_TYPE_TABLE); |
| 82 | } | 82 | } |
| 83 | #else | 83 | #else |
| 84 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pud, pgdval_t prot) | 84 | static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) |
| 85 | { | 85 | { |
| 86 | BUILD_BUG(); | 86 | BUILD_BUG(); |
| 87 | } | 87 | } |
| 88 | #endif /* CONFIG_PGTABLE_LEVELS > 3 */ | 88 | #endif /* CONFIG_PGTABLE_LEVELS > 3 */ |
| 89 | 89 | ||
| 90 | extern pgd_t *pgd_alloc(struct mm_struct *mm); | 90 | extern pgd_t *pgd_alloc(struct mm_struct *mm); |
| 91 | extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); | 91 | extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); |
| 92 | 92 | ||
| 93 | static inline pte_t * | 93 | static inline pte_t * |
| 94 | pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) | 94 | pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) |
| @@ -114,10 +114,10 @@ pte_alloc_one(struct mm_struct *mm, unsigned long addr) · 19 unmodified lines | |||
| 114 | /* | 114 | /* |
| 115 | * Free a PTE table. | 115 | * Free a PTE table. |
| 116 | */ | 116 | */ |
| 117 | static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) | 117 | static inline void pte_free_kernel(struct mm_struct *mm, pte_t *ptep) |
| 118 | { | 118 | { |
| 119 | if (pte) | 119 | if (ptep) |
| 120 | free_page((unsigned long)pte); | 120 | free_page((unsigned long)ptep); |
| 121 | } | 121 | } |
| 122 | 122 | ||
| 123 | static inline void pte_free(struct mm_struct *mm, pgtable_t pte) | 123 | static inline void pte_free(struct mm_struct *mm, pgtable_t pte) |
| @@ -126,10 +126,10 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pte) · 2 unmodified lines | |||
| 126 | __free_page(pte); | 126 | __free_page(pte); |
| 127 | } | 127 | } |
| 128 | 128 | ||
| 129 | static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t pte, | 129 | static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t ptep, |
| 130 | pmdval_t prot) | 130 | pmdval_t prot) |
| 131 | { | 131 | { |
| 132 | set_pmd(pmdp, __pmd(__phys_to_pmd_val(pte) | prot)); | 132 | set_pmd(pmdp, __pmd(__phys_to_pmd_val(ptep) | prot)); |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | /* | 135 | /* |
| @@ -218,7 +218,7 @@ static inline pmd_t pmd_mkcont(pmd_t pmd) · 217 unmodified lines | |||
| 218 | 218 | ||
| 219 | static inline void set_pte(pte_t *ptep, pte_t pte) | 219 | static inline void set_pte(pte_t *ptep, pte_t pte) |
| 220 | { | 220 | { |
| 221 | *ptep = pte; | 221 | WRITE_ONCE(*ptep, pte); |
| 222 | 222 | ||
| 223 | /* | 223 | /* |
| 224 | * Only if the new pte is valid and kernel, otherwise TLB maintenance | 224 | * Only if the new pte is valid and kernel, otherwise TLB maintenance |
| @@ -250,6 +250,8 @@ extern void __sync_icache_dcache(pte_t pteval, unsigned long addr); · 25 unmodified lines | |||
| 250 | static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, | 250 | static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, |
| 251 | pte_t *ptep, pte_t pte) | 251 | pte_t *ptep, pte_t pte) |
| 252 | { | 252 | { |
| 253 | pte_t old_pte; | ||
| 254 | |||
| 253 | if (pte_present(pte) && pte_user_exec(pte) && !pte_special(pte)) | 255 | if (pte_present(pte) && pte_user_exec(pte) && !pte_special(pte)) |
| 254 | __sync_icache_dcache(pte, addr); | 256 | __sync_icache_dcache(pte, addr); |
| 255 | 257 | ||
| @@ -258,14 +260,15 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines | |||
| 258 | * hardware updates of the pte (ptep_set_access_flags safely changes | 260 | * hardware updates of the pte (ptep_set_access_flags safely changes |
| 259 | * valid ptes without going through an invalid entry). | 261 | * valid ptes without going through an invalid entry). |
| 260 | */ | 262 | */ |
| 261 | if (IS_ENABLED(CONFIG_DEBUG_VM) && pte_valid(*ptep) && pte_valid(pte) && | 263 | old_pte = READ_ONCE(*ptep); |
| 264 | if (IS_ENABLED(CONFIG_DEBUG_VM) && pte_valid(old_pte) && pte_valid(pte) && | ||
| 262 | (mm == current->active_mm || atomic_read(&mm->mm_users) > 1)) { | 265 | (mm == current->active_mm || atomic_read(&mm->mm_users) > 1)) { |
| 263 | VM_WARN_ONCE(!pte_young(pte), | 266 | VM_WARN_ONCE(!pte_young(pte), |
| 264 | "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", | 267 | "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", |
| 265 | __func__, pte_val(*ptep), pte_val(pte)); | 268 | __func__, pte_val(old_pte), pte_val(pte)); |
| 266 | VM_WARN_ONCE(pte_write(*ptep) && !pte_dirty(pte), | 269 | VM_WARN_ONCE(pte_write(old_pte) && !pte_dirty(pte), |
| 267 | "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", | 270 | "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", |
| 268 | __func__, pte_val(*ptep), pte_val(pte)); | 271 | __func__, pte_val(old_pte), pte_val(pte)); |
| 269 | } | 272 | } |
| 270 | 273 | ||
| 271 | set_pte(ptep, pte); | 274 | set_pte(ptep, pte); |
| @@ -431,7 +434,7 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, · 159 unmodified lines | |||
| 431 | 434 | ||
| 432 | static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) | 435 | static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) |
| 433 | { | 436 | { |
| 434 | *pmdp = pmd; | 437 | WRITE_ONCE(*pmdp, pmd); |
| 435 | dsb(ishst); | 438 | dsb(ishst); |
| 436 | isb(); | 439 | isb(); |
| 437 | } | 440 | } |
| @@ -482,7 +485,7 @@ static inline phys_addr_t pmd_page_paddr(pmd_t pmd) · 44 unmodified lines | |||
| 482 | 485 | ||
| 483 | static inline void set_pud(pud_t *pudp, pud_t pud) | 486 | static inline void set_pud(pud_t *pudp, pud_t pud) |
| 484 | { | 487 | { |
| 485 | *pudp = pud; | 488 | WRITE_ONCE(*pudp, pud); |
| 486 | dsb(ishst); | 489 | dsb(ishst); |
| 487 | isb(); | 490 | isb(); |
| 488 | } | 491 | } |
| @@ -500,7 +503,7 @@ static inline phys_addr_t pud_page_paddr(pud_t pud) · 11 unmodified lines | |||
| 500 | /* Find an entry in the second-level page table. */ | 503 | /* Find an entry in the second-level page table. */ |
| 501 | #define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1)) | 504 | #define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1)) |
| 502 | 505 | ||
| 503 | #define pmd_offset_phys(dir, addr) (pud_page_paddr(*(dir)) + pmd_index(addr) * sizeof(pmd_t)) | 506 | #define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t)) |
| 504 | #define pmd_offset(dir, addr) ((pmd_t *)__va(pmd_offset_phys((dir), (addr)))) | 507 | #define pmd_offset(dir, addr) ((pmd_t *)__va(pmd_offset_phys((dir), (addr)))) |
| 505 | 508 | ||
| 506 | #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) | 509 | #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) |
| @@ -535,7 +538,7 @@ static inline phys_addr_t pud_page_paddr(pud_t pud) · 28 unmodified lines | |||
| 535 | 538 | ||
| 536 | static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) | 539 | static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) |
| 537 | { | 540 | { |
| 538 | *pgdp = pgd; | 541 | WRITE_ONCE(*pgdp, pgd); |
| 539 | dsb(ishst); | 542 | dsb(ishst); |
| 540 | } | 543 | } |
| 541 | 544 | ||
| @@ -552,7 +555,7 @@ static inline phys_addr_t pgd_page_paddr(pgd_t pgd) · 10 unmodified lines | |||
| 552 | /* Find an entry in the frst-level page table. */ | 555 | /* Find an entry in the frst-level page table. */ |
| 553 | #define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) | 556 | #define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) |
| 554 | 557 | ||
| 555 | #define pud_offset_phys(dir, addr) (pgd_page_paddr(*(dir)) + pud_index(addr) * sizeof(pud_t)) | 558 | #define pud_offset_phys(dir, addr) (pgd_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t)) |
| 556 | #define pud_offset(dir, addr) ((pud_t *)__va(pud_offset_phys((dir), (addr)))) | 559 | #define pud_offset(dir, addr) ((pud_t *)__va(pud_offset_phys((dir), (addr)))) |
| 557 | 560 | ||
| 558 | #define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) | 561 | #define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) |
| @@ -28,7 +28,7 @@ struct stackframe { · 27 unmodified lines | |||
| 28 | unsigned long fp; | 28 | unsigned long fp; |
| 29 | unsigned long pc; | 29 | unsigned long pc; |
| 30 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER | 30 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER |
| 31 | unsigned int graph; | 31 | int graph; |
| 32 | #endif | 32 | #endif |
| 33 | }; | 33 | }; |
| 34 | 34 | ||
| @@ -72,15 +72,15 @@ static inline void set_fs(mm_segment_t fs) · 71 unmodified lines | |||
| 72 | * This is equivalent to the following test: | 72 | * This is equivalent to the following test: |
| 73 | * (u65)addr + (u65)size <= (u65)current->addr_limit + 1 | 73 | * (u65)addr + (u65)size <= (u65)current->addr_limit + 1 |
| 74 | */ | 74 | */ |
| 75 | static inline unsigned long __range_ok(unsigned long addr, unsigned long size) | 75 | static inline unsigned long __range_ok(const void __user *addr, unsigned long size) |
| 76 | { | 76 | { |
| 77 | unsigned long limit = current_thread_info()->addr_limit; | 77 | unsigned long ret, limit = current_thread_info()->addr_limit; |
| 78 | 78 | ||
| 79 | __chk_user_ptr(addr); | 79 | __chk_user_ptr(addr); |
| 80 | asm volatile( | 80 | asm volatile( |
| 81 | // A + B <= C + 1 for all A,B,C, in four easy steps: | 81 | // A + B <= C + 1 for all A,B,C, in four easy steps: |
| 82 | // 1: X = A + B; X' = X % 2^64 | 82 | // 1: X = A + B; X' = X % 2^64 |
| 83 | " adds %0, %0, %2\n" | 83 | " adds %0, %3, %2\n" |
| 84 | // 2: Set C = 0 if X > 2^64, to guarantee X' > C in step 4 | 84 | // 2: Set C = 0 if X > 2^64, to guarantee X' > C in step 4 |
| 85 | " csel %1, xzr, %1, hi\n" | 85 | " csel %1, xzr, %1, hi\n" |
| 86 | // 3: Set X' = ~0 if X >= 2^64. For X == 2^64, this decrements X' | 86 | // 3: Set X' = ~0 if X >= 2^64. For X == 2^64, this decrements X' |
| @@ -92,9 +92,9 @@ static inline unsigned long __range_ok(unsigned long addr, unsigned long size) · 5 unmodified lines | |||
| 92 | // testing X' - C == 0, subject to the previous adjustments. | 92 | // testing X' - C == 0, subject to the previous adjustments. |
| 93 | " sbcs xzr, %0, %1\n" | 93 | " sbcs xzr, %0, %1\n" |
| 94 | " cset %0, ls\n" | 94 | " cset %0, ls\n" |
| 95 | : "+r" (addr), "+r" (limit) : "Ir" (size) : "cc"); | 95 | : "=&r" (ret), "+r" (limit) : "Ir" (size), "0" (addr) : "cc"); |
| 96 | 96 | ||
| 97 | return addr; | 97 | return ret; |
| 98 | } | 98 | } |
| 99 | 99 | ||
| 100 | /* | 100 | /* |
| @@ -104,7 +104,7 @@ static inline unsigned long __range_ok(unsigned long addr, unsigned long size) · 3 unmodified lines | |||
| 104 | */ | 104 | */ |
| 105 | #define untagged_addr(addr) sign_extend64(addr, 55) | 105 | #define untagged_addr(addr) sign_extend64(addr, 55) |
| 106 | 106 | ||
| 107 | #define access_ok(type, addr, size) __range_ok((unsigned long)(addr), size) | 107 | #define access_ok(type, addr, size) __range_ok(addr, size) |
| 108 | #define user_addr_max get_fs | 108 | #define user_addr_max get_fs |
| 109 | 109 | ||
| 110 | #define _ASM_EXTABLE(from, to) \ | 110 | #define _ASM_EXTABLE(from, to) \ |
| @@ -370,6 +370,7 @@ static unsigned int __kprobes aarch32_check_condition(u32 opcode, u32 psr) · 369 unmodified lines | |||
| 370 | static int swp_handler(struct pt_regs *regs, u32 instr) | 370 | static int swp_handler(struct pt_regs *regs, u32 instr) |
| 371 | { | 371 | { |
| 372 | u32 destreg, data, type, address = 0; | 372 | u32 destreg, data, type, address = 0; |
| 373 | const void __user *user_ptr; | ||
| 373 | int rn, rt2, res = 0; | 374 | int rn, rt2, res = 0; |
| 374 | 375 | ||
| 375 | perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS, 1, regs, regs->pc); | 376 | perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS, 1, regs, regs->pc); |
| @@ -401,7 +402,8 @@ static int swp_handler(struct pt_regs *regs, u32 instr) · 25 unmodified lines | |||
| 401 | aarch32_insn_extract_reg_num(instr, A32_RT2_OFFSET), data); | 402 | aarch32_insn_extract_reg_num(instr, A32_RT2_OFFSET), data); |
| 402 | 403 | ||
| 403 | /* Check access in reasonable access range for both SWP and SWPB */ | 404 | /* Check access in reasonable access range for both SWP and SWPB */ |
| 404 | if (!access_ok(VERIFY_WRITE, (address & ~3), 4)) { | 405 | user_ptr = (const void __user *)(unsigned long)(address & ~3); |
| 406 | if (!access_ok(VERIFY_WRITE, user_ptr, 4)) { | ||
| 405 | pr_debug("SWP{B} emulation: access to 0x%08x not allowed!\n", | 407 | pr_debug("SWP{B} emulation: access to 0x%08x not allowed!\n", |
| 406 | address); | 408 | address); |
| 407 | goto fault; | 409 | goto fault; |
| @@ -178,7 +178,7 @@ static int enable_smccc_arch_workaround_1(void *data) · 177 unmodified lines | |||
| 178 | case PSCI_CONDUIT_HVC: | 178 | case PSCI_CONDUIT_HVC: |
| 179 | arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, | 179 | arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, |
| 180 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); | 180 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); |
| 181 | if (res.a0) | 181 | if ((int)res.a0 < 0) |
| 182 | return 0; | 182 | return 0; |
| 183 | cb = call_hvc_arch_workaround_1; | 183 | cb = call_hvc_arch_workaround_1; |
| 184 | smccc_start = __smccc_workaround_1_hvc_start; | 184 | smccc_start = __smccc_workaround_1_hvc_start; |
| @@ -188,7 +188,7 @@ static int enable_smccc_arch_workaround_1(void *data) · 3 unmodified lines | |||
| 188 | case PSCI_CONDUIT_SMC: | 188 | case PSCI_CONDUIT_SMC: |
| 189 | arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, | 189 | arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, |
| 190 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); | 190 | ARM_SMCCC_ARCH_WORKAROUND_1, &res); |
| 191 | if (res.a0) | 191 | if ((int)res.a0 < 0) |
| 192 | return 0; | 192 | return 0; |
| 193 | cb = call_smc_arch_workaround_1; | 193 | cb = call_smc_arch_workaround_1; |
| 194 | smccc_start = __smccc_workaround_1_smc_start; | 194 | smccc_start = __smccc_workaround_1_smc_start; |
| @@ -406,6 +406,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = { · 211 unmodified lines | |||
| 406 | .capability = ARM64_HARDEN_BP_POST_GUEST_EXIT, | 406 | .capability = ARM64_HARDEN_BP_POST_GUEST_EXIT, |
| 407 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR_V1), | 407 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR_V1), |
| 408 | }, | 408 | }, |
| 409 | { | ||
| 410 | .capability = ARM64_HARDEN_BRANCH_PREDICTOR, | ||
| 411 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR), | ||
| 412 | .enable = qcom_enable_link_stack_sanitization, | ||
| 413 | }, | ||
| 414 | { | ||
| 415 | .capability = ARM64_HARDEN_BP_POST_GUEST_EXIT, | ||
| 416 | MIDR_ALL_VERSIONS(MIDR_QCOM_FALKOR), | ||
| 417 | }, | ||
| 409 | { | 418 | { |
| 410 | .capability = ARM64_HARDEN_BRANCH_PREDICTOR, | 419 | .capability = ARM64_HARDEN_BRANCH_PREDICTOR, |
| 411 | MIDR_ALL_VERSIONS(MIDR_BRCM_VULCAN), | 420 | MIDR_ALL_VERSIONS(MIDR_BRCM_VULCAN), |
| @@ -199,9 +199,11 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr2[] = { · 198 unmodified lines | |||
| 199 | }; | 199 | }; |
| 200 | 200 | ||
| 201 | static const struct arm64_ftr_bits ftr_ctr[] = { | 201 | static const struct arm64_ftr_bits ftr_ctr[] = { |
| 202 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RAO */ | 202 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RES1 */ |
| 203 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 29, 1, 1), /* DIC */ | ||
| 204 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 28, 1, 1), /* IDC */ | ||
| 203 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */ | 205 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */ |
| 204 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 20, 4, 0), /* ERG */ | 206 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_HIGHER_SAFE, 20, 4, 0), /* ERG */ |
| 205 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */ | 207 | ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */ |
| 206 | /* | 208 | /* |
| 207 | * Linux can handle differing I-cache policies. Userspace JITs will | 209 | * Linux can handle differing I-cache policies. Userspace JITs will |
| @@ -90,7 +90,7 @@ static int __init set_permissions(pte_t *ptep, pgtable_t token, · 89 unmodified lines | |||
| 90 | unsigned long addr, void *data) | 90 | unsigned long addr, void *data) |
| 91 | { | 91 | { |
| 92 | efi_memory_desc_t *md = data; | 92 | efi_memory_desc_t *md = data; |
| 93 | pte_t pte = *ptep; | 93 | pte_t pte = READ_ONCE(*ptep); |
| 94 | 94 | ||
| 95 | if (md->attribute & EFI_MEMORY_RO) | 95 | if (md->attribute & EFI_MEMORY_RO) |
| 96 | pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); | 96 | pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); |
| @@ -202,10 +202,10 @@ static int create_safe_exec_page(void *src_start, size_t length, · 201 unmodified lines | |||
| 202 | gfp_t mask) | 202 | gfp_t mask) |
| 203 | { | 203 | { |
| 204 | int rc = 0; | 204 | int rc = 0; |
| 205 | pgd_t *pgd; | 205 | pgd_t *pgdp; |
| 206 | pud_t *pud; | 206 | pud_t *pudp; |
| 207 | pmd_t *pmd; | 207 | pmd_t *pmdp; |
| 208 | pte_t *pte; | 208 | pte_t *ptep; |
| 209 | unsigned long dst = (unsigned long)allocator(mask); | 209 | unsigned long dst = (unsigned long)allocator(mask); |
| 210 | 210 | ||
| 211 | if (!dst) { | 211 | if (!dst) { |
| @@ -216,38 +216,38 @@ static int create_safe_exec_page(void *src_start, size_t length, · 4 unmodified lines | |||
| 216 | memcpy((void *)dst, src_start, length); | 216 | memcpy((void *)dst, src_start, length); |
| 217 | flush_icache_range(dst, dst + length); | 217 | flush_icache_range(dst, dst + length); |
| 218 | 218 | ||
| 219 | pgd = pgd_offset_raw(allocator(mask), dst_addr); | 219 | pgdp = pgd_offset_raw(allocator(mask), dst_addr); |
| 220 | if (pgd_none(*pgd)) { | 220 | if (pgd_none(READ_ONCE(*pgdp))) { |
| 221 | pud = allocator(mask); | 221 | pudp = allocator(mask); |
| 222 | if (!pud) { | 222 | if (!pudp) { |
| 223 | rc = -ENOMEM; | 223 | rc = -ENOMEM; |
| 224 | goto out; | 224 | goto out; |
| 225 | } | 225 | } |
| 226 | pgd_populate(&init_mm, pgd, pud); | 226 | pgd_populate(&init_mm, pgdp, pudp); |
| 227 | } | 227 | } |
| 228 | 228 | ||
| 229 | pud = pud_offset(pgd, dst_addr); | 229 | pudp = pud_offset(pgdp, dst_addr); |
| 230 | if (pud_none(*pud)) { | 230 | if (pud_none(READ_ONCE(*pudp))) { |
| 231 | pmd = allocator(mask); | 231 | pmdp = allocator(mask); |
| 232 | if (!pmd) { | 232 | if (!pmdp) { |
| 233 | rc = -ENOMEM; | 233 | rc = -ENOMEM; |
| 234 | goto out; | 234 | goto out; |
| 235 | } | 235 | } |
| 236 | pud_populate(&init_mm, pud, pmd); | 236 | pud_populate(&init_mm, pudp, pmdp); |
| 237 | } | 237 | } |
| 238 | 238 | ||
| 239 | pmd = pmd_offset(pud, dst_addr); | 239 | pmdp = pmd_offset(pudp, dst_addr); |
| 240 | if (pmd_none(*pmd)) { | 240 | if (pmd_none(READ_ONCE(*pmdp))) { |
| 241 | pte = allocator(mask); | 241 | ptep = allocator(mask); |
| 242 | if (!pte) { | 242 | if (!ptep) { |
| 243 | rc = -ENOMEM; | 243 | rc = -ENOMEM; |
| 244 | goto out; | 244 | goto out; |
| 245 | } | 245 | } |
| 246 | pmd_populate_kernel(&init_mm, pmd, pte); | 246 | pmd_populate_kernel(&init_mm, pmdp, ptep); |
| 247 | } | 247 | } |
| 248 | 248 | ||
| 249 | pte = pte_offset_kernel(pmd, dst_addr); | 249 | ptep = pte_offset_kernel(pmdp, dst_addr); |
| 250 | set_pte(pte, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC)); | 250 | set_pte(ptep, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC)); |
| 251 | 251 | ||
| 252 | /* | 252 | /* |
| 253 | * Load our new page tables. A strict BBM approach requires that we | 253 | * Load our new page tables. A strict BBM approach requires that we |
| @@ -263,7 +263,7 @@ static int create_safe_exec_page(void *src_start, size_t length, · 9 unmodified lines | |||
| 263 | */ | 263 | */ |
| 264 | cpu_set_reserved_ttbr0(); | 264 | cpu_set_reserved_ttbr0(); |
| 265 | local_flush_tlb_all(); | 265 | local_flush_tlb_all(); |
| 266 | write_sysreg(phys_to_ttbr(virt_to_phys(pgd)), ttbr0_el1); | 266 | write_sysreg(phys_to_ttbr(virt_to_phys(pgdp)), ttbr0_el1); |
| 267 | isb(); | 267 | isb(); |
| 268 | 268 | ||
| 269 | *phys_dst_addr = virt_to_phys((void *)dst); | 269 | *phys_dst_addr = virt_to_phys((void *)dst); |
| @@ -320,9 +320,9 @@ int swsusp_arch_suspend(void) · 50 unmodified lines | |||
| 320 | return ret; | 320 | return ret; |
| 321 | } | 321 | } |
| 322 | 322 | ||
| 323 | static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr) | 323 | static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) |
| 324 | { | 324 | { |
| 325 | pte_t pte = *src_pte; | 325 | pte_t pte = READ_ONCE(*src_ptep); |
| 326 | 326 | ||
| 327 | if (pte_valid(pte)) { | 327 | if (pte_valid(pte)) { |
| 328 | /* | 328 | /* |
| @@ -330,7 +330,7 @@ static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr) · 1 unmodified line | |||
| 330 | * read only (code, rodata). Clear the RDONLY bit from | 330 | * read only (code, rodata). Clear the RDONLY bit from |
| 331 | * the temporary mappings we use during restore. | 331 | * the temporary mappings we use during restore. |
| 332 | */ | 332 | */ |
| 333 | set_pte(dst_pte, pte_mkwrite(pte)); | 333 | set_pte(dst_ptep, pte_mkwrite(pte)); |
| 334 | } else if (debug_pagealloc_enabled() && !pte_none(pte)) { | 334 | } else if (debug_pagealloc_enabled() && !pte_none(pte)) { |
| 335 | /* | 335 | /* |
| 336 | * debug_pagealloc will removed the PTE_VALID bit if | 336 | * debug_pagealloc will removed the PTE_VALID bit if |
| @@ -343,112 +343,116 @@ static void _copy_pte(pte_t *dst_pte, pte_t *src_pte, unsigned long addr) · 6 unmodified lines | |||
| 343 | */ | 343 | */ |
| 344 | BUG_ON(!pfn_valid(pte_pfn(pte))); | 344 | BUG_ON(!pfn_valid(pte_pfn(pte))); |
| 345 | 345 | ||
| 346 | set_pte(dst_pte, pte_mkpresent(pte_mkwrite(pte))); | 346 | set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte))); |
| 347 | } | 347 | } |
| 348 | } | 348 | } |
| 349 | 349 | ||
| 350 | static int copy_pte(pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long start, | 350 | static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start, |
| 351 | unsigned long end) | 351 | unsigned long end) |
| 352 | { | 352 | { |
| 353 | pte_t *src_pte; | 353 | pte_t *src_ptep; |
| 354 | pte_t *dst_pte; | 354 | pte_t *dst_ptep; |
| 355 | unsigned long addr = start; | 355 | unsigned long addr = start; |
| 356 | 356 | ||
| 357 | dst_pte = (pte_t *)get_safe_page(GFP_ATOMIC); | 357 | dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); |
| 358 | if (!dst_pte) | 358 | if (!dst_ptep) |
| 359 | return -ENOMEM; | 359 | return -ENOMEM; |
| 360 | pmd_populate_kernel(&init_mm, dst_pmd, dst_pte); | 360 | pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep); |
| 361 | dst_pte = pte_offset_kernel(dst_pmd, start); | 361 | dst_ptep = pte_offset_kernel(dst_pmdp, start); |
| 362 | 362 | ||
| 363 | src_pte = pte_offset_kernel(src_pmd, start); | 363 | src_ptep = pte_offset_kernel(src_pmdp, start); |
| 364 | do { | 364 | do { |
| 365 | _copy_pte(dst_pte, src_pte, addr); | 365 | _copy_pte(dst_ptep, src_ptep, addr); |
| 366 | } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); | 366 | } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end); |
| 367 | 367 | ||
| 368 | return 0; | 368 | return 0; |
| 369 | } | 369 | } |
| 370 | 370 | ||
| 371 | static int copy_pmd(pud_t *dst_pud, pud_t *src_pud, unsigned long start, | 371 | static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start, |
| 372 | unsigned long end) | 372 | unsigned long end) |
| 373 | { | 373 | { |
| 374 | pmd_t *src_pmd; | 374 | pmd_t *src_pmdp; |
| 375 | pmd_t *dst_pmd; | 375 | pmd_t *dst_pmdp; |
| 376 | unsigned long next; | 376 | unsigned long next; |
| 377 | unsigned long addr = start; | 377 | unsigned long addr = start; |
| 378 | 378 | ||
| 379 | if (pud_none(*dst_pud)) { | 379 | if (pud_none(READ_ONCE(*dst_pudp))) { |
| 380 | dst_pmd = (pmd_t *)get_safe_page(GFP_ATOMIC); | 380 | dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); |
| 381 | if (!dst_pmd) | 381 | if (!dst_pmdp) |
| 382 | return -ENOMEM; | 382 | return -ENOMEM; |
| 383 | pud_populate(&init_mm, dst_pud, dst_pmd); | 383 | pud_populate(&init_mm, dst_pudp, dst_pmdp); |
| 384 | } | 384 | } |
| 385 | dst_pmd = pmd_offset(dst_pud, start); | 385 | dst_pmdp = pmd_offset(dst_pudp, start); |
| 386 | 386 | ||
| 387 | src_pmd = pmd_offset(src_pud, start); | 387 | src_pmdp = pmd_offset(src_pudp, start); |
| 388 | do { | 388 | do { |
| 389 | pmd_t pmd = READ_ONCE(*src_pmdp); | ||
| 390 | |||
| 389 | next = pmd_addr_end(addr, end); | 391 | next = pmd_addr_end(addr, end); |
| 390 | if (pmd_none(*src_pmd)) | 392 | if (pmd_none(pmd)) |
| 391 | continue; | 393 | continue; |
| 392 | if (pmd_table(*src_pmd)) { | 394 | if (pmd_table(pmd)) { |
| 393 | if (copy_pte(dst_pmd, src_pmd, addr, next)) | 395 | if (copy_pte(dst_pmdp, src_pmdp, addr, next)) |
| 394 | return -ENOMEM; | 396 | return -ENOMEM; |
| 395 | } else { | 397 | } else { |
| 396 | set_pmd(dst_pmd, | 398 | set_pmd(dst_pmdp, |
| 397 | __pmd(pmd_val(*src_pmd) & ~PMD_SECT_RDONLY)); | 399 | __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY)); |
| 398 | } | 400 | } |
| 399 | } while (dst_pmd++, src_pmd++, addr = next, addr != end); | 401 | } while (dst_pmdp++, src_pmdp++, addr = next, addr != end); |
| 400 | 402 | ||
| 401 | return 0; | 403 | return 0; |
| 402 | } | 404 | } |
| 403 | 405 | ||
| 404 | static int copy_pud(pgd_t *dst_pgd, pgd_t *src_pgd, unsigned long start, | 406 | static int copy_pud(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start, |
| 405 | unsigned long end) | 407 | unsigned long end) |
| 406 | { | 408 | { |
| 407 | pud_t *dst_pud; | 409 | pud_t *dst_pudp; |
| 408 | pud_t *src_pud; | 410 | pud_t *src_pudp; |
| 409 | unsigned long next; | 411 | unsigned long next; |
| 410 | unsigned long addr = start; | 412 | unsigned long addr = start; |
| 411 | 413 | ||
| 412 | if (pgd_none(*dst_pgd)) { | 414 | if (pgd_none(READ_ONCE(*dst_pgdp))) { |
| 413 | dst_pud = (pud_t *)get_safe_page(GFP_ATOMIC); | 415 | dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); |
| 414 | if (!dst_pud) | 416 | if (!dst_pudp) |
| 415 | return -ENOMEM; | 417 | return -ENOMEM; |
| 416 | pgd_populate(&init_mm, dst_pgd, dst_pud); | 418 | pgd_populate(&init_mm, dst_pgdp, dst_pudp); |
| 417 | } | 419 | } |
| 418 | dst_pud = pud_offset(dst_pgd, start); | 420 | dst_pudp = pud_offset(dst_pgdp, start); |
| 419 | 421 | ||
| 420 | src_pud = pud_offset(src_pgd, start); | 422 | src_pudp = pud_offset(src_pgdp, start); |
| 421 | do { | 423 | do { |
| 424 | pud_t pud = READ_ONCE(*src_pudp); | ||
| 425 | |||
| 422 | next = pud_addr_end(addr, end); | 426 | next = pud_addr_end(addr, end); |
| 423 | if (pud_none(*src_pud)) | 427 | if (pud_none(pud)) |
| 424 | continue; | 428 | continue; |
| 425 | if (pud_table(*(src_pud))) { | 429 | if (pud_table(pud)) { |
| 426 | if (copy_pmd(dst_pud, src_pud, addr, next)) | 430 | if (copy_pmd(dst_pudp, src_pudp, addr, next)) |
| 427 | return -ENOMEM; | 431 | return -ENOMEM; |
| 428 | } else { | 432 | } else { |
| 429 | set_pud(dst_pud, | 433 | set_pud(dst_pudp, |
| 430 | __pud(pud_val(*src_pud) & ~PMD_SECT_RDONLY)); | 434 | __pud(pud_val(pud) & ~PMD_SECT_RDONLY)); |
| 431 | } | 435 | } |
| 432 | } while (dst_pud++, src_pud++, addr = next, addr != end); | 436 | } while (dst_pudp++, src_pudp++, addr = next, addr != end); |
| 433 | 437 | ||
| 434 | return 0; | 438 | return 0; |
| 435 | } | 439 | } |
| 436 | 440 | ||
| 437 | static int copy_page_tables(pgd_t *dst_pgd, unsigned long start, | 441 | static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start, |
| 438 | unsigned long end) | 442 | unsigned long end) |
| 439 | { | 443 | { |
| 440 | unsigned long next; | 444 | unsigned long next; |
| 441 | unsigned long addr = start; | 445 | unsigned long addr = start; |
| 442 | pgd_t *src_pgd = pgd_offset_k(start); | 446 | pgd_t *src_pgdp = pgd_offset_k(start); |
| 443 | 447 | ||
| 444 | dst_pgd = pgd_offset_raw(dst_pgd, start); | 448 | dst_pgdp = pgd_offset_raw(dst_pgdp, start); |
| 445 | do { | 449 | do { |
| 446 | next = pgd_addr_end(addr, end); | 450 | next = pgd_addr_end(addr, end); |
| 447 | if (pgd_none(*src_pgd)) | 451 | if (pgd_none(READ_ONCE(*src_pgdp))) |
| 448 | continue; | 452 | continue; |
| 449 | if (copy_pud(dst_pgd, src_pgd, addr, next)) | 453 | if (copy_pud(dst_pgdp, src_pgdp, addr, next)) |
| 450 | return -ENOMEM; | 454 | return -ENOMEM; |
| 451 | } while (dst_pgd++, src_pgd++, addr = next, addr != end); | 455 | } while (dst_pgdp++, src_pgdp++, addr = next, addr != end); |
| 452 | 456 | ||
| 453 | return 0; | 457 | return 0; |
| 454 | } | 458 | } |
| @@ -908,9 +908,9 @@ static void __armv8pmu_probe_pmu(void *info) · 907 unmodified lines | |||
| 908 | int pmuver; | 908 | int pmuver; |
| 909 | 909 | ||
| 910 | dfr0 = read_sysreg(id_aa64dfr0_el1); | 910 | dfr0 = read_sysreg(id_aa64dfr0_el1); |
| 911 | pmuver = cpuid_feature_extract_signed_field(dfr0, | 911 | pmuver = cpuid_feature_extract_unsigned_field(dfr0, |
| 912 | ID_AA64DFR0_PMUVER_SHIFT); | 912 | ID_AA64DFR0_PMUVER_SHIFT); |
| 913 | if (pmuver < 1) | 913 | if (pmuver == 0xf || pmuver == 0) |
| 914 | return; | 914 | return; |
| 915 | 915 | ||
| 916 | probe->present = true; | 916 | probe->present = true; |
| @@ -220,8 +220,15 @@ void __show_regs(struct pt_regs *regs) · 219 unmodified lines | |||
| 220 | 220 | ||
| 221 | show_regs_print_info(KERN_DEFAULT); | 221 | show_regs_print_info(KERN_DEFAULT); |
| 222 | print_pstate(regs); | 222 | print_pstate(regs); |
| 223 | printk("pc : %pS\n", (void *)regs->pc); | 223 | |
| 224 | printk("lr : %pS\n", (void *)lr); | 224 | if (!user_mode(regs)) { |
| 225 | printk("pc : %pS\n", (void *)regs->pc); | ||
| 226 | printk("lr : %pS\n", (void *)lr); | ||
| 227 | } else { | ||
| 228 | printk("pc : %016llx\n", regs->pc); | ||
| 229 | printk("lr : %016llx\n", lr); | ||
| 230 | } | ||
| 231 | |||
| 225 | printk("sp : %016llx\n", sp); | 232 | printk("sp : %016llx\n", sp); |
| 226 | 233 | ||
| 227 | i = top_reg; | 234 | i = top_reg; |
| @@ -1419,7 +1419,7 @@ static int compat_ptrace_hbp_get(unsigned int note_type, · 1418 unmodified lines | |||
| 1419 | u64 addr = 0; | 1419 | u64 addr = 0; |
| 1420 | u32 ctrl = 0; | 1420 | u32 ctrl = 0; |
| 1421 | 1421 | ||
| 1422 | int err, idx = compat_ptrace_hbp_num_to_idx(num);; | 1422 | int err, idx = compat_ptrace_hbp_num_to_idx(num); |
| 1423 | 1423 | ||
| 1424 | if (num & 1) { | 1424 | if (num & 1) { |
| 1425 | err = ptrace_hbp_get_addr(note_type, tsk, idx, &addr); | 1425 | err = ptrace_hbp_get_addr(note_type, tsk, idx, &addr); |
| @@ -59,6 +59,11 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) · 58 unmodified lines | |||
| 59 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER | 59 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER |
| 60 | if (tsk->ret_stack && | 60 | if (tsk->ret_stack && |
| 61 | (frame->pc == (unsigned long)return_to_handler)) { | 61 | (frame->pc == (unsigned long)return_to_handler)) { |
| 62 | if (WARN_ON_ONCE(frame->graph == -1)) | ||
| 63 | return -EINVAL; | ||
| 64 | if (frame->graph < -1) | ||
| 65 | frame->graph += FTRACE_NOTRACE_DEPTH; | ||
| 66 | |||
| 62 | /* | 67 | /* |
| 63 | * This is a case where function graph tracer has | 68 | * This is a case where function graph tracer has |
| 64 | * modified a return address (LR) in a stack frame | 69 | * modified a return address (LR) in a stack frame |
| @@ -57,7 +57,7 @@ do_compat_cache_op(unsigned long start, unsigned long end, int flags) · 56 unmodified lines | |||
| 57 | if (end < start || flags) | 57 | if (end < start || flags) |
| 58 | return -EINVAL; | 58 | return -EINVAL; |
| 59 | 59 | ||
| 60 | if (!access_ok(VERIFY_READ, start, end - start)) | 60 | if (!access_ok(VERIFY_READ, (const void __user *)start, end - start)) |
| 61 | return -EFAULT; | 61 | return -EFAULT; |
| 62 | 62 | ||
| 63 | return __do_compat_cache_op(start, end); | 63 | return __do_compat_cache_op(start, end); |
| @@ -52,7 +52,7 @@ unsigned long profile_pc(struct pt_regs *regs) · 51 unmodified lines | |||
| 52 | frame.fp = regs->regs[29]; | 52 | frame.fp = regs->regs[29]; |
| 53 | frame.pc = regs->pc; | 53 | frame.pc = regs->pc; |
| 54 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER | 54 | #ifdef CONFIG_FUNCTION_GRAPH_TRACER |
| 55 | frame.graph = -1; /* no task info */ | 55 | frame.graph = current->curr_ret_stack; |
| 56 | #endif | 56 | #endif |
| 57 | do { | 57 | do { |
| 58 | int ret = unwind_frame(NULL, &frame); | 58 | int ret = unwind_frame(NULL, &frame); |
| @@ -57,7 +57,7 @@ static const char *handler[]= { · 56 unmodified lines | |||
| 57 | "Error" | 57 | "Error" |
| 58 | }; | 58 | }; |
| 59 | 59 | ||
| 60 | int show_unhandled_signals = 1; | 60 | int show_unhandled_signals = 0; |
| 61 | 61 | ||
| 62 | static void dump_backtrace_entry(unsigned long where) | 62 | static void dump_backtrace_entry(unsigned long where) |
| 63 | { | 63 | { |
| @@ -526,14 +526,6 @@ asmlinkage long do_ni_syscall(struct pt_regs *regs) · 462 unmodified lines | |||
| 526 | } | 526 | } |
| 527 | #endif | 527 | #endif |
| 528 | 528 | ||
| 529 | if (show_unhandled_signals_ratelimited()) { | ||
| 530 | pr_info("%s[%d]: syscall %d\n", current->comm, | ||
| 531 | task_pid_nr(current), regs->syscallno); | ||
| 532 | dump_instr("", regs); | ||
| 533 | if (user_mode(regs)) | ||
| 534 | __show_regs(regs); | ||
| 535 | } | ||
| 536 | |||
| 537 | return sys_ni_syscall(); | 529 | return sys_ni_syscall(); |
| 538 | } | 530 | } |
| 539 | 531 | ||
| @@ -407,8 +407,10 @@ again: · 406 unmodified lines | |||
| 407 | u32 midr = read_cpuid_id(); | 407 | u32 midr = read_cpuid_id(); |
| 408 | 408 | ||
| 409 | /* Apply BTAC predictors mitigation to all Falkor chips */ | 409 | /* Apply BTAC predictors mitigation to all Falkor chips */ |
| 410 | if ((midr & MIDR_CPU_MODEL_MASK) == MIDR_QCOM_FALKOR_V1) | 410 | if (((midr & MIDR_CPU_MODEL_MASK) == MIDR_QCOM_FALKOR) || |
| 411 | ((midr & MIDR_CPU_MODEL_MASK) == MIDR_QCOM_FALKOR_V1)) { | ||
| 411 | __qcom_hyp_sanitize_btac_predictors(); | 412 | __qcom_hyp_sanitize_btac_predictors(); |
| 413 | } | ||
| 412 | } | 414 | } |
| 413 | 415 | ||
| 414 | fp_enabled = __fpsimd_enabled(); | 416 | fp_enabled = __fpsimd_enabled(); |
| @@ -286,48 +286,52 @@ static void note_page(struct pg_state *st, unsigned long addr, unsigned level, · 285 unmodified lines | |||
| 286 | 286 | ||
| 287 | } | 287 | } |
| 288 | 288 | ||
| 289 | static void walk_pte(struct pg_state *st, pmd_t *pmd, unsigned long start) | 289 | static void walk_pte(struct pg_state *st, pmd_t *pmdp, unsigned long start) |
| 290 | { | 290 | { |
| 291 | pte_t *pte = pte_offset_kernel(pmd, 0UL); | 291 | pte_t *ptep = pte_offset_kernel(pmdp, 0UL); |
| 292 | unsigned long addr; | 292 | unsigned long addr; |
| 293 | unsigned i; | 293 | unsigned i; |
| 294 | 294 | ||
| 295 | for (i = 0; i < PTRS_PER_PTE; i++, pte++) { | 295 | for (i = 0; i < PTRS_PER_PTE; i++, ptep++) { |
| 296 | addr = start + i * PAGE_SIZE; | 296 | addr = start + i * PAGE_SIZE; |
| 297 | note_page(st, addr, 4, pte_val(*pte)); | 297 | note_page(st, addr, 4, READ_ONCE(pte_val(*ptep))); |
| 298 | } | 298 | } |
| 299 | } | 299 | } |
| 300 | 300 | ||
| 301 | static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) | 301 | static void walk_pmd(struct pg_state *st, pud_t *pudp, unsigned long start) |
| 302 | { | 302 | { |
| 303 | pmd_t *pmd = pmd_offset(pud, 0UL); | 303 | pmd_t *pmdp = pmd_offset(pudp, 0UL); |
| 304 | unsigned long addr; | 304 | unsigned long addr; |
| 305 | unsigned i; | 305 | unsigned i; |
| 306 | 306 | ||
| 307 | for (i = 0; i < PTRS_PER_PMD; i++, pmd++) { | 307 | for (i = 0; i < PTRS_PER_PMD; i++, pmdp++) { |
| 308 | pmd_t pmd = READ_ONCE(*pmdp); | ||
| 309 | |||
| 308 | addr = start + i * PMD_SIZE; | 310 | addr = start + i * PMD_SIZE; |
| 309 | if (pmd_none(*pmd) || pmd_sect(*pmd)) { | 311 | if (pmd_none(pmd) || pmd_sect(pmd)) { |
| 310 | note_page(st, addr, 3, pmd_val(*pmd)); | 312 | note_page(st, addr, 3, pmd_val(pmd)); |
| 311 | } else { | 313 | } else { |
| 312 | BUG_ON(pmd_bad(*pmd)); | 314 | BUG_ON(pmd_bad(pmd)); |
| 313 | walk_pte(st, pmd, addr); | 315 | walk_pte(st, pmdp, addr); |
| 314 | } | 316 | } |
| 315 | } | 317 | } |
| 316 | } | 318 | } |
| 317 | 319 | ||
| 318 | static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) | 320 | static void walk_pud(struct pg_state *st, pgd_t *pgdp, unsigned long start) |
| 319 | { | 321 | { |
| 320 | pud_t *pud = pud_offset(pgd, 0UL); | 322 | pud_t *pudp = pud_offset(pgdp, 0UL); |
| 321 | unsigned long addr; | 323 | unsigned long addr; |
| 322 | unsigned i; | 324 | unsigned i; |
| 323 | 325 | ||
| 324 | for (i = 0; i < PTRS_PER_PUD; i++, pud++) { | 326 | for (i = 0; i < PTRS_PER_PUD; i++, pudp++) { |
| 327 | pud_t pud = READ_ONCE(*pudp); | ||
| 328 | |||
| 325 | addr = start + i * PUD_SIZE; | 329 | addr = start + i * PUD_SIZE; |
| 326 | if (pud_none(*pud) || pud_sect(*pud)) { | 330 | if (pud_none(pud) || pud_sect(pud)) { |
| 327 | note_page(st, addr, 2, pud_val(*pud)); | 331 | note_page(st, addr, 2, pud_val(pud)); |
| 328 | } else { | 332 | } else { |
| 329 | BUG_ON(pud_bad(*pud)); | 333 | BUG_ON(pud_bad(pud)); |
| 330 | walk_pmd(st, pud, addr); | 334 | walk_pmd(st, pudp, addr); |
| 331 | } | 335 | } |
| 332 | } | 336 | } |
| 333 | } | 337 | } |
| @@ -335,17 +339,19 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) · 1 unmodified line | |||
| 335 | static void walk_pgd(struct pg_state *st, struct mm_struct *mm, | 339 | static void walk_pgd(struct pg_state *st, struct mm_struct *mm, |
| 336 | unsigned long start) | 340 | unsigned long start) |
| 337 | { | 341 | { |
| 338 | pgd_t *pgd = pgd_offset(mm, 0UL); | 342 | pgd_t *pgdp = pgd_offset(mm, 0UL); |
| 339 | unsigned i; | 343 | unsigned i; |
| 340 | unsigned long addr; | 344 | unsigned long addr; |
| 341 | 345 | ||
| 342 | for (i = 0; i < PTRS_PER_PGD; i++, pgd++) { | 346 | for (i = 0; i < PTRS_PER_PGD; i++, pgdp++) { |
| 347 | pgd_t pgd = READ_ONCE(*pgdp); | ||
| 348 | |||
| 343 | addr = start + i * PGDIR_SIZE; | 349 | addr = start + i * PGDIR_SIZE; |
| 344 | if (pgd_none(*pgd)) { | 350 | if (pgd_none(pgd)) { |
| 345 | note_page(st, addr, 1, pgd_val(*pgd)); | 351 | note_page(st, addr, 1, pgd_val(pgd)); |
| 346 | } else { | 352 | } else { |
| 347 | BUG_ON(pgd_bad(*pgd)); | 353 | BUG_ON(pgd_bad(pgd)); |
| 348 | walk_pud(st, pgd, addr); | 354 | walk_pud(st, pgdp, addr); |
| 349 | } | 355 | } |
| 350 | } | 356 | } |
| 351 | } | 357 | } |
| @@ -130,7 +130,8 @@ static void mem_abort_decode(unsigned int esr) · 129 unmodified lines | |||
| 130 | void show_pte(unsigned long addr) | 130 | void show_pte(unsigned long addr) |
| 131 | { | 131 | { |
| 132 | struct mm_struct *mm; | 132 | struct mm_struct *mm; |
| 133 | pgd_t *pgd; | 133 | pgd_t *pgdp; |
| 134 | pgd_t pgd; | ||
| 134 | 135 | ||
| 135 | if (addr < TASK_SIZE) { | 136 | if (addr < TASK_SIZE) { |
| 136 | /* TTBR0 */ | 137 | /* TTBR0 */ |
| @@ -149,33 +150,37 @@ void show_pte(unsigned long addr) · 12 unmodified lines | |||
| 149 | return; | 150 | return; |
| 150 | } | 151 | } |
| 151 | 152 | ||
| 152 | pr_alert("%s pgtable: %luk pages, %u-bit VAs, pgd = %p\n", | 153 | pr_alert("%s pgtable: %luk pages, %u-bit VAs, pgdp = %p\n", |
| 153 | mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K, | 154 | mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K, |
| 154 | VA_BITS, mm->pgd); | 155 | VA_BITS, mm->pgd); |
| 155 | pgd = pgd_offset(mm, addr); | 156 | pgdp = pgd_offset(mm, addr); |
| 156 | pr_alert("[%016lx] *pgd=%016llx", addr, pgd_val(*pgd)); | 157 | pgd = READ_ONCE(*pgdp); |
| 158 | pr_alert("[%016lx] pgd=%016llx", addr, pgd_val(pgd)); | ||
| 157 | 159 | ||
| 158 | do { | 160 | do { |
| 159 | pud_t *pud; | 161 | pud_t *pudp, pud; |
| 160 | pmd_t *pmd; | 162 | pmd_t *pmdp, pmd; |
| 161 | pte_t *pte; | 163 | pte_t *ptep, pte; |
| 162 | 164 | ||
| 163 | if (pgd_none(*pgd) || pgd_bad(*pgd)) | 165 | if (pgd_none(pgd) || pgd_bad(pgd)) |
| 164 | break; | 166 | break; |
| 165 | 167 | ||
| 166 | pud = pud_offset(pgd, addr); | 168 | pudp = pud_offset(pgdp, addr); |
| 167 | pr_cont(", *pud=%016llx", pud_val(*pud)); | 169 | pud = READ_ONCE(*pudp); |
| 168 | if (pud_none(*pud) || pud_bad(*pud)) | 170 | pr_cont(", pud=%016llx", pud_val(pud)); |
| 171 | if (pud_none(pud) || pud_bad(pud)) | ||
| 169 | break; | 172 | break; |
| 170 | 173 | ||
| 171 | pmd = pmd_offset(pud, addr); | 174 | pmdp = pmd_offset(pudp, addr); |
| 172 | pr_cont(", *pmd=%016llx", pmd_val(*pmd)); | 175 | pmd = READ_ONCE(*pmdp); |
| 173 | if (pmd_none(*pmd) || pmd_bad(*pmd)) | 176 | pr_cont(", pmd=%016llx", pmd_val(pmd)); |
| 177 | if (pmd_none(pmd) || pmd_bad(pmd)) | ||
| 174 | break; | 178 | break; |
| 175 | 179 | ||
| 176 | pte = pte_offset_map(pmd, addr); | 180 | ptep = pte_offset_map(pmdp, addr); |
| 177 | pr_cont(", *pte=%016llx", pte_val(*pte)); | 181 | pte = READ_ONCE(*ptep); |
| 178 | pte_unmap(pte); | 182 | pr_cont(", pte=%016llx", pte_val(pte)); |
| 183 | pte_unmap(ptep); | ||
| 179 | } while(0); | 184 | } while(0); |
| 180 | 185 | ||
| 181 | pr_cont("\n"); | 186 | pr_cont("\n"); |
| @@ -196,8 +201,9 @@ int ptep_set_access_flags(struct vm_area_struct *vma, · 14 unmodified lines | |||
| 196 | pte_t entry, int dirty) | 201 | pte_t entry, int dirty) |
| 197 | { | 202 | { |
| 198 | pteval_t old_pteval, pteval; | 203 | pteval_t old_pteval, pteval; |
| 204 | pte_t pte = READ_ONCE(*ptep); | ||
| 199 | 205 | ||
| 200 | if (pte_same(*ptep, entry)) | 206 | if (pte_same(pte, entry)) |
| 201 | return 0; | 207 | return 0; |
| 202 | 208 | ||
| 203 | /* only preserve the access flags and write permission */ | 209 | /* only preserve the access flags and write permission */ |
| @@ -210,7 +216,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, · 6 unmodified lines | |||
| 210 | * (calculated as: a & b == ~(~a | ~b)). | 216 | * (calculated as: a & b == ~(~a | ~b)). |
| 211 | */ | 217 | */ |
| 212 | pte_val(entry) ^= PTE_RDONLY; | 218 | pte_val(entry) ^= PTE_RDONLY; |
| 213 | pteval = READ_ONCE(pte_val(*ptep)); | 219 | pteval = pte_val(pte); |
| 214 | do { | 220 | do { |
| 215 | old_pteval = pteval; | 221 | old_pteval = pteval; |
| 216 | pteval ^= PTE_RDONLY; | 222 | pteval ^= PTE_RDONLY; |
| @@ -54,14 +54,14 @@ static inline pgprot_t pte_pgprot(pte_t pte) · 53 unmodified lines | |||
| 54 | static int find_num_contig(struct mm_struct *mm, unsigned long addr, | 54 | static int find_num_contig(struct mm_struct *mm, unsigned long addr, |
| 55 | pte_t *ptep, size_t *pgsize) | 55 | pte_t *ptep, size_t *pgsize) |
| 56 | { | 56 | { |
| 57 | pgd_t *pgd = pgd_offset(mm, addr); | 57 | pgd_t *pgdp = pgd_offset(mm, addr); |
| 58 | pud_t *pud; | 58 | pud_t *pudp; |
| 59 | pmd_t *pmd; | 59 | pmd_t *pmdp; |
| 60 | 60 | ||
| 61 | *pgsize = PAGE_SIZE; | 61 | *pgsize = PAGE_SIZE; |
| 62 | pud = pud_offset(pgd, addr); | 62 | pudp = pud_offset(pgdp, addr); |
| 63 | pmd = pmd_offset(pud, addr); | 63 | pmdp = pmd_offset(pudp, addr); |
| 64 | if ((pte_t *)pmd == ptep) { | 64 | if ((pte_t *)pmdp == ptep) { |
| 65 | *pgsize = PMD_SIZE; | 65 | *pgsize = PMD_SIZE; |
| 66 | return CONT_PMDS; | 66 | return CONT_PMDS; |
| 67 | } | 67 | } |
| @@ -181,11 +181,8 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 113 unmodified lines | |||
| 181 | 181 | ||
| 182 | clear_flush(mm, addr, ptep, pgsize, ncontig); | 182 | clear_flush(mm, addr, ptep, pgsize, ncontig); |
| 183 | 183 | ||
| 184 | for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) { | 184 | for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) |
| 185 | pr_debug("%s: set pte %p to 0x%llx\n", __func__, ptep, | ||
| 186 | pte_val(pfn_pte(pfn, hugeprot))); | ||
| 187 | set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); | 185 | set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); |
| 188 | } | ||
| 189 | } | 186 | } |
| 190 | 187 | ||
| 191 | void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, | 188 | void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, |
| @@ -203,20 +200,20 @@ void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, · 11 unmodified lines | |||
| 203 | pte_t *huge_pte_alloc(struct mm_struct *mm, | 200 | pte_t *huge_pte_alloc(struct mm_struct *mm, |
| 204 | unsigned long addr, unsigned long sz) | 201 | unsigned long addr, unsigned long sz) |
| 205 | { | 202 | { |
| 206 | pgd_t *pgd; | 203 | pgd_t *pgdp; |
| 207 | pud_t *pud; | 204 | pud_t *pudp; |
| 208 | pte_t *pte = NULL; | 205 | pmd_t *pmdp; |
| 209 | 206 | pte_t *ptep = NULL; | |
| 210 | pr_debug("%s: addr:0x%lx sz:0x%lx\n", __func__, addr, sz); | 207 | |
| 211 | pgd = pgd_offset(mm, addr); | 208 | pgdp = pgd_offset(mm, addr); |
| 212 | pud = pud_alloc(mm, pgd, addr); | 209 | pudp = pud_alloc(mm, pgdp, addr); |
| 213 | if (!pud) | 210 | if (!pudp) |
| 214 | return NULL; | 211 | return NULL; |
| 215 | 212 | ||
| 216 | if (sz == PUD_SIZE) { | 213 | if (sz == PUD_SIZE) { |
| 217 | pte = (pte_t *)pud; | 214 | ptep = (pte_t *)pudp; |
| 218 | } else if (sz == (PAGE_SIZE * CONT_PTES)) { | 215 | } else if (sz == (PAGE_SIZE * CONT_PTES)) { |
| 219 | pmd_t *pmd = pmd_alloc(mm, pud, addr); | 216 | pmdp = pmd_alloc(mm, pudp, addr); |
| 220 | 217 | ||
| 221 | WARN_ON(addr & (sz - 1)); | 218 | WARN_ON(addr & (sz - 1)); |
| 222 | /* | 219 | /* |
| @@ -226,60 +223,55 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, · 3 unmodified lines | |||
| 226 | * will be no pte_unmap() to correspond with this | 223 | * will be no pte_unmap() to correspond with this |
| 227 | * pte_alloc_map(). | 224 | * pte_alloc_map(). |
| 228 | */ | 225 | */ |
| 229 | pte = pte_alloc_map(mm, pmd, addr); | 226 | ptep = pte_alloc_map(mm, pmdp, addr); |
| 230 | } else if (sz == PMD_SIZE) { | 227 | } else if (sz == PMD_SIZE) { |
| 231 | if (IS_ENABLED(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && | 228 | if (IS_ENABLED(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && |
| 232 | pud_none(*pud)) | 229 | pud_none(READ_ONCE(*pudp))) |
| 233 | pte = huge_pmd_share(mm, addr, pud); | 230 | ptep = huge_pmd_share(mm, addr, pudp); |
| 234 | else | 231 | else |
| 235 | pte = (pte_t *)pmd_alloc(mm, pud, addr); | 232 | ptep = (pte_t *)pmd_alloc(mm, pudp, addr); |
| 236 | } else if (sz == (PMD_SIZE * CONT_PMDS)) { | 233 | } else if (sz == (PMD_SIZE * CONT_PMDS)) { |
| 237 | pmd_t *pmd; | 234 | pmdp = pmd_alloc(mm, pudp, addr); |
| 238 | |||
| 239 | pmd = pmd_alloc(mm, pud, addr); | ||
| 240 | WARN_ON(addr & (sz - 1)); | 235 | WARN_ON(addr & (sz - 1)); |
| 241 | return (pte_t *)pmd; | 236 | return (pte_t *)pmdp; |
| 242 | } | 237 | } |
| 243 | 238 | ||
| 244 | pr_debug("%s: addr:0x%lx sz:0x%lx ret pte=%p/0x%llx\n", __func__, addr, | 239 | return ptep; |
| 245 | sz, pte, pte_val(*pte)); | ||
| 246 | return pte; | ||
| 247 | } | 240 | } |
| 248 | 241 | ||
| 249 | pte_t *huge_pte_offset(struct mm_struct *mm, | 242 | pte_t *huge_pte_offset(struct mm_struct *mm, |
| 250 | unsigned long addr, unsigned long sz) | 243 | unsigned long addr, unsigned long sz) |
| 251 | { | 244 | { |
| 252 | pgd_t *pgd; | 245 | pgd_t *pgdp; |
| 253 | pud_t *pud; | 246 | pud_t *pudp, pud; |
| 254 | pmd_t *pmd; | 247 | pmd_t *pmdp, pmd; |
| 255 | 248 | ||
| 256 | pgd = pgd_offset(mm, addr); | 249 | pgdp = pgd_offset(mm, addr); |
| 257 | pr_debug("%s: addr:0x%lx pgd:%p\n", __func__, addr, pgd); | 250 | if (!pgd_present(READ_ONCE(*pgdp))) |
| 258 | if (!pgd_present(*pgd)) | ||
| 259 | return NULL; | 251 | return NULL; |
| 260 | 252 | ||
| 261 | pud = pud_offset(pgd, addr); | 253 | pudp = pud_offset(pgdp, addr); |
| 262 | if (sz != PUD_SIZE && pud_none(*pud)) | 254 | pud = READ_ONCE(*pudp); |
| 255 | if (sz != PUD_SIZE && pud_none(pud)) | ||
| 263 | return NULL; | 256 | return NULL; |
| 264 | /* hugepage or swap? */ | 257 | /* hugepage or swap? */ |
| 265 | if (pud_huge(*pud) || !pud_present(*pud)) | 258 | if (pud_huge(pud) || !pud_present(pud)) |
| 266 | return (pte_t *)pud; | 259 | return (pte_t *)pudp; |
| 267 | /* table; check the next level */ | 260 | /* table; check the next level */ |
| 268 | 261 | ||
| 269 | if (sz == CONT_PMD_SIZE) | 262 | if (sz == CONT_PMD_SIZE) |
| 270 | addr &= CONT_PMD_MASK; | 263 | addr &= CONT_PMD_MASK; |
| 271 | 264 | ||
| 272 | pmd = pmd_offset(pud, addr); | 265 | pmdp = pmd_offset(pudp, addr); |
| 266 | pmd = READ_ONCE(*pmdp); | ||
| 273 | if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) && | 267 | if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) && |
| 274 | pmd_none(*pmd)) | 268 | pmd_none(pmd)) |
| 275 | return NULL; | 269 | return NULL; |
| 276 | if (pmd_huge(*pmd) || !pmd_present(*pmd)) | 270 | if (pmd_huge(pmd) || !pmd_present(pmd)) |
| 277 | return (pte_t *)pmd; | 271 | return (pte_t *)pmdp; |
| 278 | 272 | ||
| 279 | if (sz == CONT_PTE_SIZE) { | 273 | if (sz == CONT_PTE_SIZE) |
| 280 | pte_t *pte = pte_offset_kernel(pmd, (addr & CONT_PTE_MASK)); | 274 | return pte_offset_kernel(pmdp, (addr & CONT_PTE_MASK)); |
| 281 | return pte; | ||
| 282 | } | ||
| 283 | 275 | ||
| 284 | return NULL; | 276 | return NULL; |
| 285 | } | 277 | } |
| @@ -367,7 +359,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, · 81 unmodified lines | |||
| 367 | size_t pgsize; | 359 | size_t pgsize; |
| 368 | pte_t pte; | 360 | pte_t pte; |
| 369 | 361 | ||
| 370 | if (!pte_cont(*ptep)) { | 362 | if (!pte_cont(READ_ONCE(*ptep))) { |
| 371 | ptep_set_wrprotect(mm, addr, ptep); | 363 | ptep_set_wrprotect(mm, addr, ptep); |
| 372 | return; | 364 | return; |
| 373 | } | 365 | } |
| @@ -391,7 +383,7 @@ void huge_ptep_clear_flush(struct vm_area_struct *vma, · 17 unmodified lines | |||
| 391 | size_t pgsize; | 383 | size_t pgsize; |
| 392 | int ncontig; | 384 | int ncontig; |
| 393 | 385 | ||
| 394 | if (!pte_cont(*ptep)) { | 386 | if (!pte_cont(READ_ONCE(*ptep))) { |
| 395 | ptep_clear_flush(vma, addr, ptep); | 387 | ptep_clear_flush(vma, addr, ptep); |
| 396 | return; | 388 | return; |
| 397 | } | 389 | } |
| @@ -44,92 +44,92 @@ static phys_addr_t __init kasan_alloc_zeroed_page(int node) · 43 unmodified lines | |||
| 44 | return __pa(p); | 44 | return __pa(p); |
| 45 | } | 45 | } |
| 46 | 46 | ||
| 47 | static pte_t *__init kasan_pte_offset(pmd_t *pmd, unsigned long addr, int node, | 47 | static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, |
| 48 | bool early) | 48 | bool early) |
| 49 | { | 49 | { |
| 50 | if (pmd_none(*pmd)) { | 50 | if (pmd_none(READ_ONCE(*pmdp))) { |
| 51 | phys_addr_t pte_phys = early ? __pa_symbol(kasan_zero_pte) | 51 | phys_addr_t pte_phys = early ? __pa_symbol(kasan_zero_pte) |
| 52 | : kasan_alloc_zeroed_page(node); | 52 | : kasan_alloc_zeroed_page(node); |
| 53 | __pmd_populate(pmd, pte_phys, PMD_TYPE_TABLE); | 53 | __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); |
| 54 | } | 54 | } |
| 55 | 55 | ||
| 56 | return early ? pte_offset_kimg(pmd, addr) | 56 | return early ? pte_offset_kimg(pmdp, addr) |
| 57 | : pte_offset_kernel(pmd, addr); | 57 | : pte_offset_kernel(pmdp, addr); |
| 58 | } | 58 | } |
| 59 | 59 | ||
| 60 | static pmd_t *__init kasan_pmd_offset(pud_t *pud, unsigned long addr, int node, | 60 | static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, |
| 61 | bool early) | 61 | bool early) |
| 62 | { | 62 | { |
| 63 | if (pud_none(*pud)) { | 63 | if (pud_none(READ_ONCE(*pudp))) { |
| 64 | phys_addr_t pmd_phys = early ? __pa_symbol(kasan_zero_pmd) | 64 | phys_addr_t pmd_phys = early ? __pa_symbol(kasan_zero_pmd) |
| 65 | : kasan_alloc_zeroed_page(node); | 65 | : kasan_alloc_zeroed_page(node); |
| 66 | __pud_populate(pud, pmd_phys, PMD_TYPE_TABLE); | 66 | __pud_populate(pudp, pmd_phys, PMD_TYPE_TABLE); |
| 67 | } | 67 | } |
| 68 | 68 | ||
| 69 | return early ? pmd_offset_kimg(pud, addr) : pmd_offset(pud, addr); | 69 | return early ? pmd_offset_kimg(pudp, addr) : pmd_offset(pudp, addr); |
| 70 | } | 70 | } |
| 71 | 71 | ||
| 72 | static pud_t *__init kasan_pud_offset(pgd_t *pgd, unsigned long addr, int node, | 72 | static pud_t *__init kasan_pud_offset(pgd_t *pgdp, unsigned long addr, int node, |
| 73 | bool early) | 73 | bool early) |
| 74 | { | 74 | { |
| 75 | if (pgd_none(*pgd)) { | 75 | if (pgd_none(READ_ONCE(*pgdp))) { |
| 76 | phys_addr_t pud_phys = early ? __pa_symbol(kasan_zero_pud) | 76 | phys_addr_t pud_phys = early ? __pa_symbol(kasan_zero_pud) |
| 77 | : kasan_alloc_zeroed_page(node); | 77 | : kasan_alloc_zeroed_page(node); |
| 78 | __pgd_populate(pgd, pud_phys, PMD_TYPE_TABLE); | 78 | __pgd_populate(pgdp, pud_phys, PMD_TYPE_TABLE); |
| 79 | } | 79 | } |
| 80 | 80 | ||
| 81 | return early ? pud_offset_kimg(pgd, addr) : pud_offset(pgd, addr); | 81 | return early ? pud_offset_kimg(pgdp, addr) : pud_offset(pgdp, addr); |
| 82 | } | 82 | } |
| 83 | 83 | ||
| 84 | static void __init kasan_pte_populate(pmd_t *pmd, unsigned long addr, | 84 | static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, |
| 85 | unsigned long end, int node, bool early) | 85 | unsigned long end, int node, bool early) |
| 86 | { | 86 | { |
| 87 | unsigned long next; | 87 | unsigned long next; |
| 88 | pte_t *pte = kasan_pte_offset(pmd, addr, node, early); | 88 | pte_t *ptep = kasan_pte_offset(pmdp, addr, node, early); |
| 89 | 89 | ||
| 90 | do { | 90 | do { |
| 91 | phys_addr_t page_phys = early ? __pa_symbol(kasan_zero_page) | 91 | phys_addr_t page_phys = early ? __pa_symbol(kasan_zero_page) |
| 92 | : kasan_alloc_zeroed_page(node); | 92 | : kasan_alloc_zeroed_page(node); |
| 93 | next = addr + PAGE_SIZE; | 93 | next = addr + PAGE_SIZE; |
| 94 | set_pte(pte, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); | 94 | set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); |
| 95 | } while (pte++, addr = next, addr != end && pte_none(*pte)); | 95 | } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); |
| 96 | } | 96 | } |
| 97 | 97 | ||
| 98 | static void __init kasan_pmd_populate(pud_t *pud, unsigned long addr, | 98 | static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, |
| 99 | unsigned long end, int node, bool early) | 99 | unsigned long end, int node, bool early) |
| 100 | { | 100 | { |
| 101 | unsigned long next; | 101 | unsigned long next; |
| 102 | pmd_t *pmd = kasan_pmd_offset(pud, addr, node, early); | 102 | pmd_t *pmdp = kasan_pmd_offset(pudp, addr, node, early); |
| 103 | 103 | ||
| 104 | do { | 104 | do { |
| 105 | next = pmd_addr_end(addr, end); | 105 | next = pmd_addr_end(addr, end); |
| 106 | kasan_pte_populate(pmd, addr, next, node, early); | 106 | kasan_pte_populate(pmdp, addr, next, node, early); |
| 107 | } while (pmd++, addr = next, addr != end && pmd_none(*pmd)); | 107 | } while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp))); |
| 108 | } | 108 | } |
| 109 | 109 | ||
| 110 | static void __init kasan_pud_populate(pgd_t *pgd, unsigned long addr, | 110 | static void __init kasan_pud_populate(pgd_t *pgdp, unsigned long addr, |
| 111 | unsigned long end, int node, bool early) | 111 | unsigned long end, int node, bool early) |
| 112 | { | 112 | { |
| 113 | unsigned long next; | 113 | unsigned long next; |
| 114 | pud_t *pud = kasan_pud_offset(pgd, addr, node, early); | 114 | pud_t *pudp = kasan_pud_offset(pgdp, addr, node, early); |
| 115 | 115 | ||
| 116 | do { | 116 | do { |
| 117 | next = pud_addr_end(addr, end); | 117 | next = pud_addr_end(addr, end); |
| 118 | kasan_pmd_populate(pud, addr, next, node, early); | 118 | kasan_pmd_populate(pudp, addr, next, node, early); |
| 119 | } while (pud++, addr = next, addr != end && pud_none(*pud)); | 119 | } while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp))); |
| 120 | } | 120 | } |
| 121 | 121 | ||
| 122 | static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, | 122 | static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, |
| 123 | int node, bool early) | 123 | int node, bool early) |
| 124 | { | 124 | { |
| 125 | unsigned long next; | 125 | unsigned long next; |
| 126 | pgd_t *pgd; | 126 | pgd_t *pgdp; |
| 127 | 127 | ||
| 128 | pgd = pgd_offset_k(addr); | 128 | pgdp = pgd_offset_k(addr); |
| 129 | do { | 129 | do { |
| 130 | next = pgd_addr_end(addr, end); | 130 | next = pgd_addr_end(addr, end); |
| 131 | kasan_pud_populate(pgd, addr, next, node, early); | 131 | kasan_pud_populate(pgdp, addr, next, node, early); |
| 132 | } while (pgd++, addr = next, addr != end); | 132 | } while (pgdp++, addr = next, addr != end); |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | /* The early shadow maps everything to a single page of zeroes */ | 135 | /* The early shadow maps everything to a single page of zeroes */ |
| @@ -155,14 +155,14 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, · 19 unmodified lines | |||
| 155 | */ | 155 | */ |
| 156 | void __init kasan_copy_shadow(pgd_t *pgdir) | 156 | void __init kasan_copy_shadow(pgd_t *pgdir) |
| 157 | { | 157 | { |
| 158 | pgd_t *pgd, *pgd_new, *pgd_end; | 158 | pgd_t *pgdp, *pgdp_new, *pgdp_end; |
| 159 | 159 | ||
| 160 | pgd = pgd_offset_k(KASAN_SHADOW_START); | 160 | pgdp = pgd_offset_k(KASAN_SHADOW_START); |
| 161 | pgd_end = pgd_offset_k(KASAN_SHADOW_END); | 161 | pgdp_end = pgd_offset_k(KASAN_SHADOW_END); |
| 162 | pgd_new = pgd_offset_raw(pgdir, KASAN_SHADOW_START); | 162 | pgdp_new = pgd_offset_raw(pgdir, KASAN_SHADOW_START); |
| 163 | do { | 163 | do { |
| 164 | set_pgd(pgd_new, *pgd); | 164 | set_pgd(pgdp_new, READ_ONCE(*pgdp)); |
| 165 | } while (pgd++, pgd_new++, pgd != pgd_end); | 165 | } while (pgdp++, pgdp_new++, pgdp != pgdp_end); |
| 166 | } | 166 | } |
| 167 | 167 | ||
| 168 | static void __init clear_pgds(unsigned long start, | 168 | static void __init clear_pgds(unsigned long start, |
| @@ -108,7 +108,7 @@ static bool pgattr_change_is_safe(u64 old, u64 new) · 107 unmodified lines | |||
| 108 | * The following mapping attributes may be updated in live | 108 | * The following mapping attributes may be updated in live |
| 109 | * kernel mappings without the need for break-before-make. | 109 | * kernel mappings without the need for break-before-make. |
| 110 | */ | 110 | */ |
| 111 | static const pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE; | 111 | static const pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; |
| 112 | 112 | ||
| 113 | /* creating or taking down mappings is always safe */ | 113 | /* creating or taking down mappings is always safe */ |
| 114 | if (old == 0 || new == 0) | 114 | if (old == 0 || new == 0) |
| @@ -118,52 +118,55 @@ static bool pgattr_change_is_safe(u64 old, u64 new) · 3 unmodified lines | |||
| 118 | if ((old | new) & PTE_CONT) | 118 | if ((old | new) & PTE_CONT) |
| 119 | return false; | 119 | return false; |
| 120 | 120 | ||
| 121 | /* Transitioning from Global to Non-Global is safe */ | 121 | /* Transitioning from Non-Global to Global is unsafe */ |
| 122 | if (((old ^ new) == PTE_NG) && (new & PTE_NG)) | 122 | if (old & ~new & PTE_NG) |
| 123 | return true; | 123 | return false; |
| 124 | 124 | ||
| 125 | return ((old ^ new) & ~mask) == 0; | 125 | return ((old ^ new) & ~mask) == 0; |
| 126 | } | 126 | } |
| 127 | 127 | ||
| 128 | static void init_pte(pmd_t *pmd, unsigned long addr, unsigned long end, | 128 | static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, |
| 129 | phys_addr_t phys, pgprot_t prot) | 129 | phys_addr_t phys, pgprot_t prot) |
| 130 | { | 130 | { |
| 131 | pte_t *pte; | 131 | pte_t *ptep; |
| 132 | 132 | ||
| 133 | pte = pte_set_fixmap_offset(pmd, addr); | 133 | ptep = pte_set_fixmap_offset(pmdp, addr); |
| 134 | do { | 134 | do { |
| 135 | pte_t old_pte = *pte; | 135 | pte_t old_pte = READ_ONCE(*ptep); |
| 136 | 136 | ||
| 137 | set_pte(pte, pfn_pte(__phys_to_pfn(phys), prot)); | 137 | set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); |
| 138 | 138 | ||
| 139 | /* | 139 | /* |
| 140 | * After the PTE entry has been populated once, we | 140 | * After the PTE entry has been populated once, we |
| 141 | * only allow updates to the permission attributes. | 141 | * only allow updates to the permission attributes. |
| 142 | */ | 142 | */ |
| 143 | BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), pte_val(*pte))); | 143 | BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), |
| 144 | READ_ONCE(pte_val(*ptep)))); | ||
| 144 | 145 | ||
| 145 | phys += PAGE_SIZE; | 146 | phys += PAGE_SIZE; |
| 146 | } while (pte++, addr += PAGE_SIZE, addr != end); | 147 | } while (ptep++, addr += PAGE_SIZE, addr != end); |
| 147 | 148 | ||
| 148 | pte_clear_fixmap(); | 149 | pte_clear_fixmap(); |
| 149 | } | 150 | } |
| 150 | 151 | ||
| 151 | static void alloc_init_cont_pte(pmd_t *pmd, unsigned long addr, | 152 | static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, |
| 152 | unsigned long end, phys_addr_t phys, | 153 | unsigned long end, phys_addr_t phys, |
| 153 | pgprot_t prot, | 154 | pgprot_t prot, |
| 154 | phys_addr_t (*pgtable_alloc)(void), | 155 | phys_addr_t (*pgtable_alloc)(void), |
| 155 | int flags) | 156 | int flags) |
| 156 | { | 157 | { |
| 157 | unsigned long next; | 158 | unsigned long next; |
| 159 | pmd_t pmd = READ_ONCE(*pmdp); | ||
| 158 | 160 | ||
| 159 | BUG_ON(pmd_sect(*pmd)); | 161 | BUG_ON(pmd_sect(pmd)); |
| 160 | if (pmd_none(*pmd)) { | 162 | if (pmd_none(pmd)) { |
| 161 | phys_addr_t pte_phys; | 163 | phys_addr_t pte_phys; |
| 162 | BUG_ON(!pgtable_alloc); | 164 | BUG_ON(!pgtable_alloc); |
| 163 | pte_phys = pgtable_alloc(); | 165 | pte_phys = pgtable_alloc(); |
| 164 | __pmd_populate(pmd, pte_phys, PMD_TYPE_TABLE); | 166 | __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); |
| 167 | pmd = READ_ONCE(*pmdp); | ||
| 165 | } | 168 | } |
| 166 | BUG_ON(pmd_bad(*pmd)); | 169 | BUG_ON(pmd_bad(pmd)); |
| 167 | 170 | ||
| 168 | do { | 171 | do { |
| 169 | pgprot_t __prot = prot; | 172 | pgprot_t __prot = prot; |
| @@ -175,67 +178,69 @@ static void alloc_init_cont_pte(pmd_t *pmd, unsigned long addr, · 5 unmodified lines | |||
| 175 | (flags & NO_CONT_MAPPINGS) == 0) | 178 | (flags & NO_CONT_MAPPINGS) == 0) |
| 176 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); | 179 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); |
| 177 | 180 | ||
| 178 | init_pte(pmd, addr, next, phys, __prot); | 181 | init_pte(pmdp, addr, next, phys, __prot); |
| 179 | 182 | ||
| 180 | phys += next - addr; | 183 | phys += next - addr; |
| 181 | } while (addr = next, addr != end); | 184 | } while (addr = next, addr != end); |
| 182 | } | 185 | } |
| 183 | 186 | ||
| 184 | static void init_pmd(pud_t *pud, unsigned long addr, unsigned long end, | 187 | static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, |
| 185 | phys_addr_t phys, pgprot_t prot, | 188 | phys_addr_t phys, pgprot_t prot, |
| 186 | phys_addr_t (*pgtable_alloc)(void), int flags) | 189 | phys_addr_t (*pgtable_alloc)(void), int flags) |
| 187 | { | 190 | { |
| 188 | unsigned long next; | 191 | unsigned long next; |
| 189 | pmd_t *pmd; | 192 | pmd_t *pmdp; |
| 190 | 193 | ||
| 191 | pmd = pmd_set_fixmap_offset(pud, addr); | 194 | pmdp = pmd_set_fixmap_offset(pudp, addr); |
| 192 | do { | 195 | do { |
| 193 | pmd_t old_pmd = *pmd; | 196 | pmd_t old_pmd = READ_ONCE(*pmdp); |
| 194 | 197 | ||
| 195 | next = pmd_addr_end(addr, end); | 198 | next = pmd_addr_end(addr, end); |
| 196 | 199 | ||
| 197 | /* try section mapping first */ | 200 | /* try section mapping first */ |
| 198 | if (((addr | next | phys) & ~SECTION_MASK) == 0 && | 201 | if (((addr | next | phys) & ~SECTION_MASK) == 0 && |
| 199 | (flags & NO_BLOCK_MAPPINGS) == 0) { | 202 | (flags & NO_BLOCK_MAPPINGS) == 0) { |
| 200 | pmd_set_huge(pmd, phys, prot); | 203 | pmd_set_huge(pmdp, phys, prot); |
| 201 | 204 | ||
| 202 | /* | 205 | /* |
| 203 | * After the PMD entry has been populated once, we | 206 | * After the PMD entry has been populated once, we |
| 204 | * only allow updates to the permission attributes. | 207 | * only allow updates to the permission attributes. |
| 205 | */ | 208 | */ |
| 206 | BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), | 209 | BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), |
| 207 | pmd_val(*pmd))); | 210 | READ_ONCE(pmd_val(*pmdp)))); |
| 208 | } else { | 211 | } else { |
| 209 | alloc_init_cont_pte(pmd, addr, next, phys, prot, | 212 | alloc_init_cont_pte(pmdp, addr, next, phys, prot, |
| 210 | pgtable_alloc, flags); | 213 | pgtable_alloc, flags); |
| 211 | 214 | ||
| 212 | BUG_ON(pmd_val(old_pmd) != 0 && | 215 | BUG_ON(pmd_val(old_pmd) != 0 && |
| 213 | pmd_val(old_pmd) != pmd_val(*pmd)); | 216 | pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); |
| 214 | } | 217 | } |
| 215 | phys += next - addr; | 218 | phys += next - addr; |
| 216 | } while (pmd++, addr = next, addr != end); | 219 | } while (pmdp++, addr = next, addr != end); |
| 217 | 220 | ||
| 218 | pmd_clear_fixmap(); | 221 | pmd_clear_fixmap(); |
| 219 | } | 222 | } |
| 220 | 223 | ||
| 221 | static void alloc_init_cont_pmd(pud_t *pud, unsigned long addr, | 224 | static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, |
| 222 | unsigned long end, phys_addr_t phys, | 225 | unsigned long end, phys_addr_t phys, |
| 223 | pgprot_t prot, | 226 | pgprot_t prot, |
| 224 | phys_addr_t (*pgtable_alloc)(void), int flags) | 227 | phys_addr_t (*pgtable_alloc)(void), int flags) |
| 225 | { | 228 | { |
| 226 | unsigned long next; | 229 | unsigned long next; |
| 230 | pud_t pud = READ_ONCE(*pudp); | ||
| 227 | 231 | ||
| 228 | /* | 232 | /* |
| 229 | * Check for initial section mappings in the pgd/pud. | 233 | * Check for initial section mappings in the pgd/pud. |
| 230 | */ | 234 | */ |
| 231 | BUG_ON(pud_sect(*pud)); | 235 | BUG_ON(pud_sect(pud)); |
| 232 | if (pud_none(*pud)) { | 236 | if (pud_none(pud)) { |
| 233 | phys_addr_t pmd_phys; | 237 | phys_addr_t pmd_phys; |
| 234 | BUG_ON(!pgtable_alloc); | 238 | BUG_ON(!pgtable_alloc); |
| 235 | pmd_phys = pgtable_alloc(); | 239 | pmd_phys = pgtable_alloc(); |
| 236 | __pud_populate(pud, pmd_phys, PUD_TYPE_TABLE); | 240 | __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); |
| 241 | pud = READ_ONCE(*pudp); | ||
| 237 | } | 242 | } |
| 238 | BUG_ON(pud_bad(*pud)); | 243 | BUG_ON(pud_bad(pud)); |
| 239 | 244 | ||
| 240 | do { | 245 | do { |
| 241 | pgprot_t __prot = prot; | 246 | pgprot_t __prot = prot; |
| @@ -247,7 +252,7 @@ static void alloc_init_cont_pmd(pud_t *pud, unsigned long addr, · 5 unmodified lines | |||
| 247 | (flags & NO_CONT_MAPPINGS) == 0) | 252 | (flags & NO_CONT_MAPPINGS) == 0) |
| 248 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); | 253 | __prot = __pgprot(pgprot_val(prot) | PTE_CONT); |
| 249 | 254 | ||
| 250 | init_pmd(pud, addr, next, phys, __prot, pgtable_alloc, flags); | 255 | init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); |
| 251 | 256 | ||
| 252 | phys += next - addr; | 257 | phys += next - addr; |
| 253 | } while (addr = next, addr != end); | 258 | } while (addr = next, addr != end); |
| @@ -265,25 +270,27 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, · 11 unmodified lines | |||
| 265 | return true; | 270 | return true; |
| 266 | } | 271 | } |
| 267 | 272 | ||
| 268 | static void alloc_init_pud(pgd_t *pgd, unsigned long addr, unsigned long end, | 273 | static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, |
| 269 | phys_addr_t phys, pgprot_t prot, | 274 | phys_addr_t phys, pgprot_t prot, |
| 270 | phys_addr_t (*pgtable_alloc)(void), | 275 | phys_addr_t (*pgtable_alloc)(void), |
| 271 | int flags) | 276 | int flags) |
| 272 | { | 277 | { |
| 273 | pud_t *pud; | ||
| 274 | unsigned long next; | 278 | unsigned long next; |
| 279 | pud_t *pudp; | ||
| 280 | pgd_t pgd = READ_ONCE(*pgdp); | ||
| 275 | 281 | ||
| 276 | if (pgd_none(*pgd)) { | 282 | if (pgd_none(pgd)) { |
| 277 | phys_addr_t pud_phys; | 283 | phys_addr_t pud_phys; |
| 278 | BUG_ON(!pgtable_alloc); | 284 | BUG_ON(!pgtable_alloc); |
| 279 | pud_phys = pgtable_alloc(); | 285 | pud_phys = pgtable_alloc(); |
| 280 | __pgd_populate(pgd, pud_phys, PUD_TYPE_TABLE); | 286 | __pgd_populate(pgdp, pud_phys, PUD_TYPE_TABLE); |
| 287 | pgd = READ_ONCE(*pgdp); | ||
| 281 | } | 288 | } |
| 282 | BUG_ON(pgd_bad(*pgd)); | 289 | BUG_ON(pgd_bad(pgd)); |
| 283 | 290 | ||
| 284 | pud = pud_set_fixmap_offset(pgd, addr); | 291 | pudp = pud_set_fixmap_offset(pgdp, addr); |
| 285 | do { | 292 | do { |
| 286 | pud_t old_pud = *pud; | 293 | pud_t old_pud = READ_ONCE(*pudp); |
| 287 | 294 | ||
| 288 | next = pud_addr_end(addr, end); | 295 | next = pud_addr_end(addr, end); |
| 289 | 296 | ||
| @@ -292,23 +299,23 @@ static void alloc_init_pud(pgd_t *pgd, unsigned long addr, unsigned long end, · 2 unmodified lines | |||
| 292 | */ | 299 | */ |
| 293 | if (use_1G_block(addr, next, phys) && | 300 | if (use_1G_block(addr, next, phys) && |
| 294 | (flags & NO_BLOCK_MAPPINGS) == 0) { | 301 | (flags & NO_BLOCK_MAPPINGS) == 0) { |
| 295 | pud_set_huge(pud, phys, prot); | 302 | pud_set_huge(pudp, phys, prot); |
| 296 | 303 | ||
| 297 | /* | 304 | /* |
| 298 | * After the PUD entry has been populated once, we | 305 | * After the PUD entry has been populated once, we |
| 299 | * only allow updates to the permission attributes. | 306 | * only allow updates to the permission attributes. |
| 300 | */ | 307 | */ |
| 301 | BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), | 308 | BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), |
| 302 | pud_val(*pud))); | 309 | READ_ONCE(pud_val(*pudp)))); |
| 303 | } else { | 310 | } else { |
| 304 | alloc_init_cont_pmd(pud, addr, next, phys, prot, | 311 | alloc_init_cont_pmd(pudp, addr, next, phys, prot, |
| 305 | pgtable_alloc, flags); | 312 | pgtable_alloc, flags); |
| 306 | 313 | ||
| 307 | BUG_ON(pud_val(old_pud) != 0 && | 314 | BUG_ON(pud_val(old_pud) != 0 && |
| 308 | pud_val(old_pud) != pud_val(*pud)); | 315 | pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); |
| 309 | } | 316 | } |
| 310 | phys += next - addr; | 317 | phys += next - addr; |
| 311 | } while (pud++, addr = next, addr != end); | 318 | } while (pudp++, addr = next, addr != end); |
| 312 | 319 | ||
| 313 | pud_clear_fixmap(); | 320 | pud_clear_fixmap(); |
| 314 | } | 321 | } |
| @@ -320,7 +327,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, · 5 unmodified lines | |||
| 320 | int flags) | 327 | int flags) |
| 321 | { | 328 | { |
| 322 | unsigned long addr, length, end, next; | 329 | unsigned long addr, length, end, next; |
| 323 | pgd_t *pgd = pgd_offset_raw(pgdir, virt); | 330 | pgd_t *pgdp = pgd_offset_raw(pgdir, virt); |
| 324 | 331 | ||
| 325 | /* | 332 | /* |
| 326 | * If the virtual and physical address don't have the same offset | 333 | * If the virtual and physical address don't have the same offset |
| @@ -336,10 +343,10 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, · 9 unmodified lines | |||
| 336 | end = addr + length; | 343 | end = addr + length; |
| 337 | do { | 344 | do { |
| 338 | next = pgd_addr_end(addr, end); | 345 | next = pgd_addr_end(addr, end); |
| 339 | alloc_init_pud(pgd, addr, next, phys, prot, pgtable_alloc, | 346 | alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, |
| 340 | flags); | 347 | flags); |
| 341 | phys += next - addr; | 348 | phys += next - addr; |
| 342 | } while (pgd++, addr = next, addr != end); | 349 | } while (pgdp++, addr = next, addr != end); |
| 343 | } | 350 | } |
| 344 | 351 | ||
| 345 | static phys_addr_t pgd_pgtable_alloc(void) | 352 | static phys_addr_t pgd_pgtable_alloc(void) |
| @@ -401,10 +408,10 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, · 55 unmodified lines | |||
| 401 | flush_tlb_kernel_range(virt, virt + size); | 408 | flush_tlb_kernel_range(virt, virt + size); |
| 402 | } | 409 | } |
| 403 | 410 | ||
| 404 | static void __init __map_memblock(pgd_t *pgd, phys_addr_t start, | 411 | static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, |
| 405 | phys_addr_t end, pgprot_t prot, int flags) | 412 | phys_addr_t end, pgprot_t prot, int flags) |
| 406 | { | 413 | { |
| 407 | __create_pgd_mapping(pgd, start, __phys_to_virt(start), end - start, | 414 | __create_pgd_mapping(pgdp, start, __phys_to_virt(start), end - start, |
| 408 | prot, early_pgtable_alloc, flags); | 415 | prot, early_pgtable_alloc, flags); |
| 409 | } | 416 | } |
| 410 | 417 | ||
| @@ -418,7 +425,7 @@ void __init mark_linear_text_alias_ro(void) · 7 unmodified lines | |||
| 418 | PAGE_KERNEL_RO); | 425 | PAGE_KERNEL_RO); |
| 419 | } | 426 | } |
| 420 | 427 | ||
| 421 | static void __init map_mem(pgd_t *pgd) | 428 | static void __init map_mem(pgd_t *pgdp) |
| 422 | { | 429 | { |
| 423 | phys_addr_t kernel_start = __pa_symbol(_text); | 430 | phys_addr_t kernel_start = __pa_symbol(_text); |
| 424 | phys_addr_t kernel_end = __pa_symbol(__init_begin); | 431 | phys_addr_t kernel_end = __pa_symbol(__init_begin); |
| @@ -451,7 +458,7 @@ static void __init map_mem(pgd_t *pgd) · 26 unmodified lines | |||
| 451 | if (memblock_is_nomap(reg)) | 458 | if (memblock_is_nomap(reg)) |
| 452 | continue; | 459 | continue; |
| 453 | 460 | ||
| 454 | __map_memblock(pgd, start, end, PAGE_KERNEL, flags); | 461 | __map_memblock(pgdp, start, end, PAGE_KERNEL, flags); |
| 455 | } | 462 | } |
| 456 | 463 | ||
| 457 | /* | 464 | /* |
| @@ -464,7 +471,7 @@ static void __init map_mem(pgd_t *pgd) · 6 unmodified lines | |||
| 464 | * Note that contiguous mappings cannot be remapped in this way, | 471 | * Note that contiguous mappings cannot be remapped in this way, |
| 465 | * so we should avoid them here. | 472 | * so we should avoid them here. |
| 466 | */ | 473 | */ |
| 467 | __map_memblock(pgd, kernel_start, kernel_end, | 474 | __map_memblock(pgdp, kernel_start, kernel_end, |
| 468 | PAGE_KERNEL, NO_CONT_MAPPINGS); | 475 | PAGE_KERNEL, NO_CONT_MAPPINGS); |
| 469 | memblock_clear_nomap(kernel_start, kernel_end - kernel_start); | 476 | memblock_clear_nomap(kernel_start, kernel_end - kernel_start); |
| 470 | 477 | ||
| @@ -475,7 +482,7 @@ static void __init map_mem(pgd_t *pgd) · 4 unmodified lines | |||
| 475 | * through /sys/kernel/kexec_crash_size interface. | 482 | * through /sys/kernel/kexec_crash_size interface. |
| 476 | */ | 483 | */ |
| 477 | if (crashk_res.end) { | 484 | if (crashk_res.end) { |
| 478 | __map_memblock(pgd, crashk_res.start, crashk_res.end + 1, | 485 | __map_memblock(pgdp, crashk_res.start, crashk_res.end + 1, |
| 479 | PAGE_KERNEL, | 486 | PAGE_KERNEL, |
| 480 | NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); | 487 | NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); |
| 481 | memblock_clear_nomap(crashk_res.start, | 488 | memblock_clear_nomap(crashk_res.start, |
| @@ -499,7 +506,7 @@ void mark_rodata_ro(void) · 17 unmodified lines | |||
| 499 | debug_checkwx(); | 506 | debug_checkwx(); |
| 500 | } | 507 | } |
| 501 | 508 | ||
| 502 | static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end, | 509 | static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, |
| 503 | pgprot_t prot, struct vm_struct *vma, | 510 | pgprot_t prot, struct vm_struct *vma, |
| 504 | int flags, unsigned long vm_flags) | 511 | int flags, unsigned long vm_flags) |
| 505 | { | 512 | { |
| @@ -509,7 +516,7 @@ static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end, · 3 unmodified lines | |||
| 509 | BUG_ON(!PAGE_ALIGNED(pa_start)); | 516 | BUG_ON(!PAGE_ALIGNED(pa_start)); |
| 510 | BUG_ON(!PAGE_ALIGNED(size)); | 517 | BUG_ON(!PAGE_ALIGNED(size)); |
| 511 | 518 | ||
| 512 | __create_pgd_mapping(pgd, pa_start, (unsigned long)va_start, size, prot, | 519 | __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, |
| 513 | early_pgtable_alloc, flags); | 520 | early_pgtable_alloc, flags); |
| 514 | 521 | ||
| 515 | if (!(vm_flags & VM_NO_GUARD)) | 522 | if (!(vm_flags & VM_NO_GUARD)) |
| @@ -562,7 +569,7 @@ core_initcall(map_entry_trampoline); · 46 unmodified lines | |||
| 562 | /* | 569 | /* |
| 563 | * Create fine-grained mappings for the kernel. | 570 | * Create fine-grained mappings for the kernel. |
| 564 | */ | 571 | */ |
| 565 | static void __init map_kernel(pgd_t *pgd) | 572 | static void __init map_kernel(pgd_t *pgdp) |
| 566 | { | 573 | { |
| 567 | static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, | 574 | static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, |
| 568 | vmlinux_initdata, vmlinux_data; | 575 | vmlinux_initdata, vmlinux_data; |
| @@ -578,24 +585,24 @@ static void __init map_kernel(pgd_t *pgd) · 9 unmodified lines | |||
| 578 | * Only rodata will be remapped with different permissions later on, | 585 | * Only rodata will be remapped with different permissions later on, |
| 579 | * all other segments are allowed to use contiguous mappings. | 586 | * all other segments are allowed to use contiguous mappings. |
| 580 | */ | 587 | */ |
| 581 | map_kernel_segment(pgd, _text, _etext, text_prot, &vmlinux_text, 0, | 588 | map_kernel_segment(pgdp, _text, _etext, text_prot, &vmlinux_text, 0, |
| 582 | VM_NO_GUARD); | 589 | VM_NO_GUARD); |
| 583 | map_kernel_segment(pgd, __start_rodata, __inittext_begin, PAGE_KERNEL, | 590 | map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, |
| 584 | &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); | 591 | &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); |
| 585 | map_kernel_segment(pgd, __inittext_begin, __inittext_end, text_prot, | 592 | map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, |
| 586 | &vmlinux_inittext, 0, VM_NO_GUARD); | 593 | &vmlinux_inittext, 0, VM_NO_GUARD); |
| 587 | map_kernel_segment(pgd, __initdata_begin, __initdata_end, PAGE_KERNEL, | 594 | map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, |
| 588 | &vmlinux_initdata, 0, VM_NO_GUARD); | 595 | &vmlinux_initdata, 0, VM_NO_GUARD); |
| 589 | map_kernel_segment(pgd, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); | 596 | map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); |
| 590 | 597 | ||
| 591 | if (!pgd_val(*pgd_offset_raw(pgd, FIXADDR_START))) { | 598 | if (!READ_ONCE(pgd_val(*pgd_offset_raw(pgdp, FIXADDR_START)))) { |
| 592 | /* | 599 | /* |
| 593 | * The fixmap falls in a separate pgd to the kernel, and doesn't | 600 | * The fixmap falls in a separate pgd to the kernel, and doesn't |
| 594 | * live in the carveout for the swapper_pg_dir. We can simply | 601 | * live in the carveout for the swapper_pg_dir. We can simply |
| 595 | * re-use the existing dir for the fixmap. | 602 | * re-use the existing dir for the fixmap. |
| 596 | */ | 603 | */ |
| 597 | set_pgd(pgd_offset_raw(pgd, FIXADDR_START), | 604 | set_pgd(pgd_offset_raw(pgdp, FIXADDR_START), |
| 598 | *pgd_offset_k(FIXADDR_START)); | 605 | READ_ONCE(*pgd_offset_k(FIXADDR_START))); |
| 599 | } else if (CONFIG_PGTABLE_LEVELS > 3) { | 606 | } else if (CONFIG_PGTABLE_LEVELS > 3) { |
| 600 | /* | 607 | /* |
| 601 | * The fixmap shares its top level pgd entry with the kernel | 608 | * The fixmap shares its top level pgd entry with the kernel |
| @@ -604,14 +611,15 @@ static void __init map_kernel(pgd_t *pgd) · 2 unmodified lines | |||
| 604 | * entry instead. | 611 | * entry instead. |
| 605 | */ | 612 | */ |
| 606 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); | 613 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); |
| 607 | pud_populate(&init_mm, pud_set_fixmap_offset(pgd, FIXADDR_START), | 614 | pud_populate(&init_mm, |
| 615 | pud_set_fixmap_offset(pgdp, FIXADDR_START), | ||
| 608 | lm_alias(bm_pmd)); | 616 | lm_alias(bm_pmd)); |
| 609 | pud_clear_fixmap(); | 617 | pud_clear_fixmap(); |
| 610 | } else { | 618 | } else { |
| 611 | BUG(); | 619 | BUG(); |
| 612 | } | 620 | } |
| 613 | 621 | ||
| 614 | kasan_copy_shadow(pgd); | 622 | kasan_copy_shadow(pgdp); |
| 615 | } | 623 | } |
| 616 | 624 | ||
| 617 | /* | 625 | /* |
| @@ -621,10 +629,10 @@ static void __init map_kernel(pgd_t *pgd) · 3 unmodified lines | |||
| 621 | void __init paging_init(void) | 629 | void __init paging_init(void) |
| 622 | { | 630 | { |
| 623 | phys_addr_t pgd_phys = early_pgtable_alloc(); | 631 | phys_addr_t pgd_phys = early_pgtable_alloc(); |
| 624 | pgd_t *pgd = pgd_set_fixmap(pgd_phys); | 632 | pgd_t *pgdp = pgd_set_fixmap(pgd_phys); |
| 625 | 633 | ||
| 626 | map_kernel(pgd); | 634 | map_kernel(pgdp); |
| 627 | map_mem(pgd); | 635 | map_mem(pgdp); |
| 628 | 636 | ||
| 629 | /* | 637 | /* |
| 630 | * We want to reuse the original swapper_pg_dir so we don't have to | 638 | * We want to reuse the original swapper_pg_dir so we don't have to |
| @@ -635,7 +643,7 @@ void __init paging_init(void) · 4 unmodified lines | |||
| 635 | * To do this we need to go via a temporary pgd. | 643 | * To do this we need to go via a temporary pgd. |
| 636 | */ | 644 | */ |
| 637 | cpu_replace_ttbr1(__va(pgd_phys)); | 645 | cpu_replace_ttbr1(__va(pgd_phys)); |
| 638 | memcpy(swapper_pg_dir, pgd, PGD_SIZE); | 646 | memcpy(swapper_pg_dir, pgdp, PGD_SIZE); |
| 639 | cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); | 647 | cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); |
| 640 | 648 | ||
| 641 | pgd_clear_fixmap(); | 649 | pgd_clear_fixmap(); |
| @@ -655,37 +663,40 @@ void __init paging_init(void) · 13 unmodified lines | |||
| 655 | */ | 663 | */ |
| 656 | int kern_addr_valid(unsigned long addr) | 664 | int kern_addr_valid(unsigned long addr) |
| 657 | { | 665 | { |
| 658 | pgd_t *pgd; | 666 | pgd_t *pgdp; |
| 659 | pud_t *pud; | 667 | pud_t *pudp, pud; |
| 660 | pmd_t *pmd; | 668 | pmd_t *pmdp, pmd; |
| 661 | pte_t *pte; | 669 | pte_t *ptep, pte; |
| 662 | 670 | ||
| 663 | if ((((long)addr) >> VA_BITS) != -1UL) | 671 | if ((((long)addr) >> VA_BITS) != -1UL) |
| 664 | return 0; | 672 | return 0; |
| 665 | 673 | ||
| 666 | pgd = pgd_offset_k(addr); | 674 | pgdp = pgd_offset_k(addr); |
| 667 | if (pgd_none(*pgd)) | 675 | if (pgd_none(READ_ONCE(*pgdp))) |
| 668 | return 0; | 676 | return 0; |
| 669 | 677 | ||
| 670 | pud = pud_offset(pgd, addr); | 678 | pudp = pud_offset(pgdp, addr); |
| 671 | if (pud_none(*pud)) | 679 | pud = READ_ONCE(*pudp); |
| 680 | if (pud_none(pud)) | ||
| 672 | return 0; | 681 | return 0; |
| 673 | 682 | ||
| 674 | if (pud_sect(*pud)) | 683 | if (pud_sect(pud)) |
| 675 | return pfn_valid(pud_pfn(*pud)); | 684 | return pfn_valid(pud_pfn(pud)); |
| 676 | 685 | ||
| 677 | pmd = pmd_offset(pud, addr); | 686 | pmdp = pmd_offset(pudp, addr); |
| 678 | if (pmd_none(*pmd)) | 687 | pmd = READ_ONCE(*pmdp); |
| 688 | if (pmd_none(pmd)) | ||
| 679 | return 0; | 689 | return 0; |
| 680 | 690 | ||
| 681 | if (pmd_sect(*pmd)) | 691 | if (pmd_sect(pmd)) |
| 682 | return pfn_valid(pmd_pfn(*pmd)); | 692 | return pfn_valid(pmd_pfn(pmd)); |
| 683 | 693 | ||
| 684 | pte = pte_offset_kernel(pmd, addr); | 694 | ptep = pte_offset_kernel(pmdp, addr); |
| 685 | if (pte_none(*pte)) | 695 | pte = READ_ONCE(*ptep); |
| 696 | if (pte_none(pte)) | ||
| 686 | return 0; | 697 | return 0; |
| 687 | 698 | ||
| 688 | return pfn_valid(pte_pfn(*pte)); | 699 | return pfn_valid(pte_pfn(pte)); |
| 689 | } | 700 | } |
| 690 | #ifdef CONFIG_SPARSEMEM_VMEMMAP | 701 | #ifdef CONFIG_SPARSEMEM_VMEMMAP |
| 691 | #if !ARM64_SWAPPER_USES_SECTION_MAPS | 702 | #if !ARM64_SWAPPER_USES_SECTION_MAPS |
| @@ -700,32 +711,32 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, · 8 unmodified lines | |||
| 700 | { | 711 | { |
| 701 | unsigned long addr = start; | 712 | unsigned long addr = start; |
| 702 | unsigned long next; | 713 | unsigned long next; |
| 703 | pgd_t *pgd; | 714 | pgd_t *pgdp; |
| 704 | pud_t *pud; | 715 | pud_t *pudp; |
| 705 | pmd_t *pmd; | 716 | pmd_t *pmdp; |
| 706 | 717 | ||
| 707 | do { | 718 | do { |
| 708 | next = pmd_addr_end(addr, end); | 719 | next = pmd_addr_end(addr, end); |
| 709 | 720 | ||
| 710 | pgd = vmemmap_pgd_populate(addr, node); | 721 | pgdp = vmemmap_pgd_populate(addr, node); |
| 711 | if (!pgd) | 722 | if (!pgdp) |
| 712 | return -ENOMEM; | 723 | return -ENOMEM; |
| 713 | 724 | ||
| 714 | pud = vmemmap_pud_populate(pgd, addr, node); | 725 | pudp = vmemmap_pud_populate(pgdp, addr, node); |
| 715 | if (!pud) | 726 | if (!pudp) |
| 716 | return -ENOMEM; | 727 | return -ENOMEM; |
| 717 | 728 | ||
| 718 | pmd = pmd_offset(pud, addr); | 729 | pmdp = pmd_offset(pudp, addr); |
| 719 | if (pmd_none(*pmd)) { | 730 | if (pmd_none(READ_ONCE(*pmdp))) { |
| 720 | void *p = NULL; | 731 | void *p = NULL; |
| 721 | 732 | ||
| 722 | p = vmemmap_alloc_block_buf(PMD_SIZE, node); | 733 | p = vmemmap_alloc_block_buf(PMD_SIZE, node); |
| 723 | if (!p) | 734 | if (!p) |
| 724 | return -ENOMEM; | 735 | return -ENOMEM; |
| 725 | 736 | ||
| 726 | pmd_set_huge(pmd, __pa(p), __pgprot(PROT_SECT_NORMAL)); | 737 | pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL)); |
| 727 | } else | 738 | } else |
| 728 | vmemmap_verify((pte_t *)pmd, node, addr, next); | 739 | vmemmap_verify((pte_t *)pmdp, node, addr, next); |
| 729 | } while (addr = next, addr != end); | 740 | } while (addr = next, addr != end); |
| 730 | 741 | ||
| 731 | return 0; | 742 | return 0; |
| @@ -739,20 +750,22 @@ void vmemmap_free(unsigned long start, unsigned long end, · 7 unmodified lines | |||
| 739 | 750 | ||
| 740 | static inline pud_t * fixmap_pud(unsigned long addr) | 751 | static inline pud_t * fixmap_pud(unsigned long addr) |
| 741 | { | 752 | { |
| 742 | pgd_t *pgd = pgd_offset_k(addr); | 753 | pgd_t *pgdp = pgd_offset_k(addr); |
| 754 | pgd_t pgd = READ_ONCE(*pgdp); | ||
| 743 | 755 | ||
| 744 | BUG_ON(pgd_none(*pgd) || pgd_bad(*pgd)); | 756 | BUG_ON(pgd_none(pgd) || pgd_bad(pgd)); |
| 745 | 757 | ||
| 746 | return pud_offset_kimg(pgd, addr); | 758 | return pud_offset_kimg(pgdp, addr); |
| 747 | } | 759 | } |
| 748 | 760 | ||
| 749 | static inline pmd_t * fixmap_pmd(unsigned long addr) | 761 | static inline pmd_t * fixmap_pmd(unsigned long addr) |
| 750 | { | 762 | { |
| 751 | pud_t *pud = fixmap_pud(addr); | 763 | pud_t *pudp = fixmap_pud(addr); |
| 764 | pud_t pud = READ_ONCE(*pudp); | ||
| 752 | 765 | ||
| 753 | BUG_ON(pud_none(*pud) || pud_bad(*pud)); | 766 | BUG_ON(pud_none(pud) || pud_bad(pud)); |
| 754 | 767 | ||
| 755 | return pmd_offset_kimg(pud, addr); | 768 | return pmd_offset_kimg(pudp, addr); |
| 756 | } | 769 | } |
| 757 | 770 | ||
| 758 | static inline pte_t * fixmap_pte(unsigned long addr) | 771 | static inline pte_t * fixmap_pte(unsigned long addr) |
| @@ -768,30 +781,31 @@ static inline pte_t * fixmap_pte(unsigned long addr) · 9 unmodified lines | |||
| 768 | */ | 781 | */ |
| 769 | void __init early_fixmap_init(void) | 782 | void __init early_fixmap_init(void) |
| 770 | { | 783 | { |
| 771 | pgd_t *pgd; | 784 | pgd_t *pgdp, pgd; |
| 772 | pud_t *pud; | 785 | pud_t *pudp; |
| 773 | pmd_t *pmd; | 786 | pmd_t *pmdp; |
| 774 | unsigned long addr = FIXADDR_START; | 787 | unsigned long addr = FIXADDR_START; |
| 775 | 788 | ||
| 776 | pgd = pgd_offset_k(addr); | 789 | pgdp = pgd_offset_k(addr); |
| 790 | pgd = READ_ONCE(*pgdp); | ||
| 777 | if (CONFIG_PGTABLE_LEVELS > 3 && | 791 | if (CONFIG_PGTABLE_LEVELS > 3 && |
| 778 | !(pgd_none(*pgd) || pgd_page_paddr(*pgd) == __pa_symbol(bm_pud))) { | 792 | !(pgd_none(pgd) || pgd_page_paddr(pgd) == __pa_symbol(bm_pud))) { |
| 779 | /* | 793 | /* |
| 780 | * We only end up here if the kernel mapping and the fixmap | 794 | * We only end up here if the kernel mapping and the fixmap |
| 781 | * share the top level pgd entry, which should only happen on | 795 | * share the top level pgd entry, which should only happen on |
| 782 | * 16k/4 levels configurations. | 796 | * 16k/4 levels configurations. |
| 783 | */ | 797 | */ |
| 784 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); | 798 | BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); |
| 785 | pud = pud_offset_kimg(pgd, addr); | 799 | pudp = pud_offset_kimg(pgdp, addr); |
| 786 | } else { | 800 | } else { |
| 787 | if (pgd_none(*pgd)) | 801 | if (pgd_none(pgd)) |
| 788 | __pgd_populate(pgd, __pa_symbol(bm_pud), PUD_TYPE_TABLE); | 802 | __pgd_populate(pgdp, __pa_symbol(bm_pud), PUD_TYPE_TABLE); |
| 789 | pud = fixmap_pud(addr); | 803 | pudp = fixmap_pud(addr); |
| 790 | } | 804 | } |
| 791 | if (pud_none(*pud)) | 805 | if (pud_none(READ_ONCE(*pudp))) |
| 792 | __pud_populate(pud, __pa_symbol(bm_pmd), PMD_TYPE_TABLE); | 806 | __pud_populate(pudp, __pa_symbol(bm_pmd), PMD_TYPE_TABLE); |
| 793 | pmd = fixmap_pmd(addr); | 807 | pmdp = fixmap_pmd(addr); |
| 794 | __pmd_populate(pmd, __pa_symbol(bm_pte), PMD_TYPE_TABLE); | 808 | __pmd_populate(pmdp, __pa_symbol(bm_pte), PMD_TYPE_TABLE); |
| 795 | 809 | ||
| 796 | /* | 810 | /* |
| 797 | * The boot-ioremap range spans multiple pmds, for which | 811 | * The boot-ioremap range spans multiple pmds, for which |
| @@ -800,11 +814,11 @@ void __init early_fixmap_init(void) · 2 unmodified lines | |||
| 800 | BUILD_BUG_ON((__fix_to_virt(FIX_BTMAP_BEGIN) >> PMD_SHIFT) | 814 | BUILD_BUG_ON((__fix_to_virt(FIX_BTMAP_BEGIN) >> PMD_SHIFT) |
| 801 | != (__fix_to_virt(FIX_BTMAP_END) >> PMD_SHIFT)); | 815 | != (__fix_to_virt(FIX_BTMAP_END) >> PMD_SHIFT)); |
| 802 | 816 | ||
| 803 | if ((pmd != fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN))) | 817 | if ((pmdp != fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN))) |
| 804 | || pmd != fixmap_pmd(fix_to_virt(FIX_BTMAP_END))) { | 818 | || pmdp != fixmap_pmd(fix_to_virt(FIX_BTMAP_END))) { |
| 805 | WARN_ON(1); | 819 | WARN_ON(1); |
| 806 | pr_warn("pmd %p != %p, %p\n", | 820 | pr_warn("pmdp %p != %p, %p\n", |
| 807 | pmd, fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN)), | 821 | pmdp, fixmap_pmd(fix_to_virt(FIX_BTMAP_BEGIN)), |
| 808 | fixmap_pmd(fix_to_virt(FIX_BTMAP_END))); | 822 | fixmap_pmd(fix_to_virt(FIX_BTMAP_END))); |
| 809 | pr_warn("fix_to_virt(FIX_BTMAP_BEGIN): %08lx\n", | 823 | pr_warn("fix_to_virt(FIX_BTMAP_BEGIN): %08lx\n", |
| 810 | fix_to_virt(FIX_BTMAP_BEGIN)); | 824 | fix_to_virt(FIX_BTMAP_BEGIN)); |
| @@ -824,16 +838,16 @@ void __set_fixmap(enum fixed_addresses idx, · 13 unmodified lines | |||
| 824 | phys_addr_t phys, pgprot_t flags) | 838 | phys_addr_t phys, pgprot_t flags) |
| 825 | { | 839 | { |
| 826 | unsigned long addr = __fix_to_virt(idx); | 840 | unsigned long addr = __fix_to_virt(idx); |
| 827 | pte_t *pte; | 841 | pte_t *ptep; |
| 828 | 842 | ||
| 829 | BUG_ON(idx <= FIX_HOLE || idx >= __end_of_fixed_addresses); | 843 | BUG_ON(idx <= FIX_HOLE || idx >= __end_of_fixed_addresses); |
| 830 | 844 | ||
| 831 | pte = fixmap_pte(addr); | 845 | ptep = fixmap_pte(addr); |
| 832 | 846 | ||
| 833 | if (pgprot_val(flags)) { | 847 | if (pgprot_val(flags)) { |
| 834 | set_pte(pte, pfn_pte(phys >> PAGE_SHIFT, flags)); | 848 | set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); |
| 835 | } else { | 849 | } else { |
| 836 | pte_clear(&init_mm, addr, pte); | 850 | pte_clear(&init_mm, addr, ptep); |
| 837 | flush_tlb_kernel_range(addr, addr+PAGE_SIZE); | 851 | flush_tlb_kernel_range(addr, addr+PAGE_SIZE); |
| 838 | } | 852 | } |
| 839 | } | 853 | } |
| @@ -915,36 +929,46 @@ int __init arch_ioremap_pmd_supported(void) · 75 unmodified lines | |||
| 915 | return 1; | 929 | return 1; |
| 916 | } | 930 | } |
| 917 | 931 | ||
| 918 | int pud_set_huge(pud_t *pud, phys_addr_t phys, pgprot_t prot) | 932 | int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) |
| 919 | { | 933 | { |
| 920 | pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT | | 934 | pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT | |
| 921 | pgprot_val(mk_sect_prot(prot))); | 935 | pgprot_val(mk_sect_prot(prot))); |
| 936 | |||
| 937 | /* ioremap_page_range doesn't honour BBM */ | ||
| 938 | if (pud_present(READ_ONCE(*pudp))) | ||
| 939 | return 0; | ||
| 940 | |||
| 922 | BUG_ON(phys & ~PUD_MASK); | 941 | BUG_ON(phys & ~PUD_MASK); |
| 923 | set_pud(pud, pfn_pud(__phys_to_pfn(phys), sect_prot)); | 942 | set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot)); |
| 924 | return 1; | 943 | return 1; |
| 925 | } | 944 | } |
| 926 | 945 | ||
| 927 | int pmd_set_huge(pmd_t *pmd, phys_addr_t phys, pgprot_t prot) | 946 | int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) |
| 928 | { | 947 | { |
| 929 | pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT | | 948 | pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT | |
| 930 | pgprot_val(mk_sect_prot(prot))); | 949 | pgprot_val(mk_sect_prot(prot))); |
| 950 | |||
| 951 | /* ioremap_page_range doesn't honour BBM */ | ||
| 952 | if (pmd_present(READ_ONCE(*pmdp))) | ||
| 953 | return 0; | ||
| 954 | |||
| 931 | BUG_ON(phys & ~PMD_MASK); | 955 | BUG_ON(phys & ~PMD_MASK); |
| 932 | set_pmd(pmd, pfn_pmd(__phys_to_pfn(phys), sect_prot)); | 956 | set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot)); |
| 933 | return 1; | 957 | return 1; |
| 934 | } | 958 | } |
| 935 | 959 | ||
| 936 | int pud_clear_huge(pud_t *pud) | 960 | int pud_clear_huge(pud_t *pudp) |
| 937 | { | 961 | { |
| 938 | if (!pud_sect(*pud)) | 962 | if (!pud_sect(READ_ONCE(*pudp))) |
| 939 | return 0; | 963 | return 0; |
| 940 | pud_clear(pud); | 964 | pud_clear(pudp); |
| 941 | return 1; | 965 | return 1; |
| 942 | } | 966 | } |
| 943 | 967 | ||
| 944 | int pmd_clear_huge(pmd_t *pmd) | 968 | int pmd_clear_huge(pmd_t *pmdp) |
| 945 | { | 969 | { |
| 946 | if (!pmd_sect(*pmd)) | 970 | if (!pmd_sect(READ_ONCE(*pmdp))) |
| 947 | return 0; | 971 | return 0; |
| 948 | pmd_clear(pmd); | 972 | pmd_clear(pmdp); |
| 949 | return 1; | 973 | return 1; |
| 950 | } | 974 | } |
| @@ -29,7 +29,7 @@ static int change_page_range(pte_t *ptep, pgtable_t token, unsigned long addr, · 28 unmodified lines | |||
| 29 | void *data) | 29 | void *data) |
| 30 | { | 30 | { |
| 31 | struct page_change_data *cdata = data; | 31 | struct page_change_data *cdata = data; |
| 32 | pte_t pte = *ptep; | 32 | pte_t pte = READ_ONCE(*ptep); |
| 33 | 33 | ||
| 34 | pte = clear_pte_bit(pte, cdata->clear_mask); | 34 | pte = clear_pte_bit(pte, cdata->clear_mask); |
| 35 | pte = set_pte_bit(pte, cdata->set_mask); | 35 | pte = set_pte_bit(pte, cdata->set_mask); |
| @@ -156,30 +156,32 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) · 120 unmodified lines | |||
| 156 | */ | 156 | */ |
| 157 | bool kernel_page_present(struct page *page) | 157 | bool kernel_page_present(struct page *page) |
| 158 | { | 158 | { |
| 159 | pgd_t *pgd; | 159 | pgd_t *pgdp; |
| 160 | pud_t *pud; | 160 | pud_t *pudp, pud; |
| 161 | pmd_t *pmd; | 161 | pmd_t *pmdp, pmd; |
| 162 | pte_t *pte; | 162 | pte_t *ptep; |
| 163 | unsigned long addr = (unsigned long)page_address(page); | 163 | unsigned long addr = (unsigned long)page_address(page); |
| 164 | 164 | ||
| 165 | pgd = pgd_offset_k(addr); | 165 | pgdp = pgd_offset_k(addr); |
| 166 | if (pgd_none(*pgd)) | 166 | if (pgd_none(READ_ONCE(*pgdp))) |
| 167 | return false; | 167 | return false; |
| 168 | 168 | ||
| 169 | pud = pud_offset(pgd, addr); | 169 | pudp = pud_offset(pgdp, addr); |
| 170 | if (pud_none(*pud)) | 170 | pud = READ_ONCE(*pudp); |
| 171 | if (pud_none(pud)) | ||
| 171 | return false; | 172 | return false; |
| 172 | if (pud_sect(*pud)) | 173 | if (pud_sect(pud)) |
| 173 | return true; | 174 | return true; |
| 174 | 175 | ||
| 175 | pmd = pmd_offset(pud, addr); | 176 | pmdp = pmd_offset(pudp, addr); |
| 176 | if (pmd_none(*pmd)) | 177 | pmd = READ_ONCE(*pmdp); |
| 178 | if (pmd_none(pmd)) | ||
| 177 | return false; | 179 | return false; |
| 178 | if (pmd_sect(*pmd)) | 180 | if (pmd_sect(pmd)) |
| 179 | return true; | 181 | return true; |
| 180 | 182 | ||
| 181 | pte = pte_offset_kernel(pmd, addr); | 183 | ptep = pte_offset_kernel(pmdp, addr); |
| 182 | return pte_valid(*pte); | 184 | return pte_valid(READ_ONCE(*ptep)); |
| 183 | } | 185 | } |
| 184 | #endif /* CONFIG_HIBERNATION */ | 186 | #endif /* CONFIG_HIBERNATION */ |
| 185 | #endif /* CONFIG_DEBUG_PAGEALLOC */ | 187 | #endif /* CONFIG_DEBUG_PAGEALLOC */ |
| @@ -205,7 +205,8 @@ ENDPROC(idmap_cpu_replace_ttbr1) · 204 unmodified lines | |||
| 205 | dc cvac, cur_\()\type\()p // Ensure any existing dirty | 205 | dc cvac, cur_\()\type\()p // Ensure any existing dirty |
| 206 | dmb sy // lines are written back before | 206 | dmb sy // lines are written back before |
| 207 | ldr \type, [cur_\()\type\()p] // loading the entry | 207 | ldr \type, [cur_\()\type\()p] // loading the entry |
| 208 | tbz \type, #0, next_\()\type // Skip invalid entries | 208 | tbz \type, #0, skip_\()\type // Skip invalid and |
| 209 | tbnz \type, #11, skip_\()\type // non-global entries | ||
| 209 | .endm | 210 | .endm |
| 210 | 211 | ||
| 211 | .macro __idmap_kpti_put_pgtable_ent_ng, type | 212 | .macro __idmap_kpti_put_pgtable_ent_ng, type |
| @@ -265,8 +266,9 @@ ENTRY(idmap_kpti_install_ng_mappings) · 53 unmodified lines | |||
| 265 | add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) | 266 | add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) |
| 266 | do_pgd: __idmap_kpti_get_pgtable_ent pgd | 267 | do_pgd: __idmap_kpti_get_pgtable_ent pgd |
| 267 | tbnz pgd, #1, walk_puds | 268 | tbnz pgd, #1, walk_puds |
| 268 | __idmap_kpti_put_pgtable_ent_ng pgd | ||
| 269 | next_pgd: | 269 | next_pgd: |
| 270 | __idmap_kpti_put_pgtable_ent_ng pgd | ||
| 271 | skip_pgd: | ||
| 270 | add cur_pgdp, cur_pgdp, #8 | 272 | add cur_pgdp, cur_pgdp, #8 |
| 271 | cmp cur_pgdp, end_pgdp | 273 | cmp cur_pgdp, end_pgdp |
| 272 | b.ne do_pgd | 274 | b.ne do_pgd |
| @@ -294,8 +296,9 @@ walk_puds: · 21 unmodified lines | |||
| 294 | add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) | 296 | add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) |
| 295 | do_pud: __idmap_kpti_get_pgtable_ent pud | 297 | do_pud: __idmap_kpti_get_pgtable_ent pud |
| 296 | tbnz pud, #1, walk_pmds | 298 | tbnz pud, #1, walk_pmds |
| 297 | __idmap_kpti_put_pgtable_ent_ng pud | ||
| 298 | next_pud: | 299 | next_pud: |
| 300 | __idmap_kpti_put_pgtable_ent_ng pud | ||
| 301 | skip_pud: | ||
| 299 | add cur_pudp, cur_pudp, 8 | 302 | add cur_pudp, cur_pudp, 8 |
| 300 | cmp cur_pudp, end_pudp | 303 | cmp cur_pudp, end_pudp |
| 301 | b.ne do_pud | 304 | b.ne do_pud |
| @@ -314,8 +317,9 @@ walk_pmds: · 12 unmodified lines | |||
| 314 | add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) | 317 | add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) |
| 315 | do_pmd: __idmap_kpti_get_pgtable_ent pmd | 318 | do_pmd: __idmap_kpti_get_pgtable_ent pmd |
| 316 | tbnz pmd, #1, walk_ptes | 319 | tbnz pmd, #1, walk_ptes |
| 317 | __idmap_kpti_put_pgtable_ent_ng pmd | ||
| 318 | next_pmd: | 320 | next_pmd: |
| 321 | __idmap_kpti_put_pgtable_ent_ng pmd | ||
| 322 | skip_pmd: | ||
| 319 | add cur_pmdp, cur_pmdp, #8 | 323 | add cur_pmdp, cur_pmdp, #8 |
| 320 | cmp cur_pmdp, end_pmdp | 324 | cmp cur_pmdp, end_pmdp |
| 321 | b.ne do_pmd | 325 | b.ne do_pmd |
| @@ -333,7 +337,7 @@ walk_ptes: · 11 unmodified lines | |||
| 333 | add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) | 337 | add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) |
| 334 | do_pte: __idmap_kpti_get_pgtable_ent pte | 338 | do_pte: __idmap_kpti_get_pgtable_ent pte |
| 335 | __idmap_kpti_put_pgtable_ent_ng pte | 339 | __idmap_kpti_put_pgtable_ent_ng pte |
| 336 | next_pte: | 340 | skip_pte: |
| 337 | add cur_ptep, cur_ptep, #8 | 341 | add cur_ptep, cur_ptep, #8 |
| 338 | cmp cur_ptep, end_ptep | 342 | cmp cur_ptep, end_ptep |
| 339 | b.ne do_pte | 343 | b.ne do_pte |
| @@ -250,8 +250,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 249 unmodified lines | |||
| 250 | off = offsetof(struct bpf_array, map.max_entries); | 250 | off = offsetof(struct bpf_array, map.max_entries); |
| 251 | emit_a64_mov_i64(tmp, off, ctx); | 251 | emit_a64_mov_i64(tmp, off, ctx); |
| 252 | emit(A64_LDR32(tmp, r2, tmp), ctx); | 252 | emit(A64_LDR32(tmp, r2, tmp), ctx); |
| 253 | emit(A64_MOV(0, r3, r3), ctx); | ||
| 253 | emit(A64_CMP(0, r3, tmp), ctx); | 254 | emit(A64_CMP(0, r3, tmp), ctx); |
| 254 | emit(A64_B_(A64_COND_GE, jmp_offset), ctx); | 255 | emit(A64_B_(A64_COND_CS, jmp_offset), ctx); |
| 255 | 256 | ||
| 256 | /* if (tail_call_cnt > MAX_TAIL_CALL_CNT) | 257 | /* if (tail_call_cnt > MAX_TAIL_CALL_CNT) |
| 257 | * goto out; | 258 | * goto out; |
| @@ -259,7 +260,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 1 unmodified line | |||
| 259 | */ | 260 | */ |
| 260 | emit_a64_mov_i64(tmp, MAX_TAIL_CALL_CNT, ctx); | 261 | emit_a64_mov_i64(tmp, MAX_TAIL_CALL_CNT, ctx); |
| 261 | emit(A64_CMP(1, tcc, tmp), ctx); | 262 | emit(A64_CMP(1, tcc, tmp), ctx); |
| 262 | emit(A64_B_(A64_COND_GT, jmp_offset), ctx); | 263 | emit(A64_B_(A64_COND_HI, jmp_offset), ctx); |
| 263 | emit(A64_ADD_I(1, tcc, tcc, 1), ctx); | 264 | emit(A64_ADD_I(1, tcc, tcc, 1), ctx); |
| 264 | 265 | ||
| 265 | /* prog = array->ptrs[index]; | 266 | /* prog = array->ptrs[index]; |
| @@ -44,18 +44,25 @@ struct bug_frame { · 43 unmodified lines | |||
| 44 | * not be used like this with newer versions of gcc. | 44 | * not be used like this with newer versions of gcc. |
| 45 | */ | 45 | */ |
| 46 | #define BUG() \ | 46 | #define BUG() \ |
| 47 | do { \ | ||
| 47 | __asm__ __volatile__ ("clear.d [" __stringify(BUG_MAGIC) "]\n\t"\ | 48 | __asm__ __volatile__ ("clear.d [" __stringify(BUG_MAGIC) "]\n\t"\ |
| 48 | "movu.w " __stringify(__LINE__) ",$r0\n\t"\ | 49 | "movu.w " __stringify(__LINE__) ",$r0\n\t"\ |
| 49 | "jump 0f\n\t" \ | 50 | "jump 0f\n\t" \ |
| 50 | ".section .rodata\n" \ | 51 | ".section .rodata\n" \ |
| 51 | "0:\t.string \"" __FILE__ "\"\n\t" \ | 52 | "0:\t.string \"" __FILE__ "\"\n\t" \ |
| 52 | ".previous") | 53 | ".previous"); \ |
| 54 | unreachable(); \ | ||
| 55 | } while (0) | ||
| 53 | #endif | 56 | #endif |
| 54 | 57 | ||
| 55 | #else | 58 | #else |
| 56 | 59 | ||
| 57 | /* This just causes an oops. */ | 60 | /* This just causes an oops. */ |
| 58 | #define BUG() (*(int *)0 = 0) | 61 | #define BUG() \ |
| 62 | do { \ | ||
| 63 | barrier_before_unreachable(); \ | ||
| 64 | __builtin_trap(); \ | ||
| 65 | } while (0) | ||
| 59 | 66 | ||
| 60 | #endif | 67 | #endif |
| 61 | 68 | ||
| @@ -66,38 +66,35 @@ ATOMIC_OPS(add, +) · 65 unmodified lines | |||
| 66 | ATOMIC_OPS(sub, -) | 66 | ATOMIC_OPS(sub, -) |
| 67 | 67 | ||
| 68 | #ifdef __OPTIMIZE__ | 68 | #ifdef __OPTIMIZE__ |
| 69 | #define __ia64_atomic_const(i) __builtin_constant_p(i) ? \ | 69 | #define __ia64_atomic_const(i) \ |
| 70 | static const int __ia64_atomic_p = __builtin_constant_p(i) ? \ | ||
| 70 | ((i) == 1 || (i) == 4 || (i) == 8 || (i) == 16 || \ | 71 | ((i) == 1 || (i) == 4 || (i) == 8 || (i) == 16 || \ |
| 71 | (i) == -1 || (i) == -4 || (i) == -8 || (i) == -16) : 0 | 72 | (i) == -1 || (i) == -4 || (i) == -8 || (i) == -16) : 0;\ |
| 73 | __ia64_atomic_p | ||
| 74 | #else | ||
| 75 | #define __ia64_atomic_const(i) 0 | ||
| 76 | #endif | ||
| 72 | 77 | ||
| 73 | #define atomic_add_return(i, v) \ | 78 | #define atomic_add_return(i,v) \ |
| 74 | ({ \ | 79 | ({ \ |
| 75 | int __i = (i); \ | 80 | int __ia64_aar_i = (i); \ |
| 76 | static const int __ia64_atomic_p = __ia64_atomic_const(i); \ | 81 | __ia64_atomic_const(i) \ |
| 77 | __ia64_atomic_p ? ia64_fetch_and_add(__i, &(v)->counter) : \ | 82 | ? ia64_fetch_and_add(__ia64_aar_i, &(v)->counter) \ |
| 78 | ia64_atomic_add(__i, v); \ | 83 | : ia64_atomic_add(__ia64_aar_i, v); \ |
| 79 | }) | 84 | }) |
| 80 | 85 | ||
| 81 | #define atomic_sub_return(i, v) \ | 86 | #define atomic_sub_return(i,v) \ |
| 82 | ({ \ | 87 | ({ \ |
| 83 | int __i = (i); \ | 88 | int __ia64_asr_i = (i); \ |
| 84 | static const int __ia64_atomic_p = __ia64_atomic_const(i); \ | 89 | __ia64_atomic_const(i) \ |
| 85 | __ia64_atomic_p ? ia64_fetch_and_add(-__i, &(v)->counter) : \ | 90 | ? ia64_fetch_and_add(-__ia64_asr_i, &(v)->counter) \ |
| 86 | ia64_atomic_sub(__i, v); \ | 91 | : ia64_atomic_sub(__ia64_asr_i, v); \ |
| 87 | }) | 92 | }) |
| 88 | #else | ||
| 89 | #define atomic_add_return(i, v) ia64_atomic_add(i, v) | ||
| 90 | #define atomic_sub_return(i, v) ia64_atomic_sub(i, v) | ||
| 91 | #endif | ||
| 92 | 93 | ||
| 93 | #define atomic_fetch_add(i,v) \ | 94 | #define atomic_fetch_add(i,v) \ |
| 94 | ({ \ | 95 | ({ \ |
| 95 | int __ia64_aar_i = (i); \ | 96 | int __ia64_aar_i = (i); \ |
| 96 | (__builtin_constant_p(i) \ | 97 | __ia64_atomic_const(i) \ |
| 97 | && ( (__ia64_aar_i == 1) || (__ia64_aar_i == 4) \ | ||
| 98 | || (__ia64_aar_i == 8) || (__ia64_aar_i == 16) \ | ||
| 99 | || (__ia64_aar_i == -1) || (__ia64_aar_i == -4) \ | ||
| 100 | || (__ia64_aar_i == -8) || (__ia64_aar_i == -16))) \ | ||
| 101 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ | 98 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ |
| 102 | : ia64_atomic_fetch_add(__ia64_aar_i, v); \ | 99 | : ia64_atomic_fetch_add(__ia64_aar_i, v); \ |
| 103 | }) | 100 | }) |
| @@ -105,11 +102,7 @@ ATOMIC_OPS(sub, -) · 1 unmodified line | |||
| 105 | #define atomic_fetch_sub(i,v) \ | 102 | #define atomic_fetch_sub(i,v) \ |
| 106 | ({ \ | 103 | ({ \ |
| 107 | int __ia64_asr_i = (i); \ | 104 | int __ia64_asr_i = (i); \ |
| 108 | (__builtin_constant_p(i) \ | 105 | __ia64_atomic_const(i) \ |
| 109 | && ( (__ia64_asr_i == 1) || (__ia64_asr_i == 4) \ | ||
| 110 | || (__ia64_asr_i == 8) || (__ia64_asr_i == 16) \ | ||
| 111 | || (__ia64_asr_i == -1) || (__ia64_asr_i == -4) \ | ||
| 112 | || (__ia64_asr_i == -8) || (__ia64_asr_i == -16))) \ | ||
| 113 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ | 106 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ |
| 114 | : ia64_atomic_fetch_sub(__ia64_asr_i, v); \ | 107 | : ia64_atomic_fetch_sub(__ia64_asr_i, v); \ |
| 115 | }) | 108 | }) |
| @@ -170,11 +163,7 @@ ATOMIC64_OPS(sub, -) · 54 unmodified lines | |||
| 170 | #define atomic64_add_return(i,v) \ | 163 | #define atomic64_add_return(i,v) \ |
| 171 | ({ \ | 164 | ({ \ |
| 172 | long __ia64_aar_i = (i); \ | 165 | long __ia64_aar_i = (i); \ |
| 173 | (__builtin_constant_p(i) \ | 166 | __ia64_atomic_const(i) \ |
| 174 | && ( (__ia64_aar_i == 1) || (__ia64_aar_i == 4) \ | ||
| 175 | || (__ia64_aar_i == 8) || (__ia64_aar_i == 16) \ | ||
| 176 | || (__ia64_aar_i == -1) || (__ia64_aar_i == -4) \ | ||
| 177 | || (__ia64_aar_i == -8) || (__ia64_aar_i == -16))) \ | ||
| 178 | ? ia64_fetch_and_add(__ia64_aar_i, &(v)->counter) \ | 167 | ? ia64_fetch_and_add(__ia64_aar_i, &(v)->counter) \ |
| 179 | : ia64_atomic64_add(__ia64_aar_i, v); \ | 168 | : ia64_atomic64_add(__ia64_aar_i, v); \ |
| 180 | }) | 169 | }) |
| @@ -182,11 +171,7 @@ ATOMIC64_OPS(sub, -) · 1 unmodified line | |||
| 182 | #define atomic64_sub_return(i,v) \ | 171 | #define atomic64_sub_return(i,v) \ |
| 183 | ({ \ | 172 | ({ \ |
| 184 | long __ia64_asr_i = (i); \ | 173 | long __ia64_asr_i = (i); \ |
| 185 | (__builtin_constant_p(i) \ | 174 | __ia64_atomic_const(i) \ |
| 186 | && ( (__ia64_asr_i == 1) || (__ia64_asr_i == 4) \ | ||
| 187 | || (__ia64_asr_i == 8) || (__ia64_asr_i == 16) \ | ||
| 188 | || (__ia64_asr_i == -1) || (__ia64_asr_i == -4) \ | ||
| 189 | || (__ia64_asr_i == -8) || (__ia64_asr_i == -16))) \ | ||
| 190 | ? ia64_fetch_and_add(-__ia64_asr_i, &(v)->counter) \ | 175 | ? ia64_fetch_and_add(-__ia64_asr_i, &(v)->counter) \ |
| 191 | : ia64_atomic64_sub(__ia64_asr_i, v); \ | 176 | : ia64_atomic64_sub(__ia64_asr_i, v); \ |
| 192 | }) | 177 | }) |
| @@ -194,11 +179,7 @@ ATOMIC64_OPS(sub, -) · 1 unmodified line | |||
| 194 | #define atomic64_fetch_add(i,v) \ | 179 | #define atomic64_fetch_add(i,v) \ |
| 195 | ({ \ | 180 | ({ \ |
| 196 | long __ia64_aar_i = (i); \ | 181 | long __ia64_aar_i = (i); \ |
| 197 | (__builtin_constant_p(i) \ | 182 | __ia64_atomic_const(i) \ |
| 198 | && ( (__ia64_aar_i == 1) || (__ia64_aar_i == 4) \ | ||
| 199 | || (__ia64_aar_i == 8) || (__ia64_aar_i == 16) \ | ||
| 200 | || (__ia64_aar_i == -1) || (__ia64_aar_i == -4) \ | ||
| 201 | || (__ia64_aar_i == -8) || (__ia64_aar_i == -16))) \ | ||
| 202 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ | 183 | ? ia64_fetchadd(__ia64_aar_i, &(v)->counter, acq) \ |
| 203 | : ia64_atomic64_fetch_add(__ia64_aar_i, v); \ | 184 | : ia64_atomic64_fetch_add(__ia64_aar_i, v); \ |
| 204 | }) | 185 | }) |
| @@ -206,11 +187,7 @@ ATOMIC64_OPS(sub, -) · 1 unmodified line | |||
| 206 | #define atomic64_fetch_sub(i,v) \ | 187 | #define atomic64_fetch_sub(i,v) \ |
| 207 | ({ \ | 188 | ({ \ |
| 208 | long __ia64_asr_i = (i); \ | 189 | long __ia64_asr_i = (i); \ |
| 209 | (__builtin_constant_p(i) \ | 190 | __ia64_atomic_const(i) \ |
| 210 | && ( (__ia64_asr_i == 1) || (__ia64_asr_i == 4) \ | ||
| 211 | || (__ia64_asr_i == 8) || (__ia64_asr_i == 16) \ | ||
| 212 | || (__ia64_asr_i == -1) || (__ia64_asr_i == -4) \ | ||
| 213 | || (__ia64_asr_i == -8) || (__ia64_asr_i == -16))) \ | ||
| 214 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ | 191 | ? ia64_fetchadd(-__ia64_asr_i, &(v)->counter, acq) \ |
| 215 | : ia64_atomic64_fetch_sub(__ia64_asr_i, v); \ | 192 | : ia64_atomic64_fetch_sub(__ia64_asr_i, v); \ |
| 216 | }) | 193 | }) |
| @@ -4,7 +4,11 @@ · 3 unmodified lines | |||
| 4 | 4 | ||
| 5 | #ifdef CONFIG_BUG | 5 | #ifdef CONFIG_BUG |
| 6 | #define ia64_abort() __builtin_trap() | 6 | #define ia64_abort() __builtin_trap() |
| 7 | #define BUG() do { printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); ia64_abort(); } while (0) | 7 | #define BUG() do { \ |
| 8 | printk("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ | ||
| 9 | barrier_before_unreachable(); \ | ||
| 10 | ia64_abort(); \ | ||
| 11 | } while (0) | ||
| 8 | 12 | ||
| 9 | /* should this BUG be made generic? */ | 13 | /* should this BUG be made generic? */ |
| 10 | #define HAVE_ARCH_BUG | 14 | #define HAVE_ARCH_BUG |
| @@ -41,7 +41,6 @@ ifneq ($(CONFIG_IA64_ESI),) · 40 unmodified lines | |||
| 41 | obj-y += esi_stub.o # must be in kernel proper | 41 | obj-y += esi_stub.o # must be in kernel proper |
| 42 | endif | 42 | endif |
| 43 | obj-$(CONFIG_INTEL_IOMMU) += pci-dma.o | 43 | obj-$(CONFIG_INTEL_IOMMU) += pci-dma.o |
| 44 | obj-$(CONFIG_SWIOTLB) += pci-swiotlb.o | ||
| 45 | 44 | ||
| 46 | obj-$(CONFIG_BINFMT_ELF) += elfcore.o | 45 | obj-$(CONFIG_BINFMT_ELF) += elfcore.o |
| 47 | 46 | ||
| @@ -117,7 +117,7 @@ store_call_start(struct device *dev, struct device_attribute *attr, · 116 unmodified lines | |||
| 117 | 117 | ||
| 118 | #ifdef ERR_INJ_DEBUG | 118 | #ifdef ERR_INJ_DEBUG |
| 119 | printk(KERN_DEBUG "Returns: status=%d,\n", (int)status[cpu]); | 119 | printk(KERN_DEBUG "Returns: status=%d,\n", (int)status[cpu]); |
| 120 | printk(KERN_DEBUG "capapbilities=%lx,\n", capabilities[cpu]); | 120 | printk(KERN_DEBUG "capabilities=%lx,\n", capabilities[cpu]); |
| 121 | printk(KERN_DEBUG "resources=%lx\n", resources[cpu]); | 121 | printk(KERN_DEBUG "resources=%lx\n", resources[cpu]); |
| 122 | #endif | 122 | #endif |
| 123 | return size; | 123 | return size; |
| @@ -142,7 +142,7 @@ store_virtual_to_phys(struct device *dev, struct device_attribute *attr, · 18 unmodified lines | |||
| 142 | u64 virt_addr=simple_strtoull(buf, NULL, 16); | 142 | u64 virt_addr=simple_strtoull(buf, NULL, 16); |
| 143 | int ret; | 143 | int ret; |
| 144 | 144 | ||
| 145 | ret = get_user_pages(virt_addr, 1, FOLL_WRITE, NULL, NULL); | 145 | ret = get_user_pages_fast(virt_addr, 1, FOLL_WRITE, NULL); |
| 146 | if (ret<=0) { | 146 | if (ret<=0) { |
| 147 | #ifdef ERR_INJ_DEBUG | 147 | #ifdef ERR_INJ_DEBUG |
| 148 | printk("Virtual address %lx is not existing.\n",virt_addr); | 148 | printk("Virtual address %lx is not existing.\n",virt_addr); |
| @@ -16,7 +16,7 @@ import re · 15 unmodified lines | |||
| 16 | import sys | 16 | import sys |
| 17 | 17 | ||
| 18 | if len(sys.argv) != 2: | 18 | if len(sys.argv) != 2: |
| 19 | print "Usage: %s FILE" % sys.argv[0] | 19 | print("Usage: %s FILE" % sys.argv[0]) |
| 20 | sys.exit(2) | 20 | sys.exit(2) |
| 21 | 21 | ||
| 22 | readelf = os.getenv("READELF", "readelf") | 22 | readelf = os.getenv("READELF", "readelf") |
| @@ -29,7 +29,7 @@ def check_func (func, slots, rlen_sum): · 6 unmodified lines | |||
| 29 | global num_errors | 29 | global num_errors |
| 30 | num_errors += 1 | 30 | num_errors += 1 |
| 31 | if not func: func = "[%#x-%#x]" % (start, end) | 31 | if not func: func = "[%#x-%#x]" % (start, end) |
| 32 | print "ERROR: %s: %lu slots, total region length = %lu" % (func, slots, rlen_sum) | 32 | print("ERROR: %s: %lu slots, total region length = %lu" % (func, slots, rlen_sum)) |
| 33 | return | 33 | return |
| 34 | 34 | ||
| 35 | num_funcs = 0 | 35 | num_funcs = 0 |
| @@ -43,23 +43,23 @@ for line in os.popen("%s -u %s" % (readelf, sys.argv[1])): · 7 unmodified lines | |||
| 43 | check_func(func, slots, rlen_sum) | 43 | check_func(func, slots, rlen_sum) |
| 44 | 44 | ||
| 45 | func = m.group(1) | 45 | func = m.group(1) |
| 46 | start = long(m.group(2), 16) | 46 | start = int(m.group(2), 16) |
| 47 | end = long(m.group(3), 16) | 47 | end = int(m.group(3), 16) |
| 48 | slots = 3 * (end - start) / 16 | 48 | slots = 3 * (end - start) / 16 |
| 49 | rlen_sum = 0L | 49 | rlen_sum = 0 |
| 50 | num_funcs += 1 | 50 | num_funcs += 1 |
| 51 | else: | 51 | else: |
| 52 | m = rlen_pattern.match(line) | 52 | m = rlen_pattern.match(line) |
| 53 | if m: | 53 | if m: |
| 54 | rlen_sum += long(m.group(1)) | 54 | rlen_sum += int(m.group(1)) |
| 55 | check_func(func, slots, rlen_sum) | 55 | check_func(func, slots, rlen_sum) |
| 56 | 56 | ||
| 57 | if num_errors == 0: | 57 | if num_errors == 0: |
| 58 | print "No errors detected in %u functions." % num_funcs | 58 | print("No errors detected in %u functions." % num_funcs) |
| 59 | else: | 59 | else: |
| 60 | if num_errors > 1: | 60 | if num_errors > 1: |
| 61 | err="errors" | 61 | err="errors" |
| 62 | else: | 62 | else: |
| 63 | err="error" | 63 | err="error" |
| 64 | print "%u %s detected in %u functions." % (num_errors, err, num_funcs) | 64 | print("%u %s detected in %u functions." % (num_errors, err, num_funcs)) |
| 65 | sys.exit(1) | 65 | sys.exit(1) |
| @@ -8,16 +8,19 @@ · 7 unmodified lines | |||
| 8 | #ifndef CONFIG_SUN3 | 8 | #ifndef CONFIG_SUN3 |
| 9 | #define BUG() do { \ | 9 | #define BUG() do { \ |
| 10 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ | 10 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ |
| 11 | barrier_before_unreachable(); \ | ||
| 11 | __builtin_trap(); \ | 12 | __builtin_trap(); \ |
| 12 | } while (0) | 13 | } while (0) |
| 13 | #else | 14 | #else |
| 14 | #define BUG() do { \ | 15 | #define BUG() do { \ |
| 15 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ | 16 | pr_crit("kernel BUG at %s:%d!\n", __FILE__, __LINE__); \ |
| 17 | barrier_before_unreachable(); \ | ||
| 16 | panic("BUG!"); \ | 18 | panic("BUG!"); \ |
| 17 | } while (0) | 19 | } while (0) |
| 18 | #endif | 20 | #endif |
| 19 | #else | 21 | #else |
| 20 | #define BUG() do { \ | 22 | #define BUG() do { \ |
| 23 | barrier_before_unreachable(); \ | ||
| 21 | __builtin_trap(); \ | 24 | __builtin_trap(); \ |
| 22 | } while (0) | 25 | } while (0) |
| 23 | #endif | 26 | #endif |
| @@ -135,6 +135,8 @@ int __init ath25_find_config(phys_addr_t base, unsigned long size) · 134 unmodified lines | |||
| 135 | } | 135 | } |
| 136 | 136 | ||
| 137 | board_data = kzalloc(BOARD_CONFIG_BUFSZ, GFP_KERNEL); | 137 | board_data = kzalloc(BOARD_CONFIG_BUFSZ, GFP_KERNEL); |
| 138 | if (!board_data) | ||
| 139 | goto error; | ||
| 138 | ath25_board.config = (struct ath25_boarddata *)board_data; | 140 | ath25_board.config = (struct ath25_boarddata *)board_data; |
| 139 | memcpy_fromio(board_data, bcfg, 0x100); | 141 | memcpy_fromio(board_data, bcfg, 0x100); |
| 140 | if (broken_boarddata) { | 142 | if (broken_boarddata) { |
| @@ -126,6 +126,7 @@ $(obj)/vmlinux.its.S: $(addprefix $(srctree)/arch/mips/$(PLATFORM)/,$(ITS_INPUTS · 125 unmodified lines | |||
| 126 | 126 | ||
| 127 | quiet_cmd_cpp_its_S = ITS $@ | 127 | quiet_cmd_cpp_its_S = ITS $@ |
| 128 | cmd_cpp_its_S = $(CPP) $(cpp_flags) -P -C -o $@ $< \ | 128 | cmd_cpp_its_S = $(CPP) $(cpp_flags) -P -C -o $@ $< \ |
| 129 | -D__ASSEMBLY__ \ | ||
| 129 | -DKERNEL_NAME="\"Linux $(KERNELRELEASE)\"" \ | 130 | -DKERNEL_NAME="\"Linux $(KERNELRELEASE)\"" \ |
| 130 | -DVMLINUX_BINARY="\"$(3)\"" \ | 131 | -DVMLINUX_BINARY="\"$(3)\"" \ |
| 131 | -DVMLINUX_COMPRESSION="\"$(2)\"" \ | 132 | -DVMLINUX_COMPRESSION="\"$(2)\"" \ |
| @@ -2277,6 +2277,8 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node, · 2276 unmodified lines | |||
| 2277 | } | 2277 | } |
| 2278 | 2278 | ||
| 2279 | host_data = kzalloc(sizeof(*host_data), GFP_KERNEL); | 2279 | host_data = kzalloc(sizeof(*host_data), GFP_KERNEL); |
| 2280 | if (!host_data) | ||
| 2281 | return -ENOMEM; | ||
| 2280 | raw_spin_lock_init(&host_data->lock); | 2282 | raw_spin_lock_init(&host_data->lock); |
| 2281 | 2283 | ||
| 2282 | addr = of_get_address(ciu_node, 0, NULL, NULL); | 2284 | addr = of_get_address(ciu_node, 0, NULL, NULL); |
| @@ -86,7 +86,6 @@ struct compat_flock { · 85 unmodified lines | |||
| 86 | compat_off_t l_len; | 86 | compat_off_t l_len; |
| 87 | s32 l_sysid; | 87 | s32 l_sysid; |
| 88 | compat_pid_t l_pid; | 88 | compat_pid_t l_pid; |
| 89 | short __unused; | ||
| 90 | s32 pad[4]; | 89 | s32 pad[4]; |
| 91 | }; | 90 | }; |
| 92 | 91 | ||
| @@ -10,6 +10,8 @@ · 9 unmodified lines | |||
| 10 | 10 | ||
| 11 | #include <linux/errno.h> | 11 | #include <linux/errno.h> |
| 12 | #include <linux/percpu.h> | 12 | #include <linux/percpu.h> |
| 13 | #include <linux/of.h> | ||
| 14 | #include <linux/of_address.h> | ||
| 13 | #include <linux/spinlock.h> | 15 | #include <linux/spinlock.h> |
| 14 | 16 | ||
| 15 | #include <asm/mips-cps.h> | 17 | #include <asm/mips-cps.h> |
| @@ -22,6 +24,17 @@ static DEFINE_PER_CPU_ALIGNED(unsigned long, cpc_core_lock_flags); · 6 unmodified lines | |||
| 22 | 24 | ||
| 23 | phys_addr_t __weak mips_cpc_default_phys_base(void) | 25 | phys_addr_t __weak mips_cpc_default_phys_base(void) |
| 24 | { | 26 | { |
| 27 | struct device_node *cpc_node; | ||
| 28 | struct resource res; | ||
| 29 | int err; | ||
| 30 | |||
| 31 | cpc_node = of_find_compatible_node(of_root, NULL, "mti,mips-cpc"); | ||
| 32 | if (cpc_node) { | ||
| 33 | err = of_address_to_resource(cpc_node, 0, &res); | ||
| 34 | if (!err) | ||
| 35 | return res.start; | ||
| 36 | } | ||
| 37 | |||
| 25 | return 0; | 38 | return 0; |
| 26 | } | 39 | } |
| 27 | 40 | ||
| @@ -375,6 +375,7 @@ static void __init bootmem_init(void) · 374 unmodified lines | |||
| 375 | unsigned long reserved_end; | 375 | unsigned long reserved_end; |
| 376 | unsigned long mapstart = ~0UL; | 376 | unsigned long mapstart = ~0UL; |
| 377 | unsigned long bootmap_size; | 377 | unsigned long bootmap_size; |
| 378 | phys_addr_t ramstart = (phys_addr_t)ULLONG_MAX; | ||
| 378 | bool bootmap_valid = false; | 379 | bool bootmap_valid = false; |
| 379 | int i; | 380 | int i; |
| 380 | 381 | ||
| @@ -395,7 +396,8 @@ static void __init bootmem_init(void) · 14 unmodified lines | |||
| 395 | max_low_pfn = 0; | 396 | max_low_pfn = 0; |
| 396 | 397 | ||
| 397 | /* | 398 | /* |
| 398 | * Find the highest page frame number we have available. | 399 | * Find the highest page frame number we have available |
| 400 | * and the lowest used RAM address | ||
| 399 | */ | 401 | */ |
| 400 | for (i = 0; i < boot_mem_map.nr_map; i++) { | 402 | for (i = 0; i < boot_mem_map.nr_map; i++) { |
| 401 | unsigned long start, end; | 403 | unsigned long start, end; |
| @@ -407,6 +409,8 @@ static void __init bootmem_init(void) · 5 unmodified lines | |||
| 407 | end = PFN_DOWN(boot_mem_map.map[i].addr | 409 | end = PFN_DOWN(boot_mem_map.map[i].addr |
| 408 | + boot_mem_map.map[i].size); | 410 | + boot_mem_map.map[i].size); |
| 409 | 411 | ||
| 412 | ramstart = min(ramstart, boot_mem_map.map[i].addr); | ||
| 413 | |||
| 410 | #ifndef CONFIG_HIGHMEM | 414 | #ifndef CONFIG_HIGHMEM |
| 411 | /* | 415 | /* |
| 412 | * Skip highmem here so we get an accurate max_low_pfn if low | 416 | * Skip highmem here so we get an accurate max_low_pfn if low |
| @@ -436,6 +440,13 @@ static void __init bootmem_init(void) · 23 unmodified lines | |||
| 436 | mapstart = max(reserved_end, start); | 440 | mapstart = max(reserved_end, start); |
| 437 | } | 441 | } |
| 438 | 442 | ||
| 443 | /* | ||
| 444 | * Reserve any memory between the start of RAM and PHYS_OFFSET | ||
| 445 | */ | ||
| 446 | if (ramstart > PHYS_OFFSET) | ||
| 447 | add_memory_region(PHYS_OFFSET, ramstart - PHYS_OFFSET, | ||
| 448 | BOOT_MEM_RESERVED); | ||
| 449 | |||
| 439 | if (min_low_pfn >= max_low_pfn) | 450 | if (min_low_pfn >= max_low_pfn) |
| 440 | panic("Incorrect memory mapping !!!"); | 451 | panic("Incorrect memory mapping !!!"); |
| 441 | if (min_low_pfn > ARCH_PFN_OFFSET) { | 452 | if (min_low_pfn > ARCH_PFN_OFFSET) { |
| @@ -664,9 +675,6 @@ static int __init early_parse_mem(char *p) · 222 unmodified lines | |||
| 664 | 675 | ||
| 665 | add_memory_region(start, size, BOOT_MEM_RAM); | 676 | add_memory_region(start, size, BOOT_MEM_RAM); |
| 666 | 677 | ||
| 667 | if (start && start > PHYS_OFFSET) | ||
| 668 | add_memory_region(PHYS_OFFSET, start - PHYS_OFFSET, | ||
| 669 | BOOT_MEM_RESERVED); | ||
| 670 | return 0; | 678 | return 0; |
| 671 | } | 679 | } |
| 672 | early_param("mem", early_parse_mem); | 680 | early_param("mem", early_parse_mem); |
| @@ -168,11 +168,11 @@ static void bmips_prepare_cpus(unsigned int max_cpus) · 167 unmodified lines | |||
| 168 | return; | 168 | return; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | if (request_irq(IPI0_IRQ, bmips_ipi_interrupt, IRQF_PERCPU, | 171 | if (request_irq(IPI0_IRQ, bmips_ipi_interrupt, |
| 172 | "smp_ipi0", NULL)) | 172 | IRQF_PERCPU | IRQF_NO_SUSPEND, "smp_ipi0", NULL)) |
| 173 | panic("Can't request IPI0 interrupt"); | 173 | panic("Can't request IPI0 interrupt"); |
| 174 | if (request_irq(IPI1_IRQ, bmips_ipi_interrupt, IRQF_PERCPU, | 174 | if (request_irq(IPI1_IRQ, bmips_ipi_interrupt, |
| 175 | "smp_ipi1", NULL)) | 175 | IRQF_PERCPU | IRQF_NO_SUSPEND, "smp_ipi1", NULL)) |
| 176 | panic("Can't request IPI1 interrupt"); | 176 | panic("Can't request IPI1 interrupt"); |
| 177 | } | 177 | } |
| 178 | 178 | ||
| @@ -572,7 +572,7 @@ asmlinkage void __weak plat_wired_tlb_setup(void) · 393 unmodified lines | |||
| 572 | */ | 572 | */ |
| 573 | } | 573 | } |
| 574 | 574 | ||
| 575 | void __init bmips_cpu_setup(void) | 575 | void bmips_cpu_setup(void) |
| 576 | { | 576 | { |
| 577 | void __iomem __maybe_unused *cbr = BMIPS_GET_CBR(); | 577 | void __iomem __maybe_unused *cbr = BMIPS_GET_CBR(); |
| 578 | u32 __maybe_unused cfg; | 578 | u32 __maybe_unused cfg; |
| @@ -7,6 +7,8 @@ choice · 6 unmodified lines | |||
| 7 | config LEMOTE_FULOONG2E | 7 | config LEMOTE_FULOONG2E |
| 8 | bool "Lemote Fuloong(2e) mini-PC" | 8 | bool "Lemote Fuloong(2e) mini-PC" |
| 9 | select ARCH_SPARSEMEM_ENABLE | 9 | select ARCH_SPARSEMEM_ENABLE |
| 10 | select ARCH_MIGHT_HAVE_PC_PARPORT | ||
| 11 | select ARCH_MIGHT_HAVE_PC_SERIO | ||
| 10 | select CEVT_R4K | 12 | select CEVT_R4K |
| 11 | select CSRC_R4K | 13 | select CSRC_R4K |
| 12 | select SYS_HAS_CPU_LOONGSON2E | 14 | select SYS_HAS_CPU_LOONGSON2E |
| @@ -33,6 +35,8 @@ config LEMOTE_FULOONG2E · 20 unmodified lines | |||
| 33 | config LEMOTE_MACH2F | 35 | config LEMOTE_MACH2F |
| 34 | bool "Lemote Loongson 2F family machines" | 36 | bool "Lemote Loongson 2F family machines" |
| 35 | select ARCH_SPARSEMEM_ENABLE | 37 | select ARCH_SPARSEMEM_ENABLE |
| 38 | select ARCH_MIGHT_HAVE_PC_PARPORT | ||
| 39 | select ARCH_MIGHT_HAVE_PC_SERIO | ||
| 36 | select BOARD_SCACHE | 40 | select BOARD_SCACHE |
| 37 | select BOOT_ELF32 | 41 | select BOOT_ELF32 |
| 38 | select CEVT_R4K if ! MIPS_EXTERNAL_TIMER | 42 | select CEVT_R4K if ! MIPS_EXTERNAL_TIMER |
| @@ -62,6 +66,8 @@ config LEMOTE_MACH2F · 23 unmodified lines | |||
| 62 | config LOONGSON_MACH3X | 66 | config LOONGSON_MACH3X |
| 63 | bool "Generic Loongson 3 family machines" | 67 | bool "Generic Loongson 3 family machines" |
| 64 | select ARCH_SPARSEMEM_ENABLE | 68 | select ARCH_SPARSEMEM_ENABLE |
| 69 | select ARCH_MIGHT_HAVE_PC_PARPORT | ||
| 70 | select ARCH_MIGHT_HAVE_PC_SERIO | ||
| 65 | select GENERIC_ISA_DMA_SUPPORT_BROKEN | 71 | select GENERIC_ISA_DMA_SUPPORT_BROKEN |
| 66 | select BOOT_ELF32 | 72 | select BOOT_ELF32 |
| 67 | select BOARD_SCACHE | 73 | select BOARD_SCACHE |
| @@ -26,6 +26,7 @@ void flush_user_icache_range_asm(unsigned long, unsigned long); · 25 unmodified lines | |||
| 26 | void flush_kernel_icache_range_asm(unsigned long, unsigned long); | 26 | void flush_kernel_icache_range_asm(unsigned long, unsigned long); |
| 27 | void flush_user_dcache_range_asm(unsigned long, unsigned long); | 27 | void flush_user_dcache_range_asm(unsigned long, unsigned long); |
| 28 | void flush_kernel_dcache_range_asm(unsigned long, unsigned long); | 28 | void flush_kernel_dcache_range_asm(unsigned long, unsigned long); |
| 29 | void purge_kernel_dcache_range_asm(unsigned long, unsigned long); | ||
| 29 | void flush_kernel_dcache_page_asm(void *); | 30 | void flush_kernel_dcache_page_asm(void *); |
| 30 | void flush_kernel_icache_page(void *); | 31 | void flush_kernel_icache_page(void *); |
| 31 | 32 | ||
| @@ -316,6 +316,8 @@ extern int _parisc_requires_coherency; · 315 unmodified lines | |||
| 316 | #define parisc_requires_coherency() (0) | 316 | #define parisc_requires_coherency() (0) |
| 317 | #endif | 317 | #endif |
| 318 | 318 | ||
| 319 | extern int running_on_qemu; | ||
| 320 | |||
| 319 | #endif /* __ASSEMBLY__ */ | 321 | #endif /* __ASSEMBLY__ */ |
| 320 | 322 | ||
| 321 | #endif /* __ASM_PARISC_PROCESSOR_H */ | 323 | #endif /* __ASM_PARISC_PROCESSOR_H */ |
| @@ -465,10 +465,10 @@ EXPORT_SYMBOL(copy_user_page); · 464 unmodified lines | |||
| 465 | int __flush_tlb_range(unsigned long sid, unsigned long start, | 465 | int __flush_tlb_range(unsigned long sid, unsigned long start, |
| 466 | unsigned long end) | 466 | unsigned long end) |
| 467 | { | 467 | { |
| 468 | unsigned long flags, size; | 468 | unsigned long flags; |
| 469 | 469 | ||
| 470 | size = (end - start); | 470 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 471 | if (size >= parisc_tlb_flush_threshold) { | 471 | end - start >= parisc_tlb_flush_threshold) { |
| 472 | flush_tlb_all(); | 472 | flush_tlb_all(); |
| 473 | return 1; | 473 | return 1; |
| 474 | } | 474 | } |
| @@ -539,13 +539,11 @@ void flush_cache_mm(struct mm_struct *mm) · 64 unmodified lines | |||
| 539 | struct vm_area_struct *vma; | 539 | struct vm_area_struct *vma; |
| 540 | pgd_t *pgd; | 540 | pgd_t *pgd; |
| 541 | 541 | ||
| 542 | /* Flush the TLB to avoid speculation if coherency is required. */ | ||
| 543 | if (parisc_requires_coherency()) | ||
| 544 | flush_tlb_all(); | ||
| 545 | |||
| 546 | /* Flushing the whole cache on each cpu takes forever on | 542 | /* Flushing the whole cache on each cpu takes forever on |
| 547 | rp3440, etc. So, avoid it if the mm isn't too big. */ | 543 | rp3440, etc. So, avoid it if the mm isn't too big. */ |
| 548 | if (mm_total_size(mm) >= parisc_cache_flush_threshold) { | 544 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 545 | mm_total_size(mm) >= parisc_cache_flush_threshold) { | ||
| 546 | flush_tlb_all(); | ||
| 549 | flush_cache_all(); | 547 | flush_cache_all(); |
| 550 | return; | 548 | return; |
| 551 | } | 549 | } |
| @@ -553,9 +551,9 @@ void flush_cache_mm(struct mm_struct *mm) · 1 unmodified line | |||
| 553 | if (mm->context == mfsp(3)) { | 551 | if (mm->context == mfsp(3)) { |
| 554 | for (vma = mm->mmap; vma; vma = vma->vm_next) { | 552 | for (vma = mm->mmap; vma; vma = vma->vm_next) { |
| 555 | flush_user_dcache_range_asm(vma->vm_start, vma->vm_end); | 553 | flush_user_dcache_range_asm(vma->vm_start, vma->vm_end); |
| 556 | if ((vma->vm_flags & VM_EXEC) == 0) | 554 | if (vma->vm_flags & VM_EXEC) |
| 557 | continue; | 555 | flush_user_icache_range_asm(vma->vm_start, vma->vm_end); |
| 558 | flush_user_icache_range_asm(vma->vm_start, vma->vm_end); | 556 | flush_tlb_range(vma, vma->vm_start, vma->vm_end); |
| 559 | } | 557 | } |
| 560 | return; | 558 | return; |
| 561 | } | 559 | } |
| @@ -581,14 +579,9 @@ void flush_cache_mm(struct mm_struct *mm) · 19 unmodified lines | |||
| 581 | void flush_cache_range(struct vm_area_struct *vma, | 579 | void flush_cache_range(struct vm_area_struct *vma, |
| 582 | unsigned long start, unsigned long end) | 580 | unsigned long start, unsigned long end) |
| 583 | { | 581 | { |
| 584 | BUG_ON(!vma->vm_mm->context); | 582 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 585 | 583 | end - start >= parisc_cache_flush_threshold) { | |
| 586 | /* Flush the TLB to avoid speculation if coherency is required. */ | ||
| 587 | if (parisc_requires_coherency()) | ||
| 588 | flush_tlb_range(vma, start, end); | 584 | flush_tlb_range(vma, start, end); |
| 589 | |||
| 590 | if ((end - start) >= parisc_cache_flush_threshold | ||
| 591 | || vma->vm_mm->context != mfsp(3)) { | ||
| 592 | flush_cache_all(); | 585 | flush_cache_all(); |
| 593 | return; | 586 | return; |
| 594 | } | 587 | } |
| @@ -596,6 +589,7 @@ void flush_cache_range(struct vm_area_struct *vma, · 1 unmodified line | |||
| 596 | flush_user_dcache_range_asm(start, end); | 589 | flush_user_dcache_range_asm(start, end); |
| 597 | if (vma->vm_flags & VM_EXEC) | 590 | if (vma->vm_flags & VM_EXEC) |
| 598 | flush_user_icache_range_asm(start, end); | 591 | flush_user_icache_range_asm(start, end); |
| 592 | flush_tlb_range(vma, start, end); | ||
| 599 | } | 593 | } |
| 600 | 594 | ||
| 601 | void | 595 | void |
| @@ -604,8 +598,7 @@ flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long · 2 unmodified lines | |||
| 604 | BUG_ON(!vma->vm_mm->context); | 598 | BUG_ON(!vma->vm_mm->context); |
| 605 | 599 | ||
| 606 | if (pfn_valid(pfn)) { | 600 | if (pfn_valid(pfn)) { |
| 607 | if (parisc_requires_coherency()) | 601 | flush_tlb_page(vma, vmaddr); |
| 608 | flush_tlb_page(vma, vmaddr); | ||
| 609 | __flush_cache_page(vma, vmaddr, PFN_PHYS(pfn)); | 602 | __flush_cache_page(vma, vmaddr, PFN_PHYS(pfn)); |
| 610 | } | 603 | } |
| 611 | } | 604 | } |
| @@ -613,21 +606,33 @@ flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long · 1 unmodified line | |||
| 613 | void flush_kernel_vmap_range(void *vaddr, int size) | 606 | void flush_kernel_vmap_range(void *vaddr, int size) |
| 614 | { | 607 | { |
| 615 | unsigned long start = (unsigned long)vaddr; | 608 | unsigned long start = (unsigned long)vaddr; |
| 609 | unsigned long end = start + size; | ||
| 616 | 610 | ||
| 617 | if ((unsigned long)size > parisc_cache_flush_threshold) | 611 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 612 | (unsigned long)size >= parisc_cache_flush_threshold) { | ||
| 613 | flush_tlb_kernel_range(start, end); | ||
| 618 | flush_data_cache(); | 614 | flush_data_cache(); |
| 619 | else | 615 | return; |
| 620 | flush_kernel_dcache_range_asm(start, start + size); | 616 | } |
| 617 | |||
| 618 | flush_kernel_dcache_range_asm(start, end); | ||
| 619 | flush_tlb_kernel_range(start, end); | ||
| 621 | } | 620 | } |
| 622 | EXPORT_SYMBOL(flush_kernel_vmap_range); | 621 | EXPORT_SYMBOL(flush_kernel_vmap_range); |
| 623 | 622 | ||
| 624 | void invalidate_kernel_vmap_range(void *vaddr, int size) | 623 | void invalidate_kernel_vmap_range(void *vaddr, int size) |
| 625 | { | 624 | { |
| 626 | unsigned long start = (unsigned long)vaddr; | 625 | unsigned long start = (unsigned long)vaddr; |
| 626 | unsigned long end = start + size; | ||
| 627 | 627 | ||
| 628 | if ((unsigned long)size > parisc_cache_flush_threshold) | 628 | if ((!IS_ENABLED(CONFIG_SMP) || !arch_irqs_disabled()) && |
| 629 | (unsigned long)size >= parisc_cache_flush_threshold) { | ||
| 630 | flush_tlb_kernel_range(start, end); | ||
| 629 | flush_data_cache(); | 631 | flush_data_cache(); |
| 630 | else | 632 | return; |
| 631 | flush_kernel_dcache_range_asm(start, start + size); | 633 | } |
| 634 | |||
| 635 | purge_kernel_dcache_range_asm(start, end); | ||
| 636 | flush_tlb_kernel_range(start, end); | ||
| 632 | } | 637 | } |
| 633 | EXPORT_SYMBOL(invalidate_kernel_vmap_range); | 638 | EXPORT_SYMBOL(invalidate_kernel_vmap_range); |
| @@ -138,6 +138,16 @@ $pgt_fill_loop: · 137 unmodified lines | |||
| 138 | std %dp,0x18(%r10) | 138 | std %dp,0x18(%r10) |
| 139 | #endif | 139 | #endif |
| 140 | 140 | ||
| 141 | #ifdef CONFIG_64BIT | ||
| 142 | /* Get PDCE_PROC for monarch CPU. */ | ||
| 143 | #define MEM_PDC_LO 0x388 | ||
| 144 | #define MEM_PDC_HI 0x35C | ||
| 145 | ldw MEM_PDC_LO(%r0),%r3 | ||
| 146 | ldw MEM_PDC_HI(%r0),%r10 | ||
| 147 | depd %r10, 31, 32, %r3 /* move to upper word */ | ||
| 148 | #endif | ||
| 149 | |||
| 150 | |||
| 141 | #ifdef CONFIG_SMP | 151 | #ifdef CONFIG_SMP |
| 142 | /* Set the smp rendezvous address into page zero. | 152 | /* Set the smp rendezvous address into page zero. |
| 143 | ** It would be safer to do this in init_smp_config() but | 153 | ** It would be safer to do this in init_smp_config() but |
| @@ -196,12 +206,6 @@ common_stext: · 52 unmodified lines | |||
| 196 | ** Someday, palo might not do this for the Monarch either. | 206 | ** Someday, palo might not do this for the Monarch either. |
| 197 | */ | 207 | */ |
| 198 | 2: | 208 | 2: |
| 199 | #define MEM_PDC_LO 0x388 | ||
| 200 | #define MEM_PDC_HI 0x35C | ||
| 201 | ldw MEM_PDC_LO(%r0),%r3 | ||
| 202 | ldw MEM_PDC_HI(%r0),%r6 | ||
| 203 | depd %r6, 31, 32, %r3 /* move to upper word */ | ||
| 204 | |||
| 205 | mfctl %cr30,%r6 /* PCX-W2 firmware bug */ | 209 | mfctl %cr30,%r6 /* PCX-W2 firmware bug */ |
| 206 | 210 | ||
| 207 | ldo PDC_PSW(%r0),%arg0 /* 21 */ | 211 | ldo PDC_PSW(%r0),%arg0 /* 21 */ |
| @@ -268,6 +272,8 @@ $install_iva: · 60 unmodified lines | |||
| 268 | aligned_rfi: | 272 | aligned_rfi: |
| 269 | pcxt_ssm_bug | 273 | pcxt_ssm_bug |
| 270 | 274 | ||
| 275 | copy %r3, %arg0 /* PDCE_PROC for smp_callin() */ | ||
| 276 | |||
| 271 | rsm PSW_SM_QUIET,%r0 /* off troublesome PSW bits */ | 277 | rsm PSW_SM_QUIET,%r0 /* off troublesome PSW bits */ |
| 272 | /* Don't need NOPs, have 8 compliant insn before rfi */ | 278 | /* Don't need NOPs, have 8 compliant insn before rfi */ |
| 273 | 279 | ||
| @@ -1110,6 +1110,28 @@ ENTRY_CFI(flush_kernel_dcache_range_asm) · 1109 unmodified lines | |||
| 1110 | .procend | 1110 | .procend |
| 1111 | ENDPROC_CFI(flush_kernel_dcache_range_asm) | 1111 | ENDPROC_CFI(flush_kernel_dcache_range_asm) |
| 1112 | 1112 | ||
| 1113 | ENTRY_CFI(purge_kernel_dcache_range_asm) | ||
| 1114 | .proc | ||
| 1115 | .callinfo NO_CALLS | ||
| 1116 | .entry | ||
| 1117 | |||
| 1118 | ldil L%dcache_stride, %r1 | ||
| 1119 | ldw R%dcache_stride(%r1), %r23 | ||
| 1120 | ldo -1(%r23), %r21 | ||
| 1121 | ANDCM %r26, %r21, %r26 | ||
| 1122 | |||
| 1123 | 1: cmpb,COND(<<),n %r26, %r25,1b | ||
| 1124 | pdc,m %r23(%r26) | ||
| 1125 | |||
| 1126 | sync | ||
| 1127 | syncdma | ||
| 1128 | bv %r0(%r2) | ||
| 1129 | nop | ||
| 1130 | .exit | ||
| 1131 | |||
| 1132 | .procend | ||
| 1133 | ENDPROC_CFI(purge_kernel_dcache_range_asm) | ||
| 1134 | |||
| 1113 | ENTRY_CFI(flush_user_icache_range_asm) | 1135 | ENTRY_CFI(flush_user_icache_range_asm) |
| 1114 | .proc | 1136 | .proc |
| 1115 | .callinfo NO_CALLS | 1137 | .callinfo NO_CALLS |
| @@ -292,10 +292,15 @@ smp_cpu_init(int cpunum) · 291 unmodified lines | |||
| 292 | * Slaves start using C here. Indirectly called from smp_slave_stext. | 292 | * Slaves start using C here. Indirectly called from smp_slave_stext. |
| 293 | * Do what start_kernel() and main() do for boot strap processor (aka monarch) | 293 | * Do what start_kernel() and main() do for boot strap processor (aka monarch) |
| 294 | */ | 294 | */ |
| 295 | void __init smp_callin(void) | 295 | void __init smp_callin(unsigned long pdce_proc) |
| 296 | { | 296 | { |
| 297 | int slave_id = cpu_now_booting; | 297 | int slave_id = cpu_now_booting; |
| 298 | 298 | ||
| 299 | #ifdef CONFIG_64BIT | ||
| 300 | WARN_ON(((unsigned long)(PAGE0->mem_pdc_hi) << 32 | ||
| 301 | | PAGE0->mem_pdc) != pdce_proc); | ||
| 302 | #endif | ||
| 303 | |||
| 299 | smp_cpu_init(slave_id); | 304 | smp_cpu_init(slave_id); |
| 300 | preempt_disable(); | 305 | preempt_disable(); |
| 301 | 306 | ||
| @@ -76,10 +76,10 @@ irqreturn_t __irq_entry timer_interrupt(int irq, void *dev_id) · 75 unmodified lines | |||
| 76 | next_tick = cpuinfo->it_value; | 76 | next_tick = cpuinfo->it_value; |
| 77 | 77 | ||
| 78 | /* Calculate how many ticks have elapsed. */ | 78 | /* Calculate how many ticks have elapsed. */ |
| 79 | now = mfctl(16); | ||
| 79 | do { | 80 | do { |
| 80 | ++ticks_elapsed; | 81 | ++ticks_elapsed; |
| 81 | next_tick += cpt; | 82 | next_tick += cpt; |
| 82 | now = mfctl(16); | ||
| 83 | } while (next_tick - now > cpt); | 83 | } while (next_tick - now > cpt); |
| 84 | 84 | ||
| 85 | /* Store (in CR16 cycles) up to when we are accounting right now. */ | 85 | /* Store (in CR16 cycles) up to when we are accounting right now. */ |
| @@ -103,16 +103,17 @@ irqreturn_t __irq_entry timer_interrupt(int irq, void *dev_id) · 17 unmodified lines | |||
| 103 | * if one or the other wrapped. If "now" is "bigger" we'll end up | 103 | * if one or the other wrapped. If "now" is "bigger" we'll end up |
| 104 | * with a very large unsigned number. | 104 | * with a very large unsigned number. |
| 105 | */ | 105 | */ |
| 106 | while (next_tick - mfctl(16) > cpt) | 106 | now = mfctl(16); |
| 107 | while (next_tick - now > cpt) | ||
| 107 | next_tick += cpt; | 108 | next_tick += cpt; |
| 108 | 109 | ||
| 109 | /* Program the IT when to deliver the next interrupt. | 110 | /* Program the IT when to deliver the next interrupt. |
| 110 | * Only bottom 32-bits of next_tick are writable in CR16! | 111 | * Only bottom 32-bits of next_tick are writable in CR16! |
| 111 | * Timer interrupt will be delivered at least a few hundred cycles | 112 | * Timer interrupt will be delivered at least a few hundred cycles |
| 112 | * after the IT fires, so if we are too close (<= 500 cycles) to the | 113 | * after the IT fires, so if we are too close (<= 8000 cycles) to the |
| 113 | * next cycle, simply skip it. | 114 | * next cycle, simply skip it. |
| 114 | */ | 115 | */ |
| 115 | if (next_tick - mfctl(16) <= 500) | 116 | if (next_tick - now <= 8000) |
| 116 | next_tick += cpt; | 117 | next_tick += cpt; |
| 117 | mtctl(next_tick, 16); | 118 | mtctl(next_tick, 16); |
| 118 | 119 | ||
| @@ -248,7 +249,7 @@ static int __init init_cr16_clocksource(void) · 129 unmodified lines | |||
| 248 | * different sockets, so mark them unstable and lower rating on | 249 | * different sockets, so mark them unstable and lower rating on |
| 249 | * multi-socket SMP systems. | 250 | * multi-socket SMP systems. |
| 250 | */ | 251 | */ |
| 251 | if (num_online_cpus() > 1) { | 252 | if (num_online_cpus() > 1 && !running_on_qemu) { |
| 252 | int cpu; | 253 | int cpu; |
| 253 | unsigned long cpu0_loc; | 254 | unsigned long cpu0_loc; |
| 254 | cpu0_loc = per_cpu(cpu_data, 0).cpu_loc; | 255 | cpu0_loc = per_cpu(cpu_data, 0).cpu_loc; |
| @@ -629,7 +629,12 @@ void __init mem_init(void) · 628 unmodified lines | |||
| 629 | #endif | 629 | #endif |
| 630 | 630 | ||
| 631 | mem_init_print_info(NULL); | 631 | mem_init_print_info(NULL); |
| 632 | #ifdef CONFIG_DEBUG_KERNEL /* double-sanity-check paranoia */ | 632 | |
| 633 | #if 0 | ||
| 634 | /* | ||
| 635 | * Do not expose the virtual kernel memory layout to userspace. | ||
| 636 | * But keep code for debugging purposes. | ||
| 637 | */ | ||
| 633 | printk("virtual kernel memory layout:\n" | 638 | printk("virtual kernel memory layout:\n" |
| 634 | " vmalloc : 0x%px - 0x%px (%4ld MB)\n" | 639 | " vmalloc : 0x%px - 0x%px (%4ld MB)\n" |
| 635 | " memory : 0x%px - 0x%px (%4ld MB)\n" | 640 | " memory : 0x%px - 0x%px (%4ld MB)\n" |
| @@ -101,7 +101,8 @@ $(addprefix $(obj)/,$(zlib-y)): \ · 100 unmodified lines | |||
| 101 | libfdt := fdt.c fdt_ro.c fdt_wip.c fdt_sw.c fdt_rw.c fdt_strerror.c | 101 | libfdt := fdt.c fdt_ro.c fdt_wip.c fdt_sw.c fdt_rw.c fdt_strerror.c |
| 102 | libfdtheader := fdt.h libfdt.h libfdt_internal.h | 102 | libfdtheader := fdt.h libfdt.h libfdt_internal.h |
| 103 | 103 | ||
| 104 | $(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o): \ | 104 | $(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o \ |
| 105 | treeboot-akebono.o treeboot-currituck.o treeboot-iss4xx.o): \ | ||
| 105 | $(addprefix $(obj)/,$(libfdtheader)) | 106 | $(addprefix $(obj)/,$(libfdtheader)) |
| 106 | 107 | ||
| 107 | src-wlib-y := string.S crt0.S stdio.c decompress.c main.c \ | 108 | src-wlib-y := string.S crt0.S stdio.c decompress.c main.c \ |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | |||
| 16 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) | 16 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) |
| 17 | 17 | ||
| 18 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE | 18 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE |
| 19 | #define PUD_CACHE_INDEX PUD_INDEX_SIZE | ||
| 19 | 20 | ||
| 20 | #ifndef __ASSEMBLY__ | 21 | #ifndef __ASSEMBLY__ |
| 21 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) | 22 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) |
| @@ -63,7 +63,8 @@ static inline int hash__hugepd_ok(hugepd_t hpd) · 62 unmodified lines | |||
| 63 | * keeping the prototype consistent across the two formats. | 63 | * keeping the prototype consistent across the two formats. |
| 64 | */ | 64 | */ |
| 65 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, | 65 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, |
| 66 | unsigned int subpg_index, unsigned long hidx) | 66 | unsigned int subpg_index, unsigned long hidx, |
| 67 | int offset) | ||
| 67 | { | 68 | { |
| 68 | return (hidx << H_PAGE_F_GIX_SHIFT) & | 69 | return (hidx << H_PAGE_F_GIX_SHIFT) & |
| 69 | (H_PAGE_F_SECOND | H_PAGE_F_GIX); | 70 | (H_PAGE_F_SECOND | H_PAGE_F_GIX); |
| @@ -45,7 +45,7 @@ · 44 unmodified lines | |||
| 45 | * generic accessors and iterators here | 45 | * generic accessors and iterators here |
| 46 | */ | 46 | */ |
| 47 | #define __real_pte __real_pte | 47 | #define __real_pte __real_pte |
| 48 | static inline real_pte_t __real_pte(pte_t pte, pte_t *ptep) | 48 | static inline real_pte_t __real_pte(pte_t pte, pte_t *ptep, int offset) |
| 49 | { | 49 | { |
| 50 | real_pte_t rpte; | 50 | real_pte_t rpte; |
| 51 | unsigned long *hidxp; | 51 | unsigned long *hidxp; |
| @@ -59,7 +59,7 @@ static inline real_pte_t __real_pte(pte_t pte, pte_t *ptep) · 7 unmodified lines | |||
| 59 | */ | 59 | */ |
| 60 | smp_rmb(); | 60 | smp_rmb(); |
| 61 | 61 | ||
| 62 | hidxp = (unsigned long *)(ptep + PTRS_PER_PTE); | 62 | hidxp = (unsigned long *)(ptep + offset); |
| 63 | rpte.hidx = *hidxp; | 63 | rpte.hidx = *hidxp; |
| 64 | return rpte; | 64 | return rpte; |
| 65 | } | 65 | } |
| @@ -86,9 +86,10 @@ static inline unsigned long __rpte_to_hidx(real_pte_t rpte, unsigned long index) · 20 unmodified lines | |||
| 86 | * expected to modify the PTE bits accordingly and commit the PTE to memory. | 86 | * expected to modify the PTE bits accordingly and commit the PTE to memory. |
| 87 | */ | 87 | */ |
| 88 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, | 88 | static inline unsigned long pte_set_hidx(pte_t *ptep, real_pte_t rpte, |
| 89 | unsigned int subpg_index, unsigned long hidx) | 89 | unsigned int subpg_index, |
| 90 | unsigned long hidx, int offset) | ||
| 90 | { | 91 | { |
| 91 | unsigned long *hidxp = (unsigned long *)(ptep + PTRS_PER_PTE); | 92 | unsigned long *hidxp = (unsigned long *)(ptep + offset); |
| 92 | 93 | ||
| 93 | rpte.hidx &= ~HIDX_BITS(0xfUL, subpg_index); | 94 | rpte.hidx &= ~HIDX_BITS(0xfUL, subpg_index); |
| 94 | *hidxp = rpte.hidx | HIDX_BITS(HIDX_SHIFT_BY_ONE(hidx), subpg_index); | 95 | *hidxp = rpte.hidx | HIDX_BITS(HIDX_SHIFT_BY_ONE(hidx), subpg_index); |
| @@ -140,13 +141,18 @@ static inline int hash__remap_4k_pfn(struct vm_area_struct *vma, unsigned long a · 45 unmodified lines | |||
| 140 | } | 141 | } |
| 141 | 142 | ||
| 142 | #define H_PTE_TABLE_SIZE PTE_FRAG_SIZE | 143 | #define H_PTE_TABLE_SIZE PTE_FRAG_SIZE |
| 143 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE | 144 | #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined (CONFIG_HUGETLB_PAGE) |
| 144 | #define H_PMD_TABLE_SIZE ((sizeof(pmd_t) << PMD_INDEX_SIZE) + \ | 145 | #define H_PMD_TABLE_SIZE ((sizeof(pmd_t) << PMD_INDEX_SIZE) + \ |
| 145 | (sizeof(unsigned long) << PMD_INDEX_SIZE)) | 146 | (sizeof(unsigned long) << PMD_INDEX_SIZE)) |
| 146 | #else | 147 | #else |
| 147 | #define H_PMD_TABLE_SIZE (sizeof(pmd_t) << PMD_INDEX_SIZE) | 148 | #define H_PMD_TABLE_SIZE (sizeof(pmd_t) << PMD_INDEX_SIZE) |
| 148 | #endif | 149 | #endif |
| 150 | #ifdef CONFIG_HUGETLB_PAGE | ||
| 151 | #define H_PUD_TABLE_SIZE ((sizeof(pud_t) << PUD_INDEX_SIZE) + \ | ||
| 152 | (sizeof(unsigned long) << PUD_INDEX_SIZE)) | ||
| 153 | #else | ||
| 149 | #define H_PUD_TABLE_SIZE (sizeof(pud_t) << PUD_INDEX_SIZE) | 154 | #define H_PUD_TABLE_SIZE (sizeof(pud_t) << PUD_INDEX_SIZE) |
| 155 | #endif | ||
| 150 | #define H_PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE) | 156 | #define H_PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE) |
| 151 | 157 | ||
| 152 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE | 158 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| @@ -23,7 +23,8 @@ · 22 unmodified lines | |||
| 23 | H_PUD_INDEX_SIZE + H_PGD_INDEX_SIZE + PAGE_SHIFT) | 23 | H_PUD_INDEX_SIZE + H_PGD_INDEX_SIZE + PAGE_SHIFT) |
| 24 | #define H_PGTABLE_RANGE (ASM_CONST(1) << H_PGTABLE_EADDR_SIZE) | 24 | #define H_PGTABLE_RANGE (ASM_CONST(1) << H_PGTABLE_EADDR_SIZE) |
| 25 | 25 | ||
| 26 | #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_PPC_64K_PAGES) | 26 | #if (defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE)) && \ |
| 27 | defined(CONFIG_PPC_64K_PAGES) | ||
| 27 | /* | 28 | /* |
| 28 | * only with hash 64k we need to use the second half of pmd page table | 29 | * only with hash 64k we need to use the second half of pmd page table |
| 29 | * to store pointer to deposited pgtable_t | 30 | * to store pointer to deposited pgtable_t |
| @@ -32,6 +33,16 @@ · 2 unmodified lines | |||
| 32 | #else | 33 | #else |
| 33 | #define H_PMD_CACHE_INDEX H_PMD_INDEX_SIZE | 34 | #define H_PMD_CACHE_INDEX H_PMD_INDEX_SIZE |
| 34 | #endif | 35 | #endif |
| 36 | /* | ||
| 37 | * We store the slot details in the second half of page table. | ||
| 38 | * Increase the pud level table so that hugetlb ptes can be stored | ||
| 39 | * at pud level. | ||
| 40 | */ | ||
| 41 | #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_PPC_64K_PAGES) | ||
| 42 | #define H_PUD_CACHE_INDEX (H_PUD_INDEX_SIZE + 1) | ||
| 43 | #else | ||
| 44 | #define H_PUD_CACHE_INDEX (H_PUD_INDEX_SIZE) | ||
| 45 | #endif | ||
| 35 | /* | 46 | /* |
| 36 | * Define the address range of the kernel non-linear virtual area | 47 | * Define the address range of the kernel non-linear virtual area |
| 37 | */ | 48 | */ |
| @@ -73,10 +73,16 @@ static inline void radix__pgd_free(struct mm_struct *mm, pgd_t *pgd) · 72 unmodified lines | |||
| 73 | 73 | ||
| 74 | static inline pgd_t *pgd_alloc(struct mm_struct *mm) | 74 | static inline pgd_t *pgd_alloc(struct mm_struct *mm) |
| 75 | { | 75 | { |
| 76 | pgd_t *pgd; | ||
| 77 | |||
| 76 | if (radix_enabled()) | 78 | if (radix_enabled()) |
| 77 | return radix__pgd_alloc(mm); | 79 | return radix__pgd_alloc(mm); |
| 78 | return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), | 80 | |
| 79 | pgtable_gfp_flags(mm, GFP_KERNEL)); | 81 | pgd = kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), |
| 82 | pgtable_gfp_flags(mm, GFP_KERNEL)); | ||
| 83 | memset(pgd, 0, PGD_TABLE_SIZE); | ||
| 84 | |||
| 85 | return pgd; | ||
| 80 | } | 86 | } |
| 81 | 87 | ||
| 82 | static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) | 88 | static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) |
| @@ -93,13 +99,13 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud) · 10 unmodified lines | |||
| 93 | 99 | ||
| 94 | static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) | 100 | static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) |
| 95 | { | 101 | { |
| 96 | return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), | 102 | return kmem_cache_alloc(PGT_CACHE(PUD_CACHE_INDEX), |
| 97 | pgtable_gfp_flags(mm, GFP_KERNEL)); | 103 | pgtable_gfp_flags(mm, GFP_KERNEL)); |
| 98 | } | 104 | } |
| 99 | 105 | ||
| 100 | static inline void pud_free(struct mm_struct *mm, pud_t *pud) | 106 | static inline void pud_free(struct mm_struct *mm, pud_t *pud) |
| 101 | { | 107 | { |
| 102 | kmem_cache_free(PGT_CACHE(PUD_INDEX_SIZE), pud); | 108 | kmem_cache_free(PGT_CACHE(PUD_CACHE_INDEX), pud); |
| 103 | } | 109 | } |
| 104 | 110 | ||
| 105 | static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) | 111 | static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) |
| @@ -115,7 +121,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, · 9 unmodified lines | |||
| 115 | * ahead and flush the page walk cache | 121 | * ahead and flush the page walk cache |
| 116 | */ | 122 | */ |
| 117 | flush_tlb_pgtable(tlb, address); | 123 | flush_tlb_pgtable(tlb, address); |
| 118 | pgtable_free_tlb(tlb, pud, PUD_INDEX_SIZE); | 124 | pgtable_free_tlb(tlb, pud, PUD_CACHE_INDEX); |
| 119 | } | 125 | } |
| 120 | 126 | ||
| 121 | static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) | 127 | static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) |
| @@ -232,11 +232,13 @@ extern unsigned long __pmd_index_size; · 231 unmodified lines | |||
| 232 | extern unsigned long __pud_index_size; | 232 | extern unsigned long __pud_index_size; |
| 233 | extern unsigned long __pgd_index_size; | 233 | extern unsigned long __pgd_index_size; |
| 234 | extern unsigned long __pmd_cache_index; | 234 | extern unsigned long __pmd_cache_index; |
| 235 | extern unsigned long __pud_cache_index; | ||
| 235 | #define PTE_INDEX_SIZE __pte_index_size | 236 | #define PTE_INDEX_SIZE __pte_index_size |
| 236 | #define PMD_INDEX_SIZE __pmd_index_size | 237 | #define PMD_INDEX_SIZE __pmd_index_size |
| 237 | #define PUD_INDEX_SIZE __pud_index_size | 238 | #define PUD_INDEX_SIZE __pud_index_size |
| 238 | #define PGD_INDEX_SIZE __pgd_index_size | 239 | #define PGD_INDEX_SIZE __pgd_index_size |
| 239 | #define PMD_CACHE_INDEX __pmd_cache_index | 240 | #define PMD_CACHE_INDEX __pmd_cache_index |
| 241 | #define PUD_CACHE_INDEX __pud_cache_index | ||
| 240 | /* | 242 | /* |
| 241 | * Because of use of pte fragments and THP, size of page table | 243 | * Because of use of pte fragments and THP, size of page table |
| 242 | * are not always derived out of index size above. | 244 | * are not always derived out of index size above. |
| @@ -348,7 +350,7 @@ extern unsigned long pci_io_base; · 105 unmodified lines | |||
| 348 | */ | 350 | */ |
| 349 | #ifndef __real_pte | 351 | #ifndef __real_pte |
| 350 | 352 | ||
| 351 | #define __real_pte(e,p) ((real_pte_t){(e)}) | 353 | #define __real_pte(e, p, o) ((real_pte_t){(e)}) |
| 352 | #define __rpte_to_pte(r) ((r).pte) | 354 | #define __rpte_to_pte(r) ((r).pte) |
| 353 | #define __rpte_to_hidx(r,index) (pte_val(__rpte_to_pte(r)) >> H_PAGE_F_GIX_SHIFT) | 355 | #define __rpte_to_hidx(r,index) (pte_val(__rpte_to_pte(r)) >> H_PAGE_F_GIX_SHIFT) |
| 354 | 356 | ||
| @@ -645,7 +645,7 @@ END_FTR_SECTION_NESTED(ftr,ftr,943) · 644 unmodified lines | |||
| 645 | EXC_HV, SOFTEN_TEST_HV, bitmask) | 645 | EXC_HV, SOFTEN_TEST_HV, bitmask) |
| 646 | 646 | ||
| 647 | #define MASKABLE_RELON_EXCEPTION_HV_OOL(vec, label, bitmask) \ | 647 | #define MASKABLE_RELON_EXCEPTION_HV_OOL(vec, label, bitmask) \ |
| 648 | MASKABLE_EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_NOTEST_HV, vec, bitmask);\ | 648 | MASKABLE_EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_TEST_HV, vec, bitmask);\ |
| 649 | EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_HV) | 649 | EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_HV) |
| 650 | 650 | ||
| 651 | /* | 651 | /* |
| @@ -52,7 +52,7 @@ · 51 unmodified lines | |||
| 52 | #define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000) | 52 | #define FW_FEATURE_TYPE1_AFFINITY ASM_CONST(0x0000000100000000) |
| 53 | #define FW_FEATURE_PRRN ASM_CONST(0x0000000200000000) | 53 | #define FW_FEATURE_PRRN ASM_CONST(0x0000000200000000) |
| 54 | #define FW_FEATURE_DRMEM_V2 ASM_CONST(0x0000000400000000) | 54 | #define FW_FEATURE_DRMEM_V2 ASM_CONST(0x0000000400000000) |
| 55 | #define FW_FEATURE_DRC_INFO ASM_CONST(0x0000000400000000) | 55 | #define FW_FEATURE_DRC_INFO ASM_CONST(0x0000000800000000) |
| 56 | 56 | ||
| 57 | #ifndef __ASSEMBLY__ | 57 | #ifndef __ASSEMBLY__ |
| 58 | 58 | ||
| @@ -29,6 +29,16 @@ · 28 unmodified lines | |||
| 29 | #define PACA_IRQ_HMI 0x20 | 29 | #define PACA_IRQ_HMI 0x20 |
| 30 | #define PACA_IRQ_PMI 0x40 | 30 | #define PACA_IRQ_PMI 0x40 |
| 31 | 31 | ||
| 32 | /* | ||
| 33 | * Some soft-masked interrupts must be hard masked until they are replayed | ||
| 34 | * (e.g., because the soft-masked handler does not clear the exception). | ||
| 35 | */ | ||
| 36 | #ifdef CONFIG_PPC_BOOK3S | ||
| 37 | #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE|PACA_IRQ_PMI) | ||
| 38 | #else | ||
| 39 | #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE) | ||
| 40 | #endif | ||
| 41 | |||
| 32 | /* | 42 | /* |
| 33 | * flags for paca->irq_soft_mask | 43 | * flags for paca->irq_soft_mask |
| 34 | */ | 44 | */ |
| @@ -244,7 +254,7 @@ static inline bool lazy_irq_pending(void) · 209 unmodified lines | |||
| 244 | static inline void may_hard_irq_enable(void) | 254 | static inline void may_hard_irq_enable(void) |
| 245 | { | 255 | { |
| 246 | get_paca()->irq_happened &= ~PACA_IRQ_HARD_DIS; | 256 | get_paca()->irq_happened &= ~PACA_IRQ_HARD_DIS; |
| 247 | if (!(get_paca()->irq_happened & PACA_IRQ_EE)) | 257 | if (!(get_paca()->irq_happened & PACA_IRQ_MUST_HARD_MASK)) |
| 248 | __hard_irq_enable(); | 258 | __hard_irq_enable(); |
| 249 | } | 259 | } |
| 250 | 260 | ||
| @@ -140,6 +140,12 @@ static inline bool kdump_in_progress(void) · 139 unmodified lines | |||
| 140 | return false; | 140 | return false; |
| 141 | } | 141 | } |
| 142 | 142 | ||
| 143 | static inline void crash_ipi_callback(struct pt_regs *regs) { } | ||
| 144 | |||
| 145 | static inline void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *)) | ||
| 146 | { | ||
| 147 | } | ||
| 148 | |||
| 143 | #endif /* CONFIG_KEXEC_CORE */ | 149 | #endif /* CONFIG_KEXEC_CORE */ |
| 144 | #endif /* ! __ASSEMBLY__ */ | 150 | #endif /* ! __ASSEMBLY__ */ |
| 145 | #endif /* __KERNEL__ */ | 151 | #endif /* __KERNEL__ */ |
| @@ -24,6 +24,7 @@ extern int icache_44x_need_flush; · 23 unmodified lines | |||
| 24 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) | 24 | #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) |
| 25 | 25 | ||
| 26 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE | 26 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE |
| 27 | #define PUD_CACHE_INDEX PUD_INDEX_SIZE | ||
| 27 | 28 | ||
| 28 | #ifndef __ASSEMBLY__ | 29 | #ifndef __ASSEMBLY__ |
| 29 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) | 30 | #define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE) |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | |||
| 27 | #else | 27 | #else |
| 28 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE | 28 | #define PMD_CACHE_INDEX PMD_INDEX_SIZE |
| 29 | #endif | 29 | #endif |
| 30 | #define PUD_CACHE_INDEX PUD_INDEX_SIZE | ||
| 30 | 31 | ||
| 31 | /* | 32 | /* |
| 32 | * Define the address range of the kernel non-linear virtual area | 33 | * Define the address range of the kernel non-linear virtual area |
| @@ -44,6 +44,11 @@ extern int sysfs_add_device_to_node(struct device *dev, int nid); · 43 unmodified lines | |||
| 44 | extern void sysfs_remove_device_from_node(struct device *dev, int nid); | 44 | extern void sysfs_remove_device_from_node(struct device *dev, int nid); |
| 45 | extern int numa_update_cpu_topology(bool cpus_locked); | 45 | extern int numa_update_cpu_topology(bool cpus_locked); |
| 46 | 46 | ||
| 47 | static inline void update_numa_cpu_lookup_table(unsigned int cpu, int node) | ||
| 48 | { | ||
| 49 | numa_cpu_lookup_table[cpu] = node; | ||
| 50 | } | ||
| 51 | |||
| 47 | static inline int early_cpu_to_node(int cpu) | 52 | static inline int early_cpu_to_node(int cpu) |
| 48 | { | 53 | { |
| 49 | int nid; | 54 | int nid; |
| @@ -76,12 +81,16 @@ static inline int numa_update_cpu_topology(bool cpus_locked) · 26 unmodified lines | |||
| 76 | { | 81 | { |
| 77 | return 0; | 82 | return 0; |
| 78 | } | 83 | } |
| 84 | |||
| 85 | static inline void update_numa_cpu_lookup_table(unsigned int cpu, int node) {} | ||
| 86 | |||
| 79 | #endif /* CONFIG_NUMA */ | 87 | #endif /* CONFIG_NUMA */ |
| 80 | 88 | ||
| 81 | #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR) | 89 | #if defined(CONFIG_NUMA) && defined(CONFIG_PPC_SPLPAR) |
| 82 | extern int start_topology_update(void); | 90 | extern int start_topology_update(void); |
| 83 | extern int stop_topology_update(void); | 91 | extern int stop_topology_update(void); |
| 84 | extern int prrn_is_enabled(void); | 92 | extern int prrn_is_enabled(void); |
| 93 | extern int find_and_online_cpu_nid(int cpu); | ||
| 85 | #else | 94 | #else |
| 86 | static inline int start_topology_update(void) | 95 | static inline int start_topology_update(void) |
| 87 | { | 96 | { |
| @@ -95,6 +104,10 @@ static inline int prrn_is_enabled(void) · 7 unmodified lines | |||
| 95 | { | 104 | { |
| 96 | return 0; | 105 | return 0; |
| 97 | } | 106 | } |
| 107 | static inline int find_and_online_cpu_nid(int cpu) | ||
| 108 | { | ||
| 109 | return 0; | ||
| 110 | } | ||
| 98 | #endif /* CONFIG_NUMA && CONFIG_PPC_SPLPAR */ | 111 | #endif /* CONFIG_NUMA && CONFIG_PPC_SPLPAR */ |
| 99 | 112 | ||
| 100 | #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_NEED_MULTIPLE_NODES) | 113 | #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_NEED_MULTIPLE_NODES) |
| @@ -384,7 +384,8 @@ static void *eeh_report_resume(void *data, void *userdata) · 383 unmodified lines | |||
| 384 | eeh_pcid_put(dev); | 384 | eeh_pcid_put(dev); |
| 385 | pci_uevent_ers(dev, PCI_ERS_RESULT_RECOVERED); | 385 | pci_uevent_ers(dev, PCI_ERS_RESULT_RECOVERED); |
| 386 | #ifdef CONFIG_PCI_IOV | 386 | #ifdef CONFIG_PCI_IOV |
| 387 | eeh_ops->notify_resume(eeh_dev_to_pdn(edev)); | 387 | if (eeh_ops->notify_resume && eeh_dev_to_pdn(edev)) |
| 388 | eeh_ops->notify_resume(eeh_dev_to_pdn(edev)); | ||
| 388 | #endif | 389 | #endif |
| 389 | return NULL; | 390 | return NULL; |
| 390 | } | 391 | } |
| @@ -943,6 +943,8 @@ kernel_dbg_exc: · 942 unmodified lines | |||
| 943 | /* | 943 | /* |
| 944 | * An interrupt came in while soft-disabled; We mark paca->irq_happened | 944 | * An interrupt came in while soft-disabled; We mark paca->irq_happened |
| 945 | * accordingly and if the interrupt is level sensitive, we hard disable | 945 | * accordingly and if the interrupt is level sensitive, we hard disable |
| 946 | * hard disable (full_mask) corresponds to PACA_IRQ_MUST_HARD_MASK, so | ||
| 947 | * keep these in synch. | ||
| 946 | */ | 948 | */ |
| 947 | 949 | ||
| 948 | .macro masked_interrupt_book3e paca_irq full_mask | 950 | .macro masked_interrupt_book3e paca_irq full_mask |
| @@ -1426,7 +1426,7 @@ EXC_COMMON_BEGIN(soft_nmi_common) · 1425 unmodified lines | |||
| 1426 | * triggered and won't automatically refire. | 1426 | * triggered and won't automatically refire. |
| 1427 | * - If it was a HMI we return immediately since we handled it in realmode | 1427 | * - If it was a HMI we return immediately since we handled it in realmode |
| 1428 | * and it won't refire. | 1428 | * and it won't refire. |
| 1429 | * - else we hard disable and return. | 1429 | * - Else it is one of PACA_IRQ_MUST_HARD_MASK, so hard disable and return. |
| 1430 | * This is called with r10 containing the value to OR to the paca field. | 1430 | * This is called with r10 containing the value to OR to the paca field. |
| 1431 | */ | 1431 | */ |
| 1432 | #define MASKED_INTERRUPT(_H) \ | 1432 | #define MASKED_INTERRUPT(_H) \ |
| @@ -1441,8 +1441,8 @@ masked_##_H##interrupt: \ · 8 unmodified lines | |||
| 1441 | ori r10,r10,0xffff; \ | 1441 | ori r10,r10,0xffff; \ |
| 1442 | mtspr SPRN_DEC,r10; \ | 1442 | mtspr SPRN_DEC,r10; \ |
| 1443 | b MASKED_DEC_HANDLER_LABEL; \ | 1443 | b MASKED_DEC_HANDLER_LABEL; \ |
| 1444 | 1: andi. r10,r10,(PACA_IRQ_DBELL|PACA_IRQ_HMI); \ | 1444 | 1: andi. r10,r10,PACA_IRQ_MUST_HARD_MASK; \ |
| 1445 | bne 2f; \ | 1445 | beq 2f; \ |
| 1446 | mfspr r10,SPRN_##_H##SRR1; \ | 1446 | mfspr r10,SPRN_##_H##SRR1; \ |
| 1447 | xori r10,r10,MSR_EE; /* clear MSR_EE */ \ | 1447 | xori r10,r10,MSR_EE; /* clear MSR_EE */ \ |
| 1448 | mtspr SPRN_##_H##SRR1,r10; \ | 1448 | mtspr SPRN_##_H##SRR1,r10; \ |
| @@ -874,7 +874,6 @@ struct ibm_arch_vec __cacheline_aligned ibm_architecture_vec = { · 873 unmodified lines | |||
| 874 | .mmu = 0, | 874 | .mmu = 0, |
| 875 | .hash_ext = 0, | 875 | .hash_ext = 0, |
| 876 | .radix_ext = 0, | 876 | .radix_ext = 0, |
| 877 | .byte22 = OV5_FEAT(OV5_DRC_INFO), | ||
| 878 | }, | 877 | }, |
| 879 | 878 | ||
| 880 | /* option vector 6: IBM PAPR hints */ | 879 | /* option vector 6: IBM PAPR hints */ |
| @@ -788,7 +788,8 @@ static int register_cpu_online(unsigned int cpu) · 787 unmodified lines | |||
| 788 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) | 788 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) |
| 789 | device_create_file(s, &dev_attr_pir); | 789 | device_create_file(s, &dev_attr_pir); |
| 790 | 790 | ||
| 791 | if (cpu_has_feature(CPU_FTR_ARCH_206)) | 791 | if (cpu_has_feature(CPU_FTR_ARCH_206) && |
| 792 | !firmware_has_feature(FW_FEATURE_LPAR)) | ||
| 792 | device_create_file(s, &dev_attr_tscr); | 793 | device_create_file(s, &dev_attr_tscr); |
| 793 | #endif /* CONFIG_PPC64 */ | 794 | #endif /* CONFIG_PPC64 */ |
| 794 | 795 | ||
| @@ -873,7 +874,8 @@ static int unregister_cpu_online(unsigned int cpu) · 78 unmodified lines | |||
| 873 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) | 874 | if (cpu_has_feature(CPU_FTR_PPCAS_ARCH_V2)) |
| 874 | device_remove_file(s, &dev_attr_pir); | 875 | device_remove_file(s, &dev_attr_pir); |
| 875 | 876 | ||
| 876 | if (cpu_has_feature(CPU_FTR_ARCH_206)) | 877 | if (cpu_has_feature(CPU_FTR_ARCH_206) && |
| 878 | !firmware_has_feature(FW_FEATURE_LPAR)) | ||
| 877 | device_remove_file(s, &dev_attr_tscr); | 879 | device_remove_file(s, &dev_attr_tscr); |
| 878 | #endif /* CONFIG_PPC64 */ | 880 | #endif /* CONFIG_PPC64 */ |
| 879 | 881 | ||
| @@ -195,6 +195,12 @@ static void kvmppc_pte_free(pte_t *ptep) · 194 unmodified lines | |||
| 195 | kmem_cache_free(kvm_pte_cache, ptep); | 195 | kmem_cache_free(kvm_pte_cache, ptep); |
| 196 | } | 196 | } |
| 197 | 197 | ||
| 198 | /* Like pmd_huge() and pmd_large(), but works regardless of config options */ | ||
| 199 | static inline int pmd_is_leaf(pmd_t pmd) | ||
| 200 | { | ||
| 201 | return !!(pmd_val(pmd) & _PAGE_PTE); | ||
| 202 | } | ||
| 203 | |||
| 198 | static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, | 204 | static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, |
| 199 | unsigned int level, unsigned long mmu_seq) | 205 | unsigned int level, unsigned long mmu_seq) |
| 200 | { | 206 | { |
| @@ -219,7 +225,7 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, · 18 unmodified lines | |||
| 219 | else | 225 | else |
| 220 | new_pmd = pmd_alloc_one(kvm->mm, gpa); | 226 | new_pmd = pmd_alloc_one(kvm->mm, gpa); |
| 221 | 227 | ||
| 222 | if (level == 0 && !(pmd && pmd_present(*pmd))) | 228 | if (level == 0 && !(pmd && pmd_present(*pmd) && !pmd_is_leaf(*pmd))) |
| 223 | new_ptep = kvmppc_pte_alloc(); | 229 | new_ptep = kvmppc_pte_alloc(); |
| 224 | 230 | ||
| 225 | /* Check if we might have been invalidated; let the guest retry if so */ | 231 | /* Check if we might have been invalidated; let the guest retry if so */ |
| @@ -244,12 +250,30 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, · 18 unmodified lines | |||
| 244 | new_pmd = NULL; | 250 | new_pmd = NULL; |
| 245 | } | 251 | } |
| 246 | pmd = pmd_offset(pud, gpa); | 252 | pmd = pmd_offset(pud, gpa); |
| 247 | if (pmd_large(*pmd)) { | 253 | if (pmd_is_leaf(*pmd)) { |
| 248 | /* Someone else has instantiated a large page here; retry */ | 254 | unsigned long lgpa = gpa & PMD_MASK; |
| 249 | ret = -EAGAIN; | 255 | |
| 250 | goto out_unlock; | 256 | /* |
| 251 | } | 257 | * If we raced with another CPU which has just put |
| 252 | if (level == 1 && !pmd_none(*pmd)) { | 258 | * a 2MB pte in after we saw a pte page, try again. |
| 259 | */ | ||
| 260 | if (level == 0 && !new_ptep) { | ||
| 261 | ret = -EAGAIN; | ||
| 262 | goto out_unlock; | ||
| 263 | } | ||
| 264 | /* Valid 2MB page here already, remove it */ | ||
| 265 | old = kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), | ||
| 266 | ~0UL, 0, lgpa, PMD_SHIFT); | ||
| 267 | kvmppc_radix_tlbie_page(kvm, lgpa, PMD_SHIFT); | ||
| 268 | if (old & _PAGE_DIRTY) { | ||
| 269 | unsigned long gfn = lgpa >> PAGE_SHIFT; | ||
| 270 | struct kvm_memory_slot *memslot; | ||
| 271 | memslot = gfn_to_memslot(kvm, gfn); | ||
| 272 | if (memslot && memslot->dirty_bitmap) | ||
| 273 | kvmppc_update_dirty_map(memslot, | ||
| 274 | gfn, PMD_SIZE); | ||
| 275 | } | ||
| 276 | } else if (level == 1 && !pmd_none(*pmd)) { | ||
| 253 | /* | 277 | /* |
| 254 | * There's a page table page here, but we wanted | 278 | * There's a page table page here, but we wanted |
| 255 | * to install a large page. Tell the caller and let | 279 | * to install a large page. Tell the caller and let |
| @@ -412,28 +436,24 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, · 156 unmodified lines | |||
| 412 | } else { | 436 | } else { |
| 413 | page = pages[0]; | 437 | page = pages[0]; |
| 414 | pfn = page_to_pfn(page); | 438 | pfn = page_to_pfn(page); |
| 415 | if (PageHuge(page)) { | 439 | if (PageCompound(page)) { |
| 416 | page = compound_head(page); | 440 | pte_size <<= compound_order(compound_head(page)); |
| 417 | pte_size <<= compound_order(page); | ||
| 418 | /* See if we can insert a 2MB large-page PTE here */ | 441 | /* See if we can insert a 2MB large-page PTE here */ |
| 419 | if (pte_size >= PMD_SIZE && | 442 | if (pte_size >= PMD_SIZE && |
| 420 | (gpa & PMD_MASK & PAGE_MASK) == | 443 | (gpa & (PMD_SIZE - PAGE_SIZE)) == |
| 421 | (hva & PMD_MASK & PAGE_MASK)) { | 444 | (hva & (PMD_SIZE - PAGE_SIZE))) { |
| 422 | level = 1; | 445 | level = 1; |
| 423 | pfn &= ~((PMD_SIZE >> PAGE_SHIFT) - 1); | 446 | pfn &= ~((PMD_SIZE >> PAGE_SHIFT) - 1); |
| 424 | } | 447 | } |
| 425 | } | 448 | } |
| 426 | /* See if we can provide write access */ | 449 | /* See if we can provide write access */ |
| 427 | if (writing) { | 450 | if (writing) { |
| 428 | /* | ||
| 429 | * We assume gup_fast has set dirty on the host PTE. | ||
| 430 | */ | ||
| 431 | pgflags |= _PAGE_WRITE; | 451 | pgflags |= _PAGE_WRITE; |
| 432 | } else { | 452 | } else { |
| 433 | local_irq_save(flags); | 453 | local_irq_save(flags); |
| 434 | ptep = find_current_mm_pte(current->mm->pgd, | 454 | ptep = find_current_mm_pte(current->mm->pgd, |
| 435 | hva, NULL, NULL); | 455 | hva, NULL, NULL); |
| 436 | if (ptep && pte_write(*ptep) && pte_dirty(*ptep)) | 456 | if (ptep && pte_write(*ptep)) |
| 437 | pgflags |= _PAGE_WRITE; | 457 | pgflags |= _PAGE_WRITE; |
| 438 | local_irq_restore(flags); | 458 | local_irq_restore(flags); |
| 439 | } | 459 | } |
| @@ -459,18 +479,15 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, · 19 unmodified lines | |||
| 459 | pte = pfn_pte(pfn, __pgprot(pgflags)); | 479 | pte = pfn_pte(pfn, __pgprot(pgflags)); |
| 460 | ret = kvmppc_create_pte(kvm, pte, gpa, level, mmu_seq); | 480 | ret = kvmppc_create_pte(kvm, pte, gpa, level, mmu_seq); |
| 461 | } | 481 | } |
| 462 | if (ret == 0 || ret == -EAGAIN) | ||
| 463 | ret = RESUME_GUEST; | ||
| 464 | 482 | ||
| 465 | if (page) { | 483 | if (page) { |
| 466 | /* | 484 | if (!ret && (pgflags & _PAGE_WRITE)) |
| 467 | * We drop pages[0] here, not page because page might | 485 | set_page_dirty_lock(page); |
| 468 | * have been set to the head page of a compound, but | 486 | put_page(page); |
| 469 | * we have to drop the reference on the correct tail | ||
| 470 | * page to match the get inside gup() | ||
| 471 | */ | ||
| 472 | put_page(pages[0]); | ||
| 473 | } | 487 | } |
| 488 | |||
| 489 | if (ret == 0 || ret == -EAGAIN) | ||
| 490 | ret = RESUME_GUEST; | ||
| 474 | return ret; | 491 | return ret; |
| 475 | } | 492 | } |
| 476 | 493 | ||
| @@ -644,7 +661,7 @@ void kvmppc_free_radix(struct kvm *kvm) · 167 unmodified lines | |||
| 644 | continue; | 661 | continue; |
| 645 | pmd = pmd_offset(pud, 0); | 662 | pmd = pmd_offset(pud, 0); |
| 646 | for (im = 0; im < PTRS_PER_PMD; ++im, ++pmd) { | 663 | for (im = 0; im < PTRS_PER_PMD; ++im, ++pmd) { |
| 647 | if (pmd_huge(*pmd)) { | 664 | if (pmd_is_leaf(*pmd)) { |
| 648 | pmd_clear(pmd); | 665 | pmd_clear(pmd); |
| 649 | continue; | 666 | continue; |
| 650 | } | 667 | } |
| @@ -2885,7 +2885,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) · 2884 unmodified lines | |||
| 2885 | */ | 2885 | */ |
| 2886 | trace_hardirqs_on(); | 2886 | trace_hardirqs_on(); |
| 2887 | 2887 | ||
| 2888 | guest_enter(); | 2888 | guest_enter_irqoff(); |
| 2889 | 2889 | ||
| 2890 | srcu_idx = srcu_read_lock(&vc->kvm->srcu); | 2890 | srcu_idx = srcu_read_lock(&vc->kvm->srcu); |
| 2891 | 2891 | ||
| @@ -2893,8 +2893,6 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) · 1 unmodified line | |||
| 2893 | 2893 | ||
| 2894 | srcu_read_unlock(&vc->kvm->srcu, srcu_idx); | 2894 | srcu_read_unlock(&vc->kvm->srcu, srcu_idx); |
| 2895 | 2895 | ||
| 2896 | guest_exit(); | ||
| 2897 | |||
| 2898 | trace_hardirqs_off(); | 2896 | trace_hardirqs_off(); |
| 2899 | set_irq_happened(trap); | 2897 | set_irq_happened(trap); |
| 2900 | 2898 | ||
| @@ -2937,6 +2935,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) · 36 unmodified lines | |||
| 2937 | kvmppc_set_host_core(pcpu); | 2935 | kvmppc_set_host_core(pcpu); |
| 2938 | 2936 | ||
| 2939 | local_irq_enable(); | 2937 | local_irq_enable(); |
| 2938 | guest_exit(); | ||
| 2940 | 2939 | ||
| 2941 | /* Let secondaries go back to the offline loop */ | 2940 | /* Let secondaries go back to the offline loop */ |
| 2942 | for (i = 0; i < controlled_threads; ++i) { | 2941 | for (i = 0; i < controlled_threads; ++i) { |
| @@ -3656,15 +3655,17 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu) · 713 unmodified lines | |||
| 3656 | goto up_out; | 3655 | goto up_out; |
| 3657 | 3656 | ||
| 3658 | psize = vma_kernel_pagesize(vma); | 3657 | psize = vma_kernel_pagesize(vma); |
| 3659 | porder = __ilog2(psize); | ||
| 3660 | 3658 | ||
| 3661 | up_read(¤t->mm->mmap_sem); | 3659 | up_read(¤t->mm->mmap_sem); |
| 3662 | 3660 | ||
| 3663 | /* We can handle 4k, 64k or 16M pages in the VRMA */ | 3661 | /* We can handle 4k, 64k or 16M pages in the VRMA */ |
| 3664 | err = -EINVAL; | 3662 | if (psize >= 0x1000000) |
| 3665 | if (!(psize == 0x1000 || psize == 0x10000 || | 3663 | psize = 0x1000000; |
| 3666 | psize == 0x1000000)) | 3664 | else if (psize >= 0x10000) |
| 3667 | goto out_srcu; | 3665 | psize = 0x10000; |
| 3666 | else | ||
| 3667 | psize = 0x1000; | ||
| 3668 | porder = __ilog2(psize); | ||
| 3668 | 3669 | ||
| 3669 | senc = slb_pgsize_encoding(psize); | 3670 | senc = slb_pgsize_encoding(psize); |
| 3670 | kvm->arch.vrma_slb_v = senc | SLB_VSID_B_1T | | 3671 | kvm->arch.vrma_slb_v = senc | SLB_VSID_B_1T | |
| @@ -188,7 +188,7 @@ static int xive_provision_queue(struct kvm_vcpu *vcpu, u8 prio) · 187 unmodified lines | |||
| 188 | if (!qpage) { | 188 | if (!qpage) { |
| 189 | pr_err("Failed to allocate queue %d for VCPU %d\n", | 189 | pr_err("Failed to allocate queue %d for VCPU %d\n", |
| 190 | prio, xc->server_num); | 190 | prio, xc->server_num); |
| 191 | return -ENOMEM;; | 191 | return -ENOMEM; |
| 192 | } | 192 | } |
| 193 | memset(qpage, 0, 1 << xive->q_order); | 193 | memset(qpage, 0, 1 << xive->q_order); |
| 194 | 194 | ||
| @@ -1345,7 +1345,7 @@ static int kvmppc_emulate_mmio_vsx_loadstore(struct kvm_vcpu *vcpu, · 1344 unmodified lines | |||
| 1345 | int kvmppc_handle_load128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, | 1345 | int kvmppc_handle_load128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, |
| 1346 | unsigned int rt, int is_default_endian) | 1346 | unsigned int rt, int is_default_endian) |
| 1347 | { | 1347 | { |
| 1348 | enum emulation_result emulated; | 1348 | enum emulation_result emulated = EMULATE_DONE; |
| 1349 | 1349 | ||
| 1350 | while (vcpu->arch.mmio_vmx_copy_nums) { | 1350 | while (vcpu->arch.mmio_vmx_copy_nums) { |
| 1351 | emulated = __kvmppc_handle_load(run, vcpu, rt, 8, | 1351 | emulated = __kvmppc_handle_load(run, vcpu, rt, 8, |
| @@ -1608,7 +1608,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · 256 unmodified lines | |||
| 1608 | 1608 | ||
| 1609 | kvm_sigset_deactivate(vcpu); | 1609 | kvm_sigset_deactivate(vcpu); |
| 1610 | 1610 | ||
| 1611 | #ifdef CONFIG_ALTIVEC | ||
| 1611 | out: | 1612 | out: |
| 1613 | #endif | ||
| 1612 | vcpu_put(vcpu); | 1614 | vcpu_put(vcpu); |
| 1613 | return r; | 1615 | return r; |
| 1614 | } | 1616 | } |
| @@ -98,7 +98,7 @@ static void init_drconf_v2_cell(struct of_drconf_cell_v2 *dr_cell, · 97 unmodified lines | |||
| 98 | dr_cell->base_addr = cpu_to_be64(lmb->base_addr); | 98 | dr_cell->base_addr = cpu_to_be64(lmb->base_addr); |
| 99 | dr_cell->drc_index = cpu_to_be32(lmb->drc_index); | 99 | dr_cell->drc_index = cpu_to_be32(lmb->drc_index); |
| 100 | dr_cell->aa_index = cpu_to_be32(lmb->aa_index); | 100 | dr_cell->aa_index = cpu_to_be32(lmb->aa_index); |
| 101 | dr_cell->flags = cpu_to_be32(lmb->flags); | 101 | dr_cell->flags = cpu_to_be32(drmem_lmb_flags(lmb)); |
| 102 | } | 102 | } |
| 103 | 103 | ||
| 104 | static int drmem_update_dt_v2(struct device_node *memory, | 104 | static int drmem_update_dt_v2(struct device_node *memory, |
| @@ -121,7 +121,7 @@ static int drmem_update_dt_v2(struct device_node *memory, · 16 unmodified lines | |||
| 121 | } | 121 | } |
| 122 | 122 | ||
| 123 | if (prev_lmb->aa_index != lmb->aa_index || | 123 | if (prev_lmb->aa_index != lmb->aa_index || |
| 124 | prev_lmb->flags != lmb->flags) | 124 | drmem_lmb_flags(prev_lmb) != drmem_lmb_flags(lmb)) |
| 125 | lmb_sets++; | 125 | lmb_sets++; |
| 126 | 126 | ||
| 127 | prev_lmb = lmb; | 127 | prev_lmb = lmb; |
| @@ -150,7 +150,7 @@ static int drmem_update_dt_v2(struct device_node *memory, · 22 unmodified lines | |||
| 150 | } | 150 | } |
| 151 | 151 | ||
| 152 | if (prev_lmb->aa_index != lmb->aa_index || | 152 | if (prev_lmb->aa_index != lmb->aa_index || |
| 153 | prev_lmb->flags != lmb->flags) { | 153 | drmem_lmb_flags(prev_lmb) != drmem_lmb_flags(lmb)) { |
| 154 | /* end of one set, start of another */ | 154 | /* end of one set, start of another */ |
| 155 | dr_cell->seq_lmbs = cpu_to_be32(seq_lmbs); | 155 | dr_cell->seq_lmbs = cpu_to_be32(seq_lmbs); |
| 156 | dr_cell++; | 156 | dr_cell++; |
| @@ -216,6 +216,8 @@ static void __init __walk_drmem_v1_lmbs(const __be32 *prop, const __be32 *usm, · 59 unmodified lines | |||
| 216 | u32 i, n_lmbs; | 216 | u32 i, n_lmbs; |
| 217 | 217 | ||
| 218 | n_lmbs = of_read_number(prop++, 1); | 218 | n_lmbs = of_read_number(prop++, 1); |
| 219 | if (n_lmbs == 0) | ||
| 220 | return; | ||
| 219 | 221 | ||
| 220 | for (i = 0; i < n_lmbs; i++) { | 222 | for (i = 0; i < n_lmbs; i++) { |
| 221 | read_drconf_v1_cell(&lmb, &prop); | 223 | read_drconf_v1_cell(&lmb, &prop); |
| @@ -245,6 +247,8 @@ static void __init __walk_drmem_v2_lmbs(const __be32 *prop, const __be32 *usm, · 23 unmodified lines | |||
| 245 | u32 i, j, lmb_sets; | 247 | u32 i, j, lmb_sets; |
| 246 | 248 | ||
| 247 | lmb_sets = of_read_number(prop++, 1); | 249 | lmb_sets = of_read_number(prop++, 1); |
| 250 | if (lmb_sets == 0) | ||
| 251 | return; | ||
| 248 | 252 | ||
| 249 | for (i = 0; i < lmb_sets; i++) { | 253 | for (i = 0; i < lmb_sets; i++) { |
| 250 | read_drconf_v2_cell(&dr_cell, &prop); | 254 | read_drconf_v2_cell(&dr_cell, &prop); |
| @@ -354,6 +358,8 @@ static void __init init_drmem_v1_lmbs(const __be32 *prop) · 103 unmodified lines | |||
| 354 | struct drmem_lmb *lmb; | 358 | struct drmem_lmb *lmb; |
| 355 | 359 | ||
| 356 | drmem_info->n_lmbs = of_read_number(prop++, 1); | 360 | drmem_info->n_lmbs = of_read_number(prop++, 1); |
| 361 | if (drmem_info->n_lmbs == 0) | ||
| 362 | return; | ||
| 357 | 363 | ||
| 358 | drmem_info->lmbs = kcalloc(drmem_info->n_lmbs, sizeof(*lmb), | 364 | drmem_info->lmbs = kcalloc(drmem_info->n_lmbs, sizeof(*lmb), |
| 359 | GFP_KERNEL); | 365 | GFP_KERNEL); |
| @@ -373,6 +379,8 @@ static void __init init_drmem_v2_lmbs(const __be32 *prop) · 13 unmodified lines | |||
| 373 | int lmb_index; | 379 | int lmb_index; |
| 374 | 380 | ||
| 375 | lmb_sets = of_read_number(prop++, 1); | 381 | lmb_sets = of_read_number(prop++, 1); |
| 382 | if (lmb_sets == 0) | ||
| 383 | return; | ||
| 376 | 384 | ||
| 377 | /* first pass, calculate the number of LMBs */ | 385 | /* first pass, calculate the number of LMBs */ |
| 378 | p = prop; | 386 | p = prop; |
| @@ -55,7 +55,7 @@ int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, · 54 unmodified lines | |||
| 55 | * need to add in 0x1 if it's a read-only user page | 55 | * need to add in 0x1 if it's a read-only user page |
| 56 | */ | 56 | */ |
| 57 | rflags = htab_convert_pte_flags(new_pte); | 57 | rflags = htab_convert_pte_flags(new_pte); |
| 58 | rpte = __real_pte(__pte(old_pte), ptep); | 58 | rpte = __real_pte(__pte(old_pte), ptep, PTRS_PER_PTE); |
| 59 | 59 | ||
| 60 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && | 60 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && |
| 61 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) | 61 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) |
| @@ -117,7 +117,7 @@ repeat: · 55 unmodified lines | |||
| 117 | return -1; | 117 | return -1; |
| 118 | } | 118 | } |
| 119 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; | 119 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; |
| 120 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot); | 120 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot, PTRS_PER_PTE); |
| 121 | } | 121 | } |
| 122 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); | 122 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); |
| 123 | return 0; | 123 | return 0; |
| @@ -86,7 +86,7 @@ int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, · 85 unmodified lines | |||
| 86 | 86 | ||
| 87 | subpg_index = (ea & (PAGE_SIZE - 1)) >> shift; | 87 | subpg_index = (ea & (PAGE_SIZE - 1)) >> shift; |
| 88 | vpn = hpt_vpn(ea, vsid, ssize); | 88 | vpn = hpt_vpn(ea, vsid, ssize); |
| 89 | rpte = __real_pte(__pte(old_pte), ptep); | 89 | rpte = __real_pte(__pte(old_pte), ptep, PTRS_PER_PTE); |
| 90 | /* | 90 | /* |
| 91 | *None of the sub 4k page is hashed | 91 | *None of the sub 4k page is hashed |
| 92 | */ | 92 | */ |
| @@ -214,7 +214,7 @@ repeat: · 121 unmodified lines | |||
| 214 | return -1; | 214 | return -1; |
| 215 | } | 215 | } |
| 216 | 216 | ||
| 217 | new_pte |= pte_set_hidx(ptep, rpte, subpg_index, slot); | 217 | new_pte |= pte_set_hidx(ptep, rpte, subpg_index, slot, PTRS_PER_PTE); |
| 218 | new_pte |= H_PAGE_HASHPTE; | 218 | new_pte |= H_PAGE_HASHPTE; |
| 219 | 219 | ||
| 220 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); | 220 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); |
| @@ -262,7 +262,7 @@ int __hash_page_64K(unsigned long ea, unsigned long access, · 41 unmodified lines | |||
| 262 | } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); | 262 | } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); |
| 263 | 263 | ||
| 264 | rflags = htab_convert_pte_flags(new_pte); | 264 | rflags = htab_convert_pte_flags(new_pte); |
| 265 | rpte = __real_pte(__pte(old_pte), ptep); | 265 | rpte = __real_pte(__pte(old_pte), ptep, PTRS_PER_PTE); |
| 266 | 266 | ||
| 267 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && | 267 | if (cpu_has_feature(CPU_FTR_NOEXECUTE) && |
| 268 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) | 268 | !cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) |
| @@ -327,7 +327,7 @@ repeat: · 58 unmodified lines | |||
| 327 | } | 327 | } |
| 328 | 328 | ||
| 329 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; | 329 | new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE; |
| 330 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot); | 330 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot, PTRS_PER_PTE); |
| 331 | } | 331 | } |
| 332 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); | 332 | *ptep = __pte(new_pte & ~H_PAGE_BUSY); |
| 333 | return 0; | 333 | return 0; |
| @@ -1008,6 +1008,7 @@ void __init hash__early_init_mmu(void) · 1007 unmodified lines | |||
| 1008 | __pmd_index_size = H_PMD_INDEX_SIZE; | 1008 | __pmd_index_size = H_PMD_INDEX_SIZE; |
| 1009 | __pud_index_size = H_PUD_INDEX_SIZE; | 1009 | __pud_index_size = H_PUD_INDEX_SIZE; |
| 1010 | __pgd_index_size = H_PGD_INDEX_SIZE; | 1010 | __pgd_index_size = H_PGD_INDEX_SIZE; |
| 1011 | __pud_cache_index = H_PUD_CACHE_INDEX; | ||
| 1011 | __pmd_cache_index = H_PMD_CACHE_INDEX; | 1012 | __pmd_cache_index = H_PMD_CACHE_INDEX; |
| 1012 | __pte_table_size = H_PTE_TABLE_SIZE; | 1013 | __pte_table_size = H_PTE_TABLE_SIZE; |
| 1013 | __pmd_table_size = H_PMD_TABLE_SIZE; | 1014 | __pmd_table_size = H_PMD_TABLE_SIZE; |
| @@ -27,7 +27,7 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 26 unmodified lines | |||
| 27 | unsigned long vpn; | 27 | unsigned long vpn; |
| 28 | unsigned long old_pte, new_pte; | 28 | unsigned long old_pte, new_pte; |
| 29 | unsigned long rflags, pa, sz; | 29 | unsigned long rflags, pa, sz; |
| 30 | long slot; | 30 | long slot, offset; |
| 31 | 31 | ||
| 32 | BUG_ON(shift != mmu_psize_defs[mmu_psize].shift); | 32 | BUG_ON(shift != mmu_psize_defs[mmu_psize].shift); |
| 33 | 33 | ||
| @@ -63,7 +63,11 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 29 unmodified lines | |||
| 63 | } while(!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); | 63 | } while(!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); |
| 64 | 64 | ||
| 65 | rflags = htab_convert_pte_flags(new_pte); | 65 | rflags = htab_convert_pte_flags(new_pte); |
| 66 | rpte = __real_pte(__pte(old_pte), ptep); | 66 | if (unlikely(mmu_psize == MMU_PAGE_16G)) |
| 67 | offset = PTRS_PER_PUD; | ||
| 68 | else | ||
| 69 | offset = PTRS_PER_PMD; | ||
| 70 | rpte = __real_pte(__pte(old_pte), ptep, offset); | ||
| 67 | 71 | ||
| 68 | sz = ((1UL) << shift); | 72 | sz = ((1UL) << shift); |
| 69 | if (!cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) | 73 | if (!cpu_has_feature(CPU_FTR_COHERENT_ICACHE)) |
| @@ -104,7 +108,7 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 34 unmodified lines | |||
| 104 | return -1; | 108 | return -1; |
| 105 | } | 109 | } |
| 106 | 110 | ||
| 107 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot); | 111 | new_pte |= pte_set_hidx(ptep, rpte, 0, slot, offset); |
| 108 | } | 112 | } |
| 109 | 113 | ||
| 110 | /* | 114 | /* |
| @@ -100,6 +100,6 @@ void pgtable_cache_init(void) · 99 unmodified lines | |||
| 100 | * same size as either the pgd or pmd index except with THP enabled | 100 | * same size as either the pgd or pmd index except with THP enabled |
| 101 | * on book3s 64 | 101 | * on book3s 64 |
| 102 | */ | 102 | */ |
| 103 | if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE)) | 103 | if (PUD_CACHE_INDEX && !PGT_CACHE(PUD_CACHE_INDEX)) |
| 104 | pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor); | 104 | pgtable_cache_add(PUD_CACHE_INDEX, pud_ctor); |
| 105 | } | 105 | } |
| @@ -143,11 +143,6 @@ static void reset_numa_cpu_lookup_table(void) · 142 unmodified lines | |||
| 143 | numa_cpu_lookup_table[cpu] = -1; | 143 | numa_cpu_lookup_table[cpu] = -1; |
| 144 | } | 144 | } |
| 145 | 145 | ||
| 146 | static void update_numa_cpu_lookup_table(unsigned int cpu, int node) | ||
| 147 | { | ||
| 148 | numa_cpu_lookup_table[cpu] = node; | ||
| 149 | } | ||
| 150 | |||
| 151 | static void map_cpu_to_node(int cpu, int node) | 146 | static void map_cpu_to_node(int cpu, int node) |
| 152 | { | 147 | { |
| 153 | update_numa_cpu_lookup_table(cpu, node); | 148 | update_numa_cpu_lookup_table(cpu, node); |
| @@ -17,9 +17,11 @@ · 16 unmodified lines | |||
| 17 | #include <linux/of_fdt.h> | 17 | #include <linux/of_fdt.h> |
| 18 | #include <linux/mm.h> | 18 | #include <linux/mm.h> |
| 19 | #include <linux/string_helpers.h> | 19 | #include <linux/string_helpers.h> |
| 20 | #include <linux/stop_machine.h> | ||
| 20 | 21 | ||
| 21 | #include <asm/pgtable.h> | 22 | #include <asm/pgtable.h> |
| 22 | #include <asm/pgalloc.h> | 23 | #include <asm/pgalloc.h> |
| 24 | #include <asm/mmu_context.h> | ||
| 23 | #include <asm/dma.h> | 25 | #include <asm/dma.h> |
| 24 | #include <asm/machdep.h> | 26 | #include <asm/machdep.h> |
| 25 | #include <asm/mmu.h> | 27 | #include <asm/mmu.h> |
| @@ -333,6 +335,22 @@ static void __init radix_init_pgtable(void) · 307 unmodified lines | |||
| 333 | "r" (TLBIEL_INVAL_SET_LPID), "r" (0)); | 335 | "r" (TLBIEL_INVAL_SET_LPID), "r" (0)); |
| 334 | asm volatile("eieio; tlbsync; ptesync" : : : "memory"); | 336 | asm volatile("eieio; tlbsync; ptesync" : : : "memory"); |
| 335 | trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1); | 337 | trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1); |
| 338 | |||
| 339 | /* | ||
| 340 | * The init_mm context is given the first available (non-zero) PID, | ||
| 341 | * which is the "guard PID" and contains no page table. PIDR should | ||
| 342 | * never be set to zero because that duplicates the kernel address | ||
| 343 | * space at the 0x0... offset (quadrant 0)! | ||
| 344 | * | ||
| 345 | * An arbitrary PID that may later be allocated by the PID allocator | ||
| 346 | * for userspace processes must not be used either, because that | ||
| 347 | * would cause stale user mappings for that PID on CPUs outside of | ||
| 348 | * the TLB invalidation scheme (because it won't be in mm_cpumask). | ||
| 349 | * | ||
| 350 | * So permanently carve out one PID for the purpose of a guard PID. | ||
| 351 | */ | ||
| 352 | init_mm.context.id = mmu_base_pid; | ||
| 353 | mmu_base_pid++; | ||
| 336 | } | 354 | } |
| 337 | 355 | ||
| 338 | static void __init radix_init_partition_table(void) | 356 | static void __init radix_init_partition_table(void) |
| @@ -535,6 +553,7 @@ void __init radix__early_init_mmu(void) · 196 unmodified lines | |||
| 535 | __pmd_index_size = RADIX_PMD_INDEX_SIZE; | 553 | __pmd_index_size = RADIX_PMD_INDEX_SIZE; |
| 536 | __pud_index_size = RADIX_PUD_INDEX_SIZE; | 554 | __pud_index_size = RADIX_PUD_INDEX_SIZE; |
| 537 | __pgd_index_size = RADIX_PGD_INDEX_SIZE; | 555 | __pgd_index_size = RADIX_PGD_INDEX_SIZE; |
| 556 | __pud_cache_index = RADIX_PUD_INDEX_SIZE; | ||
| 538 | __pmd_cache_index = RADIX_PMD_INDEX_SIZE; | 557 | __pmd_cache_index = RADIX_PMD_INDEX_SIZE; |
| 539 | __pte_table_size = RADIX_PTE_TABLE_SIZE; | 558 | __pte_table_size = RADIX_PTE_TABLE_SIZE; |
| 540 | __pmd_table_size = RADIX_PMD_TABLE_SIZE; | 559 | __pmd_table_size = RADIX_PMD_TABLE_SIZE; |
| @@ -579,7 +598,8 @@ void __init radix__early_init_mmu(void) · 38 unmodified lines | |||
| 579 | 598 | ||
| 580 | radix_init_iamr(); | 599 | radix_init_iamr(); |
| 581 | radix_init_pgtable(); | 600 | radix_init_pgtable(); |
| 582 | 601 | /* Switch to the guard PID before turning on MMU */ | |
| 602 | radix__switch_mmu_context(NULL, &init_mm); | ||
| 583 | if (cpu_has_feature(CPU_FTR_HVMODE)) | 603 | if (cpu_has_feature(CPU_FTR_HVMODE)) |
| 584 | tlbiel_all(); | 604 | tlbiel_all(); |
| 585 | } | 605 | } |
| @@ -604,6 +624,7 @@ void radix__early_init_mmu_secondary(void) · 18 unmodified lines | |||
| 604 | } | 624 | } |
| 605 | radix_init_iamr(); | 625 | radix_init_iamr(); |
| 606 | 626 | ||
| 627 | radix__switch_mmu_context(NULL, &init_mm); | ||
| 607 | if (cpu_has_feature(CPU_FTR_HVMODE)) | 628 | if (cpu_has_feature(CPU_FTR_HVMODE)) |
| 608 | tlbiel_all(); | 629 | tlbiel_all(); |
| 609 | } | 630 | } |
| @@ -666,6 +687,30 @@ static void free_pmd_table(pmd_t *pmd_start, pud_t *pud) · 56 unmodified lines | |||
| 666 | pud_clear(pud); | 687 | pud_clear(pud); |
| 667 | } | 688 | } |
| 668 | 689 | ||
| 690 | struct change_mapping_params { | ||
| 691 | pte_t *pte; | ||
| 692 | unsigned long start; | ||
| 693 | unsigned long end; | ||
| 694 | unsigned long aligned_start; | ||
| 695 | unsigned long aligned_end; | ||
| 696 | }; | ||
| 697 | |||
| 698 | static int stop_machine_change_mapping(void *data) | ||
| 699 | { | ||
| 700 | struct change_mapping_params *params = | ||
| 701 | (struct change_mapping_params *)data; | ||
| 702 | |||
| 703 | if (!data) | ||
| 704 | return -1; | ||
| 705 | |||
| 706 | spin_unlock(&init_mm.page_table_lock); | ||
| 707 | pte_clear(&init_mm, params->aligned_start, params->pte); | ||
| 708 | create_physical_mapping(params->aligned_start, params->start); | ||
| 709 | create_physical_mapping(params->end, params->aligned_end); | ||
| 710 | spin_lock(&init_mm.page_table_lock); | ||
| 711 | return 0; | ||
| 712 | } | ||
| 713 | |||
| 669 | static void remove_pte_table(pte_t *pte_start, unsigned long addr, | 714 | static void remove_pte_table(pte_t *pte_start, unsigned long addr, |
| 670 | unsigned long end) | 715 | unsigned long end) |
| 671 | { | 716 | { |
| @@ -694,6 +739,52 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr, · 22 unmodified lines | |||
| 694 | } | 739 | } |
| 695 | } | 740 | } |
| 696 | 741 | ||
| 742 | /* | ||
| 743 | * clear the pte and potentially split the mapping helper | ||
| 744 | */ | ||
| 745 | static void split_kernel_mapping(unsigned long addr, unsigned long end, | ||
| 746 | unsigned long size, pte_t *pte) | ||
| 747 | { | ||
| 748 | unsigned long mask = ~(size - 1); | ||
| 749 | unsigned long aligned_start = addr & mask; | ||
| 750 | unsigned long aligned_end = addr + size; | ||
| 751 | struct change_mapping_params params; | ||
| 752 | bool split_region = false; | ||
| 753 | |||
| 754 | if ((end - addr) < size) { | ||
| 755 | /* | ||
| 756 | * We're going to clear the PTE, but not flushed | ||
| 757 | * the mapping, time to remap and flush. The | ||
| 758 | * effects if visible outside the processor or | ||
| 759 | * if we are running in code close to the | ||
| 760 | * mapping we cleared, we are in trouble. | ||
| 761 | */ | ||
| 762 | if (overlaps_kernel_text(aligned_start, addr) || | ||
| 763 | overlaps_kernel_text(end, aligned_end)) { | ||
| 764 | /* | ||
| 765 | * Hack, just return, don't pte_clear | ||
| 766 | */ | ||
| 767 | WARN_ONCE(1, "Linear mapping %lx->%lx overlaps kernel " | ||
| 768 | "text, not splitting\n", addr, end); | ||
| 769 | return; | ||
| 770 | } | ||
| 771 | split_region = true; | ||
| 772 | } | ||
| 773 | |||
| 774 | if (split_region) { | ||
| 775 | params.pte = pte; | ||
| 776 | params.start = addr; | ||
| 777 | params.end = end; | ||
| 778 | params.aligned_start = addr & ~(size - 1); | ||
| 779 | params.aligned_end = min_t(unsigned long, aligned_end, | ||
| 780 | (unsigned long)__va(memblock_end_of_DRAM())); | ||
| 781 | stop_machine(stop_machine_change_mapping, ¶ms, NULL); | ||
| 782 | return; | ||
| 783 | } | ||
| 784 | |||
| 785 | pte_clear(&init_mm, addr, pte); | ||
| 786 | } | ||
| 787 | |||
| 697 | static void remove_pmd_table(pmd_t *pmd_start, unsigned long addr, | 788 | static void remove_pmd_table(pmd_t *pmd_start, unsigned long addr, |
| 698 | unsigned long end) | 789 | unsigned long end) |
| 699 | { | 790 | { |
| @@ -709,13 +800,7 @@ static void remove_pmd_table(pmd_t *pmd_start, unsigned long addr, · 9 unmodified lines | |||
| 709 | continue; | 800 | continue; |
| 710 | 801 | ||
| 711 | if (pmd_huge(*pmd)) { | 802 | if (pmd_huge(*pmd)) { |
| 712 | if (!IS_ALIGNED(addr, PMD_SIZE) || | 803 | split_kernel_mapping(addr, end, PMD_SIZE, (pte_t *)pmd); |
| 713 | !IS_ALIGNED(next, PMD_SIZE)) { | ||
| 714 | WARN_ONCE(1, "%s: unaligned range\n", __func__); | ||
| 715 | continue; | ||
| 716 | } | ||
| 717 | |||
| 718 | pte_clear(&init_mm, addr, (pte_t *)pmd); | ||
| 719 | continue; | 804 | continue; |
| 720 | } | 805 | } |
| 721 | 806 | ||
| @@ -740,13 +825,7 @@ static void remove_pud_table(pud_t *pud_start, unsigned long addr, · 18 unmodified lines | |||
| 740 | continue; | 825 | continue; |
| 741 | 826 | ||
| 742 | if (pud_huge(*pud)) { | 827 | if (pud_huge(*pud)) { |
| 743 | if (!IS_ALIGNED(addr, PUD_SIZE) || | 828 | split_kernel_mapping(addr, end, PUD_SIZE, (pte_t *)pud); |
| 744 | !IS_ALIGNED(next, PUD_SIZE)) { | ||
| 745 | WARN_ONCE(1, "%s: unaligned range\n", __func__); | ||
| 746 | continue; | ||
| 747 | } | ||
| 748 | |||
| 749 | pte_clear(&init_mm, addr, (pte_t *)pud); | ||
| 750 | continue; | 829 | continue; |
| 751 | } | 830 | } |
| 752 | 831 | ||
| @@ -772,13 +851,7 @@ static void remove_pagetable(unsigned long start, unsigned long end) · 19 unmodified lines | |||
| 772 | continue; | 851 | continue; |
| 773 | 852 | ||
| 774 | if (pgd_huge(*pgd)) { | 853 | if (pgd_huge(*pgd)) { |
| 775 | if (!IS_ALIGNED(addr, PGDIR_SIZE) || | 854 | split_kernel_mapping(addr, end, PGDIR_SIZE, (pte_t *)pgd); |
| 776 | !IS_ALIGNED(next, PGDIR_SIZE)) { | ||
| 777 | WARN_ONCE(1, "%s: unaligned range\n", __func__); | ||
| 778 | continue; | ||
| 779 | } | ||
| 780 | |||
| 781 | pte_clear(&init_mm, addr, (pte_t *)pgd); | ||
| 782 | continue; | 855 | continue; |
| 783 | } | 856 | } |
| 784 | 857 | ||
| @@ -82,6 +82,8 @@ unsigned long __pgd_index_size; · 81 unmodified lines | |||
| 82 | EXPORT_SYMBOL(__pgd_index_size); | 82 | EXPORT_SYMBOL(__pgd_index_size); |
| 83 | unsigned long __pmd_cache_index; | 83 | unsigned long __pmd_cache_index; |
| 84 | EXPORT_SYMBOL(__pmd_cache_index); | 84 | EXPORT_SYMBOL(__pmd_cache_index); |
| 85 | unsigned long __pud_cache_index; | ||
| 86 | EXPORT_SYMBOL(__pud_cache_index); | ||
| 85 | unsigned long __pte_table_size; | 87 | unsigned long __pte_table_size; |
| 86 | EXPORT_SYMBOL(__pte_table_size); | 88 | EXPORT_SYMBOL(__pte_table_size); |
| 87 | unsigned long __pmd_table_size; | 89 | unsigned long __pmd_table_size; |
| @@ -471,6 +473,8 @@ void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0, · 383 unmodified lines | |||
| 471 | if (old & PATB_HR) { | 473 | if (old & PATB_HR) { |
| 472 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : : | 474 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : : |
| 473 | "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); | 475 | "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); |
| 476 | asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : : | ||
| 477 | "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); | ||
| 474 | trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1); | 478 | trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1); |
| 475 | } else { | 479 | } else { |
| 476 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : : | 480 | asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : : |
| @@ -51,7 +51,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 50 unmodified lines | |||
| 51 | unsigned int psize; | 51 | unsigned int psize; |
| 52 | int ssize; | 52 | int ssize; |
| 53 | real_pte_t rpte; | 53 | real_pte_t rpte; |
| 54 | int i; | 54 | int i, offset; |
| 55 | 55 | ||
| 56 | i = batch->index; | 56 | i = batch->index; |
| 57 | 57 | ||
| @@ -67,6 +67,10 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 9 unmodified lines | |||
| 67 | psize = get_slice_psize(mm, addr); | 67 | psize = get_slice_psize(mm, addr); |
| 68 | /* Mask the address for the correct page size */ | 68 | /* Mask the address for the correct page size */ |
| 69 | addr &= ~((1UL << mmu_psize_defs[psize].shift) - 1); | 69 | addr &= ~((1UL << mmu_psize_defs[psize].shift) - 1); |
| 70 | if (unlikely(psize == MMU_PAGE_16G)) | ||
| 71 | offset = PTRS_PER_PUD; | ||
| 72 | else | ||
| 73 | offset = PTRS_PER_PMD; | ||
| 70 | #else | 74 | #else |
| 71 | BUG(); | 75 | BUG(); |
| 72 | psize = pte_pagesize_index(mm, addr, pte); /* shutup gcc */ | 76 | psize = pte_pagesize_index(mm, addr, pte); /* shutup gcc */ |
| @@ -78,6 +82,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 5 unmodified lines | |||
| 78 | * support 64k pages, this might be different from the | 82 | * support 64k pages, this might be different from the |
| 79 | * hardware page size encoded in the slice table. */ | 83 | * hardware page size encoded in the slice table. */ |
| 80 | addr &= PAGE_MASK; | 84 | addr &= PAGE_MASK; |
| 85 | offset = PTRS_PER_PTE; | ||
| 81 | } | 86 | } |
| 82 | 87 | ||
| 83 | 88 | ||
| @@ -91,7 +96,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 7 unmodified lines | |||
| 91 | } | 96 | } |
| 92 | WARN_ON(vsid == 0); | 97 | WARN_ON(vsid == 0); |
| 93 | vpn = hpt_vpn(addr, vsid, ssize); | 98 | vpn = hpt_vpn(addr, vsid, ssize); |
| 94 | rpte = __real_pte(__pte(pte), ptep); | 99 | rpte = __real_pte(__pte(pte), ptep, offset); |
| 95 | 100 | ||
| 96 | /* | 101 | /* |
| 97 | * Check if we have an active batch on this CPU. If not, just | 102 | * Check if we have an active batch on this CPU. If not, just |
| @@ -327,6 +327,9 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, · 326 unmodified lines | |||
| 327 | BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4); | 327 | BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4); |
| 328 | PPC_LWZ_OFFS(r_A, r_skb, offsetof(struct sk_buff, len)); | 328 | PPC_LWZ_OFFS(r_A, r_skb, offsetof(struct sk_buff, len)); |
| 329 | break; | 329 | break; |
| 330 | case BPF_LDX | BPF_W | BPF_ABS: /* A = *((u32 *)(seccomp_data + K)); */ | ||
| 331 | PPC_LWZ_OFFS(r_A, r_skb, K); | ||
| 332 | break; | ||
| 330 | case BPF_LDX | BPF_W | BPF_LEN: /* X = skb->len; */ | 333 | case BPF_LDX | BPF_W | BPF_LEN: /* X = skb->len; */ |
| 331 | PPC_LWZ_OFFS(r_X, r_skb, offsetof(struct sk_buff, len)); | 334 | PPC_LWZ_OFFS(r_X, r_skb, offsetof(struct sk_buff, len)); |
| 332 | break; | 335 | break; |
| @@ -240,6 +240,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 · 239 unmodified lines | |||
| 240 | * goto out; | 240 | * goto out; |
| 241 | */ | 241 | */ |
| 242 | PPC_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)); | 242 | PPC_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)); |
| 243 | PPC_RLWINM(b2p_index, b2p_index, 0, 0, 31); | ||
| 243 | PPC_CMPLW(b2p_index, b2p[TMP_REG_1]); | 244 | PPC_CMPLW(b2p_index, b2p[TMP_REG_1]); |
| 244 | PPC_BCC(COND_GE, out); | 245 | PPC_BCC(COND_GE, out); |
| 245 | 246 | ||
| @@ -199,9 +199,11 @@ static void disable_nest_pmu_counters(void) · 198 unmodified lines | |||
| 199 | const struct cpumask *l_cpumask; | 199 | const struct cpumask *l_cpumask; |
| 200 | 200 | ||
| 201 | get_online_cpus(); | 201 | get_online_cpus(); |
| 202 | for_each_online_node(nid) { | 202 | for_each_node_with_cpus(nid) { |
| 203 | l_cpumask = cpumask_of_node(nid); | 203 | l_cpumask = cpumask_of_node(nid); |
| 204 | cpu = cpumask_first(l_cpumask); | 204 | cpu = cpumask_first_and(l_cpumask, cpu_online_mask); |
| 205 | if (cpu >= nr_cpu_ids) | ||
| 206 | continue; | ||
| 205 | opal_imc_counters_stop(OPAL_IMC_COUNTERS_NEST, | 207 | opal_imc_counters_stop(OPAL_IMC_COUNTERS_NEST, |
| 206 | get_hard_smp_processor_id(cpu)); | 208 | get_hard_smp_processor_id(cpu)); |
| 207 | } | 209 | } |
| @@ -1854,7 +1854,7 @@ static int pnv_pci_ioda_dma_set_mask(struct pci_dev *pdev, u64 dma_mask) · 1853 unmodified lines | |||
| 1854 | s64 rc; | 1854 | s64 rc; |
| 1855 | 1855 | ||
| 1856 | if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE)) | 1856 | if (WARN_ON(!pdn || pdn->pe_number == IODA_INVALID_PE)) |
| 1857 | return -ENODEV;; | 1857 | return -ENODEV; |
| 1858 | 1858 | ||
| 1859 | pe = &phb->ioda.pe_array[pdn->pe_number]; | 1859 | pe = &phb->ioda.pe_array[pdn->pe_number]; |
| 1860 | if (pe->tce_bypass_enabled) { | 1860 | if (pe->tce_bypass_enabled) { |
| @@ -80,6 +80,10 @@ static void pnv_setup_rfi_flush(void) · 79 unmodified lines | |||
| 80 | if (np && of_property_read_bool(np, "disabled")) | 80 | if (np && of_property_read_bool(np, "disabled")) |
| 81 | enable--; | 81 | enable--; |
| 82 | 82 | ||
| 83 | np = of_get_child_by_name(fw_features, "speculation-policy-favor-security"); | ||
| 84 | if (np && of_property_read_bool(np, "disabled")) | ||
| 85 | enable = 0; | ||
| 86 | |||
| 83 | of_node_put(np); | 87 | of_node_put(np); |
| 84 | of_node_put(fw_features); | 88 | of_node_put(fw_features); |
| 85 | } | 89 | } |
| @@ -1063,16 +1063,16 @@ struct vas_window *vas_tx_win_open(int vasid, enum vas_cop_type cop, · 1062 unmodified lines | |||
| 1063 | rc = PTR_ERR(txwin->paste_kaddr); | 1063 | rc = PTR_ERR(txwin->paste_kaddr); |
| 1064 | goto free_window; | 1064 | goto free_window; |
| 1065 | } | 1065 | } |
| 1066 | } else { | ||
| 1067 | /* | ||
| 1068 | * A user mapping must ensure that context switch issues | ||
| 1069 | * CP_ABORT for this thread. | ||
| 1070 | */ | ||
| 1071 | rc = set_thread_uses_vas(); | ||
| 1072 | if (rc) | ||
| 1073 | goto free_window; | ||
| 1066 | } | 1074 | } |
| 1067 | 1075 | ||
| 1068 | /* | ||
| 1069 | * Now that we have a send window, ensure context switch issues | ||
| 1070 | * CP_ABORT for this thread. | ||
| 1071 | */ | ||
| 1072 | rc = -EINVAL; | ||
| 1073 | if (set_thread_uses_vas() < 0) | ||
| 1074 | goto free_window; | ||
| 1075 | |||
| 1076 | set_vinst_win(vinst, txwin); | 1076 | set_vinst_win(vinst, txwin); |
| 1077 | 1077 | ||
| 1078 | return txwin; | 1078 | return txwin; |
| @@ -36,6 +36,7 @@ · 35 unmodified lines | |||
| 36 | #include <asm/xics.h> | 36 | #include <asm/xics.h> |
| 37 | #include <asm/xive.h> | 37 | #include <asm/xive.h> |
| 38 | #include <asm/plpar_wrappers.h> | 38 | #include <asm/plpar_wrappers.h> |
| 39 | #include <asm/topology.h> | ||
| 39 | 40 | ||
| 40 | #include "pseries.h" | 41 | #include "pseries.h" |
| 41 | #include "offline_states.h" | 42 | #include "offline_states.h" |
| @@ -331,6 +332,7 @@ static void pseries_remove_processor(struct device_node *np) · 289 unmodified lines | |||
| 331 | BUG_ON(cpu_online(cpu)); | 332 | BUG_ON(cpu_online(cpu)); |
| 332 | set_cpu_present(cpu, false); | 333 | set_cpu_present(cpu, false); |
| 333 | set_hard_smp_processor_id(cpu, -1); | 334 | set_hard_smp_processor_id(cpu, -1); |
| 335 | update_numa_cpu_lookup_table(cpu, -1); | ||
| 334 | break; | 336 | break; |
| 335 | } | 337 | } |
| 336 | if (cpu >= nr_cpu_ids) | 338 | if (cpu >= nr_cpu_ids) |
| @@ -340,8 +342,6 @@ static void pseries_remove_processor(struct device_node *np) · 3 unmodified lines | |||
| 340 | cpu_maps_update_done(); | 342 | cpu_maps_update_done(); |
| 341 | } | 343 | } |
| 342 | 344 | ||
| 343 | extern int find_and_online_cpu_nid(int cpu); | ||
| 344 | |||
| 345 | static int dlpar_online_cpu(struct device_node *dn) | 345 | static int dlpar_online_cpu(struct device_node *dn) |
| 346 | { | 346 | { |
| 347 | int rc = 0; | 347 | int rc = 0; |
| @@ -48,6 +48,28 @@ static irqreturn_t ras_epow_interrupt(int irq, void *dev_id); · 47 unmodified lines | |||
| 48 | static irqreturn_t ras_error_interrupt(int irq, void *dev_id); | 48 | static irqreturn_t ras_error_interrupt(int irq, void *dev_id); |
| 49 | 49 | ||
| 50 | 50 | ||
| 51 | /* | ||
| 52 | * Enable the hotplug interrupt late because processing them may touch other | ||
| 53 | * devices or systems (e.g. hugepages) that have not been initialized at the | ||
| 54 | * subsys stage. | ||
| 55 | */ | ||
| 56 | int __init init_ras_hotplug_IRQ(void) | ||
| 57 | { | ||
| 58 | struct device_node *np; | ||
| 59 | |||
| 60 | /* Hotplug Events */ | ||
| 61 | np = of_find_node_by_path("/event-sources/hot-plug-events"); | ||
| 62 | if (np != NULL) { | ||
| 63 | if (dlpar_workqueue_init() == 0) | ||
| 64 | request_event_sources_irqs(np, ras_hotplug_interrupt, | ||
| 65 | "RAS_HOTPLUG"); | ||
| 66 | of_node_put(np); | ||
| 67 | } | ||
| 68 | |||
| 69 | return 0; | ||
| 70 | } | ||
| 71 | machine_late_initcall(pseries, init_ras_hotplug_IRQ); | ||
| 72 | |||
| 51 | /* | 73 | /* |
| 52 | * Initialize handlers for the set of interrupts caused by hardware errors | 74 | * Initialize handlers for the set of interrupts caused by hardware errors |
| 53 | * and power system events. | 75 | * and power system events. |
| @@ -66,15 +88,6 @@ static int __init init_ras_IRQ(void) · 12 unmodified lines | |||
| 66 | of_node_put(np); | 88 | of_node_put(np); |
| 67 | } | 89 | } |
| 68 | 90 | ||
| 69 | /* Hotplug Events */ | ||
| 70 | np = of_find_node_by_path("/event-sources/hot-plug-events"); | ||
| 71 | if (np != NULL) { | ||
| 72 | if (dlpar_workqueue_init() == 0) | ||
| 73 | request_event_sources_irqs(np, ras_hotplug_interrupt, | ||
| 74 | "RAS_HOTPLUG"); | ||
| 75 | of_node_put(np); | ||
| 76 | } | ||
| 77 | |||
| 78 | /* EPOW Events */ | 91 | /* EPOW Events */ |
| 79 | np = of_find_node_by_path("/event-sources/epow-events"); | 92 | np = of_find_node_by_path("/event-sources/epow-events"); |
| 80 | if (np != NULL) { | 93 | if (np != NULL) { |
| @@ -482,7 +482,8 @@ static void pseries_setup_rfi_flush(void) · 481 unmodified lines | |||
| 482 | if (types == L1D_FLUSH_NONE) | 482 | if (types == L1D_FLUSH_NONE) |
| 483 | types = L1D_FLUSH_FALLBACK; | 483 | types = L1D_FLUSH_FALLBACK; |
| 484 | 484 | ||
| 485 | if (!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) | 485 | if ((!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) || |
| 486 | (!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY))) | ||
| 486 | enable = false; | 487 | enable = false; |
| 487 | } else { | 488 | } else { |
| 488 | /* Default to fallback if case hcall is not available */ | 489 | /* Default to fallback if case hcall is not available */ |
| @@ -356,7 +356,8 @@ static int xive_spapr_configure_queue(u32 target, struct xive_q *q, u8 prio, · 355 unmodified lines | |||
| 356 | 356 | ||
| 357 | rc = plpar_int_get_queue_info(0, target, prio, &esn_page, &esn_size); | 357 | rc = plpar_int_get_queue_info(0, target, prio, &esn_page, &esn_size); |
| 358 | if (rc) { | 358 | if (rc) { |
| 359 | pr_err("Error %lld getting queue info prio %d\n", rc, prio); | 359 | pr_err("Error %lld getting queue info CPU %d prio %d\n", rc, |
| 360 | target, prio); | ||
| 360 | rc = -EIO; | 361 | rc = -EIO; |
| 361 | goto fail; | 362 | goto fail; |
| 362 | } | 363 | } |
| @@ -370,7 +371,8 @@ static int xive_spapr_configure_queue(u32 target, struct xive_q *q, u8 prio, · 7 unmodified lines | |||
| 370 | /* Configure and enable the queue in HW */ | 371 | /* Configure and enable the queue in HW */ |
| 371 | rc = plpar_int_set_queue_config(flags, target, prio, qpage_phys, order); | 372 | rc = plpar_int_set_queue_config(flags, target, prio, qpage_phys, order); |
| 372 | if (rc) { | 373 | if (rc) { |
| 373 | pr_err("Error %lld setting queue for prio %d\n", rc, prio); | 374 | pr_err("Error %lld setting queue for CPU %d prio %d\n", rc, |
| 375 | target, prio); | ||
| 374 | rc = -EIO; | 376 | rc = -EIO; |
| 375 | } else { | 377 | } else { |
| 376 | q->qpage = qpage; | 378 | q->qpage = qpage; |
| @@ -389,8 +391,8 @@ static int xive_spapr_setup_queue(unsigned int cpu, struct xive_cpu *xc, · 12 unmodified lines | |||
| 389 | if (IS_ERR(qpage)) | 391 | if (IS_ERR(qpage)) |
| 390 | return PTR_ERR(qpage); | 392 | return PTR_ERR(qpage); |
| 391 | 393 | ||
| 392 | return xive_spapr_configure_queue(cpu, q, prio, qpage, | 394 | return xive_spapr_configure_queue(get_hard_smp_processor_id(cpu), |
| 393 | xive_queue_shift); | 395 | q, prio, qpage, xive_queue_shift); |
| 394 | } | 396 | } |
| 395 | 397 | ||
| 396 | static void xive_spapr_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, | 398 | static void xive_spapr_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, |
| @@ -399,10 +401,12 @@ static void xive_spapr_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, · 2 unmodified lines | |||
| 399 | struct xive_q *q = &xc->queue[prio]; | 401 | struct xive_q *q = &xc->queue[prio]; |
| 400 | unsigned int alloc_order; | 402 | unsigned int alloc_order; |
| 401 | long rc; | 403 | long rc; |
| 404 | int hw_cpu = get_hard_smp_processor_id(cpu); | ||
| 402 | 405 | ||
| 403 | rc = plpar_int_set_queue_config(0, cpu, prio, 0, 0); | 406 | rc = plpar_int_set_queue_config(0, hw_cpu, prio, 0, 0); |
| 404 | if (rc) | 407 | if (rc) |
| 405 | pr_err("Error %ld setting queue for prio %d\n", rc, prio); | 408 | pr_err("Error %ld setting queue for CPU %d prio %d\n", rc, |
| 409 | hw_cpu, prio); | ||
| 406 | 410 | ||
| 407 | alloc_order = xive_alloc_order(xive_queue_shift); | 411 | alloc_order = xive_alloc_order(xive_queue_shift); |
| 408 | free_pages((unsigned long)q->qpage, alloc_order); | 412 | free_pages((unsigned long)q->qpage, alloc_order); |
| @@ -8,7 +8,6 @@ config RISCV · 7 unmodified lines | |||
| 8 | select OF | 8 | select OF |
| 9 | select OF_EARLY_FLATTREE | 9 | select OF_EARLY_FLATTREE |
| 10 | select OF_IRQ | 10 | select OF_IRQ |
| 11 | select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE | ||
| 12 | select ARCH_WANT_FRAME_POINTERS | 11 | select ARCH_WANT_FRAME_POINTERS |
| 13 | select CLONE_BACKWARDS | 12 | select CLONE_BACKWARDS |
| 14 | select COMMON_CLK | 13 | select COMMON_CLK |
| @@ -20,7 +19,6 @@ config RISCV · 5 unmodified lines | |||
| 20 | select GENERIC_STRNLEN_USER | 19 | select GENERIC_STRNLEN_USER |
| 21 | select GENERIC_SMP_IDLE_THREAD | 20 | select GENERIC_SMP_IDLE_THREAD |
| 22 | select GENERIC_ATOMIC64 if !64BIT || !RISCV_ISA_A | 21 | select GENERIC_ATOMIC64 if !64BIT || !RISCV_ISA_A |
| 23 | select ARCH_WANT_OPTIONAL_GPIOLIB | ||
| 24 | select HAVE_MEMBLOCK | 22 | select HAVE_MEMBLOCK |
| 25 | select HAVE_MEMBLOCK_NODE_MAP | 23 | select HAVE_MEMBLOCK_NODE_MAP |
| 26 | select HAVE_DMA_API_DEBUG | 24 | select HAVE_DMA_API_DEBUG |
| @@ -34,7 +32,6 @@ config RISCV · 7 unmodified lines | |||
| 34 | select HAVE_ARCH_TRACEHOOK | 32 | select HAVE_ARCH_TRACEHOOK |
| 35 | select MODULES_USE_ELF_RELA if MODULES | 33 | select MODULES_USE_ELF_RELA if MODULES |
| 36 | select THREAD_INFO_IN_TASK | 34 | select THREAD_INFO_IN_TASK |
| 37 | select RISCV_IRQ_INTC | ||
| 38 | select RISCV_TIMER | 35 | select RISCV_TIMER |
| 39 | 36 | ||
| 40 | config MMU | 37 | config MMU |
| @@ -34,9 +34,9 @@ · 33 unmodified lines | |||
| 34 | #define wmb() RISCV_FENCE(ow,ow) | 34 | #define wmb() RISCV_FENCE(ow,ow) |
| 35 | 35 | ||
| 36 | /* These barriers do not need to enforce ordering on devices, just memory. */ | 36 | /* These barriers do not need to enforce ordering on devices, just memory. */ |
| 37 | #define smp_mb() RISCV_FENCE(rw,rw) | 37 | #define __smp_mb() RISCV_FENCE(rw,rw) |
| 38 | #define smp_rmb() RISCV_FENCE(r,r) | 38 | #define __smp_rmb() RISCV_FENCE(r,r) |
| 39 | #define smp_wmb() RISCV_FENCE(w,w) | 39 | #define __smp_wmb() RISCV_FENCE(w,w) |
| 40 | 40 | ||
| 41 | /* | 41 | /* |
| 42 | * This is a very specific barrier: it's currently only used in two places in | 42 | * This is a very specific barrier: it's currently only used in two places in |
| @@ -172,6 +172,9 @@ ENTRY(handle_exception) · 171 unmodified lines | |||
| 172 | move a1, sp /* pt_regs */ | 172 | move a1, sp /* pt_regs */ |
| 173 | tail do_IRQ | 173 | tail do_IRQ |
| 174 | 1: | 174 | 1: |
| 175 | /* Exceptions run with interrupts enabled */ | ||
| 176 | csrs sstatus, SR_SIE | ||
| 177 | |||
| 175 | /* Handle syscalls */ | 178 | /* Handle syscalls */ |
| 176 | li t0, EXC_SYSCALL | 179 | li t0, EXC_SYSCALL |
| 177 | beq s4, t0, handle_syscall | 180 | beq s4, t0, handle_syscall |
| @@ -198,8 +201,6 @@ handle_syscall: · 20 unmodified lines | |||
| 198 | */ | 201 | */ |
| 199 | addi s2, s2, 0x4 | 202 | addi s2, s2, 0x4 |
| 200 | REG_S s2, PT_SEPC(sp) | 203 | REG_S s2, PT_SEPC(sp) |
| 201 | /* System calls run with interrupts enabled */ | ||
| 202 | csrs sstatus, SR_SIE | ||
| 203 | /* Trace syscalls, but only if requested by the user. */ | 204 | /* Trace syscalls, but only if requested by the user. */ |
| 204 | REG_L t0, TASK_TI_FLAGS(tp) | 205 | REG_L t0, TASK_TI_FLAGS(tp) |
| 205 | andi t0, t0, _TIF_SYSCALL_TRACE | 206 | andi t0, t0, _TIF_SYSCALL_TRACE |
| @@ -64,7 +64,7 @@ ENTRY(_start) · 63 unmodified lines | |||
| 64 | /* Start the kernel */ | 64 | /* Start the kernel */ |
| 65 | mv a0, s0 | 65 | mv a0, s0 |
| 66 | mv a1, s1 | 66 | mv a1, s1 |
| 67 | call sbi_save | 67 | call parse_dtb |
| 68 | tail start_kernel | 68 | tail start_kernel |
| 69 | 69 | ||
| 70 | relocate: | 70 | relocate: |
| @@ -144,7 +144,7 @@ asmlinkage void __init setup_vm(void) · 143 unmodified lines | |||
| 144 | #endif | 144 | #endif |
| 145 | } | 145 | } |
| 146 | 146 | ||
| 147 | void __init sbi_save(unsigned int hartid, void *dtb) | 147 | void __init parse_dtb(unsigned int hartid, void *dtb) |
| 148 | { | 148 | { |
| 149 | early_init_dt_scan(__va(dtb)); | 149 | early_init_dt_scan(__va(dtb)); |
| 150 | } | 150 | } |
| @@ -63,6 +63,7 @@ static inline int init_new_context(struct task_struct *tsk, · 62 unmodified lines | |||
| 63 | _ASCE_USER_BITS | _ASCE_TYPE_SEGMENT; | 63 | _ASCE_USER_BITS | _ASCE_TYPE_SEGMENT; |
| 64 | /* pgd_alloc() did not account this pmd */ | 64 | /* pgd_alloc() did not account this pmd */ |
| 65 | mm_inc_nr_pmds(mm); | 65 | mm_inc_nr_pmds(mm); |
| 66 | mm_inc_nr_puds(mm); | ||
| 66 | } | 67 | } |
| 67 | crst_table_init((unsigned long *) mm->pgd, pgd_entry_type(mm)); | 68 | crst_table_init((unsigned long *) mm->pgd, pgd_entry_type(mm)); |
| 68 | return 0; | 69 | return 0; |
| @@ -14,6 +14,7 @@ · 13 unmodified lines | |||
| 14 | #include <asm/processor.h> | 14 | #include <asm/processor.h> |
| 15 | #include <asm/cache.h> | 15 | #include <asm/cache.h> |
| 16 | #include <asm/ctl_reg.h> | 16 | #include <asm/ctl_reg.h> |
| 17 | #include <asm/dwarf.h> | ||
| 17 | #include <asm/errno.h> | 18 | #include <asm/errno.h> |
| 18 | #include <asm/ptrace.h> | 19 | #include <asm/ptrace.h> |
| 19 | #include <asm/thread_info.h> | 20 | #include <asm/thread_info.h> |
| @@ -230,7 +231,7 @@ _PIF_WORK = (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART) · 210 unmodified lines | |||
| 230 | .hidden \name | 231 | .hidden \name |
| 231 | .type \name,@function | 232 | .type \name,@function |
| 232 | \name: | 233 | \name: |
| 233 | .cfi_startproc | 234 | CFI_STARTPROC |
| 234 | #ifdef CONFIG_HAVE_MARCH_Z10_FEATURES | 235 | #ifdef CONFIG_HAVE_MARCH_Z10_FEATURES |
| 235 | exrl 0,0f | 236 | exrl 0,0f |
| 236 | #else | 237 | #else |
| @@ -239,7 +240,7 @@ _PIF_WORK = (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART) · 2 unmodified lines | |||
| 239 | #endif | 240 | #endif |
| 240 | j . | 241 | j . |
| 241 | 0: br \reg | 242 | 0: br \reg |
| 242 | .cfi_endproc | 243 | CFI_ENDPROC |
| 243 | .endm | 244 | .endm |
| 244 | 245 | ||
| 245 | GEN_BR_THUNK __s390x_indirect_jump_r1use_r9,%r9,%r1 | 246 | GEN_BR_THUNK __s390x_indirect_jump_r1use_r9,%r9,%r1 |
| @@ -426,13 +427,13 @@ ENTRY(system_call) · 180 unmodified lines | |||
| 426 | UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER | 427 | UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER |
| 427 | BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP | 428 | BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP |
| 428 | stmg %r0,%r7,__PT_R0(%r11) | 429 | stmg %r0,%r7,__PT_R0(%r11) |
| 429 | # clear user controlled register to prevent speculative use | ||
| 430 | xgr %r0,%r0 | ||
| 431 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC | 430 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC |
| 432 | mvc __PT_PSW(16,%r11),__LC_SVC_OLD_PSW | 431 | mvc __PT_PSW(16,%r11),__LC_SVC_OLD_PSW |
| 433 | mvc __PT_INT_CODE(4,%r11),__LC_SVC_ILC | 432 | mvc __PT_INT_CODE(4,%r11),__LC_SVC_ILC |
| 434 | stg %r14,__PT_FLAGS(%r11) | 433 | stg %r14,__PT_FLAGS(%r11) |
| 435 | .Lsysc_do_svc: | 434 | .Lsysc_do_svc: |
| 435 | # clear user controlled register to prevent speculative use | ||
| 436 | xgr %r0,%r0 | ||
| 436 | # load address of system call table | 437 | # load address of system call table |
| 437 | lg %r10,__THREAD_sysc_table(%r13,%r12) | 438 | lg %r10,__THREAD_sysc_table(%r13,%r12) |
| 438 | llgh %r8,__PT_INT_CODE+2(%r11) | 439 | llgh %r8,__PT_INT_CODE+2(%r11) |
| @@ -1439,6 +1440,7 @@ cleanup_critical: · 1000 unmodified lines | |||
| 1439 | stg %r15,__LC_SYSTEM_TIMER | 1440 | stg %r15,__LC_SYSTEM_TIMER |
| 1440 | 0: # update accounting time stamp | 1441 | 0: # update accounting time stamp |
| 1441 | mvc __LC_LAST_UPDATE_TIMER(8),__LC_SYNC_ENTER_TIMER | 1442 | mvc __LC_LAST_UPDATE_TIMER(8),__LC_SYNC_ENTER_TIMER |
| 1443 | BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP | ||
| 1442 | # set up saved register r11 | 1444 | # set up saved register r11 |
| 1443 | lg %r15,__LC_KERNEL_STACK | 1445 | lg %r15,__LC_KERNEL_STACK |
| 1444 | la %r9,STACK_FRAME_OVERHEAD(%r15) | 1446 | la %r9,STACK_FRAME_OVERHEAD(%r15) |
| @@ -2,8 +2,8 @@ · 1 unmodified line | |||
| 2 | #include <linux/module.h> | 2 | #include <linux/module.h> |
| 3 | #include <asm/nospec-branch.h> | 3 | #include <asm/nospec-branch.h> |
| 4 | 4 | ||
| 5 | int nospec_call_disable = IS_ENABLED(EXPOLINE_OFF); | 5 | int nospec_call_disable = IS_ENABLED(CONFIG_EXPOLINE_OFF); |
| 6 | int nospec_return_disable = !IS_ENABLED(EXPOLINE_FULL); | 6 | int nospec_return_disable = !IS_ENABLED(CONFIG_EXPOLINE_FULL); |
| 7 | 7 | ||
| 8 | static int __init nospectre_v2_setup_early(char *str) | 8 | static int __init nospectre_v2_setup_early(char *str) |
| 9 | { | 9 | { |
| @@ -22,22 +22,6 @@ · 21 unmodified lines | |||
| 22 | #include "trace.h" | 22 | #include "trace.h" |
| 23 | #include "trace-s390.h" | 23 | #include "trace-s390.h" |
| 24 | 24 | ||
| 25 | |||
| 26 | static const intercept_handler_t instruction_handlers[256] = { | ||
| 27 | [0x01] = kvm_s390_handle_01, | ||
| 28 | [0x82] = kvm_s390_handle_lpsw, | ||
| 29 | [0x83] = kvm_s390_handle_diag, | ||
| 30 | [0xaa] = kvm_s390_handle_aa, | ||
| 31 | [0xae] = kvm_s390_handle_sigp, | ||
| 32 | [0xb2] = kvm_s390_handle_b2, | ||
| 33 | [0xb6] = kvm_s390_handle_stctl, | ||
| 34 | [0xb7] = kvm_s390_handle_lctl, | ||
| 35 | [0xb9] = kvm_s390_handle_b9, | ||
| 36 | [0xe3] = kvm_s390_handle_e3, | ||
| 37 | [0xe5] = kvm_s390_handle_e5, | ||
| 38 | [0xeb] = kvm_s390_handle_eb, | ||
| 39 | }; | ||
| 40 | |||
| 41 | u8 kvm_s390_get_ilen(struct kvm_vcpu *vcpu) | 25 | u8 kvm_s390_get_ilen(struct kvm_vcpu *vcpu) |
| 42 | { | 26 | { |
| 43 | struct kvm_s390_sie_block *sie_block = vcpu->arch.sie_block; | 27 | struct kvm_s390_sie_block *sie_block = vcpu->arch.sie_block; |
| @@ -129,16 +113,39 @@ static int handle_validity(struct kvm_vcpu *vcpu) · 85 unmodified lines | |||
| 129 | 113 | ||
| 130 | static int handle_instruction(struct kvm_vcpu *vcpu) | 114 | static int handle_instruction(struct kvm_vcpu *vcpu) |
| 131 | { | 115 | { |
| 132 | intercept_handler_t handler; | ||
| 133 | |||
| 134 | vcpu->stat.exit_instruction++; | 116 | vcpu->stat.exit_instruction++; |
| 135 | trace_kvm_s390_intercept_instruction(vcpu, | 117 | trace_kvm_s390_intercept_instruction(vcpu, |
| 136 | vcpu->arch.sie_block->ipa, | 118 | vcpu->arch.sie_block->ipa, |
| 137 | vcpu->arch.sie_block->ipb); | 119 | vcpu->arch.sie_block->ipb); |
| 138 | handler = instruction_handlers[vcpu->arch.sie_block->ipa >> 8]; | 120 | |
| 139 | if (handler) | 121 | switch (vcpu->arch.sie_block->ipa >> 8) { |
| 140 | return handler(vcpu); | 122 | case 0x01: |
| 141 | return -EOPNOTSUPP; | 123 | return kvm_s390_handle_01(vcpu); |
| 124 | case 0x82: | ||
| 125 | return kvm_s390_handle_lpsw(vcpu); | ||
| 126 | case 0x83: | ||
| 127 | return kvm_s390_handle_diag(vcpu); | ||
| 128 | case 0xaa: | ||
| 129 | return kvm_s390_handle_aa(vcpu); | ||
| 130 | case 0xae: | ||
| 131 | return kvm_s390_handle_sigp(vcpu); | ||
| 132 | case 0xb2: | ||
| 133 | return kvm_s390_handle_b2(vcpu); | ||
| 134 | case 0xb6: | ||
| 135 | return kvm_s390_handle_stctl(vcpu); | ||
| 136 | case 0xb7: | ||
| 137 | return kvm_s390_handle_lctl(vcpu); | ||
| 138 | case 0xb9: | ||
| 139 | return kvm_s390_handle_b9(vcpu); | ||
| 140 | case 0xe3: | ||
| 141 | return kvm_s390_handle_e3(vcpu); | ||
| 142 | case 0xe5: | ||
| 143 | return kvm_s390_handle_e5(vcpu); | ||
| 144 | case 0xeb: | ||
| 145 | return kvm_s390_handle_eb(vcpu); | ||
| 146 | default: | ||
| 147 | return -EOPNOTSUPP; | ||
| 148 | } | ||
| 142 | } | 149 | } |
| 143 | 150 | ||
| 144 | static int inject_prog_on_prog_intercept(struct kvm_vcpu *vcpu) | 151 | static int inject_prog_on_prog_intercept(struct kvm_vcpu *vcpu) |
| @@ -169,8 +169,15 @@ static int ckc_interrupts_enabled(struct kvm_vcpu *vcpu) · 168 unmodified lines | |||
| 169 | 169 | ||
| 170 | static int ckc_irq_pending(struct kvm_vcpu *vcpu) | 170 | static int ckc_irq_pending(struct kvm_vcpu *vcpu) |
| 171 | { | 171 | { |
| 172 | if (vcpu->arch.sie_block->ckc >= kvm_s390_get_tod_clock_fast(vcpu->kvm)) | 172 | const u64 now = kvm_s390_get_tod_clock_fast(vcpu->kvm); |
| 173 | const u64 ckc = vcpu->arch.sie_block->ckc; | ||
| 174 | |||
| 175 | if (vcpu->arch.sie_block->gcr[0] & 0x0020000000000000ul) { | ||
| 176 | if ((s64)ckc >= (s64)now) | ||
| 177 | return 0; | ||
| 178 | } else if (ckc >= now) { | ||
| 173 | return 0; | 179 | return 0; |
| 180 | } | ||
| 174 | return ckc_interrupts_enabled(vcpu); | 181 | return ckc_interrupts_enabled(vcpu); |
| 175 | } | 182 | } |
| 176 | 183 | ||
| @@ -187,12 +194,6 @@ static int cpu_timer_irq_pending(struct kvm_vcpu *vcpu) · 10 unmodified lines | |||
| 187 | return kvm_s390_get_cpu_timer(vcpu) >> 63; | 194 | return kvm_s390_get_cpu_timer(vcpu) >> 63; |
| 188 | } | 195 | } |
| 189 | 196 | ||
| 190 | static inline int is_ioirq(unsigned long irq_type) | ||
| 191 | { | ||
| 192 | return ((irq_type >= IRQ_PEND_IO_ISC_7) && | ||
| 193 | (irq_type <= IRQ_PEND_IO_ISC_0)); | ||
| 194 | } | ||
| 195 | |||
| 196 | static uint64_t isc_to_isc_bits(int isc) | 197 | static uint64_t isc_to_isc_bits(int isc) |
| 197 | { | 198 | { |
| 198 | return (0x80 >> isc) << 24; | 199 | return (0x80 >> isc) << 24; |
| @@ -236,10 +237,15 @@ static inline int kvm_s390_gisa_tac_ipm_gisc(struct kvm_s390_gisa *gisa, u32 gis · 37 unmodified lines | |||
| 236 | return test_and_clear_bit_inv(IPM_BIT_OFFSET + gisc, (unsigned long *) gisa); | 237 | return test_and_clear_bit_inv(IPM_BIT_OFFSET + gisc, (unsigned long *) gisa); |
| 237 | } | 238 | } |
| 238 | 239 | ||
| 239 | static inline unsigned long pending_irqs(struct kvm_vcpu *vcpu) | 240 | static inline unsigned long pending_irqs_no_gisa(struct kvm_vcpu *vcpu) |
| 240 | { | 241 | { |
| 241 | return vcpu->kvm->arch.float_int.pending_irqs | | 242 | return vcpu->kvm->arch.float_int.pending_irqs | |
| 242 | vcpu->arch.local_int.pending_irqs | | 243 | vcpu->arch.local_int.pending_irqs; |
| 244 | } | ||
| 245 | |||
| 246 | static inline unsigned long pending_irqs(struct kvm_vcpu *vcpu) | ||
| 247 | { | ||
| 248 | return pending_irqs_no_gisa(vcpu) | | ||
| 243 | kvm_s390_gisa_get_ipm(vcpu->kvm->arch.gisa) << IRQ_PEND_IO_ISC_7; | 249 | kvm_s390_gisa_get_ipm(vcpu->kvm->arch.gisa) << IRQ_PEND_IO_ISC_7; |
| 244 | } | 250 | } |
| 245 | 251 | ||
| @@ -337,7 +343,7 @@ static void __reset_intercept_indicators(struct kvm_vcpu *vcpu) · 91 unmodified lines | |||
| 337 | 343 | ||
| 338 | static void set_intercept_indicators_io(struct kvm_vcpu *vcpu) | 344 | static void set_intercept_indicators_io(struct kvm_vcpu *vcpu) |
| 339 | { | 345 | { |
| 340 | if (!(pending_irqs(vcpu) & IRQ_PEND_IO_MASK)) | 346 | if (!(pending_irqs_no_gisa(vcpu) & IRQ_PEND_IO_MASK)) |
| 341 | return; | 347 | return; |
| 342 | else if (psw_ioint_disabled(vcpu)) | 348 | else if (psw_ioint_disabled(vcpu)) |
| 343 | kvm_s390_set_cpuflags(vcpu, CPUSTAT_IO_INT); | 349 | kvm_s390_set_cpuflags(vcpu, CPUSTAT_IO_INT); |
| @@ -1011,24 +1017,6 @@ out: · 667 unmodified lines | |||
| 1011 | return rc; | 1017 | return rc; |
| 1012 | } | 1018 | } |
| 1013 | 1019 | ||
| 1014 | typedef int (*deliver_irq_t)(struct kvm_vcpu *vcpu); | ||
| 1015 | |||
| 1016 | static const deliver_irq_t deliver_irq_funcs[] = { | ||
| 1017 | [IRQ_PEND_MCHK_EX] = __deliver_machine_check, | ||
| 1018 | [IRQ_PEND_MCHK_REP] = __deliver_machine_check, | ||
| 1019 | [IRQ_PEND_PROG] = __deliver_prog, | ||
| 1020 | [IRQ_PEND_EXT_EMERGENCY] = __deliver_emergency_signal, | ||
| 1021 | [IRQ_PEND_EXT_EXTERNAL] = __deliver_external_call, | ||
| 1022 | [IRQ_PEND_EXT_CLOCK_COMP] = __deliver_ckc, | ||
| 1023 | [IRQ_PEND_EXT_CPU_TIMER] = __deliver_cpu_timer, | ||
| 1024 | [IRQ_PEND_RESTART] = __deliver_restart, | ||
| 1025 | [IRQ_PEND_SET_PREFIX] = __deliver_set_prefix, | ||
| 1026 | [IRQ_PEND_PFAULT_INIT] = __deliver_pfault_init, | ||
| 1027 | [IRQ_PEND_EXT_SERVICE] = __deliver_service, | ||
| 1028 | [IRQ_PEND_PFAULT_DONE] = __deliver_pfault_done, | ||
| 1029 | [IRQ_PEND_VIRTIO] = __deliver_virtio, | ||
| 1030 | }; | ||
| 1031 | |||
| 1032 | /* Check whether an external call is pending (deliverable or not) */ | 1020 | /* Check whether an external call is pending (deliverable or not) */ |
| 1033 | int kvm_s390_ext_call_pending(struct kvm_vcpu *vcpu) | 1021 | int kvm_s390_ext_call_pending(struct kvm_vcpu *vcpu) |
| 1034 | { | 1022 | { |
| @@ -1066,13 +1054,19 @@ int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu) · 31 unmodified lines | |||
| 1066 | 1054 | ||
| 1067 | static u64 __calculate_sltime(struct kvm_vcpu *vcpu) | 1055 | static u64 __calculate_sltime(struct kvm_vcpu *vcpu) |
| 1068 | { | 1056 | { |
| 1069 | u64 now, cputm, sltime = 0; | 1057 | const u64 now = kvm_s390_get_tod_clock_fast(vcpu->kvm); |
| 1058 | const u64 ckc = vcpu->arch.sie_block->ckc; | ||
| 1059 | u64 cputm, sltime = 0; | ||
| 1070 | 1060 | ||
| 1071 | if (ckc_interrupts_enabled(vcpu)) { | 1061 | if (ckc_interrupts_enabled(vcpu)) { |
| 1072 | now = kvm_s390_get_tod_clock_fast(vcpu->kvm); | 1062 | if (vcpu->arch.sie_block->gcr[0] & 0x0020000000000000ul) { |
| 1073 | sltime = tod_to_ns(vcpu->arch.sie_block->ckc - now); | 1063 | if ((s64)now < (s64)ckc) |
| 1074 | /* already expired or overflow? */ | 1064 | sltime = tod_to_ns((s64)ckc - (s64)now); |
| 1075 | if (!sltime || vcpu->arch.sie_block->ckc <= now) | 1065 | } else if (now < ckc) { |
| 1066 | sltime = tod_to_ns(ckc - now); | ||
| 1067 | } | ||
| 1068 | /* already expired */ | ||
| 1069 | if (!sltime) | ||
| 1076 | return 0; | 1070 | return 0; |
| 1077 | if (cpu_timer_interrupts_enabled(vcpu)) { | 1071 | if (cpu_timer_interrupts_enabled(vcpu)) { |
| 1078 | cputm = kvm_s390_get_cpu_timer(vcpu); | 1072 | cputm = kvm_s390_get_cpu_timer(vcpu); |
| @@ -1192,7 +1186,6 @@ void kvm_s390_clear_local_irqs(struct kvm_vcpu *vcpu) · 113 unmodified lines | |||
| 1192 | int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) | 1186 | int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) |
| 1193 | { | 1187 | { |
| 1194 | struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; | 1188 | struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; |
| 1195 | deliver_irq_t func; | ||
| 1196 | int rc = 0; | 1189 | int rc = 0; |
| 1197 | unsigned long irq_type; | 1190 | unsigned long irq_type; |
| 1198 | unsigned long irqs; | 1191 | unsigned long irqs; |
| @@ -1212,16 +1205,57 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) · 13 unmodified lines | |||
| 1212 | while ((irqs = deliverable_irqs(vcpu)) && !rc) { | 1205 | while ((irqs = deliverable_irqs(vcpu)) && !rc) { |
| 1213 | /* bits are in the reverse order of interrupt priority */ | 1206 | /* bits are in the reverse order of interrupt priority */ |
| 1214 | irq_type = find_last_bit(&irqs, IRQ_PEND_COUNT); | 1207 | irq_type = find_last_bit(&irqs, IRQ_PEND_COUNT); |
| 1215 | if (is_ioirq(irq_type)) { | 1208 | switch (irq_type) { |
| 1209 | case IRQ_PEND_IO_ISC_0: | ||
| 1210 | case IRQ_PEND_IO_ISC_1: | ||
| 1211 | case IRQ_PEND_IO_ISC_2: | ||
| 1212 | case IRQ_PEND_IO_ISC_3: | ||
| 1213 | case IRQ_PEND_IO_ISC_4: | ||
| 1214 | case IRQ_PEND_IO_ISC_5: | ||
| 1215 | case IRQ_PEND_IO_ISC_6: | ||
| 1216 | case IRQ_PEND_IO_ISC_7: | ||
| 1216 | rc = __deliver_io(vcpu, irq_type); | 1217 | rc = __deliver_io(vcpu, irq_type); |
| 1217 | } else { | 1218 | break; |
| 1218 | func = deliver_irq_funcs[irq_type]; | 1219 | case IRQ_PEND_MCHK_EX: |
| 1219 | if (!func) { | 1220 | case IRQ_PEND_MCHK_REP: |
| 1220 | WARN_ON_ONCE(func == NULL); | 1221 | rc = __deliver_machine_check(vcpu); |
| 1221 | clear_bit(irq_type, &li->pending_irqs); | 1222 | break; |
| 1222 | continue; | 1223 | case IRQ_PEND_PROG: |
| 1223 | } | 1224 | rc = __deliver_prog(vcpu); |
| 1224 | rc = func(vcpu); | 1225 | break; |
| 1226 | case IRQ_PEND_EXT_EMERGENCY: | ||
| 1227 | rc = __deliver_emergency_signal(vcpu); | ||
| 1228 | break; | ||
| 1229 | case IRQ_PEND_EXT_EXTERNAL: | ||
| 1230 | rc = __deliver_external_call(vcpu); | ||
| 1231 | break; | ||
| 1232 | case IRQ_PEND_EXT_CLOCK_COMP: | ||
| 1233 | rc = __deliver_ckc(vcpu); | ||
| 1234 | break; | ||
| 1235 | case IRQ_PEND_EXT_CPU_TIMER: | ||
| 1236 | rc = __deliver_cpu_timer(vcpu); | ||
| 1237 | break; | ||
| 1238 | case IRQ_PEND_RESTART: | ||
| 1239 | rc = __deliver_restart(vcpu); | ||
| 1240 | break; | ||
| 1241 | case IRQ_PEND_SET_PREFIX: | ||
| 1242 | rc = __deliver_set_prefix(vcpu); | ||
| 1243 | break; | ||
| 1244 | case IRQ_PEND_PFAULT_INIT: | ||
| 1245 | rc = __deliver_pfault_init(vcpu); | ||
| 1246 | break; | ||
| 1247 | case IRQ_PEND_EXT_SERVICE: | ||
| 1248 | rc = __deliver_service(vcpu); | ||
| 1249 | break; | ||
| 1250 | case IRQ_PEND_PFAULT_DONE: | ||
| 1251 | rc = __deliver_pfault_done(vcpu); | ||
| 1252 | break; | ||
| 1253 | case IRQ_PEND_VIRTIO: | ||
| 1254 | rc = __deliver_virtio(vcpu); | ||
| 1255 | break; | ||
| 1256 | default: | ||
| 1257 | WARN_ONCE(1, "Unknown pending irq type %ld", irq_type); | ||
| 1258 | clear_bit(irq_type, &li->pending_irqs); | ||
| 1225 | } | 1259 | } |
| 1226 | } | 1260 | } |
| 1227 | 1261 | ||
| @@ -1701,7 +1735,8 @@ static void __floating_irq_kick(struct kvm *kvm, u64 type) · 473 unmodified lines | |||
| 1701 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_STOP_INT); | 1735 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_STOP_INT); |
| 1702 | break; | 1736 | break; |
| 1703 | case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX: | 1737 | case KVM_S390_INT_IO_MIN...KVM_S390_INT_IO_MAX: |
| 1704 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT); | 1738 | if (!(type & KVM_S390_INT_IO_AI_MASK && kvm->arch.gisa)) |
| 1739 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_IO_INT); | ||
| 1705 | break; | 1740 | break; |
| 1706 | default: | 1741 | default: |
| 1707 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_EXT_INT); | 1742 | kvm_s390_set_cpuflags(dst_vcpu, CPUSTAT_EXT_INT); |
| @@ -86,6 +86,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { · 85 unmodified lines | |||
| 86 | { "deliver_prefix_signal", VCPU_STAT(deliver_prefix_signal) }, | 86 | { "deliver_prefix_signal", VCPU_STAT(deliver_prefix_signal) }, |
| 87 | { "deliver_restart_signal", VCPU_STAT(deliver_restart_signal) }, | 87 | { "deliver_restart_signal", VCPU_STAT(deliver_restart_signal) }, |
| 88 | { "deliver_program_interruption", VCPU_STAT(deliver_program_int) }, | 88 | { "deliver_program_interruption", VCPU_STAT(deliver_program_int) }, |
| 89 | { "deliver_io_interrupt", VCPU_STAT(deliver_io_int) }, | ||
| 89 | { "exit_wait_state", VCPU_STAT(exit_wait_state) }, | 90 | { "exit_wait_state", VCPU_STAT(exit_wait_state) }, |
| 90 | { "instruction_epsw", VCPU_STAT(instruction_epsw) }, | 91 | { "instruction_epsw", VCPU_STAT(instruction_epsw) }, |
| 91 | { "instruction_gs", VCPU_STAT(instruction_gs) }, | 92 | { "instruction_gs", VCPU_STAT(instruction_gs) }, |
| @@ -179,6 +180,28 @@ int kvm_arch_hardware_enable(void) · 87 unmodified lines | |||
| 179 | static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start, | 180 | static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start, |
| 180 | unsigned long end); | 181 | unsigned long end); |
| 181 | 182 | ||
| 183 | static void kvm_clock_sync_scb(struct kvm_s390_sie_block *scb, u64 delta) | ||
| 184 | { | ||
| 185 | u8 delta_idx = 0; | ||
| 186 | |||
| 187 | /* | ||
| 188 | * The TOD jumps by delta, we have to compensate this by adding | ||
| 189 | * -delta to the epoch. | ||
| 190 | */ | ||
| 191 | delta = -delta; | ||
| 192 | |||
| 193 | /* sign-extension - we're adding to signed values below */ | ||
| 194 | if ((s64)delta < 0) | ||
| 195 | delta_idx = -1; | ||
| 196 | |||
| 197 | scb->epoch += delta; | ||
| 198 | if (scb->ecd & ECD_MEF) { | ||
| 199 | scb->epdx += delta_idx; | ||
| 200 | if (scb->epoch < delta) | ||
| 201 | scb->epdx += 1; | ||
| 202 | } | ||
| 203 | } | ||
| 204 | |||
| 182 | /* | 205 | /* |
| 183 | * This callback is executed during stop_machine(). All CPUs are therefore | 206 | * This callback is executed during stop_machine(). All CPUs are therefore |
| 184 | * temporarily stopped. In order not to change guest behavior, we have to | 207 | * temporarily stopped. In order not to change guest behavior, we have to |
| @@ -194,13 +217,17 @@ static int kvm_clock_sync(struct notifier_block *notifier, unsigned long val, · 9 unmodified lines | |||
| 194 | unsigned long long *delta = v; | 217 | unsigned long long *delta = v; |
| 195 | 218 | ||
| 196 | list_for_each_entry(kvm, &vm_list, vm_list) { | 219 | list_for_each_entry(kvm, &vm_list, vm_list) { |
| 197 | kvm->arch.epoch -= *delta; | ||
| 198 | kvm_for_each_vcpu(i, vcpu, kvm) { | 220 | kvm_for_each_vcpu(i, vcpu, kvm) { |
| 199 | vcpu->arch.sie_block->epoch -= *delta; | 221 | kvm_clock_sync_scb(vcpu->arch.sie_block, *delta); |
| 222 | if (i == 0) { | ||
| 223 | kvm->arch.epoch = vcpu->arch.sie_block->epoch; | ||
| 224 | kvm->arch.epdx = vcpu->arch.sie_block->epdx; | ||
| 225 | } | ||
| 200 | if (vcpu->arch.cputm_enabled) | 226 | if (vcpu->arch.cputm_enabled) |
| 201 | vcpu->arch.cputm_start += *delta; | 227 | vcpu->arch.cputm_start += *delta; |
| 202 | if (vcpu->arch.vsie_block) | 228 | if (vcpu->arch.vsie_block) |
| 203 | vcpu->arch.vsie_block->epoch -= *delta; | 229 | kvm_clock_sync_scb(vcpu->arch.vsie_block, |
| 230 | *delta); | ||
| 204 | } | 231 | } |
| 205 | } | 232 | } |
| 206 | return NOTIFY_OK; | 233 | return NOTIFY_OK; |
| @@ -902,12 +929,9 @@ static int kvm_s390_set_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr) · 695 unmodified lines | |||
| 902 | if (copy_from_user(>od, (void __user *)attr->addr, sizeof(gtod))) | 929 | if (copy_from_user(>od, (void __user *)attr->addr, sizeof(gtod))) |
| 903 | return -EFAULT; | 930 | return -EFAULT; |
| 904 | 931 | ||
| 905 | if (test_kvm_facility(kvm, 139)) | 932 | if (!test_kvm_facility(kvm, 139) && gtod.epoch_idx) |
| 906 | kvm_s390_set_tod_clock_ext(kvm, >od); | ||
| 907 | else if (gtod.epoch_idx == 0) | ||
| 908 | kvm_s390_set_tod_clock(kvm, gtod.tod); | ||
| 909 | else | ||
| 910 | return -EINVAL; | 933 | return -EINVAL; |
| 934 | kvm_s390_set_tod_clock(kvm, >od); | ||
| 911 | 935 | ||
| 912 | VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx", | 936 | VM_EVENT(kvm, 3, "SET: TOD extension: 0x%x, TOD base: 0x%llx", |
| 913 | gtod.epoch_idx, gtod.tod); | 937 | gtod.epoch_idx, gtod.tod); |
| @@ -932,13 +956,14 @@ static int kvm_s390_set_tod_high(struct kvm *kvm, struct kvm_device_attr *attr) · 18 unmodified lines | |||
| 932 | 956 | ||
| 933 | static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr) | 957 | static int kvm_s390_set_tod_low(struct kvm *kvm, struct kvm_device_attr *attr) |
| 934 | { | 958 | { |
| 935 | u64 gtod; | 959 | struct kvm_s390_vm_tod_clock gtod = { 0 }; |
| 936 | 960 | ||
| 937 | if (copy_from_user(>od, (void __user *)attr->addr, sizeof(gtod))) | 961 | if (copy_from_user(>od.tod, (void __user *)attr->addr, |
| 962 | sizeof(gtod.tod))) | ||
| 938 | return -EFAULT; | 963 | return -EFAULT; |
| 939 | 964 | ||
| 940 | kvm_s390_set_tod_clock(kvm, gtod); | 965 | kvm_s390_set_tod_clock(kvm, >od); |
| 941 | VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod); | 966 | VM_EVENT(kvm, 3, "SET: TOD base: 0x%llx", gtod.tod); |
| 942 | return 0; | 967 | return 0; |
| 943 | } | 968 | } |
| 944 | 969 | ||
| @@ -2122,6 +2147,7 @@ static void sca_add_vcpu(struct kvm_vcpu *vcpu) · 1177 unmodified lines | |||
| 2122 | /* we still need the basic sca for the ipte control */ | 2147 | /* we still need the basic sca for the ipte control */ |
| 2123 | vcpu->arch.sie_block->scaoh = (__u32)(((__u64)sca) >> 32); | 2148 | vcpu->arch.sie_block->scaoh = (__u32)(((__u64)sca) >> 32); |
| 2124 | vcpu->arch.sie_block->scaol = (__u32)(__u64)sca; | 2149 | vcpu->arch.sie_block->scaol = (__u32)(__u64)sca; |
| 2150 | return; | ||
| 2125 | } | 2151 | } |
| 2126 | read_lock(&vcpu->kvm->arch.sca_lock); | 2152 | read_lock(&vcpu->kvm->arch.sca_lock); |
| 2127 | if (vcpu->kvm->arch.use_esca) { | 2153 | if (vcpu->kvm->arch.use_esca) { |
| @@ -2389,6 +2415,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) · 261 unmodified lines | |||
| 2389 | mutex_lock(&vcpu->kvm->lock); | 2415 | mutex_lock(&vcpu->kvm->lock); |
| 2390 | preempt_disable(); | 2416 | preempt_disable(); |
| 2391 | vcpu->arch.sie_block->epoch = vcpu->kvm->arch.epoch; | 2417 | vcpu->arch.sie_block->epoch = vcpu->kvm->arch.epoch; |
| 2418 | vcpu->arch.sie_block->epdx = vcpu->kvm->arch.epdx; | ||
| 2392 | preempt_enable(); | 2419 | preempt_enable(); |
| 2393 | mutex_unlock(&vcpu->kvm->lock); | 2420 | mutex_unlock(&vcpu->kvm->lock); |
| 2394 | if (!kvm_is_ucontrol(vcpu->kvm)) { | 2421 | if (!kvm_is_ucontrol(vcpu->kvm)) { |
| @@ -3021,8 +3048,8 @@ retry: · 626 unmodified lines | |||
| 3021 | return 0; | 3048 | return 0; |
| 3022 | } | 3049 | } |
| 3023 | 3050 | ||
| 3024 | void kvm_s390_set_tod_clock_ext(struct kvm *kvm, | 3051 | void kvm_s390_set_tod_clock(struct kvm *kvm, |
| 3025 | const struct kvm_s390_vm_tod_clock *gtod) | 3052 | const struct kvm_s390_vm_tod_clock *gtod) |
| 3026 | { | 3053 | { |
| 3027 | struct kvm_vcpu *vcpu; | 3054 | struct kvm_vcpu *vcpu; |
| 3028 | struct kvm_s390_tod_clock_ext htod; | 3055 | struct kvm_s390_tod_clock_ext htod; |
| @@ -3034,10 +3061,12 @@ void kvm_s390_set_tod_clock_ext(struct kvm *kvm, · 5 unmodified lines | |||
| 3034 | get_tod_clock_ext((char *)&htod); | 3061 | get_tod_clock_ext((char *)&htod); |
| 3035 | 3062 | ||
| 3036 | kvm->arch.epoch = gtod->tod - htod.tod; | 3063 | kvm->arch.epoch = gtod->tod - htod.tod; |
| 3037 | kvm->arch.epdx = gtod->epoch_idx - htod.epoch_idx; | 3064 | kvm->arch.epdx = 0; |
| 3038 | 3065 | if (test_kvm_facility(kvm, 139)) { | |
| 3039 | if (kvm->arch.epoch > gtod->tod) | 3066 | kvm->arch.epdx = gtod->epoch_idx - htod.epoch_idx; |
| 3040 | kvm->arch.epdx -= 1; | 3067 | if (kvm->arch.epoch > gtod->tod) |
| 3068 | kvm->arch.epdx -= 1; | ||
| 3069 | } | ||
| 3041 | 3070 | ||
| 3042 | kvm_s390_vcpu_block_all(kvm); | 3071 | kvm_s390_vcpu_block_all(kvm); |
| 3043 | kvm_for_each_vcpu(i, vcpu, kvm) { | 3072 | kvm_for_each_vcpu(i, vcpu, kvm) { |
| @@ -3050,22 +3079,6 @@ void kvm_s390_set_tod_clock_ext(struct kvm *kvm, · 6 unmodified lines | |||
| 3050 | mutex_unlock(&kvm->lock); | 3079 | mutex_unlock(&kvm->lock); |
| 3051 | } | 3080 | } |
| 3052 | 3081 | ||
| 3053 | void kvm_s390_set_tod_clock(struct kvm *kvm, u64 tod) | ||
| 3054 | { | ||
| 3055 | struct kvm_vcpu *vcpu; | ||
| 3056 | int i; | ||
| 3057 | |||
| 3058 | mutex_lock(&kvm->lock); | ||
| 3059 | preempt_disable(); | ||
| 3060 | kvm->arch.epoch = tod - get_tod_clock(); | ||
| 3061 | kvm_s390_vcpu_block_all(kvm); | ||
| 3062 | kvm_for_each_vcpu(i, vcpu, kvm) | ||
| 3063 | vcpu->arch.sie_block->epoch = kvm->arch.epoch; | ||
| 3064 | kvm_s390_vcpu_unblock_all(kvm); | ||
| 3065 | preempt_enable(); | ||
| 3066 | mutex_unlock(&kvm->lock); | ||
| 3067 | } | ||
| 3068 | |||
| 3069 | /** | 3082 | /** |
| 3070 | * kvm_arch_fault_in_page - fault-in guest page if necessary | 3083 | * kvm_arch_fault_in_page - fault-in guest page if necessary |
| 3071 | * @vcpu: The corresponding virtual cpu | 3084 | * @vcpu: The corresponding virtual cpu |
| @@ -19,8 +19,6 @@ · 18 unmodified lines | |||
| 19 | #include <asm/processor.h> | 19 | #include <asm/processor.h> |
| 20 | #include <asm/sclp.h> | 20 | #include <asm/sclp.h> |
| 21 | 21 | ||
| 22 | typedef int (*intercept_handler_t)(struct kvm_vcpu *vcpu); | ||
| 23 | |||
| 24 | /* Transactional Memory Execution related macros */ | 22 | /* Transactional Memory Execution related macros */ |
| 25 | #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE)) | 23 | #define IS_TE_ENABLED(vcpu) ((vcpu->arch.sie_block->ecb & ECB_TE)) |
| 26 | #define TDB_FORMAT1 1 | 24 | #define TDB_FORMAT1 1 |
| @@ -283,9 +281,8 @@ int kvm_s390_handle_sigp(struct kvm_vcpu *vcpu); · 256 unmodified lines | |||
| 283 | int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu); | 281 | int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu); |
| 284 | 282 | ||
| 285 | /* implemented in kvm-s390.c */ | 283 | /* implemented in kvm-s390.c */ |
| 286 | void kvm_s390_set_tod_clock_ext(struct kvm *kvm, | 284 | void kvm_s390_set_tod_clock(struct kvm *kvm, |
| 287 | const struct kvm_s390_vm_tod_clock *gtod); | 285 | const struct kvm_s390_vm_tod_clock *gtod); |
| 288 | void kvm_s390_set_tod_clock(struct kvm *kvm, u64 tod); | ||
| 289 | long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable); | 286 | long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable); |
| 290 | int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr); | 287 | int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr); |
| 291 | int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr); | 288 | int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr); |
| @@ -85,9 +85,10 @@ int kvm_s390_handle_e3(struct kvm_vcpu *vcpu) · 84 unmodified lines | |||
| 85 | /* Handle SCK (SET CLOCK) interception */ | 85 | /* Handle SCK (SET CLOCK) interception */ |
| 86 | static int handle_set_clock(struct kvm_vcpu *vcpu) | 86 | static int handle_set_clock(struct kvm_vcpu *vcpu) |
| 87 | { | 87 | { |
| 88 | struct kvm_s390_vm_tod_clock gtod = { 0 }; | ||
| 88 | int rc; | 89 | int rc; |
| 89 | u8 ar; | 90 | u8 ar; |
| 90 | u64 op2, val; | 91 | u64 op2; |
| 91 | 92 | ||
| 92 | vcpu->stat.instruction_sck++; | 93 | vcpu->stat.instruction_sck++; |
| 93 | 94 | ||
| @@ -97,12 +98,12 @@ static int handle_set_clock(struct kvm_vcpu *vcpu) · 3 unmodified lines | |||
| 97 | op2 = kvm_s390_get_base_disp_s(vcpu, &ar); | 98 | op2 = kvm_s390_get_base_disp_s(vcpu, &ar); |
| 98 | if (op2 & 7) /* Operand must be on a doubleword boundary */ | 99 | if (op2 & 7) /* Operand must be on a doubleword boundary */ |
| 99 | return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION); | 100 | return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION); |
| 100 | rc = read_guest(vcpu, op2, ar, &val, sizeof(val)); | 101 | rc = read_guest(vcpu, op2, ar, >od.tod, sizeof(gtod.tod)); |
| 101 | if (rc) | 102 | if (rc) |
| 102 | return kvm_s390_inject_prog_cond(vcpu, rc); | 103 | return kvm_s390_inject_prog_cond(vcpu, rc); |
| 103 | 104 | ||
| 104 | VCPU_EVENT(vcpu, 3, "SCK: setting guest TOD to 0x%llx", val); | 105 | VCPU_EVENT(vcpu, 3, "SCK: setting guest TOD to 0x%llx", gtod.tod); |
| 105 | kvm_s390_set_tod_clock(vcpu->kvm, val); | 106 | kvm_s390_set_tod_clock(vcpu->kvm, >od); |
| 106 | 107 | ||
| 107 | kvm_s390_set_psw_cc(vcpu, 0); | 108 | kvm_s390_set_psw_cc(vcpu, 0); |
| 108 | return 0; | 109 | return 0; |
| @@ -795,55 +796,60 @@ out: · 686 unmodified lines | |||
| 795 | return rc; | 796 | return rc; |
| 796 | } | 797 | } |
| 797 | 798 | ||
| 798 | static const intercept_handler_t b2_handlers[256] = { | ||
| 799 | [0x02] = handle_stidp, | ||
| 800 | [0x04] = handle_set_clock, | ||
| 801 | [0x10] = handle_set_prefix, | ||
| 802 | [0x11] = handle_store_prefix, | ||
| 803 | [0x12] = handle_store_cpu_address, | ||
| 804 | [0x14] = kvm_s390_handle_vsie, | ||
| 805 | [0x21] = handle_ipte_interlock, | ||
| 806 | [0x29] = handle_iske, | ||
| 807 | [0x2a] = handle_rrbe, | ||
| 808 | [0x2b] = handle_sske, | ||
| 809 | [0x2c] = handle_test_block, | ||
| 810 | [0x30] = handle_io_inst, | ||
| 811 | [0x31] = handle_io_inst, | ||
| 812 | [0x32] = handle_io_inst, | ||
| 813 | [0x33] = handle_io_inst, | ||
| 814 | [0x34] = handle_io_inst, | ||
| 815 | [0x35] = handle_io_inst, | ||
| 816 | [0x36] = handle_io_inst, | ||
| 817 | [0x37] = handle_io_inst, | ||
| 818 | [0x38] = handle_io_inst, | ||
| 819 | [0x39] = handle_io_inst, | ||
| 820 | [0x3a] = handle_io_inst, | ||
| 821 | [0x3b] = handle_io_inst, | ||
| 822 | [0x3c] = handle_io_inst, | ||
| 823 | [0x50] = handle_ipte_interlock, | ||
| 824 | [0x56] = handle_sthyi, | ||
| 825 | [0x5f] = handle_io_inst, | ||
| 826 | [0x74] = handle_io_inst, | ||
| 827 | [0x76] = handle_io_inst, | ||
| 828 | [0x7d] = handle_stsi, | ||
| 829 | [0xb1] = handle_stfl, | ||
| 830 | [0xb2] = handle_lpswe, | ||
| 831 | }; | ||
| 832 | |||
| 833 | int kvm_s390_handle_b2(struct kvm_vcpu *vcpu) | 799 | int kvm_s390_handle_b2(struct kvm_vcpu *vcpu) |
| 834 | { | 800 | { |
| 835 | intercept_handler_t handler; | 801 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 836 | 802 | case 0x02: | |
| 837 | /* | 803 | return handle_stidp(vcpu); |
| 838 | * A lot of B2 instructions are priviledged. Here we check for | 804 | case 0x04: |
| 839 | * the privileged ones, that we can handle in the kernel. | 805 | return handle_set_clock(vcpu); |
| 840 | * Anything else goes to userspace. | 806 | case 0x10: |
| 841 | */ | 807 | return handle_set_prefix(vcpu); |
| 842 | handler = b2_handlers[vcpu->arch.sie_block->ipa & 0x00ff]; | 808 | case 0x11: |
| 843 | if (handler) | 809 | return handle_store_prefix(vcpu); |
| 844 | return handler(vcpu); | 810 | case 0x12: |
| 845 | 811 | return handle_store_cpu_address(vcpu); | |
| 846 | return -EOPNOTSUPP; | 812 | case 0x14: |
| 813 | return kvm_s390_handle_vsie(vcpu); | ||
| 814 | case 0x21: | ||
| 815 | case 0x50: | ||
| 816 | return handle_ipte_interlock(vcpu); | ||
| 817 | case 0x29: | ||
| 818 | return handle_iske(vcpu); | ||
| 819 | case 0x2a: | ||
| 820 | return handle_rrbe(vcpu); | ||
| 821 | case 0x2b: | ||
| 822 | return handle_sske(vcpu); | ||
| 823 | case 0x2c: | ||
| 824 | return handle_test_block(vcpu); | ||
| 825 | case 0x30: | ||
| 826 | case 0x31: | ||
| 827 | case 0x32: | ||
| 828 | case 0x33: | ||
| 829 | case 0x34: | ||
| 830 | case 0x35: | ||
| 831 | case 0x36: | ||
| 832 | case 0x37: | ||
| 833 | case 0x38: | ||
| 834 | case 0x39: | ||
| 835 | case 0x3a: | ||
| 836 | case 0x3b: | ||
| 837 | case 0x3c: | ||
| 838 | case 0x5f: | ||
| 839 | case 0x74: | ||
| 840 | case 0x76: | ||
| 841 | return handle_io_inst(vcpu); | ||
| 842 | case 0x56: | ||
| 843 | return handle_sthyi(vcpu); | ||
| 844 | case 0x7d: | ||
| 845 | return handle_stsi(vcpu); | ||
| 846 | case 0xb1: | ||
| 847 | return handle_stfl(vcpu); | ||
| 848 | case 0xb2: | ||
| 849 | return handle_lpswe(vcpu); | ||
| 850 | default: | ||
| 851 | return -EOPNOTSUPP; | ||
| 852 | } | ||
| 847 | } | 853 | } |
| 848 | 854 | ||
| 849 | static int handle_epsw(struct kvm_vcpu *vcpu) | 855 | static int handle_epsw(struct kvm_vcpu *vcpu) |
| @@ -1105,25 +1111,22 @@ static int handle_essa(struct kvm_vcpu *vcpu) · 255 unmodified lines | |||
| 1105 | return 0; | 1111 | return 0; |
| 1106 | } | 1112 | } |
| 1107 | 1113 | ||
| 1108 | static const intercept_handler_t b9_handlers[256] = { | ||
| 1109 | [0x8a] = handle_ipte_interlock, | ||
| 1110 | [0x8d] = handle_epsw, | ||
| 1111 | [0x8e] = handle_ipte_interlock, | ||
| 1112 | [0x8f] = handle_ipte_interlock, | ||
| 1113 | [0xab] = handle_essa, | ||
| 1114 | [0xaf] = handle_pfmf, | ||
| 1115 | }; | ||
| 1116 | |||
| 1117 | int kvm_s390_handle_b9(struct kvm_vcpu *vcpu) | 1114 | int kvm_s390_handle_b9(struct kvm_vcpu *vcpu) |
| 1118 | { | 1115 | { |
| 1119 | intercept_handler_t handler; | 1116 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 1120 | 1117 | case 0x8a: | |
| 1121 | /* This is handled just as for the B2 instructions. */ | 1118 | case 0x8e: |
| 1122 | handler = b9_handlers[vcpu->arch.sie_block->ipa & 0x00ff]; | 1119 | case 0x8f: |
| 1123 | if (handler) | 1120 | return handle_ipte_interlock(vcpu); |
| 1124 | return handler(vcpu); | 1121 | case 0x8d: |
| 1125 | 1122 | return handle_epsw(vcpu); | |
| 1126 | return -EOPNOTSUPP; | 1123 | case 0xab: |
| 1124 | return handle_essa(vcpu); | ||
| 1125 | case 0xaf: | ||
| 1126 | return handle_pfmf(vcpu); | ||
| 1127 | default: | ||
| 1128 | return -EOPNOTSUPP; | ||
| 1129 | } | ||
| 1127 | } | 1130 | } |
| 1128 | 1131 | ||
| 1129 | int kvm_s390_handle_lctl(struct kvm_vcpu *vcpu) | 1132 | int kvm_s390_handle_lctl(struct kvm_vcpu *vcpu) |
| @@ -1271,22 +1274,20 @@ static int handle_stctg(struct kvm_vcpu *vcpu) · 141 unmodified lines | |||
| 1271 | return rc ? kvm_s390_inject_prog_cond(vcpu, rc) : 0; | 1274 | return rc ? kvm_s390_inject_prog_cond(vcpu, rc) : 0; |
| 1272 | } | 1275 | } |
| 1273 | 1276 | ||
| 1274 | static const intercept_handler_t eb_handlers[256] = { | ||
| 1275 | [0x2f] = handle_lctlg, | ||
| 1276 | [0x25] = handle_stctg, | ||
| 1277 | [0x60] = handle_ri, | ||
| 1278 | [0x61] = handle_ri, | ||
| 1279 | [0x62] = handle_ri, | ||
| 1280 | }; | ||
| 1281 | |||
| 1282 | int kvm_s390_handle_eb(struct kvm_vcpu *vcpu) | 1277 | int kvm_s390_handle_eb(struct kvm_vcpu *vcpu) |
| 1283 | { | 1278 | { |
| 1284 | intercept_handler_t handler; | 1279 | switch (vcpu->arch.sie_block->ipb & 0x000000ff) { |
| 1285 | 1280 | case 0x25: | |
| 1286 | handler = eb_handlers[vcpu->arch.sie_block->ipb & 0xff]; | 1281 | return handle_stctg(vcpu); |
| 1287 | if (handler) | 1282 | case 0x2f: |
| 1288 | return handler(vcpu); | 1283 | return handle_lctlg(vcpu); |
| 1289 | return -EOPNOTSUPP; | 1284 | case 0x60: |
| 1285 | case 0x61: | ||
| 1286 | case 0x62: | ||
| 1287 | return handle_ri(vcpu); | ||
| 1288 | default: | ||
| 1289 | return -EOPNOTSUPP; | ||
| 1290 | } | ||
| 1290 | } | 1291 | } |
| 1291 | 1292 | ||
| 1292 | static int handle_tprot(struct kvm_vcpu *vcpu) | 1293 | static int handle_tprot(struct kvm_vcpu *vcpu) |
| @@ -1346,10 +1347,12 @@ out_unlock: · 53 unmodified lines | |||
| 1346 | 1347 | ||
| 1347 | int kvm_s390_handle_e5(struct kvm_vcpu *vcpu) | 1348 | int kvm_s390_handle_e5(struct kvm_vcpu *vcpu) |
| 1348 | { | 1349 | { |
| 1349 | /* For e5xx... instructions we only handle TPROT */ | 1350 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 1350 | if ((vcpu->arch.sie_block->ipa & 0x00ff) == 0x01) | 1351 | case 0x01: |
| 1351 | return handle_tprot(vcpu); | 1352 | return handle_tprot(vcpu); |
| 1352 | return -EOPNOTSUPP; | 1353 | default: |
| 1354 | return -EOPNOTSUPP; | ||
| 1355 | } | ||
| 1353 | } | 1356 | } |
| 1354 | 1357 | ||
| 1355 | static int handle_sckpf(struct kvm_vcpu *vcpu) | 1358 | static int handle_sckpf(struct kvm_vcpu *vcpu) |
| @@ -1380,17 +1383,14 @@ static int handle_ptff(struct kvm_vcpu *vcpu) · 24 unmodified lines | |||
| 1380 | return 0; | 1383 | return 0; |
| 1381 | } | 1384 | } |
| 1382 | 1385 | ||
| 1383 | static const intercept_handler_t x01_handlers[256] = { | ||
| 1384 | [0x04] = handle_ptff, | ||
| 1385 | [0x07] = handle_sckpf, | ||
| 1386 | }; | ||
| 1387 | |||
| 1388 | int kvm_s390_handle_01(struct kvm_vcpu *vcpu) | 1386 | int kvm_s390_handle_01(struct kvm_vcpu *vcpu) |
| 1389 | { | 1387 | { |
| 1390 | intercept_handler_t handler; | 1388 | switch (vcpu->arch.sie_block->ipa & 0x00ff) { |
| 1391 | 1389 | case 0x04: | |
| 1392 | handler = x01_handlers[vcpu->arch.sie_block->ipa & 0x00ff]; | 1390 | return handle_ptff(vcpu); |
| 1393 | if (handler) | 1391 | case 0x07: |
| 1394 | return handler(vcpu); | 1392 | return handle_sckpf(vcpu); |
| 1395 | return -EOPNOTSUPP; | 1393 | default: |
| 1394 | return -EOPNOTSUPP; | ||
| 1395 | } | ||
| 1396 | } | 1396 | } |
| @@ -821,6 +821,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 820 unmodified lines | |||
| 821 | { | 821 | { |
| 822 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; | 822 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; |
| 823 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; | 823 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; |
| 824 | int guest_bp_isolation; | ||
| 824 | int rc; | 825 | int rc; |
| 825 | 826 | ||
| 826 | handle_last_fault(vcpu, vsie_page); | 827 | handle_last_fault(vcpu, vsie_page); |
| @@ -831,6 +832,20 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 4 unmodified lines | |||
| 831 | s390_handle_mcck(); | 832 | s390_handle_mcck(); |
| 832 | 833 | ||
| 833 | srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); | 834 | srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx); |
| 835 | |||
| 836 | /* save current guest state of bp isolation override */ | ||
| 837 | guest_bp_isolation = test_thread_flag(TIF_ISOLATE_BP_GUEST); | ||
| 838 | |||
| 839 | /* | ||
| 840 | * The guest is running with BPBC, so we have to force it on for our | ||
| 841 | * nested guest. This is done by enabling BPBC globally, so the BPBC | ||
| 842 | * control in the SCB (which the nested guest can modify) is simply | ||
| 843 | * ignored. | ||
| 844 | */ | ||
| 845 | if (test_kvm_facility(vcpu->kvm, 82) && | ||
| 846 | vcpu->arch.sie_block->fpf & FPF_BPBC) | ||
| 847 | set_thread_flag(TIF_ISOLATE_BP_GUEST); | ||
| 848 | |||
| 834 | local_irq_disable(); | 849 | local_irq_disable(); |
| 835 | guest_enter_irqoff(); | 850 | guest_enter_irqoff(); |
| 836 | local_irq_enable(); | 851 | local_irq_enable(); |
| @@ -840,6 +855,11 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 3 unmodified lines | |||
| 840 | local_irq_disable(); | 855 | local_irq_disable(); |
| 841 | guest_exit_irqoff(); | 856 | guest_exit_irqoff(); |
| 842 | local_irq_enable(); | 857 | local_irq_enable(); |
| 858 | |||
| 859 | /* restore guest state for bp isolation override */ | ||
| 860 | if (!guest_bp_isolation) | ||
| 861 | clear_thread_flag(TIF_ISOLATE_BP_GUEST); | ||
| 862 | |||
| 843 | vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu); | 863 | vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu); |
| 844 | 864 | ||
| 845 | if (rc == -EINTR) { | 865 | if (rc == -EINTR) { |
| @@ -1 +1,3 @@ | |||
| 1 | obj-$(CONFIG_USE_BUILTIN_DTB) += $(patsubst "%",%,$(CONFIG_BUILTIN_DTB_SOURCE)).dtb.o | 1 | ifneq ($(CONFIG_BUILTIN_DTB_SOURCE),"") |
| 2 | obj-y += $(patsubst "%",%,$(CONFIG_BUILTIN_DTB_SOURCE)).dtb.o | ||
| 3 | endif | ||
| @@ -430,6 +430,8 @@ config SPARC_LEON · 429 unmodified lines | |||
| 430 | depends on SPARC32 | 430 | depends on SPARC32 |
| 431 | select USB_EHCI_BIG_ENDIAN_MMIO | 431 | select USB_EHCI_BIG_ENDIAN_MMIO |
| 432 | select USB_EHCI_BIG_ENDIAN_DESC | 432 | select USB_EHCI_BIG_ENDIAN_DESC |
| 433 | select USB_UHCI_BIG_ENDIAN_MMIO | ||
| 434 | select USB_UHCI_BIG_ENDIAN_DESC | ||
| 433 | ---help--- | 435 | ---help--- |
| 434 | If you say Y here if you are running on a SPARC-LEON processor. | 436 | If you say Y here if you are running on a SPARC-LEON processor. |
| 435 | The LEON processor is a synthesizable VHDL model of the | 437 | The LEON processor is a synthesizable VHDL model of the |
| @@ -9,10 +9,14 @@ · 8 unmodified lines | |||
| 9 | void do_BUG(const char *file, int line); | 9 | void do_BUG(const char *file, int line); |
| 10 | #define BUG() do { \ | 10 | #define BUG() do { \ |
| 11 | do_BUG(__FILE__, __LINE__); \ | 11 | do_BUG(__FILE__, __LINE__); \ |
| 12 | barrier_before_unreachable(); \ | ||
| 12 | __builtin_trap(); \ | 13 | __builtin_trap(); \ |
| 13 | } while (0) | 14 | } while (0) |
| 14 | #else | 15 | #else |
| 15 | #define BUG() __builtin_trap() | 16 | #define BUG() do { \ |
| 17 | barrier_before_unreachable(); \ | ||
| 18 | __builtin_trap(); \ | ||
| 19 | } while (0) | ||
| 16 | #endif | 20 | #endif |
| 17 | 21 | ||
| 18 | #define HAVE_ARCH_BUG | 22 | #define HAVE_ARCH_BUG |
| @@ -1,6 +1,7 @@ | |||
| 1 | boot/compressed/vmlinux | 1 | boot/compressed/vmlinux |
| 2 | tools/test_get_len | 2 | tools/test_get_len |
| 3 | tools/insn_sanity | 3 | tools/insn_sanity |
| 4 | tools/insn_decoder_test | ||
| 4 | purgatory/kexec-purgatory.c | 5 | purgatory/kexec-purgatory.c |
| 5 | purgatory/purgatory.ro | 6 | purgatory/purgatory.ro |
| 6 | 7 | ||
| @@ -423,12 +423,6 @@ config X86_MPPARSE · 422 unmodified lines | |||
| 423 | For old smp systems that do not have proper acpi support. Newer systems | 423 | For old smp systems that do not have proper acpi support. Newer systems |
| 424 | (esp with 64bit cpus) with acpi support, MADT and DSDT will override it | 424 | (esp with 64bit cpus) with acpi support, MADT and DSDT will override it |
| 425 | 425 | ||
| 426 | config X86_BIGSMP | ||
| 427 | bool "Support for big SMP systems with more than 8 CPUs" | ||
| 428 | depends on X86_32 && SMP | ||
| 429 | ---help--- | ||
| 430 | This option is needed for the systems that have more than 8 CPUs | ||
| 431 | |||
| 432 | config GOLDFISH | 426 | config GOLDFISH |
| 433 | def_bool y | 427 | def_bool y |
| 434 | depends on X86_GOLDFISH | 428 | depends on X86_GOLDFISH |
| @@ -436,6 +430,7 @@ config GOLDFISH · 1 unmodified line | |||
| 436 | config RETPOLINE | 430 | config RETPOLINE |
| 437 | bool "Avoid speculative indirect branches in kernel" | 431 | bool "Avoid speculative indirect branches in kernel" |
| 438 | default y | 432 | default y |
| 433 | select STACK_VALIDATION if HAVE_STACK_VALIDATION | ||
| 439 | help | 434 | help |
| 440 | Compile kernel with the retpoline compiler options to guard against | 435 | Compile kernel with the retpoline compiler options to guard against |
| 441 | kernel-to-user data leaks by avoiding speculative indirect | 436 | kernel-to-user data leaks by avoiding speculative indirect |
| @@ -460,6 +455,12 @@ config INTEL_RDT · 18 unmodified lines | |||
| 460 | Say N if unsure. | 455 | Say N if unsure. |
| 461 | 456 | ||
| 462 | if X86_32 | 457 | if X86_32 |
| 458 | config X86_BIGSMP | ||
| 459 | bool "Support for big SMP systems with more than 8 CPUs" | ||
| 460 | depends on SMP | ||
| 461 | ---help--- | ||
| 462 | This option is needed for the systems that have more than 8 CPUs | ||
| 463 | |||
| 463 | config X86_EXTENDED_PLATFORM | 464 | config X86_EXTENDED_PLATFORM |
| 464 | bool "Support for extended (non-PC) x86 platforms" | 465 | bool "Support for extended (non-PC) x86 platforms" |
| 465 | default y | 466 | default y |
| @@ -949,25 +950,66 @@ config MAXSMP · 483 unmodified lines | |||
| 949 | Enable maximum number of CPUS and NUMA Nodes for this architecture. | 950 | Enable maximum number of CPUS and NUMA Nodes for this architecture. |
| 950 | If unsure, say N. | 951 | If unsure, say N. |
| 951 | 952 | ||
| 953 | # | ||
| 954 | # The maximum number of CPUs supported: | ||
| 955 | # | ||
| 956 | # The main config value is NR_CPUS, which defaults to NR_CPUS_DEFAULT, | ||
| 957 | # and which can be configured interactively in the | ||
| 958 | # [NR_CPUS_RANGE_BEGIN ... NR_CPUS_RANGE_END] range. | ||
| 959 | # | ||
| 960 | # The ranges are different on 32-bit and 64-bit kernels, depending on | ||
| 961 | # hardware capabilities and scalability features of the kernel. | ||
| 962 | # | ||
| 963 | # ( If MAXSMP is enabled we just use the highest possible value and disable | ||
| 964 | # interactive configuration. ) | ||
| 965 | # | ||
| 966 | |||
| 967 | config NR_CPUS_RANGE_BEGIN | ||
| 968 | int | ||
| 969 | default NR_CPUS_RANGE_END if MAXSMP | ||
| 970 | default 1 if !SMP | ||
| 971 | default 2 | ||
| 972 | |||
| 973 | config NR_CPUS_RANGE_END | ||
| 974 | int | ||
| 975 | depends on X86_32 | ||
| 976 | default 64 if SMP && X86_BIGSMP | ||
| 977 | default 8 if SMP && !X86_BIGSMP | ||
| 978 | default 1 if !SMP | ||
| 979 | |||
| 980 | config NR_CPUS_RANGE_END | ||
| 981 | int | ||
| 982 | depends on X86_64 | ||
| 983 | default 8192 if SMP && ( MAXSMP || CPUMASK_OFFSTACK) | ||
| 984 | default 512 if SMP && (!MAXSMP && !CPUMASK_OFFSTACK) | ||
| 985 | default 1 if !SMP | ||
| 986 | |||
| 987 | config NR_CPUS_DEFAULT | ||
| 988 | int | ||
| 989 | depends on X86_32 | ||
| 990 | default 32 if X86_BIGSMP | ||
| 991 | default 8 if SMP | ||
| 992 | default 1 if !SMP | ||
| 993 | |||
| 994 | config NR_CPUS_DEFAULT | ||
| 995 | int | ||
| 996 | depends on X86_64 | ||
| 997 | default 8192 if MAXSMP | ||
| 998 | default 64 if SMP | ||
| 999 | default 1 if !SMP | ||
| 1000 | |||
| 952 | config NR_CPUS | 1001 | config NR_CPUS |
| 953 | int "Maximum number of CPUs" if SMP && !MAXSMP | 1002 | int "Maximum number of CPUs" if SMP && !MAXSMP |
| 954 | range 2 8 if SMP && X86_32 && !X86_BIGSMP | 1003 | range NR_CPUS_RANGE_BEGIN NR_CPUS_RANGE_END |
| 955 | range 2 64 if SMP && X86_32 && X86_BIGSMP | 1004 | default NR_CPUS_DEFAULT |
| 956 | range 2 512 if SMP && !MAXSMP && !CPUMASK_OFFSTACK && X86_64 | ||
| 957 | range 2 8192 if SMP && !MAXSMP && CPUMASK_OFFSTACK && X86_64 | ||
| 958 | default "1" if !SMP | ||
| 959 | default "8192" if MAXSMP | ||
| 960 | default "32" if SMP && X86_BIGSMP | ||
| 961 | default "8" if SMP && X86_32 | ||
| 962 | default "64" if SMP | ||
| 963 | ---help--- | 1005 | ---help--- |
| 964 | This allows you to specify the maximum number of CPUs which this | 1006 | This allows you to specify the maximum number of CPUs which this |
| 965 | kernel will support. If CPUMASK_OFFSTACK is enabled, the maximum | 1007 | kernel will support. If CPUMASK_OFFSTACK is enabled, the maximum |
| 966 | supported value is 8192, otherwise the maximum value is 512. The | 1008 | supported value is 8192, otherwise the maximum value is 512. The |
| 967 | minimum value which makes sense is 2. | 1009 | minimum value which makes sense is 2. |
| 968 | 1010 | ||
| 969 | This is purely to save memory - each supported CPU adds | 1011 | This is purely to save memory: each supported CPU adds about 8KB |
| 970 | approximately eight kilobytes to the kernel image. | 1012 | to the kernel image. |
| 971 | 1013 | ||
| 972 | config SCHED_SMT | 1014 | config SCHED_SMT |
| 973 | bool "SMT (Hyperthreading) scheduler support" | 1015 | bool "SMT (Hyperthreading) scheduler support" |
| @@ -1363,7 +1405,7 @@ config HIGHMEM4G · 389 unmodified lines | |||
| 1363 | 1405 | ||
| 1364 | config HIGHMEM64G | 1406 | config HIGHMEM64G |
| 1365 | bool "64GB" | 1407 | bool "64GB" |
| 1366 | depends on !M486 | 1408 | depends on !M486 && !M586 && !M586TSC && !M586MMX && !MGEODE_LX && !MGEODEGX1 && !MCYRIXIII && !MELAN && !MWINCHIPC6 && !WINCHIP3D && !MK6 |
| 1367 | select X86_PAE | 1409 | select X86_PAE |
| 1368 | ---help--- | 1410 | ---help--- |
| 1369 | Select this if you have a 32-bit processor and more than 4 | 1411 | Select this if you have a 32-bit processor and more than 4 |
| @@ -2265,7 +2307,7 @@ choice · 895 unmodified lines | |||
| 2265 | it can be used to assist security vulnerability exploitation. | 2307 | it can be used to assist security vulnerability exploitation. |
| 2266 | 2308 | ||
| 2267 | This setting can be changed at boot time via the kernel command | 2309 | This setting can be changed at boot time via the kernel command |
| 2268 | line parameter vsyscall=[native|emulate|none]. | 2310 | line parameter vsyscall=[emulate|none]. |
| 2269 | 2311 | ||
| 2270 | On a system with recent enough glibc (2.14 or newer) and no | 2312 | On a system with recent enough glibc (2.14 or newer) and no |
| 2271 | static binaries, you can say None without a performance penalty | 2313 | static binaries, you can say None without a performance penalty |
| @@ -2273,15 +2315,6 @@ choice · 1 unmodified line | |||
| 2273 | 2315 | ||
| 2274 | If unsure, select "Emulate". | 2316 | If unsure, select "Emulate". |
| 2275 | 2317 | ||
| 2276 | config LEGACY_VSYSCALL_NATIVE | ||
| 2277 | bool "Native" | ||
| 2278 | help | ||
| 2279 | Actual executable code is located in the fixed vsyscall | ||
| 2280 | address mapping, implementing time() efficiently. Since | ||
| 2281 | this makes the mapping executable, it can be used during | ||
| 2282 | security vulnerability exploitation (traditionally as | ||
| 2283 | ROP gadgets). This configuration is not recommended. | ||
| 2284 | |||
| 2285 | config LEGACY_VSYSCALL_EMULATE | 2318 | config LEGACY_VSYSCALL_EMULATE |
| 2286 | bool "Emulate" | 2319 | bool "Emulate" |
| 2287 | help | 2320 | help |
| @@ -374,7 +374,7 @@ config X86_TSC · 373 unmodified lines | |||
| 374 | 374 | ||
| 375 | config X86_CMPXCHG64 | 375 | config X86_CMPXCHG64 |
| 376 | def_bool y | 376 | def_bool y |
| 377 | depends on X86_PAE || X86_64 || MCORE2 || MPENTIUM4 || MPENTIUMM || MPENTIUMIII || MPENTIUMII || M686 || MATOM | 377 | depends on X86_PAE || X86_64 || MCORE2 || MPENTIUM4 || MPENTIUMM || MPENTIUMIII || MPENTIUMII || M686 || M586TSC || M586MMX || MATOM || MGEODE_LX || MGEODEGX1 || MK6 || MK7 || MK8 |
| 378 | 378 | ||
| 379 | # this should be set for all -march=.. options where the compiler | 379 | # this should be set for all -march=.. options where the compiler |
| 380 | # generates cmov. | 380 | # generates cmov. |
| @@ -385,7 +385,7 @@ config X86_CMOV · 4 unmodified lines | |||
| 385 | config X86_MINIMUM_CPU_FAMILY | 385 | config X86_MINIMUM_CPU_FAMILY |
| 386 | int | 386 | int |
| 387 | default "64" if X86_64 | 387 | default "64" if X86_64 |
| 388 | default "6" if X86_32 && X86_P6_NOP | 388 | default "6" if X86_32 && (MPENTIUM4 || MPENTIUMM || MPENTIUMIII || MPENTIUMII || M686 || MVIAC3_2 || MVIAC7 || MEFFICEON || MATOM || MCRUSOE || MCORE2 || MK7 || MK8) |
| 389 | default "5" if X86_32 && X86_CMPXCHG64 | 389 | default "5" if X86_32 && X86_CMPXCHG64 |
| 390 | default "4" | 390 | default "4" |
| 391 | 391 | ||
| @@ -232,10 +232,9 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables · 231 unmodified lines | |||
| 232 | 232 | ||
| 233 | # Avoid indirect branches in kernel to deal with Spectre | 233 | # Avoid indirect branches in kernel to deal with Spectre |
| 234 | ifdef CONFIG_RETPOLINE | 234 | ifdef CONFIG_RETPOLINE |
| 235 | RETPOLINE_CFLAGS += $(call cc-option,-mindirect-branch=thunk-extern -mindirect-branch-register) | 235 | ifneq ($(RETPOLINE_CFLAGS),) |
| 236 | ifneq ($(RETPOLINE_CFLAGS),) | 236 | KBUILD_CFLAGS += $(RETPOLINE_CFLAGS) -DRETPOLINE |
| 237 | KBUILD_CFLAGS += $(RETPOLINE_CFLAGS) -DRETPOLINE | 237 | endif |
| 238 | endif | ||
| 239 | endif | 238 | endif |
| 240 | 239 | ||
| 241 | archscripts: scripts_basic | 240 | archscripts: scripts_basic |
| @@ -439,7 +439,7 @@ setup_uga32(void **uga_handle, unsigned long size, u32 *width, u32 *height) · 438 unmodified lines | |||
| 439 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; | 439 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; |
| 440 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; | 440 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; |
| 441 | unsigned long nr_ugas; | 441 | unsigned long nr_ugas; |
| 442 | u32 *handles = (u32 *)uga_handle;; | 442 | u32 *handles = (u32 *)uga_handle; |
| 443 | efi_status_t status = EFI_INVALID_PARAMETER; | 443 | efi_status_t status = EFI_INVALID_PARAMETER; |
| 444 | int i; | 444 | int i; |
| 445 | 445 | ||
| @@ -484,7 +484,7 @@ setup_uga64(void **uga_handle, unsigned long size, u32 *width, u32 *height) · 38 unmodified lines | |||
| 484 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; | 484 | struct efi_uga_draw_protocol *uga = NULL, *first_uga; |
| 485 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; | 485 | efi_guid_t uga_proto = EFI_UGA_PROTOCOL_GUID; |
| 486 | unsigned long nr_ugas; | 486 | unsigned long nr_ugas; |
| 487 | u64 *handles = (u64 *)uga_handle;; | 487 | u64 *handles = (u64 *)uga_handle; |
| 488 | efi_status_t status = EFI_INVALID_PARAMETER; | 488 | efi_status_t status = EFI_INVALID_PARAMETER; |
| 489 | int i; | 489 | int i; |
| 490 | 490 | ||
| @@ -57,10 +57,12 @@ void sha512_mb_mgr_init_avx2(struct sha512_mb_mgr *state) · 56 unmodified lines | |||
| 57 | { | 57 | { |
| 58 | unsigned int j; | 58 | unsigned int j; |
| 59 | 59 | ||
| 60 | state->lens[0] = 0; | 60 | /* initially all lanes are unused */ |
| 61 | state->lens[1] = 1; | 61 | state->lens[0] = 0xFFFFFFFF00000000; |
| 62 | state->lens[2] = 2; | 62 | state->lens[1] = 0xFFFFFFFF00000001; |
| 63 | state->lens[3] = 3; | 63 | state->lens[2] = 0xFFFFFFFF00000002; |
| 64 | state->lens[3] = 0xFFFFFFFF00000003; | ||
| 65 | |||
| 64 | state->unused_lanes = 0xFF03020100; | 66 | state->unused_lanes = 0xFF03020100; |
| 65 | for (j = 0; j < 4; j++) | 67 | for (j = 0; j < 4; j++) |
| 66 | state->ldata[j].job_in_lane = NULL; | 68 | state->ldata[j].job_in_lane = NULL; |
| @@ -97,80 +97,78 @@ For 32-bit we have the following conventions - kernel is built with · 96 unmodified lines | |||
| 97 | 97 | ||
| 98 | #define SIZEOF_PTREGS 21*8 | 98 | #define SIZEOF_PTREGS 21*8 |
| 99 | 99 | ||
| 100 | .macro ALLOC_PT_GPREGS_ON_STACK | 100 | .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax save_ret=0 |
| 101 | addq $-(15*8), %rsp | 101 | /* |
| 102 | .endm | 102 | * Push registers and sanitize registers of values that a |
| 103 | 103 | * speculation attack might otherwise want to exploit. The | |
| 104 | .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8910=1 r11=1 | 104 | * lower registers are likely clobbered well before they |
| 105 | .if \r11 | 105 | * could be put to use in a speculative execution gadget. |
| 106 | movq %r11, 6*8+\offset(%rsp) | 106 | * Interleave XOR with PUSH for better uop scheduling: |
| 107 | .endif | 107 | */ |
| 108 | .if \r8910 | 108 | .if \save_ret |
| 109 | movq %r10, 7*8+\offset(%rsp) | 109 | pushq %rsi /* pt_regs->si */ |
| 110 | movq %r9, 8*8+\offset(%rsp) | 110 | movq 8(%rsp), %rsi /* temporarily store the return address in %rsi */ |
| 111 | movq %r8, 9*8+\offset(%rsp) | 111 | movq %rdi, 8(%rsp) /* pt_regs->di (overwriting original return address) */ |
| 112 | .endif | 112 | .else |
| 113 | .if \rax | 113 | pushq %rdi /* pt_regs->di */ |
| 114 | movq %rax, 10*8+\offset(%rsp) | 114 | pushq %rsi /* pt_regs->si */ |
| 115 | .endif | 115 | .endif |
| 116 | .if \rcx | 116 | pushq \rdx /* pt_regs->dx */ |
| 117 | movq %rcx, 11*8+\offset(%rsp) | 117 | pushq %rcx /* pt_regs->cx */ |
| 118 | pushq \rax /* pt_regs->ax */ | ||
| 119 | pushq %r8 /* pt_regs->r8 */ | ||
| 120 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 121 | pushq %r9 /* pt_regs->r9 */ | ||
| 122 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 123 | pushq %r10 /* pt_regs->r10 */ | ||
| 124 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 125 | pushq %r11 /* pt_regs->r11 */ | ||
| 126 | xorl %r11d, %r11d /* nospec r11*/ | ||
| 127 | pushq %rbx /* pt_regs->rbx */ | ||
| 128 | xorl %ebx, %ebx /* nospec rbx*/ | ||
| 129 | pushq %rbp /* pt_regs->rbp */ | ||
| 130 | xorl %ebp, %ebp /* nospec rbp*/ | ||
| 131 | pushq %r12 /* pt_regs->r12 */ | ||
| 132 | xorl %r12d, %r12d /* nospec r12*/ | ||
| 133 | pushq %r13 /* pt_regs->r13 */ | ||
| 134 | xorl %r13d, %r13d /* nospec r13*/ | ||
| 135 | pushq %r14 /* pt_regs->r14 */ | ||
| 136 | xorl %r14d, %r14d /* nospec r14*/ | ||
| 137 | pushq %r15 /* pt_regs->r15 */ | ||
| 138 | xorl %r15d, %r15d /* nospec r15*/ | ||
| 139 | UNWIND_HINT_REGS | ||
| 140 | .if \save_ret | ||
| 141 | pushq %rsi /* return address on top of stack */ | ||
| 118 | .endif | 142 | .endif |
| 119 | movq %rdx, 12*8+\offset(%rsp) | 143 | .endm |
| 120 | movq %rsi, 13*8+\offset(%rsp) | 144 | |
| 121 | movq %rdi, 14*8+\offset(%rsp) | 145 | .macro POP_REGS pop_rdi=1 skip_r11rcx=0 |
| 122 | UNWIND_HINT_REGS offset=\offset extra=0 | ||
| 123 | .endm | ||
| 124 | .macro SAVE_C_REGS offset=0 | ||
| 125 | SAVE_C_REGS_HELPER \offset, 1, 1, 1, 1 | ||
| 126 | .endm | ||
| 127 | .macro SAVE_C_REGS_EXCEPT_RAX_RCX offset=0 | ||
| 128 | SAVE_C_REGS_HELPER \offset, 0, 0, 1, 1 | ||
| 129 | .endm | ||
| 130 | .macro SAVE_C_REGS_EXCEPT_R891011 | ||
| 131 | SAVE_C_REGS_HELPER 0, 1, 1, 0, 0 | ||
| 132 | .endm | ||
| 133 | .macro SAVE_C_REGS_EXCEPT_RCX_R891011 | ||
| 134 | SAVE_C_REGS_HELPER 0, 1, 0, 0, 0 | ||
| 135 | .endm | ||
| 136 | .macro SAVE_C_REGS_EXCEPT_RAX_RCX_R11 | ||
| 137 | SAVE_C_REGS_HELPER 0, 0, 0, 1, 0 | ||
| 138 | .endm | ||
| 139 | |||
| 140 | .macro SAVE_EXTRA_REGS offset=0 | ||
| 141 | movq %r15, 0*8+\offset(%rsp) | ||
| 142 | movq %r14, 1*8+\offset(%rsp) | ||
| 143 | movq %r13, 2*8+\offset(%rsp) | ||
| 144 | movq %r12, 3*8+\offset(%rsp) | ||
| 145 | movq %rbp, 4*8+\offset(%rsp) | ||
| 146 | movq %rbx, 5*8+\offset(%rsp) | ||
| 147 | UNWIND_HINT_REGS offset=\offset | ||
| 148 | .endm | ||
| 149 | |||
| 150 | .macro POP_EXTRA_REGS | ||
| 151 | popq %r15 | 146 | popq %r15 |
| 152 | popq %r14 | 147 | popq %r14 |
| 153 | popq %r13 | 148 | popq %r13 |
| 154 | popq %r12 | 149 | popq %r12 |
| 155 | popq %rbp | 150 | popq %rbp |
| 156 | popq %rbx | 151 | popq %rbx |
| 157 | .endm | 152 | .if \skip_r11rcx |
| 158 | 153 | popq %rsi | |
| 159 | .macro POP_C_REGS | 154 | .else |
| 160 | popq %r11 | 155 | popq %r11 |
| 156 | .endif | ||
| 161 | popq %r10 | 157 | popq %r10 |
| 162 | popq %r9 | 158 | popq %r9 |
| 163 | popq %r8 | 159 | popq %r8 |
| 164 | popq %rax | 160 | popq %rax |
| 161 | .if \skip_r11rcx | ||
| 162 | popq %rsi | ||
| 163 | .else | ||
| 165 | popq %rcx | 164 | popq %rcx |
| 165 | .endif | ||
| 166 | popq %rdx | 166 | popq %rdx |
| 167 | popq %rsi | 167 | popq %rsi |
| 168 | .if \pop_rdi | ||
| 168 | popq %rdi | 169 | popq %rdi |
| 169 | .endm | 170 | .endif |
| 170 | 171 | .endm | |
| 171 | .macro icebp | ||
| 172 | .byte 0xf1 | ||
| 173 | .endm | ||
| 174 | 172 | ||
| 175 | /* | 173 | /* |
| 176 | * This is a sneaky trick to help the unwinder find pt_regs on the stack. The | 174 | * This is a sneaky trick to help the unwinder find pt_regs on the stack. The |
| @@ -178,17 +176,12 @@ For 32-bit we have the following conventions - kernel is built with · 1 unmodified line | |||
| 178 | * is just setting the LSB, which makes it an invalid stack address and is also | 176 | * is just setting the LSB, which makes it an invalid stack address and is also |
| 179 | * a signal to the unwinder that it's a pt_regs pointer in disguise. | 177 | * a signal to the unwinder that it's a pt_regs pointer in disguise. |
| 180 | * | 178 | * |
| 181 | * NOTE: This macro must be used *after* SAVE_EXTRA_REGS because it corrupts | 179 | * NOTE: This macro must be used *after* PUSH_AND_CLEAR_REGS because it corrupts |
| 182 | * the original rbp. | 180 | * the original rbp. |
| 183 | */ | 181 | */ |
| 184 | .macro ENCODE_FRAME_POINTER ptregs_offset=0 | 182 | .macro ENCODE_FRAME_POINTER ptregs_offset=0 |
| 185 | #ifdef CONFIG_FRAME_POINTER | 183 | #ifdef CONFIG_FRAME_POINTER |
| 186 | .if \ptregs_offset | 184 | leaq 1+\ptregs_offset(%rsp), %rbp |
| 187 | leaq \ptregs_offset(%rsp), %rbp | ||
| 188 | .else | ||
| 189 | mov %rsp, %rbp | ||
| 190 | .endif | ||
| 191 | orq $0x1, %rbp | ||
| 192 | #endif | 185 | #endif |
| 193 | .endm | 186 | .endm |
| 194 | 187 | ||
| @@ -252,8 +252,7 @@ ENTRY(__switch_to_asm) · 251 unmodified lines | |||
| 252 | * exist, overwrite the RSB with entries which capture | 252 | * exist, overwrite the RSB with entries which capture |
| 253 | * speculative execution to prevent attack. | 253 | * speculative execution to prevent attack. |
| 254 | */ | 254 | */ |
| 255 | /* Clobbers %ebx */ | 255 | FILL_RETURN_BUFFER %ebx, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW |
| 256 | FILL_RETURN_BUFFER RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW | ||
| 257 | #endif | 256 | #endif |
| 258 | 257 | ||
| 259 | /* restore callee-saved registers */ | 258 | /* restore callee-saved registers */ |
| @@ -55,7 +55,7 @@ END(native_usergs_sysret64) · 54 unmodified lines | |||
| 55 | 55 | ||
| 56 | .macro TRACE_IRQS_FLAGS flags:req | 56 | .macro TRACE_IRQS_FLAGS flags:req |
| 57 | #ifdef CONFIG_TRACE_IRQFLAGS | 57 | #ifdef CONFIG_TRACE_IRQFLAGS |
| 58 | bt $9, \flags /* interrupts off? */ | 58 | btl $9, \flags /* interrupts off? */ |
| 59 | jnc 1f | 59 | jnc 1f |
| 60 | TRACE_IRQS_ON | 60 | TRACE_IRQS_ON |
| 61 | 1: | 61 | 1: |
| @@ -213,7 +213,7 @@ ENTRY(entry_SYSCALL_64) · 151 unmodified lines | |||
| 213 | 213 | ||
| 214 | swapgs | 214 | swapgs |
| 215 | /* | 215 | /* |
| 216 | * This path is not taken when PAGE_TABLE_ISOLATION is disabled so it | 216 | * This path is only taken when PAGE_TABLE_ISOLATION is disabled so it |
| 217 | * is not required to switch CR3. | 217 | * is not required to switch CR3. |
| 218 | */ | 218 | */ |
| 219 | movq %rsp, PER_CPU_VAR(rsp_scratch) | 219 | movq %rsp, PER_CPU_VAR(rsp_scratch) |
| @@ -227,22 +227,8 @@ ENTRY(entry_SYSCALL_64) · 7 unmodified lines | |||
| 227 | pushq %rcx /* pt_regs->ip */ | 227 | pushq %rcx /* pt_regs->ip */ |
| 228 | GLOBAL(entry_SYSCALL_64_after_hwframe) | 228 | GLOBAL(entry_SYSCALL_64_after_hwframe) |
| 229 | pushq %rax /* pt_regs->orig_ax */ | 229 | pushq %rax /* pt_regs->orig_ax */ |
| 230 | pushq %rdi /* pt_regs->di */ | 230 | |
| 231 | pushq %rsi /* pt_regs->si */ | 231 | PUSH_AND_CLEAR_REGS rax=$-ENOSYS |
| 232 | pushq %rdx /* pt_regs->dx */ | ||
| 233 | pushq %rcx /* pt_regs->cx */ | ||
| 234 | pushq $-ENOSYS /* pt_regs->ax */ | ||
| 235 | pushq %r8 /* pt_regs->r8 */ | ||
| 236 | pushq %r9 /* pt_regs->r9 */ | ||
| 237 | pushq %r10 /* pt_regs->r10 */ | ||
| 238 | pushq %r11 /* pt_regs->r11 */ | ||
| 239 | pushq %rbx /* pt_regs->rbx */ | ||
| 240 | pushq %rbp /* pt_regs->rbp */ | ||
| 241 | pushq %r12 /* pt_regs->r12 */ | ||
| 242 | pushq %r13 /* pt_regs->r13 */ | ||
| 243 | pushq %r14 /* pt_regs->r14 */ | ||
| 244 | pushq %r15 /* pt_regs->r15 */ | ||
| 245 | UNWIND_HINT_REGS | ||
| 246 | 232 | ||
| 247 | TRACE_IRQS_OFF | 233 | TRACE_IRQS_OFF |
| 248 | 234 | ||
| @@ -321,15 +307,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe) · 72 unmodified lines | |||
| 321 | syscall_return_via_sysret: | 307 | syscall_return_via_sysret: |
| 322 | /* rcx and r11 are already restored (see code above) */ | 308 | /* rcx and r11 are already restored (see code above) */ |
| 323 | UNWIND_HINT_EMPTY | 309 | UNWIND_HINT_EMPTY |
| 324 | POP_EXTRA_REGS | 310 | POP_REGS pop_rdi=0 skip_r11rcx=1 |
| 325 | popq %rsi /* skip r11 */ | ||
| 326 | popq %r10 | ||
| 327 | popq %r9 | ||
| 328 | popq %r8 | ||
| 329 | popq %rax | ||
| 330 | popq %rsi /* skip rcx */ | ||
| 331 | popq %rdx | ||
| 332 | popq %rsi | ||
| 333 | 311 | ||
| 334 | /* | 312 | /* |
| 335 | * Now all regs are restored except RSP and RDI. | 313 | * Now all regs are restored except RSP and RDI. |
| @@ -386,8 +364,7 @@ ENTRY(__switch_to_asm) · 50 unmodified lines | |||
| 386 | * exist, overwrite the RSB with entries which capture | 364 | * exist, overwrite the RSB with entries which capture |
| 387 | * speculative execution to prevent attack. | 365 | * speculative execution to prevent attack. |
| 388 | */ | 366 | */ |
| 389 | /* Clobbers %rbx */ | 367 | FILL_RETURN_BUFFER %r12, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW |
| 390 | FILL_RETURN_BUFFER RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW | ||
| 391 | #endif | 368 | #endif |
| 392 | 369 | ||
| 393 | /* restore callee-saved registers */ | 370 | /* restore callee-saved registers */ |
| @@ -471,9 +448,19 @@ END(irq_entries_start) · 77 unmodified lines | |||
| 471 | * | 448 | * |
| 472 | * The invariant is that, if irq_count != -1, then the IRQ stack is in use. | 449 | * The invariant is that, if irq_count != -1, then the IRQ stack is in use. |
| 473 | */ | 450 | */ |
| 474 | .macro ENTER_IRQ_STACK regs=1 old_rsp | 451 | .macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0 |
| 475 | DEBUG_ENTRY_ASSERT_IRQS_OFF | 452 | DEBUG_ENTRY_ASSERT_IRQS_OFF |
| 453 | |||
| 454 | .if \save_ret | ||
| 455 | /* | ||
| 456 | * If save_ret is set, the original stack contains one additional | ||
| 457 | * entry -- the return address. Therefore, move the address one | ||
| 458 | * entry below %rsp to \old_rsp. | ||
| 459 | */ | ||
| 460 | leaq 8(%rsp), \old_rsp | ||
| 461 | .else | ||
| 476 | movq %rsp, \old_rsp | 462 | movq %rsp, \old_rsp |
| 463 | .endif | ||
| 477 | 464 | ||
| 478 | .if \regs | 465 | .if \regs |
| 479 | UNWIND_HINT_REGS base=\old_rsp | 466 | UNWIND_HINT_REGS base=\old_rsp |
| @@ -519,6 +506,15 @@ END(irq_entries_start) · 39 unmodified lines | |||
| 519 | .if \regs | 506 | .if \regs |
| 520 | UNWIND_HINT_REGS indirect=1 | 507 | UNWIND_HINT_REGS indirect=1 |
| 521 | .endif | 508 | .endif |
| 509 | |||
| 510 | .if \save_ret | ||
| 511 | /* | ||
| 512 | * Push the return address to the stack. This return address can | ||
| 513 | * be found at the "real" original RSP, which was offset by 8 at | ||
| 514 | * the beginning of this macro. | ||
| 515 | */ | ||
| 516 | pushq -8(\old_rsp) | ||
| 517 | .endif | ||
| 522 | .endm | 518 | .endm |
| 523 | 519 | ||
| 524 | /* | 520 | /* |
| @@ -542,29 +538,65 @@ END(irq_entries_start) · 17 unmodified lines | |||
| 542 | .endm | 538 | .endm |
| 543 | 539 | ||
| 544 | /* | 540 | /* |
| 545 | * Interrupt entry/exit. | 541 | * Interrupt entry helper function. |
| 546 | * | ||
| 547 | * Interrupt entry points save only callee clobbered registers in fast path. | ||
| 548 | * | 542 | * |
| 549 | * Entry runs with interrupts off. | 543 | * Entry runs with interrupts off. Stack layout at entry: |
| 544 | * +----------------------------------------------------+ | ||
| 545 | * | regs->ss | | ||
| 546 | * | regs->rsp | | ||
| 547 | * | regs->eflags | | ||
| 548 | * | regs->cs | | ||
| 549 | * | regs->ip | | ||
| 550 | * +----------------------------------------------------+ | ||
| 551 | * | regs->orig_ax = ~(interrupt number) | | ||
| 552 | * +----------------------------------------------------+ | ||
| 553 | * | return address | | ||
| 554 | * +----------------------------------------------------+ | ||
| 550 | */ | 555 | */ |
| 551 | 556 | ENTRY(interrupt_entry) | |
| 552 | /* 0(%rsp): ~(interrupt number) */ | 557 | UNWIND_HINT_FUNC |
| 553 | .macro interrupt func | 558 | ASM_CLAC |
| 554 | cld | 559 | cld |
| 555 | 560 | ||
| 556 | testb $3, CS-ORIG_RAX(%rsp) | 561 | testb $3, CS-ORIG_RAX+8(%rsp) |
| 557 | jz 1f | 562 | jz 1f |
| 558 | SWAPGS | 563 | SWAPGS |
| 559 | call switch_to_thread_stack | 564 | |
| 565 | /* | ||
| 566 | * Switch to the thread stack. The IRET frame and orig_ax are | ||
| 567 | * on the stack, as well as the return address. RDI..R12 are | ||
| 568 | * not (yet) on the stack and space has not (yet) been | ||
| 569 | * allocated for them. | ||
| 570 | */ | ||
| 571 | pushq %rdi | ||
| 572 | |||
| 573 | /* Need to switch before accessing the thread stack. */ | ||
| 574 | SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi | ||
| 575 | movq %rsp, %rdi | ||
| 576 | movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp | ||
| 577 | |||
| 578 | /* | ||
| 579 | * We have RDI, return address, and orig_ax on the stack on | ||
| 580 | * top of the IRET frame. That means offset=24 | ||
| 581 | */ | ||
| 582 | UNWIND_HINT_IRET_REGS base=%rdi offset=24 | ||
| 583 | |||
| 584 | pushq 7*8(%rdi) /* regs->ss */ | ||
| 585 | pushq 6*8(%rdi) /* regs->rsp */ | ||
| 586 | pushq 5*8(%rdi) /* regs->eflags */ | ||
| 587 | pushq 4*8(%rdi) /* regs->cs */ | ||
| 588 | pushq 3*8(%rdi) /* regs->ip */ | ||
| 589 | pushq 2*8(%rdi) /* regs->orig_ax */ | ||
| 590 | pushq 8(%rdi) /* return address */ | ||
| 591 | UNWIND_HINT_FUNC | ||
| 592 | |||
| 593 | movq (%rdi), %rdi | ||
| 560 | 1: | 594 | 1: |
| 561 | 595 | ||
| 562 | ALLOC_PT_GPREGS_ON_STACK | 596 | PUSH_AND_CLEAR_REGS save_ret=1 |
| 563 | SAVE_C_REGS | 597 | ENCODE_FRAME_POINTER 8 |
| 564 | SAVE_EXTRA_REGS | ||
| 565 | ENCODE_FRAME_POINTER | ||
| 566 | 598 | ||
| 567 | testb $3, CS(%rsp) | 599 | testb $3, CS+8(%rsp) |
| 568 | jz 1f | 600 | jz 1f |
| 569 | 601 | ||
| 570 | /* | 602 | /* |
| @@ -572,7 +604,7 @@ END(irq_entries_start) · 1 unmodified line | |||
| 572 | * | 604 | * |
| 573 | * We need to tell lockdep that IRQs are off. We can't do this until | 605 | * We need to tell lockdep that IRQs are off. We can't do this until |
| 574 | * we fix gsbase, and we should do it before enter_from_user_mode | 606 | * we fix gsbase, and we should do it before enter_from_user_mode |
| 575 | * (which can take locks). Since TRACE_IRQS_OFF idempotent, | 607 | * (which can take locks). Since TRACE_IRQS_OFF is idempotent, |
| 576 | * the simplest way to handle it is to just call it twice if | 608 | * the simplest way to handle it is to just call it twice if |
| 577 | * we enter from user mode. There's no reason to optimize this since | 609 | * we enter from user mode. There's no reason to optimize this since |
| 578 | * TRACE_IRQS_OFF is a no-op if lockdep is off. | 610 | * TRACE_IRQS_OFF is a no-op if lockdep is off. |
| @@ -582,12 +614,15 @@ END(irq_entries_start) · 3 unmodified lines | |||
| 582 | CALL_enter_from_user_mode | 614 | CALL_enter_from_user_mode |
| 583 | 615 | ||
| 584 | 1: | 616 | 1: |
| 585 | ENTER_IRQ_STACK old_rsp=%rdi | 617 | ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 |
| 586 | /* We entered an interrupt context - irqs are off: */ | 618 | /* We entered an interrupt context - irqs are off: */ |
| 587 | TRACE_IRQS_OFF | 619 | TRACE_IRQS_OFF |
| 588 | 620 | ||
| 589 | call \func /* rdi points to pt_regs */ | 621 | ret |
| 590 | .endm | 622 | END(interrupt_entry) |
| 623 | |||
| 624 | |||
| 625 | /* Interrupt entry/exit. */ | ||
| 591 | 626 | ||
| 592 | /* | 627 | /* |
| 593 | * The interrupt stubs push (~vector+0x80) onto the stack and | 628 | * The interrupt stubs push (~vector+0x80) onto the stack and |
| @@ -595,9 +630,10 @@ END(irq_entries_start) · 1 unmodified line | |||
| 595 | */ | 630 | */ |
| 596 | .p2align CONFIG_X86_L1_CACHE_SHIFT | 631 | .p2align CONFIG_X86_L1_CACHE_SHIFT |
| 597 | common_interrupt: | 632 | common_interrupt: |
| 598 | ASM_CLAC | ||
| 599 | addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ | 633 | addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ |
| 600 | interrupt do_IRQ | 634 | call interrupt_entry |
| 635 | UNWIND_HINT_REGS indirect=1 | ||
| 636 | call do_IRQ /* rdi points to pt_regs */ | ||
| 601 | /* 0(%rsp): old RSP */ | 637 | /* 0(%rsp): old RSP */ |
| 602 | ret_from_intr: | 638 | ret_from_intr: |
| 603 | DISABLE_INTERRUPTS(CLBR_ANY) | 639 | DISABLE_INTERRUPTS(CLBR_ANY) |
| @@ -622,15 +658,7 @@ GLOBAL(swapgs_restore_regs_and_return_to_usermode) · 18 unmodified lines | |||
| 622 | ud2 | 658 | ud2 |
| 623 | 1: | 659 | 1: |
| 624 | #endif | 660 | #endif |
| 625 | POP_EXTRA_REGS | 661 | POP_REGS pop_rdi=0 |
| 626 | popq %r11 | ||
| 627 | popq %r10 | ||
| 628 | popq %r9 | ||
| 629 | popq %r8 | ||
| 630 | popq %rax | ||
| 631 | popq %rcx | ||
| 632 | popq %rdx | ||
| 633 | popq %rsi | ||
| 634 | 662 | ||
| 635 | /* | 663 | /* |
| 636 | * The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS. | 664 | * The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS. |
| @@ -688,8 +716,7 @@ GLOBAL(restore_regs_and_return_to_kernel) · 51 unmodified lines | |||
| 688 | ud2 | 716 | ud2 |
| 689 | 1: | 717 | 1: |
| 690 | #endif | 718 | #endif |
| 691 | POP_EXTRA_REGS | 719 | POP_REGS |
| 692 | POP_C_REGS | ||
| 693 | addq $8, %rsp /* skip regs->orig_ax */ | 720 | addq $8, %rsp /* skip regs->orig_ax */ |
| 694 | /* | 721 | /* |
| 695 | * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization | 722 | * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization |
| @@ -799,10 +826,11 @@ END(common_interrupt) · 103 unmodified lines | |||
| 799 | .macro apicinterrupt3 num sym do_sym | 826 | .macro apicinterrupt3 num sym do_sym |
| 800 | ENTRY(\sym) | 827 | ENTRY(\sym) |
| 801 | UNWIND_HINT_IRET_REGS | 828 | UNWIND_HINT_IRET_REGS |
| 802 | ASM_CLAC | ||
| 803 | pushq $~(\num) | 829 | pushq $~(\num) |
| 804 | .Lcommon_\sym: | 830 | .Lcommon_\sym: |
| 805 | interrupt \do_sym | 831 | call interrupt_entry |
| 832 | UNWIND_HINT_REGS indirect=1 | ||
| 833 | call \do_sym /* rdi points to pt_regs */ | ||
| 806 | jmp ret_from_intr | 834 | jmp ret_from_intr |
| 807 | END(\sym) | 835 | END(\sym) |
| 808 | .endm | 836 | .endm |
| @@ -865,34 +893,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt · 56 unmodified lines | |||
| 865 | */ | 893 | */ |
| 866 | #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) | 894 | #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + ((x) - 1) * 8) |
| 867 | 895 | ||
| 868 | /* | ||
| 869 | * Switch to the thread stack. This is called with the IRET frame and | ||
| 870 | * orig_ax on the stack. (That is, RDI..R12 are not on the stack and | ||
| 871 | * space has not been allocated for them.) | ||
| 872 | */ | ||
| 873 | ENTRY(switch_to_thread_stack) | ||
| 874 | UNWIND_HINT_FUNC | ||
| 875 | |||
| 876 | pushq %rdi | ||
| 877 | /* Need to switch before accessing the thread stack. */ | ||
| 878 | SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi | ||
| 879 | movq %rsp, %rdi | ||
| 880 | movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp | ||
| 881 | UNWIND_HINT sp_offset=16 sp_reg=ORC_REG_DI | ||
| 882 | |||
| 883 | pushq 7*8(%rdi) /* regs->ss */ | ||
| 884 | pushq 6*8(%rdi) /* regs->rsp */ | ||
| 885 | pushq 5*8(%rdi) /* regs->eflags */ | ||
| 886 | pushq 4*8(%rdi) /* regs->cs */ | ||
| 887 | pushq 3*8(%rdi) /* regs->ip */ | ||
| 888 | pushq 2*8(%rdi) /* regs->orig_ax */ | ||
| 889 | pushq 8(%rdi) /* return address */ | ||
| 890 | UNWIND_HINT_FUNC | ||
| 891 | |||
| 892 | movq (%rdi), %rdi | ||
| 893 | ret | ||
| 894 | END(switch_to_thread_stack) | ||
| 895 | |||
| 896 | .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 | 896 | .macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 |
| 897 | ENTRY(\sym) | 897 | ENTRY(\sym) |
| 898 | UNWIND_HINT_IRET_REGS offset=\has_error_code*8 | 898 | UNWIND_HINT_IRET_REGS offset=\has_error_code*8 |
| @@ -908,10 +908,8 @@ ENTRY(\sym) · 9 unmodified lines | |||
| 908 | pushq $-1 /* ORIG_RAX: no syscall to restart */ | 908 | pushq $-1 /* ORIG_RAX: no syscall to restart */ |
| 909 | .endif | 909 | .endif |
| 910 | 910 | ||
| 911 | ALLOC_PT_GPREGS_ON_STACK | ||
| 912 | |||
| 913 | .if \paranoid < 2 | 911 | .if \paranoid < 2 |
| 914 | testb $3, CS(%rsp) /* If coming from userspace, switch stacks */ | 912 | testb $3, CS-ORIG_RAX(%rsp) /* If coming from userspace, switch stacks */ |
| 915 | jnz .Lfrom_usermode_switch_stack_\@ | 913 | jnz .Lfrom_usermode_switch_stack_\@ |
| 916 | .endif | 914 | .endif |
| 917 | 915 | ||
| @@ -1121,9 +1119,7 @@ ENTRY(xen_failsafe_callback) · 203 unmodified lines | |||
| 1121 | addq $0x30, %rsp | 1119 | addq $0x30, %rsp |
| 1122 | UNWIND_HINT_IRET_REGS | 1120 | UNWIND_HINT_IRET_REGS |
| 1123 | pushq $-1 /* orig_ax = -1 => not a system call */ | 1121 | pushq $-1 /* orig_ax = -1 => not a system call */ |
| 1124 | ALLOC_PT_GPREGS_ON_STACK | 1122 | PUSH_AND_CLEAR_REGS |
| 1125 | SAVE_C_REGS | ||
| 1126 | SAVE_EXTRA_REGS | ||
| 1127 | ENCODE_FRAME_POINTER | 1123 | ENCODE_FRAME_POINTER |
| 1128 | jmp error_exit | 1124 | jmp error_exit |
| 1129 | END(xen_failsafe_callback) | 1125 | END(xen_failsafe_callback) |
| @@ -1170,8 +1166,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1 · 40 unmodified lines | |||
| 1170 | ENTRY(paranoid_entry) | 1166 | ENTRY(paranoid_entry) |
| 1171 | UNWIND_HINT_FUNC | 1167 | UNWIND_HINT_FUNC |
| 1172 | cld | 1168 | cld |
| 1173 | SAVE_C_REGS 8 | 1169 | PUSH_AND_CLEAR_REGS save_ret=1 |
| 1174 | SAVE_EXTRA_REGS 8 | ||
| 1175 | ENCODE_FRAME_POINTER 8 | 1170 | ENCODE_FRAME_POINTER 8 |
| 1176 | movl $1, %ebx | 1171 | movl $1, %ebx |
| 1177 | movl $MSR_GS_BASE, %ecx | 1172 | movl $MSR_GS_BASE, %ecx |
| @@ -1211,21 +1206,20 @@ ENTRY(paranoid_exit) · 33 unmodified lines | |||
| 1211 | jmp .Lparanoid_exit_restore | 1206 | jmp .Lparanoid_exit_restore |
| 1212 | .Lparanoid_exit_no_swapgs: | 1207 | .Lparanoid_exit_no_swapgs: |
| 1213 | TRACE_IRQS_IRETQ_DEBUG | 1208 | TRACE_IRQS_IRETQ_DEBUG |
| 1209 | RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 | ||
| 1214 | .Lparanoid_exit_restore: | 1210 | .Lparanoid_exit_restore: |
| 1215 | jmp restore_regs_and_return_to_kernel | 1211 | jmp restore_regs_and_return_to_kernel |
| 1216 | END(paranoid_exit) | 1212 | END(paranoid_exit) |
| 1217 | 1213 | ||
| 1218 | /* | 1214 | /* |
| 1219 | * Save all registers in pt_regs, and switch gs if needed. | 1215 | * Save all registers in pt_regs, and switch GS if needed. |
| 1220 | * Return: EBX=0: came from user mode; EBX=1: otherwise | 1216 | * Return: EBX=0: came from user mode; EBX=1: otherwise |
| 1221 | */ | 1217 | */ |
| 1222 | ENTRY(error_entry) | 1218 | ENTRY(error_entry) |
| 1223 | UNWIND_HINT_FUNC | 1219 | UNWIND_HINT_FUNC |
| 1224 | cld | 1220 | cld |
| 1225 | SAVE_C_REGS 8 | 1221 | PUSH_AND_CLEAR_REGS save_ret=1 |
| 1226 | SAVE_EXTRA_REGS 8 | ||
| 1227 | ENCODE_FRAME_POINTER 8 | 1222 | ENCODE_FRAME_POINTER 8 |
| 1228 | xorl %ebx, %ebx | ||
| 1229 | testb $3, CS+8(%rsp) | 1223 | testb $3, CS+8(%rsp) |
| 1230 | jz .Lerror_kernelspace | 1224 | jz .Lerror_kernelspace |
| 1231 | 1225 | ||
| @@ -1406,22 +1400,7 @@ ENTRY(nmi) · 174 unmodified lines | |||
| 1406 | pushq 1*8(%rdx) /* pt_regs->rip */ | 1400 | pushq 1*8(%rdx) /* pt_regs->rip */ |
| 1407 | UNWIND_HINT_IRET_REGS | 1401 | UNWIND_HINT_IRET_REGS |
| 1408 | pushq $-1 /* pt_regs->orig_ax */ | 1402 | pushq $-1 /* pt_regs->orig_ax */ |
| 1409 | pushq %rdi /* pt_regs->di */ | 1403 | PUSH_AND_CLEAR_REGS rdx=(%rdx) |
| 1410 | pushq %rsi /* pt_regs->si */ | ||
| 1411 | pushq (%rdx) /* pt_regs->dx */ | ||
| 1412 | pushq %rcx /* pt_regs->cx */ | ||
| 1413 | pushq %rax /* pt_regs->ax */ | ||
| 1414 | pushq %r8 /* pt_regs->r8 */ | ||
| 1415 | pushq %r9 /* pt_regs->r9 */ | ||
| 1416 | pushq %r10 /* pt_regs->r10 */ | ||
| 1417 | pushq %r11 /* pt_regs->r11 */ | ||
| 1418 | pushq %rbx /* pt_regs->rbx */ | ||
| 1419 | pushq %rbp /* pt_regs->rbp */ | ||
| 1420 | pushq %r12 /* pt_regs->r12 */ | ||
| 1421 | pushq %r13 /* pt_regs->r13 */ | ||
| 1422 | pushq %r14 /* pt_regs->r14 */ | ||
| 1423 | pushq %r15 /* pt_regs->r15 */ | ||
| 1424 | UNWIND_HINT_REGS | ||
| 1425 | ENCODE_FRAME_POINTER | 1404 | ENCODE_FRAME_POINTER |
| 1426 | 1405 | ||
| 1427 | /* | 1406 | /* |
| @@ -1631,7 +1610,6 @@ end_repeat_nmi: · 203 unmodified lines | |||
| 1631 | * frame to point back to repeat_nmi. | 1610 | * frame to point back to repeat_nmi. |
| 1632 | */ | 1611 | */ |
| 1633 | pushq $-1 /* ORIG_RAX: no syscall to restart */ | 1612 | pushq $-1 /* ORIG_RAX: no syscall to restart */ |
| 1634 | ALLOC_PT_GPREGS_ON_STACK | ||
| 1635 | 1613 | ||
| 1636 | /* | 1614 | /* |
| 1637 | * Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit | 1615 | * Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit |
| @@ -1655,8 +1633,7 @@ end_repeat_nmi: · 17 unmodified lines | |||
| 1655 | nmi_swapgs: | 1633 | nmi_swapgs: |
| 1656 | SWAPGS_UNSAFE_STACK | 1634 | SWAPGS_UNSAFE_STACK |
| 1657 | nmi_restore: | 1635 | nmi_restore: |
| 1658 | POP_EXTRA_REGS | 1636 | POP_REGS |
| 1659 | POP_C_REGS | ||
| 1660 | 1637 | ||
| 1661 | /* | 1638 | /* |
| 1662 | * Skip orig_ax and the "outermost" frame to point RSP at the "iret" | 1639 | * Skip orig_ax and the "outermost" frame to point RSP at the "iret" |
| @@ -85,15 +85,25 @@ ENTRY(entry_SYSENTER_compat) · 84 unmodified lines | |||
| 85 | pushq %rcx /* pt_regs->cx */ | 85 | pushq %rcx /* pt_regs->cx */ |
| 86 | pushq $-ENOSYS /* pt_regs->ax */ | 86 | pushq $-ENOSYS /* pt_regs->ax */ |
| 87 | pushq $0 /* pt_regs->r8 = 0 */ | 87 | pushq $0 /* pt_regs->r8 = 0 */ |
| 88 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 88 | pushq $0 /* pt_regs->r9 = 0 */ | 89 | pushq $0 /* pt_regs->r9 = 0 */ |
| 90 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 89 | pushq $0 /* pt_regs->r10 = 0 */ | 91 | pushq $0 /* pt_regs->r10 = 0 */ |
| 92 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 90 | pushq $0 /* pt_regs->r11 = 0 */ | 93 | pushq $0 /* pt_regs->r11 = 0 */ |
| 94 | xorl %r11d, %r11d /* nospec r11 */ | ||
| 91 | pushq %rbx /* pt_regs->rbx */ | 95 | pushq %rbx /* pt_regs->rbx */ |
| 96 | xorl %ebx, %ebx /* nospec rbx */ | ||
| 92 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ | 97 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ |
| 98 | xorl %ebp, %ebp /* nospec rbp */ | ||
| 93 | pushq $0 /* pt_regs->r12 = 0 */ | 99 | pushq $0 /* pt_regs->r12 = 0 */ |
| 100 | xorl %r12d, %r12d /* nospec r12 */ | ||
| 94 | pushq $0 /* pt_regs->r13 = 0 */ | 101 | pushq $0 /* pt_regs->r13 = 0 */ |
| 102 | xorl %r13d, %r13d /* nospec r13 */ | ||
| 95 | pushq $0 /* pt_regs->r14 = 0 */ | 103 | pushq $0 /* pt_regs->r14 = 0 */ |
| 104 | xorl %r14d, %r14d /* nospec r14 */ | ||
| 96 | pushq $0 /* pt_regs->r15 = 0 */ | 105 | pushq $0 /* pt_regs->r15 = 0 */ |
| 106 | xorl %r15d, %r15d /* nospec r15 */ | ||
| 97 | cld | 107 | cld |
| 98 | 108 | ||
| 99 | /* | 109 | /* |
| @@ -214,15 +224,25 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe) · 114 unmodified lines | |||
| 214 | pushq %rbp /* pt_regs->cx (stashed in bp) */ | 224 | pushq %rbp /* pt_regs->cx (stashed in bp) */ |
| 215 | pushq $-ENOSYS /* pt_regs->ax */ | 225 | pushq $-ENOSYS /* pt_regs->ax */ |
| 216 | pushq $0 /* pt_regs->r8 = 0 */ | 226 | pushq $0 /* pt_regs->r8 = 0 */ |
| 227 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 217 | pushq $0 /* pt_regs->r9 = 0 */ | 228 | pushq $0 /* pt_regs->r9 = 0 */ |
| 229 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 218 | pushq $0 /* pt_regs->r10 = 0 */ | 230 | pushq $0 /* pt_regs->r10 = 0 */ |
| 231 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 219 | pushq $0 /* pt_regs->r11 = 0 */ | 232 | pushq $0 /* pt_regs->r11 = 0 */ |
| 233 | xorl %r11d, %r11d /* nospec r11 */ | ||
| 220 | pushq %rbx /* pt_regs->rbx */ | 234 | pushq %rbx /* pt_regs->rbx */ |
| 235 | xorl %ebx, %ebx /* nospec rbx */ | ||
| 221 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ | 236 | pushq %rbp /* pt_regs->rbp (will be overwritten) */ |
| 237 | xorl %ebp, %ebp /* nospec rbp */ | ||
| 222 | pushq $0 /* pt_regs->r12 = 0 */ | 238 | pushq $0 /* pt_regs->r12 = 0 */ |
| 239 | xorl %r12d, %r12d /* nospec r12 */ | ||
| 223 | pushq $0 /* pt_regs->r13 = 0 */ | 240 | pushq $0 /* pt_regs->r13 = 0 */ |
| 241 | xorl %r13d, %r13d /* nospec r13 */ | ||
| 224 | pushq $0 /* pt_regs->r14 = 0 */ | 242 | pushq $0 /* pt_regs->r14 = 0 */ |
| 243 | xorl %r14d, %r14d /* nospec r14 */ | ||
| 225 | pushq $0 /* pt_regs->r15 = 0 */ | 244 | pushq $0 /* pt_regs->r15 = 0 */ |
| 245 | xorl %r15d, %r15d /* nospec r15 */ | ||
| 226 | 246 | ||
| 227 | /* | 247 | /* |
| 228 | * User mode is traced as though IRQs are on, and SYSENTER | 248 | * User mode is traced as though IRQs are on, and SYSENTER |
| @@ -278,9 +298,9 @@ sysret32_from_system_call: · 49 unmodified lines | |||
| 278 | */ | 298 | */ |
| 279 | SWITCH_TO_USER_CR3_NOSTACK scratch_reg=%r8 scratch_reg2=%r9 | 299 | SWITCH_TO_USER_CR3_NOSTACK scratch_reg=%r8 scratch_reg2=%r9 |
| 280 | 300 | ||
| 281 | xorq %r8, %r8 | 301 | xorl %r8d, %r8d |
| 282 | xorq %r9, %r9 | 302 | xorl %r9d, %r9d |
| 283 | xorq %r10, %r10 | 303 | xorl %r10d, %r10d |
| 284 | swapgs | 304 | swapgs |
| 285 | sysretl | 305 | sysretl |
| 286 | END(entry_SYSCALL_compat) | 306 | END(entry_SYSCALL_compat) |
| @@ -327,26 +347,47 @@ ENTRY(entry_INT80_compat) · 40 unmodified lines | |||
| 327 | */ | 347 | */ |
| 328 | movl %eax, %eax | 348 | movl %eax, %eax |
| 329 | 349 | ||
| 350 | /* switch to thread stack expects orig_ax and rdi to be pushed */ | ||
| 330 | pushq %rax /* pt_regs->orig_ax */ | 351 | pushq %rax /* pt_regs->orig_ax */ |
| 352 | pushq %rdi /* pt_regs->di */ | ||
| 331 | 353 | ||
| 332 | /* switch to thread stack expects orig_ax to be pushed */ | 354 | /* Need to switch before accessing the thread stack. */ |
| 333 | call switch_to_thread_stack | 355 | SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi |
| 356 | movq %rsp, %rdi | ||
| 357 | movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp | ||
| 334 | 358 | ||
| 335 | pushq %rdi /* pt_regs->di */ | 359 | pushq 6*8(%rdi) /* regs->ss */ |
| 360 | pushq 5*8(%rdi) /* regs->rsp */ | ||
| 361 | pushq 4*8(%rdi) /* regs->eflags */ | ||
| 362 | pushq 3*8(%rdi) /* regs->cs */ | ||
| 363 | pushq 2*8(%rdi) /* regs->ip */ | ||
| 364 | pushq 1*8(%rdi) /* regs->orig_ax */ | ||
| 365 | |||
| 366 | pushq (%rdi) /* pt_regs->di */ | ||
| 336 | pushq %rsi /* pt_regs->si */ | 367 | pushq %rsi /* pt_regs->si */ |
| 337 | pushq %rdx /* pt_regs->dx */ | 368 | pushq %rdx /* pt_regs->dx */ |
| 338 | pushq %rcx /* pt_regs->cx */ | 369 | pushq %rcx /* pt_regs->cx */ |
| 339 | pushq $-ENOSYS /* pt_regs->ax */ | 370 | pushq $-ENOSYS /* pt_regs->ax */ |
| 340 | pushq $0 /* pt_regs->r8 = 0 */ | 371 | pushq $0 /* pt_regs->r8 = 0 */ |
| 372 | xorl %r8d, %r8d /* nospec r8 */ | ||
| 341 | pushq $0 /* pt_regs->r9 = 0 */ | 373 | pushq $0 /* pt_regs->r9 = 0 */ |
| 374 | xorl %r9d, %r9d /* nospec r9 */ | ||
| 342 | pushq $0 /* pt_regs->r10 = 0 */ | 375 | pushq $0 /* pt_regs->r10 = 0 */ |
| 376 | xorl %r10d, %r10d /* nospec r10 */ | ||
| 343 | pushq $0 /* pt_regs->r11 = 0 */ | 377 | pushq $0 /* pt_regs->r11 = 0 */ |
| 378 | xorl %r11d, %r11d /* nospec r11 */ | ||
| 344 | pushq %rbx /* pt_regs->rbx */ | 379 | pushq %rbx /* pt_regs->rbx */ |
| 380 | xorl %ebx, %ebx /* nospec rbx */ | ||
| 345 | pushq %rbp /* pt_regs->rbp */ | 381 | pushq %rbp /* pt_regs->rbp */ |
| 382 | xorl %ebp, %ebp /* nospec rbp */ | ||
| 346 | pushq %r12 /* pt_regs->r12 */ | 383 | pushq %r12 /* pt_regs->r12 */ |
| 384 | xorl %r12d, %r12d /* nospec r12 */ | ||
| 347 | pushq %r13 /* pt_regs->r13 */ | 385 | pushq %r13 /* pt_regs->r13 */ |
| 386 | xorl %r13d, %r13d /* nospec r13 */ | ||
| 348 | pushq %r14 /* pt_regs->r14 */ | 387 | pushq %r14 /* pt_regs->r14 */ |
| 388 | xorl %r14d, %r14d /* nospec r14 */ | ||
| 349 | pushq %r15 /* pt_regs->r15 */ | 389 | pushq %r15 /* pt_regs->r15 */ |
| 390 | xorl %r15d, %r15d /* nospec r15 */ | ||
| 350 | cld | 391 | cld |
| 351 | 392 | ||
| 352 | /* | 393 | /* |
| @@ -363,15 +404,3 @@ ENTRY(entry_INT80_compat) · 10 unmodified lines | |||
| 363 | TRACE_IRQS_ON | 404 | TRACE_IRQS_ON |
| 364 | jmp swapgs_restore_regs_and_return_to_usermode | 405 | jmp swapgs_restore_regs_and_return_to_usermode |
| 365 | END(entry_INT80_compat) | 406 | END(entry_INT80_compat) |
| 366 | |||
| 367 | ENTRY(stub32_clone) | ||
| 368 | /* | ||
| 369 | * The 32-bit clone ABI is: clone(..., int tls_val, int *child_tidptr). | ||
| 370 | * The 64-bit clone ABI is: clone(..., int *child_tidptr, int tls_val). | ||
| 371 | * | ||
| 372 | * The native 64-bit kernel's sys_clone() implements the latter, | ||
| 373 | * so we need to swap arguments here before calling it: | ||
| 374 | */ | ||
| 375 | xchg %r8, %rcx | ||
| 376 | jmp sys_clone | ||
| 377 | ENDPROC(stub32_clone) | ||
| @@ -8,12 +8,12 @@ · 7 unmodified lines | |||
| 8 | # | 8 | # |
| 9 | 0 i386 restart_syscall sys_restart_syscall | 9 | 0 i386 restart_syscall sys_restart_syscall |
| 10 | 1 i386 exit sys_exit | 10 | 1 i386 exit sys_exit |
| 11 | 2 i386 fork sys_fork sys_fork | 11 | 2 i386 fork sys_fork |
| 12 | 3 i386 read sys_read | 12 | 3 i386 read sys_read |
| 13 | 4 i386 write sys_write | 13 | 4 i386 write sys_write |
| 14 | 5 i386 open sys_open compat_sys_open | 14 | 5 i386 open sys_open compat_sys_open |
| 15 | 6 i386 close sys_close | 15 | 6 i386 close sys_close |
| 16 | 7 i386 waitpid sys_waitpid sys32_waitpid | 16 | 7 i386 waitpid sys_waitpid compat_sys_x86_waitpid |
| 17 | 8 i386 creat sys_creat | 17 | 8 i386 creat sys_creat |
| 18 | 9 i386 link sys_link | 18 | 9 i386 link sys_link |
| 19 | 10 i386 unlink sys_unlink | 19 | 10 i386 unlink sys_unlink |
| @@ -78,7 +78,7 @@ · 58 unmodified lines | |||
| 78 | 69 i386 ssetmask sys_ssetmask | 78 | 69 i386 ssetmask sys_ssetmask |
| 79 | 70 i386 setreuid sys_setreuid16 | 79 | 70 i386 setreuid sys_setreuid16 |
| 80 | 71 i386 setregid sys_setregid16 | 80 | 71 i386 setregid sys_setregid16 |
| 81 | 72 i386 sigsuspend sys_sigsuspend sys_sigsuspend | 81 | 72 i386 sigsuspend sys_sigsuspend |
| 82 | 73 i386 sigpending sys_sigpending compat_sys_sigpending | 82 | 73 i386 sigpending sys_sigpending compat_sys_sigpending |
| 83 | 74 i386 sethostname sys_sethostname | 83 | 74 i386 sethostname sys_sethostname |
| 84 | 75 i386 setrlimit sys_setrlimit compat_sys_setrlimit | 84 | 75 i386 setrlimit sys_setrlimit compat_sys_setrlimit |
| @@ -96,7 +96,7 @@ · 11 unmodified lines | |||
| 96 | 87 i386 swapon sys_swapon | 96 | 87 i386 swapon sys_swapon |
| 97 | 88 i386 reboot sys_reboot | 97 | 88 i386 reboot sys_reboot |
| 98 | 89 i386 readdir sys_old_readdir compat_sys_old_readdir | 98 | 89 i386 readdir sys_old_readdir compat_sys_old_readdir |
| 99 | 90 i386 mmap sys_old_mmap sys32_mmap | 99 | 90 i386 mmap sys_old_mmap compat_sys_x86_mmap |
| 100 | 91 i386 munmap sys_munmap | 100 | 91 i386 munmap sys_munmap |
| 101 | 92 i386 truncate sys_truncate compat_sys_truncate | 101 | 92 i386 truncate sys_truncate compat_sys_truncate |
| 102 | 93 i386 ftruncate sys_ftruncate compat_sys_ftruncate | 102 | 93 i386 ftruncate sys_ftruncate compat_sys_ftruncate |
| @@ -126,7 +126,7 @@ · 23 unmodified lines | |||
| 126 | 117 i386 ipc sys_ipc compat_sys_ipc | 126 | 117 i386 ipc sys_ipc compat_sys_ipc |
| 127 | 118 i386 fsync sys_fsync | 127 | 118 i386 fsync sys_fsync |
| 128 | 119 i386 sigreturn sys_sigreturn sys32_sigreturn | 128 | 119 i386 sigreturn sys_sigreturn sys32_sigreturn |
| 129 | 120 i386 clone sys_clone stub32_clone | 129 | 120 i386 clone sys_clone compat_sys_x86_clone |
| 130 | 121 i386 setdomainname sys_setdomainname | 130 | 121 i386 setdomainname sys_setdomainname |
| 131 | 122 i386 uname sys_newuname | 131 | 122 i386 uname sys_newuname |
| 132 | 123 i386 modify_ldt sys_modify_ldt | 132 | 123 i386 modify_ldt sys_modify_ldt |
| @@ -186,8 +186,8 @@ · 53 unmodified lines | |||
| 186 | 177 i386 rt_sigtimedwait sys_rt_sigtimedwait compat_sys_rt_sigtimedwait | 186 | 177 i386 rt_sigtimedwait sys_rt_sigtimedwait compat_sys_rt_sigtimedwait |
| 187 | 178 i386 rt_sigqueueinfo sys_rt_sigqueueinfo compat_sys_rt_sigqueueinfo | 187 | 178 i386 rt_sigqueueinfo sys_rt_sigqueueinfo compat_sys_rt_sigqueueinfo |
| 188 | 179 i386 rt_sigsuspend sys_rt_sigsuspend | 188 | 179 i386 rt_sigsuspend sys_rt_sigsuspend |
| 189 | 180 i386 pread64 sys_pread64 sys32_pread | 189 | 180 i386 pread64 sys_pread64 compat_sys_x86_pread |
| 190 | 181 i386 pwrite64 sys_pwrite64 sys32_pwrite | 190 | 181 i386 pwrite64 sys_pwrite64 compat_sys_x86_pwrite |
| 191 | 182 i386 chown sys_chown16 | 191 | 182 i386 chown sys_chown16 |
| 192 | 183 i386 getcwd sys_getcwd | 192 | 183 i386 getcwd sys_getcwd |
| 193 | 184 i386 capget sys_capget | 193 | 184 i386 capget sys_capget |
| @@ -196,14 +196,14 @@ · 2 unmodified lines | |||
| 196 | 187 i386 sendfile sys_sendfile compat_sys_sendfile | 196 | 187 i386 sendfile sys_sendfile compat_sys_sendfile |
| 197 | 188 i386 getpmsg | 197 | 188 i386 getpmsg |
| 198 | 189 i386 putpmsg | 198 | 189 i386 putpmsg |
| 199 | 190 i386 vfork sys_vfork sys_vfork | 199 | 190 i386 vfork sys_vfork |
| 200 | 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit | 200 | 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit |
| 201 | 192 i386 mmap2 sys_mmap_pgoff | 201 | 192 i386 mmap2 sys_mmap_pgoff |
| 202 | 193 i386 truncate64 sys_truncate64 sys32_truncate64 | 202 | 193 i386 truncate64 sys_truncate64 compat_sys_x86_truncate64 |
| 203 | 194 i386 ftruncate64 sys_ftruncate64 sys32_ftruncate64 | 203 | 194 i386 ftruncate64 sys_ftruncate64 compat_sys_x86_ftruncate64 |
| 204 | 195 i386 stat64 sys_stat64 sys32_stat64 | 204 | 195 i386 stat64 sys_stat64 compat_sys_x86_stat64 |
| 205 | 196 i386 lstat64 sys_lstat64 sys32_lstat64 | 205 | 196 i386 lstat64 sys_lstat64 compat_sys_x86_lstat64 |
| 206 | 197 i386 fstat64 sys_fstat64 sys32_fstat64 | 206 | 197 i386 fstat64 sys_fstat64 compat_sys_x86_fstat64 |
| 207 | 198 i386 lchown32 sys_lchown | 207 | 198 i386 lchown32 sys_lchown |
| 208 | 199 i386 getuid32 sys_getuid | 208 | 199 i386 getuid32 sys_getuid |
| 209 | 200 i386 getgid32 sys_getgid | 209 | 200 i386 getgid32 sys_getgid |
| @@ -231,7 +231,7 @@ · 21 unmodified lines | |||
| 231 | # 222 is unused | 231 | # 222 is unused |
| 232 | # 223 is unused | 232 | # 223 is unused |
| 233 | 224 i386 gettid sys_gettid | 233 | 224 i386 gettid sys_gettid |
| 234 | 225 i386 readahead sys_readahead sys32_readahead | 234 | 225 i386 readahead sys_readahead compat_sys_x86_readahead |
| 235 | 226 i386 setxattr sys_setxattr | 235 | 226 i386 setxattr sys_setxattr |
| 236 | 227 i386 lsetxattr sys_lsetxattr | 236 | 227 i386 lsetxattr sys_lsetxattr |
| 237 | 228 i386 fsetxattr sys_fsetxattr | 237 | 228 i386 fsetxattr sys_fsetxattr |
| @@ -256,7 +256,7 @@ · 18 unmodified lines | |||
| 256 | 247 i386 io_getevents sys_io_getevents compat_sys_io_getevents | 256 | 247 i386 io_getevents sys_io_getevents compat_sys_io_getevents |
| 257 | 248 i386 io_submit sys_io_submit compat_sys_io_submit | 257 | 248 i386 io_submit sys_io_submit compat_sys_io_submit |
| 258 | 249 i386 io_cancel sys_io_cancel | 258 | 249 i386 io_cancel sys_io_cancel |
| 259 | 250 i386 fadvise64 sys_fadvise64 sys32_fadvise64 | 259 | 250 i386 fadvise64 sys_fadvise64 compat_sys_x86_fadvise64 |
| 260 | # 251 is available for reuse (was briefly sys_set_zone_reclaim) | 260 | # 251 is available for reuse (was briefly sys_set_zone_reclaim) |
| 261 | 252 i386 exit_group sys_exit_group | 261 | 252 i386 exit_group sys_exit_group |
| 262 | 253 i386 lookup_dcookie sys_lookup_dcookie compat_sys_lookup_dcookie | 262 | 253 i386 lookup_dcookie sys_lookup_dcookie compat_sys_lookup_dcookie |
| @@ -278,7 +278,7 @@ · 15 unmodified lines | |||
| 278 | 269 i386 fstatfs64 sys_fstatfs64 compat_sys_fstatfs64 | 278 | 269 i386 fstatfs64 sys_fstatfs64 compat_sys_fstatfs64 |
| 279 | 270 i386 tgkill sys_tgkill | 279 | 270 i386 tgkill sys_tgkill |
| 280 | 271 i386 utimes sys_utimes compat_sys_utimes | 280 | 271 i386 utimes sys_utimes compat_sys_utimes |
| 281 | 272 i386 fadvise64_64 sys_fadvise64_64 sys32_fadvise64_64 | 281 | 272 i386 fadvise64_64 sys_fadvise64_64 compat_sys_x86_fadvise64_64 |
| 282 | 273 i386 vserver | 282 | 273 i386 vserver |
| 283 | 274 i386 mbind sys_mbind | 283 | 274 i386 mbind sys_mbind |
| 284 | 275 i386 get_mempolicy sys_get_mempolicy compat_sys_get_mempolicy | 284 | 275 i386 get_mempolicy sys_get_mempolicy compat_sys_get_mempolicy |
| @@ -306,7 +306,7 @@ · 21 unmodified lines | |||
| 306 | 297 i386 mknodat sys_mknodat | 306 | 297 i386 mknodat sys_mknodat |
| 307 | 298 i386 fchownat sys_fchownat | 307 | 298 i386 fchownat sys_fchownat |
| 308 | 299 i386 futimesat sys_futimesat compat_sys_futimesat | 308 | 299 i386 futimesat sys_futimesat compat_sys_futimesat |
| 309 | 300 i386 fstatat64 sys_fstatat64 sys32_fstatat | 309 | 300 i386 fstatat64 sys_fstatat64 compat_sys_x86_fstatat |
| 310 | 301 i386 unlinkat sys_unlinkat | 310 | 301 i386 unlinkat sys_unlinkat |
| 311 | 302 i386 renameat sys_renameat | 311 | 302 i386 renameat sys_renameat |
| 312 | 303 i386 linkat sys_linkat | 312 | 303 i386 linkat sys_linkat |
| @@ -320,7 +320,7 @@ · 7 unmodified lines | |||
| 320 | 311 i386 set_robust_list sys_set_robust_list compat_sys_set_robust_list | 320 | 311 i386 set_robust_list sys_set_robust_list compat_sys_set_robust_list |
| 321 | 312 i386 get_robust_list sys_get_robust_list compat_sys_get_robust_list | 321 | 312 i386 get_robust_list sys_get_robust_list compat_sys_get_robust_list |
| 322 | 313 i386 splice sys_splice | 322 | 313 i386 splice sys_splice |
| 323 | 314 i386 sync_file_range sys_sync_file_range sys32_sync_file_range | 323 | 314 i386 sync_file_range sys_sync_file_range compat_sys_x86_sync_file_range |
| 324 | 315 i386 tee sys_tee | 324 | 315 i386 tee sys_tee |
| 325 | 316 i386 vmsplice sys_vmsplice compat_sys_vmsplice | 325 | 316 i386 vmsplice sys_vmsplice compat_sys_vmsplice |
| 326 | 317 i386 move_pages sys_move_pages compat_sys_move_pages | 326 | 317 i386 move_pages sys_move_pages compat_sys_move_pages |
| @@ -330,7 +330,7 @@ · 3 unmodified lines | |||
| 330 | 321 i386 signalfd sys_signalfd compat_sys_signalfd | 330 | 321 i386 signalfd sys_signalfd compat_sys_signalfd |
| 331 | 322 i386 timerfd_create sys_timerfd_create | 331 | 322 i386 timerfd_create sys_timerfd_create |
| 332 | 323 i386 eventfd sys_eventfd | 332 | 323 i386 eventfd sys_eventfd |
| 333 | 324 i386 fallocate sys_fallocate sys32_fallocate | 333 | 324 i386 fallocate sys_fallocate compat_sys_x86_fallocate |
| 334 | 325 i386 timerfd_settime sys_timerfd_settime compat_sys_timerfd_settime | 334 | 325 i386 timerfd_settime sys_timerfd_settime compat_sys_timerfd_settime |
| 335 | 326 i386 timerfd_gettime sys_timerfd_gettime compat_sys_timerfd_gettime | 335 | 326 i386 timerfd_gettime sys_timerfd_gettime compat_sys_timerfd_gettime |
| 336 | 327 i386 signalfd4 sys_signalfd4 compat_sys_signalfd4 | 336 | 327 i386 signalfd4 sys_signalfd4 compat_sys_signalfd4 |
| @@ -42,10 +42,8 @@ · 41 unmodified lines | |||
| 42 | #define CREATE_TRACE_POINTS | 42 | #define CREATE_TRACE_POINTS |
| 43 | #include "vsyscall_trace.h" | 43 | #include "vsyscall_trace.h" |
| 44 | 44 | ||
| 45 | static enum { EMULATE, NATIVE, NONE } vsyscall_mode = | 45 | static enum { EMULATE, NONE } vsyscall_mode = |
| 46 | #if defined(CONFIG_LEGACY_VSYSCALL_NATIVE) | 46 | #ifdef CONFIG_LEGACY_VSYSCALL_NONE |
| 47 | NATIVE; | ||
| 48 | #elif defined(CONFIG_LEGACY_VSYSCALL_NONE) | ||
| 49 | NONE; | 47 | NONE; |
| 50 | #else | 48 | #else |
| 51 | EMULATE; | 49 | EMULATE; |
| @@ -56,8 +54,6 @@ static int __init vsyscall_setup(char *str) · 4 unmodified lines | |||
| 56 | if (str) { | 54 | if (str) { |
| 57 | if (!strcmp("emulate", str)) | 55 | if (!strcmp("emulate", str)) |
| 58 | vsyscall_mode = EMULATE; | 56 | vsyscall_mode = EMULATE; |
| 59 | else if (!strcmp("native", str)) | ||
| 60 | vsyscall_mode = NATIVE; | ||
| 61 | else if (!strcmp("none", str)) | 57 | else if (!strcmp("none", str)) |
| 62 | vsyscall_mode = NONE; | 58 | vsyscall_mode = NONE; |
| 63 | else | 59 | else |
| @@ -139,10 +135,6 @@ bool emulate_vsyscall(struct pt_regs *regs, unsigned long address) · 75 unmodified lines | |||
| 139 | 135 | ||
| 140 | WARN_ON_ONCE(address != regs->ip); | 136 | WARN_ON_ONCE(address != regs->ip); |
| 141 | 137 | ||
| 142 | /* This should be unreachable in NATIVE mode. */ | ||
| 143 | if (WARN_ON(vsyscall_mode == NATIVE)) | ||
| 144 | return false; | ||
| 145 | |||
| 146 | if (vsyscall_mode == NONE) { | 138 | if (vsyscall_mode == NONE) { |
| 147 | warn_bad_vsyscall(KERN_INFO, regs, | 139 | warn_bad_vsyscall(KERN_INFO, regs, |
| 148 | "vsyscall attempted with vsyscall=none"); | 140 | "vsyscall attempted with vsyscall=none"); |
| @@ -370,9 +362,7 @@ void __init map_vsyscall(void) · 221 unmodified lines | |||
| 370 | 362 | ||
| 371 | if (vsyscall_mode != NONE) { | 363 | if (vsyscall_mode != NONE) { |
| 372 | __set_fixmap(VSYSCALL_PAGE, physaddr_vsyscall, | 364 | __set_fixmap(VSYSCALL_PAGE, physaddr_vsyscall, |
| 373 | vsyscall_mode == NATIVE | 365 | PAGE_KERNEL_VVAR); |
| 374 | ? PAGE_KERNEL_VSYSCALL | ||
| 375 | : PAGE_KERNEL_VVAR); | ||
| 376 | set_vsyscall_pgtable_user_bits(swapper_pg_dir); | 366 | set_vsyscall_pgtable_user_bits(swapper_pg_dir); |
| 377 | } | 367 | } |
| 378 | 368 | ||
| @@ -3559,7 +3559,7 @@ static int intel_snb_pebs_broken(int cpu) · 3558 unmodified lines | |||
| 3559 | break; | 3559 | break; |
| 3560 | 3560 | ||
| 3561 | case INTEL_FAM6_SANDYBRIDGE_X: | 3561 | case INTEL_FAM6_SANDYBRIDGE_X: |
| 3562 | switch (cpu_data(cpu).x86_mask) { | 3562 | switch (cpu_data(cpu).x86_stepping) { |
| 3563 | case 6: rev = 0x618; break; | 3563 | case 6: rev = 0x618; break; |
| 3564 | case 7: rev = 0x70c; break; | 3564 | case 7: rev = 0x70c; break; |
| 3565 | } | 3565 | } |
| @@ -1186,7 +1186,7 @@ void __init intel_pmu_lbr_init_atom(void) · 1185 unmodified lines | |||
| 1186 | * on PMU interrupt | 1186 | * on PMU interrupt |
| 1187 | */ | 1187 | */ |
| 1188 | if (boot_cpu_data.x86_model == 28 | 1188 | if (boot_cpu_data.x86_model == 28 |
| 1189 | && boot_cpu_data.x86_mask < 10) { | 1189 | && boot_cpu_data.x86_stepping < 10) { |
| 1190 | pr_cont("LBR disabled due to erratum"); | 1190 | pr_cont("LBR disabled due to erratum"); |
| 1191 | return; | 1191 | return; |
| 1192 | } | 1192 | } |
| @@ -234,7 +234,7 @@ static __initconst const struct x86_pmu p6_pmu = { · 233 unmodified lines | |||
| 234 | 234 | ||
| 235 | static __init void p6_pmu_rdpmc_quirk(void) | 235 | static __init void p6_pmu_rdpmc_quirk(void) |
| 236 | { | 236 | { |
| 237 | if (boot_cpu_data.x86_mask < 9) { | 237 | if (boot_cpu_data.x86_stepping < 9) { |
| 238 | /* | 238 | /* |
| 239 | * PPro erratum 26; fixed in stepping 9 and above. | 239 | * PPro erratum 26; fixed in stepping 9 and above. |
| 240 | */ | 240 | */ |
| @@ -3606,7 +3606,7 @@ static struct intel_uncore_type skx_uncore_imc = { · 3605 unmodified lines | |||
| 3606 | }; | 3606 | }; |
| 3607 | 3607 | ||
| 3608 | static struct attribute *skx_upi_uncore_formats_attr[] = { | 3608 | static struct attribute *skx_upi_uncore_formats_attr[] = { |
| 3609 | &format_attr_event_ext.attr, | 3609 | &format_attr_event.attr, |
| 3610 | &format_attr_umask_ext.attr, | 3610 | &format_attr_umask_ext.attr, |
| 3611 | &format_attr_edge.attr, | 3611 | &format_attr_edge.attr, |
| 3612 | &format_attr_inv.attr, | 3612 | &format_attr_inv.attr, |
| @@ -51,15 +51,14 @@ · 50 unmodified lines | |||
| 51 | #define AA(__x) ((unsigned long)(__x)) | 51 | #define AA(__x) ((unsigned long)(__x)) |
| 52 | 52 | ||
| 53 | 53 | ||
| 54 | asmlinkage long sys32_truncate64(const char __user *filename, | 54 | COMPAT_SYSCALL_DEFINE3(x86_truncate64, const char __user *, filename, |
| 55 | unsigned long offset_low, | 55 | unsigned long, offset_low, unsigned long, offset_high) |
| 56 | unsigned long offset_high) | ||
| 57 | { | 56 | { |
| 58 | return sys_truncate(filename, ((loff_t) offset_high << 32) | offset_low); | 57 | return sys_truncate(filename, ((loff_t) offset_high << 32) | offset_low); |
| 59 | } | 58 | } |
| 60 | 59 | ||
| 61 | asmlinkage long sys32_ftruncate64(unsigned int fd, unsigned long offset_low, | 60 | COMPAT_SYSCALL_DEFINE3(x86_ftruncate64, unsigned int, fd, |
| 62 | unsigned long offset_high) | 61 | unsigned long, offset_low, unsigned long, offset_high) |
| 63 | { | 62 | { |
| 64 | return sys_ftruncate(fd, ((loff_t) offset_high << 32) | offset_low); | 63 | return sys_ftruncate(fd, ((loff_t) offset_high << 32) | offset_low); |
| 65 | } | 64 | } |
| @@ -96,8 +95,8 @@ static int cp_stat64(struct stat64 __user *ubuf, struct kstat *stat) · 30 unmodified lines | |||
| 96 | return 0; | 95 | return 0; |
| 97 | } | 96 | } |
| 98 | 97 | ||
| 99 | asmlinkage long sys32_stat64(const char __user *filename, | 98 | COMPAT_SYSCALL_DEFINE2(x86_stat64, const char __user *, filename, |
| 100 | struct stat64 __user *statbuf) | 99 | struct stat64 __user *, statbuf) |
| 101 | { | 100 | { |
| 102 | struct kstat stat; | 101 | struct kstat stat; |
| 103 | int ret = vfs_stat(filename, &stat); | 102 | int ret = vfs_stat(filename, &stat); |
| @@ -107,8 +106,8 @@ asmlinkage long sys32_stat64(const char __user *filename, · 3 unmodified lines | |||
| 107 | return ret; | 106 | return ret; |
| 108 | } | 107 | } |
| 109 | 108 | ||
| 110 | asmlinkage long sys32_lstat64(const char __user *filename, | 109 | COMPAT_SYSCALL_DEFINE2(x86_lstat64, const char __user *, filename, |
| 111 | struct stat64 __user *statbuf) | 110 | struct stat64 __user *, statbuf) |
| 112 | { | 111 | { |
| 113 | struct kstat stat; | 112 | struct kstat stat; |
| 114 | int ret = vfs_lstat(filename, &stat); | 113 | int ret = vfs_lstat(filename, &stat); |
| @@ -117,7 +116,8 @@ asmlinkage long sys32_lstat64(const char __user *filename, · 2 unmodified lines | |||
| 117 | return ret; | 116 | return ret; |
| 118 | } | 117 | } |
| 119 | 118 | ||
| 120 | asmlinkage long sys32_fstat64(unsigned int fd, struct stat64 __user *statbuf) | 119 | COMPAT_SYSCALL_DEFINE2(x86_fstat64, unsigned int, fd, |
| 120 | struct stat64 __user *, statbuf) | ||
| 121 | { | 121 | { |
| 122 | struct kstat stat; | 122 | struct kstat stat; |
| 123 | int ret = vfs_fstat(fd, &stat); | 123 | int ret = vfs_fstat(fd, &stat); |
| @@ -126,8 +126,9 @@ asmlinkage long sys32_fstat64(unsigned int fd, struct stat64 __user *statbuf) · 2 unmodified lines | |||
| 126 | return ret; | 126 | return ret; |
| 127 | } | 127 | } |
| 128 | 128 | ||
| 129 | asmlinkage long sys32_fstatat(unsigned int dfd, const char __user *filename, | 129 | COMPAT_SYSCALL_DEFINE4(x86_fstatat, unsigned int, dfd, |
| 130 | struct stat64 __user *statbuf, int flag) | 130 | const char __user *, filename, |
| 131 | struct stat64 __user *, statbuf, int, flag) | ||
| 131 | { | 132 | { |
| 132 | struct kstat stat; | 133 | struct kstat stat; |
| 133 | int error; | 134 | int error; |
| @@ -153,7 +154,7 @@ struct mmap_arg_struct32 { · 19 unmodified lines | |||
| 153 | unsigned int offset; | 154 | unsigned int offset; |
| 154 | }; | 155 | }; |
| 155 | 156 | ||
| 156 | asmlinkage long sys32_mmap(struct mmap_arg_struct32 __user *arg) | 157 | COMPAT_SYSCALL_DEFINE1(x86_mmap, struct mmap_arg_struct32 __user *, arg) |
| 157 | { | 158 | { |
| 158 | struct mmap_arg_struct32 a; | 159 | struct mmap_arg_struct32 a; |
| 159 | 160 | ||
| @@ -167,22 +168,22 @@ asmlinkage long sys32_mmap(struct mmap_arg_struct32 __user *arg) · 7 unmodified lines | |||
| 167 | a.offset>>PAGE_SHIFT); | 168 | a.offset>>PAGE_SHIFT); |
| 168 | } | 169 | } |
| 169 | 170 | ||
| 170 | asmlinkage long sys32_waitpid(compat_pid_t pid, unsigned int __user *stat_addr, | 171 | COMPAT_SYSCALL_DEFINE3(x86_waitpid, compat_pid_t, pid, unsigned int __user *, |
| 171 | int options) | 172 | stat_addr, int, options) |
| 172 | { | 173 | { |
| 173 | return compat_sys_wait4(pid, stat_addr, options, NULL); | 174 | return compat_sys_wait4(pid, stat_addr, options, NULL); |
| 174 | } | 175 | } |
| 175 | 176 | ||
| 176 | /* warning: next two assume little endian */ | 177 | /* warning: next two assume little endian */ |
| 177 | asmlinkage long sys32_pread(unsigned int fd, char __user *ubuf, u32 count, | 178 | COMPAT_SYSCALL_DEFINE5(x86_pread, unsigned int, fd, char __user *, ubuf, |
| 178 | u32 poslo, u32 poshi) | 179 | u32, count, u32, poslo, u32, poshi) |
| 179 | { | 180 | { |
| 180 | return sys_pread64(fd, ubuf, count, | 181 | return sys_pread64(fd, ubuf, count, |
| 181 | ((loff_t)AA(poshi) << 32) | AA(poslo)); | 182 | ((loff_t)AA(poshi) << 32) | AA(poslo)); |
| 182 | } | 183 | } |
| 183 | 184 | ||
| 184 | asmlinkage long sys32_pwrite(unsigned int fd, const char __user *ubuf, | 185 | COMPAT_SYSCALL_DEFINE5(x86_pwrite, unsigned int, fd, const char __user *, ubuf, |
| 185 | u32 count, u32 poslo, u32 poshi) | 186 | u32, count, u32, poslo, u32, poshi) |
| 186 | { | 187 | { |
| 187 | return sys_pwrite64(fd, ubuf, count, | 188 | return sys_pwrite64(fd, ubuf, count, |
| 188 | ((loff_t)AA(poshi) << 32) | AA(poslo)); | 189 | ((loff_t)AA(poshi) << 32) | AA(poslo)); |
| @@ -193,8 +194,9 @@ asmlinkage long sys32_pwrite(unsigned int fd, const char __user *ubuf, · 4 unmodified lines | |||
| 193 | * Some system calls that need sign extended arguments. This could be | 194 | * Some system calls that need sign extended arguments. This could be |
| 194 | * done by a generic wrapper. | 195 | * done by a generic wrapper. |
| 195 | */ | 196 | */ |
| 196 | long sys32_fadvise64_64(int fd, __u32 offset_low, __u32 offset_high, | 197 | COMPAT_SYSCALL_DEFINE6(x86_fadvise64_64, int, fd, __u32, offset_low, |
| 197 | __u32 len_low, __u32 len_high, int advice) | 198 | __u32, offset_high, __u32, len_low, __u32, len_high, |
| 199 | int, advice) | ||
| 198 | { | 200 | { |
| 199 | return sys_fadvise64_64(fd, | 201 | return sys_fadvise64_64(fd, |
| 200 | (((u64)offset_high)<<32) | offset_low, | 202 | (((u64)offset_high)<<32) | offset_low, |
| @@ -202,31 +204,43 @@ long sys32_fadvise64_64(int fd, __u32 offset_low, __u32 offset_high, · 1 unmodified line | |||
| 202 | advice); | 204 | advice); |
| 203 | } | 205 | } |
| 204 | 206 | ||
| 205 | asmlinkage ssize_t sys32_readahead(int fd, unsigned off_lo, unsigned off_hi, | 207 | COMPAT_SYSCALL_DEFINE4(x86_readahead, int, fd, unsigned int, off_lo, |
| 206 | size_t count) | 208 | unsigned int, off_hi, size_t, count) |
| 207 | { | 209 | { |
| 208 | return sys_readahead(fd, ((u64)off_hi << 32) | off_lo, count); | 210 | return sys_readahead(fd, ((u64)off_hi << 32) | off_lo, count); |
| 209 | } | 211 | } |
| 210 | 212 | ||
| 211 | asmlinkage long sys32_sync_file_range(int fd, unsigned off_low, unsigned off_hi, | 213 | COMPAT_SYSCALL_DEFINE6(x86_sync_file_range, int, fd, unsigned int, off_low, |
| 212 | unsigned n_low, unsigned n_hi, int flags) | 214 | unsigned int, off_hi, unsigned int, n_low, |
| 215 | unsigned int, n_hi, int, flags) | ||
| 213 | { | 216 | { |
| 214 | return sys_sync_file_range(fd, | 217 | return sys_sync_file_range(fd, |
| 215 | ((u64)off_hi << 32) | off_low, | 218 | ((u64)off_hi << 32) | off_low, |
| 216 | ((u64)n_hi << 32) | n_low, flags); | 219 | ((u64)n_hi << 32) | n_low, flags); |
| 217 | } | 220 | } |
| 218 | 221 | ||
| 219 | asmlinkage long sys32_fadvise64(int fd, unsigned offset_lo, unsigned offset_hi, | 222 | COMPAT_SYSCALL_DEFINE5(x86_fadvise64, int, fd, unsigned int, offset_lo, |
| 220 | size_t len, int advice) | 223 | unsigned int, offset_hi, size_t, len, int, advice) |
| 221 | { | 224 | { |
| 222 | return sys_fadvise64_64(fd, ((u64)offset_hi << 32) | offset_lo, | 225 | return sys_fadvise64_64(fd, ((u64)offset_hi << 32) | offset_lo, |
| 223 | len, advice); | 226 | len, advice); |
| 224 | } | 227 | } |
| 225 | 228 | ||
| 226 | asmlinkage long sys32_fallocate(int fd, int mode, unsigned offset_lo, | 229 | COMPAT_SYSCALL_DEFINE6(x86_fallocate, int, fd, int, mode, |
| 227 | unsigned offset_hi, unsigned len_lo, | 230 | unsigned int, offset_lo, unsigned int, offset_hi, |
| 228 | unsigned len_hi) | 231 | unsigned int, len_lo, unsigned int, len_hi) |
| 229 | { | 232 | { |
| 230 | return sys_fallocate(fd, mode, ((u64)offset_hi << 32) | offset_lo, | 233 | return sys_fallocate(fd, mode, ((u64)offset_hi << 32) | offset_lo, |
| 231 | ((u64)len_hi << 32) | len_lo); | 234 | ((u64)len_hi << 32) | len_lo); |
| 232 | } | 235 | } |
| 236 | |||
| 237 | /* | ||
| 238 | * The 32-bit clone ABI is CONFIG_CLONE_BACKWARDS | ||
| 239 | */ | ||
| 240 | COMPAT_SYSCALL_DEFINE5(x86_clone, unsigned long, clone_flags, | ||
| 241 | unsigned long, newsp, int __user *, parent_tidptr, | ||
| 242 | unsigned long, tls_val, int __user *, child_tidptr) | ||
| 243 | { | ||
| 244 | return sys_clone(clone_flags, newsp, parent_tidptr, child_tidptr, | ||
| 245 | tls_val); | ||
| 246 | } | ||
| @@ -94,7 +94,7 @@ static inline unsigned int acpi_processor_cstate_check(unsigned int max_cstate) · 93 unmodified lines | |||
| 94 | if (boot_cpu_data.x86 == 0x0F && | 94 | if (boot_cpu_data.x86 == 0x0F && |
| 95 | boot_cpu_data.x86_vendor == X86_VENDOR_AMD && | 95 | boot_cpu_data.x86_vendor == X86_VENDOR_AMD && |
| 96 | boot_cpu_data.x86_model <= 0x05 && | 96 | boot_cpu_data.x86_model <= 0x05 && |
| 97 | boot_cpu_data.x86_mask < 0x0A) | 97 | boot_cpu_data.x86_stepping < 0x0A) |
| 98 | return 1; | 98 | return 1; |
| 99 | else if (boot_cpu_has(X86_BUG_AMD_APIC_C1E)) | 99 | else if (boot_cpu_has(X86_BUG_AMD_APIC_C1E)) |
| 100 | return 1; | 100 | return 1; |
| @@ -7,6 +7,8 @@ · 6 unmodified lines | |||
| 7 | #ifndef _ASM_X86_MACH_DEFAULT_APM_H | 7 | #ifndef _ASM_X86_MACH_DEFAULT_APM_H |
| 8 | #define _ASM_X86_MACH_DEFAULT_APM_H | 8 | #define _ASM_X86_MACH_DEFAULT_APM_H |
| 9 | 9 | ||
| 10 | #include <asm/nospec-branch.h> | ||
| 11 | |||
| 10 | #ifdef APM_ZERO_SEGS | 12 | #ifdef APM_ZERO_SEGS |
| 11 | # define APM_DO_ZERO_SEGS \ | 13 | # define APM_DO_ZERO_SEGS \ |
| 12 | "pushl %%ds\n\t" \ | 14 | "pushl %%ds\n\t" \ |
| @@ -32,6 +34,7 @@ static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in, · 19 unmodified lines | |||
| 32 | * N.B. We do NOT need a cld after the BIOS call | 34 | * N.B. We do NOT need a cld after the BIOS call |
| 33 | * because we always save and restore the flags. | 35 | * because we always save and restore the flags. |
| 34 | */ | 36 | */ |
| 37 | firmware_restrict_branch_speculation_start(); | ||
| 35 | __asm__ __volatile__(APM_DO_ZERO_SEGS | 38 | __asm__ __volatile__(APM_DO_ZERO_SEGS |
| 36 | "pushl %%edi\n\t" | 39 | "pushl %%edi\n\t" |
| 37 | "pushl %%ebp\n\t" | 40 | "pushl %%ebp\n\t" |
| @@ -44,6 +47,7 @@ static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in, · 6 unmodified lines | |||
| 44 | "=S" (*esi) | 47 | "=S" (*esi) |
| 45 | : "a" (func), "b" (ebx_in), "c" (ecx_in) | 48 | : "a" (func), "b" (ebx_in), "c" (ecx_in) |
| 46 | : "memory", "cc"); | 49 | : "memory", "cc"); |
| 50 | firmware_restrict_branch_speculation_end(); | ||
| 47 | } | 51 | } |
| 48 | 52 | ||
| 49 | static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, | 53 | static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, |
| @@ -56,6 +60,7 @@ static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, · 6 unmodified lines | |||
| 56 | * N.B. We do NOT need a cld after the BIOS call | 60 | * N.B. We do NOT need a cld after the BIOS call |
| 57 | * because we always save and restore the flags. | 61 | * because we always save and restore the flags. |
| 58 | */ | 62 | */ |
| 63 | firmware_restrict_branch_speculation_start(); | ||
| 59 | __asm__ __volatile__(APM_DO_ZERO_SEGS | 64 | __asm__ __volatile__(APM_DO_ZERO_SEGS |
| 60 | "pushl %%edi\n\t" | 65 | "pushl %%edi\n\t" |
| 61 | "pushl %%ebp\n\t" | 66 | "pushl %%ebp\n\t" |
| @@ -68,6 +73,7 @@ static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in, · 6 unmodified lines | |||
| 68 | "=S" (si) | 73 | "=S" (si) |
| 69 | : "a" (func), "b" (ebx_in), "c" (ecx_in) | 74 | : "a" (func), "b" (ebx_in), "c" (ecx_in) |
| 70 | : "memory", "cc"); | 75 | : "memory", "cc"); |
| 76 | firmware_restrict_branch_speculation_end(); | ||
| 71 | return error; | 77 | return error; |
| 72 | } | 78 | } |
| 73 | 79 | ||
| @@ -38,7 +38,4 @@ INDIRECT_THUNK(dx) · 37 unmodified lines | |||
| 38 | INDIRECT_THUNK(si) | 38 | INDIRECT_THUNK(si) |
| 39 | INDIRECT_THUNK(di) | 39 | INDIRECT_THUNK(di) |
| 40 | INDIRECT_THUNK(bp) | 40 | INDIRECT_THUNK(bp) |
| 41 | asmlinkage void __fill_rsb(void); | ||
| 42 | asmlinkage void __clear_rsb(void); | ||
| 43 | |||
| 44 | #endif /* CONFIG_RETPOLINE */ | 41 | #endif /* CONFIG_RETPOLINE */ |
| @@ -40,7 +40,7 @@ static inline unsigned long array_index_mask_nospec(unsigned long index, · 39 unmodified lines | |||
| 40 | 40 | ||
| 41 | asm ("cmp %1,%2; sbb %0,%0;" | 41 | asm ("cmp %1,%2; sbb %0,%0;" |
| 42 | :"=r" (mask) | 42 | :"=r" (mask) |
| 43 | :"r"(size),"r" (index) | 43 | :"g"(size),"r" (index) |
| 44 | :"cc"); | 44 | :"cc"); |
| 45 | return mask; | 45 | return mask; |
| 46 | } | 46 | } |
| @@ -78,7 +78,7 @@ set_bit(long nr, volatile unsigned long *addr) · 77 unmodified lines | |||
| 78 | : "iq" ((u8)CONST_MASK(nr)) | 78 | : "iq" ((u8)CONST_MASK(nr)) |
| 79 | : "memory"); | 79 | : "memory"); |
| 80 | } else { | 80 | } else { |
| 81 | asm volatile(LOCK_PREFIX "bts %1,%0" | 81 | asm volatile(LOCK_PREFIX __ASM_SIZE(bts) " %1,%0" |
| 82 | : BITOP_ADDR(addr) : "Ir" (nr) : "memory"); | 82 | : BITOP_ADDR(addr) : "Ir" (nr) : "memory"); |
| 83 | } | 83 | } |
| 84 | } | 84 | } |
| @@ -94,7 +94,7 @@ set_bit(long nr, volatile unsigned long *addr) · 9 unmodified lines | |||
| 94 | */ | 94 | */ |
| 95 | static __always_inline void __set_bit(long nr, volatile unsigned long *addr) | 95 | static __always_inline void __set_bit(long nr, volatile unsigned long *addr) |
| 96 | { | 96 | { |
| 97 | asm volatile("bts %1,%0" : ADDR : "Ir" (nr) : "memory"); | 97 | asm volatile(__ASM_SIZE(bts) " %1,%0" : ADDR : "Ir" (nr) : "memory"); |
| 98 | } | 98 | } |
| 99 | 99 | ||
| 100 | /** | 100 | /** |
| @@ -115,7 +115,7 @@ clear_bit(long nr, volatile unsigned long *addr) · 14 unmodified lines | |||
| 115 | : CONST_MASK_ADDR(nr, addr) | 115 | : CONST_MASK_ADDR(nr, addr) |
| 116 | : "iq" ((u8)~CONST_MASK(nr))); | 116 | : "iq" ((u8)~CONST_MASK(nr))); |
| 117 | } else { | 117 | } else { |
| 118 | asm volatile(LOCK_PREFIX "btr %1,%0" | 118 | asm volatile(LOCK_PREFIX __ASM_SIZE(btr) " %1,%0" |
| 119 | : BITOP_ADDR(addr) | 119 | : BITOP_ADDR(addr) |
| 120 | : "Ir" (nr)); | 120 | : "Ir" (nr)); |
| 121 | } | 121 | } |
| @@ -137,7 +137,7 @@ static __always_inline void clear_bit_unlock(long nr, volatile unsigned long *ad · 15 unmodified lines | |||
| 137 | 137 | ||
| 138 | static __always_inline void __clear_bit(long nr, volatile unsigned long *addr) | 138 | static __always_inline void __clear_bit(long nr, volatile unsigned long *addr) |
| 139 | { | 139 | { |
| 140 | asm volatile("btr %1,%0" : ADDR : "Ir" (nr)); | 140 | asm volatile(__ASM_SIZE(btr) " %1,%0" : ADDR : "Ir" (nr)); |
| 141 | } | 141 | } |
| 142 | 142 | ||
| 143 | static __always_inline bool clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) | 143 | static __always_inline bool clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) |
| @@ -182,7 +182,7 @@ static __always_inline void __clear_bit_unlock(long nr, volatile unsigned long * · 38 unmodified lines | |||
| 182 | */ | 182 | */ |
| 183 | static __always_inline void __change_bit(long nr, volatile unsigned long *addr) | 183 | static __always_inline void __change_bit(long nr, volatile unsigned long *addr) |
| 184 | { | 184 | { |
| 185 | asm volatile("btc %1,%0" : ADDR : "Ir" (nr)); | 185 | asm volatile(__ASM_SIZE(btc) " %1,%0" : ADDR : "Ir" (nr)); |
| 186 | } | 186 | } |
| 187 | 187 | ||
| 188 | /** | 188 | /** |
| @@ -201,7 +201,7 @@ static __always_inline void change_bit(long nr, volatile unsigned long *addr) · 12 unmodified lines | |||
| 201 | : CONST_MASK_ADDR(nr, addr) | 201 | : CONST_MASK_ADDR(nr, addr) |
| 202 | : "iq" ((u8)CONST_MASK(nr))); | 202 | : "iq" ((u8)CONST_MASK(nr))); |
| 203 | } else { | 203 | } else { |
| 204 | asm volatile(LOCK_PREFIX "btc %1,%0" | 204 | asm volatile(LOCK_PREFIX __ASM_SIZE(btc) " %1,%0" |
| 205 | : BITOP_ADDR(addr) | 205 | : BITOP_ADDR(addr) |
| 206 | : "Ir" (nr)); | 206 | : "Ir" (nr)); |
| 207 | } | 207 | } |
| @@ -217,7 +217,8 @@ static __always_inline void change_bit(long nr, volatile unsigned long *addr) · 9 unmodified lines | |||
| 217 | */ | 217 | */ |
| 218 | static __always_inline bool test_and_set_bit(long nr, volatile unsigned long *addr) | 218 | static __always_inline bool test_and_set_bit(long nr, volatile unsigned long *addr) |
| 219 | { | 219 | { |
| 220 | GEN_BINARY_RMWcc(LOCK_PREFIX "bts", *addr, "Ir", nr, "%0", c); | 220 | GEN_BINARY_RMWcc(LOCK_PREFIX __ASM_SIZE(bts), |
| 221 | *addr, "Ir", nr, "%0", c); | ||
| 221 | } | 222 | } |
| 222 | 223 | ||
| 223 | /** | 224 | /** |
| @@ -246,7 +247,7 @@ static __always_inline bool __test_and_set_bit(long nr, volatile unsigned long * · 22 unmodified lines | |||
| 246 | { | 247 | { |
| 247 | bool oldbit; | 248 | bool oldbit; |
| 248 | 249 | ||
| 249 | asm("bts %2,%1" | 250 | asm(__ASM_SIZE(bts) " %2,%1" |
| 250 | CC_SET(c) | 251 | CC_SET(c) |
| 251 | : CC_OUT(c) (oldbit), ADDR | 252 | : CC_OUT(c) (oldbit), ADDR |
| 252 | : "Ir" (nr)); | 253 | : "Ir" (nr)); |
| @@ -263,7 +264,8 @@ static __always_inline bool __test_and_set_bit(long nr, volatile unsigned long * · 10 unmodified lines | |||
| 263 | */ | 264 | */ |
| 264 | static __always_inline bool test_and_clear_bit(long nr, volatile unsigned long *addr) | 265 | static __always_inline bool test_and_clear_bit(long nr, volatile unsigned long *addr) |
| 265 | { | 266 | { |
| 266 | GEN_BINARY_RMWcc(LOCK_PREFIX "btr", *addr, "Ir", nr, "%0", c); | 267 | GEN_BINARY_RMWcc(LOCK_PREFIX __ASM_SIZE(btr), |
| 268 | *addr, "Ir", nr, "%0", c); | ||
| 267 | } | 269 | } |
| 268 | 270 | ||
| 269 | /** | 271 | /** |
| @@ -286,7 +288,7 @@ static __always_inline bool __test_and_clear_bit(long nr, volatile unsigned long · 16 unmodified lines | |||
| 286 | { | 288 | { |
| 287 | bool oldbit; | 289 | bool oldbit; |
| 288 | 290 | ||
| 289 | asm volatile("btr %2,%1" | 291 | asm volatile(__ASM_SIZE(btr) " %2,%1" |
| 290 | CC_SET(c) | 292 | CC_SET(c) |
| 291 | : CC_OUT(c) (oldbit), ADDR | 293 | : CC_OUT(c) (oldbit), ADDR |
| 292 | : "Ir" (nr)); | 294 | : "Ir" (nr)); |
| @@ -298,7 +300,7 @@ static __always_inline bool __test_and_change_bit(long nr, volatile unsigned lon · 5 unmodified lines | |||
| 298 | { | 300 | { |
| 299 | bool oldbit; | 301 | bool oldbit; |
| 300 | 302 | ||
| 301 | asm volatile("btc %2,%1" | 303 | asm volatile(__ASM_SIZE(btc) " %2,%1" |
| 302 | CC_SET(c) | 304 | CC_SET(c) |
| 303 | : CC_OUT(c) (oldbit), ADDR | 305 | : CC_OUT(c) (oldbit), ADDR |
| 304 | : "Ir" (nr) : "memory"); | 306 | : "Ir" (nr) : "memory"); |
| @@ -316,7 +318,8 @@ static __always_inline bool __test_and_change_bit(long nr, volatile unsigned lon · 11 unmodified lines | |||
| 316 | */ | 318 | */ |
| 317 | static __always_inline bool test_and_change_bit(long nr, volatile unsigned long *addr) | 319 | static __always_inline bool test_and_change_bit(long nr, volatile unsigned long *addr) |
| 318 | { | 320 | { |
| 319 | GEN_BINARY_RMWcc(LOCK_PREFIX "btc", *addr, "Ir", nr, "%0", c); | 321 | GEN_BINARY_RMWcc(LOCK_PREFIX __ASM_SIZE(btc), |
| 322 | *addr, "Ir", nr, "%0", c); | ||
| 320 | } | 323 | } |
| 321 | 324 | ||
| 322 | static __always_inline bool constant_test_bit(long nr, const volatile unsigned long *addr) | 325 | static __always_inline bool constant_test_bit(long nr, const volatile unsigned long *addr) |
| @@ -329,7 +332,7 @@ static __always_inline bool variable_test_bit(long nr, volatile const unsigned l · 6 unmodified lines | |||
| 329 | { | 332 | { |
| 330 | bool oldbit; | 333 | bool oldbit; |
| 331 | 334 | ||
| 332 | asm volatile("bt %2,%1" | 335 | asm volatile(__ASM_SIZE(bt) " %2,%1" |
| 333 | CC_SET(c) | 336 | CC_SET(c) |
| 334 | : CC_OUT(c) (oldbit) | 337 | : CC_OUT(c) (oldbit) |
| 335 | : "m" (*(unsigned long *)addr), "Ir" (nr)); | 338 | : "m" (*(unsigned long *)addr), "Ir" (nr)); |
| @@ -5,23 +5,20 @@ · 4 unmodified lines | |||
| 5 | #include <linux/stringify.h> | 5 | #include <linux/stringify.h> |
| 6 | 6 | ||
| 7 | /* | 7 | /* |
| 8 | * Since some emulators terminate on UD2, we cannot use it for WARN. | 8 | * Despite that some emulators terminate on UD2, we use it for WARN(). |
| 9 | * Since various instruction decoders disagree on the length of UD1, | ||
| 10 | * we cannot use it either. So use UD0 for WARN. | ||
| 11 | * | 9 | * |
| 12 | * (binutils knows about "ud1" but {en,de}codes it as 2 bytes, whereas | 10 | * Since various instruction decoders/specs disagree on the encoding of |
| 13 | * our kernel decoder thinks it takes a ModRM byte, which seems consistent | 11 | * UD0/UD1. |
| 14 | * with various things like the Intel SDM instruction encoding rules) | ||
| 15 | */ | 12 | */ |
| 16 | 13 | ||
| 17 | #define ASM_UD0 ".byte 0x0f, 0xff" | 14 | #define ASM_UD0 ".byte 0x0f, 0xff" /* + ModRM (for Intel) */ |
| 18 | #define ASM_UD1 ".byte 0x0f, 0xb9" /* + ModRM */ | 15 | #define ASM_UD1 ".byte 0x0f, 0xb9" /* + ModRM */ |
| 19 | #define ASM_UD2 ".byte 0x0f, 0x0b" | 16 | #define ASM_UD2 ".byte 0x0f, 0x0b" |
| 20 | 17 | ||
| 21 | #define INSN_UD0 0xff0f | 18 | #define INSN_UD0 0xff0f |
| 22 | #define INSN_UD2 0x0b0f | 19 | #define INSN_UD2 0x0b0f |
| 23 | 20 | ||
| 24 | #define LEN_UD0 2 | 21 | #define LEN_UD2 2 |
| 25 | 22 | ||
| 26 | #ifdef CONFIG_GENERIC_BUG | 23 | #ifdef CONFIG_GENERIC_BUG |
| 27 | 24 | ||
| @@ -77,7 +74,11 @@ do { \ · 49 unmodified lines | |||
| 77 | unreachable(); \ | 74 | unreachable(); \ |
| 78 | } while (0) | 75 | } while (0) |
| 79 | 76 | ||
| 80 | #define __WARN_FLAGS(flags) _BUG_FLAGS(ASM_UD0, BUGFLAG_WARNING|(flags)) | 77 | #define __WARN_FLAGS(flags) \ |
| 78 | do { \ | ||
| 79 | _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags)); \ | ||
| 80 | annotate_reachable(); \ | ||
| 81 | } while (0) | ||
| 81 | 82 | ||
| 82 | #include <asm-generic/bug.h> | 83 | #include <asm-generic/bug.h> |
| 83 | 84 | ||
| @@ -148,45 +148,46 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit); · 147 unmodified lines | |||
| 148 | */ | 148 | */ |
| 149 | static __always_inline __pure bool _static_cpu_has(u16 bit) | 149 | static __always_inline __pure bool _static_cpu_has(u16 bit) |
| 150 | { | 150 | { |
| 151 | asm_volatile_goto("1: jmp 6f\n" | 151 | asm_volatile_goto("1: jmp 6f\n" |
| 152 | "2:\n" | 152 | "2:\n" |
| 153 | ".skip -(((5f-4f) - (2b-1b)) > 0) * " | 153 | ".skip -(((5f-4f) - (2b-1b)) > 0) * " |
| 154 | "((5f-4f) - (2b-1b)),0x90\n" | 154 | "((5f-4f) - (2b-1b)),0x90\n" |
| 155 | "3:\n" | 155 | "3:\n" |
| 156 | ".section .altinstructions,\"a\"\n" | 156 | ".section .altinstructions,\"a\"\n" |
| 157 | " .long 1b - .\n" /* src offset */ | 157 | " .long 1b - .\n" /* src offset */ |
| 158 | " .long 4f - .\n" /* repl offset */ | 158 | " .long 4f - .\n" /* repl offset */ |
| 159 | " .word %P1\n" /* always replace */ | 159 | " .word %P[always]\n" /* always replace */ |
| 160 | " .byte 3b - 1b\n" /* src len */ | 160 | " .byte 3b - 1b\n" /* src len */ |
| 161 | " .byte 5f - 4f\n" /* repl len */ | 161 | " .byte 5f - 4f\n" /* repl len */ |
| 162 | " .byte 3b - 2b\n" /* pad len */ | 162 | " .byte 3b - 2b\n" /* pad len */ |
| 163 | ".previous\n" | 163 | ".previous\n" |
| 164 | ".section .altinstr_replacement,\"ax\"\n" | 164 | ".section .altinstr_replacement,\"ax\"\n" |
| 165 | "4: jmp %l[t_no]\n" | 165 | "4: jmp %l[t_no]\n" |
| 166 | "5:\n" | 166 | "5:\n" |
| 167 | ".previous\n" | 167 | ".previous\n" |
| 168 | ".section .altinstructions,\"a\"\n" | 168 | ".section .altinstructions,\"a\"\n" |
| 169 | " .long 1b - .\n" /* src offset */ | 169 | " .long 1b - .\n" /* src offset */ |
| 170 | " .long 0\n" /* no replacement */ | 170 | " .long 0\n" /* no replacement */ |
| 171 | " .word %P0\n" /* feature bit */ | 171 | " .word %P[feature]\n" /* feature bit */ |
| 172 | " .byte 3b - 1b\n" /* src len */ | 172 | " .byte 3b - 1b\n" /* src len */ |
| 173 | " .byte 0\n" /* repl len */ | 173 | " .byte 0\n" /* repl len */ |
| 174 | " .byte 0\n" /* pad len */ | 174 | " .byte 0\n" /* pad len */ |
| 175 | ".previous\n" | 175 | ".previous\n" |
| 176 | ".section .altinstr_aux,\"ax\"\n" | 176 | ".section .altinstr_aux,\"ax\"\n" |
| 177 | "6:\n" | 177 | "6:\n" |
| 178 | " testb %[bitnum],%[cap_byte]\n" | 178 | " testb %[bitnum],%[cap_byte]\n" |
| 179 | " jnz %l[t_yes]\n" | 179 | " jnz %l[t_yes]\n" |
| 180 | " jmp %l[t_no]\n" | 180 | " jmp %l[t_no]\n" |
| 181 | ".previous\n" | 181 | ".previous\n" |
| 182 | : : "i" (bit), "i" (X86_FEATURE_ALWAYS), | 182 | : : [feature] "i" (bit), |
| 183 | [bitnum] "i" (1 << (bit & 7)), | 183 | [always] "i" (X86_FEATURE_ALWAYS), |
| 184 | [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >> 3]) | 184 | [bitnum] "i" (1 << (bit & 7)), |
| 185 | : : t_yes, t_no); | 185 | [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >> 3]) |
| 186 | t_yes: | 186 | : : t_yes, t_no); |
| 187 | return true; | 187 | t_yes: |
| 188 | t_no: | 188 | return true; |
| 189 | return false; | 189 | t_no: |
| 190 | return false; | ||
| 190 | } | 191 | } |
| 191 | 192 | ||
| 192 | #define static_cpu_has(bit) \ | 193 | #define static_cpu_has(bit) \ |
| @@ -213,6 +213,7 @@ · 212 unmodified lines | |||
| 213 | #define X86_FEATURE_SEV ( 7*32+20) /* AMD Secure Encrypted Virtualization */ | 213 | #define X86_FEATURE_SEV ( 7*32+20) /* AMD Secure Encrypted Virtualization */ |
| 214 | 214 | ||
| 215 | #define X86_FEATURE_USE_IBPB ( 7*32+21) /* "" Indirect Branch Prediction Barrier enabled */ | 215 | #define X86_FEATURE_USE_IBPB ( 7*32+21) /* "" Indirect Branch Prediction Barrier enabled */ |
| 216 | #define X86_FEATURE_USE_IBRS_FW ( 7*32+22) /* "" Use IBRS during runtime firmware calls */ | ||
| 216 | 217 | ||
| 217 | /* Virtualization flags: Linux defined, word 8 */ | 218 | /* Virtualization flags: Linux defined, word 8 */ |
| 218 | #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */ | 219 | #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */ |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | |||
| 6 | #include <asm/pgtable.h> | 6 | #include <asm/pgtable.h> |
| 7 | #include <asm/processor-flags.h> | 7 | #include <asm/processor-flags.h> |
| 8 | #include <asm/tlb.h> | 8 | #include <asm/tlb.h> |
| 9 | #include <asm/nospec-branch.h> | ||
| 9 | 10 | ||
| 10 | /* | 11 | /* |
| 11 | * We map the EFI regions needed for runtime services non-contiguously, | 12 | * We map the EFI regions needed for runtime services non-contiguously, |
| @@ -36,8 +37,18 @@ · 24 unmodified lines | |||
| 36 | 37 | ||
| 37 | extern asmlinkage unsigned long efi_call_phys(void *, ...); | 38 | extern asmlinkage unsigned long efi_call_phys(void *, ...); |
| 38 | 39 | ||
| 39 | #define arch_efi_call_virt_setup() kernel_fpu_begin() | 40 | #define arch_efi_call_virt_setup() \ |
| 40 | #define arch_efi_call_virt_teardown() kernel_fpu_end() | 41 | ({ \ |
| 42 | kernel_fpu_begin(); \ | ||
| 43 | firmware_restrict_branch_speculation_start(); \ | ||
| 44 | }) | ||
| 45 | |||
| 46 | #define arch_efi_call_virt_teardown() \ | ||
| 47 | ({ \ | ||
| 48 | firmware_restrict_branch_speculation_end(); \ | ||
| 49 | kernel_fpu_end(); \ | ||
| 50 | }) | ||
| 51 | |||
| 41 | 52 | ||
| 42 | /* | 53 | /* |
| 43 | * Wrap all the virtual calls in a way that forces the parameters on the stack. | 54 | * Wrap all the virtual calls in a way that forces the parameters on the stack. |
| @@ -73,6 +84,7 @@ struct efi_scratch { · 29 unmodified lines | |||
| 73 | efi_sync_low_kernel_mappings(); \ | 84 | efi_sync_low_kernel_mappings(); \ |
| 74 | preempt_disable(); \ | 85 | preempt_disable(); \ |
| 75 | __kernel_fpu_begin(); \ | 86 | __kernel_fpu_begin(); \ |
| 87 | firmware_restrict_branch_speculation_start(); \ | ||
| 76 | \ | 88 | \ |
| 77 | if (efi_scratch.use_pgd) { \ | 89 | if (efi_scratch.use_pgd) { \ |
| 78 | efi_scratch.prev_cr3 = __read_cr3(); \ | 90 | efi_scratch.prev_cr3 = __read_cr3(); \ |
| @@ -91,6 +103,7 @@ struct efi_scratch { · 12 unmodified lines | |||
| 91 | __flush_tlb_all(); \ | 103 | __flush_tlb_all(); \ |
| 92 | } \ | 104 | } \ |
| 93 | \ | 105 | \ |
| 106 | firmware_restrict_branch_speculation_end(); \ | ||
| 94 | __kernel_fpu_end(); \ | 107 | __kernel_fpu_end(); \ |
| 95 | preempt_enable(); \ | 108 | preempt_enable(); \ |
| 96 | }) | 109 | }) |
| @@ -507,6 +507,7 @@ struct kvm_vcpu_arch { · 506 unmodified lines | |||
| 507 | u64 smi_count; | 507 | u64 smi_count; |
| 508 | bool tpr_access_reporting; | 508 | bool tpr_access_reporting; |
| 509 | u64 ia32_xss; | 509 | u64 ia32_xss; |
| 510 | u64 microcode_version; | ||
| 510 | 511 | ||
| 511 | /* | 512 | /* |
| 512 | * Paging state of the vcpu | 513 | * Paging state of the vcpu |
| @@ -1095,6 +1096,8 @@ struct kvm_x86_ops { · 582 unmodified lines | |||
| 1095 | int (*mem_enc_op)(struct kvm *kvm, void __user *argp); | 1096 | int (*mem_enc_op)(struct kvm *kvm, void __user *argp); |
| 1096 | int (*mem_enc_reg_region)(struct kvm *kvm, struct kvm_enc_region *argp); | 1097 | int (*mem_enc_reg_region)(struct kvm *kvm, struct kvm_enc_region *argp); |
| 1097 | int (*mem_enc_unreg_region)(struct kvm *kvm, struct kvm_enc_region *argp); | 1098 | int (*mem_enc_unreg_region)(struct kvm *kvm, struct kvm_enc_region *argp); |
| 1099 | |||
| 1100 | int (*get_msr_feature)(struct kvm_msr_entry *entry); | ||
| 1098 | }; | 1101 | }; |
| 1099 | 1102 | ||
| 1100 | struct kvm_arch_async_pf { | 1103 | struct kvm_arch_async_pf { |
| @@ -1464,7 +1467,4 @@ static inline int kvm_cpu_get_apicid(int mps_cpu) · 363 unmodified lines | |||
| 1464 | #define put_smstate(type, buf, offset, val) \ | 1467 | #define put_smstate(type, buf, offset, val) \ |
| 1465 | *(type *)((buf) + (offset) - 0x7e00) = val | 1468 | *(type *)((buf) + (offset) - 0x7e00) = val |
| 1466 | 1469 | ||
| 1467 | void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, | ||
| 1468 | unsigned long start, unsigned long end); | ||
| 1469 | |||
| 1470 | #endif /* _ASM_X86_KVM_HOST_H */ | 1470 | #endif /* _ASM_X86_KVM_HOST_H */ |
| @@ -37,7 +37,12 @@ struct cpu_signature { · 36 unmodified lines | |||
| 37 | 37 | ||
| 38 | struct device; | 38 | struct device; |
| 39 | 39 | ||
| 40 | enum ucode_state { UCODE_ERROR, UCODE_OK, UCODE_NFOUND }; | 40 | enum ucode_state { |
| 41 | UCODE_OK = 0, | ||
| 42 | UCODE_UPDATED, | ||
| 43 | UCODE_NFOUND, | ||
| 44 | UCODE_ERROR, | ||
| 45 | }; | ||
| 41 | 46 | ||
| 42 | struct microcode_ops { | 47 | struct microcode_ops { |
| 43 | enum ucode_state (*request_microcode_user) (int cpu, | 48 | enum ucode_state (*request_microcode_user) (int cpu, |
| @@ -54,7 +59,7 @@ struct microcode_ops { · 10 unmodified lines | |||
| 54 | * are being called. | 59 | * are being called. |
| 55 | * See also the "Synchronization" section in microcode_core.c. | 60 | * See also the "Synchronization" section in microcode_core.c. |
| 56 | */ | 61 | */ |
| 57 | int (*apply_microcode) (int cpu); | 62 | enum ucode_state (*apply_microcode) (int cpu); |
| 58 | int (*collect_cpu_info) (int cpu, struct cpu_signature *csig); | 63 | int (*collect_cpu_info) (int cpu, struct cpu_signature *csig); |
| 59 | }; | 64 | }; |
| 60 | 65 | ||
| @@ -74,6 +74,7 @@ static inline void *ldt_slot_va(int slot) · 73 unmodified lines | |||
| 74 | return (void *)(LDT_BASE_ADDR + LDT_SLOT_STRIDE * slot); | 74 | return (void *)(LDT_BASE_ADDR + LDT_SLOT_STRIDE * slot); |
| 75 | #else | 75 | #else |
| 76 | BUG(); | 76 | BUG(); |
| 77 | return (void *)fix_to_virt(FIX_HOLE); | ||
| 77 | #endif | 78 | #endif |
| 78 | } | 79 | } |
| 79 | 80 | ||
| @@ -6,6 +6,51 @@ · 5 unmodified lines | |||
| 6 | #include <asm/alternative.h> | 6 | #include <asm/alternative.h> |
| 7 | #include <asm/alternative-asm.h> | 7 | #include <asm/alternative-asm.h> |
| 8 | #include <asm/cpufeatures.h> | 8 | #include <asm/cpufeatures.h> |
| 9 | #include <asm/msr-index.h> | ||
| 10 | |||
| 11 | /* | ||
| 12 | * Fill the CPU return stack buffer. | ||
| 13 | * | ||
| 14 | * Each entry in the RSB, if used for a speculative 'ret', contains an | ||
| 15 | * infinite 'pause; lfence; jmp' loop to capture speculative execution. | ||
| 16 | * | ||
| 17 | * This is required in various cases for retpoline and IBRS-based | ||
| 18 | * mitigations for the Spectre variant 2 vulnerability. Sometimes to | ||
| 19 | * eliminate potentially bogus entries from the RSB, and sometimes | ||
| 20 | * purely to ensure that it doesn't get empty, which on some CPUs would | ||
| 21 | * allow predictions from other (unwanted!) sources to be used. | ||
| 22 | * | ||
| 23 | * We define a CPP macro such that it can be used from both .S files and | ||
| 24 | * inline assembly. It's possible to do a .macro and then include that | ||
| 25 | * from C via asm(".include <asm/nospec-branch.h>") but let's not go there. | ||
| 26 | */ | ||
| 27 | |||
| 28 | #define RSB_CLEAR_LOOPS 32 /* To forcibly overwrite all entries */ | ||
| 29 | #define RSB_FILL_LOOPS 16 /* To avoid underflow */ | ||
| 30 | |||
| 31 | /* | ||
| 32 | * Google experimented with loop-unrolling and this turned out to be | ||
| 33 | * the optimal version — two calls, each with their own speculation | ||
| 34 | * trap should their return address end up getting used, in a loop. | ||
| 35 | */ | ||
| 36 | #define __FILL_RETURN_BUFFER(reg, nr, sp) \ | ||
| 37 | mov $(nr/2), reg; \ | ||
| 38 | 771: \ | ||
| 39 | call 772f; \ | ||
| 40 | 773: /* speculation trap */ \ | ||
| 41 | pause; \ | ||
| 42 | lfence; \ | ||
| 43 | jmp 773b; \ | ||
| 44 | 772: \ | ||
| 45 | call 774f; \ | ||
| 46 | 775: /* speculation trap */ \ | ||
| 47 | pause; \ | ||
| 48 | lfence; \ | ||
| 49 | jmp 775b; \ | ||
| 50 | 774: \ | ||
| 51 | dec reg; \ | ||
| 52 | jnz 771b; \ | ||
| 53 | add $(BITS_PER_LONG/8) * nr, sp; | ||
| 9 | 54 | ||
| 10 | #ifdef __ASSEMBLY__ | 55 | #ifdef __ASSEMBLY__ |
| 11 | 56 | ||
| @@ -22,6 +67,18 @@ · 10 unmodified lines | |||
| 22 | .popsection | 67 | .popsection |
| 23 | .endm | 68 | .endm |
| 24 | 69 | ||
| 70 | /* | ||
| 71 | * This should be used immediately before an indirect jump/call. It tells | ||
| 72 | * objtool the subsequent indirect jump/call is vouched safe for retpoline | ||
| 73 | * builds. | ||
| 74 | */ | ||
| 75 | .macro ANNOTATE_RETPOLINE_SAFE | ||
| 76 | .Lannotate_\@: | ||
| 77 | .pushsection .discard.retpoline_safe | ||
| 78 | _ASM_PTR .Lannotate_\@ | ||
| 79 | .popsection | ||
| 80 | .endm | ||
| 81 | |||
| 25 | /* | 82 | /* |
| 26 | * These are the bare retpoline primitives for indirect jmp and call. | 83 | * These are the bare retpoline primitives for indirect jmp and call. |
| 27 | * Do not use these directly; they only exist to make the ALTERNATIVE | 84 | * Do not use these directly; they only exist to make the ALTERNATIVE |
| @@ -58,9 +115,9 @@ · 30 unmodified lines | |||
| 58 | .macro JMP_NOSPEC reg:req | 115 | .macro JMP_NOSPEC reg:req |
| 59 | #ifdef CONFIG_RETPOLINE | 116 | #ifdef CONFIG_RETPOLINE |
| 60 | ANNOTATE_NOSPEC_ALTERNATIVE | 117 | ANNOTATE_NOSPEC_ALTERNATIVE |
| 61 | ALTERNATIVE_2 __stringify(jmp *\reg), \ | 118 | ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *\reg), \ |
| 62 | __stringify(RETPOLINE_JMP \reg), X86_FEATURE_RETPOLINE, \ | 119 | __stringify(RETPOLINE_JMP \reg), X86_FEATURE_RETPOLINE, \ |
| 63 | __stringify(lfence; jmp *\reg), X86_FEATURE_RETPOLINE_AMD | 120 | __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *\reg), X86_FEATURE_RETPOLINE_AMD |
| 64 | #else | 121 | #else |
| 65 | jmp *\reg | 122 | jmp *\reg |
| 66 | #endif | 123 | #endif |
| @@ -69,18 +126,25 @@ · 2 unmodified lines | |||
| 69 | .macro CALL_NOSPEC reg:req | 126 | .macro CALL_NOSPEC reg:req |
| 70 | #ifdef CONFIG_RETPOLINE | 127 | #ifdef CONFIG_RETPOLINE |
| 71 | ANNOTATE_NOSPEC_ALTERNATIVE | 128 | ANNOTATE_NOSPEC_ALTERNATIVE |
| 72 | ALTERNATIVE_2 __stringify(call *\reg), \ | 129 | ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; call *\reg), \ |
| 73 | __stringify(RETPOLINE_CALL \reg), X86_FEATURE_RETPOLINE,\ | 130 | __stringify(RETPOLINE_CALL \reg), X86_FEATURE_RETPOLINE,\ |
| 74 | __stringify(lfence; call *\reg), X86_FEATURE_RETPOLINE_AMD | 131 | __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; call *\reg), X86_FEATURE_RETPOLINE_AMD |
| 75 | #else | 132 | #else |
| 76 | call *\reg | 133 | call *\reg |
| 77 | #endif | 134 | #endif |
| 78 | .endm | 135 | .endm |
| 79 | 136 | ||
| 80 | /* This clobbers the BX register */ | 137 | /* |
| 81 | .macro FILL_RETURN_BUFFER nr:req ftr:req | 138 | * A simpler FILL_RETURN_BUFFER macro. Don't make people use the CPP |
| 139 | * monstrosity above, manually. | ||
| 140 | */ | ||
| 141 | .macro FILL_RETURN_BUFFER reg:req nr:req ftr:req | ||
| 82 | #ifdef CONFIG_RETPOLINE | 142 | #ifdef CONFIG_RETPOLINE |
| 83 | ALTERNATIVE "", "call __clear_rsb", \ftr | 143 | ANNOTATE_NOSPEC_ALTERNATIVE |
| 144 | ALTERNATIVE "jmp .Lskip_rsb_\@", \ | ||
| 145 | __stringify(__FILL_RETURN_BUFFER(\reg,\nr,%_ASM_SP)) \ | ||
| 146 | \ftr | ||
| 147 | .Lskip_rsb_\@: | ||
| 84 | #endif | 148 | #endif |
| 85 | .endm | 149 | .endm |
| 86 | 150 | ||
| @@ -92,6 +156,12 @@ · 5 unmodified lines | |||
| 92 | ".long 999b - .\n\t" \ | 156 | ".long 999b - .\n\t" \ |
| 93 | ".popsection\n\t" | 157 | ".popsection\n\t" |
| 94 | 158 | ||
| 159 | #define ANNOTATE_RETPOLINE_SAFE \ | ||
| 160 | "999:\n\t" \ | ||
| 161 | ".pushsection .discard.retpoline_safe\n\t" \ | ||
| 162 | _ASM_PTR " 999b\n\t" \ | ||
| 163 | ".popsection\n\t" | ||
| 164 | |||
| 95 | #if defined(CONFIG_X86_64) && defined(RETPOLINE) | 165 | #if defined(CONFIG_X86_64) && defined(RETPOLINE) |
| 96 | 166 | ||
| 97 | /* | 167 | /* |
| @@ -101,6 +171,7 @@ · 3 unmodified lines | |||
| 101 | # define CALL_NOSPEC \ | 171 | # define CALL_NOSPEC \ |
| 102 | ANNOTATE_NOSPEC_ALTERNATIVE \ | 172 | ANNOTATE_NOSPEC_ALTERNATIVE \ |
| 103 | ALTERNATIVE( \ | 173 | ALTERNATIVE( \ |
| 174 | ANNOTATE_RETPOLINE_SAFE \ | ||
| 104 | "call *%[thunk_target]\n", \ | 175 | "call *%[thunk_target]\n", \ |
| 105 | "call __x86_indirect_thunk_%V[thunk_target]\n", \ | 176 | "call __x86_indirect_thunk_%V[thunk_target]\n", \ |
| 106 | X86_FEATURE_RETPOLINE) | 177 | X86_FEATURE_RETPOLINE) |
| @@ -155,20 +226,90 @@ extern char __indirect_thunk_end[]; · 48 unmodified lines | |||
| 155 | static inline void vmexit_fill_RSB(void) | 226 | static inline void vmexit_fill_RSB(void) |
| 156 | { | 227 | { |
| 157 | #ifdef CONFIG_RETPOLINE | 228 | #ifdef CONFIG_RETPOLINE |
| 158 | alternative_input("", | 229 | unsigned long loops; |
| 159 | "call __fill_rsb", | 230 | |
| 160 | X86_FEATURE_RETPOLINE, | 231 | asm volatile (ANNOTATE_NOSPEC_ALTERNATIVE |
| 161 | ASM_NO_INPUT_CLOBBER(_ASM_BX, "memory")); | 232 | ALTERNATIVE("jmp 910f", |
| 233 | __stringify(__FILL_RETURN_BUFFER(%0, RSB_CLEAR_LOOPS, %1)), | ||
| 234 | X86_FEATURE_RETPOLINE) | ||
| 235 | "910:" | ||
| 236 | : "=r" (loops), ASM_CALL_CONSTRAINT | ||
| 237 | : : "memory" ); | ||
| 162 | #endif | 238 | #endif |
| 163 | } | 239 | } |
| 164 | 240 | ||
| 241 | #define alternative_msr_write(_msr, _val, _feature) \ | ||
| 242 | asm volatile(ALTERNATIVE("", \ | ||
| 243 | "movl %[msr], %%ecx\n\t" \ | ||
| 244 | "movl %[val], %%eax\n\t" \ | ||
| 245 | "movl $0, %%edx\n\t" \ | ||
| 246 | "wrmsr", \ | ||
| 247 | _feature) \ | ||
| 248 | : : [msr] "i" (_msr), [val] "i" (_val) \ | ||
| 249 | : "eax", "ecx", "edx", "memory") | ||
| 250 | |||
| 165 | static inline void indirect_branch_prediction_barrier(void) | 251 | static inline void indirect_branch_prediction_barrier(void) |
| 166 | { | 252 | { |
| 167 | alternative_input("", | 253 | alternative_msr_write(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, |
| 168 | "call __ibp_barrier", | 254 | X86_FEATURE_USE_IBPB); |
| 169 | X86_FEATURE_USE_IBPB, | ||
| 170 | ASM_NO_INPUT_CLOBBER("eax", "ecx", "edx", "memory")); | ||
| 171 | } | 255 | } |
| 172 | 256 | ||
| 257 | /* | ||
| 258 | * With retpoline, we must use IBRS to restrict branch prediction | ||
| 259 | * before calling into firmware. | ||
| 260 | * | ||
| 261 | * (Implemented as CPP macros due to header hell.) | ||
| 262 | */ | ||
| 263 | #define firmware_restrict_branch_speculation_start() \ | ||
| 264 | do { \ | ||
| 265 | preempt_disable(); \ | ||
| 266 | alternative_msr_write(MSR_IA32_SPEC_CTRL, SPEC_CTRL_IBRS, \ | ||
| 267 | X86_FEATURE_USE_IBRS_FW); \ | ||
| 268 | } while (0) | ||
| 269 | |||
| 270 | #define firmware_restrict_branch_speculation_end() \ | ||
| 271 | do { \ | ||
| 272 | alternative_msr_write(MSR_IA32_SPEC_CTRL, 0, \ | ||
| 273 | X86_FEATURE_USE_IBRS_FW); \ | ||
| 274 | preempt_enable(); \ | ||
| 275 | } while (0) | ||
| 276 | |||
| 173 | #endif /* __ASSEMBLY__ */ | 277 | #endif /* __ASSEMBLY__ */ |
| 278 | |||
| 279 | /* | ||
| 280 | * Below is used in the eBPF JIT compiler and emits the byte sequence | ||
| 281 | * for the following assembly: | ||
| 282 | * | ||
| 283 | * With retpolines configured: | ||
| 284 | * | ||
| 285 | * callq do_rop | ||
| 286 | * spec_trap: | ||
| 287 | * pause | ||
| 288 | * lfence | ||
| 289 | * jmp spec_trap | ||
| 290 | * do_rop: | ||
| 291 | * mov %rax,(%rsp) | ||
| 292 | * retq | ||
| 293 | * | ||
| 294 | * Without retpolines configured: | ||
| 295 | * | ||
| 296 | * jmp *%rax | ||
| 297 | */ | ||
| 298 | #ifdef CONFIG_RETPOLINE | ||
| 299 | # define RETPOLINE_RAX_BPF_JIT_SIZE 17 | ||
| 300 | # define RETPOLINE_RAX_BPF_JIT() \ | ||
| 301 | EMIT1_off32(0xE8, 7); /* callq do_rop */ \ | ||
| 302 | /* spec_trap: */ \ | ||
| 303 | EMIT2(0xF3, 0x90); /* pause */ \ | ||
| 304 | EMIT3(0x0F, 0xAE, 0xE8); /* lfence */ \ | ||
| 305 | EMIT2(0xEB, 0xF9); /* jmp spec_trap */ \ | ||
| 306 | /* do_rop: */ \ | ||
| 307 | EMIT4(0x48, 0x89, 0x04, 0x24); /* mov %rax,(%rsp) */ \ | ||
| 308 | EMIT1(0xC3); /* retq */ | ||
| 309 | #else | ||
| 310 | # define RETPOLINE_RAX_BPF_JIT_SIZE 2 | ||
| 311 | # define RETPOLINE_RAX_BPF_JIT() \ | ||
| 312 | EMIT2(0xFF, 0xE0); /* jmp *%rax */ | ||
| 313 | #endif | ||
| 314 | |||
| 174 | #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ | 315 | #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ |
| @@ -52,10 +52,6 @@ static inline void clear_page(void *page) · 51 unmodified lines | |||
| 52 | 52 | ||
| 53 | void copy_page(void *to, void *from); | 53 | void copy_page(void *to, void *from); |
| 54 | 54 | ||
| 55 | #ifdef CONFIG_X86_MCE | ||
| 56 | #define arch_unmap_kpfn arch_unmap_kpfn | ||
| 57 | #endif | ||
| 58 | |||
| 59 | #endif /* !__ASSEMBLY__ */ | 55 | #endif /* !__ASSEMBLY__ */ |
| 60 | 56 | ||
| 61 | #ifdef CONFIG_X86_VSYSCALL_EMULATION | 57 | #ifdef CONFIG_X86_VSYSCALL_EMULATION |
| @@ -7,6 +7,7 @@ · 6 unmodified lines | |||
| 7 | #ifdef CONFIG_PARAVIRT | 7 | #ifdef CONFIG_PARAVIRT |
| 8 | #include <asm/pgtable_types.h> | 8 | #include <asm/pgtable_types.h> |
| 9 | #include <asm/asm.h> | 9 | #include <asm/asm.h> |
| 10 | #include <asm/nospec-branch.h> | ||
| 10 | 11 | ||
| 11 | #include <asm/paravirt_types.h> | 12 | #include <asm/paravirt_types.h> |
| 12 | 13 | ||
| @@ -297,9 +298,9 @@ static inline void __flush_tlb_global(void) · 284 unmodified lines | |||
| 297 | { | 298 | { |
| 298 | PVOP_VCALL0(pv_mmu_ops.flush_tlb_kernel); | 299 | PVOP_VCALL0(pv_mmu_ops.flush_tlb_kernel); |
| 299 | } | 300 | } |
| 300 | static inline void __flush_tlb_single(unsigned long addr) | 301 | static inline void __flush_tlb_one_user(unsigned long addr) |
| 301 | { | 302 | { |
| 302 | PVOP_VCALL1(pv_mmu_ops.flush_tlb_single, addr); | 303 | PVOP_VCALL1(pv_mmu_ops.flush_tlb_one_user, addr); |
| 303 | } | 304 | } |
| 304 | 305 | ||
| 305 | static inline void flush_tlb_others(const struct cpumask *cpumask, | 306 | static inline void flush_tlb_others(const struct cpumask *cpumask, |
| @@ -879,23 +880,27 @@ extern void default_banner(void); · 573 unmodified lines | |||
| 879 | 880 | ||
| 880 | #define INTERRUPT_RETURN \ | 881 | #define INTERRUPT_RETURN \ |
| 881 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_iret), CLBR_NONE, \ | 882 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_iret), CLBR_NONE, \ |
| 882 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_iret)) | 883 | ANNOTATE_RETPOLINE_SAFE; \ |
| 884 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_iret);) | ||
| 883 | 885 | ||
| 884 | #define DISABLE_INTERRUPTS(clobbers) \ | 886 | #define DISABLE_INTERRUPTS(clobbers) \ |
| 885 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_disable), clobbers, \ | 887 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_disable), clobbers, \ |
| 886 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ | 888 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ |
| 889 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 887 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_disable); \ | 890 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_disable); \ |
| 888 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) | 891 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) |
| 889 | 892 | ||
| 890 | #define ENABLE_INTERRUPTS(clobbers) \ | 893 | #define ENABLE_INTERRUPTS(clobbers) \ |
| 891 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_enable), clobbers, \ | 894 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_irq_enable), clobbers, \ |
| 892 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ | 895 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ |
| 896 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 893 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_enable); \ | 897 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_irq_enable); \ |
| 894 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) | 898 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) |
| 895 | 899 | ||
| 896 | #ifdef CONFIG_X86_32 | 900 | #ifdef CONFIG_X86_32 |
| 897 | #define GET_CR0_INTO_EAX \ | 901 | #define GET_CR0_INTO_EAX \ |
| 898 | push %ecx; push %edx; \ | 902 | push %ecx; push %edx; \ |
| 903 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 899 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_read_cr0); \ | 904 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_read_cr0); \ |
| 900 | pop %edx; pop %ecx | 905 | pop %edx; pop %ecx |
| 901 | #else /* !CONFIG_X86_32 */ | 906 | #else /* !CONFIG_X86_32 */ |
| @@ -917,21 +922,25 @@ extern void default_banner(void); · 15 unmodified lines | |||
| 917 | */ | 922 | */ |
| 918 | #define SWAPGS \ | 923 | #define SWAPGS \ |
| 919 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_swapgs), CLBR_NONE, \ | 924 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_swapgs), CLBR_NONE, \ |
| 920 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_swapgs) \ | 925 | ANNOTATE_RETPOLINE_SAFE; \ |
| 926 | call PARA_INDIRECT(pv_cpu_ops+PV_CPU_swapgs); \ | ||
| 921 | ) | 927 | ) |
| 922 | 928 | ||
| 923 | #define GET_CR2_INTO_RAX \ | 929 | #define GET_CR2_INTO_RAX \ |
| 924 | call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2) | 930 | ANNOTATE_RETPOLINE_SAFE; \ |
| 931 | call PARA_INDIRECT(pv_mmu_ops+PV_MMU_read_cr2); | ||
| 925 | 932 | ||
| 926 | #define USERGS_SYSRET64 \ | 933 | #define USERGS_SYSRET64 \ |
| 927 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_usergs_sysret64), \ | 934 | PARA_SITE(PARA_PATCH(pv_cpu_ops, PV_CPU_usergs_sysret64), \ |
| 928 | CLBR_NONE, \ | 935 | CLBR_NONE, \ |
| 929 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_usergs_sysret64)) | 936 | ANNOTATE_RETPOLINE_SAFE; \ |
| 937 | jmp PARA_INDIRECT(pv_cpu_ops+PV_CPU_usergs_sysret64);) | ||
| 930 | 938 | ||
| 931 | #ifdef CONFIG_DEBUG_ENTRY | 939 | #ifdef CONFIG_DEBUG_ENTRY |
| 932 | #define SAVE_FLAGS(clobbers) \ | 940 | #define SAVE_FLAGS(clobbers) \ |
| 933 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_save_fl), clobbers, \ | 941 | PARA_SITE(PARA_PATCH(pv_irq_ops, PV_IRQ_save_fl), clobbers, \ |
| 934 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ | 942 | PV_SAVE_REGS(clobbers | CLBR_CALLEE_SAVE); \ |
| 943 | ANNOTATE_RETPOLINE_SAFE; \ | ||
| 935 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_save_fl); \ | 944 | call PARA_INDIRECT(pv_irq_ops+PV_IRQ_save_fl); \ |
| 936 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) | 945 | PV_RESTORE_REGS(clobbers | CLBR_CALLEE_SAVE);) |
| 937 | #endif | 946 | #endif |
| @@ -43,6 +43,7 @@ · 42 unmodified lines | |||
| 43 | #include <asm/desc_defs.h> | 43 | #include <asm/desc_defs.h> |
| 44 | #include <asm/kmap_types.h> | 44 | #include <asm/kmap_types.h> |
| 45 | #include <asm/pgtable_types.h> | 45 | #include <asm/pgtable_types.h> |
| 46 | #include <asm/nospec-branch.h> | ||
| 46 | 47 | ||
| 47 | struct page; | 48 | struct page; |
| 48 | struct thread_struct; | 49 | struct thread_struct; |
| @@ -217,7 +218,7 @@ struct pv_mmu_ops { · 168 unmodified lines | |||
| 217 | /* TLB operations */ | 218 | /* TLB operations */ |
| 218 | void (*flush_tlb_user)(void); | 219 | void (*flush_tlb_user)(void); |
| 219 | void (*flush_tlb_kernel)(void); | 220 | void (*flush_tlb_kernel)(void); |
| 220 | void (*flush_tlb_single)(unsigned long addr); | 221 | void (*flush_tlb_one_user)(unsigned long addr); |
| 221 | void (*flush_tlb_others)(const struct cpumask *cpus, | 222 | void (*flush_tlb_others)(const struct cpumask *cpus, |
| 222 | const struct flush_tlb_info *info); | 223 | const struct flush_tlb_info *info); |
| 223 | 224 | ||
| @@ -392,7 +393,9 @@ int paravirt_disable_iospace(void); · 168 unmodified lines | |||
| 392 | * offset into the paravirt_patch_template structure, and can therefore be | 393 | * offset into the paravirt_patch_template structure, and can therefore be |
| 393 | * freely converted back into a structure offset. | 394 | * freely converted back into a structure offset. |
| 394 | */ | 395 | */ |
| 395 | #define PARAVIRT_CALL "call *%c[paravirt_opptr];" | 396 | #define PARAVIRT_CALL \ |
| 397 | ANNOTATE_RETPOLINE_SAFE \ | ||
| 398 | "call *%c[paravirt_opptr];" | ||
| 396 | 399 | ||
| 397 | /* | 400 | /* |
| 398 | * These macros are intended to wrap calls through one of the paravirt | 401 | * These macros are intended to wrap calls through one of the paravirt |
| @@ -526,7 +526,7 @@ static inline bool x86_this_cpu_variable_test_bit(int nr, · 525 unmodified lines | |||
| 526 | { | 526 | { |
| 527 | bool oldbit; | 527 | bool oldbit; |
| 528 | 528 | ||
| 529 | asm volatile("bt "__percpu_arg(2)",%1" | 529 | asm volatile("btl "__percpu_arg(2)",%1" |
| 530 | CC_SET(c) | 530 | CC_SET(c) |
| 531 | : CC_OUT(c) (oldbit) | 531 | : CC_OUT(c) (oldbit) |
| 532 | : "m" (*(unsigned long __percpu *)addr), "Ir" (nr)); | 532 | : "m" (*(unsigned long __percpu *)addr), "Ir" (nr)); |
| @@ -350,14 +350,14 @@ static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) · 349 unmodified lines | |||
| 350 | { | 350 | { |
| 351 | pmdval_t v = native_pmd_val(pmd); | 351 | pmdval_t v = native_pmd_val(pmd); |
| 352 | 352 | ||
| 353 | return __pmd(v | set); | 353 | return native_make_pmd(v | set); |
| 354 | } | 354 | } |
| 355 | 355 | ||
| 356 | static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) | 356 | static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) |
| 357 | { | 357 | { |
| 358 | pmdval_t v = native_pmd_val(pmd); | 358 | pmdval_t v = native_pmd_val(pmd); |
| 359 | 359 | ||
| 360 | return __pmd(v & ~clear); | 360 | return native_make_pmd(v & ~clear); |
| 361 | } | 361 | } |
| 362 | 362 | ||
| 363 | static inline pmd_t pmd_mkold(pmd_t pmd) | 363 | static inline pmd_t pmd_mkold(pmd_t pmd) |
| @@ -409,14 +409,14 @@ static inline pud_t pud_set_flags(pud_t pud, pudval_t set) · 45 unmodified lines | |||
| 409 | { | 409 | { |
| 410 | pudval_t v = native_pud_val(pud); | 410 | pudval_t v = native_pud_val(pud); |
| 411 | 411 | ||
| 412 | return __pud(v | set); | 412 | return native_make_pud(v | set); |
| 413 | } | 413 | } |
| 414 | 414 | ||
| 415 | static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) | 415 | static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) |
| 416 | { | 416 | { |
| 417 | pudval_t v = native_pud_val(pud); | 417 | pudval_t v = native_pud_val(pud); |
| 418 | 418 | ||
| 419 | return __pud(v & ~clear); | 419 | return native_make_pud(v & ~clear); |
| 420 | } | 420 | } |
| 421 | 421 | ||
| 422 | static inline pud_t pud_mkold(pud_t pud) | 422 | static inline pud_t pud_mkold(pud_t pud) |
| @@ -32,6 +32,7 @@ extern pmd_t initial_pg_pmd[]; · 31 unmodified lines | |||
| 32 | static inline void pgtable_cache_init(void) { } | 32 | static inline void pgtable_cache_init(void) { } |
| 33 | static inline void check_pgt_cache(void) { } | 33 | static inline void check_pgt_cache(void) { } |
| 34 | void paging_init(void); | 34 | void paging_init(void); |
| 35 | void sync_initial_page_table(void); | ||
| 35 | 36 | ||
| 36 | /* | 37 | /* |
| 37 | * Define this if things work differently on an i386 and an i486: | 38 | * Define this if things work differently on an i386 and an i486: |
| @@ -61,7 +62,7 @@ void paging_init(void); · 23 unmodified lines | |||
| 61 | #define kpte_clear_flush(ptep, vaddr) \ | 62 | #define kpte_clear_flush(ptep, vaddr) \ |
| 62 | do { \ | 63 | do { \ |
| 63 | pte_clear(&init_mm, (vaddr), (ptep)); \ | 64 | pte_clear(&init_mm, (vaddr), (ptep)); \ |
| 64 | __flush_tlb_one((vaddr)); \ | 65 | __flush_tlb_one_kernel((vaddr)); \ |
| 65 | } while (0) | 66 | } while (0) |
| 66 | 67 | ||
| 67 | #endif /* !__ASSEMBLY__ */ | 68 | #endif /* !__ASSEMBLY__ */ |
| @@ -28,6 +28,7 @@ extern pgd_t init_top_pgt[]; · 27 unmodified lines | |||
| 28 | #define swapper_pg_dir init_top_pgt | 28 | #define swapper_pg_dir init_top_pgt |
| 29 | 29 | ||
| 30 | extern void paging_init(void); | 30 | extern void paging_init(void); |
| 31 | static inline void sync_initial_page_table(void) { } | ||
| 31 | 32 | ||
| 32 | #define pte_ERROR(e) \ | 33 | #define pte_ERROR(e) \ |
| 33 | pr_err("%s:%d: bad pte %p(%016lx)\n", \ | 34 | pr_err("%s:%d: bad pte %p(%016lx)\n", \ |
| @@ -174,7 +174,6 @@ enum page_cache_mode { · 173 unmodified lines | |||
| 174 | #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) | 174 | #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) |
| 175 | #define __PAGE_KERNEL_RX (__PAGE_KERNEL_EXEC & ~_PAGE_RW) | 175 | #define __PAGE_KERNEL_RX (__PAGE_KERNEL_EXEC & ~_PAGE_RW) |
| 176 | #define __PAGE_KERNEL_NOCACHE (__PAGE_KERNEL | _PAGE_NOCACHE) | 176 | #define __PAGE_KERNEL_NOCACHE (__PAGE_KERNEL | _PAGE_NOCACHE) |
| 177 | #define __PAGE_KERNEL_VSYSCALL (__PAGE_KERNEL_RX | _PAGE_USER) | ||
| 178 | #define __PAGE_KERNEL_VVAR (__PAGE_KERNEL_RO | _PAGE_USER) | 177 | #define __PAGE_KERNEL_VVAR (__PAGE_KERNEL_RO | _PAGE_USER) |
| 179 | #define __PAGE_KERNEL_LARGE (__PAGE_KERNEL | _PAGE_PSE) | 178 | #define __PAGE_KERNEL_LARGE (__PAGE_KERNEL | _PAGE_PSE) |
| 180 | #define __PAGE_KERNEL_LARGE_EXEC (__PAGE_KERNEL_EXEC | _PAGE_PSE) | 179 | #define __PAGE_KERNEL_LARGE_EXEC (__PAGE_KERNEL_EXEC | _PAGE_PSE) |
| @@ -206,7 +205,6 @@ enum page_cache_mode { · 25 unmodified lines | |||
| 206 | #define PAGE_KERNEL_NOCACHE __pgprot(__PAGE_KERNEL_NOCACHE | _PAGE_ENC) | 205 | #define PAGE_KERNEL_NOCACHE __pgprot(__PAGE_KERNEL_NOCACHE | _PAGE_ENC) |
| 207 | #define PAGE_KERNEL_LARGE __pgprot(__PAGE_KERNEL_LARGE | _PAGE_ENC) | 206 | #define PAGE_KERNEL_LARGE __pgprot(__PAGE_KERNEL_LARGE | _PAGE_ENC) |
| 208 | #define PAGE_KERNEL_LARGE_EXEC __pgprot(__PAGE_KERNEL_LARGE_EXEC | _PAGE_ENC) | 207 | #define PAGE_KERNEL_LARGE_EXEC __pgprot(__PAGE_KERNEL_LARGE_EXEC | _PAGE_ENC) |
| 209 | #define PAGE_KERNEL_VSYSCALL __pgprot(__PAGE_KERNEL_VSYSCALL | _PAGE_ENC) | ||
| 210 | #define PAGE_KERNEL_VVAR __pgprot(__PAGE_KERNEL_VVAR | _PAGE_ENC) | 208 | #define PAGE_KERNEL_VVAR __pgprot(__PAGE_KERNEL_VVAR | _PAGE_ENC) |
| 211 | 209 | ||
| 212 | #define PAGE_KERNEL_IO __pgprot(__PAGE_KERNEL_IO) | 210 | #define PAGE_KERNEL_IO __pgprot(__PAGE_KERNEL_IO) |
| @@ -323,6 +321,11 @@ static inline pudval_t native_pud_val(pud_t pud) · 110 unmodified lines | |||
| 323 | #else | 321 | #else |
| 324 | #include <asm-generic/pgtable-nopud.h> | 322 | #include <asm-generic/pgtable-nopud.h> |
| 325 | 323 | ||
| 324 | static inline pud_t native_make_pud(pudval_t val) | ||
| 325 | { | ||
| 326 | return (pud_t) { .p4d.pgd = native_make_pgd(val) }; | ||
| 327 | } | ||
| 328 | |||
| 326 | static inline pudval_t native_pud_val(pud_t pud) | 329 | static inline pudval_t native_pud_val(pud_t pud) |
| 327 | { | 330 | { |
| 328 | return native_pgd_val(pud.p4d.pgd); | 331 | return native_pgd_val(pud.p4d.pgd); |
| @@ -344,6 +347,11 @@ static inline pmdval_t native_pmd_val(pmd_t pmd) · 15 unmodified lines | |||
| 344 | #else | 347 | #else |
| 345 | #include <asm-generic/pgtable-nopmd.h> | 348 | #include <asm-generic/pgtable-nopmd.h> |
| 346 | 349 | ||
| 350 | static inline pmd_t native_make_pmd(pmdval_t val) | ||
| 351 | { | ||
| 352 | return (pmd_t) { .pud.p4d.pgd = native_make_pgd(val) }; | ||
| 353 | } | ||
| 354 | |||
| 347 | static inline pmdval_t native_pmd_val(pmd_t pmd) | 355 | static inline pmdval_t native_pmd_val(pmd_t pmd) |
| 348 | { | 356 | { |
| 349 | return native_pgd_val(pmd.pud.p4d.pgd); | 357 | return native_pgd_val(pmd.pud.p4d.pgd); |
| @@ -91,7 +91,7 @@ struct cpuinfo_x86 { · 90 unmodified lines | |||
| 91 | __u8 x86; /* CPU family */ | 91 | __u8 x86; /* CPU family */ |
| 92 | __u8 x86_vendor; /* CPU vendor */ | 92 | __u8 x86_vendor; /* CPU vendor */ |
| 93 | __u8 x86_model; | 93 | __u8 x86_model; |
| 94 | __u8 x86_mask; | 94 | __u8 x86_stepping; |
| 95 | #ifdef CONFIG_X86_64 | 95 | #ifdef CONFIG_X86_64 |
| 96 | /* Number of 4K pages in DTLB/ITLB combined(in pages): */ | 96 | /* Number of 4K pages in DTLB/ITLB combined(in pages): */ |
| 97 | int x86_tlbsize; | 97 | int x86_tlbsize; |
| @@ -109,7 +109,7 @@ struct cpuinfo_x86 { · 11 unmodified lines | |||
| 109 | char x86_vendor_id[16]; | 109 | char x86_vendor_id[16]; |
| 110 | char x86_model_id[64]; | 110 | char x86_model_id[64]; |
| 111 | /* in KB - valid for CPUS which support this call: */ | 111 | /* in KB - valid for CPUS which support this call: */ |
| 112 | int x86_cache_size; | 112 | unsigned int x86_cache_size; |
| 113 | int x86_cache_alignment; /* In bytes */ | 113 | int x86_cache_alignment; /* In bytes */ |
| 114 | /* Cache QoS architectural values: */ | 114 | /* Cache QoS architectural values: */ |
| 115 | int x86_cache_max_rmid; /* max index */ | 115 | int x86_cache_max_rmid; /* max index */ |
| @@ -977,7 +977,5 @@ bool xen_set_default_idle(void); · 861 unmodified lines | |||
| 977 | 977 | ||
| 978 | void stop_this_cpu(void *dummy); | 978 | void stop_this_cpu(void *dummy); |
| 979 | void df_debug(struct pt_regs *regs, long error_code); | 979 | void df_debug(struct pt_regs *regs, long error_code); |
| 980 | 980 | void microcode_check(void); | |
| 981 | void __ibp_barrier(void); | ||
| 982 | |||
| 983 | #endif /* _ASM_X86_PROCESSOR_H */ | 981 | #endif /* _ASM_X86_PROCESSOR_H */ |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | |||
| 17 | #define _REFCOUNT_EXCEPTION \ | 17 | #define _REFCOUNT_EXCEPTION \ |
| 18 | ".pushsection .text..refcount\n" \ | 18 | ".pushsection .text..refcount\n" \ |
| 19 | "111:\tlea %[counter], %%" _ASM_CX "\n" \ | 19 | "111:\tlea %[counter], %%" _ASM_CX "\n" \ |
| 20 | "112:\t" ASM_UD0 "\n" \ | 20 | "112:\t" ASM_UD2 "\n" \ |
| 21 | ASM_UNREACHABLE \ | 21 | ASM_UNREACHABLE \ |
| 22 | ".popsection\n" \ | 22 | ".popsection\n" \ |
| 23 | "113:\n" \ | 23 | "113:\n" \ |
| @@ -67,13 +67,13 @@ static __always_inline __must_check · 43 unmodified lines | |||
| 67 | bool refcount_sub_and_test(unsigned int i, refcount_t *r) | 67 | bool refcount_sub_and_test(unsigned int i, refcount_t *r) |
| 68 | { | 68 | { |
| 69 | GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", REFCOUNT_CHECK_LT_ZERO, | 69 | GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", REFCOUNT_CHECK_LT_ZERO, |
| 70 | r->refs.counter, "er", i, "%0", e); | 70 | r->refs.counter, "er", i, "%0", e, "cx"); |
| 71 | } | 71 | } |
| 72 | 72 | ||
| 73 | static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r) | 73 | static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r) |
| 74 | { | 74 | { |
| 75 | GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", REFCOUNT_CHECK_LT_ZERO, | 75 | GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", REFCOUNT_CHECK_LT_ZERO, |
| 76 | r->refs.counter, "%0", e); | 76 | r->refs.counter, "%0", e, "cx"); |
| 77 | } | 77 | } |
| 78 | 78 | ||
| 79 | static __always_inline __must_check | 79 | static __always_inline __must_check |
| @@ -2,8 +2,7 @@ · 1 unmodified line | |||
| 2 | #ifndef _ASM_X86_RMWcc | 2 | #ifndef _ASM_X86_RMWcc |
| 3 | #define _ASM_X86_RMWcc | 3 | #define _ASM_X86_RMWcc |
| 4 | 4 | ||
| 5 | #define __CLOBBERS_MEM "memory" | 5 | #define __CLOBBERS_MEM(clb...) "memory", ## clb |
| 6 | #define __CLOBBERS_MEM_CC_CX "memory", "cc", "cx" | ||
| 7 | 6 | ||
| 8 | #if !defined(__GCC_ASM_FLAG_OUTPUTS__) && defined(CC_HAVE_ASM_GOTO) | 7 | #if !defined(__GCC_ASM_FLAG_OUTPUTS__) && defined(CC_HAVE_ASM_GOTO) |
| 9 | 8 | ||
| @@ -40,18 +39,19 @@ do { \ · 30 unmodified lines | |||
| 40 | #endif /* defined(__GCC_ASM_FLAG_OUTPUTS__) || !defined(CC_HAVE_ASM_GOTO) */ | 39 | #endif /* defined(__GCC_ASM_FLAG_OUTPUTS__) || !defined(CC_HAVE_ASM_GOTO) */ |
| 41 | 40 | ||
| 42 | #define GEN_UNARY_RMWcc(op, var, arg0, cc) \ | 41 | #define GEN_UNARY_RMWcc(op, var, arg0, cc) \ |
| 43 | __GEN_RMWcc(op " " arg0, var, cc, __CLOBBERS_MEM) | 42 | __GEN_RMWcc(op " " arg0, var, cc, __CLOBBERS_MEM()) |
| 44 | 43 | ||
| 45 | #define GEN_UNARY_SUFFIXED_RMWcc(op, suffix, var, arg0, cc) \ | 44 | #define GEN_UNARY_SUFFIXED_RMWcc(op, suffix, var, arg0, cc, clobbers...)\ |
| 46 | __GEN_RMWcc(op " " arg0 "\n\t" suffix, var, cc, \ | 45 | __GEN_RMWcc(op " " arg0 "\n\t" suffix, var, cc, \ |
| 47 | __CLOBBERS_MEM_CC_CX) | 46 | __CLOBBERS_MEM(clobbers)) |
| 48 | 47 | ||
| 49 | #define GEN_BINARY_RMWcc(op, var, vcon, val, arg0, cc) \ | 48 | #define GEN_BINARY_RMWcc(op, var, vcon, val, arg0, cc) \ |
| 50 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0, var, cc, \ | 49 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0, var, cc, \ |
| 51 | __CLOBBERS_MEM, vcon (val)) | 50 | __CLOBBERS_MEM(), vcon (val)) |
| 52 | 51 | ||
| 53 | #define GEN_BINARY_SUFFIXED_RMWcc(op, suffix, var, vcon, val, arg0, cc) \ | 52 | #define GEN_BINARY_SUFFIXED_RMWcc(op, suffix, var, vcon, val, arg0, cc, \ |
| 53 | clobbers...) \ | ||
| 54 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0 "\n\t" suffix, var, cc, \ | 54 | __GEN_RMWcc(op __BINARY_RMWcc_ARG arg0 "\n\t" suffix, var, cc, \ |
| 55 | __CLOBBERS_MEM_CC_CX, vcon (val)) | 55 | __CLOBBERS_MEM(clobbers), vcon (val)) |
| 56 | 56 | ||
| 57 | #endif /* _ASM_X86_RMWcc */ | 57 | #endif /* _ASM_X86_RMWcc */ |
| @@ -10,6 +10,7 @@ extern struct exception_table_entry __stop___ex_table[]; · 9 unmodified lines | |||
| 10 | 10 | ||
| 11 | #if defined(CONFIG_X86_64) | 11 | #if defined(CONFIG_X86_64) |
| 12 | extern char __end_rodata_hpage_align[]; | 12 | extern char __end_rodata_hpage_align[]; |
| 13 | extern char __entry_trampoline_start[], __entry_trampoline_end[]; | ||
| 13 | #endif | 14 | #endif |
| 14 | 15 | ||
| 15 | #endif /* _ASM_X86_SECTIONS_H */ | 16 | #endif /* _ASM_X86_SECTIONS_H */ |
| @@ -129,6 +129,7 @@ static inline void arch_send_call_function_ipi_mask(const struct cpumask *mask) · 128 unmodified lines | |||
| 129 | void cpu_disable_common(void); | 129 | void cpu_disable_common(void); |
| 130 | void native_smp_prepare_boot_cpu(void); | 130 | void native_smp_prepare_boot_cpu(void); |
| 131 | void native_smp_prepare_cpus(unsigned int max_cpus); | 131 | void native_smp_prepare_cpus(unsigned int max_cpus); |
| 132 | void calculate_max_logical_packages(void); | ||
| 132 | void native_smp_cpus_done(unsigned int max_cpus); | 133 | void native_smp_cpus_done(unsigned int max_cpus); |
| 133 | void common_cpu_up(unsigned int cpunum, struct task_struct *tidle); | 134 | void common_cpu_up(unsigned int cpunum, struct task_struct *tidle); |
| 134 | int native_cpu_up(unsigned int cpunum, struct task_struct *tidle); | 135 | int native_cpu_up(unsigned int cpunum, struct task_struct *tidle); |
| @@ -20,31 +20,43 @@ · 19 unmodified lines | |||
| 20 | #include <asm/ia32.h> | 20 | #include <asm/ia32.h> |
| 21 | 21 | ||
| 22 | /* ia32/sys_ia32.c */ | 22 | /* ia32/sys_ia32.c */ |
| 23 | asmlinkage long sys32_truncate64(const char __user *, unsigned long, unsigned long); | 23 | asmlinkage long compat_sys_x86_truncate64(const char __user *, unsigned long, |
| 24 | asmlinkage long sys32_ftruncate64(unsigned int, unsigned long, unsigned long); | 24 | unsigned long); |
| 25 | asmlinkage long compat_sys_x86_ftruncate64(unsigned int, unsigned long, | ||
| 26 | unsigned long); | ||
| 25 | 27 | ||
| 26 | asmlinkage long sys32_stat64(const char __user *, struct stat64 __user *); | 28 | asmlinkage long compat_sys_x86_stat64(const char __user *, |
| 27 | asmlinkage long sys32_lstat64(const char __user *, struct stat64 __user *); | 29 | struct stat64 __user *); |
| 28 | asmlinkage long sys32_fstat64(unsigned int, struct stat64 __user *); | 30 | asmlinkage long compat_sys_x86_lstat64(const char __user *, |
| 29 | asmlinkage long sys32_fstatat(unsigned int, const char __user *, | 31 | struct stat64 __user *); |
| 32 | asmlinkage long compat_sys_x86_fstat64(unsigned int, struct stat64 __user *); | ||
| 33 | asmlinkage long compat_sys_x86_fstatat(unsigned int, const char __user *, | ||
| 30 | struct stat64 __user *, int); | 34 | struct stat64 __user *, int); |
| 31 | struct mmap_arg_struct32; | 35 | struct mmap_arg_struct32; |
| 32 | asmlinkage long sys32_mmap(struct mmap_arg_struct32 __user *); | 36 | asmlinkage long compat_sys_x86_mmap(struct mmap_arg_struct32 __user *); |
| 33 | 37 | ||
| 34 | asmlinkage long sys32_waitpid(compat_pid_t, unsigned int __user *, int); | 38 | asmlinkage long compat_sys_x86_waitpid(compat_pid_t, unsigned int __user *, |
| 39 | int); | ||
| 35 | 40 | ||
| 36 | asmlinkage long sys32_pread(unsigned int, char __user *, u32, u32, u32); | 41 | asmlinkage long compat_sys_x86_pread(unsigned int, char __user *, u32, u32, |
| 37 | asmlinkage long sys32_pwrite(unsigned int, const char __user *, u32, u32, u32); | 42 | u32); |
| 43 | asmlinkage long compat_sys_x86_pwrite(unsigned int, const char __user *, u32, | ||
| 44 | u32, u32); | ||
| 38 | 45 | ||
| 39 | long sys32_fadvise64_64(int, __u32, __u32, __u32, __u32, int); | 46 | asmlinkage long compat_sys_x86_fadvise64_64(int, __u32, __u32, __u32, __u32, |
| 40 | long sys32_vm86_warning(void); | 47 | int); |
| 41 | 48 | ||
| 42 | asmlinkage ssize_t sys32_readahead(int, unsigned, unsigned, size_t); | 49 | asmlinkage ssize_t compat_sys_x86_readahead(int, unsigned int, unsigned int, |
| 43 | asmlinkage long sys32_sync_file_range(int, unsigned, unsigned, | 50 | size_t); |
| 44 | unsigned, unsigned, int); | 51 | asmlinkage long compat_sys_x86_sync_file_range(int, unsigned int, unsigned int, |
| 45 | asmlinkage long sys32_fadvise64(int, unsigned, unsigned, size_t, int); | 52 | unsigned int, unsigned int, |
| 46 | asmlinkage long sys32_fallocate(int, int, unsigned, | 53 | int); |
| 47 | unsigned, unsigned, unsigned); | 54 | asmlinkage long compat_sys_x86_fadvise64(int, unsigned int, unsigned int, |
| 55 | size_t, int); | ||
| 56 | asmlinkage long compat_sys_x86_fallocate(int, int, unsigned int, unsigned int, | ||
| 57 | unsigned int, unsigned int); | ||
| 58 | asmlinkage long compat_sys_x86_clone(unsigned long, unsigned long, int __user *, | ||
| 59 | unsigned long, int __user *); | ||
| 48 | 60 | ||
| 49 | /* ia32/ia32_signal.c */ | 61 | /* ia32/ia32_signal.c */ |
| 50 | asmlinkage long sys32_sigreturn(void); | 62 | asmlinkage long sys32_sigreturn(void); |
| @@ -140,7 +140,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) · 139 unmodified lines | |||
| 140 | #else | 140 | #else |
| 141 | #define __flush_tlb() __native_flush_tlb() | 141 | #define __flush_tlb() __native_flush_tlb() |
| 142 | #define __flush_tlb_global() __native_flush_tlb_global() | 142 | #define __flush_tlb_global() __native_flush_tlb_global() |
| 143 | #define __flush_tlb_single(addr) __native_flush_tlb_single(addr) | 143 | #define __flush_tlb_one_user(addr) __native_flush_tlb_one_user(addr) |
| 144 | #endif | 144 | #endif |
| 145 | 145 | ||
| 146 | static inline bool tlb_defer_switch_to_init_mm(void) | 146 | static inline bool tlb_defer_switch_to_init_mm(void) |
| @@ -400,7 +400,7 @@ static inline void __native_flush_tlb_global(void) · 253 unmodified lines | |||
| 400 | /* | 400 | /* |
| 401 | * flush one page in the user mapping | 401 | * flush one page in the user mapping |
| 402 | */ | 402 | */ |
| 403 | static inline void __native_flush_tlb_single(unsigned long addr) | 403 | static inline void __native_flush_tlb_one_user(unsigned long addr) |
| 404 | { | 404 | { |
| 405 | u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); | 405 | u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); |
| 406 | 406 | ||
| @@ -437,18 +437,31 @@ static inline void __flush_tlb_all(void) · 30 unmodified lines | |||
| 437 | /* | 437 | /* |
| 438 | * flush one page in the kernel mapping | 438 | * flush one page in the kernel mapping |
| 439 | */ | 439 | */ |
| 440 | static inline void __flush_tlb_one(unsigned long addr) | 440 | static inline void __flush_tlb_one_kernel(unsigned long addr) |
| 441 | { | 441 | { |
| 442 | count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ONE); | 442 | count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ONE); |
| 443 | __flush_tlb_single(addr); | 443 | |
| 444 | /* | ||
| 445 | * If PTI is off, then __flush_tlb_one_user() is just INVLPG or its | ||
| 446 | * paravirt equivalent. Even with PCID, this is sufficient: we only | ||
| 447 | * use PCID if we also use global PTEs for the kernel mapping, and | ||
| 448 | * INVLPG flushes global translations across all address spaces. | ||
| 449 | * | ||
| 450 | * If PTI is on, then the kernel is mapped with non-global PTEs, and | ||
| 451 | * __flush_tlb_one_user() will flush the given address for the current | ||
| 452 | * kernel address space and for its usermode counterpart, but it does | ||
| 453 | * not flush it for other address spaces. | ||
| 454 | */ | ||
| 455 | __flush_tlb_one_user(addr); | ||
| 444 | 456 | ||
| 445 | if (!static_cpu_has(X86_FEATURE_PTI)) | 457 | if (!static_cpu_has(X86_FEATURE_PTI)) |
| 446 | return; | 458 | return; |
| 447 | 459 | ||
| 448 | /* | 460 | /* |
| 449 | * __flush_tlb_single() will have cleared the TLB entry for this ASID, | 461 | * See above. We need to propagate the flush to all other address |
| 450 | * but since kernel space is replicated across all, we must also | 462 | * spaces. In principle, we only need to propagate it to kernelmode |
| 451 | * invalidate all others. | 463 | * address spaces, but the extra bookkeeping we would need is not |
| 464 | * worth it. | ||
| 452 | */ | 465 | */ |
| 453 | invalidate_other_asid(); | 466 | invalidate_other_asid(); |
| 454 | } | 467 | } |
| @@ -241,24 +241,24 @@ · 240 unmodified lines | |||
| 241 | #define HV_X64_MSR_REENLIGHTENMENT_CONTROL 0x40000106 | 241 | #define HV_X64_MSR_REENLIGHTENMENT_CONTROL 0x40000106 |
| 242 | 242 | ||
| 243 | struct hv_reenlightenment_control { | 243 | struct hv_reenlightenment_control { |
| 244 | u64 vector:8; | 244 | __u64 vector:8; |
| 245 | u64 reserved1:8; | 245 | __u64 reserved1:8; |
| 246 | u64 enabled:1; | 246 | __u64 enabled:1; |
| 247 | u64 reserved2:15; | 247 | __u64 reserved2:15; |
| 248 | u64 target_vp:32; | 248 | __u64 target_vp:32; |
| 249 | }; | 249 | }; |
| 250 | 250 | ||
| 251 | #define HV_X64_MSR_TSC_EMULATION_CONTROL 0x40000107 | 251 | #define HV_X64_MSR_TSC_EMULATION_CONTROL 0x40000107 |
| 252 | #define HV_X64_MSR_TSC_EMULATION_STATUS 0x40000108 | 252 | #define HV_X64_MSR_TSC_EMULATION_STATUS 0x40000108 |
| 253 | 253 | ||
| 254 | struct hv_tsc_emulation_control { | 254 | struct hv_tsc_emulation_control { |
| 255 | u64 enabled:1; | 255 | __u64 enabled:1; |
| 256 | u64 reserved:63; | 256 | __u64 reserved:63; |
| 257 | }; | 257 | }; |
| 258 | 258 | ||
| 259 | struct hv_tsc_emulation_status { | 259 | struct hv_tsc_emulation_status { |
| 260 | u64 inprogress:1; | 260 | __u64 inprogress:1; |
| 261 | u64 reserved:63; | 261 | __u64 reserved:63; |
| 262 | }; | 262 | }; |
| 263 | 263 | ||
| 264 | #define HV_X64_MSR_HYPERCALL_ENABLE 0x00000001 | 264 | #define HV_X64_MSR_HYPERCALL_ENABLE 0x00000001 |
| @@ -26,6 +26,7 @@ · 25 unmodified lines | |||
| 26 | #define KVM_FEATURE_PV_EOI 6 | 26 | #define KVM_FEATURE_PV_EOI 6 |
| 27 | #define KVM_FEATURE_PV_UNHALT 7 | 27 | #define KVM_FEATURE_PV_UNHALT 7 |
| 28 | #define KVM_FEATURE_PV_TLB_FLUSH 9 | 28 | #define KVM_FEATURE_PV_TLB_FLUSH 9 |
| 29 | #define KVM_FEATURE_ASYNC_PF_VMEXIT 10 | ||
| 29 | 30 | ||
| 30 | /* The last 8 bits are used to indicate how to interpret the flags field | 31 | /* The last 8 bits are used to indicate how to interpret the flags field |
| 31 | * in pvclock structure. If no bits are set, all flags are ignored. | 32 | * in pvclock structure. If no bits are set, all flags are ignored. |
| @@ -30,6 +30,7 @@ struct mce { · 29 unmodified lines | |||
| 30 | __u64 synd; /* MCA_SYND MSR: only valid on SMCA systems */ | 30 | __u64 synd; /* MCA_SYND MSR: only valid on SMCA systems */ |
| 31 | __u64 ipid; /* MCA_IPID MSR: only valid on SMCA systems */ | 31 | __u64 ipid; /* MCA_IPID MSR: only valid on SMCA systems */ |
| 32 | __u64 ppin; /* Protected Processor Inventory Number */ | 32 | __u64 ppin; /* Protected Processor Inventory Number */ |
| 33 | __u32 microcode;/* Microcode revision */ | ||
| 33 | }; | 34 | }; |
| 34 | 35 | ||
| 35 | #define MCE_GET_RECORD_LEN _IOR('M', 1, int) | 36 | #define MCE_GET_RECORD_LEN _IOR('M', 1, int) |
| @@ -235,7 +235,7 @@ int amd_cache_northbridges(void) · 234 unmodified lines | |||
| 235 | if (boot_cpu_data.x86 == 0x10 && | 235 | if (boot_cpu_data.x86 == 0x10 && |
| 236 | boot_cpu_data.x86_model >= 0x8 && | 236 | boot_cpu_data.x86_model >= 0x8 && |
| 237 | (boot_cpu_data.x86_model > 0x9 || | 237 | (boot_cpu_data.x86_model > 0x9 || |
| 238 | boot_cpu_data.x86_mask >= 0x1)) | 238 | boot_cpu_data.x86_stepping >= 0x1)) |
| 239 | amd_northbridges.flags |= AMD_NB_L3_INDEX_DISABLE; | 239 | amd_northbridges.flags |= AMD_NB_L3_INDEX_DISABLE; |
| 240 | 240 | ||
| 241 | if (boot_cpu_data.x86 == 0x15) | 241 | if (boot_cpu_data.x86 == 0x15) |
| @@ -546,7 +546,7 @@ static DEFINE_PER_CPU(struct clock_event_device, lapic_events); · 545 unmodified lines | |||
| 546 | 546 | ||
| 547 | static u32 hsx_deadline_rev(void) | 547 | static u32 hsx_deadline_rev(void) |
| 548 | { | 548 | { |
| 549 | switch (boot_cpu_data.x86_mask) { | 549 | switch (boot_cpu_data.x86_stepping) { |
| 550 | case 0x02: return 0x3a; /* EP */ | 550 | case 0x02: return 0x3a; /* EP */ |
| 551 | case 0x04: return 0x0f; /* EX */ | 551 | case 0x04: return 0x0f; /* EX */ |
| 552 | } | 552 | } |
| @@ -556,7 +556,7 @@ static u32 hsx_deadline_rev(void) · 3 unmodified lines | |||
| 556 | 556 | ||
| 557 | static u32 bdx_deadline_rev(void) | 557 | static u32 bdx_deadline_rev(void) |
| 558 | { | 558 | { |
| 559 | switch (boot_cpu_data.x86_mask) { | 559 | switch (boot_cpu_data.x86_stepping) { |
| 560 | case 0x02: return 0x00000011; | 560 | case 0x02: return 0x00000011; |
| 561 | case 0x03: return 0x0700000e; | 561 | case 0x03: return 0x0700000e; |
| 562 | case 0x04: return 0x0f00000c; | 562 | case 0x04: return 0x0f00000c; |
| @@ -568,7 +568,7 @@ static u32 bdx_deadline_rev(void) · 5 unmodified lines | |||
| 568 | 568 | ||
| 569 | static u32 skx_deadline_rev(void) | 569 | static u32 skx_deadline_rev(void) |
| 570 | { | 570 | { |
| 571 | switch (boot_cpu_data.x86_mask) { | 571 | switch (boot_cpu_data.x86_stepping) { |
| 572 | case 0x03: return 0x01000136; | 572 | case 0x03: return 0x01000136; |
| 573 | case 0x04: return 0x02000014; | 573 | case 0x04: return 0x02000014; |
| 574 | } | 574 | } |
| @@ -1603,7 +1603,7 @@ static void __init delay_with_tsc(void) · 1602 unmodified lines | |||
| 1603 | do { | 1603 | do { |
| 1604 | rep_nop(); | 1604 | rep_nop(); |
| 1605 | now = rdtsc(); | 1605 | now = rdtsc(); |
| 1606 | } while ((now - start) < 40000000000UL / HZ && | 1606 | } while ((now - start) < 40000000000ULL / HZ && |
| 1607 | time_before_eq(jiffies, end)); | 1607 | time_before_eq(jiffies, end)); |
| 1608 | } | 1608 | } |
| 1609 | 1609 | ||
| @@ -134,21 +134,40 @@ static void apic_update_vector(struct irq_data *irqd, unsigned int newvec, · 133 unmodified lines | |||
| 134 | { | 134 | { |
| 135 | struct apic_chip_data *apicd = apic_chip_data(irqd); | 135 | struct apic_chip_data *apicd = apic_chip_data(irqd); |
| 136 | struct irq_desc *desc = irq_data_to_desc(irqd); | 136 | struct irq_desc *desc = irq_data_to_desc(irqd); |
| 137 | bool managed = irqd_affinity_is_managed(irqd); | ||
| 137 | 138 | ||
| 138 | lockdep_assert_held(&vector_lock); | 139 | lockdep_assert_held(&vector_lock); |
| 139 | 140 | ||
| 140 | trace_vector_update(irqd->irq, newvec, newcpu, apicd->vector, | 141 | trace_vector_update(irqd->irq, newvec, newcpu, apicd->vector, |
| 141 | apicd->cpu); | 142 | apicd->cpu); |
| 142 | 143 | ||
| 143 | /* Setup the vector move, if required */ | 144 | /* |
| 144 | if (apicd->vector && cpu_online(apicd->cpu)) { | 145 | * If there is no vector associated or if the associated vector is |
| 146 | * the shutdown vector, which is associated to make PCI/MSI | ||
| 147 | * shutdown mode work, then there is nothing to release. Clear out | ||
| 148 | * prev_vector for this and the offlined target case. | ||
| 149 | */ | ||
| 150 | apicd->prev_vector = 0; | ||
| 151 | if (!apicd->vector || apicd->vector == MANAGED_IRQ_SHUTDOWN_VECTOR) | ||
| 152 | goto setnew; | ||
| 153 | /* | ||
| 154 | * If the target CPU of the previous vector is online, then mark | ||
| 155 | * the vector as move in progress and store it for cleanup when the | ||
| 156 | * first interrupt on the new vector arrives. If the target CPU is | ||
| 157 | * offline then the regular release mechanism via the cleanup | ||
| 158 | * vector is not possible and the vector can be immediately freed | ||
| 159 | * in the underlying matrix allocator. | ||
| 160 | */ | ||
| 161 | if (cpu_online(apicd->cpu)) { | ||
| 145 | apicd->move_in_progress = true; | 162 | apicd->move_in_progress = true; |
| 146 | apicd->prev_vector = apicd->vector; | 163 | apicd->prev_vector = apicd->vector; |
| 147 | apicd->prev_cpu = apicd->cpu; | 164 | apicd->prev_cpu = apicd->cpu; |
| 148 | } else { | 165 | } else { |
| 149 | apicd->prev_vector = 0; | 166 | irq_matrix_free(vector_matrix, apicd->cpu, apicd->vector, |
| 167 | managed); | ||
| 150 | } | 168 | } |
| 151 | 169 | ||
| 170 | setnew: | ||
| 152 | apicd->vector = newvec; | 171 | apicd->vector = newvec; |
| 153 | apicd->cpu = newcpu; | 172 | apicd->cpu = newcpu; |
| 154 | BUG_ON(!IS_ERR_OR_NULL(per_cpu(vector_irq, newcpu)[newvec])); | 173 | BUG_ON(!IS_ERR_OR_NULL(per_cpu(vector_irq, newcpu)[newvec])); |
| @@ -1176,16 +1176,25 @@ static void __init decode_gam_rng_tbl(unsigned long ptr) · 1175 unmodified lines | |||
| 1176 | 1176 | ||
| 1177 | uv_gre_table = gre; | 1177 | uv_gre_table = gre; |
| 1178 | for (; gre->type != UV_GAM_RANGE_TYPE_UNUSED; gre++) { | 1178 | for (; gre->type != UV_GAM_RANGE_TYPE_UNUSED; gre++) { |
| 1179 | unsigned long size = ((unsigned long)(gre->limit - lgre) | ||
| 1180 | << UV_GAM_RANGE_SHFT); | ||
| 1181 | int order = 0; | ||
| 1182 | char suffix[] = " KMGTPE"; | ||
| 1183 | |||
| 1184 | while (size > 9999 && order < sizeof(suffix)) { | ||
| 1185 | size /= 1024; | ||
| 1186 | order++; | ||
| 1187 | } | ||
| 1188 | |||
| 1179 | if (!index) { | 1189 | if (!index) { |
| 1180 | pr_info("UV: GAM Range Table...\n"); | 1190 | pr_info("UV: GAM Range Table...\n"); |
| 1181 | pr_info("UV: # %20s %14s %5s %4s %5s %3s %2s\n", "Range", "", "Size", "Type", "NASID", "SID", "PN"); | 1191 | pr_info("UV: # %20s %14s %5s %4s %5s %3s %2s\n", "Range", "", "Size", "Type", "NASID", "SID", "PN"); |
| 1182 | } | 1192 | } |
| 1183 | pr_info("UV: %2d: 0x%014lx-0x%014lx %5luG %3d %04x %02x %02x\n", | 1193 | pr_info("UV: %2d: 0x%014lx-0x%014lx %5lu%c %3d %04x %02x %02x\n", |
| 1184 | index++, | 1194 | index++, |
| 1185 | (unsigned long)lgre << UV_GAM_RANGE_SHFT, | 1195 | (unsigned long)lgre << UV_GAM_RANGE_SHFT, |
| 1186 | (unsigned long)gre->limit << UV_GAM_RANGE_SHFT, | 1196 | (unsigned long)gre->limit << UV_GAM_RANGE_SHFT, |
| 1187 | ((unsigned long)(gre->limit - lgre)) >> | 1197 | size, suffix[order], |
| 1188 | (30 - UV_GAM_RANGE_SHFT), /* 64M -> 1G */ | ||
| 1189 | gre->type, gre->nasid, gre->sockid, gre->pnode); | 1198 | gre->type, gre->nasid, gre->sockid, gre->pnode); |
| 1190 | 1199 | ||
| 1191 | lgre = gre->limit; | 1200 | lgre = gre->limit; |
| @@ -18,7 +18,7 @@ void foo(void) · 17 unmodified lines | |||
| 18 | OFFSET(CPUINFO_x86, cpuinfo_x86, x86); | 18 | OFFSET(CPUINFO_x86, cpuinfo_x86, x86); |
| 19 | OFFSET(CPUINFO_x86_vendor, cpuinfo_x86, x86_vendor); | 19 | OFFSET(CPUINFO_x86_vendor, cpuinfo_x86, x86_vendor); |
| 20 | OFFSET(CPUINFO_x86_model, cpuinfo_x86, x86_model); | 20 | OFFSET(CPUINFO_x86_model, cpuinfo_x86, x86_model); |
| 21 | OFFSET(CPUINFO_x86_mask, cpuinfo_x86, x86_mask); | 21 | OFFSET(CPUINFO_x86_stepping, cpuinfo_x86, x86_stepping); |
| 22 | OFFSET(CPUINFO_cpuid_level, cpuinfo_x86, cpuid_level); | 22 | OFFSET(CPUINFO_cpuid_level, cpuinfo_x86, cpuid_level); |
| 23 | OFFSET(CPUINFO_x86_capability, cpuinfo_x86, x86_capability); | 23 | OFFSET(CPUINFO_x86_capability, cpuinfo_x86, x86_capability); |
| 24 | OFFSET(CPUINFO_x86_vendor_id, cpuinfo_x86, x86_vendor_id); | 24 | OFFSET(CPUINFO_x86_vendor_id, cpuinfo_x86, x86_vendor_id); |
| @@ -119,7 +119,7 @@ static void init_amd_k6(struct cpuinfo_x86 *c) · 118 unmodified lines | |||
| 119 | return; | 119 | return; |
| 120 | } | 120 | } |
| 121 | 121 | ||
| 122 | if (c->x86_model == 6 && c->x86_mask == 1) { | 122 | if (c->x86_model == 6 && c->x86_stepping == 1) { |
| 123 | const int K6_BUG_LOOP = 1000000; | 123 | const int K6_BUG_LOOP = 1000000; |
| 124 | int n; | 124 | int n; |
| 125 | void (*f_vide)(void); | 125 | void (*f_vide)(void); |
| @@ -149,7 +149,7 @@ static void init_amd_k6(struct cpuinfo_x86 *c) · 23 unmodified lines | |||
| 149 | 149 | ||
| 150 | /* K6 with old style WHCR */ | 150 | /* K6 with old style WHCR */ |
| 151 | if (c->x86_model < 8 || | 151 | if (c->x86_model < 8 || |
| 152 | (c->x86_model == 8 && c->x86_mask < 8)) { | 152 | (c->x86_model == 8 && c->x86_stepping < 8)) { |
| 153 | /* We can only write allocate on the low 508Mb */ | 153 | /* We can only write allocate on the low 508Mb */ |
| 154 | if (mbytes > 508) | 154 | if (mbytes > 508) |
| 155 | mbytes = 508; | 155 | mbytes = 508; |
| @@ -168,7 +168,7 @@ static void init_amd_k6(struct cpuinfo_x86 *c) · 12 unmodified lines | |||
| 168 | return; | 168 | return; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | if ((c->x86_model == 8 && c->x86_mask > 7) || | 171 | if ((c->x86_model == 8 && c->x86_stepping > 7) || |
| 172 | c->x86_model == 9 || c->x86_model == 13) { | 172 | c->x86_model == 9 || c->x86_model == 13) { |
| 173 | /* The more serious chips .. */ | 173 | /* The more serious chips .. */ |
| 174 | 174 | ||
| @@ -221,7 +221,7 @@ static void init_amd_k7(struct cpuinfo_x86 *c) · 46 unmodified lines | |||
| 221 | * are more robust with CLK_CTL set to 200xxxxx instead of 600xxxxx | 221 | * are more robust with CLK_CTL set to 200xxxxx instead of 600xxxxx |
| 222 | * As per AMD technical note 27212 0.2 | 222 | * As per AMD technical note 27212 0.2 |
| 223 | */ | 223 | */ |
| 224 | if ((c->x86_model == 8 && c->x86_mask >= 1) || (c->x86_model > 8)) { | 224 | if ((c->x86_model == 8 && c->x86_stepping >= 1) || (c->x86_model > 8)) { |
| 225 | rdmsr(MSR_K7_CLK_CTL, l, h); | 225 | rdmsr(MSR_K7_CLK_CTL, l, h); |
| 226 | if ((l & 0xfff00000) != 0x20000000) { | 226 | if ((l & 0xfff00000) != 0x20000000) { |
| 227 | pr_info("CPU: CLK_CTL MSR was %x. Reprogramming to %x\n", | 227 | pr_info("CPU: CLK_CTL MSR was %x. Reprogramming to %x\n", |
| @@ -241,12 +241,12 @@ static void init_amd_k7(struct cpuinfo_x86 *c) · 13 unmodified lines | |||
| 241 | * but they are not certified as MP capable. | 241 | * but they are not certified as MP capable. |
| 242 | */ | 242 | */ |
| 243 | /* Athlon 660/661 is valid. */ | 243 | /* Athlon 660/661 is valid. */ |
| 244 | if ((c->x86_model == 6) && ((c->x86_mask == 0) || | 244 | if ((c->x86_model == 6) && ((c->x86_stepping == 0) || |
| 245 | (c->x86_mask == 1))) | 245 | (c->x86_stepping == 1))) |
| 246 | return; | 246 | return; |
| 247 | 247 | ||
| 248 | /* Duron 670 is valid */ | 248 | /* Duron 670 is valid */ |
| 249 | if ((c->x86_model == 7) && (c->x86_mask == 0)) | 249 | if ((c->x86_model == 7) && (c->x86_stepping == 0)) |
| 250 | return; | 250 | return; |
| 251 | 251 | ||
| 252 | /* | 252 | /* |
| @@ -256,8 +256,8 @@ static void init_amd_k7(struct cpuinfo_x86 *c) · 3 unmodified lines | |||
| 256 | * See http://www.heise.de/newsticker/data/jow-18.10.01-000 for | 256 | * See http://www.heise.de/newsticker/data/jow-18.10.01-000 for |
| 257 | * more. | 257 | * more. |
| 258 | */ | 258 | */ |
| 259 | if (((c->x86_model == 6) && (c->x86_mask >= 2)) || | 259 | if (((c->x86_model == 6) && (c->x86_stepping >= 2)) || |
| 260 | ((c->x86_model == 7) && (c->x86_mask >= 1)) || | 260 | ((c->x86_model == 7) && (c->x86_stepping >= 1)) || |
| 261 | (c->x86_model > 7)) | 261 | (c->x86_model > 7)) |
| 262 | if (cpu_has(c, X86_FEATURE_MP)) | 262 | if (cpu_has(c, X86_FEATURE_MP)) |
| 263 | return; | 263 | return; |
| @@ -628,7 +628,7 @@ static void early_init_amd(struct cpuinfo_x86 *c) · 364 unmodified lines | |||
| 628 | /* Set MTRR capability flag if appropriate */ | 628 | /* Set MTRR capability flag if appropriate */ |
| 629 | if (c->x86 == 5) | 629 | if (c->x86 == 5) |
| 630 | if (c->x86_model == 13 || c->x86_model == 9 || | 630 | if (c->x86_model == 13 || c->x86_model == 9 || |
| 631 | (c->x86_model == 8 && c->x86_mask >= 8)) | 631 | (c->x86_model == 8 && c->x86_stepping >= 8)) |
| 632 | set_cpu_cap(c, X86_FEATURE_K6_MTRR); | 632 | set_cpu_cap(c, X86_FEATURE_K6_MTRR); |
| 633 | #endif | 633 | #endif |
| 634 | #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_PCI) | 634 | #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_PCI) |
| @@ -795,7 +795,7 @@ static void init_amd_zn(struct cpuinfo_x86 *c) · 160 unmodified lines | |||
| 795 | * Fix erratum 1076: CPB feature bit not being set in CPUID. It affects | 795 | * Fix erratum 1076: CPB feature bit not being set in CPUID. It affects |
| 796 | * all up to and including B1. | 796 | * all up to and including B1. |
| 797 | */ | 797 | */ |
| 798 | if (c->x86_model <= 1 && c->x86_mask <= 1) | 798 | if (c->x86_model <= 1 && c->x86_stepping <= 1) |
| 799 | set_cpu_cap(c, X86_FEATURE_CPB); | 799 | set_cpu_cap(c, X86_FEATURE_CPB); |
| 800 | } | 800 | } |
| 801 | 801 | ||
| @@ -906,11 +906,11 @@ static unsigned int amd_size_cache(struct cpuinfo_x86 *c, unsigned int size) · 104 unmodified lines | |||
| 906 | /* AMD errata T13 (order #21922) */ | 906 | /* AMD errata T13 (order #21922) */ |
| 907 | if ((c->x86 == 6)) { | 907 | if ((c->x86 == 6)) { |
| 908 | /* Duron Rev A0 */ | 908 | /* Duron Rev A0 */ |
| 909 | if (c->x86_model == 3 && c->x86_mask == 0) | 909 | if (c->x86_model == 3 && c->x86_stepping == 0) |
| 910 | size = 64; | 910 | size = 64; |
| 911 | /* Tbird rev A1/A2 */ | 911 | /* Tbird rev A1/A2 */ |
| 912 | if (c->x86_model == 4 && | 912 | if (c->x86_model == 4 && |
| 913 | (c->x86_mask == 0 || c->x86_mask == 1)) | 913 | (c->x86_stepping == 0 || c->x86_stepping == 1)) |
| 914 | size = 256; | 914 | size = 256; |
| 915 | } | 915 | } |
| 916 | return size; | 916 | return size; |
| @@ -1047,7 +1047,7 @@ static bool cpu_has_amd_erratum(struct cpuinfo_x86 *cpu, const int *erratum) · 130 unmodified lines | |||
| 1047 | } | 1047 | } |
| 1048 | 1048 | ||
| 1049 | /* OSVW unavailable or ID unknown, match family-model-stepping range */ | 1049 | /* OSVW unavailable or ID unknown, match family-model-stepping range */ |
| 1050 | ms = (cpu->x86_model << 4) | cpu->x86_mask; | 1050 | ms = (cpu->x86_model << 4) | cpu->x86_stepping; |
| 1051 | while ((range = *erratum++)) | 1051 | while ((range = *erratum++)) |
| 1052 | if ((cpu->x86 == AMD_MODEL_RANGE_FAMILY(range)) && | 1052 | if ((cpu->x86 == AMD_MODEL_RANGE_FAMILY(range)) && |
| 1053 | (ms >= AMD_MODEL_RANGE_START(range)) && | 1053 | (ms >= AMD_MODEL_RANGE_START(range)) && |
| @@ -162,8 +162,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) · 161 unmodified lines | |||
| 162 | if (cmdline_find_option_bool(boot_command_line, "nospectre_v2")) | 162 | if (cmdline_find_option_bool(boot_command_line, "nospectre_v2")) |
| 163 | return SPECTRE_V2_CMD_NONE; | 163 | return SPECTRE_V2_CMD_NONE; |
| 164 | else { | 164 | else { |
| 165 | ret = cmdline_find_option(boot_command_line, "spectre_v2", arg, | 165 | ret = cmdline_find_option(boot_command_line, "spectre_v2", arg, sizeof(arg)); |
| 166 | sizeof(arg)); | ||
| 167 | if (ret < 0) | 166 | if (ret < 0) |
| 168 | return SPECTRE_V2_CMD_AUTO; | 167 | return SPECTRE_V2_CMD_AUTO; |
| 169 | 168 | ||
| @@ -175,8 +174,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) · 5 unmodified lines | |||
| 175 | } | 174 | } |
| 176 | 175 | ||
| 177 | if (i >= ARRAY_SIZE(mitigation_options)) { | 176 | if (i >= ARRAY_SIZE(mitigation_options)) { |
| 178 | pr_err("unknown option (%s). Switching to AUTO select\n", | 177 | pr_err("unknown option (%s). Switching to AUTO select\n", arg); |
| 179 | mitigation_options[i].option); | ||
| 180 | return SPECTRE_V2_CMD_AUTO; | 178 | return SPECTRE_V2_CMD_AUTO; |
| 181 | } | 179 | } |
| 182 | } | 180 | } |
| @@ -185,8 +183,7 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void) · 2 unmodified lines | |||
| 185 | cmd == SPECTRE_V2_CMD_RETPOLINE_AMD || | 183 | cmd == SPECTRE_V2_CMD_RETPOLINE_AMD || |
| 186 | cmd == SPECTRE_V2_CMD_RETPOLINE_GENERIC) && | 184 | cmd == SPECTRE_V2_CMD_RETPOLINE_GENERIC) && |
| 187 | !IS_ENABLED(CONFIG_RETPOLINE)) { | 185 | !IS_ENABLED(CONFIG_RETPOLINE)) { |
| 188 | pr_err("%s selected but not compiled in. Switching to AUTO select\n", | 186 | pr_err("%s selected but not compiled in. Switching to AUTO select\n", mitigation_options[i].option); |
| 189 | mitigation_options[i].option); | ||
| 190 | return SPECTRE_V2_CMD_AUTO; | 187 | return SPECTRE_V2_CMD_AUTO; |
| 191 | } | 188 | } |
| 192 | 189 | ||
| @@ -256,14 +253,14 @@ static void __init spectre_v2_select_mitigation(void) · 63 unmodified lines | |||
| 256 | goto retpoline_auto; | 253 | goto retpoline_auto; |
| 257 | break; | 254 | break; |
| 258 | } | 255 | } |
| 259 | pr_err("kernel not compiled with retpoline; no mitigation available!"); | 256 | pr_err("Spectre mitigation: kernel not compiled with retpoline; no mitigation available!"); |
| 260 | return; | 257 | return; |
| 261 | 258 | ||
| 262 | retpoline_auto: | 259 | retpoline_auto: |
| 263 | if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) { | 260 | if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) { |
| 264 | retpoline_amd: | 261 | retpoline_amd: |
| 265 | if (!boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) { | 262 | if (!boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) { |
| 266 | pr_err("LFENCE not serializing. Switching to generic retpoline\n"); | 263 | pr_err("Spectre mitigation: LFENCE not serializing, switching to generic retpoline\n"); |
| 267 | goto retpoline_generic; | 264 | goto retpoline_generic; |
| 268 | } | 265 | } |
| 269 | mode = retp_compiler() ? SPECTRE_V2_RETPOLINE_AMD : | 266 | mode = retp_compiler() ? SPECTRE_V2_RETPOLINE_AMD : |
| @@ -281,7 +278,7 @@ retpoline_auto: · 11 unmodified lines | |||
| 281 | pr_info("%s\n", spectre_v2_strings[mode]); | 278 | pr_info("%s\n", spectre_v2_strings[mode]); |
| 282 | 279 | ||
| 283 | /* | 280 | /* |
| 284 | * If neither SMEP or KPTI are available, there is a risk of | 281 | * If neither SMEP nor PTI are available, there is a risk of |
| 285 | * hitting userspace addresses in the RSB after a context switch | 282 | * hitting userspace addresses in the RSB after a context switch |
| 286 | * from a shallow call stack to a deeper one. To prevent this fill | 283 | * from a shallow call stack to a deeper one. To prevent this fill |
| 287 | * the entire RSB, even when using IBRS. | 284 | * the entire RSB, even when using IBRS. |
| @@ -295,21 +292,29 @@ retpoline_auto: · 7 unmodified lines | |||
| 295 | if ((!boot_cpu_has(X86_FEATURE_PTI) && | 292 | if ((!boot_cpu_has(X86_FEATURE_PTI) && |
| 296 | !boot_cpu_has(X86_FEATURE_SMEP)) || is_skylake_era()) { | 293 | !boot_cpu_has(X86_FEATURE_SMEP)) || is_skylake_era()) { |
| 297 | setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); | 294 | setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); |
| 298 | pr_info("Filling RSB on context switch\n"); | 295 | pr_info("Spectre v2 mitigation: Filling RSB on context switch\n"); |
| 299 | } | 296 | } |
| 300 | 297 | ||
| 301 | /* Initialize Indirect Branch Prediction Barrier if supported */ | 298 | /* Initialize Indirect Branch Prediction Barrier if supported */ |
| 302 | if (boot_cpu_has(X86_FEATURE_IBPB)) { | 299 | if (boot_cpu_has(X86_FEATURE_IBPB)) { |
| 303 | setup_force_cpu_cap(X86_FEATURE_USE_IBPB); | 300 | setup_force_cpu_cap(X86_FEATURE_USE_IBPB); |
| 304 | pr_info("Enabling Indirect Branch Prediction Barrier\n"); | 301 | pr_info("Spectre v2 mitigation: Enabling Indirect Branch Prediction Barrier\n"); |
| 302 | } | ||
| 303 | |||
| 304 | /* | ||
| 305 | * Retpoline means the kernel is safe because it has no indirect | ||
| 306 | * branches. But firmware isn't, so use IBRS to protect that. | ||
| 307 | */ | ||
| 308 | if (boot_cpu_has(X86_FEATURE_IBRS)) { | ||
| 309 | setup_force_cpu_cap(X86_FEATURE_USE_IBRS_FW); | ||
| 310 | pr_info("Enabling Restricted Speculation for firmware calls\n"); | ||
| 305 | } | 311 | } |
| 306 | } | 312 | } |
| 307 | 313 | ||
| 308 | #undef pr_fmt | 314 | #undef pr_fmt |
| 309 | 315 | ||
| 310 | #ifdef CONFIG_SYSFS | 316 | #ifdef CONFIG_SYSFS |
| 311 | ssize_t cpu_show_meltdown(struct device *dev, | 317 | ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf) |
| 312 | struct device_attribute *attr, char *buf) | ||
| 313 | { | 318 | { |
| 314 | if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) | 319 | if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) |
| 315 | return sprintf(buf, "Not affected\n"); | 320 | return sprintf(buf, "Not affected\n"); |
| @@ -318,28 +323,21 @@ ssize_t cpu_show_meltdown(struct device *dev, · 2 unmodified lines | |||
| 318 | return sprintf(buf, "Vulnerable\n"); | 323 | return sprintf(buf, "Vulnerable\n"); |
| 319 | } | 324 | } |
| 320 | 325 | ||
| 321 | ssize_t cpu_show_spectre_v1(struct device *dev, | 326 | ssize_t cpu_show_spectre_v1(struct device *dev, struct device_attribute *attr, char *buf) |
| 322 | struct device_attribute *attr, char *buf) | ||
| 323 | { | 327 | { |
| 324 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1)) | 328 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1)) |
| 325 | return sprintf(buf, "Not affected\n"); | 329 | return sprintf(buf, "Not affected\n"); |
| 326 | return sprintf(buf, "Mitigation: __user pointer sanitization\n"); | 330 | return sprintf(buf, "Mitigation: __user pointer sanitization\n"); |
| 327 | } | 331 | } |
| 328 | 332 | ||
| 329 | ssize_t cpu_show_spectre_v2(struct device *dev, | 333 | ssize_t cpu_show_spectre_v2(struct device *dev, struct device_attribute *attr, char *buf) |
| 330 | struct device_attribute *attr, char *buf) | ||
| 331 | { | 334 | { |
| 332 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V2)) | 335 | if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V2)) |
| 333 | return sprintf(buf, "Not affected\n"); | 336 | return sprintf(buf, "Not affected\n"); |
| 334 | 337 | ||
| 335 | return sprintf(buf, "%s%s%s\n", spectre_v2_strings[spectre_v2_enabled], | 338 | return sprintf(buf, "%s%s%s%s\n", spectre_v2_strings[spectre_v2_enabled], |
| 336 | boot_cpu_has(X86_FEATURE_USE_IBPB) ? ", IBPB" : "", | 339 | boot_cpu_has(X86_FEATURE_USE_IBPB) ? ", IBPB" : "", |
| 340 | boot_cpu_has(X86_FEATURE_USE_IBRS_FW) ? ", IBRS_FW" : "", | ||
| 337 | spectre_v2_module_string()); | 341 | spectre_v2_module_string()); |
| 338 | } | 342 | } |
| 339 | #endif | 343 | #endif |
| 340 | |||
| 341 | void __ibp_barrier(void) | ||
| 342 | { | ||
| 343 | __wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0); | ||
| 344 | } | ||
| 345 | EXPORT_SYMBOL_GPL(__ibp_barrier); | ||
| @@ -140,7 +140,7 @@ static void init_centaur(struct cpuinfo_x86 *c) · 139 unmodified lines | |||
| 140 | clear_cpu_cap(c, X86_FEATURE_TSC); | 140 | clear_cpu_cap(c, X86_FEATURE_TSC); |
| 141 | break; | 141 | break; |
| 142 | case 8: | 142 | case 8: |
| 143 | switch (c->x86_mask) { | 143 | switch (c->x86_stepping) { |
| 144 | default: | 144 | default: |
| 145 | name = "2"; | 145 | name = "2"; |
| 146 | break; | 146 | break; |
| @@ -215,7 +215,7 @@ centaur_size_cache(struct cpuinfo_x86 *c, unsigned int size) · 68 unmodified lines | |||
| 215 | * - Note, it seems this may only be in engineering samples. | 215 | * - Note, it seems this may only be in engineering samples. |
| 216 | */ | 216 | */ |
| 217 | if ((c->x86 == 6) && (c->x86_model == 9) && | 217 | if ((c->x86 == 6) && (c->x86_model == 9) && |
| 218 | (c->x86_mask == 1) && (size == 65)) | 218 | (c->x86_stepping == 1) && (size == 65)) |
| 219 | size -= 1; | 219 | size -= 1; |
| 220 | return size; | 220 | return size; |
| 221 | } | 221 | } |
| @@ -731,7 +731,7 @@ void cpu_detect(struct cpuinfo_x86 *c) · 730 unmodified lines | |||
| 731 | cpuid(0x00000001, &tfms, &misc, &junk, &cap0); | 731 | cpuid(0x00000001, &tfms, &misc, &junk, &cap0); |
| 732 | c->x86 = x86_family(tfms); | 732 | c->x86 = x86_family(tfms); |
| 733 | c->x86_model = x86_model(tfms); | 733 | c->x86_model = x86_model(tfms); |
| 734 | c->x86_mask = x86_stepping(tfms); | 734 | c->x86_stepping = x86_stepping(tfms); |
| 735 | 735 | ||
| 736 | if (cap0 & (1<<19)) { | 736 | if (cap0 & (1<<19)) { |
| 737 | c->x86_clflush_size = ((misc >> 8) & 0xff) * 8; | 737 | c->x86_clflush_size = ((misc >> 8) & 0xff) * 8; |
| @@ -1184,9 +1184,9 @@ static void identify_cpu(struct cpuinfo_x86 *c) · 446 unmodified lines | |||
| 1184 | int i; | 1184 | int i; |
| 1185 | 1185 | ||
| 1186 | c->loops_per_jiffy = loops_per_jiffy; | 1186 | c->loops_per_jiffy = loops_per_jiffy; |
| 1187 | c->x86_cache_size = -1; | 1187 | c->x86_cache_size = 0; |
| 1188 | c->x86_vendor = X86_VENDOR_UNKNOWN; | 1188 | c->x86_vendor = X86_VENDOR_UNKNOWN; |
| 1189 | c->x86_model = c->x86_mask = 0; /* So far unknown... */ | 1189 | c->x86_model = c->x86_stepping = 0; /* So far unknown... */ |
| 1190 | c->x86_vendor_id[0] = '\0'; /* Unset */ | 1190 | c->x86_vendor_id[0] = '\0'; /* Unset */ |
| 1191 | c->x86_model_id[0] = '\0'; /* Unset */ | 1191 | c->x86_model_id[0] = '\0'; /* Unset */ |
| 1192 | c->x86_max_cores = 1; | 1192 | c->x86_max_cores = 1; |
| @@ -1378,8 +1378,8 @@ void print_cpu_info(struct cpuinfo_x86 *c) · 185 unmodified lines | |||
| 1378 | 1378 | ||
| 1379 | pr_cont(" (family: 0x%x, model: 0x%x", c->x86, c->x86_model); | 1379 | pr_cont(" (family: 0x%x, model: 0x%x", c->x86, c->x86_model); |
| 1380 | 1380 | ||
| 1381 | if (c->x86_mask || c->cpuid_level >= 0) | 1381 | if (c->x86_stepping || c->cpuid_level >= 0) |
| 1382 | pr_cont(", stepping: 0x%x)\n", c->x86_mask); | 1382 | pr_cont(", stepping: 0x%x)\n", c->x86_stepping); |
| 1383 | else | 1383 | else |
| 1384 | pr_cont(")\n"); | 1384 | pr_cont(")\n"); |
| 1385 | } | 1385 | } |
| @@ -1749,3 +1749,33 @@ static int __init init_cpu_syscore(void) · 363 unmodified lines | |||
| 1749 | return 0; | 1749 | return 0; |
| 1750 | } | 1750 | } |
| 1751 | core_initcall(init_cpu_syscore); | 1751 | core_initcall(init_cpu_syscore); |
| 1752 | |||
| 1753 | /* | ||
| 1754 | * The microcode loader calls this upon late microcode load to recheck features, | ||
| 1755 | * only when microcode has been updated. Caller holds microcode_mutex and CPU | ||
| 1756 | * hotplug lock. | ||
| 1757 | */ | ||
| 1758 | void microcode_check(void) | ||
| 1759 | { | ||
| 1760 | struct cpuinfo_x86 info; | ||
| 1761 | |||
| 1762 | perf_check_microcode(); | ||
| 1763 | |||
| 1764 | /* Reload CPUID max function as it might've changed. */ | ||
| 1765 | info.cpuid_level = cpuid_eax(0); | ||
| 1766 | |||
| 1767 | /* | ||
| 1768 | * Copy all capability leafs to pick up the synthetic ones so that | ||
| 1769 | * memcmp() below doesn't fail on that. The ones coming from CPUID will | ||
| 1770 | * get overwritten in get_cpu_cap(). | ||
| 1771 | */ | ||
| 1772 | memcpy(&info.x86_capability, &boot_cpu_data.x86_capability, sizeof(info.x86_capability)); | ||
| 1773 | |||
| 1774 | get_cpu_cap(&info); | ||
| 1775 | |||
| 1776 | if (!memcmp(&info.x86_capability, &boot_cpu_data.x86_capability, sizeof(info.x86_capability))) | ||
| 1777 | return; | ||
| 1778 | |||
| 1779 | pr_warn("x86/CPU: CPU features have changed after loading microcode, but might not take effect.\n"); | ||
| 1780 | pr_warn("x86/CPU: Please consider either early loading through initrd/built-in or a potential BIOS update.\n"); | ||
| 1781 | } | ||
| @@ -215,7 +215,7 @@ static void init_cyrix(struct cpuinfo_x86 *c) · 214 unmodified lines | |||
| 215 | 215 | ||
| 216 | /* common case step number/rev -- exceptions handled below */ | 216 | /* common case step number/rev -- exceptions handled below */ |
| 217 | c->x86_model = (dir1 >> 4) + 1; | 217 | c->x86_model = (dir1 >> 4) + 1; |
| 218 | c->x86_mask = dir1 & 0xf; | 218 | c->x86_stepping = dir1 & 0xf; |
| 219 | 219 | ||
| 220 | /* Now cook; the original recipe is by Channing Corn, from Cyrix. | 220 | /* Now cook; the original recipe is by Channing Corn, from Cyrix. |
| 221 | * We do the same thing for each generation: we work out | 221 | * We do the same thing for each generation: we work out |
| @@ -116,14 +116,13 @@ struct sku_microcode { · 115 unmodified lines | |||
| 116 | u32 microcode; | 116 | u32 microcode; |
| 117 | }; | 117 | }; |
| 118 | static const struct sku_microcode spectre_bad_microcodes[] = { | 118 | static const struct sku_microcode spectre_bad_microcodes[] = { |
| 119 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0B, 0x84 }, | 119 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0B, 0x80 }, |
| 120 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0A, 0x84 }, | 120 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x0A, 0x80 }, |
| 121 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x09, 0x84 }, | 121 | { INTEL_FAM6_KABYLAKE_DESKTOP, 0x09, 0x80 }, |
| 122 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x0A, 0x84 }, | 122 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x0A, 0x80 }, |
| 123 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x09, 0x84 }, | 123 | { INTEL_FAM6_KABYLAKE_MOBILE, 0x09, 0x80 }, |
| 124 | { INTEL_FAM6_SKYLAKE_X, 0x03, 0x0100013e }, | 124 | { INTEL_FAM6_SKYLAKE_X, 0x03, 0x0100013e }, |
| 125 | { INTEL_FAM6_SKYLAKE_X, 0x04, 0x0200003c }, | 125 | { INTEL_FAM6_SKYLAKE_X, 0x04, 0x0200003c }, |
| 126 | { INTEL_FAM6_SKYLAKE_MOBILE, 0x03, 0xc2 }, | ||
| 127 | { INTEL_FAM6_SKYLAKE_DESKTOP, 0x03, 0xc2 }, | 126 | { INTEL_FAM6_SKYLAKE_DESKTOP, 0x03, 0xc2 }, |
| 128 | { INTEL_FAM6_BROADWELL_CORE, 0x04, 0x28 }, | 127 | { INTEL_FAM6_BROADWELL_CORE, 0x04, 0x28 }, |
| 129 | { INTEL_FAM6_BROADWELL_GT3E, 0x01, 0x1b }, | 128 | { INTEL_FAM6_BROADWELL_GT3E, 0x01, 0x1b }, |
| @@ -136,8 +135,6 @@ static const struct sku_microcode spectre_bad_microcodes[] = { · 6 unmodified lines | |||
| 136 | { INTEL_FAM6_HASWELL_X, 0x02, 0x3b }, | 135 | { INTEL_FAM6_HASWELL_X, 0x02, 0x3b }, |
| 137 | { INTEL_FAM6_HASWELL_X, 0x04, 0x10 }, | 136 | { INTEL_FAM6_HASWELL_X, 0x04, 0x10 }, |
| 138 | { INTEL_FAM6_IVYBRIDGE_X, 0x04, 0x42a }, | 137 | { INTEL_FAM6_IVYBRIDGE_X, 0x04, 0x42a }, |
| 139 | /* Updated in the 20180108 release; blacklist until we know otherwise */ | ||
| 140 | { INTEL_FAM6_ATOM_GEMINI_LAKE, 0x01, 0x22 }, | ||
| 141 | /* Observed in the wild */ | 138 | /* Observed in the wild */ |
| 142 | { INTEL_FAM6_SANDYBRIDGE_X, 0x06, 0x61b }, | 139 | { INTEL_FAM6_SANDYBRIDGE_X, 0x06, 0x61b }, |
| 143 | { INTEL_FAM6_SANDYBRIDGE_X, 0x07, 0x712 }, | 140 | { INTEL_FAM6_SANDYBRIDGE_X, 0x07, 0x712 }, |
| @@ -147,9 +144,16 @@ static bool bad_spectre_microcode(struct cpuinfo_x86 *c) · 3 unmodified lines | |||
| 147 | { | 144 | { |
| 148 | int i; | 145 | int i; |
| 149 | 146 | ||
| 147 | /* | ||
| 148 | * We know that the hypervisor lie to us on the microcode version so | ||
| 149 | * we may as well hope that it is running the correct version. | ||
| 150 | */ | ||
| 151 | if (cpu_has(c, X86_FEATURE_HYPERVISOR)) | ||
| 152 | return false; | ||
| 153 | |||
| 150 | for (i = 0; i < ARRAY_SIZE(spectre_bad_microcodes); i++) { | 154 | for (i = 0; i < ARRAY_SIZE(spectre_bad_microcodes); i++) { |
| 151 | if (c->x86_model == spectre_bad_microcodes[i].model && | 155 | if (c->x86_model == spectre_bad_microcodes[i].model && |
| 152 | c->x86_mask == spectre_bad_microcodes[i].stepping) | 156 | c->x86_stepping == spectre_bad_microcodes[i].stepping) |
| 153 | return (c->microcode <= spectre_bad_microcodes[i].microcode); | 157 | return (c->microcode <= spectre_bad_microcodes[i].microcode); |
| 154 | } | 158 | } |
| 155 | return false; | 159 | return false; |
| @@ -196,7 +200,7 @@ static void early_init_intel(struct cpuinfo_x86 *c) · 40 unmodified lines | |||
| 196 | * need the microcode to have already been loaded... so if it is | 200 | * need the microcode to have already been loaded... so if it is |
| 197 | * not, recommend a BIOS update and disable large pages. | 201 | * not, recommend a BIOS update and disable large pages. |
| 198 | */ | 202 | */ |
| 199 | if (c->x86 == 6 && c->x86_model == 0x1c && c->x86_mask <= 2 && | 203 | if (c->x86 == 6 && c->x86_model == 0x1c && c->x86_stepping <= 2 && |
| 200 | c->microcode < 0x20e) { | 204 | c->microcode < 0x20e) { |
| 201 | pr_warn("Atom PSE erratum detected, BIOS microcode update recommended\n"); | 205 | pr_warn("Atom PSE erratum detected, BIOS microcode update recommended\n"); |
| 202 | clear_cpu_cap(c, X86_FEATURE_PSE); | 206 | clear_cpu_cap(c, X86_FEATURE_PSE); |
| @@ -212,7 +216,7 @@ static void early_init_intel(struct cpuinfo_x86 *c) · 9 unmodified lines | |||
| 212 | 216 | ||
| 213 | /* CPUID workaround for 0F33/0F34 CPU */ | 217 | /* CPUID workaround for 0F33/0F34 CPU */ |
| 214 | if (c->x86 == 0xF && c->x86_model == 0x3 | 218 | if (c->x86 == 0xF && c->x86_model == 0x3 |
| 215 | && (c->x86_mask == 0x3 || c->x86_mask == 0x4)) | 219 | && (c->x86_stepping == 0x3 || c->x86_stepping == 0x4)) |
| 216 | c->x86_phys_bits = 36; | 220 | c->x86_phys_bits = 36; |
| 217 | 221 | ||
| 218 | /* | 222 | /* |
| @@ -310,7 +314,7 @@ int ppro_with_ram_bug(void) · 91 unmodified lines | |||
| 310 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && | 314 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && |
| 311 | boot_cpu_data.x86 == 6 && | 315 | boot_cpu_data.x86 == 6 && |
| 312 | boot_cpu_data.x86_model == 1 && | 316 | boot_cpu_data.x86_model == 1 && |
| 313 | boot_cpu_data.x86_mask < 8) { | 317 | boot_cpu_data.x86_stepping < 8) { |
| 314 | pr_info("Pentium Pro with Errata#50 detected. Taking evasive action.\n"); | 318 | pr_info("Pentium Pro with Errata#50 detected. Taking evasive action.\n"); |
| 315 | return 1; | 319 | return 1; |
| 316 | } | 320 | } |
| @@ -327,7 +331,7 @@ static void intel_smp_check(struct cpuinfo_x86 *c) · 10 unmodified lines | |||
| 327 | * Mask B, Pentium, but not Pentium MMX | 331 | * Mask B, Pentium, but not Pentium MMX |
| 328 | */ | 332 | */ |
| 329 | if (c->x86 == 5 && | 333 | if (c->x86 == 5 && |
| 330 | c->x86_mask >= 1 && c->x86_mask <= 4 && | 334 | c->x86_stepping >= 1 && c->x86_stepping <= 4 && |
| 331 | c->x86_model <= 3) { | 335 | c->x86_model <= 3) { |
| 332 | /* | 336 | /* |
| 333 | * Remember we have B step Pentia with bugs | 337 | * Remember we have B step Pentia with bugs |
| @@ -370,7 +374,7 @@ static void intel_workarounds(struct cpuinfo_x86 *c) · 36 unmodified lines | |||
| 370 | * SEP CPUID bug: Pentium Pro reports SEP but doesn't have it until | 374 | * SEP CPUID bug: Pentium Pro reports SEP but doesn't have it until |
| 371 | * model 3 mask 3 | 375 | * model 3 mask 3 |
| 372 | */ | 376 | */ |
| 373 | if ((c->x86<<8 | c->x86_model<<4 | c->x86_mask) < 0x633) | 377 | if ((c->x86<<8 | c->x86_model<<4 | c->x86_stepping) < 0x633) |
| 374 | clear_cpu_cap(c, X86_FEATURE_SEP); | 378 | clear_cpu_cap(c, X86_FEATURE_SEP); |
| 375 | 379 | ||
| 376 | /* | 380 | /* |
| @@ -388,7 +392,7 @@ static void intel_workarounds(struct cpuinfo_x86 *c) · 11 unmodified lines | |||
| 388 | * P4 Xeon erratum 037 workaround. | 392 | * P4 Xeon erratum 037 workaround. |
| 389 | * Hardware prefetcher may cause stale data to be loaded into the cache. | 393 | * Hardware prefetcher may cause stale data to be loaded into the cache. |
| 390 | */ | 394 | */ |
| 391 | if ((c->x86 == 15) && (c->x86_model == 1) && (c->x86_mask == 1)) { | 395 | if ((c->x86 == 15) && (c->x86_model == 1) && (c->x86_stepping == 1)) { |
| 392 | if (msr_set_bit(MSR_IA32_MISC_ENABLE, | 396 | if (msr_set_bit(MSR_IA32_MISC_ENABLE, |
| 393 | MSR_IA32_MISC_ENABLE_PREFETCH_DISABLE_BIT) > 0) { | 397 | MSR_IA32_MISC_ENABLE_PREFETCH_DISABLE_BIT) > 0) { |
| 394 | pr_info("CPU: C0 stepping P4 Xeon detected.\n"); | 398 | pr_info("CPU: C0 stepping P4 Xeon detected.\n"); |
| @@ -403,7 +407,7 @@ static void intel_workarounds(struct cpuinfo_x86 *c) · 8 unmodified lines | |||
| 403 | * Specification Update"). | 407 | * Specification Update"). |
| 404 | */ | 408 | */ |
| 405 | if (boot_cpu_has(X86_FEATURE_APIC) && (c->x86<<8 | c->x86_model<<4) == 0x520 && | 409 | if (boot_cpu_has(X86_FEATURE_APIC) && (c->x86<<8 | c->x86_model<<4) == 0x520 && |
| 406 | (c->x86_mask < 0x6 || c->x86_mask == 0xb)) | 410 | (c->x86_stepping < 0x6 || c->x86_stepping == 0xb)) |
| 407 | set_cpu_bug(c, X86_BUG_11AP); | 411 | set_cpu_bug(c, X86_BUG_11AP); |
| 408 | 412 | ||
| 409 | 413 | ||
| @@ -650,7 +654,7 @@ static void init_intel(struct cpuinfo_x86 *c) · 240 unmodified lines | |||
| 650 | case 6: | 654 | case 6: |
| 651 | if (l2 == 128) | 655 | if (l2 == 128) |
| 652 | p = "Celeron (Mendocino)"; | 656 | p = "Celeron (Mendocino)"; |
| 653 | else if (c->x86_mask == 0 || c->x86_mask == 5) | 657 | else if (c->x86_stepping == 0 || c->x86_stepping == 5) |
| 654 | p = "Celeron-A"; | 658 | p = "Celeron-A"; |
| 655 | break; | 659 | break; |
| 656 | 660 | ||
| @@ -819,7 +819,7 @@ static __init void rdt_quirks(void) · 818 unmodified lines | |||
| 819 | cache_alloc_hsw_probe(); | 819 | cache_alloc_hsw_probe(); |
| 820 | break; | 820 | break; |
| 821 | case INTEL_FAM6_SKYLAKE_X: | 821 | case INTEL_FAM6_SKYLAKE_X: |
| 822 | if (boot_cpu_data.x86_mask <= 4) | 822 | if (boot_cpu_data.x86_stepping <= 4) |
| 823 | set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat"); | 823 | set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat"); |
| 824 | } | 824 | } |
| 825 | } | 825 | } |
| @@ -1804,6 +1804,7 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, · 1803 unmodified lines | |||
| 1804 | goto out_common_fail; | 1804 | goto out_common_fail; |
| 1805 | } | 1805 | } |
| 1806 | closid = ret; | 1806 | closid = ret; |
| 1807 | ret = 0; | ||
| 1807 | 1808 | ||
| 1808 | rdtgrp->closid = closid; | 1809 | rdtgrp->closid = closid; |
| 1809 | list_add(&rdtgrp->rdtgroup_list, &rdt_all_groups); | 1810 | list_add(&rdtgrp->rdtgroup_list, &rdt_all_groups); |
| @@ -115,4 +115,19 @@ static inline void mce_unregister_injector_chain(struct notifier_block *nb) { } · 114 unmodified lines | |||
| 115 | 115 | ||
| 116 | extern struct mca_config mca_cfg; | 116 | extern struct mca_config mca_cfg; |
| 117 | 117 | ||
| 118 | #ifndef CONFIG_X86_64 | ||
| 119 | /* | ||
| 120 | * On 32-bit systems it would be difficult to safely unmap a poison page | ||
| 121 | * from the kernel 1:1 map because there are no non-canonical addresses that | ||
| 122 | * we can use to refer to the address without risking a speculative access. | ||
| 123 | * However, this isn't much of an issue because: | ||
| 124 | * 1) Few unmappable pages are in the 1:1 map. Most are in HIGHMEM which | ||
| 125 | * are only mapped into the kernel as needed | ||
| 126 | * 2) Few people would run a 32-bit kernel on a machine that supports | ||
| 127 | * recoverable errors because they have too much memory to boot 32-bit. | ||
| 128 | */ | ||
| 129 | static inline void mce_unmap_kpfn(unsigned long pfn) {} | ||
| 130 | #define mce_unmap_kpfn mce_unmap_kpfn | ||
| 131 | #endif | ||
| 132 | |||
| 118 | #endif /* __X86_MCE_INTERNAL_H__ */ | 133 | #endif /* __X86_MCE_INTERNAL_H__ */ |
| @@ -56,6 +56,9 @@ · 55 unmodified lines | |||
| 56 | 56 | ||
| 57 | static DEFINE_MUTEX(mce_log_mutex); | 57 | static DEFINE_MUTEX(mce_log_mutex); |
| 58 | 58 | ||
| 59 | /* sysfs synchronization */ | ||
| 60 | static DEFINE_MUTEX(mce_sysfs_mutex); | ||
| 61 | |||
| 59 | #define CREATE_TRACE_POINTS | 62 | #define CREATE_TRACE_POINTS |
| 60 | #include <trace/events/mce.h> | 63 | #include <trace/events/mce.h> |
| 61 | 64 | ||
| @@ -105,6 +108,10 @@ static struct irq_work mce_irq_work; · 43 unmodified lines | |||
| 105 | 108 | ||
| 106 | static void (*quirk_no_way_out)(int bank, struct mce *m, struct pt_regs *regs); | 109 | static void (*quirk_no_way_out)(int bank, struct mce *m, struct pt_regs *regs); |
| 107 | 110 | ||
| 111 | #ifndef mce_unmap_kpfn | ||
| 112 | static void mce_unmap_kpfn(unsigned long pfn); | ||
| 113 | #endif | ||
| 114 | |||
| 108 | /* | 115 | /* |
| 109 | * CPU/chipset specific EDAC code can register a notifier call here to print | 116 | * CPU/chipset specific EDAC code can register a notifier call here to print |
| 110 | * MCE errors in a human-readable form. | 117 | * MCE errors in a human-readable form. |
| @@ -126,6 +133,8 @@ void mce_setup(struct mce *m) · 15 unmodified lines | |||
| 126 | 133 | ||
| 127 | if (this_cpu_has(X86_FEATURE_INTEL_PPIN)) | 134 | if (this_cpu_has(X86_FEATURE_INTEL_PPIN)) |
| 128 | rdmsrl(MSR_PPIN, m->ppin); | 135 | rdmsrl(MSR_PPIN, m->ppin); |
| 136 | |||
| 137 | m->microcode = boot_cpu_data.microcode; | ||
| 129 | } | 138 | } |
| 130 | 139 | ||
| 131 | DEFINE_PER_CPU(struct mce, injectm); | 140 | DEFINE_PER_CPU(struct mce, injectm); |
| @@ -234,7 +243,7 @@ static void __print_mce(struct mce *m) · 102 unmodified lines | |||
| 234 | m->cs, m->ip); | 243 | m->cs, m->ip); |
| 235 | 244 | ||
| 236 | if (m->cs == __KERNEL_CS) | 245 | if (m->cs == __KERNEL_CS) |
| 237 | pr_cont("{%pS}", (void *)m->ip); | 246 | pr_cont("{%pS}", (void *)(unsigned long)m->ip); |
| 238 | pr_cont("\n"); | 247 | pr_cont("\n"); |
| 239 | } | 248 | } |
| 240 | 249 | ||
| @@ -258,7 +267,7 @@ static void __print_mce(struct mce *m) · 17 unmodified lines | |||
| 258 | */ | 267 | */ |
| 259 | pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x microcode %x\n", | 268 | pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x microcode %x\n", |
| 260 | m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid, | 269 | m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid, |
| 261 | cpu_data(m->extcpu).microcode); | 270 | m->microcode); |
| 262 | } | 271 | } |
| 263 | 272 | ||
| 264 | static void print_mce(struct mce *m) | 273 | static void print_mce(struct mce *m) |
| @@ -590,7 +599,8 @@ static int srao_decode_notifier(struct notifier_block *nb, unsigned long val, · 325 unmodified lines | |||
| 590 | 599 | ||
| 591 | if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { | 600 | if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { |
| 592 | pfn = mce->addr >> PAGE_SHIFT; | 601 | pfn = mce->addr >> PAGE_SHIFT; |
| 593 | memory_failure(pfn, 0); | 602 | if (!memory_failure(pfn, 0)) |
| 603 | mce_unmap_kpfn(pfn); | ||
| 594 | } | 604 | } |
| 595 | 605 | ||
| 596 | return NOTIFY_OK; | 606 | return NOTIFY_OK; |
| @@ -1057,12 +1067,13 @@ static int do_memory_failure(struct mce *m) · 460 unmodified lines | |||
| 1057 | ret = memory_failure(m->addr >> PAGE_SHIFT, flags); | 1067 | ret = memory_failure(m->addr >> PAGE_SHIFT, flags); |
| 1058 | if (ret) | 1068 | if (ret) |
| 1059 | pr_err("Memory error not recovered"); | 1069 | pr_err("Memory error not recovered"); |
| 1070 | else | ||
| 1071 | mce_unmap_kpfn(m->addr >> PAGE_SHIFT); | ||
| 1060 | return ret; | 1072 | return ret; |
| 1061 | } | 1073 | } |
| 1062 | 1074 | ||
| 1063 | #if defined(arch_unmap_kpfn) && defined(CONFIG_MEMORY_FAILURE) | 1075 | #ifndef mce_unmap_kpfn |
| 1064 | 1076 | static void mce_unmap_kpfn(unsigned long pfn) | |
| 1065 | void arch_unmap_kpfn(unsigned long pfn) | ||
| 1066 | { | 1077 | { |
| 1067 | unsigned long decoy_addr; | 1078 | unsigned long decoy_addr; |
| 1068 | 1079 | ||
| @@ -1073,7 +1084,7 @@ void arch_unmap_kpfn(unsigned long pfn) · 4 unmodified lines | |||
| 1073 | * We would like to just call: | 1084 | * We would like to just call: |
| 1074 | * set_memory_np((unsigned long)pfn_to_kaddr(pfn), 1); | 1085 | * set_memory_np((unsigned long)pfn_to_kaddr(pfn), 1); |
| 1075 | * but doing that would radically increase the odds of a | 1086 | * but doing that would radically increase the odds of a |
| 1076 | * speculative access to the posion page because we'd have | 1087 | * speculative access to the poison page because we'd have |
| 1077 | * the virtual address of the kernel 1:1 mapping sitting | 1088 | * the virtual address of the kernel 1:1 mapping sitting |
| 1078 | * around in registers. | 1089 | * around in registers. |
| 1079 | * Instead we get tricky. We create a non-canonical address | 1090 | * Instead we get tricky. We create a non-canonical address |
| @@ -1098,7 +1109,6 @@ void arch_unmap_kpfn(unsigned long pfn) · 18 unmodified lines | |||
| 1098 | 1109 | ||
| 1099 | if (set_memory_np(decoy_addr, 1)) | 1110 | if (set_memory_np(decoy_addr, 1)) |
| 1100 | pr_warn("Could not invalidate pfn=0x%lx from 1:1 map\n", pfn); | 1111 | pr_warn("Could not invalidate pfn=0x%lx from 1:1 map\n", pfn); |
| 1101 | |||
| 1102 | } | 1112 | } |
| 1103 | #endif | 1113 | #endif |
| 1104 | 1114 | ||
| @@ -2081,6 +2091,7 @@ static ssize_t set_ignore_ce(struct device *s, · 976 unmodified lines | |||
| 2081 | if (kstrtou64(buf, 0, &new) < 0) | 2091 | if (kstrtou64(buf, 0, &new) < 0) |
| 2082 | return -EINVAL; | 2092 | return -EINVAL; |
| 2083 | 2093 | ||
| 2094 | mutex_lock(&mce_sysfs_mutex); | ||
| 2084 | if (mca_cfg.ignore_ce ^ !!new) { | 2095 | if (mca_cfg.ignore_ce ^ !!new) { |
| 2085 | if (new) { | 2096 | if (new) { |
| 2086 | /* disable ce features */ | 2097 | /* disable ce features */ |
| @@ -2093,6 +2104,8 @@ static ssize_t set_ignore_ce(struct device *s, · 6 unmodified lines | |||
| 2093 | on_each_cpu(mce_enable_ce, (void *)1, 1); | 2104 | on_each_cpu(mce_enable_ce, (void *)1, 1); |
| 2094 | } | 2105 | } |
| 2095 | } | 2106 | } |
| 2107 | mutex_unlock(&mce_sysfs_mutex); | ||
| 2108 | |||
| 2096 | return size; | 2109 | return size; |
| 2097 | } | 2110 | } |
| 2098 | 2111 | ||
| @@ -2105,6 +2118,7 @@ static ssize_t set_cmci_disabled(struct device *s, · 6 unmodified lines | |||
| 2105 | if (kstrtou64(buf, 0, &new) < 0) | 2118 | if (kstrtou64(buf, 0, &new) < 0) |
| 2106 | return -EINVAL; | 2119 | return -EINVAL; |
| 2107 | 2120 | ||
| 2121 | mutex_lock(&mce_sysfs_mutex); | ||
| 2108 | if (mca_cfg.cmci_disabled ^ !!new) { | 2122 | if (mca_cfg.cmci_disabled ^ !!new) { |
| 2109 | if (new) { | 2123 | if (new) { |
| 2110 | /* disable cmci */ | 2124 | /* disable cmci */ |
| @@ -2116,6 +2130,8 @@ static ssize_t set_cmci_disabled(struct device *s, · 5 unmodified lines | |||
| 2116 | on_each_cpu(mce_enable_ce, NULL, 1); | 2130 | on_each_cpu(mce_enable_ce, NULL, 1); |
| 2117 | } | 2131 | } |
| 2118 | } | 2132 | } |
| 2133 | mutex_unlock(&mce_sysfs_mutex); | ||
| 2134 | |||
| 2119 | return size; | 2135 | return size; |
| 2120 | } | 2136 | } |
| 2121 | 2137 | ||
| @@ -2123,8 +2139,19 @@ static ssize_t store_int_with_restart(struct device *s, · 1 unmodified line | |||
| 2123 | struct device_attribute *attr, | 2139 | struct device_attribute *attr, |
| 2124 | const char *buf, size_t size) | 2140 | const char *buf, size_t size) |
| 2125 | { | 2141 | { |
| 2126 | ssize_t ret = device_store_int(s, attr, buf, size); | 2142 | unsigned long old_check_interval = check_interval; |
| 2143 | ssize_t ret = device_store_ulong(s, attr, buf, size); | ||
| 2144 | |||
| 2145 | if (check_interval == old_check_interval) | ||
| 2146 | return ret; | ||
| 2147 | |||
| 2148 | if (check_interval < 1) | ||
| 2149 | check_interval = 1; | ||
| 2150 | |||
| 2151 | mutex_lock(&mce_sysfs_mutex); | ||
| 2127 | mce_restart(); | 2152 | mce_restart(); |
| 2153 | mutex_unlock(&mce_sysfs_mutex); | ||
| 2154 | |||
| 2128 | return ret; | 2155 | return ret; |
| 2129 | } | 2156 | } |
| 2130 | 2157 | ||
| @@ -498,7 +498,7 @@ static unsigned int verify_patch_size(u8 family, u32 patch_size, · 497 unmodified lines | |||
| 498 | return patch_size; | 498 | return patch_size; |
| 499 | } | 499 | } |
| 500 | 500 | ||
| 501 | static int apply_microcode_amd(int cpu) | 501 | static enum ucode_state apply_microcode_amd(int cpu) |
| 502 | { | 502 | { |
| 503 | struct cpuinfo_x86 *c = &cpu_data(cpu); | 503 | struct cpuinfo_x86 *c = &cpu_data(cpu); |
| 504 | struct microcode_amd *mc_amd; | 504 | struct microcode_amd *mc_amd; |
| @@ -512,7 +512,7 @@ static int apply_microcode_amd(int cpu) · 7 unmodified lines | |||
| 512 | 512 | ||
| 513 | p = find_patch(cpu); | 513 | p = find_patch(cpu); |
| 514 | if (!p) | 514 | if (!p) |
| 515 | return 0; | 515 | return UCODE_NFOUND; |
| 516 | 516 | ||
| 517 | mc_amd = p->data; | 517 | mc_amd = p->data; |
| 518 | uci->mc = p->data; | 518 | uci->mc = p->data; |
| @@ -523,13 +523,13 @@ static int apply_microcode_amd(int cpu) · 4 unmodified lines | |||
| 523 | if (rev >= mc_amd->hdr.patch_id) { | 523 | if (rev >= mc_amd->hdr.patch_id) { |
| 524 | c->microcode = rev; | 524 | c->microcode = rev; |
| 525 | uci->cpu_sig.rev = rev; | 525 | uci->cpu_sig.rev = rev; |
| 526 | return 0; | 526 | return UCODE_OK; |
| 527 | } | 527 | } |
| 528 | 528 | ||
| 529 | if (__apply_microcode_amd(mc_amd)) { | 529 | if (__apply_microcode_amd(mc_amd)) { |
| 530 | pr_err("CPU%d: update failed for patch_level=0x%08x\n", | 530 | pr_err("CPU%d: update failed for patch_level=0x%08x\n", |
| 531 | cpu, mc_amd->hdr.patch_id); | 531 | cpu, mc_amd->hdr.patch_id); |
| 532 | return -1; | 532 | return UCODE_ERROR; |
| 533 | } | 533 | } |
| 534 | pr_info("CPU%d: new patch_level=0x%08x\n", cpu, | 534 | pr_info("CPU%d: new patch_level=0x%08x\n", cpu, |
| 535 | mc_amd->hdr.patch_id); | 535 | mc_amd->hdr.patch_id); |
| @@ -537,7 +537,7 @@ static int apply_microcode_amd(int cpu) · 1 unmodified line | |||
| 537 | uci->cpu_sig.rev = mc_amd->hdr.patch_id; | 537 | uci->cpu_sig.rev = mc_amd->hdr.patch_id; |
| 538 | c->microcode = mc_amd->hdr.patch_id; | 538 | c->microcode = mc_amd->hdr.patch_id; |
| 539 | 539 | ||
| 540 | return 0; | 540 | return UCODE_UPDATED; |
| 541 | } | 541 | } |
| 542 | 542 | ||
| 543 | static int install_equiv_cpu_table(const u8 *buf) | 543 | static int install_equiv_cpu_table(const u8 *buf) |
| @@ -22,13 +22,16 @@ · 21 unmodified lines | |||
| 22 | #define pr_fmt(fmt) "microcode: " fmt | 22 | #define pr_fmt(fmt) "microcode: " fmt |
| 23 | 23 | ||
| 24 | #include <linux/platform_device.h> | 24 | #include <linux/platform_device.h> |
| 25 | #include <linux/stop_machine.h> | ||
| 25 | #include <linux/syscore_ops.h> | 26 | #include <linux/syscore_ops.h> |
| 26 | #include <linux/miscdevice.h> | 27 | #include <linux/miscdevice.h> |
| 27 | #include <linux/capability.h> | 28 | #include <linux/capability.h> |
| 28 | #include <linux/firmware.h> | 29 | #include <linux/firmware.h> |
| 29 | #include <linux/kernel.h> | 30 | #include <linux/kernel.h> |
| 31 | #include <linux/delay.h> | ||
| 30 | #include <linux/mutex.h> | 32 | #include <linux/mutex.h> |
| 31 | #include <linux/cpu.h> | 33 | #include <linux/cpu.h> |
| 34 | #include <linux/nmi.h> | ||
| 32 | #include <linux/fs.h> | 35 | #include <linux/fs.h> |
| 33 | #include <linux/mm.h> | 36 | #include <linux/mm.h> |
| 34 | 37 | ||
| @@ -64,6 +67,11 @@ LIST_HEAD(microcode_cache); · 29 unmodified lines | |||
| 64 | */ | 67 | */ |
| 65 | static DEFINE_MUTEX(microcode_mutex); | 68 | static DEFINE_MUTEX(microcode_mutex); |
| 66 | 69 | ||
| 70 | /* | ||
| 71 | * Serialize late loading so that CPUs get updated one-by-one. | ||
| 72 | */ | ||
| 73 | static DEFINE_SPINLOCK(update_lock); | ||
| 74 | |||
| 67 | struct ucode_cpu_info ucode_cpu_info[NR_CPUS]; | 75 | struct ucode_cpu_info ucode_cpu_info[NR_CPUS]; |
| 68 | 76 | ||
| 69 | struct cpu_info_ctx { | 77 | struct cpu_info_ctx { |
| @@ -373,26 +381,23 @@ static int collect_cpu_info(int cpu) · 303 unmodified lines | |||
| 373 | return ret; | 381 | return ret; |
| 374 | } | 382 | } |
| 375 | 383 | ||
| 376 | struct apply_microcode_ctx { | ||
| 377 | int err; | ||
| 378 | }; | ||
| 379 | |||
| 380 | static void apply_microcode_local(void *arg) | 384 | static void apply_microcode_local(void *arg) |
| 381 | { | 385 | { |
| 382 | struct apply_microcode_ctx *ctx = arg; | 386 | enum ucode_state *err = arg; |
| 383 | 387 | ||
| 384 | ctx->err = microcode_ops->apply_microcode(smp_processor_id()); | 388 | *err = microcode_ops->apply_microcode(smp_processor_id()); |
| 385 | } | 389 | } |
| 386 | 390 | ||
| 387 | static int apply_microcode_on_target(int cpu) | 391 | static int apply_microcode_on_target(int cpu) |
| 388 | { | 392 | { |
| 389 | struct apply_microcode_ctx ctx = { .err = 0 }; | 393 | enum ucode_state err; |
| 390 | int ret; | 394 | int ret; |
| 391 | 395 | ||
| 392 | ret = smp_call_function_single(cpu, apply_microcode_local, &ctx, 1); | 396 | ret = smp_call_function_single(cpu, apply_microcode_local, &err, 1); |
| 393 | if (!ret) | 397 | if (!ret) { |
| 394 | ret = ctx.err; | 398 | if (err == UCODE_ERROR) |
| 395 | 399 | ret = 1; | |
| 400 | } | ||
| 396 | return ret; | 401 | return ret; |
| 397 | } | 402 | } |
| 398 | 403 | ||
| @@ -489,31 +494,110 @@ static void __exit microcode_dev_exit(void) · 90 unmodified lines | |||
| 489 | /* fake device for request_firmware */ | 494 | /* fake device for request_firmware */ |
| 490 | static struct platform_device *microcode_pdev; | 495 | static struct platform_device *microcode_pdev; |
| 491 | 496 | ||
| 492 | static int reload_for_cpu(int cpu) | 497 | /* |
| 498 | * Late loading dance. Why the heavy-handed stomp_machine effort? | ||
| 499 | * | ||
| 500 | * - HT siblings must be idle and not execute other code while the other sibling | ||
| 501 | * is loading microcode in order to avoid any negative interactions caused by | ||
| 502 | * the loading. | ||
| 503 | * | ||
| 504 | * - In addition, microcode update on the cores must be serialized until this | ||
| 505 | * requirement can be relaxed in the future. Right now, this is conservative | ||
| 506 | * and good. | ||
| 507 | */ | ||
| 508 | #define SPINUNIT 100 /* 100 nsec */ | ||
| 509 | |||
| 510 | static int check_online_cpus(void) | ||
| 493 | { | 511 | { |
| 494 | struct ucode_cpu_info *uci = ucode_cpu_info + cpu; | 512 | if (num_online_cpus() == num_present_cpus()) |
| 495 | enum ucode_state ustate; | 513 | return 0; |
| 496 | int err = 0; | ||
| 497 | 514 | ||
| 498 | if (!uci->valid) | 515 | pr_err("Not all CPUs online, aborting microcode update.\n"); |
| 499 | return err; | ||
| 500 | 516 | ||
| 501 | ustate = microcode_ops->request_microcode_fw(cpu, µcode_pdev->dev, true); | 517 | return -EINVAL; |
| 502 | if (ustate == UCODE_OK) | 518 | } |
| 503 | apply_microcode_on_target(cpu); | 519 | |
| 504 | else | 520 | static atomic_t late_cpus; |
| 505 | if (ustate == UCODE_ERROR) | 521 | |
| 506 | err = -EINVAL; | 522 | /* |
| 507 | return err; | 523 | * Returns: |
| 524 | * < 0 - on error | ||
| 525 | * 0 - no update done | ||
| 526 | * 1 - microcode was updated | ||
| 527 | */ | ||
| 528 | static int __reload_late(void *info) | ||
| 529 | { | ||
| 530 | unsigned int timeout = NSEC_PER_SEC; | ||
| 531 | int all_cpus = num_online_cpus(); | ||
| 532 | int cpu = smp_processor_id(); | ||
| 533 | enum ucode_state err; | ||
| 534 | int ret = 0; | ||
| 535 | |||
| 536 | atomic_dec(&late_cpus); | ||
| 537 | |||
| 538 | /* | ||
| 539 | * Wait for all CPUs to arrive. A load will not be attempted unless all | ||
| 540 | * CPUs show up. | ||
| 541 | * */ | ||
| 542 | while (atomic_read(&late_cpus)) { | ||
| 543 | if (timeout < SPINUNIT) { | ||
| 544 | pr_err("Timeout while waiting for CPUs rendezvous, remaining: %d\n", | ||
| 545 | atomic_read(&late_cpus)); | ||
| 546 | return -1; | ||
| 547 | } | ||
| 548 | |||
| 549 | ndelay(SPINUNIT); | ||
| 550 | timeout -= SPINUNIT; | ||
| 551 | |||
| 552 | touch_nmi_watchdog(); | ||
| 553 | } | ||
| 554 | |||
| 555 | spin_lock(&update_lock); | ||
| 556 | apply_microcode_local(&err); | ||
| 557 | spin_unlock(&update_lock); | ||
| 558 | |||
| 559 | if (err > UCODE_NFOUND) { | ||
| 560 | pr_warn("Error reloading microcode on CPU %d\n", cpu); | ||
| 561 | ret = -1; | ||
| 562 | } else if (err == UCODE_UPDATED) { | ||
| 563 | ret = 1; | ||
| 564 | } | ||
| 565 | |||
| 566 | atomic_inc(&late_cpus); | ||
| 567 | |||
| 568 | while (atomic_read(&late_cpus) != all_cpus) | ||
| 569 | cpu_relax(); | ||
| 570 | |||
| 571 | return ret; | ||
| 572 | } | ||
| 573 | |||
| 574 | /* | ||
| 575 | * Reload microcode late on all CPUs. Wait for a sec until they | ||
| 576 | * all gather together. | ||
| 577 | */ | ||
| 578 | static int microcode_reload_late(void) | ||
| 579 | { | ||
| 580 | int ret; | ||
| 581 | |||
| 582 | atomic_set(&late_cpus, num_online_cpus()); | ||
| 583 | |||
| 584 | ret = stop_machine_cpuslocked(__reload_late, NULL, cpu_online_mask); | ||
| 585 | if (ret < 0) | ||
| 586 | return ret; | ||
| 587 | else if (ret > 0) | ||
| 588 | microcode_check(); | ||
| 589 | |||
| 590 | return ret; | ||
| 508 | } | 591 | } |
| 509 | 592 | ||
| 510 | static ssize_t reload_store(struct device *dev, | 593 | static ssize_t reload_store(struct device *dev, |
| 511 | struct device_attribute *attr, | 594 | struct device_attribute *attr, |
| 512 | const char *buf, size_t size) | 595 | const char *buf, size_t size) |
| 513 | { | 596 | { |
| 597 | enum ucode_state tmp_ret = UCODE_OK; | ||
| 598 | int bsp = boot_cpu_data.cpu_index; | ||
| 514 | unsigned long val; | 599 | unsigned long val; |
| 515 | int cpu; | 600 | ssize_t ret = 0; |
| 516 | ssize_t ret = 0, tmp_ret; | ||
| 517 | 601 | ||
| 518 | ret = kstrtoul(buf, 0, &val); | 602 | ret = kstrtoul(buf, 0, &val); |
| 519 | if (ret) | 603 | if (ret) |
| @@ -522,23 +606,24 @@ static ssize_t reload_store(struct device *dev, · 2 unmodified lines | |||
| 522 | if (val != 1) | 606 | if (val != 1) |
| 523 | return size; | 607 | return size; |
| 524 | 608 | ||
| 609 | tmp_ret = microcode_ops->request_microcode_fw(bsp, µcode_pdev->dev, true); | ||
| 610 | if (tmp_ret != UCODE_OK) | ||
| 611 | return size; | ||
| 612 | |||
| 525 | get_online_cpus(); | 613 | get_online_cpus(); |
| 526 | mutex_lock(µcode_mutex); | ||
| 527 | for_each_online_cpu(cpu) { | ||
| 528 | tmp_ret = reload_for_cpu(cpu); | ||
| 529 | if (tmp_ret != 0) | ||
| 530 | pr_warn("Error reloading microcode on CPU %d\n", cpu); | ||
| 531 | 614 | ||
| 532 | /* save retval of the first encountered reload error */ | 615 | ret = check_online_cpus(); |
| 533 | if (!ret) | 616 | if (ret) |
| 534 | ret = tmp_ret; | 617 | goto put; |
| 535 | } | 618 | |
| 536 | if (!ret) | 619 | mutex_lock(µcode_mutex); |
| 537 | perf_check_microcode(); | 620 | ret = microcode_reload_late(); |
| 538 | mutex_unlock(µcode_mutex); | 621 | mutex_unlock(µcode_mutex); |
| 622 | |||
| 623 | put: | ||
| 539 | put_online_cpus(); | 624 | put_online_cpus(); |
| 540 | 625 | ||
| 541 | if (!ret) | 626 | if (ret >= 0) |
| 542 | ret = size; | 627 | ret = size; |
| 543 | 628 | ||
| 544 | return ret; | 629 | return ret; |
| @@ -589,6 +589,23 @@ static int apply_microcode_early(struct ucode_cpu_info *uci, bool early) · 588 unmodified lines | |||
| 589 | if (!mc) | 589 | if (!mc) |
| 590 | return 0; | 590 | return 0; |
| 591 | 591 | ||
| 592 | /* | ||
| 593 | * Save us the MSR write below - which is a particular expensive | ||
| 594 | * operation - when the other hyperthread has updated the microcode | ||
| 595 | * already. | ||
| 596 | */ | ||
| 597 | rev = intel_get_microcode_revision(); | ||
| 598 | if (rev >= mc->hdr.rev) { | ||
| 599 | uci->cpu_sig.rev = rev; | ||
| 600 | return UCODE_OK; | ||
| 601 | } | ||
| 602 | |||
| 603 | /* | ||
| 604 | * Writeback and invalidate caches before updating microcode to avoid | ||
| 605 | * internal issues depending on what the microcode is updating. | ||
| 606 | */ | ||
| 607 | native_wbinvd(); | ||
| 608 | |||
| 592 | /* write microcode via MSR 0x79 */ | 609 | /* write microcode via MSR 0x79 */ |
| 593 | native_wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); | 610 | native_wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); |
| 594 | 611 | ||
| @@ -772,27 +789,44 @@ static int collect_cpu_info(int cpu_num, struct cpu_signature *csig) · 177 unmodified lines | |||
| 772 | return 0; | 789 | return 0; |
| 773 | } | 790 | } |
| 774 | 791 | ||
| 775 | static int apply_microcode_intel(int cpu) | 792 | static enum ucode_state apply_microcode_intel(int cpu) |
| 776 | { | 793 | { |
| 794 | struct ucode_cpu_info *uci = ucode_cpu_info + cpu; | ||
| 795 | struct cpuinfo_x86 *c = &cpu_data(cpu); | ||
| 777 | struct microcode_intel *mc; | 796 | struct microcode_intel *mc; |
| 778 | struct ucode_cpu_info *uci; | ||
| 779 | struct cpuinfo_x86 *c; | ||
| 780 | static int prev_rev; | 797 | static int prev_rev; |
| 781 | u32 rev; | 798 | u32 rev; |
| 782 | 799 | ||
| 783 | /* We should bind the task to the CPU */ | 800 | /* We should bind the task to the CPU */ |
| 784 | if (WARN_ON(raw_smp_processor_id() != cpu)) | 801 | if (WARN_ON(raw_smp_processor_id() != cpu)) |
| 785 | return -1; | 802 | return UCODE_ERROR; |
| 786 | 803 | ||
| 787 | uci = ucode_cpu_info + cpu; | 804 | /* Look for a newer patch in our cache: */ |
| 788 | mc = uci->mc; | 805 | mc = find_patch(uci); |
| 789 | if (!mc) { | 806 | if (!mc) { |
| 790 | /* Look for a newer patch in our cache: */ | 807 | mc = uci->mc; |
| 791 | mc = find_patch(uci); | ||
| 792 | if (!mc) | 808 | if (!mc) |
| 793 | return 0; | 809 | return UCODE_NFOUND; |
| 794 | } | 810 | } |
| 795 | 811 | ||
| 812 | /* | ||
| 813 | * Save us the MSR write below - which is a particular expensive | ||
| 814 | * operation - when the other hyperthread has updated the microcode | ||
| 815 | * already. | ||
| 816 | */ | ||
| 817 | rev = intel_get_microcode_revision(); | ||
| 818 | if (rev >= mc->hdr.rev) { | ||
| 819 | uci->cpu_sig.rev = rev; | ||
| 820 | c->microcode = rev; | ||
| 821 | return UCODE_OK; | ||
| 822 | } | ||
| 823 | |||
| 824 | /* | ||
| 825 | * Writeback and invalidate caches before updating microcode to avoid | ||
| 826 | * internal issues depending on what the microcode is updating. | ||
| 827 | */ | ||
| 828 | native_wbinvd(); | ||
| 829 | |||
| 796 | /* write microcode via MSR 0x79 */ | 830 | /* write microcode via MSR 0x79 */ |
| 797 | wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); | 831 | wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits); |
| 798 | 832 | ||
| @@ -801,7 +835,7 @@ static int apply_microcode_intel(int cpu) · 2 unmodified lines | |||
| 801 | if (rev != mc->hdr.rev) { | 835 | if (rev != mc->hdr.rev) { |
| 802 | pr_err("CPU%d update to revision 0x%x failed\n", | 836 | pr_err("CPU%d update to revision 0x%x failed\n", |
| 803 | cpu, mc->hdr.rev); | 837 | cpu, mc->hdr.rev); |
| 804 | return -1; | 838 | return UCODE_ERROR; |
| 805 | } | 839 | } |
| 806 | 840 | ||
| 807 | if (rev != prev_rev) { | 841 | if (rev != prev_rev) { |
| @@ -813,12 +847,10 @@ static int apply_microcode_intel(int cpu) · 5 unmodified lines | |||
| 813 | prev_rev = rev; | 847 | prev_rev = rev; |
| 814 | } | 848 | } |
| 815 | 849 | ||
| 816 | c = &cpu_data(cpu); | ||
| 817 | |||
| 818 | uci->cpu_sig.rev = rev; | 850 | uci->cpu_sig.rev = rev; |
| 819 | c->microcode = rev; | 851 | c->microcode = rev; |
| 820 | 852 | ||
| 821 | return 0; | 853 | return UCODE_UPDATED; |
| 822 | } | 854 | } |
| 823 | 855 | ||
| 824 | static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, | 856 | static enum ucode_state generic_load_microcode(int cpu, void *data, size_t size, |
| @@ -921,7 +953,7 @@ static bool is_blacklisted(unsigned int cpu) · 96 unmodified lines | |||
| 921 | */ | 953 | */ |
| 922 | if (c->x86 == 6 && | 954 | if (c->x86 == 6 && |
| 923 | c->x86_model == INTEL_FAM6_BROADWELL_X && | 955 | c->x86_model == INTEL_FAM6_BROADWELL_X && |
| 924 | c->x86_mask == 0x01 && | 956 | c->x86_stepping == 0x01 && |
| 925 | llc_size_per_core > 2621440 && | 957 | llc_size_per_core > 2621440 && |
| 926 | c->microcode < 0x0b000021) { | 958 | c->microcode < 0x0b000021) { |
| 927 | pr_err_once("Erratum BDF90: late loading with revision < 0x0b000021 (0x%x) disabled.\n", c->microcode); | 959 | pr_err_once("Erratum BDF90: late loading with revision < 0x0b000021 (0x%x) disabled.\n", c->microcode); |
| @@ -944,7 +976,7 @@ static enum ucode_state request_microcode_fw(int cpu, struct device *device, · 16 unmodified lines | |||
| 944 | return UCODE_NFOUND; | 976 | return UCODE_NFOUND; |
| 945 | 977 | ||
| 946 | sprintf(name, "intel-ucode/%02x-%02x-%02x", | 978 | sprintf(name, "intel-ucode/%02x-%02x-%02x", |
| 947 | c->x86, c->x86_model, c->x86_mask); | 979 | c->x86, c->x86_model, c->x86_stepping); |
| 948 | 980 | ||
| 949 | if (request_firmware_direct(&firmware, name, device)) { | 981 | if (request_firmware_direct(&firmware, name, device)) { |
| 950 | pr_debug("data file %s load failed\n", name); | 982 | pr_debug("data file %s load failed\n", name); |
| @@ -982,7 +1014,7 @@ static struct microcode_ops microcode_intel_ops = { · 31 unmodified lines | |||
| 982 | 1014 | ||
| 983 | static int __init calc_llc_size_per_core(struct cpuinfo_x86 *c) | 1015 | static int __init calc_llc_size_per_core(struct cpuinfo_x86 *c) |
| 984 | { | 1016 | { |
| 985 | u64 llc_size = c->x86_cache_size * 1024; | 1017 | u64 llc_size = c->x86_cache_size * 1024ULL; |
| 986 | 1018 | ||
| 987 | do_div(llc_size, c->x86_max_cores); | 1019 | do_div(llc_size, c->x86_max_cores); |
| 988 | 1020 | ||
| @@ -859,7 +859,7 @@ int generic_validate_add_page(unsigned long base, unsigned long size, · 858 unmodified lines | |||
| 859 | */ | 859 | */ |
| 860 | if (is_cpu(INTEL) && boot_cpu_data.x86 == 6 && | 860 | if (is_cpu(INTEL) && boot_cpu_data.x86 == 6 && |
| 861 | boot_cpu_data.x86_model == 1 && | 861 | boot_cpu_data.x86_model == 1 && |
| 862 | boot_cpu_data.x86_mask <= 7) { | 862 | boot_cpu_data.x86_stepping <= 7) { |
| 863 | if (base & ((1 << (22 - PAGE_SHIFT)) - 1)) { | 863 | if (base & ((1 << (22 - PAGE_SHIFT)) - 1)) { |
| 864 | pr_warn("mtrr: base(0x%lx000) is not 4 MiB aligned\n", base); | 864 | pr_warn("mtrr: base(0x%lx000) is not 4 MiB aligned\n", base); |
| 865 | return -EINVAL; | 865 | return -EINVAL; |
| @@ -711,8 +711,8 @@ void __init mtrr_bp_init(void) · 710 unmodified lines | |||
| 711 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && | 711 | if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && |
| 712 | boot_cpu_data.x86 == 0xF && | 712 | boot_cpu_data.x86 == 0xF && |
| 713 | boot_cpu_data.x86_model == 0x3 && | 713 | boot_cpu_data.x86_model == 0x3 && |
| 714 | (boot_cpu_data.x86_mask == 0x3 || | 714 | (boot_cpu_data.x86_stepping == 0x3 || |
| 715 | boot_cpu_data.x86_mask == 0x4)) | 715 | boot_cpu_data.x86_stepping == 0x4)) |
| 716 | phys_addr = 36; | 716 | phys_addr = 36; |
| 717 | 717 | ||
| 718 | size_or_mask = SIZE_OR_MASK_BITS(phys_addr); | 718 | size_or_mask = SIZE_OR_MASK_BITS(phys_addr); |
| @@ -72,8 +72,8 @@ static int show_cpuinfo(struct seq_file *m, void *v) · 71 unmodified lines | |||
| 72 | c->x86_model, | 72 | c->x86_model, |
| 73 | c->x86_model_id[0] ? c->x86_model_id : "unknown"); | 73 | c->x86_model_id[0] ? c->x86_model_id : "unknown"); |
| 74 | 74 | ||
| 75 | if (c->x86_mask || c->cpuid_level >= 0) | 75 | if (c->x86_stepping || c->cpuid_level >= 0) |
| 76 | seq_printf(m, "stepping\t: %d\n", c->x86_mask); | 76 | seq_printf(m, "stepping\t: %d\n", c->x86_stepping); |
| 77 | else | 77 | else |
| 78 | seq_puts(m, "stepping\t: unknown\n"); | 78 | seq_puts(m, "stepping\t: unknown\n"); |
| 79 | if (c->microcode) | 79 | if (c->microcode) |
| @@ -91,8 +91,8 @@ static int show_cpuinfo(struct seq_file *m, void *v) · 11 unmodified lines | |||
| 91 | } | 91 | } |
| 92 | 92 | ||
| 93 | /* Cache size */ | 93 | /* Cache size */ |
| 94 | if (c->x86_cache_size >= 0) | 94 | if (c->x86_cache_size) |
| 95 | seq_printf(m, "cache size\t: %d KB\n", c->x86_cache_size); | 95 | seq_printf(m, "cache size\t: %u KB\n", c->x86_cache_size); |
| 96 | 96 | ||
| 97 | show_cpuinfo_core(m, c, cpu); | 97 | show_cpuinfo_core(m, c, cpu); |
| 98 | show_cpuinfo_misc(m, c); | 98 | show_cpuinfo_misc(m, c); |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | |||
| 37 | #define X86 new_cpu_data+CPUINFO_x86 | 37 | #define X86 new_cpu_data+CPUINFO_x86 |
| 38 | #define X86_VENDOR new_cpu_data+CPUINFO_x86_vendor | 38 | #define X86_VENDOR new_cpu_data+CPUINFO_x86_vendor |
| 39 | #define X86_MODEL new_cpu_data+CPUINFO_x86_model | 39 | #define X86_MODEL new_cpu_data+CPUINFO_x86_model |
| 40 | #define X86_MASK new_cpu_data+CPUINFO_x86_mask | 40 | #define X86_STEPPING new_cpu_data+CPUINFO_x86_stepping |
| 41 | #define X86_HARD_MATH new_cpu_data+CPUINFO_hard_math | 41 | #define X86_HARD_MATH new_cpu_data+CPUINFO_hard_math |
| 42 | #define X86_CPUID new_cpu_data+CPUINFO_cpuid_level | 42 | #define X86_CPUID new_cpu_data+CPUINFO_cpuid_level |
| 43 | #define X86_CAPABILITY new_cpu_data+CPUINFO_x86_capability | 43 | #define X86_CAPABILITY new_cpu_data+CPUINFO_x86_capability |
| @@ -332,7 +332,7 @@ ENTRY(startup_32_smp) · 288 unmodified lines | |||
| 332 | shrb $4,%al | 332 | shrb $4,%al |
| 333 | movb %al,X86_MODEL | 333 | movb %al,X86_MODEL |
| 334 | andb $0x0f,%cl # mask mask revision | 334 | andb $0x0f,%cl # mask mask revision |
| 335 | movb %cl,X86_MASK | 335 | movb %cl,X86_STEPPING |
| 336 | movl %edx,X86_CAPABILITY | 336 | movl %edx,X86_CAPABILITY |
| 337 | 337 | ||
| 338 | .Lis486: | 338 | .Lis486: |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | |||
| 23 | #include <asm/nops.h> | 23 | #include <asm/nops.h> |
| 24 | #include "../entry/calling.h" | 24 | #include "../entry/calling.h" |
| 25 | #include <asm/export.h> | 25 | #include <asm/export.h> |
| 26 | #include <asm/nospec-branch.h> | ||
| 26 | 27 | ||
| 27 | #ifdef CONFIG_PARAVIRT | 28 | #ifdef CONFIG_PARAVIRT |
| 28 | #include <asm/asm-offsets.h> | 29 | #include <asm/asm-offsets.h> |
| @@ -134,6 +135,7 @@ ENTRY(secondary_startup_64) · 105 unmodified lines | |||
| 134 | 135 | ||
| 135 | /* Ensure I am executing from virtual addresses */ | 136 | /* Ensure I am executing from virtual addresses */ |
| 136 | movq $1f, %rax | 137 | movq $1f, %rax |
| 138 | ANNOTATE_RETPOLINE_SAFE | ||
| 137 | jmp *%rax | 139 | jmp *%rax |
| 138 | 1: | 140 | 1: |
| 139 | UNWIND_HINT_EMPTY | 141 | UNWIND_HINT_EMPTY |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | |||
| 23 | /* | 23 | /* |
| 24 | * this changes the io permissions bitmap in the current task. | 24 | * this changes the io permissions bitmap in the current task. |
| 25 | */ | 25 | */ |
| 26 | asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int turn_on) | 26 | SYSCALL_DEFINE3(ioperm, unsigned long, from, unsigned long, num, int, turn_on) |
| 27 | { | 27 | { |
| 28 | struct thread_struct *t = ¤t->thread; | 28 | struct thread_struct *t = ¤t->thread; |
| 29 | struct tss_struct *tss; | 29 | struct tss_struct *tss; |
| @@ -1168,10 +1168,18 @@ NOKPROBE_SYMBOL(longjmp_break_handler); · 1167 unmodified lines | |||
| 1168 | 1168 | ||
| 1169 | bool arch_within_kprobe_blacklist(unsigned long addr) | 1169 | bool arch_within_kprobe_blacklist(unsigned long addr) |
| 1170 | { | 1170 | { |
| 1171 | bool is_in_entry_trampoline_section = false; | ||
| 1172 | |||
| 1173 | #ifdef CONFIG_X86_64 | ||
| 1174 | is_in_entry_trampoline_section = | ||
| 1175 | (addr >= (unsigned long)__entry_trampoline_start && | ||
| 1176 | addr < (unsigned long)__entry_trampoline_end); | ||
| 1177 | #endif | ||
| 1171 | return (addr >= (unsigned long)__kprobes_text_start && | 1178 | return (addr >= (unsigned long)__kprobes_text_start && |
| 1172 | addr < (unsigned long)__kprobes_text_end) || | 1179 | addr < (unsigned long)__kprobes_text_end) || |
| 1173 | (addr >= (unsigned long)__entry_text_start && | 1180 | (addr >= (unsigned long)__entry_text_start && |
| 1174 | addr < (unsigned long)__entry_text_end); | 1181 | addr < (unsigned long)__entry_text_end) || |
| 1182 | is_in_entry_trampoline_section; | ||
| 1175 | } | 1183 | } |
| 1176 | 1184 | ||
| 1177 | int __init arch_init_kprobes(void) | 1185 | int __init arch_init_kprobes(void) |
| @@ -49,7 +49,7 @@ · 48 unmodified lines | |||
| 49 | 49 | ||
| 50 | static int kvmapf = 1; | 50 | static int kvmapf = 1; |
| 51 | 51 | ||
| 52 | static int parse_no_kvmapf(char *arg) | 52 | static int __init parse_no_kvmapf(char *arg) |
| 53 | { | 53 | { |
| 54 | kvmapf = 0; | 54 | kvmapf = 0; |
| 55 | return 0; | 55 | return 0; |
| @@ -58,7 +58,7 @@ static int parse_no_kvmapf(char *arg) · 2 unmodified lines | |||
| 58 | early_param("no-kvmapf", parse_no_kvmapf); | 58 | early_param("no-kvmapf", parse_no_kvmapf); |
| 59 | 59 | ||
| 60 | static int steal_acc = 1; | 60 | static int steal_acc = 1; |
| 61 | static int parse_no_stealacc(char *arg) | 61 | static int __init parse_no_stealacc(char *arg) |
| 62 | { | 62 | { |
| 63 | steal_acc = 0; | 63 | steal_acc = 0; |
| 64 | return 0; | 64 | return 0; |
| @@ -67,7 +67,7 @@ static int parse_no_stealacc(char *arg) · 2 unmodified lines | |||
| 67 | early_param("no-steal-acc", parse_no_stealacc); | 67 | early_param("no-steal-acc", parse_no_stealacc); |
| 68 | 68 | ||
| 69 | static int kvmclock_vsyscall = 1; | 69 | static int kvmclock_vsyscall = 1; |
| 70 | static int parse_no_kvmclock_vsyscall(char *arg) | 70 | static int __init parse_no_kvmclock_vsyscall(char *arg) |
| 71 | { | 71 | { |
| 72 | kvmclock_vsyscall = 0; | 72 | kvmclock_vsyscall = 0; |
| 73 | return 0; | 73 | return 0; |
| @@ -341,10 +341,10 @@ static void kvm_guest_cpu_init(void) · 267 unmodified lines | |||
| 341 | #endif | 341 | #endif |
| 342 | pa |= KVM_ASYNC_PF_ENABLED; | 342 | pa |= KVM_ASYNC_PF_ENABLED; |
| 343 | 343 | ||
| 344 | /* Async page fault support for L1 hypervisor is optional */ | 344 | if (kvm_para_has_feature(KVM_FEATURE_ASYNC_PF_VMEXIT)) |
| 345 | if (wrmsr_safe(MSR_KVM_ASYNC_PF_EN, | 345 | pa |= KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT; |
| 346 | (pa | KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT) & 0xffffffff, pa >> 32) < 0) | 346 | |
| 347 | wrmsrl(MSR_KVM_ASYNC_PF_EN, pa); | 347 | wrmsrl(MSR_KVM_ASYNC_PF_EN, pa); |
| 348 | __this_cpu_write(apf_reason.enabled, 1); | 348 | __this_cpu_write(apf_reason.enabled, 1); |
| 349 | printk(KERN_INFO"KVM setup async PF for cpu %d\n", | 349 | printk(KERN_INFO"KVM setup async PF for cpu %d\n", |
| 350 | smp_processor_id()); | 350 | smp_processor_id()); |
| @@ -545,7 +545,8 @@ static void __init kvm_guest_init(void) · 194 unmodified lines | |||
| 545 | pv_time_ops.steal_clock = kvm_steal_clock; | 545 | pv_time_ops.steal_clock = kvm_steal_clock; |
| 546 | } | 546 | } |
| 547 | 547 | ||
| 548 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH)) | 548 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && |
| 549 | !kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) | ||
| 549 | pv_mmu_ops.flush_tlb_others = kvm_flush_tlb_others; | 550 | pv_mmu_ops.flush_tlb_others = kvm_flush_tlb_others; |
| 550 | 551 | ||
| 551 | if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) | 552 | if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) |
| @@ -633,7 +634,8 @@ static __init int kvm_setup_pv_tlb_flush(void) · 81 unmodified lines | |||
| 633 | { | 634 | { |
| 634 | int cpu; | 635 | int cpu; |
| 635 | 636 | ||
| 636 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH)) { | 637 | if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && |
| 638 | !kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { | ||
| 637 | for_each_possible_cpu(cpu) { | 639 | for_each_possible_cpu(cpu) { |
| 638 | zalloc_cpumask_var_node(per_cpu_ptr(&__pv_tlb_mask, cpu), | 640 | zalloc_cpumask_var_node(per_cpu_ptr(&__pv_tlb_mask, cpu), |
| 639 | GFP_KERNEL, cpu_to_node(cpu)); | 641 | GFP_KERNEL, cpu_to_node(cpu)); |
| @@ -542,6 +542,7 @@ int arch_kexec_apply_relocations_add(const Elf64_Ehdr *ehdr, · 541 unmodified lines | |||
| 542 | goto overflow; | 542 | goto overflow; |
| 543 | break; | 543 | break; |
| 544 | case R_X86_64_PC32: | 544 | case R_X86_64_PC32: |
| 545 | case R_X86_64_PLT32: | ||
| 545 | value -= (u64)address; | 546 | value -= (u64)address; |
| 546 | *(u32 *)location = value; | 547 | *(u32 *)location = value; |
| 547 | break; | 548 | break; |
| @@ -191,6 +191,7 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, · 190 unmodified lines | |||
| 191 | goto overflow; | 191 | goto overflow; |
| 192 | break; | 192 | break; |
| 193 | case R_X86_64_PC32: | 193 | case R_X86_64_PC32: |
| 194 | case R_X86_64_PLT32: | ||
| 194 | if (*(u32 *)loc != 0) | 195 | if (*(u32 *)loc != 0) |
| 195 | goto invalid_relocation; | 196 | goto invalid_relocation; |
| 196 | val -= (u64)loc; | 197 | val -= (u64)loc; |
| @@ -410,7 +410,7 @@ static inline void __init construct_default_ISA_mptable(int mpc_default_type) · 409 unmodified lines | |||
| 410 | processor.apicver = mpc_default_type > 4 ? 0x10 : 0x01; | 410 | processor.apicver = mpc_default_type > 4 ? 0x10 : 0x01; |
| 411 | processor.cpuflag = CPU_ENABLED; | 411 | processor.cpuflag = CPU_ENABLED; |
| 412 | processor.cpufeature = (boot_cpu_data.x86 << 8) | | 412 | processor.cpufeature = (boot_cpu_data.x86 << 8) | |
| 413 | (boot_cpu_data.x86_model << 4) | boot_cpu_data.x86_mask; | 413 | (boot_cpu_data.x86_model << 4) | boot_cpu_data.x86_stepping; |
| 414 | processor.featureflag = boot_cpu_data.x86_capability[CPUID_1_EDX]; | 414 | processor.featureflag = boot_cpu_data.x86_capability[CPUID_1_EDX]; |
| 415 | processor.reserved[0] = 0; | 415 | processor.reserved[0] = 0; |
| 416 | processor.reserved[1] = 0; | 416 | processor.reserved[1] = 0; |
| @@ -200,9 +200,9 @@ static void native_flush_tlb_global(void) · 199 unmodified lines | |||
| 200 | __native_flush_tlb_global(); | 200 | __native_flush_tlb_global(); |
| 201 | } | 201 | } |
| 202 | 202 | ||
| 203 | static void native_flush_tlb_single(unsigned long addr) | 203 | static void native_flush_tlb_one_user(unsigned long addr) |
| 204 | { | 204 | { |
| 205 | __native_flush_tlb_single(addr); | 205 | __native_flush_tlb_one_user(addr); |
| 206 | } | 206 | } |
| 207 | 207 | ||
| 208 | struct static_key paravirt_steal_enabled; | 208 | struct static_key paravirt_steal_enabled; |
| @@ -401,7 +401,7 @@ struct pv_mmu_ops pv_mmu_ops __ro_after_init = { · 192 unmodified lines | |||
| 401 | 401 | ||
| 402 | .flush_tlb_user = native_flush_tlb, | 402 | .flush_tlb_user = native_flush_tlb, |
| 403 | .flush_tlb_kernel = native_flush_tlb_global, | 403 | .flush_tlb_kernel = native_flush_tlb_global, |
| 404 | .flush_tlb_single = native_flush_tlb_single, | 404 | .flush_tlb_one_user = native_flush_tlb_one_user, |
| 405 | .flush_tlb_others = native_flush_tlb_others, | 405 | .flush_tlb_others = native_flush_tlb_others, |
| 406 | 406 | ||
| 407 | .pgd_alloc = __paravirt_pgd_alloc, | 407 | .pgd_alloc = __paravirt_pgd_alloc, |
| @@ -1204,20 +1204,13 @@ void __init setup_arch(char **cmdline_p) · 1203 unmodified lines | |||
| 1204 | 1204 | ||
| 1205 | kasan_init(); | 1205 | kasan_init(); |
| 1206 | 1206 | ||
| 1207 | #ifdef CONFIG_X86_32 | ||
| 1208 | /* sync back kernel address range */ | ||
| 1209 | clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY, | ||
| 1210 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 1211 | KERNEL_PGD_PTRS); | ||
| 1212 | |||
| 1213 | /* | 1207 | /* |
| 1214 | * sync back low identity map too. It is used for example | 1208 | * Sync back kernel address range. |
| 1215 | * in the 32-bit EFI stub. | 1209 | * |
| 1210 | * FIXME: Can the later sync in setup_cpu_entry_areas() replace | ||
| 1211 | * this call? | ||
| 1216 | */ | 1212 | */ |
| 1217 | clone_pgd_range(initial_page_table, | 1213 | sync_initial_page_table(); |
| 1218 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 1219 | min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY)); | ||
| 1220 | #endif | ||
| 1221 | 1214 | ||
| 1222 | tboot_probe(); | 1215 | tboot_probe(); |
| 1223 | 1216 | ||
| @@ -287,24 +287,15 @@ void __init setup_per_cpu_areas(void) · 286 unmodified lines | |||
| 287 | /* Setup cpu initialized, callin, callout masks */ | 287 | /* Setup cpu initialized, callin, callout masks */ |
| 288 | setup_cpu_local_masks(); | 288 | setup_cpu_local_masks(); |
| 289 | 289 | ||
| 290 | #ifdef CONFIG_X86_32 | ||
| 291 | /* | 290 | /* |
| 292 | * Sync back kernel address range again. We already did this in | 291 | * Sync back kernel address range again. We already did this in |
| 293 | * setup_arch(), but percpu data also needs to be available in | 292 | * setup_arch(), but percpu data also needs to be available in |
| 294 | * the smpboot asm. We can't reliably pick up percpu mappings | 293 | * the smpboot asm. We can't reliably pick up percpu mappings |
| 295 | * using vmalloc_fault(), because exception dispatch needs | 294 | * using vmalloc_fault(), because exception dispatch needs |
| 296 | * percpu data. | 295 | * percpu data. |
| 296 | * | ||
| 297 | * FIXME: Can the later sync in setup_cpu_entry_areas() replace | ||
| 298 | * this call? | ||
| 297 | */ | 299 | */ |
| 298 | clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY, | 300 | sync_initial_page_table(); |
| 299 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 300 | KERNEL_PGD_PTRS); | ||
| 301 | |||
| 302 | /* | ||
| 303 | * sync back low identity map too. It is used for example | ||
| 304 | * in the 32-bit EFI stub. | ||
| 305 | */ | ||
| 306 | clone_pgd_range(initial_page_table, | ||
| 307 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 308 | min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY)); | ||
| 309 | #endif | ||
| 310 | } | 301 | } |
| @@ -43,6 +43,13 @@ static inline void signal_compat_build_tests(void) · 42 unmodified lines | |||
| 43 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields) != 3 * sizeof(int)); | 43 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields) != 3 * sizeof(int)); |
| 44 | #define CHECK_CSI_OFFSET(name) BUILD_BUG_ON(_sifields_offset != offsetof(compat_siginfo_t, _sifields.name)) | 44 | #define CHECK_CSI_OFFSET(name) BUILD_BUG_ON(_sifields_offset != offsetof(compat_siginfo_t, _sifields.name)) |
| 45 | 45 | ||
| 46 | BUILD_BUG_ON(offsetof(siginfo_t, si_signo) != 0); | ||
| 47 | BUILD_BUG_ON(offsetof(siginfo_t, si_errno) != 4); | ||
| 48 | BUILD_BUG_ON(offsetof(siginfo_t, si_code) != 8); | ||
| 49 | |||
| 50 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_signo) != 0); | ||
| 51 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_errno) != 4); | ||
| 52 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_code) != 8); | ||
| 46 | /* | 53 | /* |
| 47 | * Ensure that the size of each si_field never changes. | 54 | * Ensure that the size of each si_field never changes. |
| 48 | * If it does, it is a sign that the | 55 | * If it does, it is a sign that the |
| @@ -63,36 +70,94 @@ static inline void signal_compat_build_tests(void) · 14 unmodified lines | |||
| 63 | CHECK_CSI_SIZE (_kill, 2*sizeof(int)); | 70 | CHECK_CSI_SIZE (_kill, 2*sizeof(int)); |
| 64 | CHECK_SI_SIZE (_kill, 2*sizeof(int)); | 71 | CHECK_SI_SIZE (_kill, 2*sizeof(int)); |
| 65 | 72 | ||
| 73 | BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x10); | ||
| 74 | BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x14); | ||
| 75 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pid) != 0xC); | ||
| 76 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_uid) != 0x10); | ||
| 77 | |||
| 66 | CHECK_CSI_OFFSET(_timer); | 78 | CHECK_CSI_OFFSET(_timer); |
| 67 | CHECK_CSI_SIZE (_timer, 3*sizeof(int)); | 79 | CHECK_CSI_SIZE (_timer, 3*sizeof(int)); |
| 68 | CHECK_SI_SIZE (_timer, 6*sizeof(int)); | 80 | CHECK_SI_SIZE (_timer, 6*sizeof(int)); |
| 69 | 81 | ||
| 82 | BUILD_BUG_ON(offsetof(siginfo_t, si_tid) != 0x10); | ||
| 83 | BUILD_BUG_ON(offsetof(siginfo_t, si_overrun) != 0x14); | ||
| 84 | BUILD_BUG_ON(offsetof(siginfo_t, si_value) != 0x18); | ||
| 85 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_tid) != 0x0C); | ||
| 86 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_overrun) != 0x10); | ||
| 87 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_value) != 0x14); | ||
| 88 | |||
| 70 | CHECK_CSI_OFFSET(_rt); | 89 | CHECK_CSI_OFFSET(_rt); |
| 71 | CHECK_CSI_SIZE (_rt, 3*sizeof(int)); | 90 | CHECK_CSI_SIZE (_rt, 3*sizeof(int)); |
| 72 | CHECK_SI_SIZE (_rt, 4*sizeof(int)); | 91 | CHECK_SI_SIZE (_rt, 4*sizeof(int)); |
| 73 | 92 | ||
| 93 | BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x10); | ||
| 94 | BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x14); | ||
| 95 | BUILD_BUG_ON(offsetof(siginfo_t, si_value) != 0x18); | ||
| 96 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pid) != 0x0C); | ||
| 97 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_uid) != 0x10); | ||
| 98 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_value) != 0x14); | ||
| 99 | |||
| 74 | CHECK_CSI_OFFSET(_sigchld); | 100 | CHECK_CSI_OFFSET(_sigchld); |
| 75 | CHECK_CSI_SIZE (_sigchld, 5*sizeof(int)); | 101 | CHECK_CSI_SIZE (_sigchld, 5*sizeof(int)); |
| 76 | CHECK_SI_SIZE (_sigchld, 8*sizeof(int)); | 102 | CHECK_SI_SIZE (_sigchld, 8*sizeof(int)); |
| 77 | 103 | ||
| 104 | BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x10); | ||
| 105 | BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x14); | ||
| 106 | BUILD_BUG_ON(offsetof(siginfo_t, si_status) != 0x18); | ||
| 107 | BUILD_BUG_ON(offsetof(siginfo_t, si_utime) != 0x20); | ||
| 108 | BUILD_BUG_ON(offsetof(siginfo_t, si_stime) != 0x28); | ||
| 109 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pid) != 0x0C); | ||
| 110 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_uid) != 0x10); | ||
| 111 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_status) != 0x14); | ||
| 112 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_utime) != 0x18); | ||
| 113 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_stime) != 0x1C); | ||
| 114 | |||
| 78 | #ifdef CONFIG_X86_X32_ABI | 115 | #ifdef CONFIG_X86_X32_ABI |
| 79 | CHECK_CSI_OFFSET(_sigchld_x32); | 116 | CHECK_CSI_OFFSET(_sigchld_x32); |
| 80 | CHECK_CSI_SIZE (_sigchld_x32, 7*sizeof(int)); | 117 | CHECK_CSI_SIZE (_sigchld_x32, 7*sizeof(int)); |
| 81 | /* no _sigchld_x32 in the generic siginfo_t */ | 118 | /* no _sigchld_x32 in the generic siginfo_t */ |
| 119 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields._sigchld_x32._utime) != 0x18); | ||
| 120 | BUILD_BUG_ON(offsetof(compat_siginfo_t, _sifields._sigchld_x32._stime) != 0x20); | ||
| 82 | #endif | 121 | #endif |
| 83 | 122 | ||
| 84 | CHECK_CSI_OFFSET(_sigfault); | 123 | CHECK_CSI_OFFSET(_sigfault); |
| 85 | CHECK_CSI_SIZE (_sigfault, 4*sizeof(int)); | 124 | CHECK_CSI_SIZE (_sigfault, 4*sizeof(int)); |
| 86 | CHECK_SI_SIZE (_sigfault, 8*sizeof(int)); | 125 | CHECK_SI_SIZE (_sigfault, 8*sizeof(int)); |
| 87 | 126 | ||
| 127 | BUILD_BUG_ON(offsetof(siginfo_t, si_addr) != 0x10); | ||
| 128 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_addr) != 0x0C); | ||
| 129 | |||
| 130 | BUILD_BUG_ON(offsetof(siginfo_t, si_addr_lsb) != 0x18); | ||
| 131 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_addr_lsb) != 0x10); | ||
| 132 | |||
| 133 | BUILD_BUG_ON(offsetof(siginfo_t, si_lower) != 0x20); | ||
| 134 | BUILD_BUG_ON(offsetof(siginfo_t, si_upper) != 0x28); | ||
| 135 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_lower) != 0x14); | ||
| 136 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_upper) != 0x18); | ||
| 137 | |||
| 138 | BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x20); | ||
| 139 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_pkey) != 0x14); | ||
| 140 | |||
| 88 | CHECK_CSI_OFFSET(_sigpoll); | 141 | CHECK_CSI_OFFSET(_sigpoll); |
| 89 | CHECK_CSI_SIZE (_sigpoll, 2*sizeof(int)); | 142 | CHECK_CSI_SIZE (_sigpoll, 2*sizeof(int)); |
| 90 | CHECK_SI_SIZE (_sigpoll, 4*sizeof(int)); | 143 | CHECK_SI_SIZE (_sigpoll, 4*sizeof(int)); |
| 91 | 144 | ||
| 145 | BUILD_BUG_ON(offsetof(siginfo_t, si_band) != 0x10); | ||
| 146 | BUILD_BUG_ON(offsetof(siginfo_t, si_fd) != 0x18); | ||
| 147 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_band) != 0x0C); | ||
| 148 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_fd) != 0x10); | ||
| 149 | |||
| 92 | CHECK_CSI_OFFSET(_sigsys); | 150 | CHECK_CSI_OFFSET(_sigsys); |
| 93 | CHECK_CSI_SIZE (_sigsys, 3*sizeof(int)); | 151 | CHECK_CSI_SIZE (_sigsys, 3*sizeof(int)); |
| 94 | CHECK_SI_SIZE (_sigsys, 4*sizeof(int)); | 152 | CHECK_SI_SIZE (_sigsys, 4*sizeof(int)); |
| 95 | 153 | ||
| 154 | BUILD_BUG_ON(offsetof(siginfo_t, si_call_addr) != 0x10); | ||
| 155 | BUILD_BUG_ON(offsetof(siginfo_t, si_syscall) != 0x18); | ||
| 156 | BUILD_BUG_ON(offsetof(siginfo_t, si_arch) != 0x1C); | ||
| 157 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_call_addr) != 0x0C); | ||
| 158 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_syscall) != 0x10); | ||
| 159 | BUILD_BUG_ON(offsetof(compat_siginfo_t, si_arch) != 0x14); | ||
| 160 | |||
| 96 | /* any new si_fields should be added here */ | 161 | /* any new si_fields should be added here */ |
| 97 | } | 162 | } |
| 98 | 163 | ||
| @@ -1281,11 +1281,10 @@ void __init native_smp_prepare_boot_cpu(void) · 1280 unmodified lines | |||
| 1281 | cpu_set_state_online(me); | 1281 | cpu_set_state_online(me); |
| 1282 | } | 1282 | } |
| 1283 | 1283 | ||
| 1284 | void __init native_smp_cpus_done(unsigned int max_cpus) | 1284 | void __init calculate_max_logical_packages(void) |
| 1285 | { | 1285 | { |
| 1286 | int ncpus; | 1286 | int ncpus; |
| 1287 | 1287 | ||
| 1288 | pr_debug("Boot done\n"); | ||
| 1289 | /* | 1288 | /* |
| 1290 | * Today neither Intel nor AMD support heterogenous systems so | 1289 | * Today neither Intel nor AMD support heterogenous systems so |
| 1291 | * extrapolate the boot cpu's data to all packages. | 1290 | * extrapolate the boot cpu's data to all packages. |
| @@ -1293,6 +1292,13 @@ void __init native_smp_cpus_done(unsigned int max_cpus) · 1 unmodified line | |||
| 1293 | ncpus = cpu_data(0).booted_cores * topology_max_smt_threads(); | 1292 | ncpus = cpu_data(0).booted_cores * topology_max_smt_threads(); |
| 1294 | __max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus); | 1293 | __max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus); |
| 1295 | pr_info("Max logical packages: %u\n", __max_logical_packages); | 1294 | pr_info("Max logical packages: %u\n", __max_logical_packages); |
| 1295 | } | ||
| 1296 | |||
| 1297 | void __init native_smp_cpus_done(unsigned int max_cpus) | ||
| 1298 | { | ||
| 1299 | pr_debug("Boot done\n"); | ||
| 1300 | |||
| 1301 | calculate_max_logical_packages(); | ||
| 1296 | 1302 | ||
| 1297 | if (x86_has_numa_in_package) | 1303 | if (x86_has_numa_in_package) |
| 1298 | set_sched_topology(x86_numa_in_package_topology); | 1304 | set_sched_topology(x86_numa_in_package_topology); |
| @@ -1430,8 +1436,8 @@ static void remove_siblinginfo(int cpu) · 131 unmodified lines | |||
| 1430 | cpumask_clear(cpu_llc_shared_mask(cpu)); | 1436 | cpumask_clear(cpu_llc_shared_mask(cpu)); |
| 1431 | cpumask_clear(topology_sibling_cpumask(cpu)); | 1437 | cpumask_clear(topology_sibling_cpumask(cpu)); |
| 1432 | cpumask_clear(topology_core_cpumask(cpu)); | 1438 | cpumask_clear(topology_core_cpumask(cpu)); |
| 1433 | c->phys_proc_id = 0; | ||
| 1434 | c->cpu_core_id = 0; | 1439 | c->cpu_core_id = 0; |
| 1440 | c->booted_cores = 0; | ||
| 1435 | cpumask_clear_cpu(cpu, cpu_sibling_setup_mask); | 1441 | cpumask_clear_cpu(cpu, cpu_sibling_setup_mask); |
| 1436 | recompute_smt_state(); | 1442 | recompute_smt_state(); |
| 1437 | } | 1443 | } |
| @@ -181,7 +181,7 @@ int fixup_bug(struct pt_regs *regs, int trapnr) · 180 unmodified lines | |||
| 181 | break; | 181 | break; |
| 182 | 182 | ||
| 183 | case BUG_TRAP_TYPE_WARN: | 183 | case BUG_TRAP_TYPE_WARN: |
| 184 | regs->ip += LEN_UD0; | 184 | regs->ip += LEN_UD2; |
| 185 | return 1; | 185 | return 1; |
| 186 | } | 186 | } |
| 187 | 187 | ||
| @@ -5,7 +5,6 @@ · 4 unmodified lines | |||
| 5 | #include <asm/unwind.h> | 5 | #include <asm/unwind.h> |
| 6 | #include <asm/orc_types.h> | 6 | #include <asm/orc_types.h> |
| 7 | #include <asm/orc_lookup.h> | 7 | #include <asm/orc_lookup.h> |
| 8 | #include <asm/sections.h> | ||
| 9 | 8 | ||
| 10 | #define orc_warn(fmt, ...) \ | 9 | #define orc_warn(fmt, ...) \ |
| 11 | printk_deferred_once(KERN_WARNING pr_fmt("WARNING: " fmt), ##__VA_ARGS__) | 10 | printk_deferred_once(KERN_WARNING pr_fmt("WARNING: " fmt), ##__VA_ARGS__) |
| @@ -148,7 +147,7 @@ static struct orc_entry *orc_find(unsigned long ip) · 136 unmodified lines | |||
| 148 | } | 147 | } |
| 149 | 148 | ||
| 150 | /* vmlinux .init slow lookup: */ | 149 | /* vmlinux .init slow lookup: */ |
| 151 | if (ip >= (unsigned long)_sinittext && ip < (unsigned long)_einittext) | 150 | if (init_kernel_text(ip)) |
| 152 | return __orc_find(__start_orc_unwind_ip, __start_orc_unwind, | 151 | return __orc_find(__start_orc_unwind_ip, __start_orc_unwind, |
| 153 | __stop_orc_unwind_ip - __start_orc_unwind_ip, ip); | 152 | __stop_orc_unwind_ip - __start_orc_unwind_ip, ip); |
| 154 | 153 | ||
| @@ -118,9 +118,11 @@ SECTIONS · 117 unmodified lines | |||
| 118 | 118 | ||
| 119 | #ifdef CONFIG_X86_64 | 119 | #ifdef CONFIG_X86_64 |
| 120 | . = ALIGN(PAGE_SIZE); | 120 | . = ALIGN(PAGE_SIZE); |
| 121 | VMLINUX_SYMBOL(__entry_trampoline_start) = .; | ||
| 121 | _entry_trampoline = .; | 122 | _entry_trampoline = .; |
| 122 | *(.entry_trampoline) | 123 | *(.entry_trampoline) |
| 123 | . = ALIGN(PAGE_SIZE); | 124 | . = ALIGN(PAGE_SIZE); |
| 125 | VMLINUX_SYMBOL(__entry_trampoline_end) = .; | ||
| 124 | ASSERT(. - _entry_trampoline == PAGE_SIZE, "entry trampoline is too big"); | 126 | ASSERT(. - _entry_trampoline == PAGE_SIZE, "entry trampoline is too big"); |
| 125 | #endif | 127 | #endif |
| 126 | 128 | ||
| @@ -607,7 +607,8 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, · 606 unmodified lines | |||
| 607 | (1 << KVM_FEATURE_PV_EOI) | | 607 | (1 << KVM_FEATURE_PV_EOI) | |
| 608 | (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT) | | 608 | (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT) | |
| 609 | (1 << KVM_FEATURE_PV_UNHALT) | | 609 | (1 << KVM_FEATURE_PV_UNHALT) | |
| 610 | (1 << KVM_FEATURE_PV_TLB_FLUSH); | 610 | (1 << KVM_FEATURE_PV_TLB_FLUSH) | |
| 611 | (1 << KVM_FEATURE_ASYNC_PF_VMEXIT); | ||
| 611 | 612 | ||
| 612 | if (sched_info_on()) | 613 | if (sched_info_on()) |
| 613 | entry->eax |= (1 << KVM_FEATURE_STEAL_TIME); | 614 | entry->eax |= (1 << KVM_FEATURE_STEAL_TIME); |
| @@ -2002,14 +2002,13 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value) · 2001 unmodified lines | |||
| 2002 | 2002 | ||
| 2003 | void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) | 2003 | void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) |
| 2004 | { | 2004 | { |
| 2005 | struct kvm_lapic *apic; | 2005 | struct kvm_lapic *apic = vcpu->arch.apic; |
| 2006 | int i; | 2006 | int i; |
| 2007 | 2007 | ||
| 2008 | apic_debug("%s\n", __func__); | 2008 | if (!apic) |
| 2009 | return; | ||
| 2009 | 2010 | ||
| 2010 | ASSERT(vcpu); | 2011 | apic_debug("%s\n", __func__); |
| 2011 | apic = vcpu->arch.apic; | ||
| 2012 | ASSERT(apic != NULL); | ||
| 2013 | 2012 | ||
| 2014 | /* Stop the timer in case it's a reset to an active apic */ | 2013 | /* Stop the timer in case it's a reset to an active apic */ |
| 2015 | hrtimer_cancel(&apic->lapic_timer.timer); | 2014 | hrtimer_cancel(&apic->lapic_timer.timer); |
| @@ -2165,7 +2164,6 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu) · 149 unmodified lines | |||
| 2165 | */ | 2164 | */ |
| 2166 | vcpu->arch.apic_base = MSR_IA32_APICBASE_ENABLE; | 2165 | vcpu->arch.apic_base = MSR_IA32_APICBASE_ENABLE; |
| 2167 | static_key_slow_inc(&apic_sw_disabled.key); /* sw disabled at reset */ | 2166 | static_key_slow_inc(&apic_sw_disabled.key); /* sw disabled at reset */ |
| 2168 | kvm_lapic_reset(vcpu, false); | ||
| 2169 | kvm_iodevice_init(&apic->dev, &apic_mmio_ops); | 2167 | kvm_iodevice_init(&apic->dev, &apic_mmio_ops); |
| 2170 | 2168 | ||
| 2171 | return 0; | 2169 | return 0; |
| @@ -2569,7 +2567,6 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) · 397 unmodified lines | |||
| 2569 | 2567 | ||
| 2570 | pe = xchg(&apic->pending_events, 0); | 2568 | pe = xchg(&apic->pending_events, 0); |
| 2571 | if (test_bit(KVM_APIC_INIT, &pe)) { | 2569 | if (test_bit(KVM_APIC_INIT, &pe)) { |
| 2572 | kvm_lapic_reset(vcpu, true); | ||
| 2573 | kvm_vcpu_reset(vcpu, true); | 2570 | kvm_vcpu_reset(vcpu, true); |
| 2574 | if (kvm_vcpu_is_bsp(apic->vcpu)) | 2571 | if (kvm_vcpu_is_bsp(apic->vcpu)) |
| 2575 | vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; | 2572 | vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; |
| @@ -3029,7 +3029,7 @@ static int kvm_handle_bad_page(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · 3028 unmodified lines | |||
| 3029 | return RET_PF_RETRY; | 3029 | return RET_PF_RETRY; |
| 3030 | } | 3030 | } |
| 3031 | 3031 | ||
| 3032 | return -EFAULT; | 3032 | return RET_PF_EMULATE; |
| 3033 | } | 3033 | } |
| 3034 | 3034 | ||
| 3035 | static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, | 3035 | static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, |
| @@ -5080,7 +5080,7 @@ void kvm_mmu_uninit_vm(struct kvm *kvm) · 2044 unmodified lines | |||
| 5080 | typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); | 5080 | typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); |
| 5081 | 5081 | ||
| 5082 | /* The caller should hold mmu-lock before calling this function. */ | 5082 | /* The caller should hold mmu-lock before calling this function. */ |
| 5083 | static bool | 5083 | static __always_inline bool |
| 5084 | slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5084 | slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5085 | slot_level_handler fn, int start_level, int end_level, | 5085 | slot_level_handler fn, int start_level, int end_level, |
| 5086 | gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb) | 5086 | gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb) |
| @@ -5110,7 +5110,7 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, · 23 unmodified lines | |||
| 5110 | return flush; | 5110 | return flush; |
| 5111 | } | 5111 | } |
| 5112 | 5112 | ||
| 5113 | static bool | 5113 | static __always_inline bool |
| 5114 | slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5114 | slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5115 | slot_level_handler fn, int start_level, int end_level, | 5115 | slot_level_handler fn, int start_level, int end_level, |
| 5116 | bool lock_flush_tlb) | 5116 | bool lock_flush_tlb) |
| @@ -5121,7 +5121,7 @@ slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, · 4 unmodified lines | |||
| 5121 | lock_flush_tlb); | 5121 | lock_flush_tlb); |
| 5122 | } | 5122 | } |
| 5123 | 5123 | ||
| 5124 | static bool | 5124 | static __always_inline bool |
| 5125 | slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5125 | slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5126 | slot_level_handler fn, bool lock_flush_tlb) | 5126 | slot_level_handler fn, bool lock_flush_tlb) |
| 5127 | { | 5127 | { |
| @@ -5129,7 +5129,7 @@ slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, · 1 unmodified line | |||
| 5129 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); | 5129 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); |
| 5130 | } | 5130 | } |
| 5131 | 5131 | ||
| 5132 | static bool | 5132 | static __always_inline bool |
| 5133 | slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5133 | slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5134 | slot_level_handler fn, bool lock_flush_tlb) | 5134 | slot_level_handler fn, bool lock_flush_tlb) |
| 5135 | { | 5135 | { |
| @@ -5137,7 +5137,7 @@ slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, · 1 unmodified line | |||
| 5137 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); | 5137 | PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); |
| 5138 | } | 5138 | } |
| 5139 | 5139 | ||
| 5140 | static bool | 5140 | static __always_inline bool |
| 5141 | slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, | 5141 | slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, |
| 5142 | slot_level_handler fn, bool lock_flush_tlb) | 5142 | slot_level_handler fn, bool lock_flush_tlb) |
| 5143 | { | 5143 | { |
| @@ -49,6 +49,7 @@ · 48 unmodified lines | |||
| 49 | #include <asm/debugreg.h> | 49 | #include <asm/debugreg.h> |
| 50 | #include <asm/kvm_para.h> | 50 | #include <asm/kvm_para.h> |
| 51 | #include <asm/irq_remapping.h> | 51 | #include <asm/irq_remapping.h> |
| 52 | #include <asm/microcode.h> | ||
| 52 | #include <asm/nospec-branch.h> | 53 | #include <asm/nospec-branch.h> |
| 53 | 54 | ||
| 54 | #include <asm/virtext.h> | 55 | #include <asm/virtext.h> |
| @@ -178,6 +179,8 @@ struct vcpu_svm { · 123 unmodified lines | |||
| 178 | uint64_t sysenter_eip; | 179 | uint64_t sysenter_eip; |
| 179 | uint64_t tsc_aux; | 180 | uint64_t tsc_aux; |
| 180 | 181 | ||
| 182 | u64 msr_decfg; | ||
| 183 | |||
| 181 | u64 next_rip; | 184 | u64 next_rip; |
| 182 | 185 | ||
| 183 | u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS]; | 186 | u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS]; |
| @@ -300,6 +303,8 @@ module_param(vgif, int, 0444); · 116 unmodified lines | |||
| 300 | static int sev = IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT); | 303 | static int sev = IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT); |
| 301 | module_param(sev, int, 0444); | 304 | module_param(sev, int, 0444); |
| 302 | 305 | ||
| 306 | static u8 rsm_ins_bytes[] = "\x0f\xaa"; | ||
| 307 | |||
| 303 | static void svm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); | 308 | static void svm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); |
| 304 | static void svm_flush_tlb(struct kvm_vcpu *vcpu, bool invalidate_gpa); | 309 | static void svm_flush_tlb(struct kvm_vcpu *vcpu, bool invalidate_gpa); |
| 305 | static void svm_complete_interrupts(struct vcpu_svm *svm); | 310 | static void svm_complete_interrupts(struct vcpu_svm *svm); |
| @@ -1383,6 +1388,7 @@ static void init_vmcb(struct vcpu_svm *svm) · 1077 unmodified lines | |||
| 1383 | set_intercept(svm, INTERCEPT_SKINIT); | 1388 | set_intercept(svm, INTERCEPT_SKINIT); |
| 1384 | set_intercept(svm, INTERCEPT_WBINVD); | 1389 | set_intercept(svm, INTERCEPT_WBINVD); |
| 1385 | set_intercept(svm, INTERCEPT_XSETBV); | 1390 | set_intercept(svm, INTERCEPT_XSETBV); |
| 1391 | set_intercept(svm, INTERCEPT_RSM); | ||
| 1386 | 1392 | ||
| 1387 | if (!kvm_mwait_in_guest()) { | 1393 | if (!kvm_mwait_in_guest()) { |
| 1388 | set_intercept(svm, INTERCEPT_MONITOR); | 1394 | set_intercept(svm, INTERCEPT_MONITOR); |
| @@ -1902,6 +1908,7 @@ static void svm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 513 unmodified lines | |||
| 1902 | u32 dummy; | 1908 | u32 dummy; |
| 1903 | u32 eax = 1; | 1909 | u32 eax = 1; |
| 1904 | 1910 | ||
| 1911 | vcpu->arch.microcode_version = 0x01000065; | ||
| 1905 | svm->spec_ctrl = 0; | 1912 | svm->spec_ctrl = 0; |
| 1906 | 1913 | ||
| 1907 | if (!init_event) { | 1914 | if (!init_event) { |
| @@ -3699,6 +3706,12 @@ static int emulate_on_interception(struct vcpu_svm *svm) · 1791 unmodified lines | |||
| 3699 | return emulate_instruction(&svm->vcpu, 0) == EMULATE_DONE; | 3706 | return emulate_instruction(&svm->vcpu, 0) == EMULATE_DONE; |
| 3700 | } | 3707 | } |
| 3701 | 3708 | ||
| 3709 | static int rsm_interception(struct vcpu_svm *svm) | ||
| 3710 | { | ||
| 3711 | return x86_emulate_instruction(&svm->vcpu, 0, 0, | ||
| 3712 | rsm_ins_bytes, 2) == EMULATE_DONE; | ||
| 3713 | } | ||
| 3714 | |||
| 3702 | static int rdpmc_interception(struct vcpu_svm *svm) | 3715 | static int rdpmc_interception(struct vcpu_svm *svm) |
| 3703 | { | 3716 | { |
| 3704 | int err; | 3717 | int err; |
| @@ -3860,6 +3873,22 @@ static int cr8_write_interception(struct vcpu_svm *svm) · 155 unmodified lines | |||
| 3860 | return 0; | 3873 | return 0; |
| 3861 | } | 3874 | } |
| 3862 | 3875 | ||
| 3876 | static int svm_get_msr_feature(struct kvm_msr_entry *msr) | ||
| 3877 | { | ||
| 3878 | msr->data = 0; | ||
| 3879 | |||
| 3880 | switch (msr->index) { | ||
| 3881 | case MSR_F10H_DECFG: | ||
| 3882 | if (boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) | ||
| 3883 | msr->data |= MSR_F10H_DECFG_LFENCE_SERIALIZE; | ||
| 3884 | break; | ||
| 3885 | default: | ||
| 3886 | return 1; | ||
| 3887 | } | ||
| 3888 | |||
| 3889 | return 0; | ||
| 3890 | } | ||
| 3891 | |||
| 3863 | static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) | 3892 | static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) |
| 3864 | { | 3893 | { |
| 3865 | struct vcpu_svm *svm = to_svm(vcpu); | 3894 | struct vcpu_svm *svm = to_svm(vcpu); |
| @@ -3935,9 +3964,6 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 69 unmodified lines | |||
| 3935 | 3964 | ||
| 3936 | msr_info->data = svm->spec_ctrl; | 3965 | msr_info->data = svm->spec_ctrl; |
| 3937 | break; | 3966 | break; |
| 3938 | case MSR_IA32_UCODE_REV: | ||
| 3939 | msr_info->data = 0x01000065; | ||
| 3940 | break; | ||
| 3941 | case MSR_F15H_IC_CFG: { | 3967 | case MSR_F15H_IC_CFG: { |
| 3942 | 3968 | ||
| 3943 | int family, model; | 3969 | int family, model; |
| @@ -3955,6 +3981,9 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 11 unmodified lines | |||
| 3955 | msr_info->data = 0x1E; | 3981 | msr_info->data = 0x1E; |
| 3956 | } | 3982 | } |
| 3957 | break; | 3983 | break; |
| 3984 | case MSR_F10H_DECFG: | ||
| 3985 | msr_info->data = svm->msr_decfg; | ||
| 3986 | break; | ||
| 3958 | default: | 3987 | default: |
| 3959 | return kvm_get_msr_common(vcpu, msr_info); | 3988 | return kvm_get_msr_common(vcpu, msr_info); |
| 3960 | } | 3989 | } |
| @@ -4133,6 +4162,24 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) · 172 unmodified lines | |||
| 4133 | case MSR_VM_IGNNE: | 4162 | case MSR_VM_IGNNE: |
| 4134 | vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data); | 4163 | vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data); |
| 4135 | break; | 4164 | break; |
| 4165 | case MSR_F10H_DECFG: { | ||
| 4166 | struct kvm_msr_entry msr_entry; | ||
| 4167 | |||
| 4168 | msr_entry.index = msr->index; | ||
| 4169 | if (svm_get_msr_feature(&msr_entry)) | ||
| 4170 | return 1; | ||
| 4171 | |||
| 4172 | /* Check the supported bits */ | ||
| 4173 | if (data & ~msr_entry.data) | ||
| 4174 | return 1; | ||
| 4175 | |||
| 4176 | /* Don't allow the guest to change a bit, #GP */ | ||
| 4177 | if (!msr->host_initiated && (data ^ msr_entry.data)) | ||
| 4178 | return 1; | ||
| 4179 | |||
| 4180 | svm->msr_decfg = data; | ||
| 4181 | break; | ||
| 4182 | } | ||
| 4136 | case MSR_IA32_APICBASE: | 4183 | case MSR_IA32_APICBASE: |
| 4137 | if (kvm_vcpu_apicv_active(vcpu)) | 4184 | if (kvm_vcpu_apicv_active(vcpu)) |
| 4138 | avic_update_vapic_bar(to_svm(vcpu), data); | 4185 | avic_update_vapic_bar(to_svm(vcpu), data); |
| @@ -4541,7 +4588,7 @@ static int (*const svm_exit_handlers[])(struct vcpu_svm *svm) = { · 402 unmodified lines | |||
| 4541 | [SVM_EXIT_MWAIT] = mwait_interception, | 4588 | [SVM_EXIT_MWAIT] = mwait_interception, |
| 4542 | [SVM_EXIT_XSETBV] = xsetbv_interception, | 4589 | [SVM_EXIT_XSETBV] = xsetbv_interception, |
| 4543 | [SVM_EXIT_NPF] = npf_interception, | 4590 | [SVM_EXIT_NPF] = npf_interception, |
| 4544 | [SVM_EXIT_RSM] = emulate_on_interception, | 4591 | [SVM_EXIT_RSM] = rsm_interception, |
| 4545 | [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception, | 4592 | [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception, |
| 4546 | [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception, | 4593 | [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception, |
| 4547 | }; | 4594 | }; |
| @@ -5355,7 +5402,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu) · 807 unmodified lines | |||
| 5355 | * being speculatively taken. | 5402 | * being speculatively taken. |
| 5356 | */ | 5403 | */ |
| 5357 | if (svm->spec_ctrl) | 5404 | if (svm->spec_ctrl) |
| 5358 | wrmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl); | 5405 | native_wrmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl); |
| 5359 | 5406 | ||
| 5360 | asm volatile ( | 5407 | asm volatile ( |
| 5361 | "push %%" _ASM_BP "; \n\t" | 5408 | "push %%" _ASM_BP "; \n\t" |
| @@ -5464,11 +5511,11 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu) · 102 unmodified lines | |||
| 5464 | * If the L02 MSR bitmap does not intercept the MSR, then we need to | 5511 | * If the L02 MSR bitmap does not intercept the MSR, then we need to |
| 5465 | * save it. | 5512 | * save it. |
| 5466 | */ | 5513 | */ |
| 5467 | if (!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)) | 5514 | if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL))) |
| 5468 | rdmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl); | 5515 | svm->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL); |
| 5469 | 5516 | ||
| 5470 | if (svm->spec_ctrl) | 5517 | if (svm->spec_ctrl) |
| 5471 | wrmsrl(MSR_IA32_SPEC_CTRL, 0); | 5518 | native_wrmsrl(MSR_IA32_SPEC_CTRL, 0); |
| 5472 | 5519 | ||
| 5473 | /* Eliminate branch target predictions from guest mode */ | 5520 | /* Eliminate branch target predictions from guest mode */ |
| 5474 | vmexit_fill_RSB(); | 5521 | vmexit_fill_RSB(); |
| @@ -6236,16 +6283,18 @@ e_free: · 761 unmodified lines | |||
| 6236 | 6283 | ||
| 6237 | static int sev_launch_measure(struct kvm *kvm, struct kvm_sev_cmd *argp) | 6284 | static int sev_launch_measure(struct kvm *kvm, struct kvm_sev_cmd *argp) |
| 6238 | { | 6285 | { |
| 6286 | void __user *measure = (void __user *)(uintptr_t)argp->data; | ||
| 6239 | struct kvm_sev_info *sev = &kvm->arch.sev_info; | 6287 | struct kvm_sev_info *sev = &kvm->arch.sev_info; |
| 6240 | struct sev_data_launch_measure *data; | 6288 | struct sev_data_launch_measure *data; |
| 6241 | struct kvm_sev_launch_measure params; | 6289 | struct kvm_sev_launch_measure params; |
| 6290 | void __user *p = NULL; | ||
| 6242 | void *blob = NULL; | 6291 | void *blob = NULL; |
| 6243 | int ret; | 6292 | int ret; |
| 6244 | 6293 | ||
| 6245 | if (!sev_guest(kvm)) | 6294 | if (!sev_guest(kvm)) |
| 6246 | return -ENOTTY; | 6295 | return -ENOTTY; |
| 6247 | 6296 | ||
| 6248 | if (copy_from_user(¶ms, (void __user *)(uintptr_t)argp->data, sizeof(params))) | 6297 | if (copy_from_user(¶ms, measure, sizeof(params))) |
| 6249 | return -EFAULT; | 6298 | return -EFAULT; |
| 6250 | 6299 | ||
| 6251 | data = kzalloc(sizeof(*data), GFP_KERNEL); | 6300 | data = kzalloc(sizeof(*data), GFP_KERNEL); |
| @@ -6256,17 +6305,13 @@ static int sev_launch_measure(struct kvm *kvm, struct kvm_sev_cmd *argp) · 4 unmodified lines | |||
| 6256 | if (!params.len) | 6305 | if (!params.len) |
| 6257 | goto cmd; | 6306 | goto cmd; |
| 6258 | 6307 | ||
| 6259 | if (params.uaddr) { | 6308 | p = (void __user *)(uintptr_t)params.uaddr; |
| 6309 | if (p) { | ||
| 6260 | if (params.len > SEV_FW_BLOB_MAX_SIZE) { | 6310 | if (params.len > SEV_FW_BLOB_MAX_SIZE) { |
| 6261 | ret = -EINVAL; | 6311 | ret = -EINVAL; |
| 6262 | goto e_free; | 6312 | goto e_free; |
| 6263 | } | 6313 | } |
| 6264 | 6314 | ||
| 6265 | if (!access_ok(VERIFY_WRITE, params.uaddr, params.len)) { | ||
| 6266 | ret = -EFAULT; | ||
| 6267 | goto e_free; | ||
| 6268 | } | ||
| 6269 | |||
| 6270 | ret = -ENOMEM; | 6315 | ret = -ENOMEM; |
| 6271 | blob = kmalloc(params.len, GFP_KERNEL); | 6316 | blob = kmalloc(params.len, GFP_KERNEL); |
| 6272 | if (!blob) | 6317 | if (!blob) |
| @@ -6290,13 +6335,13 @@ cmd: · 17 unmodified lines | |||
| 6290 | goto e_free_blob; | 6335 | goto e_free_blob; |
| 6291 | 6336 | ||
| 6292 | if (blob) { | 6337 | if (blob) { |
| 6293 | if (copy_to_user((void __user *)(uintptr_t)params.uaddr, blob, params.len)) | 6338 | if (copy_to_user(p, blob, params.len)) |
| 6294 | ret = -EFAULT; | 6339 | ret = -EFAULT; |
| 6295 | } | 6340 | } |
| 6296 | 6341 | ||
| 6297 | done: | 6342 | done: |
| 6298 | params.len = data->len; | 6343 | params.len = data->len; |
| 6299 | if (copy_to_user((void __user *)(uintptr_t)argp->data, ¶ms, sizeof(params))) | 6344 | if (copy_to_user(measure, ¶ms, sizeof(params))) |
| 6300 | ret = -EFAULT; | 6345 | ret = -EFAULT; |
| 6301 | e_free_blob: | 6346 | e_free_blob: |
| 6302 | kfree(blob); | 6347 | kfree(blob); |
| @@ -6597,7 +6642,7 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) · 294 unmodified lines | |||
| 6597 | struct page **pages; | 6642 | struct page **pages; |
| 6598 | void *blob, *hdr; | 6643 | void *blob, *hdr; |
| 6599 | unsigned long n; | 6644 | unsigned long n; |
| 6600 | int ret; | 6645 | int ret, offset; |
| 6601 | 6646 | ||
| 6602 | if (!sev_guest(kvm)) | 6647 | if (!sev_guest(kvm)) |
| 6603 | return -ENOTTY; | 6648 | return -ENOTTY; |
| @@ -6623,6 +6668,10 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) · 19 unmodified lines | |||
| 6623 | if (!data) | 6668 | if (!data) |
| 6624 | goto e_unpin_memory; | 6669 | goto e_unpin_memory; |
| 6625 | 6670 | ||
| 6671 | offset = params.guest_uaddr & (PAGE_SIZE - 1); | ||
| 6672 | data->guest_address = __sme_page_pa(pages[0]) + offset; | ||
| 6673 | data->guest_len = params.guest_len; | ||
| 6674 | |||
| 6626 | blob = psp_copy_user_blob(params.trans_uaddr, params.trans_len); | 6675 | blob = psp_copy_user_blob(params.trans_uaddr, params.trans_len); |
| 6627 | if (IS_ERR(blob)) { | 6676 | if (IS_ERR(blob)) { |
| 6628 | ret = PTR_ERR(blob); | 6677 | ret = PTR_ERR(blob); |
| @@ -6637,8 +6686,8 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) · 8 unmodified lines | |||
| 6637 | ret = PTR_ERR(hdr); | 6686 | ret = PTR_ERR(hdr); |
| 6638 | goto e_free_blob; | 6687 | goto e_free_blob; |
| 6639 | } | 6688 | } |
| 6640 | data->trans_address = __psp_pa(blob); | 6689 | data->hdr_address = __psp_pa(hdr); |
| 6641 | data->trans_len = params.trans_len; | 6690 | data->hdr_len = params.hdr_len; |
| 6642 | 6691 | ||
| 6643 | data->handle = sev->handle; | 6692 | data->handle = sev->handle; |
| 6644 | ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_SECRET, data, &argp->error); | 6693 | ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_SECRET, data, &argp->error); |
| @@ -6821,6 +6870,7 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = { · 176 unmodified lines | |||
| 6821 | .vcpu_unblocking = svm_vcpu_unblocking, | 6870 | .vcpu_unblocking = svm_vcpu_unblocking, |
| 6822 | 6871 | ||
| 6823 | .update_bp_intercept = update_bp_intercept, | 6872 | .update_bp_intercept = update_bp_intercept, |
| 6873 | .get_msr_feature = svm_get_msr_feature, | ||
| 6824 | .get_msr = svm_get_msr, | 6874 | .get_msr = svm_get_msr, |
| 6825 | .set_msr = svm_set_msr, | 6875 | .set_msr = svm_set_msr, |
| 6826 | .get_segment_base = svm_get_segment_base, | 6876 | .get_segment_base = svm_get_segment_base, |
| @@ -51,6 +51,7 @@ · 50 unmodified lines | |||
| 51 | #include <asm/apic.h> | 51 | #include <asm/apic.h> |
| 52 | #include <asm/irq_remapping.h> | 52 | #include <asm/irq_remapping.h> |
| 53 | #include <asm/mmu_context.h> | 53 | #include <asm/mmu_context.h> |
| 54 | #include <asm/microcode.h> | ||
| 54 | #include <asm/nospec-branch.h> | 55 | #include <asm/nospec-branch.h> |
| 55 | 56 | ||
| 56 | #include "trace.h" | 57 | #include "trace.h" |
| @@ -3226,6 +3227,11 @@ static inline bool vmx_feature_control_msr_valid(struct kvm_vcpu *vcpu, · 3169 unmodified lines | |||
| 3226 | return !(val & ~valid_bits); | 3227 | return !(val & ~valid_bits); |
| 3227 | } | 3228 | } |
| 3228 | 3229 | ||
| 3230 | static int vmx_get_msr_feature(struct kvm_msr_entry *msr) | ||
| 3231 | { | ||
| 3232 | return 1; | ||
| 3233 | } | ||
| 3234 | |||
| 3229 | /* | 3235 | /* |
| 3230 | * Reads an msr value (of 'msr_index') into 'pdata'. | 3236 | * Reads an msr value (of 'msr_index') into 'pdata'. |
| 3231 | * Returns 0 on success, non-0 otherwise. | 3237 | * Returns 0 on success, non-0 otherwise. |
| @@ -4485,7 +4491,8 @@ static int vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) · 1253 unmodified lines | |||
| 4485 | vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL, | 4491 | vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL, |
| 4486 | SECONDARY_EXEC_DESC); | 4492 | SECONDARY_EXEC_DESC); |
| 4487 | hw_cr4 &= ~X86_CR4_UMIP; | 4493 | hw_cr4 &= ~X86_CR4_UMIP; |
| 4488 | } else | 4494 | } else if (!is_guest_mode(vcpu) || |
| 4495 | !nested_cpu_has2(get_vmcs12(vcpu), SECONDARY_EXEC_DESC)) | ||
| 4489 | vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL, | 4496 | vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL, |
| 4490 | SECONDARY_EXEC_DESC); | 4497 | SECONDARY_EXEC_DESC); |
| 4491 | 4498 | ||
| @@ -5765,6 +5772,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) · 1273 unmodified lines | |||
| 5765 | vmx->rmode.vm86_active = 0; | 5772 | vmx->rmode.vm86_active = 0; |
| 5766 | vmx->spec_ctrl = 0; | 5773 | vmx->spec_ctrl = 0; |
| 5767 | 5774 | ||
| 5775 | vcpu->arch.microcode_version = 0x100000000ULL; | ||
| 5768 | vmx->vcpu.arch.regs[VCPU_REGS_RDX] = get_rdx_init_val(); | 5776 | vmx->vcpu.arch.regs[VCPU_REGS_RDX] = get_rdx_init_val(); |
| 5769 | kvm_set_cr8(vcpu, 0); | 5777 | kvm_set_cr8(vcpu, 0); |
| 5770 | 5778 | ||
| @@ -9452,7 +9460,7 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) · 3681 unmodified lines | |||
| 9452 | * being speculatively taken. | 9460 | * being speculatively taken. |
| 9453 | */ | 9461 | */ |
| 9454 | if (vmx->spec_ctrl) | 9462 | if (vmx->spec_ctrl) |
| 9455 | wrmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl); | 9463 | native_wrmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl); |
| 9456 | 9464 | ||
| 9457 | vmx->__launched = vmx->loaded_vmcs->launched; | 9465 | vmx->__launched = vmx->loaded_vmcs->launched; |
| 9458 | asm( | 9466 | asm( |
| @@ -9587,11 +9595,11 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) · 128 unmodified lines | |||
| 9587 | * If the L02 MSR bitmap does not intercept the MSR, then we need to | 9595 | * If the L02 MSR bitmap does not intercept the MSR, then we need to |
| 9588 | * save it. | 9596 | * save it. |
| 9589 | */ | 9597 | */ |
| 9590 | if (!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)) | 9598 | if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL))) |
| 9591 | rdmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl); | 9599 | vmx->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL); |
| 9592 | 9600 | ||
| 9593 | if (vmx->spec_ctrl) | 9601 | if (vmx->spec_ctrl) |
| 9594 | wrmsrl(MSR_IA32_SPEC_CTRL, 0); | 9602 | native_wrmsrl(MSR_IA32_SPEC_CTRL, 0); |
| 9595 | 9603 | ||
| 9596 | /* Eliminate branch target predictions from guest mode */ | 9604 | /* Eliminate branch target predictions from guest mode */ |
| 9597 | vmexit_fill_RSB(); | 9605 | vmexit_fill_RSB(); |
| @@ -10136,7 +10144,10 @@ static void nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, · 538 unmodified lines | |||
| 10136 | (unsigned long)(vmcs12->posted_intr_desc_addr & | 10144 | (unsigned long)(vmcs12->posted_intr_desc_addr & |
| 10137 | (PAGE_SIZE - 1))); | 10145 | (PAGE_SIZE - 1))); |
| 10138 | } | 10146 | } |
| 10139 | if (!nested_vmx_prepare_msr_bitmap(vcpu, vmcs12)) | 10147 | if (nested_vmx_prepare_msr_bitmap(vcpu, vmcs12)) |
| 10148 | vmcs_set_bits(CPU_BASED_VM_EXEC_CONTROL, | ||
| 10149 | CPU_BASED_USE_MSR_BITMAPS); | ||
| 10150 | else | ||
| 10140 | vmcs_clear_bits(CPU_BASED_VM_EXEC_CONTROL, | 10151 | vmcs_clear_bits(CPU_BASED_VM_EXEC_CONTROL, |
| 10141 | CPU_BASED_USE_MSR_BITMAPS); | 10152 | CPU_BASED_USE_MSR_BITMAPS); |
| 10142 | } | 10153 | } |
| @@ -10224,8 +10235,8 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, · 81 unmodified lines | |||
| 10224 | * updated to reflect this when L1 (or its L2s) actually write to | 10235 | * updated to reflect this when L1 (or its L2s) actually write to |
| 10225 | * the MSR. | 10236 | * the MSR. |
| 10226 | */ | 10237 | */ |
| 10227 | bool pred_cmd = msr_write_intercepted_l01(vcpu, MSR_IA32_PRED_CMD); | 10238 | bool pred_cmd = !msr_write_intercepted_l01(vcpu, MSR_IA32_PRED_CMD); |
| 10228 | bool spec_ctrl = msr_write_intercepted_l01(vcpu, MSR_IA32_SPEC_CTRL); | 10239 | bool spec_ctrl = !msr_write_intercepted_l01(vcpu, MSR_IA32_SPEC_CTRL); |
| 10229 | 10240 | ||
| 10230 | /* Nothing to do if the MSR bitmap is not in use. */ | 10241 | /* Nothing to do if the MSR bitmap is not in use. */ |
| 10231 | if (!cpu_has_vmx_msr_bitmap() || | 10242 | if (!cpu_has_vmx_msr_bitmap() || |
| @@ -11196,7 +11207,12 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) · 964 unmodified lines | |||
| 11196 | if (ret) | 11207 | if (ret) |
| 11197 | return ret; | 11208 | return ret; |
| 11198 | 11209 | ||
| 11199 | if (vmcs12->guest_activity_state == GUEST_ACTIVITY_HLT) | 11210 | /* |
| 11211 | * If we're entering a halted L2 vcpu and the L2 vcpu won't be woken | ||
| 11212 | * by event injection, halt vcpu. | ||
| 11213 | */ | ||
| 11214 | if ((vmcs12->guest_activity_state == GUEST_ACTIVITY_HLT) && | ||
| 11215 | !(vmcs12->vm_entry_intr_info_field & INTR_INFO_VALID_MASK)) | ||
| 11200 | return kvm_vcpu_halt(vcpu); | 11216 | return kvm_vcpu_halt(vcpu); |
| 11201 | 11217 | ||
| 11202 | vmx->nested.nested_run_pending = 1; | 11218 | vmx->nested.nested_run_pending = 1; |
| @@ -12287,6 +12303,7 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { · 1084 unmodified lines | |||
| 12287 | .vcpu_put = vmx_vcpu_put, | 12303 | .vcpu_put = vmx_vcpu_put, |
| 12288 | 12304 | ||
| 12289 | .update_bp_intercept = update_exception_bitmap, | 12305 | .update_bp_intercept = update_exception_bitmap, |
| 12306 | .get_msr_feature = vmx_get_msr_feature, | ||
| 12290 | .get_msr = vmx_get_msr, | 12307 | .get_msr = vmx_get_msr, |
| 12291 | .set_msr = vmx_set_msr, | 12308 | .set_msr = vmx_set_msr, |
| 12292 | .get_segment_base = vmx_get_segment_base, | 12309 | .get_segment_base = vmx_get_segment_base, |
| @@ -1049,6 +1049,45 @@ static u32 emulated_msrs[] = { · 1048 unmodified lines | |||
| 1049 | 1049 | ||
| 1050 | static unsigned num_emulated_msrs; | 1050 | static unsigned num_emulated_msrs; |
| 1051 | 1051 | ||
| 1052 | /* | ||
| 1053 | * List of msr numbers which are used to expose MSR-based features that | ||
| 1054 | * can be used by a hypervisor to validate requested CPU features. | ||
| 1055 | */ | ||
| 1056 | static u32 msr_based_features[] = { | ||
| 1057 | MSR_F10H_DECFG, | ||
| 1058 | MSR_IA32_UCODE_REV, | ||
| 1059 | }; | ||
| 1060 | |||
| 1061 | static unsigned int num_msr_based_features; | ||
| 1062 | |||
| 1063 | static int kvm_get_msr_feature(struct kvm_msr_entry *msr) | ||
| 1064 | { | ||
| 1065 | switch (msr->index) { | ||
| 1066 | case MSR_IA32_UCODE_REV: | ||
| 1067 | rdmsrl(msr->index, msr->data); | ||
| 1068 | break; | ||
| 1069 | default: | ||
| 1070 | if (kvm_x86_ops->get_msr_feature(msr)) | ||
| 1071 | return 1; | ||
| 1072 | } | ||
| 1073 | return 0; | ||
| 1074 | } | ||
| 1075 | |||
| 1076 | static int do_get_msr_feature(struct kvm_vcpu *vcpu, unsigned index, u64 *data) | ||
| 1077 | { | ||
| 1078 | struct kvm_msr_entry msr; | ||
| 1079 | int r; | ||
| 1080 | |||
| 1081 | msr.index = index; | ||
| 1082 | r = kvm_get_msr_feature(&msr); | ||
| 1083 | if (r) | ||
| 1084 | return r; | ||
| 1085 | |||
| 1086 | *data = msr.data; | ||
| 1087 | |||
| 1088 | return 0; | ||
| 1089 | } | ||
| 1090 | |||
| 1052 | bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) | 1091 | bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) |
| 1053 | { | 1092 | { |
| 1054 | if (efer & efer_reserved_bits) | 1093 | if (efer & efer_reserved_bits) |
| @@ -2222,7 +2261,6 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 1167 unmodified lines | |||
| 2222 | 2261 | ||
| 2223 | switch (msr) { | 2262 | switch (msr) { |
| 2224 | case MSR_AMD64_NB_CFG: | 2263 | case MSR_AMD64_NB_CFG: |
| 2225 | case MSR_IA32_UCODE_REV: | ||
| 2226 | case MSR_IA32_UCODE_WRITE: | 2264 | case MSR_IA32_UCODE_WRITE: |
| 2227 | case MSR_VM_HSAVE_PA: | 2265 | case MSR_VM_HSAVE_PA: |
| 2228 | case MSR_AMD64_PATCH_LOADER: | 2266 | case MSR_AMD64_PATCH_LOADER: |
| @@ -2230,6 +2268,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 1 unmodified line | |||
| 2230 | case MSR_AMD64_DC_CFG: | 2268 | case MSR_AMD64_DC_CFG: |
| 2231 | break; | 2269 | break; |
| 2232 | 2270 | ||
| 2271 | case MSR_IA32_UCODE_REV: | ||
| 2272 | if (msr_info->host_initiated) | ||
| 2273 | vcpu->arch.microcode_version = data; | ||
| 2274 | break; | ||
| 2233 | case MSR_EFER: | 2275 | case MSR_EFER: |
| 2234 | return set_efer(vcpu, data); | 2276 | return set_efer(vcpu, data); |
| 2235 | case MSR_K7_HWCR: | 2277 | case MSR_K7_HWCR: |
| @@ -2525,7 +2567,7 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) · 289 unmodified lines | |||
| 2525 | msr_info->data = 0; | 2567 | msr_info->data = 0; |
| 2526 | break; | 2568 | break; |
| 2527 | case MSR_IA32_UCODE_REV: | 2569 | case MSR_IA32_UCODE_REV: |
| 2528 | msr_info->data = 0x100000000ULL; | 2570 | msr_info->data = vcpu->arch.microcode_version; |
| 2529 | break; | 2571 | break; |
| 2530 | case MSR_MTRRcap: | 2572 | case MSR_MTRRcap: |
| 2531 | case 0x200 ... 0x2ff: | 2573 | case 0x200 ... 0x2ff: |
| @@ -2680,13 +2722,11 @@ static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, · 148 unmodified lines | |||
| 2680 | int (*do_msr)(struct kvm_vcpu *vcpu, | 2722 | int (*do_msr)(struct kvm_vcpu *vcpu, |
| 2681 | unsigned index, u64 *data)) | 2723 | unsigned index, u64 *data)) |
| 2682 | { | 2724 | { |
| 2683 | int i, idx; | 2725 | int i; |
| 2684 | 2726 | ||
| 2685 | idx = srcu_read_lock(&vcpu->kvm->srcu); | ||
| 2686 | for (i = 0; i < msrs->nmsrs; ++i) | 2727 | for (i = 0; i < msrs->nmsrs; ++i) |
| 2687 | if (do_msr(vcpu, entries[i].index, &entries[i].data)) | 2728 | if (do_msr(vcpu, entries[i].index, &entries[i].data)) |
| 2688 | break; | 2729 | break; |
| 2689 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | ||
| 2690 | 2730 | ||
| 2691 | return i; | 2731 | return i; |
| 2692 | } | 2732 | } |
| @@ -2785,6 +2825,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 92 unmodified lines | |||
| 2785 | case KVM_CAP_SET_BOOT_CPU_ID: | 2825 | case KVM_CAP_SET_BOOT_CPU_ID: |
| 2786 | case KVM_CAP_SPLIT_IRQCHIP: | 2826 | case KVM_CAP_SPLIT_IRQCHIP: |
| 2787 | case KVM_CAP_IMMEDIATE_EXIT: | 2827 | case KVM_CAP_IMMEDIATE_EXIT: |
| 2828 | case KVM_CAP_GET_MSR_FEATURES: | ||
| 2788 | r = 1; | 2829 | r = 1; |
| 2789 | break; | 2830 | break; |
| 2790 | case KVM_CAP_ADJUST_CLOCK: | 2831 | case KVM_CAP_ADJUST_CLOCK: |
| @@ -2899,6 +2940,31 @@ long kvm_arch_dev_ioctl(struct file *filp, · 108 unmodified lines | |||
| 2899 | goto out; | 2940 | goto out; |
| 2900 | r = 0; | 2941 | r = 0; |
| 2901 | break; | 2942 | break; |
| 2943 | case KVM_GET_MSR_FEATURE_INDEX_LIST: { | ||
| 2944 | struct kvm_msr_list __user *user_msr_list = argp; | ||
| 2945 | struct kvm_msr_list msr_list; | ||
| 2946 | unsigned int n; | ||
| 2947 | |||
| 2948 | r = -EFAULT; | ||
| 2949 | if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) | ||
| 2950 | goto out; | ||
| 2951 | n = msr_list.nmsrs; | ||
| 2952 | msr_list.nmsrs = num_msr_based_features; | ||
| 2953 | if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) | ||
| 2954 | goto out; | ||
| 2955 | r = -E2BIG; | ||
| 2956 | if (n < msr_list.nmsrs) | ||
| 2957 | goto out; | ||
| 2958 | r = -EFAULT; | ||
| 2959 | if (copy_to_user(user_msr_list->indices, &msr_based_features, | ||
| 2960 | num_msr_based_features * sizeof(u32))) | ||
| 2961 | goto out; | ||
| 2962 | r = 0; | ||
| 2963 | break; | ||
| 2964 | } | ||
| 2965 | case KVM_GET_MSRS: | ||
| 2966 | r = msr_io(NULL, argp, do_get_msr_feature, 1); | ||
| 2967 | break; | ||
| 2902 | } | 2968 | } |
| 2903 | default: | 2969 | default: |
| 2904 | r = -EINVAL; | 2970 | r = -EINVAL; |
| @@ -3636,12 +3702,18 @@ long kvm_arch_vcpu_ioctl(struct file *filp, · 731 unmodified lines | |||
| 3636 | r = 0; | 3702 | r = 0; |
| 3637 | break; | 3703 | break; |
| 3638 | } | 3704 | } |
| 3639 | case KVM_GET_MSRS: | 3705 | case KVM_GET_MSRS: { |
| 3706 | int idx = srcu_read_lock(&vcpu->kvm->srcu); | ||
| 3640 | r = msr_io(vcpu, argp, do_get_msr, 1); | 3707 | r = msr_io(vcpu, argp, do_get_msr, 1); |
| 3708 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | ||
| 3641 | break; | 3709 | break; |
| 3642 | case KVM_SET_MSRS: | 3710 | } |
| 3711 | case KVM_SET_MSRS: { | ||
| 3712 | int idx = srcu_read_lock(&vcpu->kvm->srcu); | ||
| 3643 | r = msr_io(vcpu, argp, do_set_msr, 0); | 3713 | r = msr_io(vcpu, argp, do_set_msr, 0); |
| 3714 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | ||
| 3644 | break; | 3715 | break; |
| 3716 | } | ||
| 3645 | case KVM_TPR_ACCESS_REPORTING: { | 3717 | case KVM_TPR_ACCESS_REPORTING: { |
| 3646 | struct kvm_tpr_access_ctl tac; | 3718 | struct kvm_tpr_access_ctl tac; |
| 3647 | 3719 | ||
| @@ -4464,6 +4536,19 @@ static void kvm_init_msr_list(void) · 816 unmodified lines | |||
| 4464 | j++; | 4536 | j++; |
| 4465 | } | 4537 | } |
| 4466 | num_emulated_msrs = j; | 4538 | num_emulated_msrs = j; |
| 4539 | |||
| 4540 | for (i = j = 0; i < ARRAY_SIZE(msr_based_features); i++) { | ||
| 4541 | struct kvm_msr_entry msr; | ||
| 4542 | |||
| 4543 | msr.index = msr_based_features[i]; | ||
| 4544 | if (kvm_get_msr_feature(&msr)) | ||
| 4545 | continue; | ||
| 4546 | |||
| 4547 | if (j < i) | ||
| 4548 | msr_based_features[j] = msr_based_features[i]; | ||
| 4549 | j++; | ||
| 4550 | } | ||
| 4551 | num_msr_based_features = j; | ||
| 4467 | } | 4552 | } |
| 4468 | 4553 | ||
| 4469 | static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, | 4554 | static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, |
| @@ -8017,6 +8102,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) · 3547 unmodified lines | |||
| 8017 | 8102 | ||
| 8018 | void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) | 8103 | void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) |
| 8019 | { | 8104 | { |
| 8105 | kvm_lapic_reset(vcpu, init_event); | ||
| 8106 | |||
| 8020 | vcpu->arch.hflags = 0; | 8107 | vcpu->arch.hflags = 0; |
| 8021 | 8108 | ||
| 8022 | vcpu->arch.smi_pending = 0; | 8109 | vcpu->arch.smi_pending = 0; |
| @@ -8460,10 +8547,8 @@ int __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, u32 size) · 437 unmodified lines | |||
| 8460 | return r; | 8547 | return r; |
| 8461 | } | 8548 | } |
| 8462 | 8549 | ||
| 8463 | if (!size) { | 8550 | if (!size) |
| 8464 | r = vm_munmap(old.userspace_addr, old.npages * PAGE_SIZE); | 8551 | vm_munmap(old.userspace_addr, old.npages * PAGE_SIZE); |
| 8465 | WARN_ON(r < 0); | ||
| 8466 | } | ||
| 8467 | 8552 | ||
| 8468 | return 0; | 8553 | return 0; |
| 8469 | } | 8554 | } |
| @@ -28,7 +28,6 @@ lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o · 27 unmodified lines | |||
| 28 | lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o | 28 | lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o |
| 29 | lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o | 29 | lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o |
| 30 | lib-$(CONFIG_RETPOLINE) += retpoline.o | 30 | lib-$(CONFIG_RETPOLINE) += retpoline.o |
| 31 | OBJECT_FILES_NON_STANDARD_retpoline.o :=y | ||
| 32 | 31 | ||
| 33 | obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o | 32 | obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o |
| 34 | 33 | ||
| @@ -18,7 +18,7 @@ unsigned int x86_model(unsigned int sig) · 17 unmodified lines | |||
| 18 | { | 18 | { |
| 19 | unsigned int fam, model; | 19 | unsigned int fam, model; |
| 20 | 20 | ||
| 21 | fam = x86_family(sig); | 21 | fam = x86_family(sig); |
| 22 | 22 | ||
| 23 | model = (sig >> 4) & 0xf; | 23 | model = (sig >> 4) & 0xf; |
| 24 | 24 | ||
| @@ -7,6 +7,7 @@ asmlinkage void just_return_func(void); · 6 unmodified lines | |||
| 7 | 7 | ||
| 8 | asm( | 8 | asm( |
| 9 | ".type just_return_func, @function\n" | 9 | ".type just_return_func, @function\n" |
| 10 | ".globl just_return_func\n" | ||
| 10 | "just_return_func:\n" | 11 | "just_return_func:\n" |
| 11 | " ret\n" | 12 | " ret\n" |
| 12 | ".size just_return_func, .-just_return_func\n" | 13 | ".size just_return_func, .-just_return_func\n" |
| @@ -7,7 +7,6 @@ · 6 unmodified lines | |||
| 7 | #include <asm/alternative-asm.h> | 7 | #include <asm/alternative-asm.h> |
| 8 | #include <asm/export.h> | 8 | #include <asm/export.h> |
| 9 | #include <asm/nospec-branch.h> | 9 | #include <asm/nospec-branch.h> |
| 10 | #include <asm/bitsperlong.h> | ||
| 11 | 10 | ||
| 12 | .macro THUNK reg | 11 | .macro THUNK reg |
| 13 | .section .text.__x86.indirect_thunk | 12 | .section .text.__x86.indirect_thunk |
| @@ -47,58 +46,3 @@ GENERATE_THUNK(r13) · 33 unmodified lines | |||
| 47 | GENERATE_THUNK(r14) | 46 | GENERATE_THUNK(r14) |
| 48 | GENERATE_THUNK(r15) | 47 | GENERATE_THUNK(r15) |
| 49 | #endif | 48 | #endif |
| 50 | |||
| 51 | /* | ||
| 52 | * Fill the CPU return stack buffer. | ||
| 53 | * | ||
| 54 | * Each entry in the RSB, if used for a speculative 'ret', contains an | ||
| 55 | * infinite 'pause; lfence; jmp' loop to capture speculative execution. | ||
| 56 | * | ||
| 57 | * This is required in various cases for retpoline and IBRS-based | ||
| 58 | * mitigations for the Spectre variant 2 vulnerability. Sometimes to | ||
| 59 | * eliminate potentially bogus entries from the RSB, and sometimes | ||
| 60 | * purely to ensure that it doesn't get empty, which on some CPUs would | ||
| 61 | * allow predictions from other (unwanted!) sources to be used. | ||
| 62 | * | ||
| 63 | * Google experimented with loop-unrolling and this turned out to be | ||
| 64 | * the optimal version - two calls, each with their own speculation | ||
| 65 | * trap should their return address end up getting used, in a loop. | ||
| 66 | */ | ||
| 67 | .macro STUFF_RSB nr:req sp:req | ||
| 68 | mov $(\nr / 2), %_ASM_BX | ||
| 69 | .align 16 | ||
| 70 | 771: | ||
| 71 | call 772f | ||
| 72 | 773: /* speculation trap */ | ||
| 73 | pause | ||
| 74 | lfence | ||
| 75 | jmp 773b | ||
| 76 | .align 16 | ||
| 77 | 772: | ||
| 78 | call 774f | ||
| 79 | 775: /* speculation trap */ | ||
| 80 | pause | ||
| 81 | lfence | ||
| 82 | jmp 775b | ||
| 83 | .align 16 | ||
| 84 | 774: | ||
| 85 | dec %_ASM_BX | ||
| 86 | jnz 771b | ||
| 87 | add $((BITS_PER_LONG/8) * \nr), \sp | ||
| 88 | .endm | ||
| 89 | |||
| 90 | #define RSB_FILL_LOOPS 16 /* To avoid underflow */ | ||
| 91 | |||
| 92 | ENTRY(__fill_rsb) | ||
| 93 | STUFF_RSB RSB_FILL_LOOPS, %_ASM_SP | ||
| 94 | ret | ||
| 95 | END(__fill_rsb) | ||
| 96 | EXPORT_SYMBOL_GPL(__fill_rsb) | ||
| 97 | |||
| 98 | #define RSB_CLEAR_LOOPS 32 /* To forcibly overwrite all entries */ | ||
| 99 | |||
| 100 | ENTRY(__clear_rsb) | ||
| 101 | STUFF_RSB RSB_CLEAR_LOOPS, %_ASM_SP | ||
| 102 | ret | ||
| 103 | END(__clear_rsb) | ||
| 104 | EXPORT_SYMBOL_GPL(__clear_rsb) | ||
| @@ -163,4 +163,10 @@ void __init setup_cpu_entry_areas(void) · 162 unmodified lines | |||
| 163 | 163 | ||
| 164 | for_each_possible_cpu(cpu) | 164 | for_each_possible_cpu(cpu) |
| 165 | setup_cpu_entry_area(cpu); | 165 | setup_cpu_entry_area(cpu); |
| 166 | |||
| 167 | /* | ||
| 168 | * This is the last essential update to swapper_pgdir which needs | ||
| 169 | * to be synchronized to initial_page_table on 32bit. | ||
| 170 | */ | ||
| 171 | sync_initial_page_table(); | ||
| 166 | } | 172 | } |
| @@ -1248,10 +1248,6 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, · 1247 unmodified lines | |||
| 1248 | tsk = current; | 1248 | tsk = current; |
| 1249 | mm = tsk->mm; | 1249 | mm = tsk->mm; |
| 1250 | 1250 | ||
| 1251 | /* | ||
| 1252 | * Detect and handle instructions that would cause a page fault for | ||
| 1253 | * both a tracked kernel page and a userspace page. | ||
| 1254 | */ | ||
| 1255 | prefetchw(&mm->mmap_sem); | 1251 | prefetchw(&mm->mmap_sem); |
| 1256 | 1252 | ||
| 1257 | if (unlikely(kmmio_fault(regs, address))) | 1253 | if (unlikely(kmmio_fault(regs, address))) |
| @@ -453,6 +453,21 @@ static inline void permanent_kmaps_init(pgd_t *pgd_base) · 452 unmodified lines | |||
| 453 | } | 453 | } |
| 454 | #endif /* CONFIG_HIGHMEM */ | 454 | #endif /* CONFIG_HIGHMEM */ |
| 455 | 455 | ||
| 456 | void __init sync_initial_page_table(void) | ||
| 457 | { | ||
| 458 | clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY, | ||
| 459 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 460 | KERNEL_PGD_PTRS); | ||
| 461 | |||
| 462 | /* | ||
| 463 | * sync back low identity map too. It is used for example | ||
| 464 | * in the 32-bit EFI stub. | ||
| 465 | */ | ||
| 466 | clone_pgd_range(initial_page_table, | ||
| 467 | swapper_pg_dir + KERNEL_PGD_BOUNDARY, | ||
| 468 | min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY)); | ||
| 469 | } | ||
| 470 | |||
| 456 | void __init native_pagetable_init(void) | 471 | void __init native_pagetable_init(void) |
| 457 | { | 472 | { |
| 458 | unsigned long pfn, va; | 473 | unsigned long pfn, va; |
| @@ -256,7 +256,7 @@ static void __set_pte_vaddr(pud_t *pud, unsigned long vaddr, pte_t new_pte) · 255 unmodified lines | |||
| 256 | * It's enough to flush this one mapping. | 256 | * It's enough to flush this one mapping. |
| 257 | * (PGE mappings get flushed as well) | 257 | * (PGE mappings get flushed as well) |
| 258 | */ | 258 | */ |
| 259 | __flush_tlb_one(vaddr); | 259 | __flush_tlb_one_kernel(vaddr); |
| 260 | } | 260 | } |
| 261 | 261 | ||
| 262 | void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte) | 262 | void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte) |
| @@ -1193,8 +1193,8 @@ void __init mem_init(void) · 930 unmodified lines | |||
| 1193 | register_page_bootmem_info(); | 1193 | register_page_bootmem_info(); |
| 1194 | 1194 | ||
| 1195 | /* Register memory areas for /proc/kcore */ | 1195 | /* Register memory areas for /proc/kcore */ |
| 1196 | kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, | 1196 | if (get_gate_vma(&init_mm)) |
| 1197 | PAGE_SIZE, KCORE_OTHER); | 1197 | kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, PAGE_SIZE, KCORE_USER); |
| 1198 | 1198 | ||
| 1199 | mem_init_print_info(NULL); | 1199 | mem_init_print_info(NULL); |
| 1200 | } | 1200 | } |
| @@ -820,5 +820,5 @@ void __init __early_set_fixmap(enum fixed_addresses idx, · 819 unmodified lines | |||
| 820 | set_pte(pte, pfn_pte(phys >> PAGE_SHIFT, flags)); | 820 | set_pte(pte, pfn_pte(phys >> PAGE_SHIFT, flags)); |
| 821 | else | 821 | else |
| 822 | pte_clear(&init_mm, addr, pte); | 822 | pte_clear(&init_mm, addr, pte); |
| 823 | __flush_tlb_one(addr); | 823 | __flush_tlb_one_kernel(addr); |
| 824 | } | 824 | } |
| @@ -168,7 +168,7 @@ static int clear_page_presence(struct kmmio_fault_page *f, bool clear) · 167 unmodified lines | |||
| 168 | return -1; | 168 | return -1; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | __flush_tlb_one(f->addr); | 171 | __flush_tlb_one_kernel(f->addr); |
| 172 | return 0; | 172 | return 0; |
| 173 | } | 173 | } |
| 174 | 174 | ||
| @@ -15,6 +15,7 @@ · 14 unmodified lines | |||
| 15 | #include <asm/page.h> | 15 | #include <asm/page.h> |
| 16 | #include <asm/processor-flags.h> | 16 | #include <asm/processor-flags.h> |
| 17 | #include <asm/msr-index.h> | 17 | #include <asm/msr-index.h> |
| 18 | #include <asm/nospec-branch.h> | ||
| 18 | 19 | ||
| 19 | .text | 20 | .text |
| 20 | .code64 | 21 | .code64 |
| @@ -59,6 +60,7 @@ ENTRY(sme_encrypt_execute) · 38 unmodified lines | |||
| 59 | movq %rax, %r8 /* Workarea encryption routine */ | 60 | movq %rax, %r8 /* Workarea encryption routine */ |
| 60 | addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ | 61 | addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ |
| 61 | 62 | ||
| 63 | ANNOTATE_RETPOLINE_SAFE | ||
| 62 | call *%rax /* Call the encryption routine */ | 64 | call *%rax /* Call the encryption routine */ |
| 63 | 65 | ||
| 64 | pop %r12 | 66 | pop %r12 |
| @@ -63,7 +63,7 @@ void set_pte_vaddr(unsigned long vaddr, pte_t pteval) · 62 unmodified lines | |||
| 63 | * It's enough to flush this one mapping. | 63 | * It's enough to flush this one mapping. |
| 64 | * (PGE mappings get flushed as well) | 64 | * (PGE mappings get flushed as well) |
| 65 | */ | 65 | */ |
| 66 | __flush_tlb_one(vaddr); | 66 | __flush_tlb_one_kernel(vaddr); |
| 67 | } | 67 | } |
| 68 | 68 | ||
| 69 | unsigned long __FIXADDR_TOP = 0xfffff000; | 69 | unsigned long __FIXADDR_TOP = 0xfffff000; |
| @@ -332,7 +332,7 @@ static void __init pti_clone_user_shared(void) · 331 unmodified lines | |||
| 332 | } | 332 | } |
| 333 | 333 | ||
| 334 | /* | 334 | /* |
| 335 | * Clone the ESPFIX P4D into the user space visinble page table | 335 | * Clone the ESPFIX P4D into the user space visible page table |
| 336 | */ | 336 | */ |
| 337 | static void __init pti_setup_espfix64(void) | 337 | static void __init pti_setup_espfix64(void) |
| 338 | { | 338 | { |
| @@ -498,7 +498,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, · 497 unmodified lines | |||
| 498 | * flush that changes context.tlb_gen from 2 to 3. If they get | 498 | * flush that changes context.tlb_gen from 2 to 3. If they get |
| 499 | * processed on this CPU in reverse order, we'll see | 499 | * processed on this CPU in reverse order, we'll see |
| 500 | * local_tlb_gen == 1, mm_tlb_gen == 3, and end != TLB_FLUSH_ALL. | 500 | * local_tlb_gen == 1, mm_tlb_gen == 3, and end != TLB_FLUSH_ALL. |
| 501 | * If we were to use __flush_tlb_single() and set local_tlb_gen to | 501 | * If we were to use __flush_tlb_one_user() and set local_tlb_gen to |
| 502 | * 3, we'd be break the invariant: we'd update local_tlb_gen above | 502 | * 3, we'd be break the invariant: we'd update local_tlb_gen above |
| 503 | * 1 without the full flush that's needed for tlb_gen 2. | 503 | * 1 without the full flush that's needed for tlb_gen 2. |
| 504 | * | 504 | * |
| @@ -519,7 +519,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, · 14 unmodified lines | |||
| 519 | 519 | ||
| 520 | addr = f->start; | 520 | addr = f->start; |
| 521 | while (addr < f->end) { | 521 | while (addr < f->end) { |
| 522 | __flush_tlb_single(addr); | 522 | __flush_tlb_one_user(addr); |
| 523 | addr += PAGE_SIZE; | 523 | addr += PAGE_SIZE; |
| 524 | } | 524 | } |
| 525 | if (local) | 525 | if (local) |
| @@ -666,7 +666,7 @@ static void do_kernel_range_flush(void *info) · 140 unmodified lines | |||
| 666 | 666 | ||
| 667 | /* flush range by one by one 'invlpg' */ | 667 | /* flush range by one by one 'invlpg' */ |
| 668 | for (addr = f->start; addr < f->end; addr += PAGE_SIZE) | 668 | for (addr = f->start; addr < f->end; addr += PAGE_SIZE) |
| 669 | __flush_tlb_one(addr); | 669 | __flush_tlb_one_kernel(addr); |
| 670 | } | 670 | } |
| 671 | 671 | ||
| 672 | void flush_tlb_kernel_range(unsigned long start, unsigned long end) | 672 | void flush_tlb_kernel_range(unsigned long start, unsigned long end) |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | |||
| 13 | #include <linux/if_vlan.h> | 13 | #include <linux/if_vlan.h> |
| 14 | #include <asm/cacheflush.h> | 14 | #include <asm/cacheflush.h> |
| 15 | #include <asm/set_memory.h> | 15 | #include <asm/set_memory.h> |
| 16 | #include <asm/nospec-branch.h> | ||
| 16 | #include <linux/bpf.h> | 17 | #include <linux/bpf.h> |
| 17 | 18 | ||
| 18 | /* | 19 | /* |
| @@ -290,7 +291,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 271 unmodified lines | |||
| 290 | EMIT2(0x89, 0xD2); /* mov edx, edx */ | 291 | EMIT2(0x89, 0xD2); /* mov edx, edx */ |
| 291 | EMIT3(0x39, 0x56, /* cmp dword ptr [rsi + 16], edx */ | 292 | EMIT3(0x39, 0x56, /* cmp dword ptr [rsi + 16], edx */ |
| 292 | offsetof(struct bpf_array, map.max_entries)); | 293 | offsetof(struct bpf_array, map.max_entries)); |
| 293 | #define OFFSET1 43 /* number of bytes to jump */ | 294 | #define OFFSET1 (41 + RETPOLINE_RAX_BPF_JIT_SIZE) /* number of bytes to jump */ |
| 294 | EMIT2(X86_JBE, OFFSET1); /* jbe out */ | 295 | EMIT2(X86_JBE, OFFSET1); /* jbe out */ |
| 295 | label1 = cnt; | 296 | label1 = cnt; |
| 296 | 297 | ||
| @@ -299,7 +300,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 2 unmodified lines | |||
| 299 | */ | 300 | */ |
| 300 | EMIT2_off32(0x8B, 0x85, 36); /* mov eax, dword ptr [rbp + 36] */ | 301 | EMIT2_off32(0x8B, 0x85, 36); /* mov eax, dword ptr [rbp + 36] */ |
| 301 | EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_CNT */ | 302 | EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_CNT */ |
| 302 | #define OFFSET2 32 | 303 | #define OFFSET2 (30 + RETPOLINE_RAX_BPF_JIT_SIZE) |
| 303 | EMIT2(X86_JA, OFFSET2); /* ja out */ | 304 | EMIT2(X86_JA, OFFSET2); /* ja out */ |
| 304 | label2 = cnt; | 305 | label2 = cnt; |
| 305 | EMIT3(0x83, 0xC0, 0x01); /* add eax, 1 */ | 306 | EMIT3(0x83, 0xC0, 0x01); /* add eax, 1 */ |
| @@ -313,7 +314,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 7 unmodified lines | |||
| 313 | * goto out; | 314 | * goto out; |
| 314 | */ | 315 | */ |
| 315 | EMIT3(0x48, 0x85, 0xC0); /* test rax,rax */ | 316 | EMIT3(0x48, 0x85, 0xC0); /* test rax,rax */ |
| 316 | #define OFFSET3 10 | 317 | #define OFFSET3 (8 + RETPOLINE_RAX_BPF_JIT_SIZE) |
| 317 | EMIT2(X86_JE, OFFSET3); /* je out */ | 318 | EMIT2(X86_JE, OFFSET3); /* je out */ |
| 318 | label3 = cnt; | 319 | label3 = cnt; |
| 319 | 320 | ||
| @@ -326,7 +327,7 @@ static void emit_bpf_tail_call(u8 **pprog) · 6 unmodified lines | |||
| 326 | * rdi == ctx (1st arg) | 327 | * rdi == ctx (1st arg) |
| 327 | * rax == prog->bpf_func + prologue_size | 328 | * rax == prog->bpf_func + prologue_size |
| 328 | */ | 329 | */ |
| 329 | EMIT2(0xFF, 0xE0); /* jmp rax */ | 330 | RETPOLINE_RAX_BPF_JIT(); |
| 330 | 331 | ||
| 331 | /* out: */ | 332 | /* out: */ |
| 332 | BUILD_BUG_ON(cnt - label1 != OFFSET1); | 333 | BUILD_BUG_ON(cnt - label1 != OFFSET1); |
| @@ -460,7 +460,7 @@ static int nmi_setup(void) · 459 unmodified lines | |||
| 460 | goto fail; | 460 | goto fail; |
| 461 | 461 | ||
| 462 | for_each_possible_cpu(cpu) { | 462 | for_each_possible_cpu(cpu) { |
| 463 | if (!cpu) | 463 | if (!IS_ENABLED(CONFIG_SMP) || !cpu) |
| 464 | continue; | 464 | continue; |
| 465 | 465 | ||
| 466 | memcpy(per_cpu(cpu_msrs, cpu).counters, | 466 | memcpy(per_cpu(cpu_msrs, cpu).counters, |
| @@ -79,7 +79,7 @@ static void intel_mid_power_off(void) · 78 unmodified lines | |||
| 79 | 79 | ||
| 80 | static void intel_mid_reboot(void) | 80 | static void intel_mid_reboot(void) |
| 81 | { | 81 | { |
| 82 | intel_scu_ipc_simple_command(IPCMSG_COLD_BOOT, 0); | 82 | intel_scu_ipc_simple_command(IPCMSG_COLD_RESET, 0); |
| 83 | } | 83 | } |
| 84 | 84 | ||
| 85 | static unsigned long __init intel_mid_calibrate_tsc(void) | 85 | static unsigned long __init intel_mid_calibrate_tsc(void) |
| @@ -299,7 +299,7 @@ static void bau_process_message(struct msg_desc *mdp, struct bau_control *bcp, · 298 unmodified lines | |||
| 299 | local_flush_tlb(); | 299 | local_flush_tlb(); |
| 300 | stat->d_alltlb++; | 300 | stat->d_alltlb++; |
| 301 | } else { | 301 | } else { |
| 302 | __flush_tlb_single(msg->address); | 302 | __flush_tlb_one_user(msg->address); |
| 303 | stat->d_onetlb++; | 303 | stat->d_onetlb++; |
| 304 | } | 304 | } |
| 305 | stat->d_requestee++; | 305 | stat->d_requestee++; |
| @@ -102,7 +102,7 @@ ENTRY(startup_32) · 101 unmodified lines | |||
| 102 | * don't we'll eventually crash trying to execute encrypted | 102 | * don't we'll eventually crash trying to execute encrypted |
| 103 | * instructions. | 103 | * instructions. |
| 104 | */ | 104 | */ |
| 105 | bt $TH_FLAGS_SME_ACTIVE_BIT, pa_tr_flags | 105 | btl $TH_FLAGS_SME_ACTIVE_BIT, pa_tr_flags |
| 106 | jnc .Ldone | 106 | jnc .Ldone |
| 107 | movl $MSR_K8_SYSCFG, %ecx | 107 | movl $MSR_K8_SYSCFG, %ecx |
| 108 | rdmsr | 108 | rdmsr |
| @@ -770,9 +770,12 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, · 769 unmodified lines | |||
| 770 | break; | 770 | break; |
| 771 | 771 | ||
| 772 | case R_X86_64_PC32: | 772 | case R_X86_64_PC32: |
| 773 | case R_X86_64_PLT32: | ||
| 773 | /* | 774 | /* |
| 774 | * PC relative relocations don't need to be adjusted unless | 775 | * PC relative relocations don't need to be adjusted unless |
| 775 | * referencing a percpu symbol. | 776 | * referencing a percpu symbol. |
| 777 | * | ||
| 778 | * NB: R_X86_64_PLT32 can be treated as R_X86_64_PC32. | ||
| 776 | */ | 779 | */ |
| 777 | if (is_percpu_sym(sym, symname)) | 780 | if (is_percpu_sym(sym, symname)) |
| 778 | add_reloc(&relocs32neg, offset); | 781 | add_reloc(&relocs32neg, offset); |
| @@ -1376,8 +1376,6 @@ asmlinkage __visible void __init xen_start_kernel(void) · 1375 unmodified lines | |||
| 1376 | 1376 | ||
| 1377 | if (!xen_initial_domain()) { | 1377 | if (!xen_initial_domain()) { |
| 1378 | add_preferred_console("xenboot", 0, NULL); | 1378 | add_preferred_console("xenboot", 0, NULL); |
| 1379 | add_preferred_console("tty", 0, NULL); | ||
| 1380 | add_preferred_console("hvc", 0, NULL); | ||
| 1381 | if (pci_xen) | 1379 | if (pci_xen) |
| 1382 | x86_init.pci.arch_init = pci_xen_init; | 1380 | x86_init.pci.arch_init = pci_xen_init; |
| 1383 | } else { | 1381 | } else { |
| @@ -1410,6 +1408,10 @@ asmlinkage __visible void __init xen_start_kernel(void) · 26 unmodified lines | |||
| 1410 | 1408 | ||
| 1411 | xen_boot_params_init_edd(); | 1409 | xen_boot_params_init_edd(); |
| 1412 | } | 1410 | } |
| 1411 | |||
| 1412 | add_preferred_console("tty", 0, NULL); | ||
| 1413 | add_preferred_console("hvc", 0, NULL); | ||
| 1414 | |||
| 1413 | #ifdef CONFIG_PCI | 1415 | #ifdef CONFIG_PCI |
| 1414 | /* PCI BIOS service won't work from a PV guest. */ | 1416 | /* PCI BIOS service won't work from a PV guest. */ |
| 1415 | pci_probe &= ~PCI_PROBE_BIOS; | 1417 | pci_probe &= ~PCI_PROBE_BIOS; |
| @@ -1300,12 +1300,12 @@ static void xen_flush_tlb(void) · 1299 unmodified lines | |||
| 1300 | preempt_enable(); | 1300 | preempt_enable(); |
| 1301 | } | 1301 | } |
| 1302 | 1302 | ||
| 1303 | static void xen_flush_tlb_single(unsigned long addr) | 1303 | static void xen_flush_tlb_one_user(unsigned long addr) |
| 1304 | { | 1304 | { |
| 1305 | struct mmuext_op *op; | 1305 | struct mmuext_op *op; |
| 1306 | struct multicall_space mcs; | 1306 | struct multicall_space mcs; |
| 1307 | 1307 | ||
| 1308 | trace_xen_mmu_flush_tlb_single(addr); | 1308 | trace_xen_mmu_flush_tlb_one_user(addr); |
| 1309 | 1309 | ||
| 1310 | preempt_disable(); | 1310 | preempt_disable(); |
| 1311 | 1311 | ||
| @@ -2370,7 +2370,7 @@ static const struct pv_mmu_ops xen_mmu_ops __initconst = { · 1058 unmodified lines | |||
| 2370 | 2370 | ||
| 2371 | .flush_tlb_user = xen_flush_tlb, | 2371 | .flush_tlb_user = xen_flush_tlb, |
| 2372 | .flush_tlb_kernel = xen_flush_tlb, | 2372 | .flush_tlb_kernel = xen_flush_tlb, |
| 2373 | .flush_tlb_single = xen_flush_tlb_single, | 2373 | .flush_tlb_one_user = xen_flush_tlb_one_user, |
| 2374 | .flush_tlb_others = xen_flush_tlb_others, | 2374 | .flush_tlb_others = xen_flush_tlb_others, |
| 2375 | 2375 | ||
| 2376 | .pgd_alloc = xen_pgd_alloc, | 2376 | .pgd_alloc = xen_pgd_alloc, |
| @@ -122,6 +122,8 @@ void __init xen_smp_cpus_done(unsigned int max_cpus) · 121 unmodified lines | |||
| 122 | 122 | ||
| 123 | if (xen_hvm_domain()) | 123 | if (xen_hvm_domain()) |
| 124 | native_smp_cpus_done(max_cpus); | 124 | native_smp_cpus_done(max_cpus); |
| 125 | else | ||
| 126 | calculate_max_logical_packages(); | ||
| 125 | 127 | ||
| 126 | if (xen_have_vcpu_info_placement) | 128 | if (xen_have_vcpu_info_placement) |
| 127 | return; | 129 | return; |
| @@ -1,12 +1,15 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | #include <linux/types.h> | 2 | #include <linux/types.h> |
| 3 | #include <linux/tick.h> | 3 | #include <linux/tick.h> |
| 4 | #include <linux/percpu-defs.h> | ||
| 4 | 5 | ||
| 5 | #include <xen/xen.h> | 6 | #include <xen/xen.h> |
| 6 | #include <xen/interface/xen.h> | 7 | #include <xen/interface/xen.h> |
| 7 | #include <xen/grant_table.h> | 8 | #include <xen/grant_table.h> |
| 8 | #include <xen/events.h> | 9 | #include <xen/events.h> |
| 9 | 10 | ||
| 11 | #include <asm/cpufeatures.h> | ||
| 12 | #include <asm/msr-index.h> | ||
| 10 | #include <asm/xen/hypercall.h> | 13 | #include <asm/xen/hypercall.h> |
| 11 | #include <asm/xen/page.h> | 14 | #include <asm/xen/page.h> |
| 12 | #include <asm/fixmap.h> | 15 | #include <asm/fixmap.h> |
| @@ -15,6 +18,8 @@ · 2 unmodified lines | |||
| 15 | #include "mmu.h" | 18 | #include "mmu.h" |
| 16 | #include "pmu.h" | 19 | #include "pmu.h" |
| 17 | 20 | ||
| 21 | static DEFINE_PER_CPU(u64, spec_ctrl); | ||
| 22 | |||
| 18 | void xen_arch_pre_suspend(void) | 23 | void xen_arch_pre_suspend(void) |
| 19 | { | 24 | { |
| 20 | xen_save_time_memory_area(); | 25 | xen_save_time_memory_area(); |
| @@ -35,6 +40,9 @@ void xen_arch_post_suspend(int cancelled) · 14 unmodified lines | |||
| 35 | 40 | ||
| 36 | static void xen_vcpu_notify_restore(void *data) | 41 | static void xen_vcpu_notify_restore(void *data) |
| 37 | { | 42 | { |
| 43 | if (xen_pv_domain() && boot_cpu_has(X86_FEATURE_SPEC_CTRL)) | ||
| 44 | wrmsrl(MSR_IA32_SPEC_CTRL, this_cpu_read(spec_ctrl)); | ||
| 45 | |||
| 38 | /* Boot processor notified via generic timekeeping_resume() */ | 46 | /* Boot processor notified via generic timekeeping_resume() */ |
| 39 | if (smp_processor_id() == 0) | 47 | if (smp_processor_id() == 0) |
| 40 | return; | 48 | return; |
| @@ -44,7 +52,15 @@ static void xen_vcpu_notify_restore(void *data) · 3 unmodified lines | |||
| 44 | 52 | ||
| 45 | static void xen_vcpu_notify_suspend(void *data) | 53 | static void xen_vcpu_notify_suspend(void *data) |
| 46 | { | 54 | { |
| 55 | u64 tmp; | ||
| 56 | |||
| 47 | tick_suspend_local(); | 57 | tick_suspend_local(); |
| 58 | |||
| 59 | if (xen_pv_domain() && boot_cpu_has(X86_FEATURE_SPEC_CTRL)) { | ||
| 60 | rdmsrl(MSR_IA32_SPEC_CTRL, tmp); | ||
| 61 | this_cpu_write(spec_ctrl, tmp); | ||
| 62 | wrmsrl(MSR_IA32_SPEC_CTRL, 0); | ||
| 63 | } | ||
| 48 | } | 64 | } |
| 49 | 65 | ||
| 50 | void xen_arch_resume(void) | 66 | void xen_arch_resume(void) |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | |||
| 16 | */ | 16 | */ |
| 17 | 17 | ||
| 18 | #include <linux/dma-contiguous.h> | 18 | #include <linux/dma-contiguous.h> |
| 19 | #include <linux/dma-direct.h> | ||
| 19 | #include <linux/gfp.h> | 20 | #include <linux/gfp.h> |
| 20 | #include <linux/highmem.h> | 21 | #include <linux/highmem.h> |
| 21 | #include <linux/mm.h> | 22 | #include <linux/mm.h> |
| @@ -123,7 +124,7 @@ static void *xtensa_dma_alloc(struct device *dev, size_t size, · 101 unmodified lines | |||
| 123 | unsigned long attrs) | 124 | unsigned long attrs) |
| 124 | { | 125 | { |
| 125 | unsigned long ret; | 126 | unsigned long ret; |
| 126 | unsigned long uncached = 0; | 127 | unsigned long uncached; |
| 127 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; | 128 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; |
| 128 | struct page *page = NULL; | 129 | struct page *page = NULL; |
| 129 | 130 | ||
| @@ -144,15 +145,27 @@ static void *xtensa_dma_alloc(struct device *dev, size_t size, · 14 unmodified lines | |||
| 144 | if (!page) | 145 | if (!page) |
| 145 | return NULL; | 146 | return NULL; |
| 146 | 147 | ||
| 147 | ret = (unsigned long)page_address(page); | 148 | *handle = phys_to_dma(dev, page_to_phys(page)); |
| 148 | 149 | ||
| 149 | /* We currently don't support coherent memory outside KSEG */ | 150 | #ifdef CONFIG_MMU |
| 151 | if (PageHighMem(page)) { | ||
| 152 | void *p; | ||
| 150 | 153 | ||
| 154 | p = dma_common_contiguous_remap(page, size, VM_MAP, | ||
| 155 | pgprot_noncached(PAGE_KERNEL), | ||
| 156 | __builtin_return_address(0)); | ||
| 157 | if (!p) { | ||
| 158 | if (!dma_release_from_contiguous(dev, page, count)) | ||
| 159 | __free_pages(page, get_order(size)); | ||
| 160 | } | ||
| 161 | return p; | ||
| 162 | } | ||
| 163 | #endif | ||
| 164 | ret = (unsigned long)page_address(page); | ||
| 151 | BUG_ON(ret < XCHAL_KSEG_CACHED_VADDR || | 165 | BUG_ON(ret < XCHAL_KSEG_CACHED_VADDR || |
| 152 | ret > XCHAL_KSEG_CACHED_VADDR + XCHAL_KSEG_SIZE - 1); | 166 | ret > XCHAL_KSEG_CACHED_VADDR + XCHAL_KSEG_SIZE - 1); |
| 153 | 167 | ||
| 154 | uncached = ret + XCHAL_KSEG_BYPASS_VADDR - XCHAL_KSEG_CACHED_VADDR; | 168 | uncached = ret + XCHAL_KSEG_BYPASS_VADDR - XCHAL_KSEG_CACHED_VADDR; |
| 155 | *handle = virt_to_bus((void *)ret); | ||
| 156 | __invalidate_dcache_range(ret, size); | 169 | __invalidate_dcache_range(ret, size); |
| 157 | 170 | ||
| 158 | return (void *)uncached; | 171 | return (void *)uncached; |
| @@ -161,13 +174,20 @@ static void *xtensa_dma_alloc(struct device *dev, size_t size, · 2 unmodified lines | |||
| 161 | static void xtensa_dma_free(struct device *dev, size_t size, void *vaddr, | 174 | static void xtensa_dma_free(struct device *dev, size_t size, void *vaddr, |
| 162 | dma_addr_t dma_handle, unsigned long attrs) | 175 | dma_addr_t dma_handle, unsigned long attrs) |
| 163 | { | 176 | { |
| 164 | unsigned long addr = (unsigned long)vaddr + | ||
| 165 | XCHAL_KSEG_CACHED_VADDR - XCHAL_KSEG_BYPASS_VADDR; | ||
| 166 | struct page *page = virt_to_page(addr); | ||
| 167 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; | 177 | unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; |
| 168 | 178 | unsigned long addr = (unsigned long)vaddr; | |
| 169 | BUG_ON(addr < XCHAL_KSEG_CACHED_VADDR || | 179 | struct page *page; |
| 170 | addr > XCHAL_KSEG_CACHED_VADDR + XCHAL_KSEG_SIZE - 1); | 180 | |
| 181 | if (addr >= XCHAL_KSEG_BYPASS_VADDR && | ||
| 182 | addr - XCHAL_KSEG_BYPASS_VADDR < XCHAL_KSEG_SIZE) { | ||
| 183 | addr += XCHAL_KSEG_CACHED_VADDR - XCHAL_KSEG_BYPASS_VADDR; | ||
| 184 | page = virt_to_page(addr); | ||
| 185 | } else { | ||
| 186 | #ifdef CONFIG_MMU | ||
| 187 | dma_common_free_remap(vaddr, size, VM_MAP); | ||
| 188 | #endif | ||
| 189 | page = pfn_to_page(PHYS_PFN(dma_to_phys(dev, dma_handle))); | ||
| 190 | } | ||
| 171 | 191 | ||
| 172 | if (!dma_release_from_contiguous(dev, page, count)) | 192 | if (!dma_release_from_contiguous(dev, page, count)) |
| 173 | __free_pages(page, get_order(size)); | 193 | __free_pages(page, get_order(size)); |
| @@ -79,19 +79,75 @@ void __init zones_init(void) · 78 unmodified lines | |||
| 79 | free_area_init_node(0, zones_size, ARCH_PFN_OFFSET, NULL); | 79 | free_area_init_node(0, zones_size, ARCH_PFN_OFFSET, NULL); |
| 80 | } | 80 | } |
| 81 | 81 | ||
| 82 | #ifdef CONFIG_HIGHMEM | ||
| 83 | static void __init free_area_high(unsigned long pfn, unsigned long end) | ||
| 84 | { | ||
| 85 | for (; pfn < end; pfn++) | ||
| 86 | free_highmem_page(pfn_to_page(pfn)); | ||
| 87 | } | ||
| 88 | |||
| 89 | static void __init free_highpages(void) | ||
| 90 | { | ||
| 91 | unsigned long max_low = max_low_pfn; | ||
| 92 | struct memblock_region *mem, *res; | ||
| 93 | |||
| 94 | reset_all_zones_managed_pages(); | ||
| 95 | /* set highmem page free */ | ||
| 96 | for_each_memblock(memory, mem) { | ||
| 97 | unsigned long start = memblock_region_memory_base_pfn(mem); | ||
| 98 | unsigned long end = memblock_region_memory_end_pfn(mem); | ||
| 99 | |||
| 100 | /* Ignore complete lowmem entries */ | ||
| 101 | if (end <= max_low) | ||
| 102 | continue; | ||
| 103 | |||
| 104 | if (memblock_is_nomap(mem)) | ||
| 105 | continue; | ||
| 106 | |||
| 107 | /* Truncate partial highmem entries */ | ||
| 108 | if (start < max_low) | ||
| 109 | start = max_low; | ||
| 110 | |||
| 111 | /* Find and exclude any reserved regions */ | ||
| 112 | for_each_memblock(reserved, res) { | ||
| 113 | unsigned long res_start, res_end; | ||
| 114 | |||
| 115 | res_start = memblock_region_reserved_base_pfn(res); | ||
| 116 | res_end = memblock_region_reserved_end_pfn(res); | ||
| 117 | |||
| 118 | if (res_end < start) | ||
| 119 | continue; | ||
| 120 | if (res_start < start) | ||
| 121 | res_start = start; | ||
| 122 | if (res_start > end) | ||
| 123 | res_start = end; | ||
| 124 | if (res_end > end) | ||
| 125 | res_end = end; | ||
| 126 | if (res_start != start) | ||
| 127 | free_area_high(start, res_start); | ||
| 128 | start = res_end; | ||
| 129 | if (start == end) | ||
| 130 | break; | ||
| 131 | } | ||
| 132 | |||
| 133 | /* And now free anything which remains */ | ||
| 134 | if (start < end) | ||
| 135 | free_area_high(start, end); | ||
| 136 | } | ||
| 137 | } | ||
| 138 | #else | ||
| 139 | static void __init free_highpages(void) | ||
| 140 | { | ||
| 141 | } | ||
| 142 | #endif | ||
| 143 | |||
| 82 | /* | 144 | /* |
| 83 | * Initialize memory pages. | 145 | * Initialize memory pages. |
| 84 | */ | 146 | */ |
| 85 | 147 | ||
| 86 | void __init mem_init(void) | 148 | void __init mem_init(void) |
| 87 | { | 149 | { |
| 88 | #ifdef CONFIG_HIGHMEM | 150 | free_highpages(); |
| 89 | unsigned long tmp; | ||
| 90 | |||
| 91 | reset_all_zones_managed_pages(); | ||
| 92 | for (tmp = max_low_pfn; tmp < max_pfn; tmp++) | ||
| 93 | free_highmem_page(pfn_to_page(tmp)); | ||
| 94 | #endif | ||
| 95 | 151 | ||
| 96 | max_mapnr = max_pfn - ARCH_PFN_OFFSET; | 152 | max_mapnr = max_pfn - ARCH_PFN_OFFSET; |
| 97 | high_memory = (void *)__va(max_low_pfn << PAGE_SHIFT); | 153 | high_memory = (void *)__va(max_low_pfn << PAGE_SHIFT); |
| @@ -812,7 +812,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, · 811 unmodified lines | |||
| 812 | struct gendisk *disk; | 812 | struct gendisk *disk; |
| 813 | struct request_queue *q; | 813 | struct request_queue *q; |
| 814 | struct blkcg_gq *blkg; | 814 | struct blkcg_gq *blkg; |
| 815 | struct module *owner; | ||
| 816 | unsigned int major, minor; | 815 | unsigned int major, minor; |
| 817 | int key_len, part, ret; | 816 | int key_len, part, ret; |
| 818 | char *body; | 817 | char *body; |
| @@ -904,9 +903,7 @@ fail_unlock: · 85 unmodified lines | |||
| 904 | spin_unlock_irq(q->queue_lock); | 903 | spin_unlock_irq(q->queue_lock); |
| 905 | rcu_read_unlock(); | 904 | rcu_read_unlock(); |
| 906 | fail: | 905 | fail: |
| 907 | owner = disk->fops->owner; | 906 | put_disk_and_module(disk); |
| 908 | put_disk(disk); | ||
| 909 | module_put(owner); | ||
| 910 | /* | 907 | /* |
| 911 | * If queue was bypassing, we should retry. Do so after a | 908 | * If queue was bypassing, we should retry. Do so after a |
| 912 | * short msleep(). It isn't strictly necessary but queue | 909 | * short msleep(). It isn't strictly necessary but queue |
| @@ -931,13 +928,9 @@ EXPORT_SYMBOL_GPL(blkg_conf_prep); · 18 unmodified lines | |||
| 931 | void blkg_conf_finish(struct blkg_conf_ctx *ctx) | 928 | void blkg_conf_finish(struct blkg_conf_ctx *ctx) |
| 932 | __releases(ctx->disk->queue->queue_lock) __releases(rcu) | 929 | __releases(ctx->disk->queue->queue_lock) __releases(rcu) |
| 933 | { | 930 | { |
| 934 | struct module *owner; | ||
| 935 | |||
| 936 | spin_unlock_irq(ctx->disk->queue->queue_lock); | 931 | spin_unlock_irq(ctx->disk->queue->queue_lock); |
| 937 | rcu_read_unlock(); | 932 | rcu_read_unlock(); |
| 938 | owner = ctx->disk->fops->owner; | 933 | put_disk_and_module(ctx->disk); |
| 939 | put_disk(ctx->disk); | ||
| 940 | module_put(owner); | ||
| 941 | } | 934 | } |
| 942 | EXPORT_SYMBOL_GPL(blkg_conf_finish); | 935 | EXPORT_SYMBOL_GPL(blkg_conf_finish); |
| 943 | 936 | ||
| @@ -2434,7 +2434,7 @@ blk_qc_t submit_bio(struct bio *bio) · 2433 unmodified lines | |||
| 2434 | unsigned int count; | 2434 | unsigned int count; |
| 2435 | 2435 | ||
| 2436 | if (unlikely(bio_op(bio) == REQ_OP_WRITE_SAME)) | 2436 | if (unlikely(bio_op(bio) == REQ_OP_WRITE_SAME)) |
| 2437 | count = queue_logical_block_size(bio->bi_disk->queue); | 2437 | count = queue_logical_block_size(bio->bi_disk->queue) >> 9; |
| 2438 | else | 2438 | else |
| 2439 | count = bio_sectors(bio); | 2439 | count = bio_sectors(bio); |
| 2440 | 2440 | ||
| @@ -712,7 +712,6 @@ static void __blk_mq_requeue_request(struct request *rq) · 711 unmodified lines | |||
| 712 | 712 | ||
| 713 | trace_block_rq_requeue(q, rq); | 713 | trace_block_rq_requeue(q, rq); |
| 714 | wbt_requeue(q->rq_wb, &rq->issue_stat); | 714 | wbt_requeue(q->rq_wb, &rq->issue_stat); |
| 715 | blk_mq_sched_requeue_request(rq); | ||
| 716 | 715 | ||
| 717 | if (blk_mq_rq_state(rq) != MQ_RQ_IDLE) { | 716 | if (blk_mq_rq_state(rq) != MQ_RQ_IDLE) { |
| 718 | blk_mq_rq_update_state(rq, MQ_RQ_IDLE); | 717 | blk_mq_rq_update_state(rq, MQ_RQ_IDLE); |
| @@ -725,6 +724,9 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list) · 6 unmodified lines | |||
| 725 | { | 724 | { |
| 726 | __blk_mq_requeue_request(rq); | 725 | __blk_mq_requeue_request(rq); |
| 727 | 726 | ||
| 727 | /* this request will be re-inserted to io scheduler queue */ | ||
| 728 | blk_mq_sched_requeue_request(rq); | ||
| 729 | |||
| 728 | BUG_ON(blk_queued_rq(rq)); | 730 | BUG_ON(blk_queued_rq(rq)); |
| 729 | blk_mq_add_to_requeue_list(rq, true, kick_requeue_list); | 731 | blk_mq_add_to_requeue_list(rq, true, kick_requeue_list); |
| 730 | } | 732 | } |
| @@ -3164,6 +3166,7 @@ static bool __blk_mq_poll(struct blk_mq_hw_ctx *hctx, struct request *rq) · 2433 unmodified lines | |||
| 3164 | cpu_relax(); | 3166 | cpu_relax(); |
| 3165 | } | 3167 | } |
| 3166 | 3168 | ||
| 3169 | __set_current_state(TASK_RUNNING); | ||
| 3167 | return false; | 3170 | return false; |
| 3168 | } | 3171 | } |
| 3169 | 3172 | ||
| @@ -547,7 +547,7 @@ static int exact_lock(dev_t devt, void *data) · 546 unmodified lines | |||
| 547 | { | 547 | { |
| 548 | struct gendisk *p = data; | 548 | struct gendisk *p = data; |
| 549 | 549 | ||
| 550 | if (!get_disk(p)) | 550 | if (!get_disk_and_module(p)) |
| 551 | return -1; | 551 | return -1; |
| 552 | return 0; | 552 | return 0; |
| 553 | } | 553 | } |
| @@ -717,6 +717,11 @@ void del_gendisk(struct gendisk *disk) · 163 unmodified lines | |||
| 717 | blk_integrity_del(disk); | 717 | blk_integrity_del(disk); |
| 718 | disk_del_events(disk); | 718 | disk_del_events(disk); |
| 719 | 719 | ||
| 720 | /* | ||
| 721 | * Block lookups of the disk until all bdevs are unhashed and the | ||
| 722 | * disk is marked as dead (GENHD_FL_UP cleared). | ||
| 723 | */ | ||
| 724 | down_write(&disk->lookup_sem); | ||
| 720 | /* invalidate stuff */ | 725 | /* invalidate stuff */ |
| 721 | disk_part_iter_init(&piter, disk, | 726 | disk_part_iter_init(&piter, disk, |
| 722 | DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); | 727 | DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); |
| @@ -731,6 +736,7 @@ void del_gendisk(struct gendisk *disk) · 8 unmodified lines | |||
| 731 | bdev_unhash_inode(disk_devt(disk)); | 736 | bdev_unhash_inode(disk_devt(disk)); |
| 732 | set_capacity(disk, 0); | 737 | set_capacity(disk, 0); |
| 733 | disk->flags &= ~GENHD_FL_UP; | 738 | disk->flags &= ~GENHD_FL_UP; |
| 739 | up_write(&disk->lookup_sem); | ||
| 734 | 740 | ||
| 735 | if (!(disk->flags & GENHD_FL_HIDDEN)) | 741 | if (!(disk->flags & GENHD_FL_HIDDEN)) |
| 736 | sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi"); | 742 | sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi"); |
| @@ -809,16 +815,28 @@ struct gendisk *get_gendisk(dev_t devt, int *partno) · 72 unmodified lines | |||
| 809 | 815 | ||
| 810 | spin_lock_bh(&ext_devt_lock); | 816 | spin_lock_bh(&ext_devt_lock); |
| 811 | part = idr_find(&ext_devt_idr, blk_mangle_minor(MINOR(devt))); | 817 | part = idr_find(&ext_devt_idr, blk_mangle_minor(MINOR(devt))); |
| 812 | if (part && get_disk(part_to_disk(part))) { | 818 | if (part && get_disk_and_module(part_to_disk(part))) { |
| 813 | *partno = part->partno; | 819 | *partno = part->partno; |
| 814 | disk = part_to_disk(part); | 820 | disk = part_to_disk(part); |
| 815 | } | 821 | } |
| 816 | spin_unlock_bh(&ext_devt_lock); | 822 | spin_unlock_bh(&ext_devt_lock); |
| 817 | } | 823 | } |
| 818 | 824 | ||
| 819 | if (disk && unlikely(disk->flags & GENHD_FL_HIDDEN)) { | 825 | if (!disk) |
| 820 | put_disk(disk); | 826 | return NULL; |
| 827 | |||
| 828 | /* | ||
| 829 | * Synchronize with del_gendisk() to not return disk that is being | ||
| 830 | * destroyed. | ||
| 831 | */ | ||
| 832 | down_read(&disk->lookup_sem); | ||
| 833 | if (unlikely((disk->flags & GENHD_FL_HIDDEN) || | ||
| 834 | !(disk->flags & GENHD_FL_UP))) { | ||
| 835 | up_read(&disk->lookup_sem); | ||
| 836 | put_disk_and_module(disk); | ||
| 821 | disk = NULL; | 837 | disk = NULL; |
| 838 | } else { | ||
| 839 | up_read(&disk->lookup_sem); | ||
| 822 | } | 840 | } |
| 823 | return disk; | 841 | return disk; |
| 824 | } | 842 | } |
| @@ -1418,6 +1436,7 @@ struct gendisk *__alloc_disk_node(int minors, int node_id) · 593 unmodified lines | |||
| 1418 | kfree(disk); | 1436 | kfree(disk); |
| 1419 | return NULL; | 1437 | return NULL; |
| 1420 | } | 1438 | } |
| 1439 | init_rwsem(&disk->lookup_sem); | ||
| 1421 | disk->node_id = node_id; | 1440 | disk->node_id = node_id; |
| 1422 | if (disk_expand_part_tbl(disk, 0)) { | 1441 | if (disk_expand_part_tbl(disk, 0)) { |
| 1423 | free_part_stats(&disk->part0); | 1442 | free_part_stats(&disk->part0); |
| @@ -1453,7 +1472,7 @@ struct gendisk *__alloc_disk_node(int minors, int node_id) · 29 unmodified lines | |||
| 1453 | } | 1472 | } |
| 1454 | EXPORT_SYMBOL(__alloc_disk_node); | 1473 | EXPORT_SYMBOL(__alloc_disk_node); |
| 1455 | 1474 | ||
| 1456 | struct kobject *get_disk(struct gendisk *disk) | 1475 | struct kobject *get_disk_and_module(struct gendisk *disk) |
| 1457 | { | 1476 | { |
| 1458 | struct module *owner; | 1477 | struct module *owner; |
| 1459 | struct kobject *kobj; | 1478 | struct kobject *kobj; |
| @@ -1471,17 +1490,30 @@ struct kobject *get_disk(struct gendisk *disk) · 11 unmodified lines | |||
| 1471 | return kobj; | 1490 | return kobj; |
| 1472 | 1491 | ||
| 1473 | } | 1492 | } |
| 1474 | 1493 | EXPORT_SYMBOL(get_disk_and_module); | |
| 1475 | EXPORT_SYMBOL(get_disk); | ||
| 1476 | 1494 | ||
| 1477 | void put_disk(struct gendisk *disk) | 1495 | void put_disk(struct gendisk *disk) |
| 1478 | { | 1496 | { |
| 1479 | if (disk) | 1497 | if (disk) |
| 1480 | kobject_put(&disk_to_dev(disk)->kobj); | 1498 | kobject_put(&disk_to_dev(disk)->kobj); |
| 1481 | } | 1499 | } |
| 1482 | |||
| 1483 | EXPORT_SYMBOL(put_disk); | 1500 | EXPORT_SYMBOL(put_disk); |
| 1484 | 1501 | ||
| 1502 | /* | ||
| 1503 | * This is a counterpart of get_disk_and_module() and thus also of | ||
| 1504 | * get_gendisk(). | ||
| 1505 | */ | ||
| 1506 | void put_disk_and_module(struct gendisk *disk) | ||
| 1507 | { | ||
| 1508 | if (disk) { | ||
| 1509 | struct module *owner = disk->fops->owner; | ||
| 1510 | |||
| 1511 | put_disk(disk); | ||
| 1512 | module_put(owner); | ||
| 1513 | } | ||
| 1514 | } | ||
| 1515 | EXPORT_SYMBOL(put_disk_and_module); | ||
| 1516 | |||
| 1485 | static void set_disk_ro_uevent(struct gendisk *gd, int ro) | 1517 | static void set_disk_ro_uevent(struct gendisk *gd, int ro) |
| 1486 | { | 1518 | { |
| 1487 | char event[] = "DISK_RO=1"; | 1519 | char event[] = "DISK_RO=1"; |
| @@ -225,7 +225,7 @@ static int blk_ioctl_discard(struct block_device *bdev, fmode_t mode, · 224 unmodified lines | |||
| 225 | 225 | ||
| 226 | if (start + len > i_size_read(bdev->bd_inode)) | 226 | if (start + len > i_size_read(bdev->bd_inode)) |
| 227 | return -EINVAL; | 227 | return -EINVAL; |
| 228 | truncate_inode_pages_range(mapping, start, start + len); | 228 | truncate_inode_pages_range(mapping, start, start + len - 1); |
| 229 | return blkdev_issue_discard(bdev, start >> 9, len >> 9, | 229 | return blkdev_issue_discard(bdev, start >> 9, len >> 9, |
| 230 | GFP_KERNEL, flags); | 230 | GFP_KERNEL, flags); |
| 231 | } | 231 | } |
| @@ -833,6 +833,7 @@ static struct elevator_type kyber_sched = { · 832 unmodified lines | |||
| 833 | .limit_depth = kyber_limit_depth, | 833 | .limit_depth = kyber_limit_depth, |
| 834 | .prepare_request = kyber_prepare_request, | 834 | .prepare_request = kyber_prepare_request, |
| 835 | .finish_request = kyber_finish_request, | 835 | .finish_request = kyber_finish_request, |
| 836 | .requeue_request = kyber_finish_request, | ||
| 836 | .completed_request = kyber_completed_request, | 837 | .completed_request = kyber_completed_request, |
| 837 | .dispatch_request = kyber_dispatch_request, | 838 | .dispatch_request = kyber_dispatch_request, |
| 838 | .has_work = kyber_has_work, | 839 | .has_work = kyber_has_work, |
| @@ -535,13 +535,22 @@ static void dd_insert_requests(struct blk_mq_hw_ctx *hctx, · 534 unmodified lines | |||
| 535 | spin_unlock(&dd->lock); | 535 | spin_unlock(&dd->lock); |
| 536 | } | 536 | } |
| 537 | 537 | ||
| 538 | /* | ||
| 539 | * Nothing to do here. This is defined only to ensure that .finish_request | ||
| 540 | * method is called upon request completion. | ||
| 541 | */ | ||
| 542 | static void dd_prepare_request(struct request *rq, struct bio *bio) | ||
| 543 | { | ||
| 544 | } | ||
| 545 | |||
| 538 | /* | 546 | /* |
| 539 | * For zoned block devices, write unlock the target zone of | 547 | * For zoned block devices, write unlock the target zone of |
| 540 | * completed write requests. Do this while holding the zone lock | 548 | * completed write requests. Do this while holding the zone lock |
| 541 | * spinlock so that the zone is never unlocked while deadline_fifo_request() | 549 | * spinlock so that the zone is never unlocked while deadline_fifo_request() |
| 542 | * while deadline_next_request() are executing. | 550 | * or deadline_next_request() are executing. This function is called for |
| 551 | * all requests, whether or not these requests complete successfully. | ||
| 543 | */ | 552 | */ |
| 544 | static void dd_completed_request(struct request *rq) | 553 | static void dd_finish_request(struct request *rq) |
| 545 | { | 554 | { |
| 546 | struct request_queue *q = rq->q; | 555 | struct request_queue *q = rq->q; |
| 547 | 556 | ||
| @@ -756,7 +765,8 @@ static struct elevator_type mq_deadline = { · 208 unmodified lines | |||
| 756 | .ops.mq = { | 765 | .ops.mq = { |
| 757 | .insert_requests = dd_insert_requests, | 766 | .insert_requests = dd_insert_requests, |
| 758 | .dispatch_request = dd_dispatch_request, | 767 | .dispatch_request = dd_dispatch_request, |
| 759 | .completed_request = dd_completed_request, | 768 | .prepare_request = dd_prepare_request, |
| 769 | .finish_request = dd_finish_request, | ||
| 760 | .next_request = elv_rb_latter_request, | 770 | .next_request = elv_rb_latter_request, |
| 761 | .former_request = elv_rb_former_request, | 771 | .former_request = elv_rb_former_request, |
| 762 | .bio_merge = dd_bio_merge, | 772 | .bio_merge = dd_bio_merge, |
| @@ -51,6 +51,12 @@ const char *bdevname(struct block_device *bdev, char *buf) · 50 unmodified lines | |||
| 51 | 51 | ||
| 52 | EXPORT_SYMBOL(bdevname); | 52 | EXPORT_SYMBOL(bdevname); |
| 53 | 53 | ||
| 54 | const char *bio_devname(struct bio *bio, char *buf) | ||
| 55 | { | ||
| 56 | return disk_name(bio->bi_disk, bio->bi_partno, buf); | ||
| 57 | } | ||
| 58 | EXPORT_SYMBOL(bio_devname); | ||
| 59 | |||
| 54 | /* | 60 | /* |
| 55 | * There's very little reason to use this, you should really | 61 | * There's very little reason to use this, you should really |
| 56 | * have a struct block_device just about everywhere and use | 62 | * have a struct block_device just about everywhere and use |
| @@ -490,7 +490,7 @@ static int opal_discovery0_end(struct opal_dev *dev) · 489 unmodified lines | |||
| 490 | 490 | ||
| 491 | if (!found_com_id) { | 491 | if (!found_com_id) { |
| 492 | pr_debug("Could not find OPAL comid for device. Returning early\n"); | 492 | pr_debug("Could not find OPAL comid for device. Returning early\n"); |
| 493 | return -EOPNOTSUPP;; | 493 | return -EOPNOTSUPP; |
| 494 | } | 494 | } |
| 495 | 495 | ||
| 496 | dev->comid = comid; | 496 | dev->comid = comid; |
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | #include "blacklist.h" | 2 | #include "blacklist.h" |
| 3 | 3 | ||
| 4 | const char __initdata *const blacklist_hashes[] = { | 4 | const char __initconst *const blacklist_hashes[] = { |
| 5 | NULL | 5 | NULL |
| 6 | }; | 6 | }; |
| @@ -106,6 +106,7 @@ static int pkcs7_validate_trust_one(struct pkcs7_message *pkcs7, · 105 unmodified lines | |||
| 106 | pr_devel("sinfo %u: Direct signer is key %x\n", | 106 | pr_devel("sinfo %u: Direct signer is key %x\n", |
| 107 | sinfo->index, key_serial(key)); | 107 | sinfo->index, key_serial(key)); |
| 108 | x509 = NULL; | 108 | x509 = NULL; |
| 109 | sig = sinfo->sig; | ||
| 109 | goto matched; | 110 | goto matched; |
| 110 | } | 111 | } |
| 111 | if (PTR_ERR(key) != -ENOKEY) | 112 | if (PTR_ERR(key) != -ENOKEY) |
| @@ -270,7 +270,7 @@ static int pkcs7_verify_sig_chain(struct pkcs7_message *pkcs7, · 269 unmodified lines | |||
| 270 | sinfo->index); | 270 | sinfo->index); |
| 271 | return 0; | 271 | return 0; |
| 272 | } | 272 | } |
| 273 | ret = public_key_verify_signature(p->pub, p->sig); | 273 | ret = public_key_verify_signature(p->pub, x509->sig); |
| 274 | if (ret < 0) | 274 | if (ret < 0) |
| 275 | return ret; | 275 | return ret; |
| 276 | x509->signer = p; | 276 | x509->signer = p; |
| @@ -366,8 +366,7 @@ static int pkcs7_verify_one(struct pkcs7_message *pkcs7, · 89 unmodified lines | |||
| 366 | * | 366 | * |
| 367 | * (*) -EBADMSG if some part of the message was invalid, or: | 367 | * (*) -EBADMSG if some part of the message was invalid, or: |
| 368 | * | 368 | * |
| 369 | * (*) 0 if no signature chains were found to be blacklisted or to contain | 369 | * (*) 0 if a signature chain passed verification, or: |
| 370 | * unsupported crypto, or: | ||
| 371 | * | 370 | * |
| 372 | * (*) -EKEYREJECTED if a blacklisted key was encountered, or: | 371 | * (*) -EKEYREJECTED if a blacklisted key was encountered, or: |
| 373 | * | 372 | * |
| @@ -423,8 +422,11 @@ int pkcs7_verify(struct pkcs7_message *pkcs7, · 49 unmodified lines | |||
| 423 | 422 | ||
| 424 | for (sinfo = pkcs7->signed_infos; sinfo; sinfo = sinfo->next) { | 423 | for (sinfo = pkcs7->signed_infos; sinfo; sinfo = sinfo->next) { |
| 425 | ret = pkcs7_verify_one(pkcs7, sinfo); | 424 | ret = pkcs7_verify_one(pkcs7, sinfo); |
| 426 | if (sinfo->blacklisted && actual_ret == -ENOPKG) | 425 | if (sinfo->blacklisted) { |
| 427 | actual_ret = -EKEYREJECTED; | 426 | if (actual_ret == -ENOPKG) |
| 427 | actual_ret = -EKEYREJECTED; | ||
| 428 | continue; | ||
| 429 | } | ||
| 428 | if (ret < 0) { | 430 | if (ret < 0) { |
| 429 | if (ret == -ENOPKG) { | 431 | if (ret == -ENOPKG) { |
| 430 | sinfo->unsupported_crypto = true; | 432 | sinfo->unsupported_crypto = true; |
| @@ -79,9 +79,11 @@ int public_key_verify_signature(const struct public_key *pkey, · 78 unmodified lines | |||
| 79 | 79 | ||
| 80 | BUG_ON(!pkey); | 80 | BUG_ON(!pkey); |
| 81 | BUG_ON(!sig); | 81 | BUG_ON(!sig); |
| 82 | BUG_ON(!sig->digest); | ||
| 83 | BUG_ON(!sig->s); | 82 | BUG_ON(!sig->s); |
| 84 | 83 | ||
| 84 | if (!sig->digest) | ||
| 85 | return -ENOPKG; | ||
| 86 | |||
| 85 | alg_name = sig->pkey_algo; | 87 | alg_name = sig->pkey_algo; |
| 86 | if (strcmp(sig->pkey_algo, "rsa") == 0) { | 88 | if (strcmp(sig->pkey_algo, "rsa") == 0) { |
| 87 | /* The data wangled by the RSA algorithm is typically padded | 89 | /* The data wangled by the RSA algorithm is typically padded |
| @@ -67,8 +67,9 @@ __setup("ca_keys=", ca_keys_setup); · 66 unmodified lines | |||
| 67 | * | 67 | * |
| 68 | * Returns 0 if the new certificate was accepted, -ENOKEY if we couldn't find a | 68 | * Returns 0 if the new certificate was accepted, -ENOKEY if we couldn't find a |
| 69 | * matching parent certificate in the trusted list, -EKEYREJECTED if the | 69 | * matching parent certificate in the trusted list, -EKEYREJECTED if the |
| 70 | * signature check fails or the key is blacklisted and some other error if | 70 | * signature check fails or the key is blacklisted, -ENOPKG if the signature |
| 71 | * there is a matching certificate but the signature check cannot be performed. | 71 | * uses unsupported crypto, or some other error if there is a matching |
| 72 | * certificate but the signature check cannot be performed. | ||
| 72 | */ | 73 | */ |
| 73 | int restrict_link_by_signature(struct key *dest_keyring, | 74 | int restrict_link_by_signature(struct key *dest_keyring, |
| 74 | const struct key_type *type, | 75 | const struct key_type *type, |
| @@ -88,6 +89,8 @@ int restrict_link_by_signature(struct key *dest_keyring, · 13 unmodified lines | |||
| 88 | return -EOPNOTSUPP; | 89 | return -EOPNOTSUPP; |
| 89 | 90 | ||
| 90 | sig = payload->data[asym_auth]; | 91 | sig = payload->data[asym_auth]; |
| 92 | if (!sig) | ||
| 93 | return -ENOPKG; | ||
| 91 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) | 94 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) |
| 92 | return -ENOKEY; | 95 | return -ENOKEY; |
| 93 | 96 | ||
| @@ -139,6 +142,8 @@ static int key_or_keyring_common(struct key *dest_keyring, · 45 unmodified lines | |||
| 139 | return -EOPNOTSUPP; | 142 | return -EOPNOTSUPP; |
| 140 | 143 | ||
| 141 | sig = payload->data[asym_auth]; | 144 | sig = payload->data[asym_auth]; |
| 145 | if (!sig) | ||
| 146 | return -ENOPKG; | ||
| 142 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) | 147 | if (!sig->auth_ids[0] && !sig->auth_ids[1]) |
| 143 | return -ENOKEY; | 148 | return -ENOKEY; |
| 144 | 149 | ||
| @@ -222,9 +227,9 @@ static int key_or_keyring_common(struct key *dest_keyring, · 77 unmodified lines | |||
| 222 | * | 227 | * |
| 223 | * Returns 0 if the new certificate was accepted, -ENOKEY if we | 228 | * Returns 0 if the new certificate was accepted, -ENOKEY if we |
| 224 | * couldn't find a matching parent certificate in the trusted list, | 229 | * couldn't find a matching parent certificate in the trusted list, |
| 225 | * -EKEYREJECTED if the signature check fails, and some other error if | 230 | * -EKEYREJECTED if the signature check fails, -ENOPKG if the signature uses |
| 226 | * there is a matching certificate but the signature check cannot be | 231 | * unsupported crypto, or some other error if there is a matching certificate |
| 227 | * performed. | 232 | * but the signature check cannot be performed. |
| 228 | */ | 233 | */ |
| 229 | int restrict_link_by_key_or_keyring(struct key *dest_keyring, | 234 | int restrict_link_by_key_or_keyring(struct key *dest_keyring, |
| 230 | const struct key_type *type, | 235 | const struct key_type *type, |
| @@ -249,9 +254,9 @@ int restrict_link_by_key_or_keyring(struct key *dest_keyring, · 18 unmodified lines | |||
| 249 | * | 254 | * |
| 250 | * Returns 0 if the new certificate was accepted, -ENOKEY if we | 255 | * Returns 0 if the new certificate was accepted, -ENOKEY if we |
| 251 | * couldn't find a matching parent certificate in the trusted list, | 256 | * couldn't find a matching parent certificate in the trusted list, |
| 252 | * -EKEYREJECTED if the signature check fails, and some other error if | 257 | * -EKEYREJECTED if the signature check fails, -ENOPKG if the signature uses |
| 253 | * there is a matching certificate but the signature check cannot be | 258 | * unsupported crypto, or some other error if there is a matching certificate |
| 254 | * performed. | 259 | * but the signature check cannot be performed. |
| 255 | */ | 260 | */ |
| 256 | int restrict_link_by_key_or_keyring_chain(struct key *dest_keyring, | 261 | int restrict_link_by_key_or_keyring_chain(struct key *dest_keyring, |
| 257 | const struct key_type *type, | 262 | const struct key_type *type, |
| @@ -20,6 +20,20 @@ · 19 unmodified lines | |||
| 20 | #include <crypto/sha3.h> | 20 | #include <crypto/sha3.h> |
| 21 | #include <asm/unaligned.h> | 21 | #include <asm/unaligned.h> |
| 22 | 22 | ||
| 23 | /* | ||
| 24 | * On some 32-bit architectures (mn10300 and h8300), GCC ends up using | ||
| 25 | * over 1 KB of stack if we inline the round calculation into the loop | ||
| 26 | * in keccakf(). On the other hand, on 64-bit architectures with plenty | ||
| 27 | * of [64-bit wide] general purpose registers, not inlining it severely | ||
| 28 | * hurts performance. So let's use 64-bitness as a heuristic to decide | ||
| 29 | * whether to inline or not. | ||
| 30 | */ | ||
| 31 | #ifdef CONFIG_64BIT | ||
| 32 | #define SHA3_INLINE inline | ||
| 33 | #else | ||
| 34 | #define SHA3_INLINE noinline | ||
| 35 | #endif | ||
| 36 | |||
| 23 | #define KECCAK_ROUNDS 24 | 37 | #define KECCAK_ROUNDS 24 |
| 24 | 38 | ||
| 25 | static const u64 keccakf_rndc[24] = { | 39 | static const u64 keccakf_rndc[24] = { |
| @@ -35,111 +49,115 @@ static const u64 keccakf_rndc[24] = { · 9 unmodified lines | |||
| 35 | 49 | ||
| 36 | /* update the state with given number of rounds */ | 50 | /* update the state with given number of rounds */ |
| 37 | 51 | ||
| 38 | static void __attribute__((__optimize__("O3"))) keccakf(u64 st[25]) | 52 | static SHA3_INLINE void keccakf_round(u64 st[25]) |
| 39 | { | 53 | { |
| 40 | u64 t[5], tt, bc[5]; | 54 | u64 t[5], tt, bc[5]; |
| 41 | int round; | ||
| 42 | 55 | ||
| 43 | for (round = 0; round < KECCAK_ROUNDS; round++) { | 56 | /* Theta */ |
| 57 | bc[0] = st[0] ^ st[5] ^ st[10] ^ st[15] ^ st[20]; | ||
| 58 | bc[1] = st[1] ^ st[6] ^ st[11] ^ st[16] ^ st[21]; | ||
| 59 | bc[2] = st[2] ^ st[7] ^ st[12] ^ st[17] ^ st[22]; | ||
| 60 | bc[3] = st[3] ^ st[8] ^ st[13] ^ st[18] ^ st[23]; | ||
| 61 | bc[4] = st[4] ^ st[9] ^ st[14] ^ st[19] ^ st[24]; | ||
| 62 | |||
| 63 | t[0] = bc[4] ^ rol64(bc[1], 1); | ||
| 64 | t[1] = bc[0] ^ rol64(bc[2], 1); | ||
| 65 | t[2] = bc[1] ^ rol64(bc[3], 1); | ||
| 66 | t[3] = bc[2] ^ rol64(bc[4], 1); | ||
| 67 | t[4] = bc[3] ^ rol64(bc[0], 1); | ||
| 68 | |||
| 69 | st[0] ^= t[0]; | ||
| 70 | |||
| 71 | /* Rho Pi */ | ||
| 72 | tt = st[1]; | ||
| 73 | st[ 1] = rol64(st[ 6] ^ t[1], 44); | ||
| 74 | st[ 6] = rol64(st[ 9] ^ t[4], 20); | ||
| 75 | st[ 9] = rol64(st[22] ^ t[2], 61); | ||
| 76 | st[22] = rol64(st[14] ^ t[4], 39); | ||
| 77 | st[14] = rol64(st[20] ^ t[0], 18); | ||
| 78 | st[20] = rol64(st[ 2] ^ t[2], 62); | ||
| 79 | st[ 2] = rol64(st[12] ^ t[2], 43); | ||
| 80 | st[12] = rol64(st[13] ^ t[3], 25); | ||
| 81 | st[13] = rol64(st[19] ^ t[4], 8); | ||
| 82 | st[19] = rol64(st[23] ^ t[3], 56); | ||
| 83 | st[23] = rol64(st[15] ^ t[0], 41); | ||
| 84 | st[15] = rol64(st[ 4] ^ t[4], 27); | ||
| 85 | st[ 4] = rol64(st[24] ^ t[4], 14); | ||
| 86 | st[24] = rol64(st[21] ^ t[1], 2); | ||
| 87 | st[21] = rol64(st[ 8] ^ t[3], 55); | ||
| 88 | st[ 8] = rol64(st[16] ^ t[1], 45); | ||
| 89 | st[16] = rol64(st[ 5] ^ t[0], 36); | ||
| 90 | st[ 5] = rol64(st[ 3] ^ t[3], 28); | ||
| 91 | st[ 3] = rol64(st[18] ^ t[3], 21); | ||
| 92 | st[18] = rol64(st[17] ^ t[2], 15); | ||
| 93 | st[17] = rol64(st[11] ^ t[1], 10); | ||
| 94 | st[11] = rol64(st[ 7] ^ t[2], 6); | ||
| 95 | st[ 7] = rol64(st[10] ^ t[0], 3); | ||
| 96 | st[10] = rol64( tt ^ t[1], 1); | ||
| 97 | |||
| 98 | /* Chi */ | ||
| 99 | bc[ 0] = ~st[ 1] & st[ 2]; | ||
| 100 | bc[ 1] = ~st[ 2] & st[ 3]; | ||
| 101 | bc[ 2] = ~st[ 3] & st[ 4]; | ||
| 102 | bc[ 3] = ~st[ 4] & st[ 0]; | ||
| 103 | bc[ 4] = ~st[ 0] & st[ 1]; | ||
| 104 | st[ 0] ^= bc[ 0]; | ||
| 105 | st[ 1] ^= bc[ 1]; | ||
| 106 | st[ 2] ^= bc[ 2]; | ||
| 107 | st[ 3] ^= bc[ 3]; | ||
| 108 | st[ 4] ^= bc[ 4]; | ||
| 109 | |||
| 110 | bc[ 0] = ~st[ 6] & st[ 7]; | ||
| 111 | bc[ 1] = ~st[ 7] & st[ 8]; | ||
| 112 | bc[ 2] = ~st[ 8] & st[ 9]; | ||
| 113 | bc[ 3] = ~st[ 9] & st[ 5]; | ||
| 114 | bc[ 4] = ~st[ 5] & st[ 6]; | ||
| 115 | st[ 5] ^= bc[ 0]; | ||
| 116 | st[ 6] ^= bc[ 1]; | ||
| 117 | st[ 7] ^= bc[ 2]; | ||
| 118 | st[ 8] ^= bc[ 3]; | ||
| 119 | st[ 9] ^= bc[ 4]; | ||
| 120 | |||
| 121 | bc[ 0] = ~st[11] & st[12]; | ||
| 122 | bc[ 1] = ~st[12] & st[13]; | ||
| 123 | bc[ 2] = ~st[13] & st[14]; | ||
| 124 | bc[ 3] = ~st[14] & st[10]; | ||
| 125 | bc[ 4] = ~st[10] & st[11]; | ||
| 126 | st[10] ^= bc[ 0]; | ||
| 127 | st[11] ^= bc[ 1]; | ||
| 128 | st[12] ^= bc[ 2]; | ||
| 129 | st[13] ^= bc[ 3]; | ||
| 130 | st[14] ^= bc[ 4]; | ||
| 131 | |||
| 132 | bc[ 0] = ~st[16] & st[17]; | ||
| 133 | bc[ 1] = ~st[17] & st[18]; | ||
| 134 | bc[ 2] = ~st[18] & st[19]; | ||
| 135 | bc[ 3] = ~st[19] & st[15]; | ||
| 136 | bc[ 4] = ~st[15] & st[16]; | ||
| 137 | st[15] ^= bc[ 0]; | ||
| 138 | st[16] ^= bc[ 1]; | ||
| 139 | st[17] ^= bc[ 2]; | ||
| 140 | st[18] ^= bc[ 3]; | ||
| 141 | st[19] ^= bc[ 4]; | ||
| 142 | |||
| 143 | bc[ 0] = ~st[21] & st[22]; | ||
| 144 | bc[ 1] = ~st[22] & st[23]; | ||
| 145 | bc[ 2] = ~st[23] & st[24]; | ||
| 146 | bc[ 3] = ~st[24] & st[20]; | ||
| 147 | bc[ 4] = ~st[20] & st[21]; | ||
| 148 | st[20] ^= bc[ 0]; | ||
| 149 | st[21] ^= bc[ 1]; | ||
| 150 | st[22] ^= bc[ 2]; | ||
| 151 | st[23] ^= bc[ 3]; | ||
| 152 | st[24] ^= bc[ 4]; | ||
| 153 | } | ||
| 44 | 154 | ||
| 45 | /* Theta */ | 155 | static void __optimize("O3") keccakf(u64 st[25]) |
| 46 | bc[0] = st[0] ^ st[5] ^ st[10] ^ st[15] ^ st[20]; | 156 | { |
| 47 | bc[1] = st[1] ^ st[6] ^ st[11] ^ st[16] ^ st[21]; | 157 | int round; |
| 48 | bc[2] = st[2] ^ st[7] ^ st[12] ^ st[17] ^ st[22]; | ||
| 49 | bc[3] = st[3] ^ st[8] ^ st[13] ^ st[18] ^ st[23]; | ||
| 50 | bc[4] = st[4] ^ st[9] ^ st[14] ^ st[19] ^ st[24]; | ||
| 51 | |||
| 52 | t[0] = bc[4] ^ rol64(bc[1], 1); | ||
| 53 | t[1] = bc[0] ^ rol64(bc[2], 1); | ||
| 54 | t[2] = bc[1] ^ rol64(bc[3], 1); | ||
| 55 | t[3] = bc[2] ^ rol64(bc[4], 1); | ||
| 56 | t[4] = bc[3] ^ rol64(bc[0], 1); | ||
| 57 | |||
| 58 | st[0] ^= t[0]; | ||
| 59 | |||
| 60 | /* Rho Pi */ | ||
| 61 | tt = st[1]; | ||
| 62 | st[ 1] = rol64(st[ 6] ^ t[1], 44); | ||
| 63 | st[ 6] = rol64(st[ 9] ^ t[4], 20); | ||
| 64 | st[ 9] = rol64(st[22] ^ t[2], 61); | ||
| 65 | st[22] = rol64(st[14] ^ t[4], 39); | ||
| 66 | st[14] = rol64(st[20] ^ t[0], 18); | ||
| 67 | st[20] = rol64(st[ 2] ^ t[2], 62); | ||
| 68 | st[ 2] = rol64(st[12] ^ t[2], 43); | ||
| 69 | st[12] = rol64(st[13] ^ t[3], 25); | ||
| 70 | st[13] = rol64(st[19] ^ t[4], 8); | ||
| 71 | st[19] = rol64(st[23] ^ t[3], 56); | ||
| 72 | st[23] = rol64(st[15] ^ t[0], 41); | ||
| 73 | st[15] = rol64(st[ 4] ^ t[4], 27); | ||
| 74 | st[ 4] = rol64(st[24] ^ t[4], 14); | ||
| 75 | st[24] = rol64(st[21] ^ t[1], 2); | ||
| 76 | st[21] = rol64(st[ 8] ^ t[3], 55); | ||
| 77 | st[ 8] = rol64(st[16] ^ t[1], 45); | ||
| 78 | st[16] = rol64(st[ 5] ^ t[0], 36); | ||
| 79 | st[ 5] = rol64(st[ 3] ^ t[3], 28); | ||
| 80 | st[ 3] = rol64(st[18] ^ t[3], 21); | ||
| 81 | st[18] = rol64(st[17] ^ t[2], 15); | ||
| 82 | st[17] = rol64(st[11] ^ t[1], 10); | ||
| 83 | st[11] = rol64(st[ 7] ^ t[2], 6); | ||
| 84 | st[ 7] = rol64(st[10] ^ t[0], 3); | ||
| 85 | st[10] = rol64( tt ^ t[1], 1); | ||
| 86 | |||
| 87 | /* Chi */ | ||
| 88 | bc[ 0] = ~st[ 1] & st[ 2]; | ||
| 89 | bc[ 1] = ~st[ 2] & st[ 3]; | ||
| 90 | bc[ 2] = ~st[ 3] & st[ 4]; | ||
| 91 | bc[ 3] = ~st[ 4] & st[ 0]; | ||
| 92 | bc[ 4] = ~st[ 0] & st[ 1]; | ||
| 93 | st[ 0] ^= bc[ 0]; | ||
| 94 | st[ 1] ^= bc[ 1]; | ||
| 95 | st[ 2] ^= bc[ 2]; | ||
| 96 | st[ 3] ^= bc[ 3]; | ||
| 97 | st[ 4] ^= bc[ 4]; | ||
| 98 | |||
| 99 | bc[ 0] = ~st[ 6] & st[ 7]; | ||
| 100 | bc[ 1] = ~st[ 7] & st[ 8]; | ||
| 101 | bc[ 2] = ~st[ 8] & st[ 9]; | ||
| 102 | bc[ 3] = ~st[ 9] & st[ 5]; | ||
| 103 | bc[ 4] = ~st[ 5] & st[ 6]; | ||
| 104 | st[ 5] ^= bc[ 0]; | ||
| 105 | st[ 6] ^= bc[ 1]; | ||
| 106 | st[ 7] ^= bc[ 2]; | ||
| 107 | st[ 8] ^= bc[ 3]; | ||
| 108 | st[ 9] ^= bc[ 4]; | ||
| 109 | |||
| 110 | bc[ 0] = ~st[11] & st[12]; | ||
| 111 | bc[ 1] = ~st[12] & st[13]; | ||
| 112 | bc[ 2] = ~st[13] & st[14]; | ||
| 113 | bc[ 3] = ~st[14] & st[10]; | ||
| 114 | bc[ 4] = ~st[10] & st[11]; | ||
| 115 | st[10] ^= bc[ 0]; | ||
| 116 | st[11] ^= bc[ 1]; | ||
| 117 | st[12] ^= bc[ 2]; | ||
| 118 | st[13] ^= bc[ 3]; | ||
| 119 | st[14] ^= bc[ 4]; | ||
| 120 | |||
| 121 | bc[ 0] = ~st[16] & st[17]; | ||
| 122 | bc[ 1] = ~st[17] & st[18]; | ||
| 123 | bc[ 2] = ~st[18] & st[19]; | ||
| 124 | bc[ 3] = ~st[19] & st[15]; | ||
| 125 | bc[ 4] = ~st[15] & st[16]; | ||
| 126 | st[15] ^= bc[ 0]; | ||
| 127 | st[16] ^= bc[ 1]; | ||
| 128 | st[17] ^= bc[ 2]; | ||
| 129 | st[18] ^= bc[ 3]; | ||
| 130 | st[19] ^= bc[ 4]; | ||
| 131 | |||
| 132 | bc[ 0] = ~st[21] & st[22]; | ||
| 133 | bc[ 1] = ~st[22] & st[23]; | ||
| 134 | bc[ 2] = ~st[23] & st[24]; | ||
| 135 | bc[ 3] = ~st[24] & st[20]; | ||
| 136 | bc[ 4] = ~st[20] & st[21]; | ||
| 137 | st[20] ^= bc[ 0]; | ||
| 138 | st[21] ^= bc[ 1]; | ||
| 139 | st[22] ^= bc[ 2]; | ||
| 140 | st[23] ^= bc[ 3]; | ||
| 141 | st[24] ^= bc[ 4]; | ||
| 142 | 158 | ||
| 159 | for (round = 0; round < KECCAK_ROUNDS; round++) { | ||
| 160 | keccakf_round(st); | ||
| 143 | /* Iota */ | 161 | /* Iota */ |
| 144 | st[0] ^= keccakf_rndc[round]; | 162 | st[0] ^= keccakf_rndc[round]; |
| 145 | } | 163 | } |
| @@ -660,13 +660,15 @@ struct acpi_device *acpi_companion_match(const struct device *dev) · 659 unmodified lines | |||
| 660 | * acpi_of_match_device - Match device object using the "compatible" property. | 660 | * acpi_of_match_device - Match device object using the "compatible" property. |
| 661 | * @adev: ACPI device object to match. | 661 | * @adev: ACPI device object to match. |
| 662 | * @of_match_table: List of device IDs to match against. | 662 | * @of_match_table: List of device IDs to match against. |
| 663 | * @of_id: OF ID if matched | ||
| 663 | * | 664 | * |
| 664 | * If @dev has an ACPI companion which has ACPI_DT_NAMESPACE_HID in its list of | 665 | * If @dev has an ACPI companion which has ACPI_DT_NAMESPACE_HID in its list of |
| 665 | * identifiers and a _DSD object with the "compatible" property, use that | 666 | * identifiers and a _DSD object with the "compatible" property, use that |
| 666 | * property to match against the given list of identifiers. | 667 | * property to match against the given list of identifiers. |
| 667 | */ | 668 | */ |
| 668 | static bool acpi_of_match_device(struct acpi_device *adev, | 669 | static bool acpi_of_match_device(struct acpi_device *adev, |
| 669 | const struct of_device_id *of_match_table) | 670 | const struct of_device_id *of_match_table, |
| 671 | const struct of_device_id **of_id) | ||
| 670 | { | 672 | { |
| 671 | const union acpi_object *of_compatible, *obj; | 673 | const union acpi_object *of_compatible, *obj; |
| 672 | int i, nval; | 674 | int i, nval; |
| @@ -690,8 +692,11 @@ static bool acpi_of_match_device(struct acpi_device *adev, · 17 unmodified lines | |||
| 690 | const struct of_device_id *id; | 692 | const struct of_device_id *id; |
| 691 | 693 | ||
| 692 | for (id = of_match_table; id->compatible[0]; id++) | 694 | for (id = of_match_table; id->compatible[0]; id++) |
| 693 | if (!strcasecmp(obj->string.pointer, id->compatible)) | 695 | if (!strcasecmp(obj->string.pointer, id->compatible)) { |
| 696 | if (of_id) | ||
| 697 | *of_id = id; | ||
| 694 | return true; | 698 | return true; |
| 699 | } | ||
| 695 | } | 700 | } |
| 696 | 701 | ||
| 697 | return false; | 702 | return false; |
| @@ -762,10 +767,11 @@ static bool __acpi_match_device_cls(const struct acpi_device_id *id, · 64 unmodified lines | |||
| 762 | return true; | 767 | return true; |
| 763 | } | 768 | } |
| 764 | 769 | ||
| 765 | static const struct acpi_device_id *__acpi_match_device( | 770 | static bool __acpi_match_device(struct acpi_device *device, |
| 766 | struct acpi_device *device, | 771 | const struct acpi_device_id *acpi_ids, |
| 767 | const struct acpi_device_id *ids, | 772 | const struct of_device_id *of_ids, |
| 768 | const struct of_device_id *of_ids) | 773 | const struct acpi_device_id **acpi_id, |
| 774 | const struct of_device_id **of_id) | ||
| 769 | { | 775 | { |
| 770 | const struct acpi_device_id *id; | 776 | const struct acpi_device_id *id; |
| 771 | struct acpi_hardware_id *hwid; | 777 | struct acpi_hardware_id *hwid; |
| @@ -775,30 +781,32 @@ static const struct acpi_device_id *__acpi_match_device( · 3 unmodified lines | |||
| 775 | * driver for it. | 781 | * driver for it. |
| 776 | */ | 782 | */ |
| 777 | if (!device || !device->status.present) | 783 | if (!device || !device->status.present) |
| 778 | return NULL; | 784 | return false; |
| 779 | 785 | ||
| 780 | list_for_each_entry(hwid, &device->pnp.ids, list) { | 786 | list_for_each_entry(hwid, &device->pnp.ids, list) { |
| 781 | /* First, check the ACPI/PNP IDs provided by the caller. */ | 787 | /* First, check the ACPI/PNP IDs provided by the caller. */ |
| 782 | for (id = ids; id->id[0] || id->cls; id++) { | 788 | if (acpi_ids) { |
| 783 | if (id->id[0] && !strcmp((char *) id->id, hwid->id)) | 789 | for (id = acpi_ids; id->id[0] || id->cls; id++) { |
| 784 | return id; | 790 | if (id->id[0] && !strcmp((char *)id->id, hwid->id)) |
| 785 | else if (id->cls && __acpi_match_device_cls(id, hwid)) | 791 | goto out_acpi_match; |
| 786 | return id; | 792 | if (id->cls && __acpi_match_device_cls(id, hwid)) |
| 793 | goto out_acpi_match; | ||
| 794 | } | ||
| 787 | } | 795 | } |
| 788 | 796 | ||
| 789 | /* | 797 | /* |
| 790 | * Next, check ACPI_DT_NAMESPACE_HID and try to match the | 798 | * Next, check ACPI_DT_NAMESPACE_HID and try to match the |
| 791 | * "compatible" property if found. | 799 | * "compatible" property if found. |
| 792 | * | ||
| 793 | * The id returned by the below is not valid, but the only | ||
| 794 | * caller passing non-NULL of_ids here is only interested in | ||
| 795 | * whether or not the return value is NULL. | ||
| 796 | */ | 800 | */ |
| 797 | if (!strcmp(ACPI_DT_NAMESPACE_HID, hwid->id) | 801 | if (!strcmp(ACPI_DT_NAMESPACE_HID, hwid->id)) |
| 798 | && acpi_of_match_device(device, of_ids)) | 802 | return acpi_of_match_device(device, of_ids, of_id); |
| 799 | return id; | ||
| 800 | } | 803 | } |
| 801 | return NULL; | 804 | return false; |
| 805 | |||
| 806 | out_acpi_match: | ||
| 807 | if (acpi_id) | ||
| 808 | *acpi_id = id; | ||
| 809 | return true; | ||
| 802 | } | 810 | } |
| 803 | 811 | ||
| 804 | /** | 812 | /** |
| @@ -815,32 +823,29 @@ static const struct acpi_device_id *__acpi_match_device( · 10 unmodified lines | |||
| 815 | const struct acpi_device_id *acpi_match_device(const struct acpi_device_id *ids, | 823 | const struct acpi_device_id *acpi_match_device(const struct acpi_device_id *ids, |
| 816 | const struct device *dev) | 824 | const struct device *dev) |
| 817 | { | 825 | { |
| 818 | return __acpi_match_device(acpi_companion_match(dev), ids, NULL); | 826 | const struct acpi_device_id *id = NULL; |
| 827 | |||
| 828 | __acpi_match_device(acpi_companion_match(dev), ids, NULL, &id, NULL); | ||
| 829 | return id; | ||
| 819 | } | 830 | } |
| 820 | EXPORT_SYMBOL_GPL(acpi_match_device); | 831 | EXPORT_SYMBOL_GPL(acpi_match_device); |
| 821 | 832 | ||
| 822 | void *acpi_get_match_data(const struct device *dev) | 833 | const void *acpi_device_get_match_data(const struct device *dev) |
| 823 | { | 834 | { |
| 824 | const struct acpi_device_id *match; | 835 | const struct acpi_device_id *match; |
| 825 | 836 | ||
| 826 | if (!dev->driver) | ||
| 827 | return NULL; | ||
| 828 | |||
| 829 | if (!dev->driver->acpi_match_table) | ||
| 830 | return NULL; | ||
| 831 | |||
| 832 | match = acpi_match_device(dev->driver->acpi_match_table, dev); | 837 | match = acpi_match_device(dev->driver->acpi_match_table, dev); |
| 833 | if (!match) | 838 | if (!match) |
| 834 | return NULL; | 839 | return NULL; |
| 835 | 840 | ||
| 836 | return (void *)match->driver_data; | 841 | return (const void *)match->driver_data; |
| 837 | } | 842 | } |
| 838 | EXPORT_SYMBOL_GPL(acpi_get_match_data); | 843 | EXPORT_SYMBOL_GPL(acpi_device_get_match_data); |
| 839 | 844 | ||
| 840 | int acpi_match_device_ids(struct acpi_device *device, | 845 | int acpi_match_device_ids(struct acpi_device *device, |
| 841 | const struct acpi_device_id *ids) | 846 | const struct acpi_device_id *ids) |
| 842 | { | 847 | { |
| 843 | return __acpi_match_device(device, ids, NULL) ? 0 : -ENOENT; | 848 | return __acpi_match_device(device, ids, NULL, NULL, NULL) ? 0 : -ENOENT; |
| 844 | } | 849 | } |
| 845 | EXPORT_SYMBOL(acpi_match_device_ids); | 850 | EXPORT_SYMBOL(acpi_match_device_ids); |
| 846 | 851 | ||
| @@ -849,10 +854,12 @@ bool acpi_driver_match_device(struct device *dev, · 2 unmodified lines | |||
| 849 | { | 854 | { |
| 850 | if (!drv->acpi_match_table) | 855 | if (!drv->acpi_match_table) |
| 851 | return acpi_of_match_device(ACPI_COMPANION(dev), | 856 | return acpi_of_match_device(ACPI_COMPANION(dev), |
| 852 | drv->of_match_table); | 857 | drv->of_match_table, |
| 858 | NULL); | ||
| 853 | 859 | ||
| 854 | return !!__acpi_match_device(acpi_companion_match(dev), | 860 | return __acpi_match_device(acpi_companion_match(dev), |
| 855 | drv->acpi_match_table, drv->of_match_table); | 861 | drv->acpi_match_table, drv->of_match_table, |
| 862 | NULL, NULL); | ||
| 856 | } | 863 | } |
| 857 | EXPORT_SYMBOL_GPL(acpi_driver_match_device); | 864 | EXPORT_SYMBOL_GPL(acpi_driver_match_device); |
| 858 | 865 | ||
| @@ -1927,6 +1927,9 @@ static int acpi_ec_suspend_noirq(struct device *dev) · 1926 unmodified lines | |||
| 1927 | ec->reference_count >= 1) | 1927 | ec->reference_count >= 1) |
| 1928 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_DISABLE); | 1928 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_DISABLE); |
| 1929 | 1929 | ||
| 1930 | if (acpi_sleep_no_ec_events()) | ||
| 1931 | acpi_ec_enter_noirq(ec); | ||
| 1932 | |||
| 1930 | return 0; | 1933 | return 0; |
| 1931 | } | 1934 | } |
| 1932 | 1935 | ||
| @@ -1934,6 +1937,9 @@ static int acpi_ec_resume_noirq(struct device *dev) · 1 unmodified line | |||
| 1934 | { | 1937 | { |
| 1935 | struct acpi_ec *ec = acpi_driver_data(to_acpi_device(dev)); | 1938 | struct acpi_ec *ec = acpi_driver_data(to_acpi_device(dev)); |
| 1936 | 1939 | ||
| 1940 | if (acpi_sleep_no_ec_events()) | ||
| 1941 | acpi_ec_leave_noirq(ec); | ||
| 1942 | |||
| 1937 | if (ec_no_wakeup && test_bit(EC_FLAGS_STARTED, &ec->flags) && | 1943 | if (ec_no_wakeup && test_bit(EC_FLAGS_STARTED, &ec->flags) && |
| 1938 | ec->reference_count >= 1) | 1944 | ec->reference_count >= 1) |
| 1939 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_ENABLE); | 1945 | acpi_set_gpe(NULL, ec->gpe, ACPI_GPE_ENABLE); |
| @@ -1271,11 +1271,11 @@ static int acpi_fwnode_graph_parse_endpoint(const struct fwnode_handle *fwnode, · 1270 unmodified lines | |||
| 1271 | return 0; | 1271 | return 0; |
| 1272 | } | 1272 | } |
| 1273 | 1273 | ||
| 1274 | static void * | 1274 | static const void * |
| 1275 | acpi_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, | 1275 | acpi_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, |
| 1276 | const struct device *dev) | 1276 | const struct device *dev) |
| 1277 | { | 1277 | { |
| 1278 | return acpi_get_match_data(dev); | 1278 | return acpi_device_get_match_data(dev); |
| 1279 | } | 1279 | } |
| 1280 | 1280 | ||
| 1281 | #define DECLARE_ACPI_FWNODE_OPS(ops) \ | 1281 | #define DECLARE_ACPI_FWNODE_OPS(ops) \ |
| @@ -115,6 +115,7 @@ int __init acpi_parse_spcr(bool enable_earlycon, bool enable_console) · 114 unmodified lines | |||
| 115 | table->serial_port.access_width))) { | 115 | table->serial_port.access_width))) { |
| 116 | default: | 116 | default: |
| 117 | pr_err("Unexpected SPCR Access Width. Defaulting to byte size\n"); | 117 | pr_err("Unexpected SPCR Access Width. Defaulting to byte size\n"); |
| 118 | /* fall through */ | ||
| 118 | case 8: | 119 | case 8: |
| 119 | iotype = "mmio"; | 120 | iotype = "mmio"; |
| 120 | break; | 121 | break; |
| @@ -1991,8 +1991,14 @@ static void binder_send_failed_reply(struct binder_transaction *t, · 1990 unmodified lines | |||
| 1991 | &target_thread->reply_error.work); | 1991 | &target_thread->reply_error.work); |
| 1992 | wake_up_interruptible(&target_thread->wait); | 1992 | wake_up_interruptible(&target_thread->wait); |
| 1993 | } else { | 1993 | } else { |
| 1994 | WARN(1, "Unexpected reply error: %u\n", | 1994 | /* |
| 1995 | target_thread->reply_error.cmd); | 1995 | * Cannot get here for normal operation, but |
| 1996 | * we can if multiple synchronous transactions | ||
| 1997 | * are sent without blocking for responses. | ||
| 1998 | * Just ignore the 2nd error in this case. | ||
| 1999 | */ | ||
| 2000 | pr_warn("Unexpected reply error: %u\n", | ||
| 2001 | target_thread->reply_error.cmd); | ||
| 1996 | } | 2002 | } |
| 1997 | binder_inner_proc_unlock(target_thread->proc); | 2003 | binder_inner_proc_unlock(target_thread->proc); |
| 1998 | binder_thread_dec_tmpref(target_thread); | 2004 | binder_thread_dec_tmpref(target_thread); |
| @@ -2193,7 +2199,7 @@ static void binder_transaction_buffer_release(struct binder_proc *proc, · 194 unmodified lines | |||
| 2193 | int debug_id = buffer->debug_id; | 2199 | int debug_id = buffer->debug_id; |
| 2194 | 2200 | ||
| 2195 | binder_debug(BINDER_DEBUG_TRANSACTION, | 2201 | binder_debug(BINDER_DEBUG_TRANSACTION, |
| 2196 | "%d buffer release %d, size %zd-%zd, failed at %p\n", | 2202 | "%d buffer release %d, size %zd-%zd, failed at %pK\n", |
| 2197 | proc->pid, buffer->debug_id, | 2203 | proc->pid, buffer->debug_id, |
| 2198 | buffer->data_size, buffer->offsets_size, failed_at); | 2204 | buffer->data_size, buffer->offsets_size, failed_at); |
| 2199 | 2205 | ||
| @@ -3705,7 +3711,7 @@ static int binder_thread_write(struct binder_proc *proc, · 1505 unmodified lines | |||
| 3705 | } | 3711 | } |
| 3706 | } | 3712 | } |
| 3707 | binder_debug(BINDER_DEBUG_DEAD_BINDER, | 3713 | binder_debug(BINDER_DEBUG_DEAD_BINDER, |
| 3708 | "%d:%d BC_DEAD_BINDER_DONE %016llx found %p\n", | 3714 | "%d:%d BC_DEAD_BINDER_DONE %016llx found %pK\n", |
| 3709 | proc->pid, thread->pid, (u64)cookie, | 3715 | proc->pid, thread->pid, (u64)cookie, |
| 3710 | death); | 3716 | death); |
| 3711 | if (death == NULL) { | 3717 | if (death == NULL) { |
| @@ -4376,6 +4382,15 @@ static int binder_thread_release(struct binder_proc *proc, · 664 unmodified lines | |||
| 4376 | 4382 | ||
| 4377 | binder_inner_proc_unlock(thread->proc); | 4383 | binder_inner_proc_unlock(thread->proc); |
| 4378 | 4384 | ||
| 4385 | /* | ||
| 4386 | * This is needed to avoid races between wake_up_poll() above and | ||
| 4387 | * and ep_remove_waitqueue() called for other reasons (eg the epoll file | ||
| 4388 | * descriptor being closed); ep_remove_waitqueue() holds an RCU read | ||
| 4389 | * lock, so we can be sure it's done after calling synchronize_rcu(). | ||
| 4390 | */ | ||
| 4391 | if (thread->looper & BINDER_LOOPER_STATE_POLL) | ||
| 4392 | synchronize_rcu(); | ||
| 4393 | |||
| 4379 | if (send_reply) | 4394 | if (send_reply) |
| 4380 | binder_send_failed_reply(send_reply, BR_DEAD_REPLY); | 4395 | binder_send_failed_reply(send_reply, BR_DEAD_REPLY); |
| 4381 | binder_release_work(proc, &thread->todo); | 4396 | binder_release_work(proc, &thread->todo); |
| @@ -4391,6 +4406,8 @@ static __poll_t binder_poll(struct file *filp, · 9 unmodified lines | |||
| 4391 | bool wait_for_proc_work; | 4406 | bool wait_for_proc_work; |
| 4392 | 4407 | ||
| 4393 | thread = binder_get_thread(proc); | 4408 | thread = binder_get_thread(proc); |
| 4409 | if (!thread) | ||
| 4410 | return POLLERR; | ||
| 4394 | 4411 | ||
| 4395 | binder_inner_proc_lock(thread->proc); | 4412 | binder_inner_proc_lock(thread->proc); |
| 4396 | thread->looper |= BINDER_LOOPER_STATE_POLL; | 4413 | thread->looper |= BINDER_LOOPER_STATE_POLL; |
| @@ -5034,7 +5051,7 @@ static void print_binder_transaction_ilocked(struct seq_file *m, · 637 unmodified lines | |||
| 5034 | spin_lock(&t->lock); | 5051 | spin_lock(&t->lock); |
| 5035 | to_proc = t->to_proc; | 5052 | to_proc = t->to_proc; |
| 5036 | seq_printf(m, | 5053 | seq_printf(m, |
| 5037 | "%s %d: %p from %d:%d to %d:%d code %x flags %x pri %ld r%d", | 5054 | "%s %d: %pK from %d:%d to %d:%d code %x flags %x pri %ld r%d", |
| 5038 | prefix, t->debug_id, t, | 5055 | prefix, t->debug_id, t, |
| 5039 | t->from ? t->from->proc->pid : 0, | 5056 | t->from ? t->from->proc->pid : 0, |
| 5040 | t->from ? t->from->pid : 0, | 5057 | t->from ? t->from->pid : 0, |
| @@ -5058,7 +5075,7 @@ static void print_binder_transaction_ilocked(struct seq_file *m, · 17 unmodified lines | |||
| 5058 | } | 5075 | } |
| 5059 | if (buffer->target_node) | 5076 | if (buffer->target_node) |
| 5060 | seq_printf(m, " node %d", buffer->target_node->debug_id); | 5077 | seq_printf(m, " node %d", buffer->target_node->debug_id); |
| 5061 | seq_printf(m, " size %zd:%zd data %p\n", | 5078 | seq_printf(m, " size %zd:%zd data %pK\n", |
| 5062 | buffer->data_size, buffer->offsets_size, | 5079 | buffer->data_size, buffer->offsets_size, |
| 5063 | buffer->data); | 5080 | buffer->data); |
| 5064 | } | 5081 | } |
| @@ -310,6 +310,9 @@ static void __device_link_del(struct device_link *link) · 309 unmodified lines | |||
| 310 | dev_info(link->consumer, "Dropping the link to %s\n", | 310 | dev_info(link->consumer, "Dropping the link to %s\n", |
| 311 | dev_name(link->supplier)); | 311 | dev_name(link->supplier)); |
| 312 | 312 | ||
| 313 | if (link->flags & DL_FLAG_PM_RUNTIME) | ||
| 314 | pm_runtime_drop_link(link->consumer); | ||
| 315 | |||
| 313 | list_del(&link->s_node); | 316 | list_del(&link->s_node); |
| 314 | list_del(&link->c_node); | 317 | list_del(&link->c_node); |
| 315 | device_link_free(link); | 318 | device_link_free(link); |
| @@ -321,7 +321,8 @@ void dev_pm_arm_wake_irq(struct wake_irq *wirq) · 320 unmodified lines | |||
| 321 | return; | 321 | return; |
| 322 | 322 | ||
| 323 | if (device_may_wakeup(wirq->dev)) { | 323 | if (device_may_wakeup(wirq->dev)) { |
| 324 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED) | 324 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED && |
| 325 | !pm_runtime_status_suspended(wirq->dev)) | ||
| 325 | enable_irq(wirq->irq); | 326 | enable_irq(wirq->irq); |
| 326 | 327 | ||
| 327 | enable_irq_wake(wirq->irq); | 328 | enable_irq_wake(wirq->irq); |
| @@ -343,7 +344,8 @@ void dev_pm_disarm_wake_irq(struct wake_irq *wirq) · 15 unmodified lines | |||
| 343 | if (device_may_wakeup(wirq->dev)) { | 344 | if (device_may_wakeup(wirq->dev)) { |
| 344 | disable_irq_wake(wirq->irq); | 345 | disable_irq_wake(wirq->irq); |
| 345 | 346 | ||
| 346 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED) | 347 | if (wirq->status & WAKE_IRQ_DEDICATED_ALLOCATED && |
| 348 | !pm_runtime_status_suspended(wirq->dev)) | ||
| 347 | disable_irq_nosync(wirq->irq); | 349 | disable_irq_nosync(wirq->irq); |
| 348 | } | 350 | } |
| 349 | } | 351 | } |
| @@ -1410,9 +1410,8 @@ int fwnode_graph_parse_endpoint(const struct fwnode_handle *fwnode, · 1409 unmodified lines | |||
| 1410 | } | 1410 | } |
| 1411 | EXPORT_SYMBOL(fwnode_graph_parse_endpoint); | 1411 | EXPORT_SYMBOL(fwnode_graph_parse_endpoint); |
| 1412 | 1412 | ||
| 1413 | void *device_get_match_data(struct device *dev) | 1413 | const void *device_get_match_data(struct device *dev) |
| 1414 | { | 1414 | { |
| 1415 | return fwnode_call_ptr_op(dev_fwnode(dev), device_get_match_data, | 1415 | return fwnode_call_ptr_op(dev_fwnode(dev), device_get_match_data, dev); |
| 1416 | dev); | ||
| 1417 | } | 1416 | } |
| 1418 | EXPORT_SYMBOL_GPL(device_get_match_data); | 1417 | EXPORT_SYMBOL_GPL(device_get_match_data); |
| @@ -1758,7 +1758,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 1757 unmodified lines | |||
| 1758 | if (unit[drive].type->code == FD_NODRIVE) | 1758 | if (unit[drive].type->code == FD_NODRIVE) |
| 1759 | return NULL; | 1759 | return NULL; |
| 1760 | *part = 0; | 1760 | *part = 0; |
| 1761 | return get_disk(unit[drive].gendisk); | 1761 | return get_disk_and_module(unit[drive].gendisk); |
| 1762 | } | 1762 | } |
| 1763 | 1763 | ||
| 1764 | static int __init amiga_floppy_probe(struct platform_device *pdev) | 1764 | static int __init amiga_floppy_probe(struct platform_device *pdev) |
| @@ -1917,7 +1917,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 1916 unmodified lines | |||
| 1917 | if (drive >= FD_MAX_UNITS || type > NUM_DISK_MINORS) | 1917 | if (drive >= FD_MAX_UNITS || type > NUM_DISK_MINORS) |
| 1918 | return NULL; | 1918 | return NULL; |
| 1919 | *part = 0; | 1919 | *part = 0; |
| 1920 | return get_disk(unit[drive].disk); | 1920 | return get_disk_and_module(unit[drive].disk); |
| 1921 | } | 1921 | } |
| 1922 | 1922 | ||
| 1923 | static int __init atari_floppy_init (void) | 1923 | static int __init atari_floppy_init (void) |
| @@ -456,7 +456,7 @@ static struct kobject *brd_probe(dev_t dev, int *part, void *data) · 455 unmodified lines | |||
| 456 | 456 | ||
| 457 | mutex_lock(&brd_devices_mutex); | 457 | mutex_lock(&brd_devices_mutex); |
| 458 | brd = brd_init_one(MINOR(dev) / max_part, &new); | 458 | brd = brd_init_one(MINOR(dev) / max_part, &new); |
| 459 | kobj = brd ? get_disk(brd->brd_disk) : NULL; | 459 | kobj = brd ? get_disk_and_module(brd->brd_disk) : NULL; |
| 460 | mutex_unlock(&brd_devices_mutex); | 460 | mutex_unlock(&brd_devices_mutex); |
| 461 | 461 | ||
| 462 | if (new) | 462 | if (new) |
| @@ -4505,7 +4505,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 4504 unmodified lines | |||
| 4505 | if (((*part >> 2) & 0x1f) >= ARRAY_SIZE(floppy_type)) | 4505 | if (((*part >> 2) & 0x1f) >= ARRAY_SIZE(floppy_type)) |
| 4506 | return NULL; | 4506 | return NULL; |
| 4507 | *part = 0; | 4507 | *part = 0; |
| 4508 | return get_disk(disks[drive]); | 4508 | return get_disk_and_module(disks[drive]); |
| 4509 | } | 4509 | } |
| 4510 | 4510 | ||
| 4511 | static int __init do_floppy_init(void) | 4511 | static int __init do_floppy_init(void) |
| @@ -266,7 +266,7 @@ static int lo_write_bvec(struct file *file, struct bio_vec *bvec, loff_t *ppos) · 265 unmodified lines | |||
| 266 | struct iov_iter i; | 266 | struct iov_iter i; |
| 267 | ssize_t bw; | 267 | ssize_t bw; |
| 268 | 268 | ||
| 269 | iov_iter_bvec(&i, ITER_BVEC, bvec, 1, bvec->bv_len); | 269 | iov_iter_bvec(&i, ITER_BVEC | WRITE, bvec, 1, bvec->bv_len); |
| 270 | 270 | ||
| 271 | file_start_write(file); | 271 | file_start_write(file); |
| 272 | bw = vfs_iter_write(file, &i, ppos, 0); | 272 | bw = vfs_iter_write(file, &i, ppos, 0); |
| @@ -1922,7 +1922,7 @@ static struct kobject *loop_probe(dev_t dev, int *part, void *data) · 1649 unmodified lines | |||
| 1922 | if (err < 0) | 1922 | if (err < 0) |
| 1923 | kobj = NULL; | 1923 | kobj = NULL; |
| 1924 | else | 1924 | else |
| 1925 | kobj = get_disk(lo->lo_disk); | 1925 | kobj = get_disk_and_module(lo->lo_disk); |
| 1926 | mutex_unlock(&loop_index_mutex); | 1926 | mutex_unlock(&loop_index_mutex); |
| 1927 | 1927 | ||
| 1928 | *part = 0; | 1928 | *part = 0; |
| @@ -1591,7 +1591,7 @@ again: · 1590 unmodified lines | |||
| 1591 | if (new_index < 0) { | 1591 | if (new_index < 0) { |
| 1592 | mutex_unlock(&nbd_index_mutex); | 1592 | mutex_unlock(&nbd_index_mutex); |
| 1593 | printk(KERN_ERR "nbd: failed to add new device\n"); | 1593 | printk(KERN_ERR "nbd: failed to add new device\n"); |
| 1594 | return ret; | 1594 | return new_index; |
| 1595 | } | 1595 | } |
| 1596 | nbd = idr_find(&nbd_index_idr, new_index); | 1596 | nbd = idr_find(&nbd_index_idr, new_index); |
| 1597 | } | 1597 | } |
| @@ -1122,7 +1122,7 @@ static int pkt_start_recovery(struct packet_data *pkt) · 1121 unmodified lines | |||
| 1122 | pkt->sector = new_sector; | 1122 | pkt->sector = new_sector; |
| 1123 | 1123 | ||
| 1124 | bio_reset(pkt->bio); | 1124 | bio_reset(pkt->bio); |
| 1125 | bio_set_set(pkt->bio, pd->bdev); | 1125 | bio_set_dev(pkt->bio, pd->bdev); |
| 1126 | bio_set_op_attrs(pkt->bio, REQ_OP_WRITE, 0); | 1126 | bio_set_op_attrs(pkt->bio, REQ_OP_WRITE, 0); |
| 1127 | pkt->bio->bi_iter.bi_sector = new_sector; | 1127 | pkt->bio->bi_iter.bi_sector = new_sector; |
| 1128 | pkt->bio->bi_iter.bi_size = pkt->frames * CD_FRAMESIZE; | 1128 | pkt->bio->bi_iter.bi_size = pkt->frames * CD_FRAMESIZE; |
| @@ -799,7 +799,7 @@ static struct kobject *floppy_find(dev_t dev, int *part, void *data) · 798 unmodified lines | |||
| 799 | return NULL; | 799 | return NULL; |
| 800 | 800 | ||
| 801 | *part = 0; | 801 | *part = 0; |
| 802 | return get_disk(swd->unit[drive].disk); | 802 | return get_disk_and_module(swd->unit[drive].disk); |
| 803 | } | 803 | } |
| 804 | 804 | ||
| 805 | static int swim_add_floppy(struct swim_priv *swd, enum drive_location location) | 805 | static int swim_add_floppy(struct swim_priv *swd, enum drive_location location) |
| @@ -262,6 +262,7 @@ static DEFINE_SPINLOCK(minor_lock); · 261 unmodified lines | |||
| 262 | 262 | ||
| 263 | static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo); | 263 | static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo); |
| 264 | static void blkfront_gather_backend_features(struct blkfront_info *info); | 264 | static void blkfront_gather_backend_features(struct blkfront_info *info); |
| 265 | static int negotiate_mq(struct blkfront_info *info); | ||
| 265 | 266 | ||
| 266 | static int get_id_from_freelist(struct blkfront_ring_info *rinfo) | 267 | static int get_id_from_freelist(struct blkfront_ring_info *rinfo) |
| 267 | { | 268 | { |
| @@ -1774,11 +1775,18 @@ static int talk_to_blkback(struct xenbus_device *dev, · 1506 unmodified lines | |||
| 1774 | unsigned int i, max_page_order; | 1775 | unsigned int i, max_page_order; |
| 1775 | unsigned int ring_page_order; | 1776 | unsigned int ring_page_order; |
| 1776 | 1777 | ||
| 1778 | if (!info) | ||
| 1779 | return -ENODEV; | ||
| 1780 | |||
| 1777 | max_page_order = xenbus_read_unsigned(info->xbdev->otherend, | 1781 | max_page_order = xenbus_read_unsigned(info->xbdev->otherend, |
| 1778 | "max-ring-page-order", 0); | 1782 | "max-ring-page-order", 0); |
| 1779 | ring_page_order = min(xen_blkif_max_ring_order, max_page_order); | 1783 | ring_page_order = min(xen_blkif_max_ring_order, max_page_order); |
| 1780 | info->nr_ring_pages = 1 << ring_page_order; | 1784 | info->nr_ring_pages = 1 << ring_page_order; |
| 1781 | 1785 | ||
| 1786 | err = negotiate_mq(info); | ||
| 1787 | if (err) | ||
| 1788 | goto destroy_blkring; | ||
| 1789 | |||
| 1782 | for (i = 0; i < info->nr_rings; i++) { | 1790 | for (i = 0; i < info->nr_rings; i++) { |
| 1783 | struct blkfront_ring_info *rinfo = &info->rinfo[i]; | 1791 | struct blkfront_ring_info *rinfo = &info->rinfo[i]; |
| 1784 | 1792 | ||
| @@ -1978,11 +1986,6 @@ static int blkfront_probe(struct xenbus_device *dev, · 193 unmodified lines | |||
| 1978 | } | 1986 | } |
| 1979 | 1987 | ||
| 1980 | info->xbdev = dev; | 1988 | info->xbdev = dev; |
| 1981 | err = negotiate_mq(info); | ||
| 1982 | if (err) { | ||
| 1983 | kfree(info); | ||
| 1984 | return err; | ||
| 1985 | } | ||
| 1986 | 1989 | ||
| 1987 | mutex_init(&info->mutex); | 1990 | mutex_init(&info->mutex); |
| 1988 | info->vdevice = vdevice; | 1991 | info->vdevice = vdevice; |
| @@ -2099,10 +2102,6 @@ static int blkfront_resume(struct xenbus_device *dev) · 110 unmodified lines | |||
| 2099 | 2102 | ||
| 2100 | blkif_free(info, info->connected == BLKIF_STATE_CONNECTED); | 2103 | blkif_free(info, info->connected == BLKIF_STATE_CONNECTED); |
| 2101 | 2104 | ||
| 2102 | err = negotiate_mq(info); | ||
| 2103 | if (err) | ||
| 2104 | return err; | ||
| 2105 | |||
| 2106 | err = talk_to_blkback(dev, info); | 2105 | err = talk_to_blkback(dev, info); |
| 2107 | if (!err) | 2106 | if (!err) |
| 2108 | blk_mq_update_nr_hw_queues(&info->tag_set, info->nr_rings); | 2107 | blk_mq_update_nr_hw_queues(&info->tag_set, info->nr_rings); |
| @@ -332,7 +332,7 @@ static const struct block_device_operations z2_fops = · 331 unmodified lines | |||
| 332 | static struct kobject *z2_find(dev_t dev, int *part, void *data) | 332 | static struct kobject *z2_find(dev_t dev, int *part, void *data) |
| 333 | { | 333 | { |
| 334 | *part = 0; | 334 | *part = 0; |
| 335 | return get_disk(z2ram_gendisk); | 335 | return get_disk_and_module(z2ram_gendisk); |
| 336 | } | 336 | } |
| 337 | 337 | ||
| 338 | static struct request_queue *z2_queue; | 338 | static struct request_queue *z2_queue; |
| @@ -21,6 +21,7 @@ · 20 unmodified lines | |||
| 21 | * | 21 | * |
| 22 | */ | 22 | */ |
| 23 | 23 | ||
| 24 | #include <linux/dmi.h> | ||
| 24 | #include <linux/module.h> | 25 | #include <linux/module.h> |
| 25 | #include <linux/usb.h> | 26 | #include <linux/usb.h> |
| 26 | #include <linux/usb/quirks.h> | 27 | #include <linux/usb/quirks.h> |
| @@ -379,6 +380,21 @@ static const struct usb_device_id blacklist_table[] = { · 352 unmodified lines | |||
| 379 | { } /* Terminating entry */ | 380 | { } /* Terminating entry */ |
| 380 | }; | 381 | }; |
| 381 | 382 | ||
| 383 | /* The Bluetooth USB module build into some devices needs to be reset on resume, | ||
| 384 | * this is a problem with the platform (likely shutting off all power) not with | ||
| 385 | * the module itself. So we use a DMI list to match known broken platforms. | ||
| 386 | */ | ||
| 387 | static const struct dmi_system_id btusb_needs_reset_resume_table[] = { | ||
| 388 | { | ||
| 389 | /* Lenovo Yoga 920 (QCA Rome device 0cf3:e300) */ | ||
| 390 | .matches = { | ||
| 391 | DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), | ||
| 392 | DMI_MATCH(DMI_PRODUCT_VERSION, "Lenovo YOGA 920"), | ||
| 393 | }, | ||
| 394 | }, | ||
| 395 | {} | ||
| 396 | }; | ||
| 397 | |||
| 382 | #define BTUSB_MAX_ISOC_FRAMES 10 | 398 | #define BTUSB_MAX_ISOC_FRAMES 10 |
| 383 | 399 | ||
| 384 | #define BTUSB_INTR_RUNNING 0 | 400 | #define BTUSB_INTR_RUNNING 0 |
| @@ -2945,6 +2961,9 @@ static int btusb_probe(struct usb_interface *intf, · 2560 unmodified lines | |||
| 2945 | hdev->send = btusb_send_frame; | 2961 | hdev->send = btusb_send_frame; |
| 2946 | hdev->notify = btusb_notify; | 2962 | hdev->notify = btusb_notify; |
| 2947 | 2963 | ||
| 2964 | if (dmi_check_system(btusb_needs_reset_resume_table)) | ||
| 2965 | interface_to_usbdev(intf)->quirks |= USB_QUIRK_RESET_RESUME; | ||
| 2966 | |||
| 2948 | #ifdef CONFIG_PM | 2967 | #ifdef CONFIG_PM |
| 2949 | err = btusb_config_oob_wake(hdev); | 2968 | err = btusb_config_oob_wake(hdev); |
| 2950 | if (err) | 2969 | if (err) |
| @@ -3031,12 +3050,6 @@ static int btusb_probe(struct usb_interface *intf, · 80 unmodified lines | |||
| 3031 | if (id->driver_info & BTUSB_QCA_ROME) { | 3050 | if (id->driver_info & BTUSB_QCA_ROME) { |
| 3032 | data->setup_on_usb = btusb_setup_qca; | 3051 | data->setup_on_usb = btusb_setup_qca; |
| 3033 | hdev->set_bdaddr = btusb_set_bdaddr_ath3012; | 3052 | hdev->set_bdaddr = btusb_set_bdaddr_ath3012; |
| 3034 | |||
| 3035 | /* QCA Rome devices lose their updated firmware over suspend, | ||
| 3036 | * but the USB hub doesn't notice any status change. | ||
| 3037 | * explicitly request a device reset on resume. | ||
| 3038 | */ | ||
| 3039 | interface_to_usbdev(intf)->quirks |= USB_QUIRK_RESET_RESUME; | ||
| 3040 | } | 3053 | } |
| 3041 | 3054 | ||
| 3042 | #ifdef CONFIG_BT_HCIBTUSB_RTL | 3055 | #ifdef CONFIG_BT_HCIBTUSB_RTL |
| @@ -922,12 +922,13 @@ static int bcm_get_resources(struct bcm_device *dev) · 921 unmodified lines | |||
| 922 | 922 | ||
| 923 | dev->clk = devm_clk_get(dev->dev, NULL); | 923 | dev->clk = devm_clk_get(dev->dev, NULL); |
| 924 | 924 | ||
| 925 | dev->device_wakeup = devm_gpiod_get(dev->dev, "device-wakeup", | 925 | dev->device_wakeup = devm_gpiod_get_optional(dev->dev, "device-wakeup", |
| 926 | GPIOD_OUT_LOW); | 926 | GPIOD_OUT_LOW); |
| 927 | if (IS_ERR(dev->device_wakeup)) | 927 | if (IS_ERR(dev->device_wakeup)) |
| 928 | return PTR_ERR(dev->device_wakeup); | 928 | return PTR_ERR(dev->device_wakeup); |
| 929 | 929 | ||
| 930 | dev->shutdown = devm_gpiod_get(dev->dev, "shutdown", GPIOD_OUT_LOW); | 930 | dev->shutdown = devm_gpiod_get_optional(dev->dev, "shutdown", |
| 931 | GPIOD_OUT_LOW); | ||
| 931 | if (IS_ERR(dev->shutdown)) | 932 | if (IS_ERR(dev->shutdown)) |
| 932 | return PTR_ERR(dev->shutdown); | 933 | return PTR_ERR(dev->shutdown); |
| 933 | 934 | ||
| @@ -630,7 +630,7 @@ static int sysc_init_dts_quirks(struct sysc *ddata) · 629 unmodified lines | |||
| 630 | for (i = 0; i < ARRAY_SIZE(sysc_dts_quirks); i++) { | 630 | for (i = 0; i < ARRAY_SIZE(sysc_dts_quirks); i++) { |
| 631 | prop = of_get_property(np, sysc_dts_quirks[i].name, &len); | 631 | prop = of_get_property(np, sysc_dts_quirks[i].name, &len); |
| 632 | if (!prop) | 632 | if (!prop) |
| 633 | break; | 633 | continue; |
| 634 | 634 | ||
| 635 | ddata->cfg.quirks |= sysc_dts_quirks[i].mask; | 635 | ddata->cfg.quirks |= sysc_dts_quirks[i].mask; |
| 636 | } | 636 | } |
| @@ -162,7 +162,7 @@ static int via_rng_init(struct hwrng *rng) · 161 unmodified lines | |||
| 162 | /* Enable secondary noise source on CPUs where it is present. */ | 162 | /* Enable secondary noise source on CPUs where it is present. */ |
| 163 | 163 | ||
| 164 | /* Nehemiah stepping 8 and higher */ | 164 | /* Nehemiah stepping 8 and higher */ |
| 165 | if ((c->x86_model == 9) && (c->x86_mask > 7)) | 165 | if ((c->x86_model == 9) && (c->x86_stepping > 7)) |
| 166 | lo |= VIA_NOISESRC2; | 166 | lo |= VIA_NOISESRC2; |
| 167 | 167 | ||
| 168 | /* Esther */ | 168 | /* Esther */ |
| @@ -457,7 +457,7 @@ static int st33zp24_recv(struct tpm_chip *chip, unsigned char *buf, · 456 unmodified lines | |||
| 457 | size_t count) | 457 | size_t count) |
| 458 | { | 458 | { |
| 459 | int size = 0; | 459 | int size = 0; |
| 460 | int expected; | 460 | u32 expected; |
| 461 | 461 | ||
| 462 | if (!chip) | 462 | if (!chip) |
| 463 | return -EBUSY; | 463 | return -EBUSY; |
| @@ -474,7 +474,7 @@ static int st33zp24_recv(struct tpm_chip *chip, unsigned char *buf, · 10 unmodified lines | |||
| 474 | } | 474 | } |
| 475 | 475 | ||
| 476 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); | 476 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); |
| 477 | if (expected > count) { | 477 | if (expected > count || expected < TPM_HEADER_SIZE) { |
| 478 | size = -EIO; | 478 | size = -EIO; |
| 479 | goto out; | 479 | goto out; |
| 480 | } | 480 | } |
| @@ -1190,6 +1190,10 @@ int tpm_get_random(struct tpm_chip *chip, u8 *out, size_t max) · 1189 unmodified lines | |||
| 1190 | break; | 1190 | break; |
| 1191 | 1191 | ||
| 1192 | recd = be32_to_cpu(tpm_cmd.params.getrandom_out.rng_data_len); | 1192 | recd = be32_to_cpu(tpm_cmd.params.getrandom_out.rng_data_len); |
| 1193 | if (recd > num_bytes) { | ||
| 1194 | total = -EFAULT; | ||
| 1195 | break; | ||
| 1196 | } | ||
| 1193 | 1197 | ||
| 1194 | rlength = be32_to_cpu(tpm_cmd.header.out.length); | 1198 | rlength = be32_to_cpu(tpm_cmd.header.out.length); |
| 1195 | if (rlength < offsetof(struct tpm_getrandom_out, rng_data) + | 1199 | if (rlength < offsetof(struct tpm_getrandom_out, rng_data) + |
| @@ -683,6 +683,10 @@ static int tpm2_unseal_cmd(struct tpm_chip *chip, · 682 unmodified lines | |||
| 683 | if (!rc) { | 683 | if (!rc) { |
| 684 | data_len = be16_to_cpup( | 684 | data_len = be16_to_cpup( |
| 685 | (__be16 *) &buf.data[TPM_HEADER_SIZE + 4]); | 685 | (__be16 *) &buf.data[TPM_HEADER_SIZE + 4]); |
| 686 | if (data_len < MIN_KEY_SIZE || data_len > MAX_KEY_SIZE + 1) { | ||
| 687 | rc = -EFAULT; | ||
| 688 | goto out; | ||
| 689 | } | ||
| 686 | 690 | ||
| 687 | rlength = be32_to_cpu(((struct tpm2_cmd *)&buf) | 691 | rlength = be32_to_cpu(((struct tpm2_cmd *)&buf) |
| 688 | ->header.out.length); | 692 | ->header.out.length); |
| @@ -473,7 +473,8 @@ static int recv_data(struct tpm_chip *chip, u8 *buf, size_t count) · 472 unmodified lines | |||
| 473 | static int tpm_tis_i2c_recv(struct tpm_chip *chip, u8 *buf, size_t count) | 473 | static int tpm_tis_i2c_recv(struct tpm_chip *chip, u8 *buf, size_t count) |
| 474 | { | 474 | { |
| 475 | int size = 0; | 475 | int size = 0; |
| 476 | int expected, status; | 476 | int status; |
| 477 | u32 expected; | ||
| 477 | 478 | ||
| 478 | if (count < TPM_HEADER_SIZE) { | 479 | if (count < TPM_HEADER_SIZE) { |
| 479 | size = -EIO; | 480 | size = -EIO; |
| @@ -488,7 +489,7 @@ static int tpm_tis_i2c_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 8 unmodified lines | |||
| 488 | } | 489 | } |
| 489 | 490 | ||
| 490 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); | 491 | expected = be32_to_cpu(*(__be32 *)(buf + 2)); |
| 491 | if ((size_t) expected > count) { | 492 | if (((size_t) expected > count) || (expected < TPM_HEADER_SIZE)) { |
| 492 | size = -EIO; | 493 | size = -EIO; |
| 493 | goto out; | 494 | goto out; |
| 494 | } | 495 | } |
| @@ -281,7 +281,11 @@ static int i2c_nuvoton_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 280 unmodified lines | |||
| 281 | struct device *dev = chip->dev.parent; | 281 | struct device *dev = chip->dev.parent; |
| 282 | struct i2c_client *client = to_i2c_client(dev); | 282 | struct i2c_client *client = to_i2c_client(dev); |
| 283 | s32 rc; | 283 | s32 rc; |
| 284 | int expected, status, burst_count, retries, size = 0; | 284 | int status; |
| 285 | int burst_count; | ||
| 286 | int retries; | ||
| 287 | int size = 0; | ||
| 288 | u32 expected; | ||
| 285 | 289 | ||
| 286 | if (count < TPM_HEADER_SIZE) { | 290 | if (count < TPM_HEADER_SIZE) { |
| 287 | i2c_nuvoton_ready(chip); /* return to idle */ | 291 | i2c_nuvoton_ready(chip); /* return to idle */ |
| @@ -323,7 +327,7 @@ static int i2c_nuvoton_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 35 unmodified lines | |||
| 323 | * to machine native | 327 | * to machine native |
| 324 | */ | 328 | */ |
| 325 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); | 329 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); |
| 326 | if (expected > count) { | 330 | if (expected > count || expected < size) { |
| 327 | dev_err(dev, "%s() expected > count\n", __func__); | 331 | dev_err(dev, "%s() expected > count\n", __func__); |
| 328 | size = -EIO; | 332 | size = -EIO; |
| 329 | continue; | 333 | continue; |
| @@ -270,7 +270,8 @@ static int tpm_tis_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 269 unmodified lines | |||
| 270 | { | 270 | { |
| 271 | struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); | 271 | struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); |
| 272 | int size = 0; | 272 | int size = 0; |
| 273 | int expected, status; | 273 | int status; |
| 274 | u32 expected; | ||
| 274 | 275 | ||
| 275 | if (count < TPM_HEADER_SIZE) { | 276 | if (count < TPM_HEADER_SIZE) { |
| 276 | size = -EIO; | 277 | size = -EIO; |
| @@ -285,7 +286,7 @@ static int tpm_tis_recv(struct tpm_chip *chip, u8 *buf, size_t count) · 8 unmodified lines | |||
| 285 | } | 286 | } |
| 286 | 287 | ||
| 287 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); | 288 | expected = be32_to_cpu(*(__be32 *) (buf + 2)); |
| 288 | if (expected > count) { | 289 | if (expected > count || expected < TPM_HEADER_SIZE) { |
| 289 | size = -EIO; | 290 | size = -EIO; |
| 290 | goto out; | 291 | goto out; |
| 291 | } | 292 | } |
| @@ -386,6 +386,7 @@ config ATMEL_PIT · 385 unmodified lines | |||
| 386 | 386 | ||
| 387 | config ATMEL_ST | 387 | config ATMEL_ST |
| 388 | bool "Atmel ST timer support" if COMPILE_TEST | 388 | bool "Atmel ST timer support" if COMPILE_TEST |
| 389 | depends on HAS_IOMEM | ||
| 389 | select TIMER_OF | 390 | select TIMER_OF |
| 390 | select MFD_SYSCON | 391 | select MFD_SYSCON |
| 391 | help | 392 | help |
| @@ -251,9 +251,14 @@ static irqreturn_t timer_irq_handler(int irq, void *dev_id) · 250 unmodified lines | |||
| 251 | int irq_reenable = clockevent_state_periodic(evt); | 251 | int irq_reenable = clockevent_state_periodic(evt); |
| 252 | 252 | ||
| 253 | /* | 253 | /* |
| 254 | * Any write to CTRL reg ACks the interrupt, we rewrite the | 254 | * 1. ACK the interrupt |
| 255 | * Count when [N]ot [H]alted bit. | 255 | * - For ARC700, any write to CTRL reg ACKs it, so just rewrite |
| 256 | * And re-arm it if perioid by [I]nterrupt [E]nable bit | 256 | * Count when [N]ot [H]alted bit. |
| 257 | * - For HS3x, it is a bit subtle. On taken count-down interrupt, | ||
| 258 | * IP bit [3] is set, which needs to be cleared for ACK'ing. | ||
| 259 | * The write below can only update the other two bits, hence | ||
| 260 | * explicitly clears IP bit | ||
| 261 | * 2. Re-arm interrupt if periodic by writing to IE bit [0] | ||
| 257 | */ | 262 | */ |
| 258 | write_aux_reg(ARC_REG_TIMER0_CTRL, irq_reenable | TIMER_CTRL_NH); | 263 | write_aux_reg(ARC_REG_TIMER0_CTRL, irq_reenable | TIMER_CTRL_NH); |
| 259 | 264 | ||
| @@ -281,7 +281,7 @@ static int __init __ftm_clk_init(struct device_node *np, char *cnt_name, · 280 unmodified lines | |||
| 281 | 281 | ||
| 282 | static unsigned long __init ftm_clk_init(struct device_node *np) | 282 | static unsigned long __init ftm_clk_init(struct device_node *np) |
| 283 | { | 283 | { |
| 284 | unsigned long freq; | 284 | long freq; |
| 285 | 285 | ||
| 286 | freq = __ftm_clk_init(np, "ftm-evt-counter-en", "ftm-evt"); | 286 | freq = __ftm_clk_init(np, "ftm-evt-counter-en", "ftm-evt"); |
| 287 | if (freq <= 0) | 287 | if (freq <= 0) |
| @@ -166,7 +166,7 @@ static int __init __gic_clocksource_init(void) · 165 unmodified lines | |||
| 166 | 166 | ||
| 167 | /* Set clocksource mask. */ | 167 | /* Set clocksource mask. */ |
| 168 | count_width = read_gic_config() & GIC_CONFIG_COUNTBITS; | 168 | count_width = read_gic_config() & GIC_CONFIG_COUNTBITS; |
| 169 | count_width >>= __fls(GIC_CONFIG_COUNTBITS); | 169 | count_width >>= __ffs(GIC_CONFIG_COUNTBITS); |
| 170 | count_width *= 4; | 170 | count_width *= 4; |
| 171 | count_width += 32; | 171 | count_width += 32; |
| 172 | gic_clocksource.mask = CLOCKSOURCE_MASK(count_width); | 172 | gic_clocksource.mask = CLOCKSOURCE_MASK(count_width); |
| @@ -205,12 +205,12 @@ static int __init gic_clocksource_of_init(struct device_node *node) · 32 unmodified lines | |||
| 205 | } else if (of_property_read_u32(node, "clock-frequency", | 205 | } else if (of_property_read_u32(node, "clock-frequency", |
| 206 | &gic_frequency)) { | 206 | &gic_frequency)) { |
| 207 | pr_err("GIC frequency not specified.\n"); | 207 | pr_err("GIC frequency not specified.\n"); |
| 208 | return -EINVAL;; | 208 | return -EINVAL; |
| 209 | } | 209 | } |
| 210 | gic_timer_irq = irq_of_parse_and_map(node, 0); | 210 | gic_timer_irq = irq_of_parse_and_map(node, 0); |
| 211 | if (!gic_timer_irq) { | 211 | if (!gic_timer_irq) { |
| 212 | pr_err("GIC timer IRQ not specified.\n"); | 212 | pr_err("GIC timer IRQ not specified.\n"); |
| 213 | return -EINVAL;; | 213 | return -EINVAL; |
| 214 | } | 214 | } |
| 215 | 215 | ||
| 216 | ret = __gic_clocksource_init(); | 216 | ret = __gic_clocksource_init(); |
| @@ -334,7 +334,7 @@ static int __init sun5i_timer_init(struct device_node *node) · 333 unmodified lines | |||
| 334 | timer_base = of_io_request_and_map(node, 0, of_node_full_name(node)); | 334 | timer_base = of_io_request_and_map(node, 0, of_node_full_name(node)); |
| 335 | if (IS_ERR(timer_base)) { | 335 | if (IS_ERR(timer_base)) { |
| 336 | pr_err("Can't map registers\n"); | 336 | pr_err("Can't map registers\n"); |
| 337 | return PTR_ERR(timer_base);; | 337 | return PTR_ERR(timer_base); |
| 338 | } | 338 | } |
| 339 | 339 | ||
| 340 | irq = irq_of_parse_and_map(node, 0); | 340 | irq = irq_of_parse_and_map(node, 0); |
| @@ -44,10 +44,10 @@ config ARM_DT_BL_CPUFREQ · 43 unmodified lines | |||
| 44 | 44 | ||
| 45 | config ARM_SCPI_CPUFREQ | 45 | config ARM_SCPI_CPUFREQ |
| 46 | tristate "SCPI based CPUfreq driver" | 46 | tristate "SCPI based CPUfreq driver" |
| 47 | depends on ARM_BIG_LITTLE_CPUFREQ && ARM_SCPI_PROTOCOL && COMMON_CLK_SCPI | 47 | depends on ARM_SCPI_PROTOCOL && COMMON_CLK_SCPI |
| 48 | help | 48 | help |
| 49 | This adds the CPUfreq driver support for ARM big.LITTLE platforms | 49 | This adds the CPUfreq driver support for ARM platforms using SCPI |
| 50 | using SCPI protocol for CPU power management. | 50 | protocol for CPU power management. |
| 51 | 51 | ||
| 52 | This driver uses SCPI Message Protocol driver to interact with the | 52 | This driver uses SCPI Message Protocol driver to interact with the |
| 53 | firmware providing the CPU DVFS functionality. | 53 | firmware providing the CPU DVFS functionality. |
| @@ -629,7 +629,7 @@ static int acpi_cpufreq_blacklist(struct cpuinfo_x86 *c) · 628 unmodified lines | |||
| 629 | if (c->x86_vendor == X86_VENDOR_INTEL) { | 629 | if (c->x86_vendor == X86_VENDOR_INTEL) { |
| 630 | if ((c->x86 == 15) && | 630 | if ((c->x86 == 15) && |
| 631 | (c->x86_model == 6) && | 631 | (c->x86_model == 6) && |
| 632 | (c->x86_mask == 8)) { | 632 | (c->x86_stepping == 8)) { |
| 633 | pr_info("Intel(R) Xeon(R) 7100 Errata AL30, processors may lock up on frequency changes: disabling acpi-cpufreq\n"); | 633 | pr_info("Intel(R) Xeon(R) 7100 Errata AL30, processors may lock up on frequency changes: disabling acpi-cpufreq\n"); |
| 634 | return -ENODEV; | 634 | return -ENODEV; |
| 635 | } | 635 | } |
| @@ -775,7 +775,7 @@ static int longhaul_cpu_init(struct cpufreq_policy *policy) · 774 unmodified lines | |||
| 775 | break; | 775 | break; |
| 776 | 776 | ||
| 777 | case 7: | 777 | case 7: |
| 778 | switch (c->x86_mask) { | 778 | switch (c->x86_stepping) { |
| 779 | case 0: | 779 | case 0: |
| 780 | longhaul_version = TYPE_LONGHAUL_V1; | 780 | longhaul_version = TYPE_LONGHAUL_V1; |
| 781 | cpu_model = CPU_SAMUEL2; | 781 | cpu_model = CPU_SAMUEL2; |
| @@ -787,7 +787,7 @@ static int longhaul_cpu_init(struct cpufreq_policy *policy) · 5 unmodified lines | |||
| 787 | break; | 787 | break; |
| 788 | case 1 ... 15: | 788 | case 1 ... 15: |
| 789 | longhaul_version = TYPE_LONGHAUL_V2; | 789 | longhaul_version = TYPE_LONGHAUL_V2; |
| 790 | if (c->x86_mask < 8) { | 790 | if (c->x86_stepping < 8) { |
| 791 | cpu_model = CPU_SAMUEL2; | 791 | cpu_model = CPU_SAMUEL2; |
| 792 | cpuname = "C3 'Samuel 2' [C5B]"; | 792 | cpuname = "C3 'Samuel 2' [C5B]"; |
| 793 | } else { | 793 | } else { |
| @@ -814,7 +814,7 @@ static int longhaul_cpu_init(struct cpufreq_policy *policy) · 20 unmodified lines | |||
| 814 | numscales = 32; | 814 | numscales = 32; |
| 815 | memcpy(mults, nehemiah_mults, sizeof(nehemiah_mults)); | 815 | memcpy(mults, nehemiah_mults, sizeof(nehemiah_mults)); |
| 816 | memcpy(eblcr, nehemiah_eblcr, sizeof(nehemiah_eblcr)); | 816 | memcpy(eblcr, nehemiah_eblcr, sizeof(nehemiah_eblcr)); |
| 817 | switch (c->x86_mask) { | 817 | switch (c->x86_stepping) { |
| 818 | case 0 ... 1: | 818 | case 0 ... 1: |
| 819 | cpu_model = CPU_NEHEMIAH; | 819 | cpu_model = CPU_NEHEMIAH; |
| 820 | cpuname = "C3 'Nehemiah A' [C5XLOE]"; | 820 | cpuname = "C3 'Nehemiah A' [C5XLOE]"; |
| @@ -168,7 +168,7 @@ static int cpufreq_p4_cpu_init(struct cpufreq_policy *policy) · 167 unmodified lines | |||
| 168 | #endif | 168 | #endif |
| 169 | 169 | ||
| 170 | /* Errata workaround */ | 170 | /* Errata workaround */ |
| 171 | cpuid = (c->x86 << 8) | (c->x86_model << 4) | c->x86_mask; | 171 | cpuid = (c->x86 << 8) | (c->x86_model << 4) | c->x86_stepping; |
| 172 | switch (cpuid) { | 172 | switch (cpuid) { |
| 173 | case 0x0f07: | 173 | case 0x0f07: |
| 174 | case 0x0f0a: | 174 | case 0x0f0a: |
| @@ -131,7 +131,7 @@ static int check_powernow(void) · 130 unmodified lines | |||
| 131 | return 0; | 131 | return 0; |
| 132 | } | 132 | } |
| 133 | 133 | ||
| 134 | if ((c->x86_model == 6) && (c->x86_mask == 0)) { | 134 | if ((c->x86_model == 6) && (c->x86_stepping == 0)) { |
| 135 | pr_info("K7 660[A0] core detected, enabling errata workarounds\n"); | 135 | pr_info("K7 660[A0] core detected, enabling errata workarounds\n"); |
| 136 | have_a0 = 1; | 136 | have_a0 = 1; |
| 137 | } | 137 | } |
| @@ -351,7 +351,13 @@ struct clk *s3c_cpufreq_clk_get(struct device *dev, const char *name) · 350 unmodified lines | |||
| 351 | static int s3c_cpufreq_init(struct cpufreq_policy *policy) | 351 | static int s3c_cpufreq_init(struct cpufreq_policy *policy) |
| 352 | { | 352 | { |
| 353 | policy->clk = clk_arm; | 353 | policy->clk = clk_arm; |
| 354 | return cpufreq_generic_init(policy, ftab, cpu_cur.info->latency); | 354 | |
| 355 | policy->cpuinfo.transition_latency = cpu_cur.info->latency; | ||
| 356 | |||
| 357 | if (ftab) | ||
| 358 | return cpufreq_table_validate_and_show(policy, ftab); | ||
| 359 | |||
| 360 | return 0; | ||
| 355 | } | 361 | } |
| 356 | 362 | ||
| 357 | static int __init s3c_cpufreq_initclks(void) | 363 | static int __init s3c_cpufreq_initclks(void) |
| @@ -51,15 +51,23 @@ static unsigned int scpi_cpufreq_get_rate(unsigned int cpu) · 50 unmodified lines | |||
| 51 | static int | 51 | static int |
| 52 | scpi_cpufreq_set_target(struct cpufreq_policy *policy, unsigned int index) | 52 | scpi_cpufreq_set_target(struct cpufreq_policy *policy, unsigned int index) |
| 53 | { | 53 | { |
| 54 | unsigned long freq = policy->freq_table[index].frequency; | ||
| 54 | struct scpi_data *priv = policy->driver_data; | 55 | struct scpi_data *priv = policy->driver_data; |
| 55 | u64 rate = policy->freq_table[index].frequency * 1000; | 56 | u64 rate = freq * 1000; |
| 56 | int ret; | 57 | int ret; |
| 57 | 58 | ||
| 58 | ret = clk_set_rate(priv->clk, rate); | 59 | ret = clk_set_rate(priv->clk, rate); |
| 59 | if (!ret && (clk_get_rate(priv->clk) != rate)) | ||
| 60 | ret = -EIO; | ||
| 61 | 60 | ||
| 62 | return ret; | 61 | if (ret) |
| 62 | return ret; | ||
| 63 | |||
| 64 | if (clk_get_rate(priv->clk) != rate) | ||
| 65 | return -EIO; | ||
| 66 | |||
| 67 | arch_set_freq_scale(policy->related_cpus, freq, | ||
| 68 | policy->cpuinfo.max_freq); | ||
| 69 | |||
| 70 | return 0; | ||
| 63 | } | 71 | } |
| 64 | 72 | ||
| 65 | static int | 73 | static int |
| @@ -37,7 +37,7 @@ struct cpu_id · 36 unmodified lines | |||
| 37 | { | 37 | { |
| 38 | __u8 x86; /* CPU family */ | 38 | __u8 x86; /* CPU family */ |
| 39 | __u8 x86_model; /* model */ | 39 | __u8 x86_model; /* model */ |
| 40 | __u8 x86_mask; /* stepping */ | 40 | __u8 x86_stepping; /* stepping */ |
| 41 | }; | 41 | }; |
| 42 | 42 | ||
| 43 | enum { | 43 | enum { |
| @@ -277,7 +277,7 @@ static int centrino_verify_cpu_id(const struct cpuinfo_x86 *c, · 233 unmodified lines | |||
| 277 | { | 277 | { |
| 278 | if ((c->x86 == x->x86) && | 278 | if ((c->x86 == x->x86) && |
| 279 | (c->x86_model == x->x86_model) && | 279 | (c->x86_model == x->x86_model) && |
| 280 | (c->x86_mask == x->x86_mask)) | 280 | (c->x86_stepping == x->x86_stepping)) |
| 281 | return 1; | 281 | return 1; |
| 282 | return 0; | 282 | return 0; |
| 283 | } | 283 | } |
| @@ -272,9 +272,9 @@ unsigned int speedstep_detect_processor(void) · 271 unmodified lines | |||
| 272 | ebx = cpuid_ebx(0x00000001); | 272 | ebx = cpuid_ebx(0x00000001); |
| 273 | ebx &= 0x000000FF; | 273 | ebx &= 0x000000FF; |
| 274 | 274 | ||
| 275 | pr_debug("ebx value is %x, x86_mask is %x\n", ebx, c->x86_mask); | 275 | pr_debug("ebx value is %x, x86_stepping is %x\n", ebx, c->x86_stepping); |
| 276 | 276 | ||
| 277 | switch (c->x86_mask) { | 277 | switch (c->x86_stepping) { |
| 278 | case 4: | 278 | case 4: |
| 279 | /* | 279 | /* |
| 280 | * B-stepping [M-P4-M] | 280 | * B-stepping [M-P4-M] |
| @@ -361,7 +361,7 @@ unsigned int speedstep_detect_processor(void) · 80 unmodified lines | |||
| 361 | msr_lo, msr_hi); | 361 | msr_lo, msr_hi); |
| 362 | if ((msr_hi & (1<<18)) && | 362 | if ((msr_hi & (1<<18)) && |
| 363 | (relaxed_check ? 1 : (msr_hi & (3<<24)))) { | 363 | (relaxed_check ? 1 : (msr_hi & (3<<24)))) { |
| 364 | if (c->x86_mask == 0x01) { | 364 | if (c->x86_stepping == 0x01) { |
| 365 | pr_debug("early PIII version\n"); | 365 | pr_debug("early PIII version\n"); |
| 366 | return SPEEDSTEP_CPU_PIII_C_EARLY; | 366 | return SPEEDSTEP_CPU_PIII_C_EARLY; |
| 367 | } else | 367 | } else |
| @@ -228,12 +228,16 @@ static int instantiate_rng(struct device *ctrldev, int state_handle_mask, · 227 unmodified lines | |||
| 228 | * without any error (HW optimizations for later | 228 | * without any error (HW optimizations for later |
| 229 | * CAAM eras), then try again. | 229 | * CAAM eras), then try again. |
| 230 | */ | 230 | */ |
| 231 | if (ret) | ||
| 232 | break; | ||
| 233 | |||
| 231 | rdsta_val = rd_reg32(&ctrl->r4tst[0].rdsta) & RDSTA_IFMASK; | 234 | rdsta_val = rd_reg32(&ctrl->r4tst[0].rdsta) & RDSTA_IFMASK; |
| 232 | if ((status && status != JRSTA_SSRC_JUMP_HALT_CC) || | 235 | if ((status && status != JRSTA_SSRC_JUMP_HALT_CC) || |
| 233 | !(rdsta_val & (1 << sh_idx))) | 236 | !(rdsta_val & (1 << sh_idx))) { |
| 234 | ret = -EAGAIN; | 237 | ret = -EAGAIN; |
| 235 | if (ret) | ||
| 236 | break; | 238 | break; |
| 239 | } | ||
| 240 | |||
| 237 | dev_info(ctrldev, "Instantiated RNG4 SH%d\n", sh_idx); | 241 | dev_info(ctrldev, "Instantiated RNG4 SH%d\n", sh_idx); |
| 238 | /* Clear the contents before recreating the descriptor */ | 242 | /* Clear the contents before recreating the descriptor */ |
| 239 | memset(desc, 0x00, CAAM_CMD_SZ * 7); | 243 | memset(desc, 0x00, CAAM_CMD_SZ * 7); |
| @@ -211,7 +211,7 @@ static int __sev_platform_shutdown_locked(int *error) · 210 unmodified lines | |||
| 211 | { | 211 | { |
| 212 | int ret; | 212 | int ret; |
| 213 | 213 | ||
| 214 | ret = __sev_do_cmd_locked(SEV_CMD_SHUTDOWN, 0, error); | 214 | ret = __sev_do_cmd_locked(SEV_CMD_SHUTDOWN, NULL, error); |
| 215 | if (ret) | 215 | if (ret) |
| 216 | return ret; | 216 | return ret; |
| 217 | 217 | ||
| @@ -271,7 +271,7 @@ static int sev_ioctl_do_reset(struct sev_issue_cmd *argp) · 53 unmodified lines | |||
| 271 | return rc; | 271 | return rc; |
| 272 | } | 272 | } |
| 273 | 273 | ||
| 274 | return __sev_do_cmd_locked(SEV_CMD_FACTORY_RESET, 0, &argp->error); | 274 | return __sev_do_cmd_locked(SEV_CMD_FACTORY_RESET, NULL, &argp->error); |
| 275 | } | 275 | } |
| 276 | 276 | ||
| 277 | static int sev_ioctl_do_platform_status(struct sev_issue_cmd *argp) | 277 | static int sev_ioctl_do_platform_status(struct sev_issue_cmd *argp) |
| @@ -299,7 +299,7 @@ static int sev_ioctl_do_pek_pdh_gen(int cmd, struct sev_issue_cmd *argp) · 21 unmodified lines | |||
| 299 | return rc; | 299 | return rc; |
| 300 | } | 300 | } |
| 301 | 301 | ||
| 302 | return __sev_do_cmd_locked(cmd, 0, &argp->error); | 302 | return __sev_do_cmd_locked(cmd, NULL, &argp->error); |
| 303 | } | 303 | } |
| 304 | 304 | ||
| 305 | static int sev_ioctl_do_pek_csr(struct sev_issue_cmd *argp) | 305 | static int sev_ioctl_do_pek_csr(struct sev_issue_cmd *argp) |
| @@ -624,7 +624,7 @@ EXPORT_SYMBOL_GPL(sev_guest_decommission); · 318 unmodified lines | |||
| 624 | 624 | ||
| 625 | int sev_guest_df_flush(int *error) | 625 | int sev_guest_df_flush(int *error) |
| 626 | { | 626 | { |
| 627 | return sev_do_cmd(SEV_CMD_DF_FLUSH, 0, error); | 627 | return sev_do_cmd(SEV_CMD_DF_FLUSH, NULL, error); |
| 628 | } | 628 | } |
| 629 | EXPORT_SYMBOL_GPL(sev_guest_df_flush); | 629 | EXPORT_SYMBOL_GPL(sev_guest_df_flush); |
| 630 | 630 | ||
| @@ -512,7 +512,7 @@ static int __init padlock_init(void) · 511 unmodified lines | |||
| 512 | 512 | ||
| 513 | printk(KERN_NOTICE PFX "Using VIA PadLock ACE for AES algorithm.\n"); | 513 | printk(KERN_NOTICE PFX "Using VIA PadLock ACE for AES algorithm.\n"); |
| 514 | 514 | ||
| 515 | if (c->x86 == 6 && c->x86_model == 15 && c->x86_mask == 2) { | 515 | if (c->x86 == 6 && c->x86_model == 15 && c->x86_stepping == 2) { |
| 516 | ecb_fetch_blocks = MAX_ECB_FETCH_BLOCKS; | 516 | ecb_fetch_blocks = MAX_ECB_FETCH_BLOCKS; |
| 517 | cbc_fetch_blocks = MAX_CBC_FETCH_BLOCKS; | 517 | cbc_fetch_blocks = MAX_CBC_FETCH_BLOCKS; |
| 518 | printk(KERN_NOTICE PFX "VIA Nano stepping 2 detected: enabling workaround.\n"); | 518 | printk(KERN_NOTICE PFX "VIA Nano stepping 2 detected: enabling workaround.\n"); |
| @@ -1922,15 +1922,21 @@ static void s5p_aes_crypt_start(struct s5p_aes_dev *dev, unsigned long mode) · 1921 unmodified lines | |||
| 1922 | uint32_t aes_control; | 1922 | uint32_t aes_control; |
| 1923 | unsigned long flags; | 1923 | unsigned long flags; |
| 1924 | int err; | 1924 | int err; |
| 1925 | u8 *iv; | ||
| 1925 | 1926 | ||
| 1926 | aes_control = SSS_AES_KEY_CHANGE_MODE; | 1927 | aes_control = SSS_AES_KEY_CHANGE_MODE; |
| 1927 | if (mode & FLAGS_AES_DECRYPT) | 1928 | if (mode & FLAGS_AES_DECRYPT) |
| 1928 | aes_control |= SSS_AES_MODE_DECRYPT; | 1929 | aes_control |= SSS_AES_MODE_DECRYPT; |
| 1929 | 1930 | ||
| 1930 | if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CBC) | 1931 | if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CBC) { |
| 1931 | aes_control |= SSS_AES_CHAIN_MODE_CBC; | 1932 | aes_control |= SSS_AES_CHAIN_MODE_CBC; |
| 1932 | else if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CTR) | 1933 | iv = req->info; |
| 1934 | } else if ((mode & FLAGS_AES_MODE_MASK) == FLAGS_AES_CTR) { | ||
| 1933 | aes_control |= SSS_AES_CHAIN_MODE_CTR; | 1935 | aes_control |= SSS_AES_CHAIN_MODE_CTR; |
| 1936 | iv = req->info; | ||
| 1937 | } else { | ||
| 1938 | iv = NULL; /* AES_ECB */ | ||
| 1939 | } | ||
| 1934 | 1940 | ||
| 1935 | if (dev->ctx->keylen == AES_KEYSIZE_192) | 1941 | if (dev->ctx->keylen == AES_KEYSIZE_192) |
| 1936 | aes_control |= SSS_AES_KEY_SIZE_192; | 1942 | aes_control |= SSS_AES_KEY_SIZE_192; |
| @@ -1961,7 +1967,7 @@ static void s5p_aes_crypt_start(struct s5p_aes_dev *dev, unsigned long mode) · 24 unmodified lines | |||
| 1961 | goto outdata_error; | 1967 | goto outdata_error; |
| 1962 | 1968 | ||
| 1963 | SSS_AES_WRITE(dev, AES_CONTROL, aes_control); | 1969 | SSS_AES_WRITE(dev, AES_CONTROL, aes_control); |
| 1964 | s5p_set_aes(dev, dev->ctx->aes_key, req->info, dev->ctx->keylen); | 1970 | s5p_set_aes(dev, dev->ctx->aes_key, iv, dev->ctx->keylen); |
| 1965 | 1971 | ||
| 1966 | s5p_set_dma_indata(dev, dev->sg_src); | 1972 | s5p_set_dma_indata(dev, dev->sg_src); |
| 1967 | s5p_set_dma_outdata(dev, dev->sg_dst); | 1973 | s5p_set_dma_outdata(dev, dev->sg_dst); |
| @@ -28,7 +28,7 @@ int sun4i_ss_prng_generate(struct crypto_rng *tfm, const u8 *src, · 27 unmodified lines | |||
| 28 | algt = container_of(alg, struct sun4i_ss_alg_template, alg.rng); | 28 | algt = container_of(alg, struct sun4i_ss_alg_template, alg.rng); |
| 29 | ss = algt->ss; | 29 | ss = algt->ss; |
| 30 | 30 | ||
| 31 | spin_lock(&ss->slock); | 31 | spin_lock_bh(&ss->slock); |
| 32 | 32 | ||
| 33 | writel(mode, ss->base + SS_CTL); | 33 | writel(mode, ss->base + SS_CTL); |
| 34 | 34 | ||
| @@ -51,6 +51,6 @@ int sun4i_ss_prng_generate(struct crypto_rng *tfm, const u8 *src, · 16 unmodified lines | |||
| 51 | } | 51 | } |
| 52 | 52 | ||
| 53 | writel(0, ss->base + SS_CTL); | 53 | writel(0, ss->base + SS_CTL); |
| 54 | spin_unlock(&ss->slock); | 54 | spin_unlock_bh(&ss->slock); |
| 55 | return dlen; | 55 | return 0; |
| 56 | } | 56 | } |
| @@ -1138,6 +1138,10 @@ static int talitos_sg_map(struct device *dev, struct scatterlist *src, · 1137 unmodified lines | |||
| 1138 | struct talitos_private *priv = dev_get_drvdata(dev); | 1138 | struct talitos_private *priv = dev_get_drvdata(dev); |
| 1139 | bool is_sec1 = has_ftr_sec1(priv); | 1139 | bool is_sec1 = has_ftr_sec1(priv); |
| 1140 | 1140 | ||
| 1141 | if (!src) { | ||
| 1142 | to_talitos_ptr(ptr, 0, 0, is_sec1); | ||
| 1143 | return 1; | ||
| 1144 | } | ||
| 1141 | if (sg_count == 1) { | 1145 | if (sg_count == 1) { |
| 1142 | to_talitos_ptr(ptr, sg_dma_address(src) + offset, len, is_sec1); | 1146 | to_talitos_ptr(ptr, sg_dma_address(src) + offset, len, is_sec1); |
| 1143 | return sg_count; | 1147 | return sg_count; |
| @@ -246,12 +246,6 @@ long dax_direct_access(struct dax_device *dax_dev, pgoff_t pgoff, long nr_pages, · 245 unmodified lines | |||
| 246 | { | 246 | { |
| 247 | long avail; | 247 | long avail; |
| 248 | 248 | ||
| 249 | /* | ||
| 250 | * The device driver is allowed to sleep, in order to make the | ||
| 251 | * memory directly accessible. | ||
| 252 | */ | ||
| 253 | might_sleep(); | ||
| 254 | |||
| 255 | if (!dax_dev) | 249 | if (!dax_dev) |
| 256 | return -EOPNOTSUPP; | 250 | return -EOPNOTSUPP; |
| 257 | 251 | ||
| @@ -163,6 +163,7 @@ struct mv_xor_v2_device { · 162 unmodified lines | |||
| 163 | void __iomem *dma_base; | 163 | void __iomem *dma_base; |
| 164 | void __iomem *glob_base; | 164 | void __iomem *glob_base; |
| 165 | struct clk *clk; | 165 | struct clk *clk; |
| 166 | struct clk *reg_clk; | ||
| 166 | struct tasklet_struct irq_tasklet; | 167 | struct tasklet_struct irq_tasklet; |
| 167 | struct list_head free_sw_desc; | 168 | struct list_head free_sw_desc; |
| 168 | struct dma_device dmadev; | 169 | struct dma_device dmadev; |
| @@ -749,13 +750,26 @@ static int mv_xor_v2_probe(struct platform_device *pdev) · 580 unmodified lines | |||
| 749 | if (ret) | 750 | if (ret) |
| 750 | return ret; | 751 | return ret; |
| 751 | 752 | ||
| 753 | xor_dev->reg_clk = devm_clk_get(&pdev->dev, "reg"); | ||
| 754 | if (PTR_ERR(xor_dev->reg_clk) != -ENOENT) { | ||
| 755 | if (!IS_ERR(xor_dev->reg_clk)) { | ||
| 756 | ret = clk_prepare_enable(xor_dev->reg_clk); | ||
| 757 | if (ret) | ||
| 758 | return ret; | ||
| 759 | } else { | ||
| 760 | return PTR_ERR(xor_dev->reg_clk); | ||
| 761 | } | ||
| 762 | } | ||
| 763 | |||
| 752 | xor_dev->clk = devm_clk_get(&pdev->dev, NULL); | 764 | xor_dev->clk = devm_clk_get(&pdev->dev, NULL); |
| 753 | if (IS_ERR(xor_dev->clk) && PTR_ERR(xor_dev->clk) == -EPROBE_DEFER) | 765 | if (IS_ERR(xor_dev->clk) && PTR_ERR(xor_dev->clk) == -EPROBE_DEFER) { |
| 754 | return -EPROBE_DEFER; | 766 | ret = EPROBE_DEFER; |
| 767 | goto disable_reg_clk; | ||
| 768 | } | ||
| 755 | if (!IS_ERR(xor_dev->clk)) { | 769 | if (!IS_ERR(xor_dev->clk)) { |
| 756 | ret = clk_prepare_enable(xor_dev->clk); | 770 | ret = clk_prepare_enable(xor_dev->clk); |
| 757 | if (ret) | 771 | if (ret) |
| 758 | return ret; | 772 | goto disable_reg_clk; |
| 759 | } | 773 | } |
| 760 | 774 | ||
| 761 | ret = platform_msi_domain_alloc_irqs(&pdev->dev, 1, | 775 | ret = platform_msi_domain_alloc_irqs(&pdev->dev, 1, |
| @@ -866,8 +880,9 @@ free_hw_desq: · 104 unmodified lines | |||
| 866 | free_msi_irqs: | 880 | free_msi_irqs: |
| 867 | platform_msi_domain_free_irqs(&pdev->dev); | 881 | platform_msi_domain_free_irqs(&pdev->dev); |
| 868 | disable_clk: | 882 | disable_clk: |
| 869 | if (!IS_ERR(xor_dev->clk)) | 883 | clk_disable_unprepare(xor_dev->clk); |
| 870 | clk_disable_unprepare(xor_dev->clk); | 884 | disable_reg_clk: |
| 885 | clk_disable_unprepare(xor_dev->reg_clk); | ||
| 871 | return ret; | 886 | return ret; |
| 872 | } | 887 | } |
| 873 | 888 | ||
| @@ -917,7 +917,7 @@ rcar_dmac_chan_prep_sg(struct rcar_dmac_chan *chan, struct scatterlist *sgl, · 916 unmodified lines | |||
| 917 | 917 | ||
| 918 | rcar_dmac_chan_configure_desc(chan, desc); | 918 | rcar_dmac_chan_configure_desc(chan, desc); |
| 919 | 919 | ||
| 920 | max_chunk_size = (RCAR_DMATCR_MASK + 1) << desc->xfer_shift; | 920 | max_chunk_size = RCAR_DMATCR_MASK << desc->xfer_shift; |
| 921 | 921 | ||
| 922 | /* | 922 | /* |
| 923 | * Allocate and fill the transfer chunk descriptors. We own the only | 923 | * Allocate and fill the transfer chunk descriptors. We own the only |
| @@ -3147,7 +3147,7 @@ static struct amd64_family_type *per_family_init(struct amd64_pvt *pvt) · 3146 unmodified lines | |||
| 3147 | struct amd64_family_type *fam_type = NULL; | 3147 | struct amd64_family_type *fam_type = NULL; |
| 3148 | 3148 | ||
| 3149 | pvt->ext_model = boot_cpu_data.x86_model >> 4; | 3149 | pvt->ext_model = boot_cpu_data.x86_model >> 4; |
| 3150 | pvt->stepping = boot_cpu_data.x86_mask; | 3150 | pvt->stepping = boot_cpu_data.x86_stepping; |
| 3151 | pvt->model = boot_cpu_data.x86_model; | 3151 | pvt->model = boot_cpu_data.x86_model; |
| 3152 | pvt->fam = boot_cpu_data.x86; | 3152 | pvt->fam = boot_cpu_data.x86; |
| 3153 | 3153 | ||
| @@ -279,7 +279,7 @@ static const u32 correrrthrsld[] = { · 278 unmodified lines | |||
| 279 | * sbridge structs | 279 | * sbridge structs |
| 280 | */ | 280 | */ |
| 281 | 281 | ||
| 282 | #define NUM_CHANNELS 4 /* Max channels per MC */ | 282 | #define NUM_CHANNELS 6 /* Max channels per MC */ |
| 283 | #define MAX_DIMMS 3 /* Max DIMMS per channel */ | 283 | #define MAX_DIMMS 3 /* Max DIMMS per channel */ |
| 284 | #define KNL_MAX_CHAS 38 /* KNL max num. of Cache Home Agents */ | 284 | #define KNL_MAX_CHAS 38 /* KNL max num. of Cache Home Agents */ |
| 285 | #define KNL_MAX_CHANNELS 6 /* KNL max num. of PCI channels */ | 285 | #define KNL_MAX_CHANNELS 6 /* KNL max num. of PCI channels */ |
| @@ -1,7 +1,6 @@ | |||
| 1 | /* | 1 | /* |
| 2 | * extcon-axp288.c - X-Power AXP288 PMIC extcon cable detection driver | 2 | * extcon-axp288.c - X-Power AXP288 PMIC extcon cable detection driver |
| 3 | * | 3 | * |
| 4 | * Copyright (C) 2016-2017 Hans de Goede <hdegoede@redhat.com> | ||
| 5 | * Copyright (C) 2015 Intel Corporation | 4 | * Copyright (C) 2015 Intel Corporation |
| 6 | * Author: Ramakrishna Pallala <ramakrishna.pallala@intel.com> | 5 | * Author: Ramakrishna Pallala <ramakrishna.pallala@intel.com> |
| 7 | * | 6 | * |
| @@ -98,15 +97,13 @@ struct axp288_extcon_info { · 90 unmodified lines | |||
| 98 | struct device *dev; | 97 | struct device *dev; |
| 99 | struct regmap *regmap; | 98 | struct regmap *regmap; |
| 100 | struct regmap_irq_chip_data *regmap_irqc; | 99 | struct regmap_irq_chip_data *regmap_irqc; |
| 101 | struct delayed_work det_work; | ||
| 102 | int irq[EXTCON_IRQ_END]; | 100 | int irq[EXTCON_IRQ_END]; |
| 103 | struct extcon_dev *edev; | 101 | struct extcon_dev *edev; |
| 104 | unsigned int previous_cable; | 102 | unsigned int previous_cable; |
| 105 | bool first_detect_done; | ||
| 106 | }; | 103 | }; |
| 107 | 104 | ||
| 108 | /* Power up/down reason string array */ | 105 | /* Power up/down reason string array */ |
| 109 | static char *axp288_pwr_up_down_info[] = { | 106 | static const char * const axp288_pwr_up_down_info[] = { |
| 110 | "Last wake caused by user pressing the power button", | 107 | "Last wake caused by user pressing the power button", |
| 111 | "Last wake caused by a charger insertion", | 108 | "Last wake caused by a charger insertion", |
| 112 | "Last wake caused by a battery insertion", | 109 | "Last wake caused by a battery insertion", |
| @@ -124,7 +121,7 @@ static char *axp288_pwr_up_down_info[] = { · 11 unmodified lines | |||
| 124 | */ | 121 | */ |
| 125 | static void axp288_extcon_log_rsi(struct axp288_extcon_info *info) | 122 | static void axp288_extcon_log_rsi(struct axp288_extcon_info *info) |
| 126 | { | 123 | { |
| 127 | char **rsi; | 124 | const char * const *rsi; |
| 128 | unsigned int val, i, clear_mask = 0; | 125 | unsigned int val, i, clear_mask = 0; |
| 129 | int ret; | 126 | int ret; |
| 130 | 127 | ||
| @@ -140,25 +137,6 @@ static void axp288_extcon_log_rsi(struct axp288_extcon_info *info) · 9 unmodified lines | |||
| 140 | regmap_write(info->regmap, AXP288_PS_BOOT_REASON_REG, clear_mask); | 137 | regmap_write(info->regmap, AXP288_PS_BOOT_REASON_REG, clear_mask); |
| 141 | } | 138 | } |
| 142 | 139 | ||
| 143 | static void axp288_chrg_detect_complete(struct axp288_extcon_info *info) | ||
| 144 | { | ||
| 145 | /* | ||
| 146 | * We depend on other drivers to do things like mux the data lines, | ||
| 147 | * enable/disable vbus based on the id-pin, etc. Sometimes the BIOS has | ||
| 148 | * not set these things up correctly resulting in the initial charger | ||
| 149 | * cable type detection giving a wrong result and we end up not charging | ||
| 150 | * or charging at only 0.5A. | ||
| 151 | * | ||
| 152 | * So we schedule a second cable type detection after 2 seconds to | ||
| 153 | * give the other drivers time to load and do their thing. | ||
| 154 | */ | ||
| 155 | if (!info->first_detect_done) { | ||
| 156 | queue_delayed_work(system_wq, &info->det_work, | ||
| 157 | msecs_to_jiffies(2000)); | ||
| 158 | info->first_detect_done = true; | ||
| 159 | } | ||
| 160 | } | ||
| 161 | |||
| 162 | static int axp288_handle_chrg_det_event(struct axp288_extcon_info *info) | 140 | static int axp288_handle_chrg_det_event(struct axp288_extcon_info *info) |
| 163 | { | 141 | { |
| 164 | int ret, stat, cfg, pwr_stat; | 142 | int ret, stat, cfg, pwr_stat; |
| @@ -223,8 +201,6 @@ no_vbus: · 58 unmodified lines | |||
| 223 | info->previous_cable = cable; | 201 | info->previous_cable = cable; |
| 224 | } | 202 | } |
| 225 | 203 | ||
| 226 | axp288_chrg_detect_complete(info); | ||
| 227 | |||
| 228 | return 0; | 204 | return 0; |
| 229 | 205 | ||
| 230 | dev_det_ret: | 206 | dev_det_ret: |
| @@ -246,11 +222,8 @@ static irqreturn_t axp288_extcon_isr(int irq, void *data) · 15 unmodified lines | |||
| 246 | return IRQ_HANDLED; | 222 | return IRQ_HANDLED; |
| 247 | } | 223 | } |
| 248 | 224 | ||
| 249 | static void axp288_extcon_det_work(struct work_struct *work) | 225 | static void axp288_extcon_enable(struct axp288_extcon_info *info) |
| 250 | { | 226 | { |
| 251 | struct axp288_extcon_info *info = | ||
| 252 | container_of(work, struct axp288_extcon_info, det_work.work); | ||
| 253 | |||
| 254 | regmap_update_bits(info->regmap, AXP288_BC_GLOBAL_REG, | 227 | regmap_update_bits(info->regmap, AXP288_BC_GLOBAL_REG, |
| 255 | BC_GLOBAL_RUN, 0); | 228 | BC_GLOBAL_RUN, 0); |
| 256 | /* Enable the charger detection logic */ | 229 | /* Enable the charger detection logic */ |
| @@ -272,7 +245,6 @@ static int axp288_extcon_probe(struct platform_device *pdev) · 15 unmodified lines | |||
| 272 | info->regmap = axp20x->regmap; | 245 | info->regmap = axp20x->regmap; |
| 273 | info->regmap_irqc = axp20x->regmap_irqc; | 246 | info->regmap_irqc = axp20x->regmap_irqc; |
| 274 | info->previous_cable = EXTCON_NONE; | 247 | info->previous_cable = EXTCON_NONE; |
| 275 | INIT_DELAYED_WORK(&info->det_work, axp288_extcon_det_work); | ||
| 276 | 248 | ||
| 277 | platform_set_drvdata(pdev, info); | 249 | platform_set_drvdata(pdev, info); |
| 278 | 250 | ||
| @@ -318,7 +290,7 @@ static int axp288_extcon_probe(struct platform_device *pdev) · 39 unmodified lines | |||
| 318 | } | 290 | } |
| 319 | 291 | ||
| 320 | /* Start charger cable type detection */ | 292 | /* Start charger cable type detection */ |
| 321 | queue_delayed_work(system_wq, &info->det_work, 0); | 293 | axp288_extcon_enable(info); |
| 322 | 294 | ||
| 323 | return 0; | 295 | return 0; |
| 324 | } | 296 | } |
| @@ -153,8 +153,9 @@ static int int3496_probe(struct platform_device *pdev) · 152 unmodified lines | |||
| 153 | return ret; | 153 | return ret; |
| 154 | } | 154 | } |
| 155 | 155 | ||
| 156 | /* queue initial processing of id-pin */ | 156 | /* process id-pin so that we start with the right status */ |
| 157 | queue_delayed_work(system_wq, &data->work, 0); | 157 | queue_delayed_work(system_wq, &data->work, 0); |
| 158 | flush_delayed_work(&data->work); | ||
| 158 | 159 | ||
| 159 | platform_set_drvdata(pdev, data); | 160 | platform_set_drvdata(pdev, data); |
| 160 | 161 | ||
| @@ -14,7 +14,6 @@ · 13 unmodified lines | |||
| 14 | * GNU General Public License for more details. | 14 | * GNU General Public License for more details. |
| 15 | */ | 15 | */ |
| 16 | 16 | ||
| 17 | #include <linux/clk.h> | ||
| 18 | #include <linux/err.h> | 17 | #include <linux/err.h> |
| 19 | #include <linux/gpio.h> | 18 | #include <linux/gpio.h> |
| 20 | #include <linux/init.h> | 19 | #include <linux/init.h> |
| @@ -37,10 +36,9 @@ struct gpio_rcar_priv { · 16 unmodified lines | |||
| 37 | struct platform_device *pdev; | 36 | struct platform_device *pdev; |
| 38 | struct gpio_chip gpio_chip; | 37 | struct gpio_chip gpio_chip; |
| 39 | struct irq_chip irq_chip; | 38 | struct irq_chip irq_chip; |
| 40 | struct clk *clk; | ||
| 41 | unsigned int irq_parent; | 39 | unsigned int irq_parent; |
| 40 | atomic_t wakeup_path; | ||
| 42 | bool has_both_edge_trigger; | 41 | bool has_both_edge_trigger; |
| 43 | bool needs_clk; | ||
| 44 | }; | 42 | }; |
| 45 | 43 | ||
| 46 | #define IOINTSEL 0x00 /* General IO/Interrupt Switching Register */ | 44 | #define IOINTSEL 0x00 /* General IO/Interrupt Switching Register */ |
| @@ -186,13 +184,10 @@ static int gpio_rcar_irq_set_wake(struct irq_data *d, unsigned int on) · 139 unmodified lines | |||
| 186 | } | 184 | } |
| 187 | } | 185 | } |
| 188 | 186 | ||
| 189 | if (!p->clk) | ||
| 190 | return 0; | ||
| 191 | |||
| 192 | if (on) | 187 | if (on) |
| 193 | clk_enable(p->clk); | 188 | atomic_inc(&p->wakeup_path); |
| 194 | else | 189 | else |
| 195 | clk_disable(p->clk); | 190 | atomic_dec(&p->wakeup_path); |
| 196 | 191 | ||
| 197 | return 0; | 192 | return 0; |
| 198 | } | 193 | } |
| @@ -330,17 +325,14 @@ static int gpio_rcar_direction_output(struct gpio_chip *chip, unsigned offset, · 131 unmodified lines | |||
| 330 | 325 | ||
| 331 | struct gpio_rcar_info { | 326 | struct gpio_rcar_info { |
| 332 | bool has_both_edge_trigger; | 327 | bool has_both_edge_trigger; |
| 333 | bool needs_clk; | ||
| 334 | }; | 328 | }; |
| 335 | 329 | ||
| 336 | static const struct gpio_rcar_info gpio_rcar_info_gen1 = { | 330 | static const struct gpio_rcar_info gpio_rcar_info_gen1 = { |
| 337 | .has_both_edge_trigger = false, | 331 | .has_both_edge_trigger = false, |
| 338 | .needs_clk = false, | ||
| 339 | }; | 332 | }; |
| 340 | 333 | ||
| 341 | static const struct gpio_rcar_info gpio_rcar_info_gen2 = { | 334 | static const struct gpio_rcar_info gpio_rcar_info_gen2 = { |
| 342 | .has_both_edge_trigger = true, | 335 | .has_both_edge_trigger = true, |
| 343 | .needs_clk = true, | ||
| 344 | }; | 336 | }; |
| 345 | 337 | ||
| 346 | static const struct of_device_id gpio_rcar_of_table[] = { | 338 | static const struct of_device_id gpio_rcar_of_table[] = { |
| @@ -403,7 +395,6 @@ static int gpio_rcar_parse_dt(struct gpio_rcar_priv *p, unsigned int *npins) · 56 unmodified lines | |||
| 403 | ret = of_parse_phandle_with_fixed_args(np, "gpio-ranges", 3, 0, &args); | 395 | ret = of_parse_phandle_with_fixed_args(np, "gpio-ranges", 3, 0, &args); |
| 404 | *npins = ret == 0 ? args.args[2] : RCAR_MAX_GPIO_PER_BANK; | 396 | *npins = ret == 0 ? args.args[2] : RCAR_MAX_GPIO_PER_BANK; |
| 405 | p->has_both_edge_trigger = info->has_both_edge_trigger; | 397 | p->has_both_edge_trigger = info->has_both_edge_trigger; |
| 406 | p->needs_clk = info->needs_clk; | ||
| 407 | 398 | ||
| 408 | if (*npins == 0 || *npins > RCAR_MAX_GPIO_PER_BANK) { | 399 | if (*npins == 0 || *npins > RCAR_MAX_GPIO_PER_BANK) { |
| 409 | dev_warn(&p->pdev->dev, | 400 | dev_warn(&p->pdev->dev, |
| @@ -440,16 +431,6 @@ static int gpio_rcar_probe(struct platform_device *pdev) · 30 unmodified lines | |||
| 440 | 431 | ||
| 441 | platform_set_drvdata(pdev, p); | 432 | platform_set_drvdata(pdev, p); |
| 442 | 433 | ||
| 443 | p->clk = devm_clk_get(dev, NULL); | ||
| 444 | if (IS_ERR(p->clk)) { | ||
| 445 | if (p->needs_clk) { | ||
| 446 | dev_err(dev, "unable to get clock\n"); | ||
| 447 | ret = PTR_ERR(p->clk); | ||
| 448 | goto err0; | ||
| 449 | } | ||
| 450 | p->clk = NULL; | ||
| 451 | } | ||
| 452 | |||
| 453 | pm_runtime_enable(dev); | 434 | pm_runtime_enable(dev); |
| 454 | 435 | ||
| 455 | irq = platform_get_resource(pdev, IORESOURCE_IRQ, 0); | 436 | irq = platform_get_resource(pdev, IORESOURCE_IRQ, 0); |
| @@ -531,11 +512,24 @@ static int gpio_rcar_remove(struct platform_device *pdev) · 75 unmodified lines | |||
| 531 | return 0; | 512 | return 0; |
| 532 | } | 513 | } |
| 533 | 514 | ||
| 515 | static int __maybe_unused gpio_rcar_suspend(struct device *dev) | ||
| 516 | { | ||
| 517 | struct gpio_rcar_priv *p = dev_get_drvdata(dev); | ||
| 518 | |||
| 519 | if (atomic_read(&p->wakeup_path)) | ||
| 520 | device_set_wakeup_path(dev); | ||
| 521 | |||
| 522 | return 0; | ||
| 523 | } | ||
| 524 | |||
| 525 | static SIMPLE_DEV_PM_OPS(gpio_rcar_pm_ops, gpio_rcar_suspend, NULL); | ||
| 526 | |||
| 534 | static struct platform_driver gpio_rcar_device_driver = { | 527 | static struct platform_driver gpio_rcar_device_driver = { |
| 535 | .probe = gpio_rcar_probe, | 528 | .probe = gpio_rcar_probe, |
| 536 | .remove = gpio_rcar_remove, | 529 | .remove = gpio_rcar_remove, |
| 537 | .driver = { | 530 | .driver = { |
| 538 | .name = "gpio_rcar", | 531 | .name = "gpio_rcar", |
| 532 | .pm = &gpio_rcar_pm_ops, | ||
| 539 | .of_match_table = of_match_ptr(gpio_rcar_of_table), | 533 | .of_match_table = of_match_ptr(gpio_rcar_of_table), |
| 540 | } | 534 | } |
| 541 | }; | 535 | }; |
| @@ -241,6 +241,19 @@ struct gpio_desc *of_find_gpio(struct device *dev, const char *con_id, · 240 unmodified lines | |||
| 241 | 241 | ||
| 242 | desc = of_get_named_gpiod_flags(dev->of_node, prop_name, idx, | 242 | desc = of_get_named_gpiod_flags(dev->of_node, prop_name, idx, |
| 243 | &of_flags); | 243 | &of_flags); |
| 244 | /* | ||
| 245 | * -EPROBE_DEFER in our case means that we found a | ||
| 246 | * valid GPIO property, but no controller has been | ||
| 247 | * registered so far. | ||
| 248 | * | ||
| 249 | * This means we don't need to look any further for | ||
| 250 | * alternate name conventions, and we should really | ||
| 251 | * preserve the return code for our user to be able to | ||
| 252 | * retry probing later. | ||
| 253 | */ | ||
| 254 | if (IS_ERR(desc) && PTR_ERR(desc) == -EPROBE_DEFER) | ||
| 255 | return desc; | ||
| 256 | |||
| 244 | if (!IS_ERR(desc) || (PTR_ERR(desc) != -ENOENT)) | 257 | if (!IS_ERR(desc) || (PTR_ERR(desc) != -ENOENT)) |
| 245 | break; | 258 | break; |
| 246 | } | 259 | } |
| @@ -250,7 +263,7 @@ struct gpio_desc *of_find_gpio(struct device *dev, const char *con_id, · 3 unmodified lines | |||
| 250 | desc = of_find_spi_gpio(dev, con_id, &of_flags); | 263 | desc = of_find_spi_gpio(dev, con_id, &of_flags); |
| 251 | 264 | ||
| 252 | /* Special handling for regulator GPIOs if used */ | 265 | /* Special handling for regulator GPIOs if used */ |
| 253 | if (IS_ERR(desc)) | 266 | if (IS_ERR(desc) && PTR_ERR(desc) != -EPROBE_DEFER) |
| 254 | desc = of_find_regulator_gpio(dev, con_id, &of_flags); | 267 | desc = of_find_regulator_gpio(dev, con_id, &of_flags); |
| 255 | 268 | ||
| 256 | if (IS_ERR(desc)) | 269 | if (IS_ERR(desc)) |
| @@ -1156,7 +1156,7 @@ static inline void amdgpu_set_ib_value(struct amdgpu_cs_parser *p, · 1155 unmodified lines | |||
| 1156 | /* | 1156 | /* |
| 1157 | * Writeback | 1157 | * Writeback |
| 1158 | */ | 1158 | */ |
| 1159 | #define AMDGPU_MAX_WB 512 /* Reserve at most 512 WB slots for amdgpu-owned rings. */ | 1159 | #define AMDGPU_MAX_WB 128 /* Reserve at most 128 WB slots for amdgpu-owned rings. */ |
| 1160 | 1160 | ||
| 1161 | struct amdgpu_wb { | 1161 | struct amdgpu_wb { |
| 1162 | struct amdgpu_bo *wb_obj; | 1162 | struct amdgpu_bo *wb_obj; |
| @@ -540,6 +540,9 @@ int amdgpu_acpi_pcie_performance_request(struct amdgpu_device *adev, · 539 unmodified lines | |||
| 540 | size_t size; | 540 | size_t size; |
| 541 | u32 retry = 3; | 541 | u32 retry = 3; |
| 542 | 542 | ||
| 543 | if (amdgpu_acpi_pcie_notify_device_ready(adev)) | ||
| 544 | return -EINVAL; | ||
| 545 | |||
| 543 | /* Get the device handle */ | 546 | /* Get the device handle */ |
| 544 | handle = ACPI_HANDLE(&adev->pdev->dev); | 547 | handle = ACPI_HANDLE(&adev->pdev->dev); |
| 545 | if (!handle) | 548 | if (!handle) |
| @@ -568,6 +568,7 @@ static const struct amdgpu_px_quirk amdgpu_px_quirk_list[] = { · 567 unmodified lines | |||
| 568 | /* HG _PR3 doesn't seem to work on this A+A weston board */ | 568 | /* HG _PR3 doesn't seem to work on this A+A weston board */ |
| 569 | { 0x1002, 0x6900, 0x1002, 0x0124, AMDGPU_PX_QUIRK_FORCE_ATPX }, | 569 | { 0x1002, 0x6900, 0x1002, 0x0124, AMDGPU_PX_QUIRK_FORCE_ATPX }, |
| 570 | { 0x1002, 0x6900, 0x1028, 0x0812, AMDGPU_PX_QUIRK_FORCE_ATPX }, | 570 | { 0x1002, 0x6900, 0x1028, 0x0812, AMDGPU_PX_QUIRK_FORCE_ATPX }, |
| 571 | { 0x1002, 0x6900, 0x1028, 0x0813, AMDGPU_PX_QUIRK_FORCE_ATPX }, | ||
| 571 | { 0, 0, 0, 0, 0 }, | 572 | { 0, 0, 0, 0, 0 }, |
| 572 | }; | 573 | }; |
| 573 | 574 | ||
| @@ -736,9 +736,11 @@ amdgpu_connector_lvds_detect(struct drm_connector *connector, bool force) · 735 unmodified lines | |||
| 736 | enum drm_connector_status ret = connector_status_disconnected; | 736 | enum drm_connector_status ret = connector_status_disconnected; |
| 737 | int r; | 737 | int r; |
| 738 | 738 | ||
| 739 | r = pm_runtime_get_sync(connector->dev->dev); | 739 | if (!drm_kms_helper_is_poll_worker()) { |
| 740 | if (r < 0) | 740 | r = pm_runtime_get_sync(connector->dev->dev); |
| 741 | return connector_status_disconnected; | 741 | if (r < 0) |
| 742 | return connector_status_disconnected; | ||
| 743 | } | ||
| 742 | 744 | ||
| 743 | if (encoder) { | 745 | if (encoder) { |
| 744 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); | 746 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); |
| @@ -757,8 +759,12 @@ amdgpu_connector_lvds_detect(struct drm_connector *connector, bool force) · 12 unmodified lines | |||
| 757 | /* check acpi lid status ??? */ | 759 | /* check acpi lid status ??? */ |
| 758 | 760 | ||
| 759 | amdgpu_connector_update_scratch_regs(connector, ret); | 761 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 760 | pm_runtime_mark_last_busy(connector->dev->dev); | 762 | |
| 761 | pm_runtime_put_autosuspend(connector->dev->dev); | 763 | if (!drm_kms_helper_is_poll_worker()) { |
| 764 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 765 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 766 | } | ||
| 767 | |||
| 762 | return ret; | 768 | return ret; |
| 763 | } | 769 | } |
| 764 | 770 | ||
| @@ -868,9 +874,11 @@ amdgpu_connector_vga_detect(struct drm_connector *connector, bool force) · 103 unmodified lines | |||
| 868 | enum drm_connector_status ret = connector_status_disconnected; | 874 | enum drm_connector_status ret = connector_status_disconnected; |
| 869 | int r; | 875 | int r; |
| 870 | 876 | ||
| 871 | r = pm_runtime_get_sync(connector->dev->dev); | 877 | if (!drm_kms_helper_is_poll_worker()) { |
| 872 | if (r < 0) | 878 | r = pm_runtime_get_sync(connector->dev->dev); |
| 873 | return connector_status_disconnected; | 879 | if (r < 0) |
| 880 | return connector_status_disconnected; | ||
| 881 | } | ||
| 874 | 882 | ||
| 875 | encoder = amdgpu_connector_best_single_encoder(connector); | 883 | encoder = amdgpu_connector_best_single_encoder(connector); |
| 876 | if (!encoder) | 884 | if (!encoder) |
| @@ -924,8 +932,10 @@ amdgpu_connector_vga_detect(struct drm_connector *connector, bool force) · 47 unmodified lines | |||
| 924 | amdgpu_connector_update_scratch_regs(connector, ret); | 932 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 925 | 933 | ||
| 926 | out: | 934 | out: |
| 927 | pm_runtime_mark_last_busy(connector->dev->dev); | 935 | if (!drm_kms_helper_is_poll_worker()) { |
| 928 | pm_runtime_put_autosuspend(connector->dev->dev); | 936 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 937 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 938 | } | ||
| 929 | 939 | ||
| 930 | return ret; | 940 | return ret; |
| 931 | } | 941 | } |
| @@ -988,9 +998,11 @@ amdgpu_connector_dvi_detect(struct drm_connector *connector, bool force) · 56 unmodified lines | |||
| 988 | enum drm_connector_status ret = connector_status_disconnected; | 998 | enum drm_connector_status ret = connector_status_disconnected; |
| 989 | bool dret = false, broken_edid = false; | 999 | bool dret = false, broken_edid = false; |
| 990 | 1000 | ||
| 991 | r = pm_runtime_get_sync(connector->dev->dev); | 1001 | if (!drm_kms_helper_is_poll_worker()) { |
| 992 | if (r < 0) | 1002 | r = pm_runtime_get_sync(connector->dev->dev); |
| 993 | return connector_status_disconnected; | 1003 | if (r < 0) |
| 1004 | return connector_status_disconnected; | ||
| 1005 | } | ||
| 994 | 1006 | ||
| 995 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { | 1007 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { |
| 996 | ret = connector->status; | 1008 | ret = connector->status; |
| @@ -1115,8 +1127,10 @@ out: · 118 unmodified lines | |||
| 1115 | amdgpu_connector_update_scratch_regs(connector, ret); | 1127 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 1116 | 1128 | ||
| 1117 | exit: | 1129 | exit: |
| 1118 | pm_runtime_mark_last_busy(connector->dev->dev); | 1130 | if (!drm_kms_helper_is_poll_worker()) { |
| 1119 | pm_runtime_put_autosuspend(connector->dev->dev); | 1131 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1132 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1133 | } | ||
| 1120 | 1134 | ||
| 1121 | return ret; | 1135 | return ret; |
| 1122 | } | 1136 | } |
| @@ -1359,9 +1373,11 @@ amdgpu_connector_dp_detect(struct drm_connector *connector, bool force) · 236 unmodified lines | |||
| 1359 | struct drm_encoder *encoder = amdgpu_connector_best_single_encoder(connector); | 1373 | struct drm_encoder *encoder = amdgpu_connector_best_single_encoder(connector); |
| 1360 | int r; | 1374 | int r; |
| 1361 | 1375 | ||
| 1362 | r = pm_runtime_get_sync(connector->dev->dev); | 1376 | if (!drm_kms_helper_is_poll_worker()) { |
| 1363 | if (r < 0) | 1377 | r = pm_runtime_get_sync(connector->dev->dev); |
| 1364 | return connector_status_disconnected; | 1378 | if (r < 0) |
| 1379 | return connector_status_disconnected; | ||
| 1380 | } | ||
| 1365 | 1381 | ||
| 1366 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { | 1382 | if (!force && amdgpu_connector_check_hpd_status_unchanged(connector)) { |
| 1367 | ret = connector->status; | 1383 | ret = connector->status; |
| @@ -1429,8 +1445,10 @@ amdgpu_connector_dp_detect(struct drm_connector *connector, bool force) · 61 unmodified lines | |||
| 1429 | 1445 | ||
| 1430 | amdgpu_connector_update_scratch_regs(connector, ret); | 1446 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 1431 | out: | 1447 | out: |
| 1432 | pm_runtime_mark_last_busy(connector->dev->dev); | 1448 | if (!drm_kms_helper_is_poll_worker()) { |
| 1433 | pm_runtime_put_autosuspend(connector->dev->dev); | 1449 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1450 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1451 | } | ||
| 1434 | 1452 | ||
| 1435 | return ret; | 1453 | return ret; |
| 1436 | } | 1454 | } |
| @@ -492,7 +492,7 @@ static int amdgpu_device_wb_init(struct amdgpu_device *adev) · 491 unmodified lines | |||
| 492 | memset(&adev->wb.used, 0, sizeof(adev->wb.used)); | 492 | memset(&adev->wb.used, 0, sizeof(adev->wb.used)); |
| 493 | 493 | ||
| 494 | /* clear wb memory */ | 494 | /* clear wb memory */ |
| 495 | memset((char *)adev->wb.wb, 0, AMDGPU_MAX_WB * sizeof(uint32_t)); | 495 | memset((char *)adev->wb.wb, 0, AMDGPU_MAX_WB * sizeof(uint32_t) * 8); |
| 496 | } | 496 | } |
| 497 | 497 | ||
| 498 | return 0; | 498 | return 0; |
| @@ -530,8 +530,9 @@ int amdgpu_device_wb_get(struct amdgpu_device *adev, u32 *wb) · 31 unmodified lines | |||
| 530 | */ | 530 | */ |
| 531 | void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb) | 531 | void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb) |
| 532 | { | 532 | { |
| 533 | wb >>= 3; | ||
| 533 | if (wb < adev->wb.num_wb) | 534 | if (wb < adev->wb.num_wb) |
| 534 | __clear_bit(wb >> 3, adev->wb.used); | 535 | __clear_bit(wb, adev->wb.used); |
| 535 | } | 536 | } |
| 536 | 537 | ||
| 537 | /** | 538 | /** |
| @@ -1455,11 +1456,6 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev) · 917 unmodified lines | |||
| 1455 | for (i = adev->num_ip_blocks - 1; i >= 0; i--) { | 1456 | for (i = adev->num_ip_blocks - 1; i >= 0; i--) { |
| 1456 | if (!adev->ip_blocks[i].status.hw) | 1457 | if (!adev->ip_blocks[i].status.hw) |
| 1457 | continue; | 1458 | continue; |
| 1458 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC) { | ||
| 1459 | amdgpu_free_static_csa(adev); | ||
| 1460 | amdgpu_device_wb_fini(adev); | ||
| 1461 | amdgpu_device_vram_scratch_fini(adev); | ||
| 1462 | } | ||
| 1463 | 1459 | ||
| 1464 | if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_UVD && | 1460 | if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_UVD && |
| 1465 | adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCE) { | 1461 | adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCE) { |
| @@ -1486,6 +1482,13 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev) · 20 unmodified lines | |||
| 1486 | for (i = adev->num_ip_blocks - 1; i >= 0; i--) { | 1482 | for (i = adev->num_ip_blocks - 1; i >= 0; i--) { |
| 1487 | if (!adev->ip_blocks[i].status.sw) | 1483 | if (!adev->ip_blocks[i].status.sw) |
| 1488 | continue; | 1484 | continue; |
| 1485 | |||
| 1486 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC) { | ||
| 1487 | amdgpu_free_static_csa(adev); | ||
| 1488 | amdgpu_device_wb_fini(adev); | ||
| 1489 | amdgpu_device_vram_scratch_fini(adev); | ||
| 1490 | } | ||
| 1491 | |||
| 1489 | r = adev->ip_blocks[i].version->funcs->sw_fini((void *)adev); | 1492 | r = adev->ip_blocks[i].version->funcs->sw_fini((void *)adev); |
| 1490 | /* XXX handle errors */ | 1493 | /* XXX handle errors */ |
| 1491 | if (r) { | 1494 | if (r) { |
| @@ -2284,14 +2287,6 @@ int amdgpu_device_resume(struct drm_device *dev, bool resume, bool fbcon) · 792 unmodified lines | |||
| 2284 | drm_helper_connector_dpms(connector, DRM_MODE_DPMS_ON); | 2287 | drm_helper_connector_dpms(connector, DRM_MODE_DPMS_ON); |
| 2285 | } | 2288 | } |
| 2286 | drm_modeset_unlock_all(dev); | 2289 | drm_modeset_unlock_all(dev); |
| 2287 | } else { | ||
| 2288 | /* | ||
| 2289 | * There is no equivalent atomic helper to turn on | ||
| 2290 | * display, so we defined our own function for this, | ||
| 2291 | * once suspend resume is supported by the atomic | ||
| 2292 | * framework this will be reworked | ||
| 2293 | */ | ||
| 2294 | amdgpu_dm_display_resume(adev); | ||
| 2295 | } | 2290 | } |
| 2296 | } | 2291 | } |
| 2297 | 2292 | ||
| @@ -2726,7 +2721,6 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 428 unmodified lines | |||
| 2726 | if (amdgpu_device_has_dc_support(adev)) { | 2721 | if (amdgpu_device_has_dc_support(adev)) { |
| 2727 | if (drm_atomic_helper_resume(adev->ddev, state)) | 2722 | if (drm_atomic_helper_resume(adev->ddev, state)) |
| 2728 | dev_info(adev->dev, "drm resume failed:%d\n", r); | 2723 | dev_info(adev->dev, "drm resume failed:%d\n", r); |
| 2729 | amdgpu_dm_display_resume(adev); | ||
| 2730 | } else { | 2724 | } else { |
| 2731 | drm_helper_resume_force_mode(adev->ddev); | 2725 | drm_helper_resume_force_mode(adev->ddev); |
| 2732 | } | 2726 | } |
| @@ -75,7 +75,7 @@ static int amdgpu_gtt_mgr_init(struct ttm_mem_type_manager *man, · 74 unmodified lines | |||
| 75 | static int amdgpu_gtt_mgr_fini(struct ttm_mem_type_manager *man) | 75 | static int amdgpu_gtt_mgr_fini(struct ttm_mem_type_manager *man) |
| 76 | { | 76 | { |
| 77 | struct amdgpu_gtt_mgr *mgr = man->priv; | 77 | struct amdgpu_gtt_mgr *mgr = man->priv; |
| 78 | 78 | spin_lock(&mgr->lock); | |
| 79 | drm_mm_takedown(&mgr->mm); | 79 | drm_mm_takedown(&mgr->mm); |
| 80 | spin_unlock(&mgr->lock); | 80 | spin_unlock(&mgr->lock); |
| 81 | kfree(mgr); | 81 | kfree(mgr); |
| @@ -257,7 +257,8 @@ int amdgpu_irq_init(struct amdgpu_device *adev) · 256 unmodified lines | |||
| 257 | r = drm_irq_install(adev->ddev, adev->ddev->pdev->irq); | 257 | r = drm_irq_install(adev->ddev, adev->ddev->pdev->irq); |
| 258 | if (r) { | 258 | if (r) { |
| 259 | adev->irq.installed = false; | 259 | adev->irq.installed = false; |
| 260 | flush_work(&adev->hotplug_work); | 260 | if (!amdgpu_device_has_dc_support(adev)) |
| 261 | flush_work(&adev->hotplug_work); | ||
| 261 | cancel_work_sync(&adev->reset_work); | 262 | cancel_work_sync(&adev->reset_work); |
| 262 | return r; | 263 | return r; |
| 263 | } | 264 | } |
| @@ -282,7 +283,8 @@ void amdgpu_irq_fini(struct amdgpu_device *adev) · 18 unmodified lines | |||
| 282 | adev->irq.installed = false; | 283 | adev->irq.installed = false; |
| 283 | if (adev->irq.msi_enabled) | 284 | if (adev->irq.msi_enabled) |
| 284 | pci_disable_msi(adev->pdev); | 285 | pci_disable_msi(adev->pdev); |
| 285 | flush_work(&adev->hotplug_work); | 286 | if (!amdgpu_device_has_dc_support(adev)) |
| 287 | flush_work(&adev->hotplug_work); | ||
| 286 | cancel_work_sync(&adev->reset_work); | 288 | cancel_work_sync(&adev->reset_work); |
| 287 | } | 289 | } |
| 288 | 290 | ||
| @@ -481,7 +481,7 @@ static ssize_t amdgpu_debugfs_ring_read(struct file *f, char __user *buf, · 480 unmodified lines | |||
| 481 | result = 0; | 481 | result = 0; |
| 482 | 482 | ||
| 483 | if (*pos < 12) { | 483 | if (*pos < 12) { |
| 484 | early[0] = amdgpu_ring_get_rptr(ring); | 484 | early[0] = amdgpu_ring_get_rptr(ring) & ring->buf_mask; |
| 485 | early[1] = amdgpu_ring_get_wptr(ring) & ring->buf_mask; | 485 | early[1] = amdgpu_ring_get_wptr(ring) & ring->buf_mask; |
| 486 | early[2] = ring->wptr & ring->buf_mask; | 486 | early[2] = ring->wptr & ring->buf_mask; |
| 487 | for (i = *pos / 4; i < 3 && size; i++) { | 487 | for (i = *pos / 4; i < 3 && size; i++) { |
| @@ -299,12 +299,15 @@ int amdgpu_uvd_suspend(struct amdgpu_device *adev) · 298 unmodified lines | |||
| 299 | 299 | ||
| 300 | cancel_delayed_work_sync(&adev->uvd.idle_work); | 300 | cancel_delayed_work_sync(&adev->uvd.idle_work); |
| 301 | 301 | ||
| 302 | for (i = 0; i < adev->uvd.max_handles; ++i) | 302 | /* only valid for physical mode */ |
| 303 | if (atomic_read(&adev->uvd.handles[i])) | 303 | if (adev->asic_type < CHIP_POLARIS10) { |
| 304 | break; | 304 | for (i = 0; i < adev->uvd.max_handles; ++i) |
| 305 | if (atomic_read(&adev->uvd.handles[i])) | ||
| 306 | break; | ||
| 305 | 307 | ||
| 306 | if (i == AMDGPU_MAX_UVD_HANDLES) | 308 | if (i == adev->uvd.max_handles) |
| 307 | return 0; | 309 | return 0; |
| 310 | } | ||
| 308 | 311 | ||
| 309 | size = amdgpu_bo_size(adev->uvd.vcpu_bo); | 312 | size = amdgpu_bo_size(adev->uvd.vcpu_bo); |
| 310 | ptr = adev->uvd.cpu_addr; | 313 | ptr = adev->uvd.cpu_addr; |
| @@ -3093,7 +3093,7 @@ static int dce_v6_0_hpd_irq(struct amdgpu_device *adev, · 3092 unmodified lines | |||
| 3093 | tmp |= DC_HPD1_INT_CONTROL__DC_HPD1_INT_ACK_MASK; | 3093 | tmp |= DC_HPD1_INT_CONTROL__DC_HPD1_INT_ACK_MASK; |
| 3094 | WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], tmp); | 3094 | WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], tmp); |
| 3095 | schedule_work(&adev->hotplug_work); | 3095 | schedule_work(&adev->hotplug_work); |
| 3096 | DRM_INFO("IH: HPD%d\n", hpd + 1); | 3096 | DRM_DEBUG("IH: HPD%d\n", hpd + 1); |
| 3097 | } | 3097 | } |
| 3098 | 3098 | ||
| 3099 | return 0; | 3099 | return 0; |
| @@ -4384,34 +4384,8 @@ static void gfx_v7_0_gpu_early_init(struct amdgpu_device *adev) · 4383 unmodified lines | |||
| 4384 | case CHIP_KAVERI: | 4384 | case CHIP_KAVERI: |
| 4385 | adev->gfx.config.max_shader_engines = 1; | 4385 | adev->gfx.config.max_shader_engines = 1; |
| 4386 | adev->gfx.config.max_tile_pipes = 4; | 4386 | adev->gfx.config.max_tile_pipes = 4; |
| 4387 | if ((adev->pdev->device == 0x1304) || | 4387 | adev->gfx.config.max_cu_per_sh = 8; |
| 4388 | (adev->pdev->device == 0x1305) || | 4388 | adev->gfx.config.max_backends_per_se = 2; |
| 4389 | (adev->pdev->device == 0x130C) || | ||
| 4390 | (adev->pdev->device == 0x130F) || | ||
| 4391 | (adev->pdev->device == 0x1310) || | ||
| 4392 | (adev->pdev->device == 0x1311) || | ||
| 4393 | (adev->pdev->device == 0x131C)) { | ||
| 4394 | adev->gfx.config.max_cu_per_sh = 8; | ||
| 4395 | adev->gfx.config.max_backends_per_se = 2; | ||
| 4396 | } else if ((adev->pdev->device == 0x1309) || | ||
| 4397 | (adev->pdev->device == 0x130A) || | ||
| 4398 | (adev->pdev->device == 0x130D) || | ||
| 4399 | (adev->pdev->device == 0x1313) || | ||
| 4400 | (adev->pdev->device == 0x131D)) { | ||
| 4401 | adev->gfx.config.max_cu_per_sh = 6; | ||
| 4402 | adev->gfx.config.max_backends_per_se = 2; | ||
| 4403 | } else if ((adev->pdev->device == 0x1306) || | ||
| 4404 | (adev->pdev->device == 0x1307) || | ||
| 4405 | (adev->pdev->device == 0x130B) || | ||
| 4406 | (adev->pdev->device == 0x130E) || | ||
| 4407 | (adev->pdev->device == 0x1315) || | ||
| 4408 | (adev->pdev->device == 0x131B)) { | ||
| 4409 | adev->gfx.config.max_cu_per_sh = 4; | ||
| 4410 | adev->gfx.config.max_backends_per_se = 1; | ||
| 4411 | } else { | ||
| 4412 | adev->gfx.config.max_cu_per_sh = 3; | ||
| 4413 | adev->gfx.config.max_backends_per_se = 1; | ||
| 4414 | } | ||
| 4415 | adev->gfx.config.max_sh_per_se = 1; | 4389 | adev->gfx.config.max_sh_per_se = 1; |
| 4416 | adev->gfx.config.max_texture_channel_caches = 4; | 4390 | adev->gfx.config.max_texture_channel_caches = 4; |
| 4417 | adev->gfx.config.max_gprs = 256; | 4391 | adev->gfx.config.max_gprs = 256; |
| @@ -634,7 +634,7 @@ static int gmc_v9_0_late_init(void *handle) · 633 unmodified lines | |||
| 634 | for(i = 0; i < AMDGPU_MAX_VMHUBS; ++i) | 634 | for(i = 0; i < AMDGPU_MAX_VMHUBS; ++i) |
| 635 | BUG_ON(vm_inv_eng[i] > 16); | 635 | BUG_ON(vm_inv_eng[i] > 16); |
| 636 | 636 | ||
| 637 | if (adev->asic_type == CHIP_VEGA10) { | 637 | if (adev->asic_type == CHIP_VEGA10 && !amdgpu_sriov_vf(adev)) { |
| 638 | r = gmc_v9_0_ecc_available(adev); | 638 | r = gmc_v9_0_ecc_available(adev); |
| 639 | if (r == 1) { | 639 | if (r == 1) { |
| 640 | DRM_INFO("ECC is active.\n"); | 640 | DRM_INFO("ECC is active.\n"); |
| @@ -682,7 +682,10 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev) · 41 unmodified lines | |||
| 682 | adev->mc.vram_width = amdgpu_atomfirmware_get_vram_width(adev); | 682 | adev->mc.vram_width = amdgpu_atomfirmware_get_vram_width(adev); |
| 683 | if (!adev->mc.vram_width) { | 683 | if (!adev->mc.vram_width) { |
| 684 | /* hbm memory channel size */ | 684 | /* hbm memory channel size */ |
| 685 | chansize = 128; | 685 | if (adev->flags & AMD_IS_APU) |
| 686 | chansize = 64; | ||
| 687 | else | ||
| 688 | chansize = 128; | ||
| 686 | 689 | ||
| 687 | tmp = RREG32_SOC15(DF, 0, mmDF_CS_AON0_DramBaseAddress0); | 690 | tmp = RREG32_SOC15(DF, 0, mmDF_CS_AON0_DramBaseAddress0); |
| 688 | tmp &= DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK; | 691 | tmp &= DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK; |
| @@ -238,31 +238,27 @@ static uint64_t sdma_v4_0_ring_get_rptr(struct amdgpu_ring *ring) · 237 unmodified lines | |||
| 238 | static uint64_t sdma_v4_0_ring_get_wptr(struct amdgpu_ring *ring) | 238 | static uint64_t sdma_v4_0_ring_get_wptr(struct amdgpu_ring *ring) |
| 239 | { | 239 | { |
| 240 | struct amdgpu_device *adev = ring->adev; | 240 | struct amdgpu_device *adev = ring->adev; |
| 241 | u64 *wptr = NULL; | 241 | u64 wptr; |
| 242 | uint64_t local_wptr = 0; | ||
| 243 | 242 | ||
| 244 | if (ring->use_doorbell) { | 243 | if (ring->use_doorbell) { |
| 245 | /* XXX check if swapping is necessary on BE */ | 244 | /* XXX check if swapping is necessary on BE */ |
| 246 | wptr = ((u64 *)&adev->wb.wb[ring->wptr_offs]); | 245 | wptr = READ_ONCE(*((u64 *)&adev->wb.wb[ring->wptr_offs])); |
| 247 | DRM_DEBUG("wptr/doorbell before shift == 0x%016llx\n", *wptr); | 246 | DRM_DEBUG("wptr/doorbell before shift == 0x%016llx\n", wptr); |
| 248 | *wptr = (*wptr) >> 2; | ||
| 249 | DRM_DEBUG("wptr/doorbell after shift == 0x%016llx\n", *wptr); | ||
| 250 | } else { | 247 | } else { |
| 251 | u32 lowbit, highbit; | 248 | u32 lowbit, highbit; |
| 252 | int me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1; | 249 | int me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1; |
| 253 | 250 | ||
| 254 | wptr = &local_wptr; | ||
| 255 | lowbit = RREG32(sdma_v4_0_get_reg_offset(adev, me, mmSDMA0_GFX_RB_WPTR)) >> 2; | 251 | lowbit = RREG32(sdma_v4_0_get_reg_offset(adev, me, mmSDMA0_GFX_RB_WPTR)) >> 2; |
| 256 | highbit = RREG32(sdma_v4_0_get_reg_offset(adev, me, mmSDMA0_GFX_RB_WPTR_HI)) >> 2; | 252 | highbit = RREG32(sdma_v4_0_get_reg_offset(adev, me, mmSDMA0_GFX_RB_WPTR_HI)) >> 2; |
| 257 | 253 | ||
| 258 | DRM_DEBUG("wptr [%i]high== 0x%08x low==0x%08x\n", | 254 | DRM_DEBUG("wptr [%i]high== 0x%08x low==0x%08x\n", |
| 259 | me, highbit, lowbit); | 255 | me, highbit, lowbit); |
| 260 | *wptr = highbit; | 256 | wptr = highbit; |
| 261 | *wptr = (*wptr) << 32; | 257 | wptr = wptr << 32; |
| 262 | *wptr |= lowbit; | 258 | wptr |= lowbit; |
| 263 | } | 259 | } |
| 264 | 260 | ||
| 265 | return *wptr; | 261 | return wptr >> 2; |
| 266 | } | 262 | } |
| 267 | 263 | ||
| 268 | /** | 264 | /** |
| @@ -31,6 +31,7 @@ · 30 unmodified lines | |||
| 31 | #include "amdgpu_uvd.h" | 31 | #include "amdgpu_uvd.h" |
| 32 | #include "amdgpu_vce.h" | 32 | #include "amdgpu_vce.h" |
| 33 | #include "atom.h" | 33 | #include "atom.h" |
| 34 | #include "amd_pcie.h" | ||
| 34 | #include "amdgpu_powerplay.h" | 35 | #include "amdgpu_powerplay.h" |
| 35 | #include "sid.h" | 36 | #include "sid.h" |
| 36 | #include "si_ih.h" | 37 | #include "si_ih.h" |
| @@ -1461,8 +1462,8 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 1424 unmodified lines | |||
| 1461 | { | 1462 | { |
| 1462 | struct pci_dev *root = adev->pdev->bus->self; | 1463 | struct pci_dev *root = adev->pdev->bus->self; |
| 1463 | int bridge_pos, gpu_pos; | 1464 | int bridge_pos, gpu_pos; |
| 1464 | u32 speed_cntl, mask, current_data_rate; | 1465 | u32 speed_cntl, current_data_rate; |
| 1465 | int ret, i; | 1466 | int i; |
| 1466 | u16 tmp16; | 1467 | u16 tmp16; |
| 1467 | 1468 | ||
| 1468 | if (pci_is_root_bus(adev->pdev->bus)) | 1469 | if (pci_is_root_bus(adev->pdev->bus)) |
| @@ -1474,23 +1475,20 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 5 unmodified lines | |||
| 1474 | if (adev->flags & AMD_IS_APU) | 1475 | if (adev->flags & AMD_IS_APU) |
| 1475 | return; | 1476 | return; |
| 1476 | 1477 | ||
| 1477 | ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask); | 1478 | if (!(adev->pm.pcie_gen_mask & (CAIL_PCIE_LINK_SPEED_SUPPORT_GEN2 | |
| 1478 | if (ret != 0) | 1479 | CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3))) |
| 1479 | return; | ||
| 1480 | |||
| 1481 | if (!(mask & (DRM_PCIE_SPEED_50 | DRM_PCIE_SPEED_80))) | ||
| 1482 | return; | 1480 | return; |
| 1483 | 1481 | ||
| 1484 | speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL); | 1482 | speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL); |
| 1485 | current_data_rate = (speed_cntl & LC_CURRENT_DATA_RATE_MASK) >> | 1483 | current_data_rate = (speed_cntl & LC_CURRENT_DATA_RATE_MASK) >> |
| 1486 | LC_CURRENT_DATA_RATE_SHIFT; | 1484 | LC_CURRENT_DATA_RATE_SHIFT; |
| 1487 | if (mask & DRM_PCIE_SPEED_80) { | 1485 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) { |
| 1488 | if (current_data_rate == 2) { | 1486 | if (current_data_rate == 2) { |
| 1489 | DRM_INFO("PCIE gen 3 link speeds already enabled\n"); | 1487 | DRM_INFO("PCIE gen 3 link speeds already enabled\n"); |
| 1490 | return; | 1488 | return; |
| 1491 | } | 1489 | } |
| 1492 | DRM_INFO("enabling PCIE gen 3 link speeds, disable with amdgpu.pcie_gen2=0\n"); | 1490 | DRM_INFO("enabling PCIE gen 3 link speeds, disable with amdgpu.pcie_gen2=0\n"); |
| 1493 | } else if (mask & DRM_PCIE_SPEED_50) { | 1491 | } else if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN2) { |
| 1494 | if (current_data_rate == 1) { | 1492 | if (current_data_rate == 1) { |
| 1495 | DRM_INFO("PCIE gen 2 link speeds already enabled\n"); | 1493 | DRM_INFO("PCIE gen 2 link speeds already enabled\n"); |
| 1496 | return; | 1494 | return; |
| @@ -1506,7 +1504,7 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 9 unmodified lines | |||
| 1506 | if (!gpu_pos) | 1504 | if (!gpu_pos) |
| 1507 | return; | 1505 | return; |
| 1508 | 1506 | ||
| 1509 | if (mask & DRM_PCIE_SPEED_80) { | 1507 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) { |
| 1510 | if (current_data_rate != 2) { | 1508 | if (current_data_rate != 2) { |
| 1511 | u16 bridge_cfg, gpu_cfg; | 1509 | u16 bridge_cfg, gpu_cfg; |
| 1512 | u16 bridge_cfg2, gpu_cfg2; | 1510 | u16 bridge_cfg2, gpu_cfg2; |
| @@ -1589,9 +1587,9 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 76 unmodified lines | |||
| 1589 | 1587 | ||
| 1590 | pci_read_config_word(adev->pdev, gpu_pos + PCI_EXP_LNKCTL2, &tmp16); | 1588 | pci_read_config_word(adev->pdev, gpu_pos + PCI_EXP_LNKCTL2, &tmp16); |
| 1591 | tmp16 &= ~0xf; | 1589 | tmp16 &= ~0xf; |
| 1592 | if (mask & DRM_PCIE_SPEED_80) | 1590 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) |
| 1593 | tmp16 |= 3; | 1591 | tmp16 |= 3; |
| 1594 | else if (mask & DRM_PCIE_SPEED_50) | 1592 | else if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN2) |
| 1595 | tmp16 |= 2; | 1593 | tmp16 |= 2; |
| 1596 | else | 1594 | else |
| 1597 | tmp16 |= 1; | 1595 | tmp16 |= 1; |
| @@ -26,6 +26,7 @@ · 25 unmodified lines | |||
| 26 | #include "amdgpu_pm.h" | 26 | #include "amdgpu_pm.h" |
| 27 | #include "amdgpu_dpm.h" | 27 | #include "amdgpu_dpm.h" |
| 28 | #include "amdgpu_atombios.h" | 28 | #include "amdgpu_atombios.h" |
| 29 | #include "amd_pcie.h" | ||
| 29 | #include "sid.h" | 30 | #include "sid.h" |
| 30 | #include "r600_dpm.h" | 31 | #include "r600_dpm.h" |
| 31 | #include "si_dpm.h" | 32 | #include "si_dpm.h" |
| @@ -3331,29 +3332,6 @@ static void btc_apply_voltage_delta_rules(struct amdgpu_device *adev, · 3299 unmodified lines | |||
| 3331 | } | 3332 | } |
| 3332 | } | 3333 | } |
| 3333 | 3334 | ||
| 3334 | static enum amdgpu_pcie_gen r600_get_pcie_gen_support(struct amdgpu_device *adev, | ||
| 3335 | u32 sys_mask, | ||
| 3336 | enum amdgpu_pcie_gen asic_gen, | ||
| 3337 | enum amdgpu_pcie_gen default_gen) | ||
| 3338 | { | ||
| 3339 | switch (asic_gen) { | ||
| 3340 | case AMDGPU_PCIE_GEN1: | ||
| 3341 | return AMDGPU_PCIE_GEN1; | ||
| 3342 | case AMDGPU_PCIE_GEN2: | ||
| 3343 | return AMDGPU_PCIE_GEN2; | ||
| 3344 | case AMDGPU_PCIE_GEN3: | ||
| 3345 | return AMDGPU_PCIE_GEN3; | ||
| 3346 | default: | ||
| 3347 | if ((sys_mask & DRM_PCIE_SPEED_80) && (default_gen == AMDGPU_PCIE_GEN3)) | ||
| 3348 | return AMDGPU_PCIE_GEN3; | ||
| 3349 | else if ((sys_mask & DRM_PCIE_SPEED_50) && (default_gen == AMDGPU_PCIE_GEN2)) | ||
| 3350 | return AMDGPU_PCIE_GEN2; | ||
| 3351 | else | ||
| 3352 | return AMDGPU_PCIE_GEN1; | ||
| 3353 | } | ||
| 3354 | return AMDGPU_PCIE_GEN1; | ||
| 3355 | } | ||
| 3356 | |||
| 3357 | static void r600_calculate_u_and_p(u32 i, u32 r_c, u32 p_b, | 3335 | static void r600_calculate_u_and_p(u32 i, u32 r_c, u32 p_b, |
| 3358 | u32 *p, u32 *u) | 3336 | u32 *p, u32 *u) |
| 3359 | { | 3337 | { |
| @@ -5028,10 +5006,11 @@ static int si_populate_smc_acpi_state(struct amdgpu_device *adev, · 1668 unmodified lines | |||
| 5028 | table->ACPIState.levels[0].vddc.index, | 5006 | table->ACPIState.levels[0].vddc.index, |
| 5029 | &table->ACPIState.levels[0].std_vddc); | 5007 | &table->ACPIState.levels[0].std_vddc); |
| 5030 | } | 5008 | } |
| 5031 | table->ACPIState.levels[0].gen2PCIE = (u8)r600_get_pcie_gen_support(adev, | 5009 | table->ACPIState.levels[0].gen2PCIE = |
| 5032 | si_pi->sys_pcie_mask, | 5010 | (u8)amdgpu_get_pcie_gen_support(adev, |
| 5033 | si_pi->boot_pcie_gen, | 5011 | si_pi->sys_pcie_mask, |
| 5034 | AMDGPU_PCIE_GEN1); | 5012 | si_pi->boot_pcie_gen, |
| 5013 | AMDGPU_PCIE_GEN1); | ||
| 5035 | 5014 | ||
| 5036 | if (si_pi->vddc_phase_shed_control) | 5015 | if (si_pi->vddc_phase_shed_control) |
| 5037 | si_populate_phase_shedding_value(adev, | 5016 | si_populate_phase_shedding_value(adev, |
| @@ -7168,10 +7147,10 @@ static void si_parse_pplib_clock_info(struct amdgpu_device *adev, · 2130 unmodified lines | |||
| 7168 | pl->vddc = le16_to_cpu(clock_info->si.usVDDC); | 7147 | pl->vddc = le16_to_cpu(clock_info->si.usVDDC); |
| 7169 | pl->vddci = le16_to_cpu(clock_info->si.usVDDCI); | 7148 | pl->vddci = le16_to_cpu(clock_info->si.usVDDCI); |
| 7170 | pl->flags = le32_to_cpu(clock_info->si.ulFlags); | 7149 | pl->flags = le32_to_cpu(clock_info->si.ulFlags); |
| 7171 | pl->pcie_gen = r600_get_pcie_gen_support(adev, | 7150 | pl->pcie_gen = amdgpu_get_pcie_gen_support(adev, |
| 7172 | si_pi->sys_pcie_mask, | 7151 | si_pi->sys_pcie_mask, |
| 7173 | si_pi->boot_pcie_gen, | 7152 | si_pi->boot_pcie_gen, |
| 7174 | clock_info->si.ucPCIEGen); | 7153 | clock_info->si.ucPCIEGen); |
| 7175 | 7154 | ||
| 7176 | /* patch up vddc if necessary */ | 7155 | /* patch up vddc if necessary */ |
| 7177 | ret = si_get_leakage_voltage_from_leakage_index(adev, pl->vddc, | 7156 | ret = si_get_leakage_voltage_from_leakage_index(adev, pl->vddc, |
| @@ -7326,7 +7305,6 @@ static int si_dpm_init(struct amdgpu_device *adev) · 148 unmodified lines | |||
| 7326 | struct si_power_info *si_pi; | 7305 | struct si_power_info *si_pi; |
| 7327 | struct atom_clock_dividers dividers; | 7306 | struct atom_clock_dividers dividers; |
| 7328 | int ret; | 7307 | int ret; |
| 7329 | u32 mask; | ||
| 7330 | 7308 | ||
| 7331 | si_pi = kzalloc(sizeof(struct si_power_info), GFP_KERNEL); | 7309 | si_pi = kzalloc(sizeof(struct si_power_info), GFP_KERNEL); |
| 7332 | if (si_pi == NULL) | 7310 | if (si_pi == NULL) |
| @@ -7336,11 +7314,9 @@ static int si_dpm_init(struct amdgpu_device *adev) · 3 unmodified lines | |||
| 7336 | eg_pi = &ni_pi->eg; | 7314 | eg_pi = &ni_pi->eg; |
| 7337 | pi = &eg_pi->rv7xx; | 7315 | pi = &eg_pi->rv7xx; |
| 7338 | 7316 | ||
| 7339 | ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask); | 7317 | si_pi->sys_pcie_mask = |
| 7340 | if (ret) | 7318 | (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_MASK) >> |
| 7341 | si_pi->sys_pcie_mask = 0; | 7319 | CAIL_PCIE_LINK_SPEED_SUPPORT_SHIFT; |
| 7342 | else | ||
| 7343 | si_pi->sys_pcie_mask = mask; | ||
| 7344 | si_pi->force_pcie_gen = AMDGPU_PCIE_GEN_INVALID; | 7320 | si_pi->force_pcie_gen = AMDGPU_PCIE_GEN_INVALID; |
| 7345 | si_pi->boot_pcie_gen = si_get_current_pcie_speed(adev); | 7321 | si_pi->boot_pcie_gen = si_get_current_pcie_speed(adev); |
| 7346 | 7322 | ||
| @@ -1618,7 +1618,7 @@ static const struct amdgpu_ring_funcs uvd_v6_0_enc_ring_vm_funcs = { · 1617 unmodified lines | |||
| 1618 | .set_wptr = uvd_v6_0_enc_ring_set_wptr, | 1618 | .set_wptr = uvd_v6_0_enc_ring_set_wptr, |
| 1619 | .emit_frame_size = | 1619 | .emit_frame_size = |
| 1620 | 4 + /* uvd_v6_0_enc_ring_emit_pipeline_sync */ | 1620 | 4 + /* uvd_v6_0_enc_ring_emit_pipeline_sync */ |
| 1621 | 6 + /* uvd_v6_0_enc_ring_emit_vm_flush */ | 1621 | 5 + /* uvd_v6_0_enc_ring_emit_vm_flush */ |
| 1622 | 5 + 5 + /* uvd_v6_0_enc_ring_emit_fence x2 vm fence */ | 1622 | 5 + 5 + /* uvd_v6_0_enc_ring_emit_fence x2 vm fence */ |
| 1623 | 1, /* uvd_v6_0_enc_ring_insert_end */ | 1623 | 1, /* uvd_v6_0_enc_ring_insert_end */ |
| 1624 | .emit_ib_size = 5, /* uvd_v6_0_enc_ring_emit_ib */ | 1624 | .emit_ib_size = 5, /* uvd_v6_0_enc_ring_emit_ib */ |
| @@ -629,11 +629,13 @@ static int dm_resume(void *handle) · 628 unmodified lines | |||
| 629 | { | 629 | { |
| 630 | struct amdgpu_device *adev = handle; | 630 | struct amdgpu_device *adev = handle; |
| 631 | struct amdgpu_display_manager *dm = &adev->dm; | 631 | struct amdgpu_display_manager *dm = &adev->dm; |
| 632 | int ret = 0; | ||
| 632 | 633 | ||
| 633 | /* power on hardware */ | 634 | /* power on hardware */ |
| 634 | dc_set_power_state(dm->dc, DC_ACPI_CM_POWER_STATE_D0); | 635 | dc_set_power_state(dm->dc, DC_ACPI_CM_POWER_STATE_D0); |
| 635 | 636 | ||
| 636 | return 0; | 637 | ret = amdgpu_dm_display_resume(adev); |
| 638 | return ret; | ||
| 637 | } | 639 | } |
| 638 | 640 | ||
| 639 | int amdgpu_dm_display_resume(struct amdgpu_device *adev) | 641 | int amdgpu_dm_display_resume(struct amdgpu_device *adev) |
| @@ -1035,6 +1037,10 @@ static void handle_hpd_rx_irq(void *param) · 395 unmodified lines | |||
| 1035 | !is_mst_root_connector) { | 1037 | !is_mst_root_connector) { |
| 1036 | /* Downstream Port status changed. */ | 1038 | /* Downstream Port status changed. */ |
| 1037 | if (dc_link_detect(dc_link, DETECT_REASON_HPDRX)) { | 1039 | if (dc_link_detect(dc_link, DETECT_REASON_HPDRX)) { |
| 1040 | |||
| 1041 | if (aconnector->fake_enable) | ||
| 1042 | aconnector->fake_enable = false; | ||
| 1043 | |||
| 1038 | amdgpu_dm_update_connector_after_detect(aconnector); | 1044 | amdgpu_dm_update_connector_after_detect(aconnector); |
| 1039 | 1045 | ||
| 1040 | 1046 | ||
| @@ -2010,30 +2016,32 @@ static void update_stream_scaling_settings(const struct drm_display_mode *mode, · 969 unmodified lines | |||
| 2010 | dst.width = stream->timing.h_addressable; | 2016 | dst.width = stream->timing.h_addressable; |
| 2011 | dst.height = stream->timing.v_addressable; | 2017 | dst.height = stream->timing.v_addressable; |
| 2012 | 2018 | ||
| 2013 | rmx_type = dm_state->scaling; | 2019 | if (dm_state) { |
| 2014 | if (rmx_type == RMX_ASPECT || rmx_type == RMX_OFF) { | 2020 | rmx_type = dm_state->scaling; |
| 2015 | if (src.width * dst.height < | 2021 | if (rmx_type == RMX_ASPECT || rmx_type == RMX_OFF) { |
| 2016 | src.height * dst.width) { | 2022 | if (src.width * dst.height < |
| 2017 | /* height needs less upscaling/more downscaling */ | 2023 | src.height * dst.width) { |
| 2018 | dst.width = src.width * | 2024 | /* height needs less upscaling/more downscaling */ |
| 2019 | dst.height / src.height; | 2025 | dst.width = src.width * |
| 2020 | } else { | 2026 | dst.height / src.height; |
| 2021 | /* width needs less upscaling/more downscaling */ | 2027 | } else { |
| 2022 | dst.height = src.height * | 2028 | /* width needs less upscaling/more downscaling */ |
| 2023 | dst.width / src.width; | 2029 | dst.height = src.height * |
| 2030 | dst.width / src.width; | ||
| 2031 | } | ||
| 2032 | } else if (rmx_type == RMX_CENTER) { | ||
| 2033 | dst = src; | ||
| 2024 | } | 2034 | } |
| 2025 | } else if (rmx_type == RMX_CENTER) { | ||
| 2026 | dst = src; | ||
| 2027 | } | ||
| 2028 | 2035 | ||
| 2029 | dst.x = (stream->timing.h_addressable - dst.width) / 2; | 2036 | dst.x = (stream->timing.h_addressable - dst.width) / 2; |
| 2030 | dst.y = (stream->timing.v_addressable - dst.height) / 2; | 2037 | dst.y = (stream->timing.v_addressable - dst.height) / 2; |
| 2031 | 2038 | ||
| 2032 | if (dm_state->underscan_enable) { | 2039 | if (dm_state->underscan_enable) { |
| 2033 | dst.x += dm_state->underscan_hborder / 2; | 2040 | dst.x += dm_state->underscan_hborder / 2; |
| 2034 | dst.y += dm_state->underscan_vborder / 2; | 2041 | dst.y += dm_state->underscan_vborder / 2; |
| 2035 | dst.width -= dm_state->underscan_hborder; | 2042 | dst.width -= dm_state->underscan_hborder; |
| 2036 | dst.height -= dm_state->underscan_vborder; | 2043 | dst.height -= dm_state->underscan_vborder; |
| 2044 | } | ||
| 2037 | } | 2045 | } |
| 2038 | 2046 | ||
| 2039 | stream->src = src; | 2047 | stream->src = src; |
| @@ -2358,12 +2366,7 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, · 318 unmodified lines | |||
| 2358 | 2366 | ||
| 2359 | if (aconnector == NULL) { | 2367 | if (aconnector == NULL) { |
| 2360 | DRM_ERROR("aconnector is NULL!\n"); | 2368 | DRM_ERROR("aconnector is NULL!\n"); |
| 2361 | goto drm_connector_null; | 2369 | return stream; |
| 2362 | } | ||
| 2363 | |||
| 2364 | if (dm_state == NULL) { | ||
| 2365 | DRM_ERROR("dm_state is NULL!\n"); | ||
| 2366 | goto dm_state_null; | ||
| 2367 | } | 2370 | } |
| 2368 | 2371 | ||
| 2369 | drm_connector = &aconnector->base; | 2372 | drm_connector = &aconnector->base; |
| @@ -2375,18 +2378,18 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, · 5 unmodified lines | |||
| 2375 | */ | 2378 | */ |
| 2376 | if (aconnector->mst_port) { | 2379 | if (aconnector->mst_port) { |
| 2377 | dm_dp_mst_dc_sink_create(drm_connector); | 2380 | dm_dp_mst_dc_sink_create(drm_connector); |
| 2378 | goto mst_dc_sink_create_done; | 2381 | return stream; |
| 2379 | } | 2382 | } |
| 2380 | 2383 | ||
| 2381 | if (create_fake_sink(aconnector)) | 2384 | if (create_fake_sink(aconnector)) |
| 2382 | goto stream_create_fail; | 2385 | return stream; |
| 2383 | } | 2386 | } |
| 2384 | 2387 | ||
| 2385 | stream = dc_create_stream_for_sink(aconnector->dc_sink); | 2388 | stream = dc_create_stream_for_sink(aconnector->dc_sink); |
| 2386 | 2389 | ||
| 2387 | if (stream == NULL) { | 2390 | if (stream == NULL) { |
| 2388 | DRM_ERROR("Failed to create stream for sink!\n"); | 2391 | DRM_ERROR("Failed to create stream for sink!\n"); |
| 2389 | goto stream_create_fail; | 2392 | return stream; |
| 2390 | } | 2393 | } |
| 2391 | 2394 | ||
| 2392 | list_for_each_entry(preferred_mode, &aconnector->base.modes, head) { | 2395 | list_for_each_entry(preferred_mode, &aconnector->base.modes, head) { |
| @@ -2412,9 +2415,12 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, · 19 unmodified lines | |||
| 2412 | } else { | 2415 | } else { |
| 2413 | decide_crtc_timing_for_drm_display_mode( | 2416 | decide_crtc_timing_for_drm_display_mode( |
| 2414 | &mode, preferred_mode, | 2417 | &mode, preferred_mode, |
| 2415 | dm_state->scaling != RMX_OFF); | 2418 | dm_state ? (dm_state->scaling != RMX_OFF) : false); |
| 2416 | } | 2419 | } |
| 2417 | 2420 | ||
| 2421 | if (!dm_state) | ||
| 2422 | drm_mode_set_crtcinfo(&mode, 0); | ||
| 2423 | |||
| 2418 | fill_stream_properties_from_drm_display_mode(stream, | 2424 | fill_stream_properties_from_drm_display_mode(stream, |
| 2419 | &mode, &aconnector->base); | 2425 | &mode, &aconnector->base); |
| 2420 | update_stream_scaling_settings(&mode, dm_state, stream); | 2426 | update_stream_scaling_settings(&mode, dm_state, stream); |
| @@ -2424,10 +2430,8 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, · 3 unmodified lines | |||
| 2424 | drm_connector, | 2430 | drm_connector, |
| 2425 | aconnector->dc_sink); | 2431 | aconnector->dc_sink); |
| 2426 | 2432 | ||
| 2427 | stream_create_fail: | 2433 | update_stream_signal(stream); |
| 2428 | dm_state_null: | 2434 | |
| 2429 | drm_connector_null: | ||
| 2430 | mst_dc_sink_create_done: | ||
| 2431 | return stream; | 2435 | return stream; |
| 2432 | } | 2436 | } |
| 2433 | 2437 | ||
| @@ -2495,6 +2499,27 @@ dm_crtc_duplicate_state(struct drm_crtc *crtc) · 61 unmodified lines | |||
| 2495 | return &state->base; | 2499 | return &state->base; |
| 2496 | } | 2500 | } |
| 2497 | 2501 | ||
| 2502 | |||
| 2503 | static inline int dm_set_vblank(struct drm_crtc *crtc, bool enable) | ||
| 2504 | { | ||
| 2505 | enum dc_irq_source irq_source; | ||
| 2506 | struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc); | ||
| 2507 | struct amdgpu_device *adev = crtc->dev->dev_private; | ||
| 2508 | |||
| 2509 | irq_source = IRQ_TYPE_VBLANK + acrtc->otg_inst; | ||
| 2510 | return dc_interrupt_set(adev->dm.dc, irq_source, enable) ? 0 : -EBUSY; | ||
| 2511 | } | ||
| 2512 | |||
| 2513 | static int dm_enable_vblank(struct drm_crtc *crtc) | ||
| 2514 | { | ||
| 2515 | return dm_set_vblank(crtc, true); | ||
| 2516 | } | ||
| 2517 | |||
| 2518 | static void dm_disable_vblank(struct drm_crtc *crtc) | ||
| 2519 | { | ||
| 2520 | dm_set_vblank(crtc, false); | ||
| 2521 | } | ||
| 2522 | |||
| 2498 | /* Implemented only the options currently availible for the driver */ | 2523 | /* Implemented only the options currently availible for the driver */ |
| 2499 | static const struct drm_crtc_funcs amdgpu_dm_crtc_funcs = { | 2524 | static const struct drm_crtc_funcs amdgpu_dm_crtc_funcs = { |
| 2500 | .reset = dm_crtc_reset_state, | 2525 | .reset = dm_crtc_reset_state, |
| @@ -2504,6 +2529,8 @@ static const struct drm_crtc_funcs amdgpu_dm_crtc_funcs = { · 3 unmodified lines | |||
| 2504 | .page_flip = drm_atomic_helper_page_flip, | 2529 | .page_flip = drm_atomic_helper_page_flip, |
| 2505 | .atomic_duplicate_state = dm_crtc_duplicate_state, | 2530 | .atomic_duplicate_state = dm_crtc_duplicate_state, |
| 2506 | .atomic_destroy_state = dm_crtc_destroy_state, | 2531 | .atomic_destroy_state = dm_crtc_destroy_state, |
| 2532 | .enable_vblank = dm_enable_vblank, | ||
| 2533 | .disable_vblank = dm_disable_vblank, | ||
| 2507 | }; | 2534 | }; |
| 2508 | 2535 | ||
| 2509 | static enum drm_connector_status | 2536 | static enum drm_connector_status |
| @@ -2798,7 +2825,7 @@ int amdgpu_dm_connector_mode_valid(struct drm_connector *connector, · 288 unmodified lines | |||
| 2798 | goto fail; | 2825 | goto fail; |
| 2799 | } | 2826 | } |
| 2800 | 2827 | ||
| 2801 | stream = dc_create_stream_for_sink(dc_sink); | 2828 | stream = create_stream_for_sink(aconnector, mode, NULL); |
| 2802 | if (stream == NULL) { | 2829 | if (stream == NULL) { |
| 2803 | DRM_ERROR("Failed to create stream for sink!\n"); | 2830 | DRM_ERROR("Failed to create stream for sink!\n"); |
| 2804 | goto fail; | 2831 | goto fail; |
| @@ -3058,6 +3085,9 @@ static int dm_plane_atomic_check(struct drm_plane *plane, · 253 unmodified lines | |||
| 3058 | if (!dm_plane_state->dc_state) | 3085 | if (!dm_plane_state->dc_state) |
| 3059 | return 0; | 3086 | return 0; |
| 3060 | 3087 | ||
| 3088 | if (!fill_rects_from_plane_state(state, dm_plane_state->dc_state)) | ||
| 3089 | return -EINVAL; | ||
| 3090 | |||
| 3061 | if (dc_validate_plane(dc, dm_plane_state->dc_state) == DC_OK) | 3091 | if (dc_validate_plane(dc, dm_plane_state->dc_state) == DC_OK) |
| 3062 | return 0; | 3092 | return 0; |
| 3063 | 3093 | ||
| @@ -4630,8 +4660,6 @@ static int dm_update_planes_state(struct dc *dc, · 1566 unmodified lines | |||
| 4630 | bool pflip_needed = !state->allow_modeset; | 4660 | bool pflip_needed = !state->allow_modeset; |
| 4631 | int ret = 0; | 4661 | int ret = 0; |
| 4632 | 4662 | ||
| 4633 | if (pflip_needed) | ||
| 4634 | return ret; | ||
| 4635 | 4663 | ||
| 4636 | /* Add new planes */ | 4664 | /* Add new planes */ |
| 4637 | for_each_oldnew_plane_in_state(state, plane, old_plane_state, new_plane_state, i) { | 4665 | for_each_oldnew_plane_in_state(state, plane, old_plane_state, new_plane_state, i) { |
| @@ -4646,6 +4674,8 @@ static int dm_update_planes_state(struct dc *dc, · 8 unmodified lines | |||
| 4646 | 4674 | ||
| 4647 | /* Remove any changed/removed planes */ | 4675 | /* Remove any changed/removed planes */ |
| 4648 | if (!enable) { | 4676 | if (!enable) { |
| 4677 | if (pflip_needed) | ||
| 4678 | continue; | ||
| 4649 | 4679 | ||
| 4650 | if (!old_plane_crtc) | 4680 | if (!old_plane_crtc) |
| 4651 | continue; | 4681 | continue; |
| @@ -4677,6 +4707,7 @@ static int dm_update_planes_state(struct dc *dc, · 25 unmodified lines | |||
| 4677 | *lock_and_validation_needed = true; | 4707 | *lock_and_validation_needed = true; |
| 4678 | 4708 | ||
| 4679 | } else { /* Add new planes */ | 4709 | } else { /* Add new planes */ |
| 4710 | struct dc_plane_state *dc_new_plane_state; | ||
| 4680 | 4711 | ||
| 4681 | if (drm_atomic_plane_disabling(plane->state, new_plane_state)) | 4712 | if (drm_atomic_plane_disabling(plane->state, new_plane_state)) |
| 4682 | continue; | 4713 | continue; |
| @@ -4690,38 +4721,50 @@ static int dm_update_planes_state(struct dc *dc, · 7 unmodified lines | |||
| 4690 | if (!dm_new_crtc_state->stream) | 4721 | if (!dm_new_crtc_state->stream) |
| 4691 | continue; | 4722 | continue; |
| 4692 | 4723 | ||
| 4724 | if (pflip_needed) | ||
| 4725 | continue; | ||
| 4693 | 4726 | ||
| 4694 | WARN_ON(dm_new_plane_state->dc_state); | 4727 | WARN_ON(dm_new_plane_state->dc_state); |
| 4695 | 4728 | ||
| 4696 | dm_new_plane_state->dc_state = dc_create_plane_state(dc); | 4729 | dc_new_plane_state = dc_create_plane_state(dc); |
| 4697 | 4730 | if (!dc_new_plane_state) { | |
| 4698 | DRM_DEBUG_DRIVER("Enabling DRM plane: %d on DRM crtc %d\n", | ||
| 4699 | plane->base.id, new_plane_crtc->base.id); | ||
| 4700 | |||
| 4701 | if (!dm_new_plane_state->dc_state) { | ||
| 4702 | ret = -EINVAL; | 4731 | ret = -EINVAL; |
| 4703 | return ret; | 4732 | return ret; |
| 4704 | } | 4733 | } |
| 4705 | 4734 | ||
| 4735 | DRM_DEBUG_DRIVER("Enabling DRM plane: %d on DRM crtc %d\n", | ||
| 4736 | plane->base.id, new_plane_crtc->base.id); | ||
| 4737 | |||
| 4706 | ret = fill_plane_attributes( | 4738 | ret = fill_plane_attributes( |
| 4707 | new_plane_crtc->dev->dev_private, | 4739 | new_plane_crtc->dev->dev_private, |
| 4708 | dm_new_plane_state->dc_state, | 4740 | dc_new_plane_state, |
| 4709 | new_plane_state, | 4741 | new_plane_state, |
| 4710 | new_crtc_state); | 4742 | new_crtc_state); |
| 4711 | if (ret) | 4743 | if (ret) { |
| 4744 | dc_plane_state_release(dc_new_plane_state); | ||
| 4712 | return ret; | 4745 | return ret; |
| 4746 | } | ||
| 4713 | 4747 | ||
| 4714 | 4748 | /* | |
| 4749 | * Any atomic check errors that occur after this will | ||
| 4750 | * not need a release. The plane state will be attached | ||
| 4751 | * to the stream, and therefore part of the atomic | ||
| 4752 | * state. It'll be released when the atomic state is | ||
| 4753 | * cleaned. | ||
| 4754 | */ | ||
| 4715 | if (!dc_add_plane_to_context( | 4755 | if (!dc_add_plane_to_context( |
| 4716 | dc, | 4756 | dc, |
| 4717 | dm_new_crtc_state->stream, | 4757 | dm_new_crtc_state->stream, |
| 4718 | dm_new_plane_state->dc_state, | 4758 | dc_new_plane_state, |
| 4719 | dm_state->context)) { | 4759 | dm_state->context)) { |
| 4720 | 4760 | ||
| 4761 | dc_plane_state_release(dc_new_plane_state); | ||
| 4721 | ret = -EINVAL; | 4762 | ret = -EINVAL; |
| 4722 | return ret; | 4763 | return ret; |
| 4723 | } | 4764 | } |
| 4724 | 4765 | ||
| 4766 | dm_new_plane_state->dc_state = dc_new_plane_state; | ||
| 4767 | |||
| 4725 | /* Tell DC to do a full surface update every time there | 4768 | /* Tell DC to do a full surface update every time there |
| 4726 | * is a plane change. Inefficient, but works for now. | 4769 | * is a plane change. Inefficient, but works for now. |
| 4727 | */ | 4770 | */ |
| @@ -4735,6 +4778,30 @@ static int dm_update_planes_state(struct dc *dc, · 7 unmodified lines | |||
| 4735 | return ret; | 4778 | return ret; |
| 4736 | } | 4779 | } |
| 4737 | 4780 | ||
| 4781 | static int dm_atomic_check_plane_state_fb(struct drm_atomic_state *state, | ||
| 4782 | struct drm_crtc *crtc) | ||
| 4783 | { | ||
| 4784 | struct drm_plane *plane; | ||
| 4785 | struct drm_crtc_state *crtc_state; | ||
| 4786 | |||
| 4787 | WARN_ON(!drm_atomic_get_new_crtc_state(state, crtc)); | ||
| 4788 | |||
| 4789 | drm_for_each_plane_mask(plane, state->dev, crtc->state->plane_mask) { | ||
| 4790 | struct drm_plane_state *plane_state = | ||
| 4791 | drm_atomic_get_plane_state(state, plane); | ||
| 4792 | |||
| 4793 | if (IS_ERR(plane_state)) | ||
| 4794 | return -EDEADLK; | ||
| 4795 | |||
| 4796 | crtc_state = drm_atomic_get_crtc_state(plane_state->state, crtc); | ||
| 4797 | if (crtc->primary == plane && crtc_state->active) { | ||
| 4798 | if (!plane_state->fb) | ||
| 4799 | return -EINVAL; | ||
| 4800 | } | ||
| 4801 | } | ||
| 4802 | return 0; | ||
| 4803 | } | ||
| 4804 | |||
| 4738 | static int amdgpu_dm_atomic_check(struct drm_device *dev, | 4805 | static int amdgpu_dm_atomic_check(struct drm_device *dev, |
| 4739 | struct drm_atomic_state *state) | 4806 | struct drm_atomic_state *state) |
| 4740 | { | 4807 | { |
| @@ -4758,6 +4825,10 @@ static int amdgpu_dm_atomic_check(struct drm_device *dev, · 17 unmodified lines | |||
| 4758 | goto fail; | 4825 | goto fail; |
| 4759 | 4826 | ||
| 4760 | for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) { | 4827 | for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) { |
| 4828 | ret = dm_atomic_check_plane_state_fb(state, crtc); | ||
| 4829 | if (ret) | ||
| 4830 | goto fail; | ||
| 4831 | |||
| 4761 | if (!drm_atomic_crtc_needs_modeset(new_crtc_state) && | 4832 | if (!drm_atomic_crtc_needs_modeset(new_crtc_state) && |
| 4762 | !new_crtc_state->color_mgmt_changed) | 4833 | !new_crtc_state->color_mgmt_changed) |
| 4763 | continue; | 4834 | continue; |
| @@ -683,10 +683,8 @@ static const struct amdgpu_irq_src_funcs dm_hpd_irq_funcs = { · 682 unmodified lines | |||
| 683 | 683 | ||
| 684 | void amdgpu_dm_set_irq_funcs(struct amdgpu_device *adev) | 684 | void amdgpu_dm_set_irq_funcs(struct amdgpu_device *adev) |
| 685 | { | 685 | { |
| 686 | if (adev->mode_info.num_crtc > 0) | 686 | |
| 687 | adev->crtc_irq.num_types = AMDGPU_CRTC_IRQ_VLINE1 + adev->mode_info.num_crtc; | 687 | adev->crtc_irq.num_types = adev->mode_info.num_crtc; |
| 688 | else | ||
| 689 | adev->crtc_irq.num_types = 0; | ||
| 690 | adev->crtc_irq.funcs = &dm_crtc_irq_funcs; | 688 | adev->crtc_irq.funcs = &dm_crtc_irq_funcs; |
| 691 | 689 | ||
| 692 | adev->pageflip_irq.num_types = adev->mode_info.num_crtc; | 690 | adev->pageflip_irq.num_types = adev->mode_info.num_crtc; |
| @@ -189,6 +189,12 @@ void dm_dp_mst_dc_sink_create(struct drm_connector *connector) · 188 unmodified lines | |||
| 189 | .link = aconnector->dc_link, | 189 | .link = aconnector->dc_link, |
| 190 | .sink_signal = SIGNAL_TYPE_DISPLAY_PORT_MST }; | 190 | .sink_signal = SIGNAL_TYPE_DISPLAY_PORT_MST }; |
| 191 | 191 | ||
| 192 | /* | ||
| 193 | * TODO: Need to further figure out why ddc.algo is NULL while MST port exists | ||
| 194 | */ | ||
| 195 | if (!aconnector->port || !aconnector->port->aux.ddc.algo) | ||
| 196 | return; | ||
| 197 | |||
| 192 | edid = drm_dp_mst_get_edid(connector, &aconnector->mst_port->mst_mgr, aconnector->port); | 198 | edid = drm_dp_mst_get_edid(connector, &aconnector->mst_port->mst_mgr, aconnector->port); |
| 193 | 199 | ||
| 194 | if (!edid) { | 200 | if (!edid) { |
| @@ -1358,13 +1358,13 @@ enum dc_irq_source dc_interrupt_to_irq_source( · 1357 unmodified lines | |||
| 1358 | return dal_irq_service_to_irq_source(dc->res_pool->irqs, src_id, ext_id); | 1358 | return dal_irq_service_to_irq_source(dc->res_pool->irqs, src_id, ext_id); |
| 1359 | } | 1359 | } |
| 1360 | 1360 | ||
| 1361 | void dc_interrupt_set(struct dc *dc, enum dc_irq_source src, bool enable) | 1361 | bool dc_interrupt_set(struct dc *dc, enum dc_irq_source src, bool enable) |
| 1362 | { | 1362 | { |
| 1363 | 1363 | ||
| 1364 | if (dc == NULL) | 1364 | if (dc == NULL) |
| 1365 | return; | 1365 | return false; |
| 1366 | 1366 | ||
| 1367 | dal_irq_service_set(dc->res_pool->irqs, src, enable); | 1367 | return dal_irq_service_set(dc->res_pool->irqs, src, enable); |
| 1368 | } | 1368 | } |
| 1369 | 1369 | ||
| 1370 | void dc_interrupt_ack(struct dc *dc, enum dc_irq_source src) | 1370 | void dc_interrupt_ack(struct dc *dc, enum dc_irq_source src) |
| @@ -1749,8 +1749,7 @@ static void enable_link_hdmi(struct pipe_ctx *pipe_ctx) · 1748 unmodified lines | |||
| 1749 | link->link_enc, | 1749 | link->link_enc, |
| 1750 | pipe_ctx->clock_source->id, | 1750 | pipe_ctx->clock_source->id, |
| 1751 | display_color_depth, | 1751 | display_color_depth, |
| 1752 | pipe_ctx->stream->signal == SIGNAL_TYPE_HDMI_TYPE_A, | 1752 | pipe_ctx->stream->signal, |
| 1753 | pipe_ctx->stream->signal == SIGNAL_TYPE_DVI_DUAL_LINK, | ||
| 1754 | stream->phy_pix_clk); | 1753 | stream->phy_pix_clk); |
| 1755 | 1754 | ||
| 1756 | if (pipe_ctx->stream->signal == SIGNAL_TYPE_HDMI_TYPE_A) | 1755 | if (pipe_ctx->stream->signal == SIGNAL_TYPE_HDMI_TYPE_A) |
| @@ -718,7 +718,7 @@ static enum link_training_result perform_channel_equalization_sequence( · 717 unmodified lines | |||
| 718 | uint32_t retries_ch_eq; | 718 | uint32_t retries_ch_eq; |
| 719 | enum dc_lane_count lane_count = lt_settings->link_settings.lane_count; | 719 | enum dc_lane_count lane_count = lt_settings->link_settings.lane_count; |
| 720 | union lane_align_status_updated dpcd_lane_status_updated = {{0}}; | 720 | union lane_align_status_updated dpcd_lane_status_updated = {{0}}; |
| 721 | union lane_status dpcd_lane_status[LANE_COUNT_DP_MAX] = {{{0}}};; | 721 | union lane_status dpcd_lane_status[LANE_COUNT_DP_MAX] = {{{0}}}; |
| 722 | 722 | ||
| 723 | hw_tr_pattern = get_supported_tp(link); | 723 | hw_tr_pattern = get_supported_tp(link); |
| 724 | 724 | ||
| @@ -1465,7 +1465,7 @@ void decide_link_settings(struct dc_stream_state *stream, · 740 unmodified lines | |||
| 1465 | /* MST doesn't perform link training for now | 1465 | /* MST doesn't perform link training for now |
| 1466 | * TODO: add MST specific link training routine | 1466 | * TODO: add MST specific link training routine |
| 1467 | */ | 1467 | */ |
| 1468 | if (is_mst_supported(link)) { | 1468 | if (stream->signal == SIGNAL_TYPE_DISPLAY_PORT_MST) { |
| 1469 | *link_setting = link->verified_link_cap; | 1469 | *link_setting = link->verified_link_cap; |
| 1470 | return; | 1470 | return; |
| 1471 | } | 1471 | } |
| @@ -1360,9 +1360,6 @@ bool dc_is_stream_scaling_unchanged( · 1359 unmodified lines | |||
| 1360 | return true; | 1360 | return true; |
| 1361 | } | 1361 | } |
| 1362 | 1362 | ||
| 1363 | /* Maximum TMDS single link pixel clock 165MHz */ | ||
| 1364 | #define TMDS_MAX_PIXEL_CLOCK_IN_KHZ 165000 | ||
| 1365 | |||
| 1366 | static void update_stream_engine_usage( | 1363 | static void update_stream_engine_usage( |
| 1367 | struct resource_context *res_ctx, | 1364 | struct resource_context *res_ctx, |
| 1368 | const struct resource_pool *pool, | 1365 | const struct resource_pool *pool, |
| @@ -33,8 +33,7 @@ · 32 unmodified lines | |||
| 33 | /******************************************************************************* | 33 | /******************************************************************************* |
| 34 | * Private functions | 34 | * Private functions |
| 35 | ******************************************************************************/ | 35 | ******************************************************************************/ |
| 36 | #define TMDS_MAX_PIXEL_CLOCK_IN_KHZ_UPMOST 297000 | 36 | void update_stream_signal(struct dc_stream_state *stream) |
| 37 | static void update_stream_signal(struct dc_stream_state *stream) | ||
| 38 | { | 37 | { |
| 39 | 38 | ||
| 40 | struct dc_sink *dc_sink = stream->sink; | 39 | struct dc_sink *dc_sink = stream->sink; |
| @@ -45,8 +44,9 @@ static void update_stream_signal(struct dc_stream_state *stream) · 4 unmodified lines | |||
| 45 | stream->signal = dc_sink->sink_signal; | 44 | stream->signal = dc_sink->sink_signal; |
| 46 | 45 | ||
| 47 | if (dc_is_dvi_signal(stream->signal)) { | 46 | if (dc_is_dvi_signal(stream->signal)) { |
| 48 | if (stream->timing.pix_clk_khz > TMDS_MAX_PIXEL_CLOCK_IN_KHZ_UPMOST && | 47 | if (stream->ctx->dc->caps.dual_link_dvi && |
| 49 | stream->sink->sink_signal != SIGNAL_TYPE_DVI_SINGLE_LINK) | 48 | stream->timing.pix_clk_khz > TMDS_MAX_PIXEL_CLOCK && |
| 49 | stream->sink->sink_signal != SIGNAL_TYPE_DVI_SINGLE_LINK) | ||
| 50 | stream->signal = SIGNAL_TYPE_DVI_DUAL_LINK; | 50 | stream->signal = SIGNAL_TYPE_DVI_DUAL_LINK; |
| 51 | else | 51 | else |
| 52 | stream->signal = SIGNAL_TYPE_DVI_SINGLE_LINK; | 52 | stream->signal = SIGNAL_TYPE_DVI_SINGLE_LINK; |
| @@ -193,44 +193,20 @@ bool dc_stream_set_cursor_attributes( · 140 unmodified lines | |||
| 193 | 193 | ||
| 194 | core_dc = stream->ctx->dc; | 194 | core_dc = stream->ctx->dc; |
| 195 | res_ctx = &core_dc->current_state->res_ctx; | 195 | res_ctx = &core_dc->current_state->res_ctx; |
| 196 | stream->cursor_attributes = *attributes; | ||
| 196 | 197 | ||
| 197 | for (i = 0; i < MAX_PIPES; i++) { | 198 | for (i = 0; i < MAX_PIPES; i++) { |
| 198 | struct pipe_ctx *pipe_ctx = &res_ctx->pipe_ctx[i]; | 199 | struct pipe_ctx *pipe_ctx = &res_ctx->pipe_ctx[i]; |
| 199 | 200 | ||
| 200 | if (pipe_ctx->stream != stream || (!pipe_ctx->plane_res.xfm && !pipe_ctx->plane_res.dpp)) | 201 | if (pipe_ctx->stream != stream || (!pipe_ctx->plane_res.xfm && |
| 202 | !pipe_ctx->plane_res.dpp) || !pipe_ctx->plane_res.ipp) | ||
| 201 | continue; | 203 | continue; |
| 202 | if (pipe_ctx->top_pipe && pipe_ctx->plane_state != pipe_ctx->top_pipe->plane_state) | 204 | if (pipe_ctx->top_pipe && pipe_ctx->plane_state != pipe_ctx->top_pipe->plane_state) |
| 203 | continue; | 205 | continue; |
| 204 | 206 | ||
| 205 | 207 | ||
| 206 | if (pipe_ctx->plane_res.ipp->funcs->ipp_cursor_set_attributes != NULL) | 208 | core_dc->hwss.set_cursor_attribute(pipe_ctx); |
| 207 | pipe_ctx->plane_res.ipp->funcs->ipp_cursor_set_attributes( | ||
| 208 | pipe_ctx->plane_res.ipp, attributes); | ||
| 209 | |||
| 210 | if (pipe_ctx->plane_res.hubp != NULL && | ||
| 211 | pipe_ctx->plane_res.hubp->funcs->set_cursor_attributes != NULL) | ||
| 212 | pipe_ctx->plane_res.hubp->funcs->set_cursor_attributes( | ||
| 213 | pipe_ctx->plane_res.hubp, attributes); | ||
| 214 | |||
| 215 | if (pipe_ctx->plane_res.mi != NULL && | ||
| 216 | pipe_ctx->plane_res.mi->funcs->set_cursor_attributes != NULL) | ||
| 217 | pipe_ctx->plane_res.mi->funcs->set_cursor_attributes( | ||
| 218 | pipe_ctx->plane_res.mi, attributes); | ||
| 219 | |||
| 220 | |||
| 221 | if (pipe_ctx->plane_res.xfm != NULL && | ||
| 222 | pipe_ctx->plane_res.xfm->funcs->set_cursor_attributes != NULL) | ||
| 223 | pipe_ctx->plane_res.xfm->funcs->set_cursor_attributes( | ||
| 224 | pipe_ctx->plane_res.xfm, attributes); | ||
| 225 | |||
| 226 | if (pipe_ctx->plane_res.dpp != NULL && | ||
| 227 | pipe_ctx->plane_res.dpp->funcs->set_cursor_attributes != NULL) | ||
| 228 | pipe_ctx->plane_res.dpp->funcs->set_cursor_attributes( | ||
| 229 | pipe_ctx->plane_res.dpp, attributes->color_format); | ||
| 230 | } | 209 | } |
| 231 | |||
| 232 | stream->cursor_attributes = *attributes; | ||
| 233 | |||
| 234 | return true; | 210 | return true; |
| 235 | } | 211 | } |
| 236 | 212 | ||
| @@ -254,55 +230,21 @@ bool dc_stream_set_cursor_position( · 17 unmodified lines | |||
| 254 | 230 | ||
| 255 | core_dc = stream->ctx->dc; | 231 | core_dc = stream->ctx->dc; |
| 256 | res_ctx = &core_dc->current_state->res_ctx; | 232 | res_ctx = &core_dc->current_state->res_ctx; |
| 233 | stream->cursor_position = *position; | ||
| 257 | 234 | ||
| 258 | for (i = 0; i < MAX_PIPES; i++) { | 235 | for (i = 0; i < MAX_PIPES; i++) { |
| 259 | struct pipe_ctx *pipe_ctx = &res_ctx->pipe_ctx[i]; | 236 | struct pipe_ctx *pipe_ctx = &res_ctx->pipe_ctx[i]; |
| 260 | struct input_pixel_processor *ipp = pipe_ctx->plane_res.ipp; | ||
| 261 | struct mem_input *mi = pipe_ctx->plane_res.mi; | ||
| 262 | struct hubp *hubp = pipe_ctx->plane_res.hubp; | ||
| 263 | struct dpp *dpp = pipe_ctx->plane_res.dpp; | ||
| 264 | struct dc_cursor_position pos_cpy = *position; | ||
| 265 | struct dc_cursor_mi_param param = { | ||
| 266 | .pixel_clk_khz = stream->timing.pix_clk_khz, | ||
| 267 | .ref_clk_khz = core_dc->res_pool->ref_clock_inKhz, | ||
| 268 | .viewport_x_start = pipe_ctx->plane_res.scl_data.viewport.x, | ||
| 269 | .viewport_width = pipe_ctx->plane_res.scl_data.viewport.width, | ||
| 270 | .h_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.horz | ||
| 271 | }; | ||
| 272 | 237 | ||
| 273 | if (pipe_ctx->stream != stream || | 238 | if (pipe_ctx->stream != stream || |
| 274 | (!pipe_ctx->plane_res.mi && !pipe_ctx->plane_res.hubp) || | 239 | (!pipe_ctx->plane_res.mi && !pipe_ctx->plane_res.hubp) || |
| 275 | !pipe_ctx->plane_state || | 240 | !pipe_ctx->plane_state || |
| 276 | (!pipe_ctx->plane_res.xfm && !pipe_ctx->plane_res.dpp)) | 241 | (!pipe_ctx->plane_res.xfm && !pipe_ctx->plane_res.dpp) || |
| 277 | continue; | 242 | !pipe_ctx->plane_res.ipp) |
| 278 | |||
| 279 | if (pipe_ctx->plane_state->address.type | ||
| 280 | == PLN_ADDR_TYPE_VIDEO_PROGRESSIVE) | ||
| 281 | pos_cpy.enable = false; | ||
| 282 | |||
| 283 | if (pipe_ctx->top_pipe && pipe_ctx->plane_state != pipe_ctx->top_pipe->plane_state) | ||
| 284 | pos_cpy.enable = false; | ||
| 285 | |||
| 286 | |||
| 287 | if (ipp != NULL && ipp->funcs->ipp_cursor_set_position != NULL) | ||
| 288 | ipp->funcs->ipp_cursor_set_position(ipp, &pos_cpy, ¶m); | ||
| 289 | |||
| 290 | if (mi != NULL && mi->funcs->set_cursor_position != NULL) | ||
| 291 | mi->funcs->set_cursor_position(mi, &pos_cpy, ¶m); | ||
| 292 | |||
| 293 | if (!hubp) | ||
| 294 | continue; | 243 | continue; |
| 295 | 244 | ||
| 296 | if (hubp->funcs->set_cursor_position != NULL) | 245 | core_dc->hwss.set_cursor_position(pipe_ctx); |
| 297 | hubp->funcs->set_cursor_position(hubp, &pos_cpy, ¶m); | ||
| 298 | |||
| 299 | if (dpp != NULL && dpp->funcs->set_cursor_position != NULL) | ||
| 300 | dpp->funcs->set_cursor_position(dpp, &pos_cpy, ¶m, hubp->curs_attr.width); | ||
| 301 | |||
| 302 | } | 246 | } |
| 303 | 247 | ||
| 304 | stream->cursor_position = *position; | ||
| 305 | |||
| 306 | return true; | 248 | return true; |
| 307 | } | 249 | } |
| 308 | 250 | ||
| @@ -62,6 +62,7 @@ struct dc_caps { · 61 unmodified lines | |||
| 62 | bool dcc_const_color; | 62 | bool dcc_const_color; |
| 63 | bool dynamic_audio; | 63 | bool dynamic_audio; |
| 64 | bool is_apu; | 64 | bool is_apu; |
| 65 | bool dual_link_dvi; | ||
| 65 | }; | 66 | }; |
| 66 | 67 | ||
| 67 | struct dc_dcc_surface_param { | 68 | struct dc_dcc_surface_param { |
| @@ -672,7 +673,7 @@ enum dc_irq_source dc_interrupt_to_irq_source( · 604 unmodified lines | |||
| 672 | struct dc *dc, | 673 | struct dc *dc, |
| 673 | uint32_t src_id, | 674 | uint32_t src_id, |
| 674 | uint32_t ext_id); | 675 | uint32_t ext_id); |
| 675 | void dc_interrupt_set(struct dc *dc, enum dc_irq_source src, bool enable); | 676 | bool dc_interrupt_set(struct dc *dc, enum dc_irq_source src, bool enable); |
| 676 | void dc_interrupt_ack(struct dc *dc, enum dc_irq_source src); | 677 | void dc_interrupt_ack(struct dc *dc, enum dc_irq_source src); |
| 677 | enum dc_irq_source dc_get_hpd_irq_source_at_index( | 678 | enum dc_irq_source dc_get_hpd_irq_source_at_index( |
| 678 | struct dc *dc, uint32_t link_index); | 679 | struct dc *dc, uint32_t link_index); |
| @@ -237,6 +237,8 @@ enum surface_update_type dc_check_update_surfaces_for_stream( · 236 unmodified lines | |||
| 237 | */ | 237 | */ |
| 238 | struct dc_stream_state *dc_create_stream_for_sink(struct dc_sink *dc_sink); | 238 | struct dc_stream_state *dc_create_stream_for_sink(struct dc_sink *dc_sink); |
| 239 | 239 | ||
| 240 | void update_stream_signal(struct dc_stream_state *stream); | ||
| 241 | |||
| 240 | void dc_stream_retain(struct dc_stream_state *dc_stream); | 242 | void dc_stream_retain(struct dc_stream_state *dc_stream); |
| 241 | void dc_stream_release(struct dc_stream_state *dc_stream); | 243 | void dc_stream_release(struct dc_stream_state *dc_stream); |
| 242 | 244 | ||
| @@ -236,6 +236,7 @@ · 235 unmodified lines | |||
| 236 | SR(D2VGA_CONTROL), \ | 236 | SR(D2VGA_CONTROL), \ |
| 237 | SR(D3VGA_CONTROL), \ | 237 | SR(D3VGA_CONTROL), \ |
| 238 | SR(D4VGA_CONTROL), \ | 238 | SR(D4VGA_CONTROL), \ |
| 239 | SR(VGA_TEST_CONTROL), \ | ||
| 239 | SR(DC_IP_REQUEST_CNTL), \ | 240 | SR(DC_IP_REQUEST_CNTL), \ |
| 240 | BL_REG_LIST() | 241 | BL_REG_LIST() |
| 241 | 242 | ||
| @@ -337,6 +338,7 @@ struct dce_hwseq_registers { · 95 unmodified lines | |||
| 337 | uint32_t D2VGA_CONTROL; | 338 | uint32_t D2VGA_CONTROL; |
| 338 | uint32_t D3VGA_CONTROL; | 339 | uint32_t D3VGA_CONTROL; |
| 339 | uint32_t D4VGA_CONTROL; | 340 | uint32_t D4VGA_CONTROL; |
| 341 | uint32_t VGA_TEST_CONTROL; | ||
| 340 | /* MMHUB registers. read only. temporary hack */ | 342 | /* MMHUB registers. read only. temporary hack */ |
| 341 | uint32_t VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32; | 343 | uint32_t VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32; |
| 342 | uint32_t VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32; | 344 | uint32_t VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32; |
| @@ -493,6 +495,9 @@ struct dce_hwseq_registers { · 150 unmodified lines | |||
| 493 | HWS_SF(, DOMAIN6_PG_STATUS, DOMAIN6_PGFSM_PWR_STATUS, mask_sh), \ | 495 | HWS_SF(, DOMAIN6_PG_STATUS, DOMAIN6_PGFSM_PWR_STATUS, mask_sh), \ |
| 494 | HWS_SF(, DOMAIN7_PG_STATUS, DOMAIN7_PGFSM_PWR_STATUS, mask_sh), \ | 496 | HWS_SF(, DOMAIN7_PG_STATUS, DOMAIN7_PGFSM_PWR_STATUS, mask_sh), \ |
| 495 | HWS_SF(, DC_IP_REQUEST_CNTL, IP_REQUEST_EN, mask_sh), \ | 497 | HWS_SF(, DC_IP_REQUEST_CNTL, IP_REQUEST_EN, mask_sh), \ |
| 498 | HWS_SF(, D1VGA_CONTROL, D1VGA_MODE_ENABLE, mask_sh),\ | ||
| 499 | HWS_SF(, VGA_TEST_CONTROL, VGA_TEST_ENABLE, mask_sh),\ | ||
| 500 | HWS_SF(, VGA_TEST_CONTROL, VGA_TEST_RENDER_START, mask_sh),\ | ||
| 496 | HWS_SF(, LVTMA_PWRSEQ_CNTL, LVTMA_BLON, mask_sh), \ | 501 | HWS_SF(, LVTMA_PWRSEQ_CNTL, LVTMA_BLON, mask_sh), \ |
| 497 | HWS_SF(, LVTMA_PWRSEQ_STATE, LVTMA_PWRSEQ_TARGET_STATE_R, mask_sh) | 502 | HWS_SF(, LVTMA_PWRSEQ_STATE, LVTMA_PWRSEQ_TARGET_STATE_R, mask_sh) |
| 498 | 503 | ||
| @@ -583,7 +588,10 @@ struct dce_hwseq_registers { · 84 unmodified lines | |||
| 583 | type DCFCLK_GATE_DIS; \ | 588 | type DCFCLK_GATE_DIS; \ |
| 584 | type DCHUBBUB_GLOBAL_TIMER_REFDIV; \ | 589 | type DCHUBBUB_GLOBAL_TIMER_REFDIV; \ |
| 585 | type DENTIST_DPPCLK_WDIVIDER; \ | 590 | type DENTIST_DPPCLK_WDIVIDER; \ |
| 586 | type DENTIST_DISPCLK_WDIVIDER; | 591 | type DENTIST_DISPCLK_WDIVIDER; \ |
| 592 | type VGA_TEST_ENABLE; \ | ||
| 593 | type VGA_TEST_RENDER_START; \ | ||
| 594 | type D1VGA_MODE_ENABLE; | ||
| 587 | 595 | ||
| 588 | struct dce_hwseq_shift { | 596 | struct dce_hwseq_shift { |
| 589 | HWSEQ_REG_FIELD_LIST(uint8_t) | 597 | HWSEQ_REG_FIELD_LIST(uint8_t) |
| @@ -82,13 +82,6 @@ · 81 unmodified lines | |||
| 82 | #define DCE110_DIG_FE_SOURCE_SELECT_DIGF 0x20 | 82 | #define DCE110_DIG_FE_SOURCE_SELECT_DIGF 0x20 |
| 83 | #define DCE110_DIG_FE_SOURCE_SELECT_DIGG 0x40 | 83 | #define DCE110_DIG_FE_SOURCE_SELECT_DIGG 0x40 |
| 84 | 84 | ||
| 85 | /* Minimum pixel clock, in KHz. For TMDS signal is 25.00 MHz */ | ||
| 86 | #define TMDS_MIN_PIXEL_CLOCK 25000 | ||
| 87 | /* Maximum pixel clock, in KHz. For TMDS signal is 165.00 MHz */ | ||
| 88 | #define TMDS_MAX_PIXEL_CLOCK 165000 | ||
| 89 | /* For current ASICs pixel clock - 600MHz */ | ||
| 90 | #define MAX_ENCODER_CLOCK 600000 | ||
| 91 | |||
| 92 | enum { | 85 | enum { |
| 93 | DP_MST_UPDATE_MAX_RETRY = 50 | 86 | DP_MST_UPDATE_MAX_RETRY = 50 |
| 94 | }; | 87 | }; |
| @@ -683,6 +676,7 @@ void dce110_link_encoder_construct( · 588 unmodified lines | |||
| 683 | { | 676 | { |
| 684 | struct bp_encoder_cap_info bp_cap_info = {0}; | 677 | struct bp_encoder_cap_info bp_cap_info = {0}; |
| 685 | const struct dc_vbios_funcs *bp_funcs = init_data->ctx->dc_bios->funcs; | 678 | const struct dc_vbios_funcs *bp_funcs = init_data->ctx->dc_bios->funcs; |
| 679 | enum bp_result result = BP_RESULT_OK; | ||
| 686 | 680 | ||
| 687 | enc110->base.funcs = &dce110_lnk_enc_funcs; | 681 | enc110->base.funcs = &dce110_lnk_enc_funcs; |
| 688 | enc110->base.ctx = init_data->ctx; | 682 | enc110->base.ctx = init_data->ctx; |
| @@ -757,15 +751,24 @@ void dce110_link_encoder_construct( · 68 unmodified lines | |||
| 757 | enc110->base.preferred_engine = ENGINE_ID_UNKNOWN; | 751 | enc110->base.preferred_engine = ENGINE_ID_UNKNOWN; |
| 758 | } | 752 | } |
| 759 | 753 | ||
| 754 | /* default to one to mirror Windows behavior */ | ||
| 755 | enc110->base.features.flags.bits.HDMI_6GB_EN = 1; | ||
| 756 | |||
| 757 | result = bp_funcs->get_encoder_cap_info(enc110->base.ctx->dc_bios, | ||
| 758 | enc110->base.id, &bp_cap_info); | ||
| 759 | |||
| 760 | /* Override features with DCE-specific values */ | 760 | /* Override features with DCE-specific values */ |
| 761 | if (BP_RESULT_OK == bp_funcs->get_encoder_cap_info( | 761 | if (BP_RESULT_OK == result) { |
| 762 | enc110->base.ctx->dc_bios, enc110->base.id, | ||
| 763 | &bp_cap_info)) { | ||
| 764 | enc110->base.features.flags.bits.IS_HBR2_CAPABLE = | 762 | enc110->base.features.flags.bits.IS_HBR2_CAPABLE = |
| 765 | bp_cap_info.DP_HBR2_EN; | 763 | bp_cap_info.DP_HBR2_EN; |
| 766 | enc110->base.features.flags.bits.IS_HBR3_CAPABLE = | 764 | enc110->base.features.flags.bits.IS_HBR3_CAPABLE = |
| 767 | bp_cap_info.DP_HBR3_EN; | 765 | bp_cap_info.DP_HBR3_EN; |
| 768 | enc110->base.features.flags.bits.HDMI_6GB_EN = bp_cap_info.HDMI_6GB_EN; | 766 | enc110->base.features.flags.bits.HDMI_6GB_EN = bp_cap_info.HDMI_6GB_EN; |
| 767 | } else { | ||
| 768 | dm_logger_write(enc110->base.ctx->logger, LOG_WARNING, | ||
| 769 | "%s: Failed to get encoder_cap_info from VBIOS with error code %d!\n", | ||
| 770 | __func__, | ||
| 771 | result); | ||
| 769 | } | 772 | } |
| 770 | } | 773 | } |
| 771 | 774 | ||
| @@ -904,8 +907,7 @@ void dce110_link_encoder_enable_tmds_output( · 132 unmodified lines | |||
| 904 | struct link_encoder *enc, | 907 | struct link_encoder *enc, |
| 905 | enum clock_source_id clock_source, | 908 | enum clock_source_id clock_source, |
| 906 | enum dc_color_depth color_depth, | 909 | enum dc_color_depth color_depth, |
| 907 | bool hdmi, | 910 | enum signal_type signal, |
| 908 | bool dual_link, | ||
| 909 | uint32_t pixel_clock) | 911 | uint32_t pixel_clock) |
| 910 | { | 912 | { |
| 911 | struct dce110_link_encoder *enc110 = TO_DCE110_LINK_ENC(enc); | 913 | struct dce110_link_encoder *enc110 = TO_DCE110_LINK_ENC(enc); |
| @@ -919,16 +921,12 @@ void dce110_link_encoder_enable_tmds_output( · 7 unmodified lines | |||
| 919 | cntl.engine_id = enc->preferred_engine; | 921 | cntl.engine_id = enc->preferred_engine; |
| 920 | cntl.transmitter = enc110->base.transmitter; | 922 | cntl.transmitter = enc110->base.transmitter; |
| 921 | cntl.pll_id = clock_source; | 923 | cntl.pll_id = clock_source; |
| 922 | if (hdmi) { | 924 | cntl.signal = signal; |
| 923 | cntl.signal = SIGNAL_TYPE_HDMI_TYPE_A; | 925 | if (cntl.signal == SIGNAL_TYPE_DVI_DUAL_LINK) |
| 924 | cntl.lanes_number = 4; | ||
| 925 | } else if (dual_link) { | ||
| 926 | cntl.signal = SIGNAL_TYPE_DVI_DUAL_LINK; | ||
| 927 | cntl.lanes_number = 8; | 926 | cntl.lanes_number = 8; |
| 928 | } else { | 927 | else |
| 929 | cntl.signal = SIGNAL_TYPE_DVI_SINGLE_LINK; | ||
| 930 | cntl.lanes_number = 4; | 928 | cntl.lanes_number = 4; |
| 931 | } | 929 | |
| 932 | cntl.hpd_sel = enc110->base.hpd_source; | 930 | cntl.hpd_sel = enc110->base.hpd_source; |
| 933 | 931 | ||
| 934 | cntl.pixel_clock = pixel_clock; | 932 | cntl.pixel_clock = pixel_clock; |
| @@ -210,8 +210,7 @@ void dce110_link_encoder_enable_tmds_output( · 209 unmodified lines | |||
| 210 | struct link_encoder *enc, | 210 | struct link_encoder *enc, |
| 211 | enum clock_source_id clock_source, | 211 | enum clock_source_id clock_source, |
| 212 | enum dc_color_depth color_depth, | 212 | enum dc_color_depth color_depth, |
| 213 | bool hdmi, | 213 | enum signal_type signal, |
| 214 | bool dual_link, | ||
| 215 | uint32_t pixel_clock); | 214 | uint32_t pixel_clock); |
| 216 | 215 | ||
| 217 | /* enables DP PHY output */ | 216 | /* enables DP PHY output */ |
| @@ -852,6 +852,7 @@ static bool construct( · 851 unmodified lines | |||
| 852 | dc->caps.max_downscale_ratio = 200; | 852 | dc->caps.max_downscale_ratio = 200; |
| 853 | dc->caps.i2c_speed_in_khz = 40; | 853 | dc->caps.i2c_speed_in_khz = 40; |
| 854 | dc->caps.max_cursor_size = 128; | 854 | dc->caps.max_cursor_size = 128; |
| 855 | dc->caps.dual_link_dvi = true; | ||
| 855 | 856 | ||
| 856 | for (i = 0; i < pool->base.pipe_count; i++) { | 857 | for (i = 0; i < pool->base.pipe_count; i++) { |
| 857 | pool->base.timing_generators[i] = | 858 | pool->base.timing_generators[i] = |
| @@ -688,15 +688,22 @@ void dce110_enable_stream(struct pipe_ctx *pipe_ctx) · 687 unmodified lines | |||
| 688 | struct dc_crtc_timing *timing = &pipe_ctx->stream->timing; | 688 | struct dc_crtc_timing *timing = &pipe_ctx->stream->timing; |
| 689 | struct dc_link *link = pipe_ctx->stream->sink->link; | 689 | struct dc_link *link = pipe_ctx->stream->sink->link; |
| 690 | 690 | ||
| 691 | /* 1. update AVI info frame (HDMI, DP) | 691 | |
| 692 | * we always need to update info frame | ||
| 693 | */ | ||
| 694 | uint32_t active_total_with_borders; | 692 | uint32_t active_total_with_borders; |
| 695 | uint32_t early_control = 0; | 693 | uint32_t early_control = 0; |
| 696 | struct timing_generator *tg = pipe_ctx->stream_res.tg; | 694 | struct timing_generator *tg = pipe_ctx->stream_res.tg; |
| 697 | 695 | ||
| 698 | /* TODOFPGA may change to hwss.update_info_frame */ | 696 | /* For MST, there are multiply stream go to only one link. |
| 697 | * connect DIG back_end to front_end while enable_stream and | ||
| 698 | * disconnect them during disable_stream | ||
| 699 | * BY this, it is logic clean to separate stream and link */ | ||
| 700 | link->link_enc->funcs->connect_dig_be_to_fe(link->link_enc, | ||
| 701 | pipe_ctx->stream_res.stream_enc->id, true); | ||
| 702 | |||
| 703 | /* update AVI info frame (HDMI, DP)*/ | ||
| 704 | /* TODO: FPGA may change to hwss.update_info_frame */ | ||
| 699 | dce110_update_info_frame(pipe_ctx); | 705 | dce110_update_info_frame(pipe_ctx); |
| 706 | |||
| 700 | /* enable early control to avoid corruption on DP monitor*/ | 707 | /* enable early control to avoid corruption on DP monitor*/ |
| 701 | active_total_with_borders = | 708 | active_total_with_borders = |
| 702 | timing->h_addressable | 709 | timing->h_addressable |
| @@ -717,12 +724,8 @@ void dce110_enable_stream(struct pipe_ctx *pipe_ctx) · 14 unmodified lines | |||
| 717 | pipe_ctx->stream_res.stream_enc->funcs->dp_audio_enable(pipe_ctx->stream_res.stream_enc); | 724 | pipe_ctx->stream_res.stream_enc->funcs->dp_audio_enable(pipe_ctx->stream_res.stream_enc); |
| 718 | } | 725 | } |
| 719 | 726 | ||
| 720 | /* For MST, there are multiply stream go to only one link. | 727 | |
| 721 | * connect DIG back_end to front_end while enable_stream and | 728 | |
| 722 | * disconnect them during disable_stream | ||
| 723 | * BY this, it is logic clean to separate stream and link */ | ||
| 724 | link->link_enc->funcs->connect_dig_be_to_fe(link->link_enc, | ||
| 725 | pipe_ctx->stream_res.stream_enc->id, true); | ||
| 726 | 729 | ||
| 727 | } | 730 | } |
| 728 | 731 | ||
| @@ -1690,9 +1693,13 @@ static void apply_min_clocks( · 961 unmodified lines | |||
| 1690 | * Check if FBC can be enabled | 1693 | * Check if FBC can be enabled |
| 1691 | */ | 1694 | */ |
| 1692 | static bool should_enable_fbc(struct dc *dc, | 1695 | static bool should_enable_fbc(struct dc *dc, |
| 1693 | struct dc_state *context) | 1696 | struct dc_state *context, |
| 1697 | uint32_t *pipe_idx) | ||
| 1694 | { | 1698 | { |
| 1695 | struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[0]; | 1699 | uint32_t i; |
| 1700 | struct pipe_ctx *pipe_ctx = NULL; | ||
| 1701 | struct resource_context *res_ctx = &context->res_ctx; | ||
| 1702 | |||
| 1696 | 1703 | ||
| 1697 | ASSERT(dc->fbc_compressor); | 1704 | ASSERT(dc->fbc_compressor); |
| 1698 | 1705 | ||
| @@ -1704,6 +1711,14 @@ static bool should_enable_fbc(struct dc *dc, · 5 unmodified lines | |||
| 1704 | if (context->stream_count != 1) | 1711 | if (context->stream_count != 1) |
| 1705 | return false; | 1712 | return false; |
| 1706 | 1713 | ||
| 1714 | for (i = 0; i < dc->res_pool->pipe_count; i++) { | ||
| 1715 | if (res_ctx->pipe_ctx[i].stream) { | ||
| 1716 | pipe_ctx = &res_ctx->pipe_ctx[i]; | ||
| 1717 | *pipe_idx = i; | ||
| 1718 | break; | ||
| 1719 | } | ||
| 1720 | } | ||
| 1721 | |||
| 1707 | /* Only supports eDP */ | 1722 | /* Only supports eDP */ |
| 1708 | if (pipe_ctx->stream->sink->link->connector_signal != SIGNAL_TYPE_EDP) | 1723 | if (pipe_ctx->stream->sink->link->connector_signal != SIGNAL_TYPE_EDP) |
| 1709 | return false; | 1724 | return false; |
| @@ -1729,11 +1744,14 @@ static bool should_enable_fbc(struct dc *dc, · 19 unmodified lines | |||
| 1729 | static void enable_fbc(struct dc *dc, | 1744 | static void enable_fbc(struct dc *dc, |
| 1730 | struct dc_state *context) | 1745 | struct dc_state *context) |
| 1731 | { | 1746 | { |
| 1732 | if (should_enable_fbc(dc, context)) { | 1747 | uint32_t pipe_idx = 0; |
| 1748 | |||
| 1749 | if (should_enable_fbc(dc, context, &pipe_idx)) { | ||
| 1733 | /* Program GRPH COMPRESSED ADDRESS and PITCH */ | 1750 | /* Program GRPH COMPRESSED ADDRESS and PITCH */ |
| 1734 | struct compr_addr_and_pitch_params params = {0, 0, 0}; | 1751 | struct compr_addr_and_pitch_params params = {0, 0, 0}; |
| 1735 | struct compressor *compr = dc->fbc_compressor; | 1752 | struct compressor *compr = dc->fbc_compressor; |
| 1736 | struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[0]; | 1753 | struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[pipe_idx]; |
| 1754 | |||
| 1737 | 1755 | ||
| 1738 | params.source_view_width = pipe_ctx->stream->timing.h_addressable; | 1756 | params.source_view_width = pipe_ctx->stream->timing.h_addressable; |
| 1739 | params.source_view_height = pipe_ctx->stream->timing.v_addressable; | 1757 | params.source_view_height = pipe_ctx->stream->timing.v_addressable; |
| @@ -2915,6 +2933,49 @@ static void program_csc_matrix(struct pipe_ctx *pipe_ctx, · 1175 unmodified lines | |||
| 2915 | } | 2933 | } |
| 2916 | } | 2934 | } |
| 2917 | 2935 | ||
| 2936 | void dce110_set_cursor_position(struct pipe_ctx *pipe_ctx) | ||
| 2937 | { | ||
| 2938 | struct dc_cursor_position pos_cpy = pipe_ctx->stream->cursor_position; | ||
| 2939 | struct input_pixel_processor *ipp = pipe_ctx->plane_res.ipp; | ||
| 2940 | struct mem_input *mi = pipe_ctx->plane_res.mi; | ||
| 2941 | struct dc_cursor_mi_param param = { | ||
| 2942 | .pixel_clk_khz = pipe_ctx->stream->timing.pix_clk_khz, | ||
| 2943 | .ref_clk_khz = pipe_ctx->stream->ctx->dc->res_pool->ref_clock_inKhz, | ||
| 2944 | .viewport_x_start = pipe_ctx->plane_res.scl_data.viewport.x, | ||
| 2945 | .viewport_width = pipe_ctx->plane_res.scl_data.viewport.width, | ||
| 2946 | .h_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.horz | ||
| 2947 | }; | ||
| 2948 | |||
| 2949 | if (pipe_ctx->plane_state->address.type | ||
| 2950 | == PLN_ADDR_TYPE_VIDEO_PROGRESSIVE) | ||
| 2951 | pos_cpy.enable = false; | ||
| 2952 | |||
| 2953 | if (pipe_ctx->top_pipe && pipe_ctx->plane_state != pipe_ctx->top_pipe->plane_state) | ||
| 2954 | pos_cpy.enable = false; | ||
| 2955 | |||
| 2956 | if (ipp->funcs->ipp_cursor_set_position) | ||
| 2957 | ipp->funcs->ipp_cursor_set_position(ipp, &pos_cpy, ¶m); | ||
| 2958 | if (mi->funcs->set_cursor_position) | ||
| 2959 | mi->funcs->set_cursor_position(mi, &pos_cpy, ¶m); | ||
| 2960 | } | ||
| 2961 | |||
| 2962 | void dce110_set_cursor_attribute(struct pipe_ctx *pipe_ctx) | ||
| 2963 | { | ||
| 2964 | struct dc_cursor_attributes *attributes = &pipe_ctx->stream->cursor_attributes; | ||
| 2965 | |||
| 2966 | if (pipe_ctx->plane_res.ipp->funcs->ipp_cursor_set_attributes) | ||
| 2967 | pipe_ctx->plane_res.ipp->funcs->ipp_cursor_set_attributes( | ||
| 2968 | pipe_ctx->plane_res.ipp, attributes); | ||
| 2969 | |||
| 2970 | if (pipe_ctx->plane_res.mi->funcs->set_cursor_attributes) | ||
| 2971 | pipe_ctx->plane_res.mi->funcs->set_cursor_attributes( | ||
| 2972 | pipe_ctx->plane_res.mi, attributes); | ||
| 2973 | |||
| 2974 | if (pipe_ctx->plane_res.xfm->funcs->set_cursor_attributes) | ||
| 2975 | pipe_ctx->plane_res.xfm->funcs->set_cursor_attributes( | ||
| 2976 | pipe_ctx->plane_res.xfm, attributes); | ||
| 2977 | } | ||
| 2978 | |||
| 2918 | static void ready_shared_resources(struct dc *dc, struct dc_state *context) {} | 2979 | static void ready_shared_resources(struct dc *dc, struct dc_state *context) {} |
| 2919 | 2980 | ||
| 2920 | static void optimize_shared_resources(struct dc *dc) {} | 2981 | static void optimize_shared_resources(struct dc *dc) {} |
| @@ -2957,6 +3018,8 @@ static const struct hw_sequencer_funcs dce110_funcs = { · 36 unmodified lines | |||
| 2957 | .edp_backlight_control = hwss_edp_backlight_control, | 3018 | .edp_backlight_control = hwss_edp_backlight_control, |
| 2958 | .edp_power_control = hwss_edp_power_control, | 3019 | .edp_power_control = hwss_edp_power_control, |
| 2959 | .edp_wait_for_hpd_ready = hwss_edp_wait_for_hpd_ready, | 3020 | .edp_wait_for_hpd_ready = hwss_edp_wait_for_hpd_ready, |
| 3021 | .set_cursor_position = dce110_set_cursor_position, | ||
| 3022 | .set_cursor_attribute = dce110_set_cursor_attribute | ||
| 2960 | }; | 3023 | }; |
| 2961 | 3024 | ||
| 2962 | void dce110_hw_sequencer_construct(struct dc *dc) | 3025 | void dce110_hw_sequencer_construct(struct dc *dc) |
| @@ -846,6 +846,16 @@ static bool dce110_validate_bandwidth( · 845 unmodified lines | |||
| 846 | return result; | 846 | return result; |
| 847 | } | 847 | } |
| 848 | 848 | ||
| 849 | enum dc_status dce110_validate_plane(const struct dc_plane_state *plane_state, | ||
| 850 | struct dc_caps *caps) | ||
| 851 | { | ||
| 852 | if (((plane_state->dst_rect.width * 2) < plane_state->src_rect.width) || | ||
| 853 | ((plane_state->dst_rect.height * 2) < plane_state->src_rect.height)) | ||
| 854 | return DC_FAIL_SURFACE_VALIDATE; | ||
| 855 | |||
| 856 | return DC_OK; | ||
| 857 | } | ||
| 858 | |||
| 849 | static bool dce110_validate_surface_sets( | 859 | static bool dce110_validate_surface_sets( |
| 850 | struct dc_state *context) | 860 | struct dc_state *context) |
| 851 | { | 861 | { |
| @@ -869,6 +879,13 @@ static bool dce110_validate_surface_sets( · 17 unmodified lines | |||
| 869 | plane->src_rect.height > 1080)) | 879 | plane->src_rect.height > 1080)) |
| 870 | return false; | 880 | return false; |
| 871 | 881 | ||
| 882 | /* we don't have the logic to support underlay | ||
| 883 | * only yet so block the use case where we get | ||
| 884 | * NV12 plane as top layer | ||
| 885 | */ | ||
| 886 | if (j == 0) | ||
| 887 | return false; | ||
| 888 | |||
| 872 | /* irrespective of plane format, | 889 | /* irrespective of plane format, |
| 873 | * stream should be RGB encoded | 890 | * stream should be RGB encoded |
| 874 | */ | 891 | */ |
| @@ -1021,6 +1038,7 @@ static const struct resource_funcs dce110_res_pool_funcs = { · 146 unmodified lines | |||
| 1021 | .link_enc_create = dce110_link_encoder_create, | 1038 | .link_enc_create = dce110_link_encoder_create, |
| 1022 | .validate_guaranteed = dce110_validate_guaranteed, | 1039 | .validate_guaranteed = dce110_validate_guaranteed, |
| 1023 | .validate_bandwidth = dce110_validate_bandwidth, | 1040 | .validate_bandwidth = dce110_validate_bandwidth, |
| 1041 | .validate_plane = dce110_validate_plane, | ||
| 1024 | .acquire_idle_pipe_for_layer = dce110_acquire_underlay, | 1042 | .acquire_idle_pipe_for_layer = dce110_acquire_underlay, |
| 1025 | .add_stream_to_ctx = dce110_add_stream_to_ctx, | 1043 | .add_stream_to_ctx = dce110_add_stream_to_ctx, |
| 1026 | .validate_global = dce110_validate_global | 1044 | .validate_global = dce110_validate_global |
| @@ -1103,6 +1103,8 @@ static bool construct( · 1102 unmodified lines | |||
| 1103 | dc->caps.max_downscale_ratio = 200; | 1103 | dc->caps.max_downscale_ratio = 200; |
| 1104 | dc->caps.i2c_speed_in_khz = 100; | 1104 | dc->caps.i2c_speed_in_khz = 100; |
| 1105 | dc->caps.max_cursor_size = 128; | 1105 | dc->caps.max_cursor_size = 128; |
| 1106 | dc->caps.dual_link_dvi = true; | ||
| 1107 | |||
| 1106 | 1108 | ||
| 1107 | /************************************************* | 1109 | /************************************************* |
| 1108 | * Create resources * | 1110 | * Create resources * |
| @@ -835,6 +835,8 @@ static bool construct( · 834 unmodified lines | |||
| 835 | dc->caps.max_downscale_ratio = 200; | 835 | dc->caps.max_downscale_ratio = 200; |
| 836 | dc->caps.i2c_speed_in_khz = 100; | 836 | dc->caps.i2c_speed_in_khz = 100; |
| 837 | dc->caps.max_cursor_size = 128; | 837 | dc->caps.max_cursor_size = 128; |
| 838 | dc->caps.dual_link_dvi = true; | ||
| 839 | |||
| 838 | dc->debug = debug_defaults; | 840 | dc->debug = debug_defaults; |
| 839 | 841 | ||
| 840 | /************************************************* | 842 | /************************************************* |
| @@ -793,6 +793,7 @@ static bool dce80_construct( · 792 unmodified lines | |||
| 793 | dc->caps.max_downscale_ratio = 200; | 793 | dc->caps.max_downscale_ratio = 200; |
| 794 | dc->caps.i2c_speed_in_khz = 40; | 794 | dc->caps.i2c_speed_in_khz = 40; |
| 795 | dc->caps.max_cursor_size = 128; | 795 | dc->caps.max_cursor_size = 128; |
| 796 | dc->caps.dual_link_dvi = true; | ||
| 796 | 797 | ||
| 797 | /************************************************* | 798 | /************************************************* |
| 798 | * Create resources * | 799 | * Create resources * |
| @@ -238,10 +238,24 @@ static void enable_power_gating_plane( · 237 unmodified lines | |||
| 238 | static void disable_vga( | 238 | static void disable_vga( |
| 239 | struct dce_hwseq *hws) | 239 | struct dce_hwseq *hws) |
| 240 | { | 240 | { |
| 241 | unsigned int in_vga_mode = 0; | ||
| 242 | |||
| 243 | REG_GET(D1VGA_CONTROL, D1VGA_MODE_ENABLE, &in_vga_mode); | ||
| 244 | |||
| 245 | if (in_vga_mode == 0) | ||
| 246 | return; | ||
| 247 | |||
| 241 | REG_WRITE(D1VGA_CONTROL, 0); | 248 | REG_WRITE(D1VGA_CONTROL, 0); |
| 242 | REG_WRITE(D2VGA_CONTROL, 0); | 249 | |
| 243 | REG_WRITE(D3VGA_CONTROL, 0); | 250 | /* HW Engineer's Notes: |
| 244 | REG_WRITE(D4VGA_CONTROL, 0); | 251 | * During switch from vga->extended, if we set the VGA_TEST_ENABLE and |
| 252 | * then hit the VGA_TEST_RENDER_START, then the DCHUBP timing gets updated correctly. | ||
| 253 | * | ||
| 254 | * Then vBIOS will have it poll for the VGA_TEST_RENDER_DONE and unset | ||
| 255 | * VGA_TEST_ENABLE, to leave it in the same state as before. | ||
| 256 | */ | ||
| 257 | REG_UPDATE(VGA_TEST_CONTROL, VGA_TEST_ENABLE, 1); | ||
| 258 | REG_UPDATE(VGA_TEST_CONTROL, VGA_TEST_RENDER_START, 1); | ||
| 245 | } | 259 | } |
| 246 | 260 | ||
| 247 | static void dpp_pg_control( | 261 | static void dpp_pg_control( |
| @@ -1761,6 +1775,11 @@ static void update_dchubp_dpp( · 1513 unmodified lines | |||
| 1761 | &pipe_ctx->plane_res.scl_data.viewport_c); | 1775 | &pipe_ctx->plane_res.scl_data.viewport_c); |
| 1762 | } | 1776 | } |
| 1763 | 1777 | ||
| 1778 | if (pipe_ctx->stream->cursor_attributes.address.quad_part != 0) { | ||
| 1779 | dc->hwss.set_cursor_position(pipe_ctx); | ||
| 1780 | dc->hwss.set_cursor_attribute(pipe_ctx); | ||
| 1781 | } | ||
| 1782 | |||
| 1764 | if (plane_state->update_flags.bits.full_update) { | 1783 | if (plane_state->update_flags.bits.full_update) { |
| 1765 | /*gamut remap*/ | 1784 | /*gamut remap*/ |
| 1766 | program_gamut_remap(pipe_ctx); | 1785 | program_gamut_remap(pipe_ctx); |
| @@ -2296,7 +2315,7 @@ static bool dcn10_dummy_display_power_gating( · 529 unmodified lines | |||
| 2296 | return true; | 2315 | return true; |
| 2297 | } | 2316 | } |
| 2298 | 2317 | ||
| 2299 | void dcn10_update_pending_status(struct pipe_ctx *pipe_ctx) | 2318 | static void dcn10_update_pending_status(struct pipe_ctx *pipe_ctx) |
| 2300 | { | 2319 | { |
| 2301 | struct dc_plane_state *plane_state = pipe_ctx->plane_state; | 2320 | struct dc_plane_state *plane_state = pipe_ctx->plane_state; |
| 2302 | struct timing_generator *tg = pipe_ctx->stream_res.tg; | 2321 | struct timing_generator *tg = pipe_ctx->stream_res.tg; |
| @@ -2316,12 +2335,46 @@ void dcn10_update_pending_status(struct pipe_ctx *pipe_ctx) · 13 unmodified lines | |||
| 2316 | } | 2335 | } |
| 2317 | } | 2336 | } |
| 2318 | 2337 | ||
| 2319 | void dcn10_update_dchub(struct dce_hwseq *hws, struct dchub_init_data *dh_data) | 2338 | static void dcn10_update_dchub(struct dce_hwseq *hws, struct dchub_init_data *dh_data) |
| 2320 | { | 2339 | { |
| 2321 | if (hws->ctx->dc->res_pool->hubbub != NULL) | 2340 | if (hws->ctx->dc->res_pool->hubbub != NULL) |
| 2322 | hubbub1_update_dchub(hws->ctx->dc->res_pool->hubbub, dh_data); | 2341 | hubbub1_update_dchub(hws->ctx->dc->res_pool->hubbub, dh_data); |
| 2323 | } | 2342 | } |
| 2324 | 2343 | ||
| 2344 | static void dcn10_set_cursor_position(struct pipe_ctx *pipe_ctx) | ||
| 2345 | { | ||
| 2346 | struct dc_cursor_position pos_cpy = pipe_ctx->stream->cursor_position; | ||
| 2347 | struct hubp *hubp = pipe_ctx->plane_res.hubp; | ||
| 2348 | struct dpp *dpp = pipe_ctx->plane_res.dpp; | ||
| 2349 | struct dc_cursor_mi_param param = { | ||
| 2350 | .pixel_clk_khz = pipe_ctx->stream->timing.pix_clk_khz, | ||
| 2351 | .ref_clk_khz = pipe_ctx->stream->ctx->dc->res_pool->ref_clock_inKhz, | ||
| 2352 | .viewport_x_start = pipe_ctx->plane_res.scl_data.viewport.x, | ||
| 2353 | .viewport_width = pipe_ctx->plane_res.scl_data.viewport.width, | ||
| 2354 | .h_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.horz | ||
| 2355 | }; | ||
| 2356 | |||
| 2357 | if (pipe_ctx->plane_state->address.type | ||
| 2358 | == PLN_ADDR_TYPE_VIDEO_PROGRESSIVE) | ||
| 2359 | pos_cpy.enable = false; | ||
| 2360 | |||
| 2361 | if (pipe_ctx->top_pipe && pipe_ctx->plane_state != pipe_ctx->top_pipe->plane_state) | ||
| 2362 | pos_cpy.enable = false; | ||
| 2363 | |||
| 2364 | hubp->funcs->set_cursor_position(hubp, &pos_cpy, ¶m); | ||
| 2365 | dpp->funcs->set_cursor_position(dpp, &pos_cpy, ¶m, hubp->curs_attr.width); | ||
| 2366 | } | ||
| 2367 | |||
| 2368 | static void dcn10_set_cursor_attribute(struct pipe_ctx *pipe_ctx) | ||
| 2369 | { | ||
| 2370 | struct dc_cursor_attributes *attributes = &pipe_ctx->stream->cursor_attributes; | ||
| 2371 | |||
| 2372 | pipe_ctx->plane_res.hubp->funcs->set_cursor_attributes( | ||
| 2373 | pipe_ctx->plane_res.hubp, attributes); | ||
| 2374 | pipe_ctx->plane_res.dpp->funcs->set_cursor_attributes( | ||
| 2375 | pipe_ctx->plane_res.dpp, attributes->color_format); | ||
| 2376 | } | ||
| 2377 | |||
| 2325 | static const struct hw_sequencer_funcs dcn10_funcs = { | 2378 | static const struct hw_sequencer_funcs dcn10_funcs = { |
| 2326 | .program_gamut_remap = program_gamut_remap, | 2379 | .program_gamut_remap = program_gamut_remap, |
| 2327 | .program_csc_matrix = program_csc_matrix, | 2380 | .program_csc_matrix = program_csc_matrix, |
| @@ -2362,6 +2415,8 @@ static const struct hw_sequencer_funcs dcn10_funcs = { · 34 unmodified lines | |||
| 2362 | .edp_backlight_control = hwss_edp_backlight_control, | 2415 | .edp_backlight_control = hwss_edp_backlight_control, |
| 2363 | .edp_power_control = hwss_edp_power_control, | 2416 | .edp_power_control = hwss_edp_power_control, |
| 2364 | .edp_wait_for_hpd_ready = hwss_edp_wait_for_hpd_ready, | 2417 | .edp_wait_for_hpd_ready = hwss_edp_wait_for_hpd_ready, |
| 2418 | .set_cursor_position = dcn10_set_cursor_position, | ||
| 2419 | .set_cursor_attribute = dcn10_set_cursor_attribute | ||
| 2365 | }; | 2420 | }; |
| 2366 | 2421 | ||
| 2367 | 2422 | ||
| @@ -123,8 +123,7 @@ struct link_encoder_funcs { · 122 unmodified lines | |||
| 123 | void (*enable_tmds_output)(struct link_encoder *enc, | 123 | void (*enable_tmds_output)(struct link_encoder *enc, |
| 124 | enum clock_source_id clock_source, | 124 | enum clock_source_id clock_source, |
| 125 | enum dc_color_depth color_depth, | 125 | enum dc_color_depth color_depth, |
| 126 | bool hdmi, | 126 | enum signal_type signal, |
| 127 | bool dual_link, | ||
| 128 | uint32_t pixel_clock); | 127 | uint32_t pixel_clock); |
| 129 | void (*enable_dp_output)(struct link_encoder *enc, | 128 | void (*enable_dp_output)(struct link_encoder *enc, |
| 130 | const struct dc_link_settings *link_settings, | 129 | const struct dc_link_settings *link_settings, |
| @@ -198,6 +198,9 @@ struct hw_sequencer_funcs { · 197 unmodified lines | |||
| 198 | bool enable); | 198 | bool enable); |
| 199 | void (*edp_wait_for_hpd_ready)(struct dc_link *link, bool power_up); | 199 | void (*edp_wait_for_hpd_ready)(struct dc_link *link, bool power_up); |
| 200 | 200 | ||
| 201 | void (*set_cursor_position)(struct pipe_ctx *pipe); | ||
| 202 | void (*set_cursor_attribute)(struct pipe_ctx *pipe); | ||
| 203 | |||
| 201 | }; | 204 | }; |
| 202 | 205 | ||
| 203 | void color_space_to_black_color( | 206 | void color_space_to_black_color( |
| @@ -217,7 +217,7 @@ bool dce110_vblank_set( · 216 unmodified lines | |||
| 217 | core_dc->current_state->res_ctx.pipe_ctx[pipe_offset].stream_res.tg; | 217 | core_dc->current_state->res_ctx.pipe_ctx[pipe_offset].stream_res.tg; |
| 218 | 218 | ||
| 219 | if (enable) { | 219 | if (enable) { |
| 220 | if (!tg->funcs->arm_vert_intr(tg, 2)) { | 220 | if (!tg || !tg->funcs->arm_vert_intr(tg, 2)) { |
| 221 | DC_ERROR("Failed to get VBLANK!\n"); | 221 | DC_ERROR("Failed to get VBLANK!\n"); |
| 222 | return false; | 222 | return false; |
| 223 | } | 223 | } |
| @@ -42,8 +42,7 @@ static void virtual_link_encoder_enable_tmds_output( · 41 unmodified lines | |||
| 42 | struct link_encoder *enc, | 42 | struct link_encoder *enc, |
| 43 | enum clock_source_id clock_source, | 43 | enum clock_source_id clock_source, |
| 44 | enum dc_color_depth color_depth, | 44 | enum dc_color_depth color_depth, |
| 45 | bool hdmi, | 45 | enum signal_type signal, |
| 46 | bool dual_link, | ||
| 47 | uint32_t pixel_clock) {} | 46 | uint32_t pixel_clock) {} |
| 48 | 47 | ||
| 49 | static void virtual_link_encoder_enable_dp_output( | 48 | static void virtual_link_encoder_enable_dp_output( |
| @@ -419,11 +419,6 @@ struct bios_event_info { · 418 unmodified lines | |||
| 419 | bool backlight_changed; | 419 | bool backlight_changed; |
| 420 | }; | 420 | }; |
| 421 | 421 | ||
| 422 | enum { | ||
| 423 | HDMI_PIXEL_CLOCK_IN_KHZ_297 = 297000, | ||
| 424 | TMDS_PIXEL_CLOCK_IN_KHZ_165 = 165000 | ||
| 425 | }; | ||
| 426 | |||
| 427 | /* | 422 | /* |
| 428 | * DFS-bypass flag | 423 | * DFS-bypass flag |
| 429 | */ | 424 | */ |
| @@ -26,6 +26,11 @@ · 25 unmodified lines | |||
| 26 | #ifndef __DC_SIGNAL_TYPES_H__ | 26 | #ifndef __DC_SIGNAL_TYPES_H__ |
| 27 | #define __DC_SIGNAL_TYPES_H__ | 27 | #define __DC_SIGNAL_TYPES_H__ |
| 28 | 28 | ||
| 29 | /* Minimum pixel clock, in KHz. For TMDS signal is 25.00 MHz */ | ||
| 30 | #define TMDS_MIN_PIXEL_CLOCK 25000 | ||
| 31 | /* Maximum pixel clock, in KHz. For TMDS signal is 165.00 MHz */ | ||
| 32 | #define TMDS_MAX_PIXEL_CLOCK 165000 | ||
| 33 | |||
| 29 | enum signal_type { | 34 | enum signal_type { |
| 30 | SIGNAL_TYPE_NONE = 0L, /* no signal */ | 35 | SIGNAL_TYPE_NONE = 0L, /* no signal */ |
| 31 | SIGNAL_TYPE_DVI_SINGLE_LINK = (1 << 0), | 36 | SIGNAL_TYPE_DVI_SINGLE_LINK = (1 << 0), |
| @@ -162,7 +162,7 @@ static int pp_hw_init(void *handle) · 161 unmodified lines | |||
| 162 | if(hwmgr->smumgr_funcs->start_smu(pp_handle->hwmgr)) { | 162 | if(hwmgr->smumgr_funcs->start_smu(pp_handle->hwmgr)) { |
| 163 | pr_err("smc start failed\n"); | 163 | pr_err("smc start failed\n"); |
| 164 | hwmgr->smumgr_funcs->smu_fini(pp_handle->hwmgr); | 164 | hwmgr->smumgr_funcs->smu_fini(pp_handle->hwmgr); |
| 165 | return -EINVAL;; | 165 | return -EINVAL; |
| 166 | } | 166 | } |
| 167 | if (ret == PP_DPM_DISABLED) | 167 | if (ret == PP_DPM_DISABLED) |
| 168 | goto exit; | 168 | goto exit; |
| @@ -2756,10 +2756,13 @@ static int smu7_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, · 2755 unmodified lines | |||
| 2756 | PHM_PlatformCaps_DisableMclkSwitchingForFrameLock); | 2756 | PHM_PlatformCaps_DisableMclkSwitchingForFrameLock); |
| 2757 | 2757 | ||
| 2758 | 2758 | ||
| 2759 | disable_mclk_switching = ((1 < info.display_count) || | 2759 | if (info.display_count == 0) |
| 2760 | disable_mclk_switching_for_frame_lock || | 2760 | disable_mclk_switching = false; |
| 2761 | smu7_vblank_too_short(hwmgr, mode_info.vblank_time_us) || | 2761 | else |
| 2762 | (mode_info.refresh_rate > 120)); | 2762 | disable_mclk_switching = ((1 < info.display_count) || |
| 2763 | disable_mclk_switching_for_frame_lock || | ||
| 2764 | smu7_vblank_too_short(hwmgr, mode_info.vblank_time_us) || | ||
| 2765 | (mode_info.refresh_rate > 120)); | ||
| 2763 | 2766 | ||
| 2764 | sclk = smu7_ps->performance_levels[0].engine_clock; | 2767 | sclk = smu7_ps->performance_levels[0].engine_clock; |
| 2765 | mclk = smu7_ps->performance_levels[0].memory_clock; | 2768 | mclk = smu7_ps->performance_levels[0].memory_clock; |
| @@ -4534,13 +4537,6 @@ static int smu7_set_power_profile_state(struct pp_hwmgr *hwmgr, · 1768 unmodified lines | |||
| 4534 | int tmp_result, result = 0; | 4537 | int tmp_result, result = 0; |
| 4535 | uint32_t sclk_mask = 0, mclk_mask = 0; | 4538 | uint32_t sclk_mask = 0, mclk_mask = 0; |
| 4536 | 4539 | ||
| 4537 | if (hwmgr->chip_id == CHIP_FIJI) { | ||
| 4538 | if (request->type == AMD_PP_GFX_PROFILE) | ||
| 4539 | smu7_enable_power_containment(hwmgr); | ||
| 4540 | else if (request->type == AMD_PP_COMPUTE_PROFILE) | ||
| 4541 | smu7_disable_power_containment(hwmgr); | ||
| 4542 | } | ||
| 4543 | |||
| 4544 | if (hwmgr->dpm_level != AMD_DPM_FORCED_LEVEL_AUTO) | 4540 | if (hwmgr->dpm_level != AMD_DPM_FORCED_LEVEL_AUTO) |
| 4545 | return -EINVAL; | 4541 | return -EINVAL; |
| 4546 | 4542 | ||
| @@ -3168,10 +3168,13 @@ static int vega10_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, · 3167 unmodified lines | |||
| 3168 | disable_mclk_switching_for_vr = PP_CAP(PHM_PlatformCaps_DisableMclkSwitchForVR); | 3168 | disable_mclk_switching_for_vr = PP_CAP(PHM_PlatformCaps_DisableMclkSwitchForVR); |
| 3169 | force_mclk_high = PP_CAP(PHM_PlatformCaps_ForceMclkHigh); | 3169 | force_mclk_high = PP_CAP(PHM_PlatformCaps_ForceMclkHigh); |
| 3170 | 3170 | ||
| 3171 | disable_mclk_switching = (info.display_count > 1) || | 3171 | if (info.display_count == 0) |
| 3172 | disable_mclk_switching_for_frame_lock || | 3172 | disable_mclk_switching = false; |
| 3173 | disable_mclk_switching_for_vr || | 3173 | else |
| 3174 | force_mclk_high; | 3174 | disable_mclk_switching = (info.display_count > 1) || |
| 3175 | disable_mclk_switching_for_frame_lock || | ||
| 3176 | disable_mclk_switching_for_vr || | ||
| 3177 | force_mclk_high; | ||
| 3175 | 3178 | ||
| 3176 | sclk = vega10_ps->performance_levels[0].gfx_clock; | 3179 | sclk = vega10_ps->performance_levels[0].gfx_clock; |
| 3177 | mclk = vega10_ps->performance_levels[0].mem_clock; | 3180 | mclk = vega10_ps->performance_levels[0].mem_clock; |
| @@ -294,22 +294,7 @@ static void cirrus_crtc_prepare(struct drm_crtc *crtc) · 293 unmodified lines | |||
| 294 | { | 294 | { |
| 295 | } | 295 | } |
| 296 | 296 | ||
| 297 | /* | 297 | static void cirrus_crtc_load_lut(struct drm_crtc *crtc) |
| 298 | * This is called after a mode is programmed. It should reverse anything done | ||
| 299 | * by the prepare function | ||
| 300 | */ | ||
| 301 | static void cirrus_crtc_commit(struct drm_crtc *crtc) | ||
| 302 | { | ||
| 303 | } | ||
| 304 | |||
| 305 | /* | ||
| 306 | * The core can pass us a set of gamma values to program. We actually only | ||
| 307 | * use this for 8-bit mode so can't perform smooth fades on deeper modes, | ||
| 308 | * but it's a requirement that we provide the function | ||
| 309 | */ | ||
| 310 | static int cirrus_crtc_gamma_set(struct drm_crtc *crtc, u16 *red, u16 *green, | ||
| 311 | u16 *blue, uint32_t size, | ||
| 312 | struct drm_modeset_acquire_ctx *ctx) | ||
| 313 | { | 298 | { |
| 314 | struct drm_device *dev = crtc->dev; | 299 | struct drm_device *dev = crtc->dev; |
| 315 | struct cirrus_device *cdev = dev->dev_private; | 300 | struct cirrus_device *cdev = dev->dev_private; |
| @@ -317,7 +302,7 @@ static int cirrus_crtc_gamma_set(struct drm_crtc *crtc, u16 *red, u16 *green, · 1 unmodified line | |||
| 317 | int i; | 302 | int i; |
| 318 | 303 | ||
| 319 | if (!crtc->enabled) | 304 | if (!crtc->enabled) |
| 320 | return 0; | 305 | return; |
| 321 | 306 | ||
| 322 | r = crtc->gamma_store; | 307 | r = crtc->gamma_store; |
| 323 | g = r + crtc->gamma_size; | 308 | g = r + crtc->gamma_size; |
| @@ -330,6 +315,27 @@ static int cirrus_crtc_gamma_set(struct drm_crtc *crtc, u16 *red, u16 *green, · 6 unmodified lines | |||
| 330 | WREG8(PALETTE_DATA, *g++ >> 8); | 315 | WREG8(PALETTE_DATA, *g++ >> 8); |
| 331 | WREG8(PALETTE_DATA, *b++ >> 8); | 316 | WREG8(PALETTE_DATA, *b++ >> 8); |
| 332 | } | 317 | } |
| 318 | } | ||
| 319 | |||
| 320 | /* | ||
| 321 | * This is called after a mode is programmed. It should reverse anything done | ||
| 322 | * by the prepare function | ||
| 323 | */ | ||
| 324 | static void cirrus_crtc_commit(struct drm_crtc *crtc) | ||
| 325 | { | ||
| 326 | cirrus_crtc_load_lut(crtc); | ||
| 327 | } | ||
| 328 | |||
| 329 | /* | ||
| 330 | * The core can pass us a set of gamma values to program. We actually only | ||
| 331 | * use this for 8-bit mode so can't perform smooth fades on deeper modes, | ||
| 332 | * but it's a requirement that we provide the function | ||
| 333 | */ | ||
| 334 | static int cirrus_crtc_gamma_set(struct drm_crtc *crtc, u16 *red, u16 *green, | ||
| 335 | u16 *blue, uint32_t size, | ||
| 336 | struct drm_modeset_acquire_ctx *ctx) | ||
| 337 | { | ||
| 338 | cirrus_crtc_load_lut(crtc); | ||
| 333 | 339 | ||
| 334 | return 0; | 340 | return 0; |
| 335 | } | 341 | } |
| @@ -1878,6 +1878,8 @@ int drm_atomic_helper_setup_commit(struct drm_atomic_state *state, · 1877 unmodified lines | |||
| 1878 | new_crtc_state->event->base.completion = &commit->flip_done; | 1878 | new_crtc_state->event->base.completion = &commit->flip_done; |
| 1879 | new_crtc_state->event->base.completion_release = release_crtc_commit; | 1879 | new_crtc_state->event->base.completion_release = release_crtc_commit; |
| 1880 | drm_crtc_commit_get(commit); | 1880 | drm_crtc_commit_get(commit); |
| 1881 | |||
| 1882 | commit->abort_completion = true; | ||
| 1881 | } | 1883 | } |
| 1882 | 1884 | ||
| 1883 | for_each_oldnew_connector_in_state(state, conn, old_conn_state, new_conn_state, i) { | 1885 | for_each_oldnew_connector_in_state(state, conn, old_conn_state, new_conn_state, i) { |
| @@ -3421,8 +3423,21 @@ EXPORT_SYMBOL(drm_atomic_helper_crtc_duplicate_state); · 1537 unmodified lines | |||
| 3421 | void __drm_atomic_helper_crtc_destroy_state(struct drm_crtc_state *state) | 3423 | void __drm_atomic_helper_crtc_destroy_state(struct drm_crtc_state *state) |
| 3422 | { | 3424 | { |
| 3423 | if (state->commit) { | 3425 | if (state->commit) { |
| 3426 | /* | ||
| 3427 | * In the event that a non-blocking commit returns | ||
| 3428 | * -ERESTARTSYS before the commit_tail work is queued, we will | ||
| 3429 | * have an extra reference to the commit object. Release it, if | ||
| 3430 | * the event has not been consumed by the worker. | ||
| 3431 | * | ||
| 3432 | * state->event may be freed, so we can't directly look at | ||
| 3433 | * state->event->base.completion. | ||
| 3434 | */ | ||
| 3435 | if (state->event && state->commit->abort_completion) | ||
| 3436 | drm_crtc_commit_put(state->commit); | ||
| 3437 | |||
| 3424 | kfree(state->commit->event); | 3438 | kfree(state->commit->event); |
| 3425 | state->commit->event = NULL; | 3439 | state->commit->event = NULL; |
| 3440 | |||
| 3426 | drm_crtc_commit_put(state->commit); | 3441 | drm_crtc_commit_put(state->commit); |
| 3427 | } | 3442 | } |
| 3428 | 3443 | ||
| @@ -113,6 +113,9 @@ static const struct edid_quirk { · 112 unmodified lines | |||
| 113 | /* AEO model 0 reports 8 bpc, but is a 6 bpc panel */ | 113 | /* AEO model 0 reports 8 bpc, but is a 6 bpc panel */ |
| 114 | { "AEO", 0, EDID_QUIRK_FORCE_6BPC }, | 114 | { "AEO", 0, EDID_QUIRK_FORCE_6BPC }, |
| 115 | 115 | ||
| 116 | /* CPT panel of Asus UX303LA reports 8 bpc, but is a 6 bpc panel */ | ||
| 117 | { "CPT", 0x17df, EDID_QUIRK_FORCE_6BPC }, | ||
| 118 | |||
| 116 | /* Belinea 10 15 55 */ | 119 | /* Belinea 10 15 55 */ |
| 117 | { "MAX", 1516, EDID_QUIRK_PREFER_LARGE_60 }, | 120 | { "MAX", 1516, EDID_QUIRK_PREFER_LARGE_60 }, |
| 118 | { "MAX", 0x77e, EDID_QUIRK_PREFER_LARGE_60 }, | 121 | { "MAX", 0x77e, EDID_QUIRK_PREFER_LARGE_60 }, |
| @@ -162,6 +165,24 @@ static const struct edid_quirk { · 43 unmodified lines | |||
| 162 | 165 | ||
| 163 | /* HTC Vive VR Headset */ | 166 | /* HTC Vive VR Headset */ |
| 164 | { "HVR", 0xaa01, EDID_QUIRK_NON_DESKTOP }, | 167 | { "HVR", 0xaa01, EDID_QUIRK_NON_DESKTOP }, |
| 168 | |||
| 169 | /* Oculus Rift DK1, DK2, and CV1 VR Headsets */ | ||
| 170 | { "OVR", 0x0001, EDID_QUIRK_NON_DESKTOP }, | ||
| 171 | { "OVR", 0x0003, EDID_QUIRK_NON_DESKTOP }, | ||
| 172 | { "OVR", 0x0004, EDID_QUIRK_NON_DESKTOP }, | ||
| 173 | |||
| 174 | /* Windows Mixed Reality Headsets */ | ||
| 175 | { "ACR", 0x7fce, EDID_QUIRK_NON_DESKTOP }, | ||
| 176 | { "HPN", 0x3515, EDID_QUIRK_NON_DESKTOP }, | ||
| 177 | { "LEN", 0x0408, EDID_QUIRK_NON_DESKTOP }, | ||
| 178 | { "LEN", 0xb800, EDID_QUIRK_NON_DESKTOP }, | ||
| 179 | { "FUJ", 0x1970, EDID_QUIRK_NON_DESKTOP }, | ||
| 180 | { "DEL", 0x7fce, EDID_QUIRK_NON_DESKTOP }, | ||
| 181 | { "SEC", 0x144a, EDID_QUIRK_NON_DESKTOP }, | ||
| 182 | { "AUS", 0xc102, EDID_QUIRK_NON_DESKTOP }, | ||
| 183 | |||
| 184 | /* Sony PlayStation VR Headset */ | ||
| 185 | { "SNY", 0x0704, EDID_QUIRK_NON_DESKTOP }, | ||
| 165 | }; | 186 | }; |
| 166 | 187 | ||
| 167 | /* | 188 | /* |
| @@ -121,6 +121,10 @@ int drm_mode_addfb(struct drm_device *dev, · 120 unmodified lines | |||
| 121 | r.pixel_format = drm_mode_legacy_fb_format(or->bpp, or->depth); | 121 | r.pixel_format = drm_mode_legacy_fb_format(or->bpp, or->depth); |
| 122 | r.handles[0] = or->handle; | 122 | r.handles[0] = or->handle; |
| 123 | 123 | ||
| 124 | if (r.pixel_format == DRM_FORMAT_XRGB2101010 && | ||
| 125 | dev->driver->driver_features & DRIVER_PREFER_XBGR_30BPP) | ||
| 126 | r.pixel_format = DRM_FORMAT_XBGR2101010; | ||
| 127 | |||
| 124 | ret = drm_mode_addfb2(dev, &r, file_priv); | 128 | ret = drm_mode_addfb2(dev, &r, file_priv); |
| 125 | if (ret) | 129 | if (ret) |
| 126 | return ret; | 130 | return ret; |
| @@ -836,9 +836,24 @@ struct drm_mm_node *drm_mm_scan_color_evict(struct drm_mm_scan *scan) · 835 unmodified lines | |||
| 836 | if (!mm->color_adjust) | 836 | if (!mm->color_adjust) |
| 837 | return NULL; | 837 | return NULL; |
| 838 | 838 | ||
| 839 | hole = list_first_entry(&mm->hole_stack, typeof(*hole), hole_stack); | 839 | /* |
| 840 | hole_start = __drm_mm_hole_node_start(hole); | 840 | * The hole found during scanning should ideally be the first element |
| 841 | hole_end = hole_start + hole->hole_size; | 841 | * in the hole_stack list, but due to side-effects in the driver it |
| 842 | * may not be. | ||
| 843 | */ | ||
| 844 | list_for_each_entry(hole, &mm->hole_stack, hole_stack) { | ||
| 845 | hole_start = __drm_mm_hole_node_start(hole); | ||
| 846 | hole_end = hole_start + hole->hole_size; | ||
| 847 | |||
| 848 | if (hole_start <= scan->hit_start && | ||
| 849 | hole_end >= scan->hit_end) | ||
| 850 | break; | ||
| 851 | } | ||
| 852 | |||
| 853 | /* We should only be called after we found the hole previously */ | ||
| 854 | DRM_MM_BUG_ON(&hole->hole_stack == &mm->hole_stack); | ||
| 855 | if (unlikely(&hole->hole_stack == &mm->hole_stack)) | ||
| 856 | return NULL; | ||
| 842 | 857 | ||
| 843 | DRM_MM_BUG_ON(hole_start > scan->hit_start); | 858 | DRM_MM_BUG_ON(hole_start > scan->hit_start); |
| 844 | DRM_MM_BUG_ON(hole_end < scan->hit_end); | 859 | DRM_MM_BUG_ON(hole_end < scan->hit_end); |
| @@ -653,6 +653,26 @@ out: · 652 unmodified lines | |||
| 653 | schedule_delayed_work(delayed_work, DRM_OUTPUT_POLL_PERIOD); | 653 | schedule_delayed_work(delayed_work, DRM_OUTPUT_POLL_PERIOD); |
| 654 | } | 654 | } |
| 655 | 655 | ||
| 656 | /** | ||
| 657 | * drm_kms_helper_is_poll_worker - is %current task an output poll worker? | ||
| 658 | * | ||
| 659 | * Determine if %current task is an output poll worker. This can be used | ||
| 660 | * to select distinct code paths for output polling versus other contexts. | ||
| 661 | * | ||
| 662 | * One use case is to avoid a deadlock between the output poll worker and | ||
| 663 | * the autosuspend worker wherein the latter waits for polling to finish | ||
| 664 | * upon calling drm_kms_helper_poll_disable(), while the former waits for | ||
| 665 | * runtime suspend to finish upon calling pm_runtime_get_sync() in a | ||
| 666 | * connector ->detect hook. | ||
| 667 | */ | ||
| 668 | bool drm_kms_helper_is_poll_worker(void) | ||
| 669 | { | ||
| 670 | struct work_struct *work = current_work(); | ||
| 671 | |||
| 672 | return work && work->func == output_poll_execute; | ||
| 673 | } | ||
| 674 | EXPORT_SYMBOL(drm_kms_helper_is_poll_worker); | ||
| 675 | |||
| 656 | /** | 676 | /** |
| 657 | * drm_kms_helper_poll_disable - disable output polling | 677 | * drm_kms_helper_poll_disable - disable output polling |
| 658 | * @dev: drm_device | 678 | * @dev: drm_device |
| @@ -286,7 +286,6 @@ static int g2d_init_cmdlist(struct g2d_data *g2d) · 285 unmodified lines | |||
| 286 | 286 | ||
| 287 | node = kcalloc(G2D_CMDLIST_NUM, sizeof(*node), GFP_KERNEL); | 287 | node = kcalloc(G2D_CMDLIST_NUM, sizeof(*node), GFP_KERNEL); |
| 288 | if (!node) { | 288 | if (!node) { |
| 289 | dev_err(dev, "failed to allocate memory\n"); | ||
| 290 | ret = -ENOMEM; | 289 | ret = -ENOMEM; |
| 291 | goto err; | 290 | goto err; |
| 292 | } | 291 | } |
| @@ -926,7 +925,7 @@ static void g2d_finish_event(struct g2d_data *g2d, u32 cmdlist_no) · 633 unmodified lines | |||
| 926 | struct drm_device *drm_dev = g2d->subdrv.drm_dev; | 925 | struct drm_device *drm_dev = g2d->subdrv.drm_dev; |
| 927 | struct g2d_runqueue_node *runqueue_node = g2d->runqueue_node; | 926 | struct g2d_runqueue_node *runqueue_node = g2d->runqueue_node; |
| 928 | struct drm_exynos_pending_g2d_event *e; | 927 | struct drm_exynos_pending_g2d_event *e; |
| 929 | struct timeval now; | 928 | struct timespec64 now; |
| 930 | 929 | ||
| 931 | if (list_empty(&runqueue_node->event_list)) | 930 | if (list_empty(&runqueue_node->event_list)) |
| 932 | return; | 931 | return; |
| @@ -934,9 +933,9 @@ static void g2d_finish_event(struct g2d_data *g2d, u32 cmdlist_no) · 1 unmodified line | |||
| 934 | e = list_first_entry(&runqueue_node->event_list, | 933 | e = list_first_entry(&runqueue_node->event_list, |
| 935 | struct drm_exynos_pending_g2d_event, base.link); | 934 | struct drm_exynos_pending_g2d_event, base.link); |
| 936 | 935 | ||
| 937 | do_gettimeofday(&now); | 936 | ktime_get_ts64(&now); |
| 938 | e->event.tv_sec = now.tv_sec; | 937 | e->event.tv_sec = now.tv_sec; |
| 939 | e->event.tv_usec = now.tv_usec; | 938 | e->event.tv_usec = now.tv_nsec / NSEC_PER_USEC; |
| 940 | e->event.cmdlist_no = cmdlist_no; | 939 | e->event.cmdlist_no = cmdlist_no; |
| 941 | 940 | ||
| 942 | drm_send_event(drm_dev, &e->base); | 941 | drm_send_event(drm_dev, &e->base); |
| @@ -1358,10 +1357,9 @@ int exynos_g2d_exec_ioctl(struct drm_device *drm_dev, void *data, · 415 unmodified lines | |||
| 1358 | return -EFAULT; | 1357 | return -EFAULT; |
| 1359 | 1358 | ||
| 1360 | runqueue_node = kmem_cache_alloc(g2d->runqueue_slab, GFP_KERNEL); | 1359 | runqueue_node = kmem_cache_alloc(g2d->runqueue_slab, GFP_KERNEL); |
| 1361 | if (!runqueue_node) { | 1360 | if (!runqueue_node) |
| 1362 | dev_err(dev, "failed to allocate memory\n"); | ||
| 1363 | return -ENOMEM; | 1361 | return -ENOMEM; |
| 1364 | } | 1362 | |
| 1365 | run_cmdlist = &runqueue_node->run_cmdlist; | 1363 | run_cmdlist = &runqueue_node->run_cmdlist; |
| 1366 | event_list = &runqueue_node->event_list; | 1364 | event_list = &runqueue_node->event_list; |
| 1367 | INIT_LIST_HEAD(run_cmdlist); | 1365 | INIT_LIST_HEAD(run_cmdlist); |
| @@ -1,19 +0,0 @@ | |||
| 1 | /* | ||
| 2 | * Copyright (c) 2012 Samsung Electronics Co., Ltd. | ||
| 3 | * | ||
| 4 | * Authors: | ||
| 5 | * YoungJun Cho <yj44.cho@samsung.com> | ||
| 6 | * Eunchul Kim <chulspro.kim@samsung.com> | ||
| 7 | * | ||
| 8 | * This program is free software; you can redistribute it and/or modify it | ||
| 9 | * under the terms of the GNU General Public License as published by the | ||
| 10 | * Free Software Foundation; either version 2 of the License, or (at your | ||
| 11 | * option) any later version. | ||
| 12 | */ | ||
| 13 | |||
| 14 | #ifndef _EXYNOS_DRM_ROTATOR_H_ | ||
| 15 | #define _EXYNOS_DRM_ROTATOR_H_ | ||
| 16 | |||
| 17 | /* TODO */ | ||
| 18 | |||
| 19 | #endif | ||
| @@ -1068,10 +1068,13 @@ static void hdmi_audio_config(struct hdmi_context *hdata) · 1067 unmodified lines | |||
| 1068 | /* Configuration I2S input ports. Configure I2S_PIN_SEL_0~4 */ | 1068 | /* Configuration I2S input ports. Configure I2S_PIN_SEL_0~4 */ |
| 1069 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_0, HDMI_I2S_SEL_SCLK(5) | 1069 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_0, HDMI_I2S_SEL_SCLK(5) |
| 1070 | | HDMI_I2S_SEL_LRCK(6)); | 1070 | | HDMI_I2S_SEL_LRCK(6)); |
| 1071 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_1, HDMI_I2S_SEL_SDATA1(1) | 1071 | |
| 1072 | | HDMI_I2S_SEL_SDATA2(4)); | 1072 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_1, HDMI_I2S_SEL_SDATA1(3) |
| 1073 | | HDMI_I2S_SEL_SDATA0(4)); | ||
| 1074 | |||
| 1073 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_2, HDMI_I2S_SEL_SDATA3(1) | 1075 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_2, HDMI_I2S_SEL_SDATA3(1) |
| 1074 | | HDMI_I2S_SEL_SDATA2(2)); | 1076 | | HDMI_I2S_SEL_SDATA2(2)); |
| 1077 | |||
| 1075 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_3, HDMI_I2S_SEL_DSD(0)); | 1078 | hdmi_reg_writeb(hdata, HDMI_I2S_PIN_SEL_3, HDMI_I2S_SEL_DSD(0)); |
| 1076 | 1079 | ||
| 1077 | /* I2S_CON_1 & 2 */ | 1080 | /* I2S_CON_1 & 2 */ |
| @@ -569,7 +569,7 @@ · 568 unmodified lines | |||
| 569 | #define EXYNOS_CIIMGEFF_FIN_EMBOSSING (4 << 26) | 569 | #define EXYNOS_CIIMGEFF_FIN_EMBOSSING (4 << 26) |
| 570 | #define EXYNOS_CIIMGEFF_FIN_SILHOUETTE (5 << 26) | 570 | #define EXYNOS_CIIMGEFF_FIN_SILHOUETTE (5 << 26) |
| 571 | #define EXYNOS_CIIMGEFF_FIN_MASK (7 << 26) | 571 | #define EXYNOS_CIIMGEFF_FIN_MASK (7 << 26) |
| 572 | #define EXYNOS_CIIMGEFF_PAT_CBCR_MASK ((0xff < 13) | (0xff < 0)) | 572 | #define EXYNOS_CIIMGEFF_PAT_CBCR_MASK ((0xff << 13) | (0xff << 0)) |
| 573 | 573 | ||
| 574 | /* Real input DMA size register */ | 574 | /* Real input DMA size register */ |
| 575 | #define EXYNOS_CIREAL_ISIZE_AUTOLOAD_ENABLE (1 << 31) | 575 | #define EXYNOS_CIREAL_ISIZE_AUTOLOAD_ENABLE (1 << 31) |
| @@ -464,7 +464,7 @@ · 463 unmodified lines | |||
| 464 | 464 | ||
| 465 | /* I2S_PIN_SEL_1 */ | 465 | /* I2S_PIN_SEL_1 */ |
| 466 | #define HDMI_I2S_SEL_SDATA1(x) (((x) & 0x7) << 4) | 466 | #define HDMI_I2S_SEL_SDATA1(x) (((x) & 0x7) << 4) |
| 467 | #define HDMI_I2S_SEL_SDATA2(x) ((x) & 0x7) | 467 | #define HDMI_I2S_SEL_SDATA0(x) ((x) & 0x7) |
| 468 | 468 | ||
| 469 | /* I2S_PIN_SEL_2 */ | 469 | /* I2S_PIN_SEL_2 */ |
| 470 | #define HDMI_I2S_SEL_SDATA3(x) (((x) & 0x7) << 4) | 470 | #define HDMI_I2S_SEL_SDATA3(x) (((x) & 0x7) << 4) |
| @@ -733,6 +733,25 @@ static ssize_t intel_vgpu_rw(struct mdev_device *mdev, char *buf, · 732 unmodified lines | |||
| 733 | return ret == 0 ? count : ret; | 733 | return ret == 0 ? count : ret; |
| 734 | } | 734 | } |
| 735 | 735 | ||
| 736 | static bool gtt_entry(struct mdev_device *mdev, loff_t *ppos) | ||
| 737 | { | ||
| 738 | struct intel_vgpu *vgpu = mdev_get_drvdata(mdev); | ||
| 739 | unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos); | ||
| 740 | struct intel_gvt *gvt = vgpu->gvt; | ||
| 741 | int offset; | ||
| 742 | |||
| 743 | /* Only allow MMIO GGTT entry access */ | ||
| 744 | if (index != PCI_BASE_ADDRESS_0) | ||
| 745 | return false; | ||
| 746 | |||
| 747 | offset = (u64)(*ppos & VFIO_PCI_OFFSET_MASK) - | ||
| 748 | intel_vgpu_get_bar_gpa(vgpu, PCI_BASE_ADDRESS_0); | ||
| 749 | |||
| 750 | return (offset >= gvt->device_info.gtt_start_offset && | ||
| 751 | offset < gvt->device_info.gtt_start_offset + gvt_ggtt_sz(gvt)) ? | ||
| 752 | true : false; | ||
| 753 | } | ||
| 754 | |||
| 736 | static ssize_t intel_vgpu_read(struct mdev_device *mdev, char __user *buf, | 755 | static ssize_t intel_vgpu_read(struct mdev_device *mdev, char __user *buf, |
| 737 | size_t count, loff_t *ppos) | 756 | size_t count, loff_t *ppos) |
| 738 | { | 757 | { |
| @@ -742,7 +761,21 @@ static ssize_t intel_vgpu_read(struct mdev_device *mdev, char __user *buf, · 3 unmodified lines | |||
| 742 | while (count) { | 761 | while (count) { |
| 743 | size_t filled; | 762 | size_t filled; |
| 744 | 763 | ||
| 745 | if (count >= 4 && !(*ppos % 4)) { | 764 | /* Only support GGTT entry 8 bytes read */ |
| 765 | if (count >= 8 && !(*ppos % 8) && | ||
| 766 | gtt_entry(mdev, ppos)) { | ||
| 767 | u64 val; | ||
| 768 | |||
| 769 | ret = intel_vgpu_rw(mdev, (char *)&val, sizeof(val), | ||
| 770 | ppos, false); | ||
| 771 | if (ret <= 0) | ||
| 772 | goto read_err; | ||
| 773 | |||
| 774 | if (copy_to_user(buf, &val, sizeof(val))) | ||
| 775 | goto read_err; | ||
| 776 | |||
| 777 | filled = 8; | ||
| 778 | } else if (count >= 4 && !(*ppos % 4)) { | ||
| 746 | u32 val; | 779 | u32 val; |
| 747 | 780 | ||
| 748 | ret = intel_vgpu_rw(mdev, (char *)&val, sizeof(val), | 781 | ret = intel_vgpu_rw(mdev, (char *)&val, sizeof(val), |
| @@ -802,7 +835,21 @@ static ssize_t intel_vgpu_write(struct mdev_device *mdev, · 53 unmodified lines | |||
| 802 | while (count) { | 835 | while (count) { |
| 803 | size_t filled; | 836 | size_t filled; |
| 804 | 837 | ||
| 805 | if (count >= 4 && !(*ppos % 4)) { | 838 | /* Only support GGTT entry 8 bytes write */ |
| 839 | if (count >= 8 && !(*ppos % 8) && | ||
| 840 | gtt_entry(mdev, ppos)) { | ||
| 841 | u64 val; | ||
| 842 | |||
| 843 | if (copy_from_user(&val, buf, sizeof(val))) | ||
| 844 | goto write_err; | ||
| 845 | |||
| 846 | ret = intel_vgpu_rw(mdev, (char *)&val, sizeof(val), | ||
| 847 | ppos, true); | ||
| 848 | if (ret <= 0) | ||
| 849 | goto write_err; | ||
| 850 | |||
| 851 | filled = 8; | ||
| 852 | } else if (count >= 4 && !(*ppos % 4)) { | ||
| 806 | u32 val; | 853 | u32 val; |
| 807 | 854 | ||
| 808 | if (copy_from_user(&val, buf, sizeof(val))) | 855 | if (copy_from_user(&val, buf, sizeof(val))) |
| @@ -118,6 +118,7 @@ static struct engine_mmio gen9_engine_mmio_list[] __cacheline_aligned = { · 117 unmodified lines | |||
| 118 | {RCS, HALF_SLICE_CHICKEN3, 0xffff, true}, /* 0xe184 */ | 118 | {RCS, HALF_SLICE_CHICKEN3, 0xffff, true}, /* 0xe184 */ |
| 119 | {RCS, GEN9_HALF_SLICE_CHICKEN5, 0xffff, true}, /* 0xe188 */ | 119 | {RCS, GEN9_HALF_SLICE_CHICKEN5, 0xffff, true}, /* 0xe188 */ |
| 120 | {RCS, GEN9_HALF_SLICE_CHICKEN7, 0xffff, true}, /* 0xe194 */ | 120 | {RCS, GEN9_HALF_SLICE_CHICKEN7, 0xffff, true}, /* 0xe194 */ |
| 121 | {RCS, GEN8_ROW_CHICKEN, 0xffff, true}, /* 0xe4f0 */ | ||
| 121 | {RCS, TRVATTL3PTRDW(0), 0, false}, /* 0x4de0 */ | 122 | {RCS, TRVATTL3PTRDW(0), 0, false}, /* 0x4de0 */ |
| 122 | {RCS, TRVATTL3PTRDW(1), 0, false}, /* 0x4de4 */ | 123 | {RCS, TRVATTL3PTRDW(1), 0, false}, /* 0x4de4 */ |
| 123 | {RCS, TRNULLDETCT, 0, false}, /* 0x4de8 */ | 124 | {RCS, TRNULLDETCT, 0, false}, /* 0x4de8 */ |
| @@ -333,7 +333,7 @@ TRACE_EVENT(render_mmio, · 332 unmodified lines | |||
| 333 | TP_PROTO(int old_id, int new_id, char *action, unsigned int reg, | 333 | TP_PROTO(int old_id, int new_id, char *action, unsigned int reg, |
| 334 | unsigned int old_val, unsigned int new_val), | 334 | unsigned int old_val, unsigned int new_val), |
| 335 | 335 | ||
| 336 | TP_ARGS(old_id, new_id, action, reg, new_val, old_val), | 336 | TP_ARGS(old_id, new_id, action, reg, old_val, new_val), |
| 337 | 337 | ||
| 338 | TP_STRUCT__entry( | 338 | TP_STRUCT__entry( |
| 339 | __field(int, old_id) | 339 | __field(int, old_id) |
| @@ -1433,19 +1433,7 @@ void i915_driver_unload(struct drm_device *dev) · 1432 unmodified lines | |||
| 1433 | 1433 | ||
| 1434 | intel_modeset_cleanup(dev); | 1434 | intel_modeset_cleanup(dev); |
| 1435 | 1435 | ||
| 1436 | /* | 1436 | intel_bios_cleanup(dev_priv); |
| 1437 | * free the memory space allocated for the child device | ||
| 1438 | * config parsed from VBT | ||
| 1439 | */ | ||
| 1440 | if (dev_priv->vbt.child_dev && dev_priv->vbt.child_dev_num) { | ||
| 1441 | kfree(dev_priv->vbt.child_dev); | ||
| 1442 | dev_priv->vbt.child_dev = NULL; | ||
| 1443 | dev_priv->vbt.child_dev_num = 0; | ||
| 1444 | } | ||
| 1445 | kfree(dev_priv->vbt.sdvo_lvds_vbt_mode); | ||
| 1446 | dev_priv->vbt.sdvo_lvds_vbt_mode = NULL; | ||
| 1447 | kfree(dev_priv->vbt.lfp_lvds_vbt_mode); | ||
| 1448 | dev_priv->vbt.lfp_lvds_vbt_mode = NULL; | ||
| 1449 | 1437 | ||
| 1450 | vga_switcheroo_unregister_client(pdev); | 1438 | vga_switcheroo_unregister_client(pdev); |
| 1451 | vga_client_register(pdev, NULL, NULL, NULL); | 1439 | vga_client_register(pdev, NULL, NULL, NULL); |
| @@ -1349,6 +1349,7 @@ struct intel_vbt_data { · 1348 unmodified lines | |||
| 1349 | u32 size; | 1349 | u32 size; |
| 1350 | u8 *data; | 1350 | u8 *data; |
| 1351 | const u8 *sequence[MIPI_SEQ_MAX]; | 1351 | const u8 *sequence[MIPI_SEQ_MAX]; |
| 1352 | u8 *deassert_seq; /* Used by fixup_mipi_sequences() */ | ||
| 1352 | } dsi; | 1353 | } dsi; |
| 1353 | 1354 | ||
| 1354 | int crt_ddc_pin; | 1355 | int crt_ddc_pin; |
| @@ -3657,6 +3658,7 @@ extern void intel_i2c_reset(struct drm_i915_private *dev_priv); · 2302 unmodified lines | |||
| 3657 | 3658 | ||
| 3658 | /* intel_bios.c */ | 3659 | /* intel_bios.c */ |
| 3659 | void intel_bios_init(struct drm_i915_private *dev_priv); | 3660 | void intel_bios_init(struct drm_i915_private *dev_priv); |
| 3661 | void intel_bios_cleanup(struct drm_i915_private *dev_priv); | ||
| 3660 | bool intel_bios_is_valid_vbt(const void *buf, size_t size); | 3662 | bool intel_bios_is_valid_vbt(const void *buf, size_t size); |
| 3661 | bool intel_bios_is_tv_present(struct drm_i915_private *dev_priv); | 3663 | bool intel_bios_is_tv_present(struct drm_i915_private *dev_priv); |
| 3662 | bool intel_bios_is_lvds_present(struct drm_i915_private *dev_priv, u8 *i2c_pin); | 3664 | bool intel_bios_is_lvds_present(struct drm_i915_private *dev_priv, u8 *i2c_pin); |
| @@ -3205,8 +3205,10 @@ void i915_gem_set_wedged(struct drm_i915_private *i915) · 3204 unmodified lines | |||
| 3205 | * rolling the global seqno forward (since this would complete requests | 3205 | * rolling the global seqno forward (since this would complete requests |
| 3206 | * for which we haven't set the fence error to EIO yet). | 3206 | * for which we haven't set the fence error to EIO yet). |
| 3207 | */ | 3207 | */ |
| 3208 | for_each_engine(engine, i915, id) | 3208 | for_each_engine(engine, i915, id) { |
| 3209 | i915_gem_reset_prepare_engine(engine); | ||
| 3209 | engine->submit_request = nop_submit_request; | 3210 | engine->submit_request = nop_submit_request; |
| 3211 | } | ||
| 3210 | 3212 | ||
| 3211 | /* | 3213 | /* |
| 3212 | * Make sure no one is running the old callback before we proceed with | 3214 | * Make sure no one is running the old callback before we proceed with |
| @@ -3244,6 +3246,8 @@ void i915_gem_set_wedged(struct drm_i915_private *i915) · 31 unmodified lines | |||
| 3244 | intel_engine_init_global_seqno(engine, | 3246 | intel_engine_init_global_seqno(engine, |
| 3245 | intel_engine_last_submit(engine)); | 3247 | intel_engine_last_submit(engine)); |
| 3246 | spin_unlock_irqrestore(&engine->timeline->lock, flags); | 3248 | spin_unlock_irqrestore(&engine->timeline->lock, flags); |
| 3249 | |||
| 3250 | i915_gem_reset_finish_engine(engine); | ||
| 3247 | } | 3251 | } |
| 3248 | 3252 | ||
| 3249 | set_bit(I915_WEDGED, &i915->gpu_error.flags); | 3253 | set_bit(I915_WEDGED, &i915->gpu_error.flags); |
| @@ -803,7 +803,7 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data, · 802 unmodified lines | |||
| 803 | 803 | ||
| 804 | case I915_CONTEXT_PARAM_PRIORITY: | 804 | case I915_CONTEXT_PARAM_PRIORITY: |
| 805 | { | 805 | { |
| 806 | int priority = args->value; | 806 | s64 priority = args->value; |
| 807 | 807 | ||
| 808 | if (args->size) | 808 | if (args->size) |
| 809 | ret = -EINVAL; | 809 | ret = -EINVAL; |
| @@ -505,6 +505,8 @@ eb_add_vma(struct i915_execbuffer *eb, unsigned int i, struct i915_vma *vma) · 504 unmodified lines | |||
| 505 | list_add_tail(&vma->exec_link, &eb->unbound); | 505 | list_add_tail(&vma->exec_link, &eb->unbound); |
| 506 | if (drm_mm_node_allocated(&vma->node)) | 506 | if (drm_mm_node_allocated(&vma->node)) |
| 507 | err = i915_vma_unbind(vma); | 507 | err = i915_vma_unbind(vma); |
| 508 | if (unlikely(err)) | ||
| 509 | vma->exec_flags = NULL; | ||
| 508 | } | 510 | } |
| 509 | return err; | 511 | return err; |
| 510 | } | 512 | } |
| @@ -2410,7 +2412,7 @@ err_request: · 1899 unmodified lines | |||
| 2410 | if (out_fence) { | 2412 | if (out_fence) { |
| 2411 | if (err == 0) { | 2413 | if (err == 0) { |
| 2412 | fd_install(out_fence_fd, out_fence->file); | 2414 | fd_install(out_fence_fd, out_fence->file); |
| 2413 | args->rsvd2 &= GENMASK_ULL(0, 31); /* keep in-fence */ | 2415 | args->rsvd2 &= GENMASK_ULL(31, 0); /* keep in-fence */ |
| 2414 | args->rsvd2 |= (u64)out_fence_fd << 32; | 2416 | args->rsvd2 |= (u64)out_fence_fd << 32; |
| 2415 | out_fence_fd = -1; | 2417 | out_fence_fd = -1; |
| 2416 | } else { | 2418 | } else { |
| @@ -476,8 +476,6 @@ void __i915_gem_request_submit(struct drm_i915_gem_request *request) · 475 unmodified lines | |||
| 476 | GEM_BUG_ON(!irqs_disabled()); | 476 | GEM_BUG_ON(!irqs_disabled()); |
| 477 | lockdep_assert_held(&engine->timeline->lock); | 477 | lockdep_assert_held(&engine->timeline->lock); |
| 478 | 478 | ||
| 479 | trace_i915_gem_request_execute(request); | ||
| 480 | |||
| 481 | /* Transfer from per-context onto the global per-engine timeline */ | 479 | /* Transfer from per-context onto the global per-engine timeline */ |
| 482 | timeline = engine->timeline; | 480 | timeline = engine->timeline; |
| 483 | GEM_BUG_ON(timeline == request->timeline); | 481 | GEM_BUG_ON(timeline == request->timeline); |
| @@ -501,6 +499,8 @@ void __i915_gem_request_submit(struct drm_i915_gem_request *request) · 17 unmodified lines | |||
| 501 | list_move_tail(&request->link, &timeline->requests); | 499 | list_move_tail(&request->link, &timeline->requests); |
| 502 | spin_unlock(&request->timeline->lock); | 500 | spin_unlock(&request->timeline->lock); |
| 503 | 501 | ||
| 502 | trace_i915_gem_request_execute(request); | ||
| 503 | |||
| 504 | wake_up_all(&request->execute); | 504 | wake_up_all(&request->execute); |
| 505 | } | 505 | } |
| 506 | 506 | ||
| @@ -84,9 +84,9 @@ show_test_oa_id(struct device *kdev, struct device_attribute *attr, char *buf) · 83 unmodified lines | |||
| 84 | void | 84 | void |
| 85 | i915_perf_load_test_config_cflgt3(struct drm_i915_private *dev_priv) | 85 | i915_perf_load_test_config_cflgt3(struct drm_i915_private *dev_priv) |
| 86 | { | 86 | { |
| 87 | strncpy(dev_priv->perf.oa.test_config.uuid, | 87 | strlcpy(dev_priv->perf.oa.test_config.uuid, |
| 88 | "577e8e2c-3fa0-4875-8743-3538d585e3b0", | 88 | "577e8e2c-3fa0-4875-8743-3538d585e3b0", |
| 89 | UUID_STRING_LEN); | 89 | sizeof(dev_priv->perf.oa.test_config.uuid)); |
| 90 | dev_priv->perf.oa.test_config.id = 1; | 90 | dev_priv->perf.oa.test_config.id = 1; |
| 91 | 91 | ||
| 92 | dev_priv->perf.oa.test_config.mux_regs = mux_config_test_oa; | 92 | dev_priv->perf.oa.test_config.mux_regs = mux_config_test_oa; |
| @@ -96,9 +96,9 @@ show_test_oa_id(struct device *kdev, struct device_attribute *attr, char *buf) · 95 unmodified lines | |||
| 96 | void | 96 | void |
| 97 | i915_perf_load_test_config_cnl(struct drm_i915_private *dev_priv) | 97 | i915_perf_load_test_config_cnl(struct drm_i915_private *dev_priv) |
| 98 | { | 98 | { |
| 99 | strncpy(dev_priv->perf.oa.test_config.uuid, | 99 | strlcpy(dev_priv->perf.oa.test_config.uuid, |
| 100 | "db41edd4-d8e7-4730-ad11-b9a2d6833503", | 100 | "db41edd4-d8e7-4730-ad11-b9a2d6833503", |
| 101 | UUID_STRING_LEN); | 101 | sizeof(dev_priv->perf.oa.test_config.uuid)); |
| 102 | dev_priv->perf.oa.test_config.id = 1; | 102 | dev_priv->perf.oa.test_config.id = 1; |
| 103 | 103 | ||
| 104 | dev_priv->perf.oa.test_config.mux_regs = mux_config_test_oa; | 104 | dev_priv->perf.oa.test_config.mux_regs = mux_config_test_oa; |
| @@ -1303,9 +1303,8 @@ static void i915_oa_stream_destroy(struct i915_perf_stream *stream) · 1302 unmodified lines | |||
| 1303 | */ | 1303 | */ |
| 1304 | mutex_lock(&dev_priv->drm.struct_mutex); | 1304 | mutex_lock(&dev_priv->drm.struct_mutex); |
| 1305 | dev_priv->perf.oa.exclusive_stream = NULL; | 1305 | dev_priv->perf.oa.exclusive_stream = NULL; |
| 1306 | mutex_unlock(&dev_priv->drm.struct_mutex); | ||
| 1307 | |||
| 1308 | dev_priv->perf.oa.ops.disable_metric_set(dev_priv); | 1306 | dev_priv->perf.oa.ops.disable_metric_set(dev_priv); |
| 1307 | mutex_unlock(&dev_priv->drm.struct_mutex); | ||
| 1309 | 1308 | ||
| 1310 | free_oa_buffer(dev_priv); | 1309 | free_oa_buffer(dev_priv); |
| 1311 | 1310 | ||
| @@ -1756,22 +1755,13 @@ static int gen8_switch_to_updated_kernel_context(struct drm_i915_private *dev_pr · 444 unmodified lines | |||
| 1756 | * Note: it's only the RCS/Render context that has any OA state. | 1755 | * Note: it's only the RCS/Render context that has any OA state. |
| 1757 | */ | 1756 | */ |
| 1758 | static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv, | 1757 | static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv, |
| 1759 | const struct i915_oa_config *oa_config, | 1758 | const struct i915_oa_config *oa_config) |
| 1760 | bool interruptible) | ||
| 1761 | { | 1759 | { |
| 1762 | struct i915_gem_context *ctx; | 1760 | struct i915_gem_context *ctx; |
| 1763 | int ret; | 1761 | int ret; |
| 1764 | unsigned int wait_flags = I915_WAIT_LOCKED; | 1762 | unsigned int wait_flags = I915_WAIT_LOCKED; |
| 1765 | 1763 | ||
| 1766 | if (interruptible) { | 1764 | lockdep_assert_held(&dev_priv->drm.struct_mutex); |
| 1767 | ret = i915_mutex_lock_interruptible(&dev_priv->drm); | ||
| 1768 | if (ret) | ||
| 1769 | return ret; | ||
| 1770 | |||
| 1771 | wait_flags |= I915_WAIT_INTERRUPTIBLE; | ||
| 1772 | } else { | ||
| 1773 | mutex_lock(&dev_priv->drm.struct_mutex); | ||
| 1774 | } | ||
| 1775 | 1765 | ||
| 1776 | /* Switch away from any user context. */ | 1766 | /* Switch away from any user context. */ |
| 1777 | ret = gen8_switch_to_updated_kernel_context(dev_priv, oa_config); | 1767 | ret = gen8_switch_to_updated_kernel_context(dev_priv, oa_config); |
| @@ -1819,8 +1809,6 @@ static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv, · 41 unmodified lines | |||
| 1819 | } | 1809 | } |
| 1820 | 1810 | ||
| 1821 | out: | 1811 | out: |
| 1822 | mutex_unlock(&dev_priv->drm.struct_mutex); | ||
| 1823 | |||
| 1824 | return ret; | 1812 | return ret; |
| 1825 | } | 1813 | } |
| 1826 | 1814 | ||
| @@ -1863,7 +1851,7 @@ static int gen8_enable_metric_set(struct drm_i915_private *dev_priv, · 36 unmodified lines | |||
| 1863 | * to make sure all slices/subslices are ON before writing to NOA | 1851 | * to make sure all slices/subslices are ON before writing to NOA |
| 1864 | * registers. | 1852 | * registers. |
| 1865 | */ | 1853 | */ |
| 1866 | ret = gen8_configure_all_contexts(dev_priv, oa_config, true); | 1854 | ret = gen8_configure_all_contexts(dev_priv, oa_config); |
| 1867 | if (ret) | 1855 | if (ret) |
| 1868 | return ret; | 1856 | return ret; |
| 1869 | 1857 | ||
| @@ -1878,7 +1866,7 @@ static int gen8_enable_metric_set(struct drm_i915_private *dev_priv, · 8 unmodified lines | |||
| 1878 | static void gen8_disable_metric_set(struct drm_i915_private *dev_priv) | 1866 | static void gen8_disable_metric_set(struct drm_i915_private *dev_priv) |
| 1879 | { | 1867 | { |
| 1880 | /* Reset all contexts' slices/subslices configurations. */ | 1868 | /* Reset all contexts' slices/subslices configurations. */ |
| 1881 | gen8_configure_all_contexts(dev_priv, NULL, false); | 1869 | gen8_configure_all_contexts(dev_priv, NULL); |
| 1882 | 1870 | ||
| 1883 | I915_WRITE(GDT_CHICKEN_BITS, (I915_READ(GDT_CHICKEN_BITS) & | 1871 | I915_WRITE(GDT_CHICKEN_BITS, (I915_READ(GDT_CHICKEN_BITS) & |
| 1884 | ~GT_NOA_ENABLE)); | 1872 | ~GT_NOA_ENABLE)); |
| @@ -1888,7 +1876,7 @@ static void gen8_disable_metric_set(struct drm_i915_private *dev_priv) · 3 unmodified lines | |||
| 1888 | static void gen10_disable_metric_set(struct drm_i915_private *dev_priv) | 1876 | static void gen10_disable_metric_set(struct drm_i915_private *dev_priv) |
| 1889 | { | 1877 | { |
| 1890 | /* Reset all contexts' slices/subslices configurations. */ | 1878 | /* Reset all contexts' slices/subslices configurations. */ |
| 1891 | gen8_configure_all_contexts(dev_priv, NULL, false); | 1879 | gen8_configure_all_contexts(dev_priv, NULL); |
| 1892 | 1880 | ||
| 1893 | /* Make sure we disable noa to save power. */ | 1881 | /* Make sure we disable noa to save power. */ |
| 1894 | I915_WRITE(RPM_CONFIG1, | 1882 | I915_WRITE(RPM_CONFIG1, |
| @@ -2138,6 +2126,10 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, · 243 unmodified lines | |||
| 2138 | if (ret) | 2126 | if (ret) |
| 2139 | goto err_oa_buf_alloc; | 2127 | goto err_oa_buf_alloc; |
| 2140 | 2128 | ||
| 2129 | ret = i915_mutex_lock_interruptible(&dev_priv->drm); | ||
| 2130 | if (ret) | ||
| 2131 | goto err_lock; | ||
| 2132 | |||
| 2141 | ret = dev_priv->perf.oa.ops.enable_metric_set(dev_priv, | 2133 | ret = dev_priv->perf.oa.ops.enable_metric_set(dev_priv, |
| 2142 | stream->oa_config); | 2134 | stream->oa_config); |
| 2143 | if (ret) | 2135 | if (ret) |
| @@ -2145,23 +2137,17 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, · 1 unmodified line | |||
| 2145 | 2137 | ||
| 2146 | stream->ops = &i915_oa_stream_ops; | 2138 | stream->ops = &i915_oa_stream_ops; |
| 2147 | 2139 | ||
| 2148 | /* Lock device for exclusive_stream access late because | ||
| 2149 | * enable_metric_set() might lock as well on gen8+. | ||
| 2150 | */ | ||
| 2151 | ret = i915_mutex_lock_interruptible(&dev_priv->drm); | ||
| 2152 | if (ret) | ||
| 2153 | goto err_lock; | ||
| 2154 | |||
| 2155 | dev_priv->perf.oa.exclusive_stream = stream; | 2140 | dev_priv->perf.oa.exclusive_stream = stream; |
| 2156 | 2141 | ||
| 2157 | mutex_unlock(&dev_priv->drm.struct_mutex); | 2142 | mutex_unlock(&dev_priv->drm.struct_mutex); |
| 2158 | 2143 | ||
| 2159 | return 0; | 2144 | return 0; |
| 2160 | 2145 | ||
| 2161 | err_lock: | 2146 | err_enable: |
| 2162 | dev_priv->perf.oa.ops.disable_metric_set(dev_priv); | 2147 | dev_priv->perf.oa.ops.disable_metric_set(dev_priv); |
| 2148 | mutex_unlock(&dev_priv->drm.struct_mutex); | ||
| 2163 | 2149 | ||
| 2164 | err_enable: | 2150 | err_lock: |
| 2165 | free_oa_buffer(dev_priv); | 2151 | free_oa_buffer(dev_priv); |
| 2166 | 2152 | ||
| 2167 | err_oa_buf_alloc: | 2153 | err_oa_buf_alloc: |
| @@ -285,26 +285,41 @@ static u64 count_interrupts(struct drm_i915_private *i915) · 284 unmodified lines | |||
| 285 | return sum; | 285 | return sum; |
| 286 | } | 286 | } |
| 287 | 287 | ||
| 288 | static void i915_pmu_event_destroy(struct perf_event *event) | 288 | static void engine_event_destroy(struct perf_event *event) |
| 289 | { | 289 | { |
| 290 | WARN_ON(event->parent); | 290 | struct drm_i915_private *i915 = |
| 291 | container_of(event->pmu, typeof(*i915), pmu.base); | ||
| 292 | struct intel_engine_cs *engine; | ||
| 293 | |||
| 294 | engine = intel_engine_lookup_user(i915, | ||
| 295 | engine_event_class(event), | ||
| 296 | engine_event_instance(event)); | ||
| 297 | if (WARN_ON_ONCE(!engine)) | ||
| 298 | return; | ||
| 299 | |||
| 300 | if (engine_event_sample(event) == I915_SAMPLE_BUSY && | ||
| 301 | intel_engine_supports_stats(engine)) | ||
| 302 | intel_disable_engine_stats(engine); | ||
| 291 | } | 303 | } |
| 292 | 304 | ||
| 293 | static int engine_event_init(struct perf_event *event) | 305 | static void i915_pmu_event_destroy(struct perf_event *event) |
| 294 | { | 306 | { |
| 295 | struct drm_i915_private *i915 = | 307 | WARN_ON(event->parent); |
| 296 | container_of(event->pmu, typeof(*i915), pmu.base); | ||
| 297 | 308 | ||
| 298 | if (!intel_engine_lookup_user(i915, engine_event_class(event), | 309 | if (is_engine_event(event)) |
| 299 | engine_event_instance(event))) | 310 | engine_event_destroy(event); |
| 300 | return -ENODEV; | 311 | } |
| 301 | 312 | ||
| 302 | switch (engine_event_sample(event)) { | 313 | static int |
| 314 | engine_event_status(struct intel_engine_cs *engine, | ||
| 315 | enum drm_i915_pmu_engine_sample sample) | ||
| 316 | { | ||
| 317 | switch (sample) { | ||
| 303 | case I915_SAMPLE_BUSY: | 318 | case I915_SAMPLE_BUSY: |
| 304 | case I915_SAMPLE_WAIT: | 319 | case I915_SAMPLE_WAIT: |
| 305 | break; | 320 | break; |
| 306 | case I915_SAMPLE_SEMA: | 321 | case I915_SAMPLE_SEMA: |
| 307 | if (INTEL_GEN(i915) < 6) | 322 | if (INTEL_GEN(engine->i915) < 6) |
| 308 | return -ENODEV; | 323 | return -ENODEV; |
| 309 | break; | 324 | break; |
| 310 | default: | 325 | default: |
| @@ -314,6 +329,30 @@ static int engine_event_init(struct perf_event *event) · 3 unmodified lines | |||
| 314 | return 0; | 329 | return 0; |
| 315 | } | 330 | } |
| 316 | 331 | ||
| 332 | static int engine_event_init(struct perf_event *event) | ||
| 333 | { | ||
| 334 | struct drm_i915_private *i915 = | ||
| 335 | container_of(event->pmu, typeof(*i915), pmu.base); | ||
| 336 | struct intel_engine_cs *engine; | ||
| 337 | u8 sample; | ||
| 338 | int ret; | ||
| 339 | |||
| 340 | engine = intel_engine_lookup_user(i915, engine_event_class(event), | ||
| 341 | engine_event_instance(event)); | ||
| 342 | if (!engine) | ||
| 343 | return -ENODEV; | ||
| 344 | |||
| 345 | sample = engine_event_sample(event); | ||
| 346 | ret = engine_event_status(engine, sample); | ||
| 347 | if (ret) | ||
| 348 | return ret; | ||
| 349 | |||
| 350 | if (sample == I915_SAMPLE_BUSY && intel_engine_supports_stats(engine)) | ||
| 351 | ret = intel_enable_engine_stats(engine); | ||
| 352 | |||
| 353 | return ret; | ||
| 354 | } | ||
| 355 | |||
| 317 | static int i915_pmu_event_init(struct perf_event *event) | 356 | static int i915_pmu_event_init(struct perf_event *event) |
| 318 | { | 357 | { |
| 319 | struct drm_i915_private *i915 = | 358 | struct drm_i915_private *i915 = |
| @@ -370,7 +409,94 @@ static int i915_pmu_event_init(struct perf_event *event) · 50 unmodified lines | |||
| 370 | return 0; | 409 | return 0; |
| 371 | } | 410 | } |
| 372 | 411 | ||
| 373 | static u64 __i915_pmu_event_read(struct perf_event *event) | 412 | static u64 __get_rc6(struct drm_i915_private *i915) |
| 413 | { | ||
| 414 | u64 val; | ||
| 415 | |||
| 416 | val = intel_rc6_residency_ns(i915, | ||
| 417 | IS_VALLEYVIEW(i915) ? | ||
| 418 | VLV_GT_RENDER_RC6 : | ||
| 419 | GEN6_GT_GFX_RC6); | ||
| 420 | |||
| 421 | if (HAS_RC6p(i915)) | ||
| 422 | val += intel_rc6_residency_ns(i915, GEN6_GT_GFX_RC6p); | ||
| 423 | |||
| 424 | if (HAS_RC6pp(i915)) | ||
| 425 | val += intel_rc6_residency_ns(i915, GEN6_GT_GFX_RC6pp); | ||
| 426 | |||
| 427 | return val; | ||
| 428 | } | ||
| 429 | |||
| 430 | static u64 get_rc6(struct drm_i915_private *i915, bool locked) | ||
| 431 | { | ||
| 432 | #if IS_ENABLED(CONFIG_PM) | ||
| 433 | unsigned long flags; | ||
| 434 | u64 val; | ||
| 435 | |||
| 436 | if (intel_runtime_pm_get_if_in_use(i915)) { | ||
| 437 | val = __get_rc6(i915); | ||
| 438 | intel_runtime_pm_put(i915); | ||
| 439 | |||
| 440 | /* | ||
| 441 | * If we are coming back from being runtime suspended we must | ||
| 442 | * be careful not to report a larger value than returned | ||
| 443 | * previously. | ||
| 444 | */ | ||
| 445 | |||
| 446 | if (!locked) | ||
| 447 | spin_lock_irqsave(&i915->pmu.lock, flags); | ||
| 448 | |||
| 449 | if (val >= i915->pmu.sample[__I915_SAMPLE_RC6_ESTIMATED].cur) { | ||
| 450 | i915->pmu.sample[__I915_SAMPLE_RC6_ESTIMATED].cur = 0; | ||
| 451 | i915->pmu.sample[__I915_SAMPLE_RC6].cur = val; | ||
| 452 | } else { | ||
| 453 | val = i915->pmu.sample[__I915_SAMPLE_RC6_ESTIMATED].cur; | ||
| 454 | } | ||
| 455 | |||
| 456 | if (!locked) | ||
| 457 | spin_unlock_irqrestore(&i915->pmu.lock, flags); | ||
| 458 | } else { | ||
| 459 | struct pci_dev *pdev = i915->drm.pdev; | ||
| 460 | struct device *kdev = &pdev->dev; | ||
| 461 | unsigned long flags2; | ||
| 462 | |||
| 463 | /* | ||
| 464 | * We are runtime suspended. | ||
| 465 | * | ||
| 466 | * Report the delta from when the device was suspended to now, | ||
| 467 | * on top of the last known real value, as the approximated RC6 | ||
| 468 | * counter value. | ||
| 469 | */ | ||
| 470 | if (!locked) | ||
| 471 | spin_lock_irqsave(&i915->pmu.lock, flags); | ||
| 472 | |||
| 473 | spin_lock_irqsave(&kdev->power.lock, flags2); | ||
| 474 | |||
| 475 | if (!i915->pmu.sample[__I915_SAMPLE_RC6_ESTIMATED].cur) | ||
| 476 | i915->pmu.suspended_jiffies_last = | ||
| 477 | kdev->power.suspended_jiffies; | ||
| 478 | |||
| 479 | val = kdev->power.suspended_jiffies - | ||
| 480 | i915->pmu.suspended_jiffies_last; | ||
| 481 | val += jiffies - kdev->power.accounting_timestamp; | ||
| 482 | |||
| 483 | spin_unlock_irqrestore(&kdev->power.lock, flags2); | ||
| 484 | |||
| 485 | val = jiffies_to_nsecs(val); | ||
| 486 | val += i915->pmu.sample[__I915_SAMPLE_RC6].cur; | ||
| 487 | i915->pmu.sample[__I915_SAMPLE_RC6_ESTIMATED].cur = val; | ||
| 488 | |||
| 489 | if (!locked) | ||
| 490 | spin_unlock_irqrestore(&i915->pmu.lock, flags); | ||
| 491 | } | ||
| 492 | |||
| 493 | return val; | ||
| 494 | #else | ||
| 495 | return __get_rc6(i915); | ||
| 496 | #endif | ||
| 497 | } | ||
| 498 | |||
| 499 | static u64 __i915_pmu_event_read(struct perf_event *event, bool locked) | ||
| 374 | { | 500 | { |
| 375 | struct drm_i915_private *i915 = | 501 | struct drm_i915_private *i915 = |
| 376 | container_of(event->pmu, typeof(*i915), pmu.base); | 502 | container_of(event->pmu, typeof(*i915), pmu.base); |
| @@ -387,7 +513,7 @@ static u64 __i915_pmu_event_read(struct perf_event *event) · 10 unmodified lines | |||
| 387 | if (WARN_ON_ONCE(!engine)) { | 513 | if (WARN_ON_ONCE(!engine)) { |
| 388 | /* Do nothing */ | 514 | /* Do nothing */ |
| 389 | } else if (sample == I915_SAMPLE_BUSY && | 515 | } else if (sample == I915_SAMPLE_BUSY && |
| 390 | engine->pmu.busy_stats) { | 516 | intel_engine_supports_stats(engine)) { |
| 391 | val = ktime_to_ns(intel_engine_get_busy_time(engine)); | 517 | val = ktime_to_ns(intel_engine_get_busy_time(engine)); |
| 392 | } else { | 518 | } else { |
| 393 | val = engine->pmu.sample[sample].cur; | 519 | val = engine->pmu.sample[sample].cur; |
| @@ -408,18 +534,7 @@ static u64 __i915_pmu_event_read(struct perf_event *event) · 14 unmodified lines | |||
| 408 | val = count_interrupts(i915); | 534 | val = count_interrupts(i915); |
| 409 | break; | 535 | break; |
| 410 | case I915_PMU_RC6_RESIDENCY: | 536 | case I915_PMU_RC6_RESIDENCY: |
| 411 | intel_runtime_pm_get(i915); | 537 | val = get_rc6(i915, locked); |
| 412 | val = intel_rc6_residency_ns(i915, | ||
| 413 | IS_VALLEYVIEW(i915) ? | ||
| 414 | VLV_GT_RENDER_RC6 : | ||
| 415 | GEN6_GT_GFX_RC6); | ||
| 416 | if (HAS_RC6p(i915)) | ||
| 417 | val += intel_rc6_residency_ns(i915, | ||
| 418 | GEN6_GT_GFX_RC6p); | ||
| 419 | if (HAS_RC6pp(i915)) | ||
| 420 | val += intel_rc6_residency_ns(i915, | ||
| 421 | GEN6_GT_GFX_RC6pp); | ||
| 422 | intel_runtime_pm_put(i915); | ||
| 423 | break; | 538 | break; |
| 424 | } | 539 | } |
| 425 | } | 540 | } |
| @@ -434,7 +549,7 @@ static void i915_pmu_event_read(struct perf_event *event) · 8 unmodified lines | |||
| 434 | 549 | ||
| 435 | again: | 550 | again: |
| 436 | prev = local64_read(&hwc->prev_count); | 551 | prev = local64_read(&hwc->prev_count); |
| 437 | new = __i915_pmu_event_read(event); | 552 | new = __i915_pmu_event_read(event, false); |
| 438 | 553 | ||
| 439 | if (local64_cmpxchg(&hwc->prev_count, prev, new) != prev) | 554 | if (local64_cmpxchg(&hwc->prev_count, prev, new) != prev) |
| 440 | goto again; | 555 | goto again; |
| @@ -442,12 +557,6 @@ again: · 1 unmodified line | |||
| 442 | local64_add(new - prev, &event->count); | 557 | local64_add(new - prev, &event->count); |
| 443 | } | 558 | } |
| 444 | 559 | ||
| 445 | static bool engine_needs_busy_stats(struct intel_engine_cs *engine) | ||
| 446 | { | ||
| 447 | return intel_engine_supports_stats(engine) && | ||
| 448 | (engine->pmu.enable & BIT(I915_SAMPLE_BUSY)); | ||
| 449 | } | ||
| 450 | |||
| 451 | static void i915_pmu_enable(struct perf_event *event) | 560 | static void i915_pmu_enable(struct perf_event *event) |
| 452 | { | 561 | { |
| 453 | struct drm_i915_private *i915 = | 562 | struct drm_i915_private *i915 = |
| @@ -487,21 +596,7 @@ static void i915_pmu_enable(struct perf_event *event) · 33 unmodified lines | |||
| 487 | 596 | ||
| 488 | GEM_BUG_ON(sample >= I915_PMU_SAMPLE_BITS); | 597 | GEM_BUG_ON(sample >= I915_PMU_SAMPLE_BITS); |
| 489 | GEM_BUG_ON(engine->pmu.enable_count[sample] == ~0); | 598 | GEM_BUG_ON(engine->pmu.enable_count[sample] == ~0); |
| 490 | if (engine->pmu.enable_count[sample]++ == 0) { | 599 | engine->pmu.enable_count[sample]++; |
| 491 | /* | ||
| 492 | * Enable engine busy stats tracking if needed or | ||
| 493 | * alternatively cancel the scheduled disable. | ||
| 494 | * | ||
| 495 | * If the delayed disable was pending, cancel it and | ||
| 496 | * in this case do not enable since it already is. | ||
| 497 | */ | ||
| 498 | if (engine_needs_busy_stats(engine) && | ||
| 499 | !engine->pmu.busy_stats) { | ||
| 500 | engine->pmu.busy_stats = true; | ||
| 501 | if (!cancel_delayed_work(&engine->pmu.disable_busy_stats)) | ||
| 502 | intel_enable_engine_stats(engine); | ||
| 503 | } | ||
| 504 | } | ||
| 505 | } | 600 | } |
| 506 | 601 | ||
| 507 | /* | 602 | /* |
| @@ -509,19 +604,11 @@ static void i915_pmu_enable(struct perf_event *event) · 1 unmodified line | |||
| 509 | * for all listeners. Even when the event was already enabled and has | 604 | * for all listeners. Even when the event was already enabled and has |
| 510 | * an existing non-zero value. | 605 | * an existing non-zero value. |
| 511 | */ | 606 | */ |
| 512 | local64_set(&event->hw.prev_count, __i915_pmu_event_read(event)); | 607 | local64_set(&event->hw.prev_count, __i915_pmu_event_read(event, true)); |
| 513 | 608 | ||
| 514 | spin_unlock_irqrestore(&i915->pmu.lock, flags); | 609 | spin_unlock_irqrestore(&i915->pmu.lock, flags); |
| 515 | } | 610 | } |
| 516 | 611 | ||
| 517 | static void __disable_busy_stats(struct work_struct *work) | ||
| 518 | { | ||
| 519 | struct intel_engine_cs *engine = | ||
| 520 | container_of(work, typeof(*engine), pmu.disable_busy_stats.work); | ||
| 521 | |||
| 522 | intel_disable_engine_stats(engine); | ||
| 523 | } | ||
| 524 | |||
| 525 | static void i915_pmu_disable(struct perf_event *event) | 612 | static void i915_pmu_disable(struct perf_event *event) |
| 526 | { | 613 | { |
| 527 | struct drm_i915_private *i915 = | 614 | struct drm_i915_private *i915 = |
| @@ -545,26 +632,8 @@ static void i915_pmu_disable(struct perf_event *event) · 17 unmodified lines | |||
| 545 | * Decrement the reference count and clear the enabled | 632 | * Decrement the reference count and clear the enabled |
| 546 | * bitmask when the last listener on an event goes away. | 633 | * bitmask when the last listener on an event goes away. |
| 547 | */ | 634 | */ |
| 548 | if (--engine->pmu.enable_count[sample] == 0) { | 635 | if (--engine->pmu.enable_count[sample] == 0) |
| 549 | engine->pmu.enable &= ~BIT(sample); | 636 | engine->pmu.enable &= ~BIT(sample); |
| 550 | if (!engine_needs_busy_stats(engine) && | ||
| 551 | engine->pmu.busy_stats) { | ||
| 552 | engine->pmu.busy_stats = false; | ||
| 553 | /* | ||
| 554 | * We request a delayed disable to handle the | ||
| 555 | * rapid on/off cycles on events, which can | ||
| 556 | * happen when tools like perf stat start, in a | ||
| 557 | * nicer way. | ||
| 558 | * | ||
| 559 | * In addition, this also helps with busy stats | ||
| 560 | * accuracy with background CPU offline/online | ||
| 561 | * migration events. | ||
| 562 | */ | ||
| 563 | queue_delayed_work(system_wq, | ||
| 564 | &engine->pmu.disable_busy_stats, | ||
| 565 | round_jiffies_up_relative(HZ)); | ||
| 566 | } | ||
| 567 | } | ||
| 568 | } | 637 | } |
| 569 | 638 | ||
| 570 | GEM_BUG_ON(bit >= I915_PMU_MASK_BITS); | 639 | GEM_BUG_ON(bit >= I915_PMU_MASK_BITS); |
| @@ -797,8 +866,6 @@ static void i915_pmu_unregister_cpuhp_state(struct drm_i915_private *i915) · 226 unmodified lines | |||
| 797 | 866 | ||
| 798 | void i915_pmu_register(struct drm_i915_private *i915) | 867 | void i915_pmu_register(struct drm_i915_private *i915) |
| 799 | { | 868 | { |
| 800 | struct intel_engine_cs *engine; | ||
| 801 | enum intel_engine_id id; | ||
| 802 | int ret; | 869 | int ret; |
| 803 | 870 | ||
| 804 | if (INTEL_GEN(i915) <= 2) { | 871 | if (INTEL_GEN(i915) <= 2) { |
| @@ -820,10 +887,6 @@ void i915_pmu_register(struct drm_i915_private *i915) · 15 unmodified lines | |||
| 820 | hrtimer_init(&i915->pmu.timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); | 887 | hrtimer_init(&i915->pmu.timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); |
| 821 | i915->pmu.timer.function = i915_sample; | 888 | i915->pmu.timer.function = i915_sample; |
| 822 | 889 | ||
| 823 | for_each_engine(engine, i915, id) | ||
| 824 | INIT_DELAYED_WORK(&engine->pmu.disable_busy_stats, | ||
| 825 | __disable_busy_stats); | ||
| 826 | |||
| 827 | ret = perf_pmu_register(&i915->pmu.base, "i915", -1); | 890 | ret = perf_pmu_register(&i915->pmu.base, "i915", -1); |
| 828 | if (ret) | 891 | if (ret) |
| 829 | goto err; | 892 | goto err; |
| @@ -843,9 +906,6 @@ err: · 13 unmodified lines | |||
| 843 | 906 | ||
| 844 | void i915_pmu_unregister(struct drm_i915_private *i915) | 907 | void i915_pmu_unregister(struct drm_i915_private *i915) |
| 845 | { | 908 | { |
| 846 | struct intel_engine_cs *engine; | ||
| 847 | enum intel_engine_id id; | ||
| 848 | |||
| 849 | if (!i915->pmu.base.event_init) | 909 | if (!i915->pmu.base.event_init) |
| 850 | return; | 910 | return; |
| 851 | 911 | ||
| @@ -853,11 +913,6 @@ void i915_pmu_unregister(struct drm_i915_private *i915) · 1 unmodified line | |||
| 853 | 913 | ||
| 854 | hrtimer_cancel(&i915->pmu.timer); | 914 | hrtimer_cancel(&i915->pmu.timer); |
| 855 | 915 | ||
| 856 | for_each_engine(engine, i915, id) { | ||
| 857 | GEM_BUG_ON(engine->pmu.busy_stats); | ||
| 858 | flush_delayed_work(&engine->pmu.disable_busy_stats); | ||
| 859 | } | ||
| 860 | |||
| 861 | i915_pmu_unregister_cpuhp_state(i915); | 916 | i915_pmu_unregister_cpuhp_state(i915); |
| 862 | 917 | ||
| 863 | perf_pmu_unregister(&i915->pmu.base); | 918 | perf_pmu_unregister(&i915->pmu.base); |
| @@ -27,6 +27,8 @@ · 26 unmodified lines | |||
| 27 | enum { | 27 | enum { |
| 28 | __I915_SAMPLE_FREQ_ACT = 0, | 28 | __I915_SAMPLE_FREQ_ACT = 0, |
| 29 | __I915_SAMPLE_FREQ_REQ, | 29 | __I915_SAMPLE_FREQ_REQ, |
| 30 | __I915_SAMPLE_RC6, | ||
| 31 | __I915_SAMPLE_RC6_ESTIMATED, | ||
| 30 | __I915_NUM_PMU_SAMPLERS | 32 | __I915_NUM_PMU_SAMPLERS |
| 31 | }; | 33 | }; |
| 32 | 34 | ||
| @@ -94,6 +96,10 @@ struct i915_pmu { · 61 unmodified lines | |||
| 94 | * struct intel_engine_cs. | 96 | * struct intel_engine_cs. |
| 95 | */ | 97 | */ |
| 96 | struct i915_pmu_sample sample[__I915_NUM_PMU_SAMPLERS]; | 98 | struct i915_pmu_sample sample[__I915_NUM_PMU_SAMPLERS]; |
| 99 | /** | ||
| 100 | * @suspended_jiffies_last: Cached suspend time from PM core. | ||
| 101 | */ | ||
| 102 | unsigned long suspended_jiffies_last; | ||
| 97 | }; | 103 | }; |
| 98 | 104 | ||
| 99 | #ifdef CONFIG_PERF_EVENTS | 105 | #ifdef CONFIG_PERF_EVENTS |
| @@ -2027,7 +2027,7 @@ enum i915_power_well_id { · 2026 unmodified lines | |||
| 2027 | #define _CNL_PORT_TX_DW5_LN0_AE 0x162454 | 2027 | #define _CNL_PORT_TX_DW5_LN0_AE 0x162454 |
| 2028 | #define _CNL_PORT_TX_DW5_LN0_B 0x162654 | 2028 | #define _CNL_PORT_TX_DW5_LN0_B 0x162654 |
| 2029 | #define _CNL_PORT_TX_DW5_LN0_C 0x162C54 | 2029 | #define _CNL_PORT_TX_DW5_LN0_C 0x162C54 |
| 2030 | #define _CNL_PORT_TX_DW5_LN0_D 0x162ED4 | 2030 | #define _CNL_PORT_TX_DW5_LN0_D 0x162E54 |
| 2031 | #define _CNL_PORT_TX_DW5_LN0_F 0x162854 | 2031 | #define _CNL_PORT_TX_DW5_LN0_F 0x162854 |
| 2032 | #define CNL_PORT_TX_DW5_GRP(port) _MMIO_PORT6(port, \ | 2032 | #define CNL_PORT_TX_DW5_GRP(port) _MMIO_PORT6(port, \ |
| 2033 | _CNL_PORT_TX_DW5_GRP_AE, \ | 2033 | _CNL_PORT_TX_DW5_GRP_AE, \ |
| @@ -2058,7 +2058,7 @@ enum i915_power_well_id { · 24 unmodified lines | |||
| 2058 | #define _CNL_PORT_TX_DW7_LN0_AE 0x16245C | 2058 | #define _CNL_PORT_TX_DW7_LN0_AE 0x16245C |
| 2059 | #define _CNL_PORT_TX_DW7_LN0_B 0x16265C | 2059 | #define _CNL_PORT_TX_DW7_LN0_B 0x16265C |
| 2060 | #define _CNL_PORT_TX_DW7_LN0_C 0x162C5C | 2060 | #define _CNL_PORT_TX_DW7_LN0_C 0x162C5C |
| 2061 | #define _CNL_PORT_TX_DW7_LN0_D 0x162EDC | 2061 | #define _CNL_PORT_TX_DW7_LN0_D 0x162E5C |
| 2062 | #define _CNL_PORT_TX_DW7_LN0_F 0x16285C | 2062 | #define _CNL_PORT_TX_DW7_LN0_F 0x16285C |
| 2063 | #define CNL_PORT_TX_DW7_GRP(port) _MMIO_PORT6(port, \ | 2063 | #define CNL_PORT_TX_DW7_GRP(port) _MMIO_PORT6(port, \ |
| 2064 | _CNL_PORT_TX_DW7_GRP_AE, \ | 2064 | _CNL_PORT_TX_DW7_GRP_AE, \ |
| @@ -779,11 +779,11 @@ static struct intel_encoder *get_saved_enc(struct drm_i915_private *dev_priv, · 778 unmodified lines | |||
| 779 | { | 779 | { |
| 780 | struct intel_encoder *encoder; | 780 | struct intel_encoder *encoder; |
| 781 | 781 | ||
| 782 | if (WARN_ON(pipe >= ARRAY_SIZE(dev_priv->av_enc_map))) | ||
| 783 | return NULL; | ||
| 784 | |||
| 785 | /* MST */ | 782 | /* MST */ |
| 786 | if (pipe >= 0) { | 783 | if (pipe >= 0) { |
| 784 | if (WARN_ON(pipe >= ARRAY_SIZE(dev_priv->av_enc_map))) | ||
| 785 | return NULL; | ||
| 786 | |||
| 787 | encoder = dev_priv->av_enc_map[pipe]; | 787 | encoder = dev_priv->av_enc_map[pipe]; |
| 788 | /* | 788 | /* |
| 789 | * when bootup, audio driver may not know it is | 789 | * when bootup, audio driver may not know it is |
| @@ -947,6 +947,86 @@ static int goto_next_sequence_v3(const u8 *data, int index, int total) · 946 unmodified lines | |||
| 947 | return 0; | 947 | return 0; |
| 948 | } | 948 | } |
| 949 | 949 | ||
| 950 | /* | ||
| 951 | * Get len of pre-fixed deassert fragment from a v1 init OTP sequence, | ||
| 952 | * skip all delay + gpio operands and stop at the first DSI packet op. | ||
| 953 | */ | ||
| 954 | static int get_init_otp_deassert_fragment_len(struct drm_i915_private *dev_priv) | ||
| 955 | { | ||
| 956 | const u8 *data = dev_priv->vbt.dsi.sequence[MIPI_SEQ_INIT_OTP]; | ||
| 957 | int index, len; | ||
| 958 | |||
| 959 | if (WARN_ON(!data || dev_priv->vbt.dsi.seq_version != 1)) | ||
| 960 | return 0; | ||
| 961 | |||
| 962 | /* index = 1 to skip sequence byte */ | ||
| 963 | for (index = 1; data[index] != MIPI_SEQ_ELEM_END; index += len) { | ||
| 964 | switch (data[index]) { | ||
| 965 | case MIPI_SEQ_ELEM_SEND_PKT: | ||
| 966 | return index == 1 ? 0 : index; | ||
| 967 | case MIPI_SEQ_ELEM_DELAY: | ||
| 968 | len = 5; /* 1 byte for operand + uint32 */ | ||
| 969 | break; | ||
| 970 | case MIPI_SEQ_ELEM_GPIO: | ||
| 971 | len = 3; /* 1 byte for op, 1 for gpio_nr, 1 for value */ | ||
| 972 | break; | ||
| 973 | default: | ||
| 974 | return 0; | ||
| 975 | } | ||
| 976 | } | ||
| 977 | |||
| 978 | return 0; | ||
| 979 | } | ||
| 980 | |||
| 981 | /* | ||
| 982 | * Some v1 VBT MIPI sequences do the deassert in the init OTP sequence. | ||
| 983 | * The deassert must be done before calling intel_dsi_device_ready, so for | ||
| 984 | * these devices we split the init OTP sequence into a deassert sequence and | ||
| 985 | * the actual init OTP part. | ||
| 986 | */ | ||
| 987 | static void fixup_mipi_sequences(struct drm_i915_private *dev_priv) | ||
| 988 | { | ||
| 989 | u8 *init_otp; | ||
| 990 | int len; | ||
| 991 | |||
| 992 | /* Limit this to VLV for now. */ | ||
| 993 | if (!IS_VALLEYVIEW(dev_priv)) | ||
| 994 | return; | ||
| 995 | |||
| 996 | /* Limit this to v1 vid-mode sequences */ | ||
| 997 | if (dev_priv->vbt.dsi.config->is_cmd_mode || | ||
| 998 | dev_priv->vbt.dsi.seq_version != 1) | ||
| 999 | return; | ||
| 1000 | |||
| 1001 | /* Only do this if there are otp and assert seqs and no deassert seq */ | ||
| 1002 | if (!dev_priv->vbt.dsi.sequence[MIPI_SEQ_INIT_OTP] || | ||
| 1003 | !dev_priv->vbt.dsi.sequence[MIPI_SEQ_ASSERT_RESET] || | ||
| 1004 | dev_priv->vbt.dsi.sequence[MIPI_SEQ_DEASSERT_RESET]) | ||
| 1005 | return; | ||
| 1006 | |||
| 1007 | /* The deassert-sequence ends at the first DSI packet */ | ||
| 1008 | len = get_init_otp_deassert_fragment_len(dev_priv); | ||
| 1009 | if (!len) | ||
| 1010 | return; | ||
| 1011 | |||
| 1012 | DRM_DEBUG_KMS("Using init OTP fragment to deassert reset\n"); | ||
| 1013 | |||
| 1014 | /* Copy the fragment, update seq byte and terminate it */ | ||
| 1015 | init_otp = (u8 *)dev_priv->vbt.dsi.sequence[MIPI_SEQ_INIT_OTP]; | ||
| 1016 | dev_priv->vbt.dsi.deassert_seq = kmemdup(init_otp, len + 1, GFP_KERNEL); | ||
| 1017 | if (!dev_priv->vbt.dsi.deassert_seq) | ||
| 1018 | return; | ||
| 1019 | dev_priv->vbt.dsi.deassert_seq[0] = MIPI_SEQ_DEASSERT_RESET; | ||
| 1020 | dev_priv->vbt.dsi.deassert_seq[len] = MIPI_SEQ_ELEM_END; | ||
| 1021 | /* Use the copy for deassert */ | ||
| 1022 | dev_priv->vbt.dsi.sequence[MIPI_SEQ_DEASSERT_RESET] = | ||
| 1023 | dev_priv->vbt.dsi.deassert_seq; | ||
| 1024 | /* Replace the last byte of the fragment with init OTP seq byte */ | ||
| 1025 | init_otp[len - 1] = MIPI_SEQ_INIT_OTP; | ||
| 1026 | /* And make MIPI_MIPI_SEQ_INIT_OTP point to it */ | ||
| 1027 | dev_priv->vbt.dsi.sequence[MIPI_SEQ_INIT_OTP] = init_otp + len - 1; | ||
| 1028 | } | ||
| 1029 | |||
| 950 | static void | 1030 | static void |
| 951 | parse_mipi_sequence(struct drm_i915_private *dev_priv, | 1031 | parse_mipi_sequence(struct drm_i915_private *dev_priv, |
| 952 | const struct bdb_header *bdb) | 1032 | const struct bdb_header *bdb) |
| @@ -1016,6 +1096,8 @@ parse_mipi_sequence(struct drm_i915_private *dev_priv, · 63 unmodified lines | |||
| 1016 | dev_priv->vbt.dsi.size = seq_size; | 1096 | dev_priv->vbt.dsi.size = seq_size; |
| 1017 | dev_priv->vbt.dsi.seq_version = sequence->version; | 1097 | dev_priv->vbt.dsi.seq_version = sequence->version; |
| 1018 | 1098 | ||
| 1099 | fixup_mipi_sequences(dev_priv); | ||
| 1100 | |||
| 1019 | DRM_DEBUG_DRIVER("MIPI related VBT parsing complete\n"); | 1101 | DRM_DEBUG_DRIVER("MIPI related VBT parsing complete\n"); |
| 1020 | return; | 1102 | return; |
| 1021 | 1103 | ||
| @@ -1588,6 +1670,29 @@ out: · 566 unmodified lines | |||
| 1588 | pci_unmap_rom(pdev, bios); | 1670 | pci_unmap_rom(pdev, bios); |
| 1589 | } | 1671 | } |
| 1590 | 1672 | ||
| 1673 | /** | ||
| 1674 | * intel_bios_cleanup - Free any resources allocated by intel_bios_init() | ||
| 1675 | * @dev_priv: i915 device instance | ||
| 1676 | */ | ||
| 1677 | void intel_bios_cleanup(struct drm_i915_private *dev_priv) | ||
| 1678 | { | ||
| 1679 | kfree(dev_priv->vbt.child_dev); | ||
| 1680 | dev_priv->vbt.child_dev = NULL; | ||
| 1681 | dev_priv->vbt.child_dev_num = 0; | ||
| 1682 | kfree(dev_priv->vbt.sdvo_lvds_vbt_mode); | ||
| 1683 | dev_priv->vbt.sdvo_lvds_vbt_mode = NULL; | ||
| 1684 | kfree(dev_priv->vbt.lfp_lvds_vbt_mode); | ||
| 1685 | dev_priv->vbt.lfp_lvds_vbt_mode = NULL; | ||
| 1686 | kfree(dev_priv->vbt.dsi.data); | ||
| 1687 | dev_priv->vbt.dsi.data = NULL; | ||
| 1688 | kfree(dev_priv->vbt.dsi.pps); | ||
| 1689 | dev_priv->vbt.dsi.pps = NULL; | ||
| 1690 | kfree(dev_priv->vbt.dsi.config); | ||
| 1691 | dev_priv->vbt.dsi.config = NULL; | ||
| 1692 | kfree(dev_priv->vbt.dsi.deassert_seq); | ||
| 1693 | dev_priv->vbt.dsi.deassert_seq = NULL; | ||
| 1694 | } | ||
| 1695 | |||
| 1591 | /** | 1696 | /** |
| 1592 | * intel_bios_is_tv_present - is integrated TV present in VBT | 1697 | * intel_bios_is_tv_present - is integrated TV present in VBT |
| 1593 | * @dev_priv: i915 device instance | 1698 | * @dev_priv: i915 device instance |
| @@ -594,29 +594,16 @@ void intel_engine_remove_wait(struct intel_engine_cs *engine, · 593 unmodified lines | |||
| 594 | spin_unlock_irq(&b->rb_lock); | 594 | spin_unlock_irq(&b->rb_lock); |
| 595 | } | 595 | } |
| 596 | 596 | ||
| 597 | static bool signal_valid(const struct drm_i915_gem_request *request) | ||
| 598 | { | ||
| 599 | return intel_wait_check_request(&request->signaling.wait, request); | ||
| 600 | } | ||
| 601 | |||
| 602 | static bool signal_complete(const struct drm_i915_gem_request *request) | 597 | static bool signal_complete(const struct drm_i915_gem_request *request) |
| 603 | { | 598 | { |
| 604 | if (!request) | 599 | if (!request) |
| 605 | return false; | 600 | return false; |
| 606 | 601 | ||
| 607 | /* If another process served as the bottom-half it may have already | 602 | /* |
| 608 | * signalled that this wait is already completed. | 603 | * Carefully check if the request is complete, giving time for the |
| 609 | */ | ||
| 610 | if (intel_wait_complete(&request->signaling.wait)) | ||
| 611 | return signal_valid(request); | ||
| 612 | |||
| 613 | /* Carefully check if the request is complete, giving time for the | ||
| 614 | * seqno to be visible or if the GPU hung. | 604 | * seqno to be visible or if the GPU hung. |
| 615 | */ | 605 | */ |
| 616 | if (__i915_request_irq_complete(request)) | 606 | return __i915_request_irq_complete(request); |
| 617 | return true; | ||
| 618 | |||
| 619 | return false; | ||
| 620 | } | 607 | } |
| 621 | 608 | ||
| 622 | static struct drm_i915_gem_request *to_signaler(struct rb_node *rb) | 609 | static struct drm_i915_gem_request *to_signaler(struct rb_node *rb) |
| @@ -659,9 +646,13 @@ static int intel_breadcrumbs_signaler(void *arg) · 36 unmodified lines | |||
| 659 | request = i915_gem_request_get_rcu(request); | 646 | request = i915_gem_request_get_rcu(request); |
| 660 | rcu_read_unlock(); | 647 | rcu_read_unlock(); |
| 661 | if (signal_complete(request)) { | 648 | if (signal_complete(request)) { |
| 662 | local_bh_disable(); | 649 | if (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, |
| 663 | dma_fence_signal(&request->fence); | 650 | &request->fence.flags)) { |
| 664 | local_bh_enable(); /* kick start the tasklets */ | 651 | local_bh_disable(); |
| 652 | dma_fence_signal(&request->fence); | ||
| 653 | GEM_BUG_ON(!i915_gem_request_completed(request)); | ||
| 654 | local_bh_enable(); /* kick start the tasklets */ | ||
| 655 | } | ||
| 665 | 656 | ||
| 666 | spin_lock_irq(&b->rb_lock); | 657 | spin_lock_irq(&b->rb_lock); |
| 667 | 658 | ||
| @@ -1952,6 +1952,14 @@ int intel_crtc_compute_min_cdclk(const struct intel_crtc_state *crtc_state) · 1951 unmodified lines | |||
| 1952 | if (crtc_state->has_audio && INTEL_GEN(dev_priv) >= 9) | 1952 | if (crtc_state->has_audio && INTEL_GEN(dev_priv) >= 9) |
| 1953 | min_cdclk = max(2 * 96000, min_cdclk); | 1953 | min_cdclk = max(2 * 96000, min_cdclk); |
| 1954 | 1954 | ||
| 1955 | /* | ||
| 1956 | * On Valleyview some DSI panels lose (v|h)sync when the clock is lower | ||
| 1957 | * than 320000KHz. | ||
| 1958 | */ | ||
| 1959 | if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DSI) && | ||
| 1960 | IS_VALLEYVIEW(dev_priv)) | ||
| 1961 | min_cdclk = max(320000, min_cdclk); | ||
| 1962 | |||
| 1955 | if (min_cdclk > dev_priv->max_cdclk_freq) { | 1963 | if (min_cdclk > dev_priv->max_cdclk_freq) { |
| 1956 | DRM_DEBUG_KMS("required cdclk (%d kHz) exceeds max (%d kHz)\n", | 1964 | DRM_DEBUG_KMS("required cdclk (%d kHz) exceeds max (%d kHz)\n", |
| 1957 | min_cdclk, dev_priv->max_cdclk_freq); | 1965 | min_cdclk, dev_priv->max_cdclk_freq); |
| @@ -1458,7 +1458,9 @@ static bool ring_is_idle(struct intel_engine_cs *engine) · 1457 unmodified lines | |||
| 1458 | struct drm_i915_private *dev_priv = engine->i915; | 1458 | struct drm_i915_private *dev_priv = engine->i915; |
| 1459 | bool idle = true; | 1459 | bool idle = true; |
| 1460 | 1460 | ||
| 1461 | intel_runtime_pm_get(dev_priv); | 1461 | /* If the whole device is asleep, the engine must be idle */ |
| 1462 | if (!intel_runtime_pm_get_if_in_use(dev_priv)) | ||
| 1463 | return true; | ||
| 1462 | 1464 | ||
| 1463 | /* First check that no commands are left in the ring */ | 1465 | /* First check that no commands are left in the ring */ |
| 1464 | if ((I915_READ_HEAD(engine) & HEAD_ADDR) != | 1466 | if ((I915_READ_HEAD(engine) & HEAD_ADDR) != |
| @@ -1943,16 +1945,22 @@ intel_engine_lookup_user(struct drm_i915_private *i915, u8 class, u8 instance) · 478 unmodified lines | |||
| 1943 | */ | 1945 | */ |
| 1944 | int intel_enable_engine_stats(struct intel_engine_cs *engine) | 1946 | int intel_enable_engine_stats(struct intel_engine_cs *engine) |
| 1945 | { | 1947 | { |
| 1948 | struct intel_engine_execlists *execlists = &engine->execlists; | ||
| 1946 | unsigned long flags; | 1949 | unsigned long flags; |
| 1950 | int err = 0; | ||
| 1947 | 1951 | ||
| 1948 | if (!intel_engine_supports_stats(engine)) | 1952 | if (!intel_engine_supports_stats(engine)) |
| 1949 | return -ENODEV; | 1953 | return -ENODEV; |
| 1950 | 1954 | ||
| 1955 | tasklet_disable(&execlists->tasklet); | ||
| 1951 | spin_lock_irqsave(&engine->stats.lock, flags); | 1956 | spin_lock_irqsave(&engine->stats.lock, flags); |
| 1952 | if (engine->stats.enabled == ~0) | 1957 | |
| 1953 | goto busy; | 1958 | if (unlikely(engine->stats.enabled == ~0)) { |
| 1959 | err = -EBUSY; | ||
| 1960 | goto unlock; | ||
| 1961 | } | ||
| 1962 | |||
| 1954 | if (engine->stats.enabled++ == 0) { | 1963 | if (engine->stats.enabled++ == 0) { |
| 1955 | struct intel_engine_execlists *execlists = &engine->execlists; | ||
| 1956 | const struct execlist_port *port = execlists->port; | 1964 | const struct execlist_port *port = execlists->port; |
| 1957 | unsigned int num_ports = execlists_num_ports(execlists); | 1965 | unsigned int num_ports = execlists_num_ports(execlists); |
| 1958 | 1966 | ||
| @@ -1967,14 +1975,12 @@ int intel_enable_engine_stats(struct intel_engine_cs *engine) · 8 unmodified lines | |||
| 1967 | if (engine->stats.active) | 1975 | if (engine->stats.active) |
| 1968 | engine->stats.start = engine->stats.enabled_at; | 1976 | engine->stats.start = engine->stats.enabled_at; |
| 1969 | } | 1977 | } |
| 1970 | spin_unlock_irqrestore(&engine->stats.lock, flags); | ||
| 1971 | |||
| 1972 | return 0; | ||
| 1973 | 1978 | ||
| 1974 | busy: | 1979 | unlock: |
| 1975 | spin_unlock_irqrestore(&engine->stats.lock, flags); | 1980 | spin_unlock_irqrestore(&engine->stats.lock, flags); |
| 1981 | tasklet_enable(&execlists->tasklet); | ||
| 1976 | 1982 | ||
| 1977 | return -EBUSY; | 1983 | return err; |
| 1978 | } | 1984 | } |
| 1979 | 1985 | ||
| 1980 | static ktime_t __intel_engine_get_busy_time(struct intel_engine_cs *engine) | 1986 | static ktime_t __intel_engine_get_busy_time(struct intel_engine_cs *engine) |
| @@ -719,6 +719,8 @@ static void execlists_cancel_requests(struct intel_engine_cs *engine) · 718 unmodified lines | |||
| 719 | struct rb_node *rb; | 719 | struct rb_node *rb; |
| 720 | unsigned long flags; | 720 | unsigned long flags; |
| 721 | 721 | ||
| 722 | GEM_TRACE("%s\n", engine->name); | ||
| 723 | |||
| 722 | spin_lock_irqsave(&engine->timeline->lock, flags); | 724 | spin_lock_irqsave(&engine->timeline->lock, flags); |
| 723 | 725 | ||
| 724 | /* Cancel the requests on the HW and clear the ELSP tracker. */ | 726 | /* Cancel the requests on the HW and clear the ELSP tracker. */ |
| @@ -765,6 +767,9 @@ static void execlists_cancel_requests(struct intel_engine_cs *engine) · 40 unmodified lines | |||
| 765 | */ | 767 | */ |
| 766 | clear_bit(ENGINE_IRQ_EXECLIST, &engine->irq_posted); | 768 | clear_bit(ENGINE_IRQ_EXECLIST, &engine->irq_posted); |
| 767 | 769 | ||
| 770 | /* Mark all CS interrupts as complete */ | ||
| 771 | execlists->active = 0; | ||
| 772 | |||
| 768 | spin_unlock_irqrestore(&engine->timeline->lock, flags); | 773 | spin_unlock_irqrestore(&engine->timeline->lock, flags); |
| 769 | } | 774 | } |
| 770 | 775 | ||
| @@ -366,20 +366,6 @@ struct intel_engine_cs { · 365 unmodified lines | |||
| 366 | */ | 366 | */ |
| 367 | #define I915_ENGINE_SAMPLE_MAX (I915_SAMPLE_SEMA + 1) | 367 | #define I915_ENGINE_SAMPLE_MAX (I915_SAMPLE_SEMA + 1) |
| 368 | struct i915_pmu_sample sample[I915_ENGINE_SAMPLE_MAX]; | 368 | struct i915_pmu_sample sample[I915_ENGINE_SAMPLE_MAX]; |
| 369 | /** | ||
| 370 | * @busy_stats: Has enablement of engine stats tracking been | ||
| 371 | * requested. | ||
| 372 | */ | ||
| 373 | bool busy_stats; | ||
| 374 | /** | ||
| 375 | * @disable_busy_stats: Work item for busy stats disabling. | ||
| 376 | * | ||
| 377 | * Same as with @enable_busy_stats action, with the difference | ||
| 378 | * that we delay it in case there are rapid enable-disable | ||
| 379 | * actions, which can happen during tool startup (like perf | ||
| 380 | * stat). | ||
| 381 | */ | ||
| 382 | struct delayed_work disable_busy_stats; | ||
| 383 | } pmu; | 369 | } pmu; |
| 384 | 370 | ||
| 385 | /* | 371 | /* |
| @@ -36,6 +36,7 @@ · 35 unmodified lines | |||
| 36 | #include "meson_venc.h" | 36 | #include "meson_venc.h" |
| 37 | #include "meson_vpp.h" | 37 | #include "meson_vpp.h" |
| 38 | #include "meson_viu.h" | 38 | #include "meson_viu.h" |
| 39 | #include "meson_canvas.h" | ||
| 39 | #include "meson_registers.h" | 40 | #include "meson_registers.h" |
| 40 | 41 | ||
| 41 | /* CRTC definition */ | 42 | /* CRTC definition */ |
| @@ -192,6 +193,11 @@ void meson_crtc_irq(struct meson_drm *priv) · 150 unmodified lines | |||
| 192 | } else | 193 | } else |
| 193 | meson_vpp_disable_interlace_vscaler_osd1(priv); | 194 | meson_vpp_disable_interlace_vscaler_osd1(priv); |
| 194 | 195 | ||
| 196 | meson_canvas_setup(priv, MESON_CANVAS_ID_OSD1, | ||
| 197 | priv->viu.osd1_addr, priv->viu.osd1_stride, | ||
| 198 | priv->viu.osd1_height, MESON_CANVAS_WRAP_NONE, | ||
| 199 | MESON_CANVAS_BLKMODE_LINEAR); | ||
| 200 | |||
| 195 | /* Enable OSD1 */ | 201 | /* Enable OSD1 */ |
| 196 | writel_bits_relaxed(VPP_OSD1_POSTBLEND, VPP_OSD1_POSTBLEND, | 202 | writel_bits_relaxed(VPP_OSD1_POSTBLEND, VPP_OSD1_POSTBLEND, |
| 197 | priv->io_base + _REG(VPP_MISC)); | 203 | priv->io_base + _REG(VPP_MISC)); |
| @@ -43,6 +43,9 @@ struct meson_drm { · 42 unmodified lines | |||
| 43 | bool osd1_commit; | 43 | bool osd1_commit; |
| 44 | uint32_t osd1_ctrl_stat; | 44 | uint32_t osd1_ctrl_stat; |
| 45 | uint32_t osd1_blk0_cfg[5]; | 45 | uint32_t osd1_blk0_cfg[5]; |
| 46 | uint32_t osd1_addr; | ||
| 47 | uint32_t osd1_stride; | ||
| 48 | uint32_t osd1_height; | ||
| 46 | } viu; | 49 | } viu; |
| 47 | 50 | ||
| 48 | struct { | 51 | struct { |
| @@ -164,10 +164,9 @@ static void meson_plane_atomic_update(struct drm_plane *plane, · 163 unmodified lines | |||
| 164 | /* Update Canvas with buffer address */ | 164 | /* Update Canvas with buffer address */ |
| 165 | gem = drm_fb_cma_get_gem_obj(fb, 0); | 165 | gem = drm_fb_cma_get_gem_obj(fb, 0); |
| 166 | 166 | ||
| 167 | meson_canvas_setup(priv, MESON_CANVAS_ID_OSD1, | 167 | priv->viu.osd1_addr = gem->paddr; |
| 168 | gem->paddr, fb->pitches[0], | 168 | priv->viu.osd1_stride = fb->pitches[0]; |
| 169 | fb->height, MESON_CANVAS_WRAP_NONE, | 169 | priv->viu.osd1_height = fb->height; |
| 170 | MESON_CANVAS_BLKMODE_LINEAR); | ||
| 171 | 170 | ||
| 172 | spin_unlock_irqrestore(&priv->drm->event_lock, flags); | 171 | spin_unlock_irqrestore(&priv->drm->event_lock, flags); |
| 173 | } | 172 | } |
| @@ -680,7 +680,7 @@ struct msm_kms *mdp5_kms_init(struct drm_device *dev) · 679 unmodified lines | |||
| 680 | } else { | 680 | } else { |
| 681 | dev_info(&pdev->dev, | 681 | dev_info(&pdev->dev, |
| 682 | "no iommu, fallback to phys contig buffers for scanout\n"); | 682 | "no iommu, fallback to phys contig buffers for scanout\n"); |
| 683 | aspace = NULL;; | 683 | aspace = NULL; |
| 684 | } | 684 | } |
| 685 | 685 | ||
| 686 | pm_runtime_put_sync(&pdev->dev); | 686 | pm_runtime_put_sync(&pdev->dev); |
| @@ -570,9 +570,15 @@ nouveau_connector_detect(struct drm_connector *connector, bool force) · 569 unmodified lines | |||
| 570 | nv_connector->edid = NULL; | 570 | nv_connector->edid = NULL; |
| 571 | } | 571 | } |
| 572 | 572 | ||
| 573 | ret = pm_runtime_get_sync(connector->dev->dev); | 573 | /* Outputs are only polled while runtime active, so acquiring a |
| 574 | if (ret < 0 && ret != -EACCES) | 574 | * runtime PM ref here is unnecessary (and would deadlock upon |
| 575 | return conn_status; | 575 | * runtime suspend because it waits for polling to finish). |
| 576 | */ | ||
| 577 | if (!drm_kms_helper_is_poll_worker()) { | ||
| 578 | ret = pm_runtime_get_sync(connector->dev->dev); | ||
| 579 | if (ret < 0 && ret != -EACCES) | ||
| 580 | return conn_status; | ||
| 581 | } | ||
| 576 | 582 | ||
| 577 | nv_encoder = nouveau_connector_ddc_detect(connector); | 583 | nv_encoder = nouveau_connector_ddc_detect(connector); |
| 578 | if (nv_encoder && (i2c = nv_encoder->i2c) != NULL) { | 584 | if (nv_encoder && (i2c = nv_encoder->i2c) != NULL) { |
| @@ -647,8 +653,10 @@ detect_analog: · 68 unmodified lines | |||
| 647 | 653 | ||
| 648 | out: | 654 | out: |
| 649 | 655 | ||
| 650 | pm_runtime_mark_last_busy(connector->dev->dev); | 656 | if (!drm_kms_helper_is_poll_worker()) { |
| 651 | pm_runtime_put_autosuspend(connector->dev->dev); | 657 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 658 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 659 | } | ||
| 652 | 660 | ||
| 653 | return conn_status; | 661 | return conn_status; |
| 654 | } | 662 | } |
| @@ -4477,6 +4477,7 @@ nv50_display_create(struct drm_device *dev) · 4476 unmodified lines | |||
| 4477 | nouveau_display(dev)->fini = nv50_display_fini; | 4477 | nouveau_display(dev)->fini = nv50_display_fini; |
| 4478 | disp->disp = &nouveau_display(dev)->disp; | 4478 | disp->disp = &nouveau_display(dev)->disp; |
| 4479 | dev->mode_config.funcs = &nv50_disp_func; | 4479 | dev->mode_config.funcs = &nv50_disp_func; |
| 4480 | dev->driver->driver_features |= DRIVER_PREFER_XBGR_30BPP; | ||
| 4480 | if (nouveau_atomic) | 4481 | if (nouveau_atomic) |
| 4481 | dev->driver->driver_features |= DRIVER_ATOMIC; | 4482 | dev->driver->driver_features |= DRIVER_ATOMIC; |
| 4482 | 4483 | ||
| @@ -301,7 +301,7 @@ nvkm_therm_attr_set(struct nvkm_therm *therm, · 300 unmodified lines | |||
| 301 | void | 301 | void |
| 302 | nvkm_therm_clkgate_enable(struct nvkm_therm *therm) | 302 | nvkm_therm_clkgate_enable(struct nvkm_therm *therm) |
| 303 | { | 303 | { |
| 304 | if (!therm->func->clkgate_enable || !therm->clkgating_enabled) | 304 | if (!therm || !therm->func->clkgate_enable || !therm->clkgating_enabled) |
| 305 | return; | 305 | return; |
| 306 | 306 | ||
| 307 | nvkm_debug(&therm->subdev, | 307 | nvkm_debug(&therm->subdev, |
| @@ -312,7 +312,7 @@ nvkm_therm_clkgate_enable(struct nvkm_therm *therm) · 4 unmodified lines | |||
| 312 | void | 312 | void |
| 313 | nvkm_therm_clkgate_fini(struct nvkm_therm *therm, bool suspend) | 313 | nvkm_therm_clkgate_fini(struct nvkm_therm *therm, bool suspend) |
| 314 | { | 314 | { |
| 315 | if (!therm->func->clkgate_fini || !therm->clkgating_enabled) | 315 | if (!therm || !therm->func->clkgate_fini || !therm->clkgating_enabled) |
| 316 | return; | 316 | return; |
| 317 | 317 | ||
| 318 | nvkm_debug(&therm->subdev, | 318 | nvkm_debug(&therm->subdev, |
| @@ -395,7 +395,7 @@ void · 76 unmodified lines | |||
| 395 | nvkm_therm_clkgate_init(struct nvkm_therm *therm, | 395 | nvkm_therm_clkgate_init(struct nvkm_therm *therm, |
| 396 | const struct nvkm_therm_clkgate_pack *p) | 396 | const struct nvkm_therm_clkgate_pack *p) |
| 397 | { | 397 | { |
| 398 | if (!therm->func->clkgate_init || !therm->clkgating_enabled) | 398 | if (!therm || !therm->func->clkgate_init || !therm->clkgating_enabled) |
| 399 | return; | 399 | return; |
| 400 | 400 | ||
| 401 | therm->func->clkgate_init(therm, p); | 401 | therm->func->clkgate_init(therm, p); |
| @@ -3221,35 +3221,8 @@ static void cik_gpu_init(struct radeon_device *rdev) · 3220 unmodified lines | |||
| 3221 | case CHIP_KAVERI: | 3221 | case CHIP_KAVERI: |
| 3222 | rdev->config.cik.max_shader_engines = 1; | 3222 | rdev->config.cik.max_shader_engines = 1; |
| 3223 | rdev->config.cik.max_tile_pipes = 4; | 3223 | rdev->config.cik.max_tile_pipes = 4; |
| 3224 | if ((rdev->pdev->device == 0x1304) || | 3224 | rdev->config.cik.max_cu_per_sh = 8; |
| 3225 | (rdev->pdev->device == 0x1305) || | 3225 | rdev->config.cik.max_backends_per_se = 2; |
| 3226 | (rdev->pdev->device == 0x130C) || | ||
| 3227 | (rdev->pdev->device == 0x130F) || | ||
| 3228 | (rdev->pdev->device == 0x1310) || | ||
| 3229 | (rdev->pdev->device == 0x1311) || | ||
| 3230 | (rdev->pdev->device == 0x131C)) { | ||
| 3231 | rdev->config.cik.max_cu_per_sh = 8; | ||
| 3232 | rdev->config.cik.max_backends_per_se = 2; | ||
| 3233 | } else if ((rdev->pdev->device == 0x1309) || | ||
| 3234 | (rdev->pdev->device == 0x130A) || | ||
| 3235 | (rdev->pdev->device == 0x130D) || | ||
| 3236 | (rdev->pdev->device == 0x1313) || | ||
| 3237 | (rdev->pdev->device == 0x131D)) { | ||
| 3238 | rdev->config.cik.max_cu_per_sh = 6; | ||
| 3239 | rdev->config.cik.max_backends_per_se = 2; | ||
| 3240 | } else if ((rdev->pdev->device == 0x1306) || | ||
| 3241 | (rdev->pdev->device == 0x1307) || | ||
| 3242 | (rdev->pdev->device == 0x130B) || | ||
| 3243 | (rdev->pdev->device == 0x130E) || | ||
| 3244 | (rdev->pdev->device == 0x1315) || | ||
| 3245 | (rdev->pdev->device == 0x1318) || | ||
| 3246 | (rdev->pdev->device == 0x131B)) { | ||
| 3247 | rdev->config.cik.max_cu_per_sh = 4; | ||
| 3248 | rdev->config.cik.max_backends_per_se = 1; | ||
| 3249 | } else { | ||
| 3250 | rdev->config.cik.max_cu_per_sh = 3; | ||
| 3251 | rdev->config.cik.max_backends_per_se = 1; | ||
| 3252 | } | ||
| 3253 | rdev->config.cik.max_sh_per_se = 1; | 3226 | rdev->config.cik.max_sh_per_se = 1; |
| 3254 | rdev->config.cik.max_texture_channel_caches = 4; | 3227 | rdev->config.cik.max_texture_channel_caches = 4; |
| 3255 | rdev->config.cik.max_gprs = 256; | 3228 | rdev->config.cik.max_gprs = 256; |
| @@ -899,9 +899,11 @@ radeon_lvds_detect(struct drm_connector *connector, bool force) · 898 unmodified lines | |||
| 899 | enum drm_connector_status ret = connector_status_disconnected; | 899 | enum drm_connector_status ret = connector_status_disconnected; |
| 900 | int r; | 900 | int r; |
| 901 | 901 | ||
| 902 | r = pm_runtime_get_sync(connector->dev->dev); | 902 | if (!drm_kms_helper_is_poll_worker()) { |
| 903 | if (r < 0) | 903 | r = pm_runtime_get_sync(connector->dev->dev); |
| 904 | return connector_status_disconnected; | 904 | if (r < 0) |
| 905 | return connector_status_disconnected; | ||
| 906 | } | ||
| 905 | 907 | ||
| 906 | if (encoder) { | 908 | if (encoder) { |
| 907 | struct radeon_encoder *radeon_encoder = to_radeon_encoder(encoder); | 909 | struct radeon_encoder *radeon_encoder = to_radeon_encoder(encoder); |
| @@ -924,8 +926,12 @@ radeon_lvds_detect(struct drm_connector *connector, bool force) · 16 unmodified lines | |||
| 924 | /* check acpi lid status ??? */ | 926 | /* check acpi lid status ??? */ |
| 925 | 927 | ||
| 926 | radeon_connector_update_scratch_regs(connector, ret); | 928 | radeon_connector_update_scratch_regs(connector, ret); |
| 927 | pm_runtime_mark_last_busy(connector->dev->dev); | 929 | |
| 928 | pm_runtime_put_autosuspend(connector->dev->dev); | 930 | if (!drm_kms_helper_is_poll_worker()) { |
| 931 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 932 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 933 | } | ||
| 934 | |||
| 929 | return ret; | 935 | return ret; |
| 930 | } | 936 | } |
| 931 | 937 | ||
| @@ -1039,9 +1045,11 @@ radeon_vga_detect(struct drm_connector *connector, bool force) · 107 unmodified lines | |||
| 1039 | enum drm_connector_status ret = connector_status_disconnected; | 1045 | enum drm_connector_status ret = connector_status_disconnected; |
| 1040 | int r; | 1046 | int r; |
| 1041 | 1047 | ||
| 1042 | r = pm_runtime_get_sync(connector->dev->dev); | 1048 | if (!drm_kms_helper_is_poll_worker()) { |
| 1043 | if (r < 0) | 1049 | r = pm_runtime_get_sync(connector->dev->dev); |
| 1044 | return connector_status_disconnected; | 1050 | if (r < 0) |
| 1051 | return connector_status_disconnected; | ||
| 1052 | } | ||
| 1045 | 1053 | ||
| 1046 | encoder = radeon_best_single_encoder(connector); | 1054 | encoder = radeon_best_single_encoder(connector); |
| 1047 | if (!encoder) | 1055 | if (!encoder) |
| @@ -1108,8 +1116,10 @@ radeon_vga_detect(struct drm_connector *connector, bool force) · 60 unmodified lines | |||
| 1108 | radeon_connector_update_scratch_regs(connector, ret); | 1116 | radeon_connector_update_scratch_regs(connector, ret); |
| 1109 | 1117 | ||
| 1110 | out: | 1118 | out: |
| 1111 | pm_runtime_mark_last_busy(connector->dev->dev); | 1119 | if (!drm_kms_helper_is_poll_worker()) { |
| 1112 | pm_runtime_put_autosuspend(connector->dev->dev); | 1120 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1121 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1122 | } | ||
| 1113 | 1123 | ||
| 1114 | return ret; | 1124 | return ret; |
| 1115 | } | 1125 | } |
| @@ -1173,9 +1183,11 @@ radeon_tv_detect(struct drm_connector *connector, bool force) · 57 unmodified lines | |||
| 1173 | if (!radeon_connector->dac_load_detect) | 1183 | if (!radeon_connector->dac_load_detect) |
| 1174 | return ret; | 1184 | return ret; |
| 1175 | 1185 | ||
| 1176 | r = pm_runtime_get_sync(connector->dev->dev); | 1186 | if (!drm_kms_helper_is_poll_worker()) { |
| 1177 | if (r < 0) | 1187 | r = pm_runtime_get_sync(connector->dev->dev); |
| 1178 | return connector_status_disconnected; | 1188 | if (r < 0) |
| 1189 | return connector_status_disconnected; | ||
| 1190 | } | ||
| 1179 | 1191 | ||
| 1180 | encoder = radeon_best_single_encoder(connector); | 1192 | encoder = radeon_best_single_encoder(connector); |
| 1181 | if (!encoder) | 1193 | if (!encoder) |
| @@ -1187,8 +1199,12 @@ radeon_tv_detect(struct drm_connector *connector, bool force) · 5 unmodified lines | |||
| 1187 | if (ret == connector_status_connected) | 1199 | if (ret == connector_status_connected) |
| 1188 | ret = radeon_connector_analog_encoder_conflict_solve(connector, encoder, ret, false); | 1200 | ret = radeon_connector_analog_encoder_conflict_solve(connector, encoder, ret, false); |
| 1189 | radeon_connector_update_scratch_regs(connector, ret); | 1201 | radeon_connector_update_scratch_regs(connector, ret); |
| 1190 | pm_runtime_mark_last_busy(connector->dev->dev); | 1202 | |
| 1191 | pm_runtime_put_autosuspend(connector->dev->dev); | 1203 | if (!drm_kms_helper_is_poll_worker()) { |
| 1204 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 1205 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1206 | } | ||
| 1207 | |||
| 1192 | return ret; | 1208 | return ret; |
| 1193 | } | 1209 | } |
| 1194 | 1210 | ||
| @@ -1251,9 +1267,11 @@ radeon_dvi_detect(struct drm_connector *connector, bool force) · 56 unmodified lines | |||
| 1251 | enum drm_connector_status ret = connector_status_disconnected; | 1267 | enum drm_connector_status ret = connector_status_disconnected; |
| 1252 | bool dret = false, broken_edid = false; | 1268 | bool dret = false, broken_edid = false; |
| 1253 | 1269 | ||
| 1254 | r = pm_runtime_get_sync(connector->dev->dev); | 1270 | if (!drm_kms_helper_is_poll_worker()) { |
| 1255 | if (r < 0) | 1271 | r = pm_runtime_get_sync(connector->dev->dev); |
| 1256 | return connector_status_disconnected; | 1272 | if (r < 0) |
| 1273 | return connector_status_disconnected; | ||
| 1274 | } | ||
| 1257 | 1275 | ||
| 1258 | if (radeon_connector->detected_hpd_without_ddc) { | 1276 | if (radeon_connector->detected_hpd_without_ddc) { |
| 1259 | force = true; | 1277 | force = true; |
| @@ -1436,8 +1454,10 @@ out: · 176 unmodified lines | |||
| 1436 | } | 1454 | } |
| 1437 | 1455 | ||
| 1438 | exit: | 1456 | exit: |
| 1439 | pm_runtime_mark_last_busy(connector->dev->dev); | 1457 | if (!drm_kms_helper_is_poll_worker()) { |
| 1440 | pm_runtime_put_autosuspend(connector->dev->dev); | 1458 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1459 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1460 | } | ||
| 1441 | 1461 | ||
| 1442 | return ret; | 1462 | return ret; |
| 1443 | } | 1463 | } |
| @@ -1688,9 +1708,11 @@ radeon_dp_detect(struct drm_connector *connector, bool force) · 244 unmodified lines | |||
| 1688 | if (radeon_dig_connector->is_mst) | 1708 | if (radeon_dig_connector->is_mst) |
| 1689 | return connector_status_disconnected; | 1709 | return connector_status_disconnected; |
| 1690 | 1710 | ||
| 1691 | r = pm_runtime_get_sync(connector->dev->dev); | 1711 | if (!drm_kms_helper_is_poll_worker()) { |
| 1692 | if (r < 0) | 1712 | r = pm_runtime_get_sync(connector->dev->dev); |
| 1693 | return connector_status_disconnected; | 1713 | if (r < 0) |
| 1714 | return connector_status_disconnected; | ||
| 1715 | } | ||
| 1694 | 1716 | ||
| 1695 | if (!force && radeon_check_hpd_status_unchanged(connector)) { | 1717 | if (!force && radeon_check_hpd_status_unchanged(connector)) { |
| 1696 | ret = connector->status; | 1718 | ret = connector->status; |
| @@ -1777,8 +1799,10 @@ radeon_dp_detect(struct drm_connector *connector, bool force) · 80 unmodified lines | |||
| 1777 | } | 1799 | } |
| 1778 | 1800 | ||
| 1779 | out: | 1801 | out: |
| 1780 | pm_runtime_mark_last_busy(connector->dev->dev); | 1802 | if (!drm_kms_helper_is_poll_worker()) { |
| 1781 | pm_runtime_put_autosuspend(connector->dev->dev); | 1803 | pm_runtime_mark_last_busy(connector->dev->dev); |
| 1804 | pm_runtime_put_autosuspend(connector->dev->dev); | ||
| 1805 | } | ||
| 1782 | 1806 | ||
| 1783 | return ret; | 1807 | return ret; |
| 1784 | } | 1808 | } |
| @@ -1365,6 +1365,10 @@ int radeon_device_init(struct radeon_device *rdev, · 1364 unmodified lines | |||
| 1365 | if ((rdev->flags & RADEON_IS_PCI) && | 1365 | if ((rdev->flags & RADEON_IS_PCI) && |
| 1366 | (rdev->family <= CHIP_RS740)) | 1366 | (rdev->family <= CHIP_RS740)) |
| 1367 | rdev->need_dma32 = true; | 1367 | rdev->need_dma32 = true; |
| 1368 | #ifdef CONFIG_PPC64 | ||
| 1369 | if (rdev->family == CHIP_CEDAR) | ||
| 1370 | rdev->need_dma32 = true; | ||
| 1371 | #endif | ||
| 1368 | 1372 | ||
| 1369 | dma_bits = rdev->need_dma32 ? 32 : 40; | 1373 | dma_bits = rdev->need_dma32 ? 32 : 40; |
| 1370 | r = pci_set_dma_mask(rdev->pdev, DMA_BIT_MASK(dma_bits)); | 1374 | r = pci_set_dma_mask(rdev->pdev, DMA_BIT_MASK(dma_bits)); |
| @@ -47,7 +47,6 @@ static bool radeon_pm_in_vbl(struct radeon_device *rdev); · 46 unmodified lines | |||
| 47 | static bool radeon_pm_debug_check_in_vbl(struct radeon_device *rdev, bool finish); | 47 | static bool radeon_pm_debug_check_in_vbl(struct radeon_device *rdev, bool finish); |
| 48 | static void radeon_pm_update_profile(struct radeon_device *rdev); | 48 | static void radeon_pm_update_profile(struct radeon_device *rdev); |
| 49 | static void radeon_pm_set_clocks(struct radeon_device *rdev); | 49 | static void radeon_pm_set_clocks(struct radeon_device *rdev); |
| 50 | static void radeon_pm_compute_clocks_dpm(struct radeon_device *rdev); | ||
| 51 | 50 | ||
| 52 | int radeon_pm_get_type_index(struct radeon_device *rdev, | 51 | int radeon_pm_get_type_index(struct radeon_device *rdev, |
| 53 | enum radeon_pm_state_type ps_type, | 52 | enum radeon_pm_state_type ps_type, |
| @@ -80,8 +79,6 @@ void radeon_pm_acpi_event_handler(struct radeon_device *rdev) · 26 unmodified lines | |||
| 80 | radeon_dpm_enable_bapm(rdev, rdev->pm.dpm.ac_power); | 79 | radeon_dpm_enable_bapm(rdev, rdev->pm.dpm.ac_power); |
| 81 | } | 80 | } |
| 82 | mutex_unlock(&rdev->pm.mutex); | 81 | mutex_unlock(&rdev->pm.mutex); |
| 83 | /* allow new DPM state to be picked */ | ||
| 84 | radeon_pm_compute_clocks_dpm(rdev); | ||
| 85 | } else if (rdev->pm.pm_method == PM_METHOD_PROFILE) { | 82 | } else if (rdev->pm.pm_method == PM_METHOD_PROFILE) { |
| 86 | if (rdev->pm.profile == PM_PROFILE_AUTO) { | 83 | if (rdev->pm.profile == PM_PROFILE_AUTO) { |
| 87 | mutex_lock(&rdev->pm.mutex); | 84 | mutex_lock(&rdev->pm.mutex); |
| @@ -885,8 +882,7 @@ static struct radeon_ps *radeon_dpm_pick_power_state(struct radeon_device *rdev, · 797 unmodified lines | |||
| 885 | dpm_state = POWER_STATE_TYPE_INTERNAL_3DPERF; | 882 | dpm_state = POWER_STATE_TYPE_INTERNAL_3DPERF; |
| 886 | /* balanced states don't exist at the moment */ | 883 | /* balanced states don't exist at the moment */ |
| 887 | if (dpm_state == POWER_STATE_TYPE_BALANCED) | 884 | if (dpm_state == POWER_STATE_TYPE_BALANCED) |
| 888 | dpm_state = rdev->pm.dpm.ac_power ? | 885 | dpm_state = POWER_STATE_TYPE_PERFORMANCE; |
| 889 | POWER_STATE_TYPE_PERFORMANCE : POWER_STATE_TYPE_BATTERY; | ||
| 890 | 886 | ||
| 891 | restart_search: | 887 | restart_search: |
| 892 | /* Pick the best power state based on current conditions */ | 888 | /* Pick the best power state based on current conditions */ |
| @@ -461,7 +461,7 @@ void drm_sched_hw_job_reset(struct drm_gpu_scheduler *sched, struct drm_sched_jo · 460 unmodified lines | |||
| 461 | { | 461 | { |
| 462 | struct drm_sched_job *s_job; | 462 | struct drm_sched_job *s_job; |
| 463 | struct drm_sched_entity *entity, *tmp; | 463 | struct drm_sched_entity *entity, *tmp; |
| 464 | int i;; | 464 | int i; |
| 465 | 465 | ||
| 466 | spin_lock(&sched->job_list_lock); | 466 | spin_lock(&sched->job_list_lock); |
| 467 | list_for_each_entry_reverse(s_job, &sched->ring_mirror_list, node) { | 467 | list_for_each_entry_reverse(s_job, &sched->ring_mirror_list, node) { |
| @@ -93,6 +93,8 @@ static void sun4i_crtc_atomic_disable(struct drm_crtc *crtc, · 92 unmodified lines | |||
| 93 | 93 | ||
| 94 | DRM_DEBUG_DRIVER("Disabling the CRTC\n"); | 94 | DRM_DEBUG_DRIVER("Disabling the CRTC\n"); |
| 95 | 95 | ||
| 96 | drm_crtc_vblank_off(crtc); | ||
| 97 | |||
| 96 | sun4i_tcon_set_status(scrtc->tcon, encoder, false); | 98 | sun4i_tcon_set_status(scrtc->tcon, encoder, false); |
| 97 | 99 | ||
| 98 | if (crtc->state->event && !crtc->state->active) { | 100 | if (crtc->state->event && !crtc->state->active) { |
| @@ -113,6 +115,8 @@ static void sun4i_crtc_atomic_enable(struct drm_crtc *crtc, · 14 unmodified lines | |||
| 113 | DRM_DEBUG_DRIVER("Enabling the CRTC\n"); | 115 | DRM_DEBUG_DRIVER("Enabling the CRTC\n"); |
| 114 | 116 | ||
| 115 | sun4i_tcon_set_status(scrtc->tcon, encoder, true); | 117 | sun4i_tcon_set_status(scrtc->tcon, encoder, true); |
| 118 | |||
| 119 | drm_crtc_vblank_on(crtc); | ||
| 116 | } | 120 | } |
| 117 | 121 | ||
| 118 | static void sun4i_crtc_mode_set_nofb(struct drm_crtc *crtc) | 122 | static void sun4i_crtc_mode_set_nofb(struct drm_crtc *crtc) |
| @@ -132,10 +132,13 @@ static int sun4i_dclk_get_phase(struct clk_hw *hw) · 131 unmodified lines | |||
| 132 | static int sun4i_dclk_set_phase(struct clk_hw *hw, int degrees) | 132 | static int sun4i_dclk_set_phase(struct clk_hw *hw, int degrees) |
| 133 | { | 133 | { |
| 134 | struct sun4i_dclk *dclk = hw_to_dclk(hw); | 134 | struct sun4i_dclk *dclk = hw_to_dclk(hw); |
| 135 | u32 val = degrees / 120; | ||
| 136 | |||
| 137 | val <<= 28; | ||
| 135 | 138 | ||
| 136 | regmap_update_bits(dclk->regmap, SUN4I_TCON0_IO_POL_REG, | 139 | regmap_update_bits(dclk->regmap, SUN4I_TCON0_IO_POL_REG, |
| 137 | GENMASK(29, 28), | 140 | GENMASK(29, 28), |
| 138 | degrees / 120); | 141 | val); |
| 139 | 142 | ||
| 140 | return 0; | 143 | return 0; |
| 141 | } | 144 | } |
| @@ -92,6 +92,8 @@ static int sun4i_rgb_mode_valid(struct drm_connector *connector, · 91 unmodified lines | |||
| 92 | 92 | ||
| 93 | DRM_DEBUG_DRIVER("Vertical parameters OK\n"); | 93 | DRM_DEBUG_DRIVER("Vertical parameters OK\n"); |
| 94 | 94 | ||
| 95 | tcon->dclk_min_div = 6; | ||
| 96 | tcon->dclk_max_div = 127; | ||
| 95 | rounded_rate = clk_round_rate(tcon->dclk, rate); | 97 | rounded_rate = clk_round_rate(tcon->dclk, rate); |
| 96 | if (rounded_rate < rate) | 98 | if (rounded_rate < rate) |
| 97 | return MODE_CLOCK_LOW; | 99 | return MODE_CLOCK_LOW; |
| @@ -101,10 +101,12 @@ static void sun4i_tcon_channel_set_status(struct sun4i_tcon *tcon, int channel, · 100 unmodified lines | |||
| 101 | return; | 101 | return; |
| 102 | } | 102 | } |
| 103 | 103 | ||
| 104 | if (enabled) | 104 | if (enabled) { |
| 105 | clk_prepare_enable(clk); | 105 | clk_prepare_enable(clk); |
| 106 | else | 106 | } else { |
| 107 | clk_rate_exclusive_put(clk); | ||
| 107 | clk_disable_unprepare(clk); | 108 | clk_disable_unprepare(clk); |
| 109 | } | ||
| 108 | } | 110 | } |
| 109 | 111 | ||
| 110 | static void sun4i_tcon_lvds_set_status(struct sun4i_tcon *tcon, | 112 | static void sun4i_tcon_lvds_set_status(struct sun4i_tcon *tcon, |
| @@ -260,7 +262,7 @@ static void sun4i_tcon0_mode_set_common(struct sun4i_tcon *tcon, · 149 unmodified lines | |||
| 260 | const struct drm_display_mode *mode) | 262 | const struct drm_display_mode *mode) |
| 261 | { | 263 | { |
| 262 | /* Configure the dot clock */ | 264 | /* Configure the dot clock */ |
| 263 | clk_set_rate(tcon->dclk, mode->crtc_clock * 1000); | 265 | clk_set_rate_exclusive(tcon->dclk, mode->crtc_clock * 1000); |
| 264 | 266 | ||
| 265 | /* Set the resolution */ | 267 | /* Set the resolution */ |
| 266 | regmap_write(tcon->regs, SUN4I_TCON0_BASIC0_REG, | 268 | regmap_write(tcon->regs, SUN4I_TCON0_BASIC0_REG, |
| @@ -335,6 +337,9 @@ static void sun4i_tcon0_mode_set_lvds(struct sun4i_tcon *tcon, · 68 unmodified lines | |||
| 335 | regmap_update_bits(tcon->regs, SUN4I_TCON_GCTL_REG, | 337 | regmap_update_bits(tcon->regs, SUN4I_TCON_GCTL_REG, |
| 336 | SUN4I_TCON_GCTL_IOMAP_MASK, | 338 | SUN4I_TCON_GCTL_IOMAP_MASK, |
| 337 | SUN4I_TCON_GCTL_IOMAP_TCON0); | 339 | SUN4I_TCON_GCTL_IOMAP_TCON0); |
| 340 | |||
| 341 | /* Enable the output on the pins */ | ||
| 342 | regmap_write(tcon->regs, SUN4I_TCON0_IO_TRI_REG, 0xe0000000); | ||
| 338 | } | 343 | } |
| 339 | 344 | ||
| 340 | static void sun4i_tcon0_mode_set_rgb(struct sun4i_tcon *tcon, | 345 | static void sun4i_tcon0_mode_set_rgb(struct sun4i_tcon *tcon, |
| @@ -418,7 +423,7 @@ static void sun4i_tcon1_mode_set(struct sun4i_tcon *tcon, · 77 unmodified lines | |||
| 418 | WARN_ON(!tcon->quirks->has_channel_1); | 423 | WARN_ON(!tcon->quirks->has_channel_1); |
| 419 | 424 | ||
| 420 | /* Configure the dot clock */ | 425 | /* Configure the dot clock */ |
| 421 | clk_set_rate(tcon->sclk1, mode->crtc_clock * 1000); | 426 | clk_set_rate_exclusive(tcon->sclk1, mode->crtc_clock * 1000); |
| 422 | 427 | ||
| 423 | /* Adjust clock delay */ | 428 | /* Adjust clock delay */ |
| 424 | clk_delay = sun4i_tcon_get_clk_delay(mode, 1); | 429 | clk_delay = sun4i_tcon_get_clk_delay(mode, 1); |
| @@ -870,52 +875,56 @@ static int sun4i_tcon_bind(struct device *dev, struct device *master, · 445 unmodified lines | |||
| 870 | return ret; | 875 | return ret; |
| 871 | } | 876 | } |
| 872 | 877 | ||
| 873 | /* | 878 | if (tcon->quirks->supports_lvds) { |
| 874 | * This can only be made optional since we've had DT nodes | 879 | /* |
| 875 | * without the LVDS reset properties. | 880 | * This can only be made optional since we've had DT |
| 876 | * | 881 | * nodes without the LVDS reset properties. |
| 877 | * If the property is missing, just disable LVDS, and print a | 882 | * |
| 878 | * warning. | 883 | * If the property is missing, just disable LVDS, and |
| 879 | */ | 884 | * print a warning. |
| 880 | tcon->lvds_rst = devm_reset_control_get_optional(dev, "lvds"); | 885 | */ |
| 881 | if (IS_ERR(tcon->lvds_rst)) { | 886 | tcon->lvds_rst = devm_reset_control_get_optional(dev, "lvds"); |
| 882 | dev_err(dev, "Couldn't get our reset line\n"); | 887 | if (IS_ERR(tcon->lvds_rst)) { |
| 883 | return PTR_ERR(tcon->lvds_rst); | 888 | dev_err(dev, "Couldn't get our reset line\n"); |
| 884 | } else if (tcon->lvds_rst) { | 889 | return PTR_ERR(tcon->lvds_rst); |
| 885 | has_lvds_rst = true; | 890 | } else if (tcon->lvds_rst) { |
| 886 | reset_control_reset(tcon->lvds_rst); | 891 | has_lvds_rst = true; |
| 887 | } else { | 892 | reset_control_reset(tcon->lvds_rst); |
| 888 | has_lvds_rst = false; | 893 | } else { |
| 889 | } | 894 | has_lvds_rst = false; |
| 895 | } | ||
| 890 | 896 | ||
| 891 | /* | 897 | /* |
| 892 | * This can only be made optional since we've had DT nodes | 898 | * This can only be made optional since we've had DT |
| 893 | * without the LVDS reset properties. | 899 | * nodes without the LVDS reset properties. |
| 894 | * | 900 | * |
| 895 | * If the property is missing, just disable LVDS, and print a | 901 | * If the property is missing, just disable LVDS, and |
| 896 | * warning. | 902 | * print a warning. |
| 897 | */ | 903 | */ |
| 898 | if (tcon->quirks->has_lvds_alt) { | 904 | if (tcon->quirks->has_lvds_alt) { |
| 899 | tcon->lvds_pll = devm_clk_get(dev, "lvds-alt"); | 905 | tcon->lvds_pll = devm_clk_get(dev, "lvds-alt"); |
| 900 | if (IS_ERR(tcon->lvds_pll)) { | 906 | if (IS_ERR(tcon->lvds_pll)) { |
| 901 | if (PTR_ERR(tcon->lvds_pll) == -ENOENT) { | 907 | if (PTR_ERR(tcon->lvds_pll) == -ENOENT) { |
| 902 | has_lvds_alt = false; | 908 | has_lvds_alt = false; |
| 909 | } else { | ||
| 910 | dev_err(dev, "Couldn't get the LVDS PLL\n"); | ||
| 911 | return PTR_ERR(tcon->lvds_pll); | ||
| 912 | } | ||
| 903 | } else { | 913 | } else { |
| 904 | dev_err(dev, "Couldn't get the LVDS PLL\n"); | 914 | has_lvds_alt = true; |
| 905 | return PTR_ERR(tcon->lvds_pll); | ||
| 906 | } | 915 | } |
| 907 | } else { | ||
| 908 | has_lvds_alt = true; | ||
| 909 | } | 916 | } |
| 910 | } | ||
| 911 | 917 | ||
| 912 | if (!has_lvds_rst || (tcon->quirks->has_lvds_alt && !has_lvds_alt)) { | 918 | if (!has_lvds_rst || |
| 913 | dev_warn(dev, | 919 | (tcon->quirks->has_lvds_alt && !has_lvds_alt)) { |
| 914 | "Missing LVDS properties, Please upgrade your DT\n"); | 920 | dev_warn(dev, "Missing LVDS properties, Please upgrade your DT\n"); |
| 915 | dev_warn(dev, "LVDS output disabled\n"); | 921 | dev_warn(dev, "LVDS output disabled\n"); |
| 916 | can_lvds = false; | 922 | can_lvds = false; |
| 923 | } else { | ||
| 924 | can_lvds = true; | ||
| 925 | } | ||
| 917 | } else { | 926 | } else { |
| 918 | can_lvds = true; | 927 | can_lvds = false; |
| 919 | } | 928 | } |
| 920 | 929 | ||
| 921 | ret = sun4i_tcon_init_clocks(dev, tcon); | 930 | ret = sun4i_tcon_init_clocks(dev, tcon); |
| @@ -1134,7 +1143,7 @@ static const struct sun4i_tcon_quirks sun8i_a33_quirks = { · 212 unmodified lines | |||
| 1134 | }; | 1143 | }; |
| 1135 | 1144 | ||
| 1136 | static const struct sun4i_tcon_quirks sun8i_a83t_lcd_quirks = { | 1145 | static const struct sun4i_tcon_quirks sun8i_a83t_lcd_quirks = { |
| 1137 | /* nothing is supported */ | 1146 | .supports_lvds = true, |
| 1138 | }; | 1147 | }; |
| 1139 | 1148 | ||
| 1140 | static const struct sun4i_tcon_quirks sun8i_v3s_quirks = { | 1149 | static const struct sun4i_tcon_quirks sun8i_v3s_quirks = { |
| @@ -175,6 +175,7 @@ struct sun4i_tcon_quirks { · 174 unmodified lines | |||
| 175 | bool has_channel_1; /* a33 does not have channel 1 */ | 175 | bool has_channel_1; /* a33 does not have channel 1 */ |
| 176 | bool has_lvds_alt; /* Does the LVDS clock have a parent other than the TCON clock? */ | 176 | bool has_lvds_alt; /* Does the LVDS clock have a parent other than the TCON clock? */ |
| 177 | bool needs_de_be_mux; /* sun6i needs mux to select backend */ | 177 | bool needs_de_be_mux; /* sun6i needs mux to select backend */ |
| 178 | bool supports_lvds; /* Does the TCON support an LVDS output? */ | ||
| 178 | 179 | ||
| 179 | /* callback to handle tcon muxing options */ | 180 | /* callback to handle tcon muxing options */ |
| 180 | int (*set_mux)(struct sun4i_tcon *, const struct drm_encoder *); | 181 | int (*set_mux)(struct sun4i_tcon *, const struct drm_encoder *); |
| @@ -197,6 +197,9 @@ static int virtio_gpu_getparam_ioctl(struct drm_device *dev, void *data, · 196 unmodified lines | |||
| 197 | case VIRTGPU_PARAM_3D_FEATURES: | 197 | case VIRTGPU_PARAM_3D_FEATURES: |
| 198 | value = vgdev->has_virgl_3d == true ? 1 : 0; | 198 | value = vgdev->has_virgl_3d == true ? 1 : 0; |
| 199 | break; | 199 | break; |
| 200 | case VIRTGPU_PARAM_CAPSET_QUERY_FIX: | ||
| 201 | value = 1; | ||
| 202 | break; | ||
| 200 | default: | 203 | default: |
| 201 | return -EINVAL; | 204 | return -EINVAL; |
| 202 | } | 205 | } |
| @@ -472,7 +475,7 @@ static int virtio_gpu_get_caps_ioctl(struct drm_device *dev, · 269 unmodified lines | |||
| 472 | { | 475 | { |
| 473 | struct virtio_gpu_device *vgdev = dev->dev_private; | 476 | struct virtio_gpu_device *vgdev = dev->dev_private; |
| 474 | struct drm_virtgpu_get_caps *args = data; | 477 | struct drm_virtgpu_get_caps *args = data; |
| 475 | int size; | 478 | unsigned size, host_caps_size; |
| 476 | int i; | 479 | int i; |
| 477 | int found_valid = -1; | 480 | int found_valid = -1; |
| 478 | int ret; | 481 | int ret; |
| @@ -481,6 +484,10 @@ static int virtio_gpu_get_caps_ioctl(struct drm_device *dev, · 2 unmodified lines | |||
| 481 | if (vgdev->num_capsets == 0) | 484 | if (vgdev->num_capsets == 0) |
| 482 | return -ENOSYS; | 485 | return -ENOSYS; |
| 483 | 486 | ||
| 487 | /* don't allow userspace to pass 0 */ | ||
| 488 | if (args->size == 0) | ||
| 489 | return -EINVAL; | ||
| 490 | |||
| 484 | spin_lock(&vgdev->display_info_lock); | 491 | spin_lock(&vgdev->display_info_lock); |
| 485 | for (i = 0; i < vgdev->num_capsets; i++) { | 492 | for (i = 0; i < vgdev->num_capsets; i++) { |
| 486 | if (vgdev->capsets[i].id == args->cap_set_id) { | 493 | if (vgdev->capsets[i].id == args->cap_set_id) { |
| @@ -496,11 +503,9 @@ static int virtio_gpu_get_caps_ioctl(struct drm_device *dev, · 9 unmodified lines | |||
| 496 | return -EINVAL; | 503 | return -EINVAL; |
| 497 | } | 504 | } |
| 498 | 505 | ||
| 499 | size = vgdev->capsets[found_valid].max_size; | 506 | host_caps_size = vgdev->capsets[found_valid].max_size; |
| 500 | if (args->size > size) { | 507 | /* only copy to user the minimum of the host caps size or the guest caps size */ |
| 501 | spin_unlock(&vgdev->display_info_lock); | 508 | size = min(args->size, host_caps_size); |
| 502 | return -EINVAL; | ||
| 503 | } | ||
| 504 | 509 | ||
| 505 | list_for_each_entry(cache_ent, &vgdev->cap_cache, head) { | 510 | list_for_each_entry(cache_ent, &vgdev->cap_cache, head) { |
| 506 | if (cache_ent->id == args->cap_set_id && | 511 | if (cache_ent->id == args->cap_set_id && |
| @@ -1089,7 +1089,7 @@ static void ipu_irq_handler(struct irq_desc *desc) · 1088 unmodified lines | |||
| 1089 | { | 1089 | { |
| 1090 | struct ipu_soc *ipu = irq_desc_get_handler_data(desc); | 1090 | struct ipu_soc *ipu = irq_desc_get_handler_data(desc); |
| 1091 | struct irq_chip *chip = irq_desc_get_chip(desc); | 1091 | struct irq_chip *chip = irq_desc_get_chip(desc); |
| 1092 | const int int_reg[] = { 0, 1, 2, 3, 10, 11, 12, 13, 14}; | 1092 | static const int int_reg[] = { 0, 1, 2, 3, 10, 11, 12, 13, 14}; |
| 1093 | 1093 | ||
| 1094 | chained_irq_enter(chip, desc); | 1094 | chained_irq_enter(chip, desc); |
| 1095 | 1095 | ||
| @@ -1102,7 +1102,7 @@ static void ipu_err_irq_handler(struct irq_desc *desc) · 6 unmodified lines | |||
| 1102 | { | 1102 | { |
| 1103 | struct ipu_soc *ipu = irq_desc_get_handler_data(desc); | 1103 | struct ipu_soc *ipu = irq_desc_get_handler_data(desc); |
| 1104 | struct irq_chip *chip = irq_desc_get_chip(desc); | 1104 | struct irq_chip *chip = irq_desc_get_chip(desc); |
| 1105 | const int int_reg[] = { 4, 5, 8, 9}; | 1105 | static const int int_reg[] = { 4, 5, 8, 9}; |
| 1106 | 1106 | ||
| 1107 | chained_irq_enter(chip, desc); | 1107 | chained_irq_enter(chip, desc); |
| 1108 | 1108 | ||
| @@ -788,12 +788,14 @@ int ipu_cpmem_set_image(struct ipuv3_channel *ch, struct ipu_image *image) · 787 unmodified lines | |||
| 788 | case V4L2_PIX_FMT_SGBRG8: | 788 | case V4L2_PIX_FMT_SGBRG8: |
| 789 | case V4L2_PIX_FMT_SGRBG8: | 789 | case V4L2_PIX_FMT_SGRBG8: |
| 790 | case V4L2_PIX_FMT_SRGGB8: | 790 | case V4L2_PIX_FMT_SRGGB8: |
| 791 | case V4L2_PIX_FMT_GREY: | ||
| 791 | offset = image->rect.left + image->rect.top * pix->bytesperline; | 792 | offset = image->rect.left + image->rect.top * pix->bytesperline; |
| 792 | break; | 793 | break; |
| 793 | case V4L2_PIX_FMT_SBGGR16: | 794 | case V4L2_PIX_FMT_SBGGR16: |
| 794 | case V4L2_PIX_FMT_SGBRG16: | 795 | case V4L2_PIX_FMT_SGBRG16: |
| 795 | case V4L2_PIX_FMT_SGRBG16: | 796 | case V4L2_PIX_FMT_SGRBG16: |
| 796 | case V4L2_PIX_FMT_SRGGB16: | 797 | case V4L2_PIX_FMT_SRGGB16: |
| 798 | case V4L2_PIX_FMT_Y16: | ||
| 797 | offset = image->rect.left * 2 + | 799 | offset = image->rect.left * 2 + |
| 798 | image->rect.top * pix->bytesperline; | 800 | image->rect.top * pix->bytesperline; |
| 799 | break; | 801 | break; |
| @@ -288,6 +288,7 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config *cfg, u32 mbus_code) · 287 unmodified lines | |||
| 288 | case MEDIA_BUS_FMT_SGBRG10_1X10: | 288 | case MEDIA_BUS_FMT_SGBRG10_1X10: |
| 289 | case MEDIA_BUS_FMT_SGRBG10_1X10: | 289 | case MEDIA_BUS_FMT_SGRBG10_1X10: |
| 290 | case MEDIA_BUS_FMT_SRGGB10_1X10: | 290 | case MEDIA_BUS_FMT_SRGGB10_1X10: |
| 291 | case MEDIA_BUS_FMT_Y10_1X10: | ||
| 291 | cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER; | 292 | cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER; |
| 292 | cfg->mipi_dt = MIPI_DT_RAW10; | 293 | cfg->mipi_dt = MIPI_DT_RAW10; |
| 293 | cfg->data_width = IPU_CSI_DATA_WIDTH_10; | 294 | cfg->data_width = IPU_CSI_DATA_WIDTH_10; |
| @@ -296,6 +297,7 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config *cfg, u32 mbus_code) · 2 unmodified lines | |||
| 296 | case MEDIA_BUS_FMT_SGBRG12_1X12: | 297 | case MEDIA_BUS_FMT_SGBRG12_1X12: |
| 297 | case MEDIA_BUS_FMT_SGRBG12_1X12: | 298 | case MEDIA_BUS_FMT_SGRBG12_1X12: |
| 298 | case MEDIA_BUS_FMT_SRGGB12_1X12: | 299 | case MEDIA_BUS_FMT_SRGGB12_1X12: |
| 300 | case MEDIA_BUS_FMT_Y12_1X12: | ||
| 299 | cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER; | 301 | cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER; |
| 300 | cfg->mipi_dt = MIPI_DT_RAW12; | 302 | cfg->mipi_dt = MIPI_DT_RAW12; |
| 301 | cfg->data_width = IPU_CSI_DATA_WIDTH_12; | 303 | cfg->data_width = IPU_CSI_DATA_WIDTH_12; |
| @@ -129,11 +129,14 @@ ipu_pre_lookup_by_phandle(struct device *dev, const char *name, int index) · 128 unmodified lines | |||
| 129 | if (pre_node == pre->dev->of_node) { | 129 | if (pre_node == pre->dev->of_node) { |
| 130 | mutex_unlock(&ipu_pre_list_mutex); | 130 | mutex_unlock(&ipu_pre_list_mutex); |
| 131 | device_link_add(dev, pre->dev, DL_FLAG_AUTOREMOVE); | 131 | device_link_add(dev, pre->dev, DL_FLAG_AUTOREMOVE); |
| 132 | of_node_put(pre_node); | ||
| 132 | return pre; | 133 | return pre; |
| 133 | } | 134 | } |
| 134 | } | 135 | } |
| 135 | mutex_unlock(&ipu_pre_list_mutex); | 136 | mutex_unlock(&ipu_pre_list_mutex); |
| 136 | 137 | ||
| 138 | of_node_put(pre_node); | ||
| 139 | |||
| 137 | return NULL; | 140 | return NULL; |
| 138 | } | 141 | } |
| 139 | 142 | ||
| @@ -102,11 +102,14 @@ ipu_prg_lookup_by_phandle(struct device *dev, const char *name, int ipu_id) · 101 unmodified lines | |||
| 102 | mutex_unlock(&ipu_prg_list_mutex); | 102 | mutex_unlock(&ipu_prg_list_mutex); |
| 103 | device_link_add(dev, prg->dev, DL_FLAG_AUTOREMOVE); | 103 | device_link_add(dev, prg->dev, DL_FLAG_AUTOREMOVE); |
| 104 | prg->id = ipu_id; | 104 | prg->id = ipu_id; |
| 105 | of_node_put(prg_node); | ||
| 105 | return prg; | 106 | return prg; |
| 106 | } | 107 | } |
| 107 | } | 108 | } |
| 108 | mutex_unlock(&ipu_prg_list_mutex); | 109 | mutex_unlock(&ipu_prg_list_mutex); |
| 109 | 110 | ||
| 111 | of_node_put(prg_node); | ||
| 112 | |||
| 110 | return NULL; | 113 | return NULL; |
| 111 | } | 114 | } |
| 112 | 115 | ||
| @@ -645,6 +645,9 @@ · 644 unmodified lines | |||
| 645 | #define USB_DEVICE_ID_LD_MICROCASSYTIME 0x1033 | 645 | #define USB_DEVICE_ID_LD_MICROCASSYTIME 0x1033 |
| 646 | #define USB_DEVICE_ID_LD_MICROCASSYTEMPERATURE 0x1035 | 646 | #define USB_DEVICE_ID_LD_MICROCASSYTEMPERATURE 0x1035 |
| 647 | #define USB_DEVICE_ID_LD_MICROCASSYPH 0x1038 | 647 | #define USB_DEVICE_ID_LD_MICROCASSYPH 0x1038 |
| 648 | #define USB_DEVICE_ID_LD_POWERANALYSERCASSY 0x1040 | ||
| 649 | #define USB_DEVICE_ID_LD_CONVERTERCONTROLLERCASSY 0x1042 | ||
| 650 | #define USB_DEVICE_ID_LD_MACHINETESTCASSY 0x1043 | ||
| 648 | #define USB_DEVICE_ID_LD_JWM 0x1080 | 651 | #define USB_DEVICE_ID_LD_JWM 0x1080 |
| 649 | #define USB_DEVICE_ID_LD_DMMP 0x1081 | 652 | #define USB_DEVICE_ID_LD_DMMP 0x1081 |
| 650 | #define USB_DEVICE_ID_LD_UMIP 0x1090 | 653 | #define USB_DEVICE_ID_LD_UMIP 0x1090 |
| @@ -809,6 +809,9 @@ static const struct hid_device_id hid_ignore_list[] = { · 808 unmodified lines | |||
| 809 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MICROCASSYTIME) }, | 809 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MICROCASSYTIME) }, |
| 810 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MICROCASSYTEMPERATURE) }, | 810 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MICROCASSYTEMPERATURE) }, |
| 811 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MICROCASSYPH) }, | 811 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MICROCASSYPH) }, |
| 812 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_POWERANALYSERCASSY) }, | ||
| 813 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_CONVERTERCONTROLLERCASSY) }, | ||
| 814 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_MACHINETESTCASSY) }, | ||
| 812 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_JWM) }, | 815 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_JWM) }, |
| 813 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_DMMP) }, | 816 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_DMMP) }, |
| 814 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_UMIP) }, | 817 | { HID_USB_DEVICE(USB_VENDOR_ID_LD, USB_DEVICE_ID_LD_UMIP) }, |
| @@ -269,13 +269,13 @@ static int adjust_tjmax(struct cpuinfo_x86 *c, u32 id, struct device *dev) · 268 unmodified lines | |||
| 269 | for (i = 0; i < ARRAY_SIZE(tjmax_model_table); i++) { | 269 | for (i = 0; i < ARRAY_SIZE(tjmax_model_table); i++) { |
| 270 | const struct tjmax_model *tm = &tjmax_model_table[i]; | 270 | const struct tjmax_model *tm = &tjmax_model_table[i]; |
| 271 | if (c->x86_model == tm->model && | 271 | if (c->x86_model == tm->model && |
| 272 | (tm->mask == ANY || c->x86_mask == tm->mask)) | 272 | (tm->mask == ANY || c->x86_stepping == tm->mask)) |
| 273 | return tm->tjmax; | 273 | return tm->tjmax; |
| 274 | } | 274 | } |
| 275 | 275 | ||
| 276 | /* Early chips have no MSR for TjMax */ | 276 | /* Early chips have no MSR for TjMax */ |
| 277 | 277 | ||
| 278 | if (c->x86_model == 0xf && c->x86_mask < 4) | 278 | if (c->x86_model == 0xf && c->x86_stepping < 4) |
| 279 | usemsr_ee = 0; | 279 | usemsr_ee = 0; |
| 280 | 280 | ||
| 281 | if (c->x86_model > 0xe && usemsr_ee) { | 281 | if (c->x86_model > 0xe && usemsr_ee) { |
| @@ -426,7 +426,7 @@ static int chk_ucode_version(unsigned int cpu) · 144 unmodified lines | |||
| 426 | * Readings might stop update when processor visited too deep sleep, | 426 | * Readings might stop update when processor visited too deep sleep, |
| 427 | * fixed for stepping D0 (6EC). | 427 | * fixed for stepping D0 (6EC). |
| 428 | */ | 428 | */ |
| 429 | if (c->x86_model == 0xe && c->x86_mask < 0xc && c->microcode < 0x39) { | 429 | if (c->x86_model == 0xe && c->x86_stepping < 0xc && c->microcode < 0x39) { |
| 430 | pr_err("Errata AE18 not fixed, update BIOS or microcode of the CPU!\n"); | 430 | pr_err("Errata AE18 not fixed, update BIOS or microcode of the CPU!\n"); |
| 431 | return -ENODEV; | 431 | return -ENODEV; |
| 432 | } | 432 | } |
| @@ -293,7 +293,7 @@ u8 vid_which_vrm(void) · 292 unmodified lines | |||
| 293 | if (c->x86 < 6) /* Any CPU with family lower than 6 */ | 293 | if (c->x86 < 6) /* Any CPU with family lower than 6 */ |
| 294 | return 0; /* doesn't have VID */ | 294 | return 0; /* doesn't have VID */ |
| 295 | 295 | ||
| 296 | vrm_ret = find_vrm(c->x86, c->x86_model, c->x86_mask, c->x86_vendor); | 296 | vrm_ret = find_vrm(c->x86, c->x86_model, c->x86_stepping, c->x86_vendor); |
| 297 | if (vrm_ret == 134) | 297 | if (vrm_ret == 134) |
| 298 | vrm_ret = get_via_model_d_vrm(); | 298 | vrm_ret = get_via_model_d_vrm(); |
| 299 | if (vrm_ret == 0) | 299 | if (vrm_ret == 0) |
| @@ -129,7 +129,10 @@ static ssize_t temp1_input_show(struct device *dev, · 128 unmodified lines | |||
| 129 | 129 | ||
| 130 | data->read_tempreg(data->pdev, ®val); | 130 | data->read_tempreg(data->pdev, ®val); |
| 131 | temp = (regval >> 21) * 125; | 131 | temp = (regval >> 21) * 125; |
| 132 | temp -= data->temp_offset; | 132 | if (temp > data->temp_offset) |
| 133 | temp -= data->temp_offset; | ||
| 134 | else | ||
| 135 | temp = 0; | ||
| 133 | 136 | ||
| 134 | return sprintf(buf, "%u\n", temp); | 137 | return sprintf(buf, "%u\n", temp); |
| 135 | } | 138 | } |
| @@ -227,7 +230,7 @@ static bool has_erratum_319(struct pci_dev *pdev) · 91 unmodified lines | |||
| 227 | * and AM3 formats, but that's the best we can do. | 230 | * and AM3 formats, but that's the best we can do. |
| 228 | */ | 231 | */ |
| 229 | return boot_cpu_data.x86_model < 4 || | 232 | return boot_cpu_data.x86_model < 4 || |
| 230 | (boot_cpu_data.x86_model == 4 && boot_cpu_data.x86_mask <= 2); | 233 | (boot_cpu_data.x86_model == 4 && boot_cpu_data.x86_stepping <= 2); |
| 231 | } | 234 | } |
| 232 | 235 | ||
| 233 | static int k10temp_probe(struct pci_dev *pdev, | 236 | static int k10temp_probe(struct pci_dev *pdev, |
| @@ -187,7 +187,7 @@ static int k8temp_probe(struct pci_dev *pdev, · 186 unmodified lines | |||
| 187 | return -ENOMEM; | 187 | return -ENOMEM; |
| 188 | 188 | ||
| 189 | model = boot_cpu_data.x86_model; | 189 | model = boot_cpu_data.x86_model; |
| 190 | stepping = boot_cpu_data.x86_mask; | 190 | stepping = boot_cpu_data.x86_stepping; |
| 191 | 191 | ||
| 192 | /* feature available since SH-C0, exclude older revisions */ | 192 | /* feature available since SH-C0, exclude older revisions */ |
| 193 | if ((model == 4 && stepping == 0) || | 193 | if ((model == 4 && stepping == 0) || |
| @@ -123,8 +123,10 @@ config I2C_I801 · 122 unmodified lines | |||
| 123 | Wildcat Point (PCH) | 123 | Wildcat Point (PCH) |
| 124 | Wildcat Point-LP (PCH) | 124 | Wildcat Point-LP (PCH) |
| 125 | BayTrail (SOC) | 125 | BayTrail (SOC) |
| 126 | Braswell (SOC) | ||
| 126 | Sunrise Point-H (PCH) | 127 | Sunrise Point-H (PCH) |
| 127 | Sunrise Point-LP (PCH) | 128 | Sunrise Point-LP (PCH) |
| 129 | Kaby Lake-H (PCH) | ||
| 128 | DNV (SOC) | 130 | DNV (SOC) |
| 129 | Broxton (SOC) | 131 | Broxton (SOC) |
| 130 | Lewisburg (PCH) | 132 | Lewisburg (PCH) |
| @@ -50,6 +50,9 @@ · 49 unmodified lines | |||
| 50 | #define BCM2835_I2C_S_CLKT BIT(9) | 50 | #define BCM2835_I2C_S_CLKT BIT(9) |
| 51 | #define BCM2835_I2C_S_LEN BIT(10) /* Fake bit for SW error reporting */ | 51 | #define BCM2835_I2C_S_LEN BIT(10) /* Fake bit for SW error reporting */ |
| 52 | 52 | ||
| 53 | #define BCM2835_I2C_FEDL_SHIFT 16 | ||
| 54 | #define BCM2835_I2C_REDL_SHIFT 0 | ||
| 55 | |||
| 53 | #define BCM2835_I2C_CDIV_MIN 0x0002 | 56 | #define BCM2835_I2C_CDIV_MIN 0x0002 |
| 54 | #define BCM2835_I2C_CDIV_MAX 0xFFFE | 57 | #define BCM2835_I2C_CDIV_MAX 0xFFFE |
| 55 | 58 | ||
| @@ -81,7 +84,7 @@ static inline u32 bcm2835_i2c_readl(struct bcm2835_i2c_dev *i2c_dev, u32 reg) · 25 unmodified lines | |||
| 81 | 84 | ||
| 82 | static int bcm2835_i2c_set_divider(struct bcm2835_i2c_dev *i2c_dev) | 85 | static int bcm2835_i2c_set_divider(struct bcm2835_i2c_dev *i2c_dev) |
| 83 | { | 86 | { |
| 84 | u32 divider; | 87 | u32 divider, redl, fedl; |
| 85 | 88 | ||
| 86 | divider = DIV_ROUND_UP(clk_get_rate(i2c_dev->clk), | 89 | divider = DIV_ROUND_UP(clk_get_rate(i2c_dev->clk), |
| 87 | i2c_dev->bus_clk_rate); | 90 | i2c_dev->bus_clk_rate); |
| @@ -100,6 +103,22 @@ static int bcm2835_i2c_set_divider(struct bcm2835_i2c_dev *i2c_dev) · 12 unmodified lines | |||
| 100 | 103 | ||
| 101 | bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_DIV, divider); | 104 | bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_DIV, divider); |
| 102 | 105 | ||
| 106 | /* | ||
| 107 | * Number of core clocks to wait after falling edge before | ||
| 108 | * outputting the next data bit. Note that both FEDL and REDL | ||
| 109 | * can't be greater than CDIV/2. | ||
| 110 | */ | ||
| 111 | fedl = max(divider / 16, 1u); | ||
| 112 | |||
| 113 | /* | ||
| 114 | * Number of core clocks to wait after rising edge before | ||
| 115 | * sampling the next incoming data bit. | ||
| 116 | */ | ||
| 117 | redl = max(divider / 4, 1u); | ||
| 118 | |||
| 119 | bcm2835_i2c_writel(i2c_dev, BCM2835_I2C_DEL, | ||
| 120 | (fedl << BCM2835_I2C_FEDL_SHIFT) | | ||
| 121 | (redl << BCM2835_I2C_REDL_SHIFT)); | ||
| 103 | return 0; | 122 | return 0; |
| 104 | } | 123 | } |
| 105 | 124 | ||
| @@ -209,7 +209,7 @@ static void i2c_dw_xfer_init(struct dw_i2c_dev *dev) · 208 unmodified lines | |||
| 209 | i2c_dw_disable_int(dev); | 209 | i2c_dw_disable_int(dev); |
| 210 | 210 | ||
| 211 | /* Enable the adapter */ | 211 | /* Enable the adapter */ |
| 212 | __i2c_dw_enable(dev, true); | 212 | __i2c_dw_enable_and_wait(dev, true); |
| 213 | 213 | ||
| 214 | /* Clear and enable interrupts */ | 214 | /* Clear and enable interrupts */ |
| 215 | dw_readl(dev, DW_IC_CLR_INTR); | 215 | dw_readl(dev, DW_IC_CLR_INTR); |
| @@ -644,7 +644,7 @@ static int i2c_dw_init_recovery_info(struct dw_i2c_dev *dev) · 428 unmodified lines | |||
| 644 | gpio = devm_gpiod_get(dev->dev, "scl", GPIOD_OUT_HIGH); | 644 | gpio = devm_gpiod_get(dev->dev, "scl", GPIOD_OUT_HIGH); |
| 645 | if (IS_ERR(gpio)) { | 645 | if (IS_ERR(gpio)) { |
| 646 | r = PTR_ERR(gpio); | 646 | r = PTR_ERR(gpio); |
| 647 | if (r == -ENOENT) | 647 | if (r == -ENOENT || r == -ENOSYS) |
| 648 | return 0; | 648 | return 0; |
| 649 | return r; | 649 | return r; |
| 650 | } | 650 | } |
| @@ -58,6 +58,7 @@ · 57 unmodified lines | |||
| 58 | * Wildcat Point (PCH) 0x8ca2 32 hard yes yes yes | 58 | * Wildcat Point (PCH) 0x8ca2 32 hard yes yes yes |
| 59 | * Wildcat Point-LP (PCH) 0x9ca2 32 hard yes yes yes | 59 | * Wildcat Point-LP (PCH) 0x9ca2 32 hard yes yes yes |
| 60 | * BayTrail (SOC) 0x0f12 32 hard yes yes yes | 60 | * BayTrail (SOC) 0x0f12 32 hard yes yes yes |
| 61 | * Braswell (SOC) 0x2292 32 hard yes yes yes | ||
| 61 | * Sunrise Point-H (PCH) 0xa123 32 hard yes yes yes | 62 | * Sunrise Point-H (PCH) 0xa123 32 hard yes yes yes |
| 62 | * Sunrise Point-LP (PCH) 0x9d23 32 hard yes yes yes | 63 | * Sunrise Point-LP (PCH) 0x9d23 32 hard yes yes yes |
| 63 | * DNV (SOC) 0x19df 32 hard yes yes yes | 64 | * DNV (SOC) 0x19df 32 hard yes yes yes |
| @@ -233,6 +233,7 @@ static int octeon_i2c_check_status(struct octeon_i2c *i2c, int final_read) · 232 unmodified lines | |||
| 233 | return -EOPNOTSUPP; | 233 | return -EOPNOTSUPP; |
| 234 | 234 | ||
| 235 | case STAT_TXDATA_NAK: | 235 | case STAT_TXDATA_NAK: |
| 236 | case STAT_BUS_ERROR: | ||
| 236 | return -EIO; | 237 | return -EIO; |
| 237 | case STAT_TXADDR_NAK: | 238 | case STAT_TXADDR_NAK: |
| 238 | case STAT_RXADDR_NAK: | 239 | case STAT_RXADDR_NAK: |
| @@ -43,7 +43,7 @@ · 42 unmodified lines | |||
| 43 | #define TWSI_CTL_AAK 0x04 /* Assert ACK */ | 43 | #define TWSI_CTL_AAK 0x04 /* Assert ACK */ |
| 44 | 44 | ||
| 45 | /* Status values */ | 45 | /* Status values */ |
| 46 | #define STAT_ERROR 0x00 | 46 | #define STAT_BUS_ERROR 0x00 |
| 47 | #define STAT_START 0x08 | 47 | #define STAT_START 0x08 |
| 48 | #define STAT_REP_START 0x10 | 48 | #define STAT_REP_START 0x10 |
| 49 | #define STAT_TXADDR_ACK 0x18 | 49 | #define STAT_TXADDR_ACK 0x18 |
| @@ -341,7 +341,7 @@ static int i2c_sirfsoc_probe(struct platform_device *pdev) · 340 unmodified lines | |||
| 341 | platform_set_drvdata(pdev, adap); | 341 | platform_set_drvdata(pdev, adap); |
| 342 | init_completion(&siic->done); | 342 | init_completion(&siic->done); |
| 343 | 343 | ||
| 344 | /* Controller Initalisation */ | 344 | /* Controller initialisation */ |
| 345 | 345 | ||
| 346 | writel(SIRFSOC_I2C_RESET, siic->base + SIRFSOC_I2C_CTRL); | 346 | writel(SIRFSOC_I2C_RESET, siic->base + SIRFSOC_I2C_CTRL); |
| 347 | while (readl(siic->base + SIRFSOC_I2C_CTRL) & SIRFSOC_I2C_RESET) | 347 | while (readl(siic->base + SIRFSOC_I2C_CTRL) & SIRFSOC_I2C_RESET) |
| @@ -369,7 +369,7 @@ static int i2c_sirfsoc_probe(struct platform_device *pdev) · 21 unmodified lines | |||
| 369 | * but they start to affect the speed when clock is set to faster | 369 | * but they start to affect the speed when clock is set to faster |
| 370 | * frequencies. | 370 | * frequencies. |
| 371 | * Through the actual tests, use the different user_div value(which | 371 | * Through the actual tests, use the different user_div value(which |
| 372 | * in the divider formular 'Fio / (Fi2c * user_div)') to adapt | 372 | * in the divider formula 'Fio / (Fi2c * user_div)') to adapt |
| 373 | * the different ranges of i2c bus clock frequency, to make the SCL | 373 | * the different ranges of i2c bus clock frequency, to make the SCL |
| 374 | * more accurate. | 374 | * more accurate. |
| 375 | */ | 375 | */ |
| @@ -928,7 +928,7 @@ static int exact_lock(dev_t dev, void *data) · 927 unmodified lines | |||
| 928 | { | 928 | { |
| 929 | struct gendisk *p = data; | 929 | struct gendisk *p = data; |
| 930 | 930 | ||
| 931 | if (!get_disk(p)) | 931 | if (!get_disk_and_module(p)) |
| 932 | return -1; | 932 | return -1; |
| 933 | return 0; | 933 | return 0; |
| 934 | } | 934 | } |
| @@ -243,7 +243,7 @@ static int aspeed_adc_probe(struct platform_device *pdev) · 242 unmodified lines | |||
| 243 | ASPEED_ADC_INIT_POLLING_TIME, | 243 | ASPEED_ADC_INIT_POLLING_TIME, |
| 244 | ASPEED_ADC_INIT_TIMEOUT); | 244 | ASPEED_ADC_INIT_TIMEOUT); |
| 245 | if (ret) | 245 | if (ret) |
| 246 | goto scaler_error; | 246 | goto poll_timeout_error; |
| 247 | } | 247 | } |
| 248 | 248 | ||
| 249 | /* Start all channels in normal mode. */ | 249 | /* Start all channels in normal mode. */ |
| @@ -274,9 +274,10 @@ iio_register_error: · 24 unmodified lines | |||
| 274 | writel(ASPEED_OPERATION_MODE_POWER_DOWN, | 274 | writel(ASPEED_OPERATION_MODE_POWER_DOWN, |
| 275 | data->base + ASPEED_REG_ENGINE_CONTROL); | 275 | data->base + ASPEED_REG_ENGINE_CONTROL); |
| 276 | clk_disable_unprepare(data->clk_scaler->clk); | 276 | clk_disable_unprepare(data->clk_scaler->clk); |
| 277 | reset_error: | ||
| 278 | reset_control_assert(data->rst); | ||
| 279 | clk_enable_error: | 277 | clk_enable_error: |
| 278 | poll_timeout_error: | ||
| 279 | reset_control_assert(data->rst); | ||
| 280 | reset_error: | ||
| 280 | clk_hw_unregister_divider(data->clk_scaler); | 281 | clk_hw_unregister_divider(data->clk_scaler); |
| 281 | scaler_error: | 282 | scaler_error: |
| 282 | clk_hw_unregister_divider(data->clk_prescaler); | 283 | clk_hw_unregister_divider(data->clk_prescaler); |
| @@ -722,8 +722,6 @@ static int stm32h7_adc_enable(struct stm32_adc *adc) · 721 unmodified lines | |||
| 722 | int ret; | 722 | int ret; |
| 723 | u32 val; | 723 | u32 val; |
| 724 | 724 | ||
| 725 | /* Clear ADRDY by writing one, then enable ADC */ | ||
| 726 | stm32_adc_set_bits(adc, STM32H7_ADC_ISR, STM32H7_ADRDY); | ||
| 727 | stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADEN); | 725 | stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADEN); |
| 728 | 726 | ||
| 729 | /* Poll for ADRDY to be set (after adc startup time) */ | 727 | /* Poll for ADRDY to be set (after adc startup time) */ |
| @@ -731,8 +729,11 @@ static int stm32h7_adc_enable(struct stm32_adc *adc) · 1 unmodified line | |||
| 731 | val & STM32H7_ADRDY, | 729 | val & STM32H7_ADRDY, |
| 732 | 100, STM32_ADC_TIMEOUT_US); | 730 | 100, STM32_ADC_TIMEOUT_US); |
| 733 | if (ret) { | 731 | if (ret) { |
| 734 | stm32_adc_clr_bits(adc, STM32H7_ADC_CR, STM32H7_ADEN); | 732 | stm32_adc_set_bits(adc, STM32H7_ADC_CR, STM32H7_ADDIS); |
| 735 | dev_err(&indio_dev->dev, "Failed to enable ADC\n"); | 733 | dev_err(&indio_dev->dev, "Failed to enable ADC\n"); |
| 734 | } else { | ||
| 735 | /* Clear ADRDY by writing one */ | ||
| 736 | stm32_adc_set_bits(adc, STM32H7_ADC_ISR, STM32H7_ADRDY); | ||
| 736 | } | 737 | } |
| 737 | 738 | ||
| 738 | return ret; | 739 | return ret; |
| @@ -46,6 +46,10 @@ int adis_probe_trigger(struct adis *adis, struct iio_dev *indio_dev) · 45 unmodified lines | |||
| 46 | if (adis->trig == NULL) | 46 | if (adis->trig == NULL) |
| 47 | return -ENOMEM; | 47 | return -ENOMEM; |
| 48 | 48 | ||
| 49 | adis->trig->dev.parent = &adis->spi->dev; | ||
| 50 | adis->trig->ops = &adis_trigger_ops; | ||
| 51 | iio_trigger_set_drvdata(adis->trig, adis); | ||
| 52 | |||
| 49 | ret = request_irq(adis->spi->irq, | 53 | ret = request_irq(adis->spi->irq, |
| 50 | &iio_trigger_generic_data_rdy_poll, | 54 | &iio_trigger_generic_data_rdy_poll, |
| 51 | IRQF_TRIGGER_RISING, | 55 | IRQF_TRIGGER_RISING, |
| @@ -54,9 +58,6 @@ int adis_probe_trigger(struct adis *adis, struct iio_dev *indio_dev) · 2 unmodified lines | |||
| 54 | if (ret) | 58 | if (ret) |
| 55 | goto error_free_trig; | 59 | goto error_free_trig; |
| 56 | 60 | ||
| 57 | adis->trig->dev.parent = &adis->spi->dev; | ||
| 58 | adis->trig->ops = &adis_trigger_ops; | ||
| 59 | iio_trigger_set_drvdata(adis->trig, adis); | ||
| 60 | ret = iio_trigger_register(adis->trig); | 61 | ret = iio_trigger_register(adis->trig); |
| 61 | 62 | ||
| 62 | indio_dev->trig = iio_trigger_get(adis->trig); | 63 | indio_dev->trig = iio_trigger_get(adis->trig); |
| @@ -175,7 +175,7 @@ __poll_t iio_buffer_poll(struct file *filp, · 174 unmodified lines | |||
| 175 | struct iio_dev *indio_dev = filp->private_data; | 175 | struct iio_dev *indio_dev = filp->private_data; |
| 176 | struct iio_buffer *rb = indio_dev->buffer; | 176 | struct iio_buffer *rb = indio_dev->buffer; |
| 177 | 177 | ||
| 178 | if (!indio_dev->info) | 178 | if (!indio_dev->info || rb == NULL) |
| 179 | return 0; | 179 | return 0; |
| 180 | 180 | ||
| 181 | poll_wait(filp, &rb->pollq, wait); | 181 | poll_wait(filp, &rb->pollq, wait); |
| @@ -68,6 +68,8 @@ config SX9500 · 67 unmodified lines | |||
| 68 | 68 | ||
| 69 | config SRF08 | 69 | config SRF08 |
| 70 | tristate "Devantech SRF02/SRF08/SRF10 ultrasonic ranger sensor" | 70 | tristate "Devantech SRF02/SRF08/SRF10 ultrasonic ranger sensor" |
| 71 | select IIO_BUFFER | ||
| 72 | select IIO_TRIGGERED_BUFFER | ||
| 71 | depends on I2C | 73 | depends on I2C |
| 72 | help | 74 | help |
| 73 | Say Y here to build a driver for Devantech SRF02/SRF08/SRF10 | 75 | Say Y here to build a driver for Devantech SRF02/SRF08/SRF10 |
| @@ -550,18 +550,13 @@ static int addr_resolve(struct sockaddr *src_in, · 549 unmodified lines | |||
| 550 | dst_release(dst); | 550 | dst_release(dst); |
| 551 | } | 551 | } |
| 552 | 552 | ||
| 553 | if (ndev->flags & IFF_LOOPBACK) { | 553 | if (ndev) { |
| 554 | ret = rdma_translate_ip(dst_in, addr); | 554 | if (ndev->flags & IFF_LOOPBACK) |
| 555 | /* | 555 | ret = rdma_translate_ip(dst_in, addr); |
| 556 | * Put the loopback device and get the translated | 556 | else |
| 557 | * device instead. | 557 | addr->bound_dev_if = ndev->ifindex; |
| 558 | */ | ||
| 559 | dev_put(ndev); | 558 | dev_put(ndev); |
| 560 | ndev = dev_get_by_index(addr->net, addr->bound_dev_if); | ||
| 561 | } else { | ||
| 562 | addr->bound_dev_if = ndev->ifindex; | ||
| 563 | } | 559 | } |
| 564 | dev_put(ndev); | ||
| 565 | 560 | ||
| 566 | return ret; | 561 | return ret; |
| 567 | } | 562 | } |
| @@ -305,16 +305,21 @@ void nldev_exit(void); · 304 unmodified lines | |||
| 305 | static inline struct ib_qp *_ib_create_qp(struct ib_device *dev, | 305 | static inline struct ib_qp *_ib_create_qp(struct ib_device *dev, |
| 306 | struct ib_pd *pd, | 306 | struct ib_pd *pd, |
| 307 | struct ib_qp_init_attr *attr, | 307 | struct ib_qp_init_attr *attr, |
| 308 | struct ib_udata *udata) | 308 | struct ib_udata *udata, |
| 309 | struct ib_uobject *uobj) | ||
| 309 | { | 310 | { |
| 310 | struct ib_qp *qp; | 311 | struct ib_qp *qp; |
| 311 | 312 | ||
| 313 | if (!dev->create_qp) | ||
| 314 | return ERR_PTR(-EOPNOTSUPP); | ||
| 315 | |||
| 312 | qp = dev->create_qp(pd, attr, udata); | 316 | qp = dev->create_qp(pd, attr, udata); |
| 313 | if (IS_ERR(qp)) | 317 | if (IS_ERR(qp)) |
| 314 | return qp; | 318 | return qp; |
| 315 | 319 | ||
| 316 | qp->device = dev; | 320 | qp->device = dev; |
| 317 | qp->pd = pd; | 321 | qp->pd = pd; |
| 322 | qp->uobject = uobj; | ||
| 318 | /* | 323 | /* |
| 319 | * We don't track XRC QPs for now, because they don't have PD | 324 | * We don't track XRC QPs for now, because they don't have PD |
| 320 | * and more importantly they are created internaly by driver, | 325 | * and more importantly they are created internaly by driver, |
| @@ -17,6 +17,7 @@ · 16 unmodified lines | |||
| 17 | 17 | ||
| 18 | /* # of WCs to poll for with a single call to ib_poll_cq */ | 18 | /* # of WCs to poll for with a single call to ib_poll_cq */ |
| 19 | #define IB_POLL_BATCH 16 | 19 | #define IB_POLL_BATCH 16 |
| 20 | #define IB_POLL_BATCH_DIRECT 8 | ||
| 20 | 21 | ||
| 21 | /* # of WCs to iterate over before yielding */ | 22 | /* # of WCs to iterate over before yielding */ |
| 22 | #define IB_POLL_BUDGET_IRQ 256 | 23 | #define IB_POLL_BUDGET_IRQ 256 |
| @@ -25,18 +26,18 @@ · 2 unmodified lines | |||
| 25 | #define IB_POLL_FLAGS \ | 26 | #define IB_POLL_FLAGS \ |
| 26 | (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) | 27 | (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) |
| 27 | 28 | ||
| 28 | static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc) | 29 | static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *wcs, |
| 30 | int batch) | ||
| 29 | { | 31 | { |
| 30 | int i, n, completed = 0; | 32 | int i, n, completed = 0; |
| 31 | struct ib_wc *wcs = poll_wc ? : cq->wc; | ||
| 32 | 33 | ||
| 33 | /* | 34 | /* |
| 34 | * budget might be (-1) if the caller does not | 35 | * budget might be (-1) if the caller does not |
| 35 | * want to bound this call, thus we need unsigned | 36 | * want to bound this call, thus we need unsigned |
| 36 | * minimum here. | 37 | * minimum here. |
| 37 | */ | 38 | */ |
| 38 | while ((n = ib_poll_cq(cq, min_t(u32, IB_POLL_BATCH, | 39 | while ((n = ib_poll_cq(cq, min_t(u32, batch, |
| 39 | budget - completed), wcs)) > 0) { | 40 | budget - completed), wcs)) > 0) { |
| 40 | for (i = 0; i < n; i++) { | 41 | for (i = 0; i < n; i++) { |
| 41 | struct ib_wc *wc = &wcs[i]; | 42 | struct ib_wc *wc = &wcs[i]; |
| 42 | 43 | ||
| @@ -48,8 +49,7 @@ static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc) · 5 unmodified lines | |||
| 48 | 49 | ||
| 49 | completed += n; | 50 | completed += n; |
| 50 | 51 | ||
| 51 | if (n != IB_POLL_BATCH || | 52 | if (n != batch || (budget != -1 && completed >= budget)) |
| 52 | (budget != -1 && completed >= budget)) | ||
| 53 | break; | 53 | break; |
| 54 | } | 54 | } |
| 55 | 55 | ||
| @@ -72,9 +72,9 @@ static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc) · 16 unmodified lines | |||
| 72 | */ | 72 | */ |
| 73 | int ib_process_cq_direct(struct ib_cq *cq, int budget) | 73 | int ib_process_cq_direct(struct ib_cq *cq, int budget) |
| 74 | { | 74 | { |
| 75 | struct ib_wc wcs[IB_POLL_BATCH]; | 75 | struct ib_wc wcs[IB_POLL_BATCH_DIRECT]; |
| 76 | 76 | ||
| 77 | return __ib_process_cq(cq, budget, wcs); | 77 | return __ib_process_cq(cq, budget, wcs, IB_POLL_BATCH_DIRECT); |
| 78 | } | 78 | } |
| 79 | EXPORT_SYMBOL(ib_process_cq_direct); | 79 | EXPORT_SYMBOL(ib_process_cq_direct); |
| 80 | 80 | ||
| @@ -88,7 +88,7 @@ static int ib_poll_handler(struct irq_poll *iop, int budget) · 7 unmodified lines | |||
| 88 | struct ib_cq *cq = container_of(iop, struct ib_cq, iop); | 88 | struct ib_cq *cq = container_of(iop, struct ib_cq, iop); |
| 89 | int completed; | 89 | int completed; |
| 90 | 90 | ||
| 91 | completed = __ib_process_cq(cq, budget, NULL); | 91 | completed = __ib_process_cq(cq, budget, cq->wc, IB_POLL_BATCH); |
| 92 | if (completed < budget) { | 92 | if (completed < budget) { |
| 93 | irq_poll_complete(&cq->iop); | 93 | irq_poll_complete(&cq->iop); |
| 94 | if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) | 94 | if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) |
| @@ -108,7 +108,8 @@ static void ib_cq_poll_work(struct work_struct *work) · 13 unmodified lines | |||
| 108 | struct ib_cq *cq = container_of(work, struct ib_cq, work); | 108 | struct ib_cq *cq = container_of(work, struct ib_cq, work); |
| 109 | int completed; | 109 | int completed; |
| 110 | 110 | ||
| 111 | completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, NULL); | 111 | completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, cq->wc, |
| 112 | IB_POLL_BATCH); | ||
| 112 | if (completed >= IB_POLL_BUDGET_WORKQUEUE || | 113 | if (completed >= IB_POLL_BUDGET_WORKQUEUE || |
| 113 | ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) | 114 | ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) |
| 114 | queue_work(ib_comp_wq, &cq->work); | 115 | queue_work(ib_comp_wq, &cq->work); |
| @@ -536,14 +536,14 @@ int ib_register_device(struct ib_device *device, · 535 unmodified lines | |||
| 536 | ret = device->query_device(device, &device->attrs, &uhw); | 536 | ret = device->query_device(device, &device->attrs, &uhw); |
| 537 | if (ret) { | 537 | if (ret) { |
| 538 | pr_warn("Couldn't query the device attributes\n"); | 538 | pr_warn("Couldn't query the device attributes\n"); |
| 539 | goto cache_cleanup; | 539 | goto cg_cleanup; |
| 540 | } | 540 | } |
| 541 | 541 | ||
| 542 | ret = ib_device_register_sysfs(device, port_callback); | 542 | ret = ib_device_register_sysfs(device, port_callback); |
| 543 | if (ret) { | 543 | if (ret) { |
| 544 | pr_warn("Couldn't register device %s with driver model\n", | 544 | pr_warn("Couldn't register device %s with driver model\n", |
| 545 | device->name); | 545 | device->name); |
| 546 | goto cache_cleanup; | 546 | goto cg_cleanup; |
| 547 | } | 547 | } |
| 548 | 548 | ||
| 549 | device->reg_state = IB_DEV_REGISTERED; | 549 | device->reg_state = IB_DEV_REGISTERED; |
| @@ -559,6 +559,8 @@ int ib_register_device(struct ib_device *device, · 9 unmodified lines | |||
| 559 | mutex_unlock(&device_mutex); | 559 | mutex_unlock(&device_mutex); |
| 560 | return 0; | 560 | return 0; |
| 561 | 561 | ||
| 562 | cg_cleanup: | ||
| 563 | ib_device_unregister_rdmacg(device); | ||
| 562 | cache_cleanup: | 564 | cache_cleanup: |
| 563 | ib_cache_cleanup_one(device); | 565 | ib_cache_cleanup_one(device); |
| 564 | ib_cache_release_one(device); | 566 | ib_cache_release_one(device); |
| @@ -141,7 +141,12 @@ static struct ib_uobject *alloc_uobj(struct ib_ucontext *context, · 140 unmodified lines | |||
| 141 | */ | 141 | */ |
| 142 | uobj->context = context; | 142 | uobj->context = context; |
| 143 | uobj->type = type; | 143 | uobj->type = type; |
| 144 | atomic_set(&uobj->usecnt, 0); | 144 | /* |
| 145 | * Allocated objects start out as write locked to deny any other | ||
| 146 | * syscalls from accessing them until they are committed. See | ||
| 147 | * rdma_alloc_commit_uobject | ||
| 148 | */ | ||
| 149 | atomic_set(&uobj->usecnt, -1); | ||
| 145 | kref_init(&uobj->ref); | 150 | kref_init(&uobj->ref); |
| 146 | 151 | ||
| 147 | return uobj; | 152 | return uobj; |
| @@ -196,7 +201,15 @@ static struct ib_uobject *lookup_get_idr_uobject(const struct uverbs_obj_type *t · 48 unmodified lines | |||
| 196 | goto free; | 201 | goto free; |
| 197 | } | 202 | } |
| 198 | 203 | ||
| 199 | uverbs_uobject_get(uobj); | 204 | /* |
| 205 | * The idr_find is guaranteed to return a pointer to something that | ||
| 206 | * isn't freed yet, or NULL, as the free after idr_remove goes through | ||
| 207 | * kfree_rcu(). However the object may still have been released and | ||
| 208 | * kfree() could be called at any time. | ||
| 209 | */ | ||
| 210 | if (!kref_get_unless_zero(&uobj->ref)) | ||
| 211 | uobj = ERR_PTR(-ENOENT); | ||
| 212 | |||
| 200 | free: | 213 | free: |
| 201 | rcu_read_unlock(); | 214 | rcu_read_unlock(); |
| 202 | return uobj; | 215 | return uobj; |
| @@ -399,13 +412,13 @@ static int __must_check remove_commit_fd_uobject(struct ib_uobject *uobj, · 196 unmodified lines | |||
| 399 | return ret; | 412 | return ret; |
| 400 | } | 413 | } |
| 401 | 414 | ||
| 402 | static void lockdep_check(struct ib_uobject *uobj, bool exclusive) | 415 | static void assert_uverbs_usecnt(struct ib_uobject *uobj, bool exclusive) |
| 403 | { | 416 | { |
| 404 | #ifdef CONFIG_LOCKDEP | 417 | #ifdef CONFIG_LOCKDEP |
| 405 | if (exclusive) | 418 | if (exclusive) |
| 406 | WARN_ON(atomic_read(&uobj->usecnt) > 0); | 419 | WARN_ON(atomic_read(&uobj->usecnt) != -1); |
| 407 | else | 420 | else |
| 408 | WARN_ON(atomic_read(&uobj->usecnt) == -1); | 421 | WARN_ON(atomic_read(&uobj->usecnt) <= 0); |
| 409 | #endif | 422 | #endif |
| 410 | } | 423 | } |
| 411 | 424 | ||
| @@ -444,7 +457,7 @@ int __must_check rdma_remove_commit_uobject(struct ib_uobject *uobj) · 32 unmodified lines | |||
| 444 | WARN(true, "ib_uverbs: Cleanup is running while removing an uobject\n"); | 457 | WARN(true, "ib_uverbs: Cleanup is running while removing an uobject\n"); |
| 445 | return 0; | 458 | return 0; |
| 446 | } | 459 | } |
| 447 | lockdep_check(uobj, true); | 460 | assert_uverbs_usecnt(uobj, true); |
| 448 | ret = _rdma_remove_commit_uobject(uobj, RDMA_REMOVE_DESTROY); | 461 | ret = _rdma_remove_commit_uobject(uobj, RDMA_REMOVE_DESTROY); |
| 449 | 462 | ||
| 450 | up_read(&ucontext->cleanup_rwsem); | 463 | up_read(&ucontext->cleanup_rwsem); |
| @@ -474,16 +487,17 @@ int rdma_explicit_destroy(struct ib_uobject *uobject) · 23 unmodified lines | |||
| 474 | WARN(true, "ib_uverbs: Cleanup is running while removing an uobject\n"); | 487 | WARN(true, "ib_uverbs: Cleanup is running while removing an uobject\n"); |
| 475 | return 0; | 488 | return 0; |
| 476 | } | 489 | } |
| 477 | lockdep_check(uobject, true); | 490 | assert_uverbs_usecnt(uobject, true); |
| 478 | ret = uobject->type->type_class->remove_commit(uobject, | 491 | ret = uobject->type->type_class->remove_commit(uobject, |
| 479 | RDMA_REMOVE_DESTROY); | 492 | RDMA_REMOVE_DESTROY); |
| 480 | if (ret) | 493 | if (ret) |
| 481 | return ret; | 494 | goto out; |
| 482 | 495 | ||
| 483 | uobject->type = &null_obj_type; | 496 | uobject->type = &null_obj_type; |
| 484 | 497 | ||
| 498 | out: | ||
| 485 | up_read(&ucontext->cleanup_rwsem); | 499 | up_read(&ucontext->cleanup_rwsem); |
| 486 | return 0; | 500 | return ret; |
| 487 | } | 501 | } |
| 488 | 502 | ||
| 489 | static void alloc_commit_idr_uobject(struct ib_uobject *uobj) | 503 | static void alloc_commit_idr_uobject(struct ib_uobject *uobj) |
| @@ -527,6 +541,10 @@ int rdma_alloc_commit_uobject(struct ib_uobject *uobj) · 37 unmodified lines | |||
| 527 | return ret; | 541 | return ret; |
| 528 | } | 542 | } |
| 529 | 543 | ||
| 544 | /* matches atomic_set(-1) in alloc_uobj */ | ||
| 545 | assert_uverbs_usecnt(uobj, true); | ||
| 546 | atomic_set(&uobj->usecnt, 0); | ||
| 547 | |||
| 530 | uobj->type->type_class->alloc_commit(uobj); | 548 | uobj->type->type_class->alloc_commit(uobj); |
| 531 | up_read(&uobj->context->cleanup_rwsem); | 549 | up_read(&uobj->context->cleanup_rwsem); |
| 532 | 550 | ||
| @@ -561,7 +579,7 @@ static void lookup_put_fd_uobject(struct ib_uobject *uobj, bool exclusive) · 28 unmodified lines | |||
| 561 | 579 | ||
| 562 | void rdma_lookup_put_uobject(struct ib_uobject *uobj, bool exclusive) | 580 | void rdma_lookup_put_uobject(struct ib_uobject *uobj, bool exclusive) |
| 563 | { | 581 | { |
| 564 | lockdep_check(uobj, exclusive); | 582 | assert_uverbs_usecnt(uobj, exclusive); |
| 565 | uobj->type->type_class->lookup_put(uobj, exclusive); | 583 | uobj->type->type_class->lookup_put(uobj, exclusive); |
| 566 | /* | 584 | /* |
| 567 | * In order to unlock an object, either decrease its usecnt for | 585 | * In order to unlock an object, either decrease its usecnt for |
| @@ -7,7 +7,6 @@ · 6 unmodified lines | |||
| 7 | #include <rdma/restrack.h> | 7 | #include <rdma/restrack.h> |
| 8 | #include <linux/mutex.h> | 8 | #include <linux/mutex.h> |
| 9 | #include <linux/sched/task.h> | 9 | #include <linux/sched/task.h> |
| 10 | #include <linux/uaccess.h> | ||
| 11 | #include <linux/pid_namespace.h> | 10 | #include <linux/pid_namespace.h> |
| 12 | 11 | ||
| 13 | void rdma_restrack_init(struct rdma_restrack_root *res) | 12 | void rdma_restrack_init(struct rdma_restrack_root *res) |
| @@ -63,7 +62,6 @@ static struct ib_device *res_to_dev(struct rdma_restrack_entry *res) · 49 unmodified lines | |||
| 63 | { | 62 | { |
| 64 | enum rdma_restrack_type type = res->type; | 63 | enum rdma_restrack_type type = res->type; |
| 65 | struct ib_device *dev; | 64 | struct ib_device *dev; |
| 66 | struct ib_xrcd *xrcd; | ||
| 67 | struct ib_pd *pd; | 65 | struct ib_pd *pd; |
| 68 | struct ib_cq *cq; | 66 | struct ib_cq *cq; |
| 69 | struct ib_qp *qp; | 67 | struct ib_qp *qp; |
| @@ -81,10 +79,6 @@ static struct ib_device *res_to_dev(struct rdma_restrack_entry *res) · 11 unmodified lines | |||
| 81 | qp = container_of(res, struct ib_qp, res); | 79 | qp = container_of(res, struct ib_qp, res); |
| 82 | dev = qp->device; | 80 | dev = qp->device; |
| 83 | break; | 81 | break; |
| 84 | case RDMA_RESTRACK_XRCD: | ||
| 85 | xrcd = container_of(res, struct ib_xrcd, res); | ||
| 86 | dev = xrcd->device; | ||
| 87 | break; | ||
| 88 | default: | 82 | default: |
| 89 | WARN_ONCE(true, "Wrong resource tracking type %u\n", type); | 83 | WARN_ONCE(true, "Wrong resource tracking type %u\n", type); |
| 90 | return NULL; | 84 | return NULL; |
| @@ -93,6 +87,21 @@ static struct ib_device *res_to_dev(struct rdma_restrack_entry *res) · 2 unmodified lines | |||
| 93 | return dev; | 87 | return dev; |
| 94 | } | 88 | } |
| 95 | 89 | ||
| 90 | static bool res_is_user(struct rdma_restrack_entry *res) | ||
| 91 | { | ||
| 92 | switch (res->type) { | ||
| 93 | case RDMA_RESTRACK_PD: | ||
| 94 | return container_of(res, struct ib_pd, res)->uobject; | ||
| 95 | case RDMA_RESTRACK_CQ: | ||
| 96 | return container_of(res, struct ib_cq, res)->uobject; | ||
| 97 | case RDMA_RESTRACK_QP: | ||
| 98 | return container_of(res, struct ib_qp, res)->uobject; | ||
| 99 | default: | ||
| 100 | WARN_ONCE(true, "Wrong resource tracking type %u\n", res->type); | ||
| 101 | return false; | ||
| 102 | } | ||
| 103 | } | ||
| 104 | |||
| 96 | void rdma_restrack_add(struct rdma_restrack_entry *res) | 105 | void rdma_restrack_add(struct rdma_restrack_entry *res) |
| 97 | { | 106 | { |
| 98 | struct ib_device *dev = res_to_dev(res); | 107 | struct ib_device *dev = res_to_dev(res); |
| @@ -100,7 +109,7 @@ void rdma_restrack_add(struct rdma_restrack_entry *res) · 1 unmodified line | |||
| 100 | if (!dev) | 109 | if (!dev) |
| 101 | return; | 110 | return; |
| 102 | 111 | ||
| 103 | if (!uaccess_kernel()) { | 112 | if (res_is_user(res)) { |
| 104 | get_task_struct(current); | 113 | get_task_struct(current); |
| 105 | res->task = current; | 114 | res->task = current; |
| 106 | res->kern_name = NULL; | 115 | res->kern_name = NULL; |
| @@ -1291,10 +1291,9 @@ int ib_init_ah_attr_from_path(struct ib_device *device, u8 port_num, · 1290 unmodified lines | |||
| 1291 | 1291 | ||
| 1292 | resolved_dev = dev_get_by_index(dev_addr.net, | 1292 | resolved_dev = dev_get_by_index(dev_addr.net, |
| 1293 | dev_addr.bound_dev_if); | 1293 | dev_addr.bound_dev_if); |
| 1294 | if (resolved_dev->flags & IFF_LOOPBACK) { | 1294 | if (!resolved_dev) { |
| 1295 | dev_put(resolved_dev); | 1295 | dev_put(idev); |
| 1296 | resolved_dev = idev; | 1296 | return -ENODEV; |
| 1297 | dev_hold(resolved_dev); | ||
| 1298 | } | 1297 | } |
| 1299 | ndev = ib_get_ndev_from_path(rec); | 1298 | ndev = ib_get_ndev_from_path(rec); |
| 1300 | rcu_read_lock(); | 1299 | rcu_read_lock(); |
| @@ -1149,6 +1149,9 @@ static ssize_t ucma_init_qp_attr(struct ucma_file *file, · 1148 unmodified lines | |||
| 1149 | if (copy_from_user(&cmd, inbuf, sizeof(cmd))) | 1149 | if (copy_from_user(&cmd, inbuf, sizeof(cmd))) |
| 1150 | return -EFAULT; | 1150 | return -EFAULT; |
| 1151 | 1151 | ||
| 1152 | if (cmd.qp_state > IB_QPS_ERR) | ||
| 1153 | return -EINVAL; | ||
| 1154 | |||
| 1152 | ctx = ucma_get_ctx(file, cmd.id); | 1155 | ctx = ucma_get_ctx(file, cmd.id); |
| 1153 | if (IS_ERR(ctx)) | 1156 | if (IS_ERR(ctx)) |
| 1154 | return PTR_ERR(ctx); | 1157 | return PTR_ERR(ctx); |
| @@ -1294,6 +1297,9 @@ static ssize_t ucma_set_option(struct ucma_file *file, const char __user *inbuf, · 139 unmodified lines | |||
| 1294 | if (IS_ERR(ctx)) | 1297 | if (IS_ERR(ctx)) |
| 1295 | return PTR_ERR(ctx); | 1298 | return PTR_ERR(ctx); |
| 1296 | 1299 | ||
| 1300 | if (unlikely(cmd.optval > KMALLOC_MAX_SIZE)) | ||
| 1301 | return -EINVAL; | ||
| 1302 | |||
| 1297 | optval = memdup_user((void __user *) (unsigned long) cmd.optval, | 1303 | optval = memdup_user((void __user *) (unsigned long) cmd.optval, |
| 1298 | cmd.optlen); | 1304 | cmd.optlen); |
| 1299 | if (IS_ERR(optval)) { | 1305 | if (IS_ERR(optval)) { |
| @@ -562,9 +562,10 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file, · 561 unmodified lines | |||
| 562 | if (f.file) | 562 | if (f.file) |
| 563 | fdput(f); | 563 | fdput(f); |
| 564 | 564 | ||
| 565 | mutex_unlock(&file->device->xrcd_tree_mutex); | ||
| 566 | |||
| 565 | uobj_alloc_commit(&obj->uobject); | 567 | uobj_alloc_commit(&obj->uobject); |
| 566 | 568 | ||
| 567 | mutex_unlock(&file->device->xrcd_tree_mutex); | ||
| 568 | return in_len; | 569 | return in_len; |
| 569 | 570 | ||
| 570 | err_copy: | 571 | err_copy: |
| @@ -603,10 +604,8 @@ ssize_t ib_uverbs_close_xrcd(struct ib_uverbs_file *file, · 32 unmodified lines | |||
| 603 | 604 | ||
| 604 | uobj = uobj_get_write(uobj_get_type(xrcd), cmd.xrcd_handle, | 605 | uobj = uobj_get_write(uobj_get_type(xrcd), cmd.xrcd_handle, |
| 605 | file->ucontext); | 606 | file->ucontext); |
| 606 | if (IS_ERR(uobj)) { | 607 | if (IS_ERR(uobj)) |
| 607 | mutex_unlock(&file->device->xrcd_tree_mutex); | ||
| 608 | return PTR_ERR(uobj); | 608 | return PTR_ERR(uobj); |
| 609 | } | ||
| 610 | 609 | ||
| 611 | ret = uobj_remove_commit(uobj); | 610 | ret = uobj_remove_commit(uobj); |
| 612 | return ret ?: in_len; | 611 | return ret ?: in_len; |
| @@ -979,6 +978,9 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file, · 366 unmodified lines | |||
| 979 | struct ib_uverbs_ex_create_cq_resp resp; | 978 | struct ib_uverbs_ex_create_cq_resp resp; |
| 980 | struct ib_cq_init_attr attr = {}; | 979 | struct ib_cq_init_attr attr = {}; |
| 981 | 980 | ||
| 981 | if (!ib_dev->create_cq) | ||
| 982 | return ERR_PTR(-EOPNOTSUPP); | ||
| 983 | |||
| 982 | if (cmd->comp_vector >= file->device->num_comp_vectors) | 984 | if (cmd->comp_vector >= file->device->num_comp_vectors) |
| 983 | return ERR_PTR(-EINVAL); | 985 | return ERR_PTR(-EINVAL); |
| 984 | 986 | ||
| @@ -1030,14 +1032,14 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file, · 45 unmodified lines | |||
| 1030 | resp.response_length = offsetof(typeof(resp), response_length) + | 1032 | resp.response_length = offsetof(typeof(resp), response_length) + |
| 1031 | sizeof(resp.response_length); | 1033 | sizeof(resp.response_length); |
| 1032 | 1034 | ||
| 1035 | cq->res.type = RDMA_RESTRACK_CQ; | ||
| 1036 | rdma_restrack_add(&cq->res); | ||
| 1037 | |||
| 1033 | ret = cb(file, obj, &resp, ucore, context); | 1038 | ret = cb(file, obj, &resp, ucore, context); |
| 1034 | if (ret) | 1039 | if (ret) |
| 1035 | goto err_cb; | 1040 | goto err_cb; |
| 1036 | 1041 | ||
| 1037 | uobj_alloc_commit(&obj->uobject); | 1042 | uobj_alloc_commit(&obj->uobject); |
| 1038 | cq->res.type = RDMA_RESTRACK_CQ; | ||
| 1039 | rdma_restrack_add(&cq->res); | ||
| 1040 | |||
| 1041 | return obj; | 1043 | return obj; |
| 1042 | 1044 | ||
| 1043 | err_cb: | 1045 | err_cb: |
| @@ -1518,7 +1520,8 @@ static int create_qp(struct ib_uverbs_file *file, · 474 unmodified lines | |||
| 1518 | if (cmd->qp_type == IB_QPT_XRC_TGT) | 1520 | if (cmd->qp_type == IB_QPT_XRC_TGT) |
| 1519 | qp = ib_create_qp(pd, &attr); | 1521 | qp = ib_create_qp(pd, &attr); |
| 1520 | else | 1522 | else |
| 1521 | qp = _ib_create_qp(device, pd, &attr, uhw); | 1523 | qp = _ib_create_qp(device, pd, &attr, uhw, |
| 1524 | &obj->uevent.uobject); | ||
| 1522 | 1525 | ||
| 1523 | if (IS_ERR(qp)) { | 1526 | if (IS_ERR(qp)) { |
| 1524 | ret = PTR_ERR(qp); | 1527 | ret = PTR_ERR(qp); |
| @@ -1550,8 +1553,10 @@ static int create_qp(struct ib_uverbs_file *file, · 25 unmodified lines | |||
| 1550 | atomic_inc(&attr.srq->usecnt); | 1553 | atomic_inc(&attr.srq->usecnt); |
| 1551 | if (ind_tbl) | 1554 | if (ind_tbl) |
| 1552 | atomic_inc(&ind_tbl->usecnt); | 1555 | atomic_inc(&ind_tbl->usecnt); |
| 1556 | } else { | ||
| 1557 | /* It is done in _ib_create_qp for other QP types */ | ||
| 1558 | qp->uobject = &obj->uevent.uobject; | ||
| 1553 | } | 1559 | } |
| 1554 | qp->uobject = &obj->uevent.uobject; | ||
| 1555 | 1560 | ||
| 1556 | obj->uevent.uobject.object = qp; | 1561 | obj->uevent.uobject.object = qp; |
| 1557 | 1562 | ||
| @@ -1971,8 +1976,15 @@ static int modify_qp(struct ib_uverbs_file *file, · 413 unmodified lines | |||
| 1971 | goto release_qp; | 1976 | goto release_qp; |
| 1972 | } | 1977 | } |
| 1973 | 1978 | ||
| 1979 | if ((cmd->base.attr_mask & IB_QP_AV) && | ||
| 1980 | !rdma_is_port_valid(qp->device, cmd->base.dest.port_num)) { | ||
| 1981 | ret = -EINVAL; | ||
| 1982 | goto release_qp; | ||
| 1983 | } | ||
| 1984 | |||
| 1974 | if ((cmd->base.attr_mask & IB_QP_ALT_PATH) && | 1985 | if ((cmd->base.attr_mask & IB_QP_ALT_PATH) && |
| 1975 | !rdma_is_port_valid(qp->device, cmd->base.alt_port_num)) { | 1986 | (!rdma_is_port_valid(qp->device, cmd->base.alt_port_num) || |
| 1987 | !rdma_is_port_valid(qp->device, cmd->base.alt_dest.port_num))) { | ||
| 1976 | ret = -EINVAL; | 1988 | ret = -EINVAL; |
| 1977 | goto release_qp; | 1989 | goto release_qp; |
| 1978 | } | 1990 | } |
| @@ -2941,6 +2953,11 @@ int ib_uverbs_ex_create_wq(struct ib_uverbs_file *file, · 962 unmodified lines | |||
| 2941 | wq_init_attr.create_flags = cmd.create_flags; | 2953 | wq_init_attr.create_flags = cmd.create_flags; |
| 2942 | obj->uevent.events_reported = 0; | 2954 | obj->uevent.events_reported = 0; |
| 2943 | INIT_LIST_HEAD(&obj->uevent.event_list); | 2955 | INIT_LIST_HEAD(&obj->uevent.event_list); |
| 2956 | |||
| 2957 | if (!pd->device->create_wq) { | ||
| 2958 | err = -EOPNOTSUPP; | ||
| 2959 | goto err_put_cq; | ||
| 2960 | } | ||
| 2944 | wq = pd->device->create_wq(pd, &wq_init_attr, uhw); | 2961 | wq = pd->device->create_wq(pd, &wq_init_attr, uhw); |
| 2945 | if (IS_ERR(wq)) { | 2962 | if (IS_ERR(wq)) { |
| 2946 | err = PTR_ERR(wq); | 2963 | err = PTR_ERR(wq); |
| @@ -3084,7 +3101,12 @@ int ib_uverbs_ex_modify_wq(struct ib_uverbs_file *file, · 137 unmodified lines | |||
| 3084 | wq_attr.flags = cmd.flags; | 3101 | wq_attr.flags = cmd.flags; |
| 3085 | wq_attr.flags_mask = cmd.flags_mask; | 3102 | wq_attr.flags_mask = cmd.flags_mask; |
| 3086 | } | 3103 | } |
| 3104 | if (!wq->device->modify_wq) { | ||
| 3105 | ret = -EOPNOTSUPP; | ||
| 3106 | goto out; | ||
| 3107 | } | ||
| 3087 | ret = wq->device->modify_wq(wq, &wq_attr, cmd.attr_mask, uhw); | 3108 | ret = wq->device->modify_wq(wq, &wq_attr, cmd.attr_mask, uhw); |
| 3109 | out: | ||
| 3088 | uobj_put_obj_read(wq); | 3110 | uobj_put_obj_read(wq); |
| 3089 | return ret; | 3111 | return ret; |
| 3090 | } | 3112 | } |
| @@ -3181,6 +3203,11 @@ int ib_uverbs_ex_create_rwq_ind_table(struct ib_uverbs_file *file, · 90 unmodified lines | |||
| 3181 | 3203 | ||
| 3182 | init_attr.log_ind_tbl_size = cmd.log_ind_tbl_size; | 3204 | init_attr.log_ind_tbl_size = cmd.log_ind_tbl_size; |
| 3183 | init_attr.ind_tbl = wqs; | 3205 | init_attr.ind_tbl = wqs; |
| 3206 | |||
| 3207 | if (!ib_dev->create_rwq_ind_table) { | ||
| 3208 | err = -EOPNOTSUPP; | ||
| 3209 | goto err_uobj; | ||
| 3210 | } | ||
| 3184 | rwq_ind_tbl = ib_dev->create_rwq_ind_table(ib_dev, &init_attr, uhw); | 3211 | rwq_ind_tbl = ib_dev->create_rwq_ind_table(ib_dev, &init_attr, uhw); |
| 3185 | 3212 | ||
| 3186 | if (IS_ERR(rwq_ind_tbl)) { | 3213 | if (IS_ERR(rwq_ind_tbl)) { |
| @@ -3770,6 +3797,9 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file, · 583 unmodified lines | |||
| 3770 | struct ib_device_attr attr = {0}; | 3797 | struct ib_device_attr attr = {0}; |
| 3771 | int err; | 3798 | int err; |
| 3772 | 3799 | ||
| 3800 | if (!ib_dev->query_device) | ||
| 3801 | return -EOPNOTSUPP; | ||
| 3802 | |||
| 3773 | if (ucore->inlen < sizeof(cmd)) | 3803 | if (ucore->inlen < sizeof(cmd)) |
| 3774 | return -EINVAL; | 3804 | return -EINVAL; |
| 3775 | 3805 | ||
| @@ -59,6 +59,9 @@ static int uverbs_process_attr(struct ib_device *ibdev, · 58 unmodified lines | |||
| 59 | return 0; | 59 | return 0; |
| 60 | } | 60 | } |
| 61 | 61 | ||
| 62 | if (test_bit(attr_id, attr_bundle_h->valid_bitmap)) | ||
| 63 | return -EINVAL; | ||
| 64 | |||
| 62 | spec = &attr_spec_bucket->attrs[attr_id]; | 65 | spec = &attr_spec_bucket->attrs[attr_id]; |
| 63 | e = &elements[attr_id]; | 66 | e = &elements[attr_id]; |
| 64 | e->uattr = uattr_ptr; | 67 | e->uattr = uattr_ptr; |
| @@ -114,6 +114,7 @@ static size_t get_elements_above_id(const void **iters, · 113 unmodified lines | |||
| 114 | short min = SHRT_MAX; | 114 | short min = SHRT_MAX; |
| 115 | const void *elem; | 115 | const void *elem; |
| 116 | int i, j, last_stored = -1; | 116 | int i, j, last_stored = -1; |
| 117 | unsigned int equal_min = 0; | ||
| 117 | 118 | ||
| 118 | for_each_element(elem, i, j, elements, num_elements, num_offset, | 119 | for_each_element(elem, i, j, elements, num_elements, num_offset, |
| 119 | data_offset) { | 120 | data_offset) { |
| @@ -136,6 +137,10 @@ static size_t get_elements_above_id(const void **iters, · 16 unmodified lines | |||
| 136 | */ | 137 | */ |
| 137 | iters[last_stored == i ? num_iters - 1 : num_iters++] = elem; | 138 | iters[last_stored == i ? num_iters - 1 : num_iters++] = elem; |
| 138 | last_stored = i; | 139 | last_stored = i; |
| 140 | if (min == GET_ID(id)) | ||
| 141 | equal_min++; | ||
| 142 | else | ||
| 143 | equal_min = 1; | ||
| 139 | min = GET_ID(id); | 144 | min = GET_ID(id); |
| 140 | } | 145 | } |
| 141 | 146 | ||
| @@ -146,15 +151,10 @@ static size_t get_elements_above_id(const void **iters, · 4 unmodified lines | |||
| 146 | * Therefore, we need to clean the beginning of the array to make sure | 151 | * Therefore, we need to clean the beginning of the array to make sure |
| 147 | * all ids of final elements are equal to min. | 152 | * all ids of final elements are equal to min. |
| 148 | */ | 153 | */ |
| 149 | for (i = num_iters - 1; i >= 0 && | 154 | memmove(iters, iters + num_iters - equal_min, sizeof(*iters) * equal_min); |
| 150 | GET_ID(*(u16 *)(iters[i] + id_offset)) == min; i--) | ||
| 151 | ; | ||
| 152 | |||
| 153 | num_iters -= i + 1; | ||
| 154 | memmove(iters, iters + i + 1, sizeof(*iters) * num_iters); | ||
| 155 | 155 | ||
| 156 | *min_id = min; | 156 | *min_id = min; |
| 157 | return num_iters; | 157 | return equal_min; |
| 158 | } | 158 | } |
| 159 | 159 | ||
| 160 | #define find_max_element_entry_id(num_elements, elements, num_objects_fld, \ | 160 | #define find_max_element_entry_id(num_elements, elements, num_objects_fld, \ |
| @@ -322,7 +322,7 @@ static struct uverbs_method_spec *build_method_with_attrs(const struct uverbs_me · 161 unmodified lines | |||
| 322 | hash = kzalloc(sizeof(*hash) + | 322 | hash = kzalloc(sizeof(*hash) + |
| 323 | ALIGN(sizeof(*hash->attrs) * (attr_max_bucket + 1), | 323 | ALIGN(sizeof(*hash->attrs) * (attr_max_bucket + 1), |
| 324 | sizeof(long)) + | 324 | sizeof(long)) + |
| 325 | BITS_TO_LONGS(attr_max_bucket) * sizeof(long), | 325 | BITS_TO_LONGS(attr_max_bucket + 1) * sizeof(long), |
| 326 | GFP_KERNEL); | 326 | GFP_KERNEL); |
| 327 | if (!hash) { | 327 | if (!hash) { |
| 328 | res = -ENOMEM; | 328 | res = -ENOMEM; |
| @@ -509,7 +509,7 @@ static struct uverbs_object_spec *build_object_with_methods(const struct uverbs_ · 180 unmodified lines | |||
| 509 | * first handler which != NULL. This also defines the | 509 | * first handler which != NULL. This also defines the |
| 510 | * set of flags used for this handler. | 510 | * set of flags used for this handler. |
| 511 | */ | 511 | */ |
| 512 | for (i = num_object_defs - 1; | 512 | for (i = num_method_defs - 1; |
| 513 | i >= 0 && !method_defs[i]->handler; i--) | 513 | i >= 0 && !method_defs[i]->handler; i--) |
| 514 | ; | 514 | ; |
| 515 | hash->methods[min_id++] = method; | 515 | hash->methods[min_id++] = method; |
| @@ -650,12 +650,21 @@ static int verify_command_mask(struct ib_device *ib_dev, __u32 command) · 649 unmodified lines | |||
| 650 | return -1; | 650 | return -1; |
| 651 | } | 651 | } |
| 652 | 652 | ||
| 653 | static bool verify_command_idx(u32 command, bool extended) | ||
| 654 | { | ||
| 655 | if (extended) | ||
| 656 | return command < ARRAY_SIZE(uverbs_ex_cmd_table); | ||
| 657 | |||
| 658 | return command < ARRAY_SIZE(uverbs_cmd_table); | ||
| 659 | } | ||
| 660 | |||
| 653 | static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, | 661 | static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, |
| 654 | size_t count, loff_t *pos) | 662 | size_t count, loff_t *pos) |
| 655 | { | 663 | { |
| 656 | struct ib_uverbs_file *file = filp->private_data; | 664 | struct ib_uverbs_file *file = filp->private_data; |
| 657 | struct ib_device *ib_dev; | 665 | struct ib_device *ib_dev; |
| 658 | struct ib_uverbs_cmd_hdr hdr; | 666 | struct ib_uverbs_cmd_hdr hdr; |
| 667 | bool extended_command; | ||
| 659 | __u32 command; | 668 | __u32 command; |
| 660 | __u32 flags; | 669 | __u32 flags; |
| 661 | int srcu_key; | 670 | int srcu_key; |
| @@ -688,6 +697,15 @@ static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, · 26 unmodified lines | |||
| 688 | } | 697 | } |
| 689 | 698 | ||
| 690 | command = hdr.command & IB_USER_VERBS_CMD_COMMAND_MASK; | 699 | command = hdr.command & IB_USER_VERBS_CMD_COMMAND_MASK; |
| 700 | flags = (hdr.command & | ||
| 701 | IB_USER_VERBS_CMD_FLAGS_MASK) >> IB_USER_VERBS_CMD_FLAGS_SHIFT; | ||
| 702 | |||
| 703 | extended_command = flags & IB_USER_VERBS_CMD_FLAG_EXTENDED; | ||
| 704 | if (!verify_command_idx(command, extended_command)) { | ||
| 705 | ret = -EINVAL; | ||
| 706 | goto out; | ||
| 707 | } | ||
| 708 | |||
| 691 | if (verify_command_mask(ib_dev, command)) { | 709 | if (verify_command_mask(ib_dev, command)) { |
| 692 | ret = -EOPNOTSUPP; | 710 | ret = -EOPNOTSUPP; |
| 693 | goto out; | 711 | goto out; |
| @@ -699,12 +717,8 @@ static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, · 5 unmodified lines | |||
| 699 | goto out; | 717 | goto out; |
| 700 | } | 718 | } |
| 701 | 719 | ||
| 702 | flags = (hdr.command & | ||
| 703 | IB_USER_VERBS_CMD_FLAGS_MASK) >> IB_USER_VERBS_CMD_FLAGS_SHIFT; | ||
| 704 | |||
| 705 | if (!flags) { | 720 | if (!flags) { |
| 706 | if (command >= ARRAY_SIZE(uverbs_cmd_table) || | 721 | if (!uverbs_cmd_table[command]) { |
| 707 | !uverbs_cmd_table[command]) { | ||
| 708 | ret = -EINVAL; | 722 | ret = -EINVAL; |
| 709 | goto out; | 723 | goto out; |
| 710 | } | 724 | } |
| @@ -725,8 +739,7 @@ static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, · 14 unmodified lines | |||
| 725 | struct ib_udata uhw; | 739 | struct ib_udata uhw; |
| 726 | size_t written_count = count; | 740 | size_t written_count = count; |
| 727 | 741 | ||
| 728 | if (command >= ARRAY_SIZE(uverbs_ex_cmd_table) || | 742 | if (!uverbs_ex_cmd_table[command]) { |
| 729 | !uverbs_ex_cmd_table[command]) { | ||
| 730 | ret = -ENOSYS; | 743 | ret = -ENOSYS; |
| 731 | goto out; | 744 | goto out; |
| 732 | } | 745 | } |
| @@ -942,6 +955,7 @@ static const struct file_operations uverbs_fops = { · 209 unmodified lines | |||
| 942 | .llseek = no_llseek, | 955 | .llseek = no_llseek, |
| 943 | #if IS_ENABLED(CONFIG_INFINIBAND_EXP_USER_ACCESS) | 956 | #if IS_ENABLED(CONFIG_INFINIBAND_EXP_USER_ACCESS) |
| 944 | .unlocked_ioctl = ib_uverbs_ioctl, | 957 | .unlocked_ioctl = ib_uverbs_ioctl, |
| 958 | .compat_ioctl = ib_uverbs_ioctl, | ||
| 945 | #endif | 959 | #endif |
| 946 | }; | 960 | }; |
| 947 | 961 | ||
| @@ -954,6 +968,7 @@ static const struct file_operations uverbs_mmap_fops = { · 6 unmodified lines | |||
| 954 | .llseek = no_llseek, | 968 | .llseek = no_llseek, |
| 955 | #if IS_ENABLED(CONFIG_INFINIBAND_EXP_USER_ACCESS) | 969 | #if IS_ENABLED(CONFIG_INFINIBAND_EXP_USER_ACCESS) |
| 956 | .unlocked_ioctl = ib_uverbs_ioctl, | 970 | .unlocked_ioctl = ib_uverbs_ioctl, |
| 971 | .compat_ioctl = ib_uverbs_ioctl, | ||
| 957 | #endif | 972 | #endif |
| 958 | }; | 973 | }; |
| 959 | 974 | ||
| @@ -234,15 +234,18 @@ static void create_udata(struct uverbs_attr_bundle *ctx, · 233 unmodified lines | |||
| 234 | uverbs_attr_get(ctx, UVERBS_UHW_OUT); | 234 | uverbs_attr_get(ctx, UVERBS_UHW_OUT); |
| 235 | 235 | ||
| 236 | if (!IS_ERR(uhw_in)) { | 236 | if (!IS_ERR(uhw_in)) { |
| 237 | udata->inbuf = uhw_in->ptr_attr.ptr; | ||
| 238 | udata->inlen = uhw_in->ptr_attr.len; | 237 | udata->inlen = uhw_in->ptr_attr.len; |
| 238 | if (uverbs_attr_ptr_is_inline(uhw_in)) | ||
| 239 | udata->inbuf = &uhw_in->uattr->data; | ||
| 240 | else | ||
| 241 | udata->inbuf = u64_to_user_ptr(uhw_in->ptr_attr.data); | ||
| 239 | } else { | 242 | } else { |
| 240 | udata->inbuf = NULL; | 243 | udata->inbuf = NULL; |
| 241 | udata->inlen = 0; | 244 | udata->inlen = 0; |
| 242 | } | 245 | } |
| 243 | 246 | ||
| 244 | if (!IS_ERR(uhw_out)) { | 247 | if (!IS_ERR(uhw_out)) { |
| 245 | udata->outbuf = uhw_out->ptr_attr.ptr; | 248 | udata->outbuf = u64_to_user_ptr(uhw_out->ptr_attr.data); |
| 246 | udata->outlen = uhw_out->ptr_attr.len; | 249 | udata->outlen = uhw_out->ptr_attr.len; |
| 247 | } else { | 250 | } else { |
| 248 | udata->outbuf = NULL; | 251 | udata->outbuf = NULL; |
| @@ -323,7 +326,8 @@ static int uverbs_create_cq_handler(struct ib_device *ib_dev, · 74 unmodified lines | |||
| 323 | cq->res.type = RDMA_RESTRACK_CQ; | 326 | cq->res.type = RDMA_RESTRACK_CQ; |
| 324 | rdma_restrack_add(&cq->res); | 327 | rdma_restrack_add(&cq->res); |
| 325 | 328 | ||
| 326 | ret = uverbs_copy_to(attrs, CREATE_CQ_RESP_CQE, &cq->cqe); | 329 | ret = uverbs_copy_to(attrs, CREATE_CQ_RESP_CQE, &cq->cqe, |
| 330 | sizeof(cq->cqe)); | ||
| 327 | if (ret) | 331 | if (ret) |
| 328 | goto err_cq; | 332 | goto err_cq; |
| 329 | 333 | ||
| @@ -375,7 +379,7 @@ static int uverbs_destroy_cq_handler(struct ib_device *ib_dev, · 45 unmodified lines | |||
| 375 | resp.comp_events_reported = obj->comp_events_reported; | 379 | resp.comp_events_reported = obj->comp_events_reported; |
| 376 | resp.async_events_reported = obj->async_events_reported; | 380 | resp.async_events_reported = obj->async_events_reported; |
| 377 | 381 | ||
| 378 | return uverbs_copy_to(attrs, DESTROY_CQ_RESP, &resp); | 382 | return uverbs_copy_to(attrs, DESTROY_CQ_RESP, &resp, sizeof(resp)); |
| 379 | } | 383 | } |
| 380 | 384 | ||
| 381 | static DECLARE_UVERBS_METHOD( | 385 | static DECLARE_UVERBS_METHOD( |
| @@ -887,7 +887,7 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd, · 886 unmodified lines | |||
| 887 | if (qp_init_attr->cap.max_rdma_ctxs) | 887 | if (qp_init_attr->cap.max_rdma_ctxs) |
| 888 | rdma_rw_init_qp(device, qp_init_attr); | 888 | rdma_rw_init_qp(device, qp_init_attr); |
| 889 | 889 | ||
| 890 | qp = _ib_create_qp(device, pd, qp_init_attr, NULL); | 890 | qp = _ib_create_qp(device, pd, qp_init_attr, NULL, NULL); |
| 891 | if (IS_ERR(qp)) | 891 | if (IS_ERR(qp)) |
| 892 | return qp; | 892 | return qp; |
| 893 | 893 | ||
| @@ -898,7 +898,6 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd, · 4 unmodified lines | |||
| 898 | } | 898 | } |
| 899 | 899 | ||
| 900 | qp->real_qp = qp; | 900 | qp->real_qp = qp; |
| 901 | qp->uobject = NULL; | ||
| 902 | qp->qp_type = qp_init_attr->qp_type; | 901 | qp->qp_type = qp_init_attr->qp_type; |
| 903 | qp->rwq_ind_tbl = qp_init_attr->rwq_ind_tbl; | 902 | qp->rwq_ind_tbl = qp_init_attr->rwq_ind_tbl; |
| 904 | 903 | ||
| @@ -120,7 +120,6 @@ struct bnxt_re_dev { · 119 unmodified lines | |||
| 120 | #define BNXT_RE_FLAG_HAVE_L2_REF 3 | 120 | #define BNXT_RE_FLAG_HAVE_L2_REF 3 |
| 121 | #define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4 | 121 | #define BNXT_RE_FLAG_RCFW_CHANNEL_EN 4 |
| 122 | #define BNXT_RE_FLAG_QOS_WORK_REG 5 | 122 | #define BNXT_RE_FLAG_QOS_WORK_REG 5 |
| 123 | #define BNXT_RE_FLAG_TASK_IN_PROG 6 | ||
| 124 | #define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 | 123 | #define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 |
| 125 | struct net_device *netdev; | 124 | struct net_device *netdev; |
| 126 | unsigned int version, major, minor; | 125 | unsigned int version, major, minor; |
| @@ -158,6 +157,7 @@ struct bnxt_re_dev { · 31 unmodified lines | |||
| 158 | atomic_t srq_count; | 157 | atomic_t srq_count; |
| 159 | atomic_t mr_count; | 158 | atomic_t mr_count; |
| 160 | atomic_t mw_count; | 159 | atomic_t mw_count; |
| 160 | atomic_t sched_count; | ||
| 161 | /* Max of 2 lossless traffic class supported per port */ | 161 | /* Max of 2 lossless traffic class supported per port */ |
| 162 | u16 cosq[2]; | 162 | u16 cosq[2]; |
| 163 | 163 | ||
| @@ -174,10 +174,8 @@ int bnxt_re_query_device(struct ib_device *ibdev, · 173 unmodified lines | |||
| 174 | ib_attr->max_pd = dev_attr->max_pd; | 174 | ib_attr->max_pd = dev_attr->max_pd; |
| 175 | ib_attr->max_qp_rd_atom = dev_attr->max_qp_rd_atom; | 175 | ib_attr->max_qp_rd_atom = dev_attr->max_qp_rd_atom; |
| 176 | ib_attr->max_qp_init_rd_atom = dev_attr->max_qp_init_rd_atom; | 176 | ib_attr->max_qp_init_rd_atom = dev_attr->max_qp_init_rd_atom; |
| 177 | if (dev_attr->is_atomic) { | 177 | ib_attr->atomic_cap = IB_ATOMIC_NONE; |
| 178 | ib_attr->atomic_cap = IB_ATOMIC_HCA; | 178 | ib_attr->masked_atomic_cap = IB_ATOMIC_NONE; |
| 179 | ib_attr->masked_atomic_cap = IB_ATOMIC_HCA; | ||
| 180 | } | ||
| 181 | 179 | ||
| 182 | ib_attr->max_ee_rd_atom = 0; | 180 | ib_attr->max_ee_rd_atom = 0; |
| 183 | ib_attr->max_res_rd_atom = 0; | 181 | ib_attr->max_res_rd_atom = 0; |
| @@ -787,20 +785,51 @@ int bnxt_re_query_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr) · 603 unmodified lines | |||
| 787 | return 0; | 785 | return 0; |
| 788 | } | 786 | } |
| 789 | 787 | ||
| 788 | unsigned long bnxt_re_lock_cqs(struct bnxt_re_qp *qp) | ||
| 789 | __acquires(&qp->scq->cq_lock) __acquires(&qp->rcq->cq_lock) | ||
| 790 | { | ||
| 791 | unsigned long flags; | ||
| 792 | |||
| 793 | spin_lock_irqsave(&qp->scq->cq_lock, flags); | ||
| 794 | if (qp->rcq != qp->scq) | ||
| 795 | spin_lock(&qp->rcq->cq_lock); | ||
| 796 | else | ||
| 797 | __acquire(&qp->rcq->cq_lock); | ||
| 798 | |||
| 799 | return flags; | ||
| 800 | } | ||
| 801 | |||
| 802 | void bnxt_re_unlock_cqs(struct bnxt_re_qp *qp, | ||
| 803 | unsigned long flags) | ||
| 804 | __releases(&qp->scq->cq_lock) __releases(&qp->rcq->cq_lock) | ||
| 805 | { | ||
| 806 | if (qp->rcq != qp->scq) | ||
| 807 | spin_unlock(&qp->rcq->cq_lock); | ||
| 808 | else | ||
| 809 | __release(&qp->rcq->cq_lock); | ||
| 810 | spin_unlock_irqrestore(&qp->scq->cq_lock, flags); | ||
| 811 | } | ||
| 812 | |||
| 790 | /* Queue Pairs */ | 813 | /* Queue Pairs */ |
| 791 | int bnxt_re_destroy_qp(struct ib_qp *ib_qp) | 814 | int bnxt_re_destroy_qp(struct ib_qp *ib_qp) |
| 792 | { | 815 | { |
| 793 | struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp); | 816 | struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp); |
| 794 | struct bnxt_re_dev *rdev = qp->rdev; | 817 | struct bnxt_re_dev *rdev = qp->rdev; |
| 795 | int rc; | 818 | int rc; |
| 819 | unsigned int flags; | ||
| 796 | 820 | ||
| 797 | bnxt_qplib_flush_cqn_wq(&qp->qplib_qp); | 821 | bnxt_qplib_flush_cqn_wq(&qp->qplib_qp); |
| 798 | bnxt_qplib_del_flush_qp(&qp->qplib_qp); | ||
| 799 | rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp); | 822 | rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp); |
| 800 | if (rc) { | 823 | if (rc) { |
| 801 | dev_err(rdev_to_dev(rdev), "Failed to destroy HW QP"); | 824 | dev_err(rdev_to_dev(rdev), "Failed to destroy HW QP"); |
| 802 | return rc; | 825 | return rc; |
| 803 | } | 826 | } |
| 827 | |||
| 828 | flags = bnxt_re_lock_cqs(qp); | ||
| 829 | bnxt_qplib_clean_qp(&qp->qplib_qp); | ||
| 830 | bnxt_re_unlock_cqs(qp, flags); | ||
| 831 | bnxt_qplib_free_qp_res(&rdev->qplib_res, &qp->qplib_qp); | ||
| 832 | |||
| 804 | if (ib_qp->qp_type == IB_QPT_GSI && rdev->qp1_sqp) { | 833 | if (ib_qp->qp_type == IB_QPT_GSI && rdev->qp1_sqp) { |
| 805 | rc = bnxt_qplib_destroy_ah(&rdev->qplib_res, | 834 | rc = bnxt_qplib_destroy_ah(&rdev->qplib_res, |
| 806 | &rdev->sqp_ah->qplib_ah); | 835 | &rdev->sqp_ah->qplib_ah); |
| @@ -810,7 +839,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp) · 3 unmodified lines | |||
| 810 | return rc; | 839 | return rc; |
| 811 | } | 840 | } |
| 812 | 841 | ||
| 813 | bnxt_qplib_del_flush_qp(&qp->qplib_qp); | 842 | bnxt_qplib_clean_qp(&qp->qplib_qp); |
| 814 | rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, | 843 | rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, |
| 815 | &rdev->qp1_sqp->qplib_qp); | 844 | &rdev->qp1_sqp->qplib_qp); |
| 816 | if (rc) { | 845 | if (rc) { |
| @@ -1069,6 +1098,7 @@ struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd, · 252 unmodified lines | |||
| 1069 | goto fail; | 1098 | goto fail; |
| 1070 | } | 1099 | } |
| 1071 | qp->qplib_qp.scq = &cq->qplib_cq; | 1100 | qp->qplib_qp.scq = &cq->qplib_cq; |
| 1101 | qp->scq = cq; | ||
| 1072 | } | 1102 | } |
| 1073 | 1103 | ||
| 1074 | if (qp_init_attr->recv_cq) { | 1104 | if (qp_init_attr->recv_cq) { |
| @@ -1080,6 +1110,7 @@ struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd, · 5 unmodified lines | |||
| 1080 | goto fail; | 1110 | goto fail; |
| 1081 | } | 1111 | } |
| 1082 | qp->qplib_qp.rcq = &cq->qplib_cq; | 1112 | qp->qplib_qp.rcq = &cq->qplib_cq; |
| 1113 | qp->rcq = cq; | ||
| 1083 | } | 1114 | } |
| 1084 | 1115 | ||
| 1085 | if (qp_init_attr->srq) { | 1116 | if (qp_init_attr->srq) { |
| @@ -1185,7 +1216,7 @@ struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd, · 99 unmodified lines | |||
| 1185 | rc = bnxt_qplib_create_qp(&rdev->qplib_res, &qp->qplib_qp); | 1216 | rc = bnxt_qplib_create_qp(&rdev->qplib_res, &qp->qplib_qp); |
| 1186 | if (rc) { | 1217 | if (rc) { |
| 1187 | dev_err(rdev_to_dev(rdev), "Failed to create HW QP"); | 1218 | dev_err(rdev_to_dev(rdev), "Failed to create HW QP"); |
| 1188 | goto fail; | 1219 | goto free_umem; |
| 1189 | } | 1220 | } |
| 1190 | } | 1221 | } |
| 1191 | 1222 | ||
| @@ -1213,6 +1244,13 @@ struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd, · 21 unmodified lines | |||
| 1213 | return &qp->ib_qp; | 1244 | return &qp->ib_qp; |
| 1214 | qp_destroy: | 1245 | qp_destroy: |
| 1215 | bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp); | 1246 | bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp); |
| 1247 | free_umem: | ||
| 1248 | if (udata) { | ||
| 1249 | if (qp->rumem) | ||
| 1250 | ib_umem_release(qp->rumem); | ||
| 1251 | if (qp->sumem) | ||
| 1252 | ib_umem_release(qp->sumem); | ||
| 1253 | } | ||
| 1216 | fail: | 1254 | fail: |
| 1217 | kfree(qp); | 1255 | kfree(qp); |
| 1218 | return ERR_PTR(rc); | 1256 | return ERR_PTR(rc); |
| @@ -1568,6 +1606,7 @@ int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr, · 349 unmodified lines | |||
| 1568 | int status; | 1606 | int status; |
| 1569 | union ib_gid sgid; | 1607 | union ib_gid sgid; |
| 1570 | struct ib_gid_attr sgid_attr; | 1608 | struct ib_gid_attr sgid_attr; |
| 1609 | unsigned int flags; | ||
| 1571 | u8 nw_type; | 1610 | u8 nw_type; |
| 1572 | 1611 | ||
| 1573 | qp->qplib_qp.modify_flags = 0; | 1612 | qp->qplib_qp.modify_flags = 0; |
| @@ -1596,14 +1635,18 @@ int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr, · 22 unmodified lines | |||
| 1596 | dev_dbg(rdev_to_dev(rdev), | 1635 | dev_dbg(rdev_to_dev(rdev), |
| 1597 | "Move QP = %p to flush list\n", | 1636 | "Move QP = %p to flush list\n", |
| 1598 | qp); | 1637 | qp); |
| 1638 | flags = bnxt_re_lock_cqs(qp); | ||
| 1599 | bnxt_qplib_add_flush_qp(&qp->qplib_qp); | 1639 | bnxt_qplib_add_flush_qp(&qp->qplib_qp); |
| 1640 | bnxt_re_unlock_cqs(qp, flags); | ||
| 1600 | } | 1641 | } |
| 1601 | if (!qp->sumem && | 1642 | if (!qp->sumem && |
| 1602 | qp->qplib_qp.state == CMDQ_MODIFY_QP_NEW_STATE_RESET) { | 1643 | qp->qplib_qp.state == CMDQ_MODIFY_QP_NEW_STATE_RESET) { |
| 1603 | dev_dbg(rdev_to_dev(rdev), | 1644 | dev_dbg(rdev_to_dev(rdev), |
| 1604 | "Move QP = %p out of flush list\n", | 1645 | "Move QP = %p out of flush list\n", |
| 1605 | qp); | 1646 | qp); |
| 1606 | bnxt_qplib_del_flush_qp(&qp->qplib_qp); | 1647 | flags = bnxt_re_lock_cqs(qp); |
| 1648 | bnxt_qplib_clean_qp(&qp->qplib_qp); | ||
| 1649 | bnxt_re_unlock_cqs(qp, flags); | ||
| 1607 | } | 1650 | } |
| 1608 | } | 1651 | } |
| 1609 | if (qp_attr_mask & IB_QP_EN_SQD_ASYNC_NOTIFY) { | 1652 | if (qp_attr_mask & IB_QP_EN_SQD_ASYNC_NOTIFY) { |
| @@ -2189,10 +2232,13 @@ static int bnxt_re_build_inv_wqe(struct ib_send_wr *wr, · 579 unmodified lines | |||
| 2189 | wqe->type = BNXT_QPLIB_SWQE_TYPE_LOCAL_INV; | 2232 | wqe->type = BNXT_QPLIB_SWQE_TYPE_LOCAL_INV; |
| 2190 | wqe->local_inv.inv_l_key = wr->ex.invalidate_rkey; | 2233 | wqe->local_inv.inv_l_key = wr->ex.invalidate_rkey; |
| 2191 | 2234 | ||
| 2235 | /* Need unconditional fence for local invalidate | ||
| 2236 | * opcode to work as expected. | ||
| 2237 | */ | ||
| 2238 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_UC_FENCE; | ||
| 2239 | |||
| 2192 | if (wr->send_flags & IB_SEND_SIGNALED) | 2240 | if (wr->send_flags & IB_SEND_SIGNALED) |
| 2193 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SIGNAL_COMP; | 2241 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SIGNAL_COMP; |
| 2194 | if (wr->send_flags & IB_SEND_FENCE) | ||
| 2195 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_UC_FENCE; | ||
| 2196 | if (wr->send_flags & IB_SEND_SOLICITED) | 2242 | if (wr->send_flags & IB_SEND_SOLICITED) |
| 2197 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SOLICIT_EVENT; | 2243 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SOLICIT_EVENT; |
| 2198 | 2244 | ||
| @@ -2213,8 +2259,12 @@ static int bnxt_re_build_reg_wqe(struct ib_reg_wr *wr, · 14 unmodified lines | |||
| 2213 | wqe->frmr.levels = qplib_frpl->hwq.level + 1; | 2259 | wqe->frmr.levels = qplib_frpl->hwq.level + 1; |
| 2214 | wqe->type = BNXT_QPLIB_SWQE_TYPE_REG_MR; | 2260 | wqe->type = BNXT_QPLIB_SWQE_TYPE_REG_MR; |
| 2215 | 2261 | ||
| 2216 | if (wr->wr.send_flags & IB_SEND_FENCE) | 2262 | /* Need unconditional fence for reg_mr |
| 2217 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_UC_FENCE; | 2263 | * opcode to function as expected. |
| 2264 | */ | ||
| 2265 | |||
| 2266 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_UC_FENCE; | ||
| 2267 | |||
| 2218 | if (wr->wr.send_flags & IB_SEND_SIGNALED) | 2268 | if (wr->wr.send_flags & IB_SEND_SIGNALED) |
| 2219 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SIGNAL_COMP; | 2269 | wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SIGNAL_COMP; |
| 2220 | 2270 | ||
| @@ -89,6 +89,8 @@ struct bnxt_re_qp { · 88 unmodified lines | |||
| 89 | /* QP1 */ | 89 | /* QP1 */ |
| 90 | u32 send_psn; | 90 | u32 send_psn; |
| 91 | struct ib_ud_header qp1_hdr; | 91 | struct ib_ud_header qp1_hdr; |
| 92 | struct bnxt_re_cq *scq; | ||
| 93 | struct bnxt_re_cq *rcq; | ||
| 92 | }; | 94 | }; |
| 93 | 95 | ||
| 94 | struct bnxt_re_cq { | 96 | struct bnxt_re_cq { |
| @@ -220,4 +222,7 @@ struct ib_ucontext *bnxt_re_alloc_ucontext(struct ib_device *ibdev, · 125 unmodified lines | |||
| 220 | struct ib_udata *udata); | 222 | struct ib_udata *udata); |
| 221 | int bnxt_re_dealloc_ucontext(struct ib_ucontext *context); | 223 | int bnxt_re_dealloc_ucontext(struct ib_ucontext *context); |
| 222 | int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); | 224 | int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); |
| 225 | |||
| 226 | unsigned long bnxt_re_lock_cqs(struct bnxt_re_qp *qp); | ||
| 227 | void bnxt_re_unlock_cqs(struct bnxt_re_qp *qp, unsigned long flags); | ||
| 223 | #endif /* __BNXT_RE_IB_VERBS_H__ */ | 228 | #endif /* __BNXT_RE_IB_VERBS_H__ */ |
| @@ -656,7 +656,6 @@ static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) · 655 unmodified lines | |||
| 656 | mutex_unlock(&bnxt_re_dev_lock); | 656 | mutex_unlock(&bnxt_re_dev_lock); |
| 657 | 657 | ||
| 658 | synchronize_rcu(); | 658 | synchronize_rcu(); |
| 659 | flush_workqueue(bnxt_re_wq); | ||
| 660 | 659 | ||
| 661 | ib_dealloc_device(&rdev->ibdev); | 660 | ib_dealloc_device(&rdev->ibdev); |
| 662 | /* rdev is gone */ | 661 | /* rdev is gone */ |
| @@ -731,6 +730,13 @@ static int bnxt_re_handle_qp_async_event(struct creq_qp_event *qp_event, · 68 unmodified lines | |||
| 731 | struct bnxt_re_qp *qp) | 730 | struct bnxt_re_qp *qp) |
| 732 | { | 731 | { |
| 733 | struct ib_event event; | 732 | struct ib_event event; |
| 733 | unsigned int flags; | ||
| 734 | |||
| 735 | if (qp->qplib_qp.state == CMDQ_MODIFY_QP_NEW_STATE_ERR) { | ||
| 736 | flags = bnxt_re_lock_cqs(qp); | ||
| 737 | bnxt_qplib_add_flush_qp(&qp->qplib_qp); | ||
| 738 | bnxt_re_unlock_cqs(qp, flags); | ||
| 739 | } | ||
| 734 | 740 | ||
| 735 | memset(&event, 0, sizeof(event)); | 741 | memset(&event, 0, sizeof(event)); |
| 736 | if (qp->qplib_qp.srq) { | 742 | if (qp->qplib_qp.srq) { |
| @@ -1417,9 +1423,12 @@ static void bnxt_re_task(struct work_struct *work) · 680 unmodified lines | |||
| 1417 | switch (re_work->event) { | 1423 | switch (re_work->event) { |
| 1418 | case NETDEV_REGISTER: | 1424 | case NETDEV_REGISTER: |
| 1419 | rc = bnxt_re_ib_reg(rdev); | 1425 | rc = bnxt_re_ib_reg(rdev); |
| 1420 | if (rc) | 1426 | if (rc) { |
| 1421 | dev_err(rdev_to_dev(rdev), | 1427 | dev_err(rdev_to_dev(rdev), |
| 1422 | "Failed to register with IB: %#x", rc); | 1428 | "Failed to register with IB: %#x", rc); |
| 1429 | bnxt_re_remove_one(rdev); | ||
| 1430 | bnxt_re_dev_unreg(rdev); | ||
| 1431 | } | ||
| 1423 | break; | 1432 | break; |
| 1424 | case NETDEV_UP: | 1433 | case NETDEV_UP: |
| 1425 | bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, | 1434 | bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, |
| @@ -1441,7 +1450,7 @@ static void bnxt_re_task(struct work_struct *work) · 15 unmodified lines | |||
| 1441 | break; | 1450 | break; |
| 1442 | } | 1451 | } |
| 1443 | smp_mb__before_atomic(); | 1452 | smp_mb__before_atomic(); |
| 1444 | clear_bit(BNXT_RE_FLAG_TASK_IN_PROG, &rdev->flags); | 1453 | atomic_dec(&rdev->sched_count); |
| 1445 | kfree(re_work); | 1454 | kfree(re_work); |
| 1446 | } | 1455 | } |
| 1447 | 1456 | ||
| @@ -1503,7 +1512,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, · 55 unmodified lines | |||
| 1503 | /* netdev notifier will call NETDEV_UNREGISTER again later since | 1512 | /* netdev notifier will call NETDEV_UNREGISTER again later since |
| 1504 | * we are still holding the reference to the netdev | 1513 | * we are still holding the reference to the netdev |
| 1505 | */ | 1514 | */ |
| 1506 | if (test_bit(BNXT_RE_FLAG_TASK_IN_PROG, &rdev->flags)) | 1515 | if (atomic_read(&rdev->sched_count) > 0) |
| 1507 | goto exit; | 1516 | goto exit; |
| 1508 | bnxt_re_ib_unreg(rdev, false); | 1517 | bnxt_re_ib_unreg(rdev, false); |
| 1509 | bnxt_re_remove_one(rdev); | 1518 | bnxt_re_remove_one(rdev); |
| @@ -1523,7 +1532,7 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, · 13 unmodified lines | |||
| 1523 | re_work->vlan_dev = (real_dev == netdev ? | 1532 | re_work->vlan_dev = (real_dev == netdev ? |
| 1524 | NULL : netdev); | 1533 | NULL : netdev); |
| 1525 | INIT_WORK(&re_work->work, bnxt_re_task); | 1534 | INIT_WORK(&re_work->work, bnxt_re_task); |
| 1526 | set_bit(BNXT_RE_FLAG_TASK_IN_PROG, &rdev->flags); | 1535 | atomic_inc(&rdev->sched_count); |
| 1527 | queue_work(bnxt_re_wq, &re_work->work); | 1536 | queue_work(bnxt_re_wq, &re_work->work); |
| 1528 | } | 1537 | } |
| 1529 | } | 1538 | } |
| @@ -1578,6 +1587,11 @@ static void __exit bnxt_re_mod_exit(void) · 48 unmodified lines | |||
| 1578 | */ | 1587 | */ |
| 1579 | list_for_each_entry_safe_reverse(rdev, next, &to_be_deleted, list) { | 1588 | list_for_each_entry_safe_reverse(rdev, next, &to_be_deleted, list) { |
| 1580 | dev_info(rdev_to_dev(rdev), "Unregistering Device"); | 1589 | dev_info(rdev_to_dev(rdev), "Unregistering Device"); |
| 1590 | /* | ||
| 1591 | * Flush out any scheduled tasks before destroying the | ||
| 1592 | * resources | ||
| 1593 | */ | ||
| 1594 | flush_workqueue(bnxt_re_wq); | ||
| 1581 | bnxt_re_dev_stop(rdev); | 1595 | bnxt_re_dev_stop(rdev); |
| 1582 | bnxt_re_ib_unreg(rdev, true); | 1596 | bnxt_re_ib_unreg(rdev, true); |
| 1583 | bnxt_re_remove_one(rdev); | 1597 | bnxt_re_remove_one(rdev); |
| @@ -88,75 +88,35 @@ static void __bnxt_qplib_add_flush_qp(struct bnxt_qplib_qp *qp) · 87 unmodified lines | |||
| 88 | } | 88 | } |
| 89 | } | 89 | } |
| 90 | 90 | ||
| 91 | void bnxt_qplib_acquire_cq_locks(struct bnxt_qplib_qp *qp, | 91 | static void bnxt_qplib_acquire_cq_flush_locks(struct bnxt_qplib_qp *qp, |
| 92 | unsigned long *flags) | 92 | unsigned long *flags) |
| 93 | __acquires(&qp->scq->hwq.lock) __acquires(&qp->rcq->hwq.lock) | 93 | __acquires(&qp->scq->flush_lock) __acquires(&qp->rcq->flush_lock) |
| 94 | { | 94 | { |
| 95 | spin_lock_irqsave(&qp->scq->hwq.lock, *flags); | 95 | spin_lock_irqsave(&qp->scq->flush_lock, *flags); |
| 96 | if (qp->scq == qp->rcq) | 96 | if (qp->scq == qp->rcq) |
| 97 | __acquire(&qp->rcq->hwq.lock); | 97 | __acquire(&qp->rcq->flush_lock); |
| 98 | else | 98 | else |
| 99 | spin_lock(&qp->rcq->hwq.lock); | 99 | spin_lock(&qp->rcq->flush_lock); |
| 100 | } | 100 | } |
| 101 | 101 | ||
| 102 | void bnxt_qplib_release_cq_locks(struct bnxt_qplib_qp *qp, | 102 | static void bnxt_qplib_release_cq_flush_locks(struct bnxt_qplib_qp *qp, |
| 103 | unsigned long *flags) | 103 | unsigned long *flags) |
| 104 | __releases(&qp->scq->hwq.lock) __releases(&qp->rcq->hwq.lock) | 104 | __releases(&qp->scq->flush_lock) __releases(&qp->rcq->flush_lock) |
| 105 | { | 105 | { |
| 106 | if (qp->scq == qp->rcq) | 106 | if (qp->scq == qp->rcq) |
| 107 | __release(&qp->rcq->hwq.lock); | 107 | __release(&qp->rcq->flush_lock); |
| 108 | else | 108 | else |
| 109 | spin_unlock(&qp->rcq->hwq.lock); | 109 | spin_unlock(&qp->rcq->flush_lock); |
| 110 | spin_unlock_irqrestore(&qp->scq->hwq.lock, *flags); | 110 | spin_unlock_irqrestore(&qp->scq->flush_lock, *flags); |
| 111 | } | ||
| 112 | |||
| 113 | static struct bnxt_qplib_cq *bnxt_qplib_find_buddy_cq(struct bnxt_qplib_qp *qp, | ||
| 114 | struct bnxt_qplib_cq *cq) | ||
| 115 | { | ||
| 116 | struct bnxt_qplib_cq *buddy_cq = NULL; | ||
| 117 | |||
| 118 | if (qp->scq == qp->rcq) | ||
| 119 | buddy_cq = NULL; | ||
| 120 | else if (qp->scq == cq) | ||
| 121 | buddy_cq = qp->rcq; | ||
| 122 | else | ||
| 123 | buddy_cq = qp->scq; | ||
| 124 | return buddy_cq; | ||
| 125 | } | ||
| 126 | |||
| 127 | static void bnxt_qplib_lock_buddy_cq(struct bnxt_qplib_qp *qp, | ||
| 128 | struct bnxt_qplib_cq *cq) | ||
| 129 | __acquires(&buddy_cq->hwq.lock) | ||
| 130 | { | ||
| 131 | struct bnxt_qplib_cq *buddy_cq = NULL; | ||
| 132 | |||
| 133 | buddy_cq = bnxt_qplib_find_buddy_cq(qp, cq); | ||
| 134 | if (!buddy_cq) | ||
| 135 | __acquire(&cq->hwq.lock); | ||
| 136 | else | ||
| 137 | spin_lock(&buddy_cq->hwq.lock); | ||
| 138 | } | ||
| 139 | |||
| 140 | static void bnxt_qplib_unlock_buddy_cq(struct bnxt_qplib_qp *qp, | ||
| 141 | struct bnxt_qplib_cq *cq) | ||
| 142 | __releases(&buddy_cq->hwq.lock) | ||
| 143 | { | ||
| 144 | struct bnxt_qplib_cq *buddy_cq = NULL; | ||
| 145 | |||
| 146 | buddy_cq = bnxt_qplib_find_buddy_cq(qp, cq); | ||
| 147 | if (!buddy_cq) | ||
| 148 | __release(&cq->hwq.lock); | ||
| 149 | else | ||
| 150 | spin_unlock(&buddy_cq->hwq.lock); | ||
| 151 | } | 111 | } |
| 152 | 112 | ||
| 153 | void bnxt_qplib_add_flush_qp(struct bnxt_qplib_qp *qp) | 113 | void bnxt_qplib_add_flush_qp(struct bnxt_qplib_qp *qp) |
| 154 | { | 114 | { |
| 155 | unsigned long flags; | 115 | unsigned long flags; |
| 156 | 116 | ||
| 157 | bnxt_qplib_acquire_cq_locks(qp, &flags); | 117 | bnxt_qplib_acquire_cq_flush_locks(qp, &flags); |
| 158 | __bnxt_qplib_add_flush_qp(qp); | 118 | __bnxt_qplib_add_flush_qp(qp); |
| 159 | bnxt_qplib_release_cq_locks(qp, &flags); | 119 | bnxt_qplib_release_cq_flush_locks(qp, &flags); |
| 160 | } | 120 | } |
| 161 | 121 | ||
| 162 | static void __bnxt_qplib_del_flush_qp(struct bnxt_qplib_qp *qp) | 122 | static void __bnxt_qplib_del_flush_qp(struct bnxt_qplib_qp *qp) |
| @@ -173,11 +133,11 @@ static void __bnxt_qplib_del_flush_qp(struct bnxt_qplib_qp *qp) · 10 unmodified lines | |||
| 173 | } | 133 | } |
| 174 | } | 134 | } |
| 175 | 135 | ||
| 176 | void bnxt_qplib_del_flush_qp(struct bnxt_qplib_qp *qp) | 136 | void bnxt_qplib_clean_qp(struct bnxt_qplib_qp *qp) |
| 177 | { | 137 | { |
| 178 | unsigned long flags; | 138 | unsigned long flags; |
| 179 | 139 | ||
| 180 | bnxt_qplib_acquire_cq_locks(qp, &flags); | 140 | bnxt_qplib_acquire_cq_flush_locks(qp, &flags); |
| 181 | __clean_cq(qp->scq, (u64)(unsigned long)qp); | 141 | __clean_cq(qp->scq, (u64)(unsigned long)qp); |
| 182 | qp->sq.hwq.prod = 0; | 142 | qp->sq.hwq.prod = 0; |
| 183 | qp->sq.hwq.cons = 0; | 143 | qp->sq.hwq.cons = 0; |
| @@ -186,7 +146,7 @@ void bnxt_qplib_del_flush_qp(struct bnxt_qplib_qp *qp) · 2 unmodified lines | |||
| 186 | qp->rq.hwq.cons = 0; | 146 | qp->rq.hwq.cons = 0; |
| 187 | 147 | ||
| 188 | __bnxt_qplib_del_flush_qp(qp); | 148 | __bnxt_qplib_del_flush_qp(qp); |
| 189 | bnxt_qplib_release_cq_locks(qp, &flags); | 149 | bnxt_qplib_release_cq_flush_locks(qp, &flags); |
| 190 | } | 150 | } |
| 191 | 151 | ||
| 192 | static void bnxt_qpn_cqn_sched_task(struct work_struct *work) | 152 | static void bnxt_qpn_cqn_sched_task(struct work_struct *work) |
| @@ -1419,7 +1379,6 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, · 1226 unmodified lines | |||
| 1419 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; | 1379 | struct bnxt_qplib_rcfw *rcfw = res->rcfw; |
| 1420 | struct cmdq_destroy_qp req; | 1380 | struct cmdq_destroy_qp req; |
| 1421 | struct creq_destroy_qp_resp resp; | 1381 | struct creq_destroy_qp_resp resp; |
| 1422 | unsigned long flags; | ||
| 1423 | u16 cmd_flags = 0; | 1382 | u16 cmd_flags = 0; |
| 1424 | int rc; | 1383 | int rc; |
| 1425 | 1384 | ||
| @@ -1437,19 +1396,12 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, · 11 unmodified lines | |||
| 1437 | return rc; | 1396 | return rc; |
| 1438 | } | 1397 | } |
| 1439 | 1398 | ||
| 1440 | /* Must walk the associated CQs to nullified the QP ptr */ | 1399 | return 0; |
| 1441 | spin_lock_irqsave(&qp->scq->hwq.lock, flags); | 1400 | } |
| 1442 | |||
| 1443 | __clean_cq(qp->scq, (u64)(unsigned long)qp); | ||
| 1444 | |||
| 1445 | if (qp->rcq && qp->rcq != qp->scq) { | ||
| 1446 | spin_lock(&qp->rcq->hwq.lock); | ||
| 1447 | __clean_cq(qp->rcq, (u64)(unsigned long)qp); | ||
| 1448 | spin_unlock(&qp->rcq->hwq.lock); | ||
| 1449 | } | ||
| 1450 | |||
| 1451 | spin_unlock_irqrestore(&qp->scq->hwq.lock, flags); | ||
| 1452 | 1401 | ||
| 1402 | void bnxt_qplib_free_qp_res(struct bnxt_qplib_res *res, | ||
| 1403 | struct bnxt_qplib_qp *qp) | ||
| 1404 | { | ||
| 1453 | bnxt_qplib_free_qp_hdr_buf(res, qp); | 1405 | bnxt_qplib_free_qp_hdr_buf(res, qp); |
| 1454 | bnxt_qplib_free_hwq(res->pdev, &qp->sq.hwq); | 1406 | bnxt_qplib_free_hwq(res->pdev, &qp->sq.hwq); |
| 1455 | kfree(qp->sq.swq); | 1407 | kfree(qp->sq.swq); |
| @@ -1462,7 +1414,6 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, · 6 unmodified lines | |||
| 1462 | if (qp->orrq.max_elements) | 1414 | if (qp->orrq.max_elements) |
| 1463 | bnxt_qplib_free_hwq(res->pdev, &qp->orrq); | 1415 | bnxt_qplib_free_hwq(res->pdev, &qp->orrq); |
| 1464 | 1416 | ||
| 1465 | return 0; | ||
| 1466 | } | 1417 | } |
| 1467 | 1418 | ||
| 1468 | void *bnxt_qplib_get_qp1_sq_buf(struct bnxt_qplib_qp *qp, | 1419 | void *bnxt_qplib_get_qp1_sq_buf(struct bnxt_qplib_qp *qp, |
| @@ -2116,9 +2067,6 @@ void bnxt_qplib_mark_qp_error(void *qp_handle) · 647 unmodified lines | |||
| 2116 | /* Must block new posting of SQ and RQ */ | 2067 | /* Must block new posting of SQ and RQ */ |
| 2117 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; | 2068 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; |
| 2118 | bnxt_qplib_cancel_phantom_processing(qp); | 2069 | bnxt_qplib_cancel_phantom_processing(qp); |
| 2119 | |||
| 2120 | /* Add qp to flush list of the CQ */ | ||
| 2121 | __bnxt_qplib_add_flush_qp(qp); | ||
| 2122 | } | 2070 | } |
| 2123 | 2071 | ||
| 2124 | /* Note: SQE is valid from sw_sq_cons up to cqe_sq_cons (exclusive) | 2072 | /* Note: SQE is valid from sw_sq_cons up to cqe_sq_cons (exclusive) |
| @@ -2294,9 +2242,9 @@ static int bnxt_qplib_cq_process_req(struct bnxt_qplib_cq *cq, · 169 unmodified lines | |||
| 2294 | sw_sq_cons, cqe->wr_id, cqe->status); | 2242 | sw_sq_cons, cqe->wr_id, cqe->status); |
| 2295 | cqe++; | 2243 | cqe++; |
| 2296 | (*budget)--; | 2244 | (*budget)--; |
| 2297 | bnxt_qplib_lock_buddy_cq(qp, cq); | ||
| 2298 | bnxt_qplib_mark_qp_error(qp); | 2245 | bnxt_qplib_mark_qp_error(qp); |
| 2299 | bnxt_qplib_unlock_buddy_cq(qp, cq); | 2246 | /* Add qp to flush list of the CQ */ |
| 2247 | bnxt_qplib_add_flush_qp(qp); | ||
| 2300 | } else { | 2248 | } else { |
| 2301 | if (swq->flags & SQ_SEND_FLAGS_SIGNAL_COMP) { | 2249 | if (swq->flags & SQ_SEND_FLAGS_SIGNAL_COMP) { |
| 2302 | /* Before we complete, do WA 9060 */ | 2250 | /* Before we complete, do WA 9060 */ |
| @@ -2412,9 +2360,7 @@ static int bnxt_qplib_cq_process_res_rc(struct bnxt_qplib_cq *cq, · 109 unmodified lines | |||
| 2412 | if (hwcqe->status != CQ_RES_RC_STATUS_OK) { | 2360 | if (hwcqe->status != CQ_RES_RC_STATUS_OK) { |
| 2413 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; | 2361 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; |
| 2414 | /* Add qp to flush list of the CQ */ | 2362 | /* Add qp to flush list of the CQ */ |
| 2415 | bnxt_qplib_lock_buddy_cq(qp, cq); | 2363 | bnxt_qplib_add_flush_qp(qp); |
| 2416 | __bnxt_qplib_add_flush_qp(qp); | ||
| 2417 | bnxt_qplib_unlock_buddy_cq(qp, cq); | ||
| 2418 | } | 2364 | } |
| 2419 | } | 2365 | } |
| 2420 | 2366 | ||
| @@ -2498,9 +2444,7 @@ static int bnxt_qplib_cq_process_res_ud(struct bnxt_qplib_cq *cq, · 77 unmodified lines | |||
| 2498 | if (hwcqe->status != CQ_RES_RC_STATUS_OK) { | 2444 | if (hwcqe->status != CQ_RES_RC_STATUS_OK) { |
| 2499 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; | 2445 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; |
| 2500 | /* Add qp to flush list of the CQ */ | 2446 | /* Add qp to flush list of the CQ */ |
| 2501 | bnxt_qplib_lock_buddy_cq(qp, cq); | 2447 | bnxt_qplib_add_flush_qp(qp); |
| 2502 | __bnxt_qplib_add_flush_qp(qp); | ||
| 2503 | bnxt_qplib_unlock_buddy_cq(qp, cq); | ||
| 2504 | } | 2448 | } |
| 2505 | } | 2449 | } |
| 2506 | done: | 2450 | done: |
| @@ -2510,11 +2454,9 @@ done: · 3 unmodified lines | |||
| 2510 | bool bnxt_qplib_is_cq_empty(struct bnxt_qplib_cq *cq) | 2454 | bool bnxt_qplib_is_cq_empty(struct bnxt_qplib_cq *cq) |
| 2511 | { | 2455 | { |
| 2512 | struct cq_base *hw_cqe, **hw_cqe_ptr; | 2456 | struct cq_base *hw_cqe, **hw_cqe_ptr; |
| 2513 | unsigned long flags; | ||
| 2514 | u32 sw_cons, raw_cons; | 2457 | u32 sw_cons, raw_cons; |
| 2515 | bool rc = true; | 2458 | bool rc = true; |
| 2516 | 2459 | ||
| 2517 | spin_lock_irqsave(&cq->hwq.lock, flags); | ||
| 2518 | raw_cons = cq->hwq.cons; | 2460 | raw_cons = cq->hwq.cons; |
| 2519 | sw_cons = HWQ_CMP(raw_cons, &cq->hwq); | 2461 | sw_cons = HWQ_CMP(raw_cons, &cq->hwq); |
| 2520 | hw_cqe_ptr = (struct cq_base **)cq->hwq.pbl_ptr; | 2462 | hw_cqe_ptr = (struct cq_base **)cq->hwq.pbl_ptr; |
| @@ -2522,7 +2464,6 @@ bool bnxt_qplib_is_cq_empty(struct bnxt_qplib_cq *cq) · 1 unmodified line | |||
| 2522 | 2464 | ||
| 2523 | /* Check for Valid bit. If the CQE is valid, return false */ | 2465 | /* Check for Valid bit. If the CQE is valid, return false */ |
| 2524 | rc = !CQE_CMP_VALID(hw_cqe, raw_cons, cq->hwq.max_elements); | 2466 | rc = !CQE_CMP_VALID(hw_cqe, raw_cons, cq->hwq.max_elements); |
| 2525 | spin_unlock_irqrestore(&cq->hwq.lock, flags); | ||
| 2526 | return rc; | 2467 | return rc; |
| 2527 | } | 2468 | } |
| 2528 | 2469 | ||
| @@ -2611,9 +2552,7 @@ static int bnxt_qplib_cq_process_res_raweth_qp1(struct bnxt_qplib_cq *cq, · 82 unmodified lines | |||
| 2611 | if (hwcqe->status != CQ_RES_RC_STATUS_OK) { | 2552 | if (hwcqe->status != CQ_RES_RC_STATUS_OK) { |
| 2612 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; | 2553 | qp->state = CMDQ_MODIFY_QP_NEW_STATE_ERR; |
| 2613 | /* Add qp to flush list of the CQ */ | 2554 | /* Add qp to flush list of the CQ */ |
| 2614 | bnxt_qplib_lock_buddy_cq(qp, cq); | 2555 | bnxt_qplib_add_flush_qp(qp); |
| 2615 | __bnxt_qplib_add_flush_qp(qp); | ||
| 2616 | bnxt_qplib_unlock_buddy_cq(qp, cq); | ||
| 2617 | } | 2556 | } |
| 2618 | } | 2557 | } |
| 2619 | 2558 | ||
| @@ -2728,9 +2667,7 @@ do_rq: · 108 unmodified lines | |||
| 2728 | */ | 2667 | */ |
| 2729 | 2668 | ||
| 2730 | /* Add qp to flush list of the CQ */ | 2669 | /* Add qp to flush list of the CQ */ |
| 2731 | bnxt_qplib_lock_buddy_cq(qp, cq); | 2670 | bnxt_qplib_add_flush_qp(qp); |
| 2732 | __bnxt_qplib_add_flush_qp(qp); | ||
| 2733 | bnxt_qplib_unlock_buddy_cq(qp, cq); | ||
| 2734 | done: | 2671 | done: |
| 2735 | return rc; | 2672 | return rc; |
| 2736 | } | 2673 | } |
| @@ -2759,7 +2696,7 @@ int bnxt_qplib_process_flush_list(struct bnxt_qplib_cq *cq, · 22 unmodified lines | |||
| 2759 | u32 budget = num_cqes; | 2696 | u32 budget = num_cqes; |
| 2760 | unsigned long flags; | 2697 | unsigned long flags; |
| 2761 | 2698 | ||
| 2762 | spin_lock_irqsave(&cq->hwq.lock, flags); | 2699 | spin_lock_irqsave(&cq->flush_lock, flags); |
| 2763 | list_for_each_entry(qp, &cq->sqf_head, sq_flush) { | 2700 | list_for_each_entry(qp, &cq->sqf_head, sq_flush) { |
| 2764 | dev_dbg(&cq->hwq.pdev->dev, | 2701 | dev_dbg(&cq->hwq.pdev->dev, |
| 2765 | "QPLIB: FP: Flushing SQ QP= %p", | 2702 | "QPLIB: FP: Flushing SQ QP= %p", |
| @@ -2773,7 +2710,7 @@ int bnxt_qplib_process_flush_list(struct bnxt_qplib_cq *cq, · 7 unmodified lines | |||
| 2773 | qp); | 2710 | qp); |
| 2774 | __flush_rq(&qp->rq, qp, &cqe, &budget); | 2711 | __flush_rq(&qp->rq, qp, &cqe, &budget); |
| 2775 | } | 2712 | } |
| 2776 | spin_unlock_irqrestore(&cq->hwq.lock, flags); | 2713 | spin_unlock_irqrestore(&cq->flush_lock, flags); |
| 2777 | 2714 | ||
| 2778 | return num_cqes - budget; | 2715 | return num_cqes - budget; |
| 2779 | } | 2716 | } |
| @@ -2782,11 +2719,9 @@ int bnxt_qplib_poll_cq(struct bnxt_qplib_cq *cq, struct bnxt_qplib_cqe *cqe, · 2 unmodified lines | |||
| 2782 | int num_cqes, struct bnxt_qplib_qp **lib_qp) | 2719 | int num_cqes, struct bnxt_qplib_qp **lib_qp) |
| 2783 | { | 2720 | { |
| 2784 | struct cq_base *hw_cqe, **hw_cqe_ptr; | 2721 | struct cq_base *hw_cqe, **hw_cqe_ptr; |
| 2785 | unsigned long flags; | ||
| 2786 | u32 sw_cons, raw_cons; | 2722 | u32 sw_cons, raw_cons; |
| 2787 | int budget, rc = 0; | 2723 | int budget, rc = 0; |
| 2788 | 2724 | ||
| 2789 | spin_lock_irqsave(&cq->hwq.lock, flags); | ||
| 2790 | raw_cons = cq->hwq.cons; | 2725 | raw_cons = cq->hwq.cons; |
| 2791 | budget = num_cqes; | 2726 | budget = num_cqes; |
| 2792 | 2727 | ||
| @@ -2862,20 +2797,15 @@ int bnxt_qplib_poll_cq(struct bnxt_qplib_cq *cq, struct bnxt_qplib_cqe *cqe, · 69 unmodified lines | |||
| 2862 | bnxt_qplib_arm_cq(cq, DBR_DBR_TYPE_CQ); | 2797 | bnxt_qplib_arm_cq(cq, DBR_DBR_TYPE_CQ); |
| 2863 | } | 2798 | } |
| 2864 | exit: | 2799 | exit: |
| 2865 | spin_unlock_irqrestore(&cq->hwq.lock, flags); | ||
| 2866 | return num_cqes - budget; | 2800 | return num_cqes - budget; |
| 2867 | } | 2801 | } |
| 2868 | 2802 | ||
| 2869 | void bnxt_qplib_req_notify_cq(struct bnxt_qplib_cq *cq, u32 arm_type) | 2803 | void bnxt_qplib_req_notify_cq(struct bnxt_qplib_cq *cq, u32 arm_type) |
| 2870 | { | 2804 | { |
| 2871 | unsigned long flags; | ||
| 2872 | |||
| 2873 | spin_lock_irqsave(&cq->hwq.lock, flags); | ||
| 2874 | if (arm_type) | 2805 | if (arm_type) |
| 2875 | bnxt_qplib_arm_cq(cq, arm_type); | 2806 | bnxt_qplib_arm_cq(cq, arm_type); |
| 2876 | /* Using cq->arm_state variable to track whether to issue cq handler */ | 2807 | /* Using cq->arm_state variable to track whether to issue cq handler */ |
| 2877 | atomic_set(&cq->arm_state, 1); | 2808 | atomic_set(&cq->arm_state, 1); |
| 2878 | spin_unlock_irqrestore(&cq->hwq.lock, flags); | ||
| 2879 | } | 2809 | } |
| 2880 | 2810 | ||
| 2881 | void bnxt_qplib_flush_cqn_wq(struct bnxt_qplib_qp *qp) | 2811 | void bnxt_qplib_flush_cqn_wq(struct bnxt_qplib_qp *qp) |
| @@ -389,6 +389,18 @@ struct bnxt_qplib_cq { · 388 unmodified lines | |||
| 389 | struct list_head sqf_head, rqf_head; | 389 | struct list_head sqf_head, rqf_head; |
| 390 | atomic_t arm_state; | 390 | atomic_t arm_state; |
| 391 | spinlock_t compl_lock; /* synch CQ handlers */ | 391 | spinlock_t compl_lock; /* synch CQ handlers */ |
| 392 | /* Locking Notes: | ||
| 393 | * QP can move to error state from modify_qp, async error event or error | ||
| 394 | * CQE as part of poll_cq. When QP is moved to error state, it gets added | ||
| 395 | * to two flush lists, one each for SQ and RQ. | ||
| 396 | * Each flush list is protected by qplib_cq->flush_lock. Both scq and rcq | ||
| 397 | * flush_locks should be acquired when QP is moved to error. The control path | ||
| 398 | * operations(modify_qp and async error events) are synchronized with poll_cq | ||
| 399 | * using upper level CQ locks (bnxt_re_cq->cq_lock) of both SCQ and RCQ. | ||
| 400 | * The qplib_cq->flush_lock is required to synchronize two instances of poll_cq | ||
| 401 | * of the same QP while manipulating the flush list. | ||
| 402 | */ | ||
| 403 | spinlock_t flush_lock; /* QP flush management */ | ||
| 392 | }; | 404 | }; |
| 393 | 405 | ||
| 394 | #define BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE sizeof(struct xrrq_irrq) | 406 | #define BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE sizeof(struct xrrq_irrq) |
| @@ -478,6 +490,9 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); · 83 unmodified lines | |||
| 478 | int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); | 490 | int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); |
| 479 | int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); | 491 | int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); |
| 480 | int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); | 492 | int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); |
| 493 | void bnxt_qplib_clean_qp(struct bnxt_qplib_qp *qp); | ||
| 494 | void bnxt_qplib_free_qp_res(struct bnxt_qplib_res *res, | ||
| 495 | struct bnxt_qplib_qp *qp); | ||
| 481 | void *bnxt_qplib_get_qp1_sq_buf(struct bnxt_qplib_qp *qp, | 496 | void *bnxt_qplib_get_qp1_sq_buf(struct bnxt_qplib_qp *qp, |
| 482 | struct bnxt_qplib_sge *sge); | 497 | struct bnxt_qplib_sge *sge); |
| 483 | void *bnxt_qplib_get_qp1_rq_buf(struct bnxt_qplib_qp *qp, | 498 | void *bnxt_qplib_get_qp1_rq_buf(struct bnxt_qplib_qp *qp, |
| @@ -500,7 +515,6 @@ void bnxt_qplib_req_notify_cq(struct bnxt_qplib_cq *cq, u32 arm_type); · 16 unmodified lines | |||
| 500 | void bnxt_qplib_free_nq(struct bnxt_qplib_nq *nq); | 515 | void bnxt_qplib_free_nq(struct bnxt_qplib_nq *nq); |
| 501 | int bnxt_qplib_alloc_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq); | 516 | int bnxt_qplib_alloc_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq); |
| 502 | void bnxt_qplib_add_flush_qp(struct bnxt_qplib_qp *qp); | 517 | void bnxt_qplib_add_flush_qp(struct bnxt_qplib_qp *qp); |
| 503 | void bnxt_qplib_del_flush_qp(struct bnxt_qplib_qp *qp); | ||
| 504 | void bnxt_qplib_acquire_cq_locks(struct bnxt_qplib_qp *qp, | 518 | void bnxt_qplib_acquire_cq_locks(struct bnxt_qplib_qp *qp, |
| 505 | unsigned long *flags); | 519 | unsigned long *flags); |
| 506 | void bnxt_qplib_release_cq_locks(struct bnxt_qplib_qp *qp, | 520 | void bnxt_qplib_release_cq_locks(struct bnxt_qplib_qp *qp, |
| @@ -305,9 +305,8 @@ static int bnxt_qplib_process_qp_event(struct bnxt_qplib_rcfw *rcfw, · 304 unmodified lines | |||
| 305 | err_event->res_err_state_reason); | 305 | err_event->res_err_state_reason); |
| 306 | if (!qp) | 306 | if (!qp) |
| 307 | break; | 307 | break; |
| 308 | bnxt_qplib_acquire_cq_locks(qp, &flags); | ||
| 309 | bnxt_qplib_mark_qp_error(qp); | 308 | bnxt_qplib_mark_qp_error(qp); |
| 310 | bnxt_qplib_release_cq_locks(qp, &flags); | 309 | rcfw->aeq_handler(rcfw, qp_event, qp); |
| 311 | break; | 310 | break; |
| 312 | default: | 311 | default: |
| 313 | /* Command Response */ | 312 | /* Command Response */ |
| @@ -460,7 +459,11 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw, · 146 unmodified lines | |||
| 460 | int rc; | 459 | int rc; |
| 461 | 460 | ||
| 462 | RCFW_CMD_PREP(req, INITIALIZE_FW, cmd_flags); | 461 | RCFW_CMD_PREP(req, INITIALIZE_FW, cmd_flags); |
| 463 | 462 | /* Supply (log-base-2-of-host-page-size - base-page-shift) | |
| 463 | * to bono to adjust the doorbell page sizes. | ||
| 464 | */ | ||
| 465 | req.log2_dbr_pg_size = cpu_to_le16(PAGE_SHIFT - | ||
| 466 | RCFW_DBR_BASE_PAGE_SHIFT); | ||
| 464 | /* | 467 | /* |
| 465 | * VFs need not setup the HW context area, PF | 468 | * VFs need not setup the HW context area, PF |
| 466 | * shall setup this area for VF. Skipping the | 469 | * shall setup this area for VF. Skipping the |
| @@ -49,6 +49,7 @@ · 48 unmodified lines | |||
| 49 | #define RCFW_COMM_SIZE 0x104 | 49 | #define RCFW_COMM_SIZE 0x104 |
| 50 | 50 | ||
| 51 | #define RCFW_DBR_PCI_BAR_REGION 2 | 51 | #define RCFW_DBR_PCI_BAR_REGION 2 |
| 52 | #define RCFW_DBR_BASE_PAGE_SHIFT 12 | ||
| 52 | 53 | ||
| 53 | #define RCFW_CMD_PREP(req, CMD, cmd_flags) \ | 54 | #define RCFW_CMD_PREP(req, CMD, cmd_flags) \ |
| 54 | do { \ | 55 | do { \ |
| @@ -52,18 +52,6 @@ const struct bnxt_qplib_gid bnxt_qplib_gid_zero = {{ 0, 0, 0, 0, 0, 0, 0, 0, · 51 unmodified lines | |||
| 52 | 52 | ||
| 53 | /* Device */ | 53 | /* Device */ |
| 54 | 54 | ||
| 55 | static bool bnxt_qplib_is_atomic_cap(struct bnxt_qplib_rcfw *rcfw) | ||
| 56 | { | ||
| 57 | int rc; | ||
| 58 | u16 pcie_ctl2; | ||
| 59 | |||
| 60 | rc = pcie_capability_read_word(rcfw->pdev, PCI_EXP_DEVCTL2, | ||
| 61 | &pcie_ctl2); | ||
| 62 | if (rc) | ||
| 63 | return false; | ||
| 64 | return !!(pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ); | ||
| 65 | } | ||
| 66 | |||
| 67 | static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw, | 55 | static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw, |
| 68 | char *fw_ver) | 56 | char *fw_ver) |
| 69 | { | 57 | { |
| @@ -151,7 +139,8 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, · 81 unmodified lines | |||
| 151 | attr->max_pkey = le32_to_cpu(sb->max_pkeys); | 139 | attr->max_pkey = le32_to_cpu(sb->max_pkeys); |
| 152 | 140 | ||
| 153 | attr->max_inline_data = le32_to_cpu(sb->max_inline_data); | 141 | attr->max_inline_data = le32_to_cpu(sb->max_inline_data); |
| 154 | attr->l2_db_size = (sb->l2_db_space_size + 1) * PAGE_SIZE; | 142 | attr->l2_db_size = (sb->l2_db_space_size + 1) * |
| 143 | (0x01 << RCFW_DBR_BASE_PAGE_SHIFT); | ||
| 155 | attr->max_sgid = le32_to_cpu(sb->max_gid); | 144 | attr->max_sgid = le32_to_cpu(sb->max_gid); |
| 156 | 145 | ||
| 157 | bnxt_qplib_query_version(rcfw, attr->fw_ver); | 146 | bnxt_qplib_query_version(rcfw, attr->fw_ver); |
| @@ -165,7 +154,7 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, · 7 unmodified lines | |||
| 165 | attr->tqm_alloc_reqs[i * 4 + 3] = *(++tqm_alloc); | 154 | attr->tqm_alloc_reqs[i * 4 + 3] = *(++tqm_alloc); |
| 166 | } | 155 | } |
| 167 | 156 | ||
| 168 | attr->is_atomic = bnxt_qplib_is_atomic_cap(rcfw); | 157 | attr->is_atomic = 0; |
| 169 | bail: | 158 | bail: |
| 170 | bnxt_qplib_rcfw_free_sbuf(rcfw, sbuf); | 159 | bnxt_qplib_rcfw_free_sbuf(rcfw, sbuf); |
| 171 | return rc; | 160 | return rc; |
| @@ -1761,7 +1761,30 @@ struct cmdq_initialize_fw { · 1760 unmodified lines | |||
| 1761 | #define CMDQ_INITIALIZE_FW_TIM_PG_SIZE_PG_2M (0x3UL << 4) | 1761 | #define CMDQ_INITIALIZE_FW_TIM_PG_SIZE_PG_2M (0x3UL << 4) |
| 1762 | #define CMDQ_INITIALIZE_FW_TIM_PG_SIZE_PG_8M (0x4UL << 4) | 1762 | #define CMDQ_INITIALIZE_FW_TIM_PG_SIZE_PG_8M (0x4UL << 4) |
| 1763 | #define CMDQ_INITIALIZE_FW_TIM_PG_SIZE_PG_1G (0x5UL << 4) | 1763 | #define CMDQ_INITIALIZE_FW_TIM_PG_SIZE_PG_1G (0x5UL << 4) |
| 1764 | __le16 reserved16; | 1764 | /* This value is (log-base-2-of-DBR-page-size - 12). |
| 1765 | * 0 for 4KB. HW supported values are enumerated below. | ||
| 1766 | */ | ||
| 1767 | __le16 log2_dbr_pg_size; | ||
| 1768 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_MASK 0xfUL | ||
| 1769 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_SFT 0 | ||
| 1770 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_4K 0x0UL | ||
| 1771 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_8K 0x1UL | ||
| 1772 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_16K 0x2UL | ||
| 1773 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_32K 0x3UL | ||
| 1774 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_64K 0x4UL | ||
| 1775 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_128K 0x5UL | ||
| 1776 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_256K 0x6UL | ||
| 1777 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_512K 0x7UL | ||
| 1778 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_1M 0x8UL | ||
| 1779 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_2M 0x9UL | ||
| 1780 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_4M 0xaUL | ||
| 1781 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_8M 0xbUL | ||
| 1782 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_16M 0xcUL | ||
| 1783 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_32M 0xdUL | ||
| 1784 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_64M 0xeUL | ||
| 1785 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_128M 0xfUL | ||
| 1786 | #define CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_LAST \ | ||
| 1787 | CMDQ_INITIALIZE_FW_LOG2_DBR_PG_SIZE_PG_128M | ||
| 1765 | __le64 qpc_page_dir; | 1788 | __le64 qpc_page_dir; |
| 1766 | __le64 mrw_page_dir; | 1789 | __le64 mrw_page_dir; |
| 1767 | __le64 srq_page_dir; | 1790 | __le64 srq_page_dir; |
| @@ -601,6 +601,7 @@ static void use_tunnel_data(struct mlx4_ib_qp *qp, struct mlx4_ib_cq *cq, struct · 600 unmodified lines | |||
| 601 | wc->dlid_path_bits = 0; | 601 | wc->dlid_path_bits = 0; |
| 602 | 602 | ||
| 603 | if (is_eth) { | 603 | if (is_eth) { |
| 604 | wc->slid = 0; | ||
| 604 | wc->vlan_id = be16_to_cpu(hdr->tun.sl_vid); | 605 | wc->vlan_id = be16_to_cpu(hdr->tun.sl_vid); |
| 605 | memcpy(&(wc->smac[0]), (char *)&hdr->tun.mac_31_0, 4); | 606 | memcpy(&(wc->smac[0]), (char *)&hdr->tun.mac_31_0, 4); |
| 606 | memcpy(&(wc->smac[4]), (char *)&hdr->tun.slid_mac_47_32, 2); | 607 | memcpy(&(wc->smac[4]), (char *)&hdr->tun.slid_mac_47_32, 2); |
| @@ -851,7 +852,6 @@ repoll: · 244 unmodified lines | |||
| 851 | } | 852 | } |
| 852 | } | 853 | } |
| 853 | 854 | ||
| 854 | wc->slid = be16_to_cpu(cqe->rlid); | ||
| 855 | g_mlpath_rqpn = be32_to_cpu(cqe->g_mlpath_rqpn); | 855 | g_mlpath_rqpn = be32_to_cpu(cqe->g_mlpath_rqpn); |
| 856 | wc->src_qp = g_mlpath_rqpn & 0xffffff; | 856 | wc->src_qp = g_mlpath_rqpn & 0xffffff; |
| 857 | wc->dlid_path_bits = (g_mlpath_rqpn >> 24) & 0x7f; | 857 | wc->dlid_path_bits = (g_mlpath_rqpn >> 24) & 0x7f; |
| @@ -860,6 +860,7 @@ repoll: · 2 unmodified lines | |||
| 860 | wc->wc_flags |= mlx4_ib_ipoib_csum_ok(cqe->status, | 860 | wc->wc_flags |= mlx4_ib_ipoib_csum_ok(cqe->status, |
| 861 | cqe->checksum) ? IB_WC_IP_CSUM_OK : 0; | 861 | cqe->checksum) ? IB_WC_IP_CSUM_OK : 0; |
| 862 | if (is_eth) { | 862 | if (is_eth) { |
| 863 | wc->slid = 0; | ||
| 863 | wc->sl = be16_to_cpu(cqe->sl_vid) >> 13; | 864 | wc->sl = be16_to_cpu(cqe->sl_vid) >> 13; |
| 864 | if (be32_to_cpu(cqe->vlan_my_qpn) & | 865 | if (be32_to_cpu(cqe->vlan_my_qpn) & |
| 865 | MLX4_CQE_CVLAN_PRESENT_MASK) { | 866 | MLX4_CQE_CVLAN_PRESENT_MASK) { |
| @@ -871,6 +872,7 @@ repoll: · 5 unmodified lines | |||
| 871 | memcpy(wc->smac, cqe->smac, ETH_ALEN); | 872 | memcpy(wc->smac, cqe->smac, ETH_ALEN); |
| 872 | wc->wc_flags |= (IB_WC_WITH_VLAN | IB_WC_WITH_SMAC); | 873 | wc->wc_flags |= (IB_WC_WITH_VLAN | IB_WC_WITH_SMAC); |
| 873 | } else { | 874 | } else { |
| 875 | wc->slid = be16_to_cpu(cqe->rlid); | ||
| 874 | wc->sl = be16_to_cpu(cqe->sl_vid) >> 12; | 876 | wc->sl = be16_to_cpu(cqe->sl_vid) >> 12; |
| 875 | wc->vlan_id = 0xffff; | 877 | wc->vlan_id = 0xffff; |
| 876 | } | 878 | } |
| @@ -219,8 +219,6 @@ static int mlx4_ib_update_gids_v1_v2(struct gid_entry *gids, · 218 unmodified lines | |||
| 219 | gid_tbl[i].version = 2; | 219 | gid_tbl[i].version = 2; |
| 220 | if (!ipv6_addr_v4mapped((struct in6_addr *)&gids[i].gid)) | 220 | if (!ipv6_addr_v4mapped((struct in6_addr *)&gids[i].gid)) |
| 221 | gid_tbl[i].type = 1; | 221 | gid_tbl[i].type = 1; |
| 222 | else | ||
| 223 | memset(&gid_tbl[i].gid, 0, 12); | ||
| 224 | } | 222 | } |
| 225 | } | 223 | } |
| 226 | 224 | ||
| @@ -366,8 +364,13 @@ static int mlx4_ib_del_gid(struct ib_device *device, · 139 unmodified lines | |||
| 366 | if (!gids) { | 364 | if (!gids) { |
| 367 | ret = -ENOMEM; | 365 | ret = -ENOMEM; |
| 368 | } else { | 366 | } else { |
| 369 | for (i = 0; i < MLX4_MAX_PORT_GIDS; i++) | 367 | for (i = 0; i < MLX4_MAX_PORT_GIDS; i++) { |
| 370 | memcpy(&gids[i].gid, &port_gid_table->gids[i].gid, sizeof(union ib_gid)); | 368 | memcpy(&gids[i].gid, |
| 369 | &port_gid_table->gids[i].gid, | ||
| 370 | sizeof(union ib_gid)); | ||
| 371 | gids[i].gid_type = | ||
| 372 | port_gid_table->gids[i].gid_type; | ||
| 373 | } | ||
| 371 | } | 374 | } |
| 372 | } | 375 | } |
| 373 | spin_unlock_bh(&iboe->lock); | 376 | spin_unlock_bh(&iboe->lock); |
| @@ -226,7 +226,6 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe, · 225 unmodified lines | |||
| 226 | wc->ex.invalidate_rkey = be32_to_cpu(cqe->imm_inval_pkey); | 226 | wc->ex.invalidate_rkey = be32_to_cpu(cqe->imm_inval_pkey); |
| 227 | break; | 227 | break; |
| 228 | } | 228 | } |
| 229 | wc->slid = be16_to_cpu(cqe->slid); | ||
| 230 | wc->src_qp = be32_to_cpu(cqe->flags_rqpn) & 0xffffff; | 229 | wc->src_qp = be32_to_cpu(cqe->flags_rqpn) & 0xffffff; |
| 231 | wc->dlid_path_bits = cqe->ml_path; | 230 | wc->dlid_path_bits = cqe->ml_path; |
| 232 | g = (be32_to_cpu(cqe->flags_rqpn) >> 28) & 3; | 231 | g = (be32_to_cpu(cqe->flags_rqpn) >> 28) & 3; |
| @@ -241,10 +240,12 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe, · 8 unmodified lines | |||
| 241 | } | 240 | } |
| 242 | 241 | ||
| 243 | if (ll != IB_LINK_LAYER_ETHERNET) { | 242 | if (ll != IB_LINK_LAYER_ETHERNET) { |
| 243 | wc->slid = be16_to_cpu(cqe->slid); | ||
| 244 | wc->sl = (be32_to_cpu(cqe->flags_rqpn) >> 24) & 0xf; | 244 | wc->sl = (be32_to_cpu(cqe->flags_rqpn) >> 24) & 0xf; |
| 245 | return; | 245 | return; |
| 246 | } | 246 | } |
| 247 | 247 | ||
| 248 | wc->slid = 0; | ||
| 248 | vlan_present = cqe->l4_l3_hdr_type & 0x1; | 249 | vlan_present = cqe->l4_l3_hdr_type & 0x1; |
| 249 | roce_packet_type = (be32_to_cpu(cqe->flags_rqpn) >> 24) & 0x3; | 250 | roce_packet_type = (be32_to_cpu(cqe->flags_rqpn) >> 24) & 0x3; |
| 250 | if (vlan_present) { | 251 | if (vlan_present) { |
| @@ -1177,7 +1178,12 @@ static int resize_user(struct mlx5_ib_dev *dev, struct mlx5_ib_cq *cq, · 926 unmodified lines | |||
| 1177 | if (ucmd.reserved0 || ucmd.reserved1) | 1178 | if (ucmd.reserved0 || ucmd.reserved1) |
| 1178 | return -EINVAL; | 1179 | return -EINVAL; |
| 1179 | 1180 | ||
| 1180 | umem = ib_umem_get(context, ucmd.buf_addr, entries * ucmd.cqe_size, | 1181 | /* check multiplication overflow */ |
| 1182 | if (ucmd.cqe_size && SIZE_MAX / ucmd.cqe_size <= entries - 1) | ||
| 1183 | return -EINVAL; | ||
| 1184 | |||
| 1185 | umem = ib_umem_get(context, ucmd.buf_addr, | ||
| 1186 | (size_t)ucmd.cqe_size * entries, | ||
| 1181 | IB_ACCESS_LOCAL_WRITE, 1); | 1187 | IB_ACCESS_LOCAL_WRITE, 1); |
| 1182 | if (IS_ERR(umem)) { | 1188 | if (IS_ERR(umem)) { |
| 1183 | err = PTR_ERR(umem); | 1189 | err = PTR_ERR(umem); |
| @@ -245,12 +245,16 @@ struct mlx5_core_dev *mlx5_ib_get_native_port_mdev(struct mlx5_ib_dev *ibdev, · 244 unmodified lines | |||
| 245 | struct mlx5_ib_multiport_info *mpi; | 245 | struct mlx5_ib_multiport_info *mpi; |
| 246 | struct mlx5_ib_port *port; | 246 | struct mlx5_ib_port *port; |
| 247 | 247 | ||
| 248 | if (!mlx5_core_mp_enabled(ibdev->mdev) || | ||
| 249 | ll != IB_LINK_LAYER_ETHERNET) { | ||
| 250 | if (native_port_num) | ||
| 251 | *native_port_num = ib_port_num; | ||
| 252 | return ibdev->mdev; | ||
| 253 | } | ||
| 254 | |||
| 248 | if (native_port_num) | 255 | if (native_port_num) |
| 249 | *native_port_num = 1; | 256 | *native_port_num = 1; |
| 250 | 257 | ||
| 251 | if (!mlx5_core_mp_enabled(ibdev->mdev) || ll != IB_LINK_LAYER_ETHERNET) | ||
| 252 | return ibdev->mdev; | ||
| 253 | |||
| 254 | port = &ibdev->port[ib_port_num - 1]; | 258 | port = &ibdev->port[ib_port_num - 1]; |
| 255 | if (!port) | 259 | if (!port) |
| 256 | return NULL; | 260 | return NULL; |
| @@ -3263,7 +3267,7 @@ static void mlx5_ib_handle_event(struct work_struct *_work) · 3006 unmodified lines | |||
| 3263 | struct mlx5_ib_dev *ibdev; | 3267 | struct mlx5_ib_dev *ibdev; |
| 3264 | struct ib_event ibev; | 3268 | struct ib_event ibev; |
| 3265 | bool fatal = false; | 3269 | bool fatal = false; |
| 3266 | u8 port = 0; | 3270 | u8 port = (u8)work->param; |
| 3267 | 3271 | ||
| 3268 | if (mlx5_core_is_mp_slave(work->dev)) { | 3272 | if (mlx5_core_is_mp_slave(work->dev)) { |
| 3269 | ibdev = mlx5_ib_get_ibdev_from_mpi(work->context); | 3273 | ibdev = mlx5_ib_get_ibdev_from_mpi(work->context); |
| @@ -3283,8 +3287,6 @@ static void mlx5_ib_handle_event(struct work_struct *_work) · 13 unmodified lines | |||
| 3283 | case MLX5_DEV_EVENT_PORT_UP: | 3287 | case MLX5_DEV_EVENT_PORT_UP: |
| 3284 | case MLX5_DEV_EVENT_PORT_DOWN: | 3288 | case MLX5_DEV_EVENT_PORT_DOWN: |
| 3285 | case MLX5_DEV_EVENT_PORT_INITIALIZED: | 3289 | case MLX5_DEV_EVENT_PORT_INITIALIZED: |
| 3286 | port = (u8)work->param; | ||
| 3287 | |||
| 3288 | /* In RoCE, port up/down events are handled in | 3290 | /* In RoCE, port up/down events are handled in |
| 3289 | * mlx5_netdev_event(). | 3291 | * mlx5_netdev_event(). |
| 3290 | */ | 3292 | */ |
| @@ -3298,24 +3300,19 @@ static void mlx5_ib_handle_event(struct work_struct *_work) · 7 unmodified lines | |||
| 3298 | 3300 | ||
| 3299 | case MLX5_DEV_EVENT_LID_CHANGE: | 3301 | case MLX5_DEV_EVENT_LID_CHANGE: |
| 3300 | ibev.event = IB_EVENT_LID_CHANGE; | 3302 | ibev.event = IB_EVENT_LID_CHANGE; |
| 3301 | port = (u8)work->param; | ||
| 3302 | break; | 3303 | break; |
| 3303 | 3304 | ||
| 3304 | case MLX5_DEV_EVENT_PKEY_CHANGE: | 3305 | case MLX5_DEV_EVENT_PKEY_CHANGE: |
| 3305 | ibev.event = IB_EVENT_PKEY_CHANGE; | 3306 | ibev.event = IB_EVENT_PKEY_CHANGE; |
| 3306 | port = (u8)work->param; | ||
| 3307 | |||
| 3308 | schedule_work(&ibdev->devr.ports[port - 1].pkey_change_work); | 3307 | schedule_work(&ibdev->devr.ports[port - 1].pkey_change_work); |
| 3309 | break; | 3308 | break; |
| 3310 | 3309 | ||
| 3311 | case MLX5_DEV_EVENT_GUID_CHANGE: | 3310 | case MLX5_DEV_EVENT_GUID_CHANGE: |
| 3312 | ibev.event = IB_EVENT_GID_CHANGE; | 3311 | ibev.event = IB_EVENT_GID_CHANGE; |
| 3313 | port = (u8)work->param; | ||
| 3314 | break; | 3312 | break; |
| 3315 | 3313 | ||
| 3316 | case MLX5_DEV_EVENT_CLIENT_REREG: | 3314 | case MLX5_DEV_EVENT_CLIENT_REREG: |
| 3317 | ibev.event = IB_EVENT_CLIENT_REREGISTER; | 3315 | ibev.event = IB_EVENT_CLIENT_REREGISTER; |
| 3318 | port = (u8)work->param; | ||
| 3319 | break; | 3316 | break; |
| 3320 | case MLX5_DEV_EVENT_DELAY_DROP_TIMEOUT: | 3317 | case MLX5_DEV_EVENT_DELAY_DROP_TIMEOUT: |
| 3321 | schedule_work(&ibdev->delay_drop.delay_drop_work); | 3318 | schedule_work(&ibdev->delay_drop.delay_drop_work); |
| @@ -3327,7 +3324,7 @@ static void mlx5_ib_handle_event(struct work_struct *_work) · 5 unmodified lines | |||
| 3327 | ibev.device = &ibdev->ib_dev; | 3324 | ibev.device = &ibdev->ib_dev; |
| 3328 | ibev.element.port_num = port; | 3325 | ibev.element.port_num = port; |
| 3329 | 3326 | ||
| 3330 | if (port < 1 || port > ibdev->num_ports) { | 3327 | if (!rdma_is_port_valid(&ibdev->ib_dev, port)) { |
| 3331 | mlx5_ib_warn(ibdev, "warning: event on port %d\n", port); | 3328 | mlx5_ib_warn(ibdev, "warning: event on port %d\n", port); |
| 3332 | goto out; | 3329 | goto out; |
| 3333 | } | 3330 | } |
| @@ -1816,7 +1816,6 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr, · 1815 unmodified lines | |||
| 1816 | 1816 | ||
14 more lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 170 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 142 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
File renamed without changes.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 291 lines aren’t shown here.
This diff is large: its 265 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 250 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 105 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 173 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 158 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 166 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 127 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 257 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 382 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 255 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 223 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 240 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 248 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 575 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 454 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 134 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 232 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 267 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 350 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 257 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 178 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 226 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 413 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 949 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 265 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 391 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 221 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 232 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
Empty file added.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 2 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 206 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.