Sync drm-xe-next with drm-next to unblock some topic branches.
Unified · Split
| @@ -1,6 +1,7 @@ | ||
| 1 | 1 | Alan Cox <alan@lxorguk.ukuu.org.uk> |
| 2 | 2 | Alan Cox <root@hraefn.swansea.linux.org.uk> |
| 3 | 3 | Alyssa Rosenzweig <alyssa@rosenzweig.io> |
| 4 | +Askar Safin <safinaskar@gmail.com> | |
| 4 | 5 | Christoph Hellwig <hch@lst.de> |
| 5 | 6 | Jeff Kirsher <jeffrey.t.kirsher@intel.com> |
| 6 | 7 | Marc Gonzalez <marc.w.gonzalez@free.fr> |
| @@ -196,6 +196,7 @@ Christophe Leroy <chleroy@kernel.org> <christophe.leroy2@cs-soprasteria.com> · 195 unmodified lines | ||
| 196 | 196 | Christophe Ricard <christophe.ricard@gmail.com> |
| 197 | 197 | Christopher Obbard <christopher.obbard@linaro.org> <chris.obbard@collabora.com> |
| 198 | 198 | Christoph Hellwig <hch@lst.de> |
| 199 | +Christoph Manszewski <c.manszewski@gmail.com> <christoph.manszewski@intel.com> | |
| 199 | 200 | Chuck Lever <chuck.lever@oracle.com> <cel@kernel.org> |
| 200 | 201 | Chuck Lever <chuck.lever@oracle.com> <cel@netapp.com> |
| 201 | 202 | Chuck Lever <chuck.lever@oracle.com> <cel@citi.umich.edu> |
| @@ -316,6 +317,7 @@ Hans Verkuil <hverkuil@kernel.org> <hverkuil-cisco@xs4all.nl> · 114 unmodified lines | ||
| 316 | 317 | Hans Verkuil <hverkuil@kernel.org> <hansverk@cisco.com> |
| 317 | 318 | Hao Ge <hao.ge@linux.dev> <gehao@kylinos.cn> |
| 318 | 319 | Harry Yoo <harry.yoo@oracle.com> <42.hyeyoo@gmail.com> |
| 320 | +Harry Yoo <harry@kernel.org> <harry.yoo@oracle.com> | |
| 319 | 321 | Heiko Carstens <hca@linux.ibm.com> <h.carstens@de.ibm.com> |
| 320 | 322 | Heiko Carstens <hca@linux.ibm.com> <heiko.carstens@de.ibm.com> |
| 321 | 323 | Heiko Stuebner <heiko@sntech.de> <heiko.stuebner@bqreaders.com> |
| @@ -327,6 +329,7 @@ Henrik Rydberg <rydberg@bitmath.org> · 5 unmodified lines | ||
| 327 | 329 | Herbert Xu <herbert@gondor.apana.org.au> |
| 328 | 330 | Huacai Chen <chenhuacai@kernel.org> <chenhc@lemote.com> |
| 329 | 331 | Huacai Chen <chenhuacai@kernel.org> <chenhuacai@loongson.cn> |
| 332 | +Ignat Korchagin <ignat@linux.win> <ignat@cloudflare.com> | |
| 330 | 333 | Ike Panhc <ikepanhc@gmail.com> <ike.pan@canonical.com> |
| 331 | 334 | J. Bruce Fields <bfields@fieldses.org> <bfields@redhat.com> |
| 332 | 335 | J. Bruce Fields <bfields@fieldses.org> <bfields@citi.umich.edu> |
| @@ -498,7 +501,8 @@ Lior David <quic_liord@quicinc.com> <liord@codeaurora.org> · 165 unmodified lines | ||
| 498 | 501 | Loic Poulain <loic.poulain@oss.qualcomm.com> <loic.poulain@linaro.org> |
| 499 | 502 | Loic Poulain <loic.poulain@oss.qualcomm.com> <loic.poulain@intel.com> |
| 500 | 503 | Lorenzo Pieralisi <lpieralisi@kernel.org> <lorenzo.pieralisi@arm.com> |
| 501 | −Lorenzo Stoakes <lorenzo.stoakes@oracle.com> <lstoakes@gmail.com> | |
| 504 | +Lorenzo Stoakes <ljs@kernel.org> <lstoakes@gmail.com> | |
| 505 | +Lorenzo Stoakes <ljs@kernel.org> <lorenzo.stoakes@oracle.com> | |
| 502 | 506 | Luca Ceresoli <luca.ceresoli@bootlin.com> <luca@lucaceresoli.net> |
| 503 | 507 | Luca Weiss <luca@lucaweiss.eu> <luca@z3ntu.xyz> |
| 504 | 508 | Lucas De Marchi <demarchi@kernel.org> <lucas.demarchi@intel.com> |
| @@ -585,6 +589,7 @@ Morten Welinder <terra@gnome.org> · 80 unmodified lines | ||
| 585 | 589 | Morten Welinder <welinder@anemone.rentec.com> |
| 586 | 590 | Morten Welinder <welinder@darter.rentec.com> |
| 587 | 591 | Morten Welinder <welinder@troll.com> |
| 592 | +Muhammad Usama Anjum <usama.anjum@arm.com> <usama.anjum@collabora.com> | |
| 588 | 593 | Mukesh Ojha <quic_mojha@quicinc.com> <mojha@codeaurora.org> |
| 589 | 594 | Muna Sinada <quic_msinada@quicinc.com> <msinada@codeaurora.org> |
| 590 | 595 | Murali Nalajala <quic_mnalajal@quicinc.com> <mnalajal@codeaurora.org> |
| @@ -151,11 +151,11 @@ Description: · 150 unmodified lines | ||
| 151 | 151 | The algorithm_params file is write-only and is used to setup |
| 152 | 152 | compression algorithm parameters. |
| 153 | 153 | |
| 154 | −What: /sys/block/zram<id>/writeback_compressed | |
| 154 | +What: /sys/block/zram<id>/compressed_writeback | |
| 155 | 155 | Date: Decemeber 2025 |
| 156 | 156 | Contact: Richard Chang <richardycc@google.com> |
| 157 | 157 | Description: |
| 158 | − The writeback_compressed device atrribute toggles compressed | |
| 158 | + The compressed_writeback device atrribute toggles compressed | |
| 159 | 159 | writeback feature. |
| 160 | 160 | |
| 161 | 161 | What: /sys/block/zram<id>/writeback_batch_size |
| @@ -85,6 +85,16 @@ In the example, 'Requester ID' means the ID of the device that sent · 84 unmodified lines | ||
| 85 | 85 | the error message to the Root Port. Please refer to PCIe specs for other |
| 86 | 86 | fields. |
| 87 | 87 | |
| 88 | +The 'TLP Header' is the prefix/header of the TLP that caused the error | |
| 89 | +in raw hex format. To decode the TLP Header into human-readable form | |
| 90 | +one may use tlp-tool: | |
| 91 | + | |
| 92 | +https://github.com/mmpg-x86/tlp-tool | |
| 93 | + | |
| 94 | +Example usage:: | |
| 95 | + | |
| 96 | + curl -L https://git.kernel.org/linus/2ca1c94ce0b6 | rtlp-tool --aer | |
| 97 | + | |
| 88 | 98 | AER Ratelimits |
| 89 | 99 | -------------- |
| 90 | 100 | |
| @@ -216,7 +216,7 @@ writeback_limit WO specifies the maximum amount of write IO zram · 215 unmodified lines | ||
| 216 | 216 | writeback_limit_enable RW show and set writeback_limit feature |
| 217 | 217 | writeback_batch_size RW show and set maximum number of in-flight |
| 218 | 218 | writeback operations |
| 219 | −writeback_compressed RW show and set compressed writeback feature | |
| 219 | +compressed_writeback RW show and set compressed writeback feature | |
| 220 | 220 | comp_algorithm RW show and change the compression algorithm |
| 221 | 221 | algorithm_params WO setup compression algorithm parameters |
| 222 | 222 | compact WO trigger memory compaction |
| @@ -439,11 +439,11 @@ budget in next setting is user's job. · 216 unmodified lines | ||
| 439 | 439 | By default zram stores written back pages in decompressed (raw) form, which |
| 440 | 440 | means that writeback operation involves decompression of the page before |
| 441 | 441 | writing it to the backing device. This behavior can be changed by enabling |
| 442 | −`writeback_compressed` feature, which causes zram to write compressed pages | |
| 442 | +`compressed_writeback` feature, which causes zram to write compressed pages | |
| 443 | 443 | to the backing device, thus avoiding decompression overhead. To enable |
| 444 | 444 | this feature, execute:: |
| 445 | 445 | |
| 446 | − $ echo yes > /sys/block/zramX/writeback_compressed | |
| 446 | + $ echo yes > /sys/block/zramX/compressed_writeback | |
| 447 | 447 | |
| 448 | 448 | Note that this feature should be configured before the `zramX` device is |
| 449 | 449 | initialized. |
| @@ -8196,6 +8196,9 @@ Kernel parameters · 8195 unmodified lines | ||
| 8196 | 8196 | p = USB_QUIRK_SHORT_SET_ADDRESS_REQ_TIMEOUT |
| 8197 | 8197 | (Reduce timeout of the SET_ADDRESS |
| 8198 | 8198 | request from 5000 ms to 500 ms); |
| 8199 | + q = USB_QUIRK_FORCE_ONE_CONFIG (Device | |
| 8200 | + claims zero configurations, | |
| 8201 | + forcing to 1); | |
| 8199 | 8202 | Example: quirks=0781:5580:bk,0a5c:5834:gij |
| 8200 | 8203 | |
| 8201 | 8204 | usbhid.mousepoll= |
| @@ -149,11 +149,33 @@ For architectures that require cache flushing for DMA coherence · 148 unmodified lines | ||
| 149 | 149 | DMA_ATTR_MMIO will not perform any cache flushing. The address |
| 150 | 150 | provided must never be mapped cacheable into the CPU. |
| 151 | 151 | |
| 152 | −DMA_ATTR_CPU_CACHE_CLEAN | |
| 153 | −------------------------ | |
| 154 | − | |
| 155 | −This attribute indicates the CPU will not dirty any cacheline overlapping this | |
| 156 | −DMA_FROM_DEVICE/DMA_BIDIRECTIONAL buffer while it is mapped. This allows | |
| 157 | −multiple small buffers to safely share a cacheline without risk of data | |
| 158 | −corruption, suppressing DMA debug warnings about overlapping mappings. | |
| 159 | −All mappings sharing a cacheline should have this attribute. | |
| 152 | +DMA_ATTR_DEBUGGING_IGNORE_CACHELINES | |
| 153 | +------------------------------------ | |
| 154 | + | |
| 155 | +This attribute indicates that CPU cache lines may overlap for buffers mapped | |
| 156 | +with DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. | |
| 157 | + | |
| 158 | +Such overlap may occur when callers map multiple small buffers that reside | |
| 159 | +within the same cache line. In this case, callers must guarantee that the CPU | |
| 160 | +will not dirty these cache lines after the mappings are established. When this | |
| 161 | +condition is met, multiple buffers can safely share a cache line without risking | |
| 162 | +data corruption. | |
| 163 | + | |
| 164 | +All mappings that share a cache line must set this attribute to suppress DMA | |
| 165 | +debug warnings about overlapping mappings. | |
| 166 | + | |
| 167 | +DMA_ATTR_REQUIRE_COHERENT | |
| 168 | +------------------------- | |
| 169 | + | |
| 170 | +DMA mapping requests with the DMA_ATTR_REQUIRE_COHERENT fail on any | |
| 171 | +system where SWIOTLB or cache management is required. This should only | |
| 172 | +be used to support uAPI designs that require continuous HW DMA | |
| 173 | +coherence with userspace processes, for example RDMA and DRM. At a | |
| 174 | +minimum the memory being mapped must be userspace memory from | |
| 175 | +pin_user_pages() or similar. | |
| 176 | + | |
| 177 | +Drivers should consider using dma_mmap_pages() instead of this | |
| 178 | +interface when building their uAPIs, when possible. | |
| 179 | + | |
| 180 | +It must never be used in an in-kernel driver that only works with | |
| 181 | +kernel memory. | |
| @@ -336,6 +336,8 @@ command line arguments: · 335 unmodified lines | ||
| 336 | 336 | - ``--list_tests_attr``: If set, lists all tests that will be run and all of their |
| 337 | 337 | attributes. |
| 338 | 338 | |
| 339 | +- ``--list_suites``: If set, lists all suites that will be run. | |
| 340 | + | |
| 339 | 341 | Command-line completion |
| 340 | 342 | ============================== |
| 341 | 343 | |
| @@ -66,7 +66,7 @@ then: · 65 unmodified lines | ||
| 66 | 66 | required: |
| 67 | 67 | - refresh-rate-hz |
| 68 | 68 | |
| 69 | −additionalProperties: false | |
| 69 | +unevaluatedProperties: false | |
| 70 | 70 | |
| 71 | 71 | examples: |
| 72 | 72 | - | |
| @@ -301,6 +301,7 @@ properties: · 300 unmodified lines | ||
| 301 | 301 | maxItems: 4 |
| 302 | 302 | |
| 303 | 303 | dependencies: |
| 304 | + pd-disable: [typec-power-opmode] | |
| 304 | 305 | sink-vdos-v1: [ sink-vdos ] |
| 305 | 306 | sink-vdos: [ sink-vdos-v1 ] |
| 306 | 307 | |
| @@ -19,7 +19,9 @@ properties: · 18 unmodified lines | ||
| 19 | 19 | compatible: |
| 20 | 20 | oneOf: |
| 21 | 21 | - items: |
| 22 | − - const: arm,mali-d32 | |
| 22 | + - enum: | |
| 23 | + - arm,mali-d32 | |
| 24 | + - armchina,linlon-d6 | |
| 23 | 25 | - const: arm,mali-d71 |
| 24 | 26 | - const: arm,mali-d71 |
| 25 | 27 | |
| @@ -0,0 +1,113 @@ | ||
| 1 | +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | |
| 2 | +%YAML 1.2 | |
| 3 | +--- | |
| 4 | +$id: http://devicetree.org/schemas/display/bridge/lontium,lt8713sx.yaml# | |
| 5 | +$schema: http://devicetree.org/meta-schemas/core.yaml# | |
| 6 | + | |
| 7 | +title: Lontium LT8713SX Type-C/DP1.4 to Type-C/DP1.4/HDMI2.0/DP++ bridge-hub | |
| 8 | + | |
| 9 | +maintainers: | |
| 10 | + - Vishnu Saini <vishnu.saini@oss.qualcomm.com> | |
| 11 | + | |
| 12 | +description: | |
| 13 | + The Lontium LT8713SX is a Type-C/DP1.4 to Type-C/DP1.4/HDMI2.0 converter | |
| 14 | + that integrates one DP input and up to three configurable output interfaces | |
| 15 | + (DP1.4 / HDMI2.0 / DP++), with SST/MST functionality and audio support. | |
| 16 | + | |
| 17 | +properties: | |
| 18 | + compatible: | |
| 19 | + enum: | |
| 20 | + - lontium,lt8713sx | |
| 21 | + | |
| 22 | + reg: | |
| 23 | + maxItems: 1 | |
| 24 | + | |
| 25 | + vcc-supply: | |
| 26 | + description: Regulator for 3.3V vcc. | |
| 27 | + | |
| 28 | + vdd-supply: | |
| 29 | + description: Regulator for 1.1V vdd. | |
| 30 | + | |
| 31 | + reset-gpios: | |
| 32 | + description: GPIO connected to active low RESET pin. | |
| 33 | + | |
| 34 | + ports: | |
| 35 | + $ref: /schemas/graph.yaml#/properties/ports | |
| 36 | + | |
| 37 | + properties: | |
| 38 | + port@0: | |
| 39 | + $ref: /schemas/graph.yaml#/properties/port | |
| 40 | + description: | |
| 41 | + DP port for DP input from soc to bridge chip | |
| 42 | + | |
| 43 | + port@1: | |
| 44 | + $ref: /schemas/graph.yaml#/properties/port | |
| 45 | + description: | |
| 46 | + DP port for DP output from bridge | |
| 47 | + | |
| 48 | + port@2: | |
| 49 | + $ref: /schemas/graph.yaml#/properties/port | |
| 50 | + description: | |
| 51 | + Additional DP port for DP output from bridge | |
| 52 | + | |
| 53 | + port@3: | |
| 54 | + $ref: /schemas/graph.yaml#/properties/port | |
| 55 | + description: | |
| 56 | + Additional DP port for DP output from bridge | |
| 57 | + | |
| 58 | + required: | |
| 59 | + - port@0 | |
| 60 | + | |
| 61 | +required: | |
| 62 | + - compatible | |
| 63 | + - reg | |
| 64 | + - ports | |
| 65 | + | |
| 66 | +additionalProperties: false | |
| 67 | + | |
| 68 | +examples: | |
| 69 | + - | | |
| 70 | + #include <dt-bindings/gpio/gpio.h> | |
| 71 | + | |
| 72 | + i2c { | |
| 73 | + #address-cells = <1>; | |
| 74 | + #size-cells = <0>; | |
| 75 | + bridge@4f { | |
| 76 | + compatible = "lontium,lt8713sx"; | |
| 77 | + reg = <0x4f>; | |
| 78 | + reset-gpios = <&tlmm 6 GPIO_ACTIVE_LOW>; | |
| 79 | + | |
| 80 | + ports { | |
| 81 | + #address-cells = <1>; | |
| 82 | + #size-cells = <0>; | |
| 83 | + | |
| 84 | + port@0 { | |
| 85 | + reg = <0>; | |
| 86 | + lt8713sx_dp_in: endpoint { | |
| 87 | + remote-endpoint = <&mdss_dp0_out>; | |
| 88 | + }; | |
| 89 | + }; | |
| 90 | + | |
| 91 | + port@1 { | |
| 92 | + reg = <1>; | |
| 93 | + lt8713sx_dp0_out: endpoint { | |
| 94 | + remote-endpoint = <&dp0_connector_in>; | |
| 95 | + }; | |
| 96 | + }; | |
| 97 | + | |
| 98 | + port@2 { | |
| 99 | + reg = <2>; | |
| 100 | + lt8713sx_dp1_out: endpoint { | |
| 101 | + remote-endpoint = <&dp1_connector_in>; | |
| 102 | + }; | |
| 103 | + }; | |
| 104 | + | |
| 105 | + port@3 { | |
| 106 | + reg = <3>; | |
| 107 | + lt8713sx_dp2_out: endpoint { | |
| 108 | + remote-endpoint = <&dp2_connector_in>; | |
| 109 | + }; | |
| 110 | + }; | |
| 111 | + }; | |
| 112 | + }; | |
| 113 | + }; | |
| @@ -33,6 +33,7 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | oneOf: |
| 34 | 34 | - items: |
| 35 | 35 | - enum: |
| 36 | + - doestek,dtc34lm85am # For the Doestek DTC34LM85AM Flat Panel Display (FPD) Transmitter | |
| 36 | 37 | - onnn,fin3385 # OnSemi FIN3385 |
| 37 | 38 | - ti,ds90c185 # For the TI DS90C185 FPD-Link Serializer |
| 38 | 39 | - ti,ds90c187 # For the TI DS90C187 FPD-Link Serializer |
| @@ -33,6 +33,7 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | - enum: |
| 34 | 34 | - mediatek,mt2712-disp-aal |
| 35 | 35 | - mediatek,mt6795-disp-aal |
| 36 | + - mediatek,mt8167-disp-aal | |
| 36 | 37 | - const: mediatek,mt8173-disp-aal |
| 37 | 38 | - items: |
| 38 | 39 | - enum: |
| @@ -25,7 +25,9 @@ properties: · 24 unmodified lines | ||
| 25 | 25 | - mediatek,mt8183-disp-ccorr |
| 26 | 26 | - mediatek,mt8192-disp-ccorr |
| 27 | 27 | - items: |
| 28 | − - const: mediatek,mt8365-disp-ccorr | |
| 28 | + - enum: | |
| 29 | + - mediatek,mt8167-disp-ccorr | |
| 30 | + - mediatek,mt8365-disp-ccorr | |
| 29 | 31 | - const: mediatek,mt8183-disp-ccorr |
| 30 | 32 | - items: |
| 31 | 33 | - enum: |
| @@ -26,6 +26,7 @@ properties: · 25 unmodified lines | ||
| 26 | 26 | - mediatek,mt8183-disp-dither |
| 27 | 27 | - items: |
| 28 | 28 | - enum: |
| 29 | + - mediatek,mt8167-disp-dither | |
| 29 | 30 | - mediatek,mt8186-disp-dither |
| 30 | 31 | - mediatek,mt8188-disp-dither |
| 31 | 32 | - mediatek,mt8192-disp-dither |
| @@ -28,6 +28,7 @@ properties: · 27 unmodified lines | ||
| 28 | 28 | - items: |
| 29 | 29 | - enum: |
| 30 | 30 | - mediatek,mt6795-disp-gamma |
| 31 | + - mediatek,mt8167-disp-gamma | |
| 31 | 32 | - const: mediatek,mt8173-disp-gamma |
| 32 | 33 | - items: |
| 33 | 34 | - enum: |
| @@ -23,6 +23,7 @@ properties: · 22 unmodified lines | ||
| 23 | 23 | oneOf: |
| 24 | 24 | - enum: |
| 25 | 25 | - mediatek,mt2701-disp-ovl |
| 26 | + - mediatek,mt8167-disp-ovl | |
| 26 | 27 | - mediatek,mt8173-disp-ovl |
| 27 | 28 | - mediatek,mt8183-disp-ovl |
| 28 | 29 | - mediatek,mt8192-disp-ovl |
| @@ -36,6 +36,7 @@ properties: · 35 unmodified lines | ||
| 36 | 36 | - enum: |
| 37 | 37 | - mediatek,mt7623-disp-rdma |
| 38 | 38 | - mediatek,mt2712-disp-rdma |
| 39 | + - mediatek,mt8167-disp-rdma | |
| 39 | 40 | - const: mediatek,mt2701-disp-rdma |
| 40 | 41 | - items: |
| 41 | 42 | - enum: |
| @@ -24,7 +24,9 @@ properties: · 23 unmodified lines | ||
| 24 | 24 | - enum: |
| 25 | 25 | - mediatek,mt8173-disp-wdma |
| 26 | 26 | - items: |
| 27 | − - const: mediatek,mt6795-disp-wdma | |
| 27 | + - enum: | |
| 28 | + - mediatek,mt6795-disp-wdma | |
| 29 | + - mediatek,mt8167-disp-wdma | |
| 28 | 30 | - const: mediatek,mt8173-disp-wdma |
| 29 | 31 | |
| 30 | 32 | reg: |
| @@ -67,6 +67,7 @@ properties: · 66 unmodified lines | ||
| 67 | 67 | |
| 68 | 68 | - items: |
| 69 | 69 | - enum: |
| 70 | + - qcom,eliza-dp | |
| 70 | 71 | - qcom,sm8750-dp |
| 71 | 72 | - const: qcom,sm8650-dp |
| 72 | 73 | |
| @@ -253,7 +254,6 @@ allOf: · 180 unmodified lines | ||
| 253 | 254 | enum: |
| 254 | 255 | # these platforms support 2 streams MST on some interfaces, |
| 255 | 256 | # others are SST only |
| 256 | − - qcom,glymur-dp | |
| 257 | 257 | - qcom,sc8280xp-dp |
| 258 | 258 | - qcom,x1e80100-dp |
| 259 | 259 | then: |
| @@ -310,6 +310,26 @@ allOf: · 50 unmodified lines | ||
| 310 | 310 | minItems: 6 |
| 311 | 311 | maxItems: 8 |
| 312 | 312 | |
| 313 | + - if: | |
| 314 | + properties: | |
| 315 | + compatible: | |
| 316 | + contains: | |
| 317 | + enum: | |
| 318 | + # these platforms support 2 streams MST on some interfaces, | |
| 319 | + # others are SST only, but all controllers have 4 ports | |
| 320 | + - qcom,glymur-dp | |
| 321 | + then: | |
| 322 | + properties: | |
| 323 | + reg: | |
| 324 | + minItems: 9 | |
| 325 | + maxItems: 9 | |
| 326 | + clocks: | |
| 327 | + minItems: 5 | |
| 328 | + maxItems: 6 | |
| 329 | + clocks-names: | |
| 330 | + minItems: 5 | |
| 331 | + maxItems: 6 | |
| 332 | + | |
| 313 | 333 | unevaluatedProperties: false |
| 314 | 334 | |
| 315 | 335 | examples: |
| @@ -49,8 +49,13 @@ properties: · 48 unmodified lines | ||
| 49 | 49 | - items: |
| 50 | 50 | - enum: |
| 51 | 51 | - qcom,qcs8300-dsi-ctrl |
| 52 | + - qcom,sc8280xp-dsi-ctrl | |
| 52 | 53 | - const: qcom,sa8775p-dsi-ctrl |
| 53 | 54 | - const: qcom,mdss-dsi-ctrl |
| 55 | + - items: | |
| 56 | + - const: qcom,eliza-dsi-ctrl | |
| 57 | + - const: qcom,sm8750-dsi-ctrl | |
| 58 | + - const: qcom,mdss-dsi-ctrl | |
| 54 | 59 | - enum: |
| 55 | 60 | - qcom,dsi-ctrl-6g-qcm2290 |
| 56 | 61 | - qcom,mdss-dsi-ctrl # This should always come with an SoC-specific compatible |
| @@ -91,6 +91,7 @@ allOf: · 90 unmodified lines | ||
| 91 | 91 | compatible: |
| 92 | 92 | contains: |
| 93 | 93 | enum: |
| 94 | + - qcom,adreno-gmu-615.0 | |
| 94 | 95 | - qcom,adreno-gmu-618.0 |
| 95 | 96 | - qcom,adreno-gmu-630.2 |
| 96 | 97 | then: |
| @@ -440,13 +440,6 @@ allOf: · 439 unmodified lines | ||
| 440 | 440 | clocks: false |
| 441 | 441 | clock-names: false |
| 442 | 442 | |
| 443 | − reg-names: | |
| 444 | − minItems: 1 | |
| 445 | − items: | |
| 446 | − - const: kgsl_3d0_reg_memory | |
| 447 | − - const: cx_mem | |
| 448 | − - const: cx_dbgc | |
| 449 | − | |
| 450 | 443 | examples: |
| 451 | 444 | - | |
| 452 | 445 | |
| @@ -0,0 +1,494 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | |
| 2 | +%YAML 1.2 | |
| 3 | +--- | |
| 4 | +$id: http://devicetree.org/schemas/display/msm/qcom,eliza-mdss.yaml# | |
| 5 | +$schema: http://devicetree.org/meta-schemas/core.yaml# | |
| 6 | + | |
| 7 | +title: Qualcomm Eliza SoC Display MDSS | |
| 8 | + | |
| 9 | +maintainers: | |
| 10 | + - Krzysztof Kozlowski <krzk@kernel.org> | |
| 11 | + | |
| 12 | +description: | |
| 13 | + Eliza SoC Mobile Display Subsystem (MDSS) encapsulates sub-blocks like DPU | |
| 14 | + display controller, DSI and DP interfaces etc. | |
| 15 | + | |
| 16 | +$ref: /schemas/display/msm/mdss-common.yaml# | |
| 17 | + | |
| 18 | +properties: | |
| 19 | + compatible: | |
| 20 | + const: qcom,eliza-mdss | |
| 21 | + | |
| 22 | + clocks: | |
| 23 | + items: | |
| 24 | + - description: Display AHB | |
| 25 | + - description: Display hf AXI | |
| 26 | + - description: Display core | |
| 27 | + | |
| 28 | + iommus: | |
| 29 | + maxItems: 1 | |
| 30 | + | |
| 31 | + interconnects: | |
| 32 | + items: | |
| 33 | + - description: Interconnect path from mdp0 port to the data bus | |
| 34 | + - description: Interconnect path from CPU to the reg bus | |
| 35 | + | |
| 36 | + interconnect-names: | |
| 37 | + items: | |
| 38 | + - const: mdp0-mem | |
| 39 | + - const: cpu-cfg | |
| 40 | + | |
| 41 | +patternProperties: | |
| 42 | + "^display-controller@[0-9a-f]+$": | |
| 43 | + type: object | |
| 44 | + additionalProperties: true | |
| 45 | + properties: | |
| 46 | + compatible: | |
| 47 | + contains: | |
| 48 | + const: qcom,eliza-dpu | |
| 49 | + | |
| 50 | + "^displayport-controller@[0-9a-f]+$": | |
| 51 | + type: object | |
| 52 | + additionalProperties: true | |
| 53 | + properties: | |
| 54 | + compatible: | |
| 55 | + contains: | |
| 56 | + const: qcom,eliza-dp | |
| 57 | + | |
| 58 | + "^dsi@[0-9a-f]+$": | |
| 59 | + type: object | |
| 60 | + additionalProperties: true | |
| 61 | + properties: | |
| 62 | + compatible: | |
| 63 | + contains: | |
| 64 | + const: qcom,eliza-dsi-ctrl | |
| 65 | + | |
| 66 | + "^phy@[0-9a-f]+$": | |
| 67 | + type: object | |
| 68 | + additionalProperties: true | |
| 69 | + properties: | |
| 70 | + compatible: | |
| 71 | + contains: | |
| 72 | + const: qcom,eliza-dsi-phy-4nm | |
| 73 | + | |
| 74 | +required: | |
| 75 | + - compatible | |
| 76 | + | |
| 77 | +unevaluatedProperties: false | |
| 78 | + | |
| 79 | +examples: | |
| 80 | + - | | |
| 81 | + #include <dt-bindings/clock/qcom,dsi-phy-28nm.h> | |
| 82 | + #include <dt-bindings/clock/qcom,rpmh.h> | |
| 83 | + #include <dt-bindings/interconnect/qcom,icc.h> | |
| 84 | + #include <dt-bindings/interrupt-controller/arm-gic.h> | |
| 85 | + #include <dt-bindings/phy/phy-qcom-qmp.h> | |
| 86 | + #include <dt-bindings/power/qcom,rpmhpd.h> | |
| 87 | + | |
| 88 | + display-subsystem@ae00000 { | |
| 89 | + compatible = "qcom,eliza-mdss"; | |
| 90 | + reg = <0x0ae00000 0x1000>; | |
| 91 | + reg-names = "mdss"; | |
| 92 | + ranges; | |
| 93 | + | |
| 94 | + interrupts = <GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>; | |
| 95 | + | |
| 96 | + clocks = <&disp_cc_mdss_ahb_clk>, | |
| 97 | + <&gcc_disp_hf_axi_clk>, | |
| 98 | + <&disp_cc_mdss_mdp_clk>; | |
| 99 | + | |
| 100 | + resets = <&disp_cc_mdss_core_bcr>; | |
| 101 | + | |
| 102 | + interconnects = <&mmss_noc_master_mdp QCOM_ICC_TAG_ALWAYS | |
| 103 | + &mc_virt_slave_ebi1 QCOM_ICC_TAG_ALWAYS>, | |
| 104 | + <&gem_noc_master_appss_proc QCOM_ICC_TAG_ACTIVE_ONLY | |
| 105 | + &config_noc_slave_display_cfg QCOM_ICC_TAG_ACTIVE_ONLY>; | |
| 106 | + interconnect-names = "mdp0-mem", | |
| 107 | + "cpu-cfg"; | |
| 108 | + | |
| 109 | + power-domains = <&mdss_gdsc>; | |
| 110 | + | |
| 111 | + iommus = <&apps_smmu 0x800 0x2>; | |
| 112 | + | |
| 113 | + interrupt-controller; | |
| 114 | + #interrupt-cells = <1>; | |
| 115 | + | |
| 116 | + #address-cells = <1>; | |
| 117 | + #size-cells = <1>; | |
| 118 | + | |
| 119 | + mdss_mdp: display-controller@ae01000 { | |
| 120 | + compatible = "qcom,eliza-dpu"; | |
| 121 | + reg = <0x0ae01000 0x93000>, | |
| 122 | + <0x0aeb0000 0x2008>; | |
| 123 | + reg-names = "mdp", | |
| 124 | + "vbif"; | |
| 125 | + | |
| 126 | + interrupts-extended = <&mdss 0>; | |
| 127 | + | |
| 128 | + clocks = <&gcc_disp_hf_axi_clk>, | |
| 129 | + <&disp_cc_mdss_ahb_clk>, | |
| 130 | + <&disp_cc_mdss_mdp_lut_clk>, | |
| 131 | + <&disp_cc_mdss_mdp_clk>, | |
| 132 | + <&disp_cc_mdss_vsync_clk>; | |
| 133 | + clock-names = "nrt_bus", | |
| 134 | + "iface", | |
| 135 | + "lut", | |
| 136 | + "core", | |
| 137 | + "vsync"; | |
| 138 | + | |
| 139 | + assigned-clocks = <&disp_cc_mdss_vsync_clk>; | |
| 140 | + assigned-clock-rates = <19200000>; | |
| 141 | + | |
| 142 | + operating-points-v2 = <&mdp_opp_table>; | |
| 143 | + | |
| 144 | + power-domains = <&rpmhpd RPMHPD_MMCX>; | |
| 145 | + | |
| 146 | + ports { | |
| 147 | + #address-cells = <1>; | |
| 148 | + #size-cells = <0>; | |
| 149 | + | |
| 150 | + port@0 { | |
| 151 | + reg = <0>; | |
| 152 | + | |
| 153 | + dpu_intf1_out: endpoint { | |
| 154 | + remote-endpoint = <&mdss_dsi0_in>; | |
| 155 | + }; | |
| 156 | + }; | |
| 157 | + | |
| 158 | + port@1 { | |
| 159 | + reg = <1>; | |
| 160 | + | |
| 161 | + dpu_intf2_out: endpoint { | |
| 162 | + remote-endpoint = <&mdss_dsi1_in>; | |
| 163 | + }; | |
| 164 | + }; | |
| 165 | + | |
| 166 | + port@2 { | |
| 167 | + reg = <2>; | |
| 168 | + | |
| 169 | + dpu_intf0_out: endpoint { | |
| 170 | + remote-endpoint = <&mdss_dp0_in>; | |
| 171 | + }; | |
| 172 | + }; | |
| 173 | + }; | |
| 174 | + | |
| 175 | + mdp_opp_table: opp-table { | |
| 176 | + compatible = "operating-points-v2"; | |
| 177 | + | |
| 178 | + opp-150000000 { | |
| 179 | + opp-hz = /bits/ 64 <150000000>; | |
| 180 | + required-opps = <&rpmhpd_opp_low_svs_d1>; | |
| 181 | + }; | |
| 182 | + | |
| 183 | + opp-207000000 { | |
| 184 | + opp-hz = /bits/ 64 <207000000>; | |
| 185 | + required-opps = <&rpmhpd_opp_low_svs>; | |
| 186 | + }; | |
| 187 | + | |
| 188 | + opp-342000000 { | |
| 189 | + opp-hz = /bits/ 64 <342000000>; | |
| 190 | + required-opps = <&rpmhpd_opp_svs>; | |
| 191 | + }; | |
| 192 | + | |
| 193 | + opp-417000000 { | |
| 194 | + opp-hz = /bits/ 64 <417000000>; | |
| 195 | + required-opps = <&rpmhpd_opp_svs_l1>; | |
| 196 | + }; | |
| 197 | + | |
| 198 | + opp-532000000 { | |
| 199 | + opp-hz = /bits/ 64 <532000000>; | |
| 200 | + required-opps = <&rpmhpd_opp_nom>; | |
| 201 | + }; | |
| 202 | + | |
| 203 | + opp-600000000 { | |
| 204 | + opp-hz = /bits/ 64 <600000000>; | |
| 205 | + required-opps = <&rpmhpd_opp_nom_l1>; | |
| 206 | + }; | |
| 207 | + | |
| 208 | + opp-660000000 { | |
| 209 | + opp-hz = /bits/ 64 <660000000>; | |
| 210 | + required-opps = <&rpmhpd_opp_turbo>; | |
| 211 | + }; | |
| 212 | + }; | |
| 213 | + }; | |
| 214 | + | |
| 215 | + dsi@ae94000 { | |
| 216 | + compatible = "qcom,eliza-dsi-ctrl", "qcom,sm8750-dsi-ctrl", "qcom,mdss-dsi-ctrl"; | |
| 217 | + reg = <0x0ae94000 0x400>; | |
| 218 | + reg-names = "dsi_ctrl"; | |
| 219 | + | |
| 220 | + interrupts-extended = <&mdss 4>; | |
| 221 | + | |
| 222 | + clocks = <&disp_cc_mdss_byte0_clk>, | |
| 223 | + <&disp_cc_mdss_byte0_intf_clk>, | |
| 224 | + <&disp_cc_mdss_pclk0_clk>, | |
| 225 | + <&disp_cc_mdss_esc0_clk>, | |
| 226 | + <&disp_cc_mdss_ahb_clk>, | |
| 227 | + <&gcc_disp_hf_axi_clk>, | |
| 228 | + <&mdss_dsi0_phy DSI_PIXEL_PLL_CLK>, | |
| 229 | + <&mdss_dsi0_phy DSI_BYTE_PLL_CLK>, | |
| 230 | + <&disp_cc_esync0_clk>, | |
| 231 | + <&disp_cc_osc_clk>, | |
| 232 | + <&disp_cc_mdss_byte0_clk_src>, | |
| 233 | + <&disp_cc_mdss_pclk0_clk_src>; | |
| 234 | + clock-names = "byte", | |
| 235 | + "byte_intf", | |
| 236 | + "pixel", | |
| 237 | + "core", | |
| 238 | + "iface", | |
| 239 | + "bus", | |
| 240 | + "dsi_pll_pixel", | |
| 241 | + "dsi_pll_byte", | |
| 242 | + "esync", | |
| 243 | + "osc", | |
| 244 | + "byte_src", | |
| 245 | + "pixel_src"; | |
| 246 | + | |
| 247 | + operating-points-v2 = <&mdss_dsi_opp_table>; | |
| 248 | + | |
| 249 | + power-domains = <&rpmhpd RPMHPD_MMCX>; | |
| 250 | + | |
| 251 | + phys = <&mdss_dsi0_phy>; | |
| 252 | + phy-names = "dsi"; | |
| 253 | + | |
| 254 | + #address-cells = <1>; | |
| 255 | + #size-cells = <0>; | |
| 256 | + | |
| 257 | + ports { | |
| 258 | + #address-cells = <1>; | |
| 259 | + #size-cells = <0>; | |
| 260 | + | |
| 261 | + port@0 { | |
| 262 | + reg = <0>; | |
| 263 | + | |
| 264 | + mdss_dsi0_in: endpoint { | |
| 265 | + remote-endpoint = <&dpu_intf1_out>; | |
| 266 | + }; | |
| 267 | + }; | |
| 268 | + | |
| 269 | + port@1 { | |
| 270 | + reg = <1>; | |
| 271 | + | |
| 272 | + mdss_dsi0_out: endpoint { | |
| 273 | + remote-endpoint = <&panel0_in>; | |
| 274 | + data-lanes = <0 1 2 3>; | |
| 275 | + }; | |
| 276 | + }; | |
| 277 | + }; | |
| 278 | + | |
| 279 | + mdss_dsi_opp_table: opp-table { | |
| 280 | + compatible = "operating-points-v2"; | |
| 281 | + | |
| 282 | + opp-140630000 { | |
| 283 | + opp-hz = /bits/ 64 <140630000>; | |
| 284 | + required-opps = <&rpmhpd_opp_low_svs_d1>; | |
| 285 | + }; | |
| 286 | + | |
| 287 | + opp-187500000 { | |
| 288 | + opp-hz = /bits/ 64 <187500000>; | |
| 289 | + required-opps = <&rpmhpd_opp_low_svs>; | |
| 290 | + }; | |
| 291 | + | |
| 292 | + opp-300000000 { | |
| 293 | + opp-hz = /bits/ 64 <300000000>; | |
| 294 | + required-opps = <&rpmhpd_opp_svs>; | |
| 295 | + }; | |
| 296 | + | |
| 297 | + opp-358000000 { | |
| 298 | + opp-hz = /bits/ 64 <358000000>; | |
| 299 | + required-opps = <&rpmhpd_opp_svs_l1>; | |
| 300 | + }; | |
| 301 | + }; | |
| 302 | + }; | |
| 303 | + | |
| 304 | + mdss_dsi0_phy: phy@ae95000 { | |
| 305 | + compatible = "qcom,eliza-dsi-phy-4nm", "qcom,sm8650-dsi-phy-4nm"; | |
| 306 | + reg = <0x0ae95000 0x200>, | |
| 307 | + <0x0ae95200 0x280>, | |
| 308 | + <0x0ae95500 0x400>; | |
| 309 | + reg-names = "dsi_phy", | |
| 310 | + "dsi_phy_lane", | |
| 311 | + "dsi_pll"; | |
| 312 | + | |
| 313 | + clocks = <&disp_cc_mdss_ahb_clk>, | |
| 314 | + <&bi_tcxo_div2>; | |
| 315 | + clock-names = "iface", | |
| 316 | + "ref"; | |
| 317 | + | |
| 318 | + #clock-cells = <1>; | |
| 319 | + #phy-cells = <0>; | |
| 320 | + | |
| 321 | + vdds-supply = <&vreg_l2b>; | |
| 322 | + }; | |
| 323 | + | |
| 324 | + dsi@ae96000 { | |
| 325 | + compatible = "qcom,eliza-dsi-ctrl", "qcom,sm8750-dsi-ctrl", "qcom,mdss-dsi-ctrl"; | |
| 326 | + reg = <0x0ae96000 0x400>; | |
| 327 | + reg-names = "dsi_ctrl"; | |
| 328 | + | |
| 329 | + interrupts-extended = <&mdss 5>; | |
| 330 | + | |
| 331 | + clocks = <&disp_cc_mdss_byte1_clk>, | |
| 332 | + <&disp_cc_mdss_byte1_intf_clk>, | |
| 333 | + <&disp_cc_mdss_pclk1_clk>, | |
| 334 | + <&disp_cc_mdss_esc1_clk>, | |
| 335 | + <&disp_cc_mdss_ahb_clk>, | |
| 336 | + <&gcc_disp_hf_axi_clk>, | |
| 337 | + <&mdss_dsi1_phy DSI_PIXEL_PLL_CLK>, | |
| 338 | + <&mdss_dsi1_phy DSI_BYTE_PLL_CLK>, | |
| 339 | + <&disp_cc_esync1_clk>, | |
| 340 | + <&disp_cc_osc_clk>, | |
| 341 | + <&disp_cc_mdss_byte1_clk_src>, | |
| 342 | + <&disp_cc_mdss_pclk1_clk_src>; | |
| 343 | + clock-names = "byte", | |
| 344 | + "byte_intf", | |
| 345 | + "pixel", | |
| 346 | + "core", | |
| 347 | + "iface", | |
| 348 | + "bus", | |
| 349 | + "dsi_pll_pixel", | |
| 350 | + "dsi_pll_byte", | |
| 351 | + "esync", | |
| 352 | + "osc", | |
| 353 | + "byte_src", | |
| 354 | + "pixel_src"; | |
| 355 | + | |
| 356 | + operating-points-v2 = <&mdss_dsi_opp_table>; | |
| 357 | + | |
| 358 | + power-domains = <&rpmhpd RPMHPD_MMCX>; | |
| 359 | + | |
| 360 | + phys = <&mdss_dsi1_phy>; | |
| 361 | + phy-names = "dsi"; | |
| 362 | + | |
| 363 | + vdda-supply = <&vreg_l4b>; | |
| 364 | + | |
| 365 | + ports { | |
| 366 | + #address-cells = <1>; | |
| 367 | + #size-cells = <0>; | |
| 368 | + | |
| 369 | + port@0 { | |
| 370 | + reg = <0>; | |
| 371 | + | |
| 372 | + mdss_dsi1_in: endpoint { | |
| 373 | + remote-endpoint = <&dpu_intf2_out>; | |
| 374 | + }; | |
| 375 | + }; | |
| 376 | + | |
| 377 | + port@1 { | |
| 378 | + reg = <1>; | |
| 379 | + | |
| 380 | + mdss_dsi1_out: endpoint { | |
| 381 | + remote-endpoint = <&panel1_in>; | |
| 382 | + data-lanes = <0 1 2 3>; | |
| 383 | + }; | |
| 384 | + }; | |
| 385 | + }; | |
| 386 | + }; | |
| 387 | + | |
| 388 | + mdss_dsi1_phy: phy@ae97000 { | |
| 389 | + compatible = "qcom,eliza-dsi-phy-4nm", "qcom,sm8650-dsi-phy-4nm"; | |
| 390 | + reg = <0x0ae97000 0x200>, | |
| 391 | + <0x0ae97200 0x280>, | |
| 392 | + <0x0ae97500 0x400>; | |
| 393 | + reg-names = "dsi_phy", | |
| 394 | + "dsi_phy_lane", | |
| 395 | + "dsi_pll"; | |
| 396 | + | |
| 397 | + clocks = <&disp_cc_mdss_ahb_clk>, | |
| 398 | + <&rpmhcc RPMH_CXO_CLK>; | |
| 399 | + clock-names = "iface", | |
| 400 | + "ref"; | |
| 401 | + | |
| 402 | + #clock-cells = <1>; | |
| 403 | + #phy-cells = <0>; | |
| 404 | + | |
| 405 | + vdds-supply = <&vreg_l2b>; | |
| 406 | + }; | |
| 407 | + | |
| 408 | + displayport-controller@af54000 { | |
| 409 | + compatible = "qcom,eliza-dp", "qcom,sm8650-dp"; | |
| 410 | + reg = <0xaf54000 0x104>, | |
| 411 | + <0xaf54200 0xc0>, | |
| 412 | + <0xaf55000 0x770>, | |
| 413 | + <0xaf56000 0x9c>, | |
| 414 | + <0xaf57000 0x9c>; | |
| 415 | + | |
| 416 | + interrupts-extended = <&mdss 12>; | |
| 417 | + | |
| 418 | + clocks = <&disp_cc_mdss_ahb_clk>, | |
| 419 | + <&disp_cc_mdss_dptx0_aux_clk>, | |
| 420 | + <&disp_cc_mdss_dptx0_link_clk>, | |
| 421 | + <&disp_cc_mdss_dptx0_link_intf_clk>, | |
| 422 | + <&disp_cc_mdss_dptx0_pixel0_clk>, | |
| 423 | + <&disp_cc_mdss_dptx0_pixel1_clk>; | |
| 424 | + clock-names = "core_iface", | |
| 425 | + "core_aux", | |
| 426 | + "ctrl_link", | |
| 427 | + "ctrl_link_iface", | |
| 428 | + "stream_pixel", | |
| 429 | + "stream_1_pixel"; | |
| 430 | + | |
| 431 | + assigned-clocks = <&disp_cc_mdss_dptx0_link_clk_src>, | |
| 432 | + <&disp_cc_mdss_dptx0_pixel0_clk_src>, | |
| 433 | + <&disp_cc_mdss_dptx0_pixel1_clk_src>; | |
| 434 | + assigned-clock-parents = <&usb_dp_qmpphy QMP_USB43DP_DP_LINK_CLK>, | |
| 435 | + <&usb_dp_qmpphy QMP_USB43DP_DP_VCO_DIV_CLK>, | |
| 436 | + <&usb_dp_qmpphy QMP_USB43DP_DP_VCO_DIV_CLK>; | |
| 437 | + | |
| 438 | + operating-points-v2 = <&dp_opp_table>; | |
| 439 | + | |
| 440 | + power-domains = <&rpmhpd RPMHPD_MMCX>; | |
| 441 | + | |
| 442 | + phys = <&usb_dp_qmpphy QMP_USB43DP_DP_PHY>; | |
| 443 | + phy-names = "dp"; | |
| 444 | + | |
| 445 | + #sound-dai-cells = <0>; | |
| 446 | + | |
| 447 | + dp_opp_table: opp-table { | |
| 448 | + compatible = "operating-points-v2"; | |
| 449 | + | |
| 450 | + opp-192000000 { | |
| 451 | + opp-hz = /bits/ 64 <192000000>; | |
| 452 | + required-opps = <&rpmhpd_opp_low_svs_d1>; | |
| 453 | + }; | |
| 454 | + | |
| 455 | + opp-270000000 { | |
| 456 | + opp-hz = /bits/ 64 <270000000>; | |
| 457 | + required-opps = <&rpmhpd_opp_low_svs>; | |
| 458 | + }; | |
| 459 | + | |
| 460 | + opp-540000000 { | |
| 461 | + opp-hz = /bits/ 64 <540000000>; | |
| 462 | + required-opps = <&rpmhpd_opp_svs_l1>; | |
| 463 | + }; | |
| 464 | + | |
| 465 | + opp-810000000 { | |
| 466 | + opp-hz = /bits/ 64 <810000000>; | |
| 467 | + required-opps = <&rpmhpd_opp_nom>; | |
| 468 | + }; | |
| 469 | + }; | |
| 470 | + | |
| 471 | + ports { | |
| 472 | + #address-cells = <1>; | |
| 473 | + #size-cells = <0>; | |
| 474 | + | |
| 475 | + port@0 { | |
| 476 | + reg = <0>; | |
| 477 | + | |
| 478 | + mdss_dp0_in: endpoint { | |
| 479 | + remote-endpoint = <&dpu_intf0_out>; | |
| 480 | + }; | |
| 481 | + }; | |
| 482 | + | |
| 483 | + port@1 { | |
| 484 | + reg = <1>; | |
| 485 | + | |
| 486 | + mdss_dp0_out: endpoint { | |
| 487 | + data-lanes = <0 1 2 3>; | |
| 488 | + remote-endpoint = <&usb_dp_qmpphy_dp_in>; | |
| 489 | + link-frequencies = /bits/ 64 <1620000000 2700000000 5400000000 8100000000>; | |
| 490 | + }; | |
| 491 | + }; | |
| 492 | + }; | |
| 493 | + }; | |
| 494 | + }; | |
| @@ -176,13 +176,17 @@ examples: · 175 unmodified lines | ||
| 176 | 176 | }; |
| 177 | 177 | }; |
| 178 | 178 | |
| 179 | − displayport-controller@ae90000 { | |
| 179 | + displayport-controller@af54000 { | |
| 180 | 180 | compatible = "qcom,glymur-dp"; |
| 181 | − reg = <0xae90000 0x200>, | |
| 182 | − <0xae90200 0x200>, | |
| 183 | − <0xae90400 0x600>, | |
| 184 | − <0xae91000 0x400>, | |
| 185 | − <0xae91400 0x400>; | |
| 181 | + reg = <0xaf54000 0x200>, | |
| 182 | + <0xaf54200 0x200>, | |
| 183 | + <0xaf55000 0xc00>, | |
| 184 | + <0xaf56000 0x400>, | |
| 185 | + <0xaf57000 0x400>, | |
| 186 | + <0xaf58000 0x400>, | |
| 187 | + <0xaf59000 0x400>, | |
| 188 | + <0xaf5a000 0x600>, | |
| 189 | + <0xaf5b000 0x600>; | |
| 186 | 190 | |
| 187 | 191 | interrupt-parent = <&mdss>; |
| 188 | 192 | interrupts = <12>; |
| @@ -50,6 +50,22 @@ patternProperties: · 49 unmodified lines | ||
| 50 | 50 | - qcom,sc8280xp-dp |
| 51 | 51 | - qcom,sc8280xp-edp |
| 52 | 52 | |
| 53 | + "^dsi@[0-9a-f]+$": | |
| 54 | + type: object | |
| 55 | + additionalProperties: true | |
| 56 | + properties: | |
| 57 | + compatible: | |
| 58 | + contains: | |
| 59 | + const: qcom,sc8280xp-dsi-ctrl | |
| 60 | + | |
| 61 | + "^phy@[0-9a-f]+$": | |
| 62 | + type: object | |
| 63 | + additionalProperties: true | |
| 64 | + properties: | |
| 65 | + compatible: | |
| 66 | + contains: | |
| 67 | + const: qcom,sc8280xp-dsi-phy-5nm | |
| 68 | + | |
| 53 | 69 | unevaluatedProperties: false |
| 54 | 70 | |
| 55 | 71 | examples: |
| @@ -129,6 +145,20 @@ examples: · 73 unmodified lines | ||
| 129 | 145 | }; |
| 130 | 146 | }; |
| 131 | 147 | |
| 148 | + port@1 { | |
| 149 | + reg = <1>; | |
| 150 | + dpu_intf1_out: endpoint { | |
| 151 | + remote-endpoint = <&mdss0_dsi0_in>; | |
| 152 | + }; | |
| 153 | + }; | |
| 154 | + | |
| 155 | + port@2 { | |
| 156 | + reg = <2>; | |
| 157 | + dpu_intf2_out: endpoint { | |
| 158 | + remote-endpoint = <&mdss0_dsi1_in>; | |
| 159 | + }; | |
| 160 | + }; | |
| 161 | + | |
| 132 | 162 | port@4 { |
| 133 | 163 | reg = <4>; |
| 134 | 164 | endpoint { |
| @@ -15,6 +15,7 @@ properties: · 14 unmodified lines | ||
| 15 | 15 | compatible: |
| 16 | 16 | oneOf: |
| 17 | 17 | - enum: |
| 18 | + - qcom,eliza-dpu | |
| 18 | 19 | - qcom,glymur-dpu |
| 19 | 20 | - qcom,kaanapali-dpu |
| 20 | 21 | - qcom,sa8775p-dpu |
| @@ -10,7 +10,7 @@ maintainers: · 9 unmodified lines | ||
| 10 | 10 | - Krzysztof Kozlowski <krzk@kernel.org> |
| 11 | 11 | |
| 12 | 12 | description: |
| 13 | − SM8650 MSM Mobile Display Subsystem(MDSS), which encapsulates sub-blocks like | |
| 13 | + SM8750 MSM Mobile Display Subsystem(MDSS), which encapsulates sub-blocks like | |
| 14 | 14 | DPU display controller, DSI and DP interfaces etc. |
| 15 | 15 | |
| 16 | 16 | $ref: /schemas/display/msm/mdss-common.yaml# |
| @@ -28,7 +28,6 @@ properties: · 27 unmodified lines | ||
| 28 | 28 | |
| 29 | 29 | port: true |
| 30 | 30 | reset-gpios: true |
| 31 | − | |
| 32 | 31 | backlight: true |
| 33 | 32 | |
| 34 | 33 | required: |
| @@ -20,6 +20,8 @@ properties: · 19 unmodified lines | ||
| 20 | 20 | - boe,nv110wum-l60 |
| 21 | 21 | # CSOT pna957qt1-1 10.95" WUXGA TFT LCD panel |
| 22 | 22 | - csot,pna957qt1-1 |
| 23 | + # Holitech HTF065H045 6.517" 720x1600 TFT LCD panel | |
| 24 | + - holitech,htf065h045 | |
| 23 | 25 | # IVO t109nw41 11.0" WUXGA TFT LCD panel |
| 24 | 26 | - ivo,t109nw41 |
| 25 | 27 | # KINGDISPLAY KD110N11-51IE 10.95" WUXGA TFT LCD panel |
| @@ -0,0 +1,91 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | |
| 2 | +%YAML 1.2 | |
| 3 | +--- | |
| 4 | +$id: http://devicetree.org/schemas/display/panel/himax,hx83121a.yaml# | |
| 5 | +$schema: http://devicetree.org/meta-schemas/core.yaml# | |
| 6 | + | |
| 7 | +title: Himax HX83121A based DSI display Panels | |
| 8 | + | |
| 9 | +maintainers: | |
| 10 | + - Pengyu Luo <mitltlatltl@gmail.com> | |
| 11 | + | |
| 12 | +description: | |
| 13 | + The Himax HX83121A is a generic DSI Panel IC used to drive dsi | |
| 14 | + panels. Support video mode panels from China Star Optoelectronics | |
| 15 | + Technology (CSOT) and BOE Technology. | |
| 16 | + | |
| 17 | +allOf: | |
| 18 | + - $ref: panel-common-dual.yaml# | |
| 19 | + | |
| 20 | +properties: | |
| 21 | + compatible: | |
| 22 | + items: | |
| 23 | + - enum: | |
| 24 | + - boe,ppc357db1-4 | |
| 25 | + - csot,ppc357db1-4 | |
| 26 | + - const: himax,hx83121a | |
| 27 | + | |
| 28 | + reg: | |
| 29 | + maxItems: 1 | |
| 30 | + | |
| 31 | + reset-gpios: | |
| 32 | + maxItems: 1 | |
| 33 | + | |
| 34 | + avdd-supply: | |
| 35 | + description: analog positive supply for IC | |
| 36 | + | |
| 37 | + avee-supply: | |
| 38 | + description: analog negative supply for IC | |
| 39 | + | |
| 40 | + vddi-supply: | |
| 41 | + description: power supply for IC | |
| 42 | + | |
| 43 | + backlight: true | |
| 44 | + ports: true | |
| 45 | + | |
| 46 | +required: | |
| 47 | + - compatible | |
| 48 | + - reg | |
| 49 | + - vddi-supply | |
| 50 | + - reset-gpios | |
| 51 | + - ports | |
| 52 | + | |
| 53 | +additionalProperties: false | |
| 54 | + | |
| 55 | +examples: | |
| 56 | + - | | |
| 57 | + #include <dt-bindings/gpio/gpio.h> | |
| 58 | + | |
| 59 | + dsi { | |
| 60 | + #address-cells = <1>; | |
| 61 | + #size-cells = <0>; | |
| 62 | + | |
| 63 | + panel@0 { | |
| 64 | + compatible = "csot,ppc357db1-4", "himax,hx83121a"; | |
| 65 | + reg = <0>; | |
| 66 | + | |
| 67 | + vddi-supply = <&vreg_l2b>; | |
| 68 | + reset-gpios = <&tlmm 38 GPIO_ACTIVE_LOW>; | |
| 69 | + | |
| 70 | + ports { | |
| 71 | + #address-cells = <1>; | |
| 72 | + #size-cells = <0>; | |
| 73 | + | |
| 74 | + port@0 { | |
| 75 | + reg = <0>; | |
| 76 | + panel_in_0: endpoint { | |
| 77 | + remote-endpoint = <&dsi0_out>; | |
| 78 | + }; | |
| 79 | + }; | |
| 80 | + | |
| 81 | + port@1{ | |
| 82 | + reg = <1>; | |
| 83 | + panel_in_1: endpoint { | |
| 84 | + remote-endpoint = <&dsi1_out>; | |
| 85 | + }; | |
| 86 | + }; | |
| 87 | + }; | |
| 88 | + }; | |
| 89 | + }; | |
| 90 | + | |
| 91 | +... | |
| @@ -33,11 +33,8 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | maxItems: 1 |
| 34 | 34 | |
| 35 | 35 | reset-gpios: true |
| 36 | − | |
| 37 | 36 | backlight: true |
| 38 | − | |
| 39 | 37 | rotation: true |
| 40 | − | |
| 41 | 38 | port: true |
| 42 | 39 | |
| 43 | 40 | vcc-supply: |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | $id: http://devicetree.org/schemas/display/panel/ilitek,ili9806e.yaml# |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | −title: Ilitek ILI9806E based MIPI-DSI panels | |
| 7 | +title: Ilitek ILI9806E based panels | |
| 8 | 8 | |
| 9 | 9 | maintainers: |
| 10 | 10 | - Michael Walle <mwalle@kernel.org> |
| @@ -18,6 +18,7 @@ properties: · 7 unmodified lines | ||
| 18 | 18 | - enum: |
| 19 | 19 | - densitron,dmt028vghmcmi-1d |
| 20 | 20 | - ortustech,com35h3p70ulc |
| 21 | + - rocktech,rk050hr345-ct106a | |
| 21 | 22 | - const: ilitek,ili9806e |
| 22 | 23 | |
| 23 | 24 | reg: |
| @@ -30,11 +31,24 @@ required: · 6 unmodified lines | ||
| 30 | 31 | - compatible |
| 31 | 32 | - reg |
| 32 | 33 | - vdd-supply |
| 33 | − - vccio-supply | |
| 34 | 34 | - reset-gpios |
| 35 | 35 | - backlight |
| 36 | 36 | - port |
| 37 | 37 | |
| 38 | +if: | |
| 39 | + properties: | |
| 40 | + compatible: | |
| 41 | + contains: | |
| 42 | + enum: | |
| 43 | + - rocktech,rk050hr345-ct106a | |
| 44 | +then: | |
| 45 | + $ref: /schemas/spi/spi-peripheral-props.yaml# | |
| 46 | + required: | |
| 47 | + - spi-max-frequency | |
| 48 | +else: | |
| 49 | + required: | |
| 50 | + - vccio-supply | |
| 51 | + | |
| 38 | 52 | unevaluatedProperties: false |
| 39 | 53 | |
| 40 | 54 | examples: |
| @@ -60,5 +74,25 @@ examples: · 19 unmodified lines | ||
| 60 | 74 | }; |
| 61 | 75 | }; |
| 62 | 76 | }; |
| 77 | + - | | |
| 78 | + #include <dt-bindings/gpio/gpio.h> | |
| 79 | + | |
| 80 | + spi { | |
| 81 | + #address-cells = <1>; | |
| 82 | + #size-cells = <0>; | |
| 63 | 83 | |
| 84 | + panel@0 { | |
| 85 | + compatible = "rocktech,rk050hr345-ct106a", "ilitek,ili9806e"; | |
| 86 | + reg = <0>; | |
| 87 | + vdd-supply = <®_vdd_panel>; | |
| 88 | + spi-max-frequency = <10000000>; | |
| 89 | + reset-gpios = <&gpiob 6 GPIO_ACTIVE_LOW>; | |
| 90 | + backlight = <&backlight>; | |
| 91 | + port { | |
| 92 | + panel_in_rgb: endpoint { | |
| 93 | + remote-endpoint = <<dc_out_rgb>; | |
| 94 | + }; | |
| 95 | + }; | |
| 96 | + }; | |
| 97 | + }; | |
| 64 | 98 | ... |
| @@ -25,6 +25,7 @@ properties: · 24 unmodified lines | ||
| 25 | 25 | backlight: true |
| 26 | 26 | port: true |
| 27 | 27 | reset-gpios: true |
| 28 | + | |
| 28 | 29 | iovcc-supply: |
| 29 | 30 | description: regulator that supplies the iovcc voltage |
| 30 | 31 | vci-supply: |
| @@ -24,6 +24,7 @@ properties: · 23 unmodified lines | ||
| 24 | 24 | backlight: true |
| 25 | 25 | port: true |
| 26 | 26 | reset-gpios: true |
| 27 | + | |
| 27 | 28 | iovcc-supply: |
| 28 | 29 | description: regulator that supplies the iovcc voltage |
| 29 | 30 | vcc-supply: |
| @@ -22,7 +22,6 @@ properties: · 21 unmodified lines | ||
| 22 | 22 | - mantix,mlaf057we51-x |
| 23 | 23 | - ys,ys57pss36bh5gq |
| 24 | 24 | |
| 25 | − port: true | |
| 26 | 25 | reg: |
| 27 | 26 | maxItems: 1 |
| 28 | 27 | description: DSI virtual channel |
| @@ -36,13 +35,13 @@ properties: · 7 unmodified lines | ||
| 36 | 35 | vddi-supply: |
| 37 | 36 | description: 1.8V I/O voltage supply |
| 38 | 37 | |
| 39 | − reset-gpios: true | |
| 40 | − | |
| 41 | 38 | mantix,tp-rstn-gpios: |
| 42 | 39 | maxItems: 1 |
| 43 | 40 | description: second reset line that triggers DSI config load |
| 44 | 41 | |
| 45 | 42 | backlight: true |
| 43 | + port: true | |
| 44 | + reset-gpios: true | |
| 46 | 45 | |
| 47 | 46 | required: |
| 48 | 47 | - compatible |
| @@ -28,13 +28,14 @@ properties: · 27 unmodified lines | ||
| 28 | 28 | reg: |
| 29 | 29 | maxItems: 1 |
| 30 | 30 | |
| 31 | − reset-gpios: true | |
| 32 | 31 | vdd-supply: |
| 33 | 32 | description: regulator that supplies the vdd voltage |
| 34 | 33 | vddi-supply: |
| 35 | 34 | description: regulator that supplies the vddi voltage |
| 35 | + | |
| 36 | 36 | backlight: true |
| 37 | 37 | port: true |
| 38 | + reset-gpios: true | |
| 38 | 39 | |
| 39 | 40 | required: |
| 40 | 41 | - compatible |
| @@ -58,6 +58,10 @@ properties: · 57 unmodified lines | ||
| 58 | 58 | - hydis,hv070wx2-1e0 |
| 59 | 59 | # Jenson Display BL-JT60050-01A 7" WSVGA (1024x600) color TFT LCD LVDS panel |
| 60 | 60 | - jenson,bl-jt60050-01a |
| 61 | + # Samsung LTN070NL01 7.0" WSVGA (1024x600) TFT LCD LVDS panel | |
| 62 | + - samsung,ltn070nl01 | |
| 63 | + # Samsung LTN101AL03 10.1" WXGA (800x1280) TFT LCD LVDS panel | |
| 64 | + - samsung,ltn101al03 | |
| 61 | 65 | - tbs,a711-panel |
| 62 | 66 | # Winstar WF70A8SYJHLNGA 7" WSVGA (1024x600) color TFT LCD LVDS panel |
| 63 | 67 | - winstar,wf70a8syjhlnga |
| @@ -188,6 +188,8 @@ properties: · 187 unmodified lines | ||
| 188 | 188 | - innolux,n156bge-l21 |
| 189 | 189 | # Innolux Corporation 7.0" WSVGA (1024x600) TFT LCD panel |
| 190 | 190 | - innolux,zj070na-01p |
| 191 | + # JuTouch Technology Co.. 7" JT070TM041 WSVGA (1024 x 600) LVDS panel | |
| 192 | + - jutouch,jt070tm041 | |
| 191 | 193 | # JuTouch Technology Co.. 10" JT101TM023 WXGA (1280 x 800) LVDS panel |
| 192 | 194 | - jutouch,jt101tm023 |
| 193 | 195 | # Kaohsiung Opto-Electronics Inc. 5.7" QVGA (320 x 240) TFT LCD panel |
| @@ -310,6 +312,8 @@ properties: · 116 unmodified lines | ||
| 310 | 312 | - team-source-display,tst043015cmhx |
| 311 | 313 | # Tianma Micro-electronics P0700WXF1MBAA 7.0" WXGA (1280x800) LVDS TFT LCD panel |
| 312 | 314 | - tianma,p0700wxf1mbaa |
| 315 | + # Tianma Micro-electronics TM050RDH03 5.0" WVGA TFT LCD panel | |
| 316 | + - tianma,tm050rdh03 | |
| 313 | 317 | # Tianma Micro-electronics TM070JDHG30 7.0" WXGA TFT LCD panel |
| 314 | 318 | - tianma,tm070jdhg30 |
| 315 | 319 | # Tianma Micro-electronics TM070JDHG34-00 7.0" WXGA (1280x800) LVDS TFT LCD panel |
| @@ -33,8 +33,6 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | iovcc-supply: |
| 34 | 34 | description: Regulator for 1.8V IO power supply. |
| 35 | 35 | |
| 36 | − backlight: true | |
| 37 | − | |
| 38 | 36 | renesas,gamma: |
| 39 | 37 | $ref: /schemas/types.yaml#/definitions/uint32 |
| 40 | 38 | description: |
| @@ -51,6 +49,7 @@ properties: · 10 unmodified lines | ||
| 51 | 49 | type: boolean |
| 52 | 50 | description: digital contrast adjustment |
| 53 | 51 | |
| 52 | + backlight: true | |
| 54 | 53 | reset-gpios: true |
| 55 | 54 | port: true |
| 56 | 55 | |
| @@ -33,7 +33,6 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | description: Regulator for 1.8V IO power supply. |
| 34 | 34 | |
| 35 | 35 | backlight: true |
| 36 | − | |
| 37 | 36 | reset-gpios: true |
| 38 | 37 | port: true |
| 39 | 38 | |
| @@ -33,7 +33,6 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | # Xingbangda XBD599 5.99" 720x1440 TFT LCD panel |
| 34 | 34 | - xingbangda,xbd599 |
| 35 | 35 | |
| 36 | − port: true | |
| 37 | 36 | reg: |
| 38 | 37 | maxItems: 1 |
| 39 | 38 | description: DSI virtual channel |
| @@ -44,9 +43,9 @@ properties: · 4 unmodified lines | ||
| 44 | 43 | iovcc-supply: |
| 45 | 44 | description: I/O voltage supply |
| 46 | 45 | |
| 47 | − reset-gpios: true | |
| 48 | − | |
| 49 | 46 | backlight: true |
| 47 | + port: true | |
| 48 | + reset-gpios: true | |
| 50 | 49 | rotation: true |
| 51 | 50 | |
| 52 | 51 | required: |
| @@ -8,13 +8,16 @@ title: Samsung AMS561RA01 panel with S6E8AA5X01 controller · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | maintainers: |
| 10 | 10 | - Kaustabh Chakraborty <kauschluss@disroot.org> |
| 11 | + - Yedaya Katsman <yedaya.ka@gmail.com> | |
| 11 | 12 | |
| 12 | 13 | allOf: |
| 13 | 14 | - $ref: panel-common.yaml# |
| 14 | 15 | |
| 15 | 16 | properties: |
| 16 | 17 | compatible: |
| 17 | − const: samsung,s6e8aa5x01-ams561ra01 | |
| 18 | + enum: | |
| 19 | + - samsung,s6e8aa5x01-ams561ra01 | |
| 20 | + - samsung,s6e8fc0-m1906f9 | |
| 18 | 21 | |
| 19 | 22 | reg: |
| 20 | 23 | maxItems: 1 |
| @@ -31,9 +31,7 @@ properties: · 30 unmodified lines | ||
| 31 | 31 | description: Negative 5V supply |
| 32 | 32 | |
| 33 | 33 | reset-gpios: true |
| 34 | − | |
| 35 | 34 | enable-gpios: true |
| 36 | − | |
| 37 | 35 | port: true |
| 38 | 36 | |
| 39 | 37 | required: |
| @@ -16,8 +16,6 @@ properties: · 15 unmodified lines | ||
| 16 | 16 | compatible: |
| 17 | 17 | const: startek,kd070fhfid015 |
| 18 | 18 | |
| 19 | − enable-gpios: true | |
| 20 | − | |
| 21 | 19 | iovcc-supply: |
| 22 | 20 | description: Reference to the regulator powering the panel IO pins. |
| 23 | 21 | |
| @@ -25,11 +23,10 @@ properties: · 1 unmodified line | ||
| 25 | 23 | maxItems: 1 |
| 26 | 24 | description: DSI virtual channel |
| 27 | 25 | |
| 28 | − reset-gpios: true | |
| 29 | − | |
| 26 | + enable-gpios: true | |
| 30 | 27 | port: true |
| 31 | − | |
| 32 | 28 | power-supply: true |
| 29 | + reset-gpios: true | |
| 33 | 30 | |
| 34 | 31 | required: |
| 35 | 32 | - compatible |
| @@ -37,7 +37,7 @@ properties: · 36 unmodified lines | ||
| 37 | 37 | const: 2 |
| 38 | 38 | |
| 39 | 39 | "#interrupt-cells": |
| 40 | − const: 1 | |
| 40 | + const: 2 | |
| 41 | 41 | |
| 42 | 42 | ngpios: |
| 43 | 43 | description: |
| @@ -86,7 +86,7 @@ examples: · 42 unmodified lines | ||
| 86 | 86 | gpio-controller; |
| 87 | 87 | #gpio-cells = <2>; |
| 88 | 88 | interrupt-controller; |
| 89 | − #interrupt-cells = <1>; | |
| 89 | + #interrupt-cells = <2>; | |
| 90 | 90 | interrupts = <53>, <53>, <53>, <53>, |
| 91 | 91 | <53>, <53>, <53>, <53>, |
| 92 | 92 | <53>, <53>, <53>, <53>, |
| @@ -7,7 +7,7 @@ $schema: http://devicetree.org/meta-schemas/core.yaml# · 6 unmodified lines | ||
| 7 | 7 | title: Synopsys DesignWare APB I2C Controller |
| 8 | 8 | |
| 9 | 9 | maintainers: |
| 10 | − - Jarkko Nikula <jarkko.nikula@linux.intel.com> | |
| 10 | + - Mika Westerberg <mika.westerberg@linux.intel.com> | |
| 11 | 11 | |
| 12 | 12 | allOf: |
| 13 | 13 | - $ref: /schemas/i2c/i2c-controller.yaml# |
| @@ -19,9 +19,6 @@ description: · 18 unmodified lines | ||
| 19 | 19 | Flash sub nodes describe the memory range and optional per-flash |
| 20 | 20 | properties. |
| 21 | 21 | |
| 22 | −allOf: | |
| 23 | − - $ref: mtd.yaml# | |
| 24 | − | |
| 25 | 22 | properties: |
| 26 | 23 | compatible: |
| 27 | 24 | const: st,spear600-smi |
| @@ -42,14 +39,29 @@ properties: · 14 unmodified lines | ||
| 42 | 39 | $ref: /schemas/types.yaml#/definitions/uint32 |
| 43 | 40 | description: Functional clock rate of the SMI controller in Hz. |
| 44 | 41 | |
| 45 | − st,smi-fast-mode: | |
| 46 | − type: boolean | |
| 47 | − description: Indicates that the attached flash supports fast read mode. | |
| 42 | +patternProperties: | |
| 43 | + "^flash@.*$": | |
| 44 | + $ref: /schemas/mtd/mtd.yaml# | |
| 45 | + | |
| 46 | + properties: | |
| 47 | + reg: | |
| 48 | + maxItems: 1 | |
| 49 | + | |
| 50 | + st,smi-fast-mode: | |
| 51 | + type: boolean | |
| 52 | + description: Indicates that the attached flash supports fast read mode. | |
| 53 | + | |
| 54 | + unevaluatedProperties: false | |
| 55 | + | |
| 56 | + required: | |
| 57 | + - reg | |
| 48 | 58 | |
| 49 | 59 | required: |
| 50 | 60 | - compatible |
| 51 | 61 | - reg |
| 52 | 62 | - clock-rate |
| 63 | + - "#address-cells" | |
| 64 | + - "#size-cells" | |
| 53 | 65 | |
| 54 | 66 | unevaluatedProperties: false |
| 55 | 67 | |
| @@ -64,7 +76,7 @@ examples: · 8 unmodified lines | ||
| 64 | 76 | interrupts = <12>; |
| 65 | 77 | clock-rate = <50000000>; /* 50 MHz */ |
| 66 | 78 | |
| 67 | − flash@f8000000 { | |
| 79 | + flash@fc000000 { | |
| 68 | 80 | reg = <0xfc000000 0x1000>; |
| 69 | 81 | st,smi-fast-mode; |
| 70 | 82 | }; |
| @@ -23,6 +23,7 @@ properties: · 22 unmodified lines | ||
| 23 | 23 | - items: |
| 24 | 24 | - enum: |
| 25 | 25 | - mediatek,mt7623-mipi-tx |
| 26 | + - mediatek,mt8167-mipi-tx | |
| 26 | 27 | - const: mediatek,mt2701-mipi-tx |
| 27 | 28 | - items: |
| 28 | 29 | - enum: |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | 2 | %YAML 1.2 |
| 3 | 3 | --- |
| 4 | −$id: http://devicetree.org/schemas/display/msm/dsi-phy-10nm.yaml# | |
| 4 | +$id: http://devicetree.org/schemas/phy/qcom,dsi-phy-10nm.yaml# | |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | 7 | title: Qualcomm Display DSI 10nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | ||
| 10 | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | |
| 12 | 12 | allOf: |
| 13 | − - $ref: dsi-phy-common.yaml# | |
| 13 | + - $ref: qcom,dsi-phy-common.yaml# | |
| 14 | 14 | |
| 15 | 15 | properties: |
| 16 | 16 | compatible: |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | 2 | %YAML 1.2 |
| 3 | 3 | --- |
| 4 | −$id: http://devicetree.org/schemas/display/msm/dsi-phy-14nm.yaml# | |
| 4 | +$id: http://devicetree.org/schemas/phy/qcom,dsi-phy-14nm.yaml# | |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | 7 | title: Qualcomm Display DSI 14nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | ||
| 10 | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | |
| 12 | 12 | allOf: |
| 13 | − - $ref: dsi-phy-common.yaml# | |
| 13 | + - $ref: qcom,dsi-phy-common.yaml# | |
| 14 | 14 | |
| 15 | 15 | properties: |
| 16 | 16 | compatible: |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | 2 | %YAML 1.2 |
| 3 | 3 | --- |
| 4 | −$id: http://devicetree.org/schemas/display/msm/dsi-phy-20nm.yaml# | |
| 4 | +$id: http://devicetree.org/schemas/phy/qcom,dsi-phy-20nm.yaml# | |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | 7 | title: Qualcomm Display DSI 20nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | ||
| 10 | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | |
| 12 | 12 | allOf: |
| 13 | − - $ref: dsi-phy-common.yaml# | |
| 13 | + - $ref: qcom,dsi-phy-common.yaml# | |
| 14 | 14 | |
| 15 | 15 | properties: |
| 16 | 16 | compatible: |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | 2 | %YAML 1.2 |
| 3 | 3 | --- |
| 4 | −$id: http://devicetree.org/schemas/display/msm/dsi-phy-28nm.yaml# | |
| 4 | +$id: http://devicetree.org/schemas/phy/qcom,dsi-phy-28nm.yaml# | |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | 7 | title: Qualcomm Display DSI 28nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | ||
| 10 | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | |
| 12 | 12 | allOf: |
| 13 | − - $ref: dsi-phy-common.yaml# | |
| 13 | + - $ref: qcom,dsi-phy-common.yaml# | |
| 14 | 14 | |
| 15 | 15 | properties: |
| 16 | 16 | compatible: |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | 2 | %YAML 1.2 |
| 3 | 3 | --- |
| 4 | −$id: http://devicetree.org/schemas/display/msm/dsi-phy-7nm.yaml# | |
| 4 | +$id: http://devicetree.org/schemas/phy/qcom,dsi-phy-7nm.yaml# | |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | 7 | title: Qualcomm Display DSI 7nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | ||
| 10 | 10 | - Jonathan Marek <jonathan@marek.ca> |
| 11 | 11 | |
| 12 | 12 | allOf: |
| 13 | − - $ref: dsi-phy-common.yaml# | |
| 13 | + - $ref: qcom,dsi-phy-common.yaml# | |
| 14 | 14 | |
| 15 | 15 | properties: |
| 16 | 16 | compatible: |
| @@ -29,9 +29,14 @@ properties: · 12 unmodified lines | ||
| 29 | 29 | - qcom,sm8550-dsi-phy-4nm |
| 30 | 30 | - qcom,sm8650-dsi-phy-4nm |
| 31 | 31 | - qcom,sm8750-dsi-phy-3nm |
| 32 | + - items: | |
| 33 | + - enum: | |
| 34 | + - qcom,eliza-dsi-phy-4nm | |
| 35 | + - const: qcom,sm8650-dsi-phy-4nm | |
| 32 | 36 | - items: |
| 33 | 37 | - enum: |
| 34 | 38 | - qcom,qcs8300-dsi-phy-5nm |
| 39 | + - qcom,sc8280xp-dsi-phy-5nm | |
| 35 | 40 | - const: qcom,sa8775p-dsi-phy-5nm |
| 36 | 41 | |
| 37 | 42 | reg: |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | 2 | %YAML 1.2 |
| 3 | 3 | --- |
| 4 | −$id: http://devicetree.org/schemas/display/msm/dsi-phy-common.yaml# | |
| 4 | +$id: http://devicetree.org/schemas/phy/qcom,dsi-phy-common.yaml# | |
| 5 | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | |
| 7 | 7 | title: Qualcomm Display DSI PHY Common Properties |
| @@ -0,0 +1,93 @@ | ||
| 1 | +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | |
| 2 | +%YAML 1.2 | |
| 3 | +--- | |
| 4 | +$id: http://devicetree.org/schemas/powerpc/fsl/fsl,mpc83xx.yaml# | |
| 5 | +$schema: http://devicetree.org/meta-schemas/core.yaml# | |
| 6 | + | |
| 7 | +title: Freescale PowerQUICC II Pro (MPC83xx) platforms | |
| 8 | + | |
| 9 | +maintainers: | |
| 10 | + - J. Neuschäfer <j.ne@posteo.net> | |
| 11 | + | |
| 12 | +properties: | |
| 13 | + $nodename: | |
| 14 | + const: '/' | |
| 15 | + compatible: | |
| 16 | + oneOf: | |
| 17 | + - description: MPC83xx Reference Design Boards | |
| 18 | + items: | |
| 19 | + - enum: | |
| 20 | + - fsl,mpc8308rdb | |
| 21 | + - fsl,mpc8315erdb | |
| 22 | + - fsl,mpc8360rdk | |
| 23 | + - fsl,mpc8377rdb | |
| 24 | + - fsl,mpc8377wlan | |
| 25 | + - fsl,mpc8378rdb | |
| 26 | + - fsl,mpc8379rdb | |
| 27 | + | |
| 28 | + - description: MPC8313E Reference Design Board | |
| 29 | + items: | |
| 30 | + - const: MPC8313ERDB | |
| 31 | + - const: MPC831xRDB | |
| 32 | + - const: MPC83xxRDB | |
| 33 | + | |
| 34 | + - description: MPC8323E Reference Design Board | |
| 35 | + items: | |
| 36 | + - const: MPC8323ERDB | |
| 37 | + - const: MPC832xRDB | |
| 38 | + - const: MPC83xxRDB | |
| 39 | + | |
| 40 | + - description: MPC8349E-mITX(-GP) Reference Design Platform | |
| 41 | + items: | |
| 42 | + - enum: | |
| 43 | + - MPC8349EMITX | |
| 44 | + - MPC8349EMITXGP | |
| 45 | + - const: MPC834xMITX | |
| 46 | + - const: MPC83xxMITX | |
| 47 | + | |
| 48 | + - description: Keymile KMETER1 board | |
| 49 | + const: keymile,KMETER1 | |
| 50 | + | |
| 51 | + - description: MPC8308 P1M board | |
| 52 | + const: denx,mpc8308_p1m | |
| 53 | + | |
| 54 | +patternProperties: | |
| 55 | + "^soc@.*$": | |
| 56 | + type: object | |
| 57 | + properties: | |
| 58 | + compatible: | |
| 59 | + oneOf: | |
| 60 | + - items: | |
| 61 | + - enum: | |
| 62 | + - fsl,mpc8315-immr | |
| 63 | + - fsl,mpc8308-immr | |
| 64 | + - const: simple-bus | |
| 65 | + - items: | |
| 66 | + - const: fsl,mpc8360-immr | |
| 67 | + - const: fsl,immr | |
| 68 | + - const: fsl,soc | |
| 69 | + - const: simple-bus | |
| 70 | + - const: simple-bus | |
| 71 | + | |
| 72 | +additionalProperties: true | |
| 73 | + | |
| 74 | +examples: | |
| 75 | + - | | |
| 76 | + / { | |
| 77 | + compatible = "fsl,mpc8315erdb"; | |
| 78 | + model = "MPC8315E-RDB"; | |
| 79 | + #address-cells = <1>; | |
| 80 | + #size-cells = <1>; | |
| 81 | + | |
| 82 | + soc@e0000000 { | |
| 83 | + compatible = "fsl,mpc8315-immr", "simple-bus"; | |
| 84 | + reg = <0xe0000000 0x00000200>; | |
| 85 | + #address-cells = <1>; | |
| 86 | + #size-cells = <1>; | |
| 87 | + device_type = "soc"; | |
| 88 | + ranges = <0 0xe0000000 0x00100000>; | |
| 89 | + bus-frequency = <0>; | |
| 90 | + }; | |
| 91 | + }; | |
| 92 | + | |
| 93 | +... | |
| @@ -168,7 +168,7 @@ properties: · 167 unmodified lines | ||
| 168 | 168 | offset from voltage set to regulator. |
| 169 | 169 | |
| 170 | 170 | regulator-uv-protection-microvolt: |
| 171 | − description: Set over under voltage protection limit. This is a limit where | |
| 171 | + description: Set under voltage protection limit. This is a limit where | |
| 172 | 172 | hardware performs emergency shutdown. Zero can be passed to disable |
| 173 | 173 | protection and value '1' indicates that protection should be enabled but |
| 174 | 174 | limit setting can be omitted. Limit is given as microvolt offset from |
| @@ -182,7 +182,7 @@ properties: · 7 unmodified lines | ||
| 182 | 182 | is given as microvolt offset from voltage set to regulator. |
| 183 | 183 | |
| 184 | 184 | regulator-uv-warn-microvolt: |
| 185 | − description: Set over under voltage warning limit. This is a limit where | |
| 185 | + description: Set under voltage warning limit. This is a limit where | |
| 186 | 186 | hardware is assumed still to be functional but approaching limit where |
| 187 | 187 | it gets damaged. Recovery actions should be initiated. Zero can be passed |
| 188 | 188 | to disable detection and value '1' indicates that detection should |
| @@ -33,6 +33,7 @@ properties: · 32 unmodified lines | ||
| 33 | 33 | - const: rockchip,rk3066-spdif |
| 34 | 34 | - items: |
| 35 | 35 | - enum: |
| 36 | + - rockchip,rk3576-spdif | |
| 36 | 37 | - rockchip,rk3588-spdif |
| 37 | 38 | - const: rockchip,rk3568-spdif |
| 38 | 39 | |
| @@ -164,7 +164,7 @@ allOf: · 163 unmodified lines | ||
| 164 | 164 | properties: |
| 165 | 165 | compatible: |
| 166 | 166 | contains: |
| 167 | − const: st,stm32mph7-sai | |
| 167 | + const: st,stm32h7-sai | |
| 168 | 168 | then: |
| 169 | 169 | properties: |
| 170 | 170 | clocks: |
| @@ -6,9 +6,6 @@ $schema: http://devicetree.org/meta-schemas/core.yaml# · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | title: Allwinner A31 SPI Controller |
| 8 | 8 | |
| 9 | −allOf: | |
| 10 | − - $ref: spi-controller.yaml | |
| 11 | − | |
| 12 | 9 | maintainers: |
| 13 | 10 | - Chen-Yu Tsai <wens@csie.org> |
| 14 | 11 | - Maxime Ripard <mripard@kernel.org> |
| @@ -82,11 +79,11 @@ patternProperties: · 67 unmodified lines | ||
| 82 | 79 | |
| 83 | 80 | spi-rx-bus-width: |
| 84 | 81 | items: |
| 85 | − - const: 1 | |
| 82 | + enum: [0, 1, 2, 4] | |
| 86 | 83 | |
| 87 | 84 | spi-tx-bus-width: |
| 88 | 85 | items: |
| 89 | − - const: 1 | |
| 86 | + enum: [0, 1, 2, 4] | |
| 90 | 87 | |
| 91 | 88 | required: |
| 92 | 89 | - compatible |
| @@ -95,6 +92,28 @@ required: · 2 unmodified lines | ||
| 95 | 92 | - clocks |
| 96 | 93 | - clock-names |
| 97 | 94 | |
| 95 | +allOf: | |
| 96 | + - $ref: spi-controller.yaml | |
| 97 | + - if: | |
| 98 | + not: | |
| 99 | + properties: | |
| 100 | + compatible: | |
| 101 | + contains: | |
| 102 | + enum: | |
| 103 | + - allwinner,sun50i-r329-spi | |
| 104 | + - allwinner,sun55i-a523-spi | |
| 105 | + then: | |
| 106 | + patternProperties: | |
| 107 | + "^.*@[0-9a-f]+": | |
| 108 | + properties: | |
| 109 | + spi-rx-bus-width: | |
| 110 | + items: | |
| 111 | + enum: [0, 1] | |
| 112 | + | |
| 113 | + spi-tx-bus-width: | |
| 114 | + items: | |
| 115 | + enum: [0, 1] | |
| 116 | + | |
| 98 | 117 | unevaluatedProperties: false |
| 99 | 118 | |
| 100 | 119 | examples: |
| @@ -172,6 +172,8 @@ patternProperties: · 171 unmodified lines | ||
| 172 | 172 | description: ARM Ltd. |
| 173 | 173 | "^armadeus,.*": |
| 174 | 174 | description: ARMadeus Systems SARL |
| 175 | + "^armchina,.*": | |
| 176 | + description: Arm Technology (China) Co., Ltd. | |
| 175 | 177 | "^armsom,.*": |
| 176 | 178 | description: ArmSoM Technology Co., Ltd. |
| 177 | 179 | "^arrow,.*": |
| @@ -441,6 +443,8 @@ patternProperties: · 263 unmodified lines | ||
| 441 | 443 | description: D-Link Corporation |
| 442 | 444 | "^dmo,.*": |
| 443 | 445 | description: Data Modul AG |
| 446 | + "^doestek,.*": | |
| 447 | + description: Doestek Co., Ltd. | |
| 444 | 448 | "^domintech,.*": |
| 445 | 449 | description: Domintech Co., Ltd. |
| 446 | 450 | "^dongwoon,.*": |
| @@ -709,6 +713,8 @@ patternProperties: · 262 unmodified lines | ||
| 709 | 713 | description: Hitex Development Tools |
| 710 | 714 | "^hitron,.*": |
| 711 | 715 | description: HiTRON Electronics Corporation |
| 716 | + "^holitech,.*": | |
| 717 | + description: Jiangxi Holitech Technology Co., Ltd. | |
| 712 | 718 | "^holt,.*": |
| 713 | 719 | description: Holt Integrated Circuits, Inc. |
| 714 | 720 | "^holtek,.*": |
| @@ -99,3 +99,51 @@ of the driver is decremented. All symlinks between the two are removed. · 98 unmodified lines | ||
| 99 | 99 | When a driver is removed, the list of devices that it supports is |
| 100 | 100 | iterated over, and the driver's remove callback is called for each |
| 101 | 101 | one. The device is removed from that list and the symlinks removed. |
| 102 | + | |
| 103 | + | |
| 104 | +Driver Override | |
| 105 | +~~~~~~~~~~~~~~~ | |
| 106 | + | |
| 107 | +Userspace may override the standard matching by writing a driver name to | |
| 108 | +a device's ``driver_override`` sysfs attribute. When set, only a driver | |
| 109 | +whose name matches the override will be considered during binding. This | |
| 110 | +bypasses all bus-specific matching (OF, ACPI, ID tables, etc.). | |
| 111 | + | |
| 112 | +The override may be cleared by writing an empty string, which returns | |
| 113 | +the device to standard matching rules. Writing to ``driver_override`` | |
| 114 | +does not automatically unbind the device from its current driver or | |
| 115 | +make any attempt to load the specified driver. | |
| 116 | + | |
| 117 | +Buses opt into this mechanism by setting the ``driver_override`` flag in | |
| 118 | +their ``struct bus_type``:: | |
| 119 | + | |
| 120 | + const struct bus_type example_bus_type = { | |
| 121 | + ... | |
| 122 | + .driver_override = true, | |
| 123 | + }; | |
| 124 | + | |
| 125 | +When the flag is set, the driver core automatically creates the | |
| 126 | +``driver_override`` sysfs attribute for every device on that bus. | |
| 127 | + | |
| 128 | +The bus's ``match()`` callback should check the override before performing | |
| 129 | +its own matching, using ``device_match_driver_override()``:: | |
| 130 | + | |
| 131 | + static int example_match(struct device *dev, const struct device_driver *drv) | |
| 132 | + { | |
| 133 | + int ret; | |
| 134 | + | |
| 135 | + ret = device_match_driver_override(dev, drv); | |
| 136 | + if (ret >= 0) | |
| 137 | + return ret; | |
| 138 | + | |
| 139 | + /* Fall through to bus-specific matching... */ | |
| 140 | + } | |
| 141 | + | |
| 142 | +``device_match_driver_override()`` returns > 0 if the override matches | |
| 143 | +the given driver, 0 if the override is set but does not match, or < 0 if | |
| 144 | +no override is set at all. | |
| 145 | + | |
| 146 | +Additional helpers are available: | |
| 147 | + | |
| 148 | +- ``device_set_driver_override()`` - set or clear the override from kernel code. | |
| 149 | +- ``device_has_driver_override()`` - check whether an override is set. | |
| @@ -783,6 +783,56 @@ controlled by the "uuid" mount option, which supports these values: · 782 unmodified lines | ||
| 783 | 783 | mounted with "uuid=on". |
| 784 | 784 | |
| 785 | 785 | |
| 786 | +Durability and copy up | |
| 787 | +---------------------- | |
| 788 | + | |
| 789 | +The fsync(2) system call ensures that the data and metadata of a file | |
| 790 | +are safely written to the backing storage, which is expected to | |
| 791 | +guarantee the existence of the information post system crash. | |
| 792 | + | |
| 793 | +Without an fsync(2) call, there is no guarantee that the observed | |
| 794 | +data after a system crash will be either the old or the new data, but | |
| 795 | +in practice, the observed data after crash is often the old or new data | |
| 796 | +or a mix of both. | |
| 797 | + | |
| 798 | +When an overlayfs file is modified for the first time, copy up will | |
| 799 | +create a copy of the lower file and its parent directories in the upper | |
| 800 | +layer. Since the Linux filesystem API does not enforce any particular | |
| 801 | +ordering on storing changes without explicit fsync(2) calls, in case | |
| 802 | +of a system crash, the upper file could end up with no data at all | |
| 803 | +(i.e. zeros), which would be an unusual outcome. To avoid this | |
| 804 | +experience, overlayfs calls fsync(2) on the upper file before completing | |
| 805 | +data copy up with rename(2) or link(2) to make the copy up "atomic". | |
| 806 | + | |
| 807 | +By default, overlayfs does not explicitly call fsync(2) on copied up | |
| 808 | +directories or on metadata-only copy up, so it provides no guarantee to | |
| 809 | +persist the user's modification unless the user calls fsync(2). | |
| 810 | +The fsync during copy up only guarantees that if a copy up is observed | |
| 811 | +after a crash, the observed data is not zeroes or intermediate values | |
| 812 | +from the copy up staging area. | |
| 813 | + | |
| 814 | +On traditional local filesystems with a single journal (e.g. ext4, xfs), | |
| 815 | +fsync on a file also persists the parent directory changes, because they | |
| 816 | +are usually modified in the same transaction, so metadata durability during | |
| 817 | +data copy up effectively comes for free. Overlayfs further limits risk by | |
| 818 | +disallowing network filesystems as upper layer. | |
| 819 | + | |
| 820 | +Overlayfs can be tuned to prefer performance or durability when storing | |
| 821 | +to the underlying upper layer. This is controlled by the "fsync" mount | |
| 822 | +option, which supports these values: | |
| 823 | + | |
| 824 | +- "auto": (default) | |
| 825 | + Call fsync(2) on upper file before completion of data copy up. | |
| 826 | + No explicit fsync(2) on directory or metadata-only copy up. | |
| 827 | +- "strict": | |
| 828 | + Call fsync(2) on upper file and directories before completion of any | |
| 829 | + copy up. | |
| 830 | +- "volatile": [*] | |
| 831 | + Prefer performance over durability (see `Volatile mount`_) | |
| 832 | + | |
| 833 | +[*] The mount option "volatile" is an alias to "fsync=volatile". | |
| 834 | + | |
| 835 | + | |
| 786 | 836 | Volatile mount |
| 787 | 837 | -------------- |
| 788 | 838 | |
| @@ -1089,6 +1089,8 @@ Example output. You may not have all of these fields. · 1088 unmodified lines | ||
| 1089 | 1089 | CmaFree: 0 kB |
| 1090 | 1090 | Unaccepted: 0 kB |
| 1091 | 1091 | Balloon: 0 kB |
| 1092 | + GPUActive: 0 kB | |
| 1093 | + GPUReclaim: 0 kB | |
| 1092 | 1094 | HugePages_Total: 0 |
| 1093 | 1095 | HugePages_Free: 0 |
| 1094 | 1096 | HugePages_Rsvd: 0 |
| @@ -1269,6 +1271,12 @@ Unaccepted · 174 unmodified lines | ||
| 1269 | 1271 | Memory that has not been accepted by the guest |
| 1270 | 1272 | Balloon |
| 1271 | 1273 | Memory returned to Host by VM Balloon Drivers |
| 1274 | +GPUActive | |
| 1275 | + System memory allocated to active GPU objects | |
| 1276 | +GPUReclaim | |
| 1277 | + System memory stored in GPU pools for reuse. This memory is not | |
| 1278 | + counted in GPUActive. It is shrinker reclaimable memory kept in a reuse | |
| 1279 | + pool because it has non-standard page table attributes, like WC or UC. | |
| 1272 | 1280 | HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, Hugetlb |
| 1273 | 1281 | See Documentation/admin-guide/mm/hugetlbpage.rst. |
| 1274 | 1282 | DirectMap4k, DirectMap2M, DirectMap1G |
| @@ -104,18 +104,6 @@ VBLANK Helper Reference · 103 unmodified lines | ||
| 104 | 104 | .. kernel-doc:: drivers/gpu/drm/drm_vblank_helper.c |
| 105 | 105 | :export: |
| 106 | 106 | |
| 107 | −Simple KMS Helper Reference | |
| 108 | −=========================== | |
| 109 | − | |
| 110 | −.. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c | |
| 111 | − :doc: overview | |
| 112 | − | |
| 113 | −.. kernel-doc:: include/drm/drm_simple_kms_helper.h | |
| 114 | − :internal: | |
| 115 | − | |
| 116 | −.. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c | |
| 117 | − :export: | |
| 118 | − | |
| 119 | 107 | fbdev Helper Functions Reference |
| 120 | 108 | ================================ |
| 121 | 109 | |
| @@ -438,14 +438,14 @@ following expectations. · 437 unmodified lines | ||
| 438 | 438 | unknown consumer policy |
| 439 | 439 | =============== ======================================== |
| 440 | 440 | |
| 441 | −The only exception to this is ``WEDGED=none``, which signifies that the device | |
| 442 | −was temporarily 'wedged' at some point but was recovered from driver context | |
| 443 | −using device specific methods like reset. No explicit recovery is expected from | |
| 444 | −the consumer in this case, but it can still take additional steps like gathering | |
| 445 | −telemetry information (devcoredump, syslog). This is useful because the first | |
| 446 | −hang is usually the most critical one which can result in consequential hangs or | |
| 447 | −complete wedging. | |
| 448 | − | |
| 441 | +No Recovery | |
| 442 | +----------- | |
| 443 | + | |
| 444 | +Here ``WEDGED=none`` signifies that no recovery is expected from the consumer | |
| 445 | +but it can still try to gather telemetry information (devcoredump, syslog) for | |
| 446 | +debug purpose in order to root cause the hang. This is useful because the first | |
| 447 | +hang is usually the most critical one which can result in consequential hangs | |
| 448 | +or complete wedging. | |
| 449 | 449 | |
| 450 | 450 | Vendor Specific Recovery |
| 451 | 451 | ------------------------ |
| @@ -119,12 +119,6 @@ Simple DRM drivers to use as examples · 118 unmodified lines | ||
| 119 | 119 | The DRM subsystem contains a lot of helper functions to ease writing drivers for |
| 120 | 120 | simple graphic devices. For example, the `drivers/gpu/drm/tiny/` directory has a |
| 121 | 121 | set of drivers that are simple enough to be implemented in a single source file. |
| 122 | − | |
| 123 | −These drivers make use of the `struct drm_simple_display_pipe_funcs`, that hides | |
| 124 | −any complexity of the DRM subsystem and just requires drivers to implement a few | |
| 125 | −functions needed to operate the device. This could be used for devices that just | |
| 126 | −need a display pipeline with one full-screen scanout buffer feeding one output. | |
| 127 | − | |
| 128 | 122 | The tiny DRM drivers are good examples to understand how DRM drivers should look |
| 129 | 123 | like. Since are just a few hundreds lines of code, they are quite easy to read. |
| 130 | 124 | |
| @@ -51,82 +51,6 @@ There also have been considerations of ToPrimitive [2]. · 50 unmodified lines | ||
| 51 | 51 | | Link: https://lore.kernel.org/all/cover.1750689857.git.y.j3ms.n@gmail.com/ [1] |
| 52 | 52 | | Link: https://rust-for-linux.zulipchat.com/#narrow/channel/288089-General/topic/Implement.20.60FromPrimitive.60.20trait.20.2B.20derive.20macro.20for.20nova-core/with/541971854 [2] |
| 53 | 53 | |
| 54 | −Generic register abstraction [REGA] | |
| 55 | −----------------------------------- | |
| 56 | − | |
| 57 | −Work out how register constants and structures can be automatically generated | |
| 58 | −through generalized macros. | |
| 59 | − | |
| 60 | −Example: | |
| 61 | − | |
| 62 | −.. code-block:: rust | |
| 63 | − | |
| 64 | − register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ | |
| 65 | − MINOR_REVISION(3:0, RO), | |
| 66 | − MAJOR_REVISION(7:4, RO), | |
| 67 | − REVISION(7:0, RO), // Virtual register combining major and minor rev. | |
| 68 | − ]) | |
| 69 | − | |
| 70 | −This could expand to something like: | |
| 71 | − | |
| 72 | −.. code-block:: rust | |
| 73 | − | |
| 74 | − const BOOT0_OFFSET: usize = 0x00000000; | |
| 75 | − const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; | |
| 76 | − const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; | |
| 77 | − const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; | |
| 78 | − const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; | |
| 79 | − const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; | |
| 80 | − const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; | |
| 81 | − | |
| 82 | − struct Boot0(u32); | |
| 83 | − | |
| 84 | − impl Boot0 { | |
| 85 | − #[inline] | |
| 86 | − fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { | |
| 87 | − Self(bar.readl(BOOT0_OFFSET)) | |
| 88 | − } | |
| 89 | − | |
| 90 | − #[inline] | |
| 91 | − fn minor_revision(&self) -> u32 { | |
| 92 | − (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT | |
| 93 | − } | |
| 94 | − | |
| 95 | − #[inline] | |
| 96 | − fn major_revision(&self) -> u32 { | |
| 97 | − (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT | |
| 98 | − } | |
| 99 | − | |
| 100 | − #[inline] | |
| 101 | − fn revision(&self) -> u32 { | |
| 102 | − (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT | |
| 103 | − } | |
| 104 | − } | |
| 105 | − | |
| 106 | −Usage: | |
| 107 | − | |
| 108 | −.. code-block:: rust | |
| 109 | − | |
| 110 | − let bar = bar.try_access().ok_or(ENXIO)?; | |
| 111 | − | |
| 112 | − let boot0 = Boot0::read(&bar); | |
| 113 | − pr_info!("Revision: {}\n", boot0.revision()); | |
| 114 | − | |
| 115 | −A work-in-progress implementation currently resides in | |
| 116 | −`drivers/gpu/nova-core/regs/macros.rs` and is used in nova-core. It would be | |
| 117 | −nice to improve it (possibly using proc macros) and move it to the `kernel` | |
| 118 | −crate so it can be used by other components as well. | |
| 119 | − | |
| 120 | −Features desired before this happens: | |
| 121 | − | |
| 122 | −* Make I/O optional I/O (for field values that are not registers), | |
| 123 | −* Support other sizes than `u32`, | |
| 124 | −* Allow visibility control for registers and individual fields, | |
| 125 | −* Use Rust slice syntax to express fields ranges. | |
| 126 | − | |
| 127 | −| Complexity: Advanced | |
| 128 | −| Contact: Alexandre Courbot | |
| 129 | − | |
| 130 | 54 | Numerical operations [NUMM] |
| 131 | 55 | --------------------------- |
| 132 | 56 | |
| @@ -29,6 +29,38 @@ refactorings already and are an expert in the specific area · 28 unmodified lines | ||
| 29 | 29 | Subsystem-wide refactorings |
| 30 | 30 | =========================== |
| 31 | 31 | |
| 32 | +Open-code drm_simple_encoder_init() | |
| 33 | +----------------------------------- | |
| 34 | + | |
| 35 | +The helper drm_simple_encoder_init() was supposed to simplify encoder | |
| 36 | +initialization. Instead it only added an intermediate layer between atomic | |
| 37 | +modesetting and the DRM driver. | |
| 38 | + | |
| 39 | +The task here is to remove drm_simple_encoder_init(). Search for a driver | |
| 40 | +that calls drm_simple_encoder_init() and inline the helper. The driver will | |
| 41 | +also need its own instance of drm_encoder_funcs. | |
| 42 | + | |
| 43 | +Contact: Thomas Zimmermann, respective driver maintainer | |
| 44 | + | |
| 45 | +Level: Easy | |
| 46 | + | |
| 47 | +Replace struct drm_simple_display_pipe with regular atomic helpers | |
| 48 | +------------------------------------------------------------------ | |
| 49 | + | |
| 50 | +The data type struct drm_simple_display_pipe and its helpers were supposed | |
| 51 | +to simplify driver development. Instead they only added an intermediate layer | |
| 52 | +between atomic modesetting and the DRM driver. | |
| 53 | + | |
| 54 | +There are still drivers that use drm_simple_display_pipe. The task here is to | |
| 55 | +convert them to use regular atomic helpers. Search for a driver that calls | |
| 56 | +drm_simple_display_pipe_init() and inline all helpers from drm_simple_kms_helper.c | |
| 57 | +into the driver, such that no simple-KMS interfaces are required. Please also | |
| 58 | +rename all inlined fucntions according to driver conventions. | |
| 59 | + | |
| 60 | +Contact: Thomas Zimmermann, respective driver maintainer | |
| 61 | + | |
| 62 | +Level: Easy | |
| 63 | + | |
| 32 | 64 | Remove custom dumb_map_offset implementations |
| 33 | 65 | --------------------------------------------- |
| 34 | 66 | |
| @@ -27,10 +27,10 @@ for details. · 26 unmodified lines | ||
| 27 | 27 | Sysfs entries |
| 28 | 28 | ------------- |
| 29 | 29 | |
| 30 | −The following attributes are supported. Current maxim attribute | |
| 30 | +The following attributes are supported. Current maximum attribute | |
| 31 | 31 | is read-write, all other attributes are read-only. |
| 32 | 32 | |
| 33 | −in0_input Measured voltage in microvolts. | |
| 33 | +in0_input Measured voltage in millivolts. | |
| 34 | 34 | |
| 35 | −curr1_input Measured current in microamperes. | |
| 36 | −curr1_max_alarm Overcurrent alarm in microamperes. | |
| 35 | +curr1_input Measured current in milliamperes. | |
| 36 | +curr1_max Overcurrent shutdown threshold in milliamperes. | |
| @@ -51,8 +51,9 @@ temp1_max Provides thermal control temperature of the CPU package · 50 unmodified lines | ||
| 51 | 51 | temp1_crit Provides shutdown temperature of the CPU package which |
| 52 | 52 | is also known as the maximum processor junction |
| 53 | 53 | temperature, Tjmax or Tprochot. |
| 54 | −temp1_crit_hyst Provides the hysteresis value from Tcontrol to Tjmax of | |
| 55 | − the CPU package. | |
| 54 | +temp1_crit_hyst Provides the hysteresis temperature of the CPU | |
| 55 | + package. Returns Tcontrol, the temperature at which | |
| 56 | + the critical condition clears. | |
| 56 | 57 | |
| 57 | 58 | temp2_label "DTS" |
| 58 | 59 | temp2_input Provides current temperature of the CPU package scaled |
| @@ -62,8 +63,9 @@ temp2_max Provides thermal control temperature of the CPU package · 3 unmodified lines | ||
| 62 | 63 | temp2_crit Provides shutdown temperature of the CPU package which |
| 63 | 64 | is also known as the maximum processor junction |
| 64 | 65 | temperature, Tjmax or Tprochot. |
| 65 | −temp2_crit_hyst Provides the hysteresis value from Tcontrol to Tjmax of | |
| 66 | − the CPU package. | |
| 66 | +temp2_crit_hyst Provides the hysteresis temperature of the CPU | |
| 67 | + package. Returns Tcontrol, the temperature at which | |
| 68 | + the critical condition clears. | |
| 67 | 69 | |
| 68 | 70 | temp3_label "Tcontrol" |
| 69 | 71 | temp3_input Provides current Tcontrol temperature of the CPU |
| @@ -247,8 +247,8 @@ operations: · 246 unmodified lines | ||
| 247 | 247 | flags: [admin-perm] |
| 248 | 248 | |
| 249 | 249 | do: |
| 250 | − pre: net-shaper-nl-pre-doit | |
| 251 | − post: net-shaper-nl-post-doit | |
| 250 | + pre: net-shaper-nl-pre-doit-write | |
| 251 | + post: net-shaper-nl-post-doit-write | |
| 252 | 252 | request: |
| 253 | 253 | attributes: |
| 254 | 254 | - ifindex |
| @@ -278,8 +278,8 @@ operations: · 23 unmodified lines | ||
| 278 | 278 | flags: [admin-perm] |
| 279 | 279 | |
| 280 | 280 | do: |
| 281 | − pre: net-shaper-nl-pre-doit | |
| 282 | − post: net-shaper-nl-post-doit | |
| 281 | + pre: net-shaper-nl-pre-doit-write | |
| 282 | + post: net-shaper-nl-post-doit-write | |
| 283 | 283 | request: |
| 284 | 284 | attributes: *ns-binding |
| 285 | 285 | |
| @@ -309,8 +309,8 @@ operations: · 23 unmodified lines | ||
| 309 | 309 | flags: [admin-perm] |
| 310 | 310 | |
| 311 | 311 | do: |
| 312 | − pre: net-shaper-nl-pre-doit | |
| 313 | − post: net-shaper-nl-post-doit | |
| 312 | + pre: net-shaper-nl-pre-doit-write | |
| 313 | + post: net-shaper-nl-post-doit-write | |
| 314 | 314 | request: |
| 315 | 315 | attributes: |
| 316 | 316 | - ifindex |
| @@ -5,8 +5,138 @@ Security bugs · 4 unmodified lines | ||
| 5 | 5 | |
| 6 | 6 | Linux kernel developers take security very seriously. As such, we'd |
| 7 | 7 | like to know when a security bug is found so that it can be fixed and |
| 8 | −disclosed as quickly as possible. Please report security bugs to the | |
| 9 | −Linux kernel security team. | |
| 8 | +disclosed as quickly as possible. | |
| 9 | + | |
| 10 | +Preparing your report | |
| 11 | +--------------------- | |
| 12 | + | |
| 13 | +Like with any bug report, a security bug report requires a lot of analysis work | |
| 14 | +from the developers, so the more information you can share about the issue, the | |
| 15 | +better. Please review the procedure outlined in | |
| 16 | +Documentation/admin-guide/reporting-issues.rst if you are unclear about what | |
| 17 | +information is helpful. The following information are absolutely necessary in | |
| 18 | +**any** security bug report: | |
| 19 | + | |
| 20 | + * **affected kernel version range**: with no version indication, your report | |
| 21 | + will not be processed. A significant part of reports are for bugs that | |
| 22 | + have already been fixed, so it is extremely important that vulnerabilities | |
| 23 | + are verified on recent versions (development tree or latest stable | |
| 24 | + version), at least by verifying that the code has not changed since the | |
| 25 | + version where it was detected. | |
| 26 | + | |
| 27 | + * **description of the problem**: a detailed description of the problem, with | |
| 28 | + traces showing its manifestation, and why you consider that the observed | |
| 29 | + behavior as a problem in the kernel, is necessary. | |
| 30 | + | |
| 31 | + * **reproducer**: developers will need to be able to reproduce the problem to | |
| 32 | + consider a fix as effective. This includes both a way to trigger the issue | |
| 33 | + and a way to confirm it happens. A reproducer with low complexity | |
| 34 | + dependencies will be needed (source code, shell script, sequence of | |
| 35 | + instructions, file-system image etc). Binary-only executables are not | |
| 36 | + accepted. Working exploits are extremely helpful and will not be released | |
| 37 | + without consent from the reporter, unless they are already public. By | |
| 38 | + definition if an issue cannot be reproduced, it is not exploitable, thus it | |
| 39 | + is not a security bug. | |
| 40 | + | |
| 41 | + * **conditions**: if the bug depends on certain configuration options, | |
| 42 | + sysctls, permissions, timing, code modifications etc, these should be | |
| 43 | + indicated. | |
| 44 | + | |
| 45 | +In addition, the following information are highly desirable: | |
| 46 | + | |
| 47 | + * **suspected location of the bug**: the file names and functions where the | |
| 48 | + bug is suspected to be present are very important, at least to help forward | |
| 49 | + the report to the appropriate maintainers. When not possible (for example, | |
| 50 | + "system freezes each time I run this command"), the security team will help | |
| 51 | + identify the source of the bug. | |
| 52 | + | |
| 53 | + * **a proposed fix**: bug reporters who have analyzed the cause of a bug in | |
| 54 | + the source code almost always have an accurate idea on how to fix it, | |
| 55 | + because they spent a long time studying it and its implications. Proposing | |
| 56 | + a tested fix will save maintainers a lot of time, even if the fix ends up | |
| 57 | + not being the right one, because it helps understand the bug. When | |
| 58 | + proposing a tested fix, please always format it in a way that can be | |
| 59 | + immediately merged (see Documentation/process/submitting-patches.rst). | |
| 60 | + This will save some back-and-forth exchanges if it is accepted, and you | |
| 61 | + will be credited for finding and fixing this issue. Note that in this case | |
| 62 | + only a ``Signed-off-by:`` tag is needed, without ``Reported-by:`` when the | |
| 63 | + reporter and author are the same. | |
| 64 | + | |
| 65 | + * **mitigations**: very often during a bug analysis, some ways of mitigating | |
| 66 | + the issue appear. It is useful to share them, as they can be helpful to | |
| 67 | + keep end users protected during the time it takes them to apply the fix. | |
| 68 | + | |
| 69 | +Identifying contacts | |
| 70 | +-------------------- | |
| 71 | + | |
| 72 | +The most effective way to report a security bug is to send it directly to the | |
| 73 | +affected subsystem's maintainers and Cc: the Linux kernel security team. Do | |
| 74 | +not send it to a public list at this stage, unless you have good reasons to | |
| 75 | +consider the issue as being public or trivial to discover (e.g. result of a | |
| 76 | +widely available automated vulnerability scanning tool that can be repeated by | |
| 77 | +anyone). | |
| 78 | + | |
| 79 | +If you're sending a report for issues affecting multiple parts in the kernel, | |
| 80 | +even if they're fairly similar issues, please send individual messages (think | |
| 81 | +that maintainers will not all work on the issues at the same time). The only | |
| 82 | +exception is when an issue concerns closely related parts maintained by the | |
| 83 | +exact same subset of maintainers, and these parts are expected to be fixed all | |
| 84 | +at once by the same commit, then it may be acceptable to report them at once. | |
| 85 | + | |
| 86 | +One difficulty for most first-time reporters is to figure the right list of | |
| 87 | +recipients to send a report to. In the Linux kernel, all official maintainers | |
| 88 | +are trusted, so the consequences of accidentally including the wrong maintainer | |
| 89 | +are essentially a bit more noise for that person, i.e. nothing dramatic. As | |
| 90 | +such, a suitable method to figure the list of maintainers (which kernel | |
| 91 | +security officers use) is to rely on the get_maintainer.pl script, tuned to | |
| 92 | +only report maintainers. This script, when passed a file name, will look for | |
| 93 | +its path in the MAINTAINERS file to figure a hierarchical list of relevant | |
| 94 | +maintainers. Calling it a first time with the finest level of filtering will | |
| 95 | +most of the time return a short list of this specific file's maintainers:: | |
| 96 | + | |
| 97 | + $ ./scripts/get_maintainer.pl --no-l --no-r --pattern-depth 1 \ | |
| 98 | + drivers/example.c | |
| 99 | + Developer One <dev1@example.com> (maintainer:example driver) | |
| 100 | + Developer Two <dev2@example.org> (maintainer:example driver) | |
| 101 | + | |
| 102 | +These two maintainers should then receive the message. If the command does not | |
| 103 | +return anything, it means the affected file is part of a wider subsystem, so we | |
| 104 | +should be less specific:: | |
| 105 | + | |
| 106 | + $ ./scripts/get_maintainer.pl --no-l --no-r drivers/example.c | |
| 107 | + Developer One <dev1@example.com> (maintainer:example subsystem) | |
| 108 | + Developer Two <dev2@example.org> (maintainer:example subsystem) | |
| 109 | + Developer Three <dev3@example.com> (maintainer:example subsystem [GENERAL]) | |
| 110 | + Developer Four <dev4@example.org> (maintainer:example subsystem [GENERAL]) | |
| 111 | + | |
| 112 | +Here, picking the first, most specific ones, is sufficient. When the list is | |
| 113 | +long, it is possible to produce a comma-delimited e-mail address list on a | |
| 114 | +single line suitable for use in the To: field of a mailer like this:: | |
| 115 | + | |
| 116 | + $ ./scripts/get_maintainer.pl --no-tree --no-l --no-r --no-n --m \ | |
| 117 | + --no-git-fallback --no-substatus --no-rolestats --no-multiline \ | |
| 118 | + --pattern-depth 1 drivers/example.c | |
| 119 | + dev1@example.com, dev2@example.org | |
| 120 | + | |
| 121 | +or this for the wider list:: | |
| 122 | + | |
| 123 | + $ ./scripts/get_maintainer.pl --no-tree --no-l --no-r --no-n --m \ | |
| 124 | + --no-git-fallback --no-substatus --no-rolestats --no-multiline \ | |
| 125 | + drivers/example.c | |
| 126 | + dev1@example.com, dev2@example.org, dev3@example.com, dev4@example.org | |
| 127 | + | |
| 128 | +If at this point you're still facing difficulties spotting the right | |
| 129 | +maintainers, **and only in this case**, it's possible to send your report to | |
| 130 | +the Linux kernel security team only. Your message will be triaged, and you | |
| 131 | +will receive instructions about whom to contact, if needed. Your message may | |
| 132 | +equally be forwarded as-is to the relevant maintainers. | |
| 133 | + | |
| 134 | +Sending the report | |
| 135 | +------------------ | |
| 136 | + | |
| 137 | +Reports are to be sent over e-mail exclusively. Please use a working e-mail | |
| 138 | +address, preferably the same that you want to appear in ``Reported-by`` tags | |
| 139 | +if any. If unsure, send your report to yourself first. | |
| 10 | 140 | |
| 11 | 141 | The security team and maintainers almost always require additional |
| 12 | 142 | information beyond what was initially provided in a report and rely on |
| @@ -18,20 +148,12 @@ run additional tests. Reports where the reporter does not respond promptly · 5 unmodified lines | ||
| 18 | 148 | or cannot effectively discuss their findings may be abandoned if the |
| 19 | 149 | communication does not quickly improve. |
| 20 | 150 | |
| 21 | −As it is with any bug, the more information provided the easier it | |
| 22 | −will be to diagnose and fix. Please review the procedure outlined in | |
| 23 | −'Documentation/admin-guide/reporting-issues.rst' if you are unclear about what | |
| 24 | −information is helpful. Any exploit code is very helpful and will not | |
| 25 | −be released without consent from the reporter unless it has already been | |
| 26 | −made public. | |
| 27 | − | |
| 151 | +The report must be sent to maintainers, with the security team in ``Cc:``. | |
| 28 | 152 | The Linux kernel security team can be contacted by email at |
| 29 | 153 | <security@kernel.org>. This is a private list of security officers |
| 30 | −who will help verify the bug report and develop and release a fix. | |
| 31 | −If you already have a fix, please include it with your report, as | |
| 32 | −that can speed up the process considerably. It is possible that the | |
| 33 | −security team will bring in extra help from area maintainers to | |
| 34 | −understand and fix the security vulnerability. | |
| 154 | +who will help verify the bug report and assist developers working on a fix. | |
| 155 | +It is possible that the security team will bring in extra help from area | |
| 156 | +maintainers to understand and fix the security vulnerability. | |
| 35 | 157 | |
| 36 | 158 | Please send **plain text** emails without attachments where possible. |
| 37 | 159 | It is much harder to have a context-quoted discussion about a complex |
| @@ -42,7 +164,9 @@ reproduction steps, and follow it with a proposed fix, all in plain text. · 4 unmodified lines | ||
| 42 | 164 | Markdown, HTML and RST formatted reports are particularly frowned upon since |
| 43 | 165 | they're quite hard to read for humans and encourage to use dedicated viewers, |
| 44 | 166 | sometimes online, which by definition is not acceptable for a confidential |
| 45 | −security report. | |
| 167 | +security report. Note that some mailers tend to mangle formatting of plain | |
| 168 | +text by default, please consult Documentation/process/email-clients.rst for | |
| 169 | +more info. | |
| 46 | 170 | |
| 47 | 171 | Disclosure and embargoed information |
| 48 | 172 | ------------------------------------ |
| @@ -43,7 +43,6 @@ options should be enabled to use sched_ext: · 42 unmodified lines | ||
| 43 | 43 | CONFIG_DEBUG_INFO_BTF=y |
| 44 | 44 | CONFIG_BPF_JIT_ALWAYS_ON=y |
| 45 | 45 | CONFIG_BPF_JIT_DEFAULT_ON=y |
| 46 | − CONFIG_PAHOLE_HAS_BTF_TAG=y | |
| 47 | 46 | |
| 48 | 47 | sched_ext is used only when the BPF scheduler is loaded and running. |
| 49 | 48 | |
| @@ -58,7 +57,8 @@ in ``ops->flags``, all ``SCHED_NORMAL``, ``SCHED_BATCH``, ``SCHED_IDLE``, and · 8 unmodified lines | ||
| 58 | 57 | However, when the BPF scheduler is loaded and ``SCX_OPS_SWITCH_PARTIAL`` is |
| 59 | 58 | set in ``ops->flags``, only tasks with the ``SCHED_EXT`` policy are scheduled |
| 60 | 59 | by sched_ext, while tasks with ``SCHED_NORMAL``, ``SCHED_BATCH`` and |
| 61 | −``SCHED_IDLE`` policies are scheduled by the fair-class scheduler. | |
| 60 | +``SCHED_IDLE`` policies are scheduled by the fair-class scheduler which has | |
| 61 | +higher sched_class precedence than ``SCHED_EXT``. | |
| 62 | 62 | |
| 63 | 63 | Terminating the sched_ext scheduler program, triggering `SysRq-S`, or |
| 64 | 64 | detection of any internal error including stalled runnable tasks aborts the |
| @@ -345,6 +345,8 @@ Where to Look · 280 unmodified lines | ||
| 345 | 345 | The functions prefixed with ``scx_bpf_`` can be called from the BPF |
| 346 | 346 | scheduler. |
| 347 | 347 | |
| 348 | +* ``kernel/sched/ext_idle.c`` contains the built-in idle CPU selection policy. | |
| 349 | + | |
| 348 | 350 | * ``tools/sched_ext/`` hosts example BPF scheduler implementations. |
| 349 | 351 | |
| 350 | 352 | * ``scx_simple[.bpf].c``: Minimal global FIFO scheduler example using a |
| @@ -353,13 +355,35 @@ Where to Look · 2 unmodified lines | ||
| 353 | 355 | * ``scx_qmap[.bpf].c``: A multi-level FIFO scheduler supporting five |
| 354 | 356 | levels of priority implemented with ``BPF_MAP_TYPE_QUEUE``. |
| 355 | 357 | |
| 358 | + * ``scx_central[.bpf].c``: A central FIFO scheduler where all scheduling | |
| 359 | + decisions are made on one CPU, demonstrating ``LOCAL_ON`` dispatching, | |
| 360 | + tickless operation, and kthread preemption. | |
| 361 | + | |
| 362 | + * ``scx_cpu0[.bpf].c``: A scheduler that queues all tasks to a shared DSQ | |
| 363 | + and only dispatches them on CPU0 in FIFO order. Useful for testing bypass | |
| 364 | + behavior. | |
| 365 | + | |
| 366 | + * ``scx_flatcg[.bpf].c``: A flattened cgroup hierarchy scheduler | |
| 367 | + implementing hierarchical weight-based cgroup CPU control by compounding | |
| 368 | + each cgroup's share at every level into a single flat scheduling layer. | |
| 369 | + | |
| 370 | + * ``scx_pair[.bpf].c``: A core-scheduling example that always makes | |
| 371 | + sibling CPU pairs execute tasks from the same CPU cgroup. | |
| 372 | + | |
| 373 | + * ``scx_sdt[.bpf].c``: A variation of ``scx_simple`` demonstrating BPF | |
| 374 | + arena memory management for per-task data. | |
| 375 | + | |
| 376 | + * ``scx_userland[.bpf].c``: A minimal scheduler demonstrating user space | |
| 377 | + scheduling. Tasks with CPU affinity are direct-dispatched in FIFO order; | |
| 378 | + all others are scheduled in user space by a simple vruntime scheduler. | |
| 379 | + | |
| 356 | 380 | ABI Instability |
| 357 | 381 | =============== |
| 358 | 382 | |
| 359 | 383 | The APIs provided by sched_ext to BPF schedulers programs have no stability |
| 360 | 384 | guarantees. This includes the ops table callbacks and constants defined in |
| 361 | 385 | ``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in |
| 362 | −``kernel/sched/ext.c``. | |
| 386 | +``kernel/sched/ext.c`` and ``kernel/sched/ext_idle.c``. | |
| 363 | 387 | |
| 364 | 388 | While we will attempt to provide a relatively stable API surface when |
| 365 | 389 | possible, they are subject to change without warning between kernel |
| @@ -8,7 +8,7 @@ Landlock: unprivileged access control · 7 unmodified lines | ||
| 8 | 8 | ===================================== |
| 9 | 9 | |
| 10 | 10 | :Author: Mickaël Salaün |
| 11 | −:Date: January 2026 | |
| 11 | +:Date: March 2026 | |
| 12 | 12 | |
| 13 | 13 | The goal of Landlock is to enable restriction of ambient rights (e.g. global |
| 14 | 14 | filesystem or network access) for a set of processes. Because Landlock |
| @@ -197,12 +197,27 @@ similar backwards compatibility check is needed for the restrict flags · 182 unmodified lines | ||
| 197 | 197 | |
| 198 | 198 | .. code-block:: c |
| 199 | 199 | |
| 200 | − __u32 restrict_flags = LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON; | |
| 201 | − if (abi < 7) { | |
| 202 | − /* Clear logging flags unsupported before ABI 7. */ | |
| 200 | + __u32 restrict_flags = | |
| 201 | + LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON | | |
| 202 | + LANDLOCK_RESTRICT_SELF_TSYNC; | |
| 203 | + switch (abi) { | |
| 204 | + case 1 ... 6: | |
| 205 | + /* Removes logging flags for ABI < 7 */ | |
| 203 | 206 | restrict_flags &= ~(LANDLOCK_RESTRICT_SELF_LOG_SAME_EXEC_OFF | |
| 204 | 207 | LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON | |
| 205 | 208 | LANDLOCK_RESTRICT_SELF_LOG_SUBDOMAINS_OFF); |
| 209 | + __attribute__((fallthrough)); | |
| 210 | + case 7: | |
| 211 | + /* | |
| 212 | + * Removes multithreaded enforcement flag for ABI < 8 | |
| 213 | + * | |
| 214 | + * WARNING: Without this flag, calling landlock_restrict_self(2) is | |
| 215 | + * only equivalent if the calling process is single-threaded. Below | |
| 216 | + * ABI v8 (and as of ABI v8, when not using this flag), a Landlock | |
| 217 | + * policy would only be enforced for the calling thread and its | |
| 218 | + * children (and not for all threads, including parents and siblings). | |
| 219 | + */ | |
| 220 | + restrict_flags &= ~LANDLOCK_RESTRICT_SELF_TSYNC; | |
| 206 | 221 | } |
| 207 | 222 | |
| 208 | 223 | The next step is to restrict the current thread from gaining more privileges |
| @@ -8435,115 +8435,123 @@ KVM_CHECK_EXTENSION. · 8434 unmodified lines | ||
| 8435 | 8435 | |
| 8436 | 8436 | The valid bits in cap.args[0] are: |
| 8437 | 8437 | |
| 8438 | −=================================== ============================================ | |
| 8439 | − KVM_X86_QUIRK_LINT0_REENABLED By default, the reset value for the LVT | |
| 8440 | − LINT0 register is 0x700 (APIC_MODE_EXTINT). | |
| 8441 | − When this quirk is disabled, the reset value | |
| 8442 | − is 0x10000 (APIC_LVT_MASKED). | |
| 8443 | − | |
| 8444 | − KVM_X86_QUIRK_CD_NW_CLEARED By default, KVM clears CR0.CD and CR0.NW on | |
| 8445 | − AMD CPUs to workaround buggy guest firmware | |
| 8446 | − that runs in perpetuity with CR0.CD, i.e. | |
| 8447 | − with caches in "no fill" mode. | |
| 8448 | − | |
| 8449 | − When this quirk is disabled, KVM does not | |
| 8450 | − change the value of CR0.CD and CR0.NW. | |
| 8451 | − | |
| 8452 | − KVM_X86_QUIRK_LAPIC_MMIO_HOLE By default, the MMIO LAPIC interface is | |
| 8453 | − available even when configured for x2APIC | |
| 8454 | − mode. When this quirk is disabled, KVM | |
| 8455 | − disables the MMIO LAPIC interface if the | |
| 8456 | − LAPIC is in x2APIC mode. | |
| 8457 | − | |
| 8458 | − KVM_X86_QUIRK_OUT_7E_INC_RIP By default, KVM pre-increments %rip before | |
| 8459 | − exiting to userspace for an OUT instruction | |
| 8460 | − to port 0x7e. When this quirk is disabled, | |
| 8461 | − KVM does not pre-increment %rip before | |
| 8462 | − exiting to userspace. | |
| 8463 | − | |
| 8464 | − KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT When this quirk is disabled, KVM sets | |
| 8465 | − CPUID.01H:ECX[bit 3] (MONITOR/MWAIT) if | |
| 8466 | − IA32_MISC_ENABLE[bit 18] (MWAIT) is set. | |
| 8467 | − Additionally, when this quirk is disabled, | |
| 8468 | − KVM clears CPUID.01H:ECX[bit 3] if | |
| 8469 | − IA32_MISC_ENABLE[bit 18] is cleared. | |
| 8470 | − | |
| 8471 | − KVM_X86_QUIRK_FIX_HYPERCALL_INSN By default, KVM rewrites guest | |
| 8472 | − VMMCALL/VMCALL instructions to match the | |
| 8473 | − vendor's hypercall instruction for the | |
| 8474 | − system. When this quirk is disabled, KVM | |
| 8475 | − will no longer rewrite invalid guest | |
| 8476 | − hypercall instructions. Executing the | |
| 8477 | − incorrect hypercall instruction will | |
| 8478 | − generate a #UD within the guest. | |
| 8479 | − | |
| 8480 | −KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS By default, KVM emulates MONITOR/MWAIT (if | |
| 8481 | − they are intercepted) as NOPs regardless of | |
| 8482 | − whether or not MONITOR/MWAIT are supported | |
| 8483 | − according to guest CPUID. When this quirk | |
| 8484 | − is disabled and KVM_X86_DISABLE_EXITS_MWAIT | |
| 8485 | − is not set (MONITOR/MWAIT are intercepted), | |
| 8486 | − KVM will inject a #UD on MONITOR/MWAIT if | |
| 8487 | − they're unsupported per guest CPUID. Note, | |
| 8488 | − KVM will modify MONITOR/MWAIT support in | |
| 8489 | − guest CPUID on writes to MISC_ENABLE if | |
| 8490 | − KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT is | |
| 8491 | − disabled. | |
| 8492 | − | |
| 8493 | −KVM_X86_QUIRK_SLOT_ZAP_ALL By default, for KVM_X86_DEFAULT_VM VMs, KVM | |
| 8494 | − invalidates all SPTEs in all memslots and | |
| 8495 | − address spaces when a memslot is deleted or | |
| 8496 | − moved. When this quirk is disabled (or the | |
| 8497 | − VM type isn't KVM_X86_DEFAULT_VM), KVM only | |
| 8498 | − ensures the backing memory of the deleted | |
| 8499 | − or moved memslot isn't reachable, i.e KVM | |
| 8500 | − _may_ invalidate only SPTEs related to the | |
| 8501 | − memslot. | |
| 8502 | − | |
| 8503 | −KVM_X86_QUIRK_STUFF_FEATURE_MSRS By default, at vCPU creation, KVM sets the | |
| 8504 | − vCPU's MSR_IA32_PERF_CAPABILITIES (0x345), | |
| 8505 | − MSR_IA32_ARCH_CAPABILITIES (0x10a), | |
| 8506 | − MSR_PLATFORM_INFO (0xce), and all VMX MSRs | |
| 8507 | − (0x480..0x492) to the maximal capabilities | |
| 8508 | − supported by KVM. KVM also sets | |
| 8509 | − MSR_IA32_UCODE_REV (0x8b) to an arbitrary | |
| 8510 | − value (which is different for Intel vs. | |
| 8511 | − AMD). Lastly, when guest CPUID is set (by | |
| 8512 | − userspace), KVM modifies select VMX MSR | |
| 8513 | − fields to force consistency between guest | |
| 8514 | − CPUID and L2's effective ISA. When this | |
| 8515 | − quirk is disabled, KVM zeroes the vCPU's MSR | |
| 8516 | − values (with two exceptions, see below), | |
| 8517 | − i.e. treats the feature MSRs like CPUID | |
| 8518 | − leaves and gives userspace full control of | |
| 8519 | − the vCPU model definition. This quirk does | |
| 8520 | − not affect VMX MSRs CR0/CR4_FIXED1 (0x487 | |
| 8521 | − and 0x489), as KVM does now allow them to | |
| 8522 | − be set by userspace (KVM sets them based on | |
| 8523 | − guest CPUID, for safety purposes). | |
| 8524 | − | |
| 8525 | −KVM_X86_QUIRK_IGNORE_GUEST_PAT By default, on Intel platforms, KVM ignores | |
| 8526 | − guest PAT and forces the effective memory | |
| 8527 | − type to WB in EPT. The quirk is not available | |
| 8528 | − on Intel platforms which are incapable of | |
| 8529 | − safely honoring guest PAT (i.e., without CPU | |
| 8530 | − self-snoop, KVM always ignores guest PAT and | |
| 8531 | − forces effective memory type to WB). It is | |
| 8532 | − also ignored on AMD platforms or, on Intel, | |
| 8533 | − when a VM has non-coherent DMA devices | |
| 8534 | − assigned; KVM always honors guest PAT in | |
| 8535 | − such case. The quirk is needed to avoid | |
| 8536 | − slowdowns on certain Intel Xeon platforms | |
| 8537 | − (e.g. ICX, SPR) where self-snoop feature is | |
| 8538 | − supported but UC is slow enough to cause | |
| 8539 | − issues with some older guests that use | |
| 8540 | − UC instead of WC to map the video RAM. | |
| 8541 | − Userspace can disable the quirk to honor | |
| 8542 | − guest PAT if it knows that there is no such | |
| 8543 | − guest software, for example if it does not | |
| 8544 | − expose a bochs graphics device (which is | |
| 8545 | − known to have had a buggy driver). | |
| 8546 | −=================================== ============================================ | |
| 8438 | +======================================== ================================================ | |
| 8439 | +KVM_X86_QUIRK_LINT0_REENABLED By default, the reset value for the LVT | |
| 8440 | + LINT0 register is 0x700 (APIC_MODE_EXTINT). | |
| 8441 | + When this quirk is disabled, the reset value | |
| 8442 | + is 0x10000 (APIC_LVT_MASKED). | |
| 8443 | + | |
| 8444 | +KVM_X86_QUIRK_CD_NW_CLEARED By default, KVM clears CR0.CD and CR0.NW on | |
| 8445 | + AMD CPUs to workaround buggy guest firmware | |
| 8446 | + that runs in perpetuity with CR0.CD, i.e. | |
| 8447 | + with caches in "no fill" mode. | |
| 8448 | + | |
| 8449 | + When this quirk is disabled, KVM does not | |
| 8450 | + change the value of CR0.CD and CR0.NW. | |
| 8451 | + | |
| 8452 | +KVM_X86_QUIRK_LAPIC_MMIO_HOLE By default, the MMIO LAPIC interface is | |
| 8453 | + available even when configured for x2APIC | |
| 8454 | + mode. When this quirk is disabled, KVM | |
| 8455 | + disables the MMIO LAPIC interface if the | |
| 8456 | + LAPIC is in x2APIC mode. | |
| 8457 | + | |
| 8458 | +KVM_X86_QUIRK_OUT_7E_INC_RIP By default, KVM pre-increments %rip before | |
| 8459 | + exiting to userspace for an OUT instruction | |
| 8460 | + to port 0x7e. When this quirk is disabled, | |
| 8461 | + KVM does not pre-increment %rip before | |
| 8462 | + exiting to userspace. | |
| 8463 | + | |
| 8464 | +KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT When this quirk is disabled, KVM sets | |
| 8465 | + CPUID.01H:ECX[bit 3] (MONITOR/MWAIT) if | |
| 8466 | + IA32_MISC_ENABLE[bit 18] (MWAIT) is set. | |
| 8467 | + Additionally, when this quirk is disabled, | |
| 8468 | + KVM clears CPUID.01H:ECX[bit 3] if | |
| 8469 | + IA32_MISC_ENABLE[bit 18] is cleared. | |
| 8470 | + | |
| 8471 | +KVM_X86_QUIRK_FIX_HYPERCALL_INSN By default, KVM rewrites guest | |
| 8472 | + VMMCALL/VMCALL instructions to match the | |
| 8473 | + vendor's hypercall instruction for the | |
| 8474 | + system. When this quirk is disabled, KVM | |
| 8475 | + will no longer rewrite invalid guest | |
| 8476 | + hypercall instructions. Executing the | |
| 8477 | + incorrect hypercall instruction will | |
| 8478 | + generate a #UD within the guest. | |
| 8479 | + | |
| 8480 | +KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS By default, KVM emulates MONITOR/MWAIT (if | |
| 8481 | + they are intercepted) as NOPs regardless of | |
| 8482 | + whether or not MONITOR/MWAIT are supported | |
| 8483 | + according to guest CPUID. When this quirk | |
| 8484 | + is disabled and KVM_X86_DISABLE_EXITS_MWAIT | |
| 8485 | + is not set (MONITOR/MWAIT are intercepted), | |
| 8486 | + KVM will inject a #UD on MONITOR/MWAIT if | |
| 8487 | + they're unsupported per guest CPUID. Note, | |
| 8488 | + KVM will modify MONITOR/MWAIT support in | |
| 8489 | + guest CPUID on writes to MISC_ENABLE if | |
| 8490 | + KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT is | |
| 8491 | + disabled. | |
| 8492 | + | |
| 8493 | +KVM_X86_QUIRK_SLOT_ZAP_ALL By default, for KVM_X86_DEFAULT_VM VMs, KVM | |
| 8494 | + invalidates all SPTEs in all memslots and | |
| 8495 | + address spaces when a memslot is deleted or | |
| 8496 | + moved. When this quirk is disabled (or the | |
| 8497 | + VM type isn't KVM_X86_DEFAULT_VM), KVM only | |
| 8498 | + ensures the backing memory of the deleted | |
| 8499 | + or moved memslot isn't reachable, i.e KVM | |
| 8500 | + _may_ invalidate only SPTEs related to the | |
| 8501 | + memslot. | |
| 8502 | + | |
| 8503 | +KVM_X86_QUIRK_STUFF_FEATURE_MSRS By default, at vCPU creation, KVM sets the | |
| 8504 | + vCPU's MSR_IA32_PERF_CAPABILITIES (0x345), | |
| 8505 | + MSR_IA32_ARCH_CAPABILITIES (0x10a), | |
| 8506 | + MSR_PLATFORM_INFO (0xce), and all VMX MSRs | |
| 8507 | + (0x480..0x492) to the maximal capabilities | |
| 8508 | + supported by KVM. KVM also sets | |
| 8509 | + MSR_IA32_UCODE_REV (0x8b) to an arbitrary | |
| 8510 | + value (which is different for Intel vs. | |
| 8511 | + AMD). Lastly, when guest CPUID is set (by | |
| 8512 | + userspace), KVM modifies select VMX MSR | |
| 8513 | + fields to force consistency between guest | |
| 8514 | + CPUID and L2's effective ISA. When this | |
| 8515 | + quirk is disabled, KVM zeroes the vCPU's MSR | |
| 8516 | + values (with two exceptions, see below), | |
| 8517 | + i.e. treats the feature MSRs like CPUID | |
| 8518 | + leaves and gives userspace full control of | |
| 8519 | + the vCPU model definition. This quirk does | |
| 8520 | + not affect VMX MSRs CR0/CR4_FIXED1 (0x487 | |
| 8521 | + and 0x489), as KVM does now allow them to | |
| 8522 | + be set by userspace (KVM sets them based on | |
| 8523 | + guest CPUID, for safety purposes). | |
| 8524 | + | |
| 8525 | +KVM_X86_QUIRK_IGNORE_GUEST_PAT By default, on Intel platforms, KVM ignores | |
| 8526 | + guest PAT and forces the effective memory | |
| 8527 | + type to WB in EPT. The quirk is not available | |
| 8528 | + on Intel platforms which are incapable of | |
| 8529 | + safely honoring guest PAT (i.e., without CPU | |
| 8530 | + self-snoop, KVM always ignores guest PAT and | |
| 8531 | + forces effective memory type to WB). It is | |
| 8532 | + also ignored on AMD platforms or, on Intel, | |
| 8533 | + when a VM has non-coherent DMA devices | |
| 8534 | + assigned; KVM always honors guest PAT in | |
| 8535 | + such case. The quirk is needed to avoid | |
| 8536 | + slowdowns on certain Intel Xeon platforms | |
| 8537 | + (e.g. ICX, SPR) where self-snoop feature is | |
| 8538 | + supported but UC is slow enough to cause | |
| 8539 | + issues with some older guests that use | |
| 8540 | + UC instead of WC to map the video RAM. | |
| 8541 | + Userspace can disable the quirk to honor | |
| 8542 | + guest PAT if it knows that there is no such | |
| 8543 | + guest software, for example if it does not | |
| 8544 | + expose a bochs graphics device (which is | |
| 8545 | + known to have had a buggy driver). | |
| 8546 | + | |
| 8547 | +KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM By default, KVM relaxes the consistency | |
| 8548 | + check for GUEST_IA32_DEBUGCTL in vmcs12 | |
| 8549 | + to allow FREEZE_IN_SMM to be set. When | |
| 8550 | + this quirk is disabled, KVM requires this | |
| 8551 | + bit to be cleared. Note that the vmcs02 | |
| 8552 | + bit is still completely controlled by the | |
| 8553 | + host, regardless of the quirk setting. | |
| 8554 | +======================================== ================================================ | |
| 8547 | 8555 | |
| 8548 | 8556 | 7.32 KVM_CAP_MAX_VCPU_ID |
| 8549 | 8557 | ------------------------ |
| @@ -17,6 +17,8 @@ The acquisition orders for mutexes are as follows: · 16 unmodified lines | ||
| 17 | 17 | |
| 18 | 18 | - kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock |
| 19 | 19 | |
| 20 | +- vcpu->mutex is taken outside kvm->slots_lock and kvm->slots_arch_lock | |
| 21 | + | |
| 20 | 22 | - kvm->slots_lock is taken outside kvm->irq_lock, though acquiring |
| 21 | 23 | them together is quite rare. |
| 22 | 24 | |
| @@ -2195,6 +2195,7 @@ L: dri-devel@lists.freedesktop.org · 2194 unmodified lines | ||
| 2195 | 2195 | S: Supported |
| 2196 | 2196 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| 2197 | 2197 | F: Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml |
| 2198 | +F: drivers/gpu/drm/ci/xfails/panthor* | |
| 2198 | 2199 | F: drivers/gpu/drm/panthor/ |
| 2199 | 2200 | F: include/uapi/drm/panthor_drm.h |
| 2200 | 2201 | |
| @@ -3986,7 +3987,7 @@ F: drivers/hwmon/asus-ec-sensors.c · 1785 unmodified lines | ||
| 3986 | 3987 | ASUS NOTEBOOKS AND EEEPC ACPI/WMI EXTRAS DRIVERS |
| 3987 | 3988 | M: Corentin Chary <corentin.chary@gmail.com> |
| 3988 | 3989 | M: Luke D. Jones <luke@ljones.dev> |
| 3989 | −M: Denis Benato <benato.denis96@gmail.com> | |
| 3990 | +M: Denis Benato <denis.benato@linux.dev> | |
| 3990 | 3991 | L: platform-driver-x86@vger.kernel.org |
| 3991 | 3992 | S: Maintained |
| 3992 | 3993 | W: https://asus-linux.org/ |
| @@ -4022,7 +4023,7 @@ F: drivers/hwmon/asus_wmi_sensors.c · 29 unmodified lines | ||
| 4022 | 4023 | ASYMMETRIC KEYS |
| 4023 | 4024 | M: David Howells <dhowells@redhat.com> |
| 4024 | 4025 | M: Lukas Wunner <lukas@wunner.de> |
| 4025 | −M: Ignat Korchagin <ignat@cloudflare.com> | |
| 4026 | +M: Ignat Korchagin <ignat@linux.win> | |
| 4026 | 4027 | L: keyrings@vger.kernel.org |
| 4027 | 4028 | L: linux-crypto@vger.kernel.org |
| 4028 | 4029 | S: Maintained |
| @@ -4035,7 +4036,7 @@ F: include/linux/verification.h · 6 unmodified lines | ||
| 4035 | 4036 | |
| 4036 | 4037 | ASYMMETRIC KEYS - ECDSA |
| 4037 | 4038 | M: Lukas Wunner <lukas@wunner.de> |
| 4038 | −M: Ignat Korchagin <ignat@cloudflare.com> | |
| 4039 | +M: Ignat Korchagin <ignat@linux.win> | |
| 4039 | 4040 | R: Stefan Berger <stefanb@linux.ibm.com> |
| 4040 | 4041 | L: linux-crypto@vger.kernel.org |
| 4041 | 4042 | S: Maintained |
| @@ -4045,14 +4046,14 @@ F: include/crypto/ecc* · 3 unmodified lines | ||
| 4045 | 4046 | |
| 4046 | 4047 | ASYMMETRIC KEYS - GOST |
| 4047 | 4048 | M: Lukas Wunner <lukas@wunner.de> |
| 4048 | −M: Ignat Korchagin <ignat@cloudflare.com> | |
| 4049 | +M: Ignat Korchagin <ignat@linux.win> | |
| 4049 | 4050 | L: linux-crypto@vger.kernel.org |
| 4050 | 4051 | S: Odd fixes |
| 4051 | 4052 | F: crypto/ecrdsa* |
| 4052 | 4053 | |
| 4053 | 4054 | ASYMMETRIC KEYS - RSA |
| 4054 | 4055 | M: Lukas Wunner <lukas@wunner.de> |
| 4055 | −M: Ignat Korchagin <ignat@cloudflare.com> | |
| 4056 | +M: Ignat Korchagin <ignat@linux.win> | |
| 4056 | 4057 | L: linux-crypto@vger.kernel.org |
| 4057 | 4058 | S: Maintained |
| 4058 | 4059 | F: crypto/rsa* |
| @@ -7534,6 +7535,7 @@ F: include/linux/*fence.h · 3475 unmodified lines | ||
| 7534 | 7535 | F: include/linux/dma-buf.h |
| 7535 | 7536 | F: include/linux/dma-buf/ |
| 7536 | 7537 | F: include/linux/dma-resv.h |
| 7538 | +F: rust/helpers/dma-resv.c | |
| 7537 | 7539 | K: \bdma_(?:buf|fence|resv)\b |
| 7538 | 7540 | |
| 7539 | 7541 | DMA GENERIC OFFLOAD ENGINE SUBSYSTEM |
| @@ -7998,7 +8000,9 @@ F: Documentation/devicetree/bindings/display/himax,hx8357.yaml · 458 unmodified lines | ||
| 7998 | 8000 | F: drivers/gpu/drm/tiny/hx8357d.c |
| 7999 | 8001 | |
| 8000 | 8002 | DRM DRIVER FOR HYPERV SYNTHETIC VIDEO DEVICE |
| 8001 | −M: Deepak Rawat <drawat.floss@gmail.com> | |
| 8003 | +M: Dexuan Cui <decui@microsoft.com> | |
| 8004 | +M: Long Li <longli@microsoft.com> | |
| 8005 | +M: Saurabh Sengar <ssengar@linux.microsoft.com> | |
| 8002 | 8006 | L: linux-hyperv@vger.kernel.org |
| 8003 | 8007 | L: dri-devel@lists.freedesktop.org |
| 8004 | 8008 | S: Maintained |
| @@ -8027,8 +8031,9 @@ F: drivers/gpu/drm/panel/panel-ilitek-ili9805.c · 22 unmodified lines | ||
| 8027 | 8031 | |
| 8028 | 8032 | DRM DRIVER FOR ILITEK ILI9806E PANELS |
| 8029 | 8033 | M: Michael Walle <mwalle@kernel.org> |
| 8034 | +M: Dario Binacchi <dario.binacchi@amarulasolutions.com> | |
| 8030 | 8035 | S: Maintained |
| 8031 | −F: drivers/gpu/drm/panel/panel-ilitek-ili9806e.c | |
| 8036 | +F: drivers/gpu/drm/panel/panel-ilitek-ili9806e-* | |
| 8032 | 8037 | |
| 8033 | 8038 | DRM DRIVER FOR JADARD JD9365DA-H3 MIPI-DSI LCD PANELS |
| 8034 | 8039 | M: Jagan Teki <jagan@edgeble.ai> |
| @@ -8301,6 +8306,11 @@ S: Maintained · 266 unmodified lines | ||
| 8301 | 8306 | F: Documentation/devicetree/bindings/display/panel/samsung,s6e3ha8.yaml |
| 8302 | 8307 | F: drivers/gpu/drm/panel/panel-samsung-s6e3ha8.c |
| 8303 | 8308 | |
| 8309 | +DRM DRIVER FOR SAMSUNG S6E8FC0 PANELS | |
| 8310 | +M: Yedaya Katsman <yedaya.ka@gmail.com> | |
| 8311 | +S: Maintained | |
| 8312 | +F: drivers/gpu/drm/panel/panel-samsung-s6e8fc0-m1906f9.c | |
| 8313 | + | |
| 8304 | 8314 | DRM DRIVER FOR SAMSUNG SOFEF00 DDIC |
| 8305 | 8315 | M: David Heidelberg <david@ixit.cz> |
| 8306 | 8316 | M: Casey Connolly <casey.connolly@linaro.org> |
| @@ -8505,7 +8515,10 @@ T: git https://gitlab.freedesktop.org/drm/rust/kernel.git · 198 unmodified lines | ||
| 8505 | 8515 | F: drivers/gpu/drm/nova/ |
| 8506 | 8516 | F: drivers/gpu/drm/tyr/ |
| 8507 | 8517 | F: drivers/gpu/nova-core/ |
| 8518 | +F: rust/helpers/gpu.c | |
| 8508 | 8519 | F: rust/kernel/drm/ |
| 8520 | +F: rust/kernel/gpu.rs | |
| 8521 | +F: rust/kernel/gpu/ | |
| 8509 | 8522 | |
| 8510 | 8523 | DRM DRIVERS FOR ALLWINNER A10 |
| 8511 | 8524 | M: Chen-Yu Tsai <wens@kernel.org> |
| @@ -8632,9 +8645,14 @@ F: drivers/gpu/drm/lima/ · 120 unmodified lines | ||
| 8632 | 8645 | F: include/uapi/drm/lima_drm.h |
| 8633 | 8646 | |
| 8634 | 8647 | DRM DRIVERS FOR LOONGSON |
| 8635 | −M: Sui Jingfeng <suijingfeng@loongson.cn> | |
| 8648 | +M: Jianmin Lv <lvjianmin@loongson.cn> | |
| 8649 | +M: Qianhai Wu <wuqianhai@loongson.cn> | |
| 8650 | +R: Huacai Chen <chenhuacai@kernel.org> | |
| 8651 | +R: Mingcong Bai <jeffbai@aosc.io> | |
| 8652 | +R: Xi Ruoyao <xry111@xry111.site> | |
| 8653 | +R: Icenowy Zheng <zhengxingda@iscas.ac.cn> | |
| 8636 | 8654 | L: dri-devel@lists.freedesktop.org |
| 8637 | −S: Supported | |
| 8655 | +S: Maintained | |
| 8638 | 8656 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| 8639 | 8657 | F: drivers/gpu/drm/loongson/ |
| 8640 | 8658 | |
| @@ -8918,7 +8936,7 @@ F: include/drm/ttm/ · 277 unmodified lines | ||
| 8918 | 8936 | GPU BUDDY ALLOCATOR |
| 8919 | 8937 | M: Matthew Auld <matthew.auld@intel.com> |
| 8920 | 8938 | M: Arun Pravin <arunpravin.paneerselvam@amd.com> |
| 8921 | −R: Christian Koenig <christian.koenig@amd.com> | |
| 8939 | +R: Joel Fernandes <joelagnelf@nvidia.com> | |
| 8922 | 8940 | L: dri-devel@lists.freedesktop.org |
| 8923 | 8941 | S: Maintained |
| 8924 | 8942 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| @@ -8927,6 +8945,9 @@ F: drivers/gpu/drm/drm_buddy.c · 2 unmodified lines | ||
| 8927 | 8945 | F: drivers/gpu/tests/gpu_buddy_test.c |
| 8928 | 8946 | F: include/drm/drm_buddy.h |
| 8929 | 8947 | F: include/linux/gpu_buddy.h |
| 8948 | +F: rust/helpers/gpu.c | |
| 8949 | +F: rust/kernel/gpu.rs | |
| 8950 | +F: rust/kernel/gpu/ | |
| 8930 | 8951 | |
| 8931 | 8952 | DRM AUTOMATED TESTING |
| 8932 | 8953 | M: Helen Koike <helen.fornazier@gmail.com> |
| @@ -9628,7 +9649,12 @@ F: include/linux/ext2* · 695 unmodified lines | ||
| 9628 | 9649 | |
| 9629 | 9650 | EXT4 FILE SYSTEM |
| 9630 | 9651 | M: "Theodore Ts'o" <tytso@mit.edu> |
| 9631 | −M: Andreas Dilger <adilger.kernel@dilger.ca> | |
| 9652 | +R: Andreas Dilger <adilger.kernel@dilger.ca> | |
| 9653 | +R: Baokun Li <libaokun@linux.alibaba.com> | |
| 9654 | +R: Jan Kara <jack@suse.cz> | |
| 9655 | +R: Ojaswin Mujoo <ojaswin@linux.ibm.com> | |
| 9656 | +R: Ritesh Harjani (IBM) <ritesh.list@gmail.com> | |
| 9657 | +R: Zhang Yi <yi.zhang@huawei.com> | |
| 9632 | 9658 | L: linux-ext4@vger.kernel.org |
| 9633 | 9659 | S: Maintained |
| 9634 | 9660 | W: http://ext4.wiki.kernel.org |
| @@ -12025,7 +12051,6 @@ I2C SUBSYSTEM · 2390 unmodified lines | ||
| 12025 | 12051 | M: Wolfram Sang <wsa+renesas@sang-engineering.com> |
| 12026 | 12052 | L: linux-i2c@vger.kernel.org |
| 12027 | 12053 | S: Maintained |
| 12028 | −W: https://i2c.wiki.kernel.org/ | |
| 12029 | 12054 | Q: https://patchwork.ozlabs.org/project/linux-i2c/list/ |
| 12030 | 12055 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux.git |
| 12031 | 12056 | F: Documentation/i2c/ |
| @@ -12051,7 +12076,6 @@ I2C SUBSYSTEM HOST DRIVERS · 19 unmodified lines | ||
| 12051 | 12076 | M: Andi Shyti <andi.shyti@kernel.org> |
| 12052 | 12077 | L: linux-i2c@vger.kernel.org |
| 12053 | 12078 | S: Maintained |
| 12054 | −W: https://i2c.wiki.kernel.org/ | |
| 12055 | 12079 | Q: https://patchwork.ozlabs.org/project/linux-i2c/list/ |
| 12056 | 12080 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/andi.shyti/linux.git |
| 12057 | 12081 | F: Documentation/devicetree/bindings/i2c/ |
| @@ -16374,7 +16398,6 @@ F: net/dsa/tag_mtk.c · 4316 unmodified lines | ||
| 16374 | 16398 | |
| 16375 | 16399 | MEDIATEK T7XX 5G WWAN MODEM DRIVER |
| 16376 | 16400 | M: Chandrashekar Devegowda <chandrashekar.devegowda@intel.com> |
| 16377 | −R: Chiranjeevi Rapolu <chiranjeevi.rapolu@linux.intel.com> | |
| 16378 | 16401 | R: Liu Haijun <haijun.liu@mediatek.com> |
| 16379 | 16402 | R: Ricardo Martinez <ricardo.martinez@linux.intel.com> |
| 16380 | 16403 | L: netdev@vger.kernel.org |
| @@ -16659,7 +16682,7 @@ F: mm/balloon.c · 278 unmodified lines | ||
| 16659 | 16682 | MEMORY MANAGEMENT - CORE |
| 16660 | 16683 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16661 | 16684 | M: David Hildenbrand <david@kernel.org> |
| 16662 | −R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 16685 | +R: Lorenzo Stoakes <ljs@kernel.org> | |
| 16663 | 16686 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16664 | 16687 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16665 | 16688 | R: Mike Rapoport <rppt@kernel.org> |
| @@ -16789,7 +16812,7 @@ F: mm/workingset.c · 123 unmodified lines | ||
| 16789 | 16812 | MEMORY MANAGEMENT - MISC |
| 16790 | 16813 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16791 | 16814 | M: David Hildenbrand <david@kernel.org> |
| 16792 | −R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 16815 | +R: Lorenzo Stoakes <ljs@kernel.org> | |
| 16793 | 16816 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16794 | 16817 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16795 | 16818 | R: Mike Rapoport <rppt@kernel.org> |
| @@ -16880,7 +16903,7 @@ R: David Hildenbrand <david@kernel.org> · 84 unmodified lines | ||
| 16880 | 16903 | R: Michal Hocko <mhocko@kernel.org> |
| 16881 | 16904 | R: Qi Zheng <zhengqi.arch@bytedance.com> |
| 16882 | 16905 | R: Shakeel Butt <shakeel.butt@linux.dev> |
| 16883 | −R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 16906 | +R: Lorenzo Stoakes <ljs@kernel.org> | |
| 16884 | 16907 | L: linux-mm@kvack.org |
| 16885 | 16908 | S: Maintained |
| 16886 | 16909 | F: mm/vmscan.c |
| @@ -16889,11 +16912,11 @@ F: mm/workingset.c · 2 unmodified lines | ||
| 16889 | 16912 | MEMORY MANAGEMENT - RMAP (REVERSE MAPPING) |
| 16890 | 16913 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16891 | 16914 | M: David Hildenbrand <david@kernel.org> |
| 16892 | −M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 16915 | +M: Lorenzo Stoakes <ljs@kernel.org> | |
| 16893 | 16916 | R: Rik van Riel <riel@surriel.com> |
| 16894 | 16917 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16895 | 16918 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16896 | −R: Harry Yoo <harry.yoo@oracle.com> | |
| 16919 | +R: Harry Yoo <harry@kernel.org> | |
| 16897 | 16920 | R: Jann Horn <jannh@google.com> |
| 16898 | 16921 | L: linux-mm@kvack.org |
| 16899 | 16922 | S: Maintained |
| @@ -16934,7 +16957,7 @@ F: mm/swapfile.c · 34 unmodified lines | ||
| 16934 | 16957 | MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE) |
| 16935 | 16958 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16936 | 16959 | M: David Hildenbrand <david@kernel.org> |
| 16937 | −M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 16960 | +M: Lorenzo Stoakes <ljs@kernel.org> | |
| 16938 | 16961 | R: Zi Yan <ziy@nvidia.com> |
| 16939 | 16962 | R: Baolin Wang <baolin.wang@linux.alibaba.com> |
| 16940 | 16963 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| @@ -16974,7 +16997,7 @@ F: tools/testing/selftests/mm/uffd-*.[ch] · 33 unmodified lines | ||
| 16974 | 16997 | |
| 16975 | 16998 | MEMORY MANAGEMENT - RUST |
| 16976 | 16999 | M: Alice Ryhl <aliceryhl@google.com> |
| 16977 | −R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 17000 | +R: Lorenzo Stoakes <ljs@kernel.org> | |
| 16978 | 17001 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16979 | 17002 | L: linux-mm@kvack.org |
| 16980 | 17003 | L: rust-for-linux@vger.kernel.org |
| @@ -16990,7 +17013,7 @@ F: rust/kernel/page.rs · 9 unmodified lines | ||
| 16990 | 17013 | MEMORY MAPPING |
| 16991 | 17014 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16992 | 17015 | M: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16993 | −M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 17016 | +M: Lorenzo Stoakes <ljs@kernel.org> | |
| 16994 | 17017 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16995 | 17018 | R: Jann Horn <jannh@google.com> |
| 16996 | 17019 | R: Pedro Falcato <pfalcato@suse.de> |
| @@ -17020,7 +17043,7 @@ MEMORY MAPPING - LOCKING · 23 unmodified lines | ||
| 17020 | 17043 | M: Andrew Morton <akpm@linux-foundation.org> |
| 17021 | 17044 | M: Suren Baghdasaryan <surenb@google.com> |
| 17022 | 17045 | M: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 17023 | −M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 17046 | +M: Lorenzo Stoakes <ljs@kernel.org> | |
| 17024 | 17047 | R: Vlastimil Babka <vbabka@kernel.org> |
| 17025 | 17048 | R: Shakeel Butt <shakeel.butt@linux.dev> |
| 17026 | 17049 | L: linux-mm@kvack.org |
| @@ -17035,7 +17058,7 @@ F: mm/mmap_lock.c · 8 unmodified lines | ||
| 17035 | 17058 | MEMORY MAPPING - MADVISE (MEMORY ADVICE) |
| 17036 | 17059 | M: Andrew Morton <akpm@linux-foundation.org> |
| 17037 | 17060 | M: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 17038 | −M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 17061 | +M: Lorenzo Stoakes <ljs@kernel.org> | |
| 17039 | 17062 | M: David Hildenbrand <david@kernel.org> |
| 17040 | 17063 | R: Vlastimil Babka <vbabka@kernel.org> |
| 17041 | 17064 | R: Jann Horn <jannh@google.com> |
| @@ -21954,7 +21977,7 @@ F: drivers/media/radio/radio-tea5777.c · 4912 unmodified lines | ||
| 21954 | 21977 | |
| 21955 | 21978 | RADOS BLOCK DEVICE (RBD) |
| 21956 | 21979 | M: Ilya Dryomov <idryomov@gmail.com> |
| 21957 | −R: Dongsheng Yang <dongsheng.yang@easystack.cn> | |
| 21980 | +R: Dongsheng Yang <dongsheng.yang@linux.dev> | |
| 21958 | 21981 | L: ceph-devel@vger.kernel.org |
| 21959 | 21982 | S: Supported |
| 21960 | 21983 | W: http://ceph.com/ |
| @@ -22283,6 +22306,16 @@ L: linux-wireless@vger.kernel.org · 322 unmodified lines | ||
| 22283 | 22306 | S: Orphan |
| 22284 | 22307 | F: drivers/net/wireless/rsi/ |
| 22285 | 22308 | |
| 22309 | +RELAY | |
| 22310 | +M: Andrew Morton <akpm@linux-foundation.org> | |
| 22311 | +M: Jens Axboe <axboe@kernel.dk> | |
| 22312 | +M: Jason Xing <kernelxing@tencent.com> | |
| 22313 | +L: linux-kernel@vger.kernel.org | |
| 22314 | +S: Maintained | |
| 22315 | +F: Documentation/filesystems/relay.rst | |
| 22316 | +F: include/linux/relay.h | |
| 22317 | +F: kernel/relay.c | |
| 22318 | + | |
| 22286 | 22319 | REGISTER MAP ABSTRACTION |
| 22287 | 22320 | M: Mark Brown <broonie@kernel.org> |
| 22288 | 22321 | L: linux-kernel@vger.kernel.org |
| @@ -23173,7 +23206,7 @@ K: \b(?i:rust)\b · 884 unmodified lines | ||
| 23173 | 23206 | |
| 23174 | 23207 | RUST [ALLOC] |
| 23175 | 23208 | M: Danilo Krummrich <dakr@kernel.org> |
| 23176 | −R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | |
| 23209 | +R: Lorenzo Stoakes <ljs@kernel.org> | |
| 23177 | 23210 | R: Vlastimil Babka <vbabka@kernel.org> |
| 23178 | 23211 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 23179 | 23212 | R: Uladzislau Rezki <urezki@gmail.com> |
| @@ -23183,6 +23216,15 @@ T: git https://github.com/Rust-for-Linux/linux.git alloc-next · 3 unmodified lines | ||
| 23183 | 23216 | F: rust/kernel/alloc.rs |
| 23184 | 23217 | F: rust/kernel/alloc/ |
| 23185 | 23218 | |
| 23219 | +RUST [INTEROP] | |
| 23220 | +M: Joel Fernandes <joelagnelf@nvidia.com> | |
| 23221 | +M: Alexandre Courbot <acourbot@nvidia.com> | |
| 23222 | +L: rust-for-linux@vger.kernel.org | |
| 23223 | +S: Maintained | |
| 23224 | +T: git https://github.com/Rust-for-Linux/linux.git interop-next | |
| 23225 | +F: rust/kernel/interop.rs | |
| 23226 | +F: rust/kernel/interop/ | |
| 23227 | + | |
| 23186 | 23228 | RUST [NUM] |
| 23187 | 23229 | M: Alexandre Courbot <acourbot@nvidia.com> |
| 23188 | 23230 | R: Yury Norov <yury.norov@gmail.com> |
| @@ -24350,11 +24392,12 @@ F: drivers/nvmem/layouts/sl28vpd.c · 1161 unmodified lines | ||
| 24350 | 24392 | |
| 24351 | 24393 | SLAB ALLOCATOR |
| 24352 | 24394 | M: Vlastimil Babka <vbabka@kernel.org> |
| 24395 | +M: Harry Yoo <harry@kernel.org> | |
| 24353 | 24396 | M: Andrew Morton <akpm@linux-foundation.org> |
| 24397 | +R: Hao Li <hao.li@linux.dev> | |
| 24354 | 24398 | R: Christoph Lameter <cl@gentwo.org> |
| 24355 | 24399 | R: David Rientjes <rientjes@google.com> |
| 24356 | 24400 | R: Roman Gushchin <roman.gushchin@linux.dev> |
| 24357 | −R: Harry Yoo <harry.yoo@oracle.com> | |
| 24358 | 24401 | L: linux-mm@kvack.org |
| 24359 | 24402 | S: Maintained |
| 24360 | 24403 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git |
| @@ -24908,9 +24951,9 @@ F: drivers/clk/spear/ · 547 unmodified lines | ||
| 24908 | 24951 | F: drivers/pinctrl/spear/ |
| 24909 | 24952 | |
| 24910 | 24953 | SPI NOR SUBSYSTEM |
| 24911 | −M: Tudor Ambarus <tudor.ambarus@linaro.org> | |
| 24912 | 24954 | M: Pratyush Yadav <pratyush@kernel.org> |
| 24913 | 24955 | M: Michael Walle <mwalle@kernel.org> |
| 24956 | +R: Takahiro Kuwano <takahiro.kuwano@infineon.com> | |
| 24914 | 24957 | L: linux-mtd@lists.infradead.org |
| 24915 | 24958 | S: Maintained |
| 24916 | 24959 | W: http://www.linux-mtd.infradead.org/ |
| @@ -25765,6 +25808,7 @@ F: include/net/pkt_cls.h · 848 unmodified lines | ||
| 25765 | 25808 | F: include/net/pkt_sched.h |
| 25766 | 25809 | F: include/net/sch_priv.h |
| 25767 | 25810 | F: include/net/tc_act/ |
| 25811 | +F: include/net/tc_wrapper.h | |
| 25768 | 25812 | F: include/uapi/linux/pkt_cls.h |
| 25769 | 25813 | F: include/uapi/linux/pkt_sched.h |
| 25770 | 25814 | F: include/uapi/linux/tc_act/ |
| @@ -2,7 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | VERSION = 7 |
| 3 | 3 | PATCHLEVEL = 0 |
| 4 | 4 | SUBLEVEL = 0 |
| 5 | −EXTRAVERSION = -rc3 | |
| 5 | +EXTRAVERSION = -rc7 | |
| 6 | 6 | NAME = Baby Opossum Posse |
| 7 | 7 | |
| 8 | 8 | # *DOCUMENTATION* |
| @@ -476,6 +476,7 @@ KBUILD_USERLDFLAGS := $(USERLDFLAGS) · 467 unmodified lines | ||
| 476 | 476 | export rust_common_flags := --edition=2021 \ |
| 477 | 477 | -Zbinary_dep_depinfo=y \ |
| 478 | 478 | -Astable_features \ |
| 479 | + -Aunused_features \ | |
| 479 | 480 | -Dnon_ascii_idents \ |
| 480 | 481 | -Dunsafe_op_in_unsafe_fn \ |
| 481 | 482 | -Wmissing_docs \ |
| @@ -1113,6 +1114,9 @@ KBUILD_CFLAGS += -fno-builtin-wcslen · 631 unmodified lines | ||
| 1113 | 1114 | # change __FILE__ to the relative path to the source directory |
| 1114 | 1115 | ifdef building_out_of_srctree |
| 1115 | 1116 | KBUILD_CPPFLAGS += -fmacro-prefix-map=$(srcroot)/= |
| 1117 | +ifeq ($(call rustc-option-yn, --remap-path-scope=macro),y) | |
| 1118 | +KBUILD_RUSTFLAGS += --remap-path-prefix=$(srcroot)/= --remap-path-scope=macro | |
| 1119 | +endif | |
| 1116 | 1120 | endif |
| 1117 | 1121 | |
| 1118 | 1122 | # include additional Makefiles when needed |
| @@ -1650,7 +1654,7 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ · 531 unmodified lines | ||
| 1650 | 1654 | modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ |
| 1651 | 1655 | compile_commands.json rust/test \ |
| 1652 | 1656 | rust-project.json .vmlinux.objs .vmlinux.export.c \ |
| 1653 | − .builtin-dtbs-list .builtin-dtb.S | |
| 1657 | + .builtin-dtbs-list .builtin-dtbs.S | |
| 1654 | 1658 | |
| 1655 | 1659 | # Directories & files removed with 'make mrproper' |
| 1656 | 1660 | MRPROPER_FILES += include/config include/generated \ |
| @@ -279,7 +279,6 @@ CONFIG_TI_CPSW_SWITCHDEV=y · 278 unmodified lines | ||
| 279 | 279 | CONFIG_TI_CPTS=y |
| 280 | 280 | CONFIG_TI_KEYSTONE_NETCP=y |
| 281 | 281 | CONFIG_TI_KEYSTONE_NETCP_ETHSS=y |
| 282 | −CONFIG_TI_PRUSS=m | |
| 283 | 282 | CONFIG_TI_PRUETH=m |
| 284 | 283 | CONFIG_XILINX_EMACLITE=y |
| 285 | 284 | CONFIG_SFP=m |
| @@ -252,6 +252,7 @@ config ARM64 · 251 unmodified lines | ||
| 252 | 252 | select HAVE_RSEQ |
| 253 | 253 | select HAVE_RUST if RUSTC_SUPPORTS_ARM64 |
| 254 | 254 | select HAVE_STACKPROTECTOR |
| 255 | + select HAVE_STATIC_CALL if CFI | |
| 255 | 256 | select HAVE_SYSCALL_TRACEPOINTS |
| 256 | 257 | select HAVE_KPROBES |
| 257 | 258 | select HAVE_KRETPROBES |
| @@ -698,7 +698,7 @@ · 697 unmodified lines | ||
| 698 | 698 | compatible = "renesas,scif-r8a78000", |
| 699 | 699 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 700 | 700 | reg = <0 0xc0700000 0 0x40>; |
| 701 | − interrupts = <GIC_SPI 4074 IRQ_TYPE_LEVEL_HIGH>; | |
| 701 | + interrupts = <GIC_ESPI 10 IRQ_TYPE_LEVEL_HIGH>; | |
| 702 | 702 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 703 | 703 | clock-names = "fck", "brg_int", "scif_clk"; |
| 704 | 704 | status = "disabled"; |
| @@ -708,7 +708,7 @@ · 3 unmodified lines | ||
| 708 | 708 | compatible = "renesas,scif-r8a78000", |
| 709 | 709 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 710 | 710 | reg = <0 0xc0704000 0 0x40>; |
| 711 | − interrupts = <GIC_SPI 4075 IRQ_TYPE_LEVEL_HIGH>; | |
| 711 | + interrupts = <GIC_ESPI 11 IRQ_TYPE_LEVEL_HIGH>; | |
| 712 | 712 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 713 | 713 | clock-names = "fck", "brg_int", "scif_clk"; |
| 714 | 714 | status = "disabled"; |
| @@ -718,7 +718,7 @@ · 3 unmodified lines | ||
| 718 | 718 | compatible = "renesas,scif-r8a78000", |
| 719 | 719 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 720 | 720 | reg = <0 0xc0708000 0 0x40>; |
| 721 | − interrupts = <GIC_SPI 4076 IRQ_TYPE_LEVEL_HIGH>; | |
| 721 | + interrupts = <GIC_ESPI 12 IRQ_TYPE_LEVEL_HIGH>; | |
| 722 | 722 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 723 | 723 | clock-names = "fck", "brg_int", "scif_clk"; |
| 724 | 724 | status = "disabled"; |
| @@ -728,7 +728,7 @@ · 3 unmodified lines | ||
| 728 | 728 | compatible = "renesas,scif-r8a78000", |
| 729 | 729 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 730 | 730 | reg = <0 0xc070c000 0 0x40>; |
| 731 | − interrupts = <GIC_SPI 4077 IRQ_TYPE_LEVEL_HIGH>; | |
| 731 | + interrupts = <GIC_ESPI 13 IRQ_TYPE_LEVEL_HIGH>; | |
| 732 | 732 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 733 | 733 | clock-names = "fck", "brg_int", "scif_clk"; |
| 734 | 734 | status = "disabled"; |
| @@ -738,7 +738,7 @@ · 3 unmodified lines | ||
| 738 | 738 | compatible = "renesas,hscif-r8a78000", |
| 739 | 739 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 740 | 740 | reg = <0 0xc0710000 0 0x60>; |
| 741 | − interrupts = <GIC_SPI 4078 IRQ_TYPE_LEVEL_HIGH>; | |
| 741 | + interrupts = <GIC_ESPI 14 IRQ_TYPE_LEVEL_HIGH>; | |
| 742 | 742 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 743 | 743 | clock-names = "fck", "brg_int", "scif_clk"; |
| 744 | 744 | status = "disabled"; |
| @@ -748,7 +748,7 @@ · 3 unmodified lines | ||
| 748 | 748 | compatible = "renesas,hscif-r8a78000", |
| 749 | 749 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 750 | 750 | reg = <0 0xc0714000 0 0x60>; |
| 751 | − interrupts = <GIC_SPI 4079 IRQ_TYPE_LEVEL_HIGH>; | |
| 751 | + interrupts = <GIC_ESPI 15 IRQ_TYPE_LEVEL_HIGH>; | |
| 752 | 752 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 753 | 753 | clock-names = "fck", "brg_int", "scif_clk"; |
| 754 | 754 | status = "disabled"; |
| @@ -758,7 +758,7 @@ · 3 unmodified lines | ||
| 758 | 758 | compatible = "renesas,hscif-r8a78000", |
| 759 | 759 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 760 | 760 | reg = <0 0xc0718000 0 0x60>; |
| 761 | − interrupts = <GIC_SPI 4080 IRQ_TYPE_LEVEL_HIGH>; | |
| 761 | + interrupts = <GIC_ESPI 16 IRQ_TYPE_LEVEL_HIGH>; | |
| 762 | 762 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 763 | 763 | clock-names = "fck", "brg_int", "scif_clk"; |
| 764 | 764 | status = "disabled"; |
| @@ -768,7 +768,7 @@ · 3 unmodified lines | ||
| 768 | 768 | compatible = "renesas,hscif-r8a78000", |
| 769 | 769 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 770 | 770 | reg = <0 0xc071c000 0 0x60>; |
| 771 | − interrupts = <GIC_SPI 4081 IRQ_TYPE_LEVEL_HIGH>; | |
| 771 | + interrupts = <GIC_ESPI 17 IRQ_TYPE_LEVEL_HIGH>; | |
| 772 | 772 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 773 | 773 | clock-names = "fck", "brg_int", "scif_clk"; |
| 774 | 774 | status = "disabled"; |
| @@ -581,16 +581,6 @@ · 580 unmodified lines | ||
| 581 | 581 | status = "disabled"; |
| 582 | 582 | }; |
| 583 | 583 | |
| 584 | − wdt0: watchdog@11c00400 { | |
| 585 | − compatible = "renesas,r9a09g057-wdt"; | |
| 586 | − reg = <0 0x11c00400 0 0x400>; | |
| 587 | − clocks = <&cpg CPG_MOD 0x4b>, <&cpg CPG_MOD 0x4c>; | |
| 588 | − clock-names = "pclk", "oscclk"; | |
| 589 | − resets = <&cpg 0x75>; | |
| 590 | − power-domains = <&cpg>; | |
| 591 | − status = "disabled"; | |
| 592 | − }; | |
| 593 | − | |
| 594 | 584 | wdt1: watchdog@14400000 { |
| 595 | 585 | compatible = "renesas,r9a09g057-wdt"; |
| 596 | 586 | reg = <0 0x14400000 0 0x400>; |
| @@ -601,26 +591,6 @@ · 4 unmodified lines | ||
| 601 | 591 | status = "disabled"; |
| 602 | 592 | }; |
| 603 | 593 | |
| 604 | − wdt2: watchdog@13000000 { | |
| 605 | − compatible = "renesas,r9a09g057-wdt"; | |
| 606 | − reg = <0 0x13000000 0 0x400>; | |
| 607 | − clocks = <&cpg CPG_MOD 0x4f>, <&cpg CPG_MOD 0x50>; | |
| 608 | − clock-names = "pclk", "oscclk"; | |
| 609 | − resets = <&cpg 0x77>; | |
| 610 | − power-domains = <&cpg>; | |
| 611 | − status = "disabled"; | |
| 612 | − }; | |
| 613 | − | |
| 614 | − wdt3: watchdog@13000400 { | |
| 615 | − compatible = "renesas,r9a09g057-wdt"; | |
| 616 | − reg = <0 0x13000400 0 0x400>; | |
| 617 | − clocks = <&cpg CPG_MOD 0x51>, <&cpg CPG_MOD 0x52>; | |
| 618 | − clock-names = "pclk", "oscclk"; | |
| 619 | − resets = <&cpg 0x78>; | |
| 620 | − power-domains = <&cpg>; | |
| 621 | − status = "disabled"; | |
| 622 | − }; | |
| 623 | − | |
| 624 | 594 | rtc: rtc@11c00800 { |
| 625 | 595 | compatible = "renesas,r9a09g057-rtca3", "renesas,rz-rtca3"; |
| 626 | 596 | reg = <0 0x11c00800 0 0x400>; |
| @@ -974,8 +974,8 @@ · 973 unmodified lines | ||
| 974 | 974 | |
| 975 | 975 | cpg: clock-controller@80280000 { |
| 976 | 976 | compatible = "renesas,r9a09g077-cpg-mssr"; |
| 977 | − reg = <0 0x80280000 0 0x1000>, | |
| 978 | − <0 0x81280000 0 0x9000>; | |
| 977 | + reg = <0 0x80280000 0 0x10000>, | |
| 978 | + <0 0x81280000 0 0x10000>; | |
| 979 | 979 | clocks = <&extal_clk>; |
| 980 | 980 | clock-names = "extal"; |
| 981 | 981 | #clock-cells = <2>; |
| @@ -977,8 +977,8 @@ · 976 unmodified lines | ||
| 977 | 977 | |
| 978 | 978 | cpg: clock-controller@80280000 { |
| 979 | 979 | compatible = "renesas,r9a09g087-cpg-mssr"; |
| 980 | − reg = <0 0x80280000 0 0x1000>, | |
| 981 | − <0 0x81280000 0 0x9000>; | |
| 980 | + reg = <0 0x80280000 0 0x10000>, | |
| 981 | + <0 0x81280000 0 0x10000>; | |
| 982 | 982 | clocks = <&extal_clk>; |
| 983 | 983 | clock-names = "extal"; |
| 984 | 984 | #clock-cells = <2>; |
| @@ -162,7 +162,7 @@ · 161 unmodified lines | ||
| 162 | 162 | <100000000>; |
| 163 | 163 | renesas,settings = [ |
| 164 | 164 | 80 00 11 19 4c 42 dc 2f 06 7d 20 1a 5f 1e f2 27 |
| 165 | − 00 40 00 00 00 00 00 00 06 0c 19 02 3f f0 90 86 | |
| 165 | + 00 40 00 00 00 00 00 00 06 0c 19 02 3b f0 90 86 | |
| 166 | 166 | a0 80 30 30 9c |
| 167 | 167 | ]; |
| 168 | 168 | }; |
| @@ -53,6 +53,7 @@ · 52 unmodified lines | ||
| 53 | 53 | regulator-max-microvolt = <3300000>; |
| 54 | 54 | gpios-states = <0>; |
| 55 | 55 | states = <3300000 0>, <1800000 1>; |
| 56 | + regulator-ramp-delay = <60>; | |
| 56 | 57 | }; |
| 57 | 58 | #endif |
| 58 | 59 | |
| @@ -25,6 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | regulator-max-microvolt = <3300000>; |
| 26 | 26 | gpios-states = <0>; |
| 27 | 27 | states = <3300000 0>, <1800000 1>; |
| 28 | + regulator-ramp-delay = <60>; | |
| 28 | 29 | }; |
| 29 | 30 | }; |
| 30 | 31 | |
| @@ -76,19 +76,24 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key, · 75 unmodified lines | ||
| 76 | 76 | unsigned int key_len) |
| 77 | 77 | { |
| 78 | 78 | struct aesbs_ctx *ctx = crypto_skcipher_ctx(tfm); |
| 79 | − struct crypto_aes_ctx rk; | |
| 79 | + struct crypto_aes_ctx *rk; | |
| 80 | 80 | int err; |
| 81 | 81 | |
| 82 | − err = aes_expandkey(&rk, in_key, key_len); | |
| 82 | + rk = kmalloc(sizeof(*rk), GFP_KERNEL); | |
| 83 | + if (!rk) | |
| 84 | + return -ENOMEM; | |
| 85 | + | |
| 86 | + err = aes_expandkey(rk, in_key, key_len); | |
| 83 | 87 | if (err) |
| 84 | − return err; | |
| 88 | + goto out; | |
| 85 | 89 | |
| 86 | 90 | ctx->rounds = 6 + key_len / 4; |
| 87 | 91 | |
| 88 | 92 | scoped_ksimd() |
| 89 | − aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); | |
| 90 | − | |
| 91 | − return 0; | |
| 93 | + aesbs_convert_key(ctx->rk, rk->key_enc, ctx->rounds); | |
| 94 | +out: | |
| 95 | + kfree_sensitive(rk); | |
| 96 | + return err; | |
| 92 | 97 | } |
| 93 | 98 | |
| 94 | 99 | static int __ecb_crypt(struct skcipher_request *req, |
| @@ -133,22 +138,26 @@ static int aesbs_cbc_ctr_setkey(struct crypto_skcipher *tfm, const u8 *in_key, · 38 unmodified lines | ||
| 133 | 138 | unsigned int key_len) |
| 134 | 139 | { |
| 135 | 140 | struct aesbs_cbc_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); |
| 136 | − struct crypto_aes_ctx rk; | |
| 141 | + struct crypto_aes_ctx *rk; | |
| 137 | 142 | int err; |
| 138 | 143 | |
| 139 | − err = aes_expandkey(&rk, in_key, key_len); | |
| 144 | + rk = kmalloc(sizeof(*rk), GFP_KERNEL); | |
| 145 | + if (!rk) | |
| 146 | + return -ENOMEM; | |
| 147 | + | |
| 148 | + err = aes_expandkey(rk, in_key, key_len); | |
| 140 | 149 | if (err) |
| 141 | − return err; | |
| 150 | + goto out; | |
| 142 | 151 | |
| 143 | 152 | ctx->key.rounds = 6 + key_len / 4; |
| 144 | 153 | |
| 145 | − memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc)); | |
| 154 | + memcpy(ctx->enc, rk->key_enc, sizeof(ctx->enc)); | |
| 146 | 155 | |
| 147 | 156 | scoped_ksimd() |
| 148 | − aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); | |
| 149 | − memzero_explicit(&rk, sizeof(rk)); | |
| 150 | − | |
| 151 | − return 0; | |
| 157 | + aesbs_convert_key(ctx->key.rk, rk->key_enc, ctx->key.rounds); | |
| 158 | +out: | |
| 159 | + kfree_sensitive(rk); | |
| 160 | + return err; | |
| 152 | 161 | } |
| 153 | 162 | |
| 154 | 163 | static int cbc_encrypt(struct skcipher_request *req) |
| @@ -784,6 +784,9 @@ struct kvm_host_data { · 783 unmodified lines | ||
| 784 | 784 | /* Number of debug breakpoints/watchpoints for this CPU (minus 1) */ |
| 785 | 785 | unsigned int debug_brps; |
| 786 | 786 | unsigned int debug_wrps; |
| 787 | + | |
| 788 | + /* Last vgic_irq part of the AP list recorded in an LR */ | |
| 789 | + struct vgic_irq *last_lr_irq; | |
| 787 | 790 | }; |
| 788 | 791 | |
| 789 | 792 | struct kvm_host_psci_config { |
| @@ -0,0 +1,31 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +#ifndef _ASM_STATIC_CALL_H | |
| 3 | +#define _ASM_STATIC_CALL_H | |
| 4 | + | |
| 5 | +#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, target) \ | |
| 6 | + asm(" .pushsection .static_call.text, \"ax\" \n" \ | |
| 7 | + " .align 4 \n" \ | |
| 8 | + " .globl " name " \n" \ | |
| 9 | + name ": \n" \ | |
| 10 | + " hint 34 /* BTI C */ \n" \ | |
| 11 | + " adrp x16, 1f \n" \ | |
| 12 | + " ldr x16, [x16, :lo12:1f] \n" \ | |
| 13 | + " br x16 \n" \ | |
| 14 | + " .type " name ", %function \n" \ | |
| 15 | + " .size " name ", . - " name " \n" \ | |
| 16 | + " .popsection \n" \ | |
| 17 | + " .pushsection .rodata, \"a\" \n" \ | |
| 18 | + " .align 3 \n" \ | |
| 19 | + "1: .quad " target " \n" \ | |
| 20 | + " .popsection \n") | |
| 21 | + | |
| 22 | +#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \ | |
| 23 | + __ARCH_DEFINE_STATIC_CALL_TRAMP(STATIC_CALL_TRAMP_STR(name), #func) | |
| 24 | + | |
| 25 | +#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ | |
| 26 | + ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0) | |
| 27 | + | |
| 28 | +#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) \ | |
| 29 | + ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0) | |
| 30 | + | |
| 31 | +#endif /* _ASM_STATIC_CALL_H */ | |
| @@ -46,6 +46,7 @@ obj-$(CONFIG_MODULES) += module.o module-plts.o · 45 unmodified lines | ||
| 46 | 46 | obj-$(CONFIG_PERF_EVENTS) += perf_regs.o perf_callchain.o |
| 47 | 47 | obj-$(CONFIG_HARDLOCKUP_DETECTOR_PERF) += watchdog_hld.o |
| 48 | 48 | obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o |
| 49 | +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o | |
| 49 | 50 | obj-$(CONFIG_CPU_PM) += sleep.o suspend.o |
| 50 | 51 | obj-$(CONFIG_KGDB) += kgdb.o |
| 51 | 52 | obj-$(CONFIG_EFI) += efi.o efi-rt-wrapper.o |
| @@ -2345,6 +2345,15 @@ static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry, · 2344 unmodified lines | ||
| 2345 | 2345 | !is_midr_in_range_list(has_vgic_v3)) |
| 2346 | 2346 | return false; |
| 2347 | 2347 | |
| 2348 | + /* | |
| 2349 | + * pKVM prevents late onlining of CPUs. This means that whatever | |
| 2350 | + * state the capability is in after deprivilege cannot be affected | |
| 2351 | + * by a new CPU booting -- this is garanteed to be a CPU we have | |
| 2352 | + * already seen, and the cap is therefore unchanged. | |
| 2353 | + */ | |
| 2354 | + if (system_capabilities_finalized() && is_protected_kvm_enabled()) | |
| 2355 | + return cpus_have_final_cap(ARM64_HAS_ICH_HCR_EL2_TDIR); | |
| 2356 | + | |
| 2348 | 2357 | if (is_kernel_in_hyp_mode()) |
| 2349 | 2358 | res.a1 = read_sysreg_s(SYS_ICH_VTR_EL2); |
| 2350 | 2359 | else |
| @@ -192,6 +192,14 @@ static int scs_handle_fde_frame(const struct eh_frame *frame, · 191 unmodified lines | ||
| 192 | 192 | size -= 2; |
| 193 | 193 | break; |
| 194 | 194 | |
| 195 | + case DW_CFA_advance_loc4: | |
| 196 | + loc += *opcode++ * code_alignment_factor; | |
| 197 | + loc += (*opcode++ << 8) * code_alignment_factor; | |
| 198 | + loc += (*opcode++ << 16) * code_alignment_factor; | |
| 199 | + loc += (*opcode++ << 24) * code_alignment_factor; | |
| 200 | + size -= 4; | |
| 201 | + break; | |
| 202 | + | |
| 195 | 203 | case DW_CFA_def_cfa: |
| 196 | 204 | case DW_CFA_offset_extended: |
| 197 | 205 | size = skip_xleb128(&opcode, size); |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | |
| 13 | 13 | #include <asm/io.h> |
| 14 | 14 | #include <asm/mem_encrypt.h> |
| 15 | +#include <asm/pgtable.h> | |
| 15 | 16 | #include <asm/rsi.h> |
| 16 | 17 | |
| 17 | 18 | static struct realm_config config; |
| @@ -146,7 +147,7 @@ void __init arm64_rsi_init(void) · 128 unmodified lines | ||
| 146 | 147 | return; |
| 147 | 148 | if (WARN_ON(rsi_get_realm_config(&config))) |
| 148 | 149 | return; |
| 149 | − prot_ns_shared = BIT(config.ipa_bits - 1); | |
| 150 | + prot_ns_shared = __phys_to_pte_val(BIT(config.ipa_bits - 1)); | |
| 150 | 151 | |
| 151 | 152 | if (arm64_ioremap_prot_hook_register(realm_ioremap_hook)) |
| 152 | 153 | return; |
| @@ -0,0 +1,23 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | +#include <linux/static_call.h> | |
| 3 | +#include <linux/memory.h> | |
| 4 | +#include <asm/text-patching.h> | |
| 5 | + | |
| 6 | +void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) | |
| 7 | +{ | |
| 8 | + u64 literal; | |
| 9 | + int ret; | |
| 10 | + | |
| 11 | + if (!func) | |
| 12 | + func = __static_call_return0; | |
| 13 | + | |
| 14 | + /* decode the instructions to discover the literal address */ | |
| 15 | + literal = ALIGN_DOWN((u64)tramp + 4, SZ_4K) + | |
| 16 | + aarch64_insn_adrp_get_offset(le32_to_cpup(tramp + 4)) + | |
| 17 | + 8 * aarch64_insn_decode_immediate(AARCH64_INSN_IMM_12, | |
| 18 | + le32_to_cpup(tramp + 8)); | |
| 19 | + | |
| 20 | + ret = aarch64_insn_write_literal_u64((void *)literal, (u64)func); | |
| 21 | + WARN_ON_ONCE(ret); | |
| 22 | +} | |
| 23 | +EXPORT_SYMBOL_GPL(arch_static_call_transform); | |
| @@ -191,6 +191,7 @@ SECTIONS · 190 unmodified lines | ||
| 191 | 191 | LOCK_TEXT |
| 192 | 192 | KPROBES_TEXT |
| 193 | 193 | HYPERVISOR_TEXT |
| 194 | + STATIC_CALL_TEXT | |
| 194 | 195 | *(.gnu.warning) |
| 195 | 196 | } |
| 196 | 197 | |
| @@ -1504,8 +1504,6 @@ int __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 1503 unmodified lines | ||
| 1504 | 1504 | fail = true; |
| 1505 | 1505 | } |
| 1506 | 1506 | |
| 1507 | − isb(); | |
| 1508 | − | |
| 1509 | 1507 | if (!fail) |
| 1510 | 1508 | par = read_sysreg_par(); |
| 1511 | 1509 | |
| @@ -1755,7 +1753,7 @@ int __kvm_at_swap_desc(struct kvm *kvm, gpa_t ipa, u64 old, u64 new) · 243 unmodified lines | ||
| 1755 | 1753 | if (!writable) |
| 1756 | 1754 | return -EPERM; |
| 1757 | 1755 | |
| 1758 | − ptep = (u64 __user *)hva + offset; | |
| 1756 | + ptep = (void __user *)hva + offset; | |
| 1759 | 1757 | if (cpus_have_final_cap(ARM64_HAS_LSE_ATOMICS)) |
| 1760 | 1758 | r = __lse_swap_desc(ptep, old, new); |
| 1761 | 1759 | else |
| @@ -29,7 +29,7 @@ · 28 unmodified lines | ||
| 29 | 29 | |
| 30 | 30 | #include "trace.h" |
| 31 | 31 | |
| 32 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 32 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 33 | 33 | KVM_GENERIC_VM_STATS() |
| 34 | 34 | }; |
| 35 | 35 | |
| @@ -42,7 +42,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 6 unmodified lines | ||
| 42 | 42 | sizeof(kvm_vm_stats_desc), |
| 43 | 43 | }; |
| 44 | 44 | |
| 45 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 45 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 46 | 46 | KVM_GENERIC_VCPU_STATS(), |
| 47 | 47 | STATS_DESC_COUNTER(VCPU, hvc_exit_stat), |
| 48 | 48 | STATS_DESC_COUNTER(VCPU, wfe_exit_stat), |
| @@ -518,7 +518,7 @@ static int host_stage2_adjust_range(u64 addr, struct kvm_mem_range *range) · 517 unmodified lines | ||
| 518 | 518 | granule = kvm_granule_size(level); |
| 519 | 519 | cur.start = ALIGN_DOWN(addr, granule); |
| 520 | 520 | cur.end = cur.start + granule; |
| 521 | − if (!range_included(&cur, range)) | |
| 521 | + if (!range_included(&cur, range) && level < KVM_PGTABLE_LAST_LEVEL) | |
| 522 | 522 | continue; |
| 523 | 523 | *range = cur; |
| 524 | 524 | return 0; |
| @@ -1751,6 +1751,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 1750 unmodified lines | ||
| 1751 | 1751 | |
| 1752 | 1752 | force_pte = (max_map_size == PAGE_SIZE); |
| 1753 | 1753 | vma_pagesize = min_t(long, vma_pagesize, max_map_size); |
| 1754 | + vma_shift = __ffs(vma_pagesize); | |
| 1754 | 1755 | } |
| 1755 | 1756 | |
| 1756 | 1757 | /* |
| @@ -1837,10 +1838,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 80 unmodified lines | ||
| 1837 | 1838 | if (exec_fault && s2_force_noncacheable) |
| 1838 | 1839 | ret = -ENOEXEC; |
| 1839 | 1840 | |
| 1840 | − if (ret) { | |
| 1841 | − kvm_release_page_unused(page); | |
| 1842 | − return ret; | |
| 1843 | − } | |
| 1841 | + if (ret) | |
| 1842 | + goto out_put_page; | |
| 1844 | 1843 | |
| 1845 | 1844 | /* |
| 1846 | 1845 | * Guest performs atomic/exclusive operations on memory with unsupported |
| @@ -1850,7 +1849,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 3 unmodified lines | ||
| 1850 | 1849 | */ |
| 1851 | 1850 | if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) { |
| 1852 | 1851 | kvm_inject_dabt_excl_atomic(vcpu, kvm_vcpu_get_hfar(vcpu)); |
| 1853 | − return 1; | |
| 1852 | + ret = 1; | |
| 1853 | + goto out_put_page; | |
| 1854 | 1854 | } |
| 1855 | 1855 | |
| 1856 | 1856 | if (nested) |
| @@ -1936,6 +1936,10 @@ out_unlock: · 79 unmodified lines | ||
| 1936 | 1936 | mark_page_dirty_in_slot(kvm, memslot, gfn); |
| 1937 | 1937 | |
| 1938 | 1938 | return ret != -EAGAIN ? ret : 0; |
| 1939 | + | |
| 1940 | +out_put_page: | |
| 1941 | + kvm_release_page_unused(page); | |
| 1942 | + return ret; | |
| 1939 | 1943 | } |
| 1940 | 1944 | |
| 1941 | 1945 | /* Resolve the access fault by making the page young again. */ |
| @@ -152,31 +152,31 @@ static int get_ia_size(struct s2_walk_info *wi) · 151 unmodified lines | ||
| 152 | 152 | return 64 - wi->t0sz; |
| 153 | 153 | } |
| 154 | 154 | |
| 155 | −static int check_base_s2_limits(struct s2_walk_info *wi, | |
| 155 | +static int check_base_s2_limits(struct kvm_vcpu *vcpu, struct s2_walk_info *wi, | |
| 156 | 156 | int level, int input_size, int stride) |
| 157 | 157 | { |
| 158 | − int start_size, ia_size; | |
| 158 | + int start_size, pa_max; | |
| 159 | 159 | |
| 160 | − ia_size = get_ia_size(wi); | |
| 160 | + pa_max = kvm_get_pa_bits(vcpu->kvm); | |
| 161 | 161 | |
| 162 | 162 | /* Check translation limits */ |
| 163 | 163 | switch (BIT(wi->pgshift)) { |
| 164 | 164 | case SZ_64K: |
| 165 | − if (level == 0 || (level == 1 && ia_size <= 42)) | |
| 165 | + if (level == 0 || (level == 1 && pa_max <= 42)) | |
| 166 | 166 | return -EFAULT; |
| 167 | 167 | break; |
| 168 | 168 | case SZ_16K: |
| 169 | − if (level == 0 || (level == 1 && ia_size <= 40)) | |
| 169 | + if (level == 0 || (level == 1 && pa_max <= 40)) | |
| 170 | 170 | return -EFAULT; |
| 171 | 171 | break; |
| 172 | 172 | case SZ_4K: |
| 173 | − if (level < 0 || (level == 0 && ia_size <= 42)) | |
| 173 | + if (level < 0 || (level == 0 && pa_max <= 42)) | |
| 174 | 174 | return -EFAULT; |
| 175 | 175 | break; |
| 176 | 176 | } |
| 177 | 177 | |
| 178 | 178 | /* Check input size limits */ |
| 179 | − if (input_size > ia_size) | |
| 179 | + if (input_size > pa_max) | |
| 180 | 180 | return -EFAULT; |
| 181 | 181 | |
| 182 | 182 | /* Check number of entries in starting level table */ |
| @@ -269,16 +269,19 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, · 86 unmodified lines | ||
| 269 | 269 | if (input_size > 48 || input_size < 25) |
| 270 | 270 | return -EFAULT; |
| 271 | 271 | |
| 272 | − ret = check_base_s2_limits(wi, level, input_size, stride); | |
| 273 | − if (WARN_ON(ret)) | |
| 272 | + ret = check_base_s2_limits(vcpu, wi, level, input_size, stride); | |
| 273 | + if (WARN_ON(ret)) { | |
| 274 | + out->esr = compute_fsc(0, ESR_ELx_FSC_FAULT); | |
| 274 | 275 | return ret; |
| 276 | + } | |
| 275 | 277 | |
| 276 | 278 | base_lower_bound = 3 + input_size - ((3 - level) * stride + |
| 277 | 279 | wi->pgshift); |
| 278 | 280 | base_addr = wi->baddr & GENMASK_ULL(47, base_lower_bound); |
| 279 | 281 | |
| 280 | 282 | if (check_output_size(wi, base_addr)) { |
| 281 | − out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); | |
| 283 | + /* R_BFHQH */ | |
| 284 | + out->esr = compute_fsc(0, ESR_ELx_FSC_ADDRSZ); | |
| 282 | 285 | return 1; |
| 283 | 286 | } |
| 284 | 287 | |
| @@ -293,8 +296,10 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, · 8 unmodified lines | ||
| 293 | 296 | |
| 294 | 297 | paddr = base_addr | index; |
| 295 | 298 | ret = read_guest_s2_desc(vcpu, paddr, &desc, wi); |
| 296 | − if (ret < 0) | |
| 299 | + if (ret < 0) { | |
| 300 | + out->esr = ESR_ELx_FSC_SEA_TTW(level); | |
| 297 | 301 | return ret; |
| 302 | + } | |
| 298 | 303 | |
| 299 | 304 | new_desc = desc; |
| 300 | 305 | |
| @@ -247,6 +247,20 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) · 246 unmodified lines | ||
| 247 | 247 | kvm_vcpu_set_be(vcpu); |
| 248 | 248 | |
| 249 | 249 | *vcpu_pc(vcpu) = target_pc; |
| 250 | + | |
| 251 | + /* | |
| 252 | + * We may come from a state where either a PC update was | |
| 253 | + * pending (SMC call resulting in PC being increpented to | |
| 254 | + * skip the SMC) or a pending exception. Make sure we get | |
| 255 | + * rid of all that, as this cannot be valid out of reset. | |
| 256 | + * | |
| 257 | + * Note that clearing the exception mask also clears PC | |
| 258 | + * updates, but that's an implementation detail, and we | |
| 259 | + * really want to make it explicit. | |
| 260 | + */ | |
| 261 | + vcpu_clear_flag(vcpu, PENDING_EXCEPTION); | |
| 262 | + vcpu_clear_flag(vcpu, EXCEPT_MASK); | |
| 263 | + vcpu_clear_flag(vcpu, INCREMENT_PC); | |
| 250 | 264 | vcpu_set_reg(vcpu, 0, reset_state.r0); |
| 251 | 265 | } |
| 252 | 266 | |
| @@ -143,6 +143,21 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) · 142 unmodified lines | ||
| 143 | 143 | kvm->arch.vgic.in_kernel = true; |
| 144 | 144 | kvm->arch.vgic.vgic_model = type; |
| 145 | 145 | kvm->arch.vgic.implementation_rev = KVM_VGIC_IMP_REV_LATEST; |
| 146 | + kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF; | |
| 147 | + | |
| 148 | + aa64pfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1) & ~ID_AA64PFR0_EL1_GIC; | |
| 149 | + pfr1 = kvm_read_vm_id_reg(kvm, SYS_ID_PFR1_EL1) & ~ID_PFR1_EL1_GIC; | |
| 150 | + | |
| 151 | + if (type == KVM_DEV_TYPE_ARM_VGIC_V2) { | |
| 152 | + kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF; | |
| 153 | + } else { | |
| 154 | + INIT_LIST_HEAD(&kvm->arch.vgic.rd_regions); | |
| 155 | + aa64pfr0 |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, GIC, IMP); | |
| 156 | + pfr1 |= SYS_FIELD_PREP_ENUM(ID_PFR1_EL1, GIC, GICv3); | |
| 157 | + } | |
| 158 | + | |
| 159 | + kvm_set_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1, aa64pfr0); | |
| 160 | + kvm_set_vm_id_reg(kvm, SYS_ID_PFR1_EL1, pfr1); | |
| 146 | 161 | |
| 147 | 162 | kvm_for_each_vcpu(i, vcpu, kvm) { |
| 148 | 163 | ret = vgic_allocate_private_irqs_locked(vcpu, type); |
| @@ -157,25 +172,10 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) · 8 unmodified lines | ||
| 157 | 172 | vgic_cpu->private_irqs = NULL; |
| 158 | 173 | } |
| 159 | 174 | |
| 175 | + kvm->arch.vgic.vgic_model = 0; | |
| 160 | 176 | goto out_unlock; |
| 161 | 177 | } |
| 162 | 178 | |
| 163 | − kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF; | |
| 164 | − | |
| 165 | − aa64pfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1) & ~ID_AA64PFR0_EL1_GIC; | |
| 166 | − pfr1 = kvm_read_vm_id_reg(kvm, SYS_ID_PFR1_EL1) & ~ID_PFR1_EL1_GIC; | |
| 167 | − | |
| 168 | − if (type == KVM_DEV_TYPE_ARM_VGIC_V2) { | |
| 169 | − kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF; | |
| 170 | − } else { | |
| 171 | − INIT_LIST_HEAD(&kvm->arch.vgic.rd_regions); | |
| 172 | − aa64pfr0 |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, GIC, IMP); | |
| 173 | − pfr1 |= SYS_FIELD_PREP_ENUM(ID_PFR1_EL1, GIC, GICv3); | |
| 174 | − } | |
| 175 | − | |
| 176 | − kvm_set_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1, aa64pfr0); | |
| 177 | − kvm_set_vm_id_reg(kvm, SYS_ID_PFR1_EL1, pfr1); | |
| 178 | − | |
| 179 | 179 | if (type == KVM_DEV_TYPE_ARM_VGIC_V3) |
| 180 | 180 | kvm->arch.vgic.nassgicap = system_supports_direct_sgis(); |
| 181 | 181 | |
| @@ -115,7 +115,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) · 114 unmodified lines | ||
| 115 | 115 | struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; |
| 116 | 116 | struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2; |
| 117 | 117 | u32 eoicount = FIELD_GET(GICH_HCR_EOICOUNT, cpuif->vgic_hcr); |
| 118 | − struct vgic_irq *irq; | |
| 118 | + struct vgic_irq *irq = *host_data_ptr(last_lr_irq); | |
| 119 | 119 | |
| 120 | 120 | DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); |
| 121 | 121 | |
| @@ -123,7 +123,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) · 1 unmodified line | ||
| 123 | 123 | vgic_v2_fold_lr(vcpu, cpuif->vgic_lr[lr]); |
| 124 | 124 | |
| 125 | 125 | /* See the GICv3 equivalent for the EOIcount handling rationale */ |
| 126 | − list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { | |
| 126 | + list_for_each_entry_continue(irq, &vgic_cpu->ap_list_head, ap_list) { | |
| 127 | 127 | u32 lr; |
| 128 | 128 | |
| 129 | 129 | if (!eoicount) { |
| @@ -148,7 +148,7 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) · 147 unmodified lines | ||
| 148 | 148 | struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; |
| 149 | 149 | struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; |
| 150 | 150 | u32 eoicount = FIELD_GET(ICH_HCR_EL2_EOIcount, cpuif->vgic_hcr); |
| 151 | − struct vgic_irq *irq; | |
| 151 | + struct vgic_irq *irq = *host_data_ptr(last_lr_irq); | |
| 152 | 152 | |
| 153 | 153 | DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); |
| 154 | 154 | |
| @@ -158,12 +158,12 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) · 3 unmodified lines | ||
| 158 | 158 | /* |
| 159 | 159 | * EOIMode=0: use EOIcount to emulate deactivation. We are |
| 160 | 160 | * guaranteed to deactivate in reverse order of the activation, so |
| 161 | − * just pick one active interrupt after the other in the ap_list, | |
| 162 | − * and replay the deactivation as if the CPU was doing it. We also | |
| 163 | − * rely on priority drop to have taken place, and the list to be | |
| 164 | − * sorted by priority. | |
| 161 | + * just pick one active interrupt after the other in the tail part | |
| 162 | + * of the ap_list, past the LRs, and replay the deactivation as if | |
| 163 | + * the CPU was doing it. We also rely on priority drop to have taken | |
| 164 | + * place, and the list to be sorted by priority. | |
| 165 | 165 | */ |
| 166 | − list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { | |
| 166 | + list_for_each_entry_continue(irq, &vgic_cpu->ap_list_head, ap_list) { | |
| 167 | 167 | u64 lr; |
| 168 | 168 | |
| 169 | 169 | /* |
| @@ -814,6 +814,9 @@ retry: · 813 unmodified lines | ||
| 814 | 814 | |
| 815 | 815 | static inline void vgic_fold_lr_state(struct kvm_vcpu *vcpu) |
| 816 | 816 | { |
| 817 | + if (!*host_data_ptr(last_lr_irq)) | |
| 818 | + return; | |
| 819 | + | |
| 817 | 820 | if (kvm_vgic_global_state.type == VGIC_V2) |
| 818 | 821 | vgic_v2_fold_lr_state(vcpu); |
| 819 | 822 | else |
| @@ -960,10 +963,13 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu) · 140 unmodified lines | ||
| 960 | 963 | if (irqs_outside_lrs(&als)) |
| 961 | 964 | vgic_sort_ap_list(vcpu); |
| 962 | 965 | |
| 966 | + *host_data_ptr(last_lr_irq) = NULL; | |
| 967 | + | |
| 963 | 968 | list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { |
| 964 | 969 | scoped_guard(raw_spinlock, &irq->irq_lock) { |
| 965 | 970 | if (likely(vgic_target_oracle(irq) == vcpu)) { |
| 966 | 971 | vgic_populate_lr(vcpu, irq, count++); |
| 972 | + *host_data_ptr(last_lr_irq) = irq; | |
| 967 | 973 | } |
| 968 | 974 | } |
| 969 | 975 | |
| @@ -304,6 +304,9 @@ config AS_HAS_LBT_EXTENSION · 303 unmodified lines | ||
| 304 | 304 | config AS_HAS_LVZ_EXTENSION |
| 305 | 305 | def_bool $(as-instr,hvcl 0) |
| 306 | 306 | |
| 307 | +config AS_HAS_SCQ_EXTENSION | |
| 308 | + def_bool $(as-instr,sc.q \$t0$(comma)\$t1$(comma)\$t2) | |
| 309 | + | |
| 307 | 310 | config CC_HAS_ANNOTATE_TABLEJUMP |
| 308 | 311 | def_bool $(cc-option,-mannotate-tablejump) |
| 309 | 312 | |
| @@ -238,6 +238,8 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int · 237 unmodified lines | ||
| 238 | 238 | arch_cmpxchg((ptr), (o), (n)); \ |
| 239 | 239 | }) |
| 240 | 240 | |
| 241 | +#ifdef CONFIG_AS_HAS_SCQ_EXTENSION | |
| 242 | + | |
| 241 | 243 | union __u128_halves { |
| 242 | 244 | u128 full; |
| 243 | 245 | struct { |
| @@ -290,6 +292,9 @@ union __u128_halves { · 46 unmodified lines | ||
| 290 | 292 | BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ |
| 291 | 293 | __arch_cmpxchg128(ptr, o, n, ""); \ |
| 292 | 294 | }) |
| 295 | + | |
| 296 | +#endif /* CONFIG_AS_HAS_SCQ_EXTENSION */ | |
| 297 | + | |
| 293 | 298 | #else |
| 294 | 299 | #include <asm-generic/cmpxchg-local.h> |
| 295 | 300 | #define arch_cmpxchg64_local(ptr, o, n) __generic_cmpxchg64_local((ptr), (o), (n)) |
| @@ -41,4 +41,40 @@ · 40 unmodified lines | ||
| 41 | 41 | .cfi_endproc; \ |
| 42 | 42 | SYM_END(name, SYM_T_NONE) |
| 43 | 43 | |
| 44 | +/* | |
| 45 | + * This is for the signal handler trampoline, which is used as the return | |
| 46 | + * address of the signal handlers in userspace instead of called normally. | |
| 47 | + * The long standing libgcc bug https://gcc.gnu.org/PR124050 requires a | |
| 48 | + * nop between .cfi_startproc and the actual address of the trampoline, so | |
| 49 | + * we cannot simply use SYM_FUNC_START. | |
| 50 | + * | |
| 51 | + * This wrapper also contains all the .cfi_* directives for recovering | |
| 52 | + * the content of the GPRs and the "return address" (where the rt_sigreturn | |
| 53 | + * syscall will jump to), assuming there is a struct rt_sigframe (where | |
| 54 | + * a struct sigcontext containing those information we need to recover) at | |
| 55 | + * $sp. The "DWARF for the LoongArch(TM) Architecture" manual states | |
| 56 | + * column 0 is for $zero, but it does not make too much sense to | |
| 57 | + * save/restore the hardware zero register. Repurpose this column here | |
| 58 | + * for the return address (here it's not the content of $ra we cannot use | |
| 59 | + * the default column 3). | |
| 60 | + */ | |
| 61 | +#define SYM_SIGFUNC_START(name) \ | |
| 62 | + .cfi_startproc; \ | |
| 63 | + .cfi_signal_frame; \ | |
| 64 | + .cfi_def_cfa 3, RT_SIGFRAME_SC; \ | |
| 65 | + .cfi_return_column 0; \ | |
| 66 | + .cfi_offset 0, SC_PC; \ | |
| 67 | + \ | |
| 68 | + .irp num, 1, 2, 3, 4, 5, 6, 7, 8, \ | |
| 69 | + 9, 10, 11, 12, 13, 14, 15, 16, \ | |
| 70 | + 17, 18, 19, 20, 21, 22, 23, 24, \ | |
| 71 | + 25, 26, 27, 28, 29, 30, 31; \ | |
| 72 | + .cfi_offset \num, SC_REGS + \num * SZREG; \ | |
| 73 | + .endr; \ | |
| 74 | + \ | |
| 75 | + nop; \ | |
| 76 | + SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) | |
| 77 | + | |
| 78 | +#define SYM_SIGFUNC_END(name) SYM_FUNC_END(name) | |
| 79 | + | |
| 44 | 80 | #endif |
| @@ -0,0 +1,9 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0+ */ | |
| 2 | + | |
| 3 | +#include <asm/siginfo.h> | |
| 4 | +#include <asm/ucontext.h> | |
| 5 | + | |
| 6 | +struct rt_sigframe { | |
| 7 | + struct siginfo rs_info; | |
| 8 | + struct ucontext rs_uctx; | |
| 9 | +}; | |
| @@ -253,8 +253,13 @@ do { \ · 252 unmodified lines | ||
| 253 | 253 | \ |
| 254 | 254 | __get_kernel_common(*((type *)(dst)), sizeof(type), \ |
| 255 | 255 | (__force type *)(src)); \ |
| 256 | − if (unlikely(__gu_err)) \ | |
| 256 | + if (unlikely(__gu_err)) { \ | |
| 257 | + pr_info("%s: memory access failed, ecode 0x%x\n", \ | |
| 258 | + __func__, read_csr_excode()); \ | |
| 259 | + pr_info("%s: the caller is %pS\n", \ | |
| 260 | + __func__, __builtin_return_address(0)); \ | |
| 257 | 261 | goto err_label; \ |
| 262 | + } \ | |
| 258 | 263 | } while (0) |
| 259 | 264 | |
| 260 | 265 | #define __put_kernel_nofault(dst, src, type, err_label) \ |
| @@ -264,8 +269,13 @@ do { \ · 3 unmodified lines | ||
| 264 | 269 | \ |
| 265 | 270 | __pu_val = *(__force type *)(src); \ |
| 266 | 271 | __put_kernel_common(((type *)(dst)), sizeof(type)); \ |
| 267 | − if (unlikely(__pu_err)) \ | |
| 272 | + if (unlikely(__pu_err)) { \ | |
| 273 | + pr_info("%s: memory access failed, ecode 0x%x\n", \ | |
| 274 | + __func__, read_csr_excode()); \ | |
| 275 | + pr_info("%s: the caller is %pS\n", \ | |
| 276 | + __func__, __builtin_return_address(0)); \ | |
| 268 | 277 | goto err_label; \ |
| 278 | + } \ | |
| 269 | 279 | } while (0) |
| 270 | 280 | |
| 271 | 281 | extern unsigned long __copy_user(void *to, const void *from, __kernel_size_t n); |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <asm/ptrace.h> |
| 17 | 17 | #include <asm/processor.h> |
| 18 | 18 | #include <asm/ftrace.h> |
| 19 | +#include <asm/sigframe.h> | |
| 19 | 20 | #include <vdso/datapage.h> |
| 20 | 21 | |
| 21 | 22 | static void __used output_ptreg_defines(void) |
| @@ -220,6 +221,7 @@ static void __used output_sc_defines(void) · 198 unmodified lines | ||
| 220 | 221 | COMMENT("Linux sigcontext offsets."); |
| 221 | 222 | OFFSET(SC_REGS, sigcontext, sc_regs); |
| 222 | 223 | OFFSET(SC_PC, sigcontext, sc_pc); |
| 224 | + OFFSET(RT_SIGFRAME_SC, rt_sigframe, rs_uctx.uc_mcontext); | |
| 223 | 225 | BLANK(); |
| 224 | 226 | } |
| 225 | 227 | |
| @@ -42,16 +42,15 @@ static int __init init_cpu_fullname(void) · 41 unmodified lines | ||
| 42 | 42 | int cpu, ret; |
| 43 | 43 | char *cpuname; |
| 44 | 44 | const char *model; |
| 45 | − struct device_node *root; | |
| 46 | 45 | |
| 47 | 46 | /* Parsing cpuname from DTS model property */ |
| 48 | − root = of_find_node_by_path("/"); | |
| 49 | − ret = of_property_read_string(root, "model", &model); | |
| 47 | + ret = of_property_read_string(of_root, "model", &model); | |
| 50 | 48 | if (ret == 0) { |
| 51 | 49 | cpuname = kstrdup(model, GFP_KERNEL); |
| 50 | + if (!cpuname) | |
| 51 | + return -ENOMEM; | |
| 52 | 52 | loongson_sysconf.cpuname = strsep(&cpuname, " "); |
| 53 | 53 | } |
| 54 | − of_node_put(root); | |
| 55 | 54 | |
| 56 | 55 | if (loongson_sysconf.cpuname && !strncmp(loongson_sysconf.cpuname, "Loongson", 8)) { |
| 57 | 56 | for (cpu = 0; cpu < NR_CPUS; cpu++) |
| @@ -246,32 +246,51 @@ static int text_copy_cb(void *data) · 245 unmodified lines | ||
| 246 | 246 | |
| 247 | 247 | if (smp_processor_id() == copy->cpu) { |
| 248 | 248 | ret = copy_to_kernel_nofault(copy->dst, copy->src, copy->len); |
| 249 | − if (ret) | |
| 249 | + if (ret) { | |
| 250 | 250 | pr_err("%s: operation failed\n", __func__); |
| 251 | + return ret; | |
| 252 | + } | |
| 251 | 253 | } |
| 252 | 254 | |
| 253 | 255 | flush_icache_range((unsigned long)copy->dst, (unsigned long)copy->dst + copy->len); |
| 254 | 256 | |
| 255 | − return ret; | |
| 257 | + return 0; | |
| 256 | 258 | } |
| 257 | 259 | |
| 258 | 260 | int larch_insn_text_copy(void *dst, void *src, size_t len) |
| 259 | 261 | { |
| 260 | 262 | int ret = 0; |
| 263 | + int err = 0; | |
| 261 | 264 | size_t start, end; |
| 262 | 265 | struct insn_copy copy = { |
| 263 | 266 | .dst = dst, |
| 264 | 267 | .src = src, |
| 265 | 268 | .len = len, |
| 266 | − .cpu = smp_processor_id(), | |
| 269 | + .cpu = raw_smp_processor_id(), | |
| 267 | 270 | }; |
| 268 | 271 | |
| 272 | + /* | |
| 273 | + * Ensure copy.cpu won't be hot removed before stop_machine. | |
| 274 | + * If it is removed nobody will really update the text. | |
| 275 | + */ | |
| 276 | + lockdep_assert_cpus_held(); | |
| 277 | + | |
| 269 | 278 | start = round_down((size_t)dst, PAGE_SIZE); |
| 270 | 279 | end = round_up((size_t)dst + len, PAGE_SIZE); |
| 271 | 280 | |
| 272 | − set_memory_rw(start, (end - start) / PAGE_SIZE); | |
| 273 | − ret = stop_machine(text_copy_cb, ©, cpu_online_mask); | |
| 274 | − set_memory_rox(start, (end - start) / PAGE_SIZE); | |
| 281 | + err = set_memory_rw(start, (end - start) / PAGE_SIZE); | |
| 282 | + if (err) { | |
| 283 | + pr_info("%s: set_memory_rw() failed\n", __func__); | |
| 284 | + return err; | |
| 285 | + } | |
| 286 | + | |
| 287 | + ret = stop_machine_cpuslocked(text_copy_cb, ©, cpu_online_mask); | |
| 288 | + | |
| 289 | + err = set_memory_rox(start, (end - start) / PAGE_SIZE); | |
| 290 | + if (err) { | |
| 291 | + pr_info("%s: set_memory_rox() failed\n", __func__); | |
| 292 | + return err; | |
| 293 | + } | |
| 275 | 294 | |
| 276 | 295 | return ret; |
| 277 | 296 | } |
| @@ -35,6 +35,7 @@ · 34 unmodified lines | ||
| 35 | 35 | #include <asm/cpu-features.h> |
| 36 | 36 | #include <asm/fpu.h> |
| 37 | 37 | #include <asm/lbt.h> |
| 38 | +#include <asm/sigframe.h> | |
| 38 | 39 | #include <asm/ucontext.h> |
| 39 | 40 | #include <asm/vdso.h> |
| 40 | 41 | |
| @@ -51,11 +52,6 @@ · 10 unmodified lines | ||
| 51 | 52 | #define lock_lbt_owner() ({ preempt_disable(); pagefault_disable(); }) |
| 52 | 53 | #define unlock_lbt_owner() ({ pagefault_enable(); preempt_enable(); }) |
| 53 | 54 | |
| 54 | −struct rt_sigframe { | |
| 55 | − struct siginfo rs_info; | |
| 56 | − struct ucontext rs_uctx; | |
| 57 | −}; | |
| 58 | − | |
| 59 | 55 | struct _ctx_layout { |
| 60 | 56 | struct sctx_info *addr; |
| 61 | 57 | unsigned int size; |
| @@ -83,7 +83,7 @@ static inline void eiointc_update_sw_coremap(struct loongarch_eiointc *s, · 82 unmodified lines | ||
| 83 | 83 | |
| 84 | 84 | if (!(s->status & BIT(EIOINTC_ENABLE_CPU_ENCODE))) { |
| 85 | 85 | cpuid = ffs(cpuid) - 1; |
| 86 | − cpuid = (cpuid >= 4) ? 0 : cpuid; | |
| 86 | + cpuid = ((cpuid < 0) || (cpuid >= 4)) ? 0 : cpuid; | |
| 87 | 87 | } |
| 88 | 88 | |
| 89 | 89 | vcpu = kvm_get_vcpu_by_cpuid(s->kvm, cpuid); |
| @@ -472,34 +472,34 @@ static int kvm_eiointc_regs_access(struct kvm_device *dev, · 382 unmodified lines | ||
| 472 | 472 | switch (addr) { |
| 473 | 473 | case EIOINTC_NODETYPE_START ... EIOINTC_NODETYPE_END: |
| 474 | 474 | offset = (addr - EIOINTC_NODETYPE_START) / 4; |
| 475 | − p = s->nodetype + offset * 4; | |
| 475 | + p = (void *)s->nodetype + offset * 4; | |
| 476 | 476 | break; |
| 477 | 477 | case EIOINTC_IPMAP_START ... EIOINTC_IPMAP_END: |
| 478 | 478 | offset = (addr - EIOINTC_IPMAP_START) / 4; |
| 479 | − p = &s->ipmap + offset * 4; | |
| 479 | + p = (void *)&s->ipmap + offset * 4; | |
| 480 | 480 | break; |
| 481 | 481 | case EIOINTC_ENABLE_START ... EIOINTC_ENABLE_END: |
| 482 | 482 | offset = (addr - EIOINTC_ENABLE_START) / 4; |
| 483 | − p = s->enable + offset * 4; | |
| 483 | + p = (void *)s->enable + offset * 4; | |
| 484 | 484 | break; |
| 485 | 485 | case EIOINTC_BOUNCE_START ... EIOINTC_BOUNCE_END: |
| 486 | 486 | offset = (addr - EIOINTC_BOUNCE_START) / 4; |
| 487 | − p = s->bounce + offset * 4; | |
| 487 | + p = (void *)s->bounce + offset * 4; | |
| 488 | 488 | break; |
| 489 | 489 | case EIOINTC_ISR_START ... EIOINTC_ISR_END: |
| 490 | 490 | offset = (addr - EIOINTC_ISR_START) / 4; |
| 491 | − p = s->isr + offset * 4; | |
| 491 | + p = (void *)s->isr + offset * 4; | |
| 492 | 492 | break; |
| 493 | 493 | case EIOINTC_COREISR_START ... EIOINTC_COREISR_END: |
| 494 | 494 | if (cpu >= s->num_cpu) |
| 495 | 495 | return -EINVAL; |
| 496 | 496 | |
| 497 | 497 | offset = (addr - EIOINTC_COREISR_START) / 4; |
| 498 | − p = s->coreisr[cpu] + offset * 4; | |
| 498 | + p = (void *)s->coreisr[cpu] + offset * 4; | |
| 499 | 499 | break; |
| 500 | 500 | case EIOINTC_COREMAP_START ... EIOINTC_COREMAP_END: |
| 501 | 501 | offset = (addr - EIOINTC_COREMAP_START) / 4; |
| 502 | − p = s->coremap + offset * 4; | |
| 502 | + p = (void *)s->coremap + offset * 4; | |
| 503 | 503 | break; |
| 504 | 504 | default: |
| 505 | 505 | kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr); |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #define CREATE_TRACE_POINTS |
| 15 | 15 | #include "trace.h" |
| 16 | 16 | |
| 17 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 17 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 18 | 18 | KVM_GENERIC_VCPU_STATS(), |
| 19 | 19 | STATS_DESC_COUNTER(VCPU, int_exits), |
| 20 | 20 | STATS_DESC_COUNTER(VCPU, idle_exits), |
| @@ -588,6 +588,9 @@ struct kvm_vcpu *kvm_get_vcpu_by_cpuid(struct kvm *kvm, int cpuid) · 567 unmodified lines | ||
| 588 | 588 | { |
| 589 | 589 | struct kvm_phyid_map *map; |
| 590 | 590 | |
| 591 | + if (cpuid < 0) | |
| 592 | + return NULL; | |
| 593 | + | |
| 591 | 594 | if (cpuid >= KVM_MAX_PHYID) |
| 592 | 595 | return NULL; |
| 593 | 596 | |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <asm/kvm_eiointc.h> |
| 11 | 11 | #include <asm/kvm_pch_pic.h> |
| 12 | 12 | |
| 13 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 13 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 14 | 14 | KVM_GENERIC_VM_STATS(), |
| 15 | 15 | STATS_DESC_ICOUNTER(VM, pages), |
| 16 | 16 | STATS_DESC_ICOUNTER(VM, hugepages), |
| @@ -49,8 +49,8 @@ static void kvm_vm_init_features(struct kvm *kvm) · 32 unmodified lines | ||
| 49 | 49 | kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PMU); |
| 50 | 50 | |
| 51 | 51 | /* Enable all PV features by default */ |
| 52 | − kvm->arch.pv_features = BIT(KVM_FEATURE_IPI); | |
| 53 | − kvm->arch.kvm_features = BIT(KVM_LOONGARCH_VM_FEAT_PV_IPI); | |
| 52 | + kvm->arch.pv_features |= BIT(KVM_FEATURE_IPI); | |
| 53 | + kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_IPI); | |
| 54 | 54 | if (kvm_pvtime_supported()) { |
| 55 | 55 | kvm->arch.pv_features |= BIT(KVM_FEATURE_PREEMPT); |
| 56 | 56 | kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME); |
| @@ -1379,9 +1379,11 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len) · 1378 unmodified lines | ||
| 1379 | 1379 | { |
| 1380 | 1380 | int ret; |
| 1381 | 1381 | |
| 1382 | + cpus_read_lock(); | |
| 1382 | 1383 | mutex_lock(&text_mutex); |
| 1383 | 1384 | ret = larch_insn_text_copy(dst, src, len); |
| 1384 | 1385 | mutex_unlock(&text_mutex); |
| 1386 | + cpus_read_unlock(); | |
| 1385 | 1387 | |
| 1386 | 1388 | return ret ? ERR_PTR(-EINVAL) : dst; |
| 1387 | 1389 | } |
| @@ -1429,10 +1431,12 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, · 41 unmodified lines | ||
| 1429 | 1431 | if (ret) |
| 1430 | 1432 | return ret; |
| 1431 | 1433 | |
| 1434 | + cpus_read_lock(); | |
| 1432 | 1435 | mutex_lock(&text_mutex); |
| 1433 | 1436 | if (memcmp(ip, new_insns, LOONGARCH_LONG_JUMP_NBYTES)) |
| 1434 | 1437 | ret = larch_insn_text_copy(ip, new_insns, LOONGARCH_LONG_JUMP_NBYTES); |
| 1435 | 1438 | mutex_unlock(&text_mutex); |
| 1439 | + cpus_read_unlock(); | |
| 1436 | 1440 | |
| 1437 | 1441 | return ret; |
| 1438 | 1442 | } |
| @@ -1450,10 +1454,12 @@ int bpf_arch_text_invalidate(void *dst, size_t len) · 11 unmodified lines | ||
| 1450 | 1454 | for (i = 0; i < (len / sizeof(u32)); i++) |
| 1451 | 1455 | inst[i] = INSN_BREAK; |
| 1452 | 1456 | |
| 1457 | + cpus_read_lock(); | |
| 1453 | 1458 | mutex_lock(&text_mutex); |
| 1454 | 1459 | if (larch_insn_text_copy(dst, inst, len)) |
| 1455 | 1460 | ret = -EINVAL; |
| 1456 | 1461 | mutex_unlock(&text_mutex); |
| 1462 | + cpus_read_unlock(); | |
| 1457 | 1463 | |
| 1458 | 1464 | kvfree(inst); |
| 1459 | 1465 | |
| @@ -1568,6 +1574,11 @@ void arch_free_bpf_trampoline(void *image, unsigned int size) · 108 unmodified lines | ||
| 1568 | 1574 | bpf_prog_pack_free(image, size); |
| 1569 | 1575 | } |
| 1570 | 1576 | |
| 1577 | +int arch_protect_bpf_trampoline(void *image, unsigned int size) | |
| 1578 | +{ | |
| 1579 | + return 0; | |
| 1580 | +} | |
| 1581 | + | |
| 1571 | 1582 | /* |
| 1572 | 1583 | * Sign-extend the register if necessary |
| 1573 | 1584 | */ |
| @@ -5,9 +5,11 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <linux/kernel.h> |
| 6 | 6 | #include <linux/init.h> |
| 7 | 7 | #include <linux/acpi.h> |
| 8 | +#include <linux/delay.h> | |
| 8 | 9 | #include <linux/types.h> |
| 9 | 10 | #include <linux/pci.h> |
| 10 | 11 | #include <linux/vgaarb.h> |
| 12 | +#include <linux/io-64-nonatomic-lo-hi.h> | |
| 11 | 13 | #include <asm/cacheflush.h> |
| 12 | 14 | #include <asm/loongson.h> |
| 13 | 15 | |
| @@ -15,6 +17,9 @@ · 1 unmodified line | ||
| 15 | 17 | #define PCI_DEVICE_ID_LOONGSON_DC1 0x7a06 |
| 16 | 18 | #define PCI_DEVICE_ID_LOONGSON_DC2 0x7a36 |
| 17 | 19 | #define PCI_DEVICE_ID_LOONGSON_DC3 0x7a46 |
| 20 | +#define PCI_DEVICE_ID_LOONGSON_GPU1 0x7a15 | |
| 21 | +#define PCI_DEVICE_ID_LOONGSON_GPU2 0x7a25 | |
| 22 | +#define PCI_DEVICE_ID_LOONGSON_GPU3 0x7a35 | |
| 18 | 23 | |
| 19 | 24 | int raw_pci_read(unsigned int domain, unsigned int bus, unsigned int devfn, |
| 20 | 25 | int reg, int len, u32 *val) |
| @@ -99,3 +104,78 @@ static void pci_fixup_vgadev(struct pci_dev *pdev) · 78 unmodified lines | ||
| 99 | 104 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC1, pci_fixup_vgadev); |
| 100 | 105 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC2, pci_fixup_vgadev); |
| 101 | 106 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC3, pci_fixup_vgadev); |
| 107 | + | |
| 108 | +#define CRTC_NUM_MAX 2 | |
| 109 | +#define CRTC_OUTPUT_ENABLE 0x100 | |
| 110 | + | |
| 111 | +static void loongson_gpu_fixup_dma_hang(struct pci_dev *pdev, bool on) | |
| 112 | +{ | |
| 113 | + u32 i, val, count, crtc_offset, device; | |
| 114 | + void __iomem *crtc_reg, *base, *regbase; | |
| 115 | + static u32 crtc_status[CRTC_NUM_MAX] = { 0 }; | |
| 116 | + | |
| 117 | + base = pdev->bus->ops->map_bus(pdev->bus, pdev->devfn + 1, 0); | |
| 118 | + device = readw(base + PCI_DEVICE_ID); | |
| 119 | + | |
| 120 | + regbase = ioremap(readq(base + PCI_BASE_ADDRESS_0) & ~0xffull, SZ_64K); | |
| 121 | + if (!regbase) { | |
| 122 | + pci_err(pdev, "Failed to ioremap()\n"); | |
| 123 | + return; | |
| 124 | + } | |
| 125 | + | |
| 126 | + switch (device) { | |
| 127 | + case PCI_DEVICE_ID_LOONGSON_DC2: | |
| 128 | + crtc_reg = regbase + 0x1240; | |
| 129 | + crtc_offset = 0x10; | |
| 130 | + break; | |
| 131 | + case PCI_DEVICE_ID_LOONGSON_DC3: | |
| 132 | + crtc_reg = regbase; | |
| 133 | + crtc_offset = 0x400; | |
| 134 | + break; | |
| 135 | + } | |
| 136 | + | |
| 137 | + for (i = 0; i < CRTC_NUM_MAX; i++, crtc_reg += crtc_offset) { | |
| 138 | + val = readl(crtc_reg); | |
| 139 | + | |
| 140 | + if (!on) | |
| 141 | + crtc_status[i] = val; | |
| 142 | + | |
| 143 | + /* No need to fixup if the status is off at startup. */ | |
| 144 | + if (!(crtc_status[i] & CRTC_OUTPUT_ENABLE)) | |
| 145 | + continue; | |
| 146 | + | |
| 147 | + if (on) | |
| 148 | + val |= CRTC_OUTPUT_ENABLE; | |
| 149 | + else | |
| 150 | + val &= ~CRTC_OUTPUT_ENABLE; | |
| 151 | + | |
| 152 | + mb(); | |
| 153 | + writel(val, crtc_reg); | |
| 154 | + | |
| 155 | + for (count = 0; count < 40; count++) { | |
| 156 | + val = readl(crtc_reg) & CRTC_OUTPUT_ENABLE; | |
| 157 | + if ((on && val) || (!on && !val)) | |
| 158 | + break; | |
| 159 | + udelay(1000); | |
| 160 | + } | |
| 161 | + | |
| 162 | + pci_info(pdev, "DMA hang fixup at reg[0x%lx]: 0x%x\n", | |
| 163 | + (unsigned long)crtc_reg & 0xffff, readl(crtc_reg)); | |
| 164 | + } | |
| 165 | + | |
| 166 | + iounmap(regbase); | |
| 167 | +} | |
| 168 | + | |
| 169 | +static void pci_fixup_dma_hang_early(struct pci_dev *pdev) | |
| 170 | +{ | |
| 171 | + loongson_gpu_fixup_dma_hang(pdev, false); | |
| 172 | +} | |
| 173 | +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU2, pci_fixup_dma_hang_early); | |
| 174 | +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU3, pci_fixup_dma_hang_early); | |
| 175 | + | |
| 176 | +static void pci_fixup_dma_hang_final(struct pci_dev *pdev) | |
| 177 | +{ | |
| 178 | + loongson_gpu_fixup_dma_hang(pdev, true); | |
| 179 | +} | |
| 180 | +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU2, pci_fixup_dma_hang_final); | |
| 181 | +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU3, pci_fixup_dma_hang_final); | |
| @@ -26,7 +26,7 @@ cflags-vdso := $(ccflags-vdso) \ · 25 unmodified lines | ||
| 26 | 26 | $(filter -W%,$(filter-out -Wa$(comma)%,$(KBUILD_CFLAGS))) \ |
| 27 | 27 | -std=gnu11 -fms-extensions -O2 -g -fno-strict-aliasing -fno-common -fno-builtin \ |
| 28 | 28 | -fno-stack-protector -fno-jump-tables -DDISABLE_BRANCH_PROFILING \ |
| 29 | − $(call cc-option, -fno-asynchronous-unwind-tables) \ | |
| 29 | + $(call cc-option, -fasynchronous-unwind-tables) \ | |
| 30 | 30 | $(call cc-option, -fno-stack-protector) |
| 31 | 31 | aflags-vdso := $(ccflags-vdso) \ |
| 32 | 32 | -D__ASSEMBLY__ -Wa,-gdwarf-2 |
| @@ -41,7 +41,7 @@ endif · 8 unmodified lines | ||
| 41 | 41 | |
| 42 | 42 | # VDSO linker flags. |
| 43 | 43 | ldflags-y := -Bsymbolic --no-undefined -soname=linux-vdso.so.1 \ |
| 44 | − $(filter -E%,$(KBUILD_CFLAGS)) -shared --build-id -T | |
| 44 | + $(filter -E%,$(KBUILD_CFLAGS)) -shared --build-id --eh-frame-hdr -T | |
| 45 | 45 | |
| 46 | 46 | # |
| 47 | 47 | # Shared build commands. |
| @@ -12,13 +12,13 @@ · 11 unmodified lines | ||
| 12 | 12 | |
| 13 | 13 | #include <asm/regdef.h> |
| 14 | 14 | #include <asm/asm.h> |
| 15 | +#include <asm/asm-offsets.h> | |
| 15 | 16 | |
| 16 | 17 | .section .text |
| 17 | − .cfi_sections .debug_frame | |
| 18 | 18 | |
| 19 | −SYM_FUNC_START(__vdso_rt_sigreturn) | |
| 19 | +SYM_SIGFUNC_START(__vdso_rt_sigreturn) | |
| 20 | 20 | |
| 21 | 21 | li.w a7, __NR_rt_sigreturn |
| 22 | 22 | syscall 0 |
| 23 | 23 | |
| 24 | −SYM_FUNC_END(__vdso_rt_sigreturn) | |
| 24 | +SYM_SIGFUNC_END(__vdso_rt_sigreturn) | |
| @@ -484,7 +484,6 @@ · 483 unmodified lines | ||
| 484 | 484 | # endif |
| 485 | 485 | # ifndef cpu_vmbits |
| 486 | 486 | # define cpu_vmbits cpu_data[0].vmbits |
| 487 | −# define __NEED_VMBITS_PROBE | |
| 488 | 487 | # endif |
| 489 | 488 | #endif |
| 490 | 489 | |
| @@ -80,9 +80,7 @@ struct cpuinfo_mips { · 79 unmodified lines | ||
| 80 | 80 | int srsets; /* Shadow register sets */ |
| 81 | 81 | int package;/* physical package number */ |
| 82 | 82 | unsigned int globalnumber; |
| 83 | −#ifdef CONFIG_64BIT | |
| 84 | 83 | int vmbits; /* Virtual memory size in bits */ |
| 85 | −#endif | |
| 86 | 84 | void *data; /* Additional data */ |
| 87 | 85 | unsigned int watch_reg_count; /* Number that exist */ |
| 88 | 86 | unsigned int watch_reg_use_cnt; /* Usable by ptrace */ |
| @@ -1871,6 +1871,8 @@ do { \ · 1870 unmodified lines | ||
| 1871 | 1871 | |
| 1872 | 1872 | #define read_c0_entryhi() __read_ulong_c0_register($10, 0) |
| 1873 | 1873 | #define write_c0_entryhi(val) __write_ulong_c0_register($10, 0, val) |
| 1874 | +#define read_c0_entryhi_64() __read_64bit_c0_register($10, 0) | |
| 1875 | +#define write_c0_entryhi_64(val) __write_64bit_c0_register($10, 0, val) | |
| 1874 | 1876 | |
| 1875 | 1877 | #define read_c0_guestctl1() __read_32bit_c0_register($10, 4) |
| 1876 | 1878 | #define write_c0_guestctl1(val) __write_32bit_c0_register($10, 4, val) |
| @@ -210,11 +210,14 @@ static inline void set_elf_base_platform(const char *plat) · 209 unmodified lines | ||
| 210 | 210 | |
| 211 | 211 | static inline void cpu_probe_vmbits(struct cpuinfo_mips *c) |
| 212 | 212 | { |
| 213 | −#ifdef __NEED_VMBITS_PROBE | |
| 214 | − write_c0_entryhi(0x3fffffffffffe000ULL); | |
| 215 | − back_to_back_c0_hazard(); | |
| 216 | − c->vmbits = fls64(read_c0_entryhi() & 0x3fffffffffffe000ULL); | |
| 217 | −#endif | |
| 213 | + int vmbits = 31; | |
| 214 | + | |
| 215 | + if (cpu_has_64bits) { | |
| 216 | + write_c0_entryhi_64(0x3fffffffffffe000ULL); | |
| 217 | + back_to_back_c0_hazard(); | |
| 218 | + vmbits = fls64(read_c0_entryhi_64() & 0x3fffffffffffe000ULL); | |
| 219 | + } | |
| 220 | + c->vmbits = vmbits; | |
| 218 | 221 | } |
| 219 | 222 | |
| 220 | 223 | static void set_isa(struct cpuinfo_mips *c, unsigned int isa) |
| @@ -137,6 +137,8 @@ void cpu_probe(void) · 136 unmodified lines | ||
| 137 | 137 | else |
| 138 | 138 | cpu_set_nofpu_opts(c); |
| 139 | 139 | |
| 140 | + c->vmbits = 31; | |
| 141 | + | |
| 140 | 142 | reserve_exception_space(0, 0x400); |
| 141 | 143 | } |
| 142 | 144 | |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | ||
| 38 | 38 | #define VECTORSPACING 0x100 /* for EI/VI mode */ |
| 39 | 39 | #endif |
| 40 | 40 | |
| 41 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 41 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 42 | 42 | KVM_GENERIC_VM_STATS() |
| 43 | 43 | }; |
| 44 | 44 | |
| @@ -51,7 +51,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 6 unmodified lines | ||
| 51 | 51 | sizeof(kvm_vm_stats_desc), |
| 52 | 52 | }; |
| 53 | 53 | |
| 54 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 54 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 55 | 55 | KVM_GENERIC_VCPU_STATS(), |
| 56 | 56 | STATS_DESC_COUNTER(VCPU, wait_exits), |
| 57 | 57 | STATS_DESC_COUNTER(VCPU, cache_exits), |
| @@ -4,12 +4,12 @@ · 3 unmodified lines | ||
| 4 | 4 | #include "libgcc.h" |
| 5 | 5 | |
| 6 | 6 | /* |
| 7 | − * GCC 7 & older can suboptimally generate __multi3 calls for mips64r6, so for | |
| 7 | + * GCC 9 & older can suboptimally generate __multi3 calls for mips64r6, so for | |
| 8 | 8 | * that specific case only we implement that intrinsic here. |
| 9 | 9 | * |
| 10 | 10 | * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82981 |
| 11 | 11 | */ |
| 12 | −#if defined(CONFIG_64BIT) && defined(CONFIG_CPU_MIPSR6) && (__GNUC__ < 8) | |
| 12 | +#if defined(CONFIG_64BIT) && defined(CONFIG_CPU_MIPSR6) && (__GNUC__ < 10) | |
| 13 | 13 | |
| 14 | 14 | /* multiply 64-bit values, low 64-bits returned */ |
| 15 | 15 | static inline long long notrace dmulu(long long a, long long b) |
| @@ -51,4 +51,4 @@ ti_type notrace __multi3(ti_type a, ti_type b) · 35 unmodified lines | ||
| 51 | 51 | } |
| 52 | 52 | EXPORT_SYMBOL(__multi3); |
| 53 | 53 | |
| 54 | −#endif /* 64BIT && CPU_MIPSR6 && GCC7 */ | |
| 54 | +#endif /* 64BIT && CPU_MIPSR6 && GCC9 */ | |
| @@ -17,7 +17,9 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <linux/dma-map-ops.h> |
| 18 | 18 | #include <linux/export.h> |
| 19 | 19 | #include <linux/libfdt.h> |
| 20 | +#include <linux/minmax.h> | |
| 20 | 21 | #include <linux/pci_ids.h> |
| 22 | +#include <linux/serial_core.h> | |
| 21 | 23 | #include <linux/string_choices.h> |
| 22 | 24 | #include <asm/bootinfo.h> |
| 23 | 25 | #include <loongson.h> |
| @@ -106,9 +108,23 @@ static void __init lefi_fixup_fdt(struct system_loongson *system) · 82 unmodified lines | ||
| 106 | 108 | |
| 107 | 109 | is_loongson64g = (read_c0_prid() & PRID_IMP_MASK) == PRID_IMP_LOONGSON_64G; |
| 108 | 110 | |
| 109 | − for (i = 0; i < system->nr_uarts; i++) { | |
| 111 | + for (i = 0; i < min(system->nr_uarts, MAX_UARTS); i++) { | |
| 110 | 112 | uartdev = &system->uarts[i]; |
| 111 | 113 | |
| 114 | + /* | |
| 115 | + * Some firmware does not set nr_uarts properly and passes empty | |
| 116 | + * items. Ignore them silently. | |
| 117 | + */ | |
| 118 | + if (uartdev->uart_base == 0) | |
| 119 | + continue; | |
| 120 | + | |
| 121 | + /* Our DT only works with UPIO_MEM. */ | |
| 122 | + if (uartdev->iotype != UPIO_MEM) { | |
| 123 | + pr_warn("Ignore UART 0x%llx with iotype %u passed by firmware\n", | |
| 124 | + uartdev->uart_base, uartdev->iotype); | |
| 125 | + continue; | |
| 126 | + } | |
| 127 | + | |
| 112 | 128 | ret = lefi_fixup_fdt_serial(fdt_buf, uartdev->uart_base, |
| 113 | 129 | uartdev->uartclk); |
| 114 | 130 | /* |
| @@ -207,7 +207,8 @@ void cpu_cache_init(void) · 206 unmodified lines | ||
| 207 | 207 | { |
| 208 | 208 | if (IS_ENABLED(CONFIG_CPU_R3000) && cpu_has_3k_cache) |
| 209 | 209 | r3k_cache_init(); |
| 210 | − if (IS_ENABLED(CONFIG_CPU_R4K_CACHE_TLB) && cpu_has_4k_cache) | |
| 210 | + if ((IS_ENABLED(CONFIG_CPU_R4K_CACHE_TLB) || | |
| 211 | + IS_ENABLED(CONFIG_CPU_SB1)) && cpu_has_4k_cache) | |
| 211 | 212 | r4k_cache_init(); |
| 212 | 213 | |
| 213 | 214 | if (IS_ENABLED(CONFIG_CPU_CAVIUM_OCTEON) && cpu_has_octeon_cache) |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/sched.h> |
| 14 | 14 | #include <linux/smp.h> |
| 15 | 15 | #include <linux/memblock.h> |
| 16 | +#include <linux/minmax.h> | |
| 16 | 17 | #include <linux/mm.h> |
| 17 | 18 | #include <linux/hugetlb.h> |
| 18 | 19 | #include <linux/export.h> |
| @@ -24,6 +25,7 @@ · 5 unmodified lines | ||
| 24 | 25 | #include <asm/hazards.h> |
| 25 | 26 | #include <asm/mmu_context.h> |
| 26 | 27 | #include <asm/tlb.h> |
| 28 | +#include <asm/tlbdebug.h> | |
| 27 | 29 | #include <asm/tlbex.h> |
| 28 | 30 | #include <asm/tlbmisc.h> |
| 29 | 31 | #include <asm/setup.h> |
| @@ -511,87 +513,259 @@ static int __init set_ntlb(char *str) · 481 unmodified lines | ||
| 511 | 513 | __setup("ntlb=", set_ntlb); |
| 512 | 514 | |
| 513 | 515 | |
| 514 | −/* Comparison function for EntryHi VPN fields. */ | |
| 515 | −static int r4k_vpn_cmp(const void *a, const void *b) | |
| 516 | +/* The start bit position of VPN2 and Mask in EntryHi/PageMask registers. */ | |
| 517 | +#define VPN2_SHIFT 13 | |
| 518 | + | |
| 519 | +/* Read full EntryHi even with CONFIG_32BIT. */ | |
| 520 | +static inline unsigned long long read_c0_entryhi_native(void) | |
| 521 | +{ | |
| 522 | + return cpu_has_64bits ? read_c0_entryhi_64() : read_c0_entryhi(); | |
| 523 | +} | |
| 524 | + | |
| 525 | +/* Write full EntryHi even with CONFIG_32BIT. */ | |
| 526 | +static inline void write_c0_entryhi_native(unsigned long long v) | |
| 516 | 527 | { |
| 517 | − long v = *(unsigned long *)a - *(unsigned long *)b; | |
| 518 | − int s = sizeof(long) > sizeof(int) ? sizeof(long) * 8 - 1: 0; | |
| 519 | − return s ? (v != 0) | v >> s : v; | |
| 528 | + if (cpu_has_64bits) | |
| 529 | + write_c0_entryhi_64(v); | |
| 530 | + else | |
| 531 | + write_c0_entryhi(v); | |
| 520 | 532 | } |
| 521 | 533 | |
| 534 | +/* TLB entry state for uniquification. */ | |
| 535 | +struct tlbent { | |
| 536 | + unsigned long long wired:1; | |
| 537 | + unsigned long long global:1; | |
| 538 | + unsigned long long asid:10; | |
| 539 | + unsigned long long vpn:51; | |
| 540 | + unsigned long long pagesz:5; | |
| 541 | + unsigned long long index:14; | |
| 542 | +}; | |
| 543 | + | |
| 522 | 544 | /* |
| 523 | − * Initialise all TLB entries with unique values that do not clash with | |
| 524 | − * what we have been handed over and what we'll be using ourselves. | |
| 545 | + * Comparison function for TLB entry sorting. Place wired entries first, | |
| 546 | + * then global entries, then order by the increasing VPN/ASID and the | |
| 547 | + * decreasing page size. This lets us avoid clashes with wired entries | |
| 548 | + * easily and get entries for larger pages out of the way first. | |
| 549 | + * | |
| 550 | + * We could group bits so as to reduce the number of comparisons, but this | |
| 551 | + * is seldom executed and not performance-critical, so prefer legibility. | |
| 525 | 552 | */ |
| 526 | −static void __ref r4k_tlb_uniquify(void) | |
| 553 | +static int r4k_entry_cmp(const void *a, const void *b) | |
| 527 | 554 | { |
| 528 | − int tlbsize = current_cpu_data.tlbsize; | |
| 529 | − bool use_slab = slab_is_available(); | |
| 530 | − int start = num_wired_entries(); | |
| 531 | − phys_addr_t tlb_vpn_size; | |
| 532 | − unsigned long *tlb_vpns; | |
| 533 | − unsigned long vpn_mask; | |
| 534 | − int cnt, ent, idx, i; | |
| 535 | − | |
| 536 | − vpn_mask = GENMASK(cpu_vmbits - 1, 13); | |
| 537 | − vpn_mask |= IS_ENABLED(CONFIG_64BIT) ? 3ULL << 62 : 1 << 31; | |
| 555 | + struct tlbent ea = *(struct tlbent *)a, eb = *(struct tlbent *)b; | |
| 556 | + | |
| 557 | + if (ea.wired > eb.wired) | |
| 558 | + return -1; | |
| 559 | + else if (ea.wired < eb.wired) | |
| 560 | + return 1; | |
| 561 | + else if (ea.global > eb.global) | |
| 562 | + return -1; | |
| 563 | + else if (ea.global < eb.global) | |
| 564 | + return 1; | |
| 565 | + else if (ea.vpn < eb.vpn) | |
| 566 | + return -1; | |
| 567 | + else if (ea.vpn > eb.vpn) | |
| 568 | + return 1; | |
| 569 | + else if (ea.asid < eb.asid) | |
| 570 | + return -1; | |
| 571 | + else if (ea.asid > eb.asid) | |
| 572 | + return 1; | |
| 573 | + else if (ea.pagesz > eb.pagesz) | |
| 574 | + return -1; | |
| 575 | + else if (ea.pagesz < eb.pagesz) | |
| 576 | + return 1; | |
| 577 | + else | |
| 578 | + return 0; | |
| 579 | +} | |
| 538 | 580 | |
| 539 | − tlb_vpn_size = tlbsize * sizeof(*tlb_vpns); | |
| 540 | − tlb_vpns = (use_slab ? | |
| 541 | − kmalloc(tlb_vpn_size, GFP_KERNEL) : | |
| 542 | − memblock_alloc_raw(tlb_vpn_size, sizeof(*tlb_vpns))); | |
| 543 | − if (WARN_ON(!tlb_vpns)) | |
| 544 | − return; /* Pray local_flush_tlb_all() is good enough. */ | |
| 581 | +/* | |
| 582 | + * Fetch all the TLB entries. Mask individual VPN values retrieved with | |
| 583 | + * the corresponding page mask and ignoring any 1KiB extension as we'll | |
| 584 | + * be using 4KiB pages for uniquification. | |
| 585 | + */ | |
| 586 | +static void __ref r4k_tlb_uniquify_read(struct tlbent *tlb_vpns, int tlbsize) | |
| 587 | +{ | |
| 588 | + int start = num_wired_entries(); | |
| 589 | + unsigned long long vpn_mask; | |
| 590 | + bool global; | |
| 591 | + int i; | |
| 545 | 592 | |
| 546 | − htw_stop(); | |
| 593 | + vpn_mask = GENMASK(current_cpu_data.vmbits - 1, VPN2_SHIFT); | |
| 594 | + vpn_mask |= cpu_has_64bits ? 3ULL << 62 : 1 << 31; | |
| 547 | 595 | |
| 548 | − for (i = start, cnt = 0; i < tlbsize; i++, cnt++) { | |
| 549 | − unsigned long vpn; | |
| 596 | + for (i = 0; i < tlbsize; i++) { | |
| 597 | + unsigned long long entryhi, vpn, mask, asid; | |
| 598 | + unsigned int pagesz; | |
| 550 | 599 | |
| 551 | 600 | write_c0_index(i); |
| 552 | 601 | mtc0_tlbr_hazard(); |
| 553 | 602 | tlb_read(); |
| 554 | 603 | tlb_read_hazard(); |
| 555 | − vpn = read_c0_entryhi(); | |
| 556 | − vpn &= vpn_mask & PAGE_MASK; | |
| 557 | − tlb_vpns[cnt] = vpn; | |
| 558 | 604 | |
| 559 | − /* Prevent any large pages from overlapping regular ones. */ | |
| 560 | − write_c0_pagemask(read_c0_pagemask() & PM_DEFAULT_MASK); | |
| 561 | − mtc0_tlbw_hazard(); | |
| 562 | − tlb_write_indexed(); | |
| 563 | − tlbw_use_hazard(); | |
| 605 | + global = !!(read_c0_entrylo0() & ENTRYLO_G); | |
| 606 | + entryhi = read_c0_entryhi_native(); | |
| 607 | + mask = read_c0_pagemask(); | |
| 608 | + | |
| 609 | + asid = entryhi & cpu_asid_mask(¤t_cpu_data); | |
| 610 | + vpn = (entryhi & vpn_mask & ~mask) >> VPN2_SHIFT; | |
| 611 | + pagesz = ilog2((mask >> VPN2_SHIFT) + 1); | |
| 612 | + | |
| 613 | + tlb_vpns[i].global = global; | |
| 614 | + tlb_vpns[i].asid = global ? 0 : asid; | |
| 615 | + tlb_vpns[i].vpn = vpn; | |
| 616 | + tlb_vpns[i].pagesz = pagesz; | |
| 617 | + tlb_vpns[i].wired = i < start; | |
| 618 | + tlb_vpns[i].index = i; | |
| 564 | 619 | } |
| 620 | +} | |
| 621 | + | |
| 622 | +/* | |
| 623 | + * Write unique values to all but the wired TLB entries each, using | |
| 624 | + * the 4KiB page size. This size might not be supported with R6, but | |
| 625 | + * EHINV is mandatory for R6, so we won't ever be called in that case. | |
| 626 | + * | |
| 627 | + * A sorted table is supplied with any wired entries at the beginning, | |
| 628 | + * followed by any global entries, and then finally regular entries. | |
| 629 | + * We start at the VPN and ASID values of zero and only assign user | |
| 630 | + * addresses, therefore guaranteeing no clash with addresses produced | |
| 631 | + * by UNIQUE_ENTRYHI. We avoid any VPN values used by wired or global | |
| 632 | + * entries, by increasing the VPN value beyond the span of such entry. | |
| 633 | + * | |
| 634 | + * When a VPN/ASID clash is found with a regular entry we increment the | |
| 635 | + * ASID instead until no VPN/ASID clash has been found or the ASID space | |
| 636 | + * has been exhausted, in which case we increase the VPN value beyond | |
| 637 | + * the span of the largest clashing entry. | |
| 638 | + * | |
| 639 | + * We do not need to be concerned about FTLB or MMID configurations as | |
| 640 | + * those are required to implement the EHINV feature. | |
| 641 | + */ | |
| 642 | +static void __ref r4k_tlb_uniquify_write(struct tlbent *tlb_vpns, int tlbsize) | |
| 643 | +{ | |
| 644 | + unsigned long long asid, vpn, vpn_size, pagesz; | |
| 645 | + int widx, gidx, idx, sidx, lidx, i; | |
| 565 | 646 | |
| 566 | − sort(tlb_vpns, cnt, sizeof(tlb_vpns[0]), r4k_vpn_cmp, NULL); | |
| 647 | + vpn_size = 1ULL << (current_cpu_data.vmbits - VPN2_SHIFT); | |
| 648 | + pagesz = ilog2((PM_4K >> VPN2_SHIFT) + 1); | |
| 567 | 649 | |
| 568 | − write_c0_pagemask(PM_DEFAULT_MASK); | |
| 650 | + write_c0_pagemask(PM_4K); | |
| 569 | 651 | write_c0_entrylo0(0); |
| 570 | 652 | write_c0_entrylo1(0); |
| 571 | 653 | |
| 572 | − idx = 0; | |
| 573 | − ent = tlbsize; | |
| 574 | − for (i = start; i < tlbsize; i++) | |
| 575 | − while (1) { | |
| 576 | − unsigned long entryhi, vpn; | |
| 654 | + asid = 0; | |
| 655 | + vpn = 0; | |
| 656 | + widx = 0; | |
| 657 | + gidx = 0; | |
| 658 | + for (sidx = 0; sidx < tlbsize && tlb_vpns[sidx].wired; sidx++) | |
| 659 | + ; | |
| 660 | + for (lidx = sidx; lidx < tlbsize && tlb_vpns[lidx].global; lidx++) | |
| 661 | + ; | |
| 662 | + idx = gidx = sidx + 1; | |
| 663 | + for (i = sidx; i < tlbsize; i++) { | |
| 664 | + unsigned long long entryhi, vpn_pagesz = 0; | |
| 577 | 665 | |
| 578 | − entryhi = UNIQUE_ENTRYHI(ent); | |
| 579 | − vpn = entryhi & vpn_mask & PAGE_MASK; | |
| 666 | + while (1) { | |
| 667 | + if (WARN_ON(vpn >= vpn_size)) { | |
| 668 | + dump_tlb_all(); | |
| 669 | + /* Pray local_flush_tlb_all() will cope. */ | |
| 670 | + return; | |
| 671 | + } | |
| 580 | 672 | |
| 581 | − if (idx >= cnt || vpn < tlb_vpns[idx]) { | |
| 582 | − write_c0_entryhi(entryhi); | |
| 583 | − write_c0_index(i); | |
| 584 | − mtc0_tlbw_hazard(); | |
| 585 | − tlb_write_indexed(); | |
| 586 | − ent++; | |
| 587 | − break; | |
| 588 | − } else if (vpn == tlb_vpns[idx]) { | |
| 589 | − ent++; | |
| 590 | − } else { | |
| 673 | + /* VPN must be below the next wired entry. */ | |
| 674 | + if (widx < sidx && vpn >= tlb_vpns[widx].vpn) { | |
| 675 | + vpn = max(vpn, | |
| 676 | + (tlb_vpns[widx].vpn + | |
| 677 | + (1ULL << tlb_vpns[widx].pagesz))); | |
| 678 | + asid = 0; | |
| 679 | + widx++; | |
| 680 | + continue; | |
| 681 | + } | |
| 682 | + /* VPN must be below the next global entry. */ | |
| 683 | + if (gidx < lidx && vpn >= tlb_vpns[gidx].vpn) { | |
| 684 | + vpn = max(vpn, | |
| 685 | + (tlb_vpns[gidx].vpn + | |
| 686 | + (1ULL << tlb_vpns[gidx].pagesz))); | |
| 687 | + asid = 0; | |
| 688 | + gidx++; | |
| 689 | + continue; | |
| 690 | + } | |
| 691 | + /* Try to find a free ASID so as to conserve VPNs. */ | |
| 692 | + if (idx < tlbsize && vpn == tlb_vpns[idx].vpn && | |
| 693 | + asid == tlb_vpns[idx].asid) { | |
| 694 | + unsigned long long idx_pagesz; | |
| 695 | + | |
| 696 | + idx_pagesz = tlb_vpns[idx].pagesz; | |
| 697 | + vpn_pagesz = max(vpn_pagesz, idx_pagesz); | |
| 698 | + do | |
| 699 | + idx++; | |
| 700 | + while (idx < tlbsize && | |
| 701 | + vpn == tlb_vpns[idx].vpn && | |
| 702 | + asid == tlb_vpns[idx].asid); | |
| 703 | + asid++; | |
| 704 | + if (asid > cpu_asid_mask(¤t_cpu_data)) { | |
| 705 | + vpn += vpn_pagesz; | |
| 706 | + asid = 0; | |
| 707 | + vpn_pagesz = 0; | |
| 708 | + } | |
| 709 | + continue; | |
| 710 | + } | |
| 711 | + /* VPN mustn't be above the next regular entry. */ | |
| 712 | + if (idx < tlbsize && vpn > tlb_vpns[idx].vpn) { | |
| 713 | + vpn = max(vpn, | |
| 714 | + (tlb_vpns[idx].vpn + | |
| 715 | + (1ULL << tlb_vpns[idx].pagesz))); | |
| 716 | + asid = 0; | |
| 591 | 717 | idx++; |
| 718 | + continue; | |
| 592 | 719 | } |
| 720 | + break; | |
| 593 | 721 | } |
| 594 | 722 | |
| 723 | + entryhi = (vpn << VPN2_SHIFT) | asid; | |
| 724 | + write_c0_entryhi_native(entryhi); | |
| 725 | + write_c0_index(tlb_vpns[i].index); | |
| 726 | + mtc0_tlbw_hazard(); | |
| 727 | + tlb_write_indexed(); | |
| 728 | + | |
| 729 | + tlb_vpns[i].asid = asid; | |
| 730 | + tlb_vpns[i].vpn = vpn; | |
| 731 | + tlb_vpns[i].pagesz = pagesz; | |
| 732 | + | |
| 733 | + asid++; | |
| 734 | + if (asid > cpu_asid_mask(¤t_cpu_data)) { | |
| 735 | + vpn += 1ULL << pagesz; | |
| 736 | + asid = 0; | |
| 737 | + } | |
| 738 | + } | |
| 739 | +} | |
| 740 | + | |
| 741 | +/* | |
| 742 | + * Initialise all TLB entries with unique values that do not clash with | |
| 743 | + * what we have been handed over and what we'll be using ourselves. | |
| 744 | + */ | |
| 745 | +static void __ref r4k_tlb_uniquify(void) | |
| 746 | +{ | |
| 747 | + int tlbsize = current_cpu_data.tlbsize; | |
| 748 | + bool use_slab = slab_is_available(); | |
| 749 | + phys_addr_t tlb_vpn_size; | |
| 750 | + struct tlbent *tlb_vpns; | |
| 751 | + | |
| 752 | + tlb_vpn_size = tlbsize * sizeof(*tlb_vpns); | |
| 753 | + tlb_vpns = (use_slab ? | |
| 754 | + kmalloc(tlb_vpn_size, GFP_ATOMIC) : | |
| 755 | + memblock_alloc_raw(tlb_vpn_size, sizeof(*tlb_vpns))); | |
| 756 | + if (WARN_ON(!tlb_vpns)) | |
| 757 | + return; /* Pray local_flush_tlb_all() is good enough. */ | |
| 758 | + | |
| 759 | + htw_stop(); | |
| 760 | + | |
| 761 | + r4k_tlb_uniquify_read(tlb_vpns, tlbsize); | |
| 762 | + | |
| 763 | + sort(tlb_vpns, tlbsize, sizeof(*tlb_vpns), r4k_entry_cmp, NULL); | |
| 764 | + | |
| 765 | + r4k_tlb_uniquify_write(tlb_vpns, tlbsize); | |
| 766 | + | |
| 767 | + write_c0_pagemask(PM_DEFAULT_MASK); | |
| 768 | + | |
| 595 | 769 | tlbw_use_hazard(); |
| 596 | 770 | htw_start(); |
| 597 | 771 | flush_micro_tlb(); |
| @@ -640,7 +814,8 @@ static void r4k_tlb_configure(void) · 42 unmodified lines | ||
| 640 | 814 | temp_tlb_entry = current_cpu_data.tlbsize - 1; |
| 641 | 815 | |
| 642 | 816 | /* From this point on the ARC firmware is dead. */ |
| 643 | − r4k_tlb_uniquify(); | |
| 817 | + if (!cpu_has_tlbinv) | |
| 818 | + r4k_tlb_uniquify(); | |
| 644 | 819 | local_flush_tlb_all(); |
| 645 | 820 | |
| 646 | 821 | /* Did I tell you that ARC SUCKS? */ |
| @@ -21,16 +21,16 @@ static const char *clk_cpu(int *idx) · 20 unmodified lines | ||
| 21 | 21 | { |
| 22 | 22 | switch (ralink_soc) { |
| 23 | 23 | case RT2880_SOC: |
| 24 | − *idx = 0; | |
| 24 | + *idx = 1; | |
| 25 | 25 | return "ralink,rt2880-sysc"; |
| 26 | 26 | case RT3883_SOC: |
| 27 | − *idx = 0; | |
| 27 | + *idx = 1; | |
| 28 | 28 | return "ralink,rt3883-sysc"; |
| 29 | 29 | case RT305X_SOC_RT3050: |
| 30 | − *idx = 0; | |
| 30 | + *idx = 1; | |
| 31 | 31 | return "ralink,rt3050-sysc"; |
| 32 | 32 | case RT305X_SOC_RT3052: |
| 33 | − *idx = 0; | |
| 33 | + *idx = 1; | |
| 34 | 34 | return "ralink,rt3052-sysc"; |
| 35 | 35 | case RT305X_SOC_RT3350: |
| 36 | 36 | *idx = 1; |
| @@ -953,7 +953,7 @@ SYSCALL_DEFINE3(cacheflush, unsigned long, addr, unsigned long, bytes, · 952 unmodified lines | ||
| 953 | 953 | #else |
| 954 | 954 | "1: cmpb,<<,n %0,%2,1b\n" |
| 955 | 955 | #endif |
| 956 | − " fic,m %3(%4,%0)\n" | |
| 956 | + " fdc,m %3(%4,%0)\n" | |
| 957 | 957 | "2: sync\n" |
| 958 | 958 | ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 2b, "%1") |
| 959 | 959 | : "+r" (start), "+r" (error) |
| @@ -968,7 +968,7 @@ SYSCALL_DEFINE3(cacheflush, unsigned long, addr, unsigned long, bytes, · 8 unmodified lines | ||
| 968 | 968 | #else |
| 969 | 969 | "1: cmpb,<<,n %0,%2,1b\n" |
| 970 | 970 | #endif |
| 971 | − " fdc,m %3(%4,%0)\n" | |
| 971 | + " fic,m %3(%4,%0)\n" | |
| 972 | 972 | "2: sync\n" |
| 973 | 973 | ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 2b, "%1") |
| 974 | 974 | : "+r" (start), "+r" (error) |
| @@ -573,8 +573,8 @@ config ARCH_USING_PATCHABLE_FUNCTION_ENTRY · 572 unmodified lines | ||
| 573 | 573 | depends on FUNCTION_TRACER && (PPC32 || PPC64_ELF_ABI_V2) |
| 574 | 574 | depends on $(cc-option,-fpatchable-function-entry=2) |
| 575 | 575 | def_bool y if PPC32 |
| 576 | − def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh $(CC) -mlittle-endian) if PPC64 && CPU_LITTLE_ENDIAN | |
| 577 | − def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh $(CC) -mbig-endian) if PPC64 && CPU_BIG_ENDIAN | |
| 576 | + def_bool $(success,$(srctree)/arch/powerpc/tools/check-fpatchable-function-entry.sh $(CC) $(CLANG_FLAGS) -mlittle-endian) if PPC64 && CPU_LITTLE_ENDIAN | |
| 577 | + def_bool $(success,$(srctree)/arch/powerpc/tools/check-fpatchable-function-entry.sh $(CC) -mbig-endian) if PPC64 && CPU_BIG_ENDIAN | |
| 578 | 578 | |
| 579 | 579 | config PPC_FTRACE_OUT_OF_LINE |
| 580 | 580 | def_bool PPC64 && ARCH_USING_PATCHABLE_FUNCTION_ENTRY |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | ||
| 37 | 37 | }; |
| 38 | 38 | }; |
| 39 | 39 | |
| 40 | − memory { | |
| 40 | + memory@0 { | |
| 41 | 41 | device_type = "memory"; |
| 42 | 42 | reg = <0x00000000 0x8000000>; // 128MB at 0 |
| 43 | 43 | }; |
| @@ -1,156 +0,0 @@ | ||
| 1 | −/* T4240 Interlaken LAC Portal device tree stub with 24 portals. | |
| 2 | − * | |
| 3 | − * Copyright 2012 Freescale Semiconductor Inc. | |
| 4 | − * | |
| 5 | − * Redistribution and use in source and binary forms, with or without | |
| 6 | − * modification, are permitted provided that the following conditions are met: | |
| 7 | − * * Redistributions of source code must retain the above copyright | |
| 8 | − * notice, this list of conditions and the following disclaimer. | |
| 9 | − * * Redistributions in binary form must reproduce the above copyright | |
| 10 | − * notice, this list of conditions and the following disclaimer in the | |
| 11 | − * documentation and/or other materials provided with the distribution. | |
| 12 | − * * Neither the name of Freescale Semiconductor nor the | |
| 13 | − * names of its contributors may be used to endorse or promote products | |
| 14 | − * derived from this software without specific prior written permission. | |
| 15 | − * | |
| 16 | − * | |
| 17 | − * ALTERNATIVELY, this software may be distributed under the terms of the | |
| 18 | − * GNU General Public License ("GPL") as published by the Free Software | |
| 19 | − * Foundation, either version 2 of that License or (at your option) any | |
| 20 | − * later version. | |
| 21 | − * | |
| 22 | − * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY | |
| 23 | − * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | |
| 24 | − * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | |
| 25 | − * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | |
| 26 | − * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | |
| 27 | − * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | |
| 28 | − * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | |
| 29 | − * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | |
| 30 | − * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | |
| 31 | − * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
| 32 | − */ | |
| 33 | − | |
| 34 | −#address-cells = <0x1>; | |
| 35 | −#size-cells = <0x1>; | |
| 36 | −compatible = "fsl,interlaken-lac-portals"; | |
| 37 | − | |
| 38 | −lportal0: lac-portal@0 { | |
| 39 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 40 | − reg = <0x0 0x1000>; | |
| 41 | −}; | |
| 42 | − | |
| 43 | −lportal1: lac-portal@1000 { | |
| 44 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 45 | − reg = <0x1000 0x1000>; | |
| 46 | −}; | |
| 47 | − | |
| 48 | −lportal2: lac-portal@2000 { | |
| 49 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 50 | − reg = <0x2000 0x1000>; | |
| 51 | −}; | |
| 52 | − | |
| 53 | −lportal3: lac-portal@3000 { | |
| 54 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 55 | − reg = <0x3000 0x1000>; | |
| 56 | −}; | |
| 57 | − | |
| 58 | −lportal4: lac-portal@4000 { | |
| 59 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 60 | − reg = <0x4000 0x1000>; | |
| 61 | −}; | |
| 62 | − | |
| 63 | −lportal5: lac-portal@5000 { | |
| 64 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 65 | − reg = <0x5000 0x1000>; | |
| 66 | −}; | |
| 67 | − | |
| 68 | −lportal6: lac-portal@6000 { | |
| 69 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 70 | − reg = <0x6000 0x1000>; | |
| 71 | −}; | |
| 72 | − | |
| 73 | −lportal7: lac-portal@7000 { | |
| 74 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 75 | − reg = <0x7000 0x1000>; | |
| 76 | −}; | |
| 77 | − | |
| 78 | −lportal8: lac-portal@8000 { | |
| 79 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 80 | − reg = <0x8000 0x1000>; | |
| 81 | −}; | |
| 82 | − | |
| 83 | −lportal9: lac-portal@9000 { | |
| 84 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 85 | − reg = <0x9000 0x1000>; | |
| 86 | −}; | |
| 87 | − | |
| 88 | −lportal10: lac-portal@A000 { | |
| 89 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 90 | − reg = <0xA000 0x1000>; | |
| 91 | −}; | |
| 92 | − | |
| 93 | −lportal11: lac-portal@B000 { | |
| 94 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 95 | − reg = <0xB000 0x1000>; | |
| 96 | −}; | |
| 97 | − | |
| 98 | −lportal12: lac-portal@C000 { | |
| 99 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 100 | − reg = <0xC000 0x1000>; | |
| 101 | −}; | |
| 102 | − | |
| 103 | −lportal13: lac-portal@D000 { | |
| 104 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 105 | − reg = <0xD000 0x1000>; | |
| 106 | −}; | |
| 107 | − | |
| 108 | −lportal14: lac-portal@E000 { | |
| 109 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 110 | − reg = <0xE000 0x1000>; | |
| 111 | −}; | |
| 112 | − | |
| 113 | −lportal15: lac-portal@F000 { | |
| 114 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 115 | − reg = <0xF000 0x1000>; | |
| 116 | −}; | |
| 117 | − | |
| 118 | −lportal16: lac-portal@10000 { | |
| 119 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 120 | − reg = <0x10000 0x1000>; | |
| 121 | −}; | |
| 122 | − | |
| 123 | −lportal17: lac-portal@11000 { | |
| 124 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 125 | − reg = <0x11000 0x1000>; | |
| 126 | −}; | |
| 127 | − | |
| 128 | −lportal18: lac-portal@1200 { | |
| 129 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 130 | − reg = <0x12000 0x1000>; | |
| 131 | −}; | |
| 132 | − | |
| 133 | −lportal19: lac-portal@13000 { | |
| 134 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 135 | − reg = <0x13000 0x1000>; | |
| 136 | −}; | |
| 137 | − | |
| 138 | −lportal20: lac-portal@14000 { | |
| 139 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 140 | − reg = <0x14000 0x1000>; | |
| 141 | −}; | |
| 142 | − | |
| 143 | −lportal21: lac-portal@15000 { | |
| 144 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 145 | − reg = <0x15000 0x1000>; | |
| 146 | −}; | |
| 147 | − | |
| 148 | −lportal22: lac-portal@16000 { | |
| 149 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 150 | − reg = <0x16000 0x1000>; | |
| 151 | −}; | |
| 152 | − | |
| 153 | −lportal23: lac-portal@17000 { | |
| 154 | − compatible = "fsl,interlaken-lac-portal-v1.0"; | |
| 155 | − reg = <0x17000 0x1000>; | |
| 156 | −}; | |
| @@ -1,45 +0,0 @@ | ||
| 1 | −/* | |
| 2 | − * T4 Interlaken Look-aside Controller (LAC) device tree stub | |
| 3 | − * | |
| 4 | − * Copyright 2012 Freescale Semiconductor Inc. | |
| 5 | − * | |
| 6 | − * Redistribution and use in source and binary forms, with or without | |
| 7 | − * modification, are permitted provided that the following conditions are met: | |
| 8 | − * * Redistributions of source code must retain the above copyright | |
| 9 | − * notice, this list of conditions and the following disclaimer. | |
| 10 | − * * Redistributions in binary form must reproduce the above copyright | |
| 11 | − * notice, this list of conditions and the following disclaimer in the | |
| 12 | − * documentation and/or other materials provided with the distribution. | |
| 13 | − * * Neither the name of Freescale Semiconductor nor the | |
| 14 | − * names of its contributors may be used to endorse or promote products | |
| 15 | − * derived from this software without specific prior written permission. | |
| 16 | − * | |
| 17 | − * | |
| 18 | − * ALTERNATIVELY, this software may be distributed under the terms of the | |
| 19 | − * GNU General Public License ("GPL") as published by the Free Software | |
| 20 | − * Foundation, either version 2 of that License or (at your option) any | |
| 21 | − * later version. | |
| 22 | − * | |
| 23 | − * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY | |
| 24 | − * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | |
| 25 | − * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | |
| 26 | − * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | |
| 27 | − * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | |
| 28 | − * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | |
| 29 | − * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | |
| 30 | − * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | |
| 31 | − * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | |
| 32 | − * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
| 33 | − */ | |
| 34 | − | |
| 35 | −lac: lac@229000 { | |
| 36 | − compatible = "fsl,interlaken-lac"; | |
| 37 | − reg = <0x229000 0x1000>; | |
| 38 | − interrupts = <16 2 1 18>; | |
| 39 | −}; | |
| 40 | − | |
| 41 | −lac-hv@228000 { | |
| 42 | − compatible = "fsl,interlaken-lac-hv"; | |
| 43 | − reg = <0x228000 0x1000>; | |
| 44 | − fsl,non-hv-node = <&lac>; | |
| 45 | −}; | |
| @@ -1,43 +0,0 @@ | ||
| 1 | −/* | |
| 2 | − * PQ3 MPIC Message (Group B) device tree stub [ controller @ offset 0x42400 ] | |
| 3 | − * | |
| 4 | − * Copyright 2012 Freescale Semiconductor Inc. | |
| 5 | − * | |
| 6 | − * Redistribution and use in source and binary forms, with or without | |
| 7 | − * modification, are permitted provided that the following conditions are met: | |
| 8 | − * * Redistributions of source code must retain the above copyright | |
| 9 | − * notice, this list of conditions and the following disclaimer. | |
| 10 | − * * Redistributions in binary form must reproduce the above copyright | |
| 11 | − * notice, this list of conditions and the following disclaimer in the | |
| 12 | − * documentation and/or other materials provided with the distribution. | |
| 13 | − * * Neither the name of Freescale Semiconductor nor the | |
| 14 | − * names of its contributors may be used to endorse or promote products | |
| 15 | − * derived from this software without specific prior written permission. | |
| 16 | − * | |
| 17 | − * | |
| 18 | − * ALTERNATIVELY, this software may be distributed under the terms of the | |
| 19 | − * GNU General Public License ("GPL") as published by the Free Software | |
| 20 | − * Foundation, either version 2 of that License or (at your option) any | |
| 21 | − * later version. | |
| 22 | − * | |
| 23 | − * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY | |
| 24 | − * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | |
| 25 | − * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | |
| 26 | − * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | |
| 27 | − * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | |
| 28 | − * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | |
| 29 | − * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | |
| 30 | − * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | |
| 31 | − * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | |
| 32 | − * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
| 33 | − */ | |
| 34 | − | |
| 35 | −message@42400 { | |
| 36 | − compatible = "fsl,mpic-v3.1-msgr"; | |
| 37 | − reg = <0x42400 0x200>; | |
| 38 | − interrupts = < | |
| 39 | − 0xb4 2 0 0 | |
| 40 | − 0xb5 2 0 0 | |
| 41 | − 0xb6 2 0 0 | |
| 42 | − 0xb7 2 0 0>; | |
| 43 | −}; | |
| @@ -1,80 +0,0 @@ | ||
| 1 | −/* | |
| 2 | − * QorIQ FMan v3 1g port #1 device tree stub [ controller @ offset 0x400000 ] | |
| 3 | − * | |
| 4 | − * Copyright 2012 - 2015 Freescale Semiconductor Inc. | |
| 5 | − * | |
| 6 | − * Redistribution and use in source and binary forms, with or without | |
| 7 | − * modification, are permitted provided that the following conditions are met: | |
| 8 | − * * Redistributions of source code must retain the above copyright | |
| 9 | − * notice, this list of conditions and the following disclaimer. | |
| 10 | − * * Redistributions in binary form must reproduce the above copyright | |
| 11 | − * notice, this list of conditions and the following disclaimer in the | |
| 12 | − * documentation and/or other materials provided with the distribution. | |
| 13 | − * * Neither the name of Freescale Semiconductor nor the | |
| 14 | − * names of its contributors may be used to endorse or promote products | |
| 15 | − * derived from this software without specific prior written permission. | |
| 16 | − * | |
| 17 | − * | |
| 18 | − * ALTERNATIVELY, this software may be distributed under the terms of the | |
| 19 | − * GNU General Public License ("GPL") as published by the Free Software | |
| 20 | − * Foundation, either version 2 of that License or (at your option) any | |
| 21 | − * later version. | |
| 22 | − * | |
| 23 | − * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY | |
| 24 | − * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | |
| 25 | − * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | |
| 26 | − * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | |
| 27 | − * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | |
| 28 | − * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | |
| 29 | − * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | |
| 30 | − * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | |
| 31 | − * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | |
| 32 | − * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
| 33 | − */ | |
| 34 | − | |
| 35 | −fman@400000 { | |
| 36 | − fman0_rx_0x09: port@89000 { | |
| 37 | − cell-index = <0x9>; | |
| 38 | − compatible = "fsl,fman-v3-port-rx"; | |
| 39 | − reg = <0x89000 0x1000>; | |
| 40 | − fsl,fman-10g-port; | |
| 41 | − fsl,fman-best-effort-port; | |
| 42 | − }; | |
| 43 | − | |
| 44 | − fman0_tx_0x29: port@a9000 { | |
| 45 | − cell-index = <0x29>; | |
| 46 | − compatible = "fsl,fman-v3-port-tx"; | |
| 47 | − reg = <0xa9000 0x1000>; | |
| 48 | − fsl,fman-10g-port; | |
| 49 | − fsl,fman-best-effort-port; | |
| 50 | − }; | |
| 51 | − | |
| 52 | − ethernet@e2000 { | |
| 53 | − cell-index = <1>; | |
| 54 | − compatible = "fsl,fman-memac"; | |
| 55 | − reg = <0xe2000 0x1000>; | |
| 56 | − fsl,fman-ports = <&fman0_rx_0x09 &fman0_tx_0x29>; | |
| 57 | − ptp-timer = <&ptp_timer0>; | |
| 58 | − pcsphy-handle = <&pcsphy1>, <&qsgmiia_pcs1>; | |
| 59 | − pcs-handle-names = "sgmii", "qsgmii"; | |
| 60 | − }; | |
| 61 | − | |
| 62 | − mdio@e1000 { | |
| 63 | − qsgmiia_pcs1: ethernet-pcs@1 { | |
| 64 | − compatible = "fsl,lynx-pcs"; | |
| 65 | − reg = <1>; | |
| 66 | − }; | |
| 67 | − }; | |
| 68 | − | |
| 69 | − mdio@e3000 { | |
| 70 | − #address-cells = <1>; | |
| 71 | − #size-cells = <0>; | |
| 72 | − compatible = "fsl,fman-memac-mdio", "fsl,fman-xmdio"; | |
| 73 | − reg = <0xe3000 0x1000>; | |
| 74 | − fsl,erratum-a011043; /* must ignore read errors */ | |
| 75 | − | |
| 76 | − pcsphy1: ethernet-phy@0 { | |
| 77 | − reg = <0x0>; | |
| 78 | − }; | |
| 79 | − }; | |
| 80 | −}; | |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | ||
| 37 | 37 | }; |
| 38 | 38 | }; |
| 39 | 39 | |
| 40 | − memory { | |
| 40 | + memory@0 { | |
| 41 | 41 | device_type = "memory"; |
| 42 | 42 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 43 | 43 | }; |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | ||
| 38 | 38 | }; |
| 39 | 39 | }; |
| 40 | 40 | |
| 41 | − memory { | |
| 41 | + memory@0 { | |
| 42 | 42 | device_type = "memory"; |
| 43 | 43 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 44 | 44 | }; |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | 8 | /dts-v1/; |
| 9 | +#include <dt-bindings/interrupt-controller/irq.h> | |
| 9 | 10 | |
| 10 | 11 | / { |
| 11 | 12 | model = "MPC8313ERDB"; |
| @@ -38,7 +39,7 @@ · 26 unmodified lines | ||
| 38 | 39 | }; |
| 39 | 40 | }; |
| 40 | 41 | |
| 41 | − memory { | |
| 42 | + memory@0 { | |
| 42 | 43 | device_type = "memory"; |
| 43 | 44 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 44 | 45 | }; |
| @@ -48,7 +49,7 @@ · 3 unmodified lines | ||
| 48 | 49 | #size-cells = <1>; |
| 49 | 50 | compatible = "fsl,mpc8313-elbc", "fsl,elbc", "simple-bus"; |
| 50 | 51 | reg = <0xe0005000 0x1000>; |
| 51 | − interrupts = <77 0x8>; | |
| 52 | + interrupts = <77 IRQ_TYPE_LEVEL_LOW>; | |
| 52 | 53 | interrupt-parent = <&ipic>; |
| 53 | 54 | |
| 54 | 55 | // CS0 and CS1 are swapped when |
| @@ -118,7 +119,7 @@ · 63 unmodified lines | ||
| 118 | 119 | cell-index = <0>; |
| 119 | 120 | compatible = "fsl-i2c"; |
| 120 | 121 | reg = <0x3000 0x100>; |
| 121 | − interrupts = <14 0x8>; | |
| 122 | + interrupts = <14 IRQ_TYPE_LEVEL_LOW>; | |
| 122 | 123 | interrupt-parent = <&ipic>; |
| 123 | 124 | dfsrr; |
| 124 | 125 | rtc@68 { |
| @@ -131,7 +132,7 @@ · 6 unmodified lines | ||
| 131 | 132 | compatible = "fsl,sec2.2", "fsl,sec2.1", |
| 132 | 133 | "fsl,sec2.0"; |
| 133 | 134 | reg = <0x30000 0x10000>; |
| 134 | − interrupts = <11 0x8>; | |
| 135 | + interrupts = <11 IRQ_TYPE_LEVEL_LOW>; | |
| 135 | 136 | interrupt-parent = <&ipic>; |
| 136 | 137 | fsl,num-channels = <1>; |
| 137 | 138 | fsl,channel-fifo-len = <24>; |
| @@ -146,7 +147,7 @@ · 8 unmodified lines | ||
| 146 | 147 | cell-index = <1>; |
| 147 | 148 | compatible = "fsl-i2c"; |
| 148 | 149 | reg = <0x3100 0x100>; |
| 149 | − interrupts = <15 0x8>; | |
| 150 | + interrupts = <15 IRQ_TYPE_LEVEL_LOW>; | |
| 150 | 151 | interrupt-parent = <&ipic>; |
| 151 | 152 | dfsrr; |
| 152 | 153 | }; |
| @@ -155,7 +156,7 @@ · 2 unmodified lines | ||
| 155 | 156 | cell-index = <0>; |
| 156 | 157 | compatible = "fsl,spi"; |
| 157 | 158 | reg = <0x7000 0x1000>; |
| 158 | − interrupts = <16 0x8>; | |
| 159 | + interrupts = <16 IRQ_TYPE_LEVEL_LOW>; | |
| 159 | 160 | interrupt-parent = <&ipic>; |
| 160 | 161 | mode = "cpu"; |
| 161 | 162 | }; |
| @@ -167,7 +168,7 @@ · 5 unmodified lines | ||
| 167 | 168 | #address-cells = <1>; |
| 168 | 169 | #size-cells = <0>; |
| 169 | 170 | interrupt-parent = <&ipic>; |
| 170 | − interrupts = <38 0x8>; | |
| 171 | + interrupts = <38 IRQ_TYPE_LEVEL_LOW>; | |
| 171 | 172 | phy_type = "utmi_wide"; |
| 172 | 173 | sleep = <&pmc 0x00300000>; |
| 173 | 174 | }; |
| @@ -175,7 +176,8 @@ · 1 unmodified line | ||
| 175 | 176 | ptp_clock@24E00 { |
| 176 | 177 | compatible = "fsl,etsec-ptp"; |
| 177 | 178 | reg = <0x24E00 0xB0>; |
| 178 | − interrupts = <12 0x8 13 0x8>; | |
| 179 | + interrupts = <12 IRQ_TYPE_LEVEL_LOW>, | |
| 180 | + <13 IRQ_TYPE_LEVEL_LOW>; | |
| 179 | 181 | interrupt-parent = < &ipic >; |
| 180 | 182 | fsl,tclk-period = <10>; |
| 181 | 183 | fsl,tmr-prsc = <100>; |
| @@ -197,7 +199,9 @@ · 15 unmodified lines | ||
| 197 | 199 | compatible = "gianfar"; |
| 198 | 200 | reg = <0x24000 0x1000>; |
| 199 | 201 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 200 | − interrupts = <37 0x8 36 0x8 35 0x8>; | |
| 202 | + interrupts = <37 IRQ_TYPE_LEVEL_LOW>, | |
| 203 | + <36 IRQ_TYPE_LEVEL_LOW>, | |
| 204 | + <35 IRQ_TYPE_LEVEL_LOW>; | |
| 201 | 205 | interrupt-parent = <&ipic>; |
| 202 | 206 | tbi-handle = < &tbi0 >; |
| 203 | 207 | /* Vitesse 7385 isn't on the MDIO bus */ |
| @@ -211,7 +215,7 @@ · 7 unmodified lines | ||
| 211 | 215 | reg = <0x520 0x20>; |
| 212 | 216 | phy4: ethernet-phy@4 { |
| 213 | 217 | interrupt-parent = <&ipic>; |
| 214 | − interrupts = <20 0x8>; | |
| 218 | + interrupts = <20 IRQ_TYPE_LEVEL_LOW>; | |
| 215 | 219 | reg = <0x4>; |
| 216 | 220 | }; |
| 217 | 221 | tbi0: tbi-phy@11 { |
| @@ -231,7 +235,9 @@ · 13 unmodified lines | ||
| 231 | 235 | reg = <0x25000 0x1000>; |
| 232 | 236 | ranges = <0x0 0x25000 0x1000>; |
| 233 | 237 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 234 | − interrupts = <34 0x8 33 0x8 32 0x8>; | |
| 238 | + interrupts = <34 IRQ_TYPE_LEVEL_LOW>, | |
| 239 | + <33 IRQ_TYPE_LEVEL_LOW>, | |
| 240 | + <32 IRQ_TYPE_LEVEL_LOW>; | |
| 235 | 241 | interrupt-parent = <&ipic>; |
| 236 | 242 | tbi-handle = < &tbi1 >; |
| 237 | 243 | phy-handle = < &phy4 >; |
| @@ -259,7 +265,7 @@ · 21 unmodified lines | ||
| 259 | 265 | compatible = "fsl,ns16550", "ns16550"; |
| 260 | 266 | reg = <0x4500 0x100>; |
| 261 | 267 | clock-frequency = <0>; |
| 262 | − interrupts = <9 0x8>; | |
| 268 | + interrupts = <9 IRQ_TYPE_LEVEL_LOW>; | |
| 263 | 269 | interrupt-parent = <&ipic>; |
| 264 | 270 | }; |
| 265 | 271 | |
| @@ -269,15 +275,12 @@ · 3 unmodified lines | ||
| 269 | 275 | compatible = "fsl,ns16550", "ns16550"; |
| 270 | 276 | reg = <0x4600 0x100>; |
| 271 | 277 | clock-frequency = <0>; |
| 272 | − interrupts = <10 0x8>; | |
| 278 | + interrupts = <10 IRQ_TYPE_LEVEL_LOW>; | |
| 273 | 279 | interrupt-parent = <&ipic>; |
| 274 | 280 | }; |
| 275 | 281 | |
| 276 | 282 | /* IPIC |
| 277 | − * interrupts cell = <intr #, sense> | |
| 278 | − * sense values match linux IORESOURCE_IRQ_* defines: | |
| 279 | − * sense == 8: Level, low assertion | |
| 280 | − * sense == 2: Edge, high-to-low change | |
| 283 | + * interrupts cell = <intr #, type> | |
| 281 | 284 | */ |
| 282 | 285 | ipic: pic@700 { |
| 283 | 286 | interrupt-controller; |
| @@ -290,7 +293,7 @@ · 6 unmodified lines | ||
| 290 | 293 | pmc: power@b00 { |
| 291 | 294 | compatible = "fsl,mpc8313-pmc", "fsl,mpc8349-pmc"; |
| 292 | 295 | reg = <0xb00 0x100 0xa00 0x100>; |
| 293 | − interrupts = <80 8>; | |
| 296 | + interrupts = <80 IRQ_TYPE_LEVEL_LOW>; | |
| 294 | 297 | interrupt-parent = <&ipic>; |
| 295 | 298 | fsl,mpc8313-wakeup-timer = <>m1>; |
| 296 | 299 | |
| @@ -306,14 +309,20 @@ · 9 unmodified lines | ||
| 306 | 309 | gtm1: timer@500 { |
| 307 | 310 | compatible = "fsl,mpc8313-gtm", "fsl,gtm"; |
| 308 | 311 | reg = <0x500 0x100>; |
| 309 | − interrupts = <90 8 78 8 84 8 72 8>; | |
| 312 | + interrupts = <90 IRQ_TYPE_LEVEL_LOW>, | |
| 313 | + <78 IRQ_TYPE_LEVEL_LOW>, | |
| 314 | + <84 IRQ_TYPE_LEVEL_LOW>, | |
| 315 | + <72 IRQ_TYPE_LEVEL_LOW>; | |
| 310 | 316 | interrupt-parent = <&ipic>; |
| 311 | 317 | }; |
| 312 | 318 | |
| 313 | 319 | timer@600 { |
| 314 | 320 | compatible = "fsl,mpc8313-gtm", "fsl,gtm"; |
| 315 | 321 | reg = <0x600 0x100>; |
| 316 | − interrupts = <91 8 79 8 85 8 73 8>; | |
| 322 | + interrupts = <91 IRQ_TYPE_LEVEL_LOW>, | |
| 323 | + <79 IRQ_TYPE_LEVEL_LOW>, | |
| 324 | + <85 IRQ_TYPE_LEVEL_LOW>, | |
| 325 | + <73 IRQ_TYPE_LEVEL_LOW>; | |
| 317 | 326 | interrupt-parent = <&ipic>; |
| 318 | 327 | }; |
| 319 | 328 | }; |
| @@ -341,7 +350,7 @@ · 21 unmodified lines | ||
| 341 | 350 | 0x7800 0x0 0x0 0x3 &ipic 17 0x8 |
| 342 | 351 | 0x7800 0x0 0x0 0x4 &ipic 18 0x8>; |
| 343 | 352 | interrupt-parent = <&ipic>; |
| 344 | − interrupts = <66 0x8>; | |
| 353 | + interrupts = <66 IRQ_TYPE_LEVEL_LOW>; | |
| 345 | 354 | bus-range = <0x0 0x0>; |
| 346 | 355 | ranges = <0x02000000 0x0 0x90000000 0x90000000 0x0 0x10000000 |
| 347 | 356 | 0x42000000 0x0 0x80000000 0x80000000 0x0 0x10000000 |
| @@ -363,14 +372,14 @@ · 15 unmodified lines | ||
| 363 | 372 | reg = <0xe00082a8 4>; |
| 364 | 373 | ranges = <0 0xe0008100 0x1a8>; |
| 365 | 374 | interrupt-parent = <&ipic>; |
| 366 | − interrupts = <71 8>; | |
| 375 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 367 | 376 | |
| 368 | 377 | dma-channel@0 { |
| 369 | 378 | compatible = "fsl,mpc8313-dma-channel", |
| 370 | 379 | "fsl,elo-dma-channel"; |
| 371 | 380 | reg = <0 0x28>; |
| 372 | 381 | interrupt-parent = <&ipic>; |
| 373 | − interrupts = <71 8>; | |
| 382 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 374 | 383 | cell-index = <0>; |
| 375 | 384 | }; |
| 376 | 385 | |
| @@ -379,7 +388,7 @@ · 2 unmodified lines | ||
| 379 | 388 | "fsl,elo-dma-channel"; |
| 380 | 389 | reg = <0x80 0x28>; |
| 381 | 390 | interrupt-parent = <&ipic>; |
| 382 | − interrupts = <71 8>; | |
| 391 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 383 | 392 | cell-index = <1>; |
| 384 | 393 | }; |
| 385 | 394 | |
| @@ -388,7 +397,7 @@ · 2 unmodified lines | ||
| 388 | 397 | "fsl,elo-dma-channel"; |
| 389 | 398 | reg = <0x100 0x28>; |
| 390 | 399 | interrupt-parent = <&ipic>; |
| 391 | − interrupts = <71 8>; | |
| 400 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 392 | 401 | cell-index = <2>; |
| 393 | 402 | }; |
| 394 | 403 | |
| @@ -397,7 +406,7 @@ · 2 unmodified lines | ||
| 397 | 406 | "fsl,elo-dma-channel"; |
| 398 | 407 | reg = <0x180 0x28>; |
| 399 | 408 | interrupt-parent = <&ipic>; |
| 400 | − interrupts = <71 8>; | |
| 409 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 401 | 410 | cell-index = <3>; |
| 402 | 411 | }; |
| 403 | 412 | }; |
| @@ -40,7 +40,7 @@ · 39 unmodified lines | ||
| 40 | 40 | }; |
| 41 | 41 | }; |
| 42 | 42 | |
| 43 | − memory { | |
| 43 | + memory@0 { | |
| 44 | 44 | device_type = "memory"; |
| 45 | 45 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 46 | 46 | }; |
| @@ -50,7 +50,7 @@ · 3 unmodified lines | ||
| 50 | 50 | #size-cells = <1>; |
| 51 | 51 | compatible = "fsl,mpc8315-elbc", "fsl,elbc", "simple-bus"; |
| 52 | 52 | reg = <0xe0005000 0x1000>; |
| 53 | − interrupts = <77 0x8>; | |
| 53 | + interrupts = <77 IRQ_TYPE_LEVEL_LOW>; | |
| 54 | 54 | interrupt-parent = <&ipic>; |
| 55 | 55 | |
| 56 | 56 | // CS0 and CS1 are swapped when |
| @@ -112,7 +112,7 @@ · 55 unmodified lines | ||
| 112 | 112 | cell-index = <0>; |
| 113 | 113 | compatible = "fsl-i2c"; |
| 114 | 114 | reg = <0x3000 0x100>; |
| 115 | − interrupts = <14 0x8>; | |
| 115 | + interrupts = <14 IRQ_TYPE_LEVEL_LOW>; | |
| 116 | 116 | interrupt-parent = <&ipic>; |
| 117 | 117 | dfsrr; |
| 118 | 118 | rtc@68 { |
| @@ -133,8 +133,10 @@ · 14 unmodified lines | ||
| 133 | 133 | cell-index = <0>; |
| 134 | 134 | compatible = "fsl,spi"; |
| 135 | 135 | reg = <0x7000 0x1000>; |
| 136 | − interrupts = <16 0x8>; | |
| 136 | + interrupts = <16 IRQ_TYPE_LEVEL_LOW>; | |
| 137 | 137 | interrupt-parent = <&ipic>; |
| 138 | + #address-cells = <1>; | |
| 139 | + #size-cells = <0>; | |
| 138 | 140 | mode = "cpu"; |
| 139 | 141 | }; |
| 140 | 142 | |
| @@ -145,35 +147,35 @@ · 4 unmodified lines | ||
| 145 | 147 | reg = <0x82a8 4>; |
| 146 | 148 | ranges = <0 0x8100 0x1a8>; |
| 147 | 149 | interrupt-parent = <&ipic>; |
| 148 | − interrupts = <71 8>; | |
| 150 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 149 | 151 | cell-index = <0>; |
| 150 | 152 | dma-channel@0 { |
| 151 | 153 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 152 | 154 | reg = <0 0x80>; |
| 153 | 155 | cell-index = <0>; |
| 154 | 156 | interrupt-parent = <&ipic>; |
| 155 | − interrupts = <71 8>; | |
| 157 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 156 | 158 | }; |
| 157 | 159 | dma-channel@80 { |
| 158 | 160 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 159 | 161 | reg = <0x80 0x80>; |
| 160 | 162 | cell-index = <1>; |
| 161 | 163 | interrupt-parent = <&ipic>; |
| 162 | − interrupts = <71 8>; | |
| 164 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 163 | 165 | }; |
| 164 | 166 | dma-channel@100 { |
| 165 | 167 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 166 | 168 | reg = <0x100 0x80>; |
| 167 | 169 | cell-index = <2>; |
| 168 | 170 | interrupt-parent = <&ipic>; |
| 169 | − interrupts = <71 8>; | |
| 171 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 170 | 172 | }; |
| 171 | 173 | dma-channel@180 { |
| 172 | 174 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 173 | 175 | reg = <0x180 0x28>; |
| 174 | 176 | cell-index = <3>; |
| 175 | 177 | interrupt-parent = <&ipic>; |
| 176 | − interrupts = <71 8>; | |
| 178 | + interrupts = <71 IRQ_TYPE_LEVEL_LOW>; | |
| 177 | 179 | }; |
| 178 | 180 | }; |
| 179 | 181 | |
| @@ -183,7 +185,7 @@ · 3 unmodified lines | ||
| 183 | 185 | #address-cells = <1>; |
| 184 | 186 | #size-cells = <0>; |
| 185 | 187 | interrupt-parent = <&ipic>; |
| 186 | − interrupts = <38 0x8>; | |
| 188 | + interrupts = <38 IRQ_TYPE_LEVEL_LOW>; | |
| 187 | 189 | phy_type = "utmi"; |
| 188 | 190 | }; |
| 189 | 191 | |
| @@ -197,7 +199,9 @@ · 7 unmodified lines | ||
| 197 | 199 | reg = <0x24000 0x1000>; |
| 198 | 200 | ranges = <0x0 0x24000 0x1000>; |
| 199 | 201 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 200 | − interrupts = <32 0x8 33 0x8 34 0x8>; | |
| 202 | + interrupts = <32 IRQ_TYPE_LEVEL_LOW>, | |
| 203 | + <33 IRQ_TYPE_LEVEL_LOW>, | |
| 204 | + <34 IRQ_TYPE_LEVEL_LOW>; | |
| 201 | 205 | interrupt-parent = <&ipic>; |
| 202 | 206 | tbi-handle = <&tbi0>; |
| 203 | 207 | phy-handle = < &phy0 >; |
| @@ -238,7 +242,9 @@ · 34 unmodified lines | ||
| 238 | 242 | reg = <0x25000 0x1000>; |
| 239 | 243 | ranges = <0x0 0x25000 0x1000>; |
| 240 | 244 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 241 | − interrupts = <35 0x8 36 0x8 37 0x8>; | |
| 245 | + interrupts = <35 IRQ_TYPE_LEVEL_LOW>, | |
| 246 | + <36 IRQ_TYPE_LEVEL_LOW>, | |
| 247 | + <37 IRQ_TYPE_LEVEL_LOW>; | |
| 242 | 248 | interrupt-parent = <&ipic>; |
| 243 | 249 | tbi-handle = <&tbi1>; |
| 244 | 250 | phy-handle = < &phy1 >; |
| @@ -263,7 +269,7 @@ · 18 unmodified lines | ||
| 263 | 269 | compatible = "fsl,ns16550", "ns16550"; |
| 264 | 270 | reg = <0x4500 0x100>; |
| 265 | 271 | clock-frequency = <133333333>; |
| 266 | − interrupts = <9 0x8>; | |
| 272 | + interrupts = <9 IRQ_TYPE_LEVEL_LOW>; | |
| 267 | 273 | interrupt-parent = <&ipic>; |
| 268 | 274 | }; |
| 269 | 275 | |
| @@ -273,7 +279,7 @@ · 3 unmodified lines | ||
| 273 | 279 | compatible = "fsl,ns16550", "ns16550"; |
| 274 | 280 | reg = <0x4600 0x100>; |
| 275 | 281 | clock-frequency = <133333333>; |
| 276 | − interrupts = <10 0x8>; | |
| 282 | + interrupts = <10 IRQ_TYPE_LEVEL_LOW>; | |
| 277 | 283 | interrupt-parent = <&ipic>; |
| 278 | 284 | }; |
| 279 | 285 | |
| @@ -282,7 +288,7 @@ · 2 unmodified lines | ||
| 282 | 288 | "fsl,sec2.4", "fsl,sec2.2", "fsl,sec2.1", |
| 283 | 289 | "fsl,sec2.0"; |
| 284 | 290 | reg = <0x30000 0x10000>; |
| 285 | − interrupts = <11 0x8>; | |
| 291 | + interrupts = <11 IRQ_TYPE_LEVEL_LOW>; | |
| 286 | 292 | interrupt-parent = <&ipic>; |
| 287 | 293 | fsl,num-channels = <4>; |
| 288 | 294 | fsl,channel-fifo-len = <24>; |
| @@ -294,7 +300,7 @@ · 5 unmodified lines | ||
| 294 | 300 | compatible = "fsl,mpc8315-sata", "fsl,pq-sata"; |
| 295 | 301 | reg = <0x18000 0x1000>; |
| 296 | 302 | cell-index = <1>; |
| 297 | − interrupts = <44 0x8>; | |
| 303 | + interrupts = <44 IRQ_TYPE_LEVEL_LOW>; | |
| 298 | 304 | interrupt-parent = <&ipic>; |
| 299 | 305 | }; |
| 300 | 306 | |
| @@ -302,14 +308,17 @@ · 1 unmodified line | ||
| 302 | 308 | compatible = "fsl,mpc8315-sata", "fsl,pq-sata"; |
| 303 | 309 | reg = <0x19000 0x1000>; |
| 304 | 310 | cell-index = <2>; |
| 305 | − interrupts = <45 0x8>; | |
| 311 | + interrupts = <45 IRQ_TYPE_LEVEL_LOW>; | |
| 306 | 312 | interrupt-parent = <&ipic>; |
| 307 | 313 | }; |
| 308 | 314 | |
| 309 | 315 | gtm1: timer@500 { |
| 310 | 316 | compatible = "fsl,mpc8315-gtm", "fsl,gtm"; |
| 311 | 317 | reg = <0x500 0x100>; |
| 312 | − interrupts = <90 8 78 8 84 8 72 8>; | |
| 318 | + interrupts = <90 IRQ_TYPE_LEVEL_LOW>, | |
| 319 | + <78 IRQ_TYPE_LEVEL_LOW>, | |
| 320 | + <84 IRQ_TYPE_LEVEL_LOW>, | |
| 321 | + <72 IRQ_TYPE_LEVEL_LOW>; | |
| 313 | 322 | interrupt-parent = <&ipic>; |
| 314 | 323 | clock-frequency = <133333333>; |
| 315 | 324 | }; |
| @@ -317,16 +326,16 @@ · 1 unmodified line | ||
| 317 | 326 | timer@600 { |
| 318 | 327 | compatible = "fsl,mpc8315-gtm", "fsl,gtm"; |
| 319 | 328 | reg = <0x600 0x100>; |
| 320 | − interrupts = <91 8 79 8 85 8 73 8>; | |
| 329 | + interrupts = <91 IRQ_TYPE_LEVEL_LOW>, | |
| 330 | + <79 IRQ_TYPE_LEVEL_LOW>, | |
| 331 | + <85 IRQ_TYPE_LEVEL_LOW>, | |
| 332 | + <73 IRQ_TYPE_LEVEL_LOW>; | |
| 321 | 333 | interrupt-parent = <&ipic>; |
| 322 | 334 | clock-frequency = <133333333>; |
| 323 | 335 | }; |
| 324 | 336 | |
| 325 | 337 | /* IPIC |
| 326 | − * interrupts cell = <intr #, sense> | |
| 327 | − * sense values match linux IORESOURCE_IRQ_* defines: | |
| 328 | − * sense == 8: Level, low assertion | |
| 329 | − * sense == 2: Edge, high-to-low change | |
| 338 | + * interrupts cell = <intr #, type> | |
| 330 | 339 | */ |
| 331 | 340 | ipic: interrupt-controller@700 { |
| 332 | 341 | interrupt-controller; |
| @@ -340,14 +349,14 @@ · 7 unmodified lines | ||
| 340 | 349 | compatible = "fsl,ipic-msi"; |
| 341 | 350 | reg = <0x7c0 0x40>; |
| 342 | 351 | msi-available-ranges = <0 0x100>; |
| 343 | − interrupts = <0x43 0x8 | |
| 344 | − 0x4 0x8 | |
| 345 | − 0x51 0x8 | |
| 346 | − 0x52 0x8 | |
| 347 | − 0x56 0x8 | |
| 348 | − 0x57 0x8 | |
| 349 | − 0x58 0x8 | |
| 350 | − 0x59 0x8>; | |
| 352 | + interrupts = <0x43 IRQ_TYPE_LEVEL_LOW | |
| 353 | + 0x4 IRQ_TYPE_LEVEL_LOW | |
| 354 | + 0x51 IRQ_TYPE_LEVEL_LOW | |
| 355 | + 0x52 IRQ_TYPE_LEVEL_LOW | |
| 356 | + 0x56 IRQ_TYPE_LEVEL_LOW | |
| 357 | + 0x57 IRQ_TYPE_LEVEL_LOW | |
| 358 | + 0x58 IRQ_TYPE_LEVEL_LOW | |
| 359 | + 0x59 IRQ_TYPE_LEVEL_LOW>; | |
| 351 | 360 | interrupt-parent = < &ipic >; |
| 352 | 361 | }; |
| 353 | 362 | |
| @@ -355,7 +364,7 @@ · 1 unmodified line | ||
| 355 | 364 | compatible = "fsl,mpc8315-pmc", "fsl,mpc8313-pmc", |
| 356 | 365 | "fsl,mpc8349-pmc"; |
| 357 | 366 | reg = <0xb00 0x100 0xa00 0x100>; |
| 358 | − interrupts = <80 8>; | |
| 367 | + interrupts = <80 IRQ_TYPE_LEVEL_LOW>; | |
| 359 | 368 | interrupt-parent = <&ipic>; |
| 360 | 369 | fsl,mpc8313-wakeup-timer = <>m1>; |
| 361 | 370 | }; |
| @@ -374,24 +383,24 @@ · 12 unmodified lines | ||
| 374 | 383 | interrupt-map-mask = <0xf800 0x0 0x0 0x7>; |
| 375 | 384 | interrupt-map = < |
| 376 | 385 | /* IDSEL 0x0E -mini PCI */ |
| 377 | − 0x7000 0x0 0x0 0x1 &ipic 18 0x8 | |
| 378 | − 0x7000 0x0 0x0 0x2 &ipic 18 0x8 | |
| 379 | − 0x7000 0x0 0x0 0x3 &ipic 18 0x8 | |
| 380 | − 0x7000 0x0 0x0 0x4 &ipic 18 0x8 | |
| 386 | + 0x7000 0x0 0x0 0x1 &ipic 18 IRQ_TYPE_LEVEL_LOW | |
| 387 | + 0x7000 0x0 0x0 0x2 &ipic 18 IRQ_TYPE_LEVEL_LOW | |
| 388 | + 0x7000 0x0 0x0 0x3 &ipic 18 IRQ_TYPE_LEVEL_LOW | |
| 389 | + 0x7000 0x0 0x0 0x4 &ipic 18 IRQ_TYPE_LEVEL_LOW | |
| 381 | 390 | |
| 382 | 391 | /* IDSEL 0x0F -mini PCI */ |
| 383 | − 0x7800 0x0 0x0 0x1 &ipic 17 0x8 | |
| 384 | − 0x7800 0x0 0x0 0x2 &ipic 17 0x8 | |
| 385 | − 0x7800 0x0 0x0 0x3 &ipic 17 0x8 | |
| 386 | − 0x7800 0x0 0x0 0x4 &ipic 17 0x8 | |
| 392 | + 0x7800 0x0 0x0 0x1 &ipic 17 IRQ_TYPE_LEVEL_LOW | |
| 393 | + 0x7800 0x0 0x0 0x2 &ipic 17 IRQ_TYPE_LEVEL_LOW | |
| 394 | + 0x7800 0x0 0x0 0x3 &ipic 17 IRQ_TYPE_LEVEL_LOW | |
| 395 | + 0x7800 0x0 0x0 0x4 &ipic 17 IRQ_TYPE_LEVEL_LOW | |
| 387 | 396 | |
| 388 | 397 | /* IDSEL 0x10 - PCI slot */ |
| 389 | − 0x8000 0x0 0x0 0x1 &ipic 48 0x8 | |
| 390 | − 0x8000 0x0 0x0 0x2 &ipic 17 0x8 | |
| 391 | − 0x8000 0x0 0x0 0x3 &ipic 48 0x8 | |
| 392 | − 0x8000 0x0 0x0 0x4 &ipic 17 0x8>; | |
| 398 | + 0x8000 0x0 0x0 0x1 &ipic 48 IRQ_TYPE_LEVEL_LOW | |
| 399 | + 0x8000 0x0 0x0 0x2 &ipic 17 IRQ_TYPE_LEVEL_LOW | |
| 400 | + 0x8000 0x0 0x0 0x3 &ipic 48 IRQ_TYPE_LEVEL_LOW | |
| 401 | + 0x8000 0x0 0x0 0x4 &ipic 17 IRQ_TYPE_LEVEL_LOW>; | |
| 393 | 402 | interrupt-parent = <&ipic>; |
| 394 | − interrupts = <66 0x8>; | |
| 403 | + interrupts = <66 IRQ_TYPE_LEVEL_LOW>; | |
| 395 | 404 | bus-range = <0x0 0x0>; |
| 396 | 405 | ranges = <0x02000000 0 0x90000000 0x90000000 0 0x10000000 |
| 397 | 406 | 0x42000000 0 0x80000000 0x80000000 0 0x10000000 |
| @@ -417,10 +426,10 @@ · 19 unmodified lines | ||
| 417 | 426 | 0x01000000 0 0x00000000 0xb1000000 0 0x00800000>; |
| 418 | 427 | bus-range = <0 255>; |
| 419 | 428 | interrupt-map-mask = <0xf800 0 0 7>; |
| 420 | − interrupt-map = <0 0 0 1 &ipic 1 8 | |
| 421 | − 0 0 0 2 &ipic 1 8 | |
| 422 | − 0 0 0 3 &ipic 1 8 | |
| 423 | − 0 0 0 4 &ipic 1 8>; | |
| 429 | + interrupt-map = <0 0 0 1 &ipic 1 IRQ_TYPE_LEVEL_LOW | |
| 430 | + 0 0 0 2 &ipic 1 IRQ_TYPE_LEVEL_LOW | |
| 431 | + 0 0 0 3 &ipic 1 IRQ_TYPE_LEVEL_LOW | |
| 432 | + 0 0 0 4 &ipic 1 IRQ_TYPE_LEVEL_LOW>; | |
| 424 | 433 | clock-frequency = <0>; |
| 425 | 434 | |
| 426 | 435 | pcie@0 { |
| @@ -448,10 +457,10 @@ · 21 unmodified lines | ||
| 448 | 457 | 0x01000000 0 0x00000000 0xd1000000 0 0x00800000>; |
| 449 | 458 | bus-range = <0 255>; |
| 450 | 459 | interrupt-map-mask = <0xf800 0 0 7>; |
| 451 | − interrupt-map = <0 0 0 1 &ipic 2 8 | |
| 452 | − 0 0 0 2 &ipic 2 8 | |
| 453 | − 0 0 0 3 &ipic 2 8 | |
| 454 | − 0 0 0 4 &ipic 2 8>; | |
| 460 | + interrupt-map = <0 0 0 1 &ipic 2 IRQ_TYPE_LEVEL_LOW | |
| 461 | + 0 0 0 2 &ipic 2 IRQ_TYPE_LEVEL_LOW | |
| 462 | + 0 0 0 3 &ipic 2 IRQ_TYPE_LEVEL_LOW | |
| 463 | + 0 0 0 4 &ipic 2 IRQ_TYPE_LEVEL_LOW>; | |
| 455 | 464 | clock-frequency = <0>; |
| 456 | 465 | |
| 457 | 466 | pcie@0 { |
| @@ -471,12 +480,12 @@ · 13 unmodified lines | ||
| 471 | 480 | leds { |
| 472 | 481 | compatible = "gpio-leds"; |
| 473 | 482 | |
| 474 | − pwr { | |
| 483 | + led-pwr { | |
| 475 | 484 | gpios = <&mcu_pio 0 0>; |
| 476 | 485 | default-state = "on"; |
| 477 | 486 | }; |
| 478 | 487 | |
| 479 | − hdd { | |
| 488 | + led-hdd { | |
| 480 | 489 | gpios = <&mcu_pio 1 0>; |
| 481 | 490 | linux,default-trigger = "disk-activity"; |
| 482 | 491 | }; |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | ||
| 38 | 38 | }; |
| 39 | 39 | }; |
| 40 | 40 | |
| 41 | − memory { | |
| 41 | + memory@0 { | |
| 42 | 42 | device_type = "memory"; |
| 43 | 43 | reg = <0x00000000 0x04000000>; |
| 44 | 44 | }; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | ||
| 39 | 39 | }; |
| 40 | 40 | }; |
| 41 | 41 | |
| 42 | − memory { | |
| 42 | + memory@0 { | |
| 43 | 43 | device_type = "memory"; |
| 44 | 44 | reg = <0x00000000 0x10000000>; |
| 45 | 45 | }; |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | ||
| 37 | 37 | }; |
| 38 | 38 | }; |
| 39 | 39 | |
| 40 | − memory { | |
| 40 | + memory@0 { | |
| 41 | 41 | device_type = "memory"; |
| 42 | 42 | reg = <0x00000000 0x10000000>; |
| 43 | 43 | }; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | ||
| 39 | 39 | }; |
| 40 | 40 | }; |
| 41 | 41 | |
| 42 | − memory { | |
| 42 | + memory@0 { | |
| 43 | 43 | device_type = "memory"; |
| 44 | 44 | reg = <0x00000000 0x10000000>; // 256MB at 0 |
| 45 | 45 | }; |
| @@ -40,7 +40,7 @@ · 39 unmodified lines | ||
| 40 | 40 | }; |
| 41 | 41 | }; |
| 42 | 42 | |
| 43 | − memory { | |
| 43 | + memory@0 { | |
| 44 | 44 | device_type = "memory"; |
| 45 | 45 | reg = <0x00000000 0x20000000>; // 512MB at 0 |
| 46 | 46 | }; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | ||
| 39 | 39 | }; |
| 40 | 40 | }; |
| 41 | 41 | |
| 42 | − memory { | |
| 42 | + memory@0 { | |
| 43 | 43 | device_type = "memory"; |
| 44 | 44 | reg = <0x00000000 0x10000000>; // 256MB at 0 |
| 45 | 45 | }; |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | ||
| 37 | 37 | }; |
| 38 | 38 | }; |
| 39 | 39 | |
| 40 | − memory { | |
| 40 | + memory@0 { | |
| 41 | 41 | device_type = "memory"; |
| 42 | 42 | reg = <0x00000000 0x10000000>; // 256MB at 0 |
| 43 | 43 | }; |
| @@ -120,10 +120,8 @@ · 119 unmodified lines | ||
| 120 | 120 | |
| 121 | 121 | #if defined(CONFIG_44x) |
| 122 | 122 | #include <asm/nohash/32/pte-44x.h> |
| 123 | −#elif defined(CONFIG_PPC_85xx) && defined(CONFIG_PTE_64BIT) | |
| 124 | −#include <asm/nohash/pte-e500.h> | |
| 125 | 123 | #elif defined(CONFIG_PPC_85xx) |
| 126 | −#include <asm/nohash/32/pte-85xx.h> | |
| 124 | +#include <asm/nohash/pte-e500.h> | |
| 127 | 125 | #elif defined(CONFIG_PPC_8xx) |
| 128 | 126 | #include <asm/nohash/32/pte-8xx.h> |
| 129 | 127 | #endif |
| @@ -1,59 +0,0 @@ | ||
| 1 | −/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | −#ifndef _ASM_POWERPC_NOHASH_32_PTE_85xx_H | |
| 3 | −#define _ASM_POWERPC_NOHASH_32_PTE_85xx_H | |
| 4 | −#ifdef __KERNEL__ | |
| 5 | − | |
| 6 | −/* PTE bit definitions for Freescale BookE SW loaded TLB MMU based | |
| 7 | − * processors | |
| 8 | − * | |
| 9 | − MMU Assist Register 3: | |
| 10 | − | |
| 11 | − 32 33 34 35 36 ... 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | |
| 12 | − RPN...................... 0 0 U0 U1 U2 U3 UX SX UW SW UR SR | |
| 13 | − | |
| 14 | − - PRESENT *must* be in the bottom two bits because swap PTEs use | |
| 15 | − the top 30 bits. | |
| 16 | − | |
| 17 | −*/ | |
| 18 | − | |
| 19 | −/* Definitions for FSL Book-E Cores */ | |
| 20 | −#define _PAGE_READ 0x00001 /* H: Read permission (SR) */ | |
| 21 | −#define _PAGE_PRESENT 0x00002 /* S: PTE contains a translation */ | |
| 22 | −#define _PAGE_WRITE 0x00004 /* S: Write permission (SW) */ | |
| 23 | −#define _PAGE_DIRTY 0x00008 /* S: Page dirty */ | |
| 24 | −#define _PAGE_EXEC 0x00010 /* H: SX permission */ | |
| 25 | −#define _PAGE_ACCESSED 0x00020 /* S: Page referenced */ | |
| 26 | − | |
| 27 | −#define _PAGE_ENDIAN 0x00040 /* H: E bit */ | |
| 28 | −#define _PAGE_GUARDED 0x00080 /* H: G bit */ | |
| 29 | −#define _PAGE_COHERENT 0x00100 /* H: M bit */ | |
| 30 | −#define _PAGE_NO_CACHE 0x00200 /* H: I bit */ | |
| 31 | −#define _PAGE_WRITETHRU 0x00400 /* H: W bit */ | |
| 32 | −#define _PAGE_SPECIAL 0x00800 /* S: Special page */ | |
| 33 | − | |
| 34 | −#define _PMD_PRESENT 0 | |
| 35 | −#define _PMD_PRESENT_MASK (PAGE_MASK) | |
| 36 | −#define _PMD_BAD (~PAGE_MASK) | |
| 37 | −#define _PMD_USER 0 | |
| 38 | − | |
| 39 | −#define _PTE_NONE_MASK 0 | |
| 40 | − | |
| 41 | −#define PTE_WIMGE_SHIFT (6) | |
| 42 | − | |
| 43 | −/* | |
| 44 | − * We define 2 sets of base prot bits, one for basic pages (ie, | |
| 45 | − * cacheable kernel and user pages) and one for non cacheable | |
| 46 | − * pages. We always set _PAGE_COHERENT when SMP is enabled or | |
| 47 | − * the processor might need it for DMA coherency. | |
| 48 | − */ | |
| 49 | −#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED) | |
| 50 | −#if defined(CONFIG_SMP) || defined(CONFIG_PPC_E500MC) | |
| 51 | −#define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT) | |
| 52 | −#else | |
| 53 | −#define _PAGE_BASE (_PAGE_BASE_NC) | |
| 54 | −#endif | |
| 55 | − | |
| 56 | −#include <asm/pgtable-masks.h> | |
| 57 | − | |
| 58 | −#endif /* __KERNEL__ */ | |
| 59 | −#endif /* _ASM_POWERPC_NOHASH_32_PTE_FSL_85xx_H */ | |
| @@ -49,7 +49,7 @@ static inline unsigned long pud_val(pud_t x) · 48 unmodified lines | ||
| 49 | 49 | #endif /* CONFIG_PPC64 */ |
| 50 | 50 | |
| 51 | 51 | /* PGD level */ |
| 52 | −#if defined(CONFIG_PPC_85xx) && defined(CONFIG_PTE_64BIT) | |
| 52 | +#if defined(CONFIG_PPC_85xx) | |
| 53 | 53 | typedef struct { unsigned long long pgd; } pgd_t; |
| 54 | 54 | |
| 55 | 55 | static inline unsigned long long pgd_val(pgd_t x) |
| @@ -15,6 +15,9 @@ · 14 unmodified lines | ||
| 15 | 15 | #define TASK_SIZE_MAX TASK_SIZE_USER64 |
| 16 | 16 | #endif |
| 17 | 17 | |
| 18 | +/* Threshold above which VMX copy path is used */ | |
| 19 | +#define VMX_COPY_THRESHOLD 3328 | |
| 20 | + | |
| 18 | 21 | #include <asm-generic/access_ok.h> |
| 19 | 22 | |
| 20 | 23 | /* |
| @@ -255,7 +258,7 @@ __gus_failed: \ · 234 unmodified lines | ||
| 255 | 258 | ".section .fixup,\"ax\"\n" \ |
| 256 | 259 | "4: li %0,%3\n" \ |
| 257 | 260 | " li %1,0\n" \ |
| 258 | − " li %1+1,0\n" \ | |
| 261 | + " li %L1,0\n" \ | |
| 259 | 262 | " b 3b\n" \ |
| 260 | 263 | ".previous\n" \ |
| 261 | 264 | EX_TABLE(1b, 4b) \ |
| @@ -326,40 +329,62 @@ do { \ · 64 unmodified lines | ||
| 326 | 329 | extern unsigned long __copy_tofrom_user(void __user *to, |
| 327 | 330 | const void __user *from, unsigned long size); |
| 328 | 331 | |
| 329 | −#ifdef __powerpc64__ | |
| 330 | −static inline unsigned long | |
| 331 | −raw_copy_in_user(void __user *to, const void __user *from, unsigned long n) | |
| 332 | +unsigned long __copy_tofrom_user_base(void __user *to, | |
| 333 | + const void __user *from, unsigned long size); | |
| 334 | + | |
| 335 | +unsigned long __copy_tofrom_user_power7_vmx(void __user *to, | |
| 336 | + const void __user *from, unsigned long size); | |
| 337 | + | |
| 338 | +static __always_inline bool will_use_vmx(unsigned long n) | |
| 339 | +{ | |
| 340 | + return IS_ENABLED(CONFIG_ALTIVEC) && cpu_has_feature(CPU_FTR_VMX_COPY) && | |
| 341 | + n > VMX_COPY_THRESHOLD; | |
| 342 | +} | |
| 343 | + | |
| 344 | +static __always_inline unsigned long | |
| 345 | +raw_copy_tofrom_user(void __user *to, const void __user *from, | |
| 346 | + unsigned long n, unsigned long dir) | |
| 332 | 347 | { |
| 333 | 348 | unsigned long ret; |
| 334 | 349 | |
| 335 | − barrier_nospec(); | |
| 336 | − allow_user_access(to, KUAP_READ_WRITE); | |
| 350 | + if (will_use_vmx(n) && enter_vmx_usercopy()) { | |
| 351 | + allow_user_access(to, dir); | |
| 352 | + ret = __copy_tofrom_user_power7_vmx(to, from, n); | |
| 353 | + prevent_user_access(dir); | |
| 354 | + exit_vmx_usercopy(); | |
| 355 | + | |
| 356 | + if (unlikely(ret)) { | |
| 357 | + allow_user_access(to, dir); | |
| 358 | + ret = __copy_tofrom_user_base(to, from, n); | |
| 359 | + prevent_user_access(dir); | |
| 360 | + } | |
| 361 | + return ret; | |
| 362 | + } | |
| 363 | + | |
| 364 | + allow_user_access(to, dir); | |
| 337 | 365 | ret = __copy_tofrom_user(to, from, n); |
| 338 | − prevent_user_access(KUAP_READ_WRITE); | |
| 366 | + prevent_user_access(dir); | |
| 339 | 367 | return ret; |
| 340 | 368 | } |
| 341 | −#endif /* __powerpc64__ */ | |
| 342 | 369 | |
| 343 | −static inline unsigned long raw_copy_from_user(void *to, | |
| 344 | − const void __user *from, unsigned long n) | |
| 370 | +#ifdef CONFIG_PPC64 | |
| 371 | +static inline unsigned long | |
| 372 | +raw_copy_in_user(void __user *to, const void __user *from, unsigned long n) | |
| 345 | 373 | { |
| 346 | − unsigned long ret; | |
| 374 | + barrier_nospec(); | |
| 375 | + return raw_copy_tofrom_user(to, from, n, KUAP_READ_WRITE); | |
| 376 | +} | |
| 377 | +#endif /* CONFIG_PPC64 */ | |
| 347 | 378 | |
| 348 | − allow_user_access(NULL, KUAP_READ); | |
| 349 | − ret = __copy_tofrom_user((__force void __user *)to, from, n); | |
| 350 | − prevent_user_access(KUAP_READ); | |
| 351 | − return ret; | |
| 379 | +static inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n) | |
| 380 | +{ | |
| 381 | + return raw_copy_tofrom_user((__force void __user *)to, from, n, KUAP_READ); | |
| 352 | 382 | } |
| 353 | 383 | |
| 354 | 384 | static inline unsigned long |
| 355 | 385 | raw_copy_to_user(void __user *to, const void *from, unsigned long n) |
| 356 | 386 | { |
| 357 | − unsigned long ret; | |
| 358 | − | |
| 359 | − allow_user_access(to, KUAP_WRITE); | |
| 360 | − ret = __copy_tofrom_user(to, (__force const void __user *)from, n); | |
| 361 | − prevent_user_access(KUAP_WRITE); | |
| 362 | − return ret; | |
| 387 | + return raw_copy_tofrom_user(to, (__force const void __user *)from, n, KUAP_WRITE); | |
| 363 | 388 | } |
| 364 | 389 | |
| 365 | 390 | unsigned long __arch_clear_user(void __user *addr, unsigned long size); |
| @@ -67,7 +67,7 @@ bool arch_dma_unmap_sg_direct(struct device *dev, struct scatterlist *sg, · 66 unmodified lines | ||
| 67 | 67 | } |
| 68 | 68 | bool arch_dma_alloc_direct(struct device *dev) |
| 69 | 69 | { |
| 70 | − if (dev->dma_ops_bypass) | |
| 70 | + if (dev->dma_ops_bypass && dev->bus_dma_limit) | |
| 71 | 71 | return true; |
| 72 | 72 | |
| 73 | 73 | return false; |
| @@ -75,7 +75,7 @@ bool arch_dma_alloc_direct(struct device *dev) · 1 unmodified line | ||
| 75 | 75 | |
| 76 | 76 | bool arch_dma_free_direct(struct device *dev, dma_addr_t dma_handle) |
| 77 | 77 | { |
| 78 | − if (!dev->dma_ops_bypass) | |
| 78 | + if (!dev->dma_ops_bypass || !dev->bus_dma_limit) | |
| 79 | 79 | return false; |
| 80 | 80 | |
| 81 | 81 | return is_direct_handle(dev, dma_handle); |
| @@ -305,7 +305,6 @@ set_ivor: · 304 unmodified lines | ||
| 305 | 305 | * r12 is pointer to the pte |
| 306 | 306 | * r10 is the pshift from the PGD, if we're a hugepage |
| 307 | 307 | */ |
| 308 | −#ifdef CONFIG_PTE_64BIT | |
| 309 | 308 | #ifdef CONFIG_HUGETLB_PAGE |
| 310 | 309 | #define FIND_PTE \ |
| 311 | 310 | rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ |
| @@ -329,15 +328,6 @@ set_ivor: · 17 unmodified lines | ||
| 329 | 328 | rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ |
| 330 | 329 | lwz r11, 4(r12); /* Get pte entry */ |
| 331 | 330 | #endif /* HUGEPAGE */ |
| 332 | −#else /* !PTE_64BIT */ | |
| 333 | −#define FIND_PTE \ | |
| 334 | − rlwimi r11, r13, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ | |
| 335 | − lwz r11, 0(r11); /* Get L1 entry */ \ | |
| 336 | − rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \ | |
| 337 | − beq 2f; /* Bail if no table */ \ | |
| 338 | − rlwimi r12, r13, 22, 20, 29; /* Compute PTE address */ \ | |
| 339 | − lwz r11, 0(r12); /* Get Linux PTE */ | |
| 340 | −#endif | |
| 341 | 331 | |
| 342 | 332 | /* |
| 343 | 333 | * Interrupt vector entry code |
| @@ -473,21 +463,15 @@ END_BTB_FLUSH_SECTION · 129 unmodified lines | ||
| 473 | 463 | 4: |
| 474 | 464 | FIND_PTE |
| 475 | 465 | |
| 476 | −#ifdef CONFIG_PTE_64BIT | |
| 477 | 466 | li r13,_PAGE_PRESENT|_PAGE_BAP_SR |
| 478 | 467 | oris r13,r13,_PAGE_ACCESSED@h |
| 479 | −#else | |
| 480 | − li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED | |
| 481 | −#endif | |
| 482 | 468 | andc. r13,r13,r11 /* Check permission */ |
| 483 | 469 | |
| 484 | −#ifdef CONFIG_PTE_64BIT | |
| 485 | 470 | #ifdef CONFIG_SMP |
| 486 | 471 | subf r13,r11,r12 /* create false data dep */ |
| 487 | 472 | lwzx r13,r11,r13 /* Get upper pte bits */ |
| 488 | 473 | #else |
| 489 | 474 | lwz r13,0(r12) /* Get upper pte bits */ |
| 490 | −#endif | |
| 491 | 475 | #endif |
| 492 | 476 | |
| 493 | 477 | bne 2f /* Bail if permission/valid mismatch */ |
| @@ -552,12 +536,8 @@ END_BTB_FLUSH_SECTION · 58 unmodified lines | ||
| 552 | 536 | |
| 553 | 537 | FIND_PTE |
| 554 | 538 | /* Make up the required permissions for kernel code */ |
| 555 | −#ifdef CONFIG_PTE_64BIT | |
| 556 | 539 | li r13,_PAGE_PRESENT | _PAGE_BAP_SX |
| 557 | 540 | oris r13,r13,_PAGE_ACCESSED@h |
| 558 | −#else | |
| 559 | − li r13,_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_EXEC | |
| 560 | −#endif | |
| 561 | 541 | b 4f |
| 562 | 542 | |
| 563 | 543 | /* Get the PGD for the current thread */ |
| @@ -573,23 +553,17 @@ END_BTB_FLUSH_SECTION · 9 unmodified lines | ||
| 573 | 553 | |
| 574 | 554 | FIND_PTE |
| 575 | 555 | /* Make up the required permissions for user code */ |
| 576 | −#ifdef CONFIG_PTE_64BIT | |
| 577 | 556 | li r13,_PAGE_PRESENT | _PAGE_BAP_UX |
| 578 | 557 | oris r13,r13,_PAGE_ACCESSED@h |
| 579 | −#else | |
| 580 | − li r13,_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_EXEC | |
| 581 | −#endif | |
| 582 | 558 | |
| 583 | 559 | 4: |
| 584 | 560 | andc. r13,r13,r11 /* Check permission */ |
| 585 | 561 | |
| 586 | −#ifdef CONFIG_PTE_64BIT | |
| 587 | 562 | #ifdef CONFIG_SMP |
| 588 | 563 | subf r13,r11,r12 /* create false data dep */ |
| 589 | 564 | lwzx r13,r11,r13 /* Get upper pte bits */ |
| 590 | 565 | #else |
| 591 | 566 | lwz r13,0(r12) /* Get upper pte bits */ |
| 592 | −#endif | |
| 593 | 567 | #endif |
| 594 | 568 | |
| 595 | 569 | bne 2f /* Bail if permission mismatch */ |
| @@ -683,7 +657,7 @@ interrupt_end: · 87 unmodified lines | ||
| 683 | 657 | * r10 - tsize encoding (if HUGETLB_PAGE) or available to use |
| 684 | 658 | * r11 - TLB (info from Linux PTE) |
| 685 | 659 | * r12 - available to use |
| 686 | − * r13 - upper bits of PTE (if PTE_64BIT) or available to use | |
| 660 | + * r13 - upper bits of PTE | |
| 687 | 661 | * CR5 - results of addr >= PAGE_OFFSET |
| 688 | 662 | * MAS0, MAS1 - loaded with proper value when we get here |
| 689 | 663 | * MAS2, MAS3 - will need additional info from Linux PTE |
| @@ -751,7 +725,6 @@ finish_tlb_load: · 61 unmodified lines | ||
| 751 | 725 | * here we (properly should) assume have the appropriate value. |
| 752 | 726 | */ |
| 753 | 727 | finish_tlb_load_cont: |
| 754 | −#ifdef CONFIG_PTE_64BIT | |
| 755 | 728 | rlwinm r12, r11, 32-2, 26, 31 /* Move in perm bits */ |
| 756 | 729 | andi. r10, r11, _PAGE_DIRTY |
| 757 | 730 | bne 1f |
| @@ -764,26 +737,9 @@ BEGIN_MMU_FTR_SECTION · 6 unmodified lines | ||
| 764 | 737 | srwi r10, r13, 12 /* grab RPN[12:31] */ |
| 765 | 738 | mtspr SPRN_MAS7, r10 |
| 766 | 739 | END_MMU_FTR_SECTION_IFSET(MMU_FTR_BIG_PHYS) |
| 767 | −#else | |
| 768 | − li r10, (_PAGE_EXEC | _PAGE_READ) | |
| 769 | − mr r13, r11 | |
| 770 | − rlwimi r10, r11, 31, 29, 29 /* extract _PAGE_DIRTY into SW */ | |
| 771 | − and r12, r11, r10 | |
| 772 | − mcrf cr0, cr5 /* Test for user page */ | |
| 773 | − slwi r10, r12, 1 | |
| 774 | − or r10, r10, r12 | |
| 775 | − rlwinm r10, r10, 0, ~_PAGE_EXEC /* Clear SX on user pages */ | |
| 776 | − isellt r12, r10, r12 | |
| 777 | − rlwimi r13, r12, 0, 20, 31 /* Get RPN from PTE, merge w/ perms */ | |
| 778 | − mtspr SPRN_MAS3, r13 | |
| 779 | −#endif | |
| 780 | 740 | |
| 781 | 741 | mfspr r12, SPRN_MAS2 |
| 782 | −#ifdef CONFIG_PTE_64BIT | |
| 783 | 742 | rlwimi r12, r11, 32-19, 27, 31 /* extract WIMGE from pte */ |
| 784 | −#else | |
| 785 | − rlwimi r12, r11, 26, 27, 31 /* extract WIMGE from pte */ | |
| 786 | −#endif | |
| 787 | 743 | #ifdef CONFIG_HUGETLB_PAGE |
| 788 | 744 | beq 6, 3f /* don't mask if page isn't huge */ |
| 789 | 745 | li r13, 1 |
| @@ -1159,7 +1159,7 @@ spapr_tce_platform_iommu_attach_dev(struct iommu_domain *platform_domain, · 1158 unmodified lines | ||
| 1159 | 1159 | struct device *dev, |
| 1160 | 1160 | struct iommu_domain *old) |
| 1161 | 1161 | { |
| 1162 | − struct iommu_domain *domain = iommu_get_domain_for_dev(dev); | |
| 1162 | + struct iommu_domain *domain = iommu_driver_get_domain_for_dev(dev); | |
| 1163 | 1163 | struct iommu_table_group *table_group; |
| 1164 | 1164 | struct iommu_group *grp; |
| 1165 | 1165 | |
| @@ -2893,7 +2893,8 @@ static void __init fixup_device_tree_pmac(void) · 2892 unmodified lines | ||
| 2893 | 2893 | for (node = 0; prom_next_node(&node); ) { |
| 2894 | 2894 | type[0] = '\0'; |
| 2895 | 2895 | prom_getprop(node, "device_type", type, sizeof(type)); |
| 2896 | − if (prom_strcmp(type, "escc") && prom_strcmp(type, "i2s")) | |
| 2896 | + if (prom_strcmp(type, "escc") && prom_strcmp(type, "i2s") && | |
| 2897 | + prom_strcmp(type, "media-bay")) | |
| 2897 | 2898 | continue; |
| 2898 | 2899 | |
| 2899 | 2900 | if (prom_getproplen(node, "#size-cells") != PROM_ERROR) |
| @@ -35,7 +35,6 @@ · 34 unmodified lines | ||
| 35 | 35 | #include <linux/of_irq.h> |
| 36 | 36 | #include <linux/hugetlb.h> |
| 37 | 37 | #include <linux/pgtable.h> |
| 38 | −#include <asm/kexec.h> | |
| 39 | 38 | #include <asm/io.h> |
| 40 | 39 | #include <asm/paca.h> |
| 41 | 40 | #include <asm/processor.h> |
| @@ -995,15 +994,6 @@ void __init setup_arch(char **cmdline_p) · 953 unmodified lines | ||
| 995 | 994 | |
| 996 | 995 | initmem_init(); |
| 997 | 996 | |
| 998 | − /* | |
| 999 | − * Reserve large chunks of memory for use by CMA for kdump, fadump, KVM and | |
| 1000 | − * hugetlb. These must be called after initmem_init(), so that | |
| 1001 | − * pageblock_order is initialised. | |
| 1002 | − */ | |
| 1003 | − fadump_cma_init(); | |
| 1004 | − kdump_cma_reserve(); | |
| 1005 | − kvm_cma_reserve(); | |
| 1006 | − | |
| 1007 | 997 | early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT); |
| 1008 | 998 | |
| 1009 | 999 | if (ppc_md.setup_arch) |
| @@ -37,11 +37,29 @@ unsigned long ftrace_call_adjust(unsigned long addr) · 36 unmodified lines | ||
| 37 | 37 | if (addr >= (unsigned long)__exittext_begin && addr < (unsigned long)__exittext_end) |
| 38 | 38 | return 0; |
| 39 | 39 | |
| 40 | − if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY) && | |
| 41 | − !IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { | |
| 42 | − addr += MCOUNT_INSN_SIZE; | |
| 43 | − if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) | |
| 40 | + if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)) { | |
| 41 | + if (!IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { | |
| 44 | 42 | addr += MCOUNT_INSN_SIZE; |
| 43 | + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) | |
| 44 | + addr += MCOUNT_INSN_SIZE; | |
| 45 | + } else if (IS_ENABLED(CONFIG_CC_IS_CLANG) && IS_ENABLED(CONFIG_PPC64)) { | |
| 46 | + /* | |
| 47 | + * addr points to global entry point though the NOP was emitted at local | |
| 48 | + * entry point due to https://github.com/llvm/llvm-project/issues/163706 | |
| 49 | + * Handle that here with ppc_function_entry() for kernel symbols while | |
| 50 | + * adjusting module addresses in the else case, by looking for the below | |
| 51 | + * module global entry point sequence: | |
| 52 | + * ld r2, -8(r12) | |
| 53 | + * add r2, r2, r12 | |
| 54 | + */ | |
| 55 | + if (is_kernel_text(addr) || is_kernel_inittext(addr)) | |
| 56 | + addr = ppc_function_entry((void *)addr); | |
| 57 | + else if ((ppc_inst_val(ppc_inst_read((u32 *)addr)) == | |
| 58 | + PPC_RAW_LD(_R2, _R12, -8)) && | |
| 59 | + (ppc_inst_val(ppc_inst_read((u32 *)(addr+4))) == | |
| 60 | + PPC_RAW_ADD(_R2, _R2, _R12))) | |
| 61 | + addr += 8; | |
| 62 | + } | |
| 45 | 63 | } |
| 46 | 64 | |
| 47 | 65 | return addr; |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <asm/firmware.h> |
| 24 | 24 | |
| 25 | 25 | #define cpu_to_be_ulong __PASTE(cpu_to_be, BITS_PER_LONG) |
| 26 | +#define __be_word __PASTE(__be, BITS_PER_LONG) | |
| 26 | 27 | |
| 27 | 28 | #ifdef CONFIG_CRASH_DUMP |
| 28 | 29 | void machine_crash_shutdown(struct pt_regs *regs) |
| @@ -146,25 +147,25 @@ int __init overlaps_crashkernel(unsigned long start, unsigned long size) · 117 unmodified lines | ||
| 146 | 147 | } |
| 147 | 148 | |
| 148 | 149 | /* Values we need to export to the second kernel via the device tree. */ |
| 149 | −static phys_addr_t crashk_base; | |
| 150 | −static phys_addr_t crashk_size; | |
| 151 | −static unsigned long long mem_limit; | |
| 150 | +static __be_word crashk_base; | |
| 151 | +static __be_word crashk_size; | |
| 152 | +static __be_word mem_limit; | |
| 152 | 153 | |
| 153 | 154 | static struct property crashk_base_prop = { |
| 154 | 155 | .name = "linux,crashkernel-base", |
| 155 | − .length = sizeof(phys_addr_t), | |
| 156 | + .length = sizeof(__be_word), | |
| 156 | 157 | .value = &crashk_base |
| 157 | 158 | }; |
| 158 | 159 | |
| 159 | 160 | static struct property crashk_size_prop = { |
| 160 | 161 | .name = "linux,crashkernel-size", |
| 161 | − .length = sizeof(phys_addr_t), | |
| 162 | + .length = sizeof(__be_word), | |
| 162 | 163 | .value = &crashk_size, |
| 163 | 164 | }; |
| 164 | 165 | |
| 165 | 166 | static struct property memory_limit_prop = { |
| 166 | 167 | .name = "linux,memory-limit", |
| 167 | − .length = sizeof(unsigned long long), | |
| 168 | + .length = sizeof(__be_word), | |
| 168 | 169 | .value = &mem_limit, |
| 169 | 170 | }; |
| 170 | 171 | |
| @@ -193,11 +194,11 @@ static void __init export_crashk_values(struct device_node *node) · 22 unmodified lines | ||
| 193 | 194 | } |
| 194 | 195 | #endif /* CONFIG_CRASH_RESERVE */ |
| 195 | 196 | |
| 196 | −static phys_addr_t kernel_end; | |
| 197 | +static __be_word kernel_end; | |
| 197 | 198 | |
| 198 | 199 | static struct property kernel_end_prop = { |
| 199 | 200 | .name = "linux,kernel-end", |
| 200 | − .length = sizeof(phys_addr_t), | |
| 201 | + .length = sizeof(__be_word), | |
| 201 | 202 | .value = &kernel_end, |
| 202 | 203 | }; |
| 203 | 204 | |
| @@ -450,6 +450,11 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) · 449 unmodified lines | ||
| 450 | 450 | kbuf->buffer = headers; |
| 451 | 451 | kbuf->mem = KEXEC_BUF_MEM_UNKNOWN; |
| 452 | 452 | kbuf->bufsz = headers_sz; |
| 453 | + | |
| 454 | + /* | |
| 455 | + * Account for extra space required to accommodate additional memory | |
| 456 | + * ranges in elfcorehdr due to memory hotplug events. | |
| 457 | + */ | |
| 453 | 458 | kbuf->memsz = headers_sz + kdump_extra_elfcorehdr_size(cmem); |
| 454 | 459 | kbuf->top_down = false; |
| 455 | 460 | |
| @@ -460,7 +465,14 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) · 4 unmodified lines | ||
| 460 | 465 | } |
| 461 | 466 | |
| 462 | 467 | image->elf_load_addr = kbuf->mem; |
| 463 | − image->elf_headers_sz = headers_sz; | |
| 468 | + | |
| 469 | + /* | |
| 470 | + * If CONFIG_CRASH_HOTPLUG is enabled, the elfcorehdr kexec segment | |
| 471 | + * memsz can be larger than bufsz. Always initialize elf_headers_sz | |
| 472 | + * with memsz. This ensures the correct size is reserved for elfcorehdr | |
| 473 | + * memory in the FDT prepared for kdump. | |
| 474 | + */ | |
| 475 | + image->elf_headers_sz = kbuf->memsz; | |
| 464 | 476 | image->elf_headers = headers; |
| 465 | 477 | out: |
| 466 | 478 | kfree(cmem); |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | ||
| 38 | 38 | |
| 39 | 39 | /* #define EXIT_DEBUG */ |
| 40 | 40 | |
| 41 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 41 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 42 | 42 | KVM_GENERIC_VM_STATS(), |
| 43 | 43 | STATS_DESC_ICOUNTER(VM, num_2M_pages), |
| 44 | 44 | STATS_DESC_ICOUNTER(VM, num_1G_pages) |
| @@ -53,7 +53,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 8 unmodified lines | ||
| 53 | 53 | sizeof(kvm_vm_stats_desc), |
| 54 | 54 | }; |
| 55 | 55 | |
| 56 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 56 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 57 | 57 | KVM_GENERIC_VCPU_STATS(), |
| 58 | 58 | STATS_DESC_COUNTER(VCPU, sum_exits), |
| 59 | 59 | STATS_DESC_COUNTER(VCPU, mmio_exits), |
| @@ -36,7 +36,7 @@ · 35 unmodified lines | ||
| 36 | 36 | |
| 37 | 37 | unsigned long kvmppc_booke_handlers; |
| 38 | 38 | |
| 39 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 39 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 40 | 40 | KVM_GENERIC_VM_STATS(), |
| 41 | 41 | STATS_DESC_ICOUNTER(VM, num_2M_pages), |
| 42 | 42 | STATS_DESC_ICOUNTER(VM, num_1G_pages) |
| @@ -51,7 +51,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 8 unmodified lines | ||
| 51 | 51 | sizeof(kvm_vm_stats_desc), |
| 52 | 52 | }; |
| 53 | 53 | |
| 54 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 54 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 55 | 55 | KVM_GENERIC_VCPU_STATS(), |
| 56 | 56 | STATS_DESC_COUNTER(VCPU, sum_exits), |
| 57 | 57 | STATS_DESC_COUNTER(VCPU, mmio_exits), |
| @@ -39,15 +39,11 @@ enum vcpu_ftr { · 38 unmodified lines | ||
| 39 | 39 | /* bits [6-5] MAS2_X1 and MAS2_X0 and [4-0] bits for WIMGE */ |
| 40 | 40 | #define E500_TLB_MAS2_ATTR (0x7f) |
| 41 | 41 | |
| 42 | −struct tlbe_ref { | |
| 42 | +struct tlbe_priv { | |
| 43 | 43 | kvm_pfn_t pfn; /* valid only for TLB0, except briefly */ |
| 44 | 44 | unsigned int flags; /* E500_TLB_* */ |
| 45 | 45 | }; |
| 46 | 46 | |
| 47 | −struct tlbe_priv { | |
| 48 | − struct tlbe_ref ref; | |
| 49 | −}; | |
| 50 | − | |
| 51 | 47 | #ifdef CONFIG_KVM_E500V2 |
| 52 | 48 | struct vcpu_id_table; |
| 53 | 49 | #endif |
| @@ -920,12 +920,12 @@ int kvmppc_e500_tlb_init(struct kvmppc_vcpu_e500 *vcpu_e500) · 919 unmodified lines | ||
| 920 | 920 | vcpu_e500->gtlb_offset[0] = 0; |
| 921 | 921 | vcpu_e500->gtlb_offset[1] = KVM_E500_TLB0_SIZE; |
| 922 | 922 | |
| 923 | − vcpu_e500->gtlb_priv[0] = kzalloc_objs(struct tlbe_ref, | |
| 923 | + vcpu_e500->gtlb_priv[0] = kzalloc_objs(struct tlbe_priv, | |
| 924 | 924 | vcpu_e500->gtlb_params[0].entries); |
| 925 | 925 | if (!vcpu_e500->gtlb_priv[0]) |
| 926 | 926 | goto free_vcpu; |
| 927 | 927 | |
| 928 | − vcpu_e500->gtlb_priv[1] = kzalloc_objs(struct tlbe_ref, | |
| 928 | + vcpu_e500->gtlb_priv[1] = kzalloc_objs(struct tlbe_priv, | |
| 929 | 929 | vcpu_e500->gtlb_params[1].entries); |
| 930 | 930 | if (!vcpu_e500->gtlb_priv[1]) |
| 931 | 931 | goto free_vcpu; |
| @@ -189,16 +189,16 @@ void inval_gtlbe_on_host(struct kvmppc_vcpu_e500 *vcpu_e500, int tlbsel, · 188 unmodified lines | ||
| 189 | 189 | { |
| 190 | 190 | struct kvm_book3e_206_tlb_entry *gtlbe = |
| 191 | 191 | get_entry(vcpu_e500, tlbsel, esel); |
| 192 | − struct tlbe_ref *ref = &vcpu_e500->gtlb_priv[tlbsel][esel].ref; | |
| 192 | + struct tlbe_priv *tlbe = &vcpu_e500->gtlb_priv[tlbsel][esel]; | |
| 193 | 193 | |
| 194 | 194 | /* Don't bother with unmapped entries */ |
| 195 | − if (!(ref->flags & E500_TLB_VALID)) { | |
| 196 | − WARN(ref->flags & (E500_TLB_BITMAP | E500_TLB_TLB0), | |
| 197 | − "%s: flags %x\n", __func__, ref->flags); | |
| 195 | + if (!(tlbe->flags & E500_TLB_VALID)) { | |
| 196 | + WARN(tlbe->flags & (E500_TLB_BITMAP | E500_TLB_TLB0), | |
| 197 | + "%s: flags %x\n", __func__, tlbe->flags); | |
| 198 | 198 | WARN_ON(tlbsel == 1 && vcpu_e500->g2h_tlb1_map[esel]); |
| 199 | 199 | } |
| 200 | 200 | |
| 201 | − if (tlbsel == 1 && ref->flags & E500_TLB_BITMAP) { | |
| 201 | + if (tlbsel == 1 && tlbe->flags & E500_TLB_BITMAP) { | |
| 202 | 202 | u64 tmp = vcpu_e500->g2h_tlb1_map[esel]; |
| 203 | 203 | int hw_tlb_indx; |
| 204 | 204 | unsigned long flags; |
| @@ -216,28 +216,28 @@ void inval_gtlbe_on_host(struct kvmppc_vcpu_e500 *vcpu_e500, int tlbsel, · 11 unmodified lines | ||
| 216 | 216 | } |
| 217 | 217 | mb(); |
| 218 | 218 | vcpu_e500->g2h_tlb1_map[esel] = 0; |
| 219 | − ref->flags &= ~(E500_TLB_BITMAP | E500_TLB_VALID); | |
| 219 | + tlbe->flags &= ~(E500_TLB_BITMAP | E500_TLB_VALID); | |
| 220 | 220 | local_irq_restore(flags); |
| 221 | 221 | } |
| 222 | 222 | |
| 223 | − if (tlbsel == 1 && ref->flags & E500_TLB_TLB0) { | |
| 223 | + if (tlbsel == 1 && tlbe->flags & E500_TLB_TLB0) { | |
| 224 | 224 | /* |
| 225 | 225 | * TLB1 entry is backed by 4k pages. This should happen |
| 226 | 226 | * rarely and is not worth optimizing. Invalidate everything. |
| 227 | 227 | */ |
| 228 | 228 | kvmppc_e500_tlbil_all(vcpu_e500); |
| 229 | − ref->flags &= ~(E500_TLB_TLB0 | E500_TLB_VALID); | |
| 229 | + tlbe->flags &= ~(E500_TLB_TLB0 | E500_TLB_VALID); | |
| 230 | 230 | } |
| 231 | 231 | |
| 232 | 232 | /* |
| 233 | 233 | * If TLB entry is still valid then it's a TLB0 entry, and thus |
| 234 | 234 | * backed by at most one host tlbe per shadow pid |
| 235 | 235 | */ |
| 236 | − if (ref->flags & E500_TLB_VALID) | |
| 236 | + if (tlbe->flags & E500_TLB_VALID) | |
| 237 | 237 | kvmppc_e500_tlbil_one(vcpu_e500, gtlbe); |
| 238 | 238 | |
| 239 | 239 | /* Mark the TLB as not backed by the host anymore */ |
| 240 | − ref->flags = 0; | |
| 240 | + tlbe->flags = 0; | |
| 241 | 241 | } |
| 242 | 242 | |
| 243 | 243 | static inline int tlbe_is_writable(struct kvm_book3e_206_tlb_entry *tlbe) |
| @@ -245,26 +245,26 @@ static inline int tlbe_is_writable(struct kvm_book3e_206_tlb_entry *tlbe) · 1 unmodified line | ||
| 245 | 245 | return tlbe->mas7_3 & (MAS3_SW|MAS3_UW); |
| 246 | 246 | } |
| 247 | 247 | |
| 248 | −static inline void kvmppc_e500_ref_setup(struct tlbe_ref *ref, | |
| 249 | − struct kvm_book3e_206_tlb_entry *gtlbe, | |
| 250 | − kvm_pfn_t pfn, unsigned int wimg, | |
| 251 | − bool writable) | |
| 248 | +static inline void kvmppc_e500_tlbe_setup(struct tlbe_priv *tlbe, | |
| 249 | + struct kvm_book3e_206_tlb_entry *gtlbe, | |
| 250 | + kvm_pfn_t pfn, unsigned int wimg, | |
| 251 | + bool writable) | |
| 252 | 252 | { |
| 253 | − ref->pfn = pfn; | |
| 254 | − ref->flags = E500_TLB_VALID; | |
| 253 | + tlbe->pfn = pfn; | |
| 254 | + tlbe->flags = E500_TLB_VALID; | |
| 255 | 255 | if (writable) |
| 256 | − ref->flags |= E500_TLB_WRITABLE; | |
| 256 | + tlbe->flags |= E500_TLB_WRITABLE; | |
| 257 | 257 | |
| 258 | 258 | /* Use guest supplied MAS2_G and MAS2_E */ |
| 259 | − ref->flags |= (gtlbe->mas2 & MAS2_ATTRIB_MASK) | wimg; | |
| 259 | + tlbe->flags |= (gtlbe->mas2 & MAS2_ATTRIB_MASK) | wimg; | |
| 260 | 260 | } |
| 261 | 261 | |
| 262 | −static inline void kvmppc_e500_ref_release(struct tlbe_ref *ref) | |
| 262 | +static inline void kvmppc_e500_tlbe_release(struct tlbe_priv *tlbe) | |
| 263 | 263 | { |
| 264 | − if (ref->flags & E500_TLB_VALID) { | |
| 264 | + if (tlbe->flags & E500_TLB_VALID) { | |
| 265 | 265 | /* FIXME: don't log bogus pfn for TLB1 */ |
| 266 | − trace_kvm_booke206_ref_release(ref->pfn, ref->flags); | |
| 267 | − ref->flags = 0; | |
| 266 | + trace_kvm_booke206_ref_release(tlbe->pfn, tlbe->flags); | |
| 267 | + tlbe->flags = 0; | |
| 268 | 268 | } |
| 269 | 269 | } |
| 270 | 270 | |
| @@ -284,11 +284,8 @@ static void clear_tlb_privs(struct kvmppc_vcpu_e500 *vcpu_e500) · 13 unmodified lines | ||
| 284 | 284 | int i; |
| 285 | 285 | |
| 286 | 286 | for (tlbsel = 0; tlbsel <= 1; tlbsel++) { |
| 287 | − for (i = 0; i < vcpu_e500->gtlb_params[tlbsel].entries; i++) { | |
| 288 | − struct tlbe_ref *ref = | |
| 289 | − &vcpu_e500->gtlb_priv[tlbsel][i].ref; | |
| 290 | − kvmppc_e500_ref_release(ref); | |
| 291 | − } | |
| 287 | + for (i = 0; i < vcpu_e500->gtlb_params[tlbsel].entries; i++) | |
| 288 | + kvmppc_e500_tlbe_release(&vcpu_e500->gtlb_priv[tlbsel][i]); | |
| 292 | 289 | } |
| 293 | 290 | } |
| 294 | 291 | |
| @@ -304,18 +301,18 @@ void kvmppc_core_flush_tlb(struct kvm_vcpu *vcpu) · 9 unmodified lines | ||
| 304 | 301 | static void kvmppc_e500_setup_stlbe( |
| 305 | 302 | struct kvm_vcpu *vcpu, |
| 306 | 303 | struct kvm_book3e_206_tlb_entry *gtlbe, |
| 307 | − int tsize, struct tlbe_ref *ref, u64 gvaddr, | |
| 304 | + int tsize, struct tlbe_priv *tlbe, u64 gvaddr, | |
| 308 | 305 | struct kvm_book3e_206_tlb_entry *stlbe) |
| 309 | 306 | { |
| 310 | − kvm_pfn_t pfn = ref->pfn; | |
| 307 | + kvm_pfn_t pfn = tlbe->pfn; | |
| 311 | 308 | u32 pr = vcpu->arch.shared->msr & MSR_PR; |
| 312 | − bool writable = !!(ref->flags & E500_TLB_WRITABLE); | |
| 309 | + bool writable = !!(tlbe->flags & E500_TLB_WRITABLE); | |
| 313 | 310 | |
| 314 | − BUG_ON(!(ref->flags & E500_TLB_VALID)); | |
| 311 | + BUG_ON(!(tlbe->flags & E500_TLB_VALID)); | |
| 315 | 312 | |
| 316 | 313 | /* Force IPROT=0 for all guest mappings. */ |
| 317 | 314 | stlbe->mas1 = MAS1_TSIZE(tsize) | get_tlb_sts(gtlbe) | MAS1_VALID; |
| 318 | − stlbe->mas2 = (gvaddr & MAS2_EPN) | (ref->flags & E500_TLB_MAS2_ATTR); | |
| 315 | + stlbe->mas2 = (gvaddr & MAS2_EPN) | (tlbe->flags & E500_TLB_MAS2_ATTR); | |
| 319 | 316 | stlbe->mas7_3 = ((u64)pfn << PAGE_SHIFT) | |
| 320 | 317 | e500_shadow_mas3_attrib(gtlbe->mas7_3, writable, pr); |
| 321 | 318 | } |
| @@ -323,7 +320,7 @@ static void kvmppc_e500_setup_stlbe( · 1 unmodified line | ||
| 323 | 320 | static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, |
| 324 | 321 | u64 gvaddr, gfn_t gfn, struct kvm_book3e_206_tlb_entry *gtlbe, |
| 325 | 322 | int tlbsel, struct kvm_book3e_206_tlb_entry *stlbe, |
| 326 | − struct tlbe_ref *ref) | |
| 323 | + struct tlbe_priv *tlbe) | |
| 327 | 324 | { |
| 328 | 325 | struct kvm_memory_slot *slot; |
| 329 | 326 | unsigned int psize; |
| @@ -455,9 +452,9 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, · 125 unmodified lines | ||
| 455 | 452 | } |
| 456 | 453 | } |
| 457 | 454 | |
| 458 | − kvmppc_e500_ref_setup(ref, gtlbe, pfn, wimg, writable); | |
| 455 | + kvmppc_e500_tlbe_setup(tlbe, gtlbe, pfn, wimg, writable); | |
| 459 | 456 | kvmppc_e500_setup_stlbe(&vcpu_e500->vcpu, gtlbe, tsize, |
| 460 | − ref, gvaddr, stlbe); | |
| 457 | + tlbe, gvaddr, stlbe); | |
| 461 | 458 | writable = tlbe_is_writable(stlbe); |
| 462 | 459 | |
| 463 | 460 | /* Clear i-cache for new pages */ |
| @@ -474,17 +471,17 @@ static int kvmppc_e500_tlb0_map(struct kvmppc_vcpu_e500 *vcpu_e500, int esel, · 10 unmodified lines | ||
| 474 | 471 | struct kvm_book3e_206_tlb_entry *stlbe) |
| 475 | 472 | { |
| 476 | 473 | struct kvm_book3e_206_tlb_entry *gtlbe; |
| 477 | − struct tlbe_ref *ref; | |
| 474 | + struct tlbe_priv *tlbe; | |
| 478 | 475 | int stlbsel = 0; |
| 479 | 476 | int sesel = 0; |
| 480 | 477 | int r; |
| 481 | 478 | |
| 482 | 479 | gtlbe = get_entry(vcpu_e500, 0, esel); |
| 483 | − ref = &vcpu_e500->gtlb_priv[0][esel].ref; | |
| 480 | + tlbe = &vcpu_e500->gtlb_priv[0][esel]; | |
| 484 | 481 | |
| 485 | 482 | r = kvmppc_e500_shadow_map(vcpu_e500, get_tlb_eaddr(gtlbe), |
| 486 | 483 | get_tlb_raddr(gtlbe) >> PAGE_SHIFT, |
| 487 | − gtlbe, 0, stlbe, ref); | |
| 484 | + gtlbe, 0, stlbe, tlbe); | |
| 488 | 485 | if (r) |
| 489 | 486 | return r; |
| 490 | 487 | |
| @@ -494,7 +491,7 @@ static int kvmppc_e500_tlb0_map(struct kvmppc_vcpu_e500 *vcpu_e500, int esel, · 3 unmodified lines | ||
| 494 | 491 | } |
| 495 | 492 | |
| 496 | 493 | static int kvmppc_e500_tlb1_map_tlb1(struct kvmppc_vcpu_e500 *vcpu_e500, |
| 497 | − struct tlbe_ref *ref, | |
| 494 | + struct tlbe_priv *tlbe, | |
| 498 | 495 | int esel) |
| 499 | 496 | { |
| 500 | 497 | unsigned int sesel = vcpu_e500->host_tlb1_nv++; |
| @@ -507,10 +504,10 @@ static int kvmppc_e500_tlb1_map_tlb1(struct kvmppc_vcpu_e500 *vcpu_e500, · 6 unmodified lines | ||
| 507 | 504 | vcpu_e500->g2h_tlb1_map[idx] &= ~(1ULL << sesel); |
| 508 | 505 | } |
| 509 | 506 | |
| 510 | − vcpu_e500->gtlb_priv[1][esel].ref.flags |= E500_TLB_BITMAP; | |
| 507 | + vcpu_e500->gtlb_priv[1][esel].flags |= E500_TLB_BITMAP; | |
| 511 | 508 | vcpu_e500->g2h_tlb1_map[esel] |= (u64)1 << sesel; |
| 512 | 509 | vcpu_e500->h2g_tlb1_rmap[sesel] = esel + 1; |
| 513 | − WARN_ON(!(ref->flags & E500_TLB_VALID)); | |
| 510 | + WARN_ON(!(tlbe->flags & E500_TLB_VALID)); | |
| 514 | 511 | |
| 515 | 512 | return sesel; |
| 516 | 513 | } |
| @@ -522,24 +519,24 @@ static int kvmppc_e500_tlb1_map(struct kvmppc_vcpu_e500 *vcpu_e500, · 5 unmodified lines | ||
| 522 | 519 | u64 gvaddr, gfn_t gfn, struct kvm_book3e_206_tlb_entry *gtlbe, |
| 523 | 520 | struct kvm_book3e_206_tlb_entry *stlbe, int esel) |
| 524 | 521 | { |
| 525 | − struct tlbe_ref *ref = &vcpu_e500->gtlb_priv[1][esel].ref; | |
| 522 | + struct tlbe_priv *tlbe = &vcpu_e500->gtlb_priv[1][esel]; | |
| 526 | 523 | int sesel; |
| 527 | 524 | int r; |
| 528 | 525 | |
| 529 | 526 | r = kvmppc_e500_shadow_map(vcpu_e500, gvaddr, gfn, gtlbe, 1, stlbe, |
| 530 | − ref); | |
| 527 | + tlbe); | |
| 531 | 528 | if (r) |
| 532 | 529 | return r; |
| 533 | 530 | |
| 534 | 531 | /* Use TLB0 when we can only map a page with 4k */ |
| 535 | 532 | if (get_tlb_tsize(stlbe) == BOOK3E_PAGESZ_4K) { |
| 536 | − vcpu_e500->gtlb_priv[1][esel].ref.flags |= E500_TLB_TLB0; | |
| 533 | + vcpu_e500->gtlb_priv[1][esel].flags |= E500_TLB_TLB0; | |
| 537 | 534 | write_stlbe(vcpu_e500, gtlbe, stlbe, 0, 0); |
| 538 | 535 | return 0; |
| 539 | 536 | } |
| 540 | 537 | |
| 541 | 538 | /* Otherwise map into TLB1 */ |
| 542 | − sesel = kvmppc_e500_tlb1_map_tlb1(vcpu_e500, ref, esel); | |
| 539 | + sesel = kvmppc_e500_tlb1_map_tlb1(vcpu_e500, tlbe, esel); | |
| 543 | 540 | write_stlbe(vcpu_e500, gtlbe, stlbe, 1, sesel); |
| 544 | 541 | |
| 545 | 542 | return 0; |
| @@ -561,11 +558,11 @@ void kvmppc_mmu_map(struct kvm_vcpu *vcpu, u64 eaddr, gpa_t gpaddr, · 15 unmodified lines | ||
| 561 | 558 | priv = &vcpu_e500->gtlb_priv[tlbsel][esel]; |
| 562 | 559 | |
| 563 | 560 | /* Triggers after clear_tlb_privs or on initial mapping */ |
| 564 | − if (!(priv->ref.flags & E500_TLB_VALID)) { | |
| 561 | + if (!(priv->flags & E500_TLB_VALID)) { | |
| 565 | 562 | kvmppc_e500_tlb0_map(vcpu_e500, esel, &stlbe); |
| 566 | 563 | } else { |
| 567 | 564 | kvmppc_e500_setup_stlbe(vcpu, gtlbe, BOOK3E_PAGESZ_4K, |
| 568 | − &priv->ref, eaddr, &stlbe); | |
| 565 | + priv, eaddr, &stlbe); | |
| 569 | 566 | write_stlbe(vcpu_e500, gtlbe, &stlbe, 0, 0); |
| 570 | 567 | } |
| 571 | 568 | break; |
| @@ -562,3 +562,4 @@ exc; std r10,32(3) · 561 unmodified lines | ||
| 562 | 562 | li r5,4096 |
| 563 | 563 | b .Ldst_aligned |
| 564 | 564 | EXPORT_SYMBOL(__copy_tofrom_user) |
| 565 | +EXPORT_SYMBOL(__copy_tofrom_user_base) | |
| @@ -5,13 +5,9 @@ · 4 unmodified lines | ||
| 5 | 5 | * |
| 6 | 6 | * Author: Anton Blanchard <anton@au.ibm.com> |
| 7 | 7 | */ |
| 8 | +#include <linux/export.h> | |
| 8 | 9 | #include <asm/ppc_asm.h> |
| 9 | 10 | |
| 10 | −#ifndef SELFTEST_CASE | |
| 11 | −/* 0 == don't use VMX, 1 == use VMX */ | |
| 12 | −#define SELFTEST_CASE 0 | |
| 13 | −#endif | |
| 14 | − | |
| 15 | 11 | #ifdef __BIG_ENDIAN__ |
| 16 | 12 | #define LVS(VRT,RA,RB) lvsl VRT,RA,RB |
| 17 | 13 | #define VPERM(VRT,VRA,VRB,VRC) vperm VRT,VRA,VRB,VRC |
| @@ -47,10 +43,14 @@ · 29 unmodified lines | ||
| 47 | 43 | ld r15,STK_REG(R15)(r1) |
| 48 | 44 | ld r14,STK_REG(R14)(r1) |
| 49 | 45 | .Ldo_err3: |
| 50 | − bl CFUNC(exit_vmx_usercopy) | |
| 46 | + ld r6,STK_REG(R31)(r1) /* original destination pointer */ | |
| 47 | + ld r5,STK_REG(R29)(r1) /* original number of bytes */ | |
| 48 | + subf r7,r6,r3 /* #bytes copied */ | |
| 49 | + subf r3,r7,r5 /* #bytes not copied in r3 */ | |
| 51 | 50 | ld r0,STACKFRAMESIZE+16(r1) |
| 52 | 51 | mtlr r0 |
| 53 | − b .Lexit | |
| 52 | + addi r1,r1,STACKFRAMESIZE | |
| 53 | + blr | |
| 54 | 54 | #endif /* CONFIG_ALTIVEC */ |
| 55 | 55 | |
| 56 | 56 | .Ldo_err2: |
| @@ -74,7 +74,6 @@ · 17 unmodified lines | ||
| 74 | 74 | |
| 75 | 75 | _GLOBAL(__copy_tofrom_user_power7) |
| 76 | 76 | cmpldi r5,16 |
| 77 | − cmpldi cr1,r5,3328 | |
| 78 | 77 | |
| 79 | 78 | std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) |
| 80 | 79 | std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) |
| @@ -82,12 +81,6 @@ _GLOBAL(__copy_tofrom_user_power7) · 1 unmodified line | ||
| 82 | 81 | |
| 83 | 82 | blt .Lshort_copy |
| 84 | 83 | |
| 85 | −#ifdef CONFIG_ALTIVEC | |
| 86 | −test_feature = SELFTEST_CASE | |
| 87 | −BEGIN_FTR_SECTION | |
| 88 | − bgt cr1,.Lvmx_copy | |
| 89 | −END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) | |
| 90 | −#endif | |
| 91 | 84 | |
| 92 | 85 | .Lnonvmx_copy: |
| 93 | 86 | /* Get the source 8B aligned */ |
| @@ -263,23 +256,14 @@ err1; stb r0,0(r3) · 169 unmodified lines | ||
| 263 | 256 | 15: li r3,0 |
| 264 | 257 | blr |
| 265 | 258 | |
| 266 | −.Lunwind_stack_nonvmx_copy: | |
| 267 | − addi r1,r1,STACKFRAMESIZE | |
| 268 | − b .Lnonvmx_copy | |
| 269 | − | |
| 270 | −.Lvmx_copy: | |
| 271 | 259 | #ifdef CONFIG_ALTIVEC |
| 260 | +_GLOBAL(__copy_tofrom_user_power7_vmx) | |
| 272 | 261 | mflr r0 |
| 273 | 262 | std r0,16(r1) |
| 274 | 263 | stdu r1,-STACKFRAMESIZE(r1) |
| 275 | − bl CFUNC(enter_vmx_usercopy) | |
| 276 | − cmpwi cr1,r3,0 | |
| 277 | − ld r0,STACKFRAMESIZE+16(r1) | |
| 278 | − ld r3,STK_REG(R31)(r1) | |
| 279 | − ld r4,STK_REG(R30)(r1) | |
| 280 | − ld r5,STK_REG(R29)(r1) | |
| 281 | − mtlr r0 | |
| 282 | 264 | |
| 265 | + std r3,STK_REG(R31)(r1) | |
| 266 | + std r5,STK_REG(R29)(r1) | |
| 283 | 267 | /* |
| 284 | 268 | * We prefetch both the source and destination using enhanced touch |
| 285 | 269 | * instructions. We use a stream ID of 0 for the load side and |
| @@ -300,8 +284,6 @@ err1; stb r0,0(r3) · 14 unmodified lines | ||
| 300 | 284 | |
| 301 | 285 | DCBT_SETUP_STREAMS(r6, r7, r9, r10, r8) |
| 302 | 286 | |
| 303 | − beq cr1,.Lunwind_stack_nonvmx_copy | |
| 304 | − | |
| 305 | 287 | /* |
| 306 | 288 | * If source and destination are not relatively aligned we use a |
| 307 | 289 | * slower permute loop. |
| @@ -478,7 +460,8 @@ err3; lbz r0,0(r4) · 170 unmodified lines | ||
| 478 | 460 | err3; stb r0,0(r3) |
| 479 | 461 | |
| 480 | 462 | 15: addi r1,r1,STACKFRAMESIZE |
| 481 | − b CFUNC(exit_vmx_usercopy) /* tail call optimise */ | |
| 463 | + li r3,0 | |
| 464 | + blr | |
| 482 | 465 | |
| 483 | 466 | .Lvmx_unaligned_copy: |
| 484 | 467 | /* Get the destination 16B aligned */ |
| @@ -681,5 +664,7 @@ err3; lbz r0,0(r4) · 196 unmodified lines | ||
| 681 | 664 | err3; stb r0,0(r3) |
| 682 | 665 | |
| 683 | 666 | 15: addi r1,r1,STACKFRAMESIZE |
| 684 | − b CFUNC(exit_vmx_usercopy) /* tail call optimise */ | |
| 667 | + li r3,0 | |
| 668 | + blr | |
| 669 | +EXPORT_SYMBOL(__copy_tofrom_user_power7_vmx) | |
| 685 | 670 | #endif /* CONFIG_ALTIVEC */ |
| @@ -27,6 +27,7 @@ int enter_vmx_usercopy(void) · 26 unmodified lines | ||
| 27 | 27 | |
| 28 | 28 | return 1; |
| 29 | 29 | } |
| 30 | +EXPORT_SYMBOL(enter_vmx_usercopy); | |
| 30 | 31 | |
| 31 | 32 | /* |
| 32 | 33 | * This function must return 0 because we tail call optimise when calling |
| @@ -49,6 +50,7 @@ int exit_vmx_usercopy(void) · 16 unmodified lines | ||
| 49 | 50 | set_dec(1); |
| 50 | 51 | return 0; |
| 51 | 52 | } |
| 53 | +EXPORT_SYMBOL(exit_vmx_usercopy); | |
| 52 | 54 | |
| 53 | 55 | int enter_vmx_ops(void) |
| 54 | 56 | { |
| @@ -30,6 +30,10 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <asm/setup.h> |
| 31 | 31 | #include <asm/fixmap.h> |
| 32 | 32 | |
| 33 | +#include <asm/fadump.h> | |
| 34 | +#include <asm/kexec.h> | |
| 35 | +#include <asm/kvm_ppc.h> | |
| 36 | + | |
| 33 | 37 | #include <mm/mmu_decl.h> |
| 34 | 38 | |
| 35 | 39 | unsigned long long memory_limit __initdata; |
| @@ -268,6 +272,16 @@ void __init paging_init(void) · 232 unmodified lines | ||
| 268 | 272 | |
| 269 | 273 | void __init arch_mm_preinit(void) |
| 270 | 274 | { |
| 275 | + | |
| 276 | + /* | |
| 277 | + * Reserve large chunks of memory for use by CMA for kdump, fadump, KVM | |
| 278 | + * and hugetlb. These must be called after pageblock_order is | |
| 279 | + * initialised. | |
| 280 | + */ | |
| 281 | + fadump_cma_init(); | |
| 282 | + kdump_cma_reserve(); | |
| 283 | + kvm_cma_reserve(); | |
| 284 | + | |
| 271 | 285 | /* |
| 272 | 286 | * book3s is limited to 16 page sizes due to encoding this in |
| 273 | 287 | * a 4-bit field for slices. |
| @@ -81,9 +81,6 @@ · 80 unmodified lines | ||
| 81 | 81 | |
| 82 | 82 | #ifdef CONFIG_PPC64 |
| 83 | 83 | |
| 84 | −/* for gpr non volatile registers BPG_REG_6 to 10 */ | |
| 85 | −#define BPF_PPC_STACK_SAVE (6 * 8) | |
| 86 | − | |
| 87 | 84 | /* If dummy pass (!image), account for maximum possible instructions */ |
| 88 | 85 | #define PPC_LI64(d, i) do { \ |
| 89 | 86 | if (!image) \ |
| @@ -219,8 +216,6 @@ int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, · 129 unmodified lines | ||
| 219 | 216 | int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass, |
| 220 | 217 | struct codegen_context *ctx, int insn_idx, |
| 221 | 218 | int jmp_off, int dst_reg, u32 code); |
| 222 | − | |
| 223 | −int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx); | |
| 224 | 219 | #endif |
| 225 | 220 | |
| 226 | 221 | #endif |
| @@ -450,7 +450,7 @@ bool bpf_jit_supports_subprog_tailcalls(void) · 449 unmodified lines | ||
| 450 | 450 | |
| 451 | 451 | bool bpf_jit_supports_kfunc_call(void) |
| 452 | 452 | { |
| 453 | − return true; | |
| 453 | + return IS_ENABLED(CONFIG_PPC64); | |
| 454 | 454 | } |
| 455 | 455 | |
| 456 | 456 | bool bpf_jit_supports_arena(void) |
| @@ -638,19 +638,12 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · 181 unmodified lines | ||
| 638 | 638 | * for the traced function (BPF subprog/callee) to fetch it. |
| 639 | 639 | */ |
| 640 | 640 | static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx, |
| 641 | − int func_frame_offset, | |
| 642 | − int bpf_dummy_frame_size, int r4_off) | |
| 641 | + int bpf_frame_size, int r4_off) | |
| 643 | 642 | { |
| 644 | 643 | if (IS_ENABLED(CONFIG_PPC64)) { |
| 645 | − /* See Generated stack layout */ | |
| 646 | − int tailcallinfo_offset = BPF_PPC_TAILCALL; | |
| 647 | − | |
| 648 | − /* | |
| 649 | − * func_frame_offset = ...(1) | |
| 650 | − * bpf_dummy_frame_size + trampoline_frame_size | |
| 651 | − */ | |
| 652 | − EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset)); | |
| 653 | − EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset)); | |
| 644 | + EMIT(PPC_RAW_LD(_R4, _R1, bpf_frame_size)); | |
| 645 | + /* Refer to trampoline's Generated stack layout */ | |
| 646 | + EMIT(PPC_RAW_LD(_R3, _R4, -BPF_PPC_TAILCALL)); | |
| 654 | 647 | |
| 655 | 648 | /* |
| 656 | 649 | * Setting the tail_call_info in trampoline's frame |
| @@ -658,22 +651,14 @@ static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_conte · 1 unmodified line | ||
| 658 | 651 | */ |
| 659 | 652 | EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT)); |
| 660 | 653 | PPC_BCC_CONST_SHORT(COND_GT, 8); |
| 661 | − EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx))); | |
| 654 | + EMIT(PPC_RAW_ADDI(_R3, _R4, -BPF_PPC_TAILCALL)); | |
| 655 | + | |
| 662 | 656 | /* |
| 663 | − * From ...(1) above: | |
| 664 | − * trampoline_frame_bottom = ...(2) | |
| 665 | − * func_frame_offset - bpf_dummy_frame_size | |
| 666 | − * | |
| 667 | − * Using ...(2) derived above: | |
| 668 | − * trampoline_tail_call_info_offset = ...(3) | |
| 669 | − * trampoline_frame_bottom - tailcallinfo_offset | |
| 670 | − * | |
| 671 | − * From ...(3): | |
| 672 | − * Use trampoline_tail_call_info_offset to write reference of main's | |
| 673 | − * tail_call_info in trampoline frame. | |
| 657 | + * Trampoline's tail_call_info is at the same offset, as that of | |
| 658 | + * any bpf program, with reference to previous frame. Update the | |
| 659 | + * address of main's tail_call_info in trampoline frame. | |
| 674 | 660 | */ |
| 675 | − EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size) | |
| 676 | − - tailcallinfo_offset)); | |
| 661 | + EMIT(PPC_RAW_STL(_R3, _R1, bpf_frame_size - BPF_PPC_TAILCALL)); | |
| 677 | 662 | } else { |
| 678 | 663 | /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ |
| 679 | 664 | EMIT(PPC_RAW_LL(_R4, _R1, r4_off)); |
| @@ -681,7 +666,7 @@ static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_conte · 1 unmodified line | ||
| 681 | 666 | } |
| 682 | 667 | |
| 683 | 668 | static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx, |
| 684 | − int func_frame_offset, int r4_off) | |
| 669 | + int bpf_frame_size, int r4_off) | |
| 685 | 670 | { |
| 686 | 671 | if (IS_ENABLED(CONFIG_PPC32)) { |
| 687 | 672 | /* |
| @@ -692,12 +677,12 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont · 4 unmodified lines | ||
| 692 | 677 | } |
| 693 | 678 | } |
| 694 | 679 | |
| 695 | −static void bpf_trampoline_save_args(u32 *image, struct codegen_context *ctx, int func_frame_offset, | |
| 696 | − int nr_regs, int regs_off) | |
| 680 | +static void bpf_trampoline_save_args(u32 *image, struct codegen_context *ctx, | |
| 681 | + int bpf_frame_size, int nr_regs, int regs_off) | |
| 697 | 682 | { |
| 698 | 683 | int param_save_area_offset; |
| 699 | 684 | |
| 700 | − param_save_area_offset = func_frame_offset; /* the two frames we alloted */ | |
| 685 | + param_save_area_offset = bpf_frame_size; | |
| 701 | 686 | param_save_area_offset += STACK_FRAME_MIN_SIZE; /* param save area is past frame header */ |
| 702 | 687 | |
| 703 | 688 | for (int i = 0; i < nr_regs; i++) { |
| @@ -720,11 +705,11 @@ static void bpf_trampoline_restore_args_regs(u32 *image, struct codegen_context · 16 unmodified lines | ||
| 720 | 705 | |
| 721 | 706 | /* Used when we call into the traced function. Replicate parameter save area */ |
| 722 | 707 | static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context *ctx, |
| 723 | − int func_frame_offset, int nr_regs, int regs_off) | |
| 708 | + int bpf_frame_size, int nr_regs, int regs_off) | |
| 724 | 709 | { |
| 725 | 710 | int param_save_area_offset; |
| 726 | 711 | |
| 727 | − param_save_area_offset = func_frame_offset; /* the two frames we alloted */ | |
| 712 | + param_save_area_offset = bpf_frame_size; | |
| 728 | 713 | param_save_area_offset += STACK_FRAME_MIN_SIZE; /* param save area is past frame header */ |
| 729 | 714 | |
| 730 | 715 | for (int i = 8; i < nr_regs; i++) { |
| @@ -741,10 +726,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 10 unmodified lines | ||
| 741 | 726 | void *func_addr) |
| 742 | 727 | { |
| 743 | 728 | int regs_off, nregs_off, ip_off, run_ctx_off, retval_off, nvr_off, alt_lr_off, r4_off = 0; |
| 744 | − int i, ret, nr_regs, bpf_frame_size = 0, bpf_dummy_frame_size = 0, func_frame_offset; | |
| 745 | 729 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; |
| 746 | 730 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; |
| 747 | 731 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; |
| 732 | + int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0; | |
| 748 | 733 | struct codegen_context codegen_ctx, *ctx; |
| 749 | 734 | u32 *image = (u32 *)rw_image; |
| 750 | 735 | ppc_inst_t branch_insn; |
| @@ -770,24 +755,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 19 unmodified lines | ||
| 770 | 755 | * Generated stack layout: |
| 771 | 756 | * |
| 772 | 757 | * func prev back chain [ back chain ] |
| 773 | − * [ ] | |
| 774 | − * bpf prog redzone/tailcallcnt [ ... ] 64 bytes (64-bit powerpc) | |
| 775 | − * [ ] -- | |
| 776 | − * LR save area [ r0 save (64-bit) ] | header | |
| 777 | − * [ r0 save (32-bit) ] | | |
| 778 | − * dummy frame for unwind [ back chain 1 ] -- | |
| 779 | 758 | * [ tail_call_info ] optional - 64-bit powerpc |
| 780 | 759 | * [ padding ] align stack frame |
| 781 | 760 | * r4_off [ r4 (tailcallcnt) ] optional - 32-bit powerpc |
| 782 | 761 | * alt_lr_off [ real lr (ool stub)] optional - actual lr |
| 762 | + * retaddr_off [ return address ] | |
| 783 | 763 | * [ r26 ] |
| 784 | 764 | * nvr_off [ r25 ] nvr save area |
| 785 | 765 | * retval_off [ return value ] |
| 786 | 766 | * [ reg argN ] |
| 787 | 767 | * [ ... ] |
| 788 | − * regs_off [ reg_arg1 ] prog ctx context | |
| 789 | − * nregs_off [ args count ] | |
| 790 | − * ip_off [ traced function ] | |
| 768 | + * regs_off [ reg_arg1 ] prog_ctx | |
| 769 | + * nregs_off [ args count ] ((u64 *)prog_ctx)[-1] | |
| 770 | + * ip_off [ traced function ] ((u64 *)prog_ctx)[-2] | |
| 791 | 771 | * [ ... ] |
| 792 | 772 | * run_ctx_off [ bpf_tramp_run_ctx ] |
| 793 | 773 | * [ reg argN ] |
| @@ -843,6 +823,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 49 unmodified lines | ||
| 843 | 823 | nvr_off = bpf_frame_size; |
| 844 | 824 | bpf_frame_size += 2 * SZL; |
| 845 | 825 | |
| 826 | + /* Save area for return address */ | |
| 827 | + retaddr_off = bpf_frame_size; | |
| 828 | + bpf_frame_size += SZL; | |
| 829 | + | |
| 846 | 830 | /* Optional save area for actual LR in case of ool ftrace */ |
| 847 | 831 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { |
| 848 | 832 | alt_lr_off = bpf_frame_size; |
| @@ -869,16 +853,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 20 unmodified lines | ||
| 869 | 853 | /* Padding to align stack frame, if any */ |
| 870 | 854 | bpf_frame_size = round_up(bpf_frame_size, SZL * 2); |
| 871 | 855 | |
| 872 | − /* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */ | |
| 873 | − bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64; | |
| 874 | − | |
| 875 | − /* Offset to the traced function's stack frame */ | |
| 876 | − func_frame_offset = bpf_dummy_frame_size + bpf_frame_size; | |
| 877 | − | |
| 878 | − /* Create dummy frame for unwind, store original return value */ | |
| 856 | + /* Store original return value */ | |
| 879 | 857 | EMIT(PPC_RAW_STL(_R0, _R1, PPC_LR_STKOFF)); |
| 880 | − /* Protect red zone where tail call count goes */ | |
| 881 | − EMIT(PPC_RAW_STLU(_R1, _R1, -bpf_dummy_frame_size)); | |
| 882 | 858 | |
| 883 | 859 | /* Create our stack frame */ |
| 884 | 860 | EMIT(PPC_RAW_STLU(_R1, _R1, -bpf_frame_size)); |
| @@ -893,34 +869,44 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 8 unmodified lines | ||
| 893 | 869 | if (IS_ENABLED(CONFIG_PPC32) && nr_regs < 2) |
| 894 | 870 | EMIT(PPC_RAW_STL(_R4, _R1, r4_off)); |
| 895 | 871 | |
| 896 | − bpf_trampoline_save_args(image, ctx, func_frame_offset, nr_regs, regs_off); | |
| 872 | + bpf_trampoline_save_args(image, ctx, bpf_frame_size, nr_regs, regs_off); | |
| 897 | 873 | |
| 898 | − /* Save our return address */ | |
| 874 | + /* Save our LR/return address */ | |
| 899 | 875 | EMIT(PPC_RAW_MFLR(_R3)); |
| 900 | 876 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) |
| 901 | 877 | EMIT(PPC_RAW_STL(_R3, _R1, alt_lr_off)); |
| 902 | 878 | else |
| 903 | − EMIT(PPC_RAW_STL(_R3, _R1, bpf_frame_size + PPC_LR_STKOFF)); | |
| 879 | + EMIT(PPC_RAW_STL(_R3, _R1, retaddr_off)); | |
| 904 | 880 | |
| 905 | 881 | /* |
| 906 | − * Save ip address of the traced function. | |
| 907 | − * We could recover this from LR, but we will need to address for OOL trampoline, | |
| 908 | − * and optional GEP area. | |
| 882 | + * Derive IP address of the traced function. | |
| 883 | + * In case of CONFIG_PPC_FTRACE_OUT_OF_LINE or BPF program, LR points to the instruction | |
| 884 | + * after the 'bl' instruction in the OOL stub. Refer to ftrace_init_ool_stub() and | |
| 885 | + * bpf_arch_text_poke() for OOL stub of kernel functions and bpf programs respectively. | |
| 886 | + * Relevant stub sequence: | |
| 887 | + * | |
| 888 | + * bl <tramp> | |
| 889 | + * LR (R3) => mtlr r0 | |
| 890 | + * b <func_addr+4> | |
| 891 | + * | |
| 892 | + * Recover kernel function/bpf program address from the unconditional | |
| 893 | + * branch instruction at the end of OOL stub. | |
| 909 | 894 | */ |
| 910 | 895 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE) || flags & BPF_TRAMP_F_IP_ARG) { |
| 911 | 896 | EMIT(PPC_RAW_LWZ(_R4, _R3, 4)); |
| 912 | 897 | EMIT(PPC_RAW_SLWI(_R4, _R4, 6)); |
| 913 | 898 | EMIT(PPC_RAW_SRAWI(_R4, _R4, 6)); |
| 914 | 899 | EMIT(PPC_RAW_ADD(_R3, _R3, _R4)); |
| 915 | − EMIT(PPC_RAW_ADDI(_R3, _R3, 4)); | |
| 916 | 900 | } |
| 917 | 901 | |
| 918 | 902 | if (flags & BPF_TRAMP_F_IP_ARG) |
| 919 | 903 | EMIT(PPC_RAW_STL(_R3, _R1, ip_off)); |
| 920 | 904 | |
| 921 | − if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) | |
| 922 | − /* Fake our LR for unwind */ | |
| 923 | − EMIT(PPC_RAW_STL(_R3, _R1, bpf_frame_size + PPC_LR_STKOFF)); | |
| 905 | + if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { | |
| 906 | + /* Fake our LR for BPF_TRAMP_F_CALL_ORIG case */ | |
| 907 | + EMIT(PPC_RAW_ADDI(_R3, _R3, 4)); | |
| 908 | + EMIT(PPC_RAW_STL(_R3, _R1, retaddr_off)); | |
| 909 | + } | |
| 924 | 910 | |
| 925 | 911 | /* Save function arg count -- see bpf_get_func_arg_cnt() */ |
| 926 | 912 | EMIT(PPC_RAW_LI(_R3, nr_regs)); |
| @@ -958,20 +944,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 31 unmodified lines | ||
| 958 | 944 | /* Call the traced function */ |
| 959 | 945 | if (flags & BPF_TRAMP_F_CALL_ORIG) { |
| 960 | 946 | /* |
| 961 | − * The address in LR save area points to the correct point in the original function | |
| 947 | + * retaddr on trampoline stack points to the correct point in the original function | |
| 962 | 948 | * with both PPC_FTRACE_OUT_OF_LINE as well as with traditional ftrace instruction |
| 963 | 949 | * sequence |
| 964 | 950 | */ |
| 965 | − EMIT(PPC_RAW_LL(_R3, _R1, bpf_frame_size + PPC_LR_STKOFF)); | |
| 951 | + EMIT(PPC_RAW_LL(_R3, _R1, retaddr_off)); | |
| 966 | 952 | EMIT(PPC_RAW_MTCTR(_R3)); |
| 967 | 953 | |
| 968 | 954 | /* Replicate tail_call_cnt before calling the original BPF prog */ |
| 969 | 955 | if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) |
| 970 | − bpf_trampoline_setup_tail_call_info(image, ctx, func_frame_offset, | |
| 971 | − bpf_dummy_frame_size, r4_off); | |
| 956 | + bpf_trampoline_setup_tail_call_info(image, ctx, bpf_frame_size, r4_off); | |
| 972 | 957 | |
| 973 | 958 | /* Restore args */ |
| 974 | − bpf_trampoline_restore_args_stack(image, ctx, func_frame_offset, nr_regs, regs_off); | |
| 959 | + bpf_trampoline_restore_args_stack(image, ctx, bpf_frame_size, nr_regs, regs_off); | |
| 975 | 960 | |
| 976 | 961 | /* Restore TOC for 64-bit */ |
| 977 | 962 | if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) && !IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)) |
| @@ -985,7 +970,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 7 unmodified lines | ||
| 985 | 970 | |
| 986 | 971 | /* Restore updated tail_call_cnt */ |
| 987 | 972 | if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) |
| 988 | − bpf_trampoline_restore_tail_call_cnt(image, ctx, func_frame_offset, r4_off); | |
| 973 | + bpf_trampoline_restore_tail_call_cnt(image, ctx, bpf_frame_size, r4_off); | |
| 989 | 974 | |
| 990 | 975 | /* Reserve space to patch branch instruction to skip fexit progs */ |
| 991 | 976 | if (ro_image) /* image is NULL for dummy pass */ |
| @@ -1037,7 +1022,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 45 unmodified lines | ||
| 1037 | 1022 | EMIT(PPC_RAW_LD(_R2, _R1, 24)); |
| 1038 | 1023 | if (flags & BPF_TRAMP_F_SKIP_FRAME) { |
| 1039 | 1024 | /* Skip the traced function and return to parent */ |
| 1040 | − EMIT(PPC_RAW_ADDI(_R1, _R1, func_frame_offset)); | |
| 1025 | + EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_frame_size)); | |
| 1041 | 1026 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); |
| 1042 | 1027 | EMIT(PPC_RAW_MTLR(_R0)); |
| 1043 | 1028 | EMIT(PPC_RAW_BLR()); |
| @@ -1045,13 +1030,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line | ||
| 1045 | 1030 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { |
| 1046 | 1031 | EMIT(PPC_RAW_LL(_R0, _R1, alt_lr_off)); |
| 1047 | 1032 | EMIT(PPC_RAW_MTLR(_R0)); |
| 1048 | − EMIT(PPC_RAW_ADDI(_R1, _R1, func_frame_offset)); | |
| 1033 | + EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_frame_size)); | |
| 1049 | 1034 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); |
| 1050 | 1035 | EMIT(PPC_RAW_BLR()); |
| 1051 | 1036 | } else { |
| 1052 | − EMIT(PPC_RAW_LL(_R0, _R1, bpf_frame_size + PPC_LR_STKOFF)); | |
| 1037 | + EMIT(PPC_RAW_LL(_R0, _R1, retaddr_off)); | |
| 1053 | 1038 | EMIT(PPC_RAW_MTCTR(_R0)); |
| 1054 | − EMIT(PPC_RAW_ADDI(_R1, _R1, func_frame_offset)); | |
| 1039 | + EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_frame_size)); | |
| 1055 | 1040 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); |
| 1056 | 1041 | EMIT(PPC_RAW_MTLR(_R0)); |
| 1057 | 1042 | EMIT(PPC_RAW_BCTR()); |
| @@ -32,23 +32,27 @@ · 31 unmodified lines | ||
| 32 | 32 | * |
| 33 | 33 | * [ prev sp ] <------------- |
| 34 | 34 | * [ tail_call_info ] 8 | |
| 35 | − * [ nv gpr save area ] 6*8 + (12*8) | | |
| 35 | + * [ nv gpr save area ] (6 * 8) | | |
| 36 | + * [ addl. nv gpr save area] (12 * 8) | <--- exception boundary/callback program | |
| 36 | 37 | * [ local_tmp_var ] 24 | |
| 37 | 38 | * fp (r31) --> [ ebpf stack space ] upto 512 | |
| 38 | 39 | * [ frame header ] 32/112 | |
| 39 | 40 | * sp (r1) ---> [ stack pointer ] -------------- |
| 40 | 41 | * |
| 41 | − * Additional (12*8) in 'nv gpr save area' only in case of | |
| 42 | − * exception boundary. | |
| 42 | + * Additional (12 * 8) in 'nv gpr save area' only in case of | |
| 43 | + * exception boundary/callback. | |
| 43 | 44 | */ |
| 44 | 45 | |
| 46 | +/* BPF non-volatile registers save area size */ | |
| 47 | +#define BPF_PPC_STACK_SAVE (6 * 8) | |
| 48 | + | |
| 45 | 49 | /* for bpf JIT code internal usage */ |
| 46 | 50 | #define BPF_PPC_STACK_LOCALS 24 |
| 47 | 51 | /* |
| 48 | 52 | * for additional non volatile registers(r14-r25) to be saved |
| 49 | 53 | * at exception boundary |
| 50 | 54 | */ |
| 51 | −#define BPF_PPC_EXC_STACK_SAVE (12*8) | |
| 55 | +#define BPF_PPC_EXC_STACK_SAVE (12 * 8) | |
| 52 | 56 | |
| 53 | 57 | /* stack frame excluding BPF stack, ensure this is quadword aligned */ |
| 54 | 58 | #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ |
| @@ -125,12 +129,13 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) · 70 unmodified lines | ||
| 125 | 129 | * [ ... ] | |
| 126 | 130 | * sp (r1) ---> [ stack pointer ] -------------- |
| 127 | 131 | * [ tail_call_info ] 8 |
| 128 | − * [ nv gpr save area ] 6*8 + (12*8) | |
| 132 | + * [ nv gpr save area ] (6 * 8) | |
| 133 | + * [ addl. nv gpr save area] (12 * 8) <--- exception boundary/callback program | |
| 129 | 134 | * [ local_tmp_var ] 24 |
| 130 | 135 | * [ unused red zone ] 224 |
| 131 | 136 | * |
| 132 | − * Additional (12*8) in 'nv gpr save area' only in case of | |
| 133 | − * exception boundary. | |
| 137 | + * Additional (12 * 8) in 'nv gpr save area' only in case of | |
| 138 | + * exception boundary/callback. | |
| 134 | 139 | */ |
| 135 | 140 | static int bpf_jit_stack_local(struct codegen_context *ctx) |
| 136 | 141 | { |
| @@ -148,7 +153,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx) · 11 unmodified lines | ||
| 148 | 153 | } |
| 149 | 154 | } |
| 150 | 155 | |
| 151 | −int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx) | |
| 156 | +static int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx) | |
| 152 | 157 | { |
| 153 | 158 | return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE; |
| 154 | 159 | } |
| @@ -237,10 +242,6 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) · 82 unmodified lines | ||
| 237 | 242 | |
| 238 | 243 | if (bpf_has_stack_frame(ctx) && !ctx->exception_cb) { |
| 239 | 244 | /* |
| 240 | − * exception_cb uses boundary frame after stack walk. | |
| 241 | − * It can simply use redzone, this optimization reduces | |
| 242 | − * stack walk loop by one level. | |
| 243 | − * | |
| 244 | 245 | * We need a stack frame, but we don't necessarily need to |
| 245 | 246 | * save/restore LR unless we call other functions |
| 246 | 247 | */ |
| @@ -284,6 +285,22 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) · 37 unmodified lines | ||
| 284 | 285 | * program(main prog) as third arg |
| 285 | 286 | */ |
| 286 | 287 | EMIT(PPC_RAW_MR(_R1, _R5)); |
| 288 | + /* | |
| 289 | + * Exception callback reuses the stack frame of exception boundary. | |
| 290 | + * But BPF stack depth of exception callback and exception boundary | |
| 291 | + * don't have to be same. If BPF stack depth is different, adjust the | |
| 292 | + * stack frame size considering BPF stack depth of exception callback. | |
| 293 | + * The non-volatile register save area remains unchanged. These non- | |
| 294 | + * volatile registers are restored in exception callback's epilogue. | |
| 295 | + */ | |
| 296 | + EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R5, 0)); | |
| 297 | + EMIT(PPC_RAW_SUB(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_1), _R1)); | |
| 298 | + EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | |
| 299 | + -BPF_PPC_EXC_STACKFRAME)); | |
| 300 | + EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_2), ctx->stack_size)); | |
| 301 | + PPC_BCC_CONST_SHORT(COND_EQ, 12); | |
| 302 | + EMIT(PPC_RAW_MR(_R1, bpf_to_ppc(TMP_REG_1))); | |
| 303 | + EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_EXC_STACKFRAME + ctx->stack_size))); | |
| 287 | 304 | } |
| 288 | 305 | |
| 289 | 306 | /* |
| @@ -482,6 +499,83 @@ int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context * · 192 unmodified lines | ||
| 482 | 499 | return 0; |
| 483 | 500 | } |
| 484 | 501 | |
| 502 | +static int zero_extend(u32 *image, struct codegen_context *ctx, u32 src_reg, u32 dst_reg, u32 size) | |
| 503 | +{ | |
| 504 | + switch (size) { | |
| 505 | + case 1: | |
| 506 | + /* zero-extend 8 bits into 64 bits */ | |
| 507 | + EMIT(PPC_RAW_RLDICL(dst_reg, src_reg, 0, 56)); | |
| 508 | + return 0; | |
| 509 | + case 2: | |
| 510 | + /* zero-extend 16 bits into 64 bits */ | |
| 511 | + EMIT(PPC_RAW_RLDICL(dst_reg, src_reg, 0, 48)); | |
| 512 | + return 0; | |
| 513 | + case 4: | |
| 514 | + /* zero-extend 32 bits into 64 bits */ | |
| 515 | + EMIT(PPC_RAW_RLDICL(dst_reg, src_reg, 0, 32)); | |
| 516 | + fallthrough; | |
| 517 | + case 8: | |
| 518 | + /* Nothing to do */ | |
| 519 | + return 0; | |
| 520 | + default: | |
| 521 | + return -1; | |
| 522 | + } | |
| 523 | +} | |
| 524 | + | |
| 525 | +static int sign_extend(u32 *image, struct codegen_context *ctx, u32 src_reg, u32 dst_reg, u32 size) | |
| 526 | +{ | |
| 527 | + switch (size) { | |
| 528 | + case 1: | |
| 529 | + /* sign-extend 8 bits into 64 bits */ | |
| 530 | + EMIT(PPC_RAW_EXTSB(dst_reg, src_reg)); | |
| 531 | + return 0; | |
| 532 | + case 2: | |
| 533 | + /* sign-extend 16 bits into 64 bits */ | |
| 534 | + EMIT(PPC_RAW_EXTSH(dst_reg, src_reg)); | |
| 535 | + return 0; | |
| 536 | + case 4: | |
| 537 | + /* sign-extend 32 bits into 64 bits */ | |
| 538 | + EMIT(PPC_RAW_EXTSW(dst_reg, src_reg)); | |
| 539 | + fallthrough; | |
| 540 | + case 8: | |
| 541 | + /* Nothing to do */ | |
| 542 | + return 0; | |
| 543 | + default: | |
| 544 | + return -1; | |
| 545 | + } | |
| 546 | +} | |
| 547 | + | |
| 548 | +/* | |
| 549 | + * Handle powerpc ABI expectations from caller: | |
| 550 | + * - Unsigned arguments are zero-extended. | |
| 551 | + * - Signed arguments are sign-extended. | |
| 552 | + */ | |
| 553 | +static int prepare_for_kfunc_call(const struct bpf_prog *fp, u32 *image, | |
| 554 | + struct codegen_context *ctx, | |
| 555 | + const struct bpf_insn *insn) | |
| 556 | +{ | |
| 557 | + const struct btf_func_model *m = bpf_jit_find_kfunc_model(fp, insn); | |
| 558 | + int i; | |
| 559 | + | |
| 560 | + if (!m) | |
| 561 | + return -1; | |
| 562 | + | |
| 563 | + for (i = 0; i < m->nr_args; i++) { | |
| 564 | + /* Note that BPF ABI only allows up to 5 args for kfuncs */ | |
| 565 | + u32 reg = bpf_to_ppc(BPF_REG_1 + i), size = m->arg_size[i]; | |
| 566 | + | |
| 567 | + if (!(m->arg_flags[i] & BTF_FMODEL_SIGNED_ARG)) { | |
| 568 | + if (zero_extend(image, ctx, reg, reg, size)) | |
| 569 | + return -1; | |
| 570 | + } else { | |
| 571 | + if (sign_extend(image, ctx, reg, reg, size)) | |
| 572 | + return -1; | |
| 573 | + } | |
| 574 | + } | |
| 575 | + | |
| 576 | + return 0; | |
| 577 | +} | |
| 578 | + | |
| 485 | 579 | static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 out) |
| 486 | 580 | { |
| 487 | 581 | /* |
| @@ -522,9 +616,30 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o · 34 unmodified lines | ||
| 522 | 616 | |
| 523 | 617 | /* |
| 524 | 618 | * tail_call_info++; <- Actual value of tcc here |
| 619 | + * Writeback this updated value only if tailcall succeeds. | |
| 525 | 620 | */ |
| 526 | 621 | EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); |
| 527 | 622 | |
| 623 | + /* prog = array->ptrs[index]; */ | |
| 624 | + EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_2), b2p_index, 8)); | |
| 625 | + EMIT(PPC_RAW_ADD(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), b2p_bpf_array)); | |
| 626 | + EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | |
| 627 | + offsetof(struct bpf_array, ptrs))); | |
| 628 | + | |
| 629 | + /* | |
| 630 | + * if (prog == NULL) | |
| 631 | + * goto out; | |
| 632 | + */ | |
| 633 | + EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_2), 0)); | |
| 634 | + PPC_BCC_SHORT(COND_EQ, out); | |
| 635 | + | |
| 636 | + /* goto *(prog->bpf_func + prologue_size); */ | |
| 637 | + EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | |
| 638 | + offsetof(struct bpf_prog, bpf_func))); | |
| 639 | + EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | |
| 640 | + FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size)); | |
| 641 | + EMIT(PPC_RAW_MTCTR(bpf_to_ppc(TMP_REG_2))); | |
| 642 | + | |
| 528 | 643 | /* |
| 529 | 644 | * Before writing updated tail_call_info, distinguish if current frame |
| 530 | 645 | * is storing a reference to tail_call_info or actual tcc value in |
| @@ -539,24 +654,6 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o · 8 unmodified lines | ||
| 539 | 654 | /* Writeback updated value to tail_call_info */ |
| 540 | 655 | EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0)); |
| 541 | 656 | |
| 542 | − /* prog = array->ptrs[index]; */ | |
| 543 | − EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8)); | |
| 544 | − EMIT(PPC_RAW_ADD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), b2p_bpf_array)); | |
| 545 | − EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), offsetof(struct bpf_array, ptrs))); | |
| 546 | − | |
| 547 | − /* | |
| 548 | − * if (prog == NULL) | |
| 549 | − * goto out; | |
| 550 | − */ | |
| 551 | − EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_1), 0)); | |
| 552 | − PPC_BCC_SHORT(COND_EQ, out); | |
| 553 | − | |
| 554 | − /* goto *(prog->bpf_func + prologue_size); */ | |
| 555 | − EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), offsetof(struct bpf_prog, bpf_func))); | |
| 556 | − EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), | |
| 557 | − FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size)); | |
| 558 | − EMIT(PPC_RAW_MTCTR(bpf_to_ppc(TMP_REG_1))); | |
| 559 | − | |
| 560 | 657 | /* tear down stack, restore NVRs, ... */ |
| 561 | 658 | bpf_jit_emit_common_epilogue(image, ctx); |
| 562 | 659 | |
| @@ -1123,14 +1220,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct code · 560 unmodified lines | ||
| 1123 | 1220 | /* special mov32 for zext */ |
| 1124 | 1221 | EMIT(PPC_RAW_RLWINM(dst_reg, dst_reg, 0, 0, 31)); |
| 1125 | 1222 | break; |
| 1126 | − } else if (off == 8) { | |
| 1127 | − EMIT(PPC_RAW_EXTSB(dst_reg, src_reg)); | |
| 1128 | − } else if (off == 16) { | |
| 1129 | − EMIT(PPC_RAW_EXTSH(dst_reg, src_reg)); | |
| 1130 | − } else if (off == 32) { | |
| 1131 | − EMIT(PPC_RAW_EXTSW(dst_reg, src_reg)); | |
| 1132 | − } else if (dst_reg != src_reg) | |
| 1133 | − EMIT(PPC_RAW_MR(dst_reg, src_reg)); | |
| 1223 | + } | |
| 1224 | + if (off == 0) { | |
| 1225 | + /* MOV */ | |
| 1226 | + if (dst_reg != src_reg) | |
| 1227 | + EMIT(PPC_RAW_MR(dst_reg, src_reg)); | |
| 1228 | + } else { | |
| 1229 | + /* MOVSX: dst = (s8,s16,s32)src (off = 8,16,32) */ | |
| 1230 | + if (sign_extend(image, ctx, src_reg, dst_reg, off / 8)) | |
| 1231 | + return -1; | |
| 1232 | + } | |
| 1134 | 1233 | goto bpf_alu32_trunc; |
| 1135 | 1234 | case BPF_ALU | BPF_MOV | BPF_K: /* (u32) dst = imm */ |
| 1136 | 1235 | case BPF_ALU64 | BPF_MOV | BPF_K: /* dst = (s64) imm */ |
| @@ -1598,6 +1697,12 @@ emit_clear: · 461 unmodified lines | ||
| 1598 | 1697 | if (ret < 0) |
| 1599 | 1698 | return ret; |
| 1600 | 1699 | |
| 1700 | + /* Take care of powerpc ABI requirements before kfunc call */ | |
| 1701 | + if (insn[i].src_reg == BPF_PSEUDO_KFUNC_CALL) { | |
| 1702 | + if (prepare_for_kfunc_call(fp, image, ctx, &insn[i])) | |
| 1703 | + return -1; | |
| 1704 | + } | |
| 1705 | + | |
| 1601 | 1706 | ret = bpf_jit_emit_func_call_rel(image, fimage, ctx, func_addr); |
| 1602 | 1707 | if (ret) |
| 1603 | 1708 | return ret; |
| @@ -103,6 +103,11 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re · 102 unmodified lines | ||
| 103 | 103 | void |
| 104 | 104 | perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) |
| 105 | 105 | { |
| 106 | + perf_callchain_store(entry, perf_arch_instruction_pointer(regs)); | |
| 107 | + | |
| 108 | + if (!current->mm) | |
| 109 | + return; | |
| 110 | + | |
| 106 | 111 | if (!is_32bit_task()) |
| 107 | 112 | perf_callchain_user_64(entry, regs); |
| 108 | 113 | else |
| @@ -142,7 +142,6 @@ void perf_callchain_user_32(struct perf_callchain_entry_ctx *entry, · 141 unmodified lines | ||
| 142 | 142 | next_ip = perf_arch_instruction_pointer(regs); |
| 143 | 143 | lr = regs->link; |
| 144 | 144 | sp = regs->gpr[1]; |
| 145 | − perf_callchain_store(entry, next_ip); | |
| 146 | 145 | |
| 147 | 146 | while (entry->nr < entry->max_stack) { |
| 148 | 147 | fp = (unsigned int __user *) (unsigned long) sp; |
| @@ -77,7 +77,6 @@ void perf_callchain_user_64(struct perf_callchain_entry_ctx *entry, · 76 unmodified lines | ||
| 77 | 77 | next_ip = perf_arch_instruction_pointer(regs); |
| 78 | 78 | lr = regs->link; |
| 79 | 79 | sp = regs->gpr[1]; |
| 80 | − perf_callchain_store(entry, next_ip); | |
| 81 | 80 | |
| 82 | 81 | while (entry->nr < entry->max_stack) { |
| 83 | 82 | fp = (unsigned long __user *) sp; |
| @@ -155,8 +155,8 @@ machine_device_initcall(mpc83xx_km, mpc83xx_declare_of_platform_devices); · 154 unmodified lines | ||
| 155 | 155 | |
| 156 | 156 | /* list of the supported boards */ |
| 157 | 157 | static char *board[] __initdata = { |
| 158 | − "Keymile,KMETER1", | |
| 159 | − "Keymile,kmpbec8321", | |
| 158 | + "keymile,KMETER1", | |
| 159 | + "keymile,kmpbec8321", | |
| 160 | 160 | NULL |
| 161 | 161 | }; |
| 162 | 162 | |
| @@ -276,7 +276,7 @@ config PPC_BOOK3S · 275 unmodified lines | ||
| 276 | 276 | config PPC_E500 |
| 277 | 277 | select FSL_EMB_PERFMON |
| 278 | 278 | bool |
| 279 | − select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 | |
| 279 | + select ARCH_SUPPORTS_HUGETLBFS | |
| 280 | 280 | select PPC_SMP_MUXED_IPI |
| 281 | 281 | select PPC_DOORBELL |
| 282 | 282 | select PPC_KUEP |
| @@ -337,7 +337,7 @@ config BOOKE · 54 unmodified lines | ||
| 337 | 337 | config PTE_64BIT |
| 338 | 338 | bool |
| 339 | 339 | depends on 44x || PPC_E500 || PPC_86xx |
| 340 | − default y if PHYS_64BIT | |
| 340 | + default y if PPC_E500 || PHYS_64BIT | |
| 341 | 341 | |
| 342 | 342 | config PHYS_64BIT |
| 343 | 343 | bool 'Large physical address support' if PPC_E500 || PPC_86xx |
| @@ -605,7 +605,7 @@ static int pseries_irq_domain_alloc(struct irq_domain *domain, unsigned int virq · 604 unmodified lines | ||
| 605 | 605 | &pseries_msi_irq_chip, pseries_dev); |
| 606 | 606 | } |
| 607 | 607 | |
| 608 | − pseries_dev->msi_used++; | |
| 608 | + pseries_dev->msi_used += nr_irqs; | |
| 609 | 609 | return 0; |
| 610 | 610 | |
| 611 | 611 | out: |
File renamed without changes.
| @@ -15,9 +15,9 @@ if [ -z "$is_64bit" ]; then · 14 unmodified lines | ||
| 15 | 15 | RELOCATION=R_PPC_ADDR32 |
| 16 | 16 | fi |
| 17 | 17 | |
| 18 | −num_ool_stubs_total=$($objdump -r -j __patchable_function_entries "$vmlinux_o" | | |
| 18 | +num_ool_stubs_total=$($objdump -r -j __patchable_function_entries -d "$vmlinux_o" | | |
| 19 | 19 | grep -c "$RELOCATION") |
| 20 | −num_ool_stubs_inittext=$($objdump -r -j __patchable_function_entries "$vmlinux_o" | | |
| 20 | +num_ool_stubs_inittext=$($objdump -r -j __patchable_function_entries -d "$vmlinux_o" | | |
| 21 | 21 | grep -e ".init.text" -e ".text.startup" | grep -c "$RELOCATION") |
| 22 | 22 | num_ool_stubs_text=$((num_ool_stubs_total - num_ool_stubs_inittext)) |
| 23 | 23 | |
| @@ -428,6 +428,7 @@ · 427 unmodified lines | ||
| 428 | 428 | clocks = <&clkcfg CLK_CAN0>, <&clkcfg CLK_MSSPLL3>; |
| 429 | 429 | interrupt-parent = <&plic>; |
| 430 | 430 | interrupts = <56>; |
| 431 | + resets = <&mss_top_sysreg CLK_CAN0>; | |
| 431 | 432 | status = "disabled"; |
| 432 | 433 | }; |
| 433 | 434 | |
| @@ -437,6 +438,7 @@ · 3 unmodified lines | ||
| 437 | 438 | clocks = <&clkcfg CLK_CAN1>, <&clkcfg CLK_MSSPLL3>; |
| 438 | 439 | interrupt-parent = <&plic>; |
| 439 | 440 | interrupts = <57>; |
| 441 | + resets = <&mss_top_sysreg CLK_CAN1>; | |
| 440 | 442 | status = "disabled"; |
| 441 | 443 | }; |
| 442 | 444 | |
| @@ -2,6 +2,10 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef _ASM_RISCV_RUNTIME_CONST_H |
| 3 | 3 | #define _ASM_RISCV_RUNTIME_CONST_H |
| 4 | 4 | |
| 5 | +#ifdef MODULE | |
| 6 | + #error "Cannot use runtime-const infrastructure from modules" | |
| 7 | +#endif | |
| 8 | + | |
| 5 | 9 | #include <asm/asm.h> |
| 6 | 10 | #include <asm/alternative.h> |
| 7 | 11 | #include <asm/cacheflush.h> |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #ifndef __ASSEMBLER__ |
| 10 | 10 | |
| 11 | 11 | #include <linux/types.h> |
| 12 | +#include <linux/const.h> | |
| 12 | 13 | |
| 13 | 14 | #define PTRACE_GETFDPIC 33 |
| 14 | 15 | |
| @@ -138,12 +139,12 @@ struct __sc_riscv_cfi_state { · 123 unmodified lines | ||
| 138 | 139 | #define PTRACE_CFI_SS_LOCK_BIT 4 |
| 139 | 140 | #define PTRACE_CFI_SS_PTR_BIT 5 |
| 140 | 141 | |
| 141 | −#define PTRACE_CFI_LP_EN_STATE BIT(PTRACE_CFI_LP_EN_BIT) | |
| 142 | −#define PTRACE_CFI_LP_LOCK_STATE BIT(PTRACE_CFI_LP_LOCK_BIT) | |
| 143 | −#define PTRACE_CFI_ELP_STATE BIT(PTRACE_CFI_ELP_BIT) | |
| 144 | −#define PTRACE_CFI_SS_EN_STATE BIT(PTRACE_CFI_SS_EN_BIT) | |
| 145 | −#define PTRACE_CFI_SS_LOCK_STATE BIT(PTRACE_CFI_SS_LOCK_BIT) | |
| 146 | −#define PTRACE_CFI_SS_PTR_STATE BIT(PTRACE_CFI_SS_PTR_BIT) | |
| 142 | +#define PTRACE_CFI_LP_EN_STATE _BITUL(PTRACE_CFI_LP_EN_BIT) | |
| 143 | +#define PTRACE_CFI_LP_LOCK_STATE _BITUL(PTRACE_CFI_LP_LOCK_BIT) | |
| 144 | +#define PTRACE_CFI_ELP_STATE _BITUL(PTRACE_CFI_ELP_BIT) | |
| 145 | +#define PTRACE_CFI_SS_EN_STATE _BITUL(PTRACE_CFI_SS_EN_BIT) | |
| 146 | +#define PTRACE_CFI_SS_LOCK_STATE _BITUL(PTRACE_CFI_SS_LOCK_BIT) | |
| 147 | +#define PTRACE_CFI_SS_PTR_STATE _BITUL(PTRACE_CFI_SS_PTR_BIT) | |
| 147 | 148 | |
| 148 | 149 | #define PRACE_CFI_STATE_INVALID_MASK ~(PTRACE_CFI_LP_EN_STATE | \ |
| 149 | 150 | PTRACE_CFI_LP_LOCK_STATE | \ |
| @@ -175,7 +175,7 @@ struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] = { · 174 unmodified lines | ||
| 175 | 175 | {DBG_REG_T1, GDB_SIZEOF_REG, offsetof(struct pt_regs, t1)}, |
| 176 | 176 | {DBG_REG_T2, GDB_SIZEOF_REG, offsetof(struct pt_regs, t2)}, |
| 177 | 177 | {DBG_REG_FP, GDB_SIZEOF_REG, offsetof(struct pt_regs, s0)}, |
| 178 | − {DBG_REG_S1, GDB_SIZEOF_REG, offsetof(struct pt_regs, a1)}, | |
| 178 | + {DBG_REG_S1, GDB_SIZEOF_REG, offsetof(struct pt_regs, s1)}, | |
| 179 | 179 | {DBG_REG_A0, GDB_SIZEOF_REG, offsetof(struct pt_regs, a0)}, |
| 180 | 180 | {DBG_REG_A1, GDB_SIZEOF_REG, offsetof(struct pt_regs, a1)}, |
| 181 | 181 | {DBG_REG_A2, GDB_SIZEOF_REG, offsetof(struct pt_regs, a2)}, |
| @@ -244,8 +244,9 @@ sleeping_thread_to_gdb_regs(unsigned long *gdb_regs, struct task_struct *task) · 62 unmodified lines | ||
| 244 | 244 | gdb_regs[DBG_REG_S6_OFF] = task->thread.s[6]; |
| 245 | 245 | gdb_regs[DBG_REG_S7_OFF] = task->thread.s[7]; |
| 246 | 246 | gdb_regs[DBG_REG_S8_OFF] = task->thread.s[8]; |
| 247 | − gdb_regs[DBG_REG_S9_OFF] = task->thread.s[10]; | |
| 248 | − gdb_regs[DBG_REG_S10_OFF] = task->thread.s[11]; | |
| 247 | + gdb_regs[DBG_REG_S9_OFF] = task->thread.s[9]; | |
| 248 | + gdb_regs[DBG_REG_S10_OFF] = task->thread.s[10]; | |
| 249 | + gdb_regs[DBG_REG_S11_OFF] = task->thread.s[11]; | |
| 249 | 250 | gdb_regs[DBG_REG_EPC_OFF] = task->thread.ra; |
| 250 | 251 | } |
| 251 | 252 | |
| @@ -42,19 +42,20 @@ static inline bool is_kernel_exittext(uintptr_t addr) · 41 unmodified lines | ||
| 42 | 42 | static __always_inline void *patch_map(void *addr, const unsigned int fixmap) |
| 43 | 43 | { |
| 44 | 44 | uintptr_t uintaddr = (uintptr_t) addr; |
| 45 | − struct page *page; | |
| 45 | + phys_addr_t phys; | |
| 46 | 46 | |
| 47 | − if (core_kernel_text(uintaddr) || is_kernel_exittext(uintaddr)) | |
| 48 | − page = phys_to_page(__pa_symbol(addr)); | |
| 49 | − else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) | |
| 50 | − page = vmalloc_to_page(addr); | |
| 51 | − else | |
| 52 | − return addr; | |
| 47 | + if (core_kernel_text(uintaddr) || is_kernel_exittext(uintaddr)) { | |
| 48 | + phys = __pa_symbol(addr); | |
| 49 | + } else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) { | |
| 50 | + struct page *page = vmalloc_to_page(addr); | |
| 53 | 51 | |
| 54 | − BUG_ON(!page); | |
| 52 | + BUG_ON(!page); | |
| 53 | + phys = page_to_phys(page) + offset_in_page(addr); | |
| 54 | + } else { | |
| 55 | + return addr; | |
| 56 | + } | |
| 55 | 57 | |
| 56 | − return (void *)set_fixmap_offset(fixmap, page_to_phys(page) + | |
| 57 | − offset_in_page(addr)); | |
| 58 | + return (void *)set_fixmap_offset(fixmap, phys); | |
| 58 | 59 | } |
| 59 | 60 | |
| 60 | 61 | static void patch_unmap(int fixmap) |
| @@ -347,8 +347,10 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg) · 346 unmodified lines | ||
| 347 | 347 | if (arg & PR_TAGGED_ADDR_ENABLE && (tagged_addr_disabled || !pmlen)) |
| 348 | 348 | return -EINVAL; |
| 349 | 349 | |
| 350 | − if (!(arg & PR_TAGGED_ADDR_ENABLE)) | |
| 350 | + if (!(arg & PR_TAGGED_ADDR_ENABLE)) { | |
| 351 | 351 | pmlen = PMLEN_0; |
| 352 | + pmm = ENVCFG_PMM_PMLEN_0; | |
| 353 | + } | |
| 352 | 354 | |
| 353 | 355 | if (mmap_write_lock_killable(mm)) |
| 354 | 356 | return -EINTR; |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/irqchip/riscv-imsic.h> |
| 14 | 14 | #include <linux/irqdomain.h> |
| 15 | 15 | #include <linux/kvm_host.h> |
| 16 | +#include <linux/nospec.h> | |
| 16 | 17 | #include <linux/percpu.h> |
| 17 | 18 | #include <linux/spinlock.h> |
| 18 | 19 | #include <asm/cpufeature.h> |
| @@ -182,9 +183,14 @@ int kvm_riscv_vcpu_aia_get_csr(struct kvm_vcpu *vcpu, · 163 unmodified lines | ||
| 182 | 183 | unsigned long *out_val) |
| 183 | 184 | { |
| 184 | 185 | struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; |
| 186 | + unsigned long regs_max = sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); | |
| 185 | 187 | |
| 186 | − if (reg_num >= sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long)) | |
| 188 | + if (!riscv_isa_extension_available(vcpu->arch.isa, SSAIA)) | |
| 187 | 189 | return -ENOENT; |
| 190 | + if (reg_num >= regs_max) | |
| 191 | + return -ENOENT; | |
| 192 | + | |
| 193 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 188 | 194 | |
| 189 | 195 | *out_val = 0; |
| 190 | 196 | if (kvm_riscv_aia_available()) |
| @@ -198,9 +204,14 @@ int kvm_riscv_vcpu_aia_set_csr(struct kvm_vcpu *vcpu, · 7 unmodified lines | ||
| 198 | 204 | unsigned long val) |
| 199 | 205 | { |
| 200 | 206 | struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; |
| 207 | + unsigned long regs_max = sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); | |
| 201 | 208 | |
| 202 | − if (reg_num >= sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long)) | |
| 209 | + if (!riscv_isa_extension_available(vcpu->arch.isa, SSAIA)) | |
| 203 | 210 | return -ENOENT; |
| 211 | + if (reg_num >= regs_max) | |
| 212 | + return -ENOENT; | |
| 213 | + | |
| 214 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 204 | 215 | |
| 205 | 216 | if (kvm_riscv_aia_available()) { |
| 206 | 217 | ((unsigned long *)csr)[reg_num] = val; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/irqchip/riscv-aplic.h> |
| 11 | 11 | #include <linux/kvm_host.h> |
| 12 | 12 | #include <linux/math.h> |
| 13 | +#include <linux/nospec.h> | |
| 13 | 14 | #include <linux/spinlock.h> |
| 14 | 15 | #include <linux/swab.h> |
| 15 | 16 | #include <kvm/iodev.h> |
| @@ -45,7 +46,7 @@ static u32 aplic_read_sourcecfg(struct aplic *aplic, u32 irq) · 29 unmodified lines | ||
| 45 | 46 | |
| 46 | 47 | if (!irq || aplic->nr_irqs <= irq) |
| 47 | 48 | return 0; |
| 48 | − irqd = &aplic->irqs[irq]; | |
| 49 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 49 | 50 | |
| 50 | 51 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 51 | 52 | ret = irqd->sourcecfg; |
| @@ -61,7 +62,7 @@ static void aplic_write_sourcecfg(struct aplic *aplic, u32 irq, u32 val) · 9 unmodified lines | ||
| 61 | 62 | |
| 62 | 63 | if (!irq || aplic->nr_irqs <= irq) |
| 63 | 64 | return; |
| 64 | − irqd = &aplic->irqs[irq]; | |
| 65 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 65 | 66 | |
| 66 | 67 | if (val & APLIC_SOURCECFG_D) |
| 67 | 68 | val = 0; |
| @@ -81,7 +82,7 @@ static u32 aplic_read_target(struct aplic *aplic, u32 irq) · 13 unmodified lines | ||
| 81 | 82 | |
| 82 | 83 | if (!irq || aplic->nr_irqs <= irq) |
| 83 | 84 | return 0; |
| 84 | − irqd = &aplic->irqs[irq]; | |
| 85 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 85 | 86 | |
| 86 | 87 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 87 | 88 | ret = irqd->target; |
| @@ -97,7 +98,7 @@ static void aplic_write_target(struct aplic *aplic, u32 irq, u32 val) · 9 unmodified lines | ||
| 97 | 98 | |
| 98 | 99 | if (!irq || aplic->nr_irqs <= irq) |
| 99 | 100 | return; |
| 100 | − irqd = &aplic->irqs[irq]; | |
| 101 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 101 | 102 | |
| 102 | 103 | val &= APLIC_TARGET_EIID_MASK | |
| 103 | 104 | (APLIC_TARGET_HART_IDX_MASK << APLIC_TARGET_HART_IDX_SHIFT) | |
| @@ -116,7 +117,7 @@ static bool aplic_read_pending(struct aplic *aplic, u32 irq) · 12 unmodified lines | ||
| 116 | 117 | |
| 117 | 118 | if (!irq || aplic->nr_irqs <= irq) |
| 118 | 119 | return false; |
| 119 | − irqd = &aplic->irqs[irq]; | |
| 120 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 120 | 121 | |
| 121 | 122 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 122 | 123 | ret = (irqd->state & APLIC_IRQ_STATE_PENDING) ? true : false; |
| @@ -132,7 +133,7 @@ static void aplic_write_pending(struct aplic *aplic, u32 irq, bool pending) · 9 unmodified lines | ||
| 132 | 133 | |
| 133 | 134 | if (!irq || aplic->nr_irqs <= irq) |
| 134 | 135 | return; |
| 135 | − irqd = &aplic->irqs[irq]; | |
| 136 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 136 | 137 | |
| 137 | 138 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 138 | 139 | |
| @@ -170,7 +171,7 @@ static bool aplic_read_enabled(struct aplic *aplic, u32 irq) · 31 unmodified lines | ||
| 170 | 171 | |
| 171 | 172 | if (!irq || aplic->nr_irqs <= irq) |
| 172 | 173 | return false; |
| 173 | − irqd = &aplic->irqs[irq]; | |
| 174 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 174 | 175 | |
| 175 | 176 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 176 | 177 | ret = (irqd->state & APLIC_IRQ_STATE_ENABLED) ? true : false; |
| @@ -186,7 +187,7 @@ static void aplic_write_enabled(struct aplic *aplic, u32 irq, bool enabled) · 9 unmodified lines | ||
| 186 | 187 | |
| 187 | 188 | if (!irq || aplic->nr_irqs <= irq) |
| 188 | 189 | return; |
| 189 | − irqd = &aplic->irqs[irq]; | |
| 190 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 190 | 191 | |
| 191 | 192 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 192 | 193 | if (enabled) |
| @@ -205,7 +206,7 @@ static bool aplic_read_input(struct aplic *aplic, u32 irq) · 12 unmodified lines | ||
| 205 | 206 | |
| 206 | 207 | if (!irq || aplic->nr_irqs <= irq) |
| 207 | 208 | return false; |
| 208 | − irqd = &aplic->irqs[irq]; | |
| 209 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 209 | 210 | |
| 210 | 211 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 211 | 212 | |
| @@ -254,7 +255,7 @@ static void aplic_update_irq_range(struct kvm *kvm, u32 first, u32 last) · 42 unmodified lines | ||
| 254 | 255 | for (irq = first; irq <= last; irq++) { |
| 255 | 256 | if (!irq || aplic->nr_irqs <= irq) |
| 256 | 257 | continue; |
| 257 | − irqd = &aplic->irqs[irq]; | |
| 258 | + irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; | |
| 258 | 259 | |
| 259 | 260 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 260 | 261 | |
| @@ -283,7 +284,7 @@ int kvm_riscv_aia_aplic_inject(struct kvm *kvm, u32 source, bool level) · 22 unmodified lines | ||
| 283 | 284 | |
| 284 | 285 | if (!aplic || !source || (aplic->nr_irqs <= source)) |
| 285 | 286 | return -ENODEV; |
| 286 | − irqd = &aplic->irqs[source]; | |
| 287 | + irqd = &aplic->irqs[array_index_nospec(source, aplic->nr_irqs)]; | |
| 287 | 288 | ie = (aplic->domaincfg & APLIC_DOMAINCFG_IE) ? true : false; |
| 288 | 289 | |
| 289 | 290 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/irqchip/riscv-imsic.h> |
| 12 | 12 | #include <linux/kvm_host.h> |
| 13 | 13 | #include <linux/uaccess.h> |
| 14 | +#include <linux/cpufeature.h> | |
| 14 | 15 | |
| 15 | 16 | static int aia_create(struct kvm_device *dev, u32 type) |
| 16 | 17 | { |
| @@ -22,6 +23,9 @@ static int aia_create(struct kvm_device *dev, u32 type) · 5 unmodified lines | ||
| 22 | 23 | if (irqchip_in_kernel(kvm)) |
| 23 | 24 | return -EEXIST; |
| 24 | 25 | |
| 26 | + if (!riscv_isa_extension_available(NULL, SSAIA)) | |
| 27 | + return -ENODEV; | |
| 28 | + | |
| 25 | 29 | ret = -EBUSY; |
| 26 | 30 | if (kvm_trylock_all_vcpus(kvm)) |
| 27 | 31 | return ret; |
| @@ -437,7 +441,7 @@ static int aia_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr) · 409 unmodified lines | ||
| 437 | 441 | |
| 438 | 442 | static int aia_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr) |
| 439 | 443 | { |
| 440 | − int nr_vcpus; | |
| 444 | + int nr_vcpus, r = -ENXIO; | |
| 441 | 445 | |
| 442 | 446 | switch (attr->group) { |
| 443 | 447 | case KVM_DEV_RISCV_AIA_GRP_CONFIG: |
| @@ -466,12 +470,18 @@ static int aia_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr) · 22 unmodified lines | ||
| 466 | 470 | } |
| 467 | 471 | break; |
| 468 | 472 | case KVM_DEV_RISCV_AIA_GRP_APLIC: |
| 469 | − return kvm_riscv_aia_aplic_has_attr(dev->kvm, attr->attr); | |
| 473 | + mutex_lock(&dev->kvm->lock); | |
| 474 | + r = kvm_riscv_aia_aplic_has_attr(dev->kvm, attr->attr); | |
| 475 | + mutex_unlock(&dev->kvm->lock); | |
| 476 | + break; | |
| 470 | 477 | case KVM_DEV_RISCV_AIA_GRP_IMSIC: |
| 471 | − return kvm_riscv_aia_imsic_has_attr(dev->kvm, attr->attr); | |
| 478 | + mutex_lock(&dev->kvm->lock); | |
| 479 | + r = kvm_riscv_aia_imsic_has_attr(dev->kvm, attr->attr); | |
| 480 | + mutex_unlock(&dev->kvm->lock); | |
| 481 | + break; | |
| 472 | 482 | } |
| 473 | 483 | |
| 474 | − return -ENXIO; | |
| 484 | + return r; | |
| 475 | 485 | } |
| 476 | 486 | |
| 477 | 487 | struct kvm_device_ops kvm_riscv_aia_device_ops = { |
| @@ -908,6 +908,10 @@ int kvm_riscv_vcpu_aia_imsic_rmw(struct kvm_vcpu *vcpu, unsigned long isel, · 907 unmodified lines | ||
| 908 | 908 | int r, rc = KVM_INSN_CONTINUE_NEXT_SEPC; |
| 909 | 909 | struct imsic *imsic = vcpu->arch.aia_context.imsic_state; |
| 910 | 910 | |
| 911 | + /* If IMSIC vCPU state not initialized then forward to user space */ | |
| 912 | + if (!imsic) | |
| 913 | + return KVM_INSN_EXIT_TO_USER_SPACE; | |
| 914 | + | |
| 911 | 915 | if (isel == KVM_RISCV_AIA_IMSIC_TOPEI) { |
| 912 | 916 | /* Read pending and enabled interrupt with highest priority */ |
| 913 | 917 | topei = imsic_mrif_topei(imsic->swfile, imsic->nr_eix, |
| @@ -245,6 +245,7 @@ out: · 244 unmodified lines | ||
| 245 | 245 | bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) |
| 246 | 246 | { |
| 247 | 247 | struct kvm_gstage gstage; |
| 248 | + bool mmu_locked; | |
| 248 | 249 | |
| 249 | 250 | if (!kvm->arch.pgd) |
| 250 | 251 | return false; |
| @@ -253,9 +254,12 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · 2 unmodified lines | ||
| 253 | 254 | gstage.flags = 0; |
| 254 | 255 | gstage.vmid = READ_ONCE(kvm->arch.vmid.vmid); |
| 255 | 256 | gstage.pgd = kvm->arch.pgd; |
| 257 | + mmu_locked = spin_trylock(&kvm->mmu_lock); | |
| 256 | 258 | kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, |
| 257 | 259 | (range->end - range->start) << PAGE_SHIFT, |
| 258 | 260 | range->may_block); |
| 261 | + if (mmu_locked) | |
| 262 | + spin_unlock(&kvm->mmu_lock); | |
| 259 | 263 | return false; |
| 260 | 264 | } |
| 261 | 265 | |
| @@ -535,7 +539,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, · 273 unmodified lines | ||
| 535 | 539 | goto out_unlock; |
| 536 | 540 | |
| 537 | 541 | /* Check if we are backed by a THP and thus use block mapping if possible */ |
| 538 | − if (vma_pagesize == PAGE_SIZE) | |
| 542 | + if (!logging && (vma_pagesize == PAGE_SIZE)) | |
| 539 | 543 | vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &hfn, &gpa); |
| 540 | 544 | |
| 541 | 545 | if (writable) { |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #define CREATE_TRACE_POINTS |
| 25 | 25 | #include "trace.h" |
| 26 | 26 | |
| 27 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 27 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 28 | 28 | KVM_GENERIC_VCPU_STATS(), |
| 29 | 29 | STATS_DESC_COUNTER(VCPU, ecall_exit_stat), |
| 30 | 30 | STATS_DESC_COUNTER(VCPU, wfi_exit_stat), |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/errno.h> |
| 11 | 11 | #include <linux/err.h> |
| 12 | 12 | #include <linux/kvm_host.h> |
| 13 | +#include <linux/nospec.h> | |
| 13 | 14 | #include <linux/uaccess.h> |
| 14 | 15 | #include <asm/cpufeature.h> |
| 15 | 16 | |
| @@ -93,9 +94,11 @@ int kvm_riscv_vcpu_get_reg_fp(struct kvm_vcpu *vcpu, · 77 unmodified lines | ||
| 93 | 94 | if (reg_num == KVM_REG_RISCV_FP_F_REG(fcsr)) |
| 94 | 95 | reg_val = &cntx->fp.f.fcsr; |
| 95 | 96 | else if ((KVM_REG_RISCV_FP_F_REG(f[0]) <= reg_num) && |
| 96 | − reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) | |
| 97 | + reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) { | |
| 98 | + reg_num = array_index_nospec(reg_num, | |
| 99 | + ARRAY_SIZE(cntx->fp.f.f)); | |
| 97 | 100 | reg_val = &cntx->fp.f.f[reg_num]; |
| 98 | − else | |
| 101 | + } else | |
| 99 | 102 | return -ENOENT; |
| 100 | 103 | } else if ((rtype == KVM_REG_RISCV_FP_D) && |
| 101 | 104 | riscv_isa_extension_available(vcpu->arch.isa, d)) { |
| @@ -107,6 +110,8 @@ int kvm_riscv_vcpu_get_reg_fp(struct kvm_vcpu *vcpu, · 5 unmodified lines | ||
| 107 | 110 | reg_num <= KVM_REG_RISCV_FP_D_REG(f[31])) { |
| 108 | 111 | if (KVM_REG_SIZE(reg->id) != sizeof(u64)) |
| 109 | 112 | return -EINVAL; |
| 113 | + reg_num = array_index_nospec(reg_num, | |
| 114 | + ARRAY_SIZE(cntx->fp.d.f)); | |
| 110 | 115 | reg_val = &cntx->fp.d.f[reg_num]; |
| 111 | 116 | } else |
| 112 | 117 | return -ENOENT; |
| @@ -138,9 +143,11 @@ int kvm_riscv_vcpu_set_reg_fp(struct kvm_vcpu *vcpu, · 25 unmodified lines | ||
| 138 | 143 | if (reg_num == KVM_REG_RISCV_FP_F_REG(fcsr)) |
| 139 | 144 | reg_val = &cntx->fp.f.fcsr; |
| 140 | 145 | else if ((KVM_REG_RISCV_FP_F_REG(f[0]) <= reg_num) && |
| 141 | − reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) | |
| 146 | + reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) { | |
| 147 | + reg_num = array_index_nospec(reg_num, | |
| 148 | + ARRAY_SIZE(cntx->fp.f.f)); | |
| 142 | 149 | reg_val = &cntx->fp.f.f[reg_num]; |
| 143 | − else | |
| 150 | + } else | |
| 144 | 151 | return -ENOENT; |
| 145 | 152 | } else if ((rtype == KVM_REG_RISCV_FP_D) && |
| 146 | 153 | riscv_isa_extension_available(vcpu->arch.isa, d)) { |
| @@ -152,6 +159,8 @@ int kvm_riscv_vcpu_set_reg_fp(struct kvm_vcpu *vcpu, · 5 unmodified lines | ||
| 152 | 159 | reg_num <= KVM_REG_RISCV_FP_D_REG(f[31])) { |
| 153 | 160 | if (KVM_REG_SIZE(reg->id) != sizeof(u64)) |
| 154 | 161 | return -EINVAL; |
| 162 | + reg_num = array_index_nospec(reg_num, | |
| 163 | + ARRAY_SIZE(cntx->fp.d.f)); | |
| 155 | 164 | reg_val = &cntx->fp.d.f[reg_num]; |
| 156 | 165 | } else |
| 157 | 166 | return -ENOENT; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/bitops.h> |
| 11 | 11 | #include <linux/errno.h> |
| 12 | 12 | #include <linux/err.h> |
| 13 | +#include <linux/nospec.h> | |
| 13 | 14 | #include <linux/uaccess.h> |
| 14 | 15 | #include <linux/kvm_host.h> |
| 15 | 16 | #include <asm/cacheflush.h> |
| @@ -127,6 +128,7 @@ static int kvm_riscv_vcpu_isa_check_host(unsigned long kvm_ext, unsigned long *g · 111 unmodified lines | ||
| 127 | 128 | kvm_ext >= ARRAY_SIZE(kvm_isa_ext_arr)) |
| 128 | 129 | return -ENOENT; |
| 129 | 130 | |
| 131 | + kvm_ext = array_index_nospec(kvm_ext, ARRAY_SIZE(kvm_isa_ext_arr)); | |
| 130 | 132 | *guest_ext = kvm_isa_ext_arr[kvm_ext]; |
| 131 | 133 | switch (*guest_ext) { |
| 132 | 134 | case RISCV_ISA_EXT_SMNPM: |
| @@ -443,13 +445,16 @@ static int kvm_riscv_vcpu_get_reg_core(struct kvm_vcpu *vcpu, · 310 unmodified lines | ||
| 443 | 445 | unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | |
| 444 | 446 | KVM_REG_SIZE_MASK | |
| 445 | 447 | KVM_REG_RISCV_CORE); |
| 448 | + unsigned long regs_max = sizeof(struct kvm_riscv_core) / sizeof(unsigned long); | |
| 446 | 449 | unsigned long reg_val; |
| 447 | 450 | |
| 448 | 451 | if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) |
| 449 | 452 | return -EINVAL; |
| 450 | − if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) | |
| 453 | + if (reg_num >= regs_max) | |
| 451 | 454 | return -ENOENT; |
| 452 | 455 | |
| 456 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 457 | + | |
| 453 | 458 | if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) |
| 454 | 459 | reg_val = cntx->sepc; |
| 455 | 460 | else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && |
| @@ -476,13 +481,16 @@ static int kvm_riscv_vcpu_set_reg_core(struct kvm_vcpu *vcpu, · 20 unmodified lines | ||
| 476 | 481 | unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | |
| 477 | 482 | KVM_REG_SIZE_MASK | |
| 478 | 483 | KVM_REG_RISCV_CORE); |
| 484 | + unsigned long regs_max = sizeof(struct kvm_riscv_core) / sizeof(unsigned long); | |
| 479 | 485 | unsigned long reg_val; |
| 480 | 486 | |
| 481 | 487 | if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) |
| 482 | 488 | return -EINVAL; |
| 483 | − if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) | |
| 489 | + if (reg_num >= regs_max) | |
| 484 | 490 | return -ENOENT; |
| 485 | 491 | |
| 492 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 493 | + | |
| 486 | 494 | if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) |
| 487 | 495 | return -EFAULT; |
| 488 | 496 | |
| @@ -507,10 +515,13 @@ static int kvm_riscv_vcpu_general_get_csr(struct kvm_vcpu *vcpu, · 18 unmodified lines | ||
| 507 | 515 | unsigned long *out_val) |
| 508 | 516 | { |
| 509 | 517 | struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; |
| 518 | + unsigned long regs_max = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); | |
| 510 | 519 | |
| 511 | − if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) | |
| 520 | + if (reg_num >= regs_max) | |
| 512 | 521 | return -ENOENT; |
| 513 | 522 | |
| 523 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 524 | + | |
| 514 | 525 | if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { |
| 515 | 526 | kvm_riscv_vcpu_flush_interrupts(vcpu); |
| 516 | 527 | *out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK; |
| @@ -526,10 +537,13 @@ static int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu, · 9 unmodified lines | ||
| 526 | 537 | unsigned long reg_val) |
| 527 | 538 | { |
| 528 | 539 | struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; |
| 540 | + unsigned long regs_max = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); | |
| 529 | 541 | |
| 530 | − if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) | |
| 542 | + if (reg_num >= regs_max) | |
| 531 | 543 | return -ENOENT; |
| 532 | 544 | |
| 545 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 546 | + | |
| 533 | 547 | if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { |
| 534 | 548 | reg_val &= VSIP_VALID_MASK; |
| 535 | 549 | reg_val <<= VSIP_TO_HVIP_SHIFT; |
| @@ -548,10 +562,15 @@ static inline int kvm_riscv_vcpu_smstateen_set_csr(struct kvm_vcpu *vcpu, · 12 unmodified lines | ||
| 548 | 562 | unsigned long reg_val) |
| 549 | 563 | { |
| 550 | 564 | struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr; |
| 565 | + unsigned long regs_max = sizeof(struct kvm_riscv_smstateen_csr) / | |
| 566 | + sizeof(unsigned long); | |
| 551 | 567 | |
| 552 | − if (reg_num >= sizeof(struct kvm_riscv_smstateen_csr) / | |
| 553 | − sizeof(unsigned long)) | |
| 554 | − return -EINVAL; | |
| 568 | + if (!riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN)) | |
| 569 | + return -ENOENT; | |
| 570 | + if (reg_num >= regs_max) | |
| 571 | + return -ENOENT; | |
| 572 | + | |
| 573 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 555 | 574 | |
| 556 | 575 | ((unsigned long *)csr)[reg_num] = reg_val; |
| 557 | 576 | return 0; |
| @@ -562,10 +581,15 @@ static int kvm_riscv_vcpu_smstateen_get_csr(struct kvm_vcpu *vcpu, · 4 unmodified lines | ||
| 562 | 581 | unsigned long *out_val) |
| 563 | 582 | { |
| 564 | 583 | struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr; |
| 584 | + unsigned long regs_max = sizeof(struct kvm_riscv_smstateen_csr) / | |
| 585 | + sizeof(unsigned long); | |
| 565 | 586 | |
| 566 | − if (reg_num >= sizeof(struct kvm_riscv_smstateen_csr) / | |
| 567 | − sizeof(unsigned long)) | |
| 568 | − return -EINVAL; | |
| 587 | + if (!riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN)) | |
| 588 | + return -ENOENT; | |
| 589 | + if (reg_num >= regs_max) | |
| 590 | + return -ENOENT; | |
| 591 | + | |
| 592 | + reg_num = array_index_nospec(reg_num, regs_max); | |
| 569 | 593 | |
| 570 | 594 | *out_val = ((unsigned long *)csr)[reg_num]; |
| 571 | 595 | return 0; |
| @@ -595,10 +619,7 @@ static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu, · 23 unmodified lines | ||
| 595 | 619 | rc = kvm_riscv_vcpu_aia_get_csr(vcpu, reg_num, ®_val); |
| 596 | 620 | break; |
| 597 | 621 | case KVM_REG_RISCV_CSR_SMSTATEEN: |
| 598 | − rc = -EINVAL; | |
| 599 | − if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) | |
| 600 | − rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num, | |
| 601 | − ®_val); | |
| 622 | + rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num, ®_val); | |
| 602 | 623 | break; |
| 603 | 624 | default: |
| 604 | 625 | rc = -ENOENT; |
| @@ -640,10 +661,7 @@ static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu, · 35 unmodified lines | ||
| 640 | 661 | rc = kvm_riscv_vcpu_aia_set_csr(vcpu, reg_num, reg_val); |
| 641 | 662 | break; |
| 642 | 663 | case KVM_REG_RISCV_CSR_SMSTATEEN: |
| 643 | − rc = -EINVAL; | |
| 644 | − if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) | |
| 645 | − rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num, | |
| 646 | − reg_val); | |
| 664 | + rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num, reg_val); | |
| 647 | 665 | break; |
| 648 | 666 | default: |
| 649 | 667 | rc = -ENOENT; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/errno.h> |
| 11 | 11 | #include <linux/err.h> |
| 12 | 12 | #include <linux/kvm_host.h> |
| 13 | +#include <linux/nospec.h> | |
| 13 | 14 | #include <linux/perf/riscv_pmu.h> |
| 14 | 15 | #include <asm/csr.h> |
| 15 | 16 | #include <asm/kvm_vcpu_sbi.h> |
| @@ -87,7 +88,8 @@ static void kvm_pmu_release_perf_event(struct kvm_pmc *pmc) · 71 unmodified lines | ||
| 87 | 88 | |
| 88 | 89 | static u64 kvm_pmu_get_perf_event_hw_config(u32 sbi_event_code) |
| 89 | 90 | { |
| 90 | − return hw_event_perf_map[sbi_event_code]; | |
| 91 | + return hw_event_perf_map[array_index_nospec(sbi_event_code, | |
| 92 | + SBI_PMU_HW_GENERAL_MAX)]; | |
| 91 | 93 | } |
| 92 | 94 | |
| 93 | 95 | static u64 kvm_pmu_get_perf_event_cache_config(u32 sbi_event_code) |
| @@ -218,6 +220,7 @@ static int pmu_fw_ctr_read_hi(struct kvm_vcpu *vcpu, unsigned long cidx, · 124 unmodified lines | ||
| 218 | 220 | return -EINVAL; |
| 219 | 221 | } |
| 220 | 222 | |
| 223 | + cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS); | |
| 221 | 224 | pmc = &kvpmu->pmc[cidx]; |
| 222 | 225 | |
| 223 | 226 | if (pmc->cinfo.type != SBI_PMU_CTR_TYPE_FW) |
| @@ -244,6 +247,7 @@ static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx, · 20 unmodified lines | ||
| 244 | 247 | return -EINVAL; |
| 245 | 248 | } |
| 246 | 249 | |
| 250 | + cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS); | |
| 247 | 251 | pmc = &kvpmu->pmc[cidx]; |
| 248 | 252 | |
| 249 | 253 | if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) { |
| @@ -520,11 +524,12 @@ int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx, · 270 unmodified lines | ||
| 520 | 524 | { |
| 521 | 525 | struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu); |
| 522 | 526 | |
| 523 | − if (cidx > RISCV_KVM_MAX_COUNTERS || cidx == 1) { | |
| 527 | + if (cidx >= RISCV_KVM_MAX_COUNTERS || cidx == 1) { | |
| 524 | 528 | retdata->err_val = SBI_ERR_INVALID_PARAM; |
| 525 | 529 | return 0; |
| 526 | 530 | } |
| 527 | 531 | |
| 532 | + cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS); | |
| 528 | 533 | retdata->out_val = kvpmu->pmc[cidx].cinfo.value; |
| 529 | 534 | |
| 530 | 535 | return 0; |
| @@ -559,7 +564,8 @@ int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base, · 28 unmodified lines | ||
| 559 | 564 | } |
| 560 | 565 | /* Start the counters that have been configured and requested by the guest */ |
| 561 | 566 | for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) { |
| 562 | − pmc_index = i + ctr_base; | |
| 567 | + pmc_index = array_index_nospec(i + ctr_base, | |
| 568 | + RISCV_KVM_MAX_COUNTERS); | |
| 563 | 569 | if (!test_bit(pmc_index, kvpmu->pmc_in_use)) |
| 564 | 570 | continue; |
| 565 | 571 | /* The guest started the counter again. Reset the overflow status */ |
| @@ -630,7 +636,8 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base, · 64 unmodified lines | ||
| 630 | 636 | |
| 631 | 637 | /* Stop the counters that have been configured and requested by the guest */ |
| 632 | 638 | for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) { |
| 633 | − pmc_index = i + ctr_base; | |
| 639 | + pmc_index = array_index_nospec(i + ctr_base, | |
| 640 | + RISCV_KVM_MAX_COUNTERS); | |
| 634 | 641 | if (!test_bit(pmc_index, kvpmu->pmc_in_use)) |
| 635 | 642 | continue; |
| 636 | 643 | pmc = &kvpmu->pmc[pmc_index]; |
| @@ -761,6 +768,7 @@ int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_ba · 124 unmodified lines | ||
| 761 | 768 | } |
| 762 | 769 | } |
| 763 | 770 | |
| 771 | + ctr_idx = array_index_nospec(ctr_idx, RISCV_KVM_MAX_COUNTERS); | |
| 764 | 772 | pmc = &kvpmu->pmc[ctr_idx]; |
| 765 | 773 | pmc->idx = ctr_idx; |
| 766 | 774 | |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/kvm_host.h> |
| 14 | 14 | #include <asm/kvm_mmu.h> |
| 15 | 15 | |
| 16 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 16 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 17 | 17 | KVM_GENERIC_VM_STATS() |
| 18 | 18 | }; |
| 19 | 19 | static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == |
| @@ -62,8 +62,8 @@ do { \ · 61 unmodified lines | ||
| 62 | 62 | * @size: number of elements in array |
| 63 | 63 | */ |
| 64 | 64 | #define array_index_mask_nospec array_index_mask_nospec |
| 65 | −static inline unsigned long array_index_mask_nospec(unsigned long index, | |
| 66 | − unsigned long size) | |
| 65 | +static __always_inline unsigned long array_index_mask_nospec(unsigned long index, | |
| 66 | + unsigned long size) | |
| 67 | 67 | { |
| 68 | 68 | unsigned long mask; |
| 69 | 69 | |
| @@ -710,6 +710,9 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm); · 709 unmodified lines | ||
| 710 | 710 | void kvm_arch_crypto_set_masks(struct kvm *kvm, unsigned long *apm, |
| 711 | 711 | unsigned long *aqm, unsigned long *adm); |
| 712 | 712 | |
| 713 | +#define SIE64_RETURN_NORMAL 0 | |
| 714 | +#define SIE64_RETURN_MCCK 1 | |
| 715 | + | |
| 713 | 716 | int __sie64a(phys_addr_t sie_block_phys, struct kvm_s390_sie_block *sie_block, u64 *rsa, |
| 714 | 717 | unsigned long gasce); |
| 715 | 718 | |
| @@ -62,7 +62,7 @@ struct stack_frame { · 61 unmodified lines | ||
| 62 | 62 | struct { |
| 63 | 63 | unsigned long sie_control_block; |
| 64 | 64 | unsigned long sie_savearea; |
| 65 | − unsigned long sie_reason; | |
| 65 | + unsigned long sie_return; | |
| 66 | 66 | unsigned long sie_flags; |
| 67 | 67 | unsigned long sie_control_block_phys; |
| 68 | 68 | unsigned long sie_guest_asce; |
| @@ -63,7 +63,7 @@ int main(void) · 62 unmodified lines | ||
| 63 | 63 | OFFSET(__SF_EMPTY, stack_frame, empty[0]); |
| 64 | 64 | OFFSET(__SF_SIE_CONTROL, stack_frame, sie_control_block); |
| 65 | 65 | OFFSET(__SF_SIE_SAVEAREA, stack_frame, sie_savearea); |
| 66 | − OFFSET(__SF_SIE_REASON, stack_frame, sie_reason); | |
| 66 | + OFFSET(__SF_SIE_RETURN, stack_frame, sie_return); | |
| 67 | 67 | OFFSET(__SF_SIE_FLAGS, stack_frame, sie_flags); |
| 68 | 68 | OFFSET(__SF_SIE_CONTROL_PHYS, stack_frame, sie_control_block_phys); |
| 69 | 69 | OFFSET(__SF_SIE_GUEST_ASCE, stack_frame, sie_guest_asce); |
| @@ -200,7 +200,7 @@ SYM_FUNC_START(__sie64a) · 199 unmodified lines | ||
| 200 | 200 | stg %r3,__SF_SIE_CONTROL(%r15) # ...and virtual addresses |
| 201 | 201 | stg %r4,__SF_SIE_SAVEAREA(%r15) # save guest register save area |
| 202 | 202 | stg %r5,__SF_SIE_GUEST_ASCE(%r15) # save guest asce |
| 203 | − xc __SF_SIE_REASON(8,%r15),__SF_SIE_REASON(%r15) # reason code = 0 | |
| 203 | + xc __SF_SIE_RETURN(8,%r15),__SF_SIE_RETURN(%r15) # return code = 0 | |
| 204 | 204 | mvc __SF_SIE_FLAGS(8,%r15),__TI_flags(%r14) # copy thread flags |
| 205 | 205 | lmg %r0,%r13,0(%r4) # load guest gprs 0-13 |
| 206 | 206 | mvi __TI_sie(%r14),1 |
| @@ -237,7 +237,7 @@ SYM_INNER_LABEL(sie_exit, SYM_L_GLOBAL) · 30 unmodified lines | ||
| 237 | 237 | xgr %r4,%r4 |
| 238 | 238 | xgr %r5,%r5 |
| 239 | 239 | lmg %r6,%r14,__SF_GPRS(%r15) # restore kernel registers |
| 240 | − lg %r2,__SF_SIE_REASON(%r15) # return exit reason code | |
| 240 | + lg %r2,__SF_SIE_RETURN(%r15) # return sie return code | |
| 241 | 241 | BR_EX %r14 |
| 242 | 242 | SYM_FUNC_END(__sie64a) |
| 243 | 243 | EXPORT_SYMBOL(__sie64a) |
| @@ -271,6 +271,7 @@ SYM_CODE_START(system_call) · 27 unmodified lines | ||
| 271 | 271 | xgr %r9,%r9 |
| 272 | 272 | xgr %r10,%r10 |
| 273 | 273 | xgr %r11,%r11 |
| 274 | + xgr %r12,%r12 | |
| 274 | 275 | la %r2,STACK_FRAME_OVERHEAD(%r15) # pointer to pt_regs |
| 275 | 276 | mvc __PT_R8(64,%r2),__LC_SAVE_AREA(%r13) |
| 276 | 277 | MBEAR %r2,%r13 |
| @@ -407,6 +408,7 @@ SYM_CODE_START(\name) · 130 unmodified lines | ||
| 407 | 408 | xgr %r6,%r6 |
| 408 | 409 | xgr %r7,%r7 |
| 409 | 410 | xgr %r10,%r10 |
| 411 | + xgr %r12,%r12 | |
| 410 | 412 | xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11) |
| 411 | 413 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA(%r13) |
| 412 | 414 | MBEAR %r11,%r13 |
| @@ -496,6 +498,7 @@ SYM_CODE_START(mcck_int_handler) · 83 unmodified lines | ||
| 496 | 498 | xgr %r6,%r6 |
| 497 | 499 | xgr %r7,%r7 |
| 498 | 500 | xgr %r10,%r10 |
| 501 | + xgr %r12,%r12 | |
| 499 | 502 | stmg %r8,%r9,__PT_PSW(%r11) |
| 500 | 503 | xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11) |
| 501 | 504 | xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15) |
| @@ -147,10 +147,8 @@ void noinstr do_io_irq(struct pt_regs *regs) · 146 unmodified lines | ||
| 147 | 147 | bool from_idle; |
| 148 | 148 | |
| 149 | 149 | from_idle = test_and_clear_cpu_flag(CIF_ENABLED_WAIT); |
| 150 | − if (from_idle) { | |
| 150 | + if (from_idle) | |
| 151 | 151 | update_timer_idle(); |
| 152 | − regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | |
| 153 | − } | |
| 154 | 152 | |
| 155 | 153 | irq_enter_rcu(); |
| 156 | 154 | |
| @@ -176,6 +174,9 @@ void noinstr do_io_irq(struct pt_regs *regs) · 19 unmodified lines | ||
| 176 | 174 | |
| 177 | 175 | set_irq_regs(old_regs); |
| 178 | 176 | irqentry_exit(regs, state); |
| 177 | + | |
| 178 | + if (from_idle) | |
| 179 | + regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | |
| 179 | 180 | } |
| 180 | 181 | |
| 181 | 182 | void noinstr do_ext_irq(struct pt_regs *regs) |
| @@ -185,10 +186,8 @@ void noinstr do_ext_irq(struct pt_regs *regs) · 3 unmodified lines | ||
| 185 | 186 | bool from_idle; |
| 186 | 187 | |
| 187 | 188 | from_idle = test_and_clear_cpu_flag(CIF_ENABLED_WAIT); |
| 188 | − if (from_idle) { | |
| 189 | + if (from_idle) | |
| 189 | 190 | update_timer_idle(); |
| 190 | − regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | |
| 191 | − } | |
| 192 | 191 | |
| 193 | 192 | irq_enter_rcu(); |
| 194 | 193 | |
| @@ -210,6 +209,9 @@ void noinstr do_ext_irq(struct pt_regs *regs) · 15 unmodified lines | ||
| 210 | 209 | irq_exit_rcu(); |
| 211 | 210 | set_irq_regs(old_regs); |
| 212 | 211 | irqentry_exit(regs, state); |
| 212 | + | |
| 213 | + if (from_idle) | |
| 214 | + regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | |
| 213 | 215 | } |
| 214 | 216 | |
| 215 | 217 | static void show_msi_interrupt(struct seq_file *p, int irq) |
| @@ -487,8 +487,8 @@ void notrace s390_do_machine_check(struct pt_regs *regs) · 486 unmodified lines | ||
| 487 | 487 | mcck_dam_code = (mci.val & MCIC_SUBCLASS_MASK); |
| 488 | 488 | if (test_cpu_flag(CIF_MCCK_GUEST) && |
| 489 | 489 | (mcck_dam_code & MCCK_CODE_NO_GUEST) != mcck_dam_code) { |
| 490 | − /* Set exit reason code for host's later handling */ | |
| 491 | − *((long *)(regs->gprs[15] + __SF_SIE_REASON)) = -EINTR; | |
| 490 | + /* Set sie return code for host's later handling */ | |
| 491 | + ((struct stack_frame *)regs->gprs[15])->sie_return = SIE64_RETURN_MCCK; | |
| 492 | 492 | } |
| 493 | 493 | clear_cpu_flag(CIF_MCCK_GUEST); |
| 494 | 494 | |
| @@ -1168,6 +1168,7 @@ static void hw_collect_samples(struct perf_event *event, unsigned long *sdbt, · 1167 unmodified lines | ||
| 1168 | 1168 | static void hw_perf_event_update(struct perf_event *event, int flush_all) |
| 1169 | 1169 | { |
| 1170 | 1170 | unsigned long long event_overflow, sampl_overflow, num_sdb; |
| 1171 | + struct cpu_hw_sf *cpuhw = this_cpu_ptr(&cpu_hw_sf); | |
| 1171 | 1172 | struct hw_perf_event *hwc = &event->hw; |
| 1172 | 1173 | union hws_trailer_header prev, new; |
| 1173 | 1174 | struct hws_trailer_entry *te; |
| @@ -1247,8 +1248,11 @@ static void hw_perf_event_update(struct perf_event *event, int flush_all) · 73 unmodified lines | ||
| 1247 | 1248 | * are dropped. |
| 1248 | 1249 | * Slightly increase the interval to avoid hitting this limit. |
| 1249 | 1250 | */ |
| 1250 | − if (event_overflow) | |
| 1251 | + if (event_overflow) { | |
| 1251 | 1252 | SAMPL_RATE(hwc) += DIV_ROUND_UP(SAMPL_RATE(hwc), 10); |
| 1253 | + if (SAMPL_RATE(hwc) > cpuhw->qsi.max_sampl_rate) | |
| 1254 | + SAMPL_RATE(hwc) = cpuhw->qsi.max_sampl_rate; | |
| 1255 | + } | |
| 1252 | 1256 | } |
| 1253 | 1257 | |
| 1254 | 1258 | static inline unsigned long aux_sdb_index(struct aux_buffer *aux, |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | */ |
| 14 | 14 | |
| 15 | 15 | #include <linux/cpufeature.h> |
| 16 | +#include <linux/nospec.h> | |
| 16 | 17 | #include <linux/errno.h> |
| 17 | 18 | #include <linux/sched.h> |
| 18 | 19 | #include <linux/mm.h> |
| @@ -131,8 +132,10 @@ void noinstr __do_syscall(struct pt_regs *regs, int per_trap) · 112 unmodified lines | ||
| 131 | 132 | if (unlikely(test_and_clear_pt_regs_flag(regs, PIF_SYSCALL_RET_SET))) |
| 132 | 133 | goto out; |
| 133 | 134 | regs->gprs[2] = -ENOSYS; |
| 134 | − if (likely(nr < NR_syscalls)) | |
| 135 | + if (likely(nr < NR_syscalls)) { | |
| 136 | + nr = array_index_nospec(nr, NR_syscalls); | |
| 135 | 137 | regs->gprs[2] = sys_call_table[nr](regs); |
| 138 | + } | |
| 136 | 139 | out: |
| 137 | 140 | syscall_exit_to_user_mode(regs); |
| 138 | 141 | } |
| @@ -134,32 +134,6 @@ int dat_set_asce_limit(struct kvm_s390_mmu_cache *mc, union asce *asce, int newt · 133 unmodified lines | ||
| 134 | 134 | return 0; |
| 135 | 135 | } |
| 136 | 136 | |
| 137 | −/** | |
| 138 | − * dat_crstep_xchg() - Exchange a gmap CRSTE with another. | |
| 139 | − * @crstep: Pointer to the CRST entry | |
| 140 | − * @new: Replacement entry. | |
| 141 | − * @gfn: The affected guest address. | |
| 142 | − * @asce: The ASCE of the address space. | |
| 143 | − * | |
| 144 | − * Context: This function is assumed to be called with kvm->mmu_lock held. | |
| 145 | − */ | |
| 146 | −void dat_crstep_xchg(union crste *crstep, union crste new, gfn_t gfn, union asce asce) | |
| 147 | −{ | |
| 148 | − if (crstep->h.i) { | |
| 149 | − WRITE_ONCE(*crstep, new); | |
| 150 | − return; | |
| 151 | − } else if (cpu_has_edat2()) { | |
| 152 | − crdte_crste(crstep, *crstep, new, gfn, asce); | |
| 153 | − return; | |
| 154 | − } | |
| 155 | − | |
| 156 | − if (machine_has_tlb_guest()) | |
| 157 | − idte_crste(crstep, gfn, IDTE_GUEST_ASCE, asce, IDTE_GLOBAL); | |
| 158 | − else | |
| 159 | − idte_crste(crstep, gfn, 0, NULL_ASCE, IDTE_GLOBAL); | |
| 160 | − WRITE_ONCE(*crstep, new); | |
| 161 | −} | |
| 162 | − | |
| 163 | 137 | /** |
| 164 | 138 | * dat_crstep_xchg_atomic() - Atomically exchange a gmap CRSTE with another. |
| 165 | 139 | * @crstep: Pointer to the CRST entry. |
| @@ -175,8 +149,8 @@ void dat_crstep_xchg(union crste *crstep, union crste new, gfn_t gfn, union asce · 9 unmodified lines | ||
| 175 | 149 | * |
| 176 | 150 | * Return: %true if the exchange was successful. |
| 177 | 151 | */ |
| 178 | −bool dat_crstep_xchg_atomic(union crste *crstep, union crste old, union crste new, gfn_t gfn, | |
| 179 | − union asce asce) | |
| 152 | +bool __must_check dat_crstep_xchg_atomic(union crste *crstep, union crste old, union crste new, | |
| 153 | + gfn_t gfn, union asce asce) | |
| 180 | 154 | { |
| 181 | 155 | if (old.h.i) |
| 182 | 156 | return arch_try_cmpxchg((long *)crstep, &old.val, new.val); |
| @@ -292,6 +266,7 @@ static int dat_split_ste(struct kvm_s390_mmu_cache *mc, union pmd *pmdp, gfn_t g · 109 unmodified lines | ||
| 292 | 266 | pt->ptes[i].val = init.val | i * PAGE_SIZE; |
| 293 | 267 | /* No need to take locks as the page table is not installed yet. */ |
| 294 | 268 | pgste_init.prefix_notif = old.s.fc1.prefix_notif; |
| 269 | + pgste_init.vsie_notif = old.s.fc1.vsie_notif; | |
| 295 | 270 | pgste_init.pcl = uses_skeys && init.h.i; |
| 296 | 271 | dat_init_pgstes(pt, pgste_init.val); |
| 297 | 272 | } else { |
| @@ -893,7 +868,8 @@ static long _dat_slot_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct d · 595 unmodified lines | ||
| 893 | 868 | |
| 894 | 869 | /* This table entry needs to be updated. */ |
| 895 | 870 | if (walk->start <= gfn && walk->end >= next) { |
| 896 | − dat_crstep_xchg_atomic(crstep, crste, new_crste, gfn, walk->asce); | |
| 871 | + if (!dat_crstep_xchg_atomic(crstep, crste, new_crste, gfn, walk->asce)) | |
| 872 | + return -EINVAL; | |
| 897 | 873 | /* A lower level table was present, needs to be freed. */ |
| 898 | 874 | if (!crste.h.fc && !crste.h.i) { |
| 899 | 875 | if (is_pmd(crste)) |
| @@ -1021,67 +997,21 @@ bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end) · 121 unmodified lines | ||
| 1021 | 997 | return _dat_walk_gfn_range(start, end, asce, &test_age_ops, 0, NULL) > 0; |
| 1022 | 998 | } |
| 1023 | 999 | |
| 1024 | −int dat_link(struct kvm_s390_mmu_cache *mc, union asce asce, int level, | |
| 1025 | − bool uses_skeys, struct guest_fault *f) | |
| 1026 | −{ | |
| 1027 | − union crste oldval, newval; | |
| 1028 | − union pte newpte, oldpte; | |
| 1029 | − union pgste pgste; | |
| 1030 | − int rc = 0; | |
| 1031 | − | |
| 1032 | − rc = dat_entry_walk(mc, f->gfn, asce, DAT_WALK_ALLOC_CONTINUE, level, &f->crstep, &f->ptep); | |
| 1033 | − if (rc == -EINVAL || rc == -ENOMEM) | |
| 1034 | − return rc; | |
| 1035 | − if (rc) | |
| 1036 | − return -EAGAIN; | |
| 1037 | − | |
| 1038 | − if (WARN_ON_ONCE(unlikely(get_level(f->crstep, f->ptep) > level))) | |
| 1039 | − return -EINVAL; | |
| 1040 | − | |
| 1041 | − if (f->ptep) { | |
| 1042 | − pgste = pgste_get_lock(f->ptep); | |
| 1043 | − oldpte = *f->ptep; | |
| 1044 | − newpte = _pte(f->pfn, f->writable, f->write_attempt | oldpte.s.d, !f->page); | |
| 1045 | − newpte.s.sd = oldpte.s.sd; | |
| 1046 | − oldpte.s.sd = 0; | |
| 1047 | − if (oldpte.val == _PTE_EMPTY.val || oldpte.h.pfra == f->pfn) { | |
| 1048 | − pgste = __dat_ptep_xchg(f->ptep, pgste, newpte, f->gfn, asce, uses_skeys); | |
| 1049 | − if (f->callback) | |
| 1050 | − f->callback(f); | |
| 1051 | − } else { | |
| 1052 | − rc = -EAGAIN; | |
| 1053 | − } | |
| 1054 | − pgste_set_unlock(f->ptep, pgste); | |
| 1055 | − } else { | |
| 1056 | − oldval = READ_ONCE(*f->crstep); | |
| 1057 | − newval = _crste_fc1(f->pfn, oldval.h.tt, f->writable, | |
| 1058 | − f->write_attempt | oldval.s.fc1.d); | |
| 1059 | − newval.s.fc1.sd = oldval.s.fc1.sd; | |
| 1060 | − if (oldval.val != _CRSTE_EMPTY(oldval.h.tt).val && | |
| 1061 | − crste_origin_large(oldval) != crste_origin_large(newval)) | |
| 1062 | − return -EAGAIN; | |
| 1063 | − if (!dat_crstep_xchg_atomic(f->crstep, oldval, newval, f->gfn, asce)) | |
| 1064 | − return -EAGAIN; | |
| 1065 | − if (f->callback) | |
| 1066 | − f->callback(f); | |
| 1067 | − } | |
| 1068 | − | |
| 1069 | − return rc; | |
| 1070 | −} | |
| 1071 | − | |
| 1072 | 1000 | static long dat_set_pn_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk) |
| 1073 | 1001 | { |
| 1074 | − union crste crste = READ_ONCE(*crstep); | |
| 1002 | + union crste newcrste, oldcrste; | |
| 1075 | 1003 | int *n = walk->priv; |
| 1076 | 1004 | |
| 1077 | − if (!crste.h.fc || crste.h.i || crste.h.p) | |
| 1078 | − return 0; | |
| 1079 | − | |
| 1005 | + do { | |
| 1006 | + oldcrste = READ_ONCE(*crstep); | |
| 1007 | + if (!oldcrste.h.fc || oldcrste.h.i || oldcrste.h.p) | |
| 1008 | + return 0; | |
| 1009 | + if (oldcrste.s.fc1.prefix_notif) | |
| 1010 | + break; | |
| 1011 | + newcrste = oldcrste; | |
| 1012 | + newcrste.s.fc1.prefix_notif = 1; | |
| 1013 | + } while (!dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, walk->asce)); | |
| 1080 | 1014 | *n = 2; |
| 1081 | − if (crste.s.fc1.prefix_notif) | |
| 1082 | − return 0; | |
| 1083 | − crste.s.fc1.prefix_notif = 1; | |
| 1084 | − dat_crstep_xchg(crstep, crste, gfn, walk->asce); | |
| 1085 | 1015 | return 0; |
| 1086 | 1016 | } |
| 1087 | 1017 | |
| @@ -160,14 +160,14 @@ union pmd { · 159 unmodified lines | ||
| 160 | 160 | unsigned long :44; /* HW */ |
| 161 | 161 | unsigned long : 3; /* Unused */ |
| 162 | 162 | unsigned long : 1; /* HW */ |
| 163 | + unsigned long s : 1; /* Special */ | |
| 163 | 164 | unsigned long w : 1; /* Writable soft-bit */ |
| 164 | 165 | unsigned long r : 1; /* Readable soft-bit */ |
| 165 | 166 | unsigned long d : 1; /* Dirty */ |
| 166 | 167 | unsigned long y : 1; /* Young */ |
| 167 | − unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | |
| 168 | 168 | unsigned long : 3; /* HW */ |
| 169 | + unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | |
| 169 | 170 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ |
| 170 | − unsigned long : 1; /* Unused */ | |
| 171 | 171 | unsigned long : 4; /* HW */ |
| 172 | 172 | unsigned long sd : 1; /* Soft-Dirty */ |
| 173 | 173 | unsigned long pr : 1; /* Present */ |
| @@ -183,14 +183,14 @@ union pud { · 9 unmodified lines | ||
| 183 | 183 | unsigned long :33; /* HW */ |
| 184 | 184 | unsigned long :14; /* Unused */ |
| 185 | 185 | unsigned long : 1; /* HW */ |
| 186 | + unsigned long s : 1; /* Special */ | |
| 186 | 187 | unsigned long w : 1; /* Writable soft-bit */ |
| 187 | 188 | unsigned long r : 1; /* Readable soft-bit */ |
| 188 | 189 | unsigned long d : 1; /* Dirty */ |
| 189 | 190 | unsigned long y : 1; /* Young */ |
| 190 | − unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | |
| 191 | 191 | unsigned long : 3; /* HW */ |
| 192 | + unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | |
| 192 | 193 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ |
| 193 | − unsigned long : 1; /* Unused */ | |
| 194 | 194 | unsigned long : 4; /* HW */ |
| 195 | 195 | unsigned long sd : 1; /* Soft-Dirty */ |
| 196 | 196 | unsigned long pr : 1; /* Present */ |
| @@ -254,14 +254,14 @@ union crste { · 57 unmodified lines | ||
| 254 | 254 | struct { |
| 255 | 255 | unsigned long :47; |
| 256 | 256 | unsigned long : 1; /* HW (should be 0) */ |
| 257 | + unsigned long s : 1; /* Special */ | |
| 257 | 258 | unsigned long w : 1; /* Writable */ |
| 258 | 259 | unsigned long r : 1; /* Readable */ |
| 259 | 260 | unsigned long d : 1; /* Dirty */ |
| 260 | 261 | unsigned long y : 1; /* Young */ |
| 261 | − unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | |
| 262 | 262 | unsigned long : 3; /* HW */ |
| 263 | + unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | |
| 263 | 264 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ |
| 264 | − unsigned long : 1; | |
| 265 | 265 | unsigned long : 4; /* HW */ |
| 266 | 266 | unsigned long sd : 1; /* Soft-Dirty */ |
| 267 | 267 | unsigned long pr : 1; /* Present */ |
| @@ -540,8 +540,6 @@ int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gf · 272 unmodified lines | ||
| 540 | 540 | u16 type, u16 param); |
| 541 | 541 | int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn); |
| 542 | 542 | bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end); |
| 543 | −int dat_link(struct kvm_s390_mmu_cache *mc, union asce asce, int level, | |
| 544 | − bool uses_skeys, struct guest_fault *f); | |
| 545 | 543 | |
| 546 | 544 | int dat_perform_essa(union asce asce, gfn_t gfn, int orc, union essa_state *state, bool *dirty); |
| 547 | 545 | long dat_reset_cmma(union asce asce, gfn_t start_gfn); |
| @@ -938,11 +936,14 @@ static inline bool dat_pudp_xchg_atomic(union pud *pudp, union pud old, union pu · 390 unmodified lines | ||
| 938 | 936 | return dat_crstep_xchg_atomic(_CRSTEP(pudp), _CRSTE(old), _CRSTE(new), gfn, asce); |
| 939 | 937 | } |
| 940 | 938 | |
| 941 | −static inline void dat_crstep_clear(union crste *crstep, gfn_t gfn, union asce asce) | |
| 939 | +static inline union crste dat_crstep_clear_atomic(union crste *crstep, gfn_t gfn, union asce asce) | |
| 942 | 940 | { |
| 943 | − union crste newcrste = _CRSTE_EMPTY(crstep->h.tt); | |
| 941 | + union crste oldcrste, empty = _CRSTE_EMPTY(crstep->h.tt); | |
| 944 | 942 | |
| 945 | − dat_crstep_xchg(crstep, newcrste, gfn, asce); | |
| 943 | + do { | |
| 944 | + oldcrste = READ_ONCE(*crstep); | |
| 945 | + } while (!dat_crstep_xchg_atomic(crstep, oldcrste, empty, gfn, asce)); | |
| 946 | + return oldcrste; | |
| 946 | 947 | } |
| 947 | 948 | |
| 948 | 949 | static inline int get_level(union crste *crstep, union pte *ptep) |
| @@ -1434,17 +1434,27 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union · 1433 unmodified lines | ||
| 1434 | 1434 | if (rc) |
| 1435 | 1435 | return rc; |
| 1436 | 1436 | |
| 1437 | − pgste = pgste_get_lock(ptep_h); | |
| 1438 | − newpte = _pte(f->pfn, f->writable, !p, 0); | |
| 1439 | − newpte.s.d |= ptep->s.d; | |
| 1440 | − newpte.s.sd |= ptep->s.sd; | |
| 1441 | − newpte.h.p &= ptep->h.p; | |
| 1442 | − pgste = _gmap_ptep_xchg(sg->parent, ptep_h, newpte, pgste, f->gfn, false); | |
| 1443 | − pgste.vsie_notif = 1; | |
| 1437 | + if (!pgste_get_trylock(ptep_h, &pgste)) | |
| 1438 | + return -EAGAIN; | |
| 1439 | + newpte = _pte(f->pfn, f->writable, !p, ptep_h->s.s); | |
| 1440 | + newpte.s.d |= ptep_h->s.d; | |
| 1441 | + newpte.s.sd |= ptep_h->s.sd; | |
| 1442 | + newpte.h.p &= ptep_h->h.p; | |
| 1443 | + if (!newpte.h.p && !f->writable) { | |
| 1444 | + rc = -EOPNOTSUPP; | |
| 1445 | + } else { | |
| 1446 | + pgste = _gmap_ptep_xchg(sg->parent, ptep_h, newpte, pgste, f->gfn, false); | |
| 1447 | + pgste.vsie_notif = 1; | |
| 1448 | + } | |
| 1444 | 1449 | pgste_set_unlock(ptep_h, pgste); |
| 1450 | + if (rc) | |
| 1451 | + return rc; | |
| 1452 | + if (!sg->parent) | |
| 1453 | + return -EAGAIN; | |
| 1445 | 1454 | |
| 1446 | 1455 | newpte = _pte(f->pfn, 0, !p, 0); |
| 1447 | − pgste = pgste_get_lock(ptep); | |
| 1456 | + if (!pgste_get_trylock(ptep, &pgste)) | |
| 1457 | + return -EAGAIN; | |
| 1448 | 1458 | pgste = __dat_ptep_xchg(ptep, pgste, newpte, gpa_to_gfn(raddr), sg->asce, uses_skeys(sg)); |
| 1449 | 1459 | pgste_set_unlock(ptep, pgste); |
| 1450 | 1460 | |
| @@ -1454,7 +1464,7 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union · 3 unmodified lines | ||
| 1454 | 1464 | static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, union crste *table, |
| 1455 | 1465 | struct guest_fault *f, bool p) |
| 1456 | 1466 | { |
| 1457 | − union crste newcrste; | |
| 1467 | + union crste newcrste, oldcrste; | |
| 1458 | 1468 | gfn_t gfn; |
| 1459 | 1469 | int rc; |
| 1460 | 1470 | |
| @@ -1467,16 +1477,28 @@ static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, uni · 6 unmodified lines | ||
| 1467 | 1477 | if (rc) |
| 1468 | 1478 | return rc; |
| 1469 | 1479 | |
| 1470 | − newcrste = _crste_fc1(f->pfn, host->h.tt, f->writable, !p); | |
| 1471 | − newcrste.s.fc1.d |= host->s.fc1.d; | |
| 1472 | − newcrste.s.fc1.sd |= host->s.fc1.sd; | |
| 1473 | − newcrste.h.p &= host->h.p; | |
| 1474 | − newcrste.s.fc1.vsie_notif = 1; | |
| 1475 | − newcrste.s.fc1.prefix_notif = host->s.fc1.prefix_notif; | |
| 1476 | − _gmap_crstep_xchg(sg->parent, host, newcrste, f->gfn, false); | |
| 1480 | + do { | |
| 1481 | + /* _gmap_crstep_xchg_atomic() could have unshadowed this shadow gmap */ | |
| 1482 | + if (!sg->parent) | |
| 1483 | + return -EAGAIN; | |
| 1484 | + oldcrste = READ_ONCE(*host); | |
| 1485 | + newcrste = _crste_fc1(f->pfn, oldcrste.h.tt, f->writable, !p); | |
| 1486 | + newcrste.s.fc1.d |= oldcrste.s.fc1.d; | |
| 1487 | + newcrste.s.fc1.sd |= oldcrste.s.fc1.sd; | |
| 1488 | + newcrste.h.p &= oldcrste.h.p; | |
| 1489 | + newcrste.s.fc1.vsie_notif = 1; | |
| 1490 | + newcrste.s.fc1.prefix_notif = oldcrste.s.fc1.prefix_notif; | |
| 1491 | + newcrste.s.fc1.s = oldcrste.s.fc1.s; | |
| 1492 | + if (!newcrste.h.p && !f->writable) | |
| 1493 | + return -EOPNOTSUPP; | |
| 1494 | + } while (!_gmap_crstep_xchg_atomic(sg->parent, host, oldcrste, newcrste, f->gfn, false)); | |
| 1495 | + if (!sg->parent) | |
| 1496 | + return -EAGAIN; | |
| 1477 | 1497 | |
| 1478 | − newcrste = _crste_fc1(f->pfn, host->h.tt, 0, !p); | |
| 1479 | − dat_crstep_xchg(table, newcrste, gpa_to_gfn(raddr), sg->asce); | |
| 1498 | + newcrste = _crste_fc1(f->pfn, oldcrste.h.tt, 0, !p); | |
| 1499 | + gfn = gpa_to_gfn(raddr); | |
| 1500 | + while (!dat_crstep_xchg_atomic(table, READ_ONCE(*table), newcrste, gfn, sg->asce)) | |
| 1501 | + ; | |
| 1480 | 1502 | return 0; |
| 1481 | 1503 | } |
| 1482 | 1504 | |
| @@ -1500,21 +1522,31 @@ static int _gaccess_do_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *sg, · 17 unmodified lines | ||
| 1500 | 1522 | if (rc) |
| 1501 | 1523 | return rc; |
| 1502 | 1524 | |
| 1503 | − /* A race occourred. The shadow mapping is already valid, nothing to do */ | |
| 1504 | − if ((ptep && !ptep->h.i) || (!ptep && crste_leaf(*table))) | |
| 1525 | + /* A race occurred. The shadow mapping is already valid, nothing to do */ | |
| 1526 | + if ((ptep && !ptep->h.i && ptep->h.p == w->p) || | |
| 1527 | + (!ptep && crste_leaf(*table) && !table->h.i && table->h.p == w->p)) | |
| 1505 | 1528 | return 0; |
| 1506 | 1529 | |
| 1507 | 1530 | gl = get_level(table, ptep); |
| 1508 | 1531 | |
| 1532 | + /* In case of a real address space */ | |
| 1533 | + if (w->level <= LEVEL_MEM) { | |
| 1534 | + l = TABLE_TYPE_PAGE_TABLE; | |
| 1535 | + hl = TABLE_TYPE_REGION1; | |
| 1536 | + goto real_address_space; | |
| 1537 | + } | |
| 1538 | + | |
| 1509 | 1539 | /* |
| 1510 | 1540 | * Skip levels that are already protected. For each level, protect |
| 1511 | 1541 | * only the page containing the entry, not the whole table. |
| 1512 | 1542 | */ |
| 1513 | 1543 | for (i = gl ; i >= w->level; i--) { |
| 1514 | − rc = gmap_protect_rmap(mc, sg, entries[i - 1].gfn, gpa_to_gfn(saddr), | |
| 1515 | − entries[i - 1].pfn, i, entries[i - 1].writable); | |
| 1544 | + rc = gmap_protect_rmap(mc, sg, entries[i].gfn, gpa_to_gfn(saddr), | |
| 1545 | + entries[i].pfn, i + 1, entries[i].writable); | |
| 1516 | 1546 | if (rc) |
| 1517 | 1547 | return rc; |
| 1548 | + if (!sg->parent) | |
| 1549 | + return -EAGAIN; | |
| 1518 | 1550 | } |
| 1519 | 1551 | |
| 1520 | 1552 | rc = dat_entry_walk(NULL, entries[LEVEL_MEM].gfn, sg->parent->asce, DAT_WALK_LEAF, |
| @@ -1526,6 +1558,7 @@ static int _gaccess_do_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *sg, · 5 unmodified lines | ||
| 1526 | 1558 | /* Get the smallest granularity */ |
| 1527 | 1559 | l = min3(gl, hl, w->level); |
| 1528 | 1560 | |
| 1561 | +real_address_space: | |
| 1529 | 1562 | flags = DAT_WALK_SPLIT_ALLOC | (uses_skeys(sg->parent) ? DAT_WALK_USES_SKEYS : 0); |
| 1530 | 1563 | /* If necessary, create the shadow mapping */ |
| 1531 | 1564 | if (l < gl) { |
| @@ -313,13 +313,16 @@ static long gmap_clear_young_crste(union crste *crstep, gfn_t gfn, gfn_t end, st · 312 unmodified lines | ||
| 313 | 313 | struct clear_young_pte_priv *priv = walk->priv; |
| 314 | 314 | union crste crste, new; |
| 315 | 315 | |
| 316 | − crste = READ_ONCE(*crstep); | |
| 316 | + do { | |
| 317 | + crste = READ_ONCE(*crstep); | |
| 318 | + | |
| 319 | + if (!crste.h.fc) | |
| 320 | + return 0; | |
| 321 | + if (!crste.s.fc1.y && crste.h.i) | |
| 322 | + return 0; | |
| 323 | + if (crste_prefix(crste) && !gmap_mkold_prefix(priv->gmap, gfn, end)) | |
| 324 | + break; | |
| 317 | 325 | |
| 318 | − if (!crste.h.fc) | |
| 319 | − return 0; | |
| 320 | − if (!crste.s.fc1.y && crste.h.i) | |
| 321 | − return 0; | |
| 322 | − if (!crste_prefix(crste) || gmap_mkold_prefix(priv->gmap, gfn, end)) { | |
| 323 | 326 | new = crste; |
| 324 | 327 | new.h.i = 1; |
| 325 | 328 | new.s.fc1.y = 0; |
| @@ -328,8 +331,8 @@ static long gmap_clear_young_crste(union crste *crstep, gfn_t gfn, gfn_t end, st · 2 unmodified lines | ||
| 328 | 331 | folio_set_dirty(phys_to_folio(crste_origin_large(crste))); |
| 329 | 332 | new.s.fc1.d = 0; |
| 330 | 333 | new.h.p = 1; |
| 331 | − dat_crstep_xchg(crstep, new, gfn, walk->asce); | |
| 332 | − } | |
| 334 | + } while (!dat_crstep_xchg_atomic(crstep, crste, new, gfn, walk->asce)); | |
| 335 | + | |
| 333 | 336 | priv->young = 1; |
| 334 | 337 | return 0; |
| 335 | 338 | } |
| @@ -391,14 +394,18 @@ static long _gmap_unmap_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct · 55 unmodified lines | ||
| 391 | 394 | { |
| 392 | 395 | struct gmap_unmap_priv *priv = walk->priv; |
| 393 | 396 | struct folio *folio = NULL; |
| 397 | + union crste old = *crstep; | |
| 394 | 398 | |
| 395 | − if (crstep->h.fc) { | |
| 396 | − if (crstep->s.fc1.pr && test_bit(GMAP_FLAG_EXPORT_ON_UNMAP, &priv->gmap->flags)) | |
| 397 | − folio = phys_to_folio(crste_origin_large(*crstep)); | |
| 398 | − gmap_crstep_xchg(priv->gmap, crstep, _CRSTE_EMPTY(crstep->h.tt), gfn); | |
| 399 | − if (folio) | |
| 400 | − uv_convert_from_secure_folio(folio); | |
| 401 | − } | |
| 399 | + if (!old.h.fc) | |
| 400 | + return 0; | |
| 401 | + | |
| 402 | + if (old.s.fc1.pr && test_bit(GMAP_FLAG_EXPORT_ON_UNMAP, &priv->gmap->flags)) | |
| 403 | + folio = phys_to_folio(crste_origin_large(old)); | |
| 404 | + /* No races should happen because kvm->mmu_lock is held in write mode */ | |
| 405 | + KVM_BUG_ON(!gmap_crstep_xchg_atomic(priv->gmap, crstep, old, _CRSTE_EMPTY(old.h.tt), gfn), | |
| 406 | + priv->gmap->kvm); | |
| 407 | + if (folio) | |
| 408 | + uv_convert_from_secure_folio(folio); | |
| 402 | 409 | |
| 403 | 410 | return 0; |
| 404 | 411 | } |
| @@ -474,23 +481,24 @@ static long _crste_test_and_clear_softdirty(union crste *table, gfn_t gfn, gfn_t · 69 unmodified lines | ||
| 474 | 481 | |
| 475 | 482 | if (fatal_signal_pending(current)) |
| 476 | 483 | return 1; |
| 477 | − crste = READ_ONCE(*table); | |
| 478 | − if (!crste.h.fc) | |
| 479 | − return 0; | |
| 480 | − if (crste.h.p && !crste.s.fc1.sd) | |
| 481 | − return 0; | |
| 484 | + do { | |
| 485 | + crste = READ_ONCE(*table); | |
| 486 | + if (!crste.h.fc) | |
| 487 | + return 0; | |
| 488 | + if (crste.h.p && !crste.s.fc1.sd) | |
| 489 | + return 0; | |
| 482 | 490 | |
| 483 | − /* | |
| 484 | − * If this large page contains one or more prefixes of vCPUs that are | |
| 485 | − * currently running, do not reset the protection, leave it marked as | |
| 486 | − * dirty. | |
| 487 | − */ | |
| 488 | − if (!crste.s.fc1.prefix_notif || gmap_mkold_prefix(gmap, gfn, end)) { | |
| 491 | + /* | |
| 492 | + * If this large page contains one or more prefixes of vCPUs that are | |
| 493 | + * currently running, do not reset the protection, leave it marked as | |
| 494 | + * dirty. | |
| 495 | + */ | |
| 496 | + if (crste.s.fc1.prefix_notif && !gmap_mkold_prefix(gmap, gfn, end)) | |
| 497 | + break; | |
| 489 | 498 | new = crste; |
| 490 | 499 | new.h.p = 1; |
| 491 | 500 | new.s.fc1.sd = 0; |
| 492 | − gmap_crstep_xchg(gmap, table, new, gfn); | |
| 493 | − } | |
| 501 | + } while (!gmap_crstep_xchg_atomic(gmap, table, crste, new, gfn)); | |
| 494 | 502 | |
| 495 | 503 | for ( ; gfn < end; gfn++) |
| 496 | 504 | mark_page_dirty(gmap->kvm, gfn); |
| @@ -511,7 +519,7 @@ void gmap_sync_dirty_log(struct gmap *gmap, gfn_t start, gfn_t end) · 14 unmodified lines | ||
| 511 | 519 | _dat_walk_gfn_range(start, end, gmap->asce, &walk_ops, 0, gmap); |
| 512 | 520 | } |
| 513 | 521 | |
| 514 | −static int gmap_handle_minor_crste_fault(union asce asce, struct guest_fault *f) | |
| 522 | +static int gmap_handle_minor_crste_fault(struct gmap *gmap, struct guest_fault *f) | |
| 515 | 523 | { |
| 516 | 524 | union crste newcrste, oldcrste = READ_ONCE(*f->crstep); |
| 517 | 525 | |
| @@ -536,10 +544,8 @@ static int gmap_handle_minor_crste_fault(union asce asce, struct guest_fault *f) · 18 unmodified lines | ||
| 536 | 544 | newcrste.s.fc1.d = 1; |
| 537 | 545 | newcrste.s.fc1.sd = 1; |
| 538 | 546 | } |
| 539 | − if (!oldcrste.s.fc1.d && newcrste.s.fc1.d) | |
| 540 | − SetPageDirty(phys_to_page(crste_origin_large(newcrste))); | |
| 541 | 547 | /* In case of races, let the slow path deal with it. */ |
| 542 | − return !dat_crstep_xchg_atomic(f->crstep, oldcrste, newcrste, f->gfn, asce); | |
| 548 | + return !gmap_crstep_xchg_atomic(gmap, f->crstep, oldcrste, newcrste, f->gfn); | |
| 543 | 549 | } |
| 544 | 550 | /* Trying to write on a read-only page, let the slow path deal with it. */ |
| 545 | 551 | return 1; |
| @@ -568,8 +574,6 @@ static int _gmap_handle_minor_pte_fault(struct gmap *gmap, union pgste *pgste, · 22 unmodified lines | ||
| 568 | 574 | newpte.s.d = 1; |
| 569 | 575 | newpte.s.sd = 1; |
| 570 | 576 | } |
| 571 | − if (!oldpte.s.d && newpte.s.d) | |
| 572 | − SetPageDirty(pfn_to_page(newpte.h.pfra)); | |
| 573 | 577 | *pgste = gmap_ptep_xchg(gmap, f->ptep, newpte, *pgste, f->gfn); |
| 574 | 578 | |
| 575 | 579 | return 0; |
| @@ -606,7 +610,7 @@ int gmap_try_fixup_minor(struct gmap *gmap, struct guest_fault *fault) · 30 unmodified lines | ||
| 606 | 610 | fault->callback(fault); |
| 607 | 611 | pgste_set_unlock(fault->ptep, pgste); |
| 608 | 612 | } else { |
| 609 | − rc = gmap_handle_minor_crste_fault(gmap->asce, fault); | |
| 613 | + rc = gmap_handle_minor_crste_fault(gmap, fault); | |
| 610 | 614 | if (!rc && fault->callback) |
| 611 | 615 | fault->callback(fault); |
| 612 | 616 | } |
| @@ -623,10 +627,61 @@ static inline bool gmap_1m_allowed(struct gmap *gmap, gfn_t gfn) · 10 unmodified lines | ||
| 623 | 627 | return test_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &gmap->flags); |
| 624 | 628 | } |
| 625 | 629 | |
| 630 | +static int _gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, int level, | |
| 631 | + struct guest_fault *f) | |
| 632 | +{ | |
| 633 | + union crste oldval, newval; | |
| 634 | + union pte newpte, oldpte; | |
| 635 | + union pgste pgste; | |
| 636 | + int rc = 0; | |
| 637 | + | |
| 638 | + rc = dat_entry_walk(mc, f->gfn, gmap->asce, DAT_WALK_ALLOC_CONTINUE, level, | |
| 639 | + &f->crstep, &f->ptep); | |
| 640 | + if (rc == -ENOMEM) | |
| 641 | + return rc; | |
| 642 | + if (KVM_BUG_ON(rc == -EINVAL, gmap->kvm)) | |
| 643 | + return rc; | |
| 644 | + if (rc) | |
| 645 | + return -EAGAIN; | |
| 646 | + if (KVM_BUG_ON(get_level(f->crstep, f->ptep) > level, gmap->kvm)) | |
| 647 | + return -EINVAL; | |
| 648 | + | |
| 649 | + if (f->ptep) { | |
| 650 | + pgste = pgste_get_lock(f->ptep); | |
| 651 | + oldpte = *f->ptep; | |
| 652 | + newpte = _pte(f->pfn, f->writable, f->write_attempt | oldpte.s.d, !f->page); | |
| 653 | + newpte.s.sd = oldpte.s.sd; | |
| 654 | + oldpte.s.sd = 0; | |
| 655 | + if (oldpte.val == _PTE_EMPTY.val || oldpte.h.pfra == f->pfn) { | |
| 656 | + pgste = gmap_ptep_xchg(gmap, f->ptep, newpte, pgste, f->gfn); | |
| 657 | + if (f->callback) | |
| 658 | + f->callback(f); | |
| 659 | + } else { | |
| 660 | + rc = -EAGAIN; | |
| 661 | + } | |
| 662 | + pgste_set_unlock(f->ptep, pgste); | |
| 663 | + } else { | |
| 664 | + do { | |
| 665 | + oldval = READ_ONCE(*f->crstep); | |
| 666 | + newval = _crste_fc1(f->pfn, oldval.h.tt, f->writable, | |
| 667 | + f->write_attempt | oldval.s.fc1.d); | |
| 668 | + newval.s.fc1.s = !f->page; | |
| 669 | + newval.s.fc1.sd = oldval.s.fc1.sd; | |
| 670 | + if (oldval.val != _CRSTE_EMPTY(oldval.h.tt).val && | |
| 671 | + crste_origin_large(oldval) != crste_origin_large(newval)) | |
| 672 | + return -EAGAIN; | |
| 673 | + } while (!gmap_crstep_xchg_atomic(gmap, f->crstep, oldval, newval, f->gfn)); | |
| 674 | + if (f->callback) | |
| 675 | + f->callback(f); | |
| 676 | + } | |
| 677 | + | |
| 678 | + return rc; | |
| 679 | +} | |
| 680 | + | |
| 626 | 681 | int gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, struct guest_fault *f) |
| 627 | 682 | { |
| 628 | 683 | unsigned int order; |
| 629 | − int rc, level; | |
| 684 | + int level; | |
| 630 | 685 | |
| 631 | 686 | lockdep_assert_held(&gmap->kvm->mmu_lock); |
| 632 | 687 | |
| @@ -638,16 +693,14 @@ int gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, struct guest_fau · 5 unmodified lines | ||
| 638 | 693 | else if (order >= get_order(_SEGMENT_SIZE) && gmap_1m_allowed(gmap, f->gfn)) |
| 639 | 694 | level = TABLE_TYPE_SEGMENT; |
| 640 | 695 | } |
| 641 | − rc = dat_link(mc, gmap->asce, level, uses_skeys(gmap), f); | |
| 642 | − KVM_BUG_ON(rc == -EINVAL, gmap->kvm); | |
| 643 | − return rc; | |
| 696 | + return _gmap_link(mc, gmap, level, f); | |
| 644 | 697 | } |
| 645 | 698 | |
| 646 | 699 | static int gmap_ucas_map_one(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, |
| 647 | 700 | gfn_t p_gfn, gfn_t c_gfn, bool force_alloc) |
| 648 | 701 | { |
| 702 | + union crste newcrste, oldcrste; | |
| 649 | 703 | struct page_table *pt; |
| 650 | − union crste newcrste; | |
| 651 | 704 | union crste *crstep; |
| 652 | 705 | union pte *ptep; |
| 653 | 706 | int rc; |
| @@ -673,7 +726,11 @@ static int gmap_ucas_map_one(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, · 19 unmodified lines | ||
| 673 | 726 | &crstep, &ptep); |
| 674 | 727 | if (rc) |
| 675 | 728 | return rc; |
| 676 | − dat_crstep_xchg(crstep, newcrste, c_gfn, gmap->asce); | |
| 729 | + do { | |
| 730 | + oldcrste = READ_ONCE(*crstep); | |
| 731 | + if (oldcrste.val == newcrste.val) | |
| 732 | + break; | |
| 733 | + } while (!dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, c_gfn, gmap->asce)); | |
| 677 | 734 | return 0; |
| 678 | 735 | } |
| 679 | 736 | |
| @@ -777,8 +834,10 @@ static void gmap_ucas_unmap_one(struct gmap *gmap, gfn_t c_gfn) · 97 unmodified lines | ||
| 777 | 834 | int rc; |
| 778 | 835 | |
| 779 | 836 | rc = dat_entry_walk(NULL, c_gfn, gmap->asce, 0, TABLE_TYPE_SEGMENT, &crstep, &ptep); |
| 780 | − if (!rc) | |
| 781 | − dat_crstep_xchg(crstep, _PMD_EMPTY, c_gfn, gmap->asce); | |
| 837 | + if (rc) | |
| 838 | + return; | |
| 839 | + while (!dat_crstep_xchg_atomic(crstep, READ_ONCE(*crstep), _PMD_EMPTY, c_gfn, gmap->asce)) | |
| 840 | + ; | |
| 782 | 841 | } |
| 783 | 842 | |
| 784 | 843 | void gmap_ucas_unmap(struct gmap *gmap, gfn_t c_gfn, unsigned long count) |
| @@ -1017,8 +1076,8 @@ static void gmap_unshadow_level(struct gmap *sg, gfn_t r_gfn, int level) · 232 unmodified lines | ||
| 1017 | 1076 | dat_ptep_xchg(ptep, _PTE_EMPTY, r_gfn, sg->asce, uses_skeys(sg)); |
| 1018 | 1077 | return; |
| 1019 | 1078 | } |
| 1020 | − crste = READ_ONCE(*crstep); | |
| 1021 | − dat_crstep_clear(crstep, r_gfn, sg->asce); | |
| 1079 | + | |
| 1080 | + crste = dat_crstep_clear_atomic(crstep, r_gfn, sg->asce); | |
| 1022 | 1081 | if (crste_leaf(crste) || crste.h.i) |
| 1023 | 1082 | return; |
| 1024 | 1083 | if (is_pmd(crste)) |
| @@ -1101,6 +1160,7 @@ struct gmap_protect_asce_top_level { · 76 unmodified lines | ||
| 1101 | 1160 | static inline int __gmap_protect_asce_top_level(struct kvm_s390_mmu_cache *mc, struct gmap *sg, |
| 1102 | 1161 | struct gmap_protect_asce_top_level *context) |
| 1103 | 1162 | { |
| 1163 | + struct gmap *parent; | |
| 1104 | 1164 | int rc, i; |
| 1105 | 1165 | |
| 1106 | 1166 | guard(write_lock)(&sg->kvm->mmu_lock); |
| @@ -1108,7 +1168,12 @@ static inline int __gmap_protect_asce_top_level(struct kvm_s390_mmu_cache *mc, s · 1 unmodified line | ||
| 1108 | 1168 | if (kvm_s390_array_needs_retry_safe(sg->kvm, context->seq, context->f)) |
| 1109 | 1169 | return -EAGAIN; |
| 1110 | 1170 | |
| 1111 | − scoped_guard(spinlock, &sg->parent->children_lock) { | |
| 1171 | + parent = READ_ONCE(sg->parent); | |
| 1172 | + if (!parent) | |
| 1173 | + return -EAGAIN; | |
| 1174 | + scoped_guard(spinlock, &parent->children_lock) { | |
| 1175 | + if (READ_ONCE(sg->parent) != parent) | |
| 1176 | + return -EAGAIN; | |
| 1112 | 1177 | for (i = 0; i < CRST_TABLE_PAGES; i++) { |
| 1113 | 1178 | if (!context->f[i].valid) |
| 1114 | 1179 | continue; |
| @@ -1191,6 +1256,9 @@ struct gmap *gmap_create_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *pare · 76 unmodified lines | ||
| 1191 | 1256 | struct gmap *sg, *new; |
| 1192 | 1257 | int rc; |
| 1193 | 1258 | |
| 1259 | + if (WARN_ON(!parent)) | |
| 1260 | + return ERR_PTR(-EINVAL); | |
| 1261 | + | |
| 1194 | 1262 | scoped_guard(spinlock, &parent->children_lock) { |
| 1195 | 1263 | sg = gmap_find_shadow(parent, asce, edat_level); |
| 1196 | 1264 | if (sg) { |
| @@ -185,6 +185,8 @@ static inline union pgste _gmap_ptep_xchg(struct gmap *gmap, union pte *ptep, un · 184 unmodified lines | ||
| 185 | 185 | else |
| 186 | 186 | _gmap_handle_vsie_unshadow_event(gmap, gfn); |
| 187 | 187 | } |
| 188 | + if (!ptep->s.d && newpte.s.d && !newpte.s.s) | |
| 189 | + SetPageDirty(pfn_to_page(newpte.h.pfra)); | |
| 188 | 190 | return __dat_ptep_xchg(ptep, pgste, newpte, gfn, gmap->asce, uses_skeys(gmap)); |
| 189 | 191 | } |
| 190 | 192 | |
| @@ -194,35 +196,42 @@ static inline union pgste gmap_ptep_xchg(struct gmap *gmap, union pte *ptep, uni · 3 unmodified lines | ||
| 194 | 196 | return _gmap_ptep_xchg(gmap, ptep, newpte, pgste, gfn, true); |
| 195 | 197 | } |
| 196 | 198 | |
| 197 | −static inline void _gmap_crstep_xchg(struct gmap *gmap, union crste *crstep, union crste ne, | |
| 198 | − gfn_t gfn, bool needs_lock) | |
| 199 | +static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, union crste *crstep, | |
| 200 | + union crste oldcrste, union crste newcrste, | |
| 201 | + gfn_t gfn, bool needs_lock) | |
| 199 | 202 | { |
| 200 | − unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11); | |
| 203 | + unsigned long align = is_pmd(newcrste) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES; | |
| 204 | + | |
| 205 | + if (KVM_BUG_ON(crstep->h.tt != oldcrste.h.tt || newcrste.h.tt != oldcrste.h.tt, gmap->kvm)) | |
| 206 | + return true; | |
| 201 | 207 | |
| 202 | 208 | lockdep_assert_held(&gmap->kvm->mmu_lock); |
| 203 | 209 | if (!needs_lock) |
| 204 | 210 | lockdep_assert_held(&gmap->children_lock); |
| 205 | 211 | |
| 206 | 212 | gfn = ALIGN_DOWN(gfn, align); |
| 207 | − if (crste_prefix(*crstep) && (ne.h.p || ne.h.i || !crste_prefix(ne))) { | |
| 208 | − ne.s.fc1.prefix_notif = 0; | |
| 213 | + if (crste_prefix(oldcrste) && (newcrste.h.p || newcrste.h.i || !crste_prefix(newcrste))) { | |
| 214 | + newcrste.s.fc1.prefix_notif = 0; | |
| 209 | 215 | gmap_unmap_prefix(gmap, gfn, gfn + align); |
| 210 | 216 | } |
| 211 | − if (crste_leaf(*crstep) && crstep->s.fc1.vsie_notif && | |
| 212 | − (ne.h.p || ne.h.i || !ne.s.fc1.vsie_notif)) { | |
| 213 | − ne.s.fc1.vsie_notif = 0; | |
| 217 | + if (crste_leaf(oldcrste) && oldcrste.s.fc1.vsie_notif && | |
| 218 | + (newcrste.h.p || newcrste.h.i || !newcrste.s.fc1.vsie_notif)) { | |
| 219 | + newcrste.s.fc1.vsie_notif = 0; | |
| 214 | 220 | if (needs_lock) |
| 215 | 221 | gmap_handle_vsie_unshadow_event(gmap, gfn); |
| 216 | 222 | else |
| 217 | 223 | _gmap_handle_vsie_unshadow_event(gmap, gfn); |
| 218 | 224 | } |
| 219 | − dat_crstep_xchg(crstep, ne, gfn, gmap->asce); | |
| 225 | + if (!oldcrste.s.fc1.d && newcrste.s.fc1.d && !newcrste.s.fc1.s) | |
| 226 | + SetPageDirty(phys_to_page(crste_origin_large(newcrste))); | |
| 227 | + return dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, gmap->asce); | |
| 220 | 228 | } |
| 221 | 229 | |
| 222 | −static inline void gmap_crstep_xchg(struct gmap *gmap, union crste *crstep, union crste ne, | |
| 223 | − gfn_t gfn) | |
| 230 | +static inline bool __must_check gmap_crstep_xchg_atomic(struct gmap *gmap, union crste *crstep, | |
| 231 | + union crste oldcrste, union crste newcrste, | |
| 232 | + gfn_t gfn) | |
| 224 | 233 | { |
| 225 | − return _gmap_crstep_xchg(gmap, crstep, ne, gfn, true); | |
| 234 | + return _gmap_crstep_xchg_atomic(gmap, crstep, oldcrste, newcrste, gfn, true); | |
| 226 | 235 | } |
| 227 | 236 | |
| 228 | 237 | /** |
| @@ -2724,6 +2724,9 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · 2723 unmodified lines | ||
| 2724 | 2724 | |
| 2725 | 2725 | bit = bit_nr + (addr % PAGE_SIZE) * 8; |
| 2726 | 2726 | |
| 2727 | + /* kvm_set_routing_entry() should never allow this to happen */ | |
| 2728 | + WARN_ON_ONCE(bit > (PAGE_SIZE * BITS_PER_BYTE - 1)); | |
| 2729 | + | |
| 2727 | 2730 | return swap ? (bit ^ (BITS_PER_LONG - 1)) : bit; |
| 2728 | 2731 | } |
| 2729 | 2732 | |
| @@ -2824,6 +2827,12 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, · 94 unmodified lines | ||
| 2824 | 2827 | int rc; |
| 2825 | 2828 | |
| 2826 | 2829 | mci.val = mcck_info->mcic; |
| 2830 | + | |
| 2831 | + /* log machine checks being reinjected on all debugs */ | |
| 2832 | + VCPU_EVENT(vcpu, 2, "guest machine check %lx", mci.val); | |
| 2833 | + KVM_EVENT(2, "guest machine check %lx", mci.val); | |
| 2834 | + pr_info("guest machine check pid %d: %lx", current->pid, mci.val); | |
| 2835 | + | |
| 2827 | 2836 | if (mci.sr) |
| 2828 | 2837 | cr14 |= CR14_RECOVERY_SUBMASK; |
| 2829 | 2838 | if (mci.dg) |
| @@ -2852,6 +2861,7 @@ int kvm_set_routing_entry(struct kvm *kvm, · 22 unmodified lines | ||
| 2852 | 2861 | struct kvm_kernel_irq_routing_entry *e, |
| 2853 | 2862 | const struct kvm_irq_routing_entry *ue) |
| 2854 | 2863 | { |
| 2864 | + const struct kvm_irq_routing_s390_adapter *adapter; | |
| 2855 | 2865 | u64 uaddr_s, uaddr_i; |
| 2856 | 2866 | int idx; |
| 2857 | 2867 | |
| @@ -2862,6 +2872,14 @@ int kvm_set_routing_entry(struct kvm *kvm, · 4 unmodified lines | ||
| 2862 | 2872 | return -EINVAL; |
| 2863 | 2873 | e->set = set_adapter_int; |
| 2864 | 2874 | |
| 2875 | + adapter = &ue->u.adapter; | |
| 2876 | + if (adapter->summary_addr + (adapter->summary_offset / 8) >= | |
| 2877 | + (adapter->summary_addr & PAGE_MASK) + PAGE_SIZE) | |
| 2878 | + return -EINVAL; | |
| 2879 | + if (adapter->ind_addr + (adapter->ind_offset / 8) >= | |
| 2880 | + (adapter->ind_addr & PAGE_MASK) + PAGE_SIZE) | |
| 2881 | + return -EINVAL; | |
| 2882 | + | |
| 2865 | 2883 | idx = srcu_read_lock(&kvm->srcu); |
| 2866 | 2884 | uaddr_s = gpa_to_hva(kvm, ue->u.adapter.summary_addr); |
| 2867 | 2885 | uaddr_i = gpa_to_hva(kvm, ue->u.adapter.ind_addr); |
| @@ -65,7 +65,7 @@ · 64 unmodified lines | ||
| 65 | 65 | #define VCPU_IRQS_MAX_BUF (sizeof(struct kvm_s390_irq) * \ |
| 66 | 66 | (KVM_MAX_VCPUS + LOCAL_IRQS)) |
| 67 | 67 | |
| 68 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 68 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 69 | 69 | KVM_GENERIC_VM_STATS(), |
| 70 | 70 | STATS_DESC_COUNTER(VM, inject_io), |
| 71 | 71 | STATS_DESC_COUNTER(VM, inject_float_mchk), |
| @@ -91,7 +91,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 19 unmodified lines | ||
| 91 | 91 | sizeof(kvm_vm_stats_desc), |
| 92 | 92 | }; |
| 93 | 93 | |
| 94 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 94 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 95 | 95 | KVM_GENERIC_VCPU_STATS(), |
| 96 | 96 | STATS_DESC_COUNTER(VCPU, exit_userspace), |
| 97 | 97 | STATS_DESC_COUNTER(VCPU, exit_null), |
| @@ -4617,7 +4617,7 @@ static int vcpu_post_run_handle_fault(struct kvm_vcpu *vcpu) · 4519 unmodified lines | ||
| 4617 | 4617 | return 0; |
| 4618 | 4618 | } |
| 4619 | 4619 | |
| 4620 | −static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason) | |
| 4620 | +static int vcpu_post_run(struct kvm_vcpu *vcpu, int sie_return) | |
| 4621 | 4621 | { |
| 4622 | 4622 | struct mcck_volatile_info *mcck_info; |
| 4623 | 4623 | struct sie_page *sie_page; |
| @@ -4633,14 +4633,14 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason) · 9 unmodified lines | ||
| 4633 | 4633 | vcpu->run->s.regs.gprs[14] = vcpu->arch.sie_block->gg14; |
| 4634 | 4634 | vcpu->run->s.regs.gprs[15] = vcpu->arch.sie_block->gg15; |
| 4635 | 4635 | |
| 4636 | − if (exit_reason == -EINTR) { | |
| 4637 | − VCPU_EVENT(vcpu, 3, "%s", "machine check"); | |
| 4636 | + if (sie_return == SIE64_RETURN_MCCK) { | |
| 4638 | 4637 | sie_page = container_of(vcpu->arch.sie_block, |
| 4639 | 4638 | struct sie_page, sie_block); |
| 4640 | 4639 | mcck_info = &sie_page->mcck_info; |
| 4641 | 4640 | kvm_s390_reinject_machine_check(vcpu, mcck_info); |
| 4642 | 4641 | return 0; |
| 4643 | 4642 | } |
| 4643 | + WARN_ON_ONCE(sie_return != SIE64_RETURN_NORMAL); | |
| 4644 | 4644 | |
| 4645 | 4645 | if (vcpu->arch.sie_block->icptcode > 0) { |
| 4646 | 4646 | rc = kvm_handle_sie_intercept(vcpu); |
| @@ -4679,7 +4679,7 @@ int noinstr kvm_s390_enter_exit_sie(struct kvm_s390_sie_block *scb, · 32 unmodified lines | ||
| 4679 | 4679 | #define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK) |
| 4680 | 4680 | static int __vcpu_run(struct kvm_vcpu *vcpu) |
| 4681 | 4681 | { |
| 4682 | − int rc, exit_reason; | |
| 4682 | + int rc, sie_return; | |
| 4683 | 4683 | struct sie_page *sie_page = (struct sie_page *)vcpu->arch.sie_block; |
| 4684 | 4684 | |
| 4685 | 4685 | /* |
| @@ -4719,9 +4719,9 @@ xfer_to_guest_mode_check: · 33 unmodified lines | ||
| 4719 | 4719 | guest_timing_enter_irqoff(); |
| 4720 | 4720 | __disable_cpu_timer_accounting(vcpu); |
| 4721 | 4721 | |
| 4722 | − exit_reason = kvm_s390_enter_exit_sie(vcpu->arch.sie_block, | |
| 4723 | − vcpu->run->s.regs.gprs, | |
| 4724 | − vcpu->arch.gmap->asce.val); | |
| 4722 | + sie_return = kvm_s390_enter_exit_sie(vcpu->arch.sie_block, | |
| 4723 | + vcpu->run->s.regs.gprs, | |
| 4724 | + vcpu->arch.gmap->asce.val); | |
| 4725 | 4725 | |
| 4726 | 4726 | __enable_cpu_timer_accounting(vcpu); |
| 4727 | 4727 | guest_timing_exit_irqoff(); |
| @@ -4744,7 +4744,7 @@ xfer_to_guest_mode_check: · 16 unmodified lines | ||
| 4744 | 4744 | } |
| 4745 | 4745 | kvm_vcpu_srcu_read_lock(vcpu); |
| 4746 | 4746 | |
| 4747 | − rc = vcpu_post_run(vcpu, exit_reason); | |
| 4747 | + rc = vcpu_post_run(vcpu, sie_return); | |
| 4748 | 4748 | if (rc || guestdbg_exit_pending(vcpu)) { |
| 4749 | 4749 | kvm_vcpu_srcu_read_unlock(vcpu); |
| 4750 | 4750 | break; |
| @@ -5520,9 +5520,21 @@ long kvm_arch_vcpu_ioctl(struct file *filp, · 769 unmodified lines | ||
| 5520 | 5520 | } |
| 5521 | 5521 | #endif |
| 5522 | 5522 | case KVM_S390_VCPU_FAULT: { |
| 5523 | − idx = srcu_read_lock(&vcpu->kvm->srcu); | |
| 5524 | − r = vcpu_dat_fault_handler(vcpu, arg, 0); | |
| 5525 | − srcu_read_unlock(&vcpu->kvm->srcu, idx); | |
| 5523 | + gpa_t gaddr = arg; | |
| 5524 | + | |
| 5525 | + scoped_guard(srcu, &vcpu->kvm->srcu) { | |
| 5526 | + r = vcpu_ucontrol_translate(vcpu, &gaddr); | |
| 5527 | + if (r) | |
| 5528 | + break; | |
| 5529 | + | |
| 5530 | + r = kvm_s390_faultin_gfn_simple(vcpu, NULL, gpa_to_gfn(gaddr), false); | |
| 5531 | + if (r == PGM_ADDRESSING) | |
| 5532 | + r = -EFAULT; | |
| 5533 | + if (r <= 0) | |
| 5534 | + break; | |
| 5535 | + r = -EIO; | |
| 5536 | + KVM_BUG_ON(r, vcpu->kvm); | |
| 5537 | + } | |
| 5526 | 5538 | break; |
| 5527 | 5539 | } |
| 5528 | 5540 | case KVM_ENABLE_CAP: |
| @@ -1122,6 +1122,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page, struc · 1121 unmodified lines | ||
| 1122 | 1122 | { |
| 1123 | 1123 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; |
| 1124 | 1124 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; |
| 1125 | + unsigned long sie_return = SIE64_RETURN_NORMAL; | |
| 1125 | 1126 | int guest_bp_isolation; |
| 1126 | 1127 | int rc = 0; |
| 1127 | 1128 | |
| @@ -1163,7 +1164,7 @@ xfer_to_guest_mode_check: · 35 unmodified lines | ||
| 1163 | 1164 | goto xfer_to_guest_mode_check; |
| 1164 | 1165 | } |
| 1165 | 1166 | guest_timing_enter_irqoff(); |
| 1166 | − rc = kvm_s390_enter_exit_sie(scb_s, vcpu->run->s.regs.gprs, sg->asce.val); | |
| 1167 | + sie_return = kvm_s390_enter_exit_sie(scb_s, vcpu->run->s.regs.gprs, sg->asce.val); | |
| 1167 | 1168 | guest_timing_exit_irqoff(); |
| 1168 | 1169 | local_irq_enable(); |
| 1169 | 1170 | } |
| @@ -1178,12 +1179,13 @@ skip_sie: · 8 unmodified lines | ||
| 1178 | 1179 | |
| 1179 | 1180 | kvm_vcpu_srcu_read_lock(vcpu); |
| 1180 | 1181 | |
| 1181 | − if (rc == -EINTR) { | |
| 1182 | − VCPU_EVENT(vcpu, 3, "%s", "machine check"); | |
| 1182 | + if (sie_return == SIE64_RETURN_MCCK) { | |
| 1183 | 1183 | kvm_s390_reinject_machine_check(vcpu, &vsie_page->mcck_info); |
| 1184 | 1184 | return 0; |
| 1185 | 1185 | } |
| 1186 | 1186 | |
| 1187 | + WARN_ON_ONCE(sie_return != SIE64_RETURN_NORMAL); | |
| 1188 | + | |
| 1187 | 1189 | if (rc > 0) |
| 1188 | 1190 | rc = 0; /* we could still have an icpt */ |
| 1189 | 1191 | else if (current->thread.gmap_int_code) |
| @@ -1326,7 +1328,7 @@ static void unregister_shadow_scb(struct kvm_vcpu *vcpu) · 136 unmodified lines | ||
| 1326 | 1328 | static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) |
| 1327 | 1329 | { |
| 1328 | 1330 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; |
| 1329 | − struct gmap *sg; | |
| 1331 | + struct gmap *sg = NULL; | |
| 1330 | 1332 | int rc = 0; |
| 1331 | 1333 | |
| 1332 | 1334 | while (1) { |
| @@ -1366,6 +1368,8 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 33 unmodified lines | ||
| 1366 | 1368 | sg = gmap_put(sg); |
| 1367 | 1369 | cond_resched(); |
| 1368 | 1370 | } |
| 1371 | + if (sg) | |
| 1372 | + sg = gmap_put(sg); | |
| 1369 | 1373 | |
| 1370 | 1374 | if (rc == -EFAULT) { |
| 1371 | 1375 | /* |
| @@ -441,10 +441,17 @@ void do_secure_storage_access(struct pt_regs *regs) · 440 unmodified lines | ||
| 441 | 441 | folio = phys_to_folio(addr); |
| 442 | 442 | if (unlikely(!folio_try_get(folio))) |
| 443 | 443 | return; |
| 444 | − rc = arch_make_folio_accessible(folio); | |
| 444 | + rc = uv_convert_from_secure(folio_to_phys(folio)); | |
| 445 | + if (!rc) | |
| 446 | + clear_bit(PG_arch_1, &folio->flags.f); | |
| 445 | 447 | folio_put(folio); |
| 448 | + /* | |
| 449 | + * There are some valid fixup types for kernel | |
| 450 | + * accesses to donated secure memory. zeropad is one | |
| 451 | + * of them. | |
| 452 | + */ | |
| 446 | 453 | if (rc) |
| 447 | − BUG(); | |
| 454 | + return handle_fault_error_nolock(regs, 0); | |
| 448 | 455 | } else { |
| 449 | 456 | if (faulthandler_disabled()) |
| 450 | 457 | return handle_fault_error_nolock(regs, 0); |
| @@ -26,10 +26,6 @@ static int platform_match(struct device *dev, struct device_driver *drv) · 25 unmodified lines | ||
| 26 | 26 | struct platform_device *pdev = to_platform_device(dev); |
| 27 | 27 | struct platform_driver *pdrv = to_platform_driver(drv); |
| 28 | 28 | |
| 29 | − /* When driver_override is set, only bind to the matching driver */ | |
| 30 | − if (pdev->driver_override) | |
| 31 | − return !strcmp(pdev->driver_override, drv->name); | |
| 32 | − | |
| 33 | 29 | /* Then try to match against the id table */ |
| 34 | 30 | if (pdrv->id_table) |
| 35 | 31 | return platform_match_id(pdrv->id_table, pdev) != NULL; |
| @@ -121,6 +121,9 @@ noinstr struct ghcb *__sev_get_ghcb(struct ghcb_state *state) · 120 unmodified lines | ||
| 121 | 121 | |
| 122 | 122 | WARN_ON(!irqs_disabled()); |
| 123 | 123 | |
| 124 | + if (!sev_cfg.ghcbs_initialized) | |
| 125 | + return boot_ghcb; | |
| 126 | + | |
| 124 | 127 | data = this_cpu_read(runtime_data); |
| 125 | 128 | ghcb = &data->ghcb_page; |
| 126 | 129 | |
| @@ -164,6 +167,9 @@ noinstr void __sev_put_ghcb(struct ghcb_state *state) · 37 unmodified lines | ||
| 164 | 167 | |
| 165 | 168 | WARN_ON(!irqs_disabled()); |
| 166 | 169 | |
| 170 | + if (!sev_cfg.ghcbs_initialized) | |
| 171 | + return; | |
| 172 | + | |
| 167 | 173 | data = this_cpu_read(runtime_data); |
| 168 | 174 | ghcb = &data->ghcb_page; |
| 169 | 175 | |
| @@ -177,6 +177,16 @@ static noinstr void fred_extint(struct pt_regs *regs) · 176 unmodified lines | ||
| 177 | 177 | } |
| 178 | 178 | } |
| 179 | 179 | |
| 180 | +#ifdef CONFIG_AMD_MEM_ENCRYPT | |
| 181 | +noinstr void exc_vmm_communication(struct pt_regs *regs, unsigned long error_code) | |
| 182 | +{ | |
| 183 | + if (user_mode(regs)) | |
| 184 | + return user_exc_vmm_communication(regs, error_code); | |
| 185 | + else | |
| 186 | + return kernel_exc_vmm_communication(regs, error_code); | |
| 187 | +} | |
| 188 | +#endif | |
| 189 | + | |
| 180 | 190 | static noinstr void fred_hwexc(struct pt_regs *regs, unsigned long error_code) |
| 181 | 191 | { |
| 182 | 192 | /* Optimize for #PF. That's the only exception which matters performance wise */ |
| @@ -207,6 +217,10 @@ static noinstr void fred_hwexc(struct pt_regs *regs, unsigned long error_code) · 24 unmodified lines | ||
| 207 | 217 | #ifdef CONFIG_X86_CET |
| 208 | 218 | case X86_TRAP_CP: return exc_control_protection(regs, error_code); |
| 209 | 219 | #endif |
| 220 | +#ifdef CONFIG_AMD_MEM_ENCRYPT | |
| 221 | + case X86_TRAP_VC: return exc_vmm_communication(regs, error_code); | |
| 222 | +#endif | |
| 223 | + | |
| 210 | 224 | default: return fred_bad_type(regs, error_code); |
| 211 | 225 | } |
| 212 | 226 | |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/types.h> |
| 14 | 14 | #include <vdso/gettime.h> |
| 15 | 15 | |
| 16 | −#include "../../../../lib/vdso/gettimeofday.c" | |
| 16 | +#include "lib/vdso/gettimeofday.c" | |
| 17 | 17 | |
| 18 | 18 | int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) |
| 19 | 19 | { |
| @@ -1372,14 +1372,17 @@ static void x86_pmu_enable(struct pmu *pmu) · 1371 unmodified lines | ||
| 1372 | 1372 | else if (i < n_running) |
| 1373 | 1373 | continue; |
| 1374 | 1374 | |
| 1375 | − if (hwc->state & PERF_HES_ARCH) | |
| 1375 | + cpuc->events[hwc->idx] = event; | |
| 1376 | + | |
| 1377 | + if (hwc->state & PERF_HES_ARCH) { | |
| 1378 | + static_call(x86_pmu_set_period)(event); | |
| 1376 | 1379 | continue; |
| 1380 | + } | |
| 1377 | 1381 | |
| 1378 | 1382 | /* |
| 1379 | 1383 | * if cpuc->enabled = 0, then no wrmsr as |
| 1380 | 1384 | * per x86_pmu_enable_event() |
| 1381 | 1385 | */ |
| 1382 | − cpuc->events[hwc->idx] = event; | |
| 1383 | 1386 | x86_pmu_start(event, PERF_EF_RELOAD); |
| 1384 | 1387 | } |
| 1385 | 1388 | cpuc->n_added = 0; |
| @@ -4628,6 +4628,19 @@ static inline void intel_pmu_set_acr_caused_constr(struct perf_event *event, · 4627 unmodified lines | ||
| 4628 | 4628 | event->hw.dyn_constraint &= hybrid(event->pmu, acr_cause_mask64); |
| 4629 | 4629 | } |
| 4630 | 4630 | |
| 4631 | +static inline int intel_set_branch_counter_constr(struct perf_event *event, | |
| 4632 | + int *num) | |
| 4633 | +{ | |
| 4634 | + if (branch_sample_call_stack(event)) | |
| 4635 | + return -EINVAL; | |
| 4636 | + if (branch_sample_counters(event)) { | |
| 4637 | + (*num)++; | |
| 4638 | + event->hw.dyn_constraint &= x86_pmu.lbr_counters; | |
| 4639 | + } | |
| 4640 | + | |
| 4641 | + return 0; | |
| 4642 | +} | |
| 4643 | + | |
| 4631 | 4644 | static int intel_pmu_hw_config(struct perf_event *event) |
| 4632 | 4645 | { |
| 4633 | 4646 | int ret = x86_pmu_hw_config(event); |
| @@ -4698,21 +4711,19 @@ static int intel_pmu_hw_config(struct perf_event *event) · 64 unmodified lines | ||
| 4698 | 4711 | * group, which requires the extra space to store the counters. |
| 4699 | 4712 | */ |
| 4700 | 4713 | leader = event->group_leader; |
| 4701 | − if (branch_sample_call_stack(leader)) | |
| 4714 | + if (intel_set_branch_counter_constr(leader, &num)) | |
| 4702 | 4715 | return -EINVAL; |
| 4703 | − if (branch_sample_counters(leader)) { | |
| 4704 | − num++; | |
| 4705 | − leader->hw.dyn_constraint &= x86_pmu.lbr_counters; | |
| 4706 | − } | |
| 4707 | 4716 | leader->hw.flags |= PERF_X86_EVENT_BRANCH_COUNTERS; |
| 4708 | 4717 | |
| 4709 | 4718 | for_each_sibling_event(sibling, leader) { |
| 4710 | − if (branch_sample_call_stack(sibling)) | |
| 4719 | + if (intel_set_branch_counter_constr(sibling, &num)) | |
| 4720 | + return -EINVAL; | |
| 4721 | + } | |
| 4722 | + | |
| 4723 | + /* event isn't installed as a sibling yet. */ | |
| 4724 | + if (event != leader) { | |
| 4725 | + if (intel_set_branch_counter_constr(event, &num)) | |
| 4711 | 4726 | return -EINVAL; |
| 4712 | − if (branch_sample_counters(sibling)) { | |
| 4713 | − num++; | |
| 4714 | − sibling->hw.dyn_constraint &= x86_pmu.lbr_counters; | |
| 4715 | − } | |
| 4716 | 4727 | } |
| 4717 | 4728 | |
| 4718 | 4729 | if (num > fls(x86_pmu.lbr_counters)) |
| @@ -4844,8 +4855,10 @@ static int intel_pmu_hw_config(struct perf_event *event) · 125 unmodified lines | ||
| 4844 | 4855 | intel_pmu_set_acr_caused_constr(leader, idx++, cause_mask); |
| 4845 | 4856 | |
| 4846 | 4857 | if (leader->nr_siblings) { |
| 4847 | − for_each_sibling_event(sibling, leader) | |
| 4848 | − intel_pmu_set_acr_caused_constr(sibling, idx++, cause_mask); | |
| 4858 | + for_each_sibling_event(sibling, leader) { | |
| 4859 | + if (is_x86_event(sibling)) | |
| 4860 | + intel_pmu_set_acr_caused_constr(sibling, idx++, cause_mask); | |
| 4861 | + } | |
| 4849 | 4862 | } |
| 4850 | 4863 | |
| 4851 | 4864 | if (leader != event) |
| @@ -345,12 +345,12 @@ static u64 parse_omr_data_source(u8 dse) · 344 unmodified lines | ||
| 345 | 345 | if (omr.omr_remote) |
| 346 | 346 | val |= REM; |
| 347 | 347 | |
| 348 | − val |= omr.omr_hitm ? P(SNOOP, HITM) : P(SNOOP, HIT); | |
| 349 | − | |
| 350 | 348 | if (omr.omr_source == 0x2) { |
| 351 | − u8 snoop = omr.omr_snoop | omr.omr_promoted; | |
| 349 | + u8 snoop = omr.omr_snoop | (omr.omr_promoted << 1); | |
| 352 | 350 | |
| 353 | − if (snoop == 0x0) | |
| 351 | + if (omr.omr_hitm) | |
| 352 | + val |= P(SNOOP, HITM); | |
| 353 | + else if (snoop == 0x0) | |
| 354 | 354 | val |= P(SNOOP, NA); |
| 355 | 355 | else if (snoop == 0x1) |
| 356 | 356 | val |= P(SNOOP, MISS); |
| @@ -359,7 +359,10 @@ static u64 parse_omr_data_source(u8 dse) · 2 unmodified lines | ||
| 359 | 359 | else if (snoop == 0x3) |
| 360 | 360 | val |= P(SNOOP, NONE); |
| 361 | 361 | } else if (omr.omr_source > 0x2 && omr.omr_source < 0x7) { |
| 362 | + val |= omr.omr_hitm ? P(SNOOP, HITM) : P(SNOOP, HIT); | |
| 362 | 363 | val |= omr.omr_snoop ? P(SNOOPX, FWD) : 0; |
| 364 | + } else { | |
| 365 | + val |= P(SNOOP, NONE); | |
| 363 | 366 | } |
| 364 | 367 | |
| 365 | 368 | return val; |
| @@ -107,14 +107,12 @@ static void __noreturn hv_panic_timeout_reboot(void) · 106 unmodified lines | ||
| 107 | 107 | cpu_relax(); |
| 108 | 108 | } |
| 109 | 109 | |
| 110 | −/* This cannot be inlined as it needs stack */ | |
| 111 | −static noinline __noclone void hv_crash_restore_tss(void) | |
| 110 | +static void hv_crash_restore_tss(void) | |
| 112 | 111 | { |
| 113 | 112 | load_TR_desc(); |
| 114 | 113 | } |
| 115 | 114 | |
| 116 | −/* This cannot be inlined as it needs stack */ | |
| 117 | −static noinline void hv_crash_clear_kernpt(void) | |
| 115 | +static void hv_crash_clear_kernpt(void) | |
| 118 | 116 | { |
| 119 | 117 | pgd_t *pgd; |
| 120 | 118 | p4d_t *p4d; |
| @@ -125,6 +123,25 @@ static noinline void hv_crash_clear_kernpt(void) · 4 unmodified lines | ||
| 125 | 123 | native_p4d_clear(p4d); |
| 126 | 124 | } |
| 127 | 125 | |
| 126 | + | |
| 127 | +static void __noreturn hv_crash_handle(void) | |
| 128 | +{ | |
| 129 | + hv_crash_restore_tss(); | |
| 130 | + hv_crash_clear_kernpt(); | |
| 131 | + | |
| 132 | + /* we are now fully in devirtualized normal kernel mode */ | |
| 133 | + __crash_kexec(NULL); | |
| 134 | + | |
| 135 | + hv_panic_timeout_reboot(); | |
| 136 | +} | |
| 137 | + | |
| 138 | +/* | |
| 139 | + * __naked functions do not permit function calls, not even to __always_inline | |
| 140 | + * functions that only contain asm() blocks themselves. So use a macro instead. | |
| 141 | + */ | |
| 142 | +#define hv_wrmsr(msr, val) \ | |
| 143 | + asm volatile("wrmsr" :: "c"(msr), "a"((u32)val), "d"((u32)(val >> 32)) : "memory") | |
| 144 | + | |
| 128 | 145 | /* |
| 129 | 146 | * This is the C entry point from the asm glue code after the disable hypercall. |
| 130 | 147 | * We enter here in IA32-e long mode, ie, full 64bit mode running on kernel |
| @@ -133,51 +150,38 @@ static noinline void hv_crash_clear_kernpt(void) · 2 unmodified lines | ||
| 133 | 150 | * available. We restore kernel GDT, and rest of the context, and continue |
| 134 | 151 | * to kexec. |
| 135 | 152 | */ |
| 136 | −static asmlinkage void __noreturn hv_crash_c_entry(void) | |
| 153 | +static void __naked hv_crash_c_entry(void) | |
| 137 | 154 | { |
| 138 | − struct hv_crash_ctxt *ctxt = &hv_crash_ctxt; | |
| 139 | − | |
| 140 | 155 | /* first thing, restore kernel gdt */ |
| 141 | − native_load_gdt(&ctxt->gdtr); | |
| 156 | + asm volatile("lgdt %0" : : "m" (hv_crash_ctxt.gdtr)); | |
| 142 | 157 | |
| 143 | − asm volatile("movw %%ax, %%ss" : : "a"(ctxt->ss)); | |
| 144 | − asm volatile("movq %0, %%rsp" : : "m"(ctxt->rsp)); | |
| 158 | + asm volatile("movw %0, %%ss\n\t" | |
| 159 | + "movq %1, %%rsp" | |
| 160 | + :: "m"(hv_crash_ctxt.ss), "m"(hv_crash_ctxt.rsp)); | |
| 145 | 161 | |
| 146 | − asm volatile("movw %%ax, %%ds" : : "a"(ctxt->ds)); | |
| 147 | − asm volatile("movw %%ax, %%es" : : "a"(ctxt->es)); | |
| 148 | − asm volatile("movw %%ax, %%fs" : : "a"(ctxt->fs)); | |
| 149 | − asm volatile("movw %%ax, %%gs" : : "a"(ctxt->gs)); | |
| 162 | + asm volatile("movw %0, %%ds" : : "m"(hv_crash_ctxt.ds)); | |
| 163 | + asm volatile("movw %0, %%es" : : "m"(hv_crash_ctxt.es)); | |
| 164 | + asm volatile("movw %0, %%fs" : : "m"(hv_crash_ctxt.fs)); | |
| 165 | + asm volatile("movw %0, %%gs" : : "m"(hv_crash_ctxt.gs)); | |
| 150 | 166 | |
| 151 | − native_wrmsrq(MSR_IA32_CR_PAT, ctxt->pat); | |
| 152 | − asm volatile("movq %0, %%cr0" : : "r"(ctxt->cr0)); | |
| 167 | + hv_wrmsr(MSR_IA32_CR_PAT, hv_crash_ctxt.pat); | |
| 168 | + asm volatile("movq %0, %%cr0" : : "r"(hv_crash_ctxt.cr0)); | |
| 153 | 169 | |
| 154 | − asm volatile("movq %0, %%cr8" : : "r"(ctxt->cr8)); | |
| 155 | − asm volatile("movq %0, %%cr4" : : "r"(ctxt->cr4)); | |
| 156 | − asm volatile("movq %0, %%cr2" : : "r"(ctxt->cr4)); | |
| 170 | + asm volatile("movq %0, %%cr8" : : "r"(hv_crash_ctxt.cr8)); | |
| 171 | + asm volatile("movq %0, %%cr4" : : "r"(hv_crash_ctxt.cr4)); | |
| 172 | + asm volatile("movq %0, %%cr2" : : "r"(hv_crash_ctxt.cr2)); | |
| 157 | 173 | |
| 158 | − native_load_idt(&ctxt->idtr); | |
| 159 | − native_wrmsrq(MSR_GS_BASE, ctxt->gsbase); | |
| 160 | − native_wrmsrq(MSR_EFER, ctxt->efer); | |
| 174 | + asm volatile("lidt %0" : : "m" (hv_crash_ctxt.idtr)); | |
| 175 | + hv_wrmsr(MSR_GS_BASE, hv_crash_ctxt.gsbase); | |
| 176 | + hv_wrmsr(MSR_EFER, hv_crash_ctxt.efer); | |
| 161 | 177 | |
| 162 | 178 | /* restore the original kernel CS now via far return */ |
| 163 | − asm volatile("movzwq %0, %%rax\n\t" | |
| 164 | − "pushq %%rax\n\t" | |
| 165 | − "pushq $1f\n\t" | |
| 166 | − "lretq\n\t" | |
| 167 | − "1:nop\n\t" : : "m"(ctxt->cs) : "rax"); | |
| 168 | − | |
| 169 | − /* We are in asmlinkage without stack frame, hence make C function | |
| 170 | − * calls which will buy stack frames. | |
| 171 | − */ | |
| 172 | − hv_crash_restore_tss(); | |
| 173 | − hv_crash_clear_kernpt(); | |
| 174 | − | |
| 175 | − /* we are now fully in devirtualized normal kernel mode */ | |
| 176 | − __crash_kexec(NULL); | |
| 177 | − | |
| 178 | − hv_panic_timeout_reboot(); | |
| 179 | + asm volatile("pushq %q0\n\t" | |
| 180 | + "pushq %q1\n\t" | |
| 181 | + "lretq" | |
| 182 | + :: "r"(hv_crash_ctxt.cs), "r"(hv_crash_handle)); | |
| 179 | 183 | } |
| 180 | −/* Tell gcc we are using lretq long jump in the above function intentionally */ | |
| 184 | +/* Tell objtool we are using lretq long jump in the above function intentionally */ | |
| 181 | 185 | STACK_FRAME_NON_STANDARD(hv_crash_c_entry); |
| 182 | 186 | |
| 183 | 187 | static void hv_mark_tss_not_busy(void) |
| @@ -195,20 +199,20 @@ static void hv_hvcrash_ctxt_save(void) · 11 unmodified lines | ||
| 195 | 199 | { |
| 196 | 200 | struct hv_crash_ctxt *ctxt = &hv_crash_ctxt; |
| 197 | 201 | |
| 198 | − asm volatile("movq %%rsp,%0" : "=m"(ctxt->rsp)); | |
| 202 | + ctxt->rsp = current_stack_pointer; | |
| 199 | 203 | |
| 200 | 204 | ctxt->cr0 = native_read_cr0(); |
| 201 | 205 | ctxt->cr4 = native_read_cr4(); |
| 202 | 206 | |
| 203 | − asm volatile("movq %%cr2, %0" : "=a"(ctxt->cr2)); | |
| 204 | − asm volatile("movq %%cr8, %0" : "=a"(ctxt->cr8)); | |
| 207 | + asm volatile("movq %%cr2, %0" : "=r"(ctxt->cr2)); | |
| 208 | + asm volatile("movq %%cr8, %0" : "=r"(ctxt->cr8)); | |
| 205 | 209 | |
| 206 | − asm volatile("movl %%cs, %%eax" : "=a"(ctxt->cs)); | |
| 207 | − asm volatile("movl %%ss, %%eax" : "=a"(ctxt->ss)); | |
| 208 | − asm volatile("movl %%ds, %%eax" : "=a"(ctxt->ds)); | |
| 209 | − asm volatile("movl %%es, %%eax" : "=a"(ctxt->es)); | |
| 210 | − asm volatile("movl %%fs, %%eax" : "=a"(ctxt->fs)); | |
| 211 | − asm volatile("movl %%gs, %%eax" : "=a"(ctxt->gs)); | |
| 210 | + asm volatile("movw %%cs, %0" : "=m"(ctxt->cs)); | |
| 211 | + asm volatile("movw %%ss, %0" : "=m"(ctxt->ss)); | |
| 212 | + asm volatile("movw %%ds, %0" : "=m"(ctxt->ds)); | |
| 213 | + asm volatile("movw %%es, %0" : "=m"(ctxt->es)); | |
| 214 | + asm volatile("movw %%fs, %0" : "=m"(ctxt->fs)); | |
| 215 | + asm volatile("movw %%gs, %0" : "=m"(ctxt->gs)); | |
| 212 | 216 | |
| 213 | 217 | native_store_gdt(&ctxt->gdtr); |
| 214 | 218 | store_idt(&ctxt->idtr); |
| @@ -2485,7 +2485,8 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); · 2484 unmodified lines | ||
| 2485 | 2485 | KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ |
| 2486 | 2486 | KVM_X86_QUIRK_SLOT_ZAP_ALL | \ |
| 2487 | 2487 | KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ |
| 2488 | − KVM_X86_QUIRK_IGNORE_GUEST_PAT) | |
| 2488 | + KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ | |
| 2489 | + KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM) | |
| 2489 | 2490 | |
| 2490 | 2491 | #define KVM_X86_CONDITIONAL_QUIRKS \ |
| 2491 | 2492 | (KVM_X86_QUIRK_CD_NW_CLEARED | \ |
| @@ -476,6 +476,7 @@ struct kvm_sync_regs { · 475 unmodified lines | ||
| 476 | 476 | #define KVM_X86_QUIRK_SLOT_ZAP_ALL (1 << 7) |
| 477 | 477 | #define KVM_X86_QUIRK_STUFF_FEATURE_MSRS (1 << 8) |
| 478 | 478 | #define KVM_X86_QUIRK_IGNORE_GUEST_PAT (1 << 9) |
| 479 | +#define KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM (1 << 10) | |
| 479 | 480 | |
| 480 | 481 | #define KVM_STATE_NESTED_FORMAT_VMX 0 |
| 481 | 482 | #define KVM_STATE_NESTED_FORMAT_SVM 1 |
| @@ -44,6 +44,20 @@ KCOV_INSTRUMENT_unwind_orc.o := n · 43 unmodified lines | ||
| 44 | 44 | KCOV_INSTRUMENT_unwind_frame.o := n |
| 45 | 45 | KCOV_INSTRUMENT_unwind_guess.o := n |
| 46 | 46 | |
| 47 | +# Disable KCOV to prevent crashes during kexec: load_segments() invalidates | |
| 48 | +# the GS base, which KCOV relies on for per-CPU data. | |
| 49 | +# | |
| 50 | +# As KCOV and KEXEC compatibility should be preserved (e.g. syzkaller is | |
| 51 | +# using it to collect crash dumps during kernel fuzzing), disabling | |
| 52 | +# KCOV for KEXEC kernels is not an option. Selectively disabling KCOV | |
| 53 | +# instrumentation for individual affected functions can be fragile, while | |
| 54 | +# adding more checks to KCOV would slow it down. | |
| 55 | +# | |
| 56 | +# As a compromise solution, disable KCOV instrumentation for the whole | |
| 57 | +# source code file. If its coverage is ever needed, other approaches | |
| 58 | +# should be considered. | |
| 59 | +KCOV_INSTRUMENT_machine_kexec_64.o := n | |
| 60 | + | |
| 47 | 61 | CFLAGS_head32.o := -fno-stack-protector |
| 48 | 62 | CFLAGS_head64.o := -fno-stack-protector |
| 49 | 63 | CFLAGS_irq.o := -I $(src)/../include/asm/trace |
| @@ -1894,6 +1894,7 @@ void __init check_x2apic(void) · 1893 unmodified lines | ||
| 1894 | 1894 | |
| 1895 | 1895 | static inline void try_to_enable_x2apic(int remap_mode) { } |
| 1896 | 1896 | static inline void __x2apic_enable(void) { } |
| 1897 | +static inline void __x2apic_disable(void) { } | |
| 1897 | 1898 | #endif /* !CONFIG_X86_X2APIC */ |
| 1898 | 1899 | |
| 1899 | 1900 | void __init enable_IR_x2apic(void) |
| @@ -2456,6 +2457,11 @@ static void lapic_resume(void *data) · 556 unmodified lines | ||
| 2456 | 2457 | if (x2apic_mode) { |
| 2457 | 2458 | __x2apic_enable(); |
| 2458 | 2459 | } else { |
| 2460 | + if (x2apic_enabled()) { | |
| 2461 | + pr_warn_once("x2apic: re-enabled by firmware during resume. Disabling\n"); | |
| 2462 | + __x2apic_disable(); | |
| 2463 | + } | |
| 2464 | + | |
| 2459 | 2465 | /* |
| 2460 | 2466 | * Make sure the APICBASE points to the right address |
| 2461 | 2467 | * |
| @@ -1708,8 +1708,22 @@ static void __init uv_system_init_hub(void) · 1707 unmodified lines | ||
| 1708 | 1708 | struct uv_hub_info_s *new_hub; |
| 1709 | 1709 | |
| 1710 | 1710 | /* Allocate & fill new per hub info list */ |
| 1711 | − new_hub = (bid == 0) ? &uv_hub_info_node0 | |
| 1712 | − : kzalloc_node(bytes, GFP_KERNEL, uv_blade_to_node(bid)); | |
| 1711 | + if (bid == 0) { | |
| 1712 | + new_hub = &uv_hub_info_node0; | |
| 1713 | + } else { | |
| 1714 | + int nid; | |
| 1715 | + | |
| 1716 | + /* | |
| 1717 | + * Deconfigured sockets are mapped to SOCK_EMPTY. Use | |
| 1718 | + * NUMA_NO_NODE to allocate on a valid node. | |
| 1719 | + */ | |
| 1720 | + nid = uv_blade_to_node(bid); | |
| 1721 | + if (nid == SOCK_EMPTY) | |
| 1722 | + nid = NUMA_NO_NODE; | |
| 1723 | + | |
| 1724 | + new_hub = kzalloc_node(bytes, GFP_KERNEL, nid); | |
| 1725 | + } | |
| 1726 | + | |
| 1713 | 1727 | if (WARN_ON_ONCE(!new_hub)) { |
| 1714 | 1728 | /* do not kfree() bid 0, which is statically allocated */ |
| 1715 | 1729 | while (--bid > 0) |
| @@ -433,7 +433,20 @@ static __always_inline void setup_lass(struct cpuinfo_x86 *c) · 432 unmodified lines | ||
| 433 | 433 | |
| 434 | 434 | /* These bits should not change their value after CPU init is finished. */ |
| 435 | 435 | static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | |
| 436 | − X86_CR4_FSGSBASE | X86_CR4_CET | X86_CR4_FRED; | |
| 436 | + X86_CR4_FSGSBASE | X86_CR4_CET; | |
| 437 | + | |
| 438 | +/* | |
| 439 | + * The CR pinning protects against ROP on the 'mov %reg, %CRn' instruction(s). | |
| 440 | + * Since you can ROP directly to these instructions (barring shadow stack), | |
| 441 | + * any protection must follow immediately and unconditionally after that. | |
| 442 | + * | |
| 443 | + * Specifically, the CR[04] write functions below will have the value | |
| 444 | + * validation controlled by the @cr_pinning static_branch which is | |
| 445 | + * __ro_after_init, just like the cr4_pinned_bits value. | |
| 446 | + * | |
| 447 | + * Once set, an attacker will have to defeat page-tables to get around these | |
| 448 | + * restrictions. Which is a much bigger ask than 'simple' ROP. | |
| 449 | + */ | |
| 437 | 450 | static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning); |
| 438 | 451 | static unsigned long cr4_pinned_bits __ro_after_init; |
| 439 | 452 | |
| @@ -2050,12 +2063,6 @@ static void identify_cpu(struct cpuinfo_x86 *c) · 1610 unmodified lines | ||
| 2050 | 2063 | setup_umip(c); |
| 2051 | 2064 | setup_lass(c); |
| 2052 | 2065 | |
| 2053 | − /* Enable FSGSBASE instructions if available. */ | |
| 2054 | − if (cpu_has(c, X86_FEATURE_FSGSBASE)) { | |
| 2055 | − cr4_set_bits(X86_CR4_FSGSBASE); | |
| 2056 | − elf_hwcap2 |= HWCAP2_FSGSBASE; | |
| 2057 | − } | |
| 2058 | − | |
| 2059 | 2066 | /* |
| 2060 | 2067 | * The vendor-specific functions might have changed features. |
| 2061 | 2068 | * Now we do "generic changes." |
| @@ -2416,6 +2423,18 @@ void cpu_init_exception_handling(bool boot_cpu) · 354 unmodified lines | ||
| 2416 | 2423 | /* GHCB needs to be setup to handle #VC. */ |
| 2417 | 2424 | setup_ghcb(); |
| 2418 | 2425 | |
| 2426 | + /* | |
| 2427 | + * On CPUs with FSGSBASE support, paranoid_entry() uses | |
| 2428 | + * ALTERNATIVE-patched RDGSBASE/WRGSBASE instructions. Secondary CPUs | |
| 2429 | + * boot after alternatives are patched globally, so early exceptions | |
| 2430 | + * execute patched code that depends on FSGSBASE. Enable the feature | |
| 2431 | + * before any exceptions occur. | |
| 2432 | + */ | |
| 2433 | + if (cpu_feature_enabled(X86_FEATURE_FSGSBASE)) { | |
| 2434 | + cr4_set_bits(X86_CR4_FSGSBASE); | |
| 2435 | + elf_hwcap2 |= HWCAP2_FSGSBASE; | |
| 2436 | + } | |
| 2437 | + | |
| 2419 | 2438 | if (cpu_feature_enabled(X86_FEATURE_FRED)) { |
| 2420 | 2439 | /* The boot CPU has enabled FRED during early boot */ |
| 2421 | 2440 | if (!boot_cpu) |
| @@ -875,13 +875,18 @@ void amd_clear_bank(struct mce *m) · 874 unmodified lines | ||
| 875 | 875 | { |
| 876 | 876 | amd_reset_thr_limit(m->bank); |
| 877 | 877 | |
| 878 | − /* Clear MCA_DESTAT for all deferred errors even those logged in MCA_STATUS. */ | |
| 879 | − if (m->status & MCI_STATUS_DEFERRED) | |
| 880 | − mce_wrmsrq(MSR_AMD64_SMCA_MCx_DESTAT(m->bank), 0); | |
| 878 | + if (mce_flags.smca) { | |
| 879 | + /* | |
| 880 | + * Clear MCA_DESTAT for all deferred errors even those | |
| 881 | + * logged in MCA_STATUS. | |
| 882 | + */ | |
| 883 | + if (m->status & MCI_STATUS_DEFERRED) | |
| 884 | + mce_wrmsrq(MSR_AMD64_SMCA_MCx_DESTAT(m->bank), 0); | |
| 881 | 885 | |
| 882 | − /* Don't clear MCA_STATUS if MCA_DESTAT was used exclusively. */ | |
| 883 | − if (m->kflags & MCE_CHECK_DFR_REGS) | |
| 884 | − return; | |
| 886 | + /* Don't clear MCA_STATUS if MCA_DESTAT was used exclusively. */ | |
| 887 | + if (m->kflags & MCE_CHECK_DFR_REGS) | |
| 888 | + return; | |
| 889 | + } | |
| 885 | 890 | |
| 886 | 891 | mce_wrmsrq(mca_msr_reg(m->bank, MCA_STATUS), 0); |
| 887 | 892 | } |
| @@ -496,8 +496,9 @@ static void hv_reserve_irq_vectors(void) · 495 unmodified lines | ||
| 496 | 496 | test_and_set_bit(HYPERV_DBG_FASTFAIL_VECTOR, system_vectors)) |
| 497 | 497 | BUG(); |
| 498 | 498 | |
| 499 | − pr_info("Hyper-V: reserve vectors: %d %d %d\n", HYPERV_DBG_ASSERT_VECTOR, | |
| 500 | − HYPERV_DBG_SERVICE_VECTOR, HYPERV_DBG_FASTFAIL_VECTOR); | |
| 499 | + pr_info("Hyper-V: reserve vectors: 0x%x 0x%x 0x%x\n", | |
| 500 | + HYPERV_DBG_ASSERT_VECTOR, HYPERV_DBG_SERVICE_VECTOR, | |
| 501 | + HYPERV_DBG_FASTFAIL_VECTOR); | |
| 501 | 502 | } |
| 502 | 503 | |
| 503 | 504 | static void __init ms_hyperv_init_platform(void) |
| @@ -776,7 +776,10 @@ do { \ · 775 unmodified lines | ||
| 776 | 776 | #define SYNTHESIZED_F(name) \ |
| 777 | 777 | ({ \ |
| 778 | 778 | kvm_cpu_cap_synthesized |= feature_bit(name); \ |
| 779 | − F(name); \ | |
| 779 | + \ | |
| 780 | + BUILD_BUG_ON(X86_FEATURE_##name >= MAX_CPU_FEATURES); \ | |
| 781 | + if (boot_cpu_has(X86_FEATURE_##name)) \ | |
| 782 | + F(name); \ | |
| 780 | 783 | }) |
| 781 | 784 | |
| 782 | 785 | /* |
| @@ -1981,16 +1981,17 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu) · 1980 unmodified lines | ||
| 1981 | 1981 | if (entries[i] == KVM_HV_TLB_FLUSHALL_ENTRY) |
| 1982 | 1982 | goto out_flush_all; |
| 1983 | 1983 | |
| 1984 | − if (is_noncanonical_invlpg_address(entries[i], vcpu)) | |
| 1985 | − continue; | |
| 1986 | − | |
| 1987 | 1984 | /* |
| 1988 | 1985 | * Lower 12 bits of 'address' encode the number of additional |
| 1989 | 1986 | * pages to flush. |
| 1990 | 1987 | */ |
| 1991 | 1988 | gva = entries[i] & PAGE_MASK; |
| 1992 | − for (j = 0; j < (entries[i] & ~PAGE_MASK) + 1; j++) | |
| 1989 | + for (j = 0; j < (entries[i] & ~PAGE_MASK) + 1; j++) { | |
| 1990 | + if (is_noncanonical_invlpg_address(gva + j * PAGE_SIZE, vcpu)) | |
| 1991 | + continue; | |
| 1992 | + | |
| 1993 | 1993 | kvm_x86_call(flush_tlb_gva)(vcpu, gva + j * PAGE_SIZE); |
| 1994 | + } | |
| 1994 | 1995 | |
| 1995 | 1996 | ++vcpu->stat.tlb_flush; |
| 1996 | 1997 | } |
| @@ -321,7 +321,8 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin, · 320 unmodified lines | ||
| 321 | 321 | idx = srcu_read_lock(&kvm->irq_srcu); |
| 322 | 322 | gsi = kvm_irq_map_chip_pin(kvm, irqchip, pin); |
| 323 | 323 | if (gsi != -1) |
| 324 | − hlist_for_each_entry_rcu(kimn, &ioapic->mask_notifier_list, link) | |
| 324 | + hlist_for_each_entry_srcu(kimn, &ioapic->mask_notifier_list, link, | |
| 325 | + srcu_read_lock_held(&kvm->irq_srcu)) | |
| 325 | 326 | if (kimn->irq == gsi) |
| 326 | 327 | kimn->func(kimn, mask); |
| 327 | 328 | srcu_read_unlock(&kvm->irq_srcu, idx); |
| @@ -3044,12 +3044,6 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, · 3043 unmodified lines | ||
| 3044 | 3044 | bool prefetch = !fault || fault->prefetch; |
| 3045 | 3045 | bool write_fault = fault && fault->write; |
| 3046 | 3046 | |
| 3047 | − if (unlikely(is_noslot_pfn(pfn))) { | |
| 3048 | − vcpu->stat.pf_mmio_spte_created++; | |
| 3049 | − mark_mmio_spte(vcpu, sptep, gfn, pte_access); | |
| 3050 | − return RET_PF_EMULATE; | |
| 3051 | − } | |
| 3052 | − | |
| 3053 | 3047 | if (is_shadow_present_pte(*sptep)) { |
| 3054 | 3048 | if (prefetch && is_last_spte(*sptep, level) && |
| 3055 | 3049 | pfn == spte_to_pfn(*sptep)) |
| @@ -3066,13 +3060,22 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, · 10 unmodified lines | ||
| 3066 | 3060 | child = spte_to_child_sp(pte); |
| 3067 | 3061 | drop_parent_pte(vcpu->kvm, child, sptep); |
| 3068 | 3062 | flush = true; |
| 3069 | − } else if (WARN_ON_ONCE(pfn != spte_to_pfn(*sptep))) { | |
| 3063 | + } else if (pfn != spte_to_pfn(*sptep)) { | |
| 3064 | + WARN_ON_ONCE(vcpu->arch.mmu->root_role.direct); | |
| 3070 | 3065 | drop_spte(vcpu->kvm, sptep); |
| 3071 | 3066 | flush = true; |
| 3072 | 3067 | } else |
| 3073 | 3068 | was_rmapped = 1; |
| 3074 | 3069 | } |
| 3075 | 3070 | |
| 3071 | + if (unlikely(is_noslot_pfn(pfn))) { | |
| 3072 | + vcpu->stat.pf_mmio_spte_created++; | |
| 3073 | + mark_mmio_spte(vcpu, sptep, gfn, pte_access); | |
| 3074 | + if (flush) | |
| 3075 | + kvm_flush_remote_tlbs_gfn(vcpu->kvm, gfn, level); | |
| 3076 | + return RET_PF_EMULATE; | |
| 3077 | + } | |
| 3078 | + | |
| 3076 | 3079 | wrprot = make_spte(vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch, |
| 3077 | 3080 | false, host_writable, &spte); |
| 3078 | 3081 | |
| @@ -189,12 +189,12 @@ static void avic_activate_vmcb(struct vcpu_svm *svm) · 188 unmodified lines | ||
| 189 | 189 | struct kvm_vcpu *vcpu = &svm->vcpu; |
| 190 | 190 | |
| 191 | 191 | vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK); |
| 192 | − | |
| 193 | 192 | vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK; |
| 194 | 193 | vmcb->control.avic_physical_id |= avic_get_max_physical_id(vcpu); |
| 195 | − | |
| 196 | 194 | vmcb->control.int_ctl |= AVIC_ENABLE_MASK; |
| 197 | 195 | |
| 196 | + svm_clr_intercept(svm, INTERCEPT_CR8_WRITE); | |
| 197 | + | |
| 198 | 198 | /* |
| 199 | 199 | * Note: KVM supports hybrid-AVIC mode, where KVM emulates x2APIC MSR |
| 200 | 200 | * accesses, while interrupt injection to a running vCPU can be |
| @@ -226,6 +226,9 @@ static void avic_deactivate_vmcb(struct vcpu_svm *svm) · 25 unmodified lines | ||
| 226 | 226 | vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK); |
| 227 | 227 | vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK; |
| 228 | 228 | |
| 229 | + if (!sev_es_guest(svm->vcpu.kvm)) | |
| 230 | + svm_set_intercept(svm, INTERCEPT_CR8_WRITE); | |
| 231 | + | |
| 229 | 232 | /* |
| 230 | 233 | * If running nested and the guest uses its own MSR bitmap, there |
| 231 | 234 | * is no need to update L0's msr bitmap |
| @@ -368,7 +371,7 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb) · 136 unmodified lines | ||
| 368 | 371 | vmcb->control.avic_physical_id = __sme_set(__pa(kvm_svm->avic_physical_id_table)); |
| 369 | 372 | vmcb->control.avic_vapic_bar = APIC_DEFAULT_PHYS_BASE; |
| 370 | 373 | |
| 371 | − if (kvm_apicv_activated(svm->vcpu.kvm)) | |
| 374 | + if (kvm_vcpu_apicv_active(&svm->vcpu)) | |
| 372 | 375 | avic_activate_vmcb(svm); |
| 373 | 376 | else |
| 374 | 377 | avic_deactivate_vmcb(svm); |
| @@ -418,6 +418,15 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu) · 417 unmodified lines | ||
| 418 | 418 | return __nested_vmcb_check_controls(vcpu, ctl); |
| 419 | 419 | } |
| 420 | 420 | |
| 421 | +int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu) | |
| 422 | +{ | |
| 423 | + if (!nested_vmcb_check_save(vcpu) || | |
| 424 | + !nested_vmcb_check_controls(vcpu)) | |
| 425 | + return -EINVAL; | |
| 426 | + | |
| 427 | + return 0; | |
| 428 | +} | |
| 429 | + | |
| 421 | 430 | /* |
| 422 | 431 | * If a feature is not advertised to L1, clear the corresponding vmcb12 |
| 423 | 432 | * intercept. |
| @@ -1028,8 +1037,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) · 604 unmodified lines | ||
| 1028 | 1037 | nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); |
| 1029 | 1038 | nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); |
| 1030 | 1039 | |
| 1031 | − if (!nested_vmcb_check_save(vcpu) || | |
| 1032 | − !nested_vmcb_check_controls(vcpu)) { | |
| 1040 | + if (nested_svm_check_cached_vmcb12(vcpu) < 0) { | |
| 1033 | 1041 | vmcb12->control.exit_code = SVM_EXIT_ERR; |
| 1034 | 1042 | vmcb12->control.exit_info_1 = 0; |
| 1035 | 1043 | vmcb12->control.exit_info_2 = 0; |
| @@ -1077,8 +1077,7 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) · 1076 unmodified lines | ||
| 1077 | 1077 | svm_set_intercept(svm, INTERCEPT_CR0_WRITE); |
| 1078 | 1078 | svm_set_intercept(svm, INTERCEPT_CR3_WRITE); |
| 1079 | 1079 | svm_set_intercept(svm, INTERCEPT_CR4_WRITE); |
| 1080 | − if (!kvm_vcpu_apicv_active(vcpu)) | |
| 1081 | − svm_set_intercept(svm, INTERCEPT_CR8_WRITE); | |
| 1080 | + svm_set_intercept(svm, INTERCEPT_CR8_WRITE); | |
| 1082 | 1081 | |
| 1083 | 1082 | set_dr_intercepts(svm); |
| 1084 | 1083 | |
| @@ -1189,7 +1188,7 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) · 104 unmodified lines | ||
| 1189 | 1188 | if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS)) |
| 1190 | 1189 | svm->vmcb->control.erap_ctl |= ERAP_CONTROL_ALLOW_LARGER_RAP; |
| 1191 | 1190 | |
| 1192 | − if (kvm_vcpu_apicv_active(vcpu)) | |
| 1191 | + if (enable_apicv && irqchip_in_kernel(vcpu->kvm)) | |
| 1193 | 1192 | avic_init_vmcb(svm, vmcb); |
| 1194 | 1193 | |
| 1195 | 1194 | if (vnmi) |
| @@ -2674,9 +2673,11 @@ static int dr_interception(struct kvm_vcpu *vcpu) · 1478 unmodified lines | ||
| 2674 | 2673 | |
| 2675 | 2674 | static int cr8_write_interception(struct kvm_vcpu *vcpu) |
| 2676 | 2675 | { |
| 2676 | + u8 cr8_prev = kvm_get_cr8(vcpu); | |
| 2677 | 2677 | int r; |
| 2678 | 2678 | |
| 2679 | − u8 cr8_prev = kvm_get_cr8(vcpu); | |
| 2679 | + WARN_ON_ONCE(kvm_vcpu_apicv_active(vcpu)); | |
| 2680 | + | |
| 2680 | 2681 | /* instruction emulation calls kvm_set_cr8() */ |
| 2681 | 2682 | r = cr_interception(vcpu); |
| 2682 | 2683 | if (lapic_in_kernel(vcpu)) |
| @@ -4879,11 +4880,15 @@ static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) · 2196 unmodified lines | ||
| 4879 | 4880 | vmcb12 = map.hva; |
| 4880 | 4881 | nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); |
| 4881 | 4882 | nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); |
| 4882 | − ret = enter_svm_guest_mode(vcpu, smram64->svm_guest_vmcb_gpa, vmcb12, false); | |
| 4883 | 4883 | |
| 4884 | − if (ret) | |
| 4884 | + if (nested_svm_check_cached_vmcb12(vcpu) < 0) | |
| 4885 | + goto unmap_save; | |
| 4886 | + | |
| 4887 | + if (enter_svm_guest_mode(vcpu, smram64->svm_guest_vmcb_gpa, | |
| 4888 | + vmcb12, false) != 0) | |
| 4885 | 4889 | goto unmap_save; |
| 4886 | 4890 | |
| 4891 | + ret = 0; | |
| 4887 | 4892 | svm->nested.nested_run_pending = 1; |
| 4888 | 4893 | |
| 4889 | 4894 | unmap_save: |
| @@ -797,6 +797,7 @@ static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code) · 796 unmodified lines | ||
| 797 | 797 | |
| 798 | 798 | int nested_svm_exit_handled(struct vcpu_svm *svm); |
| 799 | 799 | int nested_svm_check_permissions(struct kvm_vcpu *vcpu); |
| 800 | +int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu); | |
| 800 | 801 | int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr, |
| 801 | 802 | bool has_error_code, u32 error_code); |
| 802 | 803 | int nested_svm_exit_special(struct vcpu_svm *svm); |
| @@ -3300,10 +3300,24 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu, · 3299 unmodified lines | ||
| 3300 | 3300 | if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP))) |
| 3301 | 3301 | return -EINVAL; |
| 3302 | 3302 | |
| 3303 | − if ((vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS) && | |
| 3304 | − (CC(!kvm_dr7_valid(vmcs12->guest_dr7)) || | |
| 3305 | − CC(!vmx_is_valid_debugctl(vcpu, vmcs12->guest_ia32_debugctl, false)))) | |
| 3306 | − return -EINVAL; | |
| 3303 | + if (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS) { | |
| 3304 | + u64 debugctl = vmcs12->guest_ia32_debugctl; | |
| 3305 | + | |
| 3306 | + /* | |
| 3307 | + * FREEZE_IN_SMM is not virtualized, but allow L1 to set it in | |
| 3308 | + * vmcs12's DEBUGCTL under a quirk for backwards compatibility. | |
| 3309 | + * Note that the quirk only relaxes the consistency check. The | |
| 3310 | + * vmcc02 bit is still under the control of the host. In | |
| 3311 | + * particular, if a host administrator decides to clear the bit, | |
| 3312 | + * then L1 has no say in the matter. | |
| 3313 | + */ | |
| 3314 | + if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM)) | |
| 3315 | + debugctl &= ~DEBUGCTLMSR_FREEZE_IN_SMM; | |
| 3316 | + | |
| 3317 | + if (CC(!kvm_dr7_valid(vmcs12->guest_dr7)) || | |
| 3318 | + CC(!vmx_is_valid_debugctl(vcpu, debugctl, false))) | |
| 3319 | + return -EINVAL; | |
| 3320 | + } | |
| 3307 | 3321 | |
| 3308 | 3322 | if ((vmcs12->vm_entry_controls & VM_ENTRY_LOAD_IA32_PAT) && |
| 3309 | 3323 | CC(!kvm_pat_valid(vmcs12->guest_ia32_pat))) |
| @@ -6842,13 +6856,34 @@ void vmx_leave_nested(struct kvm_vcpu *vcpu) · 3532 unmodified lines | ||
| 6842 | 6856 | free_nested(vcpu); |
| 6843 | 6857 | } |
| 6844 | 6858 | |
| 6859 | +int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu) | |
| 6860 | +{ | |
| 6861 | + enum vm_entry_failure_code ignored; | |
| 6862 | + struct vmcs12 *vmcs12 = get_vmcs12(vcpu); | |
| 6863 | + | |
| 6864 | + if (nested_cpu_has_shadow_vmcs(vmcs12) && | |
| 6865 | + vmcs12->vmcs_link_pointer != INVALID_GPA) { | |
| 6866 | + struct vmcs12 *shadow_vmcs12 = get_shadow_vmcs12(vcpu); | |
| 6867 | + | |
| 6868 | + if (shadow_vmcs12->hdr.revision_id != VMCS12_REVISION || | |
| 6869 | + !shadow_vmcs12->hdr.shadow_vmcs) | |
| 6870 | + return -EINVAL; | |
| 6871 | + } | |
| 6872 | + | |
| 6873 | + if (nested_vmx_check_controls(vcpu, vmcs12) || | |
| 6874 | + nested_vmx_check_host_state(vcpu, vmcs12) || | |
| 6875 | + nested_vmx_check_guest_state(vcpu, vmcs12, &ignored)) | |
| 6876 | + return -EINVAL; | |
| 6877 | + | |
| 6878 | + return 0; | |
| 6879 | +} | |
| 6880 | + | |
| 6845 | 6881 | static int vmx_set_nested_state(struct kvm_vcpu *vcpu, |
| 6846 | 6882 | struct kvm_nested_state __user *user_kvm_nested_state, |
| 6847 | 6883 | struct kvm_nested_state *kvm_state) |
| 6848 | 6884 | { |
| 6849 | 6885 | struct vcpu_vmx *vmx = to_vmx(vcpu); |
| 6850 | 6886 | struct vmcs12 *vmcs12; |
| 6851 | − enum vm_entry_failure_code ignored; | |
| 6852 | 6887 | struct kvm_vmx_nested_state_data __user *user_vmx_nested_state = |
| 6853 | 6888 | &user_kvm_nested_state->data.vmx[0]; |
| 6854 | 6889 | int ret; |
| @@ -6979,25 +7014,20 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu, · 124 unmodified lines | ||
| 6979 | 7014 | vmx->nested.mtf_pending = |
| 6980 | 7015 | !!(kvm_state->flags & KVM_STATE_NESTED_MTF_PENDING); |
| 6981 | 7016 | |
| 6982 | − ret = -EINVAL; | |
| 6983 | 7017 | if (nested_cpu_has_shadow_vmcs(vmcs12) && |
| 6984 | 7018 | vmcs12->vmcs_link_pointer != INVALID_GPA) { |
| 6985 | 7019 | struct vmcs12 *shadow_vmcs12 = get_shadow_vmcs12(vcpu); |
| 6986 | 7020 | |
| 7021 | + ret = -EINVAL; | |
| 6987 | 7022 | if (kvm_state->size < |
| 6988 | 7023 | sizeof(*kvm_state) + |
| 6989 | 7024 | sizeof(user_vmx_nested_state->vmcs12) + sizeof(*shadow_vmcs12)) |
| 6990 | 7025 | goto error_guest_mode; |
| 6991 | 7026 | |
| 7027 | + ret = -EFAULT; | |
| 6992 | 7028 | if (copy_from_user(shadow_vmcs12, |
| 6993 | 7029 | user_vmx_nested_state->shadow_vmcs12, |
| 6994 | − sizeof(*shadow_vmcs12))) { | |
| 6995 | − ret = -EFAULT; | |
| 6996 | − goto error_guest_mode; | |
| 6997 | − } | |
| 6998 | − | |
| 6999 | − if (shadow_vmcs12->hdr.revision_id != VMCS12_REVISION || | |
| 7000 | − !shadow_vmcs12->hdr.shadow_vmcs) | |
| 7030 | + sizeof(*shadow_vmcs12))) | |
| 7001 | 7031 | goto error_guest_mode; |
| 7002 | 7032 | } |
| 7003 | 7033 | |
| @@ -7008,9 +7038,8 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu, · 4 unmodified lines | ||
| 7008 | 7038 | kvm_state->hdr.vmx.preemption_timer_deadline; |
| 7009 | 7039 | } |
| 7010 | 7040 | |
| 7011 | − if (nested_vmx_check_controls(vcpu, vmcs12) || | |
| 7012 | − nested_vmx_check_host_state(vcpu, vmcs12) || | |
| 7013 | − nested_vmx_check_guest_state(vcpu, vmcs12, &ignored)) | |
| 7041 | + ret = nested_vmx_check_restored_vmcs12(vcpu); | |
| 7042 | + if (ret < 0) | |
| 7014 | 7043 | goto error_guest_mode; |
| 7015 | 7044 | |
| 7016 | 7045 | vmx->nested.dirty_vmcs12 = true; |
| @@ -22,6 +22,7 @@ void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps); · 21 unmodified lines | ||
| 22 | 22 | void nested_vmx_hardware_unsetup(void); |
| 23 | 23 | __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *)); |
| 24 | 24 | void nested_vmx_set_vmcs_shadowing_bitmap(void); |
| 25 | +int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu); | |
| 25 | 26 | void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu); |
| 26 | 27 | enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, |
| 27 | 28 | bool from_vmentry); |
| @@ -1149,7 +1149,7 @@ static void add_atomic_switch_msr(struct vcpu_vmx *vmx, unsigned msr, · 1148 unmodified lines | ||
| 1149 | 1149 | } |
| 1150 | 1150 | |
| 1151 | 1151 | vmx_add_auto_msr(&m->guest, msr, guest_val, VM_ENTRY_MSR_LOAD_COUNT, kvm); |
| 1152 | − vmx_add_auto_msr(&m->guest, msr, host_val, VM_EXIT_MSR_LOAD_COUNT, kvm); | |
| 1152 | + vmx_add_auto_msr(&m->host, msr, host_val, VM_EXIT_MSR_LOAD_COUNT, kvm); | |
| 1153 | 1153 | } |
| 1154 | 1154 | |
| 1155 | 1155 | static bool update_transition_efer(struct vcpu_vmx *vmx) |
| @@ -8528,9 +8528,13 @@ int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) · 7372 unmodified lines | ||
| 8528 | 8528 | } |
| 8529 | 8529 | |
| 8530 | 8530 | if (vmx->nested.smm.guest_mode) { |
| 8531 | + /* Triple fault if the state is invalid. */ | |
| 8532 | + if (nested_vmx_check_restored_vmcs12(vcpu) < 0) | |
| 8533 | + return 1; | |
| 8534 | + | |
| 8531 | 8535 | ret = nested_vmx_enter_non_root_mode(vcpu, false); |
| 8532 | − if (ret) | |
| 8533 | − return ret; | |
| 8536 | + if (ret != NVMX_VMENTRY_SUCCESS) | |
| 8537 | + return 1; | |
| 8534 | 8538 | |
| 8535 | 8539 | vmx->nested.nested_run_pending = 1; |
| 8536 | 8540 | vmx->nested.smm.guest_mode = false; |
| @@ -243,7 +243,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_ipiv); · 242 unmodified lines | ||
| 243 | 243 | bool __read_mostly enable_device_posted_irqs = true; |
| 244 | 244 | EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_device_posted_irqs); |
| 245 | 245 | |
| 246 | −const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 246 | +const struct kvm_stats_desc kvm_vm_stats_desc[] = { | |
| 247 | 247 | KVM_GENERIC_VM_STATS(), |
| 248 | 248 | STATS_DESC_COUNTER(VM, mmu_shadow_zapped), |
| 249 | 249 | STATS_DESC_COUNTER(VM, mmu_pte_write), |
| @@ -269,7 +269,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 19 unmodified lines | ||
| 269 | 269 | sizeof(kvm_vm_stats_desc), |
| 270 | 270 | }; |
| 271 | 271 | |
| 272 | −const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 272 | +const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { | |
| 273 | 273 | KVM_GENERIC_VCPU_STATS(), |
| 274 | 274 | STATS_DESC_COUNTER(VCPU, pf_taken), |
| 275 | 275 | STATS_DESC_COUNTER(VCPU, pf_fixed), |
| @@ -4,6 +4,8 @@ KCOV_INSTRUMENT_tlb.o := n · 3 unmodified lines | ||
| 4 | 4 | KCOV_INSTRUMENT_mem_encrypt.o := n |
| 5 | 5 | KCOV_INSTRUMENT_mem_encrypt_amd.o := n |
| 6 | 6 | KCOV_INSTRUMENT_pgprot.o := n |
| 7 | +# See the "Disable KCOV" comment in arch/x86/kernel/Makefile. | |
| 8 | +KCOV_INSTRUMENT_physaddr.o := n | |
| 7 | 9 | |
| 8 | 10 | KASAN_SANITIZE_mem_encrypt.o := n |
| 9 | 11 | KASAN_SANITIZE_mem_encrypt_amd.o := n |
| @@ -424,7 +424,7 @@ void __init efi_unmap_boot_services(void) · 423 unmodified lines | ||
| 424 | 424 | if (efi_enabled(EFI_DBG)) |
| 425 | 425 | return; |
| 426 | 426 | |
| 427 | − sz = sizeof(*ranges_to_free) * efi.memmap.nr_map + 1; | |
| 427 | + sz = sizeof(*ranges_to_free) * (efi.memmap.nr_map + 1); | |
| 428 | 428 | ranges_to_free = kzalloc(sz, GFP_KERNEL); |
| 429 | 429 | if (!ranges_to_free) { |
| 430 | 430 | pr_err("Failed to allocate storage for freeable EFI regions\n"); |
| @@ -28,8 +28,10 @@ static const struct software_node geode_gpio_keys_node = { · 27 unmodified lines | ||
| 28 | 28 | .properties = geode_gpio_keys_props, |
| 29 | 29 | }; |
| 30 | 30 | |
| 31 | −static struct property_entry geode_restart_key_props[] = { | |
| 32 | − { /* Placeholder for GPIO property */ }, | |
| 31 | +static struct software_node_ref_args geode_restart_gpio_ref; | |
| 32 | + | |
| 33 | +static const struct property_entry geode_restart_key_props[] = { | |
| 34 | + PROPERTY_ENTRY_REF_ARRAY_LEN("gpios", &geode_restart_gpio_ref, 1), | |
| 33 | 35 | PROPERTY_ENTRY_U32("linux,code", KEY_RESTART), |
| 34 | 36 | PROPERTY_ENTRY_STRING("label", "Reset button"), |
| 35 | 37 | PROPERTY_ENTRY_U32("debounce-interval", 100), |
| @@ -64,8 +66,7 @@ int __init geode_create_restart_key(unsigned int pin) · 28 unmodified lines | ||
| 64 | 66 | struct platform_device *pd; |
| 65 | 67 | int err; |
| 66 | 68 | |
| 67 | − geode_restart_key_props[0] = PROPERTY_ENTRY_GPIO("gpios", | |
| 68 | − &geode_gpiochip_node, | |
| 69 | + geode_restart_gpio_ref = SOFTWARE_NODE_REFERENCE(&geode_gpiochip_node, | |
| 69 | 70 | pin, GPIO_ACTIVE_LOW); |
| 70 | 71 | |
| 71 | 72 | err = software_node_register_node_group(geode_gpio_keys_swnodes); |
| @@ -99,6 +100,7 @@ int __init geode_create_leds(const char *label, const struct geode_led *leds, · 27 unmodified lines | ||
| 99 | 100 | const struct software_node *group[MAX_LEDS + 2] = { 0 }; |
| 100 | 101 | struct software_node *swnodes; |
| 101 | 102 | struct property_entry *props; |
| 103 | + struct software_node_ref_args *gpio_refs; | |
| 102 | 104 | struct platform_device_info led_info = { |
| 103 | 105 | .name = "leds-gpio", |
| 104 | 106 | .id = PLATFORM_DEVID_NONE, |
| @@ -127,6 +129,12 @@ int __init geode_create_leds(const char *label, const struct geode_led *leds, · 22 unmodified lines | ||
| 127 | 129 | goto err_free_swnodes; |
| 128 | 130 | } |
| 129 | 131 | |
| 132 | + gpio_refs = kzalloc_objs(*gpio_refs, n_leds); | |
| 133 | + if (!gpio_refs) { | |
| 134 | + err = -ENOMEM; | |
| 135 | + goto err_free_props; | |
| 136 | + } | |
| 137 | + | |
| 130 | 138 | group[0] = &geode_gpio_leds_node; |
| 131 | 139 | for (i = 0; i < n_leds; i++) { |
| 132 | 140 | node_name = kasprintf(GFP_KERNEL, "%s:%d", label, i); |
| @@ -135,9 +143,11 @@ int __init geode_create_leds(const char *label, const struct geode_led *leds, · 2 unmodified lines | ||
| 135 | 143 | goto err_free_names; |
| 136 | 144 | } |
| 137 | 145 | |
| 146 | + gpio_refs[i] = SOFTWARE_NODE_REFERENCE(&geode_gpiochip_node, | |
| 147 | + leds[i].pin, | |
| 148 | + GPIO_ACTIVE_LOW); | |
| 138 | 149 | props[i * 3 + 0] = |
| 139 | − PROPERTY_ENTRY_GPIO("gpios", &geode_gpiochip_node, | |
| 140 | − leds[i].pin, GPIO_ACTIVE_LOW); | |
| 150 | + PROPERTY_ENTRY_REF_ARRAY_LEN("gpios", &gpio_refs[i], 1); | |
| 141 | 151 | props[i * 3 + 1] = |
| 142 | 152 | PROPERTY_ENTRY_STRING("linux,default-trigger", |
| 143 | 153 | leds[i].default_on ? |
| @@ -171,6 +181,8 @@ err_unregister_group: · 27 unmodified lines | ||
| 171 | 181 | err_free_names: |
| 172 | 182 | while (--i >= 0) |
| 173 | 183 | kfree(swnodes[i].name); |
| 184 | + kfree(gpio_refs); | |
| 185 | +err_free_props: | |
| 174 | 186 | kfree(props); |
| 175 | 187 | err_free_swnodes: |
| 176 | 188 | kfree(swnodes); |
| @@ -623,8 +623,10 @@ static int af_alg_alloc_tsgl(struct sock *sk) · 622 unmodified lines | ||
| 623 | 623 | sg_init_table(sgl->sg, MAX_SGL_ENTS + 1); |
| 624 | 624 | sgl->cur = 0; |
| 625 | 625 | |
| 626 | − if (sg) | |
| 626 | + if (sg) { | |
| 627 | + sg_unmark_end(sg + MAX_SGL_ENTS - 1); | |
| 627 | 628 | sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg); |
| 629 | + } | |
| 628 | 630 | |
| 629 | 631 | list_add_tail(&sgl->list, &ctx->tsgl_list); |
| 630 | 632 | } |
| @@ -635,15 +637,13 @@ static int af_alg_alloc_tsgl(struct sock *sk) · 4 unmodified lines | ||
| 635 | 637 | /** |
| 636 | 638 | * af_alg_count_tsgl - Count number of TX SG entries |
| 637 | 639 | * |
| 638 | − * The counting starts from the beginning of the SGL to @bytes. If | |
| 639 | − * an @offset is provided, the counting of the SG entries starts at the @offset. | |
| 640 | + * The counting starts from the beginning of the SGL to @bytes. | |
| 640 | 641 | * |
| 641 | 642 | * @sk: socket of connection to user space |
| 642 | 643 | * @bytes: Count the number of SG entries holding given number of bytes. |
| 643 | − * @offset: Start the counting of SG entries from the given offset. | |
| 644 | 644 | * Return: Number of TX SG entries found given the constraints |
| 645 | 645 | */ |
| 646 | −unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset) | |
| 646 | +unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes) | |
| 647 | 647 | { |
| 648 | 648 | const struct alg_sock *ask = alg_sk(sk); |
| 649 | 649 | const struct af_alg_ctx *ctx = ask->private; |
| @@ -658,25 +658,11 @@ unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset) · 8 unmodified lines | ||
| 658 | 658 | const struct scatterlist *sg = sgl->sg; |
| 659 | 659 | |
| 660 | 660 | for (i = 0; i < sgl->cur; i++) { |
| 661 | − size_t bytes_count; | |
| 662 | − | |
| 663 | − /* Skip offset */ | |
| 664 | − if (offset >= sg[i].length) { | |
| 665 | − offset -= sg[i].length; | |
| 666 | − bytes -= sg[i].length; | |
| 667 | − continue; | |
| 668 | − } | |
| 669 | − | |
| 670 | − bytes_count = sg[i].length - offset; | |
| 671 | − | |
| 672 | − offset = 0; | |
| 673 | 661 | sgl_count++; |
| 674 | − | |
| 675 | − /* If we have seen requested number of bytes, stop */ | |
| 676 | − if (bytes_count >= bytes) | |
| 662 | + if (sg[i].length >= bytes) | |
| 677 | 663 | return sgl_count; |
| 678 | 664 | |
| 679 | − bytes -= bytes_count; | |
| 665 | + bytes -= sg[i].length; | |
| 680 | 666 | } |
| 681 | 667 | } |
| 682 | 668 | |
| @@ -688,19 +674,14 @@ EXPORT_SYMBOL_GPL(af_alg_count_tsgl); · 5 unmodified lines | ||
| 688 | 674 | * af_alg_pull_tsgl - Release the specified buffers from TX SGL |
| 689 | 675 | * |
| 690 | 676 | * If @dst is non-null, reassign the pages to @dst. The caller must release |
| 691 | − * the pages. If @dst_offset is given only reassign the pages to @dst starting | |
| 692 | − * at the @dst_offset (byte). The caller must ensure that @dst is large | |
| 693 | − * enough (e.g. by using af_alg_count_tsgl with the same offset). | |
| 677 | + * the pages. | |
| 694 | 678 | * |
| 695 | 679 | * @sk: socket of connection to user space |
| 696 | 680 | * @used: Number of bytes to pull from TX SGL |
| 697 | 681 | * @dst: If non-NULL, buffer is reassigned to dst SGL instead of releasing. The |
| 698 | 682 | * caller must release the buffers in dst. |
| 699 | − * @dst_offset: Reassign the TX SGL from given offset. All buffers before | |
| 700 | − * reaching the offset is released. | |
| 701 | 683 | */ |
| 702 | −void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst, | |
| 703 | − size_t dst_offset) | |
| 684 | +void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst) | |
| 704 | 685 | { |
| 705 | 686 | struct alg_sock *ask = alg_sk(sk); |
| 706 | 687 | struct af_alg_ctx *ctx = ask->private; |
| @@ -725,18 +706,10 @@ void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst, · 18 unmodified lines | ||
| 725 | 706 | * SG entries in dst. |
| 726 | 707 | */ |
| 727 | 708 | if (dst) { |
| 728 | − if (dst_offset >= plen) { | |
| 729 | − /* discard page before offset */ | |
| 730 | − dst_offset -= plen; | |
| 731 | − } else { | |
| 732 | − /* reassign page to dst after offset */ | |
| 733 | − get_page(page); | |
| 734 | − sg_set_page(dst + j, page, | |
| 735 | − plen - dst_offset, | |
| 736 | − sg[i].offset + dst_offset); | |
| 737 | − dst_offset = 0; | |
| 738 | − j++; | |
| 739 | − } | |
| 709 | + /* reassign page to dst after offset */ | |
| 710 | + get_page(page); | |
| 711 | + sg_set_page(dst + j, page, plen, sg[i].offset); | |
| 712 | + j++; | |
| 740 | 713 | } |
| 741 | 714 | |
| 742 | 715 | sg[i].length -= plen; |
| @@ -26,7 +26,6 @@ · 25 unmodified lines | ||
| 26 | 26 | #include <crypto/internal/aead.h> |
| 27 | 27 | #include <crypto/scatterwalk.h> |
| 28 | 28 | #include <crypto/if_alg.h> |
| 29 | −#include <crypto/skcipher.h> | |
| 30 | 29 | #include <linux/init.h> |
| 31 | 30 | #include <linux/list.h> |
| 32 | 31 | #include <linux/kernel.h> |
| @@ -72,9 +71,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, · 39 unmodified lines | ||
| 72 | 71 | struct alg_sock *pask = alg_sk(psk); |
| 73 | 72 | struct af_alg_ctx *ctx = ask->private; |
| 74 | 73 | struct crypto_aead *tfm = pask->private; |
| 75 | − unsigned int i, as = crypto_aead_authsize(tfm); | |
| 74 | + unsigned int as = crypto_aead_authsize(tfm); | |
| 76 | 75 | struct af_alg_async_req *areq; |
| 77 | − struct af_alg_tsgl *tsgl, *tmp; | |
| 78 | 76 | struct scatterlist *rsgl_src, *tsgl_src = NULL; |
| 79 | 77 | int err = 0; |
| 80 | 78 | size_t used = 0; /* [in] TX bufs to be en/decrypted */ |
| @@ -154,23 +152,24 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, · 73 unmodified lines | ||
| 154 | 152 | outlen -= less; |
| 155 | 153 | } |
| 156 | 154 | |
| 155 | + /* | |
| 156 | + * Create a per request TX SGL for this request which tracks the | |
| 157 | + * SG entries from the global TX SGL. | |
| 158 | + */ | |
| 157 | 159 | processed = used + ctx->aead_assoclen; |
| 158 | − list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) { | |
| 159 | − for (i = 0; i < tsgl->cur; i++) { | |
| 160 | − struct scatterlist *process_sg = tsgl->sg + i; | |
| 161 | − | |
| 162 | − if (!(process_sg->length) || !sg_page(process_sg)) | |
| 163 | − continue; | |
| 164 | − tsgl_src = process_sg; | |
| 165 | − break; | |
| 166 | − } | |
| 167 | − if (tsgl_src) | |
| 168 | − break; | |
| 169 | − } | |
| 170 | − if (processed && !tsgl_src) { | |
| 171 | − err = -EFAULT; | |
| 160 | + areq->tsgl_entries = af_alg_count_tsgl(sk, processed); | |
| 161 | + if (!areq->tsgl_entries) | |
| 162 | + areq->tsgl_entries = 1; | |
| 163 | + areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), | |
| 164 | + areq->tsgl_entries), | |
| 165 | + GFP_KERNEL); | |
| 166 | + if (!areq->tsgl) { | |
| 167 | + err = -ENOMEM; | |
| 172 | 168 | goto free; |
| 173 | 169 | } |
| 170 | + sg_init_table(areq->tsgl, areq->tsgl_entries); | |
| 171 | + af_alg_pull_tsgl(sk, processed, areq->tsgl); | |
| 172 | + tsgl_src = areq->tsgl; | |
| 174 | 173 | |
| 175 | 174 | /* |
| 176 | 175 | * Copy of AAD from source to destination |
| @@ -179,76 +178,15 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, · 2 unmodified lines | ||
| 179 | 178 | * when user space uses an in-place cipher operation, the kernel |
| 180 | 179 | * will copy the data as it does not see whether such in-place operation |
| 181 | 180 | * is initiated. |
| 182 | − * | |
| 183 | − * To ensure efficiency, the following implementation ensure that the | |
| 184 | − * ciphers are invoked to perform a crypto operation in-place. This | |
| 185 | − * is achieved by memory management specified as follows. | |
| 186 | 181 | */ |
| 187 | 182 | |
| 188 | 183 | /* Use the RX SGL as source (and destination) for crypto op. */ |
| 189 | 184 | rsgl_src = areq->first_rsgl.sgl.sgt.sgl; |
| 190 | 185 | |
| 191 | − if (ctx->enc) { | |
| 192 | − /* | |
| 193 | − * Encryption operation - The in-place cipher operation is | |
| 194 | − * achieved by the following operation: | |
| 195 | − * | |
| 196 | − * TX SGL: AAD || PT | |
| 197 | − * | | | |
| 198 | − * | copy | | |
| 199 | − * v v | |
| 200 | − * RX SGL: AAD || PT || Tag | |
| 201 | − */ | |
| 202 | − memcpy_sglist(areq->first_rsgl.sgl.sgt.sgl, tsgl_src, | |
| 203 | − processed); | |
| 204 | − af_alg_pull_tsgl(sk, processed, NULL, 0); | |
| 205 | − } else { | |
| 206 | − /* | |
| 207 | − * Decryption operation - To achieve an in-place cipher | |
| 208 | − * operation, the following SGL structure is used: | |
| 209 | − * | |
| 210 | − * TX SGL: AAD || CT || Tag | |
| 211 | − * | | ^ | |
| 212 | − * | copy | | Create SGL link. | |
| 213 | − * v v | | |
| 214 | − * RX SGL: AAD || CT ----+ | |
| 215 | − */ | |
| 216 | − | |
| 217 | − /* Copy AAD || CT to RX SGL buffer for in-place operation. */ | |
| 218 | − memcpy_sglist(areq->first_rsgl.sgl.sgt.sgl, tsgl_src, outlen); | |
| 219 | − | |
| 220 | − /* Create TX SGL for tag and chain it to RX SGL. */ | |
| 221 | − areq->tsgl_entries = af_alg_count_tsgl(sk, processed, | |
| 222 | − processed - as); | |
| 223 | − if (!areq->tsgl_entries) | |
| 224 | − areq->tsgl_entries = 1; | |
| 225 | − areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), | |
| 226 | − areq->tsgl_entries), | |
| 227 | − GFP_KERNEL); | |
| 228 | − if (!areq->tsgl) { | |
| 229 | − err = -ENOMEM; | |
| 230 | − goto free; | |
| 231 | − } | |
| 232 | − sg_init_table(areq->tsgl, areq->tsgl_entries); | |
| 233 | − | |
| 234 | − /* Release TX SGL, except for tag data and reassign tag data. */ | |
| 235 | − af_alg_pull_tsgl(sk, processed, areq->tsgl, processed - as); | |
| 236 | − | |
| 237 | − /* chain the areq TX SGL holding the tag with RX SGL */ | |
| 238 | − if (usedpages) { | |
| 239 | − /* RX SGL present */ | |
| 240 | − struct af_alg_sgl *sgl_prev = &areq->last_rsgl->sgl; | |
| 241 | − struct scatterlist *sg = sgl_prev->sgt.sgl; | |
| 242 | − | |
| 243 | − sg_unmark_end(sg + sgl_prev->sgt.nents - 1); | |
| 244 | − sg_chain(sg, sgl_prev->sgt.nents + 1, areq->tsgl); | |
| 245 | − } else | |
| 246 | − /* no RX SGL present (e.g. authentication only) */ | |
| 247 | − rsgl_src = areq->tsgl; | |
| 248 | − } | |
| 186 | + memcpy_sglist(rsgl_src, tsgl_src, ctx->aead_assoclen); | |
| 249 | 187 | |
| 250 | 188 | /* Initialize the crypto operation */ |
| 251 | − aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src, | |
| 189 | + aead_request_set_crypt(&areq->cra_u.aead_req, tsgl_src, | |
| 252 | 190 | areq->first_rsgl.sgl.sgt.sgl, used, ctx->iv); |
| 253 | 191 | aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen); |
| 254 | 192 | aead_request_set_tfm(&areq->cra_u.aead_req, tfm); |
| @@ -450,7 +388,7 @@ static void aead_sock_destruct(struct sock *sk) · 195 unmodified lines | ||
| 450 | 388 | struct crypto_aead *tfm = pask->private; |
| 451 | 389 | unsigned int ivlen = crypto_aead_ivsize(tfm); |
| 452 | 390 | |
| 453 | − af_alg_pull_tsgl(sk, ctx->used, NULL, 0); | |
| 391 | + af_alg_pull_tsgl(sk, ctx->used, NULL); | |
| 454 | 392 | sock_kzfree_s(sk, ctx->iv, ivlen); |
| 455 | 393 | sock_kfree_s(sk, ctx, ctx->len); |
| 456 | 394 | af_alg_release_parent(sk); |
| @@ -138,7 +138,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, · 137 unmodified lines | ||
| 138 | 138 | * Create a per request TX SGL for this request which tracks the |
| 139 | 139 | * SG entries from the global TX SGL. |
| 140 | 140 | */ |
| 141 | − areq->tsgl_entries = af_alg_count_tsgl(sk, len, 0); | |
| 141 | + areq->tsgl_entries = af_alg_count_tsgl(sk, len); | |
| 142 | 142 | if (!areq->tsgl_entries) |
| 143 | 143 | areq->tsgl_entries = 1; |
| 144 | 144 | areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), |
| @@ -149,7 +149,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, · 4 unmodified lines | ||
| 149 | 149 | goto free; |
| 150 | 150 | } |
| 151 | 151 | sg_init_table(areq->tsgl, areq->tsgl_entries); |
| 152 | − af_alg_pull_tsgl(sk, len, areq->tsgl, 0); | |
| 152 | + af_alg_pull_tsgl(sk, len, areq->tsgl); | |
| 153 | 153 | |
| 154 | 154 | /* Initialize the crypto operation */ |
| 155 | 155 | skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm); |
| @@ -363,7 +363,7 @@ static void skcipher_sock_destruct(struct sock *sk) · 207 unmodified lines | ||
| 363 | 363 | struct alg_sock *pask = alg_sk(psk); |
| 364 | 364 | struct crypto_skcipher *tfm = pask->private; |
| 365 | 365 | |
| 366 | − af_alg_pull_tsgl(sk, ctx->used, NULL, 0); | |
| 366 | + af_alg_pull_tsgl(sk, ctx->used, NULL); | |
| 367 | 367 | sock_kzfree_s(sk, ctx->iv, crypto_skcipher_ivsize(tfm)); |
| 368 | 368 | if (ctx->state) |
| 369 | 369 | sock_kzfree_s(sk, ctx->state, crypto_skcipher_statesize(tfm)); |
| @@ -207,6 +207,7 @@ static int crypto_authenc_esn_decrypt_tail(struct aead_request *req, · 206 unmodified lines | ||
| 207 | 207 | u8 *ohash = areq_ctx->tail; |
| 208 | 208 | unsigned int cryptlen = req->cryptlen - authsize; |
| 209 | 209 | unsigned int assoclen = req->assoclen; |
| 210 | + struct scatterlist *src = req->src; | |
| 210 | 211 | struct scatterlist *dst = req->dst; |
| 211 | 212 | u8 *ihash = ohash + crypto_ahash_digestsize(auth); |
| 212 | 213 | u32 tmp[2]; |
| @@ -214,23 +215,27 @@ static int crypto_authenc_esn_decrypt_tail(struct aead_request *req, · 1 unmodified line | ||
| 214 | 215 | if (!authsize) |
| 215 | 216 | goto decrypt; |
| 216 | 217 | |
| 217 | − /* Move high-order bits of sequence number back. */ | |
| 218 | − scatterwalk_map_and_copy(tmp, dst, 4, 4, 0); | |
| 219 | − scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 0); | |
| 220 | − scatterwalk_map_and_copy(tmp, dst, 0, 8, 1); | |
| 218 | + if (src == dst) { | |
| 219 | + /* Move high-order bits of sequence number back. */ | |
| 220 | + scatterwalk_map_and_copy(tmp, dst, 4, 4, 0); | |
| 221 | + scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 0); | |
| 222 | + scatterwalk_map_and_copy(tmp, dst, 0, 8, 1); | |
| 223 | + } else | |
| 224 | + memcpy_sglist(dst, src, assoclen); | |
| 221 | 225 | |
| 222 | 226 | if (crypto_memneq(ihash, ohash, authsize)) |
| 223 | 227 | return -EBADMSG; |
| 224 | 228 | |
| 225 | 229 | decrypt: |
| 226 | 230 | |
| 227 | − sg_init_table(areq_ctx->dst, 2); | |
| 231 | + if (src != dst) | |
| 232 | + src = scatterwalk_ffwd(areq_ctx->src, src, assoclen); | |
| 228 | 233 | dst = scatterwalk_ffwd(areq_ctx->dst, dst, assoclen); |
| 229 | 234 | |
| 230 | 235 | skcipher_request_set_tfm(skreq, ctx->enc); |
| 231 | 236 | skcipher_request_set_callback(skreq, flags, |
| 232 | 237 | req->base.complete, req->base.data); |
| 233 | − skcipher_request_set_crypt(skreq, dst, dst, cryptlen, req->iv); | |
| 238 | + skcipher_request_set_crypt(skreq, src, dst, cryptlen, req->iv); | |
| 234 | 239 | |
| 235 | 240 | return crypto_skcipher_decrypt(skreq); |
| 236 | 241 | } |
| @@ -255,6 +260,7 @@ static int crypto_authenc_esn_decrypt(struct aead_request *req) · 18 unmodified lines | ||
| 255 | 260 | unsigned int assoclen = req->assoclen; |
| 256 | 261 | unsigned int cryptlen = req->cryptlen; |
| 257 | 262 | u8 *ihash = ohash + crypto_ahash_digestsize(auth); |
| 263 | + struct scatterlist *src = req->src; | |
| 258 | 264 | struct scatterlist *dst = req->dst; |
| 259 | 265 | u32 tmp[2]; |
| 260 | 266 | int err; |
| @@ -262,24 +268,28 @@ static int crypto_authenc_esn_decrypt(struct aead_request *req) · 1 unmodified line | ||
| 262 | 268 | if (assoclen < 8) |
| 263 | 269 | return -EINVAL; |
| 264 | 270 | |
| 265 | − cryptlen -= authsize; | |
| 266 | − | |
| 267 | − if (req->src != dst) | |
| 268 | − memcpy_sglist(dst, req->src, assoclen + cryptlen); | |
| 271 | + if (!authsize) | |
| 272 | + goto tail; | |
| 269 | 273 | |
| 274 | + cryptlen -= authsize; | |
| 270 | 275 | scatterwalk_map_and_copy(ihash, req->src, assoclen + cryptlen, |
| 271 | 276 | authsize, 0); |
| 272 | 277 | |
| 273 | − if (!authsize) | |
| 274 | − goto tail; | |
| 275 | − | |
| 276 | 278 | /* Move high-order bits of sequence number to the end. */ |
| 277 | − scatterwalk_map_and_copy(tmp, dst, 0, 8, 0); | |
| 278 | − scatterwalk_map_and_copy(tmp, dst, 4, 4, 1); | |
| 279 | − scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 1); | |
| 280 | − | |
| 281 | − sg_init_table(areq_ctx->dst, 2); | |
| 282 | − dst = scatterwalk_ffwd(areq_ctx->dst, dst, 4); | |
| 279 | + scatterwalk_map_and_copy(tmp, src, 0, 8, 0); | |
| 280 | + if (src == dst) { | |
| 281 | + scatterwalk_map_and_copy(tmp, dst, 4, 4, 1); | |
| 282 | + scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 1); | |
| 283 | + dst = scatterwalk_ffwd(areq_ctx->dst, dst, 4); | |
| 284 | + } else { | |
| 285 | + scatterwalk_map_and_copy(tmp, dst, 0, 4, 1); | |
| 286 | + scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen - 4, 4, 1); | |
| 287 | + | |
| 288 | + src = scatterwalk_ffwd(areq_ctx->src, src, 8); | |
| 289 | + dst = scatterwalk_ffwd(areq_ctx->dst, dst, 4); | |
| 290 | + memcpy_sglist(dst, src, assoclen + cryptlen - 8); | |
| 291 | + dst = req->dst; | |
| 292 | + } | |
| 283 | 293 | |
| 284 | 294 | ahash_request_set_tfm(ahreq, auth); |
| 285 | 295 | ahash_request_set_crypt(ahreq, dst, ohash, assoclen + cryptlen); |
| @@ -164,18 +164,21 @@ static int deflate_decompress_one(struct acomp_req *req, · 163 unmodified lines | ||
| 164 | 164 | |
| 165 | 165 | do { |
| 166 | 166 | unsigned int dcur; |
| 167 | + unsigned long avail_in; | |
| 167 | 168 | |
| 168 | 169 | dcur = acomp_walk_next_dst(&walk); |
| 169 | − if (!dcur) { | |
| 170 | − out_of_space = true; | |
| 171 | − break; | |
| 172 | − } | |
| 173 | 170 | |
| 174 | 171 | stream->avail_out = dcur; |
| 175 | 172 | stream->next_out = walk.dst.virt.addr; |
| 173 | + avail_in = stream->avail_in; | |
| 176 | 174 | |
| 177 | 175 | ret = zlib_inflate(stream, Z_NO_FLUSH); |
| 178 | 176 | |
| 177 | + if (!dcur && avail_in == stream->avail_in) { | |
| 178 | + out_of_space = true; | |
| 179 | + break; | |
| 180 | + } | |
| 181 | + | |
| 179 | 182 | dcur -= stream->avail_out; |
| 180 | 183 | acomp_walk_done_dst(&walk, dcur); |
| 181 | 184 | } while (ret == Z_OK && stream->avail_in); |
| @@ -11,6 +11,7 @@ amdxdna-y := \ · 10 unmodified lines | ||
| 11 | 11 | aie2_solver.o \ |
| 12 | 12 | amdxdna_ctx.o \ |
| 13 | 13 | amdxdna_gem.o \ |
| 14 | + amdxdna_iommu.o \ | |
| 14 | 15 | amdxdna_mailbox.o \ |
| 15 | 16 | amdxdna_mailbox_helper.o \ |
| 16 | 17 | amdxdna_pci_drv.o \ |
| @@ -29,6 +29,16 @@ MODULE_PARM_DESC(force_cmdlist, "Force use command list (Default true)"); · 28 unmodified lines | ||
| 29 | 29 | |
| 30 | 30 | #define HWCTX_MAX_TIMEOUT 60000 /* milliseconds */ |
| 31 | 31 | |
| 32 | +struct aie2_ctx_health { | |
| 33 | + struct amdxdna_ctx_health header; | |
| 34 | + u32 txn_op_idx; | |
| 35 | + u32 ctx_pc; | |
| 36 | + u32 fatal_error_type; | |
| 37 | + u32 fatal_error_exception_type; | |
| 38 | + u32 fatal_error_exception_pc; | |
| 39 | + u32 fatal_error_app_module; | |
| 40 | +}; | |
| 41 | + | |
| 32 | 42 | static void aie2_job_release(struct kref *ref) |
| 33 | 43 | { |
| 34 | 44 | struct amdxdna_sched_job *job; |
| @@ -39,6 +49,7 @@ static void aie2_job_release(struct kref *ref) · 4 unmodified lines | ||
| 39 | 49 | wake_up(&job->hwctx->priv->job_free_wq); |
| 40 | 50 | if (job->out_fence) |
| 41 | 51 | dma_fence_put(job->out_fence); |
| 52 | + kfree(job->aie2_job_health); | |
| 42 | 53 | kfree(job); |
| 43 | 54 | } |
| 44 | 55 | |
| @@ -68,7 +79,8 @@ static int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hw · 23 unmodified lines | ||
| 68 | 79 | } |
| 69 | 80 | |
| 70 | 81 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, |
| 71 | − heap->mem.userptr, heap->mem.size); | |
| 82 | + amdxdna_obj_dma_addr(heap), | |
| 83 | + heap->mem.size); | |
| 72 | 84 | if (ret) { |
| 73 | 85 | XDNA_ERR(xdna, "Map host buf failed, ret %d", ret); |
| 74 | 86 | goto out; |
| @@ -165,7 +177,6 @@ aie2_sched_notify(struct amdxdna_sched_job *job) · 90 unmodified lines | ||
| 165 | 177 | |
| 166 | 178 | trace_xdna_job(&job->base, job->hwctx->name, "signaled fence", job->seq); |
| 167 | 179 | |
| 168 | − amdxdna_pm_suspend_put(job->hwctx->client->xdna); | |
| 169 | 180 | job->hwctx->priv->completed++; |
| 170 | 181 | dma_fence_signal(fence); |
| 171 | 182 | |
| @@ -175,6 +186,50 @@ aie2_sched_notify(struct amdxdna_sched_job *job) · 3 unmodified lines | ||
| 175 | 186 | aie2_job_put(job); |
| 176 | 187 | } |
| 177 | 188 | |
| 189 | +static void aie2_set_cmd_timeout(struct amdxdna_sched_job *job) | |
| 190 | +{ | |
| 191 | + struct aie2_ctx_health *aie2_health __free(kfree) = NULL; | |
| 192 | + struct amdxdna_dev *xdna = job->hwctx->client->xdna; | |
| 193 | + struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | |
| 194 | + struct app_health_report *report = job->aie2_job_health; | |
| 195 | + u32 fail_cmd_idx = 0; | |
| 196 | + | |
| 197 | + if (!report) | |
| 198 | + goto set_timeout; | |
| 199 | + | |
| 200 | + XDNA_ERR(xdna, "Firmware timeout state capture:"); | |
| 201 | + XDNA_ERR(xdna, "\tVersion: %d.%d", report->major, report->minor); | |
| 202 | + XDNA_ERR(xdna, "\tReport size: 0x%x", report->size); | |
| 203 | + XDNA_ERR(xdna, "\tContext ID: %d", report->context_id); | |
| 204 | + XDNA_ERR(xdna, "\tDPU PC: 0x%x", report->dpu_pc); | |
| 205 | + XDNA_ERR(xdna, "\tTXN OP ID: 0x%x", report->txn_op_id); | |
| 206 | + XDNA_ERR(xdna, "\tContext PC: 0x%x", report->ctx_pc); | |
| 207 | + XDNA_ERR(xdna, "\tFatal error type: 0x%x", report->fatal_info.fatal_type); | |
| 208 | + XDNA_ERR(xdna, "\tFatal error exception type: 0x%x", report->fatal_info.exception_type); | |
| 209 | + XDNA_ERR(xdna, "\tFatal error exception PC: 0x%x", report->fatal_info.exception_pc); | |
| 210 | + XDNA_ERR(xdna, "\tFatal error app module: 0x%x", report->fatal_info.app_module); | |
| 211 | + XDNA_ERR(xdna, "\tFatal error task ID: %d", report->fatal_info.task_index); | |
| 212 | + XDNA_ERR(xdna, "\tTimed out sub command ID: %d", report->run_list_id); | |
| 213 | + | |
| 214 | + fail_cmd_idx = report->run_list_id; | |
| 215 | + aie2_health = kzalloc_obj(*aie2_health); | |
| 216 | + if (!aie2_health) | |
| 217 | + goto set_timeout; | |
| 218 | + | |
| 219 | + aie2_health->header.version = AMDXDNA_CMD_CTX_HEALTH_V1; | |
| 220 | + aie2_health->header.npu_gen = AMDXDNA_CMD_CTX_HEALTH_AIE2; | |
| 221 | + aie2_health->txn_op_idx = report->txn_op_id; | |
| 222 | + aie2_health->ctx_pc = report->ctx_pc; | |
| 223 | + aie2_health->fatal_error_type = report->fatal_info.fatal_type; | |
| 224 | + aie2_health->fatal_error_exception_type = report->fatal_info.exception_type; | |
| 225 | + aie2_health->fatal_error_exception_pc = report->fatal_info.exception_pc; | |
| 226 | + aie2_health->fatal_error_app_module = report->fatal_info.app_module; | |
| 227 | + | |
| 228 | +set_timeout: | |
| 229 | + amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_TIMEOUT, | |
| 230 | + aie2_health, sizeof(*aie2_health)); | |
| 231 | +} | |
| 232 | + | |
| 178 | 233 | static int |
| 179 | 234 | aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) |
| 180 | 235 | { |
| @@ -186,13 +241,13 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) · 5 unmodified lines | ||
| 186 | 241 | cmd_abo = job->cmd_bo; |
| 187 | 242 | |
| 188 | 243 | if (unlikely(job->job_timeout)) { |
| 189 | − amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_TIMEOUT); | |
| 244 | + aie2_set_cmd_timeout(job); | |
| 190 | 245 | ret = -EINVAL; |
| 191 | 246 | goto out; |
| 192 | 247 | } |
| 193 | 248 | |
| 194 | 249 | if (unlikely(!data) || unlikely(size != sizeof(u32))) { |
| 195 | − amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT); | |
| 250 | + amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT, NULL, 0); | |
| 196 | 251 | ret = -EINVAL; |
| 197 | 252 | goto out; |
| 198 | 253 | } |
| @@ -202,7 +257,7 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) · 3 unmodified lines | ||
| 202 | 257 | if (status == AIE2_STATUS_SUCCESS) |
| 203 | 258 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_COMPLETED); |
| 204 | 259 | else |
| 205 | − amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ERROR); | |
| 260 | + amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ERROR, NULL, 0); | |
| 206 | 261 | |
| 207 | 262 | out: |
| 208 | 263 | aie2_sched_notify(job); |
| @@ -236,21 +291,21 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size) · 27 unmodified lines | ||
| 236 | 291 | struct amdxdna_sched_job *job = handle; |
| 237 | 292 | struct amdxdna_gem_obj *cmd_abo; |
| 238 | 293 | struct amdxdna_dev *xdna; |
| 294 | + u32 fail_cmd_idx = 0; | |
| 239 | 295 | u32 fail_cmd_status; |
| 240 | − u32 fail_cmd_idx; | |
| 241 | 296 | u32 cmd_status; |
| 242 | 297 | int ret = 0; |
| 243 | 298 | |
| 244 | 299 | cmd_abo = job->cmd_bo; |
| 245 | 300 | |
| 246 | 301 | if (unlikely(job->job_timeout)) { |
| 247 | − amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_TIMEOUT); | |
| 302 | + aie2_set_cmd_timeout(job); | |
| 248 | 303 | ret = -EINVAL; |
| 249 | 304 | goto out; |
| 250 | 305 | } |
| 251 | 306 | |
| 252 | 307 | if (unlikely(!data) || unlikely(size != sizeof(u32) * 3)) { |
| 253 | − amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT); | |
| 308 | + amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT, NULL, 0); | |
| 254 | 309 | ret = -EINVAL; |
| 255 | 310 | goto out; |
| 256 | 311 | } |
| @@ -270,10 +325,10 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size) · 13 unmodified lines | ||
| 270 | 325 | fail_cmd_idx, fail_cmd_status); |
| 271 | 326 | |
| 272 | 327 | if (fail_cmd_status == AIE2_STATUS_SUCCESS) { |
| 273 | − amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ABORT); | |
| 328 | + amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ABORT, NULL, 0); | |
| 274 | 329 | ret = -EINVAL; |
| 275 | 330 | } else { |
| 276 | − amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ERROR); | |
| 331 | + amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ERROR, NULL, 0); | |
| 277 | 332 | } |
| 278 | 333 | |
| 279 | 334 | out: |
| @@ -290,19 +345,11 @@ aie2_sched_job_run(struct drm_sched_job *sched_job) · 10 unmodified lines | ||
| 290 | 345 | struct dma_fence *fence; |
| 291 | 346 | int ret; |
| 292 | 347 | |
| 293 | − ret = amdxdna_pm_resume_get(hwctx->client->xdna); | |
| 294 | − if (ret) | |
| 348 | + if (!hwctx->priv->mbox_chann) | |
| 295 | 349 | return NULL; |
| 296 | 350 | |
| 297 | − if (!hwctx->priv->mbox_chann) { | |
| 298 | − amdxdna_pm_suspend_put(hwctx->client->xdna); | |
| 299 | − return NULL; | |
| 300 | − } | |
| 301 | − | |
| 302 | − if (!mmget_not_zero(job->mm)) { | |
| 303 | − amdxdna_pm_suspend_put(hwctx->client->xdna); | |
| 351 | + if (!mmget_not_zero(job->mm)) | |
| 304 | 352 | return ERR_PTR(-ESRCH); |
| 305 | − } | |
| 306 | 353 | |
| 307 | 354 | kref_get(&job->refcnt); |
| 308 | 355 | fence = dma_fence_get(job->fence); |
| @@ -333,7 +380,6 @@ aie2_sched_job_run(struct drm_sched_job *sched_job) · 24 unmodified lines | ||
| 333 | 380 | |
| 334 | 381 | out: |
| 335 | 382 | if (ret) { |
| 336 | − amdxdna_pm_suspend_put(hwctx->client->xdna); | |
| 337 | 383 | dma_fence_put(job->fence); |
| 338 | 384 | aie2_job_put(job); |
| 339 | 385 | mmput(job->mm); |
| @@ -362,12 +408,26 @@ aie2_sched_job_timedout(struct drm_sched_job *sched_job) · 22 unmodified lines | ||
| 362 | 408 | { |
| 363 | 409 | struct amdxdna_sched_job *job = drm_job_to_xdna_job(sched_job); |
| 364 | 410 | struct amdxdna_hwctx *hwctx = job->hwctx; |
| 411 | + struct app_health_report *report; | |
| 365 | 412 | struct amdxdna_dev *xdna; |
| 413 | + int ret; | |
| 366 | 414 | |
| 367 | 415 | xdna = hwctx->client->xdna; |
| 368 | 416 | trace_xdna_job(sched_job, hwctx->name, "job timedout", job->seq); |
| 369 | 417 | job->job_timeout = true; |
| 418 | + | |
| 370 | 419 | mutex_lock(&xdna->dev_lock); |
| 420 | + report = kzalloc_obj(*report); | |
| 421 | + if (!report) | |
| 422 | + goto reset_hwctx; | |
| 423 | + | |
| 424 | + ret = aie2_query_app_health(xdna->dev_handle, hwctx->fw_ctx_id, report); | |
| 425 | + if (ret) | |
| 426 | + kfree(report); | |
| 427 | + else | |
| 428 | + job->aie2_job_health = report; | |
| 429 | + | |
| 430 | +reset_hwctx: | |
| 371 | 431 | aie2_hwctx_stop(xdna, hwctx, sched_job); |
| 372 | 432 | |
| 373 | 433 | aie2_hwctx_restart(xdna, hwctx); |
| @@ -589,14 +649,14 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 215 unmodified lines | ||
| 589 | 649 | .size = MAX_CHAIN_CMDBUF_SIZE, |
| 590 | 650 | }; |
| 591 | 651 | |
| 592 | − abo = amdxdna_drm_alloc_dev_bo(&xdna->ddev, &args, client->filp); | |
| 652 | + abo = amdxdna_drm_create_dev_bo(&xdna->ddev, &args, client->filp); | |
| 593 | 653 | if (IS_ERR(abo)) { |
| 594 | 654 | ret = PTR_ERR(abo); |
| 595 | 655 | goto free_cmd_bufs; |
| 596 | 656 | } |
| 597 | 657 | |
| 598 | 658 | XDNA_DBG(xdna, "Command buf %d addr 0x%llx size 0x%lx", |
| 599 | − i, abo->mem.dev_addr, abo->mem.size); | |
| 659 | + i, amdxdna_gem_dev_addr(abo), abo->mem.size); | |
| 600 | 660 | priv->cmd_buf[i] = abo; |
| 601 | 661 | } |
| 602 | 662 | |
| @@ -637,7 +697,8 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 34 unmodified lines | ||
| 637 | 697 | } |
| 638 | 698 | |
| 639 | 699 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, |
| 640 | − heap->mem.userptr, heap->mem.size); | |
| 700 | + amdxdna_obj_dma_addr(heap), | |
| 701 | + heap->mem.size); | |
| 641 | 702 | if (ret) { |
| 642 | 703 | XDNA_ERR(xdna, "Map host buffer failed, ret %d", ret); |
| 643 | 704 | goto release_resource; |
| @@ -355,9 +355,8 @@ int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev) · 354 unmodified lines | ||
| 355 | 355 | return -ENOMEM; |
| 356 | 356 | |
| 357 | 357 | events->buf = aie2_alloc_msg_buffer(ndev, &total_size, &events->addr); |
| 358 | − | |
| 359 | − if (!events->buf) { | |
| 360 | − ret = -ENOMEM; | |
| 358 | + if (IS_ERR(events->buf)) { | |
| 359 | + ret = PTR_ERR(events->buf); | |
| 361 | 360 | goto free_events; |
| 362 | 361 | } |
| 363 | 362 | events->size = total_size; |
| @@ -56,16 +56,37 @@ void *aie2_alloc_msg_buffer(struct amdxdna_dev_hdl *ndev, u32 *size, · 55 unmodified lines | ||
| 56 | 56 | dma_addr_t *dma_addr) |
| 57 | 57 | { |
| 58 | 58 | struct amdxdna_dev *xdna = ndev->xdna; |
| 59 | + void *vaddr; | |
| 59 | 60 | int order; |
| 60 | 61 | |
| 61 | 62 | *size = max(*size, SZ_8K); |
| 62 | 63 | order = get_order(*size); |
| 63 | 64 | if (order > MAX_PAGE_ORDER) |
| 64 | − return NULL; | |
| 65 | + return ERR_PTR(-EINVAL); | |
| 65 | 66 | *size = PAGE_SIZE << order; |
| 66 | 67 | |
| 67 | − return dma_alloc_noncoherent(xdna->ddev.dev, *size, dma_addr, | |
| 68 | − DMA_FROM_DEVICE, GFP_KERNEL); | |
| 68 | + if (amdxdna_iova_on(xdna)) | |
| 69 | + return amdxdna_iommu_alloc(xdna, *size, dma_addr); | |
| 70 | + | |
| 71 | + vaddr = dma_alloc_noncoherent(xdna->ddev.dev, *size, dma_addr, | |
| 72 | + DMA_FROM_DEVICE, GFP_KERNEL); | |
| 73 | + if (!vaddr) | |
| 74 | + return ERR_PTR(-ENOMEM); | |
| 75 | + | |
| 76 | + return vaddr; | |
| 77 | +} | |
| 78 | + | |
| 79 | +void aie2_free_msg_buffer(struct amdxdna_dev_hdl *ndev, size_t size, | |
| 80 | + void *cpu_addr, dma_addr_t dma_addr) | |
| 81 | +{ | |
| 82 | + struct amdxdna_dev *xdna = ndev->xdna; | |
| 83 | + | |
| 84 | + if (amdxdna_iova_on(xdna)) { | |
| 85 | + amdxdna_iommu_free(xdna, size, cpu_addr, dma_addr); | |
| 86 | + return; | |
| 87 | + } | |
| 88 | + | |
| 89 | + dma_free_noncoherent(xdna->ddev.dev, size, cpu_addr, dma_addr, DMA_FROM_DEVICE); | |
| 69 | 90 | } |
| 70 | 91 | |
| 71 | 92 | int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev) |
| @@ -256,7 +277,7 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct · 184 unmodified lines | ||
| 256 | 277 | req.num_col = hwctx->num_col; |
| 257 | 278 | req.num_unused_col = hwctx->num_unused_col; |
| 258 | 279 | req.num_cq_pairs_requested = 1; |
| 259 | − req.pasid = hwctx->client->pasid; | |
| 280 | + req.pasid = amdxdna_pasid_on(hwctx->client) ? hwctx->client->pasid : 0; | |
| 260 | 281 | req.context_priority = aie2_get_context_priority(ndev, hwctx); |
| 261 | 282 | |
| 262 | 283 | ret = aie2_send_mgmt_msg_wait(ndev, &msg); |
| @@ -380,8 +401,8 @@ int aie2_query_status(struct amdxdna_dev_hdl *ndev, char __user *buf, · 117 unmodified lines | ||
| 380 | 401 | int ret; |
| 381 | 402 | |
| 382 | 403 | buff_addr = aie2_alloc_msg_buffer(ndev, &buf_sz, &dma_addr); |
| 383 | − if (!buff_addr) | |
| 384 | − return -ENOMEM; | |
| 404 | + if (IS_ERR(buff_addr)) | |
| 405 | + return PTR_ERR(buff_addr); | |
| 385 | 406 | |
| 386 | 407 | /* Go through each hardware context and mark the AIE columns that are active */ |
| 387 | 408 | list_for_each_entry(client, &xdna->client_list, node) |
| @@ -436,8 +457,8 @@ int aie2_query_telemetry(struct amdxdna_dev_hdl *ndev, · 48 unmodified lines | ||
| 436 | 457 | return -EINVAL; |
| 437 | 458 | |
| 438 | 459 | addr = aie2_alloc_msg_buffer(ndev, &buf_sz, &dma_addr); |
| 439 | − if (!addr) | |
| 440 | − return -ENOMEM; | |
| 460 | + if (IS_ERR(addr)) | |
| 461 | + return PTR_ERR(addr); | |
| 441 | 462 | |
| 442 | 463 | req.buf_addr = dma_addr; |
| 443 | 464 | req.buf_size = buf_sz; |
| @@ -532,10 +553,10 @@ int aie2_config_cu(struct amdxdna_hwctx *hwctx, · 88 unmodified lines | ||
| 532 | 553 | } |
| 533 | 554 | |
| 534 | 555 | req.cfgs[i] = FIELD_PREP(AIE2_MSG_CFG_CU_PDI_ADDR, |
| 535 | − abo->mem.dev_addr >> shift); | |
| 556 | + amdxdna_gem_dev_addr(abo) >> shift); | |
| 536 | 557 | req.cfgs[i] |= FIELD_PREP(AIE2_MSG_CFG_CU_FUNC, cu->cu_func); |
| 537 | 558 | XDNA_DBG(xdna, "CU %d full addr 0x%llx, cfg 0x%x", i, |
| 538 | − abo->mem.dev_addr, req.cfgs[i]); | |
| 559 | + amdxdna_gem_dev_addr(abo), req.cfgs[i]); | |
| 539 | 560 | drm_gem_object_put(gobj); |
| 540 | 561 | } |
| 541 | 562 | req.num_cus = hwctx->cus->num_cus; |
| @@ -982,56 +1003,78 @@ int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, · 440 unmodified lines | ||
| 982 | 1003 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 983 | 1004 | struct amdxdna_client *client = hwctx->client; |
| 984 | 1005 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 1006 | + void *cmd_buf = amdxdna_gem_vmap(cmdbuf_abo); | |
| 985 | 1007 | struct amdxdna_dev *xdna = client->xdna; |
| 986 | 1008 | struct amdxdna_cmd_chain *payload; |
| 987 | 1009 | struct xdna_mailbox_msg msg; |
| 988 | 1010 | union exec_chain_req req; |
| 989 | − u32 payload_len; | |
| 1011 | + u32 payload_len, ccnt; | |
| 990 | 1012 | u32 offset = 0; |
| 991 | 1013 | size_t size; |
| 992 | 1014 | int ret; |
| 993 | 1015 | u32 op; |
| 994 | 1016 | u32 i; |
| 995 | 1017 | |
| 1018 | + if (!cmd_buf) | |
| 1019 | + return -ENOMEM; | |
| 1020 | + | |
| 996 | 1021 | op = amdxdna_cmd_get_op(cmd_abo); |
| 997 | 1022 | payload = amdxdna_cmd_get_payload(cmd_abo, &payload_len); |
| 998 | − if (op != ERT_CMD_CHAIN || !payload || | |
| 999 | − payload_len < struct_size(payload, data, payload->command_count)) | |
| 1023 | + if (op != ERT_CMD_CHAIN) { | |
| 1024 | + XDNA_DBG(xdna, "Invalid op code %d", op); | |
| 1025 | + return -EINVAL; | |
| 1026 | + } | |
| 1027 | + | |
| 1028 | + if (!payload) { | |
| 1029 | + XDNA_DBG(xdna, "Failed to get command payload"); | |
| 1030 | + return -EINVAL; | |
| 1031 | + } | |
| 1032 | + | |
| 1033 | + ccnt = payload->command_count; | |
| 1034 | + if (payload_len < struct_size(payload, data, ccnt)) { | |
| 1035 | + XDNA_DBG(xdna, "Invalid command count %d", ccnt); | |
| 1000 | 1036 | return -EINVAL; |
| 1037 | + } | |
| 1001 | 1038 | |
| 1002 | 1039 | op = ERT_INVALID_CMD; |
| 1003 | − for (i = 0; i < payload->command_count; i++) { | |
| 1040 | + for (i = 0; i < ccnt; i++) { | |
| 1004 | 1041 | u32 boh = (u32)(payload->data[i]); |
| 1005 | 1042 | struct amdxdna_gem_obj *abo; |
| 1006 | 1043 | |
| 1007 | − abo = amdxdna_gem_get_obj(client, boh, AMDXDNA_BO_CMD); | |
| 1044 | + abo = amdxdna_gem_get_obj(client, boh, AMDXDNA_BO_SHARE); | |
| 1008 | 1045 | if (!abo) { |
| 1009 | 1046 | XDNA_ERR(xdna, "Failed to find cmd BO %d", boh); |
| 1010 | 1047 | return -ENOENT; |
| 1011 | 1048 | } |
| 1012 | 1049 | |
| 1013 | 1050 | size = cmdbuf_abo->mem.size - offset; |
| 1014 | − ret = aie2_cmdlist_fill_slot(cmdbuf_abo->mem.kva + offset, | |
| 1015 | − abo, &size, &op); | |
| 1051 | + ret = aie2_cmdlist_fill_slot(cmd_buf + offset, abo, &size, &op); | |
| 1016 | 1052 | amdxdna_gem_put_obj(abo); |
| 1017 | 1053 | if (ret) |
| 1018 | 1054 | return ret; |
| 1019 | 1055 | |
| 1020 | 1056 | offset += size; |
| 1021 | 1057 | } |
| 1058 | + | |
| 1059 | + XDNA_DBG(xdna, "Total %d commands:", ccnt); | |
| 1060 | + print_hex_dump_debug("cmdbufs: ", DUMP_PREFIX_OFFSET, 16, 4, | |
| 1061 | + cmd_buf, offset, false); | |
| 1062 | + | |
| 1022 | 1063 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); |
| 1023 | 1064 | if (msg.opcode == MSG_OP_MAX_OPCODE) |
| 1024 | 1065 | return -EOPNOTSUPP; |
| 1025 | 1066 | |
| 1026 | 1067 | /* The offset is the accumulated total size of the cmd buffer */ |
| 1027 | − EXEC_MSG_OPS(xdna)->init_chain_req(&req, cmdbuf_abo->mem.dev_addr, | |
| 1028 | − offset, payload->command_count); | |
| 1029 | − drm_clflush_virt_range(cmdbuf_abo->mem.kva, offset); | |
| 1068 | + EXEC_MSG_OPS(xdna)->init_chain_req(&req, amdxdna_gem_dev_addr(cmdbuf_abo), | |
| 1069 | + offset, ccnt); | |
| 1070 | + drm_clflush_virt_range(cmd_buf, offset); | |
| 1030 | 1071 | |
| 1031 | 1072 | msg.handle = job; |
| 1032 | 1073 | msg.notify_cb = notify_cb; |
| 1033 | 1074 | msg.send_data = (u8 *)&req; |
| 1034 | 1075 | msg.send_size = sizeof(req); |
| 1076 | + print_hex_dump_debug("cmdlist msg: ", DUMP_PREFIX_OFFSET, 16, 4, | |
| 1077 | + &req, msg.send_size, false); | |
| 1035 | 1078 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 1036 | 1079 | if (ret) { |
| 1037 | 1080 | XDNA_ERR(xdna, "Send message failed"); |
| @@ -1049,29 +1092,36 @@ int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx, · 11 unmodified lines | ||
| 1049 | 1092 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 1050 | 1093 | struct amdxdna_dev *xdna = hwctx->client->xdna; |
| 1051 | 1094 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 1095 | + void *cmd_buf = amdxdna_gem_vmap(cmdbuf_abo); | |
| 1052 | 1096 | struct xdna_mailbox_msg msg; |
| 1053 | 1097 | union exec_chain_req req; |
| 1054 | 1098 | u32 op = ERT_INVALID_CMD; |
| 1055 | 1099 | size_t size; |
| 1056 | 1100 | int ret; |
| 1057 | 1101 | |
| 1102 | + if (!cmd_buf) | |
| 1103 | + return -ENOMEM; | |
| 1104 | + | |
| 1058 | 1105 | size = cmdbuf_abo->mem.size; |
| 1059 | − ret = aie2_cmdlist_fill_slot(cmdbuf_abo->mem.kva, cmd_abo, &size, &op); | |
| 1106 | + ret = aie2_cmdlist_fill_slot(cmd_buf, cmd_abo, &size, &op); | |
| 1060 | 1107 | if (ret) |
| 1061 | 1108 | return ret; |
| 1062 | 1109 | |
| 1110 | + print_hex_dump_debug("cmdbuf: ", DUMP_PREFIX_OFFSET, 16, 4, cmd_buf, size, false); | |
| 1111 | + | |
| 1063 | 1112 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); |
| 1064 | 1113 | if (msg.opcode == MSG_OP_MAX_OPCODE) |
| 1065 | 1114 | return -EOPNOTSUPP; |
| 1066 | 1115 | |
| 1067 | − EXEC_MSG_OPS(xdna)->init_chain_req(&req, cmdbuf_abo->mem.dev_addr, | |
| 1068 | − size, 1); | |
| 1069 | − drm_clflush_virt_range(cmdbuf_abo->mem.kva, size); | |
| 1116 | + EXEC_MSG_OPS(xdna)->init_chain_req(&req, amdxdna_gem_dev_addr(cmdbuf_abo), size, 1); | |
| 1117 | + drm_clflush_virt_range(cmd_buf, size); | |
| 1070 | 1118 | |
| 1071 | 1119 | msg.handle = job; |
| 1072 | 1120 | msg.notify_cb = notify_cb; |
| 1073 | 1121 | msg.send_data = (u8 *)&req; |
| 1074 | 1122 | msg.send_size = sizeof(req); |
| 1123 | + print_hex_dump_debug("cmdlist msg: ", DUMP_PREFIX_OFFSET, 16, 4, | |
| 1124 | + &req, msg.send_size, false); | |
| 1075 | 1125 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 1076 | 1126 | if (ret) { |
| 1077 | 1127 | XDNA_ERR(hwctx->client->xdna, "Send message failed"); |
| @@ -1145,3 +1195,44 @@ int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job * · 67 unmodified lines | ||
| 1145 | 1195 | |
| 1146 | 1196 | return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 1147 | 1197 | } |
| 1198 | + | |
| 1199 | +int aie2_query_app_health(struct amdxdna_dev_hdl *ndev, u32 context_id, | |
| 1200 | + struct app_health_report *report) | |
| 1201 | +{ | |
| 1202 | + DECLARE_AIE2_MSG(get_app_health, MSG_OP_GET_APP_HEALTH); | |
| 1203 | + struct amdxdna_dev *xdna = ndev->xdna; | |
| 1204 | + struct app_health_report *buf; | |
| 1205 | + dma_addr_t dma_addr; | |
| 1206 | + u32 buf_size; | |
| 1207 | + int ret; | |
| 1208 | + | |
| 1209 | + if (!AIE2_FEATURE_ON(ndev, AIE2_APP_HEALTH)) { | |
| 1210 | + XDNA_DBG(xdna, "App health feature not supported"); | |
| 1211 | + return -EOPNOTSUPP; | |
| 1212 | + } | |
| 1213 | + | |
| 1214 | + buf_size = sizeof(*report); | |
| 1215 | + buf = aie2_alloc_msg_buffer(ndev, &buf_size, &dma_addr); | |
| 1216 | + if (IS_ERR(buf)) { | |
| 1217 | + XDNA_ERR(xdna, "Failed to allocate buffer for app health"); | |
| 1218 | + return PTR_ERR(buf); | |
| 1219 | + } | |
| 1220 | + | |
| 1221 | + req.buf_addr = dma_addr; | |
| 1222 | + req.context_id = context_id; | |
| 1223 | + req.buf_size = buf_size; | |
| 1224 | + | |
| 1225 | + drm_clflush_virt_range(buf, sizeof(*report)); | |
| 1226 | + ret = aie2_send_mgmt_msg_wait(ndev, &msg); | |
| 1227 | + if (ret) { | |
| 1228 | + XDNA_ERR(xdna, "Get app health failed, ret %d status 0x%x", ret, resp.status); | |
| 1229 | + goto free_buf; | |
| 1230 | + } | |
| 1231 | + | |
| 1232 | + /* Copy the report to caller's buffer */ | |
| 1233 | + memcpy(report, buf, sizeof(*report)); | |
| 1234 | + | |
| 1235 | +free_buf: | |
| 1236 | + aie2_free_msg_buffer(ndev, buf_size, buf, dma_addr); | |
| 1237 | + return ret; | |
| 1238 | +} | |
| @@ -31,6 +31,7 @@ enum aie2_msg_opcode { · 30 unmodified lines | ||
| 31 | 31 | MSG_OP_SET_RUNTIME_CONFIG = 0x10A, |
| 32 | 32 | MSG_OP_GET_RUNTIME_CONFIG = 0x10B, |
| 33 | 33 | MSG_OP_REGISTER_ASYNC_EVENT_MSG = 0x10C, |
| 34 | + MSG_OP_GET_APP_HEALTH = 0x114, | |
| 34 | 35 | MSG_OP_MAX_DRV_OPCODE, |
| 35 | 36 | MSG_OP_GET_PROTOCOL_VERSION = 0x301, |
| 36 | 37 | MSG_OP_MAX_OPCODE |
| @@ -451,4 +452,55 @@ struct config_debug_bo_req { · 414 unmodified lines | ||
| 451 | 452 | struct config_debug_bo_resp { |
| 452 | 453 | enum aie2_msg_status status; |
| 453 | 454 | } __packed; |
| 455 | + | |
| 456 | +struct fatal_error_info { | |
| 457 | + __u32 fatal_type; /* Fatal error type */ | |
| 458 | + __u32 exception_type; /* Only valid if fatal_type is a specific value */ | |
| 459 | + __u32 exception_argument; /* Argument based on exception type */ | |
| 460 | + __u32 exception_pc; /* Program Counter at the time of the exception */ | |
| 461 | + __u32 app_module; /* Error module name */ | |
| 462 | + __u32 task_index; /* Index of the task in which the error occurred */ | |
| 463 | + __u32 reserved[128]; | |
| 464 | +}; | |
| 465 | + | |
| 466 | +struct app_health_report { | |
| 467 | + __u16 major; | |
| 468 | + __u16 minor; | |
| 469 | + __u32 size; | |
| 470 | + __u32 context_id; | |
| 471 | + /* | |
| 472 | + * Program Counter (PC) of the last initiated DPU opcode, as reported by the ERT | |
| 473 | + * application. Before execution begins or after successful completion, the value is set | |
| 474 | + * to UINT_MAX. If execution halts prematurely due to an error, this field retains the | |
| 475 | + * opcode's PC value. | |
| 476 | + * Note: To optimize performance, the ERT may simplify certain aspects of reporting. | |
| 477 | + * Proper interpretation requires familiarity with the implementation details. | |
| 478 | + */ | |
| 479 | + __u32 dpu_pc; | |
| 480 | + /* | |
| 481 | + * Index of the last initiated TXN opcode. | |
| 482 | + * Before execution starts or after successful completion, the value is set to UINT_MAX. | |
| 483 | + * If execution halts prematurely due to an error, this field retains the opcode's ID. | |
| 484 | + * Note: To optimize performance, the ERT may simplify certain aspects of reporting. | |
| 485 | + * Proper interpretation requires familiarity with the implementation details. | |
| 486 | + */ | |
| 487 | + __u32 txn_op_id; | |
| 488 | + /* The PC of the context at the time of the report */ | |
| 489 | + __u32 ctx_pc; | |
| 490 | + struct fatal_error_info fatal_info; | |
| 491 | + /* Index of the most recently executed run list entry. */ | |
| 492 | + __u32 run_list_id; | |
| 493 | +}; | |
| 494 | + | |
| 495 | +struct get_app_health_req { | |
| 496 | + __u32 context_id; | |
| 497 | + __u32 buf_size; | |
| 498 | + __u64 buf_addr; | |
| 499 | +} __packed; | |
| 500 | + | |
| 501 | +struct get_app_health_resp { | |
| 502 | + enum aie2_msg_status status; | |
| 503 | + __u32 required_buffer_size; | |
| 504 | + __u32 reserved[7]; | |
| 505 | +} __packed; | |
| 454 | 506 | #endif /* _AIE2_MSG_PRIV_H_ */ |
| @@ -846,7 +846,10 @@ static int aie2_hwctx_status_cb(struct amdxdna_hwctx *hwctx, void *arg) · 845 unmodified lines | ||
| 846 | 846 | struct amdxdna_drm_hwctx_entry *tmp __free(kfree) = NULL; |
| 847 | 847 | struct amdxdna_drm_get_array *array_args = arg; |
| 848 | 848 | struct amdxdna_drm_hwctx_entry __user *buf; |
| 849 | + struct app_health_report report; | |
| 850 | + struct amdxdna_dev_hdl *ndev; | |
| 849 | 851 | u32 size; |
| 852 | + int ret; | |
| 850 | 853 | |
| 851 | 854 | if (!array_args->num_element) |
| 852 | 855 | return -EINVAL; |
| @@ -862,6 +865,7 @@ static int aie2_hwctx_status_cb(struct amdxdna_hwctx *hwctx, void *arg) · 9 unmodified lines | ||
| 862 | 865 | tmp->command_submissions = hwctx->priv->seq; |
| 863 | 866 | tmp->command_completions = hwctx->priv->completed; |
| 864 | 867 | tmp->pasid = hwctx->client->pasid; |
| 868 | + tmp->heap_usage = hwctx->client->heap_usage; | |
| 865 | 869 | tmp->priority = hwctx->qos.priority; |
| 866 | 870 | tmp->gops = hwctx->qos.gops; |
| 867 | 871 | tmp->fps = hwctx->qos.fps; |
| @@ -869,6 +873,17 @@ static int aie2_hwctx_status_cb(struct amdxdna_hwctx *hwctx, void *arg) · 1 unmodified line | ||
| 869 | 873 | tmp->latency = hwctx->qos.latency; |
| 870 | 874 | tmp->frame_exec_time = hwctx->qos.frame_exec_time; |
| 871 | 875 | tmp->state = AMDXDNA_HWCTX_STATE_ACTIVE; |
| 876 | + ndev = hwctx->client->xdna->dev_handle; | |
| 877 | + ret = aie2_query_app_health(ndev, hwctx->fw_ctx_id, &report); | |
| 878 | + if (!ret) { | |
| 879 | + /* Fill in app health report fields */ | |
| 880 | + tmp->txn_op_idx = report.txn_op_id; | |
| 881 | + tmp->ctx_pc = report.ctx_pc; | |
| 882 | + tmp->fatal_error_type = report.fatal_info.fatal_type; | |
| 883 | + tmp->fatal_error_exception_type = report.fatal_info.exception_type; | |
| 884 | + tmp->fatal_error_exception_pc = report.fatal_info.exception_pc; | |
| 885 | + tmp->fatal_error_app_module = report.fatal_info.app_module; | |
| 886 | + } | |
| 872 | 887 | |
| 873 | 888 | buf = u64_to_user_ptr(array_args->buffer); |
| 874 | 889 | size = min(sizeof(*tmp), array_args->element_size); |
| @@ -1134,6 +1149,9 @@ static int aie2_get_array(struct amdxdna_client *client, · 259 unmodified lines | ||
| 1134 | 1149 | case DRM_AMDXDNA_HW_LAST_ASYNC_ERR: |
| 1135 | 1150 | ret = aie2_get_array_async_error(xdna->dev_handle, args); |
| 1136 | 1151 | break; |
| 1152 | + case DRM_AMDXDNA_BO_USAGE: | |
| 1153 | + ret = amdxdna_drm_get_bo_usage(&xdna->ddev, args); | |
| 1154 | + break; | |
| 1137 | 1155 | default: |
| 1138 | 1156 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); |
| 1139 | 1157 | ret = -EOPNOTSUPP; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/limits.h> |
| 11 | 11 | #include <linux/semaphore.h> |
| 12 | 12 | |
| 13 | +#include "aie2_msg_priv.h" | |
| 13 | 14 | #include "amdxdna_mailbox.h" |
| 14 | 15 | |
| 15 | 16 | #define AIE2_INTERVAL 20000 /* us */ |
| @@ -261,6 +262,7 @@ enum aie2_fw_feature { · 245 unmodified lines | ||
| 261 | 262 | AIE2_NPU_COMMAND, |
| 262 | 263 | AIE2_PREEMPT, |
| 263 | 264 | AIE2_TEMPORAL_ONLY, |
| 265 | + AIE2_APP_HEALTH, | |
| 264 | 266 | AIE2_FEATURE_MAX |
| 265 | 267 | }; |
| 266 | 268 | |
| @@ -271,6 +273,7 @@ struct aie2_fw_feature_tbl { · 4 unmodified lines | ||
| 271 | 273 | u32 min_minor; |
| 272 | 274 | }; |
| 273 | 275 | |
| 276 | +#define AIE2_ALL_FEATURES GENMASK_ULL(AIE2_FEATURE_MAX - 1, AIE2_NPU_COMMAND) | |
| 274 | 277 | #define AIE2_FEATURE_ON(ndev, feature) test_bit(feature, &(ndev)->feature_mask) |
| 275 | 278 | |
| 276 | 279 | struct amdxdna_dev_priv { |
| @@ -341,6 +344,8 @@ int aie2_query_aie_version(struct amdxdna_dev_hdl *ndev, struct aie_version *ver · 64 unmodified lines | ||
| 341 | 344 | int aie2_query_aie_metadata(struct amdxdna_dev_hdl *ndev, struct aie_metadata *metadata); |
| 342 | 345 | int aie2_query_firmware_version(struct amdxdna_dev_hdl *ndev, |
| 343 | 346 | struct amdxdna_fw_ver *fw_ver); |
| 347 | +int aie2_query_app_health(struct amdxdna_dev_hdl *ndev, u32 context_id, | |
| 348 | + struct app_health_report *report); | |
| 344 | 349 | int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); |
| 345 | 350 | int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); |
| 346 | 351 | int aie2_map_host_buf(struct amdxdna_dev_hdl *ndev, u32 context_id, u64 addr, u64 size); |
| @@ -366,9 +371,8 @@ int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job * · 19 unmodified lines | ||
| 366 | 371 | int (*notify_cb)(void *, void __iomem *, size_t)); |
| 367 | 372 | void *aie2_alloc_msg_buffer(struct amdxdna_dev_hdl *ndev, u32 *size, |
| 368 | 373 | dma_addr_t *dma_addr); |
| 369 | −#define aie2_free_msg_buffer(ndev, size, buff_addr, dma_addr) \ | |
| 370 | − dma_free_noncoherent((ndev)->xdna->ddev.dev, size, buff_addr, \ | |
| 371 | − dma_addr, DMA_FROM_DEVICE) | |
| 374 | +void aie2_free_msg_buffer(struct amdxdna_dev_hdl *ndev, size_t size, | |
| 375 | + void *cpu_addr, dma_addr_t dma_addr); | |
| 372 | 376 | |
| 373 | 377 | /* aie2_hwctx.c */ |
| 374 | 378 | int aie2_hwctx_init(struct amdxdna_hwctx *hwctx); |
| @@ -17,6 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include "amdxdna_ctx.h" |
| 18 | 18 | #include "amdxdna_gem.h" |
| 19 | 19 | #include "amdxdna_pci_drv.h" |
| 20 | +#include "amdxdna_pm.h" | |
| 20 | 21 | |
| 21 | 22 | #define MAX_HWCTX_ID 255 |
| 22 | 23 | #define MAX_ARG_COUNT 4095 |
| @@ -94,9 +95,12 @@ int amdxdna_hwctx_walk(struct amdxdna_client *client, void *arg, · 71 unmodified lines | ||
| 94 | 95 | |
| 95 | 96 | void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size) |
| 96 | 97 | { |
| 97 | − struct amdxdna_cmd *cmd = abo->mem.kva; | |
| 98 | + struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); | |
| 98 | 99 | u32 num_masks, count; |
| 99 | 100 | |
| 101 | + if (!cmd) | |
| 102 | + return NULL; | |
| 103 | + | |
| 100 | 104 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) |
| 101 | 105 | num_masks = 0; |
| 102 | 106 | else |
| @@ -118,10 +122,13 @@ void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size) · 15 unmodified lines | ||
| 118 | 122 | |
| 119 | 123 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) |
| 120 | 124 | { |
| 121 | − struct amdxdna_cmd *cmd = abo->mem.kva; | |
| 125 | + struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); | |
| 122 | 126 | u32 num_masks, i; |
| 123 | 127 | u32 *cu_mask; |
| 124 | 128 | |
| 129 | + if (!cmd) | |
| 130 | + return INVALID_CU_IDX; | |
| 131 | + | |
| 125 | 132 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) |
| 126 | 133 | return INVALID_CU_IDX; |
| 127 | 134 | |
| @@ -137,25 +144,34 @@ u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) · 9 unmodified lines | ||
| 137 | 144 | |
| 138 | 145 | int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo, |
| 139 | 146 | struct amdxdna_sched_job *job, u32 cmd_idx, |
| 140 | − enum ert_cmd_state error_state) | |
| 147 | + enum ert_cmd_state error_state, | |
| 148 | + void *err_data, size_t size) | |
| 141 | 149 | { |
| 142 | 150 | struct amdxdna_client *client = job->hwctx->client; |
| 143 | − struct amdxdna_cmd *cmd = abo->mem.kva; | |
| 151 | + struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); | |
| 144 | 152 | struct amdxdna_cmd_chain *cc = NULL; |
| 145 | 153 | |
| 154 | + if (!cmd) | |
| 155 | + return -ENOMEM; | |
| 156 | + | |
| 146 | 157 | cmd->header &= ~AMDXDNA_CMD_STATE; |
| 147 | 158 | cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, error_state); |
| 148 | 159 | |
| 149 | 160 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) { |
| 150 | 161 | cc = amdxdna_cmd_get_payload(abo, NULL); |
| 151 | 162 | cc->error_index = (cmd_idx < cc->command_count) ? cmd_idx : 0; |
| 152 | − abo = amdxdna_gem_get_obj(client, cc->data[0], AMDXDNA_BO_CMD); | |
| 163 | + abo = amdxdna_gem_get_obj(client, cc->data[0], AMDXDNA_BO_SHARE); | |
| 153 | 164 | if (!abo) |
| 154 | 165 | return -EINVAL; |
| 155 | − cmd = abo->mem.kva; | |
| 166 | + cmd = amdxdna_gem_vmap(abo); | |
| 167 | + if (!cmd) | |
| 168 | + return -ENOMEM; | |
| 156 | 169 | } |
| 157 | 170 | |
| 158 | 171 | memset(cmd->data, 0xff, abo->mem.size - sizeof(*cmd)); |
| 172 | + if (err_data) | |
| 173 | + memcpy(cmd->data, err_data, min(size, abo->mem.size - sizeof(*cmd))); | |
| 174 | + | |
| 159 | 175 | if (cc) |
| 160 | 176 | amdxdna_gem_put_obj(abo); |
| 161 | 177 | |
| @@ -445,6 +461,7 @@ put_shmem_bo: · 283 unmodified lines | ||
| 445 | 461 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job) |
| 446 | 462 | { |
| 447 | 463 | trace_amdxdna_debug_point(job->hwctx->name, job->seq, "job release"); |
| 464 | + amdxdna_pm_suspend_put(job->hwctx->client->xdna); | |
| 448 | 465 | amdxdna_arg_bos_put(job); |
| 449 | 466 | amdxdna_gem_put_obj(job->cmd_bo); |
| 450 | 467 | dma_fence_put(job->fence); |
| @@ -468,7 +485,7 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 17 unmodified lines | ||
| 468 | 485 | job->drv_cmd = drv_cmd; |
| 469 | 486 | |
| 470 | 487 | if (cmd_bo_hdl != AMDXDNA_INVALID_BO_HANDLE) { |
| 471 | − job->cmd_bo = amdxdna_gem_get_obj(client, cmd_bo_hdl, AMDXDNA_BO_CMD); | |
| 488 | + job->cmd_bo = amdxdna_gem_get_obj(client, cmd_bo_hdl, AMDXDNA_BO_SHARE); | |
| 472 | 489 | if (!job->cmd_bo) { |
| 473 | 490 | XDNA_ERR(xdna, "Failed to get cmd bo from %d", cmd_bo_hdl); |
| 474 | 491 | ret = -EINVAL; |
| @@ -482,6 +499,12 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 7 unmodified lines | ||
| 482 | 499 | goto cmd_put; |
| 483 | 500 | } |
| 484 | 501 | |
| 502 | + ret = amdxdna_pm_resume_get(xdna); | |
| 503 | + if (ret) { | |
| 504 | + XDNA_ERR(xdna, "Resume failed, ret %d", ret); | |
| 505 | + goto put_bos; | |
| 506 | + } | |
| 507 | + | |
| 485 | 508 | idx = srcu_read_lock(&client->hwctx_srcu); |
| 486 | 509 | hwctx = xa_load(&client->hwctx_xa, hwctx_hdl); |
| 487 | 510 | if (!hwctx) { |
| @@ -522,6 +545,8 @@ put_fence: · 34 unmodified lines | ||
| 522 | 545 | dma_fence_put(job->fence); |
| 523 | 546 | unlock_srcu: |
| 524 | 547 | srcu_read_unlock(&client->hwctx_srcu, idx); |
| 548 | + amdxdna_pm_suspend_put(xdna); | |
| 549 | +put_bos: | |
| 525 | 550 | amdxdna_arg_bos_put(job); |
| 526 | 551 | cmd_put: |
| 527 | 552 | amdxdna_gem_put_obj(job->cmd_bo); |
| @@ -72,6 +72,13 @@ struct amdxdna_cmd_preempt_data { · 71 unmodified lines | ||
| 72 | 72 | u32 prop_args[]; /* properties and regular kernel arguments */ |
| 73 | 73 | }; |
| 74 | 74 | |
| 75 | +#define AMDXDNA_CMD_CTX_HEALTH_V1 1 | |
| 76 | +#define AMDXDNA_CMD_CTX_HEALTH_AIE2 0 | |
| 77 | +struct amdxdna_ctx_health { | |
| 78 | + u32 version; | |
| 79 | + u32 npu_gen; | |
| 80 | +}; | |
| 81 | + | |
| 75 | 82 | /* Exec buffer command header format */ |
| 76 | 83 | #define AMDXDNA_CMD_STATE GENMASK(3, 0) |
| 77 | 84 | #define AMDXDNA_CMD_EXTRA_CU_MASK GENMASK(11, 10) |
| @@ -122,6 +129,11 @@ struct amdxdna_drv_cmd { · 44 unmodified lines | ||
| 122 | 129 | u32 result; |
| 123 | 130 | }; |
| 124 | 131 | |
| 132 | +struct app_health_report; | |
| 133 | +union amdxdna_job_priv { | |
| 134 | + struct app_health_report *aie2_health; | |
| 135 | +}; | |
| 136 | + | |
| 125 | 137 | struct amdxdna_sched_job { |
| 126 | 138 | struct drm_sched_job base; |
| 127 | 139 | struct kref refcnt; |
| @@ -136,14 +148,20 @@ struct amdxdna_sched_job { · 8 unmodified lines | ||
| 136 | 148 | u64 seq; |
| 137 | 149 | struct amdxdna_drv_cmd *drv_cmd; |
| 138 | 150 | struct amdxdna_gem_obj *cmd_bo; |
| 151 | + union amdxdna_job_priv priv; | |
| 139 | 152 | size_t bo_cnt; |
| 140 | 153 | struct drm_gem_object *bos[] __counted_by(bo_cnt); |
| 141 | 154 | }; |
| 142 | 155 | |
| 156 | +#define aie2_job_health priv.aie2_health | |
| 157 | + | |
| 143 | 158 | static inline u32 |
| 144 | 159 | amdxdna_cmd_get_op(struct amdxdna_gem_obj *abo) |
| 145 | 160 | { |
| 146 | − struct amdxdna_cmd *cmd = abo->mem.kva; | |
| 161 | + struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); | |
| 162 | + | |
| 163 | + if (!cmd) | |
| 164 | + return ERT_INVALID_CMD; | |
| 147 | 165 | |
| 148 | 166 | return FIELD_GET(AMDXDNA_CMD_OPCODE, cmd->header); |
| 149 | 167 | } |
| @@ -151,7 +169,10 @@ amdxdna_cmd_get_op(struct amdxdna_gem_obj *abo) · 1 unmodified line | ||
| 151 | 169 | static inline void |
| 152 | 170 | amdxdna_cmd_set_state(struct amdxdna_gem_obj *abo, enum ert_cmd_state s) |
| 153 | 171 | { |
| 154 | − struct amdxdna_cmd *cmd = abo->mem.kva; | |
| 172 | + struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); | |
| 173 | + | |
| 174 | + if (!cmd) | |
| 175 | + return; | |
| 155 | 176 | |
| 156 | 177 | cmd->header &= ~AMDXDNA_CMD_STATE; |
| 157 | 178 | cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, s); |
| @@ -160,7 +181,10 @@ amdxdna_cmd_set_state(struct amdxdna_gem_obj *abo, enum ert_cmd_state s) · 2 unmodified lines | ||
| 160 | 181 | static inline enum ert_cmd_state |
| 161 | 182 | amdxdna_cmd_get_state(struct amdxdna_gem_obj *abo) |
| 162 | 183 | { |
| 163 | − struct amdxdna_cmd *cmd = abo->mem.kva; | |
| 184 | + struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); | |
| 185 | + | |
| 186 | + if (!cmd) | |
| 187 | + return ERT_CMD_STATE_INVALID; | |
| 164 | 188 | |
| 165 | 189 | return FIELD_GET(AMDXDNA_CMD_STATE, cmd->header); |
| 166 | 190 | } |
| @@ -169,7 +193,8 @@ void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size); · 2 unmodified lines | ||
| 169 | 193 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo); |
| 170 | 194 | int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo, |
| 171 | 195 | struct amdxdna_sched_job *job, u32 cmd_idx, |
| 172 | − enum ert_cmd_state error_state); | |
| 196 | + enum ert_cmd_state error_state, | |
| 197 | + void *err_data, size_t size); | |
| 173 | 198 | |
| 174 | 199 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job); |
| 175 | 200 | void amdxdna_hwctx_remove_all(struct amdxdna_client *client); |
| @@ -30,7 +30,6 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo) · 29 unmodified lines | ||
| 30 | 30 | struct amdxdna_dev *xdna = client->xdna; |
| 31 | 31 | struct amdxdna_mem *mem = &abo->mem; |
| 32 | 32 | struct amdxdna_gem_obj *heap; |
| 33 | − u64 offset; | |
| 34 | 33 | u32 align; |
| 35 | 34 | int ret; |
| 36 | 35 | |
| @@ -42,7 +41,7 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo) · 5 unmodified lines | ||
| 42 | 41 | goto unlock_out; |
| 43 | 42 | } |
| 44 | 43 | |
| 45 | − if (heap->mem.userptr == AMDXDNA_INVALID_ADDR) { | |
| 44 | + if (amdxdna_gem_uva(heap) == AMDXDNA_INVALID_ADDR) { | |
| 46 | 45 | XDNA_ERR(xdna, "Invalid dev heap userptr"); |
| 47 | 46 | ret = -EINVAL; |
| 48 | 47 | goto unlock_out; |
| @@ -64,10 +63,7 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo) · 15 unmodified lines | ||
| 64 | 63 | goto unlock_out; |
| 65 | 64 | } |
| 66 | 65 | |
| 67 | − mem->dev_addr = abo->mm_node.start; | |
| 68 | − offset = mem->dev_addr - heap->mem.dev_addr; | |
| 69 | − mem->userptr = heap->mem.userptr + offset; | |
| 70 | − mem->kva = heap->mem.kva + offset; | |
| 66 | + client->heap_usage += mem->size; | |
| 71 | 67 | |
| 72 | 68 | drm_gem_object_get(to_gobj(heap)); |
| 73 | 69 | |
| @@ -77,6 +73,45 @@ unlock_out: · 3 unmodified lines | ||
| 77 | 73 | return ret; |
| 78 | 74 | } |
| 79 | 75 | |
| 76 | +static void | |
| 77 | +amdxdna_gem_heap_free(struct amdxdna_gem_obj *abo) | |
| 78 | +{ | |
| 79 | + struct amdxdna_client *client = abo->client; | |
| 80 | + struct amdxdna_gem_obj *heap; | |
| 81 | + | |
| 82 | + mutex_lock(&client->mm_lock); | |
| 83 | + | |
| 84 | + drm_mm_remove_node(&abo->mm_node); | |
| 85 | + client->heap_usage -= abo->mem.size; | |
| 86 | + heap = client->dev_heap; | |
| 87 | + drm_gem_object_put(to_gobj(heap)); | |
| 88 | + | |
| 89 | + mutex_unlock(&client->mm_lock); | |
| 90 | +} | |
| 91 | + | |
| 92 | +static struct amdxdna_gem_obj * | |
| 93 | +amdxdna_gem_create_obj(struct drm_device *dev, size_t size) | |
| 94 | +{ | |
| 95 | + struct amdxdna_gem_obj *abo; | |
| 96 | + | |
| 97 | + abo = kzalloc_obj(*abo); | |
| 98 | + if (!abo) | |
| 99 | + return ERR_PTR(-ENOMEM); | |
| 100 | + | |
| 101 | + abo->pinned = false; | |
| 102 | + abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE; | |
| 103 | + mutex_init(&abo->lock); | |
| 104 | + | |
| 105 | + abo->mem.dma_addr = AMDXDNA_INVALID_ADDR; | |
| 106 | + abo->mem.uva = AMDXDNA_INVALID_ADDR; | |
| 107 | + abo->mem.size = size; | |
| 108 | + abo->open_ref = 0; | |
| 109 | + abo->internal = false; | |
| 110 | + INIT_LIST_HEAD(&abo->mem.umap_list); | |
| 111 | + | |
| 112 | + return abo; | |
| 113 | +} | |
| 114 | + | |
| 80 | 115 | static void |
| 81 | 116 | amdxdna_gem_destroy_obj(struct amdxdna_gem_obj *abo) |
| 82 | 117 | { |
| @@ -84,19 +119,75 @@ amdxdna_gem_destroy_obj(struct amdxdna_gem_obj *abo) · 1 unmodified line | ||
| 84 | 119 | kfree(abo); |
| 85 | 120 | } |
| 86 | 121 | |
| 87 | −static void | |
| 88 | −amdxdna_gem_heap_free(struct amdxdna_gem_obj *abo) | |
| 122 | +/* | |
| 123 | + * Obtains a kernel virtual address on the BO (usually of small size). | |
| 124 | + * The mapping is established on the first call and stays valid until | |
| 125 | + * amdxdna_gem_vunmap() is called. | |
| 126 | + */ | |
| 127 | +void *amdxdna_gem_vmap(struct amdxdna_gem_obj *abo) | |
| 89 | 128 | { |
| 90 | − struct amdxdna_gem_obj *heap; | |
| 129 | + struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); | |
| 130 | + int ret; | |
| 91 | 131 | |
| 92 | − mutex_lock(&abo->client->mm_lock); | |
| 132 | + if (abo->mem.kva) | |
| 133 | + return abo->mem.kva; | |
| 93 | 134 | |
| 94 | − drm_mm_remove_node(&abo->mm_node); | |
| 135 | + /* The first call to get the kva, taking slow path. */ | |
| 136 | + guard(mutex)(&abo->lock); | |
| 95 | 137 | |
| 96 | − heap = abo->client->dev_heap; | |
| 97 | − drm_gem_object_put(to_gobj(heap)); | |
| 138 | + if (!abo->mem.kva) { | |
| 139 | + ret = drm_gem_vmap(to_gobj(abo), &map); | |
| 140 | + if (ret) | |
| 141 | + XDNA_ERR(abo->client->xdna, "Vmap bo failed, ret %d", ret); | |
| 142 | + else | |
| 143 | + abo->mem.kva = map.vaddr; | |
| 144 | + } | |
| 145 | + return abo->mem.kva; | |
| 146 | +} | |
| 147 | + | |
| 148 | +/* | |
| 149 | + * Free mapping established through amdxdna_gem_vmap() | |
| 150 | + */ | |
| 151 | +static void amdxdna_gem_vunmap(struct amdxdna_gem_obj *abo) | |
| 152 | +{ | |
| 153 | + guard(mutex)(&abo->lock); | |
| 154 | + | |
| 155 | + if (abo->mem.kva) { | |
| 156 | + struct iosys_map map = IOSYS_MAP_INIT_VADDR(abo->mem.kva); | |
| 157 | + | |
| 158 | + drm_gem_vunmap(to_gobj(abo), &map); | |
| 159 | + abo->mem.kva = NULL; | |
| 160 | + } | |
| 161 | +} | |
| 162 | + | |
| 163 | +/* | |
| 164 | + * Obtain the user virtual address for accessing the BO. | |
| 165 | + * It can be used for device to access the BO when PASID is enabled. | |
| 166 | + */ | |
| 167 | +u64 amdxdna_gem_uva(struct amdxdna_gem_obj *abo) | |
| 168 | +{ | |
| 169 | + if (abo->type == AMDXDNA_BO_DEV) { | |
| 170 | + struct amdxdna_gem_obj *heap = abo->client->dev_heap; | |
| 171 | + u64 off = amdxdna_dev_bo_offset(abo); | |
| 172 | + | |
| 173 | + if (amdxdna_gem_uva(heap) != AMDXDNA_INVALID_ADDR) | |
| 174 | + return amdxdna_gem_uva(heap) + off; | |
| 175 | + return AMDXDNA_INVALID_ADDR; | |
| 176 | + } | |
| 98 | 177 | |
| 99 | − mutex_unlock(&abo->client->mm_lock); | |
| 178 | + return abo->mem.uva; | |
| 179 | +} | |
| 180 | + | |
| 181 | +/* | |
| 182 | + * Obtain the address for device to access the BO. | |
| 183 | + */ | |
| 184 | +u64 amdxdna_gem_dev_addr(struct amdxdna_gem_obj *abo) | |
| 185 | +{ | |
| 186 | + if (abo->type == AMDXDNA_BO_DEV_HEAP) | |
| 187 | + return abo->client->xdna->dev_info->dev_mem_base; | |
| 188 | + if (abo->type == AMDXDNA_BO_DEV) | |
| 189 | + return abo->mm_node.start; | |
| 190 | + return amdxdna_obj_dma_addr(abo); | |
| 100 | 191 | } |
| 101 | 192 | |
| 102 | 193 | static bool amdxdna_hmm_invalidate(struct mmu_interval_notifier *mni, |
| @@ -161,16 +252,19 @@ static void amdxdna_hmm_unregister(struct amdxdna_gem_obj *abo, · 58 unmodified lines | ||
| 161 | 252 | static void amdxdna_umap_release(struct kref *ref) |
| 162 | 253 | { |
| 163 | 254 | struct amdxdna_umap *mapp = container_of(ref, struct amdxdna_umap, refcnt); |
| 255 | + struct amdxdna_gem_obj *abo = mapp->abo; | |
| 164 | 256 | struct vm_area_struct *vma = mapp->vma; |
| 165 | 257 | struct amdxdna_dev *xdna; |
| 166 | 258 | |
| 167 | 259 | mmu_interval_notifier_remove(&mapp->notifier); |
| 168 | − if (is_import_bo(mapp->abo) && vma->vm_file && vma->vm_file->f_mapping) | |
| 260 | + if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping) | |
| 169 | 261 | mapping_clear_unevictable(vma->vm_file->f_mapping); |
| 170 | 262 | |
| 171 | 263 | xdna = to_xdna_dev(to_gobj(mapp->abo)->dev); |
| 172 | 264 | down_write(&xdna->notifier_lock); |
| 173 | 265 | list_del(&mapp->node); |
| 266 | + if (list_empty(&abo->mem.umap_list)) | |
| 267 | + abo->mem.uva = AMDXDNA_INVALID_ADDR; | |
| 174 | 268 | up_write(&xdna->notifier_lock); |
| 175 | 269 | |
| 176 | 270 | kvfree(mapp->range.hmm_pfns); |
| @@ -232,13 +326,13 @@ static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo, · 55 unmodified lines | ||
| 232 | 326 | mapp->abo = abo; |
| 233 | 327 | kref_init(&mapp->refcnt); |
| 234 | 328 | |
| 235 | − if (abo->mem.userptr == AMDXDNA_INVALID_ADDR) | |
| 236 | − abo->mem.userptr = addr; | |
| 237 | 329 | INIT_WORK(&mapp->hmm_unreg_work, amdxdna_hmm_unreg_work); |
| 238 | 330 | if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping) |
| 239 | 331 | mapping_set_unevictable(vma->vm_file->f_mapping); |
| 240 | 332 | |
| 241 | 333 | down_write(&xdna->notifier_lock); |
| 334 | + if (list_empty(&abo->mem.umap_list)) | |
| 335 | + abo->mem.uva = addr; | |
| 242 | 336 | list_add_tail(&mapp->node, &abo->mem.umap_list); |
| 243 | 337 | up_write(&xdna->notifier_lock); |
| 244 | 338 | |
| @@ -256,10 +350,11 @@ static void amdxdna_gem_dev_obj_free(struct drm_gem_object *gobj) · 11 unmodified lines | ||
| 256 | 350 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); |
| 257 | 351 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| 258 | 352 | |
| 259 | − XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, abo->mem.dev_addr); | |
| 353 | + XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, amdxdna_gem_dev_addr(abo)); | |
| 260 | 354 | if (abo->pinned) |
| 261 | 355 | amdxdna_gem_unpin(abo); |
| 262 | 356 | |
| 357 | + amdxdna_gem_vunmap(abo); | |
| 263 | 358 | amdxdna_gem_heap_free(abo); |
| 264 | 359 | drm_gem_object_release(gobj); |
| 265 | 360 | amdxdna_gem_destroy_obj(abo); |
| @@ -390,35 +485,6 @@ static const struct dma_buf_ops amdxdna_dmabuf_ops = { · 124 unmodified lines | ||
| 390 | 485 | .vunmap = drm_gem_dmabuf_vunmap, |
| 391 | 486 | }; |
| 392 | 487 | |
| 393 | −static int amdxdna_gem_obj_vmap(struct amdxdna_gem_obj *abo, void **vaddr) | |
| 394 | −{ | |
| 395 | − struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); | |
| 396 | − int ret; | |
| 397 | − | |
| 398 | − if (is_import_bo(abo)) | |
| 399 | − ret = dma_buf_vmap_unlocked(abo->dma_buf, &map); | |
| 400 | − else | |
| 401 | − ret = drm_gem_vmap(to_gobj(abo), &map); | |
| 402 | − | |
| 403 | − *vaddr = map.vaddr; | |
| 404 | − return ret; | |
| 405 | −} | |
| 406 | − | |
| 407 | −static void amdxdna_gem_obj_vunmap(struct amdxdna_gem_obj *abo) | |
| 408 | −{ | |
| 409 | − struct iosys_map map; | |
| 410 | − | |
| 411 | − if (!abo->mem.kva) | |
| 412 | − return; | |
| 413 | − | |
| 414 | − iosys_map_set_vaddr(&map, abo->mem.kva); | |
| 415 | − | |
| 416 | − if (is_import_bo(abo)) | |
| 417 | − dma_buf_vunmap_unlocked(abo->dma_buf, &map); | |
| 418 | − else | |
| 419 | − drm_gem_vunmap(to_gobj(abo), &map); | |
| 420 | −} | |
| 421 | − | |
| 422 | 488 | static struct dma_buf *amdxdna_gem_prime_export(struct drm_gem_object *gobj, int flags) |
| 423 | 489 | { |
| 424 | 490 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| @@ -447,13 +513,55 @@ static void amdxdna_imported_obj_free(struct amdxdna_gem_obj *abo) · 22 unmodified lines | ||
| 447 | 513 | kfree(abo); |
| 448 | 514 | } |
| 449 | 515 | |
| 516 | +static inline bool | |
| 517 | +amdxdna_gem_skip_bo_usage(struct amdxdna_gem_obj *abo) | |
| 518 | +{ | |
| 519 | + /* Do not count imported BOs since the buffer is not allocated by us. */ | |
| 520 | + if (is_import_bo(abo)) | |
| 521 | + return true; | |
| 522 | + | |
| 523 | + /* Already counted as part of HEAP BO */ | |
| 524 | + if (abo->type == AMDXDNA_BO_DEV) | |
| 525 | + return true; | |
| 526 | + | |
| 527 | + return false; | |
| 528 | +} | |
| 529 | + | |
| 530 | +static void | |
| 531 | +amdxdna_gem_add_bo_usage(struct amdxdna_gem_obj *abo) | |
| 532 | +{ | |
| 533 | + struct amdxdna_client *client = abo->client; | |
| 534 | + | |
| 535 | + if (amdxdna_gem_skip_bo_usage(abo)) | |
| 536 | + return; | |
| 537 | + | |
| 538 | + guard(mutex)(&client->mm_lock); | |
| 539 | + | |
| 540 | + client->total_bo_usage += abo->mem.size; | |
| 541 | + if (abo->internal) | |
| 542 | + client->total_int_bo_usage += abo->mem.size; | |
| 543 | +} | |
| 544 | + | |
| 545 | +static void | |
| 546 | +amdxdna_gem_del_bo_usage(struct amdxdna_gem_obj *abo) | |
| 547 | +{ | |
| 548 | + struct amdxdna_client *client = abo->client; | |
| 549 | + | |
| 550 | + if (amdxdna_gem_skip_bo_usage(abo)) | |
| 551 | + return; | |
| 552 | + | |
| 553 | + guard(mutex)(&client->mm_lock); | |
| 554 | + | |
| 555 | + client->total_bo_usage -= abo->mem.size; | |
| 556 | + if (abo->internal) | |
| 557 | + client->total_int_bo_usage -= abo->mem.size; | |
| 558 | +} | |
| 559 | + | |
| 450 | 560 | static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) |
| 451 | 561 | { |
| 452 | 562 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); |
| 453 | 563 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| 454 | 564 | |
| 455 | − XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, abo->mem.dev_addr); | |
| 456 | − | |
| 457 | 565 | amdxdna_hmm_unregister(abo, NULL); |
| 458 | 566 | flush_workqueue(xdna->notifier_wq); |
| 459 | 567 | |
| @@ -463,23 +571,76 @@ static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) · 3 unmodified lines | ||
| 463 | 571 | if (abo->type == AMDXDNA_BO_DEV_HEAP) |
| 464 | 572 | drm_mm_takedown(&abo->mm); |
| 465 | 573 | |
| 466 | − amdxdna_gem_obj_vunmap(abo); | |
| 574 | + if (amdxdna_iova_on(xdna)) | |
| 575 | + amdxdna_iommu_unmap_bo(xdna, abo); | |
| 576 | + | |
| 577 | + amdxdna_gem_vunmap(abo); | |
| 467 | 578 | mutex_destroy(&abo->lock); |
| 468 | 579 | |
| 469 | − if (is_import_bo(abo)) { | |
| 580 | + if (is_import_bo(abo)) | |
| 470 | 581 | amdxdna_imported_obj_free(abo); |
| 471 | − return; | |
| 582 | + else | |
| 583 | + drm_gem_shmem_free(&abo->base); | |
| 584 | +} | |
| 585 | + | |
| 586 | +static int amdxdna_gem_obj_open(struct drm_gem_object *gobj, struct drm_file *filp) | |
| 587 | +{ | |
| 588 | + struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); | |
| 589 | + struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | |
| 590 | + int ret; | |
| 591 | + | |
| 592 | + guard(mutex)(&abo->lock); | |
| 593 | + abo->open_ref++; | |
| 594 | + | |
| 595 | + if (abo->open_ref == 1) { | |
| 596 | + /* Attached to the client when first opened by it. */ | |
| 597 | + abo->client = filp->driver_priv; | |
| 598 | + amdxdna_gem_add_bo_usage(abo); | |
| 599 | + } | |
| 600 | + if (amdxdna_iova_on(xdna)) { | |
| 601 | + ret = amdxdna_iommu_map_bo(xdna, abo); | |
| 602 | + if (ret) | |
| 603 | + return ret; | |
| 604 | + } | |
| 605 | + | |
| 606 | + return 0; | |
| 607 | +} | |
| 608 | + | |
| 609 | +static void amdxdna_gem_obj_close(struct drm_gem_object *gobj, struct drm_file *filp) | |
| 610 | +{ | |
| 611 | + struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | |
| 612 | + | |
| 613 | + guard(mutex)(&abo->lock); | |
| 614 | + abo->open_ref--; | |
| 615 | + | |
| 616 | + if (abo->open_ref == 0) { | |
| 617 | + amdxdna_gem_del_bo_usage(abo); | |
| 618 | + /* Detach from the client when last closed by it. */ | |
| 619 | + abo->client = NULL; | |
| 472 | 620 | } |
| 621 | +} | |
| 622 | + | |
| 623 | +static int amdxdna_gem_dev_obj_vmap(struct drm_gem_object *obj, struct iosys_map *map) | |
| 624 | +{ | |
| 625 | + struct amdxdna_gem_obj *abo = to_xdna_obj(obj); | |
| 626 | + void *base = amdxdna_gem_vmap(abo->client->dev_heap); | |
| 627 | + u64 offset = amdxdna_dev_bo_offset(abo); | |
| 473 | 628 | |
| 474 | − drm_gem_shmem_free(&abo->base); | |
| 629 | + if (!base) | |
| 630 | + return -ENOMEM; | |
| 631 | + iosys_map_set_vaddr(map, base + offset); | |
| 632 | + return 0; | |
| 475 | 633 | } |
| 476 | 634 | |
| 477 | 635 | static const struct drm_gem_object_funcs amdxdna_gem_dev_obj_funcs = { |
| 478 | 636 | .free = amdxdna_gem_dev_obj_free, |
| 637 | + .vmap = amdxdna_gem_dev_obj_vmap, | |
| 479 | 638 | }; |
| 480 | 639 | |
| 481 | 640 | static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = { |
| 482 | 641 | .free = amdxdna_gem_obj_free, |
| 642 | + .open = amdxdna_gem_obj_open, | |
| 643 | + .close = amdxdna_gem_obj_close, | |
| 483 | 644 | .print_info = drm_gem_shmem_object_print_info, |
| 484 | 645 | .pin = drm_gem_shmem_object_pin, |
| 485 | 646 | .unpin = drm_gem_shmem_object_unpin, |
| @@ -491,30 +652,9 @@ static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = { · 5 unmodified lines | ||
| 491 | 652 | .export = amdxdna_gem_prime_export, |
| 492 | 653 | }; |
| 493 | 654 | |
| 494 | −static struct amdxdna_gem_obj * | |
| 495 | −amdxdna_gem_create_obj(struct drm_device *dev, size_t size) | |
| 496 | −{ | |
| 497 | − struct amdxdna_gem_obj *abo; | |
| 498 | − | |
| 499 | − abo = kzalloc_obj(*abo); | |
| 500 | − if (!abo) | |
| 501 | − return ERR_PTR(-ENOMEM); | |
| 502 | − | |
| 503 | − abo->pinned = false; | |
| 504 | − abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE; | |
| 505 | − mutex_init(&abo->lock); | |
| 506 | − | |
| 507 | − abo->mem.userptr = AMDXDNA_INVALID_ADDR; | |
| 508 | − abo->mem.dev_addr = AMDXDNA_INVALID_ADDR; | |
| 509 | − abo->mem.size = size; | |
| 510 | − INIT_LIST_HEAD(&abo->mem.umap_list); | |
| 511 | − | |
| 512 | − return abo; | |
| 513 | −} | |
| 514 | − | |
| 515 | 655 | /* For drm_driver->gem_create_object callback */ |
| 516 | 656 | struct drm_gem_object * |
| 517 | −amdxdna_gem_create_object_cb(struct drm_device *dev, size_t size) | |
| 657 | +amdxdna_gem_create_shmem_object_cb(struct drm_device *dev, size_t size) | |
| 518 | 658 | { |
| 519 | 659 | struct amdxdna_gem_obj *abo; |
| 520 | 660 | |
| @@ -528,8 +668,9 @@ amdxdna_gem_create_object_cb(struct drm_device *dev, size_t size) · 7 unmodified lines | ||
| 528 | 668 | } |
| 529 | 669 | |
| 530 | 670 | static struct amdxdna_gem_obj * |
| 531 | −amdxdna_gem_create_shmem_object(struct drm_device *dev, size_t size) | |
| 671 | +amdxdna_gem_create_shmem_object(struct drm_device *dev, struct amdxdna_drm_create_bo *args) | |
| 532 | 672 | { |
| 673 | + size_t size = args->size; | |
| 533 | 674 | struct drm_gem_shmem_object *shmem = drm_gem_shmem_create(dev, size); |
| 534 | 675 | |
| 535 | 676 | if (IS_ERR(shmem)) |
| @@ -543,7 +684,6 @@ static struct amdxdna_gem_obj * · 7 unmodified lines | ||
| 543 | 684 | amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create_bo *args) |
| 544 | 685 | { |
| 545 | 686 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 546 | − enum amdxdna_ubuf_flag flags = 0; | |
| 547 | 687 | struct amdxdna_drm_va_tbl va_tbl; |
| 548 | 688 | struct drm_gem_object *gobj; |
| 549 | 689 | struct dma_buf *dma_buf; |
| @@ -554,10 +694,7 @@ amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create · 4 unmodified lines | ||
| 554 | 694 | } |
| 555 | 695 | |
| 556 | 696 | if (va_tbl.num_entries) { |
| 557 | − if (args->type == AMDXDNA_BO_CMD) | |
| 558 | − flags |= AMDXDNA_UBUF_FLAG_MAP_DMA; | |
| 559 | − | |
| 560 | − dma_buf = amdxdna_get_ubuf(dev, flags, va_tbl.num_entries, | |
| 697 | + dma_buf = amdxdna_get_ubuf(dev, va_tbl.num_entries, | |
| 561 | 698 | u64_to_user_ptr(args->vaddr + sizeof(va_tbl))); |
| 562 | 699 | } else { |
| 563 | 700 | dma_buf = dma_buf_get(va_tbl.dmabuf_fd); |
| @@ -577,18 +714,6 @@ amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create · 13 unmodified lines | ||
| 577 | 714 | return to_xdna_obj(gobj); |
| 578 | 715 | } |
| 579 | 716 | |
| 580 | −static struct amdxdna_gem_obj * | |
| 581 | −amdxdna_gem_create_object(struct drm_device *dev, | |
| 582 | − struct amdxdna_drm_create_bo *args) | |
| 583 | −{ | |
| 584 | − size_t aligned_sz = PAGE_ALIGN(args->size); | |
| 585 | − | |
| 586 | − if (args->vaddr) | |
| 587 | − return amdxdna_gem_create_ubuf_object(dev, args); | |
| 588 | − | |
| 589 | − return amdxdna_gem_create_shmem_object(dev, aligned_sz); | |
| 590 | −} | |
| 591 | − | |
| 592 | 717 | struct drm_gem_object * |
| 593 | 718 | amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf) |
| 594 | 719 | { |
| @@ -621,6 +746,8 @@ amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf) · 26 unmodified lines | ||
| 621 | 746 | abo = to_xdna_obj(gobj); |
| 622 | 747 | abo->attach = attach; |
| 623 | 748 | abo->dma_buf = dma_buf; |
| 749 | + abo->type = AMDXDNA_BO_SHARE; | |
| 750 | + gobj->resv = dma_buf->resv; | |
| 624 | 751 | |
| 625 | 752 | return gobj; |
| 626 | 753 | |
| @@ -635,93 +762,101 @@ put_buf: · 8 unmodified lines | ||
| 635 | 762 | } |
| 636 | 763 | |
| 637 | 764 | static struct amdxdna_gem_obj * |
| 638 | −amdxdna_drm_alloc_shmem(struct drm_device *dev, | |
| 639 | − struct amdxdna_drm_create_bo *args, | |
| 640 | − struct drm_file *filp) | |
| 765 | +amdxdna_drm_create_share_bo(struct drm_device *dev, | |
| 766 | + struct amdxdna_drm_create_bo *args, struct drm_file *filp) | |
| 641 | 767 | { |
| 642 | − struct amdxdna_client *client = filp->driver_priv; | |
| 643 | 768 | struct amdxdna_gem_obj *abo; |
| 644 | 769 | |
| 645 | − abo = amdxdna_gem_create_object(dev, args); | |
| 770 | + if (args->vaddr) | |
| 771 | + abo = amdxdna_gem_create_ubuf_object(dev, args); | |
| 772 | + else | |
| 773 | + abo = amdxdna_gem_create_shmem_object(dev, args); | |
| 646 | 774 | if (IS_ERR(abo)) |
| 647 | 775 | return ERR_CAST(abo); |
| 648 | 776 | |
| 649 | − abo->client = client; | |
| 650 | − abo->type = AMDXDNA_BO_SHMEM; | |
| 777 | + if (args->type == AMDXDNA_BO_DEV_HEAP) { | |
| 778 | + abo->type = AMDXDNA_BO_DEV_HEAP; | |
| 779 | + abo->internal = true; | |
| 780 | + } else { | |
| 781 | + abo->type = AMDXDNA_BO_SHARE; | |
| 782 | + abo->internal = args->type == AMDXDNA_BO_CMD; | |
| 783 | + } | |
| 651 | 784 | |
| 652 | 785 | return abo; |
| 653 | 786 | } |
| 654 | 787 | |
| 655 | 788 | static struct amdxdna_gem_obj * |
| 656 | −amdxdna_drm_create_dev_heap(struct drm_device *dev, | |
| 657 | − struct amdxdna_drm_create_bo *args, | |
| 658 | − struct drm_file *filp) | |
| 789 | +amdxdna_drm_create_dev_heap_bo(struct drm_device *dev, | |
| 790 | + struct amdxdna_drm_create_bo *args, struct drm_file *filp) | |
| 659 | 791 | { |
| 660 | 792 | struct amdxdna_client *client = filp->driver_priv; |
| 661 | 793 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 662 | 794 | struct amdxdna_gem_obj *abo; |
| 663 | 795 | int ret; |
| 664 | 796 | |
| 665 | − if (args->size > xdna->dev_info->dev_mem_size) { | |
| 666 | − XDNA_DBG(xdna, "Invalid dev heap size 0x%llx, limit 0x%lx", | |
| 797 | + WARN_ON(!is_power_of_2(xdna->dev_info->dev_mem_size)); | |
| 798 | + XDNA_DBG(xdna, "Requested dev heap size 0x%llx", args->size); | |
| 799 | + if (!args->size || !IS_ALIGNED(args->size, xdna->dev_info->dev_mem_size)) { | |
| 800 | + XDNA_ERR(xdna, "The dev heap size 0x%llx is not multiple of 0x%lx", | |
| 667 | 801 | args->size, xdna->dev_info->dev_mem_size); |
| 668 | 802 | return ERR_PTR(-EINVAL); |
| 669 | 803 | } |
| 670 | 804 | |
| 805 | + /* HEAP BO is a special case of SHARE BO. */ | |
| 806 | + abo = amdxdna_drm_create_share_bo(dev, args, filp); | |
| 807 | + if (IS_ERR(abo)) | |
| 808 | + return ERR_CAST(abo); | |
| 809 | + | |
| 810 | + /* Set up heap for this client. */ | |
| 671 | 811 | mutex_lock(&client->mm_lock); |
| 812 | + | |
| 672 | 813 | if (client->dev_heap) { |
| 673 | 814 | XDNA_DBG(client->xdna, "dev heap is already created"); |
| 674 | 815 | ret = -EBUSY; |
| 675 | 816 | goto mm_unlock; |
| 676 | 817 | } |
| 677 | − | |
| 678 | − abo = amdxdna_gem_create_object(dev, args); | |
| 679 | − if (IS_ERR(abo)) { | |
| 680 | − ret = PTR_ERR(abo); | |
| 681 | − goto mm_unlock; | |
| 682 | − } | |
| 683 | − | |
| 684 | − abo->type = AMDXDNA_BO_DEV_HEAP; | |
| 685 | − abo->client = client; | |
| 686 | − abo->mem.dev_addr = client->xdna->dev_info->dev_mem_base; | |
| 687 | − drm_mm_init(&abo->mm, abo->mem.dev_addr, abo->mem.size); | |
| 688 | − | |
| 689 | − ret = amdxdna_gem_obj_vmap(abo, &abo->mem.kva); | |
| 690 | − if (ret) { | |
| 691 | − XDNA_ERR(xdna, "Vmap heap bo failed, ret %d", ret); | |
| 692 | − goto release_obj; | |
| 693 | − } | |
| 694 | − | |
| 695 | 818 | client->dev_heap = abo; |
| 696 | 819 | drm_gem_object_get(to_gobj(abo)); |
| 820 | + | |
| 821 | + drm_mm_init(&abo->mm, xdna->dev_info->dev_mem_base, abo->mem.size); | |
| 822 | + | |
| 697 | 823 | mutex_unlock(&client->mm_lock); |
| 698 | 824 | |
| 699 | 825 | return abo; |
| 700 | 826 | |
| 701 | −release_obj: | |
| 702 | − drm_gem_object_put(to_gobj(abo)); | |
| 703 | 827 | mm_unlock: |
| 704 | 828 | mutex_unlock(&client->mm_lock); |
| 829 | + drm_gem_object_put(to_gobj(abo)); | |
| 705 | 830 | return ERR_PTR(ret); |
| 706 | 831 | } |
| 707 | 832 | |
| 708 | 833 | struct amdxdna_gem_obj * |
| 709 | −amdxdna_drm_alloc_dev_bo(struct drm_device *dev, | |
| 710 | − struct amdxdna_drm_create_bo *args, | |
| 711 | − struct drm_file *filp) | |
| 834 | +amdxdna_drm_create_dev_bo(struct drm_device *dev, | |
| 835 | + struct amdxdna_drm_create_bo *args, struct drm_file *filp) | |
| 712 | 836 | { |
| 837 | + size_t aligned_sz = PAGE_ALIGN(args->size); | |
| 713 | 838 | struct amdxdna_client *client = filp->driver_priv; |
| 714 | 839 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 715 | − size_t aligned_sz = PAGE_ALIGN(args->size); | |
| 716 | 840 | struct amdxdna_gem_obj *abo; |
| 841 | + struct drm_gem_object *gobj; | |
| 717 | 842 | int ret; |
| 718 | 843 | |
| 719 | − abo = amdxdna_gem_create_obj(&xdna->ddev, aligned_sz); | |
| 844 | + if (!aligned_sz) { | |
| 845 | + XDNA_ERR(xdna, "Invalid BO size 0x%llx", args->size); | |
| 846 | + return ERR_PTR(-EINVAL); | |
| 847 | + } | |
| 848 | + | |
| 849 | + abo = amdxdna_gem_create_obj(dev, aligned_sz); | |
| 720 | 850 | if (IS_ERR(abo)) |
| 721 | 851 | return abo; |
| 722 | − | |
| 723 | − to_gobj(abo)->funcs = &amdxdna_gem_dev_obj_funcs; | |
| 852 | + gobj = to_gobj(abo); | |
| 853 | + gobj->funcs = &amdxdna_gem_dev_obj_funcs; | |
| 724 | 854 | abo->type = AMDXDNA_BO_DEV; |
| 855 | + abo->internal = true; | |
| 856 | + /* | |
| 857 | + * DEV BOs cannot be alive when client is gone, it's OK to | |
| 858 | + * always establish the connection. | |
| 859 | + */ | |
| 725 | 860 | abo->client = client; |
| 726 | 861 | |
| 727 | 862 | ret = amdxdna_gem_heap_alloc(abo); |
| @@ -730,31 +865,7 @@ amdxdna_drm_alloc_dev_bo(struct drm_device *dev, · 2 unmodified lines | ||
| 730 | 865 | amdxdna_gem_destroy_obj(abo); |
| 731 | 866 | return ERR_PTR(ret); |
| 732 | 867 | } |
| 733 | − | |
| 734 | − drm_gem_private_object_init(&xdna->ddev, to_gobj(abo), aligned_sz); | |
| 735 | − | |
| 736 | − return abo; | |
| 737 | −} | |
| 738 | − | |
| 739 | −static struct amdxdna_gem_obj * | |
| 740 | −amdxdna_drm_create_cmd_bo(struct drm_device *dev, | |
| 741 | − struct amdxdna_drm_create_bo *args, | |
| 742 | − struct drm_file *filp) | |
| 743 | −{ | |
| 744 | − struct amdxdna_dev *xdna = to_xdna_dev(dev); | |
| 745 | − struct amdxdna_gem_obj *abo; | |
| 746 | − | |
| 747 | − if (args->size < sizeof(struct amdxdna_cmd)) { | |
| 748 | − XDNA_DBG(xdna, "Command BO size 0x%llx too small", args->size); | |
| 749 | − return ERR_PTR(-EINVAL); | |
| 750 | − } | |
| 751 | − | |
| 752 | − abo = amdxdna_gem_create_object(dev, args); | |
| 753 | − if (IS_ERR(abo)) | |
| 754 | − return ERR_CAST(abo); | |
| 755 | − | |
| 756 | − abo->type = AMDXDNA_BO_CMD; | |
| 757 | − abo->client = filp->driver_priv; | |
| 868 | + drm_gem_private_object_init(dev, gobj, aligned_sz); | |
| 758 | 869 | |
| 759 | 870 | return abo; |
| 760 | 871 | } |
| @@ -772,17 +883,16 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f · 11 unmodified lines | ||
| 772 | 883 | XDNA_DBG(xdna, "BO arg type %d vaddr 0x%llx size 0x%llx flags 0x%llx", |
| 773 | 884 | args->type, args->vaddr, args->size, args->flags); |
| 774 | 885 | switch (args->type) { |
| 775 | − case AMDXDNA_BO_SHMEM: | |
| 776 | − abo = amdxdna_drm_alloc_shmem(dev, args, filp); | |
| 886 | + case AMDXDNA_BO_CMD: | |
| 887 | + fallthrough; | |
| 888 | + case AMDXDNA_BO_SHARE: | |
| 889 | + abo = amdxdna_drm_create_share_bo(dev, args, filp); | |
| 777 | 890 | break; |
| 778 | 891 | case AMDXDNA_BO_DEV_HEAP: |
| 779 | − abo = amdxdna_drm_create_dev_heap(dev, args, filp); | |
| 892 | + abo = amdxdna_drm_create_dev_heap_bo(dev, args, filp); | |
| 780 | 893 | break; |
| 781 | 894 | case AMDXDNA_BO_DEV: |
| 782 | − abo = amdxdna_drm_alloc_dev_bo(dev, args, filp); | |
| 783 | − break; | |
| 784 | − case AMDXDNA_BO_CMD: | |
| 785 | − abo = amdxdna_drm_create_cmd_bo(dev, args, filp); | |
| 895 | + abo = amdxdna_drm_create_dev_bo(dev, args, filp); | |
| 786 | 896 | break; |
| 787 | 897 | default: |
| 788 | 898 | return -EINVAL; |
| @@ -790,7 +900,7 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f · 1 unmodified line | ||
| 790 | 900 | if (IS_ERR(abo)) |
| 791 | 901 | return PTR_ERR(abo); |
| 792 | 902 | |
| 793 | − /* ready to publish object to userspace */ | |
| 903 | + /* Ready to publish object to userspace and count for BO usage. */ | |
| 794 | 904 | ret = drm_gem_handle_create(filp, to_gobj(abo), &args->handle); |
| 795 | 905 | if (ret) { |
| 796 | 906 | XDNA_ERR(xdna, "Create handle failed"); |
| @@ -798,8 +908,8 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f · 1 unmodified line | ||
| 798 | 908 | } |
| 799 | 909 | |
| 800 | 910 | XDNA_DBG(xdna, "BO hdl %d type %d userptr 0x%llx xdna_addr 0x%llx size 0x%lx", |
| 801 | − args->handle, args->type, abo->mem.userptr, | |
| 802 | − abo->mem.dev_addr, abo->mem.size); | |
| 911 | + args->handle, args->type, amdxdna_gem_uva(abo), | |
| 912 | + amdxdna_gem_dev_addr(abo), abo->mem.size); | |
| 803 | 913 | put_obj: |
| 804 | 914 | /* Dereference object reference. Handle holds it now. */ |
| 805 | 915 | drm_gem_object_put(to_gobj(abo)); |
| @@ -850,38 +960,19 @@ void amdxdna_gem_unpin(struct amdxdna_gem_obj *abo) · 44 unmodified lines | ||
| 850 | 960 | struct amdxdna_gem_obj *amdxdna_gem_get_obj(struct amdxdna_client *client, |
| 851 | 961 | u32 bo_hdl, u8 bo_type) |
| 852 | 962 | { |
| 853 | − struct amdxdna_dev *xdna = client->xdna; | |
| 854 | 963 | struct amdxdna_gem_obj *abo; |
| 855 | 964 | struct drm_gem_object *gobj; |
| 856 | − int ret; | |
| 857 | 965 | |
| 858 | 966 | gobj = drm_gem_object_lookup(client->filp, bo_hdl); |
| 859 | 967 | if (!gobj) { |
| 860 | − XDNA_DBG(xdna, "Can not find bo %d", bo_hdl); | |
| 968 | + XDNA_DBG(client->xdna, "Can not find bo %d", bo_hdl); | |
| 861 | 969 | return NULL; |
| 862 | 970 | } |
| 863 | 971 | |
| 864 | 972 | abo = to_xdna_obj(gobj); |
| 865 | − if (bo_type != AMDXDNA_BO_INVALID && abo->type != bo_type) | |
| 866 | − goto put_obj; | |
| 867 | − | |
| 868 | − if (bo_type != AMDXDNA_BO_CMD || abo->mem.kva) | |
| 973 | + if (bo_type == AMDXDNA_BO_INVALID || abo->type == bo_type) | |
| 869 | 974 | return abo; |
| 870 | 975 | |
| 871 | − if (abo->mem.size > SZ_32K) { | |
| 872 | − XDNA_ERR(xdna, "Cmd bo is too big %ld", abo->mem.size); | |
| 873 | − goto put_obj; | |
| 874 | − } | |
| 875 | − | |
| 876 | − ret = amdxdna_gem_obj_vmap(abo, &abo->mem.kva); | |
| 877 | − if (ret) { | |
| 878 | − XDNA_ERR(xdna, "Vmap cmd bo failed, ret %d", ret); | |
| 879 | − goto put_obj; | |
| 880 | − } | |
| 881 | − | |
| 882 | − return abo; | |
| 883 | − | |
| 884 | −put_obj: | |
| 885 | 976 | drm_gem_object_put(gobj); |
| 886 | 977 | return NULL; |
| 887 | 978 | } |
| @@ -904,8 +995,8 @@ int amdxdna_drm_get_bo_info_ioctl(struct drm_device *dev, void *data, struct drm · 16 unmodified lines | ||
| 904 | 995 | } |
| 905 | 996 | |
| 906 | 997 | abo = to_xdna_obj(gobj); |
| 907 | − args->vaddr = abo->mem.userptr; | |
| 908 | − args->xdna_addr = abo->mem.dev_addr; | |
| 998 | + args->vaddr = amdxdna_gem_uva(abo); | |
| 999 | + args->xdna_addr = amdxdna_gem_dev_addr(abo); | |
| 909 | 1000 | |
| 910 | 1001 | if (abo->type != AMDXDNA_BO_DEV) |
| 911 | 1002 | args->map_offset = drm_vma_node_offset_addr(&gobj->vma_node); |
| @@ -950,8 +1041,8 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, · 38 unmodified lines | ||
| 950 | 1041 | |
| 951 | 1042 | if (is_import_bo(abo)) |
| 952 | 1043 | drm_clflush_sg(abo->base.sgt); |
| 953 | − else if (abo->mem.kva) | |
| 954 | − drm_clflush_virt_range(abo->mem.kva + args->offset, args->size); | |
| 1044 | + else if (amdxdna_gem_vmap(abo)) | |
| 1045 | + drm_clflush_virt_range(amdxdna_gem_vmap(abo) + args->offset, args->size); | |
| 955 | 1046 | else if (abo->base.pages) |
| 956 | 1047 | drm_clflush_pages(abo->base.pages, gobj->size >> PAGE_SHIFT); |
| 957 | 1048 | else |
| @@ -969,3 +1060,43 @@ put_obj: · 11 unmodified lines | ||
| 969 | 1060 | drm_gem_object_put(gobj); |
| 970 | 1061 | return ret; |
| 971 | 1062 | } |
| 1063 | + | |
| 1064 | +int amdxdna_drm_get_bo_usage(struct drm_device *dev, struct amdxdna_drm_get_array *args) | |
| 1065 | +{ | |
| 1066 | + size_t min_sz = min(args->element_size, sizeof(struct amdxdna_drm_bo_usage)); | |
| 1067 | + char __user *buf = u64_to_user_ptr(args->buffer); | |
| 1068 | + struct amdxdna_dev *xdna = to_xdna_dev(dev); | |
| 1069 | + struct amdxdna_client *tmp_client; | |
| 1070 | + struct amdxdna_drm_bo_usage tmp; | |
| 1071 | + | |
| 1072 | + drm_WARN_ON(dev, !mutex_is_locked(&xdna->dev_lock)); | |
| 1073 | + | |
| 1074 | + if (args->num_element != 1) | |
| 1075 | + return -EINVAL; | |
| 1076 | + | |
| 1077 | + if (copy_from_user(&tmp, buf, min_sz)) | |
| 1078 | + return -EFAULT; | |
| 1079 | + | |
| 1080 | + if (!tmp.pid) | |
| 1081 | + return -EINVAL; | |
| 1082 | + | |
| 1083 | + tmp.total_usage = 0; | |
| 1084 | + tmp.internal_usage = 0; | |
| 1085 | + tmp.heap_usage = 0; | |
| 1086 | + | |
| 1087 | + list_for_each_entry(tmp_client, &xdna->client_list, node) { | |
| 1088 | + if (tmp.pid != tmp_client->pid) | |
| 1089 | + continue; | |
| 1090 | + | |
| 1091 | + mutex_lock(&tmp_client->mm_lock); | |
| 1092 | + tmp.total_usage += tmp_client->total_bo_usage; | |
| 1093 | + tmp.internal_usage += tmp_client->total_int_bo_usage; | |
| 1094 | + tmp.heap_usage += tmp_client->heap_usage; | |
| 1095 | + mutex_unlock(&tmp_client->mm_lock); | |
| 1096 | + } | |
| 1097 | + | |
| 1098 | + if (copy_to_user(buf, &tmp, min_sz)) | |
| 1099 | + return -EFAULT; | |
| 1100 | + | |
| 1101 | + return 0; | |
| 1102 | +} | |
| @@ -6,7 +6,9 @@ · 5 unmodified lines | ||
| 6 | 6 | #ifndef _AMDXDNA_GEM_H_ |
| 7 | 7 | #define _AMDXDNA_GEM_H_ |
| 8 | 8 | |
| 9 | +#include <drm/drm_gem_shmem_helper.h> | |
| 9 | 10 | #include <linux/hmm.h> |
| 11 | +#include <linux/iova.h> | |
| 10 | 12 | #include "amdxdna_pci_drv.h" |
| 11 | 13 | |
| 12 | 14 | struct amdxdna_umap { |
| @@ -22,14 +24,16 @@ struct amdxdna_umap { · 9 unmodified lines | ||
| 22 | 24 | }; |
| 23 | 25 | |
| 24 | 26 | struct amdxdna_mem { |
| 25 | − u64 userptr; | |
| 26 | 27 | void *kva; |
| 27 | − u64 dev_addr; | |
| 28 | + u64 dma_addr; | |
| 28 | 29 | size_t size; |
| 29 | − struct page **pages; | |
| 30 | − u32 nr_pages; | |
| 31 | 30 | struct list_head umap_list; |
| 32 | 31 | bool map_invalid; |
| 32 | + /* | |
| 33 | + * Cache the first mmap uva as PASID addr, which can be accessed by driver | |
| 34 | + * without taking notifier_lock. | |
| 35 | + */ | |
| 36 | + u64 uva; | |
| 33 | 37 | }; |
| 34 | 38 | |
| 35 | 39 | struct amdxdna_gem_obj { |
| @@ -37,15 +41,19 @@ struct amdxdna_gem_obj { · 1 unmodified line | ||
| 37 | 41 | struct amdxdna_client *client; |
| 38 | 42 | u8 type; |
| 39 | 43 | bool pinned; |
| 40 | − struct mutex lock; /* Protects: pinned */ | |
| 44 | + struct mutex lock; /* Protects: pinned, mem.kva, open_ref */ | |
| 41 | 45 | struct amdxdna_mem mem; |
| 46 | + int open_ref; | |
| 42 | 47 | |
| 43 | − /* Below members is uninitialized when needed */ | |
| 48 | + /* Below members are initialized when needed */ | |
| 44 | 49 | struct drm_mm mm; /* For AMDXDNA_BO_DEV_HEAP */ |
| 45 | 50 | struct drm_mm_node mm_node; /* For AMDXDNA_BO_DEV */ |
| 46 | 51 | u32 assigned_hwctx; |
| 47 | 52 | struct dma_buf *dma_buf; |
| 48 | 53 | struct dma_buf_attachment *attach; |
| 54 | + | |
| 55 | + /* True, if BO is managed by XRT, not application */ | |
| 56 | + bool internal; | |
| 49 | 57 | }; |
| 50 | 58 | |
| 51 | 59 | #define to_gobj(obj) (&(obj)->base.base) |
| @@ -63,21 +71,29 @@ static inline void amdxdna_gem_put_obj(struct amdxdna_gem_obj *abo) · 11 unmodified lines | ||
| 63 | 71 | drm_gem_object_put(to_gobj(abo)); |
| 64 | 72 | } |
| 65 | 73 | |
| 74 | +void *amdxdna_gem_vmap(struct amdxdna_gem_obj *abo); | |
| 75 | +u64 amdxdna_gem_uva(struct amdxdna_gem_obj *abo); | |
| 76 | +u64 amdxdna_gem_dev_addr(struct amdxdna_gem_obj *abo); | |
| 77 | + | |
| 66 | 78 | static inline u64 amdxdna_dev_bo_offset(struct amdxdna_gem_obj *abo) |
| 67 | 79 | { |
| 68 | − return abo->mem.dev_addr - abo->client->dev_heap->mem.dev_addr; | |
| 80 | + return amdxdna_gem_dev_addr(abo) - amdxdna_gem_dev_addr(abo->client->dev_heap); | |
| 81 | +} | |
| 82 | + | |
| 83 | +static inline u64 amdxdna_obj_dma_addr(struct amdxdna_gem_obj *abo) | |
| 84 | +{ | |
| 85 | + return amdxdna_pasid_on(abo->client) ? amdxdna_gem_uva(abo) : abo->mem.dma_addr; | |
| 69 | 86 | } |
| 70 | 87 | |
| 71 | 88 | void amdxdna_umap_put(struct amdxdna_umap *mapp); |
| 72 | 89 | |
| 73 | 90 | struct drm_gem_object * |
| 74 | −amdxdna_gem_create_object_cb(struct drm_device *dev, size_t size); | |
| 91 | +amdxdna_gem_create_shmem_object_cb(struct drm_device *dev, size_t size); | |
| 75 | 92 | struct drm_gem_object * |
| 76 | 93 | amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf); |
| 77 | 94 | struct amdxdna_gem_obj * |
| 78 | −amdxdna_drm_alloc_dev_bo(struct drm_device *dev, | |
| 79 | − struct amdxdna_drm_create_bo *args, | |
| 80 | − struct drm_file *filp); | |
| 95 | +amdxdna_drm_create_dev_bo(struct drm_device *dev, | |
| 96 | + struct amdxdna_drm_create_bo *args, struct drm_file *filp); | |
| 81 | 97 | |
| 82 | 98 | int amdxdna_gem_pin_nolock(struct amdxdna_gem_obj *abo); |
| 83 | 99 | int amdxdna_gem_pin(struct amdxdna_gem_obj *abo); |
| @@ -86,5 +102,6 @@ void amdxdna_gem_unpin(struct amdxdna_gem_obj *abo); · 2 unmodified lines | ||
| 86 | 102 | int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); |
| 87 | 103 | int amdxdna_drm_get_bo_info_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); |
| 88 | 104 | int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); |
| 105 | +int amdxdna_drm_get_bo_usage(struct drm_device *dev, struct amdxdna_drm_get_array *args); | |
| 89 | 106 | |
| 90 | 107 | #endif /* _AMDXDNA_GEM_H_ */ |
| @@ -0,0 +1,184 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | +/* | |
| 3 | + * Copyright (C) 2025, Advanced Micro Devices, Inc. | |
| 4 | + */ | |
| 5 | + | |
| 6 | +#include <drm/amdxdna_accel.h> | |
| 7 | +#include <linux/iommu.h> | |
| 8 | +#include <linux/iova.h> | |
| 9 | + | |
| 10 | +#include "amdxdna_gem.h" | |
| 11 | +#include "amdxdna_pci_drv.h" | |
| 12 | + | |
| 13 | +static bool force_iova; | |
| 14 | +module_param(force_iova, bool, 0600); | |
| 15 | +MODULE_PARM_DESC(force_iova, "Force use IOVA (Default false)"); | |
| 16 | + | |
| 17 | +static struct iova *amdxdna_iommu_alloc_iova(struct amdxdna_dev *xdna, | |
| 18 | + size_t size, | |
| 19 | + dma_addr_t *dma_addr, | |
| 20 | + bool size_aligned) | |
| 21 | +{ | |
| 22 | + unsigned long shift, end; | |
| 23 | + struct iova *iova; | |
| 24 | + | |
| 25 | + end = xdna->domain->geometry.aperture_end; | |
| 26 | + shift = iova_shift(&xdna->iovad); | |
| 27 | + size = iova_align(&xdna->iovad, size); | |
| 28 | + | |
| 29 | + iova = alloc_iova(&xdna->iovad, size >> shift, end >> shift, size_aligned); | |
| 30 | + if (!iova) | |
| 31 | + return ERR_PTR(-ENOMEM); | |
| 32 | + | |
| 33 | + *dma_addr = iova_dma_addr(&xdna->iovad, iova); | |
| 34 | + | |
| 35 | + return iova; | |
| 36 | +} | |
| 37 | + | |
| 38 | +int amdxdna_iommu_map_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo) | |
| 39 | +{ | |
| 40 | + struct sg_table *sgt; | |
| 41 | + dma_addr_t dma_addr; | |
| 42 | + struct iova *iova; | |
| 43 | + size_t size; | |
| 44 | + | |
| 45 | + if (abo->type != AMDXDNA_BO_DEV_HEAP && abo->type != AMDXDNA_BO_SHMEM) | |
| 46 | + return 0; | |
| 47 | + | |
| 48 | + sgt = drm_gem_shmem_get_pages_sgt(&abo->base); | |
| 49 | + if (IS_ERR(sgt)) { | |
| 50 | + XDNA_ERR(xdna, "Get sgt failed, ret %ld", PTR_ERR(sgt)); | |
| 51 | + return PTR_ERR(sgt); | |
| 52 | + } | |
| 53 | + | |
| 54 | + if (!sgt->orig_nents || !sg_page(sgt->sgl)) { | |
| 55 | + XDNA_ERR(xdna, "sgl is zero length or not page backed"); | |
| 56 | + return -EOPNOTSUPP; | |
| 57 | + } | |
| 58 | + | |
| 59 | + iova = amdxdna_iommu_alloc_iova(xdna, abo->mem.size, &dma_addr, | |
| 60 | + (abo->type == AMDXDNA_BO_DEV_HEAP)); | |
| 61 | + if (IS_ERR(iova)) { | |
| 62 | + XDNA_ERR(xdna, "Alloc iova failed, ret %ld", PTR_ERR(iova)); | |
| 63 | + return PTR_ERR(iova); | |
| 64 | + } | |
| 65 | + | |
| 66 | + size = iommu_map_sgtable(xdna->domain, dma_addr, sgt, | |
| 67 | + IOMMU_READ | IOMMU_WRITE); | |
| 68 | + if (size < abo->mem.size) { | |
| 69 | + __free_iova(&xdna->iovad, iova); | |
| 70 | + return -ENXIO; | |
| 71 | + } | |
| 72 | + | |
| 73 | + abo->mem.dma_addr = dma_addr; | |
| 74 | + | |
| 75 | + return 0; | |
| 76 | +} | |
| 77 | + | |
| 78 | +void amdxdna_iommu_unmap_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo) | |
| 79 | +{ | |
| 80 | + size_t size; | |
| 81 | + | |
| 82 | + if (abo->mem.dma_addr == AMDXDNA_INVALID_ADDR) | |
| 83 | + return; | |
| 84 | + | |
| 85 | + size = iova_align(&xdna->iovad, abo->mem.size); | |
| 86 | + iommu_unmap(xdna->domain, abo->mem.dma_addr, size); | |
| 87 | + free_iova(&xdna->iovad, iova_pfn(&xdna->iovad, abo->mem.dma_addr)); | |
| 88 | + abo->mem.dma_addr = AMDXDNA_INVALID_ADDR; | |
| 89 | +} | |
| 90 | + | |
| 91 | +void *amdxdna_iommu_alloc(struct amdxdna_dev *xdna, size_t size, dma_addr_t *dma_addr) | |
| 92 | +{ | |
| 93 | + struct iova *iova; | |
| 94 | + void *cpu_addr; | |
| 95 | + int ret; | |
| 96 | + | |
| 97 | + iova = amdxdna_iommu_alloc_iova(xdna, size, dma_addr, true); | |
| 98 | + if (IS_ERR(iova)) { | |
| 99 | + XDNA_ERR(xdna, "Alloc iova failed, ret %ld", PTR_ERR(iova)); | |
| 100 | + return iova; | |
| 101 | + } | |
| 102 | + | |
| 103 | + cpu_addr = (void *)__get_free_pages(GFP_KERNEL, get_order(size)); | |
| 104 | + if (!cpu_addr) { | |
| 105 | + ret = -ENOMEM; | |
| 106 | + goto free_iova; | |
| 107 | + } | |
| 108 | + | |
| 109 | + ret = iommu_map(xdna->domain, *dma_addr, virt_to_phys(cpu_addr), | |
| 110 | + iova_align(&xdna->iovad, size), | |
| 111 | + IOMMU_READ | IOMMU_WRITE, GFP_KERNEL); | |
| 112 | + if (ret) | |
| 113 | + goto free_iova; | |
| 114 | + | |
| 115 | + return cpu_addr; | |
| 116 | + | |
| 117 | +free_iova: | |
| 118 | + __free_iova(&xdna->iovad, iova); | |
| 119 | + return ERR_PTR(ret); | |
| 120 | +} | |
| 121 | + | |
| 122 | +void amdxdna_iommu_free(struct amdxdna_dev *xdna, size_t size, | |
| 123 | + void *cpu_addr, dma_addr_t dma_addr) | |
| 124 | +{ | |
| 125 | + iommu_unmap(xdna->domain, dma_addr, iova_align(&xdna->iovad, size)); | |
| 126 | + free_iova(&xdna->iovad, iova_pfn(&xdna->iovad, dma_addr)); | |
| 127 | + free_pages((unsigned long)cpu_addr, get_order(size)); | |
| 128 | +} | |
| 129 | + | |
| 130 | +int amdxdna_iommu_init(struct amdxdna_dev *xdna) | |
| 131 | +{ | |
| 132 | + unsigned long order; | |
| 133 | + int ret; | |
| 134 | + | |
| 135 | + xdna->group = iommu_group_get(xdna->ddev.dev); | |
| 136 | + if (!xdna->group || !force_iova) | |
| 137 | + return 0; | |
| 138 | + | |
| 139 | + XDNA_WARN(xdna, "Enabled force_iova mode."); | |
| 140 | + xdna->domain = iommu_paging_domain_alloc_flags(xdna->ddev.dev, | |
| 141 | + IOMMU_HWPT_ALLOC_PASID); | |
| 142 | + if (IS_ERR(xdna->domain)) { | |
| 143 | + XDNA_ERR(xdna, "Failed to alloc iommu domain"); | |
| 144 | + ret = PTR_ERR(xdna->domain); | |
| 145 | + goto put_group; | |
| 146 | + } | |
| 147 | + | |
| 148 | + ret = iova_cache_get(); | |
| 149 | + if (ret) | |
| 150 | + goto free_domain; | |
| 151 | + | |
| 152 | + order = __ffs(xdna->domain->pgsize_bitmap); | |
| 153 | + init_iova_domain(&xdna->iovad, 1UL << order, 0); | |
| 154 | + | |
| 155 | + ret = iommu_attach_group(xdna->domain, xdna->group); | |
| 156 | + if (ret) | |
| 157 | + goto put_iova; | |
| 158 | + | |
| 159 | + return 0; | |
| 160 | + | |
| 161 | +put_iova: | |
| 162 | + put_iova_domain(&xdna->iovad); | |
| 163 | + iova_cache_put(); | |
| 164 | +free_domain: | |
| 165 | + iommu_domain_free(xdna->domain); | |
| 166 | +put_group: | |
| 167 | + iommu_group_put(xdna->group); | |
| 168 | + xdna->domain = NULL; | |
| 169 | + | |
| 170 | + return ret; | |
| 171 | +} | |
| 172 | + | |
| 173 | +void amdxdna_iommu_fini(struct amdxdna_dev *xdna) | |
| 174 | +{ | |
| 175 | + if (xdna->domain) { | |
| 176 | + iommu_detach_group(xdna->domain, xdna->group); | |
| 177 | + put_iova_domain(&xdna->iovad); | |
| 178 | + iova_cache_put(); | |
| 179 | + iommu_domain_free(xdna->domain); | |
| 180 | + } | |
| 181 | + | |
| 182 | + if (xdna->group) | |
| 183 | + iommu_group_put(xdna->group); | |
| 184 | +} | |
| @@ -499,7 +499,7 @@ xdna_mailbox_start_channel(struct mailbox_channel *mb_chann, · 498 unmodified lines | ||
| 499 | 499 | int ret; |
| 500 | 500 | |
| 501 | 501 | if (!is_power_of_2(x2i->rb_size) || !is_power_of_2(i2x->rb_size)) { |
| 502 | − pr_err("Ring buf size must be power of 2"); | |
| 502 | + pr_err("Ring buf size must be power of 2\n"); | |
| 503 | 503 | return -EINVAL; |
| 504 | 504 | } |
| 505 | 505 | |
| @@ -36,9 +36,10 @@ MODULE_FIRMWARE("amdnpu/17f0_11/npu_7.sbin"); · 35 unmodified lines | ||
| 36 | 36 | * 0.5: Support getting telemetry data |
| 37 | 37 | * 0.6: Support preemption |
| 38 | 38 | * 0.7: Support getting power and utilization data |
| 39 | + * 0.8: Support BO usage query | |
| 39 | 40 | */ |
| 40 | 41 | #define AMDXDNA_DRIVER_MAJOR 0 |
| 41 | −#define AMDXDNA_DRIVER_MINOR 7 | |
| 42 | +#define AMDXDNA_DRIVER_MINOR 8 | |
| 42 | 43 | |
| 43 | 44 | /* |
| 44 | 45 | * Bind the driver base on (vendor_id, device_id) pair and later use the |
| @@ -73,18 +74,21 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp) · 28 unmodified lines | ||
| 73 | 74 | |
| 74 | 75 | client->pid = pid_nr(rcu_access_pointer(filp->pid)); |
| 75 | 76 | client->xdna = xdna; |
| 76 | − | |
| 77 | − client->sva = iommu_sva_bind_device(xdna->ddev.dev, current->mm); | |
| 78 | − if (IS_ERR(client->sva)) { | |
| 79 | − ret = PTR_ERR(client->sva); | |
| 80 | − XDNA_ERR(xdna, "SVA bind device failed, ret %d", ret); | |
| 81 | − goto failed; | |
| 82 | − } | |
| 83 | − client->pasid = iommu_sva_get_pasid(client->sva); | |
| 84 | − if (client->pasid == IOMMU_PASID_INVALID) { | |
| 85 | − XDNA_ERR(xdna, "SVA get pasid failed"); | |
| 86 | − ret = -ENODEV; | |
| 87 | − goto unbind_sva; | |
| 77 | + client->pasid = IOMMU_PASID_INVALID; | |
| 78 | + | |
| 79 | + if (!amdxdna_iova_on(xdna)) { | |
| 80 | + client->sva = iommu_sva_bind_device(xdna->ddev.dev, current->mm); | |
| 81 | + if (IS_ERR(client->sva)) { | |
| 82 | + ret = PTR_ERR(client->sva); | |
| 83 | + XDNA_ERR(xdna, "SVA bind device failed, ret %d", ret); | |
| 84 | + goto failed; | |
| 85 | + } | |
| 86 | + client->pasid = iommu_sva_get_pasid(client->sva); | |
| 87 | + if (client->pasid == IOMMU_PASID_INVALID) { | |
| 88 | + XDNA_ERR(xdna, "SVA get pasid failed"); | |
| 89 | + ret = -ENODEV; | |
| 90 | + goto unbind_sva; | |
| 91 | + } | |
| 88 | 92 | } |
| 89 | 93 | client->mm = current->mm; |
| 90 | 94 | mmgrab(client->mm); |
| @@ -103,7 +107,8 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp) · 12 unmodified lines | ||
| 103 | 107 | return 0; |
| 104 | 108 | |
| 105 | 109 | unbind_sva: |
| 106 | − iommu_sva_unbind_device(client->sva); | |
| 110 | + if (!IS_ERR_OR_NULL(client->sva)) | |
| 111 | + iommu_sva_unbind_device(client->sva); | |
| 107 | 112 | failed: |
| 108 | 113 | kfree(client); |
| 109 | 114 | |
| @@ -116,12 +121,14 @@ static void amdxdna_client_cleanup(struct amdxdna_client *client) · 6 unmodified lines | ||
| 116 | 121 | amdxdna_hwctx_remove_all(client); |
| 117 | 122 | xa_destroy(&client->hwctx_xa); |
| 118 | 123 | cleanup_srcu_struct(&client->hwctx_srcu); |
| 119 | − mutex_destroy(&client->mm_lock); | |
| 120 | 124 | |
| 121 | 125 | if (client->dev_heap) |
| 122 | 126 | drm_gem_object_put(to_gobj(client->dev_heap)); |
| 123 | 127 | |
| 124 | − iommu_sva_unbind_device(client->sva); | |
| 128 | + mutex_destroy(&client->mm_lock); | |
| 129 | + | |
| 130 | + if (!IS_ERR_OR_NULL(client->sva)) | |
| 131 | + iommu_sva_unbind_device(client->sva); | |
| 125 | 132 | mmdrop(client->mm); |
| 126 | 133 | |
| 127 | 134 | kfree(client); |
| @@ -240,7 +247,7 @@ const struct drm_driver amdxdna_drm_drv = { · 112 unmodified lines | ||
| 240 | 247 | .ioctls = amdxdna_drm_ioctls, |
| 241 | 248 | .num_ioctls = ARRAY_SIZE(amdxdna_drm_ioctls), |
| 242 | 249 | |
| 243 | − .gem_create_object = amdxdna_gem_create_object_cb, | |
| 250 | + .gem_create_object = amdxdna_gem_create_shmem_object_cb, | |
| 244 | 251 | .gem_prime_import = amdxdna_gem_prime_import, |
| 245 | 252 | }; |
| 246 | 253 | |
| @@ -282,9 +289,15 @@ static int amdxdna_probe(struct pci_dev *pdev, const struct pci_device_id *id) · 35 unmodified lines | ||
| 282 | 289 | fs_reclaim_release(GFP_KERNEL); |
| 283 | 290 | } |
| 284 | 291 | |
| 292 | + ret = amdxdna_iommu_init(xdna); | |
| 293 | + if (ret) | |
| 294 | + return ret; | |
| 295 | + | |
| 285 | 296 | xdna->notifier_wq = alloc_ordered_workqueue("notifier_wq", WQ_MEM_RECLAIM); |
| 286 | − if (!xdna->notifier_wq) | |
| 287 | − return -ENOMEM; | |
| 297 | + if (!xdna->notifier_wq) { | |
| 298 | + ret = -ENOMEM; | |
| 299 | + goto iommu_fini; | |
| 300 | + } | |
| 288 | 301 | |
| 289 | 302 | mutex_lock(&xdna->dev_lock); |
| 290 | 303 | ret = xdna->dev_info->ops->init(xdna); |
| @@ -316,6 +329,8 @@ failed_dev_fini: · 25 unmodified lines | ||
| 316 | 329 | mutex_unlock(&xdna->dev_lock); |
| 317 | 330 | destroy_notifier_wq: |
| 318 | 331 | destroy_workqueue(xdna->notifier_wq); |
| 332 | +iommu_fini: | |
| 333 | + amdxdna_iommu_fini(xdna); | |
| 319 | 334 | return ret; |
| 320 | 335 | } |
| 321 | 336 | |
| @@ -341,6 +356,8 @@ static void amdxdna_remove(struct pci_dev *pdev) · 19 unmodified lines | ||
| 341 | 356 | |
| 342 | 357 | xdna->dev_info->ops->fini(xdna); |
| 343 | 358 | mutex_unlock(&xdna->dev_lock); |
| 359 | + | |
| 360 | + amdxdna_iommu_fini(xdna); | |
| 344 | 361 | } |
| 345 | 362 | |
| 346 | 363 | static const struct dev_pm_ops amdxdna_pm_ops = { |
| @@ -6,7 +6,10 @@ · 5 unmodified lines | ||
| 6 | 6 | #ifndef _AMDXDNA_PCI_DRV_H_ |
| 7 | 7 | #define _AMDXDNA_PCI_DRV_H_ |
| 8 | 8 | |
| 9 | +#include <drm/amdxdna_accel.h> | |
| 9 | 10 | #include <drm/drm_print.h> |
| 11 | +#include <linux/iommu.h> | |
| 12 | +#include <linux/iova.h> | |
| 10 | 13 | #include <linux/workqueue.h> |
| 11 | 14 | #include <linux/xarray.h> |
| 12 | 15 | |
| @@ -101,6 +104,10 @@ struct amdxdna_dev { · 88 unmodified lines | ||
| 101 | 104 | struct amdxdna_fw_ver fw_ver; |
| 102 | 105 | struct rw_semaphore notifier_lock; /* for mmu notifier*/ |
| 103 | 106 | struct workqueue_struct *notifier_wq; |
| 107 | + | |
| 108 | + struct iommu_group *group; | |
| 109 | + struct iommu_domain *domain; | |
| 110 | + struct iova_domain iovad; | |
| 104 | 111 | }; |
| 105 | 112 | |
| 106 | 113 | /* |
| @@ -131,6 +138,10 @@ struct amdxdna_client { · 24 unmodified lines | ||
| 131 | 138 | struct iommu_sva *sva; |
| 132 | 139 | int pasid; |
| 133 | 140 | struct mm_struct *mm; |
| 141 | + | |
| 142 | + size_t heap_usage; | |
| 143 | + size_t total_bo_usage; | |
| 144 | + size_t total_int_bo_usage; | |
| 134 | 145 | }; |
| 135 | 146 | |
| 136 | 147 | #define amdxdna_for_each_hwctx(client, hwctx_id, entry) \ |
| @@ -145,4 +156,21 @@ extern const struct amdxdna_dev_info dev_npu6_info; · 8 unmodified lines | ||
| 145 | 156 | int amdxdna_sysfs_init(struct amdxdna_dev *xdna); |
| 146 | 157 | void amdxdna_sysfs_fini(struct amdxdna_dev *xdna); |
| 147 | 158 | |
| 159 | +int amdxdna_iommu_init(struct amdxdna_dev *xdna); | |
| 160 | +void amdxdna_iommu_fini(struct amdxdna_dev *xdna); | |
| 161 | +int amdxdna_iommu_map_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo); | |
| 162 | +void amdxdna_iommu_unmap_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo); | |
| 163 | +void *amdxdna_iommu_alloc(struct amdxdna_dev *xdna, size_t size, dma_addr_t *dma_addr); | |
| 164 | +void amdxdna_iommu_free(struct amdxdna_dev *xdna, size_t size, | |
| 165 | + void *cpu_addr, dma_addr_t dma_addr); | |
| 166 | + | |
| 167 | +static inline bool amdxdna_iova_on(struct amdxdna_dev *xdna) | |
| 168 | +{ | |
| 169 | + return !!xdna->domain; | |
| 170 | +} | |
| 171 | + | |
| 172 | +static inline bool amdxdna_pasid_on(struct amdxdna_client *client) | |
| 173 | +{ | |
| 174 | + return client->pasid != IOMMU_PASID_INVALID; | |
| 175 | +} | |
| 148 | 176 | #endif /* _AMDXDNA_PCI_DRV_H_ */ |
| @@ -17,7 +17,6 @@ · 16 unmodified lines | ||
| 17 | 17 | struct amdxdna_ubuf_priv { |
| 18 | 18 | struct page **pages; |
| 19 | 19 | u64 nr_pages; |
| 20 | − enum amdxdna_ubuf_flag flags; | |
| 21 | 20 | struct mm_struct *mm; |
| 22 | 21 | }; |
| 23 | 22 | |
| @@ -37,11 +36,9 @@ static struct sg_table *amdxdna_ubuf_map(struct dma_buf_attachment *attach, · 13 unmodified lines | ||
| 37 | 36 | if (ret) |
| 38 | 37 | goto err_free_sg; |
| 39 | 38 | |
| 40 | − if (ubuf->flags & AMDXDNA_UBUF_FLAG_MAP_DMA) { | |
| 41 | − ret = dma_map_sgtable(attach->dev, sg, direction, 0); | |
| 42 | − if (ret) | |
| 43 | − goto err_free_table; | |
| 44 | − } | |
| 39 | + ret = dma_map_sgtable(attach->dev, sg, direction, 0); | |
| 40 | + if (ret) | |
| 41 | + goto err_free_table; | |
| 45 | 42 | |
| 46 | 43 | return sg; |
| 47 | 44 | |
| @@ -56,11 +53,7 @@ static void amdxdna_ubuf_unmap(struct dma_buf_attachment *attach, · 8 unmodified lines | ||
| 56 | 53 | struct sg_table *sg, |
| 57 | 54 | enum dma_data_direction direction) |
| 58 | 55 | { |
| 59 | − struct amdxdna_ubuf_priv *ubuf = attach->dmabuf->priv; | |
| 60 | − | |
| 61 | − if (ubuf->flags & AMDXDNA_UBUF_FLAG_MAP_DMA) | |
| 62 | − dma_unmap_sgtable(attach->dev, sg, direction, 0); | |
| 63 | − | |
| 56 | + dma_unmap_sgtable(attach->dev, sg, direction, 0); | |
| 64 | 57 | sg_free_table(sg); |
| 65 | 58 | kfree(sg); |
| 66 | 59 | } |
| @@ -133,7 +126,6 @@ static const struct dma_buf_ops amdxdna_ubuf_dmabuf_ops = { · 66 unmodified lines | ||
| 133 | 126 | }; |
| 134 | 127 | |
| 135 | 128 | struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, |
| 136 | − enum amdxdna_ubuf_flag flags, | |
| 137 | 129 | u32 num_entries, void __user *va_entries) |
| 138 | 130 | { |
| 139 | 131 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| @@ -152,7 +144,6 @@ struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, · 12 unmodified lines | ||
| 152 | 144 | if (!ubuf) |
| 153 | 145 | return ERR_PTR(-ENOMEM); |
| 154 | 146 | |
| 155 | − ubuf->flags = flags; | |
| 156 | 147 | ubuf->mm = current->mm; |
| 157 | 148 | mmgrab(ubuf->mm); |
| 158 | 149 | |
| @@ -8,12 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <drm/drm_device.h> |
| 9 | 9 | #include <linux/dma-buf.h> |
| 10 | 10 | |
| 11 | −enum amdxdna_ubuf_flag { | |
| 12 | − AMDXDNA_UBUF_FLAG_MAP_DMA = 1, | |
| 13 | −}; | |
| 14 | − | |
| 15 | 11 | struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, |
| 16 | − enum amdxdna_ubuf_flag flags, | |
| 17 | 12 | u32 num_entries, void __user *va_entries); |
| 18 | 13 | |
| 19 | 14 | #endif /* _AMDXDNA_UBUF_H_ */ |
| @@ -93,7 +93,8 @@ const struct aie2_fw_feature_tbl npu4_fw_feature_table[] = { · 92 unmodified lines | ||
| 93 | 93 | { .features = BIT_U64(AIE2_NPU_COMMAND), .major = 6, .min_minor = 15 }, |
| 94 | 94 | { .features = BIT_U64(AIE2_PREEMPT), .major = 6, .min_minor = 12 }, |
| 95 | 95 | { .features = BIT_U64(AIE2_TEMPORAL_ONLY), .major = 6, .min_minor = 12 }, |
| 96 | − { .features = GENMASK_ULL(AIE2_TEMPORAL_ONLY, AIE2_NPU_COMMAND), .major = 7 }, | |
| 96 | + { .features = BIT_U64(AIE2_APP_HEALTH), .major = 6, .min_minor = 18 }, | |
| 97 | + { .features = AIE2_ALL_FEATURES, .major = 7 }, | |
| 97 | 98 | { 0 } |
| 98 | 99 | }; |
| 99 | 100 | |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | /* |
| 3 | − * Copyright (C) 2020-2024 Intel Corporation | |
| 3 | + * Copyright (C) 2020-2026 Intel Corporation | |
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #include <linux/debugfs.h> |
| @@ -127,6 +127,14 @@ static int firewall_irq_counter_show(struct seq_file *s, void *v) · 120 unmodified lines | ||
| 127 | 127 | return 0; |
| 128 | 128 | } |
| 129 | 129 | |
| 130 | +static int engine_reset_counter_show(struct seq_file *s, void *v) | |
| 131 | +{ | |
| 132 | + struct ivpu_device *vdev = seq_to_ivpu(s); | |
| 133 | + | |
| 134 | + seq_printf(s, "%d\n", atomic_read(&vdev->pm->engine_reset_counter)); | |
| 135 | + return 0; | |
| 136 | +} | |
| 137 | + | |
| 130 | 138 | static const struct drm_debugfs_info vdev_debugfs_list[] = { |
| 131 | 139 | {"bo_list", bo_list_show, 0}, |
| 132 | 140 | {"fw_name", fw_name_show, 0}, |
| @@ -137,6 +145,7 @@ static const struct drm_debugfs_info vdev_debugfs_list[] = { · 4 unmodified lines | ||
| 137 | 145 | {"reset_counter", reset_counter_show, 0}, |
| 138 | 146 | {"reset_pending", reset_pending_show, 0}, |
| 139 | 147 | {"firewall_irq_counter", firewall_irq_counter_show, 0}, |
| 148 | + {"engine_reset_counter", engine_reset_counter_show, 0}, | |
| 140 | 149 | }; |
| 141 | 150 | |
| 142 | 151 | static int dvfs_mode_get(void *data, u64 *dvfs_mode) |
| @@ -352,8 +361,9 @@ static const struct file_operations ivpu_force_recovery_fops = { · 209 unmodified lines | ||
| 352 | 361 | static int ivpu_reset_engine_fn(void *data, u64 val) |
| 353 | 362 | { |
| 354 | 363 | struct ivpu_device *vdev = (struct ivpu_device *)data; |
| 364 | + struct vpu_jsm_msg resp; | |
| 355 | 365 | |
| 356 | − return ivpu_jsm_reset_engine(vdev, (u32)val); | |
| 366 | + return ivpu_jsm_reset_engine(vdev, (u32)val, &resp); | |
| 357 | 367 | } |
| 358 | 368 | |
| 359 | 369 | DEFINE_DEBUGFS_ATTRIBUTE(ivpu_reset_engine_fops, NULL, ivpu_reset_engine_fn, "0x%02llx\n"); |
| @@ -665,6 +665,7 @@ static int ivpu_dev_init(struct ivpu_device *vdev) · 664 unmodified lines | ||
| 665 | 665 | vdev->context_xa_limit.max = IVPU_USER_CONTEXT_MAX_SSID; |
| 666 | 666 | atomic64_set(&vdev->unique_id_counter, 0); |
| 667 | 667 | atomic_set(&vdev->job_timeout_counter, 0); |
| 668 | + atomic_set(&vdev->faults_detected, 0); | |
| 668 | 669 | xa_init_flags(&vdev->context_xa, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ); |
| 669 | 670 | xa_init_flags(&vdev->submitted_jobs_xa, XA_FLAGS_ALLOC1); |
| 670 | 671 | xa_init_flags(&vdev->db_xa, XA_FLAGS_ALLOC1); |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | /* SPDX-License-Identifier: GPL-2.0-only */ |
| 2 | 2 | /* |
| 3 | − * Copyright (C) 2020-2025 Intel Corporation | |
| 3 | + * Copyright (C) 2020-2026 Intel Corporation | |
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #ifndef __IVPU_DRV_H__ |
| @@ -36,6 +36,7 @@ · 29 unmodified lines | ||
| 36 | 36 | #define IVPU_HW_IP_60XX 60 |
| 37 | 37 | |
| 38 | 38 | #define IVPU_HW_IP_REV_LNL_B0 4 |
| 39 | +#define IVPU_HW_IP_REV_NVL_A0 0 | |
| 39 | 40 | |
| 40 | 41 | #define IVPU_HW_BTRS_MTL 1 |
| 41 | 42 | #define IVPU_HW_BTRS_LNL 2 |
| @@ -168,6 +169,7 @@ struct ivpu_device { · 126 unmodified lines | ||
| 168 | 169 | struct xarray submitted_jobs_xa; |
| 169 | 170 | struct ivpu_ipc_consumer job_done_consumer; |
| 170 | 171 | atomic_t job_timeout_counter; |
| 172 | + atomic_t faults_detected; | |
| 171 | 173 | |
| 172 | 174 | atomic64_t unique_id_counter; |
| 173 | 175 | |
| @@ -70,8 +70,10 @@ static void wa_init(struct ivpu_device *vdev) · 69 unmodified lines | ||
| 70 | 70 | if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL) |
| 71 | 71 | vdev->wa.interrupt_clear_with_0 = ivpu_hw_btrs_irqs_clear_with_0_mtl(vdev); |
| 72 | 72 | |
| 73 | − if (ivpu_device_id(vdev) == PCI_DEVICE_ID_LNL && | |
| 74 | − ivpu_revision(vdev) < IVPU_HW_IP_REV_LNL_B0) | |
| 73 | + if ((ivpu_device_id(vdev) == PCI_DEVICE_ID_LNL && | |
| 74 | + ivpu_revision(vdev) < IVPU_HW_IP_REV_LNL_B0) || | |
| 75 | + (ivpu_device_id(vdev) == PCI_DEVICE_ID_NVL && | |
| 76 | + ivpu_revision(vdev) == IVPU_HW_IP_REV_NVL_A0)) | |
| 75 | 77 | vdev->wa.disable_clock_relinquish = true; |
| 76 | 78 | |
| 77 | 79 | if (ivpu_test_mode & IVPU_TEST_MODE_CLK_RELINQ_ENABLE) |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | /* |
| 3 | − * Copyright (C) 2020-2025 Intel Corporation | |
| 3 | + * Copyright (C) 2020-2026 Intel Corporation | |
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #include <drm/drm_file.h> |
| @@ -607,6 +607,7 @@ bool ivpu_job_handle_engine_error(struct ivpu_device *vdev, u32 job_id, u32 job_ · 600 unmodified lines | ||
| 607 | 607 | * status and ensure both are handled in the same way |
| 608 | 608 | */ |
| 609 | 609 | job->file_priv->has_mmu_faults = true; |
| 610 | + atomic_set(&vdev->faults_detected, 1); | |
| 610 | 611 | queue_work(system_percpu_wq, &vdev->context_abort_work); |
| 611 | 612 | return true; |
| 612 | 613 | } |
| @@ -1115,6 +1116,51 @@ void ivpu_job_done_consumer_fini(struct ivpu_device *vdev) · 502 unmodified lines | ||
| 1115 | 1116 | ivpu_ipc_consumer_del(vdev, &vdev->job_done_consumer); |
| 1116 | 1117 | } |
| 1117 | 1118 | |
| 1119 | +static int reset_engine_and_mark_faulty_contexts(struct ivpu_device *vdev) | |
| 1120 | +{ | |
| 1121 | + u32 num_impacted_contexts; | |
| 1122 | + struct vpu_jsm_msg resp; | |
| 1123 | + int ret; | |
| 1124 | + u32 i; | |
| 1125 | + | |
| 1126 | + ret = ivpu_jsm_reset_engine(vdev, 0, &resp); | |
| 1127 | + if (ret) | |
| 1128 | + return ret; | |
| 1129 | + | |
| 1130 | + /* | |
| 1131 | + * If faults are detected, ignore guilty contexts from engine reset as NPU may not be stuck | |
| 1132 | + * and could return currently running good context and faulty contexts are already marked | |
| 1133 | + */ | |
| 1134 | + if (atomic_cmpxchg(&vdev->faults_detected, 1, 0) == 1) | |
| 1135 | + return 0; | |
| 1136 | + | |
| 1137 | + num_impacted_contexts = resp.payload.engine_reset_done.num_impacted_contexts; | |
| 1138 | + | |
| 1139 | + ivpu_warn_ratelimited(vdev, "Engine reset performed, impacted contexts: %u\n", | |
| 1140 | + num_impacted_contexts); | |
| 1141 | + | |
| 1142 | + if (!in_range(num_impacted_contexts, 1, VPU_MAX_ENGINE_RESET_IMPACTED_CONTEXTS - 1)) { | |
| 1143 | + ivpu_pm_trigger_recovery(vdev, "Cannot determine guilty contexts"); | |
| 1144 | + return -EIO; | |
| 1145 | + } | |
| 1146 | + | |
| 1147 | + /* No faults detected, NPU likely got stuck. Mark returned contexts as guilty */ | |
| 1148 | + guard(mutex)(&vdev->context_list_lock); | |
| 1149 | + | |
| 1150 | + for (i = 0; i < num_impacted_contexts; i++) { | |
| 1151 | + u32 ssid = resp.payload.engine_reset_done.impacted_contexts[i].host_ssid; | |
| 1152 | + struct ivpu_file_priv *file_priv = xa_load(&vdev->context_xa, ssid); | |
| 1153 | + | |
| 1154 | + if (file_priv) { | |
| 1155 | + mutex_lock(&file_priv->lock); | |
| 1156 | + file_priv->has_mmu_faults = true; | |
| 1157 | + mutex_unlock(&file_priv->lock); | |
| 1158 | + } | |
| 1159 | + } | |
| 1160 | + | |
| 1161 | + return 0; | |
| 1162 | +} | |
| 1163 | + | |
| 1118 | 1164 | void ivpu_context_abort_work_fn(struct work_struct *work) |
| 1119 | 1165 | { |
| 1120 | 1166 | struct ivpu_device *vdev = container_of(work, struct ivpu_device, context_abort_work); |
| @@ -1127,7 +1173,7 @@ void ivpu_context_abort_work_fn(struct work_struct *work) · 6 unmodified lines | ||
| 1127 | 1173 | return; |
| 1128 | 1174 | |
| 1129 | 1175 | if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) |
| 1130 | − if (ivpu_jsm_reset_engine(vdev, 0)) | |
| 1176 | + if (reset_engine_and_mark_faulty_contexts(vdev)) | |
| 1131 | 1177 | goto runtime_put; |
| 1132 | 1178 | |
| 1133 | 1179 | mutex_lock(&vdev->context_list_lock); |
| @@ -151,10 +151,9 @@ int ivpu_jsm_get_heartbeat(struct ivpu_device *vdev, u32 engine, u64 *heartbeat) · 150 unmodified lines | ||
| 151 | 151 | return ret; |
| 152 | 152 | } |
| 153 | 153 | |
| 154 | −int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine) | |
| 154 | +int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine, struct vpu_jsm_msg *resp) | |
| 155 | 155 | { |
| 156 | 156 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_ENGINE_RESET }; |
| 157 | − struct vpu_jsm_msg resp; | |
| 158 | 157 | int ret; |
| 159 | 158 | |
| 160 | 159 | if (engine != VPU_ENGINE_COMPUTE) |
| @@ -162,14 +161,17 @@ int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine) · 1 unmodified line | ||
| 162 | 161 | |
| 163 | 162 | req.payload.engine_reset.engine_idx = engine; |
| 164 | 163 | |
| 165 | − ret = ivpu_ipc_send_receive(vdev, &req, VPU_JSM_MSG_ENGINE_RESET_DONE, &resp, | |
| 164 | + ret = ivpu_ipc_send_receive(vdev, &req, VPU_JSM_MSG_ENGINE_RESET_DONE, resp, | |
| 166 | 165 | VPU_IPC_CHAN_ASYNC_CMD, vdev->timeout.jsm); |
| 167 | 166 | if (ret) { |
| 168 | 167 | ivpu_err_ratelimited(vdev, "Failed to reset engine %d: %d\n", engine, ret); |
| 169 | 168 | ivpu_pm_trigger_recovery(vdev, "Engine reset failed"); |
| 169 | + return ret; | |
| 170 | 170 | } |
| 171 | 171 | |
| 172 | − return ret; | |
| 172 | + atomic_inc(&vdev->pm->engine_reset_counter); | |
| 173 | + | |
| 174 | + return 0; | |
| 173 | 175 | } |
| 174 | 176 | |
| 175 | 177 | int ivpu_jsm_preempt_engine(struct ivpu_device *vdev, u32 engine, u32 preempt_id) |
| @@ -554,6 +556,15 @@ int ivpu_jsm_dct_disable(struct ivpu_device *vdev) · 378 unmodified lines | ||
| 554 | 556 | } |
| 555 | 557 | |
| 556 | 558 | int ivpu_jsm_state_dump(struct ivpu_device *vdev) |
| 559 | +{ | |
| 560 | + struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_STATE_DUMP }; | |
| 561 | + struct vpu_jsm_msg resp; | |
| 562 | + | |
| 563 | + return ivpu_ipc_send_receive_internal(vdev, &req, VPU_JSM_MSG_STATE_DUMP_RSP, &resp, | |
| 564 | + VPU_IPC_CHAN_ASYNC_CMD, vdev->timeout.jsm); | |
| 565 | +} | |
| 566 | + | |
| 567 | +int ivpu_jsm_state_dump_no_reply(struct ivpu_device *vdev) | |
| 557 | 568 | { |
| 558 | 569 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_STATE_DUMP }; |
| 559 | 570 | |
| @@ -14,7 +14,7 @@ int ivpu_jsm_register_db(struct ivpu_device *vdev, u32 ctx_id, u32 db_id, · 13 unmodified lines | ||
| 14 | 14 | u64 jobq_base, u32 jobq_size); |
| 15 | 15 | int ivpu_jsm_unregister_db(struct ivpu_device *vdev, u32 db_id); |
| 16 | 16 | int ivpu_jsm_get_heartbeat(struct ivpu_device *vdev, u32 engine, u64 *heartbeat); |
| 17 | −int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine); | |
| 17 | +int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine, struct vpu_jsm_msg *response); | |
| 18 | 18 | int ivpu_jsm_preempt_engine(struct ivpu_device *vdev, u32 engine, u32 preempt_id); |
| 19 | 19 | int ivpu_jsm_dyndbg_control(struct ivpu_device *vdev, char *command, size_t size); |
| 20 | 20 | int ivpu_jsm_trace_get_capability(struct ivpu_device *vdev, u32 *trace_destination_mask, |
| @@ -44,5 +44,6 @@ int ivpu_jsm_metric_streamer_info(struct ivpu_device *vdev, u64 metric_group_mas · 23 unmodified lines | ||
| 44 | 44 | int ivpu_jsm_dct_enable(struct ivpu_device *vdev, u32 active_us, u32 inactive_us); |
| 45 | 45 | int ivpu_jsm_dct_disable(struct ivpu_device *vdev); |
| 46 | 46 | int ivpu_jsm_state_dump(struct ivpu_device *vdev); |
| 47 | +int ivpu_jsm_state_dump_no_reply(struct ivpu_device *vdev); | |
| 47 | 48 | |
| 48 | 49 | #endif |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | /* |
| 3 | − * Copyright (C) 2020-2024 Intel Corporation | |
| 3 | + * Copyright (C) 2020-2026 Intel Corporation | |
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #include <linux/circ_buf.h> |
| @@ -964,6 +964,7 @@ void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) · 957 unmodified lines | ||
| 964 | 964 | file_priv = xa_load(&vdev->context_xa, ssid); |
| 965 | 965 | if (file_priv) { |
| 966 | 966 | if (!READ_ONCE(file_priv->has_mmu_faults)) { |
| 967 | + atomic_set(&vdev->faults_detected, 1); | |
| 967 | 968 | ivpu_mmu_dump_event(vdev, event); |
| 968 | 969 | WRITE_ONCE(file_priv->has_mmu_faults, true); |
| 969 | 970 | } |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | /* |
| 3 | − * Copyright (C) 2020-2024 Intel Corporation | |
| 3 | + * Copyright (C) 2020-2026 Intel Corporation | |
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #include <linux/highmem.h> |
| @@ -166,7 +166,7 @@ static void ivpu_pm_recovery_work(struct work_struct *work) · 159 unmodified lines | ||
| 166 | 166 | ivpu_pm_reset_begin(vdev); |
| 167 | 167 | |
| 168 | 168 | if (!pm_runtime_status_suspended(vdev->drm.dev)) { |
| 169 | − ivpu_jsm_state_dump(vdev); | |
| 169 | + ivpu_jsm_state_dump_no_reply(vdev); | |
| 170 | 170 | ivpu_dev_coredump(vdev); |
| 171 | 171 | ivpu_suspend(vdev); |
| 172 | 172 | } |
| @@ -205,23 +205,25 @@ static void ivpu_job_timeout_work(struct work_struct *work) · 32 unmodified lines | ||
| 205 | 205 | |
| 206 | 206 | if (ivpu_jsm_get_heartbeat(vdev, 0, &heartbeat) || heartbeat <= vdev->fw->last_heartbeat) { |
| 207 | 207 | ivpu_err(vdev, "Job timeout detected, heartbeat not progressed\n"); |
| 208 | − goto recovery; | |
| 208 | + goto abort; | |
| 209 | 209 | } |
| 210 | 210 | |
| 211 | 211 | inference_max_retries = DIV_ROUND_UP(inference_timeout_ms, timeout_ms); |
| 212 | 212 | if (atomic_fetch_inc(&vdev->job_timeout_counter) >= inference_max_retries) { |
| 213 | 213 | ivpu_err(vdev, "Job timeout detected, heartbeat limit (%lld) exceeded\n", |
| 214 | 214 | inference_max_retries); |
| 215 | − goto recovery; | |
| 215 | + goto abort; | |
| 216 | 216 | } |
| 217 | 217 | |
| 218 | 218 | vdev->fw->last_heartbeat = heartbeat; |
| 219 | 219 | ivpu_start_job_timeout_detection(vdev); |
| 220 | 220 | return; |
| 221 | 221 | |
| 222 | −recovery: | |
| 222 | +abort: | |
| 223 | 223 | atomic_set(&vdev->job_timeout_counter, 0); |
| 224 | − ivpu_pm_trigger_recovery(vdev, "TDR"); | |
| 224 | + ivpu_jsm_state_dump(vdev); | |
| 225 | + ivpu_dev_coredump(vdev); | |
| 226 | + queue_work(system_percpu_wq, &vdev->context_abort_work); | |
| 225 | 227 | } |
| 226 | 228 | |
| 227 | 229 | void ivpu_start_job_timeout_detection(struct ivpu_device *vdev) |
| @@ -404,6 +406,7 @@ void ivpu_pm_init(struct ivpu_device *vdev) · 176 unmodified lines | ||
| 404 | 406 | init_rwsem(&pm->reset_lock); |
| 405 | 407 | atomic_set(&pm->reset_pending, 0); |
| 406 | 408 | atomic_set(&pm->reset_counter, 0); |
| 409 | + atomic_set(&pm->engine_reset_counter, 0); | |
| 407 | 410 | |
| 408 | 411 | INIT_WORK(&pm->recovery_work, ivpu_pm_recovery_work); |
| 409 | 412 | INIT_DELAYED_WORK(&pm->job_timeout_work, ivpu_job_timeout_work); |
| @@ -18,6 +18,7 @@ struct ivpu_pm_info { · 17 unmodified lines | ||
| 18 | 18 | struct rw_semaphore reset_lock; |
| 19 | 19 | atomic_t reset_counter; |
| 20 | 20 | atomic_t reset_pending; |
| 21 | + atomic_t engine_reset_counter; | |
| 21 | 22 | u8 dct_active_percent; |
| 22 | 23 | }; |
| 23 | 24 | |
| @@ -914,7 +914,7 @@ static int decode_deactivate(struct qaic_device *qdev, void *trans, u32 *msg_len · 913 unmodified lines | ||
| 914 | 914 | */ |
| 915 | 915 | return -ENODEV; |
| 916 | 916 | |
| 917 | − if (status) { | |
| 917 | + if (usr && status) { | |
| 918 | 918 | /* |
| 919 | 919 | * Releasing resources failed on the device side, which puts |
| 920 | 920 | * us in a bind since they may still be in use, so enable the |
| @@ -1109,6 +1109,9 @@ static void *msg_xfer(struct qaic_device *qdev, struct wrapper_list *wrappers, u · 188 unmodified lines | ||
| 1109 | 1109 | mutex_lock(&qdev->cntl_mutex); |
| 1110 | 1110 | if (!list_empty(&elem.list)) |
| 1111 | 1111 | list_del(&elem.list); |
| 1112 | + /* resp_worker() processed the response but the wait was interrupted */ | |
| 1113 | + else if (ret == -ERESTARTSYS) | |
| 1114 | + ret = 0; | |
| 1112 | 1115 | if (!ret && !elem.buf) |
| 1113 | 1116 | ret = -ETIMEDOUT; |
| 1114 | 1117 | else if (ret > 0 && !elem.buf) |
| @@ -1419,9 +1422,49 @@ static void resp_worker(struct work_struct *work) · 304 unmodified lines | ||
| 1419 | 1422 | } |
| 1420 | 1423 | mutex_unlock(&qdev->cntl_mutex); |
| 1421 | 1424 | |
| 1422 | − if (!found) | |
| 1425 | + if (!found) { | |
| 1426 | + /* | |
| 1427 | + * The user might have gone away at this point without waiting | |
| 1428 | + * for QAIC_TRANS_DEACTIVATE_FROM_DEV transaction coming from | |
| 1429 | + * the device. If this is not handled correctly, the host will | |
| 1430 | + * not know that the DBC[n] has been freed on the device. | |
| 1431 | + * Due to this failure in synchronization between the device and | |
| 1432 | + * the host, if another user requests to activate a network, and | |
| 1433 | + * the device assigns DBC[n] again, save_dbc_buf() will hang, | |
| 1434 | + * waiting for dbc[n]->in_use to be set to false, which will not | |
| 1435 | + * happen unless the qaic_dev_reset_clean_local_state() gets | |
| 1436 | + * called by resetting the device (or re-inserting the module). | |
| 1437 | + * | |
| 1438 | + * As a solution, we look for QAIC_TRANS_DEACTIVATE_FROM_DEV | |
| 1439 | + * transactions in the message before disposing of it, then | |
| 1440 | + * handle releasing the DBC resources. | |
| 1441 | + * | |
| 1442 | + * Since the user has gone away, if the device could not | |
| 1443 | + * deactivate the network (status != 0), there is no way to | |
| 1444 | + * enable and reassign the DBC to the user. We can put trust in | |
| 1445 | + * the device that it will release all the active DBCs in | |
| 1446 | + * response to the QAIC_TRANS_TERMINATE_TO_DEV transaction, | |
| 1447 | + * otherwise, the user can issue an soc_reset to the device. | |
| 1448 | + */ | |
| 1449 | + u32 msg_count = le32_to_cpu(msg->hdr.count); | |
| 1450 | + u32 msg_len = le32_to_cpu(msg->hdr.len); | |
| 1451 | + u32 len = 0; | |
| 1452 | + int j; | |
| 1453 | + | |
| 1454 | + for (j = 0; j < msg_count && len < msg_len; ++j) { | |
| 1455 | + struct wire_trans_hdr *trans_hdr; | |
| 1456 | + | |
| 1457 | + trans_hdr = (struct wire_trans_hdr *)(msg->data + len); | |
| 1458 | + if (le32_to_cpu(trans_hdr->type) == QAIC_TRANS_DEACTIVATE_FROM_DEV) { | |
| 1459 | + if (decode_deactivate(qdev, trans_hdr, &len, NULL)) | |
| 1460 | + len += le32_to_cpu(trans_hdr->len); | |
| 1461 | + } else { | |
| 1462 | + len += le32_to_cpu(trans_hdr->len); | |
| 1463 | + } | |
| 1464 | + } | |
| 1423 | 1465 | /* request must have timed out, drop packet */ |
| 1424 | 1466 | kfree(msg); |
| 1467 | + } | |
| 1425 | 1468 | |
| 1426 | 1469 | kfree(resp); |
| 1427 | 1470 | } |
| @@ -9,6 +9,7 @@ config ARCH_SUPPORTS_ACPI · 8 unmodified lines | ||
| 9 | 9 | menuconfig ACPI |
| 10 | 10 | bool "ACPI (Advanced Configuration and Power Interface) Support" |
| 11 | 11 | depends on ARCH_SUPPORTS_ACPI |
| 12 | + select AUXILIARY_BUS | |
| 12 | 13 | select PNP |
| 13 | 14 | select NLS |
| 14 | 15 | select CRC32 |
| @@ -135,7 +135,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev, · 134 unmodified lines | ||
| 135 | 135 | } |
| 136 | 136 | } |
| 137 | 137 | |
| 138 | − if (adev->device_type == ACPI_BUS_TYPE_DEVICE && !adev->pnp.type.backlight) { | |
| 138 | + if (adev->device_type == ACPI_BUS_TYPE_DEVICE) { | |
| 139 | 139 | LIST_HEAD(resource_list); |
| 140 | 140 | |
| 141 | 141 | count = acpi_dev_get_resources(adev, &resource_list, NULL, NULL); |
| @@ -113,6 +113,10 @@ static int acpi_processor_errata_piix4(struct pci_dev *dev) · 112 unmodified lines | ||
| 113 | 113 | PCI_ANY_ID, PCI_ANY_ID, NULL); |
| 114 | 114 | if (ide_dev) { |
| 115 | 115 | errata.piix4.bmisx = pci_resource_start(ide_dev, 4); |
| 116 | + if (errata.piix4.bmisx) | |
| 117 | + dev_dbg(&ide_dev->dev, | |
| 118 | + "Bus master activity detection (BM-IDE) erratum enabled\n"); | |
| 119 | + | |
| 116 | 120 | pci_dev_put(ide_dev); |
| 117 | 121 | } |
| 118 | 122 | |
| @@ -131,20 +135,17 @@ static int acpi_processor_errata_piix4(struct pci_dev *dev) · 12 unmodified lines | ||
| 131 | 135 | if (isa_dev) { |
| 132 | 136 | pci_read_config_byte(isa_dev, 0x76, &value1); |
| 133 | 137 | pci_read_config_byte(isa_dev, 0x77, &value2); |
| 134 | − if ((value1 & 0x80) || (value2 & 0x80)) | |
| 138 | + if ((value1 & 0x80) || (value2 & 0x80)) { | |
| 135 | 139 | errata.piix4.fdma = 1; |
| 140 | + dev_dbg(&isa_dev->dev, | |
| 141 | + "Type-F DMA livelock erratum (C3 disabled)\n"); | |
| 142 | + } | |
| 136 | 143 | pci_dev_put(isa_dev); |
| 137 | 144 | } |
| 138 | 145 | |
| 139 | 146 | break; |
| 140 | 147 | } |
| 141 | 148 | |
| 142 | − if (ide_dev) | |
| 143 | − dev_dbg(&ide_dev->dev, "Bus master activity detection (BM-IDE) erratum enabled\n"); | |
| 144 | − | |
| 145 | − if (isa_dev) | |
| 146 | − dev_dbg(&isa_dev->dev, "Type-F DMA livelock erratum (C3 disabled)\n"); | |
| 147 | − | |
| 148 | 149 | return 0; |
| 149 | 150 | } |
| 150 | 151 | |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | |
| 10 | 10 | #define pr_fmt(fmt) "ACPI: video: " fmt |
| 11 | 11 | |
| 12 | +#include <linux/auxiliary_bus.h> | |
| 12 | 13 | #include <linux/kernel.h> |
| 13 | 14 | #include <linux/module.h> |
| 14 | 15 | #include <linux/init.h> |
| @@ -21,7 +22,6 @@ · 6 unmodified lines | ||
| 21 | 22 | #include <linux/sort.h> |
| 22 | 23 | #include <linux/pci.h> |
| 23 | 24 | #include <linux/pci_ids.h> |
| 24 | −#include <linux/platform_device.h> | |
| 25 | 25 | #include <linux/slab.h> |
| 26 | 26 | #include <linux/dmi.h> |
| 27 | 27 | #include <linux/suspend.h> |
| @@ -77,8 +77,9 @@ static int register_count; · 49 unmodified lines | ||
| 77 | 77 | static DEFINE_MUTEX(register_count_mutex); |
| 78 | 78 | static DEFINE_MUTEX(video_list_lock); |
| 79 | 79 | static LIST_HEAD(video_bus_head); |
| 80 | −static int acpi_video_bus_probe(struct platform_device *pdev); | |
| 81 | −static void acpi_video_bus_remove(struct platform_device *pdev); | |
| 80 | +static int acpi_video_bus_probe(struct auxiliary_device *aux_dev, | |
| 81 | + const struct auxiliary_device_id *id); | |
| 82 | +static void acpi_video_bus_remove(struct auxiliary_device *aux); | |
| 82 | 83 | static void acpi_video_bus_notify(acpi_handle handle, u32 event, void *data); |
| 83 | 84 | |
| 84 | 85 | /* |
| @@ -93,19 +94,16 @@ enum acpi_video_level_idx { · 8 unmodified lines | ||
| 93 | 94 | ACPI_VIDEO_FIRST_LEVEL, /* actual supported levels begin here */ |
| 94 | 95 | }; |
| 95 | 96 | |
| 96 | −static const struct acpi_device_id video_device_ids[] = { | |
| 97 | − {ACPI_VIDEO_HID, 0}, | |
| 98 | − {"", 0}, | |
| 97 | +static const struct auxiliary_device_id video_bus_auxiliary_id_table[] = { | |
| 98 | + { .name = "acpi.video_bus" }, | |
| 99 | + {}, | |
| 99 | 100 | }; |
| 100 | −MODULE_DEVICE_TABLE(acpi, video_device_ids); | |
| 101 | +MODULE_DEVICE_TABLE(auxiliary, video_bus_auxiliary_id_table); | |
| 101 | 102 | |
| 102 | −static struct platform_driver acpi_video_bus = { | |
| 103 | +static struct auxiliary_driver acpi_video_bus = { | |
| 103 | 104 | .probe = acpi_video_bus_probe, |
| 104 | 105 | .remove = acpi_video_bus_remove, |
| 105 | − .driver = { | |
| 106 | − .name = "acpi-video", | |
| 107 | − .acpi_match_table = video_device_ids, | |
| 108 | − }, | |
| 106 | + .id_table = video_bus_auxiliary_id_table, | |
| 109 | 107 | }; |
| 110 | 108 | |
| 111 | 109 | struct acpi_video_bus_flags { |
| @@ -1885,7 +1883,7 @@ static void acpi_video_dev_add_notify_handler(struct acpi_video_device *device) · 1773 unmodified lines | ||
| 1885 | 1883 | } |
| 1886 | 1884 | |
| 1887 | 1885 | static int acpi_video_bus_add_notify_handler(struct acpi_video_bus *video, |
| 1888 | − struct platform_device *pdev) | |
| 1886 | + struct device *parent) | |
| 1889 | 1887 | { |
| 1890 | 1888 | struct input_dev *input; |
| 1891 | 1889 | struct acpi_video_device *dev; |
| @@ -1908,7 +1906,7 @@ static int acpi_video_bus_add_notify_handler(struct acpi_video_bus *video, · 16 unmodified lines | ||
| 1908 | 1906 | input->phys = video->phys; |
| 1909 | 1907 | input->id.bustype = BUS_HOST; |
| 1910 | 1908 | input->id.product = 0x06; |
| 1911 | − input->dev.parent = &pdev->dev; | |
| 1909 | + input->dev.parent = parent; | |
| 1912 | 1910 | input->evbit[0] = BIT(EV_KEY); |
| 1913 | 1911 | set_bit(KEY_SWITCHVIDEOMODE, input->keybit); |
| 1914 | 1912 | set_bit(KEY_VIDEO_NEXT, input->keybit); |
| @@ -1980,9 +1978,10 @@ static int acpi_video_bus_put_devices(struct acpi_video_bus *video) · 65 unmodified lines | ||
| 1980 | 1978 | |
| 1981 | 1979 | static int instance; |
| 1982 | 1980 | |
| 1983 | −static int acpi_video_bus_probe(struct platform_device *pdev) | |
| 1981 | +static int acpi_video_bus_probe(struct auxiliary_device *aux_dev, | |
| 1982 | + const struct auxiliary_device_id *id_unused) | |
| 1984 | 1983 | { |
| 1985 | − struct acpi_device *device = ACPI_COMPANION(&pdev->dev); | |
| 1984 | + struct acpi_device *device = ACPI_COMPANION(&aux_dev->dev); | |
| 1986 | 1985 | struct acpi_video_bus *video; |
| 1987 | 1986 | bool auto_detect; |
| 1988 | 1987 | int error; |
| @@ -2019,7 +2018,7 @@ static int acpi_video_bus_probe(struct platform_device *pdev) · 30 unmodified lines | ||
| 2019 | 2018 | instance++; |
| 2020 | 2019 | } |
| 2021 | 2020 | |
| 2022 | − platform_set_drvdata(pdev, video); | |
| 2021 | + auxiliary_set_drvdata(aux_dev, video); | |
| 2023 | 2022 | |
| 2024 | 2023 | video->device = device; |
| 2025 | 2024 | strscpy(acpi_device_name(device), ACPI_VIDEO_BUS_NAME); |
| @@ -2068,7 +2067,7 @@ static int acpi_video_bus_probe(struct platform_device *pdev) · 42 unmodified lines | ||
| 2068 | 2067 | !auto_detect) |
| 2069 | 2068 | acpi_video_bus_register_backlight(video); |
| 2070 | 2069 | |
| 2071 | − error = acpi_video_bus_add_notify_handler(video, pdev); | |
| 2070 | + error = acpi_video_bus_add_notify_handler(video, &aux_dev->dev); | |
| 2072 | 2071 | if (error) |
| 2073 | 2072 | goto err_del; |
| 2074 | 2073 | |
| @@ -2096,10 +2095,10 @@ err_free_video: · 21 unmodified lines | ||
| 2096 | 2095 | return error; |
| 2097 | 2096 | } |
| 2098 | 2097 | |
| 2099 | −static void acpi_video_bus_remove(struct platform_device *pdev) | |
| 2098 | +static void acpi_video_bus_remove(struct auxiliary_device *aux_dev) | |
| 2100 | 2099 | { |
| 2101 | − struct acpi_video_bus *video = platform_get_drvdata(pdev); | |
| 2102 | − struct acpi_device *device = ACPI_COMPANION(&pdev->dev); | |
| 2100 | + struct acpi_video_bus *video = auxiliary_get_drvdata(aux_dev); | |
| 2101 | + struct acpi_device *device = ACPI_COMPANION(&aux_dev->dev); | |
| 2103 | 2102 | |
| 2104 | 2103 | acpi_dev_remove_notify_handler(device, ACPI_DEVICE_NOTIFY, |
| 2105 | 2104 | acpi_video_bus_notify); |
| @@ -2163,7 +2162,7 @@ int acpi_video_register(void) · 57 unmodified lines | ||
| 2163 | 2162 | |
| 2164 | 2163 | dmi_check_system(video_dmi_table); |
| 2165 | 2164 | |
| 2166 | − ret = platform_driver_register(&acpi_video_bus); | |
| 2165 | + ret = auxiliary_driver_register(&acpi_video_bus); | |
| 2167 | 2166 | if (ret) |
| 2168 | 2167 | goto leave; |
| 2169 | 2168 | |
| @@ -2183,7 +2182,7 @@ void acpi_video_unregister(void) · 13 unmodified lines | ||
| 2183 | 2182 | { |
| 2184 | 2183 | mutex_lock(®ister_count_mutex); |
| 2185 | 2184 | if (register_count) { |
| 2186 | − platform_driver_unregister(&acpi_video_bus); | |
| 2185 | + auxiliary_driver_unregister(&acpi_video_bus); | |
| 2187 | 2186 | register_count = 0; |
| 2188 | 2187 | may_report_brightness_keys = false; |
| 2189 | 2188 | } |
| @@ -451,7 +451,7 @@ const union acpi_predefined_info acpi_gbl_predefined_methods[] = { · 450 unmodified lines | ||
| 451 | 451 | |
| 452 | 452 | {{"_DSM", |
| 453 | 453 | METHOD_4ARGS(ACPI_TYPE_BUFFER, ACPI_TYPE_INTEGER, ACPI_TYPE_INTEGER, |
| 454 | − ACPI_TYPE_ANY | ACPI_TYPE_PACKAGE) | | |
| 454 | + ACPI_TYPE_PACKAGE | ACPI_TYPE_ANY) | | |
| 455 | 455 | ARG_COUNT_IS_MINIMUM, |
| 456 | 456 | METHOD_RETURNS(ACPI_RTYPE_ALL)}}, /* Must return a value, but it can be of any type */ |
| 457 | 457 | |
| @@ -818,9 +818,6 @@ const struct acpi_device *acpi_companion_match(const struct device *dev) · 817 unmodified lines | ||
| 818 | 818 | if (list_empty(&adev->pnp.ids)) |
| 819 | 819 | return NULL; |
| 820 | 820 | |
| 821 | − if (adev->pnp.type.backlight) | |
| 822 | − return adev; | |
| 823 | − | |
| 824 | 821 | return acpi_primary_dev_companion(adev, dev); |
| 825 | 822 | } |
| 826 | 823 | |
| @@ -1656,6 +1656,8 @@ static int acpi_ec_setup(struct acpi_ec *ec, struct acpi_device *device, bool ca · 1655 unmodified lines | ||
| 1656 | 1656 | |
| 1657 | 1657 | ret = ec_install_handlers(ec, device, call_reg); |
| 1658 | 1658 | if (ret) { |
| 1659 | + ec_remove_handlers(ec); | |
| 1660 | + | |
| 1659 | 1661 | if (ec == first_ec) |
| 1660 | 1662 | first_ec = NULL; |
| 1661 | 1663 | |
| @@ -1681,7 +1681,7 @@ acpi_status __init acpi_os_initialize(void) · 1680 unmodified lines | ||
| 1681 | 1681 | * Use acpi_os_map_generic_address to pre-map the reset |
| 1682 | 1682 | * register if it's in system memory. |
| 1683 | 1683 | */ |
| 1684 | − void *rv; | |
| 1684 | + void __iomem *rv; | |
| 1685 | 1685 | |
| 1686 | 1686 | rv = acpi_os_map_generic_address(&acpi_gbl_FADT.reset_register); |
| 1687 | 1687 | pr_debug("%s: Reset register mapping %s\n", __func__, |
| @@ -263,6 +263,13 @@ static int rimt_iommu_xlate(struct device *dev, struct acpi_rimt_node *node, u32 · 262 unmodified lines | ||
| 263 | 263 | if (!rimt_fwnode) |
| 264 | 264 | return -EPROBE_DEFER; |
| 265 | 265 | |
| 266 | + /* | |
| 267 | + * EPROBE_DEFER ensures IOMMU is probed before the devices that | |
| 268 | + * depend on them. During shutdown, however, the IOMMU may be removed | |
| 269 | + * first, leading to issues. To avoid this, a device link is added | |
| 270 | + * which enforces the correct removal order. | |
| 271 | + */ | |
| 272 | + device_link_add(dev, rimt_fwnode->dev, DL_FLAG_AUTOREMOVE_CONSUMER); | |
| 266 | 273 | return acpi_iommu_fwspec_init(dev, deviceid, rimt_fwnode); |
| 267 | 274 | } |
| 268 | 275 | |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #define pr_fmt(fmt) "ACPI: " fmt |
| 7 | 7 | |
| 8 | 8 | #include <linux/async.h> |
| 9 | +#include <linux/auxiliary_bus.h> | |
| 9 | 10 | #include <linux/module.h> |
| 10 | 11 | #include <linux/init.h> |
| 11 | 12 | #include <linux/slab.h> |
| @@ -2192,6 +2193,44 @@ static acpi_status acpi_bus_check_add_2(acpi_handle handle, u32 lvl_not_used, · 2180 unmodified lines | ||
| 2192 | 2193 | return acpi_bus_check_add(handle, false, (struct acpi_device **)ret_p); |
| 2193 | 2194 | } |
| 2194 | 2195 | |
| 2196 | +static void acpi_video_bus_device_release(struct device *dev) | |
| 2197 | +{ | |
| 2198 | + struct auxiliary_device *aux_dev = to_auxiliary_dev(dev); | |
| 2199 | + | |
| 2200 | + kfree(aux_dev); | |
| 2201 | +} | |
| 2202 | + | |
| 2203 | +static void acpi_create_video_bus_device(struct acpi_device *adev, | |
| 2204 | + struct acpi_device *parent) | |
| 2205 | +{ | |
| 2206 | + struct auxiliary_device *aux_dev; | |
| 2207 | + static unsigned int aux_dev_id; | |
| 2208 | + | |
| 2209 | + aux_dev = kzalloc_obj(*aux_dev); | |
| 2210 | + if (!aux_dev) | |
| 2211 | + return; | |
| 2212 | + | |
| 2213 | + aux_dev->id = aux_dev_id++; | |
| 2214 | + aux_dev->name = "video_bus"; | |
| 2215 | + aux_dev->dev.parent = acpi_get_first_physical_node(parent); | |
| 2216 | + if (!aux_dev->dev.parent) | |
| 2217 | + goto err; | |
| 2218 | + | |
| 2219 | + aux_dev->dev.release = acpi_video_bus_device_release; | |
| 2220 | + | |
| 2221 | + if (auxiliary_device_init(aux_dev)) | |
| 2222 | + goto err; | |
| 2223 | + | |
| 2224 | + ACPI_COMPANION_SET(&aux_dev->dev, adev); | |
| 2225 | + if (__auxiliary_device_add(aux_dev, "acpi")) | |
| 2226 | + auxiliary_device_uninit(aux_dev); | |
| 2227 | + | |
| 2228 | + return; | |
| 2229 | + | |
| 2230 | +err: | |
| 2231 | + kfree(aux_dev); | |
| 2232 | +} | |
| 2233 | + | |
| 2195 | 2234 | struct acpi_scan_system_dev { |
| 2196 | 2235 | struct list_head node; |
| 2197 | 2236 | struct acpi_device *adev; |
| @@ -2229,6 +2268,12 @@ static void acpi_default_enumeration(struct acpi_device *device) · 31 unmodified lines | ||
| 2229 | 2268 | sd->adev = device; |
| 2230 | 2269 | list_add_tail(&sd->node, &acpi_scan_system_dev_list); |
| 2231 | 2270 | } |
| 2271 | + } else if (device->pnp.type.backlight) { | |
| 2272 | + struct acpi_device *parent; | |
| 2273 | + | |
| 2274 | + parent = acpi_dev_parent(device); | |
| 2275 | + if (parent) | |
| 2276 | + acpi_create_video_bus_device(device, parent); | |
| 2232 | 2277 | } else { |
| 2233 | 2278 | /* For a regular device object, create a platform device. */ |
| 2234 | 2279 | acpi_create_platform_device(device, NULL); |
| @@ -13,6 +13,8 @@ · 12 unmodified lines | ||
| 13 | 13 | // |
| 14 | 14 | // The shrinker will use trylock methods because it locks them in a different order. |
| 15 | 15 | |
| 16 | +use crate::AssertSync; | |
| 17 | + | |
| 16 | 18 | use core::{ |
| 17 | 19 | marker::PhantomPinned, |
| 18 | 20 | mem::{size_of, size_of_val, MaybeUninit}, |
| @@ -142,6 +144,30 @@ pub(crate) struct ShrinkablePageRange { · 123 unmodified lines | ||
| 142 | 144 | _pin: PhantomPinned, |
| 143 | 145 | } |
| 144 | 146 | |
| 147 | +// We do not define any ops. For now, used only to check identity of vmas. | |
| 148 | +static BINDER_VM_OPS: AssertSync<bindings::vm_operations_struct> = AssertSync(pin_init::zeroed()); | |
| 149 | + | |
| 150 | +// To ensure that we do not accidentally install pages into or zap pages from the wrong vma, we | |
| 151 | +// check its vm_ops and private data before using it. | |
| 152 | +fn check_vma(vma: &virt::VmaRef, owner: *const ShrinkablePageRange) -> Option<&virt::VmaMixedMap> { | |
| 153 | + // SAFETY: Just reading the vm_ops pointer of any active vma is safe. | |
| 154 | + let vm_ops = unsafe { (*vma.as_ptr()).vm_ops }; | |
| 155 | + if !ptr::eq(vm_ops, &BINDER_VM_OPS.0) { | |
| 156 | + return None; | |
| 157 | + } | |
| 158 | + | |
| 159 | + // SAFETY: Reading the vm_private_data pointer of a binder-owned vma is safe. | |
| 160 | + let vm_private_data = unsafe { (*vma.as_ptr()).vm_private_data }; | |
| 161 | + // The ShrinkablePageRange is only dropped when the Process is dropped, which only happens once | |
| 162 | + // the file's ->release handler is invoked, which means the ShrinkablePageRange outlives any | |
| 163 | + // VMA associated with it, so there can't be any false positives due to pointer reuse here. | |
| 164 | + if !ptr::eq(vm_private_data, owner.cast()) { | |
| 165 | + return None; | |
| 166 | + } | |
| 167 | + | |
| 168 | + vma.as_mixedmap_vma() | |
| 169 | +} | |
| 170 | + | |
| 145 | 171 | struct Inner { |
| 146 | 172 | /// Array of pages. |
| 147 | 173 | /// |
| @@ -308,6 +334,18 @@ impl ShrinkablePageRange { · 160 unmodified lines | ||
| 308 | 334 | inner.size = num_pages; |
| 309 | 335 | inner.vma_addr = vma.start(); |
| 310 | 336 | |
| 337 | + // This pointer is only used for comparison - it's not dereferenced. | |
| 338 | + // | |
| 339 | + // SAFETY: We own the vma, and we don't use any methods on VmaNew that rely on | |
| 340 | + // `vm_private_data`. | |
| 341 | + unsafe { | |
| 342 | + (*vma.as_ptr()).vm_private_data = ptr::from_ref(self).cast_mut().cast::<c_void>() | |
| 343 | + }; | |
| 344 | + | |
| 345 | + // SAFETY: We own the vma, and we don't use any methods on VmaNew that rely on | |
| 346 | + // `vm_ops`. | |
| 347 | + unsafe { (*vma.as_ptr()).vm_ops = &BINDER_VM_OPS.0 }; | |
| 348 | + | |
| 311 | 349 | Ok(num_pages) |
| 312 | 350 | } |
| 313 | 351 | |
| @@ -399,22 +437,25 @@ impl ShrinkablePageRange { · 85 unmodified lines | ||
| 399 | 437 | // |
| 400 | 438 | // Using `mmput_async` avoids this, because then the `mm` cleanup is instead queued to a |
| 401 | 439 | // workqueue. |
| 402 | − MmWithUser::into_mmput_async(self.mm.mmget_not_zero().ok_or(ESRCH)?) | |
| 403 | − .mmap_read_lock() | |
| 404 | − .vma_lookup(vma_addr) | |
| 405 | − .ok_or(ESRCH)? | |
| 406 | − .as_mixedmap_vma() | |
| 407 | − .ok_or(ESRCH)? | |
| 408 | − .vm_insert_page(user_page_addr, &new_page) | |
| 409 | − .inspect_err(|err| { | |
| 410 | − pr_warn!( | |
| 411 | − "Failed to vm_insert_page({}): vma_addr:{} i:{} err:{:?}", | |
| 412 | − user_page_addr, | |
| 413 | − vma_addr, | |
| 414 | − i, | |
| 415 | − err | |
| 416 | − ) | |
| 417 | − })?; | |
| 440 | + let mm = MmWithUser::into_mmput_async(self.mm.mmget_not_zero().ok_or(ESRCH)?); | |
| 441 | + { | |
| 442 | + let vma_read; | |
| 443 | + let mmap_read; | |
| 444 | + let vma = if let Some(ret) = mm.lock_vma_under_rcu(vma_addr) { | |
| 445 | + vma_read = ret; | |
| 446 | + check_vma(&vma_read, self) | |
| 447 | + } else { | |
| 448 | + mmap_read = mm.mmap_read_lock(); | |
| 449 | + mmap_read | |
| 450 | + .vma_lookup(vma_addr) | |
| 451 | + .and_then(|vma| check_vma(vma, self)) | |
| 452 | + }; | |
| 453 | + | |
| 454 | + match vma { | |
| 455 | + Some(vma) => vma.vm_insert_page(user_page_addr, &new_page)?, | |
| 456 | + None => return Err(ESRCH), | |
| 457 | + } | |
| 458 | + } | |
| 418 | 459 | |
| 419 | 460 | let inner = self.lock.lock(); |
| 420 | 461 | |
| @@ -667,12 +708,15 @@ unsafe extern "C" fn rust_shrink_free_page( · 246 unmodified lines | ||
| 667 | 708 | let mmap_read; |
| 668 | 709 | let mm_mutex; |
| 669 | 710 | let vma_addr; |
| 711 | + let range_ptr; | |
| 670 | 712 | |
| 671 | 713 | { |
| 672 | 714 | // CAST: The `list_head` field is first in `PageInfo`. |
| 673 | 715 | let info = item as *mut PageInfo; |
| 674 | 716 | // SAFETY: The `range` field of `PageInfo` is immutable. |
| 675 | − let range = unsafe { &*((*info).range) }; | |
| 717 | + range_ptr = unsafe { (*info).range }; | |
| 718 | + // SAFETY: The `range` outlives its `PageInfo` values. | |
| 719 | + let range = unsafe { &*range_ptr }; | |
| 676 | 720 | |
| 677 | 721 | mm = match range.mm.mmget_not_zero() { |
| 678 | 722 | Some(mm) => MmWithUser::into_mmput_async(mm), |
| @@ -717,9 +761,11 @@ unsafe extern "C" fn rust_shrink_free_page( · 38 unmodified lines | ||
| 717 | 761 | // SAFETY: The lru lock is locked when this method is called. |
| 718 | 762 | unsafe { bindings::spin_unlock(&raw mut (*lru).lock) }; |
| 719 | 763 | |
| 720 | − if let Some(vma) = mmap_read.vma_lookup(vma_addr) { | |
| 721 | − let user_page_addr = vma_addr + (page_index << PAGE_SHIFT); | |
| 722 | − vma.zap_page_range_single(user_page_addr, PAGE_SIZE); | |
| 764 | + if let Some(unchecked_vma) = mmap_read.vma_lookup(vma_addr) { | |
| 765 | + if let Some(vma) = check_vma(unchecked_vma, range_ptr) { | |
| 766 | + let user_page_addr = vma_addr + (page_index << PAGE_SHIFT); | |
| 767 | + vma.zap_page_range_single(user_page_addr, PAGE_SIZE); | |
| 768 | + } | |
| 723 | 769 | } |
| 724 | 770 | |
| 725 | 771 | drop(mmap_read); |
| @@ -1295,7 +1295,8 @@ impl Process { · 1294 unmodified lines | ||
| 1295 | 1295 | } |
| 1296 | 1296 | |
| 1297 | 1297 | pub(crate) fn dead_binder_done(&self, cookie: u64, thread: &Thread) { |
| 1298 | − if let Some(death) = self.inner.lock().pull_delivered_death(cookie) { | |
| 1298 | + let death = self.inner.lock().pull_delivered_death(cookie); | |
| 1299 | + if let Some(death) = death { | |
| 1299 | 1300 | death.set_notification_done(thread); |
| 1300 | 1301 | } |
| 1301 | 1302 | } |
| @@ -118,7 +118,7 @@ impl<T> ArrayRangeAllocator<T> { · 117 unmodified lines | ||
| 118 | 118 | size: usize, |
| 119 | 119 | is_oneway: bool, |
| 120 | 120 | pid: Pid, |
| 121 | − ) -> Result<usize> { | |
| 121 | + ) -> Result<(usize, bool)> { | |
| 122 | 122 | // Compute new value of free_oneway_space, which is set only on success. |
| 123 | 123 | let new_oneway_space = if is_oneway { |
| 124 | 124 | match self.free_oneway_space.checked_sub(size) { |
| @@ -146,7 +146,38 @@ impl<T> ArrayRangeAllocator<T> { · 21 unmodified lines | ||
| 146 | 146 | .ok() |
| 147 | 147 | .unwrap(); |
| 148 | 148 | |
| 149 | − Ok(insert_at_offset) | |
| 149 | + // Start detecting spammers once we have less than 20% | |
| 150 | + // of async space left (which is less than 10% of total | |
| 151 | + // buffer size). | |
| 152 | + // | |
| 153 | + // (This will short-circuit, so `low_oneway_space` is | |
| 154 | + // only called when necessary.) | |
| 155 | + let oneway_spam_detected = | |
| 156 | + is_oneway && new_oneway_space < self.size / 10 && self.low_oneway_space(pid); | |
| 157 | + | |
| 158 | + Ok((insert_at_offset, oneway_spam_detected)) | |
| 159 | + } | |
| 160 | + | |
| 161 | + /// Find the amount and size of buffers allocated by the current caller. | |
| 162 | + /// | |
| 163 | + /// The idea is that once we cross the threshold, whoever is responsible | |
| 164 | + /// for the low async space is likely to try to send another async transaction, | |
| 165 | + /// and at some point we'll catch them in the act. This is more efficient | |
| 166 | + /// than keeping a map per pid. | |
| 167 | + fn low_oneway_space(&self, calling_pid: Pid) -> bool { | |
| 168 | + let mut total_alloc_size = 0; | |
| 169 | + let mut num_buffers = 0; | |
| 170 | + | |
| 171 | + // Warn if this pid has more than 50 transactions, or more than 50% of | |
| 172 | + // async space (which is 25% of total buffer size). Oneway spam is only | |
| 173 | + // detected when the threshold is exceeded. | |
| 174 | + for range in &self.ranges { | |
| 175 | + if range.state.is_oneway() && range.state.pid() == calling_pid { | |
| 176 | + total_alloc_size += range.size; | |
| 177 | + num_buffers += 1; | |
| 178 | + } | |
| 179 | + } | |
| 180 | + num_buffers > 50 || total_alloc_size > self.size / 4 | |
| 150 | 181 | } |
| 151 | 182 | |
| 152 | 183 | pub(crate) fn reservation_abort(&mut self, offset: usize) -> Result<FreedRange> { |
| @@ -188,11 +188,11 @@ impl<T> RangeAllocator<T> { · 187 unmodified lines | ||
| 188 | 188 | self.reserve_new(args) |
| 189 | 189 | } |
| 190 | 190 | Impl::Array(array) => { |
| 191 | − let offset = | |
| 191 | + let (offset, oneway_spam_detected) = | |
| 192 | 192 | array.reserve_new(args.debug_id, args.size, args.is_oneway, args.pid)?; |
| 193 | 193 | Ok(ReserveNew::Success(ReserveNewSuccess { |
| 194 | 194 | offset, |
| 195 | − oneway_spam_detected: false, | |
| 195 | + oneway_spam_detected, | |
| 196 | 196 | _empty_array_alloc: args.empty_array_alloc, |
| 197 | 197 | _new_tree_alloc: args.new_tree_alloc, |
| 198 | 198 | _tree_alloc: args.tree_alloc, |
| @@ -164,15 +164,6 @@ impl<T> TreeRangeAllocator<T> { · 163 unmodified lines | ||
| 164 | 164 | self.free_oneway_space |
| 165 | 165 | }; |
| 166 | 166 | |
| 167 | − // Start detecting spammers once we have less than 20% | |
| 168 | − // of async space left (which is less than 10% of total | |
| 169 | − // buffer size). | |
| 170 | − // | |
| 171 | − // (This will short-circut, so `low_oneway_space` is | |
| 172 | − // only called when necessary.) | |
| 173 | − let oneway_spam_detected = | |
| 174 | − is_oneway && new_oneway_space < self.size / 10 && self.low_oneway_space(pid); | |
| 175 | − | |
| 176 | 167 | let (found_size, found_off, tree_node, free_tree_node) = match self.find_best_match(size) { |
| 177 | 168 | None => { |
| 178 | 169 | pr_warn!("ENOSPC from range_alloc.reserve_new - size: {}", size); |
| @@ -203,6 +194,15 @@ impl<T> TreeRangeAllocator<T> { · 24 unmodified lines | ||
| 203 | 194 | self.free_tree.insert(free_tree_node); |
| 204 | 195 | } |
| 205 | 196 | |
| 197 | + // Start detecting spammers once we have less than 20% | |
| 198 | + // of async space left (which is less than 10% of total | |
| 199 | + // buffer size). | |
| 200 | + // | |
| 201 | + // (This will short-circuit, so `low_oneway_space` is | |
| 202 | + // only called when necessary.) | |
| 203 | + let oneway_spam_detected = | |
| 204 | + is_oneway && new_oneway_space < self.size / 10 && self.low_oneway_space(pid); | |
| 205 | + | |
| 206 | 206 | Ok((found_off, oneway_spam_detected)) |
| 207 | 207 | } |
| 208 | 208 | |
| @@ -306,7 +306,7 @@ impl kernel::Module for BinderModule { · 305 unmodified lines | ||
| 306 | 306 | /// Makes the inner type Sync. |
| 307 | 307 | #[repr(transparent)] |
| 308 | 308 | pub struct AssertSync<T>(T); |
| 309 | −// SAFETY: Used only to insert `file_operations` into a global, which is safe. | |
| 309 | +// SAFETY: Used only to insert C bindings types into globals, which is safe. | |
| 310 | 310 | unsafe impl<T> Sync for AssertSync<T> {} |
| 311 | 311 | |
| 312 | 312 | /// File operations that rust_binderfs.c can use. |
| @@ -1015,12 +1015,9 @@ impl Thread { · 1014 unmodified lines | ||
| 1015 | 1015 | |
| 1016 | 1016 | // Copy offsets if there are any. |
| 1017 | 1017 | if offsets_size > 0 { |
| 1018 | − { | |
| 1019 | − let mut reader = | |
| 1020 | − UserSlice::new(UserPtr::from_addr(trd_data_ptr.offsets as _), offsets_size) | |
| 1021 | − .reader(); | |
| 1022 | − alloc.copy_into(&mut reader, aligned_data_size, offsets_size)?; | |
| 1023 | − } | |
| 1018 | + let mut offsets_reader = | |
| 1019 | + UserSlice::new(UserPtr::from_addr(trd_data_ptr.offsets as _), offsets_size) | |
| 1020 | + .reader(); | |
| 1024 | 1021 | |
| 1025 | 1022 | let offsets_start = aligned_data_size; |
| 1026 | 1023 | let offsets_end = aligned_data_size + offsets_size; |
| @@ -1041,11 +1038,9 @@ impl Thread { · 14 unmodified lines | ||
| 1041 | 1038 | .step_by(size_of::<u64>()) |
| 1042 | 1039 | .enumerate() |
| 1043 | 1040 | { |
| 1044 | − let offset: usize = view | |
| 1045 | − .alloc | |
| 1046 | − .read::<u64>(index_offset)? | |
| 1047 | − .try_into() | |
| 1048 | − .map_err(|_| EINVAL)?; | |
| 1041 | + let offset = offsets_reader.read::<u64>()?; | |
| 1042 | + view.alloc.write(index_offset, &offset)?; | |
| 1043 | + let offset: usize = offset.try_into().map_err(|_| EINVAL)?; | |
| 1049 | 1044 | |
| 1050 | 1045 | if offset < end_of_previous_object || !is_aligned(offset, size_of::<u32>()) { |
| 1051 | 1046 | pr_warn!("Got transaction with invalid offset."); |
| @@ -4188,6 +4188,9 @@ static const struct ata_dev_quirks_entry __ata_dev_quirks[] = { · 4187 unmodified lines | ||
| 4188 | 4188 | { "ST3320[68]13AS", "SD1[5-9]", ATA_QUIRK_NONCQ | |
| 4189 | 4189 | ATA_QUIRK_FIRMWARE_WARN }, |
| 4190 | 4190 | |
| 4191 | + /* ADATA devices with LPM issues. */ | |
| 4192 | + { "ADATA SU680", NULL, ATA_QUIRK_NOLPM }, | |
| 4193 | + | |
| 4191 | 4194 | /* Seagate disks with LPM issues */ |
| 4192 | 4195 | { "ST1000DM010-2EP102", NULL, ATA_QUIRK_NOLPM }, |
| 4193 | 4196 | { "ST2000DM008-2FR102", NULL, ATA_QUIRK_NOLPM }, |
| @@ -3600,7 +3600,7 @@ static unsigned int ata_scsiop_maint_in(struct ata_device *dev, · 3599 unmodified lines | ||
| 3600 | 3600 | |
| 3601 | 3601 | if (cdb[2] != 1 && cdb[2] != 3) { |
| 3602 | 3602 | ata_dev_warn(dev, "invalid command format %d\n", cdb[2]); |
| 3603 | − ata_scsi_set_invalid_field(dev, cmd, 1, 0xff); | |
| 3603 | + ata_scsi_set_invalid_field(dev, cmd, 2, 0xff); | |
| 3604 | 3604 | return 0; |
| 3605 | 3605 | } |
| 3606 | 3606 | |
| @@ -99,8 +99,13 @@ static int lcd2s_print(struct charlcd *lcd, int c) · 98 unmodified lines | ||
| 99 | 99 | { |
| 100 | 100 | struct lcd2s_data *lcd2s = lcd->drvdata; |
| 101 | 101 | u8 buf[2] = { LCD2S_CMD_WRITE, c }; |
| 102 | + int ret; | |
| 102 | 103 | |
| 103 | − lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); | |
| 104 | + ret = lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); | |
| 105 | + if (ret < 0) | |
| 106 | + return ret; | |
| 107 | + if (ret != sizeof(buf)) | |
| 108 | + return -EIO; | |
| 104 | 109 | return 0; |
| 105 | 110 | } |
| 106 | 111 | |
| @@ -108,9 +113,13 @@ static int lcd2s_gotoxy(struct charlcd *lcd, unsigned int x, unsigned int y) · 1 unmodified line | ||
| 108 | 113 | { |
| 109 | 114 | struct lcd2s_data *lcd2s = lcd->drvdata; |
| 110 | 115 | u8 buf[3] = { LCD2S_CMD_CUR_POS, y + 1, x + 1 }; |
| 116 | + int ret; | |
| 111 | 117 | |
| 112 | − lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); | |
| 113 | − | |
| 118 | + ret = lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); | |
| 119 | + if (ret < 0) | |
| 120 | + return ret; | |
| 121 | + if (ret != sizeof(buf)) | |
| 122 | + return -EIO; | |
| 114 | 123 | return 0; |
| 115 | 124 | } |
| 116 | 125 | |
| @@ -365,7 +365,7 @@ static DEFINE_IDA(linedisp_id); · 364 unmodified lines | ||
| 365 | 365 | |
| 366 | 366 | static void linedisp_release(struct device *dev) |
| 367 | 367 | { |
| 368 | − struct linedisp *linedisp = to_linedisp(dev); | |
| 368 | + struct linedisp *linedisp = container_of(dev, struct linedisp, dev); | |
| 369 | 369 | |
| 370 | 370 | kfree(linedisp->map); |
| 371 | 371 | kfree(linedisp->message); |
| @@ -504,6 +504,36 @@ int bus_for_each_drv(const struct bus_type *bus, struct device_driver *start, · 503 unmodified lines | ||
| 504 | 504 | } |
| 505 | 505 | EXPORT_SYMBOL_GPL(bus_for_each_drv); |
| 506 | 506 | |
| 507 | +static ssize_t driver_override_store(struct device *dev, | |
| 508 | + struct device_attribute *attr, | |
| 509 | + const char *buf, size_t count) | |
| 510 | +{ | |
| 511 | + int ret; | |
| 512 | + | |
| 513 | + ret = __device_set_driver_override(dev, buf, count); | |
| 514 | + if (ret) | |
| 515 | + return ret; | |
| 516 | + | |
| 517 | + return count; | |
| 518 | +} | |
| 519 | + | |
| 520 | +static ssize_t driver_override_show(struct device *dev, | |
| 521 | + struct device_attribute *attr, char *buf) | |
| 522 | +{ | |
| 523 | + guard(spinlock)(&dev->driver_override.lock); | |
| 524 | + return sysfs_emit(buf, "%s\n", dev->driver_override.name); | |
| 525 | +} | |
| 526 | +static DEVICE_ATTR_RW(driver_override); | |
| 527 | + | |
| 528 | +static struct attribute *driver_override_dev_attrs[] = { | |
| 529 | + &dev_attr_driver_override.attr, | |
| 530 | + NULL, | |
| 531 | +}; | |
| 532 | + | |
| 533 | +static const struct attribute_group driver_override_dev_group = { | |
| 534 | + .attrs = driver_override_dev_attrs, | |
| 535 | +}; | |
| 536 | + | |
| 507 | 537 | /** |
| 508 | 538 | * bus_add_device - add device to bus |
| 509 | 539 | * @dev: device being added |
| @@ -537,9 +567,15 @@ int bus_add_device(struct device *dev) · 27 unmodified lines | ||
| 537 | 567 | if (error) |
| 538 | 568 | goto out_put; |
| 539 | 569 | |
| 570 | + if (dev->bus->driver_override) { | |
| 571 | + error = device_add_group(dev, &driver_override_dev_group); | |
| 572 | + if (error) | |
| 573 | + goto out_groups; | |
| 574 | + } | |
| 575 | + | |
| 540 | 576 | error = sysfs_create_link(&sp->devices_kset->kobj, &dev->kobj, dev_name(dev)); |
| 541 | 577 | if (error) |
| 542 | − goto out_groups; | |
| 578 | + goto out_override; | |
| 543 | 579 | |
| 544 | 580 | error = sysfs_create_link(&dev->kobj, &sp->subsys.kobj, "subsystem"); |
| 545 | 581 | if (error) |
| @@ -550,6 +586,9 @@ int bus_add_device(struct device *dev) · 4 unmodified lines | ||
| 550 | 586 | |
| 551 | 587 | out_subsys: |
| 552 | 588 | sysfs_remove_link(&sp->devices_kset->kobj, dev_name(dev)); |
| 589 | +out_override: | |
| 590 | + if (dev->bus->driver_override) | |
| 591 | + device_remove_group(dev, &driver_override_dev_group); | |
| 553 | 592 | out_groups: |
| 554 | 593 | device_remove_groups(dev, sp->bus->dev_groups); |
| 555 | 594 | out_put: |
| @@ -607,6 +646,8 @@ void bus_remove_device(struct device *dev) · 51 unmodified lines | ||
| 607 | 646 | |
| 608 | 647 | sysfs_remove_link(&dev->kobj, "subsystem"); |
| 609 | 648 | sysfs_remove_link(&sp->devices_kset->kobj, dev_name(dev)); |
| 649 | + if (dev->bus->driver_override) | |
| 650 | + device_remove_group(dev, &driver_override_dev_group); | |
| 610 | 651 | device_remove_groups(dev, dev->bus->dev_groups); |
| 611 | 652 | if (klist_node_attached(&dev->p->knode_bus)) |
| 612 | 653 | klist_del(&dev->p->knode_bus); |
| @@ -2556,6 +2556,7 @@ static void device_release(struct kobject *kobj) · 2555 unmodified lines | ||
| 2556 | 2556 | devres_release_all(dev); |
| 2557 | 2557 | |
| 2558 | 2558 | kfree(dev->dma_range_map); |
| 2559 | + kfree(dev->driver_override.name); | |
| 2559 | 2560 | |
| 2560 | 2561 | if (dev->release) |
| 2561 | 2562 | dev->release(dev); |
| @@ -3159,6 +3160,7 @@ void device_initialize(struct device *dev) · 597 unmodified lines | ||
| 3159 | 3160 | kobject_init(&dev->kobj, &device_ktype); |
| 3160 | 3161 | INIT_LIST_HEAD(&dev->dma_pools); |
| 3161 | 3162 | mutex_init(&dev->mutex); |
| 3163 | + spin_lock_init(&dev->driver_override.lock); | |
| 3162 | 3164 | lockdep_set_novalidate_class(&dev->mutex); |
| 3163 | 3165 | spin_lock_init(&dev->devres_lock); |
| 3164 | 3166 | INIT_LIST_HEAD(&dev->devres_head); |
| @@ -381,6 +381,66 @@ static void __exit deferred_probe_exit(void) · 380 unmodified lines | ||
| 381 | 381 | } |
| 382 | 382 | __exitcall(deferred_probe_exit); |
| 383 | 383 | |
| 384 | +int __device_set_driver_override(struct device *dev, const char *s, size_t len) | |
| 385 | +{ | |
| 386 | + const char *new, *old; | |
| 387 | + char *cp; | |
| 388 | + | |
| 389 | + if (!s) | |
| 390 | + return -EINVAL; | |
| 391 | + | |
| 392 | + /* | |
| 393 | + * The stored value will be used in sysfs show callback (sysfs_emit()), | |
| 394 | + * which has a length limit of PAGE_SIZE and adds a trailing newline. | |
| 395 | + * Thus we can store one character less to avoid truncation during sysfs | |
| 396 | + * show. | |
| 397 | + */ | |
| 398 | + if (len >= (PAGE_SIZE - 1)) | |
| 399 | + return -EINVAL; | |
| 400 | + | |
| 401 | + /* | |
| 402 | + * Compute the real length of the string in case userspace sends us a | |
| 403 | + * bunch of \0 characters like python likes to do. | |
| 404 | + */ | |
| 405 | + len = strlen(s); | |
| 406 | + | |
| 407 | + if (!len) { | |
| 408 | + /* Empty string passed - clear override */ | |
| 409 | + spin_lock(&dev->driver_override.lock); | |
| 410 | + old = dev->driver_override.name; | |
| 411 | + dev->driver_override.name = NULL; | |
| 412 | + spin_unlock(&dev->driver_override.lock); | |
| 413 | + kfree(old); | |
| 414 | + | |
| 415 | + return 0; | |
| 416 | + } | |
| 417 | + | |
| 418 | + cp = strnchr(s, len, '\n'); | |
| 419 | + if (cp) | |
| 420 | + len = cp - s; | |
| 421 | + | |
| 422 | + new = kstrndup(s, len, GFP_KERNEL); | |
| 423 | + if (!new) | |
| 424 | + return -ENOMEM; | |
| 425 | + | |
| 426 | + spin_lock(&dev->driver_override.lock); | |
| 427 | + old = dev->driver_override.name; | |
| 428 | + if (cp != s) { | |
| 429 | + dev->driver_override.name = new; | |
| 430 | + spin_unlock(&dev->driver_override.lock); | |
| 431 | + } else { | |
| 432 | + /* "\n" passed - clear override */ | |
| 433 | + dev->driver_override.name = NULL; | |
| 434 | + spin_unlock(&dev->driver_override.lock); | |
| 435 | + | |
| 436 | + kfree(new); | |
| 437 | + } | |
| 438 | + kfree(old); | |
| 439 | + | |
| 440 | + return 0; | |
| 441 | +} | |
| 442 | +EXPORT_SYMBOL_GPL(__device_set_driver_override); | |
| 443 | + | |
| 384 | 444 | /** |
| 385 | 445 | * device_is_bound() - Check if device is bound to a driver |
| 386 | 446 | * @dev: device to check |
| @@ -523,6 +523,8 @@ static ssize_t node_read_meminfo(struct device *dev, · 522 unmodified lines | ||
| 523 | 523 | #ifdef CONFIG_UNACCEPTED_MEMORY |
| 524 | 524 | "Node %d Unaccepted: %8lu kB\n" |
| 525 | 525 | #endif |
| 526 | + "Node %d GPUActive: %8lu kB\n" | |
| 527 | + "Node %d GPUReclaim: %8lu kB\n" | |
| 526 | 528 | , |
| 527 | 529 | nid, K(node_page_state(pgdat, NR_FILE_DIRTY)), |
| 528 | 530 | nid, K(node_page_state(pgdat, NR_WRITEBACK)), |
| @@ -556,6 +558,9 @@ static ssize_t node_read_meminfo(struct device *dev, · 27 unmodified lines | ||
| 556 | 558 | , |
| 557 | 559 | nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED)) |
| 558 | 560 | #endif |
| 561 | + , | |
| 562 | + nid, K(node_page_state(pgdat, NR_GPU_ACTIVE)), | |
| 563 | + nid, K(node_page_state(pgdat, NR_GPU_RECLAIM)) | |
| 559 | 564 | ); |
| 560 | 565 | len += hugetlb_report_node_meminfo(buf, len, nid); |
| 561 | 566 | return len; |
| @@ -603,7 +603,6 @@ static void platform_device_release(struct device *dev) · 602 unmodified lines | ||
| 603 | 603 | kfree(pa->pdev.dev.platform_data); |
| 604 | 604 | kfree(pa->pdev.mfd_cell); |
| 605 | 605 | kfree(pa->pdev.resource); |
| 606 | − kfree(pa->pdev.driver_override); | |
| 607 | 606 | kfree(pa); |
| 608 | 607 | } |
| 609 | 608 | |
| @@ -1306,38 +1305,9 @@ static ssize_t numa_node_show(struct device *dev, · 696 unmodified lines | ||
| 1306 | 1305 | } |
| 1307 | 1306 | static DEVICE_ATTR_RO(numa_node); |
| 1308 | 1307 | |
| 1309 | −static ssize_t driver_override_show(struct device *dev, | |
| 1310 | − struct device_attribute *attr, char *buf) | |
| 1311 | −{ | |
| 1312 | − struct platform_device *pdev = to_platform_device(dev); | |
| 1313 | − ssize_t len; | |
| 1314 | − | |
| 1315 | − device_lock(dev); | |
| 1316 | − len = sysfs_emit(buf, "%s\n", pdev->driver_override); | |
| 1317 | − device_unlock(dev); | |
| 1318 | − | |
| 1319 | − return len; | |
| 1320 | −} | |
| 1321 | − | |
| 1322 | −static ssize_t driver_override_store(struct device *dev, | |
| 1323 | − struct device_attribute *attr, | |
| 1324 | − const char *buf, size_t count) | |
| 1325 | −{ | |
| 1326 | − struct platform_device *pdev = to_platform_device(dev); | |
| 1327 | − int ret; | |
| 1328 | − | |
| 1329 | − ret = driver_set_override(dev, &pdev->driver_override, buf, count); | |
| 1330 | − if (ret) | |
| 1331 | − return ret; | |
| 1332 | − | |
| 1333 | − return count; | |
| 1334 | −} | |
| 1335 | −static DEVICE_ATTR_RW(driver_override); | |
| 1336 | − | |
| 1337 | 1308 | static struct attribute *platform_dev_attrs[] = { |
| 1338 | 1309 | &dev_attr_modalias.attr, |
| 1339 | 1310 | &dev_attr_numa_node.attr, |
| 1340 | − &dev_attr_driver_override.attr, | |
| 1341 | 1311 | NULL, |
| 1342 | 1312 | }; |
| 1343 | 1313 | |
| @@ -1377,10 +1347,12 @@ static int platform_match(struct device *dev, const struct device_driver *drv) · 33 unmodified lines | ||
| 1377 | 1347 | { |
| 1378 | 1348 | struct platform_device *pdev = to_platform_device(dev); |
| 1379 | 1349 | struct platform_driver *pdrv = to_platform_driver(drv); |
| 1350 | + int ret; | |
| 1380 | 1351 | |
| 1381 | 1352 | /* When driver_override is set, only bind to the matching driver */ |
| 1382 | − if (pdev->driver_override) | |
| 1383 | − return !strcmp(pdev->driver_override, drv->name); | |
| 1353 | + ret = device_match_driver_override(dev, drv); | |
| 1354 | + if (ret >= 0) | |
| 1355 | + return ret; | |
| 1384 | 1356 | |
| 1385 | 1357 | /* Attempt an OF style match first */ |
| 1386 | 1358 | if (of_driver_match_device(dev, drv)) |
| @@ -1516,6 +1488,7 @@ static const struct dev_pm_ops platform_dev_pm_ops = { · 129 unmodified lines | ||
| 1516 | 1488 | const struct bus_type platform_bus_type = { |
| 1517 | 1489 | .name = "platform", |
| 1518 | 1490 | .dev_groups = platform_dev_groups, |
| 1491 | + .driver_override = true, | |
| 1519 | 1492 | .match = platform_match, |
| 1520 | 1493 | .uevent = platform_uevent, |
| 1521 | 1494 | .probe = platform_probe, |
| @@ -1895,6 +1895,7 @@ void pm_runtime_reinit(struct device *dev) · 1894 unmodified lines | ||
| 1895 | 1895 | void pm_runtime_remove(struct device *dev) |
| 1896 | 1896 | { |
| 1897 | 1897 | __pm_runtime_disable(dev, false); |
| 1898 | + flush_work(&dev->power.work); | |
| 1898 | 1899 | pm_runtime_reinit(dev); |
| 1899 | 1900 | } |
| 1900 | 1901 | |
| @@ -1545,6 +1545,7 @@ static int _regmap_select_page(struct regmap *map, unsigned int *reg, · 1544 unmodified lines | ||
| 1545 | 1545 | unsigned int val_num) |
| 1546 | 1546 | { |
| 1547 | 1547 | void *orig_work_buf; |
| 1548 | + unsigned int selector_reg; | |
| 1548 | 1549 | unsigned int win_offset; |
| 1549 | 1550 | unsigned int win_page; |
| 1550 | 1551 | bool page_chg; |
| @@ -1563,10 +1564,31 @@ static int _regmap_select_page(struct regmap *map, unsigned int *reg, · 12 unmodified lines | ||
| 1563 | 1564 | return -EINVAL; |
| 1564 | 1565 | } |
| 1565 | 1566 | |
| 1566 | − /* It is possible to have selector register inside data window. | |
| 1567 | − In that case, selector register is located on every page and | |
| 1568 | − it needs no page switching, when accessed alone. */ | |
| 1567 | + /* | |
| 1568 | + * Calculate the address of the selector register in the corresponding | |
| 1569 | + * data window if it is located on every page. | |
| 1570 | + */ | |
| 1571 | + page_chg = in_range(range->selector_reg, range->window_start, range->window_len); | |
| 1572 | + if (page_chg) | |
| 1573 | + selector_reg = range->range_min + win_page * range->window_len + | |
| 1574 | + range->selector_reg - range->window_start; | |
| 1575 | + | |
| 1576 | + /* | |
| 1577 | + * It is possible to have selector register inside data window. | |
| 1578 | + * In that case, selector register is located on every page and it | |
| 1579 | + * needs no page switching, when accessed alone. | |
| 1580 | + * | |
| 1581 | + * Nevertheless we should synchronize the cache values for it. | |
| 1582 | + * This can't be properly achieved if the selector register is | |
| 1583 | + * the first and the only one to be read inside the data window. | |
| 1584 | + * That's why we update it in that case as well. | |
| 1585 | + * | |
| 1586 | + * However, we specifically avoid updating it for the default page, | |
| 1587 | + * when it's overlapped with the real data window, to prevent from | |
| 1588 | + * infinite looping. | |
| 1589 | + */ | |
| 1569 | 1590 | if (val_num > 1 || |
| 1591 | + (page_chg && selector_reg != range->selector_reg) || | |
| 1570 | 1592 | range->window_start + win_offset != range->selector_reg) { |
| 1571 | 1593 | /* Use separate work_buf during page switching */ |
| 1572 | 1594 | orig_work_buf = map->work_buf; |
| @@ -1575,7 +1597,7 @@ static int _regmap_select_page(struct regmap *map, unsigned int *reg, · 2 unmodified lines | ||
| 1575 | 1597 | ret = _regmap_update_bits(map, range->selector_reg, |
| 1576 | 1598 | range->selector_mask, |
| 1577 | 1599 | win_page << range->selector_shift, |
| 1578 | − &page_chg, false); | |
| 1600 | + NULL, false); | |
| 1579 | 1601 | |
| 1580 | 1602 | map->work_buf = orig_work_buf; |
| 1581 | 1603 | |
| @@ -4443,7 +4443,9 @@ static int ublk_ctrl_start_dev(struct ublk_device *ub, · 4442 unmodified lines | ||
| 4443 | 4443 | |
| 4444 | 4444 | /* Skip partition scan if disabled by user */ |
| 4445 | 4445 | if (ub->dev_info.flags & UBLK_F_NO_AUTO_PART_SCAN) { |
| 4446 | − clear_bit(GD_SUPPRESS_PART_SCAN, &disk->state); | |
| 4446 | + /* Not clear for unprivileged daemons, see comment above */ | |
| 4447 | + if (!ub->unprivileged_daemons) | |
| 4448 | + clear_bit(GD_SUPPRESS_PART_SCAN, &disk->state); | |
| 4447 | 4449 | } else { |
| 4448 | 4450 | /* Schedule async partition scan for trusted daemons */ |
| 4449 | 4451 | if (!ub->unprivileged_daemons) |
| @@ -5006,15 +5008,22 @@ static int ublk_ctrl_get_features(const struct ublksrv_ctrl_cmd *header) · 556 unmodified lines | ||
| 5006 | 5008 | return 0; |
| 5007 | 5009 | } |
| 5008 | 5010 | |
| 5009 | −static void ublk_ctrl_set_size(struct ublk_device *ub, const struct ublksrv_ctrl_cmd *header) | |
| 5011 | +static int ublk_ctrl_set_size(struct ublk_device *ub, const struct ublksrv_ctrl_cmd *header) | |
| 5010 | 5012 | { |
| 5011 | 5013 | struct ublk_param_basic *p = &ub->params.basic; |
| 5012 | 5014 | u64 new_size = header->data[0]; |
| 5015 | + int ret = 0; | |
| 5013 | 5016 | |
| 5014 | 5017 | mutex_lock(&ub->mutex); |
| 5018 | + if (!ub->ub_disk) { | |
| 5019 | + ret = -ENODEV; | |
| 5020 | + goto out; | |
| 5021 | + } | |
| 5015 | 5022 | p->dev_sectors = new_size; |
| 5016 | 5023 | set_capacity_and_notify(ub->ub_disk, p->dev_sectors); |
| 5024 | +out: | |
| 5017 | 5025 | mutex_unlock(&ub->mutex); |
| 5026 | + return ret; | |
| 5018 | 5027 | } |
| 5019 | 5028 | |
| 5020 | 5029 | struct count_busy { |
| @@ -5335,8 +5344,7 @@ static int ublk_ctrl_uring_cmd(struct io_uring_cmd *cmd, · 314 unmodified lines | ||
| 5335 | 5344 | ret = ublk_ctrl_end_recovery(ub, &header); |
| 5336 | 5345 | break; |
| 5337 | 5346 | case UBLK_CMD_UPDATE_SIZE: |
| 5338 | − ublk_ctrl_set_size(ub, &header); | |
| 5339 | − ret = 0; | |
| 5347 | + ret = ublk_ctrl_set_size(ub, &header); | |
| 5340 | 5348 | break; |
| 5341 | 5349 | case UBLK_CMD_QUIESCE_DEV: |
| 5342 | 5350 | ret = ublk_ctrl_quiesce_dev(ub, &header); |
| @@ -549,7 +549,7 @@ static ssize_t bd_stat_show(struct device *dev, struct device_attribute *attr, · 548 unmodified lines | ||
| 549 | 549 | return ret; |
| 550 | 550 | } |
| 551 | 551 | |
| 552 | −static ssize_t writeback_compressed_store(struct device *dev, | |
| 552 | +static ssize_t compressed_writeback_store(struct device *dev, | |
| 553 | 553 | struct device_attribute *attr, |
| 554 | 554 | const char *buf, size_t len) |
| 555 | 555 | { |
| @@ -564,12 +564,12 @@ static ssize_t writeback_compressed_store(struct device *dev, · 8 unmodified lines | ||
| 564 | 564 | return -EBUSY; |
| 565 | 565 | } |
| 566 | 566 | |
| 567 | − zram->wb_compressed = val; | |
| 567 | + zram->compressed_wb = val; | |
| 568 | 568 | |
| 569 | 569 | return len; |
| 570 | 570 | } |
| 571 | 571 | |
| 572 | −static ssize_t writeback_compressed_show(struct device *dev, | |
| 572 | +static ssize_t compressed_writeback_show(struct device *dev, | |
| 573 | 573 | struct device_attribute *attr, |
| 574 | 574 | char *buf) |
| 575 | 575 | { |
| @@ -577,7 +577,7 @@ static ssize_t writeback_compressed_show(struct device *dev, · 1 unmodified line | ||
| 577 | 577 | struct zram *zram = dev_to_zram(dev); |
| 578 | 578 | |
| 579 | 579 | guard(rwsem_read)(&zram->dev_lock); |
| 580 | − val = zram->wb_compressed; | |
| 580 | + val = zram->compressed_wb; | |
| 581 | 581 | |
| 582 | 582 | return sysfs_emit(buf, "%d\n", val); |
| 583 | 583 | } |
| @@ -917,9 +917,8 @@ static void zram_account_writeback_submit(struct zram *zram) · 333 unmodified lines | ||
| 917 | 917 | |
| 918 | 918 | static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) |
| 919 | 919 | { |
| 920 | − u32 size, index = req->pps->index; | |
| 921 | − int err, prio; | |
| 922 | − bool huge; | |
| 920 | + u32 index = req->pps->index; | |
| 921 | + int err; | |
| 923 | 922 | |
| 924 | 923 | err = blk_status_to_errno(req->bio.bi_status); |
| 925 | 924 | if (err) { |
| @@ -946,28 +945,13 @@ static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) · 20 unmodified lines | ||
| 946 | 945 | goto out; |
| 947 | 946 | } |
| 948 | 947 | |
| 949 | − if (zram->wb_compressed) { | |
| 950 | − /* | |
| 951 | − * ZRAM_WB slots get freed, we need to preserve data required | |
| 952 | − * for read decompression. | |
| 953 | − */ | |
| 954 | − size = get_slot_size(zram, index); | |
| 955 | − prio = get_slot_comp_priority(zram, index); | |
| 956 | − huge = test_slot_flag(zram, index, ZRAM_HUGE); | |
| 957 | − } | |
| 958 | − | |
| 959 | − slot_free(zram, index); | |
| 960 | − set_slot_flag(zram, index, ZRAM_WB); | |
| 948 | + clear_slot_flag(zram, index, ZRAM_IDLE); | |
| 949 | + if (test_slot_flag(zram, index, ZRAM_HUGE)) | |
| 950 | + atomic64_dec(&zram->stats.huge_pages); | |
| 951 | + atomic64_sub(get_slot_size(zram, index), &zram->stats.compr_data_size); | |
| 952 | + zs_free(zram->mem_pool, get_slot_handle(zram, index)); | |
| 961 | 953 | set_slot_handle(zram, index, req->blk_idx); |
| 962 | − | |
| 963 | − if (zram->wb_compressed) { | |
| 964 | − if (huge) | |
| 965 | − set_slot_flag(zram, index, ZRAM_HUGE); | |
| 966 | − set_slot_size(zram, index, size); | |
| 967 | − set_slot_comp_priority(zram, index, prio); | |
| 968 | − } | |
| 969 | − | |
| 970 | − atomic64_inc(&zram->stats.pages_stored); | |
| 954 | + set_slot_flag(zram, index, ZRAM_WB); | |
| 971 | 955 | |
| 972 | 956 | out: |
| 973 | 957 | slot_unlock(zram, index); |
| @@ -1100,7 +1084,7 @@ static int zram_writeback_slots(struct zram *zram, · 126 unmodified lines | ||
| 1100 | 1084 | */ |
| 1101 | 1085 | if (!test_slot_flag(zram, index, ZRAM_PP_SLOT)) |
| 1102 | 1086 | goto next; |
| 1103 | − if (zram->wb_compressed) | |
| 1087 | + if (zram->compressed_wb) | |
| 1104 | 1088 | err = read_from_zspool_raw(zram, req->page, index); |
| 1105 | 1089 | else |
| 1106 | 1090 | err = read_from_zspool(zram, req->page, index); |
| @@ -1429,7 +1413,7 @@ static void zram_async_read_endio(struct bio *bio) · 322 unmodified lines | ||
| 1429 | 1413 | * |
| 1430 | 1414 | * Keep the existing behavior for now. |
| 1431 | 1415 | */ |
| 1432 | − if (zram->wb_compressed == false) { | |
| 1416 | + if (zram->compressed_wb == false) { | |
| 1433 | 1417 | /* No decompression needed, complete the parent IO */ |
| 1434 | 1418 | bio_endio(req->parent); |
| 1435 | 1419 | bio_put(bio); |
| @@ -1508,7 +1492,7 @@ static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index, · 72 unmodified lines | ||
| 1508 | 1492 | flush_work(&req.work); |
| 1509 | 1493 | destroy_work_on_stack(&req.work); |
| 1510 | 1494 | |
| 1511 | − if (req.error || zram->wb_compressed == false) | |
| 1495 | + if (req.error || zram->compressed_wb == false) | |
| 1512 | 1496 | return req.error; |
| 1513 | 1497 | |
| 1514 | 1498 | return decompress_bdev_page(zram, page, index); |
| @@ -2010,8 +1994,13 @@ static void slot_free(struct zram *zram, u32 index) · 495 unmodified lines | ||
| 2010 | 1994 | set_slot_comp_priority(zram, index, 0); |
| 2011 | 1995 | |
| 2012 | 1996 | if (test_slot_flag(zram, index, ZRAM_HUGE)) { |
| 1997 | + /* | |
| 1998 | + * Writeback completion decrements ->huge_pages but keeps | |
| 1999 | + * ZRAM_HUGE flag for deferred decompression path. | |
| 2000 | + */ | |
| 2001 | + if (!test_slot_flag(zram, index, ZRAM_WB)) | |
| 2002 | + atomic64_dec(&zram->stats.huge_pages); | |
| 2013 | 2003 | clear_slot_flag(zram, index, ZRAM_HUGE); |
| 2014 | − atomic64_dec(&zram->stats.huge_pages); | |
| 2015 | 2004 | } |
| 2016 | 2005 | |
| 2017 | 2006 | if (test_slot_flag(zram, index, ZRAM_WB)) { |
| @@ -3007,7 +2996,7 @@ static DEVICE_ATTR_WO(writeback); · 989 unmodified lines | ||
| 3007 | 2996 | static DEVICE_ATTR_RW(writeback_limit); |
| 3008 | 2997 | static DEVICE_ATTR_RW(writeback_limit_enable); |
| 3009 | 2998 | static DEVICE_ATTR_RW(writeback_batch_size); |
| 3010 | −static DEVICE_ATTR_RW(writeback_compressed); | |
| 2999 | +static DEVICE_ATTR_RW(compressed_writeback); | |
| 3011 | 3000 | #endif |
| 3012 | 3001 | #ifdef CONFIG_ZRAM_MULTI_COMP |
| 3013 | 3002 | static DEVICE_ATTR_RW(recomp_algorithm); |
| @@ -3031,7 +3020,7 @@ static struct attribute *zram_disk_attrs[] = { · 17 unmodified lines | ||
| 3031 | 3020 | &dev_attr_writeback_limit.attr, |
| 3032 | 3021 | &dev_attr_writeback_limit_enable.attr, |
| 3033 | 3022 | &dev_attr_writeback_batch_size.attr, |
| 3034 | − &dev_attr_writeback_compressed.attr, | |
| 3023 | + &dev_attr_compressed_writeback.attr, | |
| 3035 | 3024 | #endif |
| 3036 | 3025 | &dev_attr_io_stat.attr, |
| 3037 | 3026 | &dev_attr_mm_stat.attr, |
| @@ -3091,7 +3080,7 @@ static int zram_add(void) · 53 unmodified lines | ||
| 3091 | 3080 | init_rwsem(&zram->dev_lock); |
| 3092 | 3081 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 3093 | 3082 | zram->wb_batch_size = 32; |
| 3094 | − zram->wb_compressed = false; | |
| 3083 | + zram->compressed_wb = false; | |
| 3095 | 3084 | #endif |
| 3096 | 3085 | |
| 3097 | 3086 | /* gendisk structure */ |
| @@ -133,7 +133,7 @@ struct zram { · 132 unmodified lines | ||
| 133 | 133 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 134 | 134 | struct file *backing_dev; |
| 135 | 135 | bool wb_limit_enable; |
| 136 | − bool wb_compressed; | |
| 136 | + bool compressed_wb; | |
| 137 | 137 | u32 wb_batch_size; |
| 138 | 138 | u64 bd_wb_limit; |
| 139 | 139 | struct block_device *bdev; |
| @@ -251,11 +251,13 @@ void btintel_hw_error(struct hci_dev *hdev, u8 code) · 250 unmodified lines | ||
| 251 | 251 | |
| 252 | 252 | bt_dev_err(hdev, "Hardware error 0x%2.2x", code); |
| 253 | 253 | |
| 254 | + hci_req_sync_lock(hdev); | |
| 255 | + | |
| 254 | 256 | skb = __hci_cmd_sync(hdev, HCI_OP_RESET, 0, NULL, HCI_INIT_TIMEOUT); |
| 255 | 257 | if (IS_ERR(skb)) { |
| 256 | 258 | bt_dev_err(hdev, "Reset after hardware error failed (%ld)", |
| 257 | 259 | PTR_ERR(skb)); |
| 258 | − return; | |
| 260 | + goto unlock; | |
| 259 | 261 | } |
| 260 | 262 | kfree_skb(skb); |
| 261 | 263 | |
| @@ -263,18 +265,21 @@ void btintel_hw_error(struct hci_dev *hdev, u8 code) · 1 unmodified line | ||
| 263 | 265 | if (IS_ERR(skb)) { |
| 264 | 266 | bt_dev_err(hdev, "Retrieving Intel exception info failed (%ld)", |
| 265 | 267 | PTR_ERR(skb)); |
| 266 | − return; | |
| 268 | + goto unlock; | |
| 267 | 269 | } |
| 268 | 270 | |
| 269 | 271 | if (skb->len != 13) { |
| 270 | 272 | bt_dev_err(hdev, "Exception info size mismatch"); |
| 271 | 273 | kfree_skb(skb); |
| 272 | − return; | |
| 274 | + goto unlock; | |
| 273 | 275 | } |
| 274 | 276 | |
| 275 | 277 | bt_dev_err(hdev, "Exception info %s", (char *)(skb->data + 1)); |
| 276 | 278 | |
| 277 | 279 | kfree_skb(skb); |
| 280 | + | |
| 281 | +unlock: | |
| 282 | + hci_req_sync_unlock(hdev); | |
| 278 | 283 | } |
| 279 | 284 | EXPORT_SYMBOL_GPL(btintel_hw_error); |
| 280 | 285 | |
| @@ -787,6 +787,8 @@ int qca_uart_setup(struct hci_dev *hdev, uint8_t baudrate, · 786 unmodified lines | ||
| 787 | 787 | */ |
| 788 | 788 | if (soc_type == QCA_WCN3988) |
| 789 | 789 | rom_ver = ((soc_ver & 0x00000f00) >> 0x05) | (soc_ver & 0x0000000f); |
| 790 | + else if (soc_type == QCA_WCN3998) | |
| 791 | + rom_ver = ((soc_ver & 0x0000f000) >> 0x07) | (soc_ver & 0x0000000f); | |
| 790 | 792 | else |
| 791 | 793 | rom_ver = ((soc_ver & 0x00000f00) >> 0x04) | (soc_ver & 0x0000000f); |
| 792 | 794 | |
| @@ -2376,8 +2376,11 @@ static void btusb_work(struct work_struct *work) · 2375 unmodified lines | ||
| 2376 | 2376 | if (data->air_mode == HCI_NOTIFY_ENABLE_SCO_CVSD) { |
| 2377 | 2377 | if (hdev->voice_setting & 0x0020) { |
| 2378 | 2378 | static const int alts[3] = { 2, 4, 5 }; |
| 2379 | + unsigned int sco_idx; | |
| 2379 | 2380 | |
| 2380 | − new_alts = alts[data->sco_num - 1]; | |
| 2381 | + sco_idx = min_t(unsigned int, data->sco_num - 1, | |
| 2382 | + ARRAY_SIZE(alts) - 1); | |
| 2383 | + new_alts = alts[sco_idx]; | |
| 2381 | 2384 | } else { |
| 2382 | 2385 | new_alts = data->sco_num; |
| 2383 | 2386 | } |
| @@ -109,9 +109,6 @@ static int h4_recv(struct hci_uart *hu, const void *data, int count) · 108 unmodified lines | ||
| 109 | 109 | { |
| 110 | 110 | struct h4_struct *h4 = hu->priv; |
| 111 | 111 | |
| 112 | − if (!test_bit(HCI_UART_REGISTERED, &hu->flags)) | |
| 113 | − return -EUNATCH; | |
| 114 | − | |
| 115 | 112 | h4->rx_skb = h4_recv_buf(hu, h4->rx_skb, data, count, |
| 116 | 113 | h4_recv_pkts, ARRAY_SIZE(h4_recv_pkts)); |
| 117 | 114 | if (IS_ERR(h4->rx_skb)) { |
| @@ -541,6 +541,8 @@ static int download_firmware(struct ll_device *lldev) · 540 unmodified lines | ||
| 541 | 541 | if (err || !fw->data || !fw->size) { |
| 542 | 542 | bt_dev_err(lldev->hu.hdev, "request_firmware failed(errno %d) for %s", |
| 543 | 543 | err, bts_scr_name); |
| 544 | + if (!err) | |
| 545 | + release_firmware(fw); | |
| 544 | 546 | return -EINVAL; |
| 545 | 547 | } |
| 546 | 548 | ptr = (void *)fw->data; |
| @@ -36,7 +36,7 @@ static int simple_pm_bus_probe(struct platform_device *pdev) · 35 unmodified lines | ||
| 36 | 36 | * that's not listed in simple_pm_bus_of_match. We don't want to do any |
| 37 | 37 | * of the simple-pm-bus tasks for these devices, so return early. |
| 38 | 38 | */ |
| 39 | − if (pdev->driver_override) | |
| 39 | + if (device_has_driver_override(&pdev->dev)) | |
| 40 | 40 | return 0; |
| 41 | 41 | |
| 42 | 42 | match = of_match_device(dev->driver->of_match_table, dev); |
| @@ -78,7 +78,7 @@ static void simple_pm_bus_remove(struct platform_device *pdev) · 35 unmodified lines | ||
| 78 | 78 | { |
| 79 | 79 | const void *data = of_device_get_match_data(&pdev->dev); |
| 80 | 80 | |
| 81 | − if (pdev->driver_override || data) | |
| 81 | + if (device_has_driver_override(&pdev->dev) || data) | |
| 82 | 82 | return; |
| 83 | 83 | |
| 84 | 84 | dev_dbg(&pdev->dev, "%s\n", __func__); |
| @@ -178,11 +178,11 @@ static const struct of_device_id ax45mp_cache_ids[] = { · 177 unmodified lines | ||
| 178 | 178 | |
| 179 | 179 | static int __init ax45mp_cache_init(void) |
| 180 | 180 | { |
| 181 | − struct device_node *np; | |
| 182 | 181 | struct resource res; |
| 183 | 182 | int ret; |
| 184 | 183 | |
| 185 | − np = of_find_matching_node(NULL, ax45mp_cache_ids); | |
| 184 | + struct device_node *np __free(device_node) = | |
| 185 | + of_find_matching_node(NULL, ax45mp_cache_ids); | |
| 186 | 186 | if (!of_device_is_available(np)) |
| 187 | 187 | return -ENODEV; |
| 188 | 188 | |
| @@ -102,11 +102,11 @@ static const struct of_device_id starlink_cache_ids[] = { · 101 unmodified lines | ||
| 102 | 102 | |
| 103 | 103 | static int __init starlink_cache_init(void) |
| 104 | 104 | { |
| 105 | − struct device_node *np; | |
| 106 | 105 | u32 block_size; |
| 107 | 106 | int ret; |
| 108 | 107 | |
| 109 | − np = of_find_matching_node(NULL, starlink_cache_ids); | |
| 108 | + struct device_node *np __free(device_node) = | |
| 109 | + of_find_matching_node(NULL, starlink_cache_ids); | |
| 110 | 110 | if (!of_device_is_available(np)) |
| 111 | 111 | return -ENODEV; |
| 112 | 112 | |
| @@ -706,8 +706,7 @@ struct clk_hw *imx_clk_scu_alloc_dev(const char *name, · 705 unmodified lines | ||
| 706 | 706 | if (ret) |
| 707 | 707 | goto put_device; |
| 708 | 708 | |
| 709 | − ret = driver_set_override(&pdev->dev, &pdev->driver_override, | |
| 710 | − "imx-scu-clk", strlen("imx-scu-clk")); | |
| 709 | + ret = device_set_driver_override(&pdev->dev, "imx-scu-clk"); | |
| 711 | 710 | if (ret) |
| 712 | 711 | goto put_device; |
| 713 | 712 | |
| @@ -793,13 +793,15 @@ static void do_become_nonbusy(struct comedi_device *dev, · 792 unmodified lines | ||
| 793 | 793 | __comedi_clear_subdevice_runflags(s, COMEDI_SRF_RUNNING | |
| 794 | 794 | COMEDI_SRF_BUSY); |
| 795 | 795 | spin_unlock_irqrestore(&s->spin_lock, flags); |
| 796 | − if (comedi_is_runflags_busy(runflags)) { | |
| 796 | + if (async) { | |
| 797 | 797 | /* |
| 798 | 798 | * "Run active" counter was set to 1 when setting up the |
| 799 | 799 | * command. Decrement it and wait for it to become 0. |
| 800 | 800 | */ |
| 801 | − comedi_put_is_subdevice_running(s); | |
| 802 | − wait_for_completion(&async->run_complete); | |
| 801 | + if (comedi_is_runflags_busy(runflags)) { | |
| 802 | + comedi_put_is_subdevice_running(s); | |
| 803 | + wait_for_completion(&async->run_complete); | |
| 804 | + } | |
| 803 | 805 | comedi_buf_reset(s); |
| 804 | 806 | async->inttrig = NULL; |
| 805 | 807 | kfree(async->cmd.chanlist); |
| @@ -1063,6 +1063,14 @@ int comedi_device_attach(struct comedi_device *dev, struct comedi_devconfig *it) · 1062 unmodified lines | ||
| 1063 | 1063 | ret = -EIO; |
| 1064 | 1064 | goto out; |
| 1065 | 1065 | } |
| 1066 | + if (IS_ENABLED(CONFIG_LOCKDEP)) { | |
| 1067 | + /* | |
| 1068 | + * dev->spinlock is for private use by the attached low-level | |
| 1069 | + * driver. Reinitialize it to stop lock-dependency tracking | |
| 1070 | + * between attachments to different low-level drivers. | |
| 1071 | + */ | |
| 1072 | + spin_lock_init(&dev->spinlock); | |
| 1073 | + } | |
| 1066 | 1074 | dev->driver = driv; |
| 1067 | 1075 | dev->board_name = dev->board_ptr ? *(const char **)dev->board_ptr |
| 1068 | 1076 | : dev->driver->driver_name; |
| @@ -175,6 +175,18 @@ static int dt2815_attach(struct comedi_device *dev, struct comedi_devconfig *it) · 174 unmodified lines | ||
| 175 | 175 | ? current_range_type : voltage_range_type; |
| 176 | 176 | } |
| 177 | 177 | |
| 178 | + /* | |
| 179 | + * Check if hardware is present before attempting any I/O operations. | |
| 180 | + * Reading 0xff from status register typically indicates no hardware | |
| 181 | + * on the bus (floating bus reads as all 1s). | |
| 182 | + */ | |
| 183 | + if (inb(dev->iobase + DT2815_STATUS) == 0xff) { | |
| 184 | + dev_err(dev->class_dev, | |
| 185 | + "No hardware detected at I/O base 0x%lx\n", | |
| 186 | + dev->iobase); | |
| 187 | + return -ENODEV; | |
| 188 | + } | |
| 189 | + | |
| 178 | 190 | /* Init the 2815 */ |
| 179 | 191 | outb(0x00, dev->iobase + DT2815_STATUS); |
| 180 | 192 | for (i = 0; i < 100; i++) { |
| @@ -315,6 +315,18 @@ static int me4000_xilinx_download(struct comedi_device *dev, · 314 unmodified lines | ||
| 315 | 315 | unsigned int val; |
| 316 | 316 | unsigned int i; |
| 317 | 317 | |
| 318 | + /* Get data stream length from header. */ | |
| 319 | + if (size >= 4) { | |
| 320 | + file_length = (((unsigned int)data[0] & 0xff) << 24) + | |
| 321 | + (((unsigned int)data[1] & 0xff) << 16) + | |
| 322 | + (((unsigned int)data[2] & 0xff) << 8) + | |
| 323 | + ((unsigned int)data[3] & 0xff); | |
| 324 | + } | |
| 325 | + if (size < 16 || file_length > size - 16) { | |
| 326 | + dev_err(dev->class_dev, "Firmware length inconsistency\n"); | |
| 327 | + return -EINVAL; | |
| 328 | + } | |
| 329 | + | |
| 318 | 330 | if (!xilinx_iobase) |
| 319 | 331 | return -ENODEV; |
| 320 | 332 | |
| @@ -346,10 +358,6 @@ static int me4000_xilinx_download(struct comedi_device *dev, · 25 unmodified lines | ||
| 346 | 358 | outl(val, devpriv->plx_regbase + PLX9052_CNTRL); |
| 347 | 359 | |
| 348 | 360 | /* Download Xilinx firmware */ |
| 349 | − file_length = (((unsigned int)data[0] & 0xff) << 24) + | |
| 350 | − (((unsigned int)data[1] & 0xff) << 16) + | |
| 351 | − (((unsigned int)data[2] & 0xff) << 8) + | |
| 352 | − ((unsigned int)data[3] & 0xff); | |
| 353 | 361 | usleep_range(10, 1000); |
| 354 | 362 | |
| 355 | 363 | for (i = 0; i < file_length; i++) { |
| @@ -344,6 +344,25 @@ static int me2600_xilinx_download(struct comedi_device *dev, · 343 unmodified lines | ||
| 344 | 344 | unsigned int file_length; |
| 345 | 345 | unsigned int i; |
| 346 | 346 | |
| 347 | + /* | |
| 348 | + * Format of the firmware | |
| 349 | + * Build longs from the byte-wise coded header | |
| 350 | + * Byte 1-3: length of the array | |
| 351 | + * Byte 4-7: version | |
| 352 | + * Byte 8-11: date | |
| 353 | + * Byte 12-15: reserved | |
| 354 | + */ | |
| 355 | + if (size >= 4) { | |
| 356 | + file_length = (((unsigned int)data[0] & 0xff) << 24) + | |
| 357 | + (((unsigned int)data[1] & 0xff) << 16) + | |
| 358 | + (((unsigned int)data[2] & 0xff) << 8) + | |
| 359 | + ((unsigned int)data[3] & 0xff); | |
| 360 | + } | |
| 361 | + if (size < 16 || file_length > size - 16) { | |
| 362 | + dev_err(dev->class_dev, "Firmware length inconsistency\n"); | |
| 363 | + return -EINVAL; | |
| 364 | + } | |
| 365 | + | |
| 347 | 366 | /* disable irq's on PLX */ |
| 348 | 367 | writel(0x00, devpriv->plx_regbase + PLX9052_INTCSR); |
| 349 | 368 | |
| @@ -357,22 +376,6 @@ static int me2600_xilinx_download(struct comedi_device *dev, · 7 unmodified lines | ||
| 357 | 376 | writeb(0x00, dev->mmio + 0x0); |
| 358 | 377 | sleep(1); |
| 359 | 378 | |
| 360 | − /* | |
| 361 | − * Format of the firmware | |
| 362 | − * Build longs from the byte-wise coded header | |
| 363 | − * Byte 1-3: length of the array | |
| 364 | − * Byte 4-7: version | |
| 365 | − * Byte 8-11: date | |
| 366 | − * Byte 12-15: reserved | |
| 367 | − */ | |
| 368 | − if (size < 16) | |
| 369 | − return -EINVAL; | |
| 370 | − | |
| 371 | − file_length = (((unsigned int)data[0] & 0xff) << 24) + | |
| 372 | − (((unsigned int)data[1] & 0xff) << 16) + | |
| 373 | − (((unsigned int)data[2] & 0xff) << 8) + | |
| 374 | − ((unsigned int)data[3] & 0xff); | |
| 375 | − | |
| 376 | 379 | /* |
| 377 | 380 | * Loop for writing firmware byte by byte to xilinx |
| 378 | 381 | * Firmware data start at offset 16 |
| @@ -698,7 +698,8 @@ static int atmio16d_attach(struct comedi_device *dev, · 697 unmodified lines | ||
| 698 | 698 | |
| 699 | 699 | static void atmio16d_detach(struct comedi_device *dev) |
| 700 | 700 | { |
| 701 | − reset_atmio16d(dev); | |
| 701 | + if (dev->private) | |
| 702 | + reset_atmio16d(dev); | |
| 702 | 703 | comedi_legacy_detach(dev); |
| 703 | 704 | } |
| 704 | 705 | |
| @@ -107,9 +107,9 @@ static bool rz_mtu3_is_counter_invalid(struct counter_device *counter, int id) · 106 unmodified lines | ||
| 107 | 107 | struct rz_mtu3_cnt *const priv = counter_priv(counter); |
| 108 | 108 | unsigned long tmdr; |
| 109 | 109 | |
| 110 | − pm_runtime_get_sync(priv->ch->dev); | |
| 110 | + pm_runtime_get_sync(counter->parent); | |
| 111 | 111 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); |
| 112 | − pm_runtime_put(priv->ch->dev); | |
| 112 | + pm_runtime_put(counter->parent); | |
| 113 | 113 | |
| 114 | 114 | if (id == RZ_MTU3_32_BIT_CH && test_bit(RZ_MTU3_TMDR3_LWA, &tmdr)) |
| 115 | 115 | return false; |
| @@ -165,12 +165,12 @@ static int rz_mtu3_count_read(struct counter_device *counter, · 49 unmodified lines | ||
| 165 | 165 | if (ret) |
| 166 | 166 | return ret; |
| 167 | 167 | |
| 168 | − pm_runtime_get_sync(ch->dev); | |
| 168 | + pm_runtime_get_sync(counter->parent); | |
| 169 | 169 | if (count->id == RZ_MTU3_32_BIT_CH) |
| 170 | 170 | *val = rz_mtu3_32bit_ch_read(ch, RZ_MTU3_TCNTLW); |
| 171 | 171 | else |
| 172 | 172 | *val = rz_mtu3_16bit_ch_read(ch, RZ_MTU3_TCNT); |
| 173 | − pm_runtime_put(ch->dev); | |
| 173 | + pm_runtime_put(counter->parent); | |
| 174 | 174 | mutex_unlock(&priv->lock); |
| 175 | 175 | |
| 176 | 176 | return 0; |
| @@ -187,26 +187,26 @@ static int rz_mtu3_count_write(struct counter_device *counter, · 10 unmodified lines | ||
| 187 | 187 | if (ret) |
| 188 | 188 | return ret; |
| 189 | 189 | |
| 190 | − pm_runtime_get_sync(ch->dev); | |
| 190 | + pm_runtime_get_sync(counter->parent); | |
| 191 | 191 | if (count->id == RZ_MTU3_32_BIT_CH) |
| 192 | 192 | rz_mtu3_32bit_ch_write(ch, RZ_MTU3_TCNTLW, val); |
| 193 | 193 | else |
| 194 | 194 | rz_mtu3_16bit_ch_write(ch, RZ_MTU3_TCNT, val); |
| 195 | − pm_runtime_put(ch->dev); | |
| 195 | + pm_runtime_put(counter->parent); | |
| 196 | 196 | mutex_unlock(&priv->lock); |
| 197 | 197 | |
| 198 | 198 | return 0; |
| 199 | 199 | } |
| 200 | 200 | |
| 201 | 201 | static int rz_mtu3_count_function_read_helper(struct rz_mtu3_channel *const ch, |
| 202 | − struct rz_mtu3_cnt *const priv, | |
| 202 | + struct counter_device *const counter, | |
| 203 | 203 | enum counter_function *function) |
| 204 | 204 | { |
| 205 | 205 | u8 timer_mode; |
| 206 | 206 | |
| 207 | − pm_runtime_get_sync(ch->dev); | |
| 207 | + pm_runtime_get_sync(counter->parent); | |
| 208 | 208 | timer_mode = rz_mtu3_8bit_ch_read(ch, RZ_MTU3_TMDR1); |
| 209 | − pm_runtime_put(ch->dev); | |
| 209 | + pm_runtime_put(counter->parent); | |
| 210 | 210 | |
| 211 | 211 | switch (timer_mode & RZ_MTU3_TMDR1_PH_CNT_MODE_MASK) { |
| 212 | 212 | case RZ_MTU3_TMDR1_PH_CNT_MODE_1: |
| @@ -240,7 +240,7 @@ static int rz_mtu3_count_function_read(struct counter_device *counter, · 27 unmodified lines | ||
| 240 | 240 | if (ret) |
| 241 | 241 | return ret; |
| 242 | 242 | |
| 243 | − ret = rz_mtu3_count_function_read_helper(ch, priv, function); | |
| 243 | + ret = rz_mtu3_count_function_read_helper(ch, counter, function); | |
| 244 | 244 | mutex_unlock(&priv->lock); |
| 245 | 245 | |
| 246 | 246 | return ret; |
| @@ -279,9 +279,9 @@ static int rz_mtu3_count_function_write(struct counter_device *counter, · 32 unmodified lines | ||
| 279 | 279 | return -EINVAL; |
| 280 | 280 | } |
| 281 | 281 | |
| 282 | − pm_runtime_get_sync(ch->dev); | |
| 282 | + pm_runtime_get_sync(counter->parent); | |
| 283 | 283 | rz_mtu3_8bit_ch_write(ch, RZ_MTU3_TMDR1, timer_mode); |
| 284 | − pm_runtime_put(ch->dev); | |
| 284 | + pm_runtime_put(counter->parent); | |
| 285 | 285 | mutex_unlock(&priv->lock); |
| 286 | 286 | |
| 287 | 287 | return 0; |
| @@ -300,9 +300,9 @@ static int rz_mtu3_count_direction_read(struct counter_device *counter, · 12 unmodified lines | ||
| 300 | 300 | if (ret) |
| 301 | 301 | return ret; |
| 302 | 302 | |
| 303 | − pm_runtime_get_sync(ch->dev); | |
| 303 | + pm_runtime_get_sync(counter->parent); | |
| 304 | 304 | tsr = rz_mtu3_8bit_ch_read(ch, RZ_MTU3_TSR); |
| 305 | − pm_runtime_put(ch->dev); | |
| 305 | + pm_runtime_put(counter->parent); | |
| 306 | 306 | |
| 307 | 307 | *direction = (tsr & RZ_MTU3_TSR_TCFD) ? |
| 308 | 308 | COUNTER_COUNT_DIRECTION_FORWARD : COUNTER_COUNT_DIRECTION_BACKWARD; |
| @@ -377,14 +377,14 @@ static int rz_mtu3_count_ceiling_write(struct counter_device *counter, · 68 unmodified lines | ||
| 377 | 377 | return -EINVAL; |
| 378 | 378 | } |
| 379 | 379 | |
| 380 | − pm_runtime_get_sync(ch->dev); | |
| 380 | + pm_runtime_get_sync(counter->parent); | |
| 381 | 381 | if (count->id == RZ_MTU3_32_BIT_CH) |
| 382 | 382 | rz_mtu3_32bit_ch_write(ch, RZ_MTU3_TGRALW, ceiling); |
| 383 | 383 | else |
| 384 | 384 | rz_mtu3_16bit_ch_write(ch, RZ_MTU3_TGRA, ceiling); |
| 385 | 385 | |
| 386 | 386 | rz_mtu3_8bit_ch_write(ch, RZ_MTU3_TCR, RZ_MTU3_TCR_CCLR_TGRA); |
| 387 | − pm_runtime_put(ch->dev); | |
| 387 | + pm_runtime_put(counter->parent); | |
| 388 | 388 | mutex_unlock(&priv->lock); |
| 389 | 389 | |
| 390 | 390 | return 0; |
| @@ -495,25 +495,28 @@ static int rz_mtu3_count_enable_read(struct counter_device *counter, · 104 unmodified lines | ||
| 495 | 495 | static int rz_mtu3_count_enable_write(struct counter_device *counter, |
| 496 | 496 | struct counter_count *count, u8 enable) |
| 497 | 497 | { |
| 498 | − struct rz_mtu3_channel *const ch = rz_mtu3_get_ch(counter, count->id); | |
| 499 | 498 | struct rz_mtu3_cnt *const priv = counter_priv(counter); |
| 500 | 499 | int ret = 0; |
| 501 | 500 | |
| 501 | + mutex_lock(&priv->lock); | |
| 502 | + | |
| 503 | + if (priv->count_is_enabled[count->id] == enable) | |
| 504 | + goto exit; | |
| 505 | + | |
| 502 | 506 | if (enable) { |
| 503 | − mutex_lock(&priv->lock); | |
| 504 | − pm_runtime_get_sync(ch->dev); | |
| 507 | + pm_runtime_get_sync(counter->parent); | |
| 505 | 508 | ret = rz_mtu3_initialize_counter(counter, count->id); |
| 506 | 509 | if (ret == 0) |
| 507 | 510 | priv->count_is_enabled[count->id] = true; |
| 508 | − mutex_unlock(&priv->lock); | |
| 509 | 511 | } else { |
| 510 | − mutex_lock(&priv->lock); | |
| 511 | 512 | rz_mtu3_terminate_counter(counter, count->id); |
| 512 | 513 | priv->count_is_enabled[count->id] = false; |
| 513 | − pm_runtime_put(ch->dev); | |
| 514 | − mutex_unlock(&priv->lock); | |
| 514 | + pm_runtime_put(counter->parent); | |
| 515 | 515 | } |
| 516 | 516 | |
| 517 | +exit: | |
| 518 | + mutex_unlock(&priv->lock); | |
| 519 | + | |
| 517 | 520 | return ret; |
| 518 | 521 | } |
| 519 | 522 | |
| @@ -540,9 +543,9 @@ static int rz_mtu3_cascade_counts_enable_get(struct counter_device *counter, · 20 unmodified lines | ||
| 540 | 543 | if (ret) |
| 541 | 544 | return ret; |
| 542 | 545 | |
| 543 | − pm_runtime_get_sync(priv->ch->dev); | |
| 546 | + pm_runtime_get_sync(counter->parent); | |
| 544 | 547 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); |
| 545 | − pm_runtime_put(priv->ch->dev); | |
| 548 | + pm_runtime_put(counter->parent); | |
| 546 | 549 | *cascade_enable = test_bit(RZ_MTU3_TMDR3_LWA, &tmdr); |
| 547 | 550 | mutex_unlock(&priv->lock); |
| 548 | 551 | |
| @@ -559,10 +562,10 @@ static int rz_mtu3_cascade_counts_enable_set(struct counter_device *counter, · 10 unmodified lines | ||
| 559 | 562 | if (ret) |
| 560 | 563 | return ret; |
| 561 | 564 | |
| 562 | − pm_runtime_get_sync(priv->ch->dev); | |
| 565 | + pm_runtime_get_sync(counter->parent); | |
| 563 | 566 | rz_mtu3_shared_reg_update_bit(priv->ch, RZ_MTU3_TMDR3, |
| 564 | 567 | RZ_MTU3_TMDR3_LWA, cascade_enable); |
| 565 | − pm_runtime_put(priv->ch->dev); | |
| 568 | + pm_runtime_put(counter->parent); | |
| 566 | 569 | mutex_unlock(&priv->lock); |
| 567 | 570 | |
| 568 | 571 | return 0; |
| @@ -579,9 +582,9 @@ static int rz_mtu3_ext_input_phase_clock_select_get(struct counter_device *count · 10 unmodified lines | ||
| 579 | 582 | if (ret) |
| 580 | 583 | return ret; |
| 581 | 584 | |
| 582 | − pm_runtime_get_sync(priv->ch->dev); | |
| 585 | + pm_runtime_get_sync(counter->parent); | |
| 583 | 586 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); |
| 584 | − pm_runtime_put(priv->ch->dev); | |
| 587 | + pm_runtime_put(counter->parent); | |
| 585 | 588 | *ext_input_phase_clock_select = test_bit(RZ_MTU3_TMDR3_PHCKSEL, &tmdr); |
| 586 | 589 | mutex_unlock(&priv->lock); |
| 587 | 590 | |
| @@ -598,11 +601,11 @@ static int rz_mtu3_ext_input_phase_clock_select_set(struct counter_device *count · 10 unmodified lines | ||
| 598 | 601 | if (ret) |
| 599 | 602 | return ret; |
| 600 | 603 | |
| 601 | − pm_runtime_get_sync(priv->ch->dev); | |
| 604 | + pm_runtime_get_sync(counter->parent); | |
| 602 | 605 | rz_mtu3_shared_reg_update_bit(priv->ch, RZ_MTU3_TMDR3, |
| 603 | 606 | RZ_MTU3_TMDR3_PHCKSEL, |
| 604 | 607 | ext_input_phase_clock_select); |
| 605 | − pm_runtime_put(priv->ch->dev); | |
| 608 | + pm_runtime_put(counter->parent); | |
| 606 | 609 | mutex_unlock(&priv->lock); |
| 607 | 610 | |
| 608 | 611 | return 0; |
| @@ -640,7 +643,7 @@ static int rz_mtu3_action_read(struct counter_device *counter, · 31 unmodified lines | ||
| 640 | 643 | if (ret) |
| 641 | 644 | return ret; |
| 642 | 645 | |
| 643 | − ret = rz_mtu3_count_function_read_helper(ch, priv, &function); | |
| 646 | + ret = rz_mtu3_count_function_read_helper(ch, counter, &function); | |
| 644 | 647 | if (ret) { |
| 645 | 648 | mutex_unlock(&priv->lock); |
| 646 | 649 | return ret; |
| @@ -1427,12 +1427,9 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, · 1426 unmodified lines | ||
| 1427 | 1427 | * If there is a problem with its frequency table, take it |
| 1428 | 1428 | * offline and drop it. |
| 1429 | 1429 | */ |
| 1430 | − if (policy->freq_table_sorted != CPUFREQ_TABLE_SORTED_ASCENDING && | |
| 1431 | − policy->freq_table_sorted != CPUFREQ_TABLE_SORTED_DESCENDING) { | |
| 1432 | − ret = cpufreq_table_validate_and_sort(policy); | |
| 1433 | − if (ret) | |
| 1434 | − goto out_offline_policy; | |
| 1435 | − } | |
| 1430 | + ret = cpufreq_table_validate_and_sort(policy); | |
| 1431 | + if (ret) | |
| 1432 | + goto out_offline_policy; | |
| 1436 | 1433 | |
| 1437 | 1434 | /* related_cpus should at least include policy->cpus. */ |
| 1438 | 1435 | cpumask_copy(policy->related_cpus, policy->cpus); |
| @@ -313,6 +313,17 @@ static void cs_start(struct cpufreq_policy *policy) · 312 unmodified lines | ||
| 313 | 313 | dbs_info->requested_freq = policy->cur; |
| 314 | 314 | } |
| 315 | 315 | |
| 316 | +static void cs_limits(struct cpufreq_policy *policy) | |
| 317 | +{ | |
| 318 | + struct cs_policy_dbs_info *dbs_info = to_dbs_info(policy->governor_data); | |
| 319 | + | |
| 320 | + /* | |
| 321 | + * The limits have changed, so may have the current frequency. Reset | |
| 322 | + * requested_freq to avoid any unintended outcomes due to the mismatch. | |
| 323 | + */ | |
| 324 | + dbs_info->requested_freq = policy->cur; | |
| 325 | +} | |
| 326 | + | |
| 316 | 327 | static struct dbs_governor cs_governor = { |
| 317 | 328 | .gov = CPUFREQ_DBS_GOVERNOR_INITIALIZER("conservative"), |
| 318 | 329 | .kobj_type = { .default_groups = cs_groups }, |
| @@ -322,6 +333,7 @@ static struct dbs_governor cs_governor = { · 3 unmodified lines | ||
| 322 | 333 | .init = cs_init, |
| 323 | 334 | .exit = cs_exit, |
| 324 | 335 | .start = cs_start, |
| 336 | + .limits = cs_limits, | |
| 325 | 337 | }; |
| 326 | 338 | |
| 327 | 339 | #define CPU_FREQ_GOV_CONSERVATIVE (cs_governor.gov) |
| @@ -468,13 +468,13 @@ int cpufreq_dbs_governor_init(struct cpufreq_policy *policy) · 467 unmodified lines | ||
| 468 | 468 | /* Failure, so roll back. */ |
| 469 | 469 | pr_err("initialization failed (dbs_data kobject init error %d)\n", ret); |
| 470 | 470 | |
| 471 | − kobject_put(&dbs_data->attr_set.kobj); | |
| 472 | − | |
| 473 | 471 | policy->governor_data = NULL; |
| 474 | 472 | |
| 475 | 473 | if (!have_governor_per_policy()) |
| 476 | 474 | gov->gdbs_data = NULL; |
| 477 | − gov->exit(dbs_data); | |
| 475 | + | |
| 476 | + kobject_put(&dbs_data->attr_set.kobj); | |
| 477 | + goto free_policy_dbs_info; | |
| 478 | 478 | |
| 479 | 479 | free_dbs_data: |
| 480 | 480 | kfree(dbs_data); |
| @@ -563,6 +563,7 @@ EXPORT_SYMBOL_GPL(cpufreq_dbs_governor_stop); · 82 unmodified lines | ||
| 563 | 563 | |
| 564 | 564 | void cpufreq_dbs_governor_limits(struct cpufreq_policy *policy) |
| 565 | 565 | { |
| 566 | + struct dbs_governor *gov = dbs_governor_of(policy); | |
| 566 | 567 | struct policy_dbs_info *policy_dbs; |
| 567 | 568 | |
| 568 | 569 | /* Protect gov->gdbs_data against cpufreq_dbs_governor_exit() */ |
| @@ -574,6 +575,8 @@ void cpufreq_dbs_governor_limits(struct cpufreq_policy *policy) · 5 unmodified lines | ||
| 574 | 575 | mutex_lock(&policy_dbs->update_mutex); |
| 575 | 576 | cpufreq_policy_apply_limits(policy); |
| 576 | 577 | gov_update_sample_delay(policy_dbs, 0); |
| 578 | + if (gov->limits) | |
| 579 | + gov->limits(policy); | |
| 577 | 580 | mutex_unlock(&policy_dbs->update_mutex); |
| 578 | 581 | |
| 579 | 582 | out: |
| @@ -138,6 +138,7 @@ struct dbs_governor { · 137 unmodified lines | ||
| 138 | 138 | int (*init)(struct dbs_data *dbs_data); |
| 139 | 139 | void (*exit)(struct dbs_data *dbs_data); |
| 140 | 140 | void (*start)(struct cpufreq_policy *policy); |
| 141 | + void (*limits)(struct cpufreq_policy *policy); | |
| 141 | 142 | }; |
| 142 | 143 | |
| 143 | 144 | static inline struct dbs_governor *dbs_governor_of(struct cpufreq_policy *policy) |
| @@ -360,6 +360,10 @@ int cpufreq_table_validate_and_sort(struct cpufreq_policy *policy) · 359 unmodified lines | ||
| 360 | 360 | if (policy_has_boost_freq(policy)) |
| 361 | 361 | policy->boost_supported = true; |
| 362 | 362 | |
| 363 | + if (policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_ASCENDING || | |
| 364 | + policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_DESCENDING) | |
| 365 | + return 0; | |
| 366 | + | |
| 363 | 367 | return set_freq_table_sorted(policy); |
| 364 | 368 | } |
| 365 | 369 | |
| @@ -359,16 +359,6 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, · 358 unmodified lines | ||
| 359 | 359 | int cpuidle_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, |
| 360 | 360 | bool *stop_tick) |
| 361 | 361 | { |
| 362 | − /* | |
| 363 | − * If there is only a single idle state (or none), there is nothing | |
| 364 | − * meaningful for the governor to choose. Skip the governor and | |
| 365 | − * always use state 0 with the tick running. | |
| 366 | − */ | |
| 367 | − if (drv->state_count <= 1) { | |
| 368 | − *stop_tick = false; | |
| 369 | − return 0; | |
| 370 | − } | |
| 371 | − | |
| 372 | 362 | return cpuidle_curr_governor->select(drv, dev, stop_tick); |
| 373 | 363 | } |
| 374 | 364 | |
| @@ -3326,9 +3326,10 @@ static int ahash_setkey(struct crypto_ahash *ahash, const u8 *key, · 3325 unmodified lines | ||
| 3326 | 3326 | if (aligned_len < keylen) |
| 3327 | 3327 | return -EOVERFLOW; |
| 3328 | 3328 | |
| 3329 | − hashed_key = kmemdup(key, aligned_len, GFP_KERNEL); | |
| 3329 | + hashed_key = kmalloc(aligned_len, GFP_KERNEL); | |
| 3330 | 3330 | if (!hashed_key) |
| 3331 | 3331 | return -ENOMEM; |
| 3332 | + memcpy(hashed_key, key, keylen); | |
| 3332 | 3333 | ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize); |
| 3333 | 3334 | if (ret) |
| 3334 | 3335 | goto bad_free_key; |
| @@ -441,9 +441,10 @@ static int ahash_setkey(struct crypto_ahash *ahash, · 440 unmodified lines | ||
| 441 | 441 | if (aligned_len < keylen) |
| 442 | 442 | return -EOVERFLOW; |
| 443 | 443 | |
| 444 | − hashed_key = kmemdup(key, keylen, GFP_KERNEL); | |
| 444 | + hashed_key = kmalloc(aligned_len, GFP_KERNEL); | |
| 445 | 445 | if (!hashed_key) |
| 446 | 446 | return -ENOMEM; |
| 447 | + memcpy(hashed_key, key, keylen); | |
| 447 | 448 | ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize); |
| 448 | 449 | if (ret) |
| 449 | 450 | goto bad_free_key; |
| @@ -2408,10 +2408,8 @@ static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) · 2407 unmodified lines | ||
| 2408 | 2408 | * in Firmware state on failure. Use snp_reclaim_pages() to |
| 2409 | 2409 | * transition either case back to Hypervisor-owned state. |
| 2410 | 2410 | */ |
| 2411 | − if (snp_reclaim_pages(__pa(data), 1, true)) { | |
| 2412 | − snp_leak_pages(__page_to_pfn(status_page), 1); | |
| 2411 | + if (snp_reclaim_pages(__pa(data), 1, true)) | |
| 2413 | 2412 | return -EFAULT; |
| 2414 | − } | |
| 2415 | 2413 | } |
| 2416 | 2414 | |
| 2417 | 2415 | if (ret) |
| @@ -332,6 +332,13 @@ static int __init padlock_init(void) · 331 unmodified lines | ||
| 332 | 332 | if (!x86_match_cpu(padlock_sha_ids) || !boot_cpu_has(X86_FEATURE_PHE_EN)) |
| 333 | 333 | return -ENODEV; |
| 334 | 334 | |
| 335 | + /* | |
| 336 | + * Skip family 0x07 and newer used by Zhaoxin processors, | |
| 337 | + * as the driver's self-tests fail on these CPUs. | |
| 338 | + */ | |
| 339 | + if (c->x86 >= 0x07) | |
| 340 | + return -ENODEV; | |
| 341 | + | |
| 335 | 342 | /* Register the newly added algorithm module if on * |
| 336 | 343 | * VIA Nano processor, or else just do as before */ |
| 337 | 344 | if (c->x86_model < 0x0f) { |
| @@ -529,7 +529,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 528 unmodified lines | ||
| 529 | 529 | .cra_name = "cbc(aes)", |
| 530 | 530 | .cra_driver_name = "cbc-aes-tegra", |
| 531 | 531 | .cra_priority = 500, |
| 532 | − .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | CRYPTO_ALG_ASYNC, | |
| 532 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 533 | 533 | .cra_blocksize = AES_BLOCK_SIZE, |
| 534 | 534 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 535 | 535 | .cra_alignmask = 0xf, |
| @@ -550,7 +550,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 14 unmodified lines | ||
| 550 | 550 | .cra_name = "ecb(aes)", |
| 551 | 551 | .cra_driver_name = "ecb-aes-tegra", |
| 552 | 552 | .cra_priority = 500, |
| 553 | − .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | CRYPTO_ALG_ASYNC, | |
| 553 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 554 | 554 | .cra_blocksize = AES_BLOCK_SIZE, |
| 555 | 555 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 556 | 556 | .cra_alignmask = 0xf, |
| @@ -572,7 +572,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 15 unmodified lines | ||
| 572 | 572 | .cra_name = "ctr(aes)", |
| 573 | 573 | .cra_driver_name = "ctr-aes-tegra", |
| 574 | 574 | .cra_priority = 500, |
| 575 | − .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | CRYPTO_ALG_ASYNC, | |
| 575 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 576 | 576 | .cra_blocksize = 1, |
| 577 | 577 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 578 | 578 | .cra_alignmask = 0xf, |
| @@ -594,6 +594,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 15 unmodified lines | ||
| 594 | 594 | .cra_name = "xts(aes)", |
| 595 | 595 | .cra_driver_name = "xts-aes-tegra", |
| 596 | 596 | .cra_priority = 500, |
| 597 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 597 | 598 | .cra_blocksize = AES_BLOCK_SIZE, |
| 598 | 599 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 599 | 600 | .cra_alignmask = (__alignof__(u64) - 1), |
| @@ -1922,6 +1923,7 @@ static struct tegra_se_alg tegra_aead_algs[] = { · 1322 unmodified lines | ||
| 1922 | 1923 | .cra_name = "gcm(aes)", |
| 1923 | 1924 | .cra_driver_name = "gcm-aes-tegra", |
| 1924 | 1925 | .cra_priority = 500, |
| 1926 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 1925 | 1927 | .cra_blocksize = 1, |
| 1926 | 1928 | .cra_ctxsize = sizeof(struct tegra_aead_ctx), |
| 1927 | 1929 | .cra_alignmask = 0xf, |
| @@ -1944,6 +1946,7 @@ static struct tegra_se_alg tegra_aead_algs[] = { · 16 unmodified lines | ||
| 1944 | 1946 | .cra_name = "ccm(aes)", |
| 1945 | 1947 | .cra_driver_name = "ccm-aes-tegra", |
| 1946 | 1948 | .cra_priority = 500, |
| 1949 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 1947 | 1950 | .cra_blocksize = 1, |
| 1948 | 1951 | .cra_ctxsize = sizeof(struct tegra_aead_ctx), |
| 1949 | 1952 | .cra_alignmask = 0xf, |
| @@ -1971,7 +1974,7 @@ static struct tegra_se_alg tegra_cmac_algs[] = { · 21 unmodified lines | ||
| 1971 | 1974 | .cra_name = "cmac(aes)", |
| 1972 | 1975 | .cra_driver_name = "tegra-se-cmac", |
| 1973 | 1976 | .cra_priority = 300, |
| 1974 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 1977 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 1975 | 1978 | .cra_blocksize = AES_BLOCK_SIZE, |
| 1976 | 1979 | .cra_ctxsize = sizeof(struct tegra_cmac_ctx), |
| 1977 | 1980 | .cra_alignmask = 0, |
| @@ -761,7 +761,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 760 unmodified lines | ||
| 761 | 761 | .cra_name = "sha1", |
| 762 | 762 | .cra_driver_name = "tegra-se-sha1", |
| 763 | 763 | .cra_priority = 300, |
| 764 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 764 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 765 | 765 | .cra_blocksize = SHA1_BLOCK_SIZE, |
| 766 | 766 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 767 | 767 | .cra_alignmask = 0, |
| @@ -786,7 +786,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 786 | 786 | .cra_name = "sha224", |
| 787 | 787 | .cra_driver_name = "tegra-se-sha224", |
| 788 | 788 | .cra_priority = 300, |
| 789 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 789 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 790 | 790 | .cra_blocksize = SHA224_BLOCK_SIZE, |
| 791 | 791 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 792 | 792 | .cra_alignmask = 0, |
| @@ -811,7 +811,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 811 | 811 | .cra_name = "sha256", |
| 812 | 812 | .cra_driver_name = "tegra-se-sha256", |
| 813 | 813 | .cra_priority = 300, |
| 814 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 814 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 815 | 815 | .cra_blocksize = SHA256_BLOCK_SIZE, |
| 816 | 816 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 817 | 817 | .cra_alignmask = 0, |
| @@ -836,7 +836,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 836 | 836 | .cra_name = "sha384", |
| 837 | 837 | .cra_driver_name = "tegra-se-sha384", |
| 838 | 838 | .cra_priority = 300, |
| 839 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 839 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 840 | 840 | .cra_blocksize = SHA384_BLOCK_SIZE, |
| 841 | 841 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 842 | 842 | .cra_alignmask = 0, |
| @@ -861,7 +861,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 861 | 861 | .cra_name = "sha512", |
| 862 | 862 | .cra_driver_name = "tegra-se-sha512", |
| 863 | 863 | .cra_priority = 300, |
| 864 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 864 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 865 | 865 | .cra_blocksize = SHA512_BLOCK_SIZE, |
| 866 | 866 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 867 | 867 | .cra_alignmask = 0, |
| @@ -886,7 +886,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 886 | 886 | .cra_name = "sha3-224", |
| 887 | 887 | .cra_driver_name = "tegra-se-sha3-224", |
| 888 | 888 | .cra_priority = 300, |
| 889 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 889 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 890 | 890 | .cra_blocksize = SHA3_224_BLOCK_SIZE, |
| 891 | 891 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 892 | 892 | .cra_alignmask = 0, |
| @@ -911,7 +911,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 911 | 911 | .cra_name = "sha3-256", |
| 912 | 912 | .cra_driver_name = "tegra-se-sha3-256", |
| 913 | 913 | .cra_priority = 300, |
| 914 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 914 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 915 | 915 | .cra_blocksize = SHA3_256_BLOCK_SIZE, |
| 916 | 916 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 917 | 917 | .cra_alignmask = 0, |
| @@ -936,7 +936,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 936 | 936 | .cra_name = "sha3-384", |
| 937 | 937 | .cra_driver_name = "tegra-se-sha3-384", |
| 938 | 938 | .cra_priority = 300, |
| 939 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 939 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 940 | 940 | .cra_blocksize = SHA3_384_BLOCK_SIZE, |
| 941 | 941 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 942 | 942 | .cra_alignmask = 0, |
| @@ -961,7 +961,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | ||
| 961 | 961 | .cra_name = "sha3-512", |
| 962 | 962 | .cra_driver_name = "tegra-se-sha3-512", |
| 963 | 963 | .cra_priority = 300, |
| 964 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH, | |
| 964 | + .cra_flags = CRYPTO_ALG_ASYNC, | |
| 965 | 965 | .cra_blocksize = SHA3_512_BLOCK_SIZE, |
| 966 | 966 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 967 | 967 | .cra_alignmask = 0, |
| @@ -988,7 +988,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | ||
| 988 | 988 | .cra_name = "hmac(sha224)", |
| 989 | 989 | .cra_driver_name = "tegra-se-hmac-sha224", |
| 990 | 990 | .cra_priority = 300, |
| 991 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | |
| 991 | + .cra_flags = CRYPTO_ALG_ASYNC | | |
| 992 | + CRYPTO_ALG_NEED_FALLBACK, | |
| 992 | 993 | .cra_blocksize = SHA224_BLOCK_SIZE, |
| 993 | 994 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 994 | 995 | .cra_alignmask = 0, |
| @@ -1015,7 +1016,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | ||
| 1015 | 1016 | .cra_name = "hmac(sha256)", |
| 1016 | 1017 | .cra_driver_name = "tegra-se-hmac-sha256", |
| 1017 | 1018 | .cra_priority = 300, |
| 1018 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | |
| 1019 | + .cra_flags = CRYPTO_ALG_ASYNC | | |
| 1020 | + CRYPTO_ALG_NEED_FALLBACK, | |
| 1019 | 1021 | .cra_blocksize = SHA256_BLOCK_SIZE, |
| 1020 | 1022 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 1021 | 1023 | .cra_alignmask = 0, |
| @@ -1042,7 +1044,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | ||
| 1042 | 1044 | .cra_name = "hmac(sha384)", |
| 1043 | 1045 | .cra_driver_name = "tegra-se-hmac-sha384", |
| 1044 | 1046 | .cra_priority = 300, |
| 1045 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | |
| 1047 | + .cra_flags = CRYPTO_ALG_ASYNC | | |
| 1048 | + CRYPTO_ALG_NEED_FALLBACK, | |
| 1046 | 1049 | .cra_blocksize = SHA384_BLOCK_SIZE, |
| 1047 | 1050 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 1048 | 1051 | .cra_alignmask = 0, |
| @@ -1069,7 +1072,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | ||
| 1069 | 1072 | .cra_name = "hmac(sha512)", |
| 1070 | 1073 | .cra_driver_name = "tegra-se-hmac-sha512", |
| 1071 | 1074 | .cra_priority = 300, |
| 1072 | − .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | |
| 1075 | + .cra_flags = CRYPTO_ALG_ASYNC | | |
| 1076 | + CRYPTO_ALG_NEED_FALLBACK, | |
| 1073 | 1077 | .cra_blocksize = SHA512_BLOCK_SIZE, |
| 1074 | 1078 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 1075 | 1079 | .cra_alignmask = 0, |
| @@ -59,6 +59,7 @@ config CXL_ACPI · 58 unmodified lines | ||
| 59 | 59 | tristate "CXL ACPI: Platform Support" |
| 60 | 60 | depends on ACPI |
| 61 | 61 | depends on ACPI_NUMA |
| 62 | + depends on CXL_PMEM || !CXL_PMEM | |
| 62 | 63 | default CXL_BUS |
| 63 | 64 | select ACPI_TABLE_LIB |
| 64 | 65 | select ACPI_HMAT |
| @@ -94,7 +94,6 @@ static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info) · 93 unmodified lines | ||
| 94 | 94 | struct cxl_hdm *cxlhdm; |
| 95 | 95 | void __iomem *hdm; |
| 96 | 96 | u32 ctrl; |
| 97 | − int i; | |
| 98 | 97 | |
| 99 | 98 | if (!info) |
| 100 | 99 | return false; |
| @@ -113,22 +112,16 @@ static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info) · 12 unmodified lines | ||
| 113 | 112 | return false; |
| 114 | 113 | |
| 115 | 114 | /* |
| 116 | − * If any decoders are committed already, there should not be any | |
| 117 | − * emulated DVSEC decoders. | |
| 115 | + * If HDM decoders are globally enabled, do not fall back to DVSEC | |
| 116 | + * range emulation. Zeroed decoder registers after region teardown | |
| 117 | + * do not imply absence of HDM capability. | |
| 118 | + * | |
| 119 | + * Falling back to DVSEC here would treat the decoder as AUTO and | |
| 120 | + * may incorrectly latch default interleave settings. | |
| 118 | 121 | */ |
| 119 | − for (i = 0; i < cxlhdm->decoder_count; i++) { | |
| 120 | − ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); | |
| 121 | − dev_dbg(&info->port->dev, | |
| 122 | − "decoder%d.%d: committed: %ld base: %#x_%.8x size: %#x_%.8x\n", | |
| 123 | − info->port->id, i, | |
| 124 | − FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl), | |
| 125 | − readl(hdm + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i)), | |
| 126 | − readl(hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(i)), | |
| 127 | − readl(hdm + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i)), | |
| 128 | − readl(hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i))); | |
| 129 | − if (FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl)) | |
| 130 | − return false; | |
| 131 | − } | |
| 122 | + ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); | |
| 123 | + if (ctrl & CXL_HDM_DECODER_ENABLE) | |
| 124 | + return false; | |
| 132 | 125 | |
| 133 | 126 | return true; |
| 134 | 127 | } |
| @@ -1301,7 +1301,7 @@ int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd) · 1300 unmodified lines | ||
| 1301 | 1301 | * Require an endpoint to be safe otherwise the driver can not |
| 1302 | 1302 | * be sure that the device is unmapped. |
| 1303 | 1303 | */ |
| 1304 | − if (endpoint && cxl_num_decoders_committed(endpoint) == 0) | |
| 1304 | + if (cxlmd->dev.driver && cxl_num_decoders_committed(endpoint) == 0) | |
| 1305 | 1305 | return __cxl_mem_sanitize(mds, cmd); |
| 1306 | 1306 | |
| 1307 | 1307 | return -EBUSY; |
| @@ -552,10 +552,13 @@ static void cxl_port_release(struct device *dev) · 551 unmodified lines | ||
| 552 | 552 | xa_destroy(&port->dports); |
| 553 | 553 | xa_destroy(&port->regions); |
| 554 | 554 | ida_free(&cxl_port_ida, port->id); |
| 555 | − if (is_cxl_root(port)) | |
| 555 | + | |
| 556 | + if (is_cxl_root(port)) { | |
| 556 | 557 | kfree(to_cxl_root(port)); |
| 557 | − else | |
| 558 | + } else { | |
| 559 | + put_device(dev->parent); | |
| 558 | 560 | kfree(port); |
| 561 | + } | |
| 559 | 562 | } |
| 560 | 563 | |
| 561 | 564 | static ssize_t decoders_committed_show(struct device *dev, |
| @@ -707,6 +710,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport_dev, · 145 unmodified lines | ||
| 707 | 710 | struct cxl_port *iter; |
| 708 | 711 | |
| 709 | 712 | dev->parent = &parent_port->dev; |
| 713 | + get_device(dev->parent); | |
| 710 | 714 | port->depth = parent_port->depth + 1; |
| 711 | 715 | port->parent_dport = parent_dport; |
| 712 | 716 | |
| @@ -3854,8 +3854,10 @@ static int __construct_region(struct cxl_region *cxlr, · 3853 unmodified lines | ||
| 3854 | 3854 | } |
| 3855 | 3855 | |
| 3856 | 3856 | rc = sysfs_update_group(&cxlr->dev.kobj, &cxl_region_group); |
| 3857 | − if (rc) | |
| 3857 | + if (rc) { | |
| 3858 | + kfree(res); | |
| 3858 | 3859 | return rc; |
| 3860 | + } | |
| 3859 | 3861 | |
| 3860 | 3862 | rc = insert_resource(cxlrd->res, res); |
| 3861 | 3863 | if (rc) { |
| @@ -554,7 +554,7 @@ static __exit void cxl_pmem_exit(void) · 553 unmodified lines | ||
| 554 | 554 | |
| 555 | 555 | MODULE_DESCRIPTION("CXL PMEM: Persistent Memory Support"); |
| 556 | 556 | MODULE_LICENSE("GPL v2"); |
| 557 | −module_init(cxl_pmem_init); | |
| 557 | +subsys_initcall(cxl_pmem_init); | |
| 558 | 558 | module_exit(cxl_pmem_exit); |
| 559 | 559 | MODULE_IMPORT_NS("CXL"); |
| 560 | 560 | MODULE_ALIAS_CXL(CXL_DEVICE_NVDIMM_BRIDGE); |
| @@ -844,6 +844,7 @@ static int dw_edma_irq_request(struct dw_edma *dw, · 843 unmodified lines | ||
| 844 | 844 | { |
| 845 | 845 | struct dw_edma_chip *chip = dw->chip; |
| 846 | 846 | struct device *dev = dw->chip->dev; |
| 847 | + struct msi_desc *msi_desc; | |
| 847 | 848 | u32 wr_mask = 1; |
| 848 | 849 | u32 rd_mask = 1; |
| 849 | 850 | int i, err = 0; |
| @@ -895,9 +896,12 @@ static int dw_edma_irq_request(struct dw_edma *dw, · 45 unmodified lines | ||
| 895 | 896 | &dw->irq[i]); |
| 896 | 897 | if (err) |
| 897 | 898 | goto err_irq_free; |
| 898 | − | |
| 899 | − if (irq_get_msi_desc(irq)) | |
| 899 | + msi_desc = irq_get_msi_desc(irq); | |
| 900 | + if (msi_desc) { | |
| 900 | 901 | get_cached_msi_msg(irq, &dw->irq[i].msi); |
| 902 | + if (!msi_desc->pci.msi_attrib.is_msix) | |
| 903 | + dw->irq[i].msi.data = dw->irq[0].msi.data + i; | |
| 904 | + } | |
| 901 | 905 | } |
| 902 | 906 | |
| 903 | 907 | dw->nr_irqs = i; |
| @@ -252,10 +252,10 @@ static void dw_hdma_v0_core_start(struct dw_edma_chunk *chunk, bool first) · 251 unmodified lines | ||
| 252 | 252 | lower_32_bits(chunk->ll_region.paddr)); |
| 253 | 253 | SET_CH_32(dw, chan->dir, chan->id, llp.msb, |
| 254 | 254 | upper_32_bits(chunk->ll_region.paddr)); |
| 255 | + /* Set consumer cycle */ | |
| 256 | + SET_CH_32(dw, chan->dir, chan->id, cycle_sync, | |
| 257 | + HDMA_V0_CONSUMER_CYCLE_STAT | HDMA_V0_CONSUMER_CYCLE_BIT); | |
| 255 | 258 | } |
| 256 | − /* Set consumer cycle */ | |
| 257 | − SET_CH_32(dw, chan->dir, chan->id, cycle_sync, | |
| 258 | − HDMA_V0_CONSUMER_CYCLE_STAT | HDMA_V0_CONSUMER_CYCLE_BIT); | |
| 259 | 259 | |
| 260 | 260 | dw_hdma_v0_sync_ll_data(chunk); |
| 261 | 261 | |
| @@ -317,10 +317,8 @@ static struct dma_chan *fsl_edma3_xlate(struct of_phandle_args *dma_spec, · 316 unmodified lines | ||
| 317 | 317 | return NULL; |
| 318 | 318 | i = fsl_chan - fsl_edma->chans; |
| 319 | 319 | |
| 320 | − fsl_chan->priority = dma_spec->args[1]; | |
| 321 | − fsl_chan->is_rxchan = dma_spec->args[2] & FSL_EDMA_RX; | |
| 322 | − fsl_chan->is_remote = dma_spec->args[2] & FSL_EDMA_REMOTE; | |
| 323 | − fsl_chan->is_multi_fifo = dma_spec->args[2] & FSL_EDMA_MULTI_FIFO; | |
| 320 | + if (!b_chmux && i != dma_spec->args[0]) | |
| 321 | + continue; | |
| 324 | 322 | |
| 325 | 323 | if ((dma_spec->args[2] & FSL_EDMA_EVEN_CH) && (i & 0x1)) |
| 326 | 324 | continue; |
| @@ -328,17 +326,15 @@ static struct dma_chan *fsl_edma3_xlate(struct of_phandle_args *dma_spec, · 1 unmodified line | ||
| 328 | 326 | if ((dma_spec->args[2] & FSL_EDMA_ODD_CH) && !(i & 0x1)) |
| 329 | 327 | continue; |
| 330 | 328 | |
| 331 | − if (!b_chmux && i == dma_spec->args[0]) { | |
| 332 | − chan = dma_get_slave_channel(chan); | |
| 333 | − chan->device->privatecnt++; | |
| 334 | − return chan; | |
| 335 | − } else if (b_chmux && !fsl_chan->srcid) { | |
| 336 | − /* if controller support channel mux, choose a free channel */ | |
| 337 | − chan = dma_get_slave_channel(chan); | |
| 338 | − chan->device->privatecnt++; | |
| 339 | − fsl_chan->srcid = dma_spec->args[0]; | |
| 340 | − return chan; | |
| 341 | − } | |
| 329 | + fsl_chan->srcid = dma_spec->args[0]; | |
| 330 | + fsl_chan->priority = dma_spec->args[1]; | |
| 331 | + fsl_chan->is_rxchan = dma_spec->args[2] & FSL_EDMA_RX; | |
| 332 | + fsl_chan->is_remote = dma_spec->args[2] & FSL_EDMA_REMOTE; | |
| 333 | + fsl_chan->is_multi_fifo = dma_spec->args[2] & FSL_EDMA_MULTI_FIFO; | |
| 334 | + | |
| 335 | + chan = dma_get_slave_channel(chan); | |
| 336 | + chan->device->privatecnt++; | |
| 337 | + return chan; | |
| 342 | 338 | } |
| 343 | 339 | return NULL; |
| 344 | 340 | } |
| @@ -158,11 +158,7 @@ static const struct device_type idxd_cdev_file_type = { · 157 unmodified lines | ||
| 158 | 158 | static void idxd_cdev_dev_release(struct device *dev) |
| 159 | 159 | { |
| 160 | 160 | struct idxd_cdev *idxd_cdev = dev_to_cdev(dev); |
| 161 | − struct idxd_cdev_context *cdev_ctx; | |
| 162 | − struct idxd_wq *wq = idxd_cdev->wq; | |
| 163 | 161 | |
| 164 | − cdev_ctx = &ictx[wq->idxd->data->type]; | |
| 165 | − ida_free(&cdev_ctx->minor_ida, idxd_cdev->minor); | |
| 166 | 162 | kfree(idxd_cdev); |
| 167 | 163 | } |
| 168 | 164 | |
| @@ -582,11 +578,15 @@ int idxd_wq_add_cdev(struct idxd_wq *wq) · 413 unmodified lines | ||
| 582 | 578 | |
| 583 | 579 | void idxd_wq_del_cdev(struct idxd_wq *wq) |
| 584 | 580 | { |
| 581 | + struct idxd_cdev_context *cdev_ctx; | |
| 585 | 582 | struct idxd_cdev *idxd_cdev; |
| 586 | 583 | |
| 587 | 584 | idxd_cdev = wq->idxd_cdev; |
| 588 | 585 | wq->idxd_cdev = NULL; |
| 589 | 586 | cdev_device_del(&idxd_cdev->cdev, cdev_dev(idxd_cdev)); |
| 587 | + | |
| 588 | + cdev_ctx = &ictx[wq->idxd->data->type]; | |
| 589 | + ida_free(&cdev_ctx->minor_ida, idxd_cdev->minor); | |
| 590 | 590 | put_device(cdev_dev(idxd_cdev)); |
| 591 | 591 | } |
| 592 | 592 | |
| @@ -175,6 +175,7 @@ void idxd_wq_free_resources(struct idxd_wq *wq) · 174 unmodified lines | ||
| 175 | 175 | free_descs(wq); |
| 176 | 176 | dma_free_coherent(dev, wq->compls_size, wq->compls, wq->compls_addr); |
| 177 | 177 | sbitmap_queue_free(&wq->sbq); |
| 178 | + wq->type = IDXD_WQT_NONE; | |
| 178 | 179 | } |
| 179 | 180 | EXPORT_SYMBOL_NS_GPL(idxd_wq_free_resources, "IDXD"); |
| 180 | 181 | |
| @@ -382,7 +383,6 @@ static void idxd_wq_disable_cleanup(struct idxd_wq *wq) · 201 unmodified lines | ||
| 382 | 383 | lockdep_assert_held(&wq->wq_lock); |
| 383 | 384 | wq->state = IDXD_WQ_DISABLED; |
| 384 | 385 | memset(wq->wqcfg, 0, idxd->wqcfg_size); |
| 385 | − wq->type = IDXD_WQT_NONE; | |
| 386 | 386 | wq->threshold = 0; |
| 387 | 387 | wq->priority = 0; |
| 388 | 388 | wq->enqcmds_retries = IDXD_ENQCMDS_RETRIES; |
| @@ -831,8 +831,7 @@ static void idxd_device_evl_free(struct idxd_device *idxd) · 442 unmodified lines | ||
| 831 | 831 | struct device *dev = &idxd->pdev->dev; |
| 832 | 832 | struct idxd_evl *evl = idxd->evl; |
| 833 | 833 | |
| 834 | − gencfg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET); | |
| 835 | − if (!gencfg.evl_en) | |
| 834 | + if (!evl) | |
| 836 | 835 | return; |
| 837 | 836 | |
| 838 | 837 | mutex_lock(&evl->lock); |
| @@ -1125,7 +1124,11 @@ int idxd_device_config(struct idxd_device *idxd) · 286 unmodified lines | ||
| 1125 | 1124 | { |
| 1126 | 1125 | int rc; |
| 1127 | 1126 | |
| 1128 | − lockdep_assert_held(&idxd->dev_lock); | |
| 1127 | + guard(spinlock)(&idxd->dev_lock); | |
| 1128 | + | |
| 1129 | + if (!test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) | |
| 1130 | + return 0; | |
| 1131 | + | |
| 1129 | 1132 | rc = idxd_wqs_setup(idxd); |
| 1130 | 1133 | if (rc < 0) |
| 1131 | 1134 | return rc; |
| @@ -1332,6 +1335,11 @@ void idxd_wq_free_irq(struct idxd_wq *wq) · 200 unmodified lines | ||
| 1332 | 1335 | |
| 1333 | 1336 | free_irq(ie->vector, ie); |
| 1334 | 1337 | idxd_flush_pending_descs(ie); |
| 1338 | + | |
| 1339 | + /* The interrupt might have been already released by FLR */ | |
| 1340 | + if (ie->int_handle == INVALID_INT_HANDLE) | |
| 1341 | + return; | |
| 1342 | + | |
| 1335 | 1343 | if (idxd->request_int_handles) |
| 1336 | 1344 | idxd_device_release_int_handle(idxd, ie->int_handle, IDXD_IRQ_MSIX); |
| 1337 | 1345 | idxd_device_clear_perm_entry(idxd, ie); |
| @@ -1340,6 +1348,23 @@ void idxd_wq_free_irq(struct idxd_wq *wq) · 2 unmodified lines | ||
| 1340 | 1348 | ie->pasid = IOMMU_PASID_INVALID; |
| 1341 | 1349 | } |
| 1342 | 1350 | |
| 1351 | +void idxd_wq_flush_descs(struct idxd_wq *wq) | |
| 1352 | +{ | |
| 1353 | + struct idxd_irq_entry *ie = &wq->ie; | |
| 1354 | + struct idxd_device *idxd = wq->idxd; | |
| 1355 | + | |
| 1356 | + guard(mutex)(&wq->wq_lock); | |
| 1357 | + | |
| 1358 | + if (wq->state != IDXD_WQ_ENABLED || wq->type != IDXD_WQT_KERNEL) | |
| 1359 | + return; | |
| 1360 | + | |
| 1361 | + idxd_flush_pending_descs(ie); | |
| 1362 | + if (idxd->request_int_handles) | |
| 1363 | + idxd_device_release_int_handle(idxd, ie->int_handle, IDXD_IRQ_MSIX); | |
| 1364 | + idxd_device_clear_perm_entry(idxd, ie); | |
| 1365 | + ie->int_handle = INVALID_INT_HANDLE; | |
| 1366 | +} | |
| 1367 | + | |
| 1343 | 1368 | int idxd_wq_request_irq(struct idxd_wq *wq) |
| 1344 | 1369 | { |
| 1345 | 1370 | struct idxd_device *idxd = wq->idxd; |
| @@ -1454,11 +1479,7 @@ int idxd_drv_enable_wq(struct idxd_wq *wq) · 108 unmodified lines | ||
| 1454 | 1479 | } |
| 1455 | 1480 | } |
| 1456 | 1481 | |
| 1457 | − rc = 0; | |
| 1458 | − spin_lock(&idxd->dev_lock); | |
| 1459 | − if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) | |
| 1460 | − rc = idxd_device_config(idxd); | |
| 1461 | − spin_unlock(&idxd->dev_lock); | |
| 1482 | + rc = idxd_device_config(idxd); | |
| 1462 | 1483 | if (rc < 0) { |
| 1463 | 1484 | dev_dbg(dev, "Writing wq %d config failed: %d\n", wq->id, rc); |
| 1464 | 1485 | goto err; |
| @@ -1533,7 +1554,6 @@ void idxd_drv_disable_wq(struct idxd_wq *wq) · 68 unmodified lines | ||
| 1533 | 1554 | idxd_wq_reset(wq); |
| 1534 | 1555 | idxd_wq_free_resources(wq); |
| 1535 | 1556 | percpu_ref_exit(&wq->wq_active); |
| 1536 | − wq->type = IDXD_WQT_NONE; | |
| 1537 | 1557 | wq->client_count = 0; |
| 1538 | 1558 | } |
| 1539 | 1559 | EXPORT_SYMBOL_NS_GPL(idxd_drv_disable_wq, "IDXD"); |
| @@ -1554,10 +1574,7 @@ int idxd_device_drv_probe(struct idxd_dev *idxd_dev) · 14 unmodified lines | ||
| 1554 | 1574 | } |
| 1555 | 1575 | |
| 1556 | 1576 | /* Device configuration */ |
| 1557 | − spin_lock(&idxd->dev_lock); | |
| 1558 | − if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) | |
| 1559 | − rc = idxd_device_config(idxd); | |
| 1560 | − spin_unlock(&idxd->dev_lock); | |
| 1577 | + rc = idxd_device_config(idxd); | |
| 1561 | 1578 | if (rc < 0) |
| 1562 | 1579 | return -ENXIO; |
| 1563 | 1580 | |
| @@ -194,6 +194,22 @@ static void idxd_dma_release(struct dma_device *device) · 193 unmodified lines | ||
| 194 | 194 | kfree(idxd_dma); |
| 195 | 195 | } |
| 196 | 196 | |
| 197 | +static int idxd_dma_terminate_all(struct dma_chan *c) | |
| 198 | +{ | |
| 199 | + struct idxd_wq *wq = to_idxd_wq(c); | |
| 200 | + | |
| 201 | + idxd_wq_flush_descs(wq); | |
| 202 | + | |
| 203 | + return 0; | |
| 204 | +} | |
| 205 | + | |
| 206 | +static void idxd_dma_synchronize(struct dma_chan *c) | |
| 207 | +{ | |
| 208 | + struct idxd_wq *wq = to_idxd_wq(c); | |
| 209 | + | |
| 210 | + idxd_wq_drain(wq); | |
| 211 | +} | |
| 212 | + | |
| 197 | 213 | int idxd_register_dma_device(struct idxd_device *idxd) |
| 198 | 214 | { |
| 199 | 215 | struct idxd_dma_dev *idxd_dma; |
| @@ -224,6 +240,8 @@ int idxd_register_dma_device(struct idxd_device *idxd) · 24 unmodified lines | ||
| 224 | 240 | dma->device_issue_pending = idxd_dma_issue_pending; |
| 225 | 241 | dma->device_alloc_chan_resources = idxd_dma_alloc_chan_resources; |
| 226 | 242 | dma->device_free_chan_resources = idxd_dma_free_chan_resources; |
| 243 | + dma->device_terminate_all = idxd_dma_terminate_all; | |
| 244 | + dma->device_synchronize = idxd_dma_synchronize; | |
| 227 | 245 | |
| 228 | 246 | rc = dma_async_device_register(dma); |
| 229 | 247 | if (rc < 0) { |
| @@ -803,6 +803,7 @@ void idxd_wq_quiesce(struct idxd_wq *wq); · 802 unmodified lines | ||
| 803 | 803 | int idxd_wq_init_percpu_ref(struct idxd_wq *wq); |
| 804 | 804 | void idxd_wq_free_irq(struct idxd_wq *wq); |
| 805 | 805 | int idxd_wq_request_irq(struct idxd_wq *wq); |
| 806 | +void idxd_wq_flush_descs(struct idxd_wq *wq); | |
| 806 | 807 | |
| 807 | 808 | /* submission */ |
| 808 | 809 | int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc); |
| @@ -973,7 +973,8 @@ static void idxd_device_config_restore(struct idxd_device *idxd, · 972 unmodified lines | ||
| 973 | 973 | |
| 974 | 974 | idxd->rdbuf_limit = idxd_saved->saved_idxd.rdbuf_limit; |
| 975 | 975 | |
| 976 | − idxd->evl->size = saved_evl->size; | |
| 976 | + if (idxd->evl) | |
| 977 | + idxd->evl->size = saved_evl->size; | |
| 977 | 978 | |
| 978 | 979 | for (i = 0; i < idxd->max_groups; i++) { |
| 979 | 980 | struct idxd_group *saved_group, *group; |
| @@ -1104,12 +1105,10 @@ static void idxd_reset_done(struct pci_dev *pdev) · 124 unmodified lines | ||
| 1104 | 1105 | idxd_device_config_restore(idxd, idxd->idxd_saved); |
| 1105 | 1106 | |
| 1106 | 1107 | /* Re-configure IDXD device if allowed. */ |
| 1107 | − if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) { | |
| 1108 | − rc = idxd_device_config(idxd); | |
| 1109 | − if (rc < 0) { | |
| 1110 | − dev_err(dev, "HALT: %s config fails\n", idxd_name); | |
| 1111 | − goto out; | |
| 1112 | − } | |
| 1108 | + rc = idxd_device_config(idxd); | |
| 1109 | + if (rc < 0) { | |
| 1110 | + dev_err(dev, "HALT: %s config fails\n", idxd_name); | |
| 1111 | + goto out; | |
| 1113 | 1112 | } |
| 1114 | 1113 | |
| 1115 | 1114 | /* Bind IDXD device to driver. */ |
| @@ -1147,6 +1146,7 @@ static void idxd_reset_done(struct pci_dev *pdev) · 31 unmodified lines | ||
| 1147 | 1146 | } |
| 1148 | 1147 | out: |
| 1149 | 1148 | kfree(idxd->idxd_saved); |
| 1149 | + idxd->idxd_saved = NULL; | |
| 1150 | 1150 | } |
| 1151 | 1151 | |
| 1152 | 1152 | static const struct pci_error_handlers idxd_error_handler = { |
| @@ -397,6 +397,17 @@ static void idxd_device_flr(struct work_struct *work) · 396 unmodified lines | ||
| 397 | 397 | dev_err(&idxd->pdev->dev, "FLR failed\n"); |
| 398 | 398 | } |
| 399 | 399 | |
| 400 | +static void idxd_wqs_flush_descs(struct idxd_device *idxd) | |
| 401 | +{ | |
| 402 | + int i; | |
| 403 | + | |
| 404 | + for (i = 0; i < idxd->max_wqs; i++) { | |
| 405 | + struct idxd_wq *wq = idxd->wqs[i]; | |
| 406 | + | |
| 407 | + idxd_wq_flush_descs(wq); | |
| 408 | + } | |
| 409 | +} | |
| 410 | + | |
| 400 | 411 | static irqreturn_t idxd_halt(struct idxd_device *idxd) |
| 401 | 412 | { |
| 402 | 413 | union gensts_reg gensts; |
| @@ -415,6 +426,11 @@ static irqreturn_t idxd_halt(struct idxd_device *idxd) · 12 unmodified lines | ||
| 415 | 426 | } else if (gensts.reset_type == IDXD_DEVICE_RESET_FLR) { |
| 416 | 427 | idxd->state = IDXD_DEV_HALTED; |
| 417 | 428 | idxd_mask_error_interrupts(idxd); |
| 429 | + /* Flush all pending descriptors, and disable | |
| 430 | + * interrupts, they will be re-enabled when FLR | |
| 431 | + * concludes. | |
| 432 | + */ | |
| 433 | + idxd_wqs_flush_descs(idxd); | |
| 418 | 434 | dev_dbg(&idxd->pdev->dev, |
| 419 | 435 | "idxd halted, doing FLR. After FLR, configs are restored\n"); |
| 420 | 436 | INIT_WORK(&idxd->work, idxd_device_flr); |
| @@ -138,7 +138,7 @@ static void llist_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie, · 137 unmodified lines | ||
| 138 | 138 | */ |
| 139 | 139 | list_for_each_entry_safe(d, t, &flist, list) { |
| 140 | 140 | list_del_init(&d->list); |
| 141 | − idxd_dma_complete_txd(found, IDXD_COMPLETE_ABORT, true, | |
| 141 | + idxd_dma_complete_txd(d, IDXD_COMPLETE_ABORT, true, | |
| 142 | 142 | NULL, NULL); |
| 143 | 143 | } |
| 144 | 144 | } |
| @@ -1836,6 +1836,7 @@ static void idxd_conf_device_release(struct device *dev) · 1835 unmodified lines | ||
| 1836 | 1836 | { |
| 1837 | 1837 | struct idxd_device *idxd = confdev_to_idxd(dev); |
| 1838 | 1838 | |
| 1839 | + destroy_workqueue(idxd->wq); | |
| 1839 | 1840 | kfree(idxd->groups); |
| 1840 | 1841 | bitmap_free(idxd->wq_enable_map); |
| 1841 | 1842 | kfree(idxd->wqs); |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | 12 | #include <linux/bitfield.h> |
| 13 | +#include <linux/cleanup.h> | |
| 13 | 14 | #include <linux/dma-mapping.h> |
| 14 | 15 | #include <linux/dmaengine.h> |
| 15 | 16 | #include <linux/interrupt.h> |
| @@ -296,13 +297,10 @@ static void rz_dmac_disable_hw(struct rz_dmac_chan *channel) · 280 unmodified lines | ||
| 296 | 297 | { |
| 297 | 298 | struct dma_chan *chan = &channel->vc.chan; |
| 298 | 299 | struct rz_dmac *dmac = to_rz_dmac(chan->device); |
| 299 | − unsigned long flags; | |
| 300 | 300 | |
| 301 | 301 | dev_dbg(dmac->dev, "%s channel %d\n", __func__, channel->index); |
| 302 | 302 | |
| 303 | − local_irq_save(flags); | |
| 304 | 303 | rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); |
| 305 | − local_irq_restore(flags); | |
| 306 | 304 | } |
| 307 | 305 | |
| 308 | 306 | static void rz_dmac_set_dmars_register(struct rz_dmac *dmac, int nr, u32 dmars) |
| @@ -447,6 +445,7 @@ static int rz_dmac_alloc_chan_resources(struct dma_chan *chan) · 138 unmodified lines | ||
| 447 | 445 | if (!desc) |
| 448 | 446 | break; |
| 449 | 447 | |
| 448 | + /* No need to lock. This is called only for the 1st client. */ | |
| 450 | 449 | list_add_tail(&desc->node, &channel->ld_free); |
| 451 | 450 | channel->descs_allocated++; |
| 452 | 451 | } |
| @@ -502,18 +501,21 @@ rz_dmac_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, · 49 unmodified lines | ||
| 502 | 501 | dev_dbg(dmac->dev, "%s channel: %d src=0x%pad dst=0x%pad len=%zu\n", |
| 503 | 502 | __func__, channel->index, &src, &dest, len); |
| 504 | 503 | |
| 505 | − if (list_empty(&channel->ld_free)) | |
| 506 | − return NULL; | |
| 504 | + scoped_guard(spinlock_irqsave, &channel->vc.lock) { | |
| 505 | + if (list_empty(&channel->ld_free)) | |
| 506 | + return NULL; | |
| 507 | 507 | |
| 508 | − desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); | |
| 508 | + desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); | |
| 509 | 509 | |
| 510 | − desc->type = RZ_DMAC_DESC_MEMCPY; | |
| 511 | − desc->src = src; | |
| 512 | − desc->dest = dest; | |
| 513 | − desc->len = len; | |
| 514 | − desc->direction = DMA_MEM_TO_MEM; | |
| 510 | + desc->type = RZ_DMAC_DESC_MEMCPY; | |
| 511 | + desc->src = src; | |
| 512 | + desc->dest = dest; | |
| 513 | + desc->len = len; | |
| 514 | + desc->direction = DMA_MEM_TO_MEM; | |
| 515 | + | |
| 516 | + list_move_tail(channel->ld_free.next, &channel->ld_queue); | |
| 517 | + } | |
| 515 | 518 | |
| 516 | − list_move_tail(channel->ld_free.next, &channel->ld_queue); | |
| 517 | 519 | return vchan_tx_prep(&channel->vc, &desc->vd, flags); |
| 518 | 520 | } |
| 519 | 521 | |
| @@ -529,27 +531,29 @@ rz_dmac_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, · 9 unmodified lines | ||
| 529 | 531 | int dma_length = 0; |
| 530 | 532 | int i = 0; |
| 531 | 533 | |
| 532 | − if (list_empty(&channel->ld_free)) | |
| 533 | − return NULL; | |
| 534 | + scoped_guard(spinlock_irqsave, &channel->vc.lock) { | |
| 535 | + if (list_empty(&channel->ld_free)) | |
| 536 | + return NULL; | |
| 534 | 537 | |
| 535 | − desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); | |
| 538 | + desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); | |
| 536 | 539 | |
| 537 | − for_each_sg(sgl, sg, sg_len, i) { | |
| 538 | − dma_length += sg_dma_len(sg); | |
| 539 | − } | |
| 540 | + for_each_sg(sgl, sg, sg_len, i) | |
| 541 | + dma_length += sg_dma_len(sg); | |
| 540 | 542 | |
| 541 | − desc->type = RZ_DMAC_DESC_SLAVE_SG; | |
| 542 | − desc->sg = sgl; | |
| 543 | − desc->sgcount = sg_len; | |
| 544 | − desc->len = dma_length; | |
| 545 | − desc->direction = direction; | |
| 543 | + desc->type = RZ_DMAC_DESC_SLAVE_SG; | |
| 544 | + desc->sg = sgl; | |
| 545 | + desc->sgcount = sg_len; | |
| 546 | + desc->len = dma_length; | |
| 547 | + desc->direction = direction; | |
| 546 | 548 | |
| 547 | − if (direction == DMA_DEV_TO_MEM) | |
| 548 | − desc->src = channel->src_per_address; | |
| 549 | − else | |
| 550 | − desc->dest = channel->dst_per_address; | |
| 549 | + if (direction == DMA_DEV_TO_MEM) | |
| 550 | + desc->src = channel->src_per_address; | |
| 551 | + else | |
| 552 | + desc->dest = channel->dst_per_address; | |
| 553 | + | |
| 554 | + list_move_tail(channel->ld_free.next, &channel->ld_queue); | |
| 555 | + } | |
| 551 | 556 | |
| 552 | − list_move_tail(channel->ld_free.next, &channel->ld_queue); | |
| 553 | 557 | return vchan_tx_prep(&channel->vc, &desc->vd, flags); |
| 554 | 558 | } |
| 555 | 559 | |
| @@ -561,8 +565,8 @@ static int rz_dmac_terminate_all(struct dma_chan *chan) · 5 unmodified lines | ||
| 561 | 565 | unsigned int i; |
| 562 | 566 | LIST_HEAD(head); |
| 563 | 567 | |
| 564 | − rz_dmac_disable_hw(channel); | |
| 565 | 568 | spin_lock_irqsave(&channel->vc.lock, flags); |
| 569 | + rz_dmac_disable_hw(channel); | |
| 566 | 570 | for (i = 0; i < DMAC_NR_LMDESC; i++) |
| 567 | 571 | lmdesc[i].header = 0; |
| 568 | 572 | |
| @@ -699,7 +703,9 @@ static void rz_dmac_irq_handle_channel(struct rz_dmac_chan *channel) · 130 unmodified lines | ||
| 699 | 703 | if (chstat & CHSTAT_ER) { |
| 700 | 704 | dev_err(dmac->dev, "DMAC err CHSTAT_%d = %08X\n", |
| 701 | 705 | channel->index, chstat); |
| 702 | − rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); | |
| 706 | + | |
| 707 | + scoped_guard(spinlock_irqsave, &channel->vc.lock) | |
| 708 | + rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); | |
| 703 | 709 | goto done; |
| 704 | 710 | } |
| 705 | 711 | |
| @@ -1234,8 +1234,8 @@ static int xdma_probe(struct platform_device *pdev) · 1233 unmodified lines | ||
| 1234 | 1234 | |
| 1235 | 1235 | xdev->rmap = devm_regmap_init_mmio(&pdev->dev, reg_base, |
| 1236 | 1236 | &xdma_regmap_config); |
| 1237 | − if (!xdev->rmap) { | |
| 1238 | − xdma_err(xdev, "config regmap failed: %d", ret); | |
| 1237 | + if (IS_ERR(xdev->rmap)) { | |
| 1238 | + xdma_err(xdev, "config regmap failed: %pe", xdev->rmap); | |
| 1239 | 1239 | goto failed; |
| 1240 | 1240 | } |
| 1241 | 1241 | INIT_LIST_HEAD(&xdev->dma_dev.channels); |
| @@ -997,16 +997,16 @@ static u32 xilinx_dma_get_residue(struct xilinx_dma_chan *chan, · 996 unmodified lines | ||
| 997 | 997 | struct xilinx_cdma_tx_segment, |
| 998 | 998 | node); |
| 999 | 999 | cdma_hw = &cdma_seg->hw; |
| 1000 | − residue += (cdma_hw->control - cdma_hw->status) & | |
| 1001 | − chan->xdev->max_buffer_len; | |
| 1000 | + residue += (cdma_hw->control & chan->xdev->max_buffer_len) - | |
| 1001 | + (cdma_hw->status & chan->xdev->max_buffer_len); | |
| 1002 | 1002 | } else if (chan->xdev->dma_config->dmatype == |
| 1003 | 1003 | XDMA_TYPE_AXIDMA) { |
| 1004 | 1004 | axidma_seg = list_entry(entry, |
| 1005 | 1005 | struct xilinx_axidma_tx_segment, |
| 1006 | 1006 | node); |
| 1007 | 1007 | axidma_hw = &axidma_seg->hw; |
| 1008 | − residue += (axidma_hw->control - axidma_hw->status) & | |
| 1009 | − chan->xdev->max_buffer_len; | |
| 1008 | + residue += (axidma_hw->control & chan->xdev->max_buffer_len) - | |
| 1009 | + (axidma_hw->status & chan->xdev->max_buffer_len); | |
| 1010 | 1010 | } else { |
| 1011 | 1011 | aximcdma_seg = |
| 1012 | 1012 | list_entry(entry, |
| @@ -1014,8 +1014,8 @@ static u32 xilinx_dma_get_residue(struct xilinx_dma_chan *chan, · 1 unmodified line | ||
| 1014 | 1014 | node); |
| 1015 | 1015 | aximcdma_hw = &aximcdma_seg->hw; |
| 1016 | 1016 | residue += |
| 1017 | − (aximcdma_hw->control - aximcdma_hw->status) & | |
| 1018 | − chan->xdev->max_buffer_len; | |
| 1017 | + (aximcdma_hw->control & chan->xdev->max_buffer_len) - | |
| 1018 | + (aximcdma_hw->status & chan->xdev->max_buffer_len); | |
| 1019 | 1019 | } |
| 1020 | 1020 | } |
| 1021 | 1021 | |
| @@ -1235,14 +1235,6 @@ static int xilinx_dma_alloc_chan_resources(struct dma_chan *dchan) · 213 unmodified lines | ||
| 1235 | 1235 | |
| 1236 | 1236 | dma_cookie_init(dchan); |
| 1237 | 1237 | |
| 1238 | − if (chan->xdev->dma_config->dmatype == XDMA_TYPE_AXIDMA) { | |
| 1239 | − /* For AXI DMA resetting once channel will reset the | |
| 1240 | − * other channel as well so enable the interrupts here. | |
| 1241 | − */ | |
| 1242 | − dma_ctrl_set(chan, XILINX_DMA_REG_DMACR, | |
| 1243 | − XILINX_DMA_DMAXR_ALL_IRQ_MASK); | |
| 1244 | − } | |
| 1245 | − | |
| 1246 | 1238 | if ((chan->xdev->dma_config->dmatype == XDMA_TYPE_CDMA) && chan->has_sg) |
| 1247 | 1239 | dma_ctrl_set(chan, XILINX_DMA_REG_DMACR, |
| 1248 | 1240 | XILINX_CDMA_CR_SGMODE); |
| @@ -1564,8 +1556,29 @@ static void xilinx_dma_start_transfer(struct xilinx_dma_chan *chan) · 315 unmodified lines | ||
| 1564 | 1556 | if (chan->err) |
| 1565 | 1557 | return; |
| 1566 | 1558 | |
| 1567 | − if (list_empty(&chan->pending_list)) | |
| 1559 | + if (list_empty(&chan->pending_list)) { | |
| 1560 | + if (chan->cyclic) { | |
| 1561 | + struct xilinx_dma_tx_descriptor *desc; | |
| 1562 | + struct list_head *entry; | |
| 1563 | + | |
| 1564 | + desc = list_last_entry(&chan->done_list, | |
| 1565 | + struct xilinx_dma_tx_descriptor, node); | |
| 1566 | + list_for_each(entry, &desc->segments) { | |
| 1567 | + struct xilinx_axidma_tx_segment *axidma_seg; | |
| 1568 | + struct xilinx_axidma_desc_hw *axidma_hw; | |
| 1569 | + axidma_seg = list_entry(entry, | |
| 1570 | + struct xilinx_axidma_tx_segment, | |
| 1571 | + node); | |
| 1572 | + axidma_hw = &axidma_seg->hw; | |
| 1573 | + axidma_hw->status = 0; | |
| 1574 | + } | |
| 1575 | + | |
| 1576 | + list_splice_tail_init(&chan->done_list, &chan->active_list); | |
| 1577 | + chan->desc_pendingcount = 0; | |
| 1578 | + chan->idle = false; | |
| 1579 | + } | |
| 1568 | 1580 | return; |
| 1581 | + } | |
| 1569 | 1582 | |
| 1570 | 1583 | if (!chan->idle) |
| 1571 | 1584 | return; |
| @@ -1591,6 +1604,7 @@ static void xilinx_dma_start_transfer(struct xilinx_dma_chan *chan) · 19 unmodified lines | ||
| 1591 | 1604 | head_desc->async_tx.phys); |
| 1592 | 1605 | reg &= ~XILINX_DMA_CR_DELAY_MAX; |
| 1593 | 1606 | reg |= chan->irq_delay << XILINX_DMA_CR_DELAY_SHIFT; |
| 1607 | + reg |= XILINX_DMA_DMAXR_ALL_IRQ_MASK; | |
| 1594 | 1608 | dma_ctrl_write(chan, XILINX_DMA_REG_DMACR, reg); |
| 1595 | 1609 | |
| 1596 | 1610 | xilinx_dma_start(chan); |
| @@ -3024,7 +3038,7 @@ static int xilinx_dma_chan_probe(struct xilinx_dma_device *xdev, · 1427 unmodified lines | ||
| 3024 | 3038 | return -EINVAL; |
| 3025 | 3039 | } |
| 3026 | 3040 | |
| 3027 | − xdev->common.directions |= chan->direction; | |
| 3041 | + xdev->common.directions |= BIT(chan->direction); | |
| 3028 | 3042 | |
| 3029 | 3043 | /* Request the interrupt */ |
| 3030 | 3044 | chan->irq = of_irq_get(node, chan->tdest); |
| @@ -257,9 +257,10 @@ static void fwnet_header_cache_update(struct hh_cache *hh, · 256 unmodified lines | ||
| 257 | 257 | memcpy((u8 *)hh->hh_data + HH_DATA_OFF(FWNET_HLEN), haddr, net->addr_len); |
| 258 | 258 | } |
| 259 | 259 | |
| 260 | −static int fwnet_header_parse(const struct sk_buff *skb, unsigned char *haddr) | |
| 260 | +static int fwnet_header_parse(const struct sk_buff *skb, const struct net_device *dev, | |
| 261 | + unsigned char *haddr) | |
| 261 | 262 | { |
| 262 | − memcpy(haddr, skb->dev->dev_addr, FWNET_ALEN); | |
| 263 | + memcpy(haddr, dev->dev_addr, FWNET_ALEN); | |
| 263 | 264 | |
| 264 | 265 | return FWNET_ALEN; |
| 265 | 266 | } |
| @@ -205,12 +205,12 @@ static int ffa_rxtx_map(phys_addr_t tx_buf, phys_addr_t rx_buf, u32 pg_cnt) · 204 unmodified lines | ||
| 205 | 205 | return 0; |
| 206 | 206 | } |
| 207 | 207 | |
| 208 | −static int ffa_rxtx_unmap(u16 vm_id) | |
| 208 | +static int ffa_rxtx_unmap(void) | |
| 209 | 209 | { |
| 210 | 210 | ffa_value_t ret; |
| 211 | 211 | |
| 212 | 212 | invoke_ffa_fn((ffa_value_t){ |
| 213 | − .a0 = FFA_RXTX_UNMAP, .a1 = PACK_TARGET_INFO(vm_id, 0), | |
| 213 | + .a0 = FFA_RXTX_UNMAP, | |
| 214 | 214 | }, &ret); |
| 215 | 215 | |
| 216 | 216 | if (ret.a0 == FFA_ERROR) |
| @@ -2097,7 +2097,7 @@ static int __init ffa_init(void) · 1880 unmodified lines | ||
| 2097 | 2097 | |
| 2098 | 2098 | pr_err("failed to setup partitions\n"); |
| 2099 | 2099 | ffa_notifications_cleanup(); |
| 2100 | − ffa_rxtx_unmap(drv_info->vm_id); | |
| 2100 | + ffa_rxtx_unmap(); | |
| 2101 | 2101 | free_pages: |
| 2102 | 2102 | if (drv_info->tx_buffer) |
| 2103 | 2103 | free_pages_exact(drv_info->tx_buffer, rxtx_bufsz); |
| @@ -2112,7 +2112,7 @@ static void __exit ffa_exit(void) · 8 unmodified lines | ||
| 2112 | 2112 | { |
| 2113 | 2113 | ffa_notifications_cleanup(); |
| 2114 | 2114 | ffa_partitions_cleanup(); |
| 2115 | − ffa_rxtx_unmap(drv_info->vm_id); | |
| 2115 | + ffa_rxtx_unmap(); | |
| 2116 | 2116 | free_pages_exact(drv_info->tx_buffer, drv_info->rxtx_bufsz); |
| 2117 | 2117 | free_pages_exact(drv_info->rx_buffer, drv_info->rxtx_bufsz); |
| 2118 | 2118 | kfree(drv_info); |
| @@ -1066,7 +1066,7 @@ static int scmi_register_event_handler(struct scmi_notify_instance *ni, · 1065 unmodified lines | ||
| 1066 | 1066 | * since at creation time we usually want to have all setup and ready before |
| 1067 | 1067 | * events really start flowing. |
| 1068 | 1068 | * |
| 1069 | − * Return: A properly refcounted handler on Success, NULL on Failure | |
| 1069 | + * Return: A properly refcounted handler on Success, ERR_PTR on Failure | |
| 1070 | 1070 | */ |
| 1071 | 1071 | static inline struct scmi_event_handler * |
| 1072 | 1072 | __scmi_event_handler_get_ops(struct scmi_notify_instance *ni, |
| @@ -1113,7 +1113,7 @@ __scmi_event_handler_get_ops(struct scmi_notify_instance *ni, · 40 unmodified lines | ||
| 1113 | 1113 | } |
| 1114 | 1114 | mutex_unlock(&ni->pending_mtx); |
| 1115 | 1115 | |
| 1116 | − return hndl; | |
| 1116 | + return hndl ?: ERR_PTR(-ENODEV); | |
| 1117 | 1117 | } |
| 1118 | 1118 | |
| 1119 | 1119 | static struct scmi_event_handler * |
| @@ -189,13 +189,13 @@ struct scmi_protocol_handle { · 188 unmodified lines | ||
| 189 | 189 | |
| 190 | 190 | /** |
| 191 | 191 | * struct scmi_iterator_state - Iterator current state descriptor |
| 192 | − * @desc_index: Starting index for the current mulit-part request. | |
| 192 | + * @desc_index: Starting index for the current multi-part request. | |
| 193 | 193 | * @num_returned: Number of returned items in the last multi-part reply. |
| 194 | 194 | * @num_remaining: Number of remaining items in the multi-part message. |
| 195 | 195 | * @max_resources: Maximum acceptable number of items, configured by the caller |
| 196 | 196 | * depending on the underlying resources that it is querying. |
| 197 | 197 | * @loop_idx: The iterator loop index in the current multi-part reply. |
| 198 | − * @rx_len: Size in bytes of the currenly processed message; it can be used by | |
| 198 | + * @rx_len: Size in bytes of the currently processed message; it can be used by | |
| 199 | 199 | * the user of the iterator to verify a reply size. |
| 200 | 200 | * @priv: Optional pointer to some additional state-related private data setup |
| 201 | 201 | * by the caller during the iterations. |
| @@ -18,6 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | |
| 19 | 19 | #include <linux/bitmap.h> |
| 20 | 20 | #include <linux/bitfield.h> |
| 21 | +#include <linux/cleanup.h> | |
| 21 | 22 | #include <linux/device.h> |
| 22 | 23 | #include <linux/err.h> |
| 23 | 24 | #include <linux/export.h> |
| @@ -940,13 +941,13 @@ static int scpi_probe(struct platform_device *pdev) · 916 unmodified lines | ||
| 940 | 941 | int idx = scpi_drvinfo->num_chans; |
| 941 | 942 | struct scpi_chan *pchan = scpi_drvinfo->channels + idx; |
| 942 | 943 | struct mbox_client *cl = &pchan->cl; |
| 943 | − struct device_node *shmem = of_parse_phandle(np, "shmem", idx); | |
| 944 | + struct device_node *shmem __free(device_node) = | |
| 945 | + of_parse_phandle(np, "shmem", idx); | |
| 944 | 946 | |
| 945 | 947 | if (!of_match_node(shmem_of_match, shmem)) |
| 946 | 948 | return -ENXIO; |
| 947 | 949 | |
| 948 | 950 | ret = of_address_to_resource(shmem, 0, &res); |
| 949 | − of_node_put(shmem); | |
| 950 | 951 | if (ret) { |
| 951 | 952 | dev_err(dev, "failed to get SCPI payload mem resource\n"); |
| 952 | 953 | return ret; |
| @@ -1610,11 +1610,17 @@ static int cs_dsp_load(struct cs_dsp *dsp, const struct firmware *firmware, · 1609 unmodified lines | ||
| 1610 | 1610 | region_name); |
| 1611 | 1611 | |
| 1612 | 1612 | if (reg) { |
| 1613 | + /* | |
| 1614 | + * Although we expect the underlying bus does not require | |
| 1615 | + * physically-contiguous buffers, we pessimistically use | |
| 1616 | + * a temporary buffer instead of trusting that the | |
| 1617 | + * alignment of region->data is ok. | |
| 1618 | + */ | |
| 1613 | 1619 | region_len = le32_to_cpu(region->len); |
| 1614 | 1620 | if (region_len > buf_len) { |
| 1615 | 1621 | buf_len = round_up(region_len, PAGE_SIZE); |
| 1616 | − kfree(buf); | |
| 1617 | − buf = kmalloc(buf_len, GFP_KERNEL | GFP_DMA); | |
| 1622 | + vfree(buf); | |
| 1623 | + buf = vmalloc(buf_len); | |
| 1618 | 1624 | if (!buf) { |
| 1619 | 1625 | ret = -ENOMEM; |
| 1620 | 1626 | goto out_fw; |
| @@ -1643,7 +1649,7 @@ static int cs_dsp_load(struct cs_dsp *dsp, const struct firmware *firmware, · 22 unmodified lines | ||
| 1643 | 1649 | |
| 1644 | 1650 | ret = 0; |
| 1645 | 1651 | out_fw: |
| 1646 | − kfree(buf); | |
| 1652 | + vfree(buf); | |
| 1647 | 1653 | |
| 1648 | 1654 | if (ret == -EOVERFLOW) |
| 1649 | 1655 | cs_dsp_err(dsp, "%s: file content overflows file data\n", file); |
| @@ -2331,11 +2337,17 @@ static int cs_dsp_load_coeff(struct cs_dsp *dsp, const struct firmware *firmware · 681 unmodified lines | ||
| 2331 | 2337 | } |
| 2332 | 2338 | |
| 2333 | 2339 | if (reg) { |
| 2340 | + /* | |
| 2341 | + * Although we expect the underlying bus does not require | |
| 2342 | + * physically-contiguous buffers, we pessimistically use | |
| 2343 | + * a temporary buffer instead of trusting that the | |
| 2344 | + * alignment of blk->data is ok. | |
| 2345 | + */ | |
| 2334 | 2346 | region_len = le32_to_cpu(blk->len); |
| 2335 | 2347 | if (region_len > buf_len) { |
| 2336 | 2348 | buf_len = round_up(region_len, PAGE_SIZE); |
| 2337 | − kfree(buf); | |
| 2338 | − buf = kmalloc(buf_len, GFP_KERNEL | GFP_DMA); | |
| 2349 | + vfree(buf); | |
| 2350 | + buf = vmalloc(buf_len); | |
| 2339 | 2351 | if (!buf) { |
| 2340 | 2352 | ret = -ENOMEM; |
| 2341 | 2353 | goto out_fw; |
| @@ -2366,7 +2378,7 @@ static int cs_dsp_load_coeff(struct cs_dsp *dsp, const struct firmware *firmware · 24 unmodified lines | ||
| 2366 | 2378 | |
| 2367 | 2379 | ret = 0; |
| 2368 | 2380 | out_fw: |
| 2369 | − kfree(buf); | |
| 2381 | + vfree(buf); | |
| 2370 | 2382 | |
| 2371 | 2383 | if (ret == -EOVERFLOW) |
| 2372 | 2384 | cs_dsp_err(dsp, "%s: file content overflows file data\n", file); |
| @@ -768,7 +768,9 @@ static int stratix10_rsu_probe(struct platform_device *pdev) · 767 unmodified lines | ||
| 768 | 768 | rsu_async_status_callback); |
| 769 | 769 | if (ret) { |
| 770 | 770 | dev_err(dev, "Error, getting RSU status %i\n", ret); |
| 771 | + stratix10_svc_remove_async_client(priv->chan); | |
| 771 | 772 | stratix10_svc_free_channel(priv->chan); |
| 773 | + return ret; | |
| 772 | 774 | } |
| 773 | 775 | |
| 774 | 776 | /* get DCMF version from firmware */ |
| @@ -37,15 +37,14 @@ · 36 unmodified lines | ||
| 37 | 37 | * service layer will return error to FPGA manager when timeout occurs, |
| 38 | 38 | * timeout is set to 30 seconds (30 * 1000) at Intel Stratix10 SoC. |
| 39 | 39 | */ |
| 40 | −#define SVC_NUM_DATA_IN_FIFO 32 | |
| 40 | +#define SVC_NUM_DATA_IN_FIFO 8 | |
| 41 | 41 | #define SVC_NUM_CHANNEL 4 |
| 42 | −#define FPGA_CONFIG_DATA_CLAIM_TIMEOUT_MS 200 | |
| 42 | +#define FPGA_CONFIG_DATA_CLAIM_TIMEOUT_MS 2000 | |
| 43 | 43 | #define FPGA_CONFIG_STATUS_TIMEOUT_SEC 30 |
| 44 | 44 | #define BYTE_TO_WORD_SIZE 4 |
| 45 | 45 | |
| 46 | 46 | /* stratix10 service layer clients */ |
| 47 | 47 | #define STRATIX10_RSU "stratix10-rsu" |
| 48 | −#define INTEL_FCS "intel-fcs" | |
| 49 | 48 | |
| 50 | 49 | /* Maximum number of SDM client IDs. */ |
| 51 | 50 | #define MAX_SDM_CLIENT_IDS 16 |
| @@ -105,11 +104,9 @@ struct stratix10_svc_chan; · 53 unmodified lines | ||
| 105 | 104 | /** |
| 106 | 105 | * struct stratix10_svc - svc private data |
| 107 | 106 | * @stratix10_svc_rsu: pointer to stratix10 RSU device |
| 108 | − * @intel_svc_fcs: pointer to the FCS device | |
| 109 | 107 | */ |
| 110 | 108 | struct stratix10_svc { |
| 111 | 109 | struct platform_device *stratix10_svc_rsu; |
| 112 | − struct platform_device *intel_svc_fcs; | |
| 113 | 110 | }; |
| 114 | 111 | |
| 115 | 112 | /** |
| @@ -251,12 +248,10 @@ struct stratix10_async_ctrl { · 135 unmodified lines | ||
| 251 | 248 | * @num_active_client: number of active service client |
| 252 | 249 | * @node: list management |
| 253 | 250 | * @genpool: memory pool pointing to the memory region |
| 254 | − * @task: pointer to the thread task which handles SMC or HVC call | |
| 255 | − * @svc_fifo: a queue for storing service message data | |
| 256 | 251 | * @complete_status: state for completion |
| 257 | − * @svc_fifo_lock: protect access to service message data queue | |
| 258 | 252 | * @invoke_fn: function to issue secure monitor call or hypervisor call |
| 259 | 253 | * @svc: manages the list of client svc drivers |
| 254 | + * @sdm_lock: only allows a single command single response to SDM | |
| 260 | 255 | * @actrl: async control structure |
| 261 | 256 | * |
| 262 | 257 | * This struct is used to create communication channels for service clients, to |
| @@ -269,12 +264,10 @@ struct stratix10_svc_controller { · 6 unmodified lines | ||
| 269 | 264 | int num_active_client; |
| 270 | 265 | struct list_head node; |
| 271 | 266 | struct gen_pool *genpool; |
| 272 | − struct task_struct *task; | |
| 273 | − struct kfifo svc_fifo; | |
| 274 | 267 | struct completion complete_status; |
| 275 | − spinlock_t svc_fifo_lock; | |
| 276 | 268 | svc_invoke_fn *invoke_fn; |
| 277 | 269 | struct stratix10_svc *svc; |
| 270 | + struct mutex sdm_lock; | |
| 278 | 271 | struct stratix10_async_ctrl actrl; |
| 279 | 272 | }; |
| 280 | 273 | |
| @@ -283,6 +276,9 @@ struct stratix10_svc_controller { · 2 unmodified lines | ||
| 283 | 276 | * @ctrl: pointer to service controller which is the provider of this channel |
| 284 | 277 | * @scl: pointer to service client which owns the channel |
| 285 | 278 | * @name: service client name associated with the channel |
| 279 | + * @task: pointer to the thread task which handles SMC or HVC call | |
| 280 | + * @svc_fifo: a queue for storing service message data (separate fifo for every channel) | |
| 281 | + * @svc_fifo_lock: protect access to service message data queue (locking pending fifo) | |
| 286 | 282 | * @lock: protect access to the channel |
| 287 | 283 | * @async_chan: reference to asynchronous channel object for this channel |
| 288 | 284 | * |
| @@ -293,6 +289,9 @@ struct stratix10_svc_chan { · 4 unmodified lines | ||
| 293 | 289 | struct stratix10_svc_controller *ctrl; |
| 294 | 290 | struct stratix10_svc_client *scl; |
| 295 | 291 | char *name; |
| 292 | + struct task_struct *task; | |
| 293 | + struct kfifo svc_fifo; | |
| 294 | + spinlock_t svc_fifo_lock; | |
| 296 | 295 | spinlock_t lock; |
| 297 | 296 | struct stratix10_async_chan *async_chan; |
| 298 | 297 | }; |
| @@ -527,10 +526,10 @@ static void svc_thread_recv_status_ok(struct stratix10_svc_data *p_data, · 228 unmodified lines | ||
| 527 | 526 | */ |
| 528 | 527 | static int svc_normal_to_secure_thread(void *data) |
| 529 | 528 | { |
| 530 | − struct stratix10_svc_controller | |
| 531 | − *ctrl = (struct stratix10_svc_controller *)data; | |
| 532 | − struct stratix10_svc_data *pdata; | |
| 533 | − struct stratix10_svc_cb_data *cbdata; | |
| 529 | + struct stratix10_svc_chan *chan = (struct stratix10_svc_chan *)data; | |
| 530 | + struct stratix10_svc_controller *ctrl = chan->ctrl; | |
| 531 | + struct stratix10_svc_data *pdata = NULL; | |
| 532 | + struct stratix10_svc_cb_data *cbdata = NULL; | |
| 534 | 533 | struct arm_smccc_res res; |
| 535 | 534 | unsigned long a0, a1, a2, a3, a4, a5, a6, a7; |
| 536 | 535 | int ret_fifo = 0; |
| @@ -555,12 +554,12 @@ static int svc_normal_to_secure_thread(void *data) · 18 unmodified lines | ||
| 555 | 554 | a6 = 0; |
| 556 | 555 | a7 = 0; |
| 557 | 556 | |
| 558 | − pr_debug("smc_hvc_shm_thread is running\n"); | |
| 557 | + pr_debug("%s: %s: Thread is running!\n", __func__, chan->name); | |
| 559 | 558 | |
| 560 | 559 | while (!kthread_should_stop()) { |
| 561 | − ret_fifo = kfifo_out_spinlocked(&ctrl->svc_fifo, | |
| 560 | + ret_fifo = kfifo_out_spinlocked(&chan->svc_fifo, | |
| 562 | 561 | pdata, sizeof(*pdata), |
| 563 | − &ctrl->svc_fifo_lock); | |
| 562 | + &chan->svc_fifo_lock); | |
| 564 | 563 | |
| 565 | 564 | if (!ret_fifo) |
| 566 | 565 | continue; |
| @@ -569,9 +568,25 @@ static int svc_normal_to_secure_thread(void *data) · 2 unmodified lines | ||
| 569 | 568 | (unsigned int)pdata->paddr, pdata->command, |
| 570 | 569 | (unsigned int)pdata->size); |
| 571 | 570 | |
| 571 | + /* SDM can only process one command at a time */ | |
| 572 | + pr_debug("%s: %s: Thread is waiting for mutex!\n", | |
| 573 | + __func__, chan->name); | |
| 574 | + if (mutex_lock_interruptible(&ctrl->sdm_lock)) { | |
| 575 | + /* item already dequeued; notify client to unblock it */ | |
| 576 | + cbdata->status = BIT(SVC_STATUS_ERROR); | |
| 577 | + cbdata->kaddr1 = NULL; | |
| 578 | + cbdata->kaddr2 = NULL; | |
| 579 | + cbdata->kaddr3 = NULL; | |
| 580 | + if (pdata->chan->scl) | |
| 581 | + pdata->chan->scl->receive_cb(pdata->chan->scl, | |
| 582 | + cbdata); | |
| 583 | + break; | |
| 584 | + } | |
| 585 | + | |
| 572 | 586 | switch (pdata->command) { |
| 573 | 587 | case COMMAND_RECONFIG_DATA_CLAIM: |
| 574 | 588 | svc_thread_cmd_data_claim(ctrl, pdata, cbdata); |
| 589 | + mutex_unlock(&ctrl->sdm_lock); | |
| 575 | 590 | continue; |
| 576 | 591 | case COMMAND_RECONFIG: |
| 577 | 592 | a0 = INTEL_SIP_SMC_FPGA_CONFIG_START; |
| @@ -700,10 +715,11 @@ static int svc_normal_to_secure_thread(void *data) · 122 unmodified lines | ||
| 700 | 715 | break; |
| 701 | 716 | default: |
| 702 | 717 | pr_warn("it shouldn't happen\n"); |
| 703 | − break; | |
| 718 | + mutex_unlock(&ctrl->sdm_lock); | |
| 719 | + continue; | |
| 704 | 720 | } |
| 705 | − pr_debug("%s: before SMC call -- a0=0x%016x a1=0x%016x", | |
| 706 | − __func__, | |
| 721 | + pr_debug("%s: %s: before SMC call -- a0=0x%016x a1=0x%016x", | |
| 722 | + __func__, chan->name, | |
| 707 | 723 | (unsigned int)a0, |
| 708 | 724 | (unsigned int)a1); |
| 709 | 725 | pr_debug(" a2=0x%016x\n", (unsigned int)a2); |
| @@ -712,8 +728,8 @@ static int svc_normal_to_secure_thread(void *data) · 2 unmodified lines | ||
| 712 | 728 | pr_debug(" a5=0x%016x\n", (unsigned int)a5); |
| 713 | 729 | ctrl->invoke_fn(a0, a1, a2, a3, a4, a5, a6, a7, &res); |
| 714 | 730 | |
| 715 | − pr_debug("%s: after SMC call -- res.a0=0x%016x", | |
| 716 | − __func__, (unsigned int)res.a0); | |
| 731 | + pr_debug("%s: %s: after SMC call -- res.a0=0x%016x", | |
| 732 | + __func__, chan->name, (unsigned int)res.a0); | |
| 717 | 733 | pr_debug(" res.a1=0x%016x, res.a2=0x%016x", |
| 718 | 734 | (unsigned int)res.a1, (unsigned int)res.a2); |
| 719 | 735 | pr_debug(" res.a3=0x%016x\n", (unsigned int)res.a3); |
| @@ -728,6 +744,7 @@ static int svc_normal_to_secure_thread(void *data) · 8 unmodified lines | ||
| 728 | 744 | cbdata->kaddr2 = NULL; |
| 729 | 745 | cbdata->kaddr3 = NULL; |
| 730 | 746 | pdata->chan->scl->receive_cb(pdata->chan->scl, cbdata); |
| 747 | + mutex_unlock(&ctrl->sdm_lock); | |
| 731 | 748 | continue; |
| 732 | 749 | } |
| 733 | 750 | |
| @@ -801,6 +818,8 @@ static int svc_normal_to_secure_thread(void *data) · 67 unmodified lines | ||
| 801 | 818 | break; |
| 802 | 819 | |
| 803 | 820 | } |
| 821 | + | |
| 822 | + mutex_unlock(&ctrl->sdm_lock); | |
| 804 | 823 | } |
| 805 | 824 | |
| 806 | 825 | kfree(cbdata); |
| @@ -1696,22 +1715,33 @@ int stratix10_svc_send(struct stratix10_svc_chan *chan, void *msg) · 889 unmodified lines | ||
| 1696 | 1715 | if (!p_data) |
| 1697 | 1716 | return -ENOMEM; |
| 1698 | 1717 | |
| 1699 | − /* first client will create kernel thread */ | |
| 1700 | − if (!chan->ctrl->task) { | |
| 1701 | − chan->ctrl->task = | |
| 1702 | − kthread_run_on_cpu(svc_normal_to_secure_thread, | |
| 1703 | − (void *)chan->ctrl, | |
| 1704 | − cpu, "svc_smc_hvc_thread"); | |
| 1705 | − if (IS_ERR(chan->ctrl->task)) { | |
| 1718 | + /* first caller creates the per-channel kthread */ | |
| 1719 | + if (!chan->task) { | |
| 1720 | + struct task_struct *task; | |
| 1721 | + | |
| 1722 | + task = kthread_run_on_cpu(svc_normal_to_secure_thread, | |
| 1723 | + (void *)chan, | |
| 1724 | + cpu, "svc_smc_hvc_thread"); | |
| 1725 | + if (IS_ERR(task)) { | |
| 1706 | 1726 | dev_err(chan->ctrl->dev, |
| 1707 | 1727 | "failed to create svc_smc_hvc_thread\n"); |
| 1708 | 1728 | kfree(p_data); |
| 1709 | 1729 | return -EINVAL; |
| 1710 | 1730 | } |
| 1731 | + | |
| 1732 | + spin_lock(&chan->lock); | |
| 1733 | + if (chan->task) { | |
| 1734 | + /* another caller won the race; discard our thread */ | |
| 1735 | + spin_unlock(&chan->lock); | |
| 1736 | + kthread_stop(task); | |
| 1737 | + } else { | |
| 1738 | + chan->task = task; | |
| 1739 | + spin_unlock(&chan->lock); | |
| 1740 | + } | |
| 1711 | 1741 | } |
| 1712 | 1742 | |
| 1713 | − pr_debug("%s: sent P-va=%p, P-com=%x, P-size=%u\n", __func__, | |
| 1714 | − p_msg->payload, p_msg->command, | |
| 1743 | + pr_debug("%s: %s: sent P-va=%p, P-com=%x, P-size=%u\n", __func__, | |
| 1744 | + chan->name, p_msg->payload, p_msg->command, | |
| 1715 | 1745 | (unsigned int)p_msg->payload_length); |
| 1716 | 1746 | |
| 1717 | 1747 | if (list_empty(&svc_data_mem)) { |
| @@ -1747,12 +1777,16 @@ int stratix10_svc_send(struct stratix10_svc_chan *chan, void *msg) · 29 unmodified lines | ||
| 1747 | 1777 | p_data->arg[2] = p_msg->arg[2]; |
| 1748 | 1778 | p_data->size = p_msg->payload_length; |
| 1749 | 1779 | p_data->chan = chan; |
| 1750 | − pr_debug("%s: put to FIFO pa=0x%016x, cmd=%x, size=%u\n", __func__, | |
| 1751 | − (unsigned int)p_data->paddr, p_data->command, | |
| 1752 | − (unsigned int)p_data->size); | |
| 1753 | − ret = kfifo_in_spinlocked(&chan->ctrl->svc_fifo, p_data, | |
| 1780 | + pr_debug("%s: %s: put to FIFO pa=0x%016x, cmd=%x, size=%u\n", | |
| 1781 | + __func__, | |
| 1782 | + chan->name, | |
| 1783 | + (unsigned int)p_data->paddr, | |
| 1784 | + p_data->command, | |
| 1785 | + (unsigned int)p_data->size); | |
| 1786 | + | |
| 1787 | + ret = kfifo_in_spinlocked(&chan->svc_fifo, p_data, | |
| 1754 | 1788 | sizeof(*p_data), |
| 1755 | − &chan->ctrl->svc_fifo_lock); | |
| 1789 | + &chan->svc_fifo_lock); | |
| 1756 | 1790 | |
| 1757 | 1791 | kfree(p_data); |
| 1758 | 1792 | |
| @@ -1773,11 +1807,12 @@ EXPORT_SYMBOL_GPL(stratix10_svc_send); · 14 unmodified lines | ||
| 1773 | 1807 | */ |
| 1774 | 1808 | void stratix10_svc_done(struct stratix10_svc_chan *chan) |
| 1775 | 1809 | { |
| 1776 | − /* stop thread when thread is running AND only one active client */ | |
| 1777 | − if (chan->ctrl->task && chan->ctrl->num_active_client <= 1) { | |
| 1778 | − pr_debug("svc_smc_hvc_shm_thread is stopped\n"); | |
| 1779 | − kthread_stop(chan->ctrl->task); | |
| 1780 | − chan->ctrl->task = NULL; | |
| 1810 | + /* stop thread when thread is running */ | |
| 1811 | + if (chan->task) { | |
| 1812 | + pr_debug("%s: %s: svc_smc_hvc_shm_thread is stopping\n", | |
| 1813 | + __func__, chan->name); | |
| 1814 | + kthread_stop(chan->task); | |
| 1815 | + chan->task = NULL; | |
| 1781 | 1816 | } |
| 1782 | 1817 | } |
| 1783 | 1818 | EXPORT_SYMBOL_GPL(stratix10_svc_done); |
| @@ -1817,8 +1852,8 @@ void *stratix10_svc_allocate_memory(struct stratix10_svc_chan *chan, · 33 unmodified lines | ||
| 1817 | 1852 | pmem->paddr = pa; |
| 1818 | 1853 | pmem->size = s; |
| 1819 | 1854 | list_add_tail(&pmem->node, &svc_data_mem); |
| 1820 | − pr_debug("%s: va=%p, pa=0x%016x\n", __func__, | |
| 1821 | − pmem->vaddr, (unsigned int)pmem->paddr); | |
| 1855 | + pr_debug("%s: %s: va=%p, pa=0x%016x\n", __func__, | |
| 1856 | + chan->name, pmem->vaddr, (unsigned int)pmem->paddr); | |
| 1822 | 1857 | |
| 1823 | 1858 | return (void *)va; |
| 1824 | 1859 | } |
| @@ -1855,6 +1890,13 @@ static const struct of_device_id stratix10_svc_drv_match[] = { · 30 unmodified lines | ||
| 1855 | 1890 | {}, |
| 1856 | 1891 | }; |
| 1857 | 1892 | |
| 1893 | +static const char * const chan_names[SVC_NUM_CHANNEL] = { | |
| 1894 | + SVC_CLIENT_FPGA, | |
| 1895 | + SVC_CLIENT_RSU, | |
| 1896 | + SVC_CLIENT_FCS, | |
| 1897 | + SVC_CLIENT_HWMON | |
| 1898 | +}; | |
| 1899 | + | |
| 1858 | 1900 | static int stratix10_svc_drv_probe(struct platform_device *pdev) |
| 1859 | 1901 | { |
| 1860 | 1902 | struct device *dev = &pdev->dev; |
| @@ -1862,11 +1904,11 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 1 unmodified line | ||
| 1862 | 1904 | struct stratix10_svc_chan *chans; |
| 1863 | 1905 | struct gen_pool *genpool; |
| 1864 | 1906 | struct stratix10_svc_sh_memory *sh_memory; |
| 1865 | − struct stratix10_svc *svc; | |
| 1907 | + struct stratix10_svc *svc = NULL; | |
| 1866 | 1908 | |
| 1867 | 1909 | svc_invoke_fn *invoke_fn; |
| 1868 | 1910 | size_t fifo_size; |
| 1869 | − int ret; | |
| 1911 | + int ret, i = 0; | |
| 1870 | 1912 | |
| 1871 | 1913 | /* get SMC or HVC function */ |
| 1872 | 1914 | invoke_fn = get_invoke_func(dev); |
| @@ -1905,8 +1947,8 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 32 unmodified lines | ||
| 1905 | 1947 | controller->num_active_client = 0; |
| 1906 | 1948 | controller->chans = chans; |
| 1907 | 1949 | controller->genpool = genpool; |
| 1908 | − controller->task = NULL; | |
| 1909 | 1950 | controller->invoke_fn = invoke_fn; |
| 1951 | + INIT_LIST_HEAD(&controller->node); | |
| 1910 | 1952 | init_completion(&controller->complete_status); |
| 1911 | 1953 | |
| 1912 | 1954 | ret = stratix10_svc_async_init(controller); |
| @@ -1917,32 +1959,20 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 4 unmodified lines | ||
| 1917 | 1959 | } |
| 1918 | 1960 | |
| 1919 | 1961 | fifo_size = sizeof(struct stratix10_svc_data) * SVC_NUM_DATA_IN_FIFO; |
| 1920 | − ret = kfifo_alloc(&controller->svc_fifo, fifo_size, GFP_KERNEL); | |
| 1921 | − if (ret) { | |
| 1922 | − dev_err(dev, "failed to allocate FIFO\n"); | |
| 1923 | − goto err_async_exit; | |
| 1924 | − } | |
| 1925 | − spin_lock_init(&controller->svc_fifo_lock); | |
| 1926 | − | |
| 1927 | − chans[0].scl = NULL; | |
| 1928 | − chans[0].ctrl = controller; | |
| 1929 | − chans[0].name = SVC_CLIENT_FPGA; | |
| 1930 | − spin_lock_init(&chans[0].lock); | |
| 1962 | + mutex_init(&controller->sdm_lock); | |
| 1931 | 1963 | |
| 1932 | − chans[1].scl = NULL; | |
| 1933 | − chans[1].ctrl = controller; | |
| 1934 | − chans[1].name = SVC_CLIENT_RSU; | |
| 1935 | − spin_lock_init(&chans[1].lock); | |
| 1936 | − | |
| 1937 | − chans[2].scl = NULL; | |
| 1938 | − chans[2].ctrl = controller; | |
| 1939 | − chans[2].name = SVC_CLIENT_FCS; | |
| 1940 | − spin_lock_init(&chans[2].lock); | |
| 1941 | − | |
| 1942 | − chans[3].scl = NULL; | |
| 1943 | − chans[3].ctrl = controller; | |
| 1944 | − chans[3].name = SVC_CLIENT_HWMON; | |
| 1945 | − spin_lock_init(&chans[3].lock); | |
| 1964 | + for (i = 0; i < SVC_NUM_CHANNEL; i++) { | |
| 1965 | + chans[i].scl = NULL; | |
| 1966 | + chans[i].ctrl = controller; | |
| 1967 | + chans[i].name = (char *)chan_names[i]; | |
| 1968 | + spin_lock_init(&chans[i].lock); | |
| 1969 | + ret = kfifo_alloc(&chans[i].svc_fifo, fifo_size, GFP_KERNEL); | |
| 1970 | + if (ret) { | |
| 1971 | + dev_err(dev, "failed to allocate FIFO %d\n", i); | |
| 1972 | + goto err_free_fifos; | |
| 1973 | + } | |
| 1974 | + spin_lock_init(&chans[i].svc_fifo_lock); | |
| 1975 | + } | |
| 1946 | 1976 | |
| 1947 | 1977 | list_add_tail(&controller->node, &svc_ctrl); |
| 1948 | 1978 | platform_set_drvdata(pdev, controller); |
| @@ -1951,7 +1981,7 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 2 unmodified lines | ||
| 1951 | 1981 | svc = devm_kzalloc(dev, sizeof(*svc), GFP_KERNEL); |
| 1952 | 1982 | if (!svc) { |
| 1953 | 1983 | ret = -ENOMEM; |
| 1954 | − goto err_free_kfifo; | |
| 1984 | + goto err_free_fifos; | |
| 1955 | 1985 | } |
| 1956 | 1986 | controller->svc = svc; |
| 1957 | 1987 | |
| @@ -1959,51 +1989,43 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 1 unmodified line | ||
| 1959 | 1989 | if (!svc->stratix10_svc_rsu) { |
| 1960 | 1990 | dev_err(dev, "failed to allocate %s device\n", STRATIX10_RSU); |
| 1961 | 1991 | ret = -ENOMEM; |
| 1962 | − goto err_free_kfifo; | |
| 1992 | + goto err_free_fifos; | |
| 1963 | 1993 | } |
| 1964 | 1994 | |
| 1965 | 1995 | ret = platform_device_add(svc->stratix10_svc_rsu); |
| 1966 | − if (ret) { | |
| 1967 | − platform_device_put(svc->stratix10_svc_rsu); | |
| 1968 | − goto err_free_kfifo; | |
| 1969 | − } | |
| 1970 | − | |
| 1971 | − svc->intel_svc_fcs = platform_device_alloc(INTEL_FCS, 1); | |
| 1972 | − if (!svc->intel_svc_fcs) { | |
| 1973 | − dev_err(dev, "failed to allocate %s device\n", INTEL_FCS); | |
| 1974 | − ret = -ENOMEM; | |
| 1975 | − goto err_unregister_rsu_dev; | |
| 1976 | − } | |
| 1977 | − | |
| 1978 | − ret = platform_device_add(svc->intel_svc_fcs); | |
| 1979 | − if (ret) { | |
| 1980 | − platform_device_put(svc->intel_svc_fcs); | |
| 1981 | − goto err_unregister_rsu_dev; | |
| 1982 | − } | |
| 1996 | + if (ret) | |
| 1997 | + goto err_put_device; | |
| 1983 | 1998 | |
| 1984 | 1999 | ret = of_platform_default_populate(dev_of_node(dev), NULL, dev); |
| 1985 | 2000 | if (ret) |
| 1986 | − goto err_unregister_fcs_dev; | |
| 2001 | + goto err_unregister_rsu_dev; | |
| 1987 | 2002 | |
| 1988 | 2003 | pr_info("Intel Service Layer Driver Initialized\n"); |
| 1989 | 2004 | |
| 1990 | 2005 | return 0; |
| 1991 | 2006 | |
| 1992 | −err_unregister_fcs_dev: | |
| 1993 | − platform_device_unregister(svc->intel_svc_fcs); | |
| 1994 | 2007 | err_unregister_rsu_dev: |
| 1995 | 2008 | platform_device_unregister(svc->stratix10_svc_rsu); |
| 1996 | −err_free_kfifo: | |
| 1997 | − kfifo_free(&controller->svc_fifo); | |
| 1998 | −err_async_exit: | |
| 2009 | + goto err_free_fifos; | |
| 2010 | +err_put_device: | |
| 2011 | + platform_device_put(svc->stratix10_svc_rsu); | |
| 2012 | +err_free_fifos: | |
| 2013 | + /* only remove from list if list_add_tail() was reached */ | |
| 2014 | + if (!list_empty(&controller->node)) | |
| 2015 | + list_del(&controller->node); | |
| 2016 | + /* free only the FIFOs that were successfully allocated */ | |
| 2017 | + while (i--) | |
| 2018 | + kfifo_free(&chans[i].svc_fifo); | |
| 1999 | 2019 | stratix10_svc_async_exit(controller); |
| 2000 | 2020 | err_destroy_pool: |
| 2001 | 2021 | gen_pool_destroy(genpool); |
| 2022 | + | |
| 2002 | 2023 | return ret; |
| 2003 | 2024 | } |
| 2004 | 2025 | |
| 2005 | 2026 | static void stratix10_svc_drv_remove(struct platform_device *pdev) |
| 2006 | 2027 | { |
| 2028 | + int i; | |
| 2007 | 2029 | struct stratix10_svc_controller *ctrl = platform_get_drvdata(pdev); |
| 2008 | 2030 | struct stratix10_svc *svc = ctrl->svc; |
| 2009 | 2031 | |
| @@ -2011,14 +2033,16 @@ static void stratix10_svc_drv_remove(struct platform_device *pdev) · 1 unmodified line | ||
| 2011 | 2033 | |
| 2012 | 2034 | of_platform_depopulate(ctrl->dev); |
| 2013 | 2035 | |
| 2014 | − platform_device_unregister(svc->intel_svc_fcs); | |
| 2015 | 2036 | platform_device_unregister(svc->stratix10_svc_rsu); |
| 2016 | 2037 | |
| 2017 | − kfifo_free(&ctrl->svc_fifo); | |
| 2018 | − if (ctrl->task) { | |
| 2019 | − kthread_stop(ctrl->task); | |
| 2020 | − ctrl->task = NULL; | |
| 2038 | + for (i = 0; i < SVC_NUM_CHANNEL; i++) { | |
| 2039 | + if (ctrl->chans[i].task) { | |
| 2040 | + kthread_stop(ctrl->chans[i].task); | |
| 2041 | + ctrl->chans[i].task = NULL; | |
| 2042 | + } | |
| 2043 | + kfifo_free(&ctrl->chans[i].svc_fifo); | |
| 2021 | 2044 | } |
| 2045 | + | |
| 2022 | 2046 | if (ctrl->genpool) |
| 2023 | 2047 | gen_pool_destroy(ctrl->genpool); |
| 2024 | 2048 | list_del(&ctrl->node); |
| @@ -122,6 +122,7 @@ config GPIB_FLUKE · 121 unmodified lines | ||
| 122 | 122 | depends on OF |
| 123 | 123 | select GPIB_COMMON |
| 124 | 124 | select GPIB_NEC7210 |
| 125 | + depends on HAS_IOMEM | |
| 125 | 126 | help |
| 126 | 127 | GPIB driver for Fluke based cda devices. |
| 127 | 128 | |
| @@ -888,10 +888,6 @@ static int read_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 887 unmodified lines | ||
| 888 | 888 | if (read_cmd.completed_transfer_count > read_cmd.requested_transfer_count) |
| 889 | 889 | return -EINVAL; |
| 890 | 890 | |
| 891 | − desc = handle_to_descriptor(file_priv, read_cmd.handle); | |
| 892 | − if (!desc) | |
| 893 | − return -EINVAL; | |
| 894 | − | |
| 895 | 891 | if (WARN_ON_ONCE(sizeof(userbuf) > sizeof(read_cmd.buffer_ptr))) |
| 896 | 892 | return -EFAULT; |
| 897 | 893 | |
| @@ -904,6 +900,17 @@ static int read_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 6 unmodified lines | ||
| 904 | 900 | if (!access_ok(userbuf, remain)) |
| 905 | 901 | return -EFAULT; |
| 906 | 902 | |
| 903 | + /* Lock descriptors to prevent concurrent close from freeing descriptor */ | |
| 904 | + if (mutex_lock_interruptible(&file_priv->descriptors_mutex)) | |
| 905 | + return -ERESTARTSYS; | |
| 906 | + desc = handle_to_descriptor(file_priv, read_cmd.handle); | |
| 907 | + if (!desc) { | |
| 908 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 909 | + return -EINVAL; | |
| 910 | + } | |
| 911 | + atomic_inc(&desc->descriptor_busy); | |
| 912 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 913 | + | |
| 907 | 914 | atomic_set(&desc->io_in_progress, 1); |
| 908 | 915 | |
| 909 | 916 | /* Read buffer loads till we fill the user supplied buffer */ |
| @@ -937,6 +944,7 @@ static int read_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 27 unmodified lines | ||
| 937 | 944 | retval = copy_to_user((void __user *)arg, &read_cmd, sizeof(read_cmd)); |
| 938 | 945 | |
| 939 | 946 | atomic_set(&desc->io_in_progress, 0); |
| 947 | + atomic_dec(&desc->descriptor_busy); | |
| 940 | 948 | |
| 941 | 949 | wake_up_interruptible(&board->wait); |
| 942 | 950 | if (retval) |
| @@ -964,10 +972,6 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 21 unmodified lines | ||
| 964 | 972 | if (cmd.completed_transfer_count > cmd.requested_transfer_count) |
| 965 | 973 | return -EINVAL; |
| 966 | 974 | |
| 967 | − desc = handle_to_descriptor(file_priv, cmd.handle); | |
| 968 | − if (!desc) | |
| 969 | − return -EINVAL; | |
| 970 | − | |
| 971 | 975 | userbuf = (u8 __user *)(unsigned long)cmd.buffer_ptr; |
| 972 | 976 | userbuf += cmd.completed_transfer_count; |
| 973 | 977 | |
| @@ -980,6 +984,17 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 6 unmodified lines | ||
| 980 | 984 | if (!access_ok(userbuf, remain)) |
| 981 | 985 | return -EFAULT; |
| 982 | 986 | |
| 987 | + /* Lock descriptors to prevent concurrent close from freeing descriptor */ | |
| 988 | + if (mutex_lock_interruptible(&file_priv->descriptors_mutex)) | |
| 989 | + return -ERESTARTSYS; | |
| 990 | + desc = handle_to_descriptor(file_priv, cmd.handle); | |
| 991 | + if (!desc) { | |
| 992 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 993 | + return -EINVAL; | |
| 994 | + } | |
| 995 | + atomic_inc(&desc->descriptor_busy); | |
| 996 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 997 | + | |
| 983 | 998 | /* |
| 984 | 999 | * Write buffer loads till we empty the user supplied buffer. |
| 985 | 1000 | * Call drivers at least once, even if remain is zero, in |
| @@ -1003,6 +1018,7 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 17 unmodified lines | ||
| 1003 | 1018 | userbuf += bytes_written; |
| 1004 | 1019 | if (retval < 0) { |
| 1005 | 1020 | atomic_set(&desc->io_in_progress, 0); |
| 1021 | + atomic_dec(&desc->descriptor_busy); | |
| 1006 | 1022 | |
| 1007 | 1023 | wake_up_interruptible(&board->wait); |
| 1008 | 1024 | break; |
| @@ -1022,6 +1038,7 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 13 unmodified lines | ||
| 1022 | 1038 | */ |
| 1023 | 1039 | if (!no_clear_io_in_prog || fault) |
| 1024 | 1040 | atomic_set(&desc->io_in_progress, 0); |
| 1041 | + atomic_dec(&desc->descriptor_busy); | |
| 1025 | 1042 | |
| 1026 | 1043 | wake_up_interruptible(&board->wait); |
| 1027 | 1044 | if (fault) |
| @@ -1047,10 +1064,6 @@ static int write_ioctl(struct gpib_file_private *file_priv, struct gpib_board *b · 19 unmodified lines | ||
| 1047 | 1064 | if (write_cmd.completed_transfer_count > write_cmd.requested_transfer_count) |
| 1048 | 1065 | return -EINVAL; |
| 1049 | 1066 | |
| 1050 | − desc = handle_to_descriptor(file_priv, write_cmd.handle); | |
| 1051 | − if (!desc) | |
| 1052 | − return -EINVAL; | |
| 1053 | − | |
| 1054 | 1067 | userbuf = (u8 __user *)(unsigned long)write_cmd.buffer_ptr; |
| 1055 | 1068 | userbuf += write_cmd.completed_transfer_count; |
| 1056 | 1069 | |
| @@ -1060,6 +1073,17 @@ static int write_ioctl(struct gpib_file_private *file_priv, struct gpib_board *b · 3 unmodified lines | ||
| 1060 | 1073 | if (!access_ok(userbuf, remain)) |
| 1061 | 1074 | return -EFAULT; |
| 1062 | 1075 | |
| 1076 | + /* Lock descriptors to prevent concurrent close from freeing descriptor */ | |
| 1077 | + if (mutex_lock_interruptible(&file_priv->descriptors_mutex)) | |
| 1078 | + return -ERESTARTSYS; | |
| 1079 | + desc = handle_to_descriptor(file_priv, write_cmd.handle); | |
| 1080 | + if (!desc) { | |
| 1081 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1082 | + return -EINVAL; | |
| 1083 | + } | |
| 1084 | + atomic_inc(&desc->descriptor_busy); | |
| 1085 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1086 | + | |
| 1063 | 1087 | atomic_set(&desc->io_in_progress, 1); |
| 1064 | 1088 | |
| 1065 | 1089 | /* Write buffer loads till we empty the user supplied buffer */ |
| @@ -1094,6 +1118,7 @@ static int write_ioctl(struct gpib_file_private *file_priv, struct gpib_board *b · 28 unmodified lines | ||
| 1094 | 1118 | fault = copy_to_user((void __user *)arg, &write_cmd, sizeof(write_cmd)); |
| 1095 | 1119 | |
| 1096 | 1120 | atomic_set(&desc->io_in_progress, 0); |
| 1121 | + atomic_dec(&desc->descriptor_busy); | |
| 1097 | 1122 | |
| 1098 | 1123 | wake_up_interruptible(&board->wait); |
| 1099 | 1124 | if (fault) |
| @@ -1276,6 +1301,9 @@ static int close_dev_ioctl(struct file *filep, struct gpib_board *board, unsigne · 176 unmodified lines | ||
| 1276 | 1301 | { |
| 1277 | 1302 | struct gpib_close_dev_ioctl cmd; |
| 1278 | 1303 | struct gpib_file_private *file_priv = filep->private_data; |
| 1304 | + struct gpib_descriptor *desc; | |
| 1305 | + unsigned int pad; | |
| 1306 | + int sad; | |
| 1279 | 1307 | int retval; |
| 1280 | 1308 | |
| 1281 | 1309 | retval = copy_from_user(&cmd, (void __user *)arg, sizeof(cmd)); |
| @@ -1284,19 +1312,27 @@ static int close_dev_ioctl(struct file *filep, struct gpib_board *board, unsigne · 2 unmodified lines | ||
| 1284 | 1312 | |
| 1285 | 1313 | if (cmd.handle >= GPIB_MAX_NUM_DESCRIPTORS) |
| 1286 | 1314 | return -EINVAL; |
| 1287 | − if (!file_priv->descriptors[cmd.handle]) | |
| 1288 | − return -EINVAL; | |
| 1289 | 1315 | |
| 1290 | − retval = decrement_open_device_count(board, &board->device_list, | |
| 1291 | − file_priv->descriptors[cmd.handle]->pad, | |
| 1292 | − file_priv->descriptors[cmd.handle]->sad); | |
| 1293 | − if (retval < 0) | |
| 1294 | − return retval; | |
| 1295 | − | |
| 1296 | − kfree(file_priv->descriptors[cmd.handle]); | |
| 1316 | + mutex_lock(&file_priv->descriptors_mutex); | |
| 1317 | + desc = file_priv->descriptors[cmd.handle]; | |
| 1318 | + if (!desc) { | |
| 1319 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1320 | + return -EINVAL; | |
| 1321 | + } | |
| 1322 | + if (atomic_read(&desc->descriptor_busy)) { | |
| 1323 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1324 | + return -EBUSY; | |
| 1325 | + } | |
| 1326 | + /* Remove from table while holding lock to prevent new IO from starting */ | |
| 1297 | 1327 | file_priv->descriptors[cmd.handle] = NULL; |
| 1328 | + pad = desc->pad; | |
| 1329 | + sad = desc->sad; | |
| 1330 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1298 | 1331 | |
| 1299 | − return 0; | |
| 1332 | + retval = decrement_open_device_count(board, &board->device_list, pad, sad); | |
| 1333 | + | |
| 1334 | + kfree(desc); | |
| 1335 | + return retval; | |
| 1300 | 1336 | } |
| 1301 | 1337 | |
| 1302 | 1338 | static int serial_poll_ioctl(struct gpib_board *board, unsigned long arg) |
| @@ -1331,12 +1367,25 @@ static int wait_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 28 unmodified lines | ||
| 1331 | 1367 | if (retval) |
| 1332 | 1368 | return -EFAULT; |
| 1333 | 1369 | |
| 1370 | + /* | |
| 1371 | + * Lock descriptors to prevent concurrent close from freeing | |
| 1372 | + * descriptor. ibwait() releases big_gpib_mutex when wait_mask | |
| 1373 | + * is non-zero, so desc must be pinned with descriptor_busy. | |
| 1374 | + */ | |
| 1375 | + mutex_lock(&file_priv->descriptors_mutex); | |
| 1334 | 1376 | desc = handle_to_descriptor(file_priv, wait_cmd.handle); |
| 1335 | − if (!desc) | |
| 1377 | + if (!desc) { | |
| 1378 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1336 | 1379 | return -EINVAL; |
| 1380 | + } | |
| 1381 | + atomic_inc(&desc->descriptor_busy); | |
| 1382 | + mutex_unlock(&file_priv->descriptors_mutex); | |
| 1337 | 1383 | |
| 1338 | 1384 | retval = ibwait(board, wait_cmd.wait_mask, wait_cmd.clear_mask, |
| 1339 | 1385 | wait_cmd.set_mask, &wait_cmd.ibsta, wait_cmd.usec_timeout, desc); |
| 1386 | + | |
| 1387 | + atomic_dec(&desc->descriptor_busy); | |
| 1388 | + | |
| 1340 | 1389 | if (retval < 0) |
| 1341 | 1390 | return retval; |
| 1342 | 1391 | |
| @@ -2035,6 +2084,7 @@ void init_gpib_descriptor(struct gpib_descriptor *desc) · 692 unmodified lines | ||
| 2035 | 2084 | desc->is_board = 0; |
| 2036 | 2085 | desc->autopoll_enabled = 0; |
| 2037 | 2086 | atomic_set(&desc->io_in_progress, 0); |
| 2087 | + atomic_set(&desc->descriptor_busy, 0); | |
| 2038 | 2088 | } |
| 2039 | 2089 | |
| 2040 | 2090 | int gpib_register_driver(struct gpib_interface *interface, struct module *provider_module) |
| @@ -364,6 +364,14 @@ struct gpib_descriptor { · 363 unmodified lines | ||
| 364 | 364 | unsigned int pad; /* primary gpib address */ |
| 365 | 365 | int sad; /* secondary gpib address (negative means disabled) */ |
| 366 | 366 | atomic_t io_in_progress; |
| 367 | + /* | |
| 368 | + * Kernel-only reference count to prevent descriptor from being | |
| 369 | + * freed while IO handlers hold a pointer to it. Incremented | |
| 370 | + * before each IO operation, decremented when done. Unlike | |
| 371 | + * io_in_progress, this cannot be modified from userspace via | |
| 372 | + * general_ibstatus(). | |
| 373 | + */ | |
| 374 | + atomic_t descriptor_busy; | |
| 367 | 375 | unsigned is_board : 1; |
| 368 | 376 | unsigned autopoll_enabled : 1; |
| 369 | 377 | }; |
| @@ -38,8 +38,10 @@ MODULE_DESCRIPTION("GPIB driver for LPVO usb devices"); · 37 unmodified lines | ||
| 38 | 38 | /* |
| 39 | 39 | * Table of devices that work with this driver. |
| 40 | 40 | * |
| 41 | − * Currently, only one device is known to be used in the | |
| 42 | − * lpvo_usb_gpib adapter (FTDI 0403:6001). | |
| 41 | + * Currently, only one device is known to be used in the lpvo_usb_gpib | |
| 42 | + * adapter (FTDI 0403:6001) but as this device id is already handled by the | |
| 43 | + * ftdi_sio USB serial driver the LPVO driver must not bind to it by default. | |
| 44 | + * | |
| 43 | 45 | * If your adapter uses a different chip, insert a line |
| 44 | 46 | * in the following table with proper <Vendor-id>, <Product-id>. |
| 45 | 47 | * |
| @@ -50,7 +52,6 @@ MODULE_DESCRIPTION("GPIB driver for LPVO usb devices"); · 4 unmodified lines | ||
| 50 | 52 | */ |
| 51 | 53 | |
| 52 | 54 | static const struct usb_device_id skel_table[] = { |
| 53 | − { USB_DEVICE(0x0403, 0x6001) }, | |
| 54 | 55 | { } /* Terminating entry */ |
| 55 | 56 | }; |
| 56 | 57 | MODULE_DEVICE_TABLE(usb, skel_table); |
| @@ -405,7 +406,7 @@ static int usb_gpib_attach(struct gpib_board *board, const struct gpib_board_con · 348 unmodified lines | ||
| 405 | 406 | for (j = 0 ; j < MAX_DEV ; j++) { |
| 406 | 407 | if ((assigned_usb_minors & 1 << j) == 0) |
| 407 | 408 | continue; |
| 408 | − udev = usb_get_dev(interface_to_usbdev(lpvo_usb_interfaces[j])); | |
| 409 | + udev = interface_to_usbdev(lpvo_usb_interfaces[j]); | |
| 409 | 410 | device_path = kobject_get_path(&udev->dev.kobj, GFP_KERNEL); |
| 410 | 411 | match = gpib_match_device_path(&lpvo_usb_interfaces[j]->dev, |
| 411 | 412 | config->device_path); |
| @@ -420,7 +421,7 @@ static int usb_gpib_attach(struct gpib_board *board, const struct gpib_board_con · 8 unmodified lines | ||
| 420 | 421 | for (j = 0 ; j < MAX_DEV ; j++) { |
| 421 | 422 | if ((assigned_usb_minors & 1 << j) == 0) |
| 422 | 423 | continue; |
| 423 | − udev = usb_get_dev(interface_to_usbdev(lpvo_usb_interfaces[j])); | |
| 424 | + udev = interface_to_usbdev(lpvo_usb_interfaces[j]); | |
| 424 | 425 | DIA_LOG(1, "dev. %d: bus %d -> %d dev: %d -> %d\n", j, |
| 425 | 426 | udev->bus->busnum, config->pci_bus, udev->devnum, config->pci_slot); |
| 426 | 427 | if (config->pci_bus == udev->bus->busnum && |
| @@ -584,12 +584,13 @@ static bool mxc_gpio_set_pad_wakeup(struct mxc_gpio_port *port, bool enable) · 583 unmodified lines | ||
| 584 | 584 | unsigned long config; |
| 585 | 585 | bool ret = false; |
| 586 | 586 | int i, type; |
| 587 | + bool is_imx8qm = of_device_is_compatible(port->dev->of_node, "fsl,imx8qm-gpio"); | |
| 587 | 588 | |
| 588 | 589 | static const u32 pad_type_map[] = { |
| 589 | 590 | IMX_SCU_WAKEUP_OFF, /* 0 */ |
| 590 | 591 | IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_RISING */ |
| 591 | 592 | IMX_SCU_WAKEUP_FALL_EDGE, /* IRQ_TYPE_EDGE_FALLING */ |
| 592 | − IMX_SCU_WAKEUP_FALL_EDGE, /* IRQ_TYPE_EDGE_BOTH */ | |
| 593 | + IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_BOTH */ | |
| 593 | 594 | IMX_SCU_WAKEUP_HIGH_LVL, /* IRQ_TYPE_LEVEL_HIGH */ |
| 594 | 595 | IMX_SCU_WAKEUP_OFF, /* 5 */ |
| 595 | 596 | IMX_SCU_WAKEUP_OFF, /* 6 */ |
| @@ -604,6 +605,13 @@ static bool mxc_gpio_set_pad_wakeup(struct mxc_gpio_port *port, bool enable) · 8 unmodified lines | ||
| 604 | 605 | config = pad_type_map[type]; |
| 605 | 606 | else |
| 606 | 607 | config = IMX_SCU_WAKEUP_OFF; |
| 608 | + | |
| 609 | + if (is_imx8qm && config == IMX_SCU_WAKEUP_FALL_EDGE) { | |
| 610 | + dev_warn_once(port->dev, | |
| 611 | + "No falling-edge support for wakeup on i.MX8QM\n"); | |
| 612 | + config = IMX_SCU_WAKEUP_OFF; | |
| 613 | + } | |
| 614 | + | |
| 607 | 615 | ret |= mxc_gpio_generic_config(port, i, config); |
| 608 | 616 | } |
| 609 | 617 | } |
| @@ -60,8 +60,8 @@ static int qixis_cpld_gpio_probe(struct platform_device *pdev) · 59 unmodified lines | ||
| 60 | 60 | return PTR_ERR(reg); |
| 61 | 61 | |
| 62 | 62 | regmap = devm_regmap_init_mmio(&pdev->dev, reg, ®map_config_8r_8v); |
| 63 | − if (!regmap) | |
| 64 | − return -ENODEV; | |
| 63 | + if (IS_ERR(regmap)) | |
| 64 | + return PTR_ERR(regmap); | |
| 65 | 65 | |
| 66 | 66 | /* In this case, the offset of our register is 0 inside the |
| 67 | 67 | * regmap area that we just created. |
| @@ -443,8 +443,8 @@ static bool gpio_shared_dev_is_reset_gpio(struct device *consumer, · 442 unmodified lines | ||
| 443 | 443 | } |
| 444 | 444 | #endif /* CONFIG_RESET_GPIO */ |
| 445 | 445 | |
| 446 | −int gpio_shared_add_proxy_lookup(struct device *consumer, const char *con_id, | |
| 447 | − unsigned long lflags) | |
| 446 | +int gpio_shared_add_proxy_lookup(struct device *consumer, struct fwnode_handle *fwnode, | |
| 447 | + const char *con_id, unsigned long lflags) | |
| 448 | 448 | { |
| 449 | 449 | const char *dev_id = dev_name(consumer); |
| 450 | 450 | struct gpiod_lookup_table *lookup; |
| @@ -458,7 +458,7 @@ int gpio_shared_add_proxy_lookup(struct device *consumer, const char *con_id, · 7 unmodified lines | ||
| 458 | 458 | if (!ref->fwnode && device_is_compatible(consumer, "reset-gpio")) { |
| 459 | 459 | if (!gpio_shared_dev_is_reset_gpio(consumer, entry, ref)) |
| 460 | 460 | continue; |
| 461 | − } else if (!device_match_fwnode(consumer, ref->fwnode)) { | |
| 461 | + } else if (fwnode != ref->fwnode) { | |
| 462 | 462 | continue; |
| 463 | 463 | } |
| 464 | 464 | |
| @@ -506,8 +506,9 @@ static void gpio_shared_remove_adev(struct auxiliary_device *adev) · 41 unmodified lines | ||
| 506 | 506 | auxiliary_device_uninit(adev); |
| 507 | 507 | } |
| 508 | 508 | |
| 509 | −int gpio_device_setup_shared(struct gpio_device *gdev) | |
| 509 | +int gpiochip_setup_shared(struct gpio_chip *gc) | |
| 510 | 510 | { |
| 511 | + struct gpio_device *gdev = gc->gpiodev; | |
| 511 | 512 | struct gpio_shared_entry *entry; |
| 512 | 513 | struct gpio_shared_ref *ref; |
| 513 | 514 | struct gpio_desc *desc; |
| @@ -538,20 +539,42 @@ int gpio_device_setup_shared(struct gpio_device *gdev) · 24 unmodified lines | ||
| 538 | 539 | if (list_count_nodes(&entry->refs) <= 1) |
| 539 | 540 | continue; |
| 540 | 541 | |
| 541 | − desc = &gdev->descs[entry->offset]; | |
| 542 | + scoped_guard(mutex, &entry->lock) { | |
| 543 | +#if IS_ENABLED(CONFIG_OF) | |
| 544 | + if (is_of_node(entry->fwnode) && gc->of_xlate) { | |
| 545 | + /* | |
| 546 | + * This is the earliest that we can tranlate the | |
| 547 | + * devicetree offset to the chip offset. | |
| 548 | + */ | |
| 549 | + struct of_phandle_args gpiospec = { }; | |
| 542 | 550 | |
| 543 | − __set_bit(GPIOD_FLAG_SHARED, &desc->flags); | |
| 544 | − /* | |
| 545 | − * Shared GPIOs are not requested via the normal path. Make | |
| 546 | − * them inaccessible to anyone even before we register the | |
| 547 | − * chip. | |
| 548 | − */ | |
| 549 | − ret = gpiod_request_commit(desc, "shared"); | |
| 550 | − if (ret) | |
| 551 | − return ret; | |
| 551 | + gpiospec.np = to_of_node(entry->fwnode); | |
| 552 | + gpiospec.args_count = 2; | |
| 553 | + gpiospec.args[0] = entry->offset; | |
| 554 | + | |
| 555 | + ret = gc->of_xlate(gc, &gpiospec, NULL); | |
| 556 | + if (ret < 0) | |
| 557 | + return ret; | |
| 558 | + | |
| 559 | + entry->offset = ret; | |
| 560 | + } | |
| 561 | +#endif /* CONFIG_OF */ | |
| 562 | + | |
| 563 | + desc = &gdev->descs[entry->offset]; | |
| 564 | + | |
| 565 | + __set_bit(GPIOD_FLAG_SHARED, &desc->flags); | |
| 566 | + /* | |
| 567 | + * Shared GPIOs are not requested via the normal path. Make | |
| 568 | + * them inaccessible to anyone even before we register the | |
| 569 | + * chip. | |
| 570 | + */ | |
| 571 | + ret = gpiod_request_commit(desc, "shared"); | |
| 572 | + if (ret) | |
| 573 | + return ret; | |
| 552 | 574 | |
| 553 | − pr_debug("GPIO %u owned by %s is shared by multiple consumers\n", | |
| 554 | − entry->offset, gpio_device_get_label(gdev)); | |
| 575 | + pr_debug("GPIO %u owned by %s is shared by multiple consumers\n", | |
| 576 | + entry->offset, gpio_device_get_label(gdev)); | |
| 577 | + } | |
| 555 | 578 | |
| 556 | 579 | list_for_each_entry(ref, &entry->refs, list) { |
| 557 | 580 | pr_debug("Setting up a shared GPIO entry for %s (con_id: '%s')\n", |
| @@ -575,6 +598,8 @@ void gpio_device_teardown_shared(struct gpio_device *gdev) · 17 unmodified lines | ||
| 575 | 598 | struct gpio_shared_ref *ref; |
| 576 | 599 | |
| 577 | 600 | list_for_each_entry(entry, &gpio_shared_list, list) { |
| 601 | + guard(mutex)(&entry->lock); | |
| 602 | + | |
| 578 | 603 | if (!device_match_fwnode(&gdev->dev, entry->fwnode)) |
| 579 | 604 | continue; |
| 580 | 605 | |
| @@ -11,17 +11,19 @@ · 10 unmodified lines | ||
| 11 | 11 | struct gpio_device; |
| 12 | 12 | struct gpio_desc; |
| 13 | 13 | struct device; |
| 14 | +struct fwnode_handle; | |
| 14 | 15 | |
| 15 | 16 | #if IS_ENABLED(CONFIG_GPIO_SHARED) |
| 16 | 17 | |
| 17 | −int gpio_device_setup_shared(struct gpio_device *gdev); | |
| 18 | +int gpiochip_setup_shared(struct gpio_chip *gc); | |
| 18 | 19 | void gpio_device_teardown_shared(struct gpio_device *gdev); |
| 19 | −int gpio_shared_add_proxy_lookup(struct device *consumer, const char *con_id, | |
| 20 | − unsigned long lflags); | |
| 20 | +int gpio_shared_add_proxy_lookup(struct device *consumer, | |
| 21 | + struct fwnode_handle *fwnode, | |
| 22 | + const char *con_id, unsigned long lflags); | |
| 21 | 23 | |
| 22 | 24 | #else |
| 23 | 25 | |
| 24 | −static inline int gpio_device_setup_shared(struct gpio_device *gdev) | |
| 26 | +static inline int gpiochip_setup_shared(struct gpio_chip *gc) | |
| 25 | 27 | { |
| 26 | 28 | return 0; |
| 27 | 29 | } |
| @@ -29,6 +31,7 @@ static inline int gpio_device_setup_shared(struct gpio_device *gdev) · 1 unmodified line | ||
| 29 | 31 | static inline void gpio_device_teardown_shared(struct gpio_device *gdev) { } |
| 30 | 32 | |
| 31 | 33 | static inline int gpio_shared_add_proxy_lookup(struct device *consumer, |
| 34 | + struct fwnode_handle *fwnode, | |
| 32 | 35 | const char *con_id, |
| 33 | 36 | unsigned long lflags) |
| 34 | 37 | { |
| @@ -892,13 +892,15 @@ static const struct device_type gpio_dev_type = { · 891 unmodified lines | ||
| 892 | 892 | #define gcdev_unregister(gdev) device_del(&(gdev)->dev) |
| 893 | 893 | #endif |
| 894 | 894 | |
| 895 | +/* | |
| 896 | + * An initial reference count has been held in gpiochip_add_data_with_key(). | |
| 897 | + * The caller should drop the reference via gpio_device_put() on errors. | |
| 898 | + */ | |
| 895 | 899 | static int gpiochip_setup_dev(struct gpio_device *gdev) |
| 896 | 900 | { |
| 897 | 901 | struct fwnode_handle *fwnode = dev_fwnode(&gdev->dev); |
| 898 | 902 | int ret; |
| 899 | 903 | |
| 900 | − device_initialize(&gdev->dev); | |
| 901 | − | |
| 902 | 904 | /* |
| 903 | 905 | * If fwnode doesn't belong to another device, it's safe to clear its |
| 904 | 906 | * initialized flag. |
| @@ -964,9 +966,11 @@ static void gpiochip_setup_devs(void) · 59 unmodified lines | ||
| 964 | 966 | list_for_each_entry_srcu(gdev, &gpio_devices, list, |
| 965 | 967 | srcu_read_lock_held(&gpio_devices_srcu)) { |
| 966 | 968 | ret = gpiochip_setup_dev(gdev); |
| 967 | − if (ret) | |
| 969 | + if (ret) { | |
| 970 | + gpio_device_put(gdev); | |
| 968 | 971 | dev_err(&gdev->dev, |
| 969 | 972 | "Failed to initialize gpio device (%d)\n", ret); |
| 973 | + } | |
| 970 | 974 | } |
| 971 | 975 | } |
| 972 | 976 | |
| @@ -1047,71 +1051,72 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 74 unmodified lines | ||
| 1047 | 1051 | int base = 0; |
| 1048 | 1052 | int ret; |
| 1049 | 1053 | |
| 1050 | − /* | |
| 1051 | − * First: allocate and populate the internal stat container, and | |
| 1052 | − * set up the struct device. | |
| 1053 | − */ | |
| 1054 | 1054 | gdev = kzalloc(sizeof(*gdev), GFP_KERNEL); |
| 1055 | 1055 | if (!gdev) |
| 1056 | 1056 | return -ENOMEM; |
| 1057 | − | |
| 1058 | − gdev->dev.type = &gpio_dev_type; | |
| 1059 | − gdev->dev.bus = &gpio_bus_type; | |
| 1060 | − gdev->dev.parent = gc->parent; | |
| 1061 | − rcu_assign_pointer(gdev->chip, gc); | |
| 1062 | − | |
| 1063 | 1057 | gc->gpiodev = gdev; |
| 1064 | 1058 | gpiochip_set_data(gc, data); |
| 1065 | 1059 | |
| 1066 | − device_set_node(&gdev->dev, gpiochip_choose_fwnode(gc)); | |
| 1067 | − | |
| 1068 | 1060 | ret = ida_alloc(&gpio_ida, GFP_KERNEL); |
| 1069 | 1061 | if (ret < 0) |
| 1070 | 1062 | goto err_free_gdev; |
| 1071 | 1063 | gdev->id = ret; |
| 1072 | 1064 | |
| 1073 | − ret = dev_set_name(&gdev->dev, GPIOCHIP_NAME "%d", gdev->id); | |
| 1065 | + ret = init_srcu_struct(&gdev->srcu); | |
| 1074 | 1066 | if (ret) |
| 1075 | 1067 | goto err_free_ida; |
| 1068 | + rcu_assign_pointer(gdev->chip, gc); | |
| 1076 | 1069 | |
| 1077 | − if (gc->parent && gc->parent->driver) | |
| 1078 | − gdev->owner = gc->parent->driver->owner; | |
| 1079 | − else if (gc->owner) | |
| 1080 | − /* TODO: remove chip->owner */ | |
| 1081 | − gdev->owner = gc->owner; | |
| 1082 | − else | |
| 1083 | − gdev->owner = THIS_MODULE; | |
| 1070 | + ret = init_srcu_struct(&gdev->desc_srcu); | |
| 1071 | + if (ret) | |
| 1072 | + goto err_cleanup_gdev_srcu; | |
| 1073 | + | |
| 1074 | + ret = dev_set_name(&gdev->dev, GPIOCHIP_NAME "%d", gdev->id); | |
| 1075 | + if (ret) | |
| 1076 | + goto err_cleanup_desc_srcu; | |
| 1077 | + | |
| 1078 | + device_initialize(&gdev->dev); | |
| 1079 | + /* | |
| 1080 | + * After this point any allocated resources to `gdev` will be | |
| 1081 | + * free():ed by gpiodev_release(). If you add new resources | |
| 1082 | + * then make sure they get free():ed there. | |
| 1083 | + */ | |
| 1084 | + gdev->dev.type = &gpio_dev_type; | |
| 1085 | + gdev->dev.bus = &gpio_bus_type; | |
| 1086 | + gdev->dev.parent = gc->parent; | |
| 1087 | + device_set_node(&gdev->dev, gpiochip_choose_fwnode(gc)); | |
| 1084 | 1088 | |
| 1085 | 1089 | ret = gpiochip_get_ngpios(gc, &gdev->dev); |
| 1086 | 1090 | if (ret) |
| 1087 | − goto err_free_dev_name; | |
| 1091 | + goto err_put_device; | |
| 1092 | + gdev->ngpio = gc->ngpio; | |
| 1088 | 1093 | |
| 1089 | 1094 | gdev->descs = kcalloc(gc->ngpio, sizeof(*gdev->descs), GFP_KERNEL); |
| 1090 | 1095 | if (!gdev->descs) { |
| 1091 | 1096 | ret = -ENOMEM; |
| 1092 | − goto err_free_dev_name; | |
| 1097 | + goto err_put_device; | |
| 1093 | 1098 | } |
| 1094 | 1099 | |
| 1095 | 1100 | gdev->label = kstrdup_const(gc->label ?: "unknown", GFP_KERNEL); |
| 1096 | 1101 | if (!gdev->label) { |
| 1097 | 1102 | ret = -ENOMEM; |
| 1098 | − goto err_free_descs; | |
| 1103 | + goto err_put_device; | |
| 1099 | 1104 | } |
| 1100 | 1105 | |
| 1101 | − gdev->ngpio = gc->ngpio; | |
| 1102 | 1106 | gdev->can_sleep = gc->can_sleep; |
| 1103 | − | |
| 1104 | 1107 | rwlock_init(&gdev->line_state_lock); |
| 1105 | 1108 | RAW_INIT_NOTIFIER_HEAD(&gdev->line_state_notifier); |
| 1106 | 1109 | BLOCKING_INIT_NOTIFIER_HEAD(&gdev->device_notifier); |
| 1107 | − | |
| 1108 | − ret = init_srcu_struct(&gdev->srcu); | |
| 1109 | − if (ret) | |
| 1110 | − goto err_free_label; | |
| 1111 | − | |
| 1112 | − ret = init_srcu_struct(&gdev->desc_srcu); | |
| 1113 | − if (ret) | |
| 1114 | − goto err_cleanup_gdev_srcu; | |
| 1110 | +#ifdef CONFIG_PINCTRL | |
| 1111 | + INIT_LIST_HEAD(&gdev->pin_ranges); | |
| 1112 | +#endif | |
| 1113 | + if (gc->parent && gc->parent->driver) | |
| 1114 | + gdev->owner = gc->parent->driver->owner; | |
| 1115 | + else if (gc->owner) | |
| 1116 | + /* TODO: remove chip->owner */ | |
| 1117 | + gdev->owner = gc->owner; | |
| 1118 | + else | |
| 1119 | + gdev->owner = THIS_MODULE; | |
| 1115 | 1120 | |
| 1116 | 1121 | scoped_guard(mutex, &gpio_devices_lock) { |
| 1117 | 1122 | /* |
| @@ -1127,7 +1132,7 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 9 unmodified lines | ||
| 1127 | 1132 | if (base < 0) { |
| 1128 | 1133 | ret = base; |
| 1129 | 1134 | base = 0; |
| 1130 | − goto err_cleanup_desc_srcu; | |
| 1135 | + goto err_put_device; | |
| 1131 | 1136 | } |
| 1132 | 1137 | |
| 1133 | 1138 | /* |
| @@ -1147,14 +1152,10 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 13 unmodified lines | ||
| 1147 | 1152 | ret = gpiodev_add_to_list_unlocked(gdev); |
| 1148 | 1153 | if (ret) { |
| 1149 | 1154 | gpiochip_err(gc, "GPIO integer space overlap, cannot add chip\n"); |
| 1150 | − goto err_cleanup_desc_srcu; | |
| 1155 | + goto err_put_device; | |
| 1151 | 1156 | } |
| 1152 | 1157 | } |
| 1153 | 1158 | |
| 1154 | −#ifdef CONFIG_PINCTRL | |
| 1155 | − INIT_LIST_HEAD(&gdev->pin_ranges); | |
| 1156 | −#endif | |
| 1157 | − | |
| 1158 | 1159 | if (gc->names) |
| 1159 | 1160 | gpiochip_set_desc_names(gc); |
| 1160 | 1161 | |
| @@ -1210,7 +1211,7 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 49 unmodified lines | ||
| 1210 | 1211 | if (ret) |
| 1211 | 1212 | goto err_remove_irqchip_mask; |
| 1212 | 1213 | |
| 1213 | − ret = gpio_device_setup_shared(gdev); | |
| 1214 | + ret = gpiochip_setup_shared(gc); | |
| 1214 | 1215 | if (ret) |
| 1215 | 1216 | goto err_remove_irqchip; |
| 1216 | 1217 | |
| @@ -1248,25 +1249,19 @@ err_remove_from_list: · 31 unmodified lines | ||
| 1248 | 1249 | scoped_guard(mutex, &gpio_devices_lock) |
| 1249 | 1250 | list_del_rcu(&gdev->list); |
| 1250 | 1251 | synchronize_srcu(&gpio_devices_srcu); |
| 1251 | − if (gdev->dev.release) { | |
| 1252 | − /* release() has been registered by gpiochip_setup_dev() */ | |
| 1253 | − gpio_device_put(gdev); | |
| 1254 | − goto err_print_message; | |
| 1255 | − } | |
| 1252 | +err_put_device: | |
| 1253 | + gpio_device_put(gdev); | |
| 1254 | + goto err_print_message; | |
| 1255 | + | |
| 1256 | 1256 | err_cleanup_desc_srcu: |
| 1257 | 1257 | cleanup_srcu_struct(&gdev->desc_srcu); |
| 1258 | 1258 | err_cleanup_gdev_srcu: |
| 1259 | 1259 | cleanup_srcu_struct(&gdev->srcu); |
| 1260 | −err_free_label: | |
| 1261 | − kfree_const(gdev->label); | |
| 1262 | −err_free_descs: | |
| 1263 | − kfree(gdev->descs); | |
| 1264 | −err_free_dev_name: | |
| 1265 | − kfree(dev_name(&gdev->dev)); | |
| 1266 | 1260 | err_free_ida: |
| 1267 | 1261 | ida_free(&gpio_ida, gdev->id); |
| 1268 | 1262 | err_free_gdev: |
| 1269 | 1263 | kfree(gdev); |
| 1264 | + | |
| 1270 | 1265 | err_print_message: |
| 1271 | 1266 | /* failures here can mean systems won't boot... */ |
| 1272 | 1267 | if (ret != -EPROBE_DEFER) { |
| @@ -2465,8 +2460,10 @@ int gpiod_request_commit(struct gpio_desc *desc, const char *label) · 1192 unmodified lines | ||
| 2465 | 2460 | return -EBUSY; |
| 2466 | 2461 | |
| 2467 | 2462 | offset = gpiod_hwgpio(desc); |
| 2468 | − if (!gpiochip_line_is_valid(guard.gc, offset)) | |
| 2469 | − return -EINVAL; | |
| 2463 | + if (!gpiochip_line_is_valid(guard.gc, offset)) { | |
| 2464 | + ret = -EINVAL; | |
| 2465 | + goto out_clear_bit; | |
| 2466 | + } | |
| 2470 | 2467 | |
| 2471 | 2468 | /* NOTE: gpio_request() can be called in early boot, |
| 2472 | 2469 | * before IRQs are enabled, for non-sleeping (SOC) GPIOs. |
| @@ -4717,8 +4714,8 @@ struct gpio_desc *gpiod_find_and_request(struct device *consumer, · 2244 unmodified lines | ||
| 4717 | 4714 | * lookup table for the proxy device as previously |
| 4718 | 4715 | * we only knew the consumer's fwnode. |
| 4719 | 4716 | */ |
| 4720 | − ret = gpio_shared_add_proxy_lookup(consumer, con_id, | |
| 4721 | − lookupflags); | |
| 4717 | + ret = gpio_shared_add_proxy_lookup(consumer, fwnode, | |
| 4718 | + con_id, lookupflags); | |
| 4722 | 4719 | if (ret) |
| 4723 | 4720 | return ERR_PTR(ret); |
| 4724 | 4721 | |
| @@ -268,6 +268,13 @@ config DRM_GEM_SHMEM_HELPER · 267 unmodified lines | ||
| 268 | 268 | help |
| 269 | 269 | Choose this if you need the GEM shmem helper functions |
| 270 | 270 | |
| 271 | +config RUST_DRM_GEM_SHMEM_HELPER | |
| 272 | + bool | |
| 273 | + depends on DRM && MMU | |
| 274 | + select DRM_GEM_SHMEM_HELPER | |
| 275 | + help | |
| 276 | + Choose this if you need the GEM shmem helper functions In Rust | |
| 277 | + | |
| 271 | 278 | config DRM_SUBALLOC_HELPER |
| 272 | 279 | tristate |
| 273 | 280 | depends on DRM |
| @@ -85,7 +85,6 @@ config DRM_KUNIT_TEST · 84 unmodified lines | ||
| 85 | 85 | |
| 86 | 86 | config DRM_TTM_KUNIT_TEST |
| 87 | 87 | tristate "KUnit tests for TTM" if !KUNIT_ALL_TESTS |
| 88 | − default n | |
| 89 | 88 | depends on DRM && KUNIT && MMU && (UML || COMPILE_TEST) |
| 90 | 89 | select DRM_TTM |
| 91 | 90 | select DRM_BUDDY |
| @@ -213,6 +213,7 @@ amdgpu-y += \ · 212 unmodified lines | ||
| 213 | 213 | vcn_v4_0_5.o \ |
| 214 | 214 | vcn_v5_0_0.o \ |
| 215 | 215 | vcn_v5_0_1.o \ |
| 216 | + vcn_v5_0_2.o \ | |
| 216 | 217 | amdgpu_jpeg.o \ |
| 217 | 218 | jpeg_v1_0.o \ |
| 218 | 219 | jpeg_v2_0.o \ |
| @@ -223,6 +224,7 @@ amdgpu-y += \ · 4 unmodified lines | ||
| 223 | 224 | jpeg_v4_0_5.o \ |
| 224 | 225 | jpeg_v5_0_0.o \ |
| 225 | 226 | jpeg_v5_0_1.o \ |
| 227 | + jpeg_v5_0_2.o \ | |
| 226 | 228 | jpeg_v5_3_0.o |
| 227 | 229 | |
| 228 | 230 | # add VPE block |
| @@ -327,6 +327,7 @@ struct kfd_vm_fault_info; · 326 unmodified lines | ||
| 327 | 327 | struct amdgpu_hive_info; |
| 328 | 328 | struct amdgpu_reset_context; |
| 329 | 329 | struct amdgpu_reset_control; |
| 330 | +struct amdgpu_coredump_info; | |
| 330 | 331 | |
| 331 | 332 | enum amdgpu_cp_irq { |
| 332 | 333 | AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP = 0, |
| @@ -689,6 +690,7 @@ enum amdgpu_uid_type { · 356 unmodified lines | ||
| 689 | 690 | AMDGPU_UID_TYPE_XCD, |
| 690 | 691 | AMDGPU_UID_TYPE_AID, |
| 691 | 692 | AMDGPU_UID_TYPE_SOC, |
| 693 | + AMDGPU_UID_TYPE_MID, | |
| 692 | 694 | AMDGPU_UID_TYPE_MAX |
| 693 | 695 | }; |
| 694 | 696 | |
| @@ -1147,6 +1149,11 @@ struct amdgpu_device { · 452 unmodified lines | ||
| 1147 | 1149 | |
| 1148 | 1150 | struct amdgpu_reset_domain *reset_domain; |
| 1149 | 1151 | |
| 1152 | +#ifdef CONFIG_DEV_COREDUMP | |
| 1153 | + struct amdgpu_coredump_info *coredump; | |
| 1154 | + struct work_struct coredump_work; | |
| 1155 | +#endif | |
| 1156 | + | |
| 1150 | 1157 | struct mutex benchmark_mutex; |
| 1151 | 1158 | |
| 1152 | 1159 | bool scpm_enabled; |
| @@ -692,9 +692,9 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device *adev, · 691 unmodified lines | ||
| 692 | 692 | goto err_ib_sched; |
| 693 | 693 | } |
| 694 | 694 | |
| 695 | − /* Drop the initial kref_init count (see drm_sched_main as example) */ | |
| 696 | − dma_fence_put(f); | |
| 697 | 695 | ret = dma_fence_wait(f, false); |
| 696 | + /* Drop the returned fence reference after the wait completes */ | |
| 697 | + dma_fence_put(f); | |
| 698 | 698 | |
| 699 | 699 | err_ib_sched: |
| 700 | 700 | amdgpu_job_free(job); |
| @@ -330,7 +330,7 @@ static uint32_t kgd_gfx_v12_1_set_address_watch(struct amdgpu_device *adev, · 329 unmodified lines | ||
| 330 | 330 | |
| 331 | 331 | watch_address_cntl = 0; |
| 332 | 332 | watch_address_low = lower_32_bits(watch_address); |
| 333 | − watch_address_high = upper_32_bits(watch_address) & 0xffff; | |
| 333 | + watch_address_high = upper_32_bits(watch_address) & 0x1ffffff; | |
| 334 | 334 | |
| 335 | 335 | watch_address_cntl = REG_SET_FIELD(watch_address_cntl, |
| 336 | 336 | TCP_WATCH0_CNTL, |
| @@ -373,249 +373,280 @@ int amdgpu_atomfirmware_get_uma_carveout_info(struct amdgpu_device *adev, · 372 unmodified lines | ||
| 373 | 373 | return -ENODEV; |
| 374 | 374 | } |
| 375 | 375 | |
| 376 | −int | |
| 377 | −amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, | |
| 376 | +int amdgpu_atomfirmware_get_integrated_system_info(struct amdgpu_device *adev, | |
| 378 | 377 | int *vram_width, int *vram_type, |
| 379 | 378 | int *vram_vendor) |
| 380 | 379 | { |
| 381 | 380 | struct amdgpu_mode_info *mode_info = &adev->mode_info; |
| 382 | − int index, i = 0; | |
| 381 | + int index; | |
| 383 | 382 | u16 data_offset, size; |
| 384 | 383 | union igp_info *igp_info; |
| 385 | − union vram_info *vram_info; | |
| 386 | − union umc_info *umc_info; | |
| 387 | − union vram_module *vram_module; | |
| 388 | 384 | u8 frev, crev; |
| 389 | 385 | u8 mem_type; |
| 390 | − u8 mem_vendor; | |
| 391 | 386 | u32 mem_channel_number; |
| 392 | 387 | u32 mem_channel_width; |
| 393 | − u32 module_id; | |
| 394 | 388 | |
| 395 | − if (adev->flags & AMD_IS_APU) | |
| 396 | − index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, | |
| 389 | + index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, | |
| 397 | 390 | integratedsysteminfo); |
| 398 | − else { | |
| 399 | − switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | |
| 400 | − case IP_VERSION(12, 0, 0): | |
| 401 | − case IP_VERSION(12, 0, 1): | |
| 402 | − index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, umc_info); | |
| 391 | + if (amdgpu_atom_parse_data_header(mode_info->atom_context, | |
| 392 | + index, &size, | |
| 393 | + &frev, &crev, &data_offset)) { | |
| 394 | + igp_info = (union igp_info *) | |
| 395 | + (mode_info->atom_context->bios + data_offset); | |
| 396 | + switch (frev) { | |
| 397 | + case 1: | |
| 398 | + switch (crev) { | |
| 399 | + case 11: | |
| 400 | + case 12: | |
| 401 | + mem_channel_number = igp_info->v11.umachannelnumber; | |
| 402 | + if (!mem_channel_number) | |
| 403 | + mem_channel_number = 1; | |
| 404 | + mem_type = igp_info->v11.memorytype; | |
| 405 | + if (mem_type == LpDdr5MemType) | |
| 406 | + mem_channel_width = 32; | |
| 407 | + else | |
| 408 | + mem_channel_width = 64; | |
| 409 | + if (vram_width) | |
| 410 | + *vram_width = mem_channel_number * mem_channel_width; | |
| 411 | + if (vram_type) | |
| 412 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 413 | + break; | |
| 414 | + default: | |
| 415 | + return -EINVAL; | |
| 416 | + } | |
| 417 | + break; | |
| 418 | + case 2: | |
| 419 | + switch (crev) { | |
| 420 | + case 1: | |
| 421 | + case 2: | |
| 422 | + mem_channel_number = igp_info->v21.umachannelnumber; | |
| 423 | + if (!mem_channel_number) | |
| 424 | + mem_channel_number = 1; | |
| 425 | + mem_type = igp_info->v21.memorytype; | |
| 426 | + if (mem_type == LpDdr5MemType) | |
| 427 | + mem_channel_width = 32; | |
| 428 | + else | |
| 429 | + mem_channel_width = 64; | |
| 430 | + if (vram_width) | |
| 431 | + *vram_width = mem_channel_number * mem_channel_width; | |
| 432 | + if (vram_type) | |
| 433 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 434 | + break; | |
| 435 | + case 3: | |
| 436 | + mem_channel_number = igp_info->v23.umachannelnumber; | |
| 437 | + if (!mem_channel_number) | |
| 438 | + mem_channel_number = 1; | |
| 439 | + mem_type = igp_info->v23.memorytype; | |
| 440 | + if (mem_type == LpDdr5MemType) | |
| 441 | + mem_channel_width = 32; | |
| 442 | + else | |
| 443 | + mem_channel_width = 64; | |
| 444 | + if (vram_width) | |
| 445 | + *vram_width = mem_channel_number * mem_channel_width; | |
| 446 | + if (vram_type) | |
| 447 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 448 | + break; | |
| 449 | + default: | |
| 450 | + return -EINVAL; | |
| 451 | + } | |
| 403 | 452 | break; |
| 404 | 453 | default: |
| 405 | − index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, vram_info); | |
| 454 | + return -EINVAL; | |
| 406 | 455 | } |
| 456 | + } else { | |
| 457 | + return -EINVAL; | |
| 407 | 458 | } |
| 459 | + return 0; | |
| 460 | +} | |
| 461 | + | |
| 462 | +int amdgpu_atomfirmware_get_umc_info(struct amdgpu_device *adev, | |
| 463 | + int *vram_width, int *vram_type, | |
| 464 | + int *vram_vendor) | |
| 465 | +{ | |
| 466 | + struct amdgpu_mode_info *mode_info = &adev->mode_info; | |
| 467 | + int index; | |
| 468 | + u16 data_offset, size; | |
| 469 | + union umc_info *umc_info; | |
| 470 | + u8 frev, crev; | |
| 471 | + u8 mem_type; | |
| 472 | + u8 mem_vendor; | |
| 473 | + u32 mem_channel_number; | |
| 474 | + u32 mem_channel_width; | |
| 475 | + | |
| 476 | + index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, umc_info); | |
| 477 | + | |
| 408 | 478 | if (amdgpu_atom_parse_data_header(mode_info->atom_context, |
| 409 | 479 | index, &size, |
| 410 | 480 | &frev, &crev, &data_offset)) { |
| 411 | − if (adev->flags & AMD_IS_APU) { | |
| 412 | − igp_info = (union igp_info *) | |
| 413 | − (mode_info->atom_context->bios + data_offset); | |
| 414 | − switch (frev) { | |
| 415 | − case 1: | |
| 416 | − switch (crev) { | |
| 417 | − case 11: | |
| 418 | − case 12: | |
| 419 | − mem_channel_number = igp_info->v11.umachannelnumber; | |
| 420 | − if (!mem_channel_number) | |
| 421 | − mem_channel_number = 1; | |
| 422 | − mem_type = igp_info->v11.memorytype; | |
| 423 | − if (mem_type == LpDdr5MemType) | |
| 424 | − mem_channel_width = 32; | |
| 425 | − else | |
| 426 | − mem_channel_width = 64; | |
| 427 | − if (vram_width) | |
| 428 | − *vram_width = mem_channel_number * mem_channel_width; | |
| 429 | − if (vram_type) | |
| 430 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 431 | − break; | |
| 432 | − default: | |
| 433 | − return -EINVAL; | |
| 434 | − } | |
| 435 | − break; | |
| 436 | − case 2: | |
| 437 | − switch (crev) { | |
| 438 | − case 1: | |
| 439 | − case 2: | |
| 440 | − mem_channel_number = igp_info->v21.umachannelnumber; | |
| 441 | − if (!mem_channel_number) | |
| 442 | − mem_channel_number = 1; | |
| 443 | − mem_type = igp_info->v21.memorytype; | |
| 444 | − if (mem_type == LpDdr5MemType) | |
| 445 | − mem_channel_width = 32; | |
| 446 | − else | |
| 447 | − mem_channel_width = 64; | |
| 448 | − if (vram_width) | |
| 449 | − *vram_width = mem_channel_number * mem_channel_width; | |
| 450 | − if (vram_type) | |
| 451 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 452 | − break; | |
| 453 | − case 3: | |
| 454 | − mem_channel_number = igp_info->v23.umachannelnumber; | |
| 455 | − if (!mem_channel_number) | |
| 456 | − mem_channel_number = 1; | |
| 457 | − mem_type = igp_info->v23.memorytype; | |
| 458 | − if (mem_type == LpDdr5MemType) | |
| 459 | − mem_channel_width = 32; | |
| 460 | − else | |
| 461 | − mem_channel_width = 64; | |
| 462 | − if (vram_width) | |
| 463 | − *vram_width = mem_channel_number * mem_channel_width; | |
| 464 | − if (vram_type) | |
| 465 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 466 | − break; | |
| 467 | − default: | |
| 468 | − return -EINVAL; | |
| 469 | − } | |
| 481 | + umc_info = (union umc_info *)(mode_info->atom_context->bios + data_offset); | |
| 482 | + | |
| 483 | + if (frev == 4) { | |
| 484 | + switch (crev) { | |
| 485 | + case 0: | |
| 486 | + mem_channel_number = le32_to_cpu(umc_info->v40.channel_num); | |
| 487 | + mem_type = le32_to_cpu(umc_info->v40.vram_type); | |
| 488 | + mem_channel_width = le32_to_cpu(umc_info->v40.channel_width); | |
| 489 | + mem_vendor = RREG32(adev->bios_scratch_reg_offset + 4) & 0xF; | |
| 490 | + if (vram_vendor) | |
| 491 | + *vram_vendor = mem_vendor; | |
| 492 | + if (vram_type) | |
| 493 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 494 | + if (vram_width) | |
| 495 | + *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 470 | 496 | break; |
| 471 | 497 | default: |
| 472 | 498 | return -EINVAL; |
| 473 | 499 | } |
| 474 | 500 | } else { |
| 475 | − switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | |
| 476 | − case IP_VERSION(12, 0, 0): | |
| 477 | − case IP_VERSION(12, 0, 1): | |
| 478 | − umc_info = (union umc_info *)(mode_info->atom_context->bios + data_offset); | |
| 479 | − | |
| 480 | − if (frev == 4) { | |
| 481 | − switch (crev) { | |
| 482 | − case 0: | |
| 483 | − mem_channel_number = le32_to_cpu(umc_info->v40.channel_num); | |
| 484 | − mem_type = le32_to_cpu(umc_info->v40.vram_type); | |
| 485 | − mem_channel_width = le32_to_cpu(umc_info->v40.channel_width); | |
| 486 | − mem_vendor = RREG32(adev->bios_scratch_reg_offset + 4) & 0xF; | |
| 487 | − if (vram_vendor) | |
| 488 | − *vram_vendor = mem_vendor; | |
| 489 | − if (vram_type) | |
| 490 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 491 | − if (vram_width) | |
| 492 | − *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 493 | − break; | |
| 494 | − default: | |
| 495 | − return -EINVAL; | |
| 496 | − } | |
| 497 | − } else | |
| 498 | − return -EINVAL; | |
| 501 | + return -EINVAL; | |
| 502 | + } | |
| 503 | + } else { | |
| 504 | + return -EINVAL; | |
| 505 | + } | |
| 506 | + | |
| 507 | + return 0; | |
| 508 | +} | |
| 509 | + | |
| 510 | +int amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, | |
| 511 | + int *vram_width, int *vram_type, | |
| 512 | + int *vram_vendor) | |
| 513 | +{ | |
| 514 | + struct amdgpu_mode_info *mode_info = &adev->mode_info; | |
| 515 | + int index, i = 0; | |
| 516 | + u16 data_offset, size; | |
| 517 | + union vram_info *vram_info; | |
| 518 | + union vram_module *vram_module; | |
| 519 | + u8 frev, crev; | |
| 520 | + u8 mem_type; | |
| 521 | + u8 mem_vendor; | |
| 522 | + u32 mem_channel_number; | |
| 523 | + u32 mem_channel_width; | |
| 524 | + u32 module_id; | |
| 525 | + | |
| 526 | + index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, vram_info); | |
| 527 | + | |
| 528 | + if (amdgpu_atom_parse_data_header(mode_info->atom_context, | |
| 529 | + index, &size, | |
| 530 | + &frev, &crev, &data_offset)) { | |
| 531 | + vram_info = (union vram_info *) | |
| 532 | + (mode_info->atom_context->bios + data_offset); | |
| 533 | + | |
| 534 | + module_id = (RREG32(adev->bios_scratch_reg_offset + 4) & 0x00ff0000) >> 16; | |
| 535 | + if (frev == 3) { | |
| 536 | + switch (crev) { | |
| 537 | + /* v30 */ | |
| 538 | + case 0: | |
| 539 | + vram_module = (union vram_module *)vram_info->v30.vram_module; | |
| 540 | + mem_vendor = (vram_module->v30.dram_vendor_id) & 0xF; | |
| 541 | + if (vram_vendor) | |
| 542 | + *vram_vendor = mem_vendor; | |
| 543 | + mem_type = vram_info->v30.memory_type; | |
| 544 | + if (vram_type) | |
| 545 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 546 | + mem_channel_number = vram_info->v30.channel_num; | |
| 547 | + mem_channel_width = vram_info->v30.channel_width; | |
| 548 | + if (vram_width) | |
| 549 | + *vram_width = mem_channel_number * 16; | |
| 499 | 550 | break; |
| 500 | 551 | default: |
| 501 | − vram_info = (union vram_info *) | |
| 502 | − (mode_info->atom_context->bios + data_offset); | |
| 503 | − | |
| 504 | − module_id = (RREG32(adev->bios_scratch_reg_offset + 4) & 0x00ff0000) >> 16; | |
| 505 | − if (frev == 3) { | |
| 506 | − switch (crev) { | |
| 507 | − /* v30 */ | |
| 508 | − case 0: | |
| 509 | − vram_module = (union vram_module *)vram_info->v30.vram_module; | |
| 510 | − mem_vendor = (vram_module->v30.dram_vendor_id) & 0xF; | |
| 511 | − if (vram_vendor) | |
| 512 | − *vram_vendor = mem_vendor; | |
| 513 | − mem_type = vram_info->v30.memory_type; | |
| 514 | − if (vram_type) | |
| 515 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 516 | − mem_channel_number = vram_info->v30.channel_num; | |
| 517 | − mem_channel_width = vram_info->v30.channel_width; | |
| 518 | − if (vram_width) | |
| 519 | − *vram_width = mem_channel_number * 16; | |
| 520 | − break; | |
| 521 | − default: | |
| 522 | − return -EINVAL; | |
| 523 | − } | |
| 524 | − } else if (frev == 2) { | |
| 525 | − switch (crev) { | |
| 526 | − /* v23 */ | |
| 527 | − case 3: | |
| 528 | − if (module_id > vram_info->v23.vram_module_num) | |
| 529 | − module_id = 0; | |
| 530 | − vram_module = (union vram_module *)vram_info->v23.vram_module; | |
| 531 | − while (i < module_id) { | |
| 532 | − vram_module = (union vram_module *) | |
| 533 | − ((u8 *)vram_module + vram_module->v9.vram_module_size); | |
| 534 | − i++; | |
| 535 | − } | |
| 536 | − mem_type = vram_module->v9.memory_type; | |
| 537 | − if (vram_type) | |
| 538 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 539 | − mem_channel_number = vram_module->v9.channel_num; | |
| 540 | − mem_channel_width = vram_module->v9.channel_width; | |
| 541 | − if (vram_width) | |
| 542 | − *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 543 | − mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | |
| 544 | − if (vram_vendor) | |
| 545 | − *vram_vendor = mem_vendor; | |
| 546 | − break; | |
| 547 | − /* v24 */ | |
| 548 | − case 4: | |
| 549 | − if (module_id > vram_info->v24.vram_module_num) | |
| 550 | − module_id = 0; | |
| 551 | − vram_module = (union vram_module *)vram_info->v24.vram_module; | |
| 552 | − while (i < module_id) { | |
| 553 | − vram_module = (union vram_module *) | |
| 554 | − ((u8 *)vram_module + vram_module->v10.vram_module_size); | |
| 555 | − i++; | |
| 556 | − } | |
| 557 | − mem_type = vram_module->v10.memory_type; | |
| 558 | − if (vram_type) | |
| 559 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 560 | − mem_channel_number = vram_module->v10.channel_num; | |
| 561 | − mem_channel_width = vram_module->v10.channel_width; | |
| 562 | − if (vram_width) | |
| 563 | − *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 564 | − mem_vendor = (vram_module->v10.vender_rev_id) & 0xF; | |
| 565 | − if (vram_vendor) | |
| 566 | − *vram_vendor = mem_vendor; | |
| 567 | − break; | |
| 568 | − /* v25 */ | |
| 569 | − case 5: | |
| 570 | − if (module_id > vram_info->v25.vram_module_num) | |
| 571 | − module_id = 0; | |
| 572 | − vram_module = (union vram_module *)vram_info->v25.vram_module; | |
| 573 | − while (i < module_id) { | |
| 574 | − vram_module = (union vram_module *) | |
| 575 | − ((u8 *)vram_module + vram_module->v11.vram_module_size); | |
| 576 | − i++; | |
| 577 | − } | |
| 578 | − mem_type = vram_module->v11.memory_type; | |
| 579 | − if (vram_type) | |
| 580 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 581 | − mem_channel_number = vram_module->v11.channel_num; | |
| 582 | − mem_channel_width = vram_module->v11.channel_width; | |
| 583 | − if (vram_width) | |
| 584 | − *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 585 | − mem_vendor = (vram_module->v11.vender_rev_id) & 0xF; | |
| 586 | − if (vram_vendor) | |
| 587 | − *vram_vendor = mem_vendor; | |
| 588 | − break; | |
| 589 | − /* v26 */ | |
| 590 | − case 6: | |
| 591 | − if (module_id > vram_info->v26.vram_module_num) | |
| 592 | − module_id = 0; | |
| 593 | − vram_module = (union vram_module *)vram_info->v26.vram_module; | |
| 594 | − while (i < module_id) { | |
| 595 | − vram_module = (union vram_module *) | |
| 596 | − ((u8 *)vram_module + vram_module->v9.vram_module_size); | |
| 597 | − i++; | |
| 598 | − } | |
| 599 | − mem_type = vram_module->v9.memory_type; | |
| 600 | − if (vram_type) | |
| 601 | − *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 602 | − mem_channel_number = vram_module->v9.channel_num; | |
| 603 | − mem_channel_width = vram_module->v9.channel_width; | |
| 604 | − if (vram_width) | |
| 605 | − *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 606 | − mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | |
| 607 | − if (vram_vendor) | |
| 608 | − *vram_vendor = mem_vendor; | |
| 609 | − break; | |
| 610 | − default: | |
| 611 | − return -EINVAL; | |
| 612 | − } | |
| 613 | − } else { | |
| 614 | − /* invalid frev */ | |
| 615 | − return -EINVAL; | |
| 552 | + return -EINVAL; | |
| 553 | + } | |
| 554 | + } else if (frev == 2) { | |
| 555 | + switch (crev) { | |
| 556 | + /* v23 */ | |
| 557 | + case 3: | |
| 558 | + if (module_id > vram_info->v23.vram_module_num) | |
| 559 | + module_id = 0; | |
| 560 | + vram_module = (union vram_module *)vram_info->v23.vram_module; | |
| 561 | + while (i < module_id) { | |
| 562 | + vram_module = (union vram_module *) | |
| 563 | + ((u8 *)vram_module + vram_module->v9.vram_module_size); | |
| 564 | + i++; | |
| 616 | 565 | } |
| 566 | + mem_type = vram_module->v9.memory_type; | |
| 567 | + if (vram_type) | |
| 568 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 569 | + mem_channel_number = vram_module->v9.channel_num; | |
| 570 | + mem_channel_width = vram_module->v9.channel_width; | |
| 571 | + if (vram_width) | |
| 572 | + *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 573 | + mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | |
| 574 | + if (vram_vendor) | |
| 575 | + *vram_vendor = mem_vendor; | |
| 576 | + break; | |
| 577 | + /* v24 */ | |
| 578 | + case 4: | |
| 579 | + if (module_id > vram_info->v24.vram_module_num) | |
| 580 | + module_id = 0; | |
| 581 | + vram_module = (union vram_module *)vram_info->v24.vram_module; | |
| 582 | + while (i < module_id) { | |
| 583 | + vram_module = (union vram_module *) | |
| 584 | + ((u8 *)vram_module + vram_module->v10.vram_module_size); | |
| 585 | + i++; | |
| 586 | + } | |
| 587 | + mem_type = vram_module->v10.memory_type; | |
| 588 | + if (vram_type) | |
| 589 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 590 | + mem_channel_number = vram_module->v10.channel_num; | |
| 591 | + mem_channel_width = vram_module->v10.channel_width; | |
| 592 | + if (vram_width) | |
| 593 | + *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 594 | + mem_vendor = (vram_module->v10.vender_rev_id) & 0xF; | |
| 595 | + if (vram_vendor) | |
| 596 | + *vram_vendor = mem_vendor; | |
| 597 | + break; | |
| 598 | + /* v25 */ | |
| 599 | + case 5: | |
| 600 | + if (module_id > vram_info->v25.vram_module_num) | |
| 601 | + module_id = 0; | |
| 602 | + vram_module = (union vram_module *)vram_info->v25.vram_module; | |
| 603 | + while (i < module_id) { | |
| 604 | + vram_module = (union vram_module *) | |
| 605 | + ((u8 *)vram_module + vram_module->v11.vram_module_size); | |
| 606 | + i++; | |
| 607 | + } | |
| 608 | + mem_type = vram_module->v11.memory_type; | |
| 609 | + if (vram_type) | |
| 610 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 611 | + mem_channel_number = vram_module->v11.channel_num; | |
| 612 | + mem_channel_width = vram_module->v11.channel_width; | |
| 613 | + if (vram_width) | |
| 614 | + *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 615 | + mem_vendor = (vram_module->v11.vender_rev_id) & 0xF; | |
| 616 | + if (vram_vendor) | |
| 617 | + *vram_vendor = mem_vendor; | |
| 618 | + break; | |
| 619 | + /* v26 */ | |
| 620 | + case 6: | |
| 621 | + if (module_id > vram_info->v26.vram_module_num) | |
| 622 | + module_id = 0; | |
| 623 | + vram_module = (union vram_module *)vram_info->v26.vram_module; | |
| 624 | + while (i < module_id) { | |
| 625 | + vram_module = (union vram_module *) | |
| 626 | + ((u8 *)vram_module + vram_module->v9.vram_module_size); | |
| 627 | + i++; | |
| 628 | + } | |
| 629 | + mem_type = vram_module->v9.memory_type; | |
| 630 | + if (vram_type) | |
| 631 | + *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | |
| 632 | + mem_channel_number = vram_module->v9.channel_num; | |
| 633 | + mem_channel_width = vram_module->v9.channel_width; | |
| 634 | + if (vram_width) | |
| 635 | + *vram_width = mem_channel_number * (1 << mem_channel_width); | |
| 636 | + mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | |
| 637 | + if (vram_vendor) | |
| 638 | + *vram_vendor = mem_vendor; | |
| 639 | + break; | |
| 640 | + default: | |
| 641 | + return -EINVAL; | |
| 617 | 642 | } |
| 643 | + } else { | |
| 644 | + /* invalid frev */ | |
| 645 | + return -EINVAL; | |
| 618 | 646 | } |
| 647 | + | |
| 648 | + } else { | |
| 649 | + return -EINVAL; | |
| 619 | 650 | } |
| 620 | 651 | |
| 621 | 652 | return 0; |
| @@ -30,6 +30,10 @@ uint32_t amdgpu_atomfirmware_query_firmware_capability(struct amdgpu_device *ade · 29 unmodified lines | ||
| 30 | 30 | bool amdgpu_atomfirmware_gpu_virtualization_supported(struct amdgpu_device *adev); |
| 31 | 31 | void amdgpu_atomfirmware_scratch_regs_init(struct amdgpu_device *adev); |
| 32 | 32 | int amdgpu_atomfirmware_allocate_fb_scratch(struct amdgpu_device *adev); |
| 33 | +int amdgpu_atomfirmware_get_integrated_system_info(struct amdgpu_device *adev, | |
| 34 | + int *vram_width, int *vram_type, int *vram_vendor); | |
| 35 | +int amdgpu_atomfirmware_get_umc_info(struct amdgpu_device *adev, | |
| 36 | + int *vram_width, int *vram_type, int *vram_vendor); | |
| 33 | 37 | int amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, |
| 34 | 38 | int *vram_width, int *vram_type, int *vram_vendor); |
| 35 | 39 | int amdgpu_atomfirmware_get_uma_carveout_info(struct amdgpu_device *adev, |
| @@ -36,6 +36,7 @@ · 35 unmodified lines | ||
| 36 | 36 | |
| 37 | 37 | #define AMDGPU_BO_LIST_MAX_PRIORITY 32u |
| 38 | 38 | #define AMDGPU_BO_LIST_NUM_BUCKETS (AMDGPU_BO_LIST_MAX_PRIORITY + 1) |
| 39 | +#define AMDGPU_BO_LIST_MAX_ENTRIES (128 * 1024) | |
| 39 | 40 | |
| 40 | 41 | static void amdgpu_bo_list_free_rcu(struct rcu_head *rcu) |
| 41 | 42 | { |
| @@ -188,6 +189,9 @@ int amdgpu_bo_create_list_entry_array(struct drm_amdgpu_bo_list_in *in, · 146 unmodified lines | ||
| 188 | 189 | const uint32_t bo_number = in->bo_number; |
| 189 | 190 | struct drm_amdgpu_bo_list_entry *info; |
| 190 | 191 | |
| 192 | + if (bo_number > AMDGPU_BO_LIST_MAX_ENTRIES) | |
| 193 | + return -EINVAL; | |
| 194 | + | |
| 191 | 195 | /* copy the handle array from userspace to a kernel buffer */ |
| 192 | 196 | if (likely(info_size == bo_info_size)) { |
| 193 | 197 | info = vmemdup_array_user(uptr, bo_number, info_size); |
| @@ -1231,6 +1231,8 @@ static enum drm_mode_status amdgpu_connector_dvi_mode_valid(struct drm_connector · 1230 unmodified lines | ||
| 1231 | 1231 | case CONNECTOR_OBJECT_ID_HDMI_TYPE_B: |
| 1232 | 1232 | max_digital_pixel_clock_khz = max_dvi_single_link_pixel_clock * 2; |
| 1233 | 1233 | break; |
| 1234 | + default: | |
| 1235 | + return MODE_BAD; | |
| 1234 | 1236 | } |
| 1235 | 1237 | |
| 1236 | 1238 | /* When the display EDID claims that it's an HDMI display, |
| @@ -908,9 +908,8 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 907 unmodified lines | ||
| 908 | 908 | goto out_free_user_pages; |
| 909 | 909 | |
| 910 | 910 | amdgpu_bo_list_for_each_entry(e, p->bo_list) { |
| 911 | − /* One fence for TTM and one for each CS job */ | |
| 912 | 911 | r = drm_exec_prepare_obj(&p->exec, &e->bo->tbo.base, |
| 913 | − 1 + p->gang_size); | |
| 912 | + TTM_NUM_MOVE_FENCES + p->gang_size); | |
| 914 | 913 | drm_exec_retry_on_contention(&p->exec); |
| 915 | 914 | if (unlikely(r)) |
| 916 | 915 | goto out_free_user_pages; |
| @@ -920,7 +919,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 3 unmodified lines | ||
| 920 | 919 | |
| 921 | 920 | if (p->uf_bo) { |
| 922 | 921 | r = drm_exec_prepare_obj(&p->exec, &p->uf_bo->tbo.base, |
| 923 | − 1 + p->gang_size); | |
| 922 | + TTM_NUM_MOVE_FENCES + p->gang_size); | |
| 924 | 923 | drm_exec_retry_on_contention(&p->exec); |
| 925 | 924 | if (unlikely(r)) |
| 926 | 925 | goto out_free_user_pages; |
| @@ -1740,6 +1739,13 @@ int amdgpu_cs_wait_fences_ioctl(struct drm_device *dev, void *data, · 813 unmodified lines | ||
| 1740 | 1739 | struct drm_amdgpu_fence *fences; |
| 1741 | 1740 | int r; |
| 1742 | 1741 | |
| 1742 | + /* | |
| 1743 | + * fence_count must be non-zero; dma_fence_wait_any_timeout() | |
| 1744 | + * does not accept an empty fence array. | |
| 1745 | + */ | |
| 1746 | + if (!wait->in.fence_count) | |
| 1747 | + return -EINVAL; | |
| 1748 | + | |
| 1743 | 1749 | /* Get the fences from userspace */ |
| 1744 | 1750 | fences = memdup_array_user(u64_to_user_ptr(wait->in.fences), |
| 1745 | 1751 | wait->in.fence_count, |
| @@ -617,6 +617,110 @@ out: · 616 unmodified lines | ||
| 617 | 617 | return r; |
| 618 | 618 | } |
| 619 | 619 | |
| 620 | +/** | |
| 621 | + * amdgpu_debugfs_regs_pcie64_read - Read from a 64-bit PCIE register | |
| 622 | + * | |
| 623 | + * @f: open file handle | |
| 624 | + * @buf: User buffer to store read data in | |
| 625 | + * @size: Number of bytes to read | |
| 626 | + * @pos: Offset to seek to | |
| 627 | + */ | |
| 628 | +static ssize_t amdgpu_debugfs_regs_pcie64_read(struct file *f, char __user *buf, | |
| 629 | + size_t size, loff_t *pos) | |
| 630 | +{ | |
| 631 | + struct amdgpu_device *adev = file_inode(f)->i_private; | |
| 632 | + ssize_t result = 0; | |
| 633 | + int r; | |
| 634 | + | |
| 635 | + if (size & 0x7 || *pos & 0x7) | |
| 636 | + return -EINVAL; | |
| 637 | + | |
| 638 | + r = pm_runtime_get_sync(adev_to_drm(adev)->dev); | |
| 639 | + if (r < 0) { | |
| 640 | + pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | |
| 641 | + return r; | |
| 642 | + } | |
| 643 | + | |
| 644 | + r = amdgpu_virt_enable_access_debugfs(adev); | |
| 645 | + if (r < 0) { | |
| 646 | + pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | |
| 647 | + return r; | |
| 648 | + } | |
| 649 | + | |
| 650 | + while (size) { | |
| 651 | + uint64_t value; | |
| 652 | + | |
| 653 | + value = RREG64_PCIE_EXT(*pos); | |
| 654 | + | |
| 655 | + r = put_user(value, (uint64_t *)buf); | |
| 656 | + if (r) | |
| 657 | + goto out; | |
| 658 | + | |
| 659 | + result += 8; | |
| 660 | + buf += 8; | |
| 661 | + *pos += 8; | |
| 662 | + size -= 8; | |
| 663 | + } | |
| 664 | + | |
| 665 | + r = result; | |
| 666 | +out: | |
| 667 | + pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | |
| 668 | + amdgpu_virt_disable_access_debugfs(adev); | |
| 669 | + return r; | |
| 670 | +} | |
| 671 | + | |
| 672 | +/** | |
| 673 | + * amdgpu_debugfs_regs_pcie64_write - Write to a 64-bit PCIE register | |
| 674 | + * | |
| 675 | + * @f: open file handle | |
| 676 | + * @buf: User buffer to write data from | |
| 677 | + * @size: Number of bytes to write | |
| 678 | + * @pos: Offset to seek to | |
| 679 | + */ | |
| 680 | +static ssize_t amdgpu_debugfs_regs_pcie64_write(struct file *f, const char __user *buf, | |
| 681 | + size_t size, loff_t *pos) | |
| 682 | +{ | |
| 683 | + struct amdgpu_device *adev = file_inode(f)->i_private; | |
| 684 | + ssize_t result = 0; | |
| 685 | + int r; | |
| 686 | + | |
| 687 | + if (size & 0x7 || *pos & 0x7) | |
| 688 | + return -EINVAL; | |
| 689 | + | |
| 690 | + r = pm_runtime_get_sync(adev_to_drm(adev)->dev); | |
| 691 | + if (r < 0) { | |
| 692 | + pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | |
| 693 | + return r; | |
| 694 | + } | |
| 695 | + | |
| 696 | + r = amdgpu_virt_enable_access_debugfs(adev); | |
| 697 | + if (r < 0) { | |
| 698 | + pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | |
| 699 | + return r; | |
| 700 | + } | |
| 701 | + | |
| 702 | + while (size) { | |
| 703 | + uint64_t value; | |
| 704 | + | |
| 705 | + r = get_user(value, (uint64_t *)buf); | |
| 706 | + if (r) | |
| 707 | + goto out; | |
| 708 | + | |
| 709 | + WREG64_PCIE_EXT(*pos, value); | |
| 710 | + | |
| 711 | + result += 8; | |
| 712 | + buf += 8; | |
| 713 | + *pos += 8; | |
| 714 | + size -= 8; | |
| 715 | + } | |
| 716 | + | |
| 717 | + r = result; | |
| 718 | +out: | |
| 719 | + pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | |
| 720 | + amdgpu_virt_disable_access_debugfs(adev); | |
| 721 | + return r; | |
| 722 | +} | |
| 723 | + | |
| 620 | 724 | /** |
| 621 | 725 | * amdgpu_debugfs_regs_didt_read - Read from a DIDT register |
| 622 | 726 | * |
| @@ -1525,6 +1629,12 @@ static const struct file_operations amdgpu_debugfs_regs_pcie_fops = { · 902 unmodified lines | ||
| 1525 | 1629 | .write = amdgpu_debugfs_regs_pcie_write, |
| 1526 | 1630 | .llseek = default_llseek |
| 1527 | 1631 | }; |
| 1632 | +static const struct file_operations amdgpu_debugfs_regs_pcie64_fops = { | |
| 1633 | + .owner = THIS_MODULE, | |
| 1634 | + .read = amdgpu_debugfs_regs_pcie64_read, | |
| 1635 | + .write = amdgpu_debugfs_regs_pcie64_write, | |
| 1636 | + .llseek = default_llseek | |
| 1637 | +}; | |
| 1528 | 1638 | static const struct file_operations amdgpu_debugfs_regs_smc_fops = { |
| 1529 | 1639 | .owner = THIS_MODULE, |
| 1530 | 1640 | .read = amdgpu_debugfs_regs_smc_read, |
| @@ -1587,6 +1697,7 @@ static const struct file_operations *debugfs_regs[] = { · 56 unmodified lines | ||
| 1587 | 1697 | &amdgpu_debugfs_gprwave_fops, |
| 1588 | 1698 | &amdgpu_debugfs_regs_didt_fops, |
| 1589 | 1699 | &amdgpu_debugfs_regs_pcie_fops, |
| 1700 | + &amdgpu_debugfs_regs_pcie64_fops, | |
| 1590 | 1701 | &amdgpu_debugfs_regs_smc_fops, |
| 1591 | 1702 | &amdgpu_debugfs_gca_config_fops, |
| 1592 | 1703 | &amdgpu_debugfs_sensors_fops, |
| @@ -1604,6 +1715,7 @@ static const char * const debugfs_regs_names[] = { · 11 unmodified lines | ||
| 1604 | 1715 | "amdgpu_gprwave", |
| 1605 | 1716 | "amdgpu_regs_didt", |
| 1606 | 1717 | "amdgpu_regs_pcie", |
| 1718 | + "amdgpu_regs_pcie64", | |
| 1607 | 1719 | "amdgpu_regs_smc", |
| 1608 | 1720 | "amdgpu_gca_config", |
| 1609 | 1721 | "amdgpu_sensors", |
| @@ -32,8 +32,16 @@ void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, · 31 unmodified lines | ||
| 32 | 32 | bool vram_lost, struct amdgpu_job *job) |
| 33 | 33 | { |
| 34 | 34 | } |
| 35 | +void amdgpu_coredump_init(struct amdgpu_device *adev) | |
| 36 | +{ | |
| 37 | +} | |
| 38 | +void amdgpu_coredump_fini(struct amdgpu_device *adev) | |
| 39 | +{ | |
| 40 | +} | |
| 35 | 41 | #else |
| 36 | 42 | |
| 43 | +#define AMDGPU_CORE_DUMP_SIZE_MAX (256 * 1024 * 1024) | |
| 44 | + | |
| 37 | 45 | const char *hw_ip_names[MAX_HWIP] = { |
| 38 | 46 | [GC_HWIP] = "GC", |
| 39 | 47 | [HDP_HWIP] = "HDP", |
| @@ -187,20 +195,22 @@ static void amdgpu_devcoredump_fw_info(struct amdgpu_device *adev, · 147 unmodified lines | ||
| 187 | 195 | drm_printf(p, "VPE feature version: %u, fw version: 0x%08x\n", |
| 188 | 196 | adev->vpe.feature_version, adev->vpe.fw_version); |
| 189 | 197 | |
| 190 | − drm_printf(p, "\nVBIOS Information\n"); | |
| 191 | − drm_printf(p, "vbios name : %s\n", ctx->name); | |
| 192 | − drm_printf(p, "vbios pn : %s\n", ctx->vbios_pn); | |
| 193 | − drm_printf(p, "vbios version : %d\n", ctx->version); | |
| 194 | − drm_printf(p, "vbios ver_str : %s\n", ctx->vbios_ver_str); | |
| 195 | − drm_printf(p, "vbios date : %s\n", ctx->date); | |
| 198 | + if (adev->bios) { | |
| 199 | + drm_printf(p, "\nVBIOS Information\n"); | |
| 200 | + drm_printf(p, "vbios name : %s\n", ctx->name); | |
| 201 | + drm_printf(p, "vbios pn : %s\n", ctx->vbios_pn); | |
| 202 | + drm_printf(p, "vbios version : %d\n", ctx->version); | |
| 203 | + drm_printf(p, "vbios ver_str : %s\n", ctx->vbios_ver_str); | |
| 204 | + drm_printf(p, "vbios date : %s\n", ctx->date); | |
| 205 | + }else { | |
| 206 | + drm_printf(p, "\nVBIOS Information: NA\n"); | |
| 207 | + } | |
| 196 | 208 | } |
| 197 | 209 | |
| 198 | 210 | static ssize_t |
| 199 | −amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, | |
| 200 | − void *data, size_t datalen) | |
| 211 | +amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_info *coredump) | |
| 201 | 212 | { |
| 202 | 213 | struct drm_printer p; |
| 203 | − struct amdgpu_coredump_info *coredump = data; | |
| 204 | 214 | struct drm_print_iterator iter; |
| 205 | 215 | struct amdgpu_vm_fault_info *fault_info; |
| 206 | 216 | struct amdgpu_ip_block *ip_block; |
| @@ -208,7 +218,6 @@ amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, · 1 unmodified line | ||
| 208 | 218 | |
| 209 | 219 | iter.data = buffer; |
| 210 | 220 | iter.offset = 0; |
| 211 | − iter.start = offset; | |
| 212 | 221 | iter.remain = count; |
| 213 | 222 | |
| 214 | 223 | p = drm_coredump_printer(&iter); |
| @@ -322,9 +331,63 @@ amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, · 107 unmodified lines | ||
| 322 | 331 | return count - iter.remain; |
| 323 | 332 | } |
| 324 | 333 | |
| 334 | +static ssize_t | |
| 335 | +amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, | |
| 336 | + void *data, size_t datalen) | |
| 337 | +{ | |
| 338 | + struct amdgpu_coredump_info *coredump = data; | |
| 339 | + ssize_t byte_copied; | |
| 340 | + | |
| 341 | + if (!coredump) | |
| 342 | + return -ENODEV; | |
| 343 | + | |
| 344 | + if (!coredump->formatted) | |
| 345 | + return -ENODEV; | |
| 346 | + | |
| 347 | + if (offset >= coredump->formatted_size) | |
| 348 | + return 0; | |
| 349 | + | |
| 350 | + byte_copied = count < coredump->formatted_size - offset ? count : | |
| 351 | + coredump->formatted_size - offset; | |
| 352 | + memcpy(buffer, coredump->formatted + offset, byte_copied); | |
| 353 | + | |
| 354 | + return byte_copied; | |
| 355 | +} | |
| 356 | + | |
| 325 | 357 | static void amdgpu_devcoredump_free(void *data) |
| 326 | 358 | { |
| 327 | − kfree(data); | |
| 359 | + struct amdgpu_coredump_info *coredump = data; | |
| 360 | + | |
| 361 | + kvfree(coredump->formatted); | |
| 362 | + kvfree(data); | |
| 363 | +} | |
| 364 | + | |
| 365 | +static void amdgpu_devcoredump_deferred_work(struct work_struct *work) | |
| 366 | +{ | |
| 367 | + struct amdgpu_device *adev = container_of(work, typeof(*adev), coredump_work); | |
| 368 | + struct amdgpu_coredump_info *coredump = adev->coredump; | |
| 369 | + | |
| 370 | + /* Do a one-time preparation of the coredump output because | |
| 371 | + * repeatingly calling drm_coredump_printer is very slow. | |
| 372 | + */ | |
| 373 | + coredump->formatted_size = amdgpu_devcoredump_format( | |
| 374 | + NULL, AMDGPU_CORE_DUMP_SIZE_MAX, coredump); | |
| 375 | + coredump->formatted = kvzalloc(coredump->formatted_size, GFP_KERNEL); | |
| 376 | + if (!coredump->formatted) { | |
| 377 | + amdgpu_devcoredump_free(coredump); | |
| 378 | + goto end; | |
| 379 | + } | |
| 380 | + | |
| 381 | + amdgpu_devcoredump_format(coredump->formatted, coredump->formatted_size, coredump); | |
| 382 | + | |
| 383 | + /* If there's an existing coredump for this device, the free function will be | |
| 384 | + * called immediately so coredump might be invalid after the call to dev_coredumpm. | |
| 385 | + */ | |
| 386 | + dev_coredumpm(coredump->adev->dev, THIS_MODULE, coredump, 0, GFP_NOWAIT, | |
| 387 | + amdgpu_devcoredump_read, amdgpu_devcoredump_free); | |
| 388 | + | |
| 389 | +end: | |
| 390 | + adev->coredump = NULL; | |
| 328 | 391 | } |
| 329 | 392 | |
| 330 | 393 | void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, |
| @@ -334,6 +397,10 @@ void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, · 3 unmodified lines | ||
| 334 | 397 | struct amdgpu_coredump_info *coredump; |
| 335 | 398 | struct drm_sched_job *s_job; |
| 336 | 399 | |
| 400 | + /* No need to generate a new coredump if there's one in progress already. */ | |
| 401 | + if (work_pending(&adev->coredump_work)) | |
| 402 | + return; | |
| 403 | + | |
| 337 | 404 | coredump = kzalloc_obj(*coredump, GFP_NOWAIT); |
| 338 | 405 | if (!coredump) |
| 339 | 406 | return; |
| @@ -360,11 +427,26 @@ void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, · 20 unmodified lines | ||
| 360 | 427 | |
| 361 | 428 | ktime_get_ts64(&coredump->reset_time); |
| 362 | 429 | |
| 363 | − dev_coredumpm(dev->dev, THIS_MODULE, coredump, 0, GFP_NOWAIT, | |
| 364 | − amdgpu_devcoredump_read, amdgpu_devcoredump_free); | |
| 430 | + /* Update the current coredump pointer (no lock needed, this function can only be called | |
| 431 | + * from a single thread) | |
| 432 | + */ | |
| 433 | + adev->coredump = coredump; | |
| 434 | + /* Kick off coredump formatting to a worker thread. */ | |
| 435 | + queue_work(system_unbound_wq, &adev->coredump_work); | |
| 365 | 436 | |
| 366 | 437 | drm_info(dev, "AMDGPU device coredump file has been created\n"); |
| 367 | 438 | drm_info(dev, "Check your /sys/class/drm/card%d/device/devcoredump/data\n", |
| 368 | 439 | dev->primary->index); |
| 369 | 440 | } |
| 441 | + | |
| 442 | +void amdgpu_coredump_init(struct amdgpu_device *adev) | |
| 443 | +{ | |
| 444 | + INIT_WORK(&adev->coredump_work, amdgpu_devcoredump_deferred_work); | |
| 445 | +} | |
| 446 | + | |
| 447 | +void amdgpu_coredump_fini(struct amdgpu_device *adev) | |
| 448 | +{ | |
| 449 | + /* Finish deferred coredump formatting before HW/IP teardown. */ | |
| 450 | + flush_work(&adev->coredump_work); | |
| 451 | +} | |
| 370 | 452 | #endif |
| @@ -35,12 +35,20 @@ struct amdgpu_coredump_info { · 34 unmodified lines | ||
| 35 | 35 | struct amdgpu_device *adev; |
| 36 | 36 | struct amdgpu_task_info reset_task_info; |
| 37 | 37 | struct timespec64 reset_time; |
| 38 | + | |
| 38 | 39 | bool skip_vram_check; |
| 39 | 40 | bool reset_vram_lost; |
| 40 | 41 | struct amdgpu_ring *ring; |
| 42 | + /* Readable form of coredevdump, generate once to speed up | |
| 43 | + * reading it (see drm_coredump_printer's documentation). | |
| 44 | + */ | |
| 45 | + ssize_t formatted_size; | |
| 46 | + char *formatted; | |
| 41 | 47 | }; |
| 42 | 48 | #endif |
| 43 | 49 | |
| 44 | 50 | void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, |
| 45 | 51 | bool vram_lost, struct amdgpu_job *job); |
| 52 | +void amdgpu_coredump_init(struct amdgpu_device *adev); | |
| 53 | +void amdgpu_coredump_fini(struct amdgpu_device *adev); | |
| 46 | 54 | #endif |
| @@ -3498,7 +3498,8 @@ fail: · 3497 unmodified lines | ||
| 3498 | 3498 | |
| 3499 | 3499 | static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) |
| 3500 | 3500 | { |
| 3501 | − char *input = amdgpu_lockup_timeout; | |
| 3501 | + char buf[AMDGPU_MAX_TIMEOUT_PARAM_LENGTH]; | |
| 3502 | + char *input = buf; | |
| 3502 | 3503 | char *timeout_setting = NULL; |
| 3503 | 3504 | int index = 0; |
| 3504 | 3505 | long timeout; |
| @@ -3508,9 +3509,17 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) · 3 unmodified lines | ||
| 3508 | 3509 | adev->gfx_timeout = adev->compute_timeout = adev->sdma_timeout = |
| 3509 | 3510 | adev->video_timeout = msecs_to_jiffies(2000); |
| 3510 | 3511 | |
| 3511 | − if (!strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) | |
| 3512 | + if (!strnlen(amdgpu_lockup_timeout, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) | |
| 3512 | 3513 | return 0; |
| 3513 | 3514 | |
| 3515 | + /* | |
| 3516 | + * strsep() destructively modifies its input by replacing delimiters | |
| 3517 | + * with '\0'. Use a stack copy so the global module parameter buffer | |
| 3518 | + * remains intact for multi-GPU systems where this function is called | |
| 3519 | + * once per device. | |
| 3520 | + */ | |
| 3521 | + strscpy(buf, amdgpu_lockup_timeout, sizeof(buf)); | |
| 3522 | + | |
| 3514 | 3523 | while ((timeout_setting = strsep(&input, ",")) && |
| 3515 | 3524 | strnlen(timeout_setting, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) { |
| 3516 | 3525 | ret = kstrtol(timeout_setting, 0, &timeout); |
| @@ -3781,6 +3790,8 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 264 unmodified lines | ||
| 3781 | 3790 | INIT_WORK(&adev->xgmi_reset_work, amdgpu_device_xgmi_reset_func); |
| 3782 | 3791 | INIT_WORK(&adev->userq_reset_work, amdgpu_userq_reset_work); |
| 3783 | 3792 | |
| 3793 | + amdgpu_coredump_init(adev); | |
| 3794 | + | |
| 3784 | 3795 | adev->gfx.gfx_off_req_count = 1; |
| 3785 | 3796 | adev->gfx.gfx_off_residency = 0; |
| 3786 | 3797 | adev->gfx.gfx_off_entrycount = 0; |
| @@ -3878,9 +3889,21 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 91 unmodified lines | ||
| 3878 | 3889 | amdgpu_gmc_noretry_set(adev); |
| 3879 | 3890 | /* Need to get xgmi info early to decide the reset behavior*/ |
| 3880 | 3891 | if (adev->gmc.xgmi.supported) { |
| 3881 | − r = adev->gfxhub.funcs->get_xgmi_info(adev); | |
| 3882 | − if (r) | |
| 3883 | − return r; | |
| 3892 | + if (adev->gfxhub.funcs && | |
| 3893 | + adev->gfxhub.funcs->get_xgmi_info) { | |
| 3894 | + r = adev->gfxhub.funcs->get_xgmi_info(adev); | |
| 3895 | + if (r) | |
| 3896 | + return r; | |
| 3897 | + } | |
| 3898 | + } | |
| 3899 | + | |
| 3900 | + if (adev->gmc.xgmi.connected_to_cpu) { | |
| 3901 | + if (adev->mmhub.funcs && | |
| 3902 | + adev->mmhub.funcs->get_xgmi_info) { | |
| 3903 | + r = adev->mmhub.funcs->get_xgmi_info(adev); | |
| 3904 | + if (r) | |
| 3905 | + return r; | |
| 3906 | + } | |
| 3884 | 3907 | } |
| 3885 | 3908 | |
| 3886 | 3909 | /* enable PCIE atomic ops */ |
| @@ -4202,6 +4225,7 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev) · 315 unmodified lines | ||
| 4202 | 4225 | if (pci_dev_is_disconnected(adev->pdev)) |
| 4203 | 4226 | amdgpu_amdkfd_device_fini_sw(adev); |
| 4204 | 4227 | |
| 4228 | + amdgpu_coredump_fini(adev); | |
| 4205 | 4229 | amdgpu_device_ip_fini_early(adev); |
| 4206 | 4230 | |
| 4207 | 4231 | amdgpu_irq_fini_hw(adev); |
| @@ -112,8 +112,10 @@ · 111 unmodified lines | ||
| 112 | 112 | #include "smuio_v15_0_8.h" |
| 113 | 113 | #include "vcn_v5_0_0.h" |
| 114 | 114 | #include "vcn_v5_0_1.h" |
| 115 | +#include "vcn_v5_0_2.h" | |
| 115 | 116 | #include "jpeg_v5_0_0.h" |
| 116 | 117 | #include "jpeg_v5_0_1.h" |
| 118 | +#include "jpeg_v5_0_2.h" | |
| 117 | 119 | #include "jpeg_v5_3_0.h" |
| 118 | 120 | |
| 119 | 121 | #include "amdgpu_ras_mgr.h" |
| @@ -296,13 +298,17 @@ static int amdgpu_discovery_get_tmr_info(struct amdgpu_device *adev, · 176 unmodified lines | ||
| 296 | 298 | if (vram_size) |
| 297 | 299 | adev->discovery.offset = (vram_size << 20) - DISCOVERY_TMR_OFFSET; |
| 298 | 300 | |
| 299 | − if (amdgpu_sriov_vf(adev) && adev->virt.is_dynamic_crit_regn_enabled) { | |
| 300 | − adev->discovery.offset = | |
| 301 | − adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].offset; | |
| 302 | − adev->discovery.size = | |
| 303 | − adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].size_kb << 10; | |
| 304 | − if (!adev->discovery.offset || !adev->discovery.size) | |
| 305 | − return -EINVAL; | |
| 301 | + if (amdgpu_sriov_vf(adev)) { | |
| 302 | + if (adev->virt.is_dynamic_crit_regn_enabled) { | |
| 303 | + adev->discovery.offset = | |
| 304 | + adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].offset; | |
| 305 | + adev->discovery.size = | |
| 306 | + adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].size_kb << 10; | |
| 307 | + if (!adev->discovery.offset || !adev->discovery.size) | |
| 308 | + return -EINVAL; | |
| 309 | + } else { | |
| 310 | + goto out; | |
| 311 | + } | |
| 306 | 312 | } else { |
| 307 | 313 | tmr_size = RREG32(mmDRIVER_SCRATCH_2); |
| 308 | 314 | if (tmr_size) { |
| @@ -318,11 +324,11 @@ static int amdgpu_discovery_get_tmr_info(struct amdgpu_device *adev, · 9 unmodified lines | ||
| 318 | 324 | ret = amdgpu_acpi_get_tmr_info(adev, &tmr_offset, &tmr_size); |
| 319 | 325 | if (ret) |
| 320 | 326 | return ret; |
| 321 | − adev->discovery.size = (u32)tmr_size; | |
| 327 | + adev->discovery.size = DISCOVERY_TMR_SIZE; | |
| 322 | 328 | adev->discovery.offset = tmr_offset + tmr_size - DISCOVERY_TMR_OFFSET; |
| 323 | 329 | } |
| 324 | 330 | } |
| 325 | − | |
| 331 | +out: | |
| 326 | 332 | adev->discovery.bin = kzalloc(adev->discovery.size, GFP_KERNEL); |
| 327 | 333 | if (!adev->discovery.bin) |
| 328 | 334 | return -ENOMEM; |
| @@ -556,7 +562,7 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 227 unmodified lines | ||
| 556 | 562 | checksum = le16_to_cpu(info->checksum); |
| 557 | 563 | |
| 558 | 564 | switch (table_id) { |
| 559 | − case IP_DISCOVERY: | |
| 565 | + case IP_DISCOVERY: { | |
| 560 | 566 | struct ip_discovery_header *ihdr = |
| 561 | 567 | (struct ip_discovery_header *)(discovery_bin + offset); |
| 562 | 568 | act_val = le32_to_cpu(ihdr->signature); |
| @@ -564,7 +570,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | ||
| 564 | 570 | table_size = le16_to_cpu(ihdr->size); |
| 565 | 571 | table_name = "data table"; |
| 566 | 572 | break; |
| 567 | − case GC: | |
| 573 | + } | |
| 574 | + case GC: { | |
| 568 | 575 | struct gpu_info_header *ghdr = |
| 569 | 576 | (struct gpu_info_header *)(discovery_bin + offset); |
| 570 | 577 | act_val = le32_to_cpu(ghdr->table_id); |
| @@ -572,7 +579,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | ||
| 572 | 579 | table_size = le16_to_cpu(ghdr->size); |
| 573 | 580 | table_name = "gc table"; |
| 574 | 581 | break; |
| 575 | − case HARVEST_INFO: | |
| 582 | + } | |
| 583 | + case HARVEST_INFO: { | |
| 576 | 584 | struct harvest_info_header *hhdr = |
| 577 | 585 | (struct harvest_info_header *)(discovery_bin + offset); |
| 578 | 586 | act_val = le32_to_cpu(hhdr->signature); |
| @@ -580,7 +588,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | ||
| 580 | 588 | table_size = sizeof(struct harvest_table); |
| 581 | 589 | table_name = "harvest table"; |
| 582 | 590 | break; |
| 583 | − case VCN_INFO: | |
| 591 | + } | |
| 592 | + case VCN_INFO: { | |
| 584 | 593 | struct vcn_info_header *vhdr = |
| 585 | 594 | (struct vcn_info_header *)(discovery_bin + offset); |
| 586 | 595 | act_val = le32_to_cpu(vhdr->table_id); |
| @@ -588,7 +597,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | ||
| 588 | 597 | table_size = le32_to_cpu(vhdr->size_bytes); |
| 589 | 598 | table_name = "vcn table"; |
| 590 | 599 | break; |
| 591 | − case MALL_INFO: | |
| 600 | + } | |
| 601 | + case MALL_INFO: { | |
| 592 | 602 | struct mall_info_header *mhdr = |
| 593 | 603 | (struct mall_info_header *)(discovery_bin + offset); |
| 594 | 604 | act_val = le32_to_cpu(mhdr->table_id); |
| @@ -597,6 +607,7 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 2 unmodified lines | ||
| 597 | 607 | table_name = "mall table"; |
| 598 | 608 | check_table = false; |
| 599 | 609 | break; |
| 610 | + } | |
| 600 | 611 | default: |
| 601 | 612 | dev_err(adev->dev, "invalid ip discovery table id %d specified\n", table_id); |
| 602 | 613 | check_table = false; |
| @@ -1383,6 +1394,9 @@ static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev) · 780 unmodified lines | ||
| 1383 | 1394 | struct list_head *el, *tmp; |
| 1384 | 1395 | struct kset *die_kset; |
| 1385 | 1396 | |
| 1397 | + if (!ip_top) | |
| 1398 | + return; | |
| 1399 | + | |
| 1386 | 1400 | die_kset = &ip_top->die_kset; |
| 1387 | 1401 | spin_lock(&die_kset->list_lock); |
| 1388 | 1402 | list_for_each_prev_safe(el, tmp, &die_kset->list) { |
| @@ -1407,9 +1421,13 @@ void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p) · 18 unmodified lines | ||
| 1407 | 1421 | struct ip_hw_instance *ip_inst; |
| 1408 | 1422 | int i = 0, j; |
| 1409 | 1423 | |
| 1424 | + if (!ip_top) | |
| 1425 | + return; | |
| 1426 | + | |
| 1410 | 1427 | die_kset = &ip_top->die_kset; |
| 1411 | 1428 | |
| 1412 | 1429 | drm_printf(p, "\nHW IP Discovery\n"); |
| 1430 | + | |
| 1413 | 1431 | spin_lock(&die_kset->list_lock); |
| 1414 | 1432 | list_for_each(el_die, &die_kset->list) { |
| 1415 | 1433 | drm_printf(p, "die %d\n", i++); |
| @@ -1966,11 +1984,10 @@ static int amdgpu_discovery_refresh_nps_info(struct amdgpu_device *adev, · 550 unmodified lines | ||
| 1966 | 1984 | |
| 1967 | 1985 | int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, |
| 1968 | 1986 | uint32_t *nps_type, |
| 1969 | − struct amdgpu_gmc_memrange **ranges, | |
| 1987 | + struct amdgpu_gmc_memrange *ranges, | |
| 1970 | 1988 | int *range_cnt, bool refresh) |
| 1971 | 1989 | { |
| 1972 | 1990 | uint8_t *discovery_bin = adev->discovery.bin; |
| 1973 | − struct amdgpu_gmc_memrange *mem_ranges; | |
| 1974 | 1991 | struct table_info *info; |
| 1975 | 1992 | union nps_info *nps_info; |
| 1976 | 1993 | union nps_info nps_data; |
| @@ -2008,20 +2025,22 @@ int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, · 31 unmodified lines | ||
| 2008 | 2025 | |
| 2009 | 2026 | switch (le16_to_cpu(nps_info->v1.header.version_major)) { |
| 2010 | 2027 | case 1: |
| 2011 | − mem_ranges = kvzalloc_objs(*mem_ranges, nps_info->v1.count); | |
| 2012 | − if (!mem_ranges) | |
| 2013 | − return -ENOMEM; | |
| 2014 | 2028 | *nps_type = nps_info->v1.nps_type; |
| 2029 | + if (*range_cnt < nps_info->v1.count) { | |
| 2030 | + dev_dbg(adev->dev, | |
| 2031 | + "not enough space for nps ranges: %d < %d\n", | |
| 2032 | + *range_cnt, nps_info->v1.count); | |
| 2033 | + return -ENOSPC; | |
| 2034 | + } | |
| 2015 | 2035 | *range_cnt = nps_info->v1.count; |
| 2016 | 2036 | for (i = 0; i < *range_cnt; i++) { |
| 2017 | − mem_ranges[i].base_address = | |
| 2037 | + ranges[i].base_address = | |
| 2018 | 2038 | nps_info->v1.instance_info[i].base_address; |
| 2019 | − mem_ranges[i].limit_address = | |
| 2039 | + ranges[i].limit_address = | |
| 2020 | 2040 | nps_info->v1.instance_info[i].limit_address; |
| 2021 | − mem_ranges[i].nid_mask = -1; | |
| 2022 | − mem_ranges[i].flags = 0; | |
| 2041 | + ranges[i].nid_mask = -1; | |
| 2042 | + ranges[i].flags = 0; | |
| 2023 | 2043 | } |
| 2024 | − *ranges = mem_ranges; | |
| 2025 | 2044 | break; |
| 2026 | 2045 | default: |
| 2027 | 2046 | dev_err(adev->dev, "Unhandled NPS info table %d.%d\n", |
| @@ -2323,6 +2342,7 @@ static int amdgpu_discovery_set_smu_ip_blocks(struct amdgpu_device *adev) · 295 unmodified lines | ||
| 2323 | 2342 | amdgpu_device_ip_block_add(adev, &smu_v14_0_ip_block); |
| 2324 | 2343 | break; |
| 2325 | 2344 | case IP_VERSION(15, 0, 0): |
| 2345 | + case IP_VERSION(15, 0, 8): | |
| 2326 | 2346 | amdgpu_device_ip_block_add(adev, &smu_v15_0_ip_block); |
| 2327 | 2347 | break; |
| 2328 | 2348 | default: |
| @@ -2640,6 +2660,10 @@ static int amdgpu_discovery_set_mm_ip_blocks(struct amdgpu_device *adev) · 311 unmodified lines | ||
| 2640 | 2660 | amdgpu_device_ip_block_add(adev, &vcn_v5_0_1_ip_block); |
| 2641 | 2661 | amdgpu_device_ip_block_add(adev, &jpeg_v5_0_1_ip_block); |
| 2642 | 2662 | break; |
| 2663 | + case IP_VERSION(5, 0, 2): | |
| 2664 | + amdgpu_device_ip_block_add(adev, &vcn_v5_0_2_ip_block); | |
| 2665 | + amdgpu_device_ip_block_add(adev, &jpeg_v5_0_2_ip_block); | |
| 2666 | + break; | |
| 2643 | 2667 | default: |
| 2644 | 2668 | dev_err(adev->dev, |
| 2645 | 2669 | "Failed to add vcn/jpeg ip block(UVD_HWIP:0x%x)\n", |
| @@ -46,7 +46,7 @@ int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev); · 45 unmodified lines | ||
| 46 | 46 | |
| 47 | 47 | int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, |
| 48 | 48 | uint32_t *nps_type, |
| 49 | − struct amdgpu_gmc_memrange **ranges, | |
| 49 | + struct amdgpu_gmc_memrange *ranges, | |
| 50 | 50 | int *range_cnt, bool refresh); |
| 51 | 51 | |
| 52 | 52 | void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p); |
| @@ -839,8 +839,8 @@ module_param_named_unsafe(no_queue_eviction_on_vm_fault, amdgpu_no_queue_evictio · 838 unmodified lines | ||
| 839 | 839 | /** |
| 840 | 840 | * DOC: mtype_local (int) |
| 841 | 841 | */ |
| 842 | −int amdgpu_mtype_local; | |
| 843 | −MODULE_PARM_DESC(mtype_local, "MTYPE for local memory (0 = MTYPE_RW (default), 1 = MTYPE_NC, 2 = MTYPE_CC)"); | |
| 842 | +int amdgpu_mtype_local = -1; | |
| 843 | +MODULE_PARM_DESC(mtype_local, "MTYPE for local memory (default: ASIC dependent, 0 = MTYPE_RW, 1 = MTYPE_NC, 2 = MTYPE_CC)"); | |
| 844 | 844 | module_param_named_unsafe(mtype_local, amdgpu_mtype_local, int, 0444); |
| 845 | 845 | |
| 846 | 846 | /** |
| @@ -2683,8 +2683,12 @@ static int amdgpu_pmops_freeze(struct device *dev) · 1836 unmodified lines | ||
| 2683 | 2683 | if (r) |
| 2684 | 2684 | return r; |
| 2685 | 2685 | |
| 2686 | − if (amdgpu_acpi_should_gpu_reset(adev)) | |
| 2687 | − return amdgpu_asic_reset(adev); | |
| 2686 | + if (amdgpu_acpi_should_gpu_reset(adev)) { | |
| 2687 | + amdgpu_device_lock_reset_domain(adev->reset_domain); | |
| 2688 | + r = amdgpu_asic_reset(adev); | |
| 2689 | + amdgpu_device_unlock_reset_domain(adev->reset_domain); | |
| 2690 | + return r; | |
| 2691 | + } | |
| 2688 | 2692 | return 0; |
| 2689 | 2693 | } |
| 2690 | 2694 | |
| @@ -2952,9 +2956,11 @@ static int amdgpu_drm_release(struct inode *inode, struct file *filp) · 261 unmodified lines | ||
| 2952 | 2956 | int idx; |
| 2953 | 2957 | |
| 2954 | 2958 | if (fpriv && drm_dev_enter(dev, &idx)) { |
| 2955 | − fpriv->evf_mgr.fd_closing = true; | |
| 2956 | − amdgpu_eviction_fence_destroy(&fpriv->evf_mgr); | |
| 2959 | + amdgpu_evf_mgr_shutdown(&fpriv->evf_mgr); | |
| 2960 | + amdgpu_userq_mgr_cancel_resume(&fpriv->userq_mgr); | |
| 2961 | + amdgpu_evf_mgr_flush_suspend(&fpriv->evf_mgr); | |
| 2957 | 2962 | amdgpu_userq_mgr_fini(&fpriv->userq_mgr); |
| 2963 | + amdgpu_evf_mgr_fini(&fpriv->evf_mgr); | |
| 2958 | 2964 | drm_dev_exit(idx); |
| 2959 | 2965 | } |
| 2960 | 2966 | |
| @@ -25,9 +25,6 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <drm/drm_exec.h> |
| 26 | 26 | #include "amdgpu.h" |
| 27 | 27 | |
| 28 | −#define work_to_evf_mgr(w, name) container_of(w, struct amdgpu_eviction_fence_mgr, name) | |
| 29 | −#define evf_mgr_to_fpriv(e) container_of(e, struct amdgpu_fpriv, evf_mgr) | |
| 30 | − | |
| 31 | 28 | static const char * |
| 32 | 29 | amdgpu_eviction_fence_get_driver_name(struct dma_fence *fence) |
| 33 | 30 | { |
| @@ -43,127 +40,93 @@ amdgpu_eviction_fence_get_timeline_name(struct dma_fence *f) · 9 unmodified lines | ||
| 43 | 40 | return ef->timeline_name; |
| 44 | 41 | } |
| 45 | 42 | |
| 46 | −int | |
| 47 | −amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 48 | − struct drm_exec *exec) | |
| 43 | +static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f) | |
| 49 | 44 | { |
| 50 | − struct amdgpu_eviction_fence *old_ef, *new_ef; | |
| 51 | − struct drm_gem_object *obj; | |
| 52 | − unsigned long index; | |
| 53 | − int ret; | |
| 54 | − | |
| 55 | − if (evf_mgr->ev_fence && | |
| 56 | − !dma_fence_is_signaled(&evf_mgr->ev_fence->base)) | |
| 57 | − return 0; | |
| 58 | − /* | |
| 59 | − * Steps to replace eviction fence: | |
| 60 | − * * lock all objects in exec (caller) | |
| 61 | − * * create a new eviction fence | |
| 62 | − * * update new eviction fence in evf_mgr | |
| 63 | − * * attach the new eviction fence to BOs | |
| 64 | − * * release the old fence | |
| 65 | − * * unlock the objects (caller) | |
| 66 | − */ | |
| 67 | − new_ef = amdgpu_eviction_fence_create(evf_mgr); | |
| 68 | − if (!new_ef) { | |
| 69 | − DRM_ERROR("Failed to create new eviction fence\n"); | |
| 70 | − return -ENOMEM; | |
| 71 | − } | |
| 72 | − | |
| 73 | − /* Update the eviction fence now */ | |
| 74 | − spin_lock(&evf_mgr->ev_fence_lock); | |
| 75 | − old_ef = evf_mgr->ev_fence; | |
| 76 | − evf_mgr->ev_fence = new_ef; | |
| 77 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 45 | + struct amdgpu_eviction_fence *ev_fence = to_ev_fence(f); | |
| 78 | 46 | |
| 79 | − /* Attach the new fence */ | |
| 80 | − drm_exec_for_each_locked_object(exec, index, obj) { | |
| 81 | − struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | |
| 82 | − | |
| 83 | − if (!bo) | |
| 84 | − continue; | |
| 85 | − ret = amdgpu_eviction_fence_attach(evf_mgr, bo); | |
| 86 | − if (ret) { | |
| 87 | − DRM_ERROR("Failed to attch new eviction fence\n"); | |
| 88 | − goto free_err; | |
| 89 | − } | |
| 90 | − } | |
| 91 | − | |
| 92 | − /* Free old fence */ | |
| 93 | − if (old_ef) | |
| 94 | − dma_fence_put(&old_ef->base); | |
| 95 | − return 0; | |
| 96 | − | |
| 97 | −free_err: | |
| 98 | − kfree(new_ef); | |
| 99 | − return ret; | |
| 47 | + schedule_work(&ev_fence->evf_mgr->suspend_work); | |
| 48 | + return true; | |
| 100 | 49 | } |
| 101 | 50 | |
| 51 | +static const struct dma_fence_ops amdgpu_eviction_fence_ops = { | |
| 52 | + .get_driver_name = amdgpu_eviction_fence_get_driver_name, | |
| 53 | + .get_timeline_name = amdgpu_eviction_fence_get_timeline_name, | |
| 54 | + .enable_signaling = amdgpu_eviction_fence_enable_signaling, | |
| 55 | +}; | |
| 56 | + | |
| 102 | 57 | static void |
| 103 | 58 | amdgpu_eviction_fence_suspend_worker(struct work_struct *work) |
| 104 | 59 | { |
| 105 | − struct amdgpu_eviction_fence_mgr *evf_mgr = work_to_evf_mgr(work, suspend_work.work); | |
| 106 | − struct amdgpu_fpriv *fpriv = evf_mgr_to_fpriv(evf_mgr); | |
| 60 | + struct amdgpu_eviction_fence_mgr *evf_mgr = | |
| 61 | + container_of(work, struct amdgpu_eviction_fence_mgr, | |
| 62 | + suspend_work); | |
| 63 | + struct amdgpu_fpriv *fpriv = | |
| 64 | + container_of(evf_mgr, struct amdgpu_fpriv, evf_mgr); | |
| 107 | 65 | struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr; |
| 108 | − struct amdgpu_eviction_fence *ev_fence; | |
| 66 | + struct dma_fence *ev_fence; | |
| 67 | + bool cookie; | |
| 109 | 68 | |
| 110 | 69 | mutex_lock(&uq_mgr->userq_mutex); |
| 111 | − spin_lock(&evf_mgr->ev_fence_lock); | |
| 112 | − ev_fence = evf_mgr->ev_fence; | |
| 113 | − if (ev_fence) | |
| 114 | − dma_fence_get(&ev_fence->base); | |
| 115 | − else | |
| 116 | − goto unlock; | |
| 117 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 118 | 70 | |
| 119 | − amdgpu_userq_evict(uq_mgr, ev_fence); | |
| 71 | + /* | |
| 72 | + * This is intentionally after taking the userq_mutex since we do | |
| 73 | + * allocate memory while holding this lock, but only after ensuring that | |
| 74 | + * the eviction fence is signaled. | |
| 75 | + */ | |
| 76 | + cookie = dma_fence_begin_signalling(); | |
| 120 | 77 | |
| 121 | − mutex_unlock(&uq_mgr->userq_mutex); | |
| 122 | − dma_fence_put(&ev_fence->base); | |
| 123 | − return; | |
| 78 | + ev_fence = amdgpu_evf_mgr_get_fence(evf_mgr); | |
| 79 | + amdgpu_userq_evict(uq_mgr); | |
| 80 | + | |
| 81 | + /* | |
| 82 | + * Signaling the eviction fence must be done while holding the | |
| 83 | + * userq_mutex. Otherwise we won't resume the queues before issuing the | |
| 84 | + * next fence. | |
| 85 | + */ | |
| 86 | + dma_fence_signal(ev_fence); | |
| 87 | + dma_fence_end_signalling(cookie); | |
| 88 | + dma_fence_put(ev_fence); | |
| 89 | + | |
| 90 | + if (!evf_mgr->shutdown) | |
| 91 | + schedule_delayed_work(&uq_mgr->resume_work, 0); | |
| 124 | 92 | |
| 125 | −unlock: | |
| 126 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 127 | 93 | mutex_unlock(&uq_mgr->userq_mutex); |
| 128 | 94 | } |
| 129 | 95 | |
| 130 | −static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f) | |
| 96 | +int amdgpu_evf_mgr_attach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 97 | + struct amdgpu_bo *bo) | |
| 131 | 98 | { |
| 132 | − struct amdgpu_eviction_fence_mgr *evf_mgr; | |
| 133 | − struct amdgpu_eviction_fence *ev_fence; | |
| 134 | − | |
| 135 | − if (!f) | |
| 136 | − return true; | |
| 137 | − | |
| 138 | − ev_fence = to_ev_fence(f); | |
| 139 | − evf_mgr = ev_fence->evf_mgr; | |
| 99 | + struct dma_fence *ev_fence = amdgpu_evf_mgr_get_fence(evf_mgr); | |
| 100 | + struct ttm_operation_ctx ctx = { false, false }; | |
| 101 | + struct dma_resv *resv = bo->tbo.base.resv; | |
| 102 | + int ret; | |
| 140 | 103 | |
| 141 | − schedule_delayed_work(&evf_mgr->suspend_work, 0); | |
| 142 | − return true; | |
| 143 | −} | |
| 104 | + if (!dma_fence_is_signaled(ev_fence)) { | |
| 144 | 105 | |
| 145 | −static const struct dma_fence_ops amdgpu_eviction_fence_ops = { | |
| 146 | − .get_driver_name = amdgpu_eviction_fence_get_driver_name, | |
| 147 | − .get_timeline_name = amdgpu_eviction_fence_get_timeline_name, | |
| 148 | − .enable_signaling = amdgpu_eviction_fence_enable_signaling, | |
| 149 | −}; | |
| 106 | + amdgpu_bo_placement_from_domain(bo, bo->allowed_domains); | |
| 107 | + ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); | |
| 108 | + if (!ret) | |
| 109 | + dma_resv_add_fence(resv, ev_fence, | |
| 110 | + DMA_RESV_USAGE_BOOKKEEP); | |
| 111 | + } else { | |
| 112 | + ret = 0; | |
| 113 | + } | |
| 150 | 114 | |
| 151 | −void amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 152 | − struct amdgpu_eviction_fence *ev_fence) | |
| 153 | −{ | |
| 154 | − spin_lock(&evf_mgr->ev_fence_lock); | |
| 155 | − dma_fence_signal(&ev_fence->base); | |
| 156 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 115 | + dma_fence_put(ev_fence); | |
| 116 | + return ret; | |
| 157 | 117 | } |
| 158 | 118 | |
| 159 | −struct amdgpu_eviction_fence * | |
| 160 | −amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 119 | +int amdgpu_evf_mgr_rearm(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 120 | + struct drm_exec *exec) | |
| 161 | 121 | { |
| 162 | 122 | struct amdgpu_eviction_fence *ev_fence; |
| 123 | + struct drm_gem_object *obj; | |
| 124 | + unsigned long index; | |
| 163 | 125 | |
| 126 | + /* Create and initialize a new eviction fence */ | |
| 164 | 127 | ev_fence = kzalloc_obj(*ev_fence); |
| 165 | 128 | if (!ev_fence) |
| 166 | − return NULL; | |
| 129 | + return -ENOMEM; | |
| 167 | 130 | |
| 168 | 131 | ev_fence->evf_mgr = evf_mgr; |
| 169 | 132 | get_task_comm(ev_fence->timeline_name, current); |
| @@ -171,56 +134,22 @@ amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr) · 1 unmodified line | ||
| 171 | 134 | dma_fence_init64(&ev_fence->base, &amdgpu_eviction_fence_ops, |
| 172 | 135 | &ev_fence->lock, evf_mgr->ev_fence_ctx, |
| 173 | 136 | atomic_inc_return(&evf_mgr->ev_fence_seq)); |
| 174 | − return ev_fence; | |
| 175 | −} | |
| 176 | − | |
| 177 | −void amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 178 | −{ | |
| 179 | − struct amdgpu_eviction_fence *ev_fence; | |
| 180 | − | |
| 181 | − /* Wait for any pending work to execute */ | |
| 182 | − flush_delayed_work(&evf_mgr->suspend_work); | |
| 183 | − | |
| 184 | − spin_lock(&evf_mgr->ev_fence_lock); | |
| 185 | − ev_fence = evf_mgr->ev_fence; | |
| 186 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 187 | − | |
| 188 | − if (!ev_fence) | |
| 189 | − return; | |
| 190 | − | |
| 191 | − dma_fence_wait(&ev_fence->base, false); | |
| 192 | − | |
| 193 | − /* Last unref of ev_fence */ | |
| 194 | − dma_fence_put(&ev_fence->base); | |
| 195 | −} | |
| 196 | 137 | |
| 197 | −int amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 198 | − struct amdgpu_bo *bo) | |
| 199 | −{ | |
| 200 | − struct amdgpu_eviction_fence *ev_fence; | |
| 201 | − struct dma_resv *resv = bo->tbo.base.resv; | |
| 202 | − int ret; | |
| 138 | + /* Remember it for newly added BOs */ | |
| 139 | + dma_fence_put(evf_mgr->ev_fence); | |
| 140 | + evf_mgr->ev_fence = &ev_fence->base; | |
| 203 | 141 | |
| 204 | − if (!resv) | |
| 205 | − return 0; | |
| 142 | + /* And add it to all existing BOs */ | |
| 143 | + drm_exec_for_each_locked_object(exec, index, obj) { | |
| 144 | + struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | |
| 206 | 145 | |
| 207 | − ret = dma_resv_reserve_fences(resv, 1); | |
| 208 | − if (ret) { | |
| 209 | − DRM_DEBUG_DRIVER("Failed to resv fence space\n"); | |
| 210 | − return ret; | |
| 146 | + amdgpu_evf_mgr_attach_fence(evf_mgr, bo); | |
| 211 | 147 | } |
| 212 | − | |
| 213 | − spin_lock(&evf_mgr->ev_fence_lock); | |
| 214 | − ev_fence = evf_mgr->ev_fence; | |
| 215 | − if (ev_fence) | |
| 216 | − dma_resv_add_fence(resv, &ev_fence->base, DMA_RESV_USAGE_BOOKKEEP); | |
| 217 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 218 | − | |
| 219 | 148 | return 0; |
| 220 | 149 | } |
| 221 | 150 | |
| 222 | −void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 223 | − struct amdgpu_bo *bo) | |
| 151 | +void amdgpu_evf_mgr_detach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 152 | + struct amdgpu_bo *bo) | |
| 224 | 153 | { |
| 225 | 154 | struct dma_fence *stub = dma_fence_get_stub(); |
| 226 | 155 | |
| @@ -229,13 +158,31 @@ void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, · 2 unmodified lines | ||
| 229 | 158 | dma_fence_put(stub); |
| 230 | 159 | } |
| 231 | 160 | |
| 232 | −int amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 161 | +void amdgpu_evf_mgr_init(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 233 | 162 | { |
| 234 | − /* This needs to be done one time per open */ | |
| 235 | 163 | atomic_set(&evf_mgr->ev_fence_seq, 0); |
| 236 | 164 | evf_mgr->ev_fence_ctx = dma_fence_context_alloc(1); |
| 237 | − spin_lock_init(&evf_mgr->ev_fence_lock); | |
| 165 | + evf_mgr->ev_fence = dma_fence_get_stub(); | |
| 238 | 166 | |
| 239 | − INIT_DELAYED_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker); | |
| 240 | − return 0; | |
| 167 | + INIT_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker); | |
| 168 | +} | |
| 169 | + | |
| 170 | +void amdgpu_evf_mgr_shutdown(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 171 | +{ | |
| 172 | + evf_mgr->shutdown = true; | |
| 173 | + /* Make sure that the shutdown is visible to the suspend work */ | |
| 174 | + flush_work(&evf_mgr->suspend_work); | |
| 175 | +} | |
| 176 | + | |
| 177 | +void amdgpu_evf_mgr_flush_suspend(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 178 | +{ | |
| 179 | + dma_fence_wait(rcu_dereference_protected(evf_mgr->ev_fence, true), | |
| 180 | + false); | |
| 181 | + /* Make sure that we are done with the last suspend work */ | |
| 182 | + flush_work(&evf_mgr->suspend_work); | |
| 183 | +} | |
| 184 | + | |
| 185 | +void amdgpu_evf_mgr_fini(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 186 | +{ | |
| 187 | + dma_fence_put(evf_mgr->ev_fence); | |
| 241 | 188 | } |
| @@ -25,6 +25,8 @@ · 24 unmodified lines | ||
| 25 | 25 | #ifndef AMDGPU_EV_FENCE_H_ |
| 26 | 26 | #define AMDGPU_EV_FENCE_H_ |
| 27 | 27 | |
| 28 | +#include <linux/dma-fence.h> | |
| 29 | + | |
| 28 | 30 | struct amdgpu_eviction_fence { |
| 29 | 31 | struct dma_fence base; |
| 30 | 32 | spinlock_t lock; |
| @@ -35,35 +37,36 @@ struct amdgpu_eviction_fence { · 4 unmodified lines | ||
| 35 | 37 | struct amdgpu_eviction_fence_mgr { |
| 36 | 38 | u64 ev_fence_ctx; |
| 37 | 39 | atomic_t ev_fence_seq; |
| 38 | − spinlock_t ev_fence_lock; | |
| 39 | − struct amdgpu_eviction_fence *ev_fence; | |
| 40 | − struct delayed_work suspend_work; | |
| 41 | − uint8_t fd_closing; | |
| 42 | −}; | |
| 43 | − | |
| 44 | −/* Eviction fence helper functions */ | |
| 45 | −struct amdgpu_eviction_fence * | |
| 46 | −amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 47 | 40 | |
| 48 | −void | |
| 49 | −amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 50 | − | |
| 51 | −int | |
| 52 | −amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 53 | − struct amdgpu_bo *bo); | |
| 41 | + /* | |
| 42 | + * Only updated while holding the VM resv lock. | |
| 43 | + * Only signaled while holding the userq mutex. | |
| 44 | + */ | |
| 45 | + struct dma_fence __rcu *ev_fence; | |
| 46 | + struct work_struct suspend_work; | |
| 47 | + bool shutdown; | |
| 48 | +}; | |
| 54 | 49 | |
| 55 | −void | |
| 56 | −amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 57 | − struct amdgpu_bo *bo); | |
| 50 | +static inline struct dma_fence * | |
| 51 | +amdgpu_evf_mgr_get_fence(struct amdgpu_eviction_fence_mgr *evf_mgr) | |
| 52 | +{ | |
| 53 | + struct dma_fence *ev_fence; | |
| 58 | 54 | |
| 59 | −int | |
| 60 | −amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 55 | + rcu_read_lock(); | |
| 56 | + ev_fence = dma_fence_get_rcu_safe(&evf_mgr->ev_fence); | |
| 57 | + rcu_read_unlock(); | |
| 58 | + return ev_fence; | |
| 59 | +} | |
| 61 | 60 | |
| 62 | −void | |
| 63 | −amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 64 | − struct amdgpu_eviction_fence *ev_fence); | |
| 61 | +int amdgpu_evf_mgr_attach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 62 | + struct amdgpu_bo *bo); | |
| 63 | +int amdgpu_evf_mgr_rearm(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 64 | + struct drm_exec *exec); | |
| 65 | +void amdgpu_evf_mgr_detach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 66 | + struct amdgpu_bo *bo); | |
| 67 | +void amdgpu_evf_mgr_init(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 68 | +void amdgpu_evf_mgr_shutdown(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 69 | +void amdgpu_evf_mgr_flush_suspend(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 70 | +void amdgpu_evf_mgr_fini(struct amdgpu_eviction_fence_mgr *evf_mgr); | |
| 65 | 71 | |
| 66 | −int | |
| 67 | −amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | |
| 68 | − struct drm_exec *exec); | |
| 69 | 72 | #endif |
| @@ -438,7 +438,7 @@ int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring, · 437 unmodified lines | ||
| 438 | 438 | * @ring: ring to init the fence driver on |
| 439 | 439 | * |
| 440 | 440 | * Init the fence driver for the requested ring (all asics). |
| 441 | − * Helper function for amdgpu_fence_driver_init(). | |
| 441 | + * Helper function for amdgpu_fence_driver_sw_init(). | |
| 442 | 442 | */ |
| 443 | 443 | int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring) |
| 444 | 444 | { |
| @@ -106,6 +106,7 @@ static bool is_fru_eeprom_supported(struct amdgpu_device *adev, u32 *fru_addr) · 105 unmodified lines | ||
| 106 | 106 | *fru_addr = FRU_EEPROM_MADDR_8; |
| 107 | 107 | return true; |
| 108 | 108 | case IP_VERSION(13, 0, 12): |
| 109 | + case IP_VERSION(15, 0, 8): | |
| 109 | 110 | if (fru_addr) |
| 110 | 111 | *fru_addr = FRU_EEPROM_MADDR_INV; |
| 111 | 112 | return true; |
| @@ -403,6 +403,50 @@ void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, · 402 unmodified lines | ||
| 403 | 403 | drm_dev_exit(idx); |
| 404 | 404 | } |
| 405 | 405 | |
| 406 | +/** | |
| 407 | + * amdgpu_gart_map_gfx9_mqd - map mqd and ctrl_stack dma_addresses into GART entries | |
| 408 | + * | |
| 409 | + * @adev: amdgpu_device pointer | |
| 410 | + * @offset: offset into the GPU's gart aperture | |
| 411 | + * @pages: number of pages to bind | |
| 412 | + * @dma_addr: DMA addresses of pages | |
| 413 | + * @flags: page table entry flags | |
| 414 | + * | |
| 415 | + * Map the MQD and control stack addresses into GART entries with the correct | |
| 416 | + * memory types on gfxv9. The MQD occupies the first 4KB and is followed by | |
| 417 | + * the control stack. The MQD uses UC (uncached) memory, while the control stack | |
| 418 | + * uses NC (non-coherent) memory. | |
| 419 | + */ | |
| 420 | +void amdgpu_gart_map_gfx9_mqd(struct amdgpu_device *adev, uint64_t offset, | |
| 421 | + int pages, dma_addr_t *dma_addr, uint64_t flags) | |
| 422 | +{ | |
| 423 | + uint64_t page_base; | |
| 424 | + unsigned int i, j, t; | |
| 425 | + int idx; | |
| 426 | + uint64_t ctrl_flags = AMDGPU_PTE_MTYPE_VG10(flags, AMDGPU_MTYPE_NC); | |
| 427 | + void *dst; | |
| 428 | + | |
| 429 | + if (!adev->gart.ptr) | |
| 430 | + return; | |
| 431 | + | |
| 432 | + if (!drm_dev_enter(adev_to_drm(adev), &idx)) | |
| 433 | + return; | |
| 434 | + | |
| 435 | + t = offset / AMDGPU_GPU_PAGE_SIZE; | |
| 436 | + dst = adev->gart.ptr; | |
| 437 | + for (i = 0; i < pages; i++) { | |
| 438 | + page_base = dma_addr[i]; | |
| 439 | + for (j = 0; j < AMDGPU_GPU_PAGES_IN_CPU_PAGE; j++, t++) { | |
| 440 | + if ((i == 0) && (j == 0)) | |
| 441 | + amdgpu_gmc_set_pte_pde(adev, dst, t, page_base, flags); | |
| 442 | + else | |
| 443 | + amdgpu_gmc_set_pte_pde(adev, dst, t, page_base, ctrl_flags); | |
| 444 | + page_base += AMDGPU_GPU_PAGE_SIZE; | |
| 445 | + } | |
| 446 | + } | |
| 447 | + drm_dev_exit(idx); | |
| 448 | +} | |
| 449 | + | |
| 406 | 450 | /** |
| 407 | 451 | * amdgpu_gart_bind - bind pages into the gart page table |
| 408 | 452 | * |
| @@ -62,6 +62,8 @@ void amdgpu_gart_unbind(struct amdgpu_device *adev, uint64_t offset, · 61 unmodified lines | ||
| 62 | 62 | void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset, |
| 63 | 63 | int pages, dma_addr_t *dma_addr, uint64_t flags, |
| 64 | 64 | void *dst); |
| 65 | +void amdgpu_gart_map_gfx9_mqd(struct amdgpu_device *adev, uint64_t offset, | |
| 66 | + int pages, dma_addr_t *dma_addr, uint64_t flags); | |
| 65 | 67 | void amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t offset, |
| 66 | 68 | int pages, dma_addr_t *dma_addr, uint64_t flags); |
| 67 | 69 | void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, |
| @@ -30,6 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <linux/pagemap.h> |
| 31 | 31 | #include <linux/pci.h> |
| 32 | 32 | #include <linux/dma-buf.h> |
| 33 | +#include <linux/dma-fence-unwrap.h> | |
| 33 | 34 | |
| 34 | 35 | #include <drm/amdgpu_drm.h> |
| 35 | 36 | #include <drm/drm_drv.h> |
| @@ -106,6 +107,7 @@ amdgpu_gem_update_timeline_node(struct drm_file *filp, · 70 unmodified lines | ||
| 106 | 107 | *chain = dma_fence_chain_alloc(); |
| 107 | 108 | if (!*chain) { |
| 108 | 109 | drm_syncobj_put(*syncobj); |
| 110 | + *syncobj = NULL; | |
| 109 | 111 | return -ENOMEM; |
| 110 | 112 | } |
| 111 | 113 | |
| @@ -258,18 +260,15 @@ static int amdgpu_gem_object_open(struct drm_gem_object *obj, · 146 unmodified lines | ||
| 258 | 260 | |
| 259 | 261 | amdgpu_vm_bo_update_shared(abo); |
| 260 | 262 | bo_va = amdgpu_vm_bo_find(vm, abo); |
| 261 | − if (!bo_va) | |
| 263 | + if (!bo_va) { | |
| 262 | 264 | bo_va = amdgpu_vm_bo_add(adev, vm, abo); |
| 263 | − else | |
| 265 | + r = amdgpu_evf_mgr_attach_fence(&fpriv->evf_mgr, abo); | |
| 266 | + if (r) | |
| 267 | + goto out_unlock; | |
| 268 | + } else { | |
| 264 | 269 | ++bo_va->ref_count; |
| 265 | − | |
| 266 | − /* attach gfx eviction fence */ | |
| 267 | − r = amdgpu_eviction_fence_attach(&fpriv->evf_mgr, abo); | |
| 268 | − if (r) { | |
| 269 | − DRM_DEBUG_DRIVER("Failed to attach eviction fence to BO\n"); | |
| 270 | − amdgpu_bo_unreserve(abo); | |
| 271 | − return r; | |
| 272 | 270 | } |
| 271 | + | |
| 273 | 272 | drm_exec_fini(&exec); |
| 274 | 273 | |
| 275 | 274 | /* Validate and add eviction fence to DMABuf imports with dynamic |
| @@ -337,7 +336,7 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj, · 61 unmodified lines | ||
| 337 | 336 | } |
| 338 | 337 | |
| 339 | 338 | if (!amdgpu_vm_is_bo_always_valid(vm, bo)) |
| 340 | − amdgpu_eviction_fence_detach(&fpriv->evf_mgr, bo); | |
| 339 | + amdgpu_evf_mgr_detach_fence(&fpriv->evf_mgr, bo); | |
| 341 | 340 | |
| 342 | 341 | bo_va = amdgpu_vm_bo_find(vm, bo); |
| 343 | 342 | if (!bo_va || --bo_va->ref_count) |
| @@ -744,11 +743,10 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 400 unmodified lines | ||
| 744 | 743 | struct dma_fence *fence; |
| 745 | 744 | int r = 0; |
| 746 | 745 | |
| 747 | − /* Always start from the VM's existing last update fence. */ | |
| 748 | − fence = dma_fence_get(vm->last_update); | |
| 749 | − | |
| 746 | + /* If the VM is not ready return only a stub. */ | |
| 750 | 747 | if (!amdgpu_vm_ready(vm)) |
| 751 | − return fence; | |
| 748 | + return dma_fence_get_stub(); | |
| 749 | + | |
| 752 | 750 | |
| 753 | 751 | /* |
| 754 | 752 | * First clean up any freed mappings in the VM. |
| @@ -757,7 +755,7 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 2 unmodified lines | ||
| 757 | 755 | * schedules GPU work. If nothing needs clearing, @fence can remain as |
| 758 | 756 | * the original vm->last_update. |
| 759 | 757 | */ |
| 760 | − r = amdgpu_vm_clear_freed(adev, vm, &fence); | |
| 758 | + r = amdgpu_vm_clear_freed(adev, vm, &vm->last_update); | |
| 761 | 759 | if (r) |
| 762 | 760 | goto error; |
| 763 | 761 | |
| @@ -774,47 +772,34 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 10 unmodified lines | ||
| 774 | 772 | if (r) |
| 775 | 773 | goto error; |
| 776 | 774 | |
| 777 | − /* | |
| 778 | − * Decide which fence best represents the last update: | |
| 779 | − * | |
| 780 | − * MAP/REPLACE: | |
| 781 | − * - For always-valid mappings, use vm->last_update. | |
| 782 | − * - Otherwise, export bo_va->last_pt_update. | |
| 783 | − * | |
| 784 | − * UNMAP/CLEAR: | |
| 785 | − * Keep the fence returned by amdgpu_vm_clear_freed(). If no work was | |
| 786 | − * needed, it can remain as vm->last_pt_update. | |
| 787 | − * | |
| 788 | − * The VM and BO update fences are always initialized to a valid value. | |
| 789 | − * vm->last_update and bo_va->last_pt_update always start as valid fences. | |
| 790 | − * and are never expected to be NULL. | |
| 791 | − */ | |
| 792 | − switch (operation) { | |
| 793 | − case AMDGPU_VA_OP_MAP: | |
| 794 | − case AMDGPU_VA_OP_REPLACE: | |
| 775 | + if ((operation == AMDGPU_VA_OP_MAP || | |
| 776 | + operation == AMDGPU_VA_OP_REPLACE) && | |
| 777 | + !amdgpu_vm_is_bo_always_valid(vm, bo_va->base.bo)) { | |
| 778 | + | |
| 795 | 779 | /* |
| 796 | − * For MAP/REPLACE, return the page table update fence for the | |
| 797 | − * mapping we just modified. bo_va is expected to be valid here. | |
| 780 | + * For MAP/REPLACE of non per-VM BOs we need to sync to both the | |
| 781 | + * bo_va->last_pt_update and vm->last_update or otherwise we | |
| 782 | + * potentially miss the PDE updates. | |
| 798 | 783 | */ |
| 799 | − dma_fence_put(fence); | |
| 800 | − | |
| 801 | − if (amdgpu_vm_is_bo_always_valid(vm, bo_va->base.bo)) | |
| 802 | − fence = dma_fence_get(vm->last_update); | |
| 803 | − else | |
| 804 | − fence = dma_fence_get(bo_va->last_pt_update); | |
| 805 | − break; | |
| 806 | − case AMDGPU_VA_OP_UNMAP: | |
| 807 | − case AMDGPU_VA_OP_CLEAR: | |
| 808 | − default: | |
| 809 | − /* keep @fence as returned by amdgpu_vm_clear_freed() */ | |
| 810 | − break; | |
| 784 | + fence = dma_fence_unwrap_merge(vm->last_update, | |
| 785 | + bo_va->last_pt_update); | |
| 786 | + if (!fence) { | |
| 787 | + /* As fallback in OOM situations */ | |
| 788 | + dma_fence_wait(vm->last_update, false); | |
| 789 | + dma_fence_wait(bo_va->last_pt_update, false); | |
| 790 | + fence = dma_fence_get_stub(); | |
| 791 | + } | |
| 792 | + } else { | |
| 793 | + fence = dma_fence_get(vm->last_update); | |
| 811 | 794 | } |
| 812 | 795 | |
| 796 | + return fence; | |
| 797 | + | |
| 813 | 798 | error: |
| 814 | 799 | if (r && r != -ERESTARTSYS) |
| 815 | 800 | DRM_ERROR("Couldn't update BO_VA (%d)\n", r); |
| 816 | 801 | |
| 817 | − return fence; | |
| 802 | + return dma_fence_get(vm->last_update); | |
| 818 | 803 | } |
| 819 | 804 | |
| 820 | 805 | int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, |
| @@ -835,7 +820,6 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 14 unmodified lines | ||
| 835 | 820 | struct amdgpu_bo_va *bo_va; |
| 836 | 821 | struct drm_syncobj *timeline_syncobj = NULL; |
| 837 | 822 | struct dma_fence_chain *timeline_chain = NULL; |
| 838 | − struct dma_fence *fence; | |
| 839 | 823 | struct drm_exec exec; |
| 840 | 824 | uint64_t vm_size; |
| 841 | 825 | int r = 0; |
| @@ -887,6 +871,10 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 45 unmodified lines | ||
| 887 | 871 | return -EINVAL; |
| 888 | 872 | } |
| 889 | 873 | |
| 874 | + if (args->flags & AMDGPU_VM_DELAY_UPDATE && | |
| 875 | + args->vm_timeline_syncobj_out) | |
| 876 | + return -EINVAL; | |
| 877 | + | |
| 890 | 878 | if ((args->operation != AMDGPU_VA_OP_CLEAR) && |
| 891 | 879 | !(args->flags & AMDGPU_VM_PAGE_PRT)) { |
| 892 | 880 | gobj = drm_gem_object_lookup(filp, args->handle); |
| @@ -976,11 +964,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 83 unmodified lines | ||
| 976 | 964 | * that represents the last relevant update for this mapping. This |
| 977 | 965 | * fence can then be exported to the user-visible VM timeline. |
| 978 | 966 | */ |
| 979 | − if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm) { | |
| 967 | + if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && | |
| 968 | + (!adev->debug_vm || timeline_syncobj)) { | |
| 969 | + struct dma_fence *fence; | |
| 970 | + | |
| 980 | 971 | fence = amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va, |
| 981 | 972 | args->operation); |
| 982 | − | |
| 983 | − if (timeline_syncobj && fence) { | |
| 973 | + if (timeline_syncobj) { | |
| 984 | 974 | if (!args->vm_timeline_point) { |
| 985 | 975 | /* Replace the existing fence when no point is given. */ |
| 986 | 976 | drm_syncobj_replace_fence(timeline_syncobj, |
| @@ -991,6 +981,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 4 unmodified lines | ||
| 991 | 981 | timeline_chain, |
| 992 | 982 | fence, |
| 993 | 983 | args->vm_timeline_point); |
| 984 | + timeline_chain = NULL; | |
| 994 | 985 | } |
| 995 | 986 | } |
| 996 | 987 | dma_fence_put(fence); |
| @@ -998,6 +989,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 1 unmodified line | ||
| 998 | 989 | } |
| 999 | 990 | |
| 1000 | 991 | error: |
| 992 | + dma_fence_chain_free(timeline_chain); | |
| 993 | + if (timeline_syncobj) | |
| 994 | + drm_syncobj_put(timeline_syncobj); | |
| 1001 | 995 | drm_exec_fini(&exec); |
| 1002 | 996 | error_put_gobj: |
| 1003 | 997 | drm_gem_object_put(gobj); |
| @@ -463,6 +463,7 @@ struct amdgpu_gfx { · 462 unmodified lines | ||
| 463 | 463 | struct amdgpu_irq_src cp_ecc_error_irq; |
| 464 | 464 | struct amdgpu_irq_src sq_irq; |
| 465 | 465 | struct amdgpu_irq_src rlc_gc_fed_irq; |
| 466 | + struct amdgpu_irq_src rlc_poison_irq; | |
| 466 | 467 | struct sq_work sq_work; |
| 467 | 468 | |
| 468 | 469 | /* gfx status */ |
| @@ -34,6 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include "amdgpu_ras.h" |
| 35 | 35 | #include "amdgpu_reset.h" |
| 36 | 36 | #include "amdgpu_xgmi.h" |
| 37 | +#include "amdgpu_atomfirmware.h" | |
| 37 | 38 | |
| 38 | 39 | #include <drm/drm_drv.h> |
| 39 | 40 | #include <drm/ttm/ttm_tt.h> |
| @@ -1374,18 +1375,18 @@ int amdgpu_gmc_get_nps_memranges(struct amdgpu_device *adev, · 1334 unmodified lines | ||
| 1374 | 1375 | struct amdgpu_mem_partition_info *mem_ranges, |
| 1375 | 1376 | uint8_t *exp_ranges) |
| 1376 | 1377 | { |
| 1377 | − struct amdgpu_gmc_memrange *ranges; | |
| 1378 | + struct amdgpu_gmc_memrange ranges[AMDGPU_MAX_MEM_RANGES]; | |
| 1378 | 1379 | int range_cnt, ret, i, j; |
| 1379 | 1380 | uint32_t nps_type; |
| 1380 | 1381 | bool refresh; |
| 1381 | 1382 | |
| 1382 | 1383 | if (!mem_ranges || !exp_ranges) |
| 1383 | 1384 | return -EINVAL; |
| 1384 | − | |
| 1385 | + range_cnt = AMDGPU_MAX_MEM_RANGES; | |
| 1385 | 1386 | refresh = (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) && |
| 1386 | 1387 | (adev->gmc.reset_flags & AMDGPU_GMC_INIT_RESET_NPS); |
| 1387 | − ret = amdgpu_discovery_get_nps_info(adev, &nps_type, &ranges, | |
| 1388 | − &range_cnt, refresh); | |
| 1388 | + ret = amdgpu_discovery_get_nps_info(adev, &nps_type, ranges, &range_cnt, | |
| 1389 | + refresh); | |
| 1389 | 1390 | |
| 1390 | 1391 | if (ret) |
| 1391 | 1392 | return ret; |
| @@ -1446,8 +1447,6 @@ int amdgpu_gmc_get_nps_memranges(struct amdgpu_device *adev, · 54 unmodified lines | ||
| 1446 | 1447 | if (!*exp_ranges) |
| 1447 | 1448 | *exp_ranges = range_cnt; |
| 1448 | 1449 | err: |
| 1449 | − kvfree(ranges); | |
| 1450 | − | |
| 1451 | 1450 | return ret; |
| 1452 | 1451 | } |
| 1453 | 1452 | |
| @@ -1749,3 +1748,31 @@ int amdgpu_gmc_init_mem_ranges(struct amdgpu_device *adev) · 295 unmodified lines | ||
| 1749 | 1748 | |
| 1750 | 1749 | return 0; |
| 1751 | 1750 | } |
| 1751 | + | |
| 1752 | +int amdgpu_gmc_get_vram_info(struct amdgpu_device *adev, | |
| 1753 | + int *vram_width, int *vram_type, int *vram_vendor) | |
| 1754 | +{ | |
| 1755 | + int ret = 0; | |
| 1756 | + | |
| 1757 | + if (adev->flags & AMD_IS_APU) | |
| 1758 | + return amdgpu_atomfirmware_get_integrated_system_info(adev, | |
| 1759 | + vram_width, vram_type, vram_vendor); | |
| 1760 | + switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | |
| 1761 | + case IP_VERSION(12, 0, 0): | |
| 1762 | + case IP_VERSION(12, 0, 1): | |
| 1763 | + return amdgpu_atomfirmware_get_umc_info(adev, | |
| 1764 | + vram_width, vram_type, vram_vendor); | |
| 1765 | + case IP_VERSION(9, 5, 0): | |
| 1766 | + case IP_VERSION(9, 4, 4): | |
| 1767 | + case IP_VERSION(9, 4, 3): | |
| 1768 | + ret = amdgpu_atomfirmware_get_umc_info(adev, | |
| 1769 | + vram_width, vram_type, vram_vendor); | |
| 1770 | + if (vram_width && !ret) | |
| 1771 | + *vram_width *= hweight32(adev->aid_mask); | |
| 1772 | + return ret; | |
| 1773 | + default: | |
| 1774 | + return amdgpu_atomfirmware_get_vram_info(adev, | |
| 1775 | + vram_width, vram_type, vram_vendor); | |
| 1776 | + } | |
| 1777 | + return 0; | |
| 1778 | +} | |
| @@ -482,4 +482,6 @@ amdgpu_gmc_query_memory_partition(struct amdgpu_device *adev); · 481 unmodified lines | ||
| 482 | 482 | int amdgpu_gmc_init_mem_ranges(struct amdgpu_device *adev); |
| 483 | 483 | void amdgpu_gmc_init_sw_mem_ranges(struct amdgpu_device *adev, |
| 484 | 484 | struct amdgpu_mem_partition_info *mem_ranges); |
| 485 | +int amdgpu_gmc_get_vram_info(struct amdgpu_device *adev, | |
| 486 | + int *vram_width, int *vram_type, int *vram_vendor); | |
| 485 | 487 | #endif |
| @@ -35,10 +35,13 @@ · 34 unmodified lines | ||
| 35 | 35 | * PASIDs are global address space identifiers that can be shared |
| 36 | 36 | * between the GPU, an IOMMU and the driver. VMs on different devices |
| 37 | 37 | * may use the same PASID if they share the same address |
| 38 | − * space. Therefore PASIDs are allocated using a global IDA. VMs are | |
| 39 | − * looked up from the PASID per amdgpu_device. | |
| 38 | + * space. Therefore PASIDs are allocated using IDR cyclic allocator | |
| 39 | + * (similar to kernel PID allocation) which naturally delays reuse. | |
| 40 | + * VMs are looked up from the PASID per amdgpu_device. | |
| 40 | 41 | */ |
| 41 | −static DEFINE_IDA(amdgpu_pasid_ida); | |
| 42 | + | |
| 43 | +static DEFINE_IDR(amdgpu_pasid_idr); | |
| 44 | +static DEFINE_SPINLOCK(amdgpu_pasid_idr_lock); | |
| 42 | 45 | |
| 43 | 46 | /* Helper to free pasid from a fence callback */ |
| 44 | 47 | struct amdgpu_pasid_cb { |
| @@ -50,8 +53,8 @@ struct amdgpu_pasid_cb { · 5 unmodified lines | ||
| 50 | 53 | * amdgpu_pasid_alloc - Allocate a PASID |
| 51 | 54 | * @bits: Maximum width of the PASID in bits, must be at least 1 |
| 52 | 55 | * |
| 53 | − * Allocates a PASID of the given width while keeping smaller PASIDs | |
| 54 | − * available if possible. | |
| 56 | + * Uses kernel's IDR cyclic allocator (same as PID allocation). | |
| 57 | + * Allocates sequentially with automatic wrap-around. | |
| 55 | 58 | * |
| 56 | 59 | * Returns a positive integer on success. Returns %-EINVAL if bits==0. |
| 57 | 60 | * Returns %-ENOSPC if no PASID was available. Returns %-ENOMEM on |
| @@ -59,14 +62,18 @@ struct amdgpu_pasid_cb { · 1 unmodified line | ||
| 59 | 62 | */ |
| 60 | 63 | int amdgpu_pasid_alloc(unsigned int bits) |
| 61 | 64 | { |
| 62 | − int pasid = -EINVAL; | |
| 65 | + int pasid; | |
| 63 | 66 | |
| 64 | − for (bits = min(bits, 31U); bits > 0; bits--) { | |
| 65 | − pasid = ida_alloc_range(&amdgpu_pasid_ida, 1U << (bits - 1), | |
| 66 | − (1U << bits) - 1, GFP_KERNEL); | |
| 67 | − if (pasid != -ENOSPC) | |
| 68 | − break; | |
| 69 | − } | |
| 67 | + if (bits == 0) | |
| 68 | + return -EINVAL; | |
| 69 | + | |
| 70 | + spin_lock(&amdgpu_pasid_idr_lock); | |
| 71 | + /* TODO: Need to replace the idr with an xarry, and then | |
| 72 | + * handle the internal locking with ATOMIC safe paths. | |
| 73 | + */ | |
| 74 | + pasid = idr_alloc_cyclic(&amdgpu_pasid_idr, NULL, 1, | |
| 75 | + 1U << bits, GFP_ATOMIC); | |
| 76 | + spin_unlock(&amdgpu_pasid_idr_lock); | |
| 70 | 77 | |
| 71 | 78 | if (pasid >= 0) |
| 72 | 79 | trace_amdgpu_pasid_allocated(pasid); |
| @@ -81,7 +88,10 @@ int amdgpu_pasid_alloc(unsigned int bits) · 8 unmodified lines | ||
| 81 | 88 | void amdgpu_pasid_free(u32 pasid) |
| 82 | 89 | { |
| 83 | 90 | trace_amdgpu_pasid_freed(pasid); |
| 84 | − ida_free(&amdgpu_pasid_ida, pasid); | |
| 91 | + | |
| 92 | + spin_lock(&amdgpu_pasid_idr_lock); | |
| 93 | + idr_remove(&amdgpu_pasid_idr, pasid); | |
| 94 | + spin_unlock(&amdgpu_pasid_idr_lock); | |
| 85 | 95 | } |
| 86 | 96 | |
| 87 | 97 | static void amdgpu_pasid_free_cb(struct dma_fence *fence, |
| @@ -616,3 +626,15 @@ void amdgpu_vmid_mgr_fini(struct amdgpu_device *adev) · 528 unmodified lines | ||
| 616 | 626 | } |
| 617 | 627 | } |
| 618 | 628 | } |
| 629 | + | |
| 630 | +/** | |
| 631 | + * amdgpu_pasid_mgr_cleanup - cleanup PASID manager | |
| 632 | + * | |
| 633 | + * Cleanup the IDR allocator. | |
| 634 | + */ | |
| 635 | +void amdgpu_pasid_mgr_cleanup(void) | |
| 636 | +{ | |
| 637 | + spin_lock(&amdgpu_pasid_idr_lock); | |
| 638 | + idr_destroy(&amdgpu_pasid_idr); | |
| 639 | + spin_unlock(&amdgpu_pasid_idr_lock); | |
| 640 | +} | |
| @@ -74,6 +74,7 @@ int amdgpu_pasid_alloc(unsigned int bits); · 73 unmodified lines | ||
| 74 | 74 | void amdgpu_pasid_free(u32 pasid); |
| 75 | 75 | void amdgpu_pasid_free_delayed(struct dma_resv *resv, |
| 76 | 76 | u32 pasid); |
| 77 | +void amdgpu_pasid_mgr_cleanup(void); | |
| 77 | 78 | |
| 78 | 79 | bool amdgpu_vmid_had_gpu_reset(struct amdgpu_device *adev, |
| 79 | 80 | struct amdgpu_vmid *id); |
| @@ -1522,10 +1522,7 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · 1521 unmodified lines | ||
| 1522 | 1522 | "Failed to init usermode queue manager (%d), use legacy workload submission only\n", |
| 1523 | 1523 | r); |
| 1524 | 1524 | |
| 1525 | − r = amdgpu_eviction_fence_init(&fpriv->evf_mgr); | |
| 1526 | − if (r) | |
| 1527 | − goto error_vm; | |
| 1528 | − | |
| 1525 | + amdgpu_evf_mgr_init(&fpriv->evf_mgr); | |
| 1529 | 1526 | amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev); |
| 1530 | 1527 | |
| 1531 | 1528 | file_priv->driver_priv = fpriv; |
| @@ -63,14 +63,40 @@ struct amdgpu_mmhub_funcs { · 62 unmodified lines | ||
| 63 | 63 | uint64_t page_table_base); |
| 64 | 64 | void (*update_power_gating)(struct amdgpu_device *adev, |
| 65 | 65 | bool enable); |
| 66 | + int (*get_xgmi_info)(struct amdgpu_device *adev); | |
| 67 | +}; | |
| 68 | + | |
| 69 | +struct amdgpu_mmhub_client_ids { | |
| 70 | + const char * const (*names)[2]; | |
| 71 | + unsigned int size; | |
| 66 | 72 | }; |
| 67 | 73 | |
| 68 | 74 | struct amdgpu_mmhub { |
| 69 | 75 | struct ras_common_if *ras_if; |
| 70 | 76 | const struct amdgpu_mmhub_funcs *funcs; |
| 71 | 77 | struct amdgpu_mmhub_ras *ras; |
| 78 | + struct amdgpu_mmhub_client_ids client_ids; | |
| 72 | 79 | }; |
| 73 | 80 | |
| 81 | +static inline void | |
| 82 | +amdgpu_mmhub_init_client_info(struct amdgpu_mmhub *mmhub, | |
| 83 | + const char * const (*names)[2], | |
| 84 | + unsigned int size) | |
| 85 | +{ | |
| 86 | + mmhub->client_ids.names = names; | |
| 87 | + mmhub->client_ids.size = size; | |
| 88 | +} | |
| 89 | + | |
| 90 | +static inline const char * | |
| 91 | +amdgpu_mmhub_client_name(struct amdgpu_mmhub *mmhub, | |
| 92 | + u32 cid, bool is_write) | |
| 93 | +{ | |
| 94 | + if (cid < mmhub->client_ids.size) | |
| 95 | + return mmhub->client_ids.names[cid][is_write]; | |
| 96 | + | |
| 97 | + return NULL; | |
| 98 | +} | |
| 99 | + | |
| 74 | 100 | int amdgpu_mmhub_ras_sw_init(struct amdgpu_device *adev); |
| 75 | 101 | |
| 76 | 102 | #endif |
| @@ -44,6 +44,7 @@ · 43 unmodified lines | ||
| 44 | 44 | #include <drm/display/drm_dp_mst_helper.h> |
| 45 | 45 | #include "modules/inc/mod_freesync.h" |
| 46 | 46 | #include "amdgpu_dm_irq_params.h" |
| 47 | +#include "amdgpu_dm_ism.h" | |
| 47 | 48 | |
| 48 | 49 | struct amdgpu_bo; |
| 49 | 50 | struct amdgpu_device; |
| @@ -486,6 +487,10 @@ struct amdgpu_crtc { · 436 unmodified lines | ||
| 486 | 487 | int deferred_flip_completion; |
| 487 | 488 | /* parameters access from DM IRQ handler */ |
| 488 | 489 | struct dm_irq_params dm_irq_params; |
| 490 | + | |
| 491 | + /* DM idle state manager */ | |
| 492 | + struct amdgpu_dm_ism ism; | |
| 493 | + | |
| 489 | 494 | /* pll sharing */ |
| 490 | 495 | struct amdgpu_atom_ss ss; |
| 491 | 496 | bool ss_enabled; |
| @@ -1325,7 +1325,8 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo) · 1324 unmodified lines | ||
| 1325 | 1325 | if (r) |
| 1326 | 1326 | goto out; |
| 1327 | 1327 | |
| 1328 | − r = amdgpu_fill_buffer(&adev->mman.clear_entity, abo, 0, &bo->base._resv, | |
| 1328 | + r = amdgpu_fill_buffer(amdgpu_ttm_next_clear_entity(adev), | |
| 1329 | + abo, 0, &bo->base._resv, | |
| 1329 | 1330 | &fence, AMDGPU_KERNEL_JOB_ID_CLEAR_ON_RELEASE); |
| 1330 | 1331 | if (WARN_ON(r)) |
| 1331 | 1332 | goto out; |
| @@ -2250,6 +2250,7 @@ static bool amdgpu_ras_aca_is_supported(struct amdgpu_device *adev) · 2249 unmodified lines | ||
| 2250 | 2250 | case IP_VERSION(13, 0, 6): |
| 2251 | 2251 | case IP_VERSION(13, 0, 12): |
| 2252 | 2252 | case IP_VERSION(13, 0, 14): |
| 2253 | + case IP_VERSION(13, 0, 15): | |
| 2253 | 2254 | ret = true; |
| 2254 | 2255 | break; |
| 2255 | 2256 | default: |
| @@ -4000,6 +4001,7 @@ static bool amdgpu_ras_asic_supported(struct amdgpu_device *adev) · 1744 unmodified lines | ||
| 4000 | 4001 | case IP_VERSION(13, 0, 6): |
| 4001 | 4002 | case IP_VERSION(13, 0, 12): |
| 4002 | 4003 | case IP_VERSION(13, 0, 14): |
| 4004 | + case IP_VERSION(13, 0, 15): | |
| 4003 | 4005 | return true; |
| 4004 | 4006 | default: |
| 4005 | 4007 | return false; |
| @@ -4013,6 +4015,7 @@ static bool amdgpu_ras_asic_supported(struct amdgpu_device *adev) · 7 unmodified lines | ||
| 4013 | 4015 | case IP_VERSION(13, 0, 10): |
| 4014 | 4016 | case IP_VERSION(13, 0, 12): |
| 4015 | 4017 | case IP_VERSION(13, 0, 14): |
| 4018 | + case IP_VERSION(13, 0, 15): | |
| 4016 | 4019 | case IP_VERSION(14, 0, 3): |
| 4017 | 4020 | return true; |
| 4018 | 4021 | default: |
| @@ -4182,7 +4185,8 @@ init_ras_enabled_flag: · 163 unmodified lines | ||
| 4182 | 4185 | adev->aca.is_enabled = |
| 4183 | 4186 | (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) || |
| 4184 | 4187 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) || |
| 4185 | − amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)); | |
| 4188 | + amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14) || | |
| 4189 | + amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 15)); | |
| 4186 | 4190 | } |
| 4187 | 4191 | |
| 4188 | 4192 | /* bad page feature is not applicable to specific app platform */ |
| @@ -4270,6 +4274,7 @@ static void amdgpu_ras_init_reserved_vram_size(struct amdgpu_device *adev) · 81 unmodified lines | ||
| 4270 | 4274 | case IP_VERSION(13, 0, 2): |
| 4271 | 4275 | case IP_VERSION(13, 0, 6): |
| 4272 | 4276 | case IP_VERSION(13, 0, 12): |
| 4277 | + case IP_VERSION(13, 0, 15): | |
| 4273 | 4278 | con->reserved_pages_in_bytes = AMDGPU_RAS_RESERVED_VRAM_SIZE_DEFAULT; |
| 4274 | 4279 | break; |
| 4275 | 4280 | case IP_VERSION(13, 0, 14): |
| @@ -1558,6 +1558,8 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 1557 unmodified lines | ||
| 1558 | 1558 | unsigned char buf[RAS_TABLE_HEADER_SIZE] = { 0 }; |
| 1559 | 1559 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; |
| 1560 | 1560 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); |
| 1561 | + int dev_var = adev->pdev->device & 0xF; | |
| 1562 | + uint32_t vram_type = adev->gmc.vram_type; | |
| 1561 | 1563 | int res; |
| 1562 | 1564 | |
| 1563 | 1565 | if (amdgpu_ras_smu_eeprom_supported(adev)) |
| @@ -1597,6 +1599,12 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 33 unmodified lines | ||
| 1597 | 1599 | return amdgpu_ras_eeprom_reset_table(control); |
| 1598 | 1600 | } |
| 1599 | 1601 | |
| 1602 | + if (!(adev->flags & AMD_IS_APU) && (dev_var == 0x5) && | |
| 1603 | + (vram_type == AMDGPU_VRAM_TYPE_HBM3E) && | |
| 1604 | + (hdr->version < RAS_TABLE_VER_V3)) { | |
| 1605 | + return amdgpu_ras_eeprom_reset_table(control); | |
| 1606 | + } | |
| 1607 | + | |
| 1600 | 1608 | switch (hdr->version) { |
| 1601 | 1609 | case RAS_TABLE_VER_V2_1: |
| 1602 | 1610 | case RAS_TABLE_VER_V3: |
| @@ -314,7 +314,6 @@ uint64_t amdgpu_reg_get_smn_base64(struct amdgpu_device *adev, · 313 unmodified lines | ||
| 314 | 314 | "SMN base address query not supported for this device\n"); |
| 315 | 315 | return 0; |
| 316 | 316 | } |
| 317 | − return 0; | |
| 318 | 317 | } |
| 319 | 318 | return adev->reg.smn.get_smn_base(adev, block, die_inst); |
| 320 | 319 | } |
| @@ -103,10 +103,8 @@ int amdgpu_sched_ioctl(struct drm_device *dev, void *data, · 102 unmodified lines | ||
| 103 | 103 | return -EINVAL; |
| 104 | 104 | } |
| 105 | 105 | |
| 106 | − if (!amdgpu_ctx_priority_is_valid(args->in.priority)) { | |
| 107 | − WARN(1, "Invalid context priority %d\n", args->in.priority); | |
| 106 | + if (!amdgpu_ctx_priority_is_valid(args->in.priority)) | |
| 108 | 107 | return -EINVAL; |
| 109 | − } | |
| 110 | 108 | |
| 111 | 109 | switch (args->in.op) { |
| 112 | 110 | case AMDGPU_SCHED_OP_PROCESS_PRIORITY_OVERRIDE: |
| @@ -387,9 +387,11 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo, · 386 unmodified lines | ||
| 387 | 387 | { |
| 388 | 388 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->bdev); |
| 389 | 389 | struct amdgpu_bo *abo = ttm_to_amdgpu_bo(bo); |
| 390 | + struct amdgpu_ttm_buffer_entity *entity; | |
| 390 | 391 | struct amdgpu_copy_mem src, dst; |
| 391 | 392 | struct dma_fence *fence = NULL; |
| 392 | 393 | int r; |
| 394 | + u32 e; | |
| 393 | 395 | |
| 394 | 396 | src.bo = bo; |
| 395 | 397 | dst.bo = bo; |
| @@ -398,8 +400,12 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo, · 2 unmodified lines | ||
| 398 | 400 | src.offset = 0; |
| 399 | 401 | dst.offset = 0; |
| 400 | 402 | |
| 403 | + e = atomic_inc_return(&adev->mman.next_move_entity) % | |
| 404 | + adev->mman.num_move_entities; | |
| 405 | + entity = &adev->mman.move_entities[e]; | |
| 406 | + | |
| 401 | 407 | r = amdgpu_ttm_copy_mem_to_mem(adev, |
| 402 | − &adev->mman.move_entity, | |
| 408 | + entity, | |
| 403 | 409 | &src, &dst, |
| 404 | 410 | new_mem->size, |
| 405 | 411 | amdgpu_bo_encrypted(abo), |
| @@ -411,9 +417,7 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo, · 5 unmodified lines | ||
| 411 | 417 | if (old_mem->mem_type == TTM_PL_VRAM && |
| 412 | 418 | (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) { |
| 413 | 419 | struct dma_fence *wipe_fence = NULL; |
| 414 | − | |
| 415 | − r = amdgpu_fill_buffer(&adev->mman.move_entity, | |
| 416 | − abo, 0, NULL, &wipe_fence, | |
| 420 | + r = amdgpu_fill_buffer(entity, abo, 0, NULL, &wipe_fence, | |
| 417 | 421 | AMDGPU_KERNEL_JOB_ID_MOVE_BLIT); |
| 418 | 422 | if (r) { |
| 419 | 423 | goto error; |
| @@ -854,25 +858,15 @@ static void amdgpu_ttm_gart_bind_gfx9_mqd(struct amdgpu_device *adev, · 434 unmodified lines | ||
| 854 | 858 | int num_xcc = max(1U, adev->gfx.num_xcc_per_xcp); |
| 855 | 859 | uint64_t page_idx, pages_per_xcc; |
| 856 | 860 | int i; |
| 857 | − uint64_t ctrl_flags = AMDGPU_PTE_MTYPE_VG10(flags, AMDGPU_MTYPE_NC); | |
| 858 | 861 | |
| 859 | 862 | pages_per_xcc = total_pages; |
| 860 | 863 | do_div(pages_per_xcc, num_xcc); |
| 861 | 864 | |
| 862 | 865 | for (i = 0, page_idx = 0; i < num_xcc; i++, page_idx += pages_per_xcc) { |
| 863 | − /* MQD page: use default flags */ | |
| 864 | − amdgpu_gart_bind(adev, | |
| 866 | + amdgpu_gart_map_gfx9_mqd(adev, | |
| 865 | 867 | gtt->offset + (page_idx << PAGE_SHIFT), |
| 866 | − 1, >t->ttm.dma_address[page_idx], flags); | |
| 867 | − /* | |
| 868 | − * Ctrl pages - modify the memory type to NC (ctrl_flags) from | |
| 869 | − * the second page of the BO onward. | |
| 870 | − */ | |
| 871 | − amdgpu_gart_bind(adev, | |
| 872 | − gtt->offset + ((page_idx + 1) << PAGE_SHIFT), | |
| 873 | − pages_per_xcc - 1, | |
| 874 | − >t->ttm.dma_address[page_idx + 1], | |
| 875 | − ctrl_flags); | |
| 868 | + pages_per_xcc, >t->ttm.dma_address[page_idx], | |
| 869 | + flags); | |
| 876 | 870 | } |
| 877 | 871 | } |
| 878 | 872 | |
| @@ -2345,8 +2339,9 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev) · 1466 unmodified lines | ||
| 2345 | 2339 | void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) |
| 2346 | 2340 | { |
| 2347 | 2341 | struct ttm_resource_manager *man = ttm_manager_type(&adev->mman.bdev, TTM_PL_VRAM); |
| 2342 | + u32 num_clear_entities, num_move_entities; | |
| 2348 | 2343 | uint64_t size; |
| 2349 | − int r; | |
| 2344 | + int r, i, j; | |
| 2350 | 2345 | |
| 2351 | 2346 | if (!adev->mman.initialized || amdgpu_in_reset(adev) || |
| 2352 | 2347 | adev->mman.buffer_funcs_enabled == enable || adev->gmc.is_app_apu) |
| @@ -2361,6 +2356,8 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) · 8 unmodified lines | ||
| 2361 | 2356 | return; |
| 2362 | 2357 | } |
| 2363 | 2358 | |
| 2359 | + num_clear_entities = 1; | |
| 2360 | + num_move_entities = 1; | |
| 2364 | 2361 | ring = adev->mman.buffer_funcs_ring; |
| 2365 | 2362 | sched = &ring->sched; |
| 2366 | 2363 | r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, |
| @@ -2373,36 +2370,64 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) · 6 unmodified lines | ||
| 2373 | 2370 | return; |
| 2374 | 2371 | } |
| 2375 | 2372 | |
| 2376 | − r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, | |
| 2377 | − &adev->mman.clear_entity, | |
| 2378 | − DRM_SCHED_PRIORITY_NORMAL, | |
| 2379 | − &sched, 1, 1); | |
| 2380 | − if (r < 0) { | |
| 2381 | − dev_err(adev->dev, | |
| 2382 | − "Failed setting up TTM BO clear entity (%d)\n", r); | |
| 2373 | + adev->mman.clear_entities = kcalloc(num_clear_entities, | |
| 2374 | + sizeof(struct amdgpu_ttm_buffer_entity), | |
| 2375 | + GFP_KERNEL); | |
| 2376 | + atomic_set(&adev->mman.next_clear_entity, 0); | |
| 2377 | + if (!adev->mman.clear_entities) | |
| 2383 | 2378 | goto error_free_default_entity; |
| 2379 | + | |
| 2380 | + adev->mman.num_clear_entities = num_clear_entities; | |
| 2381 | + | |
| 2382 | + for (i = 0; i < num_clear_entities; i++) { | |
| 2383 | + r = amdgpu_ttm_buffer_entity_init( | |
| 2384 | + &adev->mman.gtt_mgr, &adev->mman.clear_entities[i], | |
| 2385 | + DRM_SCHED_PRIORITY_NORMAL, &sched, 1, 1); | |
| 2386 | + | |
| 2387 | + if (r < 0) { | |
| 2388 | + for (j = 0; j < i; j++) | |
| 2389 | + amdgpu_ttm_buffer_entity_fini( | |
| 2390 | + &adev->mman.gtt_mgr, &adev->mman.clear_entities[j]); | |
| 2391 | + kfree(adev->mman.clear_entities); | |
| 2392 | + adev->mman.num_clear_entities = 0; | |
| 2393 | + adev->mman.clear_entities = NULL; | |
| 2394 | + goto error_free_default_entity; | |
| 2395 | + } | |
| 2384 | 2396 | } |
| 2385 | 2397 | |
| 2386 | − r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, | |
| 2387 | − &adev->mman.move_entity, | |
| 2388 | − DRM_SCHED_PRIORITY_NORMAL, | |
| 2389 | − &sched, 1, 2); | |
| 2390 | − if (r < 0) { | |
| 2391 | − dev_err(adev->dev, | |
| 2392 | − "Failed setting up TTM BO move entity (%d)\n", r); | |
| 2393 | − goto error_free_clear_entity; | |
| 2398 | + adev->mman.num_move_entities = num_move_entities; | |
| 2399 | + atomic_set(&adev->mman.next_move_entity, 0); | |
| 2400 | + for (i = 0; i < num_move_entities; i++) { | |
| 2401 | + r = amdgpu_ttm_buffer_entity_init( | |
| 2402 | + &adev->mman.gtt_mgr, | |
| 2403 | + &adev->mman.move_entities[i], | |
| 2404 | + DRM_SCHED_PRIORITY_NORMAL, &sched, 1, 2); | |
| 2405 | + | |
| 2406 | + if (r < 0) { | |
| 2407 | + for (j = 0; j < i; j++) | |
| 2408 | + amdgpu_ttm_buffer_entity_fini( | |
| 2409 | + &adev->mman.gtt_mgr, &adev->mman.move_entities[j]); | |
| 2410 | + adev->mman.num_move_entities = 0; | |
| 2411 | + goto error_free_clear_entities; | |
| 2412 | + } | |
| 2394 | 2413 | } |
| 2395 | 2414 | } else { |
| 2396 | 2415 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, |
| 2397 | 2416 | &adev->mman.default_entity); |
| 2398 | − amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | |
| 2399 | − &adev->mman.clear_entity); | |
| 2400 | − amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | |
| 2401 | − &adev->mman.move_entity); | |
| 2417 | + for (i = 0; i < adev->mman.num_clear_entities; i++) | |
| 2418 | + amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | |
| 2419 | + &adev->mman.clear_entities[i]); | |
| 2420 | + for (i = 0; i < adev->mman.num_move_entities; i++) | |
| 2421 | + amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | |
| 2422 | + &adev->mman.move_entities[i]); | |
| 2402 | 2423 | /* Drop all the old fences since re-creating the scheduler entities |
| 2403 | 2424 | * will allocate new contexts. |
| 2404 | 2425 | */ |
| 2405 | 2426 | ttm_resource_manager_cleanup(man); |
| 2427 | + kfree(adev->mman.clear_entities); | |
| 2428 | + adev->mman.clear_entities = NULL; | |
| 2429 | + adev->mman.num_clear_entities = 0; | |
| 2430 | + adev->mman.num_move_entities = 0; | |
| 2406 | 2431 | } |
| 2407 | 2432 | |
| 2408 | 2433 | /* this just adjusts TTM size idea, which sets lpfn to the correct value */ |
| @@ -2415,9 +2440,13 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) · 6 unmodified lines | ||
| 2415 | 2440 | |
| 2416 | 2441 | return; |
| 2417 | 2442 | |
| 2418 | −error_free_clear_entity: | |
| 2419 | − amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | |
| 2420 | − &adev->mman.clear_entity); | |
| 2443 | +error_free_clear_entities: | |
| 2444 | + for (i = 0; i < adev->mman.num_clear_entities; i++) | |
| 2445 | + amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | |
| 2446 | + &adev->mman.clear_entities[i]); | |
| 2447 | + kfree(adev->mman.clear_entities); | |
| 2448 | + adev->mman.clear_entities = NULL; | |
| 2449 | + adev->mman.num_clear_entities = 0; | |
| 2421 | 2450 | error_free_default_entity: |
| 2422 | 2451 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, |
| 2423 | 2452 | &adev->mman.default_entity); |
| @@ -2567,8 +2596,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo, · 143 unmodified lines | ||
| 2567 | 2596 | |
| 2568 | 2597 | if (!fence) |
| 2569 | 2598 | return -EINVAL; |
| 2570 | − | |
| 2571 | − entity = &adev->mman.clear_entity; | |
| 2599 | + entity = &adev->mman.clear_entities[0]; | |
| 2572 | 2600 | *fence = dma_fence_get_stub(); |
| 2573 | 2601 | |
| 2574 | 2602 | amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor); |
| @@ -2620,11 +2648,8 @@ int amdgpu_fill_buffer(struct amdgpu_ttm_buffer_entity *entity, · 45 unmodified lines | ||
| 2620 | 2648 | struct amdgpu_res_cursor dst; |
| 2621 | 2649 | int r; |
| 2622 | 2650 | |
| 2623 | − if (!adev->mman.buffer_funcs_enabled) { | |
| 2624 | − dev_err(adev->dev, | |
| 2625 | − "Trying to clear memory with ring turned off.\n"); | |
| 2651 | + if (!entity) | |
| 2626 | 2652 | return -EINVAL; |
| 2627 | − } | |
| 2628 | 2653 | |
| 2629 | 2654 | amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &dst); |
| 2630 | 2655 | |
| @@ -2660,6 +2685,20 @@ error: · 29 unmodified lines | ||
| 2660 | 2685 | return r; |
| 2661 | 2686 | } |
| 2662 | 2687 | |
| 2688 | +struct amdgpu_ttm_buffer_entity * | |
| 2689 | +amdgpu_ttm_next_clear_entity(struct amdgpu_device *adev) | |
| 2690 | +{ | |
| 2691 | + struct amdgpu_mman *mman = &adev->mman; | |
| 2692 | + u32 i; | |
| 2693 | + | |
| 2694 | + if (mman->num_clear_entities == 0) | |
| 2695 | + return NULL; | |
| 2696 | + | |
| 2697 | + i = atomic_inc_return(&mman->next_clear_entity) % | |
| 2698 | + mman->num_clear_entities; | |
| 2699 | + return &mman->clear_entities[i]; | |
| 2700 | +} | |
| 2701 | + | |
| 2663 | 2702 | /** |
| 2664 | 2703 | * amdgpu_ttm_evict_resources - evict memory buffers |
| 2665 | 2704 | * @adev: amdgpu device object |
| @@ -72,8 +72,12 @@ struct amdgpu_mman { · 71 unmodified lines | ||
| 72 | 72 | |
| 73 | 73 | /* @default_entity: for workarounds, has no gart windows */ |
| 74 | 74 | struct amdgpu_ttm_buffer_entity default_entity; |
| 75 | − struct amdgpu_ttm_buffer_entity clear_entity; | |
| 76 | − struct amdgpu_ttm_buffer_entity move_entity; | |
| 75 | + struct amdgpu_ttm_buffer_entity *clear_entities; | |
| 76 | + atomic_t next_clear_entity; | |
| 77 | + u32 num_clear_entities; | |
| 78 | + struct amdgpu_ttm_buffer_entity move_entities[TTM_NUM_MOVE_FENCES]; | |
| 79 | + atomic_t next_move_entity; | |
| 80 | + u32 num_move_entities; | |
| 77 | 81 | |
| 78 | 82 | struct amdgpu_vram_mgr vram_mgr; |
| 79 | 83 | struct amdgpu_gtt_mgr gtt_mgr; |
| @@ -191,6 +195,7 @@ int amdgpu_fill_buffer(struct amdgpu_ttm_buffer_entity *entity, · 111 unmodified lines | ||
| 191 | 195 | struct dma_resv *resv, |
| 192 | 196 | struct dma_fence **f, |
| 193 | 197 | u64 k_job_id); |
| 198 | +struct amdgpu_ttm_buffer_entity *amdgpu_ttm_next_clear_entity(struct amdgpu_device *adev); | |
| 194 | 199 | |
| 195 | 200 | int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo); |
| 196 | 201 | void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo); |
| @@ -1150,7 +1150,7 @@ int amdgpu_ucode_create_bo(struct amdgpu_device *adev) · 1149 unmodified lines | ||
| 1150 | 1150 | if ((adev->firmware.load_type != AMDGPU_FW_LOAD_DIRECT) && |
| 1151 | 1151 | (adev->firmware.load_type != AMDGPU_FW_LOAD_RLC_BACKDOOR_AUTO)) { |
| 1152 | 1152 | amdgpu_bo_create_kernel(adev, adev->firmware.fw_size, PAGE_SIZE, |
| 1153 | − (amdgpu_sriov_vf(adev) || adev->debug_use_vram_fw_buf) ? | |
| 1153 | + (amdgpu_sriov_vf(adev) || adev->debug_use_vram_fw_buf || adev->gmc.xgmi.connected_to_cpu) ? | |
| 1154 | 1154 | AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT, |
| 1155 | 1155 | &adev->firmware.fw_buf, |
| 1156 | 1156 | &adev->firmware.fw_buf_mc, |
| @@ -398,6 +398,17 @@ int amdgpu_umc_process_ecc_irq(struct amdgpu_device *adev, · 397 unmodified lines | ||
| 398 | 398 | return 0; |
| 399 | 399 | } |
| 400 | 400 | |
| 401 | +int amdgpu_umc_uniras_process_ecc_irq(struct amdgpu_device *adev, | |
| 402 | + struct amdgpu_irq_src *source, | |
| 403 | + struct amdgpu_iv_entry *entry) | |
| 404 | +{ | |
| 405 | + struct ras_ih_info ih_info = {0}; | |
| 406 | + | |
| 407 | + ih_info.block = RAS_BLOCK_ID__UMC; | |
| 408 | + amdgpu_ras_mgr_dispatch_interrupt(adev, &ih_info); | |
| 409 | + return 0; | |
| 410 | +} | |
| 411 | + | |
| 401 | 412 | int amdgpu_umc_fill_error_record(struct ras_err_data *err_data, |
| 402 | 413 | uint64_t err_addr, |
| 403 | 414 | uint64_t retired_page, |
| @@ -161,6 +161,9 @@ int amdgpu_umc_pasid_poison_handler(struct amdgpu_device *adev, · 160 unmodified lines | ||
| 161 | 161 | int amdgpu_umc_process_ecc_irq(struct amdgpu_device *adev, |
| 162 | 162 | struct amdgpu_irq_src *source, |
| 163 | 163 | struct amdgpu_iv_entry *entry); |
| 164 | +int amdgpu_umc_uniras_process_ecc_irq(struct amdgpu_device *adev, | |
| 165 | + struct amdgpu_irq_src *source, | |
| 166 | + struct amdgpu_iv_entry *entry); | |
| 164 | 167 | int amdgpu_umc_fill_error_record(struct ras_err_data *err_data, |
| 165 | 168 | uint64_t err_addr, |
| 166 | 169 | uint64_t retired_page, |
| @@ -156,7 +156,7 @@ static void amdgpu_userq_hang_detect_work(struct work_struct *work) · 155 unmodified lines | ||
| 156 | 156 | struct dma_fence *fence; |
| 157 | 157 | struct amdgpu_userq_mgr *uq_mgr; |
| 158 | 158 | |
| 159 | − if (!queue || !queue->userq_mgr) | |
| 159 | + if (!queue->userq_mgr) | |
| 160 | 160 | return; |
| 161 | 161 | |
| 162 | 162 | uq_mgr = queue->userq_mgr; |
| @@ -472,17 +472,16 @@ void · 309 unmodified lines | ||
| 472 | 472 | amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *uq_mgr, |
| 473 | 473 | struct amdgpu_eviction_fence_mgr *evf_mgr) |
| 474 | 474 | { |
| 475 | − struct amdgpu_eviction_fence *ev_fence; | |
| 475 | + struct dma_fence *ev_fence; | |
| 476 | 476 | |
| 477 | 477 | retry: |
| 478 | 478 | /* Flush any pending resume work to create ev_fence */ |
| 479 | 479 | flush_delayed_work(&uq_mgr->resume_work); |
| 480 | 480 | |
| 481 | 481 | mutex_lock(&uq_mgr->userq_mutex); |
| 482 | − spin_lock(&evf_mgr->ev_fence_lock); | |
| 483 | − ev_fence = evf_mgr->ev_fence; | |
| 484 | − spin_unlock(&evf_mgr->ev_fence_lock); | |
| 485 | − if (!ev_fence || dma_fence_is_signaled(&ev_fence->base)) { | |
| 482 | + ev_fence = amdgpu_evf_mgr_get_fence(evf_mgr); | |
| 483 | + if (dma_fence_is_signaled(ev_fence)) { | |
| 484 | + dma_fence_put(ev_fence); | |
| 486 | 485 | mutex_unlock(&uq_mgr->userq_mutex); |
| 487 | 486 | /* |
| 488 | 487 | * Looks like there was no pending resume work, |
| @@ -491,6 +490,7 @@ retry: · 2 unmodified lines | ||
| 491 | 490 | schedule_delayed_work(&uq_mgr->resume_work, 0); |
| 492 | 491 | goto retry; |
| 493 | 492 | } |
| 493 | + dma_fence_put(ev_fence); | |
| 494 | 494 | } |
| 495 | 495 | |
| 496 | 496 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, |
| @@ -600,6 +600,13 @@ amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr, · 103 unmodified lines | ||
| 600 | 600 | goto unpin_bo; |
| 601 | 601 | } |
| 602 | 602 | |
| 603 | + /* Validate doorbell_offset is within the doorbell BO */ | |
| 604 | + if ((u64)db_info->doorbell_offset * db_size + db_size > | |
| 605 | + amdgpu_bo_size(db_obj->obj)) { | |
| 606 | + r = -EINVAL; | |
| 607 | + goto unpin_bo; | |
| 608 | + } | |
| 609 | + | |
| 603 | 610 | index = amdgpu_doorbell_index_on_bar(uq_mgr->adev, db_obj->obj, |
| 604 | 611 | db_info->doorbell_offset, db_size); |
| 605 | 612 | drm_dbg_driver(adev_to_drm(uq_mgr->adev), |
| @@ -623,13 +630,14 @@ amdgpu_userq_destroy(struct amdgpu_userq_mgr *uq_mgr, struct amdgpu_usermode_que · 17 unmodified lines | ||
| 623 | 630 | int r = 0; |
| 624 | 631 | |
| 625 | 632 | cancel_delayed_work_sync(&uq_mgr->resume_work); |
| 633 | + | |
| 634 | + /* Cancel any pending hang detection work and cleanup */ | |
| 635 | + cancel_delayed_work_sync(&queue->hang_detect_work); | |
| 636 | + | |
| 626 | 637 | mutex_lock(&uq_mgr->userq_mutex); |
| 638 | + queue->hang_detect_fence = NULL; | |
| 627 | 639 | amdgpu_userq_wait_for_last_fence(queue); |
| 628 | − /* Cancel any pending hang detection work and cleanup */ | |
| 629 | − if (queue->hang_detect_fence) { | |
| 630 | − cancel_delayed_work_sync(&queue->hang_detect_work); | |
| 631 | − queue->hang_detect_fence = NULL; | |
| 632 | − } | |
| 640 | + | |
| 633 | 641 | r = amdgpu_bo_reserve(queue->db_obj.obj, true); |
| 634 | 642 | if (!r) { |
| 635 | 643 | amdgpu_bo_unpin(queue->db_obj.obj); |
| @@ -996,17 +1004,14 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · 360 unmodified lines | ||
| 996 | 1004 | unsigned long queue_id; |
| 997 | 1005 | int ret = 0, r; |
| 998 | 1006 | |
| 1007 | + mutex_lock(&uq_mgr->userq_mutex); | |
| 999 | 1008 | /* Resume all the queues for this process */ |
| 1000 | 1009 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { |
| 1001 | − queue = amdgpu_userq_get(uq_mgr, queue_id); | |
| 1002 | − if (!queue) | |
| 1003 | − continue; | |
| 1004 | 1010 | |
| 1005 | 1011 | if (!amdgpu_userq_buffer_vas_mapped(queue)) { |
| 1006 | 1012 | drm_file_err(uq_mgr->file, |
| 1007 | 1013 | "trying restore queue without va mapping\n"); |
| 1008 | 1014 | queue->state = AMDGPU_USERQ_STATE_INVALID_VA; |
| 1009 | − amdgpu_userq_put(queue); | |
| 1010 | 1015 | continue; |
| 1011 | 1016 | } |
| 1012 | 1017 | |
| @@ -1014,8 +1019,8 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · 1 unmodified line | ||
| 1014 | 1019 | if (r) |
| 1015 | 1020 | ret = r; |
| 1016 | 1021 | |
| 1017 | − amdgpu_userq_put(queue); | |
| 1018 | 1022 | } |
| 1023 | + mutex_unlock(&uq_mgr->userq_mutex); | |
| 1019 | 1024 | |
| 1020 | 1025 | if (ret) |
| 1021 | 1026 | drm_file_err(uq_mgr->file, "Failed to map all the queues\n"); |
| @@ -1040,12 +1045,12 @@ amdgpu_userq_bo_validate(struct amdgpu_device *adev, struct drm_exec *exec, · 18 unmodified lines | ||
| 1040 | 1045 | struct amdgpu_bo *bo; |
| 1041 | 1046 | int ret; |
| 1042 | 1047 | |
| 1043 | − spin_lock(&vm->invalidated_lock); | |
| 1048 | + spin_lock(&vm->status_lock); | |
| 1044 | 1049 | while (!list_empty(&vm->invalidated)) { |
| 1045 | 1050 | bo_va = list_first_entry(&vm->invalidated, |
| 1046 | 1051 | struct amdgpu_bo_va, |
| 1047 | 1052 | base.vm_status); |
| 1048 | − spin_unlock(&vm->invalidated_lock); | |
| 1053 | + spin_unlock(&vm->status_lock); | |
| 1049 | 1054 | |
| 1050 | 1055 | bo = bo_va->base.bo; |
| 1051 | 1056 | ret = drm_exec_prepare_obj(exec, &bo->tbo.base, 2); |
| @@ -1062,9 +1067,9 @@ amdgpu_userq_bo_validate(struct amdgpu_device *adev, struct drm_exec *exec, · 10 unmodified lines | ||
| 1062 | 1067 | if (ret) |
| 1063 | 1068 | return ret; |
| 1064 | 1069 | |
| 1065 | − spin_lock(&vm->invalidated_lock); | |
| 1070 | + spin_lock(&vm->status_lock); | |
| 1066 | 1071 | } |
| 1067 | − spin_unlock(&vm->invalidated_lock); | |
| 1072 | + spin_unlock(&vm->status_lock); | |
| 1068 | 1073 | |
| 1069 | 1074 | return 0; |
| 1070 | 1075 | } |
| @@ -1196,7 +1201,7 @@ retry_lock: · 125 unmodified lines | ||
| 1196 | 1201 | dma_fence_wait(bo_va->last_pt_update, false); |
| 1197 | 1202 | dma_fence_wait(vm->last_update, false); |
| 1198 | 1203 | |
| 1199 | − ret = amdgpu_eviction_fence_replace_fence(&fpriv->evf_mgr, &exec); | |
| 1204 | + ret = amdgpu_evf_mgr_rearm(&fpriv->evf_mgr, &exec); | |
| 1200 | 1205 | if (ret) |
| 1201 | 1206 | drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n"); |
| 1202 | 1207 | |
| @@ -1216,26 +1221,25 @@ static void amdgpu_userq_restore_worker(struct work_struct *work) · 13 unmodified lines | ||
| 1216 | 1221 | { |
| 1217 | 1222 | struct amdgpu_userq_mgr *uq_mgr = work_to_uq_mgr(work, resume_work.work); |
| 1218 | 1223 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); |
| 1224 | + struct dma_fence *ev_fence; | |
| 1219 | 1225 | int ret; |
| 1220 | 1226 | |
| 1221 | − flush_delayed_work(&fpriv->evf_mgr.suspend_work); | |
| 1222 | − | |
| 1223 | − mutex_lock(&uq_mgr->userq_mutex); | |
| 1227 | + ev_fence = amdgpu_evf_mgr_get_fence(&fpriv->evf_mgr); | |
| 1228 | + if (!dma_fence_is_signaled(ev_fence)) | |
| 1229 | + goto put_fence; | |
| 1224 | 1230 | |
| 1225 | 1231 | ret = amdgpu_userq_vm_validate(uq_mgr); |
| 1226 | 1232 | if (ret) { |
| 1227 | 1233 | drm_file_err(uq_mgr->file, "Failed to validate BOs to restore\n"); |
| 1228 | − goto unlock; | |
| 1234 | + goto put_fence; | |
| 1229 | 1235 | } |
| 1230 | 1236 | |
| 1231 | 1237 | ret = amdgpu_userq_restore_all(uq_mgr); |
| 1232 | − if (ret) { | |
| 1238 | + if (ret) | |
| 1233 | 1239 | drm_file_err(uq_mgr->file, "Failed to restore all queues\n"); |
| 1234 | − goto unlock; | |
| 1235 | − } | |
| 1236 | 1240 | |
| 1237 | −unlock: | |
| 1238 | − mutex_unlock(&uq_mgr->userq_mutex); | |
| 1241 | +put_fence: | |
| 1242 | + dma_fence_put(ev_fence); | |
| 1239 | 1243 | } |
| 1240 | 1244 | |
| 1241 | 1245 | static int |
| @@ -1248,13 +1252,9 @@ amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr) · 6 unmodified lines | ||
| 1248 | 1252 | amdgpu_userq_detect_and_reset_queues(uq_mgr); |
| 1249 | 1253 | /* Try to unmap all the queues in this process ctx */ |
| 1250 | 1254 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { |
| 1251 | − queue = amdgpu_userq_get(uq_mgr, queue_id); | |
| 1252 | − if (!queue) | |
| 1253 | − continue; | |
| 1254 | 1255 | r = amdgpu_userq_preempt_helper(queue); |
| 1255 | 1256 | if (r) |
| 1256 | 1257 | ret = r; |
| 1257 | − amdgpu_userq_put(queue); | |
| 1258 | 1258 | } |
| 1259 | 1259 | |
| 1260 | 1260 | if (ret) |
| @@ -1287,35 +1287,26 @@ amdgpu_userq_wait_for_signal(struct amdgpu_userq_mgr *uq_mgr) · 26 unmodified lines | ||
| 1287 | 1287 | int ret; |
| 1288 | 1288 | |
| 1289 | 1289 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { |
| 1290 | − queue = amdgpu_userq_get(uq_mgr, queue_id); | |
| 1291 | − if (!queue) | |
| 1292 | − continue; | |
| 1293 | − | |
| 1294 | 1290 | struct dma_fence *f = queue->last_fence; |
| 1295 | 1291 | |
| 1296 | − if (!f || dma_fence_is_signaled(f)) { | |
| 1297 | − amdgpu_userq_put(queue); | |
| 1292 | + if (!f || dma_fence_is_signaled(f)) | |
| 1298 | 1293 | continue; |
| 1299 | − } | |
| 1294 | + | |
| 1300 | 1295 | ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100)); |
| 1301 | 1296 | if (ret <= 0) { |
| 1302 | 1297 | drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n", |
| 1303 | 1298 | f->context, f->seqno); |
| 1304 | − amdgpu_userq_put(queue); | |
| 1299 | + | |
| 1305 | 1300 | return -ETIMEDOUT; |
| 1306 | 1301 | } |
| 1307 | − amdgpu_userq_put(queue); | |
| 1308 | 1302 | } |
| 1309 | 1303 | |
| 1310 | 1304 | return 0; |
| 1311 | 1305 | } |
| 1312 | 1306 | |
| 1313 | 1307 | void |
| 1314 | −amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, | |
| 1315 | − struct amdgpu_eviction_fence *ev_fence) | |
| 1308 | +amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr) | |
| 1316 | 1309 | { |
| 1317 | − struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); | |
| 1318 | − struct amdgpu_eviction_fence_mgr *evf_mgr = &fpriv->evf_mgr; | |
| 1319 | 1310 | struct amdgpu_device *adev = uq_mgr->adev; |
| 1320 | 1311 | int ret; |
| 1321 | 1312 | |
| @@ -1328,16 +1319,6 @@ amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, · 6 unmodified lines | ||
| 1328 | 1319 | if (ret) |
| 1329 | 1320 | dev_err(adev->dev, "Failed to evict userqueue\n"); |
| 1330 | 1321 | |
| 1331 | − /* Signal current eviction fence */ | |
| 1332 | − amdgpu_eviction_fence_signal(evf_mgr, ev_fence); | |
| 1333 | − | |
| 1334 | − if (evf_mgr->fd_closing) { | |
| 1335 | − cancel_delayed_work_sync(&uq_mgr->resume_work); | |
| 1336 | − return; | |
| 1337 | − } | |
| 1338 | − | |
| 1339 | − /* Schedule a resume work */ | |
| 1340 | − schedule_delayed_work(&uq_mgr->resume_work, 0); | |
| 1341 | 1322 | } |
| 1342 | 1323 | |
| 1343 | 1324 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, |
| @@ -1352,6 +1333,11 @@ int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *f · 8 unmodified lines | ||
| 1352 | 1333 | return 0; |
| 1353 | 1334 | } |
| 1354 | 1335 | |
| 1336 | +void amdgpu_userq_mgr_cancel_resume(struct amdgpu_userq_mgr *userq_mgr) | |
| 1337 | +{ | |
| 1338 | + cancel_delayed_work_sync(&userq_mgr->resume_work); | |
| 1339 | +} | |
| 1340 | + | |
| 1355 | 1341 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr) |
| 1356 | 1342 | { |
| 1357 | 1343 | struct amdgpu_usermode_queue *queue; |
| @@ -1475,17 +1461,19 @@ int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev, · 117 unmodified lines | ||
| 1475 | 1461 | |
| 1476 | 1462 | if (!adev->userq_halt_for_enforce_isolation) |
| 1477 | 1463 | dev_warn(adev->dev, "userq scheduling already started!\n"); |
| 1464 | + | |
| 1478 | 1465 | adev->userq_halt_for_enforce_isolation = false; |
| 1466 | + | |
| 1479 | 1467 | xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) { |
| 1480 | 1468 | uqm = queue->userq_mgr; |
| 1481 | 1469 | mutex_lock(&uqm->userq_mutex); |
| 1482 | − if (((queue->queue_type == AMDGPU_HW_IP_GFX) || | |
| 1483 | − (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) && | |
| 1484 | − (queue->xcp_id == idx)) { | |
| 1470 | + if (((queue->queue_type == AMDGPU_HW_IP_GFX) || | |
| 1471 | + (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) && | |
| 1472 | + (queue->xcp_id == idx)) { | |
| 1485 | 1473 | r = amdgpu_userq_restore_helper(queue); |
| 1486 | 1474 | if (r) |
| 1487 | 1475 | ret = r; |
| 1488 | − } | |
| 1476 | + } | |
| 1489 | 1477 | mutex_unlock(&uqm->userq_mutex); |
| 1490 | 1478 | } |
| 1491 | 1479 | |
| @@ -123,6 +123,7 @@ int amdgpu_userq_ioctl(struct drm_device *dev, void *data, struct drm_file *filp · 122 unmodified lines | ||
| 123 | 123 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, |
| 124 | 124 | struct amdgpu_device *adev); |
| 125 | 125 | |
| 126 | +void amdgpu_userq_mgr_cancel_resume(struct amdgpu_userq_mgr *userq_mgr); | |
| 126 | 127 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr); |
| 127 | 128 | |
| 128 | 129 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, |
| @@ -132,8 +133,7 @@ int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, · 3 unmodified lines | ||
| 132 | 133 | void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr, |
| 133 | 134 | struct amdgpu_userq_obj *userq_obj); |
| 134 | 135 | |
| 135 | −void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, | |
| 136 | − struct amdgpu_eviction_fence *ev_fence); | |
| 136 | +void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr); | |
| 137 | 137 | |
| 138 | 138 | void amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *userq_mgr, |
| 139 | 139 | struct amdgpu_eviction_fence_mgr *evf_mgr); |
| @@ -597,11 +597,11 @@ exec_fini: · 596 unmodified lines | ||
| 597 | 597 | put_gobj_write: |
| 598 | 598 | for (i = 0; i < num_write_bo_handles; i++) |
| 599 | 599 | drm_gem_object_put(gobj_write[i]); |
| 600 | − kfree(gobj_write); | |
| 600 | + kvfree(gobj_write); | |
| 601 | 601 | put_gobj_read: |
| 602 | 602 | for (i = 0; i < num_read_bo_handles; i++) |
| 603 | 603 | drm_gem_object_put(gobj_read[i]); |
| 604 | − kfree(gobj_read); | |
| 604 | + kvfree(gobj_read); | |
| 605 | 605 | free_syncobj: |
| 606 | 606 | while (entry-- > 0) |
| 607 | 607 | if (syncobj[entry]) |
| @@ -616,339 +616,405 @@ free_syncobj_handles: · 8 unmodified lines | ||
| 616 | 616 | return r; |
| 617 | 617 | } |
| 618 | 618 | |
| 619 | −int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data, | |
| 620 | − struct drm_file *filp) | |
| 619 | +/* Count the number of expected fences so userspace can alloc a buffer */ | |
| 620 | +static int | |
| 621 | +amdgpu_userq_wait_count_fences(struct drm_file *filp, | |
| 622 | + struct drm_amdgpu_userq_wait *wait_info, | |
| 623 | + u32 *syncobj_handles, u32 *timeline_points, | |
| 624 | + u32 *timeline_handles, | |
| 625 | + struct drm_gem_object **gobj_write, | |
| 626 | + struct drm_gem_object **gobj_read) | |
| 621 | 627 | { |
| 622 | − struct drm_amdgpu_userq_wait *wait_info = data; | |
| 623 | − const unsigned int num_write_bo_handles = wait_info->num_bo_write_handles; | |
| 624 | − const unsigned int num_read_bo_handles = wait_info->num_bo_read_handles; | |
| 625 | − struct drm_amdgpu_userq_fence_info *fence_info = NULL; | |
| 626 | − struct amdgpu_fpriv *fpriv = filp->driver_priv; | |
| 627 | − struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr; | |
| 628 | − struct drm_gem_object **gobj_write, **gobj_read; | |
| 629 | − u32 *timeline_points, *timeline_handles; | |
| 630 | − struct amdgpu_usermode_queue *waitq = NULL; | |
| 631 | − u32 *syncobj_handles, num_syncobj; | |
| 632 | − struct dma_fence **fences = NULL; | |
| 633 | − u16 num_points, num_fences = 0; | |
| 628 | + int num_read_bo_handles, num_write_bo_handles; | |
| 629 | + struct dma_fence_unwrap iter; | |
| 630 | + struct dma_fence *fence, *f; | |
| 631 | + unsigned int num_fences = 0; | |
| 634 | 632 | struct drm_exec exec; |
| 635 | − int r, i, cnt; | |
| 636 | − | |
| 637 | − if (!amdgpu_userq_enabled(dev)) | |
| 638 | − return -ENOTSUPP; | |
| 639 | − | |
| 640 | − if (wait_info->num_bo_write_handles > AMDGPU_USERQ_MAX_HANDLES || | |
| 641 | − wait_info->num_bo_read_handles > AMDGPU_USERQ_MAX_HANDLES) | |
| 642 | − return -EINVAL; | |
| 643 | − | |
| 644 | − num_syncobj = wait_info->num_syncobj_handles; | |
| 645 | − syncobj_handles = memdup_array_user(u64_to_user_ptr(wait_info->syncobj_handles), | |
| 646 | − num_syncobj, sizeof(u32)); | |
| 647 | − if (IS_ERR(syncobj_handles)) | |
| 648 | − return PTR_ERR(syncobj_handles); | |
| 649 | − | |
| 633 | + int i, r; | |
| 634 | + | |
| 635 | + /* | |
| 636 | + * This needs to be outside of the lock provided by drm_exec for | |
| 637 | + * DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT to work correctly. | |
| 638 | + */ | |
| 639 | + | |
| 640 | + /* Count timeline fences */ | |
| 641 | + for (i = 0; i < wait_info->num_syncobj_timeline_handles; i++) { | |
| 642 | + r = drm_syncobj_find_fence(filp, timeline_handles[i], | |
| 643 | + timeline_points[i], | |
| 644 | + DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | |
| 645 | + &fence); | |
| 646 | + if (r) | |
| 647 | + return r; | |
| 648 | + | |
| 649 | + dma_fence_unwrap_for_each(f, &iter, fence) | |
| 650 | + num_fences++; | |
| 650 | 651 | |
| 651 | − num_points = wait_info->num_syncobj_timeline_handles; | |
| 652 | − timeline_handles = memdup_array_user(u64_to_user_ptr(wait_info->syncobj_timeline_handles), | |
| 653 | − num_points, sizeof(u32)); | |
| 654 | − if (IS_ERR(timeline_handles)) { | |
| 655 | − r = PTR_ERR(timeline_handles); | |
| 656 | − goto free_syncobj_handles; | |
| 652 | + dma_fence_put(fence); | |
| 657 | 653 | } |
| 658 | 654 | |
| 659 | − timeline_points = memdup_array_user(u64_to_user_ptr(wait_info->syncobj_timeline_points), | |
| 660 | − num_points, sizeof(u32)); | |
| 655 | + /* Count boolean fences */ | |
| 656 | + for (i = 0; i < wait_info->num_syncobj_handles; i++) { | |
| 657 | + r = drm_syncobj_find_fence(filp, syncobj_handles[i], 0, | |
| 658 | + DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | |
| 659 | + &fence); | |
| 660 | + if (r) | |
| 661 | + return r; | |
| 661 | 662 | |
| 662 | − if (IS_ERR(timeline_points)) { | |
| 663 | − r = PTR_ERR(timeline_points); | |
| 664 | − goto free_timeline_handles; | |
| 663 | + num_fences++; | |
| 664 | + dma_fence_put(fence); | |
| 665 | 665 | } |
| 666 | 666 | |
| 667 | − r = drm_gem_objects_lookup(filp, | |
| 668 | − u64_to_user_ptr(wait_info->bo_read_handles), | |
| 669 | − num_read_bo_handles, | |
| 670 | − &gobj_read); | |
| 671 | − if (r) | |
| 672 | − goto free_timeline_points; | |
| 673 | − | |
| 674 | − r = drm_gem_objects_lookup(filp, | |
| 675 | − u64_to_user_ptr(wait_info->bo_write_handles), | |
| 676 | − num_write_bo_handles, | |
| 677 | − &gobj_write); | |
| 678 | − if (r) | |
| 679 | − goto put_gobj_read; | |
| 680 | − | |
| 667 | + /* Lock all the GEM objects */ | |
| 668 | + /* TODO: It is actually not necessary to lock them */ | |
| 669 | + num_read_bo_handles = wait_info->num_bo_read_handles; | |
| 670 | + num_write_bo_handles = wait_info->num_bo_write_handles; | |
| 681 | 671 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, |
| 682 | − (num_read_bo_handles + num_write_bo_handles)); | |
| 672 | + num_read_bo_handles + num_write_bo_handles); | |
| 683 | 673 | |
| 684 | − /* Lock all BOs with retry handling */ | |
| 685 | 674 | drm_exec_until_all_locked(&exec) { |
| 686 | − r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1); | |
| 675 | + r = drm_exec_prepare_array(&exec, gobj_read, | |
| 676 | + num_read_bo_handles, 1); | |
| 687 | 677 | drm_exec_retry_on_contention(&exec); |
| 688 | − if (r) { | |
| 689 | − drm_exec_fini(&exec); | |
| 690 | − goto put_gobj_write; | |
| 691 | − } | |
| 678 | + if (r) | |
| 679 | + goto error_unlock; | |
| 692 | 680 | |
| 693 | − r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1); | |
| 681 | + r = drm_exec_prepare_array(&exec, gobj_write, | |
| 682 | + num_write_bo_handles, 1); | |
| 694 | 683 | drm_exec_retry_on_contention(&exec); |
| 695 | − if (r) { | |
| 696 | − drm_exec_fini(&exec); | |
| 697 | − goto put_gobj_write; | |
| 698 | − } | |
| 684 | + if (r) | |
| 685 | + goto error_unlock; | |
| 699 | 686 | } |
| 700 | 687 | |
| 701 | − if (!wait_info->num_fences) { | |
| 702 | − if (num_points) { | |
| 703 | − struct dma_fence_unwrap iter; | |
| 704 | − struct dma_fence *fence; | |
| 705 | − struct dma_fence *f; | |
| 706 | − | |
| 707 | − for (i = 0; i < num_points; i++) { | |
| 708 | − r = drm_syncobj_find_fence(filp, timeline_handles[i], | |
| 709 | − timeline_points[i], | |
| 710 | − DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | |
| 711 | − &fence); | |
| 712 | − if (r) | |
| 713 | − goto exec_fini; | |
| 714 | − | |
| 715 | − dma_fence_unwrap_for_each(f, &iter, fence) | |
| 716 | − num_fences++; | |
| 717 | − | |
| 718 | − dma_fence_put(fence); | |
| 719 | − } | |
| 720 | − } | |
| 688 | + /* Count read fences */ | |
| 689 | + for (i = 0; i < num_read_bo_handles; i++) { | |
| 690 | + struct dma_resv_iter resv_cursor; | |
| 691 | + struct dma_fence *fence; | |
511 more lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 548 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 174 lines aren’t shown here.
This diff is large: its 383 lines aren’t shown here.
This diff is large: its 238 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 164 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 841 lines aren’t shown here.
This diff is large: its 112 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 473 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 1220 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 143 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 336 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 599 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 407 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 303 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 160 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 424 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 382 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 159 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 149 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 134 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 150 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 202 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 296 lines aren’t shown here.
This diff is large: its 428 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 190 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 305 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 2271 lines aren’t shown here.
This diff is large: its 314 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 245 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 599 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 192 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 209 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 145 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 266 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 4 lines aren’t shown here.
This diff is large: its 4 lines aren’t shown here.
This diff is large: its 4 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 180 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 496 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 179 lines aren’t shown here.
This diff is large: its 149 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 215 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 413 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 316 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 148 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 121 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 232 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 301 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 369 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 292 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 292 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 364 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 442 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 750 lines aren’t shown here.
This diff is large: its 135 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 295 lines aren’t shown here.
This diff is large: its 324 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 302 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 288 lines aren’t shown here.
This diff is large: its 201 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 317 lines aren’t shown here.
This diff is large: its 660 lines aren’t shown here.
This diff is large: its 409 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 181 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 287 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 190 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 448 lines aren’t shown here.
This diff is large: its 174 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 92 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 228 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 328 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 1002 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 127 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 211 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 308 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 291 lines aren’t shown here.
This diff is large: its 351 lines aren’t shown here.
This diff is large: its 170 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 211 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 757 lines aren’t shown here.
This diff is large: its 308 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 586 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 715 lines aren’t shown here.
This diff is large: its 740 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 203 lines aren’t shown here.
This diff is large: its 280 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 540 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 255 lines aren’t shown here.
This diff is large: its 299 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 302 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 157 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 485 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 174 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 301 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 135 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 189 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 213 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 112 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 216 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 303 lines aren’t shown here.
This diff is large: its 184 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 237 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 176 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 481 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 181 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 178 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 269 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 142 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 185 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 586 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 327 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 201 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 256 lines aren’t shown here.
This diff is large: its 288 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 288 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 364 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 461 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 286 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 333 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 322 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 1163 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 229 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 615 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 340 lines aren’t shown here.
This diff is large: its 948 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 1261 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 306 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 156 lines aren’t shown here.
This diff is large: its 2 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 548 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 162 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 194 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 266 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 179 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 2 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 866 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 353 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 151 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 105 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 195 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.