Sync drm-xe-next with drm-next to unblock some topic branches.
Unified · Split
| @@ -1,6 +1,7 @@ | |||
| 1 | Alan Cox <alan@lxorguk.ukuu.org.uk> | 1 | Alan Cox <alan@lxorguk.ukuu.org.uk> |
| 2 | Alan Cox <root@hraefn.swansea.linux.org.uk> | 2 | Alan Cox <root@hraefn.swansea.linux.org.uk> |
| 3 | Alyssa Rosenzweig <alyssa@rosenzweig.io> | 3 | Alyssa Rosenzweig <alyssa@rosenzweig.io> |
| 4 | Askar Safin <safinaskar@gmail.com> | ||
| 4 | Christoph Hellwig <hch@lst.de> | 5 | Christoph Hellwig <hch@lst.de> |
| 5 | Jeff Kirsher <jeffrey.t.kirsher@intel.com> | 6 | Jeff Kirsher <jeffrey.t.kirsher@intel.com> |
| 6 | Marc Gonzalez <marc.w.gonzalez@free.fr> | 7 | Marc Gonzalez <marc.w.gonzalez@free.fr> |
| @@ -196,6 +196,7 @@ Christophe Leroy <chleroy@kernel.org> <christophe.leroy2@cs-soprasteria.com> · 195 unmodified lines | |||
| 196 | Christophe Ricard <christophe.ricard@gmail.com> | 196 | Christophe Ricard <christophe.ricard@gmail.com> |
| 197 | Christopher Obbard <christopher.obbard@linaro.org> <chris.obbard@collabora.com> | 197 | Christopher Obbard <christopher.obbard@linaro.org> <chris.obbard@collabora.com> |
| 198 | Christoph Hellwig <hch@lst.de> | 198 | Christoph Hellwig <hch@lst.de> |
| 199 | Christoph Manszewski <c.manszewski@gmail.com> <christoph.manszewski@intel.com> | ||
| 199 | Chuck Lever <chuck.lever@oracle.com> <cel@kernel.org> | 200 | Chuck Lever <chuck.lever@oracle.com> <cel@kernel.org> |
| 200 | Chuck Lever <chuck.lever@oracle.com> <cel@netapp.com> | 201 | Chuck Lever <chuck.lever@oracle.com> <cel@netapp.com> |
| 201 | Chuck Lever <chuck.lever@oracle.com> <cel@citi.umich.edu> | 202 | Chuck Lever <chuck.lever@oracle.com> <cel@citi.umich.edu> |
| @@ -316,6 +317,7 @@ Hans Verkuil <hverkuil@kernel.org> <hverkuil-cisco@xs4all.nl> · 114 unmodified lines | |||
| 316 | Hans Verkuil <hverkuil@kernel.org> <hansverk@cisco.com> | 317 | Hans Verkuil <hverkuil@kernel.org> <hansverk@cisco.com> |
| 317 | Hao Ge <hao.ge@linux.dev> <gehao@kylinos.cn> | 318 | Hao Ge <hao.ge@linux.dev> <gehao@kylinos.cn> |
| 318 | Harry Yoo <harry.yoo@oracle.com> <42.hyeyoo@gmail.com> | 319 | Harry Yoo <harry.yoo@oracle.com> <42.hyeyoo@gmail.com> |
| 320 | Harry Yoo <harry@kernel.org> <harry.yoo@oracle.com> | ||
| 319 | Heiko Carstens <hca@linux.ibm.com> <h.carstens@de.ibm.com> | 321 | Heiko Carstens <hca@linux.ibm.com> <h.carstens@de.ibm.com> |
| 320 | Heiko Carstens <hca@linux.ibm.com> <heiko.carstens@de.ibm.com> | 322 | Heiko Carstens <hca@linux.ibm.com> <heiko.carstens@de.ibm.com> |
| 321 | Heiko Stuebner <heiko@sntech.de> <heiko.stuebner@bqreaders.com> | 323 | Heiko Stuebner <heiko@sntech.de> <heiko.stuebner@bqreaders.com> |
| @@ -327,6 +329,7 @@ Henrik Rydberg <rydberg@bitmath.org> · 5 unmodified lines | |||
| 327 | Herbert Xu <herbert@gondor.apana.org.au> | 329 | Herbert Xu <herbert@gondor.apana.org.au> |
| 328 | Huacai Chen <chenhuacai@kernel.org> <chenhc@lemote.com> | 330 | Huacai Chen <chenhuacai@kernel.org> <chenhc@lemote.com> |
| 329 | Huacai Chen <chenhuacai@kernel.org> <chenhuacai@loongson.cn> | 331 | Huacai Chen <chenhuacai@kernel.org> <chenhuacai@loongson.cn> |
| 332 | Ignat Korchagin <ignat@linux.win> <ignat@cloudflare.com> | ||
| 330 | Ike Panhc <ikepanhc@gmail.com> <ike.pan@canonical.com> | 333 | Ike Panhc <ikepanhc@gmail.com> <ike.pan@canonical.com> |
| 331 | J. Bruce Fields <bfields@fieldses.org> <bfields@redhat.com> | 334 | J. Bruce Fields <bfields@fieldses.org> <bfields@redhat.com> |
| 332 | J. Bruce Fields <bfields@fieldses.org> <bfields@citi.umich.edu> | 335 | J. Bruce Fields <bfields@fieldses.org> <bfields@citi.umich.edu> |
| @@ -498,7 +501,8 @@ Lior David <quic_liord@quicinc.com> <liord@codeaurora.org> · 165 unmodified lines | |||
| 498 | Loic Poulain <loic.poulain@oss.qualcomm.com> <loic.poulain@linaro.org> | 501 | Loic Poulain <loic.poulain@oss.qualcomm.com> <loic.poulain@linaro.org> |
| 499 | Loic Poulain <loic.poulain@oss.qualcomm.com> <loic.poulain@intel.com> | 502 | Loic Poulain <loic.poulain@oss.qualcomm.com> <loic.poulain@intel.com> |
| 500 | Lorenzo Pieralisi <lpieralisi@kernel.org> <lorenzo.pieralisi@arm.com> | 503 | Lorenzo Pieralisi <lpieralisi@kernel.org> <lorenzo.pieralisi@arm.com> |
| 501 | Lorenzo Stoakes <lorenzo.stoakes@oracle.com> <lstoakes@gmail.com> | 504 | Lorenzo Stoakes <ljs@kernel.org> <lstoakes@gmail.com> |
| 505 | Lorenzo Stoakes <ljs@kernel.org> <lorenzo.stoakes@oracle.com> | ||
| 502 | Luca Ceresoli <luca.ceresoli@bootlin.com> <luca@lucaceresoli.net> | 506 | Luca Ceresoli <luca.ceresoli@bootlin.com> <luca@lucaceresoli.net> |
| 503 | Luca Weiss <luca@lucaweiss.eu> <luca@z3ntu.xyz> | 507 | Luca Weiss <luca@lucaweiss.eu> <luca@z3ntu.xyz> |
| 504 | Lucas De Marchi <demarchi@kernel.org> <lucas.demarchi@intel.com> | 508 | Lucas De Marchi <demarchi@kernel.org> <lucas.demarchi@intel.com> |
| @@ -585,6 +589,7 @@ Morten Welinder <terra@gnome.org> · 80 unmodified lines | |||
| 585 | Morten Welinder <welinder@anemone.rentec.com> | 589 | Morten Welinder <welinder@anemone.rentec.com> |
| 586 | Morten Welinder <welinder@darter.rentec.com> | 590 | Morten Welinder <welinder@darter.rentec.com> |
| 587 | Morten Welinder <welinder@troll.com> | 591 | Morten Welinder <welinder@troll.com> |
| 592 | Muhammad Usama Anjum <usama.anjum@arm.com> <usama.anjum@collabora.com> | ||
| 588 | Mukesh Ojha <quic_mojha@quicinc.com> <mojha@codeaurora.org> | 593 | Mukesh Ojha <quic_mojha@quicinc.com> <mojha@codeaurora.org> |
| 589 | Muna Sinada <quic_msinada@quicinc.com> <msinada@codeaurora.org> | 594 | Muna Sinada <quic_msinada@quicinc.com> <msinada@codeaurora.org> |
| 590 | Murali Nalajala <quic_mnalajal@quicinc.com> <mnalajal@codeaurora.org> | 595 | Murali Nalajala <quic_mnalajal@quicinc.com> <mnalajal@codeaurora.org> |
| @@ -151,11 +151,11 @@ Description: · 150 unmodified lines | |||
| 151 | The algorithm_params file is write-only and is used to setup | 151 | The algorithm_params file is write-only and is used to setup |
| 152 | compression algorithm parameters. | 152 | compression algorithm parameters. |
| 153 | 153 | ||
| 154 | What: /sys/block/zram<id>/writeback_compressed | 154 | What: /sys/block/zram<id>/compressed_writeback |
| 155 | Date: Decemeber 2025 | 155 | Date: Decemeber 2025 |
| 156 | Contact: Richard Chang <richardycc@google.com> | 156 | Contact: Richard Chang <richardycc@google.com> |
| 157 | Description: | 157 | Description: |
| 158 | The writeback_compressed device atrribute toggles compressed | 158 | The compressed_writeback device atrribute toggles compressed |
| 159 | writeback feature. | 159 | writeback feature. |
| 160 | 160 | ||
| 161 | What: /sys/block/zram<id>/writeback_batch_size | 161 | What: /sys/block/zram<id>/writeback_batch_size |
| @@ -85,6 +85,16 @@ In the example, 'Requester ID' means the ID of the device that sent · 84 unmodified lines | |||
| 85 | the error message to the Root Port. Please refer to PCIe specs for other | 85 | the error message to the Root Port. Please refer to PCIe specs for other |
| 86 | fields. | 86 | fields. |
| 87 | 87 | ||
| 88 | The 'TLP Header' is the prefix/header of the TLP that caused the error | ||
| 89 | in raw hex format. To decode the TLP Header into human-readable form | ||
| 90 | one may use tlp-tool: | ||
| 91 | |||
| 92 | https://github.com/mmpg-x86/tlp-tool | ||
| 93 | |||
| 94 | Example usage:: | ||
| 95 | |||
| 96 | curl -L https://git.kernel.org/linus/2ca1c94ce0b6 | rtlp-tool --aer | ||
| 97 | |||
| 88 | AER Ratelimits | 98 | AER Ratelimits |
| 89 | -------------- | 99 | -------------- |
| 90 | 100 | ||
| @@ -216,7 +216,7 @@ writeback_limit WO specifies the maximum amount of write IO zram · 215 unmodified lines | |||
| 216 | writeback_limit_enable RW show and set writeback_limit feature | 216 | writeback_limit_enable RW show and set writeback_limit feature |
| 217 | writeback_batch_size RW show and set maximum number of in-flight | 217 | writeback_batch_size RW show and set maximum number of in-flight |
| 218 | writeback operations | 218 | writeback operations |
| 219 | writeback_compressed RW show and set compressed writeback feature | 219 | compressed_writeback RW show and set compressed writeback feature |
| 220 | comp_algorithm RW show and change the compression algorithm | 220 | comp_algorithm RW show and change the compression algorithm |
| 221 | algorithm_params WO setup compression algorithm parameters | 221 | algorithm_params WO setup compression algorithm parameters |
| 222 | compact WO trigger memory compaction | 222 | compact WO trigger memory compaction |
| @@ -439,11 +439,11 @@ budget in next setting is user's job. · 216 unmodified lines | |||
| 439 | By default zram stores written back pages in decompressed (raw) form, which | 439 | By default zram stores written back pages in decompressed (raw) form, which |
| 440 | means that writeback operation involves decompression of the page before | 440 | means that writeback operation involves decompression of the page before |
| 441 | writing it to the backing device. This behavior can be changed by enabling | 441 | writing it to the backing device. This behavior can be changed by enabling |
| 442 | `writeback_compressed` feature, which causes zram to write compressed pages | 442 | `compressed_writeback` feature, which causes zram to write compressed pages |
| 443 | to the backing device, thus avoiding decompression overhead. To enable | 443 | to the backing device, thus avoiding decompression overhead. To enable |
| 444 | this feature, execute:: | 444 | this feature, execute:: |
| 445 | 445 | ||
| 446 | $ echo yes > /sys/block/zramX/writeback_compressed | 446 | $ echo yes > /sys/block/zramX/compressed_writeback |
| 447 | 447 | ||
| 448 | Note that this feature should be configured before the `zramX` device is | 448 | Note that this feature should be configured before the `zramX` device is |
| 449 | initialized. | 449 | initialized. |
| @@ -8196,6 +8196,9 @@ Kernel parameters · 8195 unmodified lines | |||
| 8196 | p = USB_QUIRK_SHORT_SET_ADDRESS_REQ_TIMEOUT | 8196 | p = USB_QUIRK_SHORT_SET_ADDRESS_REQ_TIMEOUT |
| 8197 | (Reduce timeout of the SET_ADDRESS | 8197 | (Reduce timeout of the SET_ADDRESS |
| 8198 | request from 5000 ms to 500 ms); | 8198 | request from 5000 ms to 500 ms); |
| 8199 | q = USB_QUIRK_FORCE_ONE_CONFIG (Device | ||
| 8200 | claims zero configurations, | ||
| 8201 | forcing to 1); | ||
| 8199 | Example: quirks=0781:5580:bk,0a5c:5834:gij | 8202 | Example: quirks=0781:5580:bk,0a5c:5834:gij |
| 8200 | 8203 | ||
| 8201 | usbhid.mousepoll= | 8204 | usbhid.mousepoll= |
| @@ -149,11 +149,33 @@ For architectures that require cache flushing for DMA coherence · 148 unmodified lines | |||
| 149 | DMA_ATTR_MMIO will not perform any cache flushing. The address | 149 | DMA_ATTR_MMIO will not perform any cache flushing. The address |
| 150 | provided must never be mapped cacheable into the CPU. | 150 | provided must never be mapped cacheable into the CPU. |
| 151 | 151 | ||
| 152 | DMA_ATTR_CPU_CACHE_CLEAN | 152 | DMA_ATTR_DEBUGGING_IGNORE_CACHELINES |
| 153 | ------------------------ | 153 | ------------------------------------ |
| 154 | 154 | ||
| 155 | This attribute indicates the CPU will not dirty any cacheline overlapping this | 155 | This attribute indicates that CPU cache lines may overlap for buffers mapped |
| 156 | DMA_FROM_DEVICE/DMA_BIDIRECTIONAL buffer while it is mapped. This allows | 156 | with DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. |
| 157 | multiple small buffers to safely share a cacheline without risk of data | 157 | |
| 158 | corruption, suppressing DMA debug warnings about overlapping mappings. | 158 | Such overlap may occur when callers map multiple small buffers that reside |
| 159 | All mappings sharing a cacheline should have this attribute. | 159 | within the same cache line. In this case, callers must guarantee that the CPU |
| 160 | will not dirty these cache lines after the mappings are established. When this | ||
| 161 | condition is met, multiple buffers can safely share a cache line without risking | ||
| 162 | data corruption. | ||
| 163 | |||
| 164 | All mappings that share a cache line must set this attribute to suppress DMA | ||
| 165 | debug warnings about overlapping mappings. | ||
| 166 | |||
| 167 | DMA_ATTR_REQUIRE_COHERENT | ||
| 168 | ------------------------- | ||
| 169 | |||
| 170 | DMA mapping requests with the DMA_ATTR_REQUIRE_COHERENT fail on any | ||
| 171 | system where SWIOTLB or cache management is required. This should only | ||
| 172 | be used to support uAPI designs that require continuous HW DMA | ||
| 173 | coherence with userspace processes, for example RDMA and DRM. At a | ||
| 174 | minimum the memory being mapped must be userspace memory from | ||
| 175 | pin_user_pages() or similar. | ||
| 176 | |||
| 177 | Drivers should consider using dma_mmap_pages() instead of this | ||
| 178 | interface when building their uAPIs, when possible. | ||
| 179 | |||
| 180 | It must never be used in an in-kernel driver that only works with | ||
| 181 | kernel memory. | ||
| @@ -336,6 +336,8 @@ command line arguments: · 335 unmodified lines | |||
| 336 | - ``--list_tests_attr``: If set, lists all tests that will be run and all of their | 336 | - ``--list_tests_attr``: If set, lists all tests that will be run and all of their |
| 337 | attributes. | 337 | attributes. |
| 338 | 338 | ||
| 339 | - ``--list_suites``: If set, lists all suites that will be run. | ||
| 340 | |||
| 339 | Command-line completion | 341 | Command-line completion |
| 340 | ============================== | 342 | ============================== |
| 341 | 343 | ||
| @@ -66,7 +66,7 @@ then: · 65 unmodified lines | |||
| 66 | required: | 66 | required: |
| 67 | - refresh-rate-hz | 67 | - refresh-rate-hz |
| 68 | 68 | ||
| 69 | additionalProperties: false | 69 | unevaluatedProperties: false |
| 70 | 70 | ||
| 71 | examples: | 71 | examples: |
| 72 | - | | 72 | - | |
| @@ -301,6 +301,7 @@ properties: · 300 unmodified lines | |||
| 301 | maxItems: 4 | 301 | maxItems: 4 |
| 302 | 302 | ||
| 303 | dependencies: | 303 | dependencies: |
| 304 | pd-disable: [typec-power-opmode] | ||
| 304 | sink-vdos-v1: [ sink-vdos ] | 305 | sink-vdos-v1: [ sink-vdos ] |
| 305 | sink-vdos: [ sink-vdos-v1 ] | 306 | sink-vdos: [ sink-vdos-v1 ] |
| 306 | 307 | ||
| @@ -19,7 +19,9 @@ properties: · 18 unmodified lines | |||
| 19 | compatible: | 19 | compatible: |
| 20 | oneOf: | 20 | oneOf: |
| 21 | - items: | 21 | - items: |
| 22 | - const: arm,mali-d32 | 22 | - enum: |
| 23 | - arm,mali-d32 | ||
| 24 | - armchina,linlon-d6 | ||
| 23 | - const: arm,mali-d71 | 25 | - const: arm,mali-d71 |
| 24 | - const: arm,mali-d71 | 26 | - const: arm,mali-d71 |
| 25 | 27 | ||
| @@ -0,0 +1,113 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/bridge/lontium,lt8713sx.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Lontium LT8713SX Type-C/DP1.4 to Type-C/DP1.4/HDMI2.0/DP++ bridge-hub | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Vishnu Saini <vishnu.saini@oss.qualcomm.com> | ||
| 11 | |||
| 12 | description: | ||
| 13 | The Lontium LT8713SX is a Type-C/DP1.4 to Type-C/DP1.4/HDMI2.0 converter | ||
| 14 | that integrates one DP input and up to three configurable output interfaces | ||
| 15 | (DP1.4 / HDMI2.0 / DP++), with SST/MST functionality and audio support. | ||
| 16 | |||
| 17 | properties: | ||
| 18 | compatible: | ||
| 19 | enum: | ||
| 20 | - lontium,lt8713sx | ||
| 21 | |||
| 22 | reg: | ||
| 23 | maxItems: 1 | ||
| 24 | |||
| 25 | vcc-supply: | ||
| 26 | description: Regulator for 3.3V vcc. | ||
| 27 | |||
| 28 | vdd-supply: | ||
| 29 | description: Regulator for 1.1V vdd. | ||
| 30 | |||
| 31 | reset-gpios: | ||
| 32 | description: GPIO connected to active low RESET pin. | ||
| 33 | |||
| 34 | ports: | ||
| 35 | $ref: /schemas/graph.yaml#/properties/ports | ||
| 36 | |||
| 37 | properties: | ||
| 38 | port@0: | ||
| 39 | $ref: /schemas/graph.yaml#/properties/port | ||
| 40 | description: | ||
| 41 | DP port for DP input from soc to bridge chip | ||
| 42 | |||
| 43 | port@1: | ||
| 44 | $ref: /schemas/graph.yaml#/properties/port | ||
| 45 | description: | ||
| 46 | DP port for DP output from bridge | ||
| 47 | |||
| 48 | port@2: | ||
| 49 | $ref: /schemas/graph.yaml#/properties/port | ||
| 50 | description: | ||
| 51 | Additional DP port for DP output from bridge | ||
| 52 | |||
| 53 | port@3: | ||
| 54 | $ref: /schemas/graph.yaml#/properties/port | ||
| 55 | description: | ||
| 56 | Additional DP port for DP output from bridge | ||
| 57 | |||
| 58 | required: | ||
| 59 | - port@0 | ||
| 60 | |||
| 61 | required: | ||
| 62 | - compatible | ||
| 63 | - reg | ||
| 64 | - ports | ||
| 65 | |||
| 66 | additionalProperties: false | ||
| 67 | |||
| 68 | examples: | ||
| 69 | - | | ||
| 70 | #include <dt-bindings/gpio/gpio.h> | ||
| 71 | |||
| 72 | i2c { | ||
| 73 | #address-cells = <1>; | ||
| 74 | #size-cells = <0>; | ||
| 75 | bridge@4f { | ||
| 76 | compatible = "lontium,lt8713sx"; | ||
| 77 | reg = <0x4f>; | ||
| 78 | reset-gpios = <&tlmm 6 GPIO_ACTIVE_LOW>; | ||
| 79 | |||
| 80 | ports { | ||
| 81 | #address-cells = <1>; | ||
| 82 | #size-cells = <0>; | ||
| 83 | |||
| 84 | port@0 { | ||
| 85 | reg = <0>; | ||
| 86 | lt8713sx_dp_in: endpoint { | ||
| 87 | remote-endpoint = <&mdss_dp0_out>; | ||
| 88 | }; | ||
| 89 | }; | ||
| 90 | |||
| 91 | port@1 { | ||
| 92 | reg = <1>; | ||
| 93 | lt8713sx_dp0_out: endpoint { | ||
| 94 | remote-endpoint = <&dp0_connector_in>; | ||
| 95 | }; | ||
| 96 | }; | ||
| 97 | |||
| 98 | port@2 { | ||
| 99 | reg = <2>; | ||
| 100 | lt8713sx_dp1_out: endpoint { | ||
| 101 | remote-endpoint = <&dp1_connector_in>; | ||
| 102 | }; | ||
| 103 | }; | ||
| 104 | |||
| 105 | port@3 { | ||
| 106 | reg = <3>; | ||
| 107 | lt8713sx_dp2_out: endpoint { | ||
| 108 | remote-endpoint = <&dp2_connector_in>; | ||
| 109 | }; | ||
| 110 | }; | ||
| 111 | }; | ||
| 112 | }; | ||
| 113 | }; | ||
| @@ -33,6 +33,7 @@ properties: · 32 unmodified lines | |||
| 33 | oneOf: | 33 | oneOf: |
| 34 | - items: | 34 | - items: |
| 35 | - enum: | 35 | - enum: |
| 36 | - doestek,dtc34lm85am # For the Doestek DTC34LM85AM Flat Panel Display (FPD) Transmitter | ||
| 36 | - onnn,fin3385 # OnSemi FIN3385 | 37 | - onnn,fin3385 # OnSemi FIN3385 |
| 37 | - ti,ds90c185 # For the TI DS90C185 FPD-Link Serializer | 38 | - ti,ds90c185 # For the TI DS90C185 FPD-Link Serializer |
| 38 | - ti,ds90c187 # For the TI DS90C187 FPD-Link Serializer | 39 | - ti,ds90c187 # For the TI DS90C187 FPD-Link Serializer |
| @@ -33,6 +33,7 @@ properties: · 32 unmodified lines | |||
| 33 | - enum: | 33 | - enum: |
| 34 | - mediatek,mt2712-disp-aal | 34 | - mediatek,mt2712-disp-aal |
| 35 | - mediatek,mt6795-disp-aal | 35 | - mediatek,mt6795-disp-aal |
| 36 | - mediatek,mt8167-disp-aal | ||
| 36 | - const: mediatek,mt8173-disp-aal | 37 | - const: mediatek,mt8173-disp-aal |
| 37 | - items: | 38 | - items: |
| 38 | - enum: | 39 | - enum: |
| @@ -25,7 +25,9 @@ properties: · 24 unmodified lines | |||
| 25 | - mediatek,mt8183-disp-ccorr | 25 | - mediatek,mt8183-disp-ccorr |
| 26 | - mediatek,mt8192-disp-ccorr | 26 | - mediatek,mt8192-disp-ccorr |
| 27 | - items: | 27 | - items: |
| 28 | - const: mediatek,mt8365-disp-ccorr | 28 | - enum: |
| 29 | - mediatek,mt8167-disp-ccorr | ||
| 30 | - mediatek,mt8365-disp-ccorr | ||
| 29 | - const: mediatek,mt8183-disp-ccorr | 31 | - const: mediatek,mt8183-disp-ccorr |
| 30 | - items: | 32 | - items: |
| 31 | - enum: | 33 | - enum: |
| @@ -26,6 +26,7 @@ properties: · 25 unmodified lines | |||
| 26 | - mediatek,mt8183-disp-dither | 26 | - mediatek,mt8183-disp-dither |
| 27 | - items: | 27 | - items: |
| 28 | - enum: | 28 | - enum: |
| 29 | - mediatek,mt8167-disp-dither | ||
| 29 | - mediatek,mt8186-disp-dither | 30 | - mediatek,mt8186-disp-dither |
| 30 | - mediatek,mt8188-disp-dither | 31 | - mediatek,mt8188-disp-dither |
| 31 | - mediatek,mt8192-disp-dither | 32 | - mediatek,mt8192-disp-dither |
| @@ -28,6 +28,7 @@ properties: · 27 unmodified lines | |||
| 28 | - items: | 28 | - items: |
| 29 | - enum: | 29 | - enum: |
| 30 | - mediatek,mt6795-disp-gamma | 30 | - mediatek,mt6795-disp-gamma |
| 31 | - mediatek,mt8167-disp-gamma | ||
| 31 | - const: mediatek,mt8173-disp-gamma | 32 | - const: mediatek,mt8173-disp-gamma |
| 32 | - items: | 33 | - items: |
| 33 | - enum: | 34 | - enum: |
| @@ -23,6 +23,7 @@ properties: · 22 unmodified lines | |||
| 23 | oneOf: | 23 | oneOf: |
| 24 | - enum: | 24 | - enum: |
| 25 | - mediatek,mt2701-disp-ovl | 25 | - mediatek,mt2701-disp-ovl |
| 26 | - mediatek,mt8167-disp-ovl | ||
| 26 | - mediatek,mt8173-disp-ovl | 27 | - mediatek,mt8173-disp-ovl |
| 27 | - mediatek,mt8183-disp-ovl | 28 | - mediatek,mt8183-disp-ovl |
| 28 | - mediatek,mt8192-disp-ovl | 29 | - mediatek,mt8192-disp-ovl |
| @@ -36,6 +36,7 @@ properties: · 35 unmodified lines | |||
| 36 | - enum: | 36 | - enum: |
| 37 | - mediatek,mt7623-disp-rdma | 37 | - mediatek,mt7623-disp-rdma |
| 38 | - mediatek,mt2712-disp-rdma | 38 | - mediatek,mt2712-disp-rdma |
| 39 | - mediatek,mt8167-disp-rdma | ||
| 39 | - const: mediatek,mt2701-disp-rdma | 40 | - const: mediatek,mt2701-disp-rdma |
| 40 | - items: | 41 | - items: |
| 41 | - enum: | 42 | - enum: |
| @@ -24,7 +24,9 @@ properties: · 23 unmodified lines | |||
| 24 | - enum: | 24 | - enum: |
| 25 | - mediatek,mt8173-disp-wdma | 25 | - mediatek,mt8173-disp-wdma |
| 26 | - items: | 26 | - items: |
| 27 | - const: mediatek,mt6795-disp-wdma | 27 | - enum: |
| 28 | - mediatek,mt6795-disp-wdma | ||
| 29 | - mediatek,mt8167-disp-wdma | ||
| 28 | - const: mediatek,mt8173-disp-wdma | 30 | - const: mediatek,mt8173-disp-wdma |
| 29 | 31 | ||
| 30 | reg: | 32 | reg: |
| @@ -67,6 +67,7 @@ properties: · 66 unmodified lines | |||
| 67 | 67 | ||
| 68 | - items: | 68 | - items: |
| 69 | - enum: | 69 | - enum: |
| 70 | - qcom,eliza-dp | ||
| 70 | - qcom,sm8750-dp | 71 | - qcom,sm8750-dp |
| 71 | - const: qcom,sm8650-dp | 72 | - const: qcom,sm8650-dp |
| 72 | 73 | ||
| @@ -253,7 +254,6 @@ allOf: · 180 unmodified lines | |||
| 253 | enum: | 254 | enum: |
| 254 | # these platforms support 2 streams MST on some interfaces, | 255 | # these platforms support 2 streams MST on some interfaces, |
| 255 | # others are SST only | 256 | # others are SST only |
| 256 | - qcom,glymur-dp | ||
| 257 | - qcom,sc8280xp-dp | 257 | - qcom,sc8280xp-dp |
| 258 | - qcom,x1e80100-dp | 258 | - qcom,x1e80100-dp |
| 259 | then: | 259 | then: |
| @@ -310,6 +310,26 @@ allOf: · 50 unmodified lines | |||
| 310 | minItems: 6 | 310 | minItems: 6 |
| 311 | maxItems: 8 | 311 | maxItems: 8 |
| 312 | 312 | ||
| 313 | - if: | ||
| 314 | properties: | ||
| 315 | compatible: | ||
| 316 | contains: | ||
| 317 | enum: | ||
| 318 | # these platforms support 2 streams MST on some interfaces, | ||
| 319 | # others are SST only, but all controllers have 4 ports | ||
| 320 | - qcom,glymur-dp | ||
| 321 | then: | ||
| 322 | properties: | ||
| 323 | reg: | ||
| 324 | minItems: 9 | ||
| 325 | maxItems: 9 | ||
| 326 | clocks: | ||
| 327 | minItems: 5 | ||
| 328 | maxItems: 6 | ||
| 329 | clocks-names: | ||
| 330 | minItems: 5 | ||
| 331 | maxItems: 6 | ||
| 332 | |||
| 313 | unevaluatedProperties: false | 333 | unevaluatedProperties: false |
| 314 | 334 | ||
| 315 | examples: | 335 | examples: |
| @@ -49,8 +49,13 @@ properties: · 48 unmodified lines | |||
| 49 | - items: | 49 | - items: |
| 50 | - enum: | 50 | - enum: |
| 51 | - qcom,qcs8300-dsi-ctrl | 51 | - qcom,qcs8300-dsi-ctrl |
| 52 | - qcom,sc8280xp-dsi-ctrl | ||
| 52 | - const: qcom,sa8775p-dsi-ctrl | 53 | - const: qcom,sa8775p-dsi-ctrl |
| 53 | - const: qcom,mdss-dsi-ctrl | 54 | - const: qcom,mdss-dsi-ctrl |
| 55 | - items: | ||
| 56 | - const: qcom,eliza-dsi-ctrl | ||
| 57 | - const: qcom,sm8750-dsi-ctrl | ||
| 58 | - const: qcom,mdss-dsi-ctrl | ||
| 54 | - enum: | 59 | - enum: |
| 55 | - qcom,dsi-ctrl-6g-qcm2290 | 60 | - qcom,dsi-ctrl-6g-qcm2290 |
| 56 | - qcom,mdss-dsi-ctrl # This should always come with an SoC-specific compatible | 61 | - qcom,mdss-dsi-ctrl # This should always come with an SoC-specific compatible |
| @@ -91,6 +91,7 @@ allOf: · 90 unmodified lines | |||
| 91 | compatible: | 91 | compatible: |
| 92 | contains: | 92 | contains: |
| 93 | enum: | 93 | enum: |
| 94 | - qcom,adreno-gmu-615.0 | ||
| 94 | - qcom,adreno-gmu-618.0 | 95 | - qcom,adreno-gmu-618.0 |
| 95 | - qcom,adreno-gmu-630.2 | 96 | - qcom,adreno-gmu-630.2 |
| 96 | then: | 97 | then: |
| @@ -440,13 +440,6 @@ allOf: · 439 unmodified lines | |||
| 440 | clocks: false | 440 | clocks: false |
| 441 | clock-names: false | 441 | clock-names: false |
| 442 | 442 | ||
| 443 | reg-names: | ||
| 444 | minItems: 1 | ||
| 445 | items: | ||
| 446 | - const: kgsl_3d0_reg_memory | ||
| 447 | - const: cx_mem | ||
| 448 | - const: cx_dbgc | ||
| 449 | |||
| 450 | examples: | 443 | examples: |
| 451 | - | | 444 | - | |
| 452 | 445 | ||
| @@ -0,0 +1,494 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/msm/qcom,eliza-mdss.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Qualcomm Eliza SoC Display MDSS | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Krzysztof Kozlowski <krzk@kernel.org> | ||
| 11 | |||
| 12 | description: | ||
| 13 | Eliza SoC Mobile Display Subsystem (MDSS) encapsulates sub-blocks like DPU | ||
| 14 | display controller, DSI and DP interfaces etc. | ||
| 15 | |||
| 16 | $ref: /schemas/display/msm/mdss-common.yaml# | ||
| 17 | |||
| 18 | properties: | ||
| 19 | compatible: | ||
| 20 | const: qcom,eliza-mdss | ||
| 21 | |||
| 22 | clocks: | ||
| 23 | items: | ||
| 24 | - description: Display AHB | ||
| 25 | - description: Display hf AXI | ||
| 26 | - description: Display core | ||
| 27 | |||
| 28 | iommus: | ||
| 29 | maxItems: 1 | ||
| 30 | |||
| 31 | interconnects: | ||
| 32 | items: | ||
| 33 | - description: Interconnect path from mdp0 port to the data bus | ||
| 34 | - description: Interconnect path from CPU to the reg bus | ||
| 35 | |||
| 36 | interconnect-names: | ||
| 37 | items: | ||
| 38 | - const: mdp0-mem | ||
| 39 | - const: cpu-cfg | ||
| 40 | |||
| 41 | patternProperties: | ||
| 42 | "^display-controller@[0-9a-f]+$": | ||
| 43 | type: object | ||
| 44 | additionalProperties: true | ||
| 45 | properties: | ||
| 46 | compatible: | ||
| 47 | contains: | ||
| 48 | const: qcom,eliza-dpu | ||
| 49 | |||
| 50 | "^displayport-controller@[0-9a-f]+$": | ||
| 51 | type: object | ||
| 52 | additionalProperties: true | ||
| 53 | properties: | ||
| 54 | compatible: | ||
| 55 | contains: | ||
| 56 | const: qcom,eliza-dp | ||
| 57 | |||
| 58 | "^dsi@[0-9a-f]+$": | ||
| 59 | type: object | ||
| 60 | additionalProperties: true | ||
| 61 | properties: | ||
| 62 | compatible: | ||
| 63 | contains: | ||
| 64 | const: qcom,eliza-dsi-ctrl | ||
| 65 | |||
| 66 | "^phy@[0-9a-f]+$": | ||
| 67 | type: object | ||
| 68 | additionalProperties: true | ||
| 69 | properties: | ||
| 70 | compatible: | ||
| 71 | contains: | ||
| 72 | const: qcom,eliza-dsi-phy-4nm | ||
| 73 | |||
| 74 | required: | ||
| 75 | - compatible | ||
| 76 | |||
| 77 | unevaluatedProperties: false | ||
| 78 | |||
| 79 | examples: | ||
| 80 | - | | ||
| 81 | #include <dt-bindings/clock/qcom,dsi-phy-28nm.h> | ||
| 82 | #include <dt-bindings/clock/qcom,rpmh.h> | ||
| 83 | #include <dt-bindings/interconnect/qcom,icc.h> | ||
| 84 | #include <dt-bindings/interrupt-controller/arm-gic.h> | ||
| 85 | #include <dt-bindings/phy/phy-qcom-qmp.h> | ||
| 86 | #include <dt-bindings/power/qcom,rpmhpd.h> | ||
| 87 | |||
| 88 | display-subsystem@ae00000 { | ||
| 89 | compatible = "qcom,eliza-mdss"; | ||
| 90 | reg = <0x0ae00000 0x1000>; | ||
| 91 | reg-names = "mdss"; | ||
| 92 | ranges; | ||
| 93 | |||
| 94 | interrupts = <GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>; | ||
| 95 | |||
| 96 | clocks = <&disp_cc_mdss_ahb_clk>, | ||
| 97 | <&gcc_disp_hf_axi_clk>, | ||
| 98 | <&disp_cc_mdss_mdp_clk>; | ||
| 99 | |||
| 100 | resets = <&disp_cc_mdss_core_bcr>; | ||
| 101 | |||
| 102 | interconnects = <&mmss_noc_master_mdp QCOM_ICC_TAG_ALWAYS | ||
| 103 | &mc_virt_slave_ebi1 QCOM_ICC_TAG_ALWAYS>, | ||
| 104 | <&gem_noc_master_appss_proc QCOM_ICC_TAG_ACTIVE_ONLY | ||
| 105 | &config_noc_slave_display_cfg QCOM_ICC_TAG_ACTIVE_ONLY>; | ||
| 106 | interconnect-names = "mdp0-mem", | ||
| 107 | "cpu-cfg"; | ||
| 108 | |||
| 109 | power-domains = <&mdss_gdsc>; | ||
| 110 | |||
| 111 | iommus = <&apps_smmu 0x800 0x2>; | ||
| 112 | |||
| 113 | interrupt-controller; | ||
| 114 | #interrupt-cells = <1>; | ||
| 115 | |||
| 116 | #address-cells = <1>; | ||
| 117 | #size-cells = <1>; | ||
| 118 | |||
| 119 | mdss_mdp: display-controller@ae01000 { | ||
| 120 | compatible = "qcom,eliza-dpu"; | ||
| 121 | reg = <0x0ae01000 0x93000>, | ||
| 122 | <0x0aeb0000 0x2008>; | ||
| 123 | reg-names = "mdp", | ||
| 124 | "vbif"; | ||
| 125 | |||
| 126 | interrupts-extended = <&mdss 0>; | ||
| 127 | |||
| 128 | clocks = <&gcc_disp_hf_axi_clk>, | ||
| 129 | <&disp_cc_mdss_ahb_clk>, | ||
| 130 | <&disp_cc_mdss_mdp_lut_clk>, | ||
| 131 | <&disp_cc_mdss_mdp_clk>, | ||
| 132 | <&disp_cc_mdss_vsync_clk>; | ||
| 133 | clock-names = "nrt_bus", | ||
| 134 | "iface", | ||
| 135 | "lut", | ||
| 136 | "core", | ||
| 137 | "vsync"; | ||
| 138 | |||
| 139 | assigned-clocks = <&disp_cc_mdss_vsync_clk>; | ||
| 140 | assigned-clock-rates = <19200000>; | ||
| 141 | |||
| 142 | operating-points-v2 = <&mdp_opp_table>; | ||
| 143 | |||
| 144 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 145 | |||
| 146 | ports { | ||
| 147 | #address-cells = <1>; | ||
| 148 | #size-cells = <0>; | ||
| 149 | |||
| 150 | port@0 { | ||
| 151 | reg = <0>; | ||
| 152 | |||
| 153 | dpu_intf1_out: endpoint { | ||
| 154 | remote-endpoint = <&mdss_dsi0_in>; | ||
| 155 | }; | ||
| 156 | }; | ||
| 157 | |||
| 158 | port@1 { | ||
| 159 | reg = <1>; | ||
| 160 | |||
| 161 | dpu_intf2_out: endpoint { | ||
| 162 | remote-endpoint = <&mdss_dsi1_in>; | ||
| 163 | }; | ||
| 164 | }; | ||
| 165 | |||
| 166 | port@2 { | ||
| 167 | reg = <2>; | ||
| 168 | |||
| 169 | dpu_intf0_out: endpoint { | ||
| 170 | remote-endpoint = <&mdss_dp0_in>; | ||
| 171 | }; | ||
| 172 | }; | ||
| 173 | }; | ||
| 174 | |||
| 175 | mdp_opp_table: opp-table { | ||
| 176 | compatible = "operating-points-v2"; | ||
| 177 | |||
| 178 | opp-150000000 { | ||
| 179 | opp-hz = /bits/ 64 <150000000>; | ||
| 180 | required-opps = <&rpmhpd_opp_low_svs_d1>; | ||
| 181 | }; | ||
| 182 | |||
| 183 | opp-207000000 { | ||
| 184 | opp-hz = /bits/ 64 <207000000>; | ||
| 185 | required-opps = <&rpmhpd_opp_low_svs>; | ||
| 186 | }; | ||
| 187 | |||
| 188 | opp-342000000 { | ||
| 189 | opp-hz = /bits/ 64 <342000000>; | ||
| 190 | required-opps = <&rpmhpd_opp_svs>; | ||
| 191 | }; | ||
| 192 | |||
| 193 | opp-417000000 { | ||
| 194 | opp-hz = /bits/ 64 <417000000>; | ||
| 195 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 196 | }; | ||
| 197 | |||
| 198 | opp-532000000 { | ||
| 199 | opp-hz = /bits/ 64 <532000000>; | ||
| 200 | required-opps = <&rpmhpd_opp_nom>; | ||
| 201 | }; | ||
| 202 | |||
| 203 | opp-600000000 { | ||
| 204 | opp-hz = /bits/ 64 <600000000>; | ||
| 205 | required-opps = <&rpmhpd_opp_nom_l1>; | ||
| 206 | }; | ||
| 207 | |||
| 208 | opp-660000000 { | ||
| 209 | opp-hz = /bits/ 64 <660000000>; | ||
| 210 | required-opps = <&rpmhpd_opp_turbo>; | ||
| 211 | }; | ||
| 212 | }; | ||
| 213 | }; | ||
| 214 | |||
| 215 | dsi@ae94000 { | ||
| 216 | compatible = "qcom,eliza-dsi-ctrl", "qcom,sm8750-dsi-ctrl", "qcom,mdss-dsi-ctrl"; | ||
| 217 | reg = <0x0ae94000 0x400>; | ||
| 218 | reg-names = "dsi_ctrl"; | ||
| 219 | |||
| 220 | interrupts-extended = <&mdss 4>; | ||
| 221 | |||
| 222 | clocks = <&disp_cc_mdss_byte0_clk>, | ||
| 223 | <&disp_cc_mdss_byte0_intf_clk>, | ||
| 224 | <&disp_cc_mdss_pclk0_clk>, | ||
| 225 | <&disp_cc_mdss_esc0_clk>, | ||
| 226 | <&disp_cc_mdss_ahb_clk>, | ||
| 227 | <&gcc_disp_hf_axi_clk>, | ||
| 228 | <&mdss_dsi0_phy DSI_PIXEL_PLL_CLK>, | ||
| 229 | <&mdss_dsi0_phy DSI_BYTE_PLL_CLK>, | ||
| 230 | <&disp_cc_esync0_clk>, | ||
| 231 | <&disp_cc_osc_clk>, | ||
| 232 | <&disp_cc_mdss_byte0_clk_src>, | ||
| 233 | <&disp_cc_mdss_pclk0_clk_src>; | ||
| 234 | clock-names = "byte", | ||
| 235 | "byte_intf", | ||
| 236 | "pixel", | ||
| 237 | "core", | ||
| 238 | "iface", | ||
| 239 | "bus", | ||
| 240 | "dsi_pll_pixel", | ||
| 241 | "dsi_pll_byte", | ||
| 242 | "esync", | ||
| 243 | "osc", | ||
| 244 | "byte_src", | ||
| 245 | "pixel_src"; | ||
| 246 | |||
| 247 | operating-points-v2 = <&mdss_dsi_opp_table>; | ||
| 248 | |||
| 249 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 250 | |||
| 251 | phys = <&mdss_dsi0_phy>; | ||
| 252 | phy-names = "dsi"; | ||
| 253 | |||
| 254 | #address-cells = <1>; | ||
| 255 | #size-cells = <0>; | ||
| 256 | |||
| 257 | ports { | ||
| 258 | #address-cells = <1>; | ||
| 259 | #size-cells = <0>; | ||
| 260 | |||
| 261 | port@0 { | ||
| 262 | reg = <0>; | ||
| 263 | |||
| 264 | mdss_dsi0_in: endpoint { | ||
| 265 | remote-endpoint = <&dpu_intf1_out>; | ||
| 266 | }; | ||
| 267 | }; | ||
| 268 | |||
| 269 | port@1 { | ||
| 270 | reg = <1>; | ||
| 271 | |||
| 272 | mdss_dsi0_out: endpoint { | ||
| 273 | remote-endpoint = <&panel0_in>; | ||
| 274 | data-lanes = <0 1 2 3>; | ||
| 275 | }; | ||
| 276 | }; | ||
| 277 | }; | ||
| 278 | |||
| 279 | mdss_dsi_opp_table: opp-table { | ||
| 280 | compatible = "operating-points-v2"; | ||
| 281 | |||
| 282 | opp-140630000 { | ||
| 283 | opp-hz = /bits/ 64 <140630000>; | ||
| 284 | required-opps = <&rpmhpd_opp_low_svs_d1>; | ||
| 285 | }; | ||
| 286 | |||
| 287 | opp-187500000 { | ||
| 288 | opp-hz = /bits/ 64 <187500000>; | ||
| 289 | required-opps = <&rpmhpd_opp_low_svs>; | ||
| 290 | }; | ||
| 291 | |||
| 292 | opp-300000000 { | ||
| 293 | opp-hz = /bits/ 64 <300000000>; | ||
| 294 | required-opps = <&rpmhpd_opp_svs>; | ||
| 295 | }; | ||
| 296 | |||
| 297 | opp-358000000 { | ||
| 298 | opp-hz = /bits/ 64 <358000000>; | ||
| 299 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 300 | }; | ||
| 301 | }; | ||
| 302 | }; | ||
| 303 | |||
| 304 | mdss_dsi0_phy: phy@ae95000 { | ||
| 305 | compatible = "qcom,eliza-dsi-phy-4nm", "qcom,sm8650-dsi-phy-4nm"; | ||
| 306 | reg = <0x0ae95000 0x200>, | ||
| 307 | <0x0ae95200 0x280>, | ||
| 308 | <0x0ae95500 0x400>; | ||
| 309 | reg-names = "dsi_phy", | ||
| 310 | "dsi_phy_lane", | ||
| 311 | "dsi_pll"; | ||
| 312 | |||
| 313 | clocks = <&disp_cc_mdss_ahb_clk>, | ||
| 314 | <&bi_tcxo_div2>; | ||
| 315 | clock-names = "iface", | ||
| 316 | "ref"; | ||
| 317 | |||
| 318 | #clock-cells = <1>; | ||
| 319 | #phy-cells = <0>; | ||
| 320 | |||
| 321 | vdds-supply = <&vreg_l2b>; | ||
| 322 | }; | ||
| 323 | |||
| 324 | dsi@ae96000 { | ||
| 325 | compatible = "qcom,eliza-dsi-ctrl", "qcom,sm8750-dsi-ctrl", "qcom,mdss-dsi-ctrl"; | ||
| 326 | reg = <0x0ae96000 0x400>; | ||
| 327 | reg-names = "dsi_ctrl"; | ||
| 328 | |||
| 329 | interrupts-extended = <&mdss 5>; | ||
| 330 | |||
| 331 | clocks = <&disp_cc_mdss_byte1_clk>, | ||
| 332 | <&disp_cc_mdss_byte1_intf_clk>, | ||
| 333 | <&disp_cc_mdss_pclk1_clk>, | ||
| 334 | <&disp_cc_mdss_esc1_clk>, | ||
| 335 | <&disp_cc_mdss_ahb_clk>, | ||
| 336 | <&gcc_disp_hf_axi_clk>, | ||
| 337 | <&mdss_dsi1_phy DSI_PIXEL_PLL_CLK>, | ||
| 338 | <&mdss_dsi1_phy DSI_BYTE_PLL_CLK>, | ||
| 339 | <&disp_cc_esync1_clk>, | ||
| 340 | <&disp_cc_osc_clk>, | ||
| 341 | <&disp_cc_mdss_byte1_clk_src>, | ||
| 342 | <&disp_cc_mdss_pclk1_clk_src>; | ||
| 343 | clock-names = "byte", | ||
| 344 | "byte_intf", | ||
| 345 | "pixel", | ||
| 346 | "core", | ||
| 347 | "iface", | ||
| 348 | "bus", | ||
| 349 | "dsi_pll_pixel", | ||
| 350 | "dsi_pll_byte", | ||
| 351 | "esync", | ||
| 352 | "osc", | ||
| 353 | "byte_src", | ||
| 354 | "pixel_src"; | ||
| 355 | |||
| 356 | operating-points-v2 = <&mdss_dsi_opp_table>; | ||
| 357 | |||
| 358 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 359 | |||
| 360 | phys = <&mdss_dsi1_phy>; | ||
| 361 | phy-names = "dsi"; | ||
| 362 | |||
| 363 | vdda-supply = <&vreg_l4b>; | ||
| 364 | |||
| 365 | ports { | ||
| 366 | #address-cells = <1>; | ||
| 367 | #size-cells = <0>; | ||
| 368 | |||
| 369 | port@0 { | ||
| 370 | reg = <0>; | ||
| 371 | |||
| 372 | mdss_dsi1_in: endpoint { | ||
| 373 | remote-endpoint = <&dpu_intf2_out>; | ||
| 374 | }; | ||
| 375 | }; | ||
| 376 | |||
| 377 | port@1 { | ||
| 378 | reg = <1>; | ||
| 379 | |||
| 380 | mdss_dsi1_out: endpoint { | ||
| 381 | remote-endpoint = <&panel1_in>; | ||
| 382 | data-lanes = <0 1 2 3>; | ||
| 383 | }; | ||
| 384 | }; | ||
| 385 | }; | ||
| 386 | }; | ||
| 387 | |||
| 388 | mdss_dsi1_phy: phy@ae97000 { | ||
| 389 | compatible = "qcom,eliza-dsi-phy-4nm", "qcom,sm8650-dsi-phy-4nm"; | ||
| 390 | reg = <0x0ae97000 0x200>, | ||
| 391 | <0x0ae97200 0x280>, | ||
| 392 | <0x0ae97500 0x400>; | ||
| 393 | reg-names = "dsi_phy", | ||
| 394 | "dsi_phy_lane", | ||
| 395 | "dsi_pll"; | ||
| 396 | |||
| 397 | clocks = <&disp_cc_mdss_ahb_clk>, | ||
| 398 | <&rpmhcc RPMH_CXO_CLK>; | ||
| 399 | clock-names = "iface", | ||
| 400 | "ref"; | ||
| 401 | |||
| 402 | #clock-cells = <1>; | ||
| 403 | #phy-cells = <0>; | ||
| 404 | |||
| 405 | vdds-supply = <&vreg_l2b>; | ||
| 406 | }; | ||
| 407 | |||
| 408 | displayport-controller@af54000 { | ||
| 409 | compatible = "qcom,eliza-dp", "qcom,sm8650-dp"; | ||
| 410 | reg = <0xaf54000 0x104>, | ||
| 411 | <0xaf54200 0xc0>, | ||
| 412 | <0xaf55000 0x770>, | ||
| 413 | <0xaf56000 0x9c>, | ||
| 414 | <0xaf57000 0x9c>; | ||
| 415 | |||
| 416 | interrupts-extended = <&mdss 12>; | ||
| 417 | |||
| 418 | clocks = <&disp_cc_mdss_ahb_clk>, | ||
| 419 | <&disp_cc_mdss_dptx0_aux_clk>, | ||
| 420 | <&disp_cc_mdss_dptx0_link_clk>, | ||
| 421 | <&disp_cc_mdss_dptx0_link_intf_clk>, | ||
| 422 | <&disp_cc_mdss_dptx0_pixel0_clk>, | ||
| 423 | <&disp_cc_mdss_dptx0_pixel1_clk>; | ||
| 424 | clock-names = "core_iface", | ||
| 425 | "core_aux", | ||
| 426 | "ctrl_link", | ||
| 427 | "ctrl_link_iface", | ||
| 428 | "stream_pixel", | ||
| 429 | "stream_1_pixel"; | ||
| 430 | |||
| 431 | assigned-clocks = <&disp_cc_mdss_dptx0_link_clk_src>, | ||
| 432 | <&disp_cc_mdss_dptx0_pixel0_clk_src>, | ||
| 433 | <&disp_cc_mdss_dptx0_pixel1_clk_src>; | ||
| 434 | assigned-clock-parents = <&usb_dp_qmpphy QMP_USB43DP_DP_LINK_CLK>, | ||
| 435 | <&usb_dp_qmpphy QMP_USB43DP_DP_VCO_DIV_CLK>, | ||
| 436 | <&usb_dp_qmpphy QMP_USB43DP_DP_VCO_DIV_CLK>; | ||
| 437 | |||
| 438 | operating-points-v2 = <&dp_opp_table>; | ||
| 439 | |||
| 440 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 441 | |||
| 442 | phys = <&usb_dp_qmpphy QMP_USB43DP_DP_PHY>; | ||
| 443 | phy-names = "dp"; | ||
| 444 | |||
| 445 | #sound-dai-cells = <0>; | ||
| 446 | |||
| 447 | dp_opp_table: opp-table { | ||
| 448 | compatible = "operating-points-v2"; | ||
| 449 | |||
| 450 | opp-192000000 { | ||
| 451 | opp-hz = /bits/ 64 <192000000>; | ||
| 452 | required-opps = <&rpmhpd_opp_low_svs_d1>; | ||
| 453 | }; | ||
| 454 | |||
| 455 | opp-270000000 { | ||
| 456 | opp-hz = /bits/ 64 <270000000>; | ||
| 457 | required-opps = <&rpmhpd_opp_low_svs>; | ||
| 458 | }; | ||
| 459 | |||
| 460 | opp-540000000 { | ||
| 461 | opp-hz = /bits/ 64 <540000000>; | ||
| 462 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 463 | }; | ||
| 464 | |||
| 465 | opp-810000000 { | ||
| 466 | opp-hz = /bits/ 64 <810000000>; | ||
| 467 | required-opps = <&rpmhpd_opp_nom>; | ||
| 468 | }; | ||
| 469 | }; | ||
| 470 | |||
| 471 | ports { | ||
| 472 | #address-cells = <1>; | ||
| 473 | #size-cells = <0>; | ||
| 474 | |||
| 475 | port@0 { | ||
| 476 | reg = <0>; | ||
| 477 | |||
| 478 | mdss_dp0_in: endpoint { | ||
| 479 | remote-endpoint = <&dpu_intf0_out>; | ||
| 480 | }; | ||
| 481 | }; | ||
| 482 | |||
| 483 | port@1 { | ||
| 484 | reg = <1>; | ||
| 485 | |||
| 486 | mdss_dp0_out: endpoint { | ||
| 487 | data-lanes = <0 1 2 3>; | ||
| 488 | remote-endpoint = <&usb_dp_qmpphy_dp_in>; | ||
| 489 | link-frequencies = /bits/ 64 <1620000000 2700000000 5400000000 8100000000>; | ||
| 490 | }; | ||
| 491 | }; | ||
| 492 | }; | ||
| 493 | }; | ||
| 494 | }; | ||
| @@ -176,13 +176,17 @@ examples: · 175 unmodified lines | |||
| 176 | }; | 176 | }; |
| 177 | }; | 177 | }; |
| 178 | 178 | ||
| 179 | displayport-controller@ae90000 { | 179 | displayport-controller@af54000 { |
| 180 | compatible = "qcom,glymur-dp"; | 180 | compatible = "qcom,glymur-dp"; |
| 181 | reg = <0xae90000 0x200>, | 181 | reg = <0xaf54000 0x200>, |
| 182 | <0xae90200 0x200>, | 182 | <0xaf54200 0x200>, |
| 183 | <0xae90400 0x600>, | 183 | <0xaf55000 0xc00>, |
| 184 | <0xae91000 0x400>, | 184 | <0xaf56000 0x400>, |
| 185 | <0xae91400 0x400>; | 185 | <0xaf57000 0x400>, |
| 186 | <0xaf58000 0x400>, | ||
| 187 | <0xaf59000 0x400>, | ||
| 188 | <0xaf5a000 0x600>, | ||
| 189 | <0xaf5b000 0x600>; | ||
| 186 | 190 | ||
| 187 | interrupt-parent = <&mdss>; | 191 | interrupt-parent = <&mdss>; |
| 188 | interrupts = <12>; | 192 | interrupts = <12>; |
| @@ -50,6 +50,22 @@ patternProperties: · 49 unmodified lines | |||
| 50 | - qcom,sc8280xp-dp | 50 | - qcom,sc8280xp-dp |
| 51 | - qcom,sc8280xp-edp | 51 | - qcom,sc8280xp-edp |
| 52 | 52 | ||
| 53 | "^dsi@[0-9a-f]+$": | ||
| 54 | type: object | ||
| 55 | additionalProperties: true | ||
| 56 | properties: | ||
| 57 | compatible: | ||
| 58 | contains: | ||
| 59 | const: qcom,sc8280xp-dsi-ctrl | ||
| 60 | |||
| 61 | "^phy@[0-9a-f]+$": | ||
| 62 | type: object | ||
| 63 | additionalProperties: true | ||
| 64 | properties: | ||
| 65 | compatible: | ||
| 66 | contains: | ||
| 67 | const: qcom,sc8280xp-dsi-phy-5nm | ||
| 68 | |||
| 53 | unevaluatedProperties: false | 69 | unevaluatedProperties: false |
| 54 | 70 | ||
| 55 | examples: | 71 | examples: |
| @@ -129,6 +145,20 @@ examples: · 73 unmodified lines | |||
| 129 | }; | 145 | }; |
| 130 | }; | 146 | }; |
| 131 | 147 | ||
| 148 | port@1 { | ||
| 149 | reg = <1>; | ||
| 150 | dpu_intf1_out: endpoint { | ||
| 151 | remote-endpoint = <&mdss0_dsi0_in>; | ||
| 152 | }; | ||
| 153 | }; | ||
| 154 | |||
| 155 | port@2 { | ||
| 156 | reg = <2>; | ||
| 157 | dpu_intf2_out: endpoint { | ||
| 158 | remote-endpoint = <&mdss0_dsi1_in>; | ||
| 159 | }; | ||
| 160 | }; | ||
| 161 | |||
| 132 | port@4 { | 162 | port@4 { |
| 133 | reg = <4>; | 163 | reg = <4>; |
| 134 | endpoint { | 164 | endpoint { |
| @@ -15,6 +15,7 @@ properties: · 14 unmodified lines | |||
| 15 | compatible: | 15 | compatible: |
| 16 | oneOf: | 16 | oneOf: |
| 17 | - enum: | 17 | - enum: |
| 18 | - qcom,eliza-dpu | ||
| 18 | - qcom,glymur-dpu | 19 | - qcom,glymur-dpu |
| 19 | - qcom,kaanapali-dpu | 20 | - qcom,kaanapali-dpu |
| 20 | - qcom,sa8775p-dpu | 21 | - qcom,sa8775p-dpu |
| @@ -10,7 +10,7 @@ maintainers: · 9 unmodified lines | |||
| 10 | - Krzysztof Kozlowski <krzk@kernel.org> | 10 | - Krzysztof Kozlowski <krzk@kernel.org> |
| 11 | 11 | ||
| 12 | description: | 12 | description: |
| 13 | SM8650 MSM Mobile Display Subsystem(MDSS), which encapsulates sub-blocks like | 13 | SM8750 MSM Mobile Display Subsystem(MDSS), which encapsulates sub-blocks like |
| 14 | DPU display controller, DSI and DP interfaces etc. | 14 | DPU display controller, DSI and DP interfaces etc. |
| 15 | 15 | ||
| 16 | $ref: /schemas/display/msm/mdss-common.yaml# | 16 | $ref: /schemas/display/msm/mdss-common.yaml# |
| @@ -28,7 +28,6 @@ properties: · 27 unmodified lines | |||
| 28 | 28 | ||
| 29 | port: true | 29 | port: true |
| 30 | reset-gpios: true | 30 | reset-gpios: true |
| 31 | |||
| 32 | backlight: true | 31 | backlight: true |
| 33 | 32 | ||
| 34 | required: | 33 | required: |
| @@ -20,6 +20,8 @@ properties: · 19 unmodified lines | |||
| 20 | - boe,nv110wum-l60 | 20 | - boe,nv110wum-l60 |
| 21 | # CSOT pna957qt1-1 10.95" WUXGA TFT LCD panel | 21 | # CSOT pna957qt1-1 10.95" WUXGA TFT LCD panel |
| 22 | - csot,pna957qt1-1 | 22 | - csot,pna957qt1-1 |
| 23 | # Holitech HTF065H045 6.517" 720x1600 TFT LCD panel | ||
| 24 | - holitech,htf065h045 | ||
| 23 | # IVO t109nw41 11.0" WUXGA TFT LCD panel | 25 | # IVO t109nw41 11.0" WUXGA TFT LCD panel |
| 24 | - ivo,t109nw41 | 26 | - ivo,t109nw41 |
| 25 | # KINGDISPLAY KD110N11-51IE 10.95" WUXGA TFT LCD panel | 27 | # KINGDISPLAY KD110N11-51IE 10.95" WUXGA TFT LCD panel |
| @@ -0,0 +1,91 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/himax,hx83121a.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Himax HX83121A based DSI display Panels | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Pengyu Luo <mitltlatltl@gmail.com> | ||
| 11 | |||
| 12 | description: | ||
| 13 | The Himax HX83121A is a generic DSI Panel IC used to drive dsi | ||
| 14 | panels. Support video mode panels from China Star Optoelectronics | ||
| 15 | Technology (CSOT) and BOE Technology. | ||
| 16 | |||
| 17 | allOf: | ||
| 18 | - $ref: panel-common-dual.yaml# | ||
| 19 | |||
| 20 | properties: | ||
| 21 | compatible: | ||
| 22 | items: | ||
| 23 | - enum: | ||
| 24 | - boe,ppc357db1-4 | ||
| 25 | - csot,ppc357db1-4 | ||
| 26 | - const: himax,hx83121a | ||
| 27 | |||
| 28 | reg: | ||
| 29 | maxItems: 1 | ||
| 30 | |||
| 31 | reset-gpios: | ||
| 32 | maxItems: 1 | ||
| 33 | |||
| 34 | avdd-supply: | ||
| 35 | description: analog positive supply for IC | ||
| 36 | |||
| 37 | avee-supply: | ||
| 38 | description: analog negative supply for IC | ||
| 39 | |||
| 40 | vddi-supply: | ||
| 41 | description: power supply for IC | ||
| 42 | |||
| 43 | backlight: true | ||
| 44 | ports: true | ||
| 45 | |||
| 46 | required: | ||
| 47 | - compatible | ||
| 48 | - reg | ||
| 49 | - vddi-supply | ||
| 50 | - reset-gpios | ||
| 51 | - ports | ||
| 52 | |||
| 53 | additionalProperties: false | ||
| 54 | |||
| 55 | examples: | ||
| 56 | - | | ||
| 57 | #include <dt-bindings/gpio/gpio.h> | ||
| 58 | |||
| 59 | dsi { | ||
| 60 | #address-cells = <1>; | ||
| 61 | #size-cells = <0>; | ||
| 62 | |||
| 63 | panel@0 { | ||
| 64 | compatible = "csot,ppc357db1-4", "himax,hx83121a"; | ||
| 65 | reg = <0>; | ||
| 66 | |||
| 67 | vddi-supply = <&vreg_l2b>; | ||
| 68 | reset-gpios = <&tlmm 38 GPIO_ACTIVE_LOW>; | ||
| 69 | |||
| 70 | ports { | ||
| 71 | #address-cells = <1>; | ||
| 72 | #size-cells = <0>; | ||
| 73 | |||
| 74 | port@0 { | ||
| 75 | reg = <0>; | ||
| 76 | panel_in_0: endpoint { | ||
| 77 | remote-endpoint = <&dsi0_out>; | ||
| 78 | }; | ||
| 79 | }; | ||
| 80 | |||
| 81 | port@1{ | ||
| 82 | reg = <1>; | ||
| 83 | panel_in_1: endpoint { | ||
| 84 | remote-endpoint = <&dsi1_out>; | ||
| 85 | }; | ||
| 86 | }; | ||
| 87 | }; | ||
| 88 | }; | ||
| 89 | }; | ||
| 90 | |||
| 91 | ... | ||
| @@ -33,11 +33,8 @@ properties: · 32 unmodified lines | |||
| 33 | maxItems: 1 | 33 | maxItems: 1 |
| 34 | 34 | ||
| 35 | reset-gpios: true | 35 | reset-gpios: true |
| 36 | |||
| 37 | backlight: true | 36 | backlight: true |
| 38 | |||
| 39 | rotation: true | 37 | rotation: true |
| 40 | |||
| 41 | port: true | 38 | port: true |
| 42 | 39 | ||
| 43 | vcc-supply: | 40 | vcc-supply: |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | |||
| 4 | $id: http://devicetree.org/schemas/display/panel/ilitek,ili9806e.yaml# | 4 | $id: http://devicetree.org/schemas/display/panel/ilitek,ili9806e.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Ilitek ILI9806E based MIPI-DSI panels | 7 | title: Ilitek ILI9806E based panels |
| 8 | 8 | ||
| 9 | maintainers: | 9 | maintainers: |
| 10 | - Michael Walle <mwalle@kernel.org> | 10 | - Michael Walle <mwalle@kernel.org> |
| @@ -18,6 +18,7 @@ properties: · 7 unmodified lines | |||
| 18 | - enum: | 18 | - enum: |
| 19 | - densitron,dmt028vghmcmi-1d | 19 | - densitron,dmt028vghmcmi-1d |
| 20 | - ortustech,com35h3p70ulc | 20 | - ortustech,com35h3p70ulc |
| 21 | - rocktech,rk050hr345-ct106a | ||
| 21 | - const: ilitek,ili9806e | 22 | - const: ilitek,ili9806e |
| 22 | 23 | ||
| 23 | reg: | 24 | reg: |
| @@ -30,11 +31,24 @@ required: · 6 unmodified lines | |||
| 30 | - compatible | 31 | - compatible |
| 31 | - reg | 32 | - reg |
| 32 | - vdd-supply | 33 | - vdd-supply |
| 33 | - vccio-supply | ||
| 34 | - reset-gpios | 34 | - reset-gpios |
| 35 | - backlight | 35 | - backlight |
| 36 | - port | 36 | - port |
| 37 | 37 | ||
| 38 | if: | ||
| 39 | properties: | ||
| 40 | compatible: | ||
| 41 | contains: | ||
| 42 | enum: | ||
| 43 | - rocktech,rk050hr345-ct106a | ||
| 44 | then: | ||
| 45 | $ref: /schemas/spi/spi-peripheral-props.yaml# | ||
| 46 | required: | ||
| 47 | - spi-max-frequency | ||
| 48 | else: | ||
| 49 | required: | ||
| 50 | - vccio-supply | ||
| 51 | |||
| 38 | unevaluatedProperties: false | 52 | unevaluatedProperties: false |
| 39 | 53 | ||
| 40 | examples: | 54 | examples: |
| @@ -60,5 +74,25 @@ examples: · 19 unmodified lines | |||
| 60 | }; | 74 | }; |
| 61 | }; | 75 | }; |
| 62 | }; | 76 | }; |
| 77 | - | | ||
| 78 | #include <dt-bindings/gpio/gpio.h> | ||
| 79 | |||
| 80 | spi { | ||
| 81 | #address-cells = <1>; | ||
| 82 | #size-cells = <0>; | ||
| 63 | 83 | ||
| 84 | panel@0 { | ||
| 85 | compatible = "rocktech,rk050hr345-ct106a", "ilitek,ili9806e"; | ||
| 86 | reg = <0>; | ||
| 87 | vdd-supply = <®_vdd_panel>; | ||
| 88 | spi-max-frequency = <10000000>; | ||
| 89 | reset-gpios = <&gpiob 6 GPIO_ACTIVE_LOW>; | ||
| 90 | backlight = <&backlight>; | ||
| 91 | port { | ||
| 92 | panel_in_rgb: endpoint { | ||
| 93 | remote-endpoint = <<dc_out_rgb>; | ||
| 94 | }; | ||
| 95 | }; | ||
| 96 | }; | ||
| 97 | }; | ||
| 64 | ... | 98 | ... |
| @@ -25,6 +25,7 @@ properties: · 24 unmodified lines | |||
| 25 | backlight: true | 25 | backlight: true |
| 26 | port: true | 26 | port: true |
| 27 | reset-gpios: true | 27 | reset-gpios: true |
| 28 | |||
| 28 | iovcc-supply: | 29 | iovcc-supply: |
| 29 | description: regulator that supplies the iovcc voltage | 30 | description: regulator that supplies the iovcc voltage |
| 30 | vci-supply: | 31 | vci-supply: |
| @@ -24,6 +24,7 @@ properties: · 23 unmodified lines | |||
| 24 | backlight: true | 24 | backlight: true |
| 25 | port: true | 25 | port: true |
| 26 | reset-gpios: true | 26 | reset-gpios: true |
| 27 | |||
| 27 | iovcc-supply: | 28 | iovcc-supply: |
| 28 | description: regulator that supplies the iovcc voltage | 29 | description: regulator that supplies the iovcc voltage |
| 29 | vcc-supply: | 30 | vcc-supply: |
| @@ -22,7 +22,6 @@ properties: · 21 unmodified lines | |||
| 22 | - mantix,mlaf057we51-x | 22 | - mantix,mlaf057we51-x |
| 23 | - ys,ys57pss36bh5gq | 23 | - ys,ys57pss36bh5gq |
| 24 | 24 | ||
| 25 | port: true | ||
| 26 | reg: | 25 | reg: |
| 27 | maxItems: 1 | 26 | maxItems: 1 |
| 28 | description: DSI virtual channel | 27 | description: DSI virtual channel |
| @@ -36,13 +35,13 @@ properties: · 7 unmodified lines | |||
| 36 | vddi-supply: | 35 | vddi-supply: |
| 37 | description: 1.8V I/O voltage supply | 36 | description: 1.8V I/O voltage supply |
| 38 | 37 | ||
| 39 | reset-gpios: true | ||
| 40 | |||
| 41 | mantix,tp-rstn-gpios: | 38 | mantix,tp-rstn-gpios: |
| 42 | maxItems: 1 | 39 | maxItems: 1 |
| 43 | description: second reset line that triggers DSI config load | 40 | description: second reset line that triggers DSI config load |
| 44 | 41 | ||
| 45 | backlight: true | 42 | backlight: true |
| 43 | port: true | ||
| 44 | reset-gpios: true | ||
| 46 | 45 | ||
| 47 | required: | 46 | required: |
| 48 | - compatible | 47 | - compatible |
| @@ -28,13 +28,14 @@ properties: · 27 unmodified lines | |||
| 28 | reg: | 28 | reg: |
| 29 | maxItems: 1 | 29 | maxItems: 1 |
| 30 | 30 | ||
| 31 | reset-gpios: true | ||
| 32 | vdd-supply: | 31 | vdd-supply: |
| 33 | description: regulator that supplies the vdd voltage | 32 | description: regulator that supplies the vdd voltage |
| 34 | vddi-supply: | 33 | vddi-supply: |
| 35 | description: regulator that supplies the vddi voltage | 34 | description: regulator that supplies the vddi voltage |
| 35 | |||
| 36 | backlight: true | 36 | backlight: true |
| 37 | port: true | 37 | port: true |
| 38 | reset-gpios: true | ||
| 38 | 39 | ||
| 39 | required: | 40 | required: |
| 40 | - compatible | 41 | - compatible |
| @@ -58,6 +58,10 @@ properties: · 57 unmodified lines | |||
| 58 | - hydis,hv070wx2-1e0 | 58 | - hydis,hv070wx2-1e0 |
| 59 | # Jenson Display BL-JT60050-01A 7" WSVGA (1024x600) color TFT LCD LVDS panel | 59 | # Jenson Display BL-JT60050-01A 7" WSVGA (1024x600) color TFT LCD LVDS panel |
| 60 | - jenson,bl-jt60050-01a | 60 | - jenson,bl-jt60050-01a |
| 61 | # Samsung LTN070NL01 7.0" WSVGA (1024x600) TFT LCD LVDS panel | ||
| 62 | - samsung,ltn070nl01 | ||
| 63 | # Samsung LTN101AL03 10.1" WXGA (800x1280) TFT LCD LVDS panel | ||
| 64 | - samsung,ltn101al03 | ||
| 61 | - tbs,a711-panel | 65 | - tbs,a711-panel |
| 62 | # Winstar WF70A8SYJHLNGA 7" WSVGA (1024x600) color TFT LCD LVDS panel | 66 | # Winstar WF70A8SYJHLNGA 7" WSVGA (1024x600) color TFT LCD LVDS panel |
| 63 | - winstar,wf70a8syjhlnga | 67 | - winstar,wf70a8syjhlnga |
| @@ -188,6 +188,8 @@ properties: · 187 unmodified lines | |||
| 188 | - innolux,n156bge-l21 | 188 | - innolux,n156bge-l21 |
| 189 | # Innolux Corporation 7.0" WSVGA (1024x600) TFT LCD panel | 189 | # Innolux Corporation 7.0" WSVGA (1024x600) TFT LCD panel |
| 190 | - innolux,zj070na-01p | 190 | - innolux,zj070na-01p |
| 191 | # JuTouch Technology Co.. 7" JT070TM041 WSVGA (1024 x 600) LVDS panel | ||
| 192 | - jutouch,jt070tm041 | ||
| 191 | # JuTouch Technology Co.. 10" JT101TM023 WXGA (1280 x 800) LVDS panel | 193 | # JuTouch Technology Co.. 10" JT101TM023 WXGA (1280 x 800) LVDS panel |
| 192 | - jutouch,jt101tm023 | 194 | - jutouch,jt101tm023 |
| 193 | # Kaohsiung Opto-Electronics Inc. 5.7" QVGA (320 x 240) TFT LCD panel | 195 | # Kaohsiung Opto-Electronics Inc. 5.7" QVGA (320 x 240) TFT LCD panel |
| @@ -310,6 +312,8 @@ properties: · 116 unmodified lines | |||
| 310 | - team-source-display,tst043015cmhx | 312 | - team-source-display,tst043015cmhx |
| 311 | # Tianma Micro-electronics P0700WXF1MBAA 7.0" WXGA (1280x800) LVDS TFT LCD panel | 313 | # Tianma Micro-electronics P0700WXF1MBAA 7.0" WXGA (1280x800) LVDS TFT LCD panel |
| 312 | - tianma,p0700wxf1mbaa | 314 | - tianma,p0700wxf1mbaa |
| 315 | # Tianma Micro-electronics TM050RDH03 5.0" WVGA TFT LCD panel | ||
| 316 | - tianma,tm050rdh03 | ||
| 313 | # Tianma Micro-electronics TM070JDHG30 7.0" WXGA TFT LCD panel | 317 | # Tianma Micro-electronics TM070JDHG30 7.0" WXGA TFT LCD panel |
| 314 | - tianma,tm070jdhg30 | 318 | - tianma,tm070jdhg30 |
| 315 | # Tianma Micro-electronics TM070JDHG34-00 7.0" WXGA (1280x800) LVDS TFT LCD panel | 319 | # Tianma Micro-electronics TM070JDHG34-00 7.0" WXGA (1280x800) LVDS TFT LCD panel |
| @@ -33,8 +33,6 @@ properties: · 32 unmodified lines | |||
| 33 | iovcc-supply: | 33 | iovcc-supply: |
| 34 | description: Regulator for 1.8V IO power supply. | 34 | description: Regulator for 1.8V IO power supply. |
| 35 | 35 | ||
| 36 | backlight: true | ||
| 37 | |||
| 38 | renesas,gamma: | 36 | renesas,gamma: |
| 39 | $ref: /schemas/types.yaml#/definitions/uint32 | 37 | $ref: /schemas/types.yaml#/definitions/uint32 |
| 40 | description: | 38 | description: |
| @@ -51,6 +49,7 @@ properties: · 10 unmodified lines | |||
| 51 | type: boolean | 49 | type: boolean |
| 52 | description: digital contrast adjustment | 50 | description: digital contrast adjustment |
| 53 | 51 | ||
| 52 | backlight: true | ||
| 54 | reset-gpios: true | 53 | reset-gpios: true |
| 55 | port: true | 54 | port: true |
| 56 | 55 | ||
| @@ -33,7 +33,6 @@ properties: · 32 unmodified lines | |||
| 33 | description: Regulator for 1.8V IO power supply. | 33 | description: Regulator for 1.8V IO power supply. |
| 34 | 34 | ||
| 35 | backlight: true | 35 | backlight: true |
| 36 | |||
| 37 | reset-gpios: true | 36 | reset-gpios: true |
| 38 | port: true | 37 | port: true |
| 39 | 38 | ||
| @@ -33,7 +33,6 @@ properties: · 32 unmodified lines | |||
| 33 | # Xingbangda XBD599 5.99" 720x1440 TFT LCD panel | 33 | # Xingbangda XBD599 5.99" 720x1440 TFT LCD panel |
| 34 | - xingbangda,xbd599 | 34 | - xingbangda,xbd599 |
| 35 | 35 | ||
| 36 | port: true | ||
| 37 | reg: | 36 | reg: |
| 38 | maxItems: 1 | 37 | maxItems: 1 |
| 39 | description: DSI virtual channel | 38 | description: DSI virtual channel |
| @@ -44,9 +43,9 @@ properties: · 4 unmodified lines | |||
| 44 | iovcc-supply: | 43 | iovcc-supply: |
| 45 | description: I/O voltage supply | 44 | description: I/O voltage supply |
| 46 | 45 | ||
| 47 | reset-gpios: true | ||
| 48 | |||
| 49 | backlight: true | 46 | backlight: true |
| 47 | port: true | ||
| 48 | reset-gpios: true | ||
| 50 | rotation: true | 49 | rotation: true |
| 51 | 50 | ||
| 52 | required: | 51 | required: |
| @@ -8,13 +8,16 @@ title: Samsung AMS561RA01 panel with S6E8AA5X01 controller · 7 unmodified lines | |||
| 8 | 8 | ||
| 9 | maintainers: | 9 | maintainers: |
| 10 | - Kaustabh Chakraborty <kauschluss@disroot.org> | 10 | - Kaustabh Chakraborty <kauschluss@disroot.org> |
| 11 | - Yedaya Katsman <yedaya.ka@gmail.com> | ||
| 11 | 12 | ||
| 12 | allOf: | 13 | allOf: |
| 13 | - $ref: panel-common.yaml# | 14 | - $ref: panel-common.yaml# |
| 14 | 15 | ||
| 15 | properties: | 16 | properties: |
| 16 | compatible: | 17 | compatible: |
| 17 | const: samsung,s6e8aa5x01-ams561ra01 | 18 | enum: |
| 19 | - samsung,s6e8aa5x01-ams561ra01 | ||
| 20 | - samsung,s6e8fc0-m1906f9 | ||
| 18 | 21 | ||
| 19 | reg: | 22 | reg: |
| 20 | maxItems: 1 | 23 | maxItems: 1 |
| @@ -31,9 +31,7 @@ properties: · 30 unmodified lines | |||
| 31 | description: Negative 5V supply | 31 | description: Negative 5V supply |
| 32 | 32 | ||
| 33 | reset-gpios: true | 33 | reset-gpios: true |
| 34 | |||
| 35 | enable-gpios: true | 34 | enable-gpios: true |
| 36 | |||
| 37 | port: true | 35 | port: true |
| 38 | 36 | ||
| 39 | required: | 37 | required: |
| @@ -16,8 +16,6 @@ properties: · 15 unmodified lines | |||
| 16 | compatible: | 16 | compatible: |
| 17 | const: startek,kd070fhfid015 | 17 | const: startek,kd070fhfid015 |
| 18 | 18 | ||
| 19 | enable-gpios: true | ||
| 20 | |||
| 21 | iovcc-supply: | 19 | iovcc-supply: |
| 22 | description: Reference to the regulator powering the panel IO pins. | 20 | description: Reference to the regulator powering the panel IO pins. |
| 23 | 21 | ||
| @@ -25,11 +23,10 @@ properties: · 1 unmodified line | |||
| 25 | maxItems: 1 | 23 | maxItems: 1 |
| 26 | description: DSI virtual channel | 24 | description: DSI virtual channel |
| 27 | 25 | ||
| 28 | reset-gpios: true | 26 | enable-gpios: true |
| 29 | |||
| 30 | port: true | 27 | port: true |
| 31 | |||
| 32 | power-supply: true | 28 | power-supply: true |
| 29 | reset-gpios: true | ||
| 33 | 30 | ||
| 34 | required: | 31 | required: |
| 35 | - compatible | 32 | - compatible |
| @@ -37,7 +37,7 @@ properties: · 36 unmodified lines | |||
| 37 | const: 2 | 37 | const: 2 |
| 38 | 38 | ||
| 39 | "#interrupt-cells": | 39 | "#interrupt-cells": |
| 40 | const: 1 | 40 | const: 2 |
| 41 | 41 | ||
| 42 | ngpios: | 42 | ngpios: |
| 43 | description: | 43 | description: |
| @@ -86,7 +86,7 @@ examples: · 42 unmodified lines | |||
| 86 | gpio-controller; | 86 | gpio-controller; |
| 87 | #gpio-cells = <2>; | 87 | #gpio-cells = <2>; |
| 88 | interrupt-controller; | 88 | interrupt-controller; |
| 89 | #interrupt-cells = <1>; | 89 | #interrupt-cells = <2>; |
| 90 | interrupts = <53>, <53>, <53>, <53>, | 90 | interrupts = <53>, <53>, <53>, <53>, |
| 91 | <53>, <53>, <53>, <53>, | 91 | <53>, <53>, <53>, <53>, |
| 92 | <53>, <53>, <53>, <53>, | 92 | <53>, <53>, <53>, <53>, |
| @@ -7,7 +7,7 @@ $schema: http://devicetree.org/meta-schemas/core.yaml# · 6 unmodified lines | |||
| 7 | title: Synopsys DesignWare APB I2C Controller | 7 | title: Synopsys DesignWare APB I2C Controller |
| 8 | 8 | ||
| 9 | maintainers: | 9 | maintainers: |
| 10 | - Jarkko Nikula <jarkko.nikula@linux.intel.com> | 10 | - Mika Westerberg <mika.westerberg@linux.intel.com> |
| 11 | 11 | ||
| 12 | allOf: | 12 | allOf: |
| 13 | - $ref: /schemas/i2c/i2c-controller.yaml# | 13 | - $ref: /schemas/i2c/i2c-controller.yaml# |
| @@ -19,9 +19,6 @@ description: · 18 unmodified lines | |||
| 19 | Flash sub nodes describe the memory range and optional per-flash | 19 | Flash sub nodes describe the memory range and optional per-flash |
| 20 | properties. | 20 | properties. |
| 21 | 21 | ||
| 22 | allOf: | ||
| 23 | - $ref: mtd.yaml# | ||
| 24 | |||
| 25 | properties: | 22 | properties: |
| 26 | compatible: | 23 | compatible: |
| 27 | const: st,spear600-smi | 24 | const: st,spear600-smi |
| @@ -42,14 +39,29 @@ properties: · 14 unmodified lines | |||
| 42 | $ref: /schemas/types.yaml#/definitions/uint32 | 39 | $ref: /schemas/types.yaml#/definitions/uint32 |
| 43 | description: Functional clock rate of the SMI controller in Hz. | 40 | description: Functional clock rate of the SMI controller in Hz. |
| 44 | 41 | ||
| 45 | st,smi-fast-mode: | 42 | patternProperties: |
| 46 | type: boolean | 43 | "^flash@.*$": |
| 47 | description: Indicates that the attached flash supports fast read mode. | 44 | $ref: /schemas/mtd/mtd.yaml# |
| 45 | |||
| 46 | properties: | ||
| 47 | reg: | ||
| 48 | maxItems: 1 | ||
| 49 | |||
| 50 | st,smi-fast-mode: | ||
| 51 | type: boolean | ||
| 52 | description: Indicates that the attached flash supports fast read mode. | ||
| 53 | |||
| 54 | unevaluatedProperties: false | ||
| 55 | |||
| 56 | required: | ||
| 57 | - reg | ||
| 48 | 58 | ||
| 49 | required: | 59 | required: |
| 50 | - compatible | 60 | - compatible |
| 51 | - reg | 61 | - reg |
| 52 | - clock-rate | 62 | - clock-rate |
| 63 | - "#address-cells" | ||
| 64 | - "#size-cells" | ||
| 53 | 65 | ||
| 54 | unevaluatedProperties: false | 66 | unevaluatedProperties: false |
| 55 | 67 | ||
| @@ -64,7 +76,7 @@ examples: · 8 unmodified lines | |||
| 64 | interrupts = <12>; | 76 | interrupts = <12>; |
| 65 | clock-rate = <50000000>; /* 50 MHz */ | 77 | clock-rate = <50000000>; /* 50 MHz */ |
| 66 | 78 | ||
| 67 | flash@f8000000 { | 79 | flash@fc000000 { |
| 68 | reg = <0xfc000000 0x1000>; | 80 | reg = <0xfc000000 0x1000>; |
| 69 | st,smi-fast-mode; | 81 | st,smi-fast-mode; |
| 70 | }; | 82 | }; |
| @@ -23,6 +23,7 @@ properties: · 22 unmodified lines | |||
| 23 | - items: | 23 | - items: |
| 24 | - enum: | 24 | - enum: |
| 25 | - mediatek,mt7623-mipi-tx | 25 | - mediatek,mt7623-mipi-tx |
| 26 | - mediatek,mt8167-mipi-tx | ||
| 26 | - const: mediatek,mt2701-mipi-tx | 27 | - const: mediatek,mt2701-mipi-tx |
| 27 | - items: | 28 | - items: |
| 28 | - enum: | 29 | - enum: |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | %YAML 1.2 | 2 | %YAML 1.2 |
| 3 | --- | 3 | --- |
| 4 | $id: http://devicetree.org/schemas/display/msm/dsi-phy-10nm.yaml# | 4 | $id: http://devicetree.org/schemas/phy/qcom,dsi-phy-10nm.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Qualcomm Display DSI 10nm PHY | 7 | title: Qualcomm Display DSI 10nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | |||
| 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | ||
| 12 | allOf: | 12 | allOf: |
| 13 | - $ref: dsi-phy-common.yaml# | 13 | - $ref: qcom,dsi-phy-common.yaml# |
| 14 | 14 | ||
| 15 | properties: | 15 | properties: |
| 16 | compatible: | 16 | compatible: |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | %YAML 1.2 | 2 | %YAML 1.2 |
| 3 | --- | 3 | --- |
| 4 | $id: http://devicetree.org/schemas/display/msm/dsi-phy-14nm.yaml# | 4 | $id: http://devicetree.org/schemas/phy/qcom,dsi-phy-14nm.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Qualcomm Display DSI 14nm PHY | 7 | title: Qualcomm Display DSI 14nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | |||
| 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | ||
| 12 | allOf: | 12 | allOf: |
| 13 | - $ref: dsi-phy-common.yaml# | 13 | - $ref: qcom,dsi-phy-common.yaml# |
| 14 | 14 | ||
| 15 | properties: | 15 | properties: |
| 16 | compatible: | 16 | compatible: |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | %YAML 1.2 | 2 | %YAML 1.2 |
| 3 | --- | 3 | --- |
| 4 | $id: http://devicetree.org/schemas/display/msm/dsi-phy-20nm.yaml# | 4 | $id: http://devicetree.org/schemas/phy/qcom,dsi-phy-20nm.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Qualcomm Display DSI 20nm PHY | 7 | title: Qualcomm Display DSI 20nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | |||
| 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | ||
| 12 | allOf: | 12 | allOf: |
| 13 | - $ref: dsi-phy-common.yaml# | 13 | - $ref: qcom,dsi-phy-common.yaml# |
| 14 | 14 | ||
| 15 | properties: | 15 | properties: |
| 16 | compatible: | 16 | compatible: |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | %YAML 1.2 | 2 | %YAML 1.2 |
| 3 | --- | 3 | --- |
| 4 | $id: http://devicetree.org/schemas/display/msm/dsi-phy-28nm.yaml# | 4 | $id: http://devicetree.org/schemas/phy/qcom,dsi-phy-28nm.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Qualcomm Display DSI 28nm PHY | 7 | title: Qualcomm Display DSI 28nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | |||
| 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> | 10 | - Krishna Manikandan <quic_mkrishn@quicinc.com> |
| 11 | 11 | ||
| 12 | allOf: | 12 | allOf: |
| 13 | - $ref: dsi-phy-common.yaml# | 13 | - $ref: qcom,dsi-phy-common.yaml# |
| 14 | 14 | ||
| 15 | properties: | 15 | properties: |
| 16 | compatible: | 16 | compatible: |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | %YAML 1.2 | 2 | %YAML 1.2 |
| 3 | --- | 3 | --- |
| 4 | $id: http://devicetree.org/schemas/display/msm/dsi-phy-7nm.yaml# | 4 | $id: http://devicetree.org/schemas/phy/qcom,dsi-phy-7nm.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Qualcomm Display DSI 7nm PHY | 7 | title: Qualcomm Display DSI 7nm PHY |
| @@ -10,7 +10,7 @@ maintainers: · 2 unmodified lines | |||
| 10 | - Jonathan Marek <jonathan@marek.ca> | 10 | - Jonathan Marek <jonathan@marek.ca> |
| 11 | 11 | ||
| 12 | allOf: | 12 | allOf: |
| 13 | - $ref: dsi-phy-common.yaml# | 13 | - $ref: qcom,dsi-phy-common.yaml# |
| 14 | 14 | ||
| 15 | properties: | 15 | properties: |
| 16 | compatible: | 16 | compatible: |
| @@ -29,9 +29,14 @@ properties: · 12 unmodified lines | |||
| 29 | - qcom,sm8550-dsi-phy-4nm | 29 | - qcom,sm8550-dsi-phy-4nm |
| 30 | - qcom,sm8650-dsi-phy-4nm | 30 | - qcom,sm8650-dsi-phy-4nm |
| 31 | - qcom,sm8750-dsi-phy-3nm | 31 | - qcom,sm8750-dsi-phy-3nm |
| 32 | - items: | ||
| 33 | - enum: | ||
| 34 | - qcom,eliza-dsi-phy-4nm | ||
| 35 | - const: qcom,sm8650-dsi-phy-4nm | ||
| 32 | - items: | 36 | - items: |
| 33 | - enum: | 37 | - enum: |
| 34 | - qcom,qcs8300-dsi-phy-5nm | 38 | - qcom,qcs8300-dsi-phy-5nm |
| 39 | - qcom,sc8280xp-dsi-phy-5nm | ||
| 35 | - const: qcom,sa8775p-dsi-phy-5nm | 40 | - const: qcom,sa8775p-dsi-phy-5nm |
| 36 | 41 | ||
| 37 | reg: | 42 | reg: |
| @@ -1,7 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause |
| 2 | %YAML 1.2 | 2 | %YAML 1.2 |
| 3 | --- | 3 | --- |
| 4 | $id: http://devicetree.org/schemas/display/msm/dsi-phy-common.yaml# | 4 | $id: http://devicetree.org/schemas/phy/qcom,dsi-phy-common.yaml# |
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# |
| 6 | 6 | ||
| 7 | title: Qualcomm Display DSI PHY Common Properties | 7 | title: Qualcomm Display DSI PHY Common Properties |
| @@ -0,0 +1,93 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/powerpc/fsl/fsl,mpc83xx.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Freescale PowerQUICC II Pro (MPC83xx) platforms | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - J. Neuschäfer <j.ne@posteo.net> | ||
| 11 | |||
| 12 | properties: | ||
| 13 | $nodename: | ||
| 14 | const: '/' | ||
| 15 | compatible: | ||
| 16 | oneOf: | ||
| 17 | - description: MPC83xx Reference Design Boards | ||
| 18 | items: | ||
| 19 | - enum: | ||
| 20 | - fsl,mpc8308rdb | ||
| 21 | - fsl,mpc8315erdb | ||
| 22 | - fsl,mpc8360rdk | ||
| 23 | - fsl,mpc8377rdb | ||
| 24 | - fsl,mpc8377wlan | ||
| 25 | - fsl,mpc8378rdb | ||
| 26 | - fsl,mpc8379rdb | ||
| 27 | |||
| 28 | - description: MPC8313E Reference Design Board | ||
| 29 | items: | ||
| 30 | - const: MPC8313ERDB | ||
| 31 | - const: MPC831xRDB | ||
| 32 | - const: MPC83xxRDB | ||
| 33 | |||
| 34 | - description: MPC8323E Reference Design Board | ||
| 35 | items: | ||
| 36 | - const: MPC8323ERDB | ||
| 37 | - const: MPC832xRDB | ||
| 38 | - const: MPC83xxRDB | ||
| 39 | |||
| 40 | - description: MPC8349E-mITX(-GP) Reference Design Platform | ||
| 41 | items: | ||
| 42 | - enum: | ||
| 43 | - MPC8349EMITX | ||
| 44 | - MPC8349EMITXGP | ||
| 45 | - const: MPC834xMITX | ||
| 46 | - const: MPC83xxMITX | ||
| 47 | |||
| 48 | - description: Keymile KMETER1 board | ||
| 49 | const: keymile,KMETER1 | ||
| 50 | |||
| 51 | - description: MPC8308 P1M board | ||
| 52 | const: denx,mpc8308_p1m | ||
| 53 | |||
| 54 | patternProperties: | ||
| 55 | "^soc@.*$": | ||
| 56 | type: object | ||
| 57 | properties: | ||
| 58 | compatible: | ||
| 59 | oneOf: | ||
| 60 | - items: | ||
| 61 | - enum: | ||
| 62 | - fsl,mpc8315-immr | ||
| 63 | - fsl,mpc8308-immr | ||
| 64 | - const: simple-bus | ||
| 65 | - items: | ||
| 66 | - const: fsl,mpc8360-immr | ||
| 67 | - const: fsl,immr | ||
| 68 | - const: fsl,soc | ||
| 69 | - const: simple-bus | ||
| 70 | - const: simple-bus | ||
| 71 | |||
| 72 | additionalProperties: true | ||
| 73 | |||
| 74 | examples: | ||
| 75 | - | | ||
| 76 | / { | ||
| 77 | compatible = "fsl,mpc8315erdb"; | ||
| 78 | model = "MPC8315E-RDB"; | ||
| 79 | #address-cells = <1>; | ||
| 80 | #size-cells = <1>; | ||
| 81 | |||
| 82 | soc@e0000000 { | ||
| 83 | compatible = "fsl,mpc8315-immr", "simple-bus"; | ||
| 84 | reg = <0xe0000000 0x00000200>; | ||
| 85 | #address-cells = <1>; | ||
| 86 | #size-cells = <1>; | ||
| 87 | device_type = "soc"; | ||
| 88 | ranges = <0 0xe0000000 0x00100000>; | ||
| 89 | bus-frequency = <0>; | ||
| 90 | }; | ||
| 91 | }; | ||
| 92 | |||
| 93 | ... | ||
| @@ -168,7 +168,7 @@ properties: · 167 unmodified lines | |||
| 168 | offset from voltage set to regulator. | 168 | offset from voltage set to regulator. |
| 169 | 169 | ||
| 170 | regulator-uv-protection-microvolt: | 170 | regulator-uv-protection-microvolt: |
| 171 | description: Set over under voltage protection limit. This is a limit where | 171 | description: Set under voltage protection limit. This is a limit where |
| 172 | hardware performs emergency shutdown. Zero can be passed to disable | 172 | hardware performs emergency shutdown. Zero can be passed to disable |
| 173 | protection and value '1' indicates that protection should be enabled but | 173 | protection and value '1' indicates that protection should be enabled but |
| 174 | limit setting can be omitted. Limit is given as microvolt offset from | 174 | limit setting can be omitted. Limit is given as microvolt offset from |
| @@ -182,7 +182,7 @@ properties: · 7 unmodified lines | |||
| 182 | is given as microvolt offset from voltage set to regulator. | 182 | is given as microvolt offset from voltage set to regulator. |
| 183 | 183 | ||
| 184 | regulator-uv-warn-microvolt: | 184 | regulator-uv-warn-microvolt: |
| 185 | description: Set over under voltage warning limit. This is a limit where | 185 | description: Set under voltage warning limit. This is a limit where |
| 186 | hardware is assumed still to be functional but approaching limit where | 186 | hardware is assumed still to be functional but approaching limit where |
| 187 | it gets damaged. Recovery actions should be initiated. Zero can be passed | 187 | it gets damaged. Recovery actions should be initiated. Zero can be passed |
| 188 | to disable detection and value '1' indicates that detection should | 188 | to disable detection and value '1' indicates that detection should |
| @@ -33,6 +33,7 @@ properties: · 32 unmodified lines | |||
| 33 | - const: rockchip,rk3066-spdif | 33 | - const: rockchip,rk3066-spdif |
| 34 | - items: | 34 | - items: |
| 35 | - enum: | 35 | - enum: |
| 36 | - rockchip,rk3576-spdif | ||
| 36 | - rockchip,rk3588-spdif | 37 | - rockchip,rk3588-spdif |
| 37 | - const: rockchip,rk3568-spdif | 38 | - const: rockchip,rk3568-spdif |
| 38 | 39 | ||
| @@ -164,7 +164,7 @@ allOf: · 163 unmodified lines | |||
| 164 | properties: | 164 | properties: |
| 165 | compatible: | 165 | compatible: |
| 166 | contains: | 166 | contains: |
| 167 | const: st,stm32mph7-sai | 167 | const: st,stm32h7-sai |
| 168 | then: | 168 | then: |
| 169 | properties: | 169 | properties: |
| 170 | clocks: | 170 | clocks: |
| @@ -6,9 +6,6 @@ $schema: http://devicetree.org/meta-schemas/core.yaml# · 5 unmodified lines | |||
| 6 | 6 | ||
| 7 | title: Allwinner A31 SPI Controller | 7 | title: Allwinner A31 SPI Controller |
| 8 | 8 | ||
| 9 | allOf: | ||
| 10 | - $ref: spi-controller.yaml | ||
| 11 | |||
| 12 | maintainers: | 9 | maintainers: |
| 13 | - Chen-Yu Tsai <wens@csie.org> | 10 | - Chen-Yu Tsai <wens@csie.org> |
| 14 | - Maxime Ripard <mripard@kernel.org> | 11 | - Maxime Ripard <mripard@kernel.org> |
| @@ -82,11 +79,11 @@ patternProperties: · 67 unmodified lines | |||
| 82 | 79 | ||
| 83 | spi-rx-bus-width: | 80 | spi-rx-bus-width: |
| 84 | items: | 81 | items: |
| 85 | - const: 1 | 82 | enum: [0, 1, 2, 4] |
| 86 | 83 | ||
| 87 | spi-tx-bus-width: | 84 | spi-tx-bus-width: |
| 88 | items: | 85 | items: |
| 89 | - const: 1 | 86 | enum: [0, 1, 2, 4] |
| 90 | 87 | ||
| 91 | required: | 88 | required: |
| 92 | - compatible | 89 | - compatible |
| @@ -95,6 +92,28 @@ required: · 2 unmodified lines | |||
| 95 | - clocks | 92 | - clocks |
| 96 | - clock-names | 93 | - clock-names |
| 97 | 94 | ||
| 95 | allOf: | ||
| 96 | - $ref: spi-controller.yaml | ||
| 97 | - if: | ||
| 98 | not: | ||
| 99 | properties: | ||
| 100 | compatible: | ||
| 101 | contains: | ||
| 102 | enum: | ||
| 103 | - allwinner,sun50i-r329-spi | ||
| 104 | - allwinner,sun55i-a523-spi | ||
| 105 | then: | ||
| 106 | patternProperties: | ||
| 107 | "^.*@[0-9a-f]+": | ||
| 108 | properties: | ||
| 109 | spi-rx-bus-width: | ||
| 110 | items: | ||
| 111 | enum: [0, 1] | ||
| 112 | |||
| 113 | spi-tx-bus-width: | ||
| 114 | items: | ||
| 115 | enum: [0, 1] | ||
| 116 | |||
| 98 | unevaluatedProperties: false | 117 | unevaluatedProperties: false |
| 99 | 118 | ||
| 100 | examples: | 119 | examples: |
| @@ -172,6 +172,8 @@ patternProperties: · 171 unmodified lines | |||
| 172 | description: ARM Ltd. | 172 | description: ARM Ltd. |
| 173 | "^armadeus,.*": | 173 | "^armadeus,.*": |
| 174 | description: ARMadeus Systems SARL | 174 | description: ARMadeus Systems SARL |
| 175 | "^armchina,.*": | ||
| 176 | description: Arm Technology (China) Co., Ltd. | ||
| 175 | "^armsom,.*": | 177 | "^armsom,.*": |
| 176 | description: ArmSoM Technology Co., Ltd. | 178 | description: ArmSoM Technology Co., Ltd. |
| 177 | "^arrow,.*": | 179 | "^arrow,.*": |
| @@ -441,6 +443,8 @@ patternProperties: · 263 unmodified lines | |||
| 441 | description: D-Link Corporation | 443 | description: D-Link Corporation |
| 442 | "^dmo,.*": | 444 | "^dmo,.*": |
| 443 | description: Data Modul AG | 445 | description: Data Modul AG |
| 446 | "^doestek,.*": | ||
| 447 | description: Doestek Co., Ltd. | ||
| 444 | "^domintech,.*": | 448 | "^domintech,.*": |
| 445 | description: Domintech Co., Ltd. | 449 | description: Domintech Co., Ltd. |
| 446 | "^dongwoon,.*": | 450 | "^dongwoon,.*": |
| @@ -709,6 +713,8 @@ patternProperties: · 262 unmodified lines | |||
| 709 | description: Hitex Development Tools | 713 | description: Hitex Development Tools |
| 710 | "^hitron,.*": | 714 | "^hitron,.*": |
| 711 | description: HiTRON Electronics Corporation | 715 | description: HiTRON Electronics Corporation |
| 716 | "^holitech,.*": | ||
| 717 | description: Jiangxi Holitech Technology Co., Ltd. | ||
| 712 | "^holt,.*": | 718 | "^holt,.*": |
| 713 | description: Holt Integrated Circuits, Inc. | 719 | description: Holt Integrated Circuits, Inc. |
| 714 | "^holtek,.*": | 720 | "^holtek,.*": |
| @@ -99,3 +99,51 @@ of the driver is decremented. All symlinks between the two are removed. · 98 unmodified lines | |||
| 99 | When a driver is removed, the list of devices that it supports is | 99 | When a driver is removed, the list of devices that it supports is |
| 100 | iterated over, and the driver's remove callback is called for each | 100 | iterated over, and the driver's remove callback is called for each |
| 101 | one. The device is removed from that list and the symlinks removed. | 101 | one. The device is removed from that list and the symlinks removed. |
| 102 | |||
| 103 | |||
| 104 | Driver Override | ||
| 105 | ~~~~~~~~~~~~~~~ | ||
| 106 | |||
| 107 | Userspace may override the standard matching by writing a driver name to | ||
| 108 | a device's ``driver_override`` sysfs attribute. When set, only a driver | ||
| 109 | whose name matches the override will be considered during binding. This | ||
| 110 | bypasses all bus-specific matching (OF, ACPI, ID tables, etc.). | ||
| 111 | |||
| 112 | The override may be cleared by writing an empty string, which returns | ||
| 113 | the device to standard matching rules. Writing to ``driver_override`` | ||
| 114 | does not automatically unbind the device from its current driver or | ||
| 115 | make any attempt to load the specified driver. | ||
| 116 | |||
| 117 | Buses opt into this mechanism by setting the ``driver_override`` flag in | ||
| 118 | their ``struct bus_type``:: | ||
| 119 | |||
| 120 | const struct bus_type example_bus_type = { | ||
| 121 | ... | ||
| 122 | .driver_override = true, | ||
| 123 | }; | ||
| 124 | |||
| 125 | When the flag is set, the driver core automatically creates the | ||
| 126 | ``driver_override`` sysfs attribute for every device on that bus. | ||
| 127 | |||
| 128 | The bus's ``match()`` callback should check the override before performing | ||
| 129 | its own matching, using ``device_match_driver_override()``:: | ||
| 130 | |||
| 131 | static int example_match(struct device *dev, const struct device_driver *drv) | ||
| 132 | { | ||
| 133 | int ret; | ||
| 134 | |||
| 135 | ret = device_match_driver_override(dev, drv); | ||
| 136 | if (ret >= 0) | ||
| 137 | return ret; | ||
| 138 | |||
| 139 | /* Fall through to bus-specific matching... */ | ||
| 140 | } | ||
| 141 | |||
| 142 | ``device_match_driver_override()`` returns > 0 if the override matches | ||
| 143 | the given driver, 0 if the override is set but does not match, or < 0 if | ||
| 144 | no override is set at all. | ||
| 145 | |||
| 146 | Additional helpers are available: | ||
| 147 | |||
| 148 | - ``device_set_driver_override()`` - set or clear the override from kernel code. | ||
| 149 | - ``device_has_driver_override()`` - check whether an override is set. | ||
| @@ -783,6 +783,56 @@ controlled by the "uuid" mount option, which supports these values: · 782 unmodified lines | |||
| 783 | mounted with "uuid=on". | 783 | mounted with "uuid=on". |
| 784 | 784 | ||
| 785 | 785 | ||
| 786 | Durability and copy up | ||
| 787 | ---------------------- | ||
| 788 | |||
| 789 | The fsync(2) system call ensures that the data and metadata of a file | ||
| 790 | are safely written to the backing storage, which is expected to | ||
| 791 | guarantee the existence of the information post system crash. | ||
| 792 | |||
| 793 | Without an fsync(2) call, there is no guarantee that the observed | ||
| 794 | data after a system crash will be either the old or the new data, but | ||
| 795 | in practice, the observed data after crash is often the old or new data | ||
| 796 | or a mix of both. | ||
| 797 | |||
| 798 | When an overlayfs file is modified for the first time, copy up will | ||
| 799 | create a copy of the lower file and its parent directories in the upper | ||
| 800 | layer. Since the Linux filesystem API does not enforce any particular | ||
| 801 | ordering on storing changes without explicit fsync(2) calls, in case | ||
| 802 | of a system crash, the upper file could end up with no data at all | ||
| 803 | (i.e. zeros), which would be an unusual outcome. To avoid this | ||
| 804 | experience, overlayfs calls fsync(2) on the upper file before completing | ||
| 805 | data copy up with rename(2) or link(2) to make the copy up "atomic". | ||
| 806 | |||
| 807 | By default, overlayfs does not explicitly call fsync(2) on copied up | ||
| 808 | directories or on metadata-only copy up, so it provides no guarantee to | ||
| 809 | persist the user's modification unless the user calls fsync(2). | ||
| 810 | The fsync during copy up only guarantees that if a copy up is observed | ||
| 811 | after a crash, the observed data is not zeroes or intermediate values | ||
| 812 | from the copy up staging area. | ||
| 813 | |||
| 814 | On traditional local filesystems with a single journal (e.g. ext4, xfs), | ||
| 815 | fsync on a file also persists the parent directory changes, because they | ||
| 816 | are usually modified in the same transaction, so metadata durability during | ||
| 817 | data copy up effectively comes for free. Overlayfs further limits risk by | ||
| 818 | disallowing network filesystems as upper layer. | ||
| 819 | |||
| 820 | Overlayfs can be tuned to prefer performance or durability when storing | ||
| 821 | to the underlying upper layer. This is controlled by the "fsync" mount | ||
| 822 | option, which supports these values: | ||
| 823 | |||
| 824 | - "auto": (default) | ||
| 825 | Call fsync(2) on upper file before completion of data copy up. | ||
| 826 | No explicit fsync(2) on directory or metadata-only copy up. | ||
| 827 | - "strict": | ||
| 828 | Call fsync(2) on upper file and directories before completion of any | ||
| 829 | copy up. | ||
| 830 | - "volatile": [*] | ||
| 831 | Prefer performance over durability (see `Volatile mount`_) | ||
| 832 | |||
| 833 | [*] The mount option "volatile" is an alias to "fsync=volatile". | ||
| 834 | |||
| 835 | |||
| 786 | Volatile mount | 836 | Volatile mount |
| 787 | -------------- | 837 | -------------- |
| 788 | 838 | ||
| @@ -1089,6 +1089,8 @@ Example output. You may not have all of these fields. · 1088 unmodified lines | |||
| 1089 | CmaFree: 0 kB | 1089 | CmaFree: 0 kB |
| 1090 | Unaccepted: 0 kB | 1090 | Unaccepted: 0 kB |
| 1091 | Balloon: 0 kB | 1091 | Balloon: 0 kB |
| 1092 | GPUActive: 0 kB | ||
| 1093 | GPUReclaim: 0 kB | ||
| 1092 | HugePages_Total: 0 | 1094 | HugePages_Total: 0 |
| 1093 | HugePages_Free: 0 | 1095 | HugePages_Free: 0 |
| 1094 | HugePages_Rsvd: 0 | 1096 | HugePages_Rsvd: 0 |
| @@ -1269,6 +1271,12 @@ Unaccepted · 174 unmodified lines | |||
| 1269 | Memory that has not been accepted by the guest | 1271 | Memory that has not been accepted by the guest |
| 1270 | Balloon | 1272 | Balloon |
| 1271 | Memory returned to Host by VM Balloon Drivers | 1273 | Memory returned to Host by VM Balloon Drivers |
| 1274 | GPUActive | ||
| 1275 | System memory allocated to active GPU objects | ||
| 1276 | GPUReclaim | ||
| 1277 | System memory stored in GPU pools for reuse. This memory is not | ||
| 1278 | counted in GPUActive. It is shrinker reclaimable memory kept in a reuse | ||
| 1279 | pool because it has non-standard page table attributes, like WC or UC. | ||
| 1272 | HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, Hugetlb | 1280 | HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, Hugetlb |
| 1273 | See Documentation/admin-guide/mm/hugetlbpage.rst. | 1281 | See Documentation/admin-guide/mm/hugetlbpage.rst. |
| 1274 | DirectMap4k, DirectMap2M, DirectMap1G | 1282 | DirectMap4k, DirectMap2M, DirectMap1G |
| @@ -104,18 +104,6 @@ VBLANK Helper Reference · 103 unmodified lines | |||
| 104 | .. kernel-doc:: drivers/gpu/drm/drm_vblank_helper.c | 104 | .. kernel-doc:: drivers/gpu/drm/drm_vblank_helper.c |
| 105 | :export: | 105 | :export: |
| 106 | 106 | ||
| 107 | Simple KMS Helper Reference | ||
| 108 | =========================== | ||
| 109 | |||
| 110 | .. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c | ||
| 111 | :doc: overview | ||
| 112 | |||
| 113 | .. kernel-doc:: include/drm/drm_simple_kms_helper.h | ||
| 114 | :internal: | ||
| 115 | |||
| 116 | .. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c | ||
| 117 | :export: | ||
| 118 | |||
| 119 | fbdev Helper Functions Reference | 107 | fbdev Helper Functions Reference |
| 120 | ================================ | 108 | ================================ |
| 121 | 109 | ||
| @@ -438,14 +438,14 @@ following expectations. · 437 unmodified lines | |||
| 438 | unknown consumer policy | 438 | unknown consumer policy |
| 439 | =============== ======================================== | 439 | =============== ======================================== |
| 440 | 440 | ||
| 441 | The only exception to this is ``WEDGED=none``, which signifies that the device | 441 | No Recovery |
| 442 | was temporarily 'wedged' at some point but was recovered from driver context | 442 | ----------- |
| 443 | using device specific methods like reset. No explicit recovery is expected from | 443 | |
| 444 | the consumer in this case, but it can still take additional steps like gathering | 444 | Here ``WEDGED=none`` signifies that no recovery is expected from the consumer |
| 445 | telemetry information (devcoredump, syslog). This is useful because the first | 445 | but it can still try to gather telemetry information (devcoredump, syslog) for |
| 446 | hang is usually the most critical one which can result in consequential hangs or | 446 | debug purpose in order to root cause the hang. This is useful because the first |
| 447 | complete wedging. | 447 | hang is usually the most critical one which can result in consequential hangs |
| 448 | 448 | or complete wedging. | |
| 449 | 449 | ||
| 450 | Vendor Specific Recovery | 450 | Vendor Specific Recovery |
| 451 | ------------------------ | 451 | ------------------------ |
| @@ -119,12 +119,6 @@ Simple DRM drivers to use as examples · 118 unmodified lines | |||
| 119 | The DRM subsystem contains a lot of helper functions to ease writing drivers for | 119 | The DRM subsystem contains a lot of helper functions to ease writing drivers for |
| 120 | simple graphic devices. For example, the `drivers/gpu/drm/tiny/` directory has a | 120 | simple graphic devices. For example, the `drivers/gpu/drm/tiny/` directory has a |
| 121 | set of drivers that are simple enough to be implemented in a single source file. | 121 | set of drivers that are simple enough to be implemented in a single source file. |
| 122 | |||
| 123 | These drivers make use of the `struct drm_simple_display_pipe_funcs`, that hides | ||
| 124 | any complexity of the DRM subsystem and just requires drivers to implement a few | ||
| 125 | functions needed to operate the device. This could be used for devices that just | ||
| 126 | need a display pipeline with one full-screen scanout buffer feeding one output. | ||
| 127 | |||
| 128 | The tiny DRM drivers are good examples to understand how DRM drivers should look | 122 | The tiny DRM drivers are good examples to understand how DRM drivers should look |
| 129 | like. Since are just a few hundreds lines of code, they are quite easy to read. | 123 | like. Since are just a few hundreds lines of code, they are quite easy to read. |
| 130 | 124 | ||
| @@ -51,82 +51,6 @@ There also have been considerations of ToPrimitive [2]. · 50 unmodified lines | |||
| 51 | | Link: https://lore.kernel.org/all/cover.1750689857.git.y.j3ms.n@gmail.com/ [1] | 51 | | Link: https://lore.kernel.org/all/cover.1750689857.git.y.j3ms.n@gmail.com/ [1] |
| 52 | | Link: https://rust-for-linux.zulipchat.com/#narrow/channel/288089-General/topic/Implement.20.60FromPrimitive.60.20trait.20.2B.20derive.20macro.20for.20nova-core/with/541971854 [2] | 52 | | Link: https://rust-for-linux.zulipchat.com/#narrow/channel/288089-General/topic/Implement.20.60FromPrimitive.60.20trait.20.2B.20derive.20macro.20for.20nova-core/with/541971854 [2] |
| 53 | 53 | ||
| 54 | Generic register abstraction [REGA] | ||
| 55 | ----------------------------------- | ||
| 56 | |||
| 57 | Work out how register constants and structures can be automatically generated | ||
| 58 | through generalized macros. | ||
| 59 | |||
| 60 | Example: | ||
| 61 | |||
| 62 | .. code-block:: rust | ||
| 63 | |||
| 64 | register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ | ||
| 65 | MINOR_REVISION(3:0, RO), | ||
| 66 | MAJOR_REVISION(7:4, RO), | ||
| 67 | REVISION(7:0, RO), // Virtual register combining major and minor rev. | ||
| 68 | ]) | ||
| 69 | |||
| 70 | This could expand to something like: | ||
| 71 | |||
| 72 | .. code-block:: rust | ||
| 73 | |||
| 74 | const BOOT0_OFFSET: usize = 0x00000000; | ||
| 75 | const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; | ||
| 76 | const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; | ||
| 77 | const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; | ||
| 78 | const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; | ||
| 79 | const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; | ||
| 80 | const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; | ||
| 81 | |||
| 82 | struct Boot0(u32); | ||
| 83 | |||
| 84 | impl Boot0 { | ||
| 85 | #[inline] | ||
| 86 | fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { | ||
| 87 | Self(bar.readl(BOOT0_OFFSET)) | ||
| 88 | } | ||
| 89 | |||
| 90 | #[inline] | ||
| 91 | fn minor_revision(&self) -> u32 { | ||
| 92 | (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT | ||
| 93 | } | ||
| 94 | |||
| 95 | #[inline] | ||
| 96 | fn major_revision(&self) -> u32 { | ||
| 97 | (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT | ||
| 98 | } | ||
| 99 | |||
| 100 | #[inline] | ||
| 101 | fn revision(&self) -> u32 { | ||
| 102 | (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT | ||
| 103 | } | ||
| 104 | } | ||
| 105 | |||
| 106 | Usage: | ||
| 107 | |||
| 108 | .. code-block:: rust | ||
| 109 | |||
| 110 | let bar = bar.try_access().ok_or(ENXIO)?; | ||
| 111 | |||
| 112 | let boot0 = Boot0::read(&bar); | ||
| 113 | pr_info!("Revision: {}\n", boot0.revision()); | ||
| 114 | |||
| 115 | A work-in-progress implementation currently resides in | ||
| 116 | `drivers/gpu/nova-core/regs/macros.rs` and is used in nova-core. It would be | ||
| 117 | nice to improve it (possibly using proc macros) and move it to the `kernel` | ||
| 118 | crate so it can be used by other components as well. | ||
| 119 | |||
| 120 | Features desired before this happens: | ||
| 121 | |||
| 122 | * Make I/O optional I/O (for field values that are not registers), | ||
| 123 | * Support other sizes than `u32`, | ||
| 124 | * Allow visibility control for registers and individual fields, | ||
| 125 | * Use Rust slice syntax to express fields ranges. | ||
| 126 | |||
| 127 | | Complexity: Advanced | ||
| 128 | | Contact: Alexandre Courbot | ||
| 129 | |||
| 130 | Numerical operations [NUMM] | 54 | Numerical operations [NUMM] |
| 131 | --------------------------- | 55 | --------------------------- |
| 132 | 56 | ||
| @@ -29,6 +29,38 @@ refactorings already and are an expert in the specific area · 28 unmodified lines | |||
| 29 | Subsystem-wide refactorings | 29 | Subsystem-wide refactorings |
| 30 | =========================== | 30 | =========================== |
| 31 | 31 | ||
| 32 | Open-code drm_simple_encoder_init() | ||
| 33 | ----------------------------------- | ||
| 34 | |||
| 35 | The helper drm_simple_encoder_init() was supposed to simplify encoder | ||
| 36 | initialization. Instead it only added an intermediate layer between atomic | ||
| 37 | modesetting and the DRM driver. | ||
| 38 | |||
| 39 | The task here is to remove drm_simple_encoder_init(). Search for a driver | ||
| 40 | that calls drm_simple_encoder_init() and inline the helper. The driver will | ||
| 41 | also need its own instance of drm_encoder_funcs. | ||
| 42 | |||
| 43 | Contact: Thomas Zimmermann, respective driver maintainer | ||
| 44 | |||
| 45 | Level: Easy | ||
| 46 | |||
| 47 | Replace struct drm_simple_display_pipe with regular atomic helpers | ||
| 48 | ------------------------------------------------------------------ | ||
| 49 | |||
| 50 | The data type struct drm_simple_display_pipe and its helpers were supposed | ||
| 51 | to simplify driver development. Instead they only added an intermediate layer | ||
| 52 | between atomic modesetting and the DRM driver. | ||
| 53 | |||
| 54 | There are still drivers that use drm_simple_display_pipe. The task here is to | ||
| 55 | convert them to use regular atomic helpers. Search for a driver that calls | ||
| 56 | drm_simple_display_pipe_init() and inline all helpers from drm_simple_kms_helper.c | ||
| 57 | into the driver, such that no simple-KMS interfaces are required. Please also | ||
| 58 | rename all inlined fucntions according to driver conventions. | ||
| 59 | |||
| 60 | Contact: Thomas Zimmermann, respective driver maintainer | ||
| 61 | |||
| 62 | Level: Easy | ||
| 63 | |||
| 32 | Remove custom dumb_map_offset implementations | 64 | Remove custom dumb_map_offset implementations |
| 33 | --------------------------------------------- | 65 | --------------------------------------------- |
| 34 | 66 | ||
| @@ -27,10 +27,10 @@ for details. · 26 unmodified lines | |||
| 27 | Sysfs entries | 27 | Sysfs entries |
| 28 | ------------- | 28 | ------------- |
| 29 | 29 | ||
| 30 | The following attributes are supported. Current maxim attribute | 30 | The following attributes are supported. Current maximum attribute |
| 31 | is read-write, all other attributes are read-only. | 31 | is read-write, all other attributes are read-only. |
| 32 | 32 | ||
| 33 | in0_input Measured voltage in microvolts. | 33 | in0_input Measured voltage in millivolts. |
| 34 | 34 | ||
| 35 | curr1_input Measured current in microamperes. | 35 | curr1_input Measured current in milliamperes. |
| 36 | curr1_max_alarm Overcurrent alarm in microamperes. | 36 | curr1_max Overcurrent shutdown threshold in milliamperes. |
| @@ -51,8 +51,9 @@ temp1_max Provides thermal control temperature of the CPU package · 50 unmodified lines | |||
| 51 | temp1_crit Provides shutdown temperature of the CPU package which | 51 | temp1_crit Provides shutdown temperature of the CPU package which |
| 52 | is also known as the maximum processor junction | 52 | is also known as the maximum processor junction |
| 53 | temperature, Tjmax or Tprochot. | 53 | temperature, Tjmax or Tprochot. |
| 54 | temp1_crit_hyst Provides the hysteresis value from Tcontrol to Tjmax of | 54 | temp1_crit_hyst Provides the hysteresis temperature of the CPU |
| 55 | the CPU package. | 55 | package. Returns Tcontrol, the temperature at which |
| 56 | the critical condition clears. | ||
| 56 | 57 | ||
| 57 | temp2_label "DTS" | 58 | temp2_label "DTS" |
| 58 | temp2_input Provides current temperature of the CPU package scaled | 59 | temp2_input Provides current temperature of the CPU package scaled |
| @@ -62,8 +63,9 @@ temp2_max Provides thermal control temperature of the CPU package · 3 unmodified lines | |||
| 62 | temp2_crit Provides shutdown temperature of the CPU package which | 63 | temp2_crit Provides shutdown temperature of the CPU package which |
| 63 | is also known as the maximum processor junction | 64 | is also known as the maximum processor junction |
| 64 | temperature, Tjmax or Tprochot. | 65 | temperature, Tjmax or Tprochot. |
| 65 | temp2_crit_hyst Provides the hysteresis value from Tcontrol to Tjmax of | 66 | temp2_crit_hyst Provides the hysteresis temperature of the CPU |
| 66 | the CPU package. | 67 | package. Returns Tcontrol, the temperature at which |
| 68 | the critical condition clears. | ||
| 67 | 69 | ||
| 68 | temp3_label "Tcontrol" | 70 | temp3_label "Tcontrol" |
| 69 | temp3_input Provides current Tcontrol temperature of the CPU | 71 | temp3_input Provides current Tcontrol temperature of the CPU |
| @@ -247,8 +247,8 @@ operations: · 246 unmodified lines | |||
| 247 | flags: [admin-perm] | 247 | flags: [admin-perm] |
| 248 | 248 | ||
| 249 | do: | 249 | do: |
| 250 | pre: net-shaper-nl-pre-doit | 250 | pre: net-shaper-nl-pre-doit-write |
| 251 | post: net-shaper-nl-post-doit | 251 | post: net-shaper-nl-post-doit-write |
| 252 | request: | 252 | request: |
| 253 | attributes: | 253 | attributes: |
| 254 | - ifindex | 254 | - ifindex |
| @@ -278,8 +278,8 @@ operations: · 23 unmodified lines | |||
| 278 | flags: [admin-perm] | 278 | flags: [admin-perm] |
| 279 | 279 | ||
| 280 | do: | 280 | do: |
| 281 | pre: net-shaper-nl-pre-doit | 281 | pre: net-shaper-nl-pre-doit-write |
| 282 | post: net-shaper-nl-post-doit | 282 | post: net-shaper-nl-post-doit-write |
| 283 | request: | 283 | request: |
| 284 | attributes: *ns-binding | 284 | attributes: *ns-binding |
| 285 | 285 | ||
| @@ -309,8 +309,8 @@ operations: · 23 unmodified lines | |||
| 309 | flags: [admin-perm] | 309 | flags: [admin-perm] |
| 310 | 310 | ||
| 311 | do: | 311 | do: |
| 312 | pre: net-shaper-nl-pre-doit | 312 | pre: net-shaper-nl-pre-doit-write |
| 313 | post: net-shaper-nl-post-doit | 313 | post: net-shaper-nl-post-doit-write |
| 314 | request: | 314 | request: |
| 315 | attributes: | 315 | attributes: |
| 316 | - ifindex | 316 | - ifindex |
| @@ -5,8 +5,138 @@ Security bugs · 4 unmodified lines | |||
| 5 | 5 | ||
| 6 | Linux kernel developers take security very seriously. As such, we'd | 6 | Linux kernel developers take security very seriously. As such, we'd |
| 7 | like to know when a security bug is found so that it can be fixed and | 7 | like to know when a security bug is found so that it can be fixed and |
| 8 | disclosed as quickly as possible. Please report security bugs to the | 8 | disclosed as quickly as possible. |
| 9 | Linux kernel security team. | 9 | |
| 10 | Preparing your report | ||
| 11 | --------------------- | ||
| 12 | |||
| 13 | Like with any bug report, a security bug report requires a lot of analysis work | ||
| 14 | from the developers, so the more information you can share about the issue, the | ||
| 15 | better. Please review the procedure outlined in | ||
| 16 | Documentation/admin-guide/reporting-issues.rst if you are unclear about what | ||
| 17 | information is helpful. The following information are absolutely necessary in | ||
| 18 | **any** security bug report: | ||
| 19 | |||
| 20 | * **affected kernel version range**: with no version indication, your report | ||
| 21 | will not be processed. A significant part of reports are for bugs that | ||
| 22 | have already been fixed, so it is extremely important that vulnerabilities | ||
| 23 | are verified on recent versions (development tree or latest stable | ||
| 24 | version), at least by verifying that the code has not changed since the | ||
| 25 | version where it was detected. | ||
| 26 | |||
| 27 | * **description of the problem**: a detailed description of the problem, with | ||
| 28 | traces showing its manifestation, and why you consider that the observed | ||
| 29 | behavior as a problem in the kernel, is necessary. | ||
| 30 | |||
| 31 | * **reproducer**: developers will need to be able to reproduce the problem to | ||
| 32 | consider a fix as effective. This includes both a way to trigger the issue | ||
| 33 | and a way to confirm it happens. A reproducer with low complexity | ||
| 34 | dependencies will be needed (source code, shell script, sequence of | ||
| 35 | instructions, file-system image etc). Binary-only executables are not | ||
| 36 | accepted. Working exploits are extremely helpful and will not be released | ||
| 37 | without consent from the reporter, unless they are already public. By | ||
| 38 | definition if an issue cannot be reproduced, it is not exploitable, thus it | ||
| 39 | is not a security bug. | ||
| 40 | |||
| 41 | * **conditions**: if the bug depends on certain configuration options, | ||
| 42 | sysctls, permissions, timing, code modifications etc, these should be | ||
| 43 | indicated. | ||
| 44 | |||
| 45 | In addition, the following information are highly desirable: | ||
| 46 | |||
| 47 | * **suspected location of the bug**: the file names and functions where the | ||
| 48 | bug is suspected to be present are very important, at least to help forward | ||
| 49 | the report to the appropriate maintainers. When not possible (for example, | ||
| 50 | "system freezes each time I run this command"), the security team will help | ||
| 51 | identify the source of the bug. | ||
| 52 | |||
| 53 | * **a proposed fix**: bug reporters who have analyzed the cause of a bug in | ||
| 54 | the source code almost always have an accurate idea on how to fix it, | ||
| 55 | because they spent a long time studying it and its implications. Proposing | ||
| 56 | a tested fix will save maintainers a lot of time, even if the fix ends up | ||
| 57 | not being the right one, because it helps understand the bug. When | ||
| 58 | proposing a tested fix, please always format it in a way that can be | ||
| 59 | immediately merged (see Documentation/process/submitting-patches.rst). | ||
| 60 | This will save some back-and-forth exchanges if it is accepted, and you | ||
| 61 | will be credited for finding and fixing this issue. Note that in this case | ||
| 62 | only a ``Signed-off-by:`` tag is needed, without ``Reported-by:`` when the | ||
| 63 | reporter and author are the same. | ||
| 64 | |||
| 65 | * **mitigations**: very often during a bug analysis, some ways of mitigating | ||
| 66 | the issue appear. It is useful to share them, as they can be helpful to | ||
| 67 | keep end users protected during the time it takes them to apply the fix. | ||
| 68 | |||
| 69 | Identifying contacts | ||
| 70 | -------------------- | ||
| 71 | |||
| 72 | The most effective way to report a security bug is to send it directly to the | ||
| 73 | affected subsystem's maintainers and Cc: the Linux kernel security team. Do | ||
| 74 | not send it to a public list at this stage, unless you have good reasons to | ||
| 75 | consider the issue as being public or trivial to discover (e.g. result of a | ||
| 76 | widely available automated vulnerability scanning tool that can be repeated by | ||
| 77 | anyone). | ||
| 78 | |||
| 79 | If you're sending a report for issues affecting multiple parts in the kernel, | ||
| 80 | even if they're fairly similar issues, please send individual messages (think | ||
| 81 | that maintainers will not all work on the issues at the same time). The only | ||
| 82 | exception is when an issue concerns closely related parts maintained by the | ||
| 83 | exact same subset of maintainers, and these parts are expected to be fixed all | ||
| 84 | at once by the same commit, then it may be acceptable to report them at once. | ||
| 85 | |||
| 86 | One difficulty for most first-time reporters is to figure the right list of | ||
| 87 | recipients to send a report to. In the Linux kernel, all official maintainers | ||
| 88 | are trusted, so the consequences of accidentally including the wrong maintainer | ||
| 89 | are essentially a bit more noise for that person, i.e. nothing dramatic. As | ||
| 90 | such, a suitable method to figure the list of maintainers (which kernel | ||
| 91 | security officers use) is to rely on the get_maintainer.pl script, tuned to | ||
| 92 | only report maintainers. This script, when passed a file name, will look for | ||
| 93 | its path in the MAINTAINERS file to figure a hierarchical list of relevant | ||
| 94 | maintainers. Calling it a first time with the finest level of filtering will | ||
| 95 | most of the time return a short list of this specific file's maintainers:: | ||
| 96 | |||
| 97 | $ ./scripts/get_maintainer.pl --no-l --no-r --pattern-depth 1 \ | ||
| 98 | drivers/example.c | ||
| 99 | Developer One <dev1@example.com> (maintainer:example driver) | ||
| 100 | Developer Two <dev2@example.org> (maintainer:example driver) | ||
| 101 | |||
| 102 | These two maintainers should then receive the message. If the command does not | ||
| 103 | return anything, it means the affected file is part of a wider subsystem, so we | ||
| 104 | should be less specific:: | ||
| 105 | |||
| 106 | $ ./scripts/get_maintainer.pl --no-l --no-r drivers/example.c | ||
| 107 | Developer One <dev1@example.com> (maintainer:example subsystem) | ||
| 108 | Developer Two <dev2@example.org> (maintainer:example subsystem) | ||
| 109 | Developer Three <dev3@example.com> (maintainer:example subsystem [GENERAL]) | ||
| 110 | Developer Four <dev4@example.org> (maintainer:example subsystem [GENERAL]) | ||
| 111 | |||
| 112 | Here, picking the first, most specific ones, is sufficient. When the list is | ||
| 113 | long, it is possible to produce a comma-delimited e-mail address list on a | ||
| 114 | single line suitable for use in the To: field of a mailer like this:: | ||
| 115 | |||
| 116 | $ ./scripts/get_maintainer.pl --no-tree --no-l --no-r --no-n --m \ | ||
| 117 | --no-git-fallback --no-substatus --no-rolestats --no-multiline \ | ||
| 118 | --pattern-depth 1 drivers/example.c | ||
| 119 | dev1@example.com, dev2@example.org | ||
| 120 | |||
| 121 | or this for the wider list:: | ||
| 122 | |||
| 123 | $ ./scripts/get_maintainer.pl --no-tree --no-l --no-r --no-n --m \ | ||
| 124 | --no-git-fallback --no-substatus --no-rolestats --no-multiline \ | ||
| 125 | drivers/example.c | ||
| 126 | dev1@example.com, dev2@example.org, dev3@example.com, dev4@example.org | ||
| 127 | |||
| 128 | If at this point you're still facing difficulties spotting the right | ||
| 129 | maintainers, **and only in this case**, it's possible to send your report to | ||
| 130 | the Linux kernel security team only. Your message will be triaged, and you | ||
| 131 | will receive instructions about whom to contact, if needed. Your message may | ||
| 132 | equally be forwarded as-is to the relevant maintainers. | ||
| 133 | |||
| 134 | Sending the report | ||
| 135 | ------------------ | ||
| 136 | |||
| 137 | Reports are to be sent over e-mail exclusively. Please use a working e-mail | ||
| 138 | address, preferably the same that you want to appear in ``Reported-by`` tags | ||
| 139 | if any. If unsure, send your report to yourself first. | ||
| 10 | 140 | ||
| 11 | The security team and maintainers almost always require additional | 141 | The security team and maintainers almost always require additional |
| 12 | information beyond what was initially provided in a report and rely on | 142 | information beyond what was initially provided in a report and rely on |
| @@ -18,20 +148,12 @@ run additional tests. Reports where the reporter does not respond promptly · 5 unmodified lines | |||
| 18 | or cannot effectively discuss their findings may be abandoned if the | 148 | or cannot effectively discuss their findings may be abandoned if the |
| 19 | communication does not quickly improve. | 149 | communication does not quickly improve. |
| 20 | 150 | ||
| 21 | As it is with any bug, the more information provided the easier it | 151 | The report must be sent to maintainers, with the security team in ``Cc:``. |
| 22 | will be to diagnose and fix. Please review the procedure outlined in | ||
| 23 | 'Documentation/admin-guide/reporting-issues.rst' if you are unclear about what | ||
| 24 | information is helpful. Any exploit code is very helpful and will not | ||
| 25 | be released without consent from the reporter unless it has already been | ||
| 26 | made public. | ||
| 27 | |||
| 28 | The Linux kernel security team can be contacted by email at | 152 | The Linux kernel security team can be contacted by email at |
| 29 | <security@kernel.org>. This is a private list of security officers | 153 | <security@kernel.org>. This is a private list of security officers |
| 30 | who will help verify the bug report and develop and release a fix. | 154 | who will help verify the bug report and assist developers working on a fix. |
| 31 | If you already have a fix, please include it with your report, as | 155 | It is possible that the security team will bring in extra help from area |
| 32 | that can speed up the process considerably. It is possible that the | 156 | maintainers to understand and fix the security vulnerability. |
| 33 | security team will bring in extra help from area maintainers to | ||
| 34 | understand and fix the security vulnerability. | ||
| 35 | 157 | ||
| 36 | Please send **plain text** emails without attachments where possible. | 158 | Please send **plain text** emails without attachments where possible. |
| 37 | It is much harder to have a context-quoted discussion about a complex | 159 | It is much harder to have a context-quoted discussion about a complex |
| @@ -42,7 +164,9 @@ reproduction steps, and follow it with a proposed fix, all in plain text. · 4 unmodified lines | |||
| 42 | Markdown, HTML and RST formatted reports are particularly frowned upon since | 164 | Markdown, HTML and RST formatted reports are particularly frowned upon since |
| 43 | they're quite hard to read for humans and encourage to use dedicated viewers, | 165 | they're quite hard to read for humans and encourage to use dedicated viewers, |
| 44 | sometimes online, which by definition is not acceptable for a confidential | 166 | sometimes online, which by definition is not acceptable for a confidential |
| 45 | security report. | 167 | security report. Note that some mailers tend to mangle formatting of plain |
| 168 | text by default, please consult Documentation/process/email-clients.rst for | ||
| 169 | more info. | ||
| 46 | 170 | ||
| 47 | Disclosure and embargoed information | 171 | Disclosure and embargoed information |
| 48 | ------------------------------------ | 172 | ------------------------------------ |
| @@ -43,7 +43,6 @@ options should be enabled to use sched_ext: · 42 unmodified lines | |||
| 43 | CONFIG_DEBUG_INFO_BTF=y | 43 | CONFIG_DEBUG_INFO_BTF=y |
| 44 | CONFIG_BPF_JIT_ALWAYS_ON=y | 44 | CONFIG_BPF_JIT_ALWAYS_ON=y |
| 45 | CONFIG_BPF_JIT_DEFAULT_ON=y | 45 | CONFIG_BPF_JIT_DEFAULT_ON=y |
| 46 | CONFIG_PAHOLE_HAS_BTF_TAG=y | ||
| 47 | 46 | ||
| 48 | sched_ext is used only when the BPF scheduler is loaded and running. | 47 | sched_ext is used only when the BPF scheduler is loaded and running. |
| 49 | 48 | ||
| @@ -58,7 +57,8 @@ in ``ops->flags``, all ``SCHED_NORMAL``, ``SCHED_BATCH``, ``SCHED_IDLE``, and · 8 unmodified lines | |||
| 58 | However, when the BPF scheduler is loaded and ``SCX_OPS_SWITCH_PARTIAL`` is | 57 | However, when the BPF scheduler is loaded and ``SCX_OPS_SWITCH_PARTIAL`` is |
| 59 | set in ``ops->flags``, only tasks with the ``SCHED_EXT`` policy are scheduled | 58 | set in ``ops->flags``, only tasks with the ``SCHED_EXT`` policy are scheduled |
| 60 | by sched_ext, while tasks with ``SCHED_NORMAL``, ``SCHED_BATCH`` and | 59 | by sched_ext, while tasks with ``SCHED_NORMAL``, ``SCHED_BATCH`` and |
| 61 | ``SCHED_IDLE`` policies are scheduled by the fair-class scheduler. | 60 | ``SCHED_IDLE`` policies are scheduled by the fair-class scheduler which has |
| 61 | higher sched_class precedence than ``SCHED_EXT``. | ||
| 62 | 62 | ||
| 63 | Terminating the sched_ext scheduler program, triggering `SysRq-S`, or | 63 | Terminating the sched_ext scheduler program, triggering `SysRq-S`, or |
| 64 | detection of any internal error including stalled runnable tasks aborts the | 64 | detection of any internal error including stalled runnable tasks aborts the |
| @@ -345,6 +345,8 @@ Where to Look · 280 unmodified lines | |||
| 345 | The functions prefixed with ``scx_bpf_`` can be called from the BPF | 345 | The functions prefixed with ``scx_bpf_`` can be called from the BPF |
| 346 | scheduler. | 346 | scheduler. |
| 347 | 347 | ||
| 348 | * ``kernel/sched/ext_idle.c`` contains the built-in idle CPU selection policy. | ||
| 349 | |||
| 348 | * ``tools/sched_ext/`` hosts example BPF scheduler implementations. | 350 | * ``tools/sched_ext/`` hosts example BPF scheduler implementations. |
| 349 | 351 | ||
| 350 | * ``scx_simple[.bpf].c``: Minimal global FIFO scheduler example using a | 352 | * ``scx_simple[.bpf].c``: Minimal global FIFO scheduler example using a |
| @@ -353,13 +355,35 @@ Where to Look · 2 unmodified lines | |||
| 353 | * ``scx_qmap[.bpf].c``: A multi-level FIFO scheduler supporting five | 355 | * ``scx_qmap[.bpf].c``: A multi-level FIFO scheduler supporting five |
| 354 | levels of priority implemented with ``BPF_MAP_TYPE_QUEUE``. | 356 | levels of priority implemented with ``BPF_MAP_TYPE_QUEUE``. |
| 355 | 357 | ||
| 358 | * ``scx_central[.bpf].c``: A central FIFO scheduler where all scheduling | ||
| 359 | decisions are made on one CPU, demonstrating ``LOCAL_ON`` dispatching, | ||
| 360 | tickless operation, and kthread preemption. | ||
| 361 | |||
| 362 | * ``scx_cpu0[.bpf].c``: A scheduler that queues all tasks to a shared DSQ | ||
| 363 | and only dispatches them on CPU0 in FIFO order. Useful for testing bypass | ||
| 364 | behavior. | ||
| 365 | |||
| 366 | * ``scx_flatcg[.bpf].c``: A flattened cgroup hierarchy scheduler | ||
| 367 | implementing hierarchical weight-based cgroup CPU control by compounding | ||
| 368 | each cgroup's share at every level into a single flat scheduling layer. | ||
| 369 | |||
| 370 | * ``scx_pair[.bpf].c``: A core-scheduling example that always makes | ||
| 371 | sibling CPU pairs execute tasks from the same CPU cgroup. | ||
| 372 | |||
| 373 | * ``scx_sdt[.bpf].c``: A variation of ``scx_simple`` demonstrating BPF | ||
| 374 | arena memory management for per-task data. | ||
| 375 | |||
| 376 | * ``scx_userland[.bpf].c``: A minimal scheduler demonstrating user space | ||
| 377 | scheduling. Tasks with CPU affinity are direct-dispatched in FIFO order; | ||
| 378 | all others are scheduled in user space by a simple vruntime scheduler. | ||
| 379 | |||
| 356 | ABI Instability | 380 | ABI Instability |
| 357 | =============== | 381 | =============== |
| 358 | 382 | ||
| 359 | The APIs provided by sched_ext to BPF schedulers programs have no stability | 383 | The APIs provided by sched_ext to BPF schedulers programs have no stability |
| 360 | guarantees. This includes the ops table callbacks and constants defined in | 384 | guarantees. This includes the ops table callbacks and constants defined in |
| 361 | ``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in | 385 | ``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in |
| 362 | ``kernel/sched/ext.c``. | 386 | ``kernel/sched/ext.c`` and ``kernel/sched/ext_idle.c``. |
| 363 | 387 | ||
| 364 | While we will attempt to provide a relatively stable API surface when | 388 | While we will attempt to provide a relatively stable API surface when |
| 365 | possible, they are subject to change without warning between kernel | 389 | possible, they are subject to change without warning between kernel |
| @@ -8,7 +8,7 @@ Landlock: unprivileged access control · 7 unmodified lines | |||
| 8 | ===================================== | 8 | ===================================== |
| 9 | 9 | ||
| 10 | :Author: Mickaël Salaün | 10 | :Author: Mickaël Salaün |
| 11 | :Date: January 2026 | 11 | :Date: March 2026 |
| 12 | 12 | ||
| 13 | The goal of Landlock is to enable restriction of ambient rights (e.g. global | 13 | The goal of Landlock is to enable restriction of ambient rights (e.g. global |
| 14 | filesystem or network access) for a set of processes. Because Landlock | 14 | filesystem or network access) for a set of processes. Because Landlock |
| @@ -197,12 +197,27 @@ similar backwards compatibility check is needed for the restrict flags · 182 unmodified lines | |||
| 197 | 197 | ||
| 198 | .. code-block:: c | 198 | .. code-block:: c |
| 199 | 199 | ||
| 200 | __u32 restrict_flags = LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON; | 200 | __u32 restrict_flags = |
| 201 | if (abi < 7) { | 201 | LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON | |
| 202 | /* Clear logging flags unsupported before ABI 7. */ | 202 | LANDLOCK_RESTRICT_SELF_TSYNC; |
| 203 | switch (abi) { | ||
| 204 | case 1 ... 6: | ||
| 205 | /* Removes logging flags for ABI < 7 */ | ||
| 203 | restrict_flags &= ~(LANDLOCK_RESTRICT_SELF_LOG_SAME_EXEC_OFF | | 206 | restrict_flags &= ~(LANDLOCK_RESTRICT_SELF_LOG_SAME_EXEC_OFF | |
| 204 | LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON | | 207 | LANDLOCK_RESTRICT_SELF_LOG_NEW_EXEC_ON | |
| 205 | LANDLOCK_RESTRICT_SELF_LOG_SUBDOMAINS_OFF); | 208 | LANDLOCK_RESTRICT_SELF_LOG_SUBDOMAINS_OFF); |
| 209 | __attribute__((fallthrough)); | ||
| 210 | case 7: | ||
| 211 | /* | ||
| 212 | * Removes multithreaded enforcement flag for ABI < 8 | ||
| 213 | * | ||
| 214 | * WARNING: Without this flag, calling landlock_restrict_self(2) is | ||
| 215 | * only equivalent if the calling process is single-threaded. Below | ||
| 216 | * ABI v8 (and as of ABI v8, when not using this flag), a Landlock | ||
| 217 | * policy would only be enforced for the calling thread and its | ||
| 218 | * children (and not for all threads, including parents and siblings). | ||
| 219 | */ | ||
| 220 | restrict_flags &= ~LANDLOCK_RESTRICT_SELF_TSYNC; | ||
| 206 | } | 221 | } |
| 207 | 222 | ||
| 208 | The next step is to restrict the current thread from gaining more privileges | 223 | The next step is to restrict the current thread from gaining more privileges |
| @@ -8435,115 +8435,123 @@ KVM_CHECK_EXTENSION. · 8434 unmodified lines | |||
| 8435 | 8435 | ||
| 8436 | The valid bits in cap.args[0] are: | 8436 | The valid bits in cap.args[0] are: |
| 8437 | 8437 | ||
| 8438 | =================================== ============================================ | 8438 | ======================================== ================================================ |
| 8439 | KVM_X86_QUIRK_LINT0_REENABLED By default, the reset value for the LVT | 8439 | KVM_X86_QUIRK_LINT0_REENABLED By default, the reset value for the LVT |
| 8440 | LINT0 register is 0x700 (APIC_MODE_EXTINT). | 8440 | LINT0 register is 0x700 (APIC_MODE_EXTINT). |
| 8441 | When this quirk is disabled, the reset value | 8441 | When this quirk is disabled, the reset value |
| 8442 | is 0x10000 (APIC_LVT_MASKED). | 8442 | is 0x10000 (APIC_LVT_MASKED). |
| 8443 | 8443 | ||
| 8444 | KVM_X86_QUIRK_CD_NW_CLEARED By default, KVM clears CR0.CD and CR0.NW on | 8444 | KVM_X86_QUIRK_CD_NW_CLEARED By default, KVM clears CR0.CD and CR0.NW on |
| 8445 | AMD CPUs to workaround buggy guest firmware | 8445 | AMD CPUs to workaround buggy guest firmware |
| 8446 | that runs in perpetuity with CR0.CD, i.e. | 8446 | that runs in perpetuity with CR0.CD, i.e. |
| 8447 | with caches in "no fill" mode. | 8447 | with caches in "no fill" mode. |
| 8448 | 8448 | ||
| 8449 | When this quirk is disabled, KVM does not | 8449 | When this quirk is disabled, KVM does not |
| 8450 | change the value of CR0.CD and CR0.NW. | 8450 | change the value of CR0.CD and CR0.NW. |
| 8451 | 8451 | ||
| 8452 | KVM_X86_QUIRK_LAPIC_MMIO_HOLE By default, the MMIO LAPIC interface is | 8452 | KVM_X86_QUIRK_LAPIC_MMIO_HOLE By default, the MMIO LAPIC interface is |
| 8453 | available even when configured for x2APIC | 8453 | available even when configured for x2APIC |
| 8454 | mode. When this quirk is disabled, KVM | 8454 | mode. When this quirk is disabled, KVM |
| 8455 | disables the MMIO LAPIC interface if the | 8455 | disables the MMIO LAPIC interface if the |
| 8456 | LAPIC is in x2APIC mode. | 8456 | LAPIC is in x2APIC mode. |
| 8457 | 8457 | ||
| 8458 | KVM_X86_QUIRK_OUT_7E_INC_RIP By default, KVM pre-increments %rip before | 8458 | KVM_X86_QUIRK_OUT_7E_INC_RIP By default, KVM pre-increments %rip before |
| 8459 | exiting to userspace for an OUT instruction | 8459 | exiting to userspace for an OUT instruction |
| 8460 | to port 0x7e. When this quirk is disabled, | 8460 | to port 0x7e. When this quirk is disabled, |
| 8461 | KVM does not pre-increment %rip before | 8461 | KVM does not pre-increment %rip before |
| 8462 | exiting to userspace. | 8462 | exiting to userspace. |
| 8463 | 8463 | ||
| 8464 | KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT When this quirk is disabled, KVM sets | 8464 | KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT When this quirk is disabled, KVM sets |
| 8465 | CPUID.01H:ECX[bit 3] (MONITOR/MWAIT) if | 8465 | CPUID.01H:ECX[bit 3] (MONITOR/MWAIT) if |
| 8466 | IA32_MISC_ENABLE[bit 18] (MWAIT) is set. | 8466 | IA32_MISC_ENABLE[bit 18] (MWAIT) is set. |
| 8467 | Additionally, when this quirk is disabled, | 8467 | Additionally, when this quirk is disabled, |
| 8468 | KVM clears CPUID.01H:ECX[bit 3] if | 8468 | KVM clears CPUID.01H:ECX[bit 3] if |
| 8469 | IA32_MISC_ENABLE[bit 18] is cleared. | 8469 | IA32_MISC_ENABLE[bit 18] is cleared. |
| 8470 | 8470 | ||
| 8471 | KVM_X86_QUIRK_FIX_HYPERCALL_INSN By default, KVM rewrites guest | 8471 | KVM_X86_QUIRK_FIX_HYPERCALL_INSN By default, KVM rewrites guest |
| 8472 | VMMCALL/VMCALL instructions to match the | 8472 | VMMCALL/VMCALL instructions to match the |
| 8473 | vendor's hypercall instruction for the | 8473 | vendor's hypercall instruction for the |
| 8474 | system. When this quirk is disabled, KVM | 8474 | system. When this quirk is disabled, KVM |
| 8475 | will no longer rewrite invalid guest | 8475 | will no longer rewrite invalid guest |
| 8476 | hypercall instructions. Executing the | 8476 | hypercall instructions. Executing the |
| 8477 | incorrect hypercall instruction will | 8477 | incorrect hypercall instruction will |
| 8478 | generate a #UD within the guest. | 8478 | generate a #UD within the guest. |
| 8479 | 8479 | ||
| 8480 | KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS By default, KVM emulates MONITOR/MWAIT (if | 8480 | KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS By default, KVM emulates MONITOR/MWAIT (if |
| 8481 | they are intercepted) as NOPs regardless of | 8481 | they are intercepted) as NOPs regardless of |
| 8482 | whether or not MONITOR/MWAIT are supported | 8482 | whether or not MONITOR/MWAIT are supported |
| 8483 | according to guest CPUID. When this quirk | 8483 | according to guest CPUID. When this quirk |
| 8484 | is disabled and KVM_X86_DISABLE_EXITS_MWAIT | 8484 | is disabled and KVM_X86_DISABLE_EXITS_MWAIT |
| 8485 | is not set (MONITOR/MWAIT are intercepted), | 8485 | is not set (MONITOR/MWAIT are intercepted), |
| 8486 | KVM will inject a #UD on MONITOR/MWAIT if | 8486 | KVM will inject a #UD on MONITOR/MWAIT if |
| 8487 | they're unsupported per guest CPUID. Note, | 8487 | they're unsupported per guest CPUID. Note, |
| 8488 | KVM will modify MONITOR/MWAIT support in | 8488 | KVM will modify MONITOR/MWAIT support in |
| 8489 | guest CPUID on writes to MISC_ENABLE if | 8489 | guest CPUID on writes to MISC_ENABLE if |
| 8490 | KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT is | 8490 | KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT is |
| 8491 | disabled. | 8491 | disabled. |
| 8492 | 8492 | ||
| 8493 | KVM_X86_QUIRK_SLOT_ZAP_ALL By default, for KVM_X86_DEFAULT_VM VMs, KVM | 8493 | KVM_X86_QUIRK_SLOT_ZAP_ALL By default, for KVM_X86_DEFAULT_VM VMs, KVM |
| 8494 | invalidates all SPTEs in all memslots and | 8494 | invalidates all SPTEs in all memslots and |
| 8495 | address spaces when a memslot is deleted or | 8495 | address spaces when a memslot is deleted or |
| 8496 | moved. When this quirk is disabled (or the | 8496 | moved. When this quirk is disabled (or the |
| 8497 | VM type isn't KVM_X86_DEFAULT_VM), KVM only | 8497 | VM type isn't KVM_X86_DEFAULT_VM), KVM only |
| 8498 | ensures the backing memory of the deleted | 8498 | ensures the backing memory of the deleted |
| 8499 | or moved memslot isn't reachable, i.e KVM | 8499 | or moved memslot isn't reachable, i.e KVM |
| 8500 | _may_ invalidate only SPTEs related to the | 8500 | _may_ invalidate only SPTEs related to the |
| 8501 | memslot. | 8501 | memslot. |
| 8502 | 8502 | ||
| 8503 | KVM_X86_QUIRK_STUFF_FEATURE_MSRS By default, at vCPU creation, KVM sets the | 8503 | KVM_X86_QUIRK_STUFF_FEATURE_MSRS By default, at vCPU creation, KVM sets the |
| 8504 | vCPU's MSR_IA32_PERF_CAPABILITIES (0x345), | 8504 | vCPU's MSR_IA32_PERF_CAPABILITIES (0x345), |
| 8505 | MSR_IA32_ARCH_CAPABILITIES (0x10a), | 8505 | MSR_IA32_ARCH_CAPABILITIES (0x10a), |
| 8506 | MSR_PLATFORM_INFO (0xce), and all VMX MSRs | 8506 | MSR_PLATFORM_INFO (0xce), and all VMX MSRs |
| 8507 | (0x480..0x492) to the maximal capabilities | 8507 | (0x480..0x492) to the maximal capabilities |
| 8508 | supported by KVM. KVM also sets | 8508 | supported by KVM. KVM also sets |
| 8509 | MSR_IA32_UCODE_REV (0x8b) to an arbitrary | 8509 | MSR_IA32_UCODE_REV (0x8b) to an arbitrary |
| 8510 | value (which is different for Intel vs. | 8510 | value (which is different for Intel vs. |
| 8511 | AMD). Lastly, when guest CPUID is set (by | 8511 | AMD). Lastly, when guest CPUID is set (by |
| 8512 | userspace), KVM modifies select VMX MSR | 8512 | userspace), KVM modifies select VMX MSR |
| 8513 | fields to force consistency between guest | 8513 | fields to force consistency between guest |
| 8514 | CPUID and L2's effective ISA. When this | 8514 | CPUID and L2's effective ISA. When this |
| 8515 | quirk is disabled, KVM zeroes the vCPU's MSR | 8515 | quirk is disabled, KVM zeroes the vCPU's MSR |
| 8516 | values (with two exceptions, see below), | 8516 | values (with two exceptions, see below), |
| 8517 | i.e. treats the feature MSRs like CPUID | 8517 | i.e. treats the feature MSRs like CPUID |
| 8518 | leaves and gives userspace full control of | 8518 | leaves and gives userspace full control of |
| 8519 | the vCPU model definition. This quirk does | 8519 | the vCPU model definition. This quirk does |
| 8520 | not affect VMX MSRs CR0/CR4_FIXED1 (0x487 | 8520 | not affect VMX MSRs CR0/CR4_FIXED1 (0x487 |
| 8521 | and 0x489), as KVM does now allow them to | 8521 | and 0x489), as KVM does now allow them to |
| 8522 | be set by userspace (KVM sets them based on | 8522 | be set by userspace (KVM sets them based on |
| 8523 | guest CPUID, for safety purposes). | 8523 | guest CPUID, for safety purposes). |
| 8524 | 8524 | ||
| 8525 | KVM_X86_QUIRK_IGNORE_GUEST_PAT By default, on Intel platforms, KVM ignores | 8525 | KVM_X86_QUIRK_IGNORE_GUEST_PAT By default, on Intel platforms, KVM ignores |
| 8526 | guest PAT and forces the effective memory | 8526 | guest PAT and forces the effective memory |
| 8527 | type to WB in EPT. The quirk is not available | 8527 | type to WB in EPT. The quirk is not available |
| 8528 | on Intel platforms which are incapable of | 8528 | on Intel platforms which are incapable of |
| 8529 | safely honoring guest PAT (i.e., without CPU | 8529 | safely honoring guest PAT (i.e., without CPU |
| 8530 | self-snoop, KVM always ignores guest PAT and | 8530 | self-snoop, KVM always ignores guest PAT and |
| 8531 | forces effective memory type to WB). It is | 8531 | forces effective memory type to WB). It is |
| 8532 | also ignored on AMD platforms or, on Intel, | 8532 | also ignored on AMD platforms or, on Intel, |
| 8533 | when a VM has non-coherent DMA devices | 8533 | when a VM has non-coherent DMA devices |
| 8534 | assigned; KVM always honors guest PAT in | 8534 | assigned; KVM always honors guest PAT in |
| 8535 | such case. The quirk is needed to avoid | 8535 | such case. The quirk is needed to avoid |
| 8536 | slowdowns on certain Intel Xeon platforms | 8536 | slowdowns on certain Intel Xeon platforms |
| 8537 | (e.g. ICX, SPR) where self-snoop feature is | 8537 | (e.g. ICX, SPR) where self-snoop feature is |
| 8538 | supported but UC is slow enough to cause | 8538 | supported but UC is slow enough to cause |
| 8539 | issues with some older guests that use | 8539 | issues with some older guests that use |
| 8540 | UC instead of WC to map the video RAM. | 8540 | UC instead of WC to map the video RAM. |
| 8541 | Userspace can disable the quirk to honor | 8541 | Userspace can disable the quirk to honor |
| 8542 | guest PAT if it knows that there is no such | 8542 | guest PAT if it knows that there is no such |
| 8543 | guest software, for example if it does not | 8543 | guest software, for example if it does not |
| 8544 | expose a bochs graphics device (which is | 8544 | expose a bochs graphics device (which is |
| 8545 | known to have had a buggy driver). | 8545 | known to have had a buggy driver). |
| 8546 | =================================== ============================================ | 8546 | |
| 8547 | KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM By default, KVM relaxes the consistency | ||
| 8548 | check for GUEST_IA32_DEBUGCTL in vmcs12 | ||
| 8549 | to allow FREEZE_IN_SMM to be set. When | ||
| 8550 | this quirk is disabled, KVM requires this | ||
| 8551 | bit to be cleared. Note that the vmcs02 | ||
| 8552 | bit is still completely controlled by the | ||
| 8553 | host, regardless of the quirk setting. | ||
| 8554 | ======================================== ================================================ | ||
| 8547 | 8555 | ||
| 8548 | 7.32 KVM_CAP_MAX_VCPU_ID | 8556 | 7.32 KVM_CAP_MAX_VCPU_ID |
| 8549 | ------------------------ | 8557 | ------------------------ |
| @@ -17,6 +17,8 @@ The acquisition orders for mutexes are as follows: · 16 unmodified lines | |||
| 17 | 17 | ||
| 18 | - kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock | 18 | - kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock |
| 19 | 19 | ||
| 20 | - vcpu->mutex is taken outside kvm->slots_lock and kvm->slots_arch_lock | ||
| 21 | |||
| 20 | - kvm->slots_lock is taken outside kvm->irq_lock, though acquiring | 22 | - kvm->slots_lock is taken outside kvm->irq_lock, though acquiring |
| 21 | them together is quite rare. | 23 | them together is quite rare. |
| 22 | 24 | ||
| @@ -2195,6 +2195,7 @@ L: dri-devel@lists.freedesktop.org · 2194 unmodified lines | |||
| 2195 | S: Supported | 2195 | S: Supported |
| 2196 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 2196 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| 2197 | F: Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml | 2197 | F: Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml |
| 2198 | F: drivers/gpu/drm/ci/xfails/panthor* | ||
| 2198 | F: drivers/gpu/drm/panthor/ | 2199 | F: drivers/gpu/drm/panthor/ |
| 2199 | F: include/uapi/drm/panthor_drm.h | 2200 | F: include/uapi/drm/panthor_drm.h |
| 2200 | 2201 | ||
| @@ -3986,7 +3987,7 @@ F: drivers/hwmon/asus-ec-sensors.c · 1785 unmodified lines | |||
| 3986 | ASUS NOTEBOOKS AND EEEPC ACPI/WMI EXTRAS DRIVERS | 3987 | ASUS NOTEBOOKS AND EEEPC ACPI/WMI EXTRAS DRIVERS |
| 3987 | M: Corentin Chary <corentin.chary@gmail.com> | 3988 | M: Corentin Chary <corentin.chary@gmail.com> |
| 3988 | M: Luke D. Jones <luke@ljones.dev> | 3989 | M: Luke D. Jones <luke@ljones.dev> |
| 3989 | M: Denis Benato <benato.denis96@gmail.com> | 3990 | M: Denis Benato <denis.benato@linux.dev> |
| 3990 | L: platform-driver-x86@vger.kernel.org | 3991 | L: platform-driver-x86@vger.kernel.org |
| 3991 | S: Maintained | 3992 | S: Maintained |
| 3992 | W: https://asus-linux.org/ | 3993 | W: https://asus-linux.org/ |
| @@ -4022,7 +4023,7 @@ F: drivers/hwmon/asus_wmi_sensors.c · 29 unmodified lines | |||
| 4022 | ASYMMETRIC KEYS | 4023 | ASYMMETRIC KEYS |
| 4023 | M: David Howells <dhowells@redhat.com> | 4024 | M: David Howells <dhowells@redhat.com> |
| 4024 | M: Lukas Wunner <lukas@wunner.de> | 4025 | M: Lukas Wunner <lukas@wunner.de> |
| 4025 | M: Ignat Korchagin <ignat@cloudflare.com> | 4026 | M: Ignat Korchagin <ignat@linux.win> |
| 4026 | L: keyrings@vger.kernel.org | 4027 | L: keyrings@vger.kernel.org |
| 4027 | L: linux-crypto@vger.kernel.org | 4028 | L: linux-crypto@vger.kernel.org |
| 4028 | S: Maintained | 4029 | S: Maintained |
| @@ -4035,7 +4036,7 @@ F: include/linux/verification.h · 6 unmodified lines | |||
| 4035 | 4036 | ||
| 4036 | ASYMMETRIC KEYS - ECDSA | 4037 | ASYMMETRIC KEYS - ECDSA |
| 4037 | M: Lukas Wunner <lukas@wunner.de> | 4038 | M: Lukas Wunner <lukas@wunner.de> |
| 4038 | M: Ignat Korchagin <ignat@cloudflare.com> | 4039 | M: Ignat Korchagin <ignat@linux.win> |
| 4039 | R: Stefan Berger <stefanb@linux.ibm.com> | 4040 | R: Stefan Berger <stefanb@linux.ibm.com> |
| 4040 | L: linux-crypto@vger.kernel.org | 4041 | L: linux-crypto@vger.kernel.org |
| 4041 | S: Maintained | 4042 | S: Maintained |
| @@ -4045,14 +4046,14 @@ F: include/crypto/ecc* · 3 unmodified lines | |||
| 4045 | 4046 | ||
| 4046 | ASYMMETRIC KEYS - GOST | 4047 | ASYMMETRIC KEYS - GOST |
| 4047 | M: Lukas Wunner <lukas@wunner.de> | 4048 | M: Lukas Wunner <lukas@wunner.de> |
| 4048 | M: Ignat Korchagin <ignat@cloudflare.com> | 4049 | M: Ignat Korchagin <ignat@linux.win> |
| 4049 | L: linux-crypto@vger.kernel.org | 4050 | L: linux-crypto@vger.kernel.org |
| 4050 | S: Odd fixes | 4051 | S: Odd fixes |
| 4051 | F: crypto/ecrdsa* | 4052 | F: crypto/ecrdsa* |
| 4052 | 4053 | ||
| 4053 | ASYMMETRIC KEYS - RSA | 4054 | ASYMMETRIC KEYS - RSA |
| 4054 | M: Lukas Wunner <lukas@wunner.de> | 4055 | M: Lukas Wunner <lukas@wunner.de> |
| 4055 | M: Ignat Korchagin <ignat@cloudflare.com> | 4056 | M: Ignat Korchagin <ignat@linux.win> |
| 4056 | L: linux-crypto@vger.kernel.org | 4057 | L: linux-crypto@vger.kernel.org |
| 4057 | S: Maintained | 4058 | S: Maintained |
| 4058 | F: crypto/rsa* | 4059 | F: crypto/rsa* |
| @@ -7534,6 +7535,7 @@ F: include/linux/*fence.h · 3475 unmodified lines | |||
| 7534 | F: include/linux/dma-buf.h | 7535 | F: include/linux/dma-buf.h |
| 7535 | F: include/linux/dma-buf/ | 7536 | F: include/linux/dma-buf/ |
| 7536 | F: include/linux/dma-resv.h | 7537 | F: include/linux/dma-resv.h |
| 7538 | F: rust/helpers/dma-resv.c | ||
| 7537 | K: \bdma_(?:buf|fence|resv)\b | 7539 | K: \bdma_(?:buf|fence|resv)\b |
| 7538 | 7540 | ||
| 7539 | DMA GENERIC OFFLOAD ENGINE SUBSYSTEM | 7541 | DMA GENERIC OFFLOAD ENGINE SUBSYSTEM |
| @@ -7998,7 +8000,9 @@ F: Documentation/devicetree/bindings/display/himax,hx8357.yaml · 458 unmodified lines | |||
| 7998 | F: drivers/gpu/drm/tiny/hx8357d.c | 8000 | F: drivers/gpu/drm/tiny/hx8357d.c |
| 7999 | 8001 | ||
| 8000 | DRM DRIVER FOR HYPERV SYNTHETIC VIDEO DEVICE | 8002 | DRM DRIVER FOR HYPERV SYNTHETIC VIDEO DEVICE |
| 8001 | M: Deepak Rawat <drawat.floss@gmail.com> | 8003 | M: Dexuan Cui <decui@microsoft.com> |
| 8004 | M: Long Li <longli@microsoft.com> | ||
| 8005 | M: Saurabh Sengar <ssengar@linux.microsoft.com> | ||
| 8002 | L: linux-hyperv@vger.kernel.org | 8006 | L: linux-hyperv@vger.kernel.org |
| 8003 | L: dri-devel@lists.freedesktop.org | 8007 | L: dri-devel@lists.freedesktop.org |
| 8004 | S: Maintained | 8008 | S: Maintained |
| @@ -8027,8 +8031,9 @@ F: drivers/gpu/drm/panel/panel-ilitek-ili9805.c · 22 unmodified lines | |||
| 8027 | 8031 | ||
| 8028 | DRM DRIVER FOR ILITEK ILI9806E PANELS | 8032 | DRM DRIVER FOR ILITEK ILI9806E PANELS |
| 8029 | M: Michael Walle <mwalle@kernel.org> | 8033 | M: Michael Walle <mwalle@kernel.org> |
| 8034 | M: Dario Binacchi <dario.binacchi@amarulasolutions.com> | ||
| 8030 | S: Maintained | 8035 | S: Maintained |
| 8031 | F: drivers/gpu/drm/panel/panel-ilitek-ili9806e.c | 8036 | F: drivers/gpu/drm/panel/panel-ilitek-ili9806e-* |
| 8032 | 8037 | ||
| 8033 | DRM DRIVER FOR JADARD JD9365DA-H3 MIPI-DSI LCD PANELS | 8038 | DRM DRIVER FOR JADARD JD9365DA-H3 MIPI-DSI LCD PANELS |
| 8034 | M: Jagan Teki <jagan@edgeble.ai> | 8039 | M: Jagan Teki <jagan@edgeble.ai> |
| @@ -8301,6 +8306,11 @@ S: Maintained · 266 unmodified lines | |||
| 8301 | F: Documentation/devicetree/bindings/display/panel/samsung,s6e3ha8.yaml | 8306 | F: Documentation/devicetree/bindings/display/panel/samsung,s6e3ha8.yaml |
| 8302 | F: drivers/gpu/drm/panel/panel-samsung-s6e3ha8.c | 8307 | F: drivers/gpu/drm/panel/panel-samsung-s6e3ha8.c |
| 8303 | 8308 | ||
| 8309 | DRM DRIVER FOR SAMSUNG S6E8FC0 PANELS | ||
| 8310 | M: Yedaya Katsman <yedaya.ka@gmail.com> | ||
| 8311 | S: Maintained | ||
| 8312 | F: drivers/gpu/drm/panel/panel-samsung-s6e8fc0-m1906f9.c | ||
| 8313 | |||
| 8304 | DRM DRIVER FOR SAMSUNG SOFEF00 DDIC | 8314 | DRM DRIVER FOR SAMSUNG SOFEF00 DDIC |
| 8305 | M: David Heidelberg <david@ixit.cz> | 8315 | M: David Heidelberg <david@ixit.cz> |
| 8306 | M: Casey Connolly <casey.connolly@linaro.org> | 8316 | M: Casey Connolly <casey.connolly@linaro.org> |
| @@ -8505,7 +8515,10 @@ T: git https://gitlab.freedesktop.org/drm/rust/kernel.git · 198 unmodified lines | |||
| 8505 | F: drivers/gpu/drm/nova/ | 8515 | F: drivers/gpu/drm/nova/ |
| 8506 | F: drivers/gpu/drm/tyr/ | 8516 | F: drivers/gpu/drm/tyr/ |
| 8507 | F: drivers/gpu/nova-core/ | 8517 | F: drivers/gpu/nova-core/ |
| 8518 | F: rust/helpers/gpu.c | ||
| 8508 | F: rust/kernel/drm/ | 8519 | F: rust/kernel/drm/ |
| 8520 | F: rust/kernel/gpu.rs | ||
| 8521 | F: rust/kernel/gpu/ | ||
| 8509 | 8522 | ||
| 8510 | DRM DRIVERS FOR ALLWINNER A10 | 8523 | DRM DRIVERS FOR ALLWINNER A10 |
| 8511 | M: Chen-Yu Tsai <wens@kernel.org> | 8524 | M: Chen-Yu Tsai <wens@kernel.org> |
| @@ -8632,9 +8645,14 @@ F: drivers/gpu/drm/lima/ · 120 unmodified lines | |||
| 8632 | F: include/uapi/drm/lima_drm.h | 8645 | F: include/uapi/drm/lima_drm.h |
| 8633 | 8646 | ||
| 8634 | DRM DRIVERS FOR LOONGSON | 8647 | DRM DRIVERS FOR LOONGSON |
| 8635 | M: Sui Jingfeng <suijingfeng@loongson.cn> | 8648 | M: Jianmin Lv <lvjianmin@loongson.cn> |
| 8649 | M: Qianhai Wu <wuqianhai@loongson.cn> | ||
| 8650 | R: Huacai Chen <chenhuacai@kernel.org> | ||
| 8651 | R: Mingcong Bai <jeffbai@aosc.io> | ||
| 8652 | R: Xi Ruoyao <xry111@xry111.site> | ||
| 8653 | R: Icenowy Zheng <zhengxingda@iscas.ac.cn> | ||
| 8636 | L: dri-devel@lists.freedesktop.org | 8654 | L: dri-devel@lists.freedesktop.org |
| 8637 | S: Supported | 8655 | S: Maintained |
| 8638 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 8656 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| 8639 | F: drivers/gpu/drm/loongson/ | 8657 | F: drivers/gpu/drm/loongson/ |
| 8640 | 8658 | ||
| @@ -8918,7 +8936,7 @@ F: include/drm/ttm/ · 277 unmodified lines | |||
| 8918 | GPU BUDDY ALLOCATOR | 8936 | GPU BUDDY ALLOCATOR |
| 8919 | M: Matthew Auld <matthew.auld@intel.com> | 8937 | M: Matthew Auld <matthew.auld@intel.com> |
| 8920 | M: Arun Pravin <arunpravin.paneerselvam@amd.com> | 8938 | M: Arun Pravin <arunpravin.paneerselvam@amd.com> |
| 8921 | R: Christian Koenig <christian.koenig@amd.com> | 8939 | R: Joel Fernandes <joelagnelf@nvidia.com> |
| 8922 | L: dri-devel@lists.freedesktop.org | 8940 | L: dri-devel@lists.freedesktop.org |
| 8923 | S: Maintained | 8941 | S: Maintained |
| 8924 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 8942 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| @@ -8927,6 +8945,9 @@ F: drivers/gpu/drm/drm_buddy.c · 2 unmodified lines | |||
| 8927 | F: drivers/gpu/tests/gpu_buddy_test.c | 8945 | F: drivers/gpu/tests/gpu_buddy_test.c |
| 8928 | F: include/drm/drm_buddy.h | 8946 | F: include/drm/drm_buddy.h |
| 8929 | F: include/linux/gpu_buddy.h | 8947 | F: include/linux/gpu_buddy.h |
| 8948 | F: rust/helpers/gpu.c | ||
| 8949 | F: rust/kernel/gpu.rs | ||
| 8950 | F: rust/kernel/gpu/ | ||
| 8930 | 8951 | ||
| 8931 | DRM AUTOMATED TESTING | 8952 | DRM AUTOMATED TESTING |
| 8932 | M: Helen Koike <helen.fornazier@gmail.com> | 8953 | M: Helen Koike <helen.fornazier@gmail.com> |
| @@ -9628,7 +9649,12 @@ F: include/linux/ext2* · 695 unmodified lines | |||
| 9628 | 9649 | ||
| 9629 | EXT4 FILE SYSTEM | 9650 | EXT4 FILE SYSTEM |
| 9630 | M: "Theodore Ts'o" <tytso@mit.edu> | 9651 | M: "Theodore Ts'o" <tytso@mit.edu> |
| 9631 | M: Andreas Dilger <adilger.kernel@dilger.ca> | 9652 | R: Andreas Dilger <adilger.kernel@dilger.ca> |
| 9653 | R: Baokun Li <libaokun@linux.alibaba.com> | ||
| 9654 | R: Jan Kara <jack@suse.cz> | ||
| 9655 | R: Ojaswin Mujoo <ojaswin@linux.ibm.com> | ||
| 9656 | R: Ritesh Harjani (IBM) <ritesh.list@gmail.com> | ||
| 9657 | R: Zhang Yi <yi.zhang@huawei.com> | ||
| 9632 | L: linux-ext4@vger.kernel.org | 9658 | L: linux-ext4@vger.kernel.org |
| 9633 | S: Maintained | 9659 | S: Maintained |
| 9634 | W: http://ext4.wiki.kernel.org | 9660 | W: http://ext4.wiki.kernel.org |
| @@ -12025,7 +12051,6 @@ I2C SUBSYSTEM · 2390 unmodified lines | |||
| 12025 | M: Wolfram Sang <wsa+renesas@sang-engineering.com> | 12051 | M: Wolfram Sang <wsa+renesas@sang-engineering.com> |
| 12026 | L: linux-i2c@vger.kernel.org | 12052 | L: linux-i2c@vger.kernel.org |
| 12027 | S: Maintained | 12053 | S: Maintained |
| 12028 | W: https://i2c.wiki.kernel.org/ | ||
| 12029 | Q: https://patchwork.ozlabs.org/project/linux-i2c/list/ | 12054 | Q: https://patchwork.ozlabs.org/project/linux-i2c/list/ |
| 12030 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux.git | 12055 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux.git |
| 12031 | F: Documentation/i2c/ | 12056 | F: Documentation/i2c/ |
| @@ -12051,7 +12076,6 @@ I2C SUBSYSTEM HOST DRIVERS · 19 unmodified lines | |||
| 12051 | M: Andi Shyti <andi.shyti@kernel.org> | 12076 | M: Andi Shyti <andi.shyti@kernel.org> |
| 12052 | L: linux-i2c@vger.kernel.org | 12077 | L: linux-i2c@vger.kernel.org |
| 12053 | S: Maintained | 12078 | S: Maintained |
| 12054 | W: https://i2c.wiki.kernel.org/ | ||
| 12055 | Q: https://patchwork.ozlabs.org/project/linux-i2c/list/ | 12079 | Q: https://patchwork.ozlabs.org/project/linux-i2c/list/ |
| 12056 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/andi.shyti/linux.git | 12080 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/andi.shyti/linux.git |
| 12057 | F: Documentation/devicetree/bindings/i2c/ | 12081 | F: Documentation/devicetree/bindings/i2c/ |
| @@ -16374,7 +16398,6 @@ F: net/dsa/tag_mtk.c · 4316 unmodified lines | |||
| 16374 | 16398 | ||
| 16375 | MEDIATEK T7XX 5G WWAN MODEM DRIVER | 16399 | MEDIATEK T7XX 5G WWAN MODEM DRIVER |
| 16376 | M: Chandrashekar Devegowda <chandrashekar.devegowda@intel.com> | 16400 | M: Chandrashekar Devegowda <chandrashekar.devegowda@intel.com> |
| 16377 | R: Chiranjeevi Rapolu <chiranjeevi.rapolu@linux.intel.com> | ||
| 16378 | R: Liu Haijun <haijun.liu@mediatek.com> | 16401 | R: Liu Haijun <haijun.liu@mediatek.com> |
| 16379 | R: Ricardo Martinez <ricardo.martinez@linux.intel.com> | 16402 | R: Ricardo Martinez <ricardo.martinez@linux.intel.com> |
| 16380 | L: netdev@vger.kernel.org | 16403 | L: netdev@vger.kernel.org |
| @@ -16659,7 +16682,7 @@ F: mm/balloon.c · 278 unmodified lines | |||
| 16659 | MEMORY MANAGEMENT - CORE | 16682 | MEMORY MANAGEMENT - CORE |
| 16660 | M: Andrew Morton <akpm@linux-foundation.org> | 16683 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16661 | M: David Hildenbrand <david@kernel.org> | 16684 | M: David Hildenbrand <david@kernel.org> |
| 16662 | R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 16685 | R: Lorenzo Stoakes <ljs@kernel.org> |
| 16663 | R: Liam R. Howlett <Liam.Howlett@oracle.com> | 16686 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16664 | R: Vlastimil Babka <vbabka@kernel.org> | 16687 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16665 | R: Mike Rapoport <rppt@kernel.org> | 16688 | R: Mike Rapoport <rppt@kernel.org> |
| @@ -16789,7 +16812,7 @@ F: mm/workingset.c · 123 unmodified lines | |||
| 16789 | MEMORY MANAGEMENT - MISC | 16812 | MEMORY MANAGEMENT - MISC |
| 16790 | M: Andrew Morton <akpm@linux-foundation.org> | 16813 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16791 | M: David Hildenbrand <david@kernel.org> | 16814 | M: David Hildenbrand <david@kernel.org> |
| 16792 | R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 16815 | R: Lorenzo Stoakes <ljs@kernel.org> |
| 16793 | R: Liam R. Howlett <Liam.Howlett@oracle.com> | 16816 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16794 | R: Vlastimil Babka <vbabka@kernel.org> | 16817 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16795 | R: Mike Rapoport <rppt@kernel.org> | 16818 | R: Mike Rapoport <rppt@kernel.org> |
| @@ -16880,7 +16903,7 @@ R: David Hildenbrand <david@kernel.org> · 84 unmodified lines | |||
| 16880 | R: Michal Hocko <mhocko@kernel.org> | 16903 | R: Michal Hocko <mhocko@kernel.org> |
| 16881 | R: Qi Zheng <zhengqi.arch@bytedance.com> | 16904 | R: Qi Zheng <zhengqi.arch@bytedance.com> |
| 16882 | R: Shakeel Butt <shakeel.butt@linux.dev> | 16905 | R: Shakeel Butt <shakeel.butt@linux.dev> |
| 16883 | R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 16906 | R: Lorenzo Stoakes <ljs@kernel.org> |
| 16884 | L: linux-mm@kvack.org | 16907 | L: linux-mm@kvack.org |
| 16885 | S: Maintained | 16908 | S: Maintained |
| 16886 | F: mm/vmscan.c | 16909 | F: mm/vmscan.c |
| @@ -16889,11 +16912,11 @@ F: mm/workingset.c · 2 unmodified lines | |||
| 16889 | MEMORY MANAGEMENT - RMAP (REVERSE MAPPING) | 16912 | MEMORY MANAGEMENT - RMAP (REVERSE MAPPING) |
| 16890 | M: Andrew Morton <akpm@linux-foundation.org> | 16913 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16891 | M: David Hildenbrand <david@kernel.org> | 16914 | M: David Hildenbrand <david@kernel.org> |
| 16892 | M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 16915 | M: Lorenzo Stoakes <ljs@kernel.org> |
| 16893 | R: Rik van Riel <riel@surriel.com> | 16916 | R: Rik van Riel <riel@surriel.com> |
| 16894 | R: Liam R. Howlett <Liam.Howlett@oracle.com> | 16917 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16895 | R: Vlastimil Babka <vbabka@kernel.org> | 16918 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16896 | R: Harry Yoo <harry.yoo@oracle.com> | 16919 | R: Harry Yoo <harry@kernel.org> |
| 16897 | R: Jann Horn <jannh@google.com> | 16920 | R: Jann Horn <jannh@google.com> |
| 16898 | L: linux-mm@kvack.org | 16921 | L: linux-mm@kvack.org |
| 16899 | S: Maintained | 16922 | S: Maintained |
| @@ -16934,7 +16957,7 @@ F: mm/swapfile.c · 34 unmodified lines | |||
| 16934 | MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE) | 16957 | MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE) |
| 16935 | M: Andrew Morton <akpm@linux-foundation.org> | 16958 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16936 | M: David Hildenbrand <david@kernel.org> | 16959 | M: David Hildenbrand <david@kernel.org> |
| 16937 | M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 16960 | M: Lorenzo Stoakes <ljs@kernel.org> |
| 16938 | R: Zi Yan <ziy@nvidia.com> | 16961 | R: Zi Yan <ziy@nvidia.com> |
| 16939 | R: Baolin Wang <baolin.wang@linux.alibaba.com> | 16962 | R: Baolin Wang <baolin.wang@linux.alibaba.com> |
| 16940 | R: Liam R. Howlett <Liam.Howlett@oracle.com> | 16963 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| @@ -16974,7 +16997,7 @@ F: tools/testing/selftests/mm/uffd-*.[ch] · 33 unmodified lines | |||
| 16974 | 16997 | ||
| 16975 | MEMORY MANAGEMENT - RUST | 16998 | MEMORY MANAGEMENT - RUST |
| 16976 | M: Alice Ryhl <aliceryhl@google.com> | 16999 | M: Alice Ryhl <aliceryhl@google.com> |
| 16977 | R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 17000 | R: Lorenzo Stoakes <ljs@kernel.org> |
| 16978 | R: Liam R. Howlett <Liam.Howlett@oracle.com> | 17001 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16979 | L: linux-mm@kvack.org | 17002 | L: linux-mm@kvack.org |
| 16980 | L: rust-for-linux@vger.kernel.org | 17003 | L: rust-for-linux@vger.kernel.org |
| @@ -16990,7 +17013,7 @@ F: rust/kernel/page.rs · 9 unmodified lines | |||
| 16990 | MEMORY MAPPING | 17013 | MEMORY MAPPING |
| 16991 | M: Andrew Morton <akpm@linux-foundation.org> | 17014 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16992 | M: Liam R. Howlett <Liam.Howlett@oracle.com> | 17015 | M: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 16993 | M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 17016 | M: Lorenzo Stoakes <ljs@kernel.org> |
| 16994 | R: Vlastimil Babka <vbabka@kernel.org> | 17017 | R: Vlastimil Babka <vbabka@kernel.org> |
| 16995 | R: Jann Horn <jannh@google.com> | 17018 | R: Jann Horn <jannh@google.com> |
| 16996 | R: Pedro Falcato <pfalcato@suse.de> | 17019 | R: Pedro Falcato <pfalcato@suse.de> |
| @@ -17020,7 +17043,7 @@ MEMORY MAPPING - LOCKING · 23 unmodified lines | |||
| 17020 | M: Andrew Morton <akpm@linux-foundation.org> | 17043 | M: Andrew Morton <akpm@linux-foundation.org> |
| 17021 | M: Suren Baghdasaryan <surenb@google.com> | 17044 | M: Suren Baghdasaryan <surenb@google.com> |
| 17022 | M: Liam R. Howlett <Liam.Howlett@oracle.com> | 17045 | M: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 17023 | M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 17046 | M: Lorenzo Stoakes <ljs@kernel.org> |
| 17024 | R: Vlastimil Babka <vbabka@kernel.org> | 17047 | R: Vlastimil Babka <vbabka@kernel.org> |
| 17025 | R: Shakeel Butt <shakeel.butt@linux.dev> | 17048 | R: Shakeel Butt <shakeel.butt@linux.dev> |
| 17026 | L: linux-mm@kvack.org | 17049 | L: linux-mm@kvack.org |
| @@ -17035,7 +17058,7 @@ F: mm/mmap_lock.c · 8 unmodified lines | |||
| 17035 | MEMORY MAPPING - MADVISE (MEMORY ADVICE) | 17058 | MEMORY MAPPING - MADVISE (MEMORY ADVICE) |
| 17036 | M: Andrew Morton <akpm@linux-foundation.org> | 17059 | M: Andrew Morton <akpm@linux-foundation.org> |
| 17037 | M: Liam R. Howlett <Liam.Howlett@oracle.com> | 17060 | M: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 17038 | M: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 17061 | M: Lorenzo Stoakes <ljs@kernel.org> |
| 17039 | M: David Hildenbrand <david@kernel.org> | 17062 | M: David Hildenbrand <david@kernel.org> |
| 17040 | R: Vlastimil Babka <vbabka@kernel.org> | 17063 | R: Vlastimil Babka <vbabka@kernel.org> |
| 17041 | R: Jann Horn <jannh@google.com> | 17064 | R: Jann Horn <jannh@google.com> |
| @@ -21954,7 +21977,7 @@ F: drivers/media/radio/radio-tea5777.c · 4912 unmodified lines | |||
| 21954 | 21977 | ||
| 21955 | RADOS BLOCK DEVICE (RBD) | 21978 | RADOS BLOCK DEVICE (RBD) |
| 21956 | M: Ilya Dryomov <idryomov@gmail.com> | 21979 | M: Ilya Dryomov <idryomov@gmail.com> |
| 21957 | R: Dongsheng Yang <dongsheng.yang@easystack.cn> | 21980 | R: Dongsheng Yang <dongsheng.yang@linux.dev> |
| 21958 | L: ceph-devel@vger.kernel.org | 21981 | L: ceph-devel@vger.kernel.org |
| 21959 | S: Supported | 21982 | S: Supported |
| 21960 | W: http://ceph.com/ | 21983 | W: http://ceph.com/ |
| @@ -22283,6 +22306,16 @@ L: linux-wireless@vger.kernel.org · 322 unmodified lines | |||
| 22283 | S: Orphan | 22306 | S: Orphan |
| 22284 | F: drivers/net/wireless/rsi/ | 22307 | F: drivers/net/wireless/rsi/ |
| 22285 | 22308 | ||
| 22309 | RELAY | ||
| 22310 | M: Andrew Morton <akpm@linux-foundation.org> | ||
| 22311 | M: Jens Axboe <axboe@kernel.dk> | ||
| 22312 | M: Jason Xing <kernelxing@tencent.com> | ||
| 22313 | L: linux-kernel@vger.kernel.org | ||
| 22314 | S: Maintained | ||
| 22315 | F: Documentation/filesystems/relay.rst | ||
| 22316 | F: include/linux/relay.h | ||
| 22317 | F: kernel/relay.c | ||
| 22318 | |||
| 22286 | REGISTER MAP ABSTRACTION | 22319 | REGISTER MAP ABSTRACTION |
| 22287 | M: Mark Brown <broonie@kernel.org> | 22320 | M: Mark Brown <broonie@kernel.org> |
| 22288 | L: linux-kernel@vger.kernel.org | 22321 | L: linux-kernel@vger.kernel.org |
| @@ -23173,7 +23206,7 @@ K: \b(?i:rust)\b · 884 unmodified lines | |||
| 23173 | 23206 | ||
| 23174 | RUST [ALLOC] | 23207 | RUST [ALLOC] |
| 23175 | M: Danilo Krummrich <dakr@kernel.org> | 23208 | M: Danilo Krummrich <dakr@kernel.org> |
| 23176 | R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> | 23209 | R: Lorenzo Stoakes <ljs@kernel.org> |
| 23177 | R: Vlastimil Babka <vbabka@kernel.org> | 23210 | R: Vlastimil Babka <vbabka@kernel.org> |
| 23178 | R: Liam R. Howlett <Liam.Howlett@oracle.com> | 23211 | R: Liam R. Howlett <Liam.Howlett@oracle.com> |
| 23179 | R: Uladzislau Rezki <urezki@gmail.com> | 23212 | R: Uladzislau Rezki <urezki@gmail.com> |
| @@ -23183,6 +23216,15 @@ T: git https://github.com/Rust-for-Linux/linux.git alloc-next · 3 unmodified lines | |||
| 23183 | F: rust/kernel/alloc.rs | 23216 | F: rust/kernel/alloc.rs |
| 23184 | F: rust/kernel/alloc/ | 23217 | F: rust/kernel/alloc/ |
| 23185 | 23218 | ||
| 23219 | RUST [INTEROP] | ||
| 23220 | M: Joel Fernandes <joelagnelf@nvidia.com> | ||
| 23221 | M: Alexandre Courbot <acourbot@nvidia.com> | ||
| 23222 | L: rust-for-linux@vger.kernel.org | ||
| 23223 | S: Maintained | ||
| 23224 | T: git https://github.com/Rust-for-Linux/linux.git interop-next | ||
| 23225 | F: rust/kernel/interop.rs | ||
| 23226 | F: rust/kernel/interop/ | ||
| 23227 | |||
| 23186 | RUST [NUM] | 23228 | RUST [NUM] |
| 23187 | M: Alexandre Courbot <acourbot@nvidia.com> | 23229 | M: Alexandre Courbot <acourbot@nvidia.com> |
| 23188 | R: Yury Norov <yury.norov@gmail.com> | 23230 | R: Yury Norov <yury.norov@gmail.com> |
| @@ -24350,11 +24392,12 @@ F: drivers/nvmem/layouts/sl28vpd.c · 1161 unmodified lines | |||
| 24350 | 24392 | ||
| 24351 | SLAB ALLOCATOR | 24393 | SLAB ALLOCATOR |
| 24352 | M: Vlastimil Babka <vbabka@kernel.org> | 24394 | M: Vlastimil Babka <vbabka@kernel.org> |
| 24395 | M: Harry Yoo <harry@kernel.org> | ||
| 24353 | M: Andrew Morton <akpm@linux-foundation.org> | 24396 | M: Andrew Morton <akpm@linux-foundation.org> |
| 24397 | R: Hao Li <hao.li@linux.dev> | ||
| 24354 | R: Christoph Lameter <cl@gentwo.org> | 24398 | R: Christoph Lameter <cl@gentwo.org> |
| 24355 | R: David Rientjes <rientjes@google.com> | 24399 | R: David Rientjes <rientjes@google.com> |
| 24356 | R: Roman Gushchin <roman.gushchin@linux.dev> | 24400 | R: Roman Gushchin <roman.gushchin@linux.dev> |
| 24357 | R: Harry Yoo <harry.yoo@oracle.com> | ||
| 24358 | L: linux-mm@kvack.org | 24401 | L: linux-mm@kvack.org |
| 24359 | S: Maintained | 24402 | S: Maintained |
| 24360 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git | 24403 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git |
| @@ -24908,9 +24951,9 @@ F: drivers/clk/spear/ · 547 unmodified lines | |||
| 24908 | F: drivers/pinctrl/spear/ | 24951 | F: drivers/pinctrl/spear/ |
| 24909 | 24952 | ||
| 24910 | SPI NOR SUBSYSTEM | 24953 | SPI NOR SUBSYSTEM |
| 24911 | M: Tudor Ambarus <tudor.ambarus@linaro.org> | ||
| 24912 | M: Pratyush Yadav <pratyush@kernel.org> | 24954 | M: Pratyush Yadav <pratyush@kernel.org> |
| 24913 | M: Michael Walle <mwalle@kernel.org> | 24955 | M: Michael Walle <mwalle@kernel.org> |
| 24956 | R: Takahiro Kuwano <takahiro.kuwano@infineon.com> | ||
| 24914 | L: linux-mtd@lists.infradead.org | 24957 | L: linux-mtd@lists.infradead.org |
| 24915 | S: Maintained | 24958 | S: Maintained |
| 24916 | W: http://www.linux-mtd.infradead.org/ | 24959 | W: http://www.linux-mtd.infradead.org/ |
| @@ -25765,6 +25808,7 @@ F: include/net/pkt_cls.h · 848 unmodified lines | |||
| 25765 | F: include/net/pkt_sched.h | 25808 | F: include/net/pkt_sched.h |
| 25766 | F: include/net/sch_priv.h | 25809 | F: include/net/sch_priv.h |
| 25767 | F: include/net/tc_act/ | 25810 | F: include/net/tc_act/ |
| 25811 | F: include/net/tc_wrapper.h | ||
| 25768 | F: include/uapi/linux/pkt_cls.h | 25812 | F: include/uapi/linux/pkt_cls.h |
| 25769 | F: include/uapi/linux/pkt_sched.h | 25813 | F: include/uapi/linux/pkt_sched.h |
| 25770 | F: include/uapi/linux/tc_act/ | 25814 | F: include/uapi/linux/tc_act/ |
| @@ -2,7 +2,7 @@ · 1 unmodified line | |||
| 2 | VERSION = 7 | 2 | VERSION = 7 |
| 3 | PATCHLEVEL = 0 | 3 | PATCHLEVEL = 0 |
| 4 | SUBLEVEL = 0 | 4 | SUBLEVEL = 0 |
| 5 | EXTRAVERSION = -rc3 | 5 | EXTRAVERSION = -rc7 |
| 6 | NAME = Baby Opossum Posse | 6 | NAME = Baby Opossum Posse |
| 7 | 7 | ||
| 8 | # *DOCUMENTATION* | 8 | # *DOCUMENTATION* |
| @@ -476,6 +476,7 @@ KBUILD_USERLDFLAGS := $(USERLDFLAGS) · 467 unmodified lines | |||
| 476 | export rust_common_flags := --edition=2021 \ | 476 | export rust_common_flags := --edition=2021 \ |
| 477 | -Zbinary_dep_depinfo=y \ | 477 | -Zbinary_dep_depinfo=y \ |
| 478 | -Astable_features \ | 478 | -Astable_features \ |
| 479 | -Aunused_features \ | ||
| 479 | -Dnon_ascii_idents \ | 480 | -Dnon_ascii_idents \ |
| 480 | -Dunsafe_op_in_unsafe_fn \ | 481 | -Dunsafe_op_in_unsafe_fn \ |
| 481 | -Wmissing_docs \ | 482 | -Wmissing_docs \ |
| @@ -1113,6 +1114,9 @@ KBUILD_CFLAGS += -fno-builtin-wcslen · 631 unmodified lines | |||
| 1113 | # change __FILE__ to the relative path to the source directory | 1114 | # change __FILE__ to the relative path to the source directory |
| 1114 | ifdef building_out_of_srctree | 1115 | ifdef building_out_of_srctree |
| 1115 | KBUILD_CPPFLAGS += -fmacro-prefix-map=$(srcroot)/= | 1116 | KBUILD_CPPFLAGS += -fmacro-prefix-map=$(srcroot)/= |
| 1117 | ifeq ($(call rustc-option-yn, --remap-path-scope=macro),y) | ||
| 1118 | KBUILD_RUSTFLAGS += --remap-path-prefix=$(srcroot)/= --remap-path-scope=macro | ||
| 1119 | endif | ||
| 1116 | endif | 1120 | endif |
| 1117 | 1121 | ||
| 1118 | # include additional Makefiles when needed | 1122 | # include additional Makefiles when needed |
| @@ -1650,7 +1654,7 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ · 531 unmodified lines | |||
| 1650 | modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ | 1654 | modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ |
| 1651 | compile_commands.json rust/test \ | 1655 | compile_commands.json rust/test \ |
| 1652 | rust-project.json .vmlinux.objs .vmlinux.export.c \ | 1656 | rust-project.json .vmlinux.objs .vmlinux.export.c \ |
| 1653 | .builtin-dtbs-list .builtin-dtb.S | 1657 | .builtin-dtbs-list .builtin-dtbs.S |
| 1654 | 1658 | ||
| 1655 | # Directories & files removed with 'make mrproper' | 1659 | # Directories & files removed with 'make mrproper' |
| 1656 | MRPROPER_FILES += include/config include/generated \ | 1660 | MRPROPER_FILES += include/config include/generated \ |
| @@ -279,7 +279,6 @@ CONFIG_TI_CPSW_SWITCHDEV=y · 278 unmodified lines | |||
| 279 | CONFIG_TI_CPTS=y | 279 | CONFIG_TI_CPTS=y |
| 280 | CONFIG_TI_KEYSTONE_NETCP=y | 280 | CONFIG_TI_KEYSTONE_NETCP=y |
| 281 | CONFIG_TI_KEYSTONE_NETCP_ETHSS=y | 281 | CONFIG_TI_KEYSTONE_NETCP_ETHSS=y |
| 282 | CONFIG_TI_PRUSS=m | ||
| 283 | CONFIG_TI_PRUETH=m | 282 | CONFIG_TI_PRUETH=m |
| 284 | CONFIG_XILINX_EMACLITE=y | 283 | CONFIG_XILINX_EMACLITE=y |
| 285 | CONFIG_SFP=m | 284 | CONFIG_SFP=m |
| @@ -252,6 +252,7 @@ config ARM64 · 251 unmodified lines | |||
| 252 | select HAVE_RSEQ | 252 | select HAVE_RSEQ |
| 253 | select HAVE_RUST if RUSTC_SUPPORTS_ARM64 | 253 | select HAVE_RUST if RUSTC_SUPPORTS_ARM64 |
| 254 | select HAVE_STACKPROTECTOR | 254 | select HAVE_STACKPROTECTOR |
| 255 | select HAVE_STATIC_CALL if CFI | ||
| 255 | select HAVE_SYSCALL_TRACEPOINTS | 256 | select HAVE_SYSCALL_TRACEPOINTS |
| 256 | select HAVE_KPROBES | 257 | select HAVE_KPROBES |
| 257 | select HAVE_KRETPROBES | 258 | select HAVE_KRETPROBES |
| @@ -698,7 +698,7 @@ · 697 unmodified lines | |||
| 698 | compatible = "renesas,scif-r8a78000", | 698 | compatible = "renesas,scif-r8a78000", |
| 699 | "renesas,rcar-gen5-scif", "renesas,scif"; | 699 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 700 | reg = <0 0xc0700000 0 0x40>; | 700 | reg = <0 0xc0700000 0 0x40>; |
| 701 | interrupts = <GIC_SPI 4074 IRQ_TYPE_LEVEL_HIGH>; | 701 | interrupts = <GIC_ESPI 10 IRQ_TYPE_LEVEL_HIGH>; |
| 702 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; | 702 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 703 | clock-names = "fck", "brg_int", "scif_clk"; | 703 | clock-names = "fck", "brg_int", "scif_clk"; |
| 704 | status = "disabled"; | 704 | status = "disabled"; |
| @@ -708,7 +708,7 @@ · 3 unmodified lines | |||
| 708 | compatible = "renesas,scif-r8a78000", | 708 | compatible = "renesas,scif-r8a78000", |
| 709 | "renesas,rcar-gen5-scif", "renesas,scif"; | 709 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 710 | reg = <0 0xc0704000 0 0x40>; | 710 | reg = <0 0xc0704000 0 0x40>; |
| 711 | interrupts = <GIC_SPI 4075 IRQ_TYPE_LEVEL_HIGH>; | 711 | interrupts = <GIC_ESPI 11 IRQ_TYPE_LEVEL_HIGH>; |
| 712 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; | 712 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 713 | clock-names = "fck", "brg_int", "scif_clk"; | 713 | clock-names = "fck", "brg_int", "scif_clk"; |
| 714 | status = "disabled"; | 714 | status = "disabled"; |
| @@ -718,7 +718,7 @@ · 3 unmodified lines | |||
| 718 | compatible = "renesas,scif-r8a78000", | 718 | compatible = "renesas,scif-r8a78000", |
| 719 | "renesas,rcar-gen5-scif", "renesas,scif"; | 719 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 720 | reg = <0 0xc0708000 0 0x40>; | 720 | reg = <0 0xc0708000 0 0x40>; |
| 721 | interrupts = <GIC_SPI 4076 IRQ_TYPE_LEVEL_HIGH>; | 721 | interrupts = <GIC_ESPI 12 IRQ_TYPE_LEVEL_HIGH>; |
| 722 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; | 722 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 723 | clock-names = "fck", "brg_int", "scif_clk"; | 723 | clock-names = "fck", "brg_int", "scif_clk"; |
| 724 | status = "disabled"; | 724 | status = "disabled"; |
| @@ -728,7 +728,7 @@ · 3 unmodified lines | |||
| 728 | compatible = "renesas,scif-r8a78000", | 728 | compatible = "renesas,scif-r8a78000", |
| 729 | "renesas,rcar-gen5-scif", "renesas,scif"; | 729 | "renesas,rcar-gen5-scif", "renesas,scif"; |
| 730 | reg = <0 0xc070c000 0 0x40>; | 730 | reg = <0 0xc070c000 0 0x40>; |
| 731 | interrupts = <GIC_SPI 4077 IRQ_TYPE_LEVEL_HIGH>; | 731 | interrupts = <GIC_ESPI 13 IRQ_TYPE_LEVEL_HIGH>; |
| 732 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; | 732 | clocks = <&dummy_clk_sgasyncd16>, <&dummy_clk_sgasyncd16>, <&scif_clk>; |
| 733 | clock-names = "fck", "brg_int", "scif_clk"; | 733 | clock-names = "fck", "brg_int", "scif_clk"; |
| 734 | status = "disabled"; | 734 | status = "disabled"; |
| @@ -738,7 +738,7 @@ · 3 unmodified lines | |||
| 738 | compatible = "renesas,hscif-r8a78000", | 738 | compatible = "renesas,hscif-r8a78000", |
| 739 | "renesas,rcar-gen5-hscif", "renesas,hscif"; | 739 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 740 | reg = <0 0xc0710000 0 0x60>; | 740 | reg = <0 0xc0710000 0 0x60>; |
| 741 | interrupts = <GIC_SPI 4078 IRQ_TYPE_LEVEL_HIGH>; | 741 | interrupts = <GIC_ESPI 14 IRQ_TYPE_LEVEL_HIGH>; |
| 742 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; | 742 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 743 | clock-names = "fck", "brg_int", "scif_clk"; | 743 | clock-names = "fck", "brg_int", "scif_clk"; |
| 744 | status = "disabled"; | 744 | status = "disabled"; |
| @@ -748,7 +748,7 @@ · 3 unmodified lines | |||
| 748 | compatible = "renesas,hscif-r8a78000", | 748 | compatible = "renesas,hscif-r8a78000", |
| 749 | "renesas,rcar-gen5-hscif", "renesas,hscif"; | 749 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 750 | reg = <0 0xc0714000 0 0x60>; | 750 | reg = <0 0xc0714000 0 0x60>; |
| 751 | interrupts = <GIC_SPI 4079 IRQ_TYPE_LEVEL_HIGH>; | 751 | interrupts = <GIC_ESPI 15 IRQ_TYPE_LEVEL_HIGH>; |
| 752 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; | 752 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 753 | clock-names = "fck", "brg_int", "scif_clk"; | 753 | clock-names = "fck", "brg_int", "scif_clk"; |
| 754 | status = "disabled"; | 754 | status = "disabled"; |
| @@ -758,7 +758,7 @@ · 3 unmodified lines | |||
| 758 | compatible = "renesas,hscif-r8a78000", | 758 | compatible = "renesas,hscif-r8a78000", |
| 759 | "renesas,rcar-gen5-hscif", "renesas,hscif"; | 759 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 760 | reg = <0 0xc0718000 0 0x60>; | 760 | reg = <0 0xc0718000 0 0x60>; |
| 761 | interrupts = <GIC_SPI 4080 IRQ_TYPE_LEVEL_HIGH>; | 761 | interrupts = <GIC_ESPI 16 IRQ_TYPE_LEVEL_HIGH>; |
| 762 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; | 762 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 763 | clock-names = "fck", "brg_int", "scif_clk"; | 763 | clock-names = "fck", "brg_int", "scif_clk"; |
| 764 | status = "disabled"; | 764 | status = "disabled"; |
| @@ -768,7 +768,7 @@ · 3 unmodified lines | |||
| 768 | compatible = "renesas,hscif-r8a78000", | 768 | compatible = "renesas,hscif-r8a78000", |
| 769 | "renesas,rcar-gen5-hscif", "renesas,hscif"; | 769 | "renesas,rcar-gen5-hscif", "renesas,hscif"; |
| 770 | reg = <0 0xc071c000 0 0x60>; | 770 | reg = <0 0xc071c000 0 0x60>; |
| 771 | interrupts = <GIC_SPI 4081 IRQ_TYPE_LEVEL_HIGH>; | 771 | interrupts = <GIC_ESPI 17 IRQ_TYPE_LEVEL_HIGH>; |
| 772 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; | 772 | clocks = <&dummy_clk_sgasyncd4>, <&dummy_clk_sgasyncd4>, <&scif_clk>; |
| 773 | clock-names = "fck", "brg_int", "scif_clk"; | 773 | clock-names = "fck", "brg_int", "scif_clk"; |
| 774 | status = "disabled"; | 774 | status = "disabled"; |
| @@ -581,16 +581,6 @@ · 580 unmodified lines | |||
| 581 | status = "disabled"; | 581 | status = "disabled"; |
| 582 | }; | 582 | }; |
| 583 | 583 | ||
| 584 | wdt0: watchdog@11c00400 { | ||
| 585 | compatible = "renesas,r9a09g057-wdt"; | ||
| 586 | reg = <0 0x11c00400 0 0x400>; | ||
| 587 | clocks = <&cpg CPG_MOD 0x4b>, <&cpg CPG_MOD 0x4c>; | ||
| 588 | clock-names = "pclk", "oscclk"; | ||
| 589 | resets = <&cpg 0x75>; | ||
| 590 | power-domains = <&cpg>; | ||
| 591 | status = "disabled"; | ||
| 592 | }; | ||
| 593 | |||
| 594 | wdt1: watchdog@14400000 { | 584 | wdt1: watchdog@14400000 { |
| 595 | compatible = "renesas,r9a09g057-wdt"; | 585 | compatible = "renesas,r9a09g057-wdt"; |
| 596 | reg = <0 0x14400000 0 0x400>; | 586 | reg = <0 0x14400000 0 0x400>; |
| @@ -601,26 +591,6 @@ · 4 unmodified lines | |||
| 601 | status = "disabled"; | 591 | status = "disabled"; |
| 602 | }; | 592 | }; |
| 603 | 593 | ||
| 604 | wdt2: watchdog@13000000 { | ||
| 605 | compatible = "renesas,r9a09g057-wdt"; | ||
| 606 | reg = <0 0x13000000 0 0x400>; | ||
| 607 | clocks = <&cpg CPG_MOD 0x4f>, <&cpg CPG_MOD 0x50>; | ||
| 608 | clock-names = "pclk", "oscclk"; | ||
| 609 | resets = <&cpg 0x77>; | ||
| 610 | power-domains = <&cpg>; | ||
| 611 | status = "disabled"; | ||
| 612 | }; | ||
| 613 | |||
| 614 | wdt3: watchdog@13000400 { | ||
| 615 | compatible = "renesas,r9a09g057-wdt"; | ||
| 616 | reg = <0 0x13000400 0 0x400>; | ||
| 617 | clocks = <&cpg CPG_MOD 0x51>, <&cpg CPG_MOD 0x52>; | ||
| 618 | clock-names = "pclk", "oscclk"; | ||
| 619 | resets = <&cpg 0x78>; | ||
| 620 | power-domains = <&cpg>; | ||
| 621 | status = "disabled"; | ||
| 622 | }; | ||
| 623 | |||
| 624 | rtc: rtc@11c00800 { | 594 | rtc: rtc@11c00800 { |
| 625 | compatible = "renesas,r9a09g057-rtca3", "renesas,rz-rtca3"; | 595 | compatible = "renesas,r9a09g057-rtca3", "renesas,rz-rtca3"; |
| 626 | reg = <0 0x11c00800 0 0x400>; | 596 | reg = <0 0x11c00800 0 0x400>; |
| @@ -974,8 +974,8 @@ · 973 unmodified lines | |||
| 974 | 974 | ||
| 975 | cpg: clock-controller@80280000 { | 975 | cpg: clock-controller@80280000 { |
| 976 | compatible = "renesas,r9a09g077-cpg-mssr"; | 976 | compatible = "renesas,r9a09g077-cpg-mssr"; |
| 977 | reg = <0 0x80280000 0 0x1000>, | 977 | reg = <0 0x80280000 0 0x10000>, |
| 978 | <0 0x81280000 0 0x9000>; | 978 | <0 0x81280000 0 0x10000>; |
| 979 | clocks = <&extal_clk>; | 979 | clocks = <&extal_clk>; |
| 980 | clock-names = "extal"; | 980 | clock-names = "extal"; |
| 981 | #clock-cells = <2>; | 981 | #clock-cells = <2>; |
| @@ -977,8 +977,8 @@ · 976 unmodified lines | |||
| 977 | 977 | ||
| 978 | cpg: clock-controller@80280000 { | 978 | cpg: clock-controller@80280000 { |
| 979 | compatible = "renesas,r9a09g087-cpg-mssr"; | 979 | compatible = "renesas,r9a09g087-cpg-mssr"; |
| 980 | reg = <0 0x80280000 0 0x1000>, | 980 | reg = <0 0x80280000 0 0x10000>, |
| 981 | <0 0x81280000 0 0x9000>; | 981 | <0 0x81280000 0 0x10000>; |
| 982 | clocks = <&extal_clk>; | 982 | clocks = <&extal_clk>; |
| 983 | clock-names = "extal"; | 983 | clock-names = "extal"; |
| 984 | #clock-cells = <2>; | 984 | #clock-cells = <2>; |
| @@ -162,7 +162,7 @@ · 161 unmodified lines | |||
| 162 | <100000000>; | 162 | <100000000>; |
| 163 | renesas,settings = [ | 163 | renesas,settings = [ |
| 164 | 80 00 11 19 4c 42 dc 2f 06 7d 20 1a 5f 1e f2 27 | 164 | 80 00 11 19 4c 42 dc 2f 06 7d 20 1a 5f 1e f2 27 |
| 165 | 00 40 00 00 00 00 00 00 06 0c 19 02 3f f0 90 86 | 165 | 00 40 00 00 00 00 00 00 06 0c 19 02 3b f0 90 86 |
| 166 | a0 80 30 30 9c | 166 | a0 80 30 30 9c |
| 167 | ]; | 167 | ]; |
| 168 | }; | 168 | }; |
| @@ -53,6 +53,7 @@ · 52 unmodified lines | |||
| 53 | regulator-max-microvolt = <3300000>; | 53 | regulator-max-microvolt = <3300000>; |
| 54 | gpios-states = <0>; | 54 | gpios-states = <0>; |
| 55 | states = <3300000 0>, <1800000 1>; | 55 | states = <3300000 0>, <1800000 1>; |
| 56 | regulator-ramp-delay = <60>; | ||
| 56 | }; | 57 | }; |
| 57 | #endif | 58 | #endif |
| 58 | 59 | ||
| @@ -25,6 +25,7 @@ · 24 unmodified lines | |||
| 25 | regulator-max-microvolt = <3300000>; | 25 | regulator-max-microvolt = <3300000>; |
| 26 | gpios-states = <0>; | 26 | gpios-states = <0>; |
| 27 | states = <3300000 0>, <1800000 1>; | 27 | states = <3300000 0>, <1800000 1>; |
| 28 | regulator-ramp-delay = <60>; | ||
| 28 | }; | 29 | }; |
| 29 | }; | 30 | }; |
| 30 | 31 | ||
| @@ -76,19 +76,24 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key, · 75 unmodified lines | |||
| 76 | unsigned int key_len) | 76 | unsigned int key_len) |
| 77 | { | 77 | { |
| 78 | struct aesbs_ctx *ctx = crypto_skcipher_ctx(tfm); | 78 | struct aesbs_ctx *ctx = crypto_skcipher_ctx(tfm); |
| 79 | struct crypto_aes_ctx rk; | 79 | struct crypto_aes_ctx *rk; |
| 80 | int err; | 80 | int err; |
| 81 | 81 | ||
| 82 | err = aes_expandkey(&rk, in_key, key_len); | 82 | rk = kmalloc(sizeof(*rk), GFP_KERNEL); |
| 83 | if (!rk) | ||
| 84 | return -ENOMEM; | ||
| 85 | |||
| 86 | err = aes_expandkey(rk, in_key, key_len); | ||
| 83 | if (err) | 87 | if (err) |
| 84 | return err; | 88 | goto out; |
| 85 | 89 | ||
| 86 | ctx->rounds = 6 + key_len / 4; | 90 | ctx->rounds = 6 + key_len / 4; |
| 87 | 91 | ||
| 88 | scoped_ksimd() | 92 | scoped_ksimd() |
| 89 | aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); | 93 | aesbs_convert_key(ctx->rk, rk->key_enc, ctx->rounds); |
| 90 | 94 | out: | |
| 91 | return 0; | 95 | kfree_sensitive(rk); |
| 96 | return err; | ||
| 92 | } | 97 | } |
| 93 | 98 | ||
| 94 | static int __ecb_crypt(struct skcipher_request *req, | 99 | static int __ecb_crypt(struct skcipher_request *req, |
| @@ -133,22 +138,26 @@ static int aesbs_cbc_ctr_setkey(struct crypto_skcipher *tfm, const u8 *in_key, · 38 unmodified lines | |||
| 133 | unsigned int key_len) | 138 | unsigned int key_len) |
| 134 | { | 139 | { |
| 135 | struct aesbs_cbc_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); | 140 | struct aesbs_cbc_ctr_ctx *ctx = crypto_skcipher_ctx(tfm); |
| 136 | struct crypto_aes_ctx rk; | 141 | struct crypto_aes_ctx *rk; |
| 137 | int err; | 142 | int err; |
| 138 | 143 | ||
| 139 | err = aes_expandkey(&rk, in_key, key_len); | 144 | rk = kmalloc(sizeof(*rk), GFP_KERNEL); |
| 145 | if (!rk) | ||
| 146 | return -ENOMEM; | ||
| 147 | |||
| 148 | err = aes_expandkey(rk, in_key, key_len); | ||
| 140 | if (err) | 149 | if (err) |
| 141 | return err; | 150 | goto out; |
| 142 | 151 | ||
| 143 | ctx->key.rounds = 6 + key_len / 4; | 152 | ctx->key.rounds = 6 + key_len / 4; |
| 144 | 153 | ||
| 145 | memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc)); | 154 | memcpy(ctx->enc, rk->key_enc, sizeof(ctx->enc)); |
| 146 | 155 | ||
| 147 | scoped_ksimd() | 156 | scoped_ksimd() |
| 148 | aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); | 157 | aesbs_convert_key(ctx->key.rk, rk->key_enc, ctx->key.rounds); |
| 149 | memzero_explicit(&rk, sizeof(rk)); | 158 | out: |
| 150 | 159 | kfree_sensitive(rk); | |
| 151 | return 0; | 160 | return err; |
| 152 | } | 161 | } |
| 153 | 162 | ||
| 154 | static int cbc_encrypt(struct skcipher_request *req) | 163 | static int cbc_encrypt(struct skcipher_request *req) |
| @@ -784,6 +784,9 @@ struct kvm_host_data { · 783 unmodified lines | |||
| 784 | /* Number of debug breakpoints/watchpoints for this CPU (minus 1) */ | 784 | /* Number of debug breakpoints/watchpoints for this CPU (minus 1) */ |
| 785 | unsigned int debug_brps; | 785 | unsigned int debug_brps; |
| 786 | unsigned int debug_wrps; | 786 | unsigned int debug_wrps; |
| 787 | |||
| 788 | /* Last vgic_irq part of the AP list recorded in an LR */ | ||
| 789 | struct vgic_irq *last_lr_irq; | ||
| 787 | }; | 790 | }; |
| 788 | 791 | ||
| 789 | struct kvm_host_psci_config { | 792 | struct kvm_host_psci_config { |
| @@ -0,0 +1,31 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | #ifndef _ASM_STATIC_CALL_H | ||
| 3 | #define _ASM_STATIC_CALL_H | ||
| 4 | |||
| 5 | #define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, target) \ | ||
| 6 | asm(" .pushsection .static_call.text, \"ax\" \n" \ | ||
| 7 | " .align 4 \n" \ | ||
| 8 | " .globl " name " \n" \ | ||
| 9 | name ": \n" \ | ||
| 10 | " hint 34 /* BTI C */ \n" \ | ||
| 11 | " adrp x16, 1f \n" \ | ||
| 12 | " ldr x16, [x16, :lo12:1f] \n" \ | ||
| 13 | " br x16 \n" \ | ||
| 14 | " .type " name ", %function \n" \ | ||
| 15 | " .size " name ", . - " name " \n" \ | ||
| 16 | " .popsection \n" \ | ||
| 17 | " .pushsection .rodata, \"a\" \n" \ | ||
| 18 | " .align 3 \n" \ | ||
| 19 | "1: .quad " target " \n" \ | ||
| 20 | " .popsection \n") | ||
| 21 | |||
| 22 | #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \ | ||
| 23 | __ARCH_DEFINE_STATIC_CALL_TRAMP(STATIC_CALL_TRAMP_STR(name), #func) | ||
| 24 | |||
| 25 | #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ | ||
| 26 | ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0) | ||
| 27 | |||
| 28 | #define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) \ | ||
| 29 | ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0) | ||
| 30 | |||
| 31 | #endif /* _ASM_STATIC_CALL_H */ | ||
| @@ -46,6 +46,7 @@ obj-$(CONFIG_MODULES) += module.o module-plts.o · 45 unmodified lines | |||
| 46 | obj-$(CONFIG_PERF_EVENTS) += perf_regs.o perf_callchain.o | 46 | obj-$(CONFIG_PERF_EVENTS) += perf_regs.o perf_callchain.o |
| 47 | obj-$(CONFIG_HARDLOCKUP_DETECTOR_PERF) += watchdog_hld.o | 47 | obj-$(CONFIG_HARDLOCKUP_DETECTOR_PERF) += watchdog_hld.o |
| 48 | obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o | 48 | obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o |
| 49 | obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o | ||
| 49 | obj-$(CONFIG_CPU_PM) += sleep.o suspend.o | 50 | obj-$(CONFIG_CPU_PM) += sleep.o suspend.o |
| 50 | obj-$(CONFIG_KGDB) += kgdb.o | 51 | obj-$(CONFIG_KGDB) += kgdb.o |
| 51 | obj-$(CONFIG_EFI) += efi.o efi-rt-wrapper.o | 52 | obj-$(CONFIG_EFI) += efi.o efi-rt-wrapper.o |
| @@ -2345,6 +2345,15 @@ static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry, · 2344 unmodified lines | |||
| 2345 | !is_midr_in_range_list(has_vgic_v3)) | 2345 | !is_midr_in_range_list(has_vgic_v3)) |
| 2346 | return false; | 2346 | return false; |
| 2347 | 2347 | ||
| 2348 | /* | ||
| 2349 | * pKVM prevents late onlining of CPUs. This means that whatever | ||
| 2350 | * state the capability is in after deprivilege cannot be affected | ||
| 2351 | * by a new CPU booting -- this is garanteed to be a CPU we have | ||
| 2352 | * already seen, and the cap is therefore unchanged. | ||
| 2353 | */ | ||
| 2354 | if (system_capabilities_finalized() && is_protected_kvm_enabled()) | ||
| 2355 | return cpus_have_final_cap(ARM64_HAS_ICH_HCR_EL2_TDIR); | ||
| 2356 | |||
| 2348 | if (is_kernel_in_hyp_mode()) | 2357 | if (is_kernel_in_hyp_mode()) |
| 2349 | res.a1 = read_sysreg_s(SYS_ICH_VTR_EL2); | 2358 | res.a1 = read_sysreg_s(SYS_ICH_VTR_EL2); |
| 2350 | else | 2359 | else |
| @@ -192,6 +192,14 @@ static int scs_handle_fde_frame(const struct eh_frame *frame, · 191 unmodified lines | |||
| 192 | size -= 2; | 192 | size -= 2; |
| 193 | break; | 193 | break; |
| 194 | 194 | ||
| 195 | case DW_CFA_advance_loc4: | ||
| 196 | loc += *opcode++ * code_alignment_factor; | ||
| 197 | loc += (*opcode++ << 8) * code_alignment_factor; | ||
| 198 | loc += (*opcode++ << 16) * code_alignment_factor; | ||
| 199 | loc += (*opcode++ << 24) * code_alignment_factor; | ||
| 200 | size -= 4; | ||
| 201 | break; | ||
| 202 | |||
| 195 | case DW_CFA_def_cfa: | 203 | case DW_CFA_def_cfa: |
| 196 | case DW_CFA_offset_extended: | 204 | case DW_CFA_offset_extended: |
| 197 | size = skip_xleb128(&opcode, size); | 205 | size = skip_xleb128(&opcode, size); |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | |||
| 12 | 12 | ||
| 13 | #include <asm/io.h> | 13 | #include <asm/io.h> |
| 14 | #include <asm/mem_encrypt.h> | 14 | #include <asm/mem_encrypt.h> |
| 15 | #include <asm/pgtable.h> | ||
| 15 | #include <asm/rsi.h> | 16 | #include <asm/rsi.h> |
| 16 | 17 | ||
| 17 | static struct realm_config config; | 18 | static struct realm_config config; |
| @@ -146,7 +147,7 @@ void __init arm64_rsi_init(void) · 128 unmodified lines | |||
| 146 | return; | 147 | return; |
| 147 | if (WARN_ON(rsi_get_realm_config(&config))) | 148 | if (WARN_ON(rsi_get_realm_config(&config))) |
| 148 | return; | 149 | return; |
| 149 | prot_ns_shared = BIT(config.ipa_bits - 1); | 150 | prot_ns_shared = __phys_to_pte_val(BIT(config.ipa_bits - 1)); |
| 150 | 151 | ||
| 151 | if (arm64_ioremap_prot_hook_register(realm_ioremap_hook)) | 152 | if (arm64_ioremap_prot_hook_register(realm_ioremap_hook)) |
| 152 | return; | 153 | return; |
| @@ -0,0 +1,23 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | #include <linux/static_call.h> | ||
| 3 | #include <linux/memory.h> | ||
| 4 | #include <asm/text-patching.h> | ||
| 5 | |||
| 6 | void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) | ||
| 7 | { | ||
| 8 | u64 literal; | ||
| 9 | int ret; | ||
| 10 | |||
| 11 | if (!func) | ||
| 12 | func = __static_call_return0; | ||
| 13 | |||
| 14 | /* decode the instructions to discover the literal address */ | ||
| 15 | literal = ALIGN_DOWN((u64)tramp + 4, SZ_4K) + | ||
| 16 | aarch64_insn_adrp_get_offset(le32_to_cpup(tramp + 4)) + | ||
| 17 | 8 * aarch64_insn_decode_immediate(AARCH64_INSN_IMM_12, | ||
| 18 | le32_to_cpup(tramp + 8)); | ||
| 19 | |||
| 20 | ret = aarch64_insn_write_literal_u64((void *)literal, (u64)func); | ||
| 21 | WARN_ON_ONCE(ret); | ||
| 22 | } | ||
| 23 | EXPORT_SYMBOL_GPL(arch_static_call_transform); | ||
| @@ -191,6 +191,7 @@ SECTIONS · 190 unmodified lines | |||
| 191 | LOCK_TEXT | 191 | LOCK_TEXT |
| 192 | KPROBES_TEXT | 192 | KPROBES_TEXT |
| 193 | HYPERVISOR_TEXT | 193 | HYPERVISOR_TEXT |
| 194 | STATIC_CALL_TEXT | ||
| 194 | *(.gnu.warning) | 195 | *(.gnu.warning) |
| 195 | } | 196 | } |
| 196 | 197 | ||
| @@ -1504,8 +1504,6 @@ int __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) · 1503 unmodified lines | |||
| 1504 | fail = true; | 1504 | fail = true; |
| 1505 | } | 1505 | } |
| 1506 | 1506 | ||
| 1507 | isb(); | ||
| 1508 | |||
| 1509 | if (!fail) | 1507 | if (!fail) |
| 1510 | par = read_sysreg_par(); | 1508 | par = read_sysreg_par(); |
| 1511 | 1509 | ||
| @@ -1755,7 +1753,7 @@ int __kvm_at_swap_desc(struct kvm *kvm, gpa_t ipa, u64 old, u64 new) · 243 unmodified lines | |||
| 1755 | if (!writable) | 1753 | if (!writable) |
| 1756 | return -EPERM; | 1754 | return -EPERM; |
| 1757 | 1755 | ||
| 1758 | ptep = (u64 __user *)hva + offset; | 1756 | ptep = (void __user *)hva + offset; |
| 1759 | if (cpus_have_final_cap(ARM64_HAS_LSE_ATOMICS)) | 1757 | if (cpus_have_final_cap(ARM64_HAS_LSE_ATOMICS)) |
| 1760 | r = __lse_swap_desc(ptep, old, new); | 1758 | r = __lse_swap_desc(ptep, old, new); |
| 1761 | else | 1759 | else |
| @@ -29,7 +29,7 @@ · 28 unmodified lines | |||
| 29 | 29 | ||
| 30 | #include "trace.h" | 30 | #include "trace.h" |
| 31 | 31 | ||
| 32 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 32 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 33 | KVM_GENERIC_VM_STATS() | 33 | KVM_GENERIC_VM_STATS() |
| 34 | }; | 34 | }; |
| 35 | 35 | ||
| @@ -42,7 +42,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 6 unmodified lines | |||
| 42 | sizeof(kvm_vm_stats_desc), | 42 | sizeof(kvm_vm_stats_desc), |
| 43 | }; | 43 | }; |
| 44 | 44 | ||
| 45 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 45 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 46 | KVM_GENERIC_VCPU_STATS(), | 46 | KVM_GENERIC_VCPU_STATS(), |
| 47 | STATS_DESC_COUNTER(VCPU, hvc_exit_stat), | 47 | STATS_DESC_COUNTER(VCPU, hvc_exit_stat), |
| 48 | STATS_DESC_COUNTER(VCPU, wfe_exit_stat), | 48 | STATS_DESC_COUNTER(VCPU, wfe_exit_stat), |
| @@ -518,7 +518,7 @@ static int host_stage2_adjust_range(u64 addr, struct kvm_mem_range *range) · 517 unmodified lines | |||
| 518 | granule = kvm_granule_size(level); | 518 | granule = kvm_granule_size(level); |
| 519 | cur.start = ALIGN_DOWN(addr, granule); | 519 | cur.start = ALIGN_DOWN(addr, granule); |
| 520 | cur.end = cur.start + granule; | 520 | cur.end = cur.start + granule; |
| 521 | if (!range_included(&cur, range)) | 521 | if (!range_included(&cur, range) && level < KVM_PGTABLE_LAST_LEVEL) |
| 522 | continue; | 522 | continue; |
| 523 | *range = cur; | 523 | *range = cur; |
| 524 | return 0; | 524 | return 0; |
| @@ -1751,6 +1751,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 1750 unmodified lines | |||
| 1751 | 1751 | ||
| 1752 | force_pte = (max_map_size == PAGE_SIZE); | 1752 | force_pte = (max_map_size == PAGE_SIZE); |
| 1753 | vma_pagesize = min_t(long, vma_pagesize, max_map_size); | 1753 | vma_pagesize = min_t(long, vma_pagesize, max_map_size); |
| 1754 | vma_shift = __ffs(vma_pagesize); | ||
| 1754 | } | 1755 | } |
| 1755 | 1756 | ||
| 1756 | /* | 1757 | /* |
| @@ -1837,10 +1838,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 80 unmodified lines | |||
| 1837 | if (exec_fault && s2_force_noncacheable) | 1838 | if (exec_fault && s2_force_noncacheable) |
| 1838 | ret = -ENOEXEC; | 1839 | ret = -ENOEXEC; |
| 1839 | 1840 | ||
| 1840 | if (ret) { | 1841 | if (ret) |
| 1841 | kvm_release_page_unused(page); | 1842 | goto out_put_page; |
| 1842 | return ret; | ||
| 1843 | } | ||
| 1844 | 1843 | ||
| 1845 | /* | 1844 | /* |
| 1846 | * Guest performs atomic/exclusive operations on memory with unsupported | 1845 | * Guest performs atomic/exclusive operations on memory with unsupported |
| @@ -1850,7 +1849,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · 3 unmodified lines | |||
| 1850 | */ | 1849 | */ |
| 1851 | if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) { | 1850 | if (esr_fsc_is_excl_atomic_fault(kvm_vcpu_get_esr(vcpu))) { |
| 1852 | kvm_inject_dabt_excl_atomic(vcpu, kvm_vcpu_get_hfar(vcpu)); | 1851 | kvm_inject_dabt_excl_atomic(vcpu, kvm_vcpu_get_hfar(vcpu)); |
| 1853 | return 1; | 1852 | ret = 1; |
| 1853 | goto out_put_page; | ||
| 1854 | } | 1854 | } |
| 1855 | 1855 | ||
| 1856 | if (nested) | 1856 | if (nested) |
| @@ -1936,6 +1936,10 @@ out_unlock: · 79 unmodified lines | |||
| 1936 | mark_page_dirty_in_slot(kvm, memslot, gfn); | 1936 | mark_page_dirty_in_slot(kvm, memslot, gfn); |
| 1937 | 1937 | ||
| 1938 | return ret != -EAGAIN ? ret : 0; | 1938 | return ret != -EAGAIN ? ret : 0; |
| 1939 | |||
| 1940 | out_put_page: | ||
| 1941 | kvm_release_page_unused(page); | ||
| 1942 | return ret; | ||
| 1939 | } | 1943 | } |
| 1940 | 1944 | ||
| 1941 | /* Resolve the access fault by making the page young again. */ | 1945 | /* Resolve the access fault by making the page young again. */ |
| @@ -152,31 +152,31 @@ static int get_ia_size(struct s2_walk_info *wi) · 151 unmodified lines | |||
| 152 | return 64 - wi->t0sz; | 152 | return 64 - wi->t0sz; |
| 153 | } | 153 | } |
| 154 | 154 | ||
| 155 | static int check_base_s2_limits(struct s2_walk_info *wi, | 155 | static int check_base_s2_limits(struct kvm_vcpu *vcpu, struct s2_walk_info *wi, |
| 156 | int level, int input_size, int stride) | 156 | int level, int input_size, int stride) |
| 157 | { | 157 | { |
| 158 | int start_size, ia_size; | 158 | int start_size, pa_max; |
| 159 | 159 | ||
| 160 | ia_size = get_ia_size(wi); | 160 | pa_max = kvm_get_pa_bits(vcpu->kvm); |
| 161 | 161 | ||
| 162 | /* Check translation limits */ | 162 | /* Check translation limits */ |
| 163 | switch (BIT(wi->pgshift)) { | 163 | switch (BIT(wi->pgshift)) { |
| 164 | case SZ_64K: | 164 | case SZ_64K: |
| 165 | if (level == 0 || (level == 1 && ia_size <= 42)) | 165 | if (level == 0 || (level == 1 && pa_max <= 42)) |
| 166 | return -EFAULT; | 166 | return -EFAULT; |
| 167 | break; | 167 | break; |
| 168 | case SZ_16K: | 168 | case SZ_16K: |
| 169 | if (level == 0 || (level == 1 && ia_size <= 40)) | 169 | if (level == 0 || (level == 1 && pa_max <= 40)) |
| 170 | return -EFAULT; | 170 | return -EFAULT; |
| 171 | break; | 171 | break; |
| 172 | case SZ_4K: | 172 | case SZ_4K: |
| 173 | if (level < 0 || (level == 0 && ia_size <= 42)) | 173 | if (level < 0 || (level == 0 && pa_max <= 42)) |
| 174 | return -EFAULT; | 174 | return -EFAULT; |
| 175 | break; | 175 | break; |
| 176 | } | 176 | } |
| 177 | 177 | ||
| 178 | /* Check input size limits */ | 178 | /* Check input size limits */ |
| 179 | if (input_size > ia_size) | 179 | if (input_size > pa_max) |
| 180 | return -EFAULT; | 180 | return -EFAULT; |
| 181 | 181 | ||
| 182 | /* Check number of entries in starting level table */ | 182 | /* Check number of entries in starting level table */ |
| @@ -269,16 +269,19 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, · 86 unmodified lines | |||
| 269 | if (input_size > 48 || input_size < 25) | 269 | if (input_size > 48 || input_size < 25) |
| 270 | return -EFAULT; | 270 | return -EFAULT; |
| 271 | 271 | ||
| 272 | ret = check_base_s2_limits(wi, level, input_size, stride); | 272 | ret = check_base_s2_limits(vcpu, wi, level, input_size, stride); |
| 273 | if (WARN_ON(ret)) | 273 | if (WARN_ON(ret)) { |
| 274 | out->esr = compute_fsc(0, ESR_ELx_FSC_FAULT); | ||
| 274 | return ret; | 275 | return ret; |
| 276 | } | ||
| 275 | 277 | ||
| 276 | base_lower_bound = 3 + input_size - ((3 - level) * stride + | 278 | base_lower_bound = 3 + input_size - ((3 - level) * stride + |
| 277 | wi->pgshift); | 279 | wi->pgshift); |
| 278 | base_addr = wi->baddr & GENMASK_ULL(47, base_lower_bound); | 280 | base_addr = wi->baddr & GENMASK_ULL(47, base_lower_bound); |
| 279 | 281 | ||
| 280 | if (check_output_size(wi, base_addr)) { | 282 | if (check_output_size(wi, base_addr)) { |
| 281 | out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); | 283 | /* R_BFHQH */ |
| 284 | out->esr = compute_fsc(0, ESR_ELx_FSC_ADDRSZ); | ||
| 282 | return 1; | 285 | return 1; |
| 283 | } | 286 | } |
| 284 | 287 | ||
| @@ -293,8 +296,10 @@ static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, · 8 unmodified lines | |||
| 293 | 296 | ||
| 294 | paddr = base_addr | index; | 297 | paddr = base_addr | index; |
| 295 | ret = read_guest_s2_desc(vcpu, paddr, &desc, wi); | 298 | ret = read_guest_s2_desc(vcpu, paddr, &desc, wi); |
| 296 | if (ret < 0) | 299 | if (ret < 0) { |
| 300 | out->esr = ESR_ELx_FSC_SEA_TTW(level); | ||
| 297 | return ret; | 301 | return ret; |
| 302 | } | ||
| 298 | 303 | ||
| 299 | new_desc = desc; | 304 | new_desc = desc; |
| 300 | 305 | ||
| @@ -247,6 +247,20 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) · 246 unmodified lines | |||
| 247 | kvm_vcpu_set_be(vcpu); | 247 | kvm_vcpu_set_be(vcpu); |
| 248 | 248 | ||
| 249 | *vcpu_pc(vcpu) = target_pc; | 249 | *vcpu_pc(vcpu) = target_pc; |
| 250 | |||
| 251 | /* | ||
| 252 | * We may come from a state where either a PC update was | ||
| 253 | * pending (SMC call resulting in PC being increpented to | ||
| 254 | * skip the SMC) or a pending exception. Make sure we get | ||
| 255 | * rid of all that, as this cannot be valid out of reset. | ||
| 256 | * | ||
| 257 | * Note that clearing the exception mask also clears PC | ||
| 258 | * updates, but that's an implementation detail, and we | ||
| 259 | * really want to make it explicit. | ||
| 260 | */ | ||
| 261 | vcpu_clear_flag(vcpu, PENDING_EXCEPTION); | ||
| 262 | vcpu_clear_flag(vcpu, EXCEPT_MASK); | ||
| 263 | vcpu_clear_flag(vcpu, INCREMENT_PC); | ||
| 250 | vcpu_set_reg(vcpu, 0, reset_state.r0); | 264 | vcpu_set_reg(vcpu, 0, reset_state.r0); |
| 251 | } | 265 | } |
| 252 | 266 | ||
| @@ -143,6 +143,21 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) · 142 unmodified lines | |||
| 143 | kvm->arch.vgic.in_kernel = true; | 143 | kvm->arch.vgic.in_kernel = true; |
| 144 | kvm->arch.vgic.vgic_model = type; | 144 | kvm->arch.vgic.vgic_model = type; |
| 145 | kvm->arch.vgic.implementation_rev = KVM_VGIC_IMP_REV_LATEST; | 145 | kvm->arch.vgic.implementation_rev = KVM_VGIC_IMP_REV_LATEST; |
| 146 | kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF; | ||
| 147 | |||
| 148 | aa64pfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1) & ~ID_AA64PFR0_EL1_GIC; | ||
| 149 | pfr1 = kvm_read_vm_id_reg(kvm, SYS_ID_PFR1_EL1) & ~ID_PFR1_EL1_GIC; | ||
| 150 | |||
| 151 | if (type == KVM_DEV_TYPE_ARM_VGIC_V2) { | ||
| 152 | kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF; | ||
| 153 | } else { | ||
| 154 | INIT_LIST_HEAD(&kvm->arch.vgic.rd_regions); | ||
| 155 | aa64pfr0 |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, GIC, IMP); | ||
| 156 | pfr1 |= SYS_FIELD_PREP_ENUM(ID_PFR1_EL1, GIC, GICv3); | ||
| 157 | } | ||
| 158 | |||
| 159 | kvm_set_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1, aa64pfr0); | ||
| 160 | kvm_set_vm_id_reg(kvm, SYS_ID_PFR1_EL1, pfr1); | ||
| 146 | 161 | ||
| 147 | kvm_for_each_vcpu(i, vcpu, kvm) { | 162 | kvm_for_each_vcpu(i, vcpu, kvm) { |
| 148 | ret = vgic_allocate_private_irqs_locked(vcpu, type); | 163 | ret = vgic_allocate_private_irqs_locked(vcpu, type); |
| @@ -157,25 +172,10 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) · 8 unmodified lines | |||
| 157 | vgic_cpu->private_irqs = NULL; | 172 | vgic_cpu->private_irqs = NULL; |
| 158 | } | 173 | } |
| 159 | 174 | ||
| 175 | kvm->arch.vgic.vgic_model = 0; | ||
| 160 | goto out_unlock; | 176 | goto out_unlock; |
| 161 | } | 177 | } |
| 162 | 178 | ||
| 163 | kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF; | ||
| 164 | |||
| 165 | aa64pfr0 = kvm_read_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1) & ~ID_AA64PFR0_EL1_GIC; | ||
| 166 | pfr1 = kvm_read_vm_id_reg(kvm, SYS_ID_PFR1_EL1) & ~ID_PFR1_EL1_GIC; | ||
| 167 | |||
| 168 | if (type == KVM_DEV_TYPE_ARM_VGIC_V2) { | ||
| 169 | kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF; | ||
| 170 | } else { | ||
| 171 | INIT_LIST_HEAD(&kvm->arch.vgic.rd_regions); | ||
| 172 | aa64pfr0 |= SYS_FIELD_PREP_ENUM(ID_AA64PFR0_EL1, GIC, IMP); | ||
| 173 | pfr1 |= SYS_FIELD_PREP_ENUM(ID_PFR1_EL1, GIC, GICv3); | ||
| 174 | } | ||
| 175 | |||
| 176 | kvm_set_vm_id_reg(kvm, SYS_ID_AA64PFR0_EL1, aa64pfr0); | ||
| 177 | kvm_set_vm_id_reg(kvm, SYS_ID_PFR1_EL1, pfr1); | ||
| 178 | |||
| 179 | if (type == KVM_DEV_TYPE_ARM_VGIC_V3) | 179 | if (type == KVM_DEV_TYPE_ARM_VGIC_V3) |
| 180 | kvm->arch.vgic.nassgicap = system_supports_direct_sgis(); | 180 | kvm->arch.vgic.nassgicap = system_supports_direct_sgis(); |
| 181 | 181 | ||
| @@ -115,7 +115,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) · 114 unmodified lines | |||
| 115 | struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; | 115 | struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; |
| 116 | struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2; | 116 | struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2; |
| 117 | u32 eoicount = FIELD_GET(GICH_HCR_EOICOUNT, cpuif->vgic_hcr); | 117 | u32 eoicount = FIELD_GET(GICH_HCR_EOICOUNT, cpuif->vgic_hcr); |
| 118 | struct vgic_irq *irq; | 118 | struct vgic_irq *irq = *host_data_ptr(last_lr_irq); |
| 119 | 119 | ||
| 120 | DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); | 120 | DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); |
| 121 | 121 | ||
| @@ -123,7 +123,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) · 1 unmodified line | |||
| 123 | vgic_v2_fold_lr(vcpu, cpuif->vgic_lr[lr]); | 123 | vgic_v2_fold_lr(vcpu, cpuif->vgic_lr[lr]); |
| 124 | 124 | ||
| 125 | /* See the GICv3 equivalent for the EOIcount handling rationale */ | 125 | /* See the GICv3 equivalent for the EOIcount handling rationale */ |
| 126 | list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { | 126 | list_for_each_entry_continue(irq, &vgic_cpu->ap_list_head, ap_list) { |
| 127 | u32 lr; | 127 | u32 lr; |
| 128 | 128 | ||
| 129 | if (!eoicount) { | 129 | if (!eoicount) { |
| @@ -148,7 +148,7 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) · 147 unmodified lines | |||
| 148 | struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; | 148 | struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; |
| 149 | struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; | 149 | struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; |
| 150 | u32 eoicount = FIELD_GET(ICH_HCR_EL2_EOIcount, cpuif->vgic_hcr); | 150 | u32 eoicount = FIELD_GET(ICH_HCR_EL2_EOIcount, cpuif->vgic_hcr); |
| 151 | struct vgic_irq *irq; | 151 | struct vgic_irq *irq = *host_data_ptr(last_lr_irq); |
| 152 | 152 | ||
| 153 | DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); | 153 | DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); |
| 154 | 154 | ||
| @@ -158,12 +158,12 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) · 3 unmodified lines | |||
| 158 | /* | 158 | /* |
| 159 | * EOIMode=0: use EOIcount to emulate deactivation. We are | 159 | * EOIMode=0: use EOIcount to emulate deactivation. We are |
| 160 | * guaranteed to deactivate in reverse order of the activation, so | 160 | * guaranteed to deactivate in reverse order of the activation, so |
| 161 | * just pick one active interrupt after the other in the ap_list, | 161 | * just pick one active interrupt after the other in the tail part |
| 162 | * and replay the deactivation as if the CPU was doing it. We also | 162 | * of the ap_list, past the LRs, and replay the deactivation as if |
| 163 | * rely on priority drop to have taken place, and the list to be | 163 | * the CPU was doing it. We also rely on priority drop to have taken |
| 164 | * sorted by priority. | 164 | * place, and the list to be sorted by priority. |
| 165 | */ | 165 | */ |
| 166 | list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { | 166 | list_for_each_entry_continue(irq, &vgic_cpu->ap_list_head, ap_list) { |
| 167 | u64 lr; | 167 | u64 lr; |
| 168 | 168 | ||
| 169 | /* | 169 | /* |
| @@ -814,6 +814,9 @@ retry: · 813 unmodified lines | |||
| 814 | 814 | ||
| 815 | static inline void vgic_fold_lr_state(struct kvm_vcpu *vcpu) | 815 | static inline void vgic_fold_lr_state(struct kvm_vcpu *vcpu) |
| 816 | { | 816 | { |
| 817 | if (!*host_data_ptr(last_lr_irq)) | ||
| 818 | return; | ||
| 819 | |||
| 817 | if (kvm_vgic_global_state.type == VGIC_V2) | 820 | if (kvm_vgic_global_state.type == VGIC_V2) |
| 818 | vgic_v2_fold_lr_state(vcpu); | 821 | vgic_v2_fold_lr_state(vcpu); |
| 819 | else | 822 | else |
| @@ -960,10 +963,13 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu) · 140 unmodified lines | |||
| 960 | if (irqs_outside_lrs(&als)) | 963 | if (irqs_outside_lrs(&als)) |
| 961 | vgic_sort_ap_list(vcpu); | 964 | vgic_sort_ap_list(vcpu); |
| 962 | 965 | ||
| 966 | *host_data_ptr(last_lr_irq) = NULL; | ||
| 967 | |||
| 963 | list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { | 968 | list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list) { |
| 964 | scoped_guard(raw_spinlock, &irq->irq_lock) { | 969 | scoped_guard(raw_spinlock, &irq->irq_lock) { |
| 965 | if (likely(vgic_target_oracle(irq) == vcpu)) { | 970 | if (likely(vgic_target_oracle(irq) == vcpu)) { |
| 966 | vgic_populate_lr(vcpu, irq, count++); | 971 | vgic_populate_lr(vcpu, irq, count++); |
| 972 | *host_data_ptr(last_lr_irq) = irq; | ||
| 967 | } | 973 | } |
| 968 | } | 974 | } |
| 969 | 975 | ||
| @@ -304,6 +304,9 @@ config AS_HAS_LBT_EXTENSION · 303 unmodified lines | |||
| 304 | config AS_HAS_LVZ_EXTENSION | 304 | config AS_HAS_LVZ_EXTENSION |
| 305 | def_bool $(as-instr,hvcl 0) | 305 | def_bool $(as-instr,hvcl 0) |
| 306 | 306 | ||
| 307 | config AS_HAS_SCQ_EXTENSION | ||
| 308 | def_bool $(as-instr,sc.q \$t0$(comma)\$t1$(comma)\$t2) | ||
| 309 | |||
| 307 | config CC_HAS_ANNOTATE_TABLEJUMP | 310 | config CC_HAS_ANNOTATE_TABLEJUMP |
| 308 | def_bool $(cc-option,-mannotate-tablejump) | 311 | def_bool $(cc-option,-mannotate-tablejump) |
| 309 | 312 | ||
| @@ -238,6 +238,8 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int · 237 unmodified lines | |||
| 238 | arch_cmpxchg((ptr), (o), (n)); \ | 238 | arch_cmpxchg((ptr), (o), (n)); \ |
| 239 | }) | 239 | }) |
| 240 | 240 | ||
| 241 | #ifdef CONFIG_AS_HAS_SCQ_EXTENSION | ||
| 242 | |||
| 241 | union __u128_halves { | 243 | union __u128_halves { |
| 242 | u128 full; | 244 | u128 full; |
| 243 | struct { | 245 | struct { |
| @@ -290,6 +292,9 @@ union __u128_halves { · 46 unmodified lines | |||
| 290 | BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ | 292 | BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ |
| 291 | __arch_cmpxchg128(ptr, o, n, ""); \ | 293 | __arch_cmpxchg128(ptr, o, n, ""); \ |
| 292 | }) | 294 | }) |
| 295 | |||
| 296 | #endif /* CONFIG_AS_HAS_SCQ_EXTENSION */ | ||
| 297 | |||
| 293 | #else | 298 | #else |
| 294 | #include <asm-generic/cmpxchg-local.h> | 299 | #include <asm-generic/cmpxchg-local.h> |
| 295 | #define arch_cmpxchg64_local(ptr, o, n) __generic_cmpxchg64_local((ptr), (o), (n)) | 300 | #define arch_cmpxchg64_local(ptr, o, n) __generic_cmpxchg64_local((ptr), (o), (n)) |
| @@ -41,4 +41,40 @@ · 40 unmodified lines | |||
| 41 | .cfi_endproc; \ | 41 | .cfi_endproc; \ |
| 42 | SYM_END(name, SYM_T_NONE) | 42 | SYM_END(name, SYM_T_NONE) |
| 43 | 43 | ||
| 44 | /* | ||
| 45 | * This is for the signal handler trampoline, which is used as the return | ||
| 46 | * address of the signal handlers in userspace instead of called normally. | ||
| 47 | * The long standing libgcc bug https://gcc.gnu.org/PR124050 requires a | ||
| 48 | * nop between .cfi_startproc and the actual address of the trampoline, so | ||
| 49 | * we cannot simply use SYM_FUNC_START. | ||
| 50 | * | ||
| 51 | * This wrapper also contains all the .cfi_* directives for recovering | ||
| 52 | * the content of the GPRs and the "return address" (where the rt_sigreturn | ||
| 53 | * syscall will jump to), assuming there is a struct rt_sigframe (where | ||
| 54 | * a struct sigcontext containing those information we need to recover) at | ||
| 55 | * $sp. The "DWARF for the LoongArch(TM) Architecture" manual states | ||
| 56 | * column 0 is for $zero, but it does not make too much sense to | ||
| 57 | * save/restore the hardware zero register. Repurpose this column here | ||
| 58 | * for the return address (here it's not the content of $ra we cannot use | ||
| 59 | * the default column 3). | ||
| 60 | */ | ||
| 61 | #define SYM_SIGFUNC_START(name) \ | ||
| 62 | .cfi_startproc; \ | ||
| 63 | .cfi_signal_frame; \ | ||
| 64 | .cfi_def_cfa 3, RT_SIGFRAME_SC; \ | ||
| 65 | .cfi_return_column 0; \ | ||
| 66 | .cfi_offset 0, SC_PC; \ | ||
| 67 | \ | ||
| 68 | .irp num, 1, 2, 3, 4, 5, 6, 7, 8, \ | ||
| 69 | 9, 10, 11, 12, 13, 14, 15, 16, \ | ||
| 70 | 17, 18, 19, 20, 21, 22, 23, 24, \ | ||
| 71 | 25, 26, 27, 28, 29, 30, 31; \ | ||
| 72 | .cfi_offset \num, SC_REGS + \num * SZREG; \ | ||
| 73 | .endr; \ | ||
| 74 | \ | ||
| 75 | nop; \ | ||
| 76 | SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) | ||
| 77 | |||
| 78 | #define SYM_SIGFUNC_END(name) SYM_FUNC_END(name) | ||
| 79 | |||
| 44 | #endif | 80 | #endif |
| @@ -0,0 +1,9 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0+ */ | ||
| 2 | |||
| 3 | #include <asm/siginfo.h> | ||
| 4 | #include <asm/ucontext.h> | ||
| 5 | |||
| 6 | struct rt_sigframe { | ||
| 7 | struct siginfo rs_info; | ||
| 8 | struct ucontext rs_uctx; | ||
| 9 | }; | ||
| @@ -253,8 +253,13 @@ do { \ · 252 unmodified lines | |||
| 253 | \ | 253 | \ |
| 254 | __get_kernel_common(*((type *)(dst)), sizeof(type), \ | 254 | __get_kernel_common(*((type *)(dst)), sizeof(type), \ |
| 255 | (__force type *)(src)); \ | 255 | (__force type *)(src)); \ |
| 256 | if (unlikely(__gu_err)) \ | 256 | if (unlikely(__gu_err)) { \ |
| 257 | pr_info("%s: memory access failed, ecode 0x%x\n", \ | ||
| 258 | __func__, read_csr_excode()); \ | ||
| 259 | pr_info("%s: the caller is %pS\n", \ | ||
| 260 | __func__, __builtin_return_address(0)); \ | ||
| 257 | goto err_label; \ | 261 | goto err_label; \ |
| 262 | } \ | ||
| 258 | } while (0) | 263 | } while (0) |
| 259 | 264 | ||
| 260 | #define __put_kernel_nofault(dst, src, type, err_label) \ | 265 | #define __put_kernel_nofault(dst, src, type, err_label) \ |
| @@ -264,8 +269,13 @@ do { \ · 3 unmodified lines | |||
| 264 | \ | 269 | \ |
| 265 | __pu_val = *(__force type *)(src); \ | 270 | __pu_val = *(__force type *)(src); \ |
| 266 | __put_kernel_common(((type *)(dst)), sizeof(type)); \ | 271 | __put_kernel_common(((type *)(dst)), sizeof(type)); \ |
| 267 | if (unlikely(__pu_err)) \ | 272 | if (unlikely(__pu_err)) { \ |
| 273 | pr_info("%s: memory access failed, ecode 0x%x\n", \ | ||
| 274 | __func__, read_csr_excode()); \ | ||
| 275 | pr_info("%s: the caller is %pS\n", \ | ||
| 276 | __func__, __builtin_return_address(0)); \ | ||
| 268 | goto err_label; \ | 277 | goto err_label; \ |
| 278 | } \ | ||
| 269 | } while (0) | 279 | } while (0) |
| 270 | 280 | ||
| 271 | extern unsigned long __copy_user(void *to, const void *from, __kernel_size_t n); | 281 | extern unsigned long __copy_user(void *to, const void *from, __kernel_size_t n); |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | |||
| 16 | #include <asm/ptrace.h> | 16 | #include <asm/ptrace.h> |
| 17 | #include <asm/processor.h> | 17 | #include <asm/processor.h> |
| 18 | #include <asm/ftrace.h> | 18 | #include <asm/ftrace.h> |
| 19 | #include <asm/sigframe.h> | ||
| 19 | #include <vdso/datapage.h> | 20 | #include <vdso/datapage.h> |
| 20 | 21 | ||
| 21 | static void __used output_ptreg_defines(void) | 22 | static void __used output_ptreg_defines(void) |
| @@ -220,6 +221,7 @@ static void __used output_sc_defines(void) · 198 unmodified lines | |||
| 220 | COMMENT("Linux sigcontext offsets."); | 221 | COMMENT("Linux sigcontext offsets."); |
| 221 | OFFSET(SC_REGS, sigcontext, sc_regs); | 222 | OFFSET(SC_REGS, sigcontext, sc_regs); |
| 222 | OFFSET(SC_PC, sigcontext, sc_pc); | 223 | OFFSET(SC_PC, sigcontext, sc_pc); |
| 224 | OFFSET(RT_SIGFRAME_SC, rt_sigframe, rs_uctx.uc_mcontext); | ||
| 223 | BLANK(); | 225 | BLANK(); |
| 224 | } | 226 | } |
| 225 | 227 | ||
| @@ -42,16 +42,15 @@ static int __init init_cpu_fullname(void) · 41 unmodified lines | |||
| 42 | int cpu, ret; | 42 | int cpu, ret; |
| 43 | char *cpuname; | 43 | char *cpuname; |
| 44 | const char *model; | 44 | const char *model; |
| 45 | struct device_node *root; | ||
| 46 | 45 | ||
| 47 | /* Parsing cpuname from DTS model property */ | 46 | /* Parsing cpuname from DTS model property */ |
| 48 | root = of_find_node_by_path("/"); | 47 | ret = of_property_read_string(of_root, "model", &model); |
| 49 | ret = of_property_read_string(root, "model", &model); | ||
| 50 | if (ret == 0) { | 48 | if (ret == 0) { |
| 51 | cpuname = kstrdup(model, GFP_KERNEL); | 49 | cpuname = kstrdup(model, GFP_KERNEL); |
| 50 | if (!cpuname) | ||
| 51 | return -ENOMEM; | ||
| 52 | loongson_sysconf.cpuname = strsep(&cpuname, " "); | 52 | loongson_sysconf.cpuname = strsep(&cpuname, " "); |
| 53 | } | 53 | } |
| 54 | of_node_put(root); | ||
| 55 | 54 | ||
| 56 | if (loongson_sysconf.cpuname && !strncmp(loongson_sysconf.cpuname, "Loongson", 8)) { | 55 | if (loongson_sysconf.cpuname && !strncmp(loongson_sysconf.cpuname, "Loongson", 8)) { |
| 57 | for (cpu = 0; cpu < NR_CPUS; cpu++) | 56 | for (cpu = 0; cpu < NR_CPUS; cpu++) |
| @@ -246,32 +246,51 @@ static int text_copy_cb(void *data) · 245 unmodified lines | |||
| 246 | 246 | ||
| 247 | if (smp_processor_id() == copy->cpu) { | 247 | if (smp_processor_id() == copy->cpu) { |
| 248 | ret = copy_to_kernel_nofault(copy->dst, copy->src, copy->len); | 248 | ret = copy_to_kernel_nofault(copy->dst, copy->src, copy->len); |
| 249 | if (ret) | 249 | if (ret) { |
| 250 | pr_err("%s: operation failed\n", __func__); | 250 | pr_err("%s: operation failed\n", __func__); |
| 251 | return ret; | ||
| 252 | } | ||
| 251 | } | 253 | } |
| 252 | 254 | ||
| 253 | flush_icache_range((unsigned long)copy->dst, (unsigned long)copy->dst + copy->len); | 255 | flush_icache_range((unsigned long)copy->dst, (unsigned long)copy->dst + copy->len); |
| 254 | 256 | ||
| 255 | return ret; | 257 | return 0; |
| 256 | } | 258 | } |
| 257 | 259 | ||
| 258 | int larch_insn_text_copy(void *dst, void *src, size_t len) | 260 | int larch_insn_text_copy(void *dst, void *src, size_t len) |
| 259 | { | 261 | { |
| 260 | int ret = 0; | 262 | int ret = 0; |
| 263 | int err = 0; | ||
| 261 | size_t start, end; | 264 | size_t start, end; |
| 262 | struct insn_copy copy = { | 265 | struct insn_copy copy = { |
| 263 | .dst = dst, | 266 | .dst = dst, |
| 264 | .src = src, | 267 | .src = src, |
| 265 | .len = len, | 268 | .len = len, |
| 266 | .cpu = smp_processor_id(), | 269 | .cpu = raw_smp_processor_id(), |
| 267 | }; | 270 | }; |
| 268 | 271 | ||
| 272 | /* | ||
| 273 | * Ensure copy.cpu won't be hot removed before stop_machine. | ||
| 274 | * If it is removed nobody will really update the text. | ||
| 275 | */ | ||
| 276 | lockdep_assert_cpus_held(); | ||
| 277 | |||
| 269 | start = round_down((size_t)dst, PAGE_SIZE); | 278 | start = round_down((size_t)dst, PAGE_SIZE); |
| 270 | end = round_up((size_t)dst + len, PAGE_SIZE); | 279 | end = round_up((size_t)dst + len, PAGE_SIZE); |
| 271 | 280 | ||
| 272 | set_memory_rw(start, (end - start) / PAGE_SIZE); | 281 | err = set_memory_rw(start, (end - start) / PAGE_SIZE); |
| 273 | ret = stop_machine(text_copy_cb, ©, cpu_online_mask); | 282 | if (err) { |
| 274 | set_memory_rox(start, (end - start) / PAGE_SIZE); | 283 | pr_info("%s: set_memory_rw() failed\n", __func__); |
| 284 | return err; | ||
| 285 | } | ||
| 286 | |||
| 287 | ret = stop_machine_cpuslocked(text_copy_cb, ©, cpu_online_mask); | ||
| 288 | |||
| 289 | err = set_memory_rox(start, (end - start) / PAGE_SIZE); | ||
| 290 | if (err) { | ||
| 291 | pr_info("%s: set_memory_rox() failed\n", __func__); | ||
| 292 | return err; | ||
| 293 | } | ||
| 275 | 294 | ||
| 276 | return ret; | 295 | return ret; |
| 277 | } | 296 | } |
| @@ -35,6 +35,7 @@ · 34 unmodified lines | |||
| 35 | #include <asm/cpu-features.h> | 35 | #include <asm/cpu-features.h> |
| 36 | #include <asm/fpu.h> | 36 | #include <asm/fpu.h> |
| 37 | #include <asm/lbt.h> | 37 | #include <asm/lbt.h> |
| 38 | #include <asm/sigframe.h> | ||
| 38 | #include <asm/ucontext.h> | 39 | #include <asm/ucontext.h> |
| 39 | #include <asm/vdso.h> | 40 | #include <asm/vdso.h> |
| 40 | 41 | ||
| @@ -51,11 +52,6 @@ · 10 unmodified lines | |||
| 51 | #define lock_lbt_owner() ({ preempt_disable(); pagefault_disable(); }) | 52 | #define lock_lbt_owner() ({ preempt_disable(); pagefault_disable(); }) |
| 52 | #define unlock_lbt_owner() ({ pagefault_enable(); preempt_enable(); }) | 53 | #define unlock_lbt_owner() ({ pagefault_enable(); preempt_enable(); }) |
| 53 | 54 | ||
| 54 | struct rt_sigframe { | ||
| 55 | struct siginfo rs_info; | ||
| 56 | struct ucontext rs_uctx; | ||
| 57 | }; | ||
| 58 | |||
| 59 | struct _ctx_layout { | 55 | struct _ctx_layout { |
| 60 | struct sctx_info *addr; | 56 | struct sctx_info *addr; |
| 61 | unsigned int size; | 57 | unsigned int size; |
| @@ -83,7 +83,7 @@ static inline void eiointc_update_sw_coremap(struct loongarch_eiointc *s, · 82 unmodified lines | |||
| 83 | 83 | ||
| 84 | if (!(s->status & BIT(EIOINTC_ENABLE_CPU_ENCODE))) { | 84 | if (!(s->status & BIT(EIOINTC_ENABLE_CPU_ENCODE))) { |
| 85 | cpuid = ffs(cpuid) - 1; | 85 | cpuid = ffs(cpuid) - 1; |
| 86 | cpuid = (cpuid >= 4) ? 0 : cpuid; | 86 | cpuid = ((cpuid < 0) || (cpuid >= 4)) ? 0 : cpuid; |
| 87 | } | 87 | } |
| 88 | 88 | ||
| 89 | vcpu = kvm_get_vcpu_by_cpuid(s->kvm, cpuid); | 89 | vcpu = kvm_get_vcpu_by_cpuid(s->kvm, cpuid); |
| @@ -472,34 +472,34 @@ static int kvm_eiointc_regs_access(struct kvm_device *dev, · 382 unmodified lines | |||
| 472 | switch (addr) { | 472 | switch (addr) { |
| 473 | case EIOINTC_NODETYPE_START ... EIOINTC_NODETYPE_END: | 473 | case EIOINTC_NODETYPE_START ... EIOINTC_NODETYPE_END: |
| 474 | offset = (addr - EIOINTC_NODETYPE_START) / 4; | 474 | offset = (addr - EIOINTC_NODETYPE_START) / 4; |
| 475 | p = s->nodetype + offset * 4; | 475 | p = (void *)s->nodetype + offset * 4; |
| 476 | break; | 476 | break; |
| 477 | case EIOINTC_IPMAP_START ... EIOINTC_IPMAP_END: | 477 | case EIOINTC_IPMAP_START ... EIOINTC_IPMAP_END: |
| 478 | offset = (addr - EIOINTC_IPMAP_START) / 4; | 478 | offset = (addr - EIOINTC_IPMAP_START) / 4; |
| 479 | p = &s->ipmap + offset * 4; | 479 | p = (void *)&s->ipmap + offset * 4; |
| 480 | break; | 480 | break; |
| 481 | case EIOINTC_ENABLE_START ... EIOINTC_ENABLE_END: | 481 | case EIOINTC_ENABLE_START ... EIOINTC_ENABLE_END: |
| 482 | offset = (addr - EIOINTC_ENABLE_START) / 4; | 482 | offset = (addr - EIOINTC_ENABLE_START) / 4; |
| 483 | p = s->enable + offset * 4; | 483 | p = (void *)s->enable + offset * 4; |
| 484 | break; | 484 | break; |
| 485 | case EIOINTC_BOUNCE_START ... EIOINTC_BOUNCE_END: | 485 | case EIOINTC_BOUNCE_START ... EIOINTC_BOUNCE_END: |
| 486 | offset = (addr - EIOINTC_BOUNCE_START) / 4; | 486 | offset = (addr - EIOINTC_BOUNCE_START) / 4; |
| 487 | p = s->bounce + offset * 4; | 487 | p = (void *)s->bounce + offset * 4; |
| 488 | break; | 488 | break; |
| 489 | case EIOINTC_ISR_START ... EIOINTC_ISR_END: | 489 | case EIOINTC_ISR_START ... EIOINTC_ISR_END: |
| 490 | offset = (addr - EIOINTC_ISR_START) / 4; | 490 | offset = (addr - EIOINTC_ISR_START) / 4; |
| 491 | p = s->isr + offset * 4; | 491 | p = (void *)s->isr + offset * 4; |
| 492 | break; | 492 | break; |
| 493 | case EIOINTC_COREISR_START ... EIOINTC_COREISR_END: | 493 | case EIOINTC_COREISR_START ... EIOINTC_COREISR_END: |
| 494 | if (cpu >= s->num_cpu) | 494 | if (cpu >= s->num_cpu) |
| 495 | return -EINVAL; | 495 | return -EINVAL; |
| 496 | 496 | ||
| 497 | offset = (addr - EIOINTC_COREISR_START) / 4; | 497 | offset = (addr - EIOINTC_COREISR_START) / 4; |
| 498 | p = s->coreisr[cpu] + offset * 4; | 498 | p = (void *)s->coreisr[cpu] + offset * 4; |
| 499 | break; | 499 | break; |
| 500 | case EIOINTC_COREMAP_START ... EIOINTC_COREMAP_END: | 500 | case EIOINTC_COREMAP_START ... EIOINTC_COREMAP_END: |
| 501 | offset = (addr - EIOINTC_COREMAP_START) / 4; | 501 | offset = (addr - EIOINTC_COREMAP_START) / 4; |
| 502 | p = s->coremap + offset * 4; | 502 | p = (void *)s->coremap + offset * 4; |
| 503 | break; | 503 | break; |
| 504 | default: | 504 | default: |
| 505 | kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr); | 505 | kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr); |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | |||
| 14 | #define CREATE_TRACE_POINTS | 14 | #define CREATE_TRACE_POINTS |
| 15 | #include "trace.h" | 15 | #include "trace.h" |
| 16 | 16 | ||
| 17 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 17 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 18 | KVM_GENERIC_VCPU_STATS(), | 18 | KVM_GENERIC_VCPU_STATS(), |
| 19 | STATS_DESC_COUNTER(VCPU, int_exits), | 19 | STATS_DESC_COUNTER(VCPU, int_exits), |
| 20 | STATS_DESC_COUNTER(VCPU, idle_exits), | 20 | STATS_DESC_COUNTER(VCPU, idle_exits), |
| @@ -588,6 +588,9 @@ struct kvm_vcpu *kvm_get_vcpu_by_cpuid(struct kvm *kvm, int cpuid) · 567 unmodified lines | |||
| 588 | { | 588 | { |
| 589 | struct kvm_phyid_map *map; | 589 | struct kvm_phyid_map *map; |
| 590 | 590 | ||
| 591 | if (cpuid < 0) | ||
| 592 | return NULL; | ||
| 593 | |||
| 591 | if (cpuid >= KVM_MAX_PHYID) | 594 | if (cpuid >= KVM_MAX_PHYID) |
| 592 | return NULL; | 595 | return NULL; |
| 593 | 596 | ||
| @@ -10,7 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <asm/kvm_eiointc.h> | 10 | #include <asm/kvm_eiointc.h> |
| 11 | #include <asm/kvm_pch_pic.h> | 11 | #include <asm/kvm_pch_pic.h> |
| 12 | 12 | ||
| 13 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 13 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 14 | KVM_GENERIC_VM_STATS(), | 14 | KVM_GENERIC_VM_STATS(), |
| 15 | STATS_DESC_ICOUNTER(VM, pages), | 15 | STATS_DESC_ICOUNTER(VM, pages), |
| 16 | STATS_DESC_ICOUNTER(VM, hugepages), | 16 | STATS_DESC_ICOUNTER(VM, hugepages), |
| @@ -49,8 +49,8 @@ static void kvm_vm_init_features(struct kvm *kvm) · 32 unmodified lines | |||
| 49 | kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PMU); | 49 | kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PMU); |
| 50 | 50 | ||
| 51 | /* Enable all PV features by default */ | 51 | /* Enable all PV features by default */ |
| 52 | kvm->arch.pv_features = BIT(KVM_FEATURE_IPI); | 52 | kvm->arch.pv_features |= BIT(KVM_FEATURE_IPI); |
| 53 | kvm->arch.kvm_features = BIT(KVM_LOONGARCH_VM_FEAT_PV_IPI); | 53 | kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_IPI); |
| 54 | if (kvm_pvtime_supported()) { | 54 | if (kvm_pvtime_supported()) { |
| 55 | kvm->arch.pv_features |= BIT(KVM_FEATURE_PREEMPT); | 55 | kvm->arch.pv_features |= BIT(KVM_FEATURE_PREEMPT); |
| 56 | kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME); | 56 | kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME); |
| @@ -1379,9 +1379,11 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len) · 1378 unmodified lines | |||
| 1379 | { | 1379 | { |
| 1380 | int ret; | 1380 | int ret; |
| 1381 | 1381 | ||
| 1382 | cpus_read_lock(); | ||
| 1382 | mutex_lock(&text_mutex); | 1383 | mutex_lock(&text_mutex); |
| 1383 | ret = larch_insn_text_copy(dst, src, len); | 1384 | ret = larch_insn_text_copy(dst, src, len); |
| 1384 | mutex_unlock(&text_mutex); | 1385 | mutex_unlock(&text_mutex); |
| 1386 | cpus_read_unlock(); | ||
| 1385 | 1387 | ||
| 1386 | return ret ? ERR_PTR(-EINVAL) : dst; | 1388 | return ret ? ERR_PTR(-EINVAL) : dst; |
| 1387 | } | 1389 | } |
| @@ -1429,10 +1431,12 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t, · 41 unmodified lines | |||
| 1429 | if (ret) | 1431 | if (ret) |
| 1430 | return ret; | 1432 | return ret; |
| 1431 | 1433 | ||
| 1434 | cpus_read_lock(); | ||
| 1432 | mutex_lock(&text_mutex); | 1435 | mutex_lock(&text_mutex); |
| 1433 | if (memcmp(ip, new_insns, LOONGARCH_LONG_JUMP_NBYTES)) | 1436 | if (memcmp(ip, new_insns, LOONGARCH_LONG_JUMP_NBYTES)) |
| 1434 | ret = larch_insn_text_copy(ip, new_insns, LOONGARCH_LONG_JUMP_NBYTES); | 1437 | ret = larch_insn_text_copy(ip, new_insns, LOONGARCH_LONG_JUMP_NBYTES); |
| 1435 | mutex_unlock(&text_mutex); | 1438 | mutex_unlock(&text_mutex); |
| 1439 | cpus_read_unlock(); | ||
| 1436 | 1440 | ||
| 1437 | return ret; | 1441 | return ret; |
| 1438 | } | 1442 | } |
| @@ -1450,10 +1454,12 @@ int bpf_arch_text_invalidate(void *dst, size_t len) · 11 unmodified lines | |||
| 1450 | for (i = 0; i < (len / sizeof(u32)); i++) | 1454 | for (i = 0; i < (len / sizeof(u32)); i++) |
| 1451 | inst[i] = INSN_BREAK; | 1455 | inst[i] = INSN_BREAK; |
| 1452 | 1456 | ||
| 1457 | cpus_read_lock(); | ||
| 1453 | mutex_lock(&text_mutex); | 1458 | mutex_lock(&text_mutex); |
| 1454 | if (larch_insn_text_copy(dst, inst, len)) | 1459 | if (larch_insn_text_copy(dst, inst, len)) |
| 1455 | ret = -EINVAL; | 1460 | ret = -EINVAL; |
| 1456 | mutex_unlock(&text_mutex); | 1461 | mutex_unlock(&text_mutex); |
| 1462 | cpus_read_unlock(); | ||
| 1457 | 1463 | ||
| 1458 | kvfree(inst); | 1464 | kvfree(inst); |
| 1459 | 1465 | ||
| @@ -1568,6 +1574,11 @@ void arch_free_bpf_trampoline(void *image, unsigned int size) · 108 unmodified lines | |||
| 1568 | bpf_prog_pack_free(image, size); | 1574 | bpf_prog_pack_free(image, size); |
| 1569 | } | 1575 | } |
| 1570 | 1576 | ||
| 1577 | int arch_protect_bpf_trampoline(void *image, unsigned int size) | ||
| 1578 | { | ||
| 1579 | return 0; | ||
| 1580 | } | ||
| 1581 | |||
| 1571 | /* | 1582 | /* |
| 1572 | * Sign-extend the register if necessary | 1583 | * Sign-extend the register if necessary |
| 1573 | */ | 1584 | */ |
| @@ -5,9 +5,11 @@ · 4 unmodified lines | |||
| 5 | #include <linux/kernel.h> | 5 | #include <linux/kernel.h> |
| 6 | #include <linux/init.h> | 6 | #include <linux/init.h> |
| 7 | #include <linux/acpi.h> | 7 | #include <linux/acpi.h> |
| 8 | #include <linux/delay.h> | ||
| 8 | #include <linux/types.h> | 9 | #include <linux/types.h> |
| 9 | #include <linux/pci.h> | 10 | #include <linux/pci.h> |
| 10 | #include <linux/vgaarb.h> | 11 | #include <linux/vgaarb.h> |
| 12 | #include <linux/io-64-nonatomic-lo-hi.h> | ||
| 11 | #include <asm/cacheflush.h> | 13 | #include <asm/cacheflush.h> |
| 12 | #include <asm/loongson.h> | 14 | #include <asm/loongson.h> |
| 13 | 15 | ||
| @@ -15,6 +17,9 @@ · 1 unmodified line | |||
| 15 | #define PCI_DEVICE_ID_LOONGSON_DC1 0x7a06 | 17 | #define PCI_DEVICE_ID_LOONGSON_DC1 0x7a06 |
| 16 | #define PCI_DEVICE_ID_LOONGSON_DC2 0x7a36 | 18 | #define PCI_DEVICE_ID_LOONGSON_DC2 0x7a36 |
| 17 | #define PCI_DEVICE_ID_LOONGSON_DC3 0x7a46 | 19 | #define PCI_DEVICE_ID_LOONGSON_DC3 0x7a46 |
| 20 | #define PCI_DEVICE_ID_LOONGSON_GPU1 0x7a15 | ||
| 21 | #define PCI_DEVICE_ID_LOONGSON_GPU2 0x7a25 | ||
| 22 | #define PCI_DEVICE_ID_LOONGSON_GPU3 0x7a35 | ||
| 18 | 23 | ||
| 19 | int raw_pci_read(unsigned int domain, unsigned int bus, unsigned int devfn, | 24 | int raw_pci_read(unsigned int domain, unsigned int bus, unsigned int devfn, |
| 20 | int reg, int len, u32 *val) | 25 | int reg, int len, u32 *val) |
| @@ -99,3 +104,78 @@ static void pci_fixup_vgadev(struct pci_dev *pdev) · 78 unmodified lines | |||
| 99 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC1, pci_fixup_vgadev); | 104 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC1, pci_fixup_vgadev); |
| 100 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC2, pci_fixup_vgadev); | 105 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC2, pci_fixup_vgadev); |
| 101 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC3, pci_fixup_vgadev); | 106 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_DC3, pci_fixup_vgadev); |
| 107 | |||
| 108 | #define CRTC_NUM_MAX 2 | ||
| 109 | #define CRTC_OUTPUT_ENABLE 0x100 | ||
| 110 | |||
| 111 | static void loongson_gpu_fixup_dma_hang(struct pci_dev *pdev, bool on) | ||
| 112 | { | ||
| 113 | u32 i, val, count, crtc_offset, device; | ||
| 114 | void __iomem *crtc_reg, *base, *regbase; | ||
| 115 | static u32 crtc_status[CRTC_NUM_MAX] = { 0 }; | ||
| 116 | |||
| 117 | base = pdev->bus->ops->map_bus(pdev->bus, pdev->devfn + 1, 0); | ||
| 118 | device = readw(base + PCI_DEVICE_ID); | ||
| 119 | |||
| 120 | regbase = ioremap(readq(base + PCI_BASE_ADDRESS_0) & ~0xffull, SZ_64K); | ||
| 121 | if (!regbase) { | ||
| 122 | pci_err(pdev, "Failed to ioremap()\n"); | ||
| 123 | return; | ||
| 124 | } | ||
| 125 | |||
| 126 | switch (device) { | ||
| 127 | case PCI_DEVICE_ID_LOONGSON_DC2: | ||
| 128 | crtc_reg = regbase + 0x1240; | ||
| 129 | crtc_offset = 0x10; | ||
| 130 | break; | ||
| 131 | case PCI_DEVICE_ID_LOONGSON_DC3: | ||
| 132 | crtc_reg = regbase; | ||
| 133 | crtc_offset = 0x400; | ||
| 134 | break; | ||
| 135 | } | ||
| 136 | |||
| 137 | for (i = 0; i < CRTC_NUM_MAX; i++, crtc_reg += crtc_offset) { | ||
| 138 | val = readl(crtc_reg); | ||
| 139 | |||
| 140 | if (!on) | ||
| 141 | crtc_status[i] = val; | ||
| 142 | |||
| 143 | /* No need to fixup if the status is off at startup. */ | ||
| 144 | if (!(crtc_status[i] & CRTC_OUTPUT_ENABLE)) | ||
| 145 | continue; | ||
| 146 | |||
| 147 | if (on) | ||
| 148 | val |= CRTC_OUTPUT_ENABLE; | ||
| 149 | else | ||
| 150 | val &= ~CRTC_OUTPUT_ENABLE; | ||
| 151 | |||
| 152 | mb(); | ||
| 153 | writel(val, crtc_reg); | ||
| 154 | |||
| 155 | for (count = 0; count < 40; count++) { | ||
| 156 | val = readl(crtc_reg) & CRTC_OUTPUT_ENABLE; | ||
| 157 | if ((on && val) || (!on && !val)) | ||
| 158 | break; | ||
| 159 | udelay(1000); | ||
| 160 | } | ||
| 161 | |||
| 162 | pci_info(pdev, "DMA hang fixup at reg[0x%lx]: 0x%x\n", | ||
| 163 | (unsigned long)crtc_reg & 0xffff, readl(crtc_reg)); | ||
| 164 | } | ||
| 165 | |||
| 166 | iounmap(regbase); | ||
| 167 | } | ||
| 168 | |||
| 169 | static void pci_fixup_dma_hang_early(struct pci_dev *pdev) | ||
| 170 | { | ||
| 171 | loongson_gpu_fixup_dma_hang(pdev, false); | ||
| 172 | } | ||
| 173 | DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU2, pci_fixup_dma_hang_early); | ||
| 174 | DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU3, pci_fixup_dma_hang_early); | ||
| 175 | |||
| 176 | static void pci_fixup_dma_hang_final(struct pci_dev *pdev) | ||
| 177 | { | ||
| 178 | loongson_gpu_fixup_dma_hang(pdev, true); | ||
| 179 | } | ||
| 180 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU2, pci_fixup_dma_hang_final); | ||
| 181 | DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_DEVICE_ID_LOONGSON_GPU3, pci_fixup_dma_hang_final); | ||
| @@ -26,7 +26,7 @@ cflags-vdso := $(ccflags-vdso) \ · 25 unmodified lines | |||
| 26 | $(filter -W%,$(filter-out -Wa$(comma)%,$(KBUILD_CFLAGS))) \ | 26 | $(filter -W%,$(filter-out -Wa$(comma)%,$(KBUILD_CFLAGS))) \ |
| 27 | -std=gnu11 -fms-extensions -O2 -g -fno-strict-aliasing -fno-common -fno-builtin \ | 27 | -std=gnu11 -fms-extensions -O2 -g -fno-strict-aliasing -fno-common -fno-builtin \ |
| 28 | -fno-stack-protector -fno-jump-tables -DDISABLE_BRANCH_PROFILING \ | 28 | -fno-stack-protector -fno-jump-tables -DDISABLE_BRANCH_PROFILING \ |
| 29 | $(call cc-option, -fno-asynchronous-unwind-tables) \ | 29 | $(call cc-option, -fasynchronous-unwind-tables) \ |
| 30 | $(call cc-option, -fno-stack-protector) | 30 | $(call cc-option, -fno-stack-protector) |
| 31 | aflags-vdso := $(ccflags-vdso) \ | 31 | aflags-vdso := $(ccflags-vdso) \ |
| 32 | -D__ASSEMBLY__ -Wa,-gdwarf-2 | 32 | -D__ASSEMBLY__ -Wa,-gdwarf-2 |
| @@ -41,7 +41,7 @@ endif · 8 unmodified lines | |||
| 41 | 41 | ||
| 42 | # VDSO linker flags. | 42 | # VDSO linker flags. |
| 43 | ldflags-y := -Bsymbolic --no-undefined -soname=linux-vdso.so.1 \ | 43 | ldflags-y := -Bsymbolic --no-undefined -soname=linux-vdso.so.1 \ |
| 44 | $(filter -E%,$(KBUILD_CFLAGS)) -shared --build-id -T | 44 | $(filter -E%,$(KBUILD_CFLAGS)) -shared --build-id --eh-frame-hdr -T |
| 45 | 45 | ||
| 46 | # | 46 | # |
| 47 | # Shared build commands. | 47 | # Shared build commands. |
| @@ -12,13 +12,13 @@ · 11 unmodified lines | |||
| 12 | 12 | ||
| 13 | #include <asm/regdef.h> | 13 | #include <asm/regdef.h> |
| 14 | #include <asm/asm.h> | 14 | #include <asm/asm.h> |
| 15 | #include <asm/asm-offsets.h> | ||
| 15 | 16 | ||
| 16 | .section .text | 17 | .section .text |
| 17 | .cfi_sections .debug_frame | ||
| 18 | 18 | ||
| 19 | SYM_FUNC_START(__vdso_rt_sigreturn) | 19 | SYM_SIGFUNC_START(__vdso_rt_sigreturn) |
| 20 | 20 | ||
| 21 | li.w a7, __NR_rt_sigreturn | 21 | li.w a7, __NR_rt_sigreturn |
| 22 | syscall 0 | 22 | syscall 0 |
| 23 | 23 | ||
| 24 | SYM_FUNC_END(__vdso_rt_sigreturn) | 24 | SYM_SIGFUNC_END(__vdso_rt_sigreturn) |
| @@ -484,7 +484,6 @@ · 483 unmodified lines | |||
| 484 | # endif | 484 | # endif |
| 485 | # ifndef cpu_vmbits | 485 | # ifndef cpu_vmbits |
| 486 | # define cpu_vmbits cpu_data[0].vmbits | 486 | # define cpu_vmbits cpu_data[0].vmbits |
| 487 | # define __NEED_VMBITS_PROBE | ||
| 488 | # endif | 487 | # endif |
| 489 | #endif | 488 | #endif |
| 490 | 489 | ||
| @@ -80,9 +80,7 @@ struct cpuinfo_mips { · 79 unmodified lines | |||
| 80 | int srsets; /* Shadow register sets */ | 80 | int srsets; /* Shadow register sets */ |
| 81 | int package;/* physical package number */ | 81 | int package;/* physical package number */ |
| 82 | unsigned int globalnumber; | 82 | unsigned int globalnumber; |
| 83 | #ifdef CONFIG_64BIT | ||
| 84 | int vmbits; /* Virtual memory size in bits */ | 83 | int vmbits; /* Virtual memory size in bits */ |
| 85 | #endif | ||
| 86 | void *data; /* Additional data */ | 84 | void *data; /* Additional data */ |
| 87 | unsigned int watch_reg_count; /* Number that exist */ | 85 | unsigned int watch_reg_count; /* Number that exist */ |
| 88 | unsigned int watch_reg_use_cnt; /* Usable by ptrace */ | 86 | unsigned int watch_reg_use_cnt; /* Usable by ptrace */ |
| @@ -1871,6 +1871,8 @@ do { \ · 1870 unmodified lines | |||
| 1871 | 1871 | ||
| 1872 | #define read_c0_entryhi() __read_ulong_c0_register($10, 0) | 1872 | #define read_c0_entryhi() __read_ulong_c0_register($10, 0) |
| 1873 | #define write_c0_entryhi(val) __write_ulong_c0_register($10, 0, val) | 1873 | #define write_c0_entryhi(val) __write_ulong_c0_register($10, 0, val) |
| 1874 | #define read_c0_entryhi_64() __read_64bit_c0_register($10, 0) | ||
| 1875 | #define write_c0_entryhi_64(val) __write_64bit_c0_register($10, 0, val) | ||
| 1874 | 1876 | ||
| 1875 | #define read_c0_guestctl1() __read_32bit_c0_register($10, 4) | 1877 | #define read_c0_guestctl1() __read_32bit_c0_register($10, 4) |
| 1876 | #define write_c0_guestctl1(val) __write_32bit_c0_register($10, 4, val) | 1878 | #define write_c0_guestctl1(val) __write_32bit_c0_register($10, 4, val) |
| @@ -210,11 +210,14 @@ static inline void set_elf_base_platform(const char *plat) · 209 unmodified lines | |||
| 210 | 210 | ||
| 211 | static inline void cpu_probe_vmbits(struct cpuinfo_mips *c) | 211 | static inline void cpu_probe_vmbits(struct cpuinfo_mips *c) |
| 212 | { | 212 | { |
| 213 | #ifdef __NEED_VMBITS_PROBE | 213 | int vmbits = 31; |
| 214 | write_c0_entryhi(0x3fffffffffffe000ULL); | 214 | |
| 215 | back_to_back_c0_hazard(); | 215 | if (cpu_has_64bits) { |
| 216 | c->vmbits = fls64(read_c0_entryhi() & 0x3fffffffffffe000ULL); | 216 | write_c0_entryhi_64(0x3fffffffffffe000ULL); |
| 217 | #endif | 217 | back_to_back_c0_hazard(); |
| 218 | vmbits = fls64(read_c0_entryhi_64() & 0x3fffffffffffe000ULL); | ||
| 219 | } | ||
| 220 | c->vmbits = vmbits; | ||
| 218 | } | 221 | } |
| 219 | 222 | ||
| 220 | static void set_isa(struct cpuinfo_mips *c, unsigned int isa) | 223 | static void set_isa(struct cpuinfo_mips *c, unsigned int isa) |
| @@ -137,6 +137,8 @@ void cpu_probe(void) · 136 unmodified lines | |||
| 137 | else | 137 | else |
| 138 | cpu_set_nofpu_opts(c); | 138 | cpu_set_nofpu_opts(c); |
| 139 | 139 | ||
| 140 | c->vmbits = 31; | ||
| 141 | |||
| 140 | reserve_exception_space(0, 0x400); | 142 | reserve_exception_space(0, 0x400); |
| 141 | } | 143 | } |
| 142 | 144 | ||
| @@ -38,7 +38,7 @@ · 37 unmodified lines | |||
| 38 | #define VECTORSPACING 0x100 /* for EI/VI mode */ | 38 | #define VECTORSPACING 0x100 /* for EI/VI mode */ |
| 39 | #endif | 39 | #endif |
| 40 | 40 | ||
| 41 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 41 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 42 | KVM_GENERIC_VM_STATS() | 42 | KVM_GENERIC_VM_STATS() |
| 43 | }; | 43 | }; |
| 44 | 44 | ||
| @@ -51,7 +51,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 6 unmodified lines | |||
| 51 | sizeof(kvm_vm_stats_desc), | 51 | sizeof(kvm_vm_stats_desc), |
| 52 | }; | 52 | }; |
| 53 | 53 | ||
| 54 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 54 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 55 | KVM_GENERIC_VCPU_STATS(), | 55 | KVM_GENERIC_VCPU_STATS(), |
| 56 | STATS_DESC_COUNTER(VCPU, wait_exits), | 56 | STATS_DESC_COUNTER(VCPU, wait_exits), |
| 57 | STATS_DESC_COUNTER(VCPU, cache_exits), | 57 | STATS_DESC_COUNTER(VCPU, cache_exits), |
| @@ -4,12 +4,12 @@ · 3 unmodified lines | |||
| 4 | #include "libgcc.h" | 4 | #include "libgcc.h" |
| 5 | 5 | ||
| 6 | /* | 6 | /* |
| 7 | * GCC 7 & older can suboptimally generate __multi3 calls for mips64r6, so for | 7 | * GCC 9 & older can suboptimally generate __multi3 calls for mips64r6, so for |
| 8 | * that specific case only we implement that intrinsic here. | 8 | * that specific case only we implement that intrinsic here. |
| 9 | * | 9 | * |
| 10 | * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82981 | 10 | * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82981 |
| 11 | */ | 11 | */ |
| 12 | #if defined(CONFIG_64BIT) && defined(CONFIG_CPU_MIPSR6) && (__GNUC__ < 8) | 12 | #if defined(CONFIG_64BIT) && defined(CONFIG_CPU_MIPSR6) && (__GNUC__ < 10) |
| 13 | 13 | ||
| 14 | /* multiply 64-bit values, low 64-bits returned */ | 14 | /* multiply 64-bit values, low 64-bits returned */ |
| 15 | static inline long long notrace dmulu(long long a, long long b) | 15 | static inline long long notrace dmulu(long long a, long long b) |
| @@ -51,4 +51,4 @@ ti_type notrace __multi3(ti_type a, ti_type b) · 35 unmodified lines | |||
| 51 | } | 51 | } |
| 52 | EXPORT_SYMBOL(__multi3); | 52 | EXPORT_SYMBOL(__multi3); |
| 53 | 53 | ||
| 54 | #endif /* 64BIT && CPU_MIPSR6 && GCC7 */ | 54 | #endif /* 64BIT && CPU_MIPSR6 && GCC9 */ |
| @@ -17,7 +17,9 @@ · 16 unmodified lines | |||
| 17 | #include <linux/dma-map-ops.h> | 17 | #include <linux/dma-map-ops.h> |
| 18 | #include <linux/export.h> | 18 | #include <linux/export.h> |
| 19 | #include <linux/libfdt.h> | 19 | #include <linux/libfdt.h> |
| 20 | #include <linux/minmax.h> | ||
| 20 | #include <linux/pci_ids.h> | 21 | #include <linux/pci_ids.h> |
| 22 | #include <linux/serial_core.h> | ||
| 21 | #include <linux/string_choices.h> | 23 | #include <linux/string_choices.h> |
| 22 | #include <asm/bootinfo.h> | 24 | #include <asm/bootinfo.h> |
| 23 | #include <loongson.h> | 25 | #include <loongson.h> |
| @@ -106,9 +108,23 @@ static void __init lefi_fixup_fdt(struct system_loongson *system) · 82 unmodified lines | |||
| 106 | 108 | ||
| 107 | is_loongson64g = (read_c0_prid() & PRID_IMP_MASK) == PRID_IMP_LOONGSON_64G; | 109 | is_loongson64g = (read_c0_prid() & PRID_IMP_MASK) == PRID_IMP_LOONGSON_64G; |
| 108 | 110 | ||
| 109 | for (i = 0; i < system->nr_uarts; i++) { | 111 | for (i = 0; i < min(system->nr_uarts, MAX_UARTS); i++) { |
| 110 | uartdev = &system->uarts[i]; | 112 | uartdev = &system->uarts[i]; |
| 111 | 113 | ||
| 114 | /* | ||
| 115 | * Some firmware does not set nr_uarts properly and passes empty | ||
| 116 | * items. Ignore them silently. | ||
| 117 | */ | ||
| 118 | if (uartdev->uart_base == 0) | ||
| 119 | continue; | ||
| 120 | |||
| 121 | /* Our DT only works with UPIO_MEM. */ | ||
| 122 | if (uartdev->iotype != UPIO_MEM) { | ||
| 123 | pr_warn("Ignore UART 0x%llx with iotype %u passed by firmware\n", | ||
| 124 | uartdev->uart_base, uartdev->iotype); | ||
| 125 | continue; | ||
| 126 | } | ||
| 127 | |||
| 112 | ret = lefi_fixup_fdt_serial(fdt_buf, uartdev->uart_base, | 128 | ret = lefi_fixup_fdt_serial(fdt_buf, uartdev->uart_base, |
| 113 | uartdev->uartclk); | 129 | uartdev->uartclk); |
| 114 | /* | 130 | /* |
| @@ -207,7 +207,8 @@ void cpu_cache_init(void) · 206 unmodified lines | |||
| 207 | { | 207 | { |
| 208 | if (IS_ENABLED(CONFIG_CPU_R3000) && cpu_has_3k_cache) | 208 | if (IS_ENABLED(CONFIG_CPU_R3000) && cpu_has_3k_cache) |
| 209 | r3k_cache_init(); | 209 | r3k_cache_init(); |
| 210 | if (IS_ENABLED(CONFIG_CPU_R4K_CACHE_TLB) && cpu_has_4k_cache) | 210 | if ((IS_ENABLED(CONFIG_CPU_R4K_CACHE_TLB) || |
| 211 | IS_ENABLED(CONFIG_CPU_SB1)) && cpu_has_4k_cache) | ||
| 211 | r4k_cache_init(); | 212 | r4k_cache_init(); |
| 212 | 213 | ||
| 213 | if (IS_ENABLED(CONFIG_CPU_CAVIUM_OCTEON) && cpu_has_octeon_cache) | 214 | if (IS_ENABLED(CONFIG_CPU_CAVIUM_OCTEON) && cpu_has_octeon_cache) |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | |||
| 13 | #include <linux/sched.h> | 13 | #include <linux/sched.h> |
| 14 | #include <linux/smp.h> | 14 | #include <linux/smp.h> |
| 15 | #include <linux/memblock.h> | 15 | #include <linux/memblock.h> |
| 16 | #include <linux/minmax.h> | ||
| 16 | #include <linux/mm.h> | 17 | #include <linux/mm.h> |
| 17 | #include <linux/hugetlb.h> | 18 | #include <linux/hugetlb.h> |
| 18 | #include <linux/export.h> | 19 | #include <linux/export.h> |
| @@ -24,6 +25,7 @@ · 5 unmodified lines | |||
| 24 | #include <asm/hazards.h> | 25 | #include <asm/hazards.h> |
| 25 | #include <asm/mmu_context.h> | 26 | #include <asm/mmu_context.h> |
| 26 | #include <asm/tlb.h> | 27 | #include <asm/tlb.h> |
| 28 | #include <asm/tlbdebug.h> | ||
| 27 | #include <asm/tlbex.h> | 29 | #include <asm/tlbex.h> |
| 28 | #include <asm/tlbmisc.h> | 30 | #include <asm/tlbmisc.h> |
| 29 | #include <asm/setup.h> | 31 | #include <asm/setup.h> |
| @@ -511,87 +513,259 @@ static int __init set_ntlb(char *str) · 481 unmodified lines | |||
| 511 | __setup("ntlb=", set_ntlb); | 513 | __setup("ntlb=", set_ntlb); |
| 512 | 514 | ||
| 513 | 515 | ||
| 514 | /* Comparison function for EntryHi VPN fields. */ | 516 | /* The start bit position of VPN2 and Mask in EntryHi/PageMask registers. */ |
| 515 | static int r4k_vpn_cmp(const void *a, const void *b) | 517 | #define VPN2_SHIFT 13 |
| 518 | |||
| 519 | /* Read full EntryHi even with CONFIG_32BIT. */ | ||
| 520 | static inline unsigned long long read_c0_entryhi_native(void) | ||
| 521 | { | ||
| 522 | return cpu_has_64bits ? read_c0_entryhi_64() : read_c0_entryhi(); | ||
| 523 | } | ||
| 524 | |||
| 525 | /* Write full EntryHi even with CONFIG_32BIT. */ | ||
| 526 | static inline void write_c0_entryhi_native(unsigned long long v) | ||
| 516 | { | 527 | { |
| 517 | long v = *(unsigned long *)a - *(unsigned long *)b; | 528 | if (cpu_has_64bits) |
| 518 | int s = sizeof(long) > sizeof(int) ? sizeof(long) * 8 - 1: 0; | 529 | write_c0_entryhi_64(v); |
| 519 | return s ? (v != 0) | v >> s : v; | 530 | else |
| 531 | write_c0_entryhi(v); | ||
| 520 | } | 532 | } |
| 521 | 533 | ||
| 534 | /* TLB entry state for uniquification. */ | ||
| 535 | struct tlbent { | ||
| 536 | unsigned long long wired:1; | ||
| 537 | unsigned long long global:1; | ||
| 538 | unsigned long long asid:10; | ||
| 539 | unsigned long long vpn:51; | ||
| 540 | unsigned long long pagesz:5; | ||
| 541 | unsigned long long index:14; | ||
| 542 | }; | ||
| 543 | |||
| 522 | /* | 544 | /* |
| 523 | * Initialise all TLB entries with unique values that do not clash with | 545 | * Comparison function for TLB entry sorting. Place wired entries first, |
| 524 | * what we have been handed over and what we'll be using ourselves. | 546 | * then global entries, then order by the increasing VPN/ASID and the |
| 547 | * decreasing page size. This lets us avoid clashes with wired entries | ||
| 548 | * easily and get entries for larger pages out of the way first. | ||
| 549 | * | ||
| 550 | * We could group bits so as to reduce the number of comparisons, but this | ||
| 551 | * is seldom executed and not performance-critical, so prefer legibility. | ||
| 525 | */ | 552 | */ |
| 526 | static void __ref r4k_tlb_uniquify(void) | 553 | static int r4k_entry_cmp(const void *a, const void *b) |
| 527 | { | 554 | { |
| 528 | int tlbsize = current_cpu_data.tlbsize; | 555 | struct tlbent ea = *(struct tlbent *)a, eb = *(struct tlbent *)b; |
| 529 | bool use_slab = slab_is_available(); | 556 | |
| 530 | int start = num_wired_entries(); | 557 | if (ea.wired > eb.wired) |
| 531 | phys_addr_t tlb_vpn_size; | 558 | return -1; |
| 532 | unsigned long *tlb_vpns; | 559 | else if (ea.wired < eb.wired) |
| 533 | unsigned long vpn_mask; | 560 | return 1; |
| 534 | int cnt, ent, idx, i; | 561 | else if (ea.global > eb.global) |
| 535 | 562 | return -1; | |
| 536 | vpn_mask = GENMASK(cpu_vmbits - 1, 13); | 563 | else if (ea.global < eb.global) |
| 537 | vpn_mask |= IS_ENABLED(CONFIG_64BIT) ? 3ULL << 62 : 1 << 31; | 564 | return 1; |
| 565 | else if (ea.vpn < eb.vpn) | ||
| 566 | return -1; | ||
| 567 | else if (ea.vpn > eb.vpn) | ||
| 568 | return 1; | ||
| 569 | else if (ea.asid < eb.asid) | ||
| 570 | return -1; | ||
| 571 | else if (ea.asid > eb.asid) | ||
| 572 | return 1; | ||
| 573 | else if (ea.pagesz > eb.pagesz) | ||
| 574 | return -1; | ||
| 575 | else if (ea.pagesz < eb.pagesz) | ||
| 576 | return 1; | ||
| 577 | else | ||
| 578 | return 0; | ||
| 579 | } | ||
| 538 | 580 | ||
| 539 | tlb_vpn_size = tlbsize * sizeof(*tlb_vpns); | 581 | /* |
| 540 | tlb_vpns = (use_slab ? | 582 | * Fetch all the TLB entries. Mask individual VPN values retrieved with |
| 541 | kmalloc(tlb_vpn_size, GFP_KERNEL) : | 583 | * the corresponding page mask and ignoring any 1KiB extension as we'll |
| 542 | memblock_alloc_raw(tlb_vpn_size, sizeof(*tlb_vpns))); | 584 | * be using 4KiB pages for uniquification. |
| 543 | if (WARN_ON(!tlb_vpns)) | 585 | */ |
| 544 | return; /* Pray local_flush_tlb_all() is good enough. */ | 586 | static void __ref r4k_tlb_uniquify_read(struct tlbent *tlb_vpns, int tlbsize) |
| 587 | { | ||
| 588 | int start = num_wired_entries(); | ||
| 589 | unsigned long long vpn_mask; | ||
| 590 | bool global; | ||
| 591 | int i; | ||
| 545 | 592 | ||
| 546 | htw_stop(); | 593 | vpn_mask = GENMASK(current_cpu_data.vmbits - 1, VPN2_SHIFT); |
| 594 | vpn_mask |= cpu_has_64bits ? 3ULL << 62 : 1 << 31; | ||
| 547 | 595 | ||
| 548 | for (i = start, cnt = 0; i < tlbsize; i++, cnt++) { | 596 | for (i = 0; i < tlbsize; i++) { |
| 549 | unsigned long vpn; | 597 | unsigned long long entryhi, vpn, mask, asid; |
| 598 | unsigned int pagesz; | ||
| 550 | 599 | ||
| 551 | write_c0_index(i); | 600 | write_c0_index(i); |
| 552 | mtc0_tlbr_hazard(); | 601 | mtc0_tlbr_hazard(); |
| 553 | tlb_read(); | 602 | tlb_read(); |
| 554 | tlb_read_hazard(); | 603 | tlb_read_hazard(); |
| 555 | vpn = read_c0_entryhi(); | ||
| 556 | vpn &= vpn_mask & PAGE_MASK; | ||
| 557 | tlb_vpns[cnt] = vpn; | ||
| 558 | 604 | ||
| 559 | /* Prevent any large pages from overlapping regular ones. */ | 605 | global = !!(read_c0_entrylo0() & ENTRYLO_G); |
| 560 | write_c0_pagemask(read_c0_pagemask() & PM_DEFAULT_MASK); | 606 | entryhi = read_c0_entryhi_native(); |
| 561 | mtc0_tlbw_hazard(); | 607 | mask = read_c0_pagemask(); |
| 562 | tlb_write_indexed(); | 608 | |
| 563 | tlbw_use_hazard(); | 609 | asid = entryhi & cpu_asid_mask(¤t_cpu_data); |
| 610 | vpn = (entryhi & vpn_mask & ~mask) >> VPN2_SHIFT; | ||
| 611 | pagesz = ilog2((mask >> VPN2_SHIFT) + 1); | ||
| 612 | |||
| 613 | tlb_vpns[i].global = global; | ||
| 614 | tlb_vpns[i].asid = global ? 0 : asid; | ||
| 615 | tlb_vpns[i].vpn = vpn; | ||
| 616 | tlb_vpns[i].pagesz = pagesz; | ||
| 617 | tlb_vpns[i].wired = i < start; | ||
| 618 | tlb_vpns[i].index = i; | ||
| 564 | } | 619 | } |
| 620 | } | ||
| 621 | |||
| 622 | /* | ||
| 623 | * Write unique values to all but the wired TLB entries each, using | ||
| 624 | * the 4KiB page size. This size might not be supported with R6, but | ||
| 625 | * EHINV is mandatory for R6, so we won't ever be called in that case. | ||
| 626 | * | ||
| 627 | * A sorted table is supplied with any wired entries at the beginning, | ||
| 628 | * followed by any global entries, and then finally regular entries. | ||
| 629 | * We start at the VPN and ASID values of zero and only assign user | ||
| 630 | * addresses, therefore guaranteeing no clash with addresses produced | ||
| 631 | * by UNIQUE_ENTRYHI. We avoid any VPN values used by wired or global | ||
| 632 | * entries, by increasing the VPN value beyond the span of such entry. | ||
| 633 | * | ||
| 634 | * When a VPN/ASID clash is found with a regular entry we increment the | ||
| 635 | * ASID instead until no VPN/ASID clash has been found or the ASID space | ||
| 636 | * has been exhausted, in which case we increase the VPN value beyond | ||
| 637 | * the span of the largest clashing entry. | ||
| 638 | * | ||
| 639 | * We do not need to be concerned about FTLB or MMID configurations as | ||
| 640 | * those are required to implement the EHINV feature. | ||
| 641 | */ | ||
| 642 | static void __ref r4k_tlb_uniquify_write(struct tlbent *tlb_vpns, int tlbsize) | ||
| 643 | { | ||
| 644 | unsigned long long asid, vpn, vpn_size, pagesz; | ||
| 645 | int widx, gidx, idx, sidx, lidx, i; | ||
| 565 | 646 | ||
| 566 | sort(tlb_vpns, cnt, sizeof(tlb_vpns[0]), r4k_vpn_cmp, NULL); | 647 | vpn_size = 1ULL << (current_cpu_data.vmbits - VPN2_SHIFT); |
| 648 | pagesz = ilog2((PM_4K >> VPN2_SHIFT) + 1); | ||
| 567 | 649 | ||
| 568 | write_c0_pagemask(PM_DEFAULT_MASK); | 650 | write_c0_pagemask(PM_4K); |
| 569 | write_c0_entrylo0(0); | 651 | write_c0_entrylo0(0); |
| 570 | write_c0_entrylo1(0); | 652 | write_c0_entrylo1(0); |
| 571 | 653 | ||
| 572 | idx = 0; | 654 | asid = 0; |
| 573 | ent = tlbsize; | 655 | vpn = 0; |
| 574 | for (i = start; i < tlbsize; i++) | 656 | widx = 0; |
| 575 | while (1) { | 657 | gidx = 0; |
| 576 | unsigned long entryhi, vpn; | 658 | for (sidx = 0; sidx < tlbsize && tlb_vpns[sidx].wired; sidx++) |
| 659 | ; | ||
| 660 | for (lidx = sidx; lidx < tlbsize && tlb_vpns[lidx].global; lidx++) | ||
| 661 | ; | ||
| 662 | idx = gidx = sidx + 1; | ||
| 663 | for (i = sidx; i < tlbsize; i++) { | ||
| 664 | unsigned long long entryhi, vpn_pagesz = 0; | ||
| 577 | 665 | ||
| 578 | entryhi = UNIQUE_ENTRYHI(ent); | 666 | while (1) { |
| 579 | vpn = entryhi & vpn_mask & PAGE_MASK; | 667 | if (WARN_ON(vpn >= vpn_size)) { |
| 668 | dump_tlb_all(); | ||
| 669 | /* Pray local_flush_tlb_all() will cope. */ | ||
| 670 | return; | ||
| 671 | } | ||
| 580 | 672 | ||
| 581 | if (idx >= cnt || vpn < tlb_vpns[idx]) { | 673 | /* VPN must be below the next wired entry. */ |
| 582 | write_c0_entryhi(entryhi); | 674 | if (widx < sidx && vpn >= tlb_vpns[widx].vpn) { |
| 583 | write_c0_index(i); | 675 | vpn = max(vpn, |
| 584 | mtc0_tlbw_hazard(); | 676 | (tlb_vpns[widx].vpn + |
| 585 | tlb_write_indexed(); | 677 | (1ULL << tlb_vpns[widx].pagesz))); |
| 586 | ent++; | 678 | asid = 0; |
| 587 | break; | 679 | widx++; |
| 588 | } else if (vpn == tlb_vpns[idx]) { | 680 | continue; |
| 589 | ent++; | 681 | } |
| 590 | } else { | 682 | /* VPN must be below the next global entry. */ |
| 683 | if (gidx < lidx && vpn >= tlb_vpns[gidx].vpn) { | ||
| 684 | vpn = max(vpn, | ||
| 685 | (tlb_vpns[gidx].vpn + | ||
| 686 | (1ULL << tlb_vpns[gidx].pagesz))); | ||
| 687 | asid = 0; | ||
| 688 | gidx++; | ||
| 689 | continue; | ||
| 690 | } | ||
| 691 | /* Try to find a free ASID so as to conserve VPNs. */ | ||
| 692 | if (idx < tlbsize && vpn == tlb_vpns[idx].vpn && | ||
| 693 | asid == tlb_vpns[idx].asid) { | ||
| 694 | unsigned long long idx_pagesz; | ||
| 695 | |||
| 696 | idx_pagesz = tlb_vpns[idx].pagesz; | ||
| 697 | vpn_pagesz = max(vpn_pagesz, idx_pagesz); | ||
| 698 | do | ||
| 699 | idx++; | ||
| 700 | while (idx < tlbsize && | ||
| 701 | vpn == tlb_vpns[idx].vpn && | ||
| 702 | asid == tlb_vpns[idx].asid); | ||
| 703 | asid++; | ||
| 704 | if (asid > cpu_asid_mask(¤t_cpu_data)) { | ||
| 705 | vpn += vpn_pagesz; | ||
| 706 | asid = 0; | ||
| 707 | vpn_pagesz = 0; | ||
| 708 | } | ||
| 709 | continue; | ||
| 710 | } | ||
| 711 | /* VPN mustn't be above the next regular entry. */ | ||
| 712 | if (idx < tlbsize && vpn > tlb_vpns[idx].vpn) { | ||
| 713 | vpn = max(vpn, | ||
| 714 | (tlb_vpns[idx].vpn + | ||
| 715 | (1ULL << tlb_vpns[idx].pagesz))); | ||
| 716 | asid = 0; | ||
| 591 | idx++; | 717 | idx++; |
| 718 | continue; | ||
| 592 | } | 719 | } |
| 720 | break; | ||
| 593 | } | 721 | } |
| 594 | 722 | ||
| 723 | entryhi = (vpn << VPN2_SHIFT) | asid; | ||
| 724 | write_c0_entryhi_native(entryhi); | ||
| 725 | write_c0_index(tlb_vpns[i].index); | ||
| 726 | mtc0_tlbw_hazard(); | ||
| 727 | tlb_write_indexed(); | ||
| 728 | |||
| 729 | tlb_vpns[i].asid = asid; | ||
| 730 | tlb_vpns[i].vpn = vpn; | ||
| 731 | tlb_vpns[i].pagesz = pagesz; | ||
| 732 | |||
| 733 | asid++; | ||
| 734 | if (asid > cpu_asid_mask(¤t_cpu_data)) { | ||
| 735 | vpn += 1ULL << pagesz; | ||
| 736 | asid = 0; | ||
| 737 | } | ||
| 738 | } | ||
| 739 | } | ||
| 740 | |||
| 741 | /* | ||
| 742 | * Initialise all TLB entries with unique values that do not clash with | ||
| 743 | * what we have been handed over and what we'll be using ourselves. | ||
| 744 | */ | ||
| 745 | static void __ref r4k_tlb_uniquify(void) | ||
| 746 | { | ||
| 747 | int tlbsize = current_cpu_data.tlbsize; | ||
| 748 | bool use_slab = slab_is_available(); | ||
| 749 | phys_addr_t tlb_vpn_size; | ||
| 750 | struct tlbent *tlb_vpns; | ||
| 751 | |||
| 752 | tlb_vpn_size = tlbsize * sizeof(*tlb_vpns); | ||
| 753 | tlb_vpns = (use_slab ? | ||
| 754 | kmalloc(tlb_vpn_size, GFP_ATOMIC) : | ||
| 755 | memblock_alloc_raw(tlb_vpn_size, sizeof(*tlb_vpns))); | ||
| 756 | if (WARN_ON(!tlb_vpns)) | ||
| 757 | return; /* Pray local_flush_tlb_all() is good enough. */ | ||
| 758 | |||
| 759 | htw_stop(); | ||
| 760 | |||
| 761 | r4k_tlb_uniquify_read(tlb_vpns, tlbsize); | ||
| 762 | |||
| 763 | sort(tlb_vpns, tlbsize, sizeof(*tlb_vpns), r4k_entry_cmp, NULL); | ||
| 764 | |||
| 765 | r4k_tlb_uniquify_write(tlb_vpns, tlbsize); | ||
| 766 | |||
| 767 | write_c0_pagemask(PM_DEFAULT_MASK); | ||
| 768 | |||
| 595 | tlbw_use_hazard(); | 769 | tlbw_use_hazard(); |
| 596 | htw_start(); | 770 | htw_start(); |
| 597 | flush_micro_tlb(); | 771 | flush_micro_tlb(); |
| @@ -640,7 +814,8 @@ static void r4k_tlb_configure(void) · 42 unmodified lines | |||
| 640 | temp_tlb_entry = current_cpu_data.tlbsize - 1; | 814 | temp_tlb_entry = current_cpu_data.tlbsize - 1; |
| 641 | 815 | ||
| 642 | /* From this point on the ARC firmware is dead. */ | 816 | /* From this point on the ARC firmware is dead. */ |
| 643 | r4k_tlb_uniquify(); | 817 | if (!cpu_has_tlbinv) |
| 818 | r4k_tlb_uniquify(); | ||
| 644 | local_flush_tlb_all(); | 819 | local_flush_tlb_all(); |
| 645 | 820 | ||
| 646 | /* Did I tell you that ARC SUCKS? */ | 821 | /* Did I tell you that ARC SUCKS? */ |
| @@ -21,16 +21,16 @@ static const char *clk_cpu(int *idx) · 20 unmodified lines | |||
| 21 | { | 21 | { |
| 22 | switch (ralink_soc) { | 22 | switch (ralink_soc) { |
| 23 | case RT2880_SOC: | 23 | case RT2880_SOC: |
| 24 | *idx = 0; | 24 | *idx = 1; |
| 25 | return "ralink,rt2880-sysc"; | 25 | return "ralink,rt2880-sysc"; |
| 26 | case RT3883_SOC: | 26 | case RT3883_SOC: |
| 27 | *idx = 0; | 27 | *idx = 1; |
| 28 | return "ralink,rt3883-sysc"; | 28 | return "ralink,rt3883-sysc"; |
| 29 | case RT305X_SOC_RT3050: | 29 | case RT305X_SOC_RT3050: |
| 30 | *idx = 0; | 30 | *idx = 1; |
| 31 | return "ralink,rt3050-sysc"; | 31 | return "ralink,rt3050-sysc"; |
| 32 | case RT305X_SOC_RT3052: | 32 | case RT305X_SOC_RT3052: |
| 33 | *idx = 0; | 33 | *idx = 1; |
| 34 | return "ralink,rt3052-sysc"; | 34 | return "ralink,rt3052-sysc"; |
| 35 | case RT305X_SOC_RT3350: | 35 | case RT305X_SOC_RT3350: |
| 36 | *idx = 1; | 36 | *idx = 1; |
| @@ -953,7 +953,7 @@ SYSCALL_DEFINE3(cacheflush, unsigned long, addr, unsigned long, bytes, · 952 unmodified lines | |||
| 953 | #else | 953 | #else |
| 954 | "1: cmpb,<<,n %0,%2,1b\n" | 954 | "1: cmpb,<<,n %0,%2,1b\n" |
| 955 | #endif | 955 | #endif |
| 956 | " fic,m %3(%4,%0)\n" | 956 | " fdc,m %3(%4,%0)\n" |
| 957 | "2: sync\n" | 957 | "2: sync\n" |
| 958 | ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 2b, "%1") | 958 | ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 2b, "%1") |
| 959 | : "+r" (start), "+r" (error) | 959 | : "+r" (start), "+r" (error) |
| @@ -968,7 +968,7 @@ SYSCALL_DEFINE3(cacheflush, unsigned long, addr, unsigned long, bytes, · 8 unmodified lines | |||
| 968 | #else | 968 | #else |
| 969 | "1: cmpb,<<,n %0,%2,1b\n" | 969 | "1: cmpb,<<,n %0,%2,1b\n" |
| 970 | #endif | 970 | #endif |
| 971 | " fdc,m %3(%4,%0)\n" | 971 | " fic,m %3(%4,%0)\n" |
| 972 | "2: sync\n" | 972 | "2: sync\n" |
| 973 | ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 2b, "%1") | 973 | ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 2b, "%1") |
| 974 | : "+r" (start), "+r" (error) | 974 | : "+r" (start), "+r" (error) |
| @@ -573,8 +573,8 @@ config ARCH_USING_PATCHABLE_FUNCTION_ENTRY · 572 unmodified lines | |||
| 573 | depends on FUNCTION_TRACER && (PPC32 || PPC64_ELF_ABI_V2) | 573 | depends on FUNCTION_TRACER && (PPC32 || PPC64_ELF_ABI_V2) |
| 574 | depends on $(cc-option,-fpatchable-function-entry=2) | 574 | depends on $(cc-option,-fpatchable-function-entry=2) |
| 575 | def_bool y if PPC32 | 575 | def_bool y if PPC32 |
| 576 | def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh $(CC) -mlittle-endian) if PPC64 && CPU_LITTLE_ENDIAN | 576 | def_bool $(success,$(srctree)/arch/powerpc/tools/check-fpatchable-function-entry.sh $(CC) $(CLANG_FLAGS) -mlittle-endian) if PPC64 && CPU_LITTLE_ENDIAN |
| 577 | def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh $(CC) -mbig-endian) if PPC64 && CPU_BIG_ENDIAN | 577 | def_bool $(success,$(srctree)/arch/powerpc/tools/check-fpatchable-function-entry.sh $(CC) -mbig-endian) if PPC64 && CPU_BIG_ENDIAN |
| 578 | 578 | ||
| 579 | config PPC_FTRACE_OUT_OF_LINE | 579 | config PPC_FTRACE_OUT_OF_LINE |
| 580 | def_bool PPC64 && ARCH_USING_PATCHABLE_FUNCTION_ENTRY | 580 | def_bool PPC64 && ARCH_USING_PATCHABLE_FUNCTION_ENTRY |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | |||
| 37 | }; | 37 | }; |
| 38 | }; | 38 | }; |
| 39 | 39 | ||
| 40 | memory { | 40 | memory@0 { |
| 41 | device_type = "memory"; | 41 | device_type = "memory"; |
| 42 | reg = <0x00000000 0x8000000>; // 128MB at 0 | 42 | reg = <0x00000000 0x8000000>; // 128MB at 0 |
| 43 | }; | 43 | }; |
| @@ -1,156 +0,0 @@ | |||
| 1 | /* T4240 Interlaken LAC Portal device tree stub with 24 portals. | ||
| 2 | * | ||
| 3 | * Copyright 2012 Freescale Semiconductor Inc. | ||
| 4 | * | ||
| 5 | * Redistribution and use in source and binary forms, with or without | ||
| 6 | * modification, are permitted provided that the following conditions are met: | ||
| 7 | * * Redistributions of source code must retain the above copyright | ||
| 8 | * notice, this list of conditions and the following disclaimer. | ||
| 9 | * * Redistributions in binary form must reproduce the above copyright | ||
| 10 | * notice, this list of conditions and the following disclaimer in the | ||
| 11 | * documentation and/or other materials provided with the distribution. | ||
| 12 | * * Neither the name of Freescale Semiconductor nor the | ||
| 13 | * names of its contributors may be used to endorse or promote products | ||
| 14 | * derived from this software without specific prior written permission. | ||
| 15 | * | ||
| 16 | * | ||
| 17 | * ALTERNATIVELY, this software may be distributed under the terms of the | ||
| 18 | * GNU General Public License ("GPL") as published by the Free Software | ||
| 19 | * Foundation, either version 2 of that License or (at your option) any | ||
| 20 | * later version. | ||
| 21 | * | ||
| 22 | * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY | ||
| 23 | * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | ||
| 24 | * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
| 25 | * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | ||
| 26 | * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | ||
| 27 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | ||
| 28 | * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | ||
| 29 | * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| 30 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | ||
| 31 | * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
| 32 | */ | ||
| 33 | |||
| 34 | #address-cells = <0x1>; | ||
| 35 | #size-cells = <0x1>; | ||
| 36 | compatible = "fsl,interlaken-lac-portals"; | ||
| 37 | |||
| 38 | lportal0: lac-portal@0 { | ||
| 39 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 40 | reg = <0x0 0x1000>; | ||
| 41 | }; | ||
| 42 | |||
| 43 | lportal1: lac-portal@1000 { | ||
| 44 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 45 | reg = <0x1000 0x1000>; | ||
| 46 | }; | ||
| 47 | |||
| 48 | lportal2: lac-portal@2000 { | ||
| 49 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 50 | reg = <0x2000 0x1000>; | ||
| 51 | }; | ||
| 52 | |||
| 53 | lportal3: lac-portal@3000 { | ||
| 54 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 55 | reg = <0x3000 0x1000>; | ||
| 56 | }; | ||
| 57 | |||
| 58 | lportal4: lac-portal@4000 { | ||
| 59 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 60 | reg = <0x4000 0x1000>; | ||
| 61 | }; | ||
| 62 | |||
| 63 | lportal5: lac-portal@5000 { | ||
| 64 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 65 | reg = <0x5000 0x1000>; | ||
| 66 | }; | ||
| 67 | |||
| 68 | lportal6: lac-portal@6000 { | ||
| 69 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 70 | reg = <0x6000 0x1000>; | ||
| 71 | }; | ||
| 72 | |||
| 73 | lportal7: lac-portal@7000 { | ||
| 74 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 75 | reg = <0x7000 0x1000>; | ||
| 76 | }; | ||
| 77 | |||
| 78 | lportal8: lac-portal@8000 { | ||
| 79 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 80 | reg = <0x8000 0x1000>; | ||
| 81 | }; | ||
| 82 | |||
| 83 | lportal9: lac-portal@9000 { | ||
| 84 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 85 | reg = <0x9000 0x1000>; | ||
| 86 | }; | ||
| 87 | |||
| 88 | lportal10: lac-portal@A000 { | ||
| 89 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 90 | reg = <0xA000 0x1000>; | ||
| 91 | }; | ||
| 92 | |||
| 93 | lportal11: lac-portal@B000 { | ||
| 94 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 95 | reg = <0xB000 0x1000>; | ||
| 96 | }; | ||
| 97 | |||
| 98 | lportal12: lac-portal@C000 { | ||
| 99 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 100 | reg = <0xC000 0x1000>; | ||
| 101 | }; | ||
| 102 | |||
| 103 | lportal13: lac-portal@D000 { | ||
| 104 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 105 | reg = <0xD000 0x1000>; | ||
| 106 | }; | ||
| 107 | |||
| 108 | lportal14: lac-portal@E000 { | ||
| 109 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 110 | reg = <0xE000 0x1000>; | ||
| 111 | }; | ||
| 112 | |||
| 113 | lportal15: lac-portal@F000 { | ||
| 114 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 115 | reg = <0xF000 0x1000>; | ||
| 116 | }; | ||
| 117 | |||
| 118 | lportal16: lac-portal@10000 { | ||
| 119 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 120 | reg = <0x10000 0x1000>; | ||
| 121 | }; | ||
| 122 | |||
| 123 | lportal17: lac-portal@11000 { | ||
| 124 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 125 | reg = <0x11000 0x1000>; | ||
| 126 | }; | ||
| 127 | |||
| 128 | lportal18: lac-portal@1200 { | ||
| 129 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 130 | reg = <0x12000 0x1000>; | ||
| 131 | }; | ||
| 132 | |||
| 133 | lportal19: lac-portal@13000 { | ||
| 134 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 135 | reg = <0x13000 0x1000>; | ||
| 136 | }; | ||
| 137 | |||
| 138 | lportal20: lac-portal@14000 { | ||
| 139 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 140 | reg = <0x14000 0x1000>; | ||
| 141 | }; | ||
| 142 | |||
| 143 | lportal21: lac-portal@15000 { | ||
| 144 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 145 | reg = <0x15000 0x1000>; | ||
| 146 | }; | ||
| 147 | |||
| 148 | lportal22: lac-portal@16000 { | ||
| 149 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 150 | reg = <0x16000 0x1000>; | ||
| 151 | }; | ||
| 152 | |||
| 153 | lportal23: lac-portal@17000 { | ||
| 154 | compatible = "fsl,interlaken-lac-portal-v1.0"; | ||
| 155 | reg = <0x17000 0x1000>; | ||
| 156 | }; | ||
| @@ -1,45 +0,0 @@ | |||
| 1 | /* | ||
| 2 | * T4 Interlaken Look-aside Controller (LAC) device tree stub | ||
| 3 | * | ||
| 4 | * Copyright 2012 Freescale Semiconductor Inc. | ||
| 5 | * | ||
| 6 | * Redistribution and use in source and binary forms, with or without | ||
| 7 | * modification, are permitted provided that the following conditions are met: | ||
| 8 | * * Redistributions of source code must retain the above copyright | ||
| 9 | * notice, this list of conditions and the following disclaimer. | ||
| 10 | * * Redistributions in binary form must reproduce the above copyright | ||
| 11 | * notice, this list of conditions and the following disclaimer in the | ||
| 12 | * documentation and/or other materials provided with the distribution. | ||
| 13 | * * Neither the name of Freescale Semiconductor nor the | ||
| 14 | * names of its contributors may be used to endorse or promote products | ||
| 15 | * derived from this software without specific prior written permission. | ||
| 16 | * | ||
| 17 | * | ||
| 18 | * ALTERNATIVELY, this software may be distributed under the terms of the | ||
| 19 | * GNU General Public License ("GPL") as published by the Free Software | ||
| 20 | * Foundation, either version 2 of that License or (at your option) any | ||
| 21 | * later version. | ||
| 22 | * | ||
| 23 | * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY | ||
| 24 | * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | ||
| 25 | * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
| 26 | * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | ||
| 27 | * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | ||
| 28 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | ||
| 29 | * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | ||
| 30 | * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| 31 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | ||
| 32 | * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
| 33 | */ | ||
| 34 | |||
| 35 | lac: lac@229000 { | ||
| 36 | compatible = "fsl,interlaken-lac"; | ||
| 37 | reg = <0x229000 0x1000>; | ||
| 38 | interrupts = <16 2 1 18>; | ||
| 39 | }; | ||
| 40 | |||
| 41 | lac-hv@228000 { | ||
| 42 | compatible = "fsl,interlaken-lac-hv"; | ||
| 43 | reg = <0x228000 0x1000>; | ||
| 44 | fsl,non-hv-node = <&lac>; | ||
| 45 | }; | ||
| @@ -1,43 +0,0 @@ | |||
| 1 | /* | ||
| 2 | * PQ3 MPIC Message (Group B) device tree stub [ controller @ offset 0x42400 ] | ||
| 3 | * | ||
| 4 | * Copyright 2012 Freescale Semiconductor Inc. | ||
| 5 | * | ||
| 6 | * Redistribution and use in source and binary forms, with or without | ||
| 7 | * modification, are permitted provided that the following conditions are met: | ||
| 8 | * * Redistributions of source code must retain the above copyright | ||
| 9 | * notice, this list of conditions and the following disclaimer. | ||
| 10 | * * Redistributions in binary form must reproduce the above copyright | ||
| 11 | * notice, this list of conditions and the following disclaimer in the | ||
| 12 | * documentation and/or other materials provided with the distribution. | ||
| 13 | * * Neither the name of Freescale Semiconductor nor the | ||
| 14 | * names of its contributors may be used to endorse or promote products | ||
| 15 | * derived from this software without specific prior written permission. | ||
| 16 | * | ||
| 17 | * | ||
| 18 | * ALTERNATIVELY, this software may be distributed under the terms of the | ||
| 19 | * GNU General Public License ("GPL") as published by the Free Software | ||
| 20 | * Foundation, either version 2 of that License or (at your option) any | ||
| 21 | * later version. | ||
| 22 | * | ||
| 23 | * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY | ||
| 24 | * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | ||
| 25 | * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
| 26 | * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | ||
| 27 | * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | ||
| 28 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | ||
| 29 | * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | ||
| 30 | * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| 31 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | ||
| 32 | * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
| 33 | */ | ||
| 34 | |||
| 35 | message@42400 { | ||
| 36 | compatible = "fsl,mpic-v3.1-msgr"; | ||
| 37 | reg = <0x42400 0x200>; | ||
| 38 | interrupts = < | ||
| 39 | 0xb4 2 0 0 | ||
| 40 | 0xb5 2 0 0 | ||
| 41 | 0xb6 2 0 0 | ||
| 42 | 0xb7 2 0 0>; | ||
| 43 | }; | ||
| @@ -1,80 +0,0 @@ | |||
| 1 | /* | ||
| 2 | * QorIQ FMan v3 1g port #1 device tree stub [ controller @ offset 0x400000 ] | ||
| 3 | * | ||
| 4 | * Copyright 2012 - 2015 Freescale Semiconductor Inc. | ||
| 5 | * | ||
| 6 | * Redistribution and use in source and binary forms, with or without | ||
| 7 | * modification, are permitted provided that the following conditions are met: | ||
| 8 | * * Redistributions of source code must retain the above copyright | ||
| 9 | * notice, this list of conditions and the following disclaimer. | ||
| 10 | * * Redistributions in binary form must reproduce the above copyright | ||
| 11 | * notice, this list of conditions and the following disclaimer in the | ||
| 12 | * documentation and/or other materials provided with the distribution. | ||
| 13 | * * Neither the name of Freescale Semiconductor nor the | ||
| 14 | * names of its contributors may be used to endorse or promote products | ||
| 15 | * derived from this software without specific prior written permission. | ||
| 16 | * | ||
| 17 | * | ||
| 18 | * ALTERNATIVELY, this software may be distributed under the terms of the | ||
| 19 | * GNU General Public License ("GPL") as published by the Free Software | ||
| 20 | * Foundation, either version 2 of that License or (at your option) any | ||
| 21 | * later version. | ||
| 22 | * | ||
| 23 | * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY | ||
| 24 | * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | ||
| 25 | * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
| 26 | * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY | ||
| 27 | * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | ||
| 28 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | ||
| 29 | * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | ||
| 30 | * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| 31 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | ||
| 32 | * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
| 33 | */ | ||
| 34 | |||
| 35 | fman@400000 { | ||
| 36 | fman0_rx_0x09: port@89000 { | ||
| 37 | cell-index = <0x9>; | ||
| 38 | compatible = "fsl,fman-v3-port-rx"; | ||
| 39 | reg = <0x89000 0x1000>; | ||
| 40 | fsl,fman-10g-port; | ||
| 41 | fsl,fman-best-effort-port; | ||
| 42 | }; | ||
| 43 | |||
| 44 | fman0_tx_0x29: port@a9000 { | ||
| 45 | cell-index = <0x29>; | ||
| 46 | compatible = "fsl,fman-v3-port-tx"; | ||
| 47 | reg = <0xa9000 0x1000>; | ||
| 48 | fsl,fman-10g-port; | ||
| 49 | fsl,fman-best-effort-port; | ||
| 50 | }; | ||
| 51 | |||
| 52 | ethernet@e2000 { | ||
| 53 | cell-index = <1>; | ||
| 54 | compatible = "fsl,fman-memac"; | ||
| 55 | reg = <0xe2000 0x1000>; | ||
| 56 | fsl,fman-ports = <&fman0_rx_0x09 &fman0_tx_0x29>; | ||
| 57 | ptp-timer = <&ptp_timer0>; | ||
| 58 | pcsphy-handle = <&pcsphy1>, <&qsgmiia_pcs1>; | ||
| 59 | pcs-handle-names = "sgmii", "qsgmii"; | ||
| 60 | }; | ||
| 61 | |||
| 62 | mdio@e1000 { | ||
| 63 | qsgmiia_pcs1: ethernet-pcs@1 { | ||
| 64 | compatible = "fsl,lynx-pcs"; | ||
| 65 | reg = <1>; | ||
| 66 | }; | ||
| 67 | }; | ||
| 68 | |||
| 69 | mdio@e3000 { | ||
| 70 | #address-cells = <1>; | ||
| 71 | #size-cells = <0>; | ||
| 72 | compatible = "fsl,fman-memac-mdio", "fsl,fman-xmdio"; | ||
| 73 | reg = <0xe3000 0x1000>; | ||
| 74 | fsl,erratum-a011043; /* must ignore read errors */ | ||
| 75 | |||
| 76 | pcsphy1: ethernet-phy@0 { | ||
| 77 | reg = <0x0>; | ||
| 78 | }; | ||
| 79 | }; | ||
| 80 | }; | ||
| @@ -37,7 +37,7 @@ · 36 unmodified lines | |||
| 37 | }; | 37 | }; |
| 38 | }; | 38 | }; |
| 39 | 39 | ||
| 40 | memory { | 40 | memory@0 { |
| 41 | device_type = "memory"; | 41 | device_type = "memory"; |
| 42 | reg = <0x00000000 0x08000000>; // 128MB at 0 | 42 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 43 | }; | 43 | }; |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | |||
| 38 | }; | 38 | }; |
| 39 | }; | 39 | }; |
| 40 | 40 | ||
| 41 | memory { | 41 | memory@0 { |
| 42 | device_type = "memory"; | 42 | device_type = "memory"; |
| 43 | reg = <0x00000000 0x08000000>; // 128MB at 0 | 43 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 44 | }; | 44 | }; |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | |||
| 6 | */ | 6 | */ |
| 7 | 7 | ||
| 8 | /dts-v1/; | 8 | /dts-v1/; |
| 9 | #include <dt-bindings/interrupt-controller/irq.h> | ||
| 9 | 10 | ||
| 10 | / { | 11 | / { |
| 11 | model = "MPC8313ERDB"; | 12 | model = "MPC8313ERDB"; |
| @@ -38,7 +39,7 @@ · 26 unmodified lines | |||
| 38 | }; | 39 | }; |
| 39 | }; | 40 | }; |
| 40 | 41 | ||
| 41 | memory { | 42 | memory@0 { |
| 42 | device_type = "memory"; | 43 | device_type = "memory"; |
| 43 | reg = <0x00000000 0x08000000>; // 128MB at 0 | 44 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 44 | }; | 45 | }; |
| @@ -48,7 +49,7 @@ · 3 unmodified lines | |||
| 48 | #size-cells = <1>; | 49 | #size-cells = <1>; |
| 49 | compatible = "fsl,mpc8313-elbc", "fsl,elbc", "simple-bus"; | 50 | compatible = "fsl,mpc8313-elbc", "fsl,elbc", "simple-bus"; |
| 50 | reg = <0xe0005000 0x1000>; | 51 | reg = <0xe0005000 0x1000>; |
| 51 | interrupts = <77 0x8>; | 52 | interrupts = <77 IRQ_TYPE_LEVEL_LOW>; |
| 52 | interrupt-parent = <&ipic>; | 53 | interrupt-parent = <&ipic>; |
| 53 | 54 | ||
| 54 | // CS0 and CS1 are swapped when | 55 | // CS0 and CS1 are swapped when |
| @@ -118,7 +119,7 @@ · 63 unmodified lines | |||
| 118 | cell-index = <0>; | 119 | cell-index = <0>; |
| 119 | compatible = "fsl-i2c"; | 120 | compatible = "fsl-i2c"; |
| 120 | reg = <0x3000 0x100>; | 121 | reg = <0x3000 0x100>; |
| 121 | interrupts = <14 0x8>; | 122 | interrupts = <14 IRQ_TYPE_LEVEL_LOW>; |
| 122 | interrupt-parent = <&ipic>; | 123 | interrupt-parent = <&ipic>; |
| 123 | dfsrr; | 124 | dfsrr; |
| 124 | rtc@68 { | 125 | rtc@68 { |
| @@ -131,7 +132,7 @@ · 6 unmodified lines | |||
| 131 | compatible = "fsl,sec2.2", "fsl,sec2.1", | 132 | compatible = "fsl,sec2.2", "fsl,sec2.1", |
| 132 | "fsl,sec2.0"; | 133 | "fsl,sec2.0"; |
| 133 | reg = <0x30000 0x10000>; | 134 | reg = <0x30000 0x10000>; |
| 134 | interrupts = <11 0x8>; | 135 | interrupts = <11 IRQ_TYPE_LEVEL_LOW>; |
| 135 | interrupt-parent = <&ipic>; | 136 | interrupt-parent = <&ipic>; |
| 136 | fsl,num-channels = <1>; | 137 | fsl,num-channels = <1>; |
| 137 | fsl,channel-fifo-len = <24>; | 138 | fsl,channel-fifo-len = <24>; |
| @@ -146,7 +147,7 @@ · 8 unmodified lines | |||
| 146 | cell-index = <1>; | 147 | cell-index = <1>; |
| 147 | compatible = "fsl-i2c"; | 148 | compatible = "fsl-i2c"; |
| 148 | reg = <0x3100 0x100>; | 149 | reg = <0x3100 0x100>; |
| 149 | interrupts = <15 0x8>; | 150 | interrupts = <15 IRQ_TYPE_LEVEL_LOW>; |
| 150 | interrupt-parent = <&ipic>; | 151 | interrupt-parent = <&ipic>; |
| 151 | dfsrr; | 152 | dfsrr; |
| 152 | }; | 153 | }; |
| @@ -155,7 +156,7 @@ · 2 unmodified lines | |||
| 155 | cell-index = <0>; | 156 | cell-index = <0>; |
| 156 | compatible = "fsl,spi"; | 157 | compatible = "fsl,spi"; |
| 157 | reg = <0x7000 0x1000>; | 158 | reg = <0x7000 0x1000>; |
| 158 | interrupts = <16 0x8>; | 159 | interrupts = <16 IRQ_TYPE_LEVEL_LOW>; |
| 159 | interrupt-parent = <&ipic>; | 160 | interrupt-parent = <&ipic>; |
| 160 | mode = "cpu"; | 161 | mode = "cpu"; |
| 161 | }; | 162 | }; |
| @@ -167,7 +168,7 @@ · 5 unmodified lines | |||
| 167 | #address-cells = <1>; | 168 | #address-cells = <1>; |
| 168 | #size-cells = <0>; | 169 | #size-cells = <0>; |
| 169 | interrupt-parent = <&ipic>; | 170 | interrupt-parent = <&ipic>; |
| 170 | interrupts = <38 0x8>; | 171 | interrupts = <38 IRQ_TYPE_LEVEL_LOW>; |
| 171 | phy_type = "utmi_wide"; | 172 | phy_type = "utmi_wide"; |
| 172 | sleep = <&pmc 0x00300000>; | 173 | sleep = <&pmc 0x00300000>; |
| 173 | }; | 174 | }; |
| @@ -175,7 +176,8 @@ · 1 unmodified line | |||
| 175 | ptp_clock@24E00 { | 176 | ptp_clock@24E00 { |
| 176 | compatible = "fsl,etsec-ptp"; | 177 | compatible = "fsl,etsec-ptp"; |
| 177 | reg = <0x24E00 0xB0>; | 178 | reg = <0x24E00 0xB0>; |
| 178 | interrupts = <12 0x8 13 0x8>; | 179 | interrupts = <12 IRQ_TYPE_LEVEL_LOW>, |
| 180 | <13 IRQ_TYPE_LEVEL_LOW>; | ||
| 179 | interrupt-parent = < &ipic >; | 181 | interrupt-parent = < &ipic >; |
| 180 | fsl,tclk-period = <10>; | 182 | fsl,tclk-period = <10>; |
| 181 | fsl,tmr-prsc = <100>; | 183 | fsl,tmr-prsc = <100>; |
| @@ -197,7 +199,9 @@ · 15 unmodified lines | |||
| 197 | compatible = "gianfar"; | 199 | compatible = "gianfar"; |
| 198 | reg = <0x24000 0x1000>; | 200 | reg = <0x24000 0x1000>; |
| 199 | local-mac-address = [ 00 00 00 00 00 00 ]; | 201 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 200 | interrupts = <37 0x8 36 0x8 35 0x8>; | 202 | interrupts = <37 IRQ_TYPE_LEVEL_LOW>, |
| 203 | <36 IRQ_TYPE_LEVEL_LOW>, | ||
| 204 | <35 IRQ_TYPE_LEVEL_LOW>; | ||
| 201 | interrupt-parent = <&ipic>; | 205 | interrupt-parent = <&ipic>; |
| 202 | tbi-handle = < &tbi0 >; | 206 | tbi-handle = < &tbi0 >; |
| 203 | /* Vitesse 7385 isn't on the MDIO bus */ | 207 | /* Vitesse 7385 isn't on the MDIO bus */ |
| @@ -211,7 +215,7 @@ · 7 unmodified lines | |||
| 211 | reg = <0x520 0x20>; | 215 | reg = <0x520 0x20>; |
| 212 | phy4: ethernet-phy@4 { | 216 | phy4: ethernet-phy@4 { |
| 213 | interrupt-parent = <&ipic>; | 217 | interrupt-parent = <&ipic>; |
| 214 | interrupts = <20 0x8>; | 218 | interrupts = <20 IRQ_TYPE_LEVEL_LOW>; |
| 215 | reg = <0x4>; | 219 | reg = <0x4>; |
| 216 | }; | 220 | }; |
| 217 | tbi0: tbi-phy@11 { | 221 | tbi0: tbi-phy@11 { |
| @@ -231,7 +235,9 @@ · 13 unmodified lines | |||
| 231 | reg = <0x25000 0x1000>; | 235 | reg = <0x25000 0x1000>; |
| 232 | ranges = <0x0 0x25000 0x1000>; | 236 | ranges = <0x0 0x25000 0x1000>; |
| 233 | local-mac-address = [ 00 00 00 00 00 00 ]; | 237 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 234 | interrupts = <34 0x8 33 0x8 32 0x8>; | 238 | interrupts = <34 IRQ_TYPE_LEVEL_LOW>, |
| 239 | <33 IRQ_TYPE_LEVEL_LOW>, | ||
| 240 | <32 IRQ_TYPE_LEVEL_LOW>; | ||
| 235 | interrupt-parent = <&ipic>; | 241 | interrupt-parent = <&ipic>; |
| 236 | tbi-handle = < &tbi1 >; | 242 | tbi-handle = < &tbi1 >; |
| 237 | phy-handle = < &phy4 >; | 243 | phy-handle = < &phy4 >; |
| @@ -259,7 +265,7 @@ · 21 unmodified lines | |||
| 259 | compatible = "fsl,ns16550", "ns16550"; | 265 | compatible = "fsl,ns16550", "ns16550"; |
| 260 | reg = <0x4500 0x100>; | 266 | reg = <0x4500 0x100>; |
| 261 | clock-frequency = <0>; | 267 | clock-frequency = <0>; |
| 262 | interrupts = <9 0x8>; | 268 | interrupts = <9 IRQ_TYPE_LEVEL_LOW>; |
| 263 | interrupt-parent = <&ipic>; | 269 | interrupt-parent = <&ipic>; |
| 264 | }; | 270 | }; |
| 265 | 271 | ||
| @@ -269,15 +275,12 @@ · 3 unmodified lines | |||
| 269 | compatible = "fsl,ns16550", "ns16550"; | 275 | compatible = "fsl,ns16550", "ns16550"; |
| 270 | reg = <0x4600 0x100>; | 276 | reg = <0x4600 0x100>; |
| 271 | clock-frequency = <0>; | 277 | clock-frequency = <0>; |
| 272 | interrupts = <10 0x8>; | 278 | interrupts = <10 IRQ_TYPE_LEVEL_LOW>; |
| 273 | interrupt-parent = <&ipic>; | 279 | interrupt-parent = <&ipic>; |
| 274 | }; | 280 | }; |
| 275 | 281 | ||
| 276 | /* IPIC | 282 | /* IPIC |
| 277 | * interrupts cell = <intr #, sense> | 283 | * interrupts cell = <intr #, type> |
| 278 | * sense values match linux IORESOURCE_IRQ_* defines: | ||
| 279 | * sense == 8: Level, low assertion | ||
| 280 | * sense == 2: Edge, high-to-low change | ||
| 281 | */ | 284 | */ |
| 282 | ipic: pic@700 { | 285 | ipic: pic@700 { |
| 283 | interrupt-controller; | 286 | interrupt-controller; |
| @@ -290,7 +293,7 @@ · 6 unmodified lines | |||
| 290 | pmc: power@b00 { | 293 | pmc: power@b00 { |
| 291 | compatible = "fsl,mpc8313-pmc", "fsl,mpc8349-pmc"; | 294 | compatible = "fsl,mpc8313-pmc", "fsl,mpc8349-pmc"; |
| 292 | reg = <0xb00 0x100 0xa00 0x100>; | 295 | reg = <0xb00 0x100 0xa00 0x100>; |
| 293 | interrupts = <80 8>; | 296 | interrupts = <80 IRQ_TYPE_LEVEL_LOW>; |
| 294 | interrupt-parent = <&ipic>; | 297 | interrupt-parent = <&ipic>; |
| 295 | fsl,mpc8313-wakeup-timer = <>m1>; | 298 | fsl,mpc8313-wakeup-timer = <>m1>; |
| 296 | 299 | ||
| @@ -306,14 +309,20 @@ · 9 unmodified lines | |||
| 306 | gtm1: timer@500 { | 309 | gtm1: timer@500 { |
| 307 | compatible = "fsl,mpc8313-gtm", "fsl,gtm"; | 310 | compatible = "fsl,mpc8313-gtm", "fsl,gtm"; |
| 308 | reg = <0x500 0x100>; | 311 | reg = <0x500 0x100>; |
| 309 | interrupts = <90 8 78 8 84 8 72 8>; | 312 | interrupts = <90 IRQ_TYPE_LEVEL_LOW>, |
| 313 | <78 IRQ_TYPE_LEVEL_LOW>, | ||
| 314 | <84 IRQ_TYPE_LEVEL_LOW>, | ||
| 315 | <72 IRQ_TYPE_LEVEL_LOW>; | ||
| 310 | interrupt-parent = <&ipic>; | 316 | interrupt-parent = <&ipic>; |
| 311 | }; | 317 | }; |
| 312 | 318 | ||
| 313 | timer@600 { | 319 | timer@600 { |
| 314 | compatible = "fsl,mpc8313-gtm", "fsl,gtm"; | 320 | compatible = "fsl,mpc8313-gtm", "fsl,gtm"; |
| 315 | reg = <0x600 0x100>; | 321 | reg = <0x600 0x100>; |
| 316 | interrupts = <91 8 79 8 85 8 73 8>; | 322 | interrupts = <91 IRQ_TYPE_LEVEL_LOW>, |
| 323 | <79 IRQ_TYPE_LEVEL_LOW>, | ||
| 324 | <85 IRQ_TYPE_LEVEL_LOW>, | ||
| 325 | <73 IRQ_TYPE_LEVEL_LOW>; | ||
| 317 | interrupt-parent = <&ipic>; | 326 | interrupt-parent = <&ipic>; |
| 318 | }; | 327 | }; |
| 319 | }; | 328 | }; |
| @@ -341,7 +350,7 @@ · 21 unmodified lines | |||
| 341 | 0x7800 0x0 0x0 0x3 &ipic 17 0x8 | 350 | 0x7800 0x0 0x0 0x3 &ipic 17 0x8 |
| 342 | 0x7800 0x0 0x0 0x4 &ipic 18 0x8>; | 351 | 0x7800 0x0 0x0 0x4 &ipic 18 0x8>; |
| 343 | interrupt-parent = <&ipic>; | 352 | interrupt-parent = <&ipic>; |
| 344 | interrupts = <66 0x8>; | 353 | interrupts = <66 IRQ_TYPE_LEVEL_LOW>; |
| 345 | bus-range = <0x0 0x0>; | 354 | bus-range = <0x0 0x0>; |
| 346 | ranges = <0x02000000 0x0 0x90000000 0x90000000 0x0 0x10000000 | 355 | ranges = <0x02000000 0x0 0x90000000 0x90000000 0x0 0x10000000 |
| 347 | 0x42000000 0x0 0x80000000 0x80000000 0x0 0x10000000 | 356 | 0x42000000 0x0 0x80000000 0x80000000 0x0 0x10000000 |
| @@ -363,14 +372,14 @@ · 15 unmodified lines | |||
| 363 | reg = <0xe00082a8 4>; | 372 | reg = <0xe00082a8 4>; |
| 364 | ranges = <0 0xe0008100 0x1a8>; | 373 | ranges = <0 0xe0008100 0x1a8>; |
| 365 | interrupt-parent = <&ipic>; | 374 | interrupt-parent = <&ipic>; |
| 366 | interrupts = <71 8>; | 375 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 367 | 376 | ||
| 368 | dma-channel@0 { | 377 | dma-channel@0 { |
| 369 | compatible = "fsl,mpc8313-dma-channel", | 378 | compatible = "fsl,mpc8313-dma-channel", |
| 370 | "fsl,elo-dma-channel"; | 379 | "fsl,elo-dma-channel"; |
| 371 | reg = <0 0x28>; | 380 | reg = <0 0x28>; |
| 372 | interrupt-parent = <&ipic>; | 381 | interrupt-parent = <&ipic>; |
| 373 | interrupts = <71 8>; | 382 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 374 | cell-index = <0>; | 383 | cell-index = <0>; |
| 375 | }; | 384 | }; |
| 376 | 385 | ||
| @@ -379,7 +388,7 @@ · 2 unmodified lines | |||
| 379 | "fsl,elo-dma-channel"; | 388 | "fsl,elo-dma-channel"; |
| 380 | reg = <0x80 0x28>; | 389 | reg = <0x80 0x28>; |
| 381 | interrupt-parent = <&ipic>; | 390 | interrupt-parent = <&ipic>; |
| 382 | interrupts = <71 8>; | 391 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 383 | cell-index = <1>; | 392 | cell-index = <1>; |
| 384 | }; | 393 | }; |
| 385 | 394 | ||
| @@ -388,7 +397,7 @@ · 2 unmodified lines | |||
| 388 | "fsl,elo-dma-channel"; | 397 | "fsl,elo-dma-channel"; |
| 389 | reg = <0x100 0x28>; | 398 | reg = <0x100 0x28>; |
| 390 | interrupt-parent = <&ipic>; | 399 | interrupt-parent = <&ipic>; |
| 391 | interrupts = <71 8>; | 400 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 392 | cell-index = <2>; | 401 | cell-index = <2>; |
| 393 | }; | 402 | }; |
| 394 | 403 | ||
| @@ -397,7 +406,7 @@ · 2 unmodified lines | |||
| 397 | "fsl,elo-dma-channel"; | 406 | "fsl,elo-dma-channel"; |
| 398 | reg = <0x180 0x28>; | 407 | reg = <0x180 0x28>; |
| 399 | interrupt-parent = <&ipic>; | 408 | interrupt-parent = <&ipic>; |
| 400 | interrupts = <71 8>; | 409 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 401 | cell-index = <3>; | 410 | cell-index = <3>; |
| 402 | }; | 411 | }; |
| 403 | }; | 412 | }; |
| @@ -40,7 +40,7 @@ · 39 unmodified lines | |||
| 40 | }; | 40 | }; |
| 41 | }; | 41 | }; |
| 42 | 42 | ||
| 43 | memory { | 43 | memory@0 { |
| 44 | device_type = "memory"; | 44 | device_type = "memory"; |
| 45 | reg = <0x00000000 0x08000000>; // 128MB at 0 | 45 | reg = <0x00000000 0x08000000>; // 128MB at 0 |
| 46 | }; | 46 | }; |
| @@ -50,7 +50,7 @@ · 3 unmodified lines | |||
| 50 | #size-cells = <1>; | 50 | #size-cells = <1>; |
| 51 | compatible = "fsl,mpc8315-elbc", "fsl,elbc", "simple-bus"; | 51 | compatible = "fsl,mpc8315-elbc", "fsl,elbc", "simple-bus"; |
| 52 | reg = <0xe0005000 0x1000>; | 52 | reg = <0xe0005000 0x1000>; |
| 53 | interrupts = <77 0x8>; | 53 | interrupts = <77 IRQ_TYPE_LEVEL_LOW>; |
| 54 | interrupt-parent = <&ipic>; | 54 | interrupt-parent = <&ipic>; |
| 55 | 55 | ||
| 56 | // CS0 and CS1 are swapped when | 56 | // CS0 and CS1 are swapped when |
| @@ -112,7 +112,7 @@ · 55 unmodified lines | |||
| 112 | cell-index = <0>; | 112 | cell-index = <0>; |
| 113 | compatible = "fsl-i2c"; | 113 | compatible = "fsl-i2c"; |
| 114 | reg = <0x3000 0x100>; | 114 | reg = <0x3000 0x100>; |
| 115 | interrupts = <14 0x8>; | 115 | interrupts = <14 IRQ_TYPE_LEVEL_LOW>; |
| 116 | interrupt-parent = <&ipic>; | 116 | interrupt-parent = <&ipic>; |
| 117 | dfsrr; | 117 | dfsrr; |
| 118 | rtc@68 { | 118 | rtc@68 { |
| @@ -133,8 +133,10 @@ · 14 unmodified lines | |||
| 133 | cell-index = <0>; | 133 | cell-index = <0>; |
| 134 | compatible = "fsl,spi"; | 134 | compatible = "fsl,spi"; |
| 135 | reg = <0x7000 0x1000>; | 135 | reg = <0x7000 0x1000>; |
| 136 | interrupts = <16 0x8>; | 136 | interrupts = <16 IRQ_TYPE_LEVEL_LOW>; |
| 137 | interrupt-parent = <&ipic>; | 137 | interrupt-parent = <&ipic>; |
| 138 | #address-cells = <1>; | ||
| 139 | #size-cells = <0>; | ||
| 138 | mode = "cpu"; | 140 | mode = "cpu"; |
| 139 | }; | 141 | }; |
| 140 | 142 | ||
| @@ -145,35 +147,35 @@ · 4 unmodified lines | |||
| 145 | reg = <0x82a8 4>; | 147 | reg = <0x82a8 4>; |
| 146 | ranges = <0 0x8100 0x1a8>; | 148 | ranges = <0 0x8100 0x1a8>; |
| 147 | interrupt-parent = <&ipic>; | 149 | interrupt-parent = <&ipic>; |
| 148 | interrupts = <71 8>; | 150 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 149 | cell-index = <0>; | 151 | cell-index = <0>; |
| 150 | dma-channel@0 { | 152 | dma-channel@0 { |
| 151 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; | 153 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 152 | reg = <0 0x80>; | 154 | reg = <0 0x80>; |
| 153 | cell-index = <0>; | 155 | cell-index = <0>; |
| 154 | interrupt-parent = <&ipic>; | 156 | interrupt-parent = <&ipic>; |
| 155 | interrupts = <71 8>; | 157 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 156 | }; | 158 | }; |
| 157 | dma-channel@80 { | 159 | dma-channel@80 { |
| 158 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; | 160 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 159 | reg = <0x80 0x80>; | 161 | reg = <0x80 0x80>; |
| 160 | cell-index = <1>; | 162 | cell-index = <1>; |
| 161 | interrupt-parent = <&ipic>; | 163 | interrupt-parent = <&ipic>; |
| 162 | interrupts = <71 8>; | 164 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 163 | }; | 165 | }; |
| 164 | dma-channel@100 { | 166 | dma-channel@100 { |
| 165 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; | 167 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 166 | reg = <0x100 0x80>; | 168 | reg = <0x100 0x80>; |
| 167 | cell-index = <2>; | 169 | cell-index = <2>; |
| 168 | interrupt-parent = <&ipic>; | 170 | interrupt-parent = <&ipic>; |
| 169 | interrupts = <71 8>; | 171 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 170 | }; | 172 | }; |
| 171 | dma-channel@180 { | 173 | dma-channel@180 { |
| 172 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; | 174 | compatible = "fsl,mpc8315-dma-channel", "fsl,elo-dma-channel"; |
| 173 | reg = <0x180 0x28>; | 175 | reg = <0x180 0x28>; |
| 174 | cell-index = <3>; | 176 | cell-index = <3>; |
| 175 | interrupt-parent = <&ipic>; | 177 | interrupt-parent = <&ipic>; |
| 176 | interrupts = <71 8>; | 178 | interrupts = <71 IRQ_TYPE_LEVEL_LOW>; |
| 177 | }; | 179 | }; |
| 178 | }; | 180 | }; |
| 179 | 181 | ||
| @@ -183,7 +185,7 @@ · 3 unmodified lines | |||
| 183 | #address-cells = <1>; | 185 | #address-cells = <1>; |
| 184 | #size-cells = <0>; | 186 | #size-cells = <0>; |
| 185 | interrupt-parent = <&ipic>; | 187 | interrupt-parent = <&ipic>; |
| 186 | interrupts = <38 0x8>; | 188 | interrupts = <38 IRQ_TYPE_LEVEL_LOW>; |
| 187 | phy_type = "utmi"; | 189 | phy_type = "utmi"; |
| 188 | }; | 190 | }; |
| 189 | 191 | ||
| @@ -197,7 +199,9 @@ · 7 unmodified lines | |||
| 197 | reg = <0x24000 0x1000>; | 199 | reg = <0x24000 0x1000>; |
| 198 | ranges = <0x0 0x24000 0x1000>; | 200 | ranges = <0x0 0x24000 0x1000>; |
| 199 | local-mac-address = [ 00 00 00 00 00 00 ]; | 201 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 200 | interrupts = <32 0x8 33 0x8 34 0x8>; | 202 | interrupts = <32 IRQ_TYPE_LEVEL_LOW>, |
| 203 | <33 IRQ_TYPE_LEVEL_LOW>, | ||
| 204 | <34 IRQ_TYPE_LEVEL_LOW>; | ||
| 201 | interrupt-parent = <&ipic>; | 205 | interrupt-parent = <&ipic>; |
| 202 | tbi-handle = <&tbi0>; | 206 | tbi-handle = <&tbi0>; |
| 203 | phy-handle = < &phy0 >; | 207 | phy-handle = < &phy0 >; |
| @@ -238,7 +242,9 @@ · 34 unmodified lines | |||
| 238 | reg = <0x25000 0x1000>; | 242 | reg = <0x25000 0x1000>; |
| 239 | ranges = <0x0 0x25000 0x1000>; | 243 | ranges = <0x0 0x25000 0x1000>; |
| 240 | local-mac-address = [ 00 00 00 00 00 00 ]; | 244 | local-mac-address = [ 00 00 00 00 00 00 ]; |
| 241 | interrupts = <35 0x8 36 0x8 37 0x8>; | 245 | interrupts = <35 IRQ_TYPE_LEVEL_LOW>, |
| 246 | <36 IRQ_TYPE_LEVEL_LOW>, | ||
| 247 | <37 IRQ_TYPE_LEVEL_LOW>; | ||
| 242 | interrupt-parent = <&ipic>; | 248 | interrupt-parent = <&ipic>; |
| 243 | tbi-handle = <&tbi1>; | 249 | tbi-handle = <&tbi1>; |
| 244 | phy-handle = < &phy1 >; | 250 | phy-handle = < &phy1 >; |
| @@ -263,7 +269,7 @@ · 18 unmodified lines | |||
| 263 | compatible = "fsl,ns16550", "ns16550"; | 269 | compatible = "fsl,ns16550", "ns16550"; |
| 264 | reg = <0x4500 0x100>; | 270 | reg = <0x4500 0x100>; |
| 265 | clock-frequency = <133333333>; | 271 | clock-frequency = <133333333>; |
| 266 | interrupts = <9 0x8>; | 272 | interrupts = <9 IRQ_TYPE_LEVEL_LOW>; |
| 267 | interrupt-parent = <&ipic>; | 273 | interrupt-parent = <&ipic>; |
| 268 | }; | 274 | }; |
| 269 | 275 | ||
| @@ -273,7 +279,7 @@ · 3 unmodified lines | |||
| 273 | compatible = "fsl,ns16550", "ns16550"; | 279 | compatible = "fsl,ns16550", "ns16550"; |
| 274 | reg = <0x4600 0x100>; | 280 | reg = <0x4600 0x100>; |
| 275 | clock-frequency = <133333333>; | 281 | clock-frequency = <133333333>; |
| 276 | interrupts = <10 0x8>; | 282 | interrupts = <10 IRQ_TYPE_LEVEL_LOW>; |
| 277 | interrupt-parent = <&ipic>; | 283 | interrupt-parent = <&ipic>; |
| 278 | }; | 284 | }; |
| 279 | 285 | ||
| @@ -282,7 +288,7 @@ · 2 unmodified lines | |||
| 282 | "fsl,sec2.4", "fsl,sec2.2", "fsl,sec2.1", | 288 | "fsl,sec2.4", "fsl,sec2.2", "fsl,sec2.1", |
| 283 | "fsl,sec2.0"; | 289 | "fsl,sec2.0"; |
| 284 | reg = <0x30000 0x10000>; | 290 | reg = <0x30000 0x10000>; |
| 285 | interrupts = <11 0x8>; | 291 | interrupts = <11 IRQ_TYPE_LEVEL_LOW>; |
| 286 | interrupt-parent = <&ipic>; | 292 | interrupt-parent = <&ipic>; |
| 287 | fsl,num-channels = <4>; | 293 | fsl,num-channels = <4>; |
| 288 | fsl,channel-fifo-len = <24>; | 294 | fsl,channel-fifo-len = <24>; |
| @@ -294,7 +300,7 @@ · 5 unmodified lines | |||
| 294 | compatible = "fsl,mpc8315-sata", "fsl,pq-sata"; | 300 | compatible = "fsl,mpc8315-sata", "fsl,pq-sata"; |
| 295 | reg = <0x18000 0x1000>; | 301 | reg = <0x18000 0x1000>; |
| 296 | cell-index = <1>; | 302 | cell-index = <1>; |
| 297 | interrupts = <44 0x8>; | 303 | interrupts = <44 IRQ_TYPE_LEVEL_LOW>; |
| 298 | interrupt-parent = <&ipic>; | 304 | interrupt-parent = <&ipic>; |
| 299 | }; | 305 | }; |
| 300 | 306 | ||
| @@ -302,14 +308,17 @@ · 1 unmodified line | |||
| 302 | compatible = "fsl,mpc8315-sata", "fsl,pq-sata"; | 308 | compatible = "fsl,mpc8315-sata", "fsl,pq-sata"; |
| 303 | reg = <0x19000 0x1000>; | 309 | reg = <0x19000 0x1000>; |
| 304 | cell-index = <2>; | 310 | cell-index = <2>; |
| 305 | interrupts = <45 0x8>; | 311 | interrupts = <45 IRQ_TYPE_LEVEL_LOW>; |
| 306 | interrupt-parent = <&ipic>; | 312 | interrupt-parent = <&ipic>; |
| 307 | }; | 313 | }; |
| 308 | 314 | ||
| 309 | gtm1: timer@500 { | 315 | gtm1: timer@500 { |
| 310 | compatible = "fsl,mpc8315-gtm", "fsl,gtm"; | 316 | compatible = "fsl,mpc8315-gtm", "fsl,gtm"; |
| 311 | reg = <0x500 0x100>; | 317 | reg = <0x500 0x100>; |
| 312 | interrupts = <90 8 78 8 84 8 72 8>; | 318 | interrupts = <90 IRQ_TYPE_LEVEL_LOW>, |
| 319 | <78 IRQ_TYPE_LEVEL_LOW>, | ||
| 320 | <84 IRQ_TYPE_LEVEL_LOW>, | ||
| 321 | <72 IRQ_TYPE_LEVEL_LOW>; | ||
| 313 | interrupt-parent = <&ipic>; | 322 | interrupt-parent = <&ipic>; |
| 314 | clock-frequency = <133333333>; | 323 | clock-frequency = <133333333>; |
| 315 | }; | 324 | }; |
| @@ -317,16 +326,16 @@ · 1 unmodified line | |||
| 317 | timer@600 { | 326 | timer@600 { |
| 318 | compatible = "fsl,mpc8315-gtm", "fsl,gtm"; | 327 | compatible = "fsl,mpc8315-gtm", "fsl,gtm"; |
| 319 | reg = <0x600 0x100>; | 328 | reg = <0x600 0x100>; |
| 320 | interrupts = <91 8 79 8 85 8 73 8>; | 329 | interrupts = <91 IRQ_TYPE_LEVEL_LOW>, |
| 330 | <79 IRQ_TYPE_LEVEL_LOW>, | ||
| 331 | <85 IRQ_TYPE_LEVEL_LOW>, | ||
| 332 | <73 IRQ_TYPE_LEVEL_LOW>; | ||
| 321 | interrupt-parent = <&ipic>; | 333 | interrupt-parent = <&ipic>; |
| 322 | clock-frequency = <133333333>; | 334 | clock-frequency = <133333333>; |
| 323 | }; | 335 | }; |
| 324 | 336 | ||
| 325 | /* IPIC | 337 | /* IPIC |
| 326 | * interrupts cell = <intr #, sense> | 338 | * interrupts cell = <intr #, type> |
| 327 | * sense values match linux IORESOURCE_IRQ_* defines: | ||
| 328 | * sense == 8: Level, low assertion | ||
| 329 | * sense == 2: Edge, high-to-low change | ||
| 330 | */ | 339 | */ |
| 331 | ipic: interrupt-controller@700 { | 340 | ipic: interrupt-controller@700 { |
| 332 | interrupt-controller; | 341 | interrupt-controller; |
| @@ -340,14 +349,14 @@ · 7 unmodified lines | |||
| 340 | compatible = "fsl,ipic-msi"; | 349 | compatible = "fsl,ipic-msi"; |
| 341 | reg = <0x7c0 0x40>; | 350 | reg = <0x7c0 0x40>; |
| 342 | msi-available-ranges = <0 0x100>; | 351 | msi-available-ranges = <0 0x100>; |
| 343 | interrupts = <0x43 0x8 | 352 | interrupts = <0x43 IRQ_TYPE_LEVEL_LOW |
| 344 | 0x4 0x8 | 353 | 0x4 IRQ_TYPE_LEVEL_LOW |
| 345 | 0x51 0x8 | 354 | 0x51 IRQ_TYPE_LEVEL_LOW |
| 346 | 0x52 0x8 | 355 | 0x52 IRQ_TYPE_LEVEL_LOW |
| 347 | 0x56 0x8 | 356 | 0x56 IRQ_TYPE_LEVEL_LOW |
| 348 | 0x57 0x8 | 357 | 0x57 IRQ_TYPE_LEVEL_LOW |
| 349 | 0x58 0x8 | 358 | 0x58 IRQ_TYPE_LEVEL_LOW |
| 350 | 0x59 0x8>; | 359 | 0x59 IRQ_TYPE_LEVEL_LOW>; |
| 351 | interrupt-parent = < &ipic >; | 360 | interrupt-parent = < &ipic >; |
| 352 | }; | 361 | }; |
| 353 | 362 | ||
| @@ -355,7 +364,7 @@ · 1 unmodified line | |||
| 355 | compatible = "fsl,mpc8315-pmc", "fsl,mpc8313-pmc", | 364 | compatible = "fsl,mpc8315-pmc", "fsl,mpc8313-pmc", |
| 356 | "fsl,mpc8349-pmc"; | 365 | "fsl,mpc8349-pmc"; |
| 357 | reg = <0xb00 0x100 0xa00 0x100>; | 366 | reg = <0xb00 0x100 0xa00 0x100>; |
| 358 | interrupts = <80 8>; | 367 | interrupts = <80 IRQ_TYPE_LEVEL_LOW>; |
| 359 | interrupt-parent = <&ipic>; | 368 | interrupt-parent = <&ipic>; |
| 360 | fsl,mpc8313-wakeup-timer = <>m1>; | 369 | fsl,mpc8313-wakeup-timer = <>m1>; |
| 361 | }; | 370 | }; |
| @@ -374,24 +383,24 @@ · 12 unmodified lines | |||
| 374 | interrupt-map-mask = <0xf800 0x0 0x0 0x7>; | 383 | interrupt-map-mask = <0xf800 0x0 0x0 0x7>; |
| 375 | interrupt-map = < | 384 | interrupt-map = < |
| 376 | /* IDSEL 0x0E -mini PCI */ | 385 | /* IDSEL 0x0E -mini PCI */ |
| 377 | 0x7000 0x0 0x0 0x1 &ipic 18 0x8 | 386 | 0x7000 0x0 0x0 0x1 &ipic 18 IRQ_TYPE_LEVEL_LOW |
| 378 | 0x7000 0x0 0x0 0x2 &ipic 18 0x8 | 387 | 0x7000 0x0 0x0 0x2 &ipic 18 IRQ_TYPE_LEVEL_LOW |
| 379 | 0x7000 0x0 0x0 0x3 &ipic 18 0x8 | 388 | 0x7000 0x0 0x0 0x3 &ipic 18 IRQ_TYPE_LEVEL_LOW |
| 380 | 0x7000 0x0 0x0 0x4 &ipic 18 0x8 | 389 | 0x7000 0x0 0x0 0x4 &ipic 18 IRQ_TYPE_LEVEL_LOW |
| 381 | 390 | ||
| 382 | /* IDSEL 0x0F -mini PCI */ | 391 | /* IDSEL 0x0F -mini PCI */ |
| 383 | 0x7800 0x0 0x0 0x1 &ipic 17 0x8 | 392 | 0x7800 0x0 0x0 0x1 &ipic 17 IRQ_TYPE_LEVEL_LOW |
| 384 | 0x7800 0x0 0x0 0x2 &ipic 17 0x8 | 393 | 0x7800 0x0 0x0 0x2 &ipic 17 IRQ_TYPE_LEVEL_LOW |
| 385 | 0x7800 0x0 0x0 0x3 &ipic 17 0x8 | 394 | 0x7800 0x0 0x0 0x3 &ipic 17 IRQ_TYPE_LEVEL_LOW |
| 386 | 0x7800 0x0 0x0 0x4 &ipic 17 0x8 | 395 | 0x7800 0x0 0x0 0x4 &ipic 17 IRQ_TYPE_LEVEL_LOW |
| 387 | 396 | ||
| 388 | /* IDSEL 0x10 - PCI slot */ | 397 | /* IDSEL 0x10 - PCI slot */ |
| 389 | 0x8000 0x0 0x0 0x1 &ipic 48 0x8 | 398 | 0x8000 0x0 0x0 0x1 &ipic 48 IRQ_TYPE_LEVEL_LOW |
| 390 | 0x8000 0x0 0x0 0x2 &ipic 17 0x8 | 399 | 0x8000 0x0 0x0 0x2 &ipic 17 IRQ_TYPE_LEVEL_LOW |
| 391 | 0x8000 0x0 0x0 0x3 &ipic 48 0x8 | 400 | 0x8000 0x0 0x0 0x3 &ipic 48 IRQ_TYPE_LEVEL_LOW |
| 392 | 0x8000 0x0 0x0 0x4 &ipic 17 0x8>; | 401 | 0x8000 0x0 0x0 0x4 &ipic 17 IRQ_TYPE_LEVEL_LOW>; |
| 393 | interrupt-parent = <&ipic>; | 402 | interrupt-parent = <&ipic>; |
| 394 | interrupts = <66 0x8>; | 403 | interrupts = <66 IRQ_TYPE_LEVEL_LOW>; |
| 395 | bus-range = <0x0 0x0>; | 404 | bus-range = <0x0 0x0>; |
| 396 | ranges = <0x02000000 0 0x90000000 0x90000000 0 0x10000000 | 405 | ranges = <0x02000000 0 0x90000000 0x90000000 0 0x10000000 |
| 397 | 0x42000000 0 0x80000000 0x80000000 0 0x10000000 | 406 | 0x42000000 0 0x80000000 0x80000000 0 0x10000000 |
| @@ -417,10 +426,10 @@ · 19 unmodified lines | |||
| 417 | 0x01000000 0 0x00000000 0xb1000000 0 0x00800000>; | 426 | 0x01000000 0 0x00000000 0xb1000000 0 0x00800000>; |
| 418 | bus-range = <0 255>; | 427 | bus-range = <0 255>; |
| 419 | interrupt-map-mask = <0xf800 0 0 7>; | 428 | interrupt-map-mask = <0xf800 0 0 7>; |
| 420 | interrupt-map = <0 0 0 1 &ipic 1 8 | 429 | interrupt-map = <0 0 0 1 &ipic 1 IRQ_TYPE_LEVEL_LOW |
| 421 | 0 0 0 2 &ipic 1 8 | 430 | 0 0 0 2 &ipic 1 IRQ_TYPE_LEVEL_LOW |
| 422 | 0 0 0 3 &ipic 1 8 | 431 | 0 0 0 3 &ipic 1 IRQ_TYPE_LEVEL_LOW |
| 423 | 0 0 0 4 &ipic 1 8>; | 432 | 0 0 0 4 &ipic 1 IRQ_TYPE_LEVEL_LOW>; |
| 424 | clock-frequency = <0>; | 433 | clock-frequency = <0>; |
| 425 | 434 | ||
| 426 | pcie@0 { | 435 | pcie@0 { |
| @@ -448,10 +457,10 @@ · 21 unmodified lines | |||
| 448 | 0x01000000 0 0x00000000 0xd1000000 0 0x00800000>; | 457 | 0x01000000 0 0x00000000 0xd1000000 0 0x00800000>; |
| 449 | bus-range = <0 255>; | 458 | bus-range = <0 255>; |
| 450 | interrupt-map-mask = <0xf800 0 0 7>; | 459 | interrupt-map-mask = <0xf800 0 0 7>; |
| 451 | interrupt-map = <0 0 0 1 &ipic 2 8 | 460 | interrupt-map = <0 0 0 1 &ipic 2 IRQ_TYPE_LEVEL_LOW |
| 452 | 0 0 0 2 &ipic 2 8 | 461 | 0 0 0 2 &ipic 2 IRQ_TYPE_LEVEL_LOW |
| 453 | 0 0 0 3 &ipic 2 8 | 462 | 0 0 0 3 &ipic 2 IRQ_TYPE_LEVEL_LOW |
| 454 | 0 0 0 4 &ipic 2 8>; | 463 | 0 0 0 4 &ipic 2 IRQ_TYPE_LEVEL_LOW>; |
| 455 | clock-frequency = <0>; | 464 | clock-frequency = <0>; |
| 456 | 465 | ||
| 457 | pcie@0 { | 466 | pcie@0 { |
| @@ -471,12 +480,12 @@ · 13 unmodified lines | |||
| 471 | leds { | 480 | leds { |
| 472 | compatible = "gpio-leds"; | 481 | compatible = "gpio-leds"; |
| 473 | 482 | ||
| 474 | pwr { | 483 | led-pwr { |
| 475 | gpios = <&mcu_pio 0 0>; | 484 | gpios = <&mcu_pio 0 0>; |
| 476 | default-state = "on"; | 485 | default-state = "on"; |
| 477 | }; | 486 | }; |
| 478 | 487 | ||
| 479 | hdd { | 488 | led-hdd { |
| 480 | gpios = <&mcu_pio 1 0>; | 489 | gpios = <&mcu_pio 1 0>; |
| 481 | linux,default-trigger = "disk-activity"; | 490 | linux,default-trigger = "disk-activity"; |
| 482 | }; | 491 | }; |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | |||
| 38 | }; | 38 | }; |
| 39 | }; | 39 | }; |
| 40 | 40 | ||
| 41 | memory { | 41 | memory@0 { |
| 42 | device_type = "memory"; | 42 | device_type = "memory"; |
| 43 | reg = <0x00000000 0x04000000>; | 43 | reg = <0x00000000 0x04000000>; |
| 44 | }; | 44 | }; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | |||
| 39 | }; | 39 | }; |
| 40 | }; | 40 | }; |
| 41 | 41 | ||
| 42 | memory { | 42 | memory@0 { |
| 43 | device_type = "memory"; | 43 | device_type = "memory"; |
| 44 | reg = <0x00000000 0x10000000>; | 44 | reg = <0x00000000 0x10000000>; |
| 45 | }; | 45 | }; |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | |||
| 37 | }; | 37 | }; |
| 38 | }; | 38 | }; |
| 39 | 39 | ||
| 40 | memory { | 40 | memory@0 { |
| 41 | device_type = "memory"; | 41 | device_type = "memory"; |
| 42 | reg = <0x00000000 0x10000000>; | 42 | reg = <0x00000000 0x10000000>; |
| 43 | }; | 43 | }; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | |||
| 39 | }; | 39 | }; |
| 40 | }; | 40 | }; |
| 41 | 41 | ||
| 42 | memory { | 42 | memory@0 { |
| 43 | device_type = "memory"; | 43 | device_type = "memory"; |
| 44 | reg = <0x00000000 0x10000000>; // 256MB at 0 | 44 | reg = <0x00000000 0x10000000>; // 256MB at 0 |
| 45 | }; | 45 | }; |
| @@ -40,7 +40,7 @@ · 39 unmodified lines | |||
| 40 | }; | 40 | }; |
| 41 | }; | 41 | }; |
| 42 | 42 | ||
| 43 | memory { | 43 | memory@0 { |
| 44 | device_type = "memory"; | 44 | device_type = "memory"; |
| 45 | reg = <0x00000000 0x20000000>; // 512MB at 0 | 45 | reg = <0x00000000 0x20000000>; // 512MB at 0 |
| 46 | }; | 46 | }; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | |||
| 39 | }; | 39 | }; |
| 40 | }; | 40 | }; |
| 41 | 41 | ||
| 42 | memory { | 42 | memory@0 { |
| 43 | device_type = "memory"; | 43 | device_type = "memory"; |
| 44 | reg = <0x00000000 0x10000000>; // 256MB at 0 | 44 | reg = <0x00000000 0x10000000>; // 256MB at 0 |
| 45 | }; | 45 | }; |
| @@ -37,7 +37,7 @@ · 36 unmodified lines | |||
| 37 | }; | 37 | }; |
| 38 | }; | 38 | }; |
| 39 | 39 | ||
| 40 | memory { | 40 | memory@0 { |
| 41 | device_type = "memory"; | 41 | device_type = "memory"; |
| 42 | reg = <0x00000000 0x10000000>; // 256MB at 0 | 42 | reg = <0x00000000 0x10000000>; // 256MB at 0 |
| 43 | }; | 43 | }; |
| @@ -120,10 +120,8 @@ · 119 unmodified lines | |||
| 120 | 120 | ||
| 121 | #if defined(CONFIG_44x) | 121 | #if defined(CONFIG_44x) |
| 122 | #include <asm/nohash/32/pte-44x.h> | 122 | #include <asm/nohash/32/pte-44x.h> |
| 123 | #elif defined(CONFIG_PPC_85xx) && defined(CONFIG_PTE_64BIT) | ||
| 124 | #include <asm/nohash/pte-e500.h> | ||
| 125 | #elif defined(CONFIG_PPC_85xx) | 123 | #elif defined(CONFIG_PPC_85xx) |
| 126 | #include <asm/nohash/32/pte-85xx.h> | 124 | #include <asm/nohash/pte-e500.h> |
| 127 | #elif defined(CONFIG_PPC_8xx) | 125 | #elif defined(CONFIG_PPC_8xx) |
| 128 | #include <asm/nohash/32/pte-8xx.h> | 126 | #include <asm/nohash/32/pte-8xx.h> |
| 129 | #endif | 127 | #endif |
| @@ -1,59 +0,0 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | #ifndef _ASM_POWERPC_NOHASH_32_PTE_85xx_H | ||
| 3 | #define _ASM_POWERPC_NOHASH_32_PTE_85xx_H | ||
| 4 | #ifdef __KERNEL__ | ||
| 5 | |||
| 6 | /* PTE bit definitions for Freescale BookE SW loaded TLB MMU based | ||
| 7 | * processors | ||
| 8 | * | ||
| 9 | MMU Assist Register 3: | ||
| 10 | |||
| 11 | 32 33 34 35 36 ... 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ||
| 12 | RPN...................... 0 0 U0 U1 U2 U3 UX SX UW SW UR SR | ||
| 13 | |||
| 14 | - PRESENT *must* be in the bottom two bits because swap PTEs use | ||
| 15 | the top 30 bits. | ||
| 16 | |||
| 17 | */ | ||
| 18 | |||
| 19 | /* Definitions for FSL Book-E Cores */ | ||
| 20 | #define _PAGE_READ 0x00001 /* H: Read permission (SR) */ | ||
| 21 | #define _PAGE_PRESENT 0x00002 /* S: PTE contains a translation */ | ||
| 22 | #define _PAGE_WRITE 0x00004 /* S: Write permission (SW) */ | ||
| 23 | #define _PAGE_DIRTY 0x00008 /* S: Page dirty */ | ||
| 24 | #define _PAGE_EXEC 0x00010 /* H: SX permission */ | ||
| 25 | #define _PAGE_ACCESSED 0x00020 /* S: Page referenced */ | ||
| 26 | |||
| 27 | #define _PAGE_ENDIAN 0x00040 /* H: E bit */ | ||
| 28 | #define _PAGE_GUARDED 0x00080 /* H: G bit */ | ||
| 29 | #define _PAGE_COHERENT 0x00100 /* H: M bit */ | ||
| 30 | #define _PAGE_NO_CACHE 0x00200 /* H: I bit */ | ||
| 31 | #define _PAGE_WRITETHRU 0x00400 /* H: W bit */ | ||
| 32 | #define _PAGE_SPECIAL 0x00800 /* S: Special page */ | ||
| 33 | |||
| 34 | #define _PMD_PRESENT 0 | ||
| 35 | #define _PMD_PRESENT_MASK (PAGE_MASK) | ||
| 36 | #define _PMD_BAD (~PAGE_MASK) | ||
| 37 | #define _PMD_USER 0 | ||
| 38 | |||
| 39 | #define _PTE_NONE_MASK 0 | ||
| 40 | |||
| 41 | #define PTE_WIMGE_SHIFT (6) | ||
| 42 | |||
| 43 | /* | ||
| 44 | * We define 2 sets of base prot bits, one for basic pages (ie, | ||
| 45 | * cacheable kernel and user pages) and one for non cacheable | ||
| 46 | * pages. We always set _PAGE_COHERENT when SMP is enabled or | ||
| 47 | * the processor might need it for DMA coherency. | ||
| 48 | */ | ||
| 49 | #define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED) | ||
| 50 | #if defined(CONFIG_SMP) || defined(CONFIG_PPC_E500MC) | ||
| 51 | #define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT) | ||
| 52 | #else | ||
| 53 | #define _PAGE_BASE (_PAGE_BASE_NC) | ||
| 54 | #endif | ||
| 55 | |||
| 56 | #include <asm/pgtable-masks.h> | ||
| 57 | |||
| 58 | #endif /* __KERNEL__ */ | ||
| 59 | #endif /* _ASM_POWERPC_NOHASH_32_PTE_FSL_85xx_H */ | ||
| @@ -49,7 +49,7 @@ static inline unsigned long pud_val(pud_t x) · 48 unmodified lines | |||
| 49 | #endif /* CONFIG_PPC64 */ | 49 | #endif /* CONFIG_PPC64 */ |
| 50 | 50 | ||
| 51 | /* PGD level */ | 51 | /* PGD level */ |
| 52 | #if defined(CONFIG_PPC_85xx) && defined(CONFIG_PTE_64BIT) | 52 | #if defined(CONFIG_PPC_85xx) |
| 53 | typedef struct { unsigned long long pgd; } pgd_t; | 53 | typedef struct { unsigned long long pgd; } pgd_t; |
| 54 | 54 | ||
| 55 | static inline unsigned long long pgd_val(pgd_t x) | 55 | static inline unsigned long long pgd_val(pgd_t x) |
| @@ -15,6 +15,9 @@ · 14 unmodified lines | |||
| 15 | #define TASK_SIZE_MAX TASK_SIZE_USER64 | 15 | #define TASK_SIZE_MAX TASK_SIZE_USER64 |
| 16 | #endif | 16 | #endif |
| 17 | 17 | ||
| 18 | /* Threshold above which VMX copy path is used */ | ||
| 19 | #define VMX_COPY_THRESHOLD 3328 | ||
| 20 | |||
| 18 | #include <asm-generic/access_ok.h> | 21 | #include <asm-generic/access_ok.h> |
| 19 | 22 | ||
| 20 | /* | 23 | /* |
| @@ -255,7 +258,7 @@ __gus_failed: \ · 234 unmodified lines | |||
| 255 | ".section .fixup,\"ax\"\n" \ | 258 | ".section .fixup,\"ax\"\n" \ |
| 256 | "4: li %0,%3\n" \ | 259 | "4: li %0,%3\n" \ |
| 257 | " li %1,0\n" \ | 260 | " li %1,0\n" \ |
| 258 | " li %1+1,0\n" \ | 261 | " li %L1,0\n" \ |
| 259 | " b 3b\n" \ | 262 | " b 3b\n" \ |
| 260 | ".previous\n" \ | 263 | ".previous\n" \ |
| 261 | EX_TABLE(1b, 4b) \ | 264 | EX_TABLE(1b, 4b) \ |
| @@ -326,40 +329,62 @@ do { \ · 64 unmodified lines | |||
| 326 | extern unsigned long __copy_tofrom_user(void __user *to, | 329 | extern unsigned long __copy_tofrom_user(void __user *to, |
| 327 | const void __user *from, unsigned long size); | 330 | const void __user *from, unsigned long size); |
| 328 | 331 | ||
| 329 | #ifdef __powerpc64__ | 332 | unsigned long __copy_tofrom_user_base(void __user *to, |
| 330 | static inline unsigned long | 333 | const void __user *from, unsigned long size); |
| 331 | raw_copy_in_user(void __user *to, const void __user *from, unsigned long n) | 334 | |
| 335 | unsigned long __copy_tofrom_user_power7_vmx(void __user *to, | ||
| 336 | const void __user *from, unsigned long size); | ||
| 337 | |||
| 338 | static __always_inline bool will_use_vmx(unsigned long n) | ||
| 339 | { | ||
| 340 | return IS_ENABLED(CONFIG_ALTIVEC) && cpu_has_feature(CPU_FTR_VMX_COPY) && | ||
| 341 | n > VMX_COPY_THRESHOLD; | ||
| 342 | } | ||
| 343 | |||
| 344 | static __always_inline unsigned long | ||
| 345 | raw_copy_tofrom_user(void __user *to, const void __user *from, | ||
| 346 | unsigned long n, unsigned long dir) | ||
| 332 | { | 347 | { |
| 333 | unsigned long ret; | 348 | unsigned long ret; |
| 334 | 349 | ||
| 335 | barrier_nospec(); | 350 | if (will_use_vmx(n) && enter_vmx_usercopy()) { |
| 336 | allow_user_access(to, KUAP_READ_WRITE); | 351 | allow_user_access(to, dir); |
| 352 | ret = __copy_tofrom_user_power7_vmx(to, from, n); | ||
| 353 | prevent_user_access(dir); | ||
| 354 | exit_vmx_usercopy(); | ||
| 355 | |||
| 356 | if (unlikely(ret)) { | ||
| 357 | allow_user_access(to, dir); | ||
| 358 | ret = __copy_tofrom_user_base(to, from, n); | ||
| 359 | prevent_user_access(dir); | ||
| 360 | } | ||
| 361 | return ret; | ||
| 362 | } | ||
| 363 | |||
| 364 | allow_user_access(to, dir); | ||
| 337 | ret = __copy_tofrom_user(to, from, n); | 365 | ret = __copy_tofrom_user(to, from, n); |
| 338 | prevent_user_access(KUAP_READ_WRITE); | 366 | prevent_user_access(dir); |
| 339 | return ret; | 367 | return ret; |
| 340 | } | 368 | } |
| 341 | #endif /* __powerpc64__ */ | ||
| 342 | 369 | ||
| 343 | static inline unsigned long raw_copy_from_user(void *to, | 370 | #ifdef CONFIG_PPC64 |
| 344 | const void __user *from, unsigned long n) | 371 | static inline unsigned long |
| 372 | raw_copy_in_user(void __user *to, const void __user *from, unsigned long n) | ||
| 345 | { | 373 | { |
| 346 | unsigned long ret; | 374 | barrier_nospec(); |
| 375 | return raw_copy_tofrom_user(to, from, n, KUAP_READ_WRITE); | ||
| 376 | } | ||
| 377 | #endif /* CONFIG_PPC64 */ | ||
| 347 | 378 | ||
| 348 | allow_user_access(NULL, KUAP_READ); | 379 | static inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n) |
| 349 | ret = __copy_tofrom_user((__force void __user *)to, from, n); | 380 | { |
| 350 | prevent_user_access(KUAP_READ); | 381 | return raw_copy_tofrom_user((__force void __user *)to, from, n, KUAP_READ); |
| 351 | return ret; | ||
| 352 | } | 382 | } |
| 353 | 383 | ||
| 354 | static inline unsigned long | 384 | static inline unsigned long |
| 355 | raw_copy_to_user(void __user *to, const void *from, unsigned long n) | 385 | raw_copy_to_user(void __user *to, const void *from, unsigned long n) |
| 356 | { | 386 | { |
| 357 | unsigned long ret; | 387 | return raw_copy_tofrom_user(to, (__force const void __user *)from, n, KUAP_WRITE); |
| 358 | |||
| 359 | allow_user_access(to, KUAP_WRITE); | ||
| 360 | ret = __copy_tofrom_user(to, (__force const void __user *)from, n); | ||
| 361 | prevent_user_access(KUAP_WRITE); | ||
| 362 | return ret; | ||
| 363 | } | 388 | } |
| 364 | 389 | ||
| 365 | unsigned long __arch_clear_user(void __user *addr, unsigned long size); | 390 | unsigned long __arch_clear_user(void __user *addr, unsigned long size); |
| @@ -67,7 +67,7 @@ bool arch_dma_unmap_sg_direct(struct device *dev, struct scatterlist *sg, · 66 unmodified lines | |||
| 67 | } | 67 | } |
| 68 | bool arch_dma_alloc_direct(struct device *dev) | 68 | bool arch_dma_alloc_direct(struct device *dev) |
| 69 | { | 69 | { |
| 70 | if (dev->dma_ops_bypass) | 70 | if (dev->dma_ops_bypass && dev->bus_dma_limit) |
| 71 | return true; | 71 | return true; |
| 72 | 72 | ||
| 73 | return false; | 73 | return false; |
| @@ -75,7 +75,7 @@ bool arch_dma_alloc_direct(struct device *dev) · 1 unmodified line | |||
| 75 | 75 | ||
| 76 | bool arch_dma_free_direct(struct device *dev, dma_addr_t dma_handle) | 76 | bool arch_dma_free_direct(struct device *dev, dma_addr_t dma_handle) |
| 77 | { | 77 | { |
| 78 | if (!dev->dma_ops_bypass) | 78 | if (!dev->dma_ops_bypass || !dev->bus_dma_limit) |
| 79 | return false; | 79 | return false; |
| 80 | 80 | ||
| 81 | return is_direct_handle(dev, dma_handle); | 81 | return is_direct_handle(dev, dma_handle); |
| @@ -305,7 +305,6 @@ set_ivor: · 304 unmodified lines | |||
| 305 | * r12 is pointer to the pte | 305 | * r12 is pointer to the pte |
| 306 | * r10 is the pshift from the PGD, if we're a hugepage | 306 | * r10 is the pshift from the PGD, if we're a hugepage |
| 307 | */ | 307 | */ |
| 308 | #ifdef CONFIG_PTE_64BIT | ||
| 309 | #ifdef CONFIG_HUGETLB_PAGE | 308 | #ifdef CONFIG_HUGETLB_PAGE |
| 310 | #define FIND_PTE \ | 309 | #define FIND_PTE \ |
| 311 | rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ | 310 | rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ |
| @@ -329,15 +328,6 @@ set_ivor: · 17 unmodified lines | |||
| 329 | rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ | 328 | rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ |
| 330 | lwz r11, 4(r12); /* Get pte entry */ | 329 | lwz r11, 4(r12); /* Get pte entry */ |
| 331 | #endif /* HUGEPAGE */ | 330 | #endif /* HUGEPAGE */ |
| 332 | #else /* !PTE_64BIT */ | ||
| 333 | #define FIND_PTE \ | ||
| 334 | rlwimi r11, r13, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ | ||
| 335 | lwz r11, 0(r11); /* Get L1 entry */ \ | ||
| 336 | rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \ | ||
| 337 | beq 2f; /* Bail if no table */ \ | ||
| 338 | rlwimi r12, r13, 22, 20, 29; /* Compute PTE address */ \ | ||
| 339 | lwz r11, 0(r12); /* Get Linux PTE */ | ||
| 340 | #endif | ||
| 341 | 331 | ||
| 342 | /* | 332 | /* |
| 343 | * Interrupt vector entry code | 333 | * Interrupt vector entry code |
| @@ -473,21 +463,15 @@ END_BTB_FLUSH_SECTION · 129 unmodified lines | |||
| 473 | 4: | 463 | 4: |
| 474 | FIND_PTE | 464 | FIND_PTE |
| 475 | 465 | ||
| 476 | #ifdef CONFIG_PTE_64BIT | ||
| 477 | li r13,_PAGE_PRESENT|_PAGE_BAP_SR | 466 | li r13,_PAGE_PRESENT|_PAGE_BAP_SR |
| 478 | oris r13,r13,_PAGE_ACCESSED@h | 467 | oris r13,r13,_PAGE_ACCESSED@h |
| 479 | #else | ||
| 480 | li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED | ||
| 481 | #endif | ||
| 482 | andc. r13,r13,r11 /* Check permission */ | 468 | andc. r13,r13,r11 /* Check permission */ |
| 483 | 469 | ||
| 484 | #ifdef CONFIG_PTE_64BIT | ||
| 485 | #ifdef CONFIG_SMP | 470 | #ifdef CONFIG_SMP |
| 486 | subf r13,r11,r12 /* create false data dep */ | 471 | subf r13,r11,r12 /* create false data dep */ |
| 487 | lwzx r13,r11,r13 /* Get upper pte bits */ | 472 | lwzx r13,r11,r13 /* Get upper pte bits */ |
| 488 | #else | 473 | #else |
| 489 | lwz r13,0(r12) /* Get upper pte bits */ | 474 | lwz r13,0(r12) /* Get upper pte bits */ |
| 490 | #endif | ||
| 491 | #endif | 475 | #endif |
| 492 | 476 | ||
| 493 | bne 2f /* Bail if permission/valid mismatch */ | 477 | bne 2f /* Bail if permission/valid mismatch */ |
| @@ -552,12 +536,8 @@ END_BTB_FLUSH_SECTION · 58 unmodified lines | |||
| 552 | 536 | ||
| 553 | FIND_PTE | 537 | FIND_PTE |
| 554 | /* Make up the required permissions for kernel code */ | 538 | /* Make up the required permissions for kernel code */ |
| 555 | #ifdef CONFIG_PTE_64BIT | ||
| 556 | li r13,_PAGE_PRESENT | _PAGE_BAP_SX | 539 | li r13,_PAGE_PRESENT | _PAGE_BAP_SX |
| 557 | oris r13,r13,_PAGE_ACCESSED@h | 540 | oris r13,r13,_PAGE_ACCESSED@h |
| 558 | #else | ||
| 559 | li r13,_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_EXEC | ||
| 560 | #endif | ||
| 561 | b 4f | 541 | b 4f |
| 562 | 542 | ||
| 563 | /* Get the PGD for the current thread */ | 543 | /* Get the PGD for the current thread */ |
| @@ -573,23 +553,17 @@ END_BTB_FLUSH_SECTION · 9 unmodified lines | |||
| 573 | 553 | ||
| 574 | FIND_PTE | 554 | FIND_PTE |
| 575 | /* Make up the required permissions for user code */ | 555 | /* Make up the required permissions for user code */ |
| 576 | #ifdef CONFIG_PTE_64BIT | ||
| 577 | li r13,_PAGE_PRESENT | _PAGE_BAP_UX | 556 | li r13,_PAGE_PRESENT | _PAGE_BAP_UX |
| 578 | oris r13,r13,_PAGE_ACCESSED@h | 557 | oris r13,r13,_PAGE_ACCESSED@h |
| 579 | #else | ||
| 580 | li r13,_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_EXEC | ||
| 581 | #endif | ||
| 582 | 558 | ||
| 583 | 4: | 559 | 4: |
| 584 | andc. r13,r13,r11 /* Check permission */ | 560 | andc. r13,r13,r11 /* Check permission */ |
| 585 | 561 | ||
| 586 | #ifdef CONFIG_PTE_64BIT | ||
| 587 | #ifdef CONFIG_SMP | 562 | #ifdef CONFIG_SMP |
| 588 | subf r13,r11,r12 /* create false data dep */ | 563 | subf r13,r11,r12 /* create false data dep */ |
| 589 | lwzx r13,r11,r13 /* Get upper pte bits */ | 564 | lwzx r13,r11,r13 /* Get upper pte bits */ |
| 590 | #else | 565 | #else |
| 591 | lwz r13,0(r12) /* Get upper pte bits */ | 566 | lwz r13,0(r12) /* Get upper pte bits */ |
| 592 | #endif | ||
| 593 | #endif | 567 | #endif |
| 594 | 568 | ||
| 595 | bne 2f /* Bail if permission mismatch */ | 569 | bne 2f /* Bail if permission mismatch */ |
| @@ -683,7 +657,7 @@ interrupt_end: · 87 unmodified lines | |||
| 683 | * r10 - tsize encoding (if HUGETLB_PAGE) or available to use | 657 | * r10 - tsize encoding (if HUGETLB_PAGE) or available to use |
| 684 | * r11 - TLB (info from Linux PTE) | 658 | * r11 - TLB (info from Linux PTE) |
| 685 | * r12 - available to use | 659 | * r12 - available to use |
| 686 | * r13 - upper bits of PTE (if PTE_64BIT) or available to use | 660 | * r13 - upper bits of PTE |
| 687 | * CR5 - results of addr >= PAGE_OFFSET | 661 | * CR5 - results of addr >= PAGE_OFFSET |
| 688 | * MAS0, MAS1 - loaded with proper value when we get here | 662 | * MAS0, MAS1 - loaded with proper value when we get here |
| 689 | * MAS2, MAS3 - will need additional info from Linux PTE | 663 | * MAS2, MAS3 - will need additional info from Linux PTE |
| @@ -751,7 +725,6 @@ finish_tlb_load: · 61 unmodified lines | |||
| 751 | * here we (properly should) assume have the appropriate value. | 725 | * here we (properly should) assume have the appropriate value. |
| 752 | */ | 726 | */ |
| 753 | finish_tlb_load_cont: | 727 | finish_tlb_load_cont: |
| 754 | #ifdef CONFIG_PTE_64BIT | ||
| 755 | rlwinm r12, r11, 32-2, 26, 31 /* Move in perm bits */ | 728 | rlwinm r12, r11, 32-2, 26, 31 /* Move in perm bits */ |
| 756 | andi. r10, r11, _PAGE_DIRTY | 729 | andi. r10, r11, _PAGE_DIRTY |
| 757 | bne 1f | 730 | bne 1f |
| @@ -764,26 +737,9 @@ BEGIN_MMU_FTR_SECTION · 6 unmodified lines | |||
| 764 | srwi r10, r13, 12 /* grab RPN[12:31] */ | 737 | srwi r10, r13, 12 /* grab RPN[12:31] */ |
| 765 | mtspr SPRN_MAS7, r10 | 738 | mtspr SPRN_MAS7, r10 |
| 766 | END_MMU_FTR_SECTION_IFSET(MMU_FTR_BIG_PHYS) | 739 | END_MMU_FTR_SECTION_IFSET(MMU_FTR_BIG_PHYS) |
| 767 | #else | ||
| 768 | li r10, (_PAGE_EXEC | _PAGE_READ) | ||
| 769 | mr r13, r11 | ||
| 770 | rlwimi r10, r11, 31, 29, 29 /* extract _PAGE_DIRTY into SW */ | ||
| 771 | and r12, r11, r10 | ||
| 772 | mcrf cr0, cr5 /* Test for user page */ | ||
| 773 | slwi r10, r12, 1 | ||
| 774 | or r10, r10, r12 | ||
| 775 | rlwinm r10, r10, 0, ~_PAGE_EXEC /* Clear SX on user pages */ | ||
| 776 | isellt r12, r10, r12 | ||
| 777 | rlwimi r13, r12, 0, 20, 31 /* Get RPN from PTE, merge w/ perms */ | ||
| 778 | mtspr SPRN_MAS3, r13 | ||
| 779 | #endif | ||
| 780 | 740 | ||
| 781 | mfspr r12, SPRN_MAS2 | 741 | mfspr r12, SPRN_MAS2 |
| 782 | #ifdef CONFIG_PTE_64BIT | ||
| 783 | rlwimi r12, r11, 32-19, 27, 31 /* extract WIMGE from pte */ | 742 | rlwimi r12, r11, 32-19, 27, 31 /* extract WIMGE from pte */ |
| 784 | #else | ||
| 785 | rlwimi r12, r11, 26, 27, 31 /* extract WIMGE from pte */ | ||
| 786 | #endif | ||
| 787 | #ifdef CONFIG_HUGETLB_PAGE | 743 | #ifdef CONFIG_HUGETLB_PAGE |
| 788 | beq 6, 3f /* don't mask if page isn't huge */ | 744 | beq 6, 3f /* don't mask if page isn't huge */ |
| 789 | li r13, 1 | 745 | li r13, 1 |
| @@ -1159,7 +1159,7 @@ spapr_tce_platform_iommu_attach_dev(struct iommu_domain *platform_domain, · 1158 unmodified lines | |||
| 1159 | struct device *dev, | 1159 | struct device *dev, |
| 1160 | struct iommu_domain *old) | 1160 | struct iommu_domain *old) |
| 1161 | { | 1161 | { |
| 1162 | struct iommu_domain *domain = iommu_get_domain_for_dev(dev); | 1162 | struct iommu_domain *domain = iommu_driver_get_domain_for_dev(dev); |
| 1163 | struct iommu_table_group *table_group; | 1163 | struct iommu_table_group *table_group; |
| 1164 | struct iommu_group *grp; | 1164 | struct iommu_group *grp; |
| 1165 | 1165 | ||
| @@ -2893,7 +2893,8 @@ static void __init fixup_device_tree_pmac(void) · 2892 unmodified lines | |||
| 2893 | for (node = 0; prom_next_node(&node); ) { | 2893 | for (node = 0; prom_next_node(&node); ) { |
| 2894 | type[0] = '\0'; | 2894 | type[0] = '\0'; |
| 2895 | prom_getprop(node, "device_type", type, sizeof(type)); | 2895 | prom_getprop(node, "device_type", type, sizeof(type)); |
| 2896 | if (prom_strcmp(type, "escc") && prom_strcmp(type, "i2s")) | 2896 | if (prom_strcmp(type, "escc") && prom_strcmp(type, "i2s") && |
| 2897 | prom_strcmp(type, "media-bay")) | ||
| 2897 | continue; | 2898 | continue; |
| 2898 | 2899 | ||
| 2899 | if (prom_getproplen(node, "#size-cells") != PROM_ERROR) | 2900 | if (prom_getproplen(node, "#size-cells") != PROM_ERROR) |
| @@ -35,7 +35,6 @@ · 34 unmodified lines | |||
| 35 | #include <linux/of_irq.h> | 35 | #include <linux/of_irq.h> |
| 36 | #include <linux/hugetlb.h> | 36 | #include <linux/hugetlb.h> |
| 37 | #include <linux/pgtable.h> | 37 | #include <linux/pgtable.h> |
| 38 | #include <asm/kexec.h> | ||
| 39 | #include <asm/io.h> | 38 | #include <asm/io.h> |
| 40 | #include <asm/paca.h> | 39 | #include <asm/paca.h> |
| 41 | #include <asm/processor.h> | 40 | #include <asm/processor.h> |
| @@ -995,15 +994,6 @@ void __init setup_arch(char **cmdline_p) · 953 unmodified lines | |||
| 995 | 994 | ||
| 996 | initmem_init(); | 995 | initmem_init(); |
| 997 | 996 | ||
| 998 | /* | ||
| 999 | * Reserve large chunks of memory for use by CMA for kdump, fadump, KVM and | ||
| 1000 | * hugetlb. These must be called after initmem_init(), so that | ||
| 1001 | * pageblock_order is initialised. | ||
| 1002 | */ | ||
| 1003 | fadump_cma_init(); | ||
| 1004 | kdump_cma_reserve(); | ||
| 1005 | kvm_cma_reserve(); | ||
| 1006 | |||
| 1007 | early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT); | 997 | early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT); |
| 1008 | 998 | ||
| 1009 | if (ppc_md.setup_arch) | 999 | if (ppc_md.setup_arch) |
| @@ -37,11 +37,29 @@ unsigned long ftrace_call_adjust(unsigned long addr) · 36 unmodified lines | |||
| 37 | if (addr >= (unsigned long)__exittext_begin && addr < (unsigned long)__exittext_end) | 37 | if (addr >= (unsigned long)__exittext_begin && addr < (unsigned long)__exittext_end) |
| 38 | return 0; | 38 | return 0; |
| 39 | 39 | ||
| 40 | if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY) && | 40 | if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)) { |
| 41 | !IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { | 41 | if (!IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { |
| 42 | addr += MCOUNT_INSN_SIZE; | ||
| 43 | if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) | ||
| 44 | addr += MCOUNT_INSN_SIZE; | 42 | addr += MCOUNT_INSN_SIZE; |
| 43 | if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) | ||
| 44 | addr += MCOUNT_INSN_SIZE; | ||
| 45 | } else if (IS_ENABLED(CONFIG_CC_IS_CLANG) && IS_ENABLED(CONFIG_PPC64)) { | ||
| 46 | /* | ||
| 47 | * addr points to global entry point though the NOP was emitted at local | ||
| 48 | * entry point due to https://github.com/llvm/llvm-project/issues/163706 | ||
| 49 | * Handle that here with ppc_function_entry() for kernel symbols while | ||
| 50 | * adjusting module addresses in the else case, by looking for the below | ||
| 51 | * module global entry point sequence: | ||
| 52 | * ld r2, -8(r12) | ||
| 53 | * add r2, r2, r12 | ||
| 54 | */ | ||
| 55 | if (is_kernel_text(addr) || is_kernel_inittext(addr)) | ||
| 56 | addr = ppc_function_entry((void *)addr); | ||
| 57 | else if ((ppc_inst_val(ppc_inst_read((u32 *)addr)) == | ||
| 58 | PPC_RAW_LD(_R2, _R12, -8)) && | ||
| 59 | (ppc_inst_val(ppc_inst_read((u32 *)(addr+4))) == | ||
| 60 | PPC_RAW_ADD(_R2, _R2, _R12))) | ||
| 61 | addr += 8; | ||
| 62 | } | ||
| 45 | } | 63 | } |
| 46 | 64 | ||
| 47 | return addr; | 65 | return addr; |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | |||
| 23 | #include <asm/firmware.h> | 23 | #include <asm/firmware.h> |
| 24 | 24 | ||
| 25 | #define cpu_to_be_ulong __PASTE(cpu_to_be, BITS_PER_LONG) | 25 | #define cpu_to_be_ulong __PASTE(cpu_to_be, BITS_PER_LONG) |
| 26 | #define __be_word __PASTE(__be, BITS_PER_LONG) | ||
| 26 | 27 | ||
| 27 | #ifdef CONFIG_CRASH_DUMP | 28 | #ifdef CONFIG_CRASH_DUMP |
| 28 | void machine_crash_shutdown(struct pt_regs *regs) | 29 | void machine_crash_shutdown(struct pt_regs *regs) |
| @@ -146,25 +147,25 @@ int __init overlaps_crashkernel(unsigned long start, unsigned long size) · 117 unmodified lines | |||
| 146 | } | 147 | } |
| 147 | 148 | ||
| 148 | /* Values we need to export to the second kernel via the device tree. */ | 149 | /* Values we need to export to the second kernel via the device tree. */ |
| 149 | static phys_addr_t crashk_base; | 150 | static __be_word crashk_base; |
| 150 | static phys_addr_t crashk_size; | 151 | static __be_word crashk_size; |
| 151 | static unsigned long long mem_limit; | 152 | static __be_word mem_limit; |
| 152 | 153 | ||
| 153 | static struct property crashk_base_prop = { | 154 | static struct property crashk_base_prop = { |
| 154 | .name = "linux,crashkernel-base", | 155 | .name = "linux,crashkernel-base", |
| 155 | .length = sizeof(phys_addr_t), | 156 | .length = sizeof(__be_word), |
| 156 | .value = &crashk_base | 157 | .value = &crashk_base |
| 157 | }; | 158 | }; |
| 158 | 159 | ||
| 159 | static struct property crashk_size_prop = { | 160 | static struct property crashk_size_prop = { |
| 160 | .name = "linux,crashkernel-size", | 161 | .name = "linux,crashkernel-size", |
| 161 | .length = sizeof(phys_addr_t), | 162 | .length = sizeof(__be_word), |
| 162 | .value = &crashk_size, | 163 | .value = &crashk_size, |
| 163 | }; | 164 | }; |
| 164 | 165 | ||
| 165 | static struct property memory_limit_prop = { | 166 | static struct property memory_limit_prop = { |
| 166 | .name = "linux,memory-limit", | 167 | .name = "linux,memory-limit", |
| 167 | .length = sizeof(unsigned long long), | 168 | .length = sizeof(__be_word), |
| 168 | .value = &mem_limit, | 169 | .value = &mem_limit, |
| 169 | }; | 170 | }; |
| 170 | 171 | ||
| @@ -193,11 +194,11 @@ static void __init export_crashk_values(struct device_node *node) · 22 unmodified lines | |||
| 193 | } | 194 | } |
| 194 | #endif /* CONFIG_CRASH_RESERVE */ | 195 | #endif /* CONFIG_CRASH_RESERVE */ |
| 195 | 196 | ||
| 196 | static phys_addr_t kernel_end; | 197 | static __be_word kernel_end; |
| 197 | 198 | ||
| 198 | static struct property kernel_end_prop = { | 199 | static struct property kernel_end_prop = { |
| 199 | .name = "linux,kernel-end", | 200 | .name = "linux,kernel-end", |
| 200 | .length = sizeof(phys_addr_t), | 201 | .length = sizeof(__be_word), |
| 201 | .value = &kernel_end, | 202 | .value = &kernel_end, |
| 202 | }; | 203 | }; |
| 203 | 204 | ||
| @@ -450,6 +450,11 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) · 449 unmodified lines | |||
| 450 | kbuf->buffer = headers; | 450 | kbuf->buffer = headers; |
| 451 | kbuf->mem = KEXEC_BUF_MEM_UNKNOWN; | 451 | kbuf->mem = KEXEC_BUF_MEM_UNKNOWN; |
| 452 | kbuf->bufsz = headers_sz; | 452 | kbuf->bufsz = headers_sz; |
| 453 | |||
| 454 | /* | ||
| 455 | * Account for extra space required to accommodate additional memory | ||
| 456 | * ranges in elfcorehdr due to memory hotplug events. | ||
| 457 | */ | ||
| 453 | kbuf->memsz = headers_sz + kdump_extra_elfcorehdr_size(cmem); | 458 | kbuf->memsz = headers_sz + kdump_extra_elfcorehdr_size(cmem); |
| 454 | kbuf->top_down = false; | 459 | kbuf->top_down = false; |
| 455 | 460 | ||
| @@ -460,7 +465,14 @@ static int load_elfcorehdr_segment(struct kimage *image, struct kexec_buf *kbuf) · 4 unmodified lines | |||
| 460 | } | 465 | } |
| 461 | 466 | ||
| 462 | image->elf_load_addr = kbuf->mem; | 467 | image->elf_load_addr = kbuf->mem; |
| 463 | image->elf_headers_sz = headers_sz; | 468 | |
| 469 | /* | ||
| 470 | * If CONFIG_CRASH_HOTPLUG is enabled, the elfcorehdr kexec segment | ||
| 471 | * memsz can be larger than bufsz. Always initialize elf_headers_sz | ||
| 472 | * with memsz. This ensures the correct size is reserved for elfcorehdr | ||
| 473 | * memory in the FDT prepared for kdump. | ||
| 474 | */ | ||
| 475 | image->elf_headers_sz = kbuf->memsz; | ||
| 464 | image->elf_headers = headers; | 476 | image->elf_headers = headers; |
| 465 | out: | 477 | out: |
| 466 | kfree(cmem); | 478 | kfree(cmem); |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | |||
| 38 | 38 | ||
| 39 | /* #define EXIT_DEBUG */ | 39 | /* #define EXIT_DEBUG */ |
| 40 | 40 | ||
| 41 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 41 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 42 | KVM_GENERIC_VM_STATS(), | 42 | KVM_GENERIC_VM_STATS(), |
| 43 | STATS_DESC_ICOUNTER(VM, num_2M_pages), | 43 | STATS_DESC_ICOUNTER(VM, num_2M_pages), |
| 44 | STATS_DESC_ICOUNTER(VM, num_1G_pages) | 44 | STATS_DESC_ICOUNTER(VM, num_1G_pages) |
| @@ -53,7 +53,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 8 unmodified lines | |||
| 53 | sizeof(kvm_vm_stats_desc), | 53 | sizeof(kvm_vm_stats_desc), |
| 54 | }; | 54 | }; |
| 55 | 55 | ||
| 56 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 56 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 57 | KVM_GENERIC_VCPU_STATS(), | 57 | KVM_GENERIC_VCPU_STATS(), |
| 58 | STATS_DESC_COUNTER(VCPU, sum_exits), | 58 | STATS_DESC_COUNTER(VCPU, sum_exits), |
| 59 | STATS_DESC_COUNTER(VCPU, mmio_exits), | 59 | STATS_DESC_COUNTER(VCPU, mmio_exits), |
| @@ -36,7 +36,7 @@ · 35 unmodified lines | |||
| 36 | 36 | ||
| 37 | unsigned long kvmppc_booke_handlers; | 37 | unsigned long kvmppc_booke_handlers; |
| 38 | 38 | ||
| 39 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 39 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 40 | KVM_GENERIC_VM_STATS(), | 40 | KVM_GENERIC_VM_STATS(), |
| 41 | STATS_DESC_ICOUNTER(VM, num_2M_pages), | 41 | STATS_DESC_ICOUNTER(VM, num_2M_pages), |
| 42 | STATS_DESC_ICOUNTER(VM, num_1G_pages) | 42 | STATS_DESC_ICOUNTER(VM, num_1G_pages) |
| @@ -51,7 +51,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 8 unmodified lines | |||
| 51 | sizeof(kvm_vm_stats_desc), | 51 | sizeof(kvm_vm_stats_desc), |
| 52 | }; | 52 | }; |
| 53 | 53 | ||
| 54 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 54 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 55 | KVM_GENERIC_VCPU_STATS(), | 55 | KVM_GENERIC_VCPU_STATS(), |
| 56 | STATS_DESC_COUNTER(VCPU, sum_exits), | 56 | STATS_DESC_COUNTER(VCPU, sum_exits), |
| 57 | STATS_DESC_COUNTER(VCPU, mmio_exits), | 57 | STATS_DESC_COUNTER(VCPU, mmio_exits), |
| @@ -39,15 +39,11 @@ enum vcpu_ftr { · 38 unmodified lines | |||
| 39 | /* bits [6-5] MAS2_X1 and MAS2_X0 and [4-0] bits for WIMGE */ | 39 | /* bits [6-5] MAS2_X1 and MAS2_X0 and [4-0] bits for WIMGE */ |
| 40 | #define E500_TLB_MAS2_ATTR (0x7f) | 40 | #define E500_TLB_MAS2_ATTR (0x7f) |
| 41 | 41 | ||
| 42 | struct tlbe_ref { | 42 | struct tlbe_priv { |
| 43 | kvm_pfn_t pfn; /* valid only for TLB0, except briefly */ | 43 | kvm_pfn_t pfn; /* valid only for TLB0, except briefly */ |
| 44 | unsigned int flags; /* E500_TLB_* */ | 44 | unsigned int flags; /* E500_TLB_* */ |
| 45 | }; | 45 | }; |
| 46 | 46 | ||
| 47 | struct tlbe_priv { | ||
| 48 | struct tlbe_ref ref; | ||
| 49 | }; | ||
| 50 | |||
| 51 | #ifdef CONFIG_KVM_E500V2 | 47 | #ifdef CONFIG_KVM_E500V2 |
| 52 | struct vcpu_id_table; | 48 | struct vcpu_id_table; |
| 53 | #endif | 49 | #endif |
| @@ -920,12 +920,12 @@ int kvmppc_e500_tlb_init(struct kvmppc_vcpu_e500 *vcpu_e500) · 919 unmodified lines | |||
| 920 | vcpu_e500->gtlb_offset[0] = 0; | 920 | vcpu_e500->gtlb_offset[0] = 0; |
| 921 | vcpu_e500->gtlb_offset[1] = KVM_E500_TLB0_SIZE; | 921 | vcpu_e500->gtlb_offset[1] = KVM_E500_TLB0_SIZE; |
| 922 | 922 | ||
| 923 | vcpu_e500->gtlb_priv[0] = kzalloc_objs(struct tlbe_ref, | 923 | vcpu_e500->gtlb_priv[0] = kzalloc_objs(struct tlbe_priv, |
| 924 | vcpu_e500->gtlb_params[0].entries); | 924 | vcpu_e500->gtlb_params[0].entries); |
| 925 | if (!vcpu_e500->gtlb_priv[0]) | 925 | if (!vcpu_e500->gtlb_priv[0]) |
| 926 | goto free_vcpu; | 926 | goto free_vcpu; |
| 927 | 927 | ||
| 928 | vcpu_e500->gtlb_priv[1] = kzalloc_objs(struct tlbe_ref, | 928 | vcpu_e500->gtlb_priv[1] = kzalloc_objs(struct tlbe_priv, |
| 929 | vcpu_e500->gtlb_params[1].entries); | 929 | vcpu_e500->gtlb_params[1].entries); |
| 930 | if (!vcpu_e500->gtlb_priv[1]) | 930 | if (!vcpu_e500->gtlb_priv[1]) |
| 931 | goto free_vcpu; | 931 | goto free_vcpu; |
| @@ -189,16 +189,16 @@ void inval_gtlbe_on_host(struct kvmppc_vcpu_e500 *vcpu_e500, int tlbsel, · 188 unmodified lines | |||
| 189 | { | 189 | { |
| 190 | struct kvm_book3e_206_tlb_entry *gtlbe = | 190 | struct kvm_book3e_206_tlb_entry *gtlbe = |
| 191 | get_entry(vcpu_e500, tlbsel, esel); | 191 | get_entry(vcpu_e500, tlbsel, esel); |
| 192 | struct tlbe_ref *ref = &vcpu_e500->gtlb_priv[tlbsel][esel].ref; | 192 | struct tlbe_priv *tlbe = &vcpu_e500->gtlb_priv[tlbsel][esel]; |
| 193 | 193 | ||
| 194 | /* Don't bother with unmapped entries */ | 194 | /* Don't bother with unmapped entries */ |
| 195 | if (!(ref->flags & E500_TLB_VALID)) { | 195 | if (!(tlbe->flags & E500_TLB_VALID)) { |
| 196 | WARN(ref->flags & (E500_TLB_BITMAP | E500_TLB_TLB0), | 196 | WARN(tlbe->flags & (E500_TLB_BITMAP | E500_TLB_TLB0), |
| 197 | "%s: flags %x\n", __func__, ref->flags); | 197 | "%s: flags %x\n", __func__, tlbe->flags); |
| 198 | WARN_ON(tlbsel == 1 && vcpu_e500->g2h_tlb1_map[esel]); | 198 | WARN_ON(tlbsel == 1 && vcpu_e500->g2h_tlb1_map[esel]); |
| 199 | } | 199 | } |
| 200 | 200 | ||
| 201 | if (tlbsel == 1 && ref->flags & E500_TLB_BITMAP) { | 201 | if (tlbsel == 1 && tlbe->flags & E500_TLB_BITMAP) { |
| 202 | u64 tmp = vcpu_e500->g2h_tlb1_map[esel]; | 202 | u64 tmp = vcpu_e500->g2h_tlb1_map[esel]; |
| 203 | int hw_tlb_indx; | 203 | int hw_tlb_indx; |
| 204 | unsigned long flags; | 204 | unsigned long flags; |
| @@ -216,28 +216,28 @@ void inval_gtlbe_on_host(struct kvmppc_vcpu_e500 *vcpu_e500, int tlbsel, · 11 unmodified lines | |||
| 216 | } | 216 | } |
| 217 | mb(); | 217 | mb(); |
| 218 | vcpu_e500->g2h_tlb1_map[esel] = 0; | 218 | vcpu_e500->g2h_tlb1_map[esel] = 0; |
| 219 | ref->flags &= ~(E500_TLB_BITMAP | E500_TLB_VALID); | 219 | tlbe->flags &= ~(E500_TLB_BITMAP | E500_TLB_VALID); |
| 220 | local_irq_restore(flags); | 220 | local_irq_restore(flags); |
| 221 | } | 221 | } |
| 222 | 222 | ||
| 223 | if (tlbsel == 1 && ref->flags & E500_TLB_TLB0) { | 223 | if (tlbsel == 1 && tlbe->flags & E500_TLB_TLB0) { |
| 224 | /* | 224 | /* |
| 225 | * TLB1 entry is backed by 4k pages. This should happen | 225 | * TLB1 entry is backed by 4k pages. This should happen |
| 226 | * rarely and is not worth optimizing. Invalidate everything. | 226 | * rarely and is not worth optimizing. Invalidate everything. |
| 227 | */ | 227 | */ |
| 228 | kvmppc_e500_tlbil_all(vcpu_e500); | 228 | kvmppc_e500_tlbil_all(vcpu_e500); |
| 229 | ref->flags &= ~(E500_TLB_TLB0 | E500_TLB_VALID); | 229 | tlbe->flags &= ~(E500_TLB_TLB0 | E500_TLB_VALID); |
| 230 | } | 230 | } |
| 231 | 231 | ||
| 232 | /* | 232 | /* |
| 233 | * If TLB entry is still valid then it's a TLB0 entry, and thus | 233 | * If TLB entry is still valid then it's a TLB0 entry, and thus |
| 234 | * backed by at most one host tlbe per shadow pid | 234 | * backed by at most one host tlbe per shadow pid |
| 235 | */ | 235 | */ |
| 236 | if (ref->flags & E500_TLB_VALID) | 236 | if (tlbe->flags & E500_TLB_VALID) |
| 237 | kvmppc_e500_tlbil_one(vcpu_e500, gtlbe); | 237 | kvmppc_e500_tlbil_one(vcpu_e500, gtlbe); |
| 238 | 238 | ||
| 239 | /* Mark the TLB as not backed by the host anymore */ | 239 | /* Mark the TLB as not backed by the host anymore */ |
| 240 | ref->flags = 0; | 240 | tlbe->flags = 0; |
| 241 | } | 241 | } |
| 242 | 242 | ||
| 243 | static inline int tlbe_is_writable(struct kvm_book3e_206_tlb_entry *tlbe) | 243 | static inline int tlbe_is_writable(struct kvm_book3e_206_tlb_entry *tlbe) |
| @@ -245,26 +245,26 @@ static inline int tlbe_is_writable(struct kvm_book3e_206_tlb_entry *tlbe) · 1 unmodified line | |||
| 245 | return tlbe->mas7_3 & (MAS3_SW|MAS3_UW); | 245 | return tlbe->mas7_3 & (MAS3_SW|MAS3_UW); |
| 246 | } | 246 | } |
| 247 | 247 | ||
| 248 | static inline void kvmppc_e500_ref_setup(struct tlbe_ref *ref, | 248 | static inline void kvmppc_e500_tlbe_setup(struct tlbe_priv *tlbe, |
| 249 | struct kvm_book3e_206_tlb_entry *gtlbe, | 249 | struct kvm_book3e_206_tlb_entry *gtlbe, |
| 250 | kvm_pfn_t pfn, unsigned int wimg, | 250 | kvm_pfn_t pfn, unsigned int wimg, |
| 251 | bool writable) | 251 | bool writable) |
| 252 | { | 252 | { |
| 253 | ref->pfn = pfn; | 253 | tlbe->pfn = pfn; |
| 254 | ref->flags = E500_TLB_VALID; | 254 | tlbe->flags = E500_TLB_VALID; |
| 255 | if (writable) | 255 | if (writable) |
| 256 | ref->flags |= E500_TLB_WRITABLE; | 256 | tlbe->flags |= E500_TLB_WRITABLE; |
| 257 | 257 | ||
| 258 | /* Use guest supplied MAS2_G and MAS2_E */ | 258 | /* Use guest supplied MAS2_G and MAS2_E */ |
| 259 | ref->flags |= (gtlbe->mas2 & MAS2_ATTRIB_MASK) | wimg; | 259 | tlbe->flags |= (gtlbe->mas2 & MAS2_ATTRIB_MASK) | wimg; |
| 260 | } | 260 | } |
| 261 | 261 | ||
| 262 | static inline void kvmppc_e500_ref_release(struct tlbe_ref *ref) | 262 | static inline void kvmppc_e500_tlbe_release(struct tlbe_priv *tlbe) |
| 263 | { | 263 | { |
| 264 | if (ref->flags & E500_TLB_VALID) { | 264 | if (tlbe->flags & E500_TLB_VALID) { |
| 265 | /* FIXME: don't log bogus pfn for TLB1 */ | 265 | /* FIXME: don't log bogus pfn for TLB1 */ |
| 266 | trace_kvm_booke206_ref_release(ref->pfn, ref->flags); | 266 | trace_kvm_booke206_ref_release(tlbe->pfn, tlbe->flags); |
| 267 | ref->flags = 0; | 267 | tlbe->flags = 0; |
| 268 | } | 268 | } |
| 269 | } | 269 | } |
| 270 | 270 | ||
| @@ -284,11 +284,8 @@ static void clear_tlb_privs(struct kvmppc_vcpu_e500 *vcpu_e500) · 13 unmodified lines | |||
| 284 | int i; | 284 | int i; |
| 285 | 285 | ||
| 286 | for (tlbsel = 0; tlbsel <= 1; tlbsel++) { | 286 | for (tlbsel = 0; tlbsel <= 1; tlbsel++) { |
| 287 | for (i = 0; i < vcpu_e500->gtlb_params[tlbsel].entries; i++) { | 287 | for (i = 0; i < vcpu_e500->gtlb_params[tlbsel].entries; i++) |
| 288 | struct tlbe_ref *ref = | 288 | kvmppc_e500_tlbe_release(&vcpu_e500->gtlb_priv[tlbsel][i]); |
| 289 | &vcpu_e500->gtlb_priv[tlbsel][i].ref; | ||
| 290 | kvmppc_e500_ref_release(ref); | ||
| 291 | } | ||
| 292 | } | 289 | } |
| 293 | } | 290 | } |
| 294 | 291 | ||
| @@ -304,18 +301,18 @@ void kvmppc_core_flush_tlb(struct kvm_vcpu *vcpu) · 9 unmodified lines | |||
| 304 | static void kvmppc_e500_setup_stlbe( | 301 | static void kvmppc_e500_setup_stlbe( |
| 305 | struct kvm_vcpu *vcpu, | 302 | struct kvm_vcpu *vcpu, |
| 306 | struct kvm_book3e_206_tlb_entry *gtlbe, | 303 | struct kvm_book3e_206_tlb_entry *gtlbe, |
| 307 | int tsize, struct tlbe_ref *ref, u64 gvaddr, | 304 | int tsize, struct tlbe_priv *tlbe, u64 gvaddr, |
| 308 | struct kvm_book3e_206_tlb_entry *stlbe) | 305 | struct kvm_book3e_206_tlb_entry *stlbe) |
| 309 | { | 306 | { |
| 310 | kvm_pfn_t pfn = ref->pfn; | 307 | kvm_pfn_t pfn = tlbe->pfn; |
| 311 | u32 pr = vcpu->arch.shared->msr & MSR_PR; | 308 | u32 pr = vcpu->arch.shared->msr & MSR_PR; |
| 312 | bool writable = !!(ref->flags & E500_TLB_WRITABLE); | 309 | bool writable = !!(tlbe->flags & E500_TLB_WRITABLE); |
| 313 | 310 | ||
| 314 | BUG_ON(!(ref->flags & E500_TLB_VALID)); | 311 | BUG_ON(!(tlbe->flags & E500_TLB_VALID)); |
| 315 | 312 | ||
| 316 | /* Force IPROT=0 for all guest mappings. */ | 313 | /* Force IPROT=0 for all guest mappings. */ |
| 317 | stlbe->mas1 = MAS1_TSIZE(tsize) | get_tlb_sts(gtlbe) | MAS1_VALID; | 314 | stlbe->mas1 = MAS1_TSIZE(tsize) | get_tlb_sts(gtlbe) | MAS1_VALID; |
| 318 | stlbe->mas2 = (gvaddr & MAS2_EPN) | (ref->flags & E500_TLB_MAS2_ATTR); | 315 | stlbe->mas2 = (gvaddr & MAS2_EPN) | (tlbe->flags & E500_TLB_MAS2_ATTR); |
| 319 | stlbe->mas7_3 = ((u64)pfn << PAGE_SHIFT) | | 316 | stlbe->mas7_3 = ((u64)pfn << PAGE_SHIFT) | |
| 320 | e500_shadow_mas3_attrib(gtlbe->mas7_3, writable, pr); | 317 | e500_shadow_mas3_attrib(gtlbe->mas7_3, writable, pr); |
| 321 | } | 318 | } |
| @@ -323,7 +320,7 @@ static void kvmppc_e500_setup_stlbe( · 1 unmodified line | |||
| 323 | static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, | 320 | static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, |
| 324 | u64 gvaddr, gfn_t gfn, struct kvm_book3e_206_tlb_entry *gtlbe, | 321 | u64 gvaddr, gfn_t gfn, struct kvm_book3e_206_tlb_entry *gtlbe, |
| 325 | int tlbsel, struct kvm_book3e_206_tlb_entry *stlbe, | 322 | int tlbsel, struct kvm_book3e_206_tlb_entry *stlbe, |
| 326 | struct tlbe_ref *ref) | 323 | struct tlbe_priv *tlbe) |
| 327 | { | 324 | { |
| 328 | struct kvm_memory_slot *slot; | 325 | struct kvm_memory_slot *slot; |
| 329 | unsigned int psize; | 326 | unsigned int psize; |
| @@ -455,9 +452,9 @@ static inline int kvmppc_e500_shadow_map(struct kvmppc_vcpu_e500 *vcpu_e500, · 125 unmodified lines | |||
| 455 | } | 452 | } |
| 456 | } | 453 | } |
| 457 | 454 | ||
| 458 | kvmppc_e500_ref_setup(ref, gtlbe, pfn, wimg, writable); | 455 | kvmppc_e500_tlbe_setup(tlbe, gtlbe, pfn, wimg, writable); |
| 459 | kvmppc_e500_setup_stlbe(&vcpu_e500->vcpu, gtlbe, tsize, | 456 | kvmppc_e500_setup_stlbe(&vcpu_e500->vcpu, gtlbe, tsize, |
| 460 | ref, gvaddr, stlbe); | 457 | tlbe, gvaddr, stlbe); |
| 461 | writable = tlbe_is_writable(stlbe); | 458 | writable = tlbe_is_writable(stlbe); |
| 462 | 459 | ||
| 463 | /* Clear i-cache for new pages */ | 460 | /* Clear i-cache for new pages */ |
| @@ -474,17 +471,17 @@ static int kvmppc_e500_tlb0_map(struct kvmppc_vcpu_e500 *vcpu_e500, int esel, · 10 unmodified lines | |||
| 474 | struct kvm_book3e_206_tlb_entry *stlbe) | 471 | struct kvm_book3e_206_tlb_entry *stlbe) |
| 475 | { | 472 | { |
| 476 | struct kvm_book3e_206_tlb_entry *gtlbe; | 473 | struct kvm_book3e_206_tlb_entry *gtlbe; |
| 477 | struct tlbe_ref *ref; | 474 | struct tlbe_priv *tlbe; |
| 478 | int stlbsel = 0; | 475 | int stlbsel = 0; |
| 479 | int sesel = 0; | 476 | int sesel = 0; |
| 480 | int r; | 477 | int r; |
| 481 | 478 | ||
| 482 | gtlbe = get_entry(vcpu_e500, 0, esel); | 479 | gtlbe = get_entry(vcpu_e500, 0, esel); |
| 483 | ref = &vcpu_e500->gtlb_priv[0][esel].ref; | 480 | tlbe = &vcpu_e500->gtlb_priv[0][esel]; |
| 484 | 481 | ||
| 485 | r = kvmppc_e500_shadow_map(vcpu_e500, get_tlb_eaddr(gtlbe), | 482 | r = kvmppc_e500_shadow_map(vcpu_e500, get_tlb_eaddr(gtlbe), |
| 486 | get_tlb_raddr(gtlbe) >> PAGE_SHIFT, | 483 | get_tlb_raddr(gtlbe) >> PAGE_SHIFT, |
| 487 | gtlbe, 0, stlbe, ref); | 484 | gtlbe, 0, stlbe, tlbe); |
| 488 | if (r) | 485 | if (r) |
| 489 | return r; | 486 | return r; |
| 490 | 487 | ||
| @@ -494,7 +491,7 @@ static int kvmppc_e500_tlb0_map(struct kvmppc_vcpu_e500 *vcpu_e500, int esel, · 3 unmodified lines | |||
| 494 | } | 491 | } |
| 495 | 492 | ||
| 496 | static int kvmppc_e500_tlb1_map_tlb1(struct kvmppc_vcpu_e500 *vcpu_e500, | 493 | static int kvmppc_e500_tlb1_map_tlb1(struct kvmppc_vcpu_e500 *vcpu_e500, |
| 497 | struct tlbe_ref *ref, | 494 | struct tlbe_priv *tlbe, |
| 498 | int esel) | 495 | int esel) |
| 499 | { | 496 | { |
| 500 | unsigned int sesel = vcpu_e500->host_tlb1_nv++; | 497 | unsigned int sesel = vcpu_e500->host_tlb1_nv++; |
| @@ -507,10 +504,10 @@ static int kvmppc_e500_tlb1_map_tlb1(struct kvmppc_vcpu_e500 *vcpu_e500, · 6 unmodified lines | |||
| 507 | vcpu_e500->g2h_tlb1_map[idx] &= ~(1ULL << sesel); | 504 | vcpu_e500->g2h_tlb1_map[idx] &= ~(1ULL << sesel); |
| 508 | } | 505 | } |
| 509 | 506 | ||
| 510 | vcpu_e500->gtlb_priv[1][esel].ref.flags |= E500_TLB_BITMAP; | 507 | vcpu_e500->gtlb_priv[1][esel].flags |= E500_TLB_BITMAP; |
| 511 | vcpu_e500->g2h_tlb1_map[esel] |= (u64)1 << sesel; | 508 | vcpu_e500->g2h_tlb1_map[esel] |= (u64)1 << sesel; |
| 512 | vcpu_e500->h2g_tlb1_rmap[sesel] = esel + 1; | 509 | vcpu_e500->h2g_tlb1_rmap[sesel] = esel + 1; |
| 513 | WARN_ON(!(ref->flags & E500_TLB_VALID)); | 510 | WARN_ON(!(tlbe->flags & E500_TLB_VALID)); |
| 514 | 511 | ||
| 515 | return sesel; | 512 | return sesel; |
| 516 | } | 513 | } |
| @@ -522,24 +519,24 @@ static int kvmppc_e500_tlb1_map(struct kvmppc_vcpu_e500 *vcpu_e500, · 5 unmodified lines | |||
| 522 | u64 gvaddr, gfn_t gfn, struct kvm_book3e_206_tlb_entry *gtlbe, | 519 | u64 gvaddr, gfn_t gfn, struct kvm_book3e_206_tlb_entry *gtlbe, |
| 523 | struct kvm_book3e_206_tlb_entry *stlbe, int esel) | 520 | struct kvm_book3e_206_tlb_entry *stlbe, int esel) |
| 524 | { | 521 | { |
| 525 | struct tlbe_ref *ref = &vcpu_e500->gtlb_priv[1][esel].ref; | 522 | struct tlbe_priv *tlbe = &vcpu_e500->gtlb_priv[1][esel]; |
| 526 | int sesel; | 523 | int sesel; |
| 527 | int r; | 524 | int r; |
| 528 | 525 | ||
| 529 | r = kvmppc_e500_shadow_map(vcpu_e500, gvaddr, gfn, gtlbe, 1, stlbe, | 526 | r = kvmppc_e500_shadow_map(vcpu_e500, gvaddr, gfn, gtlbe, 1, stlbe, |
| 530 | ref); | 527 | tlbe); |
| 531 | if (r) | 528 | if (r) |
| 532 | return r; | 529 | return r; |
| 533 | 530 | ||
| 534 | /* Use TLB0 when we can only map a page with 4k */ | 531 | /* Use TLB0 when we can only map a page with 4k */ |
| 535 | if (get_tlb_tsize(stlbe) == BOOK3E_PAGESZ_4K) { | 532 | if (get_tlb_tsize(stlbe) == BOOK3E_PAGESZ_4K) { |
| 536 | vcpu_e500->gtlb_priv[1][esel].ref.flags |= E500_TLB_TLB0; | 533 | vcpu_e500->gtlb_priv[1][esel].flags |= E500_TLB_TLB0; |
| 537 | write_stlbe(vcpu_e500, gtlbe, stlbe, 0, 0); | 534 | write_stlbe(vcpu_e500, gtlbe, stlbe, 0, 0); |
| 538 | return 0; | 535 | return 0; |
| 539 | } | 536 | } |
| 540 | 537 | ||
| 541 | /* Otherwise map into TLB1 */ | 538 | /* Otherwise map into TLB1 */ |
| 542 | sesel = kvmppc_e500_tlb1_map_tlb1(vcpu_e500, ref, esel); | 539 | sesel = kvmppc_e500_tlb1_map_tlb1(vcpu_e500, tlbe, esel); |
| 543 | write_stlbe(vcpu_e500, gtlbe, stlbe, 1, sesel); | 540 | write_stlbe(vcpu_e500, gtlbe, stlbe, 1, sesel); |
| 544 | 541 | ||
| 545 | return 0; | 542 | return 0; |
| @@ -561,11 +558,11 @@ void kvmppc_mmu_map(struct kvm_vcpu *vcpu, u64 eaddr, gpa_t gpaddr, · 15 unmodified lines | |||
| 561 | priv = &vcpu_e500->gtlb_priv[tlbsel][esel]; | 558 | priv = &vcpu_e500->gtlb_priv[tlbsel][esel]; |
| 562 | 559 | ||
| 563 | /* Triggers after clear_tlb_privs or on initial mapping */ | 560 | /* Triggers after clear_tlb_privs or on initial mapping */ |
| 564 | if (!(priv->ref.flags & E500_TLB_VALID)) { | 561 | if (!(priv->flags & E500_TLB_VALID)) { |
| 565 | kvmppc_e500_tlb0_map(vcpu_e500, esel, &stlbe); | 562 | kvmppc_e500_tlb0_map(vcpu_e500, esel, &stlbe); |
| 566 | } else { | 563 | } else { |
| 567 | kvmppc_e500_setup_stlbe(vcpu, gtlbe, BOOK3E_PAGESZ_4K, | 564 | kvmppc_e500_setup_stlbe(vcpu, gtlbe, BOOK3E_PAGESZ_4K, |
| 568 | &priv->ref, eaddr, &stlbe); | 565 | priv, eaddr, &stlbe); |
| 569 | write_stlbe(vcpu_e500, gtlbe, &stlbe, 0, 0); | 566 | write_stlbe(vcpu_e500, gtlbe, &stlbe, 0, 0); |
| 570 | } | 567 | } |
| 571 | break; | 568 | break; |
| @@ -562,3 +562,4 @@ exc; std r10,32(3) · 561 unmodified lines | |||
| 562 | li r5,4096 | 562 | li r5,4096 |
| 563 | b .Ldst_aligned | 563 | b .Ldst_aligned |
| 564 | EXPORT_SYMBOL(__copy_tofrom_user) | 564 | EXPORT_SYMBOL(__copy_tofrom_user) |
| 565 | EXPORT_SYMBOL(__copy_tofrom_user_base) | ||
| @@ -5,13 +5,9 @@ · 4 unmodified lines | |||
| 5 | * | 5 | * |
| 6 | * Author: Anton Blanchard <anton@au.ibm.com> | 6 | * Author: Anton Blanchard <anton@au.ibm.com> |
| 7 | */ | 7 | */ |
| 8 | #include <linux/export.h> | ||
| 8 | #include <asm/ppc_asm.h> | 9 | #include <asm/ppc_asm.h> |
| 9 | 10 | ||
| 10 | #ifndef SELFTEST_CASE | ||
| 11 | /* 0 == don't use VMX, 1 == use VMX */ | ||
| 12 | #define SELFTEST_CASE 0 | ||
| 13 | #endif | ||
| 14 | |||
| 15 | #ifdef __BIG_ENDIAN__ | 11 | #ifdef __BIG_ENDIAN__ |
| 16 | #define LVS(VRT,RA,RB) lvsl VRT,RA,RB | 12 | #define LVS(VRT,RA,RB) lvsl VRT,RA,RB |
| 17 | #define VPERM(VRT,VRA,VRB,VRC) vperm VRT,VRA,VRB,VRC | 13 | #define VPERM(VRT,VRA,VRB,VRC) vperm VRT,VRA,VRB,VRC |
| @@ -47,10 +43,14 @@ · 29 unmodified lines | |||
| 47 | ld r15,STK_REG(R15)(r1) | 43 | ld r15,STK_REG(R15)(r1) |
| 48 | ld r14,STK_REG(R14)(r1) | 44 | ld r14,STK_REG(R14)(r1) |
| 49 | .Ldo_err3: | 45 | .Ldo_err3: |
| 50 | bl CFUNC(exit_vmx_usercopy) | 46 | ld r6,STK_REG(R31)(r1) /* original destination pointer */ |
| 47 | ld r5,STK_REG(R29)(r1) /* original number of bytes */ | ||
| 48 | subf r7,r6,r3 /* #bytes copied */ | ||
| 49 | subf r3,r7,r5 /* #bytes not copied in r3 */ | ||
| 51 | ld r0,STACKFRAMESIZE+16(r1) | 50 | ld r0,STACKFRAMESIZE+16(r1) |
| 52 | mtlr r0 | 51 | mtlr r0 |
| 53 | b .Lexit | 52 | addi r1,r1,STACKFRAMESIZE |
| 53 | blr | ||
| 54 | #endif /* CONFIG_ALTIVEC */ | 54 | #endif /* CONFIG_ALTIVEC */ |
| 55 | 55 | ||
| 56 | .Ldo_err2: | 56 | .Ldo_err2: |
| @@ -74,7 +74,6 @@ · 17 unmodified lines | |||
| 74 | 74 | ||
| 75 | _GLOBAL(__copy_tofrom_user_power7) | 75 | _GLOBAL(__copy_tofrom_user_power7) |
| 76 | cmpldi r5,16 | 76 | cmpldi r5,16 |
| 77 | cmpldi cr1,r5,3328 | ||
| 78 | 77 | ||
| 79 | std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) | 78 | std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) |
| 80 | std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) | 79 | std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) |
| @@ -82,12 +81,6 @@ _GLOBAL(__copy_tofrom_user_power7) · 1 unmodified line | |||
| 82 | 81 | ||
| 83 | blt .Lshort_copy | 82 | blt .Lshort_copy |
| 84 | 83 | ||
| 85 | #ifdef CONFIG_ALTIVEC | ||
| 86 | test_feature = SELFTEST_CASE | ||
| 87 | BEGIN_FTR_SECTION | ||
| 88 | bgt cr1,.Lvmx_copy | ||
| 89 | END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) | ||
| 90 | #endif | ||
| 91 | 84 | ||
| 92 | .Lnonvmx_copy: | 85 | .Lnonvmx_copy: |
| 93 | /* Get the source 8B aligned */ | 86 | /* Get the source 8B aligned */ |
| @@ -263,23 +256,14 @@ err1; stb r0,0(r3) · 169 unmodified lines | |||
| 263 | 15: li r3,0 | 256 | 15: li r3,0 |
| 264 | blr | 257 | blr |
| 265 | 258 | ||
| 266 | .Lunwind_stack_nonvmx_copy: | ||
| 267 | addi r1,r1,STACKFRAMESIZE | ||
| 268 | b .Lnonvmx_copy | ||
| 269 | |||
| 270 | .Lvmx_copy: | ||
| 271 | #ifdef CONFIG_ALTIVEC | 259 | #ifdef CONFIG_ALTIVEC |
| 260 | _GLOBAL(__copy_tofrom_user_power7_vmx) | ||
| 272 | mflr r0 | 261 | mflr r0 |
| 273 | std r0,16(r1) | 262 | std r0,16(r1) |
| 274 | stdu r1,-STACKFRAMESIZE(r1) | 263 | stdu r1,-STACKFRAMESIZE(r1) |
| 275 | bl CFUNC(enter_vmx_usercopy) | ||
| 276 | cmpwi cr1,r3,0 | ||
| 277 | ld r0,STACKFRAMESIZE+16(r1) | ||
| 278 | ld r3,STK_REG(R31)(r1) | ||
| 279 | ld r4,STK_REG(R30)(r1) | ||
| 280 | ld r5,STK_REG(R29)(r1) | ||
| 281 | mtlr r0 | ||
| 282 | 264 | ||
| 265 | std r3,STK_REG(R31)(r1) | ||
| 266 | std r5,STK_REG(R29)(r1) | ||
| 283 | /* | 267 | /* |
| 284 | * We prefetch both the source and destination using enhanced touch | 268 | * We prefetch both the source and destination using enhanced touch |
| 285 | * instructions. We use a stream ID of 0 for the load side and | 269 | * instructions. We use a stream ID of 0 for the load side and |
| @@ -300,8 +284,6 @@ err1; stb r0,0(r3) · 14 unmodified lines | |||
| 300 | 284 | ||
| 301 | DCBT_SETUP_STREAMS(r6, r7, r9, r10, r8) | 285 | DCBT_SETUP_STREAMS(r6, r7, r9, r10, r8) |
| 302 | 286 | ||
| 303 | beq cr1,.Lunwind_stack_nonvmx_copy | ||
| 304 | |||
| 305 | /* | 287 | /* |
| 306 | * If source and destination are not relatively aligned we use a | 288 | * If source and destination are not relatively aligned we use a |
| 307 | * slower permute loop. | 289 | * slower permute loop. |
| @@ -478,7 +460,8 @@ err3; lbz r0,0(r4) · 170 unmodified lines | |||
| 478 | err3; stb r0,0(r3) | 460 | err3; stb r0,0(r3) |
| 479 | 461 | ||
| 480 | 15: addi r1,r1,STACKFRAMESIZE | 462 | 15: addi r1,r1,STACKFRAMESIZE |
| 481 | b CFUNC(exit_vmx_usercopy) /* tail call optimise */ | 463 | li r3,0 |
| 464 | blr | ||
| 482 | 465 | ||
| 483 | .Lvmx_unaligned_copy: | 466 | .Lvmx_unaligned_copy: |
| 484 | /* Get the destination 16B aligned */ | 467 | /* Get the destination 16B aligned */ |
| @@ -681,5 +664,7 @@ err3; lbz r0,0(r4) · 196 unmodified lines | |||
| 681 | err3; stb r0,0(r3) | 664 | err3; stb r0,0(r3) |
| 682 | 665 | ||
| 683 | 15: addi r1,r1,STACKFRAMESIZE | 666 | 15: addi r1,r1,STACKFRAMESIZE |
| 684 | b CFUNC(exit_vmx_usercopy) /* tail call optimise */ | 667 | li r3,0 |
| 668 | blr | ||
| 669 | EXPORT_SYMBOL(__copy_tofrom_user_power7_vmx) | ||
| 685 | #endif /* CONFIG_ALTIVEC */ | 670 | #endif /* CONFIG_ALTIVEC */ |
| @@ -27,6 +27,7 @@ int enter_vmx_usercopy(void) · 26 unmodified lines | |||
| 27 | 27 | ||
| 28 | return 1; | 28 | return 1; |
| 29 | } | 29 | } |
| 30 | EXPORT_SYMBOL(enter_vmx_usercopy); | ||
| 30 | 31 | ||
| 31 | /* | 32 | /* |
| 32 | * This function must return 0 because we tail call optimise when calling | 33 | * This function must return 0 because we tail call optimise when calling |
| @@ -49,6 +50,7 @@ int exit_vmx_usercopy(void) · 16 unmodified lines | |||
| 49 | set_dec(1); | 50 | set_dec(1); |
| 50 | return 0; | 51 | return 0; |
| 51 | } | 52 | } |
| 53 | EXPORT_SYMBOL(exit_vmx_usercopy); | ||
| 52 | 54 | ||
| 53 | int enter_vmx_ops(void) | 55 | int enter_vmx_ops(void) |
| 54 | { | 56 | { |
| @@ -30,6 +30,10 @@ · 29 unmodified lines | |||
| 30 | #include <asm/setup.h> | 30 | #include <asm/setup.h> |
| 31 | #include <asm/fixmap.h> | 31 | #include <asm/fixmap.h> |
| 32 | 32 | ||
| 33 | #include <asm/fadump.h> | ||
| 34 | #include <asm/kexec.h> | ||
| 35 | #include <asm/kvm_ppc.h> | ||
| 36 | |||
| 33 | #include <mm/mmu_decl.h> | 37 | #include <mm/mmu_decl.h> |
| 34 | 38 | ||
| 35 | unsigned long long memory_limit __initdata; | 39 | unsigned long long memory_limit __initdata; |
| @@ -268,6 +272,16 @@ void __init paging_init(void) · 232 unmodified lines | |||
| 268 | 272 | ||
| 269 | void __init arch_mm_preinit(void) | 273 | void __init arch_mm_preinit(void) |
| 270 | { | 274 | { |
| 275 | |||
| 276 | /* | ||
| 277 | * Reserve large chunks of memory for use by CMA for kdump, fadump, KVM | ||
| 278 | * and hugetlb. These must be called after pageblock_order is | ||
| 279 | * initialised. | ||
| 280 | */ | ||
| 281 | fadump_cma_init(); | ||
| 282 | kdump_cma_reserve(); | ||
| 283 | kvm_cma_reserve(); | ||
| 284 | |||
| 271 | /* | 285 | /* |
| 272 | * book3s is limited to 16 page sizes due to encoding this in | 286 | * book3s is limited to 16 page sizes due to encoding this in |
| 273 | * a 4-bit field for slices. | 287 | * a 4-bit field for slices. |
| @@ -81,9 +81,6 @@ · 80 unmodified lines | |||
| 81 | 81 | ||
| 82 | #ifdef CONFIG_PPC64 | 82 | #ifdef CONFIG_PPC64 |
| 83 | 83 | ||
| 84 | /* for gpr non volatile registers BPG_REG_6 to 10 */ | ||
| 85 | #define BPF_PPC_STACK_SAVE (6 * 8) | ||
| 86 | |||
| 87 | /* If dummy pass (!image), account for maximum possible instructions */ | 84 | /* If dummy pass (!image), account for maximum possible instructions */ |
| 88 | #define PPC_LI64(d, i) do { \ | 85 | #define PPC_LI64(d, i) do { \ |
| 89 | if (!image) \ | 86 | if (!image) \ |
| @@ -219,8 +216,6 @@ int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, · 129 unmodified lines | |||
| 219 | int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass, | 216 | int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass, |
| 220 | struct codegen_context *ctx, int insn_idx, | 217 | struct codegen_context *ctx, int insn_idx, |
| 221 | int jmp_off, int dst_reg, u32 code); | 218 | int jmp_off, int dst_reg, u32 code); |
| 222 | |||
| 223 | int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx); | ||
| 224 | #endif | 219 | #endif |
| 225 | 220 | ||
| 226 | #endif | 221 | #endif |
| @@ -450,7 +450,7 @@ bool bpf_jit_supports_subprog_tailcalls(void) · 449 unmodified lines | |||
| 450 | 450 | ||
| 451 | bool bpf_jit_supports_kfunc_call(void) | 451 | bool bpf_jit_supports_kfunc_call(void) |
| 452 | { | 452 | { |
| 453 | return true; | 453 | return IS_ENABLED(CONFIG_PPC64); |
| 454 | } | 454 | } |
| 455 | 455 | ||
| 456 | bool bpf_jit_supports_arena(void) | 456 | bool bpf_jit_supports_arena(void) |
| @@ -638,19 +638,12 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · 181 unmodified lines | |||
| 638 | * for the traced function (BPF subprog/callee) to fetch it. | 638 | * for the traced function (BPF subprog/callee) to fetch it. |
| 639 | */ | 639 | */ |
| 640 | static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx, | 640 | static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx, |
| 641 | int func_frame_offset, | 641 | int bpf_frame_size, int r4_off) |
| 642 | int bpf_dummy_frame_size, int r4_off) | ||
| 643 | { | 642 | { |
| 644 | if (IS_ENABLED(CONFIG_PPC64)) { | 643 | if (IS_ENABLED(CONFIG_PPC64)) { |
| 645 | /* See Generated stack layout */ | 644 | EMIT(PPC_RAW_LD(_R4, _R1, bpf_frame_size)); |
| 646 | int tailcallinfo_offset = BPF_PPC_TAILCALL; | 645 | /* Refer to trampoline's Generated stack layout */ |
| 647 | 646 | EMIT(PPC_RAW_LD(_R3, _R4, -BPF_PPC_TAILCALL)); | |
| 648 | /* | ||
| 649 | * func_frame_offset = ...(1) | ||
| 650 | * bpf_dummy_frame_size + trampoline_frame_size | ||
| 651 | */ | ||
| 652 | EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset)); | ||
| 653 | EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset)); | ||
| 654 | 647 | ||
| 655 | /* | 648 | /* |
| 656 | * Setting the tail_call_info in trampoline's frame | 649 | * Setting the tail_call_info in trampoline's frame |
| @@ -658,22 +651,14 @@ static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_conte · 1 unmodified line | |||
| 658 | */ | 651 | */ |
| 659 | EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT)); | 652 | EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT)); |
| 660 | PPC_BCC_CONST_SHORT(COND_GT, 8); | 653 | PPC_BCC_CONST_SHORT(COND_GT, 8); |
| 661 | EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx))); | 654 | EMIT(PPC_RAW_ADDI(_R3, _R4, -BPF_PPC_TAILCALL)); |
| 655 | |||
| 662 | /* | 656 | /* |
| 663 | * From ...(1) above: | 657 | * Trampoline's tail_call_info is at the same offset, as that of |
| 664 | * trampoline_frame_bottom = ...(2) | 658 | * any bpf program, with reference to previous frame. Update the |
| 665 | * func_frame_offset - bpf_dummy_frame_size | 659 | * address of main's tail_call_info in trampoline frame. |
| 666 | * | ||
| 667 | * Using ...(2) derived above: | ||
| 668 | * trampoline_tail_call_info_offset = ...(3) | ||
| 669 | * trampoline_frame_bottom - tailcallinfo_offset | ||
| 670 | * | ||
| 671 | * From ...(3): | ||
| 672 | * Use trampoline_tail_call_info_offset to write reference of main's | ||
| 673 | * tail_call_info in trampoline frame. | ||
| 674 | */ | 660 | */ |
| 675 | EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size) | 661 | EMIT(PPC_RAW_STL(_R3, _R1, bpf_frame_size - BPF_PPC_TAILCALL)); |
| 676 | - tailcallinfo_offset)); | ||
| 677 | } else { | 662 | } else { |
| 678 | /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ | 663 | /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */ |
| 679 | EMIT(PPC_RAW_LL(_R4, _R1, r4_off)); | 664 | EMIT(PPC_RAW_LL(_R4, _R1, r4_off)); |
| @@ -681,7 +666,7 @@ static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_conte · 1 unmodified line | |||
| 681 | } | 666 | } |
| 682 | 667 | ||
| 683 | static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx, | 668 | static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx, |
| 684 | int func_frame_offset, int r4_off) | 669 | int bpf_frame_size, int r4_off) |
| 685 | { | 670 | { |
| 686 | if (IS_ENABLED(CONFIG_PPC32)) { | 671 | if (IS_ENABLED(CONFIG_PPC32)) { |
| 687 | /* | 672 | /* |
| @@ -692,12 +677,12 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont · 4 unmodified lines | |||
| 692 | } | 677 | } |
| 693 | } | 678 | } |
| 694 | 679 | ||
| 695 | static void bpf_trampoline_save_args(u32 *image, struct codegen_context *ctx, int func_frame_offset, | 680 | static void bpf_trampoline_save_args(u32 *image, struct codegen_context *ctx, |
| 696 | int nr_regs, int regs_off) | 681 | int bpf_frame_size, int nr_regs, int regs_off) |
| 697 | { | 682 | { |
| 698 | int param_save_area_offset; | 683 | int param_save_area_offset; |
| 699 | 684 | ||
| 700 | param_save_area_offset = func_frame_offset; /* the two frames we alloted */ | 685 | param_save_area_offset = bpf_frame_size; |
| 701 | param_save_area_offset += STACK_FRAME_MIN_SIZE; /* param save area is past frame header */ | 686 | param_save_area_offset += STACK_FRAME_MIN_SIZE; /* param save area is past frame header */ |
| 702 | 687 | ||
| 703 | for (int i = 0; i < nr_regs; i++) { | 688 | for (int i = 0; i < nr_regs; i++) { |
| @@ -720,11 +705,11 @@ static void bpf_trampoline_restore_args_regs(u32 *image, struct codegen_context · 16 unmodified lines | |||
| 720 | 705 | ||
| 721 | /* Used when we call into the traced function. Replicate parameter save area */ | 706 | /* Used when we call into the traced function. Replicate parameter save area */ |
| 722 | static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context *ctx, | 707 | static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context *ctx, |
| 723 | int func_frame_offset, int nr_regs, int regs_off) | 708 | int bpf_frame_size, int nr_regs, int regs_off) |
| 724 | { | 709 | { |
| 725 | int param_save_area_offset; | 710 | int param_save_area_offset; |
| 726 | 711 | ||
| 727 | param_save_area_offset = func_frame_offset; /* the two frames we alloted */ | 712 | param_save_area_offset = bpf_frame_size; |
| 728 | param_save_area_offset += STACK_FRAME_MIN_SIZE; /* param save area is past frame header */ | 713 | param_save_area_offset += STACK_FRAME_MIN_SIZE; /* param save area is past frame header */ |
| 729 | 714 | ||
| 730 | for (int i = 8; i < nr_regs; i++) { | 715 | for (int i = 8; i < nr_regs; i++) { |
| @@ -741,10 +726,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 10 unmodified lines | |||
| 741 | void *func_addr) | 726 | void *func_addr) |
| 742 | { | 727 | { |
| 743 | int regs_off, nregs_off, ip_off, run_ctx_off, retval_off, nvr_off, alt_lr_off, r4_off = 0; | 728 | int regs_off, nregs_off, ip_off, run_ctx_off, retval_off, nvr_off, alt_lr_off, r4_off = 0; |
| 744 | int i, ret, nr_regs, bpf_frame_size = 0, bpf_dummy_frame_size = 0, func_frame_offset; | ||
| 745 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 729 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; |
| 746 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 730 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; |
| 747 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 731 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; |
| 732 | int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0; | ||
| 748 | struct codegen_context codegen_ctx, *ctx; | 733 | struct codegen_context codegen_ctx, *ctx; |
| 749 | u32 *image = (u32 *)rw_image; | 734 | u32 *image = (u32 *)rw_image; |
| 750 | ppc_inst_t branch_insn; | 735 | ppc_inst_t branch_insn; |
| @@ -770,24 +755,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 19 unmodified lines | |||
| 770 | * Generated stack layout: | 755 | * Generated stack layout: |
| 771 | * | 756 | * |
| 772 | * func prev back chain [ back chain ] | 757 | * func prev back chain [ back chain ] |
| 773 | * [ ] | ||
| 774 | * bpf prog redzone/tailcallcnt [ ... ] 64 bytes (64-bit powerpc) | ||
| 775 | * [ ] -- | ||
| 776 | * LR save area [ r0 save (64-bit) ] | header | ||
| 777 | * [ r0 save (32-bit) ] | | ||
| 778 | * dummy frame for unwind [ back chain 1 ] -- | ||
| 779 | * [ tail_call_info ] optional - 64-bit powerpc | 758 | * [ tail_call_info ] optional - 64-bit powerpc |
| 780 | * [ padding ] align stack frame | 759 | * [ padding ] align stack frame |
| 781 | * r4_off [ r4 (tailcallcnt) ] optional - 32-bit powerpc | 760 | * r4_off [ r4 (tailcallcnt) ] optional - 32-bit powerpc |
| 782 | * alt_lr_off [ real lr (ool stub)] optional - actual lr | 761 | * alt_lr_off [ real lr (ool stub)] optional - actual lr |
| 762 | * retaddr_off [ return address ] | ||
| 783 | * [ r26 ] | 763 | * [ r26 ] |
| 784 | * nvr_off [ r25 ] nvr save area | 764 | * nvr_off [ r25 ] nvr save area |
| 785 | * retval_off [ return value ] | 765 | * retval_off [ return value ] |
| 786 | * [ reg argN ] | 766 | * [ reg argN ] |
| 787 | * [ ... ] | 767 | * [ ... ] |
| 788 | * regs_off [ reg_arg1 ] prog ctx context | 768 | * regs_off [ reg_arg1 ] prog_ctx |
| 789 | * nregs_off [ args count ] | 769 | * nregs_off [ args count ] ((u64 *)prog_ctx)[-1] |
| 790 | * ip_off [ traced function ] | 770 | * ip_off [ traced function ] ((u64 *)prog_ctx)[-2] |
| 791 | * [ ... ] | 771 | * [ ... ] |
| 792 | * run_ctx_off [ bpf_tramp_run_ctx ] | 772 | * run_ctx_off [ bpf_tramp_run_ctx ] |
| 793 | * [ reg argN ] | 773 | * [ reg argN ] |
| @@ -843,6 +823,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 49 unmodified lines | |||
| 843 | nvr_off = bpf_frame_size; | 823 | nvr_off = bpf_frame_size; |
| 844 | bpf_frame_size += 2 * SZL; | 824 | bpf_frame_size += 2 * SZL; |
| 845 | 825 | ||
| 826 | /* Save area for return address */ | ||
| 827 | retaddr_off = bpf_frame_size; | ||
| 828 | bpf_frame_size += SZL; | ||
| 829 | |||
| 846 | /* Optional save area for actual LR in case of ool ftrace */ | 830 | /* Optional save area for actual LR in case of ool ftrace */ |
| 847 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { | 831 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { |
| 848 | alt_lr_off = bpf_frame_size; | 832 | alt_lr_off = bpf_frame_size; |
| @@ -869,16 +853,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 20 unmodified lines | |||
| 869 | /* Padding to align stack frame, if any */ | 853 | /* Padding to align stack frame, if any */ |
| 870 | bpf_frame_size = round_up(bpf_frame_size, SZL * 2); | 854 | bpf_frame_size = round_up(bpf_frame_size, SZL * 2); |
| 871 | 855 | ||
| 872 | /* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */ | 856 | /* Store original return value */ |
| 873 | bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64; | ||
| 874 | |||
| 875 | /* Offset to the traced function's stack frame */ | ||
| 876 | func_frame_offset = bpf_dummy_frame_size + bpf_frame_size; | ||
| 877 | |||
| 878 | /* Create dummy frame for unwind, store original return value */ | ||
| 879 | EMIT(PPC_RAW_STL(_R0, _R1, PPC_LR_STKOFF)); | 857 | EMIT(PPC_RAW_STL(_R0, _R1, PPC_LR_STKOFF)); |
| 880 | /* Protect red zone where tail call count goes */ | ||
| 881 | EMIT(PPC_RAW_STLU(_R1, _R1, -bpf_dummy_frame_size)); | ||
| 882 | 858 | ||
| 883 | /* Create our stack frame */ | 859 | /* Create our stack frame */ |
| 884 | EMIT(PPC_RAW_STLU(_R1, _R1, -bpf_frame_size)); | 860 | EMIT(PPC_RAW_STLU(_R1, _R1, -bpf_frame_size)); |
| @@ -893,34 +869,44 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 8 unmodified lines | |||
| 893 | if (IS_ENABLED(CONFIG_PPC32) && nr_regs < 2) | 869 | if (IS_ENABLED(CONFIG_PPC32) && nr_regs < 2) |
| 894 | EMIT(PPC_RAW_STL(_R4, _R1, r4_off)); | 870 | EMIT(PPC_RAW_STL(_R4, _R1, r4_off)); |
| 895 | 871 | ||
| 896 | bpf_trampoline_save_args(image, ctx, func_frame_offset, nr_regs, regs_off); | 872 | bpf_trampoline_save_args(image, ctx, bpf_frame_size, nr_regs, regs_off); |
| 897 | 873 | ||
| 898 | /* Save our return address */ | 874 | /* Save our LR/return address */ |
| 899 | EMIT(PPC_RAW_MFLR(_R3)); | 875 | EMIT(PPC_RAW_MFLR(_R3)); |
| 900 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) | 876 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) |
| 901 | EMIT(PPC_RAW_STL(_R3, _R1, alt_lr_off)); | 877 | EMIT(PPC_RAW_STL(_R3, _R1, alt_lr_off)); |
| 902 | else | 878 | else |
| 903 | EMIT(PPC_RAW_STL(_R3, _R1, bpf_frame_size + PPC_LR_STKOFF)); | 879 | EMIT(PPC_RAW_STL(_R3, _R1, retaddr_off)); |
| 904 | 880 | ||
| 905 | /* | 881 | /* |
| 906 | * Save ip address of the traced function. | 882 | * Derive IP address of the traced function. |
| 907 | * We could recover this from LR, but we will need to address for OOL trampoline, | 883 | * In case of CONFIG_PPC_FTRACE_OUT_OF_LINE or BPF program, LR points to the instruction |
| 908 | * and optional GEP area. | 884 | * after the 'bl' instruction in the OOL stub. Refer to ftrace_init_ool_stub() and |
| 885 | * bpf_arch_text_poke() for OOL stub of kernel functions and bpf programs respectively. | ||
| 886 | * Relevant stub sequence: | ||
| 887 | * | ||
| 888 | * bl <tramp> | ||
| 889 | * LR (R3) => mtlr r0 | ||
| 890 | * b <func_addr+4> | ||
| 891 | * | ||
| 892 | * Recover kernel function/bpf program address from the unconditional | ||
| 893 | * branch instruction at the end of OOL stub. | ||
| 909 | */ | 894 | */ |
| 910 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE) || flags & BPF_TRAMP_F_IP_ARG) { | 895 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE) || flags & BPF_TRAMP_F_IP_ARG) { |
| 911 | EMIT(PPC_RAW_LWZ(_R4, _R3, 4)); | 896 | EMIT(PPC_RAW_LWZ(_R4, _R3, 4)); |
| 912 | EMIT(PPC_RAW_SLWI(_R4, _R4, 6)); | 897 | EMIT(PPC_RAW_SLWI(_R4, _R4, 6)); |
| 913 | EMIT(PPC_RAW_SRAWI(_R4, _R4, 6)); | 898 | EMIT(PPC_RAW_SRAWI(_R4, _R4, 6)); |
| 914 | EMIT(PPC_RAW_ADD(_R3, _R3, _R4)); | 899 | EMIT(PPC_RAW_ADD(_R3, _R3, _R4)); |
| 915 | EMIT(PPC_RAW_ADDI(_R3, _R3, 4)); | ||
| 916 | } | 900 | } |
| 917 | 901 | ||
| 918 | if (flags & BPF_TRAMP_F_IP_ARG) | 902 | if (flags & BPF_TRAMP_F_IP_ARG) |
| 919 | EMIT(PPC_RAW_STL(_R3, _R1, ip_off)); | 903 | EMIT(PPC_RAW_STL(_R3, _R1, ip_off)); |
| 920 | 904 | ||
| 921 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) | 905 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { |
| 922 | /* Fake our LR for unwind */ | 906 | /* Fake our LR for BPF_TRAMP_F_CALL_ORIG case */ |
| 923 | EMIT(PPC_RAW_STL(_R3, _R1, bpf_frame_size + PPC_LR_STKOFF)); | 907 | EMIT(PPC_RAW_ADDI(_R3, _R3, 4)); |
| 908 | EMIT(PPC_RAW_STL(_R3, _R1, retaddr_off)); | ||
| 909 | } | ||
| 924 | 910 | ||
| 925 | /* Save function arg count -- see bpf_get_func_arg_cnt() */ | 911 | /* Save function arg count -- see bpf_get_func_arg_cnt() */ |
| 926 | EMIT(PPC_RAW_LI(_R3, nr_regs)); | 912 | EMIT(PPC_RAW_LI(_R3, nr_regs)); |
| @@ -958,20 +944,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 31 unmodified lines | |||
| 958 | /* Call the traced function */ | 944 | /* Call the traced function */ |
| 959 | if (flags & BPF_TRAMP_F_CALL_ORIG) { | 945 | if (flags & BPF_TRAMP_F_CALL_ORIG) { |
| 960 | /* | 946 | /* |
| 961 | * The address in LR save area points to the correct point in the original function | 947 | * retaddr on trampoline stack points to the correct point in the original function |
| 962 | * with both PPC_FTRACE_OUT_OF_LINE as well as with traditional ftrace instruction | 948 | * with both PPC_FTRACE_OUT_OF_LINE as well as with traditional ftrace instruction |
| 963 | * sequence | 949 | * sequence |
| 964 | */ | 950 | */ |
| 965 | EMIT(PPC_RAW_LL(_R3, _R1, bpf_frame_size + PPC_LR_STKOFF)); | 951 | EMIT(PPC_RAW_LL(_R3, _R1, retaddr_off)); |
| 966 | EMIT(PPC_RAW_MTCTR(_R3)); | 952 | EMIT(PPC_RAW_MTCTR(_R3)); |
| 967 | 953 | ||
| 968 | /* Replicate tail_call_cnt before calling the original BPF prog */ | 954 | /* Replicate tail_call_cnt before calling the original BPF prog */ |
| 969 | if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) | 955 | if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) |
| 970 | bpf_trampoline_setup_tail_call_info(image, ctx, func_frame_offset, | 956 | bpf_trampoline_setup_tail_call_info(image, ctx, bpf_frame_size, r4_off); |
| 971 | bpf_dummy_frame_size, r4_off); | ||
| 972 | 957 | ||
| 973 | /* Restore args */ | 958 | /* Restore args */ |
| 974 | bpf_trampoline_restore_args_stack(image, ctx, func_frame_offset, nr_regs, regs_off); | 959 | bpf_trampoline_restore_args_stack(image, ctx, bpf_frame_size, nr_regs, regs_off); |
| 975 | 960 | ||
| 976 | /* Restore TOC for 64-bit */ | 961 | /* Restore TOC for 64-bit */ |
| 977 | if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) && !IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)) | 962 | if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) && !IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)) |
| @@ -985,7 +970,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 7 unmodified lines | |||
| 985 | 970 | ||
| 986 | /* Restore updated tail_call_cnt */ | 971 | /* Restore updated tail_call_cnt */ |
| 987 | if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) | 972 | if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) |
| 988 | bpf_trampoline_restore_tail_call_cnt(image, ctx, func_frame_offset, r4_off); | 973 | bpf_trampoline_restore_tail_call_cnt(image, ctx, bpf_frame_size, r4_off); |
| 989 | 974 | ||
| 990 | /* Reserve space to patch branch instruction to skip fexit progs */ | 975 | /* Reserve space to patch branch instruction to skip fexit progs */ |
| 991 | if (ro_image) /* image is NULL for dummy pass */ | 976 | if (ro_image) /* image is NULL for dummy pass */ |
| @@ -1037,7 +1022,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 45 unmodified lines | |||
| 1037 | EMIT(PPC_RAW_LD(_R2, _R1, 24)); | 1022 | EMIT(PPC_RAW_LD(_R2, _R1, 24)); |
| 1038 | if (flags & BPF_TRAMP_F_SKIP_FRAME) { | 1023 | if (flags & BPF_TRAMP_F_SKIP_FRAME) { |
| 1039 | /* Skip the traced function and return to parent */ | 1024 | /* Skip the traced function and return to parent */ |
| 1040 | EMIT(PPC_RAW_ADDI(_R1, _R1, func_frame_offset)); | 1025 | EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_frame_size)); |
| 1041 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); | 1026 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); |
| 1042 | EMIT(PPC_RAW_MTLR(_R0)); | 1027 | EMIT(PPC_RAW_MTLR(_R0)); |
| 1043 | EMIT(PPC_RAW_BLR()); | 1028 | EMIT(PPC_RAW_BLR()); |
| @@ -1045,13 +1030,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line | |||
| 1045 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { | 1030 | if (IS_ENABLED(CONFIG_PPC_FTRACE_OUT_OF_LINE)) { |
| 1046 | EMIT(PPC_RAW_LL(_R0, _R1, alt_lr_off)); | 1031 | EMIT(PPC_RAW_LL(_R0, _R1, alt_lr_off)); |
| 1047 | EMIT(PPC_RAW_MTLR(_R0)); | 1032 | EMIT(PPC_RAW_MTLR(_R0)); |
| 1048 | EMIT(PPC_RAW_ADDI(_R1, _R1, func_frame_offset)); | 1033 | EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_frame_size)); |
| 1049 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); | 1034 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); |
| 1050 | EMIT(PPC_RAW_BLR()); | 1035 | EMIT(PPC_RAW_BLR()); |
| 1051 | } else { | 1036 | } else { |
| 1052 | EMIT(PPC_RAW_LL(_R0, _R1, bpf_frame_size + PPC_LR_STKOFF)); | 1037 | EMIT(PPC_RAW_LL(_R0, _R1, retaddr_off)); |
| 1053 | EMIT(PPC_RAW_MTCTR(_R0)); | 1038 | EMIT(PPC_RAW_MTCTR(_R0)); |
| 1054 | EMIT(PPC_RAW_ADDI(_R1, _R1, func_frame_offset)); | 1039 | EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_frame_size)); |
| 1055 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); | 1040 | EMIT(PPC_RAW_LL(_R0, _R1, PPC_LR_STKOFF)); |
| 1056 | EMIT(PPC_RAW_MTLR(_R0)); | 1041 | EMIT(PPC_RAW_MTLR(_R0)); |
| 1057 | EMIT(PPC_RAW_BCTR()); | 1042 | EMIT(PPC_RAW_BCTR()); |
| @@ -32,23 +32,27 @@ · 31 unmodified lines | |||
| 32 | * | 32 | * |
| 33 | * [ prev sp ] <------------- | 33 | * [ prev sp ] <------------- |
| 34 | * [ tail_call_info ] 8 | | 34 | * [ tail_call_info ] 8 | |
| 35 | * [ nv gpr save area ] 6*8 + (12*8) | | 35 | * [ nv gpr save area ] (6 * 8) | |
| 36 | * [ addl. nv gpr save area] (12 * 8) | <--- exception boundary/callback program | ||
| 36 | * [ local_tmp_var ] 24 | | 37 | * [ local_tmp_var ] 24 | |
| 37 | * fp (r31) --> [ ebpf stack space ] upto 512 | | 38 | * fp (r31) --> [ ebpf stack space ] upto 512 | |
| 38 | * [ frame header ] 32/112 | | 39 | * [ frame header ] 32/112 | |
| 39 | * sp (r1) ---> [ stack pointer ] -------------- | 40 | * sp (r1) ---> [ stack pointer ] -------------- |
| 40 | * | 41 | * |
| 41 | * Additional (12*8) in 'nv gpr save area' only in case of | 42 | * Additional (12 * 8) in 'nv gpr save area' only in case of |
| 42 | * exception boundary. | 43 | * exception boundary/callback. |
| 43 | */ | 44 | */ |
| 44 | 45 | ||
| 46 | /* BPF non-volatile registers save area size */ | ||
| 47 | #define BPF_PPC_STACK_SAVE (6 * 8) | ||
| 48 | |||
| 45 | /* for bpf JIT code internal usage */ | 49 | /* for bpf JIT code internal usage */ |
| 46 | #define BPF_PPC_STACK_LOCALS 24 | 50 | #define BPF_PPC_STACK_LOCALS 24 |
| 47 | /* | 51 | /* |
| 48 | * for additional non volatile registers(r14-r25) to be saved | 52 | * for additional non volatile registers(r14-r25) to be saved |
| 49 | * at exception boundary | 53 | * at exception boundary |
| 50 | */ | 54 | */ |
| 51 | #define BPF_PPC_EXC_STACK_SAVE (12*8) | 55 | #define BPF_PPC_EXC_STACK_SAVE (12 * 8) |
| 52 | 56 | ||
| 53 | /* stack frame excluding BPF stack, ensure this is quadword aligned */ | 57 | /* stack frame excluding BPF stack, ensure this is quadword aligned */ |
| 54 | #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ | 58 | #define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \ |
| @@ -125,12 +129,13 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) · 70 unmodified lines | |||
| 125 | * [ ... ] | | 129 | * [ ... ] | |
| 126 | * sp (r1) ---> [ stack pointer ] -------------- | 130 | * sp (r1) ---> [ stack pointer ] -------------- |
| 127 | * [ tail_call_info ] 8 | 131 | * [ tail_call_info ] 8 |
| 128 | * [ nv gpr save area ] 6*8 + (12*8) | 132 | * [ nv gpr save area ] (6 * 8) |
| 133 | * [ addl. nv gpr save area] (12 * 8) <--- exception boundary/callback program | ||
| 129 | * [ local_tmp_var ] 24 | 134 | * [ local_tmp_var ] 24 |
| 130 | * [ unused red zone ] 224 | 135 | * [ unused red zone ] 224 |
| 131 | * | 136 | * |
| 132 | * Additional (12*8) in 'nv gpr save area' only in case of | 137 | * Additional (12 * 8) in 'nv gpr save area' only in case of |
| 133 | * exception boundary. | 138 | * exception boundary/callback. |
| 134 | */ | 139 | */ |
| 135 | static int bpf_jit_stack_local(struct codegen_context *ctx) | 140 | static int bpf_jit_stack_local(struct codegen_context *ctx) |
| 136 | { | 141 | { |
| @@ -148,7 +153,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx) · 11 unmodified lines | |||
| 148 | } | 153 | } |
| 149 | } | 154 | } |
| 150 | 155 | ||
| 151 | int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx) | 156 | static int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx) |
| 152 | { | 157 | { |
| 153 | return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE; | 158 | return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE; |
| 154 | } | 159 | } |
| @@ -237,10 +242,6 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) · 82 unmodified lines | |||
| 237 | 242 | ||
| 238 | if (bpf_has_stack_frame(ctx) && !ctx->exception_cb) { | 243 | if (bpf_has_stack_frame(ctx) && !ctx->exception_cb) { |
| 239 | /* | 244 | /* |
| 240 | * exception_cb uses boundary frame after stack walk. | ||
| 241 | * It can simply use redzone, this optimization reduces | ||
| 242 | * stack walk loop by one level. | ||
| 243 | * | ||
| 244 | * We need a stack frame, but we don't necessarily need to | 245 | * We need a stack frame, but we don't necessarily need to |
| 245 | * save/restore LR unless we call other functions | 246 | * save/restore LR unless we call other functions |
| 246 | */ | 247 | */ |
| @@ -284,6 +285,22 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) · 37 unmodified lines | |||
| 284 | * program(main prog) as third arg | 285 | * program(main prog) as third arg |
| 285 | */ | 286 | */ |
| 286 | EMIT(PPC_RAW_MR(_R1, _R5)); | 287 | EMIT(PPC_RAW_MR(_R1, _R5)); |
| 288 | /* | ||
| 289 | * Exception callback reuses the stack frame of exception boundary. | ||
| 290 | * But BPF stack depth of exception callback and exception boundary | ||
| 291 | * don't have to be same. If BPF stack depth is different, adjust the | ||
| 292 | * stack frame size considering BPF stack depth of exception callback. | ||
| 293 | * The non-volatile register save area remains unchanged. These non- | ||
| 294 | * volatile registers are restored in exception callback's epilogue. | ||
| 295 | */ | ||
| 296 | EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R5, 0)); | ||
| 297 | EMIT(PPC_RAW_SUB(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_1), _R1)); | ||
| 298 | EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | ||
| 299 | -BPF_PPC_EXC_STACKFRAME)); | ||
| 300 | EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_2), ctx->stack_size)); | ||
| 301 | PPC_BCC_CONST_SHORT(COND_EQ, 12); | ||
| 302 | EMIT(PPC_RAW_MR(_R1, bpf_to_ppc(TMP_REG_1))); | ||
| 303 | EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_EXC_STACKFRAME + ctx->stack_size))); | ||
| 287 | } | 304 | } |
| 288 | 305 | ||
| 289 | /* | 306 | /* |
| @@ -482,6 +499,83 @@ int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context * · 192 unmodified lines | |||
| 482 | return 0; | 499 | return 0; |
| 483 | } | 500 | } |
| 484 | 501 | ||
| 502 | static int zero_extend(u32 *image, struct codegen_context *ctx, u32 src_reg, u32 dst_reg, u32 size) | ||
| 503 | { | ||
| 504 | switch (size) { | ||
| 505 | case 1: | ||
| 506 | /* zero-extend 8 bits into 64 bits */ | ||
| 507 | EMIT(PPC_RAW_RLDICL(dst_reg, src_reg, 0, 56)); | ||
| 508 | return 0; | ||
| 509 | case 2: | ||
| 510 | /* zero-extend 16 bits into 64 bits */ | ||
| 511 | EMIT(PPC_RAW_RLDICL(dst_reg, src_reg, 0, 48)); | ||
| 512 | return 0; | ||
| 513 | case 4: | ||
| 514 | /* zero-extend 32 bits into 64 bits */ | ||
| 515 | EMIT(PPC_RAW_RLDICL(dst_reg, src_reg, 0, 32)); | ||
| 516 | fallthrough; | ||
| 517 | case 8: | ||
| 518 | /* Nothing to do */ | ||
| 519 | return 0; | ||
| 520 | default: | ||
| 521 | return -1; | ||
| 522 | } | ||
| 523 | } | ||
| 524 | |||
| 525 | static int sign_extend(u32 *image, struct codegen_context *ctx, u32 src_reg, u32 dst_reg, u32 size) | ||
| 526 | { | ||
| 527 | switch (size) { | ||
| 528 | case 1: | ||
| 529 | /* sign-extend 8 bits into 64 bits */ | ||
| 530 | EMIT(PPC_RAW_EXTSB(dst_reg, src_reg)); | ||
| 531 | return 0; | ||
| 532 | case 2: | ||
| 533 | /* sign-extend 16 bits into 64 bits */ | ||
| 534 | EMIT(PPC_RAW_EXTSH(dst_reg, src_reg)); | ||
| 535 | return 0; | ||
| 536 | case 4: | ||
| 537 | /* sign-extend 32 bits into 64 bits */ | ||
| 538 | EMIT(PPC_RAW_EXTSW(dst_reg, src_reg)); | ||
| 539 | fallthrough; | ||
| 540 | case 8: | ||
| 541 | /* Nothing to do */ | ||
| 542 | return 0; | ||
| 543 | default: | ||
| 544 | return -1; | ||
| 545 | } | ||
| 546 | } | ||
| 547 | |||
| 548 | /* | ||
| 549 | * Handle powerpc ABI expectations from caller: | ||
| 550 | * - Unsigned arguments are zero-extended. | ||
| 551 | * - Signed arguments are sign-extended. | ||
| 552 | */ | ||
| 553 | static int prepare_for_kfunc_call(const struct bpf_prog *fp, u32 *image, | ||
| 554 | struct codegen_context *ctx, | ||
| 555 | const struct bpf_insn *insn) | ||
| 556 | { | ||
| 557 | const struct btf_func_model *m = bpf_jit_find_kfunc_model(fp, insn); | ||
| 558 | int i; | ||
| 559 | |||
| 560 | if (!m) | ||
| 561 | return -1; | ||
| 562 | |||
| 563 | for (i = 0; i < m->nr_args; i++) { | ||
| 564 | /* Note that BPF ABI only allows up to 5 args for kfuncs */ | ||
| 565 | u32 reg = bpf_to_ppc(BPF_REG_1 + i), size = m->arg_size[i]; | ||
| 566 | |||
| 567 | if (!(m->arg_flags[i] & BTF_FMODEL_SIGNED_ARG)) { | ||
| 568 | if (zero_extend(image, ctx, reg, reg, size)) | ||
| 569 | return -1; | ||
| 570 | } else { | ||
| 571 | if (sign_extend(image, ctx, reg, reg, size)) | ||
| 572 | return -1; | ||
| 573 | } | ||
| 574 | } | ||
| 575 | |||
| 576 | return 0; | ||
| 577 | } | ||
| 578 | |||
| 485 | static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 out) | 579 | static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 out) |
| 486 | { | 580 | { |
| 487 | /* | 581 | /* |
| @@ -522,9 +616,30 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o · 34 unmodified lines | |||
| 522 | 616 | ||
| 523 | /* | 617 | /* |
| 524 | * tail_call_info++; <- Actual value of tcc here | 618 | * tail_call_info++; <- Actual value of tcc here |
| 619 | * Writeback this updated value only if tailcall succeeds. | ||
| 525 | */ | 620 | */ |
| 526 | EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); | 621 | EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1)); |
| 527 | 622 | ||
| 623 | /* prog = array->ptrs[index]; */ | ||
| 624 | EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_2), b2p_index, 8)); | ||
| 625 | EMIT(PPC_RAW_ADD(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), b2p_bpf_array)); | ||
| 626 | EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | ||
| 627 | offsetof(struct bpf_array, ptrs))); | ||
| 628 | |||
| 629 | /* | ||
| 630 | * if (prog == NULL) | ||
| 631 | * goto out; | ||
| 632 | */ | ||
| 633 | EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_2), 0)); | ||
| 634 | PPC_BCC_SHORT(COND_EQ, out); | ||
| 635 | |||
| 636 | /* goto *(prog->bpf_func + prologue_size); */ | ||
| 637 | EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | ||
| 638 | offsetof(struct bpf_prog, bpf_func))); | ||
| 639 | EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), bpf_to_ppc(TMP_REG_2), | ||
| 640 | FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size)); | ||
| 641 | EMIT(PPC_RAW_MTCTR(bpf_to_ppc(TMP_REG_2))); | ||
| 642 | |||
| 528 | /* | 643 | /* |
| 529 | * Before writing updated tail_call_info, distinguish if current frame | 644 | * Before writing updated tail_call_info, distinguish if current frame |
| 530 | * is storing a reference to tail_call_info or actual tcc value in | 645 | * is storing a reference to tail_call_info or actual tcc value in |
| @@ -539,24 +654,6 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o · 8 unmodified lines | |||
| 539 | /* Writeback updated value to tail_call_info */ | 654 | /* Writeback updated value to tail_call_info */ |
| 540 | EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0)); | 655 | EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0)); |
| 541 | 656 | ||
| 542 | /* prog = array->ptrs[index]; */ | ||
| 543 | EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8)); | ||
| 544 | EMIT(PPC_RAW_ADD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), b2p_bpf_array)); | ||
| 545 | EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), offsetof(struct bpf_array, ptrs))); | ||
| 546 | |||
| 547 | /* | ||
| 548 | * if (prog == NULL) | ||
| 549 | * goto out; | ||
| 550 | */ | ||
| 551 | EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_1), 0)); | ||
| 552 | PPC_BCC_SHORT(COND_EQ, out); | ||
| 553 | |||
| 554 | /* goto *(prog->bpf_func + prologue_size); */ | ||
| 555 | EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), offsetof(struct bpf_prog, bpf_func))); | ||
| 556 | EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), | ||
| 557 | FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size)); | ||
| 558 | EMIT(PPC_RAW_MTCTR(bpf_to_ppc(TMP_REG_1))); | ||
| 559 | |||
| 560 | /* tear down stack, restore NVRs, ... */ | 657 | /* tear down stack, restore NVRs, ... */ |
| 561 | bpf_jit_emit_common_epilogue(image, ctx); | 658 | bpf_jit_emit_common_epilogue(image, ctx); |
| 562 | 659 | ||
| @@ -1123,14 +1220,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct code · 560 unmodified lines | |||
| 1123 | /* special mov32 for zext */ | 1220 | /* special mov32 for zext */ |
| 1124 | EMIT(PPC_RAW_RLWINM(dst_reg, dst_reg, 0, 0, 31)); | 1221 | EMIT(PPC_RAW_RLWINM(dst_reg, dst_reg, 0, 0, 31)); |
| 1125 | break; | 1222 | break; |
| 1126 | } else if (off == 8) { | 1223 | } |
| 1127 | EMIT(PPC_RAW_EXTSB(dst_reg, src_reg)); | 1224 | if (off == 0) { |
| 1128 | } else if (off == 16) { | 1225 | /* MOV */ |
| 1129 | EMIT(PPC_RAW_EXTSH(dst_reg, src_reg)); | 1226 | if (dst_reg != src_reg) |
| 1130 | } else if (off == 32) { | 1227 | EMIT(PPC_RAW_MR(dst_reg, src_reg)); |
| 1131 | EMIT(PPC_RAW_EXTSW(dst_reg, src_reg)); | 1228 | } else { |
| 1132 | } else if (dst_reg != src_reg) | 1229 | /* MOVSX: dst = (s8,s16,s32)src (off = 8,16,32) */ |
| 1133 | EMIT(PPC_RAW_MR(dst_reg, src_reg)); | 1230 | if (sign_extend(image, ctx, src_reg, dst_reg, off / 8)) |
| 1231 | return -1; | ||
| 1232 | } | ||
| 1134 | goto bpf_alu32_trunc; | 1233 | goto bpf_alu32_trunc; |
| 1135 | case BPF_ALU | BPF_MOV | BPF_K: /* (u32) dst = imm */ | 1234 | case BPF_ALU | BPF_MOV | BPF_K: /* (u32) dst = imm */ |
| 1136 | case BPF_ALU64 | BPF_MOV | BPF_K: /* dst = (s64) imm */ | 1235 | case BPF_ALU64 | BPF_MOV | BPF_K: /* dst = (s64) imm */ |
| @@ -1598,6 +1697,12 @@ emit_clear: · 461 unmodified lines | |||
| 1598 | if (ret < 0) | 1697 | if (ret < 0) |
| 1599 | return ret; | 1698 | return ret; |
| 1600 | 1699 | ||
| 1700 | /* Take care of powerpc ABI requirements before kfunc call */ | ||
| 1701 | if (insn[i].src_reg == BPF_PSEUDO_KFUNC_CALL) { | ||
| 1702 | if (prepare_for_kfunc_call(fp, image, ctx, &insn[i])) | ||
| 1703 | return -1; | ||
| 1704 | } | ||
| 1705 | |||
| 1601 | ret = bpf_jit_emit_func_call_rel(image, fimage, ctx, func_addr); | 1706 | ret = bpf_jit_emit_func_call_rel(image, fimage, ctx, func_addr); |
| 1602 | if (ret) | 1707 | if (ret) |
| 1603 | return ret; | 1708 | return ret; |
| @@ -103,6 +103,11 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re · 102 unmodified lines | |||
| 103 | void | 103 | void |
| 104 | perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) | 104 | perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs) |
| 105 | { | 105 | { |
| 106 | perf_callchain_store(entry, perf_arch_instruction_pointer(regs)); | ||
| 107 | |||
| 108 | if (!current->mm) | ||
| 109 | return; | ||
| 110 | |||
| 106 | if (!is_32bit_task()) | 111 | if (!is_32bit_task()) |
| 107 | perf_callchain_user_64(entry, regs); | 112 | perf_callchain_user_64(entry, regs); |
| 108 | else | 113 | else |
| @@ -142,7 +142,6 @@ void perf_callchain_user_32(struct perf_callchain_entry_ctx *entry, · 141 unmodified lines | |||
| 142 | next_ip = perf_arch_instruction_pointer(regs); | 142 | next_ip = perf_arch_instruction_pointer(regs); |
| 143 | lr = regs->link; | 143 | lr = regs->link; |
| 144 | sp = regs->gpr[1]; | 144 | sp = regs->gpr[1]; |
| 145 | perf_callchain_store(entry, next_ip); | ||
| 146 | 145 | ||
| 147 | while (entry->nr < entry->max_stack) { | 146 | while (entry->nr < entry->max_stack) { |
| 148 | fp = (unsigned int __user *) (unsigned long) sp; | 147 | fp = (unsigned int __user *) (unsigned long) sp; |
| @@ -77,7 +77,6 @@ void perf_callchain_user_64(struct perf_callchain_entry_ctx *entry, · 76 unmodified lines | |||
| 77 | next_ip = perf_arch_instruction_pointer(regs); | 77 | next_ip = perf_arch_instruction_pointer(regs); |
| 78 | lr = regs->link; | 78 | lr = regs->link; |
| 79 | sp = regs->gpr[1]; | 79 | sp = regs->gpr[1]; |
| 80 | perf_callchain_store(entry, next_ip); | ||
| 81 | 80 | ||
| 82 | while (entry->nr < entry->max_stack) { | 81 | while (entry->nr < entry->max_stack) { |
| 83 | fp = (unsigned long __user *) sp; | 82 | fp = (unsigned long __user *) sp; |
| @@ -155,8 +155,8 @@ machine_device_initcall(mpc83xx_km, mpc83xx_declare_of_platform_devices); · 154 unmodified lines | |||
| 155 | 155 | ||
| 156 | /* list of the supported boards */ | 156 | /* list of the supported boards */ |
| 157 | static char *board[] __initdata = { | 157 | static char *board[] __initdata = { |
| 158 | "Keymile,KMETER1", | 158 | "keymile,KMETER1", |
| 159 | "Keymile,kmpbec8321", | 159 | "keymile,kmpbec8321", |
| 160 | NULL | 160 | NULL |
| 161 | }; | 161 | }; |
| 162 | 162 | ||
| @@ -276,7 +276,7 @@ config PPC_BOOK3S · 275 unmodified lines | |||
| 276 | config PPC_E500 | 276 | config PPC_E500 |
| 277 | select FSL_EMB_PERFMON | 277 | select FSL_EMB_PERFMON |
| 278 | bool | 278 | bool |
| 279 | select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 | 279 | select ARCH_SUPPORTS_HUGETLBFS |
| 280 | select PPC_SMP_MUXED_IPI | 280 | select PPC_SMP_MUXED_IPI |
| 281 | select PPC_DOORBELL | 281 | select PPC_DOORBELL |
| 282 | select PPC_KUEP | 282 | select PPC_KUEP |
| @@ -337,7 +337,7 @@ config BOOKE · 54 unmodified lines | |||
| 337 | config PTE_64BIT | 337 | config PTE_64BIT |
| 338 | bool | 338 | bool |
| 339 | depends on 44x || PPC_E500 || PPC_86xx | 339 | depends on 44x || PPC_E500 || PPC_86xx |
| 340 | default y if PHYS_64BIT | 340 | default y if PPC_E500 || PHYS_64BIT |
| 341 | 341 | ||
| 342 | config PHYS_64BIT | 342 | config PHYS_64BIT |
| 343 | bool 'Large physical address support' if PPC_E500 || PPC_86xx | 343 | bool 'Large physical address support' if PPC_E500 || PPC_86xx |
| @@ -605,7 +605,7 @@ static int pseries_irq_domain_alloc(struct irq_domain *domain, unsigned int virq · 604 unmodified lines | |||
| 605 | &pseries_msi_irq_chip, pseries_dev); | 605 | &pseries_msi_irq_chip, pseries_dev); |
| 606 | } | 606 | } |
| 607 | 607 | ||
| 608 | pseries_dev->msi_used++; | 608 | pseries_dev->msi_used += nr_irqs; |
| 609 | return 0; | 609 | return 0; |
| 610 | 610 | ||
| 611 | out: | 611 | out: |
File renamed without changes.
| @@ -15,9 +15,9 @@ if [ -z "$is_64bit" ]; then · 14 unmodified lines | |||
| 15 | RELOCATION=R_PPC_ADDR32 | 15 | RELOCATION=R_PPC_ADDR32 |
| 16 | fi | 16 | fi |
| 17 | 17 | ||
| 18 | num_ool_stubs_total=$($objdump -r -j __patchable_function_entries "$vmlinux_o" | | 18 | num_ool_stubs_total=$($objdump -r -j __patchable_function_entries -d "$vmlinux_o" | |
| 19 | grep -c "$RELOCATION") | 19 | grep -c "$RELOCATION") |
| 20 | num_ool_stubs_inittext=$($objdump -r -j __patchable_function_entries "$vmlinux_o" | | 20 | num_ool_stubs_inittext=$($objdump -r -j __patchable_function_entries -d "$vmlinux_o" | |
| 21 | grep -e ".init.text" -e ".text.startup" | grep -c "$RELOCATION") | 21 | grep -e ".init.text" -e ".text.startup" | grep -c "$RELOCATION") |
| 22 | num_ool_stubs_text=$((num_ool_stubs_total - num_ool_stubs_inittext)) | 22 | num_ool_stubs_text=$((num_ool_stubs_total - num_ool_stubs_inittext)) |
| 23 | 23 | ||
| @@ -428,6 +428,7 @@ · 427 unmodified lines | |||
| 428 | clocks = <&clkcfg CLK_CAN0>, <&clkcfg CLK_MSSPLL3>; | 428 | clocks = <&clkcfg CLK_CAN0>, <&clkcfg CLK_MSSPLL3>; |
| 429 | interrupt-parent = <&plic>; | 429 | interrupt-parent = <&plic>; |
| 430 | interrupts = <56>; | 430 | interrupts = <56>; |
| 431 | resets = <&mss_top_sysreg CLK_CAN0>; | ||
| 431 | status = "disabled"; | 432 | status = "disabled"; |
| 432 | }; | 433 | }; |
| 433 | 434 | ||
| @@ -437,6 +438,7 @@ · 3 unmodified lines | |||
| 437 | clocks = <&clkcfg CLK_CAN1>, <&clkcfg CLK_MSSPLL3>; | 438 | clocks = <&clkcfg CLK_CAN1>, <&clkcfg CLK_MSSPLL3>; |
| 438 | interrupt-parent = <&plic>; | 439 | interrupt-parent = <&plic>; |
| 439 | interrupts = <57>; | 440 | interrupts = <57>; |
| 441 | resets = <&mss_top_sysreg CLK_CAN1>; | ||
| 440 | status = "disabled"; | 442 | status = "disabled"; |
| 441 | }; | 443 | }; |
| 442 | 444 | ||
| @@ -2,6 +2,10 @@ · 1 unmodified line | |||
| 2 | #ifndef _ASM_RISCV_RUNTIME_CONST_H | 2 | #ifndef _ASM_RISCV_RUNTIME_CONST_H |
| 3 | #define _ASM_RISCV_RUNTIME_CONST_H | 3 | #define _ASM_RISCV_RUNTIME_CONST_H |
| 4 | 4 | ||
| 5 | #ifdef MODULE | ||
| 6 | #error "Cannot use runtime-const infrastructure from modules" | ||
| 7 | #endif | ||
| 8 | |||
| 5 | #include <asm/asm.h> | 9 | #include <asm/asm.h> |
| 6 | #include <asm/alternative.h> | 10 | #include <asm/alternative.h> |
| 7 | #include <asm/cacheflush.h> | 11 | #include <asm/cacheflush.h> |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | #ifndef __ASSEMBLER__ | 9 | #ifndef __ASSEMBLER__ |
| 10 | 10 | ||
| 11 | #include <linux/types.h> | 11 | #include <linux/types.h> |
| 12 | #include <linux/const.h> | ||
| 12 | 13 | ||
| 13 | #define PTRACE_GETFDPIC 33 | 14 | #define PTRACE_GETFDPIC 33 |
| 14 | 15 | ||
| @@ -138,12 +139,12 @@ struct __sc_riscv_cfi_state { · 123 unmodified lines | |||
| 138 | #define PTRACE_CFI_SS_LOCK_BIT 4 | 139 | #define PTRACE_CFI_SS_LOCK_BIT 4 |
| 139 | #define PTRACE_CFI_SS_PTR_BIT 5 | 140 | #define PTRACE_CFI_SS_PTR_BIT 5 |
| 140 | 141 | ||
| 141 | #define PTRACE_CFI_LP_EN_STATE BIT(PTRACE_CFI_LP_EN_BIT) | 142 | #define PTRACE_CFI_LP_EN_STATE _BITUL(PTRACE_CFI_LP_EN_BIT) |
| 142 | #define PTRACE_CFI_LP_LOCK_STATE BIT(PTRACE_CFI_LP_LOCK_BIT) | 143 | #define PTRACE_CFI_LP_LOCK_STATE _BITUL(PTRACE_CFI_LP_LOCK_BIT) |
| 143 | #define PTRACE_CFI_ELP_STATE BIT(PTRACE_CFI_ELP_BIT) | 144 | #define PTRACE_CFI_ELP_STATE _BITUL(PTRACE_CFI_ELP_BIT) |
| 144 | #define PTRACE_CFI_SS_EN_STATE BIT(PTRACE_CFI_SS_EN_BIT) | 145 | #define PTRACE_CFI_SS_EN_STATE _BITUL(PTRACE_CFI_SS_EN_BIT) |
| 145 | #define PTRACE_CFI_SS_LOCK_STATE BIT(PTRACE_CFI_SS_LOCK_BIT) | 146 | #define PTRACE_CFI_SS_LOCK_STATE _BITUL(PTRACE_CFI_SS_LOCK_BIT) |
| 146 | #define PTRACE_CFI_SS_PTR_STATE BIT(PTRACE_CFI_SS_PTR_BIT) | 147 | #define PTRACE_CFI_SS_PTR_STATE _BITUL(PTRACE_CFI_SS_PTR_BIT) |
| 147 | 148 | ||
| 148 | #define PRACE_CFI_STATE_INVALID_MASK ~(PTRACE_CFI_LP_EN_STATE | \ | 149 | #define PRACE_CFI_STATE_INVALID_MASK ~(PTRACE_CFI_LP_EN_STATE | \ |
| 149 | PTRACE_CFI_LP_LOCK_STATE | \ | 150 | PTRACE_CFI_LP_LOCK_STATE | \ |
| @@ -175,7 +175,7 @@ struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] = { · 174 unmodified lines | |||
| 175 | {DBG_REG_T1, GDB_SIZEOF_REG, offsetof(struct pt_regs, t1)}, | 175 | {DBG_REG_T1, GDB_SIZEOF_REG, offsetof(struct pt_regs, t1)}, |
| 176 | {DBG_REG_T2, GDB_SIZEOF_REG, offsetof(struct pt_regs, t2)}, | 176 | {DBG_REG_T2, GDB_SIZEOF_REG, offsetof(struct pt_regs, t2)}, |
| 177 | {DBG_REG_FP, GDB_SIZEOF_REG, offsetof(struct pt_regs, s0)}, | 177 | {DBG_REG_FP, GDB_SIZEOF_REG, offsetof(struct pt_regs, s0)}, |
| 178 | {DBG_REG_S1, GDB_SIZEOF_REG, offsetof(struct pt_regs, a1)}, | 178 | {DBG_REG_S1, GDB_SIZEOF_REG, offsetof(struct pt_regs, s1)}, |
| 179 | {DBG_REG_A0, GDB_SIZEOF_REG, offsetof(struct pt_regs, a0)}, | 179 | {DBG_REG_A0, GDB_SIZEOF_REG, offsetof(struct pt_regs, a0)}, |
| 180 | {DBG_REG_A1, GDB_SIZEOF_REG, offsetof(struct pt_regs, a1)}, | 180 | {DBG_REG_A1, GDB_SIZEOF_REG, offsetof(struct pt_regs, a1)}, |
| 181 | {DBG_REG_A2, GDB_SIZEOF_REG, offsetof(struct pt_regs, a2)}, | 181 | {DBG_REG_A2, GDB_SIZEOF_REG, offsetof(struct pt_regs, a2)}, |
| @@ -244,8 +244,9 @@ sleeping_thread_to_gdb_regs(unsigned long *gdb_regs, struct task_struct *task) · 62 unmodified lines | |||
| 244 | gdb_regs[DBG_REG_S6_OFF] = task->thread.s[6]; | 244 | gdb_regs[DBG_REG_S6_OFF] = task->thread.s[6]; |
| 245 | gdb_regs[DBG_REG_S7_OFF] = task->thread.s[7]; | 245 | gdb_regs[DBG_REG_S7_OFF] = task->thread.s[7]; |
| 246 | gdb_regs[DBG_REG_S8_OFF] = task->thread.s[8]; | 246 | gdb_regs[DBG_REG_S8_OFF] = task->thread.s[8]; |
| 247 | gdb_regs[DBG_REG_S9_OFF] = task->thread.s[10]; | 247 | gdb_regs[DBG_REG_S9_OFF] = task->thread.s[9]; |
| 248 | gdb_regs[DBG_REG_S10_OFF] = task->thread.s[11]; | 248 | gdb_regs[DBG_REG_S10_OFF] = task->thread.s[10]; |
| 249 | gdb_regs[DBG_REG_S11_OFF] = task->thread.s[11]; | ||
| 249 | gdb_regs[DBG_REG_EPC_OFF] = task->thread.ra; | 250 | gdb_regs[DBG_REG_EPC_OFF] = task->thread.ra; |
| 250 | } | 251 | } |
| 251 | 252 | ||
| @@ -42,19 +42,20 @@ static inline bool is_kernel_exittext(uintptr_t addr) · 41 unmodified lines | |||
| 42 | static __always_inline void *patch_map(void *addr, const unsigned int fixmap) | 42 | static __always_inline void *patch_map(void *addr, const unsigned int fixmap) |
| 43 | { | 43 | { |
| 44 | uintptr_t uintaddr = (uintptr_t) addr; | 44 | uintptr_t uintaddr = (uintptr_t) addr; |
| 45 | struct page *page; | 45 | phys_addr_t phys; |
| 46 | 46 | ||
| 47 | if (core_kernel_text(uintaddr) || is_kernel_exittext(uintaddr)) | 47 | if (core_kernel_text(uintaddr) || is_kernel_exittext(uintaddr)) { |
| 48 | page = phys_to_page(__pa_symbol(addr)); | 48 | phys = __pa_symbol(addr); |
| 49 | else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) | 49 | } else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) { |
| 50 | page = vmalloc_to_page(addr); | 50 | struct page *page = vmalloc_to_page(addr); |
| 51 | else | ||
| 52 | return addr; | ||
| 53 | 51 | ||
| 54 | BUG_ON(!page); | 52 | BUG_ON(!page); |
| 53 | phys = page_to_phys(page) + offset_in_page(addr); | ||
| 54 | } else { | ||
| 55 | return addr; | ||
| 56 | } | ||
| 55 | 57 | ||
| 56 | return (void *)set_fixmap_offset(fixmap, page_to_phys(page) + | 58 | return (void *)set_fixmap_offset(fixmap, phys); |
| 57 | offset_in_page(addr)); | ||
| 58 | } | 59 | } |
| 59 | 60 | ||
| 60 | static void patch_unmap(int fixmap) | 61 | static void patch_unmap(int fixmap) |
| @@ -347,8 +347,10 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg) · 346 unmodified lines | |||
| 347 | if (arg & PR_TAGGED_ADDR_ENABLE && (tagged_addr_disabled || !pmlen)) | 347 | if (arg & PR_TAGGED_ADDR_ENABLE && (tagged_addr_disabled || !pmlen)) |
| 348 | return -EINVAL; | 348 | return -EINVAL; |
| 349 | 349 | ||
| 350 | if (!(arg & PR_TAGGED_ADDR_ENABLE)) | 350 | if (!(arg & PR_TAGGED_ADDR_ENABLE)) { |
| 351 | pmlen = PMLEN_0; | 351 | pmlen = PMLEN_0; |
| 352 | pmm = ENVCFG_PMM_PMLEN_0; | ||
| 353 | } | ||
| 352 | 354 | ||
| 353 | if (mmap_write_lock_killable(mm)) | 355 | if (mmap_write_lock_killable(mm)) |
| 354 | return -EINTR; | 356 | return -EINTR; |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | |||
| 13 | #include <linux/irqchip/riscv-imsic.h> | 13 | #include <linux/irqchip/riscv-imsic.h> |
| 14 | #include <linux/irqdomain.h> | 14 | #include <linux/irqdomain.h> |
| 15 | #include <linux/kvm_host.h> | 15 | #include <linux/kvm_host.h> |
| 16 | #include <linux/nospec.h> | ||
| 16 | #include <linux/percpu.h> | 17 | #include <linux/percpu.h> |
| 17 | #include <linux/spinlock.h> | 18 | #include <linux/spinlock.h> |
| 18 | #include <asm/cpufeature.h> | 19 | #include <asm/cpufeature.h> |
| @@ -182,9 +183,14 @@ int kvm_riscv_vcpu_aia_get_csr(struct kvm_vcpu *vcpu, · 163 unmodified lines | |||
| 182 | unsigned long *out_val) | 183 | unsigned long *out_val) |
| 183 | { | 184 | { |
| 184 | struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; | 185 | struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; |
| 186 | unsigned long regs_max = sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); | ||
| 185 | 187 | ||
| 186 | if (reg_num >= sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long)) | 188 | if (!riscv_isa_extension_available(vcpu->arch.isa, SSAIA)) |
| 187 | return -ENOENT; | 189 | return -ENOENT; |
| 190 | if (reg_num >= regs_max) | ||
| 191 | return -ENOENT; | ||
| 192 | |||
| 193 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 188 | 194 | ||
| 189 | *out_val = 0; | 195 | *out_val = 0; |
| 190 | if (kvm_riscv_aia_available()) | 196 | if (kvm_riscv_aia_available()) |
| @@ -198,9 +204,14 @@ int kvm_riscv_vcpu_aia_set_csr(struct kvm_vcpu *vcpu, · 7 unmodified lines | |||
| 198 | unsigned long val) | 204 | unsigned long val) |
| 199 | { | 205 | { |
| 200 | struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; | 206 | struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; |
| 207 | unsigned long regs_max = sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); | ||
| 201 | 208 | ||
| 202 | if (reg_num >= sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long)) | 209 | if (!riscv_isa_extension_available(vcpu->arch.isa, SSAIA)) |
| 203 | return -ENOENT; | 210 | return -ENOENT; |
| 211 | if (reg_num >= regs_max) | ||
| 212 | return -ENOENT; | ||
| 213 | |||
| 214 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 204 | 215 | ||
| 205 | if (kvm_riscv_aia_available()) { | 216 | if (kvm_riscv_aia_available()) { |
| 206 | ((unsigned long *)csr)[reg_num] = val; | 217 | ((unsigned long *)csr)[reg_num] = val; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <linux/irqchip/riscv-aplic.h> | 10 | #include <linux/irqchip/riscv-aplic.h> |
| 11 | #include <linux/kvm_host.h> | 11 | #include <linux/kvm_host.h> |
| 12 | #include <linux/math.h> | 12 | #include <linux/math.h> |
| 13 | #include <linux/nospec.h> | ||
| 13 | #include <linux/spinlock.h> | 14 | #include <linux/spinlock.h> |
| 14 | #include <linux/swab.h> | 15 | #include <linux/swab.h> |
| 15 | #include <kvm/iodev.h> | 16 | #include <kvm/iodev.h> |
| @@ -45,7 +46,7 @@ static u32 aplic_read_sourcecfg(struct aplic *aplic, u32 irq) · 29 unmodified lines | |||
| 45 | 46 | ||
| 46 | if (!irq || aplic->nr_irqs <= irq) | 47 | if (!irq || aplic->nr_irqs <= irq) |
| 47 | return 0; | 48 | return 0; |
| 48 | irqd = &aplic->irqs[irq]; | 49 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 49 | 50 | ||
| 50 | raw_spin_lock_irqsave(&irqd->lock, flags); | 51 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 51 | ret = irqd->sourcecfg; | 52 | ret = irqd->sourcecfg; |
| @@ -61,7 +62,7 @@ static void aplic_write_sourcecfg(struct aplic *aplic, u32 irq, u32 val) · 9 unmodified lines | |||
| 61 | 62 | ||
| 62 | if (!irq || aplic->nr_irqs <= irq) | 63 | if (!irq || aplic->nr_irqs <= irq) |
| 63 | return; | 64 | return; |
| 64 | irqd = &aplic->irqs[irq]; | 65 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 65 | 66 | ||
| 66 | if (val & APLIC_SOURCECFG_D) | 67 | if (val & APLIC_SOURCECFG_D) |
| 67 | val = 0; | 68 | val = 0; |
| @@ -81,7 +82,7 @@ static u32 aplic_read_target(struct aplic *aplic, u32 irq) · 13 unmodified lines | |||
| 81 | 82 | ||
| 82 | if (!irq || aplic->nr_irqs <= irq) | 83 | if (!irq || aplic->nr_irqs <= irq) |
| 83 | return 0; | 84 | return 0; |
| 84 | irqd = &aplic->irqs[irq]; | 85 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 85 | 86 | ||
| 86 | raw_spin_lock_irqsave(&irqd->lock, flags); | 87 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 87 | ret = irqd->target; | 88 | ret = irqd->target; |
| @@ -97,7 +98,7 @@ static void aplic_write_target(struct aplic *aplic, u32 irq, u32 val) · 9 unmodified lines | |||
| 97 | 98 | ||
| 98 | if (!irq || aplic->nr_irqs <= irq) | 99 | if (!irq || aplic->nr_irqs <= irq) |
| 99 | return; | 100 | return; |
| 100 | irqd = &aplic->irqs[irq]; | 101 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 101 | 102 | ||
| 102 | val &= APLIC_TARGET_EIID_MASK | | 103 | val &= APLIC_TARGET_EIID_MASK | |
| 103 | (APLIC_TARGET_HART_IDX_MASK << APLIC_TARGET_HART_IDX_SHIFT) | | 104 | (APLIC_TARGET_HART_IDX_MASK << APLIC_TARGET_HART_IDX_SHIFT) | |
| @@ -116,7 +117,7 @@ static bool aplic_read_pending(struct aplic *aplic, u32 irq) · 12 unmodified lines | |||
| 116 | 117 | ||
| 117 | if (!irq || aplic->nr_irqs <= irq) | 118 | if (!irq || aplic->nr_irqs <= irq) |
| 118 | return false; | 119 | return false; |
| 119 | irqd = &aplic->irqs[irq]; | 120 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 120 | 121 | ||
| 121 | raw_spin_lock_irqsave(&irqd->lock, flags); | 122 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 122 | ret = (irqd->state & APLIC_IRQ_STATE_PENDING) ? true : false; | 123 | ret = (irqd->state & APLIC_IRQ_STATE_PENDING) ? true : false; |
| @@ -132,7 +133,7 @@ static void aplic_write_pending(struct aplic *aplic, u32 irq, bool pending) · 9 unmodified lines | |||
| 132 | 133 | ||
| 133 | if (!irq || aplic->nr_irqs <= irq) | 134 | if (!irq || aplic->nr_irqs <= irq) |
| 134 | return; | 135 | return; |
| 135 | irqd = &aplic->irqs[irq]; | 136 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 136 | 137 | ||
| 137 | raw_spin_lock_irqsave(&irqd->lock, flags); | 138 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 138 | 139 | ||
| @@ -170,7 +171,7 @@ static bool aplic_read_enabled(struct aplic *aplic, u32 irq) · 31 unmodified lines | |||
| 170 | 171 | ||
| 171 | if (!irq || aplic->nr_irqs <= irq) | 172 | if (!irq || aplic->nr_irqs <= irq) |
| 172 | return false; | 173 | return false; |
| 173 | irqd = &aplic->irqs[irq]; | 174 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 174 | 175 | ||
| 175 | raw_spin_lock_irqsave(&irqd->lock, flags); | 176 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 176 | ret = (irqd->state & APLIC_IRQ_STATE_ENABLED) ? true : false; | 177 | ret = (irqd->state & APLIC_IRQ_STATE_ENABLED) ? true : false; |
| @@ -186,7 +187,7 @@ static void aplic_write_enabled(struct aplic *aplic, u32 irq, bool enabled) · 9 unmodified lines | |||
| 186 | 187 | ||
| 187 | if (!irq || aplic->nr_irqs <= irq) | 188 | if (!irq || aplic->nr_irqs <= irq) |
| 188 | return; | 189 | return; |
| 189 | irqd = &aplic->irqs[irq]; | 190 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 190 | 191 | ||
| 191 | raw_spin_lock_irqsave(&irqd->lock, flags); | 192 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 192 | if (enabled) | 193 | if (enabled) |
| @@ -205,7 +206,7 @@ static bool aplic_read_input(struct aplic *aplic, u32 irq) · 12 unmodified lines | |||
| 205 | 206 | ||
| 206 | if (!irq || aplic->nr_irqs <= irq) | 207 | if (!irq || aplic->nr_irqs <= irq) |
| 207 | return false; | 208 | return false; |
| 208 | irqd = &aplic->irqs[irq]; | 209 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 209 | 210 | ||
| 210 | raw_spin_lock_irqsave(&irqd->lock, flags); | 211 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 211 | 212 | ||
| @@ -254,7 +255,7 @@ static void aplic_update_irq_range(struct kvm *kvm, u32 first, u32 last) · 42 unmodified lines | |||
| 254 | for (irq = first; irq <= last; irq++) { | 255 | for (irq = first; irq <= last; irq++) { |
| 255 | if (!irq || aplic->nr_irqs <= irq) | 256 | if (!irq || aplic->nr_irqs <= irq) |
| 256 | continue; | 257 | continue; |
| 257 | irqd = &aplic->irqs[irq]; | 258 | irqd = &aplic->irqs[array_index_nospec(irq, aplic->nr_irqs)]; |
| 258 | 259 | ||
| 259 | raw_spin_lock_irqsave(&irqd->lock, flags); | 260 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| 260 | 261 | ||
| @@ -283,7 +284,7 @@ int kvm_riscv_aia_aplic_inject(struct kvm *kvm, u32 source, bool level) · 22 unmodified lines | |||
| 283 | 284 | ||
| 284 | if (!aplic || !source || (aplic->nr_irqs <= source)) | 285 | if (!aplic || !source || (aplic->nr_irqs <= source)) |
| 285 | return -ENODEV; | 286 | return -ENODEV; |
| 286 | irqd = &aplic->irqs[source]; | 287 | irqd = &aplic->irqs[array_index_nospec(source, aplic->nr_irqs)]; |
| 287 | ie = (aplic->domaincfg & APLIC_DOMAINCFG_IE) ? true : false; | 288 | ie = (aplic->domaincfg & APLIC_DOMAINCFG_IE) ? true : false; |
| 288 | 289 | ||
| 289 | raw_spin_lock_irqsave(&irqd->lock, flags); | 290 | raw_spin_lock_irqsave(&irqd->lock, flags); |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | |||
| 11 | #include <linux/irqchip/riscv-imsic.h> | 11 | #include <linux/irqchip/riscv-imsic.h> |
| 12 | #include <linux/kvm_host.h> | 12 | #include <linux/kvm_host.h> |
| 13 | #include <linux/uaccess.h> | 13 | #include <linux/uaccess.h> |
| 14 | #include <linux/cpufeature.h> | ||
| 14 | 15 | ||
| 15 | static int aia_create(struct kvm_device *dev, u32 type) | 16 | static int aia_create(struct kvm_device *dev, u32 type) |
| 16 | { | 17 | { |
| @@ -22,6 +23,9 @@ static int aia_create(struct kvm_device *dev, u32 type) · 5 unmodified lines | |||
| 22 | if (irqchip_in_kernel(kvm)) | 23 | if (irqchip_in_kernel(kvm)) |
| 23 | return -EEXIST; | 24 | return -EEXIST; |
| 24 | 25 | ||
| 26 | if (!riscv_isa_extension_available(NULL, SSAIA)) | ||
| 27 | return -ENODEV; | ||
| 28 | |||
| 25 | ret = -EBUSY; | 29 | ret = -EBUSY; |
| 26 | if (kvm_trylock_all_vcpus(kvm)) | 30 | if (kvm_trylock_all_vcpus(kvm)) |
| 27 | return ret; | 31 | return ret; |
| @@ -437,7 +441,7 @@ static int aia_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr) · 409 unmodified lines | |||
| 437 | 441 | ||
| 438 | static int aia_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr) | 442 | static int aia_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr) |
| 439 | { | 443 | { |
| 440 | int nr_vcpus; | 444 | int nr_vcpus, r = -ENXIO; |
| 441 | 445 | ||
| 442 | switch (attr->group) { | 446 | switch (attr->group) { |
| 443 | case KVM_DEV_RISCV_AIA_GRP_CONFIG: | 447 | case KVM_DEV_RISCV_AIA_GRP_CONFIG: |
| @@ -466,12 +470,18 @@ static int aia_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr) · 22 unmodified lines | |||
| 466 | } | 470 | } |
| 467 | break; | 471 | break; |
| 468 | case KVM_DEV_RISCV_AIA_GRP_APLIC: | 472 | case KVM_DEV_RISCV_AIA_GRP_APLIC: |
| 469 | return kvm_riscv_aia_aplic_has_attr(dev->kvm, attr->attr); | 473 | mutex_lock(&dev->kvm->lock); |
| 474 | r = kvm_riscv_aia_aplic_has_attr(dev->kvm, attr->attr); | ||
| 475 | mutex_unlock(&dev->kvm->lock); | ||
| 476 | break; | ||
| 470 | case KVM_DEV_RISCV_AIA_GRP_IMSIC: | 477 | case KVM_DEV_RISCV_AIA_GRP_IMSIC: |
| 471 | return kvm_riscv_aia_imsic_has_attr(dev->kvm, attr->attr); | 478 | mutex_lock(&dev->kvm->lock); |
| 479 | r = kvm_riscv_aia_imsic_has_attr(dev->kvm, attr->attr); | ||
| 480 | mutex_unlock(&dev->kvm->lock); | ||
| 481 | break; | ||
| 472 | } | 482 | } |
| 473 | 483 | ||
| 474 | return -ENXIO; | 484 | return r; |
| 475 | } | 485 | } |
| 476 | 486 | ||
| 477 | struct kvm_device_ops kvm_riscv_aia_device_ops = { | 487 | struct kvm_device_ops kvm_riscv_aia_device_ops = { |
| @@ -908,6 +908,10 @@ int kvm_riscv_vcpu_aia_imsic_rmw(struct kvm_vcpu *vcpu, unsigned long isel, · 907 unmodified lines | |||
| 908 | int r, rc = KVM_INSN_CONTINUE_NEXT_SEPC; | 908 | int r, rc = KVM_INSN_CONTINUE_NEXT_SEPC; |
| 909 | struct imsic *imsic = vcpu->arch.aia_context.imsic_state; | 909 | struct imsic *imsic = vcpu->arch.aia_context.imsic_state; |
| 910 | 910 | ||
| 911 | /* If IMSIC vCPU state not initialized then forward to user space */ | ||
| 912 | if (!imsic) | ||
| 913 | return KVM_INSN_EXIT_TO_USER_SPACE; | ||
| 914 | |||
| 911 | if (isel == KVM_RISCV_AIA_IMSIC_TOPEI) { | 915 | if (isel == KVM_RISCV_AIA_IMSIC_TOPEI) { |
| 912 | /* Read pending and enabled interrupt with highest priority */ | 916 | /* Read pending and enabled interrupt with highest priority */ |
| 913 | topei = imsic_mrif_topei(imsic->swfile, imsic->nr_eix, | 917 | topei = imsic_mrif_topei(imsic->swfile, imsic->nr_eix, |
| @@ -245,6 +245,7 @@ out: · 244 unmodified lines | |||
| 245 | bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) | 245 | bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) |
| 246 | { | 246 | { |
| 247 | struct kvm_gstage gstage; | 247 | struct kvm_gstage gstage; |
| 248 | bool mmu_locked; | ||
| 248 | 249 | ||
| 249 | if (!kvm->arch.pgd) | 250 | if (!kvm->arch.pgd) |
| 250 | return false; | 251 | return false; |
| @@ -253,9 +254,12 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · 2 unmodified lines | |||
| 253 | gstage.flags = 0; | 254 | gstage.flags = 0; |
| 254 | gstage.vmid = READ_ONCE(kvm->arch.vmid.vmid); | 255 | gstage.vmid = READ_ONCE(kvm->arch.vmid.vmid); |
| 255 | gstage.pgd = kvm->arch.pgd; | 256 | gstage.pgd = kvm->arch.pgd; |
| 257 | mmu_locked = spin_trylock(&kvm->mmu_lock); | ||
| 256 | kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, | 258 | kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, |
| 257 | (range->end - range->start) << PAGE_SHIFT, | 259 | (range->end - range->start) << PAGE_SHIFT, |
| 258 | range->may_block); | 260 | range->may_block); |
| 261 | if (mmu_locked) | ||
| 262 | spin_unlock(&kvm->mmu_lock); | ||
| 259 | return false; | 263 | return false; |
| 260 | } | 264 | } |
| 261 | 265 | ||
| @@ -535,7 +539,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, · 273 unmodified lines | |||
| 535 | goto out_unlock; | 539 | goto out_unlock; |
| 536 | 540 | ||
| 537 | /* Check if we are backed by a THP and thus use block mapping if possible */ | 541 | /* Check if we are backed by a THP and thus use block mapping if possible */ |
| 538 | if (vma_pagesize == PAGE_SIZE) | 542 | if (!logging && (vma_pagesize == PAGE_SIZE)) |
| 539 | vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &hfn, &gpa); | 543 | vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &hfn, &gpa); |
| 540 | 544 | ||
| 541 | if (writable) { | 545 | if (writable) { |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | |||
| 24 | #define CREATE_TRACE_POINTS | 24 | #define CREATE_TRACE_POINTS |
| 25 | #include "trace.h" | 25 | #include "trace.h" |
| 26 | 26 | ||
| 27 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 27 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 28 | KVM_GENERIC_VCPU_STATS(), | 28 | KVM_GENERIC_VCPU_STATS(), |
| 29 | STATS_DESC_COUNTER(VCPU, ecall_exit_stat), | 29 | STATS_DESC_COUNTER(VCPU, ecall_exit_stat), |
| 30 | STATS_DESC_COUNTER(VCPU, wfi_exit_stat), | 30 | STATS_DESC_COUNTER(VCPU, wfi_exit_stat), |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <linux/errno.h> | 10 | #include <linux/errno.h> |
| 11 | #include <linux/err.h> | 11 | #include <linux/err.h> |
| 12 | #include <linux/kvm_host.h> | 12 | #include <linux/kvm_host.h> |
| 13 | #include <linux/nospec.h> | ||
| 13 | #include <linux/uaccess.h> | 14 | #include <linux/uaccess.h> |
| 14 | #include <asm/cpufeature.h> | 15 | #include <asm/cpufeature.h> |
| 15 | 16 | ||
| @@ -93,9 +94,11 @@ int kvm_riscv_vcpu_get_reg_fp(struct kvm_vcpu *vcpu, · 77 unmodified lines | |||
| 93 | if (reg_num == KVM_REG_RISCV_FP_F_REG(fcsr)) | 94 | if (reg_num == KVM_REG_RISCV_FP_F_REG(fcsr)) |
| 94 | reg_val = &cntx->fp.f.fcsr; | 95 | reg_val = &cntx->fp.f.fcsr; |
| 95 | else if ((KVM_REG_RISCV_FP_F_REG(f[0]) <= reg_num) && | 96 | else if ((KVM_REG_RISCV_FP_F_REG(f[0]) <= reg_num) && |
| 96 | reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) | 97 | reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) { |
| 98 | reg_num = array_index_nospec(reg_num, | ||
| 99 | ARRAY_SIZE(cntx->fp.f.f)); | ||
| 97 | reg_val = &cntx->fp.f.f[reg_num]; | 100 | reg_val = &cntx->fp.f.f[reg_num]; |
| 98 | else | 101 | } else |
| 99 | return -ENOENT; | 102 | return -ENOENT; |
| 100 | } else if ((rtype == KVM_REG_RISCV_FP_D) && | 103 | } else if ((rtype == KVM_REG_RISCV_FP_D) && |
| 101 | riscv_isa_extension_available(vcpu->arch.isa, d)) { | 104 | riscv_isa_extension_available(vcpu->arch.isa, d)) { |
| @@ -107,6 +110,8 @@ int kvm_riscv_vcpu_get_reg_fp(struct kvm_vcpu *vcpu, · 5 unmodified lines | |||
| 107 | reg_num <= KVM_REG_RISCV_FP_D_REG(f[31])) { | 110 | reg_num <= KVM_REG_RISCV_FP_D_REG(f[31])) { |
| 108 | if (KVM_REG_SIZE(reg->id) != sizeof(u64)) | 111 | if (KVM_REG_SIZE(reg->id) != sizeof(u64)) |
| 109 | return -EINVAL; | 112 | return -EINVAL; |
| 113 | reg_num = array_index_nospec(reg_num, | ||
| 114 | ARRAY_SIZE(cntx->fp.d.f)); | ||
| 110 | reg_val = &cntx->fp.d.f[reg_num]; | 115 | reg_val = &cntx->fp.d.f[reg_num]; |
| 111 | } else | 116 | } else |
| 112 | return -ENOENT; | 117 | return -ENOENT; |
| @@ -138,9 +143,11 @@ int kvm_riscv_vcpu_set_reg_fp(struct kvm_vcpu *vcpu, · 25 unmodified lines | |||
| 138 | if (reg_num == KVM_REG_RISCV_FP_F_REG(fcsr)) | 143 | if (reg_num == KVM_REG_RISCV_FP_F_REG(fcsr)) |
| 139 | reg_val = &cntx->fp.f.fcsr; | 144 | reg_val = &cntx->fp.f.fcsr; |
| 140 | else if ((KVM_REG_RISCV_FP_F_REG(f[0]) <= reg_num) && | 145 | else if ((KVM_REG_RISCV_FP_F_REG(f[0]) <= reg_num) && |
| 141 | reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) | 146 | reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) { |
| 147 | reg_num = array_index_nospec(reg_num, | ||
| 148 | ARRAY_SIZE(cntx->fp.f.f)); | ||
| 142 | reg_val = &cntx->fp.f.f[reg_num]; | 149 | reg_val = &cntx->fp.f.f[reg_num]; |
| 143 | else | 150 | } else |
| 144 | return -ENOENT; | 151 | return -ENOENT; |
| 145 | } else if ((rtype == KVM_REG_RISCV_FP_D) && | 152 | } else if ((rtype == KVM_REG_RISCV_FP_D) && |
| 146 | riscv_isa_extension_available(vcpu->arch.isa, d)) { | 153 | riscv_isa_extension_available(vcpu->arch.isa, d)) { |
| @@ -152,6 +159,8 @@ int kvm_riscv_vcpu_set_reg_fp(struct kvm_vcpu *vcpu, · 5 unmodified lines | |||
| 152 | reg_num <= KVM_REG_RISCV_FP_D_REG(f[31])) { | 159 | reg_num <= KVM_REG_RISCV_FP_D_REG(f[31])) { |
| 153 | if (KVM_REG_SIZE(reg->id) != sizeof(u64)) | 160 | if (KVM_REG_SIZE(reg->id) != sizeof(u64)) |
| 154 | return -EINVAL; | 161 | return -EINVAL; |
| 162 | reg_num = array_index_nospec(reg_num, | ||
| 163 | ARRAY_SIZE(cntx->fp.d.f)); | ||
| 155 | reg_val = &cntx->fp.d.f[reg_num]; | 164 | reg_val = &cntx->fp.d.f[reg_num]; |
| 156 | } else | 165 | } else |
| 157 | return -ENOENT; | 166 | return -ENOENT; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <linux/bitops.h> | 10 | #include <linux/bitops.h> |
| 11 | #include <linux/errno.h> | 11 | #include <linux/errno.h> |
| 12 | #include <linux/err.h> | 12 | #include <linux/err.h> |
| 13 | #include <linux/nospec.h> | ||
| 13 | #include <linux/uaccess.h> | 14 | #include <linux/uaccess.h> |
| 14 | #include <linux/kvm_host.h> | 15 | #include <linux/kvm_host.h> |
| 15 | #include <asm/cacheflush.h> | 16 | #include <asm/cacheflush.h> |
| @@ -127,6 +128,7 @@ static int kvm_riscv_vcpu_isa_check_host(unsigned long kvm_ext, unsigned long *g · 111 unmodified lines | |||
| 127 | kvm_ext >= ARRAY_SIZE(kvm_isa_ext_arr)) | 128 | kvm_ext >= ARRAY_SIZE(kvm_isa_ext_arr)) |
| 128 | return -ENOENT; | 129 | return -ENOENT; |
| 129 | 130 | ||
| 131 | kvm_ext = array_index_nospec(kvm_ext, ARRAY_SIZE(kvm_isa_ext_arr)); | ||
| 130 | *guest_ext = kvm_isa_ext_arr[kvm_ext]; | 132 | *guest_ext = kvm_isa_ext_arr[kvm_ext]; |
| 131 | switch (*guest_ext) { | 133 | switch (*guest_ext) { |
| 132 | case RISCV_ISA_EXT_SMNPM: | 134 | case RISCV_ISA_EXT_SMNPM: |
| @@ -443,13 +445,16 @@ static int kvm_riscv_vcpu_get_reg_core(struct kvm_vcpu *vcpu, · 310 unmodified lines | |||
| 443 | unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | | 445 | unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | |
| 444 | KVM_REG_SIZE_MASK | | 446 | KVM_REG_SIZE_MASK | |
| 445 | KVM_REG_RISCV_CORE); | 447 | KVM_REG_RISCV_CORE); |
| 448 | unsigned long regs_max = sizeof(struct kvm_riscv_core) / sizeof(unsigned long); | ||
| 446 | unsigned long reg_val; | 449 | unsigned long reg_val; |
| 447 | 450 | ||
| 448 | if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) | 451 | if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) |
| 449 | return -EINVAL; | 452 | return -EINVAL; |
| 450 | if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) | 453 | if (reg_num >= regs_max) |
| 451 | return -ENOENT; | 454 | return -ENOENT; |
| 452 | 455 | ||
| 456 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 457 | |||
| 453 | if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) | 458 | if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) |
| 454 | reg_val = cntx->sepc; | 459 | reg_val = cntx->sepc; |
| 455 | else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && | 460 | else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && |
| @@ -476,13 +481,16 @@ static int kvm_riscv_vcpu_set_reg_core(struct kvm_vcpu *vcpu, · 20 unmodified lines | |||
| 476 | unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | | 481 | unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | |
| 477 | KVM_REG_SIZE_MASK | | 482 | KVM_REG_SIZE_MASK | |
| 478 | KVM_REG_RISCV_CORE); | 483 | KVM_REG_RISCV_CORE); |
| 484 | unsigned long regs_max = sizeof(struct kvm_riscv_core) / sizeof(unsigned long); | ||
| 479 | unsigned long reg_val; | 485 | unsigned long reg_val; |
| 480 | 486 | ||
| 481 | if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) | 487 | if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) |
| 482 | return -EINVAL; | 488 | return -EINVAL; |
| 483 | if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) | 489 | if (reg_num >= regs_max) |
| 484 | return -ENOENT; | 490 | return -ENOENT; |
| 485 | 491 | ||
| 492 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 493 | |||
| 486 | if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) | 494 | if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) |
| 487 | return -EFAULT; | 495 | return -EFAULT; |
| 488 | 496 | ||
| @@ -507,10 +515,13 @@ static int kvm_riscv_vcpu_general_get_csr(struct kvm_vcpu *vcpu, · 18 unmodified lines | |||
| 507 | unsigned long *out_val) | 515 | unsigned long *out_val) |
| 508 | { | 516 | { |
| 509 | struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; | 517 | struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; |
| 518 | unsigned long regs_max = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); | ||
| 510 | 519 | ||
| 511 | if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) | 520 | if (reg_num >= regs_max) |
| 512 | return -ENOENT; | 521 | return -ENOENT; |
| 513 | 522 | ||
| 523 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 524 | |||
| 514 | if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { | 525 | if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { |
| 515 | kvm_riscv_vcpu_flush_interrupts(vcpu); | 526 | kvm_riscv_vcpu_flush_interrupts(vcpu); |
| 516 | *out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK; | 527 | *out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK; |
| @@ -526,10 +537,13 @@ static int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu, · 9 unmodified lines | |||
| 526 | unsigned long reg_val) | 537 | unsigned long reg_val) |
| 527 | { | 538 | { |
| 528 | struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; | 539 | struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; |
| 540 | unsigned long regs_max = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); | ||
| 529 | 541 | ||
| 530 | if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) | 542 | if (reg_num >= regs_max) |
| 531 | return -ENOENT; | 543 | return -ENOENT; |
| 532 | 544 | ||
| 545 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 546 | |||
| 533 | if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { | 547 | if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { |
| 534 | reg_val &= VSIP_VALID_MASK; | 548 | reg_val &= VSIP_VALID_MASK; |
| 535 | reg_val <<= VSIP_TO_HVIP_SHIFT; | 549 | reg_val <<= VSIP_TO_HVIP_SHIFT; |
| @@ -548,10 +562,15 @@ static inline int kvm_riscv_vcpu_smstateen_set_csr(struct kvm_vcpu *vcpu, · 12 unmodified lines | |||
| 548 | unsigned long reg_val) | 562 | unsigned long reg_val) |
| 549 | { | 563 | { |
| 550 | struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr; | 564 | struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr; |
| 565 | unsigned long regs_max = sizeof(struct kvm_riscv_smstateen_csr) / | ||
| 566 | sizeof(unsigned long); | ||
| 551 | 567 | ||
| 552 | if (reg_num >= sizeof(struct kvm_riscv_smstateen_csr) / | 568 | if (!riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN)) |
| 553 | sizeof(unsigned long)) | 569 | return -ENOENT; |
| 554 | return -EINVAL; | 570 | if (reg_num >= regs_max) |
| 571 | return -ENOENT; | ||
| 572 | |||
| 573 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 555 | 574 | ||
| 556 | ((unsigned long *)csr)[reg_num] = reg_val; | 575 | ((unsigned long *)csr)[reg_num] = reg_val; |
| 557 | return 0; | 576 | return 0; |
| @@ -562,10 +581,15 @@ static int kvm_riscv_vcpu_smstateen_get_csr(struct kvm_vcpu *vcpu, · 4 unmodified lines | |||
| 562 | unsigned long *out_val) | 581 | unsigned long *out_val) |
| 563 | { | 582 | { |
| 564 | struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr; | 583 | struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr; |
| 584 | unsigned long regs_max = sizeof(struct kvm_riscv_smstateen_csr) / | ||
| 585 | sizeof(unsigned long); | ||
| 565 | 586 | ||
| 566 | if (reg_num >= sizeof(struct kvm_riscv_smstateen_csr) / | 587 | if (!riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN)) |
| 567 | sizeof(unsigned long)) | 588 | return -ENOENT; |
| 568 | return -EINVAL; | 589 | if (reg_num >= regs_max) |
| 590 | return -ENOENT; | ||
| 591 | |||
| 592 | reg_num = array_index_nospec(reg_num, regs_max); | ||
| 569 | 593 | ||
| 570 | *out_val = ((unsigned long *)csr)[reg_num]; | 594 | *out_val = ((unsigned long *)csr)[reg_num]; |
| 571 | return 0; | 595 | return 0; |
| @@ -595,10 +619,7 @@ static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu, · 23 unmodified lines | |||
| 595 | rc = kvm_riscv_vcpu_aia_get_csr(vcpu, reg_num, ®_val); | 619 | rc = kvm_riscv_vcpu_aia_get_csr(vcpu, reg_num, ®_val); |
| 596 | break; | 620 | break; |
| 597 | case KVM_REG_RISCV_CSR_SMSTATEEN: | 621 | case KVM_REG_RISCV_CSR_SMSTATEEN: |
| 598 | rc = -EINVAL; | 622 | rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num, ®_val); |
| 599 | if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) | ||
| 600 | rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num, | ||
| 601 | ®_val); | ||
| 602 | break; | 623 | break; |
| 603 | default: | 624 | default: |
| 604 | rc = -ENOENT; | 625 | rc = -ENOENT; |
| @@ -640,10 +661,7 @@ static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu, · 35 unmodified lines | |||
| 640 | rc = kvm_riscv_vcpu_aia_set_csr(vcpu, reg_num, reg_val); | 661 | rc = kvm_riscv_vcpu_aia_set_csr(vcpu, reg_num, reg_val); |
| 641 | break; | 662 | break; |
| 642 | case KVM_REG_RISCV_CSR_SMSTATEEN: | 663 | case KVM_REG_RISCV_CSR_SMSTATEEN: |
| 643 | rc = -EINVAL; | 664 | rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num, reg_val); |
| 644 | if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) | ||
| 645 | rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num, | ||
| 646 | reg_val); | ||
| 647 | break; | 665 | break; |
| 648 | default: | 666 | default: |
| 649 | rc = -ENOENT; | 667 | rc = -ENOENT; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <linux/errno.h> | 10 | #include <linux/errno.h> |
| 11 | #include <linux/err.h> | 11 | #include <linux/err.h> |
| 12 | #include <linux/kvm_host.h> | 12 | #include <linux/kvm_host.h> |
| 13 | #include <linux/nospec.h> | ||
| 13 | #include <linux/perf/riscv_pmu.h> | 14 | #include <linux/perf/riscv_pmu.h> |
| 14 | #include <asm/csr.h> | 15 | #include <asm/csr.h> |
| 15 | #include <asm/kvm_vcpu_sbi.h> | 16 | #include <asm/kvm_vcpu_sbi.h> |
| @@ -87,7 +88,8 @@ static void kvm_pmu_release_perf_event(struct kvm_pmc *pmc) · 71 unmodified lines | |||
| 87 | 88 | ||
| 88 | static u64 kvm_pmu_get_perf_event_hw_config(u32 sbi_event_code) | 89 | static u64 kvm_pmu_get_perf_event_hw_config(u32 sbi_event_code) |
| 89 | { | 90 | { |
| 90 | return hw_event_perf_map[sbi_event_code]; | 91 | return hw_event_perf_map[array_index_nospec(sbi_event_code, |
| 92 | SBI_PMU_HW_GENERAL_MAX)]; | ||
| 91 | } | 93 | } |
| 92 | 94 | ||
| 93 | static u64 kvm_pmu_get_perf_event_cache_config(u32 sbi_event_code) | 95 | static u64 kvm_pmu_get_perf_event_cache_config(u32 sbi_event_code) |
| @@ -218,6 +220,7 @@ static int pmu_fw_ctr_read_hi(struct kvm_vcpu *vcpu, unsigned long cidx, · 124 unmodified lines | |||
| 218 | return -EINVAL; | 220 | return -EINVAL; |
| 219 | } | 221 | } |
| 220 | 222 | ||
| 223 | cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS); | ||
| 221 | pmc = &kvpmu->pmc[cidx]; | 224 | pmc = &kvpmu->pmc[cidx]; |
| 222 | 225 | ||
| 223 | if (pmc->cinfo.type != SBI_PMU_CTR_TYPE_FW) | 226 | if (pmc->cinfo.type != SBI_PMU_CTR_TYPE_FW) |
| @@ -244,6 +247,7 @@ static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx, · 20 unmodified lines | |||
| 244 | return -EINVAL; | 247 | return -EINVAL; |
| 245 | } | 248 | } |
| 246 | 249 | ||
| 250 | cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS); | ||
| 247 | pmc = &kvpmu->pmc[cidx]; | 251 | pmc = &kvpmu->pmc[cidx]; |
| 248 | 252 | ||
| 249 | if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) { | 253 | if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) { |
| @@ -520,11 +524,12 @@ int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx, · 270 unmodified lines | |||
| 520 | { | 524 | { |
| 521 | struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu); | 525 | struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu); |
| 522 | 526 | ||
| 523 | if (cidx > RISCV_KVM_MAX_COUNTERS || cidx == 1) { | 527 | if (cidx >= RISCV_KVM_MAX_COUNTERS || cidx == 1) { |
| 524 | retdata->err_val = SBI_ERR_INVALID_PARAM; | 528 | retdata->err_val = SBI_ERR_INVALID_PARAM; |
| 525 | return 0; | 529 | return 0; |
| 526 | } | 530 | } |
| 527 | 531 | ||
| 532 | cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS); | ||
| 528 | retdata->out_val = kvpmu->pmc[cidx].cinfo.value; | 533 | retdata->out_val = kvpmu->pmc[cidx].cinfo.value; |
| 529 | 534 | ||
| 530 | return 0; | 535 | return 0; |
| @@ -559,7 +564,8 @@ int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base, · 28 unmodified lines | |||
| 559 | } | 564 | } |
| 560 | /* Start the counters that have been configured and requested by the guest */ | 565 | /* Start the counters that have been configured and requested by the guest */ |
| 561 | for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) { | 566 | for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) { |
| 562 | pmc_index = i + ctr_base; | 567 | pmc_index = array_index_nospec(i + ctr_base, |
| 568 | RISCV_KVM_MAX_COUNTERS); | ||
| 563 | if (!test_bit(pmc_index, kvpmu->pmc_in_use)) | 569 | if (!test_bit(pmc_index, kvpmu->pmc_in_use)) |
| 564 | continue; | 570 | continue; |
| 565 | /* The guest started the counter again. Reset the overflow status */ | 571 | /* The guest started the counter again. Reset the overflow status */ |
| @@ -630,7 +636,8 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base, · 64 unmodified lines | |||
| 630 | 636 | ||
| 631 | /* Stop the counters that have been configured and requested by the guest */ | 637 | /* Stop the counters that have been configured and requested by the guest */ |
| 632 | for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) { | 638 | for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) { |
| 633 | pmc_index = i + ctr_base; | 639 | pmc_index = array_index_nospec(i + ctr_base, |
| 640 | RISCV_KVM_MAX_COUNTERS); | ||
| 634 | if (!test_bit(pmc_index, kvpmu->pmc_in_use)) | 641 | if (!test_bit(pmc_index, kvpmu->pmc_in_use)) |
| 635 | continue; | 642 | continue; |
| 636 | pmc = &kvpmu->pmc[pmc_index]; | 643 | pmc = &kvpmu->pmc[pmc_index]; |
| @@ -761,6 +768,7 @@ int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_ba · 124 unmodified lines | |||
| 761 | } | 768 | } |
| 762 | } | 769 | } |
| 763 | 770 | ||
| 771 | ctr_idx = array_index_nospec(ctr_idx, RISCV_KVM_MAX_COUNTERS); | ||
| 764 | pmc = &kvpmu->pmc[ctr_idx]; | 772 | pmc = &kvpmu->pmc[ctr_idx]; |
| 765 | pmc->idx = ctr_idx; | 773 | pmc->idx = ctr_idx; |
| 766 | 774 | ||
| @@ -13,7 +13,7 @@ · 12 unmodified lines | |||
| 13 | #include <linux/kvm_host.h> | 13 | #include <linux/kvm_host.h> |
| 14 | #include <asm/kvm_mmu.h> | 14 | #include <asm/kvm_mmu.h> |
| 15 | 15 | ||
| 16 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 16 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 17 | KVM_GENERIC_VM_STATS() | 17 | KVM_GENERIC_VM_STATS() |
| 18 | }; | 18 | }; |
| 19 | static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == | 19 | static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == |
| @@ -62,8 +62,8 @@ do { \ · 61 unmodified lines | |||
| 62 | * @size: number of elements in array | 62 | * @size: number of elements in array |
| 63 | */ | 63 | */ |
| 64 | #define array_index_mask_nospec array_index_mask_nospec | 64 | #define array_index_mask_nospec array_index_mask_nospec |
| 65 | static inline unsigned long array_index_mask_nospec(unsigned long index, | 65 | static __always_inline unsigned long array_index_mask_nospec(unsigned long index, |
| 66 | unsigned long size) | 66 | unsigned long size) |
| 67 | { | 67 | { |
| 68 | unsigned long mask; | 68 | unsigned long mask; |
| 69 | 69 | ||
| @@ -710,6 +710,9 @@ void kvm_arch_crypto_clear_masks(struct kvm *kvm); · 709 unmodified lines | |||
| 710 | void kvm_arch_crypto_set_masks(struct kvm *kvm, unsigned long *apm, | 710 | void kvm_arch_crypto_set_masks(struct kvm *kvm, unsigned long *apm, |
| 711 | unsigned long *aqm, unsigned long *adm); | 711 | unsigned long *aqm, unsigned long *adm); |
| 712 | 712 | ||
| 713 | #define SIE64_RETURN_NORMAL 0 | ||
| 714 | #define SIE64_RETURN_MCCK 1 | ||
| 715 | |||
| 713 | int __sie64a(phys_addr_t sie_block_phys, struct kvm_s390_sie_block *sie_block, u64 *rsa, | 716 | int __sie64a(phys_addr_t sie_block_phys, struct kvm_s390_sie_block *sie_block, u64 *rsa, |
| 714 | unsigned long gasce); | 717 | unsigned long gasce); |
| 715 | 718 | ||
| @@ -62,7 +62,7 @@ struct stack_frame { · 61 unmodified lines | |||
| 62 | struct { | 62 | struct { |
| 63 | unsigned long sie_control_block; | 63 | unsigned long sie_control_block; |
| 64 | unsigned long sie_savearea; | 64 | unsigned long sie_savearea; |
| 65 | unsigned long sie_reason; | 65 | unsigned long sie_return; |
| 66 | unsigned long sie_flags; | 66 | unsigned long sie_flags; |
| 67 | unsigned long sie_control_block_phys; | 67 | unsigned long sie_control_block_phys; |
| 68 | unsigned long sie_guest_asce; | 68 | unsigned long sie_guest_asce; |
| @@ -63,7 +63,7 @@ int main(void) · 62 unmodified lines | |||
| 63 | OFFSET(__SF_EMPTY, stack_frame, empty[0]); | 63 | OFFSET(__SF_EMPTY, stack_frame, empty[0]); |
| 64 | OFFSET(__SF_SIE_CONTROL, stack_frame, sie_control_block); | 64 | OFFSET(__SF_SIE_CONTROL, stack_frame, sie_control_block); |
| 65 | OFFSET(__SF_SIE_SAVEAREA, stack_frame, sie_savearea); | 65 | OFFSET(__SF_SIE_SAVEAREA, stack_frame, sie_savearea); |
| 66 | OFFSET(__SF_SIE_REASON, stack_frame, sie_reason); | 66 | OFFSET(__SF_SIE_RETURN, stack_frame, sie_return); |
| 67 | OFFSET(__SF_SIE_FLAGS, stack_frame, sie_flags); | 67 | OFFSET(__SF_SIE_FLAGS, stack_frame, sie_flags); |
| 68 | OFFSET(__SF_SIE_CONTROL_PHYS, stack_frame, sie_control_block_phys); | 68 | OFFSET(__SF_SIE_CONTROL_PHYS, stack_frame, sie_control_block_phys); |
| 69 | OFFSET(__SF_SIE_GUEST_ASCE, stack_frame, sie_guest_asce); | 69 | OFFSET(__SF_SIE_GUEST_ASCE, stack_frame, sie_guest_asce); |
| @@ -200,7 +200,7 @@ SYM_FUNC_START(__sie64a) · 199 unmodified lines | |||
| 200 | stg %r3,__SF_SIE_CONTROL(%r15) # ...and virtual addresses | 200 | stg %r3,__SF_SIE_CONTROL(%r15) # ...and virtual addresses |
| 201 | stg %r4,__SF_SIE_SAVEAREA(%r15) # save guest register save area | 201 | stg %r4,__SF_SIE_SAVEAREA(%r15) # save guest register save area |
| 202 | stg %r5,__SF_SIE_GUEST_ASCE(%r15) # save guest asce | 202 | stg %r5,__SF_SIE_GUEST_ASCE(%r15) # save guest asce |
| 203 | xc __SF_SIE_REASON(8,%r15),__SF_SIE_REASON(%r15) # reason code = 0 | 203 | xc __SF_SIE_RETURN(8,%r15),__SF_SIE_RETURN(%r15) # return code = 0 |
| 204 | mvc __SF_SIE_FLAGS(8,%r15),__TI_flags(%r14) # copy thread flags | 204 | mvc __SF_SIE_FLAGS(8,%r15),__TI_flags(%r14) # copy thread flags |
| 205 | lmg %r0,%r13,0(%r4) # load guest gprs 0-13 | 205 | lmg %r0,%r13,0(%r4) # load guest gprs 0-13 |
| 206 | mvi __TI_sie(%r14),1 | 206 | mvi __TI_sie(%r14),1 |
| @@ -237,7 +237,7 @@ SYM_INNER_LABEL(sie_exit, SYM_L_GLOBAL) · 30 unmodified lines | |||
| 237 | xgr %r4,%r4 | 237 | xgr %r4,%r4 |
| 238 | xgr %r5,%r5 | 238 | xgr %r5,%r5 |
| 239 | lmg %r6,%r14,__SF_GPRS(%r15) # restore kernel registers | 239 | lmg %r6,%r14,__SF_GPRS(%r15) # restore kernel registers |
| 240 | lg %r2,__SF_SIE_REASON(%r15) # return exit reason code | 240 | lg %r2,__SF_SIE_RETURN(%r15) # return sie return code |
| 241 | BR_EX %r14 | 241 | BR_EX %r14 |
| 242 | SYM_FUNC_END(__sie64a) | 242 | SYM_FUNC_END(__sie64a) |
| 243 | EXPORT_SYMBOL(__sie64a) | 243 | EXPORT_SYMBOL(__sie64a) |
| @@ -271,6 +271,7 @@ SYM_CODE_START(system_call) · 27 unmodified lines | |||
| 271 | xgr %r9,%r9 | 271 | xgr %r9,%r9 |
| 272 | xgr %r10,%r10 | 272 | xgr %r10,%r10 |
| 273 | xgr %r11,%r11 | 273 | xgr %r11,%r11 |
| 274 | xgr %r12,%r12 | ||
| 274 | la %r2,STACK_FRAME_OVERHEAD(%r15) # pointer to pt_regs | 275 | la %r2,STACK_FRAME_OVERHEAD(%r15) # pointer to pt_regs |
| 275 | mvc __PT_R8(64,%r2),__LC_SAVE_AREA(%r13) | 276 | mvc __PT_R8(64,%r2),__LC_SAVE_AREA(%r13) |
| 276 | MBEAR %r2,%r13 | 277 | MBEAR %r2,%r13 |
| @@ -407,6 +408,7 @@ SYM_CODE_START(\name) · 130 unmodified lines | |||
| 407 | xgr %r6,%r6 | 408 | xgr %r6,%r6 |
| 408 | xgr %r7,%r7 | 409 | xgr %r7,%r7 |
| 409 | xgr %r10,%r10 | 410 | xgr %r10,%r10 |
| 411 | xgr %r12,%r12 | ||
| 410 | xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11) | 412 | xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11) |
| 411 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA(%r13) | 413 | mvc __PT_R8(64,%r11),__LC_SAVE_AREA(%r13) |
| 412 | MBEAR %r11,%r13 | 414 | MBEAR %r11,%r13 |
| @@ -496,6 +498,7 @@ SYM_CODE_START(mcck_int_handler) · 83 unmodified lines | |||
| 496 | xgr %r6,%r6 | 498 | xgr %r6,%r6 |
| 497 | xgr %r7,%r7 | 499 | xgr %r7,%r7 |
| 498 | xgr %r10,%r10 | 500 | xgr %r10,%r10 |
| 501 | xgr %r12,%r12 | ||
| 499 | stmg %r8,%r9,__PT_PSW(%r11) | 502 | stmg %r8,%r9,__PT_PSW(%r11) |
| 500 | xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11) | 503 | xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11) |
| 501 | xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15) | 504 | xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15) |
| @@ -147,10 +147,8 @@ void noinstr do_io_irq(struct pt_regs *regs) · 146 unmodified lines | |||
| 147 | bool from_idle; | 147 | bool from_idle; |
| 148 | 148 | ||
| 149 | from_idle = test_and_clear_cpu_flag(CIF_ENABLED_WAIT); | 149 | from_idle = test_and_clear_cpu_flag(CIF_ENABLED_WAIT); |
| 150 | if (from_idle) { | 150 | if (from_idle) |
| 151 | update_timer_idle(); | 151 | update_timer_idle(); |
| 152 | regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | ||
| 153 | } | ||
| 154 | 152 | ||
| 155 | irq_enter_rcu(); | 153 | irq_enter_rcu(); |
| 156 | 154 | ||
| @@ -176,6 +174,9 @@ void noinstr do_io_irq(struct pt_regs *regs) · 19 unmodified lines | |||
| 176 | 174 | ||
| 177 | set_irq_regs(old_regs); | 175 | set_irq_regs(old_regs); |
| 178 | irqentry_exit(regs, state); | 176 | irqentry_exit(regs, state); |
| 177 | |||
| 178 | if (from_idle) | ||
| 179 | regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | ||
| 179 | } | 180 | } |
| 180 | 181 | ||
| 181 | void noinstr do_ext_irq(struct pt_regs *regs) | 182 | void noinstr do_ext_irq(struct pt_regs *regs) |
| @@ -185,10 +186,8 @@ void noinstr do_ext_irq(struct pt_regs *regs) · 3 unmodified lines | |||
| 185 | bool from_idle; | 186 | bool from_idle; |
| 186 | 187 | ||
| 187 | from_idle = test_and_clear_cpu_flag(CIF_ENABLED_WAIT); | 188 | from_idle = test_and_clear_cpu_flag(CIF_ENABLED_WAIT); |
| 188 | if (from_idle) { | 189 | if (from_idle) |
| 189 | update_timer_idle(); | 190 | update_timer_idle(); |
| 190 | regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | ||
| 191 | } | ||
| 192 | 191 | ||
| 193 | irq_enter_rcu(); | 192 | irq_enter_rcu(); |
| 194 | 193 | ||
| @@ -210,6 +209,9 @@ void noinstr do_ext_irq(struct pt_regs *regs) · 15 unmodified lines | |||
| 210 | irq_exit_rcu(); | 209 | irq_exit_rcu(); |
| 211 | set_irq_regs(old_regs); | 210 | set_irq_regs(old_regs); |
| 212 | irqentry_exit(regs, state); | 211 | irqentry_exit(regs, state); |
| 212 | |||
| 213 | if (from_idle) | ||
| 214 | regs->psw.mask &= ~(PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_WAIT); | ||
| 213 | } | 215 | } |
| 214 | 216 | ||
| 215 | static void show_msi_interrupt(struct seq_file *p, int irq) | 217 | static void show_msi_interrupt(struct seq_file *p, int irq) |
| @@ -487,8 +487,8 @@ void notrace s390_do_machine_check(struct pt_regs *regs) · 486 unmodified lines | |||
| 487 | mcck_dam_code = (mci.val & MCIC_SUBCLASS_MASK); | 487 | mcck_dam_code = (mci.val & MCIC_SUBCLASS_MASK); |
| 488 | if (test_cpu_flag(CIF_MCCK_GUEST) && | 488 | if (test_cpu_flag(CIF_MCCK_GUEST) && |
| 489 | (mcck_dam_code & MCCK_CODE_NO_GUEST) != mcck_dam_code) { | 489 | (mcck_dam_code & MCCK_CODE_NO_GUEST) != mcck_dam_code) { |
| 490 | /* Set exit reason code for host's later handling */ | 490 | /* Set sie return code for host's later handling */ |
| 491 | *((long *)(regs->gprs[15] + __SF_SIE_REASON)) = -EINTR; | 491 | ((struct stack_frame *)regs->gprs[15])->sie_return = SIE64_RETURN_MCCK; |
| 492 | } | 492 | } |
| 493 | clear_cpu_flag(CIF_MCCK_GUEST); | 493 | clear_cpu_flag(CIF_MCCK_GUEST); |
| 494 | 494 | ||
| @@ -1168,6 +1168,7 @@ static void hw_collect_samples(struct perf_event *event, unsigned long *sdbt, · 1167 unmodified lines | |||
| 1168 | static void hw_perf_event_update(struct perf_event *event, int flush_all) | 1168 | static void hw_perf_event_update(struct perf_event *event, int flush_all) |
| 1169 | { | 1169 | { |
| 1170 | unsigned long long event_overflow, sampl_overflow, num_sdb; | 1170 | unsigned long long event_overflow, sampl_overflow, num_sdb; |
| 1171 | struct cpu_hw_sf *cpuhw = this_cpu_ptr(&cpu_hw_sf); | ||
| 1171 | struct hw_perf_event *hwc = &event->hw; | 1172 | struct hw_perf_event *hwc = &event->hw; |
| 1172 | union hws_trailer_header prev, new; | 1173 | union hws_trailer_header prev, new; |
| 1173 | struct hws_trailer_entry *te; | 1174 | struct hws_trailer_entry *te; |
| @@ -1247,8 +1248,11 @@ static void hw_perf_event_update(struct perf_event *event, int flush_all) · 73 unmodified lines | |||
| 1247 | * are dropped. | 1248 | * are dropped. |
| 1248 | * Slightly increase the interval to avoid hitting this limit. | 1249 | * Slightly increase the interval to avoid hitting this limit. |
| 1249 | */ | 1250 | */ |
| 1250 | if (event_overflow) | 1251 | if (event_overflow) { |
| 1251 | SAMPL_RATE(hwc) += DIV_ROUND_UP(SAMPL_RATE(hwc), 10); | 1252 | SAMPL_RATE(hwc) += DIV_ROUND_UP(SAMPL_RATE(hwc), 10); |
| 1253 | if (SAMPL_RATE(hwc) > cpuhw->qsi.max_sampl_rate) | ||
| 1254 | SAMPL_RATE(hwc) = cpuhw->qsi.max_sampl_rate; | ||
| 1255 | } | ||
| 1252 | } | 1256 | } |
| 1253 | 1257 | ||
| 1254 | static inline unsigned long aux_sdb_index(struct aux_buffer *aux, | 1258 | static inline unsigned long aux_sdb_index(struct aux_buffer *aux, |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | |||
| 13 | */ | 13 | */ |
| 14 | 14 | ||
| 15 | #include <linux/cpufeature.h> | 15 | #include <linux/cpufeature.h> |
| 16 | #include <linux/nospec.h> | ||
| 16 | #include <linux/errno.h> | 17 | #include <linux/errno.h> |
| 17 | #include <linux/sched.h> | 18 | #include <linux/sched.h> |
| 18 | #include <linux/mm.h> | 19 | #include <linux/mm.h> |
| @@ -131,8 +132,10 @@ void noinstr __do_syscall(struct pt_regs *regs, int per_trap) · 112 unmodified lines | |||
| 131 | if (unlikely(test_and_clear_pt_regs_flag(regs, PIF_SYSCALL_RET_SET))) | 132 | if (unlikely(test_and_clear_pt_regs_flag(regs, PIF_SYSCALL_RET_SET))) |
| 132 | goto out; | 133 | goto out; |
| 133 | regs->gprs[2] = -ENOSYS; | 134 | regs->gprs[2] = -ENOSYS; |
| 134 | if (likely(nr < NR_syscalls)) | 135 | if (likely(nr < NR_syscalls)) { |
| 136 | nr = array_index_nospec(nr, NR_syscalls); | ||
| 135 | regs->gprs[2] = sys_call_table[nr](regs); | 137 | regs->gprs[2] = sys_call_table[nr](regs); |
| 138 | } | ||
| 136 | out: | 139 | out: |
| 137 | syscall_exit_to_user_mode(regs); | 140 | syscall_exit_to_user_mode(regs); |
| 138 | } | 141 | } |
| @@ -134,32 +134,6 @@ int dat_set_asce_limit(struct kvm_s390_mmu_cache *mc, union asce *asce, int newt · 133 unmodified lines | |||
| 134 | return 0; | 134 | return 0; |
| 135 | } | 135 | } |
| 136 | 136 | ||
| 137 | /** | ||
| 138 | * dat_crstep_xchg() - Exchange a gmap CRSTE with another. | ||
| 139 | * @crstep: Pointer to the CRST entry | ||
| 140 | * @new: Replacement entry. | ||
| 141 | * @gfn: The affected guest address. | ||
| 142 | * @asce: The ASCE of the address space. | ||
| 143 | * | ||
| 144 | * Context: This function is assumed to be called with kvm->mmu_lock held. | ||
| 145 | */ | ||
| 146 | void dat_crstep_xchg(union crste *crstep, union crste new, gfn_t gfn, union asce asce) | ||
| 147 | { | ||
| 148 | if (crstep->h.i) { | ||
| 149 | WRITE_ONCE(*crstep, new); | ||
| 150 | return; | ||
| 151 | } else if (cpu_has_edat2()) { | ||
| 152 | crdte_crste(crstep, *crstep, new, gfn, asce); | ||
| 153 | return; | ||
| 154 | } | ||
| 155 | |||
| 156 | if (machine_has_tlb_guest()) | ||
| 157 | idte_crste(crstep, gfn, IDTE_GUEST_ASCE, asce, IDTE_GLOBAL); | ||
| 158 | else | ||
| 159 | idte_crste(crstep, gfn, 0, NULL_ASCE, IDTE_GLOBAL); | ||
| 160 | WRITE_ONCE(*crstep, new); | ||
| 161 | } | ||
| 162 | |||
| 163 | /** | 137 | /** |
| 164 | * dat_crstep_xchg_atomic() - Atomically exchange a gmap CRSTE with another. | 138 | * dat_crstep_xchg_atomic() - Atomically exchange a gmap CRSTE with another. |
| 165 | * @crstep: Pointer to the CRST entry. | 139 | * @crstep: Pointer to the CRST entry. |
| @@ -175,8 +149,8 @@ void dat_crstep_xchg(union crste *crstep, union crste new, gfn_t gfn, union asce · 9 unmodified lines | |||
| 175 | * | 149 | * |
| 176 | * Return: %true if the exchange was successful. | 150 | * Return: %true if the exchange was successful. |
| 177 | */ | 151 | */ |
| 178 | bool dat_crstep_xchg_atomic(union crste *crstep, union crste old, union crste new, gfn_t gfn, | 152 | bool __must_check dat_crstep_xchg_atomic(union crste *crstep, union crste old, union crste new, |
| 179 | union asce asce) | 153 | gfn_t gfn, union asce asce) |
| 180 | { | 154 | { |
| 181 | if (old.h.i) | 155 | if (old.h.i) |
| 182 | return arch_try_cmpxchg((long *)crstep, &old.val, new.val); | 156 | return arch_try_cmpxchg((long *)crstep, &old.val, new.val); |
| @@ -292,6 +266,7 @@ static int dat_split_ste(struct kvm_s390_mmu_cache *mc, union pmd *pmdp, gfn_t g · 109 unmodified lines | |||
| 292 | pt->ptes[i].val = init.val | i * PAGE_SIZE; | 266 | pt->ptes[i].val = init.val | i * PAGE_SIZE; |
| 293 | /* No need to take locks as the page table is not installed yet. */ | 267 | /* No need to take locks as the page table is not installed yet. */ |
| 294 | pgste_init.prefix_notif = old.s.fc1.prefix_notif; | 268 | pgste_init.prefix_notif = old.s.fc1.prefix_notif; |
| 269 | pgste_init.vsie_notif = old.s.fc1.vsie_notif; | ||
| 295 | pgste_init.pcl = uses_skeys && init.h.i; | 270 | pgste_init.pcl = uses_skeys && init.h.i; |
| 296 | dat_init_pgstes(pt, pgste_init.val); | 271 | dat_init_pgstes(pt, pgste_init.val); |
| 297 | } else { | 272 | } else { |
| @@ -893,7 +868,8 @@ static long _dat_slot_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct d · 595 unmodified lines | |||
| 893 | 868 | ||
| 894 | /* This table entry needs to be updated. */ | 869 | /* This table entry needs to be updated. */ |
| 895 | if (walk->start <= gfn && walk->end >= next) { | 870 | if (walk->start <= gfn && walk->end >= next) { |
| 896 | dat_crstep_xchg_atomic(crstep, crste, new_crste, gfn, walk->asce); | 871 | if (!dat_crstep_xchg_atomic(crstep, crste, new_crste, gfn, walk->asce)) |
| 872 | return -EINVAL; | ||
| 897 | /* A lower level table was present, needs to be freed. */ | 873 | /* A lower level table was present, needs to be freed. */ |
| 898 | if (!crste.h.fc && !crste.h.i) { | 874 | if (!crste.h.fc && !crste.h.i) { |
| 899 | if (is_pmd(crste)) | 875 | if (is_pmd(crste)) |
| @@ -1021,67 +997,21 @@ bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end) · 121 unmodified lines | |||
| 1021 | return _dat_walk_gfn_range(start, end, asce, &test_age_ops, 0, NULL) > 0; | 997 | return _dat_walk_gfn_range(start, end, asce, &test_age_ops, 0, NULL) > 0; |
| 1022 | } | 998 | } |
| 1023 | 999 | ||
| 1024 | int dat_link(struct kvm_s390_mmu_cache *mc, union asce asce, int level, | ||
| 1025 | bool uses_skeys, struct guest_fault *f) | ||
| 1026 | { | ||
| 1027 | union crste oldval, newval; | ||
| 1028 | union pte newpte, oldpte; | ||
| 1029 | union pgste pgste; | ||
| 1030 | int rc = 0; | ||
| 1031 | |||
| 1032 | rc = dat_entry_walk(mc, f->gfn, asce, DAT_WALK_ALLOC_CONTINUE, level, &f->crstep, &f->ptep); | ||
| 1033 | if (rc == -EINVAL || rc == -ENOMEM) | ||
| 1034 | return rc; | ||
| 1035 | if (rc) | ||
| 1036 | return -EAGAIN; | ||
| 1037 | |||
| 1038 | if (WARN_ON_ONCE(unlikely(get_level(f->crstep, f->ptep) > level))) | ||
| 1039 | return -EINVAL; | ||
| 1040 | |||
| 1041 | if (f->ptep) { | ||
| 1042 | pgste = pgste_get_lock(f->ptep); | ||
| 1043 | oldpte = *f->ptep; | ||
| 1044 | newpte = _pte(f->pfn, f->writable, f->write_attempt | oldpte.s.d, !f->page); | ||
| 1045 | newpte.s.sd = oldpte.s.sd; | ||
| 1046 | oldpte.s.sd = 0; | ||
| 1047 | if (oldpte.val == _PTE_EMPTY.val || oldpte.h.pfra == f->pfn) { | ||
| 1048 | pgste = __dat_ptep_xchg(f->ptep, pgste, newpte, f->gfn, asce, uses_skeys); | ||
| 1049 | if (f->callback) | ||
| 1050 | f->callback(f); | ||
| 1051 | } else { | ||
| 1052 | rc = -EAGAIN; | ||
| 1053 | } | ||
| 1054 | pgste_set_unlock(f->ptep, pgste); | ||
| 1055 | } else { | ||
| 1056 | oldval = READ_ONCE(*f->crstep); | ||
| 1057 | newval = _crste_fc1(f->pfn, oldval.h.tt, f->writable, | ||
| 1058 | f->write_attempt | oldval.s.fc1.d); | ||
| 1059 | newval.s.fc1.sd = oldval.s.fc1.sd; | ||
| 1060 | if (oldval.val != _CRSTE_EMPTY(oldval.h.tt).val && | ||
| 1061 | crste_origin_large(oldval) != crste_origin_large(newval)) | ||
| 1062 | return -EAGAIN; | ||
| 1063 | if (!dat_crstep_xchg_atomic(f->crstep, oldval, newval, f->gfn, asce)) | ||
| 1064 | return -EAGAIN; | ||
| 1065 | if (f->callback) | ||
| 1066 | f->callback(f); | ||
| 1067 | } | ||
| 1068 | |||
| 1069 | return rc; | ||
| 1070 | } | ||
| 1071 | |||
| 1072 | static long dat_set_pn_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk) | 1000 | static long dat_set_pn_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk) |
| 1073 | { | 1001 | { |
| 1074 | union crste crste = READ_ONCE(*crstep); | 1002 | union crste newcrste, oldcrste; |
| 1075 | int *n = walk->priv; | 1003 | int *n = walk->priv; |
| 1076 | 1004 | ||
| 1077 | if (!crste.h.fc || crste.h.i || crste.h.p) | 1005 | do { |
| 1078 | return 0; | 1006 | oldcrste = READ_ONCE(*crstep); |
| 1079 | 1007 | if (!oldcrste.h.fc || oldcrste.h.i || oldcrste.h.p) | |
| 1008 | return 0; | ||
| 1009 | if (oldcrste.s.fc1.prefix_notif) | ||
| 1010 | break; | ||
| 1011 | newcrste = oldcrste; | ||
| 1012 | newcrste.s.fc1.prefix_notif = 1; | ||
| 1013 | } while (!dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, walk->asce)); | ||
| 1080 | *n = 2; | 1014 | *n = 2; |
| 1081 | if (crste.s.fc1.prefix_notif) | ||
| 1082 | return 0; | ||
| 1083 | crste.s.fc1.prefix_notif = 1; | ||
| 1084 | dat_crstep_xchg(crstep, crste, gfn, walk->asce); | ||
| 1085 | return 0; | 1015 | return 0; |
| 1086 | } | 1016 | } |
| 1087 | 1017 | ||
| @@ -160,14 +160,14 @@ union pmd { · 159 unmodified lines | |||
| 160 | unsigned long :44; /* HW */ | 160 | unsigned long :44; /* HW */ |
| 161 | unsigned long : 3; /* Unused */ | 161 | unsigned long : 3; /* Unused */ |
| 162 | unsigned long : 1; /* HW */ | 162 | unsigned long : 1; /* HW */ |
| 163 | unsigned long s : 1; /* Special */ | ||
| 163 | unsigned long w : 1; /* Writable soft-bit */ | 164 | unsigned long w : 1; /* Writable soft-bit */ |
| 164 | unsigned long r : 1; /* Readable soft-bit */ | 165 | unsigned long r : 1; /* Readable soft-bit */ |
| 165 | unsigned long d : 1; /* Dirty */ | 166 | unsigned long d : 1; /* Dirty */ |
| 166 | unsigned long y : 1; /* Young */ | 167 | unsigned long y : 1; /* Young */ |
| 167 | unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | ||
| 168 | unsigned long : 3; /* HW */ | 168 | unsigned long : 3; /* HW */ |
| 169 | unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | ||
| 169 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ | 170 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ |
| 170 | unsigned long : 1; /* Unused */ | ||
| 171 | unsigned long : 4; /* HW */ | 171 | unsigned long : 4; /* HW */ |
| 172 | unsigned long sd : 1; /* Soft-Dirty */ | 172 | unsigned long sd : 1; /* Soft-Dirty */ |
| 173 | unsigned long pr : 1; /* Present */ | 173 | unsigned long pr : 1; /* Present */ |
| @@ -183,14 +183,14 @@ union pud { · 9 unmodified lines | |||
| 183 | unsigned long :33; /* HW */ | 183 | unsigned long :33; /* HW */ |
| 184 | unsigned long :14; /* Unused */ | 184 | unsigned long :14; /* Unused */ |
| 185 | unsigned long : 1; /* HW */ | 185 | unsigned long : 1; /* HW */ |
| 186 | unsigned long s : 1; /* Special */ | ||
| 186 | unsigned long w : 1; /* Writable soft-bit */ | 187 | unsigned long w : 1; /* Writable soft-bit */ |
| 187 | unsigned long r : 1; /* Readable soft-bit */ | 188 | unsigned long r : 1; /* Readable soft-bit */ |
| 188 | unsigned long d : 1; /* Dirty */ | 189 | unsigned long d : 1; /* Dirty */ |
| 189 | unsigned long y : 1; /* Young */ | 190 | unsigned long y : 1; /* Young */ |
| 190 | unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | ||
| 191 | unsigned long : 3; /* HW */ | 191 | unsigned long : 3; /* HW */ |
| 192 | unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | ||
| 192 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ | 193 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ |
| 193 | unsigned long : 1; /* Unused */ | ||
| 194 | unsigned long : 4; /* HW */ | 194 | unsigned long : 4; /* HW */ |
| 195 | unsigned long sd : 1; /* Soft-Dirty */ | 195 | unsigned long sd : 1; /* Soft-Dirty */ |
| 196 | unsigned long pr : 1; /* Present */ | 196 | unsigned long pr : 1; /* Present */ |
| @@ -254,14 +254,14 @@ union crste { · 57 unmodified lines | |||
| 254 | struct { | 254 | struct { |
| 255 | unsigned long :47; | 255 | unsigned long :47; |
| 256 | unsigned long : 1; /* HW (should be 0) */ | 256 | unsigned long : 1; /* HW (should be 0) */ |
| 257 | unsigned long s : 1; /* Special */ | ||
| 257 | unsigned long w : 1; /* Writable */ | 258 | unsigned long w : 1; /* Writable */ |
| 258 | unsigned long r : 1; /* Readable */ | 259 | unsigned long r : 1; /* Readable */ |
| 259 | unsigned long d : 1; /* Dirty */ | 260 | unsigned long d : 1; /* Dirty */ |
| 260 | unsigned long y : 1; /* Young */ | 261 | unsigned long y : 1; /* Young */ |
| 261 | unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | ||
| 262 | unsigned long : 3; /* HW */ | 262 | unsigned long : 3; /* HW */ |
| 263 | unsigned long prefix_notif : 1; /* Guest prefix invalidation notification */ | ||
| 263 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ | 264 | unsigned long vsie_notif : 1; /* Referenced in a shadow table */ |
| 264 | unsigned long : 1; | ||
| 265 | unsigned long : 4; /* HW */ | 265 | unsigned long : 4; /* HW */ |
| 266 | unsigned long sd : 1; /* Soft-Dirty */ | 266 | unsigned long sd : 1; /* Soft-Dirty */ |
| 267 | unsigned long pr : 1; /* Present */ | 267 | unsigned long pr : 1; /* Present */ |
| @@ -540,8 +540,6 @@ int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gf · 272 unmodified lines | |||
| 540 | u16 type, u16 param); | 540 | u16 type, u16 param); |
| 541 | int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn); | 541 | int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn); |
| 542 | bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end); | 542 | bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end); |
| 543 | int dat_link(struct kvm_s390_mmu_cache *mc, union asce asce, int level, | ||
| 544 | bool uses_skeys, struct guest_fault *f); | ||
| 545 | 543 | ||
| 546 | int dat_perform_essa(union asce asce, gfn_t gfn, int orc, union essa_state *state, bool *dirty); | 544 | int dat_perform_essa(union asce asce, gfn_t gfn, int orc, union essa_state *state, bool *dirty); |
| 547 | long dat_reset_cmma(union asce asce, gfn_t start_gfn); | 545 | long dat_reset_cmma(union asce asce, gfn_t start_gfn); |
| @@ -938,11 +936,14 @@ static inline bool dat_pudp_xchg_atomic(union pud *pudp, union pud old, union pu · 390 unmodified lines | |||
| 938 | return dat_crstep_xchg_atomic(_CRSTEP(pudp), _CRSTE(old), _CRSTE(new), gfn, asce); | 936 | return dat_crstep_xchg_atomic(_CRSTEP(pudp), _CRSTE(old), _CRSTE(new), gfn, asce); |
| 939 | } | 937 | } |
| 940 | 938 | ||
| 941 | static inline void dat_crstep_clear(union crste *crstep, gfn_t gfn, union asce asce) | 939 | static inline union crste dat_crstep_clear_atomic(union crste *crstep, gfn_t gfn, union asce asce) |
| 942 | { | 940 | { |
| 943 | union crste newcrste = _CRSTE_EMPTY(crstep->h.tt); | 941 | union crste oldcrste, empty = _CRSTE_EMPTY(crstep->h.tt); |
| 944 | 942 | ||
| 945 | dat_crstep_xchg(crstep, newcrste, gfn, asce); | 943 | do { |
| 944 | oldcrste = READ_ONCE(*crstep); | ||
| 945 | } while (!dat_crstep_xchg_atomic(crstep, oldcrste, empty, gfn, asce)); | ||
| 946 | return oldcrste; | ||
| 946 | } | 947 | } |
| 947 | 948 | ||
| 948 | static inline int get_level(union crste *crstep, union pte *ptep) | 949 | static inline int get_level(union crste *crstep, union pte *ptep) |
| @@ -1434,17 +1434,27 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union · 1433 unmodified lines | |||
| 1434 | if (rc) | 1434 | if (rc) |
| 1435 | return rc; | 1435 | return rc; |
| 1436 | 1436 | ||
| 1437 | pgste = pgste_get_lock(ptep_h); | 1437 | if (!pgste_get_trylock(ptep_h, &pgste)) |
| 1438 | newpte = _pte(f->pfn, f->writable, !p, 0); | 1438 | return -EAGAIN; |
| 1439 | newpte.s.d |= ptep->s.d; | 1439 | newpte = _pte(f->pfn, f->writable, !p, ptep_h->s.s); |
| 1440 | newpte.s.sd |= ptep->s.sd; | 1440 | newpte.s.d |= ptep_h->s.d; |
| 1441 | newpte.h.p &= ptep->h.p; | 1441 | newpte.s.sd |= ptep_h->s.sd; |
| 1442 | pgste = _gmap_ptep_xchg(sg->parent, ptep_h, newpte, pgste, f->gfn, false); | 1442 | newpte.h.p &= ptep_h->h.p; |
| 1443 | pgste.vsie_notif = 1; | 1443 | if (!newpte.h.p && !f->writable) { |
| 1444 | rc = -EOPNOTSUPP; | ||
| 1445 | } else { | ||
| 1446 | pgste = _gmap_ptep_xchg(sg->parent, ptep_h, newpte, pgste, f->gfn, false); | ||
| 1447 | pgste.vsie_notif = 1; | ||
| 1448 | } | ||
| 1444 | pgste_set_unlock(ptep_h, pgste); | 1449 | pgste_set_unlock(ptep_h, pgste); |
| 1450 | if (rc) | ||
| 1451 | return rc; | ||
| 1452 | if (!sg->parent) | ||
| 1453 | return -EAGAIN; | ||
| 1445 | 1454 | ||
| 1446 | newpte = _pte(f->pfn, 0, !p, 0); | 1455 | newpte = _pte(f->pfn, 0, !p, 0); |
| 1447 | pgste = pgste_get_lock(ptep); | 1456 | if (!pgste_get_trylock(ptep, &pgste)) |
| 1457 | return -EAGAIN; | ||
| 1448 | pgste = __dat_ptep_xchg(ptep, pgste, newpte, gpa_to_gfn(raddr), sg->asce, uses_skeys(sg)); | 1458 | pgste = __dat_ptep_xchg(ptep, pgste, newpte, gpa_to_gfn(raddr), sg->asce, uses_skeys(sg)); |
| 1449 | pgste_set_unlock(ptep, pgste); | 1459 | pgste_set_unlock(ptep, pgste); |
| 1450 | 1460 | ||
| @@ -1454,7 +1464,7 @@ static int _do_shadow_pte(struct gmap *sg, gpa_t raddr, union pte *ptep_h, union · 3 unmodified lines | |||
| 1454 | static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, union crste *table, | 1464 | static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, union crste *table, |
| 1455 | struct guest_fault *f, bool p) | 1465 | struct guest_fault *f, bool p) |
| 1456 | { | 1466 | { |
| 1457 | union crste newcrste; | 1467 | union crste newcrste, oldcrste; |
| 1458 | gfn_t gfn; | 1468 | gfn_t gfn; |
| 1459 | int rc; | 1469 | int rc; |
| 1460 | 1470 | ||
| @@ -1467,16 +1477,28 @@ static int _do_shadow_crste(struct gmap *sg, gpa_t raddr, union crste *host, uni · 6 unmodified lines | |||
| 1467 | if (rc) | 1477 | if (rc) |
| 1468 | return rc; | 1478 | return rc; |
| 1469 | 1479 | ||
| 1470 | newcrste = _crste_fc1(f->pfn, host->h.tt, f->writable, !p); | 1480 | do { |
| 1471 | newcrste.s.fc1.d |= host->s.fc1.d; | 1481 | /* _gmap_crstep_xchg_atomic() could have unshadowed this shadow gmap */ |
| 1472 | newcrste.s.fc1.sd |= host->s.fc1.sd; | 1482 | if (!sg->parent) |
| 1473 | newcrste.h.p &= host->h.p; | 1483 | return -EAGAIN; |
| 1474 | newcrste.s.fc1.vsie_notif = 1; | 1484 | oldcrste = READ_ONCE(*host); |
| 1475 | newcrste.s.fc1.prefix_notif = host->s.fc1.prefix_notif; | 1485 | newcrste = _crste_fc1(f->pfn, oldcrste.h.tt, f->writable, !p); |
| 1476 | _gmap_crstep_xchg(sg->parent, host, newcrste, f->gfn, false); | 1486 | newcrste.s.fc1.d |= oldcrste.s.fc1.d; |
| 1487 | newcrste.s.fc1.sd |= oldcrste.s.fc1.sd; | ||
| 1488 | newcrste.h.p &= oldcrste.h.p; | ||
| 1489 | newcrste.s.fc1.vsie_notif = 1; | ||
| 1490 | newcrste.s.fc1.prefix_notif = oldcrste.s.fc1.prefix_notif; | ||
| 1491 | newcrste.s.fc1.s = oldcrste.s.fc1.s; | ||
| 1492 | if (!newcrste.h.p && !f->writable) | ||
| 1493 | return -EOPNOTSUPP; | ||
| 1494 | } while (!_gmap_crstep_xchg_atomic(sg->parent, host, oldcrste, newcrste, f->gfn, false)); | ||
| 1495 | if (!sg->parent) | ||
| 1496 | return -EAGAIN; | ||
| 1477 | 1497 | ||
| 1478 | newcrste = _crste_fc1(f->pfn, host->h.tt, 0, !p); | 1498 | newcrste = _crste_fc1(f->pfn, oldcrste.h.tt, 0, !p); |
| 1479 | dat_crstep_xchg(table, newcrste, gpa_to_gfn(raddr), sg->asce); | 1499 | gfn = gpa_to_gfn(raddr); |
| 1500 | while (!dat_crstep_xchg_atomic(table, READ_ONCE(*table), newcrste, gfn, sg->asce)) | ||
| 1501 | ; | ||
| 1480 | return 0; | 1502 | return 0; |
| 1481 | } | 1503 | } |
| 1482 | 1504 | ||
| @@ -1500,21 +1522,31 @@ static int _gaccess_do_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *sg, · 17 unmodified lines | |||
| 1500 | if (rc) | 1522 | if (rc) |
| 1501 | return rc; | 1523 | return rc; |
| 1502 | 1524 | ||
| 1503 | /* A race occourred. The shadow mapping is already valid, nothing to do */ | 1525 | /* A race occurred. The shadow mapping is already valid, nothing to do */ |
| 1504 | if ((ptep && !ptep->h.i) || (!ptep && crste_leaf(*table))) | 1526 | if ((ptep && !ptep->h.i && ptep->h.p == w->p) || |
| 1527 | (!ptep && crste_leaf(*table) && !table->h.i && table->h.p == w->p)) | ||
| 1505 | return 0; | 1528 | return 0; |
| 1506 | 1529 | ||
| 1507 | gl = get_level(table, ptep); | 1530 | gl = get_level(table, ptep); |
| 1508 | 1531 | ||
| 1532 | /* In case of a real address space */ | ||
| 1533 | if (w->level <= LEVEL_MEM) { | ||
| 1534 | l = TABLE_TYPE_PAGE_TABLE; | ||
| 1535 | hl = TABLE_TYPE_REGION1; | ||
| 1536 | goto real_address_space; | ||
| 1537 | } | ||
| 1538 | |||
| 1509 | /* | 1539 | /* |
| 1510 | * Skip levels that are already protected. For each level, protect | 1540 | * Skip levels that are already protected. For each level, protect |
| 1511 | * only the page containing the entry, not the whole table. | 1541 | * only the page containing the entry, not the whole table. |
| 1512 | */ | 1542 | */ |
| 1513 | for (i = gl ; i >= w->level; i--) { | 1543 | for (i = gl ; i >= w->level; i--) { |
| 1514 | rc = gmap_protect_rmap(mc, sg, entries[i - 1].gfn, gpa_to_gfn(saddr), | 1544 | rc = gmap_protect_rmap(mc, sg, entries[i].gfn, gpa_to_gfn(saddr), |
| 1515 | entries[i - 1].pfn, i, entries[i - 1].writable); | 1545 | entries[i].pfn, i + 1, entries[i].writable); |
| 1516 | if (rc) | 1546 | if (rc) |
| 1517 | return rc; | 1547 | return rc; |
| 1548 | if (!sg->parent) | ||
| 1549 | return -EAGAIN; | ||
| 1518 | } | 1550 | } |
| 1519 | 1551 | ||
| 1520 | rc = dat_entry_walk(NULL, entries[LEVEL_MEM].gfn, sg->parent->asce, DAT_WALK_LEAF, | 1552 | rc = dat_entry_walk(NULL, entries[LEVEL_MEM].gfn, sg->parent->asce, DAT_WALK_LEAF, |
| @@ -1526,6 +1558,7 @@ static int _gaccess_do_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *sg, · 5 unmodified lines | |||
| 1526 | /* Get the smallest granularity */ | 1558 | /* Get the smallest granularity */ |
| 1527 | l = min3(gl, hl, w->level); | 1559 | l = min3(gl, hl, w->level); |
| 1528 | 1560 | ||
| 1561 | real_address_space: | ||
| 1529 | flags = DAT_WALK_SPLIT_ALLOC | (uses_skeys(sg->parent) ? DAT_WALK_USES_SKEYS : 0); | 1562 | flags = DAT_WALK_SPLIT_ALLOC | (uses_skeys(sg->parent) ? DAT_WALK_USES_SKEYS : 0); |
| 1530 | /* If necessary, create the shadow mapping */ | 1563 | /* If necessary, create the shadow mapping */ |
| 1531 | if (l < gl) { | 1564 | if (l < gl) { |
| @@ -313,13 +313,16 @@ static long gmap_clear_young_crste(union crste *crstep, gfn_t gfn, gfn_t end, st · 312 unmodified lines | |||
| 313 | struct clear_young_pte_priv *priv = walk->priv; | 313 | struct clear_young_pte_priv *priv = walk->priv; |
| 314 | union crste crste, new; | 314 | union crste crste, new; |
| 315 | 315 | ||
| 316 | crste = READ_ONCE(*crstep); | 316 | do { |
| 317 | crste = READ_ONCE(*crstep); | ||
| 318 | |||
| 319 | if (!crste.h.fc) | ||
| 320 | return 0; | ||
| 321 | if (!crste.s.fc1.y && crste.h.i) | ||
| 322 | return 0; | ||
| 323 | if (crste_prefix(crste) && !gmap_mkold_prefix(priv->gmap, gfn, end)) | ||
| 324 | break; | ||
| 317 | 325 | ||
| 318 | if (!crste.h.fc) | ||
| 319 | return 0; | ||
| 320 | if (!crste.s.fc1.y && crste.h.i) | ||
| 321 | return 0; | ||
| 322 | if (!crste_prefix(crste) || gmap_mkold_prefix(priv->gmap, gfn, end)) { | ||
| 323 | new = crste; | 326 | new = crste; |
| 324 | new.h.i = 1; | 327 | new.h.i = 1; |
| 325 | new.s.fc1.y = 0; | 328 | new.s.fc1.y = 0; |
| @@ -328,8 +331,8 @@ static long gmap_clear_young_crste(union crste *crstep, gfn_t gfn, gfn_t end, st · 2 unmodified lines | |||
| 328 | folio_set_dirty(phys_to_folio(crste_origin_large(crste))); | 331 | folio_set_dirty(phys_to_folio(crste_origin_large(crste))); |
| 329 | new.s.fc1.d = 0; | 332 | new.s.fc1.d = 0; |
| 330 | new.h.p = 1; | 333 | new.h.p = 1; |
| 331 | dat_crstep_xchg(crstep, new, gfn, walk->asce); | 334 | } while (!dat_crstep_xchg_atomic(crstep, crste, new, gfn, walk->asce)); |
| 332 | } | 335 | |
| 333 | priv->young = 1; | 336 | priv->young = 1; |
| 334 | return 0; | 337 | return 0; |
| 335 | } | 338 | } |
| @@ -391,14 +394,18 @@ static long _gmap_unmap_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct · 55 unmodified lines | |||
| 391 | { | 394 | { |
| 392 | struct gmap_unmap_priv *priv = walk->priv; | 395 | struct gmap_unmap_priv *priv = walk->priv; |
| 393 | struct folio *folio = NULL; | 396 | struct folio *folio = NULL; |
| 397 | union crste old = *crstep; | ||
| 394 | 398 | ||
| 395 | if (crstep->h.fc) { | 399 | if (!old.h.fc) |
| 396 | if (crstep->s.fc1.pr && test_bit(GMAP_FLAG_EXPORT_ON_UNMAP, &priv->gmap->flags)) | 400 | return 0; |
| 397 | folio = phys_to_folio(crste_origin_large(*crstep)); | 401 | |
| 398 | gmap_crstep_xchg(priv->gmap, crstep, _CRSTE_EMPTY(crstep->h.tt), gfn); | 402 | if (old.s.fc1.pr && test_bit(GMAP_FLAG_EXPORT_ON_UNMAP, &priv->gmap->flags)) |
| 399 | if (folio) | 403 | folio = phys_to_folio(crste_origin_large(old)); |
| 400 | uv_convert_from_secure_folio(folio); | 404 | /* No races should happen because kvm->mmu_lock is held in write mode */ |
| 401 | } | 405 | KVM_BUG_ON(!gmap_crstep_xchg_atomic(priv->gmap, crstep, old, _CRSTE_EMPTY(old.h.tt), gfn), |
| 406 | priv->gmap->kvm); | ||
| 407 | if (folio) | ||
| 408 | uv_convert_from_secure_folio(folio); | ||
| 402 | 409 | ||
| 403 | return 0; | 410 | return 0; |
| 404 | } | 411 | } |
| @@ -474,23 +481,24 @@ static long _crste_test_and_clear_softdirty(union crste *table, gfn_t gfn, gfn_t · 69 unmodified lines | |||
| 474 | 481 | ||
| 475 | if (fatal_signal_pending(current)) | 482 | if (fatal_signal_pending(current)) |
| 476 | return 1; | 483 | return 1; |
| 477 | crste = READ_ONCE(*table); | 484 | do { |
| 478 | if (!crste.h.fc) | 485 | crste = READ_ONCE(*table); |
| 479 | return 0; | 486 | if (!crste.h.fc) |
| 480 | if (crste.h.p && !crste.s.fc1.sd) | 487 | return 0; |
| 481 | return 0; | 488 | if (crste.h.p && !crste.s.fc1.sd) |
| 489 | return 0; | ||
| 482 | 490 | ||
| 483 | /* | 491 | /* |
| 484 | * If this large page contains one or more prefixes of vCPUs that are | 492 | * If this large page contains one or more prefixes of vCPUs that are |
| 485 | * currently running, do not reset the protection, leave it marked as | 493 | * currently running, do not reset the protection, leave it marked as |
| 486 | * dirty. | 494 | * dirty. |
| 487 | */ | 495 | */ |
| 488 | if (!crste.s.fc1.prefix_notif || gmap_mkold_prefix(gmap, gfn, end)) { | 496 | if (crste.s.fc1.prefix_notif && !gmap_mkold_prefix(gmap, gfn, end)) |
| 497 | break; | ||
| 489 | new = crste; | 498 | new = crste; |
| 490 | new.h.p = 1; | 499 | new.h.p = 1; |
| 491 | new.s.fc1.sd = 0; | 500 | new.s.fc1.sd = 0; |
| 492 | gmap_crstep_xchg(gmap, table, new, gfn); | 501 | } while (!gmap_crstep_xchg_atomic(gmap, table, crste, new, gfn)); |
| 493 | } | ||
| 494 | 502 | ||
| 495 | for ( ; gfn < end; gfn++) | 503 | for ( ; gfn < end; gfn++) |
| 496 | mark_page_dirty(gmap->kvm, gfn); | 504 | mark_page_dirty(gmap->kvm, gfn); |
| @@ -511,7 +519,7 @@ void gmap_sync_dirty_log(struct gmap *gmap, gfn_t start, gfn_t end) · 14 unmodified lines | |||
| 511 | _dat_walk_gfn_range(start, end, gmap->asce, &walk_ops, 0, gmap); | 519 | _dat_walk_gfn_range(start, end, gmap->asce, &walk_ops, 0, gmap); |
| 512 | } | 520 | } |
| 513 | 521 | ||
| 514 | static int gmap_handle_minor_crste_fault(union asce asce, struct guest_fault *f) | 522 | static int gmap_handle_minor_crste_fault(struct gmap *gmap, struct guest_fault *f) |
| 515 | { | 523 | { |
| 516 | union crste newcrste, oldcrste = READ_ONCE(*f->crstep); | 524 | union crste newcrste, oldcrste = READ_ONCE(*f->crstep); |
| 517 | 525 | ||
| @@ -536,10 +544,8 @@ static int gmap_handle_minor_crste_fault(union asce asce, struct guest_fault *f) · 18 unmodified lines | |||
| 536 | newcrste.s.fc1.d = 1; | 544 | newcrste.s.fc1.d = 1; |
| 537 | newcrste.s.fc1.sd = 1; | 545 | newcrste.s.fc1.sd = 1; |
| 538 | } | 546 | } |
| 539 | if (!oldcrste.s.fc1.d && newcrste.s.fc1.d) | ||
| 540 | SetPageDirty(phys_to_page(crste_origin_large(newcrste))); | ||
| 541 | /* In case of races, let the slow path deal with it. */ | 547 | /* In case of races, let the slow path deal with it. */ |
| 542 | return !dat_crstep_xchg_atomic(f->crstep, oldcrste, newcrste, f->gfn, asce); | 548 | return !gmap_crstep_xchg_atomic(gmap, f->crstep, oldcrste, newcrste, f->gfn); |
| 543 | } | 549 | } |
| 544 | /* Trying to write on a read-only page, let the slow path deal with it. */ | 550 | /* Trying to write on a read-only page, let the slow path deal with it. */ |
| 545 | return 1; | 551 | return 1; |
| @@ -568,8 +574,6 @@ static int _gmap_handle_minor_pte_fault(struct gmap *gmap, union pgste *pgste, · 22 unmodified lines | |||
| 568 | newpte.s.d = 1; | 574 | newpte.s.d = 1; |
| 569 | newpte.s.sd = 1; | 575 | newpte.s.sd = 1; |
| 570 | } | 576 | } |
| 571 | if (!oldpte.s.d && newpte.s.d) | ||
| 572 | SetPageDirty(pfn_to_page(newpte.h.pfra)); | ||
| 573 | *pgste = gmap_ptep_xchg(gmap, f->ptep, newpte, *pgste, f->gfn); | 577 | *pgste = gmap_ptep_xchg(gmap, f->ptep, newpte, *pgste, f->gfn); |
| 574 | 578 | ||
| 575 | return 0; | 579 | return 0; |
| @@ -606,7 +610,7 @@ int gmap_try_fixup_minor(struct gmap *gmap, struct guest_fault *fault) · 30 unmodified lines | |||
| 606 | fault->callback(fault); | 610 | fault->callback(fault); |
| 607 | pgste_set_unlock(fault->ptep, pgste); | 611 | pgste_set_unlock(fault->ptep, pgste); |
| 608 | } else { | 612 | } else { |
| 609 | rc = gmap_handle_minor_crste_fault(gmap->asce, fault); | 613 | rc = gmap_handle_minor_crste_fault(gmap, fault); |
| 610 | if (!rc && fault->callback) | 614 | if (!rc && fault->callback) |
| 611 | fault->callback(fault); | 615 | fault->callback(fault); |
| 612 | } | 616 | } |
| @@ -623,10 +627,61 @@ static inline bool gmap_1m_allowed(struct gmap *gmap, gfn_t gfn) · 10 unmodified lines | |||
| 623 | return test_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &gmap->flags); | 627 | return test_bit(GMAP_FLAG_ALLOW_HPAGE_1M, &gmap->flags); |
| 624 | } | 628 | } |
| 625 | 629 | ||
| 630 | static int _gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, int level, | ||
| 631 | struct guest_fault *f) | ||
| 632 | { | ||
| 633 | union crste oldval, newval; | ||
| 634 | union pte newpte, oldpte; | ||
| 635 | union pgste pgste; | ||
| 636 | int rc = 0; | ||
| 637 | |||
| 638 | rc = dat_entry_walk(mc, f->gfn, gmap->asce, DAT_WALK_ALLOC_CONTINUE, level, | ||
| 639 | &f->crstep, &f->ptep); | ||
| 640 | if (rc == -ENOMEM) | ||
| 641 | return rc; | ||
| 642 | if (KVM_BUG_ON(rc == -EINVAL, gmap->kvm)) | ||
| 643 | return rc; | ||
| 644 | if (rc) | ||
| 645 | return -EAGAIN; | ||
| 646 | if (KVM_BUG_ON(get_level(f->crstep, f->ptep) > level, gmap->kvm)) | ||
| 647 | return -EINVAL; | ||
| 648 | |||
| 649 | if (f->ptep) { | ||
| 650 | pgste = pgste_get_lock(f->ptep); | ||
| 651 | oldpte = *f->ptep; | ||
| 652 | newpte = _pte(f->pfn, f->writable, f->write_attempt | oldpte.s.d, !f->page); | ||
| 653 | newpte.s.sd = oldpte.s.sd; | ||
| 654 | oldpte.s.sd = 0; | ||
| 655 | if (oldpte.val == _PTE_EMPTY.val || oldpte.h.pfra == f->pfn) { | ||
| 656 | pgste = gmap_ptep_xchg(gmap, f->ptep, newpte, pgste, f->gfn); | ||
| 657 | if (f->callback) | ||
| 658 | f->callback(f); | ||
| 659 | } else { | ||
| 660 | rc = -EAGAIN; | ||
| 661 | } | ||
| 662 | pgste_set_unlock(f->ptep, pgste); | ||
| 663 | } else { | ||
| 664 | do { | ||
| 665 | oldval = READ_ONCE(*f->crstep); | ||
| 666 | newval = _crste_fc1(f->pfn, oldval.h.tt, f->writable, | ||
| 667 | f->write_attempt | oldval.s.fc1.d); | ||
| 668 | newval.s.fc1.s = !f->page; | ||
| 669 | newval.s.fc1.sd = oldval.s.fc1.sd; | ||
| 670 | if (oldval.val != _CRSTE_EMPTY(oldval.h.tt).val && | ||
| 671 | crste_origin_large(oldval) != crste_origin_large(newval)) | ||
| 672 | return -EAGAIN; | ||
| 673 | } while (!gmap_crstep_xchg_atomic(gmap, f->crstep, oldval, newval, f->gfn)); | ||
| 674 | if (f->callback) | ||
| 675 | f->callback(f); | ||
| 676 | } | ||
| 677 | |||
| 678 | return rc; | ||
| 679 | } | ||
| 680 | |||
| 626 | int gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, struct guest_fault *f) | 681 | int gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, struct guest_fault *f) |
| 627 | { | 682 | { |
| 628 | unsigned int order; | 683 | unsigned int order; |
| 629 | int rc, level; | 684 | int level; |
| 630 | 685 | ||
| 631 | lockdep_assert_held(&gmap->kvm->mmu_lock); | 686 | lockdep_assert_held(&gmap->kvm->mmu_lock); |
| 632 | 687 | ||
| @@ -638,16 +693,14 @@ int gmap_link(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, struct guest_fau · 5 unmodified lines | |||
| 638 | else if (order >= get_order(_SEGMENT_SIZE) && gmap_1m_allowed(gmap, f->gfn)) | 693 | else if (order >= get_order(_SEGMENT_SIZE) && gmap_1m_allowed(gmap, f->gfn)) |
| 639 | level = TABLE_TYPE_SEGMENT; | 694 | level = TABLE_TYPE_SEGMENT; |
| 640 | } | 695 | } |
| 641 | rc = dat_link(mc, gmap->asce, level, uses_skeys(gmap), f); | 696 | return _gmap_link(mc, gmap, level, f); |
| 642 | KVM_BUG_ON(rc == -EINVAL, gmap->kvm); | ||
| 643 | return rc; | ||
| 644 | } | 697 | } |
| 645 | 698 | ||
| 646 | static int gmap_ucas_map_one(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, | 699 | static int gmap_ucas_map_one(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, |
| 647 | gfn_t p_gfn, gfn_t c_gfn, bool force_alloc) | 700 | gfn_t p_gfn, gfn_t c_gfn, bool force_alloc) |
| 648 | { | 701 | { |
| 702 | union crste newcrste, oldcrste; | ||
| 649 | struct page_table *pt; | 703 | struct page_table *pt; |
| 650 | union crste newcrste; | ||
| 651 | union crste *crstep; | 704 | union crste *crstep; |
| 652 | union pte *ptep; | 705 | union pte *ptep; |
| 653 | int rc; | 706 | int rc; |
| @@ -673,7 +726,11 @@ static int gmap_ucas_map_one(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, · 19 unmodified lines | |||
| 673 | &crstep, &ptep); | 726 | &crstep, &ptep); |
| 674 | if (rc) | 727 | if (rc) |
| 675 | return rc; | 728 | return rc; |
| 676 | dat_crstep_xchg(crstep, newcrste, c_gfn, gmap->asce); | 729 | do { |
| 730 | oldcrste = READ_ONCE(*crstep); | ||
| 731 | if (oldcrste.val == newcrste.val) | ||
| 732 | break; | ||
| 733 | } while (!dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, c_gfn, gmap->asce)); | ||
| 677 | return 0; | 734 | return 0; |
| 678 | } | 735 | } |
| 679 | 736 | ||
| @@ -777,8 +834,10 @@ static void gmap_ucas_unmap_one(struct gmap *gmap, gfn_t c_gfn) · 97 unmodified lines | |||
| 777 | int rc; | 834 | int rc; |
| 778 | 835 | ||
| 779 | rc = dat_entry_walk(NULL, c_gfn, gmap->asce, 0, TABLE_TYPE_SEGMENT, &crstep, &ptep); | 836 | rc = dat_entry_walk(NULL, c_gfn, gmap->asce, 0, TABLE_TYPE_SEGMENT, &crstep, &ptep); |
| 780 | if (!rc) | 837 | if (rc) |
| 781 | dat_crstep_xchg(crstep, _PMD_EMPTY, c_gfn, gmap->asce); | 838 | return; |
| 839 | while (!dat_crstep_xchg_atomic(crstep, READ_ONCE(*crstep), _PMD_EMPTY, c_gfn, gmap->asce)) | ||
| 840 | ; | ||
| 782 | } | 841 | } |
| 783 | 842 | ||
| 784 | void gmap_ucas_unmap(struct gmap *gmap, gfn_t c_gfn, unsigned long count) | 843 | void gmap_ucas_unmap(struct gmap *gmap, gfn_t c_gfn, unsigned long count) |
| @@ -1017,8 +1076,8 @@ static void gmap_unshadow_level(struct gmap *sg, gfn_t r_gfn, int level) · 232 unmodified lines | |||
| 1017 | dat_ptep_xchg(ptep, _PTE_EMPTY, r_gfn, sg->asce, uses_skeys(sg)); | 1076 | dat_ptep_xchg(ptep, _PTE_EMPTY, r_gfn, sg->asce, uses_skeys(sg)); |
| 1018 | return; | 1077 | return; |
| 1019 | } | 1078 | } |
| 1020 | crste = READ_ONCE(*crstep); | 1079 | |
| 1021 | dat_crstep_clear(crstep, r_gfn, sg->asce); | 1080 | crste = dat_crstep_clear_atomic(crstep, r_gfn, sg->asce); |
| 1022 | if (crste_leaf(crste) || crste.h.i) | 1081 | if (crste_leaf(crste) || crste.h.i) |
| 1023 | return; | 1082 | return; |
| 1024 | if (is_pmd(crste)) | 1083 | if (is_pmd(crste)) |
| @@ -1101,6 +1160,7 @@ struct gmap_protect_asce_top_level { · 76 unmodified lines | |||
| 1101 | static inline int __gmap_protect_asce_top_level(struct kvm_s390_mmu_cache *mc, struct gmap *sg, | 1160 | static inline int __gmap_protect_asce_top_level(struct kvm_s390_mmu_cache *mc, struct gmap *sg, |
| 1102 | struct gmap_protect_asce_top_level *context) | 1161 | struct gmap_protect_asce_top_level *context) |
| 1103 | { | 1162 | { |
| 1163 | struct gmap *parent; | ||
| 1104 | int rc, i; | 1164 | int rc, i; |
| 1105 | 1165 | ||
| 1106 | guard(write_lock)(&sg->kvm->mmu_lock); | 1166 | guard(write_lock)(&sg->kvm->mmu_lock); |
| @@ -1108,7 +1168,12 @@ static inline int __gmap_protect_asce_top_level(struct kvm_s390_mmu_cache *mc, s · 1 unmodified line | |||
| 1108 | if (kvm_s390_array_needs_retry_safe(sg->kvm, context->seq, context->f)) | 1168 | if (kvm_s390_array_needs_retry_safe(sg->kvm, context->seq, context->f)) |
| 1109 | return -EAGAIN; | 1169 | return -EAGAIN; |
| 1110 | 1170 | ||
| 1111 | scoped_guard(spinlock, &sg->parent->children_lock) { | 1171 | parent = READ_ONCE(sg->parent); |
| 1172 | if (!parent) | ||
| 1173 | return -EAGAIN; | ||
| 1174 | scoped_guard(spinlock, &parent->children_lock) { | ||
| 1175 | if (READ_ONCE(sg->parent) != parent) | ||
| 1176 | return -EAGAIN; | ||
| 1112 | for (i = 0; i < CRST_TABLE_PAGES; i++) { | 1177 | for (i = 0; i < CRST_TABLE_PAGES; i++) { |
| 1113 | if (!context->f[i].valid) | 1178 | if (!context->f[i].valid) |
| 1114 | continue; | 1179 | continue; |
| @@ -1191,6 +1256,9 @@ struct gmap *gmap_create_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *pare · 76 unmodified lines | |||
| 1191 | struct gmap *sg, *new; | 1256 | struct gmap *sg, *new; |
| 1192 | int rc; | 1257 | int rc; |
| 1193 | 1258 | ||
| 1259 | if (WARN_ON(!parent)) | ||
| 1260 | return ERR_PTR(-EINVAL); | ||
| 1261 | |||
| 1194 | scoped_guard(spinlock, &parent->children_lock) { | 1262 | scoped_guard(spinlock, &parent->children_lock) { |
| 1195 | sg = gmap_find_shadow(parent, asce, edat_level); | 1263 | sg = gmap_find_shadow(parent, asce, edat_level); |
| 1196 | if (sg) { | 1264 | if (sg) { |
| @@ -185,6 +185,8 @@ static inline union pgste _gmap_ptep_xchg(struct gmap *gmap, union pte *ptep, un · 184 unmodified lines | |||
| 185 | else | 185 | else |
| 186 | _gmap_handle_vsie_unshadow_event(gmap, gfn); | 186 | _gmap_handle_vsie_unshadow_event(gmap, gfn); |
| 187 | } | 187 | } |
| 188 | if (!ptep->s.d && newpte.s.d && !newpte.s.s) | ||
| 189 | SetPageDirty(pfn_to_page(newpte.h.pfra)); | ||
| 188 | return __dat_ptep_xchg(ptep, pgste, newpte, gfn, gmap->asce, uses_skeys(gmap)); | 190 | return __dat_ptep_xchg(ptep, pgste, newpte, gfn, gmap->asce, uses_skeys(gmap)); |
| 189 | } | 191 | } |
| 190 | 192 | ||
| @@ -194,35 +196,42 @@ static inline union pgste gmap_ptep_xchg(struct gmap *gmap, union pte *ptep, uni · 3 unmodified lines | |||
| 194 | return _gmap_ptep_xchg(gmap, ptep, newpte, pgste, gfn, true); | 196 | return _gmap_ptep_xchg(gmap, ptep, newpte, pgste, gfn, true); |
| 195 | } | 197 | } |
| 196 | 198 | ||
| 197 | static inline void _gmap_crstep_xchg(struct gmap *gmap, union crste *crstep, union crste ne, | 199 | static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, union crste *crstep, |
| 198 | gfn_t gfn, bool needs_lock) | 200 | union crste oldcrste, union crste newcrste, |
| 201 | gfn_t gfn, bool needs_lock) | ||
| 199 | { | 202 | { |
| 200 | unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11); | 203 | unsigned long align = is_pmd(newcrste) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES; |
| 204 | |||
| 205 | if (KVM_BUG_ON(crstep->h.tt != oldcrste.h.tt || newcrste.h.tt != oldcrste.h.tt, gmap->kvm)) | ||
| 206 | return true; | ||
| 201 | 207 | ||
| 202 | lockdep_assert_held(&gmap->kvm->mmu_lock); | 208 | lockdep_assert_held(&gmap->kvm->mmu_lock); |
| 203 | if (!needs_lock) | 209 | if (!needs_lock) |
| 204 | lockdep_assert_held(&gmap->children_lock); | 210 | lockdep_assert_held(&gmap->children_lock); |
| 205 | 211 | ||
| 206 | gfn = ALIGN_DOWN(gfn, align); | 212 | gfn = ALIGN_DOWN(gfn, align); |
| 207 | if (crste_prefix(*crstep) && (ne.h.p || ne.h.i || !crste_prefix(ne))) { | 213 | if (crste_prefix(oldcrste) && (newcrste.h.p || newcrste.h.i || !crste_prefix(newcrste))) { |
| 208 | ne.s.fc1.prefix_notif = 0; | 214 | newcrste.s.fc1.prefix_notif = 0; |
| 209 | gmap_unmap_prefix(gmap, gfn, gfn + align); | 215 | gmap_unmap_prefix(gmap, gfn, gfn + align); |
| 210 | } | 216 | } |
| 211 | if (crste_leaf(*crstep) && crstep->s.fc1.vsie_notif && | 217 | if (crste_leaf(oldcrste) && oldcrste.s.fc1.vsie_notif && |
| 212 | (ne.h.p || ne.h.i || !ne.s.fc1.vsie_notif)) { | 218 | (newcrste.h.p || newcrste.h.i || !newcrste.s.fc1.vsie_notif)) { |
| 213 | ne.s.fc1.vsie_notif = 0; | 219 | newcrste.s.fc1.vsie_notif = 0; |
| 214 | if (needs_lock) | 220 | if (needs_lock) |
| 215 | gmap_handle_vsie_unshadow_event(gmap, gfn); | 221 | gmap_handle_vsie_unshadow_event(gmap, gfn); |
| 216 | else | 222 | else |
| 217 | _gmap_handle_vsie_unshadow_event(gmap, gfn); | 223 | _gmap_handle_vsie_unshadow_event(gmap, gfn); |
| 218 | } | 224 | } |
| 219 | dat_crstep_xchg(crstep, ne, gfn, gmap->asce); | 225 | if (!oldcrste.s.fc1.d && newcrste.s.fc1.d && !newcrste.s.fc1.s) |
| 226 | SetPageDirty(phys_to_page(crste_origin_large(newcrste))); | ||
| 227 | return dat_crstep_xchg_atomic(crstep, oldcrste, newcrste, gfn, gmap->asce); | ||
| 220 | } | 228 | } |
| 221 | 229 | ||
| 222 | static inline void gmap_crstep_xchg(struct gmap *gmap, union crste *crstep, union crste ne, | 230 | static inline bool __must_check gmap_crstep_xchg_atomic(struct gmap *gmap, union crste *crstep, |
| 223 | gfn_t gfn) | 231 | union crste oldcrste, union crste newcrste, |
| 232 | gfn_t gfn) | ||
| 224 | { | 233 | { |
| 225 | return _gmap_crstep_xchg(gmap, crstep, ne, gfn, true); | 234 | return _gmap_crstep_xchg_atomic(gmap, crstep, oldcrste, newcrste, gfn, true); |
| 226 | } | 235 | } |
| 227 | 236 | ||
| 228 | /** | 237 | /** |
| @@ -2724,6 +2724,9 @@ static unsigned long get_ind_bit(__u64 addr, unsigned long bit_nr, bool swap) · 2723 unmodified lines | |||
| 2724 | 2724 | ||
| 2725 | bit = bit_nr + (addr % PAGE_SIZE) * 8; | 2725 | bit = bit_nr + (addr % PAGE_SIZE) * 8; |
| 2726 | 2726 | ||
| 2727 | /* kvm_set_routing_entry() should never allow this to happen */ | ||
| 2728 | WARN_ON_ONCE(bit > (PAGE_SIZE * BITS_PER_BYTE - 1)); | ||
| 2729 | |||
| 2727 | return swap ? (bit ^ (BITS_PER_LONG - 1)) : bit; | 2730 | return swap ? (bit ^ (BITS_PER_LONG - 1)) : bit; |
| 2728 | } | 2731 | } |
| 2729 | 2732 | ||
| @@ -2824,6 +2827,12 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, · 94 unmodified lines | |||
| 2824 | int rc; | 2827 | int rc; |
| 2825 | 2828 | ||
| 2826 | mci.val = mcck_info->mcic; | 2829 | mci.val = mcck_info->mcic; |
| 2830 | |||
| 2831 | /* log machine checks being reinjected on all debugs */ | ||
| 2832 | VCPU_EVENT(vcpu, 2, "guest machine check %lx", mci.val); | ||
| 2833 | KVM_EVENT(2, "guest machine check %lx", mci.val); | ||
| 2834 | pr_info("guest machine check pid %d: %lx", current->pid, mci.val); | ||
| 2835 | |||
| 2827 | if (mci.sr) | 2836 | if (mci.sr) |
| 2828 | cr14 |= CR14_RECOVERY_SUBMASK; | 2837 | cr14 |= CR14_RECOVERY_SUBMASK; |
| 2829 | if (mci.dg) | 2838 | if (mci.dg) |
| @@ -2852,6 +2861,7 @@ int kvm_set_routing_entry(struct kvm *kvm, · 22 unmodified lines | |||
| 2852 | struct kvm_kernel_irq_routing_entry *e, | 2861 | struct kvm_kernel_irq_routing_entry *e, |
| 2853 | const struct kvm_irq_routing_entry *ue) | 2862 | const struct kvm_irq_routing_entry *ue) |
| 2854 | { | 2863 | { |
| 2864 | const struct kvm_irq_routing_s390_adapter *adapter; | ||
| 2855 | u64 uaddr_s, uaddr_i; | 2865 | u64 uaddr_s, uaddr_i; |
| 2856 | int idx; | 2866 | int idx; |
| 2857 | 2867 | ||
| @@ -2862,6 +2872,14 @@ int kvm_set_routing_entry(struct kvm *kvm, · 4 unmodified lines | |||
| 2862 | return -EINVAL; | 2872 | return -EINVAL; |
| 2863 | e->set = set_adapter_int; | 2873 | e->set = set_adapter_int; |
| 2864 | 2874 | ||
| 2875 | adapter = &ue->u.adapter; | ||
| 2876 | if (adapter->summary_addr + (adapter->summary_offset / 8) >= | ||
| 2877 | (adapter->summary_addr & PAGE_MASK) + PAGE_SIZE) | ||
| 2878 | return -EINVAL; | ||
| 2879 | if (adapter->ind_addr + (adapter->ind_offset / 8) >= | ||
| 2880 | (adapter->ind_addr & PAGE_MASK) + PAGE_SIZE) | ||
| 2881 | return -EINVAL; | ||
| 2882 | |||
| 2865 | idx = srcu_read_lock(&kvm->srcu); | 2883 | idx = srcu_read_lock(&kvm->srcu); |
| 2866 | uaddr_s = gpa_to_hva(kvm, ue->u.adapter.summary_addr); | 2884 | uaddr_s = gpa_to_hva(kvm, ue->u.adapter.summary_addr); |
| 2867 | uaddr_i = gpa_to_hva(kvm, ue->u.adapter.ind_addr); | 2885 | uaddr_i = gpa_to_hva(kvm, ue->u.adapter.ind_addr); |
| @@ -65,7 +65,7 @@ · 64 unmodified lines | |||
| 65 | #define VCPU_IRQS_MAX_BUF (sizeof(struct kvm_s390_irq) * \ | 65 | #define VCPU_IRQS_MAX_BUF (sizeof(struct kvm_s390_irq) * \ |
| 66 | (KVM_MAX_VCPUS + LOCAL_IRQS)) | 66 | (KVM_MAX_VCPUS + LOCAL_IRQS)) |
| 67 | 67 | ||
| 68 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 68 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 69 | KVM_GENERIC_VM_STATS(), | 69 | KVM_GENERIC_VM_STATS(), |
| 70 | STATS_DESC_COUNTER(VM, inject_io), | 70 | STATS_DESC_COUNTER(VM, inject_io), |
| 71 | STATS_DESC_COUNTER(VM, inject_float_mchk), | 71 | STATS_DESC_COUNTER(VM, inject_float_mchk), |
| @@ -91,7 +91,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 19 unmodified lines | |||
| 91 | sizeof(kvm_vm_stats_desc), | 91 | sizeof(kvm_vm_stats_desc), |
| 92 | }; | 92 | }; |
| 93 | 93 | ||
| 94 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 94 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 95 | KVM_GENERIC_VCPU_STATS(), | 95 | KVM_GENERIC_VCPU_STATS(), |
| 96 | STATS_DESC_COUNTER(VCPU, exit_userspace), | 96 | STATS_DESC_COUNTER(VCPU, exit_userspace), |
| 97 | STATS_DESC_COUNTER(VCPU, exit_null), | 97 | STATS_DESC_COUNTER(VCPU, exit_null), |
| @@ -4617,7 +4617,7 @@ static int vcpu_post_run_handle_fault(struct kvm_vcpu *vcpu) · 4519 unmodified lines | |||
| 4617 | return 0; | 4617 | return 0; |
| 4618 | } | 4618 | } |
| 4619 | 4619 | ||
| 4620 | static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason) | 4620 | static int vcpu_post_run(struct kvm_vcpu *vcpu, int sie_return) |
| 4621 | { | 4621 | { |
| 4622 | struct mcck_volatile_info *mcck_info; | 4622 | struct mcck_volatile_info *mcck_info; |
| 4623 | struct sie_page *sie_page; | 4623 | struct sie_page *sie_page; |
| @@ -4633,14 +4633,14 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason) · 9 unmodified lines | |||
| 4633 | vcpu->run->s.regs.gprs[14] = vcpu->arch.sie_block->gg14; | 4633 | vcpu->run->s.regs.gprs[14] = vcpu->arch.sie_block->gg14; |
| 4634 | vcpu->run->s.regs.gprs[15] = vcpu->arch.sie_block->gg15; | 4634 | vcpu->run->s.regs.gprs[15] = vcpu->arch.sie_block->gg15; |
| 4635 | 4635 | ||
| 4636 | if (exit_reason == -EINTR) { | 4636 | if (sie_return == SIE64_RETURN_MCCK) { |
| 4637 | VCPU_EVENT(vcpu, 3, "%s", "machine check"); | ||
| 4638 | sie_page = container_of(vcpu->arch.sie_block, | 4637 | sie_page = container_of(vcpu->arch.sie_block, |
| 4639 | struct sie_page, sie_block); | 4638 | struct sie_page, sie_block); |
| 4640 | mcck_info = &sie_page->mcck_info; | 4639 | mcck_info = &sie_page->mcck_info; |
| 4641 | kvm_s390_reinject_machine_check(vcpu, mcck_info); | 4640 | kvm_s390_reinject_machine_check(vcpu, mcck_info); |
| 4642 | return 0; | 4641 | return 0; |
| 4643 | } | 4642 | } |
| 4643 | WARN_ON_ONCE(sie_return != SIE64_RETURN_NORMAL); | ||
| 4644 | 4644 | ||
| 4645 | if (vcpu->arch.sie_block->icptcode > 0) { | 4645 | if (vcpu->arch.sie_block->icptcode > 0) { |
| 4646 | rc = kvm_handle_sie_intercept(vcpu); | 4646 | rc = kvm_handle_sie_intercept(vcpu); |
| @@ -4679,7 +4679,7 @@ int noinstr kvm_s390_enter_exit_sie(struct kvm_s390_sie_block *scb, · 32 unmodified lines | |||
| 4679 | #define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK) | 4679 | #define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK) |
| 4680 | static int __vcpu_run(struct kvm_vcpu *vcpu) | 4680 | static int __vcpu_run(struct kvm_vcpu *vcpu) |
| 4681 | { | 4681 | { |
| 4682 | int rc, exit_reason; | 4682 | int rc, sie_return; |
| 4683 | struct sie_page *sie_page = (struct sie_page *)vcpu->arch.sie_block; | 4683 | struct sie_page *sie_page = (struct sie_page *)vcpu->arch.sie_block; |
| 4684 | 4684 | ||
| 4685 | /* | 4685 | /* |
| @@ -4719,9 +4719,9 @@ xfer_to_guest_mode_check: · 33 unmodified lines | |||
| 4719 | guest_timing_enter_irqoff(); | 4719 | guest_timing_enter_irqoff(); |
| 4720 | __disable_cpu_timer_accounting(vcpu); | 4720 | __disable_cpu_timer_accounting(vcpu); |
| 4721 | 4721 | ||
| 4722 | exit_reason = kvm_s390_enter_exit_sie(vcpu->arch.sie_block, | 4722 | sie_return = kvm_s390_enter_exit_sie(vcpu->arch.sie_block, |
| 4723 | vcpu->run->s.regs.gprs, | 4723 | vcpu->run->s.regs.gprs, |
| 4724 | vcpu->arch.gmap->asce.val); | 4724 | vcpu->arch.gmap->asce.val); |
| 4725 | 4725 | ||
| 4726 | __enable_cpu_timer_accounting(vcpu); | 4726 | __enable_cpu_timer_accounting(vcpu); |
| 4727 | guest_timing_exit_irqoff(); | 4727 | guest_timing_exit_irqoff(); |
| @@ -4744,7 +4744,7 @@ xfer_to_guest_mode_check: · 16 unmodified lines | |||
| 4744 | } | 4744 | } |
| 4745 | kvm_vcpu_srcu_read_lock(vcpu); | 4745 | kvm_vcpu_srcu_read_lock(vcpu); |
| 4746 | 4746 | ||
| 4747 | rc = vcpu_post_run(vcpu, exit_reason); | 4747 | rc = vcpu_post_run(vcpu, sie_return); |
| 4748 | if (rc || guestdbg_exit_pending(vcpu)) { | 4748 | if (rc || guestdbg_exit_pending(vcpu)) { |
| 4749 | kvm_vcpu_srcu_read_unlock(vcpu); | 4749 | kvm_vcpu_srcu_read_unlock(vcpu); |
| 4750 | break; | 4750 | break; |
| @@ -5520,9 +5520,21 @@ long kvm_arch_vcpu_ioctl(struct file *filp, · 769 unmodified lines | |||
| 5520 | } | 5520 | } |
| 5521 | #endif | 5521 | #endif |
| 5522 | case KVM_S390_VCPU_FAULT: { | 5522 | case KVM_S390_VCPU_FAULT: { |
| 5523 | idx = srcu_read_lock(&vcpu->kvm->srcu); | 5523 | gpa_t gaddr = arg; |
| 5524 | r = vcpu_dat_fault_handler(vcpu, arg, 0); | 5524 | |
| 5525 | srcu_read_unlock(&vcpu->kvm->srcu, idx); | 5525 | scoped_guard(srcu, &vcpu->kvm->srcu) { |
| 5526 | r = vcpu_ucontrol_translate(vcpu, &gaddr); | ||
| 5527 | if (r) | ||
| 5528 | break; | ||
| 5529 | |||
| 5530 | r = kvm_s390_faultin_gfn_simple(vcpu, NULL, gpa_to_gfn(gaddr), false); | ||
| 5531 | if (r == PGM_ADDRESSING) | ||
| 5532 | r = -EFAULT; | ||
| 5533 | if (r <= 0) | ||
| 5534 | break; | ||
| 5535 | r = -EIO; | ||
| 5536 | KVM_BUG_ON(r, vcpu->kvm); | ||
| 5537 | } | ||
| 5526 | break; | 5538 | break; |
| 5527 | } | 5539 | } |
| 5528 | case KVM_ENABLE_CAP: | 5540 | case KVM_ENABLE_CAP: |
| @@ -1122,6 +1122,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page, struc · 1121 unmodified lines | |||
| 1122 | { | 1122 | { |
| 1123 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; | 1123 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; |
| 1124 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; | 1124 | struct kvm_s390_sie_block *scb_o = vsie_page->scb_o; |
| 1125 | unsigned long sie_return = SIE64_RETURN_NORMAL; | ||
| 1125 | int guest_bp_isolation; | 1126 | int guest_bp_isolation; |
| 1126 | int rc = 0; | 1127 | int rc = 0; |
| 1127 | 1128 | ||
| @@ -1163,7 +1164,7 @@ xfer_to_guest_mode_check: · 35 unmodified lines | |||
| 1163 | goto xfer_to_guest_mode_check; | 1164 | goto xfer_to_guest_mode_check; |
| 1164 | } | 1165 | } |
| 1165 | guest_timing_enter_irqoff(); | 1166 | guest_timing_enter_irqoff(); |
| 1166 | rc = kvm_s390_enter_exit_sie(scb_s, vcpu->run->s.regs.gprs, sg->asce.val); | 1167 | sie_return = kvm_s390_enter_exit_sie(scb_s, vcpu->run->s.regs.gprs, sg->asce.val); |
| 1167 | guest_timing_exit_irqoff(); | 1168 | guest_timing_exit_irqoff(); |
| 1168 | local_irq_enable(); | 1169 | local_irq_enable(); |
| 1169 | } | 1170 | } |
| @@ -1178,12 +1179,13 @@ skip_sie: · 8 unmodified lines | |||
| 1178 | 1179 | ||
| 1179 | kvm_vcpu_srcu_read_lock(vcpu); | 1180 | kvm_vcpu_srcu_read_lock(vcpu); |
| 1180 | 1181 | ||
| 1181 | if (rc == -EINTR) { | 1182 | if (sie_return == SIE64_RETURN_MCCK) { |
| 1182 | VCPU_EVENT(vcpu, 3, "%s", "machine check"); | ||
| 1183 | kvm_s390_reinject_machine_check(vcpu, &vsie_page->mcck_info); | 1183 | kvm_s390_reinject_machine_check(vcpu, &vsie_page->mcck_info); |
| 1184 | return 0; | 1184 | return 0; |
| 1185 | } | 1185 | } |
| 1186 | 1186 | ||
| 1187 | WARN_ON_ONCE(sie_return != SIE64_RETURN_NORMAL); | ||
| 1188 | |||
| 1187 | if (rc > 0) | 1189 | if (rc > 0) |
| 1188 | rc = 0; /* we could still have an icpt */ | 1190 | rc = 0; /* we could still have an icpt */ |
| 1189 | else if (current->thread.gmap_int_code) | 1191 | else if (current->thread.gmap_int_code) |
| @@ -1326,7 +1328,7 @@ static void unregister_shadow_scb(struct kvm_vcpu *vcpu) · 136 unmodified lines | |||
| 1326 | static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) | 1328 | static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) |
| 1327 | { | 1329 | { |
| 1328 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; | 1330 | struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s; |
| 1329 | struct gmap *sg; | 1331 | struct gmap *sg = NULL; |
| 1330 | int rc = 0; | 1332 | int rc = 0; |
| 1331 | 1333 | ||
| 1332 | while (1) { | 1334 | while (1) { |
| @@ -1366,6 +1368,8 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · 33 unmodified lines | |||
| 1366 | sg = gmap_put(sg); | 1368 | sg = gmap_put(sg); |
| 1367 | cond_resched(); | 1369 | cond_resched(); |
| 1368 | } | 1370 | } |
| 1371 | if (sg) | ||
| 1372 | sg = gmap_put(sg); | ||
| 1369 | 1373 | ||
| 1370 | if (rc == -EFAULT) { | 1374 | if (rc == -EFAULT) { |
| 1371 | /* | 1375 | /* |
| @@ -441,10 +441,17 @@ void do_secure_storage_access(struct pt_regs *regs) · 440 unmodified lines | |||
| 441 | folio = phys_to_folio(addr); | 441 | folio = phys_to_folio(addr); |
| 442 | if (unlikely(!folio_try_get(folio))) | 442 | if (unlikely(!folio_try_get(folio))) |
| 443 | return; | 443 | return; |
| 444 | rc = arch_make_folio_accessible(folio); | 444 | rc = uv_convert_from_secure(folio_to_phys(folio)); |
| 445 | if (!rc) | ||
| 446 | clear_bit(PG_arch_1, &folio->flags.f); | ||
| 445 | folio_put(folio); | 447 | folio_put(folio); |
| 448 | /* | ||
| 449 | * There are some valid fixup types for kernel | ||
| 450 | * accesses to donated secure memory. zeropad is one | ||
| 451 | * of them. | ||
| 452 | */ | ||
| 446 | if (rc) | 453 | if (rc) |
| 447 | BUG(); | 454 | return handle_fault_error_nolock(regs, 0); |
| 448 | } else { | 455 | } else { |
| 449 | if (faulthandler_disabled()) | 456 | if (faulthandler_disabled()) |
| 450 | return handle_fault_error_nolock(regs, 0); | 457 | return handle_fault_error_nolock(regs, 0); |
| @@ -26,10 +26,6 @@ static int platform_match(struct device *dev, struct device_driver *drv) · 25 unmodified lines | |||
| 26 | struct platform_device *pdev = to_platform_device(dev); | 26 | struct platform_device *pdev = to_platform_device(dev); |
| 27 | struct platform_driver *pdrv = to_platform_driver(drv); | 27 | struct platform_driver *pdrv = to_platform_driver(drv); |
| 28 | 28 | ||
| 29 | /* When driver_override is set, only bind to the matching driver */ | ||
| 30 | if (pdev->driver_override) | ||
| 31 | return !strcmp(pdev->driver_override, drv->name); | ||
| 32 | |||
| 33 | /* Then try to match against the id table */ | 29 | /* Then try to match against the id table */ |
| 34 | if (pdrv->id_table) | 30 | if (pdrv->id_table) |
| 35 | return platform_match_id(pdrv->id_table, pdev) != NULL; | 31 | return platform_match_id(pdrv->id_table, pdev) != NULL; |
| @@ -121,6 +121,9 @@ noinstr struct ghcb *__sev_get_ghcb(struct ghcb_state *state) · 120 unmodified lines | |||
| 121 | 121 | ||
| 122 | WARN_ON(!irqs_disabled()); | 122 | WARN_ON(!irqs_disabled()); |
| 123 | 123 | ||
| 124 | if (!sev_cfg.ghcbs_initialized) | ||
| 125 | return boot_ghcb; | ||
| 126 | |||
| 124 | data = this_cpu_read(runtime_data); | 127 | data = this_cpu_read(runtime_data); |
| 125 | ghcb = &data->ghcb_page; | 128 | ghcb = &data->ghcb_page; |
| 126 | 129 | ||
| @@ -164,6 +167,9 @@ noinstr void __sev_put_ghcb(struct ghcb_state *state) · 37 unmodified lines | |||
| 164 | 167 | ||
| 165 | WARN_ON(!irqs_disabled()); | 168 | WARN_ON(!irqs_disabled()); |
| 166 | 169 | ||
| 170 | if (!sev_cfg.ghcbs_initialized) | ||
| 171 | return; | ||
| 172 | |||
| 167 | data = this_cpu_read(runtime_data); | 173 | data = this_cpu_read(runtime_data); |
| 168 | ghcb = &data->ghcb_page; | 174 | ghcb = &data->ghcb_page; |
| 169 | 175 | ||
| @@ -177,6 +177,16 @@ static noinstr void fred_extint(struct pt_regs *regs) · 176 unmodified lines | |||
| 177 | } | 177 | } |
| 178 | } | 178 | } |
| 179 | 179 | ||
| 180 | #ifdef CONFIG_AMD_MEM_ENCRYPT | ||
| 181 | noinstr void exc_vmm_communication(struct pt_regs *regs, unsigned long error_code) | ||
| 182 | { | ||
| 183 | if (user_mode(regs)) | ||
| 184 | return user_exc_vmm_communication(regs, error_code); | ||
| 185 | else | ||
| 186 | return kernel_exc_vmm_communication(regs, error_code); | ||
| 187 | } | ||
| 188 | #endif | ||
| 189 | |||
| 180 | static noinstr void fred_hwexc(struct pt_regs *regs, unsigned long error_code) | 190 | static noinstr void fred_hwexc(struct pt_regs *regs, unsigned long error_code) |
| 181 | { | 191 | { |
| 182 | /* Optimize for #PF. That's the only exception which matters performance wise */ | 192 | /* Optimize for #PF. That's the only exception which matters performance wise */ |
| @@ -207,6 +217,10 @@ static noinstr void fred_hwexc(struct pt_regs *regs, unsigned long error_code) · 24 unmodified lines | |||
| 207 | #ifdef CONFIG_X86_CET | 217 | #ifdef CONFIG_X86_CET |
| 208 | case X86_TRAP_CP: return exc_control_protection(regs, error_code); | 218 | case X86_TRAP_CP: return exc_control_protection(regs, error_code); |
| 209 | #endif | 219 | #endif |
| 220 | #ifdef CONFIG_AMD_MEM_ENCRYPT | ||
| 221 | case X86_TRAP_VC: return exc_vmm_communication(regs, error_code); | ||
| 222 | #endif | ||
| 223 | |||
| 210 | default: return fred_bad_type(regs, error_code); | 224 | default: return fred_bad_type(regs, error_code); |
| 211 | } | 225 | } |
| 212 | 226 | ||
| @@ -13,7 +13,7 @@ · 12 unmodified lines | |||
| 13 | #include <linux/types.h> | 13 | #include <linux/types.h> |
| 14 | #include <vdso/gettime.h> | 14 | #include <vdso/gettime.h> |
| 15 | 15 | ||
| 16 | #include "../../../../lib/vdso/gettimeofday.c" | 16 | #include "lib/vdso/gettimeofday.c" |
| 17 | 17 | ||
| 18 | int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) | 18 | int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) |
| 19 | { | 19 | { |
| @@ -1372,14 +1372,17 @@ static void x86_pmu_enable(struct pmu *pmu) · 1371 unmodified lines | |||
| 1372 | else if (i < n_running) | 1372 | else if (i < n_running) |
| 1373 | continue; | 1373 | continue; |
| 1374 | 1374 | ||
| 1375 | if (hwc->state & PERF_HES_ARCH) | 1375 | cpuc->events[hwc->idx] = event; |
| 1376 | |||
| 1377 | if (hwc->state & PERF_HES_ARCH) { | ||
| 1378 | static_call(x86_pmu_set_period)(event); | ||
| 1376 | continue; | 1379 | continue; |
| 1380 | } | ||
| 1377 | 1381 | ||
| 1378 | /* | 1382 | /* |
| 1379 | * if cpuc->enabled = 0, then no wrmsr as | 1383 | * if cpuc->enabled = 0, then no wrmsr as |
| 1380 | * per x86_pmu_enable_event() | 1384 | * per x86_pmu_enable_event() |
| 1381 | */ | 1385 | */ |
| 1382 | cpuc->events[hwc->idx] = event; | ||
| 1383 | x86_pmu_start(event, PERF_EF_RELOAD); | 1386 | x86_pmu_start(event, PERF_EF_RELOAD); |
| 1384 | } | 1387 | } |
| 1385 | cpuc->n_added = 0; | 1388 | cpuc->n_added = 0; |
| @@ -4628,6 +4628,19 @@ static inline void intel_pmu_set_acr_caused_constr(struct perf_event *event, · 4627 unmodified lines | |||
| 4628 | event->hw.dyn_constraint &= hybrid(event->pmu, acr_cause_mask64); | 4628 | event->hw.dyn_constraint &= hybrid(event->pmu, acr_cause_mask64); |
| 4629 | } | 4629 | } |
| 4630 | 4630 | ||
| 4631 | static inline int intel_set_branch_counter_constr(struct perf_event *event, | ||
| 4632 | int *num) | ||
| 4633 | { | ||
| 4634 | if (branch_sample_call_stack(event)) | ||
| 4635 | return -EINVAL; | ||
| 4636 | if (branch_sample_counters(event)) { | ||
| 4637 | (*num)++; | ||
| 4638 | event->hw.dyn_constraint &= x86_pmu.lbr_counters; | ||
| 4639 | } | ||
| 4640 | |||
| 4641 | return 0; | ||
| 4642 | } | ||
| 4643 | |||
| 4631 | static int intel_pmu_hw_config(struct perf_event *event) | 4644 | static int intel_pmu_hw_config(struct perf_event *event) |
| 4632 | { | 4645 | { |
| 4633 | int ret = x86_pmu_hw_config(event); | 4646 | int ret = x86_pmu_hw_config(event); |
| @@ -4698,21 +4711,19 @@ static int intel_pmu_hw_config(struct perf_event *event) · 64 unmodified lines | |||
| 4698 | * group, which requires the extra space to store the counters. | 4711 | * group, which requires the extra space to store the counters. |
| 4699 | */ | 4712 | */ |
| 4700 | leader = event->group_leader; | 4713 | leader = event->group_leader; |
| 4701 | if (branch_sample_call_stack(leader)) | 4714 | if (intel_set_branch_counter_constr(leader, &num)) |
| 4702 | return -EINVAL; | 4715 | return -EINVAL; |
| 4703 | if (branch_sample_counters(leader)) { | ||
| 4704 | num++; | ||
| 4705 | leader->hw.dyn_constraint &= x86_pmu.lbr_counters; | ||
| 4706 | } | ||
| 4707 | leader->hw.flags |= PERF_X86_EVENT_BRANCH_COUNTERS; | 4716 | leader->hw.flags |= PERF_X86_EVENT_BRANCH_COUNTERS; |
| 4708 | 4717 | ||
| 4709 | for_each_sibling_event(sibling, leader) { | 4718 | for_each_sibling_event(sibling, leader) { |
| 4710 | if (branch_sample_call_stack(sibling)) | 4719 | if (intel_set_branch_counter_constr(sibling, &num)) |
| 4720 | return -EINVAL; | ||
| 4721 | } | ||
| 4722 | |||
| 4723 | /* event isn't installed as a sibling yet. */ | ||
| 4724 | if (event != leader) { | ||
| 4725 | if (intel_set_branch_counter_constr(event, &num)) | ||
| 4711 | return -EINVAL; | 4726 | return -EINVAL; |
| 4712 | if (branch_sample_counters(sibling)) { | ||
| 4713 | num++; | ||
| 4714 | sibling->hw.dyn_constraint &= x86_pmu.lbr_counters; | ||
| 4715 | } | ||
| 4716 | } | 4727 | } |
| 4717 | 4728 | ||
| 4718 | if (num > fls(x86_pmu.lbr_counters)) | 4729 | if (num > fls(x86_pmu.lbr_counters)) |
| @@ -4844,8 +4855,10 @@ static int intel_pmu_hw_config(struct perf_event *event) · 125 unmodified lines | |||
| 4844 | intel_pmu_set_acr_caused_constr(leader, idx++, cause_mask); | 4855 | intel_pmu_set_acr_caused_constr(leader, idx++, cause_mask); |
| 4845 | 4856 | ||
| 4846 | if (leader->nr_siblings) { | 4857 | if (leader->nr_siblings) { |
| 4847 | for_each_sibling_event(sibling, leader) | 4858 | for_each_sibling_event(sibling, leader) { |
| 4848 | intel_pmu_set_acr_caused_constr(sibling, idx++, cause_mask); | 4859 | if (is_x86_event(sibling)) |
| 4860 | intel_pmu_set_acr_caused_constr(sibling, idx++, cause_mask); | ||
| 4861 | } | ||
| 4849 | } | 4862 | } |
| 4850 | 4863 | ||
| 4851 | if (leader != event) | 4864 | if (leader != event) |
| @@ -345,12 +345,12 @@ static u64 parse_omr_data_source(u8 dse) · 344 unmodified lines | |||
| 345 | if (omr.omr_remote) | 345 | if (omr.omr_remote) |
| 346 | val |= REM; | 346 | val |= REM; |
| 347 | 347 | ||
| 348 | val |= omr.omr_hitm ? P(SNOOP, HITM) : P(SNOOP, HIT); | ||
| 349 | |||
| 350 | if (omr.omr_source == 0x2) { | 348 | if (omr.omr_source == 0x2) { |
| 351 | u8 snoop = omr.omr_snoop | omr.omr_promoted; | 349 | u8 snoop = omr.omr_snoop | (omr.omr_promoted << 1); |
| 352 | 350 | ||
| 353 | if (snoop == 0x0) | 351 | if (omr.omr_hitm) |
| 352 | val |= P(SNOOP, HITM); | ||
| 353 | else if (snoop == 0x0) | ||
| 354 | val |= P(SNOOP, NA); | 354 | val |= P(SNOOP, NA); |
| 355 | else if (snoop == 0x1) | 355 | else if (snoop == 0x1) |
| 356 | val |= P(SNOOP, MISS); | 356 | val |= P(SNOOP, MISS); |
| @@ -359,7 +359,10 @@ static u64 parse_omr_data_source(u8 dse) · 2 unmodified lines | |||
| 359 | else if (snoop == 0x3) | 359 | else if (snoop == 0x3) |
| 360 | val |= P(SNOOP, NONE); | 360 | val |= P(SNOOP, NONE); |
| 361 | } else if (omr.omr_source > 0x2 && omr.omr_source < 0x7) { | 361 | } else if (omr.omr_source > 0x2 && omr.omr_source < 0x7) { |
| 362 | val |= omr.omr_hitm ? P(SNOOP, HITM) : P(SNOOP, HIT); | ||
| 362 | val |= omr.omr_snoop ? P(SNOOPX, FWD) : 0; | 363 | val |= omr.omr_snoop ? P(SNOOPX, FWD) : 0; |
| 364 | } else { | ||
| 365 | val |= P(SNOOP, NONE); | ||
| 363 | } | 366 | } |
| 364 | 367 | ||
| 365 | return val; | 368 | return val; |
| @@ -107,14 +107,12 @@ static void __noreturn hv_panic_timeout_reboot(void) · 106 unmodified lines | |||
| 107 | cpu_relax(); | 107 | cpu_relax(); |
| 108 | } | 108 | } |
| 109 | 109 | ||
| 110 | /* This cannot be inlined as it needs stack */ | 110 | static void hv_crash_restore_tss(void) |
| 111 | static noinline __noclone void hv_crash_restore_tss(void) | ||
| 112 | { | 111 | { |
| 113 | load_TR_desc(); | 112 | load_TR_desc(); |
| 114 | } | 113 | } |
| 115 | 114 | ||
| 116 | /* This cannot be inlined as it needs stack */ | 115 | static void hv_crash_clear_kernpt(void) |
| 117 | static noinline void hv_crash_clear_kernpt(void) | ||
| 118 | { | 116 | { |
| 119 | pgd_t *pgd; | 117 | pgd_t *pgd; |
| 120 | p4d_t *p4d; | 118 | p4d_t *p4d; |
| @@ -125,6 +123,25 @@ static noinline void hv_crash_clear_kernpt(void) · 4 unmodified lines | |||
| 125 | native_p4d_clear(p4d); | 123 | native_p4d_clear(p4d); |
| 126 | } | 124 | } |
| 127 | 125 | ||
| 126 | |||
| 127 | static void __noreturn hv_crash_handle(void) | ||
| 128 | { | ||
| 129 | hv_crash_restore_tss(); | ||
| 130 | hv_crash_clear_kernpt(); | ||
| 131 | |||
| 132 | /* we are now fully in devirtualized normal kernel mode */ | ||
| 133 | __crash_kexec(NULL); | ||
| 134 | |||
| 135 | hv_panic_timeout_reboot(); | ||
| 136 | } | ||
| 137 | |||
| 138 | /* | ||
| 139 | * __naked functions do not permit function calls, not even to __always_inline | ||
| 140 | * functions that only contain asm() blocks themselves. So use a macro instead. | ||
| 141 | */ | ||
| 142 | #define hv_wrmsr(msr, val) \ | ||
| 143 | asm volatile("wrmsr" :: "c"(msr), "a"((u32)val), "d"((u32)(val >> 32)) : "memory") | ||
| 144 | |||
| 128 | /* | 145 | /* |
| 129 | * This is the C entry point from the asm glue code after the disable hypercall. | 146 | * This is the C entry point from the asm glue code after the disable hypercall. |
| 130 | * We enter here in IA32-e long mode, ie, full 64bit mode running on kernel | 147 | * We enter here in IA32-e long mode, ie, full 64bit mode running on kernel |
| @@ -133,51 +150,38 @@ static noinline void hv_crash_clear_kernpt(void) · 2 unmodified lines | |||
| 133 | * available. We restore kernel GDT, and rest of the context, and continue | 150 | * available. We restore kernel GDT, and rest of the context, and continue |
| 134 | * to kexec. | 151 | * to kexec. |
| 135 | */ | 152 | */ |
| 136 | static asmlinkage void __noreturn hv_crash_c_entry(void) | 153 | static void __naked hv_crash_c_entry(void) |
| 137 | { | 154 | { |
| 138 | struct hv_crash_ctxt *ctxt = &hv_crash_ctxt; | ||
| 139 | |||
| 140 | /* first thing, restore kernel gdt */ | 155 | /* first thing, restore kernel gdt */ |
| 141 | native_load_gdt(&ctxt->gdtr); | 156 | asm volatile("lgdt %0" : : "m" (hv_crash_ctxt.gdtr)); |
| 142 | 157 | ||
| 143 | asm volatile("movw %%ax, %%ss" : : "a"(ctxt->ss)); | 158 | asm volatile("movw %0, %%ss\n\t" |
| 144 | asm volatile("movq %0, %%rsp" : : "m"(ctxt->rsp)); | 159 | "movq %1, %%rsp" |
| 160 | :: "m"(hv_crash_ctxt.ss), "m"(hv_crash_ctxt.rsp)); | ||
| 145 | 161 | ||
| 146 | asm volatile("movw %%ax, %%ds" : : "a"(ctxt->ds)); | 162 | asm volatile("movw %0, %%ds" : : "m"(hv_crash_ctxt.ds)); |
| 147 | asm volatile("movw %%ax, %%es" : : "a"(ctxt->es)); | 163 | asm volatile("movw %0, %%es" : : "m"(hv_crash_ctxt.es)); |
| 148 | asm volatile("movw %%ax, %%fs" : : "a"(ctxt->fs)); | 164 | asm volatile("movw %0, %%fs" : : "m"(hv_crash_ctxt.fs)); |
| 149 | asm volatile("movw %%ax, %%gs" : : "a"(ctxt->gs)); | 165 | asm volatile("movw %0, %%gs" : : "m"(hv_crash_ctxt.gs)); |
| 150 | 166 | ||
| 151 | native_wrmsrq(MSR_IA32_CR_PAT, ctxt->pat); | 167 | hv_wrmsr(MSR_IA32_CR_PAT, hv_crash_ctxt.pat); |
| 152 | asm volatile("movq %0, %%cr0" : : "r"(ctxt->cr0)); | 168 | asm volatile("movq %0, %%cr0" : : "r"(hv_crash_ctxt.cr0)); |
| 153 | 169 | ||
| 154 | asm volatile("movq %0, %%cr8" : : "r"(ctxt->cr8)); | 170 | asm volatile("movq %0, %%cr8" : : "r"(hv_crash_ctxt.cr8)); |
| 155 | asm volatile("movq %0, %%cr4" : : "r"(ctxt->cr4)); | 171 | asm volatile("movq %0, %%cr4" : : "r"(hv_crash_ctxt.cr4)); |
| 156 | asm volatile("movq %0, %%cr2" : : "r"(ctxt->cr4)); | 172 | asm volatile("movq %0, %%cr2" : : "r"(hv_crash_ctxt.cr2)); |
| 157 | 173 | ||
| 158 | native_load_idt(&ctxt->idtr); | 174 | asm volatile("lidt %0" : : "m" (hv_crash_ctxt.idtr)); |
| 159 | native_wrmsrq(MSR_GS_BASE, ctxt->gsbase); | 175 | hv_wrmsr(MSR_GS_BASE, hv_crash_ctxt.gsbase); |
| 160 | native_wrmsrq(MSR_EFER, ctxt->efer); | 176 | hv_wrmsr(MSR_EFER, hv_crash_ctxt.efer); |
| 161 | 177 | ||
| 162 | /* restore the original kernel CS now via far return */ | 178 | /* restore the original kernel CS now via far return */ |
| 163 | asm volatile("movzwq %0, %%rax\n\t" | 179 | asm volatile("pushq %q0\n\t" |
| 164 | "pushq %%rax\n\t" | 180 | "pushq %q1\n\t" |
| 165 | "pushq $1f\n\t" | 181 | "lretq" |
| 166 | "lretq\n\t" | 182 | :: "r"(hv_crash_ctxt.cs), "r"(hv_crash_handle)); |
| 167 | "1:nop\n\t" : : "m"(ctxt->cs) : "rax"); | ||
| 168 | |||
| 169 | /* We are in asmlinkage without stack frame, hence make C function | ||
| 170 | * calls which will buy stack frames. | ||
| 171 | */ | ||
| 172 | hv_crash_restore_tss(); | ||
| 173 | hv_crash_clear_kernpt(); | ||
| 174 | |||
| 175 | /* we are now fully in devirtualized normal kernel mode */ | ||
| 176 | __crash_kexec(NULL); | ||
| 177 | |||
| 178 | hv_panic_timeout_reboot(); | ||
| 179 | } | 183 | } |
| 180 | /* Tell gcc we are using lretq long jump in the above function intentionally */ | 184 | /* Tell objtool we are using lretq long jump in the above function intentionally */ |
| 181 | STACK_FRAME_NON_STANDARD(hv_crash_c_entry); | 185 | STACK_FRAME_NON_STANDARD(hv_crash_c_entry); |
| 182 | 186 | ||
| 183 | static void hv_mark_tss_not_busy(void) | 187 | static void hv_mark_tss_not_busy(void) |
| @@ -195,20 +199,20 @@ static void hv_hvcrash_ctxt_save(void) · 11 unmodified lines | |||
| 195 | { | 199 | { |
| 196 | struct hv_crash_ctxt *ctxt = &hv_crash_ctxt; | 200 | struct hv_crash_ctxt *ctxt = &hv_crash_ctxt; |
| 197 | 201 | ||
| 198 | asm volatile("movq %%rsp,%0" : "=m"(ctxt->rsp)); | 202 | ctxt->rsp = current_stack_pointer; |
| 199 | 203 | ||
| 200 | ctxt->cr0 = native_read_cr0(); | 204 | ctxt->cr0 = native_read_cr0(); |
| 201 | ctxt->cr4 = native_read_cr4(); | 205 | ctxt->cr4 = native_read_cr4(); |
| 202 | 206 | ||
| 203 | asm volatile("movq %%cr2, %0" : "=a"(ctxt->cr2)); | 207 | asm volatile("movq %%cr2, %0" : "=r"(ctxt->cr2)); |
| 204 | asm volatile("movq %%cr8, %0" : "=a"(ctxt->cr8)); | 208 | asm volatile("movq %%cr8, %0" : "=r"(ctxt->cr8)); |
| 205 | 209 | ||
| 206 | asm volatile("movl %%cs, %%eax" : "=a"(ctxt->cs)); | 210 | asm volatile("movw %%cs, %0" : "=m"(ctxt->cs)); |
| 207 | asm volatile("movl %%ss, %%eax" : "=a"(ctxt->ss)); | 211 | asm volatile("movw %%ss, %0" : "=m"(ctxt->ss)); |
| 208 | asm volatile("movl %%ds, %%eax" : "=a"(ctxt->ds)); | 212 | asm volatile("movw %%ds, %0" : "=m"(ctxt->ds)); |
| 209 | asm volatile("movl %%es, %%eax" : "=a"(ctxt->es)); | 213 | asm volatile("movw %%es, %0" : "=m"(ctxt->es)); |
| 210 | asm volatile("movl %%fs, %%eax" : "=a"(ctxt->fs)); | 214 | asm volatile("movw %%fs, %0" : "=m"(ctxt->fs)); |
| 211 | asm volatile("movl %%gs, %%eax" : "=a"(ctxt->gs)); | 215 | asm volatile("movw %%gs, %0" : "=m"(ctxt->gs)); |
| 212 | 216 | ||
| 213 | native_store_gdt(&ctxt->gdtr); | 217 | native_store_gdt(&ctxt->gdtr); |
| 214 | store_idt(&ctxt->idtr); | 218 | store_idt(&ctxt->idtr); |
| @@ -2485,7 +2485,8 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); · 2484 unmodified lines | |||
| 2485 | KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ | 2485 | KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ |
| 2486 | KVM_X86_QUIRK_SLOT_ZAP_ALL | \ | 2486 | KVM_X86_QUIRK_SLOT_ZAP_ALL | \ |
| 2487 | KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ | 2487 | KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ |
| 2488 | KVM_X86_QUIRK_IGNORE_GUEST_PAT) | 2488 | KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ |
| 2489 | KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM) | ||
| 2489 | 2490 | ||
| 2490 | #define KVM_X86_CONDITIONAL_QUIRKS \ | 2491 | #define KVM_X86_CONDITIONAL_QUIRKS \ |
| 2491 | (KVM_X86_QUIRK_CD_NW_CLEARED | \ | 2492 | (KVM_X86_QUIRK_CD_NW_CLEARED | \ |
| @@ -476,6 +476,7 @@ struct kvm_sync_regs { · 475 unmodified lines | |||
| 476 | #define KVM_X86_QUIRK_SLOT_ZAP_ALL (1 << 7) | 476 | #define KVM_X86_QUIRK_SLOT_ZAP_ALL (1 << 7) |
| 477 | #define KVM_X86_QUIRK_STUFF_FEATURE_MSRS (1 << 8) | 477 | #define KVM_X86_QUIRK_STUFF_FEATURE_MSRS (1 << 8) |
| 478 | #define KVM_X86_QUIRK_IGNORE_GUEST_PAT (1 << 9) | 478 | #define KVM_X86_QUIRK_IGNORE_GUEST_PAT (1 << 9) |
| 479 | #define KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM (1 << 10) | ||
| 479 | 480 | ||
| 480 | #define KVM_STATE_NESTED_FORMAT_VMX 0 | 481 | #define KVM_STATE_NESTED_FORMAT_VMX 0 |
| 481 | #define KVM_STATE_NESTED_FORMAT_SVM 1 | 482 | #define KVM_STATE_NESTED_FORMAT_SVM 1 |
| @@ -44,6 +44,20 @@ KCOV_INSTRUMENT_unwind_orc.o := n · 43 unmodified lines | |||
| 44 | KCOV_INSTRUMENT_unwind_frame.o := n | 44 | KCOV_INSTRUMENT_unwind_frame.o := n |
| 45 | KCOV_INSTRUMENT_unwind_guess.o := n | 45 | KCOV_INSTRUMENT_unwind_guess.o := n |
| 46 | 46 | ||
| 47 | # Disable KCOV to prevent crashes during kexec: load_segments() invalidates | ||
| 48 | # the GS base, which KCOV relies on for per-CPU data. | ||
| 49 | # | ||
| 50 | # As KCOV and KEXEC compatibility should be preserved (e.g. syzkaller is | ||
| 51 | # using it to collect crash dumps during kernel fuzzing), disabling | ||
| 52 | # KCOV for KEXEC kernels is not an option. Selectively disabling KCOV | ||
| 53 | # instrumentation for individual affected functions can be fragile, while | ||
| 54 | # adding more checks to KCOV would slow it down. | ||
| 55 | # | ||
| 56 | # As a compromise solution, disable KCOV instrumentation for the whole | ||
| 57 | # source code file. If its coverage is ever needed, other approaches | ||
| 58 | # should be considered. | ||
| 59 | KCOV_INSTRUMENT_machine_kexec_64.o := n | ||
| 60 | |||
| 47 | CFLAGS_head32.o := -fno-stack-protector | 61 | CFLAGS_head32.o := -fno-stack-protector |
| 48 | CFLAGS_head64.o := -fno-stack-protector | 62 | CFLAGS_head64.o := -fno-stack-protector |
| 49 | CFLAGS_irq.o := -I $(src)/../include/asm/trace | 63 | CFLAGS_irq.o := -I $(src)/../include/asm/trace |
| @@ -1894,6 +1894,7 @@ void __init check_x2apic(void) · 1893 unmodified lines | |||
| 1894 | 1894 | ||
| 1895 | static inline void try_to_enable_x2apic(int remap_mode) { } | 1895 | static inline void try_to_enable_x2apic(int remap_mode) { } |
| 1896 | static inline void __x2apic_enable(void) { } | 1896 | static inline void __x2apic_enable(void) { } |
| 1897 | static inline void __x2apic_disable(void) { } | ||
| 1897 | #endif /* !CONFIG_X86_X2APIC */ | 1898 | #endif /* !CONFIG_X86_X2APIC */ |
| 1898 | 1899 | ||
| 1899 | void __init enable_IR_x2apic(void) | 1900 | void __init enable_IR_x2apic(void) |
| @@ -2456,6 +2457,11 @@ static void lapic_resume(void *data) · 556 unmodified lines | |||
| 2456 | if (x2apic_mode) { | 2457 | if (x2apic_mode) { |
| 2457 | __x2apic_enable(); | 2458 | __x2apic_enable(); |
| 2458 | } else { | 2459 | } else { |
| 2460 | if (x2apic_enabled()) { | ||
| 2461 | pr_warn_once("x2apic: re-enabled by firmware during resume. Disabling\n"); | ||
| 2462 | __x2apic_disable(); | ||
| 2463 | } | ||
| 2464 | |||
| 2459 | /* | 2465 | /* |
| 2460 | * Make sure the APICBASE points to the right address | 2466 | * Make sure the APICBASE points to the right address |
| 2461 | * | 2467 | * |
| @@ -1708,8 +1708,22 @@ static void __init uv_system_init_hub(void) · 1707 unmodified lines | |||
| 1708 | struct uv_hub_info_s *new_hub; | 1708 | struct uv_hub_info_s *new_hub; |
| 1709 | 1709 | ||
| 1710 | /* Allocate & fill new per hub info list */ | 1710 | /* Allocate & fill new per hub info list */ |
| 1711 | new_hub = (bid == 0) ? &uv_hub_info_node0 | 1711 | if (bid == 0) { |
| 1712 | : kzalloc_node(bytes, GFP_KERNEL, uv_blade_to_node(bid)); | 1712 | new_hub = &uv_hub_info_node0; |
| 1713 | } else { | ||
| 1714 | int nid; | ||
| 1715 | |||
| 1716 | /* | ||
| 1717 | * Deconfigured sockets are mapped to SOCK_EMPTY. Use | ||
| 1718 | * NUMA_NO_NODE to allocate on a valid node. | ||
| 1719 | */ | ||
| 1720 | nid = uv_blade_to_node(bid); | ||
| 1721 | if (nid == SOCK_EMPTY) | ||
| 1722 | nid = NUMA_NO_NODE; | ||
| 1723 | |||
| 1724 | new_hub = kzalloc_node(bytes, GFP_KERNEL, nid); | ||
| 1725 | } | ||
| 1726 | |||
| 1713 | if (WARN_ON_ONCE(!new_hub)) { | 1727 | if (WARN_ON_ONCE(!new_hub)) { |
| 1714 | /* do not kfree() bid 0, which is statically allocated */ | 1728 | /* do not kfree() bid 0, which is statically allocated */ |
| 1715 | while (--bid > 0) | 1729 | while (--bid > 0) |
| @@ -433,7 +433,20 @@ static __always_inline void setup_lass(struct cpuinfo_x86 *c) · 432 unmodified lines | |||
| 433 | 433 | ||
| 434 | /* These bits should not change their value after CPU init is finished. */ | 434 | /* These bits should not change their value after CPU init is finished. */ |
| 435 | static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | | 435 | static const unsigned long cr4_pinned_mask = X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP | |
| 436 | X86_CR4_FSGSBASE | X86_CR4_CET | X86_CR4_FRED; | 436 | X86_CR4_FSGSBASE | X86_CR4_CET; |
| 437 | |||
| 438 | /* | ||
| 439 | * The CR pinning protects against ROP on the 'mov %reg, %CRn' instruction(s). | ||
| 440 | * Since you can ROP directly to these instructions (barring shadow stack), | ||
| 441 | * any protection must follow immediately and unconditionally after that. | ||
| 442 | * | ||
| 443 | * Specifically, the CR[04] write functions below will have the value | ||
| 444 | * validation controlled by the @cr_pinning static_branch which is | ||
| 445 | * __ro_after_init, just like the cr4_pinned_bits value. | ||
| 446 | * | ||
| 447 | * Once set, an attacker will have to defeat page-tables to get around these | ||
| 448 | * restrictions. Which is a much bigger ask than 'simple' ROP. | ||
| 449 | */ | ||
| 437 | static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning); | 450 | static DEFINE_STATIC_KEY_FALSE_RO(cr_pinning); |
| 438 | static unsigned long cr4_pinned_bits __ro_after_init; | 451 | static unsigned long cr4_pinned_bits __ro_after_init; |
| 439 | 452 | ||
| @@ -2050,12 +2063,6 @@ static void identify_cpu(struct cpuinfo_x86 *c) · 1610 unmodified lines | |||
| 2050 | setup_umip(c); | 2063 | setup_umip(c); |
| 2051 | setup_lass(c); | 2064 | setup_lass(c); |
| 2052 | 2065 | ||
| 2053 | /* Enable FSGSBASE instructions if available. */ | ||
| 2054 | if (cpu_has(c, X86_FEATURE_FSGSBASE)) { | ||
| 2055 | cr4_set_bits(X86_CR4_FSGSBASE); | ||
| 2056 | elf_hwcap2 |= HWCAP2_FSGSBASE; | ||
| 2057 | } | ||
| 2058 | |||
| 2059 | /* | 2066 | /* |
| 2060 | * The vendor-specific functions might have changed features. | 2067 | * The vendor-specific functions might have changed features. |
| 2061 | * Now we do "generic changes." | 2068 | * Now we do "generic changes." |
| @@ -2416,6 +2423,18 @@ void cpu_init_exception_handling(bool boot_cpu) · 354 unmodified lines | |||
| 2416 | /* GHCB needs to be setup to handle #VC. */ | 2423 | /* GHCB needs to be setup to handle #VC. */ |
| 2417 | setup_ghcb(); | 2424 | setup_ghcb(); |
| 2418 | 2425 | ||
| 2426 | /* | ||
| 2427 | * On CPUs with FSGSBASE support, paranoid_entry() uses | ||
| 2428 | * ALTERNATIVE-patched RDGSBASE/WRGSBASE instructions. Secondary CPUs | ||
| 2429 | * boot after alternatives are patched globally, so early exceptions | ||
| 2430 | * execute patched code that depends on FSGSBASE. Enable the feature | ||
| 2431 | * before any exceptions occur. | ||
| 2432 | */ | ||
| 2433 | if (cpu_feature_enabled(X86_FEATURE_FSGSBASE)) { | ||
| 2434 | cr4_set_bits(X86_CR4_FSGSBASE); | ||
| 2435 | elf_hwcap2 |= HWCAP2_FSGSBASE; | ||
| 2436 | } | ||
| 2437 | |||
| 2419 | if (cpu_feature_enabled(X86_FEATURE_FRED)) { | 2438 | if (cpu_feature_enabled(X86_FEATURE_FRED)) { |
| 2420 | /* The boot CPU has enabled FRED during early boot */ | 2439 | /* The boot CPU has enabled FRED during early boot */ |
| 2421 | if (!boot_cpu) | 2440 | if (!boot_cpu) |
| @@ -875,13 +875,18 @@ void amd_clear_bank(struct mce *m) · 874 unmodified lines | |||
| 875 | { | 875 | { |
| 876 | amd_reset_thr_limit(m->bank); | 876 | amd_reset_thr_limit(m->bank); |
| 877 | 877 | ||
| 878 | /* Clear MCA_DESTAT for all deferred errors even those logged in MCA_STATUS. */ | 878 | if (mce_flags.smca) { |
| 879 | if (m->status & MCI_STATUS_DEFERRED) | 879 | /* |
| 880 | mce_wrmsrq(MSR_AMD64_SMCA_MCx_DESTAT(m->bank), 0); | 880 | * Clear MCA_DESTAT for all deferred errors even those |
| 881 | * logged in MCA_STATUS. | ||
| 882 | */ | ||
| 883 | if (m->status & MCI_STATUS_DEFERRED) | ||
| 884 | mce_wrmsrq(MSR_AMD64_SMCA_MCx_DESTAT(m->bank), 0); | ||
| 881 | 885 | ||
| 882 | /* Don't clear MCA_STATUS if MCA_DESTAT was used exclusively. */ | 886 | /* Don't clear MCA_STATUS if MCA_DESTAT was used exclusively. */ |
| 883 | if (m->kflags & MCE_CHECK_DFR_REGS) | 887 | if (m->kflags & MCE_CHECK_DFR_REGS) |
| 884 | return; | 888 | return; |
| 889 | } | ||
| 885 | 890 | ||
| 886 | mce_wrmsrq(mca_msr_reg(m->bank, MCA_STATUS), 0); | 891 | mce_wrmsrq(mca_msr_reg(m->bank, MCA_STATUS), 0); |
| 887 | } | 892 | } |
| @@ -496,8 +496,9 @@ static void hv_reserve_irq_vectors(void) · 495 unmodified lines | |||
| 496 | test_and_set_bit(HYPERV_DBG_FASTFAIL_VECTOR, system_vectors)) | 496 | test_and_set_bit(HYPERV_DBG_FASTFAIL_VECTOR, system_vectors)) |
| 497 | BUG(); | 497 | BUG(); |
| 498 | 498 | ||
| 499 | pr_info("Hyper-V: reserve vectors: %d %d %d\n", HYPERV_DBG_ASSERT_VECTOR, | 499 | pr_info("Hyper-V: reserve vectors: 0x%x 0x%x 0x%x\n", |
| 500 | HYPERV_DBG_SERVICE_VECTOR, HYPERV_DBG_FASTFAIL_VECTOR); | 500 | HYPERV_DBG_ASSERT_VECTOR, HYPERV_DBG_SERVICE_VECTOR, |
| 501 | HYPERV_DBG_FASTFAIL_VECTOR); | ||
| 501 | } | 502 | } |
| 502 | 503 | ||
| 503 | static void __init ms_hyperv_init_platform(void) | 504 | static void __init ms_hyperv_init_platform(void) |
| @@ -776,7 +776,10 @@ do { \ · 775 unmodified lines | |||
| 776 | #define SYNTHESIZED_F(name) \ | 776 | #define SYNTHESIZED_F(name) \ |
| 777 | ({ \ | 777 | ({ \ |
| 778 | kvm_cpu_cap_synthesized |= feature_bit(name); \ | 778 | kvm_cpu_cap_synthesized |= feature_bit(name); \ |
| 779 | F(name); \ | 779 | \ |
| 780 | BUILD_BUG_ON(X86_FEATURE_##name >= MAX_CPU_FEATURES); \ | ||
| 781 | if (boot_cpu_has(X86_FEATURE_##name)) \ | ||
| 782 | F(name); \ | ||
| 780 | }) | 783 | }) |
| 781 | 784 | ||
| 782 | /* | 785 | /* |
| @@ -1981,16 +1981,17 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu) · 1980 unmodified lines | |||
| 1981 | if (entries[i] == KVM_HV_TLB_FLUSHALL_ENTRY) | 1981 | if (entries[i] == KVM_HV_TLB_FLUSHALL_ENTRY) |
| 1982 | goto out_flush_all; | 1982 | goto out_flush_all; |
| 1983 | 1983 | ||
| 1984 | if (is_noncanonical_invlpg_address(entries[i], vcpu)) | ||
| 1985 | continue; | ||
| 1986 | |||
| 1987 | /* | 1984 | /* |
| 1988 | * Lower 12 bits of 'address' encode the number of additional | 1985 | * Lower 12 bits of 'address' encode the number of additional |
| 1989 | * pages to flush. | 1986 | * pages to flush. |
| 1990 | */ | 1987 | */ |
| 1991 | gva = entries[i] & PAGE_MASK; | 1988 | gva = entries[i] & PAGE_MASK; |
| 1992 | for (j = 0; j < (entries[i] & ~PAGE_MASK) + 1; j++) | 1989 | for (j = 0; j < (entries[i] & ~PAGE_MASK) + 1; j++) { |
| 1990 | if (is_noncanonical_invlpg_address(gva + j * PAGE_SIZE, vcpu)) | ||
| 1991 | continue; | ||
| 1992 | |||
| 1993 | kvm_x86_call(flush_tlb_gva)(vcpu, gva + j * PAGE_SIZE); | 1993 | kvm_x86_call(flush_tlb_gva)(vcpu, gva + j * PAGE_SIZE); |
| 1994 | } | ||
| 1994 | 1995 | ||
| 1995 | ++vcpu->stat.tlb_flush; | 1996 | ++vcpu->stat.tlb_flush; |
| 1996 | } | 1997 | } |
| @@ -321,7 +321,8 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin, · 320 unmodified lines | |||
| 321 | idx = srcu_read_lock(&kvm->irq_srcu); | 321 | idx = srcu_read_lock(&kvm->irq_srcu); |
| 322 | gsi = kvm_irq_map_chip_pin(kvm, irqchip, pin); | 322 | gsi = kvm_irq_map_chip_pin(kvm, irqchip, pin); |
| 323 | if (gsi != -1) | 323 | if (gsi != -1) |
| 324 | hlist_for_each_entry_rcu(kimn, &ioapic->mask_notifier_list, link) | 324 | hlist_for_each_entry_srcu(kimn, &ioapic->mask_notifier_list, link, |
| 325 | srcu_read_lock_held(&kvm->irq_srcu)) | ||
| 325 | if (kimn->irq == gsi) | 326 | if (kimn->irq == gsi) |
| 326 | kimn->func(kimn, mask); | 327 | kimn->func(kimn, mask); |
| 327 | srcu_read_unlock(&kvm->irq_srcu, idx); | 328 | srcu_read_unlock(&kvm->irq_srcu, idx); |
| @@ -3044,12 +3044,6 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, · 3043 unmodified lines | |||
| 3044 | bool prefetch = !fault || fault->prefetch; | 3044 | bool prefetch = !fault || fault->prefetch; |
| 3045 | bool write_fault = fault && fault->write; | 3045 | bool write_fault = fault && fault->write; |
| 3046 | 3046 | ||
| 3047 | if (unlikely(is_noslot_pfn(pfn))) { | ||
| 3048 | vcpu->stat.pf_mmio_spte_created++; | ||
| 3049 | mark_mmio_spte(vcpu, sptep, gfn, pte_access); | ||
| 3050 | return RET_PF_EMULATE; | ||
| 3051 | } | ||
| 3052 | |||
| 3053 | if (is_shadow_present_pte(*sptep)) { | 3047 | if (is_shadow_present_pte(*sptep)) { |
| 3054 | if (prefetch && is_last_spte(*sptep, level) && | 3048 | if (prefetch && is_last_spte(*sptep, level) && |
| 3055 | pfn == spte_to_pfn(*sptep)) | 3049 | pfn == spte_to_pfn(*sptep)) |
| @@ -3066,13 +3060,22 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, · 10 unmodified lines | |||
| 3066 | child = spte_to_child_sp(pte); | 3060 | child = spte_to_child_sp(pte); |
| 3067 | drop_parent_pte(vcpu->kvm, child, sptep); | 3061 | drop_parent_pte(vcpu->kvm, child, sptep); |
| 3068 | flush = true; | 3062 | flush = true; |
| 3069 | } else if (WARN_ON_ONCE(pfn != spte_to_pfn(*sptep))) { | 3063 | } else if (pfn != spte_to_pfn(*sptep)) { |
| 3064 | WARN_ON_ONCE(vcpu->arch.mmu->root_role.direct); | ||
| 3070 | drop_spte(vcpu->kvm, sptep); | 3065 | drop_spte(vcpu->kvm, sptep); |
| 3071 | flush = true; | 3066 | flush = true; |
| 3072 | } else | 3067 | } else |
| 3073 | was_rmapped = 1; | 3068 | was_rmapped = 1; |
| 3074 | } | 3069 | } |
| 3075 | 3070 | ||
| 3071 | if (unlikely(is_noslot_pfn(pfn))) { | ||
| 3072 | vcpu->stat.pf_mmio_spte_created++; | ||
| 3073 | mark_mmio_spte(vcpu, sptep, gfn, pte_access); | ||
| 3074 | if (flush) | ||
| 3075 | kvm_flush_remote_tlbs_gfn(vcpu->kvm, gfn, level); | ||
| 3076 | return RET_PF_EMULATE; | ||
| 3077 | } | ||
| 3078 | |||
| 3076 | wrprot = make_spte(vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch, | 3079 | wrprot = make_spte(vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch, |
| 3077 | false, host_writable, &spte); | 3080 | false, host_writable, &spte); |
| 3078 | 3081 | ||
| @@ -189,12 +189,12 @@ static void avic_activate_vmcb(struct vcpu_svm *svm) · 188 unmodified lines | |||
| 189 | struct kvm_vcpu *vcpu = &svm->vcpu; | 189 | struct kvm_vcpu *vcpu = &svm->vcpu; |
| 190 | 190 | ||
| 191 | vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK); | 191 | vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK); |
| 192 | |||
| 193 | vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK; | 192 | vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK; |
| 194 | vmcb->control.avic_physical_id |= avic_get_max_physical_id(vcpu); | 193 | vmcb->control.avic_physical_id |= avic_get_max_physical_id(vcpu); |
| 195 | |||
| 196 | vmcb->control.int_ctl |= AVIC_ENABLE_MASK; | 194 | vmcb->control.int_ctl |= AVIC_ENABLE_MASK; |
| 197 | 195 | ||
| 196 | svm_clr_intercept(svm, INTERCEPT_CR8_WRITE); | ||
| 197 | |||
| 198 | /* | 198 | /* |
| 199 | * Note: KVM supports hybrid-AVIC mode, where KVM emulates x2APIC MSR | 199 | * Note: KVM supports hybrid-AVIC mode, where KVM emulates x2APIC MSR |
| 200 | * accesses, while interrupt injection to a running vCPU can be | 200 | * accesses, while interrupt injection to a running vCPU can be |
| @@ -226,6 +226,9 @@ static void avic_deactivate_vmcb(struct vcpu_svm *svm) · 25 unmodified lines | |||
| 226 | vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK); | 226 | vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK); |
| 227 | vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK; | 227 | vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK; |
| 228 | 228 | ||
| 229 | if (!sev_es_guest(svm->vcpu.kvm)) | ||
| 230 | svm_set_intercept(svm, INTERCEPT_CR8_WRITE); | ||
| 231 | |||
| 229 | /* | 232 | /* |
| 230 | * If running nested and the guest uses its own MSR bitmap, there | 233 | * If running nested and the guest uses its own MSR bitmap, there |
| 231 | * is no need to update L0's msr bitmap | 234 | * is no need to update L0's msr bitmap |
| @@ -368,7 +371,7 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb) · 136 unmodified lines | |||
| 368 | vmcb->control.avic_physical_id = __sme_set(__pa(kvm_svm->avic_physical_id_table)); | 371 | vmcb->control.avic_physical_id = __sme_set(__pa(kvm_svm->avic_physical_id_table)); |
| 369 | vmcb->control.avic_vapic_bar = APIC_DEFAULT_PHYS_BASE; | 372 | vmcb->control.avic_vapic_bar = APIC_DEFAULT_PHYS_BASE; |
| 370 | 373 | ||
| 371 | if (kvm_apicv_activated(svm->vcpu.kvm)) | 374 | if (kvm_vcpu_apicv_active(&svm->vcpu)) |
| 372 | avic_activate_vmcb(svm); | 375 | avic_activate_vmcb(svm); |
| 373 | else | 376 | else |
| 374 | avic_deactivate_vmcb(svm); | 377 | avic_deactivate_vmcb(svm); |
| @@ -418,6 +418,15 @@ static bool nested_vmcb_check_controls(struct kvm_vcpu *vcpu) · 417 unmodified lines | |||
| 418 | return __nested_vmcb_check_controls(vcpu, ctl); | 418 | return __nested_vmcb_check_controls(vcpu, ctl); |
| 419 | } | 419 | } |
| 420 | 420 | ||
| 421 | int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu) | ||
| 422 | { | ||
| 423 | if (!nested_vmcb_check_save(vcpu) || | ||
| 424 | !nested_vmcb_check_controls(vcpu)) | ||
| 425 | return -EINVAL; | ||
| 426 | |||
| 427 | return 0; | ||
| 428 | } | ||
| 429 | |||
| 421 | /* | 430 | /* |
| 422 | * If a feature is not advertised to L1, clear the corresponding vmcb12 | 431 | * If a feature is not advertised to L1, clear the corresponding vmcb12 |
| 423 | * intercept. | 432 | * intercept. |
| @@ -1028,8 +1037,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) · 604 unmodified lines | |||
| 1028 | nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); | 1037 | nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); |
| 1029 | nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); | 1038 | nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); |
| 1030 | 1039 | ||
| 1031 | if (!nested_vmcb_check_save(vcpu) || | 1040 | if (nested_svm_check_cached_vmcb12(vcpu) < 0) { |
| 1032 | !nested_vmcb_check_controls(vcpu)) { | ||
| 1033 | vmcb12->control.exit_code = SVM_EXIT_ERR; | 1041 | vmcb12->control.exit_code = SVM_EXIT_ERR; |
| 1034 | vmcb12->control.exit_info_1 = 0; | 1042 | vmcb12->control.exit_info_1 = 0; |
| 1035 | vmcb12->control.exit_info_2 = 0; | 1043 | vmcb12->control.exit_info_2 = 0; |
| @@ -1077,8 +1077,7 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) · 1076 unmodified lines | |||
| 1077 | svm_set_intercept(svm, INTERCEPT_CR0_WRITE); | 1077 | svm_set_intercept(svm, INTERCEPT_CR0_WRITE); |
| 1078 | svm_set_intercept(svm, INTERCEPT_CR3_WRITE); | 1078 | svm_set_intercept(svm, INTERCEPT_CR3_WRITE); |
| 1079 | svm_set_intercept(svm, INTERCEPT_CR4_WRITE); | 1079 | svm_set_intercept(svm, INTERCEPT_CR4_WRITE); |
| 1080 | if (!kvm_vcpu_apicv_active(vcpu)) | 1080 | svm_set_intercept(svm, INTERCEPT_CR8_WRITE); |
| 1081 | svm_set_intercept(svm, INTERCEPT_CR8_WRITE); | ||
| 1082 | 1081 | ||
| 1083 | set_dr_intercepts(svm); | 1082 | set_dr_intercepts(svm); |
| 1084 | 1083 | ||
| @@ -1189,7 +1188,7 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) · 104 unmodified lines | |||
| 1189 | if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS)) | 1188 | if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS)) |
| 1190 | svm->vmcb->control.erap_ctl |= ERAP_CONTROL_ALLOW_LARGER_RAP; | 1189 | svm->vmcb->control.erap_ctl |= ERAP_CONTROL_ALLOW_LARGER_RAP; |
| 1191 | 1190 | ||
| 1192 | if (kvm_vcpu_apicv_active(vcpu)) | 1191 | if (enable_apicv && irqchip_in_kernel(vcpu->kvm)) |
| 1193 | avic_init_vmcb(svm, vmcb); | 1192 | avic_init_vmcb(svm, vmcb); |
| 1194 | 1193 | ||
| 1195 | if (vnmi) | 1194 | if (vnmi) |
| @@ -2674,9 +2673,11 @@ static int dr_interception(struct kvm_vcpu *vcpu) · 1478 unmodified lines | |||
| 2674 | 2673 | ||
| 2675 | static int cr8_write_interception(struct kvm_vcpu *vcpu) | 2674 | static int cr8_write_interception(struct kvm_vcpu *vcpu) |
| 2676 | { | 2675 | { |
| 2676 | u8 cr8_prev = kvm_get_cr8(vcpu); | ||
| 2677 | int r; | 2677 | int r; |
| 2678 | 2678 | ||
| 2679 | u8 cr8_prev = kvm_get_cr8(vcpu); | 2679 | WARN_ON_ONCE(kvm_vcpu_apicv_active(vcpu)); |
| 2680 | |||
| 2680 | /* instruction emulation calls kvm_set_cr8() */ | 2681 | /* instruction emulation calls kvm_set_cr8() */ |
| 2681 | r = cr_interception(vcpu); | 2682 | r = cr_interception(vcpu); |
| 2682 | if (lapic_in_kernel(vcpu)) | 2683 | if (lapic_in_kernel(vcpu)) |
| @@ -4879,11 +4880,15 @@ static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) · 2196 unmodified lines | |||
| 4879 | vmcb12 = map.hva; | 4880 | vmcb12 = map.hva; |
| 4880 | nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); | 4881 | nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); |
| 4881 | nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); | 4882 | nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); |
| 4882 | ret = enter_svm_guest_mode(vcpu, smram64->svm_guest_vmcb_gpa, vmcb12, false); | ||
| 4883 | 4883 | ||
| 4884 | if (ret) | 4884 | if (nested_svm_check_cached_vmcb12(vcpu) < 0) |
| 4885 | goto unmap_save; | ||
| 4886 | |||
| 4887 | if (enter_svm_guest_mode(vcpu, smram64->svm_guest_vmcb_gpa, | ||
| 4888 | vmcb12, false) != 0) | ||
| 4885 | goto unmap_save; | 4889 | goto unmap_save; |
| 4886 | 4890 | ||
| 4891 | ret = 0; | ||
| 4887 | svm->nested.nested_run_pending = 1; | 4892 | svm->nested.nested_run_pending = 1; |
| 4888 | 4893 | ||
| 4889 | unmap_save: | 4894 | unmap_save: |
| @@ -797,6 +797,7 @@ static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code) · 796 unmodified lines | |||
| 797 | 797 | ||
| 798 | int nested_svm_exit_handled(struct vcpu_svm *svm); | 798 | int nested_svm_exit_handled(struct vcpu_svm *svm); |
| 799 | int nested_svm_check_permissions(struct kvm_vcpu *vcpu); | 799 | int nested_svm_check_permissions(struct kvm_vcpu *vcpu); |
| 800 | int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu); | ||
| 800 | int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr, | 801 | int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr, |
| 801 | bool has_error_code, u32 error_code); | 802 | bool has_error_code, u32 error_code); |
| 802 | int nested_svm_exit_special(struct vcpu_svm *svm); | 803 | int nested_svm_exit_special(struct vcpu_svm *svm); |
| @@ -3300,10 +3300,24 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu, · 3299 unmodified lines | |||
| 3300 | if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP))) | 3300 | if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP))) |
| 3301 | return -EINVAL; | 3301 | return -EINVAL; |
| 3302 | 3302 | ||
| 3303 | if ((vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS) && | 3303 | if (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS) { |
| 3304 | (CC(!kvm_dr7_valid(vmcs12->guest_dr7)) || | 3304 | u64 debugctl = vmcs12->guest_ia32_debugctl; |
| 3305 | CC(!vmx_is_valid_debugctl(vcpu, vmcs12->guest_ia32_debugctl, false)))) | 3305 | |
| 3306 | return -EINVAL; | 3306 | /* |
| 3307 | * FREEZE_IN_SMM is not virtualized, but allow L1 to set it in | ||
| 3308 | * vmcs12's DEBUGCTL under a quirk for backwards compatibility. | ||
| 3309 | * Note that the quirk only relaxes the consistency check. The | ||
| 3310 | * vmcc02 bit is still under the control of the host. In | ||
| 3311 | * particular, if a host administrator decides to clear the bit, | ||
| 3312 | * then L1 has no say in the matter. | ||
| 3313 | */ | ||
| 3314 | if (kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM)) | ||
| 3315 | debugctl &= ~DEBUGCTLMSR_FREEZE_IN_SMM; | ||
| 3316 | |||
| 3317 | if (CC(!kvm_dr7_valid(vmcs12->guest_dr7)) || | ||
| 3318 | CC(!vmx_is_valid_debugctl(vcpu, debugctl, false))) | ||
| 3319 | return -EINVAL; | ||
| 3320 | } | ||
| 3307 | 3321 | ||
| 3308 | if ((vmcs12->vm_entry_controls & VM_ENTRY_LOAD_IA32_PAT) && | 3322 | if ((vmcs12->vm_entry_controls & VM_ENTRY_LOAD_IA32_PAT) && |
| 3309 | CC(!kvm_pat_valid(vmcs12->guest_ia32_pat))) | 3323 | CC(!kvm_pat_valid(vmcs12->guest_ia32_pat))) |
| @@ -6842,13 +6856,34 @@ void vmx_leave_nested(struct kvm_vcpu *vcpu) · 3532 unmodified lines | |||
| 6842 | free_nested(vcpu); | 6856 | free_nested(vcpu); |
| 6843 | } | 6857 | } |
| 6844 | 6858 | ||
| 6859 | int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu) | ||
| 6860 | { | ||
| 6861 | enum vm_entry_failure_code ignored; | ||
| 6862 | struct vmcs12 *vmcs12 = get_vmcs12(vcpu); | ||
| 6863 | |||
| 6864 | if (nested_cpu_has_shadow_vmcs(vmcs12) && | ||
| 6865 | vmcs12->vmcs_link_pointer != INVALID_GPA) { | ||
| 6866 | struct vmcs12 *shadow_vmcs12 = get_shadow_vmcs12(vcpu); | ||
| 6867 | |||
| 6868 | if (shadow_vmcs12->hdr.revision_id != VMCS12_REVISION || | ||
| 6869 | !shadow_vmcs12->hdr.shadow_vmcs) | ||
| 6870 | return -EINVAL; | ||
| 6871 | } | ||
| 6872 | |||
| 6873 | if (nested_vmx_check_controls(vcpu, vmcs12) || | ||
| 6874 | nested_vmx_check_host_state(vcpu, vmcs12) || | ||
| 6875 | nested_vmx_check_guest_state(vcpu, vmcs12, &ignored)) | ||
| 6876 | return -EINVAL; | ||
| 6877 | |||
| 6878 | return 0; | ||
| 6879 | } | ||
| 6880 | |||
| 6845 | static int vmx_set_nested_state(struct kvm_vcpu *vcpu, | 6881 | static int vmx_set_nested_state(struct kvm_vcpu *vcpu, |
| 6846 | struct kvm_nested_state __user *user_kvm_nested_state, | 6882 | struct kvm_nested_state __user *user_kvm_nested_state, |
| 6847 | struct kvm_nested_state *kvm_state) | 6883 | struct kvm_nested_state *kvm_state) |
| 6848 | { | 6884 | { |
| 6849 | struct vcpu_vmx *vmx = to_vmx(vcpu); | 6885 | struct vcpu_vmx *vmx = to_vmx(vcpu); |
| 6850 | struct vmcs12 *vmcs12; | 6886 | struct vmcs12 *vmcs12; |
| 6851 | enum vm_entry_failure_code ignored; | ||
| 6852 | struct kvm_vmx_nested_state_data __user *user_vmx_nested_state = | 6887 | struct kvm_vmx_nested_state_data __user *user_vmx_nested_state = |
| 6853 | &user_kvm_nested_state->data.vmx[0]; | 6888 | &user_kvm_nested_state->data.vmx[0]; |
| 6854 | int ret; | 6889 | int ret; |
| @@ -6979,25 +7014,20 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu, · 124 unmodified lines | |||
| 6979 | vmx->nested.mtf_pending = | 7014 | vmx->nested.mtf_pending = |
| 6980 | !!(kvm_state->flags & KVM_STATE_NESTED_MTF_PENDING); | 7015 | !!(kvm_state->flags & KVM_STATE_NESTED_MTF_PENDING); |
| 6981 | 7016 | ||
| 6982 | ret = -EINVAL; | ||
| 6983 | if (nested_cpu_has_shadow_vmcs(vmcs12) && | 7017 | if (nested_cpu_has_shadow_vmcs(vmcs12) && |
| 6984 | vmcs12->vmcs_link_pointer != INVALID_GPA) { | 7018 | vmcs12->vmcs_link_pointer != INVALID_GPA) { |
| 6985 | struct vmcs12 *shadow_vmcs12 = get_shadow_vmcs12(vcpu); | 7019 | struct vmcs12 *shadow_vmcs12 = get_shadow_vmcs12(vcpu); |
| 6986 | 7020 | ||
| 7021 | ret = -EINVAL; | ||
| 6987 | if (kvm_state->size < | 7022 | if (kvm_state->size < |
| 6988 | sizeof(*kvm_state) + | 7023 | sizeof(*kvm_state) + |
| 6989 | sizeof(user_vmx_nested_state->vmcs12) + sizeof(*shadow_vmcs12)) | 7024 | sizeof(user_vmx_nested_state->vmcs12) + sizeof(*shadow_vmcs12)) |
| 6990 | goto error_guest_mode; | 7025 | goto error_guest_mode; |
| 6991 | 7026 | ||
| 7027 | ret = -EFAULT; | ||
| 6992 | if (copy_from_user(shadow_vmcs12, | 7028 | if (copy_from_user(shadow_vmcs12, |
| 6993 | user_vmx_nested_state->shadow_vmcs12, | 7029 | user_vmx_nested_state->shadow_vmcs12, |
| 6994 | sizeof(*shadow_vmcs12))) { | 7030 | sizeof(*shadow_vmcs12))) |
| 6995 | ret = -EFAULT; | ||
| 6996 | goto error_guest_mode; | ||
| 6997 | } | ||
| 6998 | |||
| 6999 | if (shadow_vmcs12->hdr.revision_id != VMCS12_REVISION || | ||
| 7000 | !shadow_vmcs12->hdr.shadow_vmcs) | ||
| 7001 | goto error_guest_mode; | 7031 | goto error_guest_mode; |
| 7002 | } | 7032 | } |
| 7003 | 7033 | ||
| @@ -7008,9 +7038,8 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu, · 4 unmodified lines | |||
| 7008 | kvm_state->hdr.vmx.preemption_timer_deadline; | 7038 | kvm_state->hdr.vmx.preemption_timer_deadline; |
| 7009 | } | 7039 | } |
| 7010 | 7040 | ||
| 7011 | if (nested_vmx_check_controls(vcpu, vmcs12) || | 7041 | ret = nested_vmx_check_restored_vmcs12(vcpu); |
| 7012 | nested_vmx_check_host_state(vcpu, vmcs12) || | 7042 | if (ret < 0) |
| 7013 | nested_vmx_check_guest_state(vcpu, vmcs12, &ignored)) | ||
| 7014 | goto error_guest_mode; | 7043 | goto error_guest_mode; |
| 7015 | 7044 | ||
| 7016 | vmx->nested.dirty_vmcs12 = true; | 7045 | vmx->nested.dirty_vmcs12 = true; |
| @@ -22,6 +22,7 @@ void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps); · 21 unmodified lines | |||
| 22 | void nested_vmx_hardware_unsetup(void); | 22 | void nested_vmx_hardware_unsetup(void); |
| 23 | __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *)); | 23 | __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *)); |
| 24 | void nested_vmx_set_vmcs_shadowing_bitmap(void); | 24 | void nested_vmx_set_vmcs_shadowing_bitmap(void); |
| 25 | int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu); | ||
| 25 | void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu); | 26 | void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu); |
| 26 | enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, | 27 | enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, |
| 27 | bool from_vmentry); | 28 | bool from_vmentry); |
| @@ -1149,7 +1149,7 @@ static void add_atomic_switch_msr(struct vcpu_vmx *vmx, unsigned msr, · 1148 unmodified lines | |||
| 1149 | } | 1149 | } |
| 1150 | 1150 | ||
| 1151 | vmx_add_auto_msr(&m->guest, msr, guest_val, VM_ENTRY_MSR_LOAD_COUNT, kvm); | 1151 | vmx_add_auto_msr(&m->guest, msr, guest_val, VM_ENTRY_MSR_LOAD_COUNT, kvm); |
| 1152 | vmx_add_auto_msr(&m->guest, msr, host_val, VM_EXIT_MSR_LOAD_COUNT, kvm); | 1152 | vmx_add_auto_msr(&m->host, msr, host_val, VM_EXIT_MSR_LOAD_COUNT, kvm); |
| 1153 | } | 1153 | } |
| 1154 | 1154 | ||
| 1155 | static bool update_transition_efer(struct vcpu_vmx *vmx) | 1155 | static bool update_transition_efer(struct vcpu_vmx *vmx) |
| @@ -8528,9 +8528,13 @@ int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) · 7372 unmodified lines | |||
| 8528 | } | 8528 | } |
| 8529 | 8529 | ||
| 8530 | if (vmx->nested.smm.guest_mode) { | 8530 | if (vmx->nested.smm.guest_mode) { |
| 8531 | /* Triple fault if the state is invalid. */ | ||
| 8532 | if (nested_vmx_check_restored_vmcs12(vcpu) < 0) | ||
| 8533 | return 1; | ||
| 8534 | |||
| 8531 | ret = nested_vmx_enter_non_root_mode(vcpu, false); | 8535 | ret = nested_vmx_enter_non_root_mode(vcpu, false); |
| 8532 | if (ret) | 8536 | if (ret != NVMX_VMENTRY_SUCCESS) |
| 8533 | return ret; | 8537 | return 1; |
| 8534 | 8538 | ||
| 8535 | vmx->nested.nested_run_pending = 1; | 8539 | vmx->nested.nested_run_pending = 1; |
| 8536 | vmx->nested.smm.guest_mode = false; | 8540 | vmx->nested.smm.guest_mode = false; |
| @@ -243,7 +243,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_ipiv); · 242 unmodified lines | |||
| 243 | bool __read_mostly enable_device_posted_irqs = true; | 243 | bool __read_mostly enable_device_posted_irqs = true; |
| 244 | EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_device_posted_irqs); | 244 | EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_device_posted_irqs); |
| 245 | 245 | ||
| 246 | const struct _kvm_stats_desc kvm_vm_stats_desc[] = { | 246 | const struct kvm_stats_desc kvm_vm_stats_desc[] = { |
| 247 | KVM_GENERIC_VM_STATS(), | 247 | KVM_GENERIC_VM_STATS(), |
| 248 | STATS_DESC_COUNTER(VM, mmu_shadow_zapped), | 248 | STATS_DESC_COUNTER(VM, mmu_shadow_zapped), |
| 249 | STATS_DESC_COUNTER(VM, mmu_pte_write), | 249 | STATS_DESC_COUNTER(VM, mmu_pte_write), |
| @@ -269,7 +269,7 @@ const struct kvm_stats_header kvm_vm_stats_header = { · 19 unmodified lines | |||
| 269 | sizeof(kvm_vm_stats_desc), | 269 | sizeof(kvm_vm_stats_desc), |
| 270 | }; | 270 | }; |
| 271 | 271 | ||
| 272 | const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { | 272 | const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { |
| 273 | KVM_GENERIC_VCPU_STATS(), | 273 | KVM_GENERIC_VCPU_STATS(), |
| 274 | STATS_DESC_COUNTER(VCPU, pf_taken), | 274 | STATS_DESC_COUNTER(VCPU, pf_taken), |
| 275 | STATS_DESC_COUNTER(VCPU, pf_fixed), | 275 | STATS_DESC_COUNTER(VCPU, pf_fixed), |
| @@ -4,6 +4,8 @@ KCOV_INSTRUMENT_tlb.o := n · 3 unmodified lines | |||
| 4 | KCOV_INSTRUMENT_mem_encrypt.o := n | 4 | KCOV_INSTRUMENT_mem_encrypt.o := n |
| 5 | KCOV_INSTRUMENT_mem_encrypt_amd.o := n | 5 | KCOV_INSTRUMENT_mem_encrypt_amd.o := n |
| 6 | KCOV_INSTRUMENT_pgprot.o := n | 6 | KCOV_INSTRUMENT_pgprot.o := n |
| 7 | # See the "Disable KCOV" comment in arch/x86/kernel/Makefile. | ||
| 8 | KCOV_INSTRUMENT_physaddr.o := n | ||
| 7 | 9 | ||
| 8 | KASAN_SANITIZE_mem_encrypt.o := n | 10 | KASAN_SANITIZE_mem_encrypt.o := n |
| 9 | KASAN_SANITIZE_mem_encrypt_amd.o := n | 11 | KASAN_SANITIZE_mem_encrypt_amd.o := n |
| @@ -424,7 +424,7 @@ void __init efi_unmap_boot_services(void) · 423 unmodified lines | |||
| 424 | if (efi_enabled(EFI_DBG)) | 424 | if (efi_enabled(EFI_DBG)) |
| 425 | return; | 425 | return; |
| 426 | 426 | ||
| 427 | sz = sizeof(*ranges_to_free) * efi.memmap.nr_map + 1; | 427 | sz = sizeof(*ranges_to_free) * (efi.memmap.nr_map + 1); |
| 428 | ranges_to_free = kzalloc(sz, GFP_KERNEL); | 428 | ranges_to_free = kzalloc(sz, GFP_KERNEL); |
| 429 | if (!ranges_to_free) { | 429 | if (!ranges_to_free) { |
| 430 | pr_err("Failed to allocate storage for freeable EFI regions\n"); | 430 | pr_err("Failed to allocate storage for freeable EFI regions\n"); |
| @@ -28,8 +28,10 @@ static const struct software_node geode_gpio_keys_node = { · 27 unmodified lines | |||
| 28 | .properties = geode_gpio_keys_props, | 28 | .properties = geode_gpio_keys_props, |
| 29 | }; | 29 | }; |
| 30 | 30 | ||
| 31 | static struct property_entry geode_restart_key_props[] = { | 31 | static struct software_node_ref_args geode_restart_gpio_ref; |
| 32 | { /* Placeholder for GPIO property */ }, | 32 | |
| 33 | static const struct property_entry geode_restart_key_props[] = { | ||
| 34 | PROPERTY_ENTRY_REF_ARRAY_LEN("gpios", &geode_restart_gpio_ref, 1), | ||
| 33 | PROPERTY_ENTRY_U32("linux,code", KEY_RESTART), | 35 | PROPERTY_ENTRY_U32("linux,code", KEY_RESTART), |
| 34 | PROPERTY_ENTRY_STRING("label", "Reset button"), | 36 | PROPERTY_ENTRY_STRING("label", "Reset button"), |
| 35 | PROPERTY_ENTRY_U32("debounce-interval", 100), | 37 | PROPERTY_ENTRY_U32("debounce-interval", 100), |
| @@ -64,8 +66,7 @@ int __init geode_create_restart_key(unsigned int pin) · 28 unmodified lines | |||
| 64 | struct platform_device *pd; | 66 | struct platform_device *pd; |
| 65 | int err; | 67 | int err; |
| 66 | 68 | ||
| 67 | geode_restart_key_props[0] = PROPERTY_ENTRY_GPIO("gpios", | 69 | geode_restart_gpio_ref = SOFTWARE_NODE_REFERENCE(&geode_gpiochip_node, |
| 68 | &geode_gpiochip_node, | ||
| 69 | pin, GPIO_ACTIVE_LOW); | 70 | pin, GPIO_ACTIVE_LOW); |
| 70 | 71 | ||
| 71 | err = software_node_register_node_group(geode_gpio_keys_swnodes); | 72 | err = software_node_register_node_group(geode_gpio_keys_swnodes); |
| @@ -99,6 +100,7 @@ int __init geode_create_leds(const char *label, const struct geode_led *leds, · 27 unmodified lines | |||
| 99 | const struct software_node *group[MAX_LEDS + 2] = { 0 }; | 100 | const struct software_node *group[MAX_LEDS + 2] = { 0 }; |
| 100 | struct software_node *swnodes; | 101 | struct software_node *swnodes; |
| 101 | struct property_entry *props; | 102 | struct property_entry *props; |
| 103 | struct software_node_ref_args *gpio_refs; | ||
| 102 | struct platform_device_info led_info = { | 104 | struct platform_device_info led_info = { |
| 103 | .name = "leds-gpio", | 105 | .name = "leds-gpio", |
| 104 | .id = PLATFORM_DEVID_NONE, | 106 | .id = PLATFORM_DEVID_NONE, |
| @@ -127,6 +129,12 @@ int __init geode_create_leds(const char *label, const struct geode_led *leds, · 22 unmodified lines | |||
| 127 | goto err_free_swnodes; | 129 | goto err_free_swnodes; |
| 128 | } | 130 | } |
| 129 | 131 | ||
| 132 | gpio_refs = kzalloc_objs(*gpio_refs, n_leds); | ||
| 133 | if (!gpio_refs) { | ||
| 134 | err = -ENOMEM; | ||
| 135 | goto err_free_props; | ||
| 136 | } | ||
| 137 | |||
| 130 | group[0] = &geode_gpio_leds_node; | 138 | group[0] = &geode_gpio_leds_node; |
| 131 | for (i = 0; i < n_leds; i++) { | 139 | for (i = 0; i < n_leds; i++) { |
| 132 | node_name = kasprintf(GFP_KERNEL, "%s:%d", label, i); | 140 | node_name = kasprintf(GFP_KERNEL, "%s:%d", label, i); |
| @@ -135,9 +143,11 @@ int __init geode_create_leds(const char *label, const struct geode_led *leds, · 2 unmodified lines | |||
| 135 | goto err_free_names; | 143 | goto err_free_names; |
| 136 | } | 144 | } |
| 137 | 145 | ||
| 146 | gpio_refs[i] = SOFTWARE_NODE_REFERENCE(&geode_gpiochip_node, | ||
| 147 | leds[i].pin, | ||
| 148 | GPIO_ACTIVE_LOW); | ||
| 138 | props[i * 3 + 0] = | 149 | props[i * 3 + 0] = |
| 139 | PROPERTY_ENTRY_GPIO("gpios", &geode_gpiochip_node, | 150 | PROPERTY_ENTRY_REF_ARRAY_LEN("gpios", &gpio_refs[i], 1); |
| 140 | leds[i].pin, GPIO_ACTIVE_LOW); | ||
| 141 | props[i * 3 + 1] = | 151 | props[i * 3 + 1] = |
| 142 | PROPERTY_ENTRY_STRING("linux,default-trigger", | 152 | PROPERTY_ENTRY_STRING("linux,default-trigger", |
| 143 | leds[i].default_on ? | 153 | leds[i].default_on ? |
| @@ -171,6 +181,8 @@ err_unregister_group: · 27 unmodified lines | |||
| 171 | err_free_names: | 181 | err_free_names: |
| 172 | while (--i >= 0) | 182 | while (--i >= 0) |
| 173 | kfree(swnodes[i].name); | 183 | kfree(swnodes[i].name); |
| 184 | kfree(gpio_refs); | ||
| 185 | err_free_props: | ||
| 174 | kfree(props); | 186 | kfree(props); |
| 175 | err_free_swnodes: | 187 | err_free_swnodes: |
| 176 | kfree(swnodes); | 188 | kfree(swnodes); |
| @@ -623,8 +623,10 @@ static int af_alg_alloc_tsgl(struct sock *sk) · 622 unmodified lines | |||
| 623 | sg_init_table(sgl->sg, MAX_SGL_ENTS + 1); | 623 | sg_init_table(sgl->sg, MAX_SGL_ENTS + 1); |
| 624 | sgl->cur = 0; | 624 | sgl->cur = 0; |
| 625 | 625 | ||
| 626 | if (sg) | 626 | if (sg) { |
| 627 | sg_unmark_end(sg + MAX_SGL_ENTS - 1); | ||
| 627 | sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg); | 628 | sg_chain(sg, MAX_SGL_ENTS + 1, sgl->sg); |
| 629 | } | ||
| 628 | 630 | ||
| 629 | list_add_tail(&sgl->list, &ctx->tsgl_list); | 631 | list_add_tail(&sgl->list, &ctx->tsgl_list); |
| 630 | } | 632 | } |
| @@ -635,15 +637,13 @@ static int af_alg_alloc_tsgl(struct sock *sk) · 4 unmodified lines | |||
| 635 | /** | 637 | /** |
| 636 | * af_alg_count_tsgl - Count number of TX SG entries | 638 | * af_alg_count_tsgl - Count number of TX SG entries |
| 637 | * | 639 | * |
| 638 | * The counting starts from the beginning of the SGL to @bytes. If | 640 | * The counting starts from the beginning of the SGL to @bytes. |
| 639 | * an @offset is provided, the counting of the SG entries starts at the @offset. | ||
| 640 | * | 641 | * |
| 641 | * @sk: socket of connection to user space | 642 | * @sk: socket of connection to user space |
| 642 | * @bytes: Count the number of SG entries holding given number of bytes. | 643 | * @bytes: Count the number of SG entries holding given number of bytes. |
| 643 | * @offset: Start the counting of SG entries from the given offset. | ||
| 644 | * Return: Number of TX SG entries found given the constraints | 644 | * Return: Number of TX SG entries found given the constraints |
| 645 | */ | 645 | */ |
| 646 | unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset) | 646 | unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes) |
| 647 | { | 647 | { |
| 648 | const struct alg_sock *ask = alg_sk(sk); | 648 | const struct alg_sock *ask = alg_sk(sk); |
| 649 | const struct af_alg_ctx *ctx = ask->private; | 649 | const struct af_alg_ctx *ctx = ask->private; |
| @@ -658,25 +658,11 @@ unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset) · 8 unmodified lines | |||
| 658 | const struct scatterlist *sg = sgl->sg; | 658 | const struct scatterlist *sg = sgl->sg; |
| 659 | 659 | ||
| 660 | for (i = 0; i < sgl->cur; i++) { | 660 | for (i = 0; i < sgl->cur; i++) { |
| 661 | size_t bytes_count; | ||
| 662 | |||
| 663 | /* Skip offset */ | ||
| 664 | if (offset >= sg[i].length) { | ||
| 665 | offset -= sg[i].length; | ||
| 666 | bytes -= sg[i].length; | ||
| 667 | continue; | ||
| 668 | } | ||
| 669 | |||
| 670 | bytes_count = sg[i].length - offset; | ||
| 671 | |||
| 672 | offset = 0; | ||
| 673 | sgl_count++; | 661 | sgl_count++; |
| 674 | 662 | if (sg[i].length >= bytes) | |
| 675 | /* If we have seen requested number of bytes, stop */ | ||
| 676 | if (bytes_count >= bytes) | ||
| 677 | return sgl_count; | 663 | return sgl_count; |
| 678 | 664 | ||
| 679 | bytes -= bytes_count; | 665 | bytes -= sg[i].length; |
| 680 | } | 666 | } |
| 681 | } | 667 | } |
| 682 | 668 | ||
| @@ -688,19 +674,14 @@ EXPORT_SYMBOL_GPL(af_alg_count_tsgl); · 5 unmodified lines | |||
| 688 | * af_alg_pull_tsgl - Release the specified buffers from TX SGL | 674 | * af_alg_pull_tsgl - Release the specified buffers from TX SGL |
| 689 | * | 675 | * |
| 690 | * If @dst is non-null, reassign the pages to @dst. The caller must release | 676 | * If @dst is non-null, reassign the pages to @dst. The caller must release |
| 691 | * the pages. If @dst_offset is given only reassign the pages to @dst starting | 677 | * the pages. |
| 692 | * at the @dst_offset (byte). The caller must ensure that @dst is large | ||
| 693 | * enough (e.g. by using af_alg_count_tsgl with the same offset). | ||
| 694 | * | 678 | * |
| 695 | * @sk: socket of connection to user space | 679 | * @sk: socket of connection to user space |
| 696 | * @used: Number of bytes to pull from TX SGL | 680 | * @used: Number of bytes to pull from TX SGL |
| 697 | * @dst: If non-NULL, buffer is reassigned to dst SGL instead of releasing. The | 681 | * @dst: If non-NULL, buffer is reassigned to dst SGL instead of releasing. The |
| 698 | * caller must release the buffers in dst. | 682 | * caller must release the buffers in dst. |
| 699 | * @dst_offset: Reassign the TX SGL from given offset. All buffers before | ||
| 700 | * reaching the offset is released. | ||
| 701 | */ | 683 | */ |
| 702 | void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst, | 684 | void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst) |
| 703 | size_t dst_offset) | ||
| 704 | { | 685 | { |
| 705 | struct alg_sock *ask = alg_sk(sk); | 686 | struct alg_sock *ask = alg_sk(sk); |
| 706 | struct af_alg_ctx *ctx = ask->private; | 687 | struct af_alg_ctx *ctx = ask->private; |
| @@ -725,18 +706,10 @@ void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst, · 18 unmodified lines | |||
| 725 | * SG entries in dst. | 706 | * SG entries in dst. |
| 726 | */ | 707 | */ |
| 727 | if (dst) { | 708 | if (dst) { |
| 728 | if (dst_offset >= plen) { | 709 | /* reassign page to dst after offset */ |
| 729 | /* discard page before offset */ | 710 | get_page(page); |
| 730 | dst_offset -= plen; | 711 | sg_set_page(dst + j, page, plen, sg[i].offset); |
| 731 | } else { | 712 | j++; |
| 732 | /* reassign page to dst after offset */ | ||
| 733 | get_page(page); | ||
| 734 | sg_set_page(dst + j, page, | ||
| 735 | plen - dst_offset, | ||
| 736 | sg[i].offset + dst_offset); | ||
| 737 | dst_offset = 0; | ||
| 738 | j++; | ||
| 739 | } | ||
| 740 | } | 713 | } |
| 741 | 714 | ||
| 742 | sg[i].length -= plen; | 715 | sg[i].length -= plen; |
| @@ -26,7 +26,6 @@ · 25 unmodified lines | |||
| 26 | #include <crypto/internal/aead.h> | 26 | #include <crypto/internal/aead.h> |
| 27 | #include <crypto/scatterwalk.h> | 27 | #include <crypto/scatterwalk.h> |
| 28 | #include <crypto/if_alg.h> | 28 | #include <crypto/if_alg.h> |
| 29 | #include <crypto/skcipher.h> | ||
| 30 | #include <linux/init.h> | 29 | #include <linux/init.h> |
| 31 | #include <linux/list.h> | 30 | #include <linux/list.h> |
| 32 | #include <linux/kernel.h> | 31 | #include <linux/kernel.h> |
| @@ -72,9 +71,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, · 39 unmodified lines | |||
| 72 | struct alg_sock *pask = alg_sk(psk); | 71 | struct alg_sock *pask = alg_sk(psk); |
| 73 | struct af_alg_ctx *ctx = ask->private; | 72 | struct af_alg_ctx *ctx = ask->private; |
| 74 | struct crypto_aead *tfm = pask->private; | 73 | struct crypto_aead *tfm = pask->private; |
| 75 | unsigned int i, as = crypto_aead_authsize(tfm); | 74 | unsigned int as = crypto_aead_authsize(tfm); |
| 76 | struct af_alg_async_req *areq; | 75 | struct af_alg_async_req *areq; |
| 77 | struct af_alg_tsgl *tsgl, *tmp; | ||
| 78 | struct scatterlist *rsgl_src, *tsgl_src = NULL; | 76 | struct scatterlist *rsgl_src, *tsgl_src = NULL; |
| 79 | int err = 0; | 77 | int err = 0; |
| 80 | size_t used = 0; /* [in] TX bufs to be en/decrypted */ | 78 | size_t used = 0; /* [in] TX bufs to be en/decrypted */ |
| @@ -154,23 +152,24 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, · 73 unmodified lines | |||
| 154 | outlen -= less; | 152 | outlen -= less; |
| 155 | } | 153 | } |
| 156 | 154 | ||
| 155 | /* | ||
| 156 | * Create a per request TX SGL for this request which tracks the | ||
| 157 | * SG entries from the global TX SGL. | ||
| 158 | */ | ||
| 157 | processed = used + ctx->aead_assoclen; | 159 | processed = used + ctx->aead_assoclen; |
| 158 | list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) { | 160 | areq->tsgl_entries = af_alg_count_tsgl(sk, processed); |
| 159 | for (i = 0; i < tsgl->cur; i++) { | 161 | if (!areq->tsgl_entries) |
| 160 | struct scatterlist *process_sg = tsgl->sg + i; | 162 | areq->tsgl_entries = 1; |
| 161 | 163 | areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), | |
| 162 | if (!(process_sg->length) || !sg_page(process_sg)) | 164 | areq->tsgl_entries), |
| 163 | continue; | 165 | GFP_KERNEL); |
| 164 | tsgl_src = process_sg; | 166 | if (!areq->tsgl) { |
| 165 | break; | 167 | err = -ENOMEM; |
| 166 | } | ||
| 167 | if (tsgl_src) | ||
| 168 | break; | ||
| 169 | } | ||
| 170 | if (processed && !tsgl_src) { | ||
| 171 | err = -EFAULT; | ||
| 172 | goto free; | 168 | goto free; |
| 173 | } | 169 | } |
| 170 | sg_init_table(areq->tsgl, areq->tsgl_entries); | ||
| 171 | af_alg_pull_tsgl(sk, processed, areq->tsgl); | ||
| 172 | tsgl_src = areq->tsgl; | ||
| 174 | 173 | ||
| 175 | /* | 174 | /* |
| 176 | * Copy of AAD from source to destination | 175 | * Copy of AAD from source to destination |
| @@ -179,76 +178,15 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, · 2 unmodified lines | |||
| 179 | * when user space uses an in-place cipher operation, the kernel | 178 | * when user space uses an in-place cipher operation, the kernel |
| 180 | * will copy the data as it does not see whether such in-place operation | 179 | * will copy the data as it does not see whether such in-place operation |
| 181 | * is initiated. | 180 | * is initiated. |
| 182 | * | ||
| 183 | * To ensure efficiency, the following implementation ensure that the | ||
| 184 | * ciphers are invoked to perform a crypto operation in-place. This | ||
| 185 | * is achieved by memory management specified as follows. | ||
| 186 | */ | 181 | */ |
| 187 | 182 | ||
| 188 | /* Use the RX SGL as source (and destination) for crypto op. */ | 183 | /* Use the RX SGL as source (and destination) for crypto op. */ |
| 189 | rsgl_src = areq->first_rsgl.sgl.sgt.sgl; | 184 | rsgl_src = areq->first_rsgl.sgl.sgt.sgl; |
| 190 | 185 | ||
| 191 | if (ctx->enc) { | 186 | memcpy_sglist(rsgl_src, tsgl_src, ctx->aead_assoclen); |
| 192 | /* | ||
| 193 | * Encryption operation - The in-place cipher operation is | ||
| 194 | * achieved by the following operation: | ||
| 195 | * | ||
| 196 | * TX SGL: AAD || PT | ||
| 197 | * | | | ||
| 198 | * | copy | | ||
| 199 | * v v | ||
| 200 | * RX SGL: AAD || PT || Tag | ||
| 201 | */ | ||
| 202 | memcpy_sglist(areq->first_rsgl.sgl.sgt.sgl, tsgl_src, | ||
| 203 | processed); | ||
| 204 | af_alg_pull_tsgl(sk, processed, NULL, 0); | ||
| 205 | } else { | ||
| 206 | /* | ||
| 207 | * Decryption operation - To achieve an in-place cipher | ||
| 208 | * operation, the following SGL structure is used: | ||
| 209 | * | ||
| 210 | * TX SGL: AAD || CT || Tag | ||
| 211 | * | | ^ | ||
| 212 | * | copy | | Create SGL link. | ||
| 213 | * v v | | ||
| 214 | * RX SGL: AAD || CT ----+ | ||
| 215 | */ | ||
| 216 | |||
| 217 | /* Copy AAD || CT to RX SGL buffer for in-place operation. */ | ||
| 218 | memcpy_sglist(areq->first_rsgl.sgl.sgt.sgl, tsgl_src, outlen); | ||
| 219 | |||
| 220 | /* Create TX SGL for tag and chain it to RX SGL. */ | ||
| 221 | areq->tsgl_entries = af_alg_count_tsgl(sk, processed, | ||
| 222 | processed - as); | ||
| 223 | if (!areq->tsgl_entries) | ||
| 224 | areq->tsgl_entries = 1; | ||
| 225 | areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), | ||
| 226 | areq->tsgl_entries), | ||
| 227 | GFP_KERNEL); | ||
| 228 | if (!areq->tsgl) { | ||
| 229 | err = -ENOMEM; | ||
| 230 | goto free; | ||
| 231 | } | ||
| 232 | sg_init_table(areq->tsgl, areq->tsgl_entries); | ||
| 233 | |||
| 234 | /* Release TX SGL, except for tag data and reassign tag data. */ | ||
| 235 | af_alg_pull_tsgl(sk, processed, areq->tsgl, processed - as); | ||
| 236 | |||
| 237 | /* chain the areq TX SGL holding the tag with RX SGL */ | ||
| 238 | if (usedpages) { | ||
| 239 | /* RX SGL present */ | ||
| 240 | struct af_alg_sgl *sgl_prev = &areq->last_rsgl->sgl; | ||
| 241 | struct scatterlist *sg = sgl_prev->sgt.sgl; | ||
| 242 | |||
| 243 | sg_unmark_end(sg + sgl_prev->sgt.nents - 1); | ||
| 244 | sg_chain(sg, sgl_prev->sgt.nents + 1, areq->tsgl); | ||
| 245 | } else | ||
| 246 | /* no RX SGL present (e.g. authentication only) */ | ||
| 247 | rsgl_src = areq->tsgl; | ||
| 248 | } | ||
| 249 | 187 | ||
| 250 | /* Initialize the crypto operation */ | 188 | /* Initialize the crypto operation */ |
| 251 | aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src, | 189 | aead_request_set_crypt(&areq->cra_u.aead_req, tsgl_src, |
| 252 | areq->first_rsgl.sgl.sgt.sgl, used, ctx->iv); | 190 | areq->first_rsgl.sgl.sgt.sgl, used, ctx->iv); |
| 253 | aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen); | 191 | aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen); |
| 254 | aead_request_set_tfm(&areq->cra_u.aead_req, tfm); | 192 | aead_request_set_tfm(&areq->cra_u.aead_req, tfm); |
| @@ -450,7 +388,7 @@ static void aead_sock_destruct(struct sock *sk) · 195 unmodified lines | |||
| 450 | struct crypto_aead *tfm = pask->private; | 388 | struct crypto_aead *tfm = pask->private; |
| 451 | unsigned int ivlen = crypto_aead_ivsize(tfm); | 389 | unsigned int ivlen = crypto_aead_ivsize(tfm); |
| 452 | 390 | ||
| 453 | af_alg_pull_tsgl(sk, ctx->used, NULL, 0); | 391 | af_alg_pull_tsgl(sk, ctx->used, NULL); |
| 454 | sock_kzfree_s(sk, ctx->iv, ivlen); | 392 | sock_kzfree_s(sk, ctx->iv, ivlen); |
| 455 | sock_kfree_s(sk, ctx, ctx->len); | 393 | sock_kfree_s(sk, ctx, ctx->len); |
| 456 | af_alg_release_parent(sk); | 394 | af_alg_release_parent(sk); |
| @@ -138,7 +138,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, · 137 unmodified lines | |||
| 138 | * Create a per request TX SGL for this request which tracks the | 138 | * Create a per request TX SGL for this request which tracks the |
| 139 | * SG entries from the global TX SGL. | 139 | * SG entries from the global TX SGL. |
| 140 | */ | 140 | */ |
| 141 | areq->tsgl_entries = af_alg_count_tsgl(sk, len, 0); | 141 | areq->tsgl_entries = af_alg_count_tsgl(sk, len); |
| 142 | if (!areq->tsgl_entries) | 142 | if (!areq->tsgl_entries) |
| 143 | areq->tsgl_entries = 1; | 143 | areq->tsgl_entries = 1; |
| 144 | areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), | 144 | areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), |
| @@ -149,7 +149,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, · 4 unmodified lines | |||
| 149 | goto free; | 149 | goto free; |
| 150 | } | 150 | } |
| 151 | sg_init_table(areq->tsgl, areq->tsgl_entries); | 151 | sg_init_table(areq->tsgl, areq->tsgl_entries); |
| 152 | af_alg_pull_tsgl(sk, len, areq->tsgl, 0); | 152 | af_alg_pull_tsgl(sk, len, areq->tsgl); |
| 153 | 153 | ||
| 154 | /* Initialize the crypto operation */ | 154 | /* Initialize the crypto operation */ |
| 155 | skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm); | 155 | skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm); |
| @@ -363,7 +363,7 @@ static void skcipher_sock_destruct(struct sock *sk) · 207 unmodified lines | |||
| 363 | struct alg_sock *pask = alg_sk(psk); | 363 | struct alg_sock *pask = alg_sk(psk); |
| 364 | struct crypto_skcipher *tfm = pask->private; | 364 | struct crypto_skcipher *tfm = pask->private; |
| 365 | 365 | ||
| 366 | af_alg_pull_tsgl(sk, ctx->used, NULL, 0); | 366 | af_alg_pull_tsgl(sk, ctx->used, NULL); |
| 367 | sock_kzfree_s(sk, ctx->iv, crypto_skcipher_ivsize(tfm)); | 367 | sock_kzfree_s(sk, ctx->iv, crypto_skcipher_ivsize(tfm)); |
| 368 | if (ctx->state) | 368 | if (ctx->state) |
| 369 | sock_kzfree_s(sk, ctx->state, crypto_skcipher_statesize(tfm)); | 369 | sock_kzfree_s(sk, ctx->state, crypto_skcipher_statesize(tfm)); |
| @@ -207,6 +207,7 @@ static int crypto_authenc_esn_decrypt_tail(struct aead_request *req, · 206 unmodified lines | |||
| 207 | u8 *ohash = areq_ctx->tail; | 207 | u8 *ohash = areq_ctx->tail; |
| 208 | unsigned int cryptlen = req->cryptlen - authsize; | 208 | unsigned int cryptlen = req->cryptlen - authsize; |
| 209 | unsigned int assoclen = req->assoclen; | 209 | unsigned int assoclen = req->assoclen; |
| 210 | struct scatterlist *src = req->src; | ||
| 210 | struct scatterlist *dst = req->dst; | 211 | struct scatterlist *dst = req->dst; |
| 211 | u8 *ihash = ohash + crypto_ahash_digestsize(auth); | 212 | u8 *ihash = ohash + crypto_ahash_digestsize(auth); |
| 212 | u32 tmp[2]; | 213 | u32 tmp[2]; |
| @@ -214,23 +215,27 @@ static int crypto_authenc_esn_decrypt_tail(struct aead_request *req, · 1 unmodified line | |||
| 214 | if (!authsize) | 215 | if (!authsize) |
| 215 | goto decrypt; | 216 | goto decrypt; |
| 216 | 217 | ||
| 217 | /* Move high-order bits of sequence number back. */ | 218 | if (src == dst) { |
| 218 | scatterwalk_map_and_copy(tmp, dst, 4, 4, 0); | 219 | /* Move high-order bits of sequence number back. */ |
| 219 | scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 0); | 220 | scatterwalk_map_and_copy(tmp, dst, 4, 4, 0); |
| 220 | scatterwalk_map_and_copy(tmp, dst, 0, 8, 1); | 221 | scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 0); |
| 222 | scatterwalk_map_and_copy(tmp, dst, 0, 8, 1); | ||
| 223 | } else | ||
| 224 | memcpy_sglist(dst, src, assoclen); | ||
| 221 | 225 | ||
| 222 | if (crypto_memneq(ihash, ohash, authsize)) | 226 | if (crypto_memneq(ihash, ohash, authsize)) |
| 223 | return -EBADMSG; | 227 | return -EBADMSG; |
| 224 | 228 | ||
| 225 | decrypt: | 229 | decrypt: |
| 226 | 230 | ||
| 227 | sg_init_table(areq_ctx->dst, 2); | 231 | if (src != dst) |
| 232 | src = scatterwalk_ffwd(areq_ctx->src, src, assoclen); | ||
| 228 | dst = scatterwalk_ffwd(areq_ctx->dst, dst, assoclen); | 233 | dst = scatterwalk_ffwd(areq_ctx->dst, dst, assoclen); |
| 229 | 234 | ||
| 230 | skcipher_request_set_tfm(skreq, ctx->enc); | 235 | skcipher_request_set_tfm(skreq, ctx->enc); |
| 231 | skcipher_request_set_callback(skreq, flags, | 236 | skcipher_request_set_callback(skreq, flags, |
| 232 | req->base.complete, req->base.data); | 237 | req->base.complete, req->base.data); |
| 233 | skcipher_request_set_crypt(skreq, dst, dst, cryptlen, req->iv); | 238 | skcipher_request_set_crypt(skreq, src, dst, cryptlen, req->iv); |
| 234 | 239 | ||
| 235 | return crypto_skcipher_decrypt(skreq); | 240 | return crypto_skcipher_decrypt(skreq); |
| 236 | } | 241 | } |
| @@ -255,6 +260,7 @@ static int crypto_authenc_esn_decrypt(struct aead_request *req) · 18 unmodified lines | |||
| 255 | unsigned int assoclen = req->assoclen; | 260 | unsigned int assoclen = req->assoclen; |
| 256 | unsigned int cryptlen = req->cryptlen; | 261 | unsigned int cryptlen = req->cryptlen; |
| 257 | u8 *ihash = ohash + crypto_ahash_digestsize(auth); | 262 | u8 *ihash = ohash + crypto_ahash_digestsize(auth); |
| 263 | struct scatterlist *src = req->src; | ||
| 258 | struct scatterlist *dst = req->dst; | 264 | struct scatterlist *dst = req->dst; |
| 259 | u32 tmp[2]; | 265 | u32 tmp[2]; |
| 260 | int err; | 266 | int err; |
| @@ -262,24 +268,28 @@ static int crypto_authenc_esn_decrypt(struct aead_request *req) · 1 unmodified line | |||
| 262 | if (assoclen < 8) | 268 | if (assoclen < 8) |
| 263 | return -EINVAL; | 269 | return -EINVAL; |
| 264 | 270 | ||
| 265 | cryptlen -= authsize; | 271 | if (!authsize) |
| 266 | 272 | goto tail; | |
| 267 | if (req->src != dst) | ||
| 268 | memcpy_sglist(dst, req->src, assoclen + cryptlen); | ||
| 269 | 273 | ||
| 274 | cryptlen -= authsize; | ||
| 270 | scatterwalk_map_and_copy(ihash, req->src, assoclen + cryptlen, | 275 | scatterwalk_map_and_copy(ihash, req->src, assoclen + cryptlen, |
| 271 | authsize, 0); | 276 | authsize, 0); |
| 272 | 277 | ||
| 273 | if (!authsize) | ||
| 274 | goto tail; | ||
| 275 | |||
| 276 | /* Move high-order bits of sequence number to the end. */ | 278 | /* Move high-order bits of sequence number to the end. */ |
| 277 | scatterwalk_map_and_copy(tmp, dst, 0, 8, 0); | 279 | scatterwalk_map_and_copy(tmp, src, 0, 8, 0); |
| 278 | scatterwalk_map_and_copy(tmp, dst, 4, 4, 1); | 280 | if (src == dst) { |
| 279 | scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 1); | 281 | scatterwalk_map_and_copy(tmp, dst, 4, 4, 1); |
| 280 | 282 | scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen, 4, 1); | |
| 281 | sg_init_table(areq_ctx->dst, 2); | 283 | dst = scatterwalk_ffwd(areq_ctx->dst, dst, 4); |
| 282 | dst = scatterwalk_ffwd(areq_ctx->dst, dst, 4); | 284 | } else { |
| 285 | scatterwalk_map_and_copy(tmp, dst, 0, 4, 1); | ||
| 286 | scatterwalk_map_and_copy(tmp + 1, dst, assoclen + cryptlen - 4, 4, 1); | ||
| 287 | |||
| 288 | src = scatterwalk_ffwd(areq_ctx->src, src, 8); | ||
| 289 | dst = scatterwalk_ffwd(areq_ctx->dst, dst, 4); | ||
| 290 | memcpy_sglist(dst, src, assoclen + cryptlen - 8); | ||
| 291 | dst = req->dst; | ||
| 292 | } | ||
| 283 | 293 | ||
| 284 | ahash_request_set_tfm(ahreq, auth); | 294 | ahash_request_set_tfm(ahreq, auth); |
| 285 | ahash_request_set_crypt(ahreq, dst, ohash, assoclen + cryptlen); | 295 | ahash_request_set_crypt(ahreq, dst, ohash, assoclen + cryptlen); |
| @@ -164,18 +164,21 @@ static int deflate_decompress_one(struct acomp_req *req, · 163 unmodified lines | |||
| 164 | 164 | ||
| 165 | do { | 165 | do { |
| 166 | unsigned int dcur; | 166 | unsigned int dcur; |
| 167 | unsigned long avail_in; | ||
| 167 | 168 | ||
| 168 | dcur = acomp_walk_next_dst(&walk); | 169 | dcur = acomp_walk_next_dst(&walk); |
| 169 | if (!dcur) { | ||
| 170 | out_of_space = true; | ||
| 171 | break; | ||
| 172 | } | ||
| 173 | 170 | ||
| 174 | stream->avail_out = dcur; | 171 | stream->avail_out = dcur; |
| 175 | stream->next_out = walk.dst.virt.addr; | 172 | stream->next_out = walk.dst.virt.addr; |
| 173 | avail_in = stream->avail_in; | ||
| 176 | 174 | ||
| 177 | ret = zlib_inflate(stream, Z_NO_FLUSH); | 175 | ret = zlib_inflate(stream, Z_NO_FLUSH); |
| 178 | 176 | ||
| 177 | if (!dcur && avail_in == stream->avail_in) { | ||
| 178 | out_of_space = true; | ||
| 179 | break; | ||
| 180 | } | ||
| 181 | |||
| 179 | dcur -= stream->avail_out; | 182 | dcur -= stream->avail_out; |
| 180 | acomp_walk_done_dst(&walk, dcur); | 183 | acomp_walk_done_dst(&walk, dcur); |
| 181 | } while (ret == Z_OK && stream->avail_in); | 184 | } while (ret == Z_OK && stream->avail_in); |
| @@ -11,6 +11,7 @@ amdxdna-y := \ · 10 unmodified lines | |||
| 11 | aie2_solver.o \ | 11 | aie2_solver.o \ |
| 12 | amdxdna_ctx.o \ | 12 | amdxdna_ctx.o \ |
| 13 | amdxdna_gem.o \ | 13 | amdxdna_gem.o \ |
| 14 | amdxdna_iommu.o \ | ||
| 14 | amdxdna_mailbox.o \ | 15 | amdxdna_mailbox.o \ |
| 15 | amdxdna_mailbox_helper.o \ | 16 | amdxdna_mailbox_helper.o \ |
| 16 | amdxdna_pci_drv.o \ | 17 | amdxdna_pci_drv.o \ |
| @@ -29,6 +29,16 @@ MODULE_PARM_DESC(force_cmdlist, "Force use command list (Default true)"); · 28 unmodified lines | |||
| 29 | 29 | ||
| 30 | #define HWCTX_MAX_TIMEOUT 60000 /* milliseconds */ | 30 | #define HWCTX_MAX_TIMEOUT 60000 /* milliseconds */ |
| 31 | 31 | ||
| 32 | struct aie2_ctx_health { | ||
| 33 | struct amdxdna_ctx_health header; | ||
| 34 | u32 txn_op_idx; | ||
| 35 | u32 ctx_pc; | ||
| 36 | u32 fatal_error_type; | ||
| 37 | u32 fatal_error_exception_type; | ||
| 38 | u32 fatal_error_exception_pc; | ||
| 39 | u32 fatal_error_app_module; | ||
| 40 | }; | ||
| 41 | |||
| 32 | static void aie2_job_release(struct kref *ref) | 42 | static void aie2_job_release(struct kref *ref) |
| 33 | { | 43 | { |
| 34 | struct amdxdna_sched_job *job; | 44 | struct amdxdna_sched_job *job; |
| @@ -39,6 +49,7 @@ static void aie2_job_release(struct kref *ref) · 4 unmodified lines | |||
| 39 | wake_up(&job->hwctx->priv->job_free_wq); | 49 | wake_up(&job->hwctx->priv->job_free_wq); |
| 40 | if (job->out_fence) | 50 | if (job->out_fence) |
| 41 | dma_fence_put(job->out_fence); | 51 | dma_fence_put(job->out_fence); |
| 52 | kfree(job->aie2_job_health); | ||
| 42 | kfree(job); | 53 | kfree(job); |
| 43 | } | 54 | } |
| 44 | 55 | ||
| @@ -68,7 +79,8 @@ static int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hw · 23 unmodified lines | |||
| 68 | } | 79 | } |
| 69 | 80 | ||
| 70 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, | 81 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, |
| 71 | heap->mem.userptr, heap->mem.size); | 82 | amdxdna_obj_dma_addr(heap), |
| 83 | heap->mem.size); | ||
| 72 | if (ret) { | 84 | if (ret) { |
| 73 | XDNA_ERR(xdna, "Map host buf failed, ret %d", ret); | 85 | XDNA_ERR(xdna, "Map host buf failed, ret %d", ret); |
| 74 | goto out; | 86 | goto out; |
| @@ -165,7 +177,6 @@ aie2_sched_notify(struct amdxdna_sched_job *job) · 90 unmodified lines | |||
| 165 | 177 | ||
| 166 | trace_xdna_job(&job->base, job->hwctx->name, "signaled fence", job->seq); | 178 | trace_xdna_job(&job->base, job->hwctx->name, "signaled fence", job->seq); |
| 167 | 179 | ||
| 168 | amdxdna_pm_suspend_put(job->hwctx->client->xdna); | ||
| 169 | job->hwctx->priv->completed++; | 180 | job->hwctx->priv->completed++; |
| 170 | dma_fence_signal(fence); | 181 | dma_fence_signal(fence); |
| 171 | 182 | ||
| @@ -175,6 +186,50 @@ aie2_sched_notify(struct amdxdna_sched_job *job) · 3 unmodified lines | |||
| 175 | aie2_job_put(job); | 186 | aie2_job_put(job); |
| 176 | } | 187 | } |
| 177 | 188 | ||
| 189 | static void aie2_set_cmd_timeout(struct amdxdna_sched_job *job) | ||
| 190 | { | ||
| 191 | struct aie2_ctx_health *aie2_health __free(kfree) = NULL; | ||
| 192 | struct amdxdna_dev *xdna = job->hwctx->client->xdna; | ||
| 193 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | ||
| 194 | struct app_health_report *report = job->aie2_job_health; | ||
| 195 | u32 fail_cmd_idx = 0; | ||
| 196 | |||
| 197 | if (!report) | ||
| 198 | goto set_timeout; | ||
| 199 | |||
| 200 | XDNA_ERR(xdna, "Firmware timeout state capture:"); | ||
| 201 | XDNA_ERR(xdna, "\tVersion: %d.%d", report->major, report->minor); | ||
| 202 | XDNA_ERR(xdna, "\tReport size: 0x%x", report->size); | ||
| 203 | XDNA_ERR(xdna, "\tContext ID: %d", report->context_id); | ||
| 204 | XDNA_ERR(xdna, "\tDPU PC: 0x%x", report->dpu_pc); | ||
| 205 | XDNA_ERR(xdna, "\tTXN OP ID: 0x%x", report->txn_op_id); | ||
| 206 | XDNA_ERR(xdna, "\tContext PC: 0x%x", report->ctx_pc); | ||
| 207 | XDNA_ERR(xdna, "\tFatal error type: 0x%x", report->fatal_info.fatal_type); | ||
| 208 | XDNA_ERR(xdna, "\tFatal error exception type: 0x%x", report->fatal_info.exception_type); | ||
| 209 | XDNA_ERR(xdna, "\tFatal error exception PC: 0x%x", report->fatal_info.exception_pc); | ||
| 210 | XDNA_ERR(xdna, "\tFatal error app module: 0x%x", report->fatal_info.app_module); | ||
| 211 | XDNA_ERR(xdna, "\tFatal error task ID: %d", report->fatal_info.task_index); | ||
| 212 | XDNA_ERR(xdna, "\tTimed out sub command ID: %d", report->run_list_id); | ||
| 213 | |||
| 214 | fail_cmd_idx = report->run_list_id; | ||
| 215 | aie2_health = kzalloc_obj(*aie2_health); | ||
| 216 | if (!aie2_health) | ||
| 217 | goto set_timeout; | ||
| 218 | |||
| 219 | aie2_health->header.version = AMDXDNA_CMD_CTX_HEALTH_V1; | ||
| 220 | aie2_health->header.npu_gen = AMDXDNA_CMD_CTX_HEALTH_AIE2; | ||
| 221 | aie2_health->txn_op_idx = report->txn_op_id; | ||
| 222 | aie2_health->ctx_pc = report->ctx_pc; | ||
| 223 | aie2_health->fatal_error_type = report->fatal_info.fatal_type; | ||
| 224 | aie2_health->fatal_error_exception_type = report->fatal_info.exception_type; | ||
| 225 | aie2_health->fatal_error_exception_pc = report->fatal_info.exception_pc; | ||
| 226 | aie2_health->fatal_error_app_module = report->fatal_info.app_module; | ||
| 227 | |||
| 228 | set_timeout: | ||
| 229 | amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_TIMEOUT, | ||
| 230 | aie2_health, sizeof(*aie2_health)); | ||
| 231 | } | ||
| 232 | |||
| 178 | static int | 233 | static int |
| 179 | aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) | 234 | aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) |
| 180 | { | 235 | { |
| @@ -186,13 +241,13 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) · 5 unmodified lines | |||
| 186 | cmd_abo = job->cmd_bo; | 241 | cmd_abo = job->cmd_bo; |
| 187 | 242 | ||
| 188 | if (unlikely(job->job_timeout)) { | 243 | if (unlikely(job->job_timeout)) { |
| 189 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_TIMEOUT); | 244 | aie2_set_cmd_timeout(job); |
| 190 | ret = -EINVAL; | 245 | ret = -EINVAL; |
| 191 | goto out; | 246 | goto out; |
| 192 | } | 247 | } |
| 193 | 248 | ||
| 194 | if (unlikely(!data) || unlikely(size != sizeof(u32))) { | 249 | if (unlikely(!data) || unlikely(size != sizeof(u32))) { |
| 195 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT); | 250 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT, NULL, 0); |
| 196 | ret = -EINVAL; | 251 | ret = -EINVAL; |
| 197 | goto out; | 252 | goto out; |
| 198 | } | 253 | } |
| @@ -202,7 +257,7 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) · 3 unmodified lines | |||
| 202 | if (status == AIE2_STATUS_SUCCESS) | 257 | if (status == AIE2_STATUS_SUCCESS) |
| 203 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_COMPLETED); | 258 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_COMPLETED); |
| 204 | else | 259 | else |
| 205 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ERROR); | 260 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ERROR, NULL, 0); |
| 206 | 261 | ||
| 207 | out: | 262 | out: |
| 208 | aie2_sched_notify(job); | 263 | aie2_sched_notify(job); |
| @@ -236,21 +291,21 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size) · 27 unmodified lines | |||
| 236 | struct amdxdna_sched_job *job = handle; | 291 | struct amdxdna_sched_job *job = handle; |
| 237 | struct amdxdna_gem_obj *cmd_abo; | 292 | struct amdxdna_gem_obj *cmd_abo; |
| 238 | struct amdxdna_dev *xdna; | 293 | struct amdxdna_dev *xdna; |
| 294 | u32 fail_cmd_idx = 0; | ||
| 239 | u32 fail_cmd_status; | 295 | u32 fail_cmd_status; |
| 240 | u32 fail_cmd_idx; | ||
| 241 | u32 cmd_status; | 296 | u32 cmd_status; |
| 242 | int ret = 0; | 297 | int ret = 0; |
| 243 | 298 | ||
| 244 | cmd_abo = job->cmd_bo; | 299 | cmd_abo = job->cmd_bo; |
| 245 | 300 | ||
| 246 | if (unlikely(job->job_timeout)) { | 301 | if (unlikely(job->job_timeout)) { |
| 247 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_TIMEOUT); | 302 | aie2_set_cmd_timeout(job); |
| 248 | ret = -EINVAL; | 303 | ret = -EINVAL; |
| 249 | goto out; | 304 | goto out; |
| 250 | } | 305 | } |
| 251 | 306 | ||
| 252 | if (unlikely(!data) || unlikely(size != sizeof(u32) * 3)) { | 307 | if (unlikely(!data) || unlikely(size != sizeof(u32) * 3)) { |
| 253 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT); | 308 | amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT, NULL, 0); |
| 254 | ret = -EINVAL; | 309 | ret = -EINVAL; |
| 255 | goto out; | 310 | goto out; |
| 256 | } | 311 | } |
| @@ -270,10 +325,10 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size) · 13 unmodified lines | |||
| 270 | fail_cmd_idx, fail_cmd_status); | 325 | fail_cmd_idx, fail_cmd_status); |
| 271 | 326 | ||
| 272 | if (fail_cmd_status == AIE2_STATUS_SUCCESS) { | 327 | if (fail_cmd_status == AIE2_STATUS_SUCCESS) { |
| 273 | amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ABORT); | 328 | amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ABORT, NULL, 0); |
| 274 | ret = -EINVAL; | 329 | ret = -EINVAL; |
| 275 | } else { | 330 | } else { |
| 276 | amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ERROR); | 331 | amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ERROR, NULL, 0); |
| 277 | } | 332 | } |
| 278 | 333 | ||
| 279 | out: | 334 | out: |
| @@ -290,19 +345,11 @@ aie2_sched_job_run(struct drm_sched_job *sched_job) · 10 unmodified lines | |||
| 290 | struct dma_fence *fence; | 345 | struct dma_fence *fence; |
| 291 | int ret; | 346 | int ret; |
| 292 | 347 | ||
| 293 | ret = amdxdna_pm_resume_get(hwctx->client->xdna); | 348 | if (!hwctx->priv->mbox_chann) |
| 294 | if (ret) | ||
| 295 | return NULL; | 349 | return NULL; |
| 296 | 350 | ||
| 297 | if (!hwctx->priv->mbox_chann) { | 351 | if (!mmget_not_zero(job->mm)) |
| 298 | amdxdna_pm_suspend_put(hwctx->client->xdna); | ||
| 299 | return NULL; | ||
| 300 | } | ||
| 301 | |||
| 302 | if (!mmget_not_zero(job->mm)) { | ||
| 303 | amdxdna_pm_suspend_put(hwctx->client->xdna); | ||
| 304 | return ERR_PTR(-ESRCH); | 352 | return ERR_PTR(-ESRCH); |
| 305 | } | ||
| 306 | 353 | ||
| 307 | kref_get(&job->refcnt); | 354 | kref_get(&job->refcnt); |
| 308 | fence = dma_fence_get(job->fence); | 355 | fence = dma_fence_get(job->fence); |
| @@ -333,7 +380,6 @@ aie2_sched_job_run(struct drm_sched_job *sched_job) · 24 unmodified lines | |||
| 333 | 380 | ||
| 334 | out: | 381 | out: |
| 335 | if (ret) { | 382 | if (ret) { |
| 336 | amdxdna_pm_suspend_put(hwctx->client->xdna); | ||
| 337 | dma_fence_put(job->fence); | 383 | dma_fence_put(job->fence); |
| 338 | aie2_job_put(job); | 384 | aie2_job_put(job); |
| 339 | mmput(job->mm); | 385 | mmput(job->mm); |
| @@ -362,12 +408,26 @@ aie2_sched_job_timedout(struct drm_sched_job *sched_job) · 22 unmodified lines | |||
| 362 | { | 408 | { |
| 363 | struct amdxdna_sched_job *job = drm_job_to_xdna_job(sched_job); | 409 | struct amdxdna_sched_job *job = drm_job_to_xdna_job(sched_job); |
| 364 | struct amdxdna_hwctx *hwctx = job->hwctx; | 410 | struct amdxdna_hwctx *hwctx = job->hwctx; |
| 411 | struct app_health_report *report; | ||
| 365 | struct amdxdna_dev *xdna; | 412 | struct amdxdna_dev *xdna; |
| 413 | int ret; | ||
| 366 | 414 | ||
| 367 | xdna = hwctx->client->xdna; | 415 | xdna = hwctx->client->xdna; |
| 368 | trace_xdna_job(sched_job, hwctx->name, "job timedout", job->seq); | 416 | trace_xdna_job(sched_job, hwctx->name, "job timedout", job->seq); |
| 369 | job->job_timeout = true; | 417 | job->job_timeout = true; |
| 418 | |||
| 370 | mutex_lock(&xdna->dev_lock); | 419 | mutex_lock(&xdna->dev_lock); |
| 420 | report = kzalloc_obj(*report); | ||
| 421 | if (!report) | ||
| 422 | goto reset_hwctx; | ||
| 423 | |||
| 424 | ret = aie2_query_app_health(xdna->dev_handle, hwctx->fw_ctx_id, report); | ||
| 425 | if (ret) | ||
| 426 | kfree(report); | ||
| 427 | else | ||
| 428 | job->aie2_job_health = report; | ||
| 429 | |||
| 430 | reset_hwctx: | ||
| 371 | aie2_hwctx_stop(xdna, hwctx, sched_job); | 431 | aie2_hwctx_stop(xdna, hwctx, sched_job); |
| 372 | 432 | ||
| 373 | aie2_hwctx_restart(xdna, hwctx); | 433 | aie2_hwctx_restart(xdna, hwctx); |
| @@ -589,14 +649,14 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 215 unmodified lines | |||
| 589 | .size = MAX_CHAIN_CMDBUF_SIZE, | 649 | .size = MAX_CHAIN_CMDBUF_SIZE, |
| 590 | }; | 650 | }; |
| 591 | 651 | ||
| 592 | abo = amdxdna_drm_alloc_dev_bo(&xdna->ddev, &args, client->filp); | 652 | abo = amdxdna_drm_create_dev_bo(&xdna->ddev, &args, client->filp); |
| 593 | if (IS_ERR(abo)) { | 653 | if (IS_ERR(abo)) { |
| 594 | ret = PTR_ERR(abo); | 654 | ret = PTR_ERR(abo); |
| 595 | goto free_cmd_bufs; | 655 | goto free_cmd_bufs; |
| 596 | } | 656 | } |
| 597 | 657 | ||
| 598 | XDNA_DBG(xdna, "Command buf %d addr 0x%llx size 0x%lx", | 658 | XDNA_DBG(xdna, "Command buf %d addr 0x%llx size 0x%lx", |
| 599 | i, abo->mem.dev_addr, abo->mem.size); | 659 | i, amdxdna_gem_dev_addr(abo), abo->mem.size); |
| 600 | priv->cmd_buf[i] = abo; | 660 | priv->cmd_buf[i] = abo; |
| 601 | } | 661 | } |
| 602 | 662 | ||
| @@ -637,7 +697,8 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 34 unmodified lines | |||
| 637 | } | 697 | } |
| 638 | 698 | ||
| 639 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, | 699 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, |
| 640 | heap->mem.userptr, heap->mem.size); | 700 | amdxdna_obj_dma_addr(heap), |
| 701 | heap->mem.size); | ||
| 641 | if (ret) { | 702 | if (ret) { |
| 642 | XDNA_ERR(xdna, "Map host buffer failed, ret %d", ret); | 703 | XDNA_ERR(xdna, "Map host buffer failed, ret %d", ret); |
| 643 | goto release_resource; | 704 | goto release_resource; |
| @@ -355,9 +355,8 @@ int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev) · 354 unmodified lines | |||
| 355 | return -ENOMEM; | 355 | return -ENOMEM; |
| 356 | 356 | ||
| 357 | events->buf = aie2_alloc_msg_buffer(ndev, &total_size, &events->addr); | 357 | events->buf = aie2_alloc_msg_buffer(ndev, &total_size, &events->addr); |
| 358 | 358 | if (IS_ERR(events->buf)) { | |
| 359 | if (!events->buf) { | 359 | ret = PTR_ERR(events->buf); |
| 360 | ret = -ENOMEM; | ||
| 361 | goto free_events; | 360 | goto free_events; |
| 362 | } | 361 | } |
| 363 | events->size = total_size; | 362 | events->size = total_size; |
| @@ -56,16 +56,37 @@ void *aie2_alloc_msg_buffer(struct amdxdna_dev_hdl *ndev, u32 *size, · 55 unmodified lines | |||
| 56 | dma_addr_t *dma_addr) | 56 | dma_addr_t *dma_addr) |
| 57 | { | 57 | { |
| 58 | struct amdxdna_dev *xdna = ndev->xdna; | 58 | struct amdxdna_dev *xdna = ndev->xdna; |
| 59 | void *vaddr; | ||
| 59 | int order; | 60 | int order; |
| 60 | 61 | ||
| 61 | *size = max(*size, SZ_8K); | 62 | *size = max(*size, SZ_8K); |
| 62 | order = get_order(*size); | 63 | order = get_order(*size); |
| 63 | if (order > MAX_PAGE_ORDER) | 64 | if (order > MAX_PAGE_ORDER) |
| 64 | return NULL; | 65 | return ERR_PTR(-EINVAL); |
| 65 | *size = PAGE_SIZE << order; | 66 | *size = PAGE_SIZE << order; |
| 66 | 67 | ||
| 67 | return dma_alloc_noncoherent(xdna->ddev.dev, *size, dma_addr, | 68 | if (amdxdna_iova_on(xdna)) |
| 68 | DMA_FROM_DEVICE, GFP_KERNEL); | 69 | return amdxdna_iommu_alloc(xdna, *size, dma_addr); |
| 70 | |||
| 71 | vaddr = dma_alloc_noncoherent(xdna->ddev.dev, *size, dma_addr, | ||
| 72 | DMA_FROM_DEVICE, GFP_KERNEL); | ||
| 73 | if (!vaddr) | ||
| 74 | return ERR_PTR(-ENOMEM); | ||
| 75 | |||
| 76 | return vaddr; | ||
| 77 | } | ||
| 78 | |||
| 79 | void aie2_free_msg_buffer(struct amdxdna_dev_hdl *ndev, size_t size, | ||
| 80 | void *cpu_addr, dma_addr_t dma_addr) | ||
| 81 | { | ||
| 82 | struct amdxdna_dev *xdna = ndev->xdna; | ||
| 83 | |||
| 84 | if (amdxdna_iova_on(xdna)) { | ||
| 85 | amdxdna_iommu_free(xdna, size, cpu_addr, dma_addr); | ||
| 86 | return; | ||
| 87 | } | ||
| 88 | |||
| 89 | dma_free_noncoherent(xdna->ddev.dev, size, cpu_addr, dma_addr, DMA_FROM_DEVICE); | ||
| 69 | } | 90 | } |
| 70 | 91 | ||
| 71 | int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev) | 92 | int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev) |
| @@ -256,7 +277,7 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct · 184 unmodified lines | |||
| 256 | req.num_col = hwctx->num_col; | 277 | req.num_col = hwctx->num_col; |
| 257 | req.num_unused_col = hwctx->num_unused_col; | 278 | req.num_unused_col = hwctx->num_unused_col; |
| 258 | req.num_cq_pairs_requested = 1; | 279 | req.num_cq_pairs_requested = 1; |
| 259 | req.pasid = hwctx->client->pasid; | 280 | req.pasid = amdxdna_pasid_on(hwctx->client) ? hwctx->client->pasid : 0; |
| 260 | req.context_priority = aie2_get_context_priority(ndev, hwctx); | 281 | req.context_priority = aie2_get_context_priority(ndev, hwctx); |
| 261 | 282 | ||
| 262 | ret = aie2_send_mgmt_msg_wait(ndev, &msg); | 283 | ret = aie2_send_mgmt_msg_wait(ndev, &msg); |
| @@ -380,8 +401,8 @@ int aie2_query_status(struct amdxdna_dev_hdl *ndev, char __user *buf, · 117 unmodified lines | |||
| 380 | int ret; | 401 | int ret; |
| 381 | 402 | ||
| 382 | buff_addr = aie2_alloc_msg_buffer(ndev, &buf_sz, &dma_addr); | 403 | buff_addr = aie2_alloc_msg_buffer(ndev, &buf_sz, &dma_addr); |
| 383 | if (!buff_addr) | 404 | if (IS_ERR(buff_addr)) |
| 384 | return -ENOMEM; | 405 | return PTR_ERR(buff_addr); |
| 385 | 406 | ||
| 386 | /* Go through each hardware context and mark the AIE columns that are active */ | 407 | /* Go through each hardware context and mark the AIE columns that are active */ |
| 387 | list_for_each_entry(client, &xdna->client_list, node) | 408 | list_for_each_entry(client, &xdna->client_list, node) |
| @@ -436,8 +457,8 @@ int aie2_query_telemetry(struct amdxdna_dev_hdl *ndev, · 48 unmodified lines | |||
| 436 | return -EINVAL; | 457 | return -EINVAL; |
| 437 | 458 | ||
| 438 | addr = aie2_alloc_msg_buffer(ndev, &buf_sz, &dma_addr); | 459 | addr = aie2_alloc_msg_buffer(ndev, &buf_sz, &dma_addr); |
| 439 | if (!addr) | 460 | if (IS_ERR(addr)) |
| 440 | return -ENOMEM; | 461 | return PTR_ERR(addr); |
| 441 | 462 | ||
| 442 | req.buf_addr = dma_addr; | 463 | req.buf_addr = dma_addr; |
| 443 | req.buf_size = buf_sz; | 464 | req.buf_size = buf_sz; |
| @@ -532,10 +553,10 @@ int aie2_config_cu(struct amdxdna_hwctx *hwctx, · 88 unmodified lines | |||
| 532 | } | 553 | } |
| 533 | 554 | ||
| 534 | req.cfgs[i] = FIELD_PREP(AIE2_MSG_CFG_CU_PDI_ADDR, | 555 | req.cfgs[i] = FIELD_PREP(AIE2_MSG_CFG_CU_PDI_ADDR, |
| 535 | abo->mem.dev_addr >> shift); | 556 | amdxdna_gem_dev_addr(abo) >> shift); |
| 536 | req.cfgs[i] |= FIELD_PREP(AIE2_MSG_CFG_CU_FUNC, cu->cu_func); | 557 | req.cfgs[i] |= FIELD_PREP(AIE2_MSG_CFG_CU_FUNC, cu->cu_func); |
| 537 | XDNA_DBG(xdna, "CU %d full addr 0x%llx, cfg 0x%x", i, | 558 | XDNA_DBG(xdna, "CU %d full addr 0x%llx, cfg 0x%x", i, |
| 538 | abo->mem.dev_addr, req.cfgs[i]); | 559 | amdxdna_gem_dev_addr(abo), req.cfgs[i]); |
| 539 | drm_gem_object_put(gobj); | 560 | drm_gem_object_put(gobj); |
| 540 | } | 561 | } |
| 541 | req.num_cus = hwctx->cus->num_cus; | 562 | req.num_cus = hwctx->cus->num_cus; |
| @@ -982,56 +1003,78 @@ int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, · 440 unmodified lines | |||
| 982 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | 1003 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 983 | struct amdxdna_client *client = hwctx->client; | 1004 | struct amdxdna_client *client = hwctx->client; |
| 984 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | 1005 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 1006 | void *cmd_buf = amdxdna_gem_vmap(cmdbuf_abo); | ||
| 985 | struct amdxdna_dev *xdna = client->xdna; | 1007 | struct amdxdna_dev *xdna = client->xdna; |
| 986 | struct amdxdna_cmd_chain *payload; | 1008 | struct amdxdna_cmd_chain *payload; |
| 987 | struct xdna_mailbox_msg msg; | 1009 | struct xdna_mailbox_msg msg; |
| 988 | union exec_chain_req req; | 1010 | union exec_chain_req req; |
| 989 | u32 payload_len; | 1011 | u32 payload_len, ccnt; |
| 990 | u32 offset = 0; | 1012 | u32 offset = 0; |
| 991 | size_t size; | 1013 | size_t size; |
| 992 | int ret; | 1014 | int ret; |
| 993 | u32 op; | 1015 | u32 op; |
| 994 | u32 i; | 1016 | u32 i; |
| 995 | 1017 | ||
| 1018 | if (!cmd_buf) | ||
| 1019 | return -ENOMEM; | ||
| 1020 | |||
| 996 | op = amdxdna_cmd_get_op(cmd_abo); | 1021 | op = amdxdna_cmd_get_op(cmd_abo); |
| 997 | payload = amdxdna_cmd_get_payload(cmd_abo, &payload_len); | 1022 | payload = amdxdna_cmd_get_payload(cmd_abo, &payload_len); |
| 998 | if (op != ERT_CMD_CHAIN || !payload || | 1023 | if (op != ERT_CMD_CHAIN) { |
| 999 | payload_len < struct_size(payload, data, payload->command_count)) | 1024 | XDNA_DBG(xdna, "Invalid op code %d", op); |
| 1025 | return -EINVAL; | ||
| 1026 | } | ||
| 1027 | |||
| 1028 | if (!payload) { | ||
| 1029 | XDNA_DBG(xdna, "Failed to get command payload"); | ||
| 1030 | return -EINVAL; | ||
| 1031 | } | ||
| 1032 | |||
| 1033 | ccnt = payload->command_count; | ||
| 1034 | if (payload_len < struct_size(payload, data, ccnt)) { | ||
| 1035 | XDNA_DBG(xdna, "Invalid command count %d", ccnt); | ||
| 1000 | return -EINVAL; | 1036 | return -EINVAL; |
| 1037 | } | ||
| 1001 | 1038 | ||
| 1002 | op = ERT_INVALID_CMD; | 1039 | op = ERT_INVALID_CMD; |
| 1003 | for (i = 0; i < payload->command_count; i++) { | 1040 | for (i = 0; i < ccnt; i++) { |
| 1004 | u32 boh = (u32)(payload->data[i]); | 1041 | u32 boh = (u32)(payload->data[i]); |
| 1005 | struct amdxdna_gem_obj *abo; | 1042 | struct amdxdna_gem_obj *abo; |
| 1006 | 1043 | ||
| 1007 | abo = amdxdna_gem_get_obj(client, boh, AMDXDNA_BO_CMD); | 1044 | abo = amdxdna_gem_get_obj(client, boh, AMDXDNA_BO_SHARE); |
| 1008 | if (!abo) { | 1045 | if (!abo) { |
| 1009 | XDNA_ERR(xdna, "Failed to find cmd BO %d", boh); | 1046 | XDNA_ERR(xdna, "Failed to find cmd BO %d", boh); |
| 1010 | return -ENOENT; | 1047 | return -ENOENT; |
| 1011 | } | 1048 | } |
| 1012 | 1049 | ||
| 1013 | size = cmdbuf_abo->mem.size - offset; | 1050 | size = cmdbuf_abo->mem.size - offset; |
| 1014 | ret = aie2_cmdlist_fill_slot(cmdbuf_abo->mem.kva + offset, | 1051 | ret = aie2_cmdlist_fill_slot(cmd_buf + offset, abo, &size, &op); |
| 1015 | abo, &size, &op); | ||
| 1016 | amdxdna_gem_put_obj(abo); | 1052 | amdxdna_gem_put_obj(abo); |
| 1017 | if (ret) | 1053 | if (ret) |
| 1018 | return ret; | 1054 | return ret; |
| 1019 | 1055 | ||
| 1020 | offset += size; | 1056 | offset += size; |
| 1021 | } | 1057 | } |
| 1058 | |||
| 1059 | XDNA_DBG(xdna, "Total %d commands:", ccnt); | ||
| 1060 | print_hex_dump_debug("cmdbufs: ", DUMP_PREFIX_OFFSET, 16, 4, | ||
| 1061 | cmd_buf, offset, false); | ||
| 1062 | |||
| 1022 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); | 1063 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); |
| 1023 | if (msg.opcode == MSG_OP_MAX_OPCODE) | 1064 | if (msg.opcode == MSG_OP_MAX_OPCODE) |
| 1024 | return -EOPNOTSUPP; | 1065 | return -EOPNOTSUPP; |
| 1025 | 1066 | ||
| 1026 | /* The offset is the accumulated total size of the cmd buffer */ | 1067 | /* The offset is the accumulated total size of the cmd buffer */ |
| 1027 | EXEC_MSG_OPS(xdna)->init_chain_req(&req, cmdbuf_abo->mem.dev_addr, | 1068 | EXEC_MSG_OPS(xdna)->init_chain_req(&req, amdxdna_gem_dev_addr(cmdbuf_abo), |
| 1028 | offset, payload->command_count); | 1069 | offset, ccnt); |
| 1029 | drm_clflush_virt_range(cmdbuf_abo->mem.kva, offset); | 1070 | drm_clflush_virt_range(cmd_buf, offset); |
| 1030 | 1071 | ||
| 1031 | msg.handle = job; | 1072 | msg.handle = job; |
| 1032 | msg.notify_cb = notify_cb; | 1073 | msg.notify_cb = notify_cb; |
| 1033 | msg.send_data = (u8 *)&req; | 1074 | msg.send_data = (u8 *)&req; |
| 1034 | msg.send_size = sizeof(req); | 1075 | msg.send_size = sizeof(req); |
| 1076 | print_hex_dump_debug("cmdlist msg: ", DUMP_PREFIX_OFFSET, 16, 4, | ||
| 1077 | &req, msg.send_size, false); | ||
| 1035 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | 1078 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 1036 | if (ret) { | 1079 | if (ret) { |
| 1037 | XDNA_ERR(xdna, "Send message failed"); | 1080 | XDNA_ERR(xdna, "Send message failed"); |
| @@ -1049,29 +1092,36 @@ int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx, · 11 unmodified lines | |||
| 1049 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | 1092 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 1050 | struct amdxdna_dev *xdna = hwctx->client->xdna; | 1093 | struct amdxdna_dev *xdna = hwctx->client->xdna; |
| 1051 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | 1094 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 1095 | void *cmd_buf = amdxdna_gem_vmap(cmdbuf_abo); | ||
| 1052 | struct xdna_mailbox_msg msg; | 1096 | struct xdna_mailbox_msg msg; |
| 1053 | union exec_chain_req req; | 1097 | union exec_chain_req req; |
| 1054 | u32 op = ERT_INVALID_CMD; | 1098 | u32 op = ERT_INVALID_CMD; |
| 1055 | size_t size; | 1099 | size_t size; |
| 1056 | int ret; | 1100 | int ret; |
| 1057 | 1101 | ||
| 1102 | if (!cmd_buf) | ||
| 1103 | return -ENOMEM; | ||
| 1104 | |||
| 1058 | size = cmdbuf_abo->mem.size; | 1105 | size = cmdbuf_abo->mem.size; |
| 1059 | ret = aie2_cmdlist_fill_slot(cmdbuf_abo->mem.kva, cmd_abo, &size, &op); | 1106 | ret = aie2_cmdlist_fill_slot(cmd_buf, cmd_abo, &size, &op); |
| 1060 | if (ret) | 1107 | if (ret) |
| 1061 | return ret; | 1108 | return ret; |
| 1062 | 1109 | ||
| 1110 | print_hex_dump_debug("cmdbuf: ", DUMP_PREFIX_OFFSET, 16, 4, cmd_buf, size, false); | ||
| 1111 | |||
| 1063 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); | 1112 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); |
| 1064 | if (msg.opcode == MSG_OP_MAX_OPCODE) | 1113 | if (msg.opcode == MSG_OP_MAX_OPCODE) |
| 1065 | return -EOPNOTSUPP; | 1114 | return -EOPNOTSUPP; |
| 1066 | 1115 | ||
| 1067 | EXEC_MSG_OPS(xdna)->init_chain_req(&req, cmdbuf_abo->mem.dev_addr, | 1116 | EXEC_MSG_OPS(xdna)->init_chain_req(&req, amdxdna_gem_dev_addr(cmdbuf_abo), size, 1); |
| 1068 | size, 1); | 1117 | drm_clflush_virt_range(cmd_buf, size); |
| 1069 | drm_clflush_virt_range(cmdbuf_abo->mem.kva, size); | ||
| 1070 | 1118 | ||
| 1071 | msg.handle = job; | 1119 | msg.handle = job; |
| 1072 | msg.notify_cb = notify_cb; | 1120 | msg.notify_cb = notify_cb; |
| 1073 | msg.send_data = (u8 *)&req; | 1121 | msg.send_data = (u8 *)&req; |
| 1074 | msg.send_size = sizeof(req); | 1122 | msg.send_size = sizeof(req); |
| 1123 | print_hex_dump_debug("cmdlist msg: ", DUMP_PREFIX_OFFSET, 16, 4, | ||
| 1124 | &req, msg.send_size, false); | ||
| 1075 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | 1125 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 1076 | if (ret) { | 1126 | if (ret) { |
| 1077 | XDNA_ERR(hwctx->client->xdna, "Send message failed"); | 1127 | XDNA_ERR(hwctx->client->xdna, "Send message failed"); |
| @@ -1145,3 +1195,44 @@ int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job * · 67 unmodified lines | |||
| 1145 | 1195 | ||
| 1146 | return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | 1196 | return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 1147 | } | 1197 | } |
| 1198 | |||
| 1199 | int aie2_query_app_health(struct amdxdna_dev_hdl *ndev, u32 context_id, | ||
| 1200 | struct app_health_report *report) | ||
| 1201 | { | ||
| 1202 | DECLARE_AIE2_MSG(get_app_health, MSG_OP_GET_APP_HEALTH); | ||
| 1203 | struct amdxdna_dev *xdna = ndev->xdna; | ||
| 1204 | struct app_health_report *buf; | ||
| 1205 | dma_addr_t dma_addr; | ||
| 1206 | u32 buf_size; | ||
| 1207 | int ret; | ||
| 1208 | |||
| 1209 | if (!AIE2_FEATURE_ON(ndev, AIE2_APP_HEALTH)) { | ||
| 1210 | XDNA_DBG(xdna, "App health feature not supported"); | ||
| 1211 | return -EOPNOTSUPP; | ||
| 1212 | } | ||
| 1213 | |||
| 1214 | buf_size = sizeof(*report); | ||
| 1215 | buf = aie2_alloc_msg_buffer(ndev, &buf_size, &dma_addr); | ||
| 1216 | if (IS_ERR(buf)) { | ||
| 1217 | XDNA_ERR(xdna, "Failed to allocate buffer for app health"); | ||
| 1218 | return PTR_ERR(buf); | ||
| 1219 | } | ||
| 1220 | |||
| 1221 | req.buf_addr = dma_addr; | ||
| 1222 | req.context_id = context_id; | ||
| 1223 | req.buf_size = buf_size; | ||
| 1224 | |||
| 1225 | drm_clflush_virt_range(buf, sizeof(*report)); | ||
| 1226 | ret = aie2_send_mgmt_msg_wait(ndev, &msg); | ||
| 1227 | if (ret) { | ||
| 1228 | XDNA_ERR(xdna, "Get app health failed, ret %d status 0x%x", ret, resp.status); | ||
| 1229 | goto free_buf; | ||
| 1230 | } | ||
| 1231 | |||
| 1232 | /* Copy the report to caller's buffer */ | ||
| 1233 | memcpy(report, buf, sizeof(*report)); | ||
| 1234 | |||
| 1235 | free_buf: | ||
| 1236 | aie2_free_msg_buffer(ndev, buf_size, buf, dma_addr); | ||
| 1237 | return ret; | ||
| 1238 | } | ||
| @@ -31,6 +31,7 @@ enum aie2_msg_opcode { · 30 unmodified lines | |||
| 31 | MSG_OP_SET_RUNTIME_CONFIG = 0x10A, | 31 | MSG_OP_SET_RUNTIME_CONFIG = 0x10A, |
| 32 | MSG_OP_GET_RUNTIME_CONFIG = 0x10B, | 32 | MSG_OP_GET_RUNTIME_CONFIG = 0x10B, |
| 33 | MSG_OP_REGISTER_ASYNC_EVENT_MSG = 0x10C, | 33 | MSG_OP_REGISTER_ASYNC_EVENT_MSG = 0x10C, |
| 34 | MSG_OP_GET_APP_HEALTH = 0x114, | ||
| 34 | MSG_OP_MAX_DRV_OPCODE, | 35 | MSG_OP_MAX_DRV_OPCODE, |
| 35 | MSG_OP_GET_PROTOCOL_VERSION = 0x301, | 36 | MSG_OP_GET_PROTOCOL_VERSION = 0x301, |
| 36 | MSG_OP_MAX_OPCODE | 37 | MSG_OP_MAX_OPCODE |
| @@ -451,4 +452,55 @@ struct config_debug_bo_req { · 414 unmodified lines | |||
| 451 | struct config_debug_bo_resp { | 452 | struct config_debug_bo_resp { |
| 452 | enum aie2_msg_status status; | 453 | enum aie2_msg_status status; |
| 453 | } __packed; | 454 | } __packed; |
| 455 | |||
| 456 | struct fatal_error_info { | ||
| 457 | __u32 fatal_type; /* Fatal error type */ | ||
| 458 | __u32 exception_type; /* Only valid if fatal_type is a specific value */ | ||
| 459 | __u32 exception_argument; /* Argument based on exception type */ | ||
| 460 | __u32 exception_pc; /* Program Counter at the time of the exception */ | ||
| 461 | __u32 app_module; /* Error module name */ | ||
| 462 | __u32 task_index; /* Index of the task in which the error occurred */ | ||
| 463 | __u32 reserved[128]; | ||
| 464 | }; | ||
| 465 | |||
| 466 | struct app_health_report { | ||
| 467 | __u16 major; | ||
| 468 | __u16 minor; | ||
| 469 | __u32 size; | ||
| 470 | __u32 context_id; | ||
| 471 | /* | ||
| 472 | * Program Counter (PC) of the last initiated DPU opcode, as reported by the ERT | ||
| 473 | * application. Before execution begins or after successful completion, the value is set | ||
| 474 | * to UINT_MAX. If execution halts prematurely due to an error, this field retains the | ||
| 475 | * opcode's PC value. | ||
| 476 | * Note: To optimize performance, the ERT may simplify certain aspects of reporting. | ||
| 477 | * Proper interpretation requires familiarity with the implementation details. | ||
| 478 | */ | ||
| 479 | __u32 dpu_pc; | ||
| 480 | /* | ||
| 481 | * Index of the last initiated TXN opcode. | ||
| 482 | * Before execution starts or after successful completion, the value is set to UINT_MAX. | ||
| 483 | * If execution halts prematurely due to an error, this field retains the opcode's ID. | ||
| 484 | * Note: To optimize performance, the ERT may simplify certain aspects of reporting. | ||
| 485 | * Proper interpretation requires familiarity with the implementation details. | ||
| 486 | */ | ||
| 487 | __u32 txn_op_id; | ||
| 488 | /* The PC of the context at the time of the report */ | ||
| 489 | __u32 ctx_pc; | ||
| 490 | struct fatal_error_info fatal_info; | ||
| 491 | /* Index of the most recently executed run list entry. */ | ||
| 492 | __u32 run_list_id; | ||
| 493 | }; | ||
| 494 | |||
| 495 | struct get_app_health_req { | ||
| 496 | __u32 context_id; | ||
| 497 | __u32 buf_size; | ||
| 498 | __u64 buf_addr; | ||
| 499 | } __packed; | ||
| 500 | |||
| 501 | struct get_app_health_resp { | ||
| 502 | enum aie2_msg_status status; | ||
| 503 | __u32 required_buffer_size; | ||
| 504 | __u32 reserved[7]; | ||
| 505 | } __packed; | ||
| 454 | #endif /* _AIE2_MSG_PRIV_H_ */ | 506 | #endif /* _AIE2_MSG_PRIV_H_ */ |
| @@ -846,7 +846,10 @@ static int aie2_hwctx_status_cb(struct amdxdna_hwctx *hwctx, void *arg) · 845 unmodified lines | |||
| 846 | struct amdxdna_drm_hwctx_entry *tmp __free(kfree) = NULL; | 846 | struct amdxdna_drm_hwctx_entry *tmp __free(kfree) = NULL; |
| 847 | struct amdxdna_drm_get_array *array_args = arg; | 847 | struct amdxdna_drm_get_array *array_args = arg; |
| 848 | struct amdxdna_drm_hwctx_entry __user *buf; | 848 | struct amdxdna_drm_hwctx_entry __user *buf; |
| 849 | struct app_health_report report; | ||
| 850 | struct amdxdna_dev_hdl *ndev; | ||
| 849 | u32 size; | 851 | u32 size; |
| 852 | int ret; | ||
| 850 | 853 | ||
| 851 | if (!array_args->num_element) | 854 | if (!array_args->num_element) |
| 852 | return -EINVAL; | 855 | return -EINVAL; |
| @@ -862,6 +865,7 @@ static int aie2_hwctx_status_cb(struct amdxdna_hwctx *hwctx, void *arg) · 9 unmodified lines | |||
| 862 | tmp->command_submissions = hwctx->priv->seq; | 865 | tmp->command_submissions = hwctx->priv->seq; |
| 863 | tmp->command_completions = hwctx->priv->completed; | 866 | tmp->command_completions = hwctx->priv->completed; |
| 864 | tmp->pasid = hwctx->client->pasid; | 867 | tmp->pasid = hwctx->client->pasid; |
| 868 | tmp->heap_usage = hwctx->client->heap_usage; | ||
| 865 | tmp->priority = hwctx->qos.priority; | 869 | tmp->priority = hwctx->qos.priority; |
| 866 | tmp->gops = hwctx->qos.gops; | 870 | tmp->gops = hwctx->qos.gops; |
| 867 | tmp->fps = hwctx->qos.fps; | 871 | tmp->fps = hwctx->qos.fps; |
| @@ -869,6 +873,17 @@ static int aie2_hwctx_status_cb(struct amdxdna_hwctx *hwctx, void *arg) · 1 unmodified line | |||
| 869 | tmp->latency = hwctx->qos.latency; | 873 | tmp->latency = hwctx->qos.latency; |
| 870 | tmp->frame_exec_time = hwctx->qos.frame_exec_time; | 874 | tmp->frame_exec_time = hwctx->qos.frame_exec_time; |
| 871 | tmp->state = AMDXDNA_HWCTX_STATE_ACTIVE; | 875 | tmp->state = AMDXDNA_HWCTX_STATE_ACTIVE; |
| 876 | ndev = hwctx->client->xdna->dev_handle; | ||
| 877 | ret = aie2_query_app_health(ndev, hwctx->fw_ctx_id, &report); | ||
| 878 | if (!ret) { | ||
| 879 | /* Fill in app health report fields */ | ||
| 880 | tmp->txn_op_idx = report.txn_op_id; | ||
| 881 | tmp->ctx_pc = report.ctx_pc; | ||
| 882 | tmp->fatal_error_type = report.fatal_info.fatal_type; | ||
| 883 | tmp->fatal_error_exception_type = report.fatal_info.exception_type; | ||
| 884 | tmp->fatal_error_exception_pc = report.fatal_info.exception_pc; | ||
| 885 | tmp->fatal_error_app_module = report.fatal_info.app_module; | ||
| 886 | } | ||
| 872 | 887 | ||
| 873 | buf = u64_to_user_ptr(array_args->buffer); | 888 | buf = u64_to_user_ptr(array_args->buffer); |
| 874 | size = min(sizeof(*tmp), array_args->element_size); | 889 | size = min(sizeof(*tmp), array_args->element_size); |
| @@ -1134,6 +1149,9 @@ static int aie2_get_array(struct amdxdna_client *client, · 259 unmodified lines | |||
| 1134 | case DRM_AMDXDNA_HW_LAST_ASYNC_ERR: | 1149 | case DRM_AMDXDNA_HW_LAST_ASYNC_ERR: |
| 1135 | ret = aie2_get_array_async_error(xdna->dev_handle, args); | 1150 | ret = aie2_get_array_async_error(xdna->dev_handle, args); |
| 1136 | break; | 1151 | break; |
| 1152 | case DRM_AMDXDNA_BO_USAGE: | ||
| 1153 | ret = amdxdna_drm_get_bo_usage(&xdna->ddev, args); | ||
| 1154 | break; | ||
| 1137 | default: | 1155 | default: |
| 1138 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); | 1156 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); |
| 1139 | ret = -EOPNOTSUPP; | 1157 | ret = -EOPNOTSUPP; |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <linux/limits.h> | 10 | #include <linux/limits.h> |
| 11 | #include <linux/semaphore.h> | 11 | #include <linux/semaphore.h> |
| 12 | 12 | ||
| 13 | #include "aie2_msg_priv.h" | ||
| 13 | #include "amdxdna_mailbox.h" | 14 | #include "amdxdna_mailbox.h" |
| 14 | 15 | ||
| 15 | #define AIE2_INTERVAL 20000 /* us */ | 16 | #define AIE2_INTERVAL 20000 /* us */ |
| @@ -261,6 +262,7 @@ enum aie2_fw_feature { · 245 unmodified lines | |||
| 261 | AIE2_NPU_COMMAND, | 262 | AIE2_NPU_COMMAND, |
| 262 | AIE2_PREEMPT, | 263 | AIE2_PREEMPT, |
| 263 | AIE2_TEMPORAL_ONLY, | 264 | AIE2_TEMPORAL_ONLY, |
| 265 | AIE2_APP_HEALTH, | ||
| 264 | AIE2_FEATURE_MAX | 266 | AIE2_FEATURE_MAX |
| 265 | }; | 267 | }; |
| 266 | 268 | ||
| @@ -271,6 +273,7 @@ struct aie2_fw_feature_tbl { · 4 unmodified lines | |||
| 271 | u32 min_minor; | 273 | u32 min_minor; |
| 272 | }; | 274 | }; |
| 273 | 275 | ||
| 276 | #define AIE2_ALL_FEATURES GENMASK_ULL(AIE2_FEATURE_MAX - 1, AIE2_NPU_COMMAND) | ||
| 274 | #define AIE2_FEATURE_ON(ndev, feature) test_bit(feature, &(ndev)->feature_mask) | 277 | #define AIE2_FEATURE_ON(ndev, feature) test_bit(feature, &(ndev)->feature_mask) |
| 275 | 278 | ||
| 276 | struct amdxdna_dev_priv { | 279 | struct amdxdna_dev_priv { |
| @@ -341,6 +344,8 @@ int aie2_query_aie_version(struct amdxdna_dev_hdl *ndev, struct aie_version *ver · 64 unmodified lines | |||
| 341 | int aie2_query_aie_metadata(struct amdxdna_dev_hdl *ndev, struct aie_metadata *metadata); | 344 | int aie2_query_aie_metadata(struct amdxdna_dev_hdl *ndev, struct aie_metadata *metadata); |
| 342 | int aie2_query_firmware_version(struct amdxdna_dev_hdl *ndev, | 345 | int aie2_query_firmware_version(struct amdxdna_dev_hdl *ndev, |
| 343 | struct amdxdna_fw_ver *fw_ver); | 346 | struct amdxdna_fw_ver *fw_ver); |
| 347 | int aie2_query_app_health(struct amdxdna_dev_hdl *ndev, u32 context_id, | ||
| 348 | struct app_health_report *report); | ||
| 344 | int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); | 349 | int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); |
| 345 | int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); | 350 | int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); |
| 346 | int aie2_map_host_buf(struct amdxdna_dev_hdl *ndev, u32 context_id, u64 addr, u64 size); | 351 | int aie2_map_host_buf(struct amdxdna_dev_hdl *ndev, u32 context_id, u64 addr, u64 size); |
| @@ -366,9 +371,8 @@ int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job * · 19 unmodified lines | |||
| 366 | int (*notify_cb)(void *, void __iomem *, size_t)); | 371 | int (*notify_cb)(void *, void __iomem *, size_t)); |
| 367 | void *aie2_alloc_msg_buffer(struct amdxdna_dev_hdl *ndev, u32 *size, | 372 | void *aie2_alloc_msg_buffer(struct amdxdna_dev_hdl *ndev, u32 *size, |
| 368 | dma_addr_t *dma_addr); | 373 | dma_addr_t *dma_addr); |
| 369 | #define aie2_free_msg_buffer(ndev, size, buff_addr, dma_addr) \ | 374 | void aie2_free_msg_buffer(struct amdxdna_dev_hdl *ndev, size_t size, |
| 370 | dma_free_noncoherent((ndev)->xdna->ddev.dev, size, buff_addr, \ | 375 | void *cpu_addr, dma_addr_t dma_addr); |
| 371 | dma_addr, DMA_FROM_DEVICE) | ||
| 372 | 376 | ||
| 373 | /* aie2_hwctx.c */ | 377 | /* aie2_hwctx.c */ |
| 374 | int aie2_hwctx_init(struct amdxdna_hwctx *hwctx); | 378 | int aie2_hwctx_init(struct amdxdna_hwctx *hwctx); |
| @@ -17,6 +17,7 @@ · 16 unmodified lines | |||
| 17 | #include "amdxdna_ctx.h" | 17 | #include "amdxdna_ctx.h" |
| 18 | #include "amdxdna_gem.h" | 18 | #include "amdxdna_gem.h" |
| 19 | #include "amdxdna_pci_drv.h" | 19 | #include "amdxdna_pci_drv.h" |
| 20 | #include "amdxdna_pm.h" | ||
| 20 | 21 | ||
| 21 | #define MAX_HWCTX_ID 255 | 22 | #define MAX_HWCTX_ID 255 |
| 22 | #define MAX_ARG_COUNT 4095 | 23 | #define MAX_ARG_COUNT 4095 |
| @@ -94,9 +95,12 @@ int amdxdna_hwctx_walk(struct amdxdna_client *client, void *arg, · 71 unmodified lines | |||
| 94 | 95 | ||
| 95 | void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size) | 96 | void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size) |
| 96 | { | 97 | { |
| 97 | struct amdxdna_cmd *cmd = abo->mem.kva; | 98 | struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); |
| 98 | u32 num_masks, count; | 99 | u32 num_masks, count; |
| 99 | 100 | ||
| 101 | if (!cmd) | ||
| 102 | return NULL; | ||
| 103 | |||
| 100 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) | 104 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) |
| 101 | num_masks = 0; | 105 | num_masks = 0; |
| 102 | else | 106 | else |
| @@ -118,10 +122,13 @@ void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size) · 15 unmodified lines | |||
| 118 | 122 | ||
| 119 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) | 123 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) |
| 120 | { | 124 | { |
| 121 | struct amdxdna_cmd *cmd = abo->mem.kva; | 125 | struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); |
| 122 | u32 num_masks, i; | 126 | u32 num_masks, i; |
| 123 | u32 *cu_mask; | 127 | u32 *cu_mask; |
| 124 | 128 | ||
| 129 | if (!cmd) | ||
| 130 | return INVALID_CU_IDX; | ||
| 131 | |||
| 125 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) | 132 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) |
| 126 | return INVALID_CU_IDX; | 133 | return INVALID_CU_IDX; |
| 127 | 134 | ||
| @@ -137,25 +144,34 @@ u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) · 9 unmodified lines | |||
| 137 | 144 | ||
| 138 | int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo, | 145 | int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo, |
| 139 | struct amdxdna_sched_job *job, u32 cmd_idx, | 146 | struct amdxdna_sched_job *job, u32 cmd_idx, |
| 140 | enum ert_cmd_state error_state) | 147 | enum ert_cmd_state error_state, |
| 148 | void *err_data, size_t size) | ||
| 141 | { | 149 | { |
| 142 | struct amdxdna_client *client = job->hwctx->client; | 150 | struct amdxdna_client *client = job->hwctx->client; |
| 143 | struct amdxdna_cmd *cmd = abo->mem.kva; | 151 | struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); |
| 144 | struct amdxdna_cmd_chain *cc = NULL; | 152 | struct amdxdna_cmd_chain *cc = NULL; |
| 145 | 153 | ||
| 154 | if (!cmd) | ||
| 155 | return -ENOMEM; | ||
| 156 | |||
| 146 | cmd->header &= ~AMDXDNA_CMD_STATE; | 157 | cmd->header &= ~AMDXDNA_CMD_STATE; |
| 147 | cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, error_state); | 158 | cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, error_state); |
| 148 | 159 | ||
| 149 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) { | 160 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) { |
| 150 | cc = amdxdna_cmd_get_payload(abo, NULL); | 161 | cc = amdxdna_cmd_get_payload(abo, NULL); |
| 151 | cc->error_index = (cmd_idx < cc->command_count) ? cmd_idx : 0; | 162 | cc->error_index = (cmd_idx < cc->command_count) ? cmd_idx : 0; |
| 152 | abo = amdxdna_gem_get_obj(client, cc->data[0], AMDXDNA_BO_CMD); | 163 | abo = amdxdna_gem_get_obj(client, cc->data[0], AMDXDNA_BO_SHARE); |
| 153 | if (!abo) | 164 | if (!abo) |
| 154 | return -EINVAL; | 165 | return -EINVAL; |
| 155 | cmd = abo->mem.kva; | 166 | cmd = amdxdna_gem_vmap(abo); |
| 167 | if (!cmd) | ||
| 168 | return -ENOMEM; | ||
| 156 | } | 169 | } |
| 157 | 170 | ||
| 158 | memset(cmd->data, 0xff, abo->mem.size - sizeof(*cmd)); | 171 | memset(cmd->data, 0xff, abo->mem.size - sizeof(*cmd)); |
| 172 | if (err_data) | ||
| 173 | memcpy(cmd->data, err_data, min(size, abo->mem.size - sizeof(*cmd))); | ||
| 174 | |||
| 159 | if (cc) | 175 | if (cc) |
| 160 | amdxdna_gem_put_obj(abo); | 176 | amdxdna_gem_put_obj(abo); |
| 161 | 177 | ||
| @@ -445,6 +461,7 @@ put_shmem_bo: · 283 unmodified lines | |||
| 445 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job) | 461 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job) |
| 446 | { | 462 | { |
| 447 | trace_amdxdna_debug_point(job->hwctx->name, job->seq, "job release"); | 463 | trace_amdxdna_debug_point(job->hwctx->name, job->seq, "job release"); |
| 464 | amdxdna_pm_suspend_put(job->hwctx->client->xdna); | ||
| 448 | amdxdna_arg_bos_put(job); | 465 | amdxdna_arg_bos_put(job); |
| 449 | amdxdna_gem_put_obj(job->cmd_bo); | 466 | amdxdna_gem_put_obj(job->cmd_bo); |
| 450 | dma_fence_put(job->fence); | 467 | dma_fence_put(job->fence); |
| @@ -468,7 +485,7 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 17 unmodified lines | |||
| 468 | job->drv_cmd = drv_cmd; | 485 | job->drv_cmd = drv_cmd; |
| 469 | 486 | ||
| 470 | if (cmd_bo_hdl != AMDXDNA_INVALID_BO_HANDLE) { | 487 | if (cmd_bo_hdl != AMDXDNA_INVALID_BO_HANDLE) { |
| 471 | job->cmd_bo = amdxdna_gem_get_obj(client, cmd_bo_hdl, AMDXDNA_BO_CMD); | 488 | job->cmd_bo = amdxdna_gem_get_obj(client, cmd_bo_hdl, AMDXDNA_BO_SHARE); |
| 472 | if (!job->cmd_bo) { | 489 | if (!job->cmd_bo) { |
| 473 | XDNA_ERR(xdna, "Failed to get cmd bo from %d", cmd_bo_hdl); | 490 | XDNA_ERR(xdna, "Failed to get cmd bo from %d", cmd_bo_hdl); |
| 474 | ret = -EINVAL; | 491 | ret = -EINVAL; |
| @@ -482,6 +499,12 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 7 unmodified lines | |||
| 482 | goto cmd_put; | 499 | goto cmd_put; |
| 483 | } | 500 | } |
| 484 | 501 | ||
| 502 | ret = amdxdna_pm_resume_get(xdna); | ||
| 503 | if (ret) { | ||
| 504 | XDNA_ERR(xdna, "Resume failed, ret %d", ret); | ||
| 505 | goto put_bos; | ||
| 506 | } | ||
| 507 | |||
| 485 | idx = srcu_read_lock(&client->hwctx_srcu); | 508 | idx = srcu_read_lock(&client->hwctx_srcu); |
| 486 | hwctx = xa_load(&client->hwctx_xa, hwctx_hdl); | 509 | hwctx = xa_load(&client->hwctx_xa, hwctx_hdl); |
| 487 | if (!hwctx) { | 510 | if (!hwctx) { |
| @@ -522,6 +545,8 @@ put_fence: · 34 unmodified lines | |||
| 522 | dma_fence_put(job->fence); | 545 | dma_fence_put(job->fence); |
| 523 | unlock_srcu: | 546 | unlock_srcu: |
| 524 | srcu_read_unlock(&client->hwctx_srcu, idx); | 547 | srcu_read_unlock(&client->hwctx_srcu, idx); |
| 548 | amdxdna_pm_suspend_put(xdna); | ||
| 549 | put_bos: | ||
| 525 | amdxdna_arg_bos_put(job); | 550 | amdxdna_arg_bos_put(job); |
| 526 | cmd_put: | 551 | cmd_put: |
| 527 | amdxdna_gem_put_obj(job->cmd_bo); | 552 | amdxdna_gem_put_obj(job->cmd_bo); |
| @@ -72,6 +72,13 @@ struct amdxdna_cmd_preempt_data { · 71 unmodified lines | |||
| 72 | u32 prop_args[]; /* properties and regular kernel arguments */ | 72 | u32 prop_args[]; /* properties and regular kernel arguments */ |
| 73 | }; | 73 | }; |
| 74 | 74 | ||
| 75 | #define AMDXDNA_CMD_CTX_HEALTH_V1 1 | ||
| 76 | #define AMDXDNA_CMD_CTX_HEALTH_AIE2 0 | ||
| 77 | struct amdxdna_ctx_health { | ||
| 78 | u32 version; | ||
| 79 | u32 npu_gen; | ||
| 80 | }; | ||
| 81 | |||
| 75 | /* Exec buffer command header format */ | 82 | /* Exec buffer command header format */ |
| 76 | #define AMDXDNA_CMD_STATE GENMASK(3, 0) | 83 | #define AMDXDNA_CMD_STATE GENMASK(3, 0) |
| 77 | #define AMDXDNA_CMD_EXTRA_CU_MASK GENMASK(11, 10) | 84 | #define AMDXDNA_CMD_EXTRA_CU_MASK GENMASK(11, 10) |
| @@ -122,6 +129,11 @@ struct amdxdna_drv_cmd { · 44 unmodified lines | |||
| 122 | u32 result; | 129 | u32 result; |
| 123 | }; | 130 | }; |
| 124 | 131 | ||
| 132 | struct app_health_report; | ||
| 133 | union amdxdna_job_priv { | ||
| 134 | struct app_health_report *aie2_health; | ||
| 135 | }; | ||
| 136 | |||
| 125 | struct amdxdna_sched_job { | 137 | struct amdxdna_sched_job { |
| 126 | struct drm_sched_job base; | 138 | struct drm_sched_job base; |
| 127 | struct kref refcnt; | 139 | struct kref refcnt; |
| @@ -136,14 +148,20 @@ struct amdxdna_sched_job { · 8 unmodified lines | |||
| 136 | u64 seq; | 148 | u64 seq; |
| 137 | struct amdxdna_drv_cmd *drv_cmd; | 149 | struct amdxdna_drv_cmd *drv_cmd; |
| 138 | struct amdxdna_gem_obj *cmd_bo; | 150 | struct amdxdna_gem_obj *cmd_bo; |
| 151 | union amdxdna_job_priv priv; | ||
| 139 | size_t bo_cnt; | 152 | size_t bo_cnt; |
| 140 | struct drm_gem_object *bos[] __counted_by(bo_cnt); | 153 | struct drm_gem_object *bos[] __counted_by(bo_cnt); |
| 141 | }; | 154 | }; |
| 142 | 155 | ||
| 156 | #define aie2_job_health priv.aie2_health | ||
| 157 | |||
| 143 | static inline u32 | 158 | static inline u32 |
| 144 | amdxdna_cmd_get_op(struct amdxdna_gem_obj *abo) | 159 | amdxdna_cmd_get_op(struct amdxdna_gem_obj *abo) |
| 145 | { | 160 | { |
| 146 | struct amdxdna_cmd *cmd = abo->mem.kva; | 161 | struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); |
| 162 | |||
| 163 | if (!cmd) | ||
| 164 | return ERT_INVALID_CMD; | ||
| 147 | 165 | ||
| 148 | return FIELD_GET(AMDXDNA_CMD_OPCODE, cmd->header); | 166 | return FIELD_GET(AMDXDNA_CMD_OPCODE, cmd->header); |
| 149 | } | 167 | } |
| @@ -151,7 +169,10 @@ amdxdna_cmd_get_op(struct amdxdna_gem_obj *abo) · 1 unmodified line | |||
| 151 | static inline void | 169 | static inline void |
| 152 | amdxdna_cmd_set_state(struct amdxdna_gem_obj *abo, enum ert_cmd_state s) | 170 | amdxdna_cmd_set_state(struct amdxdna_gem_obj *abo, enum ert_cmd_state s) |
| 153 | { | 171 | { |
| 154 | struct amdxdna_cmd *cmd = abo->mem.kva; | 172 | struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); |
| 173 | |||
| 174 | if (!cmd) | ||
| 175 | return; | ||
| 155 | 176 | ||
| 156 | cmd->header &= ~AMDXDNA_CMD_STATE; | 177 | cmd->header &= ~AMDXDNA_CMD_STATE; |
| 157 | cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, s); | 178 | cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, s); |
| @@ -160,7 +181,10 @@ amdxdna_cmd_set_state(struct amdxdna_gem_obj *abo, enum ert_cmd_state s) · 2 unmodified lines | |||
| 160 | static inline enum ert_cmd_state | 181 | static inline enum ert_cmd_state |
| 161 | amdxdna_cmd_get_state(struct amdxdna_gem_obj *abo) | 182 | amdxdna_cmd_get_state(struct amdxdna_gem_obj *abo) |
| 162 | { | 183 | { |
| 163 | struct amdxdna_cmd *cmd = abo->mem.kva; | 184 | struct amdxdna_cmd *cmd = amdxdna_gem_vmap(abo); |
| 185 | |||
| 186 | if (!cmd) | ||
| 187 | return ERT_CMD_STATE_INVALID; | ||
| 164 | 188 | ||
| 165 | return FIELD_GET(AMDXDNA_CMD_STATE, cmd->header); | 189 | return FIELD_GET(AMDXDNA_CMD_STATE, cmd->header); |
| 166 | } | 190 | } |
| @@ -169,7 +193,8 @@ void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size); · 2 unmodified lines | |||
| 169 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo); | 193 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo); |
| 170 | int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo, | 194 | int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo, |
| 171 | struct amdxdna_sched_job *job, u32 cmd_idx, | 195 | struct amdxdna_sched_job *job, u32 cmd_idx, |
| 172 | enum ert_cmd_state error_state); | 196 | enum ert_cmd_state error_state, |
| 197 | void *err_data, size_t size); | ||
| 173 | 198 | ||
| 174 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job); | 199 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job); |
| 175 | void amdxdna_hwctx_remove_all(struct amdxdna_client *client); | 200 | void amdxdna_hwctx_remove_all(struct amdxdna_client *client); |
| @@ -30,7 +30,6 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo) · 29 unmodified lines | |||
| 30 | struct amdxdna_dev *xdna = client->xdna; | 30 | struct amdxdna_dev *xdna = client->xdna; |
| 31 | struct amdxdna_mem *mem = &abo->mem; | 31 | struct amdxdna_mem *mem = &abo->mem; |
| 32 | struct amdxdna_gem_obj *heap; | 32 | struct amdxdna_gem_obj *heap; |
| 33 | u64 offset; | ||
| 34 | u32 align; | 33 | u32 align; |
| 35 | int ret; | 34 | int ret; |
| 36 | 35 | ||
| @@ -42,7 +41,7 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo) · 5 unmodified lines | |||
| 42 | goto unlock_out; | 41 | goto unlock_out; |
| 43 | } | 42 | } |
| 44 | 43 | ||
| 45 | if (heap->mem.userptr == AMDXDNA_INVALID_ADDR) { | 44 | if (amdxdna_gem_uva(heap) == AMDXDNA_INVALID_ADDR) { |
| 46 | XDNA_ERR(xdna, "Invalid dev heap userptr"); | 45 | XDNA_ERR(xdna, "Invalid dev heap userptr"); |
| 47 | ret = -EINVAL; | 46 | ret = -EINVAL; |
| 48 | goto unlock_out; | 47 | goto unlock_out; |
| @@ -64,10 +63,7 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo) · 15 unmodified lines | |||
| 64 | goto unlock_out; | 63 | goto unlock_out; |
| 65 | } | 64 | } |
| 66 | 65 | ||
| 67 | mem->dev_addr = abo->mm_node.start; | 66 | client->heap_usage += mem->size; |
| 68 | offset = mem->dev_addr - heap->mem.dev_addr; | ||
| 69 | mem->userptr = heap->mem.userptr + offset; | ||
| 70 | mem->kva = heap->mem.kva + offset; | ||
| 71 | 67 | ||
| 72 | drm_gem_object_get(to_gobj(heap)); | 68 | drm_gem_object_get(to_gobj(heap)); |
| 73 | 69 | ||
| @@ -77,6 +73,45 @@ unlock_out: · 3 unmodified lines | |||
| 77 | return ret; | 73 | return ret; |
| 78 | } | 74 | } |
| 79 | 75 | ||
| 76 | static void | ||
| 77 | amdxdna_gem_heap_free(struct amdxdna_gem_obj *abo) | ||
| 78 | { | ||
| 79 | struct amdxdna_client *client = abo->client; | ||
| 80 | struct amdxdna_gem_obj *heap; | ||
| 81 | |||
| 82 | mutex_lock(&client->mm_lock); | ||
| 83 | |||
| 84 | drm_mm_remove_node(&abo->mm_node); | ||
| 85 | client->heap_usage -= abo->mem.size; | ||
| 86 | heap = client->dev_heap; | ||
| 87 | drm_gem_object_put(to_gobj(heap)); | ||
| 88 | |||
| 89 | mutex_unlock(&client->mm_lock); | ||
| 90 | } | ||
| 91 | |||
| 92 | static struct amdxdna_gem_obj * | ||
| 93 | amdxdna_gem_create_obj(struct drm_device *dev, size_t size) | ||
| 94 | { | ||
| 95 | struct amdxdna_gem_obj *abo; | ||
| 96 | |||
| 97 | abo = kzalloc_obj(*abo); | ||
| 98 | if (!abo) | ||
| 99 | return ERR_PTR(-ENOMEM); | ||
| 100 | |||
| 101 | abo->pinned = false; | ||
| 102 | abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE; | ||
| 103 | mutex_init(&abo->lock); | ||
| 104 | |||
| 105 | abo->mem.dma_addr = AMDXDNA_INVALID_ADDR; | ||
| 106 | abo->mem.uva = AMDXDNA_INVALID_ADDR; | ||
| 107 | abo->mem.size = size; | ||
| 108 | abo->open_ref = 0; | ||
| 109 | abo->internal = false; | ||
| 110 | INIT_LIST_HEAD(&abo->mem.umap_list); | ||
| 111 | |||
| 112 | return abo; | ||
| 113 | } | ||
| 114 | |||
| 80 | static void | 115 | static void |
| 81 | amdxdna_gem_destroy_obj(struct amdxdna_gem_obj *abo) | 116 | amdxdna_gem_destroy_obj(struct amdxdna_gem_obj *abo) |
| 82 | { | 117 | { |
| @@ -84,19 +119,75 @@ amdxdna_gem_destroy_obj(struct amdxdna_gem_obj *abo) · 1 unmodified line | |||
| 84 | kfree(abo); | 119 | kfree(abo); |
| 85 | } | 120 | } |
| 86 | 121 | ||
| 87 | static void | 122 | /* |
| 88 | amdxdna_gem_heap_free(struct amdxdna_gem_obj *abo) | 123 | * Obtains a kernel virtual address on the BO (usually of small size). |
| 124 | * The mapping is established on the first call and stays valid until | ||
| 125 | * amdxdna_gem_vunmap() is called. | ||
| 126 | */ | ||
| 127 | void *amdxdna_gem_vmap(struct amdxdna_gem_obj *abo) | ||
| 89 | { | 128 | { |
| 90 | struct amdxdna_gem_obj *heap; | 129 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); |
| 130 | int ret; | ||
| 91 | 131 | ||
| 92 | mutex_lock(&abo->client->mm_lock); | 132 | if (abo->mem.kva) |
| 133 | return abo->mem.kva; | ||
| 93 | 134 | ||
| 94 | drm_mm_remove_node(&abo->mm_node); | 135 | /* The first call to get the kva, taking slow path. */ |
| 136 | guard(mutex)(&abo->lock); | ||
| 95 | 137 | ||
| 96 | heap = abo->client->dev_heap; | 138 | if (!abo->mem.kva) { |
| 97 | drm_gem_object_put(to_gobj(heap)); | 139 | ret = drm_gem_vmap(to_gobj(abo), &map); |
| 140 | if (ret) | ||
| 141 | XDNA_ERR(abo->client->xdna, "Vmap bo failed, ret %d", ret); | ||
| 142 | else | ||
| 143 | abo->mem.kva = map.vaddr; | ||
| 144 | } | ||
| 145 | return abo->mem.kva; | ||
| 146 | } | ||
| 147 | |||
| 148 | /* | ||
| 149 | * Free mapping established through amdxdna_gem_vmap() | ||
| 150 | */ | ||
| 151 | static void amdxdna_gem_vunmap(struct amdxdna_gem_obj *abo) | ||
| 152 | { | ||
| 153 | guard(mutex)(&abo->lock); | ||
| 154 | |||
| 155 | if (abo->mem.kva) { | ||
| 156 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(abo->mem.kva); | ||
| 157 | |||
| 158 | drm_gem_vunmap(to_gobj(abo), &map); | ||
| 159 | abo->mem.kva = NULL; | ||
| 160 | } | ||
| 161 | } | ||
| 162 | |||
| 163 | /* | ||
| 164 | * Obtain the user virtual address for accessing the BO. | ||
| 165 | * It can be used for device to access the BO when PASID is enabled. | ||
| 166 | */ | ||
| 167 | u64 amdxdna_gem_uva(struct amdxdna_gem_obj *abo) | ||
| 168 | { | ||
| 169 | if (abo->type == AMDXDNA_BO_DEV) { | ||
| 170 | struct amdxdna_gem_obj *heap = abo->client->dev_heap; | ||
| 171 | u64 off = amdxdna_dev_bo_offset(abo); | ||
| 172 | |||
| 173 | if (amdxdna_gem_uva(heap) != AMDXDNA_INVALID_ADDR) | ||
| 174 | return amdxdna_gem_uva(heap) + off; | ||
| 175 | return AMDXDNA_INVALID_ADDR; | ||
| 176 | } | ||
| 98 | 177 | ||
| 99 | mutex_unlock(&abo->client->mm_lock); | 178 | return abo->mem.uva; |
| 179 | } | ||
| 180 | |||
| 181 | /* | ||
| 182 | * Obtain the address for device to access the BO. | ||
| 183 | */ | ||
| 184 | u64 amdxdna_gem_dev_addr(struct amdxdna_gem_obj *abo) | ||
| 185 | { | ||
| 186 | if (abo->type == AMDXDNA_BO_DEV_HEAP) | ||
| 187 | return abo->client->xdna->dev_info->dev_mem_base; | ||
| 188 | if (abo->type == AMDXDNA_BO_DEV) | ||
| 189 | return abo->mm_node.start; | ||
| 190 | return amdxdna_obj_dma_addr(abo); | ||
| 100 | } | 191 | } |
| 101 | 192 | ||
| 102 | static bool amdxdna_hmm_invalidate(struct mmu_interval_notifier *mni, | 193 | static bool amdxdna_hmm_invalidate(struct mmu_interval_notifier *mni, |
| @@ -161,16 +252,19 @@ static void amdxdna_hmm_unregister(struct amdxdna_gem_obj *abo, · 58 unmodified lines | |||
| 161 | static void amdxdna_umap_release(struct kref *ref) | 252 | static void amdxdna_umap_release(struct kref *ref) |
| 162 | { | 253 | { |
| 163 | struct amdxdna_umap *mapp = container_of(ref, struct amdxdna_umap, refcnt); | 254 | struct amdxdna_umap *mapp = container_of(ref, struct amdxdna_umap, refcnt); |
| 255 | struct amdxdna_gem_obj *abo = mapp->abo; | ||
| 164 | struct vm_area_struct *vma = mapp->vma; | 256 | struct vm_area_struct *vma = mapp->vma; |
| 165 | struct amdxdna_dev *xdna; | 257 | struct amdxdna_dev *xdna; |
| 166 | 258 | ||
| 167 | mmu_interval_notifier_remove(&mapp->notifier); | 259 | mmu_interval_notifier_remove(&mapp->notifier); |
| 168 | if (is_import_bo(mapp->abo) && vma->vm_file && vma->vm_file->f_mapping) | 260 | if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping) |
| 169 | mapping_clear_unevictable(vma->vm_file->f_mapping); | 261 | mapping_clear_unevictable(vma->vm_file->f_mapping); |
| 170 | 262 | ||
| 171 | xdna = to_xdna_dev(to_gobj(mapp->abo)->dev); | 263 | xdna = to_xdna_dev(to_gobj(mapp->abo)->dev); |
| 172 | down_write(&xdna->notifier_lock); | 264 | down_write(&xdna->notifier_lock); |
| 173 | list_del(&mapp->node); | 265 | list_del(&mapp->node); |
| 266 | if (list_empty(&abo->mem.umap_list)) | ||
| 267 | abo->mem.uva = AMDXDNA_INVALID_ADDR; | ||
| 174 | up_write(&xdna->notifier_lock); | 268 | up_write(&xdna->notifier_lock); |
| 175 | 269 | ||
| 176 | kvfree(mapp->range.hmm_pfns); | 270 | kvfree(mapp->range.hmm_pfns); |
| @@ -232,13 +326,13 @@ static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo, · 55 unmodified lines | |||
| 232 | mapp->abo = abo; | 326 | mapp->abo = abo; |
| 233 | kref_init(&mapp->refcnt); | 327 | kref_init(&mapp->refcnt); |
| 234 | 328 | ||
| 235 | if (abo->mem.userptr == AMDXDNA_INVALID_ADDR) | ||
| 236 | abo->mem.userptr = addr; | ||
| 237 | INIT_WORK(&mapp->hmm_unreg_work, amdxdna_hmm_unreg_work); | 329 | INIT_WORK(&mapp->hmm_unreg_work, amdxdna_hmm_unreg_work); |
| 238 | if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping) | 330 | if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping) |
| 239 | mapping_set_unevictable(vma->vm_file->f_mapping); | 331 | mapping_set_unevictable(vma->vm_file->f_mapping); |
| 240 | 332 | ||
| 241 | down_write(&xdna->notifier_lock); | 333 | down_write(&xdna->notifier_lock); |
| 334 | if (list_empty(&abo->mem.umap_list)) | ||
| 335 | abo->mem.uva = addr; | ||
| 242 | list_add_tail(&mapp->node, &abo->mem.umap_list); | 336 | list_add_tail(&mapp->node, &abo->mem.umap_list); |
| 243 | up_write(&xdna->notifier_lock); | 337 | up_write(&xdna->notifier_lock); |
| 244 | 338 | ||
| @@ -256,10 +350,11 @@ static void amdxdna_gem_dev_obj_free(struct drm_gem_object *gobj) · 11 unmodified lines | |||
| 256 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); | 350 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); |
| 257 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | 351 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| 258 | 352 | ||
| 259 | XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, abo->mem.dev_addr); | 353 | XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, amdxdna_gem_dev_addr(abo)); |
| 260 | if (abo->pinned) | 354 | if (abo->pinned) |
| 261 | amdxdna_gem_unpin(abo); | 355 | amdxdna_gem_unpin(abo); |
| 262 | 356 | ||
| 357 | amdxdna_gem_vunmap(abo); | ||
| 263 | amdxdna_gem_heap_free(abo); | 358 | amdxdna_gem_heap_free(abo); |
| 264 | drm_gem_object_release(gobj); | 359 | drm_gem_object_release(gobj); |
| 265 | amdxdna_gem_destroy_obj(abo); | 360 | amdxdna_gem_destroy_obj(abo); |
| @@ -390,35 +485,6 @@ static const struct dma_buf_ops amdxdna_dmabuf_ops = { · 124 unmodified lines | |||
| 390 | .vunmap = drm_gem_dmabuf_vunmap, | 485 | .vunmap = drm_gem_dmabuf_vunmap, |
| 391 | }; | 486 | }; |
| 392 | 487 | ||
| 393 | static int amdxdna_gem_obj_vmap(struct amdxdna_gem_obj *abo, void **vaddr) | ||
| 394 | { | ||
| 395 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); | ||
| 396 | int ret; | ||
| 397 | |||
| 398 | if (is_import_bo(abo)) | ||
| 399 | ret = dma_buf_vmap_unlocked(abo->dma_buf, &map); | ||
| 400 | else | ||
| 401 | ret = drm_gem_vmap(to_gobj(abo), &map); | ||
| 402 | |||
| 403 | *vaddr = map.vaddr; | ||
| 404 | return ret; | ||
| 405 | } | ||
| 406 | |||
| 407 | static void amdxdna_gem_obj_vunmap(struct amdxdna_gem_obj *abo) | ||
| 408 | { | ||
| 409 | struct iosys_map map; | ||
| 410 | |||
| 411 | if (!abo->mem.kva) | ||
| 412 | return; | ||
| 413 | |||
| 414 | iosys_map_set_vaddr(&map, abo->mem.kva); | ||
| 415 | |||
| 416 | if (is_import_bo(abo)) | ||
| 417 | dma_buf_vunmap_unlocked(abo->dma_buf, &map); | ||
| 418 | else | ||
| 419 | drm_gem_vunmap(to_gobj(abo), &map); | ||
| 420 | } | ||
| 421 | |||
| 422 | static struct dma_buf *amdxdna_gem_prime_export(struct drm_gem_object *gobj, int flags) | 488 | static struct dma_buf *amdxdna_gem_prime_export(struct drm_gem_object *gobj, int flags) |
| 423 | { | 489 | { |
| 424 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | 490 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| @@ -447,13 +513,55 @@ static void amdxdna_imported_obj_free(struct amdxdna_gem_obj *abo) · 22 unmodified lines | |||
| 447 | kfree(abo); | 513 | kfree(abo); |
| 448 | } | 514 | } |
| 449 | 515 | ||
| 516 | static inline bool | ||
| 517 | amdxdna_gem_skip_bo_usage(struct amdxdna_gem_obj *abo) | ||
| 518 | { | ||
| 519 | /* Do not count imported BOs since the buffer is not allocated by us. */ | ||
| 520 | if (is_import_bo(abo)) | ||
| 521 | return true; | ||
| 522 | |||
| 523 | /* Already counted as part of HEAP BO */ | ||
| 524 | if (abo->type == AMDXDNA_BO_DEV) | ||
| 525 | return true; | ||
| 526 | |||
| 527 | return false; | ||
| 528 | } | ||
| 529 | |||
| 530 | static void | ||
| 531 | amdxdna_gem_add_bo_usage(struct amdxdna_gem_obj *abo) | ||
| 532 | { | ||
| 533 | struct amdxdna_client *client = abo->client; | ||
| 534 | |||
| 535 | if (amdxdna_gem_skip_bo_usage(abo)) | ||
| 536 | return; | ||
| 537 | |||
| 538 | guard(mutex)(&client->mm_lock); | ||
| 539 | |||
| 540 | client->total_bo_usage += abo->mem.size; | ||
| 541 | if (abo->internal) | ||
| 542 | client->total_int_bo_usage += abo->mem.size; | ||
| 543 | } | ||
| 544 | |||
| 545 | static void | ||
| 546 | amdxdna_gem_del_bo_usage(struct amdxdna_gem_obj *abo) | ||
| 547 | { | ||
| 548 | struct amdxdna_client *client = abo->client; | ||
| 549 | |||
| 550 | if (amdxdna_gem_skip_bo_usage(abo)) | ||
| 551 | return; | ||
| 552 | |||
| 553 | guard(mutex)(&client->mm_lock); | ||
| 554 | |||
| 555 | client->total_bo_usage -= abo->mem.size; | ||
| 556 | if (abo->internal) | ||
| 557 | client->total_int_bo_usage -= abo->mem.size; | ||
| 558 | } | ||
| 559 | |||
| 450 | static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) | 560 | static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) |
| 451 | { | 561 | { |
| 452 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); | 562 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); |
| 453 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | 563 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| 454 | 564 | ||
| 455 | XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, abo->mem.dev_addr); | ||
| 456 | |||
| 457 | amdxdna_hmm_unregister(abo, NULL); | 565 | amdxdna_hmm_unregister(abo, NULL); |
| 458 | flush_workqueue(xdna->notifier_wq); | 566 | flush_workqueue(xdna->notifier_wq); |
| 459 | 567 | ||
| @@ -463,23 +571,76 @@ static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) · 3 unmodified lines | |||
| 463 | if (abo->type == AMDXDNA_BO_DEV_HEAP) | 571 | if (abo->type == AMDXDNA_BO_DEV_HEAP) |
| 464 | drm_mm_takedown(&abo->mm); | 572 | drm_mm_takedown(&abo->mm); |
| 465 | 573 | ||
| 466 | amdxdna_gem_obj_vunmap(abo); | 574 | if (amdxdna_iova_on(xdna)) |
| 575 | amdxdna_iommu_unmap_bo(xdna, abo); | ||
| 576 | |||
| 577 | amdxdna_gem_vunmap(abo); | ||
| 467 | mutex_destroy(&abo->lock); | 578 | mutex_destroy(&abo->lock); |
| 468 | 579 | ||
| 469 | if (is_import_bo(abo)) { | 580 | if (is_import_bo(abo)) |
| 470 | amdxdna_imported_obj_free(abo); | 581 | amdxdna_imported_obj_free(abo); |
| 471 | return; | 582 | else |
| 583 | drm_gem_shmem_free(&abo->base); | ||
| 584 | } | ||
| 585 | |||
| 586 | static int amdxdna_gem_obj_open(struct drm_gem_object *gobj, struct drm_file *filp) | ||
| 587 | { | ||
| 588 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); | ||
| 589 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | ||
| 590 | int ret; | ||
| 591 | |||
| 592 | guard(mutex)(&abo->lock); | ||
| 593 | abo->open_ref++; | ||
| 594 | |||
| 595 | if (abo->open_ref == 1) { | ||
| 596 | /* Attached to the client when first opened by it. */ | ||
| 597 | abo->client = filp->driver_priv; | ||
| 598 | amdxdna_gem_add_bo_usage(abo); | ||
| 599 | } | ||
| 600 | if (amdxdna_iova_on(xdna)) { | ||
| 601 | ret = amdxdna_iommu_map_bo(xdna, abo); | ||
| 602 | if (ret) | ||
| 603 | return ret; | ||
| 604 | } | ||
| 605 | |||
| 606 | return 0; | ||
| 607 | } | ||
| 608 | |||
| 609 | static void amdxdna_gem_obj_close(struct drm_gem_object *gobj, struct drm_file *filp) | ||
| 610 | { | ||
| 611 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | ||
| 612 | |||
| 613 | guard(mutex)(&abo->lock); | ||
| 614 | abo->open_ref--; | ||
| 615 | |||
| 616 | if (abo->open_ref == 0) { | ||
| 617 | amdxdna_gem_del_bo_usage(abo); | ||
| 618 | /* Detach from the client when last closed by it. */ | ||
| 619 | abo->client = NULL; | ||
| 472 | } | 620 | } |
| 621 | } | ||
| 622 | |||
| 623 | static int amdxdna_gem_dev_obj_vmap(struct drm_gem_object *obj, struct iosys_map *map) | ||
| 624 | { | ||
| 625 | struct amdxdna_gem_obj *abo = to_xdna_obj(obj); | ||
| 626 | void *base = amdxdna_gem_vmap(abo->client->dev_heap); | ||
| 627 | u64 offset = amdxdna_dev_bo_offset(abo); | ||
| 473 | 628 | ||
| 474 | drm_gem_shmem_free(&abo->base); | 629 | if (!base) |
| 630 | return -ENOMEM; | ||
| 631 | iosys_map_set_vaddr(map, base + offset); | ||
| 632 | return 0; | ||
| 475 | } | 633 | } |
| 476 | 634 | ||
| 477 | static const struct drm_gem_object_funcs amdxdna_gem_dev_obj_funcs = { | 635 | static const struct drm_gem_object_funcs amdxdna_gem_dev_obj_funcs = { |
| 478 | .free = amdxdna_gem_dev_obj_free, | 636 | .free = amdxdna_gem_dev_obj_free, |
| 637 | .vmap = amdxdna_gem_dev_obj_vmap, | ||
| 479 | }; | 638 | }; |
| 480 | 639 | ||
| 481 | static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = { | 640 | static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = { |
| 482 | .free = amdxdna_gem_obj_free, | 641 | .free = amdxdna_gem_obj_free, |
| 642 | .open = amdxdna_gem_obj_open, | ||
| 643 | .close = amdxdna_gem_obj_close, | ||
| 483 | .print_info = drm_gem_shmem_object_print_info, | 644 | .print_info = drm_gem_shmem_object_print_info, |
| 484 | .pin = drm_gem_shmem_object_pin, | 645 | .pin = drm_gem_shmem_object_pin, |
| 485 | .unpin = drm_gem_shmem_object_unpin, | 646 | .unpin = drm_gem_shmem_object_unpin, |
| @@ -491,30 +652,9 @@ static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = { · 5 unmodified lines | |||
| 491 | .export = amdxdna_gem_prime_export, | 652 | .export = amdxdna_gem_prime_export, |
| 492 | }; | 653 | }; |
| 493 | 654 | ||
| 494 | static struct amdxdna_gem_obj * | ||
| 495 | amdxdna_gem_create_obj(struct drm_device *dev, size_t size) | ||
| 496 | { | ||
| 497 | struct amdxdna_gem_obj *abo; | ||
| 498 | |||
| 499 | abo = kzalloc_obj(*abo); | ||
| 500 | if (!abo) | ||
| 501 | return ERR_PTR(-ENOMEM); | ||
| 502 | |||
| 503 | abo->pinned = false; | ||
| 504 | abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE; | ||
| 505 | mutex_init(&abo->lock); | ||
| 506 | |||
| 507 | abo->mem.userptr = AMDXDNA_INVALID_ADDR; | ||
| 508 | abo->mem.dev_addr = AMDXDNA_INVALID_ADDR; | ||
| 509 | abo->mem.size = size; | ||
| 510 | INIT_LIST_HEAD(&abo->mem.umap_list); | ||
| 511 | |||
| 512 | return abo; | ||
| 513 | } | ||
| 514 | |||
| 515 | /* For drm_driver->gem_create_object callback */ | 655 | /* For drm_driver->gem_create_object callback */ |
| 516 | struct drm_gem_object * | 656 | struct drm_gem_object * |
| 517 | amdxdna_gem_create_object_cb(struct drm_device *dev, size_t size) | 657 | amdxdna_gem_create_shmem_object_cb(struct drm_device *dev, size_t size) |
| 518 | { | 658 | { |
| 519 | struct amdxdna_gem_obj *abo; | 659 | struct amdxdna_gem_obj *abo; |
| 520 | 660 | ||
| @@ -528,8 +668,9 @@ amdxdna_gem_create_object_cb(struct drm_device *dev, size_t size) · 7 unmodified lines | |||
| 528 | } | 668 | } |
| 529 | 669 | ||
| 530 | static struct amdxdna_gem_obj * | 670 | static struct amdxdna_gem_obj * |
| 531 | amdxdna_gem_create_shmem_object(struct drm_device *dev, size_t size) | 671 | amdxdna_gem_create_shmem_object(struct drm_device *dev, struct amdxdna_drm_create_bo *args) |
| 532 | { | 672 | { |
| 673 | size_t size = args->size; | ||
| 533 | struct drm_gem_shmem_object *shmem = drm_gem_shmem_create(dev, size); | 674 | struct drm_gem_shmem_object *shmem = drm_gem_shmem_create(dev, size); |
| 534 | 675 | ||
| 535 | if (IS_ERR(shmem)) | 676 | if (IS_ERR(shmem)) |
| @@ -543,7 +684,6 @@ static struct amdxdna_gem_obj * · 7 unmodified lines | |||
| 543 | amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create_bo *args) | 684 | amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create_bo *args) |
| 544 | { | 685 | { |
| 545 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | 686 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 546 | enum amdxdna_ubuf_flag flags = 0; | ||
| 547 | struct amdxdna_drm_va_tbl va_tbl; | 687 | struct amdxdna_drm_va_tbl va_tbl; |
| 548 | struct drm_gem_object *gobj; | 688 | struct drm_gem_object *gobj; |
| 549 | struct dma_buf *dma_buf; | 689 | struct dma_buf *dma_buf; |
| @@ -554,10 +694,7 @@ amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create · 4 unmodified lines | |||
| 554 | } | 694 | } |
| 555 | 695 | ||
| 556 | if (va_tbl.num_entries) { | 696 | if (va_tbl.num_entries) { |
| 557 | if (args->type == AMDXDNA_BO_CMD) | 697 | dma_buf = amdxdna_get_ubuf(dev, va_tbl.num_entries, |
| 558 | flags |= AMDXDNA_UBUF_FLAG_MAP_DMA; | ||
| 559 | |||
| 560 | dma_buf = amdxdna_get_ubuf(dev, flags, va_tbl.num_entries, | ||
| 561 | u64_to_user_ptr(args->vaddr + sizeof(va_tbl))); | 698 | u64_to_user_ptr(args->vaddr + sizeof(va_tbl))); |
| 562 | } else { | 699 | } else { |
| 563 | dma_buf = dma_buf_get(va_tbl.dmabuf_fd); | 700 | dma_buf = dma_buf_get(va_tbl.dmabuf_fd); |
| @@ -577,18 +714,6 @@ amdxdna_gem_create_ubuf_object(struct drm_device *dev, struct amdxdna_drm_create · 13 unmodified lines | |||
| 577 | return to_xdna_obj(gobj); | 714 | return to_xdna_obj(gobj); |
| 578 | } | 715 | } |
| 579 | 716 | ||
| 580 | static struct amdxdna_gem_obj * | ||
| 581 | amdxdna_gem_create_object(struct drm_device *dev, | ||
| 582 | struct amdxdna_drm_create_bo *args) | ||
| 583 | { | ||
| 584 | size_t aligned_sz = PAGE_ALIGN(args->size); | ||
| 585 | |||
| 586 | if (args->vaddr) | ||
| 587 | return amdxdna_gem_create_ubuf_object(dev, args); | ||
| 588 | |||
| 589 | return amdxdna_gem_create_shmem_object(dev, aligned_sz); | ||
| 590 | } | ||
| 591 | |||
| 592 | struct drm_gem_object * | 717 | struct drm_gem_object * |
| 593 | amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf) | 718 | amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf) |
| 594 | { | 719 | { |
| @@ -621,6 +746,8 @@ amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf) · 26 unmodified lines | |||
| 621 | abo = to_xdna_obj(gobj); | 746 | abo = to_xdna_obj(gobj); |
| 622 | abo->attach = attach; | 747 | abo->attach = attach; |
| 623 | abo->dma_buf = dma_buf; | 748 | abo->dma_buf = dma_buf; |
| 749 | abo->type = AMDXDNA_BO_SHARE; | ||
| 750 | gobj->resv = dma_buf->resv; | ||
| 624 | 751 | ||
| 625 | return gobj; | 752 | return gobj; |
| 626 | 753 | ||
| @@ -635,93 +762,101 @@ put_buf: · 8 unmodified lines | |||
| 635 | } | 762 | } |
| 636 | 763 | ||
| 637 | static struct amdxdna_gem_obj * | 764 | static struct amdxdna_gem_obj * |
| 638 | amdxdna_drm_alloc_shmem(struct drm_device *dev, | 765 | amdxdna_drm_create_share_bo(struct drm_device *dev, |
| 639 | struct amdxdna_drm_create_bo *args, | 766 | struct amdxdna_drm_create_bo *args, struct drm_file *filp) |
| 640 | struct drm_file *filp) | ||
| 641 | { | 767 | { |
| 642 | struct amdxdna_client *client = filp->driver_priv; | ||
| 643 | struct amdxdna_gem_obj *abo; | 768 | struct amdxdna_gem_obj *abo; |
| 644 | 769 | ||
| 645 | abo = amdxdna_gem_create_object(dev, args); | 770 | if (args->vaddr) |
| 771 | abo = amdxdna_gem_create_ubuf_object(dev, args); | ||
| 772 | else | ||
| 773 | abo = amdxdna_gem_create_shmem_object(dev, args); | ||
| 646 | if (IS_ERR(abo)) | 774 | if (IS_ERR(abo)) |
| 647 | return ERR_CAST(abo); | 775 | return ERR_CAST(abo); |
| 648 | 776 | ||
| 649 | abo->client = client; | 777 | if (args->type == AMDXDNA_BO_DEV_HEAP) { |
| 650 | abo->type = AMDXDNA_BO_SHMEM; | 778 | abo->type = AMDXDNA_BO_DEV_HEAP; |
| 779 | abo->internal = true; | ||
| 780 | } else { | ||
| 781 | abo->type = AMDXDNA_BO_SHARE; | ||
| 782 | abo->internal = args->type == AMDXDNA_BO_CMD; | ||
| 783 | } | ||
| 651 | 784 | ||
| 652 | return abo; | 785 | return abo; |
| 653 | } | 786 | } |
| 654 | 787 | ||
| 655 | static struct amdxdna_gem_obj * | 788 | static struct amdxdna_gem_obj * |
| 656 | amdxdna_drm_create_dev_heap(struct drm_device *dev, | 789 | amdxdna_drm_create_dev_heap_bo(struct drm_device *dev, |
| 657 | struct amdxdna_drm_create_bo *args, | 790 | struct amdxdna_drm_create_bo *args, struct drm_file *filp) |
| 658 | struct drm_file *filp) | ||
| 659 | { | 791 | { |
| 660 | struct amdxdna_client *client = filp->driver_priv; | 792 | struct amdxdna_client *client = filp->driver_priv; |
| 661 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | 793 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 662 | struct amdxdna_gem_obj *abo; | 794 | struct amdxdna_gem_obj *abo; |
| 663 | int ret; | 795 | int ret; |
| 664 | 796 | ||
| 665 | if (args->size > xdna->dev_info->dev_mem_size) { | 797 | WARN_ON(!is_power_of_2(xdna->dev_info->dev_mem_size)); |
| 666 | XDNA_DBG(xdna, "Invalid dev heap size 0x%llx, limit 0x%lx", | 798 | XDNA_DBG(xdna, "Requested dev heap size 0x%llx", args->size); |
| 799 | if (!args->size || !IS_ALIGNED(args->size, xdna->dev_info->dev_mem_size)) { | ||
| 800 | XDNA_ERR(xdna, "The dev heap size 0x%llx is not multiple of 0x%lx", | ||
| 667 | args->size, xdna->dev_info->dev_mem_size); | 801 | args->size, xdna->dev_info->dev_mem_size); |
| 668 | return ERR_PTR(-EINVAL); | 802 | return ERR_PTR(-EINVAL); |
| 669 | } | 803 | } |
| 670 | 804 | ||
| 805 | /* HEAP BO is a special case of SHARE BO. */ | ||
| 806 | abo = amdxdna_drm_create_share_bo(dev, args, filp); | ||
| 807 | if (IS_ERR(abo)) | ||
| 808 | return ERR_CAST(abo); | ||
| 809 | |||
| 810 | /* Set up heap for this client. */ | ||
| 671 | mutex_lock(&client->mm_lock); | 811 | mutex_lock(&client->mm_lock); |
| 812 | |||
| 672 | if (client->dev_heap) { | 813 | if (client->dev_heap) { |
| 673 | XDNA_DBG(client->xdna, "dev heap is already created"); | 814 | XDNA_DBG(client->xdna, "dev heap is already created"); |
| 674 | ret = -EBUSY; | 815 | ret = -EBUSY; |
| 675 | goto mm_unlock; | 816 | goto mm_unlock; |
| 676 | } | 817 | } |
| 677 | |||
| 678 | abo = amdxdna_gem_create_object(dev, args); | ||
| 679 | if (IS_ERR(abo)) { | ||
| 680 | ret = PTR_ERR(abo); | ||
| 681 | goto mm_unlock; | ||
| 682 | } | ||
| 683 | |||
| 684 | abo->type = AMDXDNA_BO_DEV_HEAP; | ||
| 685 | abo->client = client; | ||
| 686 | abo->mem.dev_addr = client->xdna->dev_info->dev_mem_base; | ||
| 687 | drm_mm_init(&abo->mm, abo->mem.dev_addr, abo->mem.size); | ||
| 688 | |||
| 689 | ret = amdxdna_gem_obj_vmap(abo, &abo->mem.kva); | ||
| 690 | if (ret) { | ||
| 691 | XDNA_ERR(xdna, "Vmap heap bo failed, ret %d", ret); | ||
| 692 | goto release_obj; | ||
| 693 | } | ||
| 694 | |||
| 695 | client->dev_heap = abo; | 818 | client->dev_heap = abo; |
| 696 | drm_gem_object_get(to_gobj(abo)); | 819 | drm_gem_object_get(to_gobj(abo)); |
| 820 | |||
| 821 | drm_mm_init(&abo->mm, xdna->dev_info->dev_mem_base, abo->mem.size); | ||
| 822 | |||
| 697 | mutex_unlock(&client->mm_lock); | 823 | mutex_unlock(&client->mm_lock); |
| 698 | 824 | ||
| 699 | return abo; | 825 | return abo; |
| 700 | 826 | ||
| 701 | release_obj: | ||
| 702 | drm_gem_object_put(to_gobj(abo)); | ||
| 703 | mm_unlock: | 827 | mm_unlock: |
| 704 | mutex_unlock(&client->mm_lock); | 828 | mutex_unlock(&client->mm_lock); |
| 829 | drm_gem_object_put(to_gobj(abo)); | ||
| 705 | return ERR_PTR(ret); | 830 | return ERR_PTR(ret); |
| 706 | } | 831 | } |
| 707 | 832 | ||
| 708 | struct amdxdna_gem_obj * | 833 | struct amdxdna_gem_obj * |
| 709 | amdxdna_drm_alloc_dev_bo(struct drm_device *dev, | 834 | amdxdna_drm_create_dev_bo(struct drm_device *dev, |
| 710 | struct amdxdna_drm_create_bo *args, | 835 | struct amdxdna_drm_create_bo *args, struct drm_file *filp) |
| 711 | struct drm_file *filp) | ||
| 712 | { | 836 | { |
| 837 | size_t aligned_sz = PAGE_ALIGN(args->size); | ||
| 713 | struct amdxdna_client *client = filp->driver_priv; | 838 | struct amdxdna_client *client = filp->driver_priv; |
| 714 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | 839 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 715 | size_t aligned_sz = PAGE_ALIGN(args->size); | ||
| 716 | struct amdxdna_gem_obj *abo; | 840 | struct amdxdna_gem_obj *abo; |
| 841 | struct drm_gem_object *gobj; | ||
| 717 | int ret; | 842 | int ret; |
| 718 | 843 | ||
| 719 | abo = amdxdna_gem_create_obj(&xdna->ddev, aligned_sz); | 844 | if (!aligned_sz) { |
| 845 | XDNA_ERR(xdna, "Invalid BO size 0x%llx", args->size); | ||
| 846 | return ERR_PTR(-EINVAL); | ||
| 847 | } | ||
| 848 | |||
| 849 | abo = amdxdna_gem_create_obj(dev, aligned_sz); | ||
| 720 | if (IS_ERR(abo)) | 850 | if (IS_ERR(abo)) |
| 721 | return abo; | 851 | return abo; |
| 722 | 852 | gobj = to_gobj(abo); | |
| 723 | to_gobj(abo)->funcs = &amdxdna_gem_dev_obj_funcs; | 853 | gobj->funcs = &amdxdna_gem_dev_obj_funcs; |
| 724 | abo->type = AMDXDNA_BO_DEV; | 854 | abo->type = AMDXDNA_BO_DEV; |
| 855 | abo->internal = true; | ||
| 856 | /* | ||
| 857 | * DEV BOs cannot be alive when client is gone, it's OK to | ||
| 858 | * always establish the connection. | ||
| 859 | */ | ||
| 725 | abo->client = client; | 860 | abo->client = client; |
| 726 | 861 | ||
| 727 | ret = amdxdna_gem_heap_alloc(abo); | 862 | ret = amdxdna_gem_heap_alloc(abo); |
| @@ -730,31 +865,7 @@ amdxdna_drm_alloc_dev_bo(struct drm_device *dev, · 2 unmodified lines | |||
| 730 | amdxdna_gem_destroy_obj(abo); | 865 | amdxdna_gem_destroy_obj(abo); |
| 731 | return ERR_PTR(ret); | 866 | return ERR_PTR(ret); |
| 732 | } | 867 | } |
| 733 | 868 | drm_gem_private_object_init(dev, gobj, aligned_sz); | |
| 734 | drm_gem_private_object_init(&xdna->ddev, to_gobj(abo), aligned_sz); | ||
| 735 | |||
| 736 | return abo; | ||
| 737 | } | ||
| 738 | |||
| 739 | static struct amdxdna_gem_obj * | ||
| 740 | amdxdna_drm_create_cmd_bo(struct drm_device *dev, | ||
| 741 | struct amdxdna_drm_create_bo *args, | ||
| 742 | struct drm_file *filp) | ||
| 743 | { | ||
| 744 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | ||
| 745 | struct amdxdna_gem_obj *abo; | ||
| 746 | |||
| 747 | if (args->size < sizeof(struct amdxdna_cmd)) { | ||
| 748 | XDNA_DBG(xdna, "Command BO size 0x%llx too small", args->size); | ||
| 749 | return ERR_PTR(-EINVAL); | ||
| 750 | } | ||
| 751 | |||
| 752 | abo = amdxdna_gem_create_object(dev, args); | ||
| 753 | if (IS_ERR(abo)) | ||
| 754 | return ERR_CAST(abo); | ||
| 755 | |||
| 756 | abo->type = AMDXDNA_BO_CMD; | ||
| 757 | abo->client = filp->driver_priv; | ||
| 758 | 869 | ||
| 759 | return abo; | 870 | return abo; |
| 760 | } | 871 | } |
| @@ -772,17 +883,16 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f · 11 unmodified lines | |||
| 772 | XDNA_DBG(xdna, "BO arg type %d vaddr 0x%llx size 0x%llx flags 0x%llx", | 883 | XDNA_DBG(xdna, "BO arg type %d vaddr 0x%llx size 0x%llx flags 0x%llx", |
| 773 | args->type, args->vaddr, args->size, args->flags); | 884 | args->type, args->vaddr, args->size, args->flags); |
| 774 | switch (args->type) { | 885 | switch (args->type) { |
| 775 | case AMDXDNA_BO_SHMEM: | 886 | case AMDXDNA_BO_CMD: |
| 776 | abo = amdxdna_drm_alloc_shmem(dev, args, filp); | 887 | fallthrough; |
| 888 | case AMDXDNA_BO_SHARE: | ||
| 889 | abo = amdxdna_drm_create_share_bo(dev, args, filp); | ||
| 777 | break; | 890 | break; |
| 778 | case AMDXDNA_BO_DEV_HEAP: | 891 | case AMDXDNA_BO_DEV_HEAP: |
| 779 | abo = amdxdna_drm_create_dev_heap(dev, args, filp); | 892 | abo = amdxdna_drm_create_dev_heap_bo(dev, args, filp); |
| 780 | break; | 893 | break; |
| 781 | case AMDXDNA_BO_DEV: | 894 | case AMDXDNA_BO_DEV: |
| 782 | abo = amdxdna_drm_alloc_dev_bo(dev, args, filp); | 895 | abo = amdxdna_drm_create_dev_bo(dev, args, filp); |
| 783 | break; | ||
| 784 | case AMDXDNA_BO_CMD: | ||
| 785 | abo = amdxdna_drm_create_cmd_bo(dev, args, filp); | ||
| 786 | break; | 896 | break; |
| 787 | default: | 897 | default: |
| 788 | return -EINVAL; | 898 | return -EINVAL; |
| @@ -790,7 +900,7 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f · 1 unmodified line | |||
| 790 | if (IS_ERR(abo)) | 900 | if (IS_ERR(abo)) |
| 791 | return PTR_ERR(abo); | 901 | return PTR_ERR(abo); |
| 792 | 902 | ||
| 793 | /* ready to publish object to userspace */ | 903 | /* Ready to publish object to userspace and count for BO usage. */ |
| 794 | ret = drm_gem_handle_create(filp, to_gobj(abo), &args->handle); | 904 | ret = drm_gem_handle_create(filp, to_gobj(abo), &args->handle); |
| 795 | if (ret) { | 905 | if (ret) { |
| 796 | XDNA_ERR(xdna, "Create handle failed"); | 906 | XDNA_ERR(xdna, "Create handle failed"); |
| @@ -798,8 +908,8 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f · 1 unmodified line | |||
| 798 | } | 908 | } |
| 799 | 909 | ||
| 800 | XDNA_DBG(xdna, "BO hdl %d type %d userptr 0x%llx xdna_addr 0x%llx size 0x%lx", | 910 | XDNA_DBG(xdna, "BO hdl %d type %d userptr 0x%llx xdna_addr 0x%llx size 0x%lx", |
| 801 | args->handle, args->type, abo->mem.userptr, | 911 | args->handle, args->type, amdxdna_gem_uva(abo), |
| 802 | abo->mem.dev_addr, abo->mem.size); | 912 | amdxdna_gem_dev_addr(abo), abo->mem.size); |
| 803 | put_obj: | 913 | put_obj: |
| 804 | /* Dereference object reference. Handle holds it now. */ | 914 | /* Dereference object reference. Handle holds it now. */ |
| 805 | drm_gem_object_put(to_gobj(abo)); | 915 | drm_gem_object_put(to_gobj(abo)); |
| @@ -850,38 +960,19 @@ void amdxdna_gem_unpin(struct amdxdna_gem_obj *abo) · 44 unmodified lines | |||
| 850 | struct amdxdna_gem_obj *amdxdna_gem_get_obj(struct amdxdna_client *client, | 960 | struct amdxdna_gem_obj *amdxdna_gem_get_obj(struct amdxdna_client *client, |
| 851 | u32 bo_hdl, u8 bo_type) | 961 | u32 bo_hdl, u8 bo_type) |
| 852 | { | 962 | { |
| 853 | struct amdxdna_dev *xdna = client->xdna; | ||
| 854 | struct amdxdna_gem_obj *abo; | 963 | struct amdxdna_gem_obj *abo; |
| 855 | struct drm_gem_object *gobj; | 964 | struct drm_gem_object *gobj; |
| 856 | int ret; | ||
| 857 | 965 | ||
| 858 | gobj = drm_gem_object_lookup(client->filp, bo_hdl); | 966 | gobj = drm_gem_object_lookup(client->filp, bo_hdl); |
| 859 | if (!gobj) { | 967 | if (!gobj) { |
| 860 | XDNA_DBG(xdna, "Can not find bo %d", bo_hdl); | 968 | XDNA_DBG(client->xdna, "Can not find bo %d", bo_hdl); |
| 861 | return NULL; | 969 | return NULL; |
| 862 | } | 970 | } |
| 863 | 971 | ||
| 864 | abo = to_xdna_obj(gobj); | 972 | abo = to_xdna_obj(gobj); |
| 865 | if (bo_type != AMDXDNA_BO_INVALID && abo->type != bo_type) | 973 | if (bo_type == AMDXDNA_BO_INVALID || abo->type == bo_type) |
| 866 | goto put_obj; | ||
| 867 | |||
| 868 | if (bo_type != AMDXDNA_BO_CMD || abo->mem.kva) | ||
| 869 | return abo; | 974 | return abo; |
| 870 | 975 | ||
| 871 | if (abo->mem.size > SZ_32K) { | ||
| 872 | XDNA_ERR(xdna, "Cmd bo is too big %ld", abo->mem.size); | ||
| 873 | goto put_obj; | ||
| 874 | } | ||
| 875 | |||
| 876 | ret = amdxdna_gem_obj_vmap(abo, &abo->mem.kva); | ||
| 877 | if (ret) { | ||
| 878 | XDNA_ERR(xdna, "Vmap cmd bo failed, ret %d", ret); | ||
| 879 | goto put_obj; | ||
| 880 | } | ||
| 881 | |||
| 882 | return abo; | ||
| 883 | |||
| 884 | put_obj: | ||
| 885 | drm_gem_object_put(gobj); | 976 | drm_gem_object_put(gobj); |
| 886 | return NULL; | 977 | return NULL; |
| 887 | } | 978 | } |
| @@ -904,8 +995,8 @@ int amdxdna_drm_get_bo_info_ioctl(struct drm_device *dev, void *data, struct drm · 16 unmodified lines | |||
| 904 | } | 995 | } |
| 905 | 996 | ||
| 906 | abo = to_xdna_obj(gobj); | 997 | abo = to_xdna_obj(gobj); |
| 907 | args->vaddr = abo->mem.userptr; | 998 | args->vaddr = amdxdna_gem_uva(abo); |
| 908 | args->xdna_addr = abo->mem.dev_addr; | 999 | args->xdna_addr = amdxdna_gem_dev_addr(abo); |
| 909 | 1000 | ||
| 910 | if (abo->type != AMDXDNA_BO_DEV) | 1001 | if (abo->type != AMDXDNA_BO_DEV) |
| 911 | args->map_offset = drm_vma_node_offset_addr(&gobj->vma_node); | 1002 | args->map_offset = drm_vma_node_offset_addr(&gobj->vma_node); |
| @@ -950,8 +1041,8 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, · 38 unmodified lines | |||
| 950 | 1041 | ||
| 951 | if (is_import_bo(abo)) | 1042 | if (is_import_bo(abo)) |
| 952 | drm_clflush_sg(abo->base.sgt); | 1043 | drm_clflush_sg(abo->base.sgt); |
| 953 | else if (abo->mem.kva) | 1044 | else if (amdxdna_gem_vmap(abo)) |
| 954 | drm_clflush_virt_range(abo->mem.kva + args->offset, args->size); | 1045 | drm_clflush_virt_range(amdxdna_gem_vmap(abo) + args->offset, args->size); |
| 955 | else if (abo->base.pages) | 1046 | else if (abo->base.pages) |
| 956 | drm_clflush_pages(abo->base.pages, gobj->size >> PAGE_SHIFT); | 1047 | drm_clflush_pages(abo->base.pages, gobj->size >> PAGE_SHIFT); |
| 957 | else | 1048 | else |
| @@ -969,3 +1060,43 @@ put_obj: · 11 unmodified lines | |||
| 969 | drm_gem_object_put(gobj); | 1060 | drm_gem_object_put(gobj); |
| 970 | return ret; | 1061 | return ret; |
| 971 | } | 1062 | } |
| 1063 | |||
| 1064 | int amdxdna_drm_get_bo_usage(struct drm_device *dev, struct amdxdna_drm_get_array *args) | ||
| 1065 | { | ||
| 1066 | size_t min_sz = min(args->element_size, sizeof(struct amdxdna_drm_bo_usage)); | ||
| 1067 | char __user *buf = u64_to_user_ptr(args->buffer); | ||
| 1068 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | ||
| 1069 | struct amdxdna_client *tmp_client; | ||
| 1070 | struct amdxdna_drm_bo_usage tmp; | ||
| 1071 | |||
| 1072 | drm_WARN_ON(dev, !mutex_is_locked(&xdna->dev_lock)); | ||
| 1073 | |||
| 1074 | if (args->num_element != 1) | ||
| 1075 | return -EINVAL; | ||
| 1076 | |||
| 1077 | if (copy_from_user(&tmp, buf, min_sz)) | ||
| 1078 | return -EFAULT; | ||
| 1079 | |||
| 1080 | if (!tmp.pid) | ||
| 1081 | return -EINVAL; | ||
| 1082 | |||
| 1083 | tmp.total_usage = 0; | ||
| 1084 | tmp.internal_usage = 0; | ||
| 1085 | tmp.heap_usage = 0; | ||
| 1086 | |||
| 1087 | list_for_each_entry(tmp_client, &xdna->client_list, node) { | ||
| 1088 | if (tmp.pid != tmp_client->pid) | ||
| 1089 | continue; | ||
| 1090 | |||
| 1091 | mutex_lock(&tmp_client->mm_lock); | ||
| 1092 | tmp.total_usage += tmp_client->total_bo_usage; | ||
| 1093 | tmp.internal_usage += tmp_client->total_int_bo_usage; | ||
| 1094 | tmp.heap_usage += tmp_client->heap_usage; | ||
| 1095 | mutex_unlock(&tmp_client->mm_lock); | ||
| 1096 | } | ||
| 1097 | |||
| 1098 | if (copy_to_user(buf, &tmp, min_sz)) | ||
| 1099 | return -EFAULT; | ||
| 1100 | |||
| 1101 | return 0; | ||
| 1102 | } | ||
| @@ -6,7 +6,9 @@ · 5 unmodified lines | |||
| 6 | #ifndef _AMDXDNA_GEM_H_ | 6 | #ifndef _AMDXDNA_GEM_H_ |
| 7 | #define _AMDXDNA_GEM_H_ | 7 | #define _AMDXDNA_GEM_H_ |
| 8 | 8 | ||
| 9 | #include <drm/drm_gem_shmem_helper.h> | ||
| 9 | #include <linux/hmm.h> | 10 | #include <linux/hmm.h> |
| 11 | #include <linux/iova.h> | ||
| 10 | #include "amdxdna_pci_drv.h" | 12 | #include "amdxdna_pci_drv.h" |
| 11 | 13 | ||
| 12 | struct amdxdna_umap { | 14 | struct amdxdna_umap { |
| @@ -22,14 +24,16 @@ struct amdxdna_umap { · 9 unmodified lines | |||
| 22 | }; | 24 | }; |
| 23 | 25 | ||
| 24 | struct amdxdna_mem { | 26 | struct amdxdna_mem { |
| 25 | u64 userptr; | ||
| 26 | void *kva; | 27 | void *kva; |
| 27 | u64 dev_addr; | 28 | u64 dma_addr; |
| 28 | size_t size; | 29 | size_t size; |
| 29 | struct page **pages; | ||
| 30 | u32 nr_pages; | ||
| 31 | struct list_head umap_list; | 30 | struct list_head umap_list; |
| 32 | bool map_invalid; | 31 | bool map_invalid; |
| 32 | /* | ||
| 33 | * Cache the first mmap uva as PASID addr, which can be accessed by driver | ||
| 34 | * without taking notifier_lock. | ||
| 35 | */ | ||
| 36 | u64 uva; | ||
| 33 | }; | 37 | }; |
| 34 | 38 | ||
| 35 | struct amdxdna_gem_obj { | 39 | struct amdxdna_gem_obj { |
| @@ -37,15 +41,19 @@ struct amdxdna_gem_obj { · 1 unmodified line | |||
| 37 | struct amdxdna_client *client; | 41 | struct amdxdna_client *client; |
| 38 | u8 type; | 42 | u8 type; |
| 39 | bool pinned; | 43 | bool pinned; |
| 40 | struct mutex lock; /* Protects: pinned */ | 44 | struct mutex lock; /* Protects: pinned, mem.kva, open_ref */ |
| 41 | struct amdxdna_mem mem; | 45 | struct amdxdna_mem mem; |
| 46 | int open_ref; | ||
| 42 | 47 | ||
| 43 | /* Below members is uninitialized when needed */ | 48 | /* Below members are initialized when needed */ |
| 44 | struct drm_mm mm; /* For AMDXDNA_BO_DEV_HEAP */ | 49 | struct drm_mm mm; /* For AMDXDNA_BO_DEV_HEAP */ |
| 45 | struct drm_mm_node mm_node; /* For AMDXDNA_BO_DEV */ | 50 | struct drm_mm_node mm_node; /* For AMDXDNA_BO_DEV */ |
| 46 | u32 assigned_hwctx; | 51 | u32 assigned_hwctx; |
| 47 | struct dma_buf *dma_buf; | 52 | struct dma_buf *dma_buf; |
| 48 | struct dma_buf_attachment *attach; | 53 | struct dma_buf_attachment *attach; |
| 54 | |||
| 55 | /* True, if BO is managed by XRT, not application */ | ||
| 56 | bool internal; | ||
| 49 | }; | 57 | }; |
| 50 | 58 | ||
| 51 | #define to_gobj(obj) (&(obj)->base.base) | 59 | #define to_gobj(obj) (&(obj)->base.base) |
| @@ -63,21 +71,29 @@ static inline void amdxdna_gem_put_obj(struct amdxdna_gem_obj *abo) · 11 unmodified lines | |||
| 63 | drm_gem_object_put(to_gobj(abo)); | 71 | drm_gem_object_put(to_gobj(abo)); |
| 64 | } | 72 | } |
| 65 | 73 | ||
| 74 | void *amdxdna_gem_vmap(struct amdxdna_gem_obj *abo); | ||
| 75 | u64 amdxdna_gem_uva(struct amdxdna_gem_obj *abo); | ||
| 76 | u64 amdxdna_gem_dev_addr(struct amdxdna_gem_obj *abo); | ||
| 77 | |||
| 66 | static inline u64 amdxdna_dev_bo_offset(struct amdxdna_gem_obj *abo) | 78 | static inline u64 amdxdna_dev_bo_offset(struct amdxdna_gem_obj *abo) |
| 67 | { | 79 | { |
| 68 | return abo->mem.dev_addr - abo->client->dev_heap->mem.dev_addr; | 80 | return amdxdna_gem_dev_addr(abo) - amdxdna_gem_dev_addr(abo->client->dev_heap); |
| 81 | } | ||
| 82 | |||
| 83 | static inline u64 amdxdna_obj_dma_addr(struct amdxdna_gem_obj *abo) | ||
| 84 | { | ||
| 85 | return amdxdna_pasid_on(abo->client) ? amdxdna_gem_uva(abo) : abo->mem.dma_addr; | ||
| 69 | } | 86 | } |
| 70 | 87 | ||
| 71 | void amdxdna_umap_put(struct amdxdna_umap *mapp); | 88 | void amdxdna_umap_put(struct amdxdna_umap *mapp); |
| 72 | 89 | ||
| 73 | struct drm_gem_object * | 90 | struct drm_gem_object * |
| 74 | amdxdna_gem_create_object_cb(struct drm_device *dev, size_t size); | 91 | amdxdna_gem_create_shmem_object_cb(struct drm_device *dev, size_t size); |
| 75 | struct drm_gem_object * | 92 | struct drm_gem_object * |
| 76 | amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf); | 93 | amdxdna_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf); |
| 77 | struct amdxdna_gem_obj * | 94 | struct amdxdna_gem_obj * |
| 78 | amdxdna_drm_alloc_dev_bo(struct drm_device *dev, | 95 | amdxdna_drm_create_dev_bo(struct drm_device *dev, |
| 79 | struct amdxdna_drm_create_bo *args, | 96 | struct amdxdna_drm_create_bo *args, struct drm_file *filp); |
| 80 | struct drm_file *filp); | ||
| 81 | 97 | ||
| 82 | int amdxdna_gem_pin_nolock(struct amdxdna_gem_obj *abo); | 98 | int amdxdna_gem_pin_nolock(struct amdxdna_gem_obj *abo); |
| 83 | int amdxdna_gem_pin(struct amdxdna_gem_obj *abo); | 99 | int amdxdna_gem_pin(struct amdxdna_gem_obj *abo); |
| @@ -86,5 +102,6 @@ void amdxdna_gem_unpin(struct amdxdna_gem_obj *abo); · 2 unmodified lines | |||
| 86 | int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); | 102 | int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); |
| 87 | int amdxdna_drm_get_bo_info_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); | 103 | int amdxdna_drm_get_bo_info_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); |
| 88 | int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); | 104 | int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); |
| 105 | int amdxdna_drm_get_bo_usage(struct drm_device *dev, struct amdxdna_drm_get_array *args); | ||
| 89 | 106 | ||
| 90 | #endif /* _AMDXDNA_GEM_H_ */ | 107 | #endif /* _AMDXDNA_GEM_H_ */ |
| @@ -0,0 +1,184 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* | ||
| 3 | * Copyright (C) 2025, Advanced Micro Devices, Inc. | ||
| 4 | */ | ||
| 5 | |||
| 6 | #include <drm/amdxdna_accel.h> | ||
| 7 | #include <linux/iommu.h> | ||
| 8 | #include <linux/iova.h> | ||
| 9 | |||
| 10 | #include "amdxdna_gem.h" | ||
| 11 | #include "amdxdna_pci_drv.h" | ||
| 12 | |||
| 13 | static bool force_iova; | ||
| 14 | module_param(force_iova, bool, 0600); | ||
| 15 | MODULE_PARM_DESC(force_iova, "Force use IOVA (Default false)"); | ||
| 16 | |||
| 17 | static struct iova *amdxdna_iommu_alloc_iova(struct amdxdna_dev *xdna, | ||
| 18 | size_t size, | ||
| 19 | dma_addr_t *dma_addr, | ||
| 20 | bool size_aligned) | ||
| 21 | { | ||
| 22 | unsigned long shift, end; | ||
| 23 | struct iova *iova; | ||
| 24 | |||
| 25 | end = xdna->domain->geometry.aperture_end; | ||
| 26 | shift = iova_shift(&xdna->iovad); | ||
| 27 | size = iova_align(&xdna->iovad, size); | ||
| 28 | |||
| 29 | iova = alloc_iova(&xdna->iovad, size >> shift, end >> shift, size_aligned); | ||
| 30 | if (!iova) | ||
| 31 | return ERR_PTR(-ENOMEM); | ||
| 32 | |||
| 33 | *dma_addr = iova_dma_addr(&xdna->iovad, iova); | ||
| 34 | |||
| 35 | return iova; | ||
| 36 | } | ||
| 37 | |||
| 38 | int amdxdna_iommu_map_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo) | ||
| 39 | { | ||
| 40 | struct sg_table *sgt; | ||
| 41 | dma_addr_t dma_addr; | ||
| 42 | struct iova *iova; | ||
| 43 | size_t size; | ||
| 44 | |||
| 45 | if (abo->type != AMDXDNA_BO_DEV_HEAP && abo->type != AMDXDNA_BO_SHMEM) | ||
| 46 | return 0; | ||
| 47 | |||
| 48 | sgt = drm_gem_shmem_get_pages_sgt(&abo->base); | ||
| 49 | if (IS_ERR(sgt)) { | ||
| 50 | XDNA_ERR(xdna, "Get sgt failed, ret %ld", PTR_ERR(sgt)); | ||
| 51 | return PTR_ERR(sgt); | ||
| 52 | } | ||
| 53 | |||
| 54 | if (!sgt->orig_nents || !sg_page(sgt->sgl)) { | ||
| 55 | XDNA_ERR(xdna, "sgl is zero length or not page backed"); | ||
| 56 | return -EOPNOTSUPP; | ||
| 57 | } | ||
| 58 | |||
| 59 | iova = amdxdna_iommu_alloc_iova(xdna, abo->mem.size, &dma_addr, | ||
| 60 | (abo->type == AMDXDNA_BO_DEV_HEAP)); | ||
| 61 | if (IS_ERR(iova)) { | ||
| 62 | XDNA_ERR(xdna, "Alloc iova failed, ret %ld", PTR_ERR(iova)); | ||
| 63 | return PTR_ERR(iova); | ||
| 64 | } | ||
| 65 | |||
| 66 | size = iommu_map_sgtable(xdna->domain, dma_addr, sgt, | ||
| 67 | IOMMU_READ | IOMMU_WRITE); | ||
| 68 | if (size < abo->mem.size) { | ||
| 69 | __free_iova(&xdna->iovad, iova); | ||
| 70 | return -ENXIO; | ||
| 71 | } | ||
| 72 | |||
| 73 | abo->mem.dma_addr = dma_addr; | ||
| 74 | |||
| 75 | return 0; | ||
| 76 | } | ||
| 77 | |||
| 78 | void amdxdna_iommu_unmap_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo) | ||
| 79 | { | ||
| 80 | size_t size; | ||
| 81 | |||
| 82 | if (abo->mem.dma_addr == AMDXDNA_INVALID_ADDR) | ||
| 83 | return; | ||
| 84 | |||
| 85 | size = iova_align(&xdna->iovad, abo->mem.size); | ||
| 86 | iommu_unmap(xdna->domain, abo->mem.dma_addr, size); | ||
| 87 | free_iova(&xdna->iovad, iova_pfn(&xdna->iovad, abo->mem.dma_addr)); | ||
| 88 | abo->mem.dma_addr = AMDXDNA_INVALID_ADDR; | ||
| 89 | } | ||
| 90 | |||
| 91 | void *amdxdna_iommu_alloc(struct amdxdna_dev *xdna, size_t size, dma_addr_t *dma_addr) | ||
| 92 | { | ||
| 93 | struct iova *iova; | ||
| 94 | void *cpu_addr; | ||
| 95 | int ret; | ||
| 96 | |||
| 97 | iova = amdxdna_iommu_alloc_iova(xdna, size, dma_addr, true); | ||
| 98 | if (IS_ERR(iova)) { | ||
| 99 | XDNA_ERR(xdna, "Alloc iova failed, ret %ld", PTR_ERR(iova)); | ||
| 100 | return iova; | ||
| 101 | } | ||
| 102 | |||
| 103 | cpu_addr = (void *)__get_free_pages(GFP_KERNEL, get_order(size)); | ||
| 104 | if (!cpu_addr) { | ||
| 105 | ret = -ENOMEM; | ||
| 106 | goto free_iova; | ||
| 107 | } | ||
| 108 | |||
| 109 | ret = iommu_map(xdna->domain, *dma_addr, virt_to_phys(cpu_addr), | ||
| 110 | iova_align(&xdna->iovad, size), | ||
| 111 | IOMMU_READ | IOMMU_WRITE, GFP_KERNEL); | ||
| 112 | if (ret) | ||
| 113 | goto free_iova; | ||
| 114 | |||
| 115 | return cpu_addr; | ||
| 116 | |||
| 117 | free_iova: | ||
| 118 | __free_iova(&xdna->iovad, iova); | ||
| 119 | return ERR_PTR(ret); | ||
| 120 | } | ||
| 121 | |||
| 122 | void amdxdna_iommu_free(struct amdxdna_dev *xdna, size_t size, | ||
| 123 | void *cpu_addr, dma_addr_t dma_addr) | ||
| 124 | { | ||
| 125 | iommu_unmap(xdna->domain, dma_addr, iova_align(&xdna->iovad, size)); | ||
| 126 | free_iova(&xdna->iovad, iova_pfn(&xdna->iovad, dma_addr)); | ||
| 127 | free_pages((unsigned long)cpu_addr, get_order(size)); | ||
| 128 | } | ||
| 129 | |||
| 130 | int amdxdna_iommu_init(struct amdxdna_dev *xdna) | ||
| 131 | { | ||
| 132 | unsigned long order; | ||
| 133 | int ret; | ||
| 134 | |||
| 135 | xdna->group = iommu_group_get(xdna->ddev.dev); | ||
| 136 | if (!xdna->group || !force_iova) | ||
| 137 | return 0; | ||
| 138 | |||
| 139 | XDNA_WARN(xdna, "Enabled force_iova mode."); | ||
| 140 | xdna->domain = iommu_paging_domain_alloc_flags(xdna->ddev.dev, | ||
| 141 | IOMMU_HWPT_ALLOC_PASID); | ||
| 142 | if (IS_ERR(xdna->domain)) { | ||
| 143 | XDNA_ERR(xdna, "Failed to alloc iommu domain"); | ||
| 144 | ret = PTR_ERR(xdna->domain); | ||
| 145 | goto put_group; | ||
| 146 | } | ||
| 147 | |||
| 148 | ret = iova_cache_get(); | ||
| 149 | if (ret) | ||
| 150 | goto free_domain; | ||
| 151 | |||
| 152 | order = __ffs(xdna->domain->pgsize_bitmap); | ||
| 153 | init_iova_domain(&xdna->iovad, 1UL << order, 0); | ||
| 154 | |||
| 155 | ret = iommu_attach_group(xdna->domain, xdna->group); | ||
| 156 | if (ret) | ||
| 157 | goto put_iova; | ||
| 158 | |||
| 159 | return 0; | ||
| 160 | |||
| 161 | put_iova: | ||
| 162 | put_iova_domain(&xdna->iovad); | ||
| 163 | iova_cache_put(); | ||
| 164 | free_domain: | ||
| 165 | iommu_domain_free(xdna->domain); | ||
| 166 | put_group: | ||
| 167 | iommu_group_put(xdna->group); | ||
| 168 | xdna->domain = NULL; | ||
| 169 | |||
| 170 | return ret; | ||
| 171 | } | ||
| 172 | |||
| 173 | void amdxdna_iommu_fini(struct amdxdna_dev *xdna) | ||
| 174 | { | ||
| 175 | if (xdna->domain) { | ||
| 176 | iommu_detach_group(xdna->domain, xdna->group); | ||
| 177 | put_iova_domain(&xdna->iovad); | ||
| 178 | iova_cache_put(); | ||
| 179 | iommu_domain_free(xdna->domain); | ||
| 180 | } | ||
| 181 | |||
| 182 | if (xdna->group) | ||
| 183 | iommu_group_put(xdna->group); | ||
| 184 | } | ||
| @@ -499,7 +499,7 @@ xdna_mailbox_start_channel(struct mailbox_channel *mb_chann, · 498 unmodified lines | |||
| 499 | int ret; | 499 | int ret; |
| 500 | 500 | ||
| 501 | if (!is_power_of_2(x2i->rb_size) || !is_power_of_2(i2x->rb_size)) { | 501 | if (!is_power_of_2(x2i->rb_size) || !is_power_of_2(i2x->rb_size)) { |
| 502 | pr_err("Ring buf size must be power of 2"); | 502 | pr_err("Ring buf size must be power of 2\n"); |
| 503 | return -EINVAL; | 503 | return -EINVAL; |
| 504 | } | 504 | } |
| 505 | 505 | ||
| @@ -36,9 +36,10 @@ MODULE_FIRMWARE("amdnpu/17f0_11/npu_7.sbin"); · 35 unmodified lines | |||
| 36 | * 0.5: Support getting telemetry data | 36 | * 0.5: Support getting telemetry data |
| 37 | * 0.6: Support preemption | 37 | * 0.6: Support preemption |
| 38 | * 0.7: Support getting power and utilization data | 38 | * 0.7: Support getting power and utilization data |
| 39 | * 0.8: Support BO usage query | ||
| 39 | */ | 40 | */ |
| 40 | #define AMDXDNA_DRIVER_MAJOR 0 | 41 | #define AMDXDNA_DRIVER_MAJOR 0 |
| 41 | #define AMDXDNA_DRIVER_MINOR 7 | 42 | #define AMDXDNA_DRIVER_MINOR 8 |
| 42 | 43 | ||
| 43 | /* | 44 | /* |
| 44 | * Bind the driver base on (vendor_id, device_id) pair and later use the | 45 | * Bind the driver base on (vendor_id, device_id) pair and later use the |
| @@ -73,18 +74,21 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp) · 28 unmodified lines | |||
| 73 | 74 | ||
| 74 | client->pid = pid_nr(rcu_access_pointer(filp->pid)); | 75 | client->pid = pid_nr(rcu_access_pointer(filp->pid)); |
| 75 | client->xdna = xdna; | 76 | client->xdna = xdna; |
| 76 | 77 | client->pasid = IOMMU_PASID_INVALID; | |
| 77 | client->sva = iommu_sva_bind_device(xdna->ddev.dev, current->mm); | 78 | |
| 78 | if (IS_ERR(client->sva)) { | 79 | if (!amdxdna_iova_on(xdna)) { |
| 79 | ret = PTR_ERR(client->sva); | 80 | client->sva = iommu_sva_bind_device(xdna->ddev.dev, current->mm); |
| 80 | XDNA_ERR(xdna, "SVA bind device failed, ret %d", ret); | 81 | if (IS_ERR(client->sva)) { |
| 81 | goto failed; | 82 | ret = PTR_ERR(client->sva); |
| 82 | } | 83 | XDNA_ERR(xdna, "SVA bind device failed, ret %d", ret); |
| 83 | client->pasid = iommu_sva_get_pasid(client->sva); | 84 | goto failed; |
| 84 | if (client->pasid == IOMMU_PASID_INVALID) { | 85 | } |
| 85 | XDNA_ERR(xdna, "SVA get pasid failed"); | 86 | client->pasid = iommu_sva_get_pasid(client->sva); |
| 86 | ret = -ENODEV; | 87 | if (client->pasid == IOMMU_PASID_INVALID) { |
| 87 | goto unbind_sva; | 88 | XDNA_ERR(xdna, "SVA get pasid failed"); |
| 89 | ret = -ENODEV; | ||
| 90 | goto unbind_sva; | ||
| 91 | } | ||
| 88 | } | 92 | } |
| 89 | client->mm = current->mm; | 93 | client->mm = current->mm; |
| 90 | mmgrab(client->mm); | 94 | mmgrab(client->mm); |
| @@ -103,7 +107,8 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp) · 12 unmodified lines | |||
| 103 | return 0; | 107 | return 0; |
| 104 | 108 | ||
| 105 | unbind_sva: | 109 | unbind_sva: |
| 106 | iommu_sva_unbind_device(client->sva); | 110 | if (!IS_ERR_OR_NULL(client->sva)) |
| 111 | iommu_sva_unbind_device(client->sva); | ||
| 107 | failed: | 112 | failed: |
| 108 | kfree(client); | 113 | kfree(client); |
| 109 | 114 | ||
| @@ -116,12 +121,14 @@ static void amdxdna_client_cleanup(struct amdxdna_client *client) · 6 unmodified lines | |||
| 116 | amdxdna_hwctx_remove_all(client); | 121 | amdxdna_hwctx_remove_all(client); |
| 117 | xa_destroy(&client->hwctx_xa); | 122 | xa_destroy(&client->hwctx_xa); |
| 118 | cleanup_srcu_struct(&client->hwctx_srcu); | 123 | cleanup_srcu_struct(&client->hwctx_srcu); |
| 119 | mutex_destroy(&client->mm_lock); | ||
| 120 | 124 | ||
| 121 | if (client->dev_heap) | 125 | if (client->dev_heap) |
| 122 | drm_gem_object_put(to_gobj(client->dev_heap)); | 126 | drm_gem_object_put(to_gobj(client->dev_heap)); |
| 123 | 127 | ||
| 124 | iommu_sva_unbind_device(client->sva); | 128 | mutex_destroy(&client->mm_lock); |
| 129 | |||
| 130 | if (!IS_ERR_OR_NULL(client->sva)) | ||
| 131 | iommu_sva_unbind_device(client->sva); | ||
| 125 | mmdrop(client->mm); | 132 | mmdrop(client->mm); |
| 126 | 133 | ||
| 127 | kfree(client); | 134 | kfree(client); |
| @@ -240,7 +247,7 @@ const struct drm_driver amdxdna_drm_drv = { · 112 unmodified lines | |||
| 240 | .ioctls = amdxdna_drm_ioctls, | 247 | .ioctls = amdxdna_drm_ioctls, |
| 241 | .num_ioctls = ARRAY_SIZE(amdxdna_drm_ioctls), | 248 | .num_ioctls = ARRAY_SIZE(amdxdna_drm_ioctls), |
| 242 | 249 | ||
| 243 | .gem_create_object = amdxdna_gem_create_object_cb, | 250 | .gem_create_object = amdxdna_gem_create_shmem_object_cb, |
| 244 | .gem_prime_import = amdxdna_gem_prime_import, | 251 | .gem_prime_import = amdxdna_gem_prime_import, |
| 245 | }; | 252 | }; |
| 246 | 253 | ||
| @@ -282,9 +289,15 @@ static int amdxdna_probe(struct pci_dev *pdev, const struct pci_device_id *id) · 35 unmodified lines | |||
| 282 | fs_reclaim_release(GFP_KERNEL); | 289 | fs_reclaim_release(GFP_KERNEL); |
| 283 | } | 290 | } |
| 284 | 291 | ||
| 292 | ret = amdxdna_iommu_init(xdna); | ||
| 293 | if (ret) | ||
| 294 | return ret; | ||
| 295 | |||
| 285 | xdna->notifier_wq = alloc_ordered_workqueue("notifier_wq", WQ_MEM_RECLAIM); | 296 | xdna->notifier_wq = alloc_ordered_workqueue("notifier_wq", WQ_MEM_RECLAIM); |
| 286 | if (!xdna->notifier_wq) | 297 | if (!xdna->notifier_wq) { |
| 287 | return -ENOMEM; | 298 | ret = -ENOMEM; |
| 299 | goto iommu_fini; | ||
| 300 | } | ||
| 288 | 301 | ||
| 289 | mutex_lock(&xdna->dev_lock); | 302 | mutex_lock(&xdna->dev_lock); |
| 290 | ret = xdna->dev_info->ops->init(xdna); | 303 | ret = xdna->dev_info->ops->init(xdna); |
| @@ -316,6 +329,8 @@ failed_dev_fini: · 25 unmodified lines | |||
| 316 | mutex_unlock(&xdna->dev_lock); | 329 | mutex_unlock(&xdna->dev_lock); |
| 317 | destroy_notifier_wq: | 330 | destroy_notifier_wq: |
| 318 | destroy_workqueue(xdna->notifier_wq); | 331 | destroy_workqueue(xdna->notifier_wq); |
| 332 | iommu_fini: | ||
| 333 | amdxdna_iommu_fini(xdna); | ||
| 319 | return ret; | 334 | return ret; |
| 320 | } | 335 | } |
| 321 | 336 | ||
| @@ -341,6 +356,8 @@ static void amdxdna_remove(struct pci_dev *pdev) · 19 unmodified lines | |||
| 341 | 356 | ||
| 342 | xdna->dev_info->ops->fini(xdna); | 357 | xdna->dev_info->ops->fini(xdna); |
| 343 | mutex_unlock(&xdna->dev_lock); | 358 | mutex_unlock(&xdna->dev_lock); |
| 359 | |||
| 360 | amdxdna_iommu_fini(xdna); | ||
| 344 | } | 361 | } |
| 345 | 362 | ||
| 346 | static const struct dev_pm_ops amdxdna_pm_ops = { | 363 | static const struct dev_pm_ops amdxdna_pm_ops = { |
| @@ -6,7 +6,10 @@ · 5 unmodified lines | |||
| 6 | #ifndef _AMDXDNA_PCI_DRV_H_ | 6 | #ifndef _AMDXDNA_PCI_DRV_H_ |
| 7 | #define _AMDXDNA_PCI_DRV_H_ | 7 | #define _AMDXDNA_PCI_DRV_H_ |
| 8 | 8 | ||
| 9 | #include <drm/amdxdna_accel.h> | ||
| 9 | #include <drm/drm_print.h> | 10 | #include <drm/drm_print.h> |
| 11 | #include <linux/iommu.h> | ||
| 12 | #include <linux/iova.h> | ||
| 10 | #include <linux/workqueue.h> | 13 | #include <linux/workqueue.h> |
| 11 | #include <linux/xarray.h> | 14 | #include <linux/xarray.h> |
| 12 | 15 | ||
| @@ -101,6 +104,10 @@ struct amdxdna_dev { · 88 unmodified lines | |||
| 101 | struct amdxdna_fw_ver fw_ver; | 104 | struct amdxdna_fw_ver fw_ver; |
| 102 | struct rw_semaphore notifier_lock; /* for mmu notifier*/ | 105 | struct rw_semaphore notifier_lock; /* for mmu notifier*/ |
| 103 | struct workqueue_struct *notifier_wq; | 106 | struct workqueue_struct *notifier_wq; |
| 107 | |||
| 108 | struct iommu_group *group; | ||
| 109 | struct iommu_domain *domain; | ||
| 110 | struct iova_domain iovad; | ||
| 104 | }; | 111 | }; |
| 105 | 112 | ||
| 106 | /* | 113 | /* |
| @@ -131,6 +138,10 @@ struct amdxdna_client { · 24 unmodified lines | |||
| 131 | struct iommu_sva *sva; | 138 | struct iommu_sva *sva; |
| 132 | int pasid; | 139 | int pasid; |
| 133 | struct mm_struct *mm; | 140 | struct mm_struct *mm; |
| 141 | |||
| 142 | size_t heap_usage; | ||
| 143 | size_t total_bo_usage; | ||
| 144 | size_t total_int_bo_usage; | ||
| 134 | }; | 145 | }; |
| 135 | 146 | ||
| 136 | #define amdxdna_for_each_hwctx(client, hwctx_id, entry) \ | 147 | #define amdxdna_for_each_hwctx(client, hwctx_id, entry) \ |
| @@ -145,4 +156,21 @@ extern const struct amdxdna_dev_info dev_npu6_info; · 8 unmodified lines | |||
| 145 | int amdxdna_sysfs_init(struct amdxdna_dev *xdna); | 156 | int amdxdna_sysfs_init(struct amdxdna_dev *xdna); |
| 146 | void amdxdna_sysfs_fini(struct amdxdna_dev *xdna); | 157 | void amdxdna_sysfs_fini(struct amdxdna_dev *xdna); |
| 147 | 158 | ||
| 159 | int amdxdna_iommu_init(struct amdxdna_dev *xdna); | ||
| 160 | void amdxdna_iommu_fini(struct amdxdna_dev *xdna); | ||
| 161 | int amdxdna_iommu_map_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo); | ||
| 162 | void amdxdna_iommu_unmap_bo(struct amdxdna_dev *xdna, struct amdxdna_gem_obj *abo); | ||
| 163 | void *amdxdna_iommu_alloc(struct amdxdna_dev *xdna, size_t size, dma_addr_t *dma_addr); | ||
| 164 | void amdxdna_iommu_free(struct amdxdna_dev *xdna, size_t size, | ||
| 165 | void *cpu_addr, dma_addr_t dma_addr); | ||
| 166 | |||
| 167 | static inline bool amdxdna_iova_on(struct amdxdna_dev *xdna) | ||
| 168 | { | ||
| 169 | return !!xdna->domain; | ||
| 170 | } | ||
| 171 | |||
| 172 | static inline bool amdxdna_pasid_on(struct amdxdna_client *client) | ||
| 173 | { | ||
| 174 | return client->pasid != IOMMU_PASID_INVALID; | ||
| 175 | } | ||
| 148 | #endif /* _AMDXDNA_PCI_DRV_H_ */ | 176 | #endif /* _AMDXDNA_PCI_DRV_H_ */ |
| @@ -17,7 +17,6 @@ · 16 unmodified lines | |||
| 17 | struct amdxdna_ubuf_priv { | 17 | struct amdxdna_ubuf_priv { |
| 18 | struct page **pages; | 18 | struct page **pages; |
| 19 | u64 nr_pages; | 19 | u64 nr_pages; |
| 20 | enum amdxdna_ubuf_flag flags; | ||
| 21 | struct mm_struct *mm; | 20 | struct mm_struct *mm; |
| 22 | }; | 21 | }; |
| 23 | 22 | ||
| @@ -37,11 +36,9 @@ static struct sg_table *amdxdna_ubuf_map(struct dma_buf_attachment *attach, · 13 unmodified lines | |||
| 37 | if (ret) | 36 | if (ret) |
| 38 | goto err_free_sg; | 37 | goto err_free_sg; |
| 39 | 38 | ||
| 40 | if (ubuf->flags & AMDXDNA_UBUF_FLAG_MAP_DMA) { | 39 | ret = dma_map_sgtable(attach->dev, sg, direction, 0); |
| 41 | ret = dma_map_sgtable(attach->dev, sg, direction, 0); | 40 | if (ret) |
| 42 | if (ret) | 41 | goto err_free_table; |
| 43 | goto err_free_table; | ||
| 44 | } | ||
| 45 | 42 | ||
| 46 | return sg; | 43 | return sg; |
| 47 | 44 | ||
| @@ -56,11 +53,7 @@ static void amdxdna_ubuf_unmap(struct dma_buf_attachment *attach, · 8 unmodified lines | |||
| 56 | struct sg_table *sg, | 53 | struct sg_table *sg, |
| 57 | enum dma_data_direction direction) | 54 | enum dma_data_direction direction) |
| 58 | { | 55 | { |
| 59 | struct amdxdna_ubuf_priv *ubuf = attach->dmabuf->priv; | 56 | dma_unmap_sgtable(attach->dev, sg, direction, 0); |
| 60 | |||
| 61 | if (ubuf->flags & AMDXDNA_UBUF_FLAG_MAP_DMA) | ||
| 62 | dma_unmap_sgtable(attach->dev, sg, direction, 0); | ||
| 63 | |||
| 64 | sg_free_table(sg); | 57 | sg_free_table(sg); |
| 65 | kfree(sg); | 58 | kfree(sg); |
| 66 | } | 59 | } |
| @@ -133,7 +126,6 @@ static const struct dma_buf_ops amdxdna_ubuf_dmabuf_ops = { · 66 unmodified lines | |||
| 133 | }; | 126 | }; |
| 134 | 127 | ||
| 135 | struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, | 128 | struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, |
| 136 | enum amdxdna_ubuf_flag flags, | ||
| 137 | u32 num_entries, void __user *va_entries) | 129 | u32 num_entries, void __user *va_entries) |
| 138 | { | 130 | { |
| 139 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | 131 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| @@ -152,7 +144,6 @@ struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, · 12 unmodified lines | |||
| 152 | if (!ubuf) | 144 | if (!ubuf) |
| 153 | return ERR_PTR(-ENOMEM); | 145 | return ERR_PTR(-ENOMEM); |
| 154 | 146 | ||
| 155 | ubuf->flags = flags; | ||
| 156 | ubuf->mm = current->mm; | 147 | ubuf->mm = current->mm; |
| 157 | mmgrab(ubuf->mm); | 148 | mmgrab(ubuf->mm); |
| 158 | 149 | ||
| @@ -8,12 +8,7 @@ · 7 unmodified lines | |||
| 8 | #include <drm/drm_device.h> | 8 | #include <drm/drm_device.h> |
| 9 | #include <linux/dma-buf.h> | 9 | #include <linux/dma-buf.h> |
| 10 | 10 | ||
| 11 | enum amdxdna_ubuf_flag { | ||
| 12 | AMDXDNA_UBUF_FLAG_MAP_DMA = 1, | ||
| 13 | }; | ||
| 14 | |||
| 15 | struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, | 11 | struct dma_buf *amdxdna_get_ubuf(struct drm_device *dev, |
| 16 | enum amdxdna_ubuf_flag flags, | ||
| 17 | u32 num_entries, void __user *va_entries); | 12 | u32 num_entries, void __user *va_entries); |
| 18 | 13 | ||
| 19 | #endif /* _AMDXDNA_UBUF_H_ */ | 14 | #endif /* _AMDXDNA_UBUF_H_ */ |
| @@ -93,7 +93,8 @@ const struct aie2_fw_feature_tbl npu4_fw_feature_table[] = { · 92 unmodified lines | |||
| 93 | { .features = BIT_U64(AIE2_NPU_COMMAND), .major = 6, .min_minor = 15 }, | 93 | { .features = BIT_U64(AIE2_NPU_COMMAND), .major = 6, .min_minor = 15 }, |
| 94 | { .features = BIT_U64(AIE2_PREEMPT), .major = 6, .min_minor = 12 }, | 94 | { .features = BIT_U64(AIE2_PREEMPT), .major = 6, .min_minor = 12 }, |
| 95 | { .features = BIT_U64(AIE2_TEMPORAL_ONLY), .major = 6, .min_minor = 12 }, | 95 | { .features = BIT_U64(AIE2_TEMPORAL_ONLY), .major = 6, .min_minor = 12 }, |
| 96 | { .features = GENMASK_ULL(AIE2_TEMPORAL_ONLY, AIE2_NPU_COMMAND), .major = 7 }, | 96 | { .features = BIT_U64(AIE2_APP_HEALTH), .major = 6, .min_minor = 18 }, |
| 97 | { .features = AIE2_ALL_FEATURES, .major = 7 }, | ||
| 97 | { 0 } | 98 | { 0 } |
| 98 | }; | 99 | }; |
| 99 | 100 | ||
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2024 Intel Corporation | 3 | * Copyright (C) 2020-2026 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #include <linux/debugfs.h> | 6 | #include <linux/debugfs.h> |
| @@ -127,6 +127,14 @@ static int firewall_irq_counter_show(struct seq_file *s, void *v) · 120 unmodified lines | |||
| 127 | return 0; | 127 | return 0; |
| 128 | } | 128 | } |
| 129 | 129 | ||
| 130 | static int engine_reset_counter_show(struct seq_file *s, void *v) | ||
| 131 | { | ||
| 132 | struct ivpu_device *vdev = seq_to_ivpu(s); | ||
| 133 | |||
| 134 | seq_printf(s, "%d\n", atomic_read(&vdev->pm->engine_reset_counter)); | ||
| 135 | return 0; | ||
| 136 | } | ||
| 137 | |||
| 130 | static const struct drm_debugfs_info vdev_debugfs_list[] = { | 138 | static const struct drm_debugfs_info vdev_debugfs_list[] = { |
| 131 | {"bo_list", bo_list_show, 0}, | 139 | {"bo_list", bo_list_show, 0}, |
| 132 | {"fw_name", fw_name_show, 0}, | 140 | {"fw_name", fw_name_show, 0}, |
| @@ -137,6 +145,7 @@ static const struct drm_debugfs_info vdev_debugfs_list[] = { · 4 unmodified lines | |||
| 137 | {"reset_counter", reset_counter_show, 0}, | 145 | {"reset_counter", reset_counter_show, 0}, |
| 138 | {"reset_pending", reset_pending_show, 0}, | 146 | {"reset_pending", reset_pending_show, 0}, |
| 139 | {"firewall_irq_counter", firewall_irq_counter_show, 0}, | 147 | {"firewall_irq_counter", firewall_irq_counter_show, 0}, |
| 148 | {"engine_reset_counter", engine_reset_counter_show, 0}, | ||
| 140 | }; | 149 | }; |
| 141 | 150 | ||
| 142 | static int dvfs_mode_get(void *data, u64 *dvfs_mode) | 151 | static int dvfs_mode_get(void *data, u64 *dvfs_mode) |
| @@ -352,8 +361,9 @@ static const struct file_operations ivpu_force_recovery_fops = { · 209 unmodified lines | |||
| 352 | static int ivpu_reset_engine_fn(void *data, u64 val) | 361 | static int ivpu_reset_engine_fn(void *data, u64 val) |
| 353 | { | 362 | { |
| 354 | struct ivpu_device *vdev = (struct ivpu_device *)data; | 363 | struct ivpu_device *vdev = (struct ivpu_device *)data; |
| 364 | struct vpu_jsm_msg resp; | ||
| 355 | 365 | ||
| 356 | return ivpu_jsm_reset_engine(vdev, (u32)val); | 366 | return ivpu_jsm_reset_engine(vdev, (u32)val, &resp); |
| 357 | } | 367 | } |
| 358 | 368 | ||
| 359 | DEFINE_DEBUGFS_ATTRIBUTE(ivpu_reset_engine_fops, NULL, ivpu_reset_engine_fn, "0x%02llx\n"); | 369 | DEFINE_DEBUGFS_ATTRIBUTE(ivpu_reset_engine_fops, NULL, ivpu_reset_engine_fn, "0x%02llx\n"); |
| @@ -665,6 +665,7 @@ static int ivpu_dev_init(struct ivpu_device *vdev) · 664 unmodified lines | |||
| 665 | vdev->context_xa_limit.max = IVPU_USER_CONTEXT_MAX_SSID; | 665 | vdev->context_xa_limit.max = IVPU_USER_CONTEXT_MAX_SSID; |
| 666 | atomic64_set(&vdev->unique_id_counter, 0); | 666 | atomic64_set(&vdev->unique_id_counter, 0); |
| 667 | atomic_set(&vdev->job_timeout_counter, 0); | 667 | atomic_set(&vdev->job_timeout_counter, 0); |
| 668 | atomic_set(&vdev->faults_detected, 0); | ||
| 668 | xa_init_flags(&vdev->context_xa, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ); | 669 | xa_init_flags(&vdev->context_xa, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ); |
| 669 | xa_init_flags(&vdev->submitted_jobs_xa, XA_FLAGS_ALLOC1); | 670 | xa_init_flags(&vdev->submitted_jobs_xa, XA_FLAGS_ALLOC1); |
| 670 | xa_init_flags(&vdev->db_xa, XA_FLAGS_ALLOC1); | 671 | xa_init_flags(&vdev->db_xa, XA_FLAGS_ALLOC1); |
| @@ -1,6 +1,6 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only */ | 1 | /* SPDX-License-Identifier: GPL-2.0-only */ |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2025 Intel Corporation | 3 | * Copyright (C) 2020-2026 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #ifndef __IVPU_DRV_H__ | 6 | #ifndef __IVPU_DRV_H__ |
| @@ -36,6 +36,7 @@ · 29 unmodified lines | |||
| 36 | #define IVPU_HW_IP_60XX 60 | 36 | #define IVPU_HW_IP_60XX 60 |
| 37 | 37 | ||
| 38 | #define IVPU_HW_IP_REV_LNL_B0 4 | 38 | #define IVPU_HW_IP_REV_LNL_B0 4 |
| 39 | #define IVPU_HW_IP_REV_NVL_A0 0 | ||
| 39 | 40 | ||
| 40 | #define IVPU_HW_BTRS_MTL 1 | 41 | #define IVPU_HW_BTRS_MTL 1 |
| 41 | #define IVPU_HW_BTRS_LNL 2 | 42 | #define IVPU_HW_BTRS_LNL 2 |
| @@ -168,6 +169,7 @@ struct ivpu_device { · 126 unmodified lines | |||
| 168 | struct xarray submitted_jobs_xa; | 169 | struct xarray submitted_jobs_xa; |
| 169 | struct ivpu_ipc_consumer job_done_consumer; | 170 | struct ivpu_ipc_consumer job_done_consumer; |
| 170 | atomic_t job_timeout_counter; | 171 | atomic_t job_timeout_counter; |
| 172 | atomic_t faults_detected; | ||
| 171 | 173 | ||
| 172 | atomic64_t unique_id_counter; | 174 | atomic64_t unique_id_counter; |
| 173 | 175 | ||
| @@ -70,8 +70,10 @@ static void wa_init(struct ivpu_device *vdev) · 69 unmodified lines | |||
| 70 | if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL) | 70 | if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL) |
| 71 | vdev->wa.interrupt_clear_with_0 = ivpu_hw_btrs_irqs_clear_with_0_mtl(vdev); | 71 | vdev->wa.interrupt_clear_with_0 = ivpu_hw_btrs_irqs_clear_with_0_mtl(vdev); |
| 72 | 72 | ||
| 73 | if (ivpu_device_id(vdev) == PCI_DEVICE_ID_LNL && | 73 | if ((ivpu_device_id(vdev) == PCI_DEVICE_ID_LNL && |
| 74 | ivpu_revision(vdev) < IVPU_HW_IP_REV_LNL_B0) | 74 | ivpu_revision(vdev) < IVPU_HW_IP_REV_LNL_B0) || |
| 75 | (ivpu_device_id(vdev) == PCI_DEVICE_ID_NVL && | ||
| 76 | ivpu_revision(vdev) == IVPU_HW_IP_REV_NVL_A0)) | ||
| 75 | vdev->wa.disable_clock_relinquish = true; | 77 | vdev->wa.disable_clock_relinquish = true; |
| 76 | 78 | ||
| 77 | if (ivpu_test_mode & IVPU_TEST_MODE_CLK_RELINQ_ENABLE) | 79 | if (ivpu_test_mode & IVPU_TEST_MODE_CLK_RELINQ_ENABLE) |
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2025 Intel Corporation | 3 | * Copyright (C) 2020-2026 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #include <drm/drm_file.h> | 6 | #include <drm/drm_file.h> |
| @@ -607,6 +607,7 @@ bool ivpu_job_handle_engine_error(struct ivpu_device *vdev, u32 job_id, u32 job_ · 600 unmodified lines | |||
| 607 | * status and ensure both are handled in the same way | 607 | * status and ensure both are handled in the same way |
| 608 | */ | 608 | */ |
| 609 | job->file_priv->has_mmu_faults = true; | 609 | job->file_priv->has_mmu_faults = true; |
| 610 | atomic_set(&vdev->faults_detected, 1); | ||
| 610 | queue_work(system_percpu_wq, &vdev->context_abort_work); | 611 | queue_work(system_percpu_wq, &vdev->context_abort_work); |
| 611 | return true; | 612 | return true; |
| 612 | } | 613 | } |
| @@ -1115,6 +1116,51 @@ void ivpu_job_done_consumer_fini(struct ivpu_device *vdev) · 502 unmodified lines | |||
| 1115 | ivpu_ipc_consumer_del(vdev, &vdev->job_done_consumer); | 1116 | ivpu_ipc_consumer_del(vdev, &vdev->job_done_consumer); |
| 1116 | } | 1117 | } |
| 1117 | 1118 | ||
| 1119 | static int reset_engine_and_mark_faulty_contexts(struct ivpu_device *vdev) | ||
| 1120 | { | ||
| 1121 | u32 num_impacted_contexts; | ||
| 1122 | struct vpu_jsm_msg resp; | ||
| 1123 | int ret; | ||
| 1124 | u32 i; | ||
| 1125 | |||
| 1126 | ret = ivpu_jsm_reset_engine(vdev, 0, &resp); | ||
| 1127 | if (ret) | ||
| 1128 | return ret; | ||
| 1129 | |||
| 1130 | /* | ||
| 1131 | * If faults are detected, ignore guilty contexts from engine reset as NPU may not be stuck | ||
| 1132 | * and could return currently running good context and faulty contexts are already marked | ||
| 1133 | */ | ||
| 1134 | if (atomic_cmpxchg(&vdev->faults_detected, 1, 0) == 1) | ||
| 1135 | return 0; | ||
| 1136 | |||
| 1137 | num_impacted_contexts = resp.payload.engine_reset_done.num_impacted_contexts; | ||
| 1138 | |||
| 1139 | ivpu_warn_ratelimited(vdev, "Engine reset performed, impacted contexts: %u\n", | ||
| 1140 | num_impacted_contexts); | ||
| 1141 | |||
| 1142 | if (!in_range(num_impacted_contexts, 1, VPU_MAX_ENGINE_RESET_IMPACTED_CONTEXTS - 1)) { | ||
| 1143 | ivpu_pm_trigger_recovery(vdev, "Cannot determine guilty contexts"); | ||
| 1144 | return -EIO; | ||
| 1145 | } | ||
| 1146 | |||
| 1147 | /* No faults detected, NPU likely got stuck. Mark returned contexts as guilty */ | ||
| 1148 | guard(mutex)(&vdev->context_list_lock); | ||
| 1149 | |||
| 1150 | for (i = 0; i < num_impacted_contexts; i++) { | ||
| 1151 | u32 ssid = resp.payload.engine_reset_done.impacted_contexts[i].host_ssid; | ||
| 1152 | struct ivpu_file_priv *file_priv = xa_load(&vdev->context_xa, ssid); | ||
| 1153 | |||
| 1154 | if (file_priv) { | ||
| 1155 | mutex_lock(&file_priv->lock); | ||
| 1156 | file_priv->has_mmu_faults = true; | ||
| 1157 | mutex_unlock(&file_priv->lock); | ||
| 1158 | } | ||
| 1159 | } | ||
| 1160 | |||
| 1161 | return 0; | ||
| 1162 | } | ||
| 1163 | |||
| 1118 | void ivpu_context_abort_work_fn(struct work_struct *work) | 1164 | void ivpu_context_abort_work_fn(struct work_struct *work) |
| 1119 | { | 1165 | { |
| 1120 | struct ivpu_device *vdev = container_of(work, struct ivpu_device, context_abort_work); | 1166 | struct ivpu_device *vdev = container_of(work, struct ivpu_device, context_abort_work); |
| @@ -1127,7 +1173,7 @@ void ivpu_context_abort_work_fn(struct work_struct *work) · 6 unmodified lines | |||
| 1127 | return; | 1173 | return; |
| 1128 | 1174 | ||
| 1129 | if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) | 1175 | if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) |
| 1130 | if (ivpu_jsm_reset_engine(vdev, 0)) | 1176 | if (reset_engine_and_mark_faulty_contexts(vdev)) |
| 1131 | goto runtime_put; | 1177 | goto runtime_put; |
| 1132 | 1178 | ||
| 1133 | mutex_lock(&vdev->context_list_lock); | 1179 | mutex_lock(&vdev->context_list_lock); |
| @@ -151,10 +151,9 @@ int ivpu_jsm_get_heartbeat(struct ivpu_device *vdev, u32 engine, u64 *heartbeat) · 150 unmodified lines | |||
| 151 | return ret; | 151 | return ret; |
| 152 | } | 152 | } |
| 153 | 153 | ||
| 154 | int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine) | 154 | int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine, struct vpu_jsm_msg *resp) |
| 155 | { | 155 | { |
| 156 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_ENGINE_RESET }; | 156 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_ENGINE_RESET }; |
| 157 | struct vpu_jsm_msg resp; | ||
| 158 | int ret; | 157 | int ret; |
| 159 | 158 | ||
| 160 | if (engine != VPU_ENGINE_COMPUTE) | 159 | if (engine != VPU_ENGINE_COMPUTE) |
| @@ -162,14 +161,17 @@ int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine) · 1 unmodified line | |||
| 162 | 161 | ||
| 163 | req.payload.engine_reset.engine_idx = engine; | 162 | req.payload.engine_reset.engine_idx = engine; |
| 164 | 163 | ||
| 165 | ret = ivpu_ipc_send_receive(vdev, &req, VPU_JSM_MSG_ENGINE_RESET_DONE, &resp, | 164 | ret = ivpu_ipc_send_receive(vdev, &req, VPU_JSM_MSG_ENGINE_RESET_DONE, resp, |
| 166 | VPU_IPC_CHAN_ASYNC_CMD, vdev->timeout.jsm); | 165 | VPU_IPC_CHAN_ASYNC_CMD, vdev->timeout.jsm); |
| 167 | if (ret) { | 166 | if (ret) { |
| 168 | ivpu_err_ratelimited(vdev, "Failed to reset engine %d: %d\n", engine, ret); | 167 | ivpu_err_ratelimited(vdev, "Failed to reset engine %d: %d\n", engine, ret); |
| 169 | ivpu_pm_trigger_recovery(vdev, "Engine reset failed"); | 168 | ivpu_pm_trigger_recovery(vdev, "Engine reset failed"); |
| 169 | return ret; | ||
| 170 | } | 170 | } |
| 171 | 171 | ||
| 172 | return ret; | 172 | atomic_inc(&vdev->pm->engine_reset_counter); |
| 173 | |||
| 174 | return 0; | ||
| 173 | } | 175 | } |
| 174 | 176 | ||
| 175 | int ivpu_jsm_preempt_engine(struct ivpu_device *vdev, u32 engine, u32 preempt_id) | 177 | int ivpu_jsm_preempt_engine(struct ivpu_device *vdev, u32 engine, u32 preempt_id) |
| @@ -554,6 +556,15 @@ int ivpu_jsm_dct_disable(struct ivpu_device *vdev) · 378 unmodified lines | |||
| 554 | } | 556 | } |
| 555 | 557 | ||
| 556 | int ivpu_jsm_state_dump(struct ivpu_device *vdev) | 558 | int ivpu_jsm_state_dump(struct ivpu_device *vdev) |
| 559 | { | ||
| 560 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_STATE_DUMP }; | ||
| 561 | struct vpu_jsm_msg resp; | ||
| 562 | |||
| 563 | return ivpu_ipc_send_receive_internal(vdev, &req, VPU_JSM_MSG_STATE_DUMP_RSP, &resp, | ||
| 564 | VPU_IPC_CHAN_ASYNC_CMD, vdev->timeout.jsm); | ||
| 565 | } | ||
| 566 | |||
| 567 | int ivpu_jsm_state_dump_no_reply(struct ivpu_device *vdev) | ||
| 557 | { | 568 | { |
| 558 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_STATE_DUMP }; | 569 | struct vpu_jsm_msg req = { .type = VPU_JSM_MSG_STATE_DUMP }; |
| 559 | 570 | ||
| @@ -14,7 +14,7 @@ int ivpu_jsm_register_db(struct ivpu_device *vdev, u32 ctx_id, u32 db_id, · 13 unmodified lines | |||
| 14 | u64 jobq_base, u32 jobq_size); | 14 | u64 jobq_base, u32 jobq_size); |
| 15 | int ivpu_jsm_unregister_db(struct ivpu_device *vdev, u32 db_id); | 15 | int ivpu_jsm_unregister_db(struct ivpu_device *vdev, u32 db_id); |
| 16 | int ivpu_jsm_get_heartbeat(struct ivpu_device *vdev, u32 engine, u64 *heartbeat); | 16 | int ivpu_jsm_get_heartbeat(struct ivpu_device *vdev, u32 engine, u64 *heartbeat); |
| 17 | int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine); | 17 | int ivpu_jsm_reset_engine(struct ivpu_device *vdev, u32 engine, struct vpu_jsm_msg *response); |
| 18 | int ivpu_jsm_preempt_engine(struct ivpu_device *vdev, u32 engine, u32 preempt_id); | 18 | int ivpu_jsm_preempt_engine(struct ivpu_device *vdev, u32 engine, u32 preempt_id); |
| 19 | int ivpu_jsm_dyndbg_control(struct ivpu_device *vdev, char *command, size_t size); | 19 | int ivpu_jsm_dyndbg_control(struct ivpu_device *vdev, char *command, size_t size); |
| 20 | int ivpu_jsm_trace_get_capability(struct ivpu_device *vdev, u32 *trace_destination_mask, | 20 | int ivpu_jsm_trace_get_capability(struct ivpu_device *vdev, u32 *trace_destination_mask, |
| @@ -44,5 +44,6 @@ int ivpu_jsm_metric_streamer_info(struct ivpu_device *vdev, u64 metric_group_mas · 23 unmodified lines | |||
| 44 | int ivpu_jsm_dct_enable(struct ivpu_device *vdev, u32 active_us, u32 inactive_us); | 44 | int ivpu_jsm_dct_enable(struct ivpu_device *vdev, u32 active_us, u32 inactive_us); |
| 45 | int ivpu_jsm_dct_disable(struct ivpu_device *vdev); | 45 | int ivpu_jsm_dct_disable(struct ivpu_device *vdev); |
| 46 | int ivpu_jsm_state_dump(struct ivpu_device *vdev); | 46 | int ivpu_jsm_state_dump(struct ivpu_device *vdev); |
| 47 | int ivpu_jsm_state_dump_no_reply(struct ivpu_device *vdev); | ||
| 47 | 48 | ||
| 48 | #endif | 49 | #endif |
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2024 Intel Corporation | 3 | * Copyright (C) 2020-2026 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #include <linux/circ_buf.h> | 6 | #include <linux/circ_buf.h> |
| @@ -964,6 +964,7 @@ void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) · 957 unmodified lines | |||
| 964 | file_priv = xa_load(&vdev->context_xa, ssid); | 964 | file_priv = xa_load(&vdev->context_xa, ssid); |
| 965 | if (file_priv) { | 965 | if (file_priv) { |
| 966 | if (!READ_ONCE(file_priv->has_mmu_faults)) { | 966 | if (!READ_ONCE(file_priv->has_mmu_faults)) { |
| 967 | atomic_set(&vdev->faults_detected, 1); | ||
| 967 | ivpu_mmu_dump_event(vdev, event); | 968 | ivpu_mmu_dump_event(vdev, event); |
| 968 | WRITE_ONCE(file_priv->has_mmu_faults, true); | 969 | WRITE_ONCE(file_priv->has_mmu_faults, true); |
| 969 | } | 970 | } |
| @@ -1,6 +1,6 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2024 Intel Corporation | 3 | * Copyright (C) 2020-2026 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #include <linux/highmem.h> | 6 | #include <linux/highmem.h> |
| @@ -166,7 +166,7 @@ static void ivpu_pm_recovery_work(struct work_struct *work) · 159 unmodified lines | |||
| 166 | ivpu_pm_reset_begin(vdev); | 166 | ivpu_pm_reset_begin(vdev); |
| 167 | 167 | ||
| 168 | if (!pm_runtime_status_suspended(vdev->drm.dev)) { | 168 | if (!pm_runtime_status_suspended(vdev->drm.dev)) { |
| 169 | ivpu_jsm_state_dump(vdev); | 169 | ivpu_jsm_state_dump_no_reply(vdev); |
| 170 | ivpu_dev_coredump(vdev); | 170 | ivpu_dev_coredump(vdev); |
| 171 | ivpu_suspend(vdev); | 171 | ivpu_suspend(vdev); |
| 172 | } | 172 | } |
| @@ -205,23 +205,25 @@ static void ivpu_job_timeout_work(struct work_struct *work) · 32 unmodified lines | |||
| 205 | 205 | ||
| 206 | if (ivpu_jsm_get_heartbeat(vdev, 0, &heartbeat) || heartbeat <= vdev->fw->last_heartbeat) { | 206 | if (ivpu_jsm_get_heartbeat(vdev, 0, &heartbeat) || heartbeat <= vdev->fw->last_heartbeat) { |
| 207 | ivpu_err(vdev, "Job timeout detected, heartbeat not progressed\n"); | 207 | ivpu_err(vdev, "Job timeout detected, heartbeat not progressed\n"); |
| 208 | goto recovery; | 208 | goto abort; |
| 209 | } | 209 | } |
| 210 | 210 | ||
| 211 | inference_max_retries = DIV_ROUND_UP(inference_timeout_ms, timeout_ms); | 211 | inference_max_retries = DIV_ROUND_UP(inference_timeout_ms, timeout_ms); |
| 212 | if (atomic_fetch_inc(&vdev->job_timeout_counter) >= inference_max_retries) { | 212 | if (atomic_fetch_inc(&vdev->job_timeout_counter) >= inference_max_retries) { |
| 213 | ivpu_err(vdev, "Job timeout detected, heartbeat limit (%lld) exceeded\n", | 213 | ivpu_err(vdev, "Job timeout detected, heartbeat limit (%lld) exceeded\n", |
| 214 | inference_max_retries); | 214 | inference_max_retries); |
| 215 | goto recovery; | 215 | goto abort; |
| 216 | } | 216 | } |
| 217 | 217 | ||
| 218 | vdev->fw->last_heartbeat = heartbeat; | 218 | vdev->fw->last_heartbeat = heartbeat; |
| 219 | ivpu_start_job_timeout_detection(vdev); | 219 | ivpu_start_job_timeout_detection(vdev); |
| 220 | return; | 220 | return; |
| 221 | 221 | ||
| 222 | recovery: | 222 | abort: |
| 223 | atomic_set(&vdev->job_timeout_counter, 0); | 223 | atomic_set(&vdev->job_timeout_counter, 0); |
| 224 | ivpu_pm_trigger_recovery(vdev, "TDR"); | 224 | ivpu_jsm_state_dump(vdev); |
| 225 | ivpu_dev_coredump(vdev); | ||
| 226 | queue_work(system_percpu_wq, &vdev->context_abort_work); | ||
| 225 | } | 227 | } |
| 226 | 228 | ||
| 227 | void ivpu_start_job_timeout_detection(struct ivpu_device *vdev) | 229 | void ivpu_start_job_timeout_detection(struct ivpu_device *vdev) |
| @@ -404,6 +406,7 @@ void ivpu_pm_init(struct ivpu_device *vdev) · 176 unmodified lines | |||
| 404 | init_rwsem(&pm->reset_lock); | 406 | init_rwsem(&pm->reset_lock); |
| 405 | atomic_set(&pm->reset_pending, 0); | 407 | atomic_set(&pm->reset_pending, 0); |
| 406 | atomic_set(&pm->reset_counter, 0); | 408 | atomic_set(&pm->reset_counter, 0); |
| 409 | atomic_set(&pm->engine_reset_counter, 0); | ||
| 407 | 410 | ||
| 408 | INIT_WORK(&pm->recovery_work, ivpu_pm_recovery_work); | 411 | INIT_WORK(&pm->recovery_work, ivpu_pm_recovery_work); |
| 409 | INIT_DELAYED_WORK(&pm->job_timeout_work, ivpu_job_timeout_work); | 412 | INIT_DELAYED_WORK(&pm->job_timeout_work, ivpu_job_timeout_work); |
| @@ -18,6 +18,7 @@ struct ivpu_pm_info { · 17 unmodified lines | |||
| 18 | struct rw_semaphore reset_lock; | 18 | struct rw_semaphore reset_lock; |
| 19 | atomic_t reset_counter; | 19 | atomic_t reset_counter; |
| 20 | atomic_t reset_pending; | 20 | atomic_t reset_pending; |
| 21 | atomic_t engine_reset_counter; | ||
| 21 | u8 dct_active_percent; | 22 | u8 dct_active_percent; |
| 22 | }; | 23 | }; |
| 23 | 24 | ||
| @@ -914,7 +914,7 @@ static int decode_deactivate(struct qaic_device *qdev, void *trans, u32 *msg_len · 913 unmodified lines | |||
| 914 | */ | 914 | */ |
| 915 | return -ENODEV; | 915 | return -ENODEV; |
| 916 | 916 | ||
| 917 | if (status) { | 917 | if (usr && status) { |
| 918 | /* | 918 | /* |
| 919 | * Releasing resources failed on the device side, which puts | 919 | * Releasing resources failed on the device side, which puts |
| 920 | * us in a bind since they may still be in use, so enable the | 920 | * us in a bind since they may still be in use, so enable the |
| @@ -1109,6 +1109,9 @@ static void *msg_xfer(struct qaic_device *qdev, struct wrapper_list *wrappers, u · 188 unmodified lines | |||
| 1109 | mutex_lock(&qdev->cntl_mutex); | 1109 | mutex_lock(&qdev->cntl_mutex); |
| 1110 | if (!list_empty(&elem.list)) | 1110 | if (!list_empty(&elem.list)) |
| 1111 | list_del(&elem.list); | 1111 | list_del(&elem.list); |
| 1112 | /* resp_worker() processed the response but the wait was interrupted */ | ||
| 1113 | else if (ret == -ERESTARTSYS) | ||
| 1114 | ret = 0; | ||
| 1112 | if (!ret && !elem.buf) | 1115 | if (!ret && !elem.buf) |
| 1113 | ret = -ETIMEDOUT; | 1116 | ret = -ETIMEDOUT; |
| 1114 | else if (ret > 0 && !elem.buf) | 1117 | else if (ret > 0 && !elem.buf) |
| @@ -1419,9 +1422,49 @@ static void resp_worker(struct work_struct *work) · 304 unmodified lines | |||
| 1419 | } | 1422 | } |
| 1420 | mutex_unlock(&qdev->cntl_mutex); | 1423 | mutex_unlock(&qdev->cntl_mutex); |
| 1421 | 1424 | ||
| 1422 | if (!found) | 1425 | if (!found) { |
| 1426 | /* | ||
| 1427 | * The user might have gone away at this point without waiting | ||
| 1428 | * for QAIC_TRANS_DEACTIVATE_FROM_DEV transaction coming from | ||
| 1429 | * the device. If this is not handled correctly, the host will | ||
| 1430 | * not know that the DBC[n] has been freed on the device. | ||
| 1431 | * Due to this failure in synchronization between the device and | ||
| 1432 | * the host, if another user requests to activate a network, and | ||
| 1433 | * the device assigns DBC[n] again, save_dbc_buf() will hang, | ||
| 1434 | * waiting for dbc[n]->in_use to be set to false, which will not | ||
| 1435 | * happen unless the qaic_dev_reset_clean_local_state() gets | ||
| 1436 | * called by resetting the device (or re-inserting the module). | ||
| 1437 | * | ||
| 1438 | * As a solution, we look for QAIC_TRANS_DEACTIVATE_FROM_DEV | ||
| 1439 | * transactions in the message before disposing of it, then | ||
| 1440 | * handle releasing the DBC resources. | ||
| 1441 | * | ||
| 1442 | * Since the user has gone away, if the device could not | ||
| 1443 | * deactivate the network (status != 0), there is no way to | ||
| 1444 | * enable and reassign the DBC to the user. We can put trust in | ||
| 1445 | * the device that it will release all the active DBCs in | ||
| 1446 | * response to the QAIC_TRANS_TERMINATE_TO_DEV transaction, | ||
| 1447 | * otherwise, the user can issue an soc_reset to the device. | ||
| 1448 | */ | ||
| 1449 | u32 msg_count = le32_to_cpu(msg->hdr.count); | ||
| 1450 | u32 msg_len = le32_to_cpu(msg->hdr.len); | ||
| 1451 | u32 len = 0; | ||
| 1452 | int j; | ||
| 1453 | |||
| 1454 | for (j = 0; j < msg_count && len < msg_len; ++j) { | ||
| 1455 | struct wire_trans_hdr *trans_hdr; | ||
| 1456 | |||
| 1457 | trans_hdr = (struct wire_trans_hdr *)(msg->data + len); | ||
| 1458 | if (le32_to_cpu(trans_hdr->type) == QAIC_TRANS_DEACTIVATE_FROM_DEV) { | ||
| 1459 | if (decode_deactivate(qdev, trans_hdr, &len, NULL)) | ||
| 1460 | len += le32_to_cpu(trans_hdr->len); | ||
| 1461 | } else { | ||
| 1462 | len += le32_to_cpu(trans_hdr->len); | ||
| 1463 | } | ||
| 1464 | } | ||
| 1423 | /* request must have timed out, drop packet */ | 1465 | /* request must have timed out, drop packet */ |
| 1424 | kfree(msg); | 1466 | kfree(msg); |
| 1467 | } | ||
| 1425 | 1468 | ||
| 1426 | kfree(resp); | 1469 | kfree(resp); |
| 1427 | } | 1470 | } |
| @@ -9,6 +9,7 @@ config ARCH_SUPPORTS_ACPI · 8 unmodified lines | |||
| 9 | menuconfig ACPI | 9 | menuconfig ACPI |
| 10 | bool "ACPI (Advanced Configuration and Power Interface) Support" | 10 | bool "ACPI (Advanced Configuration and Power Interface) Support" |
| 11 | depends on ARCH_SUPPORTS_ACPI | 11 | depends on ARCH_SUPPORTS_ACPI |
| 12 | select AUXILIARY_BUS | ||
| 12 | select PNP | 13 | select PNP |
| 13 | select NLS | 14 | select NLS |
| 14 | select CRC32 | 15 | select CRC32 |
| @@ -135,7 +135,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev, · 134 unmodified lines | |||
| 135 | } | 135 | } |
| 136 | } | 136 | } |
| 137 | 137 | ||
| 138 | if (adev->device_type == ACPI_BUS_TYPE_DEVICE && !adev->pnp.type.backlight) { | 138 | if (adev->device_type == ACPI_BUS_TYPE_DEVICE) { |
| 139 | LIST_HEAD(resource_list); | 139 | LIST_HEAD(resource_list); |
| 140 | 140 | ||
| 141 | count = acpi_dev_get_resources(adev, &resource_list, NULL, NULL); | 141 | count = acpi_dev_get_resources(adev, &resource_list, NULL, NULL); |
| @@ -113,6 +113,10 @@ static int acpi_processor_errata_piix4(struct pci_dev *dev) · 112 unmodified lines | |||
| 113 | PCI_ANY_ID, PCI_ANY_ID, NULL); | 113 | PCI_ANY_ID, PCI_ANY_ID, NULL); |
| 114 | if (ide_dev) { | 114 | if (ide_dev) { |
| 115 | errata.piix4.bmisx = pci_resource_start(ide_dev, 4); | 115 | errata.piix4.bmisx = pci_resource_start(ide_dev, 4); |
| 116 | if (errata.piix4.bmisx) | ||
| 117 | dev_dbg(&ide_dev->dev, | ||
| 118 | "Bus master activity detection (BM-IDE) erratum enabled\n"); | ||
| 119 | |||
| 116 | pci_dev_put(ide_dev); | 120 | pci_dev_put(ide_dev); |
| 117 | } | 121 | } |
| 118 | 122 | ||
| @@ -131,20 +135,17 @@ static int acpi_processor_errata_piix4(struct pci_dev *dev) · 12 unmodified lines | |||
| 131 | if (isa_dev) { | 135 | if (isa_dev) { |
| 132 | pci_read_config_byte(isa_dev, 0x76, &value1); | 136 | pci_read_config_byte(isa_dev, 0x76, &value1); |
| 133 | pci_read_config_byte(isa_dev, 0x77, &value2); | 137 | pci_read_config_byte(isa_dev, 0x77, &value2); |
| 134 | if ((value1 & 0x80) || (value2 & 0x80)) | 138 | if ((value1 & 0x80) || (value2 & 0x80)) { |
| 135 | errata.piix4.fdma = 1; | 139 | errata.piix4.fdma = 1; |
| 140 | dev_dbg(&isa_dev->dev, | ||
| 141 | "Type-F DMA livelock erratum (C3 disabled)\n"); | ||
| 142 | } | ||
| 136 | pci_dev_put(isa_dev); | 143 | pci_dev_put(isa_dev); |
| 137 | } | 144 | } |
| 138 | 145 | ||
| 139 | break; | 146 | break; |
| 140 | } | 147 | } |
| 141 | 148 | ||
| 142 | if (ide_dev) | ||
| 143 | dev_dbg(&ide_dev->dev, "Bus master activity detection (BM-IDE) erratum enabled\n"); | ||
| 144 | |||
| 145 | if (isa_dev) | ||
| 146 | dev_dbg(&isa_dev->dev, "Type-F DMA livelock erratum (C3 disabled)\n"); | ||
| 147 | |||
| 148 | return 0; | 149 | return 0; |
| 149 | } | 150 | } |
| 150 | 151 | ||
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | 9 | ||
| 10 | #define pr_fmt(fmt) "ACPI: video: " fmt | 10 | #define pr_fmt(fmt) "ACPI: video: " fmt |
| 11 | 11 | ||
| 12 | #include <linux/auxiliary_bus.h> | ||
| 12 | #include <linux/kernel.h> | 13 | #include <linux/kernel.h> |
| 13 | #include <linux/module.h> | 14 | #include <linux/module.h> |
| 14 | #include <linux/init.h> | 15 | #include <linux/init.h> |
| @@ -21,7 +22,6 @@ · 6 unmodified lines | |||
| 21 | #include <linux/sort.h> | 22 | #include <linux/sort.h> |
| 22 | #include <linux/pci.h> | 23 | #include <linux/pci.h> |
| 23 | #include <linux/pci_ids.h> | 24 | #include <linux/pci_ids.h> |
| 24 | #include <linux/platform_device.h> | ||
| 25 | #include <linux/slab.h> | 25 | #include <linux/slab.h> |
| 26 | #include <linux/dmi.h> | 26 | #include <linux/dmi.h> |
| 27 | #include <linux/suspend.h> | 27 | #include <linux/suspend.h> |
| @@ -77,8 +77,9 @@ static int register_count; · 49 unmodified lines | |||
| 77 | static DEFINE_MUTEX(register_count_mutex); | 77 | static DEFINE_MUTEX(register_count_mutex); |
| 78 | static DEFINE_MUTEX(video_list_lock); | 78 | static DEFINE_MUTEX(video_list_lock); |
| 79 | static LIST_HEAD(video_bus_head); | 79 | static LIST_HEAD(video_bus_head); |
| 80 | static int acpi_video_bus_probe(struct platform_device *pdev); | 80 | static int acpi_video_bus_probe(struct auxiliary_device *aux_dev, |
| 81 | static void acpi_video_bus_remove(struct platform_device *pdev); | 81 | const struct auxiliary_device_id *id); |
| 82 | static void acpi_video_bus_remove(struct auxiliary_device *aux); | ||
| 82 | static void acpi_video_bus_notify(acpi_handle handle, u32 event, void *data); | 83 | static void acpi_video_bus_notify(acpi_handle handle, u32 event, void *data); |
| 83 | 84 | ||
| 84 | /* | 85 | /* |
| @@ -93,19 +94,16 @@ enum acpi_video_level_idx { · 8 unmodified lines | |||
| 93 | ACPI_VIDEO_FIRST_LEVEL, /* actual supported levels begin here */ | 94 | ACPI_VIDEO_FIRST_LEVEL, /* actual supported levels begin here */ |
| 94 | }; | 95 | }; |
| 95 | 96 | ||
| 96 | static const struct acpi_device_id video_device_ids[] = { | 97 | static const struct auxiliary_device_id video_bus_auxiliary_id_table[] = { |
| 97 | {ACPI_VIDEO_HID, 0}, | 98 | { .name = "acpi.video_bus" }, |
| 98 | {"", 0}, | 99 | {}, |
| 99 | }; | 100 | }; |
| 100 | MODULE_DEVICE_TABLE(acpi, video_device_ids); | 101 | MODULE_DEVICE_TABLE(auxiliary, video_bus_auxiliary_id_table); |
| 101 | 102 | ||
| 102 | static struct platform_driver acpi_video_bus = { | 103 | static struct auxiliary_driver acpi_video_bus = { |
| 103 | .probe = acpi_video_bus_probe, | 104 | .probe = acpi_video_bus_probe, |
| 104 | .remove = acpi_video_bus_remove, | 105 | .remove = acpi_video_bus_remove, |
| 105 | .driver = { | 106 | .id_table = video_bus_auxiliary_id_table, |
| 106 | .name = "acpi-video", | ||
| 107 | .acpi_match_table = video_device_ids, | ||
| 108 | }, | ||
| 109 | }; | 107 | }; |
| 110 | 108 | ||
| 111 | struct acpi_video_bus_flags { | 109 | struct acpi_video_bus_flags { |
| @@ -1885,7 +1883,7 @@ static void acpi_video_dev_add_notify_handler(struct acpi_video_device *device) · 1773 unmodified lines | |||
| 1885 | } | 1883 | } |
| 1886 | 1884 | ||
| 1887 | static int acpi_video_bus_add_notify_handler(struct acpi_video_bus *video, | 1885 | static int acpi_video_bus_add_notify_handler(struct acpi_video_bus *video, |
| 1888 | struct platform_device *pdev) | 1886 | struct device *parent) |
| 1889 | { | 1887 | { |
| 1890 | struct input_dev *input; | 1888 | struct input_dev *input; |
| 1891 | struct acpi_video_device *dev; | 1889 | struct acpi_video_device *dev; |
| @@ -1908,7 +1906,7 @@ static int acpi_video_bus_add_notify_handler(struct acpi_video_bus *video, · 16 unmodified lines | |||
| 1908 | input->phys = video->phys; | 1906 | input->phys = video->phys; |
| 1909 | input->id.bustype = BUS_HOST; | 1907 | input->id.bustype = BUS_HOST; |
| 1910 | input->id.product = 0x06; | 1908 | input->id.product = 0x06; |
| 1911 | input->dev.parent = &pdev->dev; | 1909 | input->dev.parent = parent; |
| 1912 | input->evbit[0] = BIT(EV_KEY); | 1910 | input->evbit[0] = BIT(EV_KEY); |
| 1913 | set_bit(KEY_SWITCHVIDEOMODE, input->keybit); | 1911 | set_bit(KEY_SWITCHVIDEOMODE, input->keybit); |
| 1914 | set_bit(KEY_VIDEO_NEXT, input->keybit); | 1912 | set_bit(KEY_VIDEO_NEXT, input->keybit); |
| @@ -1980,9 +1978,10 @@ static int acpi_video_bus_put_devices(struct acpi_video_bus *video) · 65 unmodified lines | |||
| 1980 | 1978 | ||
| 1981 | static int instance; | 1979 | static int instance; |
| 1982 | 1980 | ||
| 1983 | static int acpi_video_bus_probe(struct platform_device *pdev) | 1981 | static int acpi_video_bus_probe(struct auxiliary_device *aux_dev, |
| 1982 | const struct auxiliary_device_id *id_unused) | ||
| 1984 | { | 1983 | { |
| 1985 | struct acpi_device *device = ACPI_COMPANION(&pdev->dev); | 1984 | struct acpi_device *device = ACPI_COMPANION(&aux_dev->dev); |
| 1986 | struct acpi_video_bus *video; | 1985 | struct acpi_video_bus *video; |
| 1987 | bool auto_detect; | 1986 | bool auto_detect; |
| 1988 | int error; | 1987 | int error; |
| @@ -2019,7 +2018,7 @@ static int acpi_video_bus_probe(struct platform_device *pdev) · 30 unmodified lines | |||
| 2019 | instance++; | 2018 | instance++; |
| 2020 | } | 2019 | } |
| 2021 | 2020 | ||
| 2022 | platform_set_drvdata(pdev, video); | 2021 | auxiliary_set_drvdata(aux_dev, video); |
| 2023 | 2022 | ||
| 2024 | video->device = device; | 2023 | video->device = device; |
| 2025 | strscpy(acpi_device_name(device), ACPI_VIDEO_BUS_NAME); | 2024 | strscpy(acpi_device_name(device), ACPI_VIDEO_BUS_NAME); |
| @@ -2068,7 +2067,7 @@ static int acpi_video_bus_probe(struct platform_device *pdev) · 42 unmodified lines | |||
| 2068 | !auto_detect) | 2067 | !auto_detect) |
| 2069 | acpi_video_bus_register_backlight(video); | 2068 | acpi_video_bus_register_backlight(video); |
| 2070 | 2069 | ||
| 2071 | error = acpi_video_bus_add_notify_handler(video, pdev); | 2070 | error = acpi_video_bus_add_notify_handler(video, &aux_dev->dev); |
| 2072 | if (error) | 2071 | if (error) |
| 2073 | goto err_del; | 2072 | goto err_del; |
| 2074 | 2073 | ||
| @@ -2096,10 +2095,10 @@ err_free_video: · 21 unmodified lines | |||
| 2096 | return error; | 2095 | return error; |
| 2097 | } | 2096 | } |
| 2098 | 2097 | ||
| 2099 | static void acpi_video_bus_remove(struct platform_device *pdev) | 2098 | static void acpi_video_bus_remove(struct auxiliary_device *aux_dev) |
| 2100 | { | 2099 | { |
| 2101 | struct acpi_video_bus *video = platform_get_drvdata(pdev); | 2100 | struct acpi_video_bus *video = auxiliary_get_drvdata(aux_dev); |
| 2102 | struct acpi_device *device = ACPI_COMPANION(&pdev->dev); | 2101 | struct acpi_device *device = ACPI_COMPANION(&aux_dev->dev); |
| 2103 | 2102 | ||
| 2104 | acpi_dev_remove_notify_handler(device, ACPI_DEVICE_NOTIFY, | 2103 | acpi_dev_remove_notify_handler(device, ACPI_DEVICE_NOTIFY, |
| 2105 | acpi_video_bus_notify); | 2104 | acpi_video_bus_notify); |
| @@ -2163,7 +2162,7 @@ int acpi_video_register(void) · 57 unmodified lines | |||
| 2163 | 2162 | ||
| 2164 | dmi_check_system(video_dmi_table); | 2163 | dmi_check_system(video_dmi_table); |
| 2165 | 2164 | ||
| 2166 | ret = platform_driver_register(&acpi_video_bus); | 2165 | ret = auxiliary_driver_register(&acpi_video_bus); |
| 2167 | if (ret) | 2166 | if (ret) |
| 2168 | goto leave; | 2167 | goto leave; |
| 2169 | 2168 | ||
| @@ -2183,7 +2182,7 @@ void acpi_video_unregister(void) · 13 unmodified lines | |||
| 2183 | { | 2182 | { |
| 2184 | mutex_lock(®ister_count_mutex); | 2183 | mutex_lock(®ister_count_mutex); |
| 2185 | if (register_count) { | 2184 | if (register_count) { |
| 2186 | platform_driver_unregister(&acpi_video_bus); | 2185 | auxiliary_driver_unregister(&acpi_video_bus); |
| 2187 | register_count = 0; | 2186 | register_count = 0; |
| 2188 | may_report_brightness_keys = false; | 2187 | may_report_brightness_keys = false; |
| 2189 | } | 2188 | } |
| @@ -451,7 +451,7 @@ const union acpi_predefined_info acpi_gbl_predefined_methods[] = { · 450 unmodified lines | |||
| 451 | 451 | ||
| 452 | {{"_DSM", | 452 | {{"_DSM", |
| 453 | METHOD_4ARGS(ACPI_TYPE_BUFFER, ACPI_TYPE_INTEGER, ACPI_TYPE_INTEGER, | 453 | METHOD_4ARGS(ACPI_TYPE_BUFFER, ACPI_TYPE_INTEGER, ACPI_TYPE_INTEGER, |
| 454 | ACPI_TYPE_ANY | ACPI_TYPE_PACKAGE) | | 454 | ACPI_TYPE_PACKAGE | ACPI_TYPE_ANY) | |
| 455 | ARG_COUNT_IS_MINIMUM, | 455 | ARG_COUNT_IS_MINIMUM, |
| 456 | METHOD_RETURNS(ACPI_RTYPE_ALL)}}, /* Must return a value, but it can be of any type */ | 456 | METHOD_RETURNS(ACPI_RTYPE_ALL)}}, /* Must return a value, but it can be of any type */ |
| 457 | 457 | ||
| @@ -818,9 +818,6 @@ const struct acpi_device *acpi_companion_match(const struct device *dev) · 817 unmodified lines | |||
| 818 | if (list_empty(&adev->pnp.ids)) | 818 | if (list_empty(&adev->pnp.ids)) |
| 819 | return NULL; | 819 | return NULL; |
| 820 | 820 | ||
| 821 | if (adev->pnp.type.backlight) | ||
| 822 | return adev; | ||
| 823 | |||
| 824 | return acpi_primary_dev_companion(adev, dev); | 821 | return acpi_primary_dev_companion(adev, dev); |
| 825 | } | 822 | } |
| 826 | 823 | ||
| @@ -1656,6 +1656,8 @@ static int acpi_ec_setup(struct acpi_ec *ec, struct acpi_device *device, bool ca · 1655 unmodified lines | |||
| 1656 | 1656 | ||
| 1657 | ret = ec_install_handlers(ec, device, call_reg); | 1657 | ret = ec_install_handlers(ec, device, call_reg); |
| 1658 | if (ret) { | 1658 | if (ret) { |
| 1659 | ec_remove_handlers(ec); | ||
| 1660 | |||
| 1659 | if (ec == first_ec) | 1661 | if (ec == first_ec) |
| 1660 | first_ec = NULL; | 1662 | first_ec = NULL; |
| 1661 | 1663 | ||
| @@ -1681,7 +1681,7 @@ acpi_status __init acpi_os_initialize(void) · 1680 unmodified lines | |||
| 1681 | * Use acpi_os_map_generic_address to pre-map the reset | 1681 | * Use acpi_os_map_generic_address to pre-map the reset |
| 1682 | * register if it's in system memory. | 1682 | * register if it's in system memory. |
| 1683 | */ | 1683 | */ |
| 1684 | void *rv; | 1684 | void __iomem *rv; |
| 1685 | 1685 | ||
| 1686 | rv = acpi_os_map_generic_address(&acpi_gbl_FADT.reset_register); | 1686 | rv = acpi_os_map_generic_address(&acpi_gbl_FADT.reset_register); |
| 1687 | pr_debug("%s: Reset register mapping %s\n", __func__, | 1687 | pr_debug("%s: Reset register mapping %s\n", __func__, |
| @@ -263,6 +263,13 @@ static int rimt_iommu_xlate(struct device *dev, struct acpi_rimt_node *node, u32 · 262 unmodified lines | |||
| 263 | if (!rimt_fwnode) | 263 | if (!rimt_fwnode) |
| 264 | return -EPROBE_DEFER; | 264 | return -EPROBE_DEFER; |
| 265 | 265 | ||
| 266 | /* | ||
| 267 | * EPROBE_DEFER ensures IOMMU is probed before the devices that | ||
| 268 | * depend on them. During shutdown, however, the IOMMU may be removed | ||
| 269 | * first, leading to issues. To avoid this, a device link is added | ||
| 270 | * which enforces the correct removal order. | ||
| 271 | */ | ||
| 272 | device_link_add(dev, rimt_fwnode->dev, DL_FLAG_AUTOREMOVE_CONSUMER); | ||
| 266 | return acpi_iommu_fwspec_init(dev, deviceid, rimt_fwnode); | 273 | return acpi_iommu_fwspec_init(dev, deviceid, rimt_fwnode); |
| 267 | } | 274 | } |
| 268 | 275 | ||
| @@ -6,6 +6,7 @@ · 5 unmodified lines | |||
| 6 | #define pr_fmt(fmt) "ACPI: " fmt | 6 | #define pr_fmt(fmt) "ACPI: " fmt |
| 7 | 7 | ||
| 8 | #include <linux/async.h> | 8 | #include <linux/async.h> |
| 9 | #include <linux/auxiliary_bus.h> | ||
| 9 | #include <linux/module.h> | 10 | #include <linux/module.h> |
| 10 | #include <linux/init.h> | 11 | #include <linux/init.h> |
| 11 | #include <linux/slab.h> | 12 | #include <linux/slab.h> |
| @@ -2192,6 +2193,44 @@ static acpi_status acpi_bus_check_add_2(acpi_handle handle, u32 lvl_not_used, · 2180 unmodified lines | |||
| 2192 | return acpi_bus_check_add(handle, false, (struct acpi_device **)ret_p); | 2193 | return acpi_bus_check_add(handle, false, (struct acpi_device **)ret_p); |
| 2193 | } | 2194 | } |
| 2194 | 2195 | ||
| 2196 | static void acpi_video_bus_device_release(struct device *dev) | ||
| 2197 | { | ||
| 2198 | struct auxiliary_device *aux_dev = to_auxiliary_dev(dev); | ||
| 2199 | |||
| 2200 | kfree(aux_dev); | ||
| 2201 | } | ||
| 2202 | |||
| 2203 | static void acpi_create_video_bus_device(struct acpi_device *adev, | ||
| 2204 | struct acpi_device *parent) | ||
| 2205 | { | ||
| 2206 | struct auxiliary_device *aux_dev; | ||
| 2207 | static unsigned int aux_dev_id; | ||
| 2208 | |||
| 2209 | aux_dev = kzalloc_obj(*aux_dev); | ||
| 2210 | if (!aux_dev) | ||
| 2211 | return; | ||
| 2212 | |||
| 2213 | aux_dev->id = aux_dev_id++; | ||
| 2214 | aux_dev->name = "video_bus"; | ||
| 2215 | aux_dev->dev.parent = acpi_get_first_physical_node(parent); | ||
| 2216 | if (!aux_dev->dev.parent) | ||
| 2217 | goto err; | ||
| 2218 | |||
| 2219 | aux_dev->dev.release = acpi_video_bus_device_release; | ||
| 2220 | |||
| 2221 | if (auxiliary_device_init(aux_dev)) | ||
| 2222 | goto err; | ||
| 2223 | |||
| 2224 | ACPI_COMPANION_SET(&aux_dev->dev, adev); | ||
| 2225 | if (__auxiliary_device_add(aux_dev, "acpi")) | ||
| 2226 | auxiliary_device_uninit(aux_dev); | ||
| 2227 | |||
| 2228 | return; | ||
| 2229 | |||
| 2230 | err: | ||
| 2231 | kfree(aux_dev); | ||
| 2232 | } | ||
| 2233 | |||
| 2195 | struct acpi_scan_system_dev { | 2234 | struct acpi_scan_system_dev { |
| 2196 | struct list_head node; | 2235 | struct list_head node; |
| 2197 | struct acpi_device *adev; | 2236 | struct acpi_device *adev; |
| @@ -2229,6 +2268,12 @@ static void acpi_default_enumeration(struct acpi_device *device) · 31 unmodified lines | |||
| 2229 | sd->adev = device; | 2268 | sd->adev = device; |
| 2230 | list_add_tail(&sd->node, &acpi_scan_system_dev_list); | 2269 | list_add_tail(&sd->node, &acpi_scan_system_dev_list); |
| 2231 | } | 2270 | } |
| 2271 | } else if (device->pnp.type.backlight) { | ||
| 2272 | struct acpi_device *parent; | ||
| 2273 | |||
| 2274 | parent = acpi_dev_parent(device); | ||
| 2275 | if (parent) | ||
| 2276 | acpi_create_video_bus_device(device, parent); | ||
| 2232 | } else { | 2277 | } else { |
| 2233 | /* For a regular device object, create a platform device. */ | 2278 | /* For a regular device object, create a platform device. */ |
| 2234 | acpi_create_platform_device(device, NULL); | 2279 | acpi_create_platform_device(device, NULL); |
| @@ -13,6 +13,8 @@ · 12 unmodified lines | |||
| 13 | // | 13 | // |
| 14 | // The shrinker will use trylock methods because it locks them in a different order. | 14 | // The shrinker will use trylock methods because it locks them in a different order. |
| 15 | 15 | ||
| 16 | use crate::AssertSync; | ||
| 17 | |||
| 16 | use core::{ | 18 | use core::{ |
| 17 | marker::PhantomPinned, | 19 | marker::PhantomPinned, |
| 18 | mem::{size_of, size_of_val, MaybeUninit}, | 20 | mem::{size_of, size_of_val, MaybeUninit}, |
| @@ -142,6 +144,30 @@ pub(crate) struct ShrinkablePageRange { · 123 unmodified lines | |||
| 142 | _pin: PhantomPinned, | 144 | _pin: PhantomPinned, |
| 143 | } | 145 | } |
| 144 | 146 | ||
| 147 | // We do not define any ops. For now, used only to check identity of vmas. | ||
| 148 | static BINDER_VM_OPS: AssertSync<bindings::vm_operations_struct> = AssertSync(pin_init::zeroed()); | ||
| 149 | |||
| 150 | // To ensure that we do not accidentally install pages into or zap pages from the wrong vma, we | ||
| 151 | // check its vm_ops and private data before using it. | ||
| 152 | fn check_vma(vma: &virt::VmaRef, owner: *const ShrinkablePageRange) -> Option<&virt::VmaMixedMap> { | ||
| 153 | // SAFETY: Just reading the vm_ops pointer of any active vma is safe. | ||
| 154 | let vm_ops = unsafe { (*vma.as_ptr()).vm_ops }; | ||
| 155 | if !ptr::eq(vm_ops, &BINDER_VM_OPS.0) { | ||
| 156 | return None; | ||
| 157 | } | ||
| 158 | |||
| 159 | // SAFETY: Reading the vm_private_data pointer of a binder-owned vma is safe. | ||
| 160 | let vm_private_data = unsafe { (*vma.as_ptr()).vm_private_data }; | ||
| 161 | // The ShrinkablePageRange is only dropped when the Process is dropped, which only happens once | ||
| 162 | // the file's ->release handler is invoked, which means the ShrinkablePageRange outlives any | ||
| 163 | // VMA associated with it, so there can't be any false positives due to pointer reuse here. | ||
| 164 | if !ptr::eq(vm_private_data, owner.cast()) { | ||
| 165 | return None; | ||
| 166 | } | ||
| 167 | |||
| 168 | vma.as_mixedmap_vma() | ||
| 169 | } | ||
| 170 | |||
| 145 | struct Inner { | 171 | struct Inner { |
| 146 | /// Array of pages. | 172 | /// Array of pages. |
| 147 | /// | 173 | /// |
| @@ -308,6 +334,18 @@ impl ShrinkablePageRange { · 160 unmodified lines | |||
| 308 | inner.size = num_pages; | 334 | inner.size = num_pages; |
| 309 | inner.vma_addr = vma.start(); | 335 | inner.vma_addr = vma.start(); |
| 310 | 336 | ||
| 337 | // This pointer is only used for comparison - it's not dereferenced. | ||
| 338 | // | ||
| 339 | // SAFETY: We own the vma, and we don't use any methods on VmaNew that rely on | ||
| 340 | // `vm_private_data`. | ||
| 341 | unsafe { | ||
| 342 | (*vma.as_ptr()).vm_private_data = ptr::from_ref(self).cast_mut().cast::<c_void>() | ||
| 343 | }; | ||
| 344 | |||
| 345 | // SAFETY: We own the vma, and we don't use any methods on VmaNew that rely on | ||
| 346 | // `vm_ops`. | ||
| 347 | unsafe { (*vma.as_ptr()).vm_ops = &BINDER_VM_OPS.0 }; | ||
| 348 | |||
| 311 | Ok(num_pages) | 349 | Ok(num_pages) |
| 312 | } | 350 | } |
| 313 | 351 | ||
| @@ -399,22 +437,25 @@ impl ShrinkablePageRange { · 85 unmodified lines | |||
| 399 | // | 437 | // |
| 400 | // Using `mmput_async` avoids this, because then the `mm` cleanup is instead queued to a | 438 | // Using `mmput_async` avoids this, because then the `mm` cleanup is instead queued to a |
| 401 | // workqueue. | 439 | // workqueue. |
| 402 | MmWithUser::into_mmput_async(self.mm.mmget_not_zero().ok_or(ESRCH)?) | 440 | let mm = MmWithUser::into_mmput_async(self.mm.mmget_not_zero().ok_or(ESRCH)?); |
| 403 | .mmap_read_lock() | 441 | { |
| 404 | .vma_lookup(vma_addr) | 442 | let vma_read; |
| 405 | .ok_or(ESRCH)? | 443 | let mmap_read; |
| 406 | .as_mixedmap_vma() | 444 | let vma = if let Some(ret) = mm.lock_vma_under_rcu(vma_addr) { |
| 407 | .ok_or(ESRCH)? | 445 | vma_read = ret; |
| 408 | .vm_insert_page(user_page_addr, &new_page) | 446 | check_vma(&vma_read, self) |
| 409 | .inspect_err(|err| { | 447 | } else { |
| 410 | pr_warn!( | 448 | mmap_read = mm.mmap_read_lock(); |
| 411 | "Failed to vm_insert_page({}): vma_addr:{} i:{} err:{:?}", | 449 | mmap_read |
| 412 | user_page_addr, | 450 | .vma_lookup(vma_addr) |
| 413 | vma_addr, | 451 | .and_then(|vma| check_vma(vma, self)) |
| 414 | i, | 452 | }; |
| 415 | err | 453 | |
| 416 | ) | 454 | match vma { |
| 417 | })?; | 455 | Some(vma) => vma.vm_insert_page(user_page_addr, &new_page)?, |
| 456 | None => return Err(ESRCH), | ||
| 457 | } | ||
| 458 | } | ||
| 418 | 459 | ||
| 419 | let inner = self.lock.lock(); | 460 | let inner = self.lock.lock(); |
| 420 | 461 | ||
| @@ -667,12 +708,15 @@ unsafe extern "C" fn rust_shrink_free_page( · 246 unmodified lines | |||
| 667 | let mmap_read; | 708 | let mmap_read; |
| 668 | let mm_mutex; | 709 | let mm_mutex; |
| 669 | let vma_addr; | 710 | let vma_addr; |
| 711 | let range_ptr; | ||
| 670 | 712 | ||
| 671 | { | 713 | { |
| 672 | // CAST: The `list_head` field is first in `PageInfo`. | 714 | // CAST: The `list_head` field is first in `PageInfo`. |
| 673 | let info = item as *mut PageInfo; | 715 | let info = item as *mut PageInfo; |
| 674 | // SAFETY: The `range` field of `PageInfo` is immutable. | 716 | // SAFETY: The `range` field of `PageInfo` is immutable. |
| 675 | let range = unsafe { &*((*info).range) }; | 717 | range_ptr = unsafe { (*info).range }; |
| 718 | // SAFETY: The `range` outlives its `PageInfo` values. | ||
| 719 | let range = unsafe { &*range_ptr }; | ||
| 676 | 720 | ||
| 677 | mm = match range.mm.mmget_not_zero() { | 721 | mm = match range.mm.mmget_not_zero() { |
| 678 | Some(mm) => MmWithUser::into_mmput_async(mm), | 722 | Some(mm) => MmWithUser::into_mmput_async(mm), |
| @@ -717,9 +761,11 @@ unsafe extern "C" fn rust_shrink_free_page( · 38 unmodified lines | |||
| 717 | // SAFETY: The lru lock is locked when this method is called. | 761 | // SAFETY: The lru lock is locked when this method is called. |
| 718 | unsafe { bindings::spin_unlock(&raw mut (*lru).lock) }; | 762 | unsafe { bindings::spin_unlock(&raw mut (*lru).lock) }; |
| 719 | 763 | ||
| 720 | if let Some(vma) = mmap_read.vma_lookup(vma_addr) { | 764 | if let Some(unchecked_vma) = mmap_read.vma_lookup(vma_addr) { |
| 721 | let user_page_addr = vma_addr + (page_index << PAGE_SHIFT); | 765 | if let Some(vma) = check_vma(unchecked_vma, range_ptr) { |
| 722 | vma.zap_page_range_single(user_page_addr, PAGE_SIZE); | 766 | let user_page_addr = vma_addr + (page_index << PAGE_SHIFT); |
| 767 | vma.zap_page_range_single(user_page_addr, PAGE_SIZE); | ||
| 768 | } | ||
| 723 | } | 769 | } |
| 724 | 770 | ||
| 725 | drop(mmap_read); | 771 | drop(mmap_read); |
| @@ -1295,7 +1295,8 @@ impl Process { · 1294 unmodified lines | |||
| 1295 | } | 1295 | } |
| 1296 | 1296 | ||
| 1297 | pub(crate) fn dead_binder_done(&self, cookie: u64, thread: &Thread) { | 1297 | pub(crate) fn dead_binder_done(&self, cookie: u64, thread: &Thread) { |
| 1298 | if let Some(death) = self.inner.lock().pull_delivered_death(cookie) { | 1298 | let death = self.inner.lock().pull_delivered_death(cookie); |
| 1299 | if let Some(death) = death { | ||
| 1299 | death.set_notification_done(thread); | 1300 | death.set_notification_done(thread); |
| 1300 | } | 1301 | } |
| 1301 | } | 1302 | } |
| @@ -118,7 +118,7 @@ impl<T> ArrayRangeAllocator<T> { · 117 unmodified lines | |||
| 118 | size: usize, | 118 | size: usize, |
| 119 | is_oneway: bool, | 119 | is_oneway: bool, |
| 120 | pid: Pid, | 120 | pid: Pid, |
| 121 | ) -> Result<usize> { | 121 | ) -> Result<(usize, bool)> { |
| 122 | // Compute new value of free_oneway_space, which is set only on success. | 122 | // Compute new value of free_oneway_space, which is set only on success. |
| 123 | let new_oneway_space = if is_oneway { | 123 | let new_oneway_space = if is_oneway { |
| 124 | match self.free_oneway_space.checked_sub(size) { | 124 | match self.free_oneway_space.checked_sub(size) { |
| @@ -146,7 +146,38 @@ impl<T> ArrayRangeAllocator<T> { · 21 unmodified lines | |||
| 146 | .ok() | 146 | .ok() |
| 147 | .unwrap(); | 147 | .unwrap(); |
| 148 | 148 | ||
| 149 | Ok(insert_at_offset) | 149 | // Start detecting spammers once we have less than 20% |
| 150 | // of async space left (which is less than 10% of total | ||
| 151 | // buffer size). | ||
| 152 | // | ||
| 153 | // (This will short-circuit, so `low_oneway_space` is | ||
| 154 | // only called when necessary.) | ||
| 155 | let oneway_spam_detected = | ||
| 156 | is_oneway && new_oneway_space < self.size / 10 && self.low_oneway_space(pid); | ||
| 157 | |||
| 158 | Ok((insert_at_offset, oneway_spam_detected)) | ||
| 159 | } | ||
| 160 | |||
| 161 | /// Find the amount and size of buffers allocated by the current caller. | ||
| 162 | /// | ||
| 163 | /// The idea is that once we cross the threshold, whoever is responsible | ||
| 164 | /// for the low async space is likely to try to send another async transaction, | ||
| 165 | /// and at some point we'll catch them in the act. This is more efficient | ||
| 166 | /// than keeping a map per pid. | ||
| 167 | fn low_oneway_space(&self, calling_pid: Pid) -> bool { | ||
| 168 | let mut total_alloc_size = 0; | ||
| 169 | let mut num_buffers = 0; | ||
| 170 | |||
| 171 | // Warn if this pid has more than 50 transactions, or more than 50% of | ||
| 172 | // async space (which is 25% of total buffer size). Oneway spam is only | ||
| 173 | // detected when the threshold is exceeded. | ||
| 174 | for range in &self.ranges { | ||
| 175 | if range.state.is_oneway() && range.state.pid() == calling_pid { | ||
| 176 | total_alloc_size += range.size; | ||
| 177 | num_buffers += 1; | ||
| 178 | } | ||
| 179 | } | ||
| 180 | num_buffers > 50 || total_alloc_size > self.size / 4 | ||
| 150 | } | 181 | } |
| 151 | 182 | ||
| 152 | pub(crate) fn reservation_abort(&mut self, offset: usize) -> Result<FreedRange> { | 183 | pub(crate) fn reservation_abort(&mut self, offset: usize) -> Result<FreedRange> { |
| @@ -188,11 +188,11 @@ impl<T> RangeAllocator<T> { · 187 unmodified lines | |||
| 188 | self.reserve_new(args) | 188 | self.reserve_new(args) |
| 189 | } | 189 | } |
| 190 | Impl::Array(array) => { | 190 | Impl::Array(array) => { |
| 191 | let offset = | 191 | let (offset, oneway_spam_detected) = |
| 192 | array.reserve_new(args.debug_id, args.size, args.is_oneway, args.pid)?; | 192 | array.reserve_new(args.debug_id, args.size, args.is_oneway, args.pid)?; |
| 193 | Ok(ReserveNew::Success(ReserveNewSuccess { | 193 | Ok(ReserveNew::Success(ReserveNewSuccess { |
| 194 | offset, | 194 | offset, |
| 195 | oneway_spam_detected: false, | 195 | oneway_spam_detected, |
| 196 | _empty_array_alloc: args.empty_array_alloc, | 196 | _empty_array_alloc: args.empty_array_alloc, |
| 197 | _new_tree_alloc: args.new_tree_alloc, | 197 | _new_tree_alloc: args.new_tree_alloc, |
| 198 | _tree_alloc: args.tree_alloc, | 198 | _tree_alloc: args.tree_alloc, |
| @@ -164,15 +164,6 @@ impl<T> TreeRangeAllocator<T> { · 163 unmodified lines | |||
| 164 | self.free_oneway_space | 164 | self.free_oneway_space |
| 165 | }; | 165 | }; |
| 166 | 166 | ||
| 167 | // Start detecting spammers once we have less than 20% | ||
| 168 | // of async space left (which is less than 10% of total | ||
| 169 | // buffer size). | ||
| 170 | // | ||
| 171 | // (This will short-circut, so `low_oneway_space` is | ||
| 172 | // only called when necessary.) | ||
| 173 | let oneway_spam_detected = | ||
| 174 | is_oneway && new_oneway_space < self.size / 10 && self.low_oneway_space(pid); | ||
| 175 | |||
| 176 | let (found_size, found_off, tree_node, free_tree_node) = match self.find_best_match(size) { | 167 | let (found_size, found_off, tree_node, free_tree_node) = match self.find_best_match(size) { |
| 177 | None => { | 168 | None => { |
| 178 | pr_warn!("ENOSPC from range_alloc.reserve_new - size: {}", size); | 169 | pr_warn!("ENOSPC from range_alloc.reserve_new - size: {}", size); |
| @@ -203,6 +194,15 @@ impl<T> TreeRangeAllocator<T> { · 24 unmodified lines | |||
| 203 | self.free_tree.insert(free_tree_node); | 194 | self.free_tree.insert(free_tree_node); |
| 204 | } | 195 | } |
| 205 | 196 | ||
| 197 | // Start detecting spammers once we have less than 20% | ||
| 198 | // of async space left (which is less than 10% of total | ||
| 199 | // buffer size). | ||
| 200 | // | ||
| 201 | // (This will short-circuit, so `low_oneway_space` is | ||
| 202 | // only called when necessary.) | ||
| 203 | let oneway_spam_detected = | ||
| 204 | is_oneway && new_oneway_space < self.size / 10 && self.low_oneway_space(pid); | ||
| 205 | |||
| 206 | Ok((found_off, oneway_spam_detected)) | 206 | Ok((found_off, oneway_spam_detected)) |
| 207 | } | 207 | } |
| 208 | 208 | ||
| @@ -306,7 +306,7 @@ impl kernel::Module for BinderModule { · 305 unmodified lines | |||
| 306 | /// Makes the inner type Sync. | 306 | /// Makes the inner type Sync. |
| 307 | #[repr(transparent)] | 307 | #[repr(transparent)] |
| 308 | pub struct AssertSync<T>(T); | 308 | pub struct AssertSync<T>(T); |
| 309 | // SAFETY: Used only to insert `file_operations` into a global, which is safe. | 309 | // SAFETY: Used only to insert C bindings types into globals, which is safe. |
| 310 | unsafe impl<T> Sync for AssertSync<T> {} | 310 | unsafe impl<T> Sync for AssertSync<T> {} |
| 311 | 311 | ||
| 312 | /// File operations that rust_binderfs.c can use. | 312 | /// File operations that rust_binderfs.c can use. |
| @@ -1015,12 +1015,9 @@ impl Thread { · 1014 unmodified lines | |||
| 1015 | 1015 | ||
| 1016 | // Copy offsets if there are any. | 1016 | // Copy offsets if there are any. |
| 1017 | if offsets_size > 0 { | 1017 | if offsets_size > 0 { |
| 1018 | { | 1018 | let mut offsets_reader = |
| 1019 | let mut reader = | 1019 | UserSlice::new(UserPtr::from_addr(trd_data_ptr.offsets as _), offsets_size) |
| 1020 | UserSlice::new(UserPtr::from_addr(trd_data_ptr.offsets as _), offsets_size) | 1020 | .reader(); |
| 1021 | .reader(); | ||
| 1022 | alloc.copy_into(&mut reader, aligned_data_size, offsets_size)?; | ||
| 1023 | } | ||
| 1024 | 1021 | ||
| 1025 | let offsets_start = aligned_data_size; | 1022 | let offsets_start = aligned_data_size; |
| 1026 | let offsets_end = aligned_data_size + offsets_size; | 1023 | let offsets_end = aligned_data_size + offsets_size; |
| @@ -1041,11 +1038,9 @@ impl Thread { · 14 unmodified lines | |||
| 1041 | .step_by(size_of::<u64>()) | 1038 | .step_by(size_of::<u64>()) |
| 1042 | .enumerate() | 1039 | .enumerate() |
| 1043 | { | 1040 | { |
| 1044 | let offset: usize = view | 1041 | let offset = offsets_reader.read::<u64>()?; |
| 1045 | .alloc | 1042 | view.alloc.write(index_offset, &offset)?; |
| 1046 | .read::<u64>(index_offset)? | 1043 | let offset: usize = offset.try_into().map_err(|_| EINVAL)?; |
| 1047 | .try_into() | ||
| 1048 | .map_err(|_| EINVAL)?; | ||
| 1049 | 1044 | ||
| 1050 | if offset < end_of_previous_object || !is_aligned(offset, size_of::<u32>()) { | 1045 | if offset < end_of_previous_object || !is_aligned(offset, size_of::<u32>()) { |
| 1051 | pr_warn!("Got transaction with invalid offset."); | 1046 | pr_warn!("Got transaction with invalid offset."); |
| @@ -4188,6 +4188,9 @@ static const struct ata_dev_quirks_entry __ata_dev_quirks[] = { · 4187 unmodified lines | |||
| 4188 | { "ST3320[68]13AS", "SD1[5-9]", ATA_QUIRK_NONCQ | | 4188 | { "ST3320[68]13AS", "SD1[5-9]", ATA_QUIRK_NONCQ | |
| 4189 | ATA_QUIRK_FIRMWARE_WARN }, | 4189 | ATA_QUIRK_FIRMWARE_WARN }, |
| 4190 | 4190 | ||
| 4191 | /* ADATA devices with LPM issues. */ | ||
| 4192 | { "ADATA SU680", NULL, ATA_QUIRK_NOLPM }, | ||
| 4193 | |||
| 4191 | /* Seagate disks with LPM issues */ | 4194 | /* Seagate disks with LPM issues */ |
| 4192 | { "ST1000DM010-2EP102", NULL, ATA_QUIRK_NOLPM }, | 4195 | { "ST1000DM010-2EP102", NULL, ATA_QUIRK_NOLPM }, |
| 4193 | { "ST2000DM008-2FR102", NULL, ATA_QUIRK_NOLPM }, | 4196 | { "ST2000DM008-2FR102", NULL, ATA_QUIRK_NOLPM }, |
| @@ -3600,7 +3600,7 @@ static unsigned int ata_scsiop_maint_in(struct ata_device *dev, · 3599 unmodified lines | |||
| 3600 | 3600 | ||
| 3601 | if (cdb[2] != 1 && cdb[2] != 3) { | 3601 | if (cdb[2] != 1 && cdb[2] != 3) { |
| 3602 | ata_dev_warn(dev, "invalid command format %d\n", cdb[2]); | 3602 | ata_dev_warn(dev, "invalid command format %d\n", cdb[2]); |
| 3603 | ata_scsi_set_invalid_field(dev, cmd, 1, 0xff); | 3603 | ata_scsi_set_invalid_field(dev, cmd, 2, 0xff); |
| 3604 | return 0; | 3604 | return 0; |
| 3605 | } | 3605 | } |
| 3606 | 3606 | ||
| @@ -99,8 +99,13 @@ static int lcd2s_print(struct charlcd *lcd, int c) · 98 unmodified lines | |||
| 99 | { | 99 | { |
| 100 | struct lcd2s_data *lcd2s = lcd->drvdata; | 100 | struct lcd2s_data *lcd2s = lcd->drvdata; |
| 101 | u8 buf[2] = { LCD2S_CMD_WRITE, c }; | 101 | u8 buf[2] = { LCD2S_CMD_WRITE, c }; |
| 102 | int ret; | ||
| 102 | 103 | ||
| 103 | lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); | 104 | ret = lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); |
| 105 | if (ret < 0) | ||
| 106 | return ret; | ||
| 107 | if (ret != sizeof(buf)) | ||
| 108 | return -EIO; | ||
| 104 | return 0; | 109 | return 0; |
| 105 | } | 110 | } |
| 106 | 111 | ||
| @@ -108,9 +113,13 @@ static int lcd2s_gotoxy(struct charlcd *lcd, unsigned int x, unsigned int y) · 1 unmodified line | |||
| 108 | { | 113 | { |
| 109 | struct lcd2s_data *lcd2s = lcd->drvdata; | 114 | struct lcd2s_data *lcd2s = lcd->drvdata; |
| 110 | u8 buf[3] = { LCD2S_CMD_CUR_POS, y + 1, x + 1 }; | 115 | u8 buf[3] = { LCD2S_CMD_CUR_POS, y + 1, x + 1 }; |
| 116 | int ret; | ||
| 111 | 117 | ||
| 112 | lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); | 118 | ret = lcd2s_i2c_master_send(lcd2s->i2c, buf, sizeof(buf)); |
| 113 | 119 | if (ret < 0) | |
| 120 | return ret; | ||
| 121 | if (ret != sizeof(buf)) | ||
| 122 | return -EIO; | ||
| 114 | return 0; | 123 | return 0; |
| 115 | } | 124 | } |
| 116 | 125 | ||
| @@ -365,7 +365,7 @@ static DEFINE_IDA(linedisp_id); · 364 unmodified lines | |||
| 365 | 365 | ||
| 366 | static void linedisp_release(struct device *dev) | 366 | static void linedisp_release(struct device *dev) |
| 367 | { | 367 | { |
| 368 | struct linedisp *linedisp = to_linedisp(dev); | 368 | struct linedisp *linedisp = container_of(dev, struct linedisp, dev); |
| 369 | 369 | ||
| 370 | kfree(linedisp->map); | 370 | kfree(linedisp->map); |
| 371 | kfree(linedisp->message); | 371 | kfree(linedisp->message); |
| @@ -504,6 +504,36 @@ int bus_for_each_drv(const struct bus_type *bus, struct device_driver *start, · 503 unmodified lines | |||
| 504 | } | 504 | } |
| 505 | EXPORT_SYMBOL_GPL(bus_for_each_drv); | 505 | EXPORT_SYMBOL_GPL(bus_for_each_drv); |
| 506 | 506 | ||
| 507 | static ssize_t driver_override_store(struct device *dev, | ||
| 508 | struct device_attribute *attr, | ||
| 509 | const char *buf, size_t count) | ||
| 510 | { | ||
| 511 | int ret; | ||
| 512 | |||
| 513 | ret = __device_set_driver_override(dev, buf, count); | ||
| 514 | if (ret) | ||
| 515 | return ret; | ||
| 516 | |||
| 517 | return count; | ||
| 518 | } | ||
| 519 | |||
| 520 | static ssize_t driver_override_show(struct device *dev, | ||
| 521 | struct device_attribute *attr, char *buf) | ||
| 522 | { | ||
| 523 | guard(spinlock)(&dev->driver_override.lock); | ||
| 524 | return sysfs_emit(buf, "%s\n", dev->driver_override.name); | ||
| 525 | } | ||
| 526 | static DEVICE_ATTR_RW(driver_override); | ||
| 527 | |||
| 528 | static struct attribute *driver_override_dev_attrs[] = { | ||
| 529 | &dev_attr_driver_override.attr, | ||
| 530 | NULL, | ||
| 531 | }; | ||
| 532 | |||
| 533 | static const struct attribute_group driver_override_dev_group = { | ||
| 534 | .attrs = driver_override_dev_attrs, | ||
| 535 | }; | ||
| 536 | |||
| 507 | /** | 537 | /** |
| 508 | * bus_add_device - add device to bus | 538 | * bus_add_device - add device to bus |
| 509 | * @dev: device being added | 539 | * @dev: device being added |
| @@ -537,9 +567,15 @@ int bus_add_device(struct device *dev) · 27 unmodified lines | |||
| 537 | if (error) | 567 | if (error) |
| 538 | goto out_put; | 568 | goto out_put; |
| 539 | 569 | ||
| 570 | if (dev->bus->driver_override) { | ||
| 571 | error = device_add_group(dev, &driver_override_dev_group); | ||
| 572 | if (error) | ||
| 573 | goto out_groups; | ||
| 574 | } | ||
| 575 | |||
| 540 | error = sysfs_create_link(&sp->devices_kset->kobj, &dev->kobj, dev_name(dev)); | 576 | error = sysfs_create_link(&sp->devices_kset->kobj, &dev->kobj, dev_name(dev)); |
| 541 | if (error) | 577 | if (error) |
| 542 | goto out_groups; | 578 | goto out_override; |
| 543 | 579 | ||
| 544 | error = sysfs_create_link(&dev->kobj, &sp->subsys.kobj, "subsystem"); | 580 | error = sysfs_create_link(&dev->kobj, &sp->subsys.kobj, "subsystem"); |
| 545 | if (error) | 581 | if (error) |
| @@ -550,6 +586,9 @@ int bus_add_device(struct device *dev) · 4 unmodified lines | |||
| 550 | 586 | ||
| 551 | out_subsys: | 587 | out_subsys: |
| 552 | sysfs_remove_link(&sp->devices_kset->kobj, dev_name(dev)); | 588 | sysfs_remove_link(&sp->devices_kset->kobj, dev_name(dev)); |
| 589 | out_override: | ||
| 590 | if (dev->bus->driver_override) | ||
| 591 | device_remove_group(dev, &driver_override_dev_group); | ||
| 553 | out_groups: | 592 | out_groups: |
| 554 | device_remove_groups(dev, sp->bus->dev_groups); | 593 | device_remove_groups(dev, sp->bus->dev_groups); |
| 555 | out_put: | 594 | out_put: |
| @@ -607,6 +646,8 @@ void bus_remove_device(struct device *dev) · 51 unmodified lines | |||
| 607 | 646 | ||
| 608 | sysfs_remove_link(&dev->kobj, "subsystem"); | 647 | sysfs_remove_link(&dev->kobj, "subsystem"); |
| 609 | sysfs_remove_link(&sp->devices_kset->kobj, dev_name(dev)); | 648 | sysfs_remove_link(&sp->devices_kset->kobj, dev_name(dev)); |
| 649 | if (dev->bus->driver_override) | ||
| 650 | device_remove_group(dev, &driver_override_dev_group); | ||
| 610 | device_remove_groups(dev, dev->bus->dev_groups); | 651 | device_remove_groups(dev, dev->bus->dev_groups); |
| 611 | if (klist_node_attached(&dev->p->knode_bus)) | 652 | if (klist_node_attached(&dev->p->knode_bus)) |
| 612 | klist_del(&dev->p->knode_bus); | 653 | klist_del(&dev->p->knode_bus); |
| @@ -2556,6 +2556,7 @@ static void device_release(struct kobject *kobj) · 2555 unmodified lines | |||
| 2556 | devres_release_all(dev); | 2556 | devres_release_all(dev); |
| 2557 | 2557 | ||
| 2558 | kfree(dev->dma_range_map); | 2558 | kfree(dev->dma_range_map); |
| 2559 | kfree(dev->driver_override.name); | ||
| 2559 | 2560 | ||
| 2560 | if (dev->release) | 2561 | if (dev->release) |
| 2561 | dev->release(dev); | 2562 | dev->release(dev); |
| @@ -3159,6 +3160,7 @@ void device_initialize(struct device *dev) · 597 unmodified lines | |||
| 3159 | kobject_init(&dev->kobj, &device_ktype); | 3160 | kobject_init(&dev->kobj, &device_ktype); |
| 3160 | INIT_LIST_HEAD(&dev->dma_pools); | 3161 | INIT_LIST_HEAD(&dev->dma_pools); |
| 3161 | mutex_init(&dev->mutex); | 3162 | mutex_init(&dev->mutex); |
| 3163 | spin_lock_init(&dev->driver_override.lock); | ||
| 3162 | lockdep_set_novalidate_class(&dev->mutex); | 3164 | lockdep_set_novalidate_class(&dev->mutex); |
| 3163 | spin_lock_init(&dev->devres_lock); | 3165 | spin_lock_init(&dev->devres_lock); |
| 3164 | INIT_LIST_HEAD(&dev->devres_head); | 3166 | INIT_LIST_HEAD(&dev->devres_head); |
| @@ -381,6 +381,66 @@ static void __exit deferred_probe_exit(void) · 380 unmodified lines | |||
| 381 | } | 381 | } |
| 382 | __exitcall(deferred_probe_exit); | 382 | __exitcall(deferred_probe_exit); |
| 383 | 383 | ||
| 384 | int __device_set_driver_override(struct device *dev, const char *s, size_t len) | ||
| 385 | { | ||
| 386 | const char *new, *old; | ||
| 387 | char *cp; | ||
| 388 | |||
| 389 | if (!s) | ||
| 390 | return -EINVAL; | ||
| 391 | |||
| 392 | /* | ||
| 393 | * The stored value will be used in sysfs show callback (sysfs_emit()), | ||
| 394 | * which has a length limit of PAGE_SIZE and adds a trailing newline. | ||
| 395 | * Thus we can store one character less to avoid truncation during sysfs | ||
| 396 | * show. | ||
| 397 | */ | ||
| 398 | if (len >= (PAGE_SIZE - 1)) | ||
| 399 | return -EINVAL; | ||
| 400 | |||
| 401 | /* | ||
| 402 | * Compute the real length of the string in case userspace sends us a | ||
| 403 | * bunch of \0 characters like python likes to do. | ||
| 404 | */ | ||
| 405 | len = strlen(s); | ||
| 406 | |||
| 407 | if (!len) { | ||
| 408 | /* Empty string passed - clear override */ | ||
| 409 | spin_lock(&dev->driver_override.lock); | ||
| 410 | old = dev->driver_override.name; | ||
| 411 | dev->driver_override.name = NULL; | ||
| 412 | spin_unlock(&dev->driver_override.lock); | ||
| 413 | kfree(old); | ||
| 414 | |||
| 415 | return 0; | ||
| 416 | } | ||
| 417 | |||
| 418 | cp = strnchr(s, len, '\n'); | ||
| 419 | if (cp) | ||
| 420 | len = cp - s; | ||
| 421 | |||
| 422 | new = kstrndup(s, len, GFP_KERNEL); | ||
| 423 | if (!new) | ||
| 424 | return -ENOMEM; | ||
| 425 | |||
| 426 | spin_lock(&dev->driver_override.lock); | ||
| 427 | old = dev->driver_override.name; | ||
| 428 | if (cp != s) { | ||
| 429 | dev->driver_override.name = new; | ||
| 430 | spin_unlock(&dev->driver_override.lock); | ||
| 431 | } else { | ||
| 432 | /* "\n" passed - clear override */ | ||
| 433 | dev->driver_override.name = NULL; | ||
| 434 | spin_unlock(&dev->driver_override.lock); | ||
| 435 | |||
| 436 | kfree(new); | ||
| 437 | } | ||
| 438 | kfree(old); | ||
| 439 | |||
| 440 | return 0; | ||
| 441 | } | ||
| 442 | EXPORT_SYMBOL_GPL(__device_set_driver_override); | ||
| 443 | |||
| 384 | /** | 444 | /** |
| 385 | * device_is_bound() - Check if device is bound to a driver | 445 | * device_is_bound() - Check if device is bound to a driver |
| 386 | * @dev: device to check | 446 | * @dev: device to check |
| @@ -523,6 +523,8 @@ static ssize_t node_read_meminfo(struct device *dev, · 522 unmodified lines | |||
| 523 | #ifdef CONFIG_UNACCEPTED_MEMORY | 523 | #ifdef CONFIG_UNACCEPTED_MEMORY |
| 524 | "Node %d Unaccepted: %8lu kB\n" | 524 | "Node %d Unaccepted: %8lu kB\n" |
| 525 | #endif | 525 | #endif |
| 526 | "Node %d GPUActive: %8lu kB\n" | ||
| 527 | "Node %d GPUReclaim: %8lu kB\n" | ||
| 526 | , | 528 | , |
| 527 | nid, K(node_page_state(pgdat, NR_FILE_DIRTY)), | 529 | nid, K(node_page_state(pgdat, NR_FILE_DIRTY)), |
| 528 | nid, K(node_page_state(pgdat, NR_WRITEBACK)), | 530 | nid, K(node_page_state(pgdat, NR_WRITEBACK)), |
| @@ -556,6 +558,9 @@ static ssize_t node_read_meminfo(struct device *dev, · 27 unmodified lines | |||
| 556 | , | 558 | , |
| 557 | nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED)) | 559 | nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED)) |
| 558 | #endif | 560 | #endif |
| 561 | , | ||
| 562 | nid, K(node_page_state(pgdat, NR_GPU_ACTIVE)), | ||
| 563 | nid, K(node_page_state(pgdat, NR_GPU_RECLAIM)) | ||
| 559 | ); | 564 | ); |
| 560 | len += hugetlb_report_node_meminfo(buf, len, nid); | 565 | len += hugetlb_report_node_meminfo(buf, len, nid); |
| 561 | return len; | 566 | return len; |
| @@ -603,7 +603,6 @@ static void platform_device_release(struct device *dev) · 602 unmodified lines | |||
| 603 | kfree(pa->pdev.dev.platform_data); | 603 | kfree(pa->pdev.dev.platform_data); |
| 604 | kfree(pa->pdev.mfd_cell); | 604 | kfree(pa->pdev.mfd_cell); |
| 605 | kfree(pa->pdev.resource); | 605 | kfree(pa->pdev.resource); |
| 606 | kfree(pa->pdev.driver_override); | ||
| 607 | kfree(pa); | 606 | kfree(pa); |
| 608 | } | 607 | } |
| 609 | 608 | ||
| @@ -1306,38 +1305,9 @@ static ssize_t numa_node_show(struct device *dev, · 696 unmodified lines | |||
| 1306 | } | 1305 | } |
| 1307 | static DEVICE_ATTR_RO(numa_node); | 1306 | static DEVICE_ATTR_RO(numa_node); |
| 1308 | 1307 | ||
| 1309 | static ssize_t driver_override_show(struct device *dev, | ||
| 1310 | struct device_attribute *attr, char *buf) | ||
| 1311 | { | ||
| 1312 | struct platform_device *pdev = to_platform_device(dev); | ||
| 1313 | ssize_t len; | ||
| 1314 | |||
| 1315 | device_lock(dev); | ||
| 1316 | len = sysfs_emit(buf, "%s\n", pdev->driver_override); | ||
| 1317 | device_unlock(dev); | ||
| 1318 | |||
| 1319 | return len; | ||
| 1320 | } | ||
| 1321 | |||
| 1322 | static ssize_t driver_override_store(struct device *dev, | ||
| 1323 | struct device_attribute *attr, | ||
| 1324 | const char *buf, size_t count) | ||
| 1325 | { | ||
| 1326 | struct platform_device *pdev = to_platform_device(dev); | ||
| 1327 | int ret; | ||
| 1328 | |||
| 1329 | ret = driver_set_override(dev, &pdev->driver_override, buf, count); | ||
| 1330 | if (ret) | ||
| 1331 | return ret; | ||
| 1332 | |||
| 1333 | return count; | ||
| 1334 | } | ||
| 1335 | static DEVICE_ATTR_RW(driver_override); | ||
| 1336 | |||
| 1337 | static struct attribute *platform_dev_attrs[] = { | 1308 | static struct attribute *platform_dev_attrs[] = { |
| 1338 | &dev_attr_modalias.attr, | 1309 | &dev_attr_modalias.attr, |
| 1339 | &dev_attr_numa_node.attr, | 1310 | &dev_attr_numa_node.attr, |
| 1340 | &dev_attr_driver_override.attr, | ||
| 1341 | NULL, | 1311 | NULL, |
| 1342 | }; | 1312 | }; |
| 1343 | 1313 | ||
| @@ -1377,10 +1347,12 @@ static int platform_match(struct device *dev, const struct device_driver *drv) · 33 unmodified lines | |||
| 1377 | { | 1347 | { |
| 1378 | struct platform_device *pdev = to_platform_device(dev); | 1348 | struct platform_device *pdev = to_platform_device(dev); |
| 1379 | struct platform_driver *pdrv = to_platform_driver(drv); | 1349 | struct platform_driver *pdrv = to_platform_driver(drv); |
| 1350 | int ret; | ||
| 1380 | 1351 | ||
| 1381 | /* When driver_override is set, only bind to the matching driver */ | 1352 | /* When driver_override is set, only bind to the matching driver */ |
| 1382 | if (pdev->driver_override) | 1353 | ret = device_match_driver_override(dev, drv); |
| 1383 | return !strcmp(pdev->driver_override, drv->name); | 1354 | if (ret >= 0) |
| 1355 | return ret; | ||
| 1384 | 1356 | ||
| 1385 | /* Attempt an OF style match first */ | 1357 | /* Attempt an OF style match first */ |
| 1386 | if (of_driver_match_device(dev, drv)) | 1358 | if (of_driver_match_device(dev, drv)) |
| @@ -1516,6 +1488,7 @@ static const struct dev_pm_ops platform_dev_pm_ops = { · 129 unmodified lines | |||
| 1516 | const struct bus_type platform_bus_type = { | 1488 | const struct bus_type platform_bus_type = { |
| 1517 | .name = "platform", | 1489 | .name = "platform", |
| 1518 | .dev_groups = platform_dev_groups, | 1490 | .dev_groups = platform_dev_groups, |
| 1491 | .driver_override = true, | ||
| 1519 | .match = platform_match, | 1492 | .match = platform_match, |
| 1520 | .uevent = platform_uevent, | 1493 | .uevent = platform_uevent, |
| 1521 | .probe = platform_probe, | 1494 | .probe = platform_probe, |
| @@ -1895,6 +1895,7 @@ void pm_runtime_reinit(struct device *dev) · 1894 unmodified lines | |||
| 1895 | void pm_runtime_remove(struct device *dev) | 1895 | void pm_runtime_remove(struct device *dev) |
| 1896 | { | 1896 | { |
| 1897 | __pm_runtime_disable(dev, false); | 1897 | __pm_runtime_disable(dev, false); |
| 1898 | flush_work(&dev->power.work); | ||
| 1898 | pm_runtime_reinit(dev); | 1899 | pm_runtime_reinit(dev); |
| 1899 | } | 1900 | } |
| 1900 | 1901 | ||
| @@ -1545,6 +1545,7 @@ static int _regmap_select_page(struct regmap *map, unsigned int *reg, · 1544 unmodified lines | |||
| 1545 | unsigned int val_num) | 1545 | unsigned int val_num) |
| 1546 | { | 1546 | { |
| 1547 | void *orig_work_buf; | 1547 | void *orig_work_buf; |
| 1548 | unsigned int selector_reg; | ||
| 1548 | unsigned int win_offset; | 1549 | unsigned int win_offset; |
| 1549 | unsigned int win_page; | 1550 | unsigned int win_page; |
| 1550 | bool page_chg; | 1551 | bool page_chg; |
| @@ -1563,10 +1564,31 @@ static int _regmap_select_page(struct regmap *map, unsigned int *reg, · 12 unmodified lines | |||
| 1563 | return -EINVAL; | 1564 | return -EINVAL; |
| 1564 | } | 1565 | } |
| 1565 | 1566 | ||
| 1566 | /* It is possible to have selector register inside data window. | 1567 | /* |
| 1567 | In that case, selector register is located on every page and | 1568 | * Calculate the address of the selector register in the corresponding |
| 1568 | it needs no page switching, when accessed alone. */ | 1569 | * data window if it is located on every page. |
| 1570 | */ | ||
| 1571 | page_chg = in_range(range->selector_reg, range->window_start, range->window_len); | ||
| 1572 | if (page_chg) | ||
| 1573 | selector_reg = range->range_min + win_page * range->window_len + | ||
| 1574 | range->selector_reg - range->window_start; | ||
| 1575 | |||
| 1576 | /* | ||
| 1577 | * It is possible to have selector register inside data window. | ||
| 1578 | * In that case, selector register is located on every page and it | ||
| 1579 | * needs no page switching, when accessed alone. | ||
| 1580 | * | ||
| 1581 | * Nevertheless we should synchronize the cache values for it. | ||
| 1582 | * This can't be properly achieved if the selector register is | ||
| 1583 | * the first and the only one to be read inside the data window. | ||
| 1584 | * That's why we update it in that case as well. | ||
| 1585 | * | ||
| 1586 | * However, we specifically avoid updating it for the default page, | ||
| 1587 | * when it's overlapped with the real data window, to prevent from | ||
| 1588 | * infinite looping. | ||
| 1589 | */ | ||
| 1569 | if (val_num > 1 || | 1590 | if (val_num > 1 || |
| 1591 | (page_chg && selector_reg != range->selector_reg) || | ||
| 1570 | range->window_start + win_offset != range->selector_reg) { | 1592 | range->window_start + win_offset != range->selector_reg) { |
| 1571 | /* Use separate work_buf during page switching */ | 1593 | /* Use separate work_buf during page switching */ |
| 1572 | orig_work_buf = map->work_buf; | 1594 | orig_work_buf = map->work_buf; |
| @@ -1575,7 +1597,7 @@ static int _regmap_select_page(struct regmap *map, unsigned int *reg, · 2 unmodified lines | |||
| 1575 | ret = _regmap_update_bits(map, range->selector_reg, | 1597 | ret = _regmap_update_bits(map, range->selector_reg, |
| 1576 | range->selector_mask, | 1598 | range->selector_mask, |
| 1577 | win_page << range->selector_shift, | 1599 | win_page << range->selector_shift, |
| 1578 | &page_chg, false); | 1600 | NULL, false); |
| 1579 | 1601 | ||
| 1580 | map->work_buf = orig_work_buf; | 1602 | map->work_buf = orig_work_buf; |
| 1581 | 1603 | ||
| @@ -4443,7 +4443,9 @@ static int ublk_ctrl_start_dev(struct ublk_device *ub, · 4442 unmodified lines | |||
| 4443 | 4443 | ||
| 4444 | /* Skip partition scan if disabled by user */ | 4444 | /* Skip partition scan if disabled by user */ |
| 4445 | if (ub->dev_info.flags & UBLK_F_NO_AUTO_PART_SCAN) { | 4445 | if (ub->dev_info.flags & UBLK_F_NO_AUTO_PART_SCAN) { |
| 4446 | clear_bit(GD_SUPPRESS_PART_SCAN, &disk->state); | 4446 | /* Not clear for unprivileged daemons, see comment above */ |
| 4447 | if (!ub->unprivileged_daemons) | ||
| 4448 | clear_bit(GD_SUPPRESS_PART_SCAN, &disk->state); | ||
| 4447 | } else { | 4449 | } else { |
| 4448 | /* Schedule async partition scan for trusted daemons */ | 4450 | /* Schedule async partition scan for trusted daemons */ |
| 4449 | if (!ub->unprivileged_daemons) | 4451 | if (!ub->unprivileged_daemons) |
| @@ -5006,15 +5008,22 @@ static int ublk_ctrl_get_features(const struct ublksrv_ctrl_cmd *header) · 556 unmodified lines | |||
| 5006 | return 0; | 5008 | return 0; |
| 5007 | } | 5009 | } |
| 5008 | 5010 | ||
| 5009 | static void ublk_ctrl_set_size(struct ublk_device *ub, const struct ublksrv_ctrl_cmd *header) | 5011 | static int ublk_ctrl_set_size(struct ublk_device *ub, const struct ublksrv_ctrl_cmd *header) |
| 5010 | { | 5012 | { |
| 5011 | struct ublk_param_basic *p = &ub->params.basic; | 5013 | struct ublk_param_basic *p = &ub->params.basic; |
| 5012 | u64 new_size = header->data[0]; | 5014 | u64 new_size = header->data[0]; |
| 5015 | int ret = 0; | ||
| 5013 | 5016 | ||
| 5014 | mutex_lock(&ub->mutex); | 5017 | mutex_lock(&ub->mutex); |
| 5018 | if (!ub->ub_disk) { | ||
| 5019 | ret = -ENODEV; | ||
| 5020 | goto out; | ||
| 5021 | } | ||
| 5015 | p->dev_sectors = new_size; | 5022 | p->dev_sectors = new_size; |
| 5016 | set_capacity_and_notify(ub->ub_disk, p->dev_sectors); | 5023 | set_capacity_and_notify(ub->ub_disk, p->dev_sectors); |
| 5024 | out: | ||
| 5017 | mutex_unlock(&ub->mutex); | 5025 | mutex_unlock(&ub->mutex); |
| 5026 | return ret; | ||
| 5018 | } | 5027 | } |
| 5019 | 5028 | ||
| 5020 | struct count_busy { | 5029 | struct count_busy { |
| @@ -5335,8 +5344,7 @@ static int ublk_ctrl_uring_cmd(struct io_uring_cmd *cmd, · 314 unmodified lines | |||
| 5335 | ret = ublk_ctrl_end_recovery(ub, &header); | 5344 | ret = ublk_ctrl_end_recovery(ub, &header); |
| 5336 | break; | 5345 | break; |
| 5337 | case UBLK_CMD_UPDATE_SIZE: | 5346 | case UBLK_CMD_UPDATE_SIZE: |
| 5338 | ublk_ctrl_set_size(ub, &header); | 5347 | ret = ublk_ctrl_set_size(ub, &header); |
| 5339 | ret = 0; | ||
| 5340 | break; | 5348 | break; |
| 5341 | case UBLK_CMD_QUIESCE_DEV: | 5349 | case UBLK_CMD_QUIESCE_DEV: |
| 5342 | ret = ublk_ctrl_quiesce_dev(ub, &header); | 5350 | ret = ublk_ctrl_quiesce_dev(ub, &header); |
| @@ -549,7 +549,7 @@ static ssize_t bd_stat_show(struct device *dev, struct device_attribute *attr, · 548 unmodified lines | |||
| 549 | return ret; | 549 | return ret; |
| 550 | } | 550 | } |
| 551 | 551 | ||
| 552 | static ssize_t writeback_compressed_store(struct device *dev, | 552 | static ssize_t compressed_writeback_store(struct device *dev, |
| 553 | struct device_attribute *attr, | 553 | struct device_attribute *attr, |
| 554 | const char *buf, size_t len) | 554 | const char *buf, size_t len) |
| 555 | { | 555 | { |
| @@ -564,12 +564,12 @@ static ssize_t writeback_compressed_store(struct device *dev, · 8 unmodified lines | |||
| 564 | return -EBUSY; | 564 | return -EBUSY; |
| 565 | } | 565 | } |
| 566 | 566 | ||
| 567 | zram->wb_compressed = val; | 567 | zram->compressed_wb = val; |
| 568 | 568 | ||
| 569 | return len; | 569 | return len; |
| 570 | } | 570 | } |
| 571 | 571 | ||
| 572 | static ssize_t writeback_compressed_show(struct device *dev, | 572 | static ssize_t compressed_writeback_show(struct device *dev, |
| 573 | struct device_attribute *attr, | 573 | struct device_attribute *attr, |
| 574 | char *buf) | 574 | char *buf) |
| 575 | { | 575 | { |
| @@ -577,7 +577,7 @@ static ssize_t writeback_compressed_show(struct device *dev, · 1 unmodified line | |||
| 577 | struct zram *zram = dev_to_zram(dev); | 577 | struct zram *zram = dev_to_zram(dev); |
| 578 | 578 | ||
| 579 | guard(rwsem_read)(&zram->dev_lock); | 579 | guard(rwsem_read)(&zram->dev_lock); |
| 580 | val = zram->wb_compressed; | 580 | val = zram->compressed_wb; |
| 581 | 581 | ||
| 582 | return sysfs_emit(buf, "%d\n", val); | 582 | return sysfs_emit(buf, "%d\n", val); |
| 583 | } | 583 | } |
| @@ -917,9 +917,8 @@ static void zram_account_writeback_submit(struct zram *zram) · 333 unmodified lines | |||
| 917 | 917 | ||
| 918 | static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) | 918 | static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) |
| 919 | { | 919 | { |
| 920 | u32 size, index = req->pps->index; | 920 | u32 index = req->pps->index; |
| 921 | int err, prio; | 921 | int err; |
| 922 | bool huge; | ||
| 923 | 922 | ||
| 924 | err = blk_status_to_errno(req->bio.bi_status); | 923 | err = blk_status_to_errno(req->bio.bi_status); |
| 925 | if (err) { | 924 | if (err) { |
| @@ -946,28 +945,13 @@ static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) · 20 unmodified lines | |||
| 946 | goto out; | 945 | goto out; |
| 947 | } | 946 | } |
| 948 | 947 | ||
| 949 | if (zram->wb_compressed) { | 948 | clear_slot_flag(zram, index, ZRAM_IDLE); |
| 950 | /* | 949 | if (test_slot_flag(zram, index, ZRAM_HUGE)) |
| 951 | * ZRAM_WB slots get freed, we need to preserve data required | 950 | atomic64_dec(&zram->stats.huge_pages); |
| 952 | * for read decompression. | 951 | atomic64_sub(get_slot_size(zram, index), &zram->stats.compr_data_size); |
| 953 | */ | 952 | zs_free(zram->mem_pool, get_slot_handle(zram, index)); |
| 954 | size = get_slot_size(zram, index); | ||
| 955 | prio = get_slot_comp_priority(zram, index); | ||
| 956 | huge = test_slot_flag(zram, index, ZRAM_HUGE); | ||
| 957 | } | ||
| 958 | |||
| 959 | slot_free(zram, index); | ||
| 960 | set_slot_flag(zram, index, ZRAM_WB); | ||
| 961 | set_slot_handle(zram, index, req->blk_idx); | 953 | set_slot_handle(zram, index, req->blk_idx); |
| 962 | 954 | set_slot_flag(zram, index, ZRAM_WB); | |
| 963 | if (zram->wb_compressed) { | ||
| 964 | if (huge) | ||
| 965 | set_slot_flag(zram, index, ZRAM_HUGE); | ||
| 966 | set_slot_size(zram, index, size); | ||
| 967 | set_slot_comp_priority(zram, index, prio); | ||
| 968 | } | ||
| 969 | |||
| 970 | atomic64_inc(&zram->stats.pages_stored); | ||
| 971 | 955 | ||
| 972 | out: | 956 | out: |
| 973 | slot_unlock(zram, index); | 957 | slot_unlock(zram, index); |
| @@ -1100,7 +1084,7 @@ static int zram_writeback_slots(struct zram *zram, · 126 unmodified lines | |||
| 1100 | */ | 1084 | */ |
| 1101 | if (!test_slot_flag(zram, index, ZRAM_PP_SLOT)) | 1085 | if (!test_slot_flag(zram, index, ZRAM_PP_SLOT)) |
| 1102 | goto next; | 1086 | goto next; |
| 1103 | if (zram->wb_compressed) | 1087 | if (zram->compressed_wb) |
| 1104 | err = read_from_zspool_raw(zram, req->page, index); | 1088 | err = read_from_zspool_raw(zram, req->page, index); |
| 1105 | else | 1089 | else |
| 1106 | err = read_from_zspool(zram, req->page, index); | 1090 | err = read_from_zspool(zram, req->page, index); |
| @@ -1429,7 +1413,7 @@ static void zram_async_read_endio(struct bio *bio) · 322 unmodified lines | |||
| 1429 | * | 1413 | * |
| 1430 | * Keep the existing behavior for now. | 1414 | * Keep the existing behavior for now. |
| 1431 | */ | 1415 | */ |
| 1432 | if (zram->wb_compressed == false) { | 1416 | if (zram->compressed_wb == false) { |
| 1433 | /* No decompression needed, complete the parent IO */ | 1417 | /* No decompression needed, complete the parent IO */ |
| 1434 | bio_endio(req->parent); | 1418 | bio_endio(req->parent); |
| 1435 | bio_put(bio); | 1419 | bio_put(bio); |
| @@ -1508,7 +1492,7 @@ static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index, · 72 unmodified lines | |||
| 1508 | flush_work(&req.work); | 1492 | flush_work(&req.work); |
| 1509 | destroy_work_on_stack(&req.work); | 1493 | destroy_work_on_stack(&req.work); |
| 1510 | 1494 | ||
| 1511 | if (req.error || zram->wb_compressed == false) | 1495 | if (req.error || zram->compressed_wb == false) |
| 1512 | return req.error; | 1496 | return req.error; |
| 1513 | 1497 | ||
| 1514 | return decompress_bdev_page(zram, page, index); | 1498 | return decompress_bdev_page(zram, page, index); |
| @@ -2010,8 +1994,13 @@ static void slot_free(struct zram *zram, u32 index) · 495 unmodified lines | |||
| 2010 | set_slot_comp_priority(zram, index, 0); | 1994 | set_slot_comp_priority(zram, index, 0); |
| 2011 | 1995 | ||
| 2012 | if (test_slot_flag(zram, index, ZRAM_HUGE)) { | 1996 | if (test_slot_flag(zram, index, ZRAM_HUGE)) { |
| 1997 | /* | ||
| 1998 | * Writeback completion decrements ->huge_pages but keeps | ||
| 1999 | * ZRAM_HUGE flag for deferred decompression path. | ||
| 2000 | */ | ||
| 2001 | if (!test_slot_flag(zram, index, ZRAM_WB)) | ||
| 2002 | atomic64_dec(&zram->stats.huge_pages); | ||
| 2013 | clear_slot_flag(zram, index, ZRAM_HUGE); | 2003 | clear_slot_flag(zram, index, ZRAM_HUGE); |
| 2014 | atomic64_dec(&zram->stats.huge_pages); | ||
| 2015 | } | 2004 | } |
| 2016 | 2005 | ||
| 2017 | if (test_slot_flag(zram, index, ZRAM_WB)) { | 2006 | if (test_slot_flag(zram, index, ZRAM_WB)) { |
| @@ -3007,7 +2996,7 @@ static DEVICE_ATTR_WO(writeback); · 989 unmodified lines | |||
| 3007 | static DEVICE_ATTR_RW(writeback_limit); | 2996 | static DEVICE_ATTR_RW(writeback_limit); |
| 3008 | static DEVICE_ATTR_RW(writeback_limit_enable); | 2997 | static DEVICE_ATTR_RW(writeback_limit_enable); |
| 3009 | static DEVICE_ATTR_RW(writeback_batch_size); | 2998 | static DEVICE_ATTR_RW(writeback_batch_size); |
| 3010 | static DEVICE_ATTR_RW(writeback_compressed); | 2999 | static DEVICE_ATTR_RW(compressed_writeback); |
| 3011 | #endif | 3000 | #endif |
| 3012 | #ifdef CONFIG_ZRAM_MULTI_COMP | 3001 | #ifdef CONFIG_ZRAM_MULTI_COMP |
| 3013 | static DEVICE_ATTR_RW(recomp_algorithm); | 3002 | static DEVICE_ATTR_RW(recomp_algorithm); |
| @@ -3031,7 +3020,7 @@ static struct attribute *zram_disk_attrs[] = { · 17 unmodified lines | |||
| 3031 | &dev_attr_writeback_limit.attr, | 3020 | &dev_attr_writeback_limit.attr, |
| 3032 | &dev_attr_writeback_limit_enable.attr, | 3021 | &dev_attr_writeback_limit_enable.attr, |
| 3033 | &dev_attr_writeback_batch_size.attr, | 3022 | &dev_attr_writeback_batch_size.attr, |
| 3034 | &dev_attr_writeback_compressed.attr, | 3023 | &dev_attr_compressed_writeback.attr, |
| 3035 | #endif | 3024 | #endif |
| 3036 | &dev_attr_io_stat.attr, | 3025 | &dev_attr_io_stat.attr, |
| 3037 | &dev_attr_mm_stat.attr, | 3026 | &dev_attr_mm_stat.attr, |
| @@ -3091,7 +3080,7 @@ static int zram_add(void) · 53 unmodified lines | |||
| 3091 | init_rwsem(&zram->dev_lock); | 3080 | init_rwsem(&zram->dev_lock); |
| 3092 | #ifdef CONFIG_ZRAM_WRITEBACK | 3081 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 3093 | zram->wb_batch_size = 32; | 3082 | zram->wb_batch_size = 32; |
| 3094 | zram->wb_compressed = false; | 3083 | zram->compressed_wb = false; |
| 3095 | #endif | 3084 | #endif |
| 3096 | 3085 | ||
| 3097 | /* gendisk structure */ | 3086 | /* gendisk structure */ |
| @@ -133,7 +133,7 @@ struct zram { · 132 unmodified lines | |||
| 133 | #ifdef CONFIG_ZRAM_WRITEBACK | 133 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 134 | struct file *backing_dev; | 134 | struct file *backing_dev; |
| 135 | bool wb_limit_enable; | 135 | bool wb_limit_enable; |
| 136 | bool wb_compressed; | 136 | bool compressed_wb; |
| 137 | u32 wb_batch_size; | 137 | u32 wb_batch_size; |
| 138 | u64 bd_wb_limit; | 138 | u64 bd_wb_limit; |
| 139 | struct block_device *bdev; | 139 | struct block_device *bdev; |
| @@ -251,11 +251,13 @@ void btintel_hw_error(struct hci_dev *hdev, u8 code) · 250 unmodified lines | |||
| 251 | 251 | ||
| 252 | bt_dev_err(hdev, "Hardware error 0x%2.2x", code); | 252 | bt_dev_err(hdev, "Hardware error 0x%2.2x", code); |
| 253 | 253 | ||
| 254 | hci_req_sync_lock(hdev); | ||
| 255 | |||
| 254 | skb = __hci_cmd_sync(hdev, HCI_OP_RESET, 0, NULL, HCI_INIT_TIMEOUT); | 256 | skb = __hci_cmd_sync(hdev, HCI_OP_RESET, 0, NULL, HCI_INIT_TIMEOUT); |
| 255 | if (IS_ERR(skb)) { | 257 | if (IS_ERR(skb)) { |
| 256 | bt_dev_err(hdev, "Reset after hardware error failed (%ld)", | 258 | bt_dev_err(hdev, "Reset after hardware error failed (%ld)", |
| 257 | PTR_ERR(skb)); | 259 | PTR_ERR(skb)); |
| 258 | return; | 260 | goto unlock; |
| 259 | } | 261 | } |
| 260 | kfree_skb(skb); | 262 | kfree_skb(skb); |
| 261 | 263 | ||
| @@ -263,18 +265,21 @@ void btintel_hw_error(struct hci_dev *hdev, u8 code) · 1 unmodified line | |||
| 263 | if (IS_ERR(skb)) { | 265 | if (IS_ERR(skb)) { |
| 264 | bt_dev_err(hdev, "Retrieving Intel exception info failed (%ld)", | 266 | bt_dev_err(hdev, "Retrieving Intel exception info failed (%ld)", |
| 265 | PTR_ERR(skb)); | 267 | PTR_ERR(skb)); |
| 266 | return; | 268 | goto unlock; |
| 267 | } | 269 | } |
| 268 | 270 | ||
| 269 | if (skb->len != 13) { | 271 | if (skb->len != 13) { |
| 270 | bt_dev_err(hdev, "Exception info size mismatch"); | 272 | bt_dev_err(hdev, "Exception info size mismatch"); |
| 271 | kfree_skb(skb); | 273 | kfree_skb(skb); |
| 272 | return; | 274 | goto unlock; |
| 273 | } | 275 | } |
| 274 | 276 | ||
| 275 | bt_dev_err(hdev, "Exception info %s", (char *)(skb->data + 1)); | 277 | bt_dev_err(hdev, "Exception info %s", (char *)(skb->data + 1)); |
| 276 | 278 | ||
| 277 | kfree_skb(skb); | 279 | kfree_skb(skb); |
| 280 | |||
| 281 | unlock: | ||
| 282 | hci_req_sync_unlock(hdev); | ||
| 278 | } | 283 | } |
| 279 | EXPORT_SYMBOL_GPL(btintel_hw_error); | 284 | EXPORT_SYMBOL_GPL(btintel_hw_error); |
| 280 | 285 | ||
| @@ -787,6 +787,8 @@ int qca_uart_setup(struct hci_dev *hdev, uint8_t baudrate, · 786 unmodified lines | |||
| 787 | */ | 787 | */ |
| 788 | if (soc_type == QCA_WCN3988) | 788 | if (soc_type == QCA_WCN3988) |
| 789 | rom_ver = ((soc_ver & 0x00000f00) >> 0x05) | (soc_ver & 0x0000000f); | 789 | rom_ver = ((soc_ver & 0x00000f00) >> 0x05) | (soc_ver & 0x0000000f); |
| 790 | else if (soc_type == QCA_WCN3998) | ||
| 791 | rom_ver = ((soc_ver & 0x0000f000) >> 0x07) | (soc_ver & 0x0000000f); | ||
| 790 | else | 792 | else |
| 791 | rom_ver = ((soc_ver & 0x00000f00) >> 0x04) | (soc_ver & 0x0000000f); | 793 | rom_ver = ((soc_ver & 0x00000f00) >> 0x04) | (soc_ver & 0x0000000f); |
| 792 | 794 | ||
| @@ -2376,8 +2376,11 @@ static void btusb_work(struct work_struct *work) · 2375 unmodified lines | |||
| 2376 | if (data->air_mode == HCI_NOTIFY_ENABLE_SCO_CVSD) { | 2376 | if (data->air_mode == HCI_NOTIFY_ENABLE_SCO_CVSD) { |
| 2377 | if (hdev->voice_setting & 0x0020) { | 2377 | if (hdev->voice_setting & 0x0020) { |
| 2378 | static const int alts[3] = { 2, 4, 5 }; | 2378 | static const int alts[3] = { 2, 4, 5 }; |
| 2379 | unsigned int sco_idx; | ||
| 2379 | 2380 | ||
| 2380 | new_alts = alts[data->sco_num - 1]; | 2381 | sco_idx = min_t(unsigned int, data->sco_num - 1, |
| 2382 | ARRAY_SIZE(alts) - 1); | ||
| 2383 | new_alts = alts[sco_idx]; | ||
| 2381 | } else { | 2384 | } else { |
| 2382 | new_alts = data->sco_num; | 2385 | new_alts = data->sco_num; |
| 2383 | } | 2386 | } |
| @@ -109,9 +109,6 @@ static int h4_recv(struct hci_uart *hu, const void *data, int count) · 108 unmodified lines | |||
| 109 | { | 109 | { |
| 110 | struct h4_struct *h4 = hu->priv; | 110 | struct h4_struct *h4 = hu->priv; |
| 111 | 111 | ||
| 112 | if (!test_bit(HCI_UART_REGISTERED, &hu->flags)) | ||
| 113 | return -EUNATCH; | ||
| 114 | |||
| 115 | h4->rx_skb = h4_recv_buf(hu, h4->rx_skb, data, count, | 112 | h4->rx_skb = h4_recv_buf(hu, h4->rx_skb, data, count, |
| 116 | h4_recv_pkts, ARRAY_SIZE(h4_recv_pkts)); | 113 | h4_recv_pkts, ARRAY_SIZE(h4_recv_pkts)); |
| 117 | if (IS_ERR(h4->rx_skb)) { | 114 | if (IS_ERR(h4->rx_skb)) { |
| @@ -541,6 +541,8 @@ static int download_firmware(struct ll_device *lldev) · 540 unmodified lines | |||
| 541 | if (err || !fw->data || !fw->size) { | 541 | if (err || !fw->data || !fw->size) { |
| 542 | bt_dev_err(lldev->hu.hdev, "request_firmware failed(errno %d) for %s", | 542 | bt_dev_err(lldev->hu.hdev, "request_firmware failed(errno %d) for %s", |
| 543 | err, bts_scr_name); | 543 | err, bts_scr_name); |
| 544 | if (!err) | ||
| 545 | release_firmware(fw); | ||
| 544 | return -EINVAL; | 546 | return -EINVAL; |
| 545 | } | 547 | } |
| 546 | ptr = (void *)fw->data; | 548 | ptr = (void *)fw->data; |
| @@ -36,7 +36,7 @@ static int simple_pm_bus_probe(struct platform_device *pdev) · 35 unmodified lines | |||
| 36 | * that's not listed in simple_pm_bus_of_match. We don't want to do any | 36 | * that's not listed in simple_pm_bus_of_match. We don't want to do any |
| 37 | * of the simple-pm-bus tasks for these devices, so return early. | 37 | * of the simple-pm-bus tasks for these devices, so return early. |
| 38 | */ | 38 | */ |
| 39 | if (pdev->driver_override) | 39 | if (device_has_driver_override(&pdev->dev)) |
| 40 | return 0; | 40 | return 0; |
| 41 | 41 | ||
| 42 | match = of_match_device(dev->driver->of_match_table, dev); | 42 | match = of_match_device(dev->driver->of_match_table, dev); |
| @@ -78,7 +78,7 @@ static void simple_pm_bus_remove(struct platform_device *pdev) · 35 unmodified lines | |||
| 78 | { | 78 | { |
| 79 | const void *data = of_device_get_match_data(&pdev->dev); | 79 | const void *data = of_device_get_match_data(&pdev->dev); |
| 80 | 80 | ||
| 81 | if (pdev->driver_override || data) | 81 | if (device_has_driver_override(&pdev->dev) || data) |
| 82 | return; | 82 | return; |
| 83 | 83 | ||
| 84 | dev_dbg(&pdev->dev, "%s\n", __func__); | 84 | dev_dbg(&pdev->dev, "%s\n", __func__); |
| @@ -178,11 +178,11 @@ static const struct of_device_id ax45mp_cache_ids[] = { · 177 unmodified lines | |||
| 178 | 178 | ||
| 179 | static int __init ax45mp_cache_init(void) | 179 | static int __init ax45mp_cache_init(void) |
| 180 | { | 180 | { |
| 181 | struct device_node *np; | ||
| 182 | struct resource res; | 181 | struct resource res; |
| 183 | int ret; | 182 | int ret; |
| 184 | 183 | ||
| 185 | np = of_find_matching_node(NULL, ax45mp_cache_ids); | 184 | struct device_node *np __free(device_node) = |
| 185 | of_find_matching_node(NULL, ax45mp_cache_ids); | ||
| 186 | if (!of_device_is_available(np)) | 186 | if (!of_device_is_available(np)) |
| 187 | return -ENODEV; | 187 | return -ENODEV; |
| 188 | 188 | ||
| @@ -102,11 +102,11 @@ static const struct of_device_id starlink_cache_ids[] = { · 101 unmodified lines | |||
| 102 | 102 | ||
| 103 | static int __init starlink_cache_init(void) | 103 | static int __init starlink_cache_init(void) |
| 104 | { | 104 | { |
| 105 | struct device_node *np; | ||
| 106 | u32 block_size; | 105 | u32 block_size; |
| 107 | int ret; | 106 | int ret; |
| 108 | 107 | ||
| 109 | np = of_find_matching_node(NULL, starlink_cache_ids); | 108 | struct device_node *np __free(device_node) = |
| 109 | of_find_matching_node(NULL, starlink_cache_ids); | ||
| 110 | if (!of_device_is_available(np)) | 110 | if (!of_device_is_available(np)) |
| 111 | return -ENODEV; | 111 | return -ENODEV; |
| 112 | 112 | ||
| @@ -706,8 +706,7 @@ struct clk_hw *imx_clk_scu_alloc_dev(const char *name, · 705 unmodified lines | |||
| 706 | if (ret) | 706 | if (ret) |
| 707 | goto put_device; | 707 | goto put_device; |
| 708 | 708 | ||
| 709 | ret = driver_set_override(&pdev->dev, &pdev->driver_override, | 709 | ret = device_set_driver_override(&pdev->dev, "imx-scu-clk"); |
| 710 | "imx-scu-clk", strlen("imx-scu-clk")); | ||
| 711 | if (ret) | 710 | if (ret) |
| 712 | goto put_device; | 711 | goto put_device; |
| 713 | 712 | ||
| @@ -793,13 +793,15 @@ static void do_become_nonbusy(struct comedi_device *dev, · 792 unmodified lines | |||
| 793 | __comedi_clear_subdevice_runflags(s, COMEDI_SRF_RUNNING | | 793 | __comedi_clear_subdevice_runflags(s, COMEDI_SRF_RUNNING | |
| 794 | COMEDI_SRF_BUSY); | 794 | COMEDI_SRF_BUSY); |
| 795 | spin_unlock_irqrestore(&s->spin_lock, flags); | 795 | spin_unlock_irqrestore(&s->spin_lock, flags); |
| 796 | if (comedi_is_runflags_busy(runflags)) { | 796 | if (async) { |
| 797 | /* | 797 | /* |
| 798 | * "Run active" counter was set to 1 when setting up the | 798 | * "Run active" counter was set to 1 when setting up the |
| 799 | * command. Decrement it and wait for it to become 0. | 799 | * command. Decrement it and wait for it to become 0. |
| 800 | */ | 800 | */ |
| 801 | comedi_put_is_subdevice_running(s); | 801 | if (comedi_is_runflags_busy(runflags)) { |
| 802 | wait_for_completion(&async->run_complete); | 802 | comedi_put_is_subdevice_running(s); |
| 803 | wait_for_completion(&async->run_complete); | ||
| 804 | } | ||
| 803 | comedi_buf_reset(s); | 805 | comedi_buf_reset(s); |
| 804 | async->inttrig = NULL; | 806 | async->inttrig = NULL; |
| 805 | kfree(async->cmd.chanlist); | 807 | kfree(async->cmd.chanlist); |
| @@ -1063,6 +1063,14 @@ int comedi_device_attach(struct comedi_device *dev, struct comedi_devconfig *it) · 1062 unmodified lines | |||
| 1063 | ret = -EIO; | 1063 | ret = -EIO; |
| 1064 | goto out; | 1064 | goto out; |
| 1065 | } | 1065 | } |
| 1066 | if (IS_ENABLED(CONFIG_LOCKDEP)) { | ||
| 1067 | /* | ||
| 1068 | * dev->spinlock is for private use by the attached low-level | ||
| 1069 | * driver. Reinitialize it to stop lock-dependency tracking | ||
| 1070 | * between attachments to different low-level drivers. | ||
| 1071 | */ | ||
| 1072 | spin_lock_init(&dev->spinlock); | ||
| 1073 | } | ||
| 1066 | dev->driver = driv; | 1074 | dev->driver = driv; |
| 1067 | dev->board_name = dev->board_ptr ? *(const char **)dev->board_ptr | 1075 | dev->board_name = dev->board_ptr ? *(const char **)dev->board_ptr |
| 1068 | : dev->driver->driver_name; | 1076 | : dev->driver->driver_name; |
| @@ -175,6 +175,18 @@ static int dt2815_attach(struct comedi_device *dev, struct comedi_devconfig *it) · 174 unmodified lines | |||
| 175 | ? current_range_type : voltage_range_type; | 175 | ? current_range_type : voltage_range_type; |
| 176 | } | 176 | } |
| 177 | 177 | ||
| 178 | /* | ||
| 179 | * Check if hardware is present before attempting any I/O operations. | ||
| 180 | * Reading 0xff from status register typically indicates no hardware | ||
| 181 | * on the bus (floating bus reads as all 1s). | ||
| 182 | */ | ||
| 183 | if (inb(dev->iobase + DT2815_STATUS) == 0xff) { | ||
| 184 | dev_err(dev->class_dev, | ||
| 185 | "No hardware detected at I/O base 0x%lx\n", | ||
| 186 | dev->iobase); | ||
| 187 | return -ENODEV; | ||
| 188 | } | ||
| 189 | |||
| 178 | /* Init the 2815 */ | 190 | /* Init the 2815 */ |
| 179 | outb(0x00, dev->iobase + DT2815_STATUS); | 191 | outb(0x00, dev->iobase + DT2815_STATUS); |
| 180 | for (i = 0; i < 100; i++) { | 192 | for (i = 0; i < 100; i++) { |
| @@ -315,6 +315,18 @@ static int me4000_xilinx_download(struct comedi_device *dev, · 314 unmodified lines | |||
| 315 | unsigned int val; | 315 | unsigned int val; |
| 316 | unsigned int i; | 316 | unsigned int i; |
| 317 | 317 | ||
| 318 | /* Get data stream length from header. */ | ||
| 319 | if (size >= 4) { | ||
| 320 | file_length = (((unsigned int)data[0] & 0xff) << 24) + | ||
| 321 | (((unsigned int)data[1] & 0xff) << 16) + | ||
| 322 | (((unsigned int)data[2] & 0xff) << 8) + | ||
| 323 | ((unsigned int)data[3] & 0xff); | ||
| 324 | } | ||
| 325 | if (size < 16 || file_length > size - 16) { | ||
| 326 | dev_err(dev->class_dev, "Firmware length inconsistency\n"); | ||
| 327 | return -EINVAL; | ||
| 328 | } | ||
| 329 | |||
| 318 | if (!xilinx_iobase) | 330 | if (!xilinx_iobase) |
| 319 | return -ENODEV; | 331 | return -ENODEV; |
| 320 | 332 | ||
| @@ -346,10 +358,6 @@ static int me4000_xilinx_download(struct comedi_device *dev, · 25 unmodified lines | |||
| 346 | outl(val, devpriv->plx_regbase + PLX9052_CNTRL); | 358 | outl(val, devpriv->plx_regbase + PLX9052_CNTRL); |
| 347 | 359 | ||
| 348 | /* Download Xilinx firmware */ | 360 | /* Download Xilinx firmware */ |
| 349 | file_length = (((unsigned int)data[0] & 0xff) << 24) + | ||
| 350 | (((unsigned int)data[1] & 0xff) << 16) + | ||
| 351 | (((unsigned int)data[2] & 0xff) << 8) + | ||
| 352 | ((unsigned int)data[3] & 0xff); | ||
| 353 | usleep_range(10, 1000); | 361 | usleep_range(10, 1000); |
| 354 | 362 | ||
| 355 | for (i = 0; i < file_length; i++) { | 363 | for (i = 0; i < file_length; i++) { |
| @@ -344,6 +344,25 @@ static int me2600_xilinx_download(struct comedi_device *dev, · 343 unmodified lines | |||
| 344 | unsigned int file_length; | 344 | unsigned int file_length; |
| 345 | unsigned int i; | 345 | unsigned int i; |
| 346 | 346 | ||
| 347 | /* | ||
| 348 | * Format of the firmware | ||
| 349 | * Build longs from the byte-wise coded header | ||
| 350 | * Byte 1-3: length of the array | ||
| 351 | * Byte 4-7: version | ||
| 352 | * Byte 8-11: date | ||
| 353 | * Byte 12-15: reserved | ||
| 354 | */ | ||
| 355 | if (size >= 4) { | ||
| 356 | file_length = (((unsigned int)data[0] & 0xff) << 24) + | ||
| 357 | (((unsigned int)data[1] & 0xff) << 16) + | ||
| 358 | (((unsigned int)data[2] & 0xff) << 8) + | ||
| 359 | ((unsigned int)data[3] & 0xff); | ||
| 360 | } | ||
| 361 | if (size < 16 || file_length > size - 16) { | ||
| 362 | dev_err(dev->class_dev, "Firmware length inconsistency\n"); | ||
| 363 | return -EINVAL; | ||
| 364 | } | ||
| 365 | |||
| 347 | /* disable irq's on PLX */ | 366 | /* disable irq's on PLX */ |
| 348 | writel(0x00, devpriv->plx_regbase + PLX9052_INTCSR); | 367 | writel(0x00, devpriv->plx_regbase + PLX9052_INTCSR); |
| 349 | 368 | ||
| @@ -357,22 +376,6 @@ static int me2600_xilinx_download(struct comedi_device *dev, · 7 unmodified lines | |||
| 357 | writeb(0x00, dev->mmio + 0x0); | 376 | writeb(0x00, dev->mmio + 0x0); |
| 358 | sleep(1); | 377 | sleep(1); |
| 359 | 378 | ||
| 360 | /* | ||
| 361 | * Format of the firmware | ||
| 362 | * Build longs from the byte-wise coded header | ||
| 363 | * Byte 1-3: length of the array | ||
| 364 | * Byte 4-7: version | ||
| 365 | * Byte 8-11: date | ||
| 366 | * Byte 12-15: reserved | ||
| 367 | */ | ||
| 368 | if (size < 16) | ||
| 369 | return -EINVAL; | ||
| 370 | |||
| 371 | file_length = (((unsigned int)data[0] & 0xff) << 24) + | ||
| 372 | (((unsigned int)data[1] & 0xff) << 16) + | ||
| 373 | (((unsigned int)data[2] & 0xff) << 8) + | ||
| 374 | ((unsigned int)data[3] & 0xff); | ||
| 375 | |||
| 376 | /* | 379 | /* |
| 377 | * Loop for writing firmware byte by byte to xilinx | 380 | * Loop for writing firmware byte by byte to xilinx |
| 378 | * Firmware data start at offset 16 | 381 | * Firmware data start at offset 16 |
| @@ -698,7 +698,8 @@ static int atmio16d_attach(struct comedi_device *dev, · 697 unmodified lines | |||
| 698 | 698 | ||
| 699 | static void atmio16d_detach(struct comedi_device *dev) | 699 | static void atmio16d_detach(struct comedi_device *dev) |
| 700 | { | 700 | { |
| 701 | reset_atmio16d(dev); | 701 | if (dev->private) |
| 702 | reset_atmio16d(dev); | ||
| 702 | comedi_legacy_detach(dev); | 703 | comedi_legacy_detach(dev); |
| 703 | } | 704 | } |
| 704 | 705 | ||
| @@ -107,9 +107,9 @@ static bool rz_mtu3_is_counter_invalid(struct counter_device *counter, int id) · 106 unmodified lines | |||
| 107 | struct rz_mtu3_cnt *const priv = counter_priv(counter); | 107 | struct rz_mtu3_cnt *const priv = counter_priv(counter); |
| 108 | unsigned long tmdr; | 108 | unsigned long tmdr; |
| 109 | 109 | ||
| 110 | pm_runtime_get_sync(priv->ch->dev); | 110 | pm_runtime_get_sync(counter->parent); |
| 111 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); | 111 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); |
| 112 | pm_runtime_put(priv->ch->dev); | 112 | pm_runtime_put(counter->parent); |
| 113 | 113 | ||
| 114 | if (id == RZ_MTU3_32_BIT_CH && test_bit(RZ_MTU3_TMDR3_LWA, &tmdr)) | 114 | if (id == RZ_MTU3_32_BIT_CH && test_bit(RZ_MTU3_TMDR3_LWA, &tmdr)) |
| 115 | return false; | 115 | return false; |
| @@ -165,12 +165,12 @@ static int rz_mtu3_count_read(struct counter_device *counter, · 49 unmodified lines | |||
| 165 | if (ret) | 165 | if (ret) |
| 166 | return ret; | 166 | return ret; |
| 167 | 167 | ||
| 168 | pm_runtime_get_sync(ch->dev); | 168 | pm_runtime_get_sync(counter->parent); |
| 169 | if (count->id == RZ_MTU3_32_BIT_CH) | 169 | if (count->id == RZ_MTU3_32_BIT_CH) |
| 170 | *val = rz_mtu3_32bit_ch_read(ch, RZ_MTU3_TCNTLW); | 170 | *val = rz_mtu3_32bit_ch_read(ch, RZ_MTU3_TCNTLW); |
| 171 | else | 171 | else |
| 172 | *val = rz_mtu3_16bit_ch_read(ch, RZ_MTU3_TCNT); | 172 | *val = rz_mtu3_16bit_ch_read(ch, RZ_MTU3_TCNT); |
| 173 | pm_runtime_put(ch->dev); | 173 | pm_runtime_put(counter->parent); |
| 174 | mutex_unlock(&priv->lock); | 174 | mutex_unlock(&priv->lock); |
| 175 | 175 | ||
| 176 | return 0; | 176 | return 0; |
| @@ -187,26 +187,26 @@ static int rz_mtu3_count_write(struct counter_device *counter, · 10 unmodified lines | |||
| 187 | if (ret) | 187 | if (ret) |
| 188 | return ret; | 188 | return ret; |
| 189 | 189 | ||
| 190 | pm_runtime_get_sync(ch->dev); | 190 | pm_runtime_get_sync(counter->parent); |
| 191 | if (count->id == RZ_MTU3_32_BIT_CH) | 191 | if (count->id == RZ_MTU3_32_BIT_CH) |
| 192 | rz_mtu3_32bit_ch_write(ch, RZ_MTU3_TCNTLW, val); | 192 | rz_mtu3_32bit_ch_write(ch, RZ_MTU3_TCNTLW, val); |
| 193 | else | 193 | else |
| 194 | rz_mtu3_16bit_ch_write(ch, RZ_MTU3_TCNT, val); | 194 | rz_mtu3_16bit_ch_write(ch, RZ_MTU3_TCNT, val); |
| 195 | pm_runtime_put(ch->dev); | 195 | pm_runtime_put(counter->parent); |
| 196 | mutex_unlock(&priv->lock); | 196 | mutex_unlock(&priv->lock); |
| 197 | 197 | ||
| 198 | return 0; | 198 | return 0; |
| 199 | } | 199 | } |
| 200 | 200 | ||
| 201 | static int rz_mtu3_count_function_read_helper(struct rz_mtu3_channel *const ch, | 201 | static int rz_mtu3_count_function_read_helper(struct rz_mtu3_channel *const ch, |
| 202 | struct rz_mtu3_cnt *const priv, | 202 | struct counter_device *const counter, |
| 203 | enum counter_function *function) | 203 | enum counter_function *function) |
| 204 | { | 204 | { |
| 205 | u8 timer_mode; | 205 | u8 timer_mode; |
| 206 | 206 | ||
| 207 | pm_runtime_get_sync(ch->dev); | 207 | pm_runtime_get_sync(counter->parent); |
| 208 | timer_mode = rz_mtu3_8bit_ch_read(ch, RZ_MTU3_TMDR1); | 208 | timer_mode = rz_mtu3_8bit_ch_read(ch, RZ_MTU3_TMDR1); |
| 209 | pm_runtime_put(ch->dev); | 209 | pm_runtime_put(counter->parent); |
| 210 | 210 | ||
| 211 | switch (timer_mode & RZ_MTU3_TMDR1_PH_CNT_MODE_MASK) { | 211 | switch (timer_mode & RZ_MTU3_TMDR1_PH_CNT_MODE_MASK) { |
| 212 | case RZ_MTU3_TMDR1_PH_CNT_MODE_1: | 212 | case RZ_MTU3_TMDR1_PH_CNT_MODE_1: |
| @@ -240,7 +240,7 @@ static int rz_mtu3_count_function_read(struct counter_device *counter, · 27 unmodified lines | |||
| 240 | if (ret) | 240 | if (ret) |
| 241 | return ret; | 241 | return ret; |
| 242 | 242 | ||
| 243 | ret = rz_mtu3_count_function_read_helper(ch, priv, function); | 243 | ret = rz_mtu3_count_function_read_helper(ch, counter, function); |
| 244 | mutex_unlock(&priv->lock); | 244 | mutex_unlock(&priv->lock); |
| 245 | 245 | ||
| 246 | return ret; | 246 | return ret; |
| @@ -279,9 +279,9 @@ static int rz_mtu3_count_function_write(struct counter_device *counter, · 32 unmodified lines | |||
| 279 | return -EINVAL; | 279 | return -EINVAL; |
| 280 | } | 280 | } |
| 281 | 281 | ||
| 282 | pm_runtime_get_sync(ch->dev); | 282 | pm_runtime_get_sync(counter->parent); |
| 283 | rz_mtu3_8bit_ch_write(ch, RZ_MTU3_TMDR1, timer_mode); | 283 | rz_mtu3_8bit_ch_write(ch, RZ_MTU3_TMDR1, timer_mode); |
| 284 | pm_runtime_put(ch->dev); | 284 | pm_runtime_put(counter->parent); |
| 285 | mutex_unlock(&priv->lock); | 285 | mutex_unlock(&priv->lock); |
| 286 | 286 | ||
| 287 | return 0; | 287 | return 0; |
| @@ -300,9 +300,9 @@ static int rz_mtu3_count_direction_read(struct counter_device *counter, · 12 unmodified lines | |||
| 300 | if (ret) | 300 | if (ret) |
| 301 | return ret; | 301 | return ret; |
| 302 | 302 | ||
| 303 | pm_runtime_get_sync(ch->dev); | 303 | pm_runtime_get_sync(counter->parent); |
| 304 | tsr = rz_mtu3_8bit_ch_read(ch, RZ_MTU3_TSR); | 304 | tsr = rz_mtu3_8bit_ch_read(ch, RZ_MTU3_TSR); |
| 305 | pm_runtime_put(ch->dev); | 305 | pm_runtime_put(counter->parent); |
| 306 | 306 | ||
| 307 | *direction = (tsr & RZ_MTU3_TSR_TCFD) ? | 307 | *direction = (tsr & RZ_MTU3_TSR_TCFD) ? |
| 308 | COUNTER_COUNT_DIRECTION_FORWARD : COUNTER_COUNT_DIRECTION_BACKWARD; | 308 | COUNTER_COUNT_DIRECTION_FORWARD : COUNTER_COUNT_DIRECTION_BACKWARD; |
| @@ -377,14 +377,14 @@ static int rz_mtu3_count_ceiling_write(struct counter_device *counter, · 68 unmodified lines | |||
| 377 | return -EINVAL; | 377 | return -EINVAL; |
| 378 | } | 378 | } |
| 379 | 379 | ||
| 380 | pm_runtime_get_sync(ch->dev); | 380 | pm_runtime_get_sync(counter->parent); |
| 381 | if (count->id == RZ_MTU3_32_BIT_CH) | 381 | if (count->id == RZ_MTU3_32_BIT_CH) |
| 382 | rz_mtu3_32bit_ch_write(ch, RZ_MTU3_TGRALW, ceiling); | 382 | rz_mtu3_32bit_ch_write(ch, RZ_MTU3_TGRALW, ceiling); |
| 383 | else | 383 | else |
| 384 | rz_mtu3_16bit_ch_write(ch, RZ_MTU3_TGRA, ceiling); | 384 | rz_mtu3_16bit_ch_write(ch, RZ_MTU3_TGRA, ceiling); |
| 385 | 385 | ||
| 386 | rz_mtu3_8bit_ch_write(ch, RZ_MTU3_TCR, RZ_MTU3_TCR_CCLR_TGRA); | 386 | rz_mtu3_8bit_ch_write(ch, RZ_MTU3_TCR, RZ_MTU3_TCR_CCLR_TGRA); |
| 387 | pm_runtime_put(ch->dev); | 387 | pm_runtime_put(counter->parent); |
| 388 | mutex_unlock(&priv->lock); | 388 | mutex_unlock(&priv->lock); |
| 389 | 389 | ||
| 390 | return 0; | 390 | return 0; |
| @@ -495,25 +495,28 @@ static int rz_mtu3_count_enable_read(struct counter_device *counter, · 104 unmodified lines | |||
| 495 | static int rz_mtu3_count_enable_write(struct counter_device *counter, | 495 | static int rz_mtu3_count_enable_write(struct counter_device *counter, |
| 496 | struct counter_count *count, u8 enable) | 496 | struct counter_count *count, u8 enable) |
| 497 | { | 497 | { |
| 498 | struct rz_mtu3_channel *const ch = rz_mtu3_get_ch(counter, count->id); | ||
| 499 | struct rz_mtu3_cnt *const priv = counter_priv(counter); | 498 | struct rz_mtu3_cnt *const priv = counter_priv(counter); |
| 500 | int ret = 0; | 499 | int ret = 0; |
| 501 | 500 | ||
| 501 | mutex_lock(&priv->lock); | ||
| 502 | |||
| 503 | if (priv->count_is_enabled[count->id] == enable) | ||
| 504 | goto exit; | ||
| 505 | |||
| 502 | if (enable) { | 506 | if (enable) { |
| 503 | mutex_lock(&priv->lock); | 507 | pm_runtime_get_sync(counter->parent); |
| 504 | pm_runtime_get_sync(ch->dev); | ||
| 505 | ret = rz_mtu3_initialize_counter(counter, count->id); | 508 | ret = rz_mtu3_initialize_counter(counter, count->id); |
| 506 | if (ret == 0) | 509 | if (ret == 0) |
| 507 | priv->count_is_enabled[count->id] = true; | 510 | priv->count_is_enabled[count->id] = true; |
| 508 | mutex_unlock(&priv->lock); | ||
| 509 | } else { | 511 | } else { |
| 510 | mutex_lock(&priv->lock); | ||
| 511 | rz_mtu3_terminate_counter(counter, count->id); | 512 | rz_mtu3_terminate_counter(counter, count->id); |
| 512 | priv->count_is_enabled[count->id] = false; | 513 | priv->count_is_enabled[count->id] = false; |
| 513 | pm_runtime_put(ch->dev); | 514 | pm_runtime_put(counter->parent); |
| 514 | mutex_unlock(&priv->lock); | ||
| 515 | } | 515 | } |
| 516 | 516 | ||
| 517 | exit: | ||
| 518 | mutex_unlock(&priv->lock); | ||
| 519 | |||
| 517 | return ret; | 520 | return ret; |
| 518 | } | 521 | } |
| 519 | 522 | ||
| @@ -540,9 +543,9 @@ static int rz_mtu3_cascade_counts_enable_get(struct counter_device *counter, · 20 unmodified lines | |||
| 540 | if (ret) | 543 | if (ret) |
| 541 | return ret; | 544 | return ret; |
| 542 | 545 | ||
| 543 | pm_runtime_get_sync(priv->ch->dev); | 546 | pm_runtime_get_sync(counter->parent); |
| 544 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); | 547 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); |
| 545 | pm_runtime_put(priv->ch->dev); | 548 | pm_runtime_put(counter->parent); |
| 546 | *cascade_enable = test_bit(RZ_MTU3_TMDR3_LWA, &tmdr); | 549 | *cascade_enable = test_bit(RZ_MTU3_TMDR3_LWA, &tmdr); |
| 547 | mutex_unlock(&priv->lock); | 550 | mutex_unlock(&priv->lock); |
| 548 | 551 | ||
| @@ -559,10 +562,10 @@ static int rz_mtu3_cascade_counts_enable_set(struct counter_device *counter, · 10 unmodified lines | |||
| 559 | if (ret) | 562 | if (ret) |
| 560 | return ret; | 563 | return ret; |
| 561 | 564 | ||
| 562 | pm_runtime_get_sync(priv->ch->dev); | 565 | pm_runtime_get_sync(counter->parent); |
| 563 | rz_mtu3_shared_reg_update_bit(priv->ch, RZ_MTU3_TMDR3, | 566 | rz_mtu3_shared_reg_update_bit(priv->ch, RZ_MTU3_TMDR3, |
| 564 | RZ_MTU3_TMDR3_LWA, cascade_enable); | 567 | RZ_MTU3_TMDR3_LWA, cascade_enable); |
| 565 | pm_runtime_put(priv->ch->dev); | 568 | pm_runtime_put(counter->parent); |
| 566 | mutex_unlock(&priv->lock); | 569 | mutex_unlock(&priv->lock); |
| 567 | 570 | ||
| 568 | return 0; | 571 | return 0; |
| @@ -579,9 +582,9 @@ static int rz_mtu3_ext_input_phase_clock_select_get(struct counter_device *count · 10 unmodified lines | |||
| 579 | if (ret) | 582 | if (ret) |
| 580 | return ret; | 583 | return ret; |
| 581 | 584 | ||
| 582 | pm_runtime_get_sync(priv->ch->dev); | 585 | pm_runtime_get_sync(counter->parent); |
| 583 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); | 586 | tmdr = rz_mtu3_shared_reg_read(priv->ch, RZ_MTU3_TMDR3); |
| 584 | pm_runtime_put(priv->ch->dev); | 587 | pm_runtime_put(counter->parent); |
| 585 | *ext_input_phase_clock_select = test_bit(RZ_MTU3_TMDR3_PHCKSEL, &tmdr); | 588 | *ext_input_phase_clock_select = test_bit(RZ_MTU3_TMDR3_PHCKSEL, &tmdr); |
| 586 | mutex_unlock(&priv->lock); | 589 | mutex_unlock(&priv->lock); |
| 587 | 590 | ||
| @@ -598,11 +601,11 @@ static int rz_mtu3_ext_input_phase_clock_select_set(struct counter_device *count · 10 unmodified lines | |||
| 598 | if (ret) | 601 | if (ret) |
| 599 | return ret; | 602 | return ret; |
| 600 | 603 | ||
| 601 | pm_runtime_get_sync(priv->ch->dev); | 604 | pm_runtime_get_sync(counter->parent); |
| 602 | rz_mtu3_shared_reg_update_bit(priv->ch, RZ_MTU3_TMDR3, | 605 | rz_mtu3_shared_reg_update_bit(priv->ch, RZ_MTU3_TMDR3, |
| 603 | RZ_MTU3_TMDR3_PHCKSEL, | 606 | RZ_MTU3_TMDR3_PHCKSEL, |
| 604 | ext_input_phase_clock_select); | 607 | ext_input_phase_clock_select); |
| 605 | pm_runtime_put(priv->ch->dev); | 608 | pm_runtime_put(counter->parent); |
| 606 | mutex_unlock(&priv->lock); | 609 | mutex_unlock(&priv->lock); |
| 607 | 610 | ||
| 608 | return 0; | 611 | return 0; |
| @@ -640,7 +643,7 @@ static int rz_mtu3_action_read(struct counter_device *counter, · 31 unmodified lines | |||
| 640 | if (ret) | 643 | if (ret) |
| 641 | return ret; | 644 | return ret; |
| 642 | 645 | ||
| 643 | ret = rz_mtu3_count_function_read_helper(ch, priv, &function); | 646 | ret = rz_mtu3_count_function_read_helper(ch, counter, &function); |
| 644 | if (ret) { | 647 | if (ret) { |
| 645 | mutex_unlock(&priv->lock); | 648 | mutex_unlock(&priv->lock); |
| 646 | return ret; | 649 | return ret; |
| @@ -1427,12 +1427,9 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, · 1426 unmodified lines | |||
| 1427 | * If there is a problem with its frequency table, take it | 1427 | * If there is a problem with its frequency table, take it |
| 1428 | * offline and drop it. | 1428 | * offline and drop it. |
| 1429 | */ | 1429 | */ |
| 1430 | if (policy->freq_table_sorted != CPUFREQ_TABLE_SORTED_ASCENDING && | 1430 | ret = cpufreq_table_validate_and_sort(policy); |
| 1431 | policy->freq_table_sorted != CPUFREQ_TABLE_SORTED_DESCENDING) { | 1431 | if (ret) |
| 1432 | ret = cpufreq_table_validate_and_sort(policy); | 1432 | goto out_offline_policy; |
| 1433 | if (ret) | ||
| 1434 | goto out_offline_policy; | ||
| 1435 | } | ||
| 1436 | 1433 | ||
| 1437 | /* related_cpus should at least include policy->cpus. */ | 1434 | /* related_cpus should at least include policy->cpus. */ |
| 1438 | cpumask_copy(policy->related_cpus, policy->cpus); | 1435 | cpumask_copy(policy->related_cpus, policy->cpus); |
| @@ -313,6 +313,17 @@ static void cs_start(struct cpufreq_policy *policy) · 312 unmodified lines | |||
| 313 | dbs_info->requested_freq = policy->cur; | 313 | dbs_info->requested_freq = policy->cur; |
| 314 | } | 314 | } |
| 315 | 315 | ||
| 316 | static void cs_limits(struct cpufreq_policy *policy) | ||
| 317 | { | ||
| 318 | struct cs_policy_dbs_info *dbs_info = to_dbs_info(policy->governor_data); | ||
| 319 | |||
| 320 | /* | ||
| 321 | * The limits have changed, so may have the current frequency. Reset | ||
| 322 | * requested_freq to avoid any unintended outcomes due to the mismatch. | ||
| 323 | */ | ||
| 324 | dbs_info->requested_freq = policy->cur; | ||
| 325 | } | ||
| 326 | |||
| 316 | static struct dbs_governor cs_governor = { | 327 | static struct dbs_governor cs_governor = { |
| 317 | .gov = CPUFREQ_DBS_GOVERNOR_INITIALIZER("conservative"), | 328 | .gov = CPUFREQ_DBS_GOVERNOR_INITIALIZER("conservative"), |
| 318 | .kobj_type = { .default_groups = cs_groups }, | 329 | .kobj_type = { .default_groups = cs_groups }, |
| @@ -322,6 +333,7 @@ static struct dbs_governor cs_governor = { · 3 unmodified lines | |||
| 322 | .init = cs_init, | 333 | .init = cs_init, |
| 323 | .exit = cs_exit, | 334 | .exit = cs_exit, |
| 324 | .start = cs_start, | 335 | .start = cs_start, |
| 336 | .limits = cs_limits, | ||
| 325 | }; | 337 | }; |
| 326 | 338 | ||
| 327 | #define CPU_FREQ_GOV_CONSERVATIVE (cs_governor.gov) | 339 | #define CPU_FREQ_GOV_CONSERVATIVE (cs_governor.gov) |
| @@ -468,13 +468,13 @@ int cpufreq_dbs_governor_init(struct cpufreq_policy *policy) · 467 unmodified lines | |||
| 468 | /* Failure, so roll back. */ | 468 | /* Failure, so roll back. */ |
| 469 | pr_err("initialization failed (dbs_data kobject init error %d)\n", ret); | 469 | pr_err("initialization failed (dbs_data kobject init error %d)\n", ret); |
| 470 | 470 | ||
| 471 | kobject_put(&dbs_data->attr_set.kobj); | ||
| 472 | |||
| 473 | policy->governor_data = NULL; | 471 | policy->governor_data = NULL; |
| 474 | 472 | ||
| 475 | if (!have_governor_per_policy()) | 473 | if (!have_governor_per_policy()) |
| 476 | gov->gdbs_data = NULL; | 474 | gov->gdbs_data = NULL; |
| 477 | gov->exit(dbs_data); | 475 | |
| 476 | kobject_put(&dbs_data->attr_set.kobj); | ||
| 477 | goto free_policy_dbs_info; | ||
| 478 | 478 | ||
| 479 | free_dbs_data: | 479 | free_dbs_data: |
| 480 | kfree(dbs_data); | 480 | kfree(dbs_data); |
| @@ -563,6 +563,7 @@ EXPORT_SYMBOL_GPL(cpufreq_dbs_governor_stop); · 82 unmodified lines | |||
| 563 | 563 | ||
| 564 | void cpufreq_dbs_governor_limits(struct cpufreq_policy *policy) | 564 | void cpufreq_dbs_governor_limits(struct cpufreq_policy *policy) |
| 565 | { | 565 | { |
| 566 | struct dbs_governor *gov = dbs_governor_of(policy); | ||
| 566 | struct policy_dbs_info *policy_dbs; | 567 | struct policy_dbs_info *policy_dbs; |
| 567 | 568 | ||
| 568 | /* Protect gov->gdbs_data against cpufreq_dbs_governor_exit() */ | 569 | /* Protect gov->gdbs_data against cpufreq_dbs_governor_exit() */ |
| @@ -574,6 +575,8 @@ void cpufreq_dbs_governor_limits(struct cpufreq_policy *policy) · 5 unmodified lines | |||
| 574 | mutex_lock(&policy_dbs->update_mutex); | 575 | mutex_lock(&policy_dbs->update_mutex); |
| 575 | cpufreq_policy_apply_limits(policy); | 576 | cpufreq_policy_apply_limits(policy); |
| 576 | gov_update_sample_delay(policy_dbs, 0); | 577 | gov_update_sample_delay(policy_dbs, 0); |
| 578 | if (gov->limits) | ||
| 579 | gov->limits(policy); | ||
| 577 | mutex_unlock(&policy_dbs->update_mutex); | 580 | mutex_unlock(&policy_dbs->update_mutex); |
| 578 | 581 | ||
| 579 | out: | 582 | out: |
| @@ -138,6 +138,7 @@ struct dbs_governor { · 137 unmodified lines | |||
| 138 | int (*init)(struct dbs_data *dbs_data); | 138 | int (*init)(struct dbs_data *dbs_data); |
| 139 | void (*exit)(struct dbs_data *dbs_data); | 139 | void (*exit)(struct dbs_data *dbs_data); |
| 140 | void (*start)(struct cpufreq_policy *policy); | 140 | void (*start)(struct cpufreq_policy *policy); |
| 141 | void (*limits)(struct cpufreq_policy *policy); | ||
| 141 | }; | 142 | }; |
| 142 | 143 | ||
| 143 | static inline struct dbs_governor *dbs_governor_of(struct cpufreq_policy *policy) | 144 | static inline struct dbs_governor *dbs_governor_of(struct cpufreq_policy *policy) |
| @@ -360,6 +360,10 @@ int cpufreq_table_validate_and_sort(struct cpufreq_policy *policy) · 359 unmodified lines | |||
| 360 | if (policy_has_boost_freq(policy)) | 360 | if (policy_has_boost_freq(policy)) |
| 361 | policy->boost_supported = true; | 361 | policy->boost_supported = true; |
| 362 | 362 | ||
| 363 | if (policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_ASCENDING || | ||
| 364 | policy->freq_table_sorted == CPUFREQ_TABLE_SORTED_DESCENDING) | ||
| 365 | return 0; | ||
| 366 | |||
| 363 | return set_freq_table_sorted(policy); | 367 | return set_freq_table_sorted(policy); |
| 364 | } | 368 | } |
| 365 | 369 | ||
| @@ -359,16 +359,6 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, · 358 unmodified lines | |||
| 359 | int cpuidle_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, | 359 | int cpuidle_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, |
| 360 | bool *stop_tick) | 360 | bool *stop_tick) |
| 361 | { | 361 | { |
| 362 | /* | ||
| 363 | * If there is only a single idle state (or none), there is nothing | ||
| 364 | * meaningful for the governor to choose. Skip the governor and | ||
| 365 | * always use state 0 with the tick running. | ||
| 366 | */ | ||
| 367 | if (drv->state_count <= 1) { | ||
| 368 | *stop_tick = false; | ||
| 369 | return 0; | ||
| 370 | } | ||
| 371 | |||
| 372 | return cpuidle_curr_governor->select(drv, dev, stop_tick); | 362 | return cpuidle_curr_governor->select(drv, dev, stop_tick); |
| 373 | } | 363 | } |
| 374 | 364 | ||
| @@ -3326,9 +3326,10 @@ static int ahash_setkey(struct crypto_ahash *ahash, const u8 *key, · 3325 unmodified lines | |||
| 3326 | if (aligned_len < keylen) | 3326 | if (aligned_len < keylen) |
| 3327 | return -EOVERFLOW; | 3327 | return -EOVERFLOW; |
| 3328 | 3328 | ||
| 3329 | hashed_key = kmemdup(key, aligned_len, GFP_KERNEL); | 3329 | hashed_key = kmalloc(aligned_len, GFP_KERNEL); |
| 3330 | if (!hashed_key) | 3330 | if (!hashed_key) |
| 3331 | return -ENOMEM; | 3331 | return -ENOMEM; |
| 3332 | memcpy(hashed_key, key, keylen); | ||
| 3332 | ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize); | 3333 | ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize); |
| 3333 | if (ret) | 3334 | if (ret) |
| 3334 | goto bad_free_key; | 3335 | goto bad_free_key; |
| @@ -441,9 +441,10 @@ static int ahash_setkey(struct crypto_ahash *ahash, · 440 unmodified lines | |||
| 441 | if (aligned_len < keylen) | 441 | if (aligned_len < keylen) |
| 442 | return -EOVERFLOW; | 442 | return -EOVERFLOW; |
| 443 | 443 | ||
| 444 | hashed_key = kmemdup(key, keylen, GFP_KERNEL); | 444 | hashed_key = kmalloc(aligned_len, GFP_KERNEL); |
| 445 | if (!hashed_key) | 445 | if (!hashed_key) |
| 446 | return -ENOMEM; | 446 | return -ENOMEM; |
| 447 | memcpy(hashed_key, key, keylen); | ||
| 447 | ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize); | 448 | ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize); |
| 448 | if (ret) | 449 | if (ret) |
| 449 | goto bad_free_key; | 450 | goto bad_free_key; |
| @@ -2408,10 +2408,8 @@ static int sev_ioctl_do_snp_platform_status(struct sev_issue_cmd *argp) · 2407 unmodified lines | |||
| 2408 | * in Firmware state on failure. Use snp_reclaim_pages() to | 2408 | * in Firmware state on failure. Use snp_reclaim_pages() to |
| 2409 | * transition either case back to Hypervisor-owned state. | 2409 | * transition either case back to Hypervisor-owned state. |
| 2410 | */ | 2410 | */ |
| 2411 | if (snp_reclaim_pages(__pa(data), 1, true)) { | 2411 | if (snp_reclaim_pages(__pa(data), 1, true)) |
| 2412 | snp_leak_pages(__page_to_pfn(status_page), 1); | ||
| 2413 | return -EFAULT; | 2412 | return -EFAULT; |
| 2414 | } | ||
| 2415 | } | 2413 | } |
| 2416 | 2414 | ||
| 2417 | if (ret) | 2415 | if (ret) |
| @@ -332,6 +332,13 @@ static int __init padlock_init(void) · 331 unmodified lines | |||
| 332 | if (!x86_match_cpu(padlock_sha_ids) || !boot_cpu_has(X86_FEATURE_PHE_EN)) | 332 | if (!x86_match_cpu(padlock_sha_ids) || !boot_cpu_has(X86_FEATURE_PHE_EN)) |
| 333 | return -ENODEV; | 333 | return -ENODEV; |
| 334 | 334 | ||
| 335 | /* | ||
| 336 | * Skip family 0x07 and newer used by Zhaoxin processors, | ||
| 337 | * as the driver's self-tests fail on these CPUs. | ||
| 338 | */ | ||
| 339 | if (c->x86 >= 0x07) | ||
| 340 | return -ENODEV; | ||
| 341 | |||
| 335 | /* Register the newly added algorithm module if on * | 342 | /* Register the newly added algorithm module if on * |
| 336 | * VIA Nano processor, or else just do as before */ | 343 | * VIA Nano processor, or else just do as before */ |
| 337 | if (c->x86_model < 0x0f) { | 344 | if (c->x86_model < 0x0f) { |
| @@ -529,7 +529,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 528 unmodified lines | |||
| 529 | .cra_name = "cbc(aes)", | 529 | .cra_name = "cbc(aes)", |
| 530 | .cra_driver_name = "cbc-aes-tegra", | 530 | .cra_driver_name = "cbc-aes-tegra", |
| 531 | .cra_priority = 500, | 531 | .cra_priority = 500, |
| 532 | .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | CRYPTO_ALG_ASYNC, | 532 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 533 | .cra_blocksize = AES_BLOCK_SIZE, | 533 | .cra_blocksize = AES_BLOCK_SIZE, |
| 534 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), | 534 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 535 | .cra_alignmask = 0xf, | 535 | .cra_alignmask = 0xf, |
| @@ -550,7 +550,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 14 unmodified lines | |||
| 550 | .cra_name = "ecb(aes)", | 550 | .cra_name = "ecb(aes)", |
| 551 | .cra_driver_name = "ecb-aes-tegra", | 551 | .cra_driver_name = "ecb-aes-tegra", |
| 552 | .cra_priority = 500, | 552 | .cra_priority = 500, |
| 553 | .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | CRYPTO_ALG_ASYNC, | 553 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 554 | .cra_blocksize = AES_BLOCK_SIZE, | 554 | .cra_blocksize = AES_BLOCK_SIZE, |
| 555 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), | 555 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 556 | .cra_alignmask = 0xf, | 556 | .cra_alignmask = 0xf, |
| @@ -572,7 +572,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 15 unmodified lines | |||
| 572 | .cra_name = "ctr(aes)", | 572 | .cra_name = "ctr(aes)", |
| 573 | .cra_driver_name = "ctr-aes-tegra", | 573 | .cra_driver_name = "ctr-aes-tegra", |
| 574 | .cra_priority = 500, | 574 | .cra_priority = 500, |
| 575 | .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | CRYPTO_ALG_ASYNC, | 575 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 576 | .cra_blocksize = 1, | 576 | .cra_blocksize = 1, |
| 577 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), | 577 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 578 | .cra_alignmask = 0xf, | 578 | .cra_alignmask = 0xf, |
| @@ -594,6 +594,7 @@ static struct tegra_se_alg tegra_aes_algs[] = { · 15 unmodified lines | |||
| 594 | .cra_name = "xts(aes)", | 594 | .cra_name = "xts(aes)", |
| 595 | .cra_driver_name = "xts-aes-tegra", | 595 | .cra_driver_name = "xts-aes-tegra", |
| 596 | .cra_priority = 500, | 596 | .cra_priority = 500, |
| 597 | .cra_flags = CRYPTO_ALG_ASYNC, | ||
| 597 | .cra_blocksize = AES_BLOCK_SIZE, | 598 | .cra_blocksize = AES_BLOCK_SIZE, |
| 598 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), | 599 | .cra_ctxsize = sizeof(struct tegra_aes_ctx), |
| 599 | .cra_alignmask = (__alignof__(u64) - 1), | 600 | .cra_alignmask = (__alignof__(u64) - 1), |
| @@ -1922,6 +1923,7 @@ static struct tegra_se_alg tegra_aead_algs[] = { · 1322 unmodified lines | |||
| 1922 | .cra_name = "gcm(aes)", | 1923 | .cra_name = "gcm(aes)", |
| 1923 | .cra_driver_name = "gcm-aes-tegra", | 1924 | .cra_driver_name = "gcm-aes-tegra", |
| 1924 | .cra_priority = 500, | 1925 | .cra_priority = 500, |
| 1926 | .cra_flags = CRYPTO_ALG_ASYNC, | ||
| 1925 | .cra_blocksize = 1, | 1927 | .cra_blocksize = 1, |
| 1926 | .cra_ctxsize = sizeof(struct tegra_aead_ctx), | 1928 | .cra_ctxsize = sizeof(struct tegra_aead_ctx), |
| 1927 | .cra_alignmask = 0xf, | 1929 | .cra_alignmask = 0xf, |
| @@ -1944,6 +1946,7 @@ static struct tegra_se_alg tegra_aead_algs[] = { · 16 unmodified lines | |||
| 1944 | .cra_name = "ccm(aes)", | 1946 | .cra_name = "ccm(aes)", |
| 1945 | .cra_driver_name = "ccm-aes-tegra", | 1947 | .cra_driver_name = "ccm-aes-tegra", |
| 1946 | .cra_priority = 500, | 1948 | .cra_priority = 500, |
| 1949 | .cra_flags = CRYPTO_ALG_ASYNC, | ||
| 1947 | .cra_blocksize = 1, | 1950 | .cra_blocksize = 1, |
| 1948 | .cra_ctxsize = sizeof(struct tegra_aead_ctx), | 1951 | .cra_ctxsize = sizeof(struct tegra_aead_ctx), |
| 1949 | .cra_alignmask = 0xf, | 1952 | .cra_alignmask = 0xf, |
| @@ -1971,7 +1974,7 @@ static struct tegra_se_alg tegra_cmac_algs[] = { · 21 unmodified lines | |||
| 1971 | .cra_name = "cmac(aes)", | 1974 | .cra_name = "cmac(aes)", |
| 1972 | .cra_driver_name = "tegra-se-cmac", | 1975 | .cra_driver_name = "tegra-se-cmac", |
| 1973 | .cra_priority = 300, | 1976 | .cra_priority = 300, |
| 1974 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 1977 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 1975 | .cra_blocksize = AES_BLOCK_SIZE, | 1978 | .cra_blocksize = AES_BLOCK_SIZE, |
| 1976 | .cra_ctxsize = sizeof(struct tegra_cmac_ctx), | 1979 | .cra_ctxsize = sizeof(struct tegra_cmac_ctx), |
| 1977 | .cra_alignmask = 0, | 1980 | .cra_alignmask = 0, |
| @@ -761,7 +761,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 760 unmodified lines | |||
| 761 | .cra_name = "sha1", | 761 | .cra_name = "sha1", |
| 762 | .cra_driver_name = "tegra-se-sha1", | 762 | .cra_driver_name = "tegra-se-sha1", |
| 763 | .cra_priority = 300, | 763 | .cra_priority = 300, |
| 764 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 764 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 765 | .cra_blocksize = SHA1_BLOCK_SIZE, | 765 | .cra_blocksize = SHA1_BLOCK_SIZE, |
| 766 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 766 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 767 | .cra_alignmask = 0, | 767 | .cra_alignmask = 0, |
| @@ -786,7 +786,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 786 | .cra_name = "sha224", | 786 | .cra_name = "sha224", |
| 787 | .cra_driver_name = "tegra-se-sha224", | 787 | .cra_driver_name = "tegra-se-sha224", |
| 788 | .cra_priority = 300, | 788 | .cra_priority = 300, |
| 789 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 789 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 790 | .cra_blocksize = SHA224_BLOCK_SIZE, | 790 | .cra_blocksize = SHA224_BLOCK_SIZE, |
| 791 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 791 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 792 | .cra_alignmask = 0, | 792 | .cra_alignmask = 0, |
| @@ -811,7 +811,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 811 | .cra_name = "sha256", | 811 | .cra_name = "sha256", |
| 812 | .cra_driver_name = "tegra-se-sha256", | 812 | .cra_driver_name = "tegra-se-sha256", |
| 813 | .cra_priority = 300, | 813 | .cra_priority = 300, |
| 814 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 814 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 815 | .cra_blocksize = SHA256_BLOCK_SIZE, | 815 | .cra_blocksize = SHA256_BLOCK_SIZE, |
| 816 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 816 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 817 | .cra_alignmask = 0, | 817 | .cra_alignmask = 0, |
| @@ -836,7 +836,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 836 | .cra_name = "sha384", | 836 | .cra_name = "sha384", |
| 837 | .cra_driver_name = "tegra-se-sha384", | 837 | .cra_driver_name = "tegra-se-sha384", |
| 838 | .cra_priority = 300, | 838 | .cra_priority = 300, |
| 839 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 839 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 840 | .cra_blocksize = SHA384_BLOCK_SIZE, | 840 | .cra_blocksize = SHA384_BLOCK_SIZE, |
| 841 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 841 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 842 | .cra_alignmask = 0, | 842 | .cra_alignmask = 0, |
| @@ -861,7 +861,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 861 | .cra_name = "sha512", | 861 | .cra_name = "sha512", |
| 862 | .cra_driver_name = "tegra-se-sha512", | 862 | .cra_driver_name = "tegra-se-sha512", |
| 863 | .cra_priority = 300, | 863 | .cra_priority = 300, |
| 864 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 864 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 865 | .cra_blocksize = SHA512_BLOCK_SIZE, | 865 | .cra_blocksize = SHA512_BLOCK_SIZE, |
| 866 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 866 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 867 | .cra_alignmask = 0, | 867 | .cra_alignmask = 0, |
| @@ -886,7 +886,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 886 | .cra_name = "sha3-224", | 886 | .cra_name = "sha3-224", |
| 887 | .cra_driver_name = "tegra-se-sha3-224", | 887 | .cra_driver_name = "tegra-se-sha3-224", |
| 888 | .cra_priority = 300, | 888 | .cra_priority = 300, |
| 889 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 889 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 890 | .cra_blocksize = SHA3_224_BLOCK_SIZE, | 890 | .cra_blocksize = SHA3_224_BLOCK_SIZE, |
| 891 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 891 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 892 | .cra_alignmask = 0, | 892 | .cra_alignmask = 0, |
| @@ -911,7 +911,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 911 | .cra_name = "sha3-256", | 911 | .cra_name = "sha3-256", |
| 912 | .cra_driver_name = "tegra-se-sha3-256", | 912 | .cra_driver_name = "tegra-se-sha3-256", |
| 913 | .cra_priority = 300, | 913 | .cra_priority = 300, |
| 914 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 914 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 915 | .cra_blocksize = SHA3_256_BLOCK_SIZE, | 915 | .cra_blocksize = SHA3_256_BLOCK_SIZE, |
| 916 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 916 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 917 | .cra_alignmask = 0, | 917 | .cra_alignmask = 0, |
| @@ -936,7 +936,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 936 | .cra_name = "sha3-384", | 936 | .cra_name = "sha3-384", |
| 937 | .cra_driver_name = "tegra-se-sha3-384", | 937 | .cra_driver_name = "tegra-se-sha3-384", |
| 938 | .cra_priority = 300, | 938 | .cra_priority = 300, |
| 939 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 939 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 940 | .cra_blocksize = SHA3_384_BLOCK_SIZE, | 940 | .cra_blocksize = SHA3_384_BLOCK_SIZE, |
| 941 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 941 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 942 | .cra_alignmask = 0, | 942 | .cra_alignmask = 0, |
| @@ -961,7 +961,7 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 18 unmodified lines | |||
| 961 | .cra_name = "sha3-512", | 961 | .cra_name = "sha3-512", |
| 962 | .cra_driver_name = "tegra-se-sha3-512", | 962 | .cra_driver_name = "tegra-se-sha3-512", |
| 963 | .cra_priority = 300, | 963 | .cra_priority = 300, |
| 964 | .cra_flags = CRYPTO_ALG_TYPE_AHASH, | 964 | .cra_flags = CRYPTO_ALG_ASYNC, |
| 965 | .cra_blocksize = SHA3_512_BLOCK_SIZE, | 965 | .cra_blocksize = SHA3_512_BLOCK_SIZE, |
| 966 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 966 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 967 | .cra_alignmask = 0, | 967 | .cra_alignmask = 0, |
| @@ -988,7 +988,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | |||
| 988 | .cra_name = "hmac(sha224)", | 988 | .cra_name = "hmac(sha224)", |
| 989 | .cra_driver_name = "tegra-se-hmac-sha224", | 989 | .cra_driver_name = "tegra-se-hmac-sha224", |
| 990 | .cra_priority = 300, | 990 | .cra_priority = 300, |
| 991 | .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | 991 | .cra_flags = CRYPTO_ALG_ASYNC | |
| 992 | CRYPTO_ALG_NEED_FALLBACK, | ||
| 992 | .cra_blocksize = SHA224_BLOCK_SIZE, | 993 | .cra_blocksize = SHA224_BLOCK_SIZE, |
| 993 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 994 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 994 | .cra_alignmask = 0, | 995 | .cra_alignmask = 0, |
| @@ -1015,7 +1016,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | |||
| 1015 | .cra_name = "hmac(sha256)", | 1016 | .cra_name = "hmac(sha256)", |
| 1016 | .cra_driver_name = "tegra-se-hmac-sha256", | 1017 | .cra_driver_name = "tegra-se-hmac-sha256", |
| 1017 | .cra_priority = 300, | 1018 | .cra_priority = 300, |
| 1018 | .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | 1019 | .cra_flags = CRYPTO_ALG_ASYNC | |
| 1020 | CRYPTO_ALG_NEED_FALLBACK, | ||
| 1019 | .cra_blocksize = SHA256_BLOCK_SIZE, | 1021 | .cra_blocksize = SHA256_BLOCK_SIZE, |
| 1020 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 1022 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 1021 | .cra_alignmask = 0, | 1023 | .cra_alignmask = 0, |
| @@ -1042,7 +1044,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | |||
| 1042 | .cra_name = "hmac(sha384)", | 1044 | .cra_name = "hmac(sha384)", |
| 1043 | .cra_driver_name = "tegra-se-hmac-sha384", | 1045 | .cra_driver_name = "tegra-se-hmac-sha384", |
| 1044 | .cra_priority = 300, | 1046 | .cra_priority = 300, |
| 1045 | .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | 1047 | .cra_flags = CRYPTO_ALG_ASYNC | |
| 1048 | CRYPTO_ALG_NEED_FALLBACK, | ||
| 1046 | .cra_blocksize = SHA384_BLOCK_SIZE, | 1049 | .cra_blocksize = SHA384_BLOCK_SIZE, |
| 1047 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 1050 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 1048 | .cra_alignmask = 0, | 1051 | .cra_alignmask = 0, |
| @@ -1069,7 +1072,8 @@ static struct tegra_se_alg tegra_hash_algs[] = { · 20 unmodified lines | |||
| 1069 | .cra_name = "hmac(sha512)", | 1072 | .cra_name = "hmac(sha512)", |
| 1070 | .cra_driver_name = "tegra-se-hmac-sha512", | 1073 | .cra_driver_name = "tegra-se-hmac-sha512", |
| 1071 | .cra_priority = 300, | 1074 | .cra_priority = 300, |
| 1072 | .cra_flags = CRYPTO_ALG_TYPE_AHASH | CRYPTO_ALG_NEED_FALLBACK, | 1075 | .cra_flags = CRYPTO_ALG_ASYNC | |
| 1076 | CRYPTO_ALG_NEED_FALLBACK, | ||
| 1073 | .cra_blocksize = SHA512_BLOCK_SIZE, | 1077 | .cra_blocksize = SHA512_BLOCK_SIZE, |
| 1074 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), | 1078 | .cra_ctxsize = sizeof(struct tegra_sha_ctx), |
| 1075 | .cra_alignmask = 0, | 1079 | .cra_alignmask = 0, |
| @@ -59,6 +59,7 @@ config CXL_ACPI · 58 unmodified lines | |||
| 59 | tristate "CXL ACPI: Platform Support" | 59 | tristate "CXL ACPI: Platform Support" |
| 60 | depends on ACPI | 60 | depends on ACPI |
| 61 | depends on ACPI_NUMA | 61 | depends on ACPI_NUMA |
| 62 | depends on CXL_PMEM || !CXL_PMEM | ||
| 62 | default CXL_BUS | 63 | default CXL_BUS |
| 63 | select ACPI_TABLE_LIB | 64 | select ACPI_TABLE_LIB |
| 64 | select ACPI_HMAT | 65 | select ACPI_HMAT |
| @@ -94,7 +94,6 @@ static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info) · 93 unmodified lines | |||
| 94 | struct cxl_hdm *cxlhdm; | 94 | struct cxl_hdm *cxlhdm; |
| 95 | void __iomem *hdm; | 95 | void __iomem *hdm; |
| 96 | u32 ctrl; | 96 | u32 ctrl; |
| 97 | int i; | ||
| 98 | 97 | ||
| 99 | if (!info) | 98 | if (!info) |
| 100 | return false; | 99 | return false; |
| @@ -113,22 +112,16 @@ static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info) · 12 unmodified lines | |||
| 113 | return false; | 112 | return false; |
| 114 | 113 | ||
| 115 | /* | 114 | /* |
| 116 | * If any decoders are committed already, there should not be any | 115 | * If HDM decoders are globally enabled, do not fall back to DVSEC |
| 117 | * emulated DVSEC decoders. | 116 | * range emulation. Zeroed decoder registers after region teardown |
| 117 | * do not imply absence of HDM capability. | ||
| 118 | * | ||
| 119 | * Falling back to DVSEC here would treat the decoder as AUTO and | ||
| 120 | * may incorrectly latch default interleave settings. | ||
| 118 | */ | 121 | */ |
| 119 | for (i = 0; i < cxlhdm->decoder_count; i++) { | 122 | ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); |
| 120 | ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); | 123 | if (ctrl & CXL_HDM_DECODER_ENABLE) |
| 121 | dev_dbg(&info->port->dev, | 124 | return false; |
| 122 | "decoder%d.%d: committed: %ld base: %#x_%.8x size: %#x_%.8x\n", | ||
| 123 | info->port->id, i, | ||
| 124 | FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl), | ||
| 125 | readl(hdm + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i)), | ||
| 126 | readl(hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(i)), | ||
| 127 | readl(hdm + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i)), | ||
| 128 | readl(hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i))); | ||
| 129 | if (FIELD_GET(CXL_HDM_DECODER0_CTRL_COMMITTED, ctrl)) | ||
| 130 | return false; | ||
| 131 | } | ||
| 132 | 125 | ||
| 133 | return true; | 126 | return true; |
| 134 | } | 127 | } |
| @@ -1301,7 +1301,7 @@ int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd) · 1300 unmodified lines | |||
| 1301 | * Require an endpoint to be safe otherwise the driver can not | 1301 | * Require an endpoint to be safe otherwise the driver can not |
| 1302 | * be sure that the device is unmapped. | 1302 | * be sure that the device is unmapped. |
| 1303 | */ | 1303 | */ |
| 1304 | if (endpoint && cxl_num_decoders_committed(endpoint) == 0) | 1304 | if (cxlmd->dev.driver && cxl_num_decoders_committed(endpoint) == 0) |
| 1305 | return __cxl_mem_sanitize(mds, cmd); | 1305 | return __cxl_mem_sanitize(mds, cmd); |
| 1306 | 1306 | ||
| 1307 | return -EBUSY; | 1307 | return -EBUSY; |
| @@ -552,10 +552,13 @@ static void cxl_port_release(struct device *dev) · 551 unmodified lines | |||
| 552 | xa_destroy(&port->dports); | 552 | xa_destroy(&port->dports); |
| 553 | xa_destroy(&port->regions); | 553 | xa_destroy(&port->regions); |
| 554 | ida_free(&cxl_port_ida, port->id); | 554 | ida_free(&cxl_port_ida, port->id); |
| 555 | if (is_cxl_root(port)) | 555 | |
| 556 | if (is_cxl_root(port)) { | ||
| 556 | kfree(to_cxl_root(port)); | 557 | kfree(to_cxl_root(port)); |
| 557 | else | 558 | } else { |
| 559 | put_device(dev->parent); | ||
| 558 | kfree(port); | 560 | kfree(port); |
| 561 | } | ||
| 559 | } | 562 | } |
| 560 | 563 | ||
| 561 | static ssize_t decoders_committed_show(struct device *dev, | 564 | static ssize_t decoders_committed_show(struct device *dev, |
| @@ -707,6 +710,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport_dev, · 145 unmodified lines | |||
| 707 | struct cxl_port *iter; | 710 | struct cxl_port *iter; |
| 708 | 711 | ||
| 709 | dev->parent = &parent_port->dev; | 712 | dev->parent = &parent_port->dev; |
| 713 | get_device(dev->parent); | ||
| 710 | port->depth = parent_port->depth + 1; | 714 | port->depth = parent_port->depth + 1; |
| 711 | port->parent_dport = parent_dport; | 715 | port->parent_dport = parent_dport; |
| 712 | 716 | ||
| @@ -3854,8 +3854,10 @@ static int __construct_region(struct cxl_region *cxlr, · 3853 unmodified lines | |||
| 3854 | } | 3854 | } |
| 3855 | 3855 | ||
| 3856 | rc = sysfs_update_group(&cxlr->dev.kobj, &cxl_region_group); | 3856 | rc = sysfs_update_group(&cxlr->dev.kobj, &cxl_region_group); |
| 3857 | if (rc) | 3857 | if (rc) { |
| 3858 | kfree(res); | ||
| 3858 | return rc; | 3859 | return rc; |
| 3860 | } | ||
| 3859 | 3861 | ||
| 3860 | rc = insert_resource(cxlrd->res, res); | 3862 | rc = insert_resource(cxlrd->res, res); |
| 3861 | if (rc) { | 3863 | if (rc) { |
| @@ -554,7 +554,7 @@ static __exit void cxl_pmem_exit(void) · 553 unmodified lines | |||
| 554 | 554 | ||
| 555 | MODULE_DESCRIPTION("CXL PMEM: Persistent Memory Support"); | 555 | MODULE_DESCRIPTION("CXL PMEM: Persistent Memory Support"); |
| 556 | MODULE_LICENSE("GPL v2"); | 556 | MODULE_LICENSE("GPL v2"); |
| 557 | module_init(cxl_pmem_init); | 557 | subsys_initcall(cxl_pmem_init); |
| 558 | module_exit(cxl_pmem_exit); | 558 | module_exit(cxl_pmem_exit); |
| 559 | MODULE_IMPORT_NS("CXL"); | 559 | MODULE_IMPORT_NS("CXL"); |
| 560 | MODULE_ALIAS_CXL(CXL_DEVICE_NVDIMM_BRIDGE); | 560 | MODULE_ALIAS_CXL(CXL_DEVICE_NVDIMM_BRIDGE); |
| @@ -844,6 +844,7 @@ static int dw_edma_irq_request(struct dw_edma *dw, · 843 unmodified lines | |||
| 844 | { | 844 | { |
| 845 | struct dw_edma_chip *chip = dw->chip; | 845 | struct dw_edma_chip *chip = dw->chip; |
| 846 | struct device *dev = dw->chip->dev; | 846 | struct device *dev = dw->chip->dev; |
| 847 | struct msi_desc *msi_desc; | ||
| 847 | u32 wr_mask = 1; | 848 | u32 wr_mask = 1; |
| 848 | u32 rd_mask = 1; | 849 | u32 rd_mask = 1; |
| 849 | int i, err = 0; | 850 | int i, err = 0; |
| @@ -895,9 +896,12 @@ static int dw_edma_irq_request(struct dw_edma *dw, · 45 unmodified lines | |||
| 895 | &dw->irq[i]); | 896 | &dw->irq[i]); |
| 896 | if (err) | 897 | if (err) |
| 897 | goto err_irq_free; | 898 | goto err_irq_free; |
| 898 | 899 | msi_desc = irq_get_msi_desc(irq); | |
| 899 | if (irq_get_msi_desc(irq)) | 900 | if (msi_desc) { |
| 900 | get_cached_msi_msg(irq, &dw->irq[i].msi); | 901 | get_cached_msi_msg(irq, &dw->irq[i].msi); |
| 902 | if (!msi_desc->pci.msi_attrib.is_msix) | ||
| 903 | dw->irq[i].msi.data = dw->irq[0].msi.data + i; | ||
| 904 | } | ||
| 901 | } | 905 | } |
| 902 | 906 | ||
| 903 | dw->nr_irqs = i; | 907 | dw->nr_irqs = i; |
| @@ -252,10 +252,10 @@ static void dw_hdma_v0_core_start(struct dw_edma_chunk *chunk, bool first) · 251 unmodified lines | |||
| 252 | lower_32_bits(chunk->ll_region.paddr)); | 252 | lower_32_bits(chunk->ll_region.paddr)); |
| 253 | SET_CH_32(dw, chan->dir, chan->id, llp.msb, | 253 | SET_CH_32(dw, chan->dir, chan->id, llp.msb, |
| 254 | upper_32_bits(chunk->ll_region.paddr)); | 254 | upper_32_bits(chunk->ll_region.paddr)); |
| 255 | /* Set consumer cycle */ | ||
| 256 | SET_CH_32(dw, chan->dir, chan->id, cycle_sync, | ||
| 257 | HDMA_V0_CONSUMER_CYCLE_STAT | HDMA_V0_CONSUMER_CYCLE_BIT); | ||
| 255 | } | 258 | } |
| 256 | /* Set consumer cycle */ | ||
| 257 | SET_CH_32(dw, chan->dir, chan->id, cycle_sync, | ||
| 258 | HDMA_V0_CONSUMER_CYCLE_STAT | HDMA_V0_CONSUMER_CYCLE_BIT); | ||
| 259 | 259 | ||
| 260 | dw_hdma_v0_sync_ll_data(chunk); | 260 | dw_hdma_v0_sync_ll_data(chunk); |
| 261 | 261 | ||
| @@ -317,10 +317,8 @@ static struct dma_chan *fsl_edma3_xlate(struct of_phandle_args *dma_spec, · 316 unmodified lines | |||
| 317 | return NULL; | 317 | return NULL; |
| 318 | i = fsl_chan - fsl_edma->chans; | 318 | i = fsl_chan - fsl_edma->chans; |
| 319 | 319 | ||
| 320 | fsl_chan->priority = dma_spec->args[1]; | 320 | if (!b_chmux && i != dma_spec->args[0]) |
| 321 | fsl_chan->is_rxchan = dma_spec->args[2] & FSL_EDMA_RX; | 321 | continue; |
| 322 | fsl_chan->is_remote = dma_spec->args[2] & FSL_EDMA_REMOTE; | ||
| 323 | fsl_chan->is_multi_fifo = dma_spec->args[2] & FSL_EDMA_MULTI_FIFO; | ||
| 324 | 322 | ||
| 325 | if ((dma_spec->args[2] & FSL_EDMA_EVEN_CH) && (i & 0x1)) | 323 | if ((dma_spec->args[2] & FSL_EDMA_EVEN_CH) && (i & 0x1)) |
| 326 | continue; | 324 | continue; |
| @@ -328,17 +326,15 @@ static struct dma_chan *fsl_edma3_xlate(struct of_phandle_args *dma_spec, · 1 unmodified line | |||
| 328 | if ((dma_spec->args[2] & FSL_EDMA_ODD_CH) && !(i & 0x1)) | 326 | if ((dma_spec->args[2] & FSL_EDMA_ODD_CH) && !(i & 0x1)) |
| 329 | continue; | 327 | continue; |
| 330 | 328 | ||
| 331 | if (!b_chmux && i == dma_spec->args[0]) { | 329 | fsl_chan->srcid = dma_spec->args[0]; |
| 332 | chan = dma_get_slave_channel(chan); | 330 | fsl_chan->priority = dma_spec->args[1]; |
| 333 | chan->device->privatecnt++; | 331 | fsl_chan->is_rxchan = dma_spec->args[2] & FSL_EDMA_RX; |
| 334 | return chan; | 332 | fsl_chan->is_remote = dma_spec->args[2] & FSL_EDMA_REMOTE; |
| 335 | } else if (b_chmux && !fsl_chan->srcid) { | 333 | fsl_chan->is_multi_fifo = dma_spec->args[2] & FSL_EDMA_MULTI_FIFO; |
| 336 | /* if controller support channel mux, choose a free channel */ | 334 | |
| 337 | chan = dma_get_slave_channel(chan); | 335 | chan = dma_get_slave_channel(chan); |
| 338 | chan->device->privatecnt++; | 336 | chan->device->privatecnt++; |
| 339 | fsl_chan->srcid = dma_spec->args[0]; | 337 | return chan; |
| 340 | return chan; | ||
| 341 | } | ||
| 342 | } | 338 | } |
| 343 | return NULL; | 339 | return NULL; |
| 344 | } | 340 | } |
| @@ -158,11 +158,7 @@ static const struct device_type idxd_cdev_file_type = { · 157 unmodified lines | |||
| 158 | static void idxd_cdev_dev_release(struct device *dev) | 158 | static void idxd_cdev_dev_release(struct device *dev) |
| 159 | { | 159 | { |
| 160 | struct idxd_cdev *idxd_cdev = dev_to_cdev(dev); | 160 | struct idxd_cdev *idxd_cdev = dev_to_cdev(dev); |
| 161 | struct idxd_cdev_context *cdev_ctx; | ||
| 162 | struct idxd_wq *wq = idxd_cdev->wq; | ||
| 163 | 161 | ||
| 164 | cdev_ctx = &ictx[wq->idxd->data->type]; | ||
| 165 | ida_free(&cdev_ctx->minor_ida, idxd_cdev->minor); | ||
| 166 | kfree(idxd_cdev); | 162 | kfree(idxd_cdev); |
| 167 | } | 163 | } |
| 168 | 164 | ||
| @@ -582,11 +578,15 @@ int idxd_wq_add_cdev(struct idxd_wq *wq) · 413 unmodified lines | |||
| 582 | 578 | ||
| 583 | void idxd_wq_del_cdev(struct idxd_wq *wq) | 579 | void idxd_wq_del_cdev(struct idxd_wq *wq) |
| 584 | { | 580 | { |
| 581 | struct idxd_cdev_context *cdev_ctx; | ||
| 585 | struct idxd_cdev *idxd_cdev; | 582 | struct idxd_cdev *idxd_cdev; |
| 586 | 583 | ||
| 587 | idxd_cdev = wq->idxd_cdev; | 584 | idxd_cdev = wq->idxd_cdev; |
| 588 | wq->idxd_cdev = NULL; | 585 | wq->idxd_cdev = NULL; |
| 589 | cdev_device_del(&idxd_cdev->cdev, cdev_dev(idxd_cdev)); | 586 | cdev_device_del(&idxd_cdev->cdev, cdev_dev(idxd_cdev)); |
| 587 | |||
| 588 | cdev_ctx = &ictx[wq->idxd->data->type]; | ||
| 589 | ida_free(&cdev_ctx->minor_ida, idxd_cdev->minor); | ||
| 590 | put_device(cdev_dev(idxd_cdev)); | 590 | put_device(cdev_dev(idxd_cdev)); |
| 591 | } | 591 | } |
| 592 | 592 | ||
| @@ -175,6 +175,7 @@ void idxd_wq_free_resources(struct idxd_wq *wq) · 174 unmodified lines | |||
| 175 | free_descs(wq); | 175 | free_descs(wq); |
| 176 | dma_free_coherent(dev, wq->compls_size, wq->compls, wq->compls_addr); | 176 | dma_free_coherent(dev, wq->compls_size, wq->compls, wq->compls_addr); |
| 177 | sbitmap_queue_free(&wq->sbq); | 177 | sbitmap_queue_free(&wq->sbq); |
| 178 | wq->type = IDXD_WQT_NONE; | ||
| 178 | } | 179 | } |
| 179 | EXPORT_SYMBOL_NS_GPL(idxd_wq_free_resources, "IDXD"); | 180 | EXPORT_SYMBOL_NS_GPL(idxd_wq_free_resources, "IDXD"); |
| 180 | 181 | ||
| @@ -382,7 +383,6 @@ static void idxd_wq_disable_cleanup(struct idxd_wq *wq) · 201 unmodified lines | |||
| 382 | lockdep_assert_held(&wq->wq_lock); | 383 | lockdep_assert_held(&wq->wq_lock); |
| 383 | wq->state = IDXD_WQ_DISABLED; | 384 | wq->state = IDXD_WQ_DISABLED; |
| 384 | memset(wq->wqcfg, 0, idxd->wqcfg_size); | 385 | memset(wq->wqcfg, 0, idxd->wqcfg_size); |
| 385 | wq->type = IDXD_WQT_NONE; | ||
| 386 | wq->threshold = 0; | 386 | wq->threshold = 0; |
| 387 | wq->priority = 0; | 387 | wq->priority = 0; |
| 388 | wq->enqcmds_retries = IDXD_ENQCMDS_RETRIES; | 388 | wq->enqcmds_retries = IDXD_ENQCMDS_RETRIES; |
| @@ -831,8 +831,7 @@ static void idxd_device_evl_free(struct idxd_device *idxd) · 442 unmodified lines | |||
| 831 | struct device *dev = &idxd->pdev->dev; | 831 | struct device *dev = &idxd->pdev->dev; |
| 832 | struct idxd_evl *evl = idxd->evl; | 832 | struct idxd_evl *evl = idxd->evl; |
| 833 | 833 | ||
| 834 | gencfg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET); | 834 | if (!evl) |
| 835 | if (!gencfg.evl_en) | ||
| 836 | return; | 835 | return; |
| 837 | 836 | ||
| 838 | mutex_lock(&evl->lock); | 837 | mutex_lock(&evl->lock); |
| @@ -1125,7 +1124,11 @@ int idxd_device_config(struct idxd_device *idxd) · 286 unmodified lines | |||
| 1125 | { | 1124 | { |
| 1126 | int rc; | 1125 | int rc; |
| 1127 | 1126 | ||
| 1128 | lockdep_assert_held(&idxd->dev_lock); | 1127 | guard(spinlock)(&idxd->dev_lock); |
| 1128 | |||
| 1129 | if (!test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) | ||
| 1130 | return 0; | ||
| 1131 | |||
| 1129 | rc = idxd_wqs_setup(idxd); | 1132 | rc = idxd_wqs_setup(idxd); |
| 1130 | if (rc < 0) | 1133 | if (rc < 0) |
| 1131 | return rc; | 1134 | return rc; |
| @@ -1332,6 +1335,11 @@ void idxd_wq_free_irq(struct idxd_wq *wq) · 200 unmodified lines | |||
| 1332 | 1335 | ||
| 1333 | free_irq(ie->vector, ie); | 1336 | free_irq(ie->vector, ie); |
| 1334 | idxd_flush_pending_descs(ie); | 1337 | idxd_flush_pending_descs(ie); |
| 1338 | |||
| 1339 | /* The interrupt might have been already released by FLR */ | ||
| 1340 | if (ie->int_handle == INVALID_INT_HANDLE) | ||
| 1341 | return; | ||
| 1342 | |||
| 1335 | if (idxd->request_int_handles) | 1343 | if (idxd->request_int_handles) |
| 1336 | idxd_device_release_int_handle(idxd, ie->int_handle, IDXD_IRQ_MSIX); | 1344 | idxd_device_release_int_handle(idxd, ie->int_handle, IDXD_IRQ_MSIX); |
| 1337 | idxd_device_clear_perm_entry(idxd, ie); | 1345 | idxd_device_clear_perm_entry(idxd, ie); |
| @@ -1340,6 +1348,23 @@ void idxd_wq_free_irq(struct idxd_wq *wq) · 2 unmodified lines | |||
| 1340 | ie->pasid = IOMMU_PASID_INVALID; | 1348 | ie->pasid = IOMMU_PASID_INVALID; |
| 1341 | } | 1349 | } |
| 1342 | 1350 | ||
| 1351 | void idxd_wq_flush_descs(struct idxd_wq *wq) | ||
| 1352 | { | ||
| 1353 | struct idxd_irq_entry *ie = &wq->ie; | ||
| 1354 | struct idxd_device *idxd = wq->idxd; | ||
| 1355 | |||
| 1356 | guard(mutex)(&wq->wq_lock); | ||
| 1357 | |||
| 1358 | if (wq->state != IDXD_WQ_ENABLED || wq->type != IDXD_WQT_KERNEL) | ||
| 1359 | return; | ||
| 1360 | |||
| 1361 | idxd_flush_pending_descs(ie); | ||
| 1362 | if (idxd->request_int_handles) | ||
| 1363 | idxd_device_release_int_handle(idxd, ie->int_handle, IDXD_IRQ_MSIX); | ||
| 1364 | idxd_device_clear_perm_entry(idxd, ie); | ||
| 1365 | ie->int_handle = INVALID_INT_HANDLE; | ||
| 1366 | } | ||
| 1367 | |||
| 1343 | int idxd_wq_request_irq(struct idxd_wq *wq) | 1368 | int idxd_wq_request_irq(struct idxd_wq *wq) |
| 1344 | { | 1369 | { |
| 1345 | struct idxd_device *idxd = wq->idxd; | 1370 | struct idxd_device *idxd = wq->idxd; |
| @@ -1454,11 +1479,7 @@ int idxd_drv_enable_wq(struct idxd_wq *wq) · 108 unmodified lines | |||
| 1454 | } | 1479 | } |
| 1455 | } | 1480 | } |
| 1456 | 1481 | ||
| 1457 | rc = 0; | 1482 | rc = idxd_device_config(idxd); |
| 1458 | spin_lock(&idxd->dev_lock); | ||
| 1459 | if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) | ||
| 1460 | rc = idxd_device_config(idxd); | ||
| 1461 | spin_unlock(&idxd->dev_lock); | ||
| 1462 | if (rc < 0) { | 1483 | if (rc < 0) { |
| 1463 | dev_dbg(dev, "Writing wq %d config failed: %d\n", wq->id, rc); | 1484 | dev_dbg(dev, "Writing wq %d config failed: %d\n", wq->id, rc); |
| 1464 | goto err; | 1485 | goto err; |
| @@ -1533,7 +1554,6 @@ void idxd_drv_disable_wq(struct idxd_wq *wq) · 68 unmodified lines | |||
| 1533 | idxd_wq_reset(wq); | 1554 | idxd_wq_reset(wq); |
| 1534 | idxd_wq_free_resources(wq); | 1555 | idxd_wq_free_resources(wq); |
| 1535 | percpu_ref_exit(&wq->wq_active); | 1556 | percpu_ref_exit(&wq->wq_active); |
| 1536 | wq->type = IDXD_WQT_NONE; | ||
| 1537 | wq->client_count = 0; | 1557 | wq->client_count = 0; |
| 1538 | } | 1558 | } |
| 1539 | EXPORT_SYMBOL_NS_GPL(idxd_drv_disable_wq, "IDXD"); | 1559 | EXPORT_SYMBOL_NS_GPL(idxd_drv_disable_wq, "IDXD"); |
| @@ -1554,10 +1574,7 @@ int idxd_device_drv_probe(struct idxd_dev *idxd_dev) · 14 unmodified lines | |||
| 1554 | } | 1574 | } |
| 1555 | 1575 | ||
| 1556 | /* Device configuration */ | 1576 | /* Device configuration */ |
| 1557 | spin_lock(&idxd->dev_lock); | 1577 | rc = idxd_device_config(idxd); |
| 1558 | if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) | ||
| 1559 | rc = idxd_device_config(idxd); | ||
| 1560 | spin_unlock(&idxd->dev_lock); | ||
| 1561 | if (rc < 0) | 1578 | if (rc < 0) |
| 1562 | return -ENXIO; | 1579 | return -ENXIO; |
| 1563 | 1580 | ||
| @@ -194,6 +194,22 @@ static void idxd_dma_release(struct dma_device *device) · 193 unmodified lines | |||
| 194 | kfree(idxd_dma); | 194 | kfree(idxd_dma); |
| 195 | } | 195 | } |
| 196 | 196 | ||
| 197 | static int idxd_dma_terminate_all(struct dma_chan *c) | ||
| 198 | { | ||
| 199 | struct idxd_wq *wq = to_idxd_wq(c); | ||
| 200 | |||
| 201 | idxd_wq_flush_descs(wq); | ||
| 202 | |||
| 203 | return 0; | ||
| 204 | } | ||
| 205 | |||
| 206 | static void idxd_dma_synchronize(struct dma_chan *c) | ||
| 207 | { | ||
| 208 | struct idxd_wq *wq = to_idxd_wq(c); | ||
| 209 | |||
| 210 | idxd_wq_drain(wq); | ||
| 211 | } | ||
| 212 | |||
| 197 | int idxd_register_dma_device(struct idxd_device *idxd) | 213 | int idxd_register_dma_device(struct idxd_device *idxd) |
| 198 | { | 214 | { |
| 199 | struct idxd_dma_dev *idxd_dma; | 215 | struct idxd_dma_dev *idxd_dma; |
| @@ -224,6 +240,8 @@ int idxd_register_dma_device(struct idxd_device *idxd) · 24 unmodified lines | |||
| 224 | dma->device_issue_pending = idxd_dma_issue_pending; | 240 | dma->device_issue_pending = idxd_dma_issue_pending; |
| 225 | dma->device_alloc_chan_resources = idxd_dma_alloc_chan_resources; | 241 | dma->device_alloc_chan_resources = idxd_dma_alloc_chan_resources; |
| 226 | dma->device_free_chan_resources = idxd_dma_free_chan_resources; | 242 | dma->device_free_chan_resources = idxd_dma_free_chan_resources; |
| 243 | dma->device_terminate_all = idxd_dma_terminate_all; | ||
| 244 | dma->device_synchronize = idxd_dma_synchronize; | ||
| 227 | 245 | ||
| 228 | rc = dma_async_device_register(dma); | 246 | rc = dma_async_device_register(dma); |
| 229 | if (rc < 0) { | 247 | if (rc < 0) { |
| @@ -803,6 +803,7 @@ void idxd_wq_quiesce(struct idxd_wq *wq); · 802 unmodified lines | |||
| 803 | int idxd_wq_init_percpu_ref(struct idxd_wq *wq); | 803 | int idxd_wq_init_percpu_ref(struct idxd_wq *wq); |
| 804 | void idxd_wq_free_irq(struct idxd_wq *wq); | 804 | void idxd_wq_free_irq(struct idxd_wq *wq); |
| 805 | int idxd_wq_request_irq(struct idxd_wq *wq); | 805 | int idxd_wq_request_irq(struct idxd_wq *wq); |
| 806 | void idxd_wq_flush_descs(struct idxd_wq *wq); | ||
| 806 | 807 | ||
| 807 | /* submission */ | 808 | /* submission */ |
| 808 | int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc); | 809 | int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc); |
| @@ -973,7 +973,8 @@ static void idxd_device_config_restore(struct idxd_device *idxd, · 972 unmodified lines | |||
| 973 | 973 | ||
| 974 | idxd->rdbuf_limit = idxd_saved->saved_idxd.rdbuf_limit; | 974 | idxd->rdbuf_limit = idxd_saved->saved_idxd.rdbuf_limit; |
| 975 | 975 | ||
| 976 | idxd->evl->size = saved_evl->size; | 976 | if (idxd->evl) |
| 977 | idxd->evl->size = saved_evl->size; | ||
| 977 | 978 | ||
| 978 | for (i = 0; i < idxd->max_groups; i++) { | 979 | for (i = 0; i < idxd->max_groups; i++) { |
| 979 | struct idxd_group *saved_group, *group; | 980 | struct idxd_group *saved_group, *group; |
| @@ -1104,12 +1105,10 @@ static void idxd_reset_done(struct pci_dev *pdev) · 124 unmodified lines | |||
| 1104 | idxd_device_config_restore(idxd, idxd->idxd_saved); | 1105 | idxd_device_config_restore(idxd, idxd->idxd_saved); |
| 1105 | 1106 | ||
| 1106 | /* Re-configure IDXD device if allowed. */ | 1107 | /* Re-configure IDXD device if allowed. */ |
| 1107 | if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) { | 1108 | rc = idxd_device_config(idxd); |
| 1108 | rc = idxd_device_config(idxd); | 1109 | if (rc < 0) { |
| 1109 | if (rc < 0) { | 1110 | dev_err(dev, "HALT: %s config fails\n", idxd_name); |
| 1110 | dev_err(dev, "HALT: %s config fails\n", idxd_name); | 1111 | goto out; |
| 1111 | goto out; | ||
| 1112 | } | ||
| 1113 | } | 1112 | } |
| 1114 | 1113 | ||
| 1115 | /* Bind IDXD device to driver. */ | 1114 | /* Bind IDXD device to driver. */ |
| @@ -1147,6 +1146,7 @@ static void idxd_reset_done(struct pci_dev *pdev) · 31 unmodified lines | |||
| 1147 | } | 1146 | } |
| 1148 | out: | 1147 | out: |
| 1149 | kfree(idxd->idxd_saved); | 1148 | kfree(idxd->idxd_saved); |
| 1149 | idxd->idxd_saved = NULL; | ||
| 1150 | } | 1150 | } |
| 1151 | 1151 | ||
| 1152 | static const struct pci_error_handlers idxd_error_handler = { | 1152 | static const struct pci_error_handlers idxd_error_handler = { |
| @@ -397,6 +397,17 @@ static void idxd_device_flr(struct work_struct *work) · 396 unmodified lines | |||
| 397 | dev_err(&idxd->pdev->dev, "FLR failed\n"); | 397 | dev_err(&idxd->pdev->dev, "FLR failed\n"); |
| 398 | } | 398 | } |
| 399 | 399 | ||
| 400 | static void idxd_wqs_flush_descs(struct idxd_device *idxd) | ||
| 401 | { | ||
| 402 | int i; | ||
| 403 | |||
| 404 | for (i = 0; i < idxd->max_wqs; i++) { | ||
| 405 | struct idxd_wq *wq = idxd->wqs[i]; | ||
| 406 | |||
| 407 | idxd_wq_flush_descs(wq); | ||
| 408 | } | ||
| 409 | } | ||
| 410 | |||
| 400 | static irqreturn_t idxd_halt(struct idxd_device *idxd) | 411 | static irqreturn_t idxd_halt(struct idxd_device *idxd) |
| 401 | { | 412 | { |
| 402 | union gensts_reg gensts; | 413 | union gensts_reg gensts; |
| @@ -415,6 +426,11 @@ static irqreturn_t idxd_halt(struct idxd_device *idxd) · 12 unmodified lines | |||
| 415 | } else if (gensts.reset_type == IDXD_DEVICE_RESET_FLR) { | 426 | } else if (gensts.reset_type == IDXD_DEVICE_RESET_FLR) { |
| 416 | idxd->state = IDXD_DEV_HALTED; | 427 | idxd->state = IDXD_DEV_HALTED; |
| 417 | idxd_mask_error_interrupts(idxd); | 428 | idxd_mask_error_interrupts(idxd); |
| 429 | /* Flush all pending descriptors, and disable | ||
| 430 | * interrupts, they will be re-enabled when FLR | ||
| 431 | * concludes. | ||
| 432 | */ | ||
| 433 | idxd_wqs_flush_descs(idxd); | ||
| 418 | dev_dbg(&idxd->pdev->dev, | 434 | dev_dbg(&idxd->pdev->dev, |
| 419 | "idxd halted, doing FLR. After FLR, configs are restored\n"); | 435 | "idxd halted, doing FLR. After FLR, configs are restored\n"); |
| 420 | INIT_WORK(&idxd->work, idxd_device_flr); | 436 | INIT_WORK(&idxd->work, idxd_device_flr); |
| @@ -138,7 +138,7 @@ static void llist_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie, · 137 unmodified lines | |||
| 138 | */ | 138 | */ |
| 139 | list_for_each_entry_safe(d, t, &flist, list) { | 139 | list_for_each_entry_safe(d, t, &flist, list) { |
| 140 | list_del_init(&d->list); | 140 | list_del_init(&d->list); |
| 141 | idxd_dma_complete_txd(found, IDXD_COMPLETE_ABORT, true, | 141 | idxd_dma_complete_txd(d, IDXD_COMPLETE_ABORT, true, |
| 142 | NULL, NULL); | 142 | NULL, NULL); |
| 143 | } | 143 | } |
| 144 | } | 144 | } |
| @@ -1836,6 +1836,7 @@ static void idxd_conf_device_release(struct device *dev) · 1835 unmodified lines | |||
| 1836 | { | 1836 | { |
| 1837 | struct idxd_device *idxd = confdev_to_idxd(dev); | 1837 | struct idxd_device *idxd = confdev_to_idxd(dev); |
| 1838 | 1838 | ||
| 1839 | destroy_workqueue(idxd->wq); | ||
| 1839 | kfree(idxd->groups); | 1840 | kfree(idxd->groups); |
| 1840 | bitmap_free(idxd->wq_enable_map); | 1841 | bitmap_free(idxd->wq_enable_map); |
| 1841 | kfree(idxd->wqs); | 1842 | kfree(idxd->wqs); |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | */ | 10 | */ |
| 11 | 11 | ||
| 12 | #include <linux/bitfield.h> | 12 | #include <linux/bitfield.h> |
| 13 | #include <linux/cleanup.h> | ||
| 13 | #include <linux/dma-mapping.h> | 14 | #include <linux/dma-mapping.h> |
| 14 | #include <linux/dmaengine.h> | 15 | #include <linux/dmaengine.h> |
| 15 | #include <linux/interrupt.h> | 16 | #include <linux/interrupt.h> |
| @@ -296,13 +297,10 @@ static void rz_dmac_disable_hw(struct rz_dmac_chan *channel) · 280 unmodified lines | |||
| 296 | { | 297 | { |
| 297 | struct dma_chan *chan = &channel->vc.chan; | 298 | struct dma_chan *chan = &channel->vc.chan; |
| 298 | struct rz_dmac *dmac = to_rz_dmac(chan->device); | 299 | struct rz_dmac *dmac = to_rz_dmac(chan->device); |
| 299 | unsigned long flags; | ||
| 300 | 300 | ||
| 301 | dev_dbg(dmac->dev, "%s channel %d\n", __func__, channel->index); | 301 | dev_dbg(dmac->dev, "%s channel %d\n", __func__, channel->index); |
| 302 | 302 | ||
| 303 | local_irq_save(flags); | ||
| 304 | rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); | 303 | rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); |
| 305 | local_irq_restore(flags); | ||
| 306 | } | 304 | } |
| 307 | 305 | ||
| 308 | static void rz_dmac_set_dmars_register(struct rz_dmac *dmac, int nr, u32 dmars) | 306 | static void rz_dmac_set_dmars_register(struct rz_dmac *dmac, int nr, u32 dmars) |
| @@ -447,6 +445,7 @@ static int rz_dmac_alloc_chan_resources(struct dma_chan *chan) · 138 unmodified lines | |||
| 447 | if (!desc) | 445 | if (!desc) |
| 448 | break; | 446 | break; |
| 449 | 447 | ||
| 448 | /* No need to lock. This is called only for the 1st client. */ | ||
| 450 | list_add_tail(&desc->node, &channel->ld_free); | 449 | list_add_tail(&desc->node, &channel->ld_free); |
| 451 | channel->descs_allocated++; | 450 | channel->descs_allocated++; |
| 452 | } | 451 | } |
| @@ -502,18 +501,21 @@ rz_dmac_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, · 49 unmodified lines | |||
| 502 | dev_dbg(dmac->dev, "%s channel: %d src=0x%pad dst=0x%pad len=%zu\n", | 501 | dev_dbg(dmac->dev, "%s channel: %d src=0x%pad dst=0x%pad len=%zu\n", |
| 503 | __func__, channel->index, &src, &dest, len); | 502 | __func__, channel->index, &src, &dest, len); |
| 504 | 503 | ||
| 505 | if (list_empty(&channel->ld_free)) | 504 | scoped_guard(spinlock_irqsave, &channel->vc.lock) { |
| 506 | return NULL; | 505 | if (list_empty(&channel->ld_free)) |
| 506 | return NULL; | ||
| 507 | 507 | ||
| 508 | desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); | 508 | desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); |
| 509 | 509 | ||
| 510 | desc->type = RZ_DMAC_DESC_MEMCPY; | 510 | desc->type = RZ_DMAC_DESC_MEMCPY; |
| 511 | desc->src = src; | 511 | desc->src = src; |
| 512 | desc->dest = dest; | 512 | desc->dest = dest; |
| 513 | desc->len = len; | 513 | desc->len = len; |
| 514 | desc->direction = DMA_MEM_TO_MEM; | 514 | desc->direction = DMA_MEM_TO_MEM; |
| 515 | |||
| 516 | list_move_tail(channel->ld_free.next, &channel->ld_queue); | ||
| 517 | } | ||
| 515 | 518 | ||
| 516 | list_move_tail(channel->ld_free.next, &channel->ld_queue); | ||
| 517 | return vchan_tx_prep(&channel->vc, &desc->vd, flags); | 519 | return vchan_tx_prep(&channel->vc, &desc->vd, flags); |
| 518 | } | 520 | } |
| 519 | 521 | ||
| @@ -529,27 +531,29 @@ rz_dmac_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, · 9 unmodified lines | |||
| 529 | int dma_length = 0; | 531 | int dma_length = 0; |
| 530 | int i = 0; | 532 | int i = 0; |
| 531 | 533 | ||
| 532 | if (list_empty(&channel->ld_free)) | 534 | scoped_guard(spinlock_irqsave, &channel->vc.lock) { |
| 533 | return NULL; | 535 | if (list_empty(&channel->ld_free)) |
| 536 | return NULL; | ||
| 534 | 537 | ||
| 535 | desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); | 538 | desc = list_first_entry(&channel->ld_free, struct rz_dmac_desc, node); |
| 536 | 539 | ||
| 537 | for_each_sg(sgl, sg, sg_len, i) { | 540 | for_each_sg(sgl, sg, sg_len, i) |
| 538 | dma_length += sg_dma_len(sg); | 541 | dma_length += sg_dma_len(sg); |
| 539 | } | ||
| 540 | 542 | ||
| 541 | desc->type = RZ_DMAC_DESC_SLAVE_SG; | 543 | desc->type = RZ_DMAC_DESC_SLAVE_SG; |
| 542 | desc->sg = sgl; | 544 | desc->sg = sgl; |
| 543 | desc->sgcount = sg_len; | 545 | desc->sgcount = sg_len; |
| 544 | desc->len = dma_length; | 546 | desc->len = dma_length; |
| 545 | desc->direction = direction; | 547 | desc->direction = direction; |
| 546 | 548 | ||
| 547 | if (direction == DMA_DEV_TO_MEM) | 549 | if (direction == DMA_DEV_TO_MEM) |
| 548 | desc->src = channel->src_per_address; | 550 | desc->src = channel->src_per_address; |
| 549 | else | 551 | else |
| 550 | desc->dest = channel->dst_per_address; | 552 | desc->dest = channel->dst_per_address; |
| 553 | |||
| 554 | list_move_tail(channel->ld_free.next, &channel->ld_queue); | ||
| 555 | } | ||
| 551 | 556 | ||
| 552 | list_move_tail(channel->ld_free.next, &channel->ld_queue); | ||
| 553 | return vchan_tx_prep(&channel->vc, &desc->vd, flags); | 557 | return vchan_tx_prep(&channel->vc, &desc->vd, flags); |
| 554 | } | 558 | } |
| 555 | 559 | ||
| @@ -561,8 +565,8 @@ static int rz_dmac_terminate_all(struct dma_chan *chan) · 5 unmodified lines | |||
| 561 | unsigned int i; | 565 | unsigned int i; |
| 562 | LIST_HEAD(head); | 566 | LIST_HEAD(head); |
| 563 | 567 | ||
| 564 | rz_dmac_disable_hw(channel); | ||
| 565 | spin_lock_irqsave(&channel->vc.lock, flags); | 568 | spin_lock_irqsave(&channel->vc.lock, flags); |
| 569 | rz_dmac_disable_hw(channel); | ||
| 566 | for (i = 0; i < DMAC_NR_LMDESC; i++) | 570 | for (i = 0; i < DMAC_NR_LMDESC; i++) |
| 567 | lmdesc[i].header = 0; | 571 | lmdesc[i].header = 0; |
| 568 | 572 | ||
| @@ -699,7 +703,9 @@ static void rz_dmac_irq_handle_channel(struct rz_dmac_chan *channel) · 130 unmodified lines | |||
| 699 | if (chstat & CHSTAT_ER) { | 703 | if (chstat & CHSTAT_ER) { |
| 700 | dev_err(dmac->dev, "DMAC err CHSTAT_%d = %08X\n", | 704 | dev_err(dmac->dev, "DMAC err CHSTAT_%d = %08X\n", |
| 701 | channel->index, chstat); | 705 | channel->index, chstat); |
| 702 | rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); | 706 | |
| 707 | scoped_guard(spinlock_irqsave, &channel->vc.lock) | ||
| 708 | rz_dmac_ch_writel(channel, CHCTRL_DEFAULT, CHCTRL, 1); | ||
| 703 | goto done; | 709 | goto done; |
| 704 | } | 710 | } |
| 705 | 711 | ||
| @@ -1234,8 +1234,8 @@ static int xdma_probe(struct platform_device *pdev) · 1233 unmodified lines | |||
| 1234 | 1234 | ||
| 1235 | xdev->rmap = devm_regmap_init_mmio(&pdev->dev, reg_base, | 1235 | xdev->rmap = devm_regmap_init_mmio(&pdev->dev, reg_base, |
| 1236 | &xdma_regmap_config); | 1236 | &xdma_regmap_config); |
| 1237 | if (!xdev->rmap) { | 1237 | if (IS_ERR(xdev->rmap)) { |
| 1238 | xdma_err(xdev, "config regmap failed: %d", ret); | 1238 | xdma_err(xdev, "config regmap failed: %pe", xdev->rmap); |
| 1239 | goto failed; | 1239 | goto failed; |
| 1240 | } | 1240 | } |
| 1241 | INIT_LIST_HEAD(&xdev->dma_dev.channels); | 1241 | INIT_LIST_HEAD(&xdev->dma_dev.channels); |
| @@ -997,16 +997,16 @@ static u32 xilinx_dma_get_residue(struct xilinx_dma_chan *chan, · 996 unmodified lines | |||
| 997 | struct xilinx_cdma_tx_segment, | 997 | struct xilinx_cdma_tx_segment, |
| 998 | node); | 998 | node); |
| 999 | cdma_hw = &cdma_seg->hw; | 999 | cdma_hw = &cdma_seg->hw; |
| 1000 | residue += (cdma_hw->control - cdma_hw->status) & | 1000 | residue += (cdma_hw->control & chan->xdev->max_buffer_len) - |
| 1001 | chan->xdev->max_buffer_len; | 1001 | (cdma_hw->status & chan->xdev->max_buffer_len); |
| 1002 | } else if (chan->xdev->dma_config->dmatype == | 1002 | } else if (chan->xdev->dma_config->dmatype == |
| 1003 | XDMA_TYPE_AXIDMA) { | 1003 | XDMA_TYPE_AXIDMA) { |
| 1004 | axidma_seg = list_entry(entry, | 1004 | axidma_seg = list_entry(entry, |
| 1005 | struct xilinx_axidma_tx_segment, | 1005 | struct xilinx_axidma_tx_segment, |
| 1006 | node); | 1006 | node); |
| 1007 | axidma_hw = &axidma_seg->hw; | 1007 | axidma_hw = &axidma_seg->hw; |
| 1008 | residue += (axidma_hw->control - axidma_hw->status) & | 1008 | residue += (axidma_hw->control & chan->xdev->max_buffer_len) - |
| 1009 | chan->xdev->max_buffer_len; | 1009 | (axidma_hw->status & chan->xdev->max_buffer_len); |
| 1010 | } else { | 1010 | } else { |
| 1011 | aximcdma_seg = | 1011 | aximcdma_seg = |
| 1012 | list_entry(entry, | 1012 | list_entry(entry, |
| @@ -1014,8 +1014,8 @@ static u32 xilinx_dma_get_residue(struct xilinx_dma_chan *chan, · 1 unmodified line | |||
| 1014 | node); | 1014 | node); |
| 1015 | aximcdma_hw = &aximcdma_seg->hw; | 1015 | aximcdma_hw = &aximcdma_seg->hw; |
| 1016 | residue += | 1016 | residue += |
| 1017 | (aximcdma_hw->control - aximcdma_hw->status) & | 1017 | (aximcdma_hw->control & chan->xdev->max_buffer_len) - |
| 1018 | chan->xdev->max_buffer_len; | 1018 | (aximcdma_hw->status & chan->xdev->max_buffer_len); |
| 1019 | } | 1019 | } |
| 1020 | } | 1020 | } |
| 1021 | 1021 | ||
| @@ -1235,14 +1235,6 @@ static int xilinx_dma_alloc_chan_resources(struct dma_chan *dchan) · 213 unmodified lines | |||
| 1235 | 1235 | ||
| 1236 | dma_cookie_init(dchan); | 1236 | dma_cookie_init(dchan); |
| 1237 | 1237 | ||
| 1238 | if (chan->xdev->dma_config->dmatype == XDMA_TYPE_AXIDMA) { | ||
| 1239 | /* For AXI DMA resetting once channel will reset the | ||
| 1240 | * other channel as well so enable the interrupts here. | ||
| 1241 | */ | ||
| 1242 | dma_ctrl_set(chan, XILINX_DMA_REG_DMACR, | ||
| 1243 | XILINX_DMA_DMAXR_ALL_IRQ_MASK); | ||
| 1244 | } | ||
| 1245 | |||
| 1246 | if ((chan->xdev->dma_config->dmatype == XDMA_TYPE_CDMA) && chan->has_sg) | 1238 | if ((chan->xdev->dma_config->dmatype == XDMA_TYPE_CDMA) && chan->has_sg) |
| 1247 | dma_ctrl_set(chan, XILINX_DMA_REG_DMACR, | 1239 | dma_ctrl_set(chan, XILINX_DMA_REG_DMACR, |
| 1248 | XILINX_CDMA_CR_SGMODE); | 1240 | XILINX_CDMA_CR_SGMODE); |
| @@ -1564,8 +1556,29 @@ static void xilinx_dma_start_transfer(struct xilinx_dma_chan *chan) · 315 unmodified lines | |||
| 1564 | if (chan->err) | 1556 | if (chan->err) |
| 1565 | return; | 1557 | return; |
| 1566 | 1558 | ||
| 1567 | if (list_empty(&chan->pending_list)) | 1559 | if (list_empty(&chan->pending_list)) { |
| 1560 | if (chan->cyclic) { | ||
| 1561 | struct xilinx_dma_tx_descriptor *desc; | ||
| 1562 | struct list_head *entry; | ||
| 1563 | |||
| 1564 | desc = list_last_entry(&chan->done_list, | ||
| 1565 | struct xilinx_dma_tx_descriptor, node); | ||
| 1566 | list_for_each(entry, &desc->segments) { | ||
| 1567 | struct xilinx_axidma_tx_segment *axidma_seg; | ||
| 1568 | struct xilinx_axidma_desc_hw *axidma_hw; | ||
| 1569 | axidma_seg = list_entry(entry, | ||
| 1570 | struct xilinx_axidma_tx_segment, | ||
| 1571 | node); | ||
| 1572 | axidma_hw = &axidma_seg->hw; | ||
| 1573 | axidma_hw->status = 0; | ||
| 1574 | } | ||
| 1575 | |||
| 1576 | list_splice_tail_init(&chan->done_list, &chan->active_list); | ||
| 1577 | chan->desc_pendingcount = 0; | ||
| 1578 | chan->idle = false; | ||
| 1579 | } | ||
| 1568 | return; | 1580 | return; |
| 1581 | } | ||
| 1569 | 1582 | ||
| 1570 | if (!chan->idle) | 1583 | if (!chan->idle) |
| 1571 | return; | 1584 | return; |
| @@ -1591,6 +1604,7 @@ static void xilinx_dma_start_transfer(struct xilinx_dma_chan *chan) · 19 unmodified lines | |||
| 1591 | head_desc->async_tx.phys); | 1604 | head_desc->async_tx.phys); |
| 1592 | reg &= ~XILINX_DMA_CR_DELAY_MAX; | 1605 | reg &= ~XILINX_DMA_CR_DELAY_MAX; |
| 1593 | reg |= chan->irq_delay << XILINX_DMA_CR_DELAY_SHIFT; | 1606 | reg |= chan->irq_delay << XILINX_DMA_CR_DELAY_SHIFT; |
| 1607 | reg |= XILINX_DMA_DMAXR_ALL_IRQ_MASK; | ||
| 1594 | dma_ctrl_write(chan, XILINX_DMA_REG_DMACR, reg); | 1608 | dma_ctrl_write(chan, XILINX_DMA_REG_DMACR, reg); |
| 1595 | 1609 | ||
| 1596 | xilinx_dma_start(chan); | 1610 | xilinx_dma_start(chan); |
| @@ -3024,7 +3038,7 @@ static int xilinx_dma_chan_probe(struct xilinx_dma_device *xdev, · 1427 unmodified lines | |||
| 3024 | return -EINVAL; | 3038 | return -EINVAL; |
| 3025 | } | 3039 | } |
| 3026 | 3040 | ||
| 3027 | xdev->common.directions |= chan->direction; | 3041 | xdev->common.directions |= BIT(chan->direction); |
| 3028 | 3042 | ||
| 3029 | /* Request the interrupt */ | 3043 | /* Request the interrupt */ |
| 3030 | chan->irq = of_irq_get(node, chan->tdest); | 3044 | chan->irq = of_irq_get(node, chan->tdest); |
| @@ -257,9 +257,10 @@ static void fwnet_header_cache_update(struct hh_cache *hh, · 256 unmodified lines | |||
| 257 | memcpy((u8 *)hh->hh_data + HH_DATA_OFF(FWNET_HLEN), haddr, net->addr_len); | 257 | memcpy((u8 *)hh->hh_data + HH_DATA_OFF(FWNET_HLEN), haddr, net->addr_len); |
| 258 | } | 258 | } |
| 259 | 259 | ||
| 260 | static int fwnet_header_parse(const struct sk_buff *skb, unsigned char *haddr) | 260 | static int fwnet_header_parse(const struct sk_buff *skb, const struct net_device *dev, |
| 261 | unsigned char *haddr) | ||
| 261 | { | 262 | { |
| 262 | memcpy(haddr, skb->dev->dev_addr, FWNET_ALEN); | 263 | memcpy(haddr, dev->dev_addr, FWNET_ALEN); |
| 263 | 264 | ||
| 264 | return FWNET_ALEN; | 265 | return FWNET_ALEN; |
| 265 | } | 266 | } |
| @@ -205,12 +205,12 @@ static int ffa_rxtx_map(phys_addr_t tx_buf, phys_addr_t rx_buf, u32 pg_cnt) · 204 unmodified lines | |||
| 205 | return 0; | 205 | return 0; |
| 206 | } | 206 | } |
| 207 | 207 | ||
| 208 | static int ffa_rxtx_unmap(u16 vm_id) | 208 | static int ffa_rxtx_unmap(void) |
| 209 | { | 209 | { |
| 210 | ffa_value_t ret; | 210 | ffa_value_t ret; |
| 211 | 211 | ||
| 212 | invoke_ffa_fn((ffa_value_t){ | 212 | invoke_ffa_fn((ffa_value_t){ |
| 213 | .a0 = FFA_RXTX_UNMAP, .a1 = PACK_TARGET_INFO(vm_id, 0), | 213 | .a0 = FFA_RXTX_UNMAP, |
| 214 | }, &ret); | 214 | }, &ret); |
| 215 | 215 | ||
| 216 | if (ret.a0 == FFA_ERROR) | 216 | if (ret.a0 == FFA_ERROR) |
| @@ -2097,7 +2097,7 @@ static int __init ffa_init(void) · 1880 unmodified lines | |||
| 2097 | 2097 | ||
| 2098 | pr_err("failed to setup partitions\n"); | 2098 | pr_err("failed to setup partitions\n"); |
| 2099 | ffa_notifications_cleanup(); | 2099 | ffa_notifications_cleanup(); |
| 2100 | ffa_rxtx_unmap(drv_info->vm_id); | 2100 | ffa_rxtx_unmap(); |
| 2101 | free_pages: | 2101 | free_pages: |
| 2102 | if (drv_info->tx_buffer) | 2102 | if (drv_info->tx_buffer) |
| 2103 | free_pages_exact(drv_info->tx_buffer, rxtx_bufsz); | 2103 | free_pages_exact(drv_info->tx_buffer, rxtx_bufsz); |
| @@ -2112,7 +2112,7 @@ static void __exit ffa_exit(void) · 8 unmodified lines | |||
| 2112 | { | 2112 | { |
| 2113 | ffa_notifications_cleanup(); | 2113 | ffa_notifications_cleanup(); |
| 2114 | ffa_partitions_cleanup(); | 2114 | ffa_partitions_cleanup(); |
| 2115 | ffa_rxtx_unmap(drv_info->vm_id); | 2115 | ffa_rxtx_unmap(); |
| 2116 | free_pages_exact(drv_info->tx_buffer, drv_info->rxtx_bufsz); | 2116 | free_pages_exact(drv_info->tx_buffer, drv_info->rxtx_bufsz); |
| 2117 | free_pages_exact(drv_info->rx_buffer, drv_info->rxtx_bufsz); | 2117 | free_pages_exact(drv_info->rx_buffer, drv_info->rxtx_bufsz); |
| 2118 | kfree(drv_info); | 2118 | kfree(drv_info); |
| @@ -1066,7 +1066,7 @@ static int scmi_register_event_handler(struct scmi_notify_instance *ni, · 1065 unmodified lines | |||
| 1066 | * since at creation time we usually want to have all setup and ready before | 1066 | * since at creation time we usually want to have all setup and ready before |
| 1067 | * events really start flowing. | 1067 | * events really start flowing. |
| 1068 | * | 1068 | * |
| 1069 | * Return: A properly refcounted handler on Success, NULL on Failure | 1069 | * Return: A properly refcounted handler on Success, ERR_PTR on Failure |
| 1070 | */ | 1070 | */ |
| 1071 | static inline struct scmi_event_handler * | 1071 | static inline struct scmi_event_handler * |
| 1072 | __scmi_event_handler_get_ops(struct scmi_notify_instance *ni, | 1072 | __scmi_event_handler_get_ops(struct scmi_notify_instance *ni, |
| @@ -1113,7 +1113,7 @@ __scmi_event_handler_get_ops(struct scmi_notify_instance *ni, · 40 unmodified lines | |||
| 1113 | } | 1113 | } |
| 1114 | mutex_unlock(&ni->pending_mtx); | 1114 | mutex_unlock(&ni->pending_mtx); |
| 1115 | 1115 | ||
| 1116 | return hndl; | 1116 | return hndl ?: ERR_PTR(-ENODEV); |
| 1117 | } | 1117 | } |
| 1118 | 1118 | ||
| 1119 | static struct scmi_event_handler * | 1119 | static struct scmi_event_handler * |
| @@ -189,13 +189,13 @@ struct scmi_protocol_handle { · 188 unmodified lines | |||
| 189 | 189 | ||
| 190 | /** | 190 | /** |
| 191 | * struct scmi_iterator_state - Iterator current state descriptor | 191 | * struct scmi_iterator_state - Iterator current state descriptor |
| 192 | * @desc_index: Starting index for the current mulit-part request. | 192 | * @desc_index: Starting index for the current multi-part request. |
| 193 | * @num_returned: Number of returned items in the last multi-part reply. | 193 | * @num_returned: Number of returned items in the last multi-part reply. |
| 194 | * @num_remaining: Number of remaining items in the multi-part message. | 194 | * @num_remaining: Number of remaining items in the multi-part message. |
| 195 | * @max_resources: Maximum acceptable number of items, configured by the caller | 195 | * @max_resources: Maximum acceptable number of items, configured by the caller |
| 196 | * depending on the underlying resources that it is querying. | 196 | * depending on the underlying resources that it is querying. |
| 197 | * @loop_idx: The iterator loop index in the current multi-part reply. | 197 | * @loop_idx: The iterator loop index in the current multi-part reply. |
| 198 | * @rx_len: Size in bytes of the currenly processed message; it can be used by | 198 | * @rx_len: Size in bytes of the currently processed message; it can be used by |
| 199 | * the user of the iterator to verify a reply size. | 199 | * the user of the iterator to verify a reply size. |
| 200 | * @priv: Optional pointer to some additional state-related private data setup | 200 | * @priv: Optional pointer to some additional state-related private data setup |
| 201 | * by the caller during the iterations. | 201 | * by the caller during the iterations. |
| @@ -18,6 +18,7 @@ · 17 unmodified lines | |||
| 18 | 18 | ||
| 19 | #include <linux/bitmap.h> | 19 | #include <linux/bitmap.h> |
| 20 | #include <linux/bitfield.h> | 20 | #include <linux/bitfield.h> |
| 21 | #include <linux/cleanup.h> | ||
| 21 | #include <linux/device.h> | 22 | #include <linux/device.h> |
| 22 | #include <linux/err.h> | 23 | #include <linux/err.h> |
| 23 | #include <linux/export.h> | 24 | #include <linux/export.h> |
| @@ -940,13 +941,13 @@ static int scpi_probe(struct platform_device *pdev) · 916 unmodified lines | |||
| 940 | int idx = scpi_drvinfo->num_chans; | 941 | int idx = scpi_drvinfo->num_chans; |
| 941 | struct scpi_chan *pchan = scpi_drvinfo->channels + idx; | 942 | struct scpi_chan *pchan = scpi_drvinfo->channels + idx; |
| 942 | struct mbox_client *cl = &pchan->cl; | 943 | struct mbox_client *cl = &pchan->cl; |
| 943 | struct device_node *shmem = of_parse_phandle(np, "shmem", idx); | 944 | struct device_node *shmem __free(device_node) = |
| 945 | of_parse_phandle(np, "shmem", idx); | ||
| 944 | 946 | ||
| 945 | if (!of_match_node(shmem_of_match, shmem)) | 947 | if (!of_match_node(shmem_of_match, shmem)) |
| 946 | return -ENXIO; | 948 | return -ENXIO; |
| 947 | 949 | ||
| 948 | ret = of_address_to_resource(shmem, 0, &res); | 950 | ret = of_address_to_resource(shmem, 0, &res); |
| 949 | of_node_put(shmem); | ||
| 950 | if (ret) { | 951 | if (ret) { |
| 951 | dev_err(dev, "failed to get SCPI payload mem resource\n"); | 952 | dev_err(dev, "failed to get SCPI payload mem resource\n"); |
| 952 | return ret; | 953 | return ret; |
| @@ -1610,11 +1610,17 @@ static int cs_dsp_load(struct cs_dsp *dsp, const struct firmware *firmware, · 1609 unmodified lines | |||
| 1610 | region_name); | 1610 | region_name); |
| 1611 | 1611 | ||
| 1612 | if (reg) { | 1612 | if (reg) { |
| 1613 | /* | ||
| 1614 | * Although we expect the underlying bus does not require | ||
| 1615 | * physically-contiguous buffers, we pessimistically use | ||
| 1616 | * a temporary buffer instead of trusting that the | ||
| 1617 | * alignment of region->data is ok. | ||
| 1618 | */ | ||
| 1613 | region_len = le32_to_cpu(region->len); | 1619 | region_len = le32_to_cpu(region->len); |
| 1614 | if (region_len > buf_len) { | 1620 | if (region_len > buf_len) { |
| 1615 | buf_len = round_up(region_len, PAGE_SIZE); | 1621 | buf_len = round_up(region_len, PAGE_SIZE); |
| 1616 | kfree(buf); | 1622 | vfree(buf); |
| 1617 | buf = kmalloc(buf_len, GFP_KERNEL | GFP_DMA); | 1623 | buf = vmalloc(buf_len); |
| 1618 | if (!buf) { | 1624 | if (!buf) { |
| 1619 | ret = -ENOMEM; | 1625 | ret = -ENOMEM; |
| 1620 | goto out_fw; | 1626 | goto out_fw; |
| @@ -1643,7 +1649,7 @@ static int cs_dsp_load(struct cs_dsp *dsp, const struct firmware *firmware, · 22 unmodified lines | |||
| 1643 | 1649 | ||
| 1644 | ret = 0; | 1650 | ret = 0; |
| 1645 | out_fw: | 1651 | out_fw: |
| 1646 | kfree(buf); | 1652 | vfree(buf); |
| 1647 | 1653 | ||
| 1648 | if (ret == -EOVERFLOW) | 1654 | if (ret == -EOVERFLOW) |
| 1649 | cs_dsp_err(dsp, "%s: file content overflows file data\n", file); | 1655 | cs_dsp_err(dsp, "%s: file content overflows file data\n", file); |
| @@ -2331,11 +2337,17 @@ static int cs_dsp_load_coeff(struct cs_dsp *dsp, const struct firmware *firmware · 681 unmodified lines | |||
| 2331 | } | 2337 | } |
| 2332 | 2338 | ||
| 2333 | if (reg) { | 2339 | if (reg) { |
| 2340 | /* | ||
| 2341 | * Although we expect the underlying bus does not require | ||
| 2342 | * physically-contiguous buffers, we pessimistically use | ||
| 2343 | * a temporary buffer instead of trusting that the | ||
| 2344 | * alignment of blk->data is ok. | ||
| 2345 | */ | ||
| 2334 | region_len = le32_to_cpu(blk->len); | 2346 | region_len = le32_to_cpu(blk->len); |
| 2335 | if (region_len > buf_len) { | 2347 | if (region_len > buf_len) { |
| 2336 | buf_len = round_up(region_len, PAGE_SIZE); | 2348 | buf_len = round_up(region_len, PAGE_SIZE); |
| 2337 | kfree(buf); | 2349 | vfree(buf); |
| 2338 | buf = kmalloc(buf_len, GFP_KERNEL | GFP_DMA); | 2350 | buf = vmalloc(buf_len); |
| 2339 | if (!buf) { | 2351 | if (!buf) { |
| 2340 | ret = -ENOMEM; | 2352 | ret = -ENOMEM; |
| 2341 | goto out_fw; | 2353 | goto out_fw; |
| @@ -2366,7 +2378,7 @@ static int cs_dsp_load_coeff(struct cs_dsp *dsp, const struct firmware *firmware · 24 unmodified lines | |||
| 2366 | 2378 | ||
| 2367 | ret = 0; | 2379 | ret = 0; |
| 2368 | out_fw: | 2380 | out_fw: |
| 2369 | kfree(buf); | 2381 | vfree(buf); |
| 2370 | 2382 | ||
| 2371 | if (ret == -EOVERFLOW) | 2383 | if (ret == -EOVERFLOW) |
| 2372 | cs_dsp_err(dsp, "%s: file content overflows file data\n", file); | 2384 | cs_dsp_err(dsp, "%s: file content overflows file data\n", file); |
| @@ -768,7 +768,9 @@ static int stratix10_rsu_probe(struct platform_device *pdev) · 767 unmodified lines | |||
| 768 | rsu_async_status_callback); | 768 | rsu_async_status_callback); |
| 769 | if (ret) { | 769 | if (ret) { |
| 770 | dev_err(dev, "Error, getting RSU status %i\n", ret); | 770 | dev_err(dev, "Error, getting RSU status %i\n", ret); |
| 771 | stratix10_svc_remove_async_client(priv->chan); | ||
| 771 | stratix10_svc_free_channel(priv->chan); | 772 | stratix10_svc_free_channel(priv->chan); |
| 773 | return ret; | ||
| 772 | } | 774 | } |
| 773 | 775 | ||
| 774 | /* get DCMF version from firmware */ | 776 | /* get DCMF version from firmware */ |
| @@ -37,15 +37,14 @@ · 36 unmodified lines | |||
| 37 | * service layer will return error to FPGA manager when timeout occurs, | 37 | * service layer will return error to FPGA manager when timeout occurs, |
| 38 | * timeout is set to 30 seconds (30 * 1000) at Intel Stratix10 SoC. | 38 | * timeout is set to 30 seconds (30 * 1000) at Intel Stratix10 SoC. |
| 39 | */ | 39 | */ |
| 40 | #define SVC_NUM_DATA_IN_FIFO 32 | 40 | #define SVC_NUM_DATA_IN_FIFO 8 |
| 41 | #define SVC_NUM_CHANNEL 4 | 41 | #define SVC_NUM_CHANNEL 4 |
| 42 | #define FPGA_CONFIG_DATA_CLAIM_TIMEOUT_MS 200 | 42 | #define FPGA_CONFIG_DATA_CLAIM_TIMEOUT_MS 2000 |
| 43 | #define FPGA_CONFIG_STATUS_TIMEOUT_SEC 30 | 43 | #define FPGA_CONFIG_STATUS_TIMEOUT_SEC 30 |
| 44 | #define BYTE_TO_WORD_SIZE 4 | 44 | #define BYTE_TO_WORD_SIZE 4 |
| 45 | 45 | ||
| 46 | /* stratix10 service layer clients */ | 46 | /* stratix10 service layer clients */ |
| 47 | #define STRATIX10_RSU "stratix10-rsu" | 47 | #define STRATIX10_RSU "stratix10-rsu" |
| 48 | #define INTEL_FCS "intel-fcs" | ||
| 49 | 48 | ||
| 50 | /* Maximum number of SDM client IDs. */ | 49 | /* Maximum number of SDM client IDs. */ |
| 51 | #define MAX_SDM_CLIENT_IDS 16 | 50 | #define MAX_SDM_CLIENT_IDS 16 |
| @@ -105,11 +104,9 @@ struct stratix10_svc_chan; · 53 unmodified lines | |||
| 105 | /** | 104 | /** |
| 106 | * struct stratix10_svc - svc private data | 105 | * struct stratix10_svc - svc private data |
| 107 | * @stratix10_svc_rsu: pointer to stratix10 RSU device | 106 | * @stratix10_svc_rsu: pointer to stratix10 RSU device |
| 108 | * @intel_svc_fcs: pointer to the FCS device | ||
| 109 | */ | 107 | */ |
| 110 | struct stratix10_svc { | 108 | struct stratix10_svc { |
| 111 | struct platform_device *stratix10_svc_rsu; | 109 | struct platform_device *stratix10_svc_rsu; |
| 112 | struct platform_device *intel_svc_fcs; | ||
| 113 | }; | 110 | }; |
| 114 | 111 | ||
| 115 | /** | 112 | /** |
| @@ -251,12 +248,10 @@ struct stratix10_async_ctrl { · 135 unmodified lines | |||
| 251 | * @num_active_client: number of active service client | 248 | * @num_active_client: number of active service client |
| 252 | * @node: list management | 249 | * @node: list management |
| 253 | * @genpool: memory pool pointing to the memory region | 250 | * @genpool: memory pool pointing to the memory region |
| 254 | * @task: pointer to the thread task which handles SMC or HVC call | ||
| 255 | * @svc_fifo: a queue for storing service message data | ||
| 256 | * @complete_status: state for completion | 251 | * @complete_status: state for completion |
| 257 | * @svc_fifo_lock: protect access to service message data queue | ||
| 258 | * @invoke_fn: function to issue secure monitor call or hypervisor call | 252 | * @invoke_fn: function to issue secure monitor call or hypervisor call |
| 259 | * @svc: manages the list of client svc drivers | 253 | * @svc: manages the list of client svc drivers |
| 254 | * @sdm_lock: only allows a single command single response to SDM | ||
| 260 | * @actrl: async control structure | 255 | * @actrl: async control structure |
| 261 | * | 256 | * |
| 262 | * This struct is used to create communication channels for service clients, to | 257 | * This struct is used to create communication channels for service clients, to |
| @@ -269,12 +264,10 @@ struct stratix10_svc_controller { · 6 unmodified lines | |||
| 269 | int num_active_client; | 264 | int num_active_client; |
| 270 | struct list_head node; | 265 | struct list_head node; |
| 271 | struct gen_pool *genpool; | 266 | struct gen_pool *genpool; |
| 272 | struct task_struct *task; | ||
| 273 | struct kfifo svc_fifo; | ||
| 274 | struct completion complete_status; | 267 | struct completion complete_status; |
| 275 | spinlock_t svc_fifo_lock; | ||
| 276 | svc_invoke_fn *invoke_fn; | 268 | svc_invoke_fn *invoke_fn; |
| 277 | struct stratix10_svc *svc; | 269 | struct stratix10_svc *svc; |
| 270 | struct mutex sdm_lock; | ||
| 278 | struct stratix10_async_ctrl actrl; | 271 | struct stratix10_async_ctrl actrl; |
| 279 | }; | 272 | }; |
| 280 | 273 | ||
| @@ -283,6 +276,9 @@ struct stratix10_svc_controller { · 2 unmodified lines | |||
| 283 | * @ctrl: pointer to service controller which is the provider of this channel | 276 | * @ctrl: pointer to service controller which is the provider of this channel |
| 284 | * @scl: pointer to service client which owns the channel | 277 | * @scl: pointer to service client which owns the channel |
| 285 | * @name: service client name associated with the channel | 278 | * @name: service client name associated with the channel |
| 279 | * @task: pointer to the thread task which handles SMC or HVC call | ||
| 280 | * @svc_fifo: a queue for storing service message data (separate fifo for every channel) | ||
| 281 | * @svc_fifo_lock: protect access to service message data queue (locking pending fifo) | ||
| 286 | * @lock: protect access to the channel | 282 | * @lock: protect access to the channel |
| 287 | * @async_chan: reference to asynchronous channel object for this channel | 283 | * @async_chan: reference to asynchronous channel object for this channel |
| 288 | * | 284 | * |
| @@ -293,6 +289,9 @@ struct stratix10_svc_chan { · 4 unmodified lines | |||
| 293 | struct stratix10_svc_controller *ctrl; | 289 | struct stratix10_svc_controller *ctrl; |
| 294 | struct stratix10_svc_client *scl; | 290 | struct stratix10_svc_client *scl; |
| 295 | char *name; | 291 | char *name; |
| 292 | struct task_struct *task; | ||
| 293 | struct kfifo svc_fifo; | ||
| 294 | spinlock_t svc_fifo_lock; | ||
| 296 | spinlock_t lock; | 295 | spinlock_t lock; |
| 297 | struct stratix10_async_chan *async_chan; | 296 | struct stratix10_async_chan *async_chan; |
| 298 | }; | 297 | }; |
| @@ -527,10 +526,10 @@ static void svc_thread_recv_status_ok(struct stratix10_svc_data *p_data, · 228 unmodified lines | |||
| 527 | */ | 526 | */ |
| 528 | static int svc_normal_to_secure_thread(void *data) | 527 | static int svc_normal_to_secure_thread(void *data) |
| 529 | { | 528 | { |
| 530 | struct stratix10_svc_controller | 529 | struct stratix10_svc_chan *chan = (struct stratix10_svc_chan *)data; |
| 531 | *ctrl = (struct stratix10_svc_controller *)data; | 530 | struct stratix10_svc_controller *ctrl = chan->ctrl; |
| 532 | struct stratix10_svc_data *pdata; | 531 | struct stratix10_svc_data *pdata = NULL; |
| 533 | struct stratix10_svc_cb_data *cbdata; | 532 | struct stratix10_svc_cb_data *cbdata = NULL; |
| 534 | struct arm_smccc_res res; | 533 | struct arm_smccc_res res; |
| 535 | unsigned long a0, a1, a2, a3, a4, a5, a6, a7; | 534 | unsigned long a0, a1, a2, a3, a4, a5, a6, a7; |
| 536 | int ret_fifo = 0; | 535 | int ret_fifo = 0; |
| @@ -555,12 +554,12 @@ static int svc_normal_to_secure_thread(void *data) · 18 unmodified lines | |||
| 555 | a6 = 0; | 554 | a6 = 0; |
| 556 | a7 = 0; | 555 | a7 = 0; |
| 557 | 556 | ||
| 558 | pr_debug("smc_hvc_shm_thread is running\n"); | 557 | pr_debug("%s: %s: Thread is running!\n", __func__, chan->name); |
| 559 | 558 | ||
| 560 | while (!kthread_should_stop()) { | 559 | while (!kthread_should_stop()) { |
| 561 | ret_fifo = kfifo_out_spinlocked(&ctrl->svc_fifo, | 560 | ret_fifo = kfifo_out_spinlocked(&chan->svc_fifo, |
| 562 | pdata, sizeof(*pdata), | 561 | pdata, sizeof(*pdata), |
| 563 | &ctrl->svc_fifo_lock); | 562 | &chan->svc_fifo_lock); |
| 564 | 563 | ||
| 565 | if (!ret_fifo) | 564 | if (!ret_fifo) |
| 566 | continue; | 565 | continue; |
| @@ -569,9 +568,25 @@ static int svc_normal_to_secure_thread(void *data) · 2 unmodified lines | |||
| 569 | (unsigned int)pdata->paddr, pdata->command, | 568 | (unsigned int)pdata->paddr, pdata->command, |
| 570 | (unsigned int)pdata->size); | 569 | (unsigned int)pdata->size); |
| 571 | 570 | ||
| 571 | /* SDM can only process one command at a time */ | ||
| 572 | pr_debug("%s: %s: Thread is waiting for mutex!\n", | ||
| 573 | __func__, chan->name); | ||
| 574 | if (mutex_lock_interruptible(&ctrl->sdm_lock)) { | ||
| 575 | /* item already dequeued; notify client to unblock it */ | ||
| 576 | cbdata->status = BIT(SVC_STATUS_ERROR); | ||
| 577 | cbdata->kaddr1 = NULL; | ||
| 578 | cbdata->kaddr2 = NULL; | ||
| 579 | cbdata->kaddr3 = NULL; | ||
| 580 | if (pdata->chan->scl) | ||
| 581 | pdata->chan->scl->receive_cb(pdata->chan->scl, | ||
| 582 | cbdata); | ||
| 583 | break; | ||
| 584 | } | ||
| 585 | |||
| 572 | switch (pdata->command) { | 586 | switch (pdata->command) { |
| 573 | case COMMAND_RECONFIG_DATA_CLAIM: | 587 | case COMMAND_RECONFIG_DATA_CLAIM: |
| 574 | svc_thread_cmd_data_claim(ctrl, pdata, cbdata); | 588 | svc_thread_cmd_data_claim(ctrl, pdata, cbdata); |
| 589 | mutex_unlock(&ctrl->sdm_lock); | ||
| 575 | continue; | 590 | continue; |
| 576 | case COMMAND_RECONFIG: | 591 | case COMMAND_RECONFIG: |
| 577 | a0 = INTEL_SIP_SMC_FPGA_CONFIG_START; | 592 | a0 = INTEL_SIP_SMC_FPGA_CONFIG_START; |
| @@ -700,10 +715,11 @@ static int svc_normal_to_secure_thread(void *data) · 122 unmodified lines | |||
| 700 | break; | 715 | break; |
| 701 | default: | 716 | default: |
| 702 | pr_warn("it shouldn't happen\n"); | 717 | pr_warn("it shouldn't happen\n"); |
| 703 | break; | 718 | mutex_unlock(&ctrl->sdm_lock); |
| 719 | continue; | ||
| 704 | } | 720 | } |
| 705 | pr_debug("%s: before SMC call -- a0=0x%016x a1=0x%016x", | 721 | pr_debug("%s: %s: before SMC call -- a0=0x%016x a1=0x%016x", |
| 706 | __func__, | 722 | __func__, chan->name, |
| 707 | (unsigned int)a0, | 723 | (unsigned int)a0, |
| 708 | (unsigned int)a1); | 724 | (unsigned int)a1); |
| 709 | pr_debug(" a2=0x%016x\n", (unsigned int)a2); | 725 | pr_debug(" a2=0x%016x\n", (unsigned int)a2); |
| @@ -712,8 +728,8 @@ static int svc_normal_to_secure_thread(void *data) · 2 unmodified lines | |||
| 712 | pr_debug(" a5=0x%016x\n", (unsigned int)a5); | 728 | pr_debug(" a5=0x%016x\n", (unsigned int)a5); |
| 713 | ctrl->invoke_fn(a0, a1, a2, a3, a4, a5, a6, a7, &res); | 729 | ctrl->invoke_fn(a0, a1, a2, a3, a4, a5, a6, a7, &res); |
| 714 | 730 | ||
| 715 | pr_debug("%s: after SMC call -- res.a0=0x%016x", | 731 | pr_debug("%s: %s: after SMC call -- res.a0=0x%016x", |
| 716 | __func__, (unsigned int)res.a0); | 732 | __func__, chan->name, (unsigned int)res.a0); |
| 717 | pr_debug(" res.a1=0x%016x, res.a2=0x%016x", | 733 | pr_debug(" res.a1=0x%016x, res.a2=0x%016x", |
| 718 | (unsigned int)res.a1, (unsigned int)res.a2); | 734 | (unsigned int)res.a1, (unsigned int)res.a2); |
| 719 | pr_debug(" res.a3=0x%016x\n", (unsigned int)res.a3); | 735 | pr_debug(" res.a3=0x%016x\n", (unsigned int)res.a3); |
| @@ -728,6 +744,7 @@ static int svc_normal_to_secure_thread(void *data) · 8 unmodified lines | |||
| 728 | cbdata->kaddr2 = NULL; | 744 | cbdata->kaddr2 = NULL; |
| 729 | cbdata->kaddr3 = NULL; | 745 | cbdata->kaddr3 = NULL; |
| 730 | pdata->chan->scl->receive_cb(pdata->chan->scl, cbdata); | 746 | pdata->chan->scl->receive_cb(pdata->chan->scl, cbdata); |
| 747 | mutex_unlock(&ctrl->sdm_lock); | ||
| 731 | continue; | 748 | continue; |
| 732 | } | 749 | } |
| 733 | 750 | ||
| @@ -801,6 +818,8 @@ static int svc_normal_to_secure_thread(void *data) · 67 unmodified lines | |||
| 801 | break; | 818 | break; |
| 802 | 819 | ||
| 803 | } | 820 | } |
| 821 | |||
| 822 | mutex_unlock(&ctrl->sdm_lock); | ||
| 804 | } | 823 | } |
| 805 | 824 | ||
| 806 | kfree(cbdata); | 825 | kfree(cbdata); |
| @@ -1696,22 +1715,33 @@ int stratix10_svc_send(struct stratix10_svc_chan *chan, void *msg) · 889 unmodified lines | |||
| 1696 | if (!p_data) | 1715 | if (!p_data) |
| 1697 | return -ENOMEM; | 1716 | return -ENOMEM; |
| 1698 | 1717 | ||
| 1699 | /* first client will create kernel thread */ | 1718 | /* first caller creates the per-channel kthread */ |
| 1700 | if (!chan->ctrl->task) { | 1719 | if (!chan->task) { |
| 1701 | chan->ctrl->task = | 1720 | struct task_struct *task; |
| 1702 | kthread_run_on_cpu(svc_normal_to_secure_thread, | 1721 | |
| 1703 | (void *)chan->ctrl, | 1722 | task = kthread_run_on_cpu(svc_normal_to_secure_thread, |
| 1704 | cpu, "svc_smc_hvc_thread"); | 1723 | (void *)chan, |
| 1705 | if (IS_ERR(chan->ctrl->task)) { | 1724 | cpu, "svc_smc_hvc_thread"); |
| 1725 | if (IS_ERR(task)) { | ||
| 1706 | dev_err(chan->ctrl->dev, | 1726 | dev_err(chan->ctrl->dev, |
| 1707 | "failed to create svc_smc_hvc_thread\n"); | 1727 | "failed to create svc_smc_hvc_thread\n"); |
| 1708 | kfree(p_data); | 1728 | kfree(p_data); |
| 1709 | return -EINVAL; | 1729 | return -EINVAL; |
| 1710 | } | 1730 | } |
| 1731 | |||
| 1732 | spin_lock(&chan->lock); | ||
| 1733 | if (chan->task) { | ||
| 1734 | /* another caller won the race; discard our thread */ | ||
| 1735 | spin_unlock(&chan->lock); | ||
| 1736 | kthread_stop(task); | ||
| 1737 | } else { | ||
| 1738 | chan->task = task; | ||
| 1739 | spin_unlock(&chan->lock); | ||
| 1740 | } | ||
| 1711 | } | 1741 | } |
| 1712 | 1742 | ||
| 1713 | pr_debug("%s: sent P-va=%p, P-com=%x, P-size=%u\n", __func__, | 1743 | pr_debug("%s: %s: sent P-va=%p, P-com=%x, P-size=%u\n", __func__, |
| 1714 | p_msg->payload, p_msg->command, | 1744 | chan->name, p_msg->payload, p_msg->command, |
| 1715 | (unsigned int)p_msg->payload_length); | 1745 | (unsigned int)p_msg->payload_length); |
| 1716 | 1746 | ||
| 1717 | if (list_empty(&svc_data_mem)) { | 1747 | if (list_empty(&svc_data_mem)) { |
| @@ -1747,12 +1777,16 @@ int stratix10_svc_send(struct stratix10_svc_chan *chan, void *msg) · 29 unmodified lines | |||
| 1747 | p_data->arg[2] = p_msg->arg[2]; | 1777 | p_data->arg[2] = p_msg->arg[2]; |
| 1748 | p_data->size = p_msg->payload_length; | 1778 | p_data->size = p_msg->payload_length; |
| 1749 | p_data->chan = chan; | 1779 | p_data->chan = chan; |
| 1750 | pr_debug("%s: put to FIFO pa=0x%016x, cmd=%x, size=%u\n", __func__, | 1780 | pr_debug("%s: %s: put to FIFO pa=0x%016x, cmd=%x, size=%u\n", |
| 1751 | (unsigned int)p_data->paddr, p_data->command, | 1781 | __func__, |
| 1752 | (unsigned int)p_data->size); | 1782 | chan->name, |
| 1753 | ret = kfifo_in_spinlocked(&chan->ctrl->svc_fifo, p_data, | 1783 | (unsigned int)p_data->paddr, |
| 1784 | p_data->command, | ||
| 1785 | (unsigned int)p_data->size); | ||
| 1786 | |||
| 1787 | ret = kfifo_in_spinlocked(&chan->svc_fifo, p_data, | ||
| 1754 | sizeof(*p_data), | 1788 | sizeof(*p_data), |
| 1755 | &chan->ctrl->svc_fifo_lock); | 1789 | &chan->svc_fifo_lock); |
| 1756 | 1790 | ||
| 1757 | kfree(p_data); | 1791 | kfree(p_data); |
| 1758 | 1792 | ||
| @@ -1773,11 +1807,12 @@ EXPORT_SYMBOL_GPL(stratix10_svc_send); · 14 unmodified lines | |||
| 1773 | */ | 1807 | */ |
| 1774 | void stratix10_svc_done(struct stratix10_svc_chan *chan) | 1808 | void stratix10_svc_done(struct stratix10_svc_chan *chan) |
| 1775 | { | 1809 | { |
| 1776 | /* stop thread when thread is running AND only one active client */ | 1810 | /* stop thread when thread is running */ |
| 1777 | if (chan->ctrl->task && chan->ctrl->num_active_client <= 1) { | 1811 | if (chan->task) { |
| 1778 | pr_debug("svc_smc_hvc_shm_thread is stopped\n"); | 1812 | pr_debug("%s: %s: svc_smc_hvc_shm_thread is stopping\n", |
| 1779 | kthread_stop(chan->ctrl->task); | 1813 | __func__, chan->name); |
| 1780 | chan->ctrl->task = NULL; | 1814 | kthread_stop(chan->task); |
| 1815 | chan->task = NULL; | ||
| 1781 | } | 1816 | } |
| 1782 | } | 1817 | } |
| 1783 | EXPORT_SYMBOL_GPL(stratix10_svc_done); | 1818 | EXPORT_SYMBOL_GPL(stratix10_svc_done); |
| @@ -1817,8 +1852,8 @@ void *stratix10_svc_allocate_memory(struct stratix10_svc_chan *chan, · 33 unmodified lines | |||
| 1817 | pmem->paddr = pa; | 1852 | pmem->paddr = pa; |
| 1818 | pmem->size = s; | 1853 | pmem->size = s; |
| 1819 | list_add_tail(&pmem->node, &svc_data_mem); | 1854 | list_add_tail(&pmem->node, &svc_data_mem); |
| 1820 | pr_debug("%s: va=%p, pa=0x%016x\n", __func__, | 1855 | pr_debug("%s: %s: va=%p, pa=0x%016x\n", __func__, |
| 1821 | pmem->vaddr, (unsigned int)pmem->paddr); | 1856 | chan->name, pmem->vaddr, (unsigned int)pmem->paddr); |
| 1822 | 1857 | ||
| 1823 | return (void *)va; | 1858 | return (void *)va; |
| 1824 | } | 1859 | } |
| @@ -1855,6 +1890,13 @@ static const struct of_device_id stratix10_svc_drv_match[] = { · 30 unmodified lines | |||
| 1855 | {}, | 1890 | {}, |
| 1856 | }; | 1891 | }; |
| 1857 | 1892 | ||
| 1893 | static const char * const chan_names[SVC_NUM_CHANNEL] = { | ||
| 1894 | SVC_CLIENT_FPGA, | ||
| 1895 | SVC_CLIENT_RSU, | ||
| 1896 | SVC_CLIENT_FCS, | ||
| 1897 | SVC_CLIENT_HWMON | ||
| 1898 | }; | ||
| 1899 | |||
| 1858 | static int stratix10_svc_drv_probe(struct platform_device *pdev) | 1900 | static int stratix10_svc_drv_probe(struct platform_device *pdev) |
| 1859 | { | 1901 | { |
| 1860 | struct device *dev = &pdev->dev; | 1902 | struct device *dev = &pdev->dev; |
| @@ -1862,11 +1904,11 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 1 unmodified line | |||
| 1862 | struct stratix10_svc_chan *chans; | 1904 | struct stratix10_svc_chan *chans; |
| 1863 | struct gen_pool *genpool; | 1905 | struct gen_pool *genpool; |
| 1864 | struct stratix10_svc_sh_memory *sh_memory; | 1906 | struct stratix10_svc_sh_memory *sh_memory; |
| 1865 | struct stratix10_svc *svc; | 1907 | struct stratix10_svc *svc = NULL; |
| 1866 | 1908 | ||
| 1867 | svc_invoke_fn *invoke_fn; | 1909 | svc_invoke_fn *invoke_fn; |
| 1868 | size_t fifo_size; | 1910 | size_t fifo_size; |
| 1869 | int ret; | 1911 | int ret, i = 0; |
| 1870 | 1912 | ||
| 1871 | /* get SMC or HVC function */ | 1913 | /* get SMC or HVC function */ |
| 1872 | invoke_fn = get_invoke_func(dev); | 1914 | invoke_fn = get_invoke_func(dev); |
| @@ -1905,8 +1947,8 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 32 unmodified lines | |||
| 1905 | controller->num_active_client = 0; | 1947 | controller->num_active_client = 0; |
| 1906 | controller->chans = chans; | 1948 | controller->chans = chans; |
| 1907 | controller->genpool = genpool; | 1949 | controller->genpool = genpool; |
| 1908 | controller->task = NULL; | ||
| 1909 | controller->invoke_fn = invoke_fn; | 1950 | controller->invoke_fn = invoke_fn; |
| 1951 | INIT_LIST_HEAD(&controller->node); | ||
| 1910 | init_completion(&controller->complete_status); | 1952 | init_completion(&controller->complete_status); |
| 1911 | 1953 | ||
| 1912 | ret = stratix10_svc_async_init(controller); | 1954 | ret = stratix10_svc_async_init(controller); |
| @@ -1917,32 +1959,20 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 4 unmodified lines | |||
| 1917 | } | 1959 | } |
| 1918 | 1960 | ||
| 1919 | fifo_size = sizeof(struct stratix10_svc_data) * SVC_NUM_DATA_IN_FIFO; | 1961 | fifo_size = sizeof(struct stratix10_svc_data) * SVC_NUM_DATA_IN_FIFO; |
| 1920 | ret = kfifo_alloc(&controller->svc_fifo, fifo_size, GFP_KERNEL); | 1962 | mutex_init(&controller->sdm_lock); |
| 1921 | if (ret) { | ||
| 1922 | dev_err(dev, "failed to allocate FIFO\n"); | ||
| 1923 | goto err_async_exit; | ||
| 1924 | } | ||
| 1925 | spin_lock_init(&controller->svc_fifo_lock); | ||
| 1926 | |||
| 1927 | chans[0].scl = NULL; | ||
| 1928 | chans[0].ctrl = controller; | ||
| 1929 | chans[0].name = SVC_CLIENT_FPGA; | ||
| 1930 | spin_lock_init(&chans[0].lock); | ||
| 1931 | 1963 | ||
| 1932 | chans[1].scl = NULL; | 1964 | for (i = 0; i < SVC_NUM_CHANNEL; i++) { |
| 1933 | chans[1].ctrl = controller; | 1965 | chans[i].scl = NULL; |
| 1934 | chans[1].name = SVC_CLIENT_RSU; | 1966 | chans[i].ctrl = controller; |
| 1935 | spin_lock_init(&chans[1].lock); | 1967 | chans[i].name = (char *)chan_names[i]; |
| 1936 | 1968 | spin_lock_init(&chans[i].lock); | |
| 1937 | chans[2].scl = NULL; | 1969 | ret = kfifo_alloc(&chans[i].svc_fifo, fifo_size, GFP_KERNEL); |
| 1938 | chans[2].ctrl = controller; | 1970 | if (ret) { |
| 1939 | chans[2].name = SVC_CLIENT_FCS; | 1971 | dev_err(dev, "failed to allocate FIFO %d\n", i); |
| 1940 | spin_lock_init(&chans[2].lock); | 1972 | goto err_free_fifos; |
| 1941 | 1973 | } | |
| 1942 | chans[3].scl = NULL; | 1974 | spin_lock_init(&chans[i].svc_fifo_lock); |
| 1943 | chans[3].ctrl = controller; | 1975 | } |
| 1944 | chans[3].name = SVC_CLIENT_HWMON; | ||
| 1945 | spin_lock_init(&chans[3].lock); | ||
| 1946 | 1976 | ||
| 1947 | list_add_tail(&controller->node, &svc_ctrl); | 1977 | list_add_tail(&controller->node, &svc_ctrl); |
| 1948 | platform_set_drvdata(pdev, controller); | 1978 | platform_set_drvdata(pdev, controller); |
| @@ -1951,7 +1981,7 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 2 unmodified lines | |||
| 1951 | svc = devm_kzalloc(dev, sizeof(*svc), GFP_KERNEL); | 1981 | svc = devm_kzalloc(dev, sizeof(*svc), GFP_KERNEL); |
| 1952 | if (!svc) { | 1982 | if (!svc) { |
| 1953 | ret = -ENOMEM; | 1983 | ret = -ENOMEM; |
| 1954 | goto err_free_kfifo; | 1984 | goto err_free_fifos; |
| 1955 | } | 1985 | } |
| 1956 | controller->svc = svc; | 1986 | controller->svc = svc; |
| 1957 | 1987 | ||
| @@ -1959,51 +1989,43 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev) · 1 unmodified line | |||
| 1959 | if (!svc->stratix10_svc_rsu) { | 1989 | if (!svc->stratix10_svc_rsu) { |
| 1960 | dev_err(dev, "failed to allocate %s device\n", STRATIX10_RSU); | 1990 | dev_err(dev, "failed to allocate %s device\n", STRATIX10_RSU); |
| 1961 | ret = -ENOMEM; | 1991 | ret = -ENOMEM; |
| 1962 | goto err_free_kfifo; | 1992 | goto err_free_fifos; |
| 1963 | } | 1993 | } |
| 1964 | 1994 | ||
| 1965 | ret = platform_device_add(svc->stratix10_svc_rsu); | 1995 | ret = platform_device_add(svc->stratix10_svc_rsu); |
| 1966 | if (ret) { | 1996 | if (ret) |
| 1967 | platform_device_put(svc->stratix10_svc_rsu); | 1997 | goto err_put_device; |
| 1968 | goto err_free_kfifo; | ||
| 1969 | } | ||
| 1970 | |||
| 1971 | svc->intel_svc_fcs = platform_device_alloc(INTEL_FCS, 1); | ||
| 1972 | if (!svc->intel_svc_fcs) { | ||
| 1973 | dev_err(dev, "failed to allocate %s device\n", INTEL_FCS); | ||
| 1974 | ret = -ENOMEM; | ||
| 1975 | goto err_unregister_rsu_dev; | ||
| 1976 | } | ||
| 1977 | |||
| 1978 | ret = platform_device_add(svc->intel_svc_fcs); | ||
| 1979 | if (ret) { | ||
| 1980 | platform_device_put(svc->intel_svc_fcs); | ||
| 1981 | goto err_unregister_rsu_dev; | ||
| 1982 | } | ||
| 1983 | 1998 | ||
| 1984 | ret = of_platform_default_populate(dev_of_node(dev), NULL, dev); | 1999 | ret = of_platform_default_populate(dev_of_node(dev), NULL, dev); |
| 1985 | if (ret) | 2000 | if (ret) |
| 1986 | goto err_unregister_fcs_dev; | 2001 | goto err_unregister_rsu_dev; |
| 1987 | 2002 | ||
| 1988 | pr_info("Intel Service Layer Driver Initialized\n"); | 2003 | pr_info("Intel Service Layer Driver Initialized\n"); |
| 1989 | 2004 | ||
| 1990 | return 0; | 2005 | return 0; |
| 1991 | 2006 | ||
| 1992 | err_unregister_fcs_dev: | ||
| 1993 | platform_device_unregister(svc->intel_svc_fcs); | ||
| 1994 | err_unregister_rsu_dev: | 2007 | err_unregister_rsu_dev: |
| 1995 | platform_device_unregister(svc->stratix10_svc_rsu); | 2008 | platform_device_unregister(svc->stratix10_svc_rsu); |
| 1996 | err_free_kfifo: | 2009 | goto err_free_fifos; |
| 1997 | kfifo_free(&controller->svc_fifo); | 2010 | err_put_device: |
| 1998 | err_async_exit: | 2011 | platform_device_put(svc->stratix10_svc_rsu); |
| 2012 | err_free_fifos: | ||
| 2013 | /* only remove from list if list_add_tail() was reached */ | ||
| 2014 | if (!list_empty(&controller->node)) | ||
| 2015 | list_del(&controller->node); | ||
| 2016 | /* free only the FIFOs that were successfully allocated */ | ||
| 2017 | while (i--) | ||
| 2018 | kfifo_free(&chans[i].svc_fifo); | ||
| 1999 | stratix10_svc_async_exit(controller); | 2019 | stratix10_svc_async_exit(controller); |
| 2000 | err_destroy_pool: | 2020 | err_destroy_pool: |
| 2001 | gen_pool_destroy(genpool); | 2021 | gen_pool_destroy(genpool); |
| 2022 | |||
| 2002 | return ret; | 2023 | return ret; |
| 2003 | } | 2024 | } |
| 2004 | 2025 | ||
| 2005 | static void stratix10_svc_drv_remove(struct platform_device *pdev) | 2026 | static void stratix10_svc_drv_remove(struct platform_device *pdev) |
| 2006 | { | 2027 | { |
| 2028 | int i; | ||
| 2007 | struct stratix10_svc_controller *ctrl = platform_get_drvdata(pdev); | 2029 | struct stratix10_svc_controller *ctrl = platform_get_drvdata(pdev); |
| 2008 | struct stratix10_svc *svc = ctrl->svc; | 2030 | struct stratix10_svc *svc = ctrl->svc; |
| 2009 | 2031 | ||
| @@ -2011,14 +2033,16 @@ static void stratix10_svc_drv_remove(struct platform_device *pdev) · 1 unmodified line | |||
| 2011 | 2033 | ||
| 2012 | of_platform_depopulate(ctrl->dev); | 2034 | of_platform_depopulate(ctrl->dev); |
| 2013 | 2035 | ||
| 2014 | platform_device_unregister(svc->intel_svc_fcs); | ||
| 2015 | platform_device_unregister(svc->stratix10_svc_rsu); | 2036 | platform_device_unregister(svc->stratix10_svc_rsu); |
| 2016 | 2037 | ||
| 2017 | kfifo_free(&ctrl->svc_fifo); | 2038 | for (i = 0; i < SVC_NUM_CHANNEL; i++) { |
| 2018 | if (ctrl->task) { | 2039 | if (ctrl->chans[i].task) { |
| 2019 | kthread_stop(ctrl->task); | 2040 | kthread_stop(ctrl->chans[i].task); |
| 2020 | ctrl->task = NULL; | 2041 | ctrl->chans[i].task = NULL; |
| 2042 | } | ||
| 2043 | kfifo_free(&ctrl->chans[i].svc_fifo); | ||
| 2021 | } | 2044 | } |
| 2045 | |||
| 2022 | if (ctrl->genpool) | 2046 | if (ctrl->genpool) |
| 2023 | gen_pool_destroy(ctrl->genpool); | 2047 | gen_pool_destroy(ctrl->genpool); |
| 2024 | list_del(&ctrl->node); | 2048 | list_del(&ctrl->node); |
| @@ -122,6 +122,7 @@ config GPIB_FLUKE · 121 unmodified lines | |||
| 122 | depends on OF | 122 | depends on OF |
| 123 | select GPIB_COMMON | 123 | select GPIB_COMMON |
| 124 | select GPIB_NEC7210 | 124 | select GPIB_NEC7210 |
| 125 | depends on HAS_IOMEM | ||
| 125 | help | 126 | help |
| 126 | GPIB driver for Fluke based cda devices. | 127 | GPIB driver for Fluke based cda devices. |
| 127 | 128 | ||
| @@ -888,10 +888,6 @@ static int read_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 887 unmodified lines | |||
| 888 | if (read_cmd.completed_transfer_count > read_cmd.requested_transfer_count) | 888 | if (read_cmd.completed_transfer_count > read_cmd.requested_transfer_count) |
| 889 | return -EINVAL; | 889 | return -EINVAL; |
| 890 | 890 | ||
| 891 | desc = handle_to_descriptor(file_priv, read_cmd.handle); | ||
| 892 | if (!desc) | ||
| 893 | return -EINVAL; | ||
| 894 | |||
| 895 | if (WARN_ON_ONCE(sizeof(userbuf) > sizeof(read_cmd.buffer_ptr))) | 891 | if (WARN_ON_ONCE(sizeof(userbuf) > sizeof(read_cmd.buffer_ptr))) |
| 896 | return -EFAULT; | 892 | return -EFAULT; |
| 897 | 893 | ||
| @@ -904,6 +900,17 @@ static int read_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 6 unmodified lines | |||
| 904 | if (!access_ok(userbuf, remain)) | 900 | if (!access_ok(userbuf, remain)) |
| 905 | return -EFAULT; | 901 | return -EFAULT; |
| 906 | 902 | ||
| 903 | /* Lock descriptors to prevent concurrent close from freeing descriptor */ | ||
| 904 | if (mutex_lock_interruptible(&file_priv->descriptors_mutex)) | ||
| 905 | return -ERESTARTSYS; | ||
| 906 | desc = handle_to_descriptor(file_priv, read_cmd.handle); | ||
| 907 | if (!desc) { | ||
| 908 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 909 | return -EINVAL; | ||
| 910 | } | ||
| 911 | atomic_inc(&desc->descriptor_busy); | ||
| 912 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 913 | |||
| 907 | atomic_set(&desc->io_in_progress, 1); | 914 | atomic_set(&desc->io_in_progress, 1); |
| 908 | 915 | ||
| 909 | /* Read buffer loads till we fill the user supplied buffer */ | 916 | /* Read buffer loads till we fill the user supplied buffer */ |
| @@ -937,6 +944,7 @@ static int read_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 27 unmodified lines | |||
| 937 | retval = copy_to_user((void __user *)arg, &read_cmd, sizeof(read_cmd)); | 944 | retval = copy_to_user((void __user *)arg, &read_cmd, sizeof(read_cmd)); |
| 938 | 945 | ||
| 939 | atomic_set(&desc->io_in_progress, 0); | 946 | atomic_set(&desc->io_in_progress, 0); |
| 947 | atomic_dec(&desc->descriptor_busy); | ||
| 940 | 948 | ||
| 941 | wake_up_interruptible(&board->wait); | 949 | wake_up_interruptible(&board->wait); |
| 942 | if (retval) | 950 | if (retval) |
| @@ -964,10 +972,6 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 21 unmodified lines | |||
| 964 | if (cmd.completed_transfer_count > cmd.requested_transfer_count) | 972 | if (cmd.completed_transfer_count > cmd.requested_transfer_count) |
| 965 | return -EINVAL; | 973 | return -EINVAL; |
| 966 | 974 | ||
| 967 | desc = handle_to_descriptor(file_priv, cmd.handle); | ||
| 968 | if (!desc) | ||
| 969 | return -EINVAL; | ||
| 970 | |||
| 971 | userbuf = (u8 __user *)(unsigned long)cmd.buffer_ptr; | 975 | userbuf = (u8 __user *)(unsigned long)cmd.buffer_ptr; |
| 972 | userbuf += cmd.completed_transfer_count; | 976 | userbuf += cmd.completed_transfer_count; |
| 973 | 977 | ||
| @@ -980,6 +984,17 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 6 unmodified lines | |||
| 980 | if (!access_ok(userbuf, remain)) | 984 | if (!access_ok(userbuf, remain)) |
| 981 | return -EFAULT; | 985 | return -EFAULT; |
| 982 | 986 | ||
| 987 | /* Lock descriptors to prevent concurrent close from freeing descriptor */ | ||
| 988 | if (mutex_lock_interruptible(&file_priv->descriptors_mutex)) | ||
| 989 | return -ERESTARTSYS; | ||
| 990 | desc = handle_to_descriptor(file_priv, cmd.handle); | ||
| 991 | if (!desc) { | ||
| 992 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 993 | return -EINVAL; | ||
| 994 | } | ||
| 995 | atomic_inc(&desc->descriptor_busy); | ||
| 996 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 997 | |||
| 983 | /* | 998 | /* |
| 984 | * Write buffer loads till we empty the user supplied buffer. | 999 | * Write buffer loads till we empty the user supplied buffer. |
| 985 | * Call drivers at least once, even if remain is zero, in | 1000 | * Call drivers at least once, even if remain is zero, in |
| @@ -1003,6 +1018,7 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 17 unmodified lines | |||
| 1003 | userbuf += bytes_written; | 1018 | userbuf += bytes_written; |
| 1004 | if (retval < 0) { | 1019 | if (retval < 0) { |
| 1005 | atomic_set(&desc->io_in_progress, 0); | 1020 | atomic_set(&desc->io_in_progress, 0); |
| 1021 | atomic_dec(&desc->descriptor_busy); | ||
| 1006 | 1022 | ||
| 1007 | wake_up_interruptible(&board->wait); | 1023 | wake_up_interruptible(&board->wait); |
| 1008 | break; | 1024 | break; |
| @@ -1022,6 +1038,7 @@ static int command_ioctl(struct gpib_file_private *file_priv, · 13 unmodified lines | |||
| 1022 | */ | 1038 | */ |
| 1023 | if (!no_clear_io_in_prog || fault) | 1039 | if (!no_clear_io_in_prog || fault) |
| 1024 | atomic_set(&desc->io_in_progress, 0); | 1040 | atomic_set(&desc->io_in_progress, 0); |
| 1041 | atomic_dec(&desc->descriptor_busy); | ||
| 1025 | 1042 | ||
| 1026 | wake_up_interruptible(&board->wait); | 1043 | wake_up_interruptible(&board->wait); |
| 1027 | if (fault) | 1044 | if (fault) |
| @@ -1047,10 +1064,6 @@ static int write_ioctl(struct gpib_file_private *file_priv, struct gpib_board *b · 19 unmodified lines | |||
| 1047 | if (write_cmd.completed_transfer_count > write_cmd.requested_transfer_count) | 1064 | if (write_cmd.completed_transfer_count > write_cmd.requested_transfer_count) |
| 1048 | return -EINVAL; | 1065 | return -EINVAL; |
| 1049 | 1066 | ||
| 1050 | desc = handle_to_descriptor(file_priv, write_cmd.handle); | ||
| 1051 | if (!desc) | ||
| 1052 | return -EINVAL; | ||
| 1053 | |||
| 1054 | userbuf = (u8 __user *)(unsigned long)write_cmd.buffer_ptr; | 1067 | userbuf = (u8 __user *)(unsigned long)write_cmd.buffer_ptr; |
| 1055 | userbuf += write_cmd.completed_transfer_count; | 1068 | userbuf += write_cmd.completed_transfer_count; |
| 1056 | 1069 | ||
| @@ -1060,6 +1073,17 @@ static int write_ioctl(struct gpib_file_private *file_priv, struct gpib_board *b · 3 unmodified lines | |||
| 1060 | if (!access_ok(userbuf, remain)) | 1073 | if (!access_ok(userbuf, remain)) |
| 1061 | return -EFAULT; | 1074 | return -EFAULT; |
| 1062 | 1075 | ||
| 1076 | /* Lock descriptors to prevent concurrent close from freeing descriptor */ | ||
| 1077 | if (mutex_lock_interruptible(&file_priv->descriptors_mutex)) | ||
| 1078 | return -ERESTARTSYS; | ||
| 1079 | desc = handle_to_descriptor(file_priv, write_cmd.handle); | ||
| 1080 | if (!desc) { | ||
| 1081 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 1082 | return -EINVAL; | ||
| 1083 | } | ||
| 1084 | atomic_inc(&desc->descriptor_busy); | ||
| 1085 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 1086 | |||
| 1063 | atomic_set(&desc->io_in_progress, 1); | 1087 | atomic_set(&desc->io_in_progress, 1); |
| 1064 | 1088 | ||
| 1065 | /* Write buffer loads till we empty the user supplied buffer */ | 1089 | /* Write buffer loads till we empty the user supplied buffer */ |
| @@ -1094,6 +1118,7 @@ static int write_ioctl(struct gpib_file_private *file_priv, struct gpib_board *b · 28 unmodified lines | |||
| 1094 | fault = copy_to_user((void __user *)arg, &write_cmd, sizeof(write_cmd)); | 1118 | fault = copy_to_user((void __user *)arg, &write_cmd, sizeof(write_cmd)); |
| 1095 | 1119 | ||
| 1096 | atomic_set(&desc->io_in_progress, 0); | 1120 | atomic_set(&desc->io_in_progress, 0); |
| 1121 | atomic_dec(&desc->descriptor_busy); | ||
| 1097 | 1122 | ||
| 1098 | wake_up_interruptible(&board->wait); | 1123 | wake_up_interruptible(&board->wait); |
| 1099 | if (fault) | 1124 | if (fault) |
| @@ -1276,6 +1301,9 @@ static int close_dev_ioctl(struct file *filep, struct gpib_board *board, unsigne · 176 unmodified lines | |||
| 1276 | { | 1301 | { |
| 1277 | struct gpib_close_dev_ioctl cmd; | 1302 | struct gpib_close_dev_ioctl cmd; |
| 1278 | struct gpib_file_private *file_priv = filep->private_data; | 1303 | struct gpib_file_private *file_priv = filep->private_data; |
| 1304 | struct gpib_descriptor *desc; | ||
| 1305 | unsigned int pad; | ||
| 1306 | int sad; | ||
| 1279 | int retval; | 1307 | int retval; |
| 1280 | 1308 | ||
| 1281 | retval = copy_from_user(&cmd, (void __user *)arg, sizeof(cmd)); | 1309 | retval = copy_from_user(&cmd, (void __user *)arg, sizeof(cmd)); |
| @@ -1284,19 +1312,27 @@ static int close_dev_ioctl(struct file *filep, struct gpib_board *board, unsigne · 2 unmodified lines | |||
| 1284 | 1312 | ||
| 1285 | if (cmd.handle >= GPIB_MAX_NUM_DESCRIPTORS) | 1313 | if (cmd.handle >= GPIB_MAX_NUM_DESCRIPTORS) |
| 1286 | return -EINVAL; | 1314 | return -EINVAL; |
| 1287 | if (!file_priv->descriptors[cmd.handle]) | ||
| 1288 | return -EINVAL; | ||
| 1289 | 1315 | ||
| 1290 | retval = decrement_open_device_count(board, &board->device_list, | 1316 | mutex_lock(&file_priv->descriptors_mutex); |
| 1291 | file_priv->descriptors[cmd.handle]->pad, | 1317 | desc = file_priv->descriptors[cmd.handle]; |
| 1292 | file_priv->descriptors[cmd.handle]->sad); | 1318 | if (!desc) { |
| 1293 | if (retval < 0) | 1319 | mutex_unlock(&file_priv->descriptors_mutex); |
| 1294 | return retval; | 1320 | return -EINVAL; |
| 1295 | 1321 | } | |
| 1296 | kfree(file_priv->descriptors[cmd.handle]); | 1322 | if (atomic_read(&desc->descriptor_busy)) { |
| 1323 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 1324 | return -EBUSY; | ||
| 1325 | } | ||
| 1326 | /* Remove from table while holding lock to prevent new IO from starting */ | ||
| 1297 | file_priv->descriptors[cmd.handle] = NULL; | 1327 | file_priv->descriptors[cmd.handle] = NULL; |
| 1328 | pad = desc->pad; | ||
| 1329 | sad = desc->sad; | ||
| 1330 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 1298 | 1331 | ||
| 1299 | return 0; | 1332 | retval = decrement_open_device_count(board, &board->device_list, pad, sad); |
| 1333 | |||
| 1334 | kfree(desc); | ||
| 1335 | return retval; | ||
| 1300 | } | 1336 | } |
| 1301 | 1337 | ||
| 1302 | static int serial_poll_ioctl(struct gpib_board *board, unsigned long arg) | 1338 | static int serial_poll_ioctl(struct gpib_board *board, unsigned long arg) |
| @@ -1331,12 +1367,25 @@ static int wait_ioctl(struct gpib_file_private *file_priv, struct gpib_board *bo · 28 unmodified lines | |||
| 1331 | if (retval) | 1367 | if (retval) |
| 1332 | return -EFAULT; | 1368 | return -EFAULT; |
| 1333 | 1369 | ||
| 1370 | /* | ||
| 1371 | * Lock descriptors to prevent concurrent close from freeing | ||
| 1372 | * descriptor. ibwait() releases big_gpib_mutex when wait_mask | ||
| 1373 | * is non-zero, so desc must be pinned with descriptor_busy. | ||
| 1374 | */ | ||
| 1375 | mutex_lock(&file_priv->descriptors_mutex); | ||
| 1334 | desc = handle_to_descriptor(file_priv, wait_cmd.handle); | 1376 | desc = handle_to_descriptor(file_priv, wait_cmd.handle); |
| 1335 | if (!desc) | 1377 | if (!desc) { |
| 1378 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 1336 | return -EINVAL; | 1379 | return -EINVAL; |
| 1380 | } | ||
| 1381 | atomic_inc(&desc->descriptor_busy); | ||
| 1382 | mutex_unlock(&file_priv->descriptors_mutex); | ||
| 1337 | 1383 | ||
| 1338 | retval = ibwait(board, wait_cmd.wait_mask, wait_cmd.clear_mask, | 1384 | retval = ibwait(board, wait_cmd.wait_mask, wait_cmd.clear_mask, |
| 1339 | wait_cmd.set_mask, &wait_cmd.ibsta, wait_cmd.usec_timeout, desc); | 1385 | wait_cmd.set_mask, &wait_cmd.ibsta, wait_cmd.usec_timeout, desc); |
| 1386 | |||
| 1387 | atomic_dec(&desc->descriptor_busy); | ||
| 1388 | |||
| 1340 | if (retval < 0) | 1389 | if (retval < 0) |
| 1341 | return retval; | 1390 | return retval; |
| 1342 | 1391 | ||
| @@ -2035,6 +2084,7 @@ void init_gpib_descriptor(struct gpib_descriptor *desc) · 692 unmodified lines | |||
| 2035 | desc->is_board = 0; | 2084 | desc->is_board = 0; |
| 2036 | desc->autopoll_enabled = 0; | 2085 | desc->autopoll_enabled = 0; |
| 2037 | atomic_set(&desc->io_in_progress, 0); | 2086 | atomic_set(&desc->io_in_progress, 0); |
| 2087 | atomic_set(&desc->descriptor_busy, 0); | ||
| 2038 | } | 2088 | } |
| 2039 | 2089 | ||
| 2040 | int gpib_register_driver(struct gpib_interface *interface, struct module *provider_module) | 2090 | int gpib_register_driver(struct gpib_interface *interface, struct module *provider_module) |
| @@ -364,6 +364,14 @@ struct gpib_descriptor { · 363 unmodified lines | |||
| 364 | unsigned int pad; /* primary gpib address */ | 364 | unsigned int pad; /* primary gpib address */ |
| 365 | int sad; /* secondary gpib address (negative means disabled) */ | 365 | int sad; /* secondary gpib address (negative means disabled) */ |
| 366 | atomic_t io_in_progress; | 366 | atomic_t io_in_progress; |
| 367 | /* | ||
| 368 | * Kernel-only reference count to prevent descriptor from being | ||
| 369 | * freed while IO handlers hold a pointer to it. Incremented | ||
| 370 | * before each IO operation, decremented when done. Unlike | ||
| 371 | * io_in_progress, this cannot be modified from userspace via | ||
| 372 | * general_ibstatus(). | ||
| 373 | */ | ||
| 374 | atomic_t descriptor_busy; | ||
| 367 | unsigned is_board : 1; | 375 | unsigned is_board : 1; |
| 368 | unsigned autopoll_enabled : 1; | 376 | unsigned autopoll_enabled : 1; |
| 369 | }; | 377 | }; |
| @@ -38,8 +38,10 @@ MODULE_DESCRIPTION("GPIB driver for LPVO usb devices"); · 37 unmodified lines | |||
| 38 | /* | 38 | /* |
| 39 | * Table of devices that work with this driver. | 39 | * Table of devices that work with this driver. |
| 40 | * | 40 | * |
| 41 | * Currently, only one device is known to be used in the | 41 | * Currently, only one device is known to be used in the lpvo_usb_gpib |
| 42 | * lpvo_usb_gpib adapter (FTDI 0403:6001). | 42 | * adapter (FTDI 0403:6001) but as this device id is already handled by the |
| 43 | * ftdi_sio USB serial driver the LPVO driver must not bind to it by default. | ||
| 44 | * | ||
| 43 | * If your adapter uses a different chip, insert a line | 45 | * If your adapter uses a different chip, insert a line |
| 44 | * in the following table with proper <Vendor-id>, <Product-id>. | 46 | * in the following table with proper <Vendor-id>, <Product-id>. |
| 45 | * | 47 | * |
| @@ -50,7 +52,6 @@ MODULE_DESCRIPTION("GPIB driver for LPVO usb devices"); · 4 unmodified lines | |||
| 50 | */ | 52 | */ |
| 51 | 53 | ||
| 52 | static const struct usb_device_id skel_table[] = { | 54 | static const struct usb_device_id skel_table[] = { |
| 53 | { USB_DEVICE(0x0403, 0x6001) }, | ||
| 54 | { } /* Terminating entry */ | 55 | { } /* Terminating entry */ |
| 55 | }; | 56 | }; |
| 56 | MODULE_DEVICE_TABLE(usb, skel_table); | 57 | MODULE_DEVICE_TABLE(usb, skel_table); |
| @@ -405,7 +406,7 @@ static int usb_gpib_attach(struct gpib_board *board, const struct gpib_board_con · 348 unmodified lines | |||
| 405 | for (j = 0 ; j < MAX_DEV ; j++) { | 406 | for (j = 0 ; j < MAX_DEV ; j++) { |
| 406 | if ((assigned_usb_minors & 1 << j) == 0) | 407 | if ((assigned_usb_minors & 1 << j) == 0) |
| 407 | continue; | 408 | continue; |
| 408 | udev = usb_get_dev(interface_to_usbdev(lpvo_usb_interfaces[j])); | 409 | udev = interface_to_usbdev(lpvo_usb_interfaces[j]); |
| 409 | device_path = kobject_get_path(&udev->dev.kobj, GFP_KERNEL); | 410 | device_path = kobject_get_path(&udev->dev.kobj, GFP_KERNEL); |
| 410 | match = gpib_match_device_path(&lpvo_usb_interfaces[j]->dev, | 411 | match = gpib_match_device_path(&lpvo_usb_interfaces[j]->dev, |
| 411 | config->device_path); | 412 | config->device_path); |
| @@ -420,7 +421,7 @@ static int usb_gpib_attach(struct gpib_board *board, const struct gpib_board_con · 8 unmodified lines | |||
| 420 | for (j = 0 ; j < MAX_DEV ; j++) { | 421 | for (j = 0 ; j < MAX_DEV ; j++) { |
| 421 | if ((assigned_usb_minors & 1 << j) == 0) | 422 | if ((assigned_usb_minors & 1 << j) == 0) |
| 422 | continue; | 423 | continue; |
| 423 | udev = usb_get_dev(interface_to_usbdev(lpvo_usb_interfaces[j])); | 424 | udev = interface_to_usbdev(lpvo_usb_interfaces[j]); |
| 424 | DIA_LOG(1, "dev. %d: bus %d -> %d dev: %d -> %d\n", j, | 425 | DIA_LOG(1, "dev. %d: bus %d -> %d dev: %d -> %d\n", j, |
| 425 | udev->bus->busnum, config->pci_bus, udev->devnum, config->pci_slot); | 426 | udev->bus->busnum, config->pci_bus, udev->devnum, config->pci_slot); |
| 426 | if (config->pci_bus == udev->bus->busnum && | 427 | if (config->pci_bus == udev->bus->busnum && |
| @@ -584,12 +584,13 @@ static bool mxc_gpio_set_pad_wakeup(struct mxc_gpio_port *port, bool enable) · 583 unmodified lines | |||
| 584 | unsigned long config; | 584 | unsigned long config; |
| 585 | bool ret = false; | 585 | bool ret = false; |
| 586 | int i, type; | 586 | int i, type; |
| 587 | bool is_imx8qm = of_device_is_compatible(port->dev->of_node, "fsl,imx8qm-gpio"); | ||
| 587 | 588 | ||
| 588 | static const u32 pad_type_map[] = { | 589 | static const u32 pad_type_map[] = { |
| 589 | IMX_SCU_WAKEUP_OFF, /* 0 */ | 590 | IMX_SCU_WAKEUP_OFF, /* 0 */ |
| 590 | IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_RISING */ | 591 | IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_RISING */ |
| 591 | IMX_SCU_WAKEUP_FALL_EDGE, /* IRQ_TYPE_EDGE_FALLING */ | 592 | IMX_SCU_WAKEUP_FALL_EDGE, /* IRQ_TYPE_EDGE_FALLING */ |
| 592 | IMX_SCU_WAKEUP_FALL_EDGE, /* IRQ_TYPE_EDGE_BOTH */ | 593 | IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_BOTH */ |
| 593 | IMX_SCU_WAKEUP_HIGH_LVL, /* IRQ_TYPE_LEVEL_HIGH */ | 594 | IMX_SCU_WAKEUP_HIGH_LVL, /* IRQ_TYPE_LEVEL_HIGH */ |
| 594 | IMX_SCU_WAKEUP_OFF, /* 5 */ | 595 | IMX_SCU_WAKEUP_OFF, /* 5 */ |
| 595 | IMX_SCU_WAKEUP_OFF, /* 6 */ | 596 | IMX_SCU_WAKEUP_OFF, /* 6 */ |
| @@ -604,6 +605,13 @@ static bool mxc_gpio_set_pad_wakeup(struct mxc_gpio_port *port, bool enable) · 8 unmodified lines | |||
| 604 | config = pad_type_map[type]; | 605 | config = pad_type_map[type]; |
| 605 | else | 606 | else |
| 606 | config = IMX_SCU_WAKEUP_OFF; | 607 | config = IMX_SCU_WAKEUP_OFF; |
| 608 | |||
| 609 | if (is_imx8qm && config == IMX_SCU_WAKEUP_FALL_EDGE) { | ||
| 610 | dev_warn_once(port->dev, | ||
| 611 | "No falling-edge support for wakeup on i.MX8QM\n"); | ||
| 612 | config = IMX_SCU_WAKEUP_OFF; | ||
| 613 | } | ||
| 614 | |||
| 607 | ret |= mxc_gpio_generic_config(port, i, config); | 615 | ret |= mxc_gpio_generic_config(port, i, config); |
| 608 | } | 616 | } |
| 609 | } | 617 | } |
| @@ -60,8 +60,8 @@ static int qixis_cpld_gpio_probe(struct platform_device *pdev) · 59 unmodified lines | |||
| 60 | return PTR_ERR(reg); | 60 | return PTR_ERR(reg); |
| 61 | 61 | ||
| 62 | regmap = devm_regmap_init_mmio(&pdev->dev, reg, ®map_config_8r_8v); | 62 | regmap = devm_regmap_init_mmio(&pdev->dev, reg, ®map_config_8r_8v); |
| 63 | if (!regmap) | 63 | if (IS_ERR(regmap)) |
| 64 | return -ENODEV; | 64 | return PTR_ERR(regmap); |
| 65 | 65 | ||
| 66 | /* In this case, the offset of our register is 0 inside the | 66 | /* In this case, the offset of our register is 0 inside the |
| 67 | * regmap area that we just created. | 67 | * regmap area that we just created. |
| @@ -443,8 +443,8 @@ static bool gpio_shared_dev_is_reset_gpio(struct device *consumer, · 442 unmodified lines | |||
| 443 | } | 443 | } |
| 444 | #endif /* CONFIG_RESET_GPIO */ | 444 | #endif /* CONFIG_RESET_GPIO */ |
| 445 | 445 | ||
| 446 | int gpio_shared_add_proxy_lookup(struct device *consumer, const char *con_id, | 446 | int gpio_shared_add_proxy_lookup(struct device *consumer, struct fwnode_handle *fwnode, |
| 447 | unsigned long lflags) | 447 | const char *con_id, unsigned long lflags) |
| 448 | { | 448 | { |
| 449 | const char *dev_id = dev_name(consumer); | 449 | const char *dev_id = dev_name(consumer); |
| 450 | struct gpiod_lookup_table *lookup; | 450 | struct gpiod_lookup_table *lookup; |
| @@ -458,7 +458,7 @@ int gpio_shared_add_proxy_lookup(struct device *consumer, const char *con_id, · 7 unmodified lines | |||
| 458 | if (!ref->fwnode && device_is_compatible(consumer, "reset-gpio")) { | 458 | if (!ref->fwnode && device_is_compatible(consumer, "reset-gpio")) { |
| 459 | if (!gpio_shared_dev_is_reset_gpio(consumer, entry, ref)) | 459 | if (!gpio_shared_dev_is_reset_gpio(consumer, entry, ref)) |
| 460 | continue; | 460 | continue; |
| 461 | } else if (!device_match_fwnode(consumer, ref->fwnode)) { | 461 | } else if (fwnode != ref->fwnode) { |
| 462 | continue; | 462 | continue; |
| 463 | } | 463 | } |
| 464 | 464 | ||
| @@ -506,8 +506,9 @@ static void gpio_shared_remove_adev(struct auxiliary_device *adev) · 41 unmodified lines | |||
| 506 | auxiliary_device_uninit(adev); | 506 | auxiliary_device_uninit(adev); |
| 507 | } | 507 | } |
| 508 | 508 | ||
| 509 | int gpio_device_setup_shared(struct gpio_device *gdev) | 509 | int gpiochip_setup_shared(struct gpio_chip *gc) |
| 510 | { | 510 | { |
| 511 | struct gpio_device *gdev = gc->gpiodev; | ||
| 511 | struct gpio_shared_entry *entry; | 512 | struct gpio_shared_entry *entry; |
| 512 | struct gpio_shared_ref *ref; | 513 | struct gpio_shared_ref *ref; |
| 513 | struct gpio_desc *desc; | 514 | struct gpio_desc *desc; |
| @@ -538,20 +539,42 @@ int gpio_device_setup_shared(struct gpio_device *gdev) · 24 unmodified lines | |||
| 538 | if (list_count_nodes(&entry->refs) <= 1) | 539 | if (list_count_nodes(&entry->refs) <= 1) |
| 539 | continue; | 540 | continue; |
| 540 | 541 | ||
| 541 | desc = &gdev->descs[entry->offset]; | 542 | scoped_guard(mutex, &entry->lock) { |
| 543 | #if IS_ENABLED(CONFIG_OF) | ||
| 544 | if (is_of_node(entry->fwnode) && gc->of_xlate) { | ||
| 545 | /* | ||
| 546 | * This is the earliest that we can tranlate the | ||
| 547 | * devicetree offset to the chip offset. | ||
| 548 | */ | ||
| 549 | struct of_phandle_args gpiospec = { }; | ||
| 542 | 550 | ||
| 543 | __set_bit(GPIOD_FLAG_SHARED, &desc->flags); | 551 | gpiospec.np = to_of_node(entry->fwnode); |
| 544 | /* | 552 | gpiospec.args_count = 2; |
| 545 | * Shared GPIOs are not requested via the normal path. Make | 553 | gpiospec.args[0] = entry->offset; |
| 546 | * them inaccessible to anyone even before we register the | 554 | |
| 547 | * chip. | 555 | ret = gc->of_xlate(gc, &gpiospec, NULL); |
| 548 | */ | 556 | if (ret < 0) |
| 549 | ret = gpiod_request_commit(desc, "shared"); | 557 | return ret; |
| 550 | if (ret) | 558 | |
| 551 | return ret; | 559 | entry->offset = ret; |
| 560 | } | ||
| 561 | #endif /* CONFIG_OF */ | ||
| 562 | |||
| 563 | desc = &gdev->descs[entry->offset]; | ||
| 564 | |||
| 565 | __set_bit(GPIOD_FLAG_SHARED, &desc->flags); | ||
| 566 | /* | ||
| 567 | * Shared GPIOs are not requested via the normal path. Make | ||
| 568 | * them inaccessible to anyone even before we register the | ||
| 569 | * chip. | ||
| 570 | */ | ||
| 571 | ret = gpiod_request_commit(desc, "shared"); | ||
| 572 | if (ret) | ||
| 573 | return ret; | ||
| 552 | 574 | ||
| 553 | pr_debug("GPIO %u owned by %s is shared by multiple consumers\n", | 575 | pr_debug("GPIO %u owned by %s is shared by multiple consumers\n", |
| 554 | entry->offset, gpio_device_get_label(gdev)); | 576 | entry->offset, gpio_device_get_label(gdev)); |
| 577 | } | ||
| 555 | 578 | ||
| 556 | list_for_each_entry(ref, &entry->refs, list) { | 579 | list_for_each_entry(ref, &entry->refs, list) { |
| 557 | pr_debug("Setting up a shared GPIO entry for %s (con_id: '%s')\n", | 580 | pr_debug("Setting up a shared GPIO entry for %s (con_id: '%s')\n", |
| @@ -575,6 +598,8 @@ void gpio_device_teardown_shared(struct gpio_device *gdev) · 17 unmodified lines | |||
| 575 | struct gpio_shared_ref *ref; | 598 | struct gpio_shared_ref *ref; |
| 576 | 599 | ||
| 577 | list_for_each_entry(entry, &gpio_shared_list, list) { | 600 | list_for_each_entry(entry, &gpio_shared_list, list) { |
| 601 | guard(mutex)(&entry->lock); | ||
| 602 | |||
| 578 | if (!device_match_fwnode(&gdev->dev, entry->fwnode)) | 603 | if (!device_match_fwnode(&gdev->dev, entry->fwnode)) |
| 579 | continue; | 604 | continue; |
| 580 | 605 | ||
| @@ -11,17 +11,19 @@ · 10 unmodified lines | |||
| 11 | struct gpio_device; | 11 | struct gpio_device; |
| 12 | struct gpio_desc; | 12 | struct gpio_desc; |
| 13 | struct device; | 13 | struct device; |
| 14 | struct fwnode_handle; | ||
| 14 | 15 | ||
| 15 | #if IS_ENABLED(CONFIG_GPIO_SHARED) | 16 | #if IS_ENABLED(CONFIG_GPIO_SHARED) |
| 16 | 17 | ||
| 17 | int gpio_device_setup_shared(struct gpio_device *gdev); | 18 | int gpiochip_setup_shared(struct gpio_chip *gc); |
| 18 | void gpio_device_teardown_shared(struct gpio_device *gdev); | 19 | void gpio_device_teardown_shared(struct gpio_device *gdev); |
| 19 | int gpio_shared_add_proxy_lookup(struct device *consumer, const char *con_id, | 20 | int gpio_shared_add_proxy_lookup(struct device *consumer, |
| 20 | unsigned long lflags); | 21 | struct fwnode_handle *fwnode, |
| 22 | const char *con_id, unsigned long lflags); | ||
| 21 | 23 | ||
| 22 | #else | 24 | #else |
| 23 | 25 | ||
| 24 | static inline int gpio_device_setup_shared(struct gpio_device *gdev) | 26 | static inline int gpiochip_setup_shared(struct gpio_chip *gc) |
| 25 | { | 27 | { |
| 26 | return 0; | 28 | return 0; |
| 27 | } | 29 | } |
| @@ -29,6 +31,7 @@ static inline int gpio_device_setup_shared(struct gpio_device *gdev) · 1 unmodified line | |||
| 29 | static inline void gpio_device_teardown_shared(struct gpio_device *gdev) { } | 31 | static inline void gpio_device_teardown_shared(struct gpio_device *gdev) { } |
| 30 | 32 | ||
| 31 | static inline int gpio_shared_add_proxy_lookup(struct device *consumer, | 33 | static inline int gpio_shared_add_proxy_lookup(struct device *consumer, |
| 34 | struct fwnode_handle *fwnode, | ||
| 32 | const char *con_id, | 35 | const char *con_id, |
| 33 | unsigned long lflags) | 36 | unsigned long lflags) |
| 34 | { | 37 | { |
| @@ -892,13 +892,15 @@ static const struct device_type gpio_dev_type = { · 891 unmodified lines | |||
| 892 | #define gcdev_unregister(gdev) device_del(&(gdev)->dev) | 892 | #define gcdev_unregister(gdev) device_del(&(gdev)->dev) |
| 893 | #endif | 893 | #endif |
| 894 | 894 | ||
| 895 | /* | ||
| 896 | * An initial reference count has been held in gpiochip_add_data_with_key(). | ||
| 897 | * The caller should drop the reference via gpio_device_put() on errors. | ||
| 898 | */ | ||
| 895 | static int gpiochip_setup_dev(struct gpio_device *gdev) | 899 | static int gpiochip_setup_dev(struct gpio_device *gdev) |
| 896 | { | 900 | { |
| 897 | struct fwnode_handle *fwnode = dev_fwnode(&gdev->dev); | 901 | struct fwnode_handle *fwnode = dev_fwnode(&gdev->dev); |
| 898 | int ret; | 902 | int ret; |
| 899 | 903 | ||
| 900 | device_initialize(&gdev->dev); | ||
| 901 | |||
| 902 | /* | 904 | /* |
| 903 | * If fwnode doesn't belong to another device, it's safe to clear its | 905 | * If fwnode doesn't belong to another device, it's safe to clear its |
| 904 | * initialized flag. | 906 | * initialized flag. |
| @@ -964,9 +966,11 @@ static void gpiochip_setup_devs(void) · 59 unmodified lines | |||
| 964 | list_for_each_entry_srcu(gdev, &gpio_devices, list, | 966 | list_for_each_entry_srcu(gdev, &gpio_devices, list, |
| 965 | srcu_read_lock_held(&gpio_devices_srcu)) { | 967 | srcu_read_lock_held(&gpio_devices_srcu)) { |
| 966 | ret = gpiochip_setup_dev(gdev); | 968 | ret = gpiochip_setup_dev(gdev); |
| 967 | if (ret) | 969 | if (ret) { |
| 970 | gpio_device_put(gdev); | ||
| 968 | dev_err(&gdev->dev, | 971 | dev_err(&gdev->dev, |
| 969 | "Failed to initialize gpio device (%d)\n", ret); | 972 | "Failed to initialize gpio device (%d)\n", ret); |
| 973 | } | ||
| 970 | } | 974 | } |
| 971 | } | 975 | } |
| 972 | 976 | ||
| @@ -1047,71 +1051,72 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 74 unmodified lines | |||
| 1047 | int base = 0; | 1051 | int base = 0; |
| 1048 | int ret; | 1052 | int ret; |
| 1049 | 1053 | ||
| 1050 | /* | ||
| 1051 | * First: allocate and populate the internal stat container, and | ||
| 1052 | * set up the struct device. | ||
| 1053 | */ | ||
| 1054 | gdev = kzalloc(sizeof(*gdev), GFP_KERNEL); | 1054 | gdev = kzalloc(sizeof(*gdev), GFP_KERNEL); |
| 1055 | if (!gdev) | 1055 | if (!gdev) |
| 1056 | return -ENOMEM; | 1056 | return -ENOMEM; |
| 1057 | |||
| 1058 | gdev->dev.type = &gpio_dev_type; | ||
| 1059 | gdev->dev.bus = &gpio_bus_type; | ||
| 1060 | gdev->dev.parent = gc->parent; | ||
| 1061 | rcu_assign_pointer(gdev->chip, gc); | ||
| 1062 | |||
| 1063 | gc->gpiodev = gdev; | 1057 | gc->gpiodev = gdev; |
| 1064 | gpiochip_set_data(gc, data); | 1058 | gpiochip_set_data(gc, data); |
| 1065 | 1059 | ||
| 1066 | device_set_node(&gdev->dev, gpiochip_choose_fwnode(gc)); | ||
| 1067 | |||
| 1068 | ret = ida_alloc(&gpio_ida, GFP_KERNEL); | 1060 | ret = ida_alloc(&gpio_ida, GFP_KERNEL); |
| 1069 | if (ret < 0) | 1061 | if (ret < 0) |
| 1070 | goto err_free_gdev; | 1062 | goto err_free_gdev; |
| 1071 | gdev->id = ret; | 1063 | gdev->id = ret; |
| 1072 | 1064 | ||
| 1073 | ret = dev_set_name(&gdev->dev, GPIOCHIP_NAME "%d", gdev->id); | 1065 | ret = init_srcu_struct(&gdev->srcu); |
| 1074 | if (ret) | 1066 | if (ret) |
| 1075 | goto err_free_ida; | 1067 | goto err_free_ida; |
| 1068 | rcu_assign_pointer(gdev->chip, gc); | ||
| 1076 | 1069 | ||
| 1077 | if (gc->parent && gc->parent->driver) | 1070 | ret = init_srcu_struct(&gdev->desc_srcu); |
| 1078 | gdev->owner = gc->parent->driver->owner; | 1071 | if (ret) |
| 1079 | else if (gc->owner) | 1072 | goto err_cleanup_gdev_srcu; |
| 1080 | /* TODO: remove chip->owner */ | 1073 | |
| 1081 | gdev->owner = gc->owner; | 1074 | ret = dev_set_name(&gdev->dev, GPIOCHIP_NAME "%d", gdev->id); |
| 1082 | else | 1075 | if (ret) |
| 1083 | gdev->owner = THIS_MODULE; | 1076 | goto err_cleanup_desc_srcu; |
| 1077 | |||
| 1078 | device_initialize(&gdev->dev); | ||
| 1079 | /* | ||
| 1080 | * After this point any allocated resources to `gdev` will be | ||
| 1081 | * free():ed by gpiodev_release(). If you add new resources | ||
| 1082 | * then make sure they get free():ed there. | ||
| 1083 | */ | ||
| 1084 | gdev->dev.type = &gpio_dev_type; | ||
| 1085 | gdev->dev.bus = &gpio_bus_type; | ||
| 1086 | gdev->dev.parent = gc->parent; | ||
| 1087 | device_set_node(&gdev->dev, gpiochip_choose_fwnode(gc)); | ||
| 1084 | 1088 | ||
| 1085 | ret = gpiochip_get_ngpios(gc, &gdev->dev); | 1089 | ret = gpiochip_get_ngpios(gc, &gdev->dev); |
| 1086 | if (ret) | 1090 | if (ret) |
| 1087 | goto err_free_dev_name; | 1091 | goto err_put_device; |
| 1092 | gdev->ngpio = gc->ngpio; | ||
| 1088 | 1093 | ||
| 1089 | gdev->descs = kcalloc(gc->ngpio, sizeof(*gdev->descs), GFP_KERNEL); | 1094 | gdev->descs = kcalloc(gc->ngpio, sizeof(*gdev->descs), GFP_KERNEL); |
| 1090 | if (!gdev->descs) { | 1095 | if (!gdev->descs) { |
| 1091 | ret = -ENOMEM; | 1096 | ret = -ENOMEM; |
| 1092 | goto err_free_dev_name; | 1097 | goto err_put_device; |
| 1093 | } | 1098 | } |
| 1094 | 1099 | ||
| 1095 | gdev->label = kstrdup_const(gc->label ?: "unknown", GFP_KERNEL); | 1100 | gdev->label = kstrdup_const(gc->label ?: "unknown", GFP_KERNEL); |
| 1096 | if (!gdev->label) { | 1101 | if (!gdev->label) { |
| 1097 | ret = -ENOMEM; | 1102 | ret = -ENOMEM; |
| 1098 | goto err_free_descs; | 1103 | goto err_put_device; |
| 1099 | } | 1104 | } |
| 1100 | 1105 | ||
| 1101 | gdev->ngpio = gc->ngpio; | ||
| 1102 | gdev->can_sleep = gc->can_sleep; | 1106 | gdev->can_sleep = gc->can_sleep; |
| 1103 | |||
| 1104 | rwlock_init(&gdev->line_state_lock); | 1107 | rwlock_init(&gdev->line_state_lock); |
| 1105 | RAW_INIT_NOTIFIER_HEAD(&gdev->line_state_notifier); | 1108 | RAW_INIT_NOTIFIER_HEAD(&gdev->line_state_notifier); |
| 1106 | BLOCKING_INIT_NOTIFIER_HEAD(&gdev->device_notifier); | 1109 | BLOCKING_INIT_NOTIFIER_HEAD(&gdev->device_notifier); |
| 1107 | 1110 | #ifdef CONFIG_PINCTRL | |
| 1108 | ret = init_srcu_struct(&gdev->srcu); | 1111 | INIT_LIST_HEAD(&gdev->pin_ranges); |
| 1109 | if (ret) | 1112 | #endif |
| 1110 | goto err_free_label; | 1113 | if (gc->parent && gc->parent->driver) |
| 1111 | 1114 | gdev->owner = gc->parent->driver->owner; | |
| 1112 | ret = init_srcu_struct(&gdev->desc_srcu); | 1115 | else if (gc->owner) |
| 1113 | if (ret) | 1116 | /* TODO: remove chip->owner */ |
| 1114 | goto err_cleanup_gdev_srcu; | 1117 | gdev->owner = gc->owner; |
| 1118 | else | ||
| 1119 | gdev->owner = THIS_MODULE; | ||
| 1115 | 1120 | ||
| 1116 | scoped_guard(mutex, &gpio_devices_lock) { | 1121 | scoped_guard(mutex, &gpio_devices_lock) { |
| 1117 | /* | 1122 | /* |
| @@ -1127,7 +1132,7 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 9 unmodified lines | |||
| 1127 | if (base < 0) { | 1132 | if (base < 0) { |
| 1128 | ret = base; | 1133 | ret = base; |
| 1129 | base = 0; | 1134 | base = 0; |
| 1130 | goto err_cleanup_desc_srcu; | 1135 | goto err_put_device; |
| 1131 | } | 1136 | } |
| 1132 | 1137 | ||
| 1133 | /* | 1138 | /* |
| @@ -1147,14 +1152,10 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 13 unmodified lines | |||
| 1147 | ret = gpiodev_add_to_list_unlocked(gdev); | 1152 | ret = gpiodev_add_to_list_unlocked(gdev); |
| 1148 | if (ret) { | 1153 | if (ret) { |
| 1149 | gpiochip_err(gc, "GPIO integer space overlap, cannot add chip\n"); | 1154 | gpiochip_err(gc, "GPIO integer space overlap, cannot add chip\n"); |
| 1150 | goto err_cleanup_desc_srcu; | 1155 | goto err_put_device; |
| 1151 | } | 1156 | } |
| 1152 | } | 1157 | } |
| 1153 | 1158 | ||
| 1154 | #ifdef CONFIG_PINCTRL | ||
| 1155 | INIT_LIST_HEAD(&gdev->pin_ranges); | ||
| 1156 | #endif | ||
| 1157 | |||
| 1158 | if (gc->names) | 1159 | if (gc->names) |
| 1159 | gpiochip_set_desc_names(gc); | 1160 | gpiochip_set_desc_names(gc); |
| 1160 | 1161 | ||
| @@ -1210,7 +1211,7 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data, · 49 unmodified lines | |||
| 1210 | if (ret) | 1211 | if (ret) |
| 1211 | goto err_remove_irqchip_mask; | 1212 | goto err_remove_irqchip_mask; |
| 1212 | 1213 | ||
| 1213 | ret = gpio_device_setup_shared(gdev); | 1214 | ret = gpiochip_setup_shared(gc); |
| 1214 | if (ret) | 1215 | if (ret) |
| 1215 | goto err_remove_irqchip; | 1216 | goto err_remove_irqchip; |
| 1216 | 1217 | ||
| @@ -1248,25 +1249,19 @@ err_remove_from_list: · 31 unmodified lines | |||
| 1248 | scoped_guard(mutex, &gpio_devices_lock) | 1249 | scoped_guard(mutex, &gpio_devices_lock) |
| 1249 | list_del_rcu(&gdev->list); | 1250 | list_del_rcu(&gdev->list); |
| 1250 | synchronize_srcu(&gpio_devices_srcu); | 1251 | synchronize_srcu(&gpio_devices_srcu); |
| 1251 | if (gdev->dev.release) { | 1252 | err_put_device: |
| 1252 | /* release() has been registered by gpiochip_setup_dev() */ | 1253 | gpio_device_put(gdev); |
| 1253 | gpio_device_put(gdev); | 1254 | goto err_print_message; |
| 1254 | goto err_print_message; | 1255 | |
| 1255 | } | ||
| 1256 | err_cleanup_desc_srcu: | 1256 | err_cleanup_desc_srcu: |
| 1257 | cleanup_srcu_struct(&gdev->desc_srcu); | 1257 | cleanup_srcu_struct(&gdev->desc_srcu); |
| 1258 | err_cleanup_gdev_srcu: | 1258 | err_cleanup_gdev_srcu: |
| 1259 | cleanup_srcu_struct(&gdev->srcu); | 1259 | cleanup_srcu_struct(&gdev->srcu); |
| 1260 | err_free_label: | ||
| 1261 | kfree_const(gdev->label); | ||
| 1262 | err_free_descs: | ||
| 1263 | kfree(gdev->descs); | ||
| 1264 | err_free_dev_name: | ||
| 1265 | kfree(dev_name(&gdev->dev)); | ||
| 1266 | err_free_ida: | 1260 | err_free_ida: |
| 1267 | ida_free(&gpio_ida, gdev->id); | 1261 | ida_free(&gpio_ida, gdev->id); |
| 1268 | err_free_gdev: | 1262 | err_free_gdev: |
| 1269 | kfree(gdev); | 1263 | kfree(gdev); |
| 1264 | |||
| 1270 | err_print_message: | 1265 | err_print_message: |
| 1271 | /* failures here can mean systems won't boot... */ | 1266 | /* failures here can mean systems won't boot... */ |
| 1272 | if (ret != -EPROBE_DEFER) { | 1267 | if (ret != -EPROBE_DEFER) { |
| @@ -2465,8 +2460,10 @@ int gpiod_request_commit(struct gpio_desc *desc, const char *label) · 1192 unmodified lines | |||
| 2465 | return -EBUSY; | 2460 | return -EBUSY; |
| 2466 | 2461 | ||
| 2467 | offset = gpiod_hwgpio(desc); | 2462 | offset = gpiod_hwgpio(desc); |
| 2468 | if (!gpiochip_line_is_valid(guard.gc, offset)) | 2463 | if (!gpiochip_line_is_valid(guard.gc, offset)) { |
| 2469 | return -EINVAL; | 2464 | ret = -EINVAL; |
| 2465 | goto out_clear_bit; | ||
| 2466 | } | ||
| 2470 | 2467 | ||
| 2471 | /* NOTE: gpio_request() can be called in early boot, | 2468 | /* NOTE: gpio_request() can be called in early boot, |
| 2472 | * before IRQs are enabled, for non-sleeping (SOC) GPIOs. | 2469 | * before IRQs are enabled, for non-sleeping (SOC) GPIOs. |
| @@ -4717,8 +4714,8 @@ struct gpio_desc *gpiod_find_and_request(struct device *consumer, · 2244 unmodified lines | |||
| 4717 | * lookup table for the proxy device as previously | 4714 | * lookup table for the proxy device as previously |
| 4718 | * we only knew the consumer's fwnode. | 4715 | * we only knew the consumer's fwnode. |
| 4719 | */ | 4716 | */ |
| 4720 | ret = gpio_shared_add_proxy_lookup(consumer, con_id, | 4717 | ret = gpio_shared_add_proxy_lookup(consumer, fwnode, |
| 4721 | lookupflags); | 4718 | con_id, lookupflags); |
| 4722 | if (ret) | 4719 | if (ret) |
| 4723 | return ERR_PTR(ret); | 4720 | return ERR_PTR(ret); |
| 4724 | 4721 | ||
| @@ -268,6 +268,13 @@ config DRM_GEM_SHMEM_HELPER · 267 unmodified lines | |||
| 268 | help | 268 | help |
| 269 | Choose this if you need the GEM shmem helper functions | 269 | Choose this if you need the GEM shmem helper functions |
| 270 | 270 | ||
| 271 | config RUST_DRM_GEM_SHMEM_HELPER | ||
| 272 | bool | ||
| 273 | depends on DRM && MMU | ||
| 274 | select DRM_GEM_SHMEM_HELPER | ||
| 275 | help | ||
| 276 | Choose this if you need the GEM shmem helper functions In Rust | ||
| 277 | |||
| 271 | config DRM_SUBALLOC_HELPER | 278 | config DRM_SUBALLOC_HELPER |
| 272 | tristate | 279 | tristate |
| 273 | depends on DRM | 280 | depends on DRM |
| @@ -85,7 +85,6 @@ config DRM_KUNIT_TEST · 84 unmodified lines | |||
| 85 | 85 | ||
| 86 | config DRM_TTM_KUNIT_TEST | 86 | config DRM_TTM_KUNIT_TEST |
| 87 | tristate "KUnit tests for TTM" if !KUNIT_ALL_TESTS | 87 | tristate "KUnit tests for TTM" if !KUNIT_ALL_TESTS |
| 88 | default n | ||
| 89 | depends on DRM && KUNIT && MMU && (UML || COMPILE_TEST) | 88 | depends on DRM && KUNIT && MMU && (UML || COMPILE_TEST) |
| 90 | select DRM_TTM | 89 | select DRM_TTM |
| 91 | select DRM_BUDDY | 90 | select DRM_BUDDY |
| @@ -213,6 +213,7 @@ amdgpu-y += \ · 212 unmodified lines | |||
| 213 | vcn_v4_0_5.o \ | 213 | vcn_v4_0_5.o \ |
| 214 | vcn_v5_0_0.o \ | 214 | vcn_v5_0_0.o \ |
| 215 | vcn_v5_0_1.o \ | 215 | vcn_v5_0_1.o \ |
| 216 | vcn_v5_0_2.o \ | ||
| 216 | amdgpu_jpeg.o \ | 217 | amdgpu_jpeg.o \ |
| 217 | jpeg_v1_0.o \ | 218 | jpeg_v1_0.o \ |
| 218 | jpeg_v2_0.o \ | 219 | jpeg_v2_0.o \ |
| @@ -223,6 +224,7 @@ amdgpu-y += \ · 4 unmodified lines | |||
| 223 | jpeg_v4_0_5.o \ | 224 | jpeg_v4_0_5.o \ |
| 224 | jpeg_v5_0_0.o \ | 225 | jpeg_v5_0_0.o \ |
| 225 | jpeg_v5_0_1.o \ | 226 | jpeg_v5_0_1.o \ |
| 227 | jpeg_v5_0_2.o \ | ||
| 226 | jpeg_v5_3_0.o | 228 | jpeg_v5_3_0.o |
| 227 | 229 | ||
| 228 | # add VPE block | 230 | # add VPE block |
| @@ -327,6 +327,7 @@ struct kfd_vm_fault_info; · 326 unmodified lines | |||
| 327 | struct amdgpu_hive_info; | 327 | struct amdgpu_hive_info; |
| 328 | struct amdgpu_reset_context; | 328 | struct amdgpu_reset_context; |
| 329 | struct amdgpu_reset_control; | 329 | struct amdgpu_reset_control; |
| 330 | struct amdgpu_coredump_info; | ||
| 330 | 331 | ||
| 331 | enum amdgpu_cp_irq { | 332 | enum amdgpu_cp_irq { |
| 332 | AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP = 0, | 333 | AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP = 0, |
| @@ -689,6 +690,7 @@ enum amdgpu_uid_type { · 356 unmodified lines | |||
| 689 | AMDGPU_UID_TYPE_XCD, | 690 | AMDGPU_UID_TYPE_XCD, |
| 690 | AMDGPU_UID_TYPE_AID, | 691 | AMDGPU_UID_TYPE_AID, |
| 691 | AMDGPU_UID_TYPE_SOC, | 692 | AMDGPU_UID_TYPE_SOC, |
| 693 | AMDGPU_UID_TYPE_MID, | ||
| 692 | AMDGPU_UID_TYPE_MAX | 694 | AMDGPU_UID_TYPE_MAX |
| 693 | }; | 695 | }; |
| 694 | 696 | ||
| @@ -1147,6 +1149,11 @@ struct amdgpu_device { · 452 unmodified lines | |||
| 1147 | 1149 | ||
| 1148 | struct amdgpu_reset_domain *reset_domain; | 1150 | struct amdgpu_reset_domain *reset_domain; |
| 1149 | 1151 | ||
| 1152 | #ifdef CONFIG_DEV_COREDUMP | ||
| 1153 | struct amdgpu_coredump_info *coredump; | ||
| 1154 | struct work_struct coredump_work; | ||
| 1155 | #endif | ||
| 1156 | |||
| 1150 | struct mutex benchmark_mutex; | 1157 | struct mutex benchmark_mutex; |
| 1151 | 1158 | ||
| 1152 | bool scpm_enabled; | 1159 | bool scpm_enabled; |
| @@ -692,9 +692,9 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device *adev, · 691 unmodified lines | |||
| 692 | goto err_ib_sched; | 692 | goto err_ib_sched; |
| 693 | } | 693 | } |
| 694 | 694 | ||
| 695 | /* Drop the initial kref_init count (see drm_sched_main as example) */ | ||
| 696 | dma_fence_put(f); | ||
| 697 | ret = dma_fence_wait(f, false); | 695 | ret = dma_fence_wait(f, false); |
| 696 | /* Drop the returned fence reference after the wait completes */ | ||
| 697 | dma_fence_put(f); | ||
| 698 | 698 | ||
| 699 | err_ib_sched: | 699 | err_ib_sched: |
| 700 | amdgpu_job_free(job); | 700 | amdgpu_job_free(job); |
| @@ -330,7 +330,7 @@ static uint32_t kgd_gfx_v12_1_set_address_watch(struct amdgpu_device *adev, · 329 unmodified lines | |||
| 330 | 330 | ||
| 331 | watch_address_cntl = 0; | 331 | watch_address_cntl = 0; |
| 332 | watch_address_low = lower_32_bits(watch_address); | 332 | watch_address_low = lower_32_bits(watch_address); |
| 333 | watch_address_high = upper_32_bits(watch_address) & 0xffff; | 333 | watch_address_high = upper_32_bits(watch_address) & 0x1ffffff; |
| 334 | 334 | ||
| 335 | watch_address_cntl = REG_SET_FIELD(watch_address_cntl, | 335 | watch_address_cntl = REG_SET_FIELD(watch_address_cntl, |
| 336 | TCP_WATCH0_CNTL, | 336 | TCP_WATCH0_CNTL, |
| @@ -373,249 +373,280 @@ int amdgpu_atomfirmware_get_uma_carveout_info(struct amdgpu_device *adev, · 372 unmodified lines | |||
| 373 | return -ENODEV; | 373 | return -ENODEV; |
| 374 | } | 374 | } |
| 375 | 375 | ||
| 376 | int | 376 | int amdgpu_atomfirmware_get_integrated_system_info(struct amdgpu_device *adev, |
| 377 | amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, | ||
| 378 | int *vram_width, int *vram_type, | 377 | int *vram_width, int *vram_type, |
| 379 | int *vram_vendor) | 378 | int *vram_vendor) |
| 380 | { | 379 | { |
| 381 | struct amdgpu_mode_info *mode_info = &adev->mode_info; | 380 | struct amdgpu_mode_info *mode_info = &adev->mode_info; |
| 382 | int index, i = 0; | 381 | int index; |
| 383 | u16 data_offset, size; | 382 | u16 data_offset, size; |
| 384 | union igp_info *igp_info; | 383 | union igp_info *igp_info; |
| 385 | union vram_info *vram_info; | ||
| 386 | union umc_info *umc_info; | ||
| 387 | union vram_module *vram_module; | ||
| 388 | u8 frev, crev; | 384 | u8 frev, crev; |
| 389 | u8 mem_type; | 385 | u8 mem_type; |
| 390 | u8 mem_vendor; | ||
| 391 | u32 mem_channel_number; | 386 | u32 mem_channel_number; |
| 392 | u32 mem_channel_width; | 387 | u32 mem_channel_width; |
| 393 | u32 module_id; | ||
| 394 | 388 | ||
| 395 | if (adev->flags & AMD_IS_APU) | 389 | index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, |
| 396 | index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, | ||
| 397 | integratedsysteminfo); | 390 | integratedsysteminfo); |
| 398 | else { | 391 | if (amdgpu_atom_parse_data_header(mode_info->atom_context, |
| 399 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | 392 | index, &size, |
| 400 | case IP_VERSION(12, 0, 0): | 393 | &frev, &crev, &data_offset)) { |
| 401 | case IP_VERSION(12, 0, 1): | 394 | igp_info = (union igp_info *) |
| 402 | index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, umc_info); | 395 | (mode_info->atom_context->bios + data_offset); |
| 396 | switch (frev) { | ||
| 397 | case 1: | ||
| 398 | switch (crev) { | ||
| 399 | case 11: | ||
| 400 | case 12: | ||
| 401 | mem_channel_number = igp_info->v11.umachannelnumber; | ||
| 402 | if (!mem_channel_number) | ||
| 403 | mem_channel_number = 1; | ||
| 404 | mem_type = igp_info->v11.memorytype; | ||
| 405 | if (mem_type == LpDdr5MemType) | ||
| 406 | mem_channel_width = 32; | ||
| 407 | else | ||
| 408 | mem_channel_width = 64; | ||
| 409 | if (vram_width) | ||
| 410 | *vram_width = mem_channel_number * mem_channel_width; | ||
| 411 | if (vram_type) | ||
| 412 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 413 | break; | ||
| 414 | default: | ||
| 415 | return -EINVAL; | ||
| 416 | } | ||
| 417 | break; | ||
| 418 | case 2: | ||
| 419 | switch (crev) { | ||
| 420 | case 1: | ||
| 421 | case 2: | ||
| 422 | mem_channel_number = igp_info->v21.umachannelnumber; | ||
| 423 | if (!mem_channel_number) | ||
| 424 | mem_channel_number = 1; | ||
| 425 | mem_type = igp_info->v21.memorytype; | ||
| 426 | if (mem_type == LpDdr5MemType) | ||
| 427 | mem_channel_width = 32; | ||
| 428 | else | ||
| 429 | mem_channel_width = 64; | ||
| 430 | if (vram_width) | ||
| 431 | *vram_width = mem_channel_number * mem_channel_width; | ||
| 432 | if (vram_type) | ||
| 433 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 434 | break; | ||
| 435 | case 3: | ||
| 436 | mem_channel_number = igp_info->v23.umachannelnumber; | ||
| 437 | if (!mem_channel_number) | ||
| 438 | mem_channel_number = 1; | ||
| 439 | mem_type = igp_info->v23.memorytype; | ||
| 440 | if (mem_type == LpDdr5MemType) | ||
| 441 | mem_channel_width = 32; | ||
| 442 | else | ||
| 443 | mem_channel_width = 64; | ||
| 444 | if (vram_width) | ||
| 445 | *vram_width = mem_channel_number * mem_channel_width; | ||
| 446 | if (vram_type) | ||
| 447 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 448 | break; | ||
| 449 | default: | ||
| 450 | return -EINVAL; | ||
| 451 | } | ||
| 403 | break; | 452 | break; |
| 404 | default: | 453 | default: |
| 405 | index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, vram_info); | 454 | return -EINVAL; |
| 406 | } | 455 | } |
| 456 | } else { | ||
| 457 | return -EINVAL; | ||
| 407 | } | 458 | } |
| 459 | return 0; | ||
| 460 | } | ||
| 461 | |||
| 462 | int amdgpu_atomfirmware_get_umc_info(struct amdgpu_device *adev, | ||
| 463 | int *vram_width, int *vram_type, | ||
| 464 | int *vram_vendor) | ||
| 465 | { | ||
| 466 | struct amdgpu_mode_info *mode_info = &adev->mode_info; | ||
| 467 | int index; | ||
| 468 | u16 data_offset, size; | ||
| 469 | union umc_info *umc_info; | ||
| 470 | u8 frev, crev; | ||
| 471 | u8 mem_type; | ||
| 472 | u8 mem_vendor; | ||
| 473 | u32 mem_channel_number; | ||
| 474 | u32 mem_channel_width; | ||
| 475 | |||
| 476 | index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, umc_info); | ||
| 477 | |||
| 408 | if (amdgpu_atom_parse_data_header(mode_info->atom_context, | 478 | if (amdgpu_atom_parse_data_header(mode_info->atom_context, |
| 409 | index, &size, | 479 | index, &size, |
| 410 | &frev, &crev, &data_offset)) { | 480 | &frev, &crev, &data_offset)) { |
| 411 | if (adev->flags & AMD_IS_APU) { | 481 | umc_info = (union umc_info *)(mode_info->atom_context->bios + data_offset); |
| 412 | igp_info = (union igp_info *) | 482 | |
| 413 | (mode_info->atom_context->bios + data_offset); | 483 | if (frev == 4) { |
| 414 | switch (frev) { | 484 | switch (crev) { |
| 415 | case 1: | 485 | case 0: |
| 416 | switch (crev) { | 486 | mem_channel_number = le32_to_cpu(umc_info->v40.channel_num); |
| 417 | case 11: | 487 | mem_type = le32_to_cpu(umc_info->v40.vram_type); |
| 418 | case 12: | 488 | mem_channel_width = le32_to_cpu(umc_info->v40.channel_width); |
| 419 | mem_channel_number = igp_info->v11.umachannelnumber; | 489 | mem_vendor = RREG32(adev->bios_scratch_reg_offset + 4) & 0xF; |
| 420 | if (!mem_channel_number) | 490 | if (vram_vendor) |
| 421 | mem_channel_number = 1; | 491 | *vram_vendor = mem_vendor; |
| 422 | mem_type = igp_info->v11.memorytype; | 492 | if (vram_type) |
| 423 | if (mem_type == LpDdr5MemType) | 493 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); |
| 424 | mem_channel_width = 32; | 494 | if (vram_width) |
| 425 | else | 495 | *vram_width = mem_channel_number * (1 << mem_channel_width); |
| 426 | mem_channel_width = 64; | ||
| 427 | if (vram_width) | ||
| 428 | *vram_width = mem_channel_number * mem_channel_width; | ||
| 429 | if (vram_type) | ||
| 430 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 431 | break; | ||
| 432 | default: | ||
| 433 | return -EINVAL; | ||
| 434 | } | ||
| 435 | break; | ||
| 436 | case 2: | ||
| 437 | switch (crev) { | ||
| 438 | case 1: | ||
| 439 | case 2: | ||
| 440 | mem_channel_number = igp_info->v21.umachannelnumber; | ||
| 441 | if (!mem_channel_number) | ||
| 442 | mem_channel_number = 1; | ||
| 443 | mem_type = igp_info->v21.memorytype; | ||
| 444 | if (mem_type == LpDdr5MemType) | ||
| 445 | mem_channel_width = 32; | ||
| 446 | else | ||
| 447 | mem_channel_width = 64; | ||
| 448 | if (vram_width) | ||
| 449 | *vram_width = mem_channel_number * mem_channel_width; | ||
| 450 | if (vram_type) | ||
| 451 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 452 | break; | ||
| 453 | case 3: | ||
| 454 | mem_channel_number = igp_info->v23.umachannelnumber; | ||
| 455 | if (!mem_channel_number) | ||
| 456 | mem_channel_number = 1; | ||
| 457 | mem_type = igp_info->v23.memorytype; | ||
| 458 | if (mem_type == LpDdr5MemType) | ||
| 459 | mem_channel_width = 32; | ||
| 460 | else | ||
| 461 | mem_channel_width = 64; | ||
| 462 | if (vram_width) | ||
| 463 | *vram_width = mem_channel_number * mem_channel_width; | ||
| 464 | if (vram_type) | ||
| 465 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 466 | break; | ||
| 467 | default: | ||
| 468 | return -EINVAL; | ||
| 469 | } | ||
| 470 | break; | 496 | break; |
| 471 | default: | 497 | default: |
| 472 | return -EINVAL; | 498 | return -EINVAL; |
| 473 | } | 499 | } |
| 474 | } else { | 500 | } else { |
| 475 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | 501 | return -EINVAL; |
| 476 | case IP_VERSION(12, 0, 0): | 502 | } |
| 477 | case IP_VERSION(12, 0, 1): | 503 | } else { |
| 478 | umc_info = (union umc_info *)(mode_info->atom_context->bios + data_offset); | 504 | return -EINVAL; |
| 479 | 505 | } | |
| 480 | if (frev == 4) { | 506 | |
| 481 | switch (crev) { | 507 | return 0; |
| 482 | case 0: | 508 | } |
| 483 | mem_channel_number = le32_to_cpu(umc_info->v40.channel_num); | 509 | |
| 484 | mem_type = le32_to_cpu(umc_info->v40.vram_type); | 510 | int amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, |
| 485 | mem_channel_width = le32_to_cpu(umc_info->v40.channel_width); | 511 | int *vram_width, int *vram_type, |
| 486 | mem_vendor = RREG32(adev->bios_scratch_reg_offset + 4) & 0xF; | 512 | int *vram_vendor) |
| 487 | if (vram_vendor) | 513 | { |
| 488 | *vram_vendor = mem_vendor; | 514 | struct amdgpu_mode_info *mode_info = &adev->mode_info; |
| 489 | if (vram_type) | 515 | int index, i = 0; |
| 490 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | 516 | u16 data_offset, size; |
| 491 | if (vram_width) | 517 | union vram_info *vram_info; |
| 492 | *vram_width = mem_channel_number * (1 << mem_channel_width); | 518 | union vram_module *vram_module; |
| 493 | break; | 519 | u8 frev, crev; |
| 494 | default: | 520 | u8 mem_type; |
| 495 | return -EINVAL; | 521 | u8 mem_vendor; |
| 496 | } | 522 | u32 mem_channel_number; |
| 497 | } else | 523 | u32 mem_channel_width; |
| 498 | return -EINVAL; | 524 | u32 module_id; |
| 525 | |||
| 526 | index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, vram_info); | ||
| 527 | |||
| 528 | if (amdgpu_atom_parse_data_header(mode_info->atom_context, | ||
| 529 | index, &size, | ||
| 530 | &frev, &crev, &data_offset)) { | ||
| 531 | vram_info = (union vram_info *) | ||
| 532 | (mode_info->atom_context->bios + data_offset); | ||
| 533 | |||
| 534 | module_id = (RREG32(adev->bios_scratch_reg_offset + 4) & 0x00ff0000) >> 16; | ||
| 535 | if (frev == 3) { | ||
| 536 | switch (crev) { | ||
| 537 | /* v30 */ | ||
| 538 | case 0: | ||
| 539 | vram_module = (union vram_module *)vram_info->v30.vram_module; | ||
| 540 | mem_vendor = (vram_module->v30.dram_vendor_id) & 0xF; | ||
| 541 | if (vram_vendor) | ||
| 542 | *vram_vendor = mem_vendor; | ||
| 543 | mem_type = vram_info->v30.memory_type; | ||
| 544 | if (vram_type) | ||
| 545 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 546 | mem_channel_number = vram_info->v30.channel_num; | ||
| 547 | mem_channel_width = vram_info->v30.channel_width; | ||
| 548 | if (vram_width) | ||
| 549 | *vram_width = mem_channel_number * 16; | ||
| 499 | break; | 550 | break; |
| 500 | default: | 551 | default: |
| 501 | vram_info = (union vram_info *) | 552 | return -EINVAL; |
| 502 | (mode_info->atom_context->bios + data_offset); | 553 | } |
| 503 | 554 | } else if (frev == 2) { | |
| 504 | module_id = (RREG32(adev->bios_scratch_reg_offset + 4) & 0x00ff0000) >> 16; | 555 | switch (crev) { |
| 505 | if (frev == 3) { | 556 | /* v23 */ |
| 506 | switch (crev) { | 557 | case 3: |
| 507 | /* v30 */ | 558 | if (module_id > vram_info->v23.vram_module_num) |
| 508 | case 0: | 559 | module_id = 0; |
| 509 | vram_module = (union vram_module *)vram_info->v30.vram_module; | 560 | vram_module = (union vram_module *)vram_info->v23.vram_module; |
| 510 | mem_vendor = (vram_module->v30.dram_vendor_id) & 0xF; | 561 | while (i < module_id) { |
| 511 | if (vram_vendor) | 562 | vram_module = (union vram_module *) |
| 512 | *vram_vendor = mem_vendor; | 563 | ((u8 *)vram_module + vram_module->v9.vram_module_size); |
| 513 | mem_type = vram_info->v30.memory_type; | 564 | i++; |
| 514 | if (vram_type) | ||
| 515 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 516 | mem_channel_number = vram_info->v30.channel_num; | ||
| 517 | mem_channel_width = vram_info->v30.channel_width; | ||
| 518 | if (vram_width) | ||
| 519 | *vram_width = mem_channel_number * 16; | ||
| 520 | break; | ||
| 521 | default: | ||
| 522 | return -EINVAL; | ||
| 523 | } | ||
| 524 | } else if (frev == 2) { | ||
| 525 | switch (crev) { | ||
| 526 | /* v23 */ | ||
| 527 | case 3: | ||
| 528 | if (module_id > vram_info->v23.vram_module_num) | ||
| 529 | module_id = 0; | ||
| 530 | vram_module = (union vram_module *)vram_info->v23.vram_module; | ||
| 531 | while (i < module_id) { | ||
| 532 | vram_module = (union vram_module *) | ||
| 533 | ((u8 *)vram_module + vram_module->v9.vram_module_size); | ||
| 534 | i++; | ||
| 535 | } | ||
| 536 | mem_type = vram_module->v9.memory_type; | ||
| 537 | if (vram_type) | ||
| 538 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 539 | mem_channel_number = vram_module->v9.channel_num; | ||
| 540 | mem_channel_width = vram_module->v9.channel_width; | ||
| 541 | if (vram_width) | ||
| 542 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 543 | mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | ||
| 544 | if (vram_vendor) | ||
| 545 | *vram_vendor = mem_vendor; | ||
| 546 | break; | ||
| 547 | /* v24 */ | ||
| 548 | case 4: | ||
| 549 | if (module_id > vram_info->v24.vram_module_num) | ||
| 550 | module_id = 0; | ||
| 551 | vram_module = (union vram_module *)vram_info->v24.vram_module; | ||
| 552 | while (i < module_id) { | ||
| 553 | vram_module = (union vram_module *) | ||
| 554 | ((u8 *)vram_module + vram_module->v10.vram_module_size); | ||
| 555 | i++; | ||
| 556 | } | ||
| 557 | mem_type = vram_module->v10.memory_type; | ||
| 558 | if (vram_type) | ||
| 559 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 560 | mem_channel_number = vram_module->v10.channel_num; | ||
| 561 | mem_channel_width = vram_module->v10.channel_width; | ||
| 562 | if (vram_width) | ||
| 563 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 564 | mem_vendor = (vram_module->v10.vender_rev_id) & 0xF; | ||
| 565 | if (vram_vendor) | ||
| 566 | *vram_vendor = mem_vendor; | ||
| 567 | break; | ||
| 568 | /* v25 */ | ||
| 569 | case 5: | ||
| 570 | if (module_id > vram_info->v25.vram_module_num) | ||
| 571 | module_id = 0; | ||
| 572 | vram_module = (union vram_module *)vram_info->v25.vram_module; | ||
| 573 | while (i < module_id) { | ||
| 574 | vram_module = (union vram_module *) | ||
| 575 | ((u8 *)vram_module + vram_module->v11.vram_module_size); | ||
| 576 | i++; | ||
| 577 | } | ||
| 578 | mem_type = vram_module->v11.memory_type; | ||
| 579 | if (vram_type) | ||
| 580 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 581 | mem_channel_number = vram_module->v11.channel_num; | ||
| 582 | mem_channel_width = vram_module->v11.channel_width; | ||
| 583 | if (vram_width) | ||
| 584 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 585 | mem_vendor = (vram_module->v11.vender_rev_id) & 0xF; | ||
| 586 | if (vram_vendor) | ||
| 587 | *vram_vendor = mem_vendor; | ||
| 588 | break; | ||
| 589 | /* v26 */ | ||
| 590 | case 6: | ||
| 591 | if (module_id > vram_info->v26.vram_module_num) | ||
| 592 | module_id = 0; | ||
| 593 | vram_module = (union vram_module *)vram_info->v26.vram_module; | ||
| 594 | while (i < module_id) { | ||
| 595 | vram_module = (union vram_module *) | ||
| 596 | ((u8 *)vram_module + vram_module->v9.vram_module_size); | ||
| 597 | i++; | ||
| 598 | } | ||
| 599 | mem_type = vram_module->v9.memory_type; | ||
| 600 | if (vram_type) | ||
| 601 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 602 | mem_channel_number = vram_module->v9.channel_num; | ||
| 603 | mem_channel_width = vram_module->v9.channel_width; | ||
| 604 | if (vram_width) | ||
| 605 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 606 | mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | ||
| 607 | if (vram_vendor) | ||
| 608 | *vram_vendor = mem_vendor; | ||
| 609 | break; | ||
| 610 | default: | ||
| 611 | return -EINVAL; | ||
| 612 | } | ||
| 613 | } else { | ||
| 614 | /* invalid frev */ | ||
| 615 | return -EINVAL; | ||
| 616 | } | 565 | } |
| 566 | mem_type = vram_module->v9.memory_type; | ||
| 567 | if (vram_type) | ||
| 568 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 569 | mem_channel_number = vram_module->v9.channel_num; | ||
| 570 | mem_channel_width = vram_module->v9.channel_width; | ||
| 571 | if (vram_width) | ||
| 572 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 573 | mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | ||
| 574 | if (vram_vendor) | ||
| 575 | *vram_vendor = mem_vendor; | ||
| 576 | break; | ||
| 577 | /* v24 */ | ||
| 578 | case 4: | ||
| 579 | if (module_id > vram_info->v24.vram_module_num) | ||
| 580 | module_id = 0; | ||
| 581 | vram_module = (union vram_module *)vram_info->v24.vram_module; | ||
| 582 | while (i < module_id) { | ||
| 583 | vram_module = (union vram_module *) | ||
| 584 | ((u8 *)vram_module + vram_module->v10.vram_module_size); | ||
| 585 | i++; | ||
| 586 | } | ||
| 587 | mem_type = vram_module->v10.memory_type; | ||
| 588 | if (vram_type) | ||
| 589 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 590 | mem_channel_number = vram_module->v10.channel_num; | ||
| 591 | mem_channel_width = vram_module->v10.channel_width; | ||
| 592 | if (vram_width) | ||
| 593 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 594 | mem_vendor = (vram_module->v10.vender_rev_id) & 0xF; | ||
| 595 | if (vram_vendor) | ||
| 596 | *vram_vendor = mem_vendor; | ||
| 597 | break; | ||
| 598 | /* v25 */ | ||
| 599 | case 5: | ||
| 600 | if (module_id > vram_info->v25.vram_module_num) | ||
| 601 | module_id = 0; | ||
| 602 | vram_module = (union vram_module *)vram_info->v25.vram_module; | ||
| 603 | while (i < module_id) { | ||
| 604 | vram_module = (union vram_module *) | ||
| 605 | ((u8 *)vram_module + vram_module->v11.vram_module_size); | ||
| 606 | i++; | ||
| 607 | } | ||
| 608 | mem_type = vram_module->v11.memory_type; | ||
| 609 | if (vram_type) | ||
| 610 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 611 | mem_channel_number = vram_module->v11.channel_num; | ||
| 612 | mem_channel_width = vram_module->v11.channel_width; | ||
| 613 | if (vram_width) | ||
| 614 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 615 | mem_vendor = (vram_module->v11.vender_rev_id) & 0xF; | ||
| 616 | if (vram_vendor) | ||
| 617 | *vram_vendor = mem_vendor; | ||
| 618 | break; | ||
| 619 | /* v26 */ | ||
| 620 | case 6: | ||
| 621 | if (module_id > vram_info->v26.vram_module_num) | ||
| 622 | module_id = 0; | ||
| 623 | vram_module = (union vram_module *)vram_info->v26.vram_module; | ||
| 624 | while (i < module_id) { | ||
| 625 | vram_module = (union vram_module *) | ||
| 626 | ((u8 *)vram_module + vram_module->v9.vram_module_size); | ||
| 627 | i++; | ||
| 628 | } | ||
| 629 | mem_type = vram_module->v9.memory_type; | ||
| 630 | if (vram_type) | ||
| 631 | *vram_type = convert_atom_mem_type_to_vram_type(adev, mem_type); | ||
| 632 | mem_channel_number = vram_module->v9.channel_num; | ||
| 633 | mem_channel_width = vram_module->v9.channel_width; | ||
| 634 | if (vram_width) | ||
| 635 | *vram_width = mem_channel_number * (1 << mem_channel_width); | ||
| 636 | mem_vendor = (vram_module->v9.vender_rev_id) & 0xF; | ||
| 637 | if (vram_vendor) | ||
| 638 | *vram_vendor = mem_vendor; | ||
| 639 | break; | ||
| 640 | default: | ||
| 641 | return -EINVAL; | ||
| 617 | } | 642 | } |
| 643 | } else { | ||
| 644 | /* invalid frev */ | ||
| 645 | return -EINVAL; | ||
| 618 | } | 646 | } |
| 647 | |||
| 648 | } else { | ||
| 649 | return -EINVAL; | ||
| 619 | } | 650 | } |
| 620 | 651 | ||
| 621 | return 0; | 652 | return 0; |
| @@ -30,6 +30,10 @@ uint32_t amdgpu_atomfirmware_query_firmware_capability(struct amdgpu_device *ade · 29 unmodified lines | |||
| 30 | bool amdgpu_atomfirmware_gpu_virtualization_supported(struct amdgpu_device *adev); | 30 | bool amdgpu_atomfirmware_gpu_virtualization_supported(struct amdgpu_device *adev); |
| 31 | void amdgpu_atomfirmware_scratch_regs_init(struct amdgpu_device *adev); | 31 | void amdgpu_atomfirmware_scratch_regs_init(struct amdgpu_device *adev); |
| 32 | int amdgpu_atomfirmware_allocate_fb_scratch(struct amdgpu_device *adev); | 32 | int amdgpu_atomfirmware_allocate_fb_scratch(struct amdgpu_device *adev); |
| 33 | int amdgpu_atomfirmware_get_integrated_system_info(struct amdgpu_device *adev, | ||
| 34 | int *vram_width, int *vram_type, int *vram_vendor); | ||
| 35 | int amdgpu_atomfirmware_get_umc_info(struct amdgpu_device *adev, | ||
| 36 | int *vram_width, int *vram_type, int *vram_vendor); | ||
| 33 | int amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, | 37 | int amdgpu_atomfirmware_get_vram_info(struct amdgpu_device *adev, |
| 34 | int *vram_width, int *vram_type, int *vram_vendor); | 38 | int *vram_width, int *vram_type, int *vram_vendor); |
| 35 | int amdgpu_atomfirmware_get_uma_carveout_info(struct amdgpu_device *adev, | 39 | int amdgpu_atomfirmware_get_uma_carveout_info(struct amdgpu_device *adev, |
| @@ -36,6 +36,7 @@ · 35 unmodified lines | |||
| 36 | 36 | ||
| 37 | #define AMDGPU_BO_LIST_MAX_PRIORITY 32u | 37 | #define AMDGPU_BO_LIST_MAX_PRIORITY 32u |
| 38 | #define AMDGPU_BO_LIST_NUM_BUCKETS (AMDGPU_BO_LIST_MAX_PRIORITY + 1) | 38 | #define AMDGPU_BO_LIST_NUM_BUCKETS (AMDGPU_BO_LIST_MAX_PRIORITY + 1) |
| 39 | #define AMDGPU_BO_LIST_MAX_ENTRIES (128 * 1024) | ||
| 39 | 40 | ||
| 40 | static void amdgpu_bo_list_free_rcu(struct rcu_head *rcu) | 41 | static void amdgpu_bo_list_free_rcu(struct rcu_head *rcu) |
| 41 | { | 42 | { |
| @@ -188,6 +189,9 @@ int amdgpu_bo_create_list_entry_array(struct drm_amdgpu_bo_list_in *in, · 146 unmodified lines | |||
| 188 | const uint32_t bo_number = in->bo_number; | 189 | const uint32_t bo_number = in->bo_number; |
| 189 | struct drm_amdgpu_bo_list_entry *info; | 190 | struct drm_amdgpu_bo_list_entry *info; |
| 190 | 191 | ||
| 192 | if (bo_number > AMDGPU_BO_LIST_MAX_ENTRIES) | ||
| 193 | return -EINVAL; | ||
| 194 | |||
| 191 | /* copy the handle array from userspace to a kernel buffer */ | 195 | /* copy the handle array from userspace to a kernel buffer */ |
| 192 | if (likely(info_size == bo_info_size)) { | 196 | if (likely(info_size == bo_info_size)) { |
| 193 | info = vmemdup_array_user(uptr, bo_number, info_size); | 197 | info = vmemdup_array_user(uptr, bo_number, info_size); |
| @@ -1231,6 +1231,8 @@ static enum drm_mode_status amdgpu_connector_dvi_mode_valid(struct drm_connector · 1230 unmodified lines | |||
| 1231 | case CONNECTOR_OBJECT_ID_HDMI_TYPE_B: | 1231 | case CONNECTOR_OBJECT_ID_HDMI_TYPE_B: |
| 1232 | max_digital_pixel_clock_khz = max_dvi_single_link_pixel_clock * 2; | 1232 | max_digital_pixel_clock_khz = max_dvi_single_link_pixel_clock * 2; |
| 1233 | break; | 1233 | break; |
| 1234 | default: | ||
| 1235 | return MODE_BAD; | ||
| 1234 | } | 1236 | } |
| 1235 | 1237 | ||
| 1236 | /* When the display EDID claims that it's an HDMI display, | 1238 | /* When the display EDID claims that it's an HDMI display, |
| @@ -908,9 +908,8 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 907 unmodified lines | |||
| 908 | goto out_free_user_pages; | 908 | goto out_free_user_pages; |
| 909 | 909 | ||
| 910 | amdgpu_bo_list_for_each_entry(e, p->bo_list) { | 910 | amdgpu_bo_list_for_each_entry(e, p->bo_list) { |
| 911 | /* One fence for TTM and one for each CS job */ | ||
| 912 | r = drm_exec_prepare_obj(&p->exec, &e->bo->tbo.base, | 911 | r = drm_exec_prepare_obj(&p->exec, &e->bo->tbo.base, |
| 913 | 1 + p->gang_size); | 912 | TTM_NUM_MOVE_FENCES + p->gang_size); |
| 914 | drm_exec_retry_on_contention(&p->exec); | 913 | drm_exec_retry_on_contention(&p->exec); |
| 915 | if (unlikely(r)) | 914 | if (unlikely(r)) |
| 916 | goto out_free_user_pages; | 915 | goto out_free_user_pages; |
| @@ -920,7 +919,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 3 unmodified lines | |||
| 920 | 919 | ||
| 921 | if (p->uf_bo) { | 920 | if (p->uf_bo) { |
| 922 | r = drm_exec_prepare_obj(&p->exec, &p->uf_bo->tbo.base, | 921 | r = drm_exec_prepare_obj(&p->exec, &p->uf_bo->tbo.base, |
| 923 | 1 + p->gang_size); | 922 | TTM_NUM_MOVE_FENCES + p->gang_size); |
| 924 | drm_exec_retry_on_contention(&p->exec); | 923 | drm_exec_retry_on_contention(&p->exec); |
| 925 | if (unlikely(r)) | 924 | if (unlikely(r)) |
| 926 | goto out_free_user_pages; | 925 | goto out_free_user_pages; |
| @@ -1740,6 +1739,13 @@ int amdgpu_cs_wait_fences_ioctl(struct drm_device *dev, void *data, · 813 unmodified lines | |||
| 1740 | struct drm_amdgpu_fence *fences; | 1739 | struct drm_amdgpu_fence *fences; |
| 1741 | int r; | 1740 | int r; |
| 1742 | 1741 | ||
| 1742 | /* | ||
| 1743 | * fence_count must be non-zero; dma_fence_wait_any_timeout() | ||
| 1744 | * does not accept an empty fence array. | ||
| 1745 | */ | ||
| 1746 | if (!wait->in.fence_count) | ||
| 1747 | return -EINVAL; | ||
| 1748 | |||
| 1743 | /* Get the fences from userspace */ | 1749 | /* Get the fences from userspace */ |
| 1744 | fences = memdup_array_user(u64_to_user_ptr(wait->in.fences), | 1750 | fences = memdup_array_user(u64_to_user_ptr(wait->in.fences), |
| 1745 | wait->in.fence_count, | 1751 | wait->in.fence_count, |
| @@ -617,6 +617,110 @@ out: · 616 unmodified lines | |||
| 617 | return r; | 617 | return r; |
| 618 | } | 618 | } |
| 619 | 619 | ||
| 620 | /** | ||
| 621 | * amdgpu_debugfs_regs_pcie64_read - Read from a 64-bit PCIE register | ||
| 622 | * | ||
| 623 | * @f: open file handle | ||
| 624 | * @buf: User buffer to store read data in | ||
| 625 | * @size: Number of bytes to read | ||
| 626 | * @pos: Offset to seek to | ||
| 627 | */ | ||
| 628 | static ssize_t amdgpu_debugfs_regs_pcie64_read(struct file *f, char __user *buf, | ||
| 629 | size_t size, loff_t *pos) | ||
| 630 | { | ||
| 631 | struct amdgpu_device *adev = file_inode(f)->i_private; | ||
| 632 | ssize_t result = 0; | ||
| 633 | int r; | ||
| 634 | |||
| 635 | if (size & 0x7 || *pos & 0x7) | ||
| 636 | return -EINVAL; | ||
| 637 | |||
| 638 | r = pm_runtime_get_sync(adev_to_drm(adev)->dev); | ||
| 639 | if (r < 0) { | ||
| 640 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 641 | return r; | ||
| 642 | } | ||
| 643 | |||
| 644 | r = amdgpu_virt_enable_access_debugfs(adev); | ||
| 645 | if (r < 0) { | ||
| 646 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 647 | return r; | ||
| 648 | } | ||
| 649 | |||
| 650 | while (size) { | ||
| 651 | uint64_t value; | ||
| 652 | |||
| 653 | value = RREG64_PCIE_EXT(*pos); | ||
| 654 | |||
| 655 | r = put_user(value, (uint64_t *)buf); | ||
| 656 | if (r) | ||
| 657 | goto out; | ||
| 658 | |||
| 659 | result += 8; | ||
| 660 | buf += 8; | ||
| 661 | *pos += 8; | ||
| 662 | size -= 8; | ||
| 663 | } | ||
| 664 | |||
| 665 | r = result; | ||
| 666 | out: | ||
| 667 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 668 | amdgpu_virt_disable_access_debugfs(adev); | ||
| 669 | return r; | ||
| 670 | } | ||
| 671 | |||
| 672 | /** | ||
| 673 | * amdgpu_debugfs_regs_pcie64_write - Write to a 64-bit PCIE register | ||
| 674 | * | ||
| 675 | * @f: open file handle | ||
| 676 | * @buf: User buffer to write data from | ||
| 677 | * @size: Number of bytes to write | ||
| 678 | * @pos: Offset to seek to | ||
| 679 | */ | ||
| 680 | static ssize_t amdgpu_debugfs_regs_pcie64_write(struct file *f, const char __user *buf, | ||
| 681 | size_t size, loff_t *pos) | ||
| 682 | { | ||
| 683 | struct amdgpu_device *adev = file_inode(f)->i_private; | ||
| 684 | ssize_t result = 0; | ||
| 685 | int r; | ||
| 686 | |||
| 687 | if (size & 0x7 || *pos & 0x7) | ||
| 688 | return -EINVAL; | ||
| 689 | |||
| 690 | r = pm_runtime_get_sync(adev_to_drm(adev)->dev); | ||
| 691 | if (r < 0) { | ||
| 692 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 693 | return r; | ||
| 694 | } | ||
| 695 | |||
| 696 | r = amdgpu_virt_enable_access_debugfs(adev); | ||
| 697 | if (r < 0) { | ||
| 698 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 699 | return r; | ||
| 700 | } | ||
| 701 | |||
| 702 | while (size) { | ||
| 703 | uint64_t value; | ||
| 704 | |||
| 705 | r = get_user(value, (uint64_t *)buf); | ||
| 706 | if (r) | ||
| 707 | goto out; | ||
| 708 | |||
| 709 | WREG64_PCIE_EXT(*pos, value); | ||
| 710 | |||
| 711 | result += 8; | ||
| 712 | buf += 8; | ||
| 713 | *pos += 8; | ||
| 714 | size -= 8; | ||
| 715 | } | ||
| 716 | |||
| 717 | r = result; | ||
| 718 | out: | ||
| 719 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 720 | amdgpu_virt_disable_access_debugfs(adev); | ||
| 721 | return r; | ||
| 722 | } | ||
| 723 | |||
| 620 | /** | 724 | /** |
| 621 | * amdgpu_debugfs_regs_didt_read - Read from a DIDT register | 725 | * amdgpu_debugfs_regs_didt_read - Read from a DIDT register |
| 622 | * | 726 | * |
| @@ -1525,6 +1629,12 @@ static const struct file_operations amdgpu_debugfs_regs_pcie_fops = { · 902 unmodified lines | |||
| 1525 | .write = amdgpu_debugfs_regs_pcie_write, | 1629 | .write = amdgpu_debugfs_regs_pcie_write, |
| 1526 | .llseek = default_llseek | 1630 | .llseek = default_llseek |
| 1527 | }; | 1631 | }; |
| 1632 | static const struct file_operations amdgpu_debugfs_regs_pcie64_fops = { | ||
| 1633 | .owner = THIS_MODULE, | ||
| 1634 | .read = amdgpu_debugfs_regs_pcie64_read, | ||
| 1635 | .write = amdgpu_debugfs_regs_pcie64_write, | ||
| 1636 | .llseek = default_llseek | ||
| 1637 | }; | ||
| 1528 | static const struct file_operations amdgpu_debugfs_regs_smc_fops = { | 1638 | static const struct file_operations amdgpu_debugfs_regs_smc_fops = { |
| 1529 | .owner = THIS_MODULE, | 1639 | .owner = THIS_MODULE, |
| 1530 | .read = amdgpu_debugfs_regs_smc_read, | 1640 | .read = amdgpu_debugfs_regs_smc_read, |
| @@ -1587,6 +1697,7 @@ static const struct file_operations *debugfs_regs[] = { · 56 unmodified lines | |||
| 1587 | &amdgpu_debugfs_gprwave_fops, | 1697 | &amdgpu_debugfs_gprwave_fops, |
| 1588 | &amdgpu_debugfs_regs_didt_fops, | 1698 | &amdgpu_debugfs_regs_didt_fops, |
| 1589 | &amdgpu_debugfs_regs_pcie_fops, | 1699 | &amdgpu_debugfs_regs_pcie_fops, |
| 1700 | &amdgpu_debugfs_regs_pcie64_fops, | ||
| 1590 | &amdgpu_debugfs_regs_smc_fops, | 1701 | &amdgpu_debugfs_regs_smc_fops, |
| 1591 | &amdgpu_debugfs_gca_config_fops, | 1702 | &amdgpu_debugfs_gca_config_fops, |
| 1592 | &amdgpu_debugfs_sensors_fops, | 1703 | &amdgpu_debugfs_sensors_fops, |
| @@ -1604,6 +1715,7 @@ static const char * const debugfs_regs_names[] = { · 11 unmodified lines | |||
| 1604 | "amdgpu_gprwave", | 1715 | "amdgpu_gprwave", |
| 1605 | "amdgpu_regs_didt", | 1716 | "amdgpu_regs_didt", |
| 1606 | "amdgpu_regs_pcie", | 1717 | "amdgpu_regs_pcie", |
| 1718 | "amdgpu_regs_pcie64", | ||
| 1607 | "amdgpu_regs_smc", | 1719 | "amdgpu_regs_smc", |
| 1608 | "amdgpu_gca_config", | 1720 | "amdgpu_gca_config", |
| 1609 | "amdgpu_sensors", | 1721 | "amdgpu_sensors", |
| @@ -32,8 +32,16 @@ void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, · 31 unmodified lines | |||
| 32 | bool vram_lost, struct amdgpu_job *job) | 32 | bool vram_lost, struct amdgpu_job *job) |
| 33 | { | 33 | { |
| 34 | } | 34 | } |
| 35 | void amdgpu_coredump_init(struct amdgpu_device *adev) | ||
| 36 | { | ||
| 37 | } | ||
| 38 | void amdgpu_coredump_fini(struct amdgpu_device *adev) | ||
| 39 | { | ||
| 40 | } | ||
| 35 | #else | 41 | #else |
| 36 | 42 | ||
| 43 | #define AMDGPU_CORE_DUMP_SIZE_MAX (256 * 1024 * 1024) | ||
| 44 | |||
| 37 | const char *hw_ip_names[MAX_HWIP] = { | 45 | const char *hw_ip_names[MAX_HWIP] = { |
| 38 | [GC_HWIP] = "GC", | 46 | [GC_HWIP] = "GC", |
| 39 | [HDP_HWIP] = "HDP", | 47 | [HDP_HWIP] = "HDP", |
| @@ -187,20 +195,22 @@ static void amdgpu_devcoredump_fw_info(struct amdgpu_device *adev, · 147 unmodified lines | |||
| 187 | drm_printf(p, "VPE feature version: %u, fw version: 0x%08x\n", | 195 | drm_printf(p, "VPE feature version: %u, fw version: 0x%08x\n", |
| 188 | adev->vpe.feature_version, adev->vpe.fw_version); | 196 | adev->vpe.feature_version, adev->vpe.fw_version); |
| 189 | 197 | ||
| 190 | drm_printf(p, "\nVBIOS Information\n"); | 198 | if (adev->bios) { |
| 191 | drm_printf(p, "vbios name : %s\n", ctx->name); | 199 | drm_printf(p, "\nVBIOS Information\n"); |
| 192 | drm_printf(p, "vbios pn : %s\n", ctx->vbios_pn); | 200 | drm_printf(p, "vbios name : %s\n", ctx->name); |
| 193 | drm_printf(p, "vbios version : %d\n", ctx->version); | 201 | drm_printf(p, "vbios pn : %s\n", ctx->vbios_pn); |
| 194 | drm_printf(p, "vbios ver_str : %s\n", ctx->vbios_ver_str); | 202 | drm_printf(p, "vbios version : %d\n", ctx->version); |
| 195 | drm_printf(p, "vbios date : %s\n", ctx->date); | 203 | drm_printf(p, "vbios ver_str : %s\n", ctx->vbios_ver_str); |
| 204 | drm_printf(p, "vbios date : %s\n", ctx->date); | ||
| 205 | }else { | ||
| 206 | drm_printf(p, "\nVBIOS Information: NA\n"); | ||
| 207 | } | ||
| 196 | } | 208 | } |
| 197 | 209 | ||
| 198 | static ssize_t | 210 | static ssize_t |
| 199 | amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, | 211 | amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_info *coredump) |
| 200 | void *data, size_t datalen) | ||
| 201 | { | 212 | { |
| 202 | struct drm_printer p; | 213 | struct drm_printer p; |
| 203 | struct amdgpu_coredump_info *coredump = data; | ||
| 204 | struct drm_print_iterator iter; | 214 | struct drm_print_iterator iter; |
| 205 | struct amdgpu_vm_fault_info *fault_info; | 215 | struct amdgpu_vm_fault_info *fault_info; |
| 206 | struct amdgpu_ip_block *ip_block; | 216 | struct amdgpu_ip_block *ip_block; |
| @@ -208,7 +218,6 @@ amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, · 1 unmodified line | |||
| 208 | 218 | ||
| 209 | iter.data = buffer; | 219 | iter.data = buffer; |
| 210 | iter.offset = 0; | 220 | iter.offset = 0; |
| 211 | iter.start = offset; | ||
| 212 | iter.remain = count; | 221 | iter.remain = count; |
| 213 | 222 | ||
| 214 | p = drm_coredump_printer(&iter); | 223 | p = drm_coredump_printer(&iter); |
| @@ -322,9 +331,63 @@ amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, · 107 unmodified lines | |||
| 322 | return count - iter.remain; | 331 | return count - iter.remain; |
| 323 | } | 332 | } |
| 324 | 333 | ||
| 334 | static ssize_t | ||
| 335 | amdgpu_devcoredump_read(char *buffer, loff_t offset, size_t count, | ||
| 336 | void *data, size_t datalen) | ||
| 337 | { | ||
| 338 | struct amdgpu_coredump_info *coredump = data; | ||
| 339 | ssize_t byte_copied; | ||
| 340 | |||
| 341 | if (!coredump) | ||
| 342 | return -ENODEV; | ||
| 343 | |||
| 344 | if (!coredump->formatted) | ||
| 345 | return -ENODEV; | ||
| 346 | |||
| 347 | if (offset >= coredump->formatted_size) | ||
| 348 | return 0; | ||
| 349 | |||
| 350 | byte_copied = count < coredump->formatted_size - offset ? count : | ||
| 351 | coredump->formatted_size - offset; | ||
| 352 | memcpy(buffer, coredump->formatted + offset, byte_copied); | ||
| 353 | |||
| 354 | return byte_copied; | ||
| 355 | } | ||
| 356 | |||
| 325 | static void amdgpu_devcoredump_free(void *data) | 357 | static void amdgpu_devcoredump_free(void *data) |
| 326 | { | 358 | { |
| 327 | kfree(data); | 359 | struct amdgpu_coredump_info *coredump = data; |
| 360 | |||
| 361 | kvfree(coredump->formatted); | ||
| 362 | kvfree(data); | ||
| 363 | } | ||
| 364 | |||
| 365 | static void amdgpu_devcoredump_deferred_work(struct work_struct *work) | ||
| 366 | { | ||
| 367 | struct amdgpu_device *adev = container_of(work, typeof(*adev), coredump_work); | ||
| 368 | struct amdgpu_coredump_info *coredump = adev->coredump; | ||
| 369 | |||
| 370 | /* Do a one-time preparation of the coredump output because | ||
| 371 | * repeatingly calling drm_coredump_printer is very slow. | ||
| 372 | */ | ||
| 373 | coredump->formatted_size = amdgpu_devcoredump_format( | ||
| 374 | NULL, AMDGPU_CORE_DUMP_SIZE_MAX, coredump); | ||
| 375 | coredump->formatted = kvzalloc(coredump->formatted_size, GFP_KERNEL); | ||
| 376 | if (!coredump->formatted) { | ||
| 377 | amdgpu_devcoredump_free(coredump); | ||
| 378 | goto end; | ||
| 379 | } | ||
| 380 | |||
| 381 | amdgpu_devcoredump_format(coredump->formatted, coredump->formatted_size, coredump); | ||
| 382 | |||
| 383 | /* If there's an existing coredump for this device, the free function will be | ||
| 384 | * called immediately so coredump might be invalid after the call to dev_coredumpm. | ||
| 385 | */ | ||
| 386 | dev_coredumpm(coredump->adev->dev, THIS_MODULE, coredump, 0, GFP_NOWAIT, | ||
| 387 | amdgpu_devcoredump_read, amdgpu_devcoredump_free); | ||
| 388 | |||
| 389 | end: | ||
| 390 | adev->coredump = NULL; | ||
| 328 | } | 391 | } |
| 329 | 392 | ||
| 330 | void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, | 393 | void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, |
| @@ -334,6 +397,10 @@ void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, · 3 unmodified lines | |||
| 334 | struct amdgpu_coredump_info *coredump; | 397 | struct amdgpu_coredump_info *coredump; |
| 335 | struct drm_sched_job *s_job; | 398 | struct drm_sched_job *s_job; |
| 336 | 399 | ||
| 400 | /* No need to generate a new coredump if there's one in progress already. */ | ||
| 401 | if (work_pending(&adev->coredump_work)) | ||
| 402 | return; | ||
| 403 | |||
| 337 | coredump = kzalloc_obj(*coredump, GFP_NOWAIT); | 404 | coredump = kzalloc_obj(*coredump, GFP_NOWAIT); |
| 338 | if (!coredump) | 405 | if (!coredump) |
| 339 | return; | 406 | return; |
| @@ -360,11 +427,26 @@ void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, · 20 unmodified lines | |||
| 360 | 427 | ||
| 361 | ktime_get_ts64(&coredump->reset_time); | 428 | ktime_get_ts64(&coredump->reset_time); |
| 362 | 429 | ||
| 363 | dev_coredumpm(dev->dev, THIS_MODULE, coredump, 0, GFP_NOWAIT, | 430 | /* Update the current coredump pointer (no lock needed, this function can only be called |
| 364 | amdgpu_devcoredump_read, amdgpu_devcoredump_free); | 431 | * from a single thread) |
| 432 | */ | ||
| 433 | adev->coredump = coredump; | ||
| 434 | /* Kick off coredump formatting to a worker thread. */ | ||
| 435 | queue_work(system_unbound_wq, &adev->coredump_work); | ||
| 365 | 436 | ||
| 366 | drm_info(dev, "AMDGPU device coredump file has been created\n"); | 437 | drm_info(dev, "AMDGPU device coredump file has been created\n"); |
| 367 | drm_info(dev, "Check your /sys/class/drm/card%d/device/devcoredump/data\n", | 438 | drm_info(dev, "Check your /sys/class/drm/card%d/device/devcoredump/data\n", |
| 368 | dev->primary->index); | 439 | dev->primary->index); |
| 369 | } | 440 | } |
| 441 | |||
| 442 | void amdgpu_coredump_init(struct amdgpu_device *adev) | ||
| 443 | { | ||
| 444 | INIT_WORK(&adev->coredump_work, amdgpu_devcoredump_deferred_work); | ||
| 445 | } | ||
| 446 | |||
| 447 | void amdgpu_coredump_fini(struct amdgpu_device *adev) | ||
| 448 | { | ||
| 449 | /* Finish deferred coredump formatting before HW/IP teardown. */ | ||
| 450 | flush_work(&adev->coredump_work); | ||
| 451 | } | ||
| 370 | #endif | 452 | #endif |
| @@ -35,12 +35,20 @@ struct amdgpu_coredump_info { · 34 unmodified lines | |||
| 35 | struct amdgpu_device *adev; | 35 | struct amdgpu_device *adev; |
| 36 | struct amdgpu_task_info reset_task_info; | 36 | struct amdgpu_task_info reset_task_info; |
| 37 | struct timespec64 reset_time; | 37 | struct timespec64 reset_time; |
| 38 | |||
| 38 | bool skip_vram_check; | 39 | bool skip_vram_check; |
| 39 | bool reset_vram_lost; | 40 | bool reset_vram_lost; |
| 40 | struct amdgpu_ring *ring; | 41 | struct amdgpu_ring *ring; |
| 42 | /* Readable form of coredevdump, generate once to speed up | ||
| 43 | * reading it (see drm_coredump_printer's documentation). | ||
| 44 | */ | ||
| 45 | ssize_t formatted_size; | ||
| 46 | char *formatted; | ||
| 41 | }; | 47 | }; |
| 42 | #endif | 48 | #endif |
| 43 | 49 | ||
| 44 | void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, | 50 | void amdgpu_coredump(struct amdgpu_device *adev, bool skip_vram_check, |
| 45 | bool vram_lost, struct amdgpu_job *job); | 51 | bool vram_lost, struct amdgpu_job *job); |
| 52 | void amdgpu_coredump_init(struct amdgpu_device *adev); | ||
| 53 | void amdgpu_coredump_fini(struct amdgpu_device *adev); | ||
| 46 | #endif | 54 | #endif |
| @@ -3498,7 +3498,8 @@ fail: · 3497 unmodified lines | |||
| 3498 | 3498 | ||
| 3499 | static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) | 3499 | static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) |
| 3500 | { | 3500 | { |
| 3501 | char *input = amdgpu_lockup_timeout; | 3501 | char buf[AMDGPU_MAX_TIMEOUT_PARAM_LENGTH]; |
| 3502 | char *input = buf; | ||
| 3502 | char *timeout_setting = NULL; | 3503 | char *timeout_setting = NULL; |
| 3503 | int index = 0; | 3504 | int index = 0; |
| 3504 | long timeout; | 3505 | long timeout; |
| @@ -3508,9 +3509,17 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) · 3 unmodified lines | |||
| 3508 | adev->gfx_timeout = adev->compute_timeout = adev->sdma_timeout = | 3509 | adev->gfx_timeout = adev->compute_timeout = adev->sdma_timeout = |
| 3509 | adev->video_timeout = msecs_to_jiffies(2000); | 3510 | adev->video_timeout = msecs_to_jiffies(2000); |
| 3510 | 3511 | ||
| 3511 | if (!strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) | 3512 | if (!strnlen(amdgpu_lockup_timeout, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) |
| 3512 | return 0; | 3513 | return 0; |
| 3513 | 3514 | ||
| 3515 | /* | ||
| 3516 | * strsep() destructively modifies its input by replacing delimiters | ||
| 3517 | * with '\0'. Use a stack copy so the global module parameter buffer | ||
| 3518 | * remains intact for multi-GPU systems where this function is called | ||
| 3519 | * once per device. | ||
| 3520 | */ | ||
| 3521 | strscpy(buf, amdgpu_lockup_timeout, sizeof(buf)); | ||
| 3522 | |||
| 3514 | while ((timeout_setting = strsep(&input, ",")) && | 3523 | while ((timeout_setting = strsep(&input, ",")) && |
| 3515 | strnlen(timeout_setting, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) { | 3524 | strnlen(timeout_setting, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) { |
| 3516 | ret = kstrtol(timeout_setting, 0, &timeout); | 3525 | ret = kstrtol(timeout_setting, 0, &timeout); |
| @@ -3781,6 +3790,8 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 264 unmodified lines | |||
| 3781 | INIT_WORK(&adev->xgmi_reset_work, amdgpu_device_xgmi_reset_func); | 3790 | INIT_WORK(&adev->xgmi_reset_work, amdgpu_device_xgmi_reset_func); |
| 3782 | INIT_WORK(&adev->userq_reset_work, amdgpu_userq_reset_work); | 3791 | INIT_WORK(&adev->userq_reset_work, amdgpu_userq_reset_work); |
| 3783 | 3792 | ||
| 3793 | amdgpu_coredump_init(adev); | ||
| 3794 | |||
| 3784 | adev->gfx.gfx_off_req_count = 1; | 3795 | adev->gfx.gfx_off_req_count = 1; |
| 3785 | adev->gfx.gfx_off_residency = 0; | 3796 | adev->gfx.gfx_off_residency = 0; |
| 3786 | adev->gfx.gfx_off_entrycount = 0; | 3797 | adev->gfx.gfx_off_entrycount = 0; |
| @@ -3878,9 +3889,21 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 91 unmodified lines | |||
| 3878 | amdgpu_gmc_noretry_set(adev); | 3889 | amdgpu_gmc_noretry_set(adev); |
| 3879 | /* Need to get xgmi info early to decide the reset behavior*/ | 3890 | /* Need to get xgmi info early to decide the reset behavior*/ |
| 3880 | if (adev->gmc.xgmi.supported) { | 3891 | if (adev->gmc.xgmi.supported) { |
| 3881 | r = adev->gfxhub.funcs->get_xgmi_info(adev); | 3892 | if (adev->gfxhub.funcs && |
| 3882 | if (r) | 3893 | adev->gfxhub.funcs->get_xgmi_info) { |
| 3883 | return r; | 3894 | r = adev->gfxhub.funcs->get_xgmi_info(adev); |
| 3895 | if (r) | ||
| 3896 | return r; | ||
| 3897 | } | ||
| 3898 | } | ||
| 3899 | |||
| 3900 | if (adev->gmc.xgmi.connected_to_cpu) { | ||
| 3901 | if (adev->mmhub.funcs && | ||
| 3902 | adev->mmhub.funcs->get_xgmi_info) { | ||
| 3903 | r = adev->mmhub.funcs->get_xgmi_info(adev); | ||
| 3904 | if (r) | ||
| 3905 | return r; | ||
| 3906 | } | ||
| 3884 | } | 3907 | } |
| 3885 | 3908 | ||
| 3886 | /* enable PCIE atomic ops */ | 3909 | /* enable PCIE atomic ops */ |
| @@ -4202,6 +4225,7 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev) · 315 unmodified lines | |||
| 4202 | if (pci_dev_is_disconnected(adev->pdev)) | 4225 | if (pci_dev_is_disconnected(adev->pdev)) |
| 4203 | amdgpu_amdkfd_device_fini_sw(adev); | 4226 | amdgpu_amdkfd_device_fini_sw(adev); |
| 4204 | 4227 | ||
| 4228 | amdgpu_coredump_fini(adev); | ||
| 4205 | amdgpu_device_ip_fini_early(adev); | 4229 | amdgpu_device_ip_fini_early(adev); |
| 4206 | 4230 | ||
| 4207 | amdgpu_irq_fini_hw(adev); | 4231 | amdgpu_irq_fini_hw(adev); |
| @@ -112,8 +112,10 @@ · 111 unmodified lines | |||
| 112 | #include "smuio_v15_0_8.h" | 112 | #include "smuio_v15_0_8.h" |
| 113 | #include "vcn_v5_0_0.h" | 113 | #include "vcn_v5_0_0.h" |
| 114 | #include "vcn_v5_0_1.h" | 114 | #include "vcn_v5_0_1.h" |
| 115 | #include "vcn_v5_0_2.h" | ||
| 115 | #include "jpeg_v5_0_0.h" | 116 | #include "jpeg_v5_0_0.h" |
| 116 | #include "jpeg_v5_0_1.h" | 117 | #include "jpeg_v5_0_1.h" |
| 118 | #include "jpeg_v5_0_2.h" | ||
| 117 | #include "jpeg_v5_3_0.h" | 119 | #include "jpeg_v5_3_0.h" |
| 118 | 120 | ||
| 119 | #include "amdgpu_ras_mgr.h" | 121 | #include "amdgpu_ras_mgr.h" |
| @@ -296,13 +298,17 @@ static int amdgpu_discovery_get_tmr_info(struct amdgpu_device *adev, · 176 unmodified lines | |||
| 296 | if (vram_size) | 298 | if (vram_size) |
| 297 | adev->discovery.offset = (vram_size << 20) - DISCOVERY_TMR_OFFSET; | 299 | adev->discovery.offset = (vram_size << 20) - DISCOVERY_TMR_OFFSET; |
| 298 | 300 | ||
| 299 | if (amdgpu_sriov_vf(adev) && adev->virt.is_dynamic_crit_regn_enabled) { | 301 | if (amdgpu_sriov_vf(adev)) { |
| 300 | adev->discovery.offset = | 302 | if (adev->virt.is_dynamic_crit_regn_enabled) { |
| 301 | adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].offset; | 303 | adev->discovery.offset = |
| 302 | adev->discovery.size = | 304 | adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].offset; |
| 303 | adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].size_kb << 10; | 305 | adev->discovery.size = |
| 304 | if (!adev->discovery.offset || !adev->discovery.size) | 306 | adev->virt.crit_regn_tbl[AMD_SRIOV_MSG_IPD_TABLE_ID].size_kb << 10; |
| 305 | return -EINVAL; | 307 | if (!adev->discovery.offset || !adev->discovery.size) |
| 308 | return -EINVAL; | ||
| 309 | } else { | ||
| 310 | goto out; | ||
| 311 | } | ||
| 306 | } else { | 312 | } else { |
| 307 | tmr_size = RREG32(mmDRIVER_SCRATCH_2); | 313 | tmr_size = RREG32(mmDRIVER_SCRATCH_2); |
| 308 | if (tmr_size) { | 314 | if (tmr_size) { |
| @@ -318,11 +324,11 @@ static int amdgpu_discovery_get_tmr_info(struct amdgpu_device *adev, · 9 unmodified lines | |||
| 318 | ret = amdgpu_acpi_get_tmr_info(adev, &tmr_offset, &tmr_size); | 324 | ret = amdgpu_acpi_get_tmr_info(adev, &tmr_offset, &tmr_size); |
| 319 | if (ret) | 325 | if (ret) |
| 320 | return ret; | 326 | return ret; |
| 321 | adev->discovery.size = (u32)tmr_size; | 327 | adev->discovery.size = DISCOVERY_TMR_SIZE; |
| 322 | adev->discovery.offset = tmr_offset + tmr_size - DISCOVERY_TMR_OFFSET; | 328 | adev->discovery.offset = tmr_offset + tmr_size - DISCOVERY_TMR_OFFSET; |
| 323 | } | 329 | } |
| 324 | } | 330 | } |
| 325 | 331 | out: | |
| 326 | adev->discovery.bin = kzalloc(adev->discovery.size, GFP_KERNEL); | 332 | adev->discovery.bin = kzalloc(adev->discovery.size, GFP_KERNEL); |
| 327 | if (!adev->discovery.bin) | 333 | if (!adev->discovery.bin) |
| 328 | return -ENOMEM; | 334 | return -ENOMEM; |
| @@ -556,7 +562,7 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 227 unmodified lines | |||
| 556 | checksum = le16_to_cpu(info->checksum); | 562 | checksum = le16_to_cpu(info->checksum); |
| 557 | 563 | ||
| 558 | switch (table_id) { | 564 | switch (table_id) { |
| 559 | case IP_DISCOVERY: | 565 | case IP_DISCOVERY: { |
| 560 | struct ip_discovery_header *ihdr = | 566 | struct ip_discovery_header *ihdr = |
| 561 | (struct ip_discovery_header *)(discovery_bin + offset); | 567 | (struct ip_discovery_header *)(discovery_bin + offset); |
| 562 | act_val = le32_to_cpu(ihdr->signature); | 568 | act_val = le32_to_cpu(ihdr->signature); |
| @@ -564,7 +570,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | |||
| 564 | table_size = le16_to_cpu(ihdr->size); | 570 | table_size = le16_to_cpu(ihdr->size); |
| 565 | table_name = "data table"; | 571 | table_name = "data table"; |
| 566 | break; | 572 | break; |
| 567 | case GC: | 573 | } |
| 574 | case GC: { | ||
| 568 | struct gpu_info_header *ghdr = | 575 | struct gpu_info_header *ghdr = |
| 569 | (struct gpu_info_header *)(discovery_bin + offset); | 576 | (struct gpu_info_header *)(discovery_bin + offset); |
| 570 | act_val = le32_to_cpu(ghdr->table_id); | 577 | act_val = le32_to_cpu(ghdr->table_id); |
| @@ -572,7 +579,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | |||
| 572 | table_size = le16_to_cpu(ghdr->size); | 579 | table_size = le16_to_cpu(ghdr->size); |
| 573 | table_name = "gc table"; | 580 | table_name = "gc table"; |
| 574 | break; | 581 | break; |
| 575 | case HARVEST_INFO: | 582 | } |
| 583 | case HARVEST_INFO: { | ||
| 576 | struct harvest_info_header *hhdr = | 584 | struct harvest_info_header *hhdr = |
| 577 | (struct harvest_info_header *)(discovery_bin + offset); | 585 | (struct harvest_info_header *)(discovery_bin + offset); |
| 578 | act_val = le32_to_cpu(hhdr->signature); | 586 | act_val = le32_to_cpu(hhdr->signature); |
| @@ -580,7 +588,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | |||
| 580 | table_size = sizeof(struct harvest_table); | 588 | table_size = sizeof(struct harvest_table); |
| 581 | table_name = "harvest table"; | 589 | table_name = "harvest table"; |
| 582 | break; | 590 | break; |
| 583 | case VCN_INFO: | 591 | } |
| 592 | case VCN_INFO: { | ||
| 584 | struct vcn_info_header *vhdr = | 593 | struct vcn_info_header *vhdr = |
| 585 | (struct vcn_info_header *)(discovery_bin + offset); | 594 | (struct vcn_info_header *)(discovery_bin + offset); |
| 586 | act_val = le32_to_cpu(vhdr->table_id); | 595 | act_val = le32_to_cpu(vhdr->table_id); |
| @@ -588,7 +597,8 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 1 unmodified line | |||
| 588 | table_size = le32_to_cpu(vhdr->size_bytes); | 597 | table_size = le32_to_cpu(vhdr->size_bytes); |
| 589 | table_name = "vcn table"; | 598 | table_name = "vcn table"; |
| 590 | break; | 599 | break; |
| 591 | case MALL_INFO: | 600 | } |
| 601 | case MALL_INFO: { | ||
| 592 | struct mall_info_header *mhdr = | 602 | struct mall_info_header *mhdr = |
| 593 | (struct mall_info_header *)(discovery_bin + offset); | 603 | (struct mall_info_header *)(discovery_bin + offset); |
| 594 | act_val = le32_to_cpu(mhdr->table_id); | 604 | act_val = le32_to_cpu(mhdr->table_id); |
| @@ -597,6 +607,7 @@ static int amdgpu_discovery_table_check(struct amdgpu_device *adev, · 2 unmodified lines | |||
| 597 | table_name = "mall table"; | 607 | table_name = "mall table"; |
| 598 | check_table = false; | 608 | check_table = false; |
| 599 | break; | 609 | break; |
| 610 | } | ||
| 600 | default: | 611 | default: |
| 601 | dev_err(adev->dev, "invalid ip discovery table id %d specified\n", table_id); | 612 | dev_err(adev->dev, "invalid ip discovery table id %d specified\n", table_id); |
| 602 | check_table = false; | 613 | check_table = false; |
| @@ -1383,6 +1394,9 @@ static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev) · 780 unmodified lines | |||
| 1383 | struct list_head *el, *tmp; | 1394 | struct list_head *el, *tmp; |
| 1384 | struct kset *die_kset; | 1395 | struct kset *die_kset; |
| 1385 | 1396 | ||
| 1397 | if (!ip_top) | ||
| 1398 | return; | ||
| 1399 | |||
| 1386 | die_kset = &ip_top->die_kset; | 1400 | die_kset = &ip_top->die_kset; |
| 1387 | spin_lock(&die_kset->list_lock); | 1401 | spin_lock(&die_kset->list_lock); |
| 1388 | list_for_each_prev_safe(el, tmp, &die_kset->list) { | 1402 | list_for_each_prev_safe(el, tmp, &die_kset->list) { |
| @@ -1407,9 +1421,13 @@ void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p) · 18 unmodified lines | |||
| 1407 | struct ip_hw_instance *ip_inst; | 1421 | struct ip_hw_instance *ip_inst; |
| 1408 | int i = 0, j; | 1422 | int i = 0, j; |
| 1409 | 1423 | ||
| 1424 | if (!ip_top) | ||
| 1425 | return; | ||
| 1426 | |||
| 1410 | die_kset = &ip_top->die_kset; | 1427 | die_kset = &ip_top->die_kset; |
| 1411 | 1428 | ||
| 1412 | drm_printf(p, "\nHW IP Discovery\n"); | 1429 | drm_printf(p, "\nHW IP Discovery\n"); |
| 1430 | |||
| 1413 | spin_lock(&die_kset->list_lock); | 1431 | spin_lock(&die_kset->list_lock); |
| 1414 | list_for_each(el_die, &die_kset->list) { | 1432 | list_for_each(el_die, &die_kset->list) { |
| 1415 | drm_printf(p, "die %d\n", i++); | 1433 | drm_printf(p, "die %d\n", i++); |
| @@ -1966,11 +1984,10 @@ static int amdgpu_discovery_refresh_nps_info(struct amdgpu_device *adev, · 550 unmodified lines | |||
| 1966 | 1984 | ||
| 1967 | int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, | 1985 | int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, |
| 1968 | uint32_t *nps_type, | 1986 | uint32_t *nps_type, |
| 1969 | struct amdgpu_gmc_memrange **ranges, | 1987 | struct amdgpu_gmc_memrange *ranges, |
| 1970 | int *range_cnt, bool refresh) | 1988 | int *range_cnt, bool refresh) |
| 1971 | { | 1989 | { |
| 1972 | uint8_t *discovery_bin = adev->discovery.bin; | 1990 | uint8_t *discovery_bin = adev->discovery.bin; |
| 1973 | struct amdgpu_gmc_memrange *mem_ranges; | ||
| 1974 | struct table_info *info; | 1991 | struct table_info *info; |
| 1975 | union nps_info *nps_info; | 1992 | union nps_info *nps_info; |
| 1976 | union nps_info nps_data; | 1993 | union nps_info nps_data; |
| @@ -2008,20 +2025,22 @@ int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, · 31 unmodified lines | |||
| 2008 | 2025 | ||
| 2009 | switch (le16_to_cpu(nps_info->v1.header.version_major)) { | 2026 | switch (le16_to_cpu(nps_info->v1.header.version_major)) { |
| 2010 | case 1: | 2027 | case 1: |
| 2011 | mem_ranges = kvzalloc_objs(*mem_ranges, nps_info->v1.count); | ||
| 2012 | if (!mem_ranges) | ||
| 2013 | return -ENOMEM; | ||
| 2014 | *nps_type = nps_info->v1.nps_type; | 2028 | *nps_type = nps_info->v1.nps_type; |
| 2029 | if (*range_cnt < nps_info->v1.count) { | ||
| 2030 | dev_dbg(adev->dev, | ||
| 2031 | "not enough space for nps ranges: %d < %d\n", | ||
| 2032 | *range_cnt, nps_info->v1.count); | ||
| 2033 | return -ENOSPC; | ||
| 2034 | } | ||
| 2015 | *range_cnt = nps_info->v1.count; | 2035 | *range_cnt = nps_info->v1.count; |
| 2016 | for (i = 0; i < *range_cnt; i++) { | 2036 | for (i = 0; i < *range_cnt; i++) { |
| 2017 | mem_ranges[i].base_address = | 2037 | ranges[i].base_address = |
| 2018 | nps_info->v1.instance_info[i].base_address; | 2038 | nps_info->v1.instance_info[i].base_address; |
| 2019 | mem_ranges[i].limit_address = | 2039 | ranges[i].limit_address = |
| 2020 | nps_info->v1.instance_info[i].limit_address; | 2040 | nps_info->v1.instance_info[i].limit_address; |
| 2021 | mem_ranges[i].nid_mask = -1; | 2041 | ranges[i].nid_mask = -1; |
| 2022 | mem_ranges[i].flags = 0; | 2042 | ranges[i].flags = 0; |
| 2023 | } | 2043 | } |
| 2024 | *ranges = mem_ranges; | ||
| 2025 | break; | 2044 | break; |
| 2026 | default: | 2045 | default: |
| 2027 | dev_err(adev->dev, "Unhandled NPS info table %d.%d\n", | 2046 | dev_err(adev->dev, "Unhandled NPS info table %d.%d\n", |
| @@ -2323,6 +2342,7 @@ static int amdgpu_discovery_set_smu_ip_blocks(struct amdgpu_device *adev) · 295 unmodified lines | |||
| 2323 | amdgpu_device_ip_block_add(adev, &smu_v14_0_ip_block); | 2342 | amdgpu_device_ip_block_add(adev, &smu_v14_0_ip_block); |
| 2324 | break; | 2343 | break; |
| 2325 | case IP_VERSION(15, 0, 0): | 2344 | case IP_VERSION(15, 0, 0): |
| 2345 | case IP_VERSION(15, 0, 8): | ||
| 2326 | amdgpu_device_ip_block_add(adev, &smu_v15_0_ip_block); | 2346 | amdgpu_device_ip_block_add(adev, &smu_v15_0_ip_block); |
| 2327 | break; | 2347 | break; |
| 2328 | default: | 2348 | default: |
| @@ -2640,6 +2660,10 @@ static int amdgpu_discovery_set_mm_ip_blocks(struct amdgpu_device *adev) · 311 unmodified lines | |||
| 2640 | amdgpu_device_ip_block_add(adev, &vcn_v5_0_1_ip_block); | 2660 | amdgpu_device_ip_block_add(adev, &vcn_v5_0_1_ip_block); |
| 2641 | amdgpu_device_ip_block_add(adev, &jpeg_v5_0_1_ip_block); | 2661 | amdgpu_device_ip_block_add(adev, &jpeg_v5_0_1_ip_block); |
| 2642 | break; | 2662 | break; |
| 2663 | case IP_VERSION(5, 0, 2): | ||
| 2664 | amdgpu_device_ip_block_add(adev, &vcn_v5_0_2_ip_block); | ||
| 2665 | amdgpu_device_ip_block_add(adev, &jpeg_v5_0_2_ip_block); | ||
| 2666 | break; | ||
| 2643 | default: | 2667 | default: |
| 2644 | dev_err(adev->dev, | 2668 | dev_err(adev->dev, |
| 2645 | "Failed to add vcn/jpeg ip block(UVD_HWIP:0x%x)\n", | 2669 | "Failed to add vcn/jpeg ip block(UVD_HWIP:0x%x)\n", |
| @@ -46,7 +46,7 @@ int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev); · 45 unmodified lines | |||
| 46 | 46 | ||
| 47 | int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, | 47 | int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, |
| 48 | uint32_t *nps_type, | 48 | uint32_t *nps_type, |
| 49 | struct amdgpu_gmc_memrange **ranges, | 49 | struct amdgpu_gmc_memrange *ranges, |
| 50 | int *range_cnt, bool refresh); | 50 | int *range_cnt, bool refresh); |
| 51 | 51 | ||
| 52 | void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p); | 52 | void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p); |
| @@ -839,8 +839,8 @@ module_param_named_unsafe(no_queue_eviction_on_vm_fault, amdgpu_no_queue_evictio · 838 unmodified lines | |||
| 839 | /** | 839 | /** |
| 840 | * DOC: mtype_local (int) | 840 | * DOC: mtype_local (int) |
| 841 | */ | 841 | */ |
| 842 | int amdgpu_mtype_local; | 842 | int amdgpu_mtype_local = -1; |
| 843 | MODULE_PARM_DESC(mtype_local, "MTYPE for local memory (0 = MTYPE_RW (default), 1 = MTYPE_NC, 2 = MTYPE_CC)"); | 843 | MODULE_PARM_DESC(mtype_local, "MTYPE for local memory (default: ASIC dependent, 0 = MTYPE_RW, 1 = MTYPE_NC, 2 = MTYPE_CC)"); |
| 844 | module_param_named_unsafe(mtype_local, amdgpu_mtype_local, int, 0444); | 844 | module_param_named_unsafe(mtype_local, amdgpu_mtype_local, int, 0444); |
| 845 | 845 | ||
| 846 | /** | 846 | /** |
| @@ -2683,8 +2683,12 @@ static int amdgpu_pmops_freeze(struct device *dev) · 1836 unmodified lines | |||
| 2683 | if (r) | 2683 | if (r) |
| 2684 | return r; | 2684 | return r; |
| 2685 | 2685 | ||
| 2686 | if (amdgpu_acpi_should_gpu_reset(adev)) | 2686 | if (amdgpu_acpi_should_gpu_reset(adev)) { |
| 2687 | return amdgpu_asic_reset(adev); | 2687 | amdgpu_device_lock_reset_domain(adev->reset_domain); |
| 2688 | r = amdgpu_asic_reset(adev); | ||
| 2689 | amdgpu_device_unlock_reset_domain(adev->reset_domain); | ||
| 2690 | return r; | ||
| 2691 | } | ||
| 2688 | return 0; | 2692 | return 0; |
| 2689 | } | 2693 | } |
| 2690 | 2694 | ||
| @@ -2952,9 +2956,11 @@ static int amdgpu_drm_release(struct inode *inode, struct file *filp) · 261 unmodified lines | |||
| 2952 | int idx; | 2956 | int idx; |
| 2953 | 2957 | ||
| 2954 | if (fpriv && drm_dev_enter(dev, &idx)) { | 2958 | if (fpriv && drm_dev_enter(dev, &idx)) { |
| 2955 | fpriv->evf_mgr.fd_closing = true; | 2959 | amdgpu_evf_mgr_shutdown(&fpriv->evf_mgr); |
| 2956 | amdgpu_eviction_fence_destroy(&fpriv->evf_mgr); | 2960 | amdgpu_userq_mgr_cancel_resume(&fpriv->userq_mgr); |
| 2961 | amdgpu_evf_mgr_flush_suspend(&fpriv->evf_mgr); | ||
| 2957 | amdgpu_userq_mgr_fini(&fpriv->userq_mgr); | 2962 | amdgpu_userq_mgr_fini(&fpriv->userq_mgr); |
| 2963 | amdgpu_evf_mgr_fini(&fpriv->evf_mgr); | ||
| 2958 | drm_dev_exit(idx); | 2964 | drm_dev_exit(idx); |
| 2959 | } | 2965 | } |
| 2960 | 2966 | ||
| @@ -25,9 +25,6 @@ · 24 unmodified lines | |||
| 25 | #include <drm/drm_exec.h> | 25 | #include <drm/drm_exec.h> |
| 26 | #include "amdgpu.h" | 26 | #include "amdgpu.h" |
| 27 | 27 | ||
| 28 | #define work_to_evf_mgr(w, name) container_of(w, struct amdgpu_eviction_fence_mgr, name) | ||
| 29 | #define evf_mgr_to_fpriv(e) container_of(e, struct amdgpu_fpriv, evf_mgr) | ||
| 30 | |||
| 31 | static const char * | 28 | static const char * |
| 32 | amdgpu_eviction_fence_get_driver_name(struct dma_fence *fence) | 29 | amdgpu_eviction_fence_get_driver_name(struct dma_fence *fence) |
| 33 | { | 30 | { |
| @@ -43,127 +40,93 @@ amdgpu_eviction_fence_get_timeline_name(struct dma_fence *f) · 9 unmodified lines | |||
| 43 | return ef->timeline_name; | 40 | return ef->timeline_name; |
| 44 | } | 41 | } |
| 45 | 42 | ||
| 46 | int | 43 | static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f) |
| 47 | amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 48 | struct drm_exec *exec) | ||
| 49 | { | 44 | { |
| 50 | struct amdgpu_eviction_fence *old_ef, *new_ef; | 45 | struct amdgpu_eviction_fence *ev_fence = to_ev_fence(f); |
| 51 | struct drm_gem_object *obj; | ||
| 52 | unsigned long index; | ||
| 53 | int ret; | ||
| 54 | |||
| 55 | if (evf_mgr->ev_fence && | ||
| 56 | !dma_fence_is_signaled(&evf_mgr->ev_fence->base)) | ||
| 57 | return 0; | ||
| 58 | /* | ||
| 59 | * Steps to replace eviction fence: | ||
| 60 | * * lock all objects in exec (caller) | ||
| 61 | * * create a new eviction fence | ||
| 62 | * * update new eviction fence in evf_mgr | ||
| 63 | * * attach the new eviction fence to BOs | ||
| 64 | * * release the old fence | ||
| 65 | * * unlock the objects (caller) | ||
| 66 | */ | ||
| 67 | new_ef = amdgpu_eviction_fence_create(evf_mgr); | ||
| 68 | if (!new_ef) { | ||
| 69 | DRM_ERROR("Failed to create new eviction fence\n"); | ||
| 70 | return -ENOMEM; | ||
| 71 | } | ||
| 72 | |||
| 73 | /* Update the eviction fence now */ | ||
| 74 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 75 | old_ef = evf_mgr->ev_fence; | ||
| 76 | evf_mgr->ev_fence = new_ef; | ||
| 77 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 78 | 46 | ||
| 79 | /* Attach the new fence */ | 47 | schedule_work(&ev_fence->evf_mgr->suspend_work); |
| 80 | drm_exec_for_each_locked_object(exec, index, obj) { | 48 | return true; |
| 81 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | ||
| 82 | |||
| 83 | if (!bo) | ||
| 84 | continue; | ||
| 85 | ret = amdgpu_eviction_fence_attach(evf_mgr, bo); | ||
| 86 | if (ret) { | ||
| 87 | DRM_ERROR("Failed to attch new eviction fence\n"); | ||
| 88 | goto free_err; | ||
| 89 | } | ||
| 90 | } | ||
| 91 | |||
| 92 | /* Free old fence */ | ||
| 93 | if (old_ef) | ||
| 94 | dma_fence_put(&old_ef->base); | ||
| 95 | return 0; | ||
| 96 | |||
| 97 | free_err: | ||
| 98 | kfree(new_ef); | ||
| 99 | return ret; | ||
| 100 | } | 49 | } |
| 101 | 50 | ||
| 51 | static const struct dma_fence_ops amdgpu_eviction_fence_ops = { | ||
| 52 | .get_driver_name = amdgpu_eviction_fence_get_driver_name, | ||
| 53 | .get_timeline_name = amdgpu_eviction_fence_get_timeline_name, | ||
| 54 | .enable_signaling = amdgpu_eviction_fence_enable_signaling, | ||
| 55 | }; | ||
| 56 | |||
| 102 | static void | 57 | static void |
| 103 | amdgpu_eviction_fence_suspend_worker(struct work_struct *work) | 58 | amdgpu_eviction_fence_suspend_worker(struct work_struct *work) |
| 104 | { | 59 | { |
| 105 | struct amdgpu_eviction_fence_mgr *evf_mgr = work_to_evf_mgr(work, suspend_work.work); | 60 | struct amdgpu_eviction_fence_mgr *evf_mgr = |
| 106 | struct amdgpu_fpriv *fpriv = evf_mgr_to_fpriv(evf_mgr); | 61 | container_of(work, struct amdgpu_eviction_fence_mgr, |
| 62 | suspend_work); | ||
| 63 | struct amdgpu_fpriv *fpriv = | ||
| 64 | container_of(evf_mgr, struct amdgpu_fpriv, evf_mgr); | ||
| 107 | struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr; | 65 | struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr; |
| 108 | struct amdgpu_eviction_fence *ev_fence; | 66 | struct dma_fence *ev_fence; |
| 67 | bool cookie; | ||
| 109 | 68 | ||
| 110 | mutex_lock(&uq_mgr->userq_mutex); | 69 | mutex_lock(&uq_mgr->userq_mutex); |
| 111 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 112 | ev_fence = evf_mgr->ev_fence; | ||
| 113 | if (ev_fence) | ||
| 114 | dma_fence_get(&ev_fence->base); | ||
| 115 | else | ||
| 116 | goto unlock; | ||
| 117 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 118 | 70 | ||
| 119 | amdgpu_userq_evict(uq_mgr, ev_fence); | 71 | /* |
| 72 | * This is intentionally after taking the userq_mutex since we do | ||
| 73 | * allocate memory while holding this lock, but only after ensuring that | ||
| 74 | * the eviction fence is signaled. | ||
| 75 | */ | ||
| 76 | cookie = dma_fence_begin_signalling(); | ||
| 120 | 77 | ||
| 121 | mutex_unlock(&uq_mgr->userq_mutex); | 78 | ev_fence = amdgpu_evf_mgr_get_fence(evf_mgr); |
| 122 | dma_fence_put(&ev_fence->base); | 79 | amdgpu_userq_evict(uq_mgr); |
| 123 | return; | 80 | |
| 81 | /* | ||
| 82 | * Signaling the eviction fence must be done while holding the | ||
| 83 | * userq_mutex. Otherwise we won't resume the queues before issuing the | ||
| 84 | * next fence. | ||
| 85 | */ | ||
| 86 | dma_fence_signal(ev_fence); | ||
| 87 | dma_fence_end_signalling(cookie); | ||
| 88 | dma_fence_put(ev_fence); | ||
| 89 | |||
| 90 | if (!evf_mgr->shutdown) | ||
| 91 | schedule_delayed_work(&uq_mgr->resume_work, 0); | ||
| 124 | 92 | ||
| 125 | unlock: | ||
| 126 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 127 | mutex_unlock(&uq_mgr->userq_mutex); | 93 | mutex_unlock(&uq_mgr->userq_mutex); |
| 128 | } | 94 | } |
| 129 | 95 | ||
| 130 | static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f) | 96 | int amdgpu_evf_mgr_attach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, |
| 97 | struct amdgpu_bo *bo) | ||
| 131 | { | 98 | { |
| 132 | struct amdgpu_eviction_fence_mgr *evf_mgr; | 99 | struct dma_fence *ev_fence = amdgpu_evf_mgr_get_fence(evf_mgr); |
| 133 | struct amdgpu_eviction_fence *ev_fence; | 100 | struct ttm_operation_ctx ctx = { false, false }; |
| 134 | 101 | struct dma_resv *resv = bo->tbo.base.resv; | |
| 135 | if (!f) | 102 | int ret; |
| 136 | return true; | ||
| 137 | |||
| 138 | ev_fence = to_ev_fence(f); | ||
| 139 | evf_mgr = ev_fence->evf_mgr; | ||
| 140 | 103 | ||
| 141 | schedule_delayed_work(&evf_mgr->suspend_work, 0); | 104 | if (!dma_fence_is_signaled(ev_fence)) { |
| 142 | return true; | ||
| 143 | } | ||
| 144 | 105 | ||
| 145 | static const struct dma_fence_ops amdgpu_eviction_fence_ops = { | 106 | amdgpu_bo_placement_from_domain(bo, bo->allowed_domains); |
| 146 | .get_driver_name = amdgpu_eviction_fence_get_driver_name, | 107 | ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); |
| 147 | .get_timeline_name = amdgpu_eviction_fence_get_timeline_name, | 108 | if (!ret) |
| 148 | .enable_signaling = amdgpu_eviction_fence_enable_signaling, | 109 | dma_resv_add_fence(resv, ev_fence, |
| 149 | }; | 110 | DMA_RESV_USAGE_BOOKKEEP); |
| 111 | } else { | ||
| 112 | ret = 0; | ||
| 113 | } | ||
| 150 | 114 | ||
| 151 | void amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr, | 115 | dma_fence_put(ev_fence); |
| 152 | struct amdgpu_eviction_fence *ev_fence) | 116 | return ret; |
| 153 | { | ||
| 154 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 155 | dma_fence_signal(&ev_fence->base); | ||
| 156 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 157 | } | 117 | } |
| 158 | 118 | ||
| 159 | struct amdgpu_eviction_fence * | 119 | int amdgpu_evf_mgr_rearm(struct amdgpu_eviction_fence_mgr *evf_mgr, |
| 160 | amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr) | 120 | struct drm_exec *exec) |
| 161 | { | 121 | { |
| 162 | struct amdgpu_eviction_fence *ev_fence; | 122 | struct amdgpu_eviction_fence *ev_fence; |
| 123 | struct drm_gem_object *obj; | ||
| 124 | unsigned long index; | ||
| 163 | 125 | ||
| 126 | /* Create and initialize a new eviction fence */ | ||
| 164 | ev_fence = kzalloc_obj(*ev_fence); | 127 | ev_fence = kzalloc_obj(*ev_fence); |
| 165 | if (!ev_fence) | 128 | if (!ev_fence) |
| 166 | return NULL; | 129 | return -ENOMEM; |
| 167 | 130 | ||
| 168 | ev_fence->evf_mgr = evf_mgr; | 131 | ev_fence->evf_mgr = evf_mgr; |
| 169 | get_task_comm(ev_fence->timeline_name, current); | 132 | get_task_comm(ev_fence->timeline_name, current); |
| @@ -171,56 +134,22 @@ amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr) · 1 unmodified line | |||
| 171 | dma_fence_init64(&ev_fence->base, &amdgpu_eviction_fence_ops, | 134 | dma_fence_init64(&ev_fence->base, &amdgpu_eviction_fence_ops, |
| 172 | &ev_fence->lock, evf_mgr->ev_fence_ctx, | 135 | &ev_fence->lock, evf_mgr->ev_fence_ctx, |
| 173 | atomic_inc_return(&evf_mgr->ev_fence_seq)); | 136 | atomic_inc_return(&evf_mgr->ev_fence_seq)); |
| 174 | return ev_fence; | ||
| 175 | } | ||
| 176 | |||
| 177 | void amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 178 | { | ||
| 179 | struct amdgpu_eviction_fence *ev_fence; | ||
| 180 | |||
| 181 | /* Wait for any pending work to execute */ | ||
| 182 | flush_delayed_work(&evf_mgr->suspend_work); | ||
| 183 | |||
| 184 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 185 | ev_fence = evf_mgr->ev_fence; | ||
| 186 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 187 | |||
| 188 | if (!ev_fence) | ||
| 189 | return; | ||
| 190 | |||
| 191 | dma_fence_wait(&ev_fence->base, false); | ||
| 192 | |||
| 193 | /* Last unref of ev_fence */ | ||
| 194 | dma_fence_put(&ev_fence->base); | ||
| 195 | } | ||
| 196 | 137 | ||
| 197 | int amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr, | 138 | /* Remember it for newly added BOs */ |
| 198 | struct amdgpu_bo *bo) | 139 | dma_fence_put(evf_mgr->ev_fence); |
| 199 | { | 140 | evf_mgr->ev_fence = &ev_fence->base; |
| 200 | struct amdgpu_eviction_fence *ev_fence; | ||
| 201 | struct dma_resv *resv = bo->tbo.base.resv; | ||
| 202 | int ret; | ||
| 203 | 141 | ||
| 204 | if (!resv) | 142 | /* And add it to all existing BOs */ |
| 205 | return 0; | 143 | drm_exec_for_each_locked_object(exec, index, obj) { |
| 144 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | ||
| 206 | 145 | ||
| 207 | ret = dma_resv_reserve_fences(resv, 1); | 146 | amdgpu_evf_mgr_attach_fence(evf_mgr, bo); |
| 208 | if (ret) { | ||
| 209 | DRM_DEBUG_DRIVER("Failed to resv fence space\n"); | ||
| 210 | return ret; | ||
| 211 | } | 147 | } |
| 212 | |||
| 213 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 214 | ev_fence = evf_mgr->ev_fence; | ||
| 215 | if (ev_fence) | ||
| 216 | dma_resv_add_fence(resv, &ev_fence->base, DMA_RESV_USAGE_BOOKKEEP); | ||
| 217 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 218 | |||
| 219 | return 0; | 148 | return 0; |
| 220 | } | 149 | } |
| 221 | 150 | ||
| 222 | void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, | 151 | void amdgpu_evf_mgr_detach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, |
| 223 | struct amdgpu_bo *bo) | 152 | struct amdgpu_bo *bo) |
| 224 | { | 153 | { |
| 225 | struct dma_fence *stub = dma_fence_get_stub(); | 154 | struct dma_fence *stub = dma_fence_get_stub(); |
| 226 | 155 | ||
| @@ -229,13 +158,31 @@ void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, · 2 unmodified lines | |||
| 229 | dma_fence_put(stub); | 158 | dma_fence_put(stub); |
| 230 | } | 159 | } |
| 231 | 160 | ||
| 232 | int amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr) | 161 | void amdgpu_evf_mgr_init(struct amdgpu_eviction_fence_mgr *evf_mgr) |
| 233 | { | 162 | { |
| 234 | /* This needs to be done one time per open */ | ||
| 235 | atomic_set(&evf_mgr->ev_fence_seq, 0); | 163 | atomic_set(&evf_mgr->ev_fence_seq, 0); |
| 236 | evf_mgr->ev_fence_ctx = dma_fence_context_alloc(1); | 164 | evf_mgr->ev_fence_ctx = dma_fence_context_alloc(1); |
| 237 | spin_lock_init(&evf_mgr->ev_fence_lock); | 165 | evf_mgr->ev_fence = dma_fence_get_stub(); |
| 238 | 166 | ||
| 239 | INIT_DELAYED_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker); | 167 | INIT_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker); |
| 240 | return 0; | 168 | } |
| 169 | |||
| 170 | void amdgpu_evf_mgr_shutdown(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 171 | { | ||
| 172 | evf_mgr->shutdown = true; | ||
| 173 | /* Make sure that the shutdown is visible to the suspend work */ | ||
| 174 | flush_work(&evf_mgr->suspend_work); | ||
| 175 | } | ||
| 176 | |||
| 177 | void amdgpu_evf_mgr_flush_suspend(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 178 | { | ||
| 179 | dma_fence_wait(rcu_dereference_protected(evf_mgr->ev_fence, true), | ||
| 180 | false); | ||
| 181 | /* Make sure that we are done with the last suspend work */ | ||
| 182 | flush_work(&evf_mgr->suspend_work); | ||
| 183 | } | ||
| 184 | |||
| 185 | void amdgpu_evf_mgr_fini(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 186 | { | ||
| 187 | dma_fence_put(evf_mgr->ev_fence); | ||
| 241 | } | 188 | } |
| @@ -25,6 +25,8 @@ · 24 unmodified lines | |||
| 25 | #ifndef AMDGPU_EV_FENCE_H_ | 25 | #ifndef AMDGPU_EV_FENCE_H_ |
| 26 | #define AMDGPU_EV_FENCE_H_ | 26 | #define AMDGPU_EV_FENCE_H_ |
| 27 | 27 | ||
| 28 | #include <linux/dma-fence.h> | ||
| 29 | |||
| 28 | struct amdgpu_eviction_fence { | 30 | struct amdgpu_eviction_fence { |
| 29 | struct dma_fence base; | 31 | struct dma_fence base; |
| 30 | spinlock_t lock; | 32 | spinlock_t lock; |
| @@ -35,35 +37,36 @@ struct amdgpu_eviction_fence { · 4 unmodified lines | |||
| 35 | struct amdgpu_eviction_fence_mgr { | 37 | struct amdgpu_eviction_fence_mgr { |
| 36 | u64 ev_fence_ctx; | 38 | u64 ev_fence_ctx; |
| 37 | atomic_t ev_fence_seq; | 39 | atomic_t ev_fence_seq; |
| 38 | spinlock_t ev_fence_lock; | ||
| 39 | struct amdgpu_eviction_fence *ev_fence; | ||
| 40 | struct delayed_work suspend_work; | ||
| 41 | uint8_t fd_closing; | ||
| 42 | }; | ||
| 43 | |||
| 44 | /* Eviction fence helper functions */ | ||
| 45 | struct amdgpu_eviction_fence * | ||
| 46 | amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 47 | 40 | ||
| 48 | void | 41 | /* |
| 49 | amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr); | 42 | * Only updated while holding the VM resv lock. |
| 50 | 43 | * Only signaled while holding the userq mutex. | |
| 51 | int | 44 | */ |
| 52 | amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr, | 45 | struct dma_fence __rcu *ev_fence; |
| 53 | struct amdgpu_bo *bo); | 46 | struct work_struct suspend_work; |
| 47 | bool shutdown; | ||
| 48 | }; | ||
| 54 | 49 | ||
| 55 | void | 50 | static inline struct dma_fence * |
| 56 | amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, | 51 | amdgpu_evf_mgr_get_fence(struct amdgpu_eviction_fence_mgr *evf_mgr) |
| 57 | struct amdgpu_bo *bo); | 52 | { |
| 53 | struct dma_fence *ev_fence; | ||
| 58 | 54 | ||
| 59 | int | 55 | rcu_read_lock(); |
| 60 | amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr); | 56 | ev_fence = dma_fence_get_rcu_safe(&evf_mgr->ev_fence); |
| 57 | rcu_read_unlock(); | ||
| 58 | return ev_fence; | ||
| 59 | } | ||
| 61 | 60 | ||
| 62 | void | 61 | int amdgpu_evf_mgr_attach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, |
| 63 | amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr, | 62 | struct amdgpu_bo *bo); |
| 64 | struct amdgpu_eviction_fence *ev_fence); | 63 | int amdgpu_evf_mgr_rearm(struct amdgpu_eviction_fence_mgr *evf_mgr, |
| 64 | struct drm_exec *exec); | ||
| 65 | void amdgpu_evf_mgr_detach_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 66 | struct amdgpu_bo *bo); | ||
| 67 | void amdgpu_evf_mgr_init(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 68 | void amdgpu_evf_mgr_shutdown(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 69 | void amdgpu_evf_mgr_flush_suspend(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 70 | void amdgpu_evf_mgr_fini(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 65 | 71 | ||
| 66 | int | ||
| 67 | amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 68 | struct drm_exec *exec); | ||
| 69 | #endif | 72 | #endif |
| @@ -438,7 +438,7 @@ int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring, · 437 unmodified lines | |||
| 438 | * @ring: ring to init the fence driver on | 438 | * @ring: ring to init the fence driver on |
| 439 | * | 439 | * |
| 440 | * Init the fence driver for the requested ring (all asics). | 440 | * Init the fence driver for the requested ring (all asics). |
| 441 | * Helper function for amdgpu_fence_driver_init(). | 441 | * Helper function for amdgpu_fence_driver_sw_init(). |
| 442 | */ | 442 | */ |
| 443 | int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring) | 443 | int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring) |
| 444 | { | 444 | { |
| @@ -106,6 +106,7 @@ static bool is_fru_eeprom_supported(struct amdgpu_device *adev, u32 *fru_addr) · 105 unmodified lines | |||
| 106 | *fru_addr = FRU_EEPROM_MADDR_8; | 106 | *fru_addr = FRU_EEPROM_MADDR_8; |
| 107 | return true; | 107 | return true; |
| 108 | case IP_VERSION(13, 0, 12): | 108 | case IP_VERSION(13, 0, 12): |
| 109 | case IP_VERSION(15, 0, 8): | ||
| 109 | if (fru_addr) | 110 | if (fru_addr) |
| 110 | *fru_addr = FRU_EEPROM_MADDR_INV; | 111 | *fru_addr = FRU_EEPROM_MADDR_INV; |
| 111 | return true; | 112 | return true; |
| @@ -403,6 +403,50 @@ void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, · 402 unmodified lines | |||
| 403 | drm_dev_exit(idx); | 403 | drm_dev_exit(idx); |
| 404 | } | 404 | } |
| 405 | 405 | ||
| 406 | /** | ||
| 407 | * amdgpu_gart_map_gfx9_mqd - map mqd and ctrl_stack dma_addresses into GART entries | ||
| 408 | * | ||
| 409 | * @adev: amdgpu_device pointer | ||
| 410 | * @offset: offset into the GPU's gart aperture | ||
| 411 | * @pages: number of pages to bind | ||
| 412 | * @dma_addr: DMA addresses of pages | ||
| 413 | * @flags: page table entry flags | ||
| 414 | * | ||
| 415 | * Map the MQD and control stack addresses into GART entries with the correct | ||
| 416 | * memory types on gfxv9. The MQD occupies the first 4KB and is followed by | ||
| 417 | * the control stack. The MQD uses UC (uncached) memory, while the control stack | ||
| 418 | * uses NC (non-coherent) memory. | ||
| 419 | */ | ||
| 420 | void amdgpu_gart_map_gfx9_mqd(struct amdgpu_device *adev, uint64_t offset, | ||
| 421 | int pages, dma_addr_t *dma_addr, uint64_t flags) | ||
| 422 | { | ||
| 423 | uint64_t page_base; | ||
| 424 | unsigned int i, j, t; | ||
| 425 | int idx; | ||
| 426 | uint64_t ctrl_flags = AMDGPU_PTE_MTYPE_VG10(flags, AMDGPU_MTYPE_NC); | ||
| 427 | void *dst; | ||
| 428 | |||
| 429 | if (!adev->gart.ptr) | ||
| 430 | return; | ||
| 431 | |||
| 432 | if (!drm_dev_enter(adev_to_drm(adev), &idx)) | ||
| 433 | return; | ||
| 434 | |||
| 435 | t = offset / AMDGPU_GPU_PAGE_SIZE; | ||
| 436 | dst = adev->gart.ptr; | ||
| 437 | for (i = 0; i < pages; i++) { | ||
| 438 | page_base = dma_addr[i]; | ||
| 439 | for (j = 0; j < AMDGPU_GPU_PAGES_IN_CPU_PAGE; j++, t++) { | ||
| 440 | if ((i == 0) && (j == 0)) | ||
| 441 | amdgpu_gmc_set_pte_pde(adev, dst, t, page_base, flags); | ||
| 442 | else | ||
| 443 | amdgpu_gmc_set_pte_pde(adev, dst, t, page_base, ctrl_flags); | ||
| 444 | page_base += AMDGPU_GPU_PAGE_SIZE; | ||
| 445 | } | ||
| 446 | } | ||
| 447 | drm_dev_exit(idx); | ||
| 448 | } | ||
| 449 | |||
| 406 | /** | 450 | /** |
| 407 | * amdgpu_gart_bind - bind pages into the gart page table | 451 | * amdgpu_gart_bind - bind pages into the gart page table |
| 408 | * | 452 | * |
| @@ -62,6 +62,8 @@ void amdgpu_gart_unbind(struct amdgpu_device *adev, uint64_t offset, · 61 unmodified lines | |||
| 62 | void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset, | 62 | void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset, |
| 63 | int pages, dma_addr_t *dma_addr, uint64_t flags, | 63 | int pages, dma_addr_t *dma_addr, uint64_t flags, |
| 64 | void *dst); | 64 | void *dst); |
| 65 | void amdgpu_gart_map_gfx9_mqd(struct amdgpu_device *adev, uint64_t offset, | ||
| 66 | int pages, dma_addr_t *dma_addr, uint64_t flags); | ||
| 65 | void amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t offset, | 67 | void amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t offset, |
| 66 | int pages, dma_addr_t *dma_addr, uint64_t flags); | 68 | int pages, dma_addr_t *dma_addr, uint64_t flags); |
| 67 | void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, | 69 | void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, |
| @@ -30,6 +30,7 @@ · 29 unmodified lines | |||
| 30 | #include <linux/pagemap.h> | 30 | #include <linux/pagemap.h> |
| 31 | #include <linux/pci.h> | 31 | #include <linux/pci.h> |
| 32 | #include <linux/dma-buf.h> | 32 | #include <linux/dma-buf.h> |
| 33 | #include <linux/dma-fence-unwrap.h> | ||
| 33 | 34 | ||
| 34 | #include <drm/amdgpu_drm.h> | 35 | #include <drm/amdgpu_drm.h> |
| 35 | #include <drm/drm_drv.h> | 36 | #include <drm/drm_drv.h> |
| @@ -106,6 +107,7 @@ amdgpu_gem_update_timeline_node(struct drm_file *filp, · 70 unmodified lines | |||
| 106 | *chain = dma_fence_chain_alloc(); | 107 | *chain = dma_fence_chain_alloc(); |
| 107 | if (!*chain) { | 108 | if (!*chain) { |
| 108 | drm_syncobj_put(*syncobj); | 109 | drm_syncobj_put(*syncobj); |
| 110 | *syncobj = NULL; | ||
| 109 | return -ENOMEM; | 111 | return -ENOMEM; |
| 110 | } | 112 | } |
| 111 | 113 | ||
| @@ -258,18 +260,15 @@ static int amdgpu_gem_object_open(struct drm_gem_object *obj, · 146 unmodified lines | |||
| 258 | 260 | ||
| 259 | amdgpu_vm_bo_update_shared(abo); | 261 | amdgpu_vm_bo_update_shared(abo); |
| 260 | bo_va = amdgpu_vm_bo_find(vm, abo); | 262 | bo_va = amdgpu_vm_bo_find(vm, abo); |
| 261 | if (!bo_va) | 263 | if (!bo_va) { |
| 262 | bo_va = amdgpu_vm_bo_add(adev, vm, abo); | 264 | bo_va = amdgpu_vm_bo_add(adev, vm, abo); |
| 263 | else | 265 | r = amdgpu_evf_mgr_attach_fence(&fpriv->evf_mgr, abo); |
| 266 | if (r) | ||
| 267 | goto out_unlock; | ||
| 268 | } else { | ||
| 264 | ++bo_va->ref_count; | 269 | ++bo_va->ref_count; |
| 265 | |||
| 266 | /* attach gfx eviction fence */ | ||
| 267 | r = amdgpu_eviction_fence_attach(&fpriv->evf_mgr, abo); | ||
| 268 | if (r) { | ||
| 269 | DRM_DEBUG_DRIVER("Failed to attach eviction fence to BO\n"); | ||
| 270 | amdgpu_bo_unreserve(abo); | ||
| 271 | return r; | ||
| 272 | } | 270 | } |
| 271 | |||
| 273 | drm_exec_fini(&exec); | 272 | drm_exec_fini(&exec); |
| 274 | 273 | ||
| 275 | /* Validate and add eviction fence to DMABuf imports with dynamic | 274 | /* Validate and add eviction fence to DMABuf imports with dynamic |
| @@ -337,7 +336,7 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj, · 61 unmodified lines | |||
| 337 | } | 336 | } |
| 338 | 337 | ||
| 339 | if (!amdgpu_vm_is_bo_always_valid(vm, bo)) | 338 | if (!amdgpu_vm_is_bo_always_valid(vm, bo)) |
| 340 | amdgpu_eviction_fence_detach(&fpriv->evf_mgr, bo); | 339 | amdgpu_evf_mgr_detach_fence(&fpriv->evf_mgr, bo); |
| 341 | 340 | ||
| 342 | bo_va = amdgpu_vm_bo_find(vm, bo); | 341 | bo_va = amdgpu_vm_bo_find(vm, bo); |
| 343 | if (!bo_va || --bo_va->ref_count) | 342 | if (!bo_va || --bo_va->ref_count) |
| @@ -744,11 +743,10 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 400 unmodified lines | |||
| 744 | struct dma_fence *fence; | 743 | struct dma_fence *fence; |
| 745 | int r = 0; | 744 | int r = 0; |
| 746 | 745 | ||
| 747 | /* Always start from the VM's existing last update fence. */ | 746 | /* If the VM is not ready return only a stub. */ |
| 748 | fence = dma_fence_get(vm->last_update); | ||
| 749 | |||
| 750 | if (!amdgpu_vm_ready(vm)) | 747 | if (!amdgpu_vm_ready(vm)) |
| 751 | return fence; | 748 | return dma_fence_get_stub(); |
| 749 | |||
| 752 | 750 | ||
| 753 | /* | 751 | /* |
| 754 | * First clean up any freed mappings in the VM. | 752 | * First clean up any freed mappings in the VM. |
| @@ -757,7 +755,7 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 2 unmodified lines | |||
| 757 | * schedules GPU work. If nothing needs clearing, @fence can remain as | 755 | * schedules GPU work. If nothing needs clearing, @fence can remain as |
| 758 | * the original vm->last_update. | 756 | * the original vm->last_update. |
| 759 | */ | 757 | */ |
| 760 | r = amdgpu_vm_clear_freed(adev, vm, &fence); | 758 | r = amdgpu_vm_clear_freed(adev, vm, &vm->last_update); |
| 761 | if (r) | 759 | if (r) |
| 762 | goto error; | 760 | goto error; |
| 763 | 761 | ||
| @@ -774,47 +772,34 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 10 unmodified lines | |||
| 774 | if (r) | 772 | if (r) |
| 775 | goto error; | 773 | goto error; |
| 776 | 774 | ||
| 777 | /* | 775 | if ((operation == AMDGPU_VA_OP_MAP || |
| 778 | * Decide which fence best represents the last update: | 776 | operation == AMDGPU_VA_OP_REPLACE) && |
| 779 | * | 777 | !amdgpu_vm_is_bo_always_valid(vm, bo_va->base.bo)) { |
| 780 | * MAP/REPLACE: | 778 | |
| 781 | * - For always-valid mappings, use vm->last_update. | ||
| 782 | * - Otherwise, export bo_va->last_pt_update. | ||
| 783 | * | ||
| 784 | * UNMAP/CLEAR: | ||
| 785 | * Keep the fence returned by amdgpu_vm_clear_freed(). If no work was | ||
| 786 | * needed, it can remain as vm->last_pt_update. | ||
| 787 | * | ||
| 788 | * The VM and BO update fences are always initialized to a valid value. | ||
| 789 | * vm->last_update and bo_va->last_pt_update always start as valid fences. | ||
| 790 | * and are never expected to be NULL. | ||
| 791 | */ | ||
| 792 | switch (operation) { | ||
| 793 | case AMDGPU_VA_OP_MAP: | ||
| 794 | case AMDGPU_VA_OP_REPLACE: | ||
| 795 | /* | 779 | /* |
| 796 | * For MAP/REPLACE, return the page table update fence for the | 780 | * For MAP/REPLACE of non per-VM BOs we need to sync to both the |
| 797 | * mapping we just modified. bo_va is expected to be valid here. | 781 | * bo_va->last_pt_update and vm->last_update or otherwise we |
| 782 | * potentially miss the PDE updates. | ||
| 798 | */ | 783 | */ |
| 799 | dma_fence_put(fence); | 784 | fence = dma_fence_unwrap_merge(vm->last_update, |
| 800 | 785 | bo_va->last_pt_update); | |
| 801 | if (amdgpu_vm_is_bo_always_valid(vm, bo_va->base.bo)) | 786 | if (!fence) { |
| 802 | fence = dma_fence_get(vm->last_update); | 787 | /* As fallback in OOM situations */ |
| 803 | else | 788 | dma_fence_wait(vm->last_update, false); |
| 804 | fence = dma_fence_get(bo_va->last_pt_update); | 789 | dma_fence_wait(bo_va->last_pt_update, false); |
| 805 | break; | 790 | fence = dma_fence_get_stub(); |
| 806 | case AMDGPU_VA_OP_UNMAP: | 791 | } |
| 807 | case AMDGPU_VA_OP_CLEAR: | 792 | } else { |
| 808 | default: | 793 | fence = dma_fence_get(vm->last_update); |
| 809 | /* keep @fence as returned by amdgpu_vm_clear_freed() */ | ||
| 810 | break; | ||
| 811 | } | 794 | } |
| 812 | 795 | ||
| 796 | return fence; | ||
| 797 | |||
| 813 | error: | 798 | error: |
| 814 | if (r && r != -ERESTARTSYS) | 799 | if (r && r != -ERESTARTSYS) |
| 815 | DRM_ERROR("Couldn't update BO_VA (%d)\n", r); | 800 | DRM_ERROR("Couldn't update BO_VA (%d)\n", r); |
| 816 | 801 | ||
| 817 | return fence; | 802 | return dma_fence_get(vm->last_update); |
| 818 | } | 803 | } |
| 819 | 804 | ||
| 820 | int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, | 805 | int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, |
| @@ -835,7 +820,6 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 14 unmodified lines | |||
| 835 | struct amdgpu_bo_va *bo_va; | 820 | struct amdgpu_bo_va *bo_va; |
| 836 | struct drm_syncobj *timeline_syncobj = NULL; | 821 | struct drm_syncobj *timeline_syncobj = NULL; |
| 837 | struct dma_fence_chain *timeline_chain = NULL; | 822 | struct dma_fence_chain *timeline_chain = NULL; |
| 838 | struct dma_fence *fence; | ||
| 839 | struct drm_exec exec; | 823 | struct drm_exec exec; |
| 840 | uint64_t vm_size; | 824 | uint64_t vm_size; |
| 841 | int r = 0; | 825 | int r = 0; |
| @@ -887,6 +871,10 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 45 unmodified lines | |||
| 887 | return -EINVAL; | 871 | return -EINVAL; |
| 888 | } | 872 | } |
| 889 | 873 | ||
| 874 | if (args->flags & AMDGPU_VM_DELAY_UPDATE && | ||
| 875 | args->vm_timeline_syncobj_out) | ||
| 876 | return -EINVAL; | ||
| 877 | |||
| 890 | if ((args->operation != AMDGPU_VA_OP_CLEAR) && | 878 | if ((args->operation != AMDGPU_VA_OP_CLEAR) && |
| 891 | !(args->flags & AMDGPU_VM_PAGE_PRT)) { | 879 | !(args->flags & AMDGPU_VM_PAGE_PRT)) { |
| 892 | gobj = drm_gem_object_lookup(filp, args->handle); | 880 | gobj = drm_gem_object_lookup(filp, args->handle); |
| @@ -976,11 +964,13 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 83 unmodified lines | |||
| 976 | * that represents the last relevant update for this mapping. This | 964 | * that represents the last relevant update for this mapping. This |
| 977 | * fence can then be exported to the user-visible VM timeline. | 965 | * fence can then be exported to the user-visible VM timeline. |
| 978 | */ | 966 | */ |
| 979 | if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm) { | 967 | if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && |
| 968 | (!adev->debug_vm || timeline_syncobj)) { | ||
| 969 | struct dma_fence *fence; | ||
| 970 | |||
| 980 | fence = amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va, | 971 | fence = amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va, |
| 981 | args->operation); | 972 | args->operation); |
| 982 | 973 | if (timeline_syncobj) { | |
| 983 | if (timeline_syncobj && fence) { | ||
| 984 | if (!args->vm_timeline_point) { | 974 | if (!args->vm_timeline_point) { |
| 985 | /* Replace the existing fence when no point is given. */ | 975 | /* Replace the existing fence when no point is given. */ |
| 986 | drm_syncobj_replace_fence(timeline_syncobj, | 976 | drm_syncobj_replace_fence(timeline_syncobj, |
| @@ -991,6 +981,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 4 unmodified lines | |||
| 991 | timeline_chain, | 981 | timeline_chain, |
| 992 | fence, | 982 | fence, |
| 993 | args->vm_timeline_point); | 983 | args->vm_timeline_point); |
| 984 | timeline_chain = NULL; | ||
| 994 | } | 985 | } |
| 995 | } | 986 | } |
| 996 | dma_fence_put(fence); | 987 | dma_fence_put(fence); |
| @@ -998,6 +989,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 1 unmodified line | |||
| 998 | } | 989 | } |
| 999 | 990 | ||
| 1000 | error: | 991 | error: |
| 992 | dma_fence_chain_free(timeline_chain); | ||
| 993 | if (timeline_syncobj) | ||
| 994 | drm_syncobj_put(timeline_syncobj); | ||
| 1001 | drm_exec_fini(&exec); | 995 | drm_exec_fini(&exec); |
| 1002 | error_put_gobj: | 996 | error_put_gobj: |
| 1003 | drm_gem_object_put(gobj); | 997 | drm_gem_object_put(gobj); |
| @@ -463,6 +463,7 @@ struct amdgpu_gfx { · 462 unmodified lines | |||
| 463 | struct amdgpu_irq_src cp_ecc_error_irq; | 463 | struct amdgpu_irq_src cp_ecc_error_irq; |
| 464 | struct amdgpu_irq_src sq_irq; | 464 | struct amdgpu_irq_src sq_irq; |
| 465 | struct amdgpu_irq_src rlc_gc_fed_irq; | 465 | struct amdgpu_irq_src rlc_gc_fed_irq; |
| 466 | struct amdgpu_irq_src rlc_poison_irq; | ||
| 466 | struct sq_work sq_work; | 467 | struct sq_work sq_work; |
| 467 | 468 | ||
| 468 | /* gfx status */ | 469 | /* gfx status */ |
| @@ -34,6 +34,7 @@ · 33 unmodified lines | |||
| 34 | #include "amdgpu_ras.h" | 34 | #include "amdgpu_ras.h" |
| 35 | #include "amdgpu_reset.h" | 35 | #include "amdgpu_reset.h" |
| 36 | #include "amdgpu_xgmi.h" | 36 | #include "amdgpu_xgmi.h" |
| 37 | #include "amdgpu_atomfirmware.h" | ||
| 37 | 38 | ||
| 38 | #include <drm/drm_drv.h> | 39 | #include <drm/drm_drv.h> |
| 39 | #include <drm/ttm/ttm_tt.h> | 40 | #include <drm/ttm/ttm_tt.h> |
| @@ -1374,18 +1375,18 @@ int amdgpu_gmc_get_nps_memranges(struct amdgpu_device *adev, · 1334 unmodified lines | |||
| 1374 | struct amdgpu_mem_partition_info *mem_ranges, | 1375 | struct amdgpu_mem_partition_info *mem_ranges, |
| 1375 | uint8_t *exp_ranges) | 1376 | uint8_t *exp_ranges) |
| 1376 | { | 1377 | { |
| 1377 | struct amdgpu_gmc_memrange *ranges; | 1378 | struct amdgpu_gmc_memrange ranges[AMDGPU_MAX_MEM_RANGES]; |
| 1378 | int range_cnt, ret, i, j; | 1379 | int range_cnt, ret, i, j; |
| 1379 | uint32_t nps_type; | 1380 | uint32_t nps_type; |
| 1380 | bool refresh; | 1381 | bool refresh; |
| 1381 | 1382 | ||
| 1382 | if (!mem_ranges || !exp_ranges) | 1383 | if (!mem_ranges || !exp_ranges) |
| 1383 | return -EINVAL; | 1384 | return -EINVAL; |
| 1384 | 1385 | range_cnt = AMDGPU_MAX_MEM_RANGES; | |
| 1385 | refresh = (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) && | 1386 | refresh = (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) && |
| 1386 | (adev->gmc.reset_flags & AMDGPU_GMC_INIT_RESET_NPS); | 1387 | (adev->gmc.reset_flags & AMDGPU_GMC_INIT_RESET_NPS); |
| 1387 | ret = amdgpu_discovery_get_nps_info(adev, &nps_type, &ranges, | 1388 | ret = amdgpu_discovery_get_nps_info(adev, &nps_type, ranges, &range_cnt, |
| 1388 | &range_cnt, refresh); | 1389 | refresh); |
| 1389 | 1390 | ||
| 1390 | if (ret) | 1391 | if (ret) |
| 1391 | return ret; | 1392 | return ret; |
| @@ -1446,8 +1447,6 @@ int amdgpu_gmc_get_nps_memranges(struct amdgpu_device *adev, · 54 unmodified lines | |||
| 1446 | if (!*exp_ranges) | 1447 | if (!*exp_ranges) |
| 1447 | *exp_ranges = range_cnt; | 1448 | *exp_ranges = range_cnt; |
| 1448 | err: | 1449 | err: |
| 1449 | kvfree(ranges); | ||
| 1450 | |||
| 1451 | return ret; | 1450 | return ret; |
| 1452 | } | 1451 | } |
| 1453 | 1452 | ||
| @@ -1749,3 +1748,31 @@ int amdgpu_gmc_init_mem_ranges(struct amdgpu_device *adev) · 295 unmodified lines | |||
| 1749 | 1748 | ||
| 1750 | return 0; | 1749 | return 0; |
| 1751 | } | 1750 | } |
| 1751 | |||
| 1752 | int amdgpu_gmc_get_vram_info(struct amdgpu_device *adev, | ||
| 1753 | int *vram_width, int *vram_type, int *vram_vendor) | ||
| 1754 | { | ||
| 1755 | int ret = 0; | ||
| 1756 | |||
| 1757 | if (adev->flags & AMD_IS_APU) | ||
| 1758 | return amdgpu_atomfirmware_get_integrated_system_info(adev, | ||
| 1759 | vram_width, vram_type, vram_vendor); | ||
| 1760 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | ||
| 1761 | case IP_VERSION(12, 0, 0): | ||
| 1762 | case IP_VERSION(12, 0, 1): | ||
| 1763 | return amdgpu_atomfirmware_get_umc_info(adev, | ||
| 1764 | vram_width, vram_type, vram_vendor); | ||
| 1765 | case IP_VERSION(9, 5, 0): | ||
| 1766 | case IP_VERSION(9, 4, 4): | ||
| 1767 | case IP_VERSION(9, 4, 3): | ||
| 1768 | ret = amdgpu_atomfirmware_get_umc_info(adev, | ||
| 1769 | vram_width, vram_type, vram_vendor); | ||
| 1770 | if (vram_width && !ret) | ||
| 1771 | *vram_width *= hweight32(adev->aid_mask); | ||
| 1772 | return ret; | ||
| 1773 | default: | ||
| 1774 | return amdgpu_atomfirmware_get_vram_info(adev, | ||
| 1775 | vram_width, vram_type, vram_vendor); | ||
| 1776 | } | ||
| 1777 | return 0; | ||
| 1778 | } | ||
| @@ -482,4 +482,6 @@ amdgpu_gmc_query_memory_partition(struct amdgpu_device *adev); · 481 unmodified lines | |||
| 482 | int amdgpu_gmc_init_mem_ranges(struct amdgpu_device *adev); | 482 | int amdgpu_gmc_init_mem_ranges(struct amdgpu_device *adev); |
| 483 | void amdgpu_gmc_init_sw_mem_ranges(struct amdgpu_device *adev, | 483 | void amdgpu_gmc_init_sw_mem_ranges(struct amdgpu_device *adev, |
| 484 | struct amdgpu_mem_partition_info *mem_ranges); | 484 | struct amdgpu_mem_partition_info *mem_ranges); |
| 485 | int amdgpu_gmc_get_vram_info(struct amdgpu_device *adev, | ||
| 486 | int *vram_width, int *vram_type, int *vram_vendor); | ||
| 485 | #endif | 487 | #endif |
| @@ -35,10 +35,13 @@ · 34 unmodified lines | |||
| 35 | * PASIDs are global address space identifiers that can be shared | 35 | * PASIDs are global address space identifiers that can be shared |
| 36 | * between the GPU, an IOMMU and the driver. VMs on different devices | 36 | * between the GPU, an IOMMU and the driver. VMs on different devices |
| 37 | * may use the same PASID if they share the same address | 37 | * may use the same PASID if they share the same address |
| 38 | * space. Therefore PASIDs are allocated using a global IDA. VMs are | 38 | * space. Therefore PASIDs are allocated using IDR cyclic allocator |
| 39 | * looked up from the PASID per amdgpu_device. | 39 | * (similar to kernel PID allocation) which naturally delays reuse. |
| 40 | * VMs are looked up from the PASID per amdgpu_device. | ||
| 40 | */ | 41 | */ |
| 41 | static DEFINE_IDA(amdgpu_pasid_ida); | 42 | |
| 43 | static DEFINE_IDR(amdgpu_pasid_idr); | ||
| 44 | static DEFINE_SPINLOCK(amdgpu_pasid_idr_lock); | ||
| 42 | 45 | ||
| 43 | /* Helper to free pasid from a fence callback */ | 46 | /* Helper to free pasid from a fence callback */ |
| 44 | struct amdgpu_pasid_cb { | 47 | struct amdgpu_pasid_cb { |
| @@ -50,8 +53,8 @@ struct amdgpu_pasid_cb { · 5 unmodified lines | |||
| 50 | * amdgpu_pasid_alloc - Allocate a PASID | 53 | * amdgpu_pasid_alloc - Allocate a PASID |
| 51 | * @bits: Maximum width of the PASID in bits, must be at least 1 | 54 | * @bits: Maximum width of the PASID in bits, must be at least 1 |
| 52 | * | 55 | * |
| 53 | * Allocates a PASID of the given width while keeping smaller PASIDs | 56 | * Uses kernel's IDR cyclic allocator (same as PID allocation). |
| 54 | * available if possible. | 57 | * Allocates sequentially with automatic wrap-around. |
| 55 | * | 58 | * |
| 56 | * Returns a positive integer on success. Returns %-EINVAL if bits==0. | 59 | * Returns a positive integer on success. Returns %-EINVAL if bits==0. |
| 57 | * Returns %-ENOSPC if no PASID was available. Returns %-ENOMEM on | 60 | * Returns %-ENOSPC if no PASID was available. Returns %-ENOMEM on |
| @@ -59,14 +62,18 @@ struct amdgpu_pasid_cb { · 1 unmodified line | |||
| 59 | */ | 62 | */ |
| 60 | int amdgpu_pasid_alloc(unsigned int bits) | 63 | int amdgpu_pasid_alloc(unsigned int bits) |
| 61 | { | 64 | { |
| 62 | int pasid = -EINVAL; | 65 | int pasid; |
| 63 | 66 | ||
| 64 | for (bits = min(bits, 31U); bits > 0; bits--) { | 67 | if (bits == 0) |
| 65 | pasid = ida_alloc_range(&amdgpu_pasid_ida, 1U << (bits - 1), | 68 | return -EINVAL; |
| 66 | (1U << bits) - 1, GFP_KERNEL); | 69 | |
| 67 | if (pasid != -ENOSPC) | 70 | spin_lock(&amdgpu_pasid_idr_lock); |
| 68 | break; | 71 | /* TODO: Need to replace the idr with an xarry, and then |
| 69 | } | 72 | * handle the internal locking with ATOMIC safe paths. |
| 73 | */ | ||
| 74 | pasid = idr_alloc_cyclic(&amdgpu_pasid_idr, NULL, 1, | ||
| 75 | 1U << bits, GFP_ATOMIC); | ||
| 76 | spin_unlock(&amdgpu_pasid_idr_lock); | ||
| 70 | 77 | ||
| 71 | if (pasid >= 0) | 78 | if (pasid >= 0) |
| 72 | trace_amdgpu_pasid_allocated(pasid); | 79 | trace_amdgpu_pasid_allocated(pasid); |
| @@ -81,7 +88,10 @@ int amdgpu_pasid_alloc(unsigned int bits) · 8 unmodified lines | |||
| 81 | void amdgpu_pasid_free(u32 pasid) | 88 | void amdgpu_pasid_free(u32 pasid) |
| 82 | { | 89 | { |
| 83 | trace_amdgpu_pasid_freed(pasid); | 90 | trace_amdgpu_pasid_freed(pasid); |
| 84 | ida_free(&amdgpu_pasid_ida, pasid); | 91 | |
| 92 | spin_lock(&amdgpu_pasid_idr_lock); | ||
| 93 | idr_remove(&amdgpu_pasid_idr, pasid); | ||
| 94 | spin_unlock(&amdgpu_pasid_idr_lock); | ||
| 85 | } | 95 | } |
| 86 | 96 | ||
| 87 | static void amdgpu_pasid_free_cb(struct dma_fence *fence, | 97 | static void amdgpu_pasid_free_cb(struct dma_fence *fence, |
| @@ -616,3 +626,15 @@ void amdgpu_vmid_mgr_fini(struct amdgpu_device *adev) · 528 unmodified lines | |||
| 616 | } | 626 | } |
| 617 | } | 627 | } |
| 618 | } | 628 | } |
| 629 | |||
| 630 | /** | ||
| 631 | * amdgpu_pasid_mgr_cleanup - cleanup PASID manager | ||
| 632 | * | ||
| 633 | * Cleanup the IDR allocator. | ||
| 634 | */ | ||
| 635 | void amdgpu_pasid_mgr_cleanup(void) | ||
| 636 | { | ||
| 637 | spin_lock(&amdgpu_pasid_idr_lock); | ||
| 638 | idr_destroy(&amdgpu_pasid_idr); | ||
| 639 | spin_unlock(&amdgpu_pasid_idr_lock); | ||
| 640 | } | ||
| @@ -74,6 +74,7 @@ int amdgpu_pasid_alloc(unsigned int bits); · 73 unmodified lines | |||
| 74 | void amdgpu_pasid_free(u32 pasid); | 74 | void amdgpu_pasid_free(u32 pasid); |
| 75 | void amdgpu_pasid_free_delayed(struct dma_resv *resv, | 75 | void amdgpu_pasid_free_delayed(struct dma_resv *resv, |
| 76 | u32 pasid); | 76 | u32 pasid); |
| 77 | void amdgpu_pasid_mgr_cleanup(void); | ||
| 77 | 78 | ||
| 78 | bool amdgpu_vmid_had_gpu_reset(struct amdgpu_device *adev, | 79 | bool amdgpu_vmid_had_gpu_reset(struct amdgpu_device *adev, |
| 79 | struct amdgpu_vmid *id); | 80 | struct amdgpu_vmid *id); |
| @@ -1522,10 +1522,7 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · 1521 unmodified lines | |||
| 1522 | "Failed to init usermode queue manager (%d), use legacy workload submission only\n", | 1522 | "Failed to init usermode queue manager (%d), use legacy workload submission only\n", |
| 1523 | r); | 1523 | r); |
| 1524 | 1524 | ||
| 1525 | r = amdgpu_eviction_fence_init(&fpriv->evf_mgr); | 1525 | amdgpu_evf_mgr_init(&fpriv->evf_mgr); |
| 1526 | if (r) | ||
| 1527 | goto error_vm; | ||
| 1528 | |||
| 1529 | amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev); | 1526 | amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev); |
| 1530 | 1527 | ||
| 1531 | file_priv->driver_priv = fpriv; | 1528 | file_priv->driver_priv = fpriv; |
| @@ -63,14 +63,40 @@ struct amdgpu_mmhub_funcs { · 62 unmodified lines | |||
| 63 | uint64_t page_table_base); | 63 | uint64_t page_table_base); |
| 64 | void (*update_power_gating)(struct amdgpu_device *adev, | 64 | void (*update_power_gating)(struct amdgpu_device *adev, |
| 65 | bool enable); | 65 | bool enable); |
| 66 | int (*get_xgmi_info)(struct amdgpu_device *adev); | ||
| 67 | }; | ||
| 68 | |||
| 69 | struct amdgpu_mmhub_client_ids { | ||
| 70 | const char * const (*names)[2]; | ||
| 71 | unsigned int size; | ||
| 66 | }; | 72 | }; |
| 67 | 73 | ||
| 68 | struct amdgpu_mmhub { | 74 | struct amdgpu_mmhub { |
| 69 | struct ras_common_if *ras_if; | 75 | struct ras_common_if *ras_if; |
| 70 | const struct amdgpu_mmhub_funcs *funcs; | 76 | const struct amdgpu_mmhub_funcs *funcs; |
| 71 | struct amdgpu_mmhub_ras *ras; | 77 | struct amdgpu_mmhub_ras *ras; |
| 78 | struct amdgpu_mmhub_client_ids client_ids; | ||
| 72 | }; | 79 | }; |
| 73 | 80 | ||
| 81 | static inline void | ||
| 82 | amdgpu_mmhub_init_client_info(struct amdgpu_mmhub *mmhub, | ||
| 83 | const char * const (*names)[2], | ||
| 84 | unsigned int size) | ||
| 85 | { | ||
| 86 | mmhub->client_ids.names = names; | ||
| 87 | mmhub->client_ids.size = size; | ||
| 88 | } | ||
| 89 | |||
| 90 | static inline const char * | ||
| 91 | amdgpu_mmhub_client_name(struct amdgpu_mmhub *mmhub, | ||
| 92 | u32 cid, bool is_write) | ||
| 93 | { | ||
| 94 | if (cid < mmhub->client_ids.size) | ||
| 95 | return mmhub->client_ids.names[cid][is_write]; | ||
| 96 | |||
| 97 | return NULL; | ||
| 98 | } | ||
| 99 | |||
| 74 | int amdgpu_mmhub_ras_sw_init(struct amdgpu_device *adev); | 100 | int amdgpu_mmhub_ras_sw_init(struct amdgpu_device *adev); |
| 75 | 101 | ||
| 76 | #endif | 102 | #endif |
| @@ -44,6 +44,7 @@ · 43 unmodified lines | |||
| 44 | #include <drm/display/drm_dp_mst_helper.h> | 44 | #include <drm/display/drm_dp_mst_helper.h> |
| 45 | #include "modules/inc/mod_freesync.h" | 45 | #include "modules/inc/mod_freesync.h" |
| 46 | #include "amdgpu_dm_irq_params.h" | 46 | #include "amdgpu_dm_irq_params.h" |
| 47 | #include "amdgpu_dm_ism.h" | ||
| 47 | 48 | ||
| 48 | struct amdgpu_bo; | 49 | struct amdgpu_bo; |
| 49 | struct amdgpu_device; | 50 | struct amdgpu_device; |
| @@ -486,6 +487,10 @@ struct amdgpu_crtc { · 436 unmodified lines | |||
| 486 | int deferred_flip_completion; | 487 | int deferred_flip_completion; |
| 487 | /* parameters access from DM IRQ handler */ | 488 | /* parameters access from DM IRQ handler */ |
| 488 | struct dm_irq_params dm_irq_params; | 489 | struct dm_irq_params dm_irq_params; |
| 490 | |||
| 491 | /* DM idle state manager */ | ||
| 492 | struct amdgpu_dm_ism ism; | ||
| 493 | |||
| 489 | /* pll sharing */ | 494 | /* pll sharing */ |
| 490 | struct amdgpu_atom_ss ss; | 495 | struct amdgpu_atom_ss ss; |
| 491 | bool ss_enabled; | 496 | bool ss_enabled; |
| @@ -1325,7 +1325,8 @@ void amdgpu_bo_release_notify(struct ttm_buffer_object *bo) · 1324 unmodified lines | |||
| 1325 | if (r) | 1325 | if (r) |
| 1326 | goto out; | 1326 | goto out; |
| 1327 | 1327 | ||
| 1328 | r = amdgpu_fill_buffer(&adev->mman.clear_entity, abo, 0, &bo->base._resv, | 1328 | r = amdgpu_fill_buffer(amdgpu_ttm_next_clear_entity(adev), |
| 1329 | abo, 0, &bo->base._resv, | ||
| 1329 | &fence, AMDGPU_KERNEL_JOB_ID_CLEAR_ON_RELEASE); | 1330 | &fence, AMDGPU_KERNEL_JOB_ID_CLEAR_ON_RELEASE); |
| 1330 | if (WARN_ON(r)) | 1331 | if (WARN_ON(r)) |
| 1331 | goto out; | 1332 | goto out; |
| @@ -2250,6 +2250,7 @@ static bool amdgpu_ras_aca_is_supported(struct amdgpu_device *adev) · 2249 unmodified lines | |||
| 2250 | case IP_VERSION(13, 0, 6): | 2250 | case IP_VERSION(13, 0, 6): |
| 2251 | case IP_VERSION(13, 0, 12): | 2251 | case IP_VERSION(13, 0, 12): |
| 2252 | case IP_VERSION(13, 0, 14): | 2252 | case IP_VERSION(13, 0, 14): |
| 2253 | case IP_VERSION(13, 0, 15): | ||
| 2253 | ret = true; | 2254 | ret = true; |
| 2254 | break; | 2255 | break; |
| 2255 | default: | 2256 | default: |
| @@ -4000,6 +4001,7 @@ static bool amdgpu_ras_asic_supported(struct amdgpu_device *adev) · 1744 unmodified lines | |||
| 4000 | case IP_VERSION(13, 0, 6): | 4001 | case IP_VERSION(13, 0, 6): |
| 4001 | case IP_VERSION(13, 0, 12): | 4002 | case IP_VERSION(13, 0, 12): |
| 4002 | case IP_VERSION(13, 0, 14): | 4003 | case IP_VERSION(13, 0, 14): |
| 4004 | case IP_VERSION(13, 0, 15): | ||
| 4003 | return true; | 4005 | return true; |
| 4004 | default: | 4006 | default: |
| 4005 | return false; | 4007 | return false; |
| @@ -4013,6 +4015,7 @@ static bool amdgpu_ras_asic_supported(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 4013 | case IP_VERSION(13, 0, 10): | 4015 | case IP_VERSION(13, 0, 10): |
| 4014 | case IP_VERSION(13, 0, 12): | 4016 | case IP_VERSION(13, 0, 12): |
| 4015 | case IP_VERSION(13, 0, 14): | 4017 | case IP_VERSION(13, 0, 14): |
| 4018 | case IP_VERSION(13, 0, 15): | ||
| 4016 | case IP_VERSION(14, 0, 3): | 4019 | case IP_VERSION(14, 0, 3): |
| 4017 | return true; | 4020 | return true; |
| 4018 | default: | 4021 | default: |
| @@ -4182,7 +4185,8 @@ init_ras_enabled_flag: · 163 unmodified lines | |||
| 4182 | adev->aca.is_enabled = | 4185 | adev->aca.is_enabled = |
| 4183 | (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) || | 4186 | (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) || |
| 4184 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) || | 4187 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) || |
| 4185 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)); | 4188 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14) || |
| 4189 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 15)); | ||
| 4186 | } | 4190 | } |
| 4187 | 4191 | ||
| 4188 | /* bad page feature is not applicable to specific app platform */ | 4192 | /* bad page feature is not applicable to specific app platform */ |
| @@ -4270,6 +4274,7 @@ static void amdgpu_ras_init_reserved_vram_size(struct amdgpu_device *adev) · 81 unmodified lines | |||
| 4270 | case IP_VERSION(13, 0, 2): | 4274 | case IP_VERSION(13, 0, 2): |
| 4271 | case IP_VERSION(13, 0, 6): | 4275 | case IP_VERSION(13, 0, 6): |
| 4272 | case IP_VERSION(13, 0, 12): | 4276 | case IP_VERSION(13, 0, 12): |
| 4277 | case IP_VERSION(13, 0, 15): | ||
| 4273 | con->reserved_pages_in_bytes = AMDGPU_RAS_RESERVED_VRAM_SIZE_DEFAULT; | 4278 | con->reserved_pages_in_bytes = AMDGPU_RAS_RESERVED_VRAM_SIZE_DEFAULT; |
| 4274 | break; | 4279 | break; |
| 4275 | case IP_VERSION(13, 0, 14): | 4280 | case IP_VERSION(13, 0, 14): |
| @@ -1558,6 +1558,8 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 1557 unmodified lines | |||
| 1558 | unsigned char buf[RAS_TABLE_HEADER_SIZE] = { 0 }; | 1558 | unsigned char buf[RAS_TABLE_HEADER_SIZE] = { 0 }; |
| 1559 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; | 1559 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; |
| 1560 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | 1560 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); |
| 1561 | int dev_var = adev->pdev->device & 0xF; | ||
| 1562 | uint32_t vram_type = adev->gmc.vram_type; | ||
| 1561 | int res; | 1563 | int res; |
| 1562 | 1564 | ||
| 1563 | if (amdgpu_ras_smu_eeprom_supported(adev)) | 1565 | if (amdgpu_ras_smu_eeprom_supported(adev)) |
| @@ -1597,6 +1599,12 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 33 unmodified lines | |||
| 1597 | return amdgpu_ras_eeprom_reset_table(control); | 1599 | return amdgpu_ras_eeprom_reset_table(control); |
| 1598 | } | 1600 | } |
| 1599 | 1601 | ||
| 1602 | if (!(adev->flags & AMD_IS_APU) && (dev_var == 0x5) && | ||
| 1603 | (vram_type == AMDGPU_VRAM_TYPE_HBM3E) && | ||
| 1604 | (hdr->version < RAS_TABLE_VER_V3)) { | ||
| 1605 | return amdgpu_ras_eeprom_reset_table(control); | ||
| 1606 | } | ||
| 1607 | |||
| 1600 | switch (hdr->version) { | 1608 | switch (hdr->version) { |
| 1601 | case RAS_TABLE_VER_V2_1: | 1609 | case RAS_TABLE_VER_V2_1: |
| 1602 | case RAS_TABLE_VER_V3: | 1610 | case RAS_TABLE_VER_V3: |
| @@ -314,7 +314,6 @@ uint64_t amdgpu_reg_get_smn_base64(struct amdgpu_device *adev, · 313 unmodified lines | |||
| 314 | "SMN base address query not supported for this device\n"); | 314 | "SMN base address query not supported for this device\n"); |
| 315 | return 0; | 315 | return 0; |
| 316 | } | 316 | } |
| 317 | return 0; | ||
| 318 | } | 317 | } |
| 319 | return adev->reg.smn.get_smn_base(adev, block, die_inst); | 318 | return adev->reg.smn.get_smn_base(adev, block, die_inst); |
| 320 | } | 319 | } |
| @@ -103,10 +103,8 @@ int amdgpu_sched_ioctl(struct drm_device *dev, void *data, · 102 unmodified lines | |||
| 103 | return -EINVAL; | 103 | return -EINVAL; |
| 104 | } | 104 | } |
| 105 | 105 | ||
| 106 | if (!amdgpu_ctx_priority_is_valid(args->in.priority)) { | 106 | if (!amdgpu_ctx_priority_is_valid(args->in.priority)) |
| 107 | WARN(1, "Invalid context priority %d\n", args->in.priority); | ||
| 108 | return -EINVAL; | 107 | return -EINVAL; |
| 109 | } | ||
| 110 | 108 | ||
| 111 | switch (args->in.op) { | 109 | switch (args->in.op) { |
| 112 | case AMDGPU_SCHED_OP_PROCESS_PRIORITY_OVERRIDE: | 110 | case AMDGPU_SCHED_OP_PROCESS_PRIORITY_OVERRIDE: |
| @@ -387,9 +387,11 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo, · 386 unmodified lines | |||
| 387 | { | 387 | { |
| 388 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->bdev); | 388 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->bdev); |
| 389 | struct amdgpu_bo *abo = ttm_to_amdgpu_bo(bo); | 389 | struct amdgpu_bo *abo = ttm_to_amdgpu_bo(bo); |
| 390 | struct amdgpu_ttm_buffer_entity *entity; | ||
| 390 | struct amdgpu_copy_mem src, dst; | 391 | struct amdgpu_copy_mem src, dst; |
| 391 | struct dma_fence *fence = NULL; | 392 | struct dma_fence *fence = NULL; |
| 392 | int r; | 393 | int r; |
| 394 | u32 e; | ||
| 393 | 395 | ||
| 394 | src.bo = bo; | 396 | src.bo = bo; |
| 395 | dst.bo = bo; | 397 | dst.bo = bo; |
| @@ -398,8 +400,12 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo, · 2 unmodified lines | |||
| 398 | src.offset = 0; | 400 | src.offset = 0; |
| 399 | dst.offset = 0; | 401 | dst.offset = 0; |
| 400 | 402 | ||
| 403 | e = atomic_inc_return(&adev->mman.next_move_entity) % | ||
| 404 | adev->mman.num_move_entities; | ||
| 405 | entity = &adev->mman.move_entities[e]; | ||
| 406 | |||
| 401 | r = amdgpu_ttm_copy_mem_to_mem(adev, | 407 | r = amdgpu_ttm_copy_mem_to_mem(adev, |
| 402 | &adev->mman.move_entity, | 408 | entity, |
| 403 | &src, &dst, | 409 | &src, &dst, |
| 404 | new_mem->size, | 410 | new_mem->size, |
| 405 | amdgpu_bo_encrypted(abo), | 411 | amdgpu_bo_encrypted(abo), |
| @@ -411,9 +417,7 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo, · 5 unmodified lines | |||
| 411 | if (old_mem->mem_type == TTM_PL_VRAM && | 417 | if (old_mem->mem_type == TTM_PL_VRAM && |
| 412 | (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) { | 418 | (abo->flags & AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE)) { |
| 413 | struct dma_fence *wipe_fence = NULL; | 419 | struct dma_fence *wipe_fence = NULL; |
| 414 | 420 | r = amdgpu_fill_buffer(entity, abo, 0, NULL, &wipe_fence, | |
| 415 | r = amdgpu_fill_buffer(&adev->mman.move_entity, | ||
| 416 | abo, 0, NULL, &wipe_fence, | ||
| 417 | AMDGPU_KERNEL_JOB_ID_MOVE_BLIT); | 421 | AMDGPU_KERNEL_JOB_ID_MOVE_BLIT); |
| 418 | if (r) { | 422 | if (r) { |
| 419 | goto error; | 423 | goto error; |
| @@ -854,25 +858,15 @@ static void amdgpu_ttm_gart_bind_gfx9_mqd(struct amdgpu_device *adev, · 434 unmodified lines | |||
| 854 | int num_xcc = max(1U, adev->gfx.num_xcc_per_xcp); | 858 | int num_xcc = max(1U, adev->gfx.num_xcc_per_xcp); |
| 855 | uint64_t page_idx, pages_per_xcc; | 859 | uint64_t page_idx, pages_per_xcc; |
| 856 | int i; | 860 | int i; |
| 857 | uint64_t ctrl_flags = AMDGPU_PTE_MTYPE_VG10(flags, AMDGPU_MTYPE_NC); | ||
| 858 | 861 | ||
| 859 | pages_per_xcc = total_pages; | 862 | pages_per_xcc = total_pages; |
| 860 | do_div(pages_per_xcc, num_xcc); | 863 | do_div(pages_per_xcc, num_xcc); |
| 861 | 864 | ||
| 862 | for (i = 0, page_idx = 0; i < num_xcc; i++, page_idx += pages_per_xcc) { | 865 | for (i = 0, page_idx = 0; i < num_xcc; i++, page_idx += pages_per_xcc) { |
| 863 | /* MQD page: use default flags */ | 866 | amdgpu_gart_map_gfx9_mqd(adev, |
| 864 | amdgpu_gart_bind(adev, | ||
| 865 | gtt->offset + (page_idx << PAGE_SHIFT), | 867 | gtt->offset + (page_idx << PAGE_SHIFT), |
| 866 | 1, >t->ttm.dma_address[page_idx], flags); | 868 | pages_per_xcc, >t->ttm.dma_address[page_idx], |
| 867 | /* | 869 | flags); |
| 868 | * Ctrl pages - modify the memory type to NC (ctrl_flags) from | ||
| 869 | * the second page of the BO onward. | ||
| 870 | */ | ||
| 871 | amdgpu_gart_bind(adev, | ||
| 872 | gtt->offset + ((page_idx + 1) << PAGE_SHIFT), | ||
| 873 | pages_per_xcc - 1, | ||
| 874 | >t->ttm.dma_address[page_idx + 1], | ||
| 875 | ctrl_flags); | ||
| 876 | } | 870 | } |
| 877 | } | 871 | } |
| 878 | 872 | ||
| @@ -2345,8 +2339,9 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev) · 1466 unmodified lines | |||
| 2345 | void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) | 2339 | void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) |
| 2346 | { | 2340 | { |
| 2347 | struct ttm_resource_manager *man = ttm_manager_type(&adev->mman.bdev, TTM_PL_VRAM); | 2341 | struct ttm_resource_manager *man = ttm_manager_type(&adev->mman.bdev, TTM_PL_VRAM); |
| 2342 | u32 num_clear_entities, num_move_entities; | ||
| 2348 | uint64_t size; | 2343 | uint64_t size; |
| 2349 | int r; | 2344 | int r, i, j; |
| 2350 | 2345 | ||
| 2351 | if (!adev->mman.initialized || amdgpu_in_reset(adev) || | 2346 | if (!adev->mman.initialized || amdgpu_in_reset(adev) || |
| 2352 | adev->mman.buffer_funcs_enabled == enable || adev->gmc.is_app_apu) | 2347 | adev->mman.buffer_funcs_enabled == enable || adev->gmc.is_app_apu) |
| @@ -2361,6 +2356,8 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) · 8 unmodified lines | |||
| 2361 | return; | 2356 | return; |
| 2362 | } | 2357 | } |
| 2363 | 2358 | ||
| 2359 | num_clear_entities = 1; | ||
| 2360 | num_move_entities = 1; | ||
| 2364 | ring = adev->mman.buffer_funcs_ring; | 2361 | ring = adev->mman.buffer_funcs_ring; |
| 2365 | sched = &ring->sched; | 2362 | sched = &ring->sched; |
| 2366 | r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, | 2363 | r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, |
| @@ -2373,36 +2370,64 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) · 6 unmodified lines | |||
| 2373 | return; | 2370 | return; |
| 2374 | } | 2371 | } |
| 2375 | 2372 | ||
| 2376 | r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, | 2373 | adev->mman.clear_entities = kcalloc(num_clear_entities, |
| 2377 | &adev->mman.clear_entity, | 2374 | sizeof(struct amdgpu_ttm_buffer_entity), |
| 2378 | DRM_SCHED_PRIORITY_NORMAL, | 2375 | GFP_KERNEL); |
| 2379 | &sched, 1, 1); | 2376 | atomic_set(&adev->mman.next_clear_entity, 0); |
| 2380 | if (r < 0) { | 2377 | if (!adev->mman.clear_entities) |
| 2381 | dev_err(adev->dev, | ||
| 2382 | "Failed setting up TTM BO clear entity (%d)\n", r); | ||
| 2383 | goto error_free_default_entity; | 2378 | goto error_free_default_entity; |
| 2379 | |||
| 2380 | adev->mman.num_clear_entities = num_clear_entities; | ||
| 2381 | |||
| 2382 | for (i = 0; i < num_clear_entities; i++) { | ||
| 2383 | r = amdgpu_ttm_buffer_entity_init( | ||
| 2384 | &adev->mman.gtt_mgr, &adev->mman.clear_entities[i], | ||
| 2385 | DRM_SCHED_PRIORITY_NORMAL, &sched, 1, 1); | ||
| 2386 | |||
| 2387 | if (r < 0) { | ||
| 2388 | for (j = 0; j < i; j++) | ||
| 2389 | amdgpu_ttm_buffer_entity_fini( | ||
| 2390 | &adev->mman.gtt_mgr, &adev->mman.clear_entities[j]); | ||
| 2391 | kfree(adev->mman.clear_entities); | ||
| 2392 | adev->mman.num_clear_entities = 0; | ||
| 2393 | adev->mman.clear_entities = NULL; | ||
| 2394 | goto error_free_default_entity; | ||
| 2395 | } | ||
| 2384 | } | 2396 | } |
| 2385 | 2397 | ||
| 2386 | r = amdgpu_ttm_buffer_entity_init(&adev->mman.gtt_mgr, | 2398 | adev->mman.num_move_entities = num_move_entities; |
| 2387 | &adev->mman.move_entity, | 2399 | atomic_set(&adev->mman.next_move_entity, 0); |
| 2388 | DRM_SCHED_PRIORITY_NORMAL, | 2400 | for (i = 0; i < num_move_entities; i++) { |
| 2389 | &sched, 1, 2); | 2401 | r = amdgpu_ttm_buffer_entity_init( |
| 2390 | if (r < 0) { | 2402 | &adev->mman.gtt_mgr, |
| 2391 | dev_err(adev->dev, | 2403 | &adev->mman.move_entities[i], |
| 2392 | "Failed setting up TTM BO move entity (%d)\n", r); | 2404 | DRM_SCHED_PRIORITY_NORMAL, &sched, 1, 2); |
| 2393 | goto error_free_clear_entity; | 2405 | |
| 2406 | if (r < 0) { | ||
| 2407 | for (j = 0; j < i; j++) | ||
| 2408 | amdgpu_ttm_buffer_entity_fini( | ||
| 2409 | &adev->mman.gtt_mgr, &adev->mman.move_entities[j]); | ||
| 2410 | adev->mman.num_move_entities = 0; | ||
| 2411 | goto error_free_clear_entities; | ||
| 2412 | } | ||
| 2394 | } | 2413 | } |
| 2395 | } else { | 2414 | } else { |
| 2396 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | 2415 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, |
| 2397 | &adev->mman.default_entity); | 2416 | &adev->mman.default_entity); |
| 2398 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | 2417 | for (i = 0; i < adev->mman.num_clear_entities; i++) |
| 2399 | &adev->mman.clear_entity); | 2418 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, |
| 2400 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | 2419 | &adev->mman.clear_entities[i]); |
| 2401 | &adev->mman.move_entity); | 2420 | for (i = 0; i < adev->mman.num_move_entities; i++) |
| 2421 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | ||
| 2422 | &adev->mman.move_entities[i]); | ||
| 2402 | /* Drop all the old fences since re-creating the scheduler entities | 2423 | /* Drop all the old fences since re-creating the scheduler entities |
| 2403 | * will allocate new contexts. | 2424 | * will allocate new contexts. |
| 2404 | */ | 2425 | */ |
| 2405 | ttm_resource_manager_cleanup(man); | 2426 | ttm_resource_manager_cleanup(man); |
| 2427 | kfree(adev->mman.clear_entities); | ||
| 2428 | adev->mman.clear_entities = NULL; | ||
| 2429 | adev->mman.num_clear_entities = 0; | ||
| 2430 | adev->mman.num_move_entities = 0; | ||
| 2406 | } | 2431 | } |
| 2407 | 2432 | ||
| 2408 | /* this just adjusts TTM size idea, which sets lpfn to the correct value */ | 2433 | /* this just adjusts TTM size idea, which sets lpfn to the correct value */ |
| @@ -2415,9 +2440,13 @@ void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable) · 6 unmodified lines | |||
| 2415 | 2440 | ||
| 2416 | return; | 2441 | return; |
| 2417 | 2442 | ||
| 2418 | error_free_clear_entity: | 2443 | error_free_clear_entities: |
| 2419 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | 2444 | for (i = 0; i < adev->mman.num_clear_entities; i++) |
| 2420 | &adev->mman.clear_entity); | 2445 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, |
| 2446 | &adev->mman.clear_entities[i]); | ||
| 2447 | kfree(adev->mman.clear_entities); | ||
| 2448 | adev->mman.clear_entities = NULL; | ||
| 2449 | adev->mman.num_clear_entities = 0; | ||
| 2421 | error_free_default_entity: | 2450 | error_free_default_entity: |
| 2422 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, | 2451 | amdgpu_ttm_buffer_entity_fini(&adev->mman.gtt_mgr, |
| 2423 | &adev->mman.default_entity); | 2452 | &adev->mman.default_entity); |
| @@ -2567,8 +2596,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo, · 143 unmodified lines | |||
| 2567 | 2596 | ||
| 2568 | if (!fence) | 2597 | if (!fence) |
| 2569 | return -EINVAL; | 2598 | return -EINVAL; |
| 2570 | 2599 | entity = &adev->mman.clear_entities[0]; | |
| 2571 | entity = &adev->mman.clear_entity; | ||
| 2572 | *fence = dma_fence_get_stub(); | 2600 | *fence = dma_fence_get_stub(); |
| 2573 | 2601 | ||
| 2574 | amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor); | 2602 | amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &cursor); |
| @@ -2620,11 +2648,8 @@ int amdgpu_fill_buffer(struct amdgpu_ttm_buffer_entity *entity, · 45 unmodified lines | |||
| 2620 | struct amdgpu_res_cursor dst; | 2648 | struct amdgpu_res_cursor dst; |
| 2621 | int r; | 2649 | int r; |
| 2622 | 2650 | ||
| 2623 | if (!adev->mman.buffer_funcs_enabled) { | 2651 | if (!entity) |
| 2624 | dev_err(adev->dev, | ||
| 2625 | "Trying to clear memory with ring turned off.\n"); | ||
| 2626 | return -EINVAL; | 2652 | return -EINVAL; |
| 2627 | } | ||
| 2628 | 2653 | ||
| 2629 | amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &dst); | 2654 | amdgpu_res_first(bo->tbo.resource, 0, amdgpu_bo_size(bo), &dst); |
| 2630 | 2655 | ||
| @@ -2660,6 +2685,20 @@ error: · 29 unmodified lines | |||
| 2660 | return r; | 2685 | return r; |
| 2661 | } | 2686 | } |
| 2662 | 2687 | ||
| 2688 | struct amdgpu_ttm_buffer_entity * | ||
| 2689 | amdgpu_ttm_next_clear_entity(struct amdgpu_device *adev) | ||
| 2690 | { | ||
| 2691 | struct amdgpu_mman *mman = &adev->mman; | ||
| 2692 | u32 i; | ||
| 2693 | |||
| 2694 | if (mman->num_clear_entities == 0) | ||
| 2695 | return NULL; | ||
| 2696 | |||
| 2697 | i = atomic_inc_return(&mman->next_clear_entity) % | ||
| 2698 | mman->num_clear_entities; | ||
| 2699 | return &mman->clear_entities[i]; | ||
| 2700 | } | ||
| 2701 | |||
| 2663 | /** | 2702 | /** |
| 2664 | * amdgpu_ttm_evict_resources - evict memory buffers | 2703 | * amdgpu_ttm_evict_resources - evict memory buffers |
| 2665 | * @adev: amdgpu device object | 2704 | * @adev: amdgpu device object |
| @@ -72,8 +72,12 @@ struct amdgpu_mman { · 71 unmodified lines | |||
| 72 | 72 | ||
| 73 | /* @default_entity: for workarounds, has no gart windows */ | 73 | /* @default_entity: for workarounds, has no gart windows */ |
| 74 | struct amdgpu_ttm_buffer_entity default_entity; | 74 | struct amdgpu_ttm_buffer_entity default_entity; |
| 75 | struct amdgpu_ttm_buffer_entity clear_entity; | 75 | struct amdgpu_ttm_buffer_entity *clear_entities; |
| 76 | struct amdgpu_ttm_buffer_entity move_entity; | 76 | atomic_t next_clear_entity; |
| 77 | u32 num_clear_entities; | ||
| 78 | struct amdgpu_ttm_buffer_entity move_entities[TTM_NUM_MOVE_FENCES]; | ||
| 79 | atomic_t next_move_entity; | ||
| 80 | u32 num_move_entities; | ||
| 77 | 81 | ||
| 78 | struct amdgpu_vram_mgr vram_mgr; | 82 | struct amdgpu_vram_mgr vram_mgr; |
| 79 | struct amdgpu_gtt_mgr gtt_mgr; | 83 | struct amdgpu_gtt_mgr gtt_mgr; |
| @@ -191,6 +195,7 @@ int amdgpu_fill_buffer(struct amdgpu_ttm_buffer_entity *entity, · 111 unmodified lines | |||
| 191 | struct dma_resv *resv, | 195 | struct dma_resv *resv, |
| 192 | struct dma_fence **f, | 196 | struct dma_fence **f, |
| 193 | u64 k_job_id); | 197 | u64 k_job_id); |
| 198 | struct amdgpu_ttm_buffer_entity *amdgpu_ttm_next_clear_entity(struct amdgpu_device *adev); | ||
| 194 | 199 | ||
| 195 | int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo); | 200 | int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo); |
| 196 | void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo); | 201 | void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo); |
| @@ -1150,7 +1150,7 @@ int amdgpu_ucode_create_bo(struct amdgpu_device *adev) · 1149 unmodified lines | |||
| 1150 | if ((adev->firmware.load_type != AMDGPU_FW_LOAD_DIRECT) && | 1150 | if ((adev->firmware.load_type != AMDGPU_FW_LOAD_DIRECT) && |
| 1151 | (adev->firmware.load_type != AMDGPU_FW_LOAD_RLC_BACKDOOR_AUTO)) { | 1151 | (adev->firmware.load_type != AMDGPU_FW_LOAD_RLC_BACKDOOR_AUTO)) { |
| 1152 | amdgpu_bo_create_kernel(adev, adev->firmware.fw_size, PAGE_SIZE, | 1152 | amdgpu_bo_create_kernel(adev, adev->firmware.fw_size, PAGE_SIZE, |
| 1153 | (amdgpu_sriov_vf(adev) || adev->debug_use_vram_fw_buf) ? | 1153 | (amdgpu_sriov_vf(adev) || adev->debug_use_vram_fw_buf || adev->gmc.xgmi.connected_to_cpu) ? |
| 1154 | AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT, | 1154 | AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT, |
| 1155 | &adev->firmware.fw_buf, | 1155 | &adev->firmware.fw_buf, |
| 1156 | &adev->firmware.fw_buf_mc, | 1156 | &adev->firmware.fw_buf_mc, |
| @@ -398,6 +398,17 @@ int amdgpu_umc_process_ecc_irq(struct amdgpu_device *adev, · 397 unmodified lines | |||
| 398 | return 0; | 398 | return 0; |
| 399 | } | 399 | } |
| 400 | 400 | ||
| 401 | int amdgpu_umc_uniras_process_ecc_irq(struct amdgpu_device *adev, | ||
| 402 | struct amdgpu_irq_src *source, | ||
| 403 | struct amdgpu_iv_entry *entry) | ||
| 404 | { | ||
| 405 | struct ras_ih_info ih_info = {0}; | ||
| 406 | |||
| 407 | ih_info.block = RAS_BLOCK_ID__UMC; | ||
| 408 | amdgpu_ras_mgr_dispatch_interrupt(adev, &ih_info); | ||
| 409 | return 0; | ||
| 410 | } | ||
| 411 | |||
| 401 | int amdgpu_umc_fill_error_record(struct ras_err_data *err_data, | 412 | int amdgpu_umc_fill_error_record(struct ras_err_data *err_data, |
| 402 | uint64_t err_addr, | 413 | uint64_t err_addr, |
| 403 | uint64_t retired_page, | 414 | uint64_t retired_page, |
| @@ -161,6 +161,9 @@ int amdgpu_umc_pasid_poison_handler(struct amdgpu_device *adev, · 160 unmodified lines | |||
| 161 | int amdgpu_umc_process_ecc_irq(struct amdgpu_device *adev, | 161 | int amdgpu_umc_process_ecc_irq(struct amdgpu_device *adev, |
| 162 | struct amdgpu_irq_src *source, | 162 | struct amdgpu_irq_src *source, |
| 163 | struct amdgpu_iv_entry *entry); | 163 | struct amdgpu_iv_entry *entry); |
| 164 | int amdgpu_umc_uniras_process_ecc_irq(struct amdgpu_device *adev, | ||
| 165 | struct amdgpu_irq_src *source, | ||
| 166 | struct amdgpu_iv_entry *entry); | ||
| 164 | int amdgpu_umc_fill_error_record(struct ras_err_data *err_data, | 167 | int amdgpu_umc_fill_error_record(struct ras_err_data *err_data, |
| 165 | uint64_t err_addr, | 168 | uint64_t err_addr, |
| 166 | uint64_t retired_page, | 169 | uint64_t retired_page, |
| @@ -156,7 +156,7 @@ static void amdgpu_userq_hang_detect_work(struct work_struct *work) · 155 unmodified lines | |||
| 156 | struct dma_fence *fence; | 156 | struct dma_fence *fence; |
| 157 | struct amdgpu_userq_mgr *uq_mgr; | 157 | struct amdgpu_userq_mgr *uq_mgr; |
| 158 | 158 | ||
| 159 | if (!queue || !queue->userq_mgr) | 159 | if (!queue->userq_mgr) |
| 160 | return; | 160 | return; |
| 161 | 161 | ||
| 162 | uq_mgr = queue->userq_mgr; | 162 | uq_mgr = queue->userq_mgr; |
| @@ -472,17 +472,16 @@ void · 309 unmodified lines | |||
| 472 | amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *uq_mgr, | 472 | amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *uq_mgr, |
| 473 | struct amdgpu_eviction_fence_mgr *evf_mgr) | 473 | struct amdgpu_eviction_fence_mgr *evf_mgr) |
| 474 | { | 474 | { |
| 475 | struct amdgpu_eviction_fence *ev_fence; | 475 | struct dma_fence *ev_fence; |
| 476 | 476 | ||
| 477 | retry: | 477 | retry: |
| 478 | /* Flush any pending resume work to create ev_fence */ | 478 | /* Flush any pending resume work to create ev_fence */ |
| 479 | flush_delayed_work(&uq_mgr->resume_work); | 479 | flush_delayed_work(&uq_mgr->resume_work); |
| 480 | 480 | ||
| 481 | mutex_lock(&uq_mgr->userq_mutex); | 481 | mutex_lock(&uq_mgr->userq_mutex); |
| 482 | spin_lock(&evf_mgr->ev_fence_lock); | 482 | ev_fence = amdgpu_evf_mgr_get_fence(evf_mgr); |
| 483 | ev_fence = evf_mgr->ev_fence; | 483 | if (dma_fence_is_signaled(ev_fence)) { |
| 484 | spin_unlock(&evf_mgr->ev_fence_lock); | 484 | dma_fence_put(ev_fence); |
| 485 | if (!ev_fence || dma_fence_is_signaled(&ev_fence->base)) { | ||
| 486 | mutex_unlock(&uq_mgr->userq_mutex); | 485 | mutex_unlock(&uq_mgr->userq_mutex); |
| 487 | /* | 486 | /* |
| 488 | * Looks like there was no pending resume work, | 487 | * Looks like there was no pending resume work, |
| @@ -491,6 +490,7 @@ retry: · 2 unmodified lines | |||
| 491 | schedule_delayed_work(&uq_mgr->resume_work, 0); | 490 | schedule_delayed_work(&uq_mgr->resume_work, 0); |
| 492 | goto retry; | 491 | goto retry; |
| 493 | } | 492 | } |
| 493 | dma_fence_put(ev_fence); | ||
| 494 | } | 494 | } |
| 495 | 495 | ||
| 496 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, | 496 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, |
| @@ -600,6 +600,13 @@ amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr, · 103 unmodified lines | |||
| 600 | goto unpin_bo; | 600 | goto unpin_bo; |
| 601 | } | 601 | } |
| 602 | 602 | ||
| 603 | /* Validate doorbell_offset is within the doorbell BO */ | ||
| 604 | if ((u64)db_info->doorbell_offset * db_size + db_size > | ||
| 605 | amdgpu_bo_size(db_obj->obj)) { | ||
| 606 | r = -EINVAL; | ||
| 607 | goto unpin_bo; | ||
| 608 | } | ||
| 609 | |||
| 603 | index = amdgpu_doorbell_index_on_bar(uq_mgr->adev, db_obj->obj, | 610 | index = amdgpu_doorbell_index_on_bar(uq_mgr->adev, db_obj->obj, |
| 604 | db_info->doorbell_offset, db_size); | 611 | db_info->doorbell_offset, db_size); |
| 605 | drm_dbg_driver(adev_to_drm(uq_mgr->adev), | 612 | drm_dbg_driver(adev_to_drm(uq_mgr->adev), |
| @@ -623,13 +630,14 @@ amdgpu_userq_destroy(struct amdgpu_userq_mgr *uq_mgr, struct amdgpu_usermode_que · 17 unmodified lines | |||
| 623 | int r = 0; | 630 | int r = 0; |
| 624 | 631 | ||
| 625 | cancel_delayed_work_sync(&uq_mgr->resume_work); | 632 | cancel_delayed_work_sync(&uq_mgr->resume_work); |
| 633 | |||
| 634 | /* Cancel any pending hang detection work and cleanup */ | ||
| 635 | cancel_delayed_work_sync(&queue->hang_detect_work); | ||
| 636 | |||
| 626 | mutex_lock(&uq_mgr->userq_mutex); | 637 | mutex_lock(&uq_mgr->userq_mutex); |
| 638 | queue->hang_detect_fence = NULL; | ||
| 627 | amdgpu_userq_wait_for_last_fence(queue); | 639 | amdgpu_userq_wait_for_last_fence(queue); |
| 628 | /* Cancel any pending hang detection work and cleanup */ | 640 | |
| 629 | if (queue->hang_detect_fence) { | ||
| 630 | cancel_delayed_work_sync(&queue->hang_detect_work); | ||
| 631 | queue->hang_detect_fence = NULL; | ||
| 632 | } | ||
| 633 | r = amdgpu_bo_reserve(queue->db_obj.obj, true); | 641 | r = amdgpu_bo_reserve(queue->db_obj.obj, true); |
| 634 | if (!r) { | 642 | if (!r) { |
| 635 | amdgpu_bo_unpin(queue->db_obj.obj); | 643 | amdgpu_bo_unpin(queue->db_obj.obj); |
| @@ -996,17 +1004,14 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · 360 unmodified lines | |||
| 996 | unsigned long queue_id; | 1004 | unsigned long queue_id; |
| 997 | int ret = 0, r; | 1005 | int ret = 0, r; |
| 998 | 1006 | ||
| 1007 | mutex_lock(&uq_mgr->userq_mutex); | ||
| 999 | /* Resume all the queues for this process */ | 1008 | /* Resume all the queues for this process */ |
| 1000 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { | 1009 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { |
| 1001 | queue = amdgpu_userq_get(uq_mgr, queue_id); | ||
| 1002 | if (!queue) | ||
| 1003 | continue; | ||
| 1004 | 1010 | ||
| 1005 | if (!amdgpu_userq_buffer_vas_mapped(queue)) { | 1011 | if (!amdgpu_userq_buffer_vas_mapped(queue)) { |
| 1006 | drm_file_err(uq_mgr->file, | 1012 | drm_file_err(uq_mgr->file, |
| 1007 | "trying restore queue without va mapping\n"); | 1013 | "trying restore queue without va mapping\n"); |
| 1008 | queue->state = AMDGPU_USERQ_STATE_INVALID_VA; | 1014 | queue->state = AMDGPU_USERQ_STATE_INVALID_VA; |
| 1009 | amdgpu_userq_put(queue); | ||
| 1010 | continue; | 1015 | continue; |
| 1011 | } | 1016 | } |
| 1012 | 1017 | ||
| @@ -1014,8 +1019,8 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · 1 unmodified line | |||
| 1014 | if (r) | 1019 | if (r) |
| 1015 | ret = r; | 1020 | ret = r; |
| 1016 | 1021 | ||
| 1017 | amdgpu_userq_put(queue); | ||
| 1018 | } | 1022 | } |
| 1023 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 1019 | 1024 | ||
| 1020 | if (ret) | 1025 | if (ret) |
| 1021 | drm_file_err(uq_mgr->file, "Failed to map all the queues\n"); | 1026 | drm_file_err(uq_mgr->file, "Failed to map all the queues\n"); |
| @@ -1040,12 +1045,12 @@ amdgpu_userq_bo_validate(struct amdgpu_device *adev, struct drm_exec *exec, · 18 unmodified lines | |||
| 1040 | struct amdgpu_bo *bo; | 1045 | struct amdgpu_bo *bo; |
| 1041 | int ret; | 1046 | int ret; |
| 1042 | 1047 | ||
| 1043 | spin_lock(&vm->invalidated_lock); | 1048 | spin_lock(&vm->status_lock); |
| 1044 | while (!list_empty(&vm->invalidated)) { | 1049 | while (!list_empty(&vm->invalidated)) { |
| 1045 | bo_va = list_first_entry(&vm->invalidated, | 1050 | bo_va = list_first_entry(&vm->invalidated, |
| 1046 | struct amdgpu_bo_va, | 1051 | struct amdgpu_bo_va, |
| 1047 | base.vm_status); | 1052 | base.vm_status); |
| 1048 | spin_unlock(&vm->invalidated_lock); | 1053 | spin_unlock(&vm->status_lock); |
| 1049 | 1054 | ||
| 1050 | bo = bo_va->base.bo; | 1055 | bo = bo_va->base.bo; |
| 1051 | ret = drm_exec_prepare_obj(exec, &bo->tbo.base, 2); | 1056 | ret = drm_exec_prepare_obj(exec, &bo->tbo.base, 2); |
| @@ -1062,9 +1067,9 @@ amdgpu_userq_bo_validate(struct amdgpu_device *adev, struct drm_exec *exec, · 10 unmodified lines | |||
| 1062 | if (ret) | 1067 | if (ret) |
| 1063 | return ret; | 1068 | return ret; |
| 1064 | 1069 | ||
| 1065 | spin_lock(&vm->invalidated_lock); | 1070 | spin_lock(&vm->status_lock); |
| 1066 | } | 1071 | } |
| 1067 | spin_unlock(&vm->invalidated_lock); | 1072 | spin_unlock(&vm->status_lock); |
| 1068 | 1073 | ||
| 1069 | return 0; | 1074 | return 0; |
| 1070 | } | 1075 | } |
| @@ -1196,7 +1201,7 @@ retry_lock: · 125 unmodified lines | |||
| 1196 | dma_fence_wait(bo_va->last_pt_update, false); | 1201 | dma_fence_wait(bo_va->last_pt_update, false); |
| 1197 | dma_fence_wait(vm->last_update, false); | 1202 | dma_fence_wait(vm->last_update, false); |
| 1198 | 1203 | ||
| 1199 | ret = amdgpu_eviction_fence_replace_fence(&fpriv->evf_mgr, &exec); | 1204 | ret = amdgpu_evf_mgr_rearm(&fpriv->evf_mgr, &exec); |
| 1200 | if (ret) | 1205 | if (ret) |
| 1201 | drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n"); | 1206 | drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n"); |
| 1202 | 1207 | ||
| @@ -1216,26 +1221,25 @@ static void amdgpu_userq_restore_worker(struct work_struct *work) · 13 unmodified lines | |||
| 1216 | { | 1221 | { |
| 1217 | struct amdgpu_userq_mgr *uq_mgr = work_to_uq_mgr(work, resume_work.work); | 1222 | struct amdgpu_userq_mgr *uq_mgr = work_to_uq_mgr(work, resume_work.work); |
| 1218 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); | 1223 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); |
| 1224 | struct dma_fence *ev_fence; | ||
| 1219 | int ret; | 1225 | int ret; |
| 1220 | 1226 | ||
| 1221 | flush_delayed_work(&fpriv->evf_mgr.suspend_work); | 1227 | ev_fence = amdgpu_evf_mgr_get_fence(&fpriv->evf_mgr); |
| 1222 | 1228 | if (!dma_fence_is_signaled(ev_fence)) | |
| 1223 | mutex_lock(&uq_mgr->userq_mutex); | 1229 | goto put_fence; |
| 1224 | 1230 | ||
| 1225 | ret = amdgpu_userq_vm_validate(uq_mgr); | 1231 | ret = amdgpu_userq_vm_validate(uq_mgr); |
| 1226 | if (ret) { | 1232 | if (ret) { |
| 1227 | drm_file_err(uq_mgr->file, "Failed to validate BOs to restore\n"); | 1233 | drm_file_err(uq_mgr->file, "Failed to validate BOs to restore\n"); |
| 1228 | goto unlock; | 1234 | goto put_fence; |
| 1229 | } | 1235 | } |
| 1230 | 1236 | ||
| 1231 | ret = amdgpu_userq_restore_all(uq_mgr); | 1237 | ret = amdgpu_userq_restore_all(uq_mgr); |
| 1232 | if (ret) { | 1238 | if (ret) |
| 1233 | drm_file_err(uq_mgr->file, "Failed to restore all queues\n"); | 1239 | drm_file_err(uq_mgr->file, "Failed to restore all queues\n"); |
| 1234 | goto unlock; | ||
| 1235 | } | ||
| 1236 | 1240 | ||
| 1237 | unlock: | 1241 | put_fence: |
| 1238 | mutex_unlock(&uq_mgr->userq_mutex); | 1242 | dma_fence_put(ev_fence); |
| 1239 | } | 1243 | } |
| 1240 | 1244 | ||
| 1241 | static int | 1245 | static int |
| @@ -1248,13 +1252,9 @@ amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr) · 6 unmodified lines | |||
| 1248 | amdgpu_userq_detect_and_reset_queues(uq_mgr); | 1252 | amdgpu_userq_detect_and_reset_queues(uq_mgr); |
| 1249 | /* Try to unmap all the queues in this process ctx */ | 1253 | /* Try to unmap all the queues in this process ctx */ |
| 1250 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { | 1254 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { |
| 1251 | queue = amdgpu_userq_get(uq_mgr, queue_id); | ||
| 1252 | if (!queue) | ||
| 1253 | continue; | ||
| 1254 | r = amdgpu_userq_preempt_helper(queue); | 1255 | r = amdgpu_userq_preempt_helper(queue); |
| 1255 | if (r) | 1256 | if (r) |
| 1256 | ret = r; | 1257 | ret = r; |
| 1257 | amdgpu_userq_put(queue); | ||
| 1258 | } | 1258 | } |
| 1259 | 1259 | ||
| 1260 | if (ret) | 1260 | if (ret) |
| @@ -1287,35 +1287,26 @@ amdgpu_userq_wait_for_signal(struct amdgpu_userq_mgr *uq_mgr) · 26 unmodified lines | |||
| 1287 | int ret; | 1287 | int ret; |
| 1288 | 1288 | ||
| 1289 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { | 1289 | xa_for_each(&uq_mgr->userq_xa, queue_id, queue) { |
| 1290 | queue = amdgpu_userq_get(uq_mgr, queue_id); | ||
| 1291 | if (!queue) | ||
| 1292 | continue; | ||
| 1293 | |||
| 1294 | struct dma_fence *f = queue->last_fence; | 1290 | struct dma_fence *f = queue->last_fence; |
| 1295 | 1291 | ||
| 1296 | if (!f || dma_fence_is_signaled(f)) { | 1292 | if (!f || dma_fence_is_signaled(f)) |
| 1297 | amdgpu_userq_put(queue); | ||
| 1298 | continue; | 1293 | continue; |
| 1299 | } | 1294 | |
| 1300 | ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100)); | 1295 | ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100)); |
| 1301 | if (ret <= 0) { | 1296 | if (ret <= 0) { |
| 1302 | drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n", | 1297 | drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n", |
| 1303 | f->context, f->seqno); | 1298 | f->context, f->seqno); |
| 1304 | amdgpu_userq_put(queue); | 1299 | |
| 1305 | return -ETIMEDOUT; | 1300 | return -ETIMEDOUT; |
| 1306 | } | 1301 | } |
| 1307 | amdgpu_userq_put(queue); | ||
| 1308 | } | 1302 | } |
| 1309 | 1303 | ||
| 1310 | return 0; | 1304 | return 0; |
| 1311 | } | 1305 | } |
| 1312 | 1306 | ||
| 1313 | void | 1307 | void |
| 1314 | amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, | 1308 | amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr) |
| 1315 | struct amdgpu_eviction_fence *ev_fence) | ||
| 1316 | { | 1309 | { |
| 1317 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); | ||
| 1318 | struct amdgpu_eviction_fence_mgr *evf_mgr = &fpriv->evf_mgr; | ||
| 1319 | struct amdgpu_device *adev = uq_mgr->adev; | 1310 | struct amdgpu_device *adev = uq_mgr->adev; |
| 1320 | int ret; | 1311 | int ret; |
| 1321 | 1312 | ||
| @@ -1328,16 +1319,6 @@ amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, · 6 unmodified lines | |||
| 1328 | if (ret) | 1319 | if (ret) |
| 1329 | dev_err(adev->dev, "Failed to evict userqueue\n"); | 1320 | dev_err(adev->dev, "Failed to evict userqueue\n"); |
| 1330 | 1321 | ||
| 1331 | /* Signal current eviction fence */ | ||
| 1332 | amdgpu_eviction_fence_signal(evf_mgr, ev_fence); | ||
| 1333 | |||
| 1334 | if (evf_mgr->fd_closing) { | ||
| 1335 | cancel_delayed_work_sync(&uq_mgr->resume_work); | ||
| 1336 | return; | ||
| 1337 | } | ||
| 1338 | |||
| 1339 | /* Schedule a resume work */ | ||
| 1340 | schedule_delayed_work(&uq_mgr->resume_work, 0); | ||
| 1341 | } | 1322 | } |
| 1342 | 1323 | ||
| 1343 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, | 1324 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, |
| @@ -1352,6 +1333,11 @@ int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *f · 8 unmodified lines | |||
| 1352 | return 0; | 1333 | return 0; |
| 1353 | } | 1334 | } |
| 1354 | 1335 | ||
| 1336 | void amdgpu_userq_mgr_cancel_resume(struct amdgpu_userq_mgr *userq_mgr) | ||
| 1337 | { | ||
| 1338 | cancel_delayed_work_sync(&userq_mgr->resume_work); | ||
| 1339 | } | ||
| 1340 | |||
| 1355 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr) | 1341 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr) |
| 1356 | { | 1342 | { |
| 1357 | struct amdgpu_usermode_queue *queue; | 1343 | struct amdgpu_usermode_queue *queue; |
| @@ -1475,17 +1461,19 @@ int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev, · 117 unmodified lines | |||
| 1475 | 1461 | ||
| 1476 | if (!adev->userq_halt_for_enforce_isolation) | 1462 | if (!adev->userq_halt_for_enforce_isolation) |
| 1477 | dev_warn(adev->dev, "userq scheduling already started!\n"); | 1463 | dev_warn(adev->dev, "userq scheduling already started!\n"); |
| 1464 | |||
| 1478 | adev->userq_halt_for_enforce_isolation = false; | 1465 | adev->userq_halt_for_enforce_isolation = false; |
| 1466 | |||
| 1479 | xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) { | 1467 | xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) { |
| 1480 | uqm = queue->userq_mgr; | 1468 | uqm = queue->userq_mgr; |
| 1481 | mutex_lock(&uqm->userq_mutex); | 1469 | mutex_lock(&uqm->userq_mutex); |
| 1482 | if (((queue->queue_type == AMDGPU_HW_IP_GFX) || | 1470 | if (((queue->queue_type == AMDGPU_HW_IP_GFX) || |
| 1483 | (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) && | 1471 | (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) && |
| 1484 | (queue->xcp_id == idx)) { | 1472 | (queue->xcp_id == idx)) { |
| 1485 | r = amdgpu_userq_restore_helper(queue); | 1473 | r = amdgpu_userq_restore_helper(queue); |
| 1486 | if (r) | 1474 | if (r) |
| 1487 | ret = r; | 1475 | ret = r; |
| 1488 | } | 1476 | } |
| 1489 | mutex_unlock(&uqm->userq_mutex); | 1477 | mutex_unlock(&uqm->userq_mutex); |
| 1490 | } | 1478 | } |
| 1491 | 1479 | ||
| @@ -123,6 +123,7 @@ int amdgpu_userq_ioctl(struct drm_device *dev, void *data, struct drm_file *filp · 122 unmodified lines | |||
| 123 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, | 123 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, |
| 124 | struct amdgpu_device *adev); | 124 | struct amdgpu_device *adev); |
| 125 | 125 | ||
| 126 | void amdgpu_userq_mgr_cancel_resume(struct amdgpu_userq_mgr *userq_mgr); | ||
| 126 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr); | 127 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr); |
| 127 | 128 | ||
| 128 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, | 129 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, |
| @@ -132,8 +133,7 @@ int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, · 3 unmodified lines | |||
| 132 | void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr, | 133 | void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr, |
| 133 | struct amdgpu_userq_obj *userq_obj); | 134 | struct amdgpu_userq_obj *userq_obj); |
| 134 | 135 | ||
| 135 | void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, | 136 | void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr); |
| 136 | struct amdgpu_eviction_fence *ev_fence); | ||
| 137 | 137 | ||
| 138 | void amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *userq_mgr, | 138 | void amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *userq_mgr, |
| 139 | struct amdgpu_eviction_fence_mgr *evf_mgr); | 139 | struct amdgpu_eviction_fence_mgr *evf_mgr); |
| @@ -597,11 +597,11 @@ exec_fini: · 596 unmodified lines | |||
| 597 | put_gobj_write: | 597 | put_gobj_write: |
| 598 | for (i = 0; i < num_write_bo_handles; i++) | 598 | for (i = 0; i < num_write_bo_handles; i++) |
| 599 | drm_gem_object_put(gobj_write[i]); | 599 | drm_gem_object_put(gobj_write[i]); |
| 600 | kfree(gobj_write); | 600 | kvfree(gobj_write); |
| 601 | put_gobj_read: | 601 | put_gobj_read: |
| 602 | for (i = 0; i < num_read_bo_handles; i++) | 602 | for (i = 0; i < num_read_bo_handles; i++) |
| 603 | drm_gem_object_put(gobj_read[i]); | 603 | drm_gem_object_put(gobj_read[i]); |
| 604 | kfree(gobj_read); | 604 | kvfree(gobj_read); |
| 605 | free_syncobj: | 605 | free_syncobj: |
| 606 | while (entry-- > 0) | 606 | while (entry-- > 0) |
| 607 | if (syncobj[entry]) | 607 | if (syncobj[entry]) |
| @@ -616,339 +616,405 @@ free_syncobj_handles: · 8 unmodified lines | |||
| 616 | return r; | 616 | return r; |
| 617 | } | 617 | } |
| 618 | 618 | ||
| 619 | int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data, | 619 | /* Count the number of expected fences so userspace can alloc a buffer */ |
| 620 | struct drm_file *filp) | 620 | static int |
| 621 | amdgpu_userq_wait_count_fences(struct drm_file *filp, | ||
| 622 | struct drm_amdgpu_userq_wait *wait_info, | ||
| 623 | u32 *syncobj_handles, u32 *timeline_points, | ||
| 624 | u32 *timeline_handles, | ||
| 625 | struct drm_gem_object **gobj_write, | ||
| 626 | struct drm_gem_object **gobj_read) | ||
| 621 | { | 627 | { |
| 622 | struct drm_amdgpu_userq_wait *wait_info = data; | 628 | int num_read_bo_handles, num_write_bo_handles; |
| 623 | const unsigned int num_write_bo_handles = wait_info->num_bo_write_handles; | 629 | struct dma_fence_unwrap iter; |
| 624 | const unsigned int num_read_bo_handles = wait_info->num_bo_read_handles; | 630 | struct dma_fence *fence, *f; |
| 625 | struct drm_amdgpu_userq_fence_info *fence_info = NULL; | 631 | unsigned int num_fences = 0; |
| 626 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | ||
| 627 | struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr; | ||
| 628 | struct drm_gem_object **gobj_write, **gobj_read; | ||
| 629 | u32 *timeline_points, *timeline_handles; | ||
| 630 | struct amdgpu_usermode_queue *waitq = NULL; | ||
| 631 | u32 *syncobj_handles, num_syncobj; | ||
| 632 | struct dma_fence **fences = NULL; | ||
| 633 | u16 num_points, num_fences = 0; | ||
| 634 | struct drm_exec exec; | 632 | struct drm_exec exec; |
| 635 | int r, i, cnt; | 633 | int i, r; |
| 636 | 634 | ||
| 637 | if (!amdgpu_userq_enabled(dev)) | 635 | /* |
| 638 | return -ENOTSUPP; | 636 | * This needs to be outside of the lock provided by drm_exec for |
| 639 | 637 | * DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT to work correctly. | |
| 640 | if (wait_info->num_bo_write_handles > AMDGPU_USERQ_MAX_HANDLES || | 638 | */ |
| 641 | wait_info->num_bo_read_handles > AMDGPU_USERQ_MAX_HANDLES) | 639 | |
| 642 | return -EINVAL; | 640 | /* Count timeline fences */ |
| 643 | 641 | for (i = 0; i < wait_info->num_syncobj_timeline_handles; i++) { | |
| 644 | num_syncobj = wait_info->num_syncobj_handles; | 642 | r = drm_syncobj_find_fence(filp, timeline_handles[i], |
| 645 | syncobj_handles = memdup_array_user(u64_to_user_ptr(wait_info->syncobj_handles), | 643 | timeline_points[i], |
| 646 | num_syncobj, sizeof(u32)); | 644 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, |
| 647 | if (IS_ERR(syncobj_handles)) | 645 | &fence); |
| 648 | return PTR_ERR(syncobj_handles); | 646 | if (r) |
| 649 | 647 | return r; | |
| 648 | |||
| 649 | dma_fence_unwrap_for_each(f, &iter, fence) | ||
| 650 | num_fences++; | ||
| 650 | 651 | ||
| 651 | num_points = wait_info->num_syncobj_timeline_handles; | 652 | dma_fence_put(fence); |
| 652 | timeline_handles = memdup_array_user(u64_to_user_ptr(wait_info->syncobj_timeline_handles), | ||
| 653 | num_points, sizeof(u32)); | ||
| 654 | if (IS_ERR(timeline_handles)) { | ||
| 655 | r = PTR_ERR(timeline_handles); | ||
| 656 | goto free_syncobj_handles; | ||
| 657 | } | 653 | } |
| 658 | 654 | ||
| 659 | timeline_points = memdup_array_user(u64_to_user_ptr(wait_info->syncobj_timeline_points), | 655 | /* Count boolean fences */ |
| 660 | num_points, sizeof(u32)); | 656 | for (i = 0; i < wait_info->num_syncobj_handles; i++) { |
| 657 | r = drm_syncobj_find_fence(filp, syncobj_handles[i], 0, | ||
| 658 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | ||
| 659 | &fence); | ||
| 660 | if (r) | ||
| 661 | return r; | ||
| 661 | 662 | ||
| 662 | if (IS_ERR(timeline_points)) { | 663 | num_fences++; |
| 663 | r = PTR_ERR(timeline_points); | 664 | dma_fence_put(fence); |
| 664 | goto free_timeline_handles; | ||
| 665 | } | 665 | } |
| 666 | 666 | ||
| 667 | r = drm_gem_objects_lookup(filp, | 667 | /* Lock all the GEM objects */ |
| 668 | u64_to_user_ptr(wait_info->bo_read_handles), | 668 | /* TODO: It is actually not necessary to lock them */ |
| 669 | num_read_bo_handles, | 669 | num_read_bo_handles = wait_info->num_bo_read_handles; |
| 670 | &gobj_read); | 670 | num_write_bo_handles = wait_info->num_bo_write_handles; |
| 671 | if (r) | ||
| 672 | goto free_timeline_points; | ||
| 673 | |||
| 674 | r = drm_gem_objects_lookup(filp, | ||
| 675 | u64_to_user_ptr(wait_info->bo_write_handles), | ||
| 676 | num_write_bo_handles, | ||
| 677 | &gobj_write); | ||
| 678 | if (r) | ||
| 679 | goto put_gobj_read; | ||
| 680 | |||
| 681 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, | 671 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, |
| 682 | (num_read_bo_handles + num_write_bo_handles)); | 672 | num_read_bo_handles + num_write_bo_handles); |
| 683 | 673 | ||
| 684 | /* Lock all BOs with retry handling */ | ||
| 685 | drm_exec_until_all_locked(&exec) { | 674 | drm_exec_until_all_locked(&exec) { |
| 686 | r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1); | 675 | r = drm_exec_prepare_array(&exec, gobj_read, |
| 676 | num_read_bo_handles, 1); | ||
| 687 | drm_exec_retry_on_contention(&exec); | 677 | drm_exec_retry_on_contention(&exec); |
| 688 | if (r) { | 678 | if (r) |
| 689 | drm_exec_fini(&exec); | 679 | goto error_unlock; |
| 690 | goto put_gobj_write; | ||
| 691 | } | ||
| 692 | 680 | ||
| 693 | r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1); | 681 | r = drm_exec_prepare_array(&exec, gobj_write, |
| 682 | num_write_bo_handles, 1); | ||
| 694 | drm_exec_retry_on_contention(&exec); | 683 | drm_exec_retry_on_contention(&exec); |
| 695 | if (r) { | 684 | if (r) |
| 696 | drm_exec_fini(&exec); | 685 | goto error_unlock; |
| 697 | goto put_gobj_write; | ||
| 698 | } | ||
| 699 | } | 686 | } |
| 700 | 687 | ||
| 701 | if (!wait_info->num_fences) { | 688 | /* Count read fences */ |
| 702 | if (num_points) { | 689 | for (i = 0; i < num_read_bo_handles; i++) { |
| 703 | struct dma_fence_unwrap iter; | 690 | struct dma_resv_iter resv_cursor; |
| 704 | struct dma_fence *fence; | 691 | struct dma_fence *fence; |
| 705 | struct dma_fence *f; | ||
| 706 | |||
| 707 | for (i = 0; i < num_points; i++) { | ||
| 708 | r = drm_syncobj_find_fence(filp, timeline_handles[i], | ||
| 709 | timeline_points[i], | ||
| 710 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | ||
| 711 | &fence); | ||
| 712 | if (r) | ||
| 713 | goto exec_fini; | ||
| 714 | |||
| 715 | dma_fence_unwrap_for_each(f, &iter, fence) | ||
| 716 | num_fences++; | ||
| 717 | |||
| 718 | dma_fence_put(fence); | ||
| 719 | } | ||
| 720 | } | ||
511 more lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 548 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 174 lines aren’t shown here.
This diff is large: its 383 lines aren’t shown here.
This diff is large: its 238 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 164 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 841 lines aren’t shown here.
This diff is large: its 112 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 473 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 1220 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 143 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 336 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 599 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 407 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 303 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 160 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 424 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 382 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 159 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 149 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 134 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 150 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 202 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 296 lines aren’t shown here.
This diff is large: its 428 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 190 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 305 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 2271 lines aren’t shown here.
This diff is large: its 314 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 245 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 599 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 192 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 209 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 145 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 266 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 4 lines aren’t shown here.
This diff is large: its 4 lines aren’t shown here.
This diff is large: its 4 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 180 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 496 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 179 lines aren’t shown here.
This diff is large: its 149 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 215 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 413 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 316 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 148 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 121 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 232 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 301 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 369 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 292 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 292 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 364 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 442 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 750 lines aren’t shown here.
This diff is large: its 135 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 295 lines aren’t shown here.
This diff is large: its 324 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 302 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 288 lines aren’t shown here.
This diff is large: its 201 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 317 lines aren’t shown here.
This diff is large: its 660 lines aren’t shown here.
This diff is large: its 409 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 181 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 287 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 190 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 448 lines aren’t shown here.
This diff is large: its 174 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 92 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 228 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 328 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 1002 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 127 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 211 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 308 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 291 lines aren’t shown here.
This diff is large: its 351 lines aren’t shown here.
This diff is large: its 170 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 211 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 757 lines aren’t shown here.
This diff is large: its 308 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 586 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 715 lines aren’t shown here.
This diff is large: its 740 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 203 lines aren’t shown here.
This diff is large: its 280 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 540 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 255 lines aren’t shown here.
This diff is large: its 299 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 302 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 157 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 485 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 174 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 301 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 135 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 189 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 213 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 112 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 216 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 303 lines aren’t shown here.
This diff is large: its 184 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 237 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 176 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 481 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 181 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 178 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 269 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 142 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 185 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 586 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 327 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 201 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 256 lines aren’t shown here.
This diff is large: its 288 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 288 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 364 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 461 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 286 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 333 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 322 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 1163 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 229 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 615 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 340 lines aren’t shown here.
This diff is large: its 948 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 1261 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 306 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 154 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 156 lines aren’t shown here.
This diff is large: its 2 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 548 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 162 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 194 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 266 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 179 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 2 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 866 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 353 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 151 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 105 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 195 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.